2024-03-30T03:59:48.281 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-30T03:59:48.281 DEBUG:teuthology.run:Teuthology command: teuthology --name rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi --verbose --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} --archive /home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629473 --owner scheduled_rishabh@teuthology -- /home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629473/orig.config.yaml 2024-03-30T03:59:48.336 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-30T03:59:48.440 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629473 branch: wip-rishabh-testing-20240327.051042-reef description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: null first_in_suite: false job_id: '7629473' kernel: flavor: default kdb: true sha1: 1df0c7e8a2354c42d738e5454ba83ea7332deac2 last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/quincy ' name: rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-rishabh-testing-20240327.051042-reef ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - OSD_DOWN - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 470100a4ccbcf7769052bf09a5413881ccf1b97f ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 470100a4ccbcf7769052bf09a5413881ccf1b97f kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-rishabh-testing-20240327.051042-reef sha1: 470100a4ccbcf7769052bf09a5413881ccf1b97f owner: scheduled_rishabh@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 8052 sha1: 470100a4ccbcf7769052bf09a5413881ccf1b97f sleep_before_teardown: 0 subset: 497/512 suite: fs suite_branch: wip-rishabh-testing-20240327.051042-reef suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_470100a4ccbcf7769052bf09a5413881ccf1b97f/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 470100a4ccbcf7769052bf09a5413881ccf1b97f targets: smithi123.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL84LCzYPCo3lePnhqemzPds+HKP3NxrBEFTSpPOsTCV3NO0gMHX/66ja2Ywr7NGE8XNXFLYQm8xOm1D+BY4hgs= smithi135.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCvGdlm5XODb+4nlZVWzc1Z3AkK8rNHpQllvV8Q7wuxG/Kk3+oIkPF97vCgUgdi2GPWH4MMiGsU+M9UZ74G+SB4= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: branch: quincy exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: cephadm_branch: quincy cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:quincy roleless: true - print: '**** done end installing quincy cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data false - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - ceph-fuse: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] variables: fail_fs: true teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-29_18:05:24 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch ps - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-03-30T03:59:48.440 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_470100a4ccbcf7769052bf09a5413881ccf1b97f/qa; will attempt to use it 2024-03-30T03:59:48.441 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_470100a4ccbcf7769052bf09a5413881ccf1b97f/qa/tasks 2024-03-30T03:59:48.441 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-30T03:59:48.442 INFO:teuthology.task.internal:Checking packages... 2024-03-30T03:59:48.467 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '470100a4ccbcf7769052bf09a5413881ccf1b97f' 2024-03-30T03:59:48.467 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-30T03:59:48.468 INFO:teuthology.packaging:ref: None 2024-03-30T03:59:48.468 INFO:teuthology.packaging:tag: None 2024-03-30T03:59:48.468 INFO:teuthology.packaging:branch: wip-rishabh-testing-20240327.051042-reef 2024-03-30T03:59:48.468 INFO:teuthology.packaging:sha1: 470100a4ccbcf7769052bf09a5413881ccf1b97f 2024-03-30T03:59:48.468 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-rishabh-testing-20240327.051042-reef 2024-03-30T03:59:48.610 INFO:teuthology.task.internal:Found packages for ceph version 18.2.2-1018.g470100a4 2024-03-30T03:59:48.612 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-30T03:59:48.618 INFO:teuthology.task.internal:no buildpackages task found 2024-03-30T03:59:48.618 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-30T03:59:48.681 INFO:teuthology.task.internal:Saving configuration 2024-03-30T03:59:48.701 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-30T03:59:48.708 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-30T03:59:48.731 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi123.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629473', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-30 03:54:40.477689', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL84LCzYPCo3lePnhqemzPds+HKP3NxrBEFTSpPOsTCV3NO0gMHX/66ja2Ywr7NGE8XNXFLYQm8xOm1D+BY4hgs='} 2024-03-30T03:59:48.751 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi135.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629473', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-30 03:54:40.479399', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCvGdlm5XODb+4nlZVWzc1Z3AkK8rNHpQllvV8Q7wuxG/Kk3+oIkPF97vCgUgdi2GPWH4MMiGsU+M9UZ74G+SB4='} 2024-03-30T03:59:48.751 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-30T03:59:48.757 INFO:teuthology.task.internal:roles: ubuntu@smithi123.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-03-30T03:59:48.757 INFO:teuthology.task.internal:roles: ubuntu@smithi135.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-03-30T03:59:48.757 INFO:teuthology.run_tasks:Running task console_log... 2024-03-30T03:59:48.824 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fee28c68160>, signals=[15]) 2024-03-30T03:59:48.824 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-30T03:59:48.831 INFO:teuthology.task.internal:Opening connections... 2024-03-30T03:59:48.831 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi123.front.sepia.ceph.com 2024-03-30T03:59:48.833 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T03:59:48.909 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi135.front.sepia.ceph.com 2024-03-30T03:59:48.910 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi135.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T03:59:48.988 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-30T03:59:48.996 DEBUG:teuthology.orchestra.run.smithi123:> uname -m 2024-03-30T03:59:49.019 INFO:teuthology.orchestra.run.smithi123.stdout:x86_64 2024-03-30T03:59:49.019 DEBUG:teuthology.orchestra.run.smithi123:> cat /etc/os-release 2024-03-30T03:59:49.080 INFO:teuthology.orchestra.run.smithi123.stdout:NAME="CentOS Stream" 2024-03-30T03:59:49.081 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION="8" 2024-03-30T03:59:49.081 INFO:teuthology.orchestra.run.smithi123.stdout:ID="centos" 2024-03-30T03:59:49.081 INFO:teuthology.orchestra.run.smithi123.stdout:ID_LIKE="rhel fedora" 2024-03-30T03:59:49.081 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION_ID="8" 2024-03-30T03:59:49.081 INFO:teuthology.orchestra.run.smithi123.stdout:PLATFORM_ID="platform:el8" 2024-03-30T03:59:49.081 INFO:teuthology.orchestra.run.smithi123.stdout:PRETTY_NAME="CentOS Stream 8" 2024-03-30T03:59:49.081 INFO:teuthology.orchestra.run.smithi123.stdout:ANSI_COLOR="0;31" 2024-03-30T03:59:49.081 INFO:teuthology.orchestra.run.smithi123.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-03-30T03:59:49.081 INFO:teuthology.orchestra.run.smithi123.stdout:HOME_URL="https://centos.org/" 2024-03-30T03:59:49.081 INFO:teuthology.orchestra.run.smithi123.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-30T03:59:49.081 INFO:teuthology.orchestra.run.smithi123.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-03-30T03:59:49.081 INFO:teuthology.orchestra.run.smithi123.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-30T03:59:49.082 INFO:teuthology.lock.ops:Updating smithi123.front.sepia.ceph.com on lock server 2024-03-30T03:59:49.107 DEBUG:teuthology.orchestra.run.smithi135:> uname -m 2024-03-30T03:59:49.130 INFO:teuthology.orchestra.run.smithi135.stdout:x86_64 2024-03-30T03:59:49.130 DEBUG:teuthology.orchestra.run.smithi135:> cat /etc/os-release 2024-03-30T03:59:49.191 INFO:teuthology.orchestra.run.smithi135.stdout:NAME="CentOS Stream" 2024-03-30T03:59:49.191 INFO:teuthology.orchestra.run.smithi135.stdout:VERSION="8" 2024-03-30T03:59:49.191 INFO:teuthology.orchestra.run.smithi135.stdout:ID="centos" 2024-03-30T03:59:49.191 INFO:teuthology.orchestra.run.smithi135.stdout:ID_LIKE="rhel fedora" 2024-03-30T03:59:49.191 INFO:teuthology.orchestra.run.smithi135.stdout:VERSION_ID="8" 2024-03-30T03:59:49.191 INFO:teuthology.orchestra.run.smithi135.stdout:PLATFORM_ID="platform:el8" 2024-03-30T03:59:49.191 INFO:teuthology.orchestra.run.smithi135.stdout:PRETTY_NAME="CentOS Stream 8" 2024-03-30T03:59:49.191 INFO:teuthology.orchestra.run.smithi135.stdout:ANSI_COLOR="0;31" 2024-03-30T03:59:49.191 INFO:teuthology.orchestra.run.smithi135.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-03-30T03:59:49.191 INFO:teuthology.orchestra.run.smithi135.stdout:HOME_URL="https://centos.org/" 2024-03-30T03:59:49.192 INFO:teuthology.orchestra.run.smithi135.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-30T03:59:49.192 INFO:teuthology.orchestra.run.smithi135.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-03-30T03:59:49.192 INFO:teuthology.orchestra.run.smithi135.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-30T03:59:49.192 INFO:teuthology.lock.ops:Updating smithi135.front.sepia.ceph.com on lock server 2024-03-30T03:59:49.215 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-30T03:59:49.225 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-30T03:59:49.230 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-30T03:59:49.230 DEBUG:teuthology.orchestra.run.smithi123:> test '!' -e /home/ubuntu/cephtest 2024-03-30T03:59:49.233 DEBUG:teuthology.orchestra.run.smithi135:> test '!' -e /home/ubuntu/cephtest 2024-03-30T03:59:49.253 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-30T03:59:49.259 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-30T03:59:49.259 DEBUG:teuthology.orchestra.run.smithi123:> test -z $(ls -A /var/lib/ceph) 2024-03-30T03:59:49.295 DEBUG:teuthology.orchestra.run.smithi135:> test -z $(ls -A /var/lib/ceph) 2024-03-30T03:59:49.325 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-30T03:59:49.372 INFO:teuthology.run_tasks:Running task kernel... 2024-03-30T03:59:49.385 INFO:teuthology.task.kernel:normalize config orig: {'flavor': 'default', 'kdb': True, 'sha1': '1df0c7e8a2354c42d738e5454ba83ea7332deac2'} 2024-03-30T03:59:49.386 INFO:teuthology.task.kernel:config {'host.a': {'flavor': 'default', 'kdb': True, 'sha1': '1df0c7e8a2354c42d738e5454ba83ea7332deac2'}, 'host.b': {'flavor': 'default', 'kdb': True, 'sha1': '1df0c7e8a2354c42d738e5454ba83ea7332deac2'}}, timeout 300 2024-03-30T03:59:49.386 DEBUG:teuthology.orchestra.run.smithi123:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-30T03:59:49.387 DEBUG:teuthology.orchestra.run.smithi135:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-30T03:59:49.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T03:59:49.407 DEBUG:teuthology.task.kernel:sha1 for host.b is 1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-30T03:59:49.407 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "1df0c7e8a2354c42d738e5454ba83ea7332deac2"... 2024-03-30T03:59:49.407 DEBUG:teuthology.orchestra.run.smithi135:> uname -r 2024-03-30T03:59:49.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T03:59:49.408 DEBUG:teuthology.task.kernel:sha1 for host.a is 1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-30T03:59:49.408 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "1df0c7e8a2354c42d738e5454ba83ea7332deac2"... 2024-03-30T03:59:49.408 DEBUG:teuthology.orchestra.run.smithi123:> uname -r 2024-03-30T03:59:49.466 INFO:teuthology.orchestra.run.smithi135.stdout:4.18.0-489.el8.x86_64 2024-03-30T03:59:49.467 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-489.el8.x86_64 vs 1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-30T03:59:49.467 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-30T03:59:49.467 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-30T03:59:49.471 INFO:teuthology.orchestra.run.smithi123.stdout:4.18.0-489.el8.x86_64 2024-03-30T03:59:49.471 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-489.el8.x86_64 vs 1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-30T03:59:49.471 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-30T03:59:49.471 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-30T03:59:49.616 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum upgrade -y linux-firmware 2024-03-30T03:59:49.621 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum upgrade -y linux-firmware 2024-03-30T03:59:56.524 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 8 - AppStream 6.4 MB/s | 28 MB 00:04 2024-03-30T03:59:56.878 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 8 - AppStream 6.2 MB/s | 28 MB 00:04 2024-03-30T04:00:04.050 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 8 - BaseOS 4.0 MB/s | 10 MB 00:02 2024-03-30T04:00:06.311 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 8 - BaseOS 2.3 MB/s | 10 MB 00:04 2024-03-30T04:00:11.502 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 8 - Extras 4.9 kB/s | 18 kB 00:03 2024-03-30T04:00:14.012 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 8 - Extras common packages 3.1 kB/s | 7.5 kB 00:02 2024-03-30T04:00:14.239 INFO:teuthology.orchestra.run.smithi135.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2024-03-30T04:00:15.885 INFO:teuthology.orchestra.run.smithi135.stdout:Extra Packages for Enterprise Linux 10 MB/s | 16 MB 00:01 2024-03-30T04:00:19.699 INFO:teuthology.orchestra.run.smithi135.stdout:lab-extras 358 kB/s | 24 kB 00:00 2024-03-30T04:00:22.931 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-30T04:00:22.932 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-30T04:00:22.932 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2024-03-30T04:00:22.932 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-30T04:00:22.932 INFO:teuthology.orchestra.run.smithi135.stdout:Upgrading: 2024-03-30T04:00:22.932 INFO:teuthology.orchestra.run.smithi135.stdout: linux-firmware noarch 20240111-121.gitb3132c18.el8 baseos 355 M 2024-03-30T04:00:22.932 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:00:22.932 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-03-30T04:00:22.932 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-30T04:00:22.932 INFO:teuthology.orchestra.run.smithi135.stdout:Upgrade 1 Package 2024-03-30T04:00:22.932 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:00:22.933 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 355 M 2024-03-30T04:00:22.933 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-03-30T04:00:32.543 INFO:teuthology.orchestra.run.smithi135.stdout:linux-firmware-20240111-121.gitb3132c18.el8.noa 49 MB/s | 355 MB 00:07 2024-03-30T04:00:32.543 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-03-30T04:00:32.543 INFO:teuthology.orchestra.run.smithi135.stdout:Total 37 MB/s | 355 MB 00:09 2024-03-30T04:00:34.641 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-03-30T04:00:34.711 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-03-30T04:00:34.711 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-03-30T04:00:36.909 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-03-30T04:00:36.933 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-03-30T04:00:37.944 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 8 - Extras 564 B/s | 18 kB 00:32 2024-03-30T04:00:38.655 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 8 - Extras common packages 13 kB/s | 7.5 kB 00:00 2024-03-30T04:00:38.873 INFO:teuthology.orchestra.run.smithi123.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2024-03-30T04:00:40.001 INFO:teuthology.orchestra.run.smithi123.stdout:Extra Packages for Enterprise Linux 16 MB/s | 16 MB 00:01 2024-03-30T04:00:43.733 INFO:teuthology.orchestra.run.smithi123.stdout:lab-extras 439 kB/s | 24 kB 00:00 2024-03-30T04:00:46.916 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-30T04:00:46.916 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-30T04:00:46.916 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repo Size 2024-03-30T04:00:46.916 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-30T04:00:46.916 INFO:teuthology.orchestra.run.smithi123.stdout:Upgrading: 2024-03-30T04:00:46.916 INFO:teuthology.orchestra.run.smithi123.stdout: linux-firmware noarch 20240111-121.gitb3132c18.el8 baseos 355 M 2024-03-30T04:00:46.917 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:00:46.917 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-30T04:00:46.917 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-30T04:00:46.917 INFO:teuthology.orchestra.run.smithi123.stdout:Upgrade 1 Package 2024-03-30T04:00:46.917 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:00:46.917 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 355 M 2024-03-30T04:00:46.917 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-30T04:00:53.708 INFO:teuthology.orchestra.run.smithi123.stdout:linux-firmware-20240111-121.gitb3132c18.el8.noa 54 MB/s | 355 MB 00:06 2024-03-30T04:00:53.709 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-30T04:00:53.709 INFO:teuthology.orchestra.run.smithi123.stdout:Total 52 MB/s | 355 MB 00:06 2024-03-30T04:00:55.809 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-30T04:00:55.874 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-30T04:00:55.874 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-30T04:00:58.061 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-30T04:00:58.094 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-30T04:00:59.804 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-03-30T04:01:00.191 INFO:teuthology.orchestra.run.smithi135.stdout: Upgrading : linux-firmware-20240111-121.gitb3132c18.el8.noarch 1/2 2024-03-30T04:01:00.469 INFO:teuthology.orchestra.run.smithi135.stdout: Cleanup : linux-firmware-20230217-113.git83f1d778.el8.noarch 2/2 2024-03-30T04:01:00.710 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: linux-firmware-20230217-113.git83f1d778.el8.noarch 2/2 2024-03-30T04:01:00.710 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : linux-firmware-20240111-121.gitb3132c18.el8.noarch 1/2 2024-03-30T04:01:01.162 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : linux-firmware-20230217-113.git83f1d778.el8.noarch 2/2 2024-03-30T04:01:01.162 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:01:01.163 INFO:teuthology.orchestra.run.smithi135.stdout:Upgraded: 2024-03-30T04:01:01.163 INFO:teuthology.orchestra.run.smithi135.stdout: linux-firmware-20240111-121.gitb3132c18.el8.noarch 2024-03-30T04:01:01.163 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:01:01.163 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-30T04:01:01.396 INFO:teuthology.task.kernel:Downloading kernel 1df0c7e8a2354c42d738e5454ba83ea7332deac2 on host.b... 2024-03-30T04:01:01.397 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-30T04:01:01.549 INFO:teuthology.task.kernel:fetching, builder baseurl is https://1.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/8/flavors/default/x86_64/ 2024-03-30T04:01:01.549 DEBUG:teuthology.orchestra.run.smithi135:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_g1df0c7e8a235-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://1.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/8/flavors/default/x86_64/ --input-file=- 2024-03-30T04:01:01.552 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi135.front.sepia.ceph.com to complete... 2024-03-30T04:01:15.360 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30 04:01:15 URL:https://1.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/8/flavors/default/x86_64/kernel-6.8.0_g1df0c7e8a235-1.x86_64.rpm [401601944/401601944] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-30T04:01:15.360 INFO:teuthology.orchestra.run.smithi135.stderr:FINISHED --2024-03-30 04:01:15-- 2024-03-30T04:01:15.360 INFO:teuthology.orchestra.run.smithi135.stderr:Total wall clock time: 14s 2024-03-30T04:01:15.361 INFO:teuthology.orchestra.run.smithi135.stderr:Downloaded: 1 files, 383M in 13s (28.6 MB/s) 2024-03-30T04:01:15.363 INFO:teuthology.task.kernel:Installing kernel 1df0c7e8a2354c42d738e5454ba83ea7332deac2 on host.b... 2024-03-30T04:01:15.363 DEBUG:teuthology.orchestra.run.smithi135:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-30T04:01:17.093 INFO:teuthology.orchestra.run.smithi135.stdout:Verifying... ######################################## 2024-03-30T04:01:17.138 INFO:teuthology.orchestra.run.smithi135.stdout:Preparing... ######################################## 2024-03-30T04:01:17.247 INFO:teuthology.orchestra.run.smithi135.stdout:Updating / installing... 2024-03-30T04:01:20.754 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-30T04:01:21.098 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : linux-firmware-20240111-121.gitb3132c18.el8.noarch 1/2 2024-03-30T04:01:21.309 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : linux-firmware-20230217-113.git83f1d778.el8.noarch 2/2 2024-03-30T04:01:21.550 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: linux-firmware-20230217-113.git83f1d778.el8.noarch 2/2 2024-03-30T04:01:21.550 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : linux-firmware-20240111-121.gitb3132c18.el8.noarch 1/2 2024-03-30T04:01:22.078 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : linux-firmware-20230217-113.git83f1d778.el8.noarch 2/2 2024-03-30T04:01:22.078 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:01:22.078 INFO:teuthology.orchestra.run.smithi123.stdout:Upgraded: 2024-03-30T04:01:22.078 INFO:teuthology.orchestra.run.smithi123.stdout: linux-firmware-20240111-121.gitb3132c18.el8.noarch 2024-03-30T04:01:22.078 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:01:22.078 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-30T04:01:22.264 INFO:teuthology.task.kernel:Downloading kernel 1df0c7e8a2354c42d738e5454ba83ea7332deac2 on host.a... 2024-03-30T04:01:22.265 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-30T04:01:22.491 INFO:teuthology.task.kernel:fetching, builder baseurl is https://1.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/8/flavors/default/x86_64/ 2024-03-30T04:01:22.491 DEBUG:teuthology.orchestra.run.smithi123:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_g1df0c7e8a235-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://1.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/8/flavors/default/x86_64/ --input-file=- 2024-03-30T04:01:22.493 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi123.front.sepia.ceph.com to complete... 2024-03-30T04:01:36.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30 04:01:36 URL:https://1.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/8/flavors/default/x86_64/kernel-6.8.0_g1df0c7e8a235-1.x86_64.rpm [401601944/401601944] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-30T04:01:36.061 INFO:teuthology.orchestra.run.smithi123.stderr:FINISHED --2024-03-30 04:01:36-- 2024-03-30T04:01:36.061 INFO:teuthology.orchestra.run.smithi123.stderr:Total wall clock time: 14s 2024-03-30T04:01:36.061 INFO:teuthology.orchestra.run.smithi123.stderr:Downloaded: 1 files, 383M in 13s (28.9 MB/s) 2024-03-30T04:01:36.065 INFO:teuthology.task.kernel:Installing kernel 1df0c7e8a2354c42d738e5454ba83ea7332deac2 on host.a... 2024-03-30T04:01:36.065 DEBUG:teuthology.orchestra.run.smithi123:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-30T04:01:37.804 INFO:teuthology.orchestra.run.smithi123.stdout:Verifying... ######################################## 2024-03-30T04:01:37.848 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing... ######################################## 2024-03-30T04:01:37.957 INFO:teuthology.orchestra.run.smithi123.stdout:Updating / installing... 2024-03-30T04:01:57.554 INFO:teuthology.orchestra.run.smithi135.stdout:kernel-6.8.0_g1df0c7e8a235-1 ######################################## 2024-03-30T04:02:01.518 INFO:teuthology.orchestra.run.smithi135.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-30T04:02:18.195 INFO:teuthology.orchestra.run.smithi123.stdout:kernel-6.8.0_g1df0c7e8a235-1 ######################################## 2024-03-30T04:02:19.513 INFO:teuthology.orchestra.run.smithi123.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-30T04:02:21.624 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi135.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-30T04:02:21.625 DEBUG:teuthology.orchestra.run.smithi135:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-30T04:02:21.691 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235 2024-03-30T04:02:21.692 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/System.map 2024-03-30T04:02:21.692 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/config 2024-03-30T04:02:21.692 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel 2024-03-30T04:02:21.692 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch 2024-03-30T04:02:21.692 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86 2024-03-30T04:02:21.692 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto 2024-03-30T04:02:21.692 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-30T04:02:21.692 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-30T04:02:21.692 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-30T04:02:21.692 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-30T04:02:21.692 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-30T04:02:21.693 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-30T04:02:21.693 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-30T04:02:21.693 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-30T04:02:21.693 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-30T04:02:21.693 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-30T04:02:21.693 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-30T04:02:21.693 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-30T04:02:21.693 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-30T04:02:21.693 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-30T04:02:21.693 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-30T04:02:21.693 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-30T04:02:21.693 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-30T04:02:21.694 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-30T04:02:21.694 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-30T04:02:21.694 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-30T04:02:21.694 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel 2024-03-30T04:02:21.694 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu 2024-03-30T04:02:21.694 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu/mce 2024-03-30T04:02:21.695 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-30T04:02:21.695 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm 2024-03-30T04:02:21.695 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-30T04:02:21.695 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-30T04:02:21.695 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm.ko 2024-03-30T04:02:21.695 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto 2024-03-30T04:02:21.695 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/ansi_cprng.ko 2024-03-30T04:02:21.695 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/anubis.ko 2024-03-30T04:02:21.695 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/arc4.ko 2024-03-30T04:02:21.695 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx 2024-03-30T04:02:21.695 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_memcpy.ko 2024-03-30T04:02:21.695 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_pq.ko 2024-03-30T04:02:21.695 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-30T04:02:21.696 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_tx.ko 2024-03-30T04:02:21.696 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_xor.ko 2024-03-30T04:02:21.696 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/raid6test.ko 2024-03-30T04:02:21.696 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/authenc.ko 2024-03-30T04:02:21.696 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/authencesn.ko 2024-03-30T04:02:21.696 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blake2b_generic.ko 2024-03-30T04:02:21.696 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blowfish_common.ko 2024-03-30T04:02:21.696 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blowfish_generic.ko 2024-03-30T04:02:21.696 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/camellia_generic.ko 2024-03-30T04:02:21.696 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast5_generic.ko 2024-03-30T04:02:21.696 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast6_generic.ko 2024-03-30T04:02:21.697 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast_common.ko 2024-03-30T04:02:21.697 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/ccm.ko 2024-03-30T04:02:21.697 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cmac.ko 2024-03-30T04:02:21.697 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crc32_generic.ko 2024-03-30T04:02:21.697 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crypto_engine.ko 2024-03-30T04:02:21.697 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crypto_user.ko 2024-03-30T04:02:21.697 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cts.ko 2024-03-30T04:02:21.697 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/des_generic.ko 2024-03-30T04:02:21.697 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/dh_generic.ko 2024-03-30T04:02:21.697 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/echainiv.ko 2024-03-30T04:02:21.697 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/essiv.ko 2024-03-30T04:02:21.698 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/fcrypt.ko 2024-03-30T04:02:21.698 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/khazad.ko 2024-03-30T04:02:21.698 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/lz4.ko 2024-03-30T04:02:21.698 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/lz4hc.ko 2024-03-30T04:02:21.698 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/md4.ko 2024-03-30T04:02:21.698 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/michael_mic.ko 2024-03-30T04:02:21.698 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/pcbc.ko 2024-03-30T04:02:21.698 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/pcrypt.ko 2024-03-30T04:02:21.698 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/rmd160.ko 2024-03-30T04:02:21.699 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/seed.ko 2024-03-30T04:02:21.699 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/serpent_generic.ko 2024-03-30T04:02:21.699 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/tcrypt.ko 2024-03-30T04:02:21.699 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/tea.ko 2024-03-30T04:02:21.699 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/twofish_common.ko 2024-03-30T04:02:21.699 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/twofish_generic.ko 2024-03-30T04:02:21.699 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/vmac.ko 2024-03-30T04:02:21.699 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/wp512.ko 2024-03-30T04:02:21.699 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xcbc.ko 2024-03-30T04:02:21.699 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xor.ko 2024-03-30T04:02:21.699 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xxhash_generic.ko 2024-03-30T04:02:21.699 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers 2024-03-30T04:02:21.700 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi 2024-03-30T04:02:21.700 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-30T04:02:21.700 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/acpi_pad.ko 2024-03-30T04:02:21.700 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/custom_method.ko 2024-03-30T04:02:21.700 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/ec_sys.ko 2024-03-30T04:02:21.700 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/platform_profile.ko 2024-03-30T04:02:21.700 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/sbs.ko 2024-03-30T04:02:21.700 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/sbshc.ko 2024-03-30T04:02:21.701 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/video.ko 2024-03-30T04:02:21.701 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata 2024-03-30T04:02:21.701 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/acard-ahci.ko 2024-03-30T04:02:21.701 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/ahci_platform.ko 2024-03-30T04:02:21.701 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/ata_generic.ko 2024-03-30T04:02:21.701 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/libahci_platform.ko 2024-03-30T04:02:21.701 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_acpi.ko 2024-03-30T04:02:21.701 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ali.ko 2024-03-30T04:02:21.702 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_amd.ko 2024-03-30T04:02:21.702 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_artop.ko 2024-03-30T04:02:21.702 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_atiixp.ko 2024-03-30T04:02:21.702 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_atp867x.ko 2024-03-30T04:02:21.702 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cmd640.ko 2024-03-30T04:02:21.702 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cmd64x.ko 2024-03-30T04:02:21.702 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cypress.ko 2024-03-30T04:02:21.702 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_efar.ko 2024-03-30T04:02:21.703 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt366.ko 2024-03-30T04:02:21.703 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt37x.ko 2024-03-30T04:02:21.703 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-30T04:02:21.703 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-30T04:02:21.703 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_it8213.ko 2024-03-30T04:02:21.703 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_it821x.ko 2024-03-30T04:02:21.704 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_jmicron.ko 2024-03-30T04:02:21.704 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_marvell.ko 2024-03-30T04:02:21.704 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_mpiix.ko 2024-03-30T04:02:21.704 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_netcell.ko 2024-03-30T04:02:21.704 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ninja32.ko 2024-03-30T04:02:21.704 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ns87410.ko 2024-03-30T04:02:21.704 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ns87415.ko 2024-03-30T04:02:21.704 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_oldpiix.ko 2024-03-30T04:02:21.705 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_opti.ko 2024-03-30T04:02:21.705 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_optidma.ko 2024-03-30T04:02:21.705 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pcmcia.ko 2024-03-30T04:02:21.705 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-30T04:02:21.705 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-30T04:02:21.705 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_piccolo.ko 2024-03-30T04:02:21.706 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_rdc.ko 2024-03-30T04:02:21.706 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sch.ko 2024-03-30T04:02:21.706 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_serverworks.ko 2024-03-30T04:02:21.709 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sil680.ko 2024-03-30T04:02:21.709 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sis.ko 2024-03-30T04:02:21.709 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sl82c105.ko 2024-03-30T04:02:21.710 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_triflex.ko 2024-03-30T04:02:21.710 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_via.ko 2024-03-30T04:02:21.710 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pdc_adma.ko 2024-03-30T04:02:21.710 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_inic162x.ko 2024-03-30T04:02:21.710 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_mv.ko 2024-03-30T04:02:21.710 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_nv.ko 2024-03-30T04:02:21.710 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_promise.ko 2024-03-30T04:02:21.710 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_qstor.ko 2024-03-30T04:02:21.710 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sil.ko 2024-03-30T04:02:21.710 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sil24.ko 2024-03-30T04:02:21.710 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sis.ko 2024-03-30T04:02:21.710 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_svw.ko 2024-03-30T04:02:21.711 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sx4.ko 2024-03-30T04:02:21.711 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_uli.ko 2024-03-30T04:02:21.711 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_via.ko 2024-03-30T04:02:21.711 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_vsc.ko 2024-03-30T04:02:21.711 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/auxdisplay 2024-03-30T04:02:21.711 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/auxdisplay/ks0108.ko 2024-03-30T04:02:21.711 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base 2024-03-30T04:02:21.711 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap 2024-03-30T04:02:21.711 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-30T04:02:21.711 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-30T04:02:21.711 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/bcma 2024-03-30T04:02:21.711 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/bcma/bcma.ko 2024-03-30T04:02:21.711 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block 2024-03-30T04:02:21.711 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/aoe 2024-03-30T04:02:21.712 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/aoe/aoe.ko 2024-03-30T04:02:21.712 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/brd.ko 2024-03-30T04:02:21.712 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/drbd 2024-03-30T04:02:21.712 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/drbd/drbd.ko 2024-03-30T04:02:21.712 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/floppy.ko 2024-03-30T04:02:21.712 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/loop.ko 2024-03-30T04:02:21.712 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/mtip32xx 2024-03-30T04:02:21.712 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-30T04:02:21.712 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/nbd.ko 2024-03-30T04:02:21.712 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/null_blk 2024-03-30T04:02:21.712 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/null_blk/null_blk.ko 2024-03-30T04:02:21.712 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/pktcdvd.ko 2024-03-30T04:02:21.712 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/rbd.ko 2024-03-30T04:02:21.712 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/virtio_blk.ko 2024-03-30T04:02:21.712 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkback 2024-03-30T04:02:21.713 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-30T04:02:21.713 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkfront.ko 2024-03-30T04:02:21.713 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/zram 2024-03-30T04:02:21.713 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/zram/zram.ko 2024-03-30T04:02:21.713 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char 2024-03-30T04:02:21.713 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hangcheck-timer.ko 2024-03-30T04:02:21.713 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random 2024-03-30T04:02:21.713 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-30T04:02:21.713 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-30T04:02:21.713 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-30T04:02:21.713 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/via-rng.ko 2024-03-30T04:02:21.713 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-30T04:02:21.713 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi 2024-03-30T04:02:21.713 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-30T04:02:21.714 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-30T04:02:21.714 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-30T04:02:21.714 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-30T04:02:21.714 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-30T04:02:21.714 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-30T04:02:21.714 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/lp.ko 2024-03-30T04:02:21.714 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/mwave 2024-03-30T04:02:21.714 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/mwave/mwave.ko 2024-03-30T04:02:21.714 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ppdev.ko 2024-03-30T04:02:21.714 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tlclk.ko 2024-03-30T04:02:21.714 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm 2024-03-30T04:02:21.714 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm.ko 2024-03-30T04:02:21.714 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-30T04:02:21.714 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-30T04:02:21.715 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-30T04:02:21.715 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-30T04:02:21.715 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-30T04:02:21.715 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/uv_mmtimer.ko 2024-03-30T04:02:21.715 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/virtio_console.ko 2024-03-30T04:02:21.715 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq 2024-03-30T04:02:21.715 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-30T04:02:21.715 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-30T04:02:21.715 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-30T04:02:21.715 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-30T04:02:21.715 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-30T04:02:21.715 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-30T04:02:21.715 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto 2024-03-30T04:02:21.715 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp 2024-03-30T04:02:21.716 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-30T04:02:21.716 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp/ccp.ko 2024-03-30T04:02:21.716 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel 2024-03-30T04:02:21.716 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat 2024-03-30T04:02:21.716 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_common 2024-03-30T04:02:21.716 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-30T04:02:21.716 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-30T04:02:21.716 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-30T04:02:21.716 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/padlock-aes.ko 2024-03-30T04:02:21.716 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/padlock-sha.ko 2024-03-30T04:02:21.716 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/virtio 2024-03-30T04:02:21.716 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-30T04:02:21.716 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dca 2024-03-30T04:02:21.716 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dca/dca.ko 2024-03-30T04:02:21.717 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/devfreq 2024-03-30T04:02:21.717 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-30T04:02:21.717 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma 2024-03-30T04:02:21.717 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/dw 2024-03-30T04:02:21.717 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-30T04:02:21.717 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/ioat 2024-03-30T04:02:21.717 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-30T04:02:21.717 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac 2024-03-30T04:02:21.717 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/amd64_edac.ko 2024-03-30T04:02:21.717 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/e752x_edac.ko 2024-03-30T04:02:21.717 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/edac_mce_amd.ko 2024-03-30T04:02:21.717 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i3000_edac.ko 2024-03-30T04:02:21.717 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i3200_edac.ko 2024-03-30T04:02:21.717 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i5100_edac.ko 2024-03-30T04:02:21.718 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i5400_edac.ko 2024-03-30T04:02:21.718 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i7300_edac.ko 2024-03-30T04:02:21.718 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i7core_edac.ko 2024-03-30T04:02:21.718 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i82975x_edac.ko 2024-03-30T04:02:21.718 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/ie31200_edac.ko 2024-03-30T04:02:21.718 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/sb_edac.ko 2024-03-30T04:02:21.718 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/x38_edac.ko 2024-03-30T04:02:21.718 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware 2024-03-30T04:02:21.718 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/edd.ko 2024-03-30T04:02:21.718 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-30T04:02:21.718 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio 2024-03-30T04:02:21.718 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-ich.ko 2024-03-30T04:02:21.718 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-30T04:02:21.719 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid 2024-03-30T04:02:21.719 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-appleir.ko 2024-03-30T04:02:21.719 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-aureal.ko 2024-03-30T04:02:21.719 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-axff.ko 2024-03-30T04:02:21.719 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-dr.ko 2024-03-30T04:02:21.719 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-elecom.ko 2024-03-30T04:02:21.719 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-elo.ko 2024-03-30T04:02:21.719 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-emsff.ko 2024-03-30T04:02:21.719 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gaff.ko 2024-03-30T04:02:21.719 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gt683r.ko 2024-03-30T04:02:21.719 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gyration.ko 2024-03-30T04:02:21.719 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-30T04:02:21.719 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-30T04:02:21.719 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtekff.ko 2024-03-30T04:02:21.719 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-hyperv.ko 2024-03-30T04:02:21.720 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-icade.ko 2024-03-30T04:02:21.720 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-keytouch.ko 2024-03-30T04:02:21.720 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-kye.ko 2024-03-30T04:02:21.720 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-lcpower.ko 2024-03-30T04:02:21.720 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-led.ko 2024-03-30T04:02:21.720 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-lenovo.ko 2024-03-30T04:02:21.720 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-30T04:02:21.720 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-30T04:02:21.720 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-multitouch.ko 2024-03-30T04:02:21.720 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-ortek.ko 2024-03-30T04:02:21.720 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-penmount.ko 2024-03-30T04:02:21.720 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-petalynx.ko 2024-03-30T04:02:21.720 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-picolcd.ko 2024-03-30T04:02:21.720 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-pl.ko 2024-03-30T04:02:21.721 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-plantronics.ko 2024-03-30T04:02:21.721 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-primax.ko 2024-03-30T04:02:21.721 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-rmi.ko 2024-03-30T04:02:21.721 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-30T04:02:21.721 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-common.ko 2024-03-30T04:02:21.721 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-30T04:02:21.721 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-30T04:02:21.721 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-30T04:02:21.721 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-30T04:02:21.721 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-30T04:02:21.721 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-30T04:02:21.721 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-30T04:02:21.721 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-30T04:02:21.721 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-30T04:02:21.722 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat.ko 2024-03-30T04:02:21.722 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-saitek.ko 2024-03-30T04:02:21.722 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-samsung.ko 2024-03-30T04:02:21.722 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-30T04:02:21.722 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sjoy.ko 2024-03-30T04:02:21.722 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sony.ko 2024-03-30T04:02:21.722 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-speedlink.ko 2024-03-30T04:02:21.722 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-steelseries.ko 2024-03-30T04:02:21.722 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sunplus.ko 2024-03-30T04:02:21.722 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-30T04:02:21.722 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-tivo.ko 2024-03-30T04:02:21.722 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-tmff.ko 2024-03-30T04:02:21.722 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-topseed.ko 2024-03-30T04:02:21.722 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-twinhan.ko 2024-03-30T04:02:21.723 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-uclogic.ko 2024-03-30T04:02:21.723 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-waltop.ko 2024-03-30T04:02:21.723 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-wiimote.ko 2024-03-30T04:02:21.723 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-xinmo.ko 2024-03-30T04:02:21.723 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-zpff.ko 2024-03-30T04:02:21.723 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-zydacron.ko 2024-03-30T04:02:21.723 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/uhid.ko 2024-03-30T04:02:21.723 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/wacom.ko 2024-03-30T04:02:21.723 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv 2024-03-30T04:02:21.723 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_balloon.ko 2024-03-30T04:02:21.723 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_utils.ko 2024-03-30T04:02:21.723 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_vmbus.ko 2024-03-30T04:02:21.723 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon 2024-03-30T04:02:21.723 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/abituguru.ko 2024-03-30T04:02:21.724 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/abituguru3.ko 2024-03-30T04:02:21.724 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-30T04:02:21.724 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ad7414.ko 2024-03-30T04:02:21.724 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ad7418.ko 2024-03-30T04:02:21.724 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adc128d818.ko 2024-03-30T04:02:21.724 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1025.ko 2024-03-30T04:02:21.724 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1026.ko 2024-03-30T04:02:21.724 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1029.ko 2024-03-30T04:02:21.724 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1031.ko 2024-03-30T04:02:21.724 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm9240.ko 2024-03-30T04:02:21.724 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ads7828.ko 2024-03-30T04:02:21.724 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7410.ko 2024-03-30T04:02:21.724 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7411.ko 2024-03-30T04:02:21.724 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7462.ko 2024-03-30T04:02:21.724 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7470.ko 2024-03-30T04:02:21.725 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7475.ko 2024-03-30T04:02:21.725 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7x10.ko 2024-03-30T04:02:21.725 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/amc6821.ko 2024-03-30T04:02:21.725 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/applesmc.ko 2024-03-30T04:02:21.725 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asb100.ko 2024-03-30T04:02:21.725 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asc7621.ko 2024-03-30T04:02:21.725 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-30T04:02:21.725 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/atxp1.ko 2024-03-30T04:02:21.725 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/coretemp.ko 2024-03-30T04:02:21.725 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-30T04:02:21.725 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/dme1737.ko 2024-03-30T04:02:21.725 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ds1621.ko 2024-03-30T04:02:21.725 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ds620.ko 2024-03-30T04:02:21.725 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/emc1403.ko 2024-03-30T04:02:21.726 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/emc6w201.ko 2024-03-30T04:02:21.726 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f71805f.ko 2024-03-30T04:02:21.726 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f71882fg.ko 2024-03-30T04:02:21.726 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f75375s.ko 2024-03-30T04:02:21.726 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/fam15h_power.ko 2024-03-30T04:02:21.726 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/fschmd.ko 2024-03-30T04:02:21.726 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/g760a.ko 2024-03-30T04:02:21.726 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/g762.ko 2024-03-30T04:02:21.726 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/gl518sm.ko 2024-03-30T04:02:21.726 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/gl520sm.ko 2024-03-30T04:02:21.726 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-30T04:02:21.726 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/i5500_temp.ko 2024-03-30T04:02:21.726 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/i5k_amb.ko 2024-03-30T04:02:21.727 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ibmaem.ko 2024-03-30T04:02:21.727 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ibmpex.ko 2024-03-30T04:02:21.727 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ina209.ko 2024-03-30T04:02:21.727 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ina2xx.ko 2024-03-30T04:02:21.727 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/it87.ko 2024-03-30T04:02:21.727 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/k10temp.ko 2024-03-30T04:02:21.727 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/k8temp.ko 2024-03-30T04:02:21.727 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lineage-pem.ko 2024-03-30T04:02:21.727 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm63.ko 2024-03-30T04:02:21.727 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm73.ko 2024-03-30T04:02:21.727 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm75.ko 2024-03-30T04:02:21.727 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm77.ko 2024-03-30T04:02:21.727 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm78.ko 2024-03-30T04:02:21.727 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm80.ko 2024-03-30T04:02:21.728 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm83.ko 2024-03-30T04:02:21.728 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm85.ko 2024-03-30T04:02:21.728 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm87.ko 2024-03-30T04:02:21.728 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm90.ko 2024-03-30T04:02:21.728 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm92.ko 2024-03-30T04:02:21.728 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm93.ko 2024-03-30T04:02:21.728 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95234.ko 2024-03-30T04:02:21.728 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95241.ko 2024-03-30T04:02:21.728 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95245.ko 2024-03-30T04:02:21.728 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc2945.ko 2024-03-30T04:02:21.728 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4151.ko 2024-03-30T04:02:21.728 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4215.ko 2024-03-30T04:02:21.728 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4222.ko 2024-03-30T04:02:21.728 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4245.ko 2024-03-30T04:02:21.729 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4260.ko 2024-03-30T04:02:21.729 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4261.ko 2024-03-30T04:02:21.729 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max16065.ko 2024-03-30T04:02:21.729 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max1619.ko 2024-03-30T04:02:21.729 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max1668.ko 2024-03-30T04:02:21.729 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max197.ko 2024-03-30T04:02:21.729 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6639.ko 2024-03-30T04:02:21.729 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6650.ko 2024-03-30T04:02:21.729 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6697.ko 2024-03-30T04:02:21.729 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/mcp3021.ko 2024-03-30T04:02:21.729 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6683.ko 2024-03-30T04:02:21.729 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6775-core.ko 2024-03-30T04:02:21.729 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6775.ko 2024-03-30T04:02:21.730 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct7802.ko 2024-03-30T04:02:21.730 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pc87360.ko 2024-03-30T04:02:21.730 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pc87427.ko 2024-03-30T04:02:21.730 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pcf8591.ko 2024-03-30T04:02:21.730 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus 2024-03-30T04:02:21.730 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-30T04:02:21.730 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-30T04:02:21.730 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-30T04:02:21.730 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-30T04:02:21.730 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-30T04:02:21.731 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-30T04:02:21.731 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-30T04:02:21.731 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-30T04:02:21.731 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-30T04:02:21.731 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-30T04:02:21.731 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-30T04:02:21.731 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-30T04:02:21.731 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/powr1220.ko 2024-03-30T04:02:21.731 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch5627.ko 2024-03-30T04:02:21.732 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch5636.ko 2024-03-30T04:02:21.732 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-30T04:02:21.732 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sht15.ko 2024-03-30T04:02:21.732 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sht21.ko 2024-03-30T04:02:21.732 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/shtc1.ko 2024-03-30T04:02:21.732 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sis5595.ko 2024-03-30T04:02:21.732 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47b397.ko 2024-03-30T04:02:21.732 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47m1.ko 2024-03-30T04:02:21.732 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47m192.ko 2024-03-30T04:02:21.732 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/thmc50.ko 2024-03-30T04:02:21.732 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp102.ko 2024-03-30T04:02:21.732 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp103.ko 2024-03-30T04:02:21.732 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp401.ko 2024-03-30T04:02:21.732 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp421.ko 2024-03-30T04:02:21.733 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/via-cputemp.ko 2024-03-30T04:02:21.733 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/via686a.ko 2024-03-30T04:02:21.733 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/vt1211.ko 2024-03-30T04:02:21.733 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/vt8231.ko 2024-03-30T04:02:21.733 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83627ehf.ko 2024-03-30T04:02:21.733 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83627hf.ko 2024-03-30T04:02:21.733 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83781d.ko 2024-03-30T04:02:21.733 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83791d.ko 2024-03-30T04:02:21.734 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83792d.ko 2024-03-30T04:02:21.734 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83793.ko 2024-03-30T04:02:21.734 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83795.ko 2024-03-30T04:02:21.734 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83l785ts.ko 2024-03-30T04:02:21.734 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83l786ng.ko 2024-03-30T04:02:21.734 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c 2024-03-30T04:02:21.734 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos 2024-03-30T04:02:21.734 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-30T04:02:21.734 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-30T04:02:21.734 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses 2024-03-30T04:02:21.734 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-30T04:02:21.734 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-30T04:02:21.734 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-30T04:02:21.735 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-30T04:02:21.735 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-30T04:02:21.735 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-30T04:02:21.735 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-30T04:02:21.735 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-30T04:02:21.735 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-30T04:02:21.735 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-30T04:02:21.735 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-30T04:02:21.735 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-30T04:02:21.736 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-30T04:02:21.736 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-30T04:02:21.736 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-30T04:02:21.736 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-30T04:02:21.736 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-30T04:02:21.736 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-30T04:02:21.736 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-30T04:02:21.736 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-30T04:02:21.736 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-30T04:02:21.736 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-30T04:02:21.736 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-30T04:02:21.736 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-dev.ko 2024-03-30T04:02:21.736 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-mux.ko 2024-03-30T04:02:21.737 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-30T04:02:21.737 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-smbus.ko 2024-03-30T04:02:21.737 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-stub.ko 2024-03-30T04:02:21.737 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband 2024-03-30T04:02:21.737 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core 2024-03-30T04:02:21.737 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-30T04:02:21.737 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_core.ko 2024-03-30T04:02:21.737 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-30T04:02:21.737 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-30T04:02:21.738 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-30T04:02:21.738 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-30T04:02:21.738 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-30T04:02:21.738 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw 2024-03-30T04:02:21.738 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/cxgb4 2024-03-30T04:02:21.738 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-30T04:02:21.738 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx4 2024-03-30T04:02:21.738 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-30T04:02:21.738 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx5 2024-03-30T04:02:21.738 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-30T04:02:21.739 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mthca 2024-03-30T04:02:21.739 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-30T04:02:21.739 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp 2024-03-30T04:02:21.739 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/ipoib 2024-03-30T04:02:21.739 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-30T04:02:21.739 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/iser 2024-03-30T04:02:21.739 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-30T04:02:21.739 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/isert 2024-03-30T04:02:21.739 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-30T04:02:21.739 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srp 2024-03-30T04:02:21.739 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-30T04:02:21.739 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srpt 2024-03-30T04:02:21.740 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-30T04:02:21.740 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input 2024-03-30T04:02:21.740 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport 2024-03-30T04:02:21.740 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-30T04:02:21.740 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-30T04:02:21.740 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/gameport.ko 2024-03-30T04:02:21.740 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/joydev.ko 2024-03-30T04:02:21.740 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard 2024-03-30T04:02:21.740 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-30T04:02:21.740 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse 2024-03-30T04:02:21.740 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/appletouch.ko 2024-03-30T04:02:21.740 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/bcm5974.ko 2024-03-30T04:02:21.740 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/cyapatp.ko 2024-03-30T04:02:21.741 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-30T04:02:21.741 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/sermouse.ko 2024-03-30T04:02:21.741 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-30T04:02:21.741 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-30T04:02:21.741 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-30T04:02:21.741 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/rmi4 2024-03-30T04:02:21.741 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-30T04:02:21.741 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio 2024-03-30T04:02:21.741 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/altera_ps2.ko 2024-03-30T04:02:21.741 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/arc_ps2.ko 2024-03-30T04:02:21.741 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-30T04:02:21.741 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/serio_raw.ko 2024-03-30T04:02:21.741 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/sparse-keymap.ko 2024-03-30T04:02:21.742 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds 2024-03-30T04:02:21.742 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-blinkm.ko 2024-03-30T04:02:21.742 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lm3530.ko 2024-03-30T04:02:21.742 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lp3944.ko 2024-03-30T04:02:21.742 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-ss4200.ko 2024-03-30T04:02:21.742 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger 2024-03-30T04:02:21.742 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-30T04:02:21.742 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-30T04:02:21.742 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-30T04:02:21.742 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-30T04:02:21.742 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-30T04:02:21.743 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-30T04:02:21.743 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-30T04:02:21.743 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-30T04:02:21.743 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-30T04:02:21.743 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md 2024-03-30T04:02:21.743 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/bcache 2024-03-30T04:02:21.743 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/bcache/bcache.ko 2024-03-30T04:02:21.743 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-bio-prison.ko 2024-03-30T04:02:21.743 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-cache-smq.ko 2024-03-30T04:02:21.744 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-cache.ko 2024-03-30T04:02:21.744 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-crypt.ko 2024-03-30T04:02:21.744 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-delay.ko 2024-03-30T04:02:21.744 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-flakey.ko 2024-03-30T04:02:21.744 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-log-userspace.ko 2024-03-30T04:02:21.744 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-multipath.ko 2024-03-30T04:02:21.744 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-queue-length.ko 2024-03-30T04:02:21.744 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-raid.ko 2024-03-30T04:02:21.744 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-round-robin.ko 2024-03-30T04:02:21.745 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-service-time.ko 2024-03-30T04:02:21.745 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-switch.ko 2024-03-30T04:02:21.745 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-thin-pool.ko 2024-03-30T04:02:21.745 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-verity.ko 2024-03-30T04:02:21.745 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/persistent-data 2024-03-30T04:02:21.745 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-30T04:02:21.745 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid0.ko 2024-03-30T04:02:21.745 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid1.ko 2024-03-30T04:02:21.745 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid10.ko 2024-03-30T04:02:21.745 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid456.ko 2024-03-30T04:02:21.746 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message 2024-03-30T04:02:21.746 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion 2024-03-30T04:02:21.746 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptbase.ko 2024-03-30T04:02:21.746 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptctl.ko 2024-03-30T04:02:21.746 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptfc.ko 2024-03-30T04:02:21.746 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptlan.ko 2024-03-30T04:02:21.746 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptsas.ko 2024-03-30T04:02:21.746 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptscsih.ko 2024-03-30T04:02:21.746 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptspi.ko 2024-03-30T04:02:21.747 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd 2024-03-30T04:02:21.747 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lpc_ich.ko 2024-03-30T04:02:21.747 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lpc_sch.ko 2024-03-30T04:02:21.747 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/mfd-core.ko 2024-03-30T04:02:21.747 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/sm501.ko 2024-03-30T04:02:21.747 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/viperboard.ko 2024-03-30T04:02:21.747 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/vx855.ko 2024-03-30T04:02:21.747 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/wl1273-core.ko 2024-03-30T04:02:21.747 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc 2024-03-30T04:02:21.747 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/altera-stapl 2024-03-30T04:02:21.747 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-30T04:02:21.747 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/apds9802als.ko 2024-03-30T04:02:21.747 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/apds990x.ko 2024-03-30T04:02:21.747 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/bh1770glc.ko 2024-03-30T04:02:21.747 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/cb710 2024-03-30T04:02:21.748 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/cb710/cb710.ko 2024-03-30T04:02:21.748 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom 2024-03-30T04:02:21.748 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/at24.ko 2024-03-30T04:02:21.748 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-30T04:02:21.748 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/max6875.ko 2024-03-30T04:02:21.748 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/enclosure.ko 2024-03-30T04:02:21.748 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/hpilo.ko 2024-03-30T04:02:21.748 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/isl29003.ko 2024-03-30T04:02:21.748 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/isl29020.ko 2024-03-30T04:02:21.748 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d 2024-03-30T04:02:21.748 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-30T04:02:21.748 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-30T04:02:21.748 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei 2024-03-30T04:02:21.749 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei-me.ko 2024-03-30T04:02:21.749 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei-txe.ko 2024-03-30T04:02:21.749 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei.ko 2024-03-30T04:02:21.749 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-gru 2024-03-30T04:02:21.749 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-30T04:02:21.749 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp 2024-03-30T04:02:21.749 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-30T04:02:21.749 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-30T04:02:21.749 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-30T04:02:21.749 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tifm_7xx1.ko 2024-03-30T04:02:21.749 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tifm_core.ko 2024-03-30T04:02:21.749 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tsl2550.ko 2024-03-30T04:02:21.749 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_balloon.ko 2024-03-30T04:02:21.749 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_vmci 2024-03-30T04:02:21.750 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-30T04:02:21.750 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net 2024-03-30T04:02:21.750 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/bonding 2024-03-30T04:02:21.750 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/bonding/bonding.ko 2024-03-30T04:02:21.750 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa 2024-03-30T04:02:21.750 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/b53 2024-03-30T04:02:21.750 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-30T04:02:21.750 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-30T04:02:21.750 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-30T04:02:21.750 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6xxx 2024-03-30T04:02:21.750 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-30T04:02:21.750 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dummy.ko 2024-03-30T04:02:21.750 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/eql.ko 2024-03-30T04:02:21.751 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet 2024-03-30T04:02:21.751 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com 2024-03-30T04:02:21.751 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-30T04:02:21.751 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-30T04:02:21.751 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-30T04:02:21.751 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-30T04:02:21.751 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390 2024-03-30T04:02:21.751 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-30T04:02:21.751 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-30T04:02:21.751 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-30T04:02:21.751 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-30T04:02:21.751 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/adaptec 2024-03-30T04:02:21.751 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-30T04:02:21.751 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/agere 2024-03-30T04:02:21.752 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-30T04:02:21.752 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/alteon 2024-03-30T04:02:21.752 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-30T04:02:21.752 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/altera 2024-03-30T04:02:21.752 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-30T04:02:21.752 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd 2024-03-30T04:02:21.752 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-30T04:02:21.752 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-30T04:02:21.752 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-30T04:02:21.752 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros 2024-03-30T04:02:21.752 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/alx 2024-03-30T04:02:21.752 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-30T04:02:21.752 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-30T04:02:21.752 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-30T04:02:21.753 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-30T04:02:21.753 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-30T04:02:21.753 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx 2024-03-30T04:02:21.753 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-30T04:02:21.753 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-30T04:02:21.753 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom 2024-03-30T04:02:21.753 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-30T04:02:21.753 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-30T04:02:21.753 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-30T04:02:21.753 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-30T04:02:21.753 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-30T04:02:21.753 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/genet 2024-03-30T04:02:21.754 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-30T04:02:21.754 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-30T04:02:21.754 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade 2024-03-30T04:02:21.754 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade/bna 2024-03-30T04:02:21.754 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-30T04:02:21.754 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio 2024-03-30T04:02:21.754 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-30T04:02:21.754 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-30T04:02:21.754 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-30T04:02:21.754 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-30T04:02:21.754 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-30T04:02:21.754 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-30T04:02:21.754 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-30T04:02:21.754 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-30T04:02:21.755 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-30T04:02:21.755 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-30T04:02:21.755 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco 2024-03-30T04:02:21.755 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco/enic 2024-03-30T04:02:21.755 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-30T04:02:21.755 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec 2024-03-30T04:02:21.755 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip 2024-03-30T04:02:21.755 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-30T04:02:21.755 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-30T04:02:21.755 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-30T04:02:21.755 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-30T04:02:21.755 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-30T04:02:21.755 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-30T04:02:21.756 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink 2024-03-30T04:02:21.756 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-30T04:02:21.756 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-30T04:02:21.756 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dnet.ko 2024-03-30T04:02:21.756 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex 2024-03-30T04:02:21.756 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex/benet 2024-03-30T04:02:21.756 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-30T04:02:21.756 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ethoc.ko 2024-03-30T04:02:21.756 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/fealnx.ko 2024-03-30T04:02:21.756 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel 2024-03-30T04:02:21.756 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-30T04:02:21.756 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000 2024-03-30T04:02:21.756 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-30T04:02:21.756 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000e 2024-03-30T04:02:21.757 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-30T04:02:21.757 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/fm10k 2024-03-30T04:02:21.757 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-30T04:02:21.757 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/i40e 2024-03-30T04:02:21.757 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-30T04:02:21.757 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/iavf 2024-03-30T04:02:21.757 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-30T04:02:21.757 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igb 2024-03-30T04:02:21.757 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-30T04:02:21.757 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igbvf 2024-03-30T04:02:21.757 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-30T04:02:21.757 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-30T04:02:21.757 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-30T04:02:21.757 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-30T04:02:21.758 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-30T04:02:21.758 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/jme.ko 2024-03-30T04:02:21.758 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell 2024-03-30T04:02:21.758 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-30T04:02:21.758 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-30T04:02:21.758 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-30T04:02:21.758 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox 2024-03-30T04:02:21.758 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-30T04:02:21.758 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-30T04:02:21.758 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-30T04:02:21.758 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-30T04:02:21.758 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-30T04:02:21.758 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-30T04:02:21.759 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel 2024-03-30T04:02:21.759 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-30T04:02:21.759 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom 2024-03-30T04:02:21.759 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-30T04:02:21.759 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-30T04:02:21.759 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi 2024-03-30T04:02:21.759 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-30T04:02:21.759 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-30T04:02:21.759 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/neterion 2024-03-30T04:02:21.759 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-30T04:02:21.759 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/nvidia 2024-03-30T04:02:21.759 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-30T04:02:21.759 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines 2024-03-30T04:02:21.759 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-30T04:02:21.759 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-30T04:02:21.760 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic 2024-03-30T04:02:21.760 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-30T04:02:21.760 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-30T04:02:21.760 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-30T04:02:21.760 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-30T04:02:21.760 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-30T04:02:21.760 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rdc 2024-03-30T04:02:21.760 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-30T04:02:21.760 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek 2024-03-30T04:02:21.760 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-30T04:02:21.760 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-30T04:02:21.760 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-30T04:02:21.760 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-30T04:02:21.761 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rocker 2024-03-30T04:02:21.761 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-30T04:02:21.761 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sfc 2024-03-30T04:02:21.761 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-30T04:02:21.761 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/silan 2024-03-30T04:02:21.761 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-30T04:02:21.761 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis 2024-03-30T04:02:21.761 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-30T04:02:21.761 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-30T04:02:21.761 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc 2024-03-30T04:02:21.761 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-30T04:02:21.761 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-30T04:02:21.761 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-30T04:02:21.761 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-30T04:02:21.762 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro 2024-03-30T04:02:21.762 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-30T04:02:21.762 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-30T04:02:21.762 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-30T04:02:21.762 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun 2024-03-30T04:02:21.762 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-30T04:02:21.762 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-30T04:02:21.762 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-30T04:02:21.762 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-30T04:02:21.762 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/tehuti 2024-03-30T04:02:21.762 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-30T04:02:21.762 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ti 2024-03-30T04:02:21.763 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-30T04:02:21.763 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via 2024-03-30T04:02:21.763 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-30T04:02:21.763 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-30T04:02:21.763 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet 2024-03-30T04:02:21.763 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-30T04:02:21.763 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-30T04:02:21.763 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/xircom 2024-03-30T04:02:21.763 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-30T04:02:21.763 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/geneve.ko 2024-03-30T04:02:21.763 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/hyperv 2024-03-30T04:02:21.763 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-30T04:02:21.763 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ieee802154 2024-03-30T04:02:21.763 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-30T04:02:21.764 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ifb.ko 2024-03-30T04:02:21.764 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ipvlan 2024-03-30T04:02:21.764 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-30T04:02:21.764 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/macvlan.ko 2024-03-30T04:02:21.764 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/macvtap.ko 2024-03-30T04:02:21.764 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio 2024-03-30T04:02:21.764 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio.ko 2024-03-30T04:02:21.764 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-30T04:02:21.764 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-30T04:02:21.764 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-30T04:02:21.764 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mii.ko 2024-03-30T04:02:21.764 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/net_failover.ko 2024-03-30T04:02:21.764 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/netconsole.ko 2024-03-30T04:02:21.765 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/nlmon.ko 2024-03-30T04:02:21.765 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs 2024-03-30T04:02:21.765 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-30T04:02:21.765 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-30T04:02:21.765 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy 2024-03-30T04:02:21.765 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/amd.ko 2024-03-30T04:02:21.765 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-30T04:02:21.765 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-30T04:02:21.765 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-30T04:02:21.765 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm87xx.ko 2024-03-30T04:02:21.765 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/broadcom.ko 2024-03-30T04:02:21.765 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/cicada.ko 2024-03-30T04:02:21.765 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/davicom.ko 2024-03-30T04:02:21.765 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/dp83640.ko 2024-03-30T04:02:21.766 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/et1011c.ko 2024-03-30T04:02:21.766 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/icplus.ko 2024-03-30T04:02:21.766 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/lxt.ko 2024-03-30T04:02:21.766 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/marvell.ko 2024-03-30T04:02:21.766 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/micrel.ko 2024-03-30T04:02:21.766 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/national.ko 2024-03-30T04:02:21.766 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/phylink.ko 2024-03-30T04:02:21.766 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/qsemi.ko 2024-03-30T04:02:21.766 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/realtek.ko 2024-03-30T04:02:21.766 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/smsc.ko 2024-03-30T04:02:21.766 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/ste10Xp.ko 2024-03-30T04:02:21.766 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/vitesse.ko 2024-03-30T04:02:21.766 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp 2024-03-30T04:02:21.766 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-30T04:02:21.767 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_async.ko 2024-03-30T04:02:21.767 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-30T04:02:21.767 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-30T04:02:21.767 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-30T04:02:21.767 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-30T04:02:21.767 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pppoe.ko 2024-03-30T04:02:21.767 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pppox.ko 2024-03-30T04:02:21.767 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pptp.ko 2024-03-30T04:02:21.767 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip 2024-03-30T04:02:21.767 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip/slhc.ko 2024-03-30T04:02:21.767 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip/slip.ko 2024-03-30T04:02:21.767 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/sungem_phy.ko 2024-03-30T04:02:21.767 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/tap.ko 2024-03-30T04:02:21.767 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team 2024-03-30T04:02:21.767 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team.ko 2024-03-30T04:02:21.768 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-30T04:02:21.768 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-30T04:02:21.768 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-30T04:02:21.768 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_random.ko 2024-03-30T04:02:21.768 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-30T04:02:21.768 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/tun.ko 2024-03-30T04:02:21.768 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/veth.ko 2024-03-30T04:02:21.768 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/virtio_net.ko 2024-03-30T04:02:21.768 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vmxnet3 2024-03-30T04:02:21.768 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-30T04:02:21.769 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vxlan 2024-03-30T04:02:21.769 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vxlan/vxlan.ko 2024-03-30T04:02:21.769 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netback 2024-03-30T04:02:21.769 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-30T04:02:21.769 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netfront.ko 2024-03-30T04:02:21.769 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ntb 2024-03-30T04:02:21.769 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ntb/ntb.ko 2024-03-30T04:02:21.769 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme 2024-03-30T04:02:21.769 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host 2024-03-30T04:02:21.769 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host/nvme-core.ko 2024-03-30T04:02:21.769 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host/nvme.ko 2024-03-30T04:02:21.769 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport 2024-03-30T04:02:21.770 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport.ko 2024-03-30T04:02:21.770 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_cs.ko 2024-03-30T04:02:21.770 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_pc.ko 2024-03-30T04:02:21.770 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_serial.ko 2024-03-30T04:02:21.770 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci 2024-03-30T04:02:21.770 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug 2024-03-30T04:02:21.770 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-30T04:02:21.770 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/pcie 2024-03-30T04:02:21.770 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-30T04:02:21.770 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/xen-pcifront.ko 2024-03-30T04:02:21.770 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia 2024-03-30T04:02:21.771 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/i82092.ko 2024-03-30T04:02:21.771 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/pd6729.ko 2024-03-30T04:02:21.771 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-30T04:02:21.771 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl 2024-03-30T04:02:21.771 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl/intel 2024-03-30T04:02:21.771 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-30T04:02:21.771 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform 2024-03-30T04:02:21.771 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86 2024-03-30T04:02:21.771 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-30T04:02:21.771 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/acerhdf.ko 2024-03-30T04:02:21.771 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-30T04:02:21.772 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-30T04:02:21.772 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-30T04:02:21.772 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-30T04:02:21.772 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-30T04:02:21.772 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-30T04:02:21.772 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-30T04:02:21.772 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-30T04:02:21.772 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-30T04:02:21.772 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/hdaps.ko 2024-03-30T04:02:21.772 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel 2024-03-30T04:02:21.772 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-30T04:02:21.772 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel_ips.ko 2024-03-30T04:02:21.772 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-30T04:02:21.773 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-30T04:02:21.773 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-30T04:02:21.773 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-30T04:02:21.773 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-30T04:02:21.773 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-30T04:02:21.773 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-30T04:02:21.773 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-30T04:02:21.773 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-30T04:02:21.773 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/wmi.ko 2024-03-30T04:02:21.774 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap 2024-03-30T04:02:21.774 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-30T04:02:21.774 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-30T04:02:21.774 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps 2024-03-30T04:02:21.774 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients 2024-03-30T04:02:21.774 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-30T04:02:21.774 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-30T04:02:21.774 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps_parport.ko 2024-03-30T04:02:21.774 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/pps_core.ko 2024-03-30T04:02:21.774 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp 2024-03-30T04:02:21.774 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp/ptp.ko 2024-03-30T04:02:21.775 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp/ptp_kvm.ko 2024-03-30T04:02:21.775 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc 2024-03-30T04:02:21.775 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-30T04:02:21.775 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-30T04:02:21.775 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-30T04:02:21.775 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-30T04:02:21.775 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-30T04:02:21.775 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-30T04:02:21.775 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-30T04:02:21.775 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-30T04:02:21.776 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-30T04:02:21.776 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-30T04:02:21.776 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-em3027.ko 2024-03-30T04:02:21.776 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-30T04:02:21.776 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-30T04:02:21.776 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-30T04:02:21.776 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-30T04:02:21.776 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-30T04:02:21.776 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-30T04:02:21.776 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-max6900.ko 2024-03-30T04:02:21.776 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-30T04:02:21.777 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-30T04:02:21.777 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-30T04:02:21.777 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-30T04:02:21.777 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-30T04:02:21.777 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-30T04:02:21.777 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-30T04:02:21.777 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-30T04:02:21.777 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-30T04:02:21.777 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-30T04:02:21.777 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-30T04:02:21.777 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-30T04:02:21.778 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-x1205.ko 2024-03-30T04:02:21.778 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi 2024-03-30T04:02:21.778 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-9xxx.ko 2024-03-30T04:02:21.778 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-sas.ko 2024-03-30T04:02:21.778 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-xxxx.ko 2024-03-30T04:02:21.778 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/BusLogic.ko 2024-03-30T04:02:21.778 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/a100u2w.ko 2024-03-30T04:02:21.778 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aacraid 2024-03-30T04:02:21.778 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-30T04:02:21.778 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/advansys.ko 2024-03-30T04:02:21.779 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx 2024-03-30T04:02:21.779 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-30T04:02:21.779 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-30T04:02:21.779 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/am53c974.ko 2024-03-30T04:02:21.779 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/arcmsr 2024-03-30T04:02:21.779 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-30T04:02:21.779 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/atp870u.ko 2024-03-30T04:02:21.779 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/be2iscsi 2024-03-30T04:02:21.779 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-30T04:02:21.780 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bfa 2024-03-30T04:02:21.780 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bfa/bfa.ko 2024-03-30T04:02:21.780 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2fc 2024-03-30T04:02:21.780 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-30T04:02:21.780 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2i 2024-03-30T04:02:21.780 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-30T04:02:21.780 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ch.ko 2024-03-30T04:02:21.780 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/csiostor 2024-03-30T04:02:21.780 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-30T04:02:21.780 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi 2024-03-30T04:02:21.781 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-30T04:02:21.781 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-30T04:02:21.781 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-30T04:02:21.781 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-30T04:02:21.781 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-30T04:02:21.781 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/dc395x.ko 2024-03-30T04:02:21.781 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler 2024-03-30T04:02:21.781 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-30T04:02:21.781 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-30T04:02:21.781 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-30T04:02:21.781 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-30T04:02:21.782 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/dmx3191d.ko 2024-03-30T04:02:21.782 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esas2r 2024-03-30T04:02:21.782 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-30T04:02:21.782 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esp_scsi.ko 2024-03-30T04:02:21.782 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe 2024-03-30T04:02:21.782 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-30T04:02:21.782 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-30T04:02:21.782 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fnic 2024-03-30T04:02:21.782 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fnic/fnic.ko 2024-03-30T04:02:21.782 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hpsa.ko 2024-03-30T04:02:21.782 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hptiop.ko 2024-03-30T04:02:21.782 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hv_storvsc.ko 2024-03-30T04:02:21.783 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/initio.ko 2024-03-30T04:02:21.783 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ipr.ko 2024-03-30T04:02:21.783 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ips.ko 2024-03-30T04:02:21.783 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/isci 2024-03-30T04:02:21.783 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/isci/isci.ko 2024-03-30T04:02:21.783 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-30T04:02:21.783 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-30T04:02:21.783 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libfc 2024-03-30T04:02:21.783 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libfc/libfc.ko 2024-03-30T04:02:21.783 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libiscsi.ko 2024-03-30T04:02:21.783 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-30T04:02:21.783 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libsas 2024-03-30T04:02:21.783 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libsas/libsas.ko 2024-03-30T04:02:21.783 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/lpfc 2024-03-30T04:02:21.784 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-30T04:02:21.784 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid 2024-03-30T04:02:21.784 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid.ko 2024-03-30T04:02:21.784 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-30T04:02:21.784 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-30T04:02:21.784 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-30T04:02:21.784 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mpt3sas 2024-03-30T04:02:21.784 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-30T04:02:21.784 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvsas 2024-03-30T04:02:21.784 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-30T04:02:21.784 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvumi.ko 2024-03-30T04:02:21.784 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pm8001 2024-03-30T04:02:21.784 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-30T04:02:21.784 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pmcraid.ko 2024-03-30T04:02:21.784 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla1280.ko 2024-03-30T04:02:21.785 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx 2024-03-30T04:02:21.785 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-30T04:02:21.785 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-30T04:02:21.785 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla4xxx 2024-03-30T04:02:21.785 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-30T04:02:21.785 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/raid_class.ko 2024-03-30T04:02:21.785 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_debug.ko 2024-03-30T04:02:21.785 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-30T04:02:21.785 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-30T04:02:21.785 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-30T04:02:21.785 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-30T04:02:21.785 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-30T04:02:21.785 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ses.ko 2024-03-30T04:02:21.786 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/st.ko 2024-03-30T04:02:21.786 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/stex.ko 2024-03-30T04:02:21.786 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/sym53c8xx_2 2024-03-30T04:02:21.786 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-30T04:02:21.786 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/virtio_scsi.ko 2024-03-30T04:02:21.786 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-30T04:02:21.786 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/wd719x.ko 2024-03-30T04:02:21.786 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ssb 2024-03-30T04:02:21.786 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ssb/ssb.ko 2024-03-30T04:02:21.786 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target 2024-03-30T04:02:21.786 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/iscsi 2024-03-30T04:02:21.786 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-30T04:02:21.786 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/loopback 2024-03-30T04:02:21.786 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-30T04:02:21.787 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_file.ko 2024-03-30T04:02:21.787 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_iblock.ko 2024-03-30T04:02:21.787 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_mod.ko 2024-03-30T04:02:21.787 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_pscsi.ko 2024-03-30T04:02:21.787 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_user.ko 2024-03-30T04:02:21.787 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/tcm_fc 2024-03-30T04:02:21.787 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-30T04:02:21.787 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal 2024-03-30T04:02:21.787 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel 2024-03-30T04:02:21.787 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal 2024-03-30T04:02:21.787 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-30T04:02:21.787 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-30T04:02:21.787 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-30T04:02:21.787 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-30T04:02:21.788 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-30T04:02:21.788 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-30T04:02:21.788 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-30T04:02:21.788 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-30T04:02:21.788 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-30T04:02:21.788 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-30T04:02:21.788 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-30T04:02:21.788 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-30T04:02:21.788 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-30T04:02:21.788 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-30T04:02:21.788 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-30T04:02:21.788 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-30T04:02:21.788 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-30T04:02:21.788 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-30T04:02:21.789 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty 2024-03-30T04:02:21.789 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/ipwireless 2024-03-30T04:02:21.789 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-30T04:02:21.789 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/n_gsm.ko 2024-03-30T04:02:21.789 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/n_hdlc.ko 2024-03-30T04:02:21.789 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/nozomi.ko 2024-03-30T04:02:21.789 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial 2024-03-30T04:02:21.789 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/8250 2024-03-30T04:02:21.789 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-30T04:02:21.789 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/arc_uart.ko 2024-03-30T04:02:21.789 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/jsm 2024-03-30T04:02:21.789 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-30T04:02:21.790 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs 2024-03-30T04:02:21.790 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/core 2024-03-30T04:02:21.790 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-30T04:02:21.790 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/host 2024-03-30T04:02:21.790 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-30T04:02:21.790 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio 2024-03-30T04:02:21.790 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio.ko 2024-03-30T04:02:21.790 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_aec.ko 2024-03-30T04:02:21.790 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_cif.ko 2024-03-30T04:02:21.790 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_pci_generic.ko 2024-03-30T04:02:21.790 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_sercos3.ko 2024-03-30T04:02:21.790 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb 2024-03-30T04:02:21.790 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class 2024-03-30T04:02:21.791 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/cdc-acm.ko 2024-03-30T04:02:21.791 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-30T04:02:21.791 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/usblp.ko 2024-03-30T04:02:21.791 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/usbtmc.ko 2024-03-30T04:02:21.791 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host 2024-03-30T04:02:21.791 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-30T04:02:21.791 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image 2024-03-30T04:02:21.791 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image/mdc800.ko 2024-03-30T04:02:21.791 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image/microtek.ko 2024-03-30T04:02:21.791 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc 2024-03-30T04:02:21.791 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/adutux.ko 2024-03-30T04:02:21.791 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/appledisplay.ko 2024-03-30T04:02:21.791 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/emi26.ko 2024-03-30T04:02:21.792 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/emi62.ko 2024-03-30T04:02:21.792 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ezusb.ko 2024-03-30T04:02:21.792 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/idmouse.ko 2024-03-30T04:02:21.792 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/iowarrior.ko 2024-03-30T04:02:21.792 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-30T04:02:21.792 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ldusb.ko 2024-03-30T04:02:21.792 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/legousbtower.ko 2024-03-30T04:02:21.792 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/sisusbvga 2024-03-30T04:02:21.792 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-30T04:02:21.792 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-30T04:02:21.792 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usb3503.ko 2024-03-30T04:02:21.792 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usblcd.ko 2024-03-30T04:02:21.792 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-30T04:02:21.792 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/uss720.ko 2024-03-30T04:02:21.793 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/yurex.ko 2024-03-30T04:02:21.793 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy 2024-03-30T04:02:21.793 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy/phy-generic.ko 2024-03-30T04:02:21.793 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial 2024-03-30T04:02:21.793 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/aircable.ko 2024-03-30T04:02:21.793 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ark3116.ko 2024-03-30T04:02:21.793 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-30T04:02:21.793 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ch341.ko 2024-03-30T04:02:21.793 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cp210x.ko 2024-03-30T04:02:21.793 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cyberjack.ko 2024-03-30T04:02:21.793 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-30T04:02:21.793 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-30T04:02:21.793 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/empeg.ko 2024-03-30T04:02:21.793 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-30T04:02:21.794 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-30T04:02:21.794 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-30T04:02:21.794 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/io_ti.ko 2024-03-30T04:02:21.794 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ipaq.ko 2024-03-30T04:02:21.794 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ipw.ko 2024-03-30T04:02:21.794 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ir-usb.ko 2024-03-30T04:02:21.794 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-30T04:02:21.794 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/keyspan.ko 2024-03-30T04:02:21.794 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-30T04:02:21.794 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-30T04:02:21.794 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-30T04:02:21.795 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mct_u232.ko 2024-03-30T04:02:21.795 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mos7720.ko 2024-03-30T04:02:21.795 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mos7840.ko 2024-03-30T04:02:21.795 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/navman.ko 2024-03-30T04:02:21.795 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/omninet.ko 2024-03-30T04:02:21.795 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/opticon.ko 2024-03-30T04:02:21.795 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/option.ko 2024-03-30T04:02:21.795 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/oti6858.ko 2024-03-30T04:02:21.795 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/pl2303.ko 2024-03-30T04:02:21.795 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/qcaux.ko 2024-03-30T04:02:21.795 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/qcserial.ko 2024-03-30T04:02:21.795 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/quatech2.ko 2024-03-30T04:02:21.795 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/safe_serial.ko 2024-03-30T04:02:21.795 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/sierra.ko 2024-03-30T04:02:21.796 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-30T04:02:21.796 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ssu100.ko 2024-03-30T04:02:21.796 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/symbolserial.ko 2024-03-30T04:02:21.796 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-30T04:02:21.796 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-30T04:02:21.796 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb_debug.ko 2024-03-30T04:02:21.796 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-30T04:02:21.796 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/visor.ko 2024-03-30T04:02:21.796 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/whiteheat.ko 2024-03-30T04:02:21.796 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-30T04:02:21.796 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage 2024-03-30T04:02:21.796 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/uas.ko 2024-03-30T04:02:21.796 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-30T04:02:21.796 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-30T04:02:21.797 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-30T04:02:21.797 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-30T04:02:21.797 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-30T04:02:21.797 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-30T04:02:21.797 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-30T04:02:21.797 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-karma.ko 2024-03-30T04:02:21.797 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-30T04:02:21.797 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-30T04:02:21.797 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-30T04:02:21.797 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-30T04:02:21.797 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-30T04:02:21.797 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/usb-storage.ko 2024-03-30T04:02:21.797 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip 2024-03-30T04:02:21.797 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-30T04:02:21.798 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-30T04:02:21.798 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-30T04:02:21.798 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio 2024-03-30T04:02:21.798 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci 2024-03-30T04:02:21.798 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-30T04:02:21.798 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-30T04:02:21.798 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/vfio.ko 2024-03-30T04:02:21.798 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-30T04:02:21.798 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost 2024-03-30T04:02:21.798 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost.ko 2024-03-30T04:02:21.798 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-30T04:02:21.798 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_net.ko 2024-03-30T04:02:21.798 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_scsi.ko 2024-03-30T04:02:21.798 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video 2024-03-30T04:02:21.799 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight 2024-03-30T04:02:21.799 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/apple_bl.ko 2024-03-30T04:02:21.799 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/lcd.ko 2024-03-30T04:02:21.799 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-30T04:02:21.799 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio 2024-03-30T04:02:21.799 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio.ko 2024-03-30T04:02:21.799 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_balloon.ko 2024-03-30T04:02:21.799 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_mmio.ko 2024-03-30T04:02:21.799 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci.ko 2024-03-30T04:02:21.799 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-30T04:02:21.799 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-30T04:02:21.799 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_ring.ko 2024-03-30T04:02:21.800 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1 2024-03-30T04:02:21.800 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters 2024-03-30T04:02:21.800 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/ds2482.ko 2024-03-30T04:02:21.800 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/ds2490.ko 2024-03-30T04:02:21.800 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves 2024-03-30T04:02:21.800 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-30T04:02:21.800 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-30T04:02:21.800 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-30T04:02:21.800 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-30T04:02:21.800 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-30T04:02:21.800 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-30T04:02:21.800 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-30T04:02:21.800 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-30T04:02:21.800 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-30T04:02:21.801 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-30T04:02:21.801 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-30T04:02:21.801 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/wire.ko 2024-03-30T04:02:21.801 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog 2024-03-30T04:02:21.801 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-30T04:02:21.801 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-30T04:02:21.801 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-30T04:02:21.801 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/hpwdt.ko 2024-03-30T04:02:21.801 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/i6300esb.ko 2024-03-30T04:02:21.801 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-30T04:02:21.801 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-30T04:02:21.801 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ib700wdt.ko 2024-03-30T04:02:21.801 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ibmasr.ko 2024-03-30T04:02:21.801 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-30T04:02:21.802 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-30T04:02:21.802 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/it87_wdt.ko 2024-03-30T04:02:21.802 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/machzwd.ko 2024-03-30T04:02:21.802 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/nv_tco.ko 2024-03-30T04:02:21.802 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-30T04:02:21.802 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-30T04:02:21.802 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-30T04:02:21.802 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-30T04:02:21.802 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/softdog.ko 2024-03-30T04:02:21.802 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-30T04:02:21.802 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/via_wdt.ko 2024-03-30T04:02:21.802 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-30T04:02:21.802 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-30T04:02:21.803 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-30T04:02:21.803 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/wdt_pci.ko 2024-03-30T04:02:21.803 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/xen_wdt.ko 2024-03-30T04:02:21.803 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen 2024-03-30T04:02:21.803 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-30T04:02:21.803 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-evtchn.ko 2024-03-30T04:02:21.803 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-gntalloc.ko 2024-03-30T04:02:21.803 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-gntdev.ko 2024-03-30T04:02:21.803 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-pciback 2024-03-30T04:02:21.803 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-30T04:02:21.803 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-privcmd.ko 2024-03-30T04:02:21.804 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xenfs 2024-03-30T04:02:21.804 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-30T04:02:21.804 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs 2024-03-30T04:02:21.804 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/9p 2024-03-30T04:02:21.804 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/9p/9p.ko 2024-03-30T04:02:21.804 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/affs 2024-03-30T04:02:21.804 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/affs/affs.ko 2024-03-30T04:02:21.804 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/befs 2024-03-30T04:02:21.804 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/befs/befs.ko 2024-03-30T04:02:21.804 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/binfmt_misc.ko 2024-03-30T04:02:21.804 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/btrfs 2024-03-30T04:02:21.805 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/btrfs/btrfs.ko 2024-03-30T04:02:21.805 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cachefiles 2024-03-30T04:02:21.805 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cachefiles/cachefiles.ko 2024-03-30T04:02:21.805 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ceph 2024-03-30T04:02:21.805 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ceph/ceph.ko 2024-03-30T04:02:21.805 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/coda 2024-03-30T04:02:21.805 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/coda/coda.ko 2024-03-30T04:02:21.805 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cramfs 2024-03-30T04:02:21.805 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cramfs/cramfs.ko 2024-03-30T04:02:21.805 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/dlm 2024-03-30T04:02:21.805 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/dlm/dlm.ko 2024-03-30T04:02:21.805 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ecryptfs 2024-03-30T04:02:21.805 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-30T04:02:21.806 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/f2fs 2024-03-30T04:02:21.806 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/f2fs/f2fs.ko 2024-03-30T04:02:21.806 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat 2024-03-30T04:02:21.806 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/fat.ko 2024-03-30T04:02:21.806 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/msdos.ko 2024-03-30T04:02:21.806 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/vfat.ko 2024-03-30T04:02:21.806 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse 2024-03-30T04:02:21.806 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse/cuse.ko 2024-03-30T04:02:21.806 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse/fuse.ko 2024-03-30T04:02:21.807 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/gfs2 2024-03-30T04:02:21.807 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/gfs2/gfs2.ko 2024-03-30T04:02:21.807 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfs 2024-03-30T04:02:21.807 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfs/hfs.ko 2024-03-30T04:02:21.807 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfsplus 2024-03-30T04:02:21.807 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfsplus/hfsplus.ko 2024-03-30T04:02:21.807 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/isofs 2024-03-30T04:02:21.807 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/isofs/isofs.ko 2024-03-30T04:02:21.807 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/jfs 2024-03-30T04:02:21.807 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/jfs/jfs.ko 2024-03-30T04:02:21.807 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/lockd 2024-03-30T04:02:21.807 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/lockd/lockd.ko 2024-03-30T04:02:21.807 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/minix 2024-03-30T04:02:21.808 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/minix/minix.ko 2024-03-30T04:02:21.808 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/netfs 2024-03-30T04:02:21.808 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/netfs/netfs.ko 2024-03-30T04:02:21.808 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs 2024-03-30T04:02:21.808 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/blocklayout 2024-03-30T04:02:21.808 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-30T04:02:21.808 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/filelayout 2024-03-30T04:02:21.808 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-30T04:02:21.808 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/flexfilelayout 2024-03-30T04:02:21.808 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-30T04:02:21.809 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfs.ko 2024-03-30T04:02:21.809 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfsv3.ko 2024-03-30T04:02:21.809 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfsv4.ko 2024-03-30T04:02:21.809 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common 2024-03-30T04:02:21.809 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common/grace.ko 2024-03-30T04:02:21.809 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common/nfs_acl.ko 2024-03-30T04:02:21.809 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfsd 2024-03-30T04:02:21.809 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfsd/nfsd.ko 2024-03-30T04:02:21.809 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nilfs2 2024-03-30T04:02:21.809 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nilfs2/nilfs2.ko 2024-03-30T04:02:21.809 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls 2024-03-30T04:02:21.810 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-celtic.ko 2024-03-30T04:02:21.810 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-centeuro.ko 2024-03-30T04:02:21.810 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-croatian.ko 2024-03-30T04:02:21.810 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-cyrillic.ko 2024-03-30T04:02:21.810 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-gaelic.ko 2024-03-30T04:02:21.810 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-greek.ko 2024-03-30T04:02:21.810 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-iceland.ko 2024-03-30T04:02:21.811 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-inuit.ko 2024-03-30T04:02:21.811 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-roman.ko 2024-03-30T04:02:21.811 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-romanian.ko 2024-03-30T04:02:21.811 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-turkish.ko 2024-03-30T04:02:21.811 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1250.ko 2024-03-30T04:02:21.811 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1251.ko 2024-03-30T04:02:21.811 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1255.ko 2024-03-30T04:02:21.811 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp737.ko 2024-03-30T04:02:21.811 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp775.ko 2024-03-30T04:02:21.811 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp850.ko 2024-03-30T04:02:21.811 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp852.ko 2024-03-30T04:02:21.811 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp855.ko 2024-03-30T04:02:21.811 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp857.ko 2024-03-30T04:02:21.812 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp860.ko 2024-03-30T04:02:21.812 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp861.ko 2024-03-30T04:02:21.812 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp862.ko 2024-03-30T04:02:21.812 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp863.ko 2024-03-30T04:02:21.812 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp864.ko 2024-03-30T04:02:21.812 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp865.ko 2024-03-30T04:02:21.812 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp866.ko 2024-03-30T04:02:21.812 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp869.ko 2024-03-30T04:02:21.812 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp874.ko 2024-03-30T04:02:21.813 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp932.ko 2024-03-30T04:02:21.813 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp936.ko 2024-03-30T04:02:21.813 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp949.ko 2024-03-30T04:02:21.813 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp950.ko 2024-03-30T04:02:21.813 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_euc-jp.ko 2024-03-30T04:02:21.813 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-1.ko 2024-03-30T04:02:21.813 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-13.ko 2024-03-30T04:02:21.813 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-14.ko 2024-03-30T04:02:21.813 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-15.ko 2024-03-30T04:02:21.813 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-2.ko 2024-03-30T04:02:21.814 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-3.ko 2024-03-30T04:02:21.814 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-4.ko 2024-03-30T04:02:21.814 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-5.ko 2024-03-30T04:02:21.814 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-6.ko 2024-03-30T04:02:21.814 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-7.ko 2024-03-30T04:02:21.814 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-9.ko 2024-03-30T04:02:21.814 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-r.ko 2024-03-30T04:02:21.814 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-ru.ko 2024-03-30T04:02:21.814 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-u.ko 2024-03-30T04:02:21.814 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-30T04:02:21.815 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_utf8.ko 2024-03-30T04:02:21.815 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2 2024-03-30T04:02:21.815 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/cluster 2024-03-30T04:02:21.815 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-30T04:02:21.815 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlm 2024-03-30T04:02:21.815 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-30T04:02:21.815 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlmfs 2024-03-30T04:02:21.815 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-30T04:02:21.815 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2.ko 2024-03-30T04:02:21.815 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-30T04:02:21.815 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-30T04:02:21.815 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-30T04:02:21.816 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/overlayfs 2024-03-30T04:02:21.816 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/overlayfs/overlay.ko 2024-03-30T04:02:21.816 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/pstore 2024-03-30T04:02:21.816 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/pstore/ramoops.ko 2024-03-30T04:02:21.816 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/reiserfs 2024-03-30T04:02:21.816 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/reiserfs/reiserfs.ko 2024-03-30T04:02:21.816 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/romfs 2024-03-30T04:02:21.816 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/romfs/romfs.ko 2024-03-30T04:02:21.816 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb 2024-03-30T04:02:21.816 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/client 2024-03-30T04:02:21.816 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/client/cifs.ko 2024-03-30T04:02:21.816 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common 2024-03-30T04:02:21.816 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common/cifs_arc4.ko 2024-03-30T04:02:21.817 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common/cifs_md4.ko 2024-03-30T04:02:21.817 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/squashfs 2024-03-30T04:02:21.817 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/squashfs/squashfs.ko 2024-03-30T04:02:21.817 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/sysv 2024-03-30T04:02:21.817 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/sysv/sysv.ko 2024-03-30T04:02:21.817 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/udf 2024-03-30T04:02:21.817 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/udf/udf.ko 2024-03-30T04:02:21.817 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ufs 2024-03-30T04:02:21.817 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ufs/ufs.ko 2024-03-30T04:02:21.817 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/xfs 2024-03-30T04:02:21.817 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/xfs/xfs.ko 2024-03-30T04:02:21.817 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel 2024-03-30T04:02:21.817 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/rcu 2024-03-30T04:02:21.818 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/rcu/rcutorture.ko 2024-03-30T04:02:21.818 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/torture.ko 2024-03-30T04:02:21.818 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/trace 2024-03-30T04:02:21.818 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-30T04:02:21.818 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib 2024-03-30T04:02:21.818 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/asn1_encoder.ko 2024-03-30T04:02:21.818 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crc-itu-t.ko 2024-03-30T04:02:21.818 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crc8.ko 2024-03-30T04:02:21.818 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto 2024-03-30T04:02:21.818 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto/libarc4.ko 2024-03-30T04:02:21.819 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto/libdes.ko 2024-03-30T04:02:21.819 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/libcrc32c.ko 2024-03-30T04:02:21.819 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lru_cache.ko 2024-03-30T04:02:21.819 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4 2024-03-30T04:02:21.819 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4/lz4_compress.ko 2024-03-30T04:02:21.819 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4/lz4hc_compress.ko 2024-03-30T04:02:21.819 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/math 2024-03-30T04:02:21.819 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/math/cordic.ko 2024-03-30T04:02:21.819 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/raid6 2024-03-30T04:02:21.819 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/raid6/raid6_pq.ko 2024-03-30T04:02:21.820 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/reed_solomon 2024-03-30T04:02:21.820 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-30T04:02:21.820 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_bm.ko 2024-03-30T04:02:21.820 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_fsm.ko 2024-03-30T04:02:21.820 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_kmp.ko 2024-03-30T04:02:21.820 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/zstd 2024-03-30T04:02:21.820 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/zstd/zstd_compress.ko 2024-03-30T04:02:21.820 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/mm 2024-03-30T04:02:21.820 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/mm/hwpoison-inject.ko 2024-03-30T04:02:21.820 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net 2024-03-30T04:02:21.820 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan 2024-03-30T04:02:21.820 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/6lowpan.ko 2024-03-30T04:02:21.820 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_dest.ko 2024-03-30T04:02:21.820 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_fragment.ko 2024-03-30T04:02:21.821 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_hop.ko 2024-03-30T04:02:21.821 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-30T04:02:21.821 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_mobility.ko 2024-03-30T04:02:21.821 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_routing.ko 2024-03-30T04:02:21.821 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_udp.ko 2024-03-30T04:02:21.821 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802 2024-03-30T04:02:21.821 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/garp.ko 2024-03-30T04:02:21.821 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/mrp.ko 2024-03-30T04:02:21.821 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/p8022.ko 2024-03-30T04:02:21.821 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/psnap.ko 2024-03-30T04:02:21.821 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/stp.ko 2024-03-30T04:02:21.821 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/8021q 2024-03-30T04:02:21.821 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/8021q/8021q.ko 2024-03-30T04:02:21.821 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p 2024-03-30T04:02:21.822 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet.ko 2024-03-30T04:02:21.822 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_fd.ko 2024-03-30T04:02:21.822 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_rdma.ko 2024-03-30T04:02:21.822 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_virtio.ko 2024-03-30T04:02:21.822 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/batman-adv 2024-03-30T04:02:21.822 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/batman-adv/batman-adv.ko 2024-03-30T04:02:21.822 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge 2024-03-30T04:02:21.822 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/br_netfilter.ko 2024-03-30T04:02:21.822 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/bridge.ko 2024-03-30T04:02:21.822 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter 2024-03-30T04:02:21.822 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-30T04:02:21.822 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-30T04:02:21.823 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-30T04:02:21.823 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-30T04:02:21.823 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-30T04:02:21.823 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-30T04:02:21.823 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-30T04:02:21.823 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-30T04:02:21.823 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-30T04:02:21.823 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-30T04:02:21.823 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-30T04:02:21.823 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-30T04:02:21.823 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-30T04:02:21.823 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-30T04:02:21.823 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-30T04:02:21.824 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-30T04:02:21.824 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-30T04:02:21.824 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-30T04:02:21.824 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-30T04:02:21.824 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-30T04:02:21.824 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtables.ko 2024-03-30T04:02:21.824 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-30T04:02:21.824 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-30T04:02:21.824 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-30T04:02:21.824 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ceph 2024-03-30T04:02:21.824 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ceph/libceph.ko 2024-03-30T04:02:21.824 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core 2024-03-30T04:02:21.824 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core/failover.ko 2024-03-30T04:02:21.824 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core/pktgen.ko 2024-03-30T04:02:21.825 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp 2024-03-30T04:02:21.825 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp.ko 2024-03-30T04:02:21.825 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_diag.ko 2024-03-30T04:02:21.825 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_ipv4.ko 2024-03-30T04:02:21.825 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_ipv6.ko 2024-03-30T04:02:21.825 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dns_resolver 2024-03-30T04:02:21.825 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dns_resolver/dns_resolver.ko 2024-03-30T04:02:21.825 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa 2024-03-30T04:02:21.825 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/dsa_core.ko 2024-03-30T04:02:21.825 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_brcm.ko 2024-03-30T04:02:21.825 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_dsa.ko 2024-03-30T04:02:21.825 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_none.ko 2024-03-30T04:02:21.825 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_trailer.ko 2024-03-30T04:02:21.825 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154 2024-03-30T04:02:21.826 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/6lowpan 2024-03-30T04:02:21.826 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-30T04:02:21.826 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/ieee802154.ko 2024-03-30T04:02:21.826 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-30T04:02:21.826 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4 2024-03-30T04:02:21.826 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ah4.ko 2024-03-30T04:02:21.826 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/esp4.ko 2024-03-30T04:02:21.826 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/fou.ko 2024-03-30T04:02:21.826 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/gre.ko 2024-03-30T04:02:21.826 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/inet_diag.ko 2024-03-30T04:02:21.826 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_gre.ko 2024-03-30T04:02:21.826 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_tunnel.ko 2024-03-30T04:02:21.827 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_vti.ko 2024-03-30T04:02:21.827 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ipcomp.ko 2024-03-30T04:02:21.827 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ipip.ko 2024-03-30T04:02:21.827 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter 2024-03-30T04:02:21.827 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-30T04:02:21.827 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-30T04:02:21.827 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-30T04:02:21.827 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-30T04:02:21.827 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-30T04:02:21.827 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-30T04:02:21.827 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-30T04:02:21.827 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-30T04:02:21.827 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-30T04:02:21.827 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-30T04:02:21.828 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-30T04:02:21.828 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-30T04:02:21.828 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-30T04:02:21.828 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-30T04:02:21.828 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-30T04:02:21.828 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-30T04:02:21.828 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-30T04:02:21.828 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-30T04:02:21.828 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-30T04:02:21.828 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-30T04:02:21.828 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-30T04:02:21.828 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_bic.ko 2024-03-30T04:02:21.828 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_dctcp.ko 2024-03-30T04:02:21.829 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_diag.ko 2024-03-30T04:02:21.829 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_highspeed.ko 2024-03-30T04:02:21.829 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_htcp.ko 2024-03-30T04:02:21.829 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_hybla.ko 2024-03-30T04:02:21.829 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_illinois.ko 2024-03-30T04:02:21.829 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_lp.ko 2024-03-30T04:02:21.829 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_scalable.ko 2024-03-30T04:02:21.829 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_vegas.ko 2024-03-30T04:02:21.829 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_veno.ko 2024-03-30T04:02:21.829 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_westwood.ko 2024-03-30T04:02:21.829 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_yeah.ko 2024-03-30T04:02:21.829 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tunnel4.ko 2024-03-30T04:02:21.829 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/udp_diag.ko 2024-03-30T04:02:21.829 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/udp_tunnel.ko 2024-03-30T04:02:21.830 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-30T04:02:21.830 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6 2024-03-30T04:02:21.830 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ah6.ko 2024-03-30T04:02:21.830 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/esp6.ko 2024-03-30T04:02:21.830 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/fou6.ko 2024-03-30T04:02:21.830 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_tunnel.ko 2024-03-30T04:02:21.830 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-30T04:02:21.830 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_vti.ko 2024-03-30T04:02:21.830 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ipcomp6.ko 2024-03-30T04:02:21.830 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter 2024-03-30T04:02:21.830 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-30T04:02:21.830 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-30T04:02:21.830 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-30T04:02:21.830 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-30T04:02:21.831 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-30T04:02:21.831 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-30T04:02:21.831 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-30T04:02:21.831 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-30T04:02:21.831 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-30T04:02:21.831 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-30T04:02:21.831 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-30T04:02:21.831 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-30T04:02:21.831 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-30T04:02:21.831 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-30T04:02:21.831 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-30T04:02:21.831 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-30T04:02:21.831 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-30T04:02:21.831 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-30T04:02:21.832 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-30T04:02:21.832 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-30T04:02:21.832 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-30T04:02:21.832 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-30T04:02:21.832 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-30T04:02:21.832 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-30T04:02:21.832 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/sit.ko 2024-03-30T04:02:21.832 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/tunnel6.ko 2024-03-30T04:02:21.832 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-30T04:02:21.832 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/key 2024-03-30T04:02:21.832 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/key/af_key.ko 2024-03-30T04:02:21.832 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp 2024-03-30T04:02:21.832 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_core.ko 2024-03-30T04:02:21.833 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-30T04:02:21.833 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_eth.ko 2024-03-30T04:02:21.833 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ip.ko 2024-03-30T04:02:21.833 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ip6.ko 2024-03-30T04:02:21.833 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_netlink.ko 2024-03-30T04:02:21.833 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ppp.ko 2024-03-30T04:02:21.833 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/llc 2024-03-30T04:02:21.833 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/llc/llc.ko 2024-03-30T04:02:21.833 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mac802154 2024-03-30T04:02:21.833 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mac802154/mac802154.ko 2024-03-30T04:02:21.833 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mpls 2024-03-30T04:02:21.833 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mpls/mpls_gso.ko 2024-03-30T04:02:21.833 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter 2024-03-30T04:02:21.833 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset 2024-03-30T04:02:21.834 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set.ko 2024-03-30T04:02:21.834 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-30T04:02:21.834 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-30T04:02:21.834 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-30T04:02:21.834 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-30T04:02:21.834 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-30T04:02:21.834 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-30T04:02:21.834 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-30T04:02:21.834 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-30T04:02:21.834 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-30T04:02:21.834 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-30T04:02:21.834 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-30T04:02:21.834 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-30T04:02:21.835 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-30T04:02:21.835 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-30T04:02:21.835 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-30T04:02:21.835 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs 2024-03-30T04:02:21.835 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-30T04:02:21.835 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-30T04:02:21.835 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-30T04:02:21.835 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-30T04:02:21.835 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-30T04:02:21.835 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-30T04:02:21.835 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-30T04:02:21.835 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-30T04:02:21.835 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-30T04:02:21.836 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-30T04:02:21.836 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-30T04:02:21.836 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-30T04:02:21.836 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-30T04:02:21.836 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-30T04:02:21.836 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conncount.ko 2024-03-30T04:02:21.836 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack.ko 2024-03-30T04:02:21.836 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-30T04:02:21.836 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-30T04:02:21.836 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-30T04:02:21.836 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-30T04:02:21.836 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-30T04:02:21.836 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-30T04:02:21.836 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-30T04:02:21.837 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-30T04:02:21.837 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-30T04:02:21.837 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-30T04:02:21.837 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-30T04:02:21.837 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-30T04:02:21.837 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-30T04:02:21.837 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_flow_table.ko 2024-03-30T04:02:21.837 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-30T04:02:21.837 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_log_syslog.ko 2024-03-30T04:02:21.837 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat.ko 2024-03-30T04:02:21.837 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-30T04:02:21.837 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-30T04:02:21.837 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_irc.ko 2024-03-30T04:02:21.838 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_sip.ko 2024-03-30T04:02:21.838 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-30T04:02:21.838 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-30T04:02:21.838 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_tables.ko 2024-03-30T04:02:21.838 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink.ko 2024-03-30T04:02:21.838 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-30T04:02:21.838 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_log.ko 2024-03-30T04:02:21.838 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-30T04:02:21.838 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-30T04:02:21.838 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_chain_nat.ko 2024-03-30T04:02:21.838 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_compat.ko 2024-03-30T04:02:21.838 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_ct.ko 2024-03-30T04:02:21.838 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-30T04:02:21.839 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_fib.ko 2024-03-30T04:02:21.839 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-30T04:02:21.839 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_hash.ko 2024-03-30T04:02:21.839 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_limit.ko 2024-03-30T04:02:21.839 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_log.ko 2024-03-30T04:02:21.839 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_masq.ko 2024-03-30T04:02:21.839 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_nat.ko 2024-03-30T04:02:21.839 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_numgen.ko 2024-03-30T04:02:21.839 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_queue.ko 2024-03-30T04:02:21.839 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_quota.ko 2024-03-30T04:02:21.839 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_redir.ko 2024-03-30T04:02:21.839 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_reject.ko 2024-03-30T04:02:21.839 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_reject_inet.ko 2024-03-30T04:02:21.839 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_socket.ko 2024-03-30T04:02:21.840 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_tunnel.ko 2024-03-30T04:02:21.840 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_AUDIT.ko 2024-03-30T04:02:21.840 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-30T04:02:21.840 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-30T04:02:21.840 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-30T04:02:21.840 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CT.ko 2024-03-30T04:02:21.840 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_DSCP.ko 2024-03-30T04:02:21.840 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_HL.ko 2024-03-30T04:02:21.840 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_HMARK.ko 2024-03-30T04:02:21.840 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-30T04:02:21.840 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_LED.ko 2024-03-30T04:02:21.840 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_LOG.ko 2024-03-30T04:02:21.841 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-30T04:02:21.841 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NETMAP.ko 2024-03-30T04:02:21.841 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NFLOG.ko 2024-03-30T04:02:21.841 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-30T04:02:21.841 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_RATEEST.ko 2024-03-30T04:02:21.841 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-30T04:02:21.841 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_SECMARK.ko 2024-03-30T04:02:21.841 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-30T04:02:21.841 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-30T04:02:21.841 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TEE.ko 2024-03-30T04:02:21.841 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TPROXY.ko 2024-03-30T04:02:21.842 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TRACE.ko 2024-03-30T04:02:21.842 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_addrtype.ko 2024-03-30T04:02:21.842 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_bpf.ko 2024-03-30T04:02:21.842 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cgroup.ko 2024-03-30T04:02:21.842 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cluster.ko 2024-03-30T04:02:21.842 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_comment.ko 2024-03-30T04:02:21.842 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connbytes.ko 2024-03-30T04:02:21.842 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connlabel.ko 2024-03-30T04:02:21.842 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connlimit.ko 2024-03-30T04:02:21.842 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connmark.ko 2024-03-30T04:02:21.842 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_conntrack.ko 2024-03-30T04:02:21.843 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cpu.ko 2024-03-30T04:02:21.843 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_dccp.ko 2024-03-30T04:02:21.843 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_devgroup.ko 2024-03-30T04:02:21.843 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_dscp.ko 2024-03-30T04:02:21.843 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ecn.ko 2024-03-30T04:02:21.843 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_esp.ko 2024-03-30T04:02:21.843 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_hashlimit.ko 2024-03-30T04:02:21.843 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_helper.ko 2024-03-30T04:02:21.843 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_hl.ko 2024-03-30T04:02:21.843 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ipcomp.ko 2024-03-30T04:02:21.843 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_iprange.ko 2024-03-30T04:02:21.843 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ipvs.ko 2024-03-30T04:02:21.843 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_l2tp.ko 2024-03-30T04:02:21.843 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_length.ko 2024-03-30T04:02:21.844 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_limit.ko 2024-03-30T04:02:21.844 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_mac.ko 2024-03-30T04:02:21.844 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_mark.ko 2024-03-30T04:02:21.844 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_multiport.ko 2024-03-30T04:02:21.844 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_nat.ko 2024-03-30T04:02:21.844 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_nfacct.ko 2024-03-30T04:02:21.844 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_osf.ko 2024-03-30T04:02:21.844 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_owner.ko 2024-03-30T04:02:21.844 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_physdev.ko 2024-03-30T04:02:21.844 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_pkttype.ko 2024-03-30T04:02:21.845 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_policy.ko 2024-03-30T04:02:21.845 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_quota.ko 2024-03-30T04:02:21.845 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_rateest.ko 2024-03-30T04:02:21.845 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_realm.ko 2024-03-30T04:02:21.845 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_recent.ko 2024-03-30T04:02:21.845 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_sctp.ko 2024-03-30T04:02:21.845 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_set.ko 2024-03-30T04:02:21.845 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_socket.ko 2024-03-30T04:02:21.845 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_state.ko 2024-03-30T04:02:21.845 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_statistic.ko 2024-03-30T04:02:21.845 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_string.ko 2024-03-30T04:02:21.845 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_tcpmss.ko 2024-03-30T04:02:21.845 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_time.ko 2024-03-30T04:02:21.845 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_u32.ko 2024-03-30T04:02:21.846 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netlink 2024-03-30T04:02:21.846 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netlink/netlink_diag.ko 2024-03-30T04:02:21.846 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/nsh 2024-03-30T04:02:21.846 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/nsh/nsh.ko 2024-03-30T04:02:21.846 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch 2024-03-30T04:02:21.846 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/openvswitch.ko 2024-03-30T04:02:21.846 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-geneve.ko 2024-03-30T04:02:21.846 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-gre.ko 2024-03-30T04:02:21.847 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-vxlan.ko 2024-03-30T04:02:21.847 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/packet 2024-03-30T04:02:21.847 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/packet/af_packet_diag.ko 2024-03-30T04:02:21.847 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds 2024-03-30T04:02:21.847 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds.ko 2024-03-30T04:02:21.847 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds_rdma.ko 2024-03-30T04:02:21.847 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds_tcp.ko 2024-03-30T04:02:21.847 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched 2024-03-30T04:02:21.847 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_csum.ko 2024-03-30T04:02:21.847 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_gact.ko 2024-03-30T04:02:21.847 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_mirred.ko 2024-03-30T04:02:21.847 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_nat.ko 2024-03-30T04:02:21.847 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_pedit.ko 2024-03-30T04:02:21.848 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_police.ko 2024-03-30T04:02:21.848 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_simple.ko 2024-03-30T04:02:21.848 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_skbedit.ko 2024-03-30T04:02:21.848 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_vlan.ko 2024-03-30T04:02:21.848 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_basic.ko 2024-03-30T04:02:21.848 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_bpf.ko 2024-03-30T04:02:21.848 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_flow.ko 2024-03-30T04:02:21.848 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_fw.ko 2024-03-30T04:02:21.848 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_route.ko 2024-03-30T04:02:21.848 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_u32.ko 2024-03-30T04:02:21.849 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_cmp.ko 2024-03-30T04:02:21.849 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_ipset.ko 2024-03-30T04:02:21.849 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_meta.ko 2024-03-30T04:02:21.849 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_nbyte.ko 2024-03-30T04:02:21.849 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_text.ko 2024-03-30T04:02:21.849 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_u32.ko 2024-03-30T04:02:21.849 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_choke.ko 2024-03-30T04:02:21.849 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_codel.ko 2024-03-30T04:02:21.849 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_drr.ko 2024-03-30T04:02:21.849 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_fq.ko 2024-03-30T04:02:21.850 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_gred.ko 2024-03-30T04:02:21.850 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_hfsc.ko 2024-03-30T04:02:21.850 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_hhf.ko 2024-03-30T04:02:21.850 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_htb.ko 2024-03-30T04:02:21.850 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_ingress.ko 2024-03-30T04:02:21.850 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_mqprio.ko 2024-03-30T04:02:21.850 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_mqprio_lib.ko 2024-03-30T04:02:21.850 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_multiq.ko 2024-03-30T04:02:21.850 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_netem.ko 2024-03-30T04:02:21.851 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_pie.ko 2024-03-30T04:02:21.851 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_plug.ko 2024-03-30T04:02:21.851 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_prio.ko 2024-03-30T04:02:21.851 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_qfq.ko 2024-03-30T04:02:21.851 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_red.ko 2024-03-30T04:02:21.851 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_sfb.ko 2024-03-30T04:02:21.851 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_sfq.ko 2024-03-30T04:02:21.851 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_tbf.ko 2024-03-30T04:02:21.851 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_teql.ko 2024-03-30T04:02:21.851 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp 2024-03-30T04:02:21.852 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp/sctp.ko 2024-03-30T04:02:21.852 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp/sctp_diag.ko 2024-03-30T04:02:21.852 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc 2024-03-30T04:02:21.852 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss 2024-03-30T04:02:21.852 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-30T04:02:21.852 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-30T04:02:21.852 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/sunrpc.ko 2024-03-30T04:02:21.852 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/xprtrdma 2024-03-30T04:02:21.852 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-30T04:02:21.852 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc 2024-03-30T04:02:21.853 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc/diag.ko 2024-03-30T04:02:21.853 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc/tipc.ko 2024-03-30T04:02:21.853 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/unix 2024-03-30T04:02:21.853 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/unix/unix_diag.ko 2024-03-30T04:02:21.853 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock 2024-03-30T04:02:21.853 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-30T04:02:21.853 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-30T04:02:21.853 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock.ko 2024-03-30T04:02:21.853 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-30T04:02:21.854 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-30T04:02:21.854 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm 2024-03-30T04:02:21.854 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-30T04:02:21.854 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security 2024-03-30T04:02:21.854 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys 2024-03-30T04:02:21.854 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/encrypted-keys 2024-03-30T04:02:21.854 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-30T04:02:21.854 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/trusted-keys 2024-03-30T04:02:21.854 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/trusted-keys/trusted.ko 2024-03-30T04:02:21.854 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt 2024-03-30T04:02:21.855 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt/lib 2024-03-30T04:02:21.855 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt/lib/irqbypass.ko 2024-03-30T04:02:21.855 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.alias 2024-03-30T04:02:21.855 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.alias.bin 2024-03-30T04:02:21.855 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin 2024-03-30T04:02:21.855 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.bin 2024-03-30T04:02:21.855 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.modinfo 2024-03-30T04:02:21.855 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.dep 2024-03-30T04:02:21.855 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.dep.bin 2024-03-30T04:02:21.855 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.devname 2024-03-30T04:02:21.855 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.order 2024-03-30T04:02:21.855 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.softdep 2024-03-30T04:02:21.855 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.symbols 2024-03-30T04:02:21.855 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.symbols.bin 2024-03-30T04:02:21.855 INFO:teuthology.orchestra.run.smithi135.stdout:/lib/modules/6.8.0-g1df0c7e8a235/vmlinuz 2024-03-30T04:02:21.859 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-g1df0c7e8a235 2024-03-30T04:02:21.860 DEBUG:teuthology.orchestra.run.smithi135:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-30T04:02:21.886 INFO:teuthology.orchestra.run.smithi135.stdout:postinstall scriptlet (using /bin/sh): 2024-03-30T04:02:21.886 INFO:teuthology.orchestra.run.smithi135.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-30T04:02:21.886 INFO:teuthology.orchestra.run.smithi135.stdout: /usr/bin/kernel-install add 6.8.0-g1df0c7e8a235 /lib/modules/6.8.0-g1df0c7e8a235/vmlinuz 2024-03-30T04:02:21.886 INFO:teuthology.orchestra.run.smithi135.stdout:fi 2024-03-30T04:02:21.886 INFO:teuthology.orchestra.run.smithi135.stdout:for file in vmlinuz System.map config; do 2024-03-30T04:02:21.886 INFO:teuthology.orchestra.run.smithi135.stdout: if ! cmp --silent "/lib/modules/6.8.0-g1df0c7e8a235/${file}" "/boot/${file}-6.8.0-g1df0c7e8a235"; then 2024-03-30T04:02:21.886 INFO:teuthology.orchestra.run.smithi135.stdout: cp "/lib/modules/6.8.0-g1df0c7e8a235/${file}" "/boot/${file}-6.8.0-g1df0c7e8a235" 2024-03-30T04:02:21.886 INFO:teuthology.orchestra.run.smithi135.stdout: fi 2024-03-30T04:02:21.887 INFO:teuthology.orchestra.run.smithi135.stdout:done 2024-03-30T04:02:21.887 INFO:teuthology.orchestra.run.smithi135.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-30T04:02:21.887 INFO:teuthology.orchestra.run.smithi135.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-30T04:02:21.887 INFO:teuthology.orchestra.run.smithi135.stdout:new-kernel-pkg --remove 6.8.0-g1df0c7e8a235 --rminitrd --initrdfile=/boot/initramfs-6.8.0-g1df0c7e8a235.img 2024-03-30T04:02:21.887 INFO:teuthology.orchestra.run.smithi135.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-30T04:02:21.887 INFO:teuthology.orchestra.run.smithi135.stdout:kernel-install remove 6.8.0-g1df0c7e8a235 2024-03-30T04:02:21.887 INFO:teuthology.orchestra.run.smithi135.stdout:fi 2024-03-30T04:02:21.887 INFO:teuthology.orchestra.run.smithi135.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-30T04:02:21.887 INFO:teuthology.orchestra.run.smithi135.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-30T04:02:21.887 INFO:teuthology.orchestra.run.smithi135.stdout:/sbin/update-bootloader --remove 6.8.0-g1df0c7e8a235 2024-03-30T04:02:21.887 INFO:teuthology.orchestra.run.smithi135.stdout:fi 2024-03-30T04:02:21.887 DEBUG:teuthology.orchestra.run.smithi135:> sudo rpm -qi grub2-tools 2024-03-30T04:02:21.963 INFO:teuthology.orchestra.run.smithi135.stdout:Name : grub2-tools 2024-03-30T04:02:21.963 INFO:teuthology.orchestra.run.smithi135.stdout:Epoch : 1 2024-03-30T04:02:21.963 INFO:teuthology.orchestra.run.smithi135.stdout:Version : 2.02 2024-03-30T04:02:21.964 INFO:teuthology.orchestra.run.smithi135.stdout:Release : 129.el8 2024-03-30T04:02:21.964 INFO:teuthology.orchestra.run.smithi135.stdout:Architecture: x86_64 2024-03-30T04:02:21.964 INFO:teuthology.orchestra.run.smithi135.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-03-30T04:02:21.964 INFO:teuthology.orchestra.run.smithi135.stdout:Group : System Environment/Base 2024-03-30T04:02:21.964 INFO:teuthology.orchestra.run.smithi135.stdout:Size : 9265586 2024-03-30T04:02:21.964 INFO:teuthology.orchestra.run.smithi135.stdout:License : GPLv3+ 2024-03-30T04:02:21.964 INFO:teuthology.orchestra.run.smithi135.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-03-30T04:02:21.964 INFO:teuthology.orchestra.run.smithi135.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-03-30T04:02:21.964 INFO:teuthology.orchestra.run.smithi135.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-03-30T04:02:21.964 INFO:teuthology.orchestra.run.smithi135.stdout:Build Host : kbuilder.bsys.centos.org 2024-03-30T04:02:21.964 INFO:teuthology.orchestra.run.smithi135.stdout:Relocations : (not relocatable) 2024-03-30T04:02:21.965 INFO:teuthology.orchestra.run.smithi135.stdout:Packager : CentOS BuildSystem 2024-03-30T04:02:21.965 INFO:teuthology.orchestra.run.smithi135.stdout:Vendor : CentOS 2024-03-30T04:02:21.965 INFO:teuthology.orchestra.run.smithi135.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-30T04:02:21.965 INFO:teuthology.orchestra.run.smithi135.stdout:Summary : Support tools for GRUB. 2024-03-30T04:02:21.965 INFO:teuthology.orchestra.run.smithi135.stdout:Description : 2024-03-30T04:02:21.965 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:02:21.965 INFO:teuthology.orchestra.run.smithi135.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-30T04:02:21.965 INFO:teuthology.orchestra.run.smithi135.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-30T04:02:21.965 INFO:teuthology.orchestra.run.smithi135.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-30T04:02:21.965 INFO:teuthology.orchestra.run.smithi135.stdout:hardware devices. 2024-03-30T04:02:21.965 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:02:21.965 INFO:teuthology.orchestra.run.smithi135.stdout:This subpackage provides tools for support of all platforms. 2024-03-30T04:02:21.966 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-30T04:02:21.967 INFO:teuthology.task.kernel:Updating grub on smithi135 to boot 6.8.0-g1df0c7e8a235 2024-03-30T04:02:21.967 DEBUG:teuthology.orchestra.run.smithi135:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-30T04:02:27.150 INFO:teuthology.orchestra.run.smithi135.stderr:Generating grub configuration file ... 2024-03-30T04:02:29.626 INFO:teuthology.orchestra.run.smithi135.stderr:done 2024-03-30T04:02:29.627 DEBUG:teuthology.orchestra.run.smithi135:> mktemp 2024-03-30T04:02:29.651 INFO:teuthology.orchestra.run.smithi135.stdout:/tmp/tmp.aoauA9Ifvn 2024-03-30T04:02:29.651 DEBUG:teuthology.orchestra.run.smithi135:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.aoauA9Ifvn 2024-03-30T04:02:29.725 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod 0666 /tmp/tmp.aoauA9Ifvn 2024-03-30T04:02:29.895 DEBUG:teuthology.orchestra.remote:smithi135:/tmp/tmp.aoauA9Ifvn is 6KB 2024-03-30T04:02:29.945 DEBUG:teuthology.orchestra.run.smithi135:> rm -fr /tmp/tmp.aoauA9Ifvn 2024-03-30T04:02:29.962 DEBUG:teuthology.orchestra.run.smithi135:> sudo /bin/ls /boot/loader/entries || true 2024-03-30T04:02:30.034 INFO:teuthology.orchestra.run.smithi135.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-03-30T04:02:30.034 INFO:teuthology.orchestra.run.smithi135.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-03-30T04:02:30.035 INFO:teuthology.orchestra.run.smithi135.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-03-30T04:02:30.035 INFO:teuthology.orchestra.run.smithi135.stdout:a1b29d6c289747bdb22930598d39874a-6.8.0-g1df0c7e8a235.conf 2024-03-30T04:02:30.036 DEBUG:teuthology.orchestra.run.smithi135:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-6.8.0-g1df0c7e8a235 2024-03-30T04:02:30.207 DEBUG:teuthology.orchestra.run.smithi135:> sudo shutdown -r now 2024-03-30T04:02:41.530 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi123.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-30T04:02:41.531 DEBUG:teuthology.orchestra.run.smithi123:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-30T04:02:41.595 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235 2024-03-30T04:02:41.595 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/System.map 2024-03-30T04:02:41.595 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/config 2024-03-30T04:02:41.595 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel 2024-03-30T04:02:41.595 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch 2024-03-30T04:02:41.595 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86 2024-03-30T04:02:41.595 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto 2024-03-30T04:02:41.595 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-30T04:02:41.595 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-30T04:02:41.596 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-30T04:02:41.596 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-30T04:02:41.596 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-30T04:02:41.596 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-30T04:02:41.596 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-30T04:02:41.596 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-30T04:02:41.596 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-30T04:02:41.596 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-30T04:02:41.596 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-30T04:02:41.596 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-30T04:02:41.596 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-30T04:02:41.596 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-30T04:02:41.597 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-30T04:02:41.597 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-30T04:02:41.597 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-30T04:02:41.597 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-30T04:02:41.597 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-30T04:02:41.597 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-30T04:02:41.597 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel 2024-03-30T04:02:41.597 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu 2024-03-30T04:02:41.597 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu/mce 2024-03-30T04:02:41.597 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-30T04:02:41.597 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm 2024-03-30T04:02:41.597 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-30T04:02:41.597 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-30T04:02:41.598 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm.ko 2024-03-30T04:02:41.598 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto 2024-03-30T04:02:41.598 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/ansi_cprng.ko 2024-03-30T04:02:41.598 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/anubis.ko 2024-03-30T04:02:41.598 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/arc4.ko 2024-03-30T04:02:41.598 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx 2024-03-30T04:02:41.598 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_memcpy.ko 2024-03-30T04:02:41.598 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_pq.ko 2024-03-30T04:02:41.598 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-30T04:02:41.598 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_tx.ko 2024-03-30T04:02:41.598 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_xor.ko 2024-03-30T04:02:41.598 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/raid6test.ko 2024-03-30T04:02:41.598 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/authenc.ko 2024-03-30T04:02:41.599 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/authencesn.ko 2024-03-30T04:02:41.599 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blake2b_generic.ko 2024-03-30T04:02:41.599 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blowfish_common.ko 2024-03-30T04:02:41.599 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blowfish_generic.ko 2024-03-30T04:02:41.599 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/camellia_generic.ko 2024-03-30T04:02:41.599 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast5_generic.ko 2024-03-30T04:02:41.599 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast6_generic.ko 2024-03-30T04:02:41.599 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast_common.ko 2024-03-30T04:02:41.599 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/ccm.ko 2024-03-30T04:02:41.599 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cmac.ko 2024-03-30T04:02:41.599 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crc32_generic.ko 2024-03-30T04:02:41.599 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crypto_engine.ko 2024-03-30T04:02:41.599 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crypto_user.ko 2024-03-30T04:02:41.599 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cts.ko 2024-03-30T04:02:41.600 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/des_generic.ko 2024-03-30T04:02:41.600 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/dh_generic.ko 2024-03-30T04:02:41.600 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/echainiv.ko 2024-03-30T04:02:41.600 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/essiv.ko 2024-03-30T04:02:41.600 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/fcrypt.ko 2024-03-30T04:02:41.600 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/khazad.ko 2024-03-30T04:02:41.600 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/lz4.ko 2024-03-30T04:02:41.600 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/lz4hc.ko 2024-03-30T04:02:41.600 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/md4.ko 2024-03-30T04:02:41.600 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/michael_mic.ko 2024-03-30T04:02:41.600 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/pcbc.ko 2024-03-30T04:02:41.600 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/pcrypt.ko 2024-03-30T04:02:41.600 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/rmd160.ko 2024-03-30T04:02:41.601 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/seed.ko 2024-03-30T04:02:41.601 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/serpent_generic.ko 2024-03-30T04:02:41.601 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/tcrypt.ko 2024-03-30T04:02:41.601 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/tea.ko 2024-03-30T04:02:41.601 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/twofish_common.ko 2024-03-30T04:02:41.601 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/twofish_generic.ko 2024-03-30T04:02:41.601 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/vmac.ko 2024-03-30T04:02:41.601 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/wp512.ko 2024-03-30T04:02:41.601 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xcbc.ko 2024-03-30T04:02:41.601 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xor.ko 2024-03-30T04:02:41.601 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xxhash_generic.ko 2024-03-30T04:02:41.601 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers 2024-03-30T04:02:41.602 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi 2024-03-30T04:02:41.602 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-30T04:02:41.602 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/acpi_pad.ko 2024-03-30T04:02:41.602 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/custom_method.ko 2024-03-30T04:02:41.602 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/ec_sys.ko 2024-03-30T04:02:41.602 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/platform_profile.ko 2024-03-30T04:02:41.602 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/sbs.ko 2024-03-30T04:02:41.602 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/sbshc.ko 2024-03-30T04:02:41.602 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/video.ko 2024-03-30T04:02:41.602 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata 2024-03-30T04:02:41.602 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/acard-ahci.ko 2024-03-30T04:02:41.602 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/ahci_platform.ko 2024-03-30T04:02:41.602 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/ata_generic.ko 2024-03-30T04:02:41.602 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/libahci_platform.ko 2024-03-30T04:02:41.603 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_acpi.ko 2024-03-30T04:02:41.603 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ali.ko 2024-03-30T04:02:41.603 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_amd.ko 2024-03-30T04:02:41.603 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_artop.ko 2024-03-30T04:02:41.603 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_atiixp.ko 2024-03-30T04:02:41.603 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_atp867x.ko 2024-03-30T04:02:41.603 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cmd640.ko 2024-03-30T04:02:41.603 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cmd64x.ko 2024-03-30T04:02:41.603 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cypress.ko 2024-03-30T04:02:41.603 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_efar.ko 2024-03-30T04:02:41.603 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt366.ko 2024-03-30T04:02:41.603 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt37x.ko 2024-03-30T04:02:41.603 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-30T04:02:41.603 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-30T04:02:41.604 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_it8213.ko 2024-03-30T04:02:41.604 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_it821x.ko 2024-03-30T04:02:41.604 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_jmicron.ko 2024-03-30T04:02:41.604 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_marvell.ko 2024-03-30T04:02:41.604 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_mpiix.ko 2024-03-30T04:02:41.604 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_netcell.ko 2024-03-30T04:02:41.604 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ninja32.ko 2024-03-30T04:02:41.604 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ns87410.ko 2024-03-30T04:02:41.604 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ns87415.ko 2024-03-30T04:02:41.604 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_oldpiix.ko 2024-03-30T04:02:41.604 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_opti.ko 2024-03-30T04:02:41.604 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_optidma.ko 2024-03-30T04:02:41.604 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pcmcia.ko 2024-03-30T04:02:41.604 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-30T04:02:41.605 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-30T04:02:41.605 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_piccolo.ko 2024-03-30T04:02:41.605 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_rdc.ko 2024-03-30T04:02:41.605 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sch.ko 2024-03-30T04:02:41.605 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_serverworks.ko 2024-03-30T04:02:41.605 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sil680.ko 2024-03-30T04:02:41.605 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sis.ko 2024-03-30T04:02:41.605 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sl82c105.ko 2024-03-30T04:02:41.605 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_triflex.ko 2024-03-30T04:02:41.605 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_via.ko 2024-03-30T04:02:41.605 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pdc_adma.ko 2024-03-30T04:02:41.605 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_inic162x.ko 2024-03-30T04:02:41.606 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_mv.ko 2024-03-30T04:02:41.606 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_nv.ko 2024-03-30T04:02:41.606 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_promise.ko 2024-03-30T04:02:41.606 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_qstor.ko 2024-03-30T04:02:41.606 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sil.ko 2024-03-30T04:02:41.606 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sil24.ko 2024-03-30T04:02:41.606 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sis.ko 2024-03-30T04:02:41.606 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_svw.ko 2024-03-30T04:02:41.606 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sx4.ko 2024-03-30T04:02:41.606 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_uli.ko 2024-03-30T04:02:41.606 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_via.ko 2024-03-30T04:02:41.606 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_vsc.ko 2024-03-30T04:02:41.606 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/auxdisplay 2024-03-30T04:02:41.607 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/auxdisplay/ks0108.ko 2024-03-30T04:02:41.607 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base 2024-03-30T04:02:41.607 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap 2024-03-30T04:02:41.607 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-30T04:02:41.607 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-30T04:02:41.607 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/bcma 2024-03-30T04:02:41.607 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/bcma/bcma.ko 2024-03-30T04:02:41.607 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block 2024-03-30T04:02:41.607 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/aoe 2024-03-30T04:02:41.607 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/aoe/aoe.ko 2024-03-30T04:02:41.607 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/brd.ko 2024-03-30T04:02:41.607 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/drbd 2024-03-30T04:02:41.607 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/drbd/drbd.ko 2024-03-30T04:02:41.607 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/floppy.ko 2024-03-30T04:02:41.608 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/loop.ko 2024-03-30T04:02:41.608 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/mtip32xx 2024-03-30T04:02:41.608 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-30T04:02:41.608 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/nbd.ko 2024-03-30T04:02:41.608 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/null_blk 2024-03-30T04:02:41.608 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/null_blk/null_blk.ko 2024-03-30T04:02:41.608 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/pktcdvd.ko 2024-03-30T04:02:41.608 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/rbd.ko 2024-03-30T04:02:41.608 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/virtio_blk.ko 2024-03-30T04:02:41.608 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkback 2024-03-30T04:02:41.608 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-30T04:02:41.608 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkfront.ko 2024-03-30T04:02:41.608 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/zram 2024-03-30T04:02:41.609 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/zram/zram.ko 2024-03-30T04:02:41.609 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char 2024-03-30T04:02:41.609 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hangcheck-timer.ko 2024-03-30T04:02:41.609 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random 2024-03-30T04:02:41.609 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-30T04:02:41.609 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-30T04:02:41.609 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-30T04:02:41.609 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/via-rng.ko 2024-03-30T04:02:41.609 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-30T04:02:41.609 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi 2024-03-30T04:02:41.609 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-30T04:02:41.609 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-30T04:02:41.609 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-30T04:02:41.609 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-30T04:02:41.610 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-30T04:02:41.610 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-30T04:02:41.610 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/lp.ko 2024-03-30T04:02:41.610 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/mwave 2024-03-30T04:02:41.610 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/mwave/mwave.ko 2024-03-30T04:02:41.610 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ppdev.ko 2024-03-30T04:02:41.610 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tlclk.ko 2024-03-30T04:02:41.610 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm 2024-03-30T04:02:41.610 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm.ko 2024-03-30T04:02:41.610 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-30T04:02:41.610 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-30T04:02:41.610 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-30T04:02:41.611 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-30T04:02:41.611 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-30T04:02:41.611 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/uv_mmtimer.ko 2024-03-30T04:02:41.611 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/virtio_console.ko 2024-03-30T04:02:41.611 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq 2024-03-30T04:02:41.611 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-30T04:02:41.611 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-30T04:02:41.611 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-30T04:02:41.611 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-30T04:02:41.611 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-30T04:02:41.611 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-30T04:02:41.611 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto 2024-03-30T04:02:41.611 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp 2024-03-30T04:02:41.611 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-30T04:02:41.612 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp/ccp.ko 2024-03-30T04:02:41.612 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel 2024-03-30T04:02:41.612 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat 2024-03-30T04:02:41.612 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_common 2024-03-30T04:02:41.612 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-30T04:02:41.612 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-30T04:02:41.612 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-30T04:02:41.612 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/padlock-aes.ko 2024-03-30T04:02:41.612 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/padlock-sha.ko 2024-03-30T04:02:41.612 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/virtio 2024-03-30T04:02:41.612 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-30T04:02:41.612 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dca 2024-03-30T04:02:41.612 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dca/dca.ko 2024-03-30T04:02:41.613 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/devfreq 2024-03-30T04:02:41.613 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-30T04:02:41.613 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma 2024-03-30T04:02:41.613 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/dw 2024-03-30T04:02:41.613 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-30T04:02:41.613 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/ioat 2024-03-30T04:02:41.613 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-30T04:02:41.613 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac 2024-03-30T04:02:41.613 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/amd64_edac.ko 2024-03-30T04:02:41.613 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/e752x_edac.ko 2024-03-30T04:02:41.613 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/edac_mce_amd.ko 2024-03-30T04:02:41.613 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i3000_edac.ko 2024-03-30T04:02:41.613 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i3200_edac.ko 2024-03-30T04:02:41.613 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i5100_edac.ko 2024-03-30T04:02:41.613 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i5400_edac.ko 2024-03-30T04:02:41.614 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i7300_edac.ko 2024-03-30T04:02:41.614 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i7core_edac.ko 2024-03-30T04:02:41.614 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i82975x_edac.ko 2024-03-30T04:02:41.614 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/ie31200_edac.ko 2024-03-30T04:02:41.614 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/sb_edac.ko 2024-03-30T04:02:41.614 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/x38_edac.ko 2024-03-30T04:02:41.614 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware 2024-03-30T04:02:41.614 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/edd.ko 2024-03-30T04:02:41.614 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-30T04:02:41.614 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio 2024-03-30T04:02:41.614 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-ich.ko 2024-03-30T04:02:41.614 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-30T04:02:41.615 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid 2024-03-30T04:02:41.615 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-appleir.ko 2024-03-30T04:02:41.615 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-aureal.ko 2024-03-30T04:02:41.615 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-axff.ko 2024-03-30T04:02:41.615 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-dr.ko 2024-03-30T04:02:41.615 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-elecom.ko 2024-03-30T04:02:41.615 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-elo.ko 2024-03-30T04:02:41.615 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-emsff.ko 2024-03-30T04:02:41.615 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gaff.ko 2024-03-30T04:02:41.615 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gt683r.ko 2024-03-30T04:02:41.615 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gyration.ko 2024-03-30T04:02:41.615 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-30T04:02:41.615 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-30T04:02:41.615 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtekff.ko 2024-03-30T04:02:41.616 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-hyperv.ko 2024-03-30T04:02:41.616 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-icade.ko 2024-03-30T04:02:41.616 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-keytouch.ko 2024-03-30T04:02:41.616 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-kye.ko 2024-03-30T04:02:41.616 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-lcpower.ko 2024-03-30T04:02:41.616 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-led.ko 2024-03-30T04:02:41.616 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-lenovo.ko 2024-03-30T04:02:41.616 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-30T04:02:41.616 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-30T04:02:41.616 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-multitouch.ko 2024-03-30T04:02:41.616 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-ortek.ko 2024-03-30T04:02:41.616 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-penmount.ko 2024-03-30T04:02:41.616 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-petalynx.ko 2024-03-30T04:02:41.616 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-picolcd.ko 2024-03-30T04:02:41.617 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-pl.ko 2024-03-30T04:02:41.617 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-plantronics.ko 2024-03-30T04:02:41.617 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-primax.ko 2024-03-30T04:02:41.617 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-rmi.ko 2024-03-30T04:02:41.617 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-30T04:02:41.617 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-common.ko 2024-03-30T04:02:41.617 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-30T04:02:41.617 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-30T04:02:41.617 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-30T04:02:41.617 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-30T04:02:41.617 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-30T04:02:41.617 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-30T04:02:41.617 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-30T04:02:41.617 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-30T04:02:41.618 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-30T04:02:41.618 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat.ko 2024-03-30T04:02:41.618 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-saitek.ko 2024-03-30T04:02:41.618 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-samsung.ko 2024-03-30T04:02:41.618 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-30T04:02:41.618 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sjoy.ko 2024-03-30T04:02:41.618 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sony.ko 2024-03-30T04:02:41.618 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-speedlink.ko 2024-03-30T04:02:41.618 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-steelseries.ko 2024-03-30T04:02:41.618 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sunplus.ko 2024-03-30T04:02:41.618 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-30T04:02:41.618 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-tivo.ko 2024-03-30T04:02:41.618 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-tmff.ko 2024-03-30T04:02:41.618 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-topseed.ko 2024-03-30T04:02:41.619 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-twinhan.ko 2024-03-30T04:02:41.619 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-uclogic.ko 2024-03-30T04:02:41.619 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-waltop.ko 2024-03-30T04:02:41.619 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-wiimote.ko 2024-03-30T04:02:41.619 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-xinmo.ko 2024-03-30T04:02:41.619 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-zpff.ko 2024-03-30T04:02:41.619 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-zydacron.ko 2024-03-30T04:02:41.619 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/uhid.ko 2024-03-30T04:02:41.619 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/wacom.ko 2024-03-30T04:02:41.619 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv 2024-03-30T04:02:41.619 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_balloon.ko 2024-03-30T04:02:41.619 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_utils.ko 2024-03-30T04:02:41.619 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_vmbus.ko 2024-03-30T04:02:41.620 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon 2024-03-30T04:02:41.620 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/abituguru.ko 2024-03-30T04:02:41.620 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/abituguru3.ko 2024-03-30T04:02:41.620 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-30T04:02:41.620 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ad7414.ko 2024-03-30T04:02:41.620 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ad7418.ko 2024-03-30T04:02:41.620 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adc128d818.ko 2024-03-30T04:02:41.620 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1025.ko 2024-03-30T04:02:41.620 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1026.ko 2024-03-30T04:02:41.620 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1029.ko 2024-03-30T04:02:41.620 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1031.ko 2024-03-30T04:02:41.620 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm9240.ko 2024-03-30T04:02:41.620 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ads7828.ko 2024-03-30T04:02:41.620 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7410.ko 2024-03-30T04:02:41.621 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7411.ko 2024-03-30T04:02:41.621 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7462.ko 2024-03-30T04:02:41.621 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7470.ko 2024-03-30T04:02:41.621 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7475.ko 2024-03-30T04:02:41.621 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7x10.ko 2024-03-30T04:02:41.621 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/amc6821.ko 2024-03-30T04:02:41.621 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/applesmc.ko 2024-03-30T04:02:41.621 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asb100.ko 2024-03-30T04:02:41.621 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asc7621.ko 2024-03-30T04:02:41.621 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-30T04:02:41.621 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/atxp1.ko 2024-03-30T04:02:41.621 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/coretemp.ko 2024-03-30T04:02:41.621 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-30T04:02:41.622 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/dme1737.ko 2024-03-30T04:02:41.622 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ds1621.ko 2024-03-30T04:02:41.622 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ds620.ko 2024-03-30T04:02:41.622 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/emc1403.ko 2024-03-30T04:02:41.622 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/emc6w201.ko 2024-03-30T04:02:41.622 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f71805f.ko 2024-03-30T04:02:41.622 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f71882fg.ko 2024-03-30T04:02:41.622 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f75375s.ko 2024-03-30T04:02:41.622 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/fam15h_power.ko 2024-03-30T04:02:41.622 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/fschmd.ko 2024-03-30T04:02:41.622 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/g760a.ko 2024-03-30T04:02:41.622 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/g762.ko 2024-03-30T04:02:41.622 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/gl518sm.ko 2024-03-30T04:02:41.623 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/gl520sm.ko 2024-03-30T04:02:41.623 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-30T04:02:41.623 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/i5500_temp.ko 2024-03-30T04:02:41.623 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/i5k_amb.ko 2024-03-30T04:02:41.623 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ibmaem.ko 2024-03-30T04:02:41.623 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ibmpex.ko 2024-03-30T04:02:41.623 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ina209.ko 2024-03-30T04:02:41.623 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ina2xx.ko 2024-03-30T04:02:41.623 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/it87.ko 2024-03-30T04:02:41.623 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/k10temp.ko 2024-03-30T04:02:41.623 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/k8temp.ko 2024-03-30T04:02:41.623 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lineage-pem.ko 2024-03-30T04:02:41.623 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm63.ko 2024-03-30T04:02:41.624 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm73.ko 2024-03-30T04:02:41.624 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm75.ko 2024-03-30T04:02:41.624 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm77.ko 2024-03-30T04:02:41.624 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm78.ko 2024-03-30T04:02:41.624 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm80.ko 2024-03-30T04:02:41.624 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm83.ko 2024-03-30T04:02:41.624 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm85.ko 2024-03-30T04:02:41.624 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm87.ko 2024-03-30T04:02:41.624 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm90.ko 2024-03-30T04:02:41.624 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm92.ko 2024-03-30T04:02:41.624 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm93.ko 2024-03-30T04:02:41.624 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95234.ko 2024-03-30T04:02:41.624 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95241.ko 2024-03-30T04:02:41.624 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95245.ko 2024-03-30T04:02:41.625 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc2945.ko 2024-03-30T04:02:41.625 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4151.ko 2024-03-30T04:02:41.625 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4215.ko 2024-03-30T04:02:41.625 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4222.ko 2024-03-30T04:02:41.625 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4245.ko 2024-03-30T04:02:41.625 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4260.ko 2024-03-30T04:02:41.625 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4261.ko 2024-03-30T04:02:41.625 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max16065.ko 2024-03-30T04:02:41.625 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max1619.ko 2024-03-30T04:02:41.625 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max1668.ko 2024-03-30T04:02:41.625 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max197.ko 2024-03-30T04:02:41.625 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6639.ko 2024-03-30T04:02:41.626 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6650.ko 2024-03-30T04:02:41.626 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6697.ko 2024-03-30T04:02:41.626 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/mcp3021.ko 2024-03-30T04:02:41.626 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6683.ko 2024-03-30T04:02:41.626 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6775-core.ko 2024-03-30T04:02:41.626 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6775.ko 2024-03-30T04:02:41.626 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct7802.ko 2024-03-30T04:02:41.626 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pc87360.ko 2024-03-30T04:02:41.626 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pc87427.ko 2024-03-30T04:02:41.626 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pcf8591.ko 2024-03-30T04:02:41.626 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus 2024-03-30T04:02:41.626 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-30T04:02:41.626 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-30T04:02:41.626 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-30T04:02:41.627 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-30T04:02:41.627 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-30T04:02:41.627 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-30T04:02:41.627 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-30T04:02:41.627 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-30T04:02:41.627 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-30T04:02:41.627 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-30T04:02:41.627 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-30T04:02:41.627 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-30T04:02:41.627 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/powr1220.ko 2024-03-30T04:02:41.627 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch5627.ko 2024-03-30T04:02:41.627 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch5636.ko 2024-03-30T04:02:41.627 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-30T04:02:41.628 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sht15.ko 2024-03-30T04:02:41.628 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sht21.ko 2024-03-30T04:02:41.628 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/shtc1.ko 2024-03-30T04:02:41.628 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sis5595.ko 2024-03-30T04:02:41.628 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47b397.ko 2024-03-30T04:02:41.628 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47m1.ko 2024-03-30T04:02:41.628 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47m192.ko 2024-03-30T04:02:41.628 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/thmc50.ko 2024-03-30T04:02:41.628 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp102.ko 2024-03-30T04:02:41.628 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp103.ko 2024-03-30T04:02:41.628 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp401.ko 2024-03-30T04:02:41.628 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp421.ko 2024-03-30T04:02:41.628 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/via-cputemp.ko 2024-03-30T04:02:41.628 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/via686a.ko 2024-03-30T04:02:41.629 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/vt1211.ko 2024-03-30T04:02:41.629 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/vt8231.ko 2024-03-30T04:02:41.629 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83627ehf.ko 2024-03-30T04:02:41.629 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83627hf.ko 2024-03-30T04:02:41.629 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83781d.ko 2024-03-30T04:02:41.629 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83791d.ko 2024-03-30T04:02:41.629 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83792d.ko 2024-03-30T04:02:41.629 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83793.ko 2024-03-30T04:02:41.629 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83795.ko 2024-03-30T04:02:41.629 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83l785ts.ko 2024-03-30T04:02:41.629 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83l786ng.ko 2024-03-30T04:02:41.629 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c 2024-03-30T04:02:41.629 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos 2024-03-30T04:02:41.629 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-30T04:02:41.630 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-30T04:02:41.630 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses 2024-03-30T04:02:41.630 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-30T04:02:41.630 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-30T04:02:41.630 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-30T04:02:41.630 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-30T04:02:41.630 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-30T04:02:41.630 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-30T04:02:41.630 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-30T04:02:41.630 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-30T04:02:41.630 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-30T04:02:41.630 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-30T04:02:41.630 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-30T04:02:41.631 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-30T04:02:41.631 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-30T04:02:41.631 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-30T04:02:41.631 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-30T04:02:41.631 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-30T04:02:41.631 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-30T04:02:41.631 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-30T04:02:41.631 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-30T04:02:41.631 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-30T04:02:41.631 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-30T04:02:41.631 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-30T04:02:41.631 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-30T04:02:41.631 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-dev.ko 2024-03-30T04:02:41.631 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-mux.ko 2024-03-30T04:02:41.632 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-30T04:02:41.632 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-smbus.ko 2024-03-30T04:02:41.632 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-stub.ko 2024-03-30T04:02:41.632 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband 2024-03-30T04:02:41.632 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core 2024-03-30T04:02:41.632 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-30T04:02:41.632 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_core.ko 2024-03-30T04:02:41.632 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-30T04:02:41.632 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-30T04:02:41.632 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-30T04:02:41.632 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-30T04:02:41.632 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-30T04:02:41.632 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw 2024-03-30T04:02:41.633 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/cxgb4 2024-03-30T04:02:41.633 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-30T04:02:41.633 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx4 2024-03-30T04:02:41.633 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-30T04:02:41.633 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx5 2024-03-30T04:02:41.633 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-30T04:02:41.633 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mthca 2024-03-30T04:02:41.633 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-30T04:02:41.633 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp 2024-03-30T04:02:41.633 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/ipoib 2024-03-30T04:02:41.633 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-30T04:02:41.633 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/iser 2024-03-30T04:02:41.633 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-30T04:02:41.634 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/isert 2024-03-30T04:02:41.634 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-30T04:02:41.634 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srp 2024-03-30T04:02:41.634 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-30T04:02:41.634 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srpt 2024-03-30T04:02:41.634 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-30T04:02:41.634 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input 2024-03-30T04:02:41.634 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport 2024-03-30T04:02:41.634 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-30T04:02:41.634 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-30T04:02:41.634 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/gameport.ko 2024-03-30T04:02:41.634 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/joydev.ko 2024-03-30T04:02:41.634 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard 2024-03-30T04:02:41.635 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-30T04:02:41.635 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse 2024-03-30T04:02:41.635 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/appletouch.ko 2024-03-30T04:02:41.635 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/bcm5974.ko 2024-03-30T04:02:41.635 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/cyapatp.ko 2024-03-30T04:02:41.635 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-30T04:02:41.635 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/sermouse.ko 2024-03-30T04:02:41.635 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-30T04:02:41.635 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-30T04:02:41.635 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-30T04:02:41.635 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/rmi4 2024-03-30T04:02:41.635 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-30T04:02:41.635 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio 2024-03-30T04:02:41.635 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/altera_ps2.ko 2024-03-30T04:02:41.636 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/arc_ps2.ko 2024-03-30T04:02:41.636 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-30T04:02:41.636 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/serio_raw.ko 2024-03-30T04:02:41.636 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/sparse-keymap.ko 2024-03-30T04:02:41.636 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds 2024-03-30T04:02:41.636 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-blinkm.ko 2024-03-30T04:02:41.636 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lm3530.ko 2024-03-30T04:02:41.636 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lp3944.ko 2024-03-30T04:02:41.636 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-ss4200.ko 2024-03-30T04:02:41.636 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger 2024-03-30T04:02:41.636 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-30T04:02:41.636 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-30T04:02:41.636 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-30T04:02:41.637 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-30T04:02:41.637 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-30T04:02:41.637 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-30T04:02:41.637 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-30T04:02:41.637 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-30T04:02:41.637 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-30T04:02:41.637 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md 2024-03-30T04:02:41.637 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/bcache 2024-03-30T04:02:41.637 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/bcache/bcache.ko 2024-03-30T04:02:41.637 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-bio-prison.ko 2024-03-30T04:02:41.637 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-cache-smq.ko 2024-03-30T04:02:41.637 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-cache.ko 2024-03-30T04:02:41.637 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-crypt.ko 2024-03-30T04:02:41.637 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-delay.ko 2024-03-30T04:02:41.638 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-flakey.ko 2024-03-30T04:02:41.638 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-log-userspace.ko 2024-03-30T04:02:41.638 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-multipath.ko 2024-03-30T04:02:41.638 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-queue-length.ko 2024-03-30T04:02:41.638 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-raid.ko 2024-03-30T04:02:41.638 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-round-robin.ko 2024-03-30T04:02:41.638 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-service-time.ko 2024-03-30T04:02:41.638 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-switch.ko 2024-03-30T04:02:41.638 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-thin-pool.ko 2024-03-30T04:02:41.638 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-verity.ko 2024-03-30T04:02:41.638 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/persistent-data 2024-03-30T04:02:41.638 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-30T04:02:41.639 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid0.ko 2024-03-30T04:02:41.639 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid1.ko 2024-03-30T04:02:41.639 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid10.ko 2024-03-30T04:02:41.639 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid456.ko 2024-03-30T04:02:41.639 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message 2024-03-30T04:02:41.639 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion 2024-03-30T04:02:41.639 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptbase.ko 2024-03-30T04:02:41.639 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptctl.ko 2024-03-30T04:02:41.639 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptfc.ko 2024-03-30T04:02:41.639 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptlan.ko 2024-03-30T04:02:41.639 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptsas.ko 2024-03-30T04:02:41.639 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptscsih.ko 2024-03-30T04:02:41.639 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptspi.ko 2024-03-30T04:02:41.639 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd 2024-03-30T04:02:41.639 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lpc_ich.ko 2024-03-30T04:02:41.640 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lpc_sch.ko 2024-03-30T04:02:41.640 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/mfd-core.ko 2024-03-30T04:02:41.640 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/sm501.ko 2024-03-30T04:02:41.640 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/viperboard.ko 2024-03-30T04:02:41.640 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/vx855.ko 2024-03-30T04:02:41.640 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/wl1273-core.ko 2024-03-30T04:02:41.640 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc 2024-03-30T04:02:41.640 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/altera-stapl 2024-03-30T04:02:41.640 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-30T04:02:41.640 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/apds9802als.ko 2024-03-30T04:02:41.640 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/apds990x.ko 2024-03-30T04:02:41.640 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/bh1770glc.ko 2024-03-30T04:02:41.640 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/cb710 2024-03-30T04:02:41.641 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/cb710/cb710.ko 2024-03-30T04:02:41.641 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom 2024-03-30T04:02:41.641 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/at24.ko 2024-03-30T04:02:41.641 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-30T04:02:41.641 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/max6875.ko 2024-03-30T04:02:41.641 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/enclosure.ko 2024-03-30T04:02:41.641 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/hpilo.ko 2024-03-30T04:02:41.641 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/isl29003.ko 2024-03-30T04:02:41.641 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/isl29020.ko 2024-03-30T04:02:41.641 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d 2024-03-30T04:02:41.641 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-30T04:02:41.641 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-30T04:02:41.641 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei 2024-03-30T04:02:41.641 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei-me.ko 2024-03-30T04:02:41.642 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei-txe.ko 2024-03-30T04:02:41.642 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei.ko 2024-03-30T04:02:41.642 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-gru 2024-03-30T04:02:41.642 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-30T04:02:41.642 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp 2024-03-30T04:02:41.642 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-30T04:02:41.642 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-30T04:02:41.642 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-30T04:02:41.642 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tifm_7xx1.ko 2024-03-30T04:02:41.642 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tifm_core.ko 2024-03-30T04:02:41.642 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tsl2550.ko 2024-03-30T04:02:41.642 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_balloon.ko 2024-03-30T04:02:41.642 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_vmci 2024-03-30T04:02:41.643 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-30T04:02:41.643 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net 2024-03-30T04:02:41.643 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/bonding 2024-03-30T04:02:41.643 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/bonding/bonding.ko 2024-03-30T04:02:41.643 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa 2024-03-30T04:02:41.643 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/b53 2024-03-30T04:02:41.643 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-30T04:02:41.643 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-30T04:02:41.643 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-30T04:02:41.643 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6xxx 2024-03-30T04:02:41.643 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-30T04:02:41.643 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dummy.ko 2024-03-30T04:02:41.643 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/eql.ko 2024-03-30T04:02:41.643 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet 2024-03-30T04:02:41.644 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com 2024-03-30T04:02:41.644 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-30T04:02:41.644 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-30T04:02:41.644 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-30T04:02:41.644 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-30T04:02:41.644 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390 2024-03-30T04:02:41.644 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-30T04:02:41.644 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-30T04:02:41.644 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-30T04:02:41.644 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-30T04:02:41.644 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/adaptec 2024-03-30T04:02:41.644 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-30T04:02:41.644 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/agere 2024-03-30T04:02:41.644 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-30T04:02:41.645 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/alteon 2024-03-30T04:02:41.645 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-30T04:02:41.645 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/altera 2024-03-30T04:02:41.645 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-30T04:02:41.645 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd 2024-03-30T04:02:41.645 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-30T04:02:41.645 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-30T04:02:41.645 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-30T04:02:41.645 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros 2024-03-30T04:02:41.645 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/alx 2024-03-30T04:02:41.645 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-30T04:02:41.645 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-30T04:02:41.645 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-30T04:02:41.646 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-30T04:02:41.646 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-30T04:02:41.646 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx 2024-03-30T04:02:41.646 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-30T04:02:41.646 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-30T04:02:41.646 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom 2024-03-30T04:02:41.646 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-30T04:02:41.646 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-30T04:02:41.646 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-30T04:02:41.646 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-30T04:02:41.646 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-30T04:02:41.646 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/genet 2024-03-30T04:02:41.646 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-30T04:02:41.646 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-30T04:02:41.647 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade 2024-03-30T04:02:41.647 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade/bna 2024-03-30T04:02:41.647 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-30T04:02:41.647 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio 2024-03-30T04:02:41.647 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-30T04:02:41.647 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-30T04:02:41.647 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-30T04:02:41.647 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-30T04:02:41.647 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-30T04:02:41.647 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-30T04:02:41.647 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-30T04:02:41.647 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-30T04:02:41.647 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-30T04:02:41.648 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-30T04:02:41.648 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco 2024-03-30T04:02:41.648 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco/enic 2024-03-30T04:02:41.648 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-30T04:02:41.648 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec 2024-03-30T04:02:41.648 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip 2024-03-30T04:02:41.648 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-30T04:02:41.648 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-30T04:02:41.648 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-30T04:02:41.648 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-30T04:02:41.648 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-30T04:02:41.648 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-30T04:02:41.648 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink 2024-03-30T04:02:41.648 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-30T04:02:41.648 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-30T04:02:41.649 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dnet.ko 2024-03-30T04:02:41.649 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex 2024-03-30T04:02:41.649 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex/benet 2024-03-30T04:02:41.649 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-30T04:02:41.649 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ethoc.ko 2024-03-30T04:02:41.649 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/fealnx.ko 2024-03-30T04:02:41.649 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel 2024-03-30T04:02:41.649 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-30T04:02:41.649 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000 2024-03-30T04:02:41.649 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-30T04:02:41.649 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000e 2024-03-30T04:02:41.649 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-30T04:02:41.649 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/fm10k 2024-03-30T04:02:41.650 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-30T04:02:41.650 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/i40e 2024-03-30T04:02:41.650 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-30T04:02:41.650 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/iavf 2024-03-30T04:02:41.650 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-30T04:02:41.650 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igb 2024-03-30T04:02:41.650 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-30T04:02:41.650 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igbvf 2024-03-30T04:02:41.650 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-30T04:02:41.650 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-30T04:02:41.650 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-30T04:02:41.650 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-30T04:02:41.650 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-30T04:02:41.650 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/jme.ko 2024-03-30T04:02:41.651 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell 2024-03-30T04:02:41.651 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-30T04:02:41.651 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-30T04:02:41.651 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-30T04:02:41.651 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox 2024-03-30T04:02:41.651 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-30T04:02:41.651 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-30T04:02:41.651 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-30T04:02:41.651 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-30T04:02:41.651 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-30T04:02:41.651 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-30T04:02:41.651 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel 2024-03-30T04:02:41.651 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-30T04:02:41.652 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom 2024-03-30T04:02:41.652 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-30T04:02:41.652 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-30T04:02:41.652 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi 2024-03-30T04:02:41.652 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-30T04:02:41.652 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-30T04:02:41.652 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/neterion 2024-03-30T04:02:41.652 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-30T04:02:41.652 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/nvidia 2024-03-30T04:02:41.652 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-30T04:02:41.652 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines 2024-03-30T04:02:41.652 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-30T04:02:41.652 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-30T04:02:41.652 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic 2024-03-30T04:02:41.653 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-30T04:02:41.653 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-30T04:02:41.653 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-30T04:02:41.653 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-30T04:02:41.653 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-30T04:02:41.653 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rdc 2024-03-30T04:02:41.653 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-30T04:02:41.653 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek 2024-03-30T04:02:41.653 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-30T04:02:41.653 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-30T04:02:41.653 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-30T04:02:41.653 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-30T04:02:41.653 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rocker 2024-03-30T04:02:41.653 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-30T04:02:41.654 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sfc 2024-03-30T04:02:41.654 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-30T04:02:41.654 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/silan 2024-03-30T04:02:41.654 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-30T04:02:41.654 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis 2024-03-30T04:02:41.654 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-30T04:02:41.654 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-30T04:02:41.654 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc 2024-03-30T04:02:41.654 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-30T04:02:41.654 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-30T04:02:41.654 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-30T04:02:41.654 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-30T04:02:41.654 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro 2024-03-30T04:02:41.654 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-30T04:02:41.655 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-30T04:02:41.655 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-30T04:02:41.655 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun 2024-03-30T04:02:41.655 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-30T04:02:41.655 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-30T04:02:41.655 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-30T04:02:41.655 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-30T04:02:41.655 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/tehuti 2024-03-30T04:02:41.655 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-30T04:02:41.655 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ti 2024-03-30T04:02:41.655 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-30T04:02:41.655 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via 2024-03-30T04:02:41.655 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-30T04:02:41.655 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-30T04:02:41.656 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet 2024-03-30T04:02:41.656 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-30T04:02:41.656 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-30T04:02:41.656 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/xircom 2024-03-30T04:02:41.656 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-30T04:02:41.656 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/geneve.ko 2024-03-30T04:02:41.656 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/hyperv 2024-03-30T04:02:41.656 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-30T04:02:41.656 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ieee802154 2024-03-30T04:02:41.656 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-30T04:02:41.656 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ifb.ko 2024-03-30T04:02:41.656 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ipvlan 2024-03-30T04:02:41.656 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-30T04:02:41.656 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/macvlan.ko 2024-03-30T04:02:41.657 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/macvtap.ko 2024-03-30T04:02:41.657 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio 2024-03-30T04:02:41.657 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio.ko 2024-03-30T04:02:41.657 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-30T04:02:41.657 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-30T04:02:41.657 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-30T04:02:41.657 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mii.ko 2024-03-30T04:02:41.657 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/net_failover.ko 2024-03-30T04:02:41.657 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/netconsole.ko 2024-03-30T04:02:41.657 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/nlmon.ko 2024-03-30T04:02:41.657 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs 2024-03-30T04:02:41.657 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-30T04:02:41.657 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-30T04:02:41.658 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy 2024-03-30T04:02:41.658 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/amd.ko 2024-03-30T04:02:41.658 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-30T04:02:41.658 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-30T04:02:41.658 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-30T04:02:41.658 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm87xx.ko 2024-03-30T04:02:41.658 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/broadcom.ko 2024-03-30T04:02:41.658 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/cicada.ko 2024-03-30T04:02:41.658 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/davicom.ko 2024-03-30T04:02:41.658 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/dp83640.ko 2024-03-30T04:02:41.658 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/et1011c.ko 2024-03-30T04:02:41.658 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/icplus.ko 2024-03-30T04:02:41.658 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/lxt.ko 2024-03-30T04:02:41.659 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/marvell.ko 2024-03-30T04:02:41.659 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/micrel.ko 2024-03-30T04:02:41.659 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/national.ko 2024-03-30T04:02:41.659 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/phylink.ko 2024-03-30T04:02:41.659 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/qsemi.ko 2024-03-30T04:02:41.659 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/realtek.ko 2024-03-30T04:02:41.659 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/smsc.ko 2024-03-30T04:02:41.659 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/ste10Xp.ko 2024-03-30T04:02:41.659 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/vitesse.ko 2024-03-30T04:02:41.659 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp 2024-03-30T04:02:41.659 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-30T04:02:41.659 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_async.ko 2024-03-30T04:02:41.659 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-30T04:02:41.659 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-30T04:02:41.660 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-30T04:02:41.660 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-30T04:02:41.660 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pppoe.ko 2024-03-30T04:02:41.660 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pppox.ko 2024-03-30T04:02:41.660 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pptp.ko 2024-03-30T04:02:41.660 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip 2024-03-30T04:02:41.660 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip/slhc.ko 2024-03-30T04:02:41.660 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip/slip.ko 2024-03-30T04:02:41.660 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/sungem_phy.ko 2024-03-30T04:02:41.660 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/tap.ko 2024-03-30T04:02:41.660 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team 2024-03-30T04:02:41.660 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team.ko 2024-03-30T04:02:41.660 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-30T04:02:41.660 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-30T04:02:41.661 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-30T04:02:41.661 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_random.ko 2024-03-30T04:02:41.661 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-30T04:02:41.661 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/tun.ko 2024-03-30T04:02:41.661 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/veth.ko 2024-03-30T04:02:41.661 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/virtio_net.ko 2024-03-30T04:02:41.661 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vmxnet3 2024-03-30T04:02:41.661 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-30T04:02:41.661 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vxlan 2024-03-30T04:02:41.661 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vxlan/vxlan.ko 2024-03-30T04:02:41.661 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netback 2024-03-30T04:02:41.661 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-30T04:02:41.661 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netfront.ko 2024-03-30T04:02:41.661 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ntb 2024-03-30T04:02:41.662 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ntb/ntb.ko 2024-03-30T04:02:41.662 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme 2024-03-30T04:02:41.662 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host 2024-03-30T04:02:41.662 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host/nvme-core.ko 2024-03-30T04:02:41.662 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host/nvme.ko 2024-03-30T04:02:41.662 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport 2024-03-30T04:02:41.662 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport.ko 2024-03-30T04:02:41.662 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_cs.ko 2024-03-30T04:02:41.662 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_pc.ko 2024-03-30T04:02:41.662 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_serial.ko 2024-03-30T04:02:41.662 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci 2024-03-30T04:02:41.662 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug 2024-03-30T04:02:41.662 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-30T04:02:41.662 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/pcie 2024-03-30T04:02:41.663 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-30T04:02:41.663 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/xen-pcifront.ko 2024-03-30T04:02:41.663 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia 2024-03-30T04:02:41.663 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/i82092.ko 2024-03-30T04:02:41.663 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/pd6729.ko 2024-03-30T04:02:41.663 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-30T04:02:41.663 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl 2024-03-30T04:02:41.663 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl/intel 2024-03-30T04:02:41.663 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-30T04:02:41.663 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform 2024-03-30T04:02:41.663 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86 2024-03-30T04:02:41.663 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-30T04:02:41.663 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/acerhdf.ko 2024-03-30T04:02:41.663 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-30T04:02:41.664 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-30T04:02:41.664 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-30T04:02:41.664 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-30T04:02:41.664 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-30T04:02:41.664 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-30T04:02:41.664 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-30T04:02:41.664 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-30T04:02:41.664 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-30T04:02:41.664 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/hdaps.ko 2024-03-30T04:02:41.664 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel 2024-03-30T04:02:41.664 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-30T04:02:41.664 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel_ips.ko 2024-03-30T04:02:41.664 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-30T04:02:41.664 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-30T04:02:41.665 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-30T04:02:41.665 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-30T04:02:41.665 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-30T04:02:41.665 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-30T04:02:41.665 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-30T04:02:41.665 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-30T04:02:41.665 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-30T04:02:41.665 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/wmi.ko 2024-03-30T04:02:41.665 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap 2024-03-30T04:02:41.665 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-30T04:02:41.665 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-30T04:02:41.665 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps 2024-03-30T04:02:41.665 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients 2024-03-30T04:02:41.665 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-30T04:02:41.666 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-30T04:02:41.666 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps_parport.ko 2024-03-30T04:02:41.666 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/pps_core.ko 2024-03-30T04:02:41.666 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp 2024-03-30T04:02:41.666 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp/ptp.ko 2024-03-30T04:02:41.666 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp/ptp_kvm.ko 2024-03-30T04:02:41.666 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc 2024-03-30T04:02:41.666 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-30T04:02:41.666 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-30T04:02:41.666 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-30T04:02:41.666 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-30T04:02:41.666 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-30T04:02:41.666 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-30T04:02:41.667 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-30T04:02:41.667 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-30T04:02:41.667 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-30T04:02:41.667 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-30T04:02:41.667 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-em3027.ko 2024-03-30T04:02:41.667 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-30T04:02:41.667 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-30T04:02:41.667 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-30T04:02:41.667 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-30T04:02:41.667 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-30T04:02:41.667 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-30T04:02:41.667 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-max6900.ko 2024-03-30T04:02:41.667 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-30T04:02:41.667 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-30T04:02:41.668 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-30T04:02:41.668 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-30T04:02:41.668 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-30T04:02:41.668 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-30T04:02:41.668 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-30T04:02:41.668 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-30T04:02:41.668 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-30T04:02:41.668 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-30T04:02:41.668 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-30T04:02:41.668 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-30T04:02:41.668 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-x1205.ko 2024-03-30T04:02:41.668 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi 2024-03-30T04:02:41.668 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-9xxx.ko 2024-03-30T04:02:41.668 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-sas.ko 2024-03-30T04:02:41.669 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-xxxx.ko 2024-03-30T04:02:41.669 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/BusLogic.ko 2024-03-30T04:02:41.669 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/a100u2w.ko 2024-03-30T04:02:41.669 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aacraid 2024-03-30T04:02:41.669 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-30T04:02:41.669 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/advansys.ko 2024-03-30T04:02:41.669 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx 2024-03-30T04:02:41.669 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-30T04:02:41.669 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-30T04:02:41.669 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/am53c974.ko 2024-03-30T04:02:41.669 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/arcmsr 2024-03-30T04:02:41.669 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-30T04:02:41.669 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/atp870u.ko 2024-03-30T04:02:41.669 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/be2iscsi 2024-03-30T04:02:41.670 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-30T04:02:41.670 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bfa 2024-03-30T04:02:41.670 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bfa/bfa.ko 2024-03-30T04:02:41.670 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2fc 2024-03-30T04:02:41.670 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-30T04:02:41.670 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2i 2024-03-30T04:02:41.670 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-30T04:02:41.670 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ch.ko 2024-03-30T04:02:41.670 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/csiostor 2024-03-30T04:02:41.670 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-30T04:02:41.670 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi 2024-03-30T04:02:41.670 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-30T04:02:41.670 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-30T04:02:41.670 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-30T04:02:41.671 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-30T04:02:41.671 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-30T04:02:41.671 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/dc395x.ko 2024-03-30T04:02:41.671 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler 2024-03-30T04:02:41.671 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-30T04:02:41.671 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-30T04:02:41.671 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-30T04:02:41.671 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-30T04:02:41.671 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/dmx3191d.ko 2024-03-30T04:02:41.671 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esas2r 2024-03-30T04:02:41.671 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-30T04:02:41.671 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esp_scsi.ko 2024-03-30T04:02:41.671 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe 2024-03-30T04:02:41.671 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-30T04:02:41.672 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-30T04:02:41.672 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fnic 2024-03-30T04:02:41.672 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fnic/fnic.ko 2024-03-30T04:02:41.672 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hpsa.ko 2024-03-30T04:02:41.672 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hptiop.ko 2024-03-30T04:02:41.672 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hv_storvsc.ko 2024-03-30T04:02:41.672 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/initio.ko 2024-03-30T04:02:41.672 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ipr.ko 2024-03-30T04:02:41.672 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ips.ko 2024-03-30T04:02:41.672 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/isci 2024-03-30T04:02:41.672 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/isci/isci.ko 2024-03-30T04:02:41.672 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-30T04:02:41.672 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-30T04:02:41.673 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libfc 2024-03-30T04:02:41.673 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libfc/libfc.ko 2024-03-30T04:02:41.673 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libiscsi.ko 2024-03-30T04:02:41.673 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-30T04:02:41.673 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libsas 2024-03-30T04:02:41.673 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libsas/libsas.ko 2024-03-30T04:02:41.673 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/lpfc 2024-03-30T04:02:41.673 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-30T04:02:41.673 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid 2024-03-30T04:02:41.673 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid.ko 2024-03-30T04:02:41.673 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-30T04:02:41.673 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-30T04:02:41.673 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-30T04:02:41.674 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mpt3sas 2024-03-30T04:02:41.674 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-30T04:02:41.674 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvsas 2024-03-30T04:02:41.674 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-30T04:02:41.674 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvumi.ko 2024-03-30T04:02:41.674 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pm8001 2024-03-30T04:02:41.674 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-30T04:02:41.674 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pmcraid.ko 2024-03-30T04:02:41.674 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla1280.ko 2024-03-30T04:02:41.674 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx 2024-03-30T04:02:41.674 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-30T04:02:41.674 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-30T04:02:41.674 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla4xxx 2024-03-30T04:02:41.674 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-30T04:02:41.675 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/raid_class.ko 2024-03-30T04:02:41.675 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_debug.ko 2024-03-30T04:02:41.675 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-30T04:02:41.675 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-30T04:02:41.675 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-30T04:02:41.675 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-30T04:02:41.675 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-30T04:02:41.675 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ses.ko 2024-03-30T04:02:41.675 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/st.ko 2024-03-30T04:02:41.675 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/stex.ko 2024-03-30T04:02:41.675 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/sym53c8xx_2 2024-03-30T04:02:41.675 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-30T04:02:41.675 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/virtio_scsi.ko 2024-03-30T04:02:41.675 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-30T04:02:41.676 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/wd719x.ko 2024-03-30T04:02:41.676 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ssb 2024-03-30T04:02:41.676 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ssb/ssb.ko 2024-03-30T04:02:41.676 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target 2024-03-30T04:02:41.676 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/iscsi 2024-03-30T04:02:41.676 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-30T04:02:41.676 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/loopback 2024-03-30T04:02:41.676 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-30T04:02:41.676 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_file.ko 2024-03-30T04:02:41.676 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_iblock.ko 2024-03-30T04:02:41.676 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_mod.ko 2024-03-30T04:02:41.676 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_pscsi.ko 2024-03-30T04:02:41.676 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_user.ko 2024-03-30T04:02:41.676 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/tcm_fc 2024-03-30T04:02:41.677 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-30T04:02:41.677 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal 2024-03-30T04:02:41.677 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel 2024-03-30T04:02:41.677 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal 2024-03-30T04:02:41.677 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-30T04:02:41.677 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-30T04:02:41.677 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-30T04:02:41.677 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-30T04:02:41.677 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-30T04:02:41.677 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-30T04:02:41.677 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-30T04:02:41.677 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-30T04:02:41.677 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-30T04:02:41.678 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-30T04:02:41.678 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-30T04:02:41.678 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-30T04:02:41.678 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-30T04:02:41.678 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-30T04:02:41.678 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-30T04:02:41.678 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-30T04:02:41.678 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-30T04:02:41.678 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-30T04:02:41.678 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty 2024-03-30T04:02:41.678 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/ipwireless 2024-03-30T04:02:41.678 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-30T04:02:41.678 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/n_gsm.ko 2024-03-30T04:02:41.678 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/n_hdlc.ko 2024-03-30T04:02:41.679 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/nozomi.ko 2024-03-30T04:02:41.679 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial 2024-03-30T04:02:41.679 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/8250 2024-03-30T04:02:41.679 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-30T04:02:41.679 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/arc_uart.ko 2024-03-30T04:02:41.679 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/jsm 2024-03-30T04:02:41.679 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-30T04:02:41.679 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs 2024-03-30T04:02:41.679 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/core 2024-03-30T04:02:41.679 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-30T04:02:41.679 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/host 2024-03-30T04:02:41.679 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-30T04:02:41.679 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio 2024-03-30T04:02:41.679 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio.ko 2024-03-30T04:02:41.680 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_aec.ko 2024-03-30T04:02:41.680 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_cif.ko 2024-03-30T04:02:41.680 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_pci_generic.ko 2024-03-30T04:02:41.680 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_sercos3.ko 2024-03-30T04:02:41.680 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb 2024-03-30T04:02:41.680 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class 2024-03-30T04:02:41.680 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/cdc-acm.ko 2024-03-30T04:02:41.680 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-30T04:02:41.680 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/usblp.ko 2024-03-30T04:02:41.680 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/usbtmc.ko 2024-03-30T04:02:41.680 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host 2024-03-30T04:02:41.680 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-30T04:02:41.680 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image 2024-03-30T04:02:41.680 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image/mdc800.ko 2024-03-30T04:02:41.681 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image/microtek.ko 2024-03-30T04:02:41.681 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc 2024-03-30T04:02:41.681 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/adutux.ko 2024-03-30T04:02:41.681 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/appledisplay.ko 2024-03-30T04:02:41.681 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/emi26.ko 2024-03-30T04:02:41.681 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/emi62.ko 2024-03-30T04:02:41.681 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ezusb.ko 2024-03-30T04:02:41.681 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/idmouse.ko 2024-03-30T04:02:41.681 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/iowarrior.ko 2024-03-30T04:02:41.681 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-30T04:02:41.681 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ldusb.ko 2024-03-30T04:02:41.681 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/legousbtower.ko 2024-03-30T04:02:41.681 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/sisusbvga 2024-03-30T04:02:41.681 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-30T04:02:41.682 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-30T04:02:41.682 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usb3503.ko 2024-03-30T04:02:41.682 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usblcd.ko 2024-03-30T04:02:41.682 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-30T04:02:41.682 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/uss720.ko 2024-03-30T04:02:41.682 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/yurex.ko 2024-03-30T04:02:41.682 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy 2024-03-30T04:02:41.682 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy/phy-generic.ko 2024-03-30T04:02:41.682 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial 2024-03-30T04:02:41.682 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/aircable.ko 2024-03-30T04:02:41.682 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ark3116.ko 2024-03-30T04:02:41.682 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-30T04:02:41.682 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ch341.ko 2024-03-30T04:02:41.682 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cp210x.ko 2024-03-30T04:02:41.683 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cyberjack.ko 2024-03-30T04:02:41.683 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-30T04:02:41.683 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-30T04:02:41.683 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/empeg.ko 2024-03-30T04:02:41.683 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-30T04:02:41.683 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-30T04:02:41.683 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-30T04:02:41.683 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/io_ti.ko 2024-03-30T04:02:41.683 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ipaq.ko 2024-03-30T04:02:41.683 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ipw.ko 2024-03-30T04:02:41.683 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ir-usb.ko 2024-03-30T04:02:41.683 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-30T04:02:41.683 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/keyspan.ko 2024-03-30T04:02:41.683 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-30T04:02:41.684 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-30T04:02:41.684 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-30T04:02:41.684 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mct_u232.ko 2024-03-30T04:02:41.684 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mos7720.ko 2024-03-30T04:02:41.684 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mos7840.ko 2024-03-30T04:02:41.684 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/navman.ko 2024-03-30T04:02:41.684 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/omninet.ko 2024-03-30T04:02:41.684 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/opticon.ko 2024-03-30T04:02:41.684 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/option.ko 2024-03-30T04:02:41.684 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/oti6858.ko 2024-03-30T04:02:41.684 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/pl2303.ko 2024-03-30T04:02:41.684 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/qcaux.ko 2024-03-30T04:02:41.684 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/qcserial.ko 2024-03-30T04:02:41.684 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/quatech2.ko 2024-03-30T04:02:41.685 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/safe_serial.ko 2024-03-30T04:02:41.685 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/sierra.ko 2024-03-30T04:02:41.685 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-30T04:02:41.685 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ssu100.ko 2024-03-30T04:02:41.685 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/symbolserial.ko 2024-03-30T04:02:41.685 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-30T04:02:41.685 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-30T04:02:41.685 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb_debug.ko 2024-03-30T04:02:41.685 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-30T04:02:41.685 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/visor.ko 2024-03-30T04:02:41.685 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/whiteheat.ko 2024-03-30T04:02:41.685 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-30T04:02:41.685 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage 2024-03-30T04:02:41.685 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/uas.ko 2024-03-30T04:02:41.686 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-30T04:02:41.686 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-30T04:02:41.686 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-30T04:02:41.686 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-30T04:02:41.686 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-30T04:02:41.686 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-30T04:02:41.686 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-30T04:02:41.686 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-karma.ko 2024-03-30T04:02:41.686 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-30T04:02:41.686 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-30T04:02:41.686 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-30T04:02:41.686 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-30T04:02:41.686 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-30T04:02:41.687 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/usb-storage.ko 2024-03-30T04:02:41.687 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip 2024-03-30T04:02:41.687 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-30T04:02:41.687 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-30T04:02:41.687 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-30T04:02:41.687 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio 2024-03-30T04:02:41.687 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci 2024-03-30T04:02:41.687 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-30T04:02:41.687 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-30T04:02:41.687 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/vfio.ko 2024-03-30T04:02:41.687 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-30T04:02:41.687 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost 2024-03-30T04:02:41.687 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost.ko 2024-03-30T04:02:41.687 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-30T04:02:41.688 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_net.ko 2024-03-30T04:02:41.688 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_scsi.ko 2024-03-30T04:02:41.688 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video 2024-03-30T04:02:41.688 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight 2024-03-30T04:02:41.688 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/apple_bl.ko 2024-03-30T04:02:41.688 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/lcd.ko 2024-03-30T04:02:41.688 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-30T04:02:41.688 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio 2024-03-30T04:02:41.688 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio.ko 2024-03-30T04:02:41.688 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_balloon.ko 2024-03-30T04:02:41.688 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_mmio.ko 2024-03-30T04:02:41.688 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci.ko 2024-03-30T04:02:41.688 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-30T04:02:41.688 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-30T04:02:41.689 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_ring.ko 2024-03-30T04:02:41.689 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1 2024-03-30T04:02:41.689 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters 2024-03-30T04:02:41.689 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/ds2482.ko 2024-03-30T04:02:41.689 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/ds2490.ko 2024-03-30T04:02:41.689 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves 2024-03-30T04:02:41.689 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-30T04:02:41.689 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-30T04:02:41.689 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-30T04:02:41.689 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-30T04:02:41.689 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-30T04:02:41.689 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-30T04:02:41.689 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-30T04:02:41.689 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-30T04:02:41.690 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-30T04:02:41.690 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-30T04:02:41.690 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-30T04:02:41.690 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/wire.ko 2024-03-30T04:02:41.690 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog 2024-03-30T04:02:41.690 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-30T04:02:41.690 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-30T04:02:41.690 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-30T04:02:41.690 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/hpwdt.ko 2024-03-30T04:02:41.690 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/i6300esb.ko 2024-03-30T04:02:41.690 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-30T04:02:41.690 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-30T04:02:41.690 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ib700wdt.ko 2024-03-30T04:02:41.690 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ibmasr.ko 2024-03-30T04:02:41.691 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-30T04:02:41.691 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-30T04:02:41.691 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/it87_wdt.ko 2024-03-30T04:02:41.691 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/machzwd.ko 2024-03-30T04:02:41.691 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/nv_tco.ko 2024-03-30T04:02:41.691 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-30T04:02:41.691 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-30T04:02:41.691 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-30T04:02:41.691 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-30T04:02:41.691 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/softdog.ko 2024-03-30T04:02:41.691 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-30T04:02:41.691 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/via_wdt.ko 2024-03-30T04:02:41.691 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-30T04:02:41.691 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-30T04:02:41.692 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-30T04:02:41.692 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/wdt_pci.ko 2024-03-30T04:02:41.692 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/xen_wdt.ko 2024-03-30T04:02:41.692 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen 2024-03-30T04:02:41.692 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-30T04:02:41.692 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-evtchn.ko 2024-03-30T04:02:41.692 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-gntalloc.ko 2024-03-30T04:02:41.692 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-gntdev.ko 2024-03-30T04:02:41.692 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-pciback 2024-03-30T04:02:41.692 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-30T04:02:41.692 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-privcmd.ko 2024-03-30T04:02:41.692 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xenfs 2024-03-30T04:02:41.692 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-30T04:02:41.692 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs 2024-03-30T04:02:41.693 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/9p 2024-03-30T04:02:41.693 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/9p/9p.ko 2024-03-30T04:02:41.693 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/affs 2024-03-30T04:02:41.693 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/affs/affs.ko 2024-03-30T04:02:41.693 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/befs 2024-03-30T04:02:41.693 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/befs/befs.ko 2024-03-30T04:02:41.693 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/binfmt_misc.ko 2024-03-30T04:02:41.693 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/btrfs 2024-03-30T04:02:41.693 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/btrfs/btrfs.ko 2024-03-30T04:02:41.693 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cachefiles 2024-03-30T04:02:41.693 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cachefiles/cachefiles.ko 2024-03-30T04:02:41.693 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ceph 2024-03-30T04:02:41.693 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ceph/ceph.ko 2024-03-30T04:02:41.693 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/coda 2024-03-30T04:02:41.694 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/coda/coda.ko 2024-03-30T04:02:41.694 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cramfs 2024-03-30T04:02:41.694 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cramfs/cramfs.ko 2024-03-30T04:02:41.694 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/dlm 2024-03-30T04:02:41.694 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/dlm/dlm.ko 2024-03-30T04:02:41.694 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ecryptfs 2024-03-30T04:02:41.694 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-30T04:02:41.694 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/f2fs 2024-03-30T04:02:41.694 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/f2fs/f2fs.ko 2024-03-30T04:02:41.694 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat 2024-03-30T04:02:41.694 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/fat.ko 2024-03-30T04:02:41.694 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/msdos.ko 2024-03-30T04:02:41.694 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/vfat.ko 2024-03-30T04:02:41.694 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse 2024-03-30T04:02:41.695 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse/cuse.ko 2024-03-30T04:02:41.695 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse/fuse.ko 2024-03-30T04:02:41.695 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/gfs2 2024-03-30T04:02:41.695 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/gfs2/gfs2.ko 2024-03-30T04:02:41.695 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfs 2024-03-30T04:02:41.695 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfs/hfs.ko 2024-03-30T04:02:41.695 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfsplus 2024-03-30T04:02:41.695 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfsplus/hfsplus.ko 2024-03-30T04:02:41.695 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/isofs 2024-03-30T04:02:41.695 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/isofs/isofs.ko 2024-03-30T04:02:41.695 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/jfs 2024-03-30T04:02:41.695 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/jfs/jfs.ko 2024-03-30T04:02:41.695 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/lockd 2024-03-30T04:02:41.695 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/lockd/lockd.ko 2024-03-30T04:02:41.696 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/minix 2024-03-30T04:02:41.696 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/minix/minix.ko 2024-03-30T04:02:41.696 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/netfs 2024-03-30T04:02:41.696 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/netfs/netfs.ko 2024-03-30T04:02:41.696 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs 2024-03-30T04:02:41.696 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/blocklayout 2024-03-30T04:02:41.696 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-30T04:02:41.696 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/filelayout 2024-03-30T04:02:41.696 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-30T04:02:41.696 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/flexfilelayout 2024-03-30T04:02:41.696 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-30T04:02:41.696 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfs.ko 2024-03-30T04:02:41.696 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfsv3.ko 2024-03-30T04:02:41.696 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfsv4.ko 2024-03-30T04:02:41.697 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common 2024-03-30T04:02:41.697 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common/grace.ko 2024-03-30T04:02:41.697 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common/nfs_acl.ko 2024-03-30T04:02:41.697 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfsd 2024-03-30T04:02:41.697 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfsd/nfsd.ko 2024-03-30T04:02:41.697 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nilfs2 2024-03-30T04:02:41.697 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nilfs2/nilfs2.ko 2024-03-30T04:02:41.697 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls 2024-03-30T04:02:41.697 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-celtic.ko 2024-03-30T04:02:41.697 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-centeuro.ko 2024-03-30T04:02:41.697 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-croatian.ko 2024-03-30T04:02:41.697 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-cyrillic.ko 2024-03-30T04:02:41.697 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-gaelic.ko 2024-03-30T04:02:41.697 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-greek.ko 2024-03-30T04:02:41.698 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-iceland.ko 2024-03-30T04:02:41.698 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-inuit.ko 2024-03-30T04:02:41.698 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-roman.ko 2024-03-30T04:02:41.698 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-romanian.ko 2024-03-30T04:02:41.698 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-turkish.ko 2024-03-30T04:02:41.698 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1250.ko 2024-03-30T04:02:41.698 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1251.ko 2024-03-30T04:02:41.698 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1255.ko 2024-03-30T04:02:41.698 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp737.ko 2024-03-30T04:02:41.698 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp775.ko 2024-03-30T04:02:41.698 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp850.ko 2024-03-30T04:02:41.698 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp852.ko 2024-03-30T04:02:41.698 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp855.ko 2024-03-30T04:02:41.698 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp857.ko 2024-03-30T04:02:41.699 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp860.ko 2024-03-30T04:02:41.699 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp861.ko 2024-03-30T04:02:41.699 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp862.ko 2024-03-30T04:02:41.699 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp863.ko 2024-03-30T04:02:41.699 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp864.ko 2024-03-30T04:02:41.699 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp865.ko 2024-03-30T04:02:41.699 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp866.ko 2024-03-30T04:02:41.699 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp869.ko 2024-03-30T04:02:41.699 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp874.ko 2024-03-30T04:02:41.699 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp932.ko 2024-03-30T04:02:41.699 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp936.ko 2024-03-30T04:02:41.699 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp949.ko 2024-03-30T04:02:41.699 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp950.ko 2024-03-30T04:02:41.700 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_euc-jp.ko 2024-03-30T04:02:41.700 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-1.ko 2024-03-30T04:02:41.700 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-13.ko 2024-03-30T04:02:41.700 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-14.ko 2024-03-30T04:02:41.700 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-15.ko 2024-03-30T04:02:41.700 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-2.ko 2024-03-30T04:02:41.700 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-3.ko 2024-03-30T04:02:41.700 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-4.ko 2024-03-30T04:02:41.700 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-5.ko 2024-03-30T04:02:41.700 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-6.ko 2024-03-30T04:02:41.700 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-7.ko 2024-03-30T04:02:41.700 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-9.ko 2024-03-30T04:02:41.700 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-r.ko 2024-03-30T04:02:41.700 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-ru.ko 2024-03-30T04:02:41.701 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-u.ko 2024-03-30T04:02:41.701 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-30T04:02:41.701 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_utf8.ko 2024-03-30T04:02:41.701 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2 2024-03-30T04:02:41.701 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/cluster 2024-03-30T04:02:41.701 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-30T04:02:41.701 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlm 2024-03-30T04:02:41.701 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-30T04:02:41.701 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlmfs 2024-03-30T04:02:41.701 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-30T04:02:41.701 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2.ko 2024-03-30T04:02:41.701 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-30T04:02:41.701 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-30T04:02:41.702 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-30T04:02:41.702 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/overlayfs 2024-03-30T04:02:41.702 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/overlayfs/overlay.ko 2024-03-30T04:02:41.702 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/pstore 2024-03-30T04:02:41.702 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/pstore/ramoops.ko 2024-03-30T04:02:41.702 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/reiserfs 2024-03-30T04:02:41.702 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/reiserfs/reiserfs.ko 2024-03-30T04:02:41.702 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/romfs 2024-03-30T04:02:41.702 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/romfs/romfs.ko 2024-03-30T04:02:41.702 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb 2024-03-30T04:02:41.702 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/client 2024-03-30T04:02:41.702 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/client/cifs.ko 2024-03-30T04:02:41.702 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common 2024-03-30T04:02:41.703 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common/cifs_arc4.ko 2024-03-30T04:02:41.703 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common/cifs_md4.ko 2024-03-30T04:02:41.703 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/squashfs 2024-03-30T04:02:41.703 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/squashfs/squashfs.ko 2024-03-30T04:02:41.703 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/sysv 2024-03-30T04:02:41.703 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/sysv/sysv.ko 2024-03-30T04:02:41.703 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/udf 2024-03-30T04:02:41.703 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/udf/udf.ko 2024-03-30T04:02:41.703 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ufs 2024-03-30T04:02:41.703 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ufs/ufs.ko 2024-03-30T04:02:41.703 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/xfs 2024-03-30T04:02:41.703 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/xfs/xfs.ko 2024-03-30T04:02:41.703 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel 2024-03-30T04:02:41.703 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/rcu 2024-03-30T04:02:41.704 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/rcu/rcutorture.ko 2024-03-30T04:02:41.704 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/torture.ko 2024-03-30T04:02:41.704 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/trace 2024-03-30T04:02:41.704 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-30T04:02:41.704 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib 2024-03-30T04:02:41.704 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/asn1_encoder.ko 2024-03-30T04:02:41.704 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crc-itu-t.ko 2024-03-30T04:02:41.704 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crc8.ko 2024-03-30T04:02:41.704 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto 2024-03-30T04:02:41.704 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto/libarc4.ko 2024-03-30T04:02:41.704 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto/libdes.ko 2024-03-30T04:02:41.704 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/libcrc32c.ko 2024-03-30T04:02:41.704 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lru_cache.ko 2024-03-30T04:02:41.704 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4 2024-03-30T04:02:41.705 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4/lz4_compress.ko 2024-03-30T04:02:41.705 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4/lz4hc_compress.ko 2024-03-30T04:02:41.705 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/math 2024-03-30T04:02:41.705 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/math/cordic.ko 2024-03-30T04:02:41.705 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/raid6 2024-03-30T04:02:41.705 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/raid6/raid6_pq.ko 2024-03-30T04:02:41.705 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/reed_solomon 2024-03-30T04:02:41.705 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-30T04:02:41.705 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_bm.ko 2024-03-30T04:02:41.705 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_fsm.ko 2024-03-30T04:02:41.705 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_kmp.ko 2024-03-30T04:02:41.705 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/zstd 2024-03-30T04:02:41.705 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/zstd/zstd_compress.ko 2024-03-30T04:02:41.705 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/mm 2024-03-30T04:02:41.705 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/mm/hwpoison-inject.ko 2024-03-30T04:02:41.706 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net 2024-03-30T04:02:41.706 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan 2024-03-30T04:02:41.706 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/6lowpan.ko 2024-03-30T04:02:41.706 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_dest.ko 2024-03-30T04:02:41.706 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_fragment.ko 2024-03-30T04:02:41.706 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_hop.ko 2024-03-30T04:02:41.706 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-30T04:02:41.706 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_mobility.ko 2024-03-30T04:02:41.706 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_routing.ko 2024-03-30T04:02:41.706 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_udp.ko 2024-03-30T04:02:41.706 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802 2024-03-30T04:02:41.706 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/garp.ko 2024-03-30T04:02:41.706 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/mrp.ko 2024-03-30T04:02:41.707 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/p8022.ko 2024-03-30T04:02:41.707 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/psnap.ko 2024-03-30T04:02:41.707 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/stp.ko 2024-03-30T04:02:41.707 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/8021q 2024-03-30T04:02:41.707 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/8021q/8021q.ko 2024-03-30T04:02:41.707 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p 2024-03-30T04:02:41.707 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet.ko 2024-03-30T04:02:41.707 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_fd.ko 2024-03-30T04:02:41.707 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_rdma.ko 2024-03-30T04:02:41.707 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_virtio.ko 2024-03-30T04:02:41.707 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/batman-adv 2024-03-30T04:02:41.707 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/batman-adv/batman-adv.ko 2024-03-30T04:02:41.707 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge 2024-03-30T04:02:41.707 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/br_netfilter.ko 2024-03-30T04:02:41.708 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/bridge.ko 2024-03-30T04:02:41.708 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter 2024-03-30T04:02:41.708 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-30T04:02:41.708 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-30T04:02:41.708 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-30T04:02:41.708 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-30T04:02:41.708 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-30T04:02:41.708 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-30T04:02:41.708 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-30T04:02:41.708 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-30T04:02:41.708 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-30T04:02:41.708 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-30T04:02:41.708 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-30T04:02:41.708 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-30T04:02:41.709 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-30T04:02:41.709 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-30T04:02:41.709 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-30T04:02:41.709 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-30T04:02:41.709 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-30T04:02:41.709 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-30T04:02:41.709 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-30T04:02:41.709 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-30T04:02:41.709 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtables.ko 2024-03-30T04:02:41.709 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-30T04:02:41.709 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-30T04:02:41.709 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-30T04:02:41.709 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ceph 2024-03-30T04:02:41.710 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ceph/libceph.ko 2024-03-30T04:02:41.710 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core 2024-03-30T04:02:41.710 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core/failover.ko 2024-03-30T04:02:41.710 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core/pktgen.ko 2024-03-30T04:02:41.710 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp 2024-03-30T04:02:41.710 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp.ko 2024-03-30T04:02:41.710 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_diag.ko 2024-03-30T04:02:41.710 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_ipv4.ko 2024-03-30T04:02:41.710 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_ipv6.ko 2024-03-30T04:02:41.710 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dns_resolver 2024-03-30T04:02:41.710 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dns_resolver/dns_resolver.ko 2024-03-30T04:02:41.710 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa 2024-03-30T04:02:41.710 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/dsa_core.ko 2024-03-30T04:02:41.710 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_brcm.ko 2024-03-30T04:02:41.711 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_dsa.ko 2024-03-30T04:02:41.711 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_none.ko 2024-03-30T04:02:41.711 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_trailer.ko 2024-03-30T04:02:41.711 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154 2024-03-30T04:02:41.711 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/6lowpan 2024-03-30T04:02:41.711 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-30T04:02:41.711 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/ieee802154.ko 2024-03-30T04:02:41.711 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-30T04:02:41.711 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4 2024-03-30T04:02:41.711 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ah4.ko 2024-03-30T04:02:41.711 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/esp4.ko 2024-03-30T04:02:41.711 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/fou.ko 2024-03-30T04:02:41.711 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/gre.ko 2024-03-30T04:02:41.712 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/inet_diag.ko 2024-03-30T04:02:41.712 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_gre.ko 2024-03-30T04:02:41.712 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_tunnel.ko 2024-03-30T04:02:41.712 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_vti.ko 2024-03-30T04:02:41.712 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ipcomp.ko 2024-03-30T04:02:41.712 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ipip.ko 2024-03-30T04:02:41.712 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter 2024-03-30T04:02:41.712 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-30T04:02:41.712 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-30T04:02:41.712 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-30T04:02:41.712 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-30T04:02:41.712 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-30T04:02:41.712 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-30T04:02:41.712 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-30T04:02:41.713 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-30T04:02:41.713 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-30T04:02:41.713 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-30T04:02:41.713 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-30T04:02:41.713 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-30T04:02:41.713 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-30T04:02:41.713 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-30T04:02:41.713 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-30T04:02:41.713 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-30T04:02:41.713 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-30T04:02:41.713 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-30T04:02:41.713 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-30T04:02:41.713 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-30T04:02:41.713 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-30T04:02:41.714 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_bic.ko 2024-03-30T04:02:41.714 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_dctcp.ko 2024-03-30T04:02:41.714 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_diag.ko 2024-03-30T04:02:41.714 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_highspeed.ko 2024-03-30T04:02:41.714 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_htcp.ko 2024-03-30T04:02:41.714 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_hybla.ko 2024-03-30T04:02:41.714 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_illinois.ko 2024-03-30T04:02:41.714 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_lp.ko 2024-03-30T04:02:41.714 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_scalable.ko 2024-03-30T04:02:41.714 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_vegas.ko 2024-03-30T04:02:41.714 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_veno.ko 2024-03-30T04:02:41.714 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_westwood.ko 2024-03-30T04:02:41.714 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_yeah.ko 2024-03-30T04:02:41.715 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tunnel4.ko 2024-03-30T04:02:41.715 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/udp_diag.ko 2024-03-30T04:02:41.715 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/udp_tunnel.ko 2024-03-30T04:02:41.715 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-30T04:02:41.715 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6 2024-03-30T04:02:41.715 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ah6.ko 2024-03-30T04:02:41.715 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/esp6.ko 2024-03-30T04:02:41.715 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/fou6.ko 2024-03-30T04:02:41.715 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_tunnel.ko 2024-03-30T04:02:41.715 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-30T04:02:41.715 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_vti.ko 2024-03-30T04:02:41.715 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ipcomp6.ko 2024-03-30T04:02:41.715 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter 2024-03-30T04:02:41.715 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-30T04:02:41.716 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-30T04:02:41.716 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-30T04:02:41.716 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-30T04:02:41.716 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-30T04:02:41.716 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-30T04:02:41.716 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-30T04:02:41.716 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-30T04:02:41.716 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-30T04:02:41.716 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-30T04:02:41.716 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-30T04:02:41.716 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-30T04:02:41.716 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-30T04:02:41.716 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-30T04:02:41.716 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-30T04:02:41.717 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-30T04:02:41.717 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-30T04:02:41.717 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-30T04:02:41.717 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-30T04:02:41.717 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-30T04:02:41.717 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-30T04:02:41.717 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-30T04:02:41.717 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-30T04:02:41.717 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-30T04:02:41.717 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/sit.ko 2024-03-30T04:02:41.717 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/tunnel6.ko 2024-03-30T04:02:41.717 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-30T04:02:41.717 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/key 2024-03-30T04:02:41.717 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/key/af_key.ko 2024-03-30T04:02:41.718 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp 2024-03-30T04:02:41.718 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_core.ko 2024-03-30T04:02:41.718 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-30T04:02:41.718 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_eth.ko 2024-03-30T04:02:41.718 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ip.ko 2024-03-30T04:02:41.718 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ip6.ko 2024-03-30T04:02:41.718 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_netlink.ko 2024-03-30T04:02:41.718 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ppp.ko 2024-03-30T04:02:41.718 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/llc 2024-03-30T04:02:41.718 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/llc/llc.ko 2024-03-30T04:02:41.718 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mac802154 2024-03-30T04:02:41.718 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mac802154/mac802154.ko 2024-03-30T04:02:41.718 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mpls 2024-03-30T04:02:41.719 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mpls/mpls_gso.ko 2024-03-30T04:02:41.719 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter 2024-03-30T04:02:41.719 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset 2024-03-30T04:02:41.719 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set.ko 2024-03-30T04:02:41.719 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-30T04:02:41.719 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-30T04:02:41.719 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-30T04:02:41.719 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-30T04:02:41.719 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-30T04:02:41.719 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-30T04:02:41.719 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-30T04:02:41.719 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-30T04:02:41.719 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-30T04:02:41.719 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-30T04:02:41.720 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-30T04:02:41.720 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-30T04:02:41.720 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-30T04:02:41.720 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-30T04:02:41.720 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-30T04:02:41.720 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs 2024-03-30T04:02:41.720 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-30T04:02:41.720 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-30T04:02:41.720 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-30T04:02:41.720 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-30T04:02:41.720 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-30T04:02:41.720 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-30T04:02:41.720 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-30T04:02:41.720 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-30T04:02:41.721 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-30T04:02:41.721 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-30T04:02:41.721 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-30T04:02:41.721 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-30T04:02:41.721 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-30T04:02:41.721 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-30T04:02:41.721 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conncount.ko 2024-03-30T04:02:41.721 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack.ko 2024-03-30T04:02:41.721 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-30T04:02:41.721 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-30T04:02:41.721 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-30T04:02:41.721 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-30T04:02:41.721 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-30T04:02:41.721 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-30T04:02:41.722 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-30T04:02:41.722 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-30T04:02:41.722 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-30T04:02:41.722 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-30T04:02:41.722 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-30T04:02:41.722 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-30T04:02:41.722 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-30T04:02:41.722 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_flow_table.ko 2024-03-30T04:02:41.722 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-30T04:02:41.722 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_log_syslog.ko 2024-03-30T04:02:41.722 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat.ko 2024-03-30T04:02:41.722 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-30T04:02:41.722 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-30T04:02:41.723 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_irc.ko 2024-03-30T04:02:41.723 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_sip.ko 2024-03-30T04:02:41.723 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-30T04:02:41.723 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-30T04:02:41.723 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_tables.ko 2024-03-30T04:02:41.723 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink.ko 2024-03-30T04:02:41.723 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-30T04:02:41.723 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_log.ko 2024-03-30T04:02:41.723 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-30T04:02:41.723 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-30T04:02:41.723 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_chain_nat.ko 2024-03-30T04:02:41.723 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_compat.ko 2024-03-30T04:02:41.723 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_ct.ko 2024-03-30T04:02:41.723 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-30T04:02:41.724 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_fib.ko 2024-03-30T04:02:41.724 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-30T04:02:41.724 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_hash.ko 2024-03-30T04:02:41.724 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_limit.ko 2024-03-30T04:02:41.724 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_log.ko 2024-03-30T04:02:41.724 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_masq.ko 2024-03-30T04:02:41.724 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_nat.ko 2024-03-30T04:02:41.724 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_numgen.ko 2024-03-30T04:02:41.724 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_queue.ko 2024-03-30T04:02:41.724 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_quota.ko 2024-03-30T04:02:41.724 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_redir.ko 2024-03-30T04:02:41.724 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_reject.ko 2024-03-30T04:02:41.724 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_reject_inet.ko 2024-03-30T04:02:41.724 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_socket.ko 2024-03-30T04:02:41.725 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_tunnel.ko 2024-03-30T04:02:41.725 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_AUDIT.ko 2024-03-30T04:02:41.725 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-30T04:02:41.725 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-30T04:02:41.725 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-30T04:02:41.725 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CT.ko 2024-03-30T04:02:41.725 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_DSCP.ko 2024-03-30T04:02:41.725 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_HL.ko 2024-03-30T04:02:41.725 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_HMARK.ko 2024-03-30T04:02:41.725 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-30T04:02:41.725 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_LED.ko 2024-03-30T04:02:41.725 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_LOG.ko 2024-03-30T04:02:41.725 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-30T04:02:41.725 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NETMAP.ko 2024-03-30T04:02:41.726 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NFLOG.ko 2024-03-30T04:02:41.726 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-30T04:02:41.726 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_RATEEST.ko 2024-03-30T04:02:41.726 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-30T04:02:41.726 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_SECMARK.ko 2024-03-30T04:02:41.726 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-30T04:02:41.726 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-30T04:02:41.726 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TEE.ko 2024-03-30T04:02:41.726 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TPROXY.ko 2024-03-30T04:02:41.726 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TRACE.ko 2024-03-30T04:02:41.726 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_addrtype.ko 2024-03-30T04:02:41.726 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_bpf.ko 2024-03-30T04:02:41.726 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cgroup.ko 2024-03-30T04:02:41.727 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cluster.ko 2024-03-30T04:02:41.727 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_comment.ko 2024-03-30T04:02:41.727 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connbytes.ko 2024-03-30T04:02:41.727 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connlabel.ko 2024-03-30T04:02:41.727 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connlimit.ko 2024-03-30T04:02:41.727 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connmark.ko 2024-03-30T04:02:41.727 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_conntrack.ko 2024-03-30T04:02:41.727 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cpu.ko 2024-03-30T04:02:41.727 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_dccp.ko 2024-03-30T04:02:41.727 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_devgroup.ko 2024-03-30T04:02:41.727 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_dscp.ko 2024-03-30T04:02:41.727 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ecn.ko 2024-03-30T04:02:41.727 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_esp.ko 2024-03-30T04:02:41.727 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_hashlimit.ko 2024-03-30T04:02:41.728 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_helper.ko 2024-03-30T04:02:41.728 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_hl.ko 2024-03-30T04:02:41.728 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ipcomp.ko 2024-03-30T04:02:41.728 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_iprange.ko 2024-03-30T04:02:41.728 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ipvs.ko 2024-03-30T04:02:41.728 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_l2tp.ko 2024-03-30T04:02:41.728 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_length.ko 2024-03-30T04:02:41.728 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_limit.ko 2024-03-30T04:02:41.728 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_mac.ko 2024-03-30T04:02:41.728 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_mark.ko 2024-03-30T04:02:41.728 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_multiport.ko 2024-03-30T04:02:41.728 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_nat.ko 2024-03-30T04:02:41.728 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_nfacct.ko 2024-03-30T04:02:41.728 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_osf.ko 2024-03-30T04:02:41.728 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_owner.ko 2024-03-30T04:02:41.729 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_physdev.ko 2024-03-30T04:02:41.729 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_pkttype.ko 2024-03-30T04:02:41.729 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_policy.ko 2024-03-30T04:02:41.729 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_quota.ko 2024-03-30T04:02:41.729 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_rateest.ko 2024-03-30T04:02:41.729 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_realm.ko 2024-03-30T04:02:41.729 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_recent.ko 2024-03-30T04:02:41.729 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_sctp.ko 2024-03-30T04:02:41.729 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_set.ko 2024-03-30T04:02:41.729 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_socket.ko 2024-03-30T04:02:41.729 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_state.ko 2024-03-30T04:02:41.729 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_statistic.ko 2024-03-30T04:02:41.729 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_string.ko 2024-03-30T04:02:41.730 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_tcpmss.ko 2024-03-30T04:02:41.730 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_time.ko 2024-03-30T04:02:41.730 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_u32.ko 2024-03-30T04:02:41.730 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netlink 2024-03-30T04:02:41.730 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netlink/netlink_diag.ko 2024-03-30T04:02:41.730 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/nsh 2024-03-30T04:02:41.730 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/nsh/nsh.ko 2024-03-30T04:02:41.730 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch 2024-03-30T04:02:41.730 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/openvswitch.ko 2024-03-30T04:02:41.730 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-geneve.ko 2024-03-30T04:02:41.730 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-gre.ko 2024-03-30T04:02:41.730 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-vxlan.ko 2024-03-30T04:02:41.730 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/packet 2024-03-30T04:02:41.730 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/packet/af_packet_diag.ko 2024-03-30T04:02:41.731 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds 2024-03-30T04:02:41.731 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds.ko 2024-03-30T04:02:41.731 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds_rdma.ko 2024-03-30T04:02:41.731 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds_tcp.ko 2024-03-30T04:02:41.731 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched 2024-03-30T04:02:41.731 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_csum.ko 2024-03-30T04:02:41.731 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_gact.ko 2024-03-30T04:02:41.731 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_mirred.ko 2024-03-30T04:02:41.731 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_nat.ko 2024-03-30T04:02:41.731 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_pedit.ko 2024-03-30T04:02:41.731 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_police.ko 2024-03-30T04:02:41.731 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_simple.ko 2024-03-30T04:02:41.731 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_skbedit.ko 2024-03-30T04:02:41.732 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_vlan.ko 2024-03-30T04:02:41.732 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_basic.ko 2024-03-30T04:02:41.732 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_bpf.ko 2024-03-30T04:02:41.732 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_flow.ko 2024-03-30T04:02:41.732 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_fw.ko 2024-03-30T04:02:41.732 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_route.ko 2024-03-30T04:02:41.732 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_u32.ko 2024-03-30T04:02:41.732 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_cmp.ko 2024-03-30T04:02:41.732 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_ipset.ko 2024-03-30T04:02:41.732 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_meta.ko 2024-03-30T04:02:41.732 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_nbyte.ko 2024-03-30T04:02:41.732 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_text.ko 2024-03-30T04:02:41.732 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_u32.ko 2024-03-30T04:02:41.732 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_choke.ko 2024-03-30T04:02:41.733 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_codel.ko 2024-03-30T04:02:41.733 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_drr.ko 2024-03-30T04:02:41.733 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_fq.ko 2024-03-30T04:02:41.733 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_gred.ko 2024-03-30T04:02:41.733 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_hfsc.ko 2024-03-30T04:02:41.733 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_hhf.ko 2024-03-30T04:02:41.733 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_htb.ko 2024-03-30T04:02:41.733 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_ingress.ko 2024-03-30T04:02:41.733 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_mqprio.ko 2024-03-30T04:02:41.733 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_mqprio_lib.ko 2024-03-30T04:02:41.733 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_multiq.ko 2024-03-30T04:02:41.733 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_netem.ko 2024-03-30T04:02:41.733 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_pie.ko 2024-03-30T04:02:41.733 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_plug.ko 2024-03-30T04:02:41.734 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_prio.ko 2024-03-30T04:02:41.734 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_qfq.ko 2024-03-30T04:02:41.734 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_red.ko 2024-03-30T04:02:41.734 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_sfb.ko 2024-03-30T04:02:41.734 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_sfq.ko 2024-03-30T04:02:41.734 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_tbf.ko 2024-03-30T04:02:41.734 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_teql.ko 2024-03-30T04:02:41.734 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp 2024-03-30T04:02:41.734 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp/sctp.ko 2024-03-30T04:02:41.734 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp/sctp_diag.ko 2024-03-30T04:02:41.734 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc 2024-03-30T04:02:41.734 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss 2024-03-30T04:02:41.734 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-30T04:02:41.735 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-30T04:02:41.735 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/sunrpc.ko 2024-03-30T04:02:41.735 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/xprtrdma 2024-03-30T04:02:41.735 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-30T04:02:41.735 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc 2024-03-30T04:02:41.735 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc/diag.ko 2024-03-30T04:02:41.735 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc/tipc.ko 2024-03-30T04:02:41.735 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/unix 2024-03-30T04:02:41.735 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/unix/unix_diag.ko 2024-03-30T04:02:41.735 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock 2024-03-30T04:02:41.735 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-30T04:02:41.735 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-30T04:02:41.735 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock.ko 2024-03-30T04:02:41.735 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-30T04:02:41.736 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-30T04:02:41.736 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm 2024-03-30T04:02:41.736 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-30T04:02:41.736 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security 2024-03-30T04:02:41.736 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys 2024-03-30T04:02:41.736 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/encrypted-keys 2024-03-30T04:02:41.736 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-30T04:02:41.736 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/trusted-keys 2024-03-30T04:02:41.736 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/trusted-keys/trusted.ko 2024-03-30T04:02:41.736 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt 2024-03-30T04:02:41.736 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt/lib 2024-03-30T04:02:41.736 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt/lib/irqbypass.ko 2024-03-30T04:02:41.736 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.alias 2024-03-30T04:02:41.736 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.alias.bin 2024-03-30T04:02:41.737 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin 2024-03-30T04:02:41.737 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.bin 2024-03-30T04:02:41.737 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.modinfo 2024-03-30T04:02:41.737 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.dep 2024-03-30T04:02:41.737 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.dep.bin 2024-03-30T04:02:41.737 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.devname 2024-03-30T04:02:41.737 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.order 2024-03-30T04:02:41.737 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.softdep 2024-03-30T04:02:41.737 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.symbols 2024-03-30T04:02:41.737 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.symbols.bin 2024-03-30T04:02:41.737 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.8.0-g1df0c7e8a235/vmlinuz 2024-03-30T04:02:41.741 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-g1df0c7e8a235 2024-03-30T04:02:41.741 DEBUG:teuthology.orchestra.run.smithi123:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-30T04:02:41.765 INFO:teuthology.orchestra.run.smithi123.stdout:postinstall scriptlet (using /bin/sh): 2024-03-30T04:02:41.766 INFO:teuthology.orchestra.run.smithi123.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-30T04:02:41.766 INFO:teuthology.orchestra.run.smithi123.stdout: /usr/bin/kernel-install add 6.8.0-g1df0c7e8a235 /lib/modules/6.8.0-g1df0c7e8a235/vmlinuz 2024-03-30T04:02:41.766 INFO:teuthology.orchestra.run.smithi123.stdout:fi 2024-03-30T04:02:41.766 INFO:teuthology.orchestra.run.smithi123.stdout:for file in vmlinuz System.map config; do 2024-03-30T04:02:41.766 INFO:teuthology.orchestra.run.smithi123.stdout: if ! cmp --silent "/lib/modules/6.8.0-g1df0c7e8a235/${file}" "/boot/${file}-6.8.0-g1df0c7e8a235"; then 2024-03-30T04:02:41.766 INFO:teuthology.orchestra.run.smithi123.stdout: cp "/lib/modules/6.8.0-g1df0c7e8a235/${file}" "/boot/${file}-6.8.0-g1df0c7e8a235" 2024-03-30T04:02:41.766 INFO:teuthology.orchestra.run.smithi123.stdout: fi 2024-03-30T04:02:41.766 INFO:teuthology.orchestra.run.smithi123.stdout:done 2024-03-30T04:02:41.766 INFO:teuthology.orchestra.run.smithi123.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-30T04:02:41.766 INFO:teuthology.orchestra.run.smithi123.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-30T04:02:41.766 INFO:teuthology.orchestra.run.smithi123.stdout:new-kernel-pkg --remove 6.8.0-g1df0c7e8a235 --rminitrd --initrdfile=/boot/initramfs-6.8.0-g1df0c7e8a235.img 2024-03-30T04:02:41.767 INFO:teuthology.orchestra.run.smithi123.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-30T04:02:41.767 INFO:teuthology.orchestra.run.smithi123.stdout:kernel-install remove 6.8.0-g1df0c7e8a235 2024-03-30T04:02:41.767 INFO:teuthology.orchestra.run.smithi123.stdout:fi 2024-03-30T04:02:41.767 INFO:teuthology.orchestra.run.smithi123.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-30T04:02:41.767 INFO:teuthology.orchestra.run.smithi123.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-30T04:02:41.767 INFO:teuthology.orchestra.run.smithi123.stdout:/sbin/update-bootloader --remove 6.8.0-g1df0c7e8a235 2024-03-30T04:02:41.767 INFO:teuthology.orchestra.run.smithi123.stdout:fi 2024-03-30T04:02:41.768 DEBUG:teuthology.orchestra.run.smithi123:> sudo rpm -qi grub2-tools 2024-03-30T04:02:41.841 INFO:teuthology.orchestra.run.smithi123.stdout:Name : grub2-tools 2024-03-30T04:02:41.842 INFO:teuthology.orchestra.run.smithi123.stdout:Epoch : 1 2024-03-30T04:02:41.842 INFO:teuthology.orchestra.run.smithi123.stdout:Version : 2.02 2024-03-30T04:02:41.842 INFO:teuthology.orchestra.run.smithi123.stdout:Release : 129.el8 2024-03-30T04:02:41.842 INFO:teuthology.orchestra.run.smithi123.stdout:Architecture: x86_64 2024-03-30T04:02:41.842 INFO:teuthology.orchestra.run.smithi123.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-03-30T04:02:41.842 INFO:teuthology.orchestra.run.smithi123.stdout:Group : System Environment/Base 2024-03-30T04:02:41.842 INFO:teuthology.orchestra.run.smithi123.stdout:Size : 9265586 2024-03-30T04:02:41.842 INFO:teuthology.orchestra.run.smithi123.stdout:License : GPLv3+ 2024-03-30T04:02:41.842 INFO:teuthology.orchestra.run.smithi123.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-03-30T04:02:41.842 INFO:teuthology.orchestra.run.smithi123.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-03-30T04:02:41.842 INFO:teuthology.orchestra.run.smithi123.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-03-30T04:02:41.842 INFO:teuthology.orchestra.run.smithi123.stdout:Build Host : kbuilder.bsys.centos.org 2024-03-30T04:02:41.843 INFO:teuthology.orchestra.run.smithi123.stdout:Relocations : (not relocatable) 2024-03-30T04:02:41.843 INFO:teuthology.orchestra.run.smithi123.stdout:Packager : CentOS BuildSystem 2024-03-30T04:02:41.843 INFO:teuthology.orchestra.run.smithi123.stdout:Vendor : CentOS 2024-03-30T04:02:41.843 INFO:teuthology.orchestra.run.smithi123.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-30T04:02:41.843 INFO:teuthology.orchestra.run.smithi123.stdout:Summary : Support tools for GRUB. 2024-03-30T04:02:41.843 INFO:teuthology.orchestra.run.smithi123.stdout:Description : 2024-03-30T04:02:41.843 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:02:41.843 INFO:teuthology.orchestra.run.smithi123.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-30T04:02:41.843 INFO:teuthology.orchestra.run.smithi123.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-30T04:02:41.843 INFO:teuthology.orchestra.run.smithi123.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-30T04:02:41.843 INFO:teuthology.orchestra.run.smithi123.stdout:hardware devices. 2024-03-30T04:02:41.843 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:02:41.843 INFO:teuthology.orchestra.run.smithi123.stdout:This subpackage provides tools for support of all platforms. 2024-03-30T04:02:41.844 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-30T04:02:41.844 INFO:teuthology.task.kernel:Updating grub on smithi123 to boot 6.8.0-g1df0c7e8a235 2024-03-30T04:02:41.845 DEBUG:teuthology.orchestra.run.smithi123:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-30T04:02:46.270 INFO:teuthology.orchestra.run.smithi123.stderr:Generating grub configuration file ... 2024-03-30T04:02:48.669 INFO:teuthology.orchestra.run.smithi123.stderr:done 2024-03-30T04:02:48.681 DEBUG:teuthology.orchestra.run.smithi123:> mktemp 2024-03-30T04:02:48.713 INFO:teuthology.orchestra.run.smithi123.stdout:/tmp/tmp.e3LIJ1RaVg 2024-03-30T04:02:48.713 DEBUG:teuthology.orchestra.run.smithi123:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.e3LIJ1RaVg 2024-03-30T04:02:48.787 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 0666 /tmp/tmp.e3LIJ1RaVg 2024-03-30T04:02:48.962 DEBUG:teuthology.orchestra.remote:smithi123:/tmp/tmp.e3LIJ1RaVg is 6KB 2024-03-30T04:02:49.009 DEBUG:teuthology.orchestra.run.smithi123:> rm -fr /tmp/tmp.e3LIJ1RaVg 2024-03-30T04:02:49.027 DEBUG:teuthology.orchestra.run.smithi123:> sudo /bin/ls /boot/loader/entries || true 2024-03-30T04:02:49.092 INFO:teuthology.orchestra.run.smithi123.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-03-30T04:02:49.092 INFO:teuthology.orchestra.run.smithi123.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-03-30T04:02:49.092 INFO:teuthology.orchestra.run.smithi123.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-03-30T04:02:49.092 INFO:teuthology.orchestra.run.smithi123.stdout:a1b29d6c289747bdb22930598d39874a-6.8.0-g1df0c7e8a235.conf 2024-03-30T04:02:49.093 DEBUG:teuthology.orchestra.run.smithi123:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-6.8.0-g1df0c7e8a235 2024-03-30T04:02:49.267 DEBUG:teuthology.orchestra.run.smithi123:> sudo shutdown -r now 2024-03-30T04:03:00.218 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-03-30T04:03:00.219 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi135.front.sepia.ceph.com' 2024-03-30T04:03:00.219 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi135.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T04:03:19.286 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-03-30T04:03:19.287 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-03-30T04:03:19.288 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T04:03:34.815 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.135 2024-03-30T04:03:43.823 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi135.front.sepia.ceph.com' 2024-03-30T04:03:43.823 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi135.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T04:04:02.206 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.135 2024-03-30T04:04:14.218 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi135.front.sepia.ceph.com' 2024-03-30T04:04:14.219 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi135.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T04:04:15.659 DEBUG:teuthology.orchestra.run.smithi135:> true 2024-03-30T04:04:16.086 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi135.front.sepia.ceph.com' 2024-03-30T04:04:16.087 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "6.8.0-g1df0c7e8a235"... 2024-03-30T04:04:16.087 DEBUG:teuthology.orchestra.run.smithi135:> uname -r 2024-03-30T04:04:16.131 INFO:teuthology.orchestra.run.smithi135.stdout:6.8.0-g1df0c7e8a235 2024-03-30T04:04:16.131 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-g1df0c7e8a235 vs 6.8.0-g1df0c7e8a235 2024-03-30T04:04:16.131 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-30T04:04:16.131 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-30T04:04:17.132 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-03-30T04:04:17.132 DEBUG:teuthology.orchestra.run.smithi135:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-30T04:04:17.244 INFO:teuthology.orchestra.run.smithi135.stdout:ttyS1 2024-03-30T04:04:17.266 DEBUG:teuthology.parallel:result is None 2024-03-30T04:04:19.290 DEBUG:teuthology.orchestra.remote:timed out 2024-03-30T04:04:28.291 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-03-30T04:04:28.292 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T04:04:31.715 DEBUG:teuthology.orchestra.run.smithi123:> true 2024-03-30T04:04:32.162 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-03-30T04:04:32.163 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "6.8.0-g1df0c7e8a235"... 2024-03-30T04:04:32.163 DEBUG:teuthology.orchestra.run.smithi123:> uname -r 2024-03-30T04:04:32.214 INFO:teuthology.orchestra.run.smithi123.stdout:6.8.0-g1df0c7e8a235 2024-03-30T04:04:32.214 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-g1df0c7e8a235 vs 6.8.0-g1df0c7e8a235 2024-03-30T04:04:32.214 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-30T04:04:32.214 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-30T04:04:33.215 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-03-30T04:04:33.215 DEBUG:teuthology.orchestra.run.smithi123:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-30T04:04:33.338 INFO:teuthology.orchestra.run.smithi123.stdout:ttyS1 2024-03-30T04:04:33.371 DEBUG:teuthology.parallel:result is None 2024-03-30T04:04:33.372 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-30T04:04:33.380 INFO:teuthology.task.internal:Creating test directory... 2024-03-30T04:04:33.380 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-30T04:04:33.384 DEBUG:teuthology.orchestra.run.smithi135:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-30T04:04:33.436 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-30T04:04:33.443 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-30T04:04:33.453 INFO:teuthology.task.internal:Creating archive directory... 2024-03-30T04:04:33.453 DEBUG:teuthology.orchestra.run.smithi123:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-30T04:04:33.457 DEBUG:teuthology.orchestra.run.smithi135:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-30T04:04:33.518 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-30T04:04:33.525 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-30T04:04:33.525 DEBUG:teuthology.orchestra.run.smithi123:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-30T04:04:33.531 DEBUG:teuthology.orchestra.run.smithi135:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-30T04:04:33.604 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T04:04:33.625 INFO:teuthology.orchestra.run.smithi135.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T04:04:33.629 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T04:04:33.650 INFO:teuthology.orchestra.run.smithi135.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T04:04:33.652 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-30T04:04:33.660 INFO:teuthology.task.internal:Configuring sudo... 2024-03-30T04:04:33.660 DEBUG:teuthology.orchestra.run.smithi123:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-30T04:04:33.665 DEBUG:teuthology.orchestra.run.smithi135:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-30T04:04:33.731 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-30T04:04:33.741 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-30T04:04:33.741 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-30T04:04:33.745 DEBUG:teuthology.orchestra.run.smithi135:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-30T04:04:33.788 DEBUG:teuthology.orchestra.run.smithi123:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-30T04:04:33.898 DEBUG:teuthology.orchestra.run.smithi123:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-30T04:04:33.982 DEBUG:teuthology.orchestra.run.smithi123:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-30T04:04:34.053 DEBUG:teuthology.orchestra.run.smithi123:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-30T04:04:34.128 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-30T04:04:34.128 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-30T04:04:34.210 DEBUG:teuthology.orchestra.run.smithi135:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-30T04:04:34.283 DEBUG:teuthology.orchestra.run.smithi135:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-30T04:04:34.353 DEBUG:teuthology.orchestra.run.smithi135:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-30T04:04:34.428 DEBUG:teuthology.orchestra.run.smithi135:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-30T04:04:34.494 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-30T04:04:34.494 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-30T04:04:34.560 DEBUG:teuthology.orchestra.run.smithi123:> sudo service rsyslog restart 2024-03-30T04:04:34.564 DEBUG:teuthology.orchestra.run.smithi135:> sudo service rsyslog restart 2024-03-30T04:04:34.688 INFO:teuthology.orchestra.run.smithi123.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-30T04:04:34.713 INFO:teuthology.orchestra.run.smithi135.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-30T04:04:35.159 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-30T04:04:35.168 INFO:teuthology.task.internal:Starting timer... 2024-03-30T04:04:35.168 INFO:teuthology.run_tasks:Running task pcp... 2024-03-30T04:04:35.207 INFO:teuthology.run_tasks:Running task selinux... 2024-03-30T04:04:35.219 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-30T04:04:35.220 DEBUG:teuthology.orchestra.run.smithi123:> sudo service auditd rotate 2024-03-30T04:04:35.338 INFO:teuthology.orchestra.run.smithi123.stdout:Rotating logs: 2024-03-30T04:04:35.342 DEBUG:teuthology.orchestra.run.smithi135:> sudo service auditd rotate 2024-03-30T04:04:35.454 INFO:teuthology.orchestra.run.smithi135.stdout:Rotating logs: 2024-03-30T04:04:35.457 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-30T04:04:35.457 DEBUG:teuthology.orchestra.run.smithi123:> /usr/sbin/getenforce 2024-03-30T04:04:35.511 INFO:teuthology.orchestra.run.smithi123.stdout:Permissive 2024-03-30T04:04:35.512 DEBUG:teuthology.orchestra.run.smithi135:> /usr/sbin/getenforce 2024-03-30T04:04:35.563 INFO:teuthology.orchestra.run.smithi135.stdout:Permissive 2024-03-30T04:04:35.563 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi123.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi135.front.sepia.ceph.com': 'permissive'} 2024-03-30T04:04:35.564 DEBUG:teuthology.orchestra.run.smithi123:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-30T04:04:35.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T04:04:35.633 DEBUG:teuthology.orchestra.run.smithi135:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-30T04:04:35.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T04:04:35.706 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-30T04:04:35.706 DEBUG:teuthology.orchestra.run.smithi123:> sudo /usr/sbin/setenforce permissive 2024-03-30T04:04:35.774 DEBUG:teuthology.orchestra.run.smithi135:> sudo /usr/sbin/setenforce permissive 2024-03-30T04:04:35.841 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-30T04:04:35.851 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-30T04:04:35.857 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-30T04:04:36.009 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-30T04:04:36.027 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-30T04:04:36.028 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi123.front.sepia.ceph.com,smithi135.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-30T04:11:46.820 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi123.front.sepia.ceph.com'), Remote(name='ubuntu@smithi135.front.sepia.ceph.com')] 2024-03-30T04:11:46.821 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-03-30T04:11:46.822 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T04:11:46.912 DEBUG:teuthology.orchestra.run.smithi123:> true 2024-03-30T04:11:47.021 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-03-30T04:11:47.022 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi135.front.sepia.ceph.com' 2024-03-30T04:11:47.022 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi135.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T04:11:47.110 DEBUG:teuthology.orchestra.run.smithi135:> true 2024-03-30T04:11:47.227 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi135.front.sepia.ceph.com' 2024-03-30T04:11:47.228 INFO:teuthology.run_tasks:Running task clock... 2024-03-30T04:11:47.238 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-30T04:11:47.239 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-30T04:11:47.239 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-30T04:11:47.243 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-30T04:11:47.243 DEBUG:teuthology.orchestra.run.smithi135:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-30T04:11:47.310 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-30T04:11:47.340 INFO:teuthology.orchestra.run.smithi135.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-30T04:11:47.341 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-30T04:11:47.371 INFO:teuthology.orchestra.run.smithi135.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-30T04:11:47.427 INFO:teuthology.orchestra.run.smithi123.stderr:sudo: ntpd: command not found 2024-03-30T04:11:47.453 INFO:teuthology.orchestra.run.smithi123.stdout:506 Cannot talk to daemon 2024-03-30T04:11:47.466 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: ntpd: command not found 2024-03-30T04:11:47.484 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-30T04:11:47.492 INFO:teuthology.orchestra.run.smithi135.stdout:506 Cannot talk to daemon 2024-03-30T04:11:47.515 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-30T04:11:47.522 INFO:teuthology.orchestra.run.smithi135.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-30T04:11:47.553 INFO:teuthology.orchestra.run.smithi135.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-30T04:11:47.602 INFO:teuthology.orchestra.run.smithi123.stderr:bash: ntpq: command not found 2024-03-30T04:11:47.609 INFO:teuthology.orchestra.run.smithi123.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-30T04:11:47.609 INFO:teuthology.orchestra.run.smithi123.stdout:=============================================================================== 2024-03-30T04:11:47.610 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T04:11:47.610 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T04:11:47.610 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T04:11:47.610 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T04:11:47.637 INFO:teuthology.orchestra.run.smithi135.stderr:bash: ntpq: command not found 2024-03-30T04:11:47.644 INFO:teuthology.orchestra.run.smithi135.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-30T04:11:47.644 INFO:teuthology.orchestra.run.smithi135.stdout:=============================================================================== 2024-03-30T04:11:47.644 INFO:teuthology.orchestra.run.smithi135.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T04:11:47.644 INFO:teuthology.orchestra.run.smithi135.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T04:11:47.644 INFO:teuthology.orchestra.run.smithi135.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T04:11:47.644 INFO:teuthology.orchestra.run.smithi135.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T04:11:47.646 INFO:teuthology.run_tasks:Running task pexec... 2024-03-30T04:11:47.656 INFO:teuthology.task.pexec:Executing custom commands... 2024-03-30T04:11:47.656 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi123.front.sepia.ceph.com 2024-03-30T04:11:47.656 DEBUG:teuthology.orchestra.run.smithi123:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-30T04:11:47.657 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi135.front.sepia.ceph.com 2024-03-30T04:11:47.657 DEBUG:teuthology.orchestra.run.smithi135:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-30T04:11:48.579 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:32 ago on Sat 30 Mar 2024 04:10:16 AM UTC. 2024-03-30T04:11:48.591 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:37 ago on Sat 30 Mar 2024 04:10:11 AM UTC. 2024-03-30T04:11:50.749 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-30T04:11:50.750 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-30T04:11:50.750 INFO:teuthology.orchestra.run.smithi123.stdout: Package Architecture Version Repository Size 2024-03-30T04:11:50.750 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-30T04:11:50.750 INFO:teuthology.orchestra.run.smithi123.stdout:Resetting modules: 2024-03-30T04:11:50.750 INFO:teuthology.orchestra.run.smithi123.stdout: container-tools 2024-03-30T04:11:50.750 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:11:50.750 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-30T04:11:50.750 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-30T04:11:50.751 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:11:50.769 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-30T04:11:50.771 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-30T04:11:50.771 INFO:teuthology.orchestra.run.smithi135.stdout: Package Architecture Version Repository Size 2024-03-30T04:11:50.771 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-30T04:11:50.771 INFO:teuthology.orchestra.run.smithi135.stdout:Resetting modules: 2024-03-30T04:11:50.771 INFO:teuthology.orchestra.run.smithi135.stdout: container-tools 2024-03-30T04:11:50.771 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:11:50.771 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-03-30T04:11:50.771 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-30T04:11:50.771 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:11:51.039 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-30T04:11:51.059 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-30T04:11:52.075 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:40 ago on Sat 30 Mar 2024 04:10:11 AM UTC. 2024-03-30T04:11:52.077 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:35 ago on Sat 30 Mar 2024 04:10:16 AM UTC. 2024-03-30T04:11:54.248 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-30T04:11:54.287 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-30T04:11:54.287 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2024-03-30T04:11:54.287 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-30T04:11:54.287 INFO:teuthology.orchestra.run.smithi123.stdout:Installing group/module packages: 2024-03-30T04:11:54.287 INFO:teuthology.orchestra.run.smithi123.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-03-30T04:11:54.288 INFO:teuthology.orchestra.run.smithi123.stdout: buildah x86_64 2:1.33.6-2.module_el8+976+11360731 appstream 9.6 M 2024-03-30T04:11:54.288 INFO:teuthology.orchestra.run.smithi123.stdout: cockpit-podman noarch 84.1-1.module_el8+977+8ab54275 appstream 689 k 2024-03-30T04:11:54.288 INFO:teuthology.orchestra.run.smithi123.stdout: crun x86_64 1.14.3-2.module_el8+968+fbb249c7 appstream 258 k 2024-03-30T04:11:54.288 INFO:teuthology.orchestra.run.smithi123.stdout: netavark x86_64 2:1.10.3-1.module_el8+951+32019cde appstream 4.1 M 2024-03-30T04:11:54.288 INFO:teuthology.orchestra.run.smithi123.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-03-30T04:11:54.288 INFO:teuthology.orchestra.run.smithi123.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-03-30T04:11:54.288 INFO:teuthology.orchestra.run.smithi123.stdout: toolbox x86_64 0.0.99.5-2.module_el8+971+3d3df00d appstream 2.5 M 2024-03-30T04:11:54.288 INFO:teuthology.orchestra.run.smithi123.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-03-30T04:11:54.288 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2024-03-30T04:11:54.288 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-03-30T04:11:54.288 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-03-30T04:11:54.288 INFO:teuthology.orchestra.run.smithi123.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-03-30T04:11:54.288 INFO:teuthology.orchestra.run.smithi123.stdout:Installing weak dependencies: 2024-03-30T04:11:54.288 INFO:teuthology.orchestra.run.smithi123.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-03-30T04:11:54.289 INFO:teuthology.orchestra.run.smithi123.stdout:Installing module profiles: 2024-03-30T04:11:54.289 INFO:teuthology.orchestra.run.smithi123.stdout: container-tools/common 2024-03-30T04:11:54.289 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:11:54.289 INFO:teuthology.orchestra.run.smithi123.stdout:Enabling module streams: 2024-03-30T04:11:54.289 INFO:teuthology.orchestra.run.smithi123.stdout: container-tools rhel8 2024-03-30T04:11:54.289 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:11:54.289 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-30T04:11:54.289 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-30T04:11:54.289 INFO:teuthology.orchestra.run.smithi123.stdout:Install 13 Packages 2024-03-30T04:11:54.289 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:11:54.290 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 27 M 2024-03-30T04:11:54.290 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 91 M 2024-03-30T04:11:54.290 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-30T04:11:54.297 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-30T04:11:54.336 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-30T04:11:54.336 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2024-03-30T04:11:54.336 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-30T04:11:54.337 INFO:teuthology.orchestra.run.smithi135.stdout:Installing group/module packages: 2024-03-30T04:11:54.337 INFO:teuthology.orchestra.run.smithi135.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-03-30T04:11:54.337 INFO:teuthology.orchestra.run.smithi135.stdout: buildah x86_64 2:1.33.6-2.module_el8+976+11360731 appstream 9.6 M 2024-03-30T04:11:54.337 INFO:teuthology.orchestra.run.smithi135.stdout: cockpit-podman noarch 84.1-1.module_el8+977+8ab54275 appstream 689 k 2024-03-30T04:11:54.337 INFO:teuthology.orchestra.run.smithi135.stdout: crun x86_64 1.14.3-2.module_el8+968+fbb249c7 appstream 258 k 2024-03-30T04:11:54.337 INFO:teuthology.orchestra.run.smithi135.stdout: netavark x86_64 2:1.10.3-1.module_el8+951+32019cde appstream 4.1 M 2024-03-30T04:11:54.337 INFO:teuthology.orchestra.run.smithi135.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-03-30T04:11:54.337 INFO:teuthology.orchestra.run.smithi135.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-03-30T04:11:54.337 INFO:teuthology.orchestra.run.smithi135.stdout: toolbox x86_64 0.0.99.5-2.module_el8+971+3d3df00d appstream 2.5 M 2024-03-30T04:11:54.337 INFO:teuthology.orchestra.run.smithi135.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-03-30T04:11:54.337 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2024-03-30T04:11:54.337 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-03-30T04:11:54.337 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-03-30T04:11:54.337 INFO:teuthology.orchestra.run.smithi135.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-03-30T04:11:54.337 INFO:teuthology.orchestra.run.smithi135.stdout:Installing weak dependencies: 2024-03-30T04:11:54.338 INFO:teuthology.orchestra.run.smithi135.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-03-30T04:11:54.338 INFO:teuthology.orchestra.run.smithi135.stdout:Installing module profiles: 2024-03-30T04:11:54.338 INFO:teuthology.orchestra.run.smithi135.stdout: container-tools/common 2024-03-30T04:11:54.338 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:11:54.338 INFO:teuthology.orchestra.run.smithi135.stdout:Enabling module streams: 2024-03-30T04:11:54.338 INFO:teuthology.orchestra.run.smithi135.stdout: container-tools rhel8 2024-03-30T04:11:54.338 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:11:54.338 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-03-30T04:11:54.338 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-30T04:11:54.338 INFO:teuthology.orchestra.run.smithi135.stdout:Install 13 Packages 2024-03-30T04:11:54.338 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:11:54.339 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 27 M 2024-03-30T04:11:54.339 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 91 M 2024-03-30T04:11:54.339 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-03-30T04:11:54.902 INFO:teuthology.orchestra.run.smithi123.stdout:(1/13): cockpit-podman-84.1-1.module_el8+977+8a 1.2 MB/s | 689 kB 00:00 2024-03-30T04:11:54.952 INFO:teuthology.orchestra.run.smithi123.stdout:(2/13): aardvark-dns-1.10.0-1.module_el8+871+55 1.7 MB/s | 1.0 MB 00:00 2024-03-30T04:11:55.002 INFO:teuthology.orchestra.run.smithi123.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 379 kB/s | 38 kB 00:00 2024-03-30T04:11:55.053 INFO:teuthology.orchestra.run.smithi123.stdout:(4/13): crun-1.14.3-2.module_el8+968+fbb249c7.x 2.5 MB/s | 258 kB 00:00 2024-03-30T04:11:55.099 INFO:teuthology.orchestra.run.smithi135.stdout:(1/13): cockpit-podman-84.1-1.module_el8+977+8a 988 kB/s | 689 kB 00:00 2024-03-30T04:11:55.132 INFO:teuthology.orchestra.run.smithi135.stdout:(2/13): aardvark-dns-1.10.0-1.module_el8+871+55 1.4 MB/s | 1.0 MB 00:00 2024-03-30T04:11:55.178 INFO:teuthology.orchestra.run.smithi123.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 1.4 MB/s | 180 kB 00:00 2024-03-30T04:11:55.199 INFO:teuthology.orchestra.run.smithi135.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 379 kB/s | 38 kB 00:00 2024-03-30T04:11:55.241 INFO:teuthology.orchestra.run.smithi135.stdout:(4/13): crun-1.14.3-2.module_el8+968+fbb249c7.x 2.3 MB/s | 258 kB 00:00 2024-03-30T04:11:55.312 INFO:teuthology.orchestra.run.smithi123.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 185 kB/s | 25 kB 00:00 2024-03-30T04:11:55.421 INFO:teuthology.orchestra.run.smithi123.stdout:(7/13): buildah-1.33.6-2.module_el8+976+1136073 9.0 MB/s | 9.6 MB 00:01 2024-03-30T04:11:55.450 INFO:teuthology.orchestra.run.smithi135.stdout:(5/13): buildah-1.33.6-2.module_el8+976+1136073 9.2 MB/s | 9.6 MB 00:01 2024-03-30T04:11:55.454 INFO:teuthology.orchestra.run.smithi123.stdout:(8/13): python3-pyxdg-0.25-16.el8.noarch.rpm 663 kB/s | 94 kB 00:00 2024-03-30T04:11:55.475 INFO:teuthology.orchestra.run.smithi135.stdout:(6/13): python3-podman-4.9.0-1.module_el8+847+7 772 kB/s | 180 kB 00:00 2024-03-30T04:11:55.521 INFO:teuthology.orchestra.run.smithi123.stdout:(9/13): netavark-1.10.3-1.module_el8+951+32019c 7.9 MB/s | 4.1 MB 00:00 2024-03-30T04:11:55.559 INFO:teuthology.orchestra.run.smithi135.stdout:(7/13): python3-pytoml-0.1.14-5.git7dea353.el8. 228 kB/s | 25 kB 00:00 2024-03-30T04:11:55.617 INFO:teuthology.orchestra.run.smithi135.stdout:(8/13): netavark-1.10.3-1.module_el8+951+32019c 9.8 MB/s | 4.1 MB 00:00 2024-03-30T04:11:55.621 INFO:teuthology.orchestra.run.smithi123.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 527 kB/s | 53 kB 00:00 2024-03-30T04:11:55.642 INFO:teuthology.orchestra.run.smithi135.stdout:(9/13): python3-pyxdg-0.25-16.el8.noarch.rpm 561 kB/s | 94 kB 00:00 2024-03-30T04:11:55.738 INFO:teuthology.orchestra.run.smithi123.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 352 kB/s | 41 kB 00:00 2024-03-30T04:11:55.743 INFO:teuthology.orchestra.run.smithi135.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 526 kB/s | 53 kB 00:00 2024-03-30T04:11:55.789 INFO:teuthology.orchestra.run.smithi123.stdout:(12/13): toolbox-0.0.99.5-2.module_el8+971+3d3d 7.6 MB/s | 2.5 MB 00:00 2024-03-30T04:11:55.835 INFO:teuthology.orchestra.run.smithi135.stdout:(11/13): toolbox-0.0.99.5-2.module_el8+971+3d3d 12 MB/s | 2.5 MB 00:00 2024-03-30T04:11:55.860 INFO:teuthology.orchestra.run.smithi135.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 350 kB/s | 41 kB 00:00 2024-03-30T04:11:55.889 INFO:teuthology.orchestra.run.smithi123.stdout:(13/13): skopeo-1.14.3-0.1.module_el8+885+7da14 19 MB/s | 8.8 MB 00:00 2024-03-30T04:11:55.889 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-30T04:11:55.889 INFO:teuthology.orchestra.run.smithi123.stdout:Total 17 MB/s | 27 MB 00:01 2024-03-30T04:11:56.077 INFO:teuthology.orchestra.run.smithi135.stdout:(13/13): skopeo-1.14.3-0.1.module_el8+885+7da14 17 MB/s | 8.8 MB 00:00 2024-03-30T04:11:56.077 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-03-30T04:11:56.077 INFO:teuthology.orchestra.run.smithi135.stdout:Total 16 MB/s | 27 MB 00:01 2024-03-30T04:11:56.297 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-30T04:11:56.323 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-30T04:11:56.323 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-30T04:11:56.490 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-03-30T04:11:56.515 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-03-30T04:11:56.516 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-03-30T04:11:56.591 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-30T04:11:56.595 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-30T04:11:56.791 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-03-30T04:11:56.795 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-03-30T04:11:57.316 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-30T04:11:57.525 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-03-30T04:11:58.326 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-03-30T04:11:58.530 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-03-30T04:11:58.532 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-03-30T04:11:58.723 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-03-30T04:11:58.751 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-03-30T04:11:58.929 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-03-30T04:11:58.952 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-03-30T04:11:59.160 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-03-30T04:11:59.187 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-03-30T04:11:59.415 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-03-30T04:11:59.740 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-03-30T04:11:59.982 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-03-30T04:12:00.872 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 7/13 2024-03-30T04:12:01.061 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 7/13 2024-03-30T04:12:01.115 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 8/13 2024-03-30T04:12:01.317 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 8/13 2024-03-30T04:12:01.362 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 9/13 2024-03-30T04:12:01.581 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 9/13 2024-03-30T04:12:01.833 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-03-30T04:12:02.047 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-03-30T04:12:02.075 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-03-30T04:12:02.293 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-03-30T04:12:02.293 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-03-30T04:12:02.462 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-03-30T04:12:02.518 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-03-30T04:12:02.725 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-03-30T04:12:03.121 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-03-30T04:12:03.121 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-03-30T04:12:03.121 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2/13 2024-03-30T04:12:03.121 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 3/13 2024-03-30T04:12:03.121 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-03-30T04:12:03.121 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 5/13 2024-03-30T04:12:03.122 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 6/13 2024-03-30T04:12:03.122 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-03-30T04:12:03.122 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-03-30T04:12:03.122 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-03-30T04:12:03.122 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-03-30T04:12:03.122 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-03-30T04:12:03.122 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-03-30T04:12:03.418 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-03-30T04:12:03.419 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-03-30T04:12:03.419 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2/13 2024-03-30T04:12:03.419 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 3/13 2024-03-30T04:12:03.419 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-03-30T04:12:03.419 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 5/13 2024-03-30T04:12:03.419 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 6/13 2024-03-30T04:12:03.419 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-03-30T04:12:03.419 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-03-30T04:12:03.419 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-03-30T04:12:03.419 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-03-30T04:12:03.419 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-03-30T04:12:03.419 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-03-30T04:12:03.636 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-03-30T04:12:03.636 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:12:03.636 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-30T04:12:03.636 INFO:teuthology.orchestra.run.smithi123.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-03-30T04:12:03.636 INFO:teuthology.orchestra.run.smithi123.stdout: buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2024-03-30T04:12:03.636 INFO:teuthology.orchestra.run.smithi123.stdout: cockpit-podman-84.1-1.module_el8+977+8ab54275.noarch 2024-03-30T04:12:03.636 INFO:teuthology.orchestra.run.smithi123.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-03-30T04:12:03.636 INFO:teuthology.orchestra.run.smithi123.stdout: crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 2024-03-30T04:12:03.636 INFO:teuthology.orchestra.run.smithi123.stdout: netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 2024-03-30T04:12:03.636 INFO:teuthology.orchestra.run.smithi123.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-03-30T04:12:03.636 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-03-30T04:12:03.637 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-03-30T04:12:03.637 INFO:teuthology.orchestra.run.smithi123.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-03-30T04:12:03.637 INFO:teuthology.orchestra.run.smithi123.stdout: toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 2024-03-30T04:12:03.637 INFO:teuthology.orchestra.run.smithi123.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-03-30T04:12:03.637 INFO:teuthology.orchestra.run.smithi123.stdout: yajl-2.1.0-12.el8.x86_64 2024-03-30T04:12:03.637 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:12:03.637 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-30T04:12:03.859 DEBUG:teuthology.parallel:result is None 2024-03-30T04:12:04.156 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-03-30T04:12:04.156 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:12:04.156 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-03-30T04:12:04.156 INFO:teuthology.orchestra.run.smithi135.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-03-30T04:12:04.156 INFO:teuthology.orchestra.run.smithi135.stdout: buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2024-03-30T04:12:04.156 INFO:teuthology.orchestra.run.smithi135.stdout: cockpit-podman-84.1-1.module_el8+977+8ab54275.noarch 2024-03-30T04:12:04.156 INFO:teuthology.orchestra.run.smithi135.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-03-30T04:12:04.156 INFO:teuthology.orchestra.run.smithi135.stdout: crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 2024-03-30T04:12:04.157 INFO:teuthology.orchestra.run.smithi135.stdout: netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 2024-03-30T04:12:04.157 INFO:teuthology.orchestra.run.smithi135.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-03-30T04:12:04.157 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-03-30T04:12:04.157 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-03-30T04:12:04.157 INFO:teuthology.orchestra.run.smithi135.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-03-30T04:12:04.157 INFO:teuthology.orchestra.run.smithi135.stdout: toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 2024-03-30T04:12:04.157 INFO:teuthology.orchestra.run.smithi135.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-03-30T04:12:04.157 INFO:teuthology.orchestra.run.smithi135.stdout: yajl-2.1.0-12.el8.x86_64 2024-03-30T04:12:04.157 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:12:04.157 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-30T04:12:04.407 DEBUG:teuthology.parallel:result is None 2024-03-30T04:12:04.407 INFO:teuthology.run_tasks:Running task install... 2024-03-30T04:12:04.417 DEBUG:teuthology.task.install:project ceph 2024-03-30T04:12:04.417 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '470100a4ccbcf7769052bf09a5413881ccf1b97f'}} 2024-03-30T04:12:04.417 DEBUG:teuthology.task.install:config {'branch': 'quincy', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': '470100a4ccbcf7769052bf09a5413881ccf1b97f'} 2024-03-30T04:12:04.417 INFO:teuthology.task.install:Using flavor: default 2024-03-30T04:12:04.425 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-30T04:12:04.425 INFO:teuthology.task.install:extra packages: [] 2024-03-30T04:12:04.426 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '470100a4ccbcf7769052bf09a5413881ccf1b97f', 'tag': None, 'wait_for_package': False} 2024-03-30T04:12:04.426 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-30T04:12:04.426 INFO:teuthology.packaging:ref: None 2024-03-30T04:12:04.426 INFO:teuthology.packaging:tag: None 2024-03-30T04:12:04.426 INFO:teuthology.packaging:branch: quincy 2024-03-30T04:12:04.426 INFO:teuthology.packaging:sha1: 470100a4ccbcf7769052bf09a5413881ccf1b97f 2024-03-30T04:12:04.426 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=quincy 2024-03-30T04:12:04.428 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '470100a4ccbcf7769052bf09a5413881ccf1b97f', 'tag': None, 'wait_for_package': False} 2024-03-30T04:12:04.428 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-30T04:12:04.428 INFO:teuthology.packaging:ref: None 2024-03-30T04:12:04.429 INFO:teuthology.packaging:tag: None 2024-03-30T04:12:04.429 INFO:teuthology.packaging:branch: quincy 2024-03-30T04:12:04.429 INFO:teuthology.packaging:sha1: 470100a4ccbcf7769052bf09a5413881ccf1b97f 2024-03-30T04:12:04.429 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=quincy 2024-03-30T04:12:04.668 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/quincy/5c02433542ab1af3975314808d208cabd22abc39/centos/8/flavors/default/ 2024-03-30T04:12:04.668 INFO:teuthology.task.install.rpm:Package version is 17.2.7-797.g5c024335 2024-03-30T04:12:04.768 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/quincy/5c02433542ab1af3975314808d208cabd22abc39/centos/8/flavors/default/ 2024-03-30T04:12:04.769 INFO:teuthology.task.install.rpm:Package version is 17.2.7-797.g5c024335 2024-03-30T04:12:04.840 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/quincy/5c02433542ab1af3975314808d208cabd22abc39/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/5c02433542ab1af3975314808d208cabd22abc39/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/5c02433542ab1af3975314808d208cabd22abc39/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-30T04:12:04.840 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-30T04:12:04.840 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-30T04:12:04.912 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-30T04:12:04.913 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-30T04:12:04.913 INFO:teuthology.packaging:ref: None 2024-03-30T04:12:04.913 INFO:teuthology.packaging:tag: None 2024-03-30T04:12:04.913 INFO:teuthology.packaging:branch: quincy 2024-03-30T04:12:04.913 INFO:teuthology.packaging:sha1: 470100a4ccbcf7769052bf09a5413881ccf1b97f 2024-03-30T04:12:04.913 DEBUG:teuthology.orchestra.run.smithi135:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-30T04:12:04.935 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/quincy/5c02433542ab1af3975314808d208cabd22abc39/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/5c02433542ab1af3975314808d208cabd22abc39/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/5c02433542ab1af3975314808d208cabd22abc39/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-30T04:12:04.935 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-30T04:12:04.936 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-30T04:12:05.003 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-30T04:12:05.003 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-30T04:12:05.003 INFO:teuthology.packaging:ref: None 2024-03-30T04:12:05.003 INFO:teuthology.packaging:tag: None 2024-03-30T04:12:05.004 INFO:teuthology.packaging:branch: quincy 2024-03-30T04:12:05.004 INFO:teuthology.packaging:sha1: 470100a4ccbcf7769052bf09a5413881ccf1b97f 2024-03-30T04:12:05.004 DEBUG:teuthology.orchestra.run.smithi123:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-30T04:12:05.023 DEBUG:teuthology.orchestra.run.smithi135:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-30T04:12:05.111 DEBUG:teuthology.orchestra.run.smithi123:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-30T04:12:05.115 DEBUG:teuthology.orchestra.run.smithi135:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-30T04:12:05.182 INFO:teuthology.orchestra.run.smithi135.stdout:check_obsoletes = 1 2024-03-30T04:12:05.185 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum clean all 2024-03-30T04:12:05.201 DEBUG:teuthology.orchestra.run.smithi123:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-30T04:12:05.266 INFO:teuthology.orchestra.run.smithi123.stdout:check_obsoletes = 1 2024-03-30T04:12:05.269 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum clean all 2024-03-30T04:12:05.533 INFO:teuthology.orchestra.run.smithi135.stdout:76 files removed 2024-03-30T04:12:05.569 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-radosgw 2024-03-30T04:12:05.617 INFO:teuthology.orchestra.run.smithi123.stdout:76 files removed 2024-03-30T04:12:05.653 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-radosgw 2024-03-30T04:12:06.365 INFO:teuthology.orchestra.run.smithi135.stdout:ceph packages for x86_64 318 kB/s | 78 kB 00:00 2024-03-30T04:12:06.549 INFO:teuthology.orchestra.run.smithi123.stdout:ceph packages for x86_64 218 kB/s | 78 kB 00:00 2024-03-30T04:12:06.682 INFO:teuthology.orchestra.run.smithi135.stdout:ceph noarch packages 33 kB/s | 9.1 kB 00:00 2024-03-30T04:12:06.924 INFO:teuthology.orchestra.run.smithi123.stdout:ceph noarch packages 27 kB/s | 9.1 kB 00:00 2024-03-30T04:12:07.041 INFO:teuthology.orchestra.run.smithi135.stdout:ceph source packages 4.4 kB/s | 1.5 kB 00:00 2024-03-30T04:12:07.233 INFO:teuthology.orchestra.run.smithi123.stdout:ceph source packages 5.1 kB/s | 1.5 kB 00:00 2024-03-30T04:12:09.042 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 8 - AppStream 14 MB/s | 28 MB 00:01 2024-03-30T04:12:09.200 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 8 - AppStream 14 MB/s | 28 MB 00:01 2024-03-30T04:12:14.860 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 8 - BaseOS 20 MB/s | 10 MB 00:00 2024-03-30T04:12:15.100 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 8 - BaseOS 19 MB/s | 10 MB 00:00 2024-03-30T04:12:16.785 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 8 - Extras 74 kB/s | 18 kB 00:00 2024-03-30T04:12:17.126 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 8 - Extras 49 kB/s | 18 kB 00:00 2024-03-30T04:12:17.353 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 8 - Extras common packages 16 kB/s | 7.5 kB 00:00 2024-03-30T04:12:17.918 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 8 - Extras common packages 11 kB/s | 7.5 kB 00:00 2024-03-30T04:12:21.087 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 8 - PowerTools 1.3 MB/s | 4.9 MB 00:03 2024-03-30T04:12:21.336 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 8 - PowerTools 1.5 MB/s | 4.9 MB 00:03 2024-03-30T04:12:22.280 INFO:teuthology.orchestra.run.smithi135.stdout:Copr repo for python3-asyncssh owned by ceph 17 kB/s | 3.5 kB 00:00 2024-03-30T04:12:22.545 INFO:teuthology.orchestra.run.smithi123.stdout:Copr repo for python3-asyncssh owned by ceph 16 kB/s | 3.5 kB 00:00 2024-03-30T04:12:23.425 INFO:teuthology.orchestra.run.smithi135.stdout:Extra Packages for Enterprise Linux 16 MB/s | 16 MB 00:01 2024-03-30T04:12:27.098 INFO:teuthology.orchestra.run.smithi123.stdout:Extra Packages for Enterprise Linux 3.6 MB/s | 16 MB 00:04 2024-03-30T04:12:27.806 INFO:teuthology.orchestra.run.smithi135.stdout:lab-extras 436 kB/s | 24 kB 00:00 2024-03-30T04:12:31.521 INFO:teuthology.orchestra.run.smithi123.stdout:lab-extras 408 kB/s | 24 kB 00:00 2024-03-30T04:12:31.550 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-30T04:12:31.553 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-30T04:12:31.553 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2024-03-30T04:12:31.553 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-30T04:12:31.553 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-03-30T04:12:31.554 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-radosgw x86_64 2:17.2.7-797.g5c024335.el8 ceph 17 M 2024-03-30T04:12:31.554 INFO:teuthology.orchestra.run.smithi135.stdout:Upgrading: 2024-03-30T04:12:31.554 INFO:teuthology.orchestra.run.smithi135.stdout: librados2 x86_64 2:17.2.7-797.g5c024335.el8 ceph 3.5 M 2024-03-30T04:12:31.554 INFO:teuthology.orchestra.run.smithi135.stdout: librbd1 x86_64 2:17.2.7-797.g5c024335.el8 ceph 3.6 M 2024-03-30T04:12:31.554 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2024-03-30T04:12:31.554 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-base x86_64 2:17.2.7-797.g5c024335.el8 ceph 5.3 M 2024-03-30T04:12:31.554 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-common x86_64 2:17.2.7-797.g5c024335.el8 ceph 24 M 2024-03-30T04:12:31.554 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-selinux x86_64 2:17.2.7-797.g5c024335.el8 ceph 24 k 2024-03-30T04:12:31.554 INFO:teuthology.orchestra.run.smithi135.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-03-30T04:12:31.554 INFO:teuthology.orchestra.run.smithi135.stdout: libcephfs2 x86_64 2:17.2.7-797.g5c024335.el8 ceph 798 k 2024-03-30T04:12:31.554 INFO:teuthology.orchestra.run.smithi135.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-03-30T04:12:31.554 INFO:teuthology.orchestra.run.smithi135.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-03-30T04:12:31.555 INFO:teuthology.orchestra.run.smithi135.stdout: libradosstriper1 x86_64 2:17.2.7-797.g5c024335.el8 ceph 505 k 2024-03-30T04:12:31.555 INFO:teuthology.orchestra.run.smithi135.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-03-30T04:12:31.555 INFO:teuthology.orchestra.run.smithi135.stdout: librgw2 x86_64 2:17.2.7-797.g5c024335.el8 ceph 5.6 M 2024-03-30T04:12:31.555 INFO:teuthology.orchestra.run.smithi135.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-03-30T04:12:31.555 INFO:teuthology.orchestra.run.smithi135.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-03-30T04:12:31.555 INFO:teuthology.orchestra.run.smithi135.stdout: python3-ceph-argparse x86_64 2:17.2.7-797.g5c024335.el8 ceph 45 k 2024-03-30T04:12:31.555 INFO:teuthology.orchestra.run.smithi135.stdout: python3-ceph-common x86_64 2:17.2.7-797.g5c024335.el8 ceph 117 k 2024-03-30T04:12:31.555 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cephfs x86_64 2:17.2.7-797.g5c024335.el8 ceph 190 k 2024-03-30T04:12:31.555 INFO:teuthology.orchestra.run.smithi135.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-03-30T04:12:31.556 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rados x86_64 2:17.2.7-797.g5c024335.el8 ceph 370 k 2024-03-30T04:12:31.556 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rbd x86_64 2:17.2.7-797.g5c024335.el8 ceph 359 k 2024-03-30T04:12:31.556 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rgw x86_64 2:17.2.7-797.g5c024335.el8 ceph 107 k 2024-03-30T04:12:31.556 INFO:teuthology.orchestra.run.smithi135.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-03-30T04:12:31.556 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:12:31.556 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-03-30T04:12:31.556 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-30T04:12:31.556 INFO:teuthology.orchestra.run.smithi135.stdout:Install 21 Packages 2024-03-30T04:12:31.556 INFO:teuthology.orchestra.run.smithi135.stdout:Upgrade 2 Packages 2024-03-30T04:12:31.556 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:12:31.557 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 64 M 2024-03-30T04:12:31.557 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-03-30T04:12:32.575 INFO:teuthology.orchestra.run.smithi135.stdout:(1/23): ceph-base-17.2.7-797.g5c024335.el8.x86_ 9.2 MB/s | 5.3 MB 00:00 2024-03-30T04:12:32.633 INFO:teuthology.orchestra.run.smithi135.stdout:(2/23): ceph-selinux-17.2.7-797.g5c024335.el8.x 423 kB/s | 24 kB 00:00 2024-03-30T04:12:32.750 INFO:teuthology.orchestra.run.smithi135.stdout:(3/23): libcephfs2-17.2.7-797.g5c024335.el8.x86 6.7 MB/s | 798 kB 00:00 2024-03-30T04:12:32.825 INFO:teuthology.orchestra.run.smithi135.stdout:(4/23): libradosstriper1-17.2.7-797.g5c024335.e 6.6 MB/s | 505 kB 00:00 2024-03-30T04:12:33.408 INFO:teuthology.orchestra.run.smithi135.stdout:(5/23): librgw2-17.2.7-797.g5c024335.el8.x86_64 9.6 MB/s | 5.6 MB 00:00 2024-03-30T04:12:33.618 INFO:teuthology.orchestra.run.smithi135.stdout:(6/23): ceph-radosgw-17.2.7-797.g5c024335.el8.x 10 MB/s | 17 MB 00:01 2024-03-30T04:12:33.642 INFO:teuthology.orchestra.run.smithi135.stdout:(7/23): python3-ceph-argparse-17.2.7-797.g5c024 191 kB/s | 45 kB 00:00 2024-03-30T04:12:33.668 INFO:teuthology.orchestra.run.smithi135.stdout:(8/23): python3-ceph-common-17.2.7-797.g5c02433 2.3 MB/s | 117 kB 00:00 2024-03-30T04:12:33.718 INFO:teuthology.orchestra.run.smithi135.stdout:(9/23): python3-cephfs-17.2.7-797.g5c024335.el8 2.5 MB/s | 190 kB 00:00 2024-03-30T04:12:33.752 INFO:teuthology.orchestra.run.smithi135.stdout:(10/23): python3-rados-17.2.7-797.g5c024335.el8 4.3 MB/s | 370 kB 00:00 2024-03-30T04:12:33.785 INFO:teuthology.orchestra.run.smithi135.stdout:(11/23): python3-rbd-17.2.7-797.g5c024335.el8.x 5.3 MB/s | 359 kB 00:00 2024-03-30T04:12:33.909 INFO:teuthology.orchestra.run.smithi135.stdout:(12/23): python3-rgw-17.2.7-797.g5c024335.el8.x 680 kB/s | 107 kB 00:00 2024-03-30T04:12:34.629 INFO:teuthology.orchestra.run.smithi135.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 361 kB/s | 259 kB 00:00 2024-03-30T04:12:34.678 INFO:teuthology.orchestra.run.smithi135.stdout:(14/23): librdkafka-1.6.1-1.el8.x86_64.rpm 758 kB/s | 677 kB 00:00 2024-03-30T04:12:34.729 INFO:teuthology.orchestra.run.smithi135.stdout:(15/23): python3-prettytable-0.7.2-14.el8.noarc 442 kB/s | 44 kB 00:00 2024-03-30T04:12:34.754 INFO:teuthology.orchestra.run.smithi135.stdout:(16/23): librabbitmq-0.9.0-5.el8.x86_64.rpm 627 kB/s | 47 kB 00:00 2024-03-30T04:12:34.804 INFO:teuthology.orchestra.run.smithi135.stdout:(17/23): liboath-2.6.2-3.el8.x86_64.rpm 1.2 MB/s | 59 kB 00:00 2024-03-30T04:12:34.846 INFO:teuthology.orchestra.run.smithi135.stdout:(18/23): libunwind-1.3.1-3.el8.x86_64.rpm 1.8 MB/s | 75 kB 00:00 2024-03-30T04:12:35.089 INFO:teuthology.orchestra.run.smithi135.stdout:(19/23): ceph-common-17.2.7-797.g5c024335.el8.x 7.6 MB/s | 24 MB 00:03 2024-03-30T04:12:35.114 INFO:teuthology.orchestra.run.smithi135.stdout:(20/23): gperftools-libs-2.7-9.el8.x86_64.rpm 797 kB/s | 306 kB 00:00 2024-03-30T04:12:35.197 INFO:teuthology.orchestra.run.smithi135.stdout:(21/23): thrift-0.13.0-2.el8.x86_64.rpm 5.0 MB/s | 1.7 MB 00:00 2024-03-30T04:12:35.249 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-30T04:12:35.252 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-30T04:12:35.252 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2024-03-30T04:12:35.253 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-30T04:12:35.253 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-30T04:12:35.253 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-radosgw x86_64 2:17.2.7-797.g5c024335.el8 ceph 17 M 2024-03-30T04:12:35.253 INFO:teuthology.orchestra.run.smithi123.stdout:Upgrading: 2024-03-30T04:12:35.253 INFO:teuthology.orchestra.run.smithi123.stdout: librados2 x86_64 2:17.2.7-797.g5c024335.el8 ceph 3.5 M 2024-03-30T04:12:35.253 INFO:teuthology.orchestra.run.smithi123.stdout: librbd1 x86_64 2:17.2.7-797.g5c024335.el8 ceph 3.6 M 2024-03-30T04:12:35.253 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2024-03-30T04:12:35.253 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-base x86_64 2:17.2.7-797.g5c024335.el8 ceph 5.3 M 2024-03-30T04:12:35.253 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-common x86_64 2:17.2.7-797.g5c024335.el8 ceph 24 M 2024-03-30T04:12:35.253 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-selinux x86_64 2:17.2.7-797.g5c024335.el8 ceph 24 k 2024-03-30T04:12:35.253 INFO:teuthology.orchestra.run.smithi123.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-03-30T04:12:35.253 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs2 x86_64 2:17.2.7-797.g5c024335.el8 ceph 798 k 2024-03-30T04:12:35.253 INFO:teuthology.orchestra.run.smithi123.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-03-30T04:12:35.254 INFO:teuthology.orchestra.run.smithi123.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-03-30T04:12:35.254 INFO:teuthology.orchestra.run.smithi123.stdout: libradosstriper1 x86_64 2:17.2.7-797.g5c024335.el8 ceph 505 k 2024-03-30T04:12:35.254 INFO:teuthology.orchestra.run.smithi123.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-03-30T04:12:35.254 INFO:teuthology.orchestra.run.smithi123.stdout: librgw2 x86_64 2:17.2.7-797.g5c024335.el8 ceph 5.6 M 2024-03-30T04:12:35.254 INFO:teuthology.orchestra.run.smithi123.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-03-30T04:12:35.254 INFO:teuthology.orchestra.run.smithi123.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-03-30T04:12:35.254 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-argparse x86_64 2:17.2.7-797.g5c024335.el8 ceph 45 k 2024-03-30T04:12:35.254 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-common x86_64 2:17.2.7-797.g5c024335.el8 ceph 117 k 2024-03-30T04:12:35.254 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cephfs x86_64 2:17.2.7-797.g5c024335.el8 ceph 190 k 2024-03-30T04:12:35.254 INFO:teuthology.orchestra.run.smithi123.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-03-30T04:12:35.254 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rados x86_64 2:17.2.7-797.g5c024335.el8 ceph 370 k 2024-03-30T04:12:35.254 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rbd x86_64 2:17.2.7-797.g5c024335.el8 ceph 359 k 2024-03-30T04:12:35.254 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rgw x86_64 2:17.2.7-797.g5c024335.el8 ceph 107 k 2024-03-30T04:12:35.254 INFO:teuthology.orchestra.run.smithi123.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-03-30T04:12:35.255 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:12:35.255 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-30T04:12:35.255 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-30T04:12:35.255 INFO:teuthology.orchestra.run.smithi123.stdout:Install 21 Packages 2024-03-30T04:12:35.255 INFO:teuthology.orchestra.run.smithi123.stdout:Upgrade 2 Packages 2024-03-30T04:12:35.255 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:12:35.255 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 64 M 2024-03-30T04:12:35.255 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-30T04:12:35.414 INFO:teuthology.orchestra.run.smithi135.stdout:(22/23): librbd1-17.2.7-797.g5c024335.el8.x86_6 12 MB/s | 3.6 MB 00:00 2024-03-30T04:12:35.473 INFO:teuthology.orchestra.run.smithi135.stdout:(23/23): librados2-17.2.7-797.g5c024335.el8.x86 9.0 MB/s | 3.5 MB 00:00 2024-03-30T04:12:35.475 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-03-30T04:12:35.475 INFO:teuthology.orchestra.run.smithi135.stdout:Total 16 MB/s | 64 MB 00:03 2024-03-30T04:12:35.561 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-03-30T04:12:35.611 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-03-30T04:12:35.611 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-03-30T04:12:36.017 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-03-30T04:12:36.021 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-03-30T04:12:36.306 INFO:teuthology.orchestra.run.smithi123.stdout:(1/23): ceph-base-17.2.7-797.g5c024335.el8.x86_ 11 MB/s | 5.3 MB 00:00 2024-03-30T04:12:36.365 INFO:teuthology.orchestra.run.smithi123.stdout:(2/23): ceph-selinux-17.2.7-797.g5c024335.el8.x 418 kB/s | 24 kB 00:00 2024-03-30T04:12:36.448 INFO:teuthology.orchestra.run.smithi123.stdout:(3/23): libcephfs2-17.2.7-797.g5c024335.el8.x86 9.4 MB/s | 798 kB 00:00 2024-03-30T04:12:36.515 INFO:teuthology.orchestra.run.smithi123.stdout:(4/23): libradosstriper1-17.2.7-797.g5c024335.e 7.4 MB/s | 505 kB 00:00 2024-03-30T04:12:36.857 INFO:teuthology.orchestra.run.smithi123.stdout:(5/23): librgw2-17.2.7-797.g5c024335.el8.x86_64 16 MB/s | 5.6 MB 00:00 2024-03-30T04:12:36.907 INFO:teuthology.orchestra.run.smithi123.stdout:(6/23): python3-ceph-argparse-17.2.7-797.g5c024 897 kB/s | 45 kB 00:00 2024-03-30T04:12:36.957 INFO:teuthology.orchestra.run.smithi123.stdout:(7/23): python3-ceph-common-17.2.7-797.g5c02433 2.3 MB/s | 117 kB 00:00 2024-03-30T04:12:37.024 INFO:teuthology.orchestra.run.smithi123.stdout:(8/23): python3-cephfs-17.2.7-797.g5c024335.el8 2.8 MB/s | 190 kB 00:00 2024-03-30T04:12:37.062 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-03-30T04:12:37.083 INFO:teuthology.orchestra.run.smithi123.stdout:(9/23): python3-rados-17.2.7-797.g5c024335.el8. 6.2 MB/s | 370 kB 00:00 2024-03-30T04:12:37.175 INFO:teuthology.orchestra.run.smithi123.stdout:(10/23): python3-rbd-17.2.7-797.g5c024335.el8.x 3.8 MB/s | 359 kB 00:00 2024-03-30T04:12:37.225 INFO:teuthology.orchestra.run.smithi123.stdout:(11/23): python3-rgw-17.2.7-797.g5c024335.el8.x 2.1 MB/s | 107 kB 00:00 2024-03-30T04:12:37.293 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2024-03-30T04:12:37.553 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-03-30T04:12:37.767 INFO:teuthology.orchestra.run.smithi123.stdout:(12/23): librdkafka-1.6.1-1.el8.x86_64.rpm 1.2 MB/s | 677 kB 00:00 2024-03-30T04:12:37.943 INFO:teuthology.orchestra.run.smithi123.stdout:(13/23): ceph-radosgw-17.2.7-797.g5c024335.el8. 8.0 MB/s | 17 MB 00:02 2024-03-30T04:12:37.967 INFO:teuthology.orchestra.run.smithi123.stdout:(14/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.3 MB/s | 259 kB 00:00 2024-03-30T04:12:37.981 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-03-30T04:12:38.035 INFO:teuthology.orchestra.run.smithi123.stdout:(15/23): librabbitmq-0.9.0-5.el8.x86_64.rpm 707 kB/s | 47 kB 00:00 2024-03-30T04:12:38.203 INFO:teuthology.orchestra.run.smithi135.stdout: Upgrading : librados2-2:17.2.7-797.g5c024335.el8.x86_64 3/25 2024-03-30T04:12:38.276 INFO:teuthology.orchestra.run.smithi123.stdout:(16/23): python3-prettytable-0.7.2-14.el8.noarc 132 kB/s | 44 kB 00:00 2024-03-30T04:12:38.317 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librados2-2:17.2.7-797.g5c024335.el8.x86_64 3/25 2024-03-30T04:12:38.385 INFO:teuthology.orchestra.run.smithi123.stdout:(17/23): gperftools-libs-2.7-9.el8.x86_64.rpm 875 kB/s | 306 kB 00:00 2024-03-30T04:12:38.435 INFO:teuthology.orchestra.run.smithi123.stdout:(18/23): liboath-2.6.2-3.el8.x86_64.rpm 371 kB/s | 59 kB 00:00 2024-03-30T04:12:38.592 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-rados-2:17.2.7-797.g5c024335.el8.x86_64 4/25 2024-03-30T04:12:38.777 INFO:teuthology.orchestra.run.smithi123.stdout:(19/23): ceph-common-17.2.7-797.g5c024335.el8.x 8.0 MB/s | 24 MB 00:02 2024-03-30T04:12:38.786 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libcephfs2-2:17.2.7-797.g5c024335.el8.x86_64 5/25 2024-03-30T04:12:38.802 INFO:teuthology.orchestra.run.smithi123.stdout:(20/23): libunwind-1.3.1-3.el8.x86_64.rpm 180 kB/s | 75 kB 00:00 2024-03-30T04:12:39.120 INFO:teuthology.orchestra.run.smithi123.stdout:(21/23): librados2-17.2.7-797.g5c024335.el8.x86 10 MB/s | 3.5 MB 00:00 2024-03-30T04:12:39.194 INFO:teuthology.orchestra.run.smithi123.stdout:(22/23): librbd1-17.2.7-797.g5c024335.el8.x86_6 9.2 MB/s | 3.6 MB 00:00 2024-03-30T04:12:39.228 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: libcephfs2-2:17.2.7-797.g5c024335.el8.x86_64 5/25 2024-03-30T04:12:39.336 INFO:teuthology.orchestra.run.smithi123.stdout:(23/23): thrift-0.13.0-2.el8.x86_64.rpm 1.9 MB/s | 1.7 MB 00:00 2024-03-30T04:12:39.339 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-30T04:12:39.339 INFO:teuthology.orchestra.run.smithi123.stdout:Total 16 MB/s | 64 MB 00:04 2024-03-30T04:12:39.425 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-30T04:12:39.428 INFO:teuthology.orchestra.run.smithi135.stdout: Upgrading : librbd1-2:17.2.7-797.g5c024335.el8.x86_64 6/25 2024-03-30T04:12:39.474 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-30T04:12:39.474 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-30T04:12:39.702 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librbd1-2:17.2.7-797.g5c024335.el8.x86_64 6/25 2024-03-30T04:12:39.870 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-30T04:12:39.873 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-30T04:12:39.884 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/25 2024-03-30T04:12:40.108 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 8/25 2024-03-30T04:12:40.804 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 9/25 2024-03-30T04:12:40.878 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-30T04:12:40.978 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : librgw2-2:17.2.7-797.g5c024335.el8.x86_64 10/25 2024-03-30T04:12:41.044 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librgw2-2:17.2.7-797.g5c024335.el8.x86_64 10/25 2024-03-30T04:12:41.175 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2024-03-30T04:12:41.198 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-ceph-argparse-2:17.2.7-797.g5c024335.el8.x 11/25 2024-03-30T04:12:41.406 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-03-30T04:12:41.421 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-cephfs-2:17.2.7-797.g5c024335.el8.x86_64 12/25 2024-03-30T04:12:41.630 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-rgw-2:17.2.7-797.g5c024335.el8.x86_64 13/25 2024-03-30T04:12:41.832 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-03-30T04:12:41.858 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-rbd-2:17.2.7-797.g5c024335.el8.x86_64 14/25 2024-03-30T04:12:42.003 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libradosstriper1-2:17.2.7-797.g5c024335.el8.x86_64 15/25 2024-03-30T04:12:42.031 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : librados2-2:17.2.7-797.g5c024335.el8.x86_64 3/25 2024-03-30T04:12:42.092 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: libradosstriper1-2:17.2.7-797.g5c024335.el8.x86_64 15/25 2024-03-30T04:12:42.148 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librados2-2:17.2.7-797.g5c024335.el8.x86_64 3/25 2024-03-30T04:12:42.335 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2024-03-30T04:12:42.385 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rados-2:17.2.7-797.g5c024335.el8.x86_64 4/25 2024-03-30T04:12:42.538 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2024-03-30T04:12:42.581 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libcephfs2-2:17.2.7-797.g5c024335.el8.x86_64 5/25 2024-03-30T04:12:42.756 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-03-30T04:12:42.945 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-ceph-common-2:17.2.7-797.g5c024335.el8.x86 19/25 2024-03-30T04:12:43.009 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libcephfs2-2:17.2.7-797.g5c024335.el8.x86_64 5/25 2024-03-30T04:12:43.206 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : librbd1-2:17.2.7-797.g5c024335.el8.x86_64 6/25 2024-03-30T04:12:43.445 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librbd1-2:17.2.7-797.g5c024335.el8.x86_64 6/25 2024-03-30T04:12:43.628 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/25 2024-03-30T04:12:43.845 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 8/25 2024-03-30T04:12:44.532 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 9/25 2024-03-30T04:12:44.689 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librgw2-2:17.2.7-797.g5c024335.el8.x86_64 10/25 2024-03-30T04:12:44.773 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librgw2-2:17.2.7-797.g5c024335.el8.x86_64 10/25 2024-03-30T04:12:44.902 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-ceph-argparse-2:17.2.7-797.g5c024335.el8.x 11/25 2024-03-30T04:12:45.088 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cephfs-2:17.2.7-797.g5c024335.el8.x86_64 12/25 2024-03-30T04:12:45.311 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rgw-2:17.2.7-797.g5c024335.el8.x86_64 13/25 2024-03-30T04:12:45.528 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rbd-2:17.2.7-797.g5c024335.el8.x86_64 14/25 2024-03-30T04:12:45.672 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libradosstriper1-2:17.2.7-797.g5c024335.el8.x86_64 15/25 2024-03-30T04:12:45.754 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libradosstriper1-2:17.2.7-797.g5c024335.el8.x86_64 15/25 2024-03-30T04:12:45.938 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-common-2:17.2.7-797.g5c024335.el8.x86_64 20/25 2024-03-30T04:12:45.969 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2024-03-30T04:12:46.145 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-common-2:17.2.7-797.g5c024335.el8.x86_64 20/25 2024-03-30T04:12:46.157 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2024-03-30T04:12:46.592 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-03-30T04:12:46.798 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-ceph-common-2:17.2.7-797.g5c024335.el8.x86 19/25 2024-03-30T04:12:46.803 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-common-2:17.2.7-797.g5c024335.el8.x86_64 20/25 2024-03-30T04:12:47.080 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-base-2:17.2.7-797.g5c024335.el8.x86_64 21/25 2024-03-30T04:12:47.290 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-base-2:17.2.7-797.g5c024335.el8.x86_64 21/25 2024-03-30T04:12:47.469 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-selinux-2:17.2.7-797.g5c024335.el8.x86_64 22/25 2024-03-30T04:12:49.681 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-common-2:17.2.7-797.g5c024335.el8.x86_64 20/25 2024-03-30T04:12:49.914 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-common-2:17.2.7-797.g5c024335.el8.x86_64 20/25 2024-03-30T04:12:50.573 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-common-2:17.2.7-797.g5c024335.el8.x86_64 20/25 2024-03-30T04:12:50.844 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-base-2:17.2.7-797.g5c024335.el8.x86_64 21/25 2024-03-30T04:12:51.037 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-base-2:17.2.7-797.g5c024335.el8.x86_64 21/25 2024-03-30T04:12:51.271 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-selinux-2:17.2.7-797.g5c024335.el8.x86_64 22/25 2024-03-30T04:13:08.900 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-selinux-2:17.2.7-797.g5c024335.el8.x86_64 22/25 2024-03-30T04:13:08.900 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /sys 2024-03-30T04:13:08.900 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /proc 2024-03-30T04:13:08.900 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /dev 2024-03-30T04:13:08.900 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /run 2024-03-30T04:13:08.900 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /mnt 2024-03-30T04:13:08.900 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /var/tmp 2024-03-30T04:13:08.900 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /home 2024-03-30T04:13:08.901 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /tmp 2024-03-30T04:13:08.901 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /dev 2024-03-30T04:13:08.901 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:13:10.710 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-radosgw-2:17.2.7-797.g5c024335.el8.x86_64 23/25 2024-03-30T04:13:10.815 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-797.g5c024335.el8.x86_64 23/25 2024-03-30T04:13:10.815 INFO:teuthology.orchestra.run.smithi135.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-03-30T04:13:11.037 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-03-30T04:13:11.037 INFO:teuthology.orchestra.run.smithi135.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-03-30T04:13:12.005 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-selinux-2:17.2.7-797.g5c024335.el8.x86_64 22/25 2024-03-30T04:13:12.005 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /sys 2024-03-30T04:13:12.005 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /proc 2024-03-30T04:13:12.005 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /dev 2024-03-30T04:13:12.005 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /run 2024-03-30T04:13:12.005 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /mnt 2024-03-30T04:13:12.005 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /var/tmp 2024-03-30T04:13:12.005 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /home 2024-03-30T04:13:12.005 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /tmp 2024-03-30T04:13:12.006 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /dev 2024-03-30T04:13:12.006 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:13:12.072 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-03-30T04:13:12.072 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-base-2:17.2.7-797.g5c024335.el8.x86_64 1/25 2024-03-30T04:13:12.072 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-common-2:17.2.7-797.g5c024335.el8.x86_64 2/25 2024-03-30T04:13:12.072 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-radosgw-2:17.2.7-797.g5c024335.el8.x86_64 3/25 2024-03-30T04:13:12.072 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-selinux-2:17.2.7-797.g5c024335.el8.x86_64 4/25 2024-03-30T04:13:12.072 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libcephfs2-2:17.2.7-797.g5c024335.el8.x86_64 5/25 2024-03-30T04:13:12.072 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libradosstriper1-2:17.2.7-797.g5c024335.el8.x86_64 6/25 2024-03-30T04:13:12.072 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librgw2-2:17.2.7-797.g5c024335.el8.x86_64 7/25 2024-03-30T04:13:12.072 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-ceph-argparse-2:17.2.7-797.g5c024335.el8.x 8/25 2024-03-30T04:13:12.072 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-ceph-common-2:17.2.7-797.g5c024335.el8.x86 9/25 2024-03-30T04:13:12.073 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-cephfs-2:17.2.7-797.g5c024335.el8.x86_64 10/25 2024-03-30T04:13:12.073 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-rados-2:17.2.7-797.g5c024335.el8.x86_64 11/25 2024-03-30T04:13:12.073 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-rbd-2:17.2.7-797.g5c024335.el8.x86_64 12/25 2024-03-30T04:13:12.073 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-rgw-2:17.2.7-797.g5c024335.el8.x86_64 13/25 2024-03-30T04:13:12.073 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2024-03-30T04:13:12.073 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2024-03-30T04:13:12.073 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-03-30T04:13:12.073 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 17/25 2024-03-30T04:13:12.073 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-03-30T04:13:12.073 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2024-03-30T04:13:12.073 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2024-03-30T04:13:12.073 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2024-03-30T04:13:12.073 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librados2-2:17.2.7-797.g5c024335.el8.x86_64 22/25 2024-03-30T04:13:12.074 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-03-30T04:13:12.074 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librbd1-2:17.2.7-797.g5c024335.el8.x86_64 24/25 2024-03-30T04:13:12.786 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-03-30T04:13:12.786 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:13:12.786 INFO:teuthology.orchestra.run.smithi135.stdout:Upgraded: 2024-03-30T04:13:12.786 INFO:teuthology.orchestra.run.smithi135.stdout: librados2-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:13:12.786 INFO:teuthology.orchestra.run.smithi135.stdout: librbd1-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:13:12.786 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-03-30T04:13:12.787 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-base-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:13:12.787 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-common-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:13:12.787 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-radosgw-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:13:12.787 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-selinux-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:13:12.787 INFO:teuthology.orchestra.run.smithi135.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-03-30T04:13:12.787 INFO:teuthology.orchestra.run.smithi135.stdout: libcephfs2-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:13:12.787 INFO:teuthology.orchestra.run.smithi135.stdout: liboath-2.6.2-3.el8.x86_64 2024-03-30T04:13:12.787 INFO:teuthology.orchestra.run.smithi135.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-03-30T04:13:12.787 INFO:teuthology.orchestra.run.smithi135.stdout: libradosstriper1-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:13:12.787 INFO:teuthology.orchestra.run.smithi135.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-03-30T04:13:12.787 INFO:teuthology.orchestra.run.smithi135.stdout: librgw2-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:13:12.787 INFO:teuthology.orchestra.run.smithi135.stdout: libunwind-1.3.1-3.el8.x86_64 2024-03-30T04:13:12.787 INFO:teuthology.orchestra.run.smithi135.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-03-30T04:13:12.787 INFO:teuthology.orchestra.run.smithi135.stdout: python3-ceph-argparse-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:13:12.787 INFO:teuthology.orchestra.run.smithi135.stdout: python3-ceph-common-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:13:12.788 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cephfs-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:13:12.788 INFO:teuthology.orchestra.run.smithi135.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-03-30T04:13:12.788 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rados-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:13:12.788 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rbd-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:13:12.788 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rgw-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:13:12.788 INFO:teuthology.orchestra.run.smithi135.stdout: thrift-0.13.0-2.el8.x86_64 2024-03-30T04:13:12.788 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:13:12.788 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-30T04:13:13.113 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-test 2024-03-30T04:13:13.807 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-radosgw-2:17.2.7-797.g5c024335.el8.x86_64 23/25 2024-03-30T04:13:13.918 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-797.g5c024335.el8.x86_64 23/25 2024-03-30T04:13:13.918 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-03-30T04:13:14.047 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:00:47 ago on Sat 30 Mar 2024 04:12:27 AM UTC. 2024-03-30T04:13:14.165 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-03-30T04:13:14.165 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-03-30T04:13:15.191 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-03-30T04:13:15.191 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-base-2:17.2.7-797.g5c024335.el8.x86_64 1/25 2024-03-30T04:13:15.191 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-common-2:17.2.7-797.g5c024335.el8.x86_64 2/25 2024-03-30T04:13:15.191 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-radosgw-2:17.2.7-797.g5c024335.el8.x86_64 3/25 2024-03-30T04:13:15.191 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-selinux-2:17.2.7-797.g5c024335.el8.x86_64 4/25 2024-03-30T04:13:15.191 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libcephfs2-2:17.2.7-797.g5c024335.el8.x86_64 5/25 2024-03-30T04:13:15.191 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libradosstriper1-2:17.2.7-797.g5c024335.el8.x86_64 6/25 2024-03-30T04:13:15.192 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librgw2-2:17.2.7-797.g5c024335.el8.x86_64 7/25 2024-03-30T04:13:15.192 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-ceph-argparse-2:17.2.7-797.g5c024335.el8.x 8/25 2024-03-30T04:13:15.192 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-ceph-common-2:17.2.7-797.g5c024335.el8.x86 9/25 2024-03-30T04:13:15.192 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cephfs-2:17.2.7-797.g5c024335.el8.x86_64 10/25 2024-03-30T04:13:15.192 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rados-2:17.2.7-797.g5c024335.el8.x86_64 11/25 2024-03-30T04:13:15.192 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rbd-2:17.2.7-797.g5c024335.el8.x86_64 12/25 2024-03-30T04:13:15.192 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rgw-2:17.2.7-797.g5c024335.el8.x86_64 13/25 2024-03-30T04:13:15.192 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2024-03-30T04:13:15.192 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2024-03-30T04:13:15.192 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-03-30T04:13:15.192 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 17/25 2024-03-30T04:13:15.192 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-03-30T04:13:15.192 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2024-03-30T04:13:15.192 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2024-03-30T04:13:15.192 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2024-03-30T04:13:15.192 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librados2-2:17.2.7-797.g5c024335.el8.x86_64 22/25 2024-03-30T04:13:15.193 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-03-30T04:13:15.193 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librbd1-2:17.2.7-797.g5c024335.el8.x86_64 24/25 2024-03-30T04:13:15.711 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-30T04:13:15.712 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-30T04:13:15.712 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2024-03-30T04:13:15.712 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-30T04:13:15.712 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-03-30T04:13:15.712 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-test x86_64 2:17.2.7-797.g5c024335.el8 ceph 53 M 2024-03-30T04:13:15.712 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2024-03-30T04:13:15.712 INFO:teuthology.orchestra.run.smithi135.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-03-30T04:13:15.713 INFO:teuthology.orchestra.run.smithi135.stdout: libcephsqlite x86_64 2:17.2.7-797.g5c024335.el8 ceph 179 k 2024-03-30T04:13:15.713 INFO:teuthology.orchestra.run.smithi135.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-03-30T04:13:15.713 INFO:teuthology.orchestra.run.smithi135.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-03-30T04:13:15.713 INFO:teuthology.orchestra.run.smithi135.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-03-30T04:13:15.713 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:13:15.713 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-03-30T04:13:15.713 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-30T04:13:15.713 INFO:teuthology.orchestra.run.smithi135.stdout:Install 6 Packages 2024-03-30T04:13:15.713 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:13:15.714 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 54 M 2024-03-30T04:13:15.715 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 236 M 2024-03-30T04:13:15.715 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-03-30T04:13:15.965 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-03-30T04:13:15.965 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:13:15.965 INFO:teuthology.orchestra.run.smithi123.stdout:Upgraded: 2024-03-30T04:13:15.965 INFO:teuthology.orchestra.run.smithi123.stdout: librados2-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:13:15.965 INFO:teuthology.orchestra.run.smithi123.stdout: librbd1-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:13:15.965 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-30T04:13:15.965 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-base-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:13:15.965 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-common-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:13:15.965 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-radosgw-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:13:15.965 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-selinux-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:13:15.965 INFO:teuthology.orchestra.run.smithi123.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-03-30T04:13:15.965 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs2-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:13:15.966 INFO:teuthology.orchestra.run.smithi123.stdout: liboath-2.6.2-3.el8.x86_64 2024-03-30T04:13:15.966 INFO:teuthology.orchestra.run.smithi123.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-03-30T04:13:15.966 INFO:teuthology.orchestra.run.smithi123.stdout: libradosstriper1-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:13:15.966 INFO:teuthology.orchestra.run.smithi123.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-03-30T04:13:15.966 INFO:teuthology.orchestra.run.smithi123.stdout: librgw2-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:13:15.966 INFO:teuthology.orchestra.run.smithi123.stdout: libunwind-1.3.1-3.el8.x86_64 2024-03-30T04:13:15.966 INFO:teuthology.orchestra.run.smithi123.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-03-30T04:13:15.966 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-argparse-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:13:15.966 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-common-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:13:15.966 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cephfs-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:13:15.966 INFO:teuthology.orchestra.run.smithi123.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-03-30T04:13:15.966 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rados-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:13:15.966 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rbd-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:13:15.966 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rgw-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:13:15.967 INFO:teuthology.orchestra.run.smithi123.stdout: thrift-0.13.0-2.el8.x86_64 2024-03-30T04:13:15.967 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:13:15.967 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-30T04:13:16.285 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-test 2024-03-30T04:13:16.416 INFO:teuthology.orchestra.run.smithi135.stdout:(1/6): jq-1.6-8.el8.x86_64.rpm 404 kB/s | 203 kB 00:00 2024-03-30T04:13:16.517 INFO:teuthology.orchestra.run.smithi135.stdout:(2/6): libcephsqlite-17.2.7-797.g5c024335.el8.x 297 kB/s | 179 kB 00:00 2024-03-30T04:13:16.542 INFO:teuthology.orchestra.run.smithi135.stdout:(3/6): oniguruma-6.8.2-3.el8.x86_64.rpm 1.5 MB/s | 188 kB 00:00 2024-03-30T04:13:16.642 INFO:teuthology.orchestra.run.smithi135.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 707 kB/s | 71 kB 00:00 2024-03-30T04:13:17.043 INFO:teuthology.orchestra.run.smithi135.stdout:(5/6): socat-1.7.4.1-1.el8.x86_64.rpm 615 kB/s | 323 kB 00:00 2024-03-30T04:13:17.207 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:00:46 ago on Sat 30 Mar 2024 04:12:31 AM UTC. 2024-03-30T04:13:18.884 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-30T04:13:18.885 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-30T04:13:18.885 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2024-03-30T04:13:18.885 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-30T04:13:18.886 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-30T04:13:18.886 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-test x86_64 2:17.2.7-797.g5c024335.el8 ceph 53 M 2024-03-30T04:13:18.886 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2024-03-30T04:13:18.886 INFO:teuthology.orchestra.run.smithi123.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-03-30T04:13:18.886 INFO:teuthology.orchestra.run.smithi123.stdout: libcephsqlite x86_64 2:17.2.7-797.g5c024335.el8 ceph 179 k 2024-03-30T04:13:18.886 INFO:teuthology.orchestra.run.smithi123.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-03-30T04:13:18.886 INFO:teuthology.orchestra.run.smithi123.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-03-30T04:13:18.886 INFO:teuthology.orchestra.run.smithi123.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-03-30T04:13:18.886 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:13:18.886 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-30T04:13:18.886 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-30T04:13:18.886 INFO:teuthology.orchestra.run.smithi123.stdout:Install 6 Packages 2024-03-30T04:13:18.887 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:13:18.887 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 54 M 2024-03-30T04:13:18.887 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 236 M 2024-03-30T04:13:18.887 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-30T04:13:19.159 INFO:teuthology.orchestra.run.smithi123.stdout:(1/6): libcephsqlite-17.2.7-797.g5c024335.el8.x 851 kB/s | 179 kB 00:00 2024-03-30T04:13:19.410 INFO:teuthology.orchestra.run.smithi123.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 441 kB/s | 203 kB 00:00 2024-03-30T04:13:19.585 INFO:teuthology.orchestra.run.smithi123.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 1.8 MB/s | 323 kB 00:00 2024-03-30T04:13:19.610 INFO:teuthology.orchestra.run.smithi123.stdout:(4/6): oniguruma-6.8.2-3.el8.x86_64.rpm 418 kB/s | 188 kB 00:00 2024-03-30T04:13:19.694 INFO:teuthology.orchestra.run.smithi123.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 652 kB/s | 71 kB 00:00 2024-03-30T04:13:21.685 INFO:teuthology.orchestra.run.smithi123.stdout:(6/6): ceph-test-17.2.7-797.g5c024335.el8.x86_6 19 MB/s | 53 MB 00:02 2024-03-30T04:13:21.688 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-30T04:13:21.688 INFO:teuthology.orchestra.run.smithi123.stdout:Total 19 MB/s | 54 MB 00:02 2024-03-30T04:13:21.775 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-30T04:13:21.799 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-30T04:13:21.799 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-30T04:13:22.120 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-30T04:13:22.123 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-30T04:13:22.636 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-30T04:13:22.846 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-03-30T04:13:23.038 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-03-30T04:13:23.183 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-03-30T04:13:23.284 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-03-30T04:13:23.502 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-03-30T04:13:23.691 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libcephsqlite-2:17.2.7-797.g5c024335.el8.x86_64 5/6 2024-03-30T04:13:27.459 INFO:teuthology.orchestra.run.smithi135.stdout:(6/6): ceph-test-17.2.7-797.g5c024335.el8.x86_6 4.6 MB/s | 53 MB 00:11 2024-03-30T04:13:27.460 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-03-30T04:13:27.461 INFO:teuthology.orchestra.run.smithi135.stdout:Total 4.6 MB/s | 54 MB 00:11 2024-03-30T04:13:27.550 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-03-30T04:13:27.574 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-03-30T04:13:27.574 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-03-30T04:13:27.894 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-03-30T04:13:27.898 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-03-30T04:13:28.420 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-03-30T04:13:28.704 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-03-30T04:13:28.884 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-03-30T04:13:29.046 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-03-30T04:13:29.062 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libcephsqlite-2:17.2.7-797.g5c024335.el8.x86_64 5/6 2024-03-30T04:13:29.141 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-03-30T04:13:29.330 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-test-2:17.2.7-797.g5c024335.el8.x86_64 6/6 2024-03-30T04:13:29.335 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-03-30T04:13:29.512 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libcephsqlite-2:17.2.7-797.g5c024335.el8.x86_64 5/6 2024-03-30T04:13:29.704 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-test-2:17.2.7-797.g5c024335.el8.x86_64 6/6 2024-03-30T04:13:29.704 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-test-2:17.2.7-797.g5c024335.el8.x86_64 1/6 2024-03-30T04:13:29.704 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libcephsqlite-2:17.2.7-797.g5c024335.el8.x86_64 2/6 2024-03-30T04:13:29.705 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-03-30T04:13:29.705 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-03-30T04:13:29.705 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-03-30T04:13:30.150 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-03-30T04:13:30.150 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:13:30.151 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-30T04:13:30.151 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-test-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:13:30.151 INFO:teuthology.orchestra.run.smithi123.stdout: jq-1.6-8.el8.x86_64 2024-03-30T04:13:30.151 INFO:teuthology.orchestra.run.smithi123.stdout: libcephsqlite-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:13:30.151 INFO:teuthology.orchestra.run.smithi123.stdout: oniguruma-6.8.2-3.el8.x86_64 2024-03-30T04:13:30.151 INFO:teuthology.orchestra.run.smithi123.stdout: socat-1.7.4.1-1.el8.x86_64 2024-03-30T04:13:30.151 INFO:teuthology.orchestra.run.smithi123.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-03-30T04:13:30.151 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:13:30.151 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-30T04:13:30.337 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph 2024-03-30T04:13:31.298 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:00 ago on Sat 30 Mar 2024 04:12:31 AM UTC. 2024-03-30T04:13:32.997 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-30T04:13:33.001 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-30T04:13:33.001 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2024-03-30T04:13:33.001 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-30T04:13:33.001 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-30T04:13:33.001 INFO:teuthology.orchestra.run.smithi123.stdout: ceph x86_64 2:17.2.7-797.g5c024335.el8 ceph 6.4 k 2024-03-30T04:13:33.001 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2024-03-30T04:13:33.001 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mds x86_64 2:17.2.7-797.g5c024335.el8 ceph 2.3 M 2024-03-30T04:13:33.001 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr x86_64 2:17.2.7-797.g5c024335.el8 ceph 1.6 M 2024-03-30T04:13:33.001 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-modules-core noarch 2:17.2.7-797.g5c024335.el8 ceph-noarch 240 k 2024-03-30T04:13:33.001 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mon x86_64 2:17.2.7-797.g5c024335.el8 ceph 4.0 M 2024-03-30T04:13:33.001 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-osd x86_64 2:17.2.7-797.g5c024335.el8 ceph 17 M 2024-03-30T04:13:33.001 INFO:teuthology.orchestra.run.smithi123.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-03-30T04:13:33.002 INFO:teuthology.orchestra.run.smithi123.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-03-30T04:13:33.002 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-03-30T04:13:33.002 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-03-30T04:13:33.002 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-03-30T04:13:33.002 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-03-30T04:13:33.002 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-03-30T04:13:33.002 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-03-30T04:13:33.002 INFO:teuthology.orchestra.run.smithi123.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-03-30T04:13:33.002 INFO:teuthology.orchestra.run.smithi123.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-03-30T04:13:33.002 INFO:teuthology.orchestra.run.smithi123.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-03-30T04:13:33.002 INFO:teuthology.orchestra.run.smithi123.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-03-30T04:13:33.002 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-03-30T04:13:33.002 INFO:teuthology.orchestra.run.smithi123.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-03-30T04:13:33.002 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-03-30T04:13:33.003 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-03-30T04:13:33.003 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-03-30T04:13:33.003 INFO:teuthology.orchestra.run.smithi123.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-03-30T04:13:33.003 INFO:teuthology.orchestra.run.smithi123.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-03-30T04:13:33.003 INFO:teuthology.orchestra.run.smithi123.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-03-30T04:13:33.003 INFO:teuthology.orchestra.run.smithi123.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-03-30T04:13:33.003 INFO:teuthology.orchestra.run.smithi123.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-03-30T04:13:33.003 INFO:teuthology.orchestra.run.smithi123.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-03-30T04:13:33.003 INFO:teuthology.orchestra.run.smithi123.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-03-30T04:13:33.003 INFO:teuthology.orchestra.run.smithi123.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-03-30T04:13:33.003 INFO:teuthology.orchestra.run.smithi123.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-03-30T04:13:33.003 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:13:33.003 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-30T04:13:33.003 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-30T04:13:33.003 INFO:teuthology.orchestra.run.smithi123.stdout:Install 32 Packages 2024-03-30T04:13:33.004 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:13:33.004 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 29 M 2024-03-30T04:13:33.004 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 113 M 2024-03-30T04:13:33.004 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-30T04:13:33.403 INFO:teuthology.orchestra.run.smithi123.stdout:(1/32): ceph-17.2.7-797.g5c024335.el8.x86_64.rp 47 kB/s | 6.4 kB 00:00 2024-03-30T04:13:33.629 INFO:teuthology.orchestra.run.smithi123.stdout:(2/32): ceph-mds-17.2.7-797.g5c024335.el8.x86_6 6.4 MB/s | 2.3 MB 00:00 2024-03-30T04:13:34.004 INFO:teuthology.orchestra.run.smithi123.stdout:(3/32): ceph-mon-17.2.7-797.g5c024335.el8.x86_6 6.7 MB/s | 4.0 MB 00:00 2024-03-30T04:13:34.087 INFO:teuthology.orchestra.run.smithi123.stdout:(4/32): ceph-mgr-modules-core-17.2.7-797.g5c024 2.8 MB/s | 240 kB 00:00 2024-03-30T04:13:34.171 INFO:teuthology.orchestra.run.smithi123.stdout:(5/32): ceph-mgr-17.2.7-797.g5c024335.el8.x86_6 1.7 MB/s | 1.6 MB 00:00 2024-03-30T04:13:34.563 INFO:teuthology.orchestra.run.smithi123.stdout:(6/32): ceph-osd-17.2.7-797.g5c024335.el8.x86_6 18 MB/s | 17 MB 00:00 2024-03-30T04:13:34.580 INFO:teuthology.orchestra.run.smithi123.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 95 kB/s | 39 kB 00:00 2024-03-30T04:13:34.655 INFO:teuthology.orchestra.run.smithi123.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 277 kB/s | 157 kB 00:00 2024-03-30T04:13:34.845 INFO:teuthology.orchestra.run.smithi123.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 215 kB/s | 57 kB 00:00 2024-03-30T04:13:34.906 INFO:teuthology.orchestra.run.smithi123.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.8 MB/s | 457 kB 00:00 2024-03-30T04:13:34.939 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: libcephsqlite-2:17.2.7-797.g5c024335.el8.x86_64 5/6 2024-03-30T04:13:34.989 INFO:teuthology.orchestra.run.smithi123.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.7 MB/s | 249 kB 00:00 2024-03-30T04:13:35.048 INFO:teuthology.orchestra.run.smithi123.stdout:(12/32): python3-cryptography-3.2.1-7.el8.x86_6 4.4 MB/s | 641 kB 00:00 2024-03-30T04:13:35.073 INFO:teuthology.orchestra.run.smithi123.stdout:(13/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 203 kB/s | 103 kB 00:00 2024-03-30T04:13:35.098 INFO:teuthology.orchestra.run.smithi123.stdout:(14/32): python3-pycparser-2.14-14.el8.noarch.r 1.0 MB/s | 109 kB 00:00 2024-03-30T04:13:35.185 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-test-2:17.2.7-797.g5c024335.el8.x86_64 6/6 2024-03-30T04:13:35.191 INFO:teuthology.orchestra.run.smithi123.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 307 kB/s | 44 kB 00:00 2024-03-30T04:13:35.233 INFO:teuthology.orchestra.run.smithi123.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.1 MB/s | 185 kB 00:00 2024-03-30T04:13:35.258 INFO:teuthology.orchestra.run.smithi123.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.1 MB/s | 173 kB 00:00 2024-03-30T04:13:35.283 INFO:teuthology.orchestra.run.smithi123.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 222 kB/s | 11 kB 00:00 2024-03-30T04:13:35.308 INFO:teuthology.orchestra.run.smithi123.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 351 kB/s | 18 kB 00:00 2024-03-30T04:13:35.333 INFO:teuthology.orchestra.run.smithi123.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 973 kB/s | 49 kB 00:00 2024-03-30T04:13:35.359 INFO:teuthology.orchestra.run.smithi123.stdout:(21/32): python3-cherrypy-18.4.0-1.el8.noarch.r 2.2 MB/s | 384 kB 00:00 2024-03-30T04:13:35.384 INFO:teuthology.orchestra.run.smithi123.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 781 kB/s | 59 kB 00:00 2024-03-30T04:13:35.409 INFO:teuthology.orchestra.run.smithi123.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 311 kB/s | 16 kB 00:00 2024-03-30T04:13:35.434 INFO:teuthology.orchestra.run.smithi123.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 373 kB/s | 19 kB 00:00 2024-03-30T04:13:35.460 INFO:teuthology.orchestra.run.smithi123.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 475 kB/s | 24 kB 00:00 2024-03-30T04:13:35.485 INFO:teuthology.orchestra.run.smithi123.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 578 kB/s | 29 kB 00:00 2024-03-30T04:13:35.511 INFO:teuthology.orchestra.run.smithi123.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 551 kB/s | 27 kB 00:00 2024-03-30T04:13:35.536 INFO:teuthology.orchestra.run.smithi123.stdout:(28/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.4 MB/s | 283 kB 00:00 2024-03-30T04:13:35.563 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-test-2:17.2.7-797.g5c024335.el8.x86_64 6/6 2024-03-30T04:13:35.563 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-test-2:17.2.7-797.g5c024335.el8.x86_64 1/6 2024-03-30T04:13:35.563 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libcephsqlite-2:17.2.7-797.g5c024335.el8.x86_64 2/6 2024-03-30T04:13:35.563 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-03-30T04:13:35.563 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-03-30T04:13:35.563 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-03-30T04:13:35.578 INFO:teuthology.orchestra.run.smithi123.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.7 MB/s | 255 kB 00:00 2024-03-30T04:13:35.603 INFO:teuthology.orchestra.run.smithi123.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.2 MB/s | 85 kB 00:00 2024-03-30T04:13:35.628 INFO:teuthology.orchestra.run.smithi123.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.1 MB/s | 251 kB 00:00 2024-03-30T04:13:35.654 INFO:teuthology.orchestra.run.smithi123.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 306 kB/s | 23 kB 00:00 2024-03-30T04:13:35.655 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-30T04:13:35.656 INFO:teuthology.orchestra.run.smithi123.stdout:Total 11 MB/s | 29 MB 00:02 2024-03-30T04:13:35.827 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-30T04:13:35.867 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-30T04:13:35.867 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-30T04:13:36.039 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-03-30T04:13:36.039 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:13:36.040 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-03-30T04:13:36.040 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-test-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:13:36.040 INFO:teuthology.orchestra.run.smithi135.stdout: jq-1.6-8.el8.x86_64 2024-03-30T04:13:36.040 INFO:teuthology.orchestra.run.smithi135.stdout: libcephsqlite-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:13:36.040 INFO:teuthology.orchestra.run.smithi135.stdout: oniguruma-6.8.2-3.el8.x86_64 2024-03-30T04:13:36.040 INFO:teuthology.orchestra.run.smithi135.stdout: socat-1.7.4.1-1.el8.x86_64 2024-03-30T04:13:36.040 INFO:teuthology.orchestra.run.smithi135.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-03-30T04:13:36.040 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:13:36.040 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-30T04:13:36.213 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-30T04:13:36.216 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-30T04:13:36.248 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph 2024-03-30T04:13:37.202 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:10 ago on Sat 30 Mar 2024 04:12:27 AM UTC. 2024-03-30T04:13:37.329 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-30T04:13:37.556 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-03-30T04:13:37.807 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-03-30T04:13:38.068 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-03-30T04:13:38.339 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-03-30T04:13:38.536 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-03-30T04:13:38.753 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-03-30T04:13:38.936 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-30T04:13:38.940 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-30T04:13:38.940 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2024-03-30T04:13:38.940 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-30T04:13:38.940 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-03-30T04:13:38.940 INFO:teuthology.orchestra.run.smithi135.stdout: ceph x86_64 2:17.2.7-797.g5c024335.el8 ceph 6.4 k 2024-03-30T04:13:38.940 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2024-03-30T04:13:38.940 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mds x86_64 2:17.2.7-797.g5c024335.el8 ceph 2.3 M 2024-03-30T04:13:38.940 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr x86_64 2:17.2.7-797.g5c024335.el8 ceph 1.6 M 2024-03-30T04:13:38.940 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-modules-core noarch 2:17.2.7-797.g5c024335.el8 ceph-noarch 240 k 2024-03-30T04:13:38.940 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mon x86_64 2:17.2.7-797.g5c024335.el8 ceph 4.0 M 2024-03-30T04:13:38.940 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-osd x86_64 2:17.2.7-797.g5c024335.el8 ceph 17 M 2024-03-30T04:13:38.940 INFO:teuthology.orchestra.run.smithi135.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-03-30T04:13:38.941 INFO:teuthology.orchestra.run.smithi135.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-03-30T04:13:38.941 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-03-30T04:13:38.941 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-03-30T04:13:38.941 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-03-30T04:13:38.941 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-03-30T04:13:38.941 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-03-30T04:13:38.941 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-03-30T04:13:38.941 INFO:teuthology.orchestra.run.smithi135.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-03-30T04:13:38.941 INFO:teuthology.orchestra.run.smithi135.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-03-30T04:13:38.941 INFO:teuthology.orchestra.run.smithi135.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-03-30T04:13:38.941 INFO:teuthology.orchestra.run.smithi135.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-03-30T04:13:38.941 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-03-30T04:13:38.941 INFO:teuthology.orchestra.run.smithi135.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-03-30T04:13:38.941 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-03-30T04:13:38.941 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-03-30T04:13:38.942 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-03-30T04:13:38.942 INFO:teuthology.orchestra.run.smithi135.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-03-30T04:13:38.942 INFO:teuthology.orchestra.run.smithi135.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-03-30T04:13:38.942 INFO:teuthology.orchestra.run.smithi135.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-03-30T04:13:38.942 INFO:teuthology.orchestra.run.smithi135.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-03-30T04:13:38.942 INFO:teuthology.orchestra.run.smithi135.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-03-30T04:13:38.942 INFO:teuthology.orchestra.run.smithi135.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-03-30T04:13:38.942 INFO:teuthology.orchestra.run.smithi135.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-03-30T04:13:38.942 INFO:teuthology.orchestra.run.smithi135.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-03-30T04:13:38.942 INFO:teuthology.orchestra.run.smithi135.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-03-30T04:13:38.942 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:13:38.942 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-03-30T04:13:38.942 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-30T04:13:38.942 INFO:teuthology.orchestra.run.smithi135.stdout:Install 32 Packages 2024-03-30T04:13:38.943 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:13:38.943 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 29 M 2024-03-30T04:13:38.943 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 113 M 2024-03-30T04:13:38.943 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-03-30T04:13:38.967 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-03-30T04:13:39.146 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-03-30T04:13:39.386 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-03-30T04:13:39.697 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-03-30T04:13:39.702 INFO:teuthology.orchestra.run.smithi135.stdout:(1/32): ceph-17.2.7-797.g5c024335.el8.x86_64.rp 22 kB/s | 6.4 kB 00:00 2024-03-30T04:13:39.902 INFO:teuthology.orchestra.run.smithi135.stdout:(2/32): ceph-mds-17.2.7-797.g5c024335.el8.x86_6 4.7 MB/s | 2.3 MB 00:00 2024-03-30T04:13:39.952 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-03-30T04:13:40.193 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-03-30T04:13:40.261 INFO:teuthology.orchestra.run.smithi135.stdout:(3/32): ceph-mgr-17.2.7-797.g5c024335.el8.x86_6 1.9 MB/s | 1.6 MB 00:00 2024-03-30T04:13:40.370 INFO:teuthology.orchestra.run.smithi135.stdout:(4/32): ceph-mgr-modules-core-17.2.7-797.g5c024 2.2 MB/s | 240 kB 00:00 2024-03-30T04:13:40.638 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-03-30T04:13:40.837 INFO:teuthology.orchestra.run.smithi135.stdout:(5/32): python3-mako-1.0.6-14.el8.noarch.rpm 337 kB/s | 157 kB 00:00 2024-03-30T04:13:40.945 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-03-30T04:13:40.945 INFO:teuthology.orchestra.run.smithi135.stdout:(6/32): ceph-mon-17.2.7-797.g5c024335.el8.x86_6 3.2 MB/s | 4.0 MB 00:01 2024-03-30T04:13:40.971 INFO:teuthology.orchestra.run.smithi135.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 291 kB/s | 39 kB 00:00 2024-03-30T04:13:41.071 INFO:teuthology.orchestra.run.smithi135.stdout:(8/32): python3-pytz-2017.2-11.el8.noarch.rpm 570 kB/s | 57 kB 00:00 2024-03-30T04:13:41.140 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-03-30T04:13:41.322 INFO:teuthology.orchestra.run.smithi135.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 1.8 MB/s | 457 kB 00:00 2024-03-30T04:13:41.347 INFO:teuthology.orchestra.run.smithi135.stdout:(10/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 258 kB/s | 103 kB 00:00 2024-03-30T04:13:41.414 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-03-30T04:13:41.430 INFO:teuthology.orchestra.run.smithi135.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 2.2 MB/s | 249 kB 00:00 2024-03-30T04:13:41.464 INFO:teuthology.orchestra.run.smithi135.stdout:(12/32): python3-cryptography-3.2.1-7.el8.x86_6 5.4 MB/s | 641 kB 00:00 2024-03-30T04:13:41.489 INFO:teuthology.orchestra.run.smithi135.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 1.8 MB/s | 109 kB 00:00 2024-03-30T04:13:41.650 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-03-30T04:13:41.748 INFO:teuthology.orchestra.run.smithi135.stdout:(14/32): ceph-osd-17.2.7-797.g5c024335.el8.x86_ 9.3 MB/s | 17 MB 00:01 2024-03-30T04:13:41.773 INFO:teuthology.orchestra.run.smithi135.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 141 kB/s | 44 kB 00:00 2024-03-30T04:13:41.982 INFO:teuthology.orchestra.run.smithi135.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 376 kB/s | 185 kB 00:00 2024-03-30T04:13:41.987 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-03-30T04:13:42.007 INFO:teuthology.orchestra.run.smithi135.stdout:(17/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.6 MB/s | 384 kB 00:00 2024-03-30T04:13:42.074 INFO:teuthology.orchestra.run.smithi135.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 121 kB/s | 11 kB 00:00 2024-03-30T04:13:42.099 INFO:teuthology.orchestra.run.smithi135.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 191 kB/s | 18 kB 00:00 2024-03-30T04:13:42.125 INFO:teuthology.orchestra.run.smithi135.stdout:(20/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 460 kB/s | 173 kB 00:00 2024-03-30T04:13:42.150 INFO:teuthology.orchestra.run.smithi135.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 648 kB/s | 49 kB 00:00 2024-03-30T04:13:42.175 INFO:teuthology.orchestra.run.smithi135.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 779 kB/s | 59 kB 00:00 2024-03-30T04:13:42.197 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-03-30T04:13:42.217 INFO:teuthology.orchestra.run.smithi135.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 233 kB/s | 16 kB 00:00 2024-03-30T04:13:42.234 INFO:teuthology.orchestra.run.smithi135.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 320 kB/s | 19 kB 00:00 2024-03-30T04:13:42.268 INFO:teuthology.orchestra.run.smithi135.stdout:(25/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.9 MB/s | 283 kB 00:00 2024-03-30T04:13:42.285 INFO:teuthology.orchestra.run.smithi135.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 357 kB/s | 24 kB 00:00 2024-03-30T04:13:42.302 INFO:teuthology.orchestra.run.smithi135.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 436 kB/s | 29 kB 00:00 2024-03-30T04:13:42.327 INFO:teuthology.orchestra.run.smithi135.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 467 kB/s | 27 kB 00:00 2024-03-30T04:13:42.386 INFO:teuthology.orchestra.run.smithi135.stdout:(29/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.9 MB/s | 251 kB 00:00 2024-03-30T04:13:42.409 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-03-30T04:13:42.411 INFO:teuthology.orchestra.run.smithi135.stdout:(30/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.0 MB/s | 255 kB 00:00 2024-03-30T04:13:42.620 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-03-30T04:13:42.823 INFO:teuthology.orchestra.run.smithi135.stdout:(31/32): python3-webtest-2.0.33-1.el8.noarch.rp 172 kB/s | 85 kB 00:00 2024-03-30T04:13:42.927 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-03-30T04:13:43.224 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-03-30T04:13:43.224 INFO:teuthology.orchestra.run.smithi135.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 27 kB/s | 23 kB 00:00 2024-03-30T04:13:43.228 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-03-30T04:13:43.228 INFO:teuthology.orchestra.run.smithi135.stdout:Total 6.8 MB/s | 29 MB 00:04 2024-03-30T04:13:43.403 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-03-30T04:13:43.445 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-03-30T04:13:43.446 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-03-30T04:13:43.459 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-03-30T04:13:43.771 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-03-30T04:13:43.775 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-03-30T04:13:43.813 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-03-30T04:13:44.156 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-03-30T04:13:44.548 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-797.g5c024335.el8.n 27/32 2024-03-30T04:13:44.760 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-2:17.2.7-797.g5c024335.el8.x86_64 28/32 2024-03-30T04:13:44.919 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-03-30T04:13:45.219 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-03-30T04:13:45.462 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-03-30T04:13:45.731 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-03-30T04:13:46.001 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-03-30T04:13:46.173 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-03-30T04:13:46.383 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-03-30T04:13:46.408 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-2:17.2.7-797.g5c024335.el8.x86_64 28/32 2024-03-30T04:13:46.597 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-03-30T04:13:46.643 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-osd-2:17.2.7-797.g5c024335.el8.x86_64 29/32 2024-03-30T04:13:46.767 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-03-30T04:13:47.041 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-03-30T04:13:47.113 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-osd-2:17.2.7-797.g5c024335.el8.x86_64 29/32 2024-03-30T04:13:47.321 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mon-2:17.2.7-797.g5c024335.el8.x86_64 30/32 2024-03-30T04:13:47.326 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-03-30T04:13:47.586 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mon-2:17.2.7-797.g5c024335.el8.x86_64 30/32 2024-03-30T04:13:47.598 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-03-30T04:13:47.796 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mds-2:17.2.7-797.g5c024335.el8.x86_64 31/32 2024-03-30T04:13:47.837 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mds-2:17.2.7-797.g5c024335.el8.x86_64 31/32 2024-03-30T04:13:47.848 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-03-30T04:13:47.971 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-2:17.2.7-797.g5c024335.el8.x86_64 32/32 2024-03-30T04:13:48.077 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-2:17.2.7-797.g5c024335.el8.x86_64 32/32 2024-03-30T04:13:48.077 INFO:teuthology.orchestra.run.smithi123.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-30T04:13:48.077 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:13:48.343 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-03-30T04:13:48.582 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-03-30T04:13:48.617 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-2:17.2.7-797.g5c024335.el8.x86_64 1/32 2024-03-30T04:13:48.617 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mds-2:17.2.7-797.g5c024335.el8.x86_64 2/32 2024-03-30T04:13:48.617 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-2:17.2.7-797.g5c024335.el8.x86_64 3/32 2024-03-30T04:13:48.617 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mon-2:17.2.7-797.g5c024335.el8.x86_64 4/32 2024-03-30T04:13:48.617 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-osd-2:17.2.7-797.g5c024335.el8.x86_64 5/32 2024-03-30T04:13:48.617 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-797.g5c024335.el8.n 6/32 2024-03-30T04:13:48.617 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-03-30T04:13:48.618 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-03-30T04:13:48.618 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-03-30T04:13:48.618 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-03-30T04:13:48.618 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-03-30T04:13:48.618 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-03-30T04:13:48.618 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-03-30T04:13:48.618 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-03-30T04:13:48.618 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-03-30T04:13:48.618 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-03-30T04:13:48.618 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-03-30T04:13:48.618 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-03-30T04:13:48.618 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-03-30T04:13:48.618 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-03-30T04:13:48.619 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-03-30T04:13:48.619 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-03-30T04:13:48.619 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-03-30T04:13:48.619 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-03-30T04:13:48.619 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-03-30T04:13:48.619 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-03-30T04:13:48.619 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-03-30T04:13:48.619 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-03-30T04:13:48.619 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-03-30T04:13:48.619 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-03-30T04:13:48.619 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-03-30T04:13:48.769 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-03-30T04:13:49.061 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-03-30T04:13:49.296 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-03-30T04:13:49.507 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-03-30T04:13:49.507 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:13:49.507 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-30T04:13:49.507 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:13:49.507 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mds-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:13:49.507 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:13:49.508 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-modules-core-2:17.2.7-797.g5c024335.el8.noarch 2024-03-30T04:13:49.508 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mon-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:13:49.508 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-osd-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:13:49.508 INFO:teuthology.orchestra.run.smithi123.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-03-30T04:13:49.508 INFO:teuthology.orchestra.run.smithi123.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-03-30T04:13:49.508 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-03-30T04:13:49.508 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-03-30T04:13:49.508 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-03-30T04:13:49.508 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-03-30T04:13:49.509 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-6.2-6.el8.noarch 2024-03-30T04:13:49.509 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-03-30T04:13:49.509 INFO:teuthology.orchestra.run.smithi123.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-03-30T04:13:49.509 INFO:teuthology.orchestra.run.smithi123.stdout: python3-mako-1.0.6-14.el8.noarch 2024-03-30T04:13:49.509 INFO:teuthology.orchestra.run.smithi123.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-03-30T04:13:49.509 INFO:teuthology.orchestra.run.smithi123.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-03-30T04:13:49.509 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-03-30T04:13:49.509 INFO:teuthology.orchestra.run.smithi123.stdout: python3-portend-2.6-1.el8.noarch 2024-03-30T04:13:49.509 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-03-30T04:13:49.510 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pycparser-2.14-14.el8.noarch 2024-03-30T04:13:49.510 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pytz-2017.2-11.el8.noarch 2024-03-30T04:13:49.510 INFO:teuthology.orchestra.run.smithi123.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-03-30T04:13:49.510 INFO:teuthology.orchestra.run.smithi123.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-03-30T04:13:49.510 INFO:teuthology.orchestra.run.smithi123.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-03-30T04:13:49.510 INFO:teuthology.orchestra.run.smithi123.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-03-30T04:13:49.510 INFO:teuthology.orchestra.run.smithi123.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-03-30T04:13:49.510 INFO:teuthology.orchestra.run.smithi123.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-03-30T04:13:49.510 INFO:teuthology.orchestra.run.smithi123.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-03-30T04:13:49.510 INFO:teuthology.orchestra.run.smithi123.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-03-30T04:13:49.511 INFO:teuthology.orchestra.run.smithi123.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-03-30T04:13:49.511 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:13:49.511 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-30T04:13:49.634 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-03-30T04:13:49.803 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-base 2024-03-30T04:13:49.892 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-03-30T04:13:50.097 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-03-30T04:13:50.307 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-03-30T04:13:50.604 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-03-30T04:13:50.729 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:19 ago on Sat 30 Mar 2024 04:12:31 AM UTC. 2024-03-30T04:13:50.890 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-03-30T04:13:51.131 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-03-30T04:13:51.505 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-03-30T04:13:51.868 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-03-30T04:13:52.260 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-797.g5c024335.el8.n 27/32 2024-03-30T04:13:52.462 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mgr-2:17.2.7-797.g5c024335.el8.x86_64 28/32 2024-03-30T04:13:52.484 INFO:teuthology.orchestra.run.smithi123.stdout:Package ceph-base-2:17.2.7-797.g5c024335.el8.x86_64 is already installed. 2024-03-30T04:13:52.535 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-30T04:13:52.537 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2024-03-30T04:13:52.537 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-30T04:13:52.669 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install cephadm 2024-03-30T04:13:53.587 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:22 ago on Sat 30 Mar 2024 04:12:31 AM UTC. 2024-03-30T04:13:54.103 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mgr-2:17.2.7-797.g5c024335.el8.x86_64 28/32 2024-03-30T04:13:54.337 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-osd-2:17.2.7-797.g5c024335.el8.x86_64 29/32 2024-03-30T04:13:54.798 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-osd-2:17.2.7-797.g5c024335.el8.x86_64 29/32 2024-03-30T04:13:55.009 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mon-2:17.2.7-797.g5c024335.el8.x86_64 30/32 2024-03-30T04:13:55.275 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mon-2:17.2.7-797.g5c024335.el8.x86_64 30/32 2024-03-30T04:13:55.293 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-30T04:13:55.294 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-30T04:13:55.294 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2024-03-30T04:13:55.294 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-30T04:13:55.294 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-30T04:13:55.295 INFO:teuthology.orchestra.run.smithi123.stdout: cephadm noarch 2:17.2.7-797.g5c024335.el8 ceph-noarch 82 k 2024-03-30T04:13:55.295 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:13:55.295 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-30T04:13:55.295 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-30T04:13:55.295 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2024-03-30T04:13:55.295 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:13:55.295 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 82 k 2024-03-30T04:13:55.295 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 372 k 2024-03-30T04:13:55.296 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-30T04:13:55.459 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mds-2:17.2.7-797.g5c024335.el8.x86_64 31/32 2024-03-30T04:13:55.500 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mds-2:17.2.7-797.g5c024335.el8.x86_64 31/32 2024-03-30T04:13:55.676 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-2:17.2.7-797.g5c024335.el8.x86_64 32/32 2024-03-30T04:13:55.779 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-2:17.2.7-797.g5c024335.el8.x86_64 32/32 2024-03-30T04:13:55.779 INFO:teuthology.orchestra.run.smithi135.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-30T04:13:55.779 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:13:55.928 INFO:teuthology.orchestra.run.smithi123.stdout:cephadm-17.2.7-797.g5c024335.el8.noarch.rpm 130 kB/s | 82 kB 00:00 2024-03-30T04:13:55.929 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-30T04:13:55.929 INFO:teuthology.orchestra.run.smithi123.stdout:Total 129 kB/s | 82 kB 00:00 2024-03-30T04:13:55.929 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-30T04:13:55.941 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-30T04:13:55.941 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-30T04:13:55.944 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-30T04:13:55.948 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-30T04:13:55.998 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-30T04:13:56.298 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-2:17.2.7-797.g5c024335.el8.x86_64 1/32 2024-03-30T04:13:56.298 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mds-2:17.2.7-797.g5c024335.el8.x86_64 2/32 2024-03-30T04:13:56.298 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mgr-2:17.2.7-797.g5c024335.el8.x86_64 3/32 2024-03-30T04:13:56.299 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mon-2:17.2.7-797.g5c024335.el8.x86_64 4/32 2024-03-30T04:13:56.299 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-osd-2:17.2.7-797.g5c024335.el8.x86_64 5/32 2024-03-30T04:13:56.299 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-797.g5c024335.el8.n 6/32 2024-03-30T04:13:56.299 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-03-30T04:13:56.299 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-03-30T04:13:56.299 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-03-30T04:13:56.299 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-03-30T04:13:56.299 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-03-30T04:13:56.299 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-03-30T04:13:56.299 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-03-30T04:13:56.299 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-03-30T04:13:56.299 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-03-30T04:13:56.299 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-03-30T04:13:56.299 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-03-30T04:13:56.300 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-03-30T04:13:56.300 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-03-30T04:13:56.300 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-03-30T04:13:56.300 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-03-30T04:13:56.300 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-03-30T04:13:56.300 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-03-30T04:13:56.300 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-03-30T04:13:56.300 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-03-30T04:13:56.301 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-03-30T04:13:56.301 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-03-30T04:13:56.301 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-03-30T04:13:56.301 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-03-30T04:13:56.301 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-03-30T04:13:56.301 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-03-30T04:13:56.541 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: cephadm-2:17.2.7-797.g5c024335.el8.noarch 1/1 2024-03-30T04:13:56.705 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : cephadm-2:17.2.7-797.g5c024335.el8.noarch 1/1 2024-03-30T04:13:57.006 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: cephadm-2:17.2.7-797.g5c024335.el8.noarch 1/1 2024-03-30T04:13:57.146 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-03-30T04:13:57.147 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:13:57.147 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-03-30T04:13:57.147 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:13:57.147 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mds-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:13:57.147 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:13:57.147 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-modules-core-2:17.2.7-797.g5c024335.el8.noarch 2024-03-30T04:13:57.147 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mon-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:13:57.147 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-osd-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:13:57.147 INFO:teuthology.orchestra.run.smithi135.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-03-30T04:13:57.147 INFO:teuthology.orchestra.run.smithi135.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-03-30T04:13:57.147 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-03-30T04:13:57.147 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-03-30T04:13:57.148 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-03-30T04:13:57.148 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-03-30T04:13:57.148 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-6.2-6.el8.noarch 2024-03-30T04:13:57.148 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-03-30T04:13:57.148 INFO:teuthology.orchestra.run.smithi135.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-03-30T04:13:57.148 INFO:teuthology.orchestra.run.smithi135.stdout: python3-mako-1.0.6-14.el8.noarch 2024-03-30T04:13:57.148 INFO:teuthology.orchestra.run.smithi135.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-03-30T04:13:57.148 INFO:teuthology.orchestra.run.smithi135.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-03-30T04:13:57.148 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-03-30T04:13:57.148 INFO:teuthology.orchestra.run.smithi135.stdout: python3-portend-2.6-1.el8.noarch 2024-03-30T04:13:57.148 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-03-30T04:13:57.148 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pycparser-2.14-14.el8.noarch 2024-03-30T04:13:57.148 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pytz-2017.2-11.el8.noarch 2024-03-30T04:13:57.148 INFO:teuthology.orchestra.run.smithi135.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-03-30T04:13:57.148 INFO:teuthology.orchestra.run.smithi135.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-03-30T04:13:57.149 INFO:teuthology.orchestra.run.smithi135.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-03-30T04:13:57.149 INFO:teuthology.orchestra.run.smithi135.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-03-30T04:13:57.149 INFO:teuthology.orchestra.run.smithi135.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-03-30T04:13:57.149 INFO:teuthology.orchestra.run.smithi135.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-03-30T04:13:57.149 INFO:teuthology.orchestra.run.smithi135.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-03-30T04:13:57.149 INFO:teuthology.orchestra.run.smithi135.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-03-30T04:13:57.149 INFO:teuthology.orchestra.run.smithi135.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-03-30T04:13:57.149 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:13:57.149 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-30T04:13:57.440 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-base 2024-03-30T04:13:57.467 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : cephadm-2:17.2.7-797.g5c024335.el8.noarch 1/1 2024-03-30T04:13:57.468 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:13:57.468 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-30T04:13:57.468 INFO:teuthology.orchestra.run.smithi123.stdout: cephadm-2:17.2.7-797.g5c024335.el8.noarch 2024-03-30T04:13:57.468 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:13:57.468 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-30T04:13:57.717 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-immutable-object-cache 2024-03-30T04:13:58.382 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:31 ago on Sat 30 Mar 2024 04:12:27 AM UTC. 2024-03-30T04:13:58.652 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:27 ago on Sat 30 Mar 2024 04:12:31 AM UTC. 2024-03-30T04:13:59.986 INFO:teuthology.orchestra.run.smithi135.stdout:Package ceph-base-2:17.2.7-797.g5c024335.el8.x86_64 is already installed. 2024-03-30T04:14:00.037 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-30T04:14:00.038 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2024-03-30T04:14:00.038 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-30T04:14:00.168 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install cephadm 2024-03-30T04:14:00.318 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-30T04:14:00.319 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-30T04:14:00.319 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repo Size 2024-03-30T04:14:00.319 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-30T04:14:00.319 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-30T04:14:00.319 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-797.g5c024335.el8 ceph 162 k 2024-03-30T04:14:00.319 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:14:00.319 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-30T04:14:00.319 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-30T04:14:00.319 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2024-03-30T04:14:00.319 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:14:00.320 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 162 k 2024-03-30T04:14:00.320 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 494 k 2024-03-30T04:14:00.320 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-30T04:14:00.533 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-immutable-object-cache-17.2.7-797.g5c02433 762 kB/s | 162 kB 00:00 2024-03-30T04:14:00.533 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-30T04:14:00.533 INFO:teuthology.orchestra.run.smithi123.stdout:Total 757 kB/s | 162 kB 00:00 2024-03-30T04:14:00.534 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-30T04:14:00.547 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-30T04:14:00.547 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-30T04:14:00.600 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-30T04:14:00.604 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-30T04:14:00.725 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-30T04:14:00.986 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-797.g5c024335.e 1/1 2024-03-30T04:14:01.095 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:34 ago on Sat 30 Mar 2024 04:12:27 AM UTC. 2024-03-30T04:14:02.612 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-797.g5c024335.e 1/1 2024-03-30T04:14:02.787 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-30T04:14:02.787 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-30T04:14:02.787 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2024-03-30T04:14:02.787 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-30T04:14:02.787 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-03-30T04:14:02.787 INFO:teuthology.orchestra.run.smithi135.stdout: cephadm noarch 2:17.2.7-797.g5c024335.el8 ceph-noarch 82 k 2024-03-30T04:14:02.788 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:14:02.788 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-03-30T04:14:02.788 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-30T04:14:02.788 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2024-03-30T04:14:02.788 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:14:02.788 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 82 k 2024-03-30T04:14:02.789 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 372 k 2024-03-30T04:14:02.789 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-03-30T04:14:03.046 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-797.g5c024335.e 1/1 2024-03-30T04:14:03.047 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:14:03.047 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-30T04:14:03.047 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-immutable-object-cache-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:14:03.047 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:14:03.047 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-30T04:14:03.072 INFO:teuthology.orchestra.run.smithi135.stdout:cephadm-17.2.7-797.g5c024335.el8.noarch.rpm 289 kB/s | 82 kB 00:00 2024-03-30T04:14:03.072 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-03-30T04:14:03.073 INFO:teuthology.orchestra.run.smithi135.stdout:Total 287 kB/s | 82 kB 00:00 2024-03-30T04:14:03.073 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-03-30T04:14:03.083 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-03-30T04:14:03.084 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-03-30T04:14:03.087 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-03-30T04:14:03.090 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-03-30T04:14:03.135 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-03-30T04:14:03.300 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-mgr 2024-03-30T04:14:04.227 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:33 ago on Sat 30 Mar 2024 04:12:31 AM UTC. 2024-03-30T04:14:05.850 INFO:teuthology.orchestra.run.smithi123.stdout:Package ceph-mgr-2:17.2.7-797.g5c024335.el8.x86_64 is already installed. 2024-03-30T04:14:05.902 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-30T04:14:05.907 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2024-03-30T04:14:05.907 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-30T04:14:05.907 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: cephadm-2:17.2.7-797.g5c024335.el8.noarch 1/1 2024-03-30T04:14:06.025 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-mgr-dashboard 2024-03-30T04:14:06.123 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : cephadm-2:17.2.7-797.g5c024335.el8.noarch 1/1 2024-03-30T04:14:06.434 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: cephadm-2:17.2.7-797.g5c024335.el8.noarch 1/1 2024-03-30T04:14:06.891 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : cephadm-2:17.2.7-797.g5c024335.el8.noarch 1/1 2024-03-30T04:14:06.891 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:14:06.891 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-03-30T04:14:06.891 INFO:teuthology.orchestra.run.smithi135.stdout: cephadm-2:17.2.7-797.g5c024335.el8.noarch 2024-03-30T04:14:06.891 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:14:06.892 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-30T04:14:06.938 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:35 ago on Sat 30 Mar 2024 04:12:31 AM UTC. 2024-03-30T04:14:07.144 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-immutable-object-cache 2024-03-30T04:14:08.084 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:41 ago on Sat 30 Mar 2024 04:12:27 AM UTC. 2024-03-30T04:14:08.657 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-30T04:14:08.659 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-30T04:14:08.659 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2024-03-30T04:14:08.659 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-30T04:14:08.659 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-30T04:14:08.659 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-dashboard noarch 2:17.2.7-797.g5c024335.el8 ceph-noarch 1.7 M 2024-03-30T04:14:08.659 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2024-03-30T04:14:08.659 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-grafana-dashboards noarch 2:17.2.7-797.g5c024335.el8 ceph-noarch 23 k 2024-03-30T04:14:08.659 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-prometheus-alerts noarch 2:17.2.7-797.g5c024335.el8 ceph-noarch 14 k 2024-03-30T04:14:08.659 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-03-30T04:14:08.659 INFO:teuthology.orchestra.run.smithi123.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-03-30T04:14:08.659 INFO:teuthology.orchestra.run.smithi123.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-03-30T04:14:08.659 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:14:08.660 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-30T04:14:08.660 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-30T04:14:08.660 INFO:teuthology.orchestra.run.smithi123.stdout:Install 6 Packages 2024-03-30T04:14:08.660 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:14:08.660 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 2.0 M 2024-03-30T04:14:08.660 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 8.0 M 2024-03-30T04:14:08.660 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-30T04:14:08.986 INFO:teuthology.orchestra.run.smithi123.stdout:(1/6): ceph-grafana-dashboards-17.2.7-797.g5c02 172 kB/s | 23 kB 00:00 2024-03-30T04:14:09.053 INFO:teuthology.orchestra.run.smithi123.stdout:(2/6): python3-jwt-1.6.1-2.el8.noarch.rpm 642 kB/s | 43 kB 00:00 2024-03-30T04:14:09.087 INFO:teuthology.orchestra.run.smithi123.stdout:(3/6): ceph-prometheus-alerts-17.2.7-797.g5c024 61 kB/s | 14 kB 00:00 2024-03-30T04:14:09.321 INFO:teuthology.orchestra.run.smithi123.stdout:(4/6): ceph-mgr-dashboard-17.2.7-797.g5c024335. 3.6 MB/s | 1.7 MB 00:00 2024-03-30T04:14:09.479 INFO:teuthology.orchestra.run.smithi123.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 79 kB/s | 34 kB 00:00 2024-03-30T04:14:09.687 INFO:teuthology.orchestra.run.smithi123.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 327 kB/s | 196 kB 00:00 2024-03-30T04:14:09.690 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-30T04:14:09.691 INFO:teuthology.orchestra.run.smithi123.stdout:Total 1.9 MB/s | 2.0 MB 00:01 2024-03-30T04:14:09.713 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-30T04:14:09.726 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-30T04:14:09.726 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-30T04:14:09.765 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-30T04:14:09.766 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-30T04:14:09.766 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2024-03-30T04:14:09.766 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-30T04:14:09.766 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-03-30T04:14:09.766 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-797.g5c024335.el8 ceph 162 k 2024-03-30T04:14:09.766 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:14:09.766 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-03-30T04:14:09.766 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-30T04:14:09.767 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2024-03-30T04:14:09.767 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:14:09.767 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 162 k 2024-03-30T04:14:09.767 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 494 k 2024-03-30T04:14:09.767 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-03-30T04:14:09.831 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-30T04:14:09.835 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-30T04:14:10.095 INFO:teuthology.orchestra.run.smithi135.stdout:ceph-immutable-object-cache-17.2.7-797.g5c02433 492 kB/s | 162 kB 00:00 2024-03-30T04:14:10.096 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-03-30T04:14:10.096 INFO:teuthology.orchestra.run.smithi135.stdout:Total 490 kB/s | 162 kB 00:00 2024-03-30T04:14:10.096 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-03-30T04:14:10.110 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-03-30T04:14:10.110 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-03-30T04:14:10.110 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-30T04:14:10.163 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-03-30T04:14:10.167 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-03-30T04:14:10.288 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-03-30T04:14:10.354 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-03-30T04:14:10.482 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-797.g5c024335.e 1/1 2024-03-30T04:14:10.605 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-03-30T04:14:10.820 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-03-30T04:14:10.983 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-797.g5c024335.el8.no 4/6 2024-03-30T04:14:11.131 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-797.g5c024335.e 1/1 2024-03-30T04:14:11.360 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-797.g5c024335.el8.n 5/6 2024-03-30T04:14:11.563 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-797.g5c024335.el8.noarch 6/6 2024-03-30T04:14:11.574 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-797.g5c024335.e 1/1 2024-03-30T04:14:11.574 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:14:11.574 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-03-30T04:14:11.574 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-immutable-object-cache-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:14:11.574 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:14:11.574 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-30T04:14:11.848 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-mgr 2024-03-30T04:14:11.897 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-797.g5c024335.el8.noarch 6/6 2024-03-30T04:14:11.897 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-797.g5c024335.el8.n 1/6 2024-03-30T04:14:11.897 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-797.g5c024335.el8.noarch 2/6 2024-03-30T04:14:11.897 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-797.g5c024335.el8.no 3/6 2024-03-30T04:14:11.897 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-03-30T04:14:11.897 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-03-30T04:14:12.468 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-03-30T04:14:12.468 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:14:12.468 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-30T04:14:12.468 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-grafana-dashboards-2:17.2.7-797.g5c024335.el8.noarch 2024-03-30T04:14:12.468 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-dashboard-2:17.2.7-797.g5c024335.el8.noarch 2024-03-30T04:14:12.468 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-prometheus-alerts-2:17.2.7-797.g5c024335.el8.noarch 2024-03-30T04:14:12.469 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-03-30T04:14:12.469 INFO:teuthology.orchestra.run.smithi123.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-03-30T04:14:12.469 INFO:teuthology.orchestra.run.smithi123.stdout: python3-routes-2.4.1-12.el8.noarch 2024-03-30T04:14:12.469 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:14:12.469 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-30T04:14:12.627 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-30T04:14:12.774 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:45 ago on Sat 30 Mar 2024 04:12:27 AM UTC. 2024-03-30T04:14:13.539 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:42 ago on Sat 30 Mar 2024 04:12:31 AM UTC. 2024-03-30T04:14:14.410 INFO:teuthology.orchestra.run.smithi135.stdout:Package ceph-mgr-2:17.2.7-797.g5c024335.el8.x86_64 is already installed. 2024-03-30T04:14:14.463 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-30T04:14:14.464 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2024-03-30T04:14:14.464 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-30T04:14:14.582 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-mgr-dashboard 2024-03-30T04:14:15.198 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-30T04:14:15.199 INFO:teuthology.orchestra.run.smithi123.stdout:============================================================================================ 2024-03-30T04:14:15.200 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2024-03-30T04:14:15.200 INFO:teuthology.orchestra.run.smithi123.stdout:============================================================================================ 2024-03-30T04:14:15.200 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-30T04:14:15.200 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-797.g5c024335.el8 ceph-noarch 7.4 M 2024-03-30T04:14:15.200 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2024-03-30T04:14:15.200 INFO:teuthology.orchestra.run.smithi123.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-03-30T04:14:15.200 INFO:teuthology.orchestra.run.smithi123.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-03-30T04:14:15.200 INFO:teuthology.orchestra.run.smithi123.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-03-30T04:14:15.200 INFO:teuthology.orchestra.run.smithi123.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-03-30T04:14:15.200 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-03-30T04:14:15.200 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-03-30T04:14:15.200 INFO:teuthology.orchestra.run.smithi123.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-03-30T04:14:15.200 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:14:15.200 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-30T04:14:15.201 INFO:teuthology.orchestra.run.smithi123.stdout:============================================================================================ 2024-03-30T04:14:15.201 INFO:teuthology.orchestra.run.smithi123.stdout:Install 8 Packages 2024-03-30T04:14:15.201 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:14:15.201 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 36 M 2024-03-30T04:14:15.201 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 210 M 2024-03-30T04:14:15.201 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-30T04:14:15.497 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:48 ago on Sat 30 Mar 2024 04:12:27 AM UTC. 2024-03-30T04:14:16.387 INFO:teuthology.orchestra.run.smithi123.stdout:(1/8): openblas-0.3.15-4.el8.x86_64.rpm 5.8 MB/s | 4.7 MB 00:00 2024-03-30T04:14:16.462 INFO:teuthology.orchestra.run.smithi123.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 5.5 MB/s | 4.9 MB 00:00 2024-03-30T04:14:16.562 INFO:teuthology.orchestra.run.smithi123.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.2 MB/s | 225 kB 00:00 2024-03-30T04:14:16.762 INFO:teuthology.orchestra.run.smithi123.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 10 MB/s | 3.7 MB 00:00 2024-03-30T04:14:16.871 INFO:teuthology.orchestra.run.smithi123.stdout:(5/8): libgfortran-8.5.0-21.el8.x86_64.rpm 5.8 MB/s | 645 kB 00:00 2024-03-30T04:14:16.913 INFO:teuthology.orchestra.run.smithi123.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 4.1 MB/s | 172 kB 00:00 2024-03-30T04:14:17.220 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-30T04:14:17.221 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-30T04:14:17.221 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2024-03-30T04:14:17.221 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-30T04:14:17.221 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-03-30T04:14:17.221 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-dashboard noarch 2:17.2.7-797.g5c024335.el8 ceph-noarch 1.7 M 2024-03-30T04:14:17.221 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2024-03-30T04:14:17.221 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-grafana-dashboards noarch 2:17.2.7-797.g5c024335.el8 ceph-noarch 23 k 2024-03-30T04:14:17.221 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-prometheus-alerts noarch 2:17.2.7-797.g5c024335.el8 ceph-noarch 14 k 2024-03-30T04:14:17.222 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-03-30T04:14:17.222 INFO:teuthology.orchestra.run.smithi135.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-03-30T04:14:17.222 INFO:teuthology.orchestra.run.smithi135.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-03-30T04:14:17.222 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:14:17.222 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-03-30T04:14:17.222 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-30T04:14:17.222 INFO:teuthology.orchestra.run.smithi135.stdout:Install 6 Packages 2024-03-30T04:14:17.222 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:14:17.223 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 2.0 M 2024-03-30T04:14:17.223 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 8.0 M 2024-03-30T04:14:17.223 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-03-30T04:14:17.280 INFO:teuthology.orchestra.run.smithi123.stdout:(7/8): python3-scipy-1.0.0-21.module_el8+762+77 20 MB/s | 14 MB 00:00 2024-03-30T04:14:17.558 INFO:teuthology.orchestra.run.smithi135.stdout:(1/6): ceph-grafana-dashboards-17.2.7-797.g5c02 109 kB/s | 23 kB 00:00 2024-03-30T04:14:17.582 INFO:teuthology.orchestra.run.smithi135.stdout:(2/6): ceph-prometheus-alerts-17.2.7-797.g5c024 61 kB/s | 14 kB 00:00 2024-03-30T04:14:17.650 INFO:teuthology.orchestra.run.smithi135.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 463 kB/s | 43 kB 00:00 2024-03-30T04:14:17.675 INFO:teuthology.orchestra.run.smithi135.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 365 kB/s | 34 kB 00:00 2024-03-30T04:14:17.826 INFO:teuthology.orchestra.run.smithi135.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.1 MB/s | 196 kB 00:00 2024-03-30T04:14:18.115 INFO:teuthology.orchestra.run.smithi123.stdout:(8/8): ceph-mgr-diskprediction-local-17.2.7-797 2.9 MB/s | 7.4 MB 00:02 2024-03-30T04:14:18.116 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-30T04:14:18.116 INFO:teuthology.orchestra.run.smithi123.stdout:Total 12 MB/s | 36 MB 00:02 2024-03-30T04:14:18.184 INFO:teuthology.orchestra.run.smithi135.stdout:(6/6): ceph-mgr-dashboard-17.2.7-797.g5c024335. 2.0 MB/s | 1.7 MB 00:00 2024-03-30T04:14:18.185 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-03-30T04:14:18.186 INFO:teuthology.orchestra.run.smithi135.stdout:Total 2.1 MB/s | 2.0 MB 00:00 2024-03-30T04:14:18.207 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-03-30T04:14:18.220 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-03-30T04:14:18.220 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-03-30T04:14:18.321 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-03-30T04:14:18.325 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-03-30T04:14:18.425 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-30T04:14:18.447 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-30T04:14:18.447 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-30T04:14:18.639 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-03-30T04:14:18.883 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-03-30T04:14:18.894 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-30T04:14:18.898 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-30T04:14:19.144 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-03-30T04:14:19.383 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-03-30T04:14:19.488 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-30T04:14:19.504 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-797.g5c024335.el8.no 4/6 2024-03-30T04:14:19.632 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-03-30T04:14:19.772 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-03-30T04:14:19.880 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-797.g5c024335.el8.n 5/6 2024-03-30T04:14:19.915 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-03-30T04:14:20.057 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-797.g5c024335.el8.noarch 6/6 2024-03-30T04:14:20.397 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-797.g5c024335.el8.noarch 6/6 2024-03-30T04:14:20.397 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-797.g5c024335.el8.n 1/6 2024-03-30T04:14:20.398 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-797.g5c024335.el8.noarch 2/6 2024-03-30T04:14:20.398 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-797.g5c024335.el8.no 3/6 2024-03-30T04:14:20.398 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-03-30T04:14:20.398 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-03-30T04:14:20.578 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-03-30T04:14:20.749 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-03-30T04:14:20.839 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-03-30T04:14:20.840 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:14:20.840 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-03-30T04:14:20.840 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-grafana-dashboards-2:17.2.7-797.g5c024335.el8.noarch 2024-03-30T04:14:20.840 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-dashboard-2:17.2.7-797.g5c024335.el8.noarch 2024-03-30T04:14:20.840 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-prometheus-alerts-2:17.2.7-797.g5c024335.el8.noarch 2024-03-30T04:14:20.840 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-03-30T04:14:20.840 INFO:teuthology.orchestra.run.smithi135.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-03-30T04:14:20.840 INFO:teuthology.orchestra.run.smithi135.stdout: python3-routes-2.4.1-12.el8.noarch 2024-03-30T04:14:20.840 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:14:20.840 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-30T04:14:21.143 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-30T04:14:21.433 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-03-30T04:14:21.632 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-03-30T04:14:22.057 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:55 ago on Sat 30 Mar 2024 04:12:27 AM UTC. 2024-03-30T04:14:22.424 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-03-30T04:14:22.815 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-03-30T04:14:23.065 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-03-30T04:14:23.833 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-30T04:14:23.834 INFO:teuthology.orchestra.run.smithi135.stdout:============================================================================================ 2024-03-30T04:14:23.834 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2024-03-30T04:14:23.834 INFO:teuthology.orchestra.run.smithi135.stdout:============================================================================================ 2024-03-30T04:14:23.834 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-03-30T04:14:23.834 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-797.g5c024335.el8 ceph-noarch 7.4 M 2024-03-30T04:14:23.834 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2024-03-30T04:14:23.834 INFO:teuthology.orchestra.run.smithi135.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-03-30T04:14:23.834 INFO:teuthology.orchestra.run.smithi135.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-03-30T04:14:23.834 INFO:teuthology.orchestra.run.smithi135.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-03-30T04:14:23.835 INFO:teuthology.orchestra.run.smithi135.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-03-30T04:14:23.835 INFO:teuthology.orchestra.run.smithi135.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-03-30T04:14:23.835 INFO:teuthology.orchestra.run.smithi135.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-03-30T04:14:23.835 INFO:teuthology.orchestra.run.smithi135.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-03-30T04:14:23.835 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:14:23.835 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-03-30T04:14:23.835 INFO:teuthology.orchestra.run.smithi135.stdout:============================================================================================ 2024-03-30T04:14:23.835 INFO:teuthology.orchestra.run.smithi135.stdout:Install 8 Packages 2024-03-30T04:14:23.835 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:14:23.836 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 36 M 2024-03-30T04:14:23.836 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 210 M 2024-03-30T04:14:23.836 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-03-30T04:14:25.008 INFO:teuthology.orchestra.run.smithi135.stdout:(1/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 4.6 MB/s | 4.9 MB 00:01 2024-03-30T04:14:25.100 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-03-30T04:14:25.167 INFO:teuthology.orchestra.run.smithi135.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 3.9 MB/s | 4.7 MB 00:01 2024-03-30T04:14:25.267 INFO:teuthology.orchestra.run.smithi135.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.2 MB/s | 225 kB 00:00 2024-03-30T04:14:25.334 INFO:teuthology.orchestra.run.smithi135.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 11 MB/s | 3.7 MB 00:00 2024-03-30T04:14:25.451 INFO:teuthology.orchestra.run.smithi135.stdout:(5/8): libgfortran-8.5.0-21.el8.x86_64.rpm 5.4 MB/s | 645 kB 00:00 2024-03-30T04:14:25.485 INFO:teuthology.orchestra.run.smithi135.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 5.1 MB/s | 172 kB 00:00 2024-03-30T04:14:25.694 INFO:teuthology.orchestra.run.smithi135.stdout:(7/8): ceph-mgr-diskprediction-local-17.2.7-797 4.3 MB/s | 7.4 MB 00:01 2024-03-30T04:14:26.466 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-03-30T04:14:26.689 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-797.g5c024335 8/8 2024-03-30T04:14:26.737 INFO:teuthology.orchestra.run.smithi135.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 9.6 MB/s | 14 MB 00:01 2024-03-30T04:14:26.738 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-03-30T04:14:26.739 INFO:teuthology.orchestra.run.smithi135.stdout:Total 12 MB/s | 36 MB 00:02 2024-03-30T04:14:27.053 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-03-30T04:14:27.074 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-797.g5c024335 8/8 2024-03-30T04:14:27.075 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-797.g5c024335 1/8 2024-03-30T04:14:27.075 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-03-30T04:14:27.075 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-03-30T04:14:27.075 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-03-30T04:14:27.075 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-03-30T04:14:27.075 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-03-30T04:14:27.075 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-03-30T04:14:27.075 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-03-30T04:14:27.076 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-03-30T04:14:27.520 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-03-30T04:14:27.523 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-03-30T04:14:27.525 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-03-30T04:14:27.525 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:14:27.525 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-30T04:14:27.526 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-diskprediction-local-2:17.2.7-797.g5c024335.el8.noarch 2024-03-30T04:14:27.526 INFO:teuthology.orchestra.run.smithi123.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-03-30T04:14:27.526 INFO:teuthology.orchestra.run.smithi123.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-03-30T04:14:27.526 INFO:teuthology.orchestra.run.smithi123.stdout: openblas-0.3.15-4.el8.x86_64 2024-03-30T04:14:27.526 INFO:teuthology.orchestra.run.smithi123.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-03-30T04:14:27.526 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-03-30T04:14:27.526 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-03-30T04:14:27.526 INFO:teuthology.orchestra.run.smithi123.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-03-30T04:14:27.526 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:14:27.526 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-30T04:14:27.707 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-mgr-rook 2024-03-30T04:14:28.101 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-03-30T04:14:28.279 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-03-30T04:14:28.437 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-03-30T04:14:28.604 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-03-30T04:14:28.623 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:57 ago on Sat 30 Mar 2024 04:12:31 AM UTC. 2024-03-30T04:14:29.251 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-03-30T04:14:29.479 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-03-30T04:14:30.155 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-03-30T04:14:30.309 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-30T04:14:30.311 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-30T04:14:30.311 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2024-03-30T04:14:30.311 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-30T04:14:30.311 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-30T04:14:30.312 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-rook noarch 2:17.2.7-797.g5c024335.el8 ceph-noarch 47 k 2024-03-30T04:14:30.312 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2024-03-30T04:14:30.312 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-03-30T04:14:30.312 INFO:teuthology.orchestra.run.smithi123.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-03-30T04:14:30.312 INFO:teuthology.orchestra.run.smithi123.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-03-30T04:14:30.312 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-03-30T04:14:30.312 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-03-30T04:14:30.312 INFO:teuthology.orchestra.run.smithi123.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-03-30T04:14:30.312 INFO:teuthology.orchestra.run.smithi123.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-03-30T04:14:30.312 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-03-30T04:14:30.312 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-03-30T04:14:30.312 INFO:teuthology.orchestra.run.smithi123.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-03-30T04:14:30.312 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-03-30T04:14:30.312 INFO:teuthology.orchestra.run.smithi123.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-03-30T04:14:30.313 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:14:30.313 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-30T04:14:30.313 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-30T04:14:30.313 INFO:teuthology.orchestra.run.smithi123.stdout:Install 13 Packages 2024-03-30T04:14:30.313 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:14:30.313 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 1.9 M 2024-03-30T04:14:30.313 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 25 M 2024-03-30T04:14:30.313 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-30T04:14:30.361 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-03-30T04:14:30.936 INFO:teuthology.orchestra.run.smithi123.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 109 kB/s | 27 kB 00:00 2024-03-30T04:14:30.953 INFO:teuthology.orchestra.run.smithi123.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 77 kB/s | 20 kB 00:00 2024-03-30T04:14:30.978 INFO:teuthology.orchestra.run.smithi123.stdout:(3/13): ceph-mgr-rook-17.2.7-797.g5c024335.el8. 166 kB/s | 47 kB 00:00 2024-03-30T04:14:31.070 INFO:teuthology.orchestra.run.smithi123.stdout:(4/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 1.7 MB/s | 155 kB 00:00 2024-03-30T04:14:31.104 INFO:teuthology.orchestra.run.smithi123.stdout:(5/13): python3-requests-oauthlib-1.0.0-1.el8.n 1.2 MB/s | 43 kB 00:00 2024-03-30T04:14:31.157 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-03-30T04:14:31.196 INFO:teuthology.orchestra.run.smithi123.stdout:(6/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 522 kB/s | 126 kB 00:00 2024-03-30T04:14:31.238 INFO:teuthology.orchestra.run.smithi123.stdout:(7/13): python3-pyasn1-modules-0.3.7-6.el8.noar 387 kB/s | 110 kB 00:00 2024-03-30T04:14:31.497 INFO:teuthology.orchestra.run.smithi123.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 84 kB/s | 33 kB 00:00 2024-03-30T04:14:31.530 INFO:teuthology.orchestra.run.smithi123.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 49 kB/s | 16 kB 00:00 2024-03-30T04:14:31.544 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-03-30T04:14:31.680 INFO:teuthology.orchestra.run.smithi123.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 184 kB/s | 81 kB 00:00 2024-03-30T04:14:31.747 INFO:teuthology.orchestra.run.smithi123.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 296 kB/s | 64 kB 00:00 2024-03-30T04:14:31.805 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-03-30T04:14:31.806 INFO:teuthology.orchestra.run.smithi123.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 489 kB/s | 61 kB 00:00 2024-03-30T04:14:31.990 INFO:teuthology.orchestra.run.smithi123.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 2.4 MB/s | 1.2 MB 00:00 2024-03-30T04:14:31.992 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-30T04:14:31.993 INFO:teuthology.orchestra.run.smithi123.stdout:Total 1.1 MB/s | 1.9 MB 00:01 2024-03-30T04:14:32.115 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-30T04:14:32.135 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-30T04:14:32.135 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-30T04:14:32.244 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-30T04:14:32.247 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-30T04:14:32.839 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-30T04:14:33.107 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-03-30T04:14:33.336 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-03-30T04:14:33.587 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-03-30T04:14:33.813 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-03-30T04:14:33.831 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-03-30T04:14:34.013 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-03-30T04:14:34.292 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-03-30T04:14:34.614 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-03-30T04:14:34.855 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-03-30T04:14:35.179 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-03-30T04:14:35.394 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-797.g5c024335 8/8 2024-03-30T04:14:35.814 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-797.g5c024335 8/8 2024-03-30T04:14:35.814 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-797.g5c024335 1/8 2024-03-30T04:14:35.814 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-03-30T04:14:35.814 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-03-30T04:14:35.815 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-03-30T04:14:35.815 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-03-30T04:14:35.815 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-03-30T04:14:35.815 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-03-30T04:14:36.117 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-03-30T04:14:36.267 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-03-30T04:14:36.267 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:14:36.267 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-03-30T04:14:36.267 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-diskprediction-local-2:17.2.7-797.g5c024335.el8.noarch 2024-03-30T04:14:36.267 INFO:teuthology.orchestra.run.smithi135.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-03-30T04:14:36.267 INFO:teuthology.orchestra.run.smithi135.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-03-30T04:14:36.267 INFO:teuthology.orchestra.run.smithi135.stdout: openblas-0.3.15-4.el8.x86_64 2024-03-30T04:14:36.267 INFO:teuthology.orchestra.run.smithi135.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-03-30T04:14:36.267 INFO:teuthology.orchestra.run.smithi135.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-03-30T04:14:36.267 INFO:teuthology.orchestra.run.smithi135.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-03-30T04:14:36.267 INFO:teuthology.orchestra.run.smithi135.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-03-30T04:14:36.267 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:14:36.267 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-30T04:14:36.405 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-03-30T04:14:36.445 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-mgr-rook 2024-03-30T04:14:36.651 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-03-30T04:14:36.892 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-03-30T04:14:37.113 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-rook-2:17.2.7-797.g5c024335.el8.noarch 13/13 2024-03-30T04:14:37.364 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:02:10 ago on Sat 30 Mar 2024 04:12:27 AM UTC. 2024-03-30T04:14:37.467 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-797.g5c024335.el8.noarch 13/13 2024-03-30T04:14:37.467 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-rook-2:17.2.7-797.g5c024335.el8.noarch 1/13 2024-03-30T04:14:37.467 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-03-30T04:14:37.467 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-03-30T04:14:37.467 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-03-30T04:14:37.467 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-03-30T04:14:37.467 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-03-30T04:14:37.467 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-03-30T04:14:37.468 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-03-30T04:14:37.468 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-03-30T04:14:37.468 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-03-30T04:14:37.468 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-03-30T04:14:37.468 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-03-30T04:14:37.947 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-03-30T04:14:37.947 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:14:37.947 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-30T04:14:37.947 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-rook-2:17.2.7-797.g5c024335.el8.noarch 2024-03-30T04:14:37.947 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-03-30T04:14:37.948 INFO:teuthology.orchestra.run.smithi123.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-03-30T04:14:37.948 INFO:teuthology.orchestra.run.smithi123.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-03-30T04:14:37.948 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-03-30T04:14:37.948 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-03-30T04:14:37.948 INFO:teuthology.orchestra.run.smithi123.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-03-30T04:14:37.948 INFO:teuthology.orchestra.run.smithi123.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-03-30T04:14:37.948 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-03-30T04:14:37.948 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-03-30T04:14:37.948 INFO:teuthology.orchestra.run.smithi123.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-03-30T04:14:37.948 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rsa-4.9-2.el8.noarch 2024-03-30T04:14:37.948 INFO:teuthology.orchestra.run.smithi123.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-03-30T04:14:37.949 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:14:37.949 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-30T04:14:38.102 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-mgr-cephadm 2024-03-30T04:14:39.014 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:08 ago on Sat 30 Mar 2024 04:12:31 AM UTC. 2024-03-30T04:14:39.050 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-30T04:14:39.052 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-30T04:14:39.052 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2024-03-30T04:14:39.052 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-30T04:14:39.052 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-03-30T04:14:39.052 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-rook noarch 2:17.2.7-797.g5c024335.el8 ceph-noarch 47 k 2024-03-30T04:14:39.052 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2024-03-30T04:14:39.052 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-03-30T04:14:39.052 INFO:teuthology.orchestra.run.smithi135.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-03-30T04:14:39.053 INFO:teuthology.orchestra.run.smithi135.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-03-30T04:14:39.053 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-03-30T04:14:39.053 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-03-30T04:14:39.053 INFO:teuthology.orchestra.run.smithi135.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-03-30T04:14:39.053 INFO:teuthology.orchestra.run.smithi135.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-03-30T04:14:39.053 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-03-30T04:14:39.053 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-03-30T04:14:39.053 INFO:teuthology.orchestra.run.smithi135.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-03-30T04:14:39.053 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-03-30T04:14:39.053 INFO:teuthology.orchestra.run.smithi135.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-03-30T04:14:39.053 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:14:39.053 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-03-30T04:14:39.053 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-30T04:14:39.054 INFO:teuthology.orchestra.run.smithi135.stdout:Install 13 Packages 2024-03-30T04:14:39.054 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:14:39.054 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 1.9 M 2024-03-30T04:14:39.054 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 25 M 2024-03-30T04:14:39.054 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-03-30T04:14:39.378 INFO:teuthology.orchestra.run.smithi135.stdout:(1/13): ceph-mgr-rook-17.2.7-797.g5c024335.el8. 276 kB/s | 47 kB 00:00 2024-03-30T04:14:39.445 INFO:teuthology.orchestra.run.smithi135.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 111 kB/s | 27 kB 00:00 2024-03-30T04:14:39.469 INFO:teuthology.orchestra.run.smithi135.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 76 kB/s | 20 kB 00:00 2024-03-30T04:14:39.553 INFO:teuthology.orchestra.run.smithi135.stdout:(4/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 1.8 MB/s | 155 kB 00:00 2024-03-30T04:14:39.587 INFO:teuthology.orchestra.run.smithi135.stdout:(5/13): python3-requests-oauthlib-1.0.0-1.el8.n 1.3 MB/s | 43 kB 00:00 2024-03-30T04:14:39.712 INFO:teuthology.orchestra.run.smithi135.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 264 kB/s | 33 kB 00:00 2024-03-30T04:14:39.779 INFO:teuthology.orchestra.run.smithi135.stdout:(7/13): python3-certifi-2018.10.15-7.el8.noarch 244 kB/s | 16 kB 00:00 2024-03-30T04:14:39.805 INFO:teuthology.orchestra.run.smithi135.stdout:(8/13): python3-pyasn1-modules-0.3.7-6.el8.noar 329 kB/s | 110 kB 00:00 2024-03-30T04:14:39.830 INFO:teuthology.orchestra.run.smithi135.stdout:(9/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 279 kB/s | 126 kB 00:00 2024-03-30T04:14:39.897 INFO:teuthology.orchestra.run.smithi135.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 691 kB/s | 81 kB 00:00 2024-03-30T04:14:39.995 INFO:teuthology.orchestra.run.smithi135.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 630 kB/s | 61 kB 00:00 2024-03-30T04:14:40.022 INFO:teuthology.orchestra.run.smithi135.stdout:(12/13): python3-rsa-4.9-2.el8.noarch.rpm 334 kB/s | 64 kB 00:00 2024-03-30T04:14:40.147 INFO:teuthology.orchestra.run.smithi135.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 3.4 MB/s | 1.2 MB 00:00 2024-03-30T04:14:40.148 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-03-30T04:14:40.148 INFO:teuthology.orchestra.run.smithi135.stdout:Total 1.8 MB/s | 1.9 MB 00:01 2024-03-30T04:14:40.266 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-03-30T04:14:40.284 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-03-30T04:14:40.284 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-03-30T04:14:40.395 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-03-30T04:14:40.399 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-03-30T04:14:40.698 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-30T04:14:40.699 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-30T04:14:40.699 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2024-03-30T04:14:40.699 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-30T04:14:40.699 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-30T04:14:40.699 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-cephadm noarch 2:17.2.7-797.g5c024335.el8 ceph-noarch 123 k 2024-03-30T04:14:40.699 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2024-03-30T04:14:40.699 INFO:teuthology.orchestra.run.smithi123.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-03-30T04:14:40.699 INFO:teuthology.orchestra.run.smithi123.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-03-30T04:14:40.699 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-03-30T04:14:40.699 INFO:teuthology.orchestra.run.smithi123.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-03-30T04:14:40.699 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:14:40.700 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-30T04:14:40.700 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-30T04:14:40.700 INFO:teuthology.orchestra.run.smithi123.stdout:Install 5 Packages 2024-03-30T04:14:40.700 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:14:40.700 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 6.0 M 2024-03-30T04:14:40.700 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 26 M 2024-03-30T04:14:40.701 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-30T04:14:40.976 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-03-30T04:14:41.203 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-03-30T04:14:41.221 INFO:teuthology.orchestra.run.smithi123.stdout:(1/5): ceph-mgr-cephadm-17.2.7-797.g5c024335.el 406 kB/s | 123 kB 00:00 2024-03-30T04:14:41.407 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-03-30T04:14:41.413 INFO:teuthology.orchestra.run.smithi123.stdout:(2/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 2.4 MB/s | 476 kB 00:00 2024-03-30T04:14:41.446 INFO:teuthology.orchestra.run.smithi123.stdout:(3/5): python3-jinja2-2.10.1-4.el8.noarch.rpm 1.1 MB/s | 578 kB 00:00 2024-03-30T04:14:41.471 INFO:teuthology.orchestra.run.smithi123.stdout:(4/5): python3-natsort-7.1.1-2.el8.noarch.rpm 1.0 MB/s | 60 kB 00:00 2024-03-30T04:14:41.641 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-03-30T04:14:41.730 INFO:teuthology.orchestra.run.smithi123.stdout:(5/5): python3-babel-2.5.1-7.el8.noarch.rpm 5.9 MB/s | 4.8 MB 00:00 2024-03-30T04:14:41.731 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-30T04:14:41.731 INFO:teuthology.orchestra.run.smithi123.stdout:Total 5.8 MB/s | 6.0 MB 00:01 2024-03-30T04:14:41.806 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-30T04:14:41.820 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-30T04:14:41.820 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-30T04:14:41.910 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-03-30T04:14:41.942 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-30T04:14:41.946 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-30T04:14:42.119 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-03-30T04:14:42.230 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-30T04:14:42.373 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-03-30T04:14:42.601 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2024-03-30T04:14:42.695 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-03-30T04:14:42.961 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-03-30T04:14:43.667 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2024-03-30T04:14:44.188 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-03-30T04:14:44.199 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2024-03-30T04:14:44.492 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-03-30T04:14:44.504 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 4/5 2024-03-30T04:14:44.730 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-03-30T04:14:44.770 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-797.g5c024335.el8.noarch 5/5 2024-03-30T04:14:45.006 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-03-30T04:14:45.092 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-797.g5c024335.el8.noarch 5/5 2024-03-30T04:14:45.092 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-797.g5c024335.el8.noarch 1/5 2024-03-30T04:14:45.092 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2024-03-30T04:14:45.092 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 3/5 2024-03-30T04:14:45.092 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2024-03-30T04:14:45.228 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mgr-rook-2:17.2.7-797.g5c024335.el8.noarch 13/13 2024-03-30T04:14:45.470 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2024-03-30T04:14:45.470 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:14:45.471 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-30T04:14:45.471 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-cephadm-2:17.2.7-797.g5c024335.el8.noarch 2024-03-30T04:14:45.471 INFO:teuthology.orchestra.run.smithi123.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-03-30T04:14:45.471 INFO:teuthology.orchestra.run.smithi123.stdout: python3-babel-2.5.1-7.el8.noarch 2024-03-30T04:14:45.471 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jinja2-2.10.1-4.el8.noarch 2024-03-30T04:14:45.471 INFO:teuthology.orchestra.run.smithi123.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-03-30T04:14:45.471 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:14:45.471 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-30T04:14:45.590 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-797.g5c024335.el8.noarch 13/13 2024-03-30T04:14:45.591 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mgr-rook-2:17.2.7-797.g5c024335.el8.noarch 1/13 2024-03-30T04:14:45.591 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-03-30T04:14:45.591 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-03-30T04:14:45.591 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-03-30T04:14:45.591 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-03-30T04:14:45.591 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-03-30T04:14:45.591 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-03-30T04:14:45.591 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-03-30T04:14:45.591 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-03-30T04:14:45.591 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-03-30T04:14:45.591 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-03-30T04:14:45.591 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-03-30T04:14:45.626 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-fuse 2024-03-30T04:14:46.078 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-03-30T04:14:46.078 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:14:46.078 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-03-30T04:14:46.078 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-rook-2:17.2.7-797.g5c024335.el8.noarch 2024-03-30T04:14:46.078 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-03-30T04:14:46.078 INFO:teuthology.orchestra.run.smithi135.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-03-30T04:14:46.078 INFO:teuthology.orchestra.run.smithi135.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-03-30T04:14:46.078 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-03-30T04:14:46.079 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-03-30T04:14:46.079 INFO:teuthology.orchestra.run.smithi135.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-03-30T04:14:46.079 INFO:teuthology.orchestra.run.smithi135.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-03-30T04:14:46.079 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-03-30T04:14:46.079 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-03-30T04:14:46.079 INFO:teuthology.orchestra.run.smithi135.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-03-30T04:14:46.079 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rsa-4.9-2.el8.noarch 2024-03-30T04:14:46.079 INFO:teuthology.orchestra.run.smithi135.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-03-30T04:14:46.079 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:14:46.079 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-30T04:14:46.237 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-mgr-cephadm 2024-03-30T04:14:46.548 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:15 ago on Sat 30 Mar 2024 04:12:31 AM UTC. 2024-03-30T04:14:47.149 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:02:20 ago on Sat 30 Mar 2024 04:12:27 AM UTC. 2024-03-30T04:14:48.254 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-30T04:14:48.254 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-30T04:14:48.254 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2024-03-30T04:14:48.254 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-30T04:14:48.255 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-30T04:14:48.255 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-fuse x86_64 2:17.2.7-797.g5c024335.el8 ceph 876 k 2024-03-30T04:14:48.255 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2024-03-30T04:14:48.255 INFO:teuthology.orchestra.run.smithi123.stdout: fuse x86_64 2.9.7-19.el8 baseos 83 k 2024-03-30T04:14:48.255 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:14:48.255 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-30T04:14:48.255 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-30T04:14:48.255 INFO:teuthology.orchestra.run.smithi123.stdout:Install 2 Packages 2024-03-30T04:14:48.255 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:14:48.256 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 960 k 2024-03-30T04:14:48.256 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 3.1 M 2024-03-30T04:14:48.256 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-30T04:14:48.405 INFO:teuthology.orchestra.run.smithi123.stdout:(1/2): fuse-2.9.7-19.el8.x86_64.rpm 954 kB/s | 83 kB 00:00 2024-03-30T04:14:48.630 INFO:teuthology.orchestra.run.smithi123.stdout:(2/2): ceph-fuse-17.2.7-797.g5c024335.el8.x86_6 2.7 MB/s | 876 kB 00:00 2024-03-30T04:14:48.630 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-30T04:14:48.631 INFO:teuthology.orchestra.run.smithi123.stdout:Total 2.5 MB/s | 960 kB 00:00 2024-03-30T04:14:48.651 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-30T04:14:48.676 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-30T04:14:48.676 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-30T04:14:48.785 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-30T04:14:48.788 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-30T04:14:48.805 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-30T04:14:48.806 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-30T04:14:48.806 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2024-03-30T04:14:48.806 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-30T04:14:48.807 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-03-30T04:14:48.807 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-cephadm noarch 2:17.2.7-797.g5c024335.el8 ceph-noarch 123 k 2024-03-30T04:14:48.807 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2024-03-30T04:14:48.807 INFO:teuthology.orchestra.run.smithi135.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-03-30T04:14:48.807 INFO:teuthology.orchestra.run.smithi135.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-03-30T04:14:48.807 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-03-30T04:14:48.807 INFO:teuthology.orchestra.run.smithi135.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-03-30T04:14:48.807 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:14:48.807 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-03-30T04:14:48.807 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-30T04:14:48.807 INFO:teuthology.orchestra.run.smithi135.stdout:Install 5 Packages 2024-03-30T04:14:48.807 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:14:48.808 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 6.0 M 2024-03-30T04:14:48.808 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 26 M 2024-03-30T04:14:48.808 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-03-30T04:14:48.988 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-30T04:14:49.302 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : fuse-2.9.7-19.el8.x86_64 1/2 2024-03-30T04:14:49.367 INFO:teuthology.orchestra.run.smithi135.stdout:(1/5): ceph-mgr-cephadm-17.2.7-797.g5c024335.el 614 kB/s | 123 kB 00:00 2024-03-30T04:14:49.519 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-fuse-2:17.2.7-797.g5c024335.el8.x86_64 2/2 2024-03-30T04:14:49.667 INFO:teuthology.orchestra.run.smithi135.stdout:(2/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 1.5 MB/s | 476 kB 00:00 2024-03-30T04:14:49.751 INFO:teuthology.orchestra.run.smithi135.stdout:(3/5): python3-natsort-7.1.1-2.el8.noarch.rpm 722 kB/s | 60 kB 00:00 2024-03-30T04:14:49.785 INFO:teuthology.orchestra.run.smithi135.stdout:(4/5): python3-jinja2-2.10.1-4.el8.noarch.rpm 935 kB/s | 578 kB 00:00 2024-03-30T04:14:50.093 INFO:teuthology.orchestra.run.smithi135.stdout:(5/5): python3-babel-2.5.1-7.el8.noarch.rpm 5.2 MB/s | 4.8 MB 00:00 2024-03-30T04:14:50.094 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-03-30T04:14:50.094 INFO:teuthology.orchestra.run.smithi135.stdout:Total 4.7 MB/s | 6.0 MB 00:01 2024-03-30T04:14:50.132 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-fuse-2:17.2.7-797.g5c024335.el8.x86_64 2/2 2024-03-30T04:14:50.132 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-fuse-2:17.2.7-797.g5c024335.el8.x86_64 1/2 2024-03-30T04:14:50.168 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-03-30T04:14:50.183 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-03-30T04:14:50.183 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-03-30T04:14:50.314 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-03-30T04:14:50.318 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-03-30T04:14:50.564 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : fuse-2.9.7-19.el8.x86_64 2/2 2024-03-30T04:14:50.564 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:14:50.564 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-30T04:14:50.564 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-fuse-2:17.2.7-797.g5c024335.el8.x86_64 fuse-2.9.7-19.el8.x86_64 2024-03-30T04:14:50.564 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:14:50.565 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-30T04:14:50.612 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-03-30T04:14:50.799 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install librados-devel 2024-03-30T04:14:50.970 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2024-03-30T04:14:51.725 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:20 ago on Sat 30 Mar 2024 04:12:31 AM UTC. 2024-03-30T04:14:52.077 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2024-03-30T04:14:52.882 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2024-03-30T04:14:53.157 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 4/5 2024-03-30T04:14:53.367 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-797.g5c024335.el8.noarch 5/5 2024-03-30T04:14:53.426 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-30T04:14:53.427 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-30T04:14:53.427 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repo Size 2024-03-30T04:14:53.427 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-30T04:14:53.427 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-30T04:14:53.427 INFO:teuthology.orchestra.run.smithi123.stdout: librados-devel x86_64 2:17.2.7-797.g5c024335.el8 ceph 130 k 2024-03-30T04:14:53.427 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:14:53.427 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-30T04:14:53.428 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-30T04:14:53.428 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2024-03-30T04:14:53.428 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:14:53.428 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 130 k 2024-03-30T04:14:53.428 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 464 k 2024-03-30T04:14:53.428 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-30T04:14:53.750 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-797.g5c024335.el8.noarch 5/5 2024-03-30T04:14:53.751 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-797.g5c024335.el8.noarch 1/5 2024-03-30T04:14:53.751 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2024-03-30T04:14:53.751 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 3/5 2024-03-30T04:14:53.751 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2024-03-30T04:14:54.183 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2024-03-30T04:14:54.183 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:14:54.183 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-03-30T04:14:54.183 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-cephadm-2:17.2.7-797.g5c024335.el8.noarch 2024-03-30T04:14:54.184 INFO:teuthology.orchestra.run.smithi135.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-03-30T04:14:54.184 INFO:teuthology.orchestra.run.smithi135.stdout: python3-babel-2.5.1-7.el8.noarch 2024-03-30T04:14:54.184 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jinja2-2.10.1-4.el8.noarch 2024-03-30T04:14:54.184 INFO:teuthology.orchestra.run.smithi135.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-03-30T04:14:54.184 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:14:54.184 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-30T04:14:54.348 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-fuse 2024-03-30T04:14:54.384 INFO:teuthology.orchestra.run.smithi123.stdout:librados-devel-17.2.7-797.g5c024335.el8.x86_64. 136 kB/s | 130 kB 00:00 2024-03-30T04:14:54.385 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-30T04:14:54.385 INFO:teuthology.orchestra.run.smithi123.stdout:Total 136 kB/s | 130 kB 00:00 2024-03-30T04:14:54.385 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-30T04:14:54.393 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-30T04:14:54.394 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-30T04:14:54.455 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-30T04:14:54.458 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-30T04:14:54.792 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-30T04:14:55.051 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librados-devel-2:17.2.7-797.g5c024335.el8.x86_64 1/1 2024-03-30T04:14:55.266 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:02:28 ago on Sat 30 Mar 2024 04:12:27 AM UTC. 2024-03-30T04:14:55.421 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librados-devel-2:17.2.7-797.g5c024335.el8.x86_64 1/1 2024-03-30T04:14:55.838 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librados-devel-2:17.2.7-797.g5c024335.el8.x86_64 1/1 2024-03-30T04:14:55.838 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:14:55.838 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-30T04:14:55.838 INFO:teuthology.orchestra.run.smithi123.stdout: librados-devel-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:14:55.839 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:14:55.839 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-30T04:14:56.105 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install libcephfs2 2024-03-30T04:14:56.946 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-30T04:14:56.946 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-30T04:14:56.947 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2024-03-30T04:14:56.947 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-30T04:14:56.947 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-03-30T04:14:56.947 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-fuse x86_64 2:17.2.7-797.g5c024335.el8 ceph 876 k 2024-03-30T04:14:56.947 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2024-03-30T04:14:56.947 INFO:teuthology.orchestra.run.smithi135.stdout: fuse x86_64 2.9.7-19.el8 baseos 83 k 2024-03-30T04:14:56.947 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:14:56.947 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-03-30T04:14:56.947 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-30T04:14:56.947 INFO:teuthology.orchestra.run.smithi135.stdout:Install 2 Packages 2024-03-30T04:14:56.947 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:14:56.948 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 960 k 2024-03-30T04:14:56.948 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 3.1 M 2024-03-30T04:14:56.948 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-03-30T04:14:57.034 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:26 ago on Sat 30 Mar 2024 04:12:31 AM UTC. 2024-03-30T04:14:57.227 INFO:teuthology.orchestra.run.smithi135.stdout:(1/2): fuse-2.9.7-19.el8.x86_64.rpm 968 kB/s | 83 kB 00:00 2024-03-30T04:14:57.427 INFO:teuthology.orchestra.run.smithi135.stdout:(2/2): ceph-fuse-17.2.7-797.g5c024335.el8.x86_6 3.0 MB/s | 876 kB 00:00 2024-03-30T04:14:57.428 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-03-30T04:14:57.428 INFO:teuthology.orchestra.run.smithi135.stdout:Total 2.0 MB/s | 960 kB 00:00 2024-03-30T04:14:57.450 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-03-30T04:14:57.475 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-03-30T04:14:57.475 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-03-30T04:14:57.585 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-03-30T04:14:57.589 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-03-30T04:14:57.795 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-03-30T04:14:58.083 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : fuse-2.9.7-19.el8.x86_64 1/2 2024-03-30T04:14:58.340 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-fuse-2:17.2.7-797.g5c024335.el8.x86_64 2/2 2024-03-30T04:14:58.778 INFO:teuthology.orchestra.run.smithi123.stdout:Package libcephfs2-2:17.2.7-797.g5c024335.el8.x86_64 is already installed. 2024-03-30T04:14:58.829 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-30T04:14:58.830 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2024-03-30T04:14:58.831 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-30T04:14:58.924 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-fuse-2:17.2.7-797.g5c024335.el8.x86_64 2/2 2024-03-30T04:14:58.924 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-fuse-2:17.2.7-797.g5c024335.el8.x86_64 1/2 2024-03-30T04:14:58.956 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install libcephfs-devel 2024-03-30T04:14:59.408 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : fuse-2.9.7-19.el8.x86_64 2/2 2024-03-30T04:14:59.408 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:14:59.408 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-03-30T04:14:59.408 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-fuse-2:17.2.7-797.g5c024335.el8.x86_64 fuse-2.9.7-19.el8.x86_64 2024-03-30T04:14:59.408 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:14:59.408 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-30T04:14:59.631 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install librados-devel 2024-03-30T04:14:59.897 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:28 ago on Sat 30 Mar 2024 04:12:31 AM UTC. 2024-03-30T04:15:00.568 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:02:33 ago on Sat 30 Mar 2024 04:12:27 AM UTC. 2024-03-30T04:15:01.615 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-30T04:15:01.616 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-30T04:15:01.616 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repo Size 2024-03-30T04:15:01.616 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-30T04:15:01.616 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-30T04:15:01.616 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs-devel x86_64 2:17.2.7-797.g5c024335.el8 ceph 25 k 2024-03-30T04:15:01.616 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:15:01.616 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-30T04:15:01.616 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-30T04:15:01.617 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2024-03-30T04:15:01.617 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:15:01.617 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 25 k 2024-03-30T04:15:01.617 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 109 k 2024-03-30T04:15:01.617 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-30T04:15:01.833 INFO:teuthology.orchestra.run.smithi123.stdout:libcephfs-devel-17.2.7-797.g5c024335.el8.x86_64 115 kB/s | 25 kB 00:00 2024-03-30T04:15:01.834 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-30T04:15:01.834 INFO:teuthology.orchestra.run.smithi123.stdout:Total 114 kB/s | 25 kB 00:00 2024-03-30T04:15:01.834 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-30T04:15:01.839 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-30T04:15:01.839 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-30T04:15:01.843 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-30T04:15:01.846 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-30T04:15:01.898 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-30T04:15:02.091 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libcephfs-devel-2:17.2.7-797.g5c024335.el8.x86_64 1/1 2024-03-30T04:15:02.241 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-30T04:15:02.242 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-30T04:15:02.242 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2024-03-30T04:15:02.242 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-30T04:15:02.242 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-03-30T04:15:02.242 INFO:teuthology.orchestra.run.smithi135.stdout: librados-devel x86_64 2:17.2.7-797.g5c024335.el8 ceph 130 k 2024-03-30T04:15:02.243 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:15:02.243 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-03-30T04:15:02.243 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-30T04:15:02.243 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2024-03-30T04:15:02.243 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:15:02.243 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 130 k 2024-03-30T04:15:02.243 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 464 k 2024-03-30T04:15:02.243 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-03-30T04:15:02.385 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-797.g5c024335.el8.x86_64 1/1 2024-03-30T04:15:02.767 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libcephfs-devel-2:17.2.7-797.g5c024335.el8.x86_64 1/1 2024-03-30T04:15:02.768 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:15:02.768 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-30T04:15:02.768 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs-devel-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:15:02.768 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:15:02.768 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-30T04:15:02.920 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install librados2 2024-03-30T04:15:03.309 INFO:teuthology.orchestra.run.smithi135.stdout:librados-devel-17.2.7-797.g5c024335.el8.x86_64. 122 kB/s | 130 kB 00:01 2024-03-30T04:15:03.310 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-03-30T04:15:03.310 INFO:teuthology.orchestra.run.smithi135.stdout:Total 122 kB/s | 130 kB 00:01 2024-03-30T04:15:03.311 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-03-30T04:15:03.319 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-03-30T04:15:03.319 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-03-30T04:15:03.382 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-03-30T04:15:03.385 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-03-30T04:15:03.509 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-03-30T04:15:03.755 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : librados-devel-2:17.2.7-797.g5c024335.el8.x86_64 1/1 2024-03-30T04:15:03.831 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:32 ago on Sat 30 Mar 2024 04:12:31 AM UTC. 2024-03-30T04:15:04.139 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librados-devel-2:17.2.7-797.g5c024335.el8.x86_64 1/1 2024-03-30T04:15:04.564 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librados-devel-2:17.2.7-797.g5c024335.el8.x86_64 1/1 2024-03-30T04:15:04.565 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:15:04.565 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-03-30T04:15:04.565 INFO:teuthology.orchestra.run.smithi135.stdout: librados-devel-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:15:04.565 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:15:04.565 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-30T04:15:04.800 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install libcephfs2 2024-03-30T04:15:05.469 INFO:teuthology.orchestra.run.smithi123.stdout:Package librados2-2:17.2.7-797.g5c024335.el8.x86_64 is already installed. 2024-03-30T04:15:05.521 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-30T04:15:05.523 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2024-03-30T04:15:05.523 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-30T04:15:05.690 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install librbd1 2024-03-30T04:15:05.735 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:02:38 ago on Sat 30 Mar 2024 04:12:27 AM UTC. 2024-03-30T04:15:06.603 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:35 ago on Sat 30 Mar 2024 04:12:31 AM UTC. 2024-03-30T04:15:07.348 INFO:teuthology.orchestra.run.smithi135.stdout:Package libcephfs2-2:17.2.7-797.g5c024335.el8.x86_64 is already installed. 2024-03-30T04:15:07.400 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-30T04:15:07.402 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2024-03-30T04:15:07.402 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-30T04:15:07.526 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install libcephfs-devel 2024-03-30T04:15:08.250 INFO:teuthology.orchestra.run.smithi123.stdout:Package librbd1-2:17.2.7-797.g5c024335.el8.x86_64 is already installed. 2024-03-30T04:15:08.304 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-30T04:15:08.305 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2024-03-30T04:15:08.305 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-30T04:15:08.441 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install python3-rados 2024-03-30T04:15:08.458 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:02:41 ago on Sat 30 Mar 2024 04:12:27 AM UTC. 2024-03-30T04:15:09.362 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:38 ago on Sat 30 Mar 2024 04:12:31 AM UTC. 2024-03-30T04:15:10.178 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-30T04:15:10.178 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-30T04:15:10.178 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2024-03-30T04:15:10.179 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-30T04:15:10.179 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-03-30T04:15:10.179 INFO:teuthology.orchestra.run.smithi135.stdout: libcephfs-devel x86_64 2:17.2.7-797.g5c024335.el8 ceph 25 k 2024-03-30T04:15:10.179 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:15:10.179 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-03-30T04:15:10.179 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-30T04:15:10.179 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2024-03-30T04:15:10.179 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:15:10.180 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 25 k 2024-03-30T04:15:10.180 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 109 k 2024-03-30T04:15:10.180 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-03-30T04:15:10.567 INFO:teuthology.orchestra.run.smithi135.stdout:libcephfs-devel-17.2.7-797.g5c024335.el8.x86_64 64 kB/s | 25 kB 00:00 2024-03-30T04:15:10.567 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-03-30T04:15:10.568 INFO:teuthology.orchestra.run.smithi135.stdout:Total 64 kB/s | 25 kB 00:00 2024-03-30T04:15:10.568 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-03-30T04:15:10.573 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-03-30T04:15:10.573 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-03-30T04:15:10.577 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-03-30T04:15:10.581 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-03-30T04:15:10.637 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-03-30T04:15:10.846 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libcephfs-devel-2:17.2.7-797.g5c024335.el8.x86_64 1/1 2024-03-30T04:15:11.118 INFO:teuthology.orchestra.run.smithi123.stdout:Package python3-rados-2:17.2.7-797.g5c024335.el8.x86_64 is already installed. 2024-03-30T04:15:11.158 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-797.g5c024335.el8.x86_64 1/1 2024-03-30T04:15:11.170 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-30T04:15:11.171 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2024-03-30T04:15:11.172 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-30T04:15:11.291 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install python3-rgw 2024-03-30T04:15:11.556 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libcephfs-devel-2:17.2.7-797.g5c024335.el8.x86_64 1/1 2024-03-30T04:15:11.557 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:15:11.557 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-03-30T04:15:11.557 INFO:teuthology.orchestra.run.smithi135.stdout: libcephfs-devel-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:15:11.557 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:15:11.557 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-30T04:15:11.707 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install librados2 2024-03-30T04:15:12.203 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:41 ago on Sat 30 Mar 2024 04:12:31 AM UTC. 2024-03-30T04:15:12.620 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:02:45 ago on Sat 30 Mar 2024 04:12:27 AM UTC. 2024-03-30T04:15:13.836 INFO:teuthology.orchestra.run.smithi123.stdout:Package python3-rgw-2:17.2.7-797.g5c024335.el8.x86_64 is already installed. 2024-03-30T04:15:13.888 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-30T04:15:13.890 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2024-03-30T04:15:13.890 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-30T04:15:14.004 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install python3-cephfs 2024-03-30T04:15:14.254 INFO:teuthology.orchestra.run.smithi135.stdout:Package librados2-2:17.2.7-797.g5c024335.el8.x86_64 is already installed. 2024-03-30T04:15:14.306 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-30T04:15:14.307 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2024-03-30T04:15:14.307 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-30T04:15:14.434 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install librbd1 2024-03-30T04:15:14.922 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:43 ago on Sat 30 Mar 2024 04:12:31 AM UTC. 2024-03-30T04:15:15.353 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:02:48 ago on Sat 30 Mar 2024 04:12:27 AM UTC. 2024-03-30T04:15:16.568 INFO:teuthology.orchestra.run.smithi123.stdout:Package python3-cephfs-2:17.2.7-797.g5c024335.el8.x86_64 is already installed. 2024-03-30T04:15:16.619 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-30T04:15:16.621 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2024-03-30T04:15:16.621 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-30T04:15:16.729 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install python3-rbd 2024-03-30T04:15:17.005 INFO:teuthology.orchestra.run.smithi135.stdout:Package librbd1-2:17.2.7-797.g5c024335.el8.x86_64 is already installed. 2024-03-30T04:15:17.056 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-30T04:15:17.058 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2024-03-30T04:15:17.058 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-30T04:15:17.167 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install python3-rados 2024-03-30T04:15:17.659 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:46 ago on Sat 30 Mar 2024 04:12:31 AM UTC. 2024-03-30T04:15:18.086 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:02:51 ago on Sat 30 Mar 2024 04:12:27 AM UTC. 2024-03-30T04:15:19.346 INFO:teuthology.orchestra.run.smithi123.stdout:Package python3-rbd-2:17.2.7-797.g5c024335.el8.x86_64 is already installed. 2024-03-30T04:15:19.401 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-30T04:15:19.402 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2024-03-30T04:15:19.402 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-30T04:15:19.528 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install rbd-fuse 2024-03-30T04:15:19.752 INFO:teuthology.orchestra.run.smithi135.stdout:Package python3-rados-2:17.2.7-797.g5c024335.el8.x86_64 is already installed. 2024-03-30T04:15:19.804 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-30T04:15:19.806 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2024-03-30T04:15:19.806 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-30T04:15:19.920 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install python3-rgw 2024-03-30T04:15:20.448 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:49 ago on Sat 30 Mar 2024 04:12:31 AM UTC. 2024-03-30T04:15:20.837 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:02:53 ago on Sat 30 Mar 2024 04:12:27 AM UTC. 2024-03-30T04:15:22.136 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-30T04:15:22.136 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-30T04:15:22.137 INFO:teuthology.orchestra.run.smithi123.stdout: Package Architecture Version Repository Size 2024-03-30T04:15:22.137 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-30T04:15:22.137 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-30T04:15:22.137 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-fuse x86_64 2:17.2.7-797.g5c024335.el8 ceph 90 k 2024-03-30T04:15:22.137 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:15:22.137 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-30T04:15:22.137 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-30T04:15:22.137 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2024-03-30T04:15:22.137 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:15:22.138 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 90 k 2024-03-30T04:15:22.138 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 268 k 2024-03-30T04:15:22.138 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-30T04:15:22.462 INFO:teuthology.orchestra.run.smithi135.stdout:Package python3-rgw-2:17.2.7-797.g5c024335.el8.x86_64 is already installed. 2024-03-30T04:15:22.513 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-30T04:15:22.515 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2024-03-30T04:15:22.515 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-30T04:15:22.646 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install python3-cephfs 2024-03-30T04:15:22.761 INFO:teuthology.orchestra.run.smithi123.stdout:rbd-fuse-17.2.7-797.g5c024335.el8.x86_64.rpm 145 kB/s | 90 kB 00:00 2024-03-30T04:15:22.762 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-30T04:15:22.763 INFO:teuthology.orchestra.run.smithi123.stdout:Total 145 kB/s | 90 kB 00:00 2024-03-30T04:15:22.763 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-30T04:15:22.775 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-30T04:15:22.775 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-30T04:15:22.833 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-30T04:15:22.837 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-30T04:15:22.953 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-30T04:15:23.179 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : rbd-fuse-2:17.2.7-797.g5c024335.el8.x86_64 1/1 2024-03-30T04:15:23.556 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: rbd-fuse-2:17.2.7-797.g5c024335.el8.x86_64 1/1 2024-03-30T04:15:23.563 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:02:56 ago on Sat 30 Mar 2024 04:12:27 AM UTC. 2024-03-30T04:15:24.030 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : rbd-fuse-2:17.2.7-797.g5c024335.el8.x86_64 1/1 2024-03-30T04:15:24.030 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:15:24.030 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-30T04:15:24.031 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-fuse-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:15:24.031 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:15:24.031 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-30T04:15:24.229 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install rbd-mirror 2024-03-30T04:15:25.148 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:54 ago on Sat 30 Mar 2024 04:12:31 AM UTC. 2024-03-30T04:15:25.190 INFO:teuthology.orchestra.run.smithi135.stdout:Package python3-cephfs-2:17.2.7-797.g5c024335.el8.x86_64 is already installed. 2024-03-30T04:15:25.242 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-30T04:15:25.243 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2024-03-30T04:15:25.243 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-30T04:15:25.355 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install python3-rbd 2024-03-30T04:15:26.279 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:02:59 ago on Sat 30 Mar 2024 04:12:27 AM UTC. 2024-03-30T04:15:26.825 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-30T04:15:26.826 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-30T04:15:26.826 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repo Size 2024-03-30T04:15:26.826 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-30T04:15:26.826 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-30T04:15:26.826 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-mirror x86_64 2:17.2.7-797.g5c024335.el8 ceph 3.5 M 2024-03-30T04:15:26.826 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:15:26.826 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-30T04:15:26.826 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-30T04:15:26.827 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2024-03-30T04:15:26.827 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:15:26.827 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 3.5 M 2024-03-30T04:15:26.827 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 16 M 2024-03-30T04:15:26.827 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-30T04:15:27.566 INFO:teuthology.orchestra.run.smithi123.stdout:rbd-mirror-17.2.7-797.g5c024335.el8.x86_64.rpm 4.7 MB/s | 3.5 MB 00:00 2024-03-30T04:15:27.567 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-30T04:15:27.568 INFO:teuthology.orchestra.run.smithi123.stdout:Total 4.7 MB/s | 3.5 MB 00:00 2024-03-30T04:15:27.568 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-30T04:15:27.582 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-30T04:15:27.583 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-30T04:15:27.652 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-30T04:15:27.655 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-30T04:15:27.907 INFO:teuthology.orchestra.run.smithi135.stdout:Package python3-rbd-2:17.2.7-797.g5c024335.el8.x86_64 is already installed. 2024-03-30T04:15:27.960 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-30T04:15:27.962 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2024-03-30T04:15:27.962 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-30T04:15:28.079 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install rbd-fuse 2024-03-30T04:15:28.138 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-30T04:15:28.330 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : rbd-mirror-2:17.2.7-797.g5c024335.el8.x86_64 1/1 2024-03-30T04:15:28.981 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: rbd-mirror-2:17.2.7-797.g5c024335.el8.x86_64 1/1 2024-03-30T04:15:28.993 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:03:01 ago on Sat 30 Mar 2024 04:12:27 AM UTC. 2024-03-30T04:15:29.408 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : rbd-mirror-2:17.2.7-797.g5c024335.el8.x86_64 1/1 2024-03-30T04:15:29.409 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:15:29.409 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-30T04:15:29.409 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-mirror-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:15:29.409 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:15:29.409 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-30T04:15:29.652 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install rbd-nbd 2024-03-30T04:15:30.582 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:59 ago on Sat 30 Mar 2024 04:12:31 AM UTC. 2024-03-30T04:15:30.696 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-30T04:15:30.696 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-30T04:15:30.696 INFO:teuthology.orchestra.run.smithi135.stdout: Package Architecture Version Repository Size 2024-03-30T04:15:30.697 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-30T04:15:30.697 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-03-30T04:15:30.697 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-fuse x86_64 2:17.2.7-797.g5c024335.el8 ceph 90 k 2024-03-30T04:15:30.697 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:15:30.697 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-03-30T04:15:30.697 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-30T04:15:30.697 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2024-03-30T04:15:30.697 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:15:30.698 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 90 k 2024-03-30T04:15:30.698 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 268 k 2024-03-30T04:15:30.698 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-03-30T04:15:30.891 INFO:teuthology.orchestra.run.smithi135.stdout:rbd-fuse-17.2.7-797.g5c024335.el8.x86_64.rpm 471 kB/s | 90 kB 00:00 2024-03-30T04:15:30.891 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-03-30T04:15:30.892 INFO:teuthology.orchestra.run.smithi135.stdout:Total 467 kB/s | 90 kB 00:00 2024-03-30T04:15:30.892 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-03-30T04:15:30.904 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-03-30T04:15:30.904 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-03-30T04:15:30.960 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-03-30T04:15:30.963 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-03-30T04:15:31.074 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-03-30T04:15:31.326 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : rbd-fuse-2:17.2.7-797.g5c024335.el8.x86_64 1/1 2024-03-30T04:15:31.700 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: rbd-fuse-2:17.2.7-797.g5c024335.el8.x86_64 1/1 2024-03-30T04:15:32.122 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : rbd-fuse-2:17.2.7-797.g5c024335.el8.x86_64 1/1 2024-03-30T04:15:32.122 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:15:32.123 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-03-30T04:15:32.123 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-fuse-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:15:32.123 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:15:32.123 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-30T04:15:32.281 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-03-30T04:15:32.282 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-30T04:15:32.282 INFO:teuthology.orchestra.run.smithi123.stdout: Package Architecture Version Repository Size 2024-03-30T04:15:32.282 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-30T04:15:32.282 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-03-30T04:15:32.282 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-nbd x86_64 2:17.2.7-797.g5c024335.el8 ceph 176 k 2024-03-30T04:15:32.282 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:15:32.282 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-03-30T04:15:32.282 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-03-30T04:15:32.283 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2024-03-30T04:15:32.283 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:15:32.283 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 176 k 2024-03-30T04:15:32.283 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 556 k 2024-03-30T04:15:32.283 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-03-30T04:15:32.376 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install rbd-mirror 2024-03-30T04:15:32.538 INFO:teuthology.orchestra.run.smithi123.stdout:rbd-nbd-17.2.7-797.g5c024335.el8.x86_64.rpm 691 kB/s | 176 kB 00:00 2024-03-30T04:15:32.538 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-30T04:15:32.539 INFO:teuthology.orchestra.run.smithi123.stdout:Total 687 kB/s | 176 kB 00:00 2024-03-30T04:15:32.539 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-03-30T04:15:32.552 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-03-30T04:15:32.552 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-03-30T04:15:32.609 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-03-30T04:15:32.612 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-03-30T04:15:32.737 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-03-30T04:15:32.927 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : rbd-nbd-2:17.2.7-797.g5c024335.el8.x86_64 1/1 2024-03-30T04:15:33.306 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: rbd-nbd-2:17.2.7-797.g5c024335.el8.x86_64 1/1 2024-03-30T04:15:33.318 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:03:06 ago on Sat 30 Mar 2024 04:12:27 AM UTC. 2024-03-30T04:15:33.730 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : rbd-nbd-2:17.2.7-797.g5c024335.el8.x86_64 1/1 2024-03-30T04:15:33.730 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:15:33.730 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-03-30T04:15:33.730 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-nbd-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:15:33.730 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:15:33.730 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-03-30T04:15:33.949 DEBUG:teuthology.parallel:result is None 2024-03-30T04:15:35.024 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-30T04:15:35.025 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-30T04:15:35.025 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2024-03-30T04:15:35.025 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-30T04:15:35.025 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-03-30T04:15:35.025 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-mirror x86_64 2:17.2.7-797.g5c024335.el8 ceph 3.5 M 2024-03-30T04:15:35.025 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:15:35.025 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-03-30T04:15:35.025 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-30T04:15:35.025 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2024-03-30T04:15:35.025 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:15:35.026 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 3.5 M 2024-03-30T04:15:35.026 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 16 M 2024-03-30T04:15:35.026 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-03-30T04:15:37.169 INFO:teuthology.orchestra.run.smithi135.stdout:rbd-mirror-17.2.7-797.g5c024335.el8.x86_64.rpm 1.6 MB/s | 3.5 MB 00:02 2024-03-30T04:15:37.170 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-03-30T04:15:37.170 INFO:teuthology.orchestra.run.smithi135.stdout:Total 1.6 MB/s | 3.5 MB 00:02 2024-03-30T04:15:37.171 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-03-30T04:15:37.186 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-03-30T04:15:37.186 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-03-30T04:15:37.260 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-03-30T04:15:37.263 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-03-30T04:15:37.747 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-03-30T04:15:37.977 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : rbd-mirror-2:17.2.7-797.g5c024335.el8.x86_64 1/1 2024-03-30T04:15:38.624 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: rbd-mirror-2:17.2.7-797.g5c024335.el8.x86_64 1/1 2024-03-30T04:15:39.045 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : rbd-mirror-2:17.2.7-797.g5c024335.el8.x86_64 1/1 2024-03-30T04:15:39.045 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:15:39.045 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-03-30T04:15:39.045 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-mirror-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:15:39.045 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:15:39.045 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-30T04:15:39.285 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install rbd-nbd 2024-03-30T04:15:40.211 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:03:13 ago on Sat 30 Mar 2024 04:12:27 AM UTC. 2024-03-30T04:15:42.016 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-30T04:15:42.016 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-30T04:15:42.017 INFO:teuthology.orchestra.run.smithi135.stdout: Package Architecture Version Repository Size 2024-03-30T04:15:42.017 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-30T04:15:42.017 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-03-30T04:15:42.017 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-nbd x86_64 2:17.2.7-797.g5c024335.el8 ceph 176 k 2024-03-30T04:15:42.017 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:15:42.017 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-03-30T04:15:42.017 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-30T04:15:42.017 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2024-03-30T04:15:42.017 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:15:42.018 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 176 k 2024-03-30T04:15:42.018 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 556 k 2024-03-30T04:15:42.018 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-03-30T04:15:42.223 INFO:teuthology.orchestra.run.smithi135.stdout:rbd-nbd-17.2.7-797.g5c024335.el8.x86_64.rpm 859 kB/s | 176 kB 00:00 2024-03-30T04:15:42.223 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-03-30T04:15:42.224 INFO:teuthology.orchestra.run.smithi135.stdout:Total 852 kB/s | 176 kB 00:00 2024-03-30T04:15:42.224 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-03-30T04:15:42.237 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-03-30T04:15:42.237 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-03-30T04:15:42.294 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-03-30T04:15:42.297 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-03-30T04:15:42.417 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-03-30T04:15:42.658 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : rbd-nbd-2:17.2.7-797.g5c024335.el8.x86_64 1/1 2024-03-30T04:15:43.050 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: rbd-nbd-2:17.2.7-797.g5c024335.el8.x86_64 1/1 2024-03-30T04:15:43.495 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : rbd-nbd-2:17.2.7-797.g5c024335.el8.x86_64 1/1 2024-03-30T04:15:43.495 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:15:43.495 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-03-30T04:15:43.495 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-nbd-2:17.2.7-797.g5c024335.el8.x86_64 2024-03-30T04:15:43.495 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:15:43.495 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-30T04:15:43.781 DEBUG:teuthology.parallel:result is None 2024-03-30T04:15:43.781 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-30T04:15:43.782 INFO:teuthology.packaging:ref: None 2024-03-30T04:15:43.782 INFO:teuthology.packaging:tag: None 2024-03-30T04:15:43.782 INFO:teuthology.packaging:branch: quincy 2024-03-30T04:15:43.782 INFO:teuthology.packaging:sha1: 470100a4ccbcf7769052bf09a5413881ccf1b97f 2024-03-30T04:15:43.782 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=quincy 2024-03-30T04:15:44.136 DEBUG:teuthology.orchestra.run.smithi123:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-30T04:15:44.196 INFO:teuthology.orchestra.run.smithi123.stdout:17.2.7-797.g5c024335.el8 2024-03-30T04:15:44.197 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-797.g5c024335.el8 2024-03-30T04:15:44.197 INFO:teuthology.task.install:The correct ceph version 17.2.7-797.g5c024335 is installed. 2024-03-30T04:15:44.199 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-30T04:15:44.199 INFO:teuthology.packaging:ref: None 2024-03-30T04:15:44.199 INFO:teuthology.packaging:tag: None 2024-03-30T04:15:44.199 INFO:teuthology.packaging:branch: quincy 2024-03-30T04:15:44.199 INFO:teuthology.packaging:sha1: 470100a4ccbcf7769052bf09a5413881ccf1b97f 2024-03-30T04:15:44.199 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=quincy 2024-03-30T04:15:44.547 DEBUG:teuthology.orchestra.run.smithi135:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-30T04:15:44.601 INFO:teuthology.orchestra.run.smithi135.stdout:17.2.7-797.g5c024335.el8 2024-03-30T04:15:44.601 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-797.g5c024335.el8 2024-03-30T04:15:44.601 INFO:teuthology.task.install:The correct ceph version 17.2.7-797.g5c024335 is installed. 2024-03-30T04:15:44.603 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-30T04:15:44.603 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-30T04:15:44.603 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-30T04:15:44.677 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-30T04:15:44.677 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-30T04:15:44.743 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-30T04:15:44.743 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-30T04:15:44.743 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/usr/bin/daemon-helper 2024-03-30T04:15:44.810 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-30T04:15:44.916 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-30T04:15:44.916 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/usr/bin/daemon-helper 2024-03-30T04:15:44.981 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-30T04:15:45.086 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-30T04:15:45.087 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-30T04:15:45.087 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-30T04:15:45.152 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-30T04:15:45.257 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-30T04:15:45.258 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-30T04:15:45.323 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-30T04:15:45.429 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-30T04:15:45.429 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-30T04:15:45.429 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/usr/bin/stdin-killer 2024-03-30T04:15:45.495 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-30T04:15:45.601 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-30T04:15:45.601 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/usr/bin/stdin-killer 2024-03-30T04:15:45.667 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-30T04:15:45.773 INFO:teuthology.run_tasks:Running task print... 2024-03-30T04:15:45.783 INFO:teuthology.task.print:**** done install task... 2024-03-30T04:15:45.783 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-30T04:15:45.957 INFO:tasks.cephadm:Config: {'cephadm_branch': 'quincy', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}}, 'image': 'quay.ceph.io/ceph-ci/ceph:quincy', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'OSD_DOWN', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '470100a4ccbcf7769052bf09a5413881ccf1b97f'} 2024-03-30T04:15:45.957 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:quincy 2024-03-30T04:15:45.958 INFO:tasks.cephadm:Cluster fsid is 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 2024-03-30T04:15:45.958 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-30T04:15:45.958 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-03-30T04:15:45.958 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi123': '172.21.15.123', 'mon.smithi135': '172.21.15.135'} 2024-03-30T04:15:45.958 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-30T04:15:45.958 DEBUG:teuthology.orchestra.run.smithi123:> sudo hostname $(hostname -s) 2024-03-30T04:15:46.027 DEBUG:teuthology.orchestra.run.smithi135:> sudo hostname $(hostname -s) 2024-03-30T04:15:46.095 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref quincy)... 2024-03-30T04:15:46.095 DEBUG:teuthology.orchestra.run.smithi123:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-30T04:15:46.321 INFO:teuthology.orchestra.run.smithi123.stdout:-rw-rw-r--. 1 ubuntu ubuntu 375153 Mar 30 04:15 /home/ubuntu/cephtest/cephadm 2024-03-30T04:15:46.321 DEBUG:teuthology.orchestra.run.smithi135:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-30T04:15:46.498 INFO:teuthology.orchestra.run.smithi135.stdout:-rw-rw-r--. 1 ubuntu ubuntu 375153 Mar 30 04:15 /home/ubuntu/cephtest/cephadm 2024-03-30T04:15:46.499 DEBUG:teuthology.orchestra.run.smithi123:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-30T04:15:46.546 DEBUG:teuthology.orchestra.run.smithi135:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-30T04:15:46.603 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:quincy on all hosts... 2024-03-30T04:15:46.603 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-03-30T04:15:46.607 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-03-30T04:15:47.933 INFO:teuthology.orchestra.run.smithi123.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-03-30T04:15:48.063 INFO:teuthology.orchestra.run.smithi135.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-03-30T04:16:19.571 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:16:19.572 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_version": "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)", 2024-03-30T04:16:19.572 INFO:teuthology.orchestra.run.smithi123.stdout: "image_id": "f77085f6aab7aa687d22e7c004be5664ddf2da91f3bf5dc0f03eedb3890924d7", 2024-03-30T04:16:19.572 INFO:teuthology.orchestra.run.smithi123.stdout: "repo_digests": [ 2024-03-30T04:16:19.572 INFO:teuthology.orchestra.run.smithi123.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:18478aca52cd73a1126b6315d0e456c4e71d2faef3c5dfce416f19af0d85aef8" 2024-03-30T04:16:19.572 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-03-30T04:16:19.572 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:16:19.677 INFO:teuthology.orchestra.run.smithi135.stdout:{ 2024-03-30T04:16:19.678 INFO:teuthology.orchestra.run.smithi135.stdout: "ceph_version": "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)", 2024-03-30T04:16:19.678 INFO:teuthology.orchestra.run.smithi135.stdout: "image_id": "f77085f6aab7aa687d22e7c004be5664ddf2da91f3bf5dc0f03eedb3890924d7", 2024-03-30T04:16:19.678 INFO:teuthology.orchestra.run.smithi135.stdout: "repo_digests": [ 2024-03-30T04:16:19.678 INFO:teuthology.orchestra.run.smithi135.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:18478aca52cd73a1126b6315d0e456c4e71d2faef3c5dfce416f19af0d85aef8" 2024-03-30T04:16:19.678 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-03-30T04:16:19.678 INFO:teuthology.orchestra.run.smithi135.stdout:} 2024-03-30T04:16:19.703 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /etc/ceph 2024-03-30T04:16:19.773 DEBUG:teuthology.orchestra.run.smithi135:> sudo mkdir -p /etc/ceph 2024-03-30T04:16:19.842 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 777 /etc/ceph 2024-03-30T04:16:19.908 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod 777 /etc/ceph 2024-03-30T04:16:19.976 INFO:tasks.cephadm:Writing seed config... 2024-03-30T04:16:19.977 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-03-30T04:16:19.977 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-03-30T04:16:19.977 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-03-30T04:16:19.977 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-03-30T04:16:19.977 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-03-30T04:16:19.978 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-30T04:16:19.978 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-30T04:16:19.978 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-03-30T04:16:19.978 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-03-30T04:16:19.978 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-30T04:16:19.978 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-30T04:16:19.978 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-03-30T04:16:19.978 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-30T04:16:19.978 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-30T04:16:19.978 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-30T04:16:19.978 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-30T04:16:19.978 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-30T04:16:19.978 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-30T04:16:19.979 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-30T04:16:19.979 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-30T04:16:19.979 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-30T04:16:19.979 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-30T04:16:19.979 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-30T04:16:19.979 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-03-30T04:16:19.979 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-30T04:16:19.979 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-30T04:16:19.979 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-30T04:16:19.979 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-30T04:16:19.979 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-30T04:16:19.979 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-30T04:16:19.979 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-30T04:16:19.979 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-30T04:16:19.979 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-30T04:16:19.980 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-30T04:16:19.980 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-30T04:16:19.980 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-30T04:16:19.980 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-30T04:16:19.980 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-30T04:16:19.980 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-30T04:16:19.980 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-30T04:16:19.981 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-30T04:16:19.981 DEBUG:teuthology.orchestra.run.smithi123:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-30T04:16:20.025 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-30T04:16:20.025 DEBUG:teuthology.orchestra.run.smithi123:mon.smithi123> sudo journalctl -f -n 0 -u ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4@mon.smithi123.service 2024-03-30T04:16:20.070 INFO:tasks.cephadm:Bootstrapping... 2024-03-30T04:16:20.070 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy -v bootstrap --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.123 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-30T04:16:20.135 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:-- Logs begin at Sat 2024-03-30 04:04:15 UTC. -- 2024-03-30T04:16:20.364 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-03-30T04:16:20.364 INFO:teuthology.orchestra.run.smithi123.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:quincy', '-v', 'bootstrap', '--fsid', '2e77a2be-ee4c-11ee-b647-cb9ed24678a4', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.123', '--skip-admin-label'] 2024-03-30T04:16:20.410 INFO:teuthology.orchestra.run.smithi123.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-30T04:16:20.410 INFO:teuthology.orchestra.run.smithi123.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-30T04:16:20.411 INFO:teuthology.orchestra.run.smithi123.stdout:Verifying podman|docker is present... 2024-03-30T04:16:20.454 INFO:teuthology.orchestra.run.smithi123.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-30T04:16:20.454 INFO:teuthology.orchestra.run.smithi123.stdout:Verifying lvm2 is present... 2024-03-30T04:16:20.455 INFO:teuthology.orchestra.run.smithi123.stdout:Verifying time synchronization is in place... 2024-03-30T04:16:20.468 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-30T04:16:20.468 INFO:teuthology.orchestra.run.smithi123.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-30T04:16:20.480 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-30T04:16:20.480 INFO:teuthology.orchestra.run.smithi123.stdout:systemctl: stdout inactive 2024-03-30T04:16:20.493 INFO:teuthology.orchestra.run.smithi123.stdout:systemctl: stdout enabled 2024-03-30T04:16:20.505 INFO:teuthology.orchestra.run.smithi123.stdout:systemctl: stdout active 2024-03-30T04:16:20.505 INFO:teuthology.orchestra.run.smithi123.stdout:Unit chronyd.service is enabled and running 2024-03-30T04:16:20.506 INFO:teuthology.orchestra.run.smithi123.stdout:Repeating the final host check... 2024-03-30T04:16:20.551 INFO:teuthology.orchestra.run.smithi123.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-30T04:16:20.551 INFO:teuthology.orchestra.run.smithi123.stdout:podman (/bin/podman) version 4.9.4 is present 2024-03-30T04:16:20.551 INFO:teuthology.orchestra.run.smithi123.stdout:systemctl is present 2024-03-30T04:16:20.552 INFO:teuthology.orchestra.run.smithi123.stdout:lvcreate is present 2024-03-30T04:16:20.564 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-30T04:16:20.564 INFO:teuthology.orchestra.run.smithi123.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-30T04:16:20.576 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-30T04:16:20.576 INFO:teuthology.orchestra.run.smithi123.stdout:systemctl: stdout inactive 2024-03-30T04:16:20.588 INFO:teuthology.orchestra.run.smithi123.stdout:systemctl: stdout enabled 2024-03-30T04:16:20.600 INFO:teuthology.orchestra.run.smithi123.stdout:systemctl: stdout active 2024-03-30T04:16:20.601 INFO:teuthology.orchestra.run.smithi123.stdout:Unit chronyd.service is enabled and running 2024-03-30T04:16:20.601 INFO:teuthology.orchestra.run.smithi123.stdout:Host looks OK 2024-03-30T04:16:20.601 INFO:teuthology.orchestra.run.smithi123.stdout:Cluster fsid: 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 2024-03-30T04:16:20.601 INFO:teuthology.orchestra.run.smithi123.stdout:Acquiring lock 140090759238544 on /run/cephadm/2e77a2be-ee4c-11ee-b647-cb9ed24678a4.lock 2024-03-30T04:16:20.601 INFO:teuthology.orchestra.run.smithi123.stdout:Lock 140090759238544 acquired on /run/cephadm/2e77a2be-ee4c-11ee-b647-cb9ed24678a4.lock 2024-03-30T04:16:20.601 INFO:teuthology.orchestra.run.smithi123.stdout:Verifying IP 172.21.15.123 port 3300 ... 2024-03-30T04:16:20.602 INFO:teuthology.orchestra.run.smithi123.stdout:Verifying IP 172.21.15.123 port 6789 ... 2024-03-30T04:16:20.602 INFO:teuthology.orchestra.run.smithi123.stdout:Base mon IP(s) is [172.21.15.123:3300, 172.21.15.123:6789], mon addrv is [v2:172.21.15.123:3300,v1:172.21.15.123:6789] 2024-03-30T04:16:20.610 INFO:teuthology.orchestra.run.smithi123.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.123 metric 100 2024-03-30T04:16:20.610 INFO:teuthology.orchestra.run.smithi123.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.123 metric 100 2024-03-30T04:16:20.616 INFO:teuthology.orchestra.run.smithi123.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-30T04:16:20.617 INFO:teuthology.orchestra.run.smithi123.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-30T04:16:20.617 INFO:teuthology.orchestra.run.smithi123.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1798sec hoplimit 64 pref medium 2024-03-30T04:16:20.623 INFO:teuthology.orchestra.run.smithi123.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-30T04:16:20.623 INFO:teuthology.orchestra.run.smithi123.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-30T04:16:20.623 INFO:teuthology.orchestra.run.smithi123.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-30T04:16:20.623 INFO:teuthology.orchestra.run.smithi123.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-30T04:16:20.623 INFO:teuthology.orchestra.run.smithi123.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d3f1/64 scope link 2024-03-30T04:16:20.623 INFO:teuthology.orchestra.run.smithi123.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-30T04:16:20.624 INFO:teuthology.orchestra.run.smithi123.stdout:Mon IP `172.21.15.123` is in CIDR network `172.21.0.0/20` 2024-03-30T04:16:20.624 INFO:teuthology.orchestra.run.smithi123.stdout:Mon IP `172.21.15.123` is in CIDR network `172.21.0.0/20` 2024-03-30T04:16:20.624 INFO:teuthology.orchestra.run.smithi123.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-30T04:16:20.625 INFO:teuthology.orchestra.run.smithi123.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-30T04:16:20.626 INFO:teuthology.orchestra.run.smithi123.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-03-30T04:16:21.115 INFO:teuthology.orchestra.run.smithi123.stdout:/bin/podman: stdout f77085f6aab7aa687d22e7c004be5664ddf2da91f3bf5dc0f03eedb3890924d7 2024-03-30T04:16:21.115 INFO:teuthology.orchestra.run.smithi123.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:quincy... 2024-03-30T04:16:21.115 INFO:teuthology.orchestra.run.smithi123.stdout:/bin/podman: stderr Getting image source signatures 2024-03-30T04:16:21.115 INFO:teuthology.orchestra.run.smithi123.stdout:/bin/podman: stderr Copying blob sha256:c94517a95f26cae013a50df29eca9563986a9ec85d7ba9e880fbb8fc5c30025e 2024-03-30T04:16:21.116 INFO:teuthology.orchestra.run.smithi123.stdout:/bin/podman: stderr Copying blob sha256:11da15500eca1f826c328eb3265caa30b0a3730510e2a32023084d6fe9cee20c 2024-03-30T04:16:21.116 INFO:teuthology.orchestra.run.smithi123.stdout:/bin/podman: stderr Copying config sha256:f77085f6aab7aa687d22e7c004be5664ddf2da91f3bf5dc0f03eedb3890924d7 2024-03-30T04:16:21.116 INFO:teuthology.orchestra.run.smithi123.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-30T04:16:22.254 INFO:teuthology.orchestra.run.smithi123.stdout:ceph: stdout ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable) 2024-03-30T04:16:22.254 INFO:teuthology.orchestra.run.smithi123.stdout:Ceph version: ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable) 2024-03-30T04:16:22.254 INFO:teuthology.orchestra.run.smithi123.stdout:Extracting ceph user uid/gid from container image... 2024-03-30T04:16:23.281 INFO:teuthology.orchestra.run.smithi123.stdout:stat: stdout 167 167 2024-03-30T04:16:23.281 INFO:teuthology.orchestra.run.smithi123.stdout:Creating initial keys... 2024-03-30T04:16:24.279 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph-authtool: stdout AQAXkgdmf/tPKRAAhL/w7cZ2Eb1A6ZSifxxCUQ== 2024-03-30T04:16:25.318 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph-authtool: stdout AQAYkgdmvEogKBAAVZsWc8PnKdYP4PQH+Z6UYg== 2024-03-30T04:16:26.372 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph-authtool: stdout AQAZkgdmYZFlKhAASw/HaWmuKeoaMjXL1XQ86Q== 2024-03-30T04:16:26.373 INFO:teuthology.orchestra.run.smithi123.stdout:Creating initial monmap... 2024-03-30T04:16:27.529 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-30T04:16:27.530 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2024-03-30T04:16:27.530 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 2024-03-30T04:16:27.530 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-30T04:16:27.530 INFO:teuthology.orchestra.run.smithi123.stdout:monmaptool for smithi123 [v2:172.21.15.123:3300,v1:172.21.15.123:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-30T04:16:27.530 INFO:teuthology.orchestra.run.smithi123.stdout:setting min_mon_release = octopus 2024-03-30T04:16:27.530 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/monmaptool: set fsid to 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 2024-03-30T04:16:27.530 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-30T04:16:27.530 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:16:27.530 INFO:teuthology.orchestra.run.smithi123.stdout:Creating mon... 2024-03-30T04:16:29.278 INFO:teuthology.orchestra.run.smithi123.stdout:create mon.smithi123 on 2024-03-30T04:16:29.473 INFO:teuthology.orchestra.run.smithi123.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-03-30T04:16:29.649 INFO:teuthology.orchestra.run.smithi123.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-30T04:16:29.847 INFO:teuthology.orchestra.run.smithi123.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4.target → /etc/systemd/system/ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4.target. 2024-03-30T04:16:29.847 INFO:teuthology.orchestra.run.smithi123.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4.target → /etc/systemd/system/ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4.target. 2024-03-30T04:16:30.129 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4@mon.smithi123 2024-03-30T04:16:30.129 INFO:teuthology.orchestra.run.smithi123.stdout:systemctl: stderr Failed to reset failed state of unit ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4@mon.smithi123.service: Unit ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4@mon.smithi123.service not loaded. 2024-03-30T04:16:30.331 INFO:teuthology.orchestra.run.smithi123.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4.target.wants/ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4@mon.smithi123.service → /etc/systemd/system/ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4@.service. 2024-03-30T04:16:30.421 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:30 smithi123 systemd[1]: Starting Ceph mon.smithi123 for 2e77a2be-ee4c-11ee-b647-cb9ed24678a4... 2024-03-30T04:16:30.969 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-30T04:16:30.969 INFO:teuthology.orchestra.run.smithi123.stdout:systemctl: stdout disabled 2024-03-30T04:16:30.985 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-30T04:16:30.985 INFO:teuthology.orchestra.run.smithi123.stdout:systemctl: stdout inactive 2024-03-30T04:16:30.985 INFO:teuthology.orchestra.run.smithi123.stdout:firewalld.service is not enabled 2024-03-30T04:16:30.985 INFO:teuthology.orchestra.run.smithi123.stdout:Not possible to enable service . firewalld.service is not available 2024-03-30T04:16:30.986 INFO:teuthology.orchestra.run.smithi123.stdout:Waiting for mon to start... 2024-03-30T04:16:30.986 INFO:teuthology.orchestra.run.smithi123.stdout:Waiting for mon... 2024-03-30T04:16:31.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:30 smithi123 bash[34720]: 8d82484e35906c17d0964a04e4648f7ec70e0c482df9421427b5ff1581aefd7d 2024-03-30T04:16:31.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:30 smithi123 systemd[1]: Started Ceph mon.smithi123 for 2e77a2be-ee4c-11ee-b647-cb9ed24678a4. 2024-03-30T04:16:31.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:30 smithi123 ceph-mon[34758]: mkfs 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 2024-03-30T04:16:31.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:30 smithi123 ceph-mon[34758]: mon.smithi123 is new leader, mons smithi123 in quorum (ranks 0) 2024-03-30T04:16:32.205 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout cluster: 2024-03-30T04:16:32.211 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout id: 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 2024-03-30T04:16:32.211 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-30T04:16:32.211 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout 2024-03-30T04:16:32.211 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout services: 2024-03-30T04:16:32.211 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi123 (age 0.661612s) 2024-03-30T04:16:32.211 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-30T04:16:32.211 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-30T04:16:32.212 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout 2024-03-30T04:16:32.212 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout data: 2024-03-30T04:16:32.212 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-30T04:16:32.212 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-30T04:16:32.212 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-30T04:16:32.212 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout pgs: 2024-03-30T04:16:32.212 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout 2024-03-30T04:16:32.212 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:31.527+0000 7fc195600700 1 Processor -- start 2024-03-30T04:16:32.212 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:31.528+0000 7fc195600700 1 -- start start 2024-03-30T04:16:32.212 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:31.528+0000 7fc195600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc190102580 0x7fc190102990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:32.212 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:31.528+0000 7fc195600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc190102ed0 con 0x7fc190102580 2024-03-30T04:16:32.212 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:31.529+0000 7fc18ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc190102580 0x7fc190102990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:32.212 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:31.529+0000 7fc18ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc190102580 0x7fc190102990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33424/0 (socket says 172.21.15.123:33424) 2024-03-30T04:16:32.213 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:31.529+0000 7fc18ea00700 1 -- 172.21.15.123:0/2269413561 learned_addr learned my addr 172.21.15.123:0/2269413561 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:16:32.213 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:31.529+0000 7fc18ea00700 1 -- 172.21.15.123:0/2269413561 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc190103010 con 0x7fc190102580 2024-03-30T04:16:32.213 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:31.529+0000 7fc18ea00700 1 --2- 172.21.15.123:0/2269413561 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc190102580 0x7fc190102990 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fc1800098e0 tx=0x7fc180009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfc0a5c682879268 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:32.213 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:31.530+0000 7fc18d600700 1 -- 172.21.15.123:0/2269413561 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc18000b860 con 0x7fc190102580 2024-03-30T04:16:32.213 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:31.530+0000 7fc18d600700 1 -- 172.21.15.123:0/2269413561 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc18000b9c0 con 0x7fc190102580 2024-03-30T04:16:32.213 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:31.530+0000 7fc18d600700 1 -- 172.21.15.123:0/2269413561 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc18000bc90 con 0x7fc190102580 2024-03-30T04:16:32.213 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:31.530+0000 7fc195600700 1 -- 172.21.15.123:0/2269413561 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc190102580 msgr2=0x7fc190102990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:32.213 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:31.530+0000 7fc195600700 1 --2- 172.21.15.123:0/2269413561 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc190102580 0x7fc190102990 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fc1800098e0 tx=0x7fc180009250 comp rx=0 tx=0).stop 2024-03-30T04:16:32.213 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:31.530+0000 7fc195600700 1 -- 172.21.15.123:0/2269413561 shutdown_connections 2024-03-30T04:16:32.213 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:31.530+0000 7fc195600700 1 --2- 172.21.15.123:0/2269413561 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc190102580 0x7fc190102990 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:32.213 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:31.530+0000 7fc195600700 1 -- 172.21.15.123:0/2269413561 >> 172.21.15.123:0/2269413561 conn(0x7fc1900fdd00 msgr2=0x7fc190100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:32.213 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:31.530+0000 7fc195600700 1 -- 172.21.15.123:0/2269413561 shutdown_connections 2024-03-30T04:16:32.213 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:31.531+0000 7fc195600700 1 -- 172.21.15.123:0/2269413561 wait complete. 2024-03-30T04:16:32.214 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:31.531+0000 7fc195600700 1 Processor -- start 2024-03-30T04:16:32.214 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:31.531+0000 7fc195600700 1 -- start start 2024-03-30T04:16:32.214 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:31.532+0000 7fc195600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc190102580 0x7fc190078c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:32.214 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:31.532+0000 7fc195600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc190079190 con 0x7fc190102580 2024-03-30T04:16:32.214 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:31.532+0000 7fc18ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc190102580 0x7fc190078c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:32.214 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:31.532+0000 7fc18ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc190102580 0x7fc190078c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33426/0 (socket says 172.21.15.123:33426) 2024-03-30T04:16:32.214 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:31.532+0000 7fc18ea00700 1 -- 172.21.15.123:0/3476996922 learned_addr learned my addr 172.21.15.123:0/3476996922 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:16:32.214 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:31.533+0000 7fc18ea00700 1 -- 172.21.15.123:0/3476996922 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1800094f0 con 0x7fc190102580 2024-03-30T04:16:32.214 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:31.533+0000 7fc18ea00700 1 --2- 172.21.15.123:0/3476996922 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc190102580 0x7fc190078c50 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fc180003080 tx=0x7fc180016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:32.214 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:31.533+0000 7fc186a00700 1 -- 172.21.15.123:0/3476996922 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc18001c5c0 con 0x7fc190102580 2024-03-30T04:16:32.214 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:31.533+0000 7fc186a00700 1 -- 172.21.15.123:0/3476996922 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc18001c720 con 0x7fc190102580 2024-03-30T04:16:32.214 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:31.533+0000 7fc195600700 1 -- 172.21.15.123:0/3476996922 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc190077200 con 0x7fc190102580 2024-03-30T04:16:32.214 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:31.533+0000 7fc186a00700 1 -- 172.21.15.123:0/3476996922 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc18001ca30 con 0x7fc190102580 2024-03-30T04:16:32.215 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:31.533+0000 7fc195600700 1 -- 172.21.15.123:0/3476996922 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1900776f0 con 0x7fc190102580 2024-03-30T04:16:32.215 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:31.533+0000 7fc186a00700 1 -- 172.21.15.123:0/3476996922 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fc18001a450 con 0x7fc190102580 2024-03-30T04:16:32.215 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:31.533+0000 7fc186a00700 1 -- 172.21.15.123:0/3476996922 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fc18002e790 con 0x7fc190102580 2024-03-30T04:16:32.215 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:31.534+0000 7fc195600700 1 -- 172.21.15.123:0/3476996922 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc19004bed0 con 0x7fc190102580 2024-03-30T04:16:32.215 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:31.537+0000 7fc186a00700 1 -- 172.21.15.123:0/3476996922 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fc18001a890 con 0x7fc190102580 2024-03-30T04:16:32.215 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:31.608+0000 7fc195600700 1 -- 172.21.15.123:0/3476996922 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fc19010a130 con 0x7fc190102580 2024-03-30T04:16:32.215 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:31.608+0000 7fc186a00700 1 -- 172.21.15.123:0/3476996922 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7fc18001aa30 con 0x7fc190102580 2024-03-30T04:16:32.215 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:31.609+0000 7fc195600700 1 -- 172.21.15.123:0/3476996922 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc190102580 msgr2=0x7fc190078c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:32.215 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:31.609+0000 7fc195600700 1 --2- 172.21.15.123:0/3476996922 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc190102580 0x7fc190078c50 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fc180003080 tx=0x7fc180016000 comp rx=0 tx=0).stop 2024-03-30T04:16:32.215 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:31.609+0000 7fc195600700 1 -- 172.21.15.123:0/3476996922 shutdown_connections 2024-03-30T04:16:32.215 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:31.609+0000 7fc195600700 1 --2- 172.21.15.123:0/3476996922 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc190102580 0x7fc190078c50 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:32.215 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:31.609+0000 7fc195600700 1 -- 172.21.15.123:0/3476996922 >> 172.21.15.123:0/3476996922 conn(0x7fc1900fdd00 msgr2=0x7fc190104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:32.215 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:31.609+0000 7fc195600700 1 -- 172.21.15.123:0/3476996922 shutdown_connections 2024-03-30T04:16:32.216 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:31.609+0000 7fc195600700 1 -- 172.21.15.123:0/3476996922 wait complete. 2024-03-30T04:16:32.216 INFO:teuthology.orchestra.run.smithi123.stdout:mon is available 2024-03-30T04:16:32.216 INFO:teuthology.orchestra.run.smithi123.stdout:Assimilating anything we can from ceph.conf... 2024-03-30T04:16:32.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:31 smithi123 ceph-mon[34758]: mon.smithi123 is new leader, mons smithi123 in quorum (ranks 0) 2024-03-30T04:16:32.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:31 smithi123 ceph-mon[34758]: monmap e1: 1 mons at {smithi123=[v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-30T04:16:32.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:31 smithi123 ceph-mon[34758]: fsmap 2024-03-30T04:16:32.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:31 smithi123 ceph-mon[34758]: osdmap e1: 0 total, 0 up, 0 in 2024-03-30T04:16:32.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:31 smithi123 ceph-mon[34758]: mgrmap e1: no daemons active 2024-03-30T04:16:32.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:31 smithi123 ceph-mon[34758]: from='client.? 172.21.15.123:0/3476996922' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-30T04:16:33.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:32 smithi123 ceph-mon[34758]: from='client.? 172.21.15.123:0/744693319' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-30T04:16:33.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:32 smithi123 ceph-mon[34758]: from='client.? 172.21.15.123:0/744693319' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-30T04:16:34.159 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout 2024-03-30T04:16:34.159 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout [global] 2024-03-30T04:16:34.159 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout fsid = 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 2024-03-30T04:16:34.159 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.123:3300,v1:172.21.15.123:6789] 2024-03-30T04:16:34.159 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-30T04:16:34.159 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-30T04:16:34.159 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-30T04:16:34.159 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-30T04:16:34.160 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout 2024-03-30T04:16:34.160 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-30T04:16:34.160 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-30T04:16:34.160 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout 2024-03-30T04:16:34.160 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout [osd] 2024-03-30T04:16:34.160 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-30T04:16:34.160 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-30T04:16:34.160 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-30T04:16:34.160 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-30T04:16:34.160 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-30T04:16:34.160 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:32.726+0000 7f5aa6400700 1 Processor -- start 2024-03-30T04:16:34.160 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:32.726+0000 7f5aa6400700 1 -- start start 2024-03-30T04:16:34.160 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:32.727+0000 7f5aa6400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5aa0102580 0x7f5aa0102990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:34.161 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:32.727+0000 7f5aa6400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa0102ed0 con 0x7f5aa0102580 2024-03-30T04:16:34.161 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:32.727+0000 7f5a9f400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5aa0102580 0x7f5aa0102990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:34.161 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:32.727+0000 7f5a9f400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5aa0102580 0x7f5aa0102990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33442/0 (socket says 172.21.15.123:33442) 2024-03-30T04:16:34.161 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:32.727+0000 7f5a9f400700 1 -- 172.21.15.123:0/4099398368 learned_addr learned my addr 172.21.15.123:0/4099398368 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:16:34.161 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:32.728+0000 7f5a9f400700 1 -- 172.21.15.123:0/4099398368 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aa0103010 con 0x7f5aa0102580 2024-03-30T04:16:34.161 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:32.728+0000 7f5a9f400700 1 --2- 172.21.15.123:0/4099398368 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5aa0102580 0x7f5aa0102990 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f5a900098e0 tx=0x7f5a90009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8baa8a82242eea62 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:34.161 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:32.728+0000 7f5a9e000700 1 -- 172.21.15.123:0/4099398368 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a9000b860 con 0x7f5aa0102580 2024-03-30T04:16:34.161 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:32.728+0000 7f5a9e000700 1 -- 172.21.15.123:0/4099398368 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a9000b9c0 con 0x7f5aa0102580 2024-03-30T04:16:34.161 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:32.728+0000 7f5a9e000700 1 -- 172.21.15.123:0/4099398368 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a9000bc90 con 0x7f5aa0102580 2024-03-30T04:16:34.161 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:32.729+0000 7f5aa6400700 1 -- 172.21.15.123:0/4099398368 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5aa0102580 msgr2=0x7f5aa0102990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:34.161 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:32.729+0000 7f5aa6400700 1 --2- 172.21.15.123:0/4099398368 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5aa0102580 0x7f5aa0102990 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f5a900098e0 tx=0x7f5a90009250 comp rx=0 tx=0).stop 2024-03-30T04:16:34.161 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:32.729+0000 7f5aa6400700 1 -- 172.21.15.123:0/4099398368 shutdown_connections 2024-03-30T04:16:34.161 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:32.729+0000 7f5aa6400700 1 --2- 172.21.15.123:0/4099398368 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5aa0102580 0x7f5aa0102990 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:34.161 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:32.729+0000 7f5aa6400700 1 -- 172.21.15.123:0/4099398368 >> 172.21.15.123:0/4099398368 conn(0x7f5aa00fdd00 msgr2=0x7f5aa0100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:34.162 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:32.729+0000 7f5aa6400700 1 -- 172.21.15.123:0/4099398368 shutdown_connections 2024-03-30T04:16:34.162 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:32.729+0000 7f5aa6400700 1 -- 172.21.15.123:0/4099398368 wait complete. 2024-03-30T04:16:34.162 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:32.730+0000 7f5aa6400700 1 Processor -- start 2024-03-30T04:16:34.162 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:32.730+0000 7f5aa6400700 1 -- start start 2024-03-30T04:16:34.162 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:32.731+0000 7f5aa6400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5aa0102580 0x7f5aa0110bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:34.162 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:32.731+0000 7f5aa6400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa0111110 con 0x7f5aa0102580 2024-03-30T04:16:34.162 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:32.731+0000 7f5a9f400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5aa0102580 0x7f5aa0110bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:34.162 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:32.731+0000 7f5a9f400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5aa0102580 0x7f5aa0110bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33454/0 (socket says 172.21.15.123:33454) 2024-03-30T04:16:34.162 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:32.731+0000 7f5a9f400700 1 -- 172.21.15.123:0/744693319 learned_addr learned my addr 172.21.15.123:0/744693319 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:16:34.162 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:32.731+0000 7f5a9f400700 1 -- 172.21.15.123:0/744693319 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a900094f0 con 0x7f5aa0102580 2024-03-30T04:16:34.162 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:32.731+0000 7f5a9f400700 1 --2- 172.21.15.123:0/744693319 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5aa0102580 0x7f5aa0110bd0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f5a90003080 tx=0x7f5a90016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:34.162 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:32.732+0000 7f5a97400700 1 -- 172.21.15.123:0/744693319 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a9001c5c0 con 0x7f5aa0102580 2024-03-30T04:16:34.162 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:32.732+0000 7f5a97400700 1 -- 172.21.15.123:0/744693319 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a9001c720 con 0x7f5aa0102580 2024-03-30T04:16:34.162 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:32.732+0000 7f5aa6400700 1 -- 172.21.15.123:0/744693319 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5aa0111310 con 0x7f5aa0102580 2024-03-30T04:16:34.163 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:32.732+0000 7f5a97400700 1 -- 172.21.15.123:0/744693319 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a9001ca30 con 0x7f5aa0102580 2024-03-30T04:16:34.163 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:32.732+0000 7f5aa6400700 1 -- 172.21.15.123:0/744693319 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aa0111e80 con 0x7f5aa0102580 2024-03-30T04:16:34.163 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:32.732+0000 7f5a97400700 1 -- 172.21.15.123:0/744693319 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f5a9001a450 con 0x7f5aa0102580 2024-03-30T04:16:34.163 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:32.732+0000 7f5a97400700 1 -- 172.21.15.123:0/744693319 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f5a9002e790 con 0x7f5aa0102580 2024-03-30T04:16:34.163 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:32.733+0000 7f5aa6400700 1 -- 172.21.15.123:0/744693319 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5aa005f5f0 con 0x7f5aa0102580 2024-03-30T04:16:34.163 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:32.736+0000 7f5a97400700 1 -- 172.21.15.123:0/744693319 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f5a9001a890 con 0x7f5aa0102580 2024-03-30T04:16:34.163 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:32.804+0000 7f5aa6400700 1 -- 172.21.15.123:0/744693319 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f5aa0103a80 con 0x7f5aa0102580 2024-03-30T04:16:34.163 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:32.808+0000 7f5a97400700 1 -- 172.21.15.123:0/744693319 <== mon.0 v2:172.21.15.123:3300/0 7 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f5a9002ee60 con 0x7f5aa0102580 2024-03-30T04:16:34.163 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:32.808+0000 7f5a97400700 1 -- 172.21.15.123:0/744693319 <== mon.0 v2:172.21.15.123:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+431 (secure 0 0 0) 0x7f5aa0103a80 con 0x7f5aa0102580 2024-03-30T04:16:34.163 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:32.810+0000 7f5aa6400700 1 -- 172.21.15.123:0/744693319 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5aa0102580 msgr2=0x7f5aa0110bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:34.163 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:32.810+0000 7f5aa6400700 1 --2- 172.21.15.123:0/744693319 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5aa0102580 0x7f5aa0110bd0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f5a90003080 tx=0x7f5a90016000 comp rx=0 tx=0).stop 2024-03-30T04:16:34.164 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:32.810+0000 7f5aa6400700 1 -- 172.21.15.123:0/744693319 shutdown_connections 2024-03-30T04:16:34.164 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:32.810+0000 7f5aa6400700 1 --2- 172.21.15.123:0/744693319 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5aa0102580 0x7f5aa0110bd0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:34.164 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:32.810+0000 7f5aa6400700 1 -- 172.21.15.123:0/744693319 >> 172.21.15.123:0/744693319 conn(0x7f5aa00fdd00 msgr2=0x7f5aa0104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:34.164 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:32.810+0000 7f5aa6400700 1 -- 172.21.15.123:0/744693319 shutdown_connections 2024-03-30T04:16:34.164 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:32.810+0000 7f5aa6400700 1 -- 172.21.15.123:0/744693319 wait complete. 2024-03-30T04:16:34.164 INFO:teuthology.orchestra.run.smithi123.stdout:Generating new minimal ceph.conf... 2024-03-30T04:16:35.143 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:34 smithi123 ceph-mon[34758]: from='client.? 172.21.15.123:0/1392957279' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:16:35.411 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:34.706+0000 7fa6f8000700 1 Processor -- start 2024-03-30T04:16:35.411 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:34.706+0000 7fa6f8000700 1 -- start start 2024-03-30T04:16:35.411 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:34.707+0000 7fa6f8000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6f0078d50 0x7fa6f0077260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:35.411 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:34.707+0000 7fa6f8000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f0079160 con 0x7fa6f0078d50 2024-03-30T04:16:35.411 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:34.707+0000 7fa6f5600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6f0078d50 0x7fa6f0077260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:35.412 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:34.707+0000 7fa6f5600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6f0078d50 0x7fa6f0077260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44796/0 (socket says 172.21.15.123:44796) 2024-03-30T04:16:35.412 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:34.707+0000 7fa6f5600700 1 -- 172.21.15.123:0/2792881820 learned_addr learned my addr 172.21.15.123:0/2792881820 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:16:35.412 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:34.708+0000 7fa6f5600700 1 -- 172.21.15.123:0/2792881820 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6f00777a0 con 0x7fa6f0078d50 2024-03-30T04:16:35.412 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:34.708+0000 7fa6f5600700 1 --2- 172.21.15.123:0/2792881820 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6f0078d50 0x7fa6f0077260 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa6e40098e0 tx=0x7fa6e4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da66e181c726a97 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:35.412 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:34.708+0000 7fa6efe00700 1 -- 172.21.15.123:0/2792881820 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6e400b860 con 0x7fa6f0078d50 2024-03-30T04:16:35.412 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:34.708+0000 7fa6efe00700 1 -- 172.21.15.123:0/2792881820 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fa6e400be30 con 0x7fa6f0078d50 2024-03-30T04:16:35.412 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:34.708+0000 7fa6efe00700 1 -- 172.21.15.123:0/2792881820 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6e4010380 con 0x7fa6f0078d50 2024-03-30T04:16:35.412 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:34.709+0000 7fa6f8000700 1 -- 172.21.15.123:0/2792881820 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6f0078d50 msgr2=0x7fa6f0077260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:35.412 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:34.709+0000 7fa6f8000700 1 --2- 172.21.15.123:0/2792881820 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6f0078d50 0x7fa6f0077260 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa6e40098e0 tx=0x7fa6e4009250 comp rx=0 tx=0).stop 2024-03-30T04:16:35.412 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:34.709+0000 7fa6f8000700 1 -- 172.21.15.123:0/2792881820 shutdown_connections 2024-03-30T04:16:35.412 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:34.709+0000 7fa6f8000700 1 --2- 172.21.15.123:0/2792881820 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6f0078d50 0x7fa6f0077260 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:35.412 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:34.709+0000 7fa6f8000700 1 -- 172.21.15.123:0/2792881820 >> 172.21.15.123:0/2792881820 conn(0x7fa6f00fdd00 msgr2=0x7fa6f0100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:35.412 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:34.709+0000 7fa6f8000700 1 -- 172.21.15.123:0/2792881820 shutdown_connections 2024-03-30T04:16:35.412 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:34.709+0000 7fa6f8000700 1 -- 172.21.15.123:0/2792881820 wait complete. 2024-03-30T04:16:35.413 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:34.710+0000 7fa6f8000700 1 Processor -- start 2024-03-30T04:16:35.413 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:34.710+0000 7fa6f8000700 1 -- start start 2024-03-30T04:16:35.413 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:34.710+0000 7fa6f8000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6f0078d50 0x7fa6f01135a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:35.413 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:34.710+0000 7fa6f8000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f0113ae0 con 0x7fa6f0078d50 2024-03-30T04:16:35.413 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:34.711+0000 7fa6f5600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6f0078d50 0x7fa6f01135a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:35.413 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:34.711+0000 7fa6f5600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6f0078d50 0x7fa6f01135a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44798/0 (socket says 172.21.15.123:44798) 2024-03-30T04:16:35.413 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:34.711+0000 7fa6f5600700 1 -- 172.21.15.123:0/1392957279 learned_addr learned my addr 172.21.15.123:0/1392957279 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:16:35.413 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:34.711+0000 7fa6f5600700 1 -- 172.21.15.123:0/1392957279 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6e40094f0 con 0x7fa6f0078d50 2024-03-30T04:16:35.413 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:34.711+0000 7fa6f5600700 1 --2- 172.21.15.123:0/1392957279 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6f0078d50 0x7fa6f01135a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa6e4006fd0 tx=0x7fa6e4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:35.413 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:34.711+0000 7fa6ed600700 1 -- 172.21.15.123:0/1392957279 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6e4010750 con 0x7fa6f0078d50 2024-03-30T04:16:35.413 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:34.711+0000 7fa6ed600700 1 -- 172.21.15.123:0/1392957279 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fa6e4010d20 con 0x7fa6f0078d50 2024-03-30T04:16:35.413 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:34.711+0000 7fa6ed600700 1 -- 172.21.15.123:0/1392957279 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6e401ada0 con 0x7fa6f0078d50 2024-03-30T04:16:35.413 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:34.711+0000 7fa6f8000700 1 -- 172.21.15.123:0/1392957279 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6f0113ce0 con 0x7fa6f0078d50 2024-03-30T04:16:35.414 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:34.712+0000 7fa6f8000700 1 -- 172.21.15.123:0/1392957279 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6f0114850 con 0x7fa6f0078d50 2024-03-30T04:16:35.414 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:34.712+0000 7fa6ed600700 1 -- 172.21.15.123:0/1392957279 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fa6e4024400 con 0x7fa6f0078d50 2024-03-30T04:16:35.414 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:34.712+0000 7fa6ed600700 1 -- 172.21.15.123:0/1392957279 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fa6e4019710 con 0x7fa6f0078d50 2024-03-30T04:16:35.414 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:34.713+0000 7fa6f8000700 1 -- 172.21.15.123:0/1392957279 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6f005f5f0 con 0x7fa6f0078d50 2024-03-30T04:16:35.414 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:34.716+0000 7fa6ed600700 1 -- 172.21.15.123:0/1392957279 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fa6e4029060 con 0x7fa6f0078d50 2024-03-30T04:16:35.414 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:34.785+0000 7fa6f8000700 1 -- 172.21.15.123:0/1392957279 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fa6f01087f0 con 0x7fa6f0078d50 2024-03-30T04:16:35.414 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:34.785+0000 7fa6ed600700 1 -- 172.21.15.123:0/1392957279 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7fa6e4033090 con 0x7fa6f0078d50 2024-03-30T04:16:35.414 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:34.786+0000 7fa6f8000700 1 -- 172.21.15.123:0/1392957279 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6f0078d50 msgr2=0x7fa6f01135a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:35.414 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:34.786+0000 7fa6f8000700 1 --2- 172.21.15.123:0/1392957279 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6f0078d50 0x7fa6f01135a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa6e4006fd0 tx=0x7fa6e4009250 comp rx=0 tx=0).stop 2024-03-30T04:16:35.414 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:34.786+0000 7fa6f8000700 1 -- 172.21.15.123:0/1392957279 shutdown_connections 2024-03-30T04:16:35.414 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:34.786+0000 7fa6f8000700 1 --2- 172.21.15.123:0/1392957279 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6f0078d50 0x7fa6f01135a0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:35.414 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:34.786+0000 7fa6f8000700 1 -- 172.21.15.123:0/1392957279 >> 172.21.15.123:0/1392957279 conn(0x7fa6f00fdd00 msgr2=0x7fa6f00fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:35.414 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:34.786+0000 7fa6f8000700 1 -- 172.21.15.123:0/1392957279 shutdown_connections 2024-03-30T04:16:35.415 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:34.786+0000 7fa6f8000700 1 -- 172.21.15.123:0/1392957279 wait complete. 2024-03-30T04:16:35.415 INFO:teuthology.orchestra.run.smithi123.stdout:Restarting the monitor... 2024-03-30T04:16:35.705 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:35 smithi123 systemd[1]: Stopping Ceph mon.smithi123 for 2e77a2be-ee4c-11ee-b647-cb9ed24678a4... 2024-03-30T04:16:35.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:35 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-mon-smithi123[34735]: 2024-03-30T04:16:35.705+0000 7fdefcc00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi123 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T04:16:35.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:35 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-mon-smithi123[34735]: 2024-03-30T04:16:35.705+0000 7fdefcc00700 -1 mon.smithi123@0(leader) e1 *** Got Signal Terminated *** 2024-03-30T04:16:36.273 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:36 smithi123 bash[35083]: ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-mon-smithi123 2024-03-30T04:16:36.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:36 smithi123 systemd[1]: ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4@mon.smithi123.service: Succeeded. 2024-03-30T04:16:36.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:36 smithi123 systemd[1]: Stopped Ceph mon.smithi123 for 2e77a2be-ee4c-11ee-b647-cb9ed24678a4. 2024-03-30T04:16:36.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:36 smithi123 systemd[1]: Starting Ceph mon.smithi123 for 2e77a2be-ee4c-11ee-b647-cb9ed24678a4... 2024-03-30T04:16:37.080 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: set uid:gid to 167:167 (ceph:ceph) 2024-03-30T04:16:37.118 INFO:teuthology.orchestra.run.smithi123.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-30T04:16:37.340 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable), process ceph-mon, pid 7 2024-03-30T04:16:37.340 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: pidfile_write: ignore empty --pid-file 2024-03-30T04:16:37.340 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: load: jerasure load: lrc 2024-03-30T04:16:37.340 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: RocksDB version: 6.15.5 2024-03-30T04:16:37.340 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-03-30T04:16:37.340 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Compile date Mar 29 2024 2024-03-30T04:16:37.340 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: DB SUMMARY 2024-03-30T04:16:37.341 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: DB Session ID: Y7PMJLCD0AA3BL4KF36V 2024-03-30T04:16:37.341 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: CURRENT file: CURRENT 2024-03-30T04:16:37.341 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: IDENTITY file: IDENTITY 2024-03-30T04:16:37.341 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-03-30T04:16:37.341 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi123/store.db dir, Total Num: 1, files: 000008.sst 2024-03-30T04:16:37.341 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi123/store.db: 000010.log size: 96780 ; 2024-03-30T04:16:37.341 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.error_if_exists: 0 2024-03-30T04:16:37.341 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.create_if_missing: 0 2024-03-30T04:16:37.341 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.paranoid_checks: 1 2024-03-30T04:16:37.341 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-30T04:16:37.341 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.env: 0x55fca77d8a80 2024-03-30T04:16:37.341 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.fs: Posix File System 2024-03-30T04:16:37.341 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.info_log: 0x55fca931bc00 2024-03-30T04:16:37.341 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.max_file_opening_threads: 16 2024-03-30T04:16:37.342 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.statistics: (nil) 2024-03-30T04:16:37.342 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.use_fsync: 0 2024-03-30T04:16:37.342 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.max_log_file_size: 0 2024-03-30T04:16:37.342 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-30T04:16:37.342 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-30T04:16:37.342 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.keep_log_file_num: 1000 2024-03-30T04:16:37.342 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.recycle_log_file_num: 0 2024-03-30T04:16:37.342 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.allow_fallocate: 1 2024-03-30T04:16:37.342 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.allow_mmap_reads: 0 2024-03-30T04:16:37.342 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.allow_mmap_writes: 0 2024-03-30T04:16:37.342 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.use_direct_reads: 0 2024-03-30T04:16:37.342 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-30T04:16:37.342 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.create_missing_column_families: 0 2024-03-30T04:16:37.343 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.db_log_dir: 2024-03-30T04:16:37.343 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi123/store.db 2024-03-30T04:16:37.343 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-30T04:16:37.343 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-30T04:16:37.343 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-30T04:16:37.343 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-30T04:16:37.343 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-30T04:16:37.343 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-30T04:16:37.343 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.advise_random_on_open: 1 2024-03-30T04:16:37.343 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.db_write_buffer_size: 0 2024-03-30T04:16:37.343 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.write_buffer_manager: 0x55fcaa090990 2024-03-30T04:16:37.343 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-30T04:16:37.343 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-03-30T04:16:37.343 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-30T04:16:37.344 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-30T04:16:37.344 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.rate_limiter: (nil) 2024-03-30T04:16:37.344 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-30T04:16:37.344 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.wal_recovery_mode: 2 2024-03-30T04:16:37.344 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.enable_thread_tracking: 0 2024-03-30T04:16:37.344 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.enable_pipelined_write: 0 2024-03-30T04:16:37.344 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.unordered_write: 0 2024-03-30T04:16:37.344 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-30T04:16:37.344 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-30T04:16:37.344 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-30T04:16:37.344 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-30T04:16:37.344 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.row_cache: None 2024-03-30T04:16:37.344 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.wal_filter: None 2024-03-30T04:16:37.344 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-30T04:16:37.345 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.allow_ingest_behind: 0 2024-03-30T04:16:37.345 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.preserve_deletes: 0 2024-03-30T04:16:37.345 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.two_write_queues: 0 2024-03-30T04:16:37.345 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.manual_wal_flush: 0 2024-03-30T04:16:37.345 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.atomic_flush: 0 2024-03-30T04:16:37.345 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-30T04:16:37.345 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-30T04:16:37.345 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-30T04:16:37.345 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.log_readahead_size: 0 2024-03-30T04:16:37.345 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-30T04:16:37.345 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.best_efforts_recovery: 0 2024-03-30T04:16:37.345 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-30T04:16:37.345 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-30T04:16:37.345 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.allow_data_in_errors: 0 2024-03-30T04:16:37.345 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.db_host_id: __hostname__ 2024-03-30T04:16:37.346 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.max_background_jobs: 2 2024-03-30T04:16:37.346 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.max_background_compactions: -1 2024-03-30T04:16:37.346 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.max_subcompactions: 1 2024-03-30T04:16:37.346 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-30T04:16:37.346 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-30T04:16:37.346 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-30T04:16:37.346 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.max_total_wal_size: 0 2024-03-30T04:16:37.346 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-30T04:16:37.346 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-30T04:16:37.346 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-30T04:16:37.346 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-30T04:16:37.346 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.max_open_files: -1 2024-03-30T04:16:37.346 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.bytes_per_sync: 0 2024-03-30T04:16:37.347 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-30T04:16:37.347 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-30T04:16:37.347 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.compaction_readahead_size: 0 2024-03-30T04:16:37.347 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.max_background_flushes: -1 2024-03-30T04:16:37.347 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Compression algorithms supported: 2024-03-30T04:16:37.347 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-30T04:16:37.347 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: kZSTD supported: 0 2024-03-30T04:16:37.347 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: kXpressCompression supported: 0 2024-03-30T04:16:37.347 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: kLZ4HCCompression supported: 1 2024-03-30T04:16:37.347 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: kZlibCompression supported: 1 2024-03-30T04:16:37.347 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: kSnappyCompression supported: 1 2024-03-30T04:16:37.347 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: kLZ4Compression supported: 1 2024-03-30T04:16:37.347 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: kBZip2Compression supported: 0 2024-03-30T04:16:37.347 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-30T04:16:37.347 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi123/store.db/MANIFEST-000009 2024-03-30T04:16:37.348 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-03-30T04:16:37.348 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-30T04:16:37.348 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.merge_operator: 2024-03-30T04:16:37.348 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.compaction_filter: None 2024-03-30T04:16:37.348 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.compaction_filter_factory: None 2024-03-30T04:16:37.348 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.sst_partitioner_factory: None 2024-03-30T04:16:37.348 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-30T04:16:37.348 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-30T04:16:37.348 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55fca9370f58) 2024-03-30T04:16:37.348 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: cache_index_and_filter_blocks: 1 2024-03-30T04:16:37.348 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-30T04:16:37.348 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-30T04:16:37.348 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: pin_top_level_index_and_filter: 1 2024-03-30T04:16:37.348 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: index_type: 0 2024-03-30T04:16:37.348 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: data_block_index_type: 0 2024-03-30T04:16:37.349 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: index_shortening: 1 2024-03-30T04:16:37.349 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-30T04:16:37.349 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: hash_index_allow_collision: 1 2024-03-30T04:16:37.349 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: checksum: 1 2024-03-30T04:16:37.349 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: no_block_cache: 0 2024-03-30T04:16:37.349 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: block_cache: 0x55fca93a3090 2024-03-30T04:16:37.349 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: block_cache_name: BinnedLRUCache 2024-03-30T04:16:37.349 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: block_cache_options: 2024-03-30T04:16:37.349 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: capacity : 536870912 2024-03-30T04:16:37.349 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: num_shard_bits : 4 2024-03-30T04:16:37.349 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: strict_capacity_limit : 0 2024-03-30T04:16:37.349 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: high_pri_pool_ratio: 0.000 2024-03-30T04:16:37.349 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: block_cache_compressed: (nil) 2024-03-30T04:16:37.350 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: persistent_cache: (nil) 2024-03-30T04:16:37.350 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: block_size: 4096 2024-03-30T04:16:37.350 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: block_size_deviation: 10 2024-03-30T04:16:37.350 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: block_restart_interval: 16 2024-03-30T04:16:37.350 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: index_block_restart_interval: 1 2024-03-30T04:16:37.350 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: metadata_block_size: 4096 2024-03-30T04:16:37.350 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: partition_filters: 0 2024-03-30T04:16:37.350 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: use_delta_encoding: 1 2024-03-30T04:16:37.350 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-03-30T04:16:37.350 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: whole_key_filtering: 1 2024-03-30T04:16:37.350 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: verify_compression: 0 2024-03-30T04:16:37.350 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: read_amp_bytes_per_bit: 0 2024-03-30T04:16:37.350 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: format_version: 4 2024-03-30T04:16:37.350 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: enable_index_compression: 1 2024-03-30T04:16:37.351 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: block_align: 0 2024-03-30T04:16:37.351 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.write_buffer_size: 33554432 2024-03-30T04:16:37.351 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.max_write_buffer_number: 2 2024-03-30T04:16:37.351 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.compression: NoCompression 2024-03-30T04:16:37.351 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.bottommost_compression: Disabled 2024-03-30T04:16:37.351 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.prefix_extractor: nullptr 2024-03-30T04:16:37.351 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-30T04:16:37.351 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.num_levels: 7 2024-03-30T04:16:37.351 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-30T04:16:37.351 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-30T04:16:37.351 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-30T04:16:37.351 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-30T04:16:37.351 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-30T04:16:37.352 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-30T04:16:37.352 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-30T04:16:37.352 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-30T04:16:37.352 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-30T04:16:37.352 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-30T04:16:37.352 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-30T04:16:37.352 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.compression_opts.level: 32767 2024-03-30T04:16:37.352 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.compression_opts.strategy: 0 2024-03-30T04:16:37.352 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-30T04:16:37.352 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-30T04:16:37.352 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-30T04:16:37.352 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.compression_opts.enabled: false 2024-03-30T04:16:37.352 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-30T04:16:37.352 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-30T04:16:37.353 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-30T04:16:37.353 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.target_file_size_base: 67108864 2024-03-30T04:16:37.353 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-30T04:16:37.353 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-30T04:16:37.353 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-30T04:16:37.353 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-30T04:16:37.353 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-30T04:16:37.353 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-30T04:16:37.353 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-30T04:16:37.353 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-30T04:16:37.353 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-30T04:16:37.353 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-30T04:16:37.353 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-30T04:16:37.353 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-30T04:16:37.354 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-30T04:16:37.354 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.arena_block_size: 4194304 2024-03-30T04:16:37.354 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-30T04:16:37.354 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-30T04:16:37.354 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-03-30T04:16:37.354 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.disable_auto_compactions: 0 2024-03-30T04:16:37.354 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-30T04:16:37.354 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-30T04:16:37.354 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-30T04:16:37.354 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-30T04:16:37.354 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-30T04:16:37.354 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-30T04:16:37.354 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-30T04:16:37.354 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-30T04:16:37.355 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-30T04:16:37.355 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-30T04:16:37.355 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 bash[35206]: 51d18587a7dfb0d1b41df0b9c6138b332541de63f5dca1d3e99145c071221ed5 2024-03-30T04:16:37.355 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.table_properties_collectors: 2024-03-30T04:16:37.355 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.inplace_update_support: 0 2024-03-30T04:16:37.355 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-30T04:16:37.355 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-30T04:16:37.355 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-30T04:16:37.355 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-30T04:16:37.355 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.bloom_locality: 0 2024-03-30T04:16:37.355 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.max_successive_merges: 0 2024-03-30T04:16:37.355 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-30T04:16:37.355 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.paranoid_file_checks: 0 2024-03-30T04:16:37.355 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.force_consistency_checks: 1 2024-03-30T04:16:37.356 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.report_bg_io_stats: 0 2024-03-30T04:16:37.356 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.ttl: 2592000 2024-03-30T04:16:37.356 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-30T04:16:37.356 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.enable_blob_files: false 2024-03-30T04:16:37.356 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.min_blob_size: 0 2024-03-30T04:16:37.356 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.blob_file_size: 268435456 2024-03-30T04:16:37.356 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-30T04:16:37.356 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-30T04:16:37.356 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-30T04:16:37.356 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi123/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-30T04:16:37.356 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2024-03-30T04:16:37.356 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2024-03-30T04:16:37.356 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711772197095333, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-03-30T04:16:37.356 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2024-03-30T04:16:37.356 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-03-30T04:16:37.357 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711772197097381, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 92149, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 90244, "index_size": 265, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13225, "raw_average_key_size": 49, "raw_value_size": 82983, "raw_average_value_size": 310, "num_data_blocks": 11, "num_entries": 267, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1711772197, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "1f886c23-95f4-477d-819d-a086b1c53672", "db_session_id": "Y7PMJLCD0AA3BL4KF36V"}} 2024-03-30T04:16:37.357 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2024-03-30T04:16:37.357 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711772197098235, "job": 1, "event": "recovery_finished"} 2024-03-30T04:16:37.357 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-smithi123/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-30T04:16:37.357 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x55fca9337180 2024-03-30T04:16:37.357 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: DB pointer 0x55fca940c000 2024-03-30T04:16:37.357 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-03-30T04:16:37.357 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-03-30T04:16:37.357 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: ** DB Stats ** 2024-03-30T04:16:37.357 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-30T04:16:37.357 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-30T04:16:37.357 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-30T04:16:37.357 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-30T04:16:37.357 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-30T04:16:37.358 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-03-30T04:16:37.358 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-30T04:16:37.358 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: 2024-03-30T04:16:37.358 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: ** Compaction Stats [default] ** 2024-03-30T04:16:37.358 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-30T04:16:37.358 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-30T04:16:37.358 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: L0 2/0 91.79 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 71.4 0.00 0.00 1 0.001 0 0 2024-03-30T04:16:37.358 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: Sum 2/0 91.79 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 71.4 0.00 0.00 1 0.001 0 0 2024-03-30T04:16:37.358 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 71.4 0.00 0.00 1 0.001 0 0 2024-03-30T04:16:37.358 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: 2024-03-30T04:16:37.358 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: ** Compaction Stats [default] ** 2024-03-30T04:16:37.358 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-30T04:16:37.358 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-30T04:16:37.359 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 71.4 0.00 0.00 1 0.001 0 0 2024-03-30T04:16:37.359 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-30T04:16:37.359 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-30T04:16:37.359 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-30T04:16:37.359 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-30T04:16:37.359 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-30T04:16:37.359 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-30T04:16:37.359 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: Cumulative compaction: 0.00 GB write, 9.99 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-30T04:16:37.359 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: Interval compaction: 0.00 GB write, 9.99 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-30T04:16:37.359 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-30T04:16:37.359 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: 2024-03-30T04:16:37.359 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-30T04:16:37.359 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: 2024-03-30T04:16:37.359 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: ** Compaction Stats [default] ** 2024-03-30T04:16:37.360 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-30T04:16:37.360 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-30T04:16:37.360 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: L0 2/0 91.79 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 71.4 0.00 0.00 1 0.001 0 0 2024-03-30T04:16:37.360 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: Sum 2/0 91.79 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 71.4 0.00 0.00 1 0.001 0 0 2024-03-30T04:16:37.360 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-03-30T04:16:37.360 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: 2024-03-30T04:16:37.360 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: ** Compaction Stats [default] ** 2024-03-30T04:16:37.360 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-30T04:16:37.360 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-30T04:16:37.360 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 71.4 0.00 0.00 1 0.001 0 0 2024-03-30T04:16:37.360 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-30T04:16:37.360 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-30T04:16:37.360 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-30T04:16:37.360 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-30T04:16:37.360 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-30T04:16:37.360 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-30T04:16:37.361 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: Cumulative compaction: 0.00 GB write, 9.95 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-30T04:16:37.361 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-30T04:16:37.361 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-30T04:16:37.361 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: 2024-03-30T04:16:37.361 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-30T04:16:37.361 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: starting mon.smithi123 rank 0 at public addrs [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] at bind addrs [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi123 fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 2024-03-30T04:16:37.361 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: mon.smithi123@-1(???) e1 preinit fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 2024-03-30T04:16:37.361 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: mon.smithi123@-1(???).mds e1 new map 2024-03-30T04:16:37.361 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: mon.smithi123@-1(???).mds e1 print_map 2024-03-30T04:16:37.361 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: e1 2024-03-30T04:16:37.361 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:16:37.361 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:16:37.361 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: legacy client fscid: -1 2024-03-30T04:16:37.361 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: 2024-03-30T04:16:37.362 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: No filesystems configured 2024-03-30T04:16:37.362 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: mon.smithi123@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-30T04:16:37.362 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: mon.smithi123@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-30T04:16:37.362 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: mon.smithi123@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-30T04:16:37.362 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: mon.smithi123@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-30T04:16:37.362 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: mon.smithi123@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-30T04:16:37.362 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 systemd[1]: Started Ceph mon.smithi123 for 2e77a2be-ee4c-11ee-b647-cb9ed24678a4. 2024-03-30T04:16:37.362 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: mon.smithi123 is new leader, mons smithi123 in quorum (ranks 0) 2024-03-30T04:16:37.362 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: monmap e1: 1 mons at {smithi123=[v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-30T04:16:37.362 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: fsmap 2024-03-30T04:16:37.362 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: osdmap e1: 0 total, 0 up, 0 in 2024-03-30T04:16:37.362 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:37 smithi123 ceph-mon[35245]: mgrmap e1: no daemons active 2024-03-30T04:16:38.320 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:37.650+0000 7f53e8600700 1 Processor -- start 2024-03-30T04:16:38.320 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:37.650+0000 7f53e8600700 1 -- start start 2024-03-30T04:16:38.320 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:37.650+0000 7f53e8600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53e0078d60 0x7f53e0077260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:38.321 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:37.650+0000 7f53e8600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53e0079170 con 0x7f53e0078d60 2024-03-30T04:16:38.321 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:37.651+0000 7f53e5c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53e0078d60 0x7f53e0077260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:38.321 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:37.651+0000 7f53e5c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53e0078d60 0x7f53e0077260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44812/0 (socket says 172.21.15.123:44812) 2024-03-30T04:16:38.321 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:37.651+0000 7f53e5c00700 1 -- 172.21.15.123:0/3647761146 learned_addr learned my addr 172.21.15.123:0/3647761146 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:16:38.321 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:37.652+0000 7f53e5c00700 1 -- 172.21.15.123:0/3647761146 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53e00777a0 con 0x7f53e0078d60 2024-03-30T04:16:38.321 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:37.652+0000 7f53e5c00700 1 --2- 172.21.15.123:0/3647761146 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53e0078d60 0x7f53e0077260 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f53d40098e0 tx=0x7f53d4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d09f77e9e878ce20 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:38.321 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:37.652+0000 7f53dfe00700 1 -- 172.21.15.123:0/3647761146 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53d400b860 con 0x7f53e0078d60 2024-03-30T04:16:38.321 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:37.652+0000 7f53dfe00700 1 -- 172.21.15.123:0/3647761146 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f53d400be30 con 0x7f53e0078d60 2024-03-30T04:16:38.322 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:37.652+0000 7f53dfe00700 1 -- 172.21.15.123:0/3647761146 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53d4010380 con 0x7f53e0078d60 2024-03-30T04:16:38.322 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:37.653+0000 7f53e8600700 1 -- 172.21.15.123:0/3647761146 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53e0078d60 msgr2=0x7f53e0077260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:38.322 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:37.653+0000 7f53e8600700 1 --2- 172.21.15.123:0/3647761146 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53e0078d60 0x7f53e0077260 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f53d40098e0 tx=0x7f53d4009250 comp rx=0 tx=0).stop 2024-03-30T04:16:38.322 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:37.653+0000 7f53e8600700 1 -- 172.21.15.123:0/3647761146 shutdown_connections 2024-03-30T04:16:38.322 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:37.653+0000 7f53e8600700 1 --2- 172.21.15.123:0/3647761146 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53e0078d60 0x7f53e0077260 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:38.322 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:37.653+0000 7f53e8600700 1 -- 172.21.15.123:0/3647761146 >> 172.21.15.123:0/3647761146 conn(0x7f53e00fdcd0 msgr2=0x7f53e0100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:38.322 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:37.653+0000 7f53e8600700 1 -- 172.21.15.123:0/3647761146 shutdown_connections 2024-03-30T04:16:38.322 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:37.653+0000 7f53e8600700 1 -- 172.21.15.123:0/3647761146 wait complete. 2024-03-30T04:16:38.323 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:37.653+0000 7f53e8600700 1 Processor -- start 2024-03-30T04:16:38.323 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:37.654+0000 7f53e8600700 1 -- start start 2024-03-30T04:16:38.323 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:37.654+0000 7f53e8600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53e0078d60 0x7f53e0113630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:38.323 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:37.654+0000 7f53e8600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53e0113b70 con 0x7f53e0078d60 2024-03-30T04:16:38.323 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:37.654+0000 7f53e5c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53e0078d60 0x7f53e0113630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:38.323 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:37.654+0000 7f53e5c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53e0078d60 0x7f53e0113630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44816/0 (socket says 172.21.15.123:44816) 2024-03-30T04:16:38.323 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:37.654+0000 7f53e5c00700 1 -- 172.21.15.123:0/779106052 learned_addr learned my addr 172.21.15.123:0/779106052 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:16:38.323 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:37.655+0000 7f53e5c00700 1 -- 172.21.15.123:0/779106052 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53d40094f0 con 0x7f53e0078d60 2024-03-30T04:16:38.324 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:37.655+0000 7f53e5c00700 1 --2- 172.21.15.123:0/779106052 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53e0078d60 0x7f53e0113630 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f53d4006b00 tx=0x7f53d4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:38.324 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:37.655+0000 7f53dd600700 1 -- 172.21.15.123:0/779106052 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53d4010750 con 0x7f53e0078d60 2024-03-30T04:16:38.324 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:37.655+0000 7f53dd600700 1 -- 172.21.15.123:0/779106052 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f53d4010d20 con 0x7f53e0078d60 2024-03-30T04:16:38.324 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:37.655+0000 7f53e8600700 1 -- 172.21.15.123:0/779106052 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53e0113d70 con 0x7f53e0078d60 2024-03-30T04:16:38.324 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:37.655+0000 7f53dd600700 1 -- 172.21.15.123:0/779106052 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53d401acf0 con 0x7f53e0078d60 2024-03-30T04:16:38.324 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:37.655+0000 7f53e8600700 1 -- 172.21.15.123:0/779106052 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53e01148e0 con 0x7f53e0078d60 2024-03-30T04:16:38.324 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:37.655+0000 7f53dd600700 1 -- 172.21.15.123:0/779106052 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f53d402b6c0 con 0x7f53e0078d60 2024-03-30T04:16:38.324 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:37.656+0000 7f53dd600700 1 -- 172.21.15.123:0/779106052 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f53d401e070 con 0x7f53e0078d60 2024-03-30T04:16:38.325 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:37.656+0000 7f53e8600700 1 -- 172.21.15.123:0/779106052 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53e005f5f0 con 0x7f53e0078d60 2024-03-30T04:16:38.325 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:37.659+0000 7f53dd600700 1 -- 172.21.15.123:0/779106052 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f53d4010e90 con 0x7f53e0078d60 2024-03-30T04:16:38.325 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:37.727+0000 7f53e8600700 1 -- 172.21.15.123:0/779106052 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f53e01087d0 con 0x7f53e0078d60 2024-03-30T04:16:38.325 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:37.730+0000 7f53dd600700 1 -- 172.21.15.123:0/779106052 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f53d40330c0 con 0x7f53e0078d60 2024-03-30T04:16:38.325 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:37.731+0000 7f53e8600700 1 -- 172.21.15.123:0/779106052 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53e0078d60 msgr2=0x7f53e0113630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:38.325 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:37.731+0000 7f53e8600700 1 --2- 172.21.15.123:0/779106052 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53e0078d60 0x7f53e0113630 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f53d4006b00 tx=0x7f53d4009250 comp rx=0 tx=0).stop 2024-03-30T04:16:38.325 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:37.731+0000 7f53e8600700 1 -- 172.21.15.123:0/779106052 shutdown_connections 2024-03-30T04:16:38.325 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:37.731+0000 7f53e8600700 1 --2- 172.21.15.123:0/779106052 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53e0078d60 0x7f53e0113630 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:38.325 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:37.731+0000 7f53e8600700 1 -- 172.21.15.123:0/779106052 >> 172.21.15.123:0/779106052 conn(0x7f53e00fdcd0 msgr2=0x7f53e00fe8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:38.326 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:37.731+0000 7f53e8600700 1 -- 172.21.15.123:0/779106052 shutdown_connections 2024-03-30T04:16:38.326 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:37.731+0000 7f53e8600700 1 -- 172.21.15.123:0/779106052 wait complete. 2024-03-30T04:16:38.326 INFO:teuthology.orchestra.run.smithi123.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-30T04:16:38.326 INFO:teuthology.orchestra.run.smithi123.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-30T04:16:38.326 INFO:teuthology.orchestra.run.smithi123.stdout:Creating mgr... 2024-03-30T04:16:38.326 INFO:teuthology.orchestra.run.smithi123.stdout:Verifying port 9283 ... 2024-03-30T04:16:38.583 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4@mgr.smithi123.xgksqq 2024-03-30T04:16:38.583 INFO:teuthology.orchestra.run.smithi123.stdout:systemctl: stderr Failed to reset failed state of unit ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4@mgr.smithi123.xgksqq.service: Unit ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4@mgr.smithi123.xgksqq.service not loaded. 2024-03-30T04:16:38.765 INFO:teuthology.orchestra.run.smithi123.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4.target.wants/ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4@mgr.smithi123.xgksqq.service → /etc/systemd/system/ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4@.service. 2024-03-30T04:16:38.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:38 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/779106052' entity='client.admin' 2024-03-30T04:16:39.494 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-30T04:16:39.494 INFO:teuthology.orchestra.run.smithi123.stdout:systemctl: stdout disabled 2024-03-30T04:16:39.511 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-30T04:16:39.511 INFO:teuthology.orchestra.run.smithi123.stdout:systemctl: stdout inactive 2024-03-30T04:16:39.511 INFO:teuthology.orchestra.run.smithi123.stdout:firewalld.service is not enabled 2024-03-30T04:16:39.511 INFO:teuthology.orchestra.run.smithi123.stdout:Not possible to enable service . firewalld.service is not available 2024-03-30T04:16:39.528 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-30T04:16:39.528 INFO:teuthology.orchestra.run.smithi123.stdout:systemctl: stdout disabled 2024-03-30T04:16:39.545 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-30T04:16:39.545 INFO:teuthology.orchestra.run.smithi123.stdout:systemctl: stdout inactive 2024-03-30T04:16:39.545 INFO:teuthology.orchestra.run.smithi123.stdout:firewalld.service is not enabled 2024-03-30T04:16:39.545 INFO:teuthology.orchestra.run.smithi123.stdout:Not possible to open ports <[9283]>. firewalld.service is not available 2024-03-30T04:16:39.545 INFO:teuthology.orchestra.run.smithi123.stdout:Waiting for mgr to start... 2024-03-30T04:16:39.545 INFO:teuthology.orchestra.run.smithi123.stdout:Waiting for mgr... 2024-03-30T04:16:40.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:40 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/3827061024' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-30T04:16:40.779 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout 2024-03-30T04:16:40.798 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout { 2024-03-30T04:16:40.798 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "fsid": "2e77a2be-ee4c-11ee-b647-cb9ed24678a4", 2024-03-30T04:16:40.798 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "health": { 2024-03-30T04:16:40.799 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-30T04:16:40.799 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-30T04:16:40.799 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-30T04:16:40.799 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout }, 2024-03-30T04:16:40.799 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-30T04:16:40.799 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-30T04:16:40.799 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout 0 2024-03-30T04:16:40.800 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout ], 2024-03-30T04:16:40.800 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-30T04:16:40.800 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "smithi123" 2024-03-30T04:16:40.800 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout ], 2024-03-30T04:16:40.800 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-03-30T04:16:40.800 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-30T04:16:40.800 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T04:16:40.801 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-03-30T04:16:40.801 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-30T04:16:40.801 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout }, 2024-03-30T04:16:40.801 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-30T04:16:40.801 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T04:16:40.801 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-30T04:16:40.802 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-30T04:16:40.802 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-30T04:16:40.802 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-30T04:16:40.802 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-30T04:16:40.802 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-30T04:16:40.802 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout }, 2024-03-30T04:16:40.803 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-30T04:16:40.803 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-30T04:16:40.803 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-30T04:16:40.803 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-30T04:16:40.803 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-30T04:16:40.803 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-30T04:16:40.803 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-30T04:16:40.804 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-30T04:16:40.804 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-30T04:16:40.804 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout }, 2024-03-30T04:16:40.804 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-30T04:16:40.804 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T04:16:40.804 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-30T04:16:40.804 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-30T04:16:40.804 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout }, 2024-03-30T04:16:40.805 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-30T04:16:40.805 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-30T04:16:40.805 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-30T04:16:40.805 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-30T04:16:40.805 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-30T04:16:40.805 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-30T04:16:40.805 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "restful" 2024-03-30T04:16:40.806 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout ], 2024-03-30T04:16:40.806 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-30T04:16:40.806 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout }, 2024-03-30T04:16:40.806 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-30T04:16:40.806 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T04:16:40.806 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "modified": "2024-03-30T04:16:30.949937+0000", 2024-03-30T04:16:40.806 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-30T04:16:40.806 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout }, 2024-03-30T04:16:40.807 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-30T04:16:40.807 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout } 2024-03-30T04:16:40.807 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:40.108+0000 7fe1a3800700 1 Processor -- start 2024-03-30T04:16:40.807 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:40.108+0000 7fe1a3800700 1 -- start start 2024-03-30T04:16:40.807 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:40.109+0000 7fe1a3800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe19c076bc0 0x7fe19c105270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:40.807 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:40.109+0000 7fe1a3800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe19c1057b0 con 0x7fe19c076bc0 2024-03-30T04:16:40.807 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:40.109+0000 7fe1a0e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe19c076bc0 0x7fe19c105270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:40.808 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:40.110+0000 7fe1a0e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe19c076bc0 0x7fe19c105270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44838/0 (socket says 172.21.15.123:44838) 2024-03-30T04:16:40.808 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:40.110+0000 7fe1a0e00700 1 -- 172.21.15.123:0/3475576008 learned_addr learned my addr 172.21.15.123:0/3475576008 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:16:40.808 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:40.110+0000 7fe1a0e00700 1 -- 172.21.15.123:0/3475576008 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe19c1058f0 con 0x7fe19c076bc0 2024-03-30T04:16:40.808 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:40.110+0000 7fe1a0e00700 1 --2- 172.21.15.123:0/3475576008 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe19c076bc0 0x7fe19c105270 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fe1900098e0 tx=0x7fe190009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44f99133a572a0bc server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:40.808 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:40.110+0000 7fe19b400700 1 -- 172.21.15.123:0/3475576008 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe19000b860 con 0x7fe19c076bc0 2024-03-30T04:16:40.808 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:40.110+0000 7fe19b400700 1 -- 172.21.15.123:0/3475576008 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fe19000be30 con 0x7fe19c076bc0 2024-03-30T04:16:40.808 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:40.110+0000 7fe19b400700 1 -- 172.21.15.123:0/3475576008 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe190010380 con 0x7fe19c076bc0 2024-03-30T04:16:40.808 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:40.111+0000 7fe1a3800700 1 -- 172.21.15.123:0/3475576008 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe19c076bc0 msgr2=0x7fe19c105270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:40.809 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:40.111+0000 7fe1a3800700 1 --2- 172.21.15.123:0/3475576008 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe19c076bc0 0x7fe19c105270 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fe1900098e0 tx=0x7fe190009250 comp rx=0 tx=0).stop 2024-03-30T04:16:40.809 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:40.111+0000 7fe1a3800700 1 -- 172.21.15.123:0/3475576008 shutdown_connections 2024-03-30T04:16:40.809 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:40.111+0000 7fe1a3800700 1 --2- 172.21.15.123:0/3475576008 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe19c076bc0 0x7fe19c105270 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:40.809 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:40.111+0000 7fe1a3800700 1 -- 172.21.15.123:0/3475576008 >> 172.21.15.123:0/3475576008 conn(0x7fe19c0fcc60 msgr2=0x7fe19c0ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:40.809 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:40.112+0000 7fe1a3800700 1 -- 172.21.15.123:0/3475576008 shutdown_connections 2024-03-30T04:16:40.809 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:40.112+0000 7fe1a3800700 1 -- 172.21.15.123:0/3475576008 wait complete. 2024-03-30T04:16:40.809 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:40.112+0000 7fe1a3800700 1 Processor -- start 2024-03-30T04:16:40.810 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:40.113+0000 7fe1a3800700 1 -- start start 2024-03-30T04:16:40.810 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:40.113+0000 7fe1a3800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe19c076bc0 0x7fe19c1135e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:40.810 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:40.113+0000 7fe1a3800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe19c113b20 con 0x7fe19c076bc0 2024-03-30T04:16:40.810 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:40.113+0000 7fe1a0e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe19c076bc0 0x7fe19c1135e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:40.810 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:40.113+0000 7fe1a0e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe19c076bc0 0x7fe19c1135e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44846/0 (socket says 172.21.15.123:44846) 2024-03-30T04:16:40.810 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:40.113+0000 7fe1a0e00700 1 -- 172.21.15.123:0/3827061024 learned_addr learned my addr 172.21.15.123:0/3827061024 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:16:40.810 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:40.114+0000 7fe1a0e00700 1 -- 172.21.15.123:0/3827061024 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1900094f0 con 0x7fe19c076bc0 2024-03-30T04:16:40.811 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:40.114+0000 7fe1a0e00700 1 --2- 172.21.15.123:0/3827061024 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe19c076bc0 0x7fe19c1135e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe190006b00 tx=0x7fe190009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:40.811 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:40.114+0000 7fe198c00700 1 -- 172.21.15.123:0/3827061024 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe190010a80 con 0x7fe19c076bc0 2024-03-30T04:16:40.811 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:40.114+0000 7fe198c00700 1 -- 172.21.15.123:0/3827061024 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fe19001a430 con 0x7fe19c076bc0 2024-03-30T04:16:40.811 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:40.114+0000 7fe1a3800700 1 -- 172.21.15.123:0/3827061024 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe19c113d20 con 0x7fe19c076bc0 2024-03-30T04:16:40.811 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:40.114+0000 7fe198c00700 1 -- 172.21.15.123:0/3827061024 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe1900245b0 con 0x7fe19c076bc0 2024-03-30T04:16:40.811 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:40.114+0000 7fe1a3800700 1 -- 172.21.15.123:0/3827061024 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe19c114840 con 0x7fe19c076bc0 2024-03-30T04:16:40.811 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:40.115+0000 7fe198c00700 1 -- 172.21.15.123:0/3827061024 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fe190024ab0 con 0x7fe19c076bc0 2024-03-30T04:16:40.812 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:40.115+0000 7fe198c00700 1 -- 172.21.15.123:0/3827061024 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fe19002eb00 con 0x7fe19c076bc0 2024-03-30T04:16:40.812 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:40.115+0000 7fe1a3800700 1 -- 172.21.15.123:0/3827061024 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe19c05f5f0 con 0x7fe19c076bc0 2024-03-30T04:16:40.812 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:40.118+0000 7fe198c00700 1 -- 172.21.15.123:0/3827061024 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fe19002ecc0 con 0x7fe19c076bc0 2024-03-30T04:16:40.812 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:40.191+0000 7fe1a3800700 1 -- 172.21.15.123:0/3827061024 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fe19c1065c0 con 0x7fe19c076bc0 2024-03-30T04:16:40.812 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:40.191+0000 7fe198c00700 1 -- 172.21.15.123:0/3827061024 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7fe19002ee60 con 0x7fe19c076bc0 2024-03-30T04:16:40.812 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:40.192+0000 7fe1a3800700 1 -- 172.21.15.123:0/3827061024 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe19c076bc0 msgr2=0x7fe19c1135e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:40.812 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:40.192+0000 7fe1a3800700 1 --2- 172.21.15.123:0/3827061024 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe19c076bc0 0x7fe19c1135e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe190006b00 tx=0x7fe190009250 comp rx=0 tx=0).stop 2024-03-30T04:16:40.812 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:40.192+0000 7fe1a3800700 1 -- 172.21.15.123:0/3827061024 shutdown_connections 2024-03-30T04:16:40.813 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:40.192+0000 7fe1a3800700 1 --2- 172.21.15.123:0/3827061024 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe19c076bc0 0x7fe19c1135e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:40.813 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:40.192+0000 7fe1a3800700 1 -- 172.21.15.123:0/3827061024 >> 172.21.15.123:0/3827061024 conn(0x7fe19c0fcc60 msgr2=0x7fe19c0fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:40.813 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:40.192+0000 7fe1a3800700 1 -- 172.21.15.123:0/3827061024 shutdown_connections 2024-03-30T04:16:40.813 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:40.192+0000 7fe1a3800700 1 -- 172.21.15.123:0/3827061024 wait complete. 2024-03-30T04:16:40.813 INFO:teuthology.orchestra.run.smithi123.stdout:mgr not available, waiting (1/15)... 2024-03-30T04:16:43.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:43 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/2617770857' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-30T04:16:47.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:47 smithi123 ceph-mon[35245]: Activating manager daemon smithi123.xgksqq 2024-03-30T04:16:47.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:47 smithi123 ceph-mon[35245]: mgrmap e2: smithi123.xgksqq(active, starting, since 0.00432219s) 2024-03-30T04:16:47.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:47 smithi123 ceph-mon[35245]: from='mgr.14100 172.21.15.123:0/1491665265' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-30T04:16:47.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:47 smithi123 ceph-mon[35245]: from='mgr.14100 172.21.15.123:0/1491665265' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-30T04:16:47.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:47 smithi123 ceph-mon[35245]: from='mgr.14100 172.21.15.123:0/1491665265' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-30T04:16:47.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:47 smithi123 ceph-mon[35245]: from='mgr.14100 172.21.15.123:0/1491665265' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mon metadata", "id": "smithi123"}]: dispatch 2024-03-30T04:16:47.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:47 smithi123 ceph-mon[35245]: from='mgr.14100 172.21.15.123:0/1491665265' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mgr metadata", "who": "smithi123.xgksqq", "id": "smithi123.xgksqq"}]: dispatch 2024-03-30T04:16:47.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:47 smithi123 ceph-mon[35245]: Manager daemon smithi123.xgksqq is now available 2024-03-30T04:16:47.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:47 smithi123 ceph-mon[35245]: from='mgr.14100 172.21.15.123:0/1491665265' entity='mgr.smithi123.xgksqq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi123.xgksqq/mirror_snapshot_schedule"}]: dispatch 2024-03-30T04:16:47.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:47 smithi123 ceph-mon[35245]: from='mgr.14100 172.21.15.123:0/1491665265' entity='mgr.smithi123.xgksqq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi123.xgksqq/trash_purge_schedule"}]: dispatch 2024-03-30T04:16:47.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:47 smithi123 ceph-mon[35245]: from='mgr.14100 172.21.15.123:0/1491665265' entity='mgr.smithi123.xgksqq' 2024-03-30T04:16:47.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:47 smithi123 ceph-mon[35245]: from='mgr.14100 172.21.15.123:0/1491665265' entity='mgr.smithi123.xgksqq' 2024-03-30T04:16:47.968 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:47 smithi123 ceph-mon[35245]: from='mgr.14100 172.21.15.123:0/1491665265' entity='mgr.smithi123.xgksqq' 2024-03-30T04:16:50.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:49 smithi123 ceph-mon[35245]: mgrmap e3: smithi123.xgksqq(active, since 1.00994s) 2024-03-30T04:16:51.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:50 smithi123 ceph-mon[35245]: mgrmap e4: smithi123.xgksqq(active, since 2s) 2024-03-30T04:16:51.255 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout 2024-03-30T04:16:51.271 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout { 2024-03-30T04:16:51.271 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "fsid": "2e77a2be-ee4c-11ee-b647-cb9ed24678a4", 2024-03-30T04:16:51.271 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "health": { 2024-03-30T04:16:51.271 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-30T04:16:51.271 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-30T04:16:51.272 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-30T04:16:51.272 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout }, 2024-03-30T04:16:51.272 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-30T04:16:51.272 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-30T04:16:51.272 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout 0 2024-03-30T04:16:51.272 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout ], 2024-03-30T04:16:51.272 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-30T04:16:51.272 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "smithi123" 2024-03-30T04:16:51.272 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout ], 2024-03-30T04:16:51.272 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-03-30T04:16:51.273 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-30T04:16:51.273 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T04:16:51.273 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-03-30T04:16:51.273 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-30T04:16:51.273 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout }, 2024-03-30T04:16:51.273 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-30T04:16:51.273 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T04:16:51.273 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-30T04:16:51.273 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-30T04:16:51.273 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-30T04:16:51.273 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-30T04:16:51.274 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-30T04:16:51.274 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-30T04:16:51.274 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout }, 2024-03-30T04:16:51.274 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-30T04:16:51.274 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-30T04:16:51.274 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-30T04:16:51.274 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-30T04:16:51.274 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-30T04:16:51.274 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-30T04:16:51.274 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-30T04:16:51.274 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-30T04:16:51.275 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-30T04:16:51.275 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout }, 2024-03-30T04:16:51.275 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-30T04:16:51.275 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T04:16:51.275 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-30T04:16:51.275 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-30T04:16:51.275 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout }, 2024-03-30T04:16:51.275 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-30T04:16:51.275 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-30T04:16:51.275 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-30T04:16:51.276 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-30T04:16:51.276 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-30T04:16:51.276 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-30T04:16:51.276 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "restful" 2024-03-30T04:16:51.276 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout ], 2024-03-30T04:16:51.276 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-30T04:16:51.276 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout }, 2024-03-30T04:16:51.276 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-30T04:16:51.276 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T04:16:51.276 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "modified": "2024-03-30T04:16:30.949937+0000", 2024-03-30T04:16:51.276 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-30T04:16:51.277 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout }, 2024-03-30T04:16:51.277 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-30T04:16:51.277 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout } 2024-03-30T04:16:51.277 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:43.368+0000 7f6fae800700 1 Processor -- start 2024-03-30T04:16:51.277 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:43.369+0000 7f6fae800700 1 -- start start 2024-03-30T04:16:51.277 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:43.369+0000 7f6fae800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fa8076c80 0x7f6fa81051b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:51.277 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:43.369+0000 7f6fae800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa81056f0 con 0x7f6fa8076c80 2024-03-30T04:16:51.277 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:43.369+0000 7f6fa7400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fa8076c80 0x7f6fa81051b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:51.277 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:43.370+0000 7f6fa7400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fa8076c80 0x7f6fa81051b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44858/0 (socket says 172.21.15.123:44858) 2024-03-30T04:16:51.278 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:43.370+0000 7f6fa7400700 1 -- 172.21.15.123:0/2339161747 learned_addr learned my addr 172.21.15.123:0/2339161747 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:16:51.278 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:43.370+0000 7f6fa7400700 1 -- 172.21.15.123:0/2339161747 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fa8105830 con 0x7f6fa8076c80 2024-03-30T04:16:51.278 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:43.370+0000 7f6fa7400700 1 --2- 172.21.15.123:0/2339161747 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fa8076c80 0x7f6fa81051b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6f980098e0 tx=0x7f6f98009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f24c542b1926201a server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:51.278 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:43.370+0000 7f6fa6000700 1 -- 172.21.15.123:0/2339161747 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f9800b860 con 0x7f6fa8076c80 2024-03-30T04:16:51.278 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:43.370+0000 7f6fa6000700 1 -- 172.21.15.123:0/2339161747 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f6f9800be30 con 0x7f6fa8076c80 2024-03-30T04:16:51.278 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:43.370+0000 7f6fa6000700 1 -- 172.21.15.123:0/2339161747 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f98010380 con 0x7f6fa8076c80 2024-03-30T04:16:51.278 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:43.371+0000 7f6fae800700 1 -- 172.21.15.123:0/2339161747 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fa8076c80 msgr2=0x7f6fa81051b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:51.278 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:43.371+0000 7f6fae800700 1 --2- 172.21.15.123:0/2339161747 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fa8076c80 0x7f6fa81051b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6f980098e0 tx=0x7f6f98009250 comp rx=0 tx=0).stop 2024-03-30T04:16:51.279 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:43.371+0000 7f6fae800700 1 -- 172.21.15.123:0/2339161747 shutdown_connections 2024-03-30T04:16:51.279 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:43.371+0000 7f6fae800700 1 --2- 172.21.15.123:0/2339161747 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fa8076c80 0x7f6fa81051b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:51.279 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:43.371+0000 7f6fae800700 1 -- 172.21.15.123:0/2339161747 >> 172.21.15.123:0/2339161747 conn(0x7f6fa80fcc60 msgr2=0x7f6fa80ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:51.279 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:43.371+0000 7f6fae800700 1 -- 172.21.15.123:0/2339161747 shutdown_connections 2024-03-30T04:16:51.279 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:43.371+0000 7f6fae800700 1 -- 172.21.15.123:0/2339161747 wait complete. 2024-03-30T04:16:51.279 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:43.372+0000 7f6fae800700 1 Processor -- start 2024-03-30T04:16:51.279 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:43.372+0000 7f6fae800700 1 -- start start 2024-03-30T04:16:51.279 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:43.372+0000 7f6fae800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fa8076c80 0x7f6fa81135e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:51.279 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:43.372+0000 7f6fae800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa8113b20 con 0x7f6fa8076c80 2024-03-30T04:16:51.280 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:43.373+0000 7f6fa7400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fa8076c80 0x7f6fa81135e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:51.280 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:43.373+0000 7f6fa7400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fa8076c80 0x7f6fa81135e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44868/0 (socket says 172.21.15.123:44868) 2024-03-30T04:16:51.280 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:43.373+0000 7f6fa7400700 1 -- 172.21.15.123:0/2617770857 learned_addr learned my addr 172.21.15.123:0/2617770857 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:16:51.280 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:43.373+0000 7f6fa7400700 1 -- 172.21.15.123:0/2617770857 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f980094f0 con 0x7f6fa8076c80 2024-03-30T04:16:51.280 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:43.373+0000 7f6fa7400700 1 --2- 172.21.15.123:0/2617770857 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fa8076c80 0x7f6fa81135e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6f98006b00 tx=0x7f6f98009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:51.280 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:43.374+0000 7f6f9f400700 1 -- 172.21.15.123:0/2617770857 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f98010a80 con 0x7f6fa8076c80 2024-03-30T04:16:51.280 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:43.374+0000 7f6f9f400700 1 -- 172.21.15.123:0/2617770857 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f6f9801a430 con 0x7f6fa8076c80 2024-03-30T04:16:51.281 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:43.374+0000 7f6fae800700 1 -- 172.21.15.123:0/2617770857 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fa8113d20 con 0x7f6fa8076c80 2024-03-30T04:16:51.281 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:43.374+0000 7f6f9f400700 1 -- 172.21.15.123:0/2617770857 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f980245b0 con 0x7f6fa8076c80 2024-03-30T04:16:51.281 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:43.374+0000 7f6fae800700 1 -- 172.21.15.123:0/2617770857 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fa8114840 con 0x7f6fa8076c80 2024-03-30T04:16:51.281 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:43.374+0000 7f6f9f400700 1 -- 172.21.15.123:0/2617770857 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f6f98024a60 con 0x7f6fa8076c80 2024-03-30T04:16:51.281 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:43.374+0000 7f6f9f400700 1 -- 172.21.15.123:0/2617770857 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f6f9802eaa0 con 0x7f6fa8076c80 2024-03-30T04:16:51.281 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:43.375+0000 7f6fae800700 1 -- 172.21.15.123:0/2617770857 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fa805f5f0 con 0x7f6fa8076c80 2024-03-30T04:16:51.281 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:43.377+0000 7f6f9f400700 1 -- 172.21.15.123:0/2617770857 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f6f9802ed00 con 0x7f6fa8076c80 2024-03-30T04:16:51.281 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:43.450+0000 7f6fae800700 1 -- 172.21.15.123:0/2617770857 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f6fa81065b0 con 0x7f6fa8076c80 2024-03-30T04:16:51.282 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:43.451+0000 7f6f9f400700 1 -- 172.21.15.123:0/2617770857 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f6f98010be0 con 0x7f6fa8076c80 2024-03-30T04:16:51.282 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:43.452+0000 7f6fae800700 1 -- 172.21.15.123:0/2617770857 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fa8076c80 msgr2=0x7f6fa81135e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:51.282 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:43.452+0000 7f6fae800700 1 --2- 172.21.15.123:0/2617770857 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fa8076c80 0x7f6fa81135e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6f98006b00 tx=0x7f6f98009250 comp rx=0 tx=0).stop 2024-03-30T04:16:51.282 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:43.452+0000 7f6fae800700 1 -- 172.21.15.123:0/2617770857 shutdown_connections 2024-03-30T04:16:51.282 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:43.452+0000 7f6fae800700 1 --2- 172.21.15.123:0/2617770857 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fa8076c80 0x7f6fa81135e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:51.282 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:43.452+0000 7f6fae800700 1 -- 172.21.15.123:0/2617770857 >> 172.21.15.123:0/2617770857 conn(0x7f6fa80fcc60 msgr2=0x7f6fa80fd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:51.282 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:43.452+0000 7f6fae800700 1 -- 172.21.15.123:0/2617770857 shutdown_connections 2024-03-30T04:16:51.282 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:43.452+0000 7f6fae800700 1 -- 172.21.15.123:0/2617770857 wait complete. 2024-03-30T04:16:51.282 INFO:teuthology.orchestra.run.smithi123.stdout:mgr not available, waiting (2/15)... 2024-03-30T04:16:54.386 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:54 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/1483378812' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-30T04:16:54.661 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout 2024-03-30T04:16:54.661 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout { 2024-03-30T04:16:54.661 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "fsid": "2e77a2be-ee4c-11ee-b647-cb9ed24678a4", 2024-03-30T04:16:54.662 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "health": { 2024-03-30T04:16:54.662 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-30T04:16:54.662 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-30T04:16:54.662 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-30T04:16:54.662 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout }, 2024-03-30T04:16:54.662 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-30T04:16:54.662 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-30T04:16:54.662 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout 0 2024-03-30T04:16:54.662 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout ], 2024-03-30T04:16:54.662 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-30T04:16:54.662 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "smithi123" 2024-03-30T04:16:54.662 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout ], 2024-03-30T04:16:54.662 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "quorum_age": 16, 2024-03-30T04:16:54.662 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-30T04:16:54.663 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T04:16:54.663 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-03-30T04:16:54.663 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-30T04:16:54.663 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout }, 2024-03-30T04:16:54.663 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-30T04:16:54.663 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T04:16:54.663 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-30T04:16:54.663 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-30T04:16:54.663 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-30T04:16:54.672 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-30T04:16:54.672 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-30T04:16:54.672 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-30T04:16:54.672 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout }, 2024-03-30T04:16:54.672 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-30T04:16:54.672 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-30T04:16:54.672 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-30T04:16:54.672 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-30T04:16:54.672 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-30T04:16:54.672 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-30T04:16:54.673 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-30T04:16:54.673 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-30T04:16:54.673 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-30T04:16:54.673 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout }, 2024-03-30T04:16:54.673 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-30T04:16:54.673 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T04:16:54.673 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-30T04:16:54.673 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-30T04:16:54.673 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout }, 2024-03-30T04:16:54.673 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-30T04:16:54.673 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-30T04:16:54.673 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-30T04:16:54.673 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-30T04:16:54.673 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-30T04:16:54.674 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-30T04:16:54.674 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "restful" 2024-03-30T04:16:54.674 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout ], 2024-03-30T04:16:54.674 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-30T04:16:54.674 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout }, 2024-03-30T04:16:54.674 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-30T04:16:54.674 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T04:16:54.674 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "modified": "2024-03-30T04:16:30.949937+0000", 2024-03-30T04:16:54.674 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-30T04:16:54.674 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout }, 2024-03-30T04:16:54.674 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-30T04:16:54.674 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout } 2024-03-30T04:16:54.674 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:53.816+0000 7f08f1a00700 1 Processor -- start 2024-03-30T04:16:54.675 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:53.816+0000 7f08f1a00700 1 -- start start 2024-03-30T04:16:54.675 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:53.816+0000 7f08f1a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08ec078d30 0x7f08ec077240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:54.675 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:53.816+0000 7f08f1a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08ec079140 con 0x7f08ec078d30 2024-03-30T04:16:54.675 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:53.816+0000 7f08ebe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08ec078d30 0x7f08ec077240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:54.675 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:53.816+0000 7f08ebe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08ec078d30 0x7f08ec077240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48444/0 (socket says 172.21.15.123:48444) 2024-03-30T04:16:54.675 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:53.816+0000 7f08ebe00700 1 -- 172.21.15.123:0/1571761559 learned_addr learned my addr 172.21.15.123:0/1571761559 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:16:54.675 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:53.817+0000 7f08ebe00700 1 -- 172.21.15.123:0/1571761559 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08ec077780 con 0x7f08ec078d30 2024-03-30T04:16:54.675 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:53.817+0000 7f08ebe00700 1 --2- 172.21.15.123:0/1571761559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08ec078d30 0x7f08ec077240 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f08dc0098e0 tx=0x7f08dc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66695916030bb3a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:54.675 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:53.817+0000 7f08eaa00700 1 -- 172.21.15.123:0/1571761559 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f08dc00b860 con 0x7f08ec078d30 2024-03-30T04:16:54.675 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:53.817+0000 7f08eaa00700 1 -- 172.21.15.123:0/1571761559 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f08dc00be30 con 0x7f08ec078d30 2024-03-30T04:16:54.675 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:53.817+0000 7f08eaa00700 1 -- 172.21.15.123:0/1571761559 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f08dc010380 con 0x7f08ec078d30 2024-03-30T04:16:54.675 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:53.818+0000 7f08f1a00700 1 -- 172.21.15.123:0/1571761559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08ec078d30 msgr2=0x7f08ec077240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:54.675 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:53.818+0000 7f08f1a00700 1 --2- 172.21.15.123:0/1571761559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08ec078d30 0x7f08ec077240 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f08dc0098e0 tx=0x7f08dc009250 comp rx=0 tx=0).stop 2024-03-30T04:16:54.676 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:53.818+0000 7f08f1a00700 1 -- 172.21.15.123:0/1571761559 shutdown_connections 2024-03-30T04:16:54.676 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:53.818+0000 7f08f1a00700 1 --2- 172.21.15.123:0/1571761559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08ec078d30 0x7f08ec077240 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:54.676 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:53.818+0000 7f08f1a00700 1 -- 172.21.15.123:0/1571761559 >> 172.21.15.123:0/1571761559 conn(0x7f08ec0fdb90 msgr2=0x7f08ec0fffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:54.676 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:53.818+0000 7f08f1a00700 1 -- 172.21.15.123:0/1571761559 shutdown_connections 2024-03-30T04:16:54.676 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:53.818+0000 7f08f1a00700 1 -- 172.21.15.123:0/1571761559 wait complete. 2024-03-30T04:16:54.676 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:53.819+0000 7f08f1a00700 1 Processor -- start 2024-03-30T04:16:54.676 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:53.819+0000 7f08f1a00700 1 -- start start 2024-03-30T04:16:54.676 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:53.819+0000 7f08f1a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08ec078d30 0x7f08ec113460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:54.676 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:53.819+0000 7f08f1a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08ec1139a0 con 0x7f08ec078d30 2024-03-30T04:16:54.676 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:53.820+0000 7f08ebe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08ec078d30 0x7f08ec113460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:54.676 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:53.820+0000 7f08ebe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08ec078d30 0x7f08ec113460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48458/0 (socket says 172.21.15.123:48458) 2024-03-30T04:16:54.676 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:53.820+0000 7f08ebe00700 1 -- 172.21.15.123:0/1483378812 learned_addr learned my addr 172.21.15.123:0/1483378812 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:16:54.676 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:53.820+0000 7f08ebe00700 1 -- 172.21.15.123:0/1483378812 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08dc0094f0 con 0x7f08ec078d30 2024-03-30T04:16:54.676 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:53.820+0000 7f08ebe00700 1 --2- 172.21.15.123:0/1483378812 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08ec078d30 0x7f08ec113460 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f08dc006b00 tx=0x7f08dc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:54.677 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:53.820+0000 7f08e3e00700 1 -- 172.21.15.123:0/1483378812 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f08dc010750 con 0x7f08ec078d30 2024-03-30T04:16:54.677 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:53.820+0000 7f08e3e00700 1 -- 172.21.15.123:0/1483378812 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f08dc010d20 con 0x7f08ec078d30 2024-03-30T04:16:54.677 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:53.820+0000 7f08f1a00700 1 -- 172.21.15.123:0/1483378812 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08ec113ba0 con 0x7f08ec078d30 2024-03-30T04:16:54.677 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:53.820+0000 7f08e3e00700 1 -- 172.21.15.123:0/1483378812 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f08dc01acf0 con 0x7f08ec078d30 2024-03-30T04:16:54.677 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:53.820+0000 7f08f1a00700 1 -- 172.21.15.123:0/1483378812 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08ec114710 con 0x7f08ec078d30 2024-03-30T04:16:54.677 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:53.821+0000 7f08e3e00700 1 -- 172.21.15.123:0/1483378812 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7f08dc02b440 con 0x7f08ec078d30 2024-03-30T04:16:54.677 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:53.821+0000 7f08f1a00700 1 -- 172.21.15.123:0/1483378812 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08ec05f5f0 con 0x7f08ec078d30 2024-03-30T04:16:54.677 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:53.822+0000 7f08e3e00700 1 --2- 172.21.15.123:0/1483378812 >> [v2:172.21.15.123:6800/1291592055,v1:172.21.15.123:6801/1291592055] conn(0x7f08d0036790 0x7f08d0038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:54.677 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:53.822+0000 7f08e3e00700 1 -- 172.21.15.123:0/1483378812 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f08dc01e070 con 0x7f08ec078d30 2024-03-30T04:16:54.677 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:53.822+0000 7f08eb400700 1 --2- 172.21.15.123:0/1483378812 >> [v2:172.21.15.123:6800/1291592055,v1:172.21.15.123:6801/1291592055] conn(0x7f08d0036790 0x7f08d0038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:54.677 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:53.823+0000 7f08eb400700 1 --2- 172.21.15.123:0/1483378812 >> [v2:172.21.15.123:6800/1291592055,v1:172.21.15.123:6801/1291592055] conn(0x7f08d0036790 0x7f08d0038c40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f08d8009990 tx=0x7f08d8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:54.677 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:53.828+0000 7f08e3e00700 1 -- 172.21.15.123:0/1483378812 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f08dc02b6f0 con 0x7f08ec078d30 2024-03-30T04:16:54.677 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:54.097+0000 7f08f1a00700 1 -- 172.21.15.123:0/1483378812 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f08ec04bed0 con 0x7f08ec078d30 2024-03-30T04:16:54.677 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:54.097+0000 7f08e3e00700 1 -- 172.21.15.123:0/1483378812 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f08dc018b00 con 0x7f08ec078d30 2024-03-30T04:16:54.678 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:54.100+0000 7f08f1a00700 1 -- 172.21.15.123:0/1483378812 >> [v2:172.21.15.123:6800/1291592055,v1:172.21.15.123:6801/1291592055] conn(0x7f08d0036790 msgr2=0x7f08d0038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:54.678 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:54.100+0000 7f08f1a00700 1 --2- 172.21.15.123:0/1483378812 >> [v2:172.21.15.123:6800/1291592055,v1:172.21.15.123:6801/1291592055] conn(0x7f08d0036790 0x7f08d0038c40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f08d8009990 tx=0x7f08d8006e40 comp rx=0 tx=0).stop 2024-03-30T04:16:54.678 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:54.100+0000 7f08f1a00700 1 -- 172.21.15.123:0/1483378812 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08ec078d30 msgr2=0x7f08ec113460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:54.678 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:54.100+0000 7f08f1a00700 1 --2- 172.21.15.123:0/1483378812 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08ec078d30 0x7f08ec113460 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f08dc006b00 tx=0x7f08dc009250 comp rx=0 tx=0).stop 2024-03-30T04:16:54.678 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:54.100+0000 7f08f1a00700 1 -- 172.21.15.123:0/1483378812 shutdown_connections 2024-03-30T04:16:54.678 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:54.100+0000 7f08f1a00700 1 --2- 172.21.15.123:0/1483378812 >> [v2:172.21.15.123:6800/1291592055,v1:172.21.15.123:6801/1291592055] conn(0x7f08d0036790 0x7f08d0038c40 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:54.678 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:54.100+0000 7f08f1a00700 1 --2- 172.21.15.123:0/1483378812 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08ec078d30 0x7f08ec113460 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:54.678 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:54.100+0000 7f08f1a00700 1 -- 172.21.15.123:0/1483378812 >> 172.21.15.123:0/1483378812 conn(0x7f08ec0fdb90 msgr2=0x7f08ec0fe720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:54.678 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:54.100+0000 7f08f1a00700 1 -- 172.21.15.123:0/1483378812 shutdown_connections 2024-03-30T04:16:54.678 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:54.100+0000 7f08f1a00700 1 -- 172.21.15.123:0/1483378812 wait complete. 2024-03-30T04:16:54.678 INFO:teuthology.orchestra.run.smithi123.stdout:mgr is available 2024-03-30T04:16:55.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:55 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/466510472' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-30T04:16:56.104 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout 2024-03-30T04:16:56.104 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout [global] 2024-03-30T04:16:56.105 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout fsid = 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 2024-03-30T04:16:56.105 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-30T04:16:56.105 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-30T04:16:56.105 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-30T04:16:56.105 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-30T04:16:56.105 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout 2024-03-30T04:16:56.105 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-30T04:16:56.105 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-30T04:16:56.105 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout 2024-03-30T04:16:56.105 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout [osd] 2024-03-30T04:16:56.105 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-30T04:16:56.105 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-30T04:16:56.105 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-30T04:16:56.106 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-30T04:16:56.106 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-30T04:16:56.106 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:55.221+0000 7f3e25c00700 1 Processor -- start 2024-03-30T04:16:56.106 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:55.222+0000 7f3e25c00700 1 -- start start 2024-03-30T04:16:56.106 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:55.222+0000 7f3e25c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e20102580 0x7f3e20102990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:56.106 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:55.222+0000 7f3e25c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e20102ed0 con 0x7f3e20102580 2024-03-30T04:16:56.106 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:55.222+0000 7f3e1ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e20102580 0x7f3e20102990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:56.106 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:55.222+0000 7f3e1ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e20102580 0x7f3e20102990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42326/0 (socket says 172.21.15.123:42326) 2024-03-30T04:16:56.106 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:55.222+0000 7f3e1ea00700 1 -- 172.21.15.123:0/989699941 learned_addr learned my addr 172.21.15.123:0/989699941 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:16:56.106 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:55.223+0000 7f3e1ea00700 1 -- 172.21.15.123:0/989699941 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e20103010 con 0x7f3e20102580 2024-03-30T04:16:56.106 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:55.223+0000 7f3e1ea00700 1 --2- 172.21.15.123:0/989699941 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e20102580 0x7f3e20102990 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f3e100098e0 tx=0x7f3e10009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84d1dc9cc8902e85 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:56.106 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:55.223+0000 7f3e1d600700 1 -- 172.21.15.123:0/989699941 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e1000b860 con 0x7f3e20102580 2024-03-30T04:16:56.106 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:55.223+0000 7f3e1d600700 1 -- 172.21.15.123:0/989699941 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f3e1000be30 con 0x7f3e20102580 2024-03-30T04:16:56.107 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:55.223+0000 7f3e1d600700 1 -- 172.21.15.123:0/989699941 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e10010380 con 0x7f3e20102580 2024-03-30T04:16:56.107 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:55.224+0000 7f3e25c00700 1 -- 172.21.15.123:0/989699941 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e20102580 msgr2=0x7f3e20102990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:56.107 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:55.224+0000 7f3e25c00700 1 --2- 172.21.15.123:0/989699941 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e20102580 0x7f3e20102990 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f3e100098e0 tx=0x7f3e10009250 comp rx=0 tx=0).stop 2024-03-30T04:16:56.107 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:55.224+0000 7f3e25c00700 1 -- 172.21.15.123:0/989699941 shutdown_connections 2024-03-30T04:16:56.107 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:55.224+0000 7f3e25c00700 1 --2- 172.21.15.123:0/989699941 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e20102580 0x7f3e20102990 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:56.107 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:55.224+0000 7f3e25c00700 1 -- 172.21.15.123:0/989699941 >> 172.21.15.123:0/989699941 conn(0x7f3e200fdd00 msgr2=0x7f3e20100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:56.107 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:55.224+0000 7f3e25c00700 1 -- 172.21.15.123:0/989699941 shutdown_connections 2024-03-30T04:16:56.107 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:55.224+0000 7f3e25c00700 1 -- 172.21.15.123:0/989699941 wait complete. 2024-03-30T04:16:56.107 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:55.225+0000 7f3e25c00700 1 Processor -- start 2024-03-30T04:16:56.107 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:55.225+0000 7f3e25c00700 1 -- start start 2024-03-30T04:16:56.107 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:55.225+0000 7f3e25c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e20102580 0x7f3e20113540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:56.107 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:55.225+0000 7f3e25c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e20113a80 con 0x7f3e20102580 2024-03-30T04:16:56.107 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:55.225+0000 7f3e1ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e20102580 0x7f3e20113540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:56.108 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:55.225+0000 7f3e1ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e20102580 0x7f3e20113540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42342/0 (socket says 172.21.15.123:42342) 2024-03-30T04:16:56.108 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:55.225+0000 7f3e1ea00700 1 -- 172.21.15.123:0/466510472 learned_addr learned my addr 172.21.15.123:0/466510472 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:16:56.108 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:55.226+0000 7f3e1ea00700 1 -- 172.21.15.123:0/466510472 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e100094f0 con 0x7f3e20102580 2024-03-30T04:16:56.108 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:55.226+0000 7f3e1ea00700 1 --2- 172.21.15.123:0/466510472 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e20102580 0x7f3e20113540 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3e10000c00 tx=0x7f3e10009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:56.108 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:55.226+0000 7f3e16a00700 1 -- 172.21.15.123:0/466510472 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e10010a80 con 0x7f3e20102580 2024-03-30T04:16:56.108 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:55.226+0000 7f3e16a00700 1 -- 172.21.15.123:0/466510472 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f3e1001a430 con 0x7f3e20102580 2024-03-30T04:16:56.108 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:55.226+0000 7f3e25c00700 1 -- 172.21.15.123:0/466510472 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e20113c80 con 0x7f3e20102580 2024-03-30T04:16:56.108 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:55.226+0000 7f3e16a00700 1 -- 172.21.15.123:0/466510472 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e100245b0 con 0x7f3e20102580 2024-03-30T04:16:56.108 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:55.226+0000 7f3e25c00700 1 -- 172.21.15.123:0/466510472 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e201147f0 con 0x7f3e20102580 2024-03-30T04:16:56.108 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:55.227+0000 7f3e16a00700 1 -- 172.21.15.123:0/466510472 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7f3e1001a5a0 con 0x7f3e20102580 2024-03-30T04:16:56.108 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:55.227+0000 7f3e25c00700 1 -- 172.21.15.123:0/466510472 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e2005f5f0 con 0x7f3e20102580 2024-03-30T04:16:56.108 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:55.228+0000 7f3e16a00700 1 --2- 172.21.15.123:0/466510472 >> [v2:172.21.15.123:6800/1291592055,v1:172.21.15.123:6801/1291592055] conn(0x7f3e04036740 0x7f3e04038bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:56.108 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:55.228+0000 7f3e16a00700 1 -- 172.21.15.123:0/466510472 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f3e1004ace0 con 0x7f3e20102580 2024-03-30T04:16:56.108 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:55.228+0000 7f3e1e000700 1 --2- 172.21.15.123:0/466510472 >> [v2:172.21.15.123:6800/1291592055,v1:172.21.15.123:6801/1291592055] conn(0x7f3e04036740 0x7f3e04038bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:56.109 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:55.228+0000 7f3e1e000700 1 --2- 172.21.15.123:0/466510472 >> [v2:172.21.15.123:6800/1291592055,v1:172.21.15.123:6801/1291592055] conn(0x7f3e04036740 0x7f3e04038bf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c009990 tx=0x7f3e0c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:56.109 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:55.232+0000 7f3e16a00700 1 -- 172.21.15.123:0/466510472 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f3e10023dd0 con 0x7f3e20102580 2024-03-30T04:16:56.109 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:55.425+0000 7f3e25c00700 1 -- 172.21.15.123:0/466510472 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f3e2004bed0 con 0x7f3e20102580 2024-03-30T04:16:56.109 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:55.426+0000 7f3e16a00700 1 -- 172.21.15.123:0/466510472 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f3e10023390 con 0x7f3e20102580 2024-03-30T04:16:56.109 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:55.428+0000 7f3e25c00700 1 -- 172.21.15.123:0/466510472 >> [v2:172.21.15.123:6800/1291592055,v1:172.21.15.123:6801/1291592055] conn(0x7f3e04036740 msgr2=0x7f3e04038bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:56.109 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:55.428+0000 7f3e25c00700 1 --2- 172.21.15.123:0/466510472 >> [v2:172.21.15.123:6800/1291592055,v1:172.21.15.123:6801/1291592055] conn(0x7f3e04036740 0x7f3e04038bf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c009990 tx=0x7f3e0c006e40 comp rx=0 tx=0).stop 2024-03-30T04:16:56.109 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:55.428+0000 7f3e25c00700 1 -- 172.21.15.123:0/466510472 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e20102580 msgr2=0x7f3e20113540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:56.109 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:55.428+0000 7f3e25c00700 1 --2- 172.21.15.123:0/466510472 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e20102580 0x7f3e20113540 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3e10000c00 tx=0x7f3e10009250 comp rx=0 tx=0).stop 2024-03-30T04:16:56.109 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:55.428+0000 7f3e25c00700 1 -- 172.21.15.123:0/466510472 shutdown_connections 2024-03-30T04:16:56.109 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:55.428+0000 7f3e25c00700 1 --2- 172.21.15.123:0/466510472 >> [v2:172.21.15.123:6800/1291592055,v1:172.21.15.123:6801/1291592055] conn(0x7f3e04036740 0x7f3e04038bf0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:56.109 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:55.428+0000 7f3e25c00700 1 --2- 172.21.15.123:0/466510472 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e20102580 0x7f3e20113540 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:56.109 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:55.428+0000 7f3e25c00700 1 -- 172.21.15.123:0/466510472 >> 172.21.15.123:0/466510472 conn(0x7f3e200fdd00 msgr2=0x7f3e200fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:56.109 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:55.428+0000 7f3e25c00700 1 -- 172.21.15.123:0/466510472 shutdown_connections 2024-03-30T04:16:56.109 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:55.428+0000 7f3e25c00700 1 -- 172.21.15.123:0/466510472 wait complete. 2024-03-30T04:16:56.110 INFO:teuthology.orchestra.run.smithi123.stdout:Enabling cephadm module... 2024-03-30T04:16:57.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:56 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/967674587' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-30T04:16:57.643 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.620+0000 7ff8bba00700 1 Processor -- start 2024-03-30T04:16:57.644 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.620+0000 7ff8bba00700 1 -- start start 2024-03-30T04:16:57.644 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.620+0000 7ff8bba00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8b4078d60 0x7ff8b4077260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:57.644 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.620+0000 7ff8bba00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b4079170 con 0x7ff8b4078d60 2024-03-30T04:16:57.644 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.621+0000 7ff8b9000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8b4078d60 0x7ff8b4077260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:57.644 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.621+0000 7ff8b9000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8b4078d60 0x7ff8b4077260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42346/0 (socket says 172.21.15.123:42346) 2024-03-30T04:16:57.645 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.621+0000 7ff8b9000700 1 -- 172.21.15.123:0/3970950859 learned_addr learned my addr 172.21.15.123:0/3970950859 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:16:57.645 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.621+0000 7ff8b9000700 1 -- 172.21.15.123:0/3970950859 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b40777a0 con 0x7ff8b4078d60 2024-03-30T04:16:57.645 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.621+0000 7ff8b9000700 1 --2- 172.21.15.123:0/3970950859 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8b4078d60 0x7ff8b4077260 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff8a80098e0 tx=0x7ff8a8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e66a64cc0c594574 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:57.645 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.622+0000 7ff8b3400700 1 -- 172.21.15.123:0/3970950859 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8a800b860 con 0x7ff8b4078d60 2024-03-30T04:16:57.645 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.622+0000 7ff8b3400700 1 -- 172.21.15.123:0/3970950859 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7ff8a800be30 con 0x7ff8b4078d60 2024-03-30T04:16:57.645 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.622+0000 7ff8b3400700 1 -- 172.21.15.123:0/3970950859 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8a8010380 con 0x7ff8b4078d60 2024-03-30T04:16:57.645 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.622+0000 7ff8bba00700 1 -- 172.21.15.123:0/3970950859 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8b4078d60 msgr2=0x7ff8b4077260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:57.646 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.622+0000 7ff8bba00700 1 --2- 172.21.15.123:0/3970950859 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8b4078d60 0x7ff8b4077260 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff8a80098e0 tx=0x7ff8a8009250 comp rx=0 tx=0).stop 2024-03-30T04:16:57.646 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.622+0000 7ff8bba00700 1 -- 172.21.15.123:0/3970950859 shutdown_connections 2024-03-30T04:16:57.646 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.622+0000 7ff8bba00700 1 --2- 172.21.15.123:0/3970950859 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8b4078d60 0x7ff8b4077260 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:57.646 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.622+0000 7ff8bba00700 1 -- 172.21.15.123:0/3970950859 >> 172.21.15.123:0/3970950859 conn(0x7ff8b40fdcd0 msgr2=0x7ff8b4100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:57.646 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.622+0000 7ff8bba00700 1 -- 172.21.15.123:0/3970950859 shutdown_connections 2024-03-30T04:16:57.647 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.622+0000 7ff8bba00700 1 -- 172.21.15.123:0/3970950859 wait complete. 2024-03-30T04:16:57.647 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.623+0000 7ff8bba00700 1 Processor -- start 2024-03-30T04:16:57.647 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.623+0000 7ff8bba00700 1 -- start start 2024-03-30T04:16:57.647 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.623+0000 7ff8bba00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8b4078d60 0x7ff8b4113630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:57.648 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.623+0000 7ff8bba00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b4113b70 con 0x7ff8b4078d60 2024-03-30T04:16:57.648 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.624+0000 7ff8b9000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8b4078d60 0x7ff8b4113630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:57.648 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.624+0000 7ff8b9000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8b4078d60 0x7ff8b4113630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42360/0 (socket says 172.21.15.123:42360) 2024-03-30T04:16:57.648 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.624+0000 7ff8b9000700 1 -- 172.21.15.123:0/967674587 learned_addr learned my addr 172.21.15.123:0/967674587 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:16:57.648 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.624+0000 7ff8b9000700 1 -- 172.21.15.123:0/967674587 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a80094f0 con 0x7ff8b4078d60 2024-03-30T04:16:57.649 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.624+0000 7ff8b9000700 1 --2- 172.21.15.123:0/967674587 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8b4078d60 0x7ff8b4113630 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff8a8006b00 tx=0x7ff8a8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:57.649 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.624+0000 7ff8b0c00700 1 -- 172.21.15.123:0/967674587 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8a8010750 con 0x7ff8b4078d60 2024-03-30T04:16:57.649 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.625+0000 7ff8b0c00700 1 -- 172.21.15.123:0/967674587 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7ff8a8010d20 con 0x7ff8b4078d60 2024-03-30T04:16:57.649 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.625+0000 7ff8bba00700 1 -- 172.21.15.123:0/967674587 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8b4113d70 con 0x7ff8b4078d60 2024-03-30T04:16:57.649 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.625+0000 7ff8b0c00700 1 -- 172.21.15.123:0/967674587 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8a801acf0 con 0x7ff8b4078d60 2024-03-30T04:16:57.649 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.625+0000 7ff8bba00700 1 -- 172.21.15.123:0/967674587 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8b41148e0 con 0x7ff8b4078d60 2024-03-30T04:16:57.649 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.626+0000 7ff8b0c00700 1 -- 172.21.15.123:0/967674587 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7ff8a802b440 con 0x7ff8b4078d60 2024-03-30T04:16:57.649 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.626+0000 7ff8bba00700 1 -- 172.21.15.123:0/967674587 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8b405f5f0 con 0x7ff8b4078d60 2024-03-30T04:16:57.649 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.626+0000 7ff8b0c00700 1 --2- 172.21.15.123:0/967674587 >> [v2:172.21.15.123:6800/1291592055,v1:172.21.15.123:6801/1291592055] conn(0x7ff89c036790 0x7ff89c038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:57.650 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.626+0000 7ff8b0c00700 1 -- 172.21.15.123:0/967674587 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7ff8a801e070 con 0x7ff8b4078d60 2024-03-30T04:16:57.650 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.626+0000 7ff8b3e00700 1 --2- 172.21.15.123:0/967674587 >> [v2:172.21.15.123:6800/1291592055,v1:172.21.15.123:6801/1291592055] conn(0x7ff89c036790 0x7ff89c038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:57.650 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.627+0000 7ff8b3e00700 1 --2- 172.21.15.123:0/967674587 >> [v2:172.21.15.123:6800/1291592055,v1:172.21.15.123:6801/1291592055] conn(0x7ff89c036790 0x7ff89c038c40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff8a0009990 tx=0x7ff8a0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:57.650 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.631+0000 7ff8b0c00700 1 -- 172.21.15.123:0/967674587 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7ff8a802b6f0 con 0x7ff8b4078d60 2024-03-30T04:16:57.650 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.874+0000 7ff8bba00700 1 -- 172.21.15.123:0/967674587 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7ff8b404bed0 con 0x7ff8b4078d60 2024-03-30T04:16:57.650 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.927+0000 7ff8b0c00700 1 -- 172.21.15.123:0/967674587 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7ff8a801a5a0 con 0x7ff8b4078d60 2024-03-30T04:16:57.650 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.929+0000 7ff8b0c00700 1 -- 172.21.15.123:0/967674587 <== mon.0 v2:172.21.15.123:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7ff8a80298e0 con 0x7ff8b4078d60 2024-03-30T04:16:57.651 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.932+0000 7ff8bba00700 1 -- 172.21.15.123:0/967674587 >> [v2:172.21.15.123:6800/1291592055,v1:172.21.15.123:6801/1291592055] conn(0x7ff89c036790 msgr2=0x7ff89c038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:57.651 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.932+0000 7ff8bba00700 1 --2- 172.21.15.123:0/967674587 >> [v2:172.21.15.123:6800/1291592055,v1:172.21.15.123:6801/1291592055] conn(0x7ff89c036790 0x7ff89c038c40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff8a0009990 tx=0x7ff8a0006e40 comp rx=0 tx=0).stop 2024-03-30T04:16:57.651 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.932+0000 7ff8bba00700 1 -- 172.21.15.123:0/967674587 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8b4078d60 msgr2=0x7ff8b4113630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:57.651 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.932+0000 7ff8bba00700 1 --2- 172.21.15.123:0/967674587 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8b4078d60 0x7ff8b4113630 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff8a8006b00 tx=0x7ff8a8009250 comp rx=0 tx=0).stop 2024-03-30T04:16:57.651 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.932+0000 7ff8bba00700 1 -- 172.21.15.123:0/967674587 shutdown_connections 2024-03-30T04:16:57.651 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.932+0000 7ff8bba00700 1 --2- 172.21.15.123:0/967674587 >> [v2:172.21.15.123:6800/1291592055,v1:172.21.15.123:6801/1291592055] conn(0x7ff89c036790 0x7ff89c038c40 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:57.651 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.932+0000 7ff8bba00700 1 --2- 172.21.15.123:0/967674587 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8b4078d60 0x7ff8b4113630 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:57.651 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.932+0000 7ff8bba00700 1 -- 172.21.15.123:0/967674587 >> 172.21.15.123:0/967674587 conn(0x7ff8b40fdcd0 msgr2=0x7ff8b40fe8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:57.652 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.932+0000 7ff8bba00700 1 -- 172.21.15.123:0/967674587 shutdown_connections 2024-03-30T04:16:57.652 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:56.932+0000 7ff8bba00700 1 -- 172.21.15.123:0/967674587 wait complete. 2024-03-30T04:16:58.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:57 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/967674587' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-30T04:16:58.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:57 smithi123 ceph-mon[35245]: mgrmap e5: smithi123.xgksqq(active, since 9s) 2024-03-30T04:16:59.047 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout { 2024-03-30T04:16:59.047 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-30T04:16:59.047 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-30T04:16:59.047 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "active_name": "smithi123.xgksqq", 2024-03-30T04:16:59.047 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-30T04:16:59.047 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout } 2024-03-30T04:16:59.047 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.180+0000 7fe445600700 1 Processor -- start 2024-03-30T04:16:59.048 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.180+0000 7fe445600700 1 -- start start 2024-03-30T04:16:59.048 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.180+0000 7fe445600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe440076c00 0x7fe440107630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:59.048 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.180+0000 7fe445600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe440103020 con 0x7fe440076c00 2024-03-30T04:16:59.048 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.181+0000 7fe43ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe440076c00 0x7fe440107630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:59.048 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.181+0000 7fe43ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe440076c00 0x7fe440107630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42380/0 (socket says 172.21.15.123:42380) 2024-03-30T04:16:59.048 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.181+0000 7fe43ea00700 1 -- 172.21.15.123:0/1784511263 learned_addr learned my addr 172.21.15.123:0/1784511263 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:16:59.048 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.181+0000 7fe43ea00700 1 -- 172.21.15.123:0/1784511263 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe440107b70 con 0x7fe440076c00 2024-03-30T04:16:59.048 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.181+0000 7fe43ea00700 1 --2- 172.21.15.123:0/1784511263 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe440076c00 0x7fe440107630 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe4300098e0 tx=0x7fe430009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7e11852b685f8bd server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:59.048 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.182+0000 7fe43d600700 1 -- 172.21.15.123:0/1784511263 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe43000b860 con 0x7fe440076c00 2024-03-30T04:16:59.048 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.182+0000 7fe43d600700 1 -- 172.21.15.123:0/1784511263 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fe43000be30 con 0x7fe440076c00 2024-03-30T04:16:59.048 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.182+0000 7fe43d600700 1 -- 172.21.15.123:0/1784511263 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe430010380 con 0x7fe440076c00 2024-03-30T04:16:59.048 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.182+0000 7fe445600700 1 -- 172.21.15.123:0/1784511263 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe440076c00 msgr2=0x7fe440107630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:59.048 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.182+0000 7fe445600700 1 --2- 172.21.15.123:0/1784511263 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe440076c00 0x7fe440107630 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe4300098e0 tx=0x7fe430009250 comp rx=0 tx=0).stop 2024-03-30T04:16:59.048 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.182+0000 7fe445600700 1 -- 172.21.15.123:0/1784511263 shutdown_connections 2024-03-30T04:16:59.049 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.182+0000 7fe445600700 1 --2- 172.21.15.123:0/1784511263 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe440076c00 0x7fe440107630 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:59.049 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.182+0000 7fe445600700 1 -- 172.21.15.123:0/1784511263 >> 172.21.15.123:0/1784511263 conn(0x7fe4400fcc60 msgr2=0x7fe4400ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:59.049 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.183+0000 7fe445600700 1 -- 172.21.15.123:0/1784511263 shutdown_connections 2024-03-30T04:16:59.049 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.183+0000 7fe445600700 1 -- 172.21.15.123:0/1784511263 wait complete. 2024-03-30T04:16:59.049 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.183+0000 7fe445600700 1 Processor -- start 2024-03-30T04:16:59.049 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.184+0000 7fe445600700 1 -- start start 2024-03-30T04:16:59.049 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.184+0000 7fe445600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe440076c00 0x7fe440113570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:59.049 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.184+0000 7fe445600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe440113ab0 con 0x7fe440076c00 2024-03-30T04:16:59.049 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.184+0000 7fe43ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe440076c00 0x7fe440113570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:59.049 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.184+0000 7fe43ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe440076c00 0x7fe440113570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42396/0 (socket says 172.21.15.123:42396) 2024-03-30T04:16:59.049 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.184+0000 7fe43ea00700 1 -- 172.21.15.123:0/561150728 learned_addr learned my addr 172.21.15.123:0/561150728 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:16:59.049 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.185+0000 7fe43ea00700 1 -- 172.21.15.123:0/561150728 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4300094f0 con 0x7fe440076c00 2024-03-30T04:16:59.049 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.185+0000 7fe43ea00700 1 --2- 172.21.15.123:0/561150728 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe440076c00 0x7fe440113570 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe430006b00 tx=0x7fe430009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:59.050 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.185+0000 7fe436a00700 1 -- 172.21.15.123:0/561150728 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe430010a80 con 0x7fe440076c00 2024-03-30T04:16:59.050 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.185+0000 7fe436a00700 1 -- 172.21.15.123:0/561150728 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fe43001a430 con 0x7fe440076c00 2024-03-30T04:16:59.050 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.185+0000 7fe436a00700 1 -- 172.21.15.123:0/561150728 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4300245b0 con 0x7fe440076c00 2024-03-30T04:16:59.050 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.185+0000 7fe445600700 1 -- 172.21.15.123:0/561150728 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe440113cb0 con 0x7fe440076c00 2024-03-30T04:16:59.050 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.185+0000 7fe445600700 1 -- 172.21.15.123:0/561150728 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe440114820 con 0x7fe440076c00 2024-03-30T04:16:59.050 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.186+0000 7fe436a00700 1 -- 172.21.15.123:0/561150728 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fe430024830 con 0x7fe440076c00 2024-03-30T04:16:59.050 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.186+0000 7fe445600700 1 -- 172.21.15.123:0/561150728 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe44005f5f0 con 0x7fe440076c00 2024-03-30T04:16:59.050 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.187+0000 7fe436a00700 1 --2- 172.21.15.123:0/561150728 >> [v2:172.21.15.123:6800/1291592055,v1:172.21.15.123:6801/1291592055] conn(0x7fe424036790 0x7fe424038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:59.050 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.187+0000 7fe436a00700 1 -- 172.21.15.123:0/561150728 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fe43004ad00 con 0x7fe440076c00 2024-03-30T04:16:59.050 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.187+0000 7fe43e000700 1 -- 172.21.15.123:0/561150728 >> [v2:172.21.15.123:6800/1291592055,v1:172.21.15.123:6801/1291592055] conn(0x7fe424036790 msgr2=0x7fe424038c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.123:6800/1291592055 2024-03-30T04:16:59.050 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.187+0000 7fe43e000700 1 --2- 172.21.15.123:0/561150728 >> [v2:172.21.15.123:6800/1291592055,v1:172.21.15.123:6801/1291592055] conn(0x7fe424036790 0x7fe424038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-30T04:16:59.050 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.192+0000 7fe436a00700 1 -- 172.21.15.123:0/561150728 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fe43002cd60 con 0x7fe440076c00 2024-03-30T04:16:59.050 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.388+0000 7fe43e000700 1 -- 172.21.15.123:0/561150728 >> [v2:172.21.15.123:6800/1291592055,v1:172.21.15.123:6801/1291592055] conn(0x7fe424036790 msgr2=0x7fe424038c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.123:6800/1291592055 2024-03-30T04:16:59.051 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.388+0000 7fe43e000700 1 --2- 172.21.15.123:0/561150728 >> [v2:172.21.15.123:6800/1291592055,v1:172.21.15.123:6801/1291592055] conn(0x7fe424036790 0x7fe424038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-30T04:16:59.051 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.448+0000 7fe445600700 1 -- 172.21.15.123:0/561150728 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fe44005f5f0 con 0x7fe440076c00 2024-03-30T04:16:59.051 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.448+0000 7fe436a00700 1 -- 172.21.15.123:0/561150728 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7fe43002cb80 con 0x7fe440076c00 2024-03-30T04:16:59.051 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.451+0000 7fe445600700 1 -- 172.21.15.123:0/561150728 >> [v2:172.21.15.123:6800/1291592055,v1:172.21.15.123:6801/1291592055] conn(0x7fe424036790 msgr2=0x7fe424038c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-30T04:16:59.051 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.451+0000 7fe445600700 1 --2- 172.21.15.123:0/561150728 >> [v2:172.21.15.123:6800/1291592055,v1:172.21.15.123:6801/1291592055] conn(0x7fe424036790 0x7fe424038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:59.051 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.451+0000 7fe445600700 1 -- 172.21.15.123:0/561150728 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe440076c00 msgr2=0x7fe440113570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:59.051 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.451+0000 7fe445600700 1 --2- 172.21.15.123:0/561150728 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe440076c00 0x7fe440113570 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe430006b00 tx=0x7fe430009250 comp rx=0 tx=0).stop 2024-03-30T04:16:59.051 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.451+0000 7fe445600700 1 -- 172.21.15.123:0/561150728 shutdown_connections 2024-03-30T04:16:59.051 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.451+0000 7fe445600700 1 --2- 172.21.15.123:0/561150728 >> [v2:172.21.15.123:6800/1291592055,v1:172.21.15.123:6801/1291592055] conn(0x7fe424036790 0x7fe424038c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:59.051 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.451+0000 7fe445600700 1 --2- 172.21.15.123:0/561150728 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe440076c00 0x7fe440113570 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:59.051 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.451+0000 7fe445600700 1 -- 172.21.15.123:0/561150728 >> 172.21.15.123:0/561150728 conn(0x7fe4400fcc60 msgr2=0x7fe4400fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:59.051 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.451+0000 7fe445600700 1 -- 172.21.15.123:0/561150728 shutdown_connections 2024-03-30T04:16:59.051 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:58.451+0000 7fe445600700 1 -- 172.21.15.123:0/561150728 wait complete. 2024-03-30T04:16:59.051 INFO:teuthology.orchestra.run.smithi123.stdout:Waiting for the mgr to restart... 2024-03-30T04:16:59.051 INFO:teuthology.orchestra.run.smithi123.stdout:Waiting for mgr epoch 5... 2024-03-30T04:16:59.052 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:16:58 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/561150728' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-30T04:17:05.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:05 smithi123 ceph-mon[35245]: Active manager daemon smithi123.xgksqq restarted 2024-03-30T04:17:05.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:05 smithi123 ceph-mon[35245]: Activating manager daemon smithi123.xgksqq 2024-03-30T04:17:05.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:05 smithi123 ceph-mon[35245]: osdmap e2: 0 total, 0 up, 0 in 2024-03-30T04:17:05.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:05 smithi123 ceph-mon[35245]: mgrmap e6: smithi123.xgksqq(active, starting, since 0.00359354s) 2024-03-30T04:17:05.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:05 smithi123 ceph-mon[35245]: from='mgr.14118 172.21.15.123:0/1102803547' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mon metadata", "id": "smithi123"}]: dispatch 2024-03-30T04:17:05.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:05 smithi123 ceph-mon[35245]: from='mgr.14118 172.21.15.123:0/1102803547' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mgr metadata", "who": "smithi123.xgksqq", "id": "smithi123.xgksqq"}]: dispatch 2024-03-30T04:17:05.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:05 smithi123 ceph-mon[35245]: from='mgr.14118 172.21.15.123:0/1102803547' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-30T04:17:05.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:05 smithi123 ceph-mon[35245]: from='mgr.14118 172.21.15.123:0/1102803547' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-30T04:17:05.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:05 smithi123 ceph-mon[35245]: from='mgr.14118 172.21.15.123:0/1102803547' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-30T04:17:05.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:05 smithi123 ceph-mon[35245]: Manager daemon smithi123.xgksqq is now available 2024-03-30T04:17:05.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:05 smithi123 ceph-mon[35245]: from='mgr.14118 172.21.15.123:0/1102803547' entity='mgr.smithi123.xgksqq' 2024-03-30T04:17:05.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:05 smithi123 ceph-mon[35245]: from='mgr.14118 172.21.15.123:0/1102803547' entity='mgr.smithi123.xgksqq' 2024-03-30T04:17:05.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:05 smithi123 ceph-mon[35245]: from='mgr.14118 172.21.15.123:0/1102803547' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:17:06.434 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:06 smithi123 ceph-mon[35245]: Found migration_current of "None". Setting to last migration. 2024-03-30T04:17:06.434 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:06 smithi123 ceph-mon[35245]: from='mgr.14118 172.21.15.123:0/1102803547' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:17:06.434 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:06 smithi123 ceph-mon[35245]: from='mgr.14118 172.21.15.123:0/1102803547' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:17:06.434 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:06 smithi123 ceph-mon[35245]: from='mgr.14118 172.21.15.123:0/1102803547' entity='mgr.smithi123.xgksqq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi123.xgksqq/mirror_snapshot_schedule"}]: dispatch 2024-03-30T04:17:06.434 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:06 smithi123 ceph-mon[35245]: from='mgr.14118 172.21.15.123:0/1102803547' entity='mgr.smithi123.xgksqq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi123.xgksqq/trash_purge_schedule"}]: dispatch 2024-03-30T04:17:06.435 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:06 smithi123 ceph-mon[35245]: [30/Mar/2024:04:17:05] ENGINE Bus STARTING 2024-03-30T04:17:06.435 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:06 smithi123 ceph-mon[35245]: [30/Mar/2024:04:17:05] ENGINE Serving on https://172.21.15.123:7150 2024-03-30T04:17:06.435 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:06 smithi123 ceph-mon[35245]: [30/Mar/2024:04:17:05] ENGINE Bus STARTED 2024-03-30T04:17:06.435 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:06 smithi123 ceph-mon[35245]: from='mgr.14118 172.21.15.123:0/1102803547' entity='mgr.smithi123.xgksqq' 2024-03-30T04:17:06.435 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:06 smithi123 ceph-mon[35245]: from='mgr.14118 172.21.15.123:0/1102803547' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:17:06.435 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:06 smithi123 ceph-mon[35245]: mgrmap e7: smithi123.xgksqq(active, since 1.0067s) 2024-03-30T04:17:06.764 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout { 2024-03-30T04:17:06.765 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-30T04:17:06.765 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-30T04:17:06.765 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout } 2024-03-30T04:17:06.765 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:59.618+0000 7fb15f000700 1 Processor -- start 2024-03-30T04:17:06.765 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:59.618+0000 7fb15f000700 1 -- start start 2024-03-30T04:17:06.765 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:59.619+0000 7fb15f000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb158076c00 0x7fb158107630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:06.765 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:59.619+0000 7fb15f000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb158103020 con 0x7fb158076c00 2024-03-30T04:17:06.765 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:59.619+0000 7fb157e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb158076c00 0x7fb158107630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:06.765 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:59.619+0000 7fb157e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb158076c00 0x7fb158107630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42402/0 (socket says 172.21.15.123:42402) 2024-03-30T04:17:06.765 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:59.619+0000 7fb157e00700 1 -- 172.21.15.123:0/1983890221 learned_addr learned my addr 172.21.15.123:0/1983890221 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:17:06.765 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:59.620+0000 7fb157e00700 1 -- 172.21.15.123:0/1983890221 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb158107b70 con 0x7fb158076c00 2024-03-30T04:17:06.765 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:59.620+0000 7fb157e00700 1 --2- 172.21.15.123:0/1983890221 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb158076c00 0x7fb158107630 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb1480098e0 tx=0x7fb148009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=934c36c2940fbe5b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:06.766 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:59.620+0000 7fb156a00700 1 -- 172.21.15.123:0/1983890221 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb14800b860 con 0x7fb158076c00 2024-03-30T04:17:06.766 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:59.620+0000 7fb156a00700 1 -- 172.21.15.123:0/1983890221 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fb14800be30 con 0x7fb158076c00 2024-03-30T04:17:06.766 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:59.620+0000 7fb156a00700 1 -- 172.21.15.123:0/1983890221 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb148010380 con 0x7fb158076c00 2024-03-30T04:17:06.766 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:59.621+0000 7fb15f000700 1 -- 172.21.15.123:0/1983890221 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb158076c00 msgr2=0x7fb158107630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:06.766 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:59.621+0000 7fb15f000700 1 --2- 172.21.15.123:0/1983890221 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb158076c00 0x7fb158107630 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb1480098e0 tx=0x7fb148009250 comp rx=0 tx=0).stop 2024-03-30T04:17:06.766 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:59.621+0000 7fb15f000700 1 -- 172.21.15.123:0/1983890221 shutdown_connections 2024-03-30T04:17:06.766 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:59.621+0000 7fb15f000700 1 --2- 172.21.15.123:0/1983890221 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb158076c00 0x7fb158107630 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:06.766 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:59.621+0000 7fb15f000700 1 -- 172.21.15.123:0/1983890221 >> 172.21.15.123:0/1983890221 conn(0x7fb1580fcc60 msgr2=0x7fb1580ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:06.766 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:59.621+0000 7fb15f000700 1 -- 172.21.15.123:0/1983890221 shutdown_connections 2024-03-30T04:17:06.766 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:59.621+0000 7fb15f000700 1 -- 172.21.15.123:0/1983890221 wait complete. 2024-03-30T04:17:06.766 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:59.622+0000 7fb15f000700 1 Processor -- start 2024-03-30T04:17:06.766 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:59.622+0000 7fb15f000700 1 -- start start 2024-03-30T04:17:06.766 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:59.622+0000 7fb15f000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb158076c00 0x7fb158113570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:06.767 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:59.622+0000 7fb15f000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb158113ab0 con 0x7fb158076c00 2024-03-30T04:17:06.767 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:59.622+0000 7fb157e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb158076c00 0x7fb158113570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:06.767 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:59.622+0000 7fb157e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb158076c00 0x7fb158113570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42416/0 (socket says 172.21.15.123:42416) 2024-03-30T04:17:06.767 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:59.622+0000 7fb157e00700 1 -- 172.21.15.123:0/1895530894 learned_addr learned my addr 172.21.15.123:0/1895530894 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:17:06.767 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:59.623+0000 7fb157e00700 1 -- 172.21.15.123:0/1895530894 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1480094f0 con 0x7fb158076c00 2024-03-30T04:17:06.767 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:59.623+0000 7fb157e00700 1 --2- 172.21.15.123:0/1895530894 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb158076c00 0x7fb158113570 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb148006b00 tx=0x7fb148009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:06.767 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:59.623+0000 7fb14fe00700 1 -- 172.21.15.123:0/1895530894 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb148010a80 con 0x7fb158076c00 2024-03-30T04:17:06.767 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:59.623+0000 7fb14fe00700 1 -- 172.21.15.123:0/1895530894 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fb14801a430 con 0x7fb158076c00 2024-03-30T04:17:06.767 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:59.623+0000 7fb14fe00700 1 -- 172.21.15.123:0/1895530894 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1480245b0 con 0x7fb158076c00 2024-03-30T04:17:06.767 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:59.623+0000 7fb15f000700 1 -- 172.21.15.123:0/1895530894 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb158113cb0 con 0x7fb158076c00 2024-03-30T04:17:06.767 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:59.623+0000 7fb15f000700 1 -- 172.21.15.123:0/1895530894 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb158114820 con 0x7fb158076c00 2024-03-30T04:17:06.767 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:59.625+0000 7fb14fe00700 1 -- 172.21.15.123:0/1895530894 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fb148010be0 con 0x7fb158076c00 2024-03-30T04:17:06.767 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:59.625+0000 7fb14fe00700 1 --2- 172.21.15.123:0/1895530894 >> [v2:172.21.15.123:6800/1291592055,v1:172.21.15.123:6801/1291592055] conn(0x7fb13c036790 0x7fb13c038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:06.767 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:59.625+0000 7fb14fe00700 1 -- 172.21.15.123:0/1895530894 --> [v2:172.21.15.123:6800/1291592055,v1:172.21.15.123:6801/1291592055] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb13c039350 con 0x7fb13c036790 2024-03-30T04:17:06.768 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:59.625+0000 7fb14fe00700 1 -- 172.21.15.123:0/1895530894 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fb14804ad20 con 0x7fb158076c00 2024-03-30T04:17:06.768 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:59.625+0000 7fb157400700 1 -- 172.21.15.123:0/1895530894 >> [v2:172.21.15.123:6800/1291592055,v1:172.21.15.123:6801/1291592055] conn(0x7fb13c036790 msgr2=0x7fb13c038c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.123:6800/1291592055 2024-03-30T04:17:06.768 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:59.625+0000 7fb157400700 1 --2- 172.21.15.123:0/1895530894 >> [v2:172.21.15.123:6800/1291592055,v1:172.21.15.123:6801/1291592055] conn(0x7fb13c036790 0x7fb13c038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-30T04:17:06.768 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:59.826+0000 7fb157400700 1 -- 172.21.15.123:0/1895530894 >> [v2:172.21.15.123:6800/1291592055,v1:172.21.15.123:6801/1291592055] conn(0x7fb13c036790 msgr2=0x7fb13c038c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.123:6800/1291592055 2024-03-30T04:17:06.768 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:16:59.826+0000 7fb157400700 1 --2- 172.21.15.123:0/1895530894 >> [v2:172.21.15.123:6800/1291592055,v1:172.21.15.123:6801/1291592055] conn(0x7fb13c036790 0x7fb13c038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-30T04:17:06.768 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:00.227+0000 7fb157400700 1 -- 172.21.15.123:0/1895530894 >> [v2:172.21.15.123:6800/1291592055,v1:172.21.15.123:6801/1291592055] conn(0x7fb13c036790 msgr2=0x7fb13c038c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.123:6800/1291592055 2024-03-30T04:17:06.768 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:00.227+0000 7fb157400700 1 --2- 172.21.15.123:0/1895530894 >> [v2:172.21.15.123:6800/1291592055,v1:172.21.15.123:6801/1291592055] conn(0x7fb13c036790 0x7fb13c038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-30T04:17:06.768 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:01.028+0000 7fb157400700 1 -- 172.21.15.123:0/1895530894 >> [v2:172.21.15.123:6800/1291592055,v1:172.21.15.123:6801/1291592055] conn(0x7fb13c036790 msgr2=0x7fb13c038c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.123:6800/1291592055 2024-03-30T04:17:06.768 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:01.028+0000 7fb157400700 1 --2- 172.21.15.123:0/1895530894 >> [v2:172.21.15.123:6800/1291592055,v1:172.21.15.123:6801/1291592055] conn(0x7fb13c036790 0x7fb13c038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-30T04:17:06.768 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:02.629+0000 7fb157400700 1 -- 172.21.15.123:0/1895530894 >> [v2:172.21.15.123:6800/1291592055,v1:172.21.15.123:6801/1291592055] conn(0x7fb13c036790 msgr2=0x7fb13c038c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.123:6800/1291592055 2024-03-30T04:17:06.768 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:02.630+0000 7fb157400700 1 --2- 172.21.15.123:0/1895530894 >> [v2:172.21.15.123:6800/1291592055,v1:172.21.15.123:6801/1291592055] conn(0x7fb13c036790 0x7fb13c038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-30T04:17:06.768 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:05.108+0000 7fb14fe00700 1 -- 172.21.15.123:0/1895530894 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mgrmap(e 6) v1 ==== 43135+0+0 (secure 0 0 0) 0x7fb14802e960 con 0x7fb158076c00 2024-03-30T04:17:06.768 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:05.108+0000 7fb14fe00700 1 -- 172.21.15.123:0/1895530894 >> [v2:172.21.15.123:6800/1291592055,v1:172.21.15.123:6801/1291592055] conn(0x7fb13c036790 msgr2=0x7fb13c038c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-30T04:17:06.769 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:05.108+0000 7fb14fe00700 1 --2- 172.21.15.123:0/1895530894 >> [v2:172.21.15.123:6800/1291592055,v1:172.21.15.123:6801/1291592055] conn(0x7fb13c036790 0x7fb13c038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:06.769 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:06.110+0000 7fb14fe00700 1 -- 172.21.15.123:0/1895530894 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mgrmap(e 7) v1 ==== 43262+0+0 (secure 0 0 0) 0x7fb148023390 con 0x7fb158076c00 2024-03-30T04:17:06.769 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:06.110+0000 7fb14fe00700 1 --2- 172.21.15.123:0/1895530894 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7fb13c039fc0 0x7fb13c03c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:06.769 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:06.110+0000 7fb14fe00700 1 -- 172.21.15.123:0/1895530894 --> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb13c039350 con 0x7fb13c039fc0 2024-03-30T04:17:06.769 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:06.111+0000 7fb157400700 1 --2- 172.21.15.123:0/1895530894 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7fb13c039fc0 0x7fb13c03c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:06.769 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:06.112+0000 7fb157400700 1 --2- 172.21.15.123:0/1895530894 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7fb13c039fc0 0x7fb13c03c3a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fb1440039d0 tx=0x7fb144009350 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:06.769 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:06.113+0000 7fb14fe00700 1 -- 172.21.15.123:0/1895530894 <== mgr.14118 v2:172.21.15.123:6800/1776532690 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7fb13c039350 con 0x7fb13c039fc0 2024-03-30T04:17:06.770 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:06.119+0000 7fb15f000700 1 -- 172.21.15.123:0/1895530894 --> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fb158114140 con 0x7fb13c039fc0 2024-03-30T04:17:06.770 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:06.119+0000 7fb14fe00700 1 -- 172.21.15.123:0/1895530894 <== mgr.14118 v2:172.21.15.123:6800/1776532690 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fb158114140 con 0x7fb13c039fc0 2024-03-30T04:17:06.770 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:06.120+0000 7fb15f000700 1 -- 172.21.15.123:0/1895530894 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7fb13c039fc0 msgr2=0x7fb13c03c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:06.770 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:06.120+0000 7fb15f000700 1 --2- 172.21.15.123:0/1895530894 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7fb13c039fc0 0x7fb13c03c3a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fb1440039d0 tx=0x7fb144009350 comp rx=0 tx=0).stop 2024-03-30T04:17:06.770 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:06.120+0000 7fb15f000700 1 -- 172.21.15.123:0/1895530894 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb158076c00 msgr2=0x7fb158113570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:06.770 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:06.120+0000 7fb15f000700 1 --2- 172.21.15.123:0/1895530894 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb158076c00 0x7fb158113570 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb148006b00 tx=0x7fb148009250 comp rx=0 tx=0).stop 2024-03-30T04:17:06.770 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:06.120+0000 7fb15f000700 1 -- 172.21.15.123:0/1895530894 shutdown_connections 2024-03-30T04:17:06.770 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:06.120+0000 7fb15f000700 1 --2- 172.21.15.123:0/1895530894 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7fb13c039fc0 0x7fb13c03c3a0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:06.770 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:06.120+0000 7fb15f000700 1 --2- 172.21.15.123:0/1895530894 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb158076c00 0x7fb158113570 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:06.770 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:06.120+0000 7fb15f000700 1 --2- 172.21.15.123:0/1895530894 >> [v2:172.21.15.123:6800/1291592055,v1:172.21.15.123:6801/1291592055] conn(0x7fb13c036790 0x7fb13c038c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:06.770 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:06.120+0000 7fb15f000700 1 -- 172.21.15.123:0/1895530894 >> 172.21.15.123:0/1895530894 conn(0x7fb1580fcc60 msgr2=0x7fb1580fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:06.770 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:06.120+0000 7fb15f000700 1 -- 172.21.15.123:0/1895530894 shutdown_connections 2024-03-30T04:17:06.770 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:06.120+0000 7fb15f000700 1 -- 172.21.15.123:0/1895530894 wait complete. 2024-03-30T04:17:06.771 INFO:teuthology.orchestra.run.smithi123.stdout:mgr epoch 5 is available 2024-03-30T04:17:06.771 INFO:teuthology.orchestra.run.smithi123.stdout:Setting orchestrator backend to cephadm... 2024-03-30T04:17:07.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:07 smithi123 ceph-mon[35245]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-30T04:17:07.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:07 smithi123 ceph-mon[35245]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-30T04:17:08.164 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:08 smithi123 ceph-mon[35245]: mgrmap e8: smithi123.xgksqq(active, since 2s) 2024-03-30T04:17:08.164 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:08 smithi123 ceph-mon[35245]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:17:08.216 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:07.329+0000 7f6d79a00700 1 Processor -- start 2024-03-30T04:17:08.216 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:07.329+0000 7f6d79a00700 1 -- start start 2024-03-30T04:17:08.216 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:07.329+0000 7f6d79a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d74102530 0x7f6d74102940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:08.216 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:07.330+0000 7f6d79a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d74102e80 con 0x7f6d74102530 2024-03-30T04:17:08.216 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:07.330+0000 7f6d72a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d74102530 0x7f6d74102940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:08.216 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:07.330+0000 7f6d72a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d74102530 0x7f6d74102940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33352/0 (socket says 172.21.15.123:33352) 2024-03-30T04:17:08.216 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:07.330+0000 7f6d72a00700 1 -- 172.21.15.123:0/3778794322 learned_addr learned my addr 172.21.15.123:0/3778794322 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:17:08.216 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:07.330+0000 7f6d72a00700 1 -- 172.21.15.123:0/3778794322 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d74102fc0 con 0x7f6d74102530 2024-03-30T04:17:08.216 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:07.330+0000 7f6d72a00700 1 --2- 172.21.15.123:0/3778794322 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d74102530 0x7f6d74102940 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6d640098e0 tx=0x7f6d64009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4737146c45607add server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:08.216 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:07.331+0000 7f6d71600700 1 -- 172.21.15.123:0/3778794322 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d6400b860 con 0x7f6d74102530 2024-03-30T04:17:08.217 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:07.331+0000 7f6d71600700 1 -- 172.21.15.123:0/3778794322 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f6d6400be30 con 0x7f6d74102530 2024-03-30T04:17:08.217 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:07.331+0000 7f6d71600700 1 -- 172.21.15.123:0/3778794322 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d64010380 con 0x7f6d74102530 2024-03-30T04:17:08.217 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:07.331+0000 7f6d79a00700 1 -- 172.21.15.123:0/3778794322 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d74102530 msgr2=0x7f6d74102940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:08.217 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:07.331+0000 7f6d79a00700 1 --2- 172.21.15.123:0/3778794322 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d74102530 0x7f6d74102940 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6d640098e0 tx=0x7f6d64009250 comp rx=0 tx=0).stop 2024-03-30T04:17:08.217 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:07.331+0000 7f6d79a00700 1 -- 172.21.15.123:0/3778794322 shutdown_connections 2024-03-30T04:17:08.217 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:07.331+0000 7f6d79a00700 1 --2- 172.21.15.123:0/3778794322 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d74102530 0x7f6d74102940 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:08.217 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:07.331+0000 7f6d79a00700 1 -- 172.21.15.123:0/3778794322 >> 172.21.15.123:0/3778794322 conn(0x7f6d740fdcd0 msgr2=0x7f6d74100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:08.217 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:07.331+0000 7f6d79a00700 1 -- 172.21.15.123:0/3778794322 shutdown_connections 2024-03-30T04:17:08.217 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:07.332+0000 7f6d79a00700 1 -- 172.21.15.123:0/3778794322 wait complete. 2024-03-30T04:17:08.217 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:07.333+0000 7f6d79a00700 1 Processor -- start 2024-03-30T04:17:08.217 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:07.333+0000 7f6d79a00700 1 -- start start 2024-03-30T04:17:08.217 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:07.333+0000 7f6d79a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d74102530 0x7f6d74111420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:08.217 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:07.333+0000 7f6d79a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d74111960 con 0x7f6d74102530 2024-03-30T04:17:08.217 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:07.333+0000 7f6d72a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d74102530 0x7f6d74111420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:08.218 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:07.334+0000 7f6d72a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d74102530 0x7f6d74111420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33356/0 (socket says 172.21.15.123:33356) 2024-03-30T04:17:08.218 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:07.334+0000 7f6d72a00700 1 -- 172.21.15.123:0/2397253756 learned_addr learned my addr 172.21.15.123:0/2397253756 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:17:08.218 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:07.334+0000 7f6d72a00700 1 -- 172.21.15.123:0/2397253756 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d640094f0 con 0x7f6d74102530 2024-03-30T04:17:08.218 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:07.334+0000 7f6d72a00700 1 --2- 172.21.15.123:0/2397253756 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d74102530 0x7f6d74111420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6d64006b00 tx=0x7f6d64009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:08.218 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:07.334+0000 7f6d6aa00700 1 -- 172.21.15.123:0/2397253756 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d64010750 con 0x7f6d74102530 2024-03-30T04:17:08.218 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:07.334+0000 7f6d6aa00700 1 -- 172.21.15.123:0/2397253756 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f6d64010d20 con 0x7f6d74102530 2024-03-30T04:17:08.218 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:07.334+0000 7f6d79a00700 1 -- 172.21.15.123:0/2397253756 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d74111b60 con 0x7f6d74102530 2024-03-30T04:17:08.218 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:07.334+0000 7f6d6aa00700 1 -- 172.21.15.123:0/2397253756 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d6401acf0 con 0x7f6d74102530 2024-03-30T04:17:08.218 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:07.334+0000 7f6d79a00700 1 -- 172.21.15.123:0/2397253756 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d741126d0 con 0x7f6d74102530 2024-03-30T04:17:08.218 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:07.335+0000 7f6d6aa00700 1 -- 172.21.15.123:0/2397253756 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f6d6401a5a0 con 0x7f6d74102530 2024-03-30T04:17:08.218 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:07.335+0000 7f6d79a00700 1 -- 172.21.15.123:0/2397253756 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d7405f5f0 con 0x7f6d74102530 2024-03-30T04:17:08.218 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:07.336+0000 7f6d6aa00700 1 --2- 172.21.15.123:0/2397253756 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f6d580367e0 0x7f6d58038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:08.219 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:07.336+0000 7f6d6aa00700 1 -- 172.21.15.123:0/2397253756 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f6d6401e070 con 0x7f6d74102530 2024-03-30T04:17:08.219 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:07.336+0000 7f6d72000700 1 --2- 172.21.15.123:0/2397253756 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f6d580367e0 0x7f6d58038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:08.219 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:07.336+0000 7f6d72000700 1 --2- 172.21.15.123:0/2397253756 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f6d580367e0 0x7f6d58038c90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6d60009990 tx=0x7f6d60006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:08.219 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:07.341+0000 7f6d6aa00700 1 -- 172.21.15.123:0/2397253756 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f6d64029340 con 0x7f6d74102530 2024-03-30T04:17:08.219 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:07.549+0000 7f6d79a00700 1 -- 172.21.15.123:0/2397253756 --> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f6d740ff050 con 0x7f6d580367e0 2024-03-30T04:17:08.219 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:07.558+0000 7f6d6aa00700 1 -- 172.21.15.123:0/2397253756 <== mgr.14118 v2:172.21.15.123:6800/1776532690 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6d740ff050 con 0x7f6d580367e0 2024-03-30T04:17:08.219 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:07.561+0000 7f6d79a00700 1 -- 172.21.15.123:0/2397253756 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f6d580367e0 msgr2=0x7f6d58038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:08.219 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:07.561+0000 7f6d79a00700 1 --2- 172.21.15.123:0/2397253756 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f6d580367e0 0x7f6d58038c90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6d60009990 tx=0x7f6d60006e40 comp rx=0 tx=0).stop 2024-03-30T04:17:08.219 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:07.561+0000 7f6d79a00700 1 -- 172.21.15.123:0/2397253756 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d74102530 msgr2=0x7f6d74111420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:08.219 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:07.561+0000 7f6d79a00700 1 --2- 172.21.15.123:0/2397253756 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d74102530 0x7f6d74111420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6d64006b00 tx=0x7f6d64009250 comp rx=0 tx=0).stop 2024-03-30T04:17:08.219 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:07.561+0000 7f6d79a00700 1 -- 172.21.15.123:0/2397253756 shutdown_connections 2024-03-30T04:17:08.219 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:07.561+0000 7f6d79a00700 1 --2- 172.21.15.123:0/2397253756 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f6d580367e0 0x7f6d58038c90 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:08.219 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:07.561+0000 7f6d79a00700 1 --2- 172.21.15.123:0/2397253756 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d74102530 0x7f6d74111420 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:08.220 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:07.561+0000 7f6d79a00700 1 -- 172.21.15.123:0/2397253756 >> 172.21.15.123:0/2397253756 conn(0x7f6d740fdcd0 msgr2=0x7f6d740fe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:08.220 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:07.561+0000 7f6d79a00700 1 -- 172.21.15.123:0/2397253756 shutdown_connections 2024-03-30T04:17:08.220 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:07.561+0000 7f6d79a00700 1 -- 172.21.15.123:0/2397253756 wait complete. 2024-03-30T04:17:08.455 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:08 smithi123 ceph-mon[35245]: from='mgr.14118 172.21.15.123:0/1102803547' entity='mgr.smithi123.xgksqq' 2024-03-30T04:17:08.455 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:08 smithi123 ceph-mon[35245]: from='mgr.14118 172.21.15.123:0/1102803547' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:17:09.270 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:09 smithi123 ceph-mon[35245]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:17:09.556 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-30T04:17:09.556 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:08.744+0000 7f58e7200700 1 Processor -- start 2024-03-30T04:17:09.556 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:08.744+0000 7f58e7200700 1 -- start start 2024-03-30T04:17:09.556 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:08.744+0000 7f58e7200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58e0102580 0x7f58e0102990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:09.556 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:08.744+0000 7f58e7200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58e0102ed0 con 0x7f58e0102580 2024-03-30T04:17:09.556 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:08.744+0000 7f58dfe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58e0102580 0x7f58e0102990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:09.556 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:08.744+0000 7f58dfe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58e0102580 0x7f58e0102990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33366/0 (socket says 172.21.15.123:33366) 2024-03-30T04:17:09.557 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:08.744+0000 7f58dfe00700 1 -- 172.21.15.123:0/3291686361 learned_addr learned my addr 172.21.15.123:0/3291686361 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:17:09.557 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:08.745+0000 7f58dfe00700 1 -- 172.21.15.123:0/3291686361 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e0103010 con 0x7f58e0102580 2024-03-30T04:17:09.557 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:08.745+0000 7f58dfe00700 1 --2- 172.21.15.123:0/3291686361 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58e0102580 0x7f58e0102990 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f58d00098e0 tx=0x7f58d0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cadcad92109f42c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:09.557 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:08.745+0000 7f58dea00700 1 -- 172.21.15.123:0/3291686361 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58d000b860 con 0x7f58e0102580 2024-03-30T04:17:09.557 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:08.745+0000 7f58dea00700 1 -- 172.21.15.123:0/3291686361 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f58d000be30 con 0x7f58e0102580 2024-03-30T04:17:09.557 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:08.745+0000 7f58dea00700 1 -- 172.21.15.123:0/3291686361 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58d0010380 con 0x7f58e0102580 2024-03-30T04:17:09.557 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:08.746+0000 7f58e7200700 1 -- 172.21.15.123:0/3291686361 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58e0102580 msgr2=0x7f58e0102990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:09.557 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:08.746+0000 7f58e7200700 1 --2- 172.21.15.123:0/3291686361 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58e0102580 0x7f58e0102990 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f58d00098e0 tx=0x7f58d0009250 comp rx=0 tx=0).stop 2024-03-30T04:17:09.557 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:08.746+0000 7f58e7200700 1 -- 172.21.15.123:0/3291686361 shutdown_connections 2024-03-30T04:17:09.557 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:08.746+0000 7f58e7200700 1 --2- 172.21.15.123:0/3291686361 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58e0102580 0x7f58e0102990 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:09.557 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:08.746+0000 7f58e7200700 1 -- 172.21.15.123:0/3291686361 >> 172.21.15.123:0/3291686361 conn(0x7f58e00fdd00 msgr2=0x7f58e0100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:09.557 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:08.746+0000 7f58e7200700 1 -- 172.21.15.123:0/3291686361 shutdown_connections 2024-03-30T04:17:09.557 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:08.746+0000 7f58e7200700 1 -- 172.21.15.123:0/3291686361 wait complete. 2024-03-30T04:17:09.558 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:08.747+0000 7f58e7200700 1 Processor -- start 2024-03-30T04:17:09.558 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:08.747+0000 7f58e7200700 1 -- start start 2024-03-30T04:17:09.558 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:08.747+0000 7f58e7200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58e0102580 0x7f58e0113540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:09.558 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:08.747+0000 7f58e7200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58e0113a80 con 0x7f58e0102580 2024-03-30T04:17:09.558 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:08.747+0000 7f58dfe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58e0102580 0x7f58e0113540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:09.558 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:08.747+0000 7f58dfe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58e0102580 0x7f58e0113540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33380/0 (socket says 172.21.15.123:33380) 2024-03-30T04:17:09.558 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:08.747+0000 7f58dfe00700 1 -- 172.21.15.123:0/3843838848 learned_addr learned my addr 172.21.15.123:0/3843838848 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:17:09.558 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:08.748+0000 7f58dfe00700 1 -- 172.21.15.123:0/3843838848 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d00094f0 con 0x7f58e0102580 2024-03-30T04:17:09.558 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:08.748+0000 7f58dfe00700 1 --2- 172.21.15.123:0/3843838848 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58e0102580 0x7f58e0113540 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f58d0006b00 tx=0x7f58d0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:09.558 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:08.748+0000 7f58d7e00700 1 -- 172.21.15.123:0/3843838848 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58d00109b0 con 0x7f58e0102580 2024-03-30T04:17:09.558 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:08.748+0000 7f58d7e00700 1 -- 172.21.15.123:0/3843838848 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f58d0024440 con 0x7f58e0102580 2024-03-30T04:17:09.558 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:08.748+0000 7f58e7200700 1 -- 172.21.15.123:0/3843838848 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58e0113c80 con 0x7f58e0102580 2024-03-30T04:17:09.558 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:08.748+0000 7f58d7e00700 1 -- 172.21.15.123:0/3843838848 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58d001a5f0 con 0x7f58e0102580 2024-03-30T04:17:09.559 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:08.748+0000 7f58e7200700 1 -- 172.21.15.123:0/3843838848 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58e01147f0 con 0x7f58e0102580 2024-03-30T04:17:09.559 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:08.750+0000 7f58e7200700 1 -- 172.21.15.123:0/3843838848 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58e005f5f0 con 0x7f58e0102580 2024-03-30T04:17:09.559 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:08.750+0000 7f58d7e00700 1 -- 172.21.15.123:0/3843838848 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f58d0010b10 con 0x7f58e0102580 2024-03-30T04:17:09.559 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:08.750+0000 7f58d7e00700 1 --2- 172.21.15.123:0/3843838848 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f58c40367e0 0x7f58c4038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:09.559 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:08.750+0000 7f58d7e00700 1 -- 172.21.15.123:0/3843838848 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f58d004a830 con 0x7f58e0102580 2024-03-30T04:17:09.559 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:08.750+0000 7f58df400700 1 --2- 172.21.15.123:0/3843838848 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f58c40367e0 0x7f58c4038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:09.559 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:08.751+0000 7f58df400700 1 --2- 172.21.15.123:0/3843838848 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f58c40367e0 0x7f58c4038c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f58cc009990 tx=0x7f58cc006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:09.559 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:08.755+0000 7f58d7e00700 1 -- 172.21.15.123:0/3843838848 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f58d00239f0 con 0x7f58e0102580 2024-03-30T04:17:09.559 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:08.951+0000 7f58e7200700 1 -- 172.21.15.123:0/3843838848 --> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f58e00fee90 con 0x7f58c40367e0 2024-03-30T04:17:09.559 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:08.952+0000 7f58d7e00700 1 -- 172.21.15.123:0/3843838848 <== mgr.14118 v2:172.21.15.123:6800/1776532690 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f58e00fee90 con 0x7f58c40367e0 2024-03-30T04:17:09.559 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:08.954+0000 7f58e7200700 1 -- 172.21.15.123:0/3843838848 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f58c40367e0 msgr2=0x7f58c4038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:09.559 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:08.954+0000 7f58e7200700 1 --2- 172.21.15.123:0/3843838848 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f58c40367e0 0x7f58c4038c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f58cc009990 tx=0x7f58cc006e40 comp rx=0 tx=0).stop 2024-03-30T04:17:09.559 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:08.954+0000 7f58e7200700 1 -- 172.21.15.123:0/3843838848 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58e0102580 msgr2=0x7f58e0113540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:09.559 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:08.954+0000 7f58e7200700 1 --2- 172.21.15.123:0/3843838848 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58e0102580 0x7f58e0113540 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f58d0006b00 tx=0x7f58d0009250 comp rx=0 tx=0).stop 2024-03-30T04:17:09.560 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:08.954+0000 7f58e7200700 1 -- 172.21.15.123:0/3843838848 shutdown_connections 2024-03-30T04:17:09.560 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:08.954+0000 7f58e7200700 1 --2- 172.21.15.123:0/3843838848 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f58c40367e0 0x7f58c4038c90 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:09.560 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:08.954+0000 7f58e7200700 1 --2- 172.21.15.123:0/3843838848 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58e0102580 0x7f58e0113540 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:09.560 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:08.954+0000 7f58e7200700 1 -- 172.21.15.123:0/3843838848 >> 172.21.15.123:0/3843838848 conn(0x7f58e00fdd00 msgr2=0x7f58e00fe780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:09.560 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:08.954+0000 7f58e7200700 1 -- 172.21.15.123:0/3843838848 shutdown_connections 2024-03-30T04:17:09.560 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:08.954+0000 7f58e7200700 1 -- 172.21.15.123:0/3843838848 wait complete. 2024-03-30T04:17:09.560 INFO:teuthology.orchestra.run.smithi123.stdout:Generating ssh key... 2024-03-30T04:17:11.077 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:10.131+0000 7f77a5000700 1 Processor -- start 2024-03-30T04:17:11.077 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:10.131+0000 7f77a5000700 1 -- start start 2024-03-30T04:17:11.077 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:10.132+0000 7f77a5000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77a0102580 0x7f77a0102990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:11.077 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:10.132+0000 7f77a5000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a0102ed0 con 0x7f77a0102580 2024-03-30T04:17:11.077 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:10.132+0000 7f779de00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77a0102580 0x7f77a0102990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:11.078 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:10.132+0000 7f779de00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77a0102580 0x7f77a0102990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33392/0 (socket says 172.21.15.123:33392) 2024-03-30T04:17:11.078 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:10.132+0000 7f779de00700 1 -- 172.21.15.123:0/2994816337 learned_addr learned my addr 172.21.15.123:0/2994816337 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:17:11.078 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:10.132+0000 7f779de00700 1 -- 172.21.15.123:0/2994816337 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77a0103010 con 0x7f77a0102580 2024-03-30T04:17:11.078 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:10.133+0000 7f779de00700 1 --2- 172.21.15.123:0/2994816337 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77a0102580 0x7f77a0102990 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f77900098e0 tx=0x7f7790009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8df094572375c349 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:11.078 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:10.133+0000 7f779ca00700 1 -- 172.21.15.123:0/2994816337 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f779000b860 con 0x7f77a0102580 2024-03-30T04:17:11.078 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:10.133+0000 7f779ca00700 1 -- 172.21.15.123:0/2994816337 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f779000be30 con 0x7f77a0102580 2024-03-30T04:17:11.078 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:10.133+0000 7f779ca00700 1 -- 172.21.15.123:0/2994816337 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7790010380 con 0x7f77a0102580 2024-03-30T04:17:11.078 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:10.133+0000 7f77a5000700 1 -- 172.21.15.123:0/2994816337 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77a0102580 msgr2=0x7f77a0102990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:11.078 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:10.133+0000 7f77a5000700 1 --2- 172.21.15.123:0/2994816337 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77a0102580 0x7f77a0102990 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f77900098e0 tx=0x7f7790009250 comp rx=0 tx=0).stop 2024-03-30T04:17:11.078 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:10.134+0000 7f77a5000700 1 -- 172.21.15.123:0/2994816337 shutdown_connections 2024-03-30T04:17:11.078 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:10.134+0000 7f77a5000700 1 --2- 172.21.15.123:0/2994816337 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77a0102580 0x7f77a0102990 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:11.078 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:10.134+0000 7f77a5000700 1 -- 172.21.15.123:0/2994816337 >> 172.21.15.123:0/2994816337 conn(0x7f77a00fdd00 msgr2=0x7f77a0100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:11.079 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:10.134+0000 7f77a5000700 1 -- 172.21.15.123:0/2994816337 shutdown_connections 2024-03-30T04:17:11.079 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:10.134+0000 7f77a5000700 1 -- 172.21.15.123:0/2994816337 wait complete. 2024-03-30T04:17:11.079 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:10.135+0000 7f77a5000700 1 Processor -- start 2024-03-30T04:17:11.079 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:10.135+0000 7f77a5000700 1 -- start start 2024-03-30T04:17:11.079 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:10.135+0000 7f77a5000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77a0102580 0x7f77a0078c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:11.079 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:10.135+0000 7f77a5000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a0079190 con 0x7f77a0102580 2024-03-30T04:17:11.079 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:10.136+0000 7f779de00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77a0102580 0x7f77a0078c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:11.079 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:10.136+0000 7f779de00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77a0102580 0x7f77a0078c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33396/0 (socket says 172.21.15.123:33396) 2024-03-30T04:17:11.079 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:10.136+0000 7f779de00700 1 -- 172.21.15.123:0/3554857890 learned_addr learned my addr 172.21.15.123:0/3554857890 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:17:11.079 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:10.136+0000 7f779de00700 1 -- 172.21.15.123:0/3554857890 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77900094f0 con 0x7f77a0102580 2024-03-30T04:17:11.079 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:10.136+0000 7f779de00700 1 --2- 172.21.15.123:0/3554857890 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77a0102580 0x7f77a0078c50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7790000c00 tx=0x7f7790009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:11.079 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:10.136+0000 7f7796000700 1 -- 172.21.15.123:0/3554857890 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7790010a80 con 0x7f77a0102580 2024-03-30T04:17:11.079 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:10.137+0000 7f7796000700 1 -- 172.21.15.123:0/3554857890 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f779001a430 con 0x7f77a0102580 2024-03-30T04:17:11.079 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:10.137+0000 7f77a5000700 1 -- 172.21.15.123:0/3554857890 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77a0077200 con 0x7f77a0102580 2024-03-30T04:17:11.080 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:10.137+0000 7f7796000700 1 -- 172.21.15.123:0/3554857890 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f77900245b0 con 0x7f77a0102580 2024-03-30T04:17:11.080 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:10.137+0000 7f77a5000700 1 -- 172.21.15.123:0/3554857890 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77a00776f0 con 0x7f77a0102580 2024-03-30T04:17:11.080 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:10.138+0000 7f7796000700 1 -- 172.21.15.123:0/3554857890 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f779001a5a0 con 0x7f77a0102580 2024-03-30T04:17:11.080 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:10.138+0000 7f77a5000700 1 -- 172.21.15.123:0/3554857890 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77a004bed0 con 0x7f77a0102580 2024-03-30T04:17:11.080 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:10.138+0000 7f7796000700 1 --2- 172.21.15.123:0/3554857890 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f7784036790 0x7f7784038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:11.080 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:10.138+0000 7f7796000700 1 -- 172.21.15.123:0/3554857890 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f779004ade0 con 0x7f77a0102580 2024-03-30T04:17:11.080 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:10.138+0000 7f779d400700 1 --2- 172.21.15.123:0/3554857890 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f7784036790 0x7f7784038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:11.080 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:10.139+0000 7f779d400700 1 --2- 172.21.15.123:0/3554857890 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f7784036790 0x7f7784038c40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f778c009990 tx=0x7f778c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:11.080 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:10.143+0000 7f7796000700 1 -- 172.21.15.123:0/3554857890 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f7790023df0 con 0x7f77a0102580 2024-03-30T04:17:11.080 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:10.338+0000 7f77a5000700 1 -- 172.21.15.123:0/3554857890 --> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f77a0077e00 con 0x7f7784036790 2024-03-30T04:17:11.080 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:10.502+0000 7f7796000700 1 -- 172.21.15.123:0/3554857890 <== mgr.14118 v2:172.21.15.123:6800/1776532690 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f77a0077e00 con 0x7f7784036790 2024-03-30T04:17:11.080 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:10.504+0000 7f77a5000700 1 -- 172.21.15.123:0/3554857890 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f7784036790 msgr2=0x7f7784038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:11.080 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:10.504+0000 7f77a5000700 1 --2- 172.21.15.123:0/3554857890 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f7784036790 0x7f7784038c40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f778c009990 tx=0x7f778c006e40 comp rx=0 tx=0).stop 2024-03-30T04:17:11.080 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:10.504+0000 7f77a5000700 1 -- 172.21.15.123:0/3554857890 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77a0102580 msgr2=0x7f77a0078c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:11.081 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:10.504+0000 7f77a5000700 1 --2- 172.21.15.123:0/3554857890 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77a0102580 0x7f77a0078c50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7790000c00 tx=0x7f7790009250 comp rx=0 tx=0).stop 2024-03-30T04:17:11.081 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:10.504+0000 7f77a5000700 1 -- 172.21.15.123:0/3554857890 shutdown_connections 2024-03-30T04:17:11.081 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:10.504+0000 7f77a5000700 1 --2- 172.21.15.123:0/3554857890 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f7784036790 0x7f7784038c40 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:11.081 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:10.504+0000 7f77a5000700 1 --2- 172.21.15.123:0/3554857890 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77a0102580 0x7f77a0078c50 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:11.081 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:10.504+0000 7f77a5000700 1 -- 172.21.15.123:0/3554857890 >> 172.21.15.123:0/3554857890 conn(0x7f77a00fdd00 msgr2=0x7f77a00fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:11.081 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:10.504+0000 7f77a5000700 1 -- 172.21.15.123:0/3554857890 shutdown_connections 2024-03-30T04:17:11.081 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:10.504+0000 7f77a5000700 1 -- 172.21.15.123:0/3554857890 wait complete. 2024-03-30T04:17:11.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:11 smithi123 ceph-mon[35245]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:17:11.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:11 smithi123 ceph-mon[35245]: Generating ssh key... 2024-03-30T04:17:11.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:11 smithi123 ceph-mon[35245]: from='mgr.14118 172.21.15.123:0/1102803547' entity='mgr.smithi123.xgksqq' 2024-03-30T04:17:11.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:11 smithi123 ceph-mon[35245]: from='mgr.14118 172.21.15.123:0/1102803547' entity='mgr.smithi123.xgksqq' 2024-03-30T04:17:12.476 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6PBnH+eOJqjBedQ7yNS5ZfkelDzSWPbkRhZYIu9oKaG1M2NnJDxWlzLMa9TATuq1JjuxwBRXqVq07qf4ZRXYRZzRSYjk38FDbeoNR8HLjOZzf0G0PUbYTTT4xYIwgJ3dEFTSuleSUCH7fqORmKjpbyHEERS6ezUmwTZMZ/tlb1cWJJvc6nBkMp0XJQ4oyFJJTGyDQ1bLlwsfXBLS/lhfRoN/thTlsjyonNkF0KLXQ9xNewp8tmL0ICLt+t3o8AXdANVobEbknPd19bb1o8x+pv/u2ABWGVRAB8Vm75/vwVv94EvqbJKco2lgR1OzKf6WWP0ci3vjdF2+EDC6LIZ0iE5ENM5FNCpfNDW4ETLBUAECoDEIgCh5A3EvgcZiJdtpErf88b1eMoE7PMK+Zg7o+vW50pFOJrXqY4lQLSSnlPPVF4/lTVieZr0iF+T+0RGHtat9y9L7b6tO/yS0xOUB82169nc4Dr+/G1ARwTrESC6/nkDphI8JFjgyXg53cyIE= ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4 2024-03-30T04:17:12.476 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:11.629+0000 7f3fe1200700 1 Processor -- start 2024-03-30T04:17:12.476 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:11.629+0000 7f3fe1200700 1 -- start start 2024-03-30T04:17:12.476 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:11.629+0000 7f3fe1200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3fdc078d70 0x7f3fdc077280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:12.476 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:11.629+0000 7f3fe1200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fdc079180 con 0x7f3fdc078d70 2024-03-30T04:17:12.477 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:11.629+0000 7f3fdb400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3fdc078d70 0x7f3fdc077280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:12.477 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:11.630+0000 7f3fdb400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3fdc078d70 0x7f3fdc077280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33402/0 (socket says 172.21.15.123:33402) 2024-03-30T04:17:12.477 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:11.630+0000 7f3fdb400700 1 -- 172.21.15.123:0/3448252586 learned_addr learned my addr 172.21.15.123:0/3448252586 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:17:12.477 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:11.630+0000 7f3fdb400700 1 -- 172.21.15.123:0/3448252586 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fdc0777c0 con 0x7f3fdc078d70 2024-03-30T04:17:12.477 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:11.630+0000 7f3fdb400700 1 --2- 172.21.15.123:0/3448252586 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3fdc078d70 0x7f3fdc077280 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc0098e0 tx=0x7f3fcc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c53e43c960b2cde9 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:12.477 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:11.630+0000 7f3fda000700 1 -- 172.21.15.123:0/3448252586 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fcc00b860 con 0x7f3fdc078d70 2024-03-30T04:17:12.477 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:11.631+0000 7f3fda000700 1 -- 172.21.15.123:0/3448252586 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f3fcc00be30 con 0x7f3fdc078d70 2024-03-30T04:17:12.477 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:11.631+0000 7f3fda000700 1 -- 172.21.15.123:0/3448252586 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fcc010380 con 0x7f3fdc078d70 2024-03-30T04:17:12.477 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:11.631+0000 7f3fe1200700 1 -- 172.21.15.123:0/3448252586 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3fdc078d70 msgr2=0x7f3fdc077280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:12.477 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:11.631+0000 7f3fe1200700 1 --2- 172.21.15.123:0/3448252586 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3fdc078d70 0x7f3fdc077280 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc0098e0 tx=0x7f3fcc009250 comp rx=0 tx=0).stop 2024-03-30T04:17:12.477 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:11.631+0000 7f3fe1200700 1 -- 172.21.15.123:0/3448252586 shutdown_connections 2024-03-30T04:17:12.477 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:11.631+0000 7f3fe1200700 1 --2- 172.21.15.123:0/3448252586 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3fdc078d70 0x7f3fdc077280 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:12.477 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:11.631+0000 7f3fe1200700 1 -- 172.21.15.123:0/3448252586 >> 172.21.15.123:0/3448252586 conn(0x7f3fdc0fdb90 msgr2=0x7f3fdc0fffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:12.478 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:11.631+0000 7f3fe1200700 1 -- 172.21.15.123:0/3448252586 shutdown_connections 2024-03-30T04:17:12.478 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:11.631+0000 7f3fe1200700 1 -- 172.21.15.123:0/3448252586 wait complete. 2024-03-30T04:17:12.478 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:11.632+0000 7f3fe1200700 1 Processor -- start 2024-03-30T04:17:12.478 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:11.633+0000 7f3fe1200700 1 -- start start 2024-03-30T04:17:12.478 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:11.633+0000 7f3fe1200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3fdc078d70 0x7f3fdc113410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:12.478 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:11.633+0000 7f3fe1200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fdc113950 con 0x7f3fdc078d70 2024-03-30T04:17:12.478 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:11.633+0000 7f3fdb400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3fdc078d70 0x7f3fdc113410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:12.478 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:11.633+0000 7f3fdb400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3fdc078d70 0x7f3fdc113410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33414/0 (socket says 172.21.15.123:33414) 2024-03-30T04:17:12.478 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:11.633+0000 7f3fdb400700 1 -- 172.21.15.123:0/2549326504 learned_addr learned my addr 172.21.15.123:0/2549326504 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:17:12.478 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:11.634+0000 7f3fdb400700 1 -- 172.21.15.123:0/2549326504 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fcc0094f0 con 0x7f3fdc078d70 2024-03-30T04:17:12.478 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:11.634+0000 7f3fdb400700 1 --2- 172.21.15.123:0/2549326504 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3fdc078d70 0x7f3fdc113410 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc00b440 tx=0x7f3fcc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:12.478 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:11.634+0000 7f3fd3400700 1 -- 172.21.15.123:0/2549326504 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fcc010960 con 0x7f3fdc078d70 2024-03-30T04:17:12.478 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:11.634+0000 7f3fe1200700 1 -- 172.21.15.123:0/2549326504 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fdc113b50 con 0x7f3fdc078d70 2024-03-30T04:17:12.478 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:11.634+0000 7f3fd3400700 1 -- 172.21.15.123:0/2549326504 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f3fcc01a430 con 0x7f3fdc078d70 2024-03-30T04:17:12.479 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:11.634+0000 7f3fd3400700 1 -- 172.21.15.123:0/2549326504 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fcc0245b0 con 0x7f3fdc078d70 2024-03-30T04:17:12.479 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:11.634+0000 7f3fe1200700 1 -- 172.21.15.123:0/2549326504 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fdc1146c0 con 0x7f3fdc078d70 2024-03-30T04:17:12.479 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:11.635+0000 7f3fd3400700 1 -- 172.21.15.123:0/2549326504 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f3fcc01a980 con 0x7f3fdc078d70 2024-03-30T04:17:12.479 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:11.635+0000 7f3fe1200700 1 -- 172.21.15.123:0/2549326504 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fdc05f5f0 con 0x7f3fdc078d70 2024-03-30T04:17:12.479 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:11.636+0000 7f3fd3400700 1 --2- 172.21.15.123:0/2549326504 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f3fc0036790 0x7f3fc0038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:12.479 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:11.636+0000 7f3fd3400700 1 -- 172.21.15.123:0/2549326504 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f3fcc04a360 con 0x7f3fdc078d70 2024-03-30T04:17:12.479 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:11.636+0000 7f3fdaa00700 1 --2- 172.21.15.123:0/2549326504 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f3fc0036790 0x7f3fc0038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:12.479 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:11.636+0000 7f3fdaa00700 1 --2- 172.21.15.123:0/2549326504 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f3fc0036790 0x7f3fc0038c40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3fc8009990 tx=0x7f3fc8006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:12.479 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:11.641+0000 7f3fd3400700 1 -- 172.21.15.123:0/2549326504 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f3fcc02b8e0 con 0x7f3fdc078d70 2024-03-30T04:17:12.479 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:11.836+0000 7f3fe1200700 1 -- 172.21.15.123:0/2549326504 --> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f3fdc0fee60 con 0x7f3fc0036790 2024-03-30T04:17:12.479 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:11.836+0000 7f3fd3400700 1 -- 172.21.15.123:0/2549326504 <== mgr.14118 v2:172.21.15.123:6800/1776532690 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f3fdc0fee60 con 0x7f3fc0036790 2024-03-30T04:17:12.480 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:11.839+0000 7f3fe1200700 1 -- 172.21.15.123:0/2549326504 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f3fc0036790 msgr2=0x7f3fc0038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:12.480 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:11.839+0000 7f3fe1200700 1 --2- 172.21.15.123:0/2549326504 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f3fc0036790 0x7f3fc0038c40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3fc8009990 tx=0x7f3fc8006e40 comp rx=0 tx=0).stop 2024-03-30T04:17:12.480 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:11.839+0000 7f3fe1200700 1 -- 172.21.15.123:0/2549326504 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3fdc078d70 msgr2=0x7f3fdc113410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:12.480 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:11.839+0000 7f3fe1200700 1 --2- 172.21.15.123:0/2549326504 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3fdc078d70 0x7f3fdc113410 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc00b440 tx=0x7f3fcc009250 comp rx=0 tx=0).stop 2024-03-30T04:17:12.480 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:11.839+0000 7f3fe1200700 1 -- 172.21.15.123:0/2549326504 shutdown_connections 2024-03-30T04:17:12.480 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:11.839+0000 7f3fe1200700 1 --2- 172.21.15.123:0/2549326504 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f3fc0036790 0x7f3fc0038c40 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:12.480 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:11.839+0000 7f3fe1200700 1 --2- 172.21.15.123:0/2549326504 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3fdc078d70 0x7f3fdc113410 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:12.480 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:11.839+0000 7f3fe1200700 1 -- 172.21.15.123:0/2549326504 >> 172.21.15.123:0/2549326504 conn(0x7f3fdc0fdb90 msgr2=0x7f3fdc0fe710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:12.480 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:11.839+0000 7f3fe1200700 1 -- 172.21.15.123:0/2549326504 shutdown_connections 2024-03-30T04:17:12.480 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:11.839+0000 7f3fe1200700 1 -- 172.21.15.123:0/2549326504 wait complete. 2024-03-30T04:17:12.480 INFO:teuthology.orchestra.run.smithi123.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-30T04:17:12.480 INFO:teuthology.orchestra.run.smithi123.stdout:Adding key to root@localhost authorized_keys... 2024-03-30T04:17:12.481 INFO:teuthology.orchestra.run.smithi123.stdout:Adding host smithi123... 2024-03-30T04:17:12.757 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:12 smithi123 ceph-mon[35245]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:17:14.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:14 smithi123 ceph-mon[35245]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi123", "addr": "172.21.15.123", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:17:15.299 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:15 smithi123 ceph-mon[35245]: Deploying cephadm binary to smithi123 2024-03-30T04:17:16.449 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout Added host 'smithi123' with addr '172.21.15.123' 2024-03-30T04:17:16.449 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:13.020+0000 7f92c6200700 1 Processor -- start 2024-03-30T04:17:16.449 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:13.020+0000 7f92c6200700 1 -- start start 2024-03-30T04:17:16.450 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:13.021+0000 7f92c6200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92c0078d50 0x7f92c0077260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:16.450 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:13.021+0000 7f92c6200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c0079160 con 0x7f92c0078d50 2024-03-30T04:17:16.450 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:13.021+0000 7f92bf400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92c0078d50 0x7f92c0077260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:16.450 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:13.021+0000 7f92bf400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92c0078d50 0x7f92c0077260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33420/0 (socket says 172.21.15.123:33420) 2024-03-30T04:17:16.450 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:13.021+0000 7f92bf400700 1 -- 172.21.15.123:0/2104259368 learned_addr learned my addr 172.21.15.123:0/2104259368 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:17:16.451 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:13.021+0000 7f92bf400700 1 -- 172.21.15.123:0/2104259368 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c00777a0 con 0x7f92c0078d50 2024-03-30T04:17:16.451 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:13.022+0000 7f92bf400700 1 --2- 172.21.15.123:0/2104259368 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92c0078d50 0x7f92c0077260 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f92b00098e0 tx=0x7f92b0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=749f5ae3c194b1c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:16.451 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:13.022+0000 7f92be000700 1 -- 172.21.15.123:0/2104259368 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f92b000b860 con 0x7f92c0078d50 2024-03-30T04:17:16.451 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:13.022+0000 7f92be000700 1 -- 172.21.15.123:0/2104259368 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f92b000be30 con 0x7f92c0078d50 2024-03-30T04:17:16.451 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:13.022+0000 7f92be000700 1 -- 172.21.15.123:0/2104259368 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f92b0010380 con 0x7f92c0078d50 2024-03-30T04:17:16.451 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:13.022+0000 7f92c6200700 1 -- 172.21.15.123:0/2104259368 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92c0078d50 msgr2=0x7f92c0077260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:16.451 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:13.022+0000 7f92c6200700 1 --2- 172.21.15.123:0/2104259368 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92c0078d50 0x7f92c0077260 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f92b00098e0 tx=0x7f92b0009250 comp rx=0 tx=0).stop 2024-03-30T04:17:16.452 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:13.023+0000 7f92c6200700 1 -- 172.21.15.123:0/2104259368 shutdown_connections 2024-03-30T04:17:16.452 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:13.023+0000 7f92c6200700 1 --2- 172.21.15.123:0/2104259368 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92c0078d50 0x7f92c0077260 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:16.452 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:13.023+0000 7f92c6200700 1 -- 172.21.15.123:0/2104259368 >> 172.21.15.123:0/2104259368 conn(0x7f92c00fdcd0 msgr2=0x7f92c0100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:16.452 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:13.023+0000 7f92c6200700 1 -- 172.21.15.123:0/2104259368 shutdown_connections 2024-03-30T04:17:16.452 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:13.023+0000 7f92c6200700 1 -- 172.21.15.123:0/2104259368 wait complete. 2024-03-30T04:17:16.452 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:13.023+0000 7f92c6200700 1 Processor -- start 2024-03-30T04:17:16.452 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:13.024+0000 7f92c6200700 1 -- start start 2024-03-30T04:17:16.452 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:13.024+0000 7f92c6200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92c0078d50 0x7f92c0113640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:16.452 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:13.024+0000 7f92c6200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c0113b80 con 0x7f92c0078d50 2024-03-30T04:17:16.452 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:13.024+0000 7f92bf400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92c0078d50 0x7f92c0113640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:16.452 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:13.024+0000 7f92bf400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92c0078d50 0x7f92c0113640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33434/0 (socket says 172.21.15.123:33434) 2024-03-30T04:17:16.453 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:13.024+0000 7f92bf400700 1 -- 172.21.15.123:0/2589704357 learned_addr learned my addr 172.21.15.123:0/2589704357 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:17:16.453 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:13.024+0000 7f92bf400700 1 -- 172.21.15.123:0/2589704357 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b00094f0 con 0x7f92c0078d50 2024-03-30T04:17:16.453 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:13.025+0000 7f92bf400700 1 --2- 172.21.15.123:0/2589704357 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92c0078d50 0x7f92c0113640 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f92b0006b00 tx=0x7f92b0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:16.453 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:13.025+0000 7f92b7400700 1 -- 172.21.15.123:0/2589704357 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f92b0010750 con 0x7f92c0078d50 2024-03-30T04:17:16.453 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:13.025+0000 7f92b7400700 1 -- 172.21.15.123:0/2589704357 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f92b0010d20 con 0x7f92c0078d50 2024-03-30T04:17:16.453 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:13.025+0000 7f92c6200700 1 -- 172.21.15.123:0/2589704357 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92c0113d80 con 0x7f92c0078d50 2024-03-30T04:17:16.453 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:13.025+0000 7f92b7400700 1 -- 172.21.15.123:0/2589704357 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f92b001acf0 con 0x7f92c0078d50 2024-03-30T04:17:16.453 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:13.025+0000 7f92c6200700 1 -- 172.21.15.123:0/2589704357 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92c01148f0 con 0x7f92c0078d50 2024-03-30T04:17:16.454 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:13.026+0000 7f92b7400700 1 -- 172.21.15.123:0/2589704357 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f92b001a5a0 con 0x7f92c0078d50 2024-03-30T04:17:16.454 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:13.026+0000 7f92c6200700 1 -- 172.21.15.123:0/2589704357 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92c005f5f0 con 0x7f92c0078d50 2024-03-30T04:17:16.454 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:13.026+0000 7f92b7400700 1 --2- 172.21.15.123:0/2589704357 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f92a40367e0 0x7f92a4038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:16.454 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:13.026+0000 7f92b7400700 1 -- 172.21.15.123:0/2589704357 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f92b001e070 con 0x7f92c0078d50 2024-03-30T04:17:16.454 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:13.027+0000 7f92bea00700 1 --2- 172.21.15.123:0/2589704357 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f92a40367e0 0x7f92a4038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:16.454 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:13.027+0000 7f92bea00700 1 --2- 172.21.15.123:0/2589704357 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f92a40367e0 0x7f92a4038c90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f92ac009990 tx=0x7f92ac006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:16.454 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:13.031+0000 7f92b7400700 1 -- 172.21.15.123:0/2589704357 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f92b0010e90 con 0x7f92c0078d50 2024-03-30T04:17:16.454 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:13.232+0000 7f92c6200700 1 -- 172.21.15.123:0/2589704357 --> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi123", "addr": "172.21.15.123", "target": ["mon-mgr", ""]}) v1 -- 0x7f92c00ff090 con 0x7f92a40367e0 2024-03-30T04:17:16.454 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:15.767+0000 7f92b7400700 1 -- 172.21.15.123:0/2589704357 <== mgr.14118 v2:172.21.15.123:6800/1776532690 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f92c00ff090 con 0x7f92a40367e0 2024-03-30T04:17:16.454 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:15.770+0000 7f92c6200700 1 -- 172.21.15.123:0/2589704357 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f92a40367e0 msgr2=0x7f92a4038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:16.454 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:15.770+0000 7f92c6200700 1 --2- 172.21.15.123:0/2589704357 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f92a40367e0 0x7f92a4038c90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f92ac009990 tx=0x7f92ac006e40 comp rx=0 tx=0).stop 2024-03-30T04:17:16.454 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:15.770+0000 7f92c6200700 1 -- 172.21.15.123:0/2589704357 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92c0078d50 msgr2=0x7f92c0113640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:16.454 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:15.770+0000 7f92c6200700 1 --2- 172.21.15.123:0/2589704357 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92c0078d50 0x7f92c0113640 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f92b0006b00 tx=0x7f92b0009250 comp rx=0 tx=0).stop 2024-03-30T04:17:16.455 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:15.770+0000 7f92c6200700 1 -- 172.21.15.123:0/2589704357 shutdown_connections 2024-03-30T04:17:16.455 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:15.770+0000 7f92c6200700 1 --2- 172.21.15.123:0/2589704357 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f92a40367e0 0x7f92a4038c90 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:16.455 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:15.770+0000 7f92c6200700 1 --2- 172.21.15.123:0/2589704357 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92c0078d50 0x7f92c0113640 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:16.455 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:15.770+0000 7f92c6200700 1 -- 172.21.15.123:0/2589704357 >> 172.21.15.123:0/2589704357 conn(0x7f92c00fdcd0 msgr2=0x7f92c00fe8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:16.455 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:15.770+0000 7f92c6200700 1 -- 172.21.15.123:0/2589704357 shutdown_connections 2024-03-30T04:17:16.455 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:15.770+0000 7f92c6200700 1 -- 172.21.15.123:0/2589704357 wait complete. 2024-03-30T04:17:16.455 INFO:teuthology.orchestra.run.smithi123.stdout:Deploying mon service with default placement... 2024-03-30T04:17:17.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:16 smithi123 ceph-mon[35245]: from='mgr.14118 172.21.15.123:0/1102803547' entity='mgr.smithi123.xgksqq' 2024-03-30T04:17:17.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:16 smithi123 ceph-mon[35245]: Added host smithi123 2024-03-30T04:17:17.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:16 smithi123 ceph-mon[35245]: from='mgr.14118 172.21.15.123:0/1102803547' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:17:18.396 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:18 smithi123 ceph-mon[35245]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:17:18.396 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:18 smithi123 ceph-mon[35245]: Saving service mon spec with placement count:5 2024-03-30T04:17:18.396 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:18 smithi123 ceph-mon[35245]: from='mgr.14118 172.21.15.123:0/1102803547' entity='mgr.smithi123.xgksqq' 2024-03-30T04:17:18.807 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-30T04:17:18.807 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:17.124+0000 7f7001c00700 1 Processor -- start 2024-03-30T04:17:18.808 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:17.124+0000 7f7001c00700 1 -- start start 2024-03-30T04:17:18.808 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:17.124+0000 7f7001c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6ffc102580 0x7f6ffc102990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:18.808 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:17.124+0000 7f7001c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ffc102ed0 con 0x7f6ffc102580 2024-03-30T04:17:18.808 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:17.125+0000 7f6ffaa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6ffc102580 0x7f6ffc102990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:18.808 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:17.125+0000 7f6ffaa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6ffc102580 0x7f6ffc102990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34114/0 (socket says 172.21.15.123:34114) 2024-03-30T04:17:18.808 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:17.125+0000 7f6ffaa00700 1 -- 172.21.15.123:0/2379087134 learned_addr learned my addr 172.21.15.123:0/2379087134 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:17:18.808 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:17.125+0000 7f6ffaa00700 1 -- 172.21.15.123:0/2379087134 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ffc103010 con 0x7f6ffc102580 2024-03-30T04:17:18.808 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:17.125+0000 7f6ffaa00700 1 --2- 172.21.15.123:0/2379087134 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6ffc102580 0x7f6ffc102990 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6fec0098e0 tx=0x7f6fec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15e2d869e85124be server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:18.808 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:17.126+0000 7f6ff9600700 1 -- 172.21.15.123:0/2379087134 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6fec00b860 con 0x7f6ffc102580 2024-03-30T04:17:18.808 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:17.126+0000 7f6ff9600700 1 -- 172.21.15.123:0/2379087134 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f6fec00be30 con 0x7f6ffc102580 2024-03-30T04:17:18.808 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:17.126+0000 7f6ff9600700 1 -- 172.21.15.123:0/2379087134 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6fec010380 con 0x7f6ffc102580 2024-03-30T04:17:18.808 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:17.126+0000 7f7001c00700 1 -- 172.21.15.123:0/2379087134 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6ffc102580 msgr2=0x7f6ffc102990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:18.809 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:17.126+0000 7f7001c00700 1 --2- 172.21.15.123:0/2379087134 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6ffc102580 0x7f6ffc102990 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6fec0098e0 tx=0x7f6fec009250 comp rx=0 tx=0).stop 2024-03-30T04:17:18.809 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:17.126+0000 7f7001c00700 1 -- 172.21.15.123:0/2379087134 shutdown_connections 2024-03-30T04:17:18.809 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:17.126+0000 7f7001c00700 1 --2- 172.21.15.123:0/2379087134 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6ffc102580 0x7f6ffc102990 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:18.809 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:17.126+0000 7f7001c00700 1 -- 172.21.15.123:0/2379087134 >> 172.21.15.123:0/2379087134 conn(0x7f6ffc0fdd00 msgr2=0x7f6ffc100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:18.809 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:17.127+0000 7f7001c00700 1 -- 172.21.15.123:0/2379087134 shutdown_connections 2024-03-30T04:17:18.809 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:17.127+0000 7f7001c00700 1 -- 172.21.15.123:0/2379087134 wait complete. 2024-03-30T04:17:18.809 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:17.127+0000 7f7001c00700 1 Processor -- start 2024-03-30T04:17:18.809 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:17.128+0000 7f7001c00700 1 -- start start 2024-03-30T04:17:18.809 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:17.128+0000 7f7001c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6ffc102580 0x7f6ffc111340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:18.809 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:17.128+0000 7f7001c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ffc111880 con 0x7f6ffc102580 2024-03-30T04:17:18.809 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:17.128+0000 7f6ffaa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6ffc102580 0x7f6ffc111340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:18.810 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:17.128+0000 7f6ffaa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6ffc102580 0x7f6ffc111340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34120/0 (socket says 172.21.15.123:34120) 2024-03-30T04:17:18.810 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:17.128+0000 7f6ffaa00700 1 -- 172.21.15.123:0/975349469 learned_addr learned my addr 172.21.15.123:0/975349469 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:17:18.810 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:17.128+0000 7f6ffaa00700 1 -- 172.21.15.123:0/975349469 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fec0094f0 con 0x7f6ffc102580 2024-03-30T04:17:18.810 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:17.129+0000 7f6ffaa00700 1 --2- 172.21.15.123:0/975349469 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6ffc102580 0x7f6ffc111340 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6fec000c00 tx=0x7f6fec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:18.810 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:17.129+0000 7f6ff2a00700 1 -- 172.21.15.123:0/975349469 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6fec010a80 con 0x7f6ffc102580 2024-03-30T04:17:18.810 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:17.129+0000 7f6ff2a00700 1 -- 172.21.15.123:0/975349469 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f6fec01a430 con 0x7f6ffc102580 2024-03-30T04:17:18.810 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:17.129+0000 7f6ff2a00700 1 -- 172.21.15.123:0/975349469 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6fec0245b0 con 0x7f6ffc102580 2024-03-30T04:17:18.810 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:17.129+0000 7f7001c00700 1 -- 172.21.15.123:0/975349469 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ffc111a80 con 0x7f6ffc102580 2024-03-30T04:17:18.810 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:17.129+0000 7f7001c00700 1 -- 172.21.15.123:0/975349469 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ffc1125f0 con 0x7f6ffc102580 2024-03-30T04:17:18.810 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:17.131+0000 7f6fe7e00700 1 -- 172.21.15.123:0/975349469 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ffc05f5f0 con 0x7f6ffc102580 2024-03-30T04:17:18.810 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:17.131+0000 7f6ff2a00700 1 -- 172.21.15.123:0/975349469 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f6fec01a5a0 con 0x7f6ffc102580 2024-03-30T04:17:18.811 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:17.132+0000 7f6ff2a00700 1 --2- 172.21.15.123:0/975349469 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f6fe00367e0 0x7f6fe0038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:18.811 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:17.132+0000 7f6ff2a00700 1 -- 172.21.15.123:0/975349469 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f6fec04abf0 con 0x7f6ffc102580 2024-03-30T04:17:18.811 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:17.132+0000 7f6ffa000700 1 --2- 172.21.15.123:0/975349469 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f6fe00367e0 0x7f6fe0038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:18.811 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:17.132+0000 7f6ffa000700 1 --2- 172.21.15.123:0/975349469 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f6fe00367e0 0x7f6fe0038c90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8009990 tx=0x7f6fe8006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:18.811 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:17.137+0000 7f6ff2a00700 1 -- 172.21.15.123:0/975349469 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f6fec023df0 con 0x7f6ffc102580 2024-03-30T04:17:18.811 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:17.372+0000 7f6fe7e00700 1 -- 172.21.15.123:0/975349469 --> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f6ffc104390 con 0x7f6fe00367e0 2024-03-30T04:17:18.811 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:17.377+0000 7f6ff2a00700 1 -- 172.21.15.123:0/975349469 <== mgr.14118 v2:172.21.15.123:6800/1776532690 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f6ffc104390 con 0x7f6fe00367e0 2024-03-30T04:17:18.811 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:17.379+0000 7f6fe7e00700 1 -- 172.21.15.123:0/975349469 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f6fe00367e0 msgr2=0x7f6fe0038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:18.811 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:17.379+0000 7f6fe7e00700 1 --2- 172.21.15.123:0/975349469 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f6fe00367e0 0x7f6fe0038c90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8009990 tx=0x7f6fe8006e40 comp rx=0 tx=0).stop 2024-03-30T04:17:18.811 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:17.379+0000 7f6fe7e00700 1 -- 172.21.15.123:0/975349469 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6ffc102580 msgr2=0x7f6ffc111340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:18.811 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:17.379+0000 7f6fe7e00700 1 --2- 172.21.15.123:0/975349469 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6ffc102580 0x7f6ffc111340 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6fec000c00 tx=0x7f6fec009250 comp rx=0 tx=0).stop 2024-03-30T04:17:18.812 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:17.379+0000 7f6fe7e00700 1 -- 172.21.15.123:0/975349469 shutdown_connections 2024-03-30T04:17:18.812 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:17.379+0000 7f6fe7e00700 1 --2- 172.21.15.123:0/975349469 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f6fe00367e0 0x7f6fe0038c90 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:18.812 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:17.379+0000 7f6fe7e00700 1 --2- 172.21.15.123:0/975349469 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6ffc102580 0x7f6ffc111340 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:18.812 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:17.379+0000 7f6fe7e00700 1 -- 172.21.15.123:0/975349469 >> 172.21.15.123:0/975349469 conn(0x7f6ffc0fdd00 msgr2=0x7f6ffc0fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:18.812 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:17.379+0000 7f6fe7e00700 1 -- 172.21.15.123:0/975349469 shutdown_connections 2024-03-30T04:17:18.812 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:17.379+0000 7f6fe7e00700 1 -- 172.21.15.123:0/975349469 wait complete. 2024-03-30T04:17:18.812 INFO:teuthology.orchestra.run.smithi123.stdout:Deploying mgr service with default placement... 2024-03-30T04:17:19.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:19 smithi123 ceph-mon[35245]: from='mgr.14118 172.21.15.123:0/1102803547' entity='mgr.smithi123.xgksqq' 2024-03-30T04:17:19.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:19 smithi123 ceph-mon[35245]: from='mgr.14118 172.21.15.123:0/1102803547' entity='mgr.smithi123.xgksqq' 2024-03-30T04:17:20.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:20 smithi123 ceph-mon[35245]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:17:20.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:20 smithi123 ceph-mon[35245]: Saving service mgr spec with placement count:2 2024-03-30T04:17:20.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:20 smithi123 ceph-mon[35245]: from='mgr.14118 172.21.15.123:0/1102803547' entity='mgr.smithi123.xgksqq' 2024-03-30T04:17:20.971 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-30T04:17:20.976 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:19.438+0000 7f9253000700 1 Processor -- start 2024-03-30T04:17:20.976 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:19.438+0000 7f9253000700 1 -- start start 2024-03-30T04:17:20.976 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:19.439+0000 7f9253000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f924c102580 0x7f924c102990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:20.976 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:19.439+0000 7f9253000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f924c102ed0 con 0x7f924c102580 2024-03-30T04:17:20.976 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:19.439+0000 7f924be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f924c102580 0x7f924c102990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:20.976 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:19.439+0000 7f924be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f924c102580 0x7f924c102990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34122/0 (socket says 172.21.15.123:34122) 2024-03-30T04:17:20.976 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:19.439+0000 7f924be00700 1 -- 172.21.15.123:0/1922352907 learned_addr learned my addr 172.21.15.123:0/1922352907 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:17:20.976 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:19.439+0000 7f924be00700 1 -- 172.21.15.123:0/1922352907 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f924c103010 con 0x7f924c102580 2024-03-30T04:17:20.976 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:19.440+0000 7f924be00700 1 --2- 172.21.15.123:0/1922352907 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f924c102580 0x7f924c102990 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f923c0098e0 tx=0x7f923c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2cf5a0b4b06ae6e server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:20.976 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:19.440+0000 7f924aa00700 1 -- 172.21.15.123:0/1922352907 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f923c00b860 con 0x7f924c102580 2024-03-30T04:17:20.977 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:19.440+0000 7f924aa00700 1 -- 172.21.15.123:0/1922352907 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f923c00be70 con 0x7f924c102580 2024-03-30T04:17:20.977 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:19.440+0000 7f924aa00700 1 -- 172.21.15.123:0/1922352907 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f923c010380 con 0x7f924c102580 2024-03-30T04:17:20.977 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:19.440+0000 7f9253000700 1 -- 172.21.15.123:0/1922352907 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f924c102580 msgr2=0x7f924c102990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:20.977 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:19.440+0000 7f9253000700 1 --2- 172.21.15.123:0/1922352907 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f924c102580 0x7f924c102990 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f923c0098e0 tx=0x7f923c009250 comp rx=0 tx=0).stop 2024-03-30T04:17:20.977 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:19.441+0000 7f9253000700 1 -- 172.21.15.123:0/1922352907 shutdown_connections 2024-03-30T04:17:20.977 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:19.441+0000 7f9253000700 1 --2- 172.21.15.123:0/1922352907 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f924c102580 0x7f924c102990 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:20.977 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:19.441+0000 7f9253000700 1 -- 172.21.15.123:0/1922352907 >> 172.21.15.123:0/1922352907 conn(0x7f924c0fdd00 msgr2=0x7f924c100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:20.977 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:19.441+0000 7f9253000700 1 -- 172.21.15.123:0/1922352907 shutdown_connections 2024-03-30T04:17:20.977 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:19.441+0000 7f9253000700 1 -- 172.21.15.123:0/1922352907 wait complete. 2024-03-30T04:17:20.977 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:19.442+0000 7f9253000700 1 Processor -- start 2024-03-30T04:17:20.977 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:19.442+0000 7f9253000700 1 -- start start 2024-03-30T04:17:20.977 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:19.442+0000 7f9253000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f924c102580 0x7f924c113580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:20.977 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:19.442+0000 7f9253000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f924c113ac0 con 0x7f924c102580 2024-03-30T04:17:20.977 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:19.442+0000 7f924be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f924c102580 0x7f924c113580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:20.978 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:19.442+0000 7f924be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f924c102580 0x7f924c113580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34126/0 (socket says 172.21.15.123:34126) 2024-03-30T04:17:20.978 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:19.442+0000 7f924be00700 1 -- 172.21.15.123:0/284889085 learned_addr learned my addr 172.21.15.123:0/284889085 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:17:20.978 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:19.443+0000 7f924be00700 1 -- 172.21.15.123:0/284889085 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f923c0094f0 con 0x7f924c102580 2024-03-30T04:17:20.978 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:19.443+0000 7f924be00700 1 --2- 172.21.15.123:0/284889085 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f924c102580 0x7f924c113580 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f923c00c010 tx=0x7f923c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:20.978 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:19.443+0000 7f9243e00700 1 -- 172.21.15.123:0/284889085 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f923c010960 con 0x7f924c102580 2024-03-30T04:17:20.978 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:19.443+0000 7f9243e00700 1 -- 172.21.15.123:0/284889085 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f923c01a430 con 0x7f924c102580 2024-03-30T04:17:20.978 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:19.443+0000 7f9253000700 1 -- 172.21.15.123:0/284889085 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f924c113cc0 con 0x7f924c102580 2024-03-30T04:17:20.978 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:19.443+0000 7f9243e00700 1 -- 172.21.15.123:0/284889085 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f923c0245b0 con 0x7f924c102580 2024-03-30T04:17:20.978 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:19.443+0000 7f9253000700 1 -- 172.21.15.123:0/284889085 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f924c114830 con 0x7f924c102580 2024-03-30T04:17:20.978 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:19.444+0000 7f9243e00700 1 -- 172.21.15.123:0/284889085 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f923c0247f0 con 0x7f924c102580 2024-03-30T04:17:20.978 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:19.445+0000 7f9253000700 1 -- 172.21.15.123:0/284889085 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f924c05f5f0 con 0x7f924c102580 2024-03-30T04:17:20.978 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:19.445+0000 7f9243e00700 1 --2- 172.21.15.123:0/284889085 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f92300367e0 0x7f9230038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:20.978 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:19.445+0000 7f9243e00700 1 -- 172.21.15.123:0/284889085 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f923c04a7d0 con 0x7f924c102580 2024-03-30T04:17:20.979 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:19.445+0000 7f924b400700 1 --2- 172.21.15.123:0/284889085 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f92300367e0 0x7f9230038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:20.979 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:19.446+0000 7f924b400700 1 --2- 172.21.15.123:0/284889085 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f92300367e0 0x7f9230038c90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9238009990 tx=0x7f9238006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:20.979 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:19.450+0000 7f9243e00700 1 -- 172.21.15.123:0/284889085 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f923c018d90 con 0x7f924c102580 2024-03-30T04:17:20.979 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:19.672+0000 7f9253000700 1 -- 172.21.15.123:0/284889085 --> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f924c0fee90 con 0x7f92300367e0 2024-03-30T04:17:20.979 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:19.680+0000 7f9243e00700 1 -- 172.21.15.123:0/284889085 <== mgr.14118 v2:172.21.15.123:6800/1776532690 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f924c0fee90 con 0x7f92300367e0 2024-03-30T04:17:20.979 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:19.682+0000 7f9253000700 1 -- 172.21.15.123:0/284889085 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f92300367e0 msgr2=0x7f9230038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:20.979 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:19.682+0000 7f9253000700 1 --2- 172.21.15.123:0/284889085 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f92300367e0 0x7f9230038c90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9238009990 tx=0x7f9238006e40 comp rx=0 tx=0).stop 2024-03-30T04:17:20.979 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:19.682+0000 7f9253000700 1 -- 172.21.15.123:0/284889085 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f924c102580 msgr2=0x7f924c113580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:20.979 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:19.682+0000 7f9253000700 1 --2- 172.21.15.123:0/284889085 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f924c102580 0x7f924c113580 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f923c00c010 tx=0x7f923c009250 comp rx=0 tx=0).stop 2024-03-30T04:17:20.979 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:19.683+0000 7f9253000700 1 -- 172.21.15.123:0/284889085 shutdown_connections 2024-03-30T04:17:20.979 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:19.683+0000 7f9253000700 1 --2- 172.21.15.123:0/284889085 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f92300367e0 0x7f9230038c90 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:20.979 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:19.683+0000 7f9253000700 1 --2- 172.21.15.123:0/284889085 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f924c102580 0x7f924c113580 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:20.979 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:19.683+0000 7f9253000700 1 -- 172.21.15.123:0/284889085 >> 172.21.15.123:0/284889085 conn(0x7f924c0fdd00 msgr2=0x7f924c0fe780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:20.979 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:19.683+0000 7f9253000700 1 -- 172.21.15.123:0/284889085 shutdown_connections 2024-03-30T04:17:20.980 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:19.683+0000 7f9253000700 1 -- 172.21.15.123:0/284889085 wait complete. 2024-03-30T04:17:20.980 INFO:teuthology.orchestra.run.smithi123.stdout:Deploying crash service with default placement... 2024-03-30T04:17:21.969 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:21 smithi123 ceph-mon[35245]: from='mgr.14118 172.21.15.123:0/1102803547' entity='mgr.smithi123.xgksqq' 2024-03-30T04:17:22.519 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-03-30T04:17:22.519 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:21.565+0000 7f4114c00700 1 Processor -- start 2024-03-30T04:17:22.519 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:21.565+0000 7f4114c00700 1 -- start start 2024-03-30T04:17:22.519 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:21.565+0000 7f4114c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4110102580 0x7f4110102990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:22.519 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:21.565+0000 7f4114c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4110102ed0 con 0x7f4110102580 2024-03-30T04:17:22.519 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:21.566+0000 7f410de00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4110102580 0x7f4110102990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:22.519 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:21.566+0000 7f410de00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4110102580 0x7f4110102990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34140/0 (socket says 172.21.15.123:34140) 2024-03-30T04:17:22.519 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:21.566+0000 7f410de00700 1 -- 172.21.15.123:0/3234974453 learned_addr learned my addr 172.21.15.123:0/3234974453 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:17:22.519 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:21.566+0000 7f410de00700 1 -- 172.21.15.123:0/3234974453 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4110103010 con 0x7f4110102580 2024-03-30T04:17:22.519 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:21.566+0000 7f410de00700 1 --2- 172.21.15.123:0/3234974453 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4110102580 0x7f4110102990 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f41000098e0 tx=0x7f4100009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1208a2eba217600 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:22.519 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:21.567+0000 7f410ca00700 1 -- 172.21.15.123:0/3234974453 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f410000b860 con 0x7f4110102580 2024-03-30T04:17:22.520 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:21.567+0000 7f410ca00700 1 -- 172.21.15.123:0/3234974453 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f410000be70 con 0x7f4110102580 2024-03-30T04:17:22.520 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:21.567+0000 7f410ca00700 1 -- 172.21.15.123:0/3234974453 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4100010380 con 0x7f4110102580 2024-03-30T04:17:22.520 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:21.567+0000 7f4114c00700 1 -- 172.21.15.123:0/3234974453 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4110102580 msgr2=0x7f4110102990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:22.520 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:21.567+0000 7f4114c00700 1 --2- 172.21.15.123:0/3234974453 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4110102580 0x7f4110102990 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f41000098e0 tx=0x7f4100009250 comp rx=0 tx=0).stop 2024-03-30T04:17:22.520 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:21.567+0000 7f4114c00700 1 -- 172.21.15.123:0/3234974453 shutdown_connections 2024-03-30T04:17:22.520 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:21.567+0000 7f4114c00700 1 --2- 172.21.15.123:0/3234974453 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4110102580 0x7f4110102990 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:22.520 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:21.567+0000 7f4114c00700 1 -- 172.21.15.123:0/3234974453 >> 172.21.15.123:0/3234974453 conn(0x7f41100fdd00 msgr2=0x7f4110100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:22.520 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:21.568+0000 7f4114c00700 1 -- 172.21.15.123:0/3234974453 shutdown_connections 2024-03-30T04:17:22.520 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:21.568+0000 7f4114c00700 1 -- 172.21.15.123:0/3234974453 wait complete. 2024-03-30T04:17:22.520 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:21.569+0000 7f4114c00700 1 Processor -- start 2024-03-30T04:17:22.520 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:21.569+0000 7f4114c00700 1 -- start start 2024-03-30T04:17:22.520 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:21.569+0000 7f4114c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4110102580 0x7f4110078c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:22.520 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:21.569+0000 7f4114c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4110079190 con 0x7f4110102580 2024-03-30T04:17:22.520 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:21.570+0000 7f410de00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4110102580 0x7f4110078c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:22.521 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:21.570+0000 7f410de00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4110102580 0x7f4110078c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34156/0 (socket says 172.21.15.123:34156) 2024-03-30T04:17:22.521 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:21.570+0000 7f410de00700 1 -- 172.21.15.123:0/1256171435 learned_addr learned my addr 172.21.15.123:0/1256171435 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:17:22.521 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:21.570+0000 7f410de00700 1 -- 172.21.15.123:0/1256171435 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41000094f0 con 0x7f4110102580 2024-03-30T04:17:22.521 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:21.570+0000 7f410de00700 1 --2- 172.21.15.123:0/1256171435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4110102580 0x7f4110078c50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f410000c010 tx=0x7f4100009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:22.521 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:21.570+0000 7f4106000700 1 -- 172.21.15.123:0/1256171435 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4100010960 con 0x7f4110102580 2024-03-30T04:17:22.521 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:21.571+0000 7f4106000700 1 -- 172.21.15.123:0/1256171435 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f410001a430 con 0x7f4110102580 2024-03-30T04:17:22.521 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:21.571+0000 7f4114c00700 1 -- 172.21.15.123:0/1256171435 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4110077200 con 0x7f4110102580 2024-03-30T04:17:22.521 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:21.571+0000 7f4106000700 1 -- 172.21.15.123:0/1256171435 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4100019580 con 0x7f4110102580 2024-03-30T04:17:22.521 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:21.571+0000 7f4114c00700 1 -- 172.21.15.123:0/1256171435 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41100776f0 con 0x7f4110102580 2024-03-30T04:17:22.521 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:21.572+0000 7f4114c00700 1 -- 172.21.15.123:0/1256171435 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f411004bed0 con 0x7f4110102580 2024-03-30T04:17:22.521 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:21.572+0000 7f4106000700 1 -- 172.21.15.123:0/1256171435 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f41000197b0 con 0x7f4110102580 2024-03-30T04:17:22.521 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:21.572+0000 7f4106000700 1 --2- 172.21.15.123:0/1256171435 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f40f40367e0 0x7f40f4038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:22.521 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:21.573+0000 7f4106000700 1 -- 172.21.15.123:0/1256171435 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f410004a640 con 0x7f4110102580 2024-03-30T04:17:22.522 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:21.573+0000 7f410d400700 1 --2- 172.21.15.123:0/1256171435 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f40f40367e0 0x7f40f4038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:22.522 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:21.573+0000 7f410d400700 1 --2- 172.21.15.123:0/1256171435 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f40f40367e0 0x7f40f4038c90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f40fc009990 tx=0x7f40fc006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:22.522 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:21.577+0000 7f4106000700 1 -- 172.21.15.123:0/1256171435 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f410002b950 con 0x7f4110102580 2024-03-30T04:17:22.522 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:21.817+0000 7f4114c00700 1 -- 172.21.15.123:0/1256171435 --> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f4110077d80 con 0x7f40f40367e0 2024-03-30T04:17:22.522 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:21.822+0000 7f4106000700 1 -- 172.21.15.123:0/1256171435 <== mgr.14118 v2:172.21.15.123:6800/1776532690 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f4110077d80 con 0x7f40f40367e0 2024-03-30T04:17:22.522 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:21.824+0000 7f4114c00700 1 -- 172.21.15.123:0/1256171435 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f40f40367e0 msgr2=0x7f40f4038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:22.522 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:21.824+0000 7f4114c00700 1 --2- 172.21.15.123:0/1256171435 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f40f40367e0 0x7f40f4038c90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f40fc009990 tx=0x7f40fc006e40 comp rx=0 tx=0).stop 2024-03-30T04:17:22.522 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:21.824+0000 7f4114c00700 1 -- 172.21.15.123:0/1256171435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4110102580 msgr2=0x7f4110078c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:22.522 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:21.824+0000 7f4114c00700 1 --2- 172.21.15.123:0/1256171435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4110102580 0x7f4110078c50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f410000c010 tx=0x7f4100009250 comp rx=0 tx=0).stop 2024-03-30T04:17:22.522 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:21.824+0000 7f4114c00700 1 -- 172.21.15.123:0/1256171435 shutdown_connections 2024-03-30T04:17:22.522 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:21.824+0000 7f4114c00700 1 --2- 172.21.15.123:0/1256171435 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f40f40367e0 0x7f40f4038c90 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:22.522 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:21.824+0000 7f4114c00700 1 --2- 172.21.15.123:0/1256171435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4110102580 0x7f4110078c50 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:22.522 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:21.824+0000 7f4114c00700 1 -- 172.21.15.123:0/1256171435 >> 172.21.15.123:0/1256171435 conn(0x7f41100fdd00 msgr2=0x7f41100fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:22.522 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:21.825+0000 7f4114c00700 1 -- 172.21.15.123:0/1256171435 shutdown_connections 2024-03-30T04:17:22.523 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:21.825+0000 7f4114c00700 1 -- 172.21.15.123:0/1256171435 wait complete. 2024-03-30T04:17:22.523 INFO:teuthology.orchestra.run.smithi123.stdout:Deploying prometheus service with default placement... 2024-03-30T04:17:23.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:22 smithi123 ceph-mon[35245]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:17:23.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:22 smithi123 ceph-mon[35245]: Saving service crash spec with placement * 2024-03-30T04:17:23.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:22 smithi123 ceph-mon[35245]: from='mgr.14118 172.21.15.123:0/1102803547' entity='mgr.smithi123.xgksqq' 2024-03-30T04:17:23.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:22 smithi123 ceph-mon[35245]: from='mgr.14118 172.21.15.123:0/1102803547' entity='mgr.smithi123.xgksqq' 2024-03-30T04:17:24.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:24 smithi123 ceph-mon[35245]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:17:24.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:24 smithi123 ceph-mon[35245]: Saving service prometheus spec with placement count:1 2024-03-30T04:17:24.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:24 smithi123 ceph-mon[35245]: from='mgr.14118 172.21.15.123:0/1102803547' entity='mgr.smithi123.xgksqq' 2024-03-30T04:17:24.808 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-03-30T04:17:24.809 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:23.122+0000 7f4e84a00700 1 Processor -- start 2024-03-30T04:17:24.809 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:23.122+0000 7f4e84a00700 1 -- start start 2024-03-30T04:17:24.809 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:23.123+0000 7f4e84a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e80078d50 0x7f4e80077260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:24.809 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:23.123+0000 7f4e84a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e80079160 con 0x7f4e80078d50 2024-03-30T04:17:24.809 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:23.123+0000 7f4e7de00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e80078d50 0x7f4e80077260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:24.809 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:23.123+0000 7f4e7de00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e80078d50 0x7f4e80077260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34160/0 (socket says 172.21.15.123:34160) 2024-03-30T04:17:24.809 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:23.123+0000 7f4e7de00700 1 -- 172.21.15.123:0/2514816410 learned_addr learned my addr 172.21.15.123:0/2514816410 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:17:24.809 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:23.123+0000 7f4e7de00700 1 -- 172.21.15.123:0/2514816410 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e800777a0 con 0x7f4e80078d50 2024-03-30T04:17:24.809 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:23.124+0000 7f4e7de00700 1 --2- 172.21.15.123:0/2514816410 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e80078d50 0x7f4e80077260 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4e700098e0 tx=0x7f4e70009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a359123fcc187999 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:24.809 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:23.124+0000 7f4e7ca00700 1 -- 172.21.15.123:0/2514816410 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e7000b860 con 0x7f4e80078d50 2024-03-30T04:17:24.809 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:23.124+0000 7f4e7ca00700 1 -- 172.21.15.123:0/2514816410 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4e7000be70 con 0x7f4e80078d50 2024-03-30T04:17:24.809 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:23.124+0000 7f4e7ca00700 1 -- 172.21.15.123:0/2514816410 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e70010380 con 0x7f4e80078d50 2024-03-30T04:17:24.810 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:23.124+0000 7f4e84a00700 1 -- 172.21.15.123:0/2514816410 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e80078d50 msgr2=0x7f4e80077260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:24.810 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:23.124+0000 7f4e84a00700 1 --2- 172.21.15.123:0/2514816410 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e80078d50 0x7f4e80077260 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4e700098e0 tx=0x7f4e70009250 comp rx=0 tx=0).stop 2024-03-30T04:17:24.810 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:23.125+0000 7f4e84a00700 1 -- 172.21.15.123:0/2514816410 shutdown_connections 2024-03-30T04:17:24.810 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:23.125+0000 7f4e84a00700 1 --2- 172.21.15.123:0/2514816410 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e80078d50 0x7f4e80077260 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:24.810 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:23.125+0000 7f4e84a00700 1 -- 172.21.15.123:0/2514816410 >> 172.21.15.123:0/2514816410 conn(0x7f4e800fdd00 msgr2=0x7f4e80100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:24.810 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:23.125+0000 7f4e84a00700 1 -- 172.21.15.123:0/2514816410 shutdown_connections 2024-03-30T04:17:24.810 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:23.125+0000 7f4e84a00700 1 -- 172.21.15.123:0/2514816410 wait complete. 2024-03-30T04:17:24.810 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:23.126+0000 7f4e84a00700 1 Processor -- start 2024-03-30T04:17:24.810 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:23.126+0000 7f4e84a00700 1 -- start start 2024-03-30T04:17:24.810 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:23.126+0000 7f4e84a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e80078d50 0x7f4e801135a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:24.811 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:23.126+0000 7f4e84a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e80113ae0 con 0x7f4e80078d50 2024-03-30T04:17:24.811 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:23.126+0000 7f4e7de00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e80078d50 0x7f4e801135a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:24.811 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:23.126+0000 7f4e7de00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e80078d50 0x7f4e801135a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34174/0 (socket says 172.21.15.123:34174) 2024-03-30T04:17:24.811 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:23.126+0000 7f4e7de00700 1 -- 172.21.15.123:0/2541232422 learned_addr learned my addr 172.21.15.123:0/2541232422 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:17:24.811 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:23.127+0000 7f4e7de00700 1 -- 172.21.15.123:0/2541232422 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e700094f0 con 0x7f4e80078d50 2024-03-30T04:17:24.811 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:23.127+0000 7f4e7de00700 1 --2- 172.21.15.123:0/2541232422 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e80078d50 0x7f4e801135a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4e70006d40 tx=0x7f4e70009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:24.811 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:23.127+0000 7f4e76000700 1 -- 172.21.15.123:0/2541232422 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e70010960 con 0x7f4e80078d50 2024-03-30T04:17:24.811 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:23.127+0000 7f4e76000700 1 -- 172.21.15.123:0/2541232422 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4e7001a430 con 0x7f4e80078d50 2024-03-30T04:17:24.811 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:23.127+0000 7f4e76000700 1 -- 172.21.15.123:0/2541232422 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e700245b0 con 0x7f4e80078d50 2024-03-30T04:17:24.811 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:23.127+0000 7f4e84a00700 1 -- 172.21.15.123:0/2541232422 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e80113ce0 con 0x7f4e80078d50 2024-03-30T04:17:24.811 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:23.127+0000 7f4e84a00700 1 -- 172.21.15.123:0/2541232422 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e80114850 con 0x7f4e80078d50 2024-03-30T04:17:24.811 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:23.129+0000 7f4e76000700 1 -- 172.21.15.123:0/2541232422 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f4e7001a9f0 con 0x7f4e80078d50 2024-03-30T04:17:24.811 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:23.129+0000 7f4e84a00700 1 -- 172.21.15.123:0/2541232422 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e8005f5f0 con 0x7f4e80078d50 2024-03-30T04:17:24.811 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:23.129+0000 7f4e76000700 1 --2- 172.21.15.123:0/2541232422 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f4e640367e0 0x7f4e64038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:24.812 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:23.129+0000 7f4e76000700 1 -- 172.21.15.123:0/2541232422 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f4e7004a310 con 0x7f4e80078d50 2024-03-30T04:17:24.812 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:23.130+0000 7f4e7d400700 1 --2- 172.21.15.123:0/2541232422 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f4e640367e0 0x7f4e64038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:24.812 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:23.130+0000 7f4e7d400700 1 --2- 172.21.15.123:0/2541232422 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f4e640367e0 0x7f4e64038c90 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c009990 tx=0x7f4e6c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:24.812 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:23.136+0000 7f4e76000700 1 -- 172.21.15.123:0/2541232422 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f4e7002b8c0 con 0x7f4e80078d50 2024-03-30T04:17:24.812 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:23.361+0000 7f4e84a00700 1 -- 172.21.15.123:0/2541232422 --> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f4e800fef50 con 0x7f4e640367e0 2024-03-30T04:17:24.812 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:23.365+0000 7f4e76000700 1 -- 172.21.15.123:0/2541232422 <== mgr.14118 v2:172.21.15.123:6800/1776532690 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f4e800fef50 con 0x7f4e640367e0 2024-03-30T04:17:24.812 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:23.368+0000 7f4e84a00700 1 -- 172.21.15.123:0/2541232422 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f4e640367e0 msgr2=0x7f4e64038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:24.812 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:23.368+0000 7f4e84a00700 1 --2- 172.21.15.123:0/2541232422 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f4e640367e0 0x7f4e64038c90 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c009990 tx=0x7f4e6c006e40 comp rx=0 tx=0).stop 2024-03-30T04:17:24.812 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:23.368+0000 7f4e84a00700 1 -- 172.21.15.123:0/2541232422 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e80078d50 msgr2=0x7f4e801135a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:24.812 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:23.368+0000 7f4e84a00700 1 --2- 172.21.15.123:0/2541232422 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e80078d50 0x7f4e801135a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4e70006d40 tx=0x7f4e70009250 comp rx=0 tx=0).stop 2024-03-30T04:17:24.812 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:23.368+0000 7f4e84a00700 1 -- 172.21.15.123:0/2541232422 shutdown_connections 2024-03-30T04:17:24.812 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:23.368+0000 7f4e84a00700 1 --2- 172.21.15.123:0/2541232422 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f4e640367e0 0x7f4e64038c90 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:24.812 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:23.368+0000 7f4e84a00700 1 --2- 172.21.15.123:0/2541232422 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e80078d50 0x7f4e801135a0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:24.813 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:23.368+0000 7f4e84a00700 1 -- 172.21.15.123:0/2541232422 >> 172.21.15.123:0/2541232422 conn(0x7f4e800fdd00 msgr2=0x7f4e800fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:24.813 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:23.368+0000 7f4e84a00700 1 -- 172.21.15.123:0/2541232422 shutdown_connections 2024-03-30T04:17:24.813 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:23.368+0000 7f4e84a00700 1 -- 172.21.15.123:0/2541232422 wait complete. 2024-03-30T04:17:24.813 INFO:teuthology.orchestra.run.smithi123.stdout:Deploying grafana service with default placement... 2024-03-30T04:17:26.208 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-03-30T04:17:26.209 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:25.402+0000 7eff50e00700 1 Processor -- start 2024-03-30T04:17:26.209 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:25.402+0000 7eff50e00700 1 -- start start 2024-03-30T04:17:26.209 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:25.402+0000 7eff50e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff4c105ab0 0x7eff4c105ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:26.209 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:25.402+0000 7eff50e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff4c106490 con 0x7eff4c105ab0 2024-03-30T04:17:26.209 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:25.402+0000 7eff4b400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff4c105ab0 0x7eff4c105ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:26.209 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:25.402+0000 7eff4b400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff4c105ab0 0x7eff4c105ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46608/0 (socket says 172.21.15.123:46608) 2024-03-30T04:17:26.209 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:25.402+0000 7eff4b400700 1 -- 172.21.15.123:0/3160011636 learned_addr learned my addr 172.21.15.123:0/3160011636 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:17:26.210 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:25.403+0000 7eff4b400700 1 -- 172.21.15.123:0/3160011636 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff4c1065d0 con 0x7eff4c105ab0 2024-03-30T04:17:26.210 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:25.403+0000 7eff4b400700 1 --2- 172.21.15.123:0/3160011636 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff4c105ab0 0x7eff4c105ec0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7eff3c0098e0 tx=0x7eff3c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0dd8edca029778f server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:26.210 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:25.403+0000 7eff4a000700 1 -- 172.21.15.123:0/3160011636 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff3c00b860 con 0x7eff4c105ab0 2024-03-30T04:17:26.210 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:25.403+0000 7eff4a000700 1 -- 172.21.15.123:0/3160011636 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7eff3c00be70 con 0x7eff4c105ab0 2024-03-30T04:17:26.210 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:25.403+0000 7eff4a000700 1 -- 172.21.15.123:0/3160011636 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff3c010380 con 0x7eff4c105ab0 2024-03-30T04:17:26.211 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:25.404+0000 7eff50e00700 1 -- 172.21.15.123:0/3160011636 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff4c105ab0 msgr2=0x7eff4c105ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:26.211 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:25.404+0000 7eff50e00700 1 --2- 172.21.15.123:0/3160011636 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff4c105ab0 0x7eff4c105ec0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7eff3c0098e0 tx=0x7eff3c009250 comp rx=0 tx=0).stop 2024-03-30T04:17:26.211 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:25.404+0000 7eff50e00700 1 -- 172.21.15.123:0/3160011636 shutdown_connections 2024-03-30T04:17:26.211 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:25.404+0000 7eff50e00700 1 --2- 172.21.15.123:0/3160011636 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff4c105ab0 0x7eff4c105ec0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:26.211 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:25.404+0000 7eff50e00700 1 -- 172.21.15.123:0/3160011636 >> 172.21.15.123:0/3160011636 conn(0x7eff4c077d60 msgr2=0x7eff4c078180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:26.211 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:25.404+0000 7eff50e00700 1 -- 172.21.15.123:0/3160011636 shutdown_connections 2024-03-30T04:17:26.211 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:25.404+0000 7eff50e00700 1 -- 172.21.15.123:0/3160011636 wait complete. 2024-03-30T04:17:26.211 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:25.405+0000 7eff50e00700 1 Processor -- start 2024-03-30T04:17:26.211 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:25.405+0000 7eff50e00700 1 -- start start 2024-03-30T04:17:26.211 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:25.405+0000 7eff50e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff4c105ab0 0x7eff4c113440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:26.211 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:25.405+0000 7eff50e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff4c113980 con 0x7eff4c105ab0 2024-03-30T04:17:26.211 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:25.405+0000 7eff4b400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff4c105ab0 0x7eff4c113440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:26.212 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:25.405+0000 7eff4b400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff4c105ab0 0x7eff4c113440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46610/0 (socket says 172.21.15.123:46610) 2024-03-30T04:17:26.212 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:25.405+0000 7eff4b400700 1 -- 172.21.15.123:0/151605483 learned_addr learned my addr 172.21.15.123:0/151605483 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:17:26.212 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:25.406+0000 7eff4b400700 1 -- 172.21.15.123:0/151605483 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff3c0094f0 con 0x7eff4c105ab0 2024-03-30T04:17:26.212 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:25.406+0000 7eff4b400700 1 --2- 172.21.15.123:0/151605483 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff4c105ab0 0x7eff4c113440 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7eff3c0090b0 tx=0x7eff3c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:26.212 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:25.406+0000 7eff43400700 1 -- 172.21.15.123:0/151605483 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff3c010800 con 0x7eff4c105ab0 2024-03-30T04:17:26.212 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:25.406+0000 7eff43400700 1 -- 172.21.15.123:0/151605483 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7eff3c010e10 con 0x7eff4c105ab0 2024-03-30T04:17:26.212 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:25.406+0000 7eff43400700 1 -- 172.21.15.123:0/151605483 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff3c01ae60 con 0x7eff4c105ab0 2024-03-30T04:17:26.212 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:25.406+0000 7eff50e00700 1 -- 172.21.15.123:0/151605483 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff4c113b80 con 0x7eff4c105ab0 2024-03-30T04:17:26.212 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:25.406+0000 7eff50e00700 1 -- 172.21.15.123:0/151605483 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff4c1146f0 con 0x7eff4c105ab0 2024-03-30T04:17:26.212 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:25.407+0000 7eff43400700 1 -- 172.21.15.123:0/151605483 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7eff3c010960 con 0x7eff4c105ab0 2024-03-30T04:17:26.212 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:25.407+0000 7eff50e00700 1 -- 172.21.15.123:0/151605483 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff4c05f5f0 con 0x7eff4c105ab0 2024-03-30T04:17:26.212 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:25.408+0000 7eff43400700 1 --2- 172.21.15.123:0/151605483 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7eff300367e0 0x7eff30038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:26.212 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:25.408+0000 7eff43400700 1 -- 172.21.15.123:0/151605483 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7eff3c04a6d0 con 0x7eff4c105ab0 2024-03-30T04:17:26.212 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:25.408+0000 7eff4aa00700 1 --2- 172.21.15.123:0/151605483 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7eff300367e0 0x7eff30038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:26.213 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:25.409+0000 7eff4aa00700 1 --2- 172.21.15.123:0/151605483 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7eff300367e0 0x7eff30038c90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7eff38009990 tx=0x7eff38006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:26.213 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:25.412+0000 7eff43400700 1 -- 172.21.15.123:0/151605483 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7eff3c018530 con 0x7eff4c105ab0 2024-03-30T04:17:26.213 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:25.617+0000 7eff50e00700 1 -- 172.21.15.123:0/151605483 --> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7eff4c102c90 con 0x7eff300367e0 2024-03-30T04:17:26.213 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:25.622+0000 7eff43400700 1 -- 172.21.15.123:0/151605483 <== mgr.14118 v2:172.21.15.123:6800/1776532690 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7eff4c102c90 con 0x7eff300367e0 2024-03-30T04:17:26.213 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:25.625+0000 7eff50e00700 1 -- 172.21.15.123:0/151605483 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7eff300367e0 msgr2=0x7eff30038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:26.213 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:25.625+0000 7eff50e00700 1 --2- 172.21.15.123:0/151605483 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7eff300367e0 0x7eff30038c90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7eff38009990 tx=0x7eff38006e40 comp rx=0 tx=0).stop 2024-03-30T04:17:26.213 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:25.625+0000 7eff50e00700 1 -- 172.21.15.123:0/151605483 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff4c105ab0 msgr2=0x7eff4c113440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:26.213 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:25.625+0000 7eff50e00700 1 --2- 172.21.15.123:0/151605483 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff4c105ab0 0x7eff4c113440 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7eff3c0090b0 tx=0x7eff3c009250 comp rx=0 tx=0).stop 2024-03-30T04:17:26.213 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:25.625+0000 7eff50e00700 1 -- 172.21.15.123:0/151605483 shutdown_connections 2024-03-30T04:17:26.213 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:25.625+0000 7eff50e00700 1 --2- 172.21.15.123:0/151605483 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7eff300367e0 0x7eff30038c90 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:26.213 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:25.625+0000 7eff50e00700 1 --2- 172.21.15.123:0/151605483 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff4c105ab0 0x7eff4c113440 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:26.213 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:25.625+0000 7eff50e00700 1 -- 172.21.15.123:0/151605483 >> 172.21.15.123:0/151605483 conn(0x7eff4c077d60 msgr2=0x7eff4c102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:26.213 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:25.625+0000 7eff50e00700 1 -- 172.21.15.123:0/151605483 shutdown_connections 2024-03-30T04:17:26.214 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:25.625+0000 7eff50e00700 1 -- 172.21.15.123:0/151605483 wait complete. 2024-03-30T04:17:26.214 INFO:teuthology.orchestra.run.smithi123.stdout:Deploying node-exporter service with default placement... 2024-03-30T04:17:26.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:26 smithi123 ceph-mon[35245]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:17:26.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:26 smithi123 ceph-mon[35245]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:17:26.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:26 smithi123 ceph-mon[35245]: Saving service grafana spec with placement count:1 2024-03-30T04:17:26.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:26 smithi123 ceph-mon[35245]: from='mgr.14118 172.21.15.123:0/1102803547' entity='mgr.smithi123.xgksqq' 2024-03-30T04:17:27.646 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-03-30T04:17:27.646 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:26.758+0000 7ff305e00700 1 Processor -- start 2024-03-30T04:17:27.647 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:26.758+0000 7ff305e00700 1 -- start start 2024-03-30T04:17:27.647 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:26.759+0000 7ff305e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff300078d50 0x7ff300077260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:27.647 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:26.759+0000 7ff305e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff300079160 con 0x7ff300078d50 2024-03-30T04:17:27.647 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:26.759+0000 7ff2fea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff300078d50 0x7ff300077260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:27.647 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:26.759+0000 7ff2fea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff300078d50 0x7ff300077260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46626/0 (socket says 172.21.15.123:46626) 2024-03-30T04:17:27.647 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:26.759+0000 7ff2fea00700 1 -- 172.21.15.123:0/1121106412 learned_addr learned my addr 172.21.15.123:0/1121106412 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:17:27.647 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:26.759+0000 7ff2fea00700 1 -- 172.21.15.123:0/1121106412 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3000777a0 con 0x7ff300078d50 2024-03-30T04:17:27.647 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:26.760+0000 7ff2fea00700 1 --2- 172.21.15.123:0/1121106412 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff300078d50 0x7ff300077260 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff2f00098e0 tx=0x7ff2f0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3664ab363cb71c59 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:27.647 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:26.760+0000 7ff2fd600700 1 -- 172.21.15.123:0/1121106412 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2f000b860 con 0x7ff300078d50 2024-03-30T04:17:27.647 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:26.760+0000 7ff2fd600700 1 -- 172.21.15.123:0/1121106412 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff2f000be70 con 0x7ff300078d50 2024-03-30T04:17:27.647 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:26.760+0000 7ff2fd600700 1 -- 172.21.15.123:0/1121106412 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2f0010380 con 0x7ff300078d50 2024-03-30T04:17:27.647 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:26.760+0000 7ff305e00700 1 -- 172.21.15.123:0/1121106412 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff300078d50 msgr2=0x7ff300077260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:27.648 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:26.760+0000 7ff305e00700 1 --2- 172.21.15.123:0/1121106412 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff300078d50 0x7ff300077260 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff2f00098e0 tx=0x7ff2f0009250 comp rx=0 tx=0).stop 2024-03-30T04:17:27.648 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:26.761+0000 7ff305e00700 1 -- 172.21.15.123:0/1121106412 shutdown_connections 2024-03-30T04:17:27.648 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:26.761+0000 7ff305e00700 1 --2- 172.21.15.123:0/1121106412 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff300078d50 0x7ff300077260 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:27.648 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:26.761+0000 7ff305e00700 1 -- 172.21.15.123:0/1121106412 >> 172.21.15.123:0/1121106412 conn(0x7ff3000fdd00 msgr2=0x7ff300100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:27.648 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:26.761+0000 7ff305e00700 1 -- 172.21.15.123:0/1121106412 shutdown_connections 2024-03-30T04:17:27.648 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:26.761+0000 7ff305e00700 1 -- 172.21.15.123:0/1121106412 wait complete. 2024-03-30T04:17:27.648 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:26.761+0000 7ff305e00700 1 Processor -- start 2024-03-30T04:17:27.648 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:26.762+0000 7ff305e00700 1 -- start start 2024-03-30T04:17:27.648 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:26.762+0000 7ff305e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff300078d50 0x7ff3001135a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:27.648 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:26.762+0000 7ff305e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff300113ae0 con 0x7ff300078d50 2024-03-30T04:17:27.648 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:26.762+0000 7ff2fea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff300078d50 0x7ff3001135a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:27.648 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:26.762+0000 7ff2fea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff300078d50 0x7ff3001135a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46640/0 (socket says 172.21.15.123:46640) 2024-03-30T04:17:27.648 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:26.762+0000 7ff2fea00700 1 -- 172.21.15.123:0/2883037869 learned_addr learned my addr 172.21.15.123:0/2883037869 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:17:27.649 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:26.762+0000 7ff2fea00700 1 -- 172.21.15.123:0/2883037869 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f00094f0 con 0x7ff300078d50 2024-03-30T04:17:27.649 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:26.763+0000 7ff2fea00700 1 --2- 172.21.15.123:0/2883037869 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff300078d50 0x7ff3001135a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff2f0006d40 tx=0x7ff2f0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:27.649 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:26.763+0000 7ff2f6a00700 1 -- 172.21.15.123:0/2883037869 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2f0010960 con 0x7ff300078d50 2024-03-30T04:17:27.649 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:26.763+0000 7ff2f6a00700 1 -- 172.21.15.123:0/2883037869 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff2f001a430 con 0x7ff300078d50 2024-03-30T04:17:27.649 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:26.763+0000 7ff305e00700 1 -- 172.21.15.123:0/2883037869 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff300113ce0 con 0x7ff300078d50 2024-03-30T04:17:27.649 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:26.763+0000 7ff2f6a00700 1 -- 172.21.15.123:0/2883037869 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2f00245b0 con 0x7ff300078d50 2024-03-30T04:17:27.649 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:26.763+0000 7ff305e00700 1 -- 172.21.15.123:0/2883037869 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff300114850 con 0x7ff300078d50 2024-03-30T04:17:27.649 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:26.764+0000 7ff2f6a00700 1 -- 172.21.15.123:0/2883037869 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7ff2f001a9f0 con 0x7ff300078d50 2024-03-30T04:17:27.649 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:26.764+0000 7ff305e00700 1 -- 172.21.15.123:0/2883037869 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff30005f5f0 con 0x7ff300078d50 2024-03-30T04:17:27.650 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:26.764+0000 7ff2f6a00700 1 --2- 172.21.15.123:0/2883037869 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7ff2e4036790 0x7ff2e4038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:27.650 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:26.764+0000 7ff2f6a00700 1 -- 172.21.15.123:0/2883037869 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7ff2f004a3b0 con 0x7ff300078d50 2024-03-30T04:17:27.650 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:26.765+0000 7ff2fe000700 1 --2- 172.21.15.123:0/2883037869 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7ff2e4036790 0x7ff2e4038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:27.650 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:26.765+0000 7ff2fe000700 1 --2- 172.21.15.123:0/2883037869 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7ff2e4036790 0x7ff2e4038c40 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec009990 tx=0x7ff2ec006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:27.650 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:26.769+0000 7ff2f6a00700 1 -- 172.21.15.123:0/2883037869 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7ff2f002b8e0 con 0x7ff300078d50 2024-03-30T04:17:27.650 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:26.975+0000 7ff305e00700 1 -- 172.21.15.123:0/2883037869 --> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7ff3000fef50 con 0x7ff2e4036790 2024-03-30T04:17:27.650 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:26.979+0000 7ff2f6a00700 1 -- 172.21.15.123:0/2883037869 <== mgr.14118 v2:172.21.15.123:6800/1776532690 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7ff3000fef50 con 0x7ff2e4036790 2024-03-30T04:17:27.650 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:26.982+0000 7ff305e00700 1 -- 172.21.15.123:0/2883037869 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7ff2e4036790 msgr2=0x7ff2e4038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:27.650 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:26.982+0000 7ff305e00700 1 --2- 172.21.15.123:0/2883037869 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7ff2e4036790 0x7ff2e4038c40 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec009990 tx=0x7ff2ec006e40 comp rx=0 tx=0).stop 2024-03-30T04:17:27.650 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:26.982+0000 7ff305e00700 1 -- 172.21.15.123:0/2883037869 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff300078d50 msgr2=0x7ff3001135a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:27.651 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:26.982+0000 7ff305e00700 1 --2- 172.21.15.123:0/2883037869 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff300078d50 0x7ff3001135a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff2f0006d40 tx=0x7ff2f0009250 comp rx=0 tx=0).stop 2024-03-30T04:17:27.651 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:26.982+0000 7ff305e00700 1 -- 172.21.15.123:0/2883037869 shutdown_connections 2024-03-30T04:17:27.651 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:26.982+0000 7ff305e00700 1 --2- 172.21.15.123:0/2883037869 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7ff2e4036790 0x7ff2e4038c40 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:27.651 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:26.982+0000 7ff305e00700 1 --2- 172.21.15.123:0/2883037869 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff300078d50 0x7ff3001135a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:27.651 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:26.982+0000 7ff305e00700 1 -- 172.21.15.123:0/2883037869 >> 172.21.15.123:0/2883037869 conn(0x7ff3000fdd00 msgr2=0x7ff3000fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:27.651 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:26.982+0000 7ff305e00700 1 -- 172.21.15.123:0/2883037869 shutdown_connections 2024-03-30T04:17:27.651 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:26.982+0000 7ff305e00700 1 -- 172.21.15.123:0/2883037869 wait complete. 2024-03-30T04:17:27.651 INFO:teuthology.orchestra.run.smithi123.stdout:Deploying alertmanager service with default placement... 2024-03-30T04:17:28.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:27 smithi123 ceph-mon[35245]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:17:28.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:27 smithi123 ceph-mon[35245]: Saving service node-exporter spec with placement * 2024-03-30T04:17:28.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:27 smithi123 ceph-mon[35245]: from='mgr.14118 172.21.15.123:0/1102803547' entity='mgr.smithi123.xgksqq' 2024-03-30T04:17:28.976 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-03-30T04:17:28.977 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:28.179+0000 7f664e000700 1 Processor -- start 2024-03-30T04:17:28.977 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:28.179+0000 7f664e000700 1 -- start start 2024-03-30T04:17:28.977 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:28.180+0000 7f664e000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6648078d50 0x7f6648077260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:28.977 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:28.180+0000 7f664e000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6648079160 con 0x7f6648078d50 2024-03-30T04:17:28.977 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:28.180+0000 7f6647400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6648078d50 0x7f6648077260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:28.977 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:28.180+0000 7f6647400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6648078d50 0x7f6648077260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46654/0 (socket says 172.21.15.123:46654) 2024-03-30T04:17:28.977 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:28.180+0000 7f6647400700 1 -- 172.21.15.123:0/175847862 learned_addr learned my addr 172.21.15.123:0/175847862 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:17:28.977 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:28.180+0000 7f6647400700 1 -- 172.21.15.123:0/175847862 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66480777a0 con 0x7f6648078d50 2024-03-30T04:17:28.977 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:28.181+0000 7f6647400700 1 --2- 172.21.15.123:0/175847862 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6648078d50 0x7f6648077260 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f66380098e0 tx=0x7f6638009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bde91dd52fcdd8b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:28.977 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:28.181+0000 7f6646000700 1 -- 172.21.15.123:0/175847862 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f663800b860 con 0x7f6648078d50 2024-03-30T04:17:28.977 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:28.181+0000 7f6646000700 1 -- 172.21.15.123:0/175847862 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f663800be70 con 0x7f6648078d50 2024-03-30T04:17:28.977 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:28.181+0000 7f6646000700 1 -- 172.21.15.123:0/175847862 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6638010380 con 0x7f6648078d50 2024-03-30T04:17:28.977 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:28.181+0000 7f664e000700 1 -- 172.21.15.123:0/175847862 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6648078d50 msgr2=0x7f6648077260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:28.978 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:28.181+0000 7f664e000700 1 --2- 172.21.15.123:0/175847862 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6648078d50 0x7f6648077260 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f66380098e0 tx=0x7f6638009250 comp rx=0 tx=0).stop 2024-03-30T04:17:28.978 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:28.182+0000 7f664e000700 1 -- 172.21.15.123:0/175847862 shutdown_connections 2024-03-30T04:17:28.978 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:28.182+0000 7f664e000700 1 --2- 172.21.15.123:0/175847862 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6648078d50 0x7f6648077260 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:28.978 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:28.182+0000 7f664e000700 1 -- 172.21.15.123:0/175847862 >> 172.21.15.123:0/175847862 conn(0x7f66480fdd00 msgr2=0x7f6648100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:28.978 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:28.182+0000 7f664e000700 1 -- 172.21.15.123:0/175847862 shutdown_connections 2024-03-30T04:17:28.978 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:28.182+0000 7f664e000700 1 -- 172.21.15.123:0/175847862 wait complete. 2024-03-30T04:17:28.978 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:28.182+0000 7f664e000700 1 Processor -- start 2024-03-30T04:17:28.978 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:28.183+0000 7f664e000700 1 -- start start 2024-03-30T04:17:28.978 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:28.183+0000 7f664e000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6648078d50 0x7f66481135a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:28.978 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:28.183+0000 7f664e000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6648113ae0 con 0x7f6648078d50 2024-03-30T04:17:28.978 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:28.183+0000 7f6647400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6648078d50 0x7f66481135a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:28.978 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:28.183+0000 7f6647400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6648078d50 0x7f66481135a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46668/0 (socket says 172.21.15.123:46668) 2024-03-30T04:17:28.978 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:28.183+0000 7f6647400700 1 -- 172.21.15.123:0/1386825899 learned_addr learned my addr 172.21.15.123:0/1386825899 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:17:28.979 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:28.183+0000 7f6647400700 1 -- 172.21.15.123:0/1386825899 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66380094f0 con 0x7f6648078d50 2024-03-30T04:17:28.979 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:28.184+0000 7f6647400700 1 --2- 172.21.15.123:0/1386825899 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6648078d50 0x7f66481135a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6638006d40 tx=0x7f6638009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:28.979 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:28.184+0000 7f663f400700 1 -- 172.21.15.123:0/1386825899 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6638010960 con 0x7f6648078d50 2024-03-30T04:17:28.979 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:28.184+0000 7f663f400700 1 -- 172.21.15.123:0/1386825899 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f663801a430 con 0x7f6648078d50 2024-03-30T04:17:28.979 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:28.184+0000 7f664e000700 1 -- 172.21.15.123:0/1386825899 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6648113ce0 con 0x7f6648078d50 2024-03-30T04:17:28.979 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:28.184+0000 7f663f400700 1 -- 172.21.15.123:0/1386825899 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66380245b0 con 0x7f6648078d50 2024-03-30T04:17:28.979 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:28.184+0000 7f664e000700 1 -- 172.21.15.123:0/1386825899 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6648114850 con 0x7f6648078d50 2024-03-30T04:17:28.979 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:28.185+0000 7f663f400700 1 -- 172.21.15.123:0/1386825899 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f663801a9f0 con 0x7f6648078d50 2024-03-30T04:17:28.979 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:28.185+0000 7f664e000700 1 -- 172.21.15.123:0/1386825899 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f664805f5f0 con 0x7f6648078d50 2024-03-30T04:17:28.979 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:28.185+0000 7f663f400700 1 --2- 172.21.15.123:0/1386825899 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f662c0367e0 0x7f662c038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:28.979 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:28.186+0000 7f663f400700 1 -- 172.21.15.123:0/1386825899 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f663804a220 con 0x7f6648078d50 2024-03-30T04:17:28.979 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:28.186+0000 7f6646a00700 1 --2- 172.21.15.123:0/1386825899 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f662c0367e0 0x7f662c038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:28.979 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:28.186+0000 7f6646a00700 1 --2- 172.21.15.123:0/1386825899 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f662c0367e0 0x7f662c038c90 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6634009990 tx=0x7f6634006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:28.979 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:28.191+0000 7f663f400700 1 -- 172.21.15.123:0/1386825899 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f663802b8c0 con 0x7f6648078d50 2024-03-30T04:17:28.980 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:28.395+0000 7f664e000700 1 -- 172.21.15.123:0/1386825899 --> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f66480fef50 con 0x7f662c0367e0 2024-03-30T04:17:28.980 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:28.400+0000 7f663f400700 1 -- 172.21.15.123:0/1386825899 <== mgr.14118 v2:172.21.15.123:6800/1776532690 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f66480fef50 con 0x7f662c0367e0 2024-03-30T04:17:28.980 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:28.402+0000 7f664e000700 1 -- 172.21.15.123:0/1386825899 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f662c0367e0 msgr2=0x7f662c038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:28.980 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:28.402+0000 7f664e000700 1 --2- 172.21.15.123:0/1386825899 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f662c0367e0 0x7f662c038c90 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6634009990 tx=0x7f6634006e40 comp rx=0 tx=0).stop 2024-03-30T04:17:28.980 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:28.402+0000 7f664e000700 1 -- 172.21.15.123:0/1386825899 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6648078d50 msgr2=0x7f66481135a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:28.980 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:28.402+0000 7f664e000700 1 --2- 172.21.15.123:0/1386825899 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6648078d50 0x7f66481135a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6638006d40 tx=0x7f6638009250 comp rx=0 tx=0).stop 2024-03-30T04:17:28.980 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:28.402+0000 7f664e000700 1 -- 172.21.15.123:0/1386825899 shutdown_connections 2024-03-30T04:17:28.980 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:28.402+0000 7f664e000700 1 --2- 172.21.15.123:0/1386825899 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f662c0367e0 0x7f662c038c90 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:28.980 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:28.402+0000 7f664e000700 1 --2- 172.21.15.123:0/1386825899 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6648078d50 0x7f66481135a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:28.980 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:28.402+0000 7f664e000700 1 -- 172.21.15.123:0/1386825899 >> 172.21.15.123:0/1386825899 conn(0x7f66480fdd00 msgr2=0x7f66480fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:28.980 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:28.403+0000 7f664e000700 1 -- 172.21.15.123:0/1386825899 shutdown_connections 2024-03-30T04:17:28.980 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:28.403+0000 7f664e000700 1 -- 172.21.15.123:0/1386825899 wait complete. 2024-03-30T04:17:29.234 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:28 smithi123 ceph-mon[35245]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:17:29.234 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:28 smithi123 ceph-mon[35245]: from='mgr.14118 172.21.15.123:0/1102803547' entity='mgr.smithi123.xgksqq' 2024-03-30T04:17:30.063 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:29 smithi123 ceph-mon[35245]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:17:30.063 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:29 smithi123 ceph-mon[35245]: Saving service alertmanager spec with placement count:1 2024-03-30T04:17:30.063 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:29 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/4233262781' entity='client.admin' 2024-03-30T04:17:30.384 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:29.529+0000 7f38e6200700 1 Processor -- start 2024-03-30T04:17:30.385 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:29.529+0000 7f38e6200700 1 -- start start 2024-03-30T04:17:30.385 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:29.529+0000 7f38e6200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38e0078d50 0x7f38e0077260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:30.385 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:29.529+0000 7f38e6200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38e0079160 con 0x7f38e0078d50 2024-03-30T04:17:30.385 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:29.530+0000 7f38df400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38e0078d50 0x7f38e0077260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:30.385 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:29.530+0000 7f38df400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38e0078d50 0x7f38e0077260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46682/0 (socket says 172.21.15.123:46682) 2024-03-30T04:17:30.385 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:29.530+0000 7f38df400700 1 -- 172.21.15.123:0/1937750887 learned_addr learned my addr 172.21.15.123:0/1937750887 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:17:30.385 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:29.530+0000 7f38df400700 1 -- 172.21.15.123:0/1937750887 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38e00777a0 con 0x7f38e0078d50 2024-03-30T04:17:30.385 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:29.530+0000 7f38df400700 1 --2- 172.21.15.123:0/1937750887 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38e0078d50 0x7f38e0077260 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f38d00098e0 tx=0x7f38d0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c13d8cd8dae0d6c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:30.385 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:29.531+0000 7f38de000700 1 -- 172.21.15.123:0/1937750887 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38d000b860 con 0x7f38e0078d50 2024-03-30T04:17:30.386 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:29.531+0000 7f38de000700 1 -- 172.21.15.123:0/1937750887 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f38d000be70 con 0x7f38e0078d50 2024-03-30T04:17:30.386 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:29.531+0000 7f38de000700 1 -- 172.21.15.123:0/1937750887 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38d0010380 con 0x7f38e0078d50 2024-03-30T04:17:30.386 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:29.531+0000 7f38e6200700 1 -- 172.21.15.123:0/1937750887 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38e0078d50 msgr2=0x7f38e0077260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:30.386 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:29.531+0000 7f38e6200700 1 --2- 172.21.15.123:0/1937750887 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38e0078d50 0x7f38e0077260 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f38d00098e0 tx=0x7f38d0009250 comp rx=0 tx=0).stop 2024-03-30T04:17:30.386 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:29.531+0000 7f38e6200700 1 -- 172.21.15.123:0/1937750887 shutdown_connections 2024-03-30T04:17:30.386 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:29.531+0000 7f38e6200700 1 --2- 172.21.15.123:0/1937750887 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38e0078d50 0x7f38e0077260 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:30.386 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:29.531+0000 7f38e6200700 1 -- 172.21.15.123:0/1937750887 >> 172.21.15.123:0/1937750887 conn(0x7f38e00fdd00 msgr2=0x7f38e0100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:30.386 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:29.531+0000 7f38e6200700 1 -- 172.21.15.123:0/1937750887 shutdown_connections 2024-03-30T04:17:30.387 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:29.531+0000 7f38e6200700 1 -- 172.21.15.123:0/1937750887 wait complete. 2024-03-30T04:17:30.387 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:29.532+0000 7f38e6200700 1 Processor -- start 2024-03-30T04:17:30.387 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:29.532+0000 7f38e6200700 1 -- start start 2024-03-30T04:17:30.387 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:29.533+0000 7f38e6200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38e0078d50 0x7f38e01135b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:30.387 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:29.533+0000 7f38e6200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38e0113af0 con 0x7f38e0078d50 2024-03-30T04:17:30.387 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:29.533+0000 7f38df400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38e0078d50 0x7f38e01135b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:30.387 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:29.533+0000 7f38df400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38e0078d50 0x7f38e01135b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46686/0 (socket says 172.21.15.123:46686) 2024-03-30T04:17:30.387 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:29.533+0000 7f38df400700 1 -- 172.21.15.123:0/4233262781 learned_addr learned my addr 172.21.15.123:0/4233262781 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:17:30.387 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:29.533+0000 7f38df400700 1 -- 172.21.15.123:0/4233262781 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38d00094f0 con 0x7f38e0078d50 2024-03-30T04:17:30.387 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:29.533+0000 7f38df400700 1 --2- 172.21.15.123:0/4233262781 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38e0078d50 0x7f38e01135b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f38d0006d40 tx=0x7f38d0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:30.388 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:29.534+0000 7f38d7400700 1 -- 172.21.15.123:0/4233262781 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38d0010960 con 0x7f38e0078d50 2024-03-30T04:17:30.388 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:29.534+0000 7f38d7400700 1 -- 172.21.15.123:0/4233262781 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f38d001a430 con 0x7f38e0078d50 2024-03-30T04:17:30.388 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:29.534+0000 7f38d7400700 1 -- 172.21.15.123:0/4233262781 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38d00245b0 con 0x7f38e0078d50 2024-03-30T04:17:30.388 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:29.534+0000 7f38e6200700 1 -- 172.21.15.123:0/4233262781 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38e0113cf0 con 0x7f38e0078d50 2024-03-30T04:17:30.388 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:29.534+0000 7f38e6200700 1 -- 172.21.15.123:0/4233262781 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38e0114860 con 0x7f38e0078d50 2024-03-30T04:17:30.388 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:29.535+0000 7f38e6200700 1 -- 172.21.15.123:0/4233262781 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38e005f5f0 con 0x7f38e0078d50 2024-03-30T04:17:30.388 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:29.535+0000 7f38d7400700 1 -- 172.21.15.123:0/4233262781 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f38d001a9f0 con 0x7f38e0078d50 2024-03-30T04:17:30.388 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:29.535+0000 7f38d7400700 1 --2- 172.21.15.123:0/4233262781 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f38c40367e0 0x7f38c4038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:30.389 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:29.535+0000 7f38d7400700 1 -- 172.21.15.123:0/4233262781 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f38d004a310 con 0x7f38e0078d50 2024-03-30T04:17:30.389 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:29.535+0000 7f38dea00700 1 --2- 172.21.15.123:0/4233262781 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f38c40367e0 0x7f38c4038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:30.389 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:29.536+0000 7f38dea00700 1 --2- 172.21.15.123:0/4233262781 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f38c40367e0 0x7f38c4038c90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f38cc009990 tx=0x7f38cc006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:30.389 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:29.540+0000 7f38d7400700 1 -- 172.21.15.123:0/4233262781 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f38d002b8c0 con 0x7f38e0078d50 2024-03-30T04:17:30.389 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:29.731+0000 7f38e6200700 1 -- 172.21.15.123:0/4233262781 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f38e004bed0 con 0x7f38e0078d50 2024-03-30T04:17:30.389 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:29.734+0000 7f38d7400700 1 -- 172.21.15.123:0/4233262781 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f38d002b2f0 con 0x7f38e0078d50 2024-03-30T04:17:30.389 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:29.737+0000 7f38e6200700 1 -- 172.21.15.123:0/4233262781 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f38c40367e0 msgr2=0x7f38c4038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:30.389 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:29.737+0000 7f38e6200700 1 --2- 172.21.15.123:0/4233262781 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f38c40367e0 0x7f38c4038c90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f38cc009990 tx=0x7f38cc006e40 comp rx=0 tx=0).stop 2024-03-30T04:17:30.390 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:29.737+0000 7f38e6200700 1 -- 172.21.15.123:0/4233262781 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38e0078d50 msgr2=0x7f38e01135b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:30.390 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:29.737+0000 7f38e6200700 1 --2- 172.21.15.123:0/4233262781 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38e0078d50 0x7f38e01135b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f38d0006d40 tx=0x7f38d0009250 comp rx=0 tx=0).stop 2024-03-30T04:17:30.390 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:29.737+0000 7f38e6200700 1 -- 172.21.15.123:0/4233262781 shutdown_connections 2024-03-30T04:17:30.390 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:29.737+0000 7f38e6200700 1 --2- 172.21.15.123:0/4233262781 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f38c40367e0 0x7f38c4038c90 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:30.390 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:29.737+0000 7f38e6200700 1 --2- 172.21.15.123:0/4233262781 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38e0078d50 0x7f38e01135b0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:30.390 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:29.737+0000 7f38e6200700 1 -- 172.21.15.123:0/4233262781 >> 172.21.15.123:0/4233262781 conn(0x7f38e00fdd00 msgr2=0x7f38e00fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:30.390 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:29.737+0000 7f38e6200700 1 -- 172.21.15.123:0/4233262781 shutdown_connections 2024-03-30T04:17:30.390 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:29.737+0000 7f38e6200700 1 -- 172.21.15.123:0/4233262781 wait complete. 2024-03-30T04:17:31.461 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:30 smithi123 ceph-mon[35245]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:17:31.737 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:30.960+0000 7fb823800700 1 Processor -- start 2024-03-30T04:17:31.737 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:30.960+0000 7fb823800700 1 -- start start 2024-03-30T04:17:31.737 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:30.961+0000 7fb823800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb81c102530 0x7fb81c102940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:31.737 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:30.961+0000 7fb823800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb81c102e80 con 0x7fb81c102530 2024-03-30T04:17:31.737 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:30.961+0000 7fb820e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb81c102530 0x7fb81c102940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:31.738 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:30.961+0000 7fb820e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb81c102530 0x7fb81c102940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46702/0 (socket says 172.21.15.123:46702) 2024-03-30T04:17:31.738 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:30.961+0000 7fb820e00700 1 -- 172.21.15.123:0/72388852 learned_addr learned my addr 172.21.15.123:0/72388852 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:17:31.738 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:30.961+0000 7fb820e00700 1 -- 172.21.15.123:0/72388852 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb81c102fc0 con 0x7fb81c102530 2024-03-30T04:17:31.738 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:30.962+0000 7fb820e00700 1 --2- 172.21.15.123:0/72388852 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb81c102530 0x7fb81c102940 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb8100098e0 tx=0x7fb810009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b324cbfd1d083314 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:31.738 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:30.962+0000 7fb81b400700 1 -- 172.21.15.123:0/72388852 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb81000b860 con 0x7fb81c102530 2024-03-30T04:17:31.738 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:30.962+0000 7fb81b400700 1 -- 172.21.15.123:0/72388852 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb81000be70 con 0x7fb81c102530 2024-03-30T04:17:31.738 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:30.962+0000 7fb81b400700 1 -- 172.21.15.123:0/72388852 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb810010380 con 0x7fb81c102530 2024-03-30T04:17:31.738 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:30.962+0000 7fb823800700 1 -- 172.21.15.123:0/72388852 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb81c102530 msgr2=0x7fb81c102940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:31.738 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:30.962+0000 7fb823800700 1 --2- 172.21.15.123:0/72388852 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb81c102530 0x7fb81c102940 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb8100098e0 tx=0x7fb810009250 comp rx=0 tx=0).stop 2024-03-30T04:17:31.738 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:30.963+0000 7fb823800700 1 -- 172.21.15.123:0/72388852 shutdown_connections 2024-03-30T04:17:31.738 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:30.963+0000 7fb823800700 1 --2- 172.21.15.123:0/72388852 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb81c102530 0x7fb81c102940 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:31.738 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:30.963+0000 7fb823800700 1 -- 172.21.15.123:0/72388852 >> 172.21.15.123:0/72388852 conn(0x7fb81c0fdcd0 msgr2=0x7fb81c100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:31.739 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:30.963+0000 7fb823800700 1 -- 172.21.15.123:0/72388852 shutdown_connections 2024-03-30T04:17:31.739 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:30.963+0000 7fb823800700 1 -- 172.21.15.123:0/72388852 wait complete. 2024-03-30T04:17:31.739 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:30.963+0000 7fb823800700 1 Processor -- start 2024-03-30T04:17:31.739 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:30.964+0000 7fb823800700 1 -- start start 2024-03-30T04:17:31.739 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:30.964+0000 7fb823800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb81c102530 0x7fb81c113670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:31.739 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:30.964+0000 7fb823800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb81c113bb0 con 0x7fb81c102530 2024-03-30T04:17:31.739 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:30.964+0000 7fb820e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb81c102530 0x7fb81c113670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:31.739 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:30.964+0000 7fb820e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb81c102530 0x7fb81c113670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46714/0 (socket says 172.21.15.123:46714) 2024-03-30T04:17:31.739 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:30.964+0000 7fb820e00700 1 -- 172.21.15.123:0/703468264 learned_addr learned my addr 172.21.15.123:0/703468264 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:17:31.739 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:30.964+0000 7fb820e00700 1 -- 172.21.15.123:0/703468264 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8100094f0 con 0x7fb81c102530 2024-03-30T04:17:31.739 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:30.965+0000 7fb820e00700 1 --2- 172.21.15.123:0/703468264 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb81c102530 0x7fb81c113670 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb810000c00 tx=0x7fb810009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:31.739 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:30.965+0000 7fb818c00700 1 -- 172.21.15.123:0/703468264 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8100105f0 con 0x7fb81c102530 2024-03-30T04:17:31.739 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:30.965+0000 7fb818c00700 1 -- 172.21.15.123:0/703468264 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb810010c00 con 0x7fb81c102530 2024-03-30T04:17:31.739 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:30.965+0000 7fb818c00700 1 -- 172.21.15.123:0/703468264 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb81001ac70 con 0x7fb81c102530 2024-03-30T04:17:31.739 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:30.965+0000 7fb823800700 1 -- 172.21.15.123:0/703468264 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb81c113db0 con 0x7fb81c102530 2024-03-30T04:17:31.740 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:30.965+0000 7fb823800700 1 -- 172.21.15.123:0/703468264 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb81c114920 con 0x7fb81c102530 2024-03-30T04:17:31.740 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:30.966+0000 7fb818c00700 1 -- 172.21.15.123:0/703468264 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fb810010d70 con 0x7fb81c102530 2024-03-30T04:17:31.740 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:30.966+0000 7fb823800700 1 -- 172.21.15.123:0/703468264 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb81c05f5f0 con 0x7fb81c102530 2024-03-30T04:17:31.740 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:30.966+0000 7fb818c00700 1 --2- 172.21.15.123:0/703468264 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7fb8040367e0 0x7fb804038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:31.740 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:30.966+0000 7fb818c00700 1 -- 172.21.15.123:0/703468264 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fb81004a310 con 0x7fb81c102530 2024-03-30T04:17:31.740 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:30.967+0000 7fb81be00700 1 --2- 172.21.15.123:0/703468264 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7fb8040367e0 0x7fb804038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:31.740 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:30.967+0000 7fb81be00700 1 --2- 172.21.15.123:0/703468264 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7fb8040367e0 0x7fb804038c90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb808009990 tx=0x7fb808006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:31.740 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:30.971+0000 7fb818c00700 1 -- 172.21.15.123:0/703468264 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fb810018350 con 0x7fb81c102530 2024-03-30T04:17:31.740 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:31.168+0000 7fb823800700 1 -- 172.21.15.123:0/703468264 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fb81c04bed0 con 0x7fb81c102530 2024-03-30T04:17:31.740 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:31.171+0000 7fb818c00700 1 -- 172.21.15.123:0/703468264 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7fb810020070 con 0x7fb81c102530 2024-03-30T04:17:31.740 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:31.174+0000 7fb823800700 1 -- 172.21.15.123:0/703468264 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7fb8040367e0 msgr2=0x7fb804038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:31.740 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:31.174+0000 7fb823800700 1 --2- 172.21.15.123:0/703468264 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7fb8040367e0 0x7fb804038c90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb808009990 tx=0x7fb808006e40 comp rx=0 tx=0).stop 2024-03-30T04:17:31.740 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:31.174+0000 7fb823800700 1 -- 172.21.15.123:0/703468264 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb81c102530 msgr2=0x7fb81c113670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:31.740 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:31.174+0000 7fb823800700 1 --2- 172.21.15.123:0/703468264 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb81c102530 0x7fb81c113670 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb810000c00 tx=0x7fb810009250 comp rx=0 tx=0).stop 2024-03-30T04:17:31.741 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:31.174+0000 7fb823800700 1 -- 172.21.15.123:0/703468264 shutdown_connections 2024-03-30T04:17:31.741 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:31.174+0000 7fb823800700 1 --2- 172.21.15.123:0/703468264 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7fb8040367e0 0x7fb804038c90 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:31.741 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:31.174+0000 7fb823800700 1 --2- 172.21.15.123:0/703468264 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb81c102530 0x7fb81c113670 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:31.741 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:31.174+0000 7fb823800700 1 -- 172.21.15.123:0/703468264 >> 172.21.15.123:0/703468264 conn(0x7fb81c0fdcd0 msgr2=0x7fb81c0fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:31.741 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:31.174+0000 7fb823800700 1 -- 172.21.15.123:0/703468264 shutdown_connections 2024-03-30T04:17:31.741 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:31.174+0000 7fb823800700 1 -- 172.21.15.123:0/703468264 wait complete. 2024-03-30T04:17:31.741 INFO:teuthology.orchestra.run.smithi123.stdout:Enabling the dashboard module... 2024-03-30T04:17:32.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:32 smithi123 ceph-mon[35245]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:17:32.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:32 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/703468264' entity='client.admin' 2024-03-30T04:17:33.459 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:33 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/3856194455' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-30T04:17:33.858 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:32.332+0000 7f75dda00700 1 Processor -- start 2024-03-30T04:17:33.864 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:32.332+0000 7f75dda00700 1 -- start start 2024-03-30T04:17:33.864 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:32.332+0000 7f75dda00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75d8078d40 0x7f75d8077240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:33.864 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:32.332+0000 7f75dda00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75d8079150 con 0x7f75d8078d40 2024-03-30T04:17:33.864 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:32.332+0000 7f75d6a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75d8078d40 0x7f75d8077240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:33.864 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:32.332+0000 7f75d6a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75d8078d40 0x7f75d8077240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46722/0 (socket says 172.21.15.123:46722) 2024-03-30T04:17:33.864 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:32.332+0000 7f75d6a00700 1 -- 172.21.15.123:0/3752103328 learned_addr learned my addr 172.21.15.123:0/3752103328 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:17:33.864 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:32.333+0000 7f75d6a00700 1 -- 172.21.15.123:0/3752103328 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d8077780 con 0x7f75d8078d40 2024-03-30T04:17:33.864 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:32.333+0000 7f75d6a00700 1 --2- 172.21.15.123:0/3752103328 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75d8078d40 0x7f75d8077240 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f75c80098e0 tx=0x7f75c8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa1c3830bced23da server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:33.864 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:32.333+0000 7f75d5600700 1 -- 172.21.15.123:0/3752103328 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75c800b860 con 0x7f75d8078d40 2024-03-30T04:17:33.864 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:32.333+0000 7f75d5600700 1 -- 172.21.15.123:0/3752103328 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f75c800be70 con 0x7f75d8078d40 2024-03-30T04:17:33.864 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:32.333+0000 7f75d5600700 1 -- 172.21.15.123:0/3752103328 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75c8010380 con 0x7f75d8078d40 2024-03-30T04:17:33.865 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:32.334+0000 7f75dda00700 1 -- 172.21.15.123:0/3752103328 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75d8078d40 msgr2=0x7f75d8077240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:33.865 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:32.334+0000 7f75dda00700 1 --2- 172.21.15.123:0/3752103328 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75d8078d40 0x7f75d8077240 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f75c80098e0 tx=0x7f75c8009250 comp rx=0 tx=0).stop 2024-03-30T04:17:33.865 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:32.334+0000 7f75dda00700 1 -- 172.21.15.123:0/3752103328 shutdown_connections 2024-03-30T04:17:33.865 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:32.334+0000 7f75dda00700 1 --2- 172.21.15.123:0/3752103328 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75d8078d40 0x7f75d8077240 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:33.865 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:32.334+0000 7f75dda00700 1 -- 172.21.15.123:0/3752103328 >> 172.21.15.123:0/3752103328 conn(0x7f75d80fdcd0 msgr2=0x7f75d8100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:33.865 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:32.334+0000 7f75dda00700 1 -- 172.21.15.123:0/3752103328 shutdown_connections 2024-03-30T04:17:33.865 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:32.334+0000 7f75dda00700 1 -- 172.21.15.123:0/3752103328 wait complete. 2024-03-30T04:17:33.865 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:32.335+0000 7f75dda00700 1 Processor -- start 2024-03-30T04:17:33.865 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:32.335+0000 7f75dda00700 1 -- start start 2024-03-30T04:17:33.865 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:32.335+0000 7f75dda00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75d8078d40 0x7f75d8113600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:33.865 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:32.335+0000 7f75dda00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75d8113b40 con 0x7f75d8078d40 2024-03-30T04:17:33.865 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:32.335+0000 7f75d6a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75d8078d40 0x7f75d8113600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:33.865 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:32.336+0000 7f75d6a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75d8078d40 0x7f75d8113600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46738/0 (socket says 172.21.15.123:46738) 2024-03-30T04:17:33.865 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:32.336+0000 7f75d6a00700 1 -- 172.21.15.123:0/3856194455 learned_addr learned my addr 172.21.15.123:0/3856194455 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:17:33.866 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:32.336+0000 7f75d6a00700 1 -- 172.21.15.123:0/3856194455 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75c80094f0 con 0x7f75d8078d40 2024-03-30T04:17:33.866 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:32.336+0000 7f75d6a00700 1 --2- 172.21.15.123:0/3856194455 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75d8078d40 0x7f75d8113600 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f75c8011b60 tx=0x7f75c8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:33.866 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:32.336+0000 7f75cea00700 1 -- 172.21.15.123:0/3856194455 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75c8010800 con 0x7f75d8078d40 2024-03-30T04:17:33.866 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:32.336+0000 7f75cea00700 1 -- 172.21.15.123:0/3856194455 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f75c8010e10 con 0x7f75d8078d40 2024-03-30T04:17:33.866 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:32.336+0000 7f75dda00700 1 -- 172.21.15.123:0/3856194455 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75d8113d40 con 0x7f75d8078d40 2024-03-30T04:17:33.866 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:32.336+0000 7f75cea00700 1 -- 172.21.15.123:0/3856194455 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75c801ae60 con 0x7f75d8078d40 2024-03-30T04:17:33.866 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:32.336+0000 7f75dda00700 1 -- 172.21.15.123:0/3856194455 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75d81148b0 con 0x7f75d8078d40 2024-03-30T04:17:33.866 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:32.337+0000 7f75cea00700 1 -- 172.21.15.123:0/3856194455 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f75c801a430 con 0x7f75d8078d40 2024-03-30T04:17:33.866 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:32.337+0000 7f75dda00700 1 -- 172.21.15.123:0/3856194455 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75d805f5f0 con 0x7f75d8078d40 2024-03-30T04:17:33.866 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:32.338+0000 7f75cea00700 1 --2- 172.21.15.123:0/3856194455 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f75bc036770 0x7f75bc038c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:33.866 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:32.338+0000 7f75cea00700 1 -- 172.21.15.123:0/3856194455 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f75c804a710 con 0x7f75d8078d40 2024-03-30T04:17:33.866 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:32.338+0000 7f75d6000700 1 --2- 172.21.15.123:0/3856194455 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f75bc036770 0x7f75bc038c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:33.866 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:32.339+0000 7f75d6000700 1 --2- 172.21.15.123:0/3856194455 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f75bc036770 0x7f75bc038c20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f75c4009990 tx=0x7f75c4006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:33.867 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:32.343+0000 7f75cea00700 1 -- 172.21.15.123:0/3856194455 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f75c802b950 con 0x7f75d8078d40 2024-03-30T04:17:33.867 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:32.593+0000 7f75dda00700 1 -- 172.21.15.123:0/3856194455 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f75d804bed0 con 0x7f75d8078d40 2024-03-30T04:17:33.867 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:33.180+0000 7f75cea00700 1 -- 172.21.15.123:0/3856194455 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f75c8021d40 con 0x7f75d8078d40 2024-03-30T04:17:33.867 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:33.182+0000 7f75cea00700 1 -- 172.21.15.123:0/3856194455 <== mon.0 v2:172.21.15.123:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f75c804c550 con 0x7f75d8078d40 2024-03-30T04:17:33.867 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:33.185+0000 7f75dda00700 1 -- 172.21.15.123:0/3856194455 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f75bc036770 msgr2=0x7f75bc038c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:33.867 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:33.185+0000 7f75dda00700 1 --2- 172.21.15.123:0/3856194455 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f75bc036770 0x7f75bc038c20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f75c4009990 tx=0x7f75c4006e40 comp rx=0 tx=0).stop 2024-03-30T04:17:33.867 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:33.185+0000 7f75dda00700 1 -- 172.21.15.123:0/3856194455 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75d8078d40 msgr2=0x7f75d8113600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:33.867 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:33.185+0000 7f75dda00700 1 --2- 172.21.15.123:0/3856194455 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75d8078d40 0x7f75d8113600 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f75c8011b60 tx=0x7f75c8009250 comp rx=0 tx=0).stop 2024-03-30T04:17:33.867 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:33.185+0000 7f75dda00700 1 -- 172.21.15.123:0/3856194455 shutdown_connections 2024-03-30T04:17:33.867 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:33.185+0000 7f75dda00700 1 --2- 172.21.15.123:0/3856194455 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7f75bc036770 0x7f75bc038c20 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:33.867 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:33.185+0000 7f75dda00700 1 --2- 172.21.15.123:0/3856194455 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75d8078d40 0x7f75d8113600 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:33.867 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:33.185+0000 7f75dda00700 1 -- 172.21.15.123:0/3856194455 >> 172.21.15.123:0/3856194455 conn(0x7f75d80fdcd0 msgr2=0x7f75d80fe8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:33.867 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:33.185+0000 7f75dda00700 1 -- 172.21.15.123:0/3856194455 shutdown_connections 2024-03-30T04:17:33.867 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:33.186+0000 7f75dda00700 1 -- 172.21.15.123:0/3856194455 wait complete. 2024-03-30T04:17:34.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:34 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/3856194455' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-30T04:17:34.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:34 smithi123 ceph-mon[35245]: mgrmap e9: smithi123.xgksqq(active, since 28s) 2024-03-30T04:17:35.286 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout { 2024-03-30T04:17:35.286 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-30T04:17:35.287 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-30T04:17:35.287 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "active_name": "smithi123.xgksqq", 2024-03-30T04:17:35.287 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-30T04:17:35.287 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout } 2024-03-30T04:17:35.287 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.452+0000 7fae9e000700 1 Processor -- start 2024-03-30T04:17:35.287 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.452+0000 7fae9e000700 1 -- start start 2024-03-30T04:17:35.287 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.452+0000 7fae9e000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae98078d50 0x7fae98077260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:35.287 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.452+0000 7fae9e000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae98079160 con 0x7fae98078d50 2024-03-30T04:17:35.287 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.453+0000 7fae97400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae98078d50 0x7fae98077260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:35.287 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.453+0000 7fae97400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae98078d50 0x7fae98077260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57938/0 (socket says 172.21.15.123:57938) 2024-03-30T04:17:35.287 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.453+0000 7fae97400700 1 -- 172.21.15.123:0/657578118 learned_addr learned my addr 172.21.15.123:0/657578118 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:17:35.287 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.453+0000 7fae97400700 1 -- 172.21.15.123:0/657578118 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae980777a0 con 0x7fae98078d50 2024-03-30T04:17:35.287 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.453+0000 7fae97400700 1 --2- 172.21.15.123:0/657578118 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae98078d50 0x7fae98077260 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fae880098e0 tx=0x7fae88009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fe737274a0fecc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:35.288 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.454+0000 7fae96000700 1 -- 172.21.15.123:0/657578118 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae8800b860 con 0x7fae98078d50 2024-03-30T04:17:35.288 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.454+0000 7fae96000700 1 -- 172.21.15.123:0/657578118 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fae8800be70 con 0x7fae98078d50 2024-03-30T04:17:35.288 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.454+0000 7fae96000700 1 -- 172.21.15.123:0/657578118 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae88010380 con 0x7fae98078d50 2024-03-30T04:17:35.288 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.454+0000 7fae9e000700 1 -- 172.21.15.123:0/657578118 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae98078d50 msgr2=0x7fae98077260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:35.288 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.454+0000 7fae9e000700 1 --2- 172.21.15.123:0/657578118 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae98078d50 0x7fae98077260 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fae880098e0 tx=0x7fae88009250 comp rx=0 tx=0).stop 2024-03-30T04:17:35.288 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.454+0000 7fae9e000700 1 -- 172.21.15.123:0/657578118 shutdown_connections 2024-03-30T04:17:35.288 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.454+0000 7fae9e000700 1 --2- 172.21.15.123:0/657578118 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae98078d50 0x7fae98077260 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:35.288 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.454+0000 7fae9e000700 1 -- 172.21.15.123:0/657578118 >> 172.21.15.123:0/657578118 conn(0x7fae980fdd00 msgr2=0x7fae98100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:35.288 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.455+0000 7fae9e000700 1 -- 172.21.15.123:0/657578118 shutdown_connections 2024-03-30T04:17:35.288 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.455+0000 7fae9e000700 1 -- 172.21.15.123:0/657578118 wait complete. 2024-03-30T04:17:35.288 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.455+0000 7fae9e000700 1 Processor -- start 2024-03-30T04:17:35.288 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.456+0000 7fae9e000700 1 -- start start 2024-03-30T04:17:35.288 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.456+0000 7fae9e000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae98078d50 0x7fae981135a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:35.289 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.456+0000 7fae9e000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae98113ae0 con 0x7fae98078d50 2024-03-30T04:17:35.289 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.456+0000 7fae97400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae98078d50 0x7fae981135a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:35.289 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.456+0000 7fae97400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae98078d50 0x7fae981135a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57948/0 (socket says 172.21.15.123:57948) 2024-03-30T04:17:35.289 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.456+0000 7fae97400700 1 -- 172.21.15.123:0/4073614114 learned_addr learned my addr 172.21.15.123:0/4073614114 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:17:35.289 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.456+0000 7fae97400700 1 -- 172.21.15.123:0/4073614114 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae880094f0 con 0x7fae98078d50 2024-03-30T04:17:35.289 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.457+0000 7fae97400700 1 --2- 172.21.15.123:0/4073614114 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae98078d50 0x7fae981135a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fae88003080 tx=0x7fae88009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:35.289 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.457+0000 7fae8f400700 1 -- 172.21.15.123:0/4073614114 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae88010960 con 0x7fae98078d50 2024-03-30T04:17:35.289 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.457+0000 7fae8f400700 1 -- 172.21.15.123:0/4073614114 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fae8801a430 con 0x7fae98078d50 2024-03-30T04:17:35.289 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.457+0000 7fae8f400700 1 -- 172.21.15.123:0/4073614114 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae880245b0 con 0x7fae98078d50 2024-03-30T04:17:35.289 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.457+0000 7fae9e000700 1 -- 172.21.15.123:0/4073614114 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae98113ce0 con 0x7fae98078d50 2024-03-30T04:17:35.289 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.457+0000 7fae9e000700 1 -- 172.21.15.123:0/4073614114 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae98114850 con 0x7fae98078d50 2024-03-30T04:17:35.289 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.459+0000 7fae9e000700 1 -- 172.21.15.123:0/4073614114 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae9805f5f0 con 0x7fae98078d50 2024-03-30T04:17:35.289 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.459+0000 7fae8f400700 1 -- 172.21.15.123:0/4073614114 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7fae8801a9f0 con 0x7fae98078d50 2024-03-30T04:17:35.290 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.460+0000 7fae8f400700 1 --2- 172.21.15.123:0/4073614114 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7fae7c0367e0 0x7fae7c038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:35.290 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.460+0000 7fae8f400700 1 -- 172.21.15.123:0/4073614114 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fae8804a410 con 0x7fae98078d50 2024-03-30T04:17:35.290 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.460+0000 7fae96a00700 1 -- 172.21.15.123:0/4073614114 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7fae7c0367e0 msgr2=0x7fae7c038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.123:6800/1776532690 2024-03-30T04:17:35.290 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.460+0000 7fae96a00700 1 --2- 172.21.15.123:0/4073614114 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7fae7c0367e0 0x7fae7c038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-30T04:17:35.290 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.464+0000 7fae8f400700 1 -- 172.21.15.123:0/4073614114 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fae8802b8e0 con 0x7fae98078d50 2024-03-30T04:17:35.290 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.661+0000 7fae96a00700 1 -- 172.21.15.123:0/4073614114 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7fae7c0367e0 msgr2=0x7fae7c038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.123:6800/1776532690 2024-03-30T04:17:35.290 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.661+0000 7fae96a00700 1 --2- 172.21.15.123:0/4073614114 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7fae7c0367e0 0x7fae7c038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-30T04:17:35.290 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.746+0000 7fae9e000700 1 -- 172.21.15.123:0/4073614114 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fae98114b90 con 0x7fae98078d50 2024-03-30T04:17:35.290 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.747+0000 7fae8f400700 1 -- 172.21.15.123:0/4073614114 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7fae8802b2f0 con 0x7fae98078d50 2024-03-30T04:17:35.290 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.749+0000 7fae9e000700 1 -- 172.21.15.123:0/4073614114 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7fae7c0367e0 msgr2=0x7fae7c038c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-30T04:17:35.290 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.749+0000 7fae9e000700 1 --2- 172.21.15.123:0/4073614114 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7fae7c0367e0 0x7fae7c038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:35.290 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.749+0000 7fae9e000700 1 -- 172.21.15.123:0/4073614114 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae98078d50 msgr2=0x7fae981135a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:35.290 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.749+0000 7fae9e000700 1 --2- 172.21.15.123:0/4073614114 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae98078d50 0x7fae981135a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fae88003080 tx=0x7fae88009250 comp rx=0 tx=0).stop 2024-03-30T04:17:35.290 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.749+0000 7fae9e000700 1 -- 172.21.15.123:0/4073614114 shutdown_connections 2024-03-30T04:17:35.291 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.749+0000 7fae9e000700 1 --2- 172.21.15.123:0/4073614114 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7fae7c0367e0 0x7fae7c038c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:35.291 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.749+0000 7fae9e000700 1 --2- 172.21.15.123:0/4073614114 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae98078d50 0x7fae981135a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:35.291 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.749+0000 7fae9e000700 1 -- 172.21.15.123:0/4073614114 >> 172.21.15.123:0/4073614114 conn(0x7fae980fdd00 msgr2=0x7fae980fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:35.291 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.750+0000 7fae9e000700 1 -- 172.21.15.123:0/4073614114 shutdown_connections 2024-03-30T04:17:35.291 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:34.750+0000 7fae9e000700 1 -- 172.21.15.123:0/4073614114 wait complete. 2024-03-30T04:17:35.291 INFO:teuthology.orchestra.run.smithi123.stdout:Waiting for the mgr to restart... 2024-03-30T04:17:35.291 INFO:teuthology.orchestra.run.smithi123.stdout:Waiting for mgr epoch 9... 2024-03-30T04:17:35.291 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:35 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/4073614114' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-30T04:17:41.582 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:41 smithi123 ceph-mon[35245]: Active manager daemon smithi123.xgksqq restarted 2024-03-30T04:17:41.582 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:41 smithi123 ceph-mon[35245]: Activating manager daemon smithi123.xgksqq 2024-03-30T04:17:41.583 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:41 smithi123 ceph-mon[35245]: osdmap e3: 0 total, 0 up, 0 in 2024-03-30T04:17:41.583 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:41 smithi123 ceph-mon[35245]: mgrmap e10: smithi123.xgksqq(active, starting, since 0.00590925s) 2024-03-30T04:17:41.583 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:41 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mon metadata", "id": "smithi123"}]: dispatch 2024-03-30T04:17:41.583 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:41 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mgr metadata", "who": "smithi123.xgksqq", "id": "smithi123.xgksqq"}]: dispatch 2024-03-30T04:17:41.583 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:41 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-30T04:17:41.583 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:41 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-30T04:17:41.583 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:41 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-30T04:17:41.583 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:41 smithi123 ceph-mon[35245]: Manager daemon smithi123.xgksqq is now available 2024-03-30T04:17:41.583 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:41 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:17:41.583 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:41 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:17:42.939 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:42 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi123.xgksqq/mirror_snapshot_schedule"}]: dispatch 2024-03-30T04:17:42.939 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:42 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi123.xgksqq/trash_purge_schedule"}]: dispatch 2024-03-30T04:17:42.939 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:42 smithi123 ceph-mon[35245]: [30/Mar/2024:04:17:42] ENGINE Bus STARTING 2024-03-30T04:17:42.940 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:42 smithi123 ceph-mon[35245]: [30/Mar/2024:04:17:42] ENGINE Serving on https://172.21.15.123:7150 2024-03-30T04:17:42.940 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:42 smithi123 ceph-mon[35245]: [30/Mar/2024:04:17:42] ENGINE Bus STARTED 2024-03-30T04:17:42.940 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:42 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' 2024-03-30T04:17:42.940 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:42 smithi123 ceph-mon[35245]: mgrmap e11: smithi123.xgksqq(active, since 1.0074s) 2024-03-30T04:17:43.585 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout { 2024-03-30T04:17:43.585 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-30T04:17:43.585 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-30T04:17:43.585 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout } 2024-03-30T04:17:43.585 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:35.847+0000 7fa617400700 1 Processor -- start 2024-03-30T04:17:43.585 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:35.847+0000 7fa617400700 1 -- start start 2024-03-30T04:17:43.585 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:35.848+0000 7fa617400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa610102580 0x7fa610102990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:43.586 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:35.848+0000 7fa617400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa610102ed0 con 0x7fa610102580 2024-03-30T04:17:43.586 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:35.848+0000 7fa614a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa610102580 0x7fa610102990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:43.586 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:35.848+0000 7fa614a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa610102580 0x7fa610102990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57962/0 (socket says 172.21.15.123:57962) 2024-03-30T04:17:43.586 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:35.848+0000 7fa614a00700 1 -- 172.21.15.123:0/543525507 learned_addr learned my addr 172.21.15.123:0/543525507 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:17:43.586 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:35.848+0000 7fa614a00700 1 -- 172.21.15.123:0/543525507 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa610103010 con 0x7fa610102580 2024-03-30T04:17:43.586 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:35.848+0000 7fa614a00700 1 --2- 172.21.15.123:0/543525507 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa610102580 0x7fa610102990 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa6040098e0 tx=0x7fa604009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6c3982f8b6f743 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:43.586 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:35.849+0000 7fa60f400700 1 -- 172.21.15.123:0/543525507 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa60400b860 con 0x7fa610102580 2024-03-30T04:17:43.586 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:35.849+0000 7fa60f400700 1 -- 172.21.15.123:0/543525507 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa60400be70 con 0x7fa610102580 2024-03-30T04:17:43.586 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:35.849+0000 7fa60f400700 1 -- 172.21.15.123:0/543525507 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa604010380 con 0x7fa610102580 2024-03-30T04:17:43.586 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:35.849+0000 7fa617400700 1 -- 172.21.15.123:0/543525507 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa610102580 msgr2=0x7fa610102990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:43.587 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:35.849+0000 7fa617400700 1 --2- 172.21.15.123:0/543525507 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa610102580 0x7fa610102990 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa6040098e0 tx=0x7fa604009250 comp rx=0 tx=0).stop 2024-03-30T04:17:43.587 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:35.849+0000 7fa617400700 1 -- 172.21.15.123:0/543525507 shutdown_connections 2024-03-30T04:17:43.587 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:35.849+0000 7fa617400700 1 --2- 172.21.15.123:0/543525507 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa610102580 0x7fa610102990 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:43.587 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:35.849+0000 7fa617400700 1 -- 172.21.15.123:0/543525507 >> 172.21.15.123:0/543525507 conn(0x7fa6100fdd00 msgr2=0x7fa610100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:43.587 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:35.850+0000 7fa617400700 1 -- 172.21.15.123:0/543525507 shutdown_connections 2024-03-30T04:17:43.587 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:35.850+0000 7fa617400700 1 -- 172.21.15.123:0/543525507 wait complete. 2024-03-30T04:17:43.587 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:35.850+0000 7fa617400700 1 Processor -- start 2024-03-30T04:17:43.587 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:35.851+0000 7fa617400700 1 -- start start 2024-03-30T04:17:43.587 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:35.851+0000 7fa617400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa610102580 0x7fa610078c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:43.587 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:35.851+0000 7fa617400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa610079190 con 0x7fa610102580 2024-03-30T04:17:43.587 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:35.851+0000 7fa614a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa610102580 0x7fa610078c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:43.587 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:35.851+0000 7fa614a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa610102580 0x7fa610078c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57964/0 (socket says 172.21.15.123:57964) 2024-03-30T04:17:43.587 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:35.851+0000 7fa614a00700 1 -- 172.21.15.123:0/3325843429 learned_addr learned my addr 172.21.15.123:0/3325843429 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:17:43.587 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:35.851+0000 7fa614a00700 1 -- 172.21.15.123:0/3325843429 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6040094f0 con 0x7fa610102580 2024-03-30T04:17:43.588 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:35.852+0000 7fa614a00700 1 --2- 172.21.15.123:0/3325843429 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa610102580 0x7fa610078c50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa60400c010 tx=0x7fa604009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:43.588 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:35.852+0000 7fa60cc00700 1 -- 172.21.15.123:0/3325843429 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa604010960 con 0x7fa610102580 2024-03-30T04:17:43.588 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:35.852+0000 7fa617400700 1 -- 172.21.15.123:0/3325843429 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa610077200 con 0x7fa610102580 2024-03-30T04:17:43.588 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:35.852+0000 7fa60cc00700 1 -- 172.21.15.123:0/3325843429 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa60401a430 con 0x7fa610102580 2024-03-30T04:17:43.588 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:35.852+0000 7fa60cc00700 1 -- 172.21.15.123:0/3325843429 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa604019580 con 0x7fa610102580 2024-03-30T04:17:43.588 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:35.852+0000 7fa617400700 1 -- 172.21.15.123:0/3325843429 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6100776f0 con 0x7fa610102580 2024-03-30T04:17:43.588 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:35.853+0000 7fa60cc00700 1 -- 172.21.15.123:0/3325843429 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7fa60401a980 con 0x7fa610102580 2024-03-30T04:17:43.588 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:35.854+0000 7fa60cc00700 1 --2- 172.21.15.123:0/3325843429 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7fa5f80367e0 0x7fa5f8038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:43.588 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:35.854+0000 7fa60cc00700 1 -- 172.21.15.123:0/3325843429 --> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa5f80393a0 con 0x7fa5f80367e0 2024-03-30T04:17:43.588 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:35.854+0000 7fa60cc00700 1 -- 172.21.15.123:0/3325843429 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fa60404a650 con 0x7fa610102580 2024-03-30T04:17:43.589 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:35.854+0000 7fa60fe00700 1 -- 172.21.15.123:0/3325843429 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7fa5f80367e0 msgr2=0x7fa5f8038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.123:6800/1776532690 2024-03-30T04:17:43.589 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:35.854+0000 7fa60fe00700 1 --2- 172.21.15.123:0/3325843429 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7fa5f80367e0 0x7fa5f8038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-30T04:17:43.589 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:36.055+0000 7fa60fe00700 1 -- 172.21.15.123:0/3325843429 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7fa5f80367e0 msgr2=0x7fa5f8038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.123:6800/1776532690 2024-03-30T04:17:43.589 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:36.055+0000 7fa60fe00700 1 --2- 172.21.15.123:0/3325843429 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7fa5f80367e0 0x7fa5f8038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-30T04:17:43.589 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:36.456+0000 7fa60fe00700 1 -- 172.21.15.123:0/3325843429 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7fa5f80367e0 msgr2=0x7fa5f8038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.123:6800/1776532690 2024-03-30T04:17:43.589 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:36.456+0000 7fa60fe00700 1 --2- 172.21.15.123:0/3325843429 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7fa5f80367e0 0x7fa5f8038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-30T04:17:43.589 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:37.257+0000 7fa60fe00700 1 -- 172.21.15.123:0/3325843429 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7fa5f80367e0 msgr2=0x7fa5f8038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.123:6800/1776532690 2024-03-30T04:17:43.589 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:37.257+0000 7fa60fe00700 1 --2- 172.21.15.123:0/3325843429 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7fa5f80367e0 0x7fa5f8038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-30T04:17:43.589 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:38.859+0000 7fa60fe00700 1 -- 172.21.15.123:0/3325843429 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7fa5f80367e0 msgr2=0x7fa5f8038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.123:6800/1776532690 2024-03-30T04:17:43.589 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:38.859+0000 7fa60fe00700 1 --2- 172.21.15.123:0/3325843429 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7fa5f80367e0 0x7fa5f8038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-30T04:17:43.589 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:41.463+0000 7fa60cc00700 1 -- 172.21.15.123:0/3325843429 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mgrmap(e 10) v1 ==== 43148+0+0 (secure 0 0 0) 0x7fa604021d20 con 0x7fa610102580 2024-03-30T04:17:43.589 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:41.463+0000 7fa60cc00700 1 -- 172.21.15.123:0/3325843429 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7fa5f80367e0 msgr2=0x7fa5f8038c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-30T04:17:43.589 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:41.463+0000 7fa60cc00700 1 --2- 172.21.15.123:0/3325843429 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7fa5f80367e0 0x7fa5f8038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:43.590 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:42.463+0000 7fa60cc00700 1 -- 172.21.15.123:0/3325843429 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mgrmap(e 11) v1 ==== 43275+0+0 (secure 0 0 0) 0x7fa604018c50 con 0x7fa610102580 2024-03-30T04:17:43.590 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:42.463+0000 7fa60cc00700 1 --2- 172.21.15.123:0/3325843429 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7fa5f8039fd0 0x7fa5f803c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:43.590 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:42.463+0000 7fa60cc00700 1 -- 172.21.15.123:0/3325843429 --> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa5f80393a0 con 0x7fa5f8039fd0 2024-03-30T04:17:43.590 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:42.463+0000 7fa60fe00700 1 --2- 172.21.15.123:0/3325843429 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7fa5f8039fd0 0x7fa5f803c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:43.590 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:42.464+0000 7fa60fe00700 1 --2- 172.21.15.123:0/3325843429 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7fa5f8039fd0 0x7fa5f803c3b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc0039d0 tx=0x7fa5fc0092f0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:43.590 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:42.465+0000 7fa60cc00700 1 -- 172.21.15.123:0/3325843429 <== mgr.14160 v2:172.21.15.123:6800/70490328 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7fa5f80393a0 con 0x7fa5f8039fd0 2024-03-30T04:17:43.590 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:42.472+0000 7fa617400700 1 -- 172.21.15.123:0/3325843429 --> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fa61004bed0 con 0x7fa5f8039fd0 2024-03-30T04:17:43.590 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:42.472+0000 7fa60cc00700 1 -- 172.21.15.123:0/3325843429 <== mgr.14160 v2:172.21.15.123:6800/70490328 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fa61004bed0 con 0x7fa5f8039fd0 2024-03-30T04:17:43.590 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:42.473+0000 7fa617400700 1 -- 172.21.15.123:0/3325843429 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7fa5f8039fd0 msgr2=0x7fa5f803c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:43.590 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:42.473+0000 7fa617400700 1 --2- 172.21.15.123:0/3325843429 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7fa5f8039fd0 0x7fa5f803c3b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc0039d0 tx=0x7fa5fc0092f0 comp rx=0 tx=0).stop 2024-03-30T04:17:43.590 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:42.473+0000 7fa617400700 1 -- 172.21.15.123:0/3325843429 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa610102580 msgr2=0x7fa610078c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:43.590 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:42.473+0000 7fa617400700 1 --2- 172.21.15.123:0/3325843429 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa610102580 0x7fa610078c50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa60400c010 tx=0x7fa604009250 comp rx=0 tx=0).stop 2024-03-30T04:17:43.590 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:42.473+0000 7fa617400700 1 -- 172.21.15.123:0/3325843429 shutdown_connections 2024-03-30T04:17:43.590 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:42.473+0000 7fa617400700 1 --2- 172.21.15.123:0/3325843429 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7fa5f8039fd0 0x7fa5f803c3b0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:43.591 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:42.473+0000 7fa617400700 1 --2- 172.21.15.123:0/3325843429 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa610102580 0x7fa610078c50 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:43.591 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:42.473+0000 7fa617400700 1 --2- 172.21.15.123:0/3325843429 >> [v2:172.21.15.123:6800/1776532690,v1:172.21.15.123:6801/1776532690] conn(0x7fa5f80367e0 0x7fa5f8038c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:43.591 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:42.473+0000 7fa617400700 1 -- 172.21.15.123:0/3325843429 >> 172.21.15.123:0/3325843429 conn(0x7fa6100fdd00 msgr2=0x7fa6100fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:43.591 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:42.473+0000 7fa617400700 1 -- 172.21.15.123:0/3325843429 shutdown_connections 2024-03-30T04:17:43.591 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:42.473+0000 7fa617400700 1 -- 172.21.15.123:0/3325843429 wait complete. 2024-03-30T04:17:43.591 INFO:teuthology.orchestra.run.smithi123.stdout:mgr epoch 9 is available 2024-03-30T04:17:43.591 INFO:teuthology.orchestra.run.smithi123.stdout:Generating a dashboard self-signed certificate... 2024-03-30T04:17:43.802 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:43 smithi123 ceph-mon[35245]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-30T04:17:43.802 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:43 smithi123 ceph-mon[35245]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-30T04:17:44.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:44 smithi123 ceph-mon[35245]: mgrmap e12: smithi123.xgksqq(active, since 2s) 2024-03-30T04:17:44.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:44 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' 2024-03-30T04:17:44.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:44 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' 2024-03-30T04:17:44.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:44 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' 2024-03-30T04:17:45.118 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-30T04:17:45.118 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:44.139+0000 7f4081400700 1 Processor -- start 2024-03-30T04:17:45.118 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:44.139+0000 7f4081400700 1 -- start start 2024-03-30T04:17:45.118 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:44.140+0000 7f4081400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f407c076c80 0x7f407c1051b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:45.118 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:44.140+0000 7f4081400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f407c1056f0 con 0x7f407c076c80 2024-03-30T04:17:45.119 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:44.140+0000 7f407a800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f407c076c80 0x7f407c1051b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:45.119 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:44.140+0000 7f407a800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f407c076c80 0x7f407c1051b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35258/0 (socket says 172.21.15.123:35258) 2024-03-30T04:17:45.119 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:44.140+0000 7f407a800700 1 -- 172.21.15.123:0/893730720 learned_addr learned my addr 172.21.15.123:0/893730720 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:17:45.119 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:44.141+0000 7f407a800700 1 -- 172.21.15.123:0/893730720 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f407c105830 con 0x7f407c076c80 2024-03-30T04:17:45.119 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:44.141+0000 7f407a800700 1 --2- 172.21.15.123:0/893730720 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f407c076c80 0x7f407c1051b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f406c0098e0 tx=0x7f406c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f168881c45e1b780 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:45.119 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:44.141+0000 7f4079e00700 1 -- 172.21.15.123:0/893730720 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f406c00b860 con 0x7f407c076c80 2024-03-30T04:17:45.119 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:44.141+0000 7f4079e00700 1 -- 172.21.15.123:0/893730720 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f406c00be70 con 0x7f407c076c80 2024-03-30T04:17:45.119 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:44.141+0000 7f4079e00700 1 -- 172.21.15.123:0/893730720 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f406c010380 con 0x7f407c076c80 2024-03-30T04:17:45.119 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:44.142+0000 7f4081400700 1 -- 172.21.15.123:0/893730720 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f407c076c80 msgr2=0x7f407c1051b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:45.119 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:44.142+0000 7f4081400700 1 --2- 172.21.15.123:0/893730720 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f407c076c80 0x7f407c1051b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f406c0098e0 tx=0x7f406c009250 comp rx=0 tx=0).stop 2024-03-30T04:17:45.119 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:44.142+0000 7f4081400700 1 -- 172.21.15.123:0/893730720 shutdown_connections 2024-03-30T04:17:45.119 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:44.142+0000 7f4081400700 1 --2- 172.21.15.123:0/893730720 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f407c076c80 0x7f407c1051b0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:45.119 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:44.142+0000 7f4081400700 1 -- 172.21.15.123:0/893730720 >> 172.21.15.123:0/893730720 conn(0x7f407c0fcc60 msgr2=0x7f407c0ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:45.120 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:44.142+0000 7f4081400700 1 -- 172.21.15.123:0/893730720 shutdown_connections 2024-03-30T04:17:45.120 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:44.142+0000 7f4081400700 1 -- 172.21.15.123:0/893730720 wait complete. 2024-03-30T04:17:45.120 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:44.143+0000 7f4081400700 1 Processor -- start 2024-03-30T04:17:45.120 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:44.143+0000 7f4081400700 1 -- start start 2024-03-30T04:17:45.120 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:44.143+0000 7f4081400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f407c076c80 0x7f407c113620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:45.120 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:44.143+0000 7f4081400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f407c113b60 con 0x7f407c076c80 2024-03-30T04:17:45.120 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:44.144+0000 7f407a800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f407c076c80 0x7f407c113620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:45.120 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:44.144+0000 7f407a800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f407c076c80 0x7f407c113620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35264/0 (socket says 172.21.15.123:35264) 2024-03-30T04:17:45.120 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:44.144+0000 7f407a800700 1 -- 172.21.15.123:0/1757349955 learned_addr learned my addr 172.21.15.123:0/1757349955 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:17:45.120 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:44.144+0000 7f407a800700 1 -- 172.21.15.123:0/1757349955 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f406c0094f0 con 0x7f407c076c80 2024-03-30T04:17:45.120 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:44.144+0000 7f407a800700 1 --2- 172.21.15.123:0/1757349955 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f407c076c80 0x7f407c113620 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f406c003080 tx=0x7f406c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:45.120 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:44.144+0000 7f4072a00700 1 -- 172.21.15.123:0/1757349955 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f406c010960 con 0x7f407c076c80 2024-03-30T04:17:45.120 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:44.144+0000 7f4072a00700 1 -- 172.21.15.123:0/1757349955 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f406c01a430 con 0x7f407c076c80 2024-03-30T04:17:45.120 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:44.144+0000 7f4081400700 1 -- 172.21.15.123:0/1757349955 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f407c113d60 con 0x7f407c076c80 2024-03-30T04:17:45.121 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:44.145+0000 7f4072a00700 1 -- 172.21.15.123:0/1757349955 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f406c0245b0 con 0x7f407c076c80 2024-03-30T04:17:45.121 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:44.145+0000 7f4081400700 1 -- 172.21.15.123:0/1757349955 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f407c114880 con 0x7f407c076c80 2024-03-30T04:17:45.121 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:44.146+0000 7f4072a00700 1 -- 172.21.15.123:0/1757349955 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 12) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f406c02d3f0 con 0x7f407c076c80 2024-03-30T04:17:45.121 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:44.146+0000 7f4081400700 1 -- 172.21.15.123:0/1757349955 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f407c05f5f0 con 0x7f407c076c80 2024-03-30T04:17:45.121 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:44.146+0000 7f4072a00700 1 --2- 172.21.15.123:0/1757349955 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f40600367e0 0x7f4060038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:45.121 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:44.146+0000 7f4072a00700 1 -- 172.21.15.123:0/1757349955 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f406c04a4a0 con 0x7f407c076c80 2024-03-30T04:17:45.121 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:44.146+0000 7f4073e00700 1 --2- 172.21.15.123:0/1757349955 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f40600367e0 0x7f4060038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:45.121 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:44.147+0000 7f4073e00700 1 --2- 172.21.15.123:0/1757349955 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f40600367e0 0x7f4060038c90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4064009990 tx=0x7f4064006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:45.121 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:44.151+0000 7f4072a00700 1 -- 172.21.15.123:0/1757349955 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f406c02b8e0 con 0x7f407c076c80 2024-03-30T04:17:45.121 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:44.357+0000 7f4081400700 1 -- 172.21.15.123:0/1757349955 --> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f407c0fddd0 con 0x7f40600367e0 2024-03-30T04:17:45.121 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:44.426+0000 7f4072a00700 1 -- 172.21.15.123:0/1757349955 <== mgr.14160 v2:172.21.15.123:6800/70490328 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f407c0fddd0 con 0x7f40600367e0 2024-03-30T04:17:45.121 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:44.429+0000 7f4081400700 1 -- 172.21.15.123:0/1757349955 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f40600367e0 msgr2=0x7f4060038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:45.121 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:44.429+0000 7f4081400700 1 --2- 172.21.15.123:0/1757349955 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f40600367e0 0x7f4060038c90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4064009990 tx=0x7f4064006e40 comp rx=0 tx=0).stop 2024-03-30T04:17:45.122 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:44.429+0000 7f4081400700 1 -- 172.21.15.123:0/1757349955 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f407c076c80 msgr2=0x7f407c113620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:45.122 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:44.429+0000 7f4081400700 1 --2- 172.21.15.123:0/1757349955 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f407c076c80 0x7f407c113620 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f406c003080 tx=0x7f406c009250 comp rx=0 tx=0).stop 2024-03-30T04:17:45.122 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:44.429+0000 7f4081400700 1 -- 172.21.15.123:0/1757349955 shutdown_connections 2024-03-30T04:17:45.122 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:44.429+0000 7f4081400700 1 --2- 172.21.15.123:0/1757349955 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f40600367e0 0x7f4060038c90 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:45.122 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:44.429+0000 7f4081400700 1 --2- 172.21.15.123:0/1757349955 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f407c076c80 0x7f407c113620 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:45.122 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:44.429+0000 7f4081400700 1 -- 172.21.15.123:0/1757349955 >> 172.21.15.123:0/1757349955 conn(0x7f407c0fcc60 msgr2=0x7f407c0fd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:45.122 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:44.429+0000 7f4081400700 1 -- 172.21.15.123:0/1757349955 shutdown_connections 2024-03-30T04:17:45.122 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:44.429+0000 7f4081400700 1 -- 172.21.15.123:0/1757349955 wait complete. 2024-03-30T04:17:45.122 INFO:teuthology.orchestra.run.smithi123.stdout:Creating initial admin user... 2024-03-30T04:17:46.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:46 smithi123 ceph-mon[35245]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:17:46.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:46 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' 2024-03-30T04:17:46.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:46 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' 2024-03-30T04:17:47.644 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$NNtsUU0tKqTGwb7a/.CVmOhwxtJS7fbiBgtgMHBDior12MaPgQLqW", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711772266, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-30T04:17:47.644 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:45.684+0000 7f6d92a00700 1 Processor -- start 2024-03-30T04:17:47.644 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:45.684+0000 7f6d92a00700 1 -- start start 2024-03-30T04:17:47.644 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:45.684+0000 7f6d92a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d8c102580 0x7f6d8c102990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:47.644 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:45.684+0000 7f6d92a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8c102ed0 con 0x7f6d8c102580 2024-03-30T04:17:47.645 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:45.684+0000 7f6d8be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d8c102580 0x7f6d8c102990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:47.645 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:45.685+0000 7f6d8be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d8c102580 0x7f6d8c102990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35270/0 (socket says 172.21.15.123:35270) 2024-03-30T04:17:47.645 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:45.685+0000 7f6d8be00700 1 -- 172.21.15.123:0/3569347333 learned_addr learned my addr 172.21.15.123:0/3569347333 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:17:47.645 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:45.685+0000 7f6d8be00700 1 -- 172.21.15.123:0/3569347333 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d8c103010 con 0x7f6d8c102580 2024-03-30T04:17:47.645 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:45.685+0000 7f6d8be00700 1 --2- 172.21.15.123:0/3569347333 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d8c102580 0x7f6d8c102990 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c0098e0 tx=0x7f6d7c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea73c735046e492 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:47.645 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:45.685+0000 7f6d8aa00700 1 -- 172.21.15.123:0/3569347333 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d7c00b860 con 0x7f6d8c102580 2024-03-30T04:17:47.645 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:45.685+0000 7f6d8aa00700 1 -- 172.21.15.123:0/3569347333 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6d7c00be70 con 0x7f6d8c102580 2024-03-30T04:17:47.645 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:45.685+0000 7f6d8aa00700 1 -- 172.21.15.123:0/3569347333 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d7c010380 con 0x7f6d8c102580 2024-03-30T04:17:47.645 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:45.686+0000 7f6d92a00700 1 -- 172.21.15.123:0/3569347333 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d8c102580 msgr2=0x7f6d8c102990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:47.645 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:45.686+0000 7f6d92a00700 1 --2- 172.21.15.123:0/3569347333 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d8c102580 0x7f6d8c102990 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c0098e0 tx=0x7f6d7c009250 comp rx=0 tx=0).stop 2024-03-30T04:17:47.645 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:45.686+0000 7f6d92a00700 1 -- 172.21.15.123:0/3569347333 shutdown_connections 2024-03-30T04:17:47.645 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:45.686+0000 7f6d92a00700 1 --2- 172.21.15.123:0/3569347333 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d8c102580 0x7f6d8c102990 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:47.645 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:45.686+0000 7f6d92a00700 1 -- 172.21.15.123:0/3569347333 >> 172.21.15.123:0/3569347333 conn(0x7f6d8c0fdd00 msgr2=0x7f6d8c100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:47.646 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:45.686+0000 7f6d92a00700 1 -- 172.21.15.123:0/3569347333 shutdown_connections 2024-03-30T04:17:47.646 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:45.686+0000 7f6d92a00700 1 -- 172.21.15.123:0/3569347333 wait complete. 2024-03-30T04:17:47.646 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:45.687+0000 7f6d92a00700 1 Processor -- start 2024-03-30T04:17:47.646 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:45.687+0000 7f6d92a00700 1 -- start start 2024-03-30T04:17:47.646 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:45.687+0000 7f6d92a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d8c102580 0x7f6d8c111370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:47.646 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:45.688+0000 7f6d92a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8c1118b0 con 0x7f6d8c102580 2024-03-30T04:17:47.646 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:45.688+0000 7f6d8be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d8c102580 0x7f6d8c111370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:47.646 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:45.688+0000 7f6d8be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d8c102580 0x7f6d8c111370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35274/0 (socket says 172.21.15.123:35274) 2024-03-30T04:17:47.646 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:45.688+0000 7f6d8be00700 1 -- 172.21.15.123:0/1191909844 learned_addr learned my addr 172.21.15.123:0/1191909844 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:17:47.646 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:45.688+0000 7f6d8be00700 1 -- 172.21.15.123:0/1191909844 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d7c0094f0 con 0x7f6d8c102580 2024-03-30T04:17:47.646 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:45.688+0000 7f6d8be00700 1 --2- 172.21.15.123:0/1191909844 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d8c102580 0x7f6d8c111370 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c011ca0 tx=0x7f6d7c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:47.646 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:45.689+0000 7f6d83e00700 1 -- 172.21.15.123:0/1191909844 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d7c010850 con 0x7f6d8c102580 2024-03-30T04:17:47.646 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:45.689+0000 7f6d83e00700 1 -- 172.21.15.123:0/1191909844 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6d7c010e60 con 0x7f6d8c102580 2024-03-30T04:17:47.646 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:45.689+0000 7f6d92a00700 1 -- 172.21.15.123:0/1191909844 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d8c111ab0 con 0x7f6d8c102580 2024-03-30T04:17:47.647 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:45.689+0000 7f6d83e00700 1 -- 172.21.15.123:0/1191909844 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d7c01ae20 con 0x7f6d8c102580 2024-03-30T04:17:47.647 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:45.689+0000 7f6d92a00700 1 -- 172.21.15.123:0/1191909844 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d8c112620 con 0x7f6d8c102580 2024-03-30T04:17:47.647 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:45.690+0000 7f6d83e00700 1 -- 172.21.15.123:0/1191909844 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 12) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f6d7c02d3f0 con 0x7f6d8c102580 2024-03-30T04:17:47.647 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:45.690+0000 7f6d92a00700 1 -- 172.21.15.123:0/1191909844 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d8c05f5f0 con 0x7f6d8c102580 2024-03-30T04:17:47.647 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:45.690+0000 7f6d83e00700 1 --2- 172.21.15.123:0/1191909844 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f6d700367e0 0x7f6d70038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:47.647 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:45.690+0000 7f6d83e00700 1 -- 172.21.15.123:0/1191909844 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f6d7c04a640 con 0x7f6d8c102580 2024-03-30T04:17:47.647 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:45.690+0000 7f6d8b400700 1 --2- 172.21.15.123:0/1191909844 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f6d700367e0 0x7f6d70038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:47.647 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:45.691+0000 7f6d8b400700 1 --2- 172.21.15.123:0/1191909844 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f6d700367e0 0x7f6d70038c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6d78009990 tx=0x7f6d78006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:47.647 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:45.695+0000 7f6d83e00700 1 -- 172.21.15.123:0/1191909844 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f6d7c02b8e0 con 0x7f6d8c102580 2024-03-30T04:17:47.647 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:45.941+0000 7f6d92a00700 1 -- 172.21.15.123:0/1191909844 --> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f6d8c05f410 con 0x7f6d700367e0 2024-03-30T04:17:47.647 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:46.167+0000 7f6d83e00700 1 -- 172.21.15.123:0/1191909844 <== mgr.14160 v2:172.21.15.123:6800/70490328 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f6d8c05f410 con 0x7f6d700367e0 2024-03-30T04:17:47.647 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:46.169+0000 7f6d92a00700 1 -- 172.21.15.123:0/1191909844 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f6d700367e0 msgr2=0x7f6d70038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:47.647 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:46.169+0000 7f6d92a00700 1 --2- 172.21.15.123:0/1191909844 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f6d700367e0 0x7f6d70038c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6d78009990 tx=0x7f6d78006e40 comp rx=0 tx=0).stop 2024-03-30T04:17:47.647 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:46.169+0000 7f6d92a00700 1 -- 172.21.15.123:0/1191909844 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d8c102580 msgr2=0x7f6d8c111370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:47.648 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:46.169+0000 7f6d92a00700 1 --2- 172.21.15.123:0/1191909844 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d8c102580 0x7f6d8c111370 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c011ca0 tx=0x7f6d7c009250 comp rx=0 tx=0).stop 2024-03-30T04:17:47.648 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:46.169+0000 7f6d92a00700 1 -- 172.21.15.123:0/1191909844 shutdown_connections 2024-03-30T04:17:47.648 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:46.170+0000 7f6d92a00700 1 --2- 172.21.15.123:0/1191909844 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f6d700367e0 0x7f6d70038c90 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:47.648 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:46.170+0000 7f6d92a00700 1 --2- 172.21.15.123:0/1191909844 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d8c102580 0x7f6d8c111370 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:47.648 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:46.170+0000 7f6d92a00700 1 -- 172.21.15.123:0/1191909844 >> 172.21.15.123:0/1191909844 conn(0x7f6d8c0fdd00 msgr2=0x7f6d8c0fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:47.648 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:46.170+0000 7f6d92a00700 1 -- 172.21.15.123:0/1191909844 shutdown_connections 2024-03-30T04:17:47.648 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:46.170+0000 7f6d92a00700 1 -- 172.21.15.123:0/1191909844 wait complete. 2024-03-30T04:17:47.648 INFO:teuthology.orchestra.run.smithi123.stdout:Fetching dashboard port number... 2024-03-30T04:17:47.648 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:47 smithi123 ceph-mon[35245]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:17:49.004 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stdout 8443 2024-03-30T04:17:49.010 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:48.232+0000 7f5fc2200700 1 Processor -- start 2024-03-30T04:17:49.010 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:48.232+0000 7f5fc2200700 1 -- start start 2024-03-30T04:17:49.011 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:48.232+0000 7f5fc2200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fbc102530 0x7f5fbc102940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:49.011 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:48.232+0000 7f5fc2200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fbc102e80 con 0x7f5fbc102530 2024-03-30T04:17:49.011 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:48.233+0000 7f5fbb400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fbc102530 0x7f5fbc102940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:49.011 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:48.233+0000 7f5fbb400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fbc102530 0x7f5fbc102940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35286/0 (socket says 172.21.15.123:35286) 2024-03-30T04:17:49.011 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:48.233+0000 7f5fbb400700 1 -- 172.21.15.123:0/3947591546 learned_addr learned my addr 172.21.15.123:0/3947591546 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:17:49.011 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:48.233+0000 7f5fbb400700 1 -- 172.21.15.123:0/3947591546 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fbc102fc0 con 0x7f5fbc102530 2024-03-30T04:17:49.011 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:48.233+0000 7f5fbb400700 1 --2- 172.21.15.123:0/3947591546 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fbc102530 0x7f5fbc102940 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5fac0098e0 tx=0x7f5fac009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=778f61f3ee93ef6a server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:49.011 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:48.233+0000 7f5fba000700 1 -- 172.21.15.123:0/3947591546 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5fac00b860 con 0x7f5fbc102530 2024-03-30T04:17:49.011 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:48.233+0000 7f5fba000700 1 -- 172.21.15.123:0/3947591546 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5fac00be70 con 0x7f5fbc102530 2024-03-30T04:17:49.011 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:48.233+0000 7f5fba000700 1 -- 172.21.15.123:0/3947591546 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5fac010380 con 0x7f5fbc102530 2024-03-30T04:17:49.011 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:48.234+0000 7f5fc2200700 1 -- 172.21.15.123:0/3947591546 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fbc102530 msgr2=0x7f5fbc102940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:49.011 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:48.234+0000 7f5fc2200700 1 --2- 172.21.15.123:0/3947591546 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fbc102530 0x7f5fbc102940 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5fac0098e0 tx=0x7f5fac009250 comp rx=0 tx=0).stop 2024-03-30T04:17:49.011 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:48.234+0000 7f5fc2200700 1 -- 172.21.15.123:0/3947591546 shutdown_connections 2024-03-30T04:17:49.011 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:48.234+0000 7f5fc2200700 1 --2- 172.21.15.123:0/3947591546 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fbc102530 0x7f5fbc102940 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:49.012 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:48.234+0000 7f5fc2200700 1 -- 172.21.15.123:0/3947591546 >> 172.21.15.123:0/3947591546 conn(0x7f5fbc0fdcd0 msgr2=0x7f5fbc100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:49.012 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:48.234+0000 7f5fc2200700 1 -- 172.21.15.123:0/3947591546 shutdown_connections 2024-03-30T04:17:49.012 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:48.234+0000 7f5fc2200700 1 -- 172.21.15.123:0/3947591546 wait complete. 2024-03-30T04:17:49.012 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:48.235+0000 7f5fc2200700 1 Processor -- start 2024-03-30T04:17:49.012 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:48.235+0000 7f5fc2200700 1 -- start start 2024-03-30T04:17:49.012 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:48.235+0000 7f5fc2200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fbc102530 0x7f5fbc078c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:49.012 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:48.235+0000 7f5fc2200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fbc079190 con 0x7f5fbc102530 2024-03-30T04:17:49.012 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:48.236+0000 7f5fbb400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fbc102530 0x7f5fbc078c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:49.012 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:48.236+0000 7f5fbb400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fbc102530 0x7f5fbc078c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35298/0 (socket says 172.21.15.123:35298) 2024-03-30T04:17:49.012 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:48.236+0000 7f5fbb400700 1 -- 172.21.15.123:0/2072057056 learned_addr learned my addr 172.21.15.123:0/2072057056 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:17:49.012 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:48.236+0000 7f5fbb400700 1 -- 172.21.15.123:0/2072057056 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fac0094f0 con 0x7f5fbc102530 2024-03-30T04:17:49.012 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:48.236+0000 7f5fbb400700 1 --2- 172.21.15.123:0/2072057056 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fbc102530 0x7f5fbc078c50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5fac000c00 tx=0x7f5fac009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:49.012 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:48.236+0000 7f5fb3400700 1 -- 172.21.15.123:0/2072057056 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5fac0105f0 con 0x7f5fbc102530 2024-03-30T04:17:49.013 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:48.236+0000 7f5fb3400700 1 -- 172.21.15.123:0/2072057056 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5fac010c00 con 0x7f5fbc102530 2024-03-30T04:17:49.013 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:48.236+0000 7f5fb3400700 1 -- 172.21.15.123:0/2072057056 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5fac01ac70 con 0x7f5fbc102530 2024-03-30T04:17:49.013 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:48.236+0000 7f5fc2200700 1 -- 172.21.15.123:0/2072057056 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fbc077200 con 0x7f5fbc102530 2024-03-30T04:17:49.013 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:48.237+0000 7f5fc2200700 1 -- 172.21.15.123:0/2072057056 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fbc0776f0 con 0x7f5fbc102530 2024-03-30T04:17:49.013 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:48.238+0000 7f5fb3400700 1 -- 172.21.15.123:0/2072057056 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f5fac010d70 con 0x7f5fbc102530 2024-03-30T04:17:49.013 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:48.238+0000 7f5fc2200700 1 -- 172.21.15.123:0/2072057056 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fbc04bed0 con 0x7f5fbc102530 2024-03-30T04:17:49.013 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:48.238+0000 7f5fb3400700 1 --2- 172.21.15.123:0/2072057056 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f5fa00368d0 0x7f5fa0038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:49.013 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:48.238+0000 7f5fb3400700 1 -- 172.21.15.123:0/2072057056 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f5fac04a530 con 0x7f5fbc102530 2024-03-30T04:17:49.013 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:48.238+0000 7f5fbaa00700 1 --2- 172.21.15.123:0/2072057056 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f5fa00368d0 0x7f5fa0038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:49.013 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:48.239+0000 7f5fbaa00700 1 --2- 172.21.15.123:0/2072057056 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f5fa00368d0 0x7f5fa0038d80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8009990 tx=0x7f5fa8006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:49.013 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:48.243+0000 7f5fb3400700 1 -- 172.21.15.123:0/2072057056 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f5fac018350 con 0x7f5fbc102530 2024-03-30T04:17:49.013 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:48.433+0000 7f5fc2200700 1 -- 172.21.15.123:0/2072057056 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f5fbc077b70 con 0x7f5fbc102530 2024-03-30T04:17:49.013 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:48.434+0000 7f5fb3400700 1 -- 172.21.15.123:0/2072057056 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f5fac020030 con 0x7f5fbc102530 2024-03-30T04:17:49.013 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:48.436+0000 7f5fc2200700 1 -- 172.21.15.123:0/2072057056 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f5fa00368d0 msgr2=0x7f5fa0038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:49.014 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:48.436+0000 7f5fc2200700 1 --2- 172.21.15.123:0/2072057056 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f5fa00368d0 0x7f5fa0038d80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8009990 tx=0x7f5fa8006e40 comp rx=0 tx=0).stop 2024-03-30T04:17:49.014 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:48.436+0000 7f5fc2200700 1 -- 172.21.15.123:0/2072057056 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fbc102530 msgr2=0x7f5fbc078c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:49.014 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:48.436+0000 7f5fc2200700 1 --2- 172.21.15.123:0/2072057056 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fbc102530 0x7f5fbc078c50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5fac000c00 tx=0x7f5fac009250 comp rx=0 tx=0).stop 2024-03-30T04:17:49.014 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:48.436+0000 7f5fc2200700 1 -- 172.21.15.123:0/2072057056 shutdown_connections 2024-03-30T04:17:49.014 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:48.436+0000 7f5fc2200700 1 --2- 172.21.15.123:0/2072057056 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f5fa00368d0 0x7f5fa0038d80 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:49.014 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:48.436+0000 7f5fc2200700 1 --2- 172.21.15.123:0/2072057056 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fbc102530 0x7f5fbc078c50 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:49.014 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:48.436+0000 7f5fc2200700 1 -- 172.21.15.123:0/2072057056 >> 172.21.15.123:0/2072057056 conn(0x7f5fbc0fdcd0 msgr2=0x7f5fbc0fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:49.014 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:48.436+0000 7f5fc2200700 1 -- 172.21.15.123:0/2072057056 shutdown_connections 2024-03-30T04:17:49.014 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:48.436+0000 7f5fc2200700 1 -- 172.21.15.123:0/2072057056 wait complete. 2024-03-30T04:17:49.026 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-30T04:17:49.026 INFO:teuthology.orchestra.run.smithi123.stdout:systemctl: stdout disabled 2024-03-30T04:17:49.043 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-30T04:17:49.043 INFO:teuthology.orchestra.run.smithi123.stdout:systemctl: stdout inactive 2024-03-30T04:17:49.043 INFO:teuthology.orchestra.run.smithi123.stdout:firewalld.service is not enabled 2024-03-30T04:17:49.043 INFO:teuthology.orchestra.run.smithi123.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-30T04:17:49.045 INFO:teuthology.orchestra.run.smithi123.stdout:Ceph Dashboard is now available at: 2024-03-30T04:17:49.045 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:17:49.046 INFO:teuthology.orchestra.run.smithi123.stdout: URL: https://smithi123.front.sepia.ceph.com:8443/ 2024-03-30T04:17:49.046 INFO:teuthology.orchestra.run.smithi123.stdout: User: admin 2024-03-30T04:17:49.046 INFO:teuthology.orchestra.run.smithi123.stdout: Password: grirvvv9gu 2024-03-30T04:17:49.046 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:17:49.046 INFO:teuthology.orchestra.run.smithi123.stdout:Saving cluster configuration to /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/config directory 2024-03-30T04:17:49.047 INFO:teuthology.orchestra.run.smithi123.stdout:Enabling autotune for osd_memory_target 2024-03-30T04:17:49.335 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:49 smithi123 ceph-mon[35245]: mgrmap e13: smithi123.xgksqq(active, since 6s) 2024-03-30T04:17:49.335 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:49 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/2072057056' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-30T04:17:50.497 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:49.654+0000 7f67f3c00700 1 Processor -- start 2024-03-30T04:17:50.497 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:49.655+0000 7f67f3c00700 1 -- start start 2024-03-30T04:17:50.497 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:49.655+0000 7f67f3c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67ec102530 0x7f67ec102940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:50.497 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:49.655+0000 7f67f3c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67ec102e80 con 0x7f67ec102530 2024-03-30T04:17:50.497 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:49.655+0000 7f67f1200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67ec102530 0x7f67ec102940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:50.497 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:49.655+0000 7f67f1200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67ec102530 0x7f67ec102940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35312/0 (socket says 172.21.15.123:35312) 2024-03-30T04:17:50.497 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:49.655+0000 7f67f1200700 1 -- 172.21.15.123:0/1496591015 learned_addr learned my addr 172.21.15.123:0/1496591015 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:17:50.497 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:49.656+0000 7f67f1200700 1 -- 172.21.15.123:0/1496591015 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67ec102fc0 con 0x7f67ec102530 2024-03-30T04:17:50.497 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:49.656+0000 7f67f1200700 1 --2- 172.21.15.123:0/1496591015 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67ec102530 0x7f67ec102940 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f67e00098e0 tx=0x7f67e0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e177b8a9f01dd6e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:50.498 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:49.656+0000 7f67eb400700 1 -- 172.21.15.123:0/1496591015 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67e000b860 con 0x7f67ec102530 2024-03-30T04:17:50.498 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:49.656+0000 7f67eb400700 1 -- 172.21.15.123:0/1496591015 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f67e000be70 con 0x7f67ec102530 2024-03-30T04:17:50.498 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:49.656+0000 7f67eb400700 1 -- 172.21.15.123:0/1496591015 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67e0010380 con 0x7f67ec102530 2024-03-30T04:17:50.498 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:49.657+0000 7f67f3c00700 1 -- 172.21.15.123:0/1496591015 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67ec102530 msgr2=0x7f67ec102940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:50.498 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:49.657+0000 7f67f3c00700 1 --2- 172.21.15.123:0/1496591015 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67ec102530 0x7f67ec102940 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f67e00098e0 tx=0x7f67e0009250 comp rx=0 tx=0).stop 2024-03-30T04:17:50.498 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:49.657+0000 7f67f3c00700 1 -- 172.21.15.123:0/1496591015 shutdown_connections 2024-03-30T04:17:50.498 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:49.657+0000 7f67f3c00700 1 --2- 172.21.15.123:0/1496591015 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67ec102530 0x7f67ec102940 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:50.498 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:49.657+0000 7f67f3c00700 1 -- 172.21.15.123:0/1496591015 >> 172.21.15.123:0/1496591015 conn(0x7f67ec0fdcd0 msgr2=0x7f67ec100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:50.498 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:49.657+0000 7f67f3c00700 1 -- 172.21.15.123:0/1496591015 shutdown_connections 2024-03-30T04:17:50.498 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:49.657+0000 7f67f3c00700 1 -- 172.21.15.123:0/1496591015 wait complete. 2024-03-30T04:17:50.498 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:49.658+0000 7f67f3c00700 1 Processor -- start 2024-03-30T04:17:50.498 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:49.658+0000 7f67f3c00700 1 -- start start 2024-03-30T04:17:50.498 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:49.658+0000 7f67f3c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67ec102530 0x7f67ec113670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:50.499 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:49.658+0000 7f67f3c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67ec113bb0 con 0x7f67ec102530 2024-03-30T04:17:50.499 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:49.658+0000 7f67f1200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67ec102530 0x7f67ec113670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:50.499 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:49.658+0000 7f67f1200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67ec102530 0x7f67ec113670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35322/0 (socket says 172.21.15.123:35322) 2024-03-30T04:17:50.499 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:49.658+0000 7f67f1200700 1 -- 172.21.15.123:0/559199633 learned_addr learned my addr 172.21.15.123:0/559199633 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:17:50.499 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:49.659+0000 7f67f1200700 1 -- 172.21.15.123:0/559199633 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67e00094f0 con 0x7f67ec102530 2024-03-30T04:17:50.499 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:49.659+0000 7f67f1200700 1 --2- 172.21.15.123:0/559199633 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67ec102530 0x7f67ec113670 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f67e0000c00 tx=0x7f67e0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:50.499 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:49.659+0000 7f67e8c00700 1 -- 172.21.15.123:0/559199633 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67e00105f0 con 0x7f67ec102530 2024-03-30T04:17:50.499 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:49.659+0000 7f67e8c00700 1 -- 172.21.15.123:0/559199633 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f67e0010c00 con 0x7f67ec102530 2024-03-30T04:17:50.499 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:49.659+0000 7f67e8c00700 1 -- 172.21.15.123:0/559199633 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67e001ac70 con 0x7f67ec102530 2024-03-30T04:17:50.499 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:49.659+0000 7f67f3c00700 1 -- 172.21.15.123:0/559199633 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67ec113db0 con 0x7f67ec102530 2024-03-30T04:17:50.499 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:49.659+0000 7f67f3c00700 1 -- 172.21.15.123:0/559199633 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67ec114920 con 0x7f67ec102530 2024-03-30T04:17:50.499 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:49.660+0000 7f67e8c00700 1 -- 172.21.15.123:0/559199633 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f67e0010750 con 0x7f67ec102530 2024-03-30T04:17:50.499 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:49.660+0000 7f67f3c00700 1 -- 172.21.15.123:0/559199633 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67ec05f5f0 con 0x7f67ec102530 2024-03-30T04:17:50.499 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:49.661+0000 7f67e8c00700 1 --2- 172.21.15.123:0/559199633 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f67d40368d0 0x7f67d4038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:50.500 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:49.661+0000 7f67e8c00700 1 -- 172.21.15.123:0/559199633 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f67e004a570 con 0x7f67ec102530 2024-03-30T04:17:50.500 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:49.661+0000 7f67ebe00700 1 --2- 172.21.15.123:0/559199633 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f67d40368d0 0x7f67d4038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:50.500 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:49.661+0000 7f67ebe00700 1 --2- 172.21.15.123:0/559199633 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f67d40368d0 0x7f67d4038d80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f67d8009990 tx=0x7f67d8006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:50.500 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:49.665+0000 7f67e8c00700 1 -- 172.21.15.123:0/559199633 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f67e002c300 con 0x7f67ec102530 2024-03-30T04:17:50.500 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:49.858+0000 7f67f3c00700 1 -- 172.21.15.123:0/559199633 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f67ec04bed0 con 0x7f67ec102530 2024-03-30T04:17:50.500 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:49.858+0000 7f67e8c00700 1 -- 172.21.15.123:0/559199633 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f67e0020030 con 0x7f67ec102530 2024-03-30T04:17:50.500 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:49.860+0000 7f67f3c00700 1 -- 172.21.15.123:0/559199633 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f67d40368d0 msgr2=0x7f67d4038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:50.500 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:49.861+0000 7f67f3c00700 1 --2- 172.21.15.123:0/559199633 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f67d40368d0 0x7f67d4038d80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f67d8009990 tx=0x7f67d8006e40 comp rx=0 tx=0).stop 2024-03-30T04:17:50.500 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:49.861+0000 7f67f3c00700 1 -- 172.21.15.123:0/559199633 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67ec102530 msgr2=0x7f67ec113670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:50.500 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:49.861+0000 7f67f3c00700 1 --2- 172.21.15.123:0/559199633 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67ec102530 0x7f67ec113670 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f67e0000c00 tx=0x7f67e0009250 comp rx=0 tx=0).stop 2024-03-30T04:17:50.500 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:49.861+0000 7f67f3c00700 1 -- 172.21.15.123:0/559199633 shutdown_connections 2024-03-30T04:17:50.500 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:49.861+0000 7f67f3c00700 1 --2- 172.21.15.123:0/559199633 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f67d40368d0 0x7f67d4038d80 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:50.500 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:49.861+0000 7f67f3c00700 1 --2- 172.21.15.123:0/559199633 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67ec102530 0x7f67ec113670 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:50.500 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:49.861+0000 7f67f3c00700 1 -- 172.21.15.123:0/559199633 >> 172.21.15.123:0/559199633 conn(0x7f67ec0fdcd0 msgr2=0x7f67ec0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:50.501 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:49.861+0000 7f67f3c00700 1 -- 172.21.15.123:0/559199633 shutdown_connections 2024-03-30T04:17:50.501 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:49.861+0000 7f67f3c00700 1 -- 172.21.15.123:0/559199633 wait complete. 2024-03-30T04:17:51.964 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:51.033+0000 7f089da00700 1 Processor -- start 2024-03-30T04:17:51.964 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:51.033+0000 7f089da00700 1 -- start start 2024-03-30T04:17:51.964 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:51.034+0000 7f089da00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0898078d60 0x7f0898077260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:51.964 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:51.034+0000 7f089da00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0898079170 con 0x7f0898078d60 2024-03-30T04:17:51.964 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:51.034+0000 7f0896a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0898078d60 0x7f0898077260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:51.964 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:51.034+0000 7f0896a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0898078d60 0x7f0898077260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35326/0 (socket says 172.21.15.123:35326) 2024-03-30T04:17:51.964 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:51.034+0000 7f0896a00700 1 -- 172.21.15.123:0/4185658003 learned_addr learned my addr 172.21.15.123:0/4185658003 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:17:51.964 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:51.034+0000 7f0896a00700 1 -- 172.21.15.123:0/4185658003 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08980777a0 con 0x7f0898078d60 2024-03-30T04:17:51.965 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:51.035+0000 7f0896a00700 1 --2- 172.21.15.123:0/4185658003 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0898078d60 0x7f0898077260 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f08880098e0 tx=0x7f0888009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f8591885619155 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:51.965 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:51.035+0000 7f0895600700 1 -- 172.21.15.123:0/4185658003 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f088800b860 con 0x7f0898078d60 2024-03-30T04:17:51.965 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:51.035+0000 7f0895600700 1 -- 172.21.15.123:0/4185658003 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f088800be70 con 0x7f0898078d60 2024-03-30T04:17:51.965 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:51.035+0000 7f0895600700 1 -- 172.21.15.123:0/4185658003 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0888010380 con 0x7f0898078d60 2024-03-30T04:17:51.965 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:51.035+0000 7f089da00700 1 -- 172.21.15.123:0/4185658003 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0898078d60 msgr2=0x7f0898077260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:51.965 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:51.035+0000 7f089da00700 1 --2- 172.21.15.123:0/4185658003 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0898078d60 0x7f0898077260 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f08880098e0 tx=0x7f0888009250 comp rx=0 tx=0).stop 2024-03-30T04:17:51.965 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:51.035+0000 7f089da00700 1 -- 172.21.15.123:0/4185658003 shutdown_connections 2024-03-30T04:17:51.965 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:51.035+0000 7f089da00700 1 --2- 172.21.15.123:0/4185658003 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0898078d60 0x7f0898077260 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:51.965 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:51.035+0000 7f089da00700 1 -- 172.21.15.123:0/4185658003 >> 172.21.15.123:0/4185658003 conn(0x7f08980fdcd0 msgr2=0x7f0898100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:51.965 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:51.036+0000 7f089da00700 1 -- 172.21.15.123:0/4185658003 shutdown_connections 2024-03-30T04:17:51.965 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:51.036+0000 7f089da00700 1 -- 172.21.15.123:0/4185658003 wait complete. 2024-03-30T04:17:51.965 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:51.036+0000 7f089da00700 1 Processor -- start 2024-03-30T04:17:51.965 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:51.036+0000 7f089da00700 1 -- start start 2024-03-30T04:17:51.965 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:51.037+0000 7f089da00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0898078d60 0x7f0898113630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:51.966 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:51.037+0000 7f089da00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0898113b70 con 0x7f0898078d60 2024-03-30T04:17:51.966 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:51.037+0000 7f0896a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0898078d60 0x7f0898113630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:51.966 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:51.037+0000 7f0896a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0898078d60 0x7f0898113630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35328/0 (socket says 172.21.15.123:35328) 2024-03-30T04:17:51.966 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:51.037+0000 7f0896a00700 1 -- 172.21.15.123:0/1429788396 learned_addr learned my addr 172.21.15.123:0/1429788396 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:17:51.966 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:51.037+0000 7f0896a00700 1 -- 172.21.15.123:0/1429788396 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08880094f0 con 0x7f0898078d60 2024-03-30T04:17:51.966 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:51.038+0000 7f0896a00700 1 --2- 172.21.15.123:0/1429788396 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0898078d60 0x7f0898113630 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0888011ca0 tx=0x7f0888009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:51.966 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:51.038+0000 7f088ea00700 1 -- 172.21.15.123:0/1429788396 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0888010850 con 0x7f0898078d60 2024-03-30T04:17:51.966 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:51.038+0000 7f088ea00700 1 -- 172.21.15.123:0/1429788396 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0888010e60 con 0x7f0898078d60 2024-03-30T04:17:51.966 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:51.038+0000 7f089da00700 1 -- 172.21.15.123:0/1429788396 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0898113d70 con 0x7f0898078d60 2024-03-30T04:17:51.966 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:51.038+0000 7f088ea00700 1 -- 172.21.15.123:0/1429788396 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f088801ae30 con 0x7f0898078d60 2024-03-30T04:17:51.966 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:51.038+0000 7f089da00700 1 -- 172.21.15.123:0/1429788396 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08981148e0 con 0x7f0898078d60 2024-03-30T04:17:51.966 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:51.039+0000 7f088ea00700 1 -- 172.21.15.123:0/1429788396 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f08880109b0 con 0x7f0898078d60 2024-03-30T04:17:51.966 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:51.039+0000 7f089da00700 1 -- 172.21.15.123:0/1429788396 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f089805f5f0 con 0x7f0898078d60 2024-03-30T04:17:51.967 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:51.039+0000 7f088ea00700 1 --2- 172.21.15.123:0/1429788396 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f087c0368d0 0x7f087c038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:51.967 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:51.039+0000 7f088ea00700 1 -- 172.21.15.123:0/1429788396 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f088804af70 con 0x7f0898078d60 2024-03-30T04:17:51.967 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:51.040+0000 7f0896000700 1 --2- 172.21.15.123:0/1429788396 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f087c0368d0 0x7f087c038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:51.967 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:51.040+0000 7f0896000700 1 --2- 172.21.15.123:0/1429788396 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f087c0368d0 0x7f087c038d80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0884009990 tx=0x7f0884006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:51.967 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:51.044+0000 7f088ea00700 1 -- 172.21.15.123:0/1429788396 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f0888018350 con 0x7f0898078d60 2024-03-30T04:17:51.967 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:51.338+0000 7f089da00700 1 -- 172.21.15.123:0/1429788396 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f089804bed0 con 0x7f0898078d60 2024-03-30T04:17:51.967 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:51.340+0000 7f088ea00700 1 -- 172.21.15.123:0/1429788396 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v33)=0 set mgr/dashboard/cluster/status v33) v1 ==== 153+0+0 (secure 0 0 0) 0x7f0888018350 con 0x7f0898078d60 2024-03-30T04:17:51.967 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:51.342+0000 7f089da00700 1 -- 172.21.15.123:0/1429788396 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f087c0368d0 msgr2=0x7f087c038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:51.967 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:51.342+0000 7f089da00700 1 --2- 172.21.15.123:0/1429788396 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f087c0368d0 0x7f087c038d80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0884009990 tx=0x7f0884006e40 comp rx=0 tx=0).stop 2024-03-30T04:17:51.967 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:51.342+0000 7f089da00700 1 -- 172.21.15.123:0/1429788396 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0898078d60 msgr2=0x7f0898113630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:51.967 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:51.342+0000 7f089da00700 1 --2- 172.21.15.123:0/1429788396 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0898078d60 0x7f0898113630 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0888011ca0 tx=0x7f0888009250 comp rx=0 tx=0).stop 2024-03-30T04:17:51.967 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:51.343+0000 7f089da00700 1 -- 172.21.15.123:0/1429788396 shutdown_connections 2024-03-30T04:17:51.967 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:51.343+0000 7f089da00700 1 --2- 172.21.15.123:0/1429788396 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f087c0368d0 0x7f087c038d80 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:51.968 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:51.343+0000 7f089da00700 1 --2- 172.21.15.123:0/1429788396 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0898078d60 0x7f0898113630 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:51.968 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:51.343+0000 7f089da00700 1 -- 172.21.15.123:0/1429788396 >> 172.21.15.123:0/1429788396 conn(0x7f08980fdcd0 msgr2=0x7f08980fe8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:51.968 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:51.343+0000 7f089da00700 1 -- 172.21.15.123:0/1429788396 shutdown_connections 2024-03-30T04:17:51.968 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr 2024-03-30T04:17:51.343+0000 7f089da00700 1 -- 172.21.15.123:0/1429788396 wait complete. 2024-03-30T04:17:51.968 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-30T04:17:51.968 INFO:teuthology.orchestra.run.smithi123.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-30T04:17:51.968 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:17:51.968 INFO:teuthology.orchestra.run.smithi123.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-30T04:17:51.968 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:17:51.968 INFO:teuthology.orchestra.run.smithi123.stdout:Or, if you are only running a single cluster on this host: 2024-03-30T04:17:51.968 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:17:51.968 INFO:teuthology.orchestra.run.smithi123.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-30T04:17:51.968 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:17:51.968 INFO:teuthology.orchestra.run.smithi123.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-30T04:17:51.968 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:17:51.969 INFO:teuthology.orchestra.run.smithi123.stdout: ceph telemetry on 2024-03-30T04:17:51.969 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:17:51.969 INFO:teuthology.orchestra.run.smithi123.stdout:For more information see: 2024-03-30T04:17:51.969 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:17:51.969 INFO:teuthology.orchestra.run.smithi123.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-03-30T04:17:51.969 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:17:51.969 INFO:teuthology.orchestra.run.smithi123.stdout:Bootstrap complete. 2024-03-30T04:17:52.021 INFO:tasks.cephadm:Fetching config... 2024-03-30T04:17:52.021 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-30T04:17:52.021 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-30T04:17:52.067 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-30T04:17:52.067 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-30T04:17:52.067 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-30T04:17:52.151 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-30T04:17:52.151 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-30T04:17:52.151 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/keyring of=/dev/stdout 2024-03-30T04:17:52.258 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-30T04:17:52.258 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-30T04:17:52.258 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-30T04:17:52.341 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-30T04:17:52.342 DEBUG:teuthology.orchestra.run.smithi123:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6PBnH+eOJqjBedQ7yNS5ZfkelDzSWPbkRhZYIu9oKaG1M2NnJDxWlzLMa9TATuq1JjuxwBRXqVq07qf4ZRXYRZzRSYjk38FDbeoNR8HLjOZzf0G0PUbYTTT4xYIwgJ3dEFTSuleSUCH7fqORmKjpbyHEERS6ezUmwTZMZ/tlb1cWJJvc6nBkMp0XJQ4oyFJJTGyDQ1bLlwsfXBLS/lhfRoN/thTlsjyonNkF0KLXQ9xNewp8tmL0ICLt+t3o8AXdANVobEbknPd19bb1o8x+pv/u2ABWGVRAB8Vm75/vwVv94EvqbJKco2lgR1OzKf6WWP0ci3vjdF2+EDC6LIZ0iE5ENM5FNCpfNDW4ETLBUAECoDEIgCh5A3EvgcZiJdtpErf88b1eMoE7PMK+Zg7o+vW50pFOJrXqY4lQLSSnlPPVF4/lTVieZr0iF+T+0RGHtat9y9L7b6tO/yS0xOUB82169nc4Dr+/G1ARwTrESC6/nkDphI8JFjgyXg53cyIE= ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-30T04:17:52.483 INFO:teuthology.orchestra.run.smithi123.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6PBnH+eOJqjBedQ7yNS5ZfkelDzSWPbkRhZYIu9oKaG1M2NnJDxWlzLMa9TATuq1JjuxwBRXqVq07qf4ZRXYRZzRSYjk38FDbeoNR8HLjOZzf0G0PUbYTTT4xYIwgJ3dEFTSuleSUCH7fqORmKjpbyHEERS6ezUmwTZMZ/tlb1cWJJvc6nBkMp0XJQ4oyFJJTGyDQ1bLlwsfXBLS/lhfRoN/thTlsjyonNkF0KLXQ9xNewp8tmL0ICLt+t3o8AXdANVobEbknPd19bb1o8x+pv/u2ABWGVRAB8Vm75/vwVv94EvqbJKco2lgR1OzKf6WWP0ci3vjdF2+EDC6LIZ0iE5ENM5FNCpfNDW4ETLBUAECoDEIgCh5A3EvgcZiJdtpErf88b1eMoE7PMK+Zg7o+vW50pFOJrXqY4lQLSSnlPPVF4/lTVieZr0iF+T+0RGHtat9y9L7b6tO/yS0xOUB82169nc4Dr+/G1ARwTrESC6/nkDphI8JFjgyXg53cyIE= ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4 2024-03-30T04:17:52.502 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:52 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/1429788396' entity='client.admin' 2024-03-30T04:17:52.510 DEBUG:teuthology.orchestra.run.smithi135:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6PBnH+eOJqjBedQ7yNS5ZfkelDzSWPbkRhZYIu9oKaG1M2NnJDxWlzLMa9TATuq1JjuxwBRXqVq07qf4ZRXYRZzRSYjk38FDbeoNR8HLjOZzf0G0PUbYTTT4xYIwgJ3dEFTSuleSUCH7fqORmKjpbyHEERS6ezUmwTZMZ/tlb1cWJJvc6nBkMp0XJQ4oyFJJTGyDQ1bLlwsfXBLS/lhfRoN/thTlsjyonNkF0KLXQ9xNewp8tmL0ICLt+t3o8AXdANVobEbknPd19bb1o8x+pv/u2ABWGVRAB8Vm75/vwVv94EvqbJKco2lgR1OzKf6WWP0ci3vjdF2+EDC6LIZ0iE5ENM5FNCpfNDW4ETLBUAECoDEIgCh5A3EvgcZiJdtpErf88b1eMoE7PMK+Zg7o+vW50pFOJrXqY4lQLSSnlPPVF4/lTVieZr0iF+T+0RGHtat9y9L7b6tO/yS0xOUB82169nc4Dr+/G1ARwTrESC6/nkDphI8JFjgyXg53cyIE= ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-30T04:17:52.611 INFO:teuthology.orchestra.run.smithi135.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6PBnH+eOJqjBedQ7yNS5ZfkelDzSWPbkRhZYIu9oKaG1M2NnJDxWlzLMa9TATuq1JjuxwBRXqVq07qf4ZRXYRZzRSYjk38FDbeoNR8HLjOZzf0G0PUbYTTT4xYIwgJ3dEFTSuleSUCH7fqORmKjpbyHEERS6ezUmwTZMZ/tlb1cWJJvc6nBkMp0XJQ4oyFJJTGyDQ1bLlwsfXBLS/lhfRoN/thTlsjyonNkF0KLXQ9xNewp8tmL0ICLt+t3o8AXdANVobEbknPd19bb1o8x+pv/u2ABWGVRAB8Vm75/vwVv94EvqbJKco2lgR1OzKf6WWP0ci3vjdF2+EDC6LIZ0iE5ENM5FNCpfNDW4ETLBUAECoDEIgCh5A3EvgcZiJdtpErf88b1eMoE7PMK+Zg7o+vW50pFOJrXqY4lQLSSnlPPVF4/lTVieZr0iF+T+0RGHtat9y9L7b6tO/yS0xOUB82169nc4Dr+/G1ARwTrESC6/nkDphI8JFjgyXg53cyIE= ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4 2024-03-30T04:17:52.638 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-30T04:17:53.056 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:17:55.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:55.665+0000 7fe6e4c00700 1 -- 172.21.15.123:0/3382890744 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6e0071ea0 msgr2=0x7fe6e00703b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:55.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:55.665+0000 7fe6e4c00700 1 --2- 172.21.15.123:0/3382890744 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6e0071ea0 0x7fe6e00703b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe6d00048c0 tx=0x7fe6d0009e30 comp rx=0 tx=0).stop 2024-03-30T04:17:55.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:55.665+0000 7fe6e4c00700 1 -- 172.21.15.123:0/3382890744 shutdown_connections 2024-03-30T04:17:55.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:55.665+0000 7fe6e4c00700 1 --2- 172.21.15.123:0/3382890744 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6e0071ea0 0x7fe6e00703b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:55.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:55.665+0000 7fe6e4c00700 1 -- 172.21.15.123:0/3382890744 >> 172.21.15.123:0/3382890744 conn(0x7fe6e00f81d0 msgr2=0x7fe6e00fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:55.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:55.665+0000 7fe6e4c00700 1 -- 172.21.15.123:0/3382890744 shutdown_connections 2024-03-30T04:17:55.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:55.665+0000 7fe6e4c00700 1 -- 172.21.15.123:0/3382890744 wait complete. 2024-03-30T04:17:55.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:55.666+0000 7fe6e4c00700 1 Processor -- start 2024-03-30T04:17:55.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:55.666+0000 7fe6e4c00700 1 -- start start 2024-03-30T04:17:55.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:55.667+0000 7fe6e4c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6e0071ea0 0x7fe6e010f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:55.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:55.667+0000 7fe6e4c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e010f780 con 0x7fe6e0071ea0 2024-03-30T04:17:55.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:55.667+0000 7fe6dde00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6e0071ea0 0x7fe6e010f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:55.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:55.667+0000 7fe6dde00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6e0071ea0 0x7fe6e010f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53710/0 (socket says 172.21.15.123:53710) 2024-03-30T04:17:55.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:55.667+0000 7fe6dde00700 1 -- 172.21.15.123:0/604470292 learned_addr learned my addr 172.21.15.123:0/604470292 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:17:55.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:55.667+0000 7fe6dde00700 1 -- 172.21.15.123:0/604470292 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6d0005040 con 0x7fe6e0071ea0 2024-03-30T04:17:55.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:55.668+0000 7fe6dde00700 1 --2- 172.21.15.123:0/604470292 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6e0071ea0 0x7fe6e010f240 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe6d000b860 tx=0x7fe6d0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:55.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:55.668+0000 7fe6d6000700 1 -- 172.21.15.123:0/604470292 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6d0009580 con 0x7fe6e0071ea0 2024-03-30T04:17:55.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:55.668+0000 7fe6d6000700 1 -- 172.21.15.123:0/604470292 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe6d00096e0 con 0x7fe6e0071ea0 2024-03-30T04:17:55.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:55.668+0000 7fe6e4c00700 1 -- 172.21.15.123:0/604470292 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6e010f980 con 0x7fe6e0071ea0 2024-03-30T04:17:55.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:55.668+0000 7fe6d6000700 1 -- 172.21.15.123:0/604470292 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6d001c5b0 con 0x7fe6e0071ea0 2024-03-30T04:17:55.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:55.668+0000 7fe6e4c00700 1 -- 172.21.15.123:0/604470292 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6e01104f0 con 0x7fe6e0071ea0 2024-03-30T04:17:55.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:55.669+0000 7fe6e4c00700 1 -- 172.21.15.123:0/604470292 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6e0105050 con 0x7fe6e0071ea0 2024-03-30T04:17:55.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:55.669+0000 7fe6d6000700 1 -- 172.21.15.123:0/604470292 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7fe6d0013420 con 0x7fe6e0071ea0 2024-03-30T04:17:55.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:55.669+0000 7fe6d6000700 1 --2- 172.21.15.123:0/604470292 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7fe6c40368d0 0x7fe6c4038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:55.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:55.669+0000 7fe6d6000700 1 -- 172.21.15.123:0/604470292 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fe6d0049ac0 con 0x7fe6e0071ea0 2024-03-30T04:17:55.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:55.670+0000 7fe6dd400700 1 --2- 172.21.15.123:0/604470292 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7fe6c40368d0 0x7fe6c4038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:55.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:55.670+0000 7fe6dd400700 1 --2- 172.21.15.123:0/604470292 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7fe6c40368d0 0x7fe6c4038d80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc009990 tx=0x7fe6cc006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:55.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:55.674+0000 7fe6d6000700 1 -- 172.21.15.123:0/604470292 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fe6d0011550 con 0x7fe6e0071ea0 2024-03-30T04:17:55.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:55.880+0000 7fe6e4c00700 1 -- 172.21.15.123:0/604470292 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fe6e005f5f0 con 0x7fe6e0071ea0 2024-03-30T04:17:55.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:55.883+0000 7fe6d6000700 1 -- 172.21.15.123:0/604470292 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7fe6d0011370 con 0x7fe6e0071ea0 2024-03-30T04:17:55.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:55.886+0000 7fe6e4c00700 1 -- 172.21.15.123:0/604470292 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7fe6c40368d0 msgr2=0x7fe6c4038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:55.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:55.886+0000 7fe6e4c00700 1 --2- 172.21.15.123:0/604470292 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7fe6c40368d0 0x7fe6c4038d80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc009990 tx=0x7fe6cc006e40 comp rx=0 tx=0).stop 2024-03-30T04:17:55.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:55.886+0000 7fe6e4c00700 1 -- 172.21.15.123:0/604470292 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6e0071ea0 msgr2=0x7fe6e010f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:55.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:55.886+0000 7fe6e4c00700 1 --2- 172.21.15.123:0/604470292 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6e0071ea0 0x7fe6e010f240 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe6d000b860 tx=0x7fe6d0009e30 comp rx=0 tx=0).stop 2024-03-30T04:17:55.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:55.886+0000 7fe6e4c00700 1 -- 172.21.15.123:0/604470292 shutdown_connections 2024-03-30T04:17:55.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:55.886+0000 7fe6e4c00700 1 --2- 172.21.15.123:0/604470292 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7fe6c40368d0 0x7fe6c4038d80 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:55.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:55.886+0000 7fe6e4c00700 1 --2- 172.21.15.123:0/604470292 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6e0071ea0 0x7fe6e010f240 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:55.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:55.886+0000 7fe6e4c00700 1 -- 172.21.15.123:0/604470292 >> 172.21.15.123:0/604470292 conn(0x7fe6e00f81d0 msgr2=0x7fe6e00f8e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:55.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:55.886+0000 7fe6e4c00700 1 -- 172.21.15.123:0/604470292 shutdown_connections 2024-03-30T04:17:55.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:55.887+0000 7fe6e4c00700 1 -- 172.21.15.123:0/604470292 wait complete. 2024-03-30T04:17:56.605 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-30T04:17:56.606 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-30T04:17:57.204 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:56 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/604470292' entity='client.admin' 2024-03-30T04:17:57.509 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:17:58.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:58 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' 2024-03-30T04:17:58.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:58 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' 2024-03-30T04:17:58.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:58 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "who": "osd/host:smithi123", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:17:58.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:58 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' 2024-03-30T04:17:58.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:58 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi123", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-30T04:17:58.968 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:58 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi123", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-03-30T04:17:58.968 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:58 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:17:58.968 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:17:58 smithi123 ceph-mon[35245]: Deploying daemon crash.smithi123 on smithi123 2024-03-30T04:17:59.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:59.525+0000 7fde1e600700 1 -- 172.21.15.123:0/3883575064 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde180fca70 msgr2=0x7fde180fce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:59.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:59.525+0000 7fde1e600700 1 --2- 172.21.15.123:0/3883575064 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde180fca70 0x7fde180fce80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fde080048c0 tx=0x7fde08009e30 comp rx=0 tx=0).stop 2024-03-30T04:17:59.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:59.525+0000 7fde1e600700 1 -- 172.21.15.123:0/3883575064 shutdown_connections 2024-03-30T04:17:59.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:59.525+0000 7fde1e600700 1 --2- 172.21.15.123:0/3883575064 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde180fca70 0x7fde180fce80 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:59.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:59.525+0000 7fde1e600700 1 -- 172.21.15.123:0/3883575064 >> 172.21.15.123:0/3883575064 conn(0x7fde180f81d0 msgr2=0x7fde180fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:59.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:59.525+0000 7fde1e600700 1 -- 172.21.15.123:0/3883575064 shutdown_connections 2024-03-30T04:17:59.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:59.526+0000 7fde1e600700 1 -- 172.21.15.123:0/3883575064 wait complete. 2024-03-30T04:17:59.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:59.526+0000 7fde1e600700 1 Processor -- start 2024-03-30T04:17:59.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:59.526+0000 7fde1e600700 1 -- start start 2024-03-30T04:17:59.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:59.527+0000 7fde1e600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde180fca70 0x7fde1810f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:59.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:59.527+0000 7fde1e600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde1810f740 con 0x7fde180fca70 2024-03-30T04:17:59.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:59.527+0000 7fde17400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde180fca70 0x7fde1810f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:59.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:59.527+0000 7fde17400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde180fca70 0x7fde1810f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53738/0 (socket says 172.21.15.123:53738) 2024-03-30T04:17:59.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:59.527+0000 7fde17400700 1 -- 172.21.15.123:0/886119685 learned_addr learned my addr 172.21.15.123:0/886119685 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:17:59.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:59.527+0000 7fde17400700 1 -- 172.21.15.123:0/886119685 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde08005040 con 0x7fde180fca70 2024-03-30T04:17:59.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:59.528+0000 7fde17400700 1 --2- 172.21.15.123:0/886119685 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde180fca70 0x7fde1810f200 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fde08000c00 tx=0x7fde08009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:59.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:59.528+0000 7fde0f400700 1 -- 172.21.15.123:0/886119685 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde08009620 con 0x7fde180fca70 2024-03-30T04:17:59.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:59.528+0000 7fde0f400700 1 -- 172.21.15.123:0/886119685 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fde08009780 con 0x7fde180fca70 2024-03-30T04:17:59.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:59.528+0000 7fde0f400700 1 -- 172.21.15.123:0/886119685 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde0801c5b0 con 0x7fde180fca70 2024-03-30T04:17:59.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:59.528+0000 7fde1e600700 1 -- 172.21.15.123:0/886119685 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde1810f940 con 0x7fde180fca70 2024-03-30T04:17:59.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:59.528+0000 7fde1e600700 1 -- 172.21.15.123:0/886119685 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde181104b0 con 0x7fde180fca70 2024-03-30T04:17:59.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:59.529+0000 7fde1e600700 1 -- 172.21.15.123:0/886119685 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde18102e40 con 0x7fde180fca70 2024-03-30T04:17:59.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:59.529+0000 7fde0f400700 1 -- 172.21.15.123:0/886119685 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7fde08013a60 con 0x7fde180fca70 2024-03-30T04:17:59.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:59.529+0000 7fde0f400700 1 --2- 172.21.15.123:0/886119685 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7fddfc0368d0 0x7fddfc038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:59.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:59.529+0000 7fde0f400700 1 -- 172.21.15.123:0/886119685 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fde08049af0 con 0x7fde180fca70 2024-03-30T04:17:59.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:59.530+0000 7fde16a00700 1 --2- 172.21.15.123:0/886119685 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7fddfc0368d0 0x7fddfc038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:59.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:59.530+0000 7fde16a00700 1 --2- 172.21.15.123:0/886119685 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7fddfc0368d0 0x7fddfc038d80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fde04009990 tx=0x7fde04006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:59.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:59.534+0000 7fde0f400700 1 -- 172.21.15.123:0/886119685 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fde08025400 con 0x7fde180fca70 2024-03-30T04:17:59.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:59.738+0000 7fde1e600700 1 -- 172.21.15.123:0/886119685 --> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7fde18070770 con 0x7fddfc0368d0 2024-03-30T04:17:59.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:59.742+0000 7fde0f400700 1 -- 172.21.15.123:0/886119685 <== mgr.14160 v2:172.21.15.123:6800/70490328 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fde18070770 con 0x7fddfc0368d0 2024-03-30T04:17:59.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:59.744+0000 7fde1e600700 1 -- 172.21.15.123:0/886119685 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7fddfc0368d0 msgr2=0x7fddfc038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:59.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:59.744+0000 7fde1e600700 1 --2- 172.21.15.123:0/886119685 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7fddfc0368d0 0x7fddfc038d80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fde04009990 tx=0x7fde04006e40 comp rx=0 tx=0).stop 2024-03-30T04:17:59.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:59.744+0000 7fde1e600700 1 -- 172.21.15.123:0/886119685 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde180fca70 msgr2=0x7fde1810f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:59.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:59.744+0000 7fde1e600700 1 --2- 172.21.15.123:0/886119685 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde180fca70 0x7fde1810f200 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fde08000c00 tx=0x7fde08009e30 comp rx=0 tx=0).stop 2024-03-30T04:17:59.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:59.744+0000 7fde1e600700 1 -- 172.21.15.123:0/886119685 shutdown_connections 2024-03-30T04:17:59.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:59.744+0000 7fde1e600700 1 --2- 172.21.15.123:0/886119685 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7fddfc0368d0 0x7fddfc038d80 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:59.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:59.744+0000 7fde1e600700 1 --2- 172.21.15.123:0/886119685 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde180fca70 0x7fde1810f200 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:59.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:59.744+0000 7fde1e600700 1 -- 172.21.15.123:0/886119685 >> 172.21.15.123:0/886119685 conn(0x7fde180f81d0 msgr2=0x7fde180fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:59.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:59.744+0000 7fde1e600700 1 -- 172.21.15.123:0/886119685 shutdown_connections 2024-03-30T04:17:59.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:17:59.745+0000 7fde1e600700 1 -- 172.21.15.123:0/886119685 wait complete. 2024-03-30T04:18:00.937 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:00 smithi123 ceph-mon[35245]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:18:00.937 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:00 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' 2024-03-30T04:18:00.967 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi135 2024-03-30T04:18:00.968 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-30T04:18:00.968 DEBUG:teuthology.orchestra.run.smithi135:> dd of=/etc/ceph/ceph.conf 2024-03-30T04:18:01.017 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-30T04:18:01.017 DEBUG:teuthology.orchestra.run.smithi135:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-30T04:18:01.103 INFO:tasks.cephadm:Adding host smithi135 to orchestrator... 2024-03-30T04:18:01.103 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph orch host add smithi135 2024-03-30T04:18:01.700 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:18:02.825 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:02 smithi123 ceph-mon[35245]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:18:02.825 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:02 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' 2024-03-30T04:18:02.825 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:02 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' 2024-03-30T04:18:02.825 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:02 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' 2024-03-30T04:18:02.825 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:02 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' 2024-03-30T04:18:02.825 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:02 smithi123 ceph-mon[35245]: Deploying daemon node-exporter.smithi123 on smithi123 2024-03-30T04:18:03.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:03.777+0000 7efe77800700 1 -- 172.21.15.123:0/2994765734 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe700fd050 msgr2=0x7efe700fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:18:03.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:03.777+0000 7efe77800700 1 --2- 172.21.15.123:0/2994765734 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe700fd050 0x7efe700fd460 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7efe640048c0 tx=0x7efe64009e30 comp rx=0 tx=0).stop 2024-03-30T04:18:03.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:03.777+0000 7efe77800700 1 -- 172.21.15.123:0/2994765734 shutdown_connections 2024-03-30T04:18:03.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:03.777+0000 7efe77800700 1 --2- 172.21.15.123:0/2994765734 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe700fd050 0x7efe700fd460 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:18:03.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:03.777+0000 7efe77800700 1 -- 172.21.15.123:0/2994765734 >> 172.21.15.123:0/2994765734 conn(0x7efe700f70d0 msgr2=0x7efe700f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:18:03.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:03.777+0000 7efe77800700 1 -- 172.21.15.123:0/2994765734 shutdown_connections 2024-03-30T04:18:03.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:03.777+0000 7efe77800700 1 -- 172.21.15.123:0/2994765734 wait complete. 2024-03-30T04:18:03.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:03.778+0000 7efe77800700 1 Processor -- start 2024-03-30T04:18:03.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:03.778+0000 7efe77800700 1 -- start start 2024-03-30T04:18:03.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:03.778+0000 7efe77800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe700fd050 0x7efe7010f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:18:03.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:03.778+0000 7efe77800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe7010f700 con 0x7efe700fd050 2024-03-30T04:18:03.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:03.778+0000 7efe74e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe700fd050 0x7efe7010f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:18:03.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:03.778+0000 7efe74e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe700fd050 0x7efe7010f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53772/0 (socket says 172.21.15.123:53772) 2024-03-30T04:18:03.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:03.778+0000 7efe74e00700 1 -- 172.21.15.123:0/1726755613 learned_addr learned my addr 172.21.15.123:0/1726755613 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:18:03.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:03.779+0000 7efe74e00700 1 -- 172.21.15.123:0/1726755613 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe64005040 con 0x7efe700fd050 2024-03-30T04:18:03.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:03.779+0000 7efe74e00700 1 --2- 172.21.15.123:0/1726755613 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe700fd050 0x7efe7010f1c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efe64000c00 tx=0x7efe64009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:18:03.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:03.779+0000 7efe6cc00700 1 -- 172.21.15.123:0/1726755613 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe64009620 con 0x7efe700fd050 2024-03-30T04:18:03.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:03.779+0000 7efe6cc00700 1 -- 172.21.15.123:0/1726755613 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efe64009780 con 0x7efe700fd050 2024-03-30T04:18:03.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:03.779+0000 7efe77800700 1 -- 172.21.15.123:0/1726755613 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe7010f900 con 0x7efe700fd050 2024-03-30T04:18:03.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:03.779+0000 7efe6cc00700 1 -- 172.21.15.123:0/1726755613 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe6401c5b0 con 0x7efe700fd050 2024-03-30T04:18:03.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:03.779+0000 7efe77800700 1 -- 172.21.15.123:0/1726755613 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe70110470 con 0x7efe700fd050 2024-03-30T04:18:03.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:03.780+0000 7efe6cc00700 1 -- 172.21.15.123:0/1726755613 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7efe64013a60 con 0x7efe700fd050 2024-03-30T04:18:03.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:03.780+0000 7efe77800700 1 -- 172.21.15.123:0/1726755613 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe70102e50 con 0x7efe700fd050 2024-03-30T04:18:03.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:03.781+0000 7efe6cc00700 1 --2- 172.21.15.123:0/1726755613 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7efe580368d0 0x7efe58038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:18:03.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:03.781+0000 7efe6cc00700 1 -- 172.21.15.123:0/1726755613 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7efe64049aa0 con 0x7efe700fd050 2024-03-30T04:18:03.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:03.781+0000 7efe6fe00700 1 --2- 172.21.15.123:0/1726755613 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7efe580368d0 0x7efe58038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:18:03.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:03.782+0000 7efe6fe00700 1 --2- 172.21.15.123:0/1726755613 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7efe580368d0 0x7efe58038d80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7efe5c009990 tx=0x7efe5c006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:18:03.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:03.786+0000 7efe6cc00700 1 -- 172.21.15.123:0/1726755613 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7efe64025400 con 0x7efe700fd050 2024-03-30T04:18:03.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:03.987+0000 7efe77800700 1 -- 172.21.15.123:0/1726755613 --> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi135", "target": ["mon-mgr", ""]}) v1 -- 0x7efe70101460 con 0x7efe580368d0 2024-03-30T04:18:04.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:04 smithi123 ceph-mon[35245]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:18:04.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:04 smithi123 ceph-mon[35245]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi135", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:18:05.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:05 smithi123 ceph-mon[35245]: Deploying cephadm binary to smithi135 2024-03-30T04:18:06.624 INFO:teuthology.orchestra.run.smithi123.stdout:Added host 'smithi135' with addr '172.21.15.135' 2024-03-30T04:18:06.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:06.625+0000 7efe6cc00700 1 -- 172.21.15.123:0/1726755613 <== mgr.14160 v2:172.21.15.123:6800/70490328 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7efe70101460 con 0x7efe580368d0 2024-03-30T04:18:06.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:06.627+0000 7efe77800700 1 -- 172.21.15.123:0/1726755613 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7efe580368d0 msgr2=0x7efe58038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:18:06.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:06.627+0000 7efe77800700 1 --2- 172.21.15.123:0/1726755613 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7efe580368d0 0x7efe58038d80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7efe5c009990 tx=0x7efe5c006e40 comp rx=0 tx=0).stop 2024-03-30T04:18:06.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:06.627+0000 7efe77800700 1 -- 172.21.15.123:0/1726755613 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe700fd050 msgr2=0x7efe7010f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:18:06.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:06.627+0000 7efe77800700 1 --2- 172.21.15.123:0/1726755613 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe700fd050 0x7efe7010f1c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efe64000c00 tx=0x7efe64009e30 comp rx=0 tx=0).stop 2024-03-30T04:18:06.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:06.628+0000 7efe77800700 1 -- 172.21.15.123:0/1726755613 shutdown_connections 2024-03-30T04:18:06.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:06.628+0000 7efe77800700 1 --2- 172.21.15.123:0/1726755613 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7efe580368d0 0x7efe58038d80 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:18:06.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:06.628+0000 7efe77800700 1 --2- 172.21.15.123:0/1726755613 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe700fd050 0x7efe7010f1c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:18:06.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:06.628+0000 7efe77800700 1 -- 172.21.15.123:0/1726755613 >> 172.21.15.123:0/1726755613 conn(0x7efe700f70d0 msgr2=0x7efe700f7cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:18:06.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:06.628+0000 7efe77800700 1 -- 172.21.15.123:0/1726755613 shutdown_connections 2024-03-30T04:18:06.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:06.628+0000 7efe77800700 1 -- 172.21.15.123:0/1726755613 wait complete. 2024-03-30T04:18:06.927 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:06 smithi123 ceph-mon[35245]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:18:06.927 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:06 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' 2024-03-30T04:18:06.927 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:06 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' 2024-03-30T04:18:06.927 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:06 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' 2024-03-30T04:18:06.927 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:06 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' 2024-03-30T04:18:06.927 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:06 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' 2024-03-30T04:18:06.927 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:06 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' 2024-03-30T04:18:07.239 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph orch host ls --format=json 2024-03-30T04:18:07.663 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:18:07.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:07 smithi123 ceph-mon[35245]: Deploying daemon alertmanager.smithi123 on smithi123 2024-03-30T04:18:07.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:07 smithi123 ceph-mon[35245]: Added host smithi135 2024-03-30T04:18:08.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:08 smithi123 ceph-mon[35245]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:18:10.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:10.516+0000 7ff44d000700 1 -- 172.21.15.123:0/2520228319 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4480fca70 msgr2=0x7ff4480fce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:18:10.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:10.516+0000 7ff44d000700 1 --2- 172.21.15.123:0/2520228319 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4480fca70 0x7ff4480fce80 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff4380048c0 tx=0x7ff438009e30 comp rx=0 tx=0).stop 2024-03-30T04:18:10.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:10.516+0000 7ff44d000700 1 -- 172.21.15.123:0/2520228319 shutdown_connections 2024-03-30T04:18:10.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:10.516+0000 7ff44d000700 1 --2- 172.21.15.123:0/2520228319 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4480fca70 0x7ff4480fce80 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:18:10.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:10.516+0000 7ff44d000700 1 -- 172.21.15.123:0/2520228319 >> 172.21.15.123:0/2520228319 conn(0x7ff4480f81d0 msgr2=0x7ff4480fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:18:10.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:10.516+0000 7ff44d000700 1 -- 172.21.15.123:0/2520228319 shutdown_connections 2024-03-30T04:18:10.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:10.516+0000 7ff44d000700 1 -- 172.21.15.123:0/2520228319 wait complete. 2024-03-30T04:18:10.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:10.517+0000 7ff44d000700 1 Processor -- start 2024-03-30T04:18:10.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:10.518+0000 7ff44d000700 1 -- start start 2024-03-30T04:18:10.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:10.518+0000 7ff44d000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4480fca70 0x7ff44810f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:18:10.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:10.518+0000 7ff44d000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff44810f740 con 0x7ff4480fca70 2024-03-30T04:18:10.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:10.518+0000 7ff445e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4480fca70 0x7ff44810f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:18:10.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:10.518+0000 7ff445e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4480fca70 0x7ff44810f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35348/0 (socket says 172.21.15.123:35348) 2024-03-30T04:18:10.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:10.518+0000 7ff445e00700 1 -- 172.21.15.123:0/916951378 learned_addr learned my addr 172.21.15.123:0/916951378 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:18:10.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:10.519+0000 7ff445e00700 1 -- 172.21.15.123:0/916951378 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff438005040 con 0x7ff4480fca70 2024-03-30T04:18:10.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:10.519+0000 7ff445e00700 1 --2- 172.21.15.123:0/916951378 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4480fca70 0x7ff44810f200 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff438000c00 tx=0x7ff438009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:18:10.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:10.519+0000 7ff43e000700 1 -- 172.21.15.123:0/916951378 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff438009620 con 0x7ff4480fca70 2024-03-30T04:18:10.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:10.519+0000 7ff43e000700 1 -- 172.21.15.123:0/916951378 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff438009780 con 0x7ff4480fca70 2024-03-30T04:18:10.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:10.519+0000 7ff43e000700 1 -- 172.21.15.123:0/916951378 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff43801c5b0 con 0x7ff4480fca70 2024-03-30T04:18:10.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:10.519+0000 7ff44d000700 1 -- 172.21.15.123:0/916951378 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff44810f940 con 0x7ff4480fca70 2024-03-30T04:18:10.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:10.519+0000 7ff44d000700 1 -- 172.21.15.123:0/916951378 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4481104b0 con 0x7ff4480fca70 2024-03-30T04:18:10.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:10.520+0000 7ff43e000700 1 -- 172.21.15.123:0/916951378 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7ff438013a60 con 0x7ff4480fca70 2024-03-30T04:18:10.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:10.521+0000 7ff44d000700 1 -- 172.21.15.123:0/916951378 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff448102e40 con 0x7ff4480fca70 2024-03-30T04:18:10.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:10.521+0000 7ff43e000700 1 --2- 172.21.15.123:0/916951378 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7ff42c036880 0x7ff42c038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:18:10.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:10.521+0000 7ff43e000700 1 -- 172.21.15.123:0/916951378 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7ff438049b40 con 0x7ff4480fca70 2024-03-30T04:18:10.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:10.521+0000 7ff445400700 1 --2- 172.21.15.123:0/916951378 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7ff42c036880 0x7ff42c038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:18:10.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:10.522+0000 7ff445400700 1 --2- 172.21.15.123:0/916951378 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7ff42c036880 0x7ff42c038d30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff434009990 tx=0x7ff434006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:18:10.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:10.529+0000 7ff43e000700 1 -- 172.21.15.123:0/916951378 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7ff438025400 con 0x7ff4480fca70 2024-03-30T04:18:10.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:10.775+0000 7ff44d000700 1 -- 172.21.15.123:0/916951378 --> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff448070750 con 0x7ff42c036880 2024-03-30T04:18:10.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:10.777+0000 7ff43e000700 1 -- 172.21.15.123:0/916951378 <== mgr.14160 v2:172.21.15.123:6800/70490328 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+161 (secure 0 0 0) 0x7ff448070750 con 0x7ff42c036880 2024-03-30T04:18:10.776 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:18:10.776 INFO:teuthology.orchestra.run.smithi123.stdout:[{"addr": "172.21.15.123", "hostname": "smithi123", "labels": [], "status": ""}, {"addr": "172.21.15.135", "hostname": "smithi135", "labels": [], "status": ""}] 2024-03-30T04:18:10.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:10.779+0000 7ff44d000700 1 -- 172.21.15.123:0/916951378 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7ff42c036880 msgr2=0x7ff42c038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:18:10.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:10.779+0000 7ff44d000700 1 --2- 172.21.15.123:0/916951378 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7ff42c036880 0x7ff42c038d30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff434009990 tx=0x7ff434006e40 comp rx=0 tx=0).stop 2024-03-30T04:18:10.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:10.779+0000 7ff44d000700 1 -- 172.21.15.123:0/916951378 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4480fca70 msgr2=0x7ff44810f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:18:10.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:10.779+0000 7ff44d000700 1 --2- 172.21.15.123:0/916951378 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4480fca70 0x7ff44810f200 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff438000c00 tx=0x7ff438009e30 comp rx=0 tx=0).stop 2024-03-30T04:18:10.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:10.779+0000 7ff44d000700 1 -- 172.21.15.123:0/916951378 shutdown_connections 2024-03-30T04:18:10.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:10.779+0000 7ff44d000700 1 --2- 172.21.15.123:0/916951378 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7ff42c036880 0x7ff42c038d30 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:18:10.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:10.779+0000 7ff44d000700 1 --2- 172.21.15.123:0/916951378 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4480fca70 0x7ff44810f200 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:18:10.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:10.779+0000 7ff44d000700 1 -- 172.21.15.123:0/916951378 >> 172.21.15.123:0/916951378 conn(0x7ff4480f81d0 msgr2=0x7ff4480fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:18:10.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:10.779+0000 7ff44d000700 1 -- 172.21.15.123:0/916951378 shutdown_connections 2024-03-30T04:18:10.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:10.780+0000 7ff44d000700 1 -- 172.21.15.123:0/916951378 wait complete. 2024-03-30T04:18:10.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:10 smithi123 ceph-mon[35245]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:18:11.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:11 smithi123 ceph-mon[35245]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T04:18:12.474 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-30T04:18:12.475 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph osd crush tunables default 2024-03-30T04:18:12.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:12 smithi123 ceph-mon[35245]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:18:14.026 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:18:14.939 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:14 smithi123 ceph-mon[35245]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:18:16.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:16 smithi123 ceph-mon[35245]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:18:16.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:16.986+0000 7f5e79400700 1 -- 172.21.15.123:0/1360934640 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e740fca70 msgr2=0x7f5e740fce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:18:16.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:16.986+0000 7f5e79400700 1 --2- 172.21.15.123:0/1360934640 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e740fca70 0x7f5e740fce80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5e640048c0 tx=0x7f5e64009e30 comp rx=0 tx=0).stop 2024-03-30T04:18:16.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:16.987+0000 7f5e79400700 1 -- 172.21.15.123:0/1360934640 shutdown_connections 2024-03-30T04:18:16.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:16.987+0000 7f5e79400700 1 --2- 172.21.15.123:0/1360934640 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e740fca70 0x7f5e740fce80 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:18:16.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:16.987+0000 7f5e79400700 1 -- 172.21.15.123:0/1360934640 >> 172.21.15.123:0/1360934640 conn(0x7f5e740f81d0 msgr2=0x7f5e740fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:18:16.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:16.987+0000 7f5e79400700 1 -- 172.21.15.123:0/1360934640 shutdown_connections 2024-03-30T04:18:16.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:16.987+0000 7f5e79400700 1 -- 172.21.15.123:0/1360934640 wait complete. 2024-03-30T04:18:16.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:16.988+0000 7f5e79400700 1 Processor -- start 2024-03-30T04:18:16.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:16.988+0000 7f5e79400700 1 -- start start 2024-03-30T04:18:16.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:16.988+0000 7f5e79400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e740fca70 0x7f5e74071d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:18:16.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:16.988+0000 7f5e79400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e740722a0 con 0x7f5e740fca70 2024-03-30T04:18:16.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:16.989+0000 7f5e72800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e740fca70 0x7f5e74071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:18:16.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:16.989+0000 7f5e72800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e740fca70 0x7f5e74071d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53898/0 (socket says 172.21.15.123:53898) 2024-03-30T04:18:16.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:16.989+0000 7f5e72800700 1 -- 172.21.15.123:0/1309856841 learned_addr learned my addr 172.21.15.123:0/1309856841 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:18:16.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:16.989+0000 7f5e72800700 1 -- 172.21.15.123:0/1309856841 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e64005040 con 0x7f5e740fca70 2024-03-30T04:18:16.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:16.989+0000 7f5e72800700 1 --2- 172.21.15.123:0/1309856841 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e740fca70 0x7f5e74071d60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5e64000c00 tx=0x7f5e64009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:18:16.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:16.990+0000 7f5e6aa00700 1 -- 172.21.15.123:0/1309856841 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e64009620 con 0x7f5e740fca70 2024-03-30T04:18:16.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:16.990+0000 7f5e6aa00700 1 -- 172.21.15.123:0/1309856841 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5e64009780 con 0x7f5e740fca70 2024-03-30T04:18:16.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:16.990+0000 7f5e6aa00700 1 -- 172.21.15.123:0/1309856841 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e6401c5b0 con 0x7f5e740fca70 2024-03-30T04:18:16.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:16.990+0000 7f5e79400700 1 -- 172.21.15.123:0/1309856841 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e74070370 con 0x7f5e740fca70 2024-03-30T04:18:16.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:16.990+0000 7f5e79400700 1 -- 172.21.15.123:0/1309856841 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e74070860 con 0x7f5e740fca70 2024-03-30T04:18:16.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:16.991+0000 7f5e79400700 1 -- 172.21.15.123:0/1309856841 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e7405f5f0 con 0x7f5e740fca70 2024-03-30T04:18:16.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:16.991+0000 7f5e6aa00700 1 -- 172.21.15.123:0/1309856841 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f5e64013a60 con 0x7f5e740fca70 2024-03-30T04:18:16.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:16.992+0000 7f5e6aa00700 1 --2- 172.21.15.123:0/1309856841 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f5e580368d0 0x7f5e58038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:18:16.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:16.992+0000 7f5e6aa00700 1 -- 172.21.15.123:0/1309856841 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f5e64049ab0 con 0x7f5e740fca70 2024-03-30T04:18:16.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:16.992+0000 7f5e71e00700 1 --2- 172.21.15.123:0/1309856841 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f5e580368d0 0x7f5e58038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:18:16.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:16.993+0000 7f5e71e00700 1 --2- 172.21.15.123:0/1309856841 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f5e580368d0 0x7f5e58038d80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5e60009990 tx=0x7f5e60006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:18:17.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:17.001+0000 7f5e6aa00700 1 -- 172.21.15.123:0/1309856841 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f5e64025400 con 0x7f5e740fca70 2024-03-30T04:18:17.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:17.226+0000 7f5e79400700 1 -- 172.21.15.123:0/1309856841 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f5e7404bed0 con 0x7f5e740fca70 2024-03-30T04:18:17.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:17.719+0000 7f5e6aa00700 1 -- 172.21.15.123:0/1309856841 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f5e64011b60 con 0x7f5e740fca70 2024-03-30T04:18:17.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:17.722+0000 7f5e79400700 1 -- 172.21.15.123:0/1309856841 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f5e580368d0 msgr2=0x7f5e58038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:18:17.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:17.722+0000 7f5e79400700 1 --2- 172.21.15.123:0/1309856841 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f5e580368d0 0x7f5e58038d80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5e60009990 tx=0x7f5e60006e40 comp rx=0 tx=0).stop 2024-03-30T04:18:17.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:17.722+0000 7f5e79400700 1 -- 172.21.15.123:0/1309856841 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e740fca70 msgr2=0x7f5e74071d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:18:17.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:17.722+0000 7f5e79400700 1 --2- 172.21.15.123:0/1309856841 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e740fca70 0x7f5e74071d60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5e64000c00 tx=0x7f5e64009e30 comp rx=0 tx=0).stop 2024-03-30T04:18:17.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:17.722+0000 7f5e79400700 1 -- 172.21.15.123:0/1309856841 shutdown_connections 2024-03-30T04:18:17.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:17.722+0000 7f5e79400700 1 --2- 172.21.15.123:0/1309856841 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f5e580368d0 0x7f5e58038d80 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:18:17.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:17.722+0000 7f5e79400700 1 --2- 172.21.15.123:0/1309856841 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e740fca70 0x7f5e74071d60 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:18:17.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:17.722+0000 7f5e79400700 1 -- 172.21.15.123:0/1309856841 >> 172.21.15.123:0/1309856841 conn(0x7f5e740f81d0 msgr2=0x7f5e740f8a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:18:17.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:17.722+0000 7f5e79400700 1 -- 172.21.15.123:0/1309856841 shutdown_connections 2024-03-30T04:18:17.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:18:17.722+0000 7f5e79400700 1 -- 172.21.15.123:0/1309856841 wait complete. 2024-03-30T04:18:17.723 INFO:teuthology.orchestra.run.smithi123.stderr:adjusted tunables profile to default 2024-03-30T04:18:17.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:17 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/1309856841' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-30T04:18:18.512 INFO:tasks.cephadm:Adding mon.smithi123 on smithi123 2024-03-30T04:18:18.512 INFO:tasks.cephadm:Adding mon.smithi135 on smithi135 2024-03-30T04:18:18.512 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph orch apply mon '2;smithi123:172.21.15.123=smithi123;smithi135:172.21.15.135=smithi135' 2024-03-30T04:18:18.914 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-30T04:18:18.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:18 smithi123 ceph-mon[35245]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:18:18.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:18 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/1309856841' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-30T04:18:18.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:18 smithi123 ceph-mon[35245]: osdmap e4: 0 total, 0 up, 0 in 2024-03-30T04:18:18.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:18 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' 2024-03-30T04:18:18.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:18 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' 2024-03-30T04:18:18.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:18 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' 2024-03-30T04:18:18.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:18 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' 2024-03-30T04:18:18.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:18 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' 2024-03-30T04:18:18.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:18 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' 2024-03-30T04:18:18.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:18 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-30T04:18:18.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:18 smithi123 ceph-mon[35245]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-30T04:18:18.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:18 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' 2024-03-30T04:18:18.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:18 smithi123 ceph-mon[35245]: Deploying daemon grafana.smithi123 on smithi123 2024-03-30T04:18:19.001 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-30T04:18:20.851 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:20.852+0000 7f5239800700 1 -- 172.21.15.135:0/1441219789 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52340fca70 msgr2=0x7f52340fce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:18:20.851 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:20.852+0000 7f5239800700 1 --2- 172.21.15.135:0/1441219789 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52340fca70 0x7f52340fce80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f52240048c0 tx=0x7f5224009e30 comp rx=0 tx=0).stop 2024-03-30T04:18:20.851 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:20.852+0000 7f5239800700 1 -- 172.21.15.135:0/1441219789 shutdown_connections 2024-03-30T04:18:20.851 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:20.852+0000 7f5239800700 1 --2- 172.21.15.135:0/1441219789 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52340fca70 0x7f52340fce80 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:18:20.851 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:20.852+0000 7f5239800700 1 -- 172.21.15.135:0/1441219789 >> 172.21.15.135:0/1441219789 conn(0x7f52340f81d0 msgr2=0x7f52340fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:18:20.851 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:20.852+0000 7f5239800700 1 -- 172.21.15.135:0/1441219789 shutdown_connections 2024-03-30T04:18:20.851 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:20.852+0000 7f5239800700 1 -- 172.21.15.135:0/1441219789 wait complete. 2024-03-30T04:18:20.852 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:20.853+0000 7f5239800700 1 Processor -- start 2024-03-30T04:18:20.852 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:20.853+0000 7f5239800700 1 -- start start 2024-03-30T04:18:20.852 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:20.854+0000 7f5239800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52340fca70 0x7f523410cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:18:20.853 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:20.854+0000 7f5239800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f523410d510 con 0x7f52340fca70 2024-03-30T04:18:20.853 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:20.854+0000 7f5232a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52340fca70 0x7f523410cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:18:20.853 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:20.854+0000 7f5232a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52340fca70 0x7f523410cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:48202/0 (socket says 172.21.15.135:48202) 2024-03-30T04:18:20.853 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:20.854+0000 7f5232a00700 1 -- 172.21.15.135:0/1047050483 learned_addr learned my addr 172.21.15.135:0/1047050483 (peer_addr_for_me v2:172.21.15.135:0/0) 2024-03-30T04:18:20.853 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:20.855+0000 7f5232a00700 1 -- 172.21.15.135:0/1047050483 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5224005040 con 0x7f52340fca70 2024-03-30T04:18:20.854 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:20.855+0000 7f5232a00700 1 --2- 172.21.15.135:0/1047050483 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52340fca70 0x7f523410cfd0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5224000c00 tx=0x7f5224009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:18:20.854 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:20.855+0000 7f522aa00700 1 -- 172.21.15.135:0/1047050483 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5224009620 con 0x7f52340fca70 2024-03-30T04:18:20.855 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:20.856+0000 7f522aa00700 1 -- 172.21.15.135:0/1047050483 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5224009780 con 0x7f52340fca70 2024-03-30T04:18:20.855 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:20.856+0000 7f5239800700 1 -- 172.21.15.135:0/1047050483 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f523410d710 con 0x7f52340fca70 2024-03-30T04:18:20.855 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:20.856+0000 7f522aa00700 1 -- 172.21.15.135:0/1047050483 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f522401c5b0 con 0x7f52340fca70 2024-03-30T04:18:20.855 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:20.856+0000 7f5239800700 1 -- 172.21.15.135:0/1047050483 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f523410e280 con 0x7f52340fca70 2024-03-30T04:18:20.855 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:20.857+0000 7f522aa00700 1 -- 172.21.15.135:0/1047050483 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f5224013420 con 0x7f52340fca70 2024-03-30T04:18:20.856 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:20.857+0000 7f5239800700 1 -- 172.21.15.135:0/1047050483 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5234102e40 con 0x7f52340fca70 2024-03-30T04:18:20.856 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:20.857+0000 7f522aa00700 1 --2- 172.21.15.135:0/1047050483 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f52180368d0 0x7f5218038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:18:20.856 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:20.857+0000 7f522aa00700 1 -- 172.21.15.135:0/1047050483 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f5224049b20 con 0x7f52340fca70 2024-03-30T04:18:20.856 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:20.857+0000 7f5232000700 1 --2- 172.21.15.135:0/1047050483 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f52180368d0 0x7f5218038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:18:20.857 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:20.858+0000 7f5232000700 1 --2- 172.21.15.135:0/1047050483 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f52180368d0 0x7f5218038d80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5220009990 tx=0x7f5220006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:18:20.863 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:20.865+0000 7f522aa00700 1 -- 172.21.15.135:0/1047050483 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f5224025400 con 0x7f52340fca70 2024-03-30T04:18:21.066 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:21.067+0000 7f5239800700 1 -- 172.21.15.135:0/1047050483 --> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi123:172.21.15.123=smithi123;smithi135:172.21.15.135=smithi135", "target": ["mon-mgr", ""]}) v1 -- 0x7f523410e580 con 0x7f52180368d0 2024-03-30T04:18:21.075 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:21.077+0000 7f522aa00700 1 -- 172.21.15.135:0/1047050483 <== mgr.14160 v2:172.21.15.123:6800/70490328 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f523410e580 con 0x7f52180368d0 2024-03-30T04:18:21.075 INFO:teuthology.orchestra.run.smithi135.stdout:Scheduled mon update... 2024-03-30T04:18:21.077 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:21.078+0000 7f5239800700 1 -- 172.21.15.135:0/1047050483 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f52180368d0 msgr2=0x7f5218038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:18:21.077 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:21.078+0000 7f5239800700 1 --2- 172.21.15.135:0/1047050483 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f52180368d0 0x7f5218038d80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5220009990 tx=0x7f5220006e40 comp rx=0 tx=0).stop 2024-03-30T04:18:21.077 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:21.078+0000 7f5239800700 1 -- 172.21.15.135:0/1047050483 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52340fca70 msgr2=0x7f523410cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:18:21.077 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:21.078+0000 7f5239800700 1 --2- 172.21.15.135:0/1047050483 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52340fca70 0x7f523410cfd0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5224000c00 tx=0x7f5224009e30 comp rx=0 tx=0).stop 2024-03-30T04:18:21.078 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:21.078+0000 7f5239800700 1 -- 172.21.15.135:0/1047050483 shutdown_connections 2024-03-30T04:18:21.078 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:21.078+0000 7f5239800700 1 --2- 172.21.15.135:0/1047050483 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f52180368d0 0x7f5218038d80 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:18:21.078 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:21.078+0000 7f5239800700 1 --2- 172.21.15.135:0/1047050483 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52340fca70 0x7f523410cfd0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:18:21.078 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:21.078+0000 7f5239800700 1 -- 172.21.15.135:0/1047050483 >> 172.21.15.135:0/1047050483 conn(0x7f52340f81d0 msgr2=0x7f52340f8a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:18:21.078 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:21.078+0000 7f5239800700 1 -- 172.21.15.135:0/1047050483 shutdown_connections 2024-03-30T04:18:21.078 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:21.079+0000 7f5239800700 1 -- 172.21.15.135:0/1047050483 wait complete. 2024-03-30T04:18:21.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:20 smithi123 ceph-mon[35245]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:18:21.751 DEBUG:teuthology.orchestra.run.smithi135:mon.smithi135> sudo journalctl -f -n 0 -u ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4@mon.smithi135.service 2024-03-30T04:18:21.755 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-30T04:18:21.755 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-03-30T04:18:21.823 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:-- Logs begin at Sat 2024-03-30 04:03:58 UTC. -- 2024-03-30T04:18:22.187 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-30T04:18:22.275 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-30T04:18:22.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:22 smithi123 ceph-mon[35245]: from='client.14192 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi123:172.21.15.123=smithi123;smithi135:172.21.15.135=smithi135", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:18:22.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:22 smithi123 ceph-mon[35245]: Saving service mon spec with placement smithi123:172.21.15.123=smithi123;smithi135:172.21.15.135=smithi135;count:2 2024-03-30T04:18:22.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:22 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' 2024-03-30T04:18:22.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:22 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' 2024-03-30T04:18:23.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:23 smithi123 ceph-mon[35245]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:18:24.064 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:24.064+0000 7fe912400700 1 -- 172.21.15.135:0/3164372208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe90c0fca70 msgr2=0x7fe90c0fce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:18:24.064 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:24.064+0000 7fe912400700 1 --2- 172.21.15.135:0/3164372208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe90c0fca70 0x7fe90c0fce80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc0048c0 tx=0x7fe8fc009e30 comp rx=0 tx=0).stop 2024-03-30T04:18:24.064 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:24.064+0000 7fe912400700 1 -- 172.21.15.135:0/3164372208 shutdown_connections 2024-03-30T04:18:24.064 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:24.064+0000 7fe912400700 1 --2- 172.21.15.135:0/3164372208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe90c0fca70 0x7fe90c0fce80 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:18:24.064 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:24.064+0000 7fe912400700 1 -- 172.21.15.135:0/3164372208 >> 172.21.15.135:0/3164372208 conn(0x7fe90c0f81d0 msgr2=0x7fe90c0fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:18:24.064 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:24.064+0000 7fe912400700 1 -- 172.21.15.135:0/3164372208 shutdown_connections 2024-03-30T04:18:24.064 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:24.064+0000 7fe912400700 1 -- 172.21.15.135:0/3164372208 wait complete. 2024-03-30T04:18:24.064 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:24.065+0000 7fe912400700 1 Processor -- start 2024-03-30T04:18:24.064 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:24.065+0000 7fe912400700 1 -- start start 2024-03-30T04:18:24.064 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:24.065+0000 7fe912400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe90c0fca70 0x7fe90c10f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:18:24.065 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:24.065+0000 7fe912400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe90c10f720 con 0x7fe90c0fca70 2024-03-30T04:18:24.066 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:24.065+0000 7fe90b400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe90c0fca70 0x7fe90c10f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:18:24.066 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:24.065+0000 7fe90b400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe90c0fca70 0x7fe90c10f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:48220/0 (socket says 172.21.15.135:48220) 2024-03-30T04:18:24.066 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:24.065+0000 7fe90b400700 1 -- 172.21.15.135:0/1158710265 learned_addr learned my addr 172.21.15.135:0/1158710265 (peer_addr_for_me v2:172.21.15.135:0/0) 2024-03-30T04:18:24.066 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:24.066+0000 7fe90b400700 1 -- 172.21.15.135:0/1158710265 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8fc005040 con 0x7fe90c0fca70 2024-03-30T04:18:24.066 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:24.066+0000 7fe90b400700 1 --2- 172.21.15.135:0/1158710265 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe90c0fca70 0x7fe90c10f1e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc000c00 tx=0x7fe8fc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:18:24.066 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:24.066+0000 7fe903400700 1 -- 172.21.15.135:0/1158710265 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8fc009620 con 0x7fe90c0fca70 2024-03-30T04:18:24.066 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:24.066+0000 7fe903400700 1 -- 172.21.15.135:0/1158710265 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe8fc009780 con 0x7fe90c0fca70 2024-03-30T04:18:24.066 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:24.066+0000 7fe903400700 1 -- 172.21.15.135:0/1158710265 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8fc01c5b0 con 0x7fe90c0fca70 2024-03-30T04:18:24.066 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:24.066+0000 7fe912400700 1 -- 172.21.15.135:0/1158710265 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe90c10f920 con 0x7fe90c0fca70 2024-03-30T04:18:24.066 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:24.066+0000 7fe912400700 1 -- 172.21.15.135:0/1158710265 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe90c110490 con 0x7fe90c0fca70 2024-03-30T04:18:24.067 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:24.067+0000 7fe903400700 1 -- 172.21.15.135:0/1158710265 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7fe8fc013420 con 0x7fe90c0fca70 2024-03-30T04:18:24.067 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:24.067+0000 7fe912400700 1 -- 172.21.15.135:0/1158710265 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe90c102e40 con 0x7fe90c0fca70 2024-03-30T04:18:24.067 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:24.068+0000 7fe903400700 1 --2- 172.21.15.135:0/1158710265 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7fe8f0036880 0x7fe8f0038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:18:24.067 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:24.068+0000 7fe903400700 1 -- 172.21.15.135:0/1158710265 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fe8fc049b80 con 0x7fe90c0fca70 2024-03-30T04:18:24.067 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:24.068+0000 7fe90aa00700 1 --2- 172.21.15.135:0/1158710265 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7fe8f0036880 0x7fe8f0038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:18:24.068 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:24.069+0000 7fe90aa00700 1 --2- 172.21.15.135:0/1158710265 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7fe8f0036880 0x7fe8f0038d30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe8f8009990 tx=0x7fe8f8006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:18:24.072 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:24.073+0000 7fe903400700 1 -- 172.21.15.135:0/1158710265 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fe8fc04be60 con 0x7fe90c0fca70 2024-03-30T04:18:24.344 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:24.345+0000 7fe912400700 1 -- 172.21.15.135:0/1158710265 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe90c1106d0 con 0x7fe90c0fca70 2024-03-30T04:18:24.344 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:24.345+0000 7fe903400700 1 -- 172.21.15.135:0/1158710265 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fe8fc025400 con 0x7fe90c0fca70 2024-03-30T04:18:24.345 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:18:24.345 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"2e77a2be-ee4c-11ee-b647-cb9ed24678a4","modified":"2024-03-30T04:16:26.797409Z","created":"2024-03-30T04:16:26.797409Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-30T04:18:24.346 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:24.347+0000 7fe912400700 1 -- 172.21.15.135:0/1158710265 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7fe8f0036880 msgr2=0x7fe8f0038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:18:24.346 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:24.347+0000 7fe912400700 1 --2- 172.21.15.135:0/1158710265 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7fe8f0036880 0x7fe8f0038d30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe8f8009990 tx=0x7fe8f8006e40 comp rx=0 tx=0).stop 2024-03-30T04:18:24.347 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:24.347+0000 7fe912400700 1 -- 172.21.15.135:0/1158710265 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe90c0fca70 msgr2=0x7fe90c10f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:18:24.347 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:24.347+0000 7fe912400700 1 --2- 172.21.15.135:0/1158710265 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe90c0fca70 0x7fe90c10f1e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc000c00 tx=0x7fe8fc009e30 comp rx=0 tx=0).stop 2024-03-30T04:18:24.347 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:24.347+0000 7fe912400700 1 -- 172.21.15.135:0/1158710265 shutdown_connections 2024-03-30T04:18:24.347 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:24.347+0000 7fe912400700 1 --2- 172.21.15.135:0/1158710265 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7fe8f0036880 0x7fe8f0038d30 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:18:24.347 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:24.347+0000 7fe912400700 1 --2- 172.21.15.135:0/1158710265 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe90c0fca70 0x7fe90c10f1e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:18:24.347 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:24.347+0000 7fe912400700 1 -- 172.21.15.135:0/1158710265 >> 172.21.15.135:0/1158710265 conn(0x7fe90c0f81d0 msgr2=0x7fe90c0fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:18:24.347 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:24.348+0000 7fe912400700 1 -- 172.21.15.135:0/1158710265 shutdown_connections 2024-03-30T04:18:24.347 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:24.348+0000 7fe912400700 1 -- 172.21.15.135:0/1158710265 wait complete. 2024-03-30T04:18:24.347 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2024-03-30T04:18:25.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:25 smithi123 ceph-mon[35245]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:18:25.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:25 smithi123 ceph-mon[35245]: from='client.? 172.21.15.135:0/1158710265' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T04:18:25.977 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-30T04:18:25.977 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-03-30T04:18:26.379 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-30T04:18:26.467 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-30T04:18:27.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:27 smithi123 ceph-mon[35245]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:18:28.063 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:28.064+0000 7f91fe400700 1 -- 172.21.15.135:0/549875573 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91f8071e40 msgr2=0x7f91f8070350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:18:28.063 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:28.064+0000 7f91fe400700 1 --2- 172.21.15.135:0/549875573 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91f8071e40 0x7f91f8070350 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f91e80048c0 tx=0x7f91e8009e30 comp rx=0 tx=0).stop 2024-03-30T04:18:28.063 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:28.064+0000 7f91fe400700 1 -- 172.21.15.135:0/549875573 shutdown_connections 2024-03-30T04:18:28.063 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:28.064+0000 7f91fe400700 1 --2- 172.21.15.135:0/549875573 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91f8071e40 0x7f91f8070350 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:18:28.063 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:28.064+0000 7f91fe400700 1 -- 172.21.15.135:0/549875573 >> 172.21.15.135:0/549875573 conn(0x7f91f80f81d0 msgr2=0x7f91f80fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:18:28.064 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:28.064+0000 7f91fe400700 1 -- 172.21.15.135:0/549875573 shutdown_connections 2024-03-30T04:18:28.064 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:28.064+0000 7f91fe400700 1 -- 172.21.15.135:0/549875573 wait complete. 2024-03-30T04:18:28.064 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:28.065+0000 7f91fe400700 1 Processor -- start 2024-03-30T04:18:28.064 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:28.065+0000 7f91fe400700 1 -- start start 2024-03-30T04:18:28.065 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:28.065+0000 7f91fe400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91f8071e40 0x7f91f810f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:18:28.065 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:28.065+0000 7f91fe400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f810f750 con 0x7f91f8071e40 2024-03-30T04:18:28.065 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:28.066+0000 7f91f7400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91f8071e40 0x7f91f810f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:18:28.065 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:28.066+0000 7f91f7400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91f8071e40 0x7f91f810f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:58770/0 (socket says 172.21.15.135:58770) 2024-03-30T04:18:28.065 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:28.066+0000 7f91f7400700 1 -- 172.21.15.135:0/4134698463 learned_addr learned my addr 172.21.15.135:0/4134698463 (peer_addr_for_me v2:172.21.15.135:0/0) 2024-03-30T04:18:28.066 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:28.066+0000 7f91f7400700 1 -- 172.21.15.135:0/4134698463 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e8005040 con 0x7f91f8071e40 2024-03-30T04:18:28.066 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:28.066+0000 7f91f7400700 1 --2- 172.21.15.135:0/4134698463 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91f8071e40 0x7f91f810f210 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f91e800b860 tx=0x7f91e8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:18:28.066 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:28.067+0000 7f91ef400700 1 -- 172.21.15.135:0/4134698463 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f91e8009580 con 0x7f91f8071e40 2024-03-30T04:18:28.066 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:28.067+0000 7f91ef400700 1 -- 172.21.15.135:0/4134698463 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f91e80096e0 con 0x7f91f8071e40 2024-03-30T04:18:28.066 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:28.067+0000 7f91fe400700 1 -- 172.21.15.135:0/4134698463 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91f810f950 con 0x7f91f8071e40 2024-03-30T04:18:28.066 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:28.067+0000 7f91ef400700 1 -- 172.21.15.135:0/4134698463 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f91e801c5b0 con 0x7f91f8071e40 2024-03-30T04:18:28.066 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:28.067+0000 7f91fe400700 1 -- 172.21.15.135:0/4134698463 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91f81104c0 con 0x7f91f8071e40 2024-03-30T04:18:28.067 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:28.068+0000 7f91fe400700 1 -- 172.21.15.135:0/4134698463 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91f8105050 con 0x7f91f8071e40 2024-03-30T04:18:28.067 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:28.068+0000 7f91ef400700 1 -- 172.21.15.135:0/4134698463 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f91e8013420 con 0x7f91f8071e40 2024-03-30T04:18:28.067 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:28.068+0000 7f91ef400700 1 --2- 172.21.15.135:0/4134698463 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f91dc036880 0x7f91dc038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:18:28.067 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:28.069+0000 7f91ef400700 1 -- 172.21.15.135:0/4134698463 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f91e8049af0 con 0x7f91f8071e40 2024-03-30T04:18:28.068 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:28.069+0000 7f91f6a00700 1 --2- 172.21.15.135:0/4134698463 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f91dc036880 0x7f91dc038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:18:28.068 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:28.069+0000 7f91f6a00700 1 --2- 172.21.15.135:0/4134698463 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f91dc036880 0x7f91dc038d30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f91e4009990 tx=0x7f91e4006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:18:28.077 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:28.079+0000 7f91ef400700 1 -- 172.21.15.135:0/4134698463 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f91e8011550 con 0x7f91f8071e40 2024-03-30T04:18:28.348 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:28.348+0000 7f91fe400700 1 -- 172.21.15.135:0/4134698463 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f91f805f5f0 con 0x7f91f8071e40 2024-03-30T04:18:28.348 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:28.348+0000 7f91ef400700 1 -- 172.21.15.135:0/4134698463 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f91e8011370 con 0x7f91f8071e40 2024-03-30T04:18:28.348 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:18:28.348 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"2e77a2be-ee4c-11ee-b647-cb9ed24678a4","modified":"2024-03-30T04:16:26.797409Z","created":"2024-03-30T04:16:26.797409Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-30T04:18:28.350 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:28.351+0000 7f91fe400700 1 -- 172.21.15.135:0/4134698463 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f91dc036880 msgr2=0x7f91dc038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:18:28.350 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:28.351+0000 7f91fe400700 1 --2- 172.21.15.135:0/4134698463 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f91dc036880 0x7f91dc038d30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f91e4009990 tx=0x7f91e4006e40 comp rx=0 tx=0).stop 2024-03-30T04:18:28.350 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:28.351+0000 7f91fe400700 1 -- 172.21.15.135:0/4134698463 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91f8071e40 msgr2=0x7f91f810f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:18:28.350 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:28.351+0000 7f91fe400700 1 --2- 172.21.15.135:0/4134698463 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91f8071e40 0x7f91f810f210 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f91e800b860 tx=0x7f91e8009e30 comp rx=0 tx=0).stop 2024-03-30T04:18:28.351 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:28.351+0000 7f91fe400700 1 -- 172.21.15.135:0/4134698463 shutdown_connections 2024-03-30T04:18:28.351 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:28.351+0000 7f91fe400700 1 --2- 172.21.15.135:0/4134698463 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f91dc036880 0x7f91dc038d30 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:18:28.351 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:28.351+0000 7f91fe400700 1 --2- 172.21.15.135:0/4134698463 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91f8071e40 0x7f91f810f210 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:18:28.351 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:28.351+0000 7f91fe400700 1 -- 172.21.15.135:0/4134698463 >> 172.21.15.135:0/4134698463 conn(0x7f91f80f81d0 msgr2=0x7f91f80f8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:18:28.351 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:28.351+0000 7f91fe400700 1 -- 172.21.15.135:0/4134698463 shutdown_connections 2024-03-30T04:18:28.351 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:28.351+0000 7f91fe400700 1 -- 172.21.15.135:0/4134698463 wait complete. 2024-03-30T04:18:28.352 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2024-03-30T04:18:29.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:29 smithi123 ceph-mon[35245]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:18:29.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:29 smithi123 ceph-mon[35245]: from='client.? 172.21.15.135:0/4134698463' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T04:18:29.981 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-30T04:18:29.981 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-03-30T04:18:30.388 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-30T04:18:30.476 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-30T04:18:31.370 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:31 smithi123 ceph-mon[35245]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:18:32.135 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:32.136+0000 7f40cf200700 1 -- 172.21.15.135:0/1173856197 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40c80fca70 msgr2=0x7f40c80fce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:18:32.135 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:32.136+0000 7f40cf200700 1 --2- 172.21.15.135:0/1173856197 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40c80fca70 0x7f40c80fce80 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f40b80048c0 tx=0x7f40b8009e30 comp rx=0 tx=0).stop 2024-03-30T04:18:32.135 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:32.136+0000 7f40cf200700 1 -- 172.21.15.135:0/1173856197 shutdown_connections 2024-03-30T04:18:32.135 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:32.136+0000 7f40cf200700 1 --2- 172.21.15.135:0/1173856197 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40c80fca70 0x7f40c80fce80 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:18:32.135 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:32.136+0000 7f40cf200700 1 -- 172.21.15.135:0/1173856197 >> 172.21.15.135:0/1173856197 conn(0x7f40c80f81d0 msgr2=0x7f40c80fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:18:32.136 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:32.136+0000 7f40cf200700 1 -- 172.21.15.135:0/1173856197 shutdown_connections 2024-03-30T04:18:32.136 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:32.136+0000 7f40cf200700 1 -- 172.21.15.135:0/1173856197 wait complete. 2024-03-30T04:18:32.136 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:32.137+0000 7f40cf200700 1 Processor -- start 2024-03-30T04:18:32.136 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:32.137+0000 7f40cf200700 1 -- start start 2024-03-30T04:18:32.137 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:32.138+0000 7f40cf200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40c80fca70 0x7f40c810f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:18:32.137 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:32.138+0000 7f40cf200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c810f740 con 0x7f40c80fca70 2024-03-30T04:18:32.137 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:32.138+0000 7f40c7e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40c80fca70 0x7f40c810f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:18:32.137 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:32.138+0000 7f40c7e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40c80fca70 0x7f40c810f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:58786/0 (socket says 172.21.15.135:58786) 2024-03-30T04:18:32.137 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:32.138+0000 7f40c7e00700 1 -- 172.21.15.135:0/1862239225 learned_addr learned my addr 172.21.15.135:0/1862239225 (peer_addr_for_me v2:172.21.15.135:0/0) 2024-03-30T04:18:32.137 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:32.139+0000 7f40c7e00700 1 -- 172.21.15.135:0/1862239225 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40b8005040 con 0x7f40c80fca70 2024-03-30T04:18:32.137 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:32.139+0000 7f40c7e00700 1 --2- 172.21.15.135:0/1862239225 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40c80fca70 0x7f40c810f200 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f40b8000c00 tx=0x7f40b8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:18:32.137 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:32.139+0000 7f40bfe00700 1 -- 172.21.15.135:0/1862239225 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40b8009620 con 0x7f40c80fca70 2024-03-30T04:18:32.138 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:32.139+0000 7f40bfe00700 1 -- 172.21.15.135:0/1862239225 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f40b8009780 con 0x7f40c80fca70 2024-03-30T04:18:32.138 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:32.139+0000 7f40bfe00700 1 -- 172.21.15.135:0/1862239225 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40b801c5b0 con 0x7f40c80fca70 2024-03-30T04:18:32.138 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:32.139+0000 7f40cf200700 1 -- 172.21.15.135:0/1862239225 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40c810f940 con 0x7f40c80fca70 2024-03-30T04:18:32.138 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:32.139+0000 7f40cf200700 1 -- 172.21.15.135:0/1862239225 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40c81104b0 con 0x7f40c80fca70 2024-03-30T04:18:32.139 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:32.140+0000 7f40bfe00700 1 -- 172.21.15.135:0/1862239225 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f40b8013420 con 0x7f40c80fca70 2024-03-30T04:18:32.139 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:32.140+0000 7f40cf200700 1 -- 172.21.15.135:0/1862239225 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40c8102e40 con 0x7f40c80fca70 2024-03-30T04:18:32.139 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:32.141+0000 7f40bfe00700 1 --2- 172.21.15.135:0/1862239225 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f40ac0368d0 0x7f40ac038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:18:32.139 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:32.141+0000 7f40bfe00700 1 -- 172.21.15.135:0/1862239225 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f40b8049b20 con 0x7f40c80fca70 2024-03-30T04:18:32.139 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:32.141+0000 7f40c7400700 1 --2- 172.21.15.135:0/1862239225 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f40ac0368d0 0x7f40ac038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:18:32.139 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:32.142+0000 7f40c7400700 1 --2- 172.21.15.135:0/1862239225 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f40ac0368d0 0x7f40ac038d80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f40b4009990 tx=0x7f40b4006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:18:32.144 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:32.146+0000 7f40bfe00700 1 -- 172.21.15.135:0/1862239225 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f40b8025400 con 0x7f40c80fca70 2024-03-30T04:18:32.419 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:32.421+0000 7f40cf200700 1 -- 172.21.15.135:0/1862239225 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f40c81107b0 con 0x7f40c80fca70 2024-03-30T04:18:32.419 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:32.421+0000 7f40bfe00700 1 -- 172.21.15.135:0/1862239225 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f40b8025400 con 0x7f40c80fca70 2024-03-30T04:18:32.419 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:18:32.420 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"2e77a2be-ee4c-11ee-b647-cb9ed24678a4","modified":"2024-03-30T04:16:26.797409Z","created":"2024-03-30T04:16:26.797409Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-30T04:18:32.421 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:32.423+0000 7f40cf200700 1 -- 172.21.15.135:0/1862239225 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f40ac0368d0 msgr2=0x7f40ac038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:18:32.422 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:32.423+0000 7f40cf200700 1 --2- 172.21.15.135:0/1862239225 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f40ac0368d0 0x7f40ac038d80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f40b4009990 tx=0x7f40b4006e40 comp rx=0 tx=0).stop 2024-03-30T04:18:32.422 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:32.423+0000 7f40cf200700 1 -- 172.21.15.135:0/1862239225 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40c80fca70 msgr2=0x7f40c810f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:18:32.422 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:32.423+0000 7f40cf200700 1 --2- 172.21.15.135:0/1862239225 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40c80fca70 0x7f40c810f200 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f40b8000c00 tx=0x7f40b8009e30 comp rx=0 tx=0).stop 2024-03-30T04:18:32.422 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:32.424+0000 7f40cf200700 1 -- 172.21.15.135:0/1862239225 shutdown_connections 2024-03-30T04:18:32.422 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:32.424+0000 7f40cf200700 1 --2- 172.21.15.135:0/1862239225 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f40ac0368d0 0x7f40ac038d80 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:18:32.422 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:32.424+0000 7f40cf200700 1 --2- 172.21.15.135:0/1862239225 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40c80fca70 0x7f40c810f200 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:18:32.422 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:32.424+0000 7f40cf200700 1 -- 172.21.15.135:0/1862239225 >> 172.21.15.135:0/1862239225 conn(0x7f40c80f81d0 msgr2=0x7f40c80fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:18:32.422 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:32.424+0000 7f40cf200700 1 -- 172.21.15.135:0/1862239225 shutdown_connections 2024-03-30T04:18:32.422 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:32.424+0000 7f40cf200700 1 -- 172.21.15.135:0/1862239225 wait complete. 2024-03-30T04:18:32.422 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2024-03-30T04:18:33.413 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:33 smithi123 ceph-mon[35245]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:18:33.413 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:33 smithi123 ceph-mon[35245]: from='client.? 172.21.15.135:0/1862239225' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T04:18:34.067 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-30T04:18:34.067 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-03-30T04:18:34.474 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-30T04:18:34.563 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-30T04:18:35.305 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:35 smithi123 ceph-mon[35245]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:18:35.305 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:35 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' 2024-03-30T04:18:35.305 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:35 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' 2024-03-30T04:18:35.305 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:35 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' 2024-03-30T04:18:35.305 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:35 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' 2024-03-30T04:18:35.305 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:35 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' 2024-03-30T04:18:35.305 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:35 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' 2024-03-30T04:18:36.270 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:36.271+0000 7f4ab9c00700 1 -- 172.21.15.135:0/1398851686 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ab40fca70 msgr2=0x7f4ab40fce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:18:36.271 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:36.271+0000 7f4ab9c00700 1 --2- 172.21.15.135:0/1398851686 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ab40fca70 0x7f4ab40fce80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4aa40048c0 tx=0x7f4aa4009e30 comp rx=0 tx=0).stop 2024-03-30T04:18:36.271 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:36.272+0000 7f4ab9c00700 1 -- 172.21.15.135:0/1398851686 shutdown_connections 2024-03-30T04:18:36.271 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:36.272+0000 7f4ab9c00700 1 --2- 172.21.15.135:0/1398851686 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ab40fca70 0x7f4ab40fce80 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:18:36.271 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:36.272+0000 7f4ab9c00700 1 -- 172.21.15.135:0/1398851686 >> 172.21.15.135:0/1398851686 conn(0x7f4ab40f81d0 msgr2=0x7f4ab40fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:18:36.271 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:36.272+0000 7f4ab9c00700 1 -- 172.21.15.135:0/1398851686 shutdown_connections 2024-03-30T04:18:36.271 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:36.272+0000 7f4ab9c00700 1 -- 172.21.15.135:0/1398851686 wait complete. 2024-03-30T04:18:36.272 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:36.273+0000 7f4ab9c00700 1 Processor -- start 2024-03-30T04:18:36.272 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:36.273+0000 7f4ab9c00700 1 -- start start 2024-03-30T04:18:36.272 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:36.273+0000 7f4ab9c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ab40fca70 0x7f4ab4071d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:18:36.272 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:36.273+0000 7f4ab9c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab40722a0 con 0x7f4ab40fca70 2024-03-30T04:18:36.272 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:36.274+0000 7f4ab2a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ab40fca70 0x7f4ab4071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:18:36.272 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:36.274+0000 7f4ab2a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ab40fca70 0x7f4ab4071d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:58808/0 (socket says 172.21.15.135:58808) 2024-03-30T04:18:36.272 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:36.274+0000 7f4ab2a00700 1 -- 172.21.15.135:0/502598625 learned_addr learned my addr 172.21.15.135:0/502598625 (peer_addr_for_me v2:172.21.15.135:0/0) 2024-03-30T04:18:36.274 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:36.274+0000 7f4ab2a00700 1 -- 172.21.15.135:0/502598625 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa4005040 con 0x7f4ab40fca70 2024-03-30T04:18:36.274 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:36.274+0000 7f4ab2a00700 1 --2- 172.21.15.135:0/502598625 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ab40fca70 0x7f4ab4071d60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4aa4000c00 tx=0x7f4aa4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:18:36.274 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:36.274+0000 7f4aaaa00700 1 -- 172.21.15.135:0/502598625 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4aa4009620 con 0x7f4ab40fca70 2024-03-30T04:18:36.274 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:36.275+0000 7f4aaaa00700 1 -- 172.21.15.135:0/502598625 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4aa4009780 con 0x7f4ab40fca70 2024-03-30T04:18:36.274 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:36.275+0000 7f4aaaa00700 1 -- 172.21.15.135:0/502598625 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4aa401c5b0 con 0x7f4ab40fca70 2024-03-30T04:18:36.274 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:36.275+0000 7f4ab9c00700 1 -- 172.21.15.135:0/502598625 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ab4070370 con 0x7f4ab40fca70 2024-03-30T04:18:36.274 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:36.275+0000 7f4ab9c00700 1 -- 172.21.15.135:0/502598625 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ab4070860 con 0x7f4ab40fca70 2024-03-30T04:18:36.274 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:36.276+0000 7f4ab9c00700 1 -- 172.21.15.135:0/502598625 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ab405f5f0 con 0x7f4ab40fca70 2024-03-30T04:18:36.274 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:36.276+0000 7f4aaaa00700 1 -- 172.21.15.135:0/502598625 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f4aa4013a60 con 0x7f4ab40fca70 2024-03-30T04:18:36.275 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:36.276+0000 7f4aaaa00700 1 --2- 172.21.15.135:0/502598625 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f4a98036880 0x7f4a98038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:18:36.275 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:36.276+0000 7f4aaaa00700 1 -- 172.21.15.135:0/502598625 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f4aa4049b40 con 0x7f4ab40fca70 2024-03-30T04:18:36.275 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:36.277+0000 7f4ab2000700 1 --2- 172.21.15.135:0/502598625 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f4a98036880 0x7f4a98038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:18:36.275 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:36.277+0000 7f4ab2000700 1 --2- 172.21.15.135:0/502598625 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f4a98036880 0x7f4a98038d30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4aa0009990 tx=0x7f4aa0006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:18:36.279 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:36.282+0000 7f4aaaa00700 1 -- 172.21.15.135:0/502598625 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f4aa4025400 con 0x7f4ab40fca70 2024-03-30T04:18:36.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:36 smithi123 ceph-mon[35245]: Deploying daemon prometheus.smithi123 on smithi123 2024-03-30T04:18:36.553 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:36.555+0000 7f4ab9c00700 1 -- 172.21.15.135:0/502598625 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4ab4106b90 con 0x7f4ab40fca70 2024-03-30T04:18:36.554 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:36.556+0000 7f4aaaa00700 1 -- 172.21.15.135:0/502598625 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f4aa4011b60 con 0x7f4ab40fca70 2024-03-30T04:18:36.554 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:18:36.555 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"2e77a2be-ee4c-11ee-b647-cb9ed24678a4","modified":"2024-03-30T04:16:26.797409Z","created":"2024-03-30T04:16:26.797409Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-30T04:18:36.557 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:36.558+0000 7f4ab9c00700 1 -- 172.21.15.135:0/502598625 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f4a98036880 msgr2=0x7f4a98038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:18:36.557 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:36.558+0000 7f4ab9c00700 1 --2- 172.21.15.135:0/502598625 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f4a98036880 0x7f4a98038d30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4aa0009990 tx=0x7f4aa0006e40 comp rx=0 tx=0).stop 2024-03-30T04:18:36.557 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:36.558+0000 7f4ab9c00700 1 -- 172.21.15.135:0/502598625 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ab40fca70 msgr2=0x7f4ab4071d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:18:36.557 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:36.558+0000 7f4ab9c00700 1 --2- 172.21.15.135:0/502598625 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ab40fca70 0x7f4ab4071d60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4aa4000c00 tx=0x7f4aa4009e30 comp rx=0 tx=0).stop 2024-03-30T04:18:36.557 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:36.558+0000 7f4ab9c00700 1 -- 172.21.15.135:0/502598625 shutdown_connections 2024-03-30T04:18:36.557 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:36.558+0000 7f4ab9c00700 1 --2- 172.21.15.135:0/502598625 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f4a98036880 0x7f4a98038d30 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:18:36.557 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:36.558+0000 7f4ab9c00700 1 --2- 172.21.15.135:0/502598625 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ab40fca70 0x7f4ab4071d60 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:18:36.557 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:36.558+0000 7f4ab9c00700 1 -- 172.21.15.135:0/502598625 >> 172.21.15.135:0/502598625 conn(0x7f4ab40f81d0 msgr2=0x7f4ab40f8a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:18:36.557 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:36.558+0000 7f4ab9c00700 1 -- 172.21.15.135:0/502598625 shutdown_connections 2024-03-30T04:18:36.557 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:36.558+0000 7f4ab9c00700 1 -- 172.21.15.135:0/502598625 wait complete. 2024-03-30T04:18:36.558 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2024-03-30T04:18:37.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:37 smithi123 ceph-mon[35245]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:18:37.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:37 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' 2024-03-30T04:18:37.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:37 smithi123 ceph-mon[35245]: from='client.? 172.21.15.135:0/502598625' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T04:18:38.206 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-30T04:18:38.207 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-03-30T04:18:38.609 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-30T04:18:38.699 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-30T04:18:39.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:39 smithi123 ceph-mon[35245]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:18:40.321 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:40.323+0000 7fd50f800700 1 -- 172.21.15.135:0/900737061 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5080fca70 msgr2=0x7fd5080fce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:18:40.321 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:40.323+0000 7fd50f800700 1 --2- 172.21.15.135:0/900737061 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5080fca70 0x7fd5080fce80 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc0048c0 tx=0x7fd4fc009e30 comp rx=0 tx=0).stop 2024-03-30T04:18:40.321 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:40.323+0000 7fd50f800700 1 -- 172.21.15.135:0/900737061 shutdown_connections 2024-03-30T04:18:40.321 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:40.323+0000 7fd50f800700 1 --2- 172.21.15.135:0/900737061 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5080fca70 0x7fd5080fce80 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:18:40.321 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:40.323+0000 7fd50f800700 1 -- 172.21.15.135:0/900737061 >> 172.21.15.135:0/900737061 conn(0x7fd5080f81d0 msgr2=0x7fd5080fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:18:40.321 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:40.323+0000 7fd50f800700 1 -- 172.21.15.135:0/900737061 shutdown_connections 2024-03-30T04:18:40.321 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:40.323+0000 7fd50f800700 1 -- 172.21.15.135:0/900737061 wait complete. 2024-03-30T04:18:40.322 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:40.324+0000 7fd50f800700 1 Processor -- start 2024-03-30T04:18:40.322 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:40.324+0000 7fd50f800700 1 -- start start 2024-03-30T04:18:40.323 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:40.324+0000 7fd50f800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5080fca70 0x7fd50810f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:18:40.323 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:40.324+0000 7fd50f800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd50810f740 con 0x7fd5080fca70 2024-03-30T04:18:40.323 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:40.324+0000 7fd50ce00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5080fca70 0x7fd50810f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:18:40.323 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:40.324+0000 7fd50ce00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5080fca70 0x7fd50810f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:59706/0 (socket says 172.21.15.135:59706) 2024-03-30T04:18:40.323 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:40.324+0000 7fd50ce00700 1 -- 172.21.15.135:0/1982334278 learned_addr learned my addr 172.21.15.135:0/1982334278 (peer_addr_for_me v2:172.21.15.135:0/0) 2024-03-30T04:18:40.323 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:40.325+0000 7fd50ce00700 1 -- 172.21.15.135:0/1982334278 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4fc005040 con 0x7fd5080fca70 2024-03-30T04:18:40.324 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:40.325+0000 7fd50ce00700 1 --2- 172.21.15.135:0/1982334278 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5080fca70 0x7fd50810f200 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc000c00 tx=0x7fd4fc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:18:40.324 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:40.325+0000 7fd504c00700 1 -- 172.21.15.135:0/1982334278 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4fc009620 con 0x7fd5080fca70 2024-03-30T04:18:40.324 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:40.325+0000 7fd50f800700 1 -- 172.21.15.135:0/1982334278 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd50810f940 con 0x7fd5080fca70 2024-03-30T04:18:40.324 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:40.325+0000 7fd504c00700 1 -- 172.21.15.135:0/1982334278 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd4fc009780 con 0x7fd5080fca70 2024-03-30T04:18:40.324 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:40.325+0000 7fd504c00700 1 -- 172.21.15.135:0/1982334278 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4fc01c5b0 con 0x7fd5080fca70 2024-03-30T04:18:40.324 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:40.325+0000 7fd50f800700 1 -- 172.21.15.135:0/1982334278 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5081104b0 con 0x7fd5080fca70 2024-03-30T04:18:40.325 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:40.326+0000 7fd504c00700 1 -- 172.21.15.135:0/1982334278 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7fd4fc013420 con 0x7fd5080fca70 2024-03-30T04:18:40.325 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:40.326+0000 7fd50f800700 1 -- 172.21.15.135:0/1982334278 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd508102e40 con 0x7fd5080fca70 2024-03-30T04:18:40.325 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:40.327+0000 7fd504c00700 1 --2- 172.21.15.135:0/1982334278 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7fd4f0036880 0x7fd4f0038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:18:40.326 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:40.327+0000 7fd504c00700 1 -- 172.21.15.135:0/1982334278 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fd4fc049b80 con 0x7fd5080fca70 2024-03-30T04:18:40.326 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:40.327+0000 7fd507e00700 1 --2- 172.21.15.135:0/1982334278 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7fd4f0036880 0x7fd4f0038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:18:40.326 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:40.328+0000 7fd507e00700 1 --2- 172.21.15.135:0/1982334278 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7fd4f0036880 0x7fd4f0038d30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd4f4009990 tx=0x7fd4f4006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:18:40.331 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:40.333+0000 7fd504c00700 1 -- 172.21.15.135:0/1982334278 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fd4fc04be60 con 0x7fd5080fca70 2024-03-30T04:18:40.602 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:40.604+0000 7fd50f800700 1 -- 172.21.15.135:0/1982334278 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd5081107b0 con 0x7fd5080fca70 2024-03-30T04:18:40.602 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:40.605+0000 7fd504c00700 1 -- 172.21.15.135:0/1982334278 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fd4fc025400 con 0x7fd5080fca70 2024-03-30T04:18:40.603 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:18:40.603 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"2e77a2be-ee4c-11ee-b647-cb9ed24678a4","modified":"2024-03-30T04:16:26.797409Z","created":"2024-03-30T04:16:26.797409Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-30T04:18:40.605 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:40.607+0000 7fd50f800700 1 -- 172.21.15.135:0/1982334278 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7fd4f0036880 msgr2=0x7fd4f0038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:18:40.605 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:40.607+0000 7fd50f800700 1 --2- 172.21.15.135:0/1982334278 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7fd4f0036880 0x7fd4f0038d30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd4f4009990 tx=0x7fd4f4006e40 comp rx=0 tx=0).stop 2024-03-30T04:18:40.605 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:40.607+0000 7fd50f800700 1 -- 172.21.15.135:0/1982334278 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5080fca70 msgr2=0x7fd50810f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:18:40.605 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:40.607+0000 7fd50f800700 1 --2- 172.21.15.135:0/1982334278 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5080fca70 0x7fd50810f200 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc000c00 tx=0x7fd4fc009e30 comp rx=0 tx=0).stop 2024-03-30T04:18:40.605 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:40.607+0000 7fd50f800700 1 -- 172.21.15.135:0/1982334278 shutdown_connections 2024-03-30T04:18:40.605 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:40.607+0000 7fd50f800700 1 --2- 172.21.15.135:0/1982334278 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7fd4f0036880 0x7fd4f0038d30 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:18:40.605 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:40.607+0000 7fd50f800700 1 --2- 172.21.15.135:0/1982334278 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5080fca70 0x7fd50810f200 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:18:40.605 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:40.607+0000 7fd50f800700 1 -- 172.21.15.135:0/1982334278 >> 172.21.15.135:0/1982334278 conn(0x7fd5080f81d0 msgr2=0x7fd5080fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:18:40.606 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:40.607+0000 7fd50f800700 1 -- 172.21.15.135:0/1982334278 shutdown_connections 2024-03-30T04:18:40.606 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:40.607+0000 7fd50f800700 1 -- 172.21.15.135:0/1982334278 wait complete. 2024-03-30T04:18:40.606 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2024-03-30T04:18:41.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:41 smithi123 ceph-mon[35245]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:18:41.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:41 smithi123 ceph-mon[35245]: from='client.? 172.21.15.135:0/1982334278' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T04:18:42.199 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-30T04:18:42.200 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-03-30T04:18:42.605 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-30T04:18:42.692 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-30T04:18:43.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:43 smithi123 ceph-mon[35245]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:18:44.333 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:44.334+0000 7f0149a00700 1 -- 172.21.15.135:0/3621299969 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01440fca70 msgr2=0x7f01440fce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:18:44.333 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:44.334+0000 7f0149a00700 1 --2- 172.21.15.135:0/3621299969 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01440fca70 0x7f01440fce80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f01340048c0 tx=0x7f0134009e30 comp rx=0 tx=0).stop 2024-03-30T04:18:44.333 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:44.334+0000 7f0149a00700 1 -- 172.21.15.135:0/3621299969 shutdown_connections 2024-03-30T04:18:44.333 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:44.334+0000 7f0149a00700 1 --2- 172.21.15.135:0/3621299969 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01440fca70 0x7f01440fce80 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:18:44.333 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:44.334+0000 7f0149a00700 1 -- 172.21.15.135:0/3621299969 >> 172.21.15.135:0/3621299969 conn(0x7f01440f81d0 msgr2=0x7f01440fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:18:44.334 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:44.334+0000 7f0149a00700 1 -- 172.21.15.135:0/3621299969 shutdown_connections 2024-03-30T04:18:44.334 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:44.334+0000 7f0149a00700 1 -- 172.21.15.135:0/3621299969 wait complete. 2024-03-30T04:18:44.335 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:44.335+0000 7f0149a00700 1 Processor -- start 2024-03-30T04:18:44.335 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:44.335+0000 7f0149a00700 1 -- start start 2024-03-30T04:18:44.335 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:44.335+0000 7f0149a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01440fca70 0x7f014410f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:18:44.335 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:44.335+0000 7f0149a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f014410f720 con 0x7f01440fca70 2024-03-30T04:18:44.335 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:44.335+0000 7f0142a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01440fca70 0x7f014410f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:18:44.336 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:44.336+0000 7f0142a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01440fca70 0x7f014410f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:59724/0 (socket says 172.21.15.135:59724) 2024-03-30T04:18:44.336 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:44.336+0000 7f0142a00700 1 -- 172.21.15.135:0/2807785282 learned_addr learned my addr 172.21.15.135:0/2807785282 (peer_addr_for_me v2:172.21.15.135:0/0) 2024-03-30T04:18:44.336 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:44.336+0000 7f0142a00700 1 -- 172.21.15.135:0/2807785282 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0134005040 con 0x7f01440fca70 2024-03-30T04:18:44.336 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:44.336+0000 7f0142a00700 1 --2- 172.21.15.135:0/2807785282 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01440fca70 0x7f014410f1e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0134000c00 tx=0x7f0134009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:18:44.336 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:44.336+0000 7f013aa00700 1 -- 172.21.15.135:0/2807785282 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0134009620 con 0x7f01440fca70 2024-03-30T04:18:44.336 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:44.336+0000 7f013aa00700 1 -- 172.21.15.135:0/2807785282 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0134009780 con 0x7f01440fca70 2024-03-30T04:18:44.336 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:44.336+0000 7f013aa00700 1 -- 172.21.15.135:0/2807785282 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f013401c5b0 con 0x7f01440fca70 2024-03-30T04:18:44.336 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:44.336+0000 7f0149a00700 1 -- 172.21.15.135:0/2807785282 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f014410f920 con 0x7f01440fca70 2024-03-30T04:18:44.336 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:44.337+0000 7f0149a00700 1 -- 172.21.15.135:0/2807785282 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0144110490 con 0x7f01440fca70 2024-03-30T04:18:44.337 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:44.337+0000 7f013aa00700 1 -- 172.21.15.135:0/2807785282 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f0134013420 con 0x7f01440fca70 2024-03-30T04:18:44.337 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:44.337+0000 7f0149a00700 1 -- 172.21.15.135:0/2807785282 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0144102e40 con 0x7f01440fca70 2024-03-30T04:18:44.337 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:44.338+0000 7f013aa00700 1 --2- 172.21.15.135:0/2807785282 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f01280368d0 0x7f0128038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:18:44.337 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:44.338+0000 7f013aa00700 1 -- 172.21.15.135:0/2807785282 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f0134049af0 con 0x7f01440fca70 2024-03-30T04:18:44.337 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:44.338+0000 7f0142000700 1 --2- 172.21.15.135:0/2807785282 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f01280368d0 0x7f0128038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:18:44.338 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:44.339+0000 7f0142000700 1 --2- 172.21.15.135:0/2807785282 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f01280368d0 0x7f0128038d80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0130009990 tx=0x7f0130006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:18:44.342 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:44.343+0000 7f013aa00700 1 -- 172.21.15.135:0/2807785282 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f0134025400 con 0x7f01440fca70 2024-03-30T04:18:44.611 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:44.612+0000 7f0149a00700 1 -- 172.21.15.135:0/2807785282 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f01441106d0 con 0x7f01440fca70 2024-03-30T04:18:44.612 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:44.613+0000 7f013aa00700 1 -- 172.21.15.135:0/2807785282 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f0134011b60 con 0x7f01440fca70 2024-03-30T04:18:44.613 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:18:44.613 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"2e77a2be-ee4c-11ee-b647-cb9ed24678a4","modified":"2024-03-30T04:16:26.797409Z","created":"2024-03-30T04:16:26.797409Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-30T04:18:44.615 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:44.615+0000 7f0149a00700 1 -- 172.21.15.135:0/2807785282 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f01280368d0 msgr2=0x7f0128038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:18:44.615 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:44.615+0000 7f0149a00700 1 --2- 172.21.15.135:0/2807785282 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f01280368d0 0x7f0128038d80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0130009990 tx=0x7f0130006e40 comp rx=0 tx=0).stop 2024-03-30T04:18:44.615 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:44.615+0000 7f0149a00700 1 -- 172.21.15.135:0/2807785282 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01440fca70 msgr2=0x7f014410f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:18:44.615 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:44.615+0000 7f0149a00700 1 --2- 172.21.15.135:0/2807785282 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01440fca70 0x7f014410f1e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0134000c00 tx=0x7f0134009e30 comp rx=0 tx=0).stop 2024-03-30T04:18:44.615 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:44.616+0000 7f0149a00700 1 -- 172.21.15.135:0/2807785282 shutdown_connections 2024-03-30T04:18:44.615 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:44.616+0000 7f0149a00700 1 --2- 172.21.15.135:0/2807785282 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7f01280368d0 0x7f0128038d80 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:18:44.615 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:44.616+0000 7f0149a00700 1 --2- 172.21.15.135:0/2807785282 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01440fca70 0x7f014410f1e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:18:44.615 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:44.616+0000 7f0149a00700 1 -- 172.21.15.135:0/2807785282 >> 172.21.15.135:0/2807785282 conn(0x7f01440f81d0 msgr2=0x7f01440fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:18:44.615 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:44.616+0000 7f0149a00700 1 -- 172.21.15.135:0/2807785282 shutdown_connections 2024-03-30T04:18:44.615 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:44.616+0000 7f0149a00700 1 -- 172.21.15.135:0/2807785282 wait complete. 2024-03-30T04:18:44.616 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2024-03-30T04:18:45.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:45 smithi123 ceph-mon[35245]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:18:45.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:45 smithi123 ceph-mon[35245]: from='client.? 172.21.15.135:0/2807785282' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T04:18:46.118 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-30T04:18:46.118 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-03-30T04:18:46.522 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-30T04:18:46.610 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-30T04:18:47.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:47 smithi123 ceph-mon[35245]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:18:47.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:47 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' 2024-03-30T04:18:47.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:47 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' 2024-03-30T04:18:47.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:47 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' 2024-03-30T04:18:47.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:47 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-30T04:18:48.420 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:48.420+0000 7fc000600700 1 -- 172.21.15.135:0/574727531 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbff80fca70 msgr2=0x7fbff80fce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:18:48.420 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:48.420+0000 7fc000600700 1 --2- 172.21.15.135:0/574727531 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbff80fca70 0x7fbff80fce80 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fbfec0048c0 tx=0x7fbfec009e30 comp rx=0 tx=0).stop 2024-03-30T04:18:48.420 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:48.420+0000 7fc000600700 1 -- 172.21.15.135:0/574727531 shutdown_connections 2024-03-30T04:18:48.421 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:48.420+0000 7fc000600700 1 --2- 172.21.15.135:0/574727531 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbff80fca70 0x7fbff80fce80 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:18:48.421 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:48.420+0000 7fc000600700 1 -- 172.21.15.135:0/574727531 >> 172.21.15.135:0/574727531 conn(0x7fbff80f81d0 msgr2=0x7fbff80fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:18:48.421 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:48.420+0000 7fc000600700 1 -- 172.21.15.135:0/574727531 shutdown_connections 2024-03-30T04:18:48.421 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:48.420+0000 7fc000600700 1 -- 172.21.15.135:0/574727531 wait complete. 2024-03-30T04:18:48.421 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:48.421+0000 7fc000600700 1 Processor -- start 2024-03-30T04:18:48.421 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:48.421+0000 7fc000600700 1 -- start start 2024-03-30T04:18:48.422 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:48.422+0000 7fc000600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbff80fca70 0x7fbff810f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:18:48.422 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:48.422+0000 7fc000600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbff810f720 con 0x7fbff80fca70 2024-03-30T04:18:48.422 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:48.422+0000 7fbffdc00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbff80fca70 0x7fbff810f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:18:48.422 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:48.422+0000 7fbffdc00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbff80fca70 0x7fbff810f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:55786/0 (socket says 172.21.15.135:55786) 2024-03-30T04:18:48.422 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:48.422+0000 7fbffdc00700 1 -- 172.21.15.135:0/1459339104 learned_addr learned my addr 172.21.15.135:0/1459339104 (peer_addr_for_me v2:172.21.15.135:0/0) 2024-03-30T04:18:48.422 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:48.422+0000 7fbffdc00700 1 -- 172.21.15.135:0/1459339104 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfec005040 con 0x7fbff80fca70 2024-03-30T04:18:48.422 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:48.423+0000 7fbffdc00700 1 --2- 172.21.15.135:0/1459339104 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbff80fca70 0x7fbff810f1e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fbfec000c00 tx=0x7fbfec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:18:48.423 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:48.423+0000 7fbff5600700 1 -- 172.21.15.135:0/1459339104 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbfec009620 con 0x7fbff80fca70 2024-03-30T04:18:48.423 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:48.423+0000 7fc000600700 1 -- 172.21.15.135:0/1459339104 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbff810f920 con 0x7fbff80fca70 2024-03-30T04:18:48.423 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:48.423+0000 7fbff5600700 1 -- 172.21.15.135:0/1459339104 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbfec009780 con 0x7fbff80fca70 2024-03-30T04:18:48.423 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:48.423+0000 7fbff5600700 1 -- 172.21.15.135:0/1459339104 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbfec01c5b0 con 0x7fbff80fca70 2024-03-30T04:18:48.423 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:48.423+0000 7fc000600700 1 -- 172.21.15.135:0/1459339104 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbff8110490 con 0x7fbff80fca70 2024-03-30T04:18:48.423 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:48.425+0000 7fbff5600700 1 -- 172.21.15.135:0/1459339104 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 14) v1 ==== 43439+0+0 (secure 0 0 0) 0x7fbfec01c7a0 con 0x7fbff80fca70 2024-03-30T04:18:48.424 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:48.425+0000 7fc000600700 1 -- 172.21.15.135:0/1459339104 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbff8102e40 con 0x7fbff80fca70 2024-03-30T04:18:48.424 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:48.425+0000 7fbff5600700 1 --2- 172.21.15.135:0/1459339104 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7fbfe0036960 0x7fbfe0038e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:18:48.425 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:48.425+0000 7fbff5600700 1 -- 172.21.15.135:0/1459339104 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fbfec04aaa0 con 0x7fbff80fca70 2024-03-30T04:18:48.425 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:48.425+0000 7fbffd200700 1 -- 172.21.15.135:0/1459339104 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7fbfe0036960 msgr2=0x7fbfe0038e10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.123:6800/70490328 2024-03-30T04:18:48.425 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:48.426+0000 7fbffd200700 1 --2- 172.21.15.135:0/1459339104 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7fbfe0036960 0x7fbfe0038e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-30T04:18:48.430 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:48.431+0000 7fbff5600700 1 -- 172.21.15.135:0/1459339104 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fbfec011dd0 con 0x7fbff80fca70 2024-03-30T04:18:48.626 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:48.627+0000 7fbffd200700 1 -- 172.21.15.135:0/1459339104 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7fbfe0036960 msgr2=0x7fbfe0038e10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.123:6800/70490328 2024-03-30T04:18:48.626 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:48.627+0000 7fbffd200700 1 --2- 172.21.15.135:0/1459339104 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7fbfe0036960 0x7fbfe0038e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-30T04:18:48.698 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:48.699+0000 7fc000600700 1 -- 172.21.15.135:0/1459339104 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbff81106d0 con 0x7fbff80fca70 2024-03-30T04:18:48.699 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:48.700+0000 7fbff5600700 1 -- 172.21.15.135:0/1459339104 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fbfec011370 con 0x7fbff80fca70 2024-03-30T04:18:48.699 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:18:48.699 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"2e77a2be-ee4c-11ee-b647-cb9ed24678a4","modified":"2024-03-30T04:16:26.797409Z","created":"2024-03-30T04:16:26.797409Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-30T04:18:48.701 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:48.702+0000 7fc000600700 1 -- 172.21.15.135:0/1459339104 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7fbfe0036960 msgr2=0x7fbfe0038e10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-30T04:18:48.701 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:48.702+0000 7fc000600700 1 --2- 172.21.15.135:0/1459339104 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7fbfe0036960 0x7fbfe0038e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:18:48.701 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:48.702+0000 7fc000600700 1 -- 172.21.15.135:0/1459339104 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbff80fca70 msgr2=0x7fbff810f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:18:48.702 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:48.702+0000 7fc000600700 1 --2- 172.21.15.135:0/1459339104 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbff80fca70 0x7fbff810f1e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fbfec000c00 tx=0x7fbfec009e30 comp rx=0 tx=0).stop 2024-03-30T04:18:48.702 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:48.702+0000 7fc000600700 1 -- 172.21.15.135:0/1459339104 shutdown_connections 2024-03-30T04:18:48.702 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:48.702+0000 7fc000600700 1 --2- 172.21.15.135:0/1459339104 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7fbfe0036960 0x7fbfe0038e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:18:48.702 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:48.702+0000 7fc000600700 1 --2- 172.21.15.135:0/1459339104 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbff80fca70 0x7fbff810f1e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:18:48.702 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:48.702+0000 7fc000600700 1 -- 172.21.15.135:0/1459339104 >> 172.21.15.135:0/1459339104 conn(0x7fbff80f81d0 msgr2=0x7fbff80fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:18:48.702 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:48.702+0000 7fc000600700 1 -- 172.21.15.135:0/1459339104 shutdown_connections 2024-03-30T04:18:48.702 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:48.702+0000 7fc000600700 1 -- 172.21.15.135:0/1459339104 wait complete. 2024-03-30T04:18:48.702 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2024-03-30T04:18:49.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:49 smithi123 ceph-mon[35245]: from='mgr.14160 172.21.15.123:0/3759324081' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-30T04:18:49.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:49 smithi123 ceph-mon[35245]: mgrmap e14: smithi123.xgksqq(active, since 66s) 2024-03-30T04:18:49.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:49 smithi123 ceph-mon[35245]: from='client.? 172.21.15.135:0/1459339104' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T04:18:50.353 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-30T04:18:50.353 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-03-30T04:18:50.762 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-30T04:18:50.851 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-30T04:18:52.510 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:52.511+0000 7fc021a00700 1 -- 172.21.15.135:0/4009705974 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc01c0fca70 msgr2=0x7fc01c0fce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:18:52.511 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:52.511+0000 7fc021a00700 1 --2- 172.21.15.135:0/4009705974 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc01c0fca70 0x7fc01c0fce80 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc00c0048c0 tx=0x7fc00c009e30 comp rx=0 tx=0).stop 2024-03-30T04:18:52.511 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:52.511+0000 7fc021a00700 1 -- 172.21.15.135:0/4009705974 shutdown_connections 2024-03-30T04:18:52.511 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:52.511+0000 7fc021a00700 1 --2- 172.21.15.135:0/4009705974 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc01c0fca70 0x7fc01c0fce80 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:18:52.511 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:52.511+0000 7fc021a00700 1 -- 172.21.15.135:0/4009705974 >> 172.21.15.135:0/4009705974 conn(0x7fc01c0f81d0 msgr2=0x7fc01c0fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:18:52.511 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:52.511+0000 7fc021a00700 1 -- 172.21.15.135:0/4009705974 shutdown_connections 2024-03-30T04:18:52.511 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:52.511+0000 7fc021a00700 1 -- 172.21.15.135:0/4009705974 wait complete. 2024-03-30T04:18:52.511 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:52.512+0000 7fc021a00700 1 Processor -- start 2024-03-30T04:18:52.511 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:52.512+0000 7fc021a00700 1 -- start start 2024-03-30T04:18:52.512 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:52.512+0000 7fc021a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc01c0fca70 0x7fc01c10f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:18:52.513 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:52.512+0000 7fc021a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc01c10f740 con 0x7fc01c0fca70 2024-03-30T04:18:52.513 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:52.512+0000 7fc01aa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc01c0fca70 0x7fc01c10f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:18:52.513 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:52.512+0000 7fc01aa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc01c0fca70 0x7fc01c10f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:55800/0 (socket says 172.21.15.135:55800) 2024-03-30T04:18:52.513 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:52.512+0000 7fc01aa00700 1 -- 172.21.15.135:0/2892181471 learned_addr learned my addr 172.21.15.135:0/2892181471 (peer_addr_for_me v2:172.21.15.135:0/0) 2024-03-30T04:18:52.513 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:52.513+0000 7fc01aa00700 1 -- 172.21.15.135:0/2892181471 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc00c005040 con 0x7fc01c0fca70 2024-03-30T04:18:52.513 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:52.513+0000 7fc01aa00700 1 --2- 172.21.15.135:0/2892181471 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc01c0fca70 0x7fc01c10f200 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc00c000c00 tx=0x7fc00c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:18:52.513 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:52.513+0000 7fc012a00700 1 -- 172.21.15.135:0/2892181471 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc00c009620 con 0x7fc01c0fca70 2024-03-30T04:18:52.513 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:52.513+0000 7fc012a00700 1 -- 172.21.15.135:0/2892181471 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc00c009780 con 0x7fc01c0fca70 2024-03-30T04:18:52.513 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:52.513+0000 7fc021a00700 1 -- 172.21.15.135:0/2892181471 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc01c10f940 con 0x7fc01c0fca70 2024-03-30T04:18:52.513 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:52.513+0000 7fc012a00700 1 -- 172.21.15.135:0/2892181471 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc00c01c5b0 con 0x7fc01c0fca70 2024-03-30T04:18:52.513 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:52.513+0000 7fc021a00700 1 -- 172.21.15.135:0/2892181471 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc01c1104b0 con 0x7fc01c0fca70 2024-03-30T04:18:52.514 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:52.514+0000 7fc021a00700 1 -- 172.21.15.135:0/2892181471 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc01c102e40 con 0x7fc01c0fca70 2024-03-30T04:18:52.514 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:52.514+0000 7fc012a00700 1 -- 172.21.15.135:0/2892181471 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 14) v1 ==== 43439+0+0 (secure 0 0 0) 0x7fc00c013a60 con 0x7fc01c0fca70 2024-03-30T04:18:52.514 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:52.515+0000 7fc012a00700 1 --2- 172.21.15.135:0/2892181471 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7fc0000368d0 0x7fc000038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:18:52.514 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:52.515+0000 7fc012a00700 1 -- 172.21.15.135:0/2892181471 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fc00c049bc0 con 0x7fc01c0fca70 2024-03-30T04:18:52.514 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:52.515+0000 7fc01a000700 1 -- 172.21.15.135:0/2892181471 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7fc0000368d0 msgr2=0x7fc000038d80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.123:6800/70490328 2024-03-30T04:18:52.515 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:52.515+0000 7fc01a000700 1 --2- 172.21.15.135:0/2892181471 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7fc0000368d0 0x7fc000038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-30T04:18:52.519 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:52.520+0000 7fc012a00700 1 -- 172.21.15.135:0/2892181471 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fc00c025400 con 0x7fc01c0fca70 2024-03-30T04:18:52.715 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:52.716+0000 7fc01a000700 1 -- 172.21.15.135:0/2892181471 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7fc0000368d0 msgr2=0x7fc000038d80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.123:6800/70490328 2024-03-30T04:18:52.715 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:52.716+0000 7fc01a000700 1 --2- 172.21.15.135:0/2892181471 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7fc0000368d0 0x7fc000038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-30T04:18:52.787 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:52.788+0000 7fc021a00700 1 -- 172.21.15.135:0/2892181471 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc01c1107b0 con 0x7fc01c0fca70 2024-03-30T04:18:52.788 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:52.789+0000 7fc012a00700 1 -- 172.21.15.135:0/2892181471 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fc00c011b60 con 0x7fc01c0fca70 2024-03-30T04:18:52.788 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:18:52.788 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"2e77a2be-ee4c-11ee-b647-cb9ed24678a4","modified":"2024-03-30T04:16:26.797409Z","created":"2024-03-30T04:16:26.797409Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-30T04:18:52.790 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:52.791+0000 7fc021a00700 1 -- 172.21.15.135:0/2892181471 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7fc0000368d0 msgr2=0x7fc000038d80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-30T04:18:52.790 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:52.791+0000 7fc021a00700 1 --2- 172.21.15.135:0/2892181471 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7fc0000368d0 0x7fc000038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:18:52.790 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:52.791+0000 7fc021a00700 1 -- 172.21.15.135:0/2892181471 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc01c0fca70 msgr2=0x7fc01c10f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:18:52.790 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:52.791+0000 7fc021a00700 1 --2- 172.21.15.135:0/2892181471 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc01c0fca70 0x7fc01c10f200 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc00c000c00 tx=0x7fc00c009e30 comp rx=0 tx=0).stop 2024-03-30T04:18:52.790 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:52.791+0000 7fc021a00700 1 -- 172.21.15.135:0/2892181471 shutdown_connections 2024-03-30T04:18:52.790 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:52.791+0000 7fc021a00700 1 --2- 172.21.15.135:0/2892181471 >> [v2:172.21.15.123:6800/70490328,v1:172.21.15.123:6801/70490328] conn(0x7fc0000368d0 0x7fc000038d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:18:52.791 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:52.791+0000 7fc021a00700 1 --2- 172.21.15.135:0/2892181471 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc01c0fca70 0x7fc01c10f200 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:18:52.791 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:52.791+0000 7fc021a00700 1 -- 172.21.15.135:0/2892181471 >> 172.21.15.135:0/2892181471 conn(0x7fc01c0f81d0 msgr2=0x7fc01c0fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:18:52.791 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:52.791+0000 7fc021a00700 1 -- 172.21.15.135:0/2892181471 shutdown_connections 2024-03-30T04:18:52.791 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:52.791+0000 7fc021a00700 1 -- 172.21.15.135:0/2892181471 wait complete. 2024-03-30T04:18:52.791 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2024-03-30T04:18:52.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:52 smithi123 ceph-mon[35245]: from='client.? 172.21.15.135:0/2892181471' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T04:18:54.444 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-30T04:18:54.444 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-03-30T04:18:54.851 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-30T04:18:54.939 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-30T04:18:56.341 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:56 smithi123 ceph-mon[35245]: Active manager daemon smithi123.xgksqq restarted 2024-03-30T04:18:56.341 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:56 smithi123 ceph-mon[35245]: Activating manager daemon smithi123.xgksqq 2024-03-30T04:18:56.341 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:56 smithi123 ceph-mon[35245]: osdmap e5: 0 total, 0 up, 0 in 2024-03-30T04:18:56.341 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:56 smithi123 ceph-mon[35245]: mgrmap e15: smithi123.xgksqq(active, starting, since 0.00404936s) 2024-03-30T04:18:56.341 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:56 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mon metadata", "id": "smithi123"}]: dispatch 2024-03-30T04:18:56.341 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:56 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mgr metadata", "who": "smithi123.xgksqq", "id": "smithi123.xgksqq"}]: dispatch 2024-03-30T04:18:56.341 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:56 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-30T04:18:56.341 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:56 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-30T04:18:56.341 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:56 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-30T04:18:56.341 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:56 smithi123 ceph-mon[35245]: Manager daemon smithi123.xgksqq is now available 2024-03-30T04:18:56.341 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:56 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:18:56.342 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:56 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:18:56.662 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:56.663+0000 7f07ff800700 1 -- 172.21.15.135:0/926890926 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07f80fca70 msgr2=0x7f07f80fce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:18:56.662 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:56.663+0000 7f07ff800700 1 --2- 172.21.15.135:0/926890926 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07f80fca70 0x7f07f80fce80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f07ec0048c0 tx=0x7f07ec009e30 comp rx=0 tx=0).stop 2024-03-30T04:18:56.662 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:56.663+0000 7f07ff800700 1 -- 172.21.15.135:0/926890926 shutdown_connections 2024-03-30T04:18:56.663 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:56.663+0000 7f07ff800700 1 --2- 172.21.15.135:0/926890926 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07f80fca70 0x7f07f80fce80 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:18:56.663 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:56.663+0000 7f07ff800700 1 -- 172.21.15.135:0/926890926 >> 172.21.15.135:0/926890926 conn(0x7f07f80f81d0 msgr2=0x7f07f80fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:18:56.663 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:56.663+0000 7f07ff800700 1 -- 172.21.15.135:0/926890926 shutdown_connections 2024-03-30T04:18:56.663 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:56.663+0000 7f07ff800700 1 -- 172.21.15.135:0/926890926 wait complete. 2024-03-30T04:18:56.663 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:56.664+0000 7f07ff800700 1 Processor -- start 2024-03-30T04:18:56.664 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:56.664+0000 7f07ff800700 1 -- start start 2024-03-30T04:18:56.664 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:56.665+0000 7f07ff800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07f80fca70 0x7f07f810f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:18:56.664 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:56.665+0000 7f07ff800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07f810f740 con 0x7f07f80fca70 2024-03-30T04:18:56.664 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:56.665+0000 7f07fce00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07f80fca70 0x7f07f810f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:18:56.664 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:56.665+0000 7f07fce00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07f80fca70 0x7f07f810f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:55828/0 (socket says 172.21.15.135:55828) 2024-03-30T04:18:56.664 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:56.665+0000 7f07fce00700 1 -- 172.21.15.135:0/3368727647 learned_addr learned my addr 172.21.15.135:0/3368727647 (peer_addr_for_me v2:172.21.15.135:0/0) 2024-03-30T04:18:56.665 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:56.665+0000 7f07fce00700 1 -- 172.21.15.135:0/3368727647 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07ec005040 con 0x7f07f80fca70 2024-03-30T04:18:56.665 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:56.665+0000 7f07fce00700 1 --2- 172.21.15.135:0/3368727647 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07f80fca70 0x7f07f810f200 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f07ec000c00 tx=0x7f07ec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:18:56.665 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:56.666+0000 7f07f4c00700 1 -- 172.21.15.135:0/3368727647 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07ec009620 con 0x7f07f80fca70 2024-03-30T04:18:56.665 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:56.666+0000 7f07f4c00700 1 -- 172.21.15.135:0/3368727647 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f07ec009780 con 0x7f07f80fca70 2024-03-30T04:18:56.665 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:56.666+0000 7f07f4c00700 1 -- 172.21.15.135:0/3368727647 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07ec01c5b0 con 0x7f07f80fca70 2024-03-30T04:18:56.665 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:56.666+0000 7f07ff800700 1 -- 172.21.15.135:0/3368727647 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07f810f940 con 0x7f07f80fca70 2024-03-30T04:18:56.665 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:56.666+0000 7f07ff800700 1 -- 172.21.15.135:0/3368727647 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07f81104b0 con 0x7f07f80fca70 2024-03-30T04:18:56.666 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:56.667+0000 7f07f4c00700 1 -- 172.21.15.135:0/3368727647 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 15) v1 ==== 43162+0+0 (secure 0 0 0) 0x7f07ec013a60 con 0x7f07f80fca70 2024-03-30T04:18:56.666 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:56.667+0000 7f07ff800700 1 -- 172.21.15.135:0/3368727647 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07f8102e40 con 0x7f07f80fca70 2024-03-30T04:18:56.666 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:56.667+0000 7f07f4c00700 1 -- 172.21.15.135:0/3368727647 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f07ec049830 con 0x7f07f80fca70 2024-03-30T04:18:56.671 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:56.672+0000 7f07f4c00700 1 -- 172.21.15.135:0/3368727647 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f07ec023300 con 0x7f07f80fca70 2024-03-30T04:18:56.966 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:56.967+0000 7f07ff800700 1 -- 172.21.15.135:0/3368727647 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f07f81106f0 con 0x7f07f80fca70 2024-03-30T04:18:56.967 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:56.968+0000 7f07f4c00700 1 -- 172.21.15.135:0/3368727647 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f07ec011b60 con 0x7f07f80fca70 2024-03-30T04:18:56.967 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:18:56.968 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"2e77a2be-ee4c-11ee-b647-cb9ed24678a4","modified":"2024-03-30T04:16:26.797409Z","created":"2024-03-30T04:16:26.797409Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-30T04:18:56.969 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:56.970+0000 7f07ff800700 1 -- 172.21.15.135:0/3368727647 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07f80fca70 msgr2=0x7f07f810f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:18:56.969 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:56.970+0000 7f07ff800700 1 --2- 172.21.15.135:0/3368727647 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07f80fca70 0x7f07f810f200 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f07ec000c00 tx=0x7f07ec009e30 comp rx=0 tx=0).stop 2024-03-30T04:18:56.969 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:56.970+0000 7f07ff800700 1 -- 172.21.15.135:0/3368727647 shutdown_connections 2024-03-30T04:18:56.970 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:56.970+0000 7f07ff800700 1 --2- 172.21.15.135:0/3368727647 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07f80fca70 0x7f07f810f200 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:18:56.970 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:56.970+0000 7f07ff800700 1 -- 172.21.15.135:0/3368727647 >> 172.21.15.135:0/3368727647 conn(0x7f07f80f81d0 msgr2=0x7f07f80fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:18:56.970 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:56.970+0000 7f07ff800700 1 -- 172.21.15.135:0/3368727647 shutdown_connections 2024-03-30T04:18:56.970 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:18:56.971+0000 7f07ff800700 1 -- 172.21.15.135:0/3368727647 wait complete. 2024-03-30T04:18:56.970 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2024-03-30T04:18:57.596 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:57 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:18:57.596 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:57 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:18:57.596 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:57 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi123.xgksqq/mirror_snapshot_schedule"}]: dispatch 2024-03-30T04:18:57.596 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:57 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi123.xgksqq/trash_purge_schedule"}]: dispatch 2024-03-30T04:18:57.597 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:57 smithi123 ceph-mon[35245]: from='client.? 172.21.15.135:0/3368727647' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T04:18:57.597 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:57 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:18:57.597 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:57 smithi123 ceph-mon[35245]: mgrmap e16: smithi123.xgksqq(active, since 1.00789s) 2024-03-30T04:18:58.651 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-30T04:18:58.652 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-03-30T04:18:58.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:58 smithi123 ceph-mon[35245]: [30/Mar/2024:04:18:57] ENGINE Bus STARTING 2024-03-30T04:18:58.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:58 smithi123 ceph-mon[35245]: [30/Mar/2024:04:18:57] ENGINE Serving on https://172.21.15.123:7150 2024-03-30T04:18:58.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:58 smithi123 ceph-mon[35245]: [30/Mar/2024:04:18:57] ENGINE Bus STARTED 2024-03-30T04:18:58.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:58 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:18:58.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:58 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:18:59.065 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-30T04:18:59.157 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-30T04:19:00.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:59 smithi123 ceph-mon[35245]: mgrmap e17: smithi123.xgksqq(active, since 2s) 2024-03-30T04:19:00.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:18:59 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:01.735 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:01.735+0000 7f5f2b800700 1 -- 172.21.15.135:0/1697516528 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f24071ea0 msgr2=0x7f5f240703b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:19:01.735 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:01.735+0000 7f5f2b800700 1 --2- 172.21.15.135:0/1697516528 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f24071ea0 0x7f5f240703b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5f180048c0 tx=0x7f5f18009e30 comp rx=0 tx=0).stop 2024-03-30T04:19:01.735 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:01.735+0000 7f5f2b800700 1 -- 172.21.15.135:0/1697516528 shutdown_connections 2024-03-30T04:19:01.735 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:01.735+0000 7f5f2b800700 1 --2- 172.21.15.135:0/1697516528 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f24071ea0 0x7f5f240703b0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:19:01.735 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:01.735+0000 7f5f2b800700 1 -- 172.21.15.135:0/1697516528 >> 172.21.15.135:0/1697516528 conn(0x7f5f240f81d0 msgr2=0x7f5f240fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:19:01.735 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:01.735+0000 7f5f2b800700 1 -- 172.21.15.135:0/1697516528 shutdown_connections 2024-03-30T04:19:01.736 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:01.735+0000 7f5f2b800700 1 -- 172.21.15.135:0/1697516528 wait complete. 2024-03-30T04:19:01.739 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:01.736+0000 7f5f2b800700 1 Processor -- start 2024-03-30T04:19:01.739 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:01.736+0000 7f5f2b800700 1 -- start start 2024-03-30T04:19:01.739 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:01.737+0000 7f5f2b800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f24071ea0 0x7f5f2410f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:19:01.740 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:01.737+0000 7f5f2b800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2410f780 con 0x7f5f24071ea0 2024-03-30T04:19:01.740 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:01.737+0000 7f5f28e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f24071ea0 0x7f5f2410f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:19:01.740 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:01.737+0000 7f5f28e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f24071ea0 0x7f5f2410f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:56354/0 (socket says 172.21.15.135:56354) 2024-03-30T04:19:01.740 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:01.737+0000 7f5f28e00700 1 -- 172.21.15.135:0/39777255 learned_addr learned my addr 172.21.15.135:0/39777255 (peer_addr_for_me v2:172.21.15.135:0/0) 2024-03-30T04:19:01.740 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:01.738+0000 7f5f28e00700 1 -- 172.21.15.135:0/39777255 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f18005040 con 0x7f5f24071ea0 2024-03-30T04:19:01.740 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:01.738+0000 7f5f28e00700 1 --2- 172.21.15.135:0/39777255 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f24071ea0 0x7f5f2410f240 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5f1800b860 tx=0x7f5f18009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:19:01.740 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:01.738+0000 7f5f20c00700 1 -- 172.21.15.135:0/39777255 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f18009580 con 0x7f5f24071ea0 2024-03-30T04:19:01.741 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:01.738+0000 7f5f20c00700 1 -- 172.21.15.135:0/39777255 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5f180096e0 con 0x7f5f24071ea0 2024-03-30T04:19:01.741 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:01.738+0000 7f5f2b800700 1 -- 172.21.15.135:0/39777255 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f2410f980 con 0x7f5f24071ea0 2024-03-30T04:19:01.741 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:01.738+0000 7f5f20c00700 1 -- 172.21.15.135:0/39777255 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f1801c5b0 con 0x7f5f24071ea0 2024-03-30T04:19:01.741 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:01.738+0000 7f5f2b800700 1 -- 172.21.15.135:0/39777255 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f241104f0 con 0x7f5f24071ea0 2024-03-30T04:19:01.741 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:01.739+0000 7f5f2b800700 1 -- 172.21.15.135:0/39777255 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f24105050 con 0x7f5f24071ea0 2024-03-30T04:19:01.741 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:01.739+0000 7f5f20c00700 1 -- 172.21.15.135:0/39777255 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 43483+0+0 (secure 0 0 0) 0x7f5f18013420 con 0x7f5f24071ea0 2024-03-30T04:19:01.741 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:01.740+0000 7f5f20c00700 1 --2- 172.21.15.135:0/39777255 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f5f0c0369c0 0x7f5f0c038e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:19:01.741 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:01.740+0000 7f5f20c00700 1 -- 172.21.15.135:0/39777255 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f5f18049cf0 con 0x7f5f24071ea0 2024-03-30T04:19:01.742 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:01.740+0000 7f5f23e00700 1 --2- 172.21.15.135:0/39777255 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f5f0c0369c0 0x7f5f0c038e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:19:01.742 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:01.741+0000 7f5f23e00700 1 --2- 172.21.15.135:0/39777255 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f5f0c0369c0 0x7f5f0c038e70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f5f10009990 tx=0x7f5f10006e40 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:19:01.744 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:01.745+0000 7f5f20c00700 1 -- 172.21.15.135:0/39777255 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f5f18011550 con 0x7f5f24071ea0 2024-03-30T04:19:02.027 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:02.027+0000 7f5f2b800700 1 -- 172.21.15.135:0/39777255 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5f2405f5f0 con 0x7f5f24071ea0 2024-03-30T04:19:02.027 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:02.028+0000 7f5f20c00700 1 -- 172.21.15.135:0/39777255 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f5f18011370 con 0x7f5f24071ea0 2024-03-30T04:19:02.028 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:19:02.028 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"2e77a2be-ee4c-11ee-b647-cb9ed24678a4","modified":"2024-03-30T04:16:26.797409Z","created":"2024-03-30T04:16:26.797409Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-30T04:19:02.029 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:02.030+0000 7f5f2b800700 1 -- 172.21.15.135:0/39777255 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f5f0c0369c0 msgr2=0x7f5f0c038e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:19:02.030 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:02.030+0000 7f5f2b800700 1 --2- 172.21.15.135:0/39777255 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f5f0c0369c0 0x7f5f0c038e70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f5f10009990 tx=0x7f5f10006e40 comp rx=0 tx=0).stop 2024-03-30T04:19:02.030 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:02.030+0000 7f5f2b800700 1 -- 172.21.15.135:0/39777255 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f24071ea0 msgr2=0x7f5f2410f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:19:02.030 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:02.030+0000 7f5f2b800700 1 --2- 172.21.15.135:0/39777255 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f24071ea0 0x7f5f2410f240 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5f1800b860 tx=0x7f5f18009e30 comp rx=0 tx=0).stop 2024-03-30T04:19:02.030 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:02.030+0000 7f5f2b800700 1 -- 172.21.15.135:0/39777255 shutdown_connections 2024-03-30T04:19:02.030 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:02.030+0000 7f5f2b800700 1 --2- 172.21.15.135:0/39777255 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f5f0c0369c0 0x7f5f0c038e70 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:19:02.030 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:02.030+0000 7f5f2b800700 1 --2- 172.21.15.135:0/39777255 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f24071ea0 0x7f5f2410f240 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:19:02.030 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:02.030+0000 7f5f2b800700 1 -- 172.21.15.135:0/39777255 >> 172.21.15.135:0/39777255 conn(0x7f5f240f81d0 msgr2=0x7f5f240f8e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:19:02.030 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:02.031+0000 7f5f2b800700 1 -- 172.21.15.135:0/39777255 shutdown_connections 2024-03-30T04:19:02.030 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:02.031+0000 7f5f2b800700 1 -- 172.21.15.135:0/39777255 wait complete. 2024-03-30T04:19:02.031 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2024-03-30T04:19:02.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:02 smithi123 ceph-mon[35245]: from='client.? 172.21.15.135:0/39777255' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T04:19:03.787 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-30T04:19:03.788 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-03-30T04:19:04.189 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-30T04:19:04.278 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-30T04:19:05.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:05 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:05.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:05 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:05.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:05 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:05.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:05 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:05.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:05 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "who": "osd/host:smithi135", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:19:05.992 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:05.993+0000 7fea6da00700 1 -- 172.21.15.135:0/744183379 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea680fca70 msgr2=0x7fea680fce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:19:05.992 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:05.993+0000 7fea6da00700 1 --2- 172.21.15.135:0/744183379 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea680fca70 0x7fea680fce80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fea580048c0 tx=0x7fea58009e30 comp rx=0 tx=0).stop 2024-03-30T04:19:05.992 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:05.993+0000 7fea6da00700 1 -- 172.21.15.135:0/744183379 shutdown_connections 2024-03-30T04:19:05.992 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:05.993+0000 7fea6da00700 1 --2- 172.21.15.135:0/744183379 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea680fca70 0x7fea680fce80 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:19:05.993 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:05.993+0000 7fea6da00700 1 -- 172.21.15.135:0/744183379 >> 172.21.15.135:0/744183379 conn(0x7fea680f81d0 msgr2=0x7fea680fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:19:05.993 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:05.993+0000 7fea6da00700 1 -- 172.21.15.135:0/744183379 shutdown_connections 2024-03-30T04:19:05.993 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:05.993+0000 7fea6da00700 1 -- 172.21.15.135:0/744183379 wait complete. 2024-03-30T04:19:05.993 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:05.994+0000 7fea6da00700 1 Processor -- start 2024-03-30T04:19:05.994 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:05.994+0000 7fea6da00700 1 -- start start 2024-03-30T04:19:05.994 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:05.994+0000 7fea6da00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea680fca70 0x7fea6810f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:19:05.994 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:05.994+0000 7fea6da00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea6810f720 con 0x7fea680fca70 2024-03-30T04:19:05.994 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:05.995+0000 7fea66a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea680fca70 0x7fea6810f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:19:05.994 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:05.995+0000 7fea66a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea680fca70 0x7fea6810f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:56370/0 (socket says 172.21.15.135:56370) 2024-03-30T04:19:05.994 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:05.995+0000 7fea66a00700 1 -- 172.21.15.135:0/2803390229 learned_addr learned my addr 172.21.15.135:0/2803390229 (peer_addr_for_me v2:172.21.15.135:0/0) 2024-03-30T04:19:05.995 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:05.995+0000 7fea66a00700 1 -- 172.21.15.135:0/2803390229 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea58005040 con 0x7fea680fca70 2024-03-30T04:19:05.995 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:05.995+0000 7fea66a00700 1 --2- 172.21.15.135:0/2803390229 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea680fca70 0x7fea6810f1e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fea58000c00 tx=0x7fea58009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:19:05.995 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:05.996+0000 7fea5ea00700 1 -- 172.21.15.135:0/2803390229 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea58009620 con 0x7fea680fca70 2024-03-30T04:19:05.995 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:05.996+0000 7fea5ea00700 1 -- 172.21.15.135:0/2803390229 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fea58009780 con 0x7fea680fca70 2024-03-30T04:19:05.995 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:05.996+0000 7fea6da00700 1 -- 172.21.15.135:0/2803390229 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea6810f920 con 0x7fea680fca70 2024-03-30T04:19:05.995 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:05.996+0000 7fea5ea00700 1 -- 172.21.15.135:0/2803390229 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea5801c5b0 con 0x7fea680fca70 2024-03-30T04:19:05.995 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:05.996+0000 7fea6da00700 1 -- 172.21.15.135:0/2803390229 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea68110490 con 0x7fea680fca70 2024-03-30T04:19:05.996 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:05.997+0000 7fea5ea00700 1 -- 172.21.15.135:0/2803390229 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 43483+0+0 (secure 0 0 0) 0x7fea58013a60 con 0x7fea680fca70 2024-03-30T04:19:05.996 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:05.997+0000 7fea6da00700 1 -- 172.21.15.135:0/2803390229 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea68102e40 con 0x7fea680fca70 2024-03-30T04:19:05.996 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:05.997+0000 7fea5ea00700 1 --2- 172.21.15.135:0/2803390229 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fea4c0369c0 0x7fea4c038e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:19:05.996 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:05.997+0000 7fea5ea00700 1 -- 172.21.15.135:0/2803390229 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fea58049d20 con 0x7fea680fca70 2024-03-30T04:19:05.996 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:05.998+0000 7fea66000700 1 --2- 172.21.15.135:0/2803390229 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fea4c0369c0 0x7fea4c038e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:19:05.997 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:05.998+0000 7fea66000700 1 --2- 172.21.15.135:0/2803390229 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fea4c0369c0 0x7fea4c038e70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fea54009990 tx=0x7fea54006e40 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:19:06.001 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:06.003+0000 7fea5ea00700 1 -- 172.21.15.135:0/2803390229 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fea58025400 con 0x7fea680fca70 2024-03-30T04:19:06.270 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:06.271+0000 7fea6da00700 1 -- 172.21.15.135:0/2803390229 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fea681106d0 con 0x7fea680fca70 2024-03-30T04:19:06.271 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:06.272+0000 7fea5ea00700 1 -- 172.21.15.135:0/2803390229 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fea58011b60 con 0x7fea680fca70 2024-03-30T04:19:06.271 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:19:06.271 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"2e77a2be-ee4c-11ee-b647-cb9ed24678a4","modified":"2024-03-30T04:16:26.797409Z","created":"2024-03-30T04:16:26.797409Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-30T04:19:06.273 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:06.274+0000 7fea6da00700 1 -- 172.21.15.135:0/2803390229 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fea4c0369c0 msgr2=0x7fea4c038e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:19:06.273 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:06.274+0000 7fea6da00700 1 --2- 172.21.15.135:0/2803390229 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fea4c0369c0 0x7fea4c038e70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fea54009990 tx=0x7fea54006e40 comp rx=0 tx=0).stop 2024-03-30T04:19:06.273 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:06.274+0000 7fea6da00700 1 -- 172.21.15.135:0/2803390229 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea680fca70 msgr2=0x7fea6810f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:19:06.273 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:06.274+0000 7fea6da00700 1 --2- 172.21.15.135:0/2803390229 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea680fca70 0x7fea6810f1e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fea58000c00 tx=0x7fea58009e30 comp rx=0 tx=0).stop 2024-03-30T04:19:06.273 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:06.274+0000 7fea6da00700 1 -- 172.21.15.135:0/2803390229 shutdown_connections 2024-03-30T04:19:06.273 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:06.274+0000 7fea6da00700 1 --2- 172.21.15.135:0/2803390229 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fea4c0369c0 0x7fea4c038e70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:19:06.273 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:06.274+0000 7fea6da00700 1 --2- 172.21.15.135:0/2803390229 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea680fca70 0x7fea6810f1e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:19:06.274 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:06.274+0000 7fea6da00700 1 -- 172.21.15.135:0/2803390229 >> 172.21.15.135:0/2803390229 conn(0x7fea680f81d0 msgr2=0x7fea680fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:19:06.274 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:06.274+0000 7fea6da00700 1 -- 172.21.15.135:0/2803390229 shutdown_connections 2024-03-30T04:19:06.274 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:06.274+0000 7fea6da00700 1 -- 172.21.15.135:0/2803390229 wait complete. 2024-03-30T04:19:06.274 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2024-03-30T04:19:06.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:06 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:06.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:06 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:06.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:06 smithi123 ceph-mon[35245]: from='client.? 172.21.15.135:0/2803390229' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T04:19:07.963 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-30T04:19:07.963 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-03-30T04:19:07.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:07 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:07.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:07 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:07.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:07 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "who": "osd/host:smithi123", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:19:07.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:07 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:19:07.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:07 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:19:07.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:07 smithi123 ceph-mon[35245]: Updating smithi123:/etc/ceph/ceph.conf 2024-03-30T04:19:07.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:07 smithi123 ceph-mon[35245]: Updating smithi135:/etc/ceph/ceph.conf 2024-03-30T04:19:07.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:07 smithi123 ceph-mon[35245]: Updating smithi123:/var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T04:19:08.377 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T04:19:08.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:08 smithi123 ceph-mon[35245]: Updating smithi135:/var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T04:19:08.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:08 smithi123 ceph-mon[35245]: Updating smithi123:/etc/ceph/ceph.client.admin.keyring 2024-03-30T04:19:08.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:08 smithi123 ceph-mon[35245]: Updating smithi135:/etc/ceph/ceph.client.admin.keyring 2024-03-30T04:19:10.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:09 smithi123 ceph-mon[35245]: Updating smithi123:/var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T04:19:10.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:09 smithi123 ceph-mon[35245]: Updating smithi135:/var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T04:19:10.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:09 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:10.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:09 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:10.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:09 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:10.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:09 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:10.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:09 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:10.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:09 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi135", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-30T04:19:10.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:09 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi135", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-03-30T04:19:10.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:09 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:19:10.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:09 smithi123 ceph-mon[35245]: Deploying daemon crash.smithi135 on smithi135 2024-03-30T04:19:10.833 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:10.833+0000 7efddb800700 1 -- 172.21.15.135:0/3183915459 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efdd4102ac0 msgr2=0x7efdd4102e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:19:10.833 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:10.833+0000 7efddb800700 1 --2- 172.21.15.135:0/3183915459 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efdd4102ac0 0x7efdd4102e90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7efdc80048c0 tx=0x7efdc8009e30 comp rx=0 tx=0).stop 2024-03-30T04:19:10.833 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:10.833+0000 7efddb800700 1 -- 172.21.15.135:0/3183915459 shutdown_connections 2024-03-30T04:19:10.833 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:10.833+0000 7efddb800700 1 --2- 172.21.15.135:0/3183915459 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efdd4102ac0 0x7efdd4102e90 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:19:10.833 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:10.833+0000 7efddb800700 1 -- 172.21.15.135:0/3183915459 >> 172.21.15.135:0/3183915459 conn(0x7efdd40f85e0 msgr2=0x7efdd40fa9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:19:10.833 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:10.833+0000 7efddb800700 1 -- 172.21.15.135:0/3183915459 shutdown_connections 2024-03-30T04:19:10.834 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:10.833+0000 7efddb800700 1 -- 172.21.15.135:0/3183915459 wait complete. 2024-03-30T04:19:10.834 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:10.834+0000 7efddb800700 1 Processor -- start 2024-03-30T04:19:10.834 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:10.834+0000 7efddb800700 1 -- start start 2024-03-30T04:19:10.834 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:10.835+0000 7efddb800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efdd4102ac0 0x7efdd41151e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:19:10.834 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:10.835+0000 7efddb800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdd4115720 con 0x7efdd4102ac0 2024-03-30T04:19:10.835 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:10.835+0000 7efdd8e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efdd4102ac0 0x7efdd41151e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:19:10.835 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:10.835+0000 7efdd8e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efdd4102ac0 0x7efdd41151e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:37298/0 (socket says 172.21.15.135:37298) 2024-03-30T04:19:10.835 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:10.835+0000 7efdd8e00700 1 -- 172.21.15.135:0/545074684 learned_addr learned my addr 172.21.15.135:0/545074684 (peer_addr_for_me v2:172.21.15.135:0/0) 2024-03-30T04:19:10.835 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:10.836+0000 7efdd8e00700 1 -- 172.21.15.135:0/545074684 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc8005040 con 0x7efdd4102ac0 2024-03-30T04:19:10.835 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:10.836+0000 7efdd8e00700 1 --2- 172.21.15.135:0/545074684 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efdd4102ac0 0x7efdd41151e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7efdc8007e30 tx=0x7efdc8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:19:10.835 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:10.836+0000 7efdd0c00700 1 -- 172.21.15.135:0/545074684 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efdc80093c0 con 0x7efdd4102ac0 2024-03-30T04:19:10.836 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:10.836+0000 7efdd0c00700 1 -- 172.21.15.135:0/545074684 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efdc8013420 con 0x7efdd4102ac0 2024-03-30T04:19:10.836 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:10.836+0000 7efddb800700 1 -- 172.21.15.135:0/545074684 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdd4115920 con 0x7efdd4102ac0 2024-03-30T04:19:10.836 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:10.836+0000 7efdd0c00700 1 -- 172.21.15.135:0/545074684 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efdc801c5b0 con 0x7efdd4102ac0 2024-03-30T04:19:10.836 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:10.836+0000 7efddb800700 1 -- 172.21.15.135:0/545074684 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdd4115e10 con 0x7efdd4102ac0 2024-03-30T04:19:10.836 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:10.837+0000 7efddb800700 1 -- 172.21.15.135:0/545074684 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdd40fe1c0 con 0x7efdd4102ac0 2024-03-30T04:19:10.836 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:10.838+0000 7efdd0c00700 1 -- 172.21.15.135:0/545074684 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 43483+0+0 (secure 0 0 0) 0x7efdc8013970 con 0x7efdd4102ac0 2024-03-30T04:19:10.837 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:10.838+0000 7efdd0c00700 1 --2- 172.21.15.135:0/545074684 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7efdbc036970 0x7efdbc038e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:19:10.837 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:10.838+0000 7efdd0c00700 1 -- 172.21.15.135:0/545074684 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7efdc8049b10 con 0x7efdd4102ac0 2024-03-30T04:19:10.838 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:10.840+0000 7efdd3e00700 1 --2- 172.21.15.135:0/545074684 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7efdbc036970 0x7efdbc038e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:19:10.839 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:10.840+0000 7efdd3e00700 1 --2- 172.21.15.135:0/545074684 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7efdbc036970 0x7efdbc038e20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7efdc0009990 tx=0x7efdc0006e40 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:19:10.843 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:10.844+0000 7efdd0c00700 1 -- 172.21.15.135:0/545074684 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7efdc8011960 con 0x7efdd4102ac0 2024-03-30T04:19:11.170 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:11.170+0000 7efddb800700 1 -- 172.21.15.135:0/545074684 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efdd405f5f0 con 0x7efdd4102ac0 2024-03-30T04:19:11.170 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:11.171+0000 7efdd0c00700 1 -- 172.21.15.135:0/545074684 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7efdc801b390 con 0x7efdd4102ac0 2024-03-30T04:19:11.171 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:19:11.171 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"2e77a2be-ee4c-11ee-b647-cb9ed24678a4","modified":"2024-03-30T04:16:26.797409Z","created":"2024-03-30T04:16:26.797409Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-30T04:19:11.173 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:11.174+0000 7efddb800700 1 -- 172.21.15.135:0/545074684 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7efdbc036970 msgr2=0x7efdbc038e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:19:11.173 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:11.174+0000 7efddb800700 1 --2- 172.21.15.135:0/545074684 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7efdbc036970 0x7efdbc038e20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7efdc0009990 tx=0x7efdc0006e40 comp rx=0 tx=0).stop 2024-03-30T04:19:11.173 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:11.174+0000 7efddb800700 1 -- 172.21.15.135:0/545074684 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efdd4102ac0 msgr2=0x7efdd41151e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:19:11.173 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:11.174+0000 7efddb800700 1 --2- 172.21.15.135:0/545074684 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efdd4102ac0 0x7efdd41151e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7efdc8007e30 tx=0x7efdc8009e30 comp rx=0 tx=0).stop 2024-03-30T04:19:11.173 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:11.174+0000 7efddb800700 1 -- 172.21.15.135:0/545074684 shutdown_connections 2024-03-30T04:19:11.173 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:11.174+0000 7efddb800700 1 --2- 172.21.15.135:0/545074684 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7efdbc036970 0x7efdbc038e20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:19:11.173 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:11.174+0000 7efddb800700 1 --2- 172.21.15.135:0/545074684 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efdd4102ac0 0x7efdd41151e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:19:11.173 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:11.174+0000 7efddb800700 1 -- 172.21.15.135:0/545074684 >> 172.21.15.135:0/545074684 conn(0x7efdd40f85e0 msgr2=0x7efdd40701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:19:11.173 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:11.174+0000 7efddb800700 1 -- 172.21.15.135:0/545074684 shutdown_connections 2024-03-30T04:19:11.173 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:11.174+0000 7efddb800700 1 -- 172.21.15.135:0/545074684 wait complete. 2024-03-30T04:19:11.174 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2024-03-30T04:19:11.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:11 smithi123 ceph-mon[35245]: from='client.? 172.21.15.135:0/545074684' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T04:19:12.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:12 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:19:12.939 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-30T04:19:12.939 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-03-30T04:19:13.371 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T04:19:14.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:14 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:14.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:14 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:14.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:14 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:14.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:14 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:14.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:14 smithi123 ceph-mon[35245]: Deploying daemon node-exporter.smithi135 on smithi135 2024-03-30T04:19:15.148 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:15.148+0000 7fa2b6000700 1 -- 172.21.15.135:0/888507753 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2b0071f90 msgr2=0x7fa2b0070360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:19:15.148 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:15.148+0000 7fa2b6000700 1 --2- 172.21.15.135:0/888507753 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2b0071f90 0x7fa2b0070360 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa2a40048c0 tx=0x7fa2a4009e30 comp rx=0 tx=0).stop 2024-03-30T04:19:15.148 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:15.148+0000 7fa2b6000700 1 -- 172.21.15.135:0/888507753 shutdown_connections 2024-03-30T04:19:15.148 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:15.148+0000 7fa2b6000700 1 --2- 172.21.15.135:0/888507753 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2b0071f90 0x7fa2b0070360 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:19:15.148 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:15.148+0000 7fa2b6000700 1 -- 172.21.15.135:0/888507753 >> 172.21.15.135:0/888507753 conn(0x7fa2b00f8400 msgr2=0x7fa2b00fa810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:19:15.148 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:15.148+0000 7fa2b6000700 1 -- 172.21.15.135:0/888507753 shutdown_connections 2024-03-30T04:19:15.148 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:15.148+0000 7fa2b6000700 1 -- 172.21.15.135:0/888507753 wait complete. 2024-03-30T04:19:15.149 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:15.149+0000 7fa2b6000700 1 Processor -- start 2024-03-30T04:19:15.149 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:15.149+0000 7fa2b6000700 1 -- start start 2024-03-30T04:19:15.149 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:15.149+0000 7fa2b6000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2b0071f90 0x7fa2b01194f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:19:15.149 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:15.149+0000 7fa2b6000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b0119a30 con 0x7fa2b0071f90 2024-03-30T04:19:15.150 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:15.150+0000 7fa2b4c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2b0071f90 0x7fa2b01194f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:19:15.150 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:15.150+0000 7fa2b4c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2b0071f90 0x7fa2b01194f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:37332/0 (socket says 172.21.15.135:37332) 2024-03-30T04:19:15.150 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:15.150+0000 7fa2b4c00700 1 -- 172.21.15.135:0/955024788 learned_addr learned my addr 172.21.15.135:0/955024788 (peer_addr_for_me v2:172.21.15.135:0/0) 2024-03-30T04:19:15.150 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:15.150+0000 7fa2b4c00700 1 -- 172.21.15.135:0/955024788 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2a4005040 con 0x7fa2b0071f90 2024-03-30T04:19:15.150 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:15.150+0000 7fa2b4c00700 1 --2- 172.21.15.135:0/955024788 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2b0071f90 0x7fa2b01194f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa2a400b8b0 tx=0x7fa2a4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:19:15.150 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:15.150+0000 7fa2acc00700 1 -- 172.21.15.135:0/955024788 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa2a4004e60 con 0x7fa2b0071f90 2024-03-30T04:19:15.150 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:15.150+0000 7fa2acc00700 1 -- 172.21.15.135:0/955024788 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa2a400bc30 con 0x7fa2b0071f90 2024-03-30T04:19:15.150 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:15.150+0000 7fa2b6000700 1 -- 172.21.15.135:0/955024788 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2b0119c30 con 0x7fa2b0071f90 2024-03-30T04:19:15.150 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:15.151+0000 7fa2acc00700 1 -- 172.21.15.135:0/955024788 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa2a401c3f0 con 0x7fa2b0071f90 2024-03-30T04:19:15.151 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:15.151+0000 7fa2b6000700 1 -- 172.21.15.135:0/955024788 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2b011a0a0 con 0x7fa2b0071f90 2024-03-30T04:19:15.151 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:15.152+0000 7fa2acc00700 1 -- 172.21.15.135:0/955024788 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 43483+0+0 (secure 0 0 0) 0x7fa2a4013420 con 0x7fa2b0071f90 2024-03-30T04:19:15.151 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:15.152+0000 7fa2b6000700 1 -- 172.21.15.135:0/955024788 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2b0071aa0 con 0x7fa2b0071f90 2024-03-30T04:19:15.151 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:15.152+0000 7fa2acc00700 1 --2- 172.21.15.135:0/955024788 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fa2980369c0 0x7fa298038e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:19:15.151 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:15.152+0000 7fa2acc00700 1 -- 172.21.15.135:0/955024788 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fa2a40495a0 con 0x7fa2b0071f90 2024-03-30T04:19:15.152 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:15.153+0000 7fa2afe00700 1 --2- 172.21.15.135:0/955024788 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fa2980369c0 0x7fa298038e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:19:15.152 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:15.153+0000 7fa2afe00700 1 --2- 172.21.15.135:0/955024788 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fa2980369c0 0x7fa298038e70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa29c009990 tx=0x7fa29c006e40 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:19:15.156 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:15.158+0000 7fa2acc00700 1 -- 172.21.15.135:0/955024788 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fa2a4011550 con 0x7fa2b0071f90 2024-03-30T04:19:15.439 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:15.440+0000 7fa2b6000700 1 -- 172.21.15.135:0/955024788 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa2b004bed0 con 0x7fa2b0071f90 2024-03-30T04:19:15.440 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:15.441+0000 7fa2acc00700 1 -- 172.21.15.135:0/955024788 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fa2a4021030 con 0x7fa2b0071f90 2024-03-30T04:19:15.440 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:19:15.440 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"2e77a2be-ee4c-11ee-b647-cb9ed24678a4","modified":"2024-03-30T04:16:26.797409Z","created":"2024-03-30T04:16:26.797409Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-30T04:19:15.442 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:15.443+0000 7fa2b6000700 1 -- 172.21.15.135:0/955024788 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fa2980369c0 msgr2=0x7fa298038e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:19:15.442 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:15.443+0000 7fa2b6000700 1 --2- 172.21.15.135:0/955024788 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fa2980369c0 0x7fa298038e70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa29c009990 tx=0x7fa29c006e40 comp rx=0 tx=0).stop 2024-03-30T04:19:15.442 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:15.443+0000 7fa2b6000700 1 -- 172.21.15.135:0/955024788 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2b0071f90 msgr2=0x7fa2b01194f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:19:15.442 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:15.443+0000 7fa2b6000700 1 --2- 172.21.15.135:0/955024788 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2b0071f90 0x7fa2b01194f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa2a400b8b0 tx=0x7fa2a4009e30 comp rx=0 tx=0).stop 2024-03-30T04:19:15.443 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:15.443+0000 7fa2b6000700 1 -- 172.21.15.135:0/955024788 shutdown_connections 2024-03-30T04:19:15.443 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:15.443+0000 7fa2b6000700 1 --2- 172.21.15.135:0/955024788 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fa2980369c0 0x7fa298038e70 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:19:15.443 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:15.443+0000 7fa2b6000700 1 --2- 172.21.15.135:0/955024788 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2b0071f90 0x7fa2b01194f0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:19:15.443 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:15.443+0000 7fa2b6000700 1 -- 172.21.15.135:0/955024788 >> 172.21.15.135:0/955024788 conn(0x7fa2b00f8400 msgr2=0x7fa2b00fa070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:19:15.443 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:15.443+0000 7fa2b6000700 1 -- 172.21.15.135:0/955024788 shutdown_connections 2024-03-30T04:19:15.443 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:15.443+0000 7fa2b6000700 1 -- 172.21.15.135:0/955024788 wait complete. 2024-03-30T04:19:15.443 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2024-03-30T04:19:15.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:15 smithi123 ceph-mon[35245]: from='client.? 172.21.15.135:0/955024788' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T04:19:17.237 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-30T04:19:17.237 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-03-30T04:19:17.649 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T04:19:17.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:17 smithi123 ceph-mon[35245]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:19:17.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:17 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:20.145 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:19.945+0000 7f5802000700 1 -- 172.21.15.135:0/1276914158 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57fc102ae0 msgr2=0x7f57fc102eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:19:20.146 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:19.945+0000 7f5802000700 1 --2- 172.21.15.135:0/1276914158 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57fc102ae0 0x7f57fc102eb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f57ec0048c0 tx=0x7f57ec009e30 comp rx=0 tx=0).stop 2024-03-30T04:19:20.146 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:19.946+0000 7f5802000700 1 -- 172.21.15.135:0/1276914158 shutdown_connections 2024-03-30T04:19:20.146 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:19.946+0000 7f5802000700 1 --2- 172.21.15.135:0/1276914158 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57fc102ae0 0x7f57fc102eb0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:19:20.146 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:19.946+0000 7f5802000700 1 -- 172.21.15.135:0/1276914158 >> 172.21.15.135:0/1276914158 conn(0x7f57fc0f85e0 msgr2=0x7f57fc0fa9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:19:20.146 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:19.946+0000 7f5802000700 1 -- 172.21.15.135:0/1276914158 shutdown_connections 2024-03-30T04:19:20.146 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:19.946+0000 7f5802000700 1 -- 172.21.15.135:0/1276914158 wait complete. 2024-03-30T04:19:20.146 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:19.947+0000 7f5802000700 1 Processor -- start 2024-03-30T04:19:20.146 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:19.947+0000 7f5802000700 1 -- start start 2024-03-30T04:19:20.146 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:19.947+0000 7f5802000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57fc102ae0 0x7f57fc1152e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:19:20.146 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:19.947+0000 7f5802000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57fc115820 con 0x7f57fc102ae0 2024-03-30T04:19:20.146 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:19.948+0000 7f57fb400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57fc102ae0 0x7f57fc1152e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:19:20.147 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:19.948+0000 7f57fb400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57fc102ae0 0x7f57fc1152e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:56908/0 (socket says 172.21.15.135:56908) 2024-03-30T04:19:20.147 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:19.948+0000 7f57fb400700 1 -- 172.21.15.135:0/2960601705 learned_addr learned my addr 172.21.15.135:0/2960601705 (peer_addr_for_me v2:172.21.15.135:0/0) 2024-03-30T04:19:20.147 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:19.948+0000 7f57fb400700 1 -- 172.21.15.135:0/2960601705 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57ec005040 con 0x7f57fc102ae0 2024-03-30T04:19:20.147 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:19.948+0000 7f57fb400700 1 --2- 172.21.15.135:0/2960601705 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57fc102ae0 0x7f57fc1152e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f57ec007e30 tx=0x7f57ec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:19:20.147 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:19.948+0000 7f57f3400700 1 -- 172.21.15.135:0/2960601705 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57ec0093c0 con 0x7f57fc102ae0 2024-03-30T04:19:20.147 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:19.949+0000 7f57f3400700 1 -- 172.21.15.135:0/2960601705 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f57ec013420 con 0x7f57fc102ae0 2024-03-30T04:19:20.147 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:19.949+0000 7f5802000700 1 -- 172.21.15.135:0/2960601705 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57fc115a20 con 0x7f57fc102ae0 2024-03-30T04:19:20.147 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:19.949+0000 7f57f3400700 1 -- 172.21.15.135:0/2960601705 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57ec01c5b0 con 0x7f57fc102ae0 2024-03-30T04:19:20.147 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:19.949+0000 7f5802000700 1 -- 172.21.15.135:0/2960601705 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57fc115e90 con 0x7f57fc102ae0 2024-03-30T04:19:20.147 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:19.950+0000 7f57f3400700 1 -- 172.21.15.135:0/2960601705 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 43483+0+0 (secure 0 0 0) 0x7f57ec009520 con 0x7f57fc102ae0 2024-03-30T04:19:20.147 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:19.950+0000 7f5802000700 1 -- 172.21.15.135:0/2960601705 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57fc0fe1e0 con 0x7f57fc102ae0 2024-03-30T04:19:20.147 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:19.950+0000 7f57f3400700 1 --2- 172.21.15.135:0/2960601705 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f57e0036970 0x7f57e0038e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:19:20.147 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:19.950+0000 7f57f3400700 1 -- 172.21.15.135:0/2960601705 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f57ec049b10 con 0x7f57fc102ae0 2024-03-30T04:19:20.147 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:19.951+0000 7f57faa00700 1 --2- 172.21.15.135:0/2960601705 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f57e0036970 0x7f57e0038e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:19:20.147 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:19.951+0000 7f57faa00700 1 --2- 172.21.15.135:0/2960601705 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f57e0036970 0x7f57e0038e20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f57e8009990 tx=0x7f57e8006e40 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:19:20.148 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:19.956+0000 7f57f3400700 1 -- 172.21.15.135:0/2960601705 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f57ec013970 con 0x7f57fc102ae0 2024-03-30T04:19:20.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:19 smithi123 ceph-mon[35245]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:19:20.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:19 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:20.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:19 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:20.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:19 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:20.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:19 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:20.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:19 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi135.kyblli", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T04:19:20.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:19 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi135.kyblli", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-30T04:19:20.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:19 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T04:19:20.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:19 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:19:20.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:19 smithi123 ceph-mon[35245]: Deploying daemon mgr.smithi135.kyblli on smithi135 2024-03-30T04:19:20.248 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:20.250+0000 7f5802000700 1 -- 172.21.15.135:0/2960601705 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f57fc05f5f0 con 0x7f57fc102ae0 2024-03-30T04:19:20.249 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:20.251+0000 7f57f3400700 1 -- 172.21.15.135:0/2960601705 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f57ec011370 con 0x7f57fc102ae0 2024-03-30T04:19:20.249 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:19:20.249 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"2e77a2be-ee4c-11ee-b647-cb9ed24678a4","modified":"2024-03-30T04:16:26.797409Z","created":"2024-03-30T04:16:26.797409Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-30T04:19:20.251 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:20.253+0000 7f5802000700 1 -- 172.21.15.135:0/2960601705 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f57e0036970 msgr2=0x7f57e0038e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:19:20.251 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:20.253+0000 7f5802000700 1 --2- 172.21.15.135:0/2960601705 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f57e0036970 0x7f57e0038e20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f57e8009990 tx=0x7f57e8006e40 comp rx=0 tx=0).stop 2024-03-30T04:19:20.251 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:20.253+0000 7f5802000700 1 -- 172.21.15.135:0/2960601705 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57fc102ae0 msgr2=0x7f57fc1152e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:19:20.252 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:20.253+0000 7f5802000700 1 --2- 172.21.15.135:0/2960601705 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57fc102ae0 0x7f57fc1152e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f57ec007e30 tx=0x7f57ec009e30 comp rx=0 tx=0).stop 2024-03-30T04:19:20.252 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:20.253+0000 7f5802000700 1 -- 172.21.15.135:0/2960601705 shutdown_connections 2024-03-30T04:19:20.252 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:20.253+0000 7f5802000700 1 --2- 172.21.15.135:0/2960601705 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f57e0036970 0x7f57e0038e20 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:19:20.252 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:20.253+0000 7f5802000700 1 --2- 172.21.15.135:0/2960601705 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57fc102ae0 0x7f57fc1152e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:19:20.252 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:20.253+0000 7f5802000700 1 -- 172.21.15.135:0/2960601705 >> 172.21.15.135:0/2960601705 conn(0x7f57fc0f85e0 msgr2=0x7f57fc071430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:19:20.252 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:20.253+0000 7f5802000700 1 -- 172.21.15.135:0/2960601705 shutdown_connections 2024-03-30T04:19:20.252 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:20.253+0000 7f5802000700 1 -- 172.21.15.135:0/2960601705 wait complete. 2024-03-30T04:19:20.253 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2024-03-30T04:19:21.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:20 smithi123 ceph-mon[35245]: from='client.? 172.21.15.135:0/2960601705' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T04:19:22.178 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-30T04:19:22.178 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-03-30T04:19:22.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:21 smithi123 ceph-mon[35245]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:19:22.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:21 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:22.789 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T04:19:24.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:23 smithi123 ceph-mon[35245]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:19:24.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:23 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:24.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:23 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:24.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:23 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:24.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:23 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:24.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:23 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T04:19:24.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:23 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:19:24.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:23 smithi123 ceph-mon[35245]: Deploying daemon mon.smithi135 on smithi135 2024-03-30T04:19:24.990 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:24.903+0000 7f35f8000700 1 -- 172.21.15.135:0/1184939218 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35f0071f90 msgr2=0x7f35f0070360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:19:24.990 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:24.903+0000 7f35f8000700 1 --2- 172.21.15.135:0/1184939218 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35f0071f90 0x7f35f0070360 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f35e40048c0 tx=0x7f35e4009e30 comp rx=0 tx=0).stop 2024-03-30T04:19:24.990 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:24.903+0000 7f35f8000700 1 -- 172.21.15.135:0/1184939218 shutdown_connections 2024-03-30T04:19:24.990 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:24.903+0000 7f35f8000700 1 --2- 172.21.15.135:0/1184939218 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35f0071f90 0x7f35f0070360 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:19:24.990 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:24.903+0000 7f35f8000700 1 -- 172.21.15.135:0/1184939218 >> 172.21.15.135:0/1184939218 conn(0x7f35f00f85e0 msgr2=0x7f35f00fa9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:19:24.990 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:24.903+0000 7f35f8000700 1 -- 172.21.15.135:0/1184939218 shutdown_connections 2024-03-30T04:19:24.990 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:24.903+0000 7f35f8000700 1 -- 172.21.15.135:0/1184939218 wait complete. 2024-03-30T04:19:24.990 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:24.904+0000 7f35f8000700 1 Processor -- start 2024-03-30T04:19:24.990 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:24.904+0000 7f35f8000700 1 -- start start 2024-03-30T04:19:24.991 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:24.905+0000 7f35f8000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35f0071f90 0x7f35f01196c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:19:24.991 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:24.905+0000 7f35f8000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f0119c00 con 0x7f35f0071f90 2024-03-30T04:19:24.991 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:24.905+0000 7f35f5600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35f0071f90 0x7f35f01196c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:19:24.991 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:24.905+0000 7f35f5600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35f0071f90 0x7f35f01196c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:56948/0 (socket says 172.21.15.135:56948) 2024-03-30T04:19:24.991 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:24.905+0000 7f35f5600700 1 -- 172.21.15.135:0/90983515 learned_addr learned my addr 172.21.15.135:0/90983515 (peer_addr_for_me v2:172.21.15.135:0/0) 2024-03-30T04:19:24.991 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:24.906+0000 7f35f5600700 1 -- 172.21.15.135:0/90983515 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e4005040 con 0x7f35f0071f90 2024-03-30T04:19:24.991 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:24.906+0000 7f35f5600700 1 --2- 172.21.15.135:0/90983515 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35f0071f90 0x7f35f01196c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f35e400b8b0 tx=0x7f35e4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:19:24.991 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:24.906+0000 7f35ed600700 1 -- 172.21.15.135:0/90983515 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35e4004e60 con 0x7f35f0071f90 2024-03-30T04:19:24.991 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:24.907+0000 7f35f8000700 1 -- 172.21.15.135:0/90983515 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35f0119e00 con 0x7f35f0071f90 2024-03-30T04:19:24.991 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:24.907+0000 7f35ed600700 1 -- 172.21.15.135:0/90983515 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f35e400bc30 con 0x7f35f0071f90 2024-03-30T04:19:24.991 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:24.907+0000 7f35ed600700 1 -- 172.21.15.135:0/90983515 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35e401c3f0 con 0x7f35f0071f90 2024-03-30T04:19:24.991 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:24.907+0000 7f35f8000700 1 -- 172.21.15.135:0/90983515 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35f011a270 con 0x7f35f0071f90 2024-03-30T04:19:24.991 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:24.908+0000 7f35ed600700 1 -- 172.21.15.135:0/90983515 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 43483+0+0 (secure 0 0 0) 0x7f35e4026440 con 0x7f35f0071f90 2024-03-30T04:19:24.992 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:24.908+0000 7f35f8000700 1 -- 172.21.15.135:0/90983515 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35f0071aa0 con 0x7f35f0071f90 2024-03-30T04:19:24.992 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:24.909+0000 7f35ed600700 1 --2- 172.21.15.135:0/90983515 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f35d80369c0 0x7f35d8038e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:19:24.992 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:24.909+0000 7f35ed600700 1 -- 172.21.15.135:0/90983515 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f35e4049600 con 0x7f35f0071f90 2024-03-30T04:19:24.992 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:24.909+0000 7f35f4c00700 1 --2- 172.21.15.135:0/90983515 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f35d80369c0 0x7f35d8038e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:19:24.992 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:24.910+0000 7f35f4c00700 1 --2- 172.21.15.135:0/90983515 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f35d80369c0 0x7f35d8038e70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f35dc009990 tx=0x7f35dc006e40 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:19:24.992 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:24.918+0000 7f35ed600700 1 -- 172.21.15.135:0/90983515 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f35e4011780 con 0x7f35f0071f90 2024-03-30T04:19:25.203 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:25.205+0000 7f35f8000700 1 -- 172.21.15.135:0/90983515 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f35f004bed0 con 0x7f35f0071f90 2024-03-30T04:19:25.203 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:25.206+0000 7f35ed600700 1 -- 172.21.15.135:0/90983515 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f35e4021030 con 0x7f35f0071f90 2024-03-30T04:19:25.204 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:19:25.204 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"2e77a2be-ee4c-11ee-b647-cb9ed24678a4","modified":"2024-03-30T04:16:26.797409Z","created":"2024-03-30T04:16:26.797409Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-30T04:19:25.206 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:25.208+0000 7f35f8000700 1 -- 172.21.15.135:0/90983515 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f35d80369c0 msgr2=0x7f35d8038e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:19:25.206 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:25.208+0000 7f35f8000700 1 --2- 172.21.15.135:0/90983515 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f35d80369c0 0x7f35d8038e70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f35dc009990 tx=0x7f35dc006e40 comp rx=0 tx=0).stop 2024-03-30T04:19:25.207 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:25.208+0000 7f35f8000700 1 -- 172.21.15.135:0/90983515 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35f0071f90 msgr2=0x7f35f01196c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:19:25.207 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:25.208+0000 7f35f8000700 1 --2- 172.21.15.135:0/90983515 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35f0071f90 0x7f35f01196c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f35e400b8b0 tx=0x7f35e4009e30 comp rx=0 tx=0).stop 2024-03-30T04:19:25.207 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:25.208+0000 7f35f8000700 1 -- 172.21.15.135:0/90983515 shutdown_connections 2024-03-30T04:19:25.207 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:25.208+0000 7f35f8000700 1 --2- 172.21.15.135:0/90983515 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f35d80369c0 0x7f35d8038e70 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:19:25.207 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:25.208+0000 7f35f8000700 1 --2- 172.21.15.135:0/90983515 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35f0071f90 0x7f35f01196c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:19:25.207 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:25.208+0000 7f35f8000700 1 -- 172.21.15.135:0/90983515 >> 172.21.15.135:0/90983515 conn(0x7f35f00f85e0 msgr2=0x7f35f00fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:19:25.207 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:25.208+0000 7f35f8000700 1 -- 172.21.15.135:0/90983515 shutdown_connections 2024-03-30T04:19:25.207 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:25.208+0000 7f35f8000700 1 -- 172.21.15.135:0/90983515 wait complete. 2024-03-30T04:19:25.208 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2024-03-30T04:19:26.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:25 smithi123 ceph-mon[35245]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:19:26.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:25 smithi123 ceph-mon[35245]: from='client.? 172.21.15.135:0/90983515' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T04:19:27.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:26 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:19:27.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:26 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:27.300 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-30T04:19:27.300 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-03-30T04:19:27.795 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi135/config 2024-03-30T04:19:28.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:27 smithi123 ceph-mon[35245]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:19:29.573 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:29 smithi135 ceph-mon[44364]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:19:29.776 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:29 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:29.776 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:29 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:29.776 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:29 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:29.776 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:29 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:29.777 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:29 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T04:19:29.777 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:29 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:19:29.777 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:29 smithi135 ceph-mon[44364]: Deploying daemon mon.smithi135 on smithi135 2024-03-30T04:19:29.777 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:29 smithi135 ceph-mon[44364]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:19:29.777 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:29 smithi135 ceph-mon[44364]: from='client.? 172.21.15.135:0/90983515' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T04:19:29.777 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:29 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:19:29.777 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:29 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:29.778 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:29 smithi135 ceph-mon[44364]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:19:29.778 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:29 smithi135 ceph-mon[44364]: mon.smithi135@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2024-03-30T04:19:29.778 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:29 smithi135 bash[44300]: f81052b09dd8a896f1392b3cc25053684265d4e3249ca397cca9f75eaeeaf400 2024-03-30T04:19:29.778 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:29 smithi135 systemd[1]: Started Ceph mon.smithi135 for 2e77a2be-ee4c-11ee-b647-cb9ed24678a4. 2024-03-30T04:19:34.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:34 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mon metadata", "id": "smithi123"}]: dispatch 2024-03-30T04:19:34.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:34 smithi123 ceph-mon[35245]: mon.smithi123 calling monitor election 2024-03-30T04:19:34.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:34 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-03-30T04:19:34.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:34 smithi123 ceph-mon[35245]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:19:34.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:34 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-03-30T04:19:34.968 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:34 smithi123 ceph-mon[35245]: from='mgr.? 172.21.15.135:0/3563300032' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi135.kyblli/crt"}]: dispatch 2024-03-30T04:19:34.968 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:34 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-03-30T04:19:34.968 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:34 smithi123 ceph-mon[35245]: mon.smithi135 calling monitor election 2024-03-30T04:19:34.968 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:34 smithi123 ceph-mon[35245]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:19:34.968 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:34 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-03-30T04:19:34.968 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:34 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-03-30T04:19:34.968 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:34 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-03-30T04:19:34.968 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:34 smithi123 ceph-mon[35245]: mon.smithi123 is new leader, mons smithi123,smithi135 in quorum (ranks 0,1) 2024-03-30T04:19:34.968 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:34 smithi123 ceph-mon[35245]: monmap e2: 2 mons at {smithi123=[v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0],smithi135=[v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-30T04:19:34.968 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:34 smithi123 ceph-mon[35245]: fsmap 2024-03-30T04:19:34.968 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:34 smithi123 ceph-mon[35245]: osdmap e5: 0 total, 0 up, 0 in 2024-03-30T04:19:34.968 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:34 smithi123 ceph-mon[35245]: mgrmap e17: smithi123.xgksqq(active, since 38s) 2024-03-30T04:19:34.968 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:34 smithi123 ceph-mon[35245]: overall HEALTH_OK 2024-03-30T04:19:34.968 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:34 smithi123 ceph-mon[35245]: Standby manager daemon smithi135.kyblli started 2024-03-30T04:19:34.969 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:34 smithi123 ceph-mon[35245]: from='mgr.? 172.21.15.135:0/3563300032' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T04:19:34.969 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:34 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:34.969 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:34 smithi123 ceph-mon[35245]: from='mgr.? 172.21.15.135:0/3563300032' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi135.kyblli/key"}]: dispatch 2024-03-30T04:19:34.969 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:34 smithi123 ceph-mon[35245]: from='mgr.? 172.21.15.135:0/3563300032' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T04:19:34.969 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:34 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:34.969 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:34 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:34.969 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:34 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:34.969 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:34 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:35.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:34 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mon metadata", "id": "smithi123"}]: dispatch 2024-03-30T04:19:35.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:34 smithi135 ceph-mon[44364]: mon.smithi123 calling monitor election 2024-03-30T04:19:35.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:34 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-03-30T04:19:35.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:34 smithi135 ceph-mon[44364]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:19:35.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:34 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-03-30T04:19:35.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:34 smithi135 ceph-mon[44364]: from='mgr.? 172.21.15.135:0/3563300032' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi135.kyblli/crt"}]: dispatch 2024-03-30T04:19:35.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:34 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-03-30T04:19:35.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:34 smithi135 ceph-mon[44364]: mon.smithi135 calling monitor election 2024-03-30T04:19:35.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:34 smithi135 ceph-mon[44364]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:19:35.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:34 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-03-30T04:19:35.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:34 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-03-30T04:19:35.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:34 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-03-30T04:19:35.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:34 smithi135 ceph-mon[44364]: mon.smithi123 is new leader, mons smithi123,smithi135 in quorum (ranks 0,1) 2024-03-30T04:19:35.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:34 smithi135 ceph-mon[44364]: monmap e2: 2 mons at {smithi123=[v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0],smithi135=[v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-30T04:19:35.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:34 smithi135 ceph-mon[44364]: fsmap 2024-03-30T04:19:35.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:34 smithi135 ceph-mon[44364]: osdmap e5: 0 total, 0 up, 0 in 2024-03-30T04:19:35.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:34 smithi135 ceph-mon[44364]: mgrmap e17: smithi123.xgksqq(active, since 38s) 2024-03-30T04:19:35.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:34 smithi135 ceph-mon[44364]: overall HEALTH_OK 2024-03-30T04:19:35.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:34 smithi135 ceph-mon[44364]: Standby manager daemon smithi135.kyblli started 2024-03-30T04:19:35.103 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:34 smithi135 ceph-mon[44364]: from='mgr.? 172.21.15.135:0/3563300032' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T04:19:35.103 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:34 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:35.103 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:34 smithi135 ceph-mon[44364]: from='mgr.? 172.21.15.135:0/3563300032' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi135.kyblli/key"}]: dispatch 2024-03-30T04:19:35.103 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:34 smithi135 ceph-mon[44364]: from='mgr.? 172.21.15.135:0/3563300032' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T04:19:35.103 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:34 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:35.103 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:34 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:35.103 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:34 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:35.103 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:34 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:35.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:35 smithi123 ceph-mon[35245]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:19:35.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:35 smithi123 ceph-mon[35245]: mgrmap e18: smithi123.xgksqq(active, since 38s), standbys: smithi135.kyblli 2024-03-30T04:19:35.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:35 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mgr metadata", "who": "smithi135.kyblli", "id": "smithi135.kyblli"}]: dispatch 2024-03-30T04:19:35.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:35 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:19:35.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:35 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-03-30T04:19:36.013 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:36.013+0000 7f662d200700 1 -- 172.21.15.135:0/3404849455 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f660c0061f0 msgr2=0x7f660c008650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:19:36.013 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:36.013+0000 7f662d200700 1 --2- 172.21.15.135:0/3404849455 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f660c0061f0 0x7f660c008650 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6628071420 tx=0x7f6618002040 comp rx=0 tx=0).stop 2024-03-30T04:19:36.013 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:36.013+0000 7f662d200700 1 -- 172.21.15.135:0/3404849455 shutdown_connections 2024-03-30T04:19:36.013 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:36.013+0000 7f662d200700 1 --2- 172.21.15.135:0/3404849455 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f660c0061f0 0x7f660c008650 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:19:36.013 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:36.013+0000 7f662d200700 1 -- 172.21.15.135:0/3404849455 >> 172.21.15.135:0/3404849455 conn(0x7f66280f8730 msgr2=0x7f66280fab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:19:36.013 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:36.013+0000 7f662d200700 1 -- 172.21.15.135:0/3404849455 shutdown_connections 2024-03-30T04:19:36.013 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:36.014+0000 7f662d200700 1 -- 172.21.15.135:0/3404849455 wait complete. 2024-03-30T04:19:36.015 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:36.014+0000 7f662d200700 1 Processor -- start 2024-03-30T04:19:36.015 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:36.014+0000 7f662d200700 1 -- start start 2024-03-30T04:19:36.015 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:36.015+0000 7f662d200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f660c0061f0 0x7f6628115510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:19:36.015 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:36.015+0000 7f662d200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6628115a50 0x7f662810f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:19:36.015 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:36.015+0000 7f662d200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6628116010 con 0x7f660c0061f0 2024-03-30T04:19:36.015 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:36.015+0000 7f662d200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f662810f9d0 con 0x7f6628115a50 2024-03-30T04:19:36.015 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:36.015+0000 7f6625400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6628115a50 0x7f662810f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:19:36.015 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:36.015+0000 7f6625e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f660c0061f0 0x7f6628115510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:19:36.015 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:36.015+0000 7f6625400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6628115a50 0x7f662810f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.135:43714/0 (socket says 172.21.15.135:43714) 2024-03-30T04:19:36.015 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:36.015+0000 7f6625400700 1 -- 172.21.15.135:0/3472497396 learned_addr learned my addr 172.21.15.135:0/3472497396 (peer_addr_for_me v2:172.21.15.135:0/0) 2024-03-30T04:19:36.015 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:36.015+0000 7f6625400700 1 -- 172.21.15.135:0/3472497396 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6628115a50 msgr2=0x7f662810f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-30T04:19:36.015 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:36.015+0000 7f6625400700 1 -- 172.21.15.135:0/3472497396 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6628115a50 msgr2=0x7f662810f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-30T04:19:36.016 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:36.015+0000 7f6625400700 1 --2- 172.21.15.135:0/3472497396 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6628115a50 0x7f662810f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-03-30T04:19:36.016 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:36.016+0000 7f6625400700 1 --2- 172.21.15.135:0/3472497396 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6628115a50 0x7f662810f490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-30T04:19:36.016 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:36.016+0000 7f6625e00700 1 -- 172.21.15.135:0/3472497396 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6628115a50 msgr2=0x7f662810f490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-30T04:19:36.016 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:36.016+0000 7f6625e00700 1 --2- 172.21.15.135:0/3472497396 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6628115a50 0x7f662810f490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:19:36.016 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:36.016+0000 7f6625e00700 1 -- 172.21.15.135:0/3472497396 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6618005040 con 0x7f660c0061f0 2024-03-30T04:19:36.016 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:36.016+0000 7f6625e00700 1 --2- 172.21.15.135:0/3472497396 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f660c0061f0 0x7f6628115510 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6618000c00 tx=0x7f6618002040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:19:36.017 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:36.016+0000 7f661e000700 1 -- 172.21.15.135:0/3472497396 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6618017410 con 0x7f660c0061f0 2024-03-30T04:19:36.017 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:36.016+0000 7f661e000700 1 -- 172.21.15.135:0/3472497396 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6618009710 con 0x7f660c0061f0 2024-03-30T04:19:36.017 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:36.016+0000 7f662d200700 1 -- 172.21.15.135:0/3472497396 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f662810fc50 con 0x7f660c0061f0 2024-03-30T04:19:36.017 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:36.016+0000 7f661e000700 1 -- 172.21.15.135:0/3472497396 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6618022ac0 con 0x7f660c0061f0 2024-03-30T04:19:36.017 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:36.017+0000 7f662d200700 1 -- 172.21.15.135:0/3472497396 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6628110130 con 0x7f660c0061f0 2024-03-30T04:19:36.017 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:36.018+0000 7f662d200700 1 -- 172.21.15.135:0/3472497396 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6628109370 con 0x7f660c0061f0 2024-03-30T04:19:36.017 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:36.018+0000 7f661e000700 1 -- 172.21.15.135:0/3472497396 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 18) v1 ==== 86567+0+0 (secure 0 0 0) 0x7f6618009880 con 0x7f660c0061f0 2024-03-30T04:19:36.018 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:36.019+0000 7f661e000700 1 --2- 172.21.15.135:0/3472497396 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f6608069360 0x7f660806b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:19:36.018 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:36.019+0000 7f661e000700 1 -- 172.21.15.135:0/3472497396 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f6618088160 con 0x7f660c0061f0 2024-03-30T04:19:36.018 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:36.020+0000 7f6625400700 1 --2- 172.21.15.135:0/3472497396 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f6608069360 0x7f660806b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:19:36.022 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:36.020+0000 7f6625400700 1 --2- 172.21.15.135:0/3472497396 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f6608069360 0x7f660806b810 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6614006fd0 tx=0x7f6614008040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:19:36.024 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:36.025+0000 7f661e000700 1 -- 172.21.15.135:0/3472497396 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f6618058ee0 con 0x7f660c0061f0 2024-03-30T04:19:36.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:35 smithi135 ceph-mon[44364]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:19:36.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:35 smithi135 ceph-mon[44364]: mgrmap e18: smithi123.xgksqq(active, since 38s), standbys: smithi135.kyblli 2024-03-30T04:19:36.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:35 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mgr metadata", "who": "smithi135.kyblli", "id": "smithi135.kyblli"}]: dispatch 2024-03-30T04:19:36.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:35 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:19:36.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:35 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-03-30T04:19:36.337 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:36.338+0000 7f662d200700 1 -- 172.21.15.135:0/3472497396 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f66281069f0 con 0x7f660c0061f0 2024-03-30T04:19:36.337 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:36.338+0000 7f661e000700 1 -- 172.21.15.135:0/3472497396 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1021 (secure 0 0 0) 0x7f6618058a70 con 0x7f660c0061f0 2024-03-30T04:19:36.337 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-30T04:19:36.337 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":2,"fsid":"2e77a2be-ee4c-11ee-b647-cb9ed24678a4","modified":"2024-03-30T04:19:29.581328Z","created":"2024-03-30T04:16:26.797409Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi135","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:3300","nonce":0},{"type":"v1","addr":"172.21.15.135:6789","nonce":0}]},"addr":"172.21.15.135:6789/0","public_addr":"172.21.15.135:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-30T04:19:36.340 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:36.341+0000 7f662d200700 1 -- 172.21.15.135:0/3472497396 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f6608069360 msgr2=0x7f660806b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:19:36.340 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:36.341+0000 7f662d200700 1 --2- 172.21.15.135:0/3472497396 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f6608069360 0x7f660806b810 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6614006fd0 tx=0x7f6614008040 comp rx=0 tx=0).stop 2024-03-30T04:19:36.340 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:36.341+0000 7f662d200700 1 -- 172.21.15.135:0/3472497396 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f660c0061f0 msgr2=0x7f6628115510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:19:36.340 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:36.341+0000 7f662d200700 1 --2- 172.21.15.135:0/3472497396 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f660c0061f0 0x7f6628115510 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6618000c00 tx=0x7f6618002040 comp rx=0 tx=0).stop 2024-03-30T04:19:36.340 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:36.341+0000 7f662d200700 1 -- 172.21.15.135:0/3472497396 shutdown_connections 2024-03-30T04:19:36.340 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:36.341+0000 7f662d200700 1 --2- 172.21.15.135:0/3472497396 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f6608069360 0x7f660806b810 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:19:36.340 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:36.341+0000 7f662d200700 1 --2- 172.21.15.135:0/3472497396 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f660c0061f0 0x7f6628115510 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:19:36.340 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:36.341+0000 7f662d200700 1 --2- 172.21.15.135:0/3472497396 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6628115a50 0x7f662810f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:19:36.341 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:36.341+0000 7f662d200700 1 -- 172.21.15.135:0/3472497396 >> 172.21.15.135:0/3472497396 conn(0x7f66280f8730 msgr2=0x7f6628104750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:19:36.341 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:36.341+0000 7f662d200700 1 -- 172.21.15.135:0/3472497396 shutdown_connections 2024-03-30T04:19:36.341 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:19:36.341+0000 7f662d200700 1 -- 172.21.15.135:0/3472497396 wait complete. 2024-03-30T04:19:36.341 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 2 2024-03-30T04:19:37.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:37 smithi135 ceph-mon[44364]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:19:37.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:37 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:37.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:37 smithi135 ceph-mon[44364]: from='client.? 172.21.15.135:0/3472497396' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T04:19:37.674 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-30T04:19:37.674 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph config generate-minimal-conf 2024-03-30T04:19:37.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:37 smithi123 ceph-mon[35245]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:19:37.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:37 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:37.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:37 smithi123 ceph-mon[35245]: from='client.? 172.21.15.135:0/3472497396' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T04:19:38.116 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:19:38.690 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:38 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:38.690 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:38 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:38.690 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:38 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:19:38.690 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:38 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:19:38.690 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:38 smithi123 ceph-mon[35245]: Updating smithi123:/etc/ceph/ceph.conf 2024-03-30T04:19:38.690 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:38 smithi123 ceph-mon[35245]: Updating smithi135:/etc/ceph/ceph.conf 2024-03-30T04:19:38.690 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:38 smithi123 ceph-mon[35245]: Updating smithi135:/var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T04:19:38.691 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:38 smithi123 ceph-mon[35245]: Updating smithi123:/var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T04:19:38.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:38 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:38.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:38 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:38.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:38 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:19:38.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:38 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:19:38.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:38 smithi135 ceph-mon[44364]: Updating smithi123:/etc/ceph/ceph.conf 2024-03-30T04:19:38.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:38 smithi135 ceph-mon[44364]: Updating smithi135:/etc/ceph/ceph.conf 2024-03-30T04:19:38.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:38 smithi135 ceph-mon[44364]: Updating smithi135:/var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T04:19:38.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:38 smithi135 ceph-mon[44364]: Updating smithi123:/var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T04:19:39.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:39 smithi123 ceph-mon[35245]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:19:39.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:39 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:39.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:39 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:39.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:39 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:39.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:39 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:39.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:39 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:39.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:39 smithi123 ceph-mon[35245]: Reconfiguring mon.smithi123 (unknown last config time)... 2024-03-30T04:19:39.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:39 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T04:19:39.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:39 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T04:19:39.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:39 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:19:39.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:39 smithi123 ceph-mon[35245]: Reconfiguring daemon mon.smithi123 on smithi123 2024-03-30T04:19:39.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:39 smithi135 ceph-mon[44364]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:19:39.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:39 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:39.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:39 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:39.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:39 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:39.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:39 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:39.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:39 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:39.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:39 smithi135 ceph-mon[44364]: Reconfiguring mon.smithi123 (unknown last config time)... 2024-03-30T04:19:39.852 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:39 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T04:19:39.852 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:39 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T04:19:39.852 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:39 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:19:39.852 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:39 smithi135 ceph-mon[44364]: Reconfiguring daemon mon.smithi123 on smithi123 2024-03-30T04:19:40.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:40.245+0000 7fb013a00700 1 -- 172.21.15.123:0/3434958213 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb00c102ac0 msgr2=0x7fb00c102e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:19:40.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:40.245+0000 7fb013a00700 1 --2- 172.21.15.123:0/3434958213 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb00c102ac0 0x7fb00c102e90 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb0000048c0 tx=0x7fb000009e30 comp rx=0 tx=0).stop 2024-03-30T04:19:40.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:40.246+0000 7fb013a00700 1 -- 172.21.15.123:0/3434958213 shutdown_connections 2024-03-30T04:19:40.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:40.246+0000 7fb013a00700 1 --2- 172.21.15.123:0/3434958213 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb00c102ac0 0x7fb00c102e90 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:19:40.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:40.246+0000 7fb013a00700 1 -- 172.21.15.123:0/3434958213 >> 172.21.15.123:0/3434958213 conn(0x7fb00c0f85e0 msgr2=0x7fb00c0fa9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:19:40.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:40.246+0000 7fb013a00700 1 -- 172.21.15.123:0/3434958213 shutdown_connections 2024-03-30T04:19:40.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:40.246+0000 7fb013a00700 1 -- 172.21.15.123:0/3434958213 wait complete. 2024-03-30T04:19:40.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:40.246+0000 7fb013a00700 1 Processor -- start 2024-03-30T04:19:40.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:40.247+0000 7fb013a00700 1 -- start start 2024-03-30T04:19:40.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:40.247+0000 7fb013a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb00c102ac0 0x7fb00c115430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:19:40.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:40.247+0000 7fb013a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb00c115970 0x7fb00c10f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:19:40.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:40.247+0000 7fb013a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb00c115f30 con 0x7fb00c102ac0 2024-03-30T04:19:40.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:40.247+0000 7fb013a00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb00c116070 con 0x7fb00c115970 2024-03-30T04:19:40.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:40.247+0000 7fb011000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb00c102ac0 0x7fb00c115430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:19:40.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:40.247+0000 7fb011000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb00c102ac0 0x7fb00c115430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39070/0 (socket says 172.21.15.123:39070) 2024-03-30T04:19:40.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:40.247+0000 7fb011000700 1 -- 172.21.15.123:0/2133577848 learned_addr learned my addr 172.21.15.123:0/2133577848 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:19:40.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:40.247+0000 7fb011000700 1 -- 172.21.15.123:0/2133577848 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb00c115970 msgr2=0x7fb00c10f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:19:40.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:40.247+0000 7fb011000700 1 --2- 172.21.15.123:0/2133577848 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb00c115970 0x7fb00c10f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:19:40.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:40.247+0000 7fb011000700 1 -- 172.21.15.123:0/2133577848 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb000005040 con 0x7fb00c102ac0 2024-03-30T04:19:40.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:40.248+0000 7fb011000700 1 --2- 172.21.15.123:0/2133577848 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb00c102ac0 0x7fb00c115430 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb00000bff0 tx=0x7fb000009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:19:40.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:40.248+0000 7fb008c00700 1 -- 172.21.15.123:0/2133577848 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb00000e040 con 0x7fb00c102ac0 2024-03-30T04:19:40.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:40.248+0000 7fb008c00700 1 -- 172.21.15.123:0/2133577848 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb000009930 con 0x7fb00c102ac0 2024-03-30T04:19:40.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:40.248+0000 7fb013a00700 1 -- 172.21.15.123:0/2133577848 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb00c10f960 con 0x7fb00c102ac0 2024-03-30T04:19:40.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:40.248+0000 7fb008c00700 1 -- 172.21.15.123:0/2133577848 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb000013e30 con 0x7fb00c102ac0 2024-03-30T04:19:40.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:40.248+0000 7fb013a00700 1 -- 172.21.15.123:0/2133577848 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb00c10fea0 con 0x7fb00c102ac0 2024-03-30T04:19:40.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:40.249+0000 7fb013a00700 1 -- 172.21.15.123:0/2133577848 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb00c0ffd20 con 0x7fb00c102ac0 2024-03-30T04:19:40.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:40.250+0000 7fb008c00700 1 -- 172.21.15.123:0/2133577848 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 18) v1 ==== 86567+0+0 (secure 0 0 0) 0x7fb000013690 con 0x7fb00c102ac0 2024-03-30T04:19:40.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:40.251+0000 7fb008c00700 1 --2- 172.21.15.123:0/2133577848 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7faff4069360 0x7faff406b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:19:40.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:40.251+0000 7fb008c00700 1 -- 172.21.15.123:0/2133577848 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fb000087580 con 0x7fb00c102ac0 2024-03-30T04:19:40.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:40.251+0000 7fb00be00700 1 --2- 172.21.15.123:0/2133577848 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7faff4069360 0x7faff406b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:19:40.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:40.251+0000 7fb00be00700 1 --2- 172.21.15.123:0/2133577848 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7faff4069360 0x7faff406b810 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7faff8006fd0 tx=0x7faff8006dd0 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:19:40.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:40.255+0000 7fb008c00700 1 -- 172.21.15.123:0/2133577848 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fb000054920 con 0x7fb00c102ac0 2024-03-30T04:19:40.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:40.462+0000 7fb013a00700 1 -- 172.21.15.123:0/2133577848 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fb00c106c70 con 0x7fb00c102ac0 2024-03-30T04:19:40.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:40.463+0000 7fb008c00700 1 -- 172.21.15.123:0/2133577848 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+227 (secure 0 0 0) 0x7fb000013420 con 0x7fb00c102ac0 2024-03-30T04:19:40.739 INFO:teuthology.orchestra.run.smithi123.stdout:# minimal ceph.conf for 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 2024-03-30T04:19:40.739 INFO:teuthology.orchestra.run.smithi123.stdout:[global] 2024-03-30T04:19:40.739 INFO:teuthology.orchestra.run.smithi123.stdout: fsid = 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 2024-03-30T04:19:40.739 INFO:teuthology.orchestra.run.smithi123.stdout: mon_host = [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-03-30T04:19:40.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:40.465+0000 7fb013a00700 1 -- 172.21.15.123:0/2133577848 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7faff4069360 msgr2=0x7faff406b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:19:40.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:40.465+0000 7fb013a00700 1 --2- 172.21.15.123:0/2133577848 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7faff4069360 0x7faff406b810 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7faff8006fd0 tx=0x7faff8006dd0 comp rx=0 tx=0).stop 2024-03-30T04:19:40.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:40.465+0000 7fb013a00700 1 -- 172.21.15.123:0/2133577848 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb00c102ac0 msgr2=0x7fb00c115430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:19:40.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:40.465+0000 7fb013a00700 1 --2- 172.21.15.123:0/2133577848 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb00c102ac0 0x7fb00c115430 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb00000bff0 tx=0x7fb000009e30 comp rx=0 tx=0).stop 2024-03-30T04:19:40.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:40.465+0000 7fb013a00700 1 -- 172.21.15.123:0/2133577848 shutdown_connections 2024-03-30T04:19:40.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:40.465+0000 7fb013a00700 1 --2- 172.21.15.123:0/2133577848 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7faff4069360 0x7faff406b810 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:19:40.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:40.465+0000 7fb013a00700 1 --2- 172.21.15.123:0/2133577848 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb00c102ac0 0x7fb00c115430 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:19:40.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:40.465+0000 7fb013a00700 1 --2- 172.21.15.123:0/2133577848 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb00c115970 0x7fb00c10f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:19:40.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:40.465+0000 7fb013a00700 1 -- 172.21.15.123:0/2133577848 >> 172.21.15.123:0/2133577848 conn(0x7fb00c0f85e0 msgr2=0x7fb00c071460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:19:40.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:40.465+0000 7fb013a00700 1 -- 172.21.15.123:0/2133577848 shutdown_connections 2024-03-30T04:19:40.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:40.465+0000 7fb013a00700 1 -- 172.21.15.123:0/2133577848 wait complete. 2024-03-30T04:19:40.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:40 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/2133577848' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:19:40.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:40 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/2133577848' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:19:41.697 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-30T04:19:41.697 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-30T04:19:41.697 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/ceph/ceph.conf 2024-03-30T04:19:41.800 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:41 smithi123 ceph-mon[35245]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:19:41.800 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:41 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:41.800 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:41 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:41.800 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:41 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:19:41.800 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:41 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi123.xgksqq", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T04:19:41.800 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:41 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T04:19:41.800 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:41 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:19:41.808 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-30T04:19:41.808 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-30T04:19:41.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:41 smithi135 ceph-mon[44364]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:19:41.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:41 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:41.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:41 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:41.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:41 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:19:41.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:41 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi123.xgksqq", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T04:19:41.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:41 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T04:19:41.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:41 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:19:41.914 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-30T04:19:41.914 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/etc/ceph/ceph.conf 2024-03-30T04:19:41.992 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-30T04:19:41.992 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-30T04:19:42.101 INFO:tasks.cephadm:Deploying OSDs... 2024-03-30T04:19:42.101 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-30T04:19:42.101 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/scratch_devs of=/dev/stdout 2024-03-30T04:19:42.145 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-30T04:19:42.145 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_1 2024-03-30T04:19:42.233 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-30T04:19:42.233 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T04:19:42.233 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 5h/5d Inode: 448 Links: 1 2024-03-30T04:19:42.233 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T04:19:42.233 INFO:teuthology.orchestra.run.smithi123.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T04:19:42.233 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2024-03-30 04:18:46.222749880 +0000 2024-03-30T04:19:42.233 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2024-03-30 04:11:10.438497780 +0000 2024-03-30T04:19:42.233 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2024-03-30 04:11:10.438497780 +0000 2024-03-30T04:19:42.233 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: 2024-03-30 04:11:10.412498387 +0000 2024-03-30T04:19:42.234 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-30T04:19:42.299 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2024-03-30T04:19:42.299 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2024-03-30T04:19:42.299 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000299594 s, 1.7 MB/s 2024-03-30T04:19:42.302 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-30T04:19:42.348 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_2 2024-03-30T04:19:42.391 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-30T04:19:42.391 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T04:19:42.391 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 5h/5d Inode: 454 Links: 1 2024-03-30T04:19:42.391 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T04:19:42.391 INFO:teuthology.orchestra.run.smithi123.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T04:19:42.392 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2024-03-30 04:18:46.223749856 +0000 2024-03-30T04:19:42.392 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2024-03-30 04:11:10.906486843 +0000 2024-03-30T04:19:42.392 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2024-03-30 04:11:10.906486843 +0000 2024-03-30T04:19:42.392 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: 2024-03-30 04:11:10.883487381 +0000 2024-03-30T04:19:42.392 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-30T04:19:42.494 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2024-03-30T04:19:42.495 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2024-03-30T04:19:42.495 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000460686 s, 1.1 MB/s 2024-03-30T04:19:42.497 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-30T04:19:42.543 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_3 2024-03-30T04:19:42.589 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-30T04:19:42.589 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T04:19:42.589 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 5h/5d Inode: 460 Links: 1 2024-03-30T04:19:42.590 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T04:19:42.590 INFO:teuthology.orchestra.run.smithi123.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T04:19:42.590 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2024-03-30 04:18:46.223749856 +0000 2024-03-30T04:19:42.590 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2024-03-30 04:11:11.371475977 +0000 2024-03-30T04:19:42.590 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2024-03-30 04:11:11.371475977 +0000 2024-03-30T04:19:42.590 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: 2024-03-30 04:11:11.345476584 +0000 2024-03-30T04:19:42.590 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-30T04:19:42.693 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2024-03-30T04:19:42.693 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2024-03-30T04:19:42.693 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000445952 s, 1.1 MB/s 2024-03-30T04:19:42.695 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-30T04:19:42.741 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_4 2024-03-30T04:19:42.826 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-30T04:19:42.826 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T04:19:42.826 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 5h/5d Inode: 466 Links: 1 2024-03-30T04:19:42.826 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T04:19:42.826 INFO:teuthology.orchestra.run.smithi123.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T04:19:42.826 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2024-03-30 04:18:46.223749856 +0000 2024-03-30T04:19:42.826 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2024-03-30 04:11:11.841464993 +0000 2024-03-30T04:19:42.826 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2024-03-30 04:11:11.841464993 +0000 2024-03-30T04:19:42.827 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: 2024-03-30 04:11:11.816465577 +0000 2024-03-30T04:19:42.827 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-30T04:19:42.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:42 smithi135 ceph-mon[44364]: Reconfiguring mgr.smithi123.xgksqq (unknown last config time)... 2024-03-30T04:19:42.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:42 smithi135 ceph-mon[44364]: Reconfiguring daemon mgr.smithi123.xgksqq on smithi123 2024-03-30T04:19:42.925 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:42 smithi123 ceph-mon[35245]: Reconfiguring mgr.smithi123.xgksqq (unknown last config time)... 2024-03-30T04:19:42.925 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:42 smithi123 ceph-mon[35245]: Reconfiguring daemon mgr.smithi123.xgksqq on smithi123 2024-03-30T04:19:42.930 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2024-03-30T04:19:42.930 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2024-03-30T04:19:42.930 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000296943 s, 1.7 MB/s 2024-03-30T04:19:42.933 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-30T04:19:42.979 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-30T04:19:42.979 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/scratch_devs of=/dev/stdout 2024-03-30T04:19:43.028 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-30T04:19:43.028 DEBUG:teuthology.orchestra.run.smithi135:> stat /dev/vg_nvme/lv_1 2024-03-30T04:19:43.075 INFO:teuthology.orchestra.run.smithi135.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-30T04:19:43.075 INFO:teuthology.orchestra.run.smithi135.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T04:19:43.075 INFO:teuthology.orchestra.run.smithi135.stdout:Device: 5h/5d Inode: 448 Links: 1 2024-03-30T04:19:43.075 INFO:teuthology.orchestra.run.smithi135.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T04:19:43.075 INFO:teuthology.orchestra.run.smithi135.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T04:19:43.075 INFO:teuthology.orchestra.run.smithi135.stdout:Access: 2024-03-30 04:19:27.981054814 +0000 2024-03-30T04:19:43.075 INFO:teuthology.orchestra.run.smithi135.stdout:Modify: 2024-03-30 04:11:08.196726836 +0000 2024-03-30T04:19:43.075 INFO:teuthology.orchestra.run.smithi135.stdout:Change: 2024-03-30 04:11:08.196726836 +0000 2024-03-30T04:19:43.075 INFO:teuthology.orchestra.run.smithi135.stdout: Birth: 2024-03-30 04:11:08.168727427 +0000 2024-03-30T04:19:43.076 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-30T04:19:43.140 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records in 2024-03-30T04:19:43.140 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records out 2024-03-30T04:19:43.141 INFO:teuthology.orchestra.run.smithi135.stderr:512 bytes copied, 0.000319235 s, 1.6 MB/s 2024-03-30T04:19:43.143 DEBUG:teuthology.orchestra.run.smithi135:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-30T04:19:43.189 DEBUG:teuthology.orchestra.run.smithi135:> stat /dev/vg_nvme/lv_2 2024-03-30T04:19:43.274 INFO:teuthology.orchestra.run.smithi135.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-30T04:19:43.274 INFO:teuthology.orchestra.run.smithi135.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T04:19:43.274 INFO:teuthology.orchestra.run.smithi135.stdout:Device: 5h/5d Inode: 454 Links: 1 2024-03-30T04:19:43.274 INFO:teuthology.orchestra.run.smithi135.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T04:19:43.274 INFO:teuthology.orchestra.run.smithi135.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T04:19:43.274 INFO:teuthology.orchestra.run.smithi135.stdout:Access: 2024-03-30 04:19:27.982054792 +0000 2024-03-30T04:19:43.275 INFO:teuthology.orchestra.run.smithi135.stdout:Modify: 2024-03-30 04:11:08.656717141 +0000 2024-03-30T04:19:43.275 INFO:teuthology.orchestra.run.smithi135.stdout:Change: 2024-03-30 04:11:08.656717141 +0000 2024-03-30T04:19:43.275 INFO:teuthology.orchestra.run.smithi135.stdout: Birth: 2024-03-30 04:11:08.630717688 +0000 2024-03-30T04:19:43.275 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-30T04:19:43.377 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records in 2024-03-30T04:19:43.377 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records out 2024-03-30T04:19:43.377 INFO:teuthology.orchestra.run.smithi135.stderr:512 bytes copied, 0.00028665 s, 1.8 MB/s 2024-03-30T04:19:43.380 DEBUG:teuthology.orchestra.run.smithi135:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-30T04:19:43.426 DEBUG:teuthology.orchestra.run.smithi135:> stat /dev/vg_nvme/lv_3 2024-03-30T04:19:43.470 INFO:teuthology.orchestra.run.smithi135.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-30T04:19:43.470 INFO:teuthology.orchestra.run.smithi135.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T04:19:43.470 INFO:teuthology.orchestra.run.smithi135.stdout:Device: 5h/5d Inode: 460 Links: 1 2024-03-30T04:19:43.470 INFO:teuthology.orchestra.run.smithi135.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T04:19:43.470 INFO:teuthology.orchestra.run.smithi135.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T04:19:43.470 INFO:teuthology.orchestra.run.smithi135.stdout:Access: 2024-03-30 04:19:27.982054792 +0000 2024-03-30T04:19:43.470 INFO:teuthology.orchestra.run.smithi135.stdout:Modify: 2024-03-30 04:11:09.106707657 +0000 2024-03-30T04:19:43.470 INFO:teuthology.orchestra.run.smithi135.stdout:Change: 2024-03-30 04:11:09.106707657 +0000 2024-03-30T04:19:43.470 INFO:teuthology.orchestra.run.smithi135.stdout: Birth: 2024-03-30 04:11:09.084708121 +0000 2024-03-30T04:19:43.471 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-30T04:19:43.573 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records in 2024-03-30T04:19:43.573 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records out 2024-03-30T04:19:43.573 INFO:teuthology.orchestra.run.smithi135.stderr:512 bytes copied, 0.000282834 s, 1.8 MB/s 2024-03-30T04:19:43.575 DEBUG:teuthology.orchestra.run.smithi135:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-30T04:19:43.622 DEBUG:teuthology.orchestra.run.smithi135:> stat /dev/vg_nvme/lv_4 2024-03-30T04:19:43.707 INFO:teuthology.orchestra.run.smithi135.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-30T04:19:43.708 INFO:teuthology.orchestra.run.smithi135.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T04:19:43.708 INFO:teuthology.orchestra.run.smithi135.stdout:Device: 5h/5d Inode: 466 Links: 1 2024-03-30T04:19:43.708 INFO:teuthology.orchestra.run.smithi135.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T04:19:43.708 INFO:teuthology.orchestra.run.smithi135.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T04:19:43.708 INFO:teuthology.orchestra.run.smithi135.stdout:Access: 2024-03-30 04:19:27.982054792 +0000 2024-03-30T04:19:43.708 INFO:teuthology.orchestra.run.smithi135.stdout:Modify: 2024-03-30 04:11:09.581697647 +0000 2024-03-30T04:19:43.708 INFO:teuthology.orchestra.run.smithi135.stdout:Change: 2024-03-30 04:11:09.581697647 +0000 2024-03-30T04:19:43.708 INFO:teuthology.orchestra.run.smithi135.stdout: Birth: 2024-03-30 04:11:09.553698237 +0000 2024-03-30T04:19:43.708 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-30T04:19:43.811 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records in 2024-03-30T04:19:43.811 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records out 2024-03-30T04:19:43.811 INFO:teuthology.orchestra.run.smithi135.stderr:512 bytes copied, 0.000284871 s, 1.8 MB/s 2024-03-30T04:19:43.813 DEBUG:teuthology.orchestra.run.smithi135:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-30T04:19:43.859 INFO:tasks.cephadm:Deploying osd.0 on smithi123 with /dev/vg_nvme/lv_4... 2024-03-30T04:19:43.860 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-30T04:19:44.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:44 smithi135 ceph-mon[44364]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:19:44.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:44 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:44.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:44 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:44.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:44 smithi135 ceph-mon[44364]: Reconfiguring crash.smithi123 (monmap changed)... 2024-03-30T04:19:44.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:44 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi123", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-30T04:19:44.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:44 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:19:44.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:44 smithi135 ceph-mon[44364]: Reconfiguring daemon crash.smithi123 on smithi123 2024-03-30T04:19:44.430 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:44 smithi123 ceph-mon[35245]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:19:44.430 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:44 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:44.430 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:44 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:44.430 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:44 smithi123 ceph-mon[35245]: Reconfiguring crash.smithi123 (monmap changed)... 2024-03-30T04:19:44.430 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:44 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi123", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-30T04:19:44.430 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:44 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:19:44.430 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:44 smithi123 ceph-mon[35245]: Reconfiguring daemon crash.smithi123 on smithi123 2024-03-30T04:19:44.776 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:19:46.088 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:45 smithi123 ceph-mon[35245]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:19:46.089 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:45 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:46.089 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:45 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:46.089 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:45 smithi123 ceph-mon[35245]: Reconfiguring alertmanager.smithi123 (dependencies changed)... 2024-03-30T04:19:46.089 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:45 smithi123 ceph-mon[35245]: Reconfiguring daemon alertmanager.smithi123 on smithi123 2024-03-30T04:19:46.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:45 smithi135 ceph-mon[44364]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:19:46.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:45 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:46.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:45 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:46.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:45 smithi135 ceph-mon[44364]: Reconfiguring alertmanager.smithi123 (dependencies changed)... 2024-03-30T04:19:46.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:45 smithi135 ceph-mon[44364]: Reconfiguring daemon alertmanager.smithi123 on smithi123 2024-03-30T04:19:48.024 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:47 smithi123 ceph-mon[35245]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:19:48.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:47 smithi135 ceph-mon[44364]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:19:49.099 INFO:teuthology.orchestra.run.smithi123.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-30T04:19:49.099 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-30T04:19:49.099 INFO:teuthology.orchestra.run.smithi123.stderr: stderr: 10+0 records in 2024-03-30T04:19:49.099 INFO:teuthology.orchestra.run.smithi123.stderr:10+0 records out 2024-03-30T04:19:49.099 INFO:teuthology.orchestra.run.smithi123.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.046753 s, 224 MB/s 2024-03-30T04:19:49.099 INFO:teuthology.orchestra.run.smithi123.stderr:--> Zapping successful for: 2024-03-30T04:19:49.931 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi123:vg_nvme/lv_4 2024-03-30T04:19:50.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:49 smithi135 ceph-mon[44364]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:19:50.150 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:49 smithi123 ceph-mon[35245]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:19:51.060 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:19:52.059 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:51 smithi123 ceph-mon[35245]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:19:52.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:51 smithi135 ceph-mon[44364]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:19:53.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:53.545+0000 7f64e1a00700 1 -- 172.21.15.123:0/156297165 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64dc071f50 msgr2=0x7f64dc070460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:19:53.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:53.545+0000 7f64e1a00700 1 --2- 172.21.15.123:0/156297165 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64dc071f50 0x7f64dc070460 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f64cc004940 tx=0x7f64cc009e30 comp rx=0 tx=0).stop 2024-03-30T04:19:53.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:53.545+0000 7f64e1a00700 1 -- 172.21.15.123:0/156297165 shutdown_connections 2024-03-30T04:19:53.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:53.545+0000 7f64e1a00700 1 --2- 172.21.15.123:0/156297165 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f64dc0709a0 0x7f64dc070e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:19:53.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:53.545+0000 7f64e1a00700 1 --2- 172.21.15.123:0/156297165 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64dc071f50 0x7f64dc070460 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:19:53.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:53.545+0000 7f64e1a00700 1 -- 172.21.15.123:0/156297165 >> 172.21.15.123:0/156297165 conn(0x7f64dc0f8560 msgr2=0x7f64dc0fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:19:53.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:53.545+0000 7f64e1a00700 1 -- 172.21.15.123:0/156297165 shutdown_connections 2024-03-30T04:19:53.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:53.545+0000 7f64e1a00700 1 -- 172.21.15.123:0/156297165 wait complete. 2024-03-30T04:19:53.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:53.546+0000 7f64e1a00700 1 Processor -- start 2024-03-30T04:19:53.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:53.546+0000 7f64e1a00700 1 -- start start 2024-03-30T04:19:53.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:53.547+0000 7f64e1a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f64dc0709a0 0x7f64dc10fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:19:53.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:53.547+0000 7f64e1a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64dc071f50 0x7f64dc110090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:19:53.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:53.547+0000 7f64e1a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64dc110690 con 0x7f64dc071f50 2024-03-30T04:19:53.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:53.547+0000 7f64e1a00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64dc110ee0 con 0x7f64dc0709a0 2024-03-30T04:19:53.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:53.547+0000 7f64d3e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64dc071f50 0x7f64dc110090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:19:53.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:53.547+0000 7f64d3e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64dc071f50 0x7f64dc110090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53856/0 (socket says 172.21.15.123:53856) 2024-03-30T04:19:53.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:53.547+0000 7f64d3e00700 1 -- 172.21.15.123:0/1140866165 learned_addr learned my addr 172.21.15.123:0/1140866165 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:19:53.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:53.547+0000 7f64daa00700 1 --2- 172.21.15.123:0/1140866165 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f64dc0709a0 0x7f64dc10fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:19:53.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:53.547+0000 7f64d3e00700 1 -- 172.21.15.123:0/1140866165 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f64dc0709a0 msgr2=0x7f64dc10fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:19:53.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:53.547+0000 7f64d3e00700 1 --2- 172.21.15.123:0/1140866165 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f64dc0709a0 0x7f64dc10fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:19:53.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:53.547+0000 7f64d3e00700 1 -- 172.21.15.123:0/1140866165 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64cc005040 con 0x7f64dc071f50 2024-03-30T04:19:53.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:53.548+0000 7f64d3e00700 1 --2- 172.21.15.123:0/1140866165 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64dc071f50 0x7f64dc110090 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f64c400c8c0 tx=0x7f64c40044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:19:53.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:53.548+0000 7f64d2a00700 1 -- 172.21.15.123:0/1140866165 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f64c400a760 con 0x7f64dc071f50 2024-03-30T04:19:53.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:53.548+0000 7f64d2a00700 1 -- 172.21.15.123:0/1140866165 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f64c400ad70 con 0x7f64dc071f50 2024-03-30T04:19:53.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:53.548+0000 7f64d2a00700 1 -- 172.21.15.123:0/1140866165 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f64c40113a0 con 0x7f64dc071f50 2024-03-30T04:19:53.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:53.548+0000 7f64e1a00700 1 -- 172.21.15.123:0/1140866165 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64dc1111c0 con 0x7f64dc071f50 2024-03-30T04:19:53.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:53.548+0000 7f64e1a00700 1 -- 172.21.15.123:0/1140866165 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64dc111700 con 0x7f64dc071f50 2024-03-30T04:19:53.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:53.549+0000 7f64e1a00700 1 -- 172.21.15.123:0/1140866165 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64dc05f5f0 con 0x7f64dc071f50 2024-03-30T04:19:53.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:53.550+0000 7f64d2a00700 1 -- 172.21.15.123:0/1140866165 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 18) v1 ==== 86567+0+0 (secure 0 0 0) 0x7f64c4012400 con 0x7f64dc071f50 2024-03-30T04:19:53.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:53.551+0000 7f64d2a00700 1 --2- 172.21.15.123:0/1140866165 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f64c0069210 0x7f64c006b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:19:53.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:53.551+0000 7f64d2a00700 1 -- 172.21.15.123:0/1140866165 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f64c4085fd0 con 0x7f64dc071f50 2024-03-30T04:19:53.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:53.551+0000 7f64daa00700 1 --2- 172.21.15.123:0/1140866165 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f64c0069210 0x7f64c006b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:19:53.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:53.552+0000 7f64daa00700 1 --2- 172.21.15.123:0/1140866165 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f64c0069210 0x7f64c006b6c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f64cc00b3b0 tx=0x7f64cc009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:19:53.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:53.556+0000 7f64d2a00700 1 -- 172.21.15.123:0/1140866165 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f64c4056f10 con 0x7f64dc071f50 2024-03-30T04:19:53.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:19:53.772+0000 7f64e1a00700 1 -- 172.21.15.123:0/1140866165 --> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f64dc111aa0 con 0x7f64c0069210 2024-03-30T04:19:54.051 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:53 smithi123 ceph-mon[35245]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:19:54.051 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:53 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:54.051 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:53 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:54.051 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:53 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T04:19:54.052 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:53 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T04:19:54.052 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:53 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:19:54.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:53 smithi135 ceph-mon[44364]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:19:54.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:53 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:54.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:53 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:19:54.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:53 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T04:19:54.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:53 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T04:19:54.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:53 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:19:54.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:54 smithi123 ceph-mon[35245]: Reconfiguring grafana.smithi123 (dependencies changed)... 2024-03-30T04:19:54.908 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:54 smithi123 ceph-mon[35245]: Reconfiguring daemon grafana.smithi123 on smithi123 2024-03-30T04:19:54.908 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:54 smithi123 ceph-mon[35245]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:19:55.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:54 smithi135 ceph-mon[44364]: Reconfiguring grafana.smithi123 (dependencies changed)... 2024-03-30T04:19:55.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:54 smithi135 ceph-mon[44364]: Reconfiguring daemon grafana.smithi123 on smithi123 2024-03-30T04:19:55.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:54 smithi135 ceph-mon[44364]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:19:56.103 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:55 smithi135 ceph-mon[44364]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:19:56.131 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:55 smithi123 ceph-mon[35245]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:19:57.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:56 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:19:57.176 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:56 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:19:58.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:19:57 smithi135 ceph-mon[44364]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:19:58.157 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:19:57 smithi123 ceph-mon[35245]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:20:00.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:00 smithi135 ceph-mon[44364]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:20:00.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:00 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:00.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:00 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:00.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:00 smithi135 ceph-mon[44364]: Reconfiguring prometheus.smithi123 (dependencies changed)... 2024-03-30T04:20:00.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:00 smithi135 ceph-mon[44364]: Reconfiguring daemon prometheus.smithi123 on smithi123 2024-03-30T04:20:00.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:00 smithi135 ceph-mon[44364]: overall HEALTH_OK 2024-03-30T04:20:00.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:00 smithi123 ceph-mon[35245]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:20:00.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:00 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:00.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:00 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:00.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:00 smithi123 ceph-mon[35245]: Reconfiguring prometheus.smithi123 (dependencies changed)... 2024-03-30T04:20:00.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:00 smithi123 ceph-mon[35245]: Reconfiguring daemon prometheus.smithi123 on smithi123 2024-03-30T04:20:00.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:00 smithi123 ceph-mon[35245]: overall HEALTH_OK 2024-03-30T04:20:01.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:01 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/469761169' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3968446b-8696-4eec-b640-c97973ca054d"}]: dispatch 2024-03-30T04:20:01.333 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:01 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/469761169' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3968446b-8696-4eec-b640-c97973ca054d"}]': finished 2024-03-30T04:20:01.334 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:01 smithi123 ceph-mon[35245]: osdmap e6: 1 total, 0 up, 1 in 2024-03-30T04:20:01.334 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:01 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T04:20:01.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:01 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/469761169' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3968446b-8696-4eec-b640-c97973ca054d"}]: dispatch 2024-03-30T04:20:01.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:01 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/469761169' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3968446b-8696-4eec-b640-c97973ca054d"}]': finished 2024-03-30T04:20:01.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:01 smithi135 ceph-mon[44364]: osdmap e6: 1 total, 0 up, 1 in 2024-03-30T04:20:01.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:01 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T04:20:02.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:02 smithi123 ceph-mon[35245]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:20:02.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:02 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/4231524623' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T04:20:02.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:02 smithi135 ceph-mon[44364]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:20:02.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:02 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/4231524623' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T04:20:04.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:04 smithi123 ceph-mon[35245]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:20:04.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:04 smithi135 ceph-mon[44364]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:20:06.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:06 smithi123 ceph-mon[35245]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:20:06.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:06 smithi135 ceph-mon[44364]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:20:08.430 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:08 smithi123 ceph-mon[35245]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:20:08.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:08 smithi135 ceph-mon[44364]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:20:10.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:10 smithi123 ceph-mon[35245]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:20:10.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:10 smithi135 ceph-mon[44364]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:20:12.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:12 smithi123 ceph-mon[35245]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:20:12.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:12 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:20:12.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:12 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:12.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:12 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:12.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:12 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi135", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-30T04:20:12.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:12 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:20:12.596 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:12 smithi135 ceph-mon[44364]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:20:12.596 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:12 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:20:12.596 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:12 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:12.597 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:12 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:12.597 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:12 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi135", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-30T04:20:12.597 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:12 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:20:13.390 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:13 smithi123 ceph-mon[35245]: Reconfiguring crash.smithi135 (monmap changed)... 2024-03-30T04:20:13.390 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:13 smithi123 ceph-mon[35245]: Reconfiguring daemon crash.smithi135 on smithi135 2024-03-30T04:20:13.562 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:13 smithi135 ceph-mon[44364]: Reconfiguring crash.smithi135 (monmap changed)... 2024-03-30T04:20:13.563 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:13 smithi135 ceph-mon[44364]: Reconfiguring daemon crash.smithi135 on smithi135 2024-03-30T04:20:14.385 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:14 smithi135 ceph-mon[44364]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:20:14.386 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:14 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-30T04:20:14.386 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:14 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:20:14.386 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:14 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:14.386 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:14 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:14.386 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:14 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi135.kyblli", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T04:20:14.386 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:14 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T04:20:14.386 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:14 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:20:14.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:14 smithi123 ceph-mon[35245]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:20:14.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:14 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-30T04:20:14.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:14 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:20:14.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:14 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:14.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:14 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:14.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:14 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi135.kyblli", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T04:20:14.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:14 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T04:20:14.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:14 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:20:15.238 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:15 smithi135 ceph-mon[44364]: Deploying daemon osd.0 on smithi123 2024-03-30T04:20:15.238 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:15 smithi135 ceph-mon[44364]: Reconfiguring mgr.smithi135.kyblli (monmap changed)... 2024-03-30T04:20:15.238 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:15 smithi135 ceph-mon[44364]: Reconfiguring daemon mgr.smithi135.kyblli on smithi135 2024-03-30T04:20:15.450 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:15 smithi123 ceph-mon[35245]: Deploying daemon osd.0 on smithi123 2024-03-30T04:20:15.450 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:15 smithi123 ceph-mon[35245]: Reconfiguring mgr.smithi135.kyblli (monmap changed)... 2024-03-30T04:20:15.450 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:15 smithi123 ceph-mon[35245]: Reconfiguring daemon mgr.smithi135.kyblli on smithi135 2024-03-30T04:20:16.180 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:16 smithi123 ceph-mon[35245]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:20:16.180 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:16 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:16.180 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:16 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:16.180 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:16 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T04:20:16.180 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:16 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T04:20:16.181 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:16 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:20:16.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:16 smithi135 ceph-mon[44364]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:20:16.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:16 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:16.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:16 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:16.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:16 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T04:20:16.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:16 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T04:20:16.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:16 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:20:17.231 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:17 smithi123 ceph-mon[35245]: Reconfiguring mon.smithi135 (monmap changed)... 2024-03-30T04:20:17.231 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:17 smithi123 ceph-mon[35245]: Reconfiguring daemon mon.smithi135 on smithi135 2024-03-30T04:20:17.231 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:17 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:17.231 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:17 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:17.231 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:17 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-30T04:20:17.231 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:17 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi123.front.sepia.ceph.com:9093"}]: dispatch 2024-03-30T04:20:17.231 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:17 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:17.231 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:17 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-30T04:20:17.231 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:17 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi123.front.sepia.ceph.com:3000"}]: dispatch 2024-03-30T04:20:17.232 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:17 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:17.232 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:17 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-30T04:20:17.232 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:17 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi123.front.sepia.ceph.com:9095"}]: dispatch 2024-03-30T04:20:17.232 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:17 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:17.232 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:17 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:20:17.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:17 smithi135 ceph-mon[44364]: Reconfiguring mon.smithi135 (monmap changed)... 2024-03-30T04:20:17.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:17 smithi135 ceph-mon[44364]: Reconfiguring daemon mon.smithi135 on smithi135 2024-03-30T04:20:17.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:17 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:17.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:17 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:17.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:17 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-30T04:20:17.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:17 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi123.front.sepia.ceph.com:9093"}]: dispatch 2024-03-30T04:20:17.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:17 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:17.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:17 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-30T04:20:17.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:17 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi123.front.sepia.ceph.com:3000"}]: dispatch 2024-03-30T04:20:17.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:17 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:17.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:17 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-30T04:20:17.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:17 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi123.front.sepia.ceph.com:9095"}]: dispatch 2024-03-30T04:20:17.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:17 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:17.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:17 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:20:18.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:18 smithi123 ceph-mon[35245]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:20:18.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:18 smithi123 ceph-mon[35245]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-30T04:20:18.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:18 smithi123 ceph-mon[35245]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi123.front.sepia.ceph.com:9093"}]: dispatch 2024-03-30T04:20:18.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:18 smithi123 ceph-mon[35245]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-30T04:20:18.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:18 smithi123 ceph-mon[35245]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi123.front.sepia.ceph.com:3000"}]: dispatch 2024-03-30T04:20:18.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:18 smithi123 ceph-mon[35245]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-30T04:20:18.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:18 smithi123 ceph-mon[35245]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi123.front.sepia.ceph.com:9095"}]: dispatch 2024-03-30T04:20:18.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:18 smithi135 ceph-mon[44364]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:20:18.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:18 smithi135 ceph-mon[44364]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-30T04:20:18.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:18 smithi135 ceph-mon[44364]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi123.front.sepia.ceph.com:9093"}]: dispatch 2024-03-30T04:20:18.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:18 smithi135 ceph-mon[44364]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-30T04:20:18.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:18 smithi135 ceph-mon[44364]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi123.front.sepia.ceph.com:3000"}]: dispatch 2024-03-30T04:20:18.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:18 smithi135 ceph-mon[44364]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-30T04:20:18.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:18 smithi135 ceph-mon[44364]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi123.front.sepia.ceph.com:9095"}]: dispatch 2024-03-30T04:20:20.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:20 smithi123 ceph-mon[35245]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:20:20.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:20 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:20.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:20 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:20.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:20 smithi135 ceph-mon[44364]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:20:20.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:20 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:20.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:20 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:21.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:21 smithi123 ceph-mon[35245]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:20:21.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:21 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:21.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:21 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:21.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:21 smithi135 ceph-mon[44364]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:20:21.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:21 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:21.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:21 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:23.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:23 smithi123 ceph-mon[35245]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:20:23.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:23 smithi135 ceph-mon[44364]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:20:24.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:24 smithi123 ceph-mon[35245]: from='osd.0 [v2:172.21.15.123:6802/1768008525,v1:172.21.15.123:6803/1768008525]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-30T04:20:24.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:24 smithi135 ceph-mon[44364]: from='osd.0 [v2:172.21.15.123:6802/1768008525,v1:172.21.15.123:6803/1768008525]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-30T04:20:25.627 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:25 smithi123 ceph-mon[35245]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:20:25.627 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:25 smithi123 ceph-mon[35245]: from='osd.0 [v2:172.21.15.123:6802/1768008525,v1:172.21.15.123:6803/1768008525]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-30T04:20:25.627 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:25 smithi123 ceph-mon[35245]: osdmap e7: 1 total, 0 up, 1 in 2024-03-30T04:20:25.627 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:25 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T04:20:25.627 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:25 smithi123 ceph-mon[35245]: from='osd.0 [v2:172.21.15.123:6802/1768008525,v1:172.21.15.123:6803/1768008525]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-03-30T04:20:25.628 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:25 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:25.628 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:25 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:25.710 INFO:teuthology.orchestra.run.smithi123.stdout:Created osd(s) 0 on host 'smithi123' 2024-03-30T04:20:25.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:25.711+0000 7f64d2a00700 1 -- 172.21.15.123:0/1140866165 <== mgr.14206 v2:172.21.15.123:6800/1564935593 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f64dc111aa0 con 0x7f64c0069210 2024-03-30T04:20:25.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:25.713+0000 7f64e1a00700 1 -- 172.21.15.123:0/1140866165 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f64c0069210 msgr2=0x7f64c006b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:20:25.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:25.713+0000 7f64e1a00700 1 --2- 172.21.15.123:0/1140866165 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f64c0069210 0x7f64c006b6c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f64cc00b3b0 tx=0x7f64cc009e30 comp rx=0 tx=0).stop 2024-03-30T04:20:25.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:25.713+0000 7f64e1a00700 1 -- 172.21.15.123:0/1140866165 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64dc071f50 msgr2=0x7f64dc110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:20:25.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:25.713+0000 7f64e1a00700 1 --2- 172.21.15.123:0/1140866165 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64dc071f50 0x7f64dc110090 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f64c400c8c0 tx=0x7f64c40044c0 comp rx=0 tx=0).stop 2024-03-30T04:20:25.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:25.714+0000 7f64e1a00700 1 -- 172.21.15.123:0/1140866165 shutdown_connections 2024-03-30T04:20:25.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:25.714+0000 7f64e1a00700 1 --2- 172.21.15.123:0/1140866165 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f64c0069210 0x7f64c006b6c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:20:25.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:25.714+0000 7f64e1a00700 1 --2- 172.21.15.123:0/1140866165 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f64dc0709a0 0x7f64dc10fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:20:25.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:25.714+0000 7f64e1a00700 1 --2- 172.21.15.123:0/1140866165 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64dc071f50 0x7f64dc110090 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:20:25.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:25.714+0000 7f64e1a00700 1 -- 172.21.15.123:0/1140866165 >> 172.21.15.123:0/1140866165 conn(0x7f64dc0f8560 msgr2=0x7f64dc0f9d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:20:25.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:25.714+0000 7f64e1a00700 1 -- 172.21.15.123:0/1140866165 shutdown_connections 2024-03-30T04:20:25.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:25.714+0000 7f64e1a00700 1 -- 172.21.15.123:0/1140866165 wait complete. 2024-03-30T04:20:25.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:25 smithi135 ceph-mon[44364]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:20:25.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:25 smithi135 ceph-mon[44364]: from='osd.0 [v2:172.21.15.123:6802/1768008525,v1:172.21.15.123:6803/1768008525]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-30T04:20:25.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:25 smithi135 ceph-mon[44364]: osdmap e7: 1 total, 0 up, 1 in 2024-03-30T04:20:25.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:25 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T04:20:25.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:25 smithi135 ceph-mon[44364]: from='osd.0 [v2:172.21.15.123:6802/1768008525,v1:172.21.15.123:6803/1768008525]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-03-30T04:20:25.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:25 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:25.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:25 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:26.414 DEBUG:teuthology.orchestra.run.smithi123:osd.0> sudo journalctl -f -n 0 -u ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4@osd.0.service 2024-03-30T04:20:26.418 INFO:tasks.cephadm:Deploying osd.1 on smithi123 with /dev/vg_nvme/lv_3... 2024-03-30T04:20:26.418 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-30T04:20:26.488 INFO:journalctl@ceph.osd.0.smithi123.stdout:-- Logs begin at Sat 2024-03-30 04:04:15 UTC. -- 2024-03-30T04:20:26.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:26 smithi123 ceph-mon[35245]: from='osd.0 [v2:172.21.15.123:6802/1768008525,v1:172.21.15.123:6803/1768008525]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-03-30T04:20:26.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:26 smithi123 ceph-mon[35245]: osdmap e8: 1 total, 0 up, 1 in 2024-03-30T04:20:26.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:26 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T04:20:26.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:26 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T04:20:26.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:26 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:26.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:26 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:26.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:26 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:20:26.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:26 smithi135 ceph-mon[44364]: from='osd.0 [v2:172.21.15.123:6802/1768008525,v1:172.21.15.123:6803/1768008525]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-03-30T04:20:26.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:26 smithi135 ceph-mon[44364]: osdmap e8: 1 total, 0 up, 1 in 2024-03-30T04:20:26.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:26 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T04:20:26.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:26 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T04:20:26.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:26 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:26.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:26 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:26.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:26 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:20:27.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:27 smithi123 ceph-mon[35245]: purged_snaps scrub starts 2024-03-30T04:20:27.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:27 smithi123 ceph-mon[35245]: purged_snaps scrub ok 2024-03-30T04:20:27.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:27 smithi123 ceph-mon[35245]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:20:27.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:27 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T04:20:27.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:27 smithi123 ceph-mon[35245]: osd.0 [v2:172.21.15.123:6802/1768008525,v1:172.21.15.123:6803/1768008525] boot 2024-03-30T04:20:27.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:27 smithi123 ceph-mon[35245]: osdmap e9: 1 total, 1 up, 1 in 2024-03-30T04:20:27.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:27 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T04:20:27.740 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:20:27.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:27 smithi135 ceph-mon[44364]: purged_snaps scrub starts 2024-03-30T04:20:27.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:27 smithi135 ceph-mon[44364]: purged_snaps scrub ok 2024-03-30T04:20:27.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:27 smithi135 ceph-mon[44364]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T04:20:27.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:27 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T04:20:27.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:27 smithi135 ceph-mon[44364]: osd.0 [v2:172.21.15.123:6802/1768008525,v1:172.21.15.123:6803/1768008525] boot 2024-03-30T04:20:27.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:27 smithi135 ceph-mon[44364]: osdmap e9: 1 total, 1 up, 1 in 2024-03-30T04:20:27.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:27 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T04:20:28.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:28 smithi123 ceph-mon[35245]: osdmap e10: 1 total, 1 up, 1 in 2024-03-30T04:20:29.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:28 smithi135 ceph-mon[44364]: osdmap e10: 1 total, 1 up, 1 in 2024-03-30T04:20:29.951 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:29 smithi123 ceph-mon[35245]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-30T04:20:30.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:29 smithi135 ceph-mon[44364]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-30T04:20:30.569 INFO:teuthology.orchestra.run.smithi123.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-30T04:20:30.569 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-30T04:20:30.569 INFO:teuthology.orchestra.run.smithi123.stderr: stderr: 10+0 records in 2024-03-30T04:20:30.569 INFO:teuthology.orchestra.run.smithi123.stderr:10+0 records out 2024-03-30T04:20:30.569 INFO:teuthology.orchestra.run.smithi123.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0308341 s, 340 MB/s 2024-03-30T04:20:30.569 INFO:teuthology.orchestra.run.smithi123.stderr:--> Zapping successful for: 2024-03-30T04:20:31.394 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi123:vg_nvme/lv_3 2024-03-30T04:20:31.978 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:31 smithi135 ceph-mon[44364]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-30T04:20:31.978 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:31 smithi135 ceph-mon[44364]: Detected new or changed devices on smithi123 2024-03-30T04:20:31.978 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:31 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:31.978 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:31 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:31.979 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:31 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:20:31.979 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:31 smithi135 ceph-mon[44364]: Adjusting osd_memory_target on smithi123 to 13046M 2024-03-30T04:20:31.979 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:31 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:31.979 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:31 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:20:31.979 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:31 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:20:31.979 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:31 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:31.979 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:31 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:20:32.028 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:31 smithi123 ceph-mon[35245]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-30T04:20:32.028 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:31 smithi123 ceph-mon[35245]: Detected new or changed devices on smithi123 2024-03-30T04:20:32.028 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:31 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:32.028 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:31 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:32.028 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:31 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:20:32.028 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:31 smithi123 ceph-mon[35245]: Adjusting osd_memory_target on smithi123 to 13046M 2024-03-30T04:20:32.028 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:31 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:32.028 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:31 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:20:32.028 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:31 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:20:32.029 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:31 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:32.029 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:31 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:20:32.337 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:20:34.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:33 smithi135 ceph-mon[44364]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-30T04:20:34.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:33 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:34.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:33 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:34.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:33 smithi123 ceph-mon[35245]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-30T04:20:34.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:33 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:34.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:33 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:34.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:34.601+0000 7fbacb000700 1 -- 172.21.15.123:0/2549818233 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbac40fdff0 msgr2=0x7fbac40fe460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:20:34.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:34.601+0000 7fbacb000700 1 --2- 172.21.15.123:0/2549818233 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbac40fdff0 0x7fbac40fe460 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbab0004940 tx=0x7fbab0009e30 comp rx=0 tx=0).stop 2024-03-30T04:20:34.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:34.601+0000 7fbacb000700 1 -- 172.21.15.123:0/2549818233 shutdown_connections 2024-03-30T04:20:34.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:34.601+0000 7fbacb000700 1 --2- 172.21.15.123:0/2549818233 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbac40fdff0 0x7fbac40fe460 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:20:34.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:34.601+0000 7fbacb000700 1 --2- 172.21.15.123:0/2549818233 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbac40fce80 0x7fbac40fd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:20:34.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:34.601+0000 7fbacb000700 1 -- 172.21.15.123:0/2549818233 >> 172.21.15.123:0/2549818233 conn(0x7fbac40f8540 msgr2=0x7fbac40fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:20:34.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:34.602+0000 7fbacb000700 1 -- 172.21.15.123:0/2549818233 shutdown_connections 2024-03-30T04:20:34.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:34.602+0000 7fbacb000700 1 -- 172.21.15.123:0/2549818233 wait complete. 2024-03-30T04:20:34.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:34.602+0000 7fbacb000700 1 Processor -- start 2024-03-30T04:20:34.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:34.602+0000 7fbacb000700 1 -- start start 2024-03-30T04:20:34.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:34.603+0000 7fbacb000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbac40fce80 0x7fbac410d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:20:34.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:34.603+0000 7fbacb000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbac40fdff0 0x7fbac410de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:20:34.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:34.603+0000 7fbacb000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac410e430 con 0x7fbac40fdff0 2024-03-30T04:20:34.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:34.603+0000 7fbacb000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac410ec80 con 0x7fbac40fce80 2024-03-30T04:20:34.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:34.603+0000 7fbac3400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbac40fdff0 0x7fbac410de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:20:34.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:34.603+0000 7fbac3400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbac40fdff0 0x7fbac410de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44302/0 (socket says 172.21.15.123:44302) 2024-03-30T04:20:34.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:34.603+0000 7fbac3400700 1 -- 172.21.15.123:0/323530171 learned_addr learned my addr 172.21.15.123:0/323530171 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:20:34.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:34.603+0000 7fbac3e00700 1 --2- 172.21.15.123:0/323530171 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbac40fce80 0x7fbac410d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:20:34.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:34.603+0000 7fbac3400700 1 -- 172.21.15.123:0/323530171 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbac40fce80 msgr2=0x7fbac410d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:20:34.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:34.603+0000 7fbac3400700 1 --2- 172.21.15.123:0/323530171 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbac40fce80 0x7fbac410d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:20:34.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:34.603+0000 7fbac3400700 1 -- 172.21.15.123:0/323530171 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbab0005040 con 0x7fbac40fdff0 2024-03-30T04:20:34.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:34.604+0000 7fbac3400700 1 --2- 172.21.15.123:0/323530171 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbac40fdff0 0x7fbac410de30 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbab000b0d0 tx=0x7fbab0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:20:34.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:34.604+0000 7fbabbe00700 1 -- 172.21.15.123:0/323530171 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbab000e070 con 0x7fbac40fdff0 2024-03-30T04:20:34.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:34.604+0000 7fbabbe00700 1 -- 172.21.15.123:0/323530171 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbab000b410 con 0x7fbac40fdff0 2024-03-30T04:20:34.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:34.604+0000 7fbacb000700 1 -- 172.21.15.123:0/323530171 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbac410ef00 con 0x7fbac40fdff0 2024-03-30T04:20:34.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:34.604+0000 7fbabbe00700 1 -- 172.21.15.123:0/323530171 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbab0012600 con 0x7fbac40fdff0 2024-03-30T04:20:34.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:34.604+0000 7fbacb000700 1 -- 172.21.15.123:0/323530171 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbac410f440 con 0x7fbac40fdff0 2024-03-30T04:20:34.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:34.605+0000 7fbacb000700 1 -- 172.21.15.123:0/323530171 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbac405f5f0 con 0x7fbac40fdff0 2024-03-30T04:20:34.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:34.606+0000 7fbabbe00700 1 -- 172.21.15.123:0/323530171 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 18) v1 ==== 86567+0+0 (secure 0 0 0) 0x7fbab000b9f0 con 0x7fbac40fdff0 2024-03-30T04:20:34.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:34.607+0000 7fbabbe00700 1 --2- 172.21.15.123:0/323530171 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fbaa8069210 0x7fbaa806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:20:34.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:34.607+0000 7fbabbe00700 1 -- 172.21.15.123:0/323530171 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1978+0+0 (secure 0 0 0) 0x7fbab0086fe0 con 0x7fbac40fdff0 2024-03-30T04:20:34.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:34.607+0000 7fbac3e00700 1 --2- 172.21.15.123:0/323530171 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fbaa8069210 0x7fbaa806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:20:34.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:34.608+0000 7fbac3e00700 1 --2- 172.21.15.123:0/323530171 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fbaa8069210 0x7fbaa806b6c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fbab4009b80 tx=0x7fbab400b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:20:34.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:34.612+0000 7fbabbe00700 1 -- 172.21.15.123:0/323530171 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fbab0057c70 con 0x7fbac40fdff0 2024-03-30T04:20:34.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:34.826+0000 7fbacb000700 1 -- 172.21.15.123:0/323530171 --> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fbac410f720 con 0x7fbaa8069210 2024-03-30T04:20:36.055 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:35 smithi123 ceph-mon[35245]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-30T04:20:36.055 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:35 smithi123 ceph-mon[35245]: from='client.14262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:20:36.055 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:35 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T04:20:36.055 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:35 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T04:20:36.056 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:35 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:20:36.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:35 smithi135 ceph-mon[44364]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-30T04:20:36.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:35 smithi135 ceph-mon[44364]: from='client.14262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:20:36.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:35 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T04:20:36.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:35 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T04:20:36.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:35 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:20:38.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:37 smithi135 ceph-mon[44364]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-30T04:20:38.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:37 smithi123 ceph-mon[35245]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-30T04:20:39.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:39 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:39.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:39 smithi135 ceph-mon[44364]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-30T04:20:39.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:39 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:39.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:39 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:20:39.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:39 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:20:39.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:39 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:39.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:39 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/1270189169' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "336be34b-bb3c-46ab-8439-af04a1366db5"}]: dispatch 2024-03-30T04:20:39.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:39 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/1270189169' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "336be34b-bb3c-46ab-8439-af04a1366db5"}]': finished 2024-03-30T04:20:39.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:39 smithi135 ceph-mon[44364]: osdmap e11: 2 total, 1 up, 2 in 2024-03-30T04:20:39.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:39 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T04:20:39.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:39 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:39.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:39 smithi123 ceph-mon[35245]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-30T04:20:39.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:39 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:39.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:39 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:20:39.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:39 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:20:39.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:39 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:39.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:39 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/1270189169' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "336be34b-bb3c-46ab-8439-af04a1366db5"}]: dispatch 2024-03-30T04:20:39.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:39 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/1270189169' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "336be34b-bb3c-46ab-8439-af04a1366db5"}]': finished 2024-03-30T04:20:39.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:39 smithi123 ceph-mon[35245]: osdmap e11: 2 total, 1 up, 2 in 2024-03-30T04:20:39.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:39 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T04:20:40.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:40 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/2277593076' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T04:20:40.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:40 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/2277593076' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T04:20:41.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:41 smithi135 ceph-mon[44364]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-30T04:20:41.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:41 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:20:41.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:41 smithi123 ceph-mon[35245]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-30T04:20:41.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:41 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:20:43.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:43 smithi135 ceph-mon[44364]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-30T04:20:43.601 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:43 smithi123 ceph-mon[35245]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-30T04:20:45.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:45 smithi135 ceph-mon[44364]: pgmap v53: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-30T04:20:45.668 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:45 smithi123 ceph-mon[35245]: pgmap v53: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-30T04:20:47.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:47 smithi135 ceph-mon[44364]: pgmap v54: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-30T04:20:47.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:47 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-30T04:20:47.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:47 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:20:47.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:47 smithi135 ceph-mon[44364]: Deploying daemon osd.1 on smithi123 2024-03-30T04:20:47.655 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:47 smithi123 ceph-mon[35245]: pgmap v54: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-30T04:20:47.655 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:47 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-30T04:20:47.655 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:47 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:20:47.655 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:47 smithi123 ceph-mon[35245]: Deploying daemon osd.1 on smithi123 2024-03-30T04:20:49.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:49 smithi135 ceph-mon[44364]: pgmap v55: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-30T04:20:49.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:49 smithi123 ceph-mon[35245]: pgmap v55: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-30T04:20:51.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:51 smithi135 ceph-mon[44364]: pgmap v56: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-30T04:20:51.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:51 smithi123 ceph-mon[35245]: pgmap v56: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-30T04:20:53.591 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:53 smithi123 ceph-mon[35245]: pgmap v57: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-30T04:20:53.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:53 smithi135 ceph-mon[44364]: pgmap v57: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-30T04:20:54.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:54 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:54.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:54 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:54.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:54 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:20:55.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:54 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:55.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:54 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:55.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:54 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:20:55.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:55 smithi123 ceph-mon[35245]: pgmap v58: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-30T04:20:56.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:55 smithi135 ceph-mon[44364]: pgmap v58: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-30T04:20:57.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:57 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:57.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:57 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:57.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:57 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:20:57.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:57 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:57.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:57 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:57.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:57 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:20:58.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:58 smithi135 ceph-mon[44364]: pgmap v59: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-30T04:20:58.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:58 smithi135 ceph-mon[44364]: from='osd.1 [v2:172.21.15.123:6810/1040012134,v1:172.21.15.123:6811/1040012134]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-30T04:20:58.404 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:58 smithi123 ceph-mon[35245]: pgmap v59: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-30T04:20:58.404 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:58 smithi123 ceph-mon[35245]: from='osd.1 [v2:172.21.15.123:6810/1040012134,v1:172.21.15.123:6811/1040012134]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-30T04:20:58.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:58.748+0000 7fbabbe00700 1 -- 172.21.15.123:0/323530171 <== mgr.14206 v2:172.21.15.123:6800/1564935593 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fbac410f720 con 0x7fbaa8069210 2024-03-30T04:20:58.747 INFO:teuthology.orchestra.run.smithi123.stdout:Created osd(s) 1 on host 'smithi123' 2024-03-30T04:20:58.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:58.751+0000 7fbacb000700 1 -- 172.21.15.123:0/323530171 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fbaa8069210 msgr2=0x7fbaa806b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:20:58.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:58.751+0000 7fbacb000700 1 --2- 172.21.15.123:0/323530171 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fbaa8069210 0x7fbaa806b6c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fbab4009b80 tx=0x7fbab400b040 comp rx=0 tx=0).stop 2024-03-30T04:20:58.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:58.751+0000 7fbacb000700 1 -- 172.21.15.123:0/323530171 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbac40fdff0 msgr2=0x7fbac410de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:20:58.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:58.751+0000 7fbacb000700 1 --2- 172.21.15.123:0/323530171 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbac40fdff0 0x7fbac410de30 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbab000b0d0 tx=0x7fbab0009e30 comp rx=0 tx=0).stop 2024-03-30T04:20:58.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:58.751+0000 7fbacb000700 1 -- 172.21.15.123:0/323530171 shutdown_connections 2024-03-30T04:20:58.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:58.751+0000 7fbacb000700 1 --2- 172.21.15.123:0/323530171 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fbaa8069210 0x7fbaa806b6c0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:20:58.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:58.751+0000 7fbacb000700 1 --2- 172.21.15.123:0/323530171 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbac40fce80 0x7fbac410d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:20:58.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:58.751+0000 7fbacb000700 1 --2- 172.21.15.123:0/323530171 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbac40fdff0 0x7fbac410de30 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:20:58.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:58.751+0000 7fbacb000700 1 -- 172.21.15.123:0/323530171 >> 172.21.15.123:0/323530171 conn(0x7fbac40f8540 msgr2=0x7fbac40f9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:20:58.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:58.751+0000 7fbacb000700 1 -- 172.21.15.123:0/323530171 shutdown_connections 2024-03-30T04:20:58.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:20:58.751+0000 7fbacb000700 1 -- 172.21.15.123:0/323530171 wait complete. 2024-03-30T04:20:59.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:59 smithi135 ceph-mon[44364]: from='osd.1 [v2:172.21.15.123:6810/1040012134,v1:172.21.15.123:6811/1040012134]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-30T04:20:59.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:59 smithi135 ceph-mon[44364]: osdmap e12: 2 total, 1 up, 2 in 2024-03-30T04:20:59.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:59 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T04:20:59.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:59 smithi135 ceph-mon[44364]: from='osd.1 [v2:172.21.15.123:6810/1040012134,v1:172.21.15.123:6811/1040012134]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-03-30T04:20:59.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:59 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:59.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:59 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:59.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:59 smithi135 ceph-mon[44364]: from='osd.1 [v2:172.21.15.123:6810/1040012134,v1:172.21.15.123:6811/1040012134]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-03-30T04:20:59.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:59 smithi135 ceph-mon[44364]: osdmap e13: 2 total, 1 up, 2 in 2024-03-30T04:20:59.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:20:59 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T04:20:59.393 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:59 smithi123 ceph-mon[35245]: from='osd.1 [v2:172.21.15.123:6810/1040012134,v1:172.21.15.123:6811/1040012134]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-30T04:20:59.393 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:59 smithi123 ceph-mon[35245]: osdmap e12: 2 total, 1 up, 2 in 2024-03-30T04:20:59.394 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:59 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T04:20:59.394 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:59 smithi123 ceph-mon[35245]: from='osd.1 [v2:172.21.15.123:6810/1040012134,v1:172.21.15.123:6811/1040012134]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-03-30T04:20:59.394 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:59 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:59.394 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:59 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:20:59.394 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:59 smithi123 ceph-mon[35245]: from='osd.1 [v2:172.21.15.123:6810/1040012134,v1:172.21.15.123:6811/1040012134]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-03-30T04:20:59.394 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:59 smithi123 ceph-mon[35245]: osdmap e13: 2 total, 1 up, 2 in 2024-03-30T04:20:59.394 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:20:59 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T04:20:59.956 DEBUG:teuthology.orchestra.run.smithi123:osd.1> sudo journalctl -f -n 0 -u ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4@osd.1.service 2024-03-30T04:20:59.961 INFO:tasks.cephadm:Deploying osd.2 on smithi123 with /dev/vg_nvme/lv_2... 2024-03-30T04:20:59.961 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-30T04:21:00.036 INFO:journalctl@ceph.osd.1.smithi123.stdout:-- Logs begin at Sat 2024-03-30 04:04:15 UTC. -- 2024-03-30T04:21:00.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:00 smithi123 ceph-mon[35245]: pgmap v61: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-30T04:21:00.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:00 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T04:21:00.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:00 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T04:21:00.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:00 smithi135 ceph-mon[44364]: pgmap v61: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-30T04:21:00.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:00 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T04:21:00.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:00 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T04:21:00.758 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:21:01.283 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:01 smithi123 ceph-mon[35245]: purged_snaps scrub starts 2024-03-30T04:21:01.283 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:01 smithi123 ceph-mon[35245]: purged_snaps scrub ok 2024-03-30T04:21:01.283 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:01 smithi123 ceph-mon[35245]: osd.1 [v2:172.21.15.123:6810/1040012134,v1:172.21.15.123:6811/1040012134] boot 2024-03-30T04:21:01.283 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:01 smithi123 ceph-mon[35245]: osdmap e14: 2 total, 2 up, 2 in 2024-03-30T04:21:01.283 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:01 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T04:21:01.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:01 smithi135 ceph-mon[44364]: purged_snaps scrub starts 2024-03-30T04:21:01.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:01 smithi135 ceph-mon[44364]: purged_snaps scrub ok 2024-03-30T04:21:01.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:01 smithi135 ceph-mon[44364]: osd.1 [v2:172.21.15.123:6810/1040012134,v1:172.21.15.123:6811/1040012134] boot 2024-03-30T04:21:01.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:01 smithi135 ceph-mon[44364]: osdmap e14: 2 total, 2 up, 2 in 2024-03-30T04:21:01.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:01 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T04:21:02.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:02 smithi135 ceph-mon[44364]: pgmap v64: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-30T04:21:02.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:02 smithi135 ceph-mon[44364]: osdmap e15: 2 total, 2 up, 2 in 2024-03-30T04:21:02.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:02 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:21:02.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:02 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:21:02.384 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:02 smithi123 ceph-mon[35245]: pgmap v64: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-30T04:21:02.384 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:02 smithi123 ceph-mon[35245]: osdmap e15: 2 total, 2 up, 2 in 2024-03-30T04:21:02.384 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:02 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:21:02.384 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:02 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:21:03.467 INFO:teuthology.orchestra.run.smithi123.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-30T04:21:03.468 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-30T04:21:03.468 INFO:teuthology.orchestra.run.smithi123.stderr: stderr: 10+0 records in 2024-03-30T04:21:03.468 INFO:teuthology.orchestra.run.smithi123.stderr:10+0 records out 2024-03-30T04:21:03.468 INFO:teuthology.orchestra.run.smithi123.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.031209 s, 336 MB/s 2024-03-30T04:21:03.468 INFO:teuthology.orchestra.run.smithi123.stderr:--> Zapping successful for: 2024-03-30T04:21:04.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:04 smithi135 ceph-mon[44364]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-30T04:21:04.412 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:04 smithi123 ceph-mon[35245]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-30T04:21:04.794 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi123:vg_nvme/lv_2 2024-03-30T04:21:05.294 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:21:06.344 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:06 smithi123 ceph-mon[35245]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-30T04:21:06.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:06 smithi135 ceph-mon[44364]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-30T04:21:07.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:21:07.735+0000 7fc468c00700 1 -- 172.21.15.123:0/2773516979 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4640708f0 msgr2=0x7fc464070d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:21:07.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:21:07.735+0000 7fc468c00700 1 --2- 172.21.15.123:0/2773516979 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4640708f0 0x7fc464070d60 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc450004940 tx=0x7fc450009e30 comp rx=0 tx=0).stop 2024-03-30T04:21:07.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:21:07.736+0000 7fc468c00700 1 -- 172.21.15.123:0/2773516979 shutdown_connections 2024-03-30T04:21:07.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:21:07.736+0000 7fc468c00700 1 --2- 172.21.15.123:0/2773516979 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4640708f0 0x7fc464070d60 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:21:07.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:21:07.736+0000 7fc468c00700 1 --2- 172.21.15.123:0/2773516979 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc464071ea0 0x7fc4640703b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:21:07.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:21:07.736+0000 7fc468c00700 1 -- 172.21.15.123:0/2773516979 >> 172.21.15.123:0/2773516979 conn(0x7fc4640f8560 msgr2=0x7fc4640fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:21:07.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:21:07.736+0000 7fc468c00700 1 -- 172.21.15.123:0/2773516979 shutdown_connections 2024-03-30T04:21:07.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:21:07.736+0000 7fc468c00700 1 -- 172.21.15.123:0/2773516979 wait complete. 2024-03-30T04:21:07.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:21:07.736+0000 7fc468c00700 1 Processor -- start 2024-03-30T04:21:07.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:21:07.737+0000 7fc468c00700 1 -- start start 2024-03-30T04:21:07.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:21:07.737+0000 7fc468c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4640708f0 0x7fc464113f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:21:07.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:21:07.737+0000 7fc468c00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc464071ea0 0x7fc4641144d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:21:07.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:21:07.737+0000 7fc468c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc464114ad0 con 0x7fc4640708f0 2024-03-30T04:21:07.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:21:07.737+0000 7fc468c00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc464115260 con 0x7fc464071ea0 2024-03-30T04:21:07.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:21:07.737+0000 7fc461e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4640708f0 0x7fc464113f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:21:07.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:21:07.737+0000 7fc461e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4640708f0 0x7fc464113f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:51520/0 (socket says 172.21.15.123:51520) 2024-03-30T04:21:07.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:21:07.737+0000 7fc461e00700 1 -- 172.21.15.123:0/518516274 learned_addr learned my addr 172.21.15.123:0/518516274 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:21:07.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:21:07.737+0000 7fc461400700 1 --2- 172.21.15.123:0/518516274 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc464071ea0 0x7fc4641144d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:21:07.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:21:07.737+0000 7fc461e00700 1 -- 172.21.15.123:0/518516274 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc464071ea0 msgr2=0x7fc4641144d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:21:07.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:21:07.737+0000 7fc461e00700 1 --2- 172.21.15.123:0/518516274 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc464071ea0 0x7fc4641144d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:21:07.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:21:07.737+0000 7fc461e00700 1 -- 172.21.15.123:0/518516274 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc450005040 con 0x7fc4640708f0 2024-03-30T04:21:07.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:21:07.737+0000 7fc461400700 1 --2- 172.21.15.123:0/518516274 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc464071ea0 0x7fc4641144d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:21:07.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:21:07.738+0000 7fc461e00700 1 --2- 172.21.15.123:0/518516274 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4640708f0 0x7fc464113f90 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc45400eb40 tx=0x7fc45400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:21:07.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:21:07.738+0000 7fc45a000700 1 -- 172.21.15.123:0/518516274 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc454007cc0 con 0x7fc4640708f0 2024-03-30T04:21:07.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:21:07.738+0000 7fc45a000700 1 -- 172.21.15.123:0/518516274 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc45400f040 con 0x7fc4640708f0 2024-03-30T04:21:07.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:21:07.738+0000 7fc468c00700 1 -- 172.21.15.123:0/518516274 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4641154e0 con 0x7fc4640708f0 2024-03-30T04:21:07.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:21:07.738+0000 7fc45a000700 1 -- 172.21.15.123:0/518516274 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc45400cad0 con 0x7fc4640708f0 2024-03-30T04:21:07.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:21:07.738+0000 7fc468c00700 1 -- 172.21.15.123:0/518516274 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc464115a20 con 0x7fc4640708f0 2024-03-30T04:21:07.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:21:07.739+0000 7fc468c00700 1 -- 172.21.15.123:0/518516274 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc46405f5f0 con 0x7fc4640708f0 2024-03-30T04:21:07.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:21:07.740+0000 7fc45a000700 1 -- 172.21.15.123:0/518516274 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 18) v1 ==== 86567+0+0 (secure 0 0 0) 0x7fc45400cc30 con 0x7fc4640708f0 2024-03-30T04:21:07.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:21:07.741+0000 7fc45a000700 1 --2- 172.21.15.123:0/518516274 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fc448069210 0x7fc44806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:21:07.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:21:07.741+0000 7fc45a000700 1 -- 172.21.15.123:0/518516274 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2410+0+0 (secure 0 0 0) 0x7fc454085fc0 con 0x7fc4640708f0 2024-03-30T04:21:07.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:21:07.741+0000 7fc461400700 1 --2- 172.21.15.123:0/518516274 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fc448069210 0x7fc44806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:21:07.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:21:07.742+0000 7fc461400700 1 --2- 172.21.15.123:0/518516274 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fc448069210 0x7fc44806b6c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc45000b0d0 tx=0x7fc450009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:21:07.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:21:07.750+0000 7fc45a000700 1 -- 172.21.15.123:0/518516274 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fc454056a50 con 0x7fc4640708f0 2024-03-30T04:21:07.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:21:07.968+0000 7fc468c00700 1 -- 172.21.15.123:0/518516274 --> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fc464115e00 con 0x7fc448069210 2024-03-30T04:21:08.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:08 smithi123 ceph-mon[35245]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-30T04:21:08.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:08 smithi123 ceph-mon[35245]: Detected new or changed devices on smithi123 2024-03-30T04:21:08.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:08 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:21:08.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:08 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:21:08.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:08 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:21:08.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:08 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:21:08.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:08 smithi123 ceph-mon[35245]: Adjusting osd_memory_target on smithi123 to 6523M 2024-03-30T04:21:08.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:08 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:21:08.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:08 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:21:08.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:08 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:21:08.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:08 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:21:08.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:08 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:21:08.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:08 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T04:21:08.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:08 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T04:21:08.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:08 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:21:08.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:08 smithi135 ceph-mon[44364]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-30T04:21:08.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:08 smithi135 ceph-mon[44364]: Detected new or changed devices on smithi123 2024-03-30T04:21:08.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:08 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:21:08.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:08 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:21:08.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:08 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:21:08.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:08 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:21:08.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:08 smithi135 ceph-mon[44364]: Adjusting osd_memory_target on smithi123 to 6523M 2024-03-30T04:21:08.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:08 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:21:08.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:08 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:21:08.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:08 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:21:08.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:08 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:21:08.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:08 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:21:08.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:08 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T04:21:08.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:08 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T04:21:08.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:08 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:21:09.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:09 smithi123 ceph-mon[35245]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:21:09.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:09 smithi135 ceph-mon[44364]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:21:10.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:10 smithi123 ceph-mon[35245]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-30T04:21:10.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:10 smithi135 ceph-mon[44364]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-30T04:21:12.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:12 smithi135 ceph-mon[44364]: pgmap v70: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-30T04:21:12.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:12 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:21:12.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:12 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/1999991148' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4386fc8a-e9ed-413e-bd3a-bb84a0e62827"}]: dispatch 2024-03-30T04:21:12.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:12 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/1999991148' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4386fc8a-e9ed-413e-bd3a-bb84a0e62827"}]': finished 2024-03-30T04:21:12.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:12 smithi135 ceph-mon[44364]: osdmap e16: 3 total, 2 up, 3 in 2024-03-30T04:21:12.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:12 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T04:21:12.686 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:12 smithi123 ceph-mon[35245]: pgmap v70: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-30T04:21:12.686 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:12 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:21:12.686 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:12 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/1999991148' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4386fc8a-e9ed-413e-bd3a-bb84a0e62827"}]: dispatch 2024-03-30T04:21:12.686 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:12 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/1999991148' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4386fc8a-e9ed-413e-bd3a-bb84a0e62827"}]': finished 2024-03-30T04:21:12.686 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:12 smithi123 ceph-mon[35245]: osdmap e16: 3 total, 2 up, 3 in 2024-03-30T04:21:12.686 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:12 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T04:21:13.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:13 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/1861725266' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T04:21:13.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:13 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/1861725266' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T04:21:14.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:14 smithi135 ceph-mon[44364]: pgmap v72: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-30T04:21:14.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:14 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:21:14.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:14 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:21:14.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:14 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:21:14.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:14 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:21:14.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:14 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:21:14.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:14 smithi123 ceph-mon[35245]: pgmap v72: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-30T04:21:14.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:14 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:21:14.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:14 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:21:14.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:14 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:21:14.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:14 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:21:14.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:14 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:21:16.421 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:16 smithi123 ceph-mon[35245]: pgmap v73: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-30T04:21:16.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:16 smithi135 ceph-mon[44364]: pgmap v73: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-30T04:21:18.367 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:18 smithi123 ceph-mon[35245]: pgmap v74: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-30T04:21:18.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:18 smithi135 ceph-mon[44364]: pgmap v74: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-30T04:21:20.359 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:20 smithi123 ceph-mon[35245]: pgmap v75: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-30T04:21:20.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:20 smithi135 ceph-mon[44364]: pgmap v75: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-30T04:21:21.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:21 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-30T04:21:21.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:21 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:21:21.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:21 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-30T04:21:21.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:21 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:21:22.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:22 smithi135 ceph-mon[44364]: pgmap v76: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-30T04:21:22.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:22 smithi135 ceph-mon[44364]: Deploying daemon osd.2 on smithi123 2024-03-30T04:21:22.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:22 smithi123 ceph-mon[35245]: pgmap v76: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-30T04:21:22.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:22 smithi123 ceph-mon[35245]: Deploying daemon osd.2 on smithi123 2024-03-30T04:21:24.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:24 smithi123 ceph-mon[35245]: pgmap v77: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-30T04:21:24.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:24 smithi135 ceph-mon[44364]: pgmap v77: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-30T04:21:26.344 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:26 smithi123 ceph-mon[35245]: pgmap v78: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-30T04:21:26.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:26 smithi135 ceph-mon[44364]: pgmap v78: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-30T04:21:27.414 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:27 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:21:27.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:27 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:21:28.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:28 smithi123 ceph-mon[35245]: pgmap v79: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-30T04:21:28.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:28 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:21:28.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:28 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:21:28.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:28 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:21:28.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:28 smithi135 ceph-mon[44364]: pgmap v79: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-30T04:21:28.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:28 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:21:28.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:28 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:21:28.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:28 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:21:30.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:30 smithi123 ceph-mon[35245]: pgmap v80: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-30T04:21:30.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:30 smithi135 ceph-mon[44364]: pgmap v80: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-30T04:21:32.137 INFO:teuthology.orchestra.run.smithi123.stdout:Created osd(s) 2 on host 'smithi123' 2024-03-30T04:21:32.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:21:32.137+0000 7fc45a000700 1 -- 172.21.15.123:0/518516274 <== mgr.14206 v2:172.21.15.123:6800/1564935593 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fc464115e00 con 0x7fc448069210 2024-03-30T04:21:32.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:21:32.140+0000 7fc468c00700 1 -- 172.21.15.123:0/518516274 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fc448069210 msgr2=0x7fc44806b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:21:32.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:21:32.140+0000 7fc468c00700 1 --2- 172.21.15.123:0/518516274 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fc448069210 0x7fc44806b6c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc45000b0d0 tx=0x7fc450009e30 comp rx=0 tx=0).stop 2024-03-30T04:21:32.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:21:32.140+0000 7fc468c00700 1 -- 172.21.15.123:0/518516274 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4640708f0 msgr2=0x7fc464113f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:21:32.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:21:32.140+0000 7fc468c00700 1 --2- 172.21.15.123:0/518516274 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4640708f0 0x7fc464113f90 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc45400eb40 tx=0x7fc45400b040 comp rx=0 tx=0).stop 2024-03-30T04:21:32.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:21:32.140+0000 7fc468c00700 1 -- 172.21.15.123:0/518516274 shutdown_connections 2024-03-30T04:21:32.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:21:32.140+0000 7fc468c00700 1 --2- 172.21.15.123:0/518516274 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fc448069210 0x7fc44806b6c0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:21:32.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:21:32.140+0000 7fc468c00700 1 --2- 172.21.15.123:0/518516274 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4640708f0 0x7fc464113f90 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:21:32.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:21:32.140+0000 7fc468c00700 1 --2- 172.21.15.123:0/518516274 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc464071ea0 0x7fc4641144d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:21:32.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:21:32.140+0000 7fc468c00700 1 -- 172.21.15.123:0/518516274 >> 172.21.15.123:0/518516274 conn(0x7fc4640f8560 msgr2=0x7fc4640f9d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:21:32.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:21:32.141+0000 7fc468c00700 1 -- 172.21.15.123:0/518516274 shutdown_connections 2024-03-30T04:21:32.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:21:32.141+0000 7fc468c00700 1 -- 172.21.15.123:0/518516274 wait complete. 2024-03-30T04:21:32.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:32 smithi135 ceph-mon[44364]: pgmap v81: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-30T04:21:32.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:32 smithi135 ceph-mon[44364]: from='osd.2 [v2:172.21.15.123:6818/1260172805,v1:172.21.15.123:6819/1260172805]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-30T04:21:32.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:32 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:21:32.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:32 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:21:32.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:32 smithi123 ceph-mon[35245]: pgmap v81: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-30T04:21:32.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:32 smithi123 ceph-mon[35245]: from='osd.2 [v2:172.21.15.123:6818/1260172805,v1:172.21.15.123:6819/1260172805]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-30T04:21:32.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:32 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:21:32.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:32 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:21:33.433 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:33 smithi123 ceph-mon[35245]: from='osd.2 [v2:172.21.15.123:6818/1260172805,v1:172.21.15.123:6819/1260172805]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-30T04:21:33.433 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:33 smithi123 ceph-mon[35245]: osdmap e17: 3 total, 2 up, 3 in 2024-03-30T04:21:33.433 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:33 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T04:21:33.434 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:33 smithi123 ceph-mon[35245]: from='osd.2 [v2:172.21.15.123:6818/1260172805,v1:172.21.15.123:6819/1260172805]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-03-30T04:21:33.434 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:33 smithi123 ceph-mon[35245]: from='osd.2 [v2:172.21.15.123:6818/1260172805,v1:172.21.15.123:6819/1260172805]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-03-30T04:21:33.434 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:33 smithi123 ceph-mon[35245]: osdmap e18: 3 total, 2 up, 3 in 2024-03-30T04:21:33.434 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:33 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T04:21:33.436 DEBUG:teuthology.orchestra.run.smithi123:osd.2> sudo journalctl -f -n 0 -u ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4@osd.2.service 2024-03-30T04:21:33.440 INFO:tasks.cephadm:Deploying osd.3 on smithi135 with /dev/vg_nvme/lv_4... 2024-03-30T04:21:33.440 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-30T04:21:33.508 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:33 smithi135 ceph-mon[44364]: from='osd.2 [v2:172.21.15.123:6818/1260172805,v1:172.21.15.123:6819/1260172805]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-30T04:21:33.508 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:33 smithi135 ceph-mon[44364]: osdmap e17: 3 total, 2 up, 3 in 2024-03-30T04:21:33.509 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:33 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T04:21:33.509 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:33 smithi135 ceph-mon[44364]: from='osd.2 [v2:172.21.15.123:6818/1260172805,v1:172.21.15.123:6819/1260172805]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-03-30T04:21:33.509 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:33 smithi135 ceph-mon[44364]: from='osd.2 [v2:172.21.15.123:6818/1260172805,v1:172.21.15.123:6819/1260172805]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-03-30T04:21:33.509 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:33 smithi135 ceph-mon[44364]: osdmap e18: 3 total, 2 up, 3 in 2024-03-30T04:21:33.509 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:33 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T04:21:33.514 INFO:journalctl@ceph.osd.2.smithi123.stdout:-- Logs begin at Sat 2024-03-30 04:04:15 UTC. -- 2024-03-30T04:21:33.858 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi135/config 2024-03-30T04:21:34.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:34 smithi123 ceph-mon[35245]: pgmap v83: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-30T04:21:34.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:34 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T04:21:34.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:34 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T04:21:34.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:34 smithi135 ceph-mon[44364]: pgmap v83: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-30T04:21:34.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:34 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T04:21:34.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:34 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T04:21:35.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:35 smithi135 ceph-mon[44364]: purged_snaps scrub starts 2024-03-30T04:21:35.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:35 smithi135 ceph-mon[44364]: purged_snaps scrub ok 2024-03-30T04:21:35.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:35 smithi135 ceph-mon[44364]: osd.2 [v2:172.21.15.123:6818/1260172805,v1:172.21.15.123:6819/1260172805] boot 2024-03-30T04:21:35.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:35 smithi135 ceph-mon[44364]: osdmap e19: 3 total, 3 up, 3 in 2024-03-30T04:21:35.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:35 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T04:21:35.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:35 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-30T04:21:35.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:35 smithi123 ceph-mon[35245]: purged_snaps scrub starts 2024-03-30T04:21:35.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:35 smithi123 ceph-mon[35245]: purged_snaps scrub ok 2024-03-30T04:21:35.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:35 smithi123 ceph-mon[35245]: osd.2 [v2:172.21.15.123:6818/1260172805,v1:172.21.15.123:6819/1260172805] boot 2024-03-30T04:21:35.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:35 smithi123 ceph-mon[35245]: osdmap e19: 3 total, 3 up, 3 in 2024-03-30T04:21:35.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:35 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T04:21:35.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:35 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-30T04:21:36.260 INFO:teuthology.orchestra.run.smithi135.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-30T04:21:36.261 INFO:teuthology.orchestra.run.smithi135.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-30T04:21:36.261 INFO:teuthology.orchestra.run.smithi135.stderr: stderr: 10+0 records in 2024-03-30T04:21:36.261 INFO:teuthology.orchestra.run.smithi135.stderr:10+0 records out 2024-03-30T04:21:36.261 INFO:teuthology.orchestra.run.smithi135.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0395472 s, 265 MB/s 2024-03-30T04:21:36.261 INFO:teuthology.orchestra.run.smithi135.stderr:--> Zapping successful for: 2024-03-30T04:21:36.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:36 smithi123 ceph-mon[35245]: pgmap v86: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-30T04:21:36.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:36 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-30T04:21:36.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:36 smithi123 ceph-mon[35245]: osdmap e20: 3 total, 3 up, 3 in 2024-03-30T04:21:36.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:36 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T04:21:36.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:36 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:21:36.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:36 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:21:36.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:36 smithi135 ceph-mon[44364]: pgmap v86: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-30T04:21:36.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:36 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-30T04:21:36.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:36 smithi135 ceph-mon[44364]: osdmap e20: 3 total, 3 up, 3 in 2024-03-30T04:21:36.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:36 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T04:21:36.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:36 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:21:36.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:36 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:21:36.997 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi135:vg_nvme/lv_4 2024-03-30T04:21:37.186 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 30 04:21:36 smithi123 sudo[79119]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-30T04:21:37.186 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 30 04:21:36 smithi123 sudo[79119]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T04:21:37.187 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 30 04:21:36 smithi123 sudo[79119]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T04:21:37.187 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 30 04:21:36 smithi123 sudo[79119]: pam_unix(sudo:session): session closed for user root 2024-03-30T04:21:37.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:37 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-30T04:21:37.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:37 smithi135 ceph-mon[44364]: osdmap e21: 3 total, 3 up, 3 in 2024-03-30T04:21:37.409 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi135/config 2024-03-30T04:21:37.457 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:37 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-30T04:21:37.457 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:37 smithi123 ceph-mon[35245]: osdmap e21: 3 total, 3 up, 3 in 2024-03-30T04:21:37.716 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 30 04:21:37 smithi123 sudo[79160]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-30T04:21:37.716 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 30 04:21:37 smithi123 sudo[79160]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T04:21:37.716 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 30 04:21:37 smithi123 sudo[79160]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T04:21:37.717 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 30 04:21:37 smithi123 sudo[79160]: pam_unix(sudo:session): session closed for user root 2024-03-30T04:21:38.427 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:38 smithi123 ceph-mon[35245]: pgmap v89: 1 pgs: 1 creating+peering; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-03-30T04:21:38.427 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:38 smithi123 ceph-mon[35245]: osdmap e22: 3 total, 3 up, 3 in 2024-03-30T04:21:38.428 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:38 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:21:38.428 INFO:journalctl@ceph.osd.1.smithi123.stdout:Mar 30 04:21:38 smithi123 sudo[79171]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-30T04:21:38.428 INFO:journalctl@ceph.osd.1.smithi123.stdout:Mar 30 04:21:38 smithi123 sudo[79171]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T04:21:38.428 INFO:journalctl@ceph.osd.1.smithi123.stdout:Mar 30 04:21:38 smithi123 sudo[79171]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T04:21:38.428 INFO:journalctl@ceph.osd.1.smithi123.stdout:Mar 30 04:21:38 smithi123 sudo[79171]: pam_unix(sudo:session): session closed for user root 2024-03-30T04:21:38.528 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:38 smithi135 ceph-mon[44364]: pgmap v89: 1 pgs: 1 creating+peering; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-03-30T04:21:38.529 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:38 smithi135 ceph-mon[44364]: osdmap e22: 3 total, 3 up, 3 in 2024-03-30T04:21:38.529 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:38 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:21:38.716 INFO:journalctl@ceph.osd.1.smithi123.stdout:Mar 30 04:21:38 smithi123 sudo[79225]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-30T04:21:38.717 INFO:journalctl@ceph.osd.1.smithi123.stdout:Mar 30 04:21:38 smithi123 sudo[79225]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T04:21:38.717 INFO:journalctl@ceph.osd.1.smithi123.stdout:Mar 30 04:21:38 smithi123 sudo[79225]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T04:21:38.717 INFO:journalctl@ceph.osd.1.smithi123.stdout:Mar 30 04:21:38 smithi123 sudo[79225]: pam_unix(sudo:session): session closed for user root 2024-03-30T04:21:39.184 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:21:39.186+0000 7f29f6400700 1 -- 172.21.15.135:0/77104374 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f29f0070890 msgr2=0x7f29f0070d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:21:39.184 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:21:39.186+0000 7f29f6400700 1 --2- 172.21.15.135:0/77104374 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f29f0070890 0x7f29f0070d00 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f29dc004970 tx=0x7f29dc009e30 comp rx=0 tx=0).stop 2024-03-30T04:21:39.184 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:21:39.186+0000 7f29f6400700 1 -- 172.21.15.135:0/77104374 shutdown_connections 2024-03-30T04:21:39.184 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:21:39.186+0000 7f29f6400700 1 --2- 172.21.15.135:0/77104374 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f29f0070890 0x7f29f0070d00 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:21:39.185 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:21:39.186+0000 7f29f6400700 1 --2- 172.21.15.135:0/77104374 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29f0071e40 0x7f29f0070350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:21:39.185 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:21:39.186+0000 7f29f6400700 1 -- 172.21.15.135:0/77104374 >> 172.21.15.135:0/77104374 conn(0x7f29f00f8560 msgr2=0x7f29f00fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:21:39.185 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:21:39.186+0000 7f29f6400700 1 -- 172.21.15.135:0/77104374 shutdown_connections 2024-03-30T04:21:39.185 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:21:39.186+0000 7f29f6400700 1 -- 172.21.15.135:0/77104374 wait complete. 2024-03-30T04:21:39.186 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:21:39.187+0000 7f29f6400700 1 Processor -- start 2024-03-30T04:21:39.186 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:21:39.187+0000 7f29f6400700 1 -- start start 2024-03-30T04:21:39.187 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:21:39.187+0000 7f29f6400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f29f0070890 0x7f29f010fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:21:39.187 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:21:39.187+0000 7f29f6400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29f0071e40 0x7f29f010fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:21:39.187 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:21:39.187+0000 7f29f6400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29f01105f0 con 0x7f29f0071e40 2024-03-30T04:21:39.187 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:21:39.187+0000 7f29f6400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29f0110e40 con 0x7f29f0070890 2024-03-30T04:21:39.187 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:21:39.188+0000 7f29ef400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f29f0070890 0x7f29f010fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:21:39.187 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:21:39.188+0000 7f29ef400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f29f0070890 0x7f29f010fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.135:46462/0 (socket says 172.21.15.135:46462) 2024-03-30T04:21:39.187 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:21:39.188+0000 7f29ef400700 1 -- 172.21.15.135:0/3394934850 learned_addr learned my addr 172.21.15.135:0/3394934850 (peer_addr_for_me v2:172.21.15.135:0/0) 2024-03-30T04:21:39.187 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:21:39.188+0000 7f29eea00700 1 --2- 172.21.15.135:0/3394934850 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29f0071e40 0x7f29f010fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:21:39.187 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:21:39.188+0000 7f29ef400700 1 -- 172.21.15.135:0/3394934850 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29f0071e40 msgr2=0x7f29f010fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:21:39.187 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:21:39.188+0000 7f29ef400700 1 --2- 172.21.15.135:0/3394934850 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29f0071e40 0x7f29f010fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:21:39.187 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:21:39.188+0000 7f29ef400700 1 -- 172.21.15.135:0/3394934850 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29dc005040 con 0x7f29f0070890 2024-03-30T04:21:39.187 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:21:39.188+0000 7f29ef400700 1 --2- 172.21.15.135:0/3394934850 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f29f0070890 0x7f29f010fab0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f29e000eb70 tx=0x7f29e000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:21:39.187 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:21:39.188+0000 7f29e7400700 1 -- 172.21.15.135:0/3394934850 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f29e0007cc0 con 0x7f29f0070890 2024-03-30T04:21:39.187 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:21:39.188+0000 7f29e7400700 1 -- 172.21.15.135:0/3394934850 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f29e000f040 con 0x7f29f0070890 2024-03-30T04:21:39.188 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:21:39.188+0000 7f29e7400700 1 -- 172.21.15.135:0/3394934850 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f29e000c9d0 con 0x7f29f0070890 2024-03-30T04:21:39.188 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:21:39.188+0000 7f29f6400700 1 -- 172.21.15.135:0/3394934850 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29f01110c0 con 0x7f29f0070890 2024-03-30T04:21:39.188 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:21:39.189+0000 7f29f6400700 1 -- 172.21.15.135:0/3394934850 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29f0111600 con 0x7f29f0070890 2024-03-30T04:21:39.188 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:21:39.190+0000 7f29f6400700 1 -- 172.21.15.135:0/3394934850 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29f005f5f0 con 0x7f29f0070890 2024-03-30T04:21:39.188 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:21:39.191+0000 7f29e7400700 1 -- 172.21.15.135:0/3394934850 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 18) v1 ==== 86567+0+0 (secure 0 0 0) 0x7f29e000cb30 con 0x7f29f0070890 2024-03-30T04:21:39.190 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:21:39.192+0000 7f29e7400700 1 --2- 172.21.15.135:0/3394934850 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f29d4069210 0x7f29d406b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:21:39.190 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:21:39.192+0000 7f29e7400700 1 -- 172.21.15.135:0/3394934850 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 3228+0+0 (secure 0 0 0) 0x7f29e0086260 con 0x7f29f0070890 2024-03-30T04:21:39.190 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:21:39.192+0000 7f29eea00700 1 --2- 172.21.15.135:0/3394934850 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f29d4069210 0x7f29d406b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:21:39.191 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:21:39.193+0000 7f29eea00700 1 --2- 172.21.15.135:0/3394934850 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f29d4069210 0x7f29d406b6c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f29dc009fd0 tx=0x7f29dc009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:21:39.195 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:21:39.198+0000 7f29e7400700 1 -- 172.21.15.135:0/3394934850 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f29e004e190 con 0x7f29f0070890 2024-03-30T04:21:39.202 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:21:39.204+0000 7f29e7400700 1 -- 172.21.15.135:0/3394934850 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f29e0053310 con 0x7f29f0070890 2024-03-30T04:21:39.396 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:21:39.398+0000 7f29f6400700 1 -- 172.21.15.135:0/3394934850 --> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi135:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f29f0111a30 con 0x7f29d4069210 2024-03-30T04:21:39.467 INFO:journalctl@ceph.osd.2.smithi123.stdout:Mar 30 04:21:39 smithi123 sudo[79245]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-30T04:21:39.467 INFO:journalctl@ceph.osd.2.smithi123.stdout:Mar 30 04:21:39 smithi123 sudo[79245]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T04:21:39.467 INFO:journalctl@ceph.osd.2.smithi123.stdout:Mar 30 04:21:39 smithi123 sudo[79245]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T04:21:39.467 INFO:journalctl@ceph.osd.2.smithi123.stdout:Mar 30 04:21:39 smithi123 sudo[79245]: pam_unix(sudo:session): session closed for user root 2024-03-30T04:21:40.193 INFO:journalctl@ceph.osd.2.smithi123.stdout:Mar 30 04:21:39 smithi123 sudo[81622]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-30T04:21:40.193 INFO:journalctl@ceph.osd.2.smithi123.stdout:Mar 30 04:21:39 smithi123 sudo[81622]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T04:21:40.193 INFO:journalctl@ceph.osd.2.smithi123.stdout:Mar 30 04:21:39 smithi123 sudo[81622]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T04:21:40.194 INFO:journalctl@ceph.osd.2.smithi123.stdout:Mar 30 04:21:39 smithi123 sudo[81622]: pam_unix(sudo:session): session closed for user root 2024-03-30T04:21:40.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:40 smithi123 sudo[82493]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-30T04:21:40.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:40 smithi123 sudo[82493]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T04:21:40.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:40 smithi123 sudo[82493]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T04:21:40.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:40 smithi123 sudo[82493]: pam_unix(sudo:session): session closed for user root 2024-03-30T04:21:40.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:40 smithi123 sudo[82498]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-30T04:21:40.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:40 smithi123 sudo[82498]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T04:21:40.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:40 smithi123 sudo[82498]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T04:21:40.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:40 smithi123 sudo[82498]: pam_unix(sudo:session): session closed for user root 2024-03-30T04:21:40.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:40 smithi123 ceph-mon[35245]: pgmap v91: 1 pgs: 1 creating+peering; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-03-30T04:21:40.968 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:40 smithi123 ceph-mon[35245]: mgrmap e19: smithi123.xgksqq(active, since 2m), standbys: smithi135.kyblli 2024-03-30T04:21:40.968 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:40 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T04:21:40.968 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:40 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T04:21:40.968 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:40 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:21:40.968 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:40 smithi123 ceph-mon[35245]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T04:21:41.020 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:40 smithi135 ceph-mon[44364]: pgmap v91: 1 pgs: 1 creating+peering; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-03-30T04:21:41.020 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:40 smithi135 ceph-mon[44364]: mgrmap e19: smithi123.xgksqq(active, since 2m), standbys: smithi135.kyblli 2024-03-30T04:21:41.020 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:40 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T04:21:41.020 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:40 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T04:21:41.020 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:40 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:21:41.021 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:40 smithi135 ceph-mon[44364]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T04:21:41.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:41 smithi135 sudo[48245]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-30T04:21:41.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:41 smithi135 sudo[48245]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T04:21:41.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:41 smithi135 sudo[48245]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T04:21:41.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:41 smithi135 sudo[48245]: pam_unix(sudo:session): session closed for user root 2024-03-30T04:21:42.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:41 smithi123 ceph-mon[35245]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi135:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:21:42.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:41 smithi123 ceph-mon[35245]: pgmap v92: 1 pgs: 1 creating+peering; 0 B data, 873 MiB used, 267 GiB / 268 GiB avail 2024-03-30T04:21:42.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:41 smithi123 ceph-mon[35245]: Detected new or changed devices on smithi123 2024-03-30T04:21:42.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:41 smithi123 ceph-mon[35245]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T04:21:42.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:41 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mon metadata", "id": "smithi123"}]: dispatch 2024-03-30T04:21:42.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:41 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-03-30T04:21:42.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:41 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:21:42.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:41 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mon metadata", "id": "smithi123"}]: dispatch 2024-03-30T04:21:42.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:41 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-03-30T04:21:42.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:41 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:21:42.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:41 smithi135 sudo[48283]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-30T04:21:42.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:41 smithi135 sudo[48283]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T04:21:42.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:41 smithi135 sudo[48283]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T04:21:42.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:41 smithi135 sudo[48283]: pam_unix(sudo:session): session closed for user root 2024-03-30T04:21:42.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:41 smithi135 ceph-mon[44364]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi135:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:21:42.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:41 smithi135 ceph-mon[44364]: pgmap v92: 1 pgs: 1 creating+peering; 0 B data, 873 MiB used, 267 GiB / 268 GiB avail 2024-03-30T04:21:42.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:41 smithi135 ceph-mon[44364]: Detected new or changed devices on smithi123 2024-03-30T04:21:42.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:41 smithi135 ceph-mon[44364]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T04:21:42.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:41 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mon metadata", "id": "smithi123"}]: dispatch 2024-03-30T04:21:42.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:41 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-03-30T04:21:42.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:41 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:21:42.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:41 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mon metadata", "id": "smithi123"}]: dispatch 2024-03-30T04:21:42.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:41 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-03-30T04:21:42.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:41 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:21:43.202 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:42 smithi123 ceph-mon[35245]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T04:21:43.203 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:42 smithi123 ceph-mon[35245]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T04:21:43.203 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:42 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:21:43.203 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:42 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:21:43.203 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:42 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:21:43.203 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:42 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:21:43.203 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:42 smithi123 ceph-mon[35245]: Adjusting osd_memory_target on smithi123 to 4348M 2024-03-30T04:21:43.203 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:42 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:21:43.203 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:42 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:21:43.203 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:42 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:21:43.203 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:42 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:21:43.203 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:42 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:21:43.203 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:42 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:21:43.204 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:42 smithi123 ceph-mon[35245]: from='client.? 172.21.15.135:0/3023199943' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "086d86d4-ff1a-47dd-8416-447ccf9064ca"}]: dispatch 2024-03-30T04:21:43.204 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:42 smithi123 ceph-mon[35245]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "086d86d4-ff1a-47dd-8416-447ccf9064ca"}]: dispatch 2024-03-30T04:21:43.204 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:42 smithi123 ceph-mon[35245]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "086d86d4-ff1a-47dd-8416-447ccf9064ca"}]': finished 2024-03-30T04:21:43.204 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:42 smithi123 ceph-mon[35245]: osdmap e23: 4 total, 3 up, 4 in 2024-03-30T04:21:43.204 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:42 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T04:21:43.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:42 smithi135 ceph-mon[44364]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T04:21:43.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:42 smithi135 ceph-mon[44364]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T04:21:43.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:42 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:21:43.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:42 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:21:43.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:42 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:21:43.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:42 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:21:43.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:42 smithi135 ceph-mon[44364]: Adjusting osd_memory_target on smithi123 to 4348M 2024-03-30T04:21:43.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:42 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:21:43.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:42 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:21:43.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:42 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:21:43.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:42 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:21:43.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:42 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:21:43.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:42 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:21:43.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:42 smithi135 ceph-mon[44364]: from='client.? 172.21.15.135:0/3023199943' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "086d86d4-ff1a-47dd-8416-447ccf9064ca"}]: dispatch 2024-03-30T04:21:43.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:42 smithi135 ceph-mon[44364]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "086d86d4-ff1a-47dd-8416-447ccf9064ca"}]: dispatch 2024-03-30T04:21:43.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:42 smithi135 ceph-mon[44364]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "086d86d4-ff1a-47dd-8416-447ccf9064ca"}]': finished 2024-03-30T04:21:43.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:42 smithi135 ceph-mon[44364]: osdmap e23: 4 total, 3 up, 4 in 2024-03-30T04:21:43.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:42 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T04:21:44.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:43 smithi135 ceph-mon[44364]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-30T04:21:44.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:43 smithi135 ceph-mon[44364]: from='client.? 172.21.15.135:0/15136305' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T04:21:44.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:43 smithi123 ceph-mon[35245]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-30T04:21:44.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:43 smithi123 ceph-mon[35245]: from='client.? 172.21.15.135:0/15136305' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T04:21:46.343 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:45 smithi123 ceph-mon[35245]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-30T04:21:46.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:45 smithi135 ceph-mon[44364]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-30T04:21:48.273 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:47 smithi135 ceph-mon[44364]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-03-30T04:21:48.273 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:47 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:21:48.273 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:47 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:21:48.273 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:47 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:21:48.273 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:47 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:21:48.273 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:47 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:21:48.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:47 smithi123 ceph-mon[35245]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-03-30T04:21:48.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:47 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:21:48.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:47 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:21:48.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:47 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:21:48.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:47 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:21:48.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:47 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:21:50.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:49 smithi135 ceph-mon[44364]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-30T04:21:50.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:49 smithi123 ceph-mon[35245]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-30T04:21:51.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:51 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-30T04:21:51.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:51 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:21:51.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:51 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-30T04:21:51.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:51 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:21:52.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:52 smithi135 ceph-mon[44364]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-30T04:21:52.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:52 smithi135 ceph-mon[44364]: Deploying daemon osd.3 on smithi135 2024-03-30T04:21:52.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:52 smithi123 ceph-mon[35245]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-30T04:21:52.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:52 smithi123 ceph-mon[35245]: Deploying daemon osd.3 on smithi135 2024-03-30T04:21:54.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:54 smithi135 ceph-mon[44364]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-30T04:21:54.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:54 smithi123 ceph-mon[35245]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-30T04:21:56.318 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:56 smithi135 ceph-mon[44364]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 67 KiB/s wr, 4 op/s 2024-03-30T04:21:56.343 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:56 smithi123 ceph-mon[35245]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 67 KiB/s wr, 4 op/s 2024-03-30T04:21:57.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:57 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:21:57.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:57 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:21:58.272 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:58 smithi135 ceph-mon[44364]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-30T04:21:58.272 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:58 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:21:58.272 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:58 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:21:58.273 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:21:58 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:21:58.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:58 smithi123 ceph-mon[35245]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-30T04:21:58.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:58 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:21:58.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:58 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:21:58.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:21:58 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:22:00.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:00 smithi135 ceph-mon[44364]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-30T04:22:00.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:00 smithi123 ceph-mon[35245]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-30T04:22:01.780 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:01.781+0000 7f29e7400700 1 -- 172.21.15.135:0/3394934850 <== mgr.14206 v2:172.21.15.123:6800/1564935593 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f29f0111a30 con 0x7f29d4069210 2024-03-30T04:22:01.780 INFO:teuthology.orchestra.run.smithi135.stdout:Created osd(s) 3 on host 'smithi135' 2024-03-30T04:22:01.783 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:01.783+0000 7f29f6400700 1 -- 172.21.15.135:0/3394934850 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f29d4069210 msgr2=0x7f29d406b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:22:01.783 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:01.784+0000 7f29f6400700 1 --2- 172.21.15.135:0/3394934850 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f29d4069210 0x7f29d406b6c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f29dc009fd0 tx=0x7f29dc009e30 comp rx=0 tx=0).stop 2024-03-30T04:22:01.783 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:01.784+0000 7f29f6400700 1 -- 172.21.15.135:0/3394934850 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f29f0070890 msgr2=0x7f29f010fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:22:01.783 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:01.784+0000 7f29f6400700 1 --2- 172.21.15.135:0/3394934850 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f29f0070890 0x7f29f010fab0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f29e000eb70 tx=0x7f29e000b040 comp rx=0 tx=0).stop 2024-03-30T04:22:01.783 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:01.784+0000 7f29f6400700 1 -- 172.21.15.135:0/3394934850 shutdown_connections 2024-03-30T04:22:01.783 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:01.784+0000 7f29f6400700 1 --2- 172.21.15.135:0/3394934850 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f29d4069210 0x7f29d406b6c0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:22:01.783 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:01.784+0000 7f29f6400700 1 --2- 172.21.15.135:0/3394934850 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f29f0070890 0x7f29f010fab0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:22:01.783 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:01.784+0000 7f29f6400700 1 --2- 172.21.15.135:0/3394934850 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29f0071e40 0x7f29f010fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:22:01.783 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:01.784+0000 7f29f6400700 1 -- 172.21.15.135:0/3394934850 >> 172.21.15.135:0/3394934850 conn(0x7f29f00f8560 msgr2=0x7f29f00f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:22:01.783 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:01.784+0000 7f29f6400700 1 -- 172.21.15.135:0/3394934850 shutdown_connections 2024-03-30T04:22:01.783 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:01.784+0000 7f29f6400700 1 -- 172.21.15.135:0/3394934850 wait complete. 2024-03-30T04:22:02.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:02 smithi135 ceph-mon[44364]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-30T04:22:02.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:02 smithi135 ceph-mon[44364]: from='osd.3 [v2:172.21.15.135:6800/2906534885,v1:172.21.15.135:6801/2906534885]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-30T04:22:02.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:02 smithi135 ceph-mon[44364]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-30T04:22:02.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:02 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:22:02.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:02 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:22:02.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:02 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:22:02.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:02 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:22:02.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:02 smithi123 ceph-mon[35245]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-30T04:22:02.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:02 smithi123 ceph-mon[35245]: from='osd.3 [v2:172.21.15.135:6800/2906534885,v1:172.21.15.135:6801/2906534885]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-30T04:22:02.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:02 smithi123 ceph-mon[35245]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-30T04:22:02.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:02 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:22:02.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:02 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:22:02.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:02 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:22:02.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:02 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:22:02.515 DEBUG:teuthology.orchestra.run.smithi135:osd.3> sudo journalctl -f -n 0 -u ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4@osd.3.service 2024-03-30T04:22:02.519 INFO:tasks.cephadm:Deploying osd.4 on smithi135 with /dev/vg_nvme/lv_3... 2024-03-30T04:22:02.519 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-30T04:22:02.595 INFO:journalctl@ceph.osd.3.smithi135.stdout:-- Logs begin at Sat 2024-03-30 04:03:58 UTC. -- 2024-03-30T04:22:03.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:03 smithi135 ceph-mon[44364]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-30T04:22:03.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:03 smithi135 ceph-mon[44364]: osdmap e24: 4 total, 3 up, 4 in 2024-03-30T04:22:03.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:03 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T04:22:03.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:03 smithi135 ceph-mon[44364]: from='osd.3 [v2:172.21.15.135:6800/2906534885,v1:172.21.15.135:6801/2906534885]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-03-30T04:22:03.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:03 smithi135 ceph-mon[44364]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-03-30T04:22:03.352 INFO:journalctl@ceph.osd.3.smithi135.stdout:Mar 30 04:22:03 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-3[50690]: 2024-03-30T04:22:03.042+0000 7f260c000700 -1 osd.3 0 waiting for initial osdmap 2024-03-30T04:22:03.352 INFO:journalctl@ceph.osd.3.smithi135.stdout:Mar 30 04:22:03 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-3[50690]: 2024-03-30T04:22:03.051+0000 7f2607a00700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T04:22:03.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:03 smithi123 ceph-mon[35245]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-30T04:22:03.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:03 smithi123 ceph-mon[35245]: osdmap e24: 4 total, 3 up, 4 in 2024-03-30T04:22:03.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:03 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T04:22:03.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:03 smithi123 ceph-mon[35245]: from='osd.3 [v2:172.21.15.135:6800/2906534885,v1:172.21.15.135:6801/2906534885]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-03-30T04:22:03.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:03 smithi123 ceph-mon[35245]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-03-30T04:22:03.911 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi135/config 2024-03-30T04:22:04.233 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:04 smithi135 ceph-mon[44364]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-30T04:22:04.233 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:04 smithi135 ceph-mon[44364]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]': finished 2024-03-30T04:22:04.233 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:04 smithi135 ceph-mon[44364]: osdmap e25: 4 total, 3 up, 4 in 2024-03-30T04:22:04.233 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:04 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T04:22:04.233 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:04 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T04:22:04.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:04 smithi123 ceph-mon[35245]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-30T04:22:04.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:04 smithi123 ceph-mon[35245]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]': finished 2024-03-30T04:22:04.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:04 smithi123 ceph-mon[35245]: osdmap e25: 4 total, 3 up, 4 in 2024-03-30T04:22:04.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:04 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T04:22:04.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:04 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T04:22:05.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:05 smithi135 ceph-mon[44364]: purged_snaps scrub starts 2024-03-30T04:22:05.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:05 smithi135 ceph-mon[44364]: purged_snaps scrub ok 2024-03-30T04:22:05.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:05 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T04:22:05.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:05 smithi135 ceph-mon[44364]: osd.3 [v2:172.21.15.135:6800/2906534885,v1:172.21.15.135:6801/2906534885] boot 2024-03-30T04:22:05.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:05 smithi135 ceph-mon[44364]: osdmap e26: 4 total, 4 up, 4 in 2024-03-30T04:22:05.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:05 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T04:22:05.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:05 smithi123 ceph-mon[35245]: purged_snaps scrub starts 2024-03-30T04:22:05.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:05 smithi123 ceph-mon[35245]: purged_snaps scrub ok 2024-03-30T04:22:05.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:05 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T04:22:05.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:05 smithi123 ceph-mon[35245]: osd.3 [v2:172.21.15.135:6800/2906534885,v1:172.21.15.135:6801/2906534885] boot 2024-03-30T04:22:05.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:05 smithi123 ceph-mon[35245]: osdmap e26: 4 total, 4 up, 4 in 2024-03-30T04:22:05.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:05 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T04:22:06.342 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:06 smithi123 ceph-mon[35245]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-30T04:22:06.343 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:06 smithi123 ceph-mon[35245]: osdmap e27: 4 total, 4 up, 4 in 2024-03-30T04:22:06.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:06 smithi135 ceph-mon[44364]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-30T04:22:06.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:06 smithi135 ceph-mon[44364]: osdmap e27: 4 total, 4 up, 4 in 2024-03-30T04:22:07.166 INFO:teuthology.orchestra.run.smithi135.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-30T04:22:07.166 INFO:teuthology.orchestra.run.smithi135.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-30T04:22:07.166 INFO:teuthology.orchestra.run.smithi135.stderr: stderr: 10+0 records in 2024-03-30T04:22:07.166 INFO:teuthology.orchestra.run.smithi135.stderr:10+0 records out 2024-03-30T04:22:07.166 INFO:teuthology.orchestra.run.smithi135.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0466606 s, 225 MB/s 2024-03-30T04:22:07.166 INFO:teuthology.orchestra.run.smithi135.stderr:--> Zapping successful for: 2024-03-30T04:22:07.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:07 smithi135 ceph-mon[44364]: osdmap e28: 4 total, 4 up, 4 in 2024-03-30T04:22:07.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:07 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:22:07.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:07 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:22:07.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:07 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:22:07.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:07 smithi123 ceph-mon[35245]: osdmap e28: 4 total, 4 up, 4 in 2024-03-30T04:22:07.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:07 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:22:07.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:07 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:22:07.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:07 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:22:07.971 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi135:vg_nvme/lv_3 2024-03-30T04:22:08.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:08 smithi135 ceph-mon[44364]: pgmap v111: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-30T04:22:08.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:08 smithi135 ceph-mon[44364]: Detected new or changed devices on smithi135 2024-03-30T04:22:08.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:08 smithi135 ceph-mon[44364]: Adjusting osd_memory_target on smithi135 to 16118M 2024-03-30T04:22:08.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:08 smithi135 ceph-mon[44364]: osdmap e29: 4 total, 4 up, 4 in 2024-03-30T04:22:08.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:08 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:22:08.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:08 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:22:08.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:08 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:22:08.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:08 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:22:08.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:08 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:22:08.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:08 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:22:08.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:08 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:22:08.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:08 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:22:08.425 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi135/config 2024-03-30T04:22:08.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:08 smithi123 ceph-mon[35245]: pgmap v111: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-30T04:22:08.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:08 smithi123 ceph-mon[35245]: Detected new or changed devices on smithi135 2024-03-30T04:22:08.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:08 smithi123 ceph-mon[35245]: Adjusting osd_memory_target on smithi135 to 16118M 2024-03-30T04:22:08.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:08 smithi123 ceph-mon[35245]: osdmap e29: 4 total, 4 up, 4 in 2024-03-30T04:22:08.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:08 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:22:08.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:08 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:22:08.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:08 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:22:08.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:08 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:22:08.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:08 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:22:08.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:08 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:22:08.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:08 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:22:08.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:08 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:22:09.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:09 smithi135 ceph-mon[44364]: osdmap e30: 4 total, 4 up, 4 in 2024-03-30T04:22:09.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:09 smithi123 ceph-mon[35245]: osdmap e30: 4 total, 4 up, 4 in 2024-03-30T04:22:10.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:10 smithi135 ceph-mon[44364]: pgmap v114: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-30T04:22:10.367 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:10.368+0000 7fb7b6200700 1 -- 172.21.15.135:0/4257380026 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb7b00fcee0 msgr2=0x7fb7b00fd310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:22:10.367 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:10.368+0000 7fb7b6200700 1 --2- 172.21.15.135:0/4257380026 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb7b00fcee0 0x7fb7b00fd310 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb7a0004970 tx=0x7fb7a0009e30 comp rx=0 tx=0).stop 2024-03-30T04:22:10.368 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:10.368+0000 7fb7b6200700 1 -- 172.21.15.135:0/4257380026 shutdown_connections 2024-03-30T04:22:10.368 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:10.368+0000 7fb7b6200700 1 --2- 172.21.15.135:0/4257380026 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7b00fe0d0 0x7fb7b00fe540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:22:10.368 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:10.368+0000 7fb7b6200700 1 --2- 172.21.15.135:0/4257380026 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb7b00fcee0 0x7fb7b00fd310 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:22:10.368 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:10.368+0000 7fb7b6200700 1 -- 172.21.15.135:0/4257380026 >> 172.21.15.135:0/4257380026 conn(0x7fb7b00f8560 msgr2=0x7fb7b00fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:22:10.368 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:10.368+0000 7fb7b6200700 1 -- 172.21.15.135:0/4257380026 shutdown_connections 2024-03-30T04:22:10.368 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:10.368+0000 7fb7b6200700 1 -- 172.21.15.135:0/4257380026 wait complete. 2024-03-30T04:22:10.369 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:10.369+0000 7fb7b6200700 1 Processor -- start 2024-03-30T04:22:10.370 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:10.369+0000 7fb7b6200700 1 -- start start 2024-03-30T04:22:10.370 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:10.370+0000 7fb7b6200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb7b00fcee0 0x7fb7b010d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:22:10.370 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:10.370+0000 7fb7b6200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7b00fe0d0 0x7fb7b010de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:22:10.370 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:10.370+0000 7fb7b6200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7b010e430 con 0x7fb7b00fe0d0 2024-03-30T04:22:10.370 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:10.370+0000 7fb7b6200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7b010ec80 con 0x7fb7b00fcee0 2024-03-30T04:22:10.370 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:10.370+0000 7fb7af400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb7b00fcee0 0x7fb7b010d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:22:10.370 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:10.370+0000 7fb7af400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb7b00fcee0 0x7fb7b010d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.135:38758/0 (socket says 172.21.15.135:38758) 2024-03-30T04:22:10.370 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:10.370+0000 7fb7af400700 1 -- 172.21.15.135:0/2496973265 learned_addr learned my addr 172.21.15.135:0/2496973265 (peer_addr_for_me v2:172.21.15.135:0/0) 2024-03-30T04:22:10.370 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:10.370+0000 7fb7aea00700 1 --2- 172.21.15.135:0/2496973265 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7b00fe0d0 0x7fb7b010de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:22:10.370 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:10.370+0000 7fb7af400700 1 -- 172.21.15.135:0/2496973265 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7b00fe0d0 msgr2=0x7fb7b010de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:22:10.370 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:10.370+0000 7fb7af400700 1 --2- 172.21.15.135:0/2496973265 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7b00fe0d0 0x7fb7b010de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:22:10.370 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:10.370+0000 7fb7af400700 1 -- 172.21.15.135:0/2496973265 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7a0005040 con 0x7fb7b00fcee0 2024-03-30T04:22:10.370 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:10.370+0000 7fb7af400700 1 --2- 172.21.15.135:0/2496973265 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb7b00fcee0 0x7fb7b010d8f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb7a000b370 tx=0x7fb7a0009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:22:10.371 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:10.371+0000 7fb7a7400700 1 -- 172.21.15.135:0/2496973265 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7a000e040 con 0x7fb7b00fcee0 2024-03-30T04:22:10.371 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:10.371+0000 7fb7a7400700 1 -- 172.21.15.135:0/2496973265 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb7a0004290 con 0x7fb7b00fcee0 2024-03-30T04:22:10.371 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:10.371+0000 7fb7b6200700 1 -- 172.21.15.135:0/2496973265 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7b010ef00 con 0x7fb7b00fcee0 2024-03-30T04:22:10.371 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:10.371+0000 7fb7a7400700 1 -- 172.21.15.135:0/2496973265 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7a001c3f0 con 0x7fb7b00fcee0 2024-03-30T04:22:10.371 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:10.371+0000 7fb7b6200700 1 -- 172.21.15.135:0/2496973265 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7b010f440 con 0x7fb7b00fcee0 2024-03-30T04:22:10.371 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:10.372+0000 7fb7b6200700 1 -- 172.21.15.135:0/2496973265 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7b005f5f0 con 0x7fb7b00fcee0 2024-03-30T04:22:10.371 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:10.373+0000 7fb7a7400700 1 -- 172.21.15.135:0/2496973265 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fb7a0013790 con 0x7fb7b00fcee0 2024-03-30T04:22:10.372 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:10.373+0000 7fb7a7400700 1 --2- 172.21.15.135:0/2496973265 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fb7940692b0 0x7fb79406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:22:10.372 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:10.374+0000 7fb7a7400700 1 -- 172.21.15.135:0/2496973265 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 3770+0+0 (secure 0 0 0) 0x7fb7a0087670 con 0x7fb7b00fcee0 2024-03-30T04:22:10.373 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:10.374+0000 7fb7aea00700 1 --2- 172.21.15.135:0/2496973265 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fb7940692b0 0x7fb79406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:22:10.373 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:10.374+0000 7fb7aea00700 1 --2- 172.21.15.135:0/2496973265 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fb7940692b0 0x7fb79406b760 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb79c009b80 tx=0x7fb79c00b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:22:10.377 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:10.379+0000 7fb7a7400700 1 -- 172.21.15.135:0/2496973265 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fb7a0057b60 con 0x7fb7b00fcee0 2024-03-30T04:22:10.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:10 smithi123 ceph-mon[35245]: pgmap v114: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-30T04:22:10.579 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:10.579+0000 7fb7b6200700 1 -- 172.21.15.135:0/2496973265 --> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi135:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fb7b010f880 con 0x7fb7940692b0 2024-03-30T04:22:11.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:11 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T04:22:11.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:11 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T04:22:11.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:11 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:22:11.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:11 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T04:22:11.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:11 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T04:22:11.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:11 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:22:12.321 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:12 smithi135 ceph-mon[44364]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-30T04:22:12.321 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:12 smithi135 ceph-mon[44364]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi135:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:22:12.321 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:12 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:22:12.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:12 smithi123 ceph-mon[35245]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-30T04:22:12.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:12 smithi123 ceph-mon[35245]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi135:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:22:12.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:12 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:22:14.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:14 smithi135 ceph-mon[44364]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-03-30T04:22:14.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:14 smithi135 ceph-mon[44364]: from='client.? 172.21.15.135:0/1926660103' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "31868cec-dd4d-44dd-a150-7ebdbd61ea4f"}]: dispatch 2024-03-30T04:22:14.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:14 smithi135 ceph-mon[44364]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "31868cec-dd4d-44dd-a150-7ebdbd61ea4f"}]: dispatch 2024-03-30T04:22:14.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:14 smithi135 ceph-mon[44364]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "31868cec-dd4d-44dd-a150-7ebdbd61ea4f"}]': finished 2024-03-30T04:22:14.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:14 smithi135 ceph-mon[44364]: osdmap e31: 5 total, 4 up, 5 in 2024-03-30T04:22:14.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:14 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T04:22:14.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:14 smithi123 ceph-mon[35245]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-03-30T04:22:14.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:14 smithi123 ceph-mon[35245]: from='client.? 172.21.15.135:0/1926660103' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "31868cec-dd4d-44dd-a150-7ebdbd61ea4f"}]: dispatch 2024-03-30T04:22:14.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:14 smithi123 ceph-mon[35245]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "31868cec-dd4d-44dd-a150-7ebdbd61ea4f"}]: dispatch 2024-03-30T04:22:14.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:14 smithi123 ceph-mon[35245]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "31868cec-dd4d-44dd-a150-7ebdbd61ea4f"}]': finished 2024-03-30T04:22:14.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:14 smithi123 ceph-mon[35245]: osdmap e31: 5 total, 4 up, 5 in 2024-03-30T04:22:14.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:14 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T04:22:15.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:15 smithi135 ceph-mon[44364]: from='client.? 172.21.15.135:0/3901973489' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T04:22:15.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:15 smithi123 ceph-mon[35245]: from='client.? 172.21.15.135:0/3901973489' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T04:22:16.343 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:16 smithi123 ceph-mon[35245]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-03-30T04:22:16.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:16 smithi135 ceph-mon[44364]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-03-30T04:22:18.246 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:18 smithi135 ceph-mon[44364]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-03-30T04:22:18.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:18 smithi123 ceph-mon[35245]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-03-30T04:22:20.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:20 smithi135 ceph-mon[44364]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-30T04:22:20.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:20 smithi123 ceph-mon[35245]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-30T04:22:22.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:22 smithi135 ceph-mon[44364]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-30T04:22:22.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:22 smithi123 ceph-mon[35245]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-30T04:22:23.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:23 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-30T04:22:23.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:23 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:22:23.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:23 smithi135 ceph-mon[44364]: Deploying daemon osd.4 on smithi135 2024-03-30T04:22:23.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:23 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-30T04:22:23.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:23 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:22:23.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:23 smithi123 ceph-mon[35245]: Deploying daemon osd.4 on smithi135 2024-03-30T04:22:24.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:24 smithi123 ceph-mon[35245]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-30T04:22:24.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:24 smithi135 ceph-mon[44364]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-30T04:22:26.138 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:26 smithi135 ceph-mon[44364]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-30T04:22:26.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:26 smithi123 ceph-mon[35245]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-30T04:22:27.426 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:27 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:22:27.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:27 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:22:28.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:28 smithi123 ceph-mon[35245]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-30T04:22:28.596 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:28 smithi135 ceph-mon[44364]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-30T04:22:30.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:29 smithi135 ceph-mon[44364]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-30T04:22:30.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:29 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:22:30.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:29 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:22:30.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:29 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:22:30.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:29 smithi123 ceph-mon[35245]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-30T04:22:30.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:29 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:22:30.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:29 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:22:30.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:29 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:22:32.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:31 smithi123 ceph-mon[35245]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-30T04:22:32.293 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:31 smithi135 ceph-mon[44364]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-30T04:22:33.060 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:32 smithi135 ceph-mon[44364]: from='osd.4 [v2:172.21.15.135:6808/1699881600,v1:172.21.15.135:6809/1699881600]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-30T04:22:33.060 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:32 smithi135 ceph-mon[44364]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-30T04:22:33.106 INFO:teuthology.orchestra.run.smithi135.stdout:Created osd(s) 4 on host 'smithi135' 2024-03-30T04:22:33.107 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:33.108+0000 7fb7a7400700 1 -- 172.21.15.135:0/2496973265 <== mgr.14206 v2:172.21.15.123:6800/1564935593 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb7b010f880 con 0x7fb7940692b0 2024-03-30T04:22:33.108 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:33.110+0000 7fb7b6200700 1 -- 172.21.15.135:0/2496973265 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fb7940692b0 msgr2=0x7fb79406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:22:33.109 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:33.110+0000 7fb7b6200700 1 --2- 172.21.15.135:0/2496973265 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fb7940692b0 0x7fb79406b760 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb79c009b80 tx=0x7fb79c00b040 comp rx=0 tx=0).stop 2024-03-30T04:22:33.109 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:33.110+0000 7fb7b6200700 1 -- 172.21.15.135:0/2496973265 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb7b00fcee0 msgr2=0x7fb7b010d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:22:33.109 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:33.110+0000 7fb7b6200700 1 --2- 172.21.15.135:0/2496973265 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb7b00fcee0 0x7fb7b010d8f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb7a000b370 tx=0x7fb7a0009e30 comp rx=0 tx=0).stop 2024-03-30T04:22:33.109 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:33.111+0000 7fb7b6200700 1 -- 172.21.15.135:0/2496973265 shutdown_connections 2024-03-30T04:22:33.109 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:33.111+0000 7fb7b6200700 1 --2- 172.21.15.135:0/2496973265 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fb7940692b0 0x7fb79406b760 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:22:33.109 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:33.111+0000 7fb7b6200700 1 --2- 172.21.15.135:0/2496973265 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb7b00fcee0 0x7fb7b010d8f0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:22:33.109 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:33.111+0000 7fb7b6200700 1 --2- 172.21.15.135:0/2496973265 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7b00fe0d0 0x7fb7b010de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:22:33.109 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:33.111+0000 7fb7b6200700 1 -- 172.21.15.135:0/2496973265 >> 172.21.15.135:0/2496973265 conn(0x7fb7b00f8560 msgr2=0x7fb7b00f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:22:33.109 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:33.111+0000 7fb7b6200700 1 -- 172.21.15.135:0/2496973265 shutdown_connections 2024-03-30T04:22:33.109 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:33.111+0000 7fb7b6200700 1 -- 172.21.15.135:0/2496973265 wait complete. 2024-03-30T04:22:33.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:32 smithi123 ceph-mon[35245]: from='osd.4 [v2:172.21.15.135:6808/1699881600,v1:172.21.15.135:6809/1699881600]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-30T04:22:33.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:32 smithi123 ceph-mon[35245]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-30T04:22:33.778 DEBUG:teuthology.orchestra.run.smithi135:osd.4> sudo journalctl -f -n 0 -u ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4@osd.4.service 2024-03-30T04:22:33.783 INFO:tasks.cephadm:Deploying osd.5 on smithi135 with /dev/vg_nvme/lv_2... 2024-03-30T04:22:33.783 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-30T04:22:33.856 INFO:journalctl@ceph.osd.4.smithi135.stdout:-- Logs begin at Sat 2024-03-30 04:03:58 UTC. -- 2024-03-30T04:22:34.101 INFO:journalctl@ceph.osd.4.smithi135.stdout:Mar 30 04:22:33 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-4[58761]: 2024-03-30T04:22:33.878+0000 7f5414000700 -1 osd.4 0 waiting for initial osdmap 2024-03-30T04:22:34.101 INFO:journalctl@ceph.osd.4.smithi135.stdout:Mar 30 04:22:33 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-4[58761]: 2024-03-30T04:22:33.889+0000 7f540fa00700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T04:22:34.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:33 smithi135 ceph-mon[44364]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-30T04:22:34.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:33 smithi135 ceph-mon[44364]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-30T04:22:34.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:33 smithi135 ceph-mon[44364]: osdmap e32: 5 total, 4 up, 5 in 2024-03-30T04:22:34.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:33 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T04:22:34.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:33 smithi135 ceph-mon[44364]: from='osd.4 [v2:172.21.15.135:6808/1699881600,v1:172.21.15.135:6809/1699881600]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-03-30T04:22:34.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:33 smithi135 ceph-mon[44364]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-03-30T04:22:34.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:33 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:22:34.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:33 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:22:34.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:33 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:22:34.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:33 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:22:34.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:33 smithi123 ceph-mon[35245]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-30T04:22:34.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:33 smithi123 ceph-mon[35245]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-30T04:22:34.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:33 smithi123 ceph-mon[35245]: osdmap e32: 5 total, 4 up, 5 in 2024-03-30T04:22:34.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:33 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T04:22:34.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:33 smithi123 ceph-mon[35245]: from='osd.4 [v2:172.21.15.135:6808/1699881600,v1:172.21.15.135:6809/1699881600]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-03-30T04:22:34.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:33 smithi123 ceph-mon[35245]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-03-30T04:22:34.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:33 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:22:34.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:33 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:22:34.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:33 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:22:34.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:33 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:22:34.256 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi135/config 2024-03-30T04:22:35.195 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:34 smithi135 ceph-mon[44364]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]': finished 2024-03-30T04:22:35.196 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:34 smithi135 ceph-mon[44364]: osdmap e33: 5 total, 4 up, 5 in 2024-03-30T04:22:35.196 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:34 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T04:22:35.196 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:34 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T04:22:35.196 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:34 smithi135 ceph-mon[44364]: osd.4 [v2:172.21.15.135:6808/1699881600,v1:172.21.15.135:6809/1699881600] boot 2024-03-30T04:22:35.196 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:34 smithi135 ceph-mon[44364]: osdmap e34: 5 total, 5 up, 5 in 2024-03-30T04:22:35.196 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:34 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T04:22:35.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:34 smithi123 ceph-mon[35245]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]': finished 2024-03-30T04:22:35.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:34 smithi123 ceph-mon[35245]: osdmap e33: 5 total, 4 up, 5 in 2024-03-30T04:22:35.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:34 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T04:22:35.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:34 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T04:22:35.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:34 smithi123 ceph-mon[35245]: osd.4 [v2:172.21.15.135:6808/1699881600,v1:172.21.15.135:6809/1699881600] boot 2024-03-30T04:22:35.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:34 smithi123 ceph-mon[35245]: osdmap e34: 5 total, 5 up, 5 in 2024-03-30T04:22:35.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:34 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T04:22:36.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:35 smithi135 ceph-mon[44364]: purged_snaps scrub starts 2024-03-30T04:22:36.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:35 smithi135 ceph-mon[44364]: purged_snaps scrub ok 2024-03-30T04:22:36.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:35 smithi135 ceph-mon[44364]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-30T04:22:36.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:35 smithi123 ceph-mon[35245]: purged_snaps scrub starts 2024-03-30T04:22:36.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:35 smithi123 ceph-mon[35245]: purged_snaps scrub ok 2024-03-30T04:22:36.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:35 smithi123 ceph-mon[35245]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-30T04:22:37.059 INFO:teuthology.orchestra.run.smithi135.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-30T04:22:37.059 INFO:teuthology.orchestra.run.smithi135.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-30T04:22:37.059 INFO:teuthology.orchestra.run.smithi135.stderr: stderr: 10+0 records in 2024-03-30T04:22:37.059 INFO:teuthology.orchestra.run.smithi135.stderr:10+0 records out 2024-03-30T04:22:37.059 INFO:teuthology.orchestra.run.smithi135.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0463785 s, 226 MB/s 2024-03-30T04:22:37.059 INFO:teuthology.orchestra.run.smithi135.stderr:--> Zapping successful for: 2024-03-30T04:22:37.095 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:36 smithi135 ceph-mon[44364]: osdmap e35: 5 total, 5 up, 5 in 2024-03-30T04:22:37.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:36 smithi123 ceph-mon[35245]: osdmap e35: 5 total, 5 up, 5 in 2024-03-30T04:22:38.169 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:37 smithi135 ceph-mon[44364]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-30T04:22:38.192 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi135:vg_nvme/lv_2 2024-03-30T04:22:38.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:37 smithi123 ceph-mon[35245]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-30T04:22:38.662 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi135/config 2024-03-30T04:22:40.047 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:39 smithi135 ceph-mon[44364]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-30T04:22:40.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:39 smithi123 ceph-mon[35245]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-30T04:22:41.189 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:41.190+0000 7f2a3ce00700 1 -- 172.21.15.135:0/1373273913 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2a380fcee0 msgr2=0x7f2a380fd310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:22:41.190 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:41.190+0000 7f2a3ce00700 1 --2- 172.21.15.135:0/1373273913 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2a380fcee0 0x7f2a380fd310 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2a28004970 tx=0x7f2a28009e30 comp rx=0 tx=0).stop 2024-03-30T04:22:41.190 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:41.190+0000 7f2a3ce00700 1 -- 172.21.15.135:0/1373273913 shutdown_connections 2024-03-30T04:22:41.190 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:41.190+0000 7f2a3ce00700 1 --2- 172.21.15.135:0/1373273913 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2a380fe0d0 0x7f2a380fe540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:22:41.190 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:41.190+0000 7f2a3ce00700 1 --2- 172.21.15.135:0/1373273913 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2a380fcee0 0x7f2a380fd310 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:22:41.190 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:41.190+0000 7f2a3ce00700 1 -- 172.21.15.135:0/1373273913 >> 172.21.15.135:0/1373273913 conn(0x7f2a380f8560 msgr2=0x7f2a380fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:22:41.190 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:41.190+0000 7f2a3ce00700 1 -- 172.21.15.135:0/1373273913 shutdown_connections 2024-03-30T04:22:41.190 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:41.190+0000 7f2a3ce00700 1 -- 172.21.15.135:0/1373273913 wait complete. 2024-03-30T04:22:41.190 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:41.191+0000 7f2a3ce00700 1 Processor -- start 2024-03-30T04:22:41.190 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:41.191+0000 7f2a3ce00700 1 -- start start 2024-03-30T04:22:41.192 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:41.191+0000 7f2a3ce00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2a380fcee0 0x7f2a3810d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:22:41.192 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:41.191+0000 7f2a3ce00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2a380fe0d0 0x7f2a3810de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:22:41.192 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:41.191+0000 7f2a3ce00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a3810e430 con 0x7f2a380fcee0 2024-03-30T04:22:41.192 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:41.191+0000 7f2a3ce00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a3810ec80 con 0x7f2a380fe0d0 2024-03-30T04:22:41.192 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:41.192+0000 7f2a35400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2a380fe0d0 0x7f2a3810de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:22:41.192 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:41.192+0000 7f2a35e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2a380fcee0 0x7f2a3810d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:22:41.192 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:41.192+0000 7f2a35400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2a380fe0d0 0x7f2a3810de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.135:41778/0 (socket says 172.21.15.135:41778) 2024-03-30T04:22:41.192 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:41.192+0000 7f2a35400700 1 -- 172.21.15.135:0/2111380984 learned_addr learned my addr 172.21.15.135:0/2111380984 (peer_addr_for_me v2:172.21.15.135:0/0) 2024-03-30T04:22:41.192 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:41.192+0000 7f2a35e00700 1 -- 172.21.15.135:0/2111380984 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2a380fe0d0 msgr2=0x7f2a3810de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:22:41.192 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:41.192+0000 7f2a35e00700 1 --2- 172.21.15.135:0/2111380984 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2a380fe0d0 0x7f2a3810de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:22:41.192 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:41.192+0000 7f2a35e00700 1 -- 172.21.15.135:0/2111380984 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a28005040 con 0x7f2a380fcee0 2024-03-30T04:22:41.193 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:41.192+0000 7f2a35e00700 1 --2- 172.21.15.135:0/2111380984 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2a380fcee0 0x7f2a3810d8f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2a28005010 tx=0x7f2a28009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:22:41.193 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:41.192+0000 7f2a2e000700 1 -- 172.21.15.135:0/2111380984 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a2800e040 con 0x7f2a380fcee0 2024-03-30T04:22:41.193 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:41.193+0000 7f2a2e000700 1 -- 172.21.15.135:0/2111380984 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2a28013420 con 0x7f2a380fcee0 2024-03-30T04:22:41.193 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:41.193+0000 7f2a3ce00700 1 -- 172.21.15.135:0/2111380984 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a3810ef00 con 0x7f2a380fcee0 2024-03-30T04:22:41.193 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:41.193+0000 7f2a2e000700 1 -- 172.21.15.135:0/2111380984 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a2801c630 con 0x7f2a380fcee0 2024-03-30T04:22:41.193 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:41.193+0000 7f2a3ce00700 1 -- 172.21.15.135:0/2111380984 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a3810f410 con 0x7f2a380fcee0 2024-03-30T04:22:41.193 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:41.194+0000 7f2a3ce00700 1 -- 172.21.15.135:0/2111380984 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a3805f5f0 con 0x7f2a380fcee0 2024-03-30T04:22:41.193 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:41.195+0000 7f2a2e000700 1 -- 172.21.15.135:0/2111380984 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f2a28013970 con 0x7f2a380fcee0 2024-03-30T04:22:41.194 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:41.195+0000 7f2a2e000700 1 --2- 172.21.15.135:0/2111380984 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f2a1c0692b0 0x7f2a1c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:22:41.194 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:41.195+0000 7f2a2e000700 1 -- 172.21.15.135:0/2111380984 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4202+0+0 (secure 0 0 0) 0x7f2a28087e10 con 0x7f2a380fcee0 2024-03-30T04:22:41.194 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:41.196+0000 7f2a35400700 1 --2- 172.21.15.135:0/2111380984 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f2a1c0692b0 0x7f2a1c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:22:41.195 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:41.196+0000 7f2a35400700 1 --2- 172.21.15.135:0/2111380984 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f2a1c0692b0 0x7f2a1c06b760 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2a24008fd0 tx=0x7f2a2400b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:22:41.199 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:41.201+0000 7f2a2e000700 1 -- 172.21.15.135:0/2111380984 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f2a280580a0 con 0x7f2a380fcee0 2024-03-30T04:22:41.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:41 smithi135 ceph-mon[44364]: Detected new or changed devices on smithi135 2024-03-30T04:22:41.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:41 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:22:41.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:41 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:22:41.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:41 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:22:41.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:41 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:22:41.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:41 smithi135 ceph-mon[44364]: Adjusting osd_memory_target on smithi135 to 8059M 2024-03-30T04:22:41.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:41 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:22:41.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:41 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:22:41.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:41 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:22:41.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:41 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:22:41.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:41 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:22:41.406 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:22:41.407+0000 7f2a3ce00700 1 -- 172.21.15.135:0/2111380984 --> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi135:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f2a3810f830 con 0x7f2a1c0692b0 2024-03-30T04:22:41.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:41 smithi123 ceph-mon[35245]: Detected new or changed devices on smithi135 2024-03-30T04:22:41.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:41 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:22:41.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:41 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:22:41.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:41 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:22:41.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:41 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:22:41.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:41 smithi123 ceph-mon[35245]: Adjusting osd_memory_target on smithi135 to 8059M 2024-03-30T04:22:41.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:41 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:22:41.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:41 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:22:41.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:41 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:22:41.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:41 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:22:41.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:41 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:22:42.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:42 smithi135 ceph-mon[44364]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-30T04:22:42.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:42 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:22:42.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:42 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:22:42.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:42 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:22:42.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:42 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:22:42.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:42 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T04:22:42.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:42 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T04:22:42.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:42 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:22:42.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:42 smithi123 ceph-mon[35245]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-30T04:22:42.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:42 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:22:42.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:42 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:22:42.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:42 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:22:42.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:42 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:22:42.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:42 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T04:22:42.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:42 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T04:22:42.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:42 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:22:43.414 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:43 smithi135 ceph-mon[44364]: from='client.14334 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi135:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:22:43.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:43 smithi123 ceph-mon[35245]: from='client.14334 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi135:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:22:44.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:44 smithi123 ceph-mon[35245]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-30T04:22:44.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:44 smithi135 ceph-mon[44364]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-30T04:22:45.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:45 smithi123 ceph-mon[35245]: from='client.? 172.21.15.135:0/177336130' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc6e11b9-4729-4504-8917-51a3bb5e2b7c"}]: dispatch 2024-03-30T04:22:45.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:45 smithi123 ceph-mon[35245]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc6e11b9-4729-4504-8917-51a3bb5e2b7c"}]: dispatch 2024-03-30T04:22:45.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:45 smithi123 ceph-mon[35245]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dc6e11b9-4729-4504-8917-51a3bb5e2b7c"}]': finished 2024-03-30T04:22:45.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:45 smithi123 ceph-mon[35245]: osdmap e36: 6 total, 5 up, 6 in 2024-03-30T04:22:45.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:45 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T04:22:45.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:45 smithi135 ceph-mon[44364]: from='client.? 172.21.15.135:0/177336130' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc6e11b9-4729-4504-8917-51a3bb5e2b7c"}]: dispatch 2024-03-30T04:22:45.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:45 smithi135 ceph-mon[44364]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc6e11b9-4729-4504-8917-51a3bb5e2b7c"}]: dispatch 2024-03-30T04:22:45.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:45 smithi135 ceph-mon[44364]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dc6e11b9-4729-4504-8917-51a3bb5e2b7c"}]': finished 2024-03-30T04:22:45.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:45 smithi135 ceph-mon[44364]: osdmap e36: 6 total, 5 up, 6 in 2024-03-30T04:22:45.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:45 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T04:22:46.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:46 smithi123 ceph-mon[35245]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-30T04:22:46.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:46 smithi123 ceph-mon[35245]: from='client.? 172.21.15.135:0/1470228635' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T04:22:46.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:46 smithi135 ceph-mon[44364]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-30T04:22:46.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:46 smithi135 ceph-mon[44364]: from='client.? 172.21.15.135:0/1470228635' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T04:22:48.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:48 smithi123 ceph-mon[35245]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-30T04:22:48.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:48 smithi135 ceph-mon[44364]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-30T04:22:50.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:50 smithi123 ceph-mon[35245]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-30T04:22:50.515 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:50 smithi135 ceph-mon[44364]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-30T04:22:52.354 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:52 smithi135 ceph-mon[44364]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-30T04:22:52.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:52 smithi123 ceph-mon[35245]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-30T04:22:53.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:53 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-30T04:22:53.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:53 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:22:53.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:53 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-30T04:22:53.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:53 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:22:54.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:54 smithi123 ceph-mon[35245]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-30T04:22:54.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:54 smithi123 ceph-mon[35245]: Deploying daemon osd.5 on smithi135 2024-03-30T04:22:54.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:54 smithi135 ceph-mon[44364]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-30T04:22:54.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:54 smithi135 ceph-mon[44364]: Deploying daemon osd.5 on smithi135 2024-03-30T04:22:56.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:56 smithi123 ceph-mon[35245]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-30T04:22:56.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:56 smithi135 ceph-mon[44364]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-30T04:22:57.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:57 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:22:57.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:57 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:22:58.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:22:58 smithi123 ceph-mon[35245]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-30T04:22:58.563 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:22:58 smithi135 ceph-mon[44364]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-30T04:23:00.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:00 smithi135 ceph-mon[44364]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-30T04:23:00.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:00 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:23:00.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:00 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:23:00.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:00 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:23:00.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:00 smithi123 ceph-mon[35245]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-30T04:23:00.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:00 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:23:00.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:00 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:23:00.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:00 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:23:02.328 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:02 smithi135 ceph-mon[44364]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-30T04:23:02.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:02 smithi123 ceph-mon[35245]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-30T04:23:03.942 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:03.943+0000 7f2a2e000700 1 -- 172.21.15.135:0/2111380984 <== mgr.14206 v2:172.21.15.123:6800/1564935593 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f2a3810f830 con 0x7f2a1c0692b0 2024-03-30T04:23:03.942 INFO:teuthology.orchestra.run.smithi135.stdout:Created osd(s) 5 on host 'smithi135' 2024-03-30T04:23:03.944 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:03.945+0000 7f2a3ce00700 1 -- 172.21.15.135:0/2111380984 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f2a1c0692b0 msgr2=0x7f2a1c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:03.944 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:03.945+0000 7f2a3ce00700 1 --2- 172.21.15.135:0/2111380984 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f2a1c0692b0 0x7f2a1c06b760 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2a24008fd0 tx=0x7f2a2400b040 comp rx=0 tx=0).stop 2024-03-30T04:23:03.944 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:03.945+0000 7f2a3ce00700 1 -- 172.21.15.135:0/2111380984 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2a380fcee0 msgr2=0x7f2a3810d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:03.944 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:03.945+0000 7f2a3ce00700 1 --2- 172.21.15.135:0/2111380984 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2a380fcee0 0x7f2a3810d8f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2a28005010 tx=0x7f2a28009e30 comp rx=0 tx=0).stop 2024-03-30T04:23:03.944 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:03.945+0000 7f2a3ce00700 1 -- 172.21.15.135:0/2111380984 shutdown_connections 2024-03-30T04:23:03.944 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:03.945+0000 7f2a3ce00700 1 --2- 172.21.15.135:0/2111380984 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f2a1c0692b0 0x7f2a1c06b760 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:03.945 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:03.945+0000 7f2a3ce00700 1 --2- 172.21.15.135:0/2111380984 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2a380fcee0 0x7f2a3810d8f0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:03.945 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:03.945+0000 7f2a3ce00700 1 --2- 172.21.15.135:0/2111380984 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2a380fe0d0 0x7f2a3810de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:03.945 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:03.945+0000 7f2a3ce00700 1 -- 172.21.15.135:0/2111380984 >> 172.21.15.135:0/2111380984 conn(0x7f2a380f8560 msgr2=0x7f2a380f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:23:03.945 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:03.945+0000 7f2a3ce00700 1 -- 172.21.15.135:0/2111380984 shutdown_connections 2024-03-30T04:23:03.945 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:03.945+0000 7f2a3ce00700 1 -- 172.21.15.135:0/2111380984 wait complete. 2024-03-30T04:23:04.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:04 smithi123 ceph-mon[35245]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-30T04:23:04.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:04 smithi123 ceph-mon[35245]: from='osd.5 [v2:172.21.15.135:6816/1609344282,v1:172.21.15.135:6817/1609344282]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-30T04:23:04.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:04 smithi123 ceph-mon[35245]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-30T04:23:04.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:04 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:23:04.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:04 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:23:04.584 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:04 smithi135 ceph-mon[44364]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-30T04:23:04.584 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:04 smithi135 ceph-mon[44364]: from='osd.5 [v2:172.21.15.135:6816/1609344282,v1:172.21.15.135:6817/1609344282]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-30T04:23:04.584 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:04 smithi135 ceph-mon[44364]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-30T04:23:04.584 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:04 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:23:04.584 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:04 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:23:04.612 DEBUG:teuthology.orchestra.run.smithi135:osd.5> sudo journalctl -f -n 0 -u ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4@osd.5.service 2024-03-30T04:23:04.616 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-03-30T04:23:04.616 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph osd stat -f json 2024-03-30T04:23:04.689 INFO:journalctl@ceph.osd.5.smithi135.stdout:-- Logs begin at Sat 2024-03-30 04:03:58 UTC. -- 2024-03-30T04:23:05.048 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:23:05.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:05 smithi123 ceph-mon[35245]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-30T04:23:05.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:05 smithi123 ceph-mon[35245]: osdmap e37: 6 total, 5 up, 6 in 2024-03-30T04:23:05.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:05 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T04:23:05.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:05 smithi123 ceph-mon[35245]: from='osd.5 [v2:172.21.15.135:6816/1609344282,v1:172.21.15.135:6817/1609344282]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-03-30T04:23:05.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:05 smithi123 ceph-mon[35245]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-03-30T04:23:05.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:05 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:23:05.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:05 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:23:05.533 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 04:23:05 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-5[67404]: 2024-03-30T04:23:05.210+0000 7f4f10c00700 -1 osd.5 0 waiting for initial osdmap 2024-03-30T04:23:05.533 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 04:23:05 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-5[67404]: 2024-03-30T04:23:05.223+0000 7f4f08000700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T04:23:05.534 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:05 smithi135 ceph-mon[44364]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-30T04:23:05.534 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:05 smithi135 ceph-mon[44364]: osdmap e37: 6 total, 5 up, 6 in 2024-03-30T04:23:05.534 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:05 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T04:23:05.534 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:05 smithi135 ceph-mon[44364]: from='osd.5 [v2:172.21.15.135:6816/1609344282,v1:172.21.15.135:6817/1609344282]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-03-30T04:23:05.534 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:05 smithi135 ceph-mon[44364]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-03-30T04:23:05.534 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:05 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:23:05.534 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:05 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:23:06.343 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:06 smithi123 ceph-mon[35245]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-30T04:23:06.343 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:06 smithi123 ceph-mon[35245]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]': finished 2024-03-30T04:23:06.343 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:06 smithi123 ceph-mon[35245]: osdmap e38: 6 total, 5 up, 6 in 2024-03-30T04:23:06.344 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:06 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T04:23:06.344 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:06 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T04:23:06.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:06 smithi135 ceph-mon[44364]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-30T04:23:06.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:06 smithi135 ceph-mon[44364]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]': finished 2024-03-30T04:23:06.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:06 smithi135 ceph-mon[44364]: osdmap e38: 6 total, 5 up, 6 in 2024-03-30T04:23:06.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:06 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T04:23:06.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:06 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T04:23:07.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:07.124+0000 7ff2bec00700 1 -- 172.21.15.123:0/1396119861 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2b80fe0d0 msgr2=0x7ff2b80fe540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:07.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:07.124+0000 7ff2bec00700 1 --2- 172.21.15.123:0/1396119861 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2b80fe0d0 0x7ff2b80fe540 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff2a4004940 tx=0x7ff2a4009e30 comp rx=0 tx=0).stop 2024-03-30T04:23:07.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:07.124+0000 7ff2bec00700 1 -- 172.21.15.123:0/1396119861 shutdown_connections 2024-03-30T04:23:07.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:07.124+0000 7ff2bec00700 1 --2- 172.21.15.123:0/1396119861 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2b80fe0d0 0x7ff2b80fe540 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:07.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:07.124+0000 7ff2bec00700 1 --2- 172.21.15.123:0/1396119861 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff2b80fcee0 0x7ff2b80fd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:07.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:07.124+0000 7ff2bec00700 1 -- 172.21.15.123:0/1396119861 >> 172.21.15.123:0/1396119861 conn(0x7ff2b80f8560 msgr2=0x7ff2b80fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:23:07.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:07.124+0000 7ff2bec00700 1 -- 172.21.15.123:0/1396119861 shutdown_connections 2024-03-30T04:23:07.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:07.124+0000 7ff2bec00700 1 -- 172.21.15.123:0/1396119861 wait complete. 2024-03-30T04:23:07.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:07.125+0000 7ff2bec00700 1 Processor -- start 2024-03-30T04:23:07.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:07.125+0000 7ff2bec00700 1 -- start start 2024-03-30T04:23:07.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:07.125+0000 7ff2bec00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff2b80fcee0 0x7ff2b810fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:07.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:07.125+0000 7ff2bec00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2b80fe0d0 0x7ff2b81100c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:07.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:07.125+0000 7ff2bec00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b81106c0 con 0x7ff2b80fe0d0 2024-03-30T04:23:07.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:07.125+0000 7ff2bec00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b8110f10 con 0x7ff2b80fcee0 2024-03-30T04:23:07.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:07.126+0000 7ff2b7400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2b80fe0d0 0x7ff2b81100c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:07.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:07.126+0000 7ff2b7e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff2b80fcee0 0x7ff2b810fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:07.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:07.126+0000 7ff2b7400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2b80fe0d0 0x7ff2b81100c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37046/0 (socket says 172.21.15.123:37046) 2024-03-30T04:23:07.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:07.126+0000 7ff2b7400700 1 -- 172.21.15.123:0/3898360352 learned_addr learned my addr 172.21.15.123:0/3898360352 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:23:07.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:07.126+0000 7ff2b7400700 1 -- 172.21.15.123:0/3898360352 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff2b80fcee0 msgr2=0x7ff2b810fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:07.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:07.126+0000 7ff2b7400700 1 --2- 172.21.15.123:0/3898360352 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff2b80fcee0 0x7ff2b810fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:07.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:07.126+0000 7ff2b7400700 1 -- 172.21.15.123:0/3898360352 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a4005040 con 0x7ff2b80fe0d0 2024-03-30T04:23:07.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:07.126+0000 7ff2b7400700 1 --2- 172.21.15.123:0/3898360352 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2b80fe0d0 0x7ff2b81100c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff2a4013870 tx=0x7ff2a4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:23:07.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:07.126+0000 7ff2afe00700 1 -- 172.21.15.123:0/3898360352 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff2a4021070 con 0x7ff2b80fe0d0 2024-03-30T04:23:07.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:07.126+0000 7ff2afe00700 1 -- 172.21.15.123:0/3898360352 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff2a4013dc0 con 0x7ff2b80fe0d0 2024-03-30T04:23:07.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:07.126+0000 7ff2bec00700 1 -- 172.21.15.123:0/3898360352 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2b8111190 con 0x7ff2b80fe0d0 2024-03-30T04:23:07.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:07.126+0000 7ff2afe00700 1 -- 172.21.15.123:0/3898360352 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff2a400e070 con 0x7ff2b80fe0d0 2024-03-30T04:23:07.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:07.127+0000 7ff2bec00700 1 -- 172.21.15.123:0/3898360352 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2b8111670 con 0x7ff2b80fe0d0 2024-03-30T04:23:07.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:07.128+0000 7ff2bec00700 1 -- 172.21.15.123:0/3898360352 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2b805f5f0 con 0x7ff2b80fe0d0 2024-03-30T04:23:07.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:07.128+0000 7ff2afe00700 1 -- 172.21.15.123:0/3898360352 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7ff2a401c600 con 0x7ff2b80fe0d0 2024-03-30T04:23:07.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:07.129+0000 7ff2afe00700 1 --2- 172.21.15.123:0/3898360352 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7ff29c0692b0 0x7ff29c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:07.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:07.129+0000 7ff2afe00700 1 -- 172.21.15.123:0/3898360352 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ff2a4087fc0 con 0x7ff2b80fe0d0 2024-03-30T04:23:07.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:07.129+0000 7ff2b7e00700 1 --2- 172.21.15.123:0/3898360352 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7ff29c0692b0 0x7ff29c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:07.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:07.130+0000 7ff2b7e00700 1 --2- 172.21.15.123:0/3898360352 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7ff29c0692b0 0x7ff29c06b760 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff2a8009b80 tx=0x7ff2a800b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:23:07.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:07.134+0000 7ff2afe00700 1 -- 172.21.15.123:0/3898360352 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7ff2a40ba190 con 0x7ff2b80fe0d0 2024-03-30T04:23:07.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:07.331+0000 7ff2bec00700 1 -- 172.21.15.123:0/3898360352 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7ff2b8111980 con 0x7ff2b80fe0d0 2024-03-30T04:23:07.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:07.332+0000 7ff2afe00700 1 -- 172.21.15.123:0/3898360352 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v40) v1 ==== 74+0+130 (secure 0 0 0) 0x7ff2a4057e10 con 0x7ff2b80fe0d0 2024-03-30T04:23:07.331 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:23:07.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:07.334+0000 7ff2bec00700 1 -- 172.21.15.123:0/3898360352 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7ff29c0692b0 msgr2=0x7ff29c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:07.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:07.334+0000 7ff2bec00700 1 --2- 172.21.15.123:0/3898360352 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7ff29c0692b0 0x7ff29c06b760 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff2a8009b80 tx=0x7ff2a800b040 comp rx=0 tx=0).stop 2024-03-30T04:23:07.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:07.334+0000 7ff2bec00700 1 -- 172.21.15.123:0/3898360352 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2b80fe0d0 msgr2=0x7ff2b81100c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:07.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:07.334+0000 7ff2bec00700 1 --2- 172.21.15.123:0/3898360352 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2b80fe0d0 0x7ff2b81100c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff2a4013870 tx=0x7ff2a4009e30 comp rx=0 tx=0).stop 2024-03-30T04:23:07.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:07.334+0000 7ff2bec00700 1 -- 172.21.15.123:0/3898360352 shutdown_connections 2024-03-30T04:23:07.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:07.334+0000 7ff2bec00700 1 --2- 172.21.15.123:0/3898360352 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7ff29c0692b0 0x7ff29c06b760 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:07.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:07.334+0000 7ff2bec00700 1 --2- 172.21.15.123:0/3898360352 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff2b80fcee0 0x7ff2b810fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:07.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:07.334+0000 7ff2bec00700 1 --2- 172.21.15.123:0/3898360352 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2b80fe0d0 0x7ff2b81100c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:07.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:07.334+0000 7ff2bec00700 1 -- 172.21.15.123:0/3898360352 >> 172.21.15.123:0/3898360352 conn(0x7ff2b80f8560 msgr2=0x7ff2b80f9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:23:07.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:07.334+0000 7ff2bec00700 1 -- 172.21.15.123:0/3898360352 shutdown_connections 2024-03-30T04:23:07.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:07.334+0000 7ff2bec00700 1 -- 172.21.15.123:0/3898360352 wait complete. 2024-03-30T04:23:07.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:07 smithi123 ceph-mon[35245]: purged_snaps scrub starts 2024-03-30T04:23:07.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:07 smithi123 ceph-mon[35245]: purged_snaps scrub ok 2024-03-30T04:23:07.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:07 smithi123 ceph-mon[35245]: osd.5 [v2:172.21.15.135:6816/1609344282,v1:172.21.15.135:6817/1609344282] boot 2024-03-30T04:23:07.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:07 smithi123 ceph-mon[35245]: osdmap e39: 6 total, 6 up, 6 in 2024-03-30T04:23:07.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:07 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T04:23:07.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:07 smithi135 ceph-mon[44364]: purged_snaps scrub starts 2024-03-30T04:23:07.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:07 smithi135 ceph-mon[44364]: purged_snaps scrub ok 2024-03-30T04:23:07.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:07 smithi135 ceph-mon[44364]: osd.5 [v2:172.21.15.135:6816/1609344282,v1:172.21.15.135:6817/1609344282] boot 2024-03-30T04:23:07.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:07 smithi135 ceph-mon[44364]: osdmap e39: 6 total, 6 up, 6 in 2024-03-30T04:23:07.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:07 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T04:23:08.033 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":40,"num_osds":6,"num_up_osds":6,"osd_up_since":1711772586,"num_in_osds":6,"osd_in_since":1711772564,"num_remapped_pgs":0} 2024-03-30T04:23:08.034 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph osd dump --format=json 2024-03-30T04:23:08.455 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:23:08.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:08 smithi123 ceph-mon[35245]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:23:08.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:08 smithi123 ceph-mon[35245]: osdmap e40: 6 total, 6 up, 6 in 2024-03-30T04:23:08.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:08 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/3898360352' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-30T04:23:08.554 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:08 smithi135 ceph-mon[44364]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:23:08.554 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:08 smithi135 ceph-mon[44364]: osdmap e40: 6 total, 6 up, 6 in 2024-03-30T04:23:08.554 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:08 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/3898360352' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-30T04:23:10.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:09 smithi123 ceph-mon[35245]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:23:10.157 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:09 smithi123 ceph-mon[35245]: Detected new or changed devices on smithi135 2024-03-30T04:23:10.157 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:09 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:23:10.157 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:09 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:23:10.157 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:09 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:23:10.157 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:09 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:23:10.157 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:09 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:23:10.157 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:09 smithi123 ceph-mon[35245]: Adjusting osd_memory_target on smithi135 to 5372M 2024-03-30T04:23:10.157 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:09 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:23:10.157 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:09 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:23:10.157 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:09 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:23:10.157 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:09 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:23:10.157 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:09 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:23:10.158 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:09 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:23:10.158 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:09 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:23:10.158 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:09 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:23:10.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:09 smithi135 ceph-mon[44364]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:23:10.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:09 smithi135 ceph-mon[44364]: Detected new or changed devices on smithi135 2024-03-30T04:23:10.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:09 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:23:10.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:09 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:23:10.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:09 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:23:10.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:09 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:23:10.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:09 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:23:10.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:09 smithi135 ceph-mon[44364]: Adjusting osd_memory_target on smithi135 to 5372M 2024-03-30T04:23:10.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:09 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:23:10.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:09 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:23:10.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:09 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:23:10.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:09 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:23:10.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:09 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:23:10.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:09 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:23:10.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:09 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:23:10.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:09 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:23:10.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:10.496+0000 7f80de600700 1 -- 172.21.15.123:0/264447901 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80d8102e00 msgr2=0x7f80d81031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:10.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:10.496+0000 7f80de600700 1 --2- 172.21.15.123:0/264447901 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80d8102e00 0x7f80d81031d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f80c8004940 tx=0x7f80c8009e30 comp rx=0 tx=0).stop 2024-03-30T04:23:10.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:10.496+0000 7f80de600700 1 -- 172.21.15.123:0/264447901 shutdown_connections 2024-03-30T04:23:10.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:10.496+0000 7f80de600700 1 --2- 172.21.15.123:0/264447901 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f80d81037a0 0x7f80d80fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:10.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:10.496+0000 7f80de600700 1 --2- 172.21.15.123:0/264447901 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80d8102e00 0x7f80d81031d0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:10.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:10.496+0000 7f80de600700 1 -- 172.21.15.123:0/264447901 >> 172.21.15.123:0/264447901 conn(0x7f80d80f89b0 msgr2=0x7f80d80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:23:10.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:10.496+0000 7f80de600700 1 -- 172.21.15.123:0/264447901 shutdown_connections 2024-03-30T04:23:10.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:10.496+0000 7f80de600700 1 -- 172.21.15.123:0/264447901 wait complete. 2024-03-30T04:23:10.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:10.497+0000 7f80de600700 1 Processor -- start 2024-03-30T04:23:10.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:10.497+0000 7f80de600700 1 -- start start 2024-03-30T04:23:10.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:10.497+0000 7f80de600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80d8102e00 0x7f80d80702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:10.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:10.497+0000 7f80de600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f80d81037a0 0x7f80d8070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:10.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:10.497+0000 7f80de600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80d8070d90 con 0x7f80d8102e00 2024-03-30T04:23:10.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:10.497+0000 7f80de600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80d8070ed0 con 0x7f80d81037a0 2024-03-30T04:23:10.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:10.497+0000 7f80d7400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80d8102e00 0x7f80d80702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:10.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:10.497+0000 7f80d7400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80d8102e00 0x7f80d80702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37062/0 (socket says 172.21.15.123:37062) 2024-03-30T04:23:10.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:10.498+0000 7f80d7400700 1 -- 172.21.15.123:0/3566709732 learned_addr learned my addr 172.21.15.123:0/3566709732 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:23:10.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:10.498+0000 7f80d7400700 1 -- 172.21.15.123:0/3566709732 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f80d81037a0 msgr2=0x7f80d8070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:10.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:10.498+0000 7f80d7400700 1 --2- 172.21.15.123:0/3566709732 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f80d81037a0 0x7f80d8070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:10.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:10.498+0000 7f80d7400700 1 -- 172.21.15.123:0/3566709732 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80c8005040 con 0x7f80d8102e00 2024-03-30T04:23:10.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:10.498+0000 7f80d7400700 1 --2- 172.21.15.123:0/3566709732 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80d8102e00 0x7f80d80702c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f80c8009390 tx=0x7f80c8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:23:10.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:10.498+0000 7f80cf400700 1 -- 172.21.15.123:0/3566709732 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f80c800e040 con 0x7f80d8102e00 2024-03-30T04:23:10.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:10.498+0000 7f80cf400700 1 -- 172.21.15.123:0/3566709732 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f80c8013420 con 0x7f80d8102e00 2024-03-30T04:23:10.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:10.498+0000 7f80de600700 1 -- 172.21.15.123:0/3566709732 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80d8071120 con 0x7f80d8102e00 2024-03-30T04:23:10.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:10.498+0000 7f80cf400700 1 -- 172.21.15.123:0/3566709732 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f80c801c630 con 0x7f80d8102e00 2024-03-30T04:23:10.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:10.498+0000 7f80de600700 1 -- 172.21.15.123:0/3566709732 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80d811a250 con 0x7f80d8102e00 2024-03-30T04:23:10.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:10.500+0000 7f80de600700 1 -- 172.21.15.123:0/3566709732 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80d80fe550 con 0x7f80d8102e00 2024-03-30T04:23:10.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:10.500+0000 7f80cf400700 1 -- 172.21.15.123:0/3566709732 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f80c80139e0 con 0x7f80d8102e00 2024-03-30T04:23:10.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:10.501+0000 7f80cf400700 1 --2- 172.21.15.123:0/3566709732 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f80bc0692b0 0x7f80bc06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:10.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:10.501+0000 7f80cf400700 1 -- 172.21.15.123:0/3566709732 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f80c8087ff0 con 0x7f80d8102e00 2024-03-30T04:23:10.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:10.501+0000 7f80d6a00700 1 --2- 172.21.15.123:0/3566709732 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f80bc0692b0 0x7f80bc06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:10.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:10.502+0000 7f80d6a00700 1 --2- 172.21.15.123:0/3566709732 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f80bc0692b0 0x7f80bc06b760 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f80d8071c00 tx=0x7f80c4009ce0 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:23:10.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:10.506+0000 7f80cf400700 1 -- 172.21.15.123:0/3566709732 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f80c8054680 con 0x7f80d8102e00 2024-03-30T04:23:10.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:10.701+0000 7f80de600700 1 -- 172.21.15.123:0/3566709732 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f80d805f5f0 con 0x7f80d8102e00 2024-03-30T04:23:10.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:10.702+0000 7f80cf400700 1 -- 172.21.15.123:0/3566709732 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11141 (secure 0 0 0) 0x7f80c8057ca0 con 0x7f80d8102e00 2024-03-30T04:23:10.701 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:23:10.701 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":40,"fsid":"2e77a2be-ee4c-11ee-b647-cb9ed24678a4","created":"2024-03-30T04:16:30.949442+0000","modified":"2024-03-30T04:23:07.209822+0000","last_up_change":"2024-03-30T04:23:06.203543+0000","last_in_change":"2024-03-30T04:22:44.504478+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T04:21:34.842970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"3968446b-8696-4eec-b640-c97973ca054d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":1768008525},{"type":"v1","addr":"172.21.15.123:6803","nonce":1768008525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":1768008525},{"type":"v1","addr":"172.21.15.123:6805","nonce":1768008525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1768008525},{"type":"v1","addr":"172.21.15.123:6809","nonce":1768008525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":1768008525},{"type":"v1","addr":"172.21.15.123:6807","nonce":1768008525}]},"public_addr":"172.21.15.123:6803/1768008525","cluster_addr":"172.21.15.123:6805/1768008525","heartbeat_back_addr":"172.21.15.123:6809/1768008525","heartbeat_front_addr":"172.21.15.123:6807/1768008525","state":["exists","up"]},{"osd":1,"uuid":"336be34b-bb3c-46ab-8439-af04a1366db5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1040012134},{"type":"v1","addr":"172.21.15.123:6811","nonce":1040012134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1040012134},{"type":"v1","addr":"172.21.15.123:6813","nonce":1040012134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1040012134},{"type":"v1","addr":"172.21.15.123:6817","nonce":1040012134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1040012134},{"type":"v1","addr":"172.21.15.123:6815","nonce":1040012134}]},"public_addr":"172.21.15.123:6811/1040012134","cluster_addr":"172.21.15.123:6813/1040012134","heartbeat_back_addr":"172.21.15.123:6817/1040012134","heartbeat_front_addr":"172.21.15.123:6815/1040012134","state":["exists","up"]},{"osd":2,"uuid":"4386fc8a-e9ed-413e-bd3a-bb84a0e62827","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1260172805},{"type":"v1","addr":"172.21.15.123:6819","nonce":1260172805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1260172805},{"type":"v1","addr":"172.21.15.123:6821","nonce":1260172805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1260172805},{"type":"v1","addr":"172.21.15.123:6825","nonce":1260172805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1260172805},{"type":"v1","addr":"172.21.15.123:6823","nonce":1260172805}]},"public_addr":"172.21.15.123:6819/1260172805","cluster_addr":"172.21.15.123:6821/1260172805","heartbeat_back_addr":"172.21.15.123:6825/1260172805","heartbeat_front_addr":"172.21.15.123:6823/1260172805","state":["exists","up"]},{"osd":3,"uuid":"086d86d4-ff1a-47dd-8416-447ccf9064ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2906534885},{"type":"v1","addr":"172.21.15.135:6801","nonce":2906534885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2906534885},{"type":"v1","addr":"172.21.15.135:6803","nonce":2906534885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2906534885},{"type":"v1","addr":"172.21.15.135:6807","nonce":2906534885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2906534885},{"type":"v1","addr":"172.21.15.135:6805","nonce":2906534885}]},"public_addr":"172.21.15.135:6801/2906534885","cluster_addr":"172.21.15.135:6803/2906534885","heartbeat_back_addr":"172.21.15.135:6807/2906534885","heartbeat_front_addr":"172.21.15.135:6805/2906534885","state":["exists","up"]},{"osd":4,"uuid":"31868cec-dd4d-44dd-a150-7ebdbd61ea4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":1699881600},{"type":"v1","addr":"172.21.15.135:6809","nonce":1699881600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":1699881600},{"type":"v1","addr":"172.21.15.135:6811","nonce":1699881600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":1699881600},{"type":"v1","addr":"172.21.15.135:6815","nonce":1699881600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":1699881600},{"type":"v1","addr":"172.21.15.135:6813","nonce":1699881600}]},"public_addr":"172.21.15.135:6809/1699881600","cluster_addr":"172.21.15.135:6811/1699881600","heartbeat_back_addr":"172.21.15.135:6815/1699881600","heartbeat_front_addr":"172.21.15.135:6813/1699881600","state":["exists","up"]},{"osd":5,"uuid":"dc6e11b9-4729-4504-8917-51a3bb5e2b7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":1609344282},{"type":"v1","addr":"172.21.15.135:6817","nonce":1609344282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":1609344282},{"type":"v1","addr":"172.21.15.135:6819","nonce":1609344282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":1609344282},{"type":"v1","addr":"172.21.15.135:6823","nonce":1609344282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":1609344282},{"type":"v1","addr":"172.21.15.135:6821","nonce":1609344282}]},"public_addr":"172.21.15.135:6817/1609344282","cluster_addr":"172.21.15.135:6819/1609344282","heartbeat_back_addr":"172.21.15.135:6823/1609344282","heartbeat_front_addr":"172.21.15.135:6821/1609344282","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T04:20:25.138939+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T04:20:58.657719+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T04:21:32.397235+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T04:22:02.245820+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T04:22:33.842250+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T04:23:04.503576+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.123:6800/70490328":"2024-03-31T04:18:56.235750+0000","172.21.15.123:6801/70490328":"2024-03-31T04:18:56.235750+0000","172.21.15.123:0/3593359364":"2024-03-31T04:18:56.235750+0000","172.21.15.123:6801/1776532690":"2024-03-31T04:17:41.459573+0000","172.21.15.123:0/1275706223":"2024-03-31T04:18:56.235750+0000","172.21.15.123:6800/1776532690":"2024-03-31T04:17:41.459573+0000","172.21.15.123:0/133647565":"2024-03-31T04:17:41.459573+0000","172.21.15.123:0/2366611723":"2024-03-31T04:17:05.107043+0000","172.21.15.123:0/518566709":"2024-03-31T04:18:56.235750+0000","172.21.15.123:6801/1291592055":"2024-03-31T04:17:05.107043+0000","172.21.15.123:0/2358069559":"2024-03-31T04:17:41.459573+0000","172.21.15.123:0/250978570":"2024-03-31T04:17:05.107043+0000","172.21.15.123:0/3037148222":"2024-03-31T04:17:41.459573+0000","172.21.15.123:0/2756071515":"2024-03-31T04:17:05.107043+0000","172.21.15.123:6800/1291592055":"2024-03-31T04:17:05.107043+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T04:23:10.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:10.704+0000 7f80de600700 1 -- 172.21.15.123:0/3566709732 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f80bc0692b0 msgr2=0x7f80bc06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:10.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:10.704+0000 7f80de600700 1 --2- 172.21.15.123:0/3566709732 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f80bc0692b0 0x7f80bc06b760 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f80d8071c00 tx=0x7f80c4009ce0 comp rx=0 tx=0).stop 2024-03-30T04:23:10.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:10.704+0000 7f80de600700 1 -- 172.21.15.123:0/3566709732 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80d8102e00 msgr2=0x7f80d80702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:10.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:10.704+0000 7f80de600700 1 --2- 172.21.15.123:0/3566709732 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80d8102e00 0x7f80d80702c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f80c8009390 tx=0x7f80c8009e30 comp rx=0 tx=0).stop 2024-03-30T04:23:10.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:10.704+0000 7f80de600700 1 -- 172.21.15.123:0/3566709732 shutdown_connections 2024-03-30T04:23:10.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:10.704+0000 7f80de600700 1 --2- 172.21.15.123:0/3566709732 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f80bc0692b0 0x7f80bc06b760 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:10.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:10.704+0000 7f80de600700 1 --2- 172.21.15.123:0/3566709732 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80d8102e00 0x7f80d80702c0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:10.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:10.704+0000 7f80de600700 1 --2- 172.21.15.123:0/3566709732 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f80d81037a0 0x7f80d8070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:10.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:10.704+0000 7f80de600700 1 -- 172.21.15.123:0/3566709732 >> 172.21.15.123:0/3566709732 conn(0x7f80d80f89b0 msgr2=0x7f80d80f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:23:10.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:10.704+0000 7f80de600700 1 -- 172.21.15.123:0/3566709732 shutdown_connections 2024-03-30T04:23:10.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:10.704+0000 7f80de600700 1 -- 172.21.15.123:0/3566709732 wait complete. 2024-03-30T04:23:11.212 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:10 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/3566709732' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:23:11.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:10 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/3566709732' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:23:11.657 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T04:21:34.842970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-03-30T04:23:11.657 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph osd pool get .mgr pg_num 2024-03-30T04:23:12.076 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:23:12.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:11 smithi123 ceph-mon[35245]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:23:12.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:11 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:23:12.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:11 smithi135 ceph-mon[44364]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:23:12.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:11 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:23:13.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:13.920+0000 7f1be8e00700 1 -- 172.21.15.123:0/1699013595 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1be4102e20 msgr2=0x7f1be41031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:13.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:13.920+0000 7f1be8e00700 1 --2- 172.21.15.123:0/1699013595 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1be4102e20 0x7f1be41031f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1bd4004940 tx=0x7f1bd4009e30 comp rx=0 tx=0).stop 2024-03-30T04:23:13.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:13.920+0000 7f1be8e00700 1 -- 172.21.15.123:0/1699013595 shutdown_connections 2024-03-30T04:23:13.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:13.920+0000 7f1be8e00700 1 --2- 172.21.15.123:0/1699013595 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1be41037c0 0x7f1be40fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:13.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:13.920+0000 7f1be8e00700 1 --2- 172.21.15.123:0/1699013595 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1be4102e20 0x7f1be41031f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:13.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:13.920+0000 7f1be8e00700 1 -- 172.21.15.123:0/1699013595 >> 172.21.15.123:0/1699013595 conn(0x7f1be40f89d0 msgr2=0x7f1be40fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:23:13.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:13.920+0000 7f1be8e00700 1 -- 172.21.15.123:0/1699013595 shutdown_connections 2024-03-30T04:23:13.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:13.920+0000 7f1be8e00700 1 -- 172.21.15.123:0/1699013595 wait complete. 2024-03-30T04:23:13.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:13.921+0000 7f1be8e00700 1 Processor -- start 2024-03-30T04:23:13.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:13.921+0000 7f1be8e00700 1 -- start start 2024-03-30T04:23:13.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:13.921+0000 7f1be8e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1be4102e20 0x7f1be4113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:13.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:13.921+0000 7f1be8e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1be41037c0 0x7f1be4114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:13.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:13.921+0000 7f1be8e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1be4114690 con 0x7f1be4102e20 2024-03-30T04:23:13.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:13.921+0000 7f1be8e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1be410da80 con 0x7f1be41037c0 2024-03-30T04:23:13.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:13.922+0000 7f1be1e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1be4102e20 0x7f1be4113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:13.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:13.922+0000 7f1be1e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1be4102e20 0x7f1be4113b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55806/0 (socket says 172.21.15.123:55806) 2024-03-30T04:23:13.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:13.922+0000 7f1be1e00700 1 -- 172.21.15.123:0/2760036582 learned_addr learned my addr 172.21.15.123:0/2760036582 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:23:13.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:13.922+0000 7f1be1400700 1 --2- 172.21.15.123:0/2760036582 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1be41037c0 0x7f1be4114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:13.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:13.922+0000 7f1be1e00700 1 -- 172.21.15.123:0/2760036582 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1be41037c0 msgr2=0x7f1be4114040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:13.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:13.922+0000 7f1be1e00700 1 --2- 172.21.15.123:0/2760036582 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1be41037c0 0x7f1be4114040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:13.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:13.922+0000 7f1be1e00700 1 -- 172.21.15.123:0/2760036582 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bd4005040 con 0x7f1be4102e20 2024-03-30T04:23:13.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:13.922+0000 7f1be1e00700 1 --2- 172.21.15.123:0/2760036582 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1be4102e20 0x7f1be4113b00 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f1bd4009390 tx=0x7f1bd4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:23:13.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:13.922+0000 7f1bda000700 1 -- 172.21.15.123:0/2760036582 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1bd400e040 con 0x7f1be4102e20 2024-03-30T04:23:13.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:13.922+0000 7f1bda000700 1 -- 172.21.15.123:0/2760036582 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1bd4013420 con 0x7f1be4102e20 2024-03-30T04:23:13.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:13.922+0000 7f1be8e00700 1 -- 172.21.15.123:0/2760036582 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1be410dd00 con 0x7f1be4102e20 2024-03-30T04:23:13.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:13.922+0000 7f1bda000700 1 -- 172.21.15.123:0/2760036582 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1bd401c630 con 0x7f1be4102e20 2024-03-30T04:23:13.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:13.923+0000 7f1be8e00700 1 -- 172.21.15.123:0/2760036582 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1be410e2c0 con 0x7f1be4102e20 2024-03-30T04:23:13.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:13.924+0000 7f1be8e00700 1 -- 172.21.15.123:0/2760036582 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1be40fe570 con 0x7f1be4102e20 2024-03-30T04:23:13.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:13.924+0000 7f1bda000700 1 -- 172.21.15.123:0/2760036582 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f1bd40139e0 con 0x7f1be4102e20 2024-03-30T04:23:13.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:13.925+0000 7f1bda000700 1 --2- 172.21.15.123:0/2760036582 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f1bc80692b0 0x7f1bc806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:13.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:13.925+0000 7f1bda000700 1 -- 172.21.15.123:0/2760036582 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1bd4087ff0 con 0x7f1be4102e20 2024-03-30T04:23:13.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:13.926+0000 7f1be1400700 1 --2- 172.21.15.123:0/2760036582 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f1bc80692b0 0x7f1bc806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:13.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:13.926+0000 7f1be1400700 1 --2- 172.21.15.123:0/2760036582 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f1bc80692b0 0x7f1bc806b760 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1be410ec40 tx=0x7f1bd0009ee0 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:23:13.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:13.930+0000 7f1bda000700 1 -- 172.21.15.123:0/2760036582 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f1bd40ba190 con 0x7f1be4102e20 2024-03-30T04:23:14.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:14.121+0000 7f1be8e00700 1 -- 172.21.15.123:0/2760036582 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1be404bed0 con 0x7f1be4102e20 2024-03-30T04:23:14.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:14.121+0000 7f1bda000700 1 -- 172.21.15.123:0/2760036582 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v40) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1bd4058180 con 0x7f1be4102e20 2024-03-30T04:23:14.120 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 1 2024-03-30T04:23:14.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:14.124+0000 7f1be8e00700 1 -- 172.21.15.123:0/2760036582 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f1bc80692b0 msgr2=0x7f1bc806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:14.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:14.124+0000 7f1be8e00700 1 --2- 172.21.15.123:0/2760036582 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f1bc80692b0 0x7f1bc806b760 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1be410ec40 tx=0x7f1bd0009ee0 comp rx=0 tx=0).stop 2024-03-30T04:23:14.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:14.124+0000 7f1be8e00700 1 -- 172.21.15.123:0/2760036582 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1be4102e20 msgr2=0x7f1be4113b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:14.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:14.124+0000 7f1be8e00700 1 --2- 172.21.15.123:0/2760036582 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1be4102e20 0x7f1be4113b00 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f1bd4009390 tx=0x7f1bd4009e30 comp rx=0 tx=0).stop 2024-03-30T04:23:14.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:14.124+0000 7f1be8e00700 1 -- 172.21.15.123:0/2760036582 shutdown_connections 2024-03-30T04:23:14.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:14.124+0000 7f1be8e00700 1 --2- 172.21.15.123:0/2760036582 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f1bc80692b0 0x7f1bc806b760 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:14.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:14.124+0000 7f1be8e00700 1 --2- 172.21.15.123:0/2760036582 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1be4102e20 0x7f1be4113b00 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:14.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:14.124+0000 7f1be8e00700 1 --2- 172.21.15.123:0/2760036582 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1be41037c0 0x7f1be4114040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:14.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:14.124+0000 7f1be8e00700 1 -- 172.21.15.123:0/2760036582 >> 172.21.15.123:0/2760036582 conn(0x7f1be40f89d0 msgr2=0x7f1be40f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:23:14.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:14.124+0000 7f1be8e00700 1 -- 172.21.15.123:0/2760036582 shutdown_connections 2024-03-30T04:23:14.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:14.124+0000 7f1be8e00700 1 -- 172.21.15.123:0/2760036582 wait complete. 2024-03-30T04:23:14.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:13 smithi135 ceph-mon[44364]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:23:14.432 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:13 smithi123 ceph-mon[35245]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:23:14.752 INFO:tasks.cephadm:Setting up client nodes... 2024-03-30T04:23:14.752 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-30T04:23:15.171 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:23:15.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:14 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/2760036582' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T04:23:15.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:14 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/2760036582' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T04:23:16.343 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:15 smithi123 ceph-mon[35245]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:23:16.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:15 smithi135 ceph-mon[44364]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:23:17.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:17.081+0000 7fd9b7e00700 1 -- 172.21.15.123:0/1886262220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9b00fce70 msgr2=0x7fd9b00fd280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:17.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:17.081+0000 7fd9b7e00700 1 --2- 172.21.15.123:0/1886262220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9b00fce70 0x7fd9b00fd280 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd9a4004940 tx=0x7fd9a4009e30 comp rx=0 tx=0).stop 2024-03-30T04:23:17.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:17.081+0000 7fd9b7e00700 1 -- 172.21.15.123:0/1886262220 shutdown_connections 2024-03-30T04:23:17.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:17.081+0000 7fd9b7e00700 1 --2- 172.21.15.123:0/1886262220 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd9b00fe070 0x7fd9b00fe4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:17.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:17.081+0000 7fd9b7e00700 1 --2- 172.21.15.123:0/1886262220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9b00fce70 0x7fd9b00fd280 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:17.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:17.081+0000 7fd9b7e00700 1 -- 172.21.15.123:0/1886262220 >> 172.21.15.123:0/1886262220 conn(0x7fd9b00f8540 msgr2=0x7fd9b00fa9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:23:17.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:17.081+0000 7fd9b7e00700 1 -- 172.21.15.123:0/1886262220 shutdown_connections 2024-03-30T04:23:17.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:17.081+0000 7fd9b7e00700 1 -- 172.21.15.123:0/1886262220 wait complete. 2024-03-30T04:23:17.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:17.082+0000 7fd9b7e00700 1 Processor -- start 2024-03-30T04:23:17.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:17.082+0000 7fd9b7e00700 1 -- start start 2024-03-30T04:23:17.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:17.082+0000 7fd9b7e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9b00fce70 0x7fd9b010f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:17.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:17.082+0000 7fd9b7e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd9b00fe070 0x7fd9b010ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:17.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:17.082+0000 7fd9b7e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b0110530 con 0x7fd9b00fce70 2024-03-30T04:23:17.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:17.082+0000 7fd9b7e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b01106a0 con 0x7fd9b00fe070 2024-03-30T04:23:17.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:17.082+0000 7fd9b5400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9b00fce70 0x7fd9b010f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:17.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:17.082+0000 7fd9b5400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9b00fce70 0x7fd9b010f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55830/0 (socket says 172.21.15.123:55830) 2024-03-30T04:23:17.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:17.082+0000 7fd9b5400700 1 -- 172.21.15.123:0/377441958 learned_addr learned my addr 172.21.15.123:0/377441958 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:23:17.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:17.083+0000 7fd9b4a00700 1 --2- 172.21.15.123:0/377441958 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd9b00fe070 0x7fd9b010ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:17.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:17.083+0000 7fd9b5400700 1 -- 172.21.15.123:0/377441958 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd9b00fe070 msgr2=0x7fd9b010ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:17.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:17.083+0000 7fd9b5400700 1 --2- 172.21.15.123:0/377441958 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd9b00fe070 0x7fd9b010ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:17.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:17.083+0000 7fd9b5400700 1 -- 172.21.15.123:0/377441958 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a4005040 con 0x7fd9b00fce70 2024-03-30T04:23:17.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:17.083+0000 7fd9b5400700 1 --2- 172.21.15.123:0/377441958 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9b00fce70 0x7fd9b010f9f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd9a4000c00 tx=0x7fd9a4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:23:17.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:17.083+0000 7fd9ad600700 1 -- 172.21.15.123:0/377441958 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd9a400e070 con 0x7fd9b00fce70 2024-03-30T04:23:17.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:17.083+0000 7fd9ad600700 1 -- 172.21.15.123:0/377441958 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd9a40093b0 con 0x7fd9b00fce70 2024-03-30T04:23:17.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:17.083+0000 7fd9b7e00700 1 -- 172.21.15.123:0/377441958 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9b01150a0 con 0x7fd9b00fce70 2024-03-30T04:23:17.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:17.083+0000 7fd9ad600700 1 -- 172.21.15.123:0/377441958 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd9a401cd70 con 0x7fd9b00fce70 2024-03-30T04:23:17.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:17.084+0000 7fd9b7e00700 1 -- 172.21.15.123:0/377441958 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9b0115600 con 0x7fd9b00fce70 2024-03-30T04:23:17.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:17.084+0000 7fd9b7e00700 1 -- 172.21.15.123:0/377441958 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9b0101a00 con 0x7fd9b00fce70 2024-03-30T04:23:17.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:17.085+0000 7fd9ad600700 1 -- 172.21.15.123:0/377441958 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fd9a401c600 con 0x7fd9b00fce70 2024-03-30T04:23:17.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:17.086+0000 7fd9ad600700 1 --2- 172.21.15.123:0/377441958 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fd9980692b0 0x7fd99806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:17.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:17.086+0000 7fd9ad600700 1 -- 172.21.15.123:0/377441958 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd9a40883e0 con 0x7fd9b00fce70 2024-03-30T04:23:17.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:17.086+0000 7fd9b4a00700 1 --2- 172.21.15.123:0/377441958 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fd9980692b0 0x7fd99806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:17.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:17.087+0000 7fd9b4a00700 1 --2- 172.21.15.123:0/377441958 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fd9980692b0 0x7fd99806b760 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd99c009b80 tx=0x7fd99c00b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:23:17.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:17.090+0000 7fd9ad600700 1 -- 172.21.15.123:0/377441958 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fd9a40584c0 con 0x7fd9b00fce70 2024-03-30T04:23:17.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:17.361+0000 7fd9b7e00700 1 -- 172.21.15.123:0/377441958 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fd9b004c710 con 0x7fd9b00fce70 2024-03-30T04:23:17.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:17.365+0000 7fd9ad600700 1 -- 172.21.15.123:0/377441958 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v14) v1 ==== 170+0+59 (secure 0 0 0) 0x7fd9a4004490 con 0x7fd9b00fce70 2024-03-30T04:23:17.364 INFO:teuthology.orchestra.run.smithi123.stdout:[client.0] 2024-03-30T04:23:17.364 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQC1kwdm3mWbFRAA0qQRi6uBEX0gZuZzGKaKFA== 2024-03-30T04:23:17.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:17.367+0000 7fd9b7e00700 1 -- 172.21.15.123:0/377441958 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fd9980692b0 msgr2=0x7fd99806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:17.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:17.367+0000 7fd9b7e00700 1 --2- 172.21.15.123:0/377441958 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fd9980692b0 0x7fd99806b760 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd99c009b80 tx=0x7fd99c00b040 comp rx=0 tx=0).stop 2024-03-30T04:23:17.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:17.367+0000 7fd9b7e00700 1 -- 172.21.15.123:0/377441958 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9b00fce70 msgr2=0x7fd9b010f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:17.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:17.367+0000 7fd9b7e00700 1 --2- 172.21.15.123:0/377441958 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9b00fce70 0x7fd9b010f9f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd9a4000c00 tx=0x7fd9a4009e30 comp rx=0 tx=0).stop 2024-03-30T04:23:17.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:17.368+0000 7fd9b7e00700 1 -- 172.21.15.123:0/377441958 shutdown_connections 2024-03-30T04:23:17.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:17.368+0000 7fd9b7e00700 1 --2- 172.21.15.123:0/377441958 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fd9980692b0 0x7fd99806b760 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:17.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:17.368+0000 7fd9b7e00700 1 --2- 172.21.15.123:0/377441958 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9b00fce70 0x7fd9b010f9f0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:17.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:17.368+0000 7fd9b7e00700 1 --2- 172.21.15.123:0/377441958 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd9b00fe070 0x7fd9b010ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:17.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:17.368+0000 7fd9b7e00700 1 -- 172.21.15.123:0/377441958 >> 172.21.15.123:0/377441958 conn(0x7fd9b00f8540 msgr2=0x7fd9b00f9ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:23:17.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:17.368+0000 7fd9b7e00700 1 -- 172.21.15.123:0/377441958 shutdown_connections 2024-03-30T04:23:17.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:17.368+0000 7fd9b7e00700 1 -- 172.21.15.123:0/377441958 wait complete. 2024-03-30T04:23:18.090 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-30T04:23:18.405 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-30T04:23:18.405 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-30T04:23:18.406 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:17 smithi135 ceph-mon[44364]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:23:18.406 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:17 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/377441958' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-30T04:23:18.406 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:17 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/377441958' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-30T04:23:18.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:17 smithi123 ceph-mon[35245]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:23:18.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:17 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/377441958' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-30T04:23:18.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:17 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/377441958' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-30T04:23:18.502 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-30T04:23:18.931 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi135/config 2024-03-30T04:23:20.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:19 smithi135 ceph-mon[44364]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:23:20.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:19 smithi123 ceph-mon[35245]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:23:20.861 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:20.863+0000 7f1104400700 1 -- 172.21.15.135:0/3603001910 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10fc0fe070 msgr2=0x7f10fc0fe4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:20.861 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:20.863+0000 7f1104400700 1 --2- 172.21.15.135:0/3603001910 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10fc0fe070 0x7f10fc0fe4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f10e8004940 tx=0x7f10e8009e30 comp rx=0 tx=0).stop 2024-03-30T04:23:20.861 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:20.863+0000 7f1104400700 1 -- 172.21.15.135:0/3603001910 shutdown_connections 2024-03-30T04:23:20.862 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:20.863+0000 7f1104400700 1 --2- 172.21.15.135:0/3603001910 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10fc0fe070 0x7f10fc0fe4e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:20.862 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:20.863+0000 7f1104400700 1 --2- 172.21.15.135:0/3603001910 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10fc0fce70 0x7f10fc0fd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:20.862 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:20.863+0000 7f1104400700 1 -- 172.21.15.135:0/3603001910 >> 172.21.15.135:0/3603001910 conn(0x7f10fc0f8540 msgr2=0x7f10fc0fa9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:23:20.862 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:20.863+0000 7f1104400700 1 -- 172.21.15.135:0/3603001910 shutdown_connections 2024-03-30T04:23:20.862 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:20.863+0000 7f1104400700 1 -- 172.21.15.135:0/3603001910 wait complete. 2024-03-30T04:23:20.863 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:20.864+0000 7f1104400700 1 Processor -- start 2024-03-30T04:23:20.863 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:20.864+0000 7f1104400700 1 -- start start 2024-03-30T04:23:20.863 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:20.865+0000 7f1104400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10fc0fce70 0x7f10fc10d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:20.863 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:20.865+0000 7f1104400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10fc0fe070 0x7f10fc10dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:20.863 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:20.865+0000 7f1104400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10fc10e370 con 0x7f10fc0fce70 2024-03-30T04:23:20.863 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:20.865+0000 7f1104400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10fc10e4e0 con 0x7f10fc0fe070 2024-03-30T04:23:20.863 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:20.865+0000 7f1101000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10fc0fe070 0x7f10fc10dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:20.864 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:20.865+0000 7f1101000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10fc0fe070 0x7f10fc10dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.135:50928/0 (socket says 172.21.15.135:50928) 2024-03-30T04:23:20.864 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:20.865+0000 7f1101000700 1 -- 172.21.15.135:0/4114438414 learned_addr learned my addr 172.21.15.135:0/4114438414 (peer_addr_for_me v2:172.21.15.135:0/0) 2024-03-30T04:23:20.864 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:20.865+0000 7f1101a00700 1 --2- 172.21.15.135:0/4114438414 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10fc0fce70 0x7f10fc10d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:20.865 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:20.866+0000 7f1101000700 1 -- 172.21.15.135:0/4114438414 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10fc0fce70 msgr2=0x7f10fc10d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:20.865 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:20.866+0000 7f1101000700 1 --2- 172.21.15.135:0/4114438414 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10fc0fce70 0x7f10fc10d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:20.865 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:20.866+0000 7f1101000700 1 -- 172.21.15.135:0/4114438414 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10e8005040 con 0x7f10fc0fe070 2024-03-30T04:23:20.865 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:20.866+0000 7f1101000700 1 --2- 172.21.15.135:0/4114438414 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10fc0fe070 0x7f10fc10dd70 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f10e8009fd0 tx=0x7f10e8009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:23:20.865 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:20.866+0000 7f10f9600700 1 -- 172.21.15.135:0/4114438414 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f10e800e040 con 0x7f10fc0fe070 2024-03-30T04:23:20.865 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:20.866+0000 7f10f9600700 1 -- 172.21.15.135:0/4114438414 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f10e8004290 con 0x7f10fc0fe070 2024-03-30T04:23:20.865 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:20.866+0000 7f1104400700 1 -- 172.21.15.135:0/4114438414 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10fc112ee0 con 0x7f10fc0fe070 2024-03-30T04:23:20.865 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:20.866+0000 7f10f9600700 1 -- 172.21.15.135:0/4114438414 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f10e801c3f0 con 0x7f10fc0fe070 2024-03-30T04:23:20.865 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:20.866+0000 7f1104400700 1 -- 172.21.15.135:0/4114438414 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10fc113470 con 0x7f10fc0fe070 2024-03-30T04:23:20.866 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:20.867+0000 7f1104400700 1 -- 172.21.15.135:0/4114438414 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10fc04c710 con 0x7f10fc0fe070 2024-03-30T04:23:20.866 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:20.868+0000 7f10f9600700 1 -- 172.21.15.135:0/4114438414 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f10e8013790 con 0x7f10fc0fe070 2024-03-30T04:23:20.867 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:20.869+0000 7f10f9600700 1 --2- 172.21.15.135:0/4114438414 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f10e40692b0 0x7f10e406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:20.867 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:20.869+0000 7f10f9600700 1 -- 172.21.15.135:0/4114438414 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f10e8088270 con 0x7f10fc0fe070 2024-03-30T04:23:20.867 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:20.870+0000 7f1101a00700 1 --2- 172.21.15.135:0/4114438414 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f10e40692b0 0x7f10e406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:20.868 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:20.870+0000 7f1101a00700 1 --2- 172.21.15.135:0/4114438414 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f10e40692b0 0x7f10e406b760 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f10fc0fdbe0 tx=0x7f10f000b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:23:20.871 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:20.874+0000 7f10f9600700 1 -- 172.21.15.135:0/4114438414 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f10e80582d0 con 0x7f10fc0fe070 2024-03-30T04:23:21.143 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:21.145+0000 7f1104400700 1 -- 172.21.15.135:0/4114438414 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f10fc1057c0 con 0x7f10fc0fe070 2024-03-30T04:23:21.148 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:21.150+0000 7f10f9600700 1 -- 172.21.15.135:0/4114438414 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v15) v1 ==== 170+0+59 (secure 0 0 0) 0x7f10e8057e60 con 0x7f10fc0fe070 2024-03-30T04:23:21.148 INFO:teuthology.orchestra.run.smithi135.stdout:[client.1] 2024-03-30T04:23:21.149 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQC5kwdmwTa6CBAAu6tDwjjtVS07Ss3py8MSFw== 2024-03-30T04:23:21.151 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:21.153+0000 7f1104400700 1 -- 172.21.15.135:0/4114438414 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f10e40692b0 msgr2=0x7f10e406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:21.151 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:21.153+0000 7f1104400700 1 --2- 172.21.15.135:0/4114438414 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f10e40692b0 0x7f10e406b760 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f10fc0fdbe0 tx=0x7f10f000b040 comp rx=0 tx=0).stop 2024-03-30T04:23:21.151 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:21.153+0000 7f1104400700 1 -- 172.21.15.135:0/4114438414 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10fc0fe070 msgr2=0x7f10fc10dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:21.151 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:21.153+0000 7f1104400700 1 --2- 172.21.15.135:0/4114438414 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10fc0fe070 0x7f10fc10dd70 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f10e8009fd0 tx=0x7f10e8009e30 comp rx=0 tx=0).stop 2024-03-30T04:23:21.151 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:21.153+0000 7f1104400700 1 -- 172.21.15.135:0/4114438414 shutdown_connections 2024-03-30T04:23:21.151 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:21.153+0000 7f1104400700 1 --2- 172.21.15.135:0/4114438414 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f10e40692b0 0x7f10e406b760 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:21.151 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:21.153+0000 7f1104400700 1 --2- 172.21.15.135:0/4114438414 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10fc0fce70 0x7f10fc10d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:21.151 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:21.153+0000 7f1104400700 1 --2- 172.21.15.135:0/4114438414 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10fc0fe070 0x7f10fc10dd70 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:21.151 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:21.153+0000 7f1104400700 1 -- 172.21.15.135:0/4114438414 >> 172.21.15.135:0/4114438414 conn(0x7f10fc0f8540 msgr2=0x7f10fc0f9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:23:21.151 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:21.153+0000 7f1104400700 1 -- 172.21.15.135:0/4114438414 shutdown_connections 2024-03-30T04:23:21.151 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-30T04:23:21.153+0000 7f1104400700 1 -- 172.21.15.135:0/4114438414 wait complete. 2024-03-30T04:23:21.872 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-30T04:23:21.872 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-30T04:23:21.872 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-30T04:23:21.966 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-30T04:23:21.966 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-30T04:23:21.966 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph mgr dump --format=json 2024-03-30T04:23:22.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:21 smithi135 ceph-mon[44364]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:23:22.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:21 smithi135 ceph-mon[44364]: from='client.? 172.21.15.135:0/4114438414' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-30T04:23:22.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:21 smithi135 ceph-mon[44364]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-30T04:23:22.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:21 smithi135 ceph-mon[44364]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-30T04:23:22.398 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:23:22.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:21 smithi123 ceph-mon[35245]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:23:22.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:21 smithi123 ceph-mon[35245]: from='client.? 172.21.15.135:0/4114438414' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-30T04:23:22.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:21 smithi123 ceph-mon[35245]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-30T04:23:22.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:21 smithi123 ceph-mon[35245]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-30T04:23:24.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:24.183+0000 7f9a01000700 1 -- 172.21.15.123:0/3400737887 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99fc102e00 msgr2=0x7f99fc1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:24.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:24.183+0000 7f9a01000700 1 --2- 172.21.15.123:0/3400737887 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99fc102e00 0x7f99fc1031d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f99ec004940 tx=0x7f99ec009e30 comp rx=0 tx=0).stop 2024-03-30T04:23:24.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:24.183+0000 7f9a01000700 1 -- 172.21.15.123:0/3400737887 shutdown_connections 2024-03-30T04:23:24.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:24.183+0000 7f9a01000700 1 --2- 172.21.15.123:0/3400737887 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f99fc1037a0 0x7f99fc0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:24.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:24.183+0000 7f9a01000700 1 --2- 172.21.15.123:0/3400737887 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99fc102e00 0x7f99fc1031d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:24.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:24.183+0000 7f9a01000700 1 -- 172.21.15.123:0/3400737887 >> 172.21.15.123:0/3400737887 conn(0x7f99fc0f89b0 msgr2=0x7f99fc0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:23:24.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:24.183+0000 7f9a01000700 1 -- 172.21.15.123:0/3400737887 shutdown_connections 2024-03-30T04:23:24.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:24.183+0000 7f9a01000700 1 -- 172.21.15.123:0/3400737887 wait complete. 2024-03-30T04:23:24.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:24.184+0000 7f9a01000700 1 Processor -- start 2024-03-30T04:23:24.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:24.184+0000 7f9a01000700 1 -- start start 2024-03-30T04:23:24.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:24.184+0000 7f9a01000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99fc102e00 0x7f99fc115d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:24.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:24.184+0000 7f9a01000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f99fc1037a0 0x7f99fc116250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:24.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:24.184+0000 7f9a01000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99fc1168a0 con 0x7f99fc102e00 2024-03-30T04:23:24.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:24.184+0000 7f9a01000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99fc10fc70 con 0x7f99fc1037a0 2024-03-30T04:23:24.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:24.184+0000 7f99f9e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99fc102e00 0x7f99fc115d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:24.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:24.184+0000 7f99f9e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99fc102e00 0x7f99fc115d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36272/0 (socket says 172.21.15.123:36272) 2024-03-30T04:23:24.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:24.184+0000 7f99f9e00700 1 -- 172.21.15.123:0/2637924136 learned_addr learned my addr 172.21.15.123:0/2637924136 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:23:24.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:24.185+0000 7f99f9400700 1 --2- 172.21.15.123:0/2637924136 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f99fc1037a0 0x7f99fc116250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:24.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:24.185+0000 7f99f9e00700 1 -- 172.21.15.123:0/2637924136 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f99fc1037a0 msgr2=0x7f99fc116250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:24.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:24.185+0000 7f99f9e00700 1 --2- 172.21.15.123:0/2637924136 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f99fc1037a0 0x7f99fc116250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:24.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:24.185+0000 7f99f9e00700 1 -- 172.21.15.123:0/2637924136 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99ec005040 con 0x7f99fc102e00 2024-03-30T04:23:24.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:24.185+0000 7f99f9e00700 1 --2- 172.21.15.123:0/2637924136 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99fc102e00 0x7f99fc115d10 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f99ec009390 tx=0x7f99ec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:23:24.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:24.185+0000 7f99f2000700 1 -- 172.21.15.123:0/2637924136 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99ec00e040 con 0x7f99fc102e00 2024-03-30T04:23:24.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:24.185+0000 7f99f2000700 1 -- 172.21.15.123:0/2637924136 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f99ec013420 con 0x7f99fc102e00 2024-03-30T04:23:24.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:24.185+0000 7f9a01000700 1 -- 172.21.15.123:0/2637924136 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99fc10fef0 con 0x7f99fc102e00 2024-03-30T04:23:24.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:24.185+0000 7f99f2000700 1 -- 172.21.15.123:0/2637924136 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99ec01c630 con 0x7f99fc102e00 2024-03-30T04:23:24.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:24.185+0000 7f9a01000700 1 -- 172.21.15.123:0/2637924136 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99fc1104b0 con 0x7f99fc102e00 2024-03-30T04:23:24.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:24.186+0000 7f9a01000700 1 -- 172.21.15.123:0/2637924136 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99fc0fe550 con 0x7f99fc102e00 2024-03-30T04:23:24.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:24.187+0000 7f99f2000700 1 -- 172.21.15.123:0/2637924136 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f99ec0139e0 con 0x7f99fc102e00 2024-03-30T04:23:24.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:24.188+0000 7f99f2000700 1 --2- 172.21.15.123:0/2637924136 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f99e00692b0 0x7f99e006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:24.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:24.188+0000 7f99f2000700 1 -- 172.21.15.123:0/2637924136 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f99ec088120 con 0x7f99fc102e00 2024-03-30T04:23:24.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:24.188+0000 7f99f9400700 1 --2- 172.21.15.123:0/2637924136 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f99e00692b0 0x7f99e006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:24.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:24.189+0000 7f99f9400700 1 --2- 172.21.15.123:0/2637924136 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f99e00692b0 0x7f99e006b760 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f99fc1115b0 tx=0x7f99e8009ee0 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:23:24.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:24.193+0000 7f99f2000700 1 -- 172.21.15.123:0/2637924136 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f99ec058230 con 0x7f99fc102e00 2024-03-30T04:23:24.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:23 smithi123 ceph-mon[35245]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:23:24.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:23 smithi135 ceph-mon[44364]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:23:24.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:24.447+0000 7f9a01000700 1 -- 172.21.15.123:0/2637924136 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f99fc04bed0 con 0x7f99fc102e00 2024-03-30T04:23:24.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:24.450+0000 7f99f2000700 1 -- 172.21.15.123:0/2637924136 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+167176 (secure 0 0 0) 0x7f99ec057dc0 con 0x7f99fc102e00 2024-03-30T04:23:24.451 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:23:24.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:24.453+0000 7f9a01000700 1 -- 172.21.15.123:0/2637924136 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f99e00692b0 msgr2=0x7f99e006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:24.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:24.453+0000 7f9a01000700 1 --2- 172.21.15.123:0/2637924136 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f99e00692b0 0x7f99e006b760 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f99fc1115b0 tx=0x7f99e8009ee0 comp rx=0 tx=0).stop 2024-03-30T04:23:24.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:24.453+0000 7f9a01000700 1 -- 172.21.15.123:0/2637924136 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99fc102e00 msgr2=0x7f99fc115d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:24.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:24.453+0000 7f9a01000700 1 --2- 172.21.15.123:0/2637924136 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99fc102e00 0x7f99fc115d10 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f99ec009390 tx=0x7f99ec009e30 comp rx=0 tx=0).stop 2024-03-30T04:23:24.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:24.453+0000 7f9a01000700 1 -- 172.21.15.123:0/2637924136 shutdown_connections 2024-03-30T04:23:24.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:24.453+0000 7f9a01000700 1 --2- 172.21.15.123:0/2637924136 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f99e00692b0 0x7f99e006b760 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:24.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:24.453+0000 7f9a01000700 1 --2- 172.21.15.123:0/2637924136 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99fc102e00 0x7f99fc115d10 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:24.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:24.453+0000 7f9a01000700 1 --2- 172.21.15.123:0/2637924136 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f99fc1037a0 0x7f99fc116250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:24.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:24.453+0000 7f9a01000700 1 -- 172.21.15.123:0/2637924136 >> 172.21.15.123:0/2637924136 conn(0x7f99fc0f89b0 msgr2=0x7f99fc0704f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:23:24.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:24.453+0000 7f9a01000700 1 -- 172.21.15.123:0/2637924136 shutdown_connections 2024-03-30T04:23:24.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:24.453+0000 7f9a01000700 1 -- 172.21.15.123:0/2637924136 wait complete. 2024-03-30T04:23:25.150 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:24 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/2637924136' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-30T04:23:25.172 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":19,"active_gid":14206,"active_name":"smithi123.xgksqq","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":1564935593},{"type":"v1","addr":"172.21.15.123:6801","nonce":1564935593}]},"active_addr":"172.21.15.123:6801/1564935593","active_change":"2024-03-30T04:18:56.235854+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":14230,"name":"smithi135.kyblli","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.123:8443/","prometheus":"http://172.21.15.123:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.123:0","nonce":2447404972}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.123:0","nonce":72410915}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.123:0","nonce":4218181674}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.123:0","nonce":1064275350}]}]} 2024-03-30T04:23:25.175 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-30T04:23:25.175 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-30T04:23:25.176 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph osd dump --format=json 2024-03-30T04:23:25.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:24 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/2637924136' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-30T04:23:25.597 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:23:26.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:25 smithi123 ceph-mon[35245]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:23:26.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:25 smithi135 ceph-mon[44364]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:23:27.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:27 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:23:27.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:27.361+0000 7f67d0400700 1 -- 172.21.15.123:0/3654554003 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67c8070c10 msgr2=0x7f67c8109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:27.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:27.361+0000 7f67d0400700 1 --2- 172.21.15.123:0/3654554003 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67c8070c10 0x7f67c8109190 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f67b4004970 tx=0x7f67b4009e30 comp rx=0 tx=0).stop 2024-03-30T04:23:27.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:27.361+0000 7f67d0400700 1 -- 172.21.15.123:0/3654554003 shutdown_connections 2024-03-30T04:23:27.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:27.361+0000 7f67d0400700 1 --2- 172.21.15.123:0/3654554003 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67c8070c10 0x7f67c8109190 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:27.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:27.361+0000 7f67d0400700 1 --2- 172.21.15.123:0/3654554003 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f67c8070300 0x7f67c80706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:27.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:27.361+0000 7f67d0400700 1 -- 172.21.15.123:0/3654554003 >> 172.21.15.123:0/3654554003 conn(0x7f67c80f89d0 msgr2=0x7f67c80fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:23:27.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:27.361+0000 7f67d0400700 1 -- 172.21.15.123:0/3654554003 shutdown_connections 2024-03-30T04:23:27.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:27.361+0000 7f67d0400700 1 -- 172.21.15.123:0/3654554003 wait complete. 2024-03-30T04:23:27.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:27.362+0000 7f67d0400700 1 Processor -- start 2024-03-30T04:23:27.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:27.362+0000 7f67d0400700 1 -- start start 2024-03-30T04:23:27.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:27.362+0000 7f67d0400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f67c8070300 0x7f67c8115ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:27.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:27.362+0000 7f67d0400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67c8070c10 0x7f67c81161e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:27.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:27.362+0000 7f67d0400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c8116830 con 0x7f67c8070c10 2024-03-30T04:23:27.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:27.362+0000 7f67d0400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c810fc00 con 0x7f67c8070300 2024-03-30T04:23:27.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:27.363+0000 7f67cd000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67c8070c10 0x7f67c81161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:27.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:27.363+0000 7f67cd000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67c8070c10 0x7f67c81161e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36290/0 (socket says 172.21.15.123:36290) 2024-03-30T04:23:27.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:27.363+0000 7f67cd000700 1 -- 172.21.15.123:0/1297734855 learned_addr learned my addr 172.21.15.123:0/1297734855 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:23:27.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:27.363+0000 7f67cda00700 1 --2- 172.21.15.123:0/1297734855 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f67c8070300 0x7f67c8115ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:27.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:27.363+0000 7f67cd000700 1 -- 172.21.15.123:0/1297734855 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f67c8070300 msgr2=0x7f67c8115ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:27.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:27.363+0000 7f67cd000700 1 --2- 172.21.15.123:0/1297734855 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f67c8070300 0x7f67c8115ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:27.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:27.363+0000 7f67cd000700 1 -- 172.21.15.123:0/1297734855 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67b4005040 con 0x7f67c8070c10 2024-03-30T04:23:27.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:27.363+0000 7f67cd000700 1 --2- 172.21.15.123:0/1297734855 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67c8070c10 0x7f67c81161e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f67b4009360 tx=0x7f67b4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:23:27.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:27.363+0000 7f67c5600700 1 -- 172.21.15.123:0/1297734855 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f67b400e040 con 0x7f67c8070c10 2024-03-30T04:23:27.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:27.364+0000 7f67c5600700 1 -- 172.21.15.123:0/1297734855 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f67b4004290 con 0x7f67c8070c10 2024-03-30T04:23:27.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:27.364+0000 7f67c5600700 1 -- 172.21.15.123:0/1297734855 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f67b4013d90 con 0x7f67c8070c10 2024-03-30T04:23:27.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:27.364+0000 7f67d0400700 1 -- 172.21.15.123:0/1297734855 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67c810fe80 con 0x7f67c8070c10 2024-03-30T04:23:27.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:27.364+0000 7f67d0400700 1 -- 172.21.15.123:0/1297734855 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67c81103c0 con 0x7f67c8070c10 2024-03-30T04:23:27.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:27.365+0000 7f67d0400700 1 -- 172.21.15.123:0/1297734855 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67c8071a40 con 0x7f67c8070c10 2024-03-30T04:23:27.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:27.366+0000 7f67c5600700 1 -- 172.21.15.123:0/1297734855 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f67b400ba40 con 0x7f67c8070c10 2024-03-30T04:23:27.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:27.366+0000 7f67c5600700 1 --2- 172.21.15.123:0/1297734855 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f67b00692b0 0x7f67b006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:27.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:27.367+0000 7f67c5600700 1 -- 172.21.15.123:0/1297734855 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f67b4088140 con 0x7f67c8070c10 2024-03-30T04:23:27.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:27.367+0000 7f67cda00700 1 --2- 172.21.15.123:0/1297734855 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f67b00692b0 0x7f67b006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:27.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:27.367+0000 7f67cda00700 1 --2- 172.21.15.123:0/1297734855 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f67b00692b0 0x7f67b006b760 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f67bc004f40 tx=0x7f67bc00b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:23:27.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:27.371+0000 7f67c5600700 1 -- 172.21.15.123:0/1297734855 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f67b40581a0 con 0x7f67c8070c10 2024-03-30T04:23:27.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:27 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:23:27.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:27.567+0000 7f67d0400700 1 -- 172.21.15.123:0/1297734855 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f67c805f5f0 con 0x7f67c8070c10 2024-03-30T04:23:27.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:27.568+0000 7f67c5600700 1 -- 172.21.15.123:0/1297734855 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11141 (secure 0 0 0) 0x7f67b400bcf0 con 0x7f67c8070c10 2024-03-30T04:23:27.567 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:23:27.567 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":40,"fsid":"2e77a2be-ee4c-11ee-b647-cb9ed24678a4","created":"2024-03-30T04:16:30.949442+0000","modified":"2024-03-30T04:23:07.209822+0000","last_up_change":"2024-03-30T04:23:06.203543+0000","last_in_change":"2024-03-30T04:22:44.504478+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T04:21:34.842970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"3968446b-8696-4eec-b640-c97973ca054d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":1768008525},{"type":"v1","addr":"172.21.15.123:6803","nonce":1768008525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":1768008525},{"type":"v1","addr":"172.21.15.123:6805","nonce":1768008525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1768008525},{"type":"v1","addr":"172.21.15.123:6809","nonce":1768008525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":1768008525},{"type":"v1","addr":"172.21.15.123:6807","nonce":1768008525}]},"public_addr":"172.21.15.123:6803/1768008525","cluster_addr":"172.21.15.123:6805/1768008525","heartbeat_back_addr":"172.21.15.123:6809/1768008525","heartbeat_front_addr":"172.21.15.123:6807/1768008525","state":["exists","up"]},{"osd":1,"uuid":"336be34b-bb3c-46ab-8439-af04a1366db5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1040012134},{"type":"v1","addr":"172.21.15.123:6811","nonce":1040012134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1040012134},{"type":"v1","addr":"172.21.15.123:6813","nonce":1040012134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1040012134},{"type":"v1","addr":"172.21.15.123:6817","nonce":1040012134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1040012134},{"type":"v1","addr":"172.21.15.123:6815","nonce":1040012134}]},"public_addr":"172.21.15.123:6811/1040012134","cluster_addr":"172.21.15.123:6813/1040012134","heartbeat_back_addr":"172.21.15.123:6817/1040012134","heartbeat_front_addr":"172.21.15.123:6815/1040012134","state":["exists","up"]},{"osd":2,"uuid":"4386fc8a-e9ed-413e-bd3a-bb84a0e62827","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1260172805},{"type":"v1","addr":"172.21.15.123:6819","nonce":1260172805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1260172805},{"type":"v1","addr":"172.21.15.123:6821","nonce":1260172805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1260172805},{"type":"v1","addr":"172.21.15.123:6825","nonce":1260172805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1260172805},{"type":"v1","addr":"172.21.15.123:6823","nonce":1260172805}]},"public_addr":"172.21.15.123:6819/1260172805","cluster_addr":"172.21.15.123:6821/1260172805","heartbeat_back_addr":"172.21.15.123:6825/1260172805","heartbeat_front_addr":"172.21.15.123:6823/1260172805","state":["exists","up"]},{"osd":3,"uuid":"086d86d4-ff1a-47dd-8416-447ccf9064ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2906534885},{"type":"v1","addr":"172.21.15.135:6801","nonce":2906534885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2906534885},{"type":"v1","addr":"172.21.15.135:6803","nonce":2906534885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2906534885},{"type":"v1","addr":"172.21.15.135:6807","nonce":2906534885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2906534885},{"type":"v1","addr":"172.21.15.135:6805","nonce":2906534885}]},"public_addr":"172.21.15.135:6801/2906534885","cluster_addr":"172.21.15.135:6803/2906534885","heartbeat_back_addr":"172.21.15.135:6807/2906534885","heartbeat_front_addr":"172.21.15.135:6805/2906534885","state":["exists","up"]},{"osd":4,"uuid":"31868cec-dd4d-44dd-a150-7ebdbd61ea4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":1699881600},{"type":"v1","addr":"172.21.15.135:6809","nonce":1699881600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":1699881600},{"type":"v1","addr":"172.21.15.135:6811","nonce":1699881600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":1699881600},{"type":"v1","addr":"172.21.15.135:6815","nonce":1699881600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":1699881600},{"type":"v1","addr":"172.21.15.135:6813","nonce":1699881600}]},"public_addr":"172.21.15.135:6809/1699881600","cluster_addr":"172.21.15.135:6811/1699881600","heartbeat_back_addr":"172.21.15.135:6815/1699881600","heartbeat_front_addr":"172.21.15.135:6813/1699881600","state":["exists","up"]},{"osd":5,"uuid":"dc6e11b9-4729-4504-8917-51a3bb5e2b7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":1609344282},{"type":"v1","addr":"172.21.15.135:6817","nonce":1609344282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":1609344282},{"type":"v1","addr":"172.21.15.135:6819","nonce":1609344282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":1609344282},{"type":"v1","addr":"172.21.15.135:6823","nonce":1609344282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":1609344282},{"type":"v1","addr":"172.21.15.135:6821","nonce":1609344282}]},"public_addr":"172.21.15.135:6817/1609344282","cluster_addr":"172.21.15.135:6819/1609344282","heartbeat_back_addr":"172.21.15.135:6823/1609344282","heartbeat_front_addr":"172.21.15.135:6821/1609344282","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T04:20:25.138939+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T04:20:58.657719+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T04:21:32.397235+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T04:22:02.245820+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T04:22:33.842250+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T04:23:04.503576+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.123:6800/70490328":"2024-03-31T04:18:56.235750+0000","172.21.15.123:6801/70490328":"2024-03-31T04:18:56.235750+0000","172.21.15.123:0/3593359364":"2024-03-31T04:18:56.235750+0000","172.21.15.123:6801/1776532690":"2024-03-31T04:17:41.459573+0000","172.21.15.123:0/1275706223":"2024-03-31T04:18:56.235750+0000","172.21.15.123:6800/1776532690":"2024-03-31T04:17:41.459573+0000","172.21.15.123:0/133647565":"2024-03-31T04:17:41.459573+0000","172.21.15.123:0/2366611723":"2024-03-31T04:17:05.107043+0000","172.21.15.123:0/518566709":"2024-03-31T04:18:56.235750+0000","172.21.15.123:6801/1291592055":"2024-03-31T04:17:05.107043+0000","172.21.15.123:0/2358069559":"2024-03-31T04:17:41.459573+0000","172.21.15.123:0/250978570":"2024-03-31T04:17:05.107043+0000","172.21.15.123:0/3037148222":"2024-03-31T04:17:41.459573+0000","172.21.15.123:0/2756071515":"2024-03-31T04:17:05.107043+0000","172.21.15.123:6800/1291592055":"2024-03-31T04:17:05.107043+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T04:23:27.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:27.571+0000 7f67d0400700 1 -- 172.21.15.123:0/1297734855 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f67b00692b0 msgr2=0x7f67b006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:27.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:27.571+0000 7f67d0400700 1 --2- 172.21.15.123:0/1297734855 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f67b00692b0 0x7f67b006b760 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f67bc004f40 tx=0x7f67bc00b040 comp rx=0 tx=0).stop 2024-03-30T04:23:27.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:27.571+0000 7f67d0400700 1 -- 172.21.15.123:0/1297734855 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67c8070c10 msgr2=0x7f67c81161e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:27.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:27.571+0000 7f67d0400700 1 --2- 172.21.15.123:0/1297734855 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67c8070c10 0x7f67c81161e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f67b4009360 tx=0x7f67b4009e30 comp rx=0 tx=0).stop 2024-03-30T04:23:27.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:27.571+0000 7f67d0400700 1 -- 172.21.15.123:0/1297734855 shutdown_connections 2024-03-30T04:23:27.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:27.571+0000 7f67d0400700 1 --2- 172.21.15.123:0/1297734855 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f67b00692b0 0x7f67b006b760 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:27.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:27.571+0000 7f67d0400700 1 --2- 172.21.15.123:0/1297734855 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f67c8070300 0x7f67c8115ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:27.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:27.571+0000 7f67d0400700 1 --2- 172.21.15.123:0/1297734855 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67c8070c10 0x7f67c81161e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:27.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:27.571+0000 7f67d0400700 1 -- 172.21.15.123:0/1297734855 >> 172.21.15.123:0/1297734855 conn(0x7f67c80f89d0 msgr2=0x7f67c80f9f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:23:27.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:27.571+0000 7f67d0400700 1 -- 172.21.15.123:0/1297734855 shutdown_connections 2024-03-30T04:23:27.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:27.571+0000 7f67d0400700 1 -- 172.21.15.123:0/1297734855 wait complete. 2024-03-30T04:23:28.129 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:28 smithi123 ceph-mon[35245]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:23:28.129 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:28 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/1297734855' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:23:28.153 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-30T04:23:28.154 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph osd dump --format=json 2024-03-30T04:23:28.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:28 smithi135 ceph-mon[44364]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:23:28.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:28 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/1297734855' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:23:28.577 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:23:30.140 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:30 smithi123 ceph-mon[35245]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:23:30.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:30 smithi135 ceph-mon[44364]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:23:30.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:30.417+0000 7f5f4f600700 1 -- 172.21.15.123:0/1775184493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f480708a0 msgr2=0x7f5f48070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:30.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:30.417+0000 7f5f4f600700 1 --2- 172.21.15.123:0/1775184493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f480708a0 0x7f5f48070d10 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5f34004970 tx=0x7f5f34009e30 comp rx=0 tx=0).stop 2024-03-30T04:23:30.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:30.417+0000 7f5f4f600700 1 -- 172.21.15.123:0/1775184493 shutdown_connections 2024-03-30T04:23:30.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:30.417+0000 7f5f4f600700 1 --2- 172.21.15.123:0/1775184493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f480708a0 0x7f5f48070d10 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:30.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:30.417+0000 7f5f4f600700 1 --2- 172.21.15.123:0/1775184493 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5f48071f90 0x7f5f48070360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:30.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:30.417+0000 7f5f4f600700 1 -- 172.21.15.123:0/1775184493 >> 172.21.15.123:0/1775184493 conn(0x7f5f480f89b0 msgr2=0x7f5f480fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:23:30.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:30.417+0000 7f5f4f600700 1 -- 172.21.15.123:0/1775184493 shutdown_connections 2024-03-30T04:23:30.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:30.417+0000 7f5f4f600700 1 -- 172.21.15.123:0/1775184493 wait complete. 2024-03-30T04:23:30.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:30.418+0000 7f5f4f600700 1 Processor -- start 2024-03-30T04:23:30.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:30.418+0000 7f5f4f600700 1 -- start start 2024-03-30T04:23:30.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:30.418+0000 7f5f4f600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f480708a0 0x7f5f4811a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:30.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:30.418+0000 7f5f4f600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5f48071f90 0x7f5f4811a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:30.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:30.418+0000 7f5f4f600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4811ac40 con 0x7f5f480708a0 2024-03-30T04:23:30.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:30.418+0000 7f5f4f600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f48114010 con 0x7f5f48071f90 2024-03-30T04:23:30.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:30.419+0000 7f5f4cc00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f480708a0 0x7f5f4811a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:30.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:30.419+0000 7f5f4cc00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f480708a0 0x7f5f4811a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36310/0 (socket says 172.21.15.123:36310) 2024-03-30T04:23:30.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:30.419+0000 7f5f4cc00700 1 -- 172.21.15.123:0/3368749145 learned_addr learned my addr 172.21.15.123:0/3368749145 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:23:30.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:30.419+0000 7f5f4cc00700 1 -- 172.21.15.123:0/3368749145 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5f48071f90 msgr2=0x7f5f4811a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:30.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:30.419+0000 7f5f4cc00700 1 --2- 172.21.15.123:0/3368749145 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5f48071f90 0x7f5f4811a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:30.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:30.419+0000 7f5f4cc00700 1 -- 172.21.15.123:0/3368749145 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f34005040 con 0x7f5f480708a0 2024-03-30T04:23:30.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:30.419+0000 7f5f4cc00700 1 --2- 172.21.15.123:0/3368749145 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f480708a0 0x7f5f4811a0b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c002a50 tx=0x7f5f3c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:23:30.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:30.419+0000 7f5f44c00700 1 -- 172.21.15.123:0/3368749145 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5f3c007cc0 con 0x7f5f480708a0 2024-03-30T04:23:30.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:30.419+0000 7f5f44c00700 1 -- 172.21.15.123:0/3368749145 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5f3c00f040 con 0x7f5f480708a0 2024-03-30T04:23:30.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:30.419+0000 7f5f4f600700 1 -- 172.21.15.123:0/3368749145 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f481142f0 con 0x7f5f480708a0 2024-03-30T04:23:30.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:30.419+0000 7f5f44c00700 1 -- 172.21.15.123:0/3368749145 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5f3c009540 con 0x7f5f480708a0 2024-03-30T04:23:30.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:30.420+0000 7f5f4f600700 1 -- 172.21.15.123:0/3368749145 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f48114830 con 0x7f5f480708a0 2024-03-30T04:23:30.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:30.421+0000 7f5f4f600700 1 -- 172.21.15.123:0/3368749145 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f48071aa0 con 0x7f5f480708a0 2024-03-30T04:23:30.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:30.422+0000 7f5f44c00700 1 -- 172.21.15.123:0/3368749145 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f5f3c0096a0 con 0x7f5f480708a0 2024-03-30T04:23:30.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:30.422+0000 7f5f44c00700 1 --2- 172.21.15.123:0/3368749145 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f5f300692a0 0x7f5f3006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:30.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:30.423+0000 7f5f44c00700 1 -- 172.21.15.123:0/3368749145 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f5f3c086c30 con 0x7f5f480708a0 2024-03-30T04:23:30.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:30.423+0000 7f5f47e00700 1 --2- 172.21.15.123:0/3368749145 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f5f300692a0 0x7f5f3006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:30.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:30.423+0000 7f5f47e00700 1 --2- 172.21.15.123:0/3368749145 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f5f300692a0 0x7f5f3006b750 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5f34009fd0 tx=0x7f5f34009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:23:30.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:30.427+0000 7f5f44c00700 1 -- 172.21.15.123:0/3368749145 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f5f3c056c90 con 0x7f5f480708a0 2024-03-30T04:23:30.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:30.622+0000 7f5f4f600700 1 -- 172.21.15.123:0/3368749145 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5f4804bed0 con 0x7f5f480708a0 2024-03-30T04:23:30.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:30.623+0000 7f5f44c00700 1 -- 172.21.15.123:0/3368749145 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11141 (secure 0 0 0) 0x7f5f3c056820 con 0x7f5f480708a0 2024-03-30T04:23:30.624 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:23:30.624 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":40,"fsid":"2e77a2be-ee4c-11ee-b647-cb9ed24678a4","created":"2024-03-30T04:16:30.949442+0000","modified":"2024-03-30T04:23:07.209822+0000","last_up_change":"2024-03-30T04:23:06.203543+0000","last_in_change":"2024-03-30T04:22:44.504478+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T04:21:34.842970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"3968446b-8696-4eec-b640-c97973ca054d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":1768008525},{"type":"v1","addr":"172.21.15.123:6803","nonce":1768008525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":1768008525},{"type":"v1","addr":"172.21.15.123:6805","nonce":1768008525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1768008525},{"type":"v1","addr":"172.21.15.123:6809","nonce":1768008525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":1768008525},{"type":"v1","addr":"172.21.15.123:6807","nonce":1768008525}]},"public_addr":"172.21.15.123:6803/1768008525","cluster_addr":"172.21.15.123:6805/1768008525","heartbeat_back_addr":"172.21.15.123:6809/1768008525","heartbeat_front_addr":"172.21.15.123:6807/1768008525","state":["exists","up"]},{"osd":1,"uuid":"336be34b-bb3c-46ab-8439-af04a1366db5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1040012134},{"type":"v1","addr":"172.21.15.123:6811","nonce":1040012134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1040012134},{"type":"v1","addr":"172.21.15.123:6813","nonce":1040012134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1040012134},{"type":"v1","addr":"172.21.15.123:6817","nonce":1040012134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1040012134},{"type":"v1","addr":"172.21.15.123:6815","nonce":1040012134}]},"public_addr":"172.21.15.123:6811/1040012134","cluster_addr":"172.21.15.123:6813/1040012134","heartbeat_back_addr":"172.21.15.123:6817/1040012134","heartbeat_front_addr":"172.21.15.123:6815/1040012134","state":["exists","up"]},{"osd":2,"uuid":"4386fc8a-e9ed-413e-bd3a-bb84a0e62827","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1260172805},{"type":"v1","addr":"172.21.15.123:6819","nonce":1260172805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1260172805},{"type":"v1","addr":"172.21.15.123:6821","nonce":1260172805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1260172805},{"type":"v1","addr":"172.21.15.123:6825","nonce":1260172805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1260172805},{"type":"v1","addr":"172.21.15.123:6823","nonce":1260172805}]},"public_addr":"172.21.15.123:6819/1260172805","cluster_addr":"172.21.15.123:6821/1260172805","heartbeat_back_addr":"172.21.15.123:6825/1260172805","heartbeat_front_addr":"172.21.15.123:6823/1260172805","state":["exists","up"]},{"osd":3,"uuid":"086d86d4-ff1a-47dd-8416-447ccf9064ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2906534885},{"type":"v1","addr":"172.21.15.135:6801","nonce":2906534885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2906534885},{"type":"v1","addr":"172.21.15.135:6803","nonce":2906534885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2906534885},{"type":"v1","addr":"172.21.15.135:6807","nonce":2906534885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2906534885},{"type":"v1","addr":"172.21.15.135:6805","nonce":2906534885}]},"public_addr":"172.21.15.135:6801/2906534885","cluster_addr":"172.21.15.135:6803/2906534885","heartbeat_back_addr":"172.21.15.135:6807/2906534885","heartbeat_front_addr":"172.21.15.135:6805/2906534885","state":["exists","up"]},{"osd":4,"uuid":"31868cec-dd4d-44dd-a150-7ebdbd61ea4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":1699881600},{"type":"v1","addr":"172.21.15.135:6809","nonce":1699881600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":1699881600},{"type":"v1","addr":"172.21.15.135:6811","nonce":1699881600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":1699881600},{"type":"v1","addr":"172.21.15.135:6815","nonce":1699881600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":1699881600},{"type":"v1","addr":"172.21.15.135:6813","nonce":1699881600}]},"public_addr":"172.21.15.135:6809/1699881600","cluster_addr":"172.21.15.135:6811/1699881600","heartbeat_back_addr":"172.21.15.135:6815/1699881600","heartbeat_front_addr":"172.21.15.135:6813/1699881600","state":["exists","up"]},{"osd":5,"uuid":"dc6e11b9-4729-4504-8917-51a3bb5e2b7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":1609344282},{"type":"v1","addr":"172.21.15.135:6817","nonce":1609344282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":1609344282},{"type":"v1","addr":"172.21.15.135:6819","nonce":1609344282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":1609344282},{"type":"v1","addr":"172.21.15.135:6823","nonce":1609344282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":1609344282},{"type":"v1","addr":"172.21.15.135:6821","nonce":1609344282}]},"public_addr":"172.21.15.135:6817/1609344282","cluster_addr":"172.21.15.135:6819/1609344282","heartbeat_back_addr":"172.21.15.135:6823/1609344282","heartbeat_front_addr":"172.21.15.135:6821/1609344282","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T04:20:25.138939+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T04:20:58.657719+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T04:21:32.397235+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T04:22:02.245820+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T04:22:33.842250+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T04:23:04.503576+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.123:6800/70490328":"2024-03-31T04:18:56.235750+0000","172.21.15.123:6801/70490328":"2024-03-31T04:18:56.235750+0000","172.21.15.123:0/3593359364":"2024-03-31T04:18:56.235750+0000","172.21.15.123:6801/1776532690":"2024-03-31T04:17:41.459573+0000","172.21.15.123:0/1275706223":"2024-03-31T04:18:56.235750+0000","172.21.15.123:6800/1776532690":"2024-03-31T04:17:41.459573+0000","172.21.15.123:0/133647565":"2024-03-31T04:17:41.459573+0000","172.21.15.123:0/2366611723":"2024-03-31T04:17:05.107043+0000","172.21.15.123:0/518566709":"2024-03-31T04:18:56.235750+0000","172.21.15.123:6801/1291592055":"2024-03-31T04:17:05.107043+0000","172.21.15.123:0/2358069559":"2024-03-31T04:17:41.459573+0000","172.21.15.123:0/250978570":"2024-03-31T04:17:05.107043+0000","172.21.15.123:0/3037148222":"2024-03-31T04:17:41.459573+0000","172.21.15.123:0/2756071515":"2024-03-31T04:17:05.107043+0000","172.21.15.123:6800/1291592055":"2024-03-31T04:17:05.107043+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T04:23:30.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:30.626+0000 7f5f4f600700 1 -- 172.21.15.123:0/3368749145 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f5f300692a0 msgr2=0x7f5f3006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:30.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:30.626+0000 7f5f4f600700 1 --2- 172.21.15.123:0/3368749145 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f5f300692a0 0x7f5f3006b750 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5f34009fd0 tx=0x7f5f34009e30 comp rx=0 tx=0).stop 2024-03-30T04:23:30.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:30.626+0000 7f5f4f600700 1 -- 172.21.15.123:0/3368749145 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f480708a0 msgr2=0x7f5f4811a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:30.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:30.626+0000 7f5f4f600700 1 --2- 172.21.15.123:0/3368749145 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f480708a0 0x7f5f4811a0b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c002a50 tx=0x7f5f3c0046b0 comp rx=0 tx=0).stop 2024-03-30T04:23:30.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:30.626+0000 7f5f4f600700 1 -- 172.21.15.123:0/3368749145 shutdown_connections 2024-03-30T04:23:30.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:30.626+0000 7f5f4f600700 1 --2- 172.21.15.123:0/3368749145 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f5f300692a0 0x7f5f3006b750 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:30.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:30.626+0000 7f5f4f600700 1 --2- 172.21.15.123:0/3368749145 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f480708a0 0x7f5f4811a0b0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:30.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:30.626+0000 7f5f4f600700 1 --2- 172.21.15.123:0/3368749145 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5f48071f90 0x7f5f4811a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:30.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:30.626+0000 7f5f4f600700 1 -- 172.21.15.123:0/3368749145 >> 172.21.15.123:0/3368749145 conn(0x7f5f480f89b0 msgr2=0x7f5f480f9b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:23:30.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:30.626+0000 7f5f4f600700 1 -- 172.21.15.123:0/3368749145 shutdown_connections 2024-03-30T04:23:30.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:30.626+0000 7f5f4f600700 1 -- 172.21.15.123:0/3368749145 wait complete. 2024-03-30T04:23:31.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:31 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/3368749145' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:23:31.417 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:31 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/3368749145' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:23:31.443 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph tell osd.0 flush_pg_stats 2024-03-30T04:23:31.444 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph tell osd.1 flush_pg_stats 2024-03-30T04:23:31.445 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph tell osd.2 flush_pg_stats 2024-03-30T04:23:31.445 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph tell osd.3 flush_pg_stats 2024-03-30T04:23:31.446 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph tell osd.4 flush_pg_stats 2024-03-30T04:23:31.447 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph tell osd.5 flush_pg_stats 2024-03-30T04:23:32.011 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:23:32.020 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:23:32.313 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:32 smithi123 ceph-mon[35245]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:23:32.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:32 smithi135 ceph-mon[44364]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:23:33.888 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:23:33.889 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:23:33.889 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:23:33.965 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:23:34.120 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:34 smithi123 ceph-mon[35245]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:23:34.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:34 smithi135 ceph-mon[44364]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:23:36.343 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:36 smithi123 ceph-mon[35245]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:23:36.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:36 smithi135 ceph-mon[44364]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:23:38.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:38 smithi135 ceph-mon[44364]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:23:38.419 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:38 smithi123 ceph-mon[35245]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:23:39.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.387+0000 7f094e400700 1 -- 172.21.15.123:0/837589085 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09481037a0 msgr2=0x7f09480fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:39.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.387+0000 7f094e400700 1 --2- 172.21.15.123:0/837589085 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09481037a0 0x7f09480fce10 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0934004940 tx=0x7f0934009e30 comp rx=0 tx=0).stop 2024-03-30T04:23:39.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.387+0000 7f094e400700 1 -- 172.21.15.123:0/837589085 shutdown_connections 2024-03-30T04:23:39.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.387+0000 7f094e400700 1 --2- 172.21.15.123:0/837589085 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09481037a0 0x7f09480fce10 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:39.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.387+0000 7f094e400700 1 --2- 172.21.15.123:0/837589085 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0948102e00 0x7f09481031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:39.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.387+0000 7f094e400700 1 -- 172.21.15.123:0/837589085 >> 172.21.15.123:0/837589085 conn(0x7f09480f89b0 msgr2=0x7f09480fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:23:39.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.387+0000 7f094e400700 1 -- 172.21.15.123:0/837589085 shutdown_connections 2024-03-30T04:23:39.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.387+0000 7f094e400700 1 -- 172.21.15.123:0/837589085 wait complete. 2024-03-30T04:23:39.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.388+0000 7f094e400700 1 Processor -- start 2024-03-30T04:23:39.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.388+0000 7f094e400700 1 -- start start 2024-03-30T04:23:39.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.389+0000 7f094e400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0948102e00 0x7f0948113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:39.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.389+0000 7f094e400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09481037a0 0x7f0948113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:39.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.389+0000 7f094e400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0948114600 con 0x7f09481037a0 2024-03-30T04:23:39.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.389+0000 7f094e400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f094810d9f0 con 0x7f0948102e00 2024-03-30T04:23:39.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.389+0000 7f0946a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09481037a0 0x7f0948113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:39.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.389+0000 7f0946a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09481037a0 0x7f0948113fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44702/0 (socket says 172.21.15.123:44702) 2024-03-30T04:23:39.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.389+0000 7f0946a00700 1 -- 172.21.15.123:0/1846822121 learned_addr learned my addr 172.21.15.123:0/1846822121 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:23:39.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.389+0000 7f0947400700 1 --2- 172.21.15.123:0/1846822121 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0948102e00 0x7f0948113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:39.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.389+0000 7f0946a00700 1 -- 172.21.15.123:0/1846822121 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0948102e00 msgr2=0x7f0948113a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:39.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.389+0000 7f0946a00700 1 --2- 172.21.15.123:0/1846822121 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0948102e00 0x7f0948113a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:39.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.389+0000 7f0946a00700 1 -- 172.21.15.123:0/1846822121 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0934005040 con 0x7f09481037a0 2024-03-30T04:23:39.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.390+0000 7f0946a00700 1 --2- 172.21.15.123:0/1846822121 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09481037a0 0x7f0948113fb0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0934008d50 tx=0x7f0934009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:23:39.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.390+0000 7f093f400700 1 -- 172.21.15.123:0/1846822121 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f093400e030 con 0x7f09481037a0 2024-03-30T04:23:39.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.390+0000 7f093f400700 1 -- 172.21.15.123:0/1846822121 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f09340093b0 con 0x7f09481037a0 2024-03-30T04:23:39.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.390+0000 7f093f400700 1 -- 172.21.15.123:0/1846822121 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f093401cbb0 con 0x7f09481037a0 2024-03-30T04:23:39.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.390+0000 7f094e400700 1 -- 172.21.15.123:0/1846822121 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f094810dc70 con 0x7f09481037a0 2024-03-30T04:23:39.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.390+0000 7f094e400700 1 -- 172.21.15.123:0/1846822121 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f094810e230 con 0x7f09481037a0 2024-03-30T04:23:39.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.391+0000 7f094e400700 1 -- 172.21.15.123:0/1846822121 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f094810e820 con 0x7f09481037a0 2024-03-30T04:23:39.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.392+0000 7f093f400700 1 -- 172.21.15.123:0/1846822121 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f0934013930 con 0x7f09481037a0 2024-03-30T04:23:39.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.393+0000 7f093f400700 1 --2- 172.21.15.123:0/1846822121 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f092c0692b0 0x7f092c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:39.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.393+0000 7f093f400700 1 -- 172.21.15.123:0/1846822121 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f09340887a0 con 0x7f09481037a0 2024-03-30T04:23:39.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.393+0000 7f0947400700 1 --2- 172.21.15.123:0/1846822121 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f092c0692b0 0x7f092c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:39.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.394+0000 7f093f400700 1 --2- 172.21.15.123:0/1846822121 >> [v2:172.21.15.123:6802/1768008525,v1:172.21.15.123:6803/1768008525] conn(0x7f092c06ef80 0x7f092c071390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:39.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.394+0000 7f093f400700 1 -- 172.21.15.123:0/1846822121 --> [v2:172.21.15.123:6802/1768008525,v1:172.21.15.123:6803/1768008525] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f092c071a40 con 0x7f092c06ef80 2024-03-30T04:23:39.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.394+0000 7f093f400700 1 -- 172.21.15.123:0/1846822121 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f093408acf0 con 0x7f09481037a0 2024-03-30T04:23:39.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.394+0000 7f0947e00700 1 --2- 172.21.15.123:0/1846822121 >> [v2:172.21.15.123:6802/1768008525,v1:172.21.15.123:6803/1768008525] conn(0x7f092c06ef80 0x7f092c071390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:39.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.394+0000 7f0947400700 1 --2- 172.21.15.123:0/1846822121 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f092c0692b0 0x7f092c06b760 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0938009b80 tx=0x7f093800b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:23:39.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.394+0000 7f0947e00700 1 --2- 172.21.15.123:0/1846822121 >> [v2:172.21.15.123:6802/1768008525,v1:172.21.15.123:6803/1768008525] conn(0x7f092c06ef80 0x7f092c071390 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:23:39.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.396+0000 7f093f400700 1 -- 172.21.15.123:0/1846822121 <== osd.0 v2:172.21.15.123:6802/1768008525 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f092c071a40 con 0x7f092c06ef80 2024-03-30T04:23:39.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.412+0000 7f094e400700 1 -- 172.21.15.123:0/1846822121 --> [v2:172.21.15.123:6802/1768008525,v1:172.21.15.123:6803/1768008525] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f09480fa3a0 con 0x7f092c06ef80 2024-03-30T04:23:39.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.412+0000 7f093f400700 1 -- 172.21.15.123:0/1846822121 <== osd.0 v2:172.21.15.123:6802/1768008525 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f09480fa3a0 con 0x7f092c06ef80 2024-03-30T04:23:39.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.413+0000 7f094e400700 1 -- 172.21.15.123:0/1846822121 >> [v2:172.21.15.123:6802/1768008525,v1:172.21.15.123:6803/1768008525] conn(0x7f092c06ef80 msgr2=0x7f092c071390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:39.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.413+0000 7f094e400700 1 --2- 172.21.15.123:0/1846822121 >> [v2:172.21.15.123:6802/1768008525,v1:172.21.15.123:6803/1768008525] conn(0x7f092c06ef80 0x7f092c071390 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:39.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.413+0000 7f094e400700 1 -- 172.21.15.123:0/1846822121 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f092c0692b0 msgr2=0x7f092c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:39.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.413+0000 7f094e400700 1 --2- 172.21.15.123:0/1846822121 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f092c0692b0 0x7f092c06b760 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0938009b80 tx=0x7f093800b040 comp rx=0 tx=0).stop 2024-03-30T04:23:39.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.413+0000 7f094e400700 1 -- 172.21.15.123:0/1846822121 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09481037a0 msgr2=0x7f0948113fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:39.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.413+0000 7f094e400700 1 --2- 172.21.15.123:0/1846822121 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09481037a0 0x7f0948113fb0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0934008d50 tx=0x7f0934009e30 comp rx=0 tx=0).stop 2024-03-30T04:23:39.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.413+0000 7f094e400700 1 -- 172.21.15.123:0/1846822121 shutdown_connections 2024-03-30T04:23:39.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.413+0000 7f094e400700 1 --2- 172.21.15.123:0/1846822121 >> [v2:172.21.15.123:6802/1768008525,v1:172.21.15.123:6803/1768008525] conn(0x7f092c06ef80 0x7f092c071390 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:39.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.413+0000 7f094e400700 1 --2- 172.21.15.123:0/1846822121 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f092c0692b0 0x7f092c06b760 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:39.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.413+0000 7f094e400700 1 --2- 172.21.15.123:0/1846822121 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0948102e00 0x7f0948113a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:39.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.413+0000 7f094e400700 1 --2- 172.21.15.123:0/1846822121 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09481037a0 0x7f0948113fb0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:39.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.413+0000 7f094e400700 1 -- 172.21.15.123:0/1846822121 >> 172.21.15.123:0/1846822121 conn(0x7f09480f89b0 msgr2=0x7f09480f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:23:39.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.413+0000 7f094e400700 1 -- 172.21.15.123:0/1846822121 shutdown_connections 2024-03-30T04:23:39.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.413+0000 7f094e400700 1 -- 172.21.15.123:0/1846822121 wait complete. 2024-03-30T04:23:40.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:40 smithi123 ceph-mon[35245]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:23:40.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:40 smithi135 ceph-mon[44364]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:23:40.572 INFO:teuthology.orchestra.run.smithi123.stdout:38654705706 2024-03-30T04:23:40.573 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.0 2024-03-30T04:23:40.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.366+0000 7f4b21400700 1 -- 172.21.15.123:0/3905736829 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b1c1035e0 msgr2=0x7f4b1c0fcc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:40.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.366+0000 7f4b21400700 1 --2- 172.21.15.123:0/3905736829 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b1c1035e0 0x7f4b1c0fcc50 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4b08004940 tx=0x7f4b08009e30 comp rx=0 tx=0).stop 2024-03-30T04:23:40.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.366+0000 7f4b21400700 1 -- 172.21.15.123:0/3905736829 shutdown_connections 2024-03-30T04:23:40.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.366+0000 7f4b21400700 1 --2- 172.21.15.123:0/3905736829 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b1c1035e0 0x7f4b1c0fcc50 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:40.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.366+0000 7f4b21400700 1 --2- 172.21.15.123:0/3905736829 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4b1c102c40 0x7f4b1c103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:40.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.366+0000 7f4b21400700 1 -- 172.21.15.123:0/3905736829 >> 172.21.15.123:0/3905736829 conn(0x7f4b1c0f8830 msgr2=0x7f4b1c0fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:23:40.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.366+0000 7f4b21400700 1 -- 172.21.15.123:0/3905736829 shutdown_connections 2024-03-30T04:23:40.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.367+0000 7f4b21400700 1 -- 172.21.15.123:0/3905736829 wait complete. 2024-03-30T04:23:40.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.367+0000 7f4b21400700 1 Processor -- start 2024-03-30T04:23:40.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.367+0000 7f4b21400700 1 -- start start 2024-03-30T04:23:40.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.368+0000 7f4b21400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b1c102c40 0x7f4b1c1138e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:40.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.368+0000 7f4b21400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4b1c1035e0 0x7f4b1c113e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:40.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.368+0000 7f4b21400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b1c114470 con 0x7f4b1c102c40 2024-03-30T04:23:40.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.368+0000 7f4b21400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b1c10d860 con 0x7f4b1c1035e0 2024-03-30T04:23:40.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.368+0000 7f4b1be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b1c102c40 0x7f4b1c1138e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:40.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.368+0000 7f4b1be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b1c102c40 0x7f4b1c1138e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44718/0 (socket says 172.21.15.123:44718) 2024-03-30T04:23:40.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.368+0000 7f4b1be00700 1 -- 172.21.15.123:0/1088941478 learned_addr learned my addr 172.21.15.123:0/1088941478 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:23:40.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.368+0000 7f4b1b400700 1 --2- 172.21.15.123:0/1088941478 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4b1c1035e0 0x7f4b1c113e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:40.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.368+0000 7f4b1be00700 1 -- 172.21.15.123:0/1088941478 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4b1c1035e0 msgr2=0x7f4b1c113e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:40.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.368+0000 7f4b1be00700 1 --2- 172.21.15.123:0/1088941478 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4b1c1035e0 0x7f4b1c113e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:40.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.368+0000 7f4b1be00700 1 -- 172.21.15.123:0/1088941478 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b08005040 con 0x7f4b1c102c40 2024-03-30T04:23:40.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.369+0000 7f4b1be00700 1 --2- 172.21.15.123:0/1088941478 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b1c102c40 0x7f4b1c1138e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c00c8c0 tx=0x7f4b0c0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:23:40.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.369+0000 7f4b13e00700 1 -- 172.21.15.123:0/1088941478 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b0c00a980 con 0x7f4b1c102c40 2024-03-30T04:23:40.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.369+0000 7f4b13e00700 1 -- 172.21.15.123:0/1088941478 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4b0c00ce60 con 0x7f4b1c102c40 2024-03-30T04:23:40.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.369+0000 7f4b13e00700 1 -- 172.21.15.123:0/1088941478 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b0c0113a0 con 0x7f4b1c102c40 2024-03-30T04:23:40.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.369+0000 7f4b21400700 1 -- 172.21.15.123:0/1088941478 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b1c10db40 con 0x7f4b1c102c40 2024-03-30T04:23:40.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.369+0000 7f4b21400700 1 -- 172.21.15.123:0/1088941478 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b1c10e0d0 con 0x7f4b1c102c40 2024-03-30T04:23:40.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.370+0000 7f4b21400700 1 -- 172.21.15.123:0/1088941478 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4b1c10e6a0 con 0x7f4b1c102c40 2024-03-30T04:23:40.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.371+0000 7f4b13e00700 1 -- 172.21.15.123:0/1088941478 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f4b0c012400 con 0x7f4b1c102c40 2024-03-30T04:23:40.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.372+0000 7f4b13e00700 1 --2- 172.21.15.123:0/1088941478 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f4b000692b0 0x7f4b0006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:40.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.372+0000 7f4b13e00700 1 -- 172.21.15.123:0/1088941478 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4b0c086ef0 con 0x7f4b1c102c40 2024-03-30T04:23:40.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.372+0000 7f4b1b400700 1 --2- 172.21.15.123:0/1088941478 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f4b000692b0 0x7f4b0006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:40.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.372+0000 7f4b13e00700 1 --2- 172.21.15.123:0/1088941478 >> [v2:172.21.15.123:6810/1040012134,v1:172.21.15.123:6811/1040012134] conn(0x7f4b0006ef80 0x7f4b00071390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:40.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.372+0000 7f4b13e00700 1 -- 172.21.15.123:0/1088941478 --> [v2:172.21.15.123:6810/1040012134,v1:172.21.15.123:6811/1040012134] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4b00071a40 con 0x7f4b0006ef80 2024-03-30T04:23:40.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.372+0000 7f4b13e00700 1 -- 172.21.15.123:0/1088941478 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4b0c08c460 con 0x7f4b1c102c40 2024-03-30T04:23:40.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.372+0000 7f4b20a00700 1 --2- 172.21.15.123:0/1088941478 >> [v2:172.21.15.123:6810/1040012134,v1:172.21.15.123:6811/1040012134] conn(0x7f4b0006ef80 0x7f4b00071390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:40.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.372+0000 7f4b1b400700 1 --2- 172.21.15.123:0/1088941478 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f4b000692b0 0x7f4b0006b760 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c10eb40 tx=0x7f4b08009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:23:40.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.373+0000 7f4b20a00700 1 --2- 172.21.15.123:0/1088941478 >> [v2:172.21.15.123:6810/1040012134,v1:172.21.15.123:6811/1040012134] conn(0x7f4b0006ef80 0x7f4b00071390 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:23:40.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.374+0000 7f4b13e00700 1 -- 172.21.15.123:0/1088941478 <== osd.1 v2:172.21.15.123:6810/1040012134 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f4b00071a40 con 0x7f4b0006ef80 2024-03-30T04:23:40.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.389+0000 7f4b21400700 1 -- 172.21.15.123:0/1088941478 --> [v2:172.21.15.123:6810/1040012134,v1:172.21.15.123:6811/1040012134] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4b1c0fa280 con 0x7f4b0006ef80 2024-03-30T04:23:40.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.389+0000 7f4b13e00700 1 -- 172.21.15.123:0/1088941478 <== osd.1 v2:172.21.15.123:6810/1040012134 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f4b1c0fa280 con 0x7f4b0006ef80 2024-03-30T04:23:40.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.390+0000 7f4b21400700 1 -- 172.21.15.123:0/1088941478 >> [v2:172.21.15.123:6810/1040012134,v1:172.21.15.123:6811/1040012134] conn(0x7f4b0006ef80 msgr2=0x7f4b00071390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:40.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.390+0000 7f4b21400700 1 --2- 172.21.15.123:0/1088941478 >> [v2:172.21.15.123:6810/1040012134,v1:172.21.15.123:6811/1040012134] conn(0x7f4b0006ef80 0x7f4b00071390 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:40.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.390+0000 7f4b21400700 1 -- 172.21.15.123:0/1088941478 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f4b000692b0 msgr2=0x7f4b0006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:40.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.390+0000 7f4b21400700 1 --2- 172.21.15.123:0/1088941478 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f4b000692b0 0x7f4b0006b760 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c10eb40 tx=0x7f4b08009e30 comp rx=0 tx=0).stop 2024-03-30T04:23:40.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.390+0000 7f4b21400700 1 -- 172.21.15.123:0/1088941478 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b1c102c40 msgr2=0x7f4b1c1138e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:40.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.390+0000 7f4b21400700 1 --2- 172.21.15.123:0/1088941478 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b1c102c40 0x7f4b1c1138e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c00c8c0 tx=0x7f4b0c0044c0 comp rx=0 tx=0).stop 2024-03-30T04:23:40.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.390+0000 7f4b21400700 1 -- 172.21.15.123:0/1088941478 shutdown_connections 2024-03-30T04:23:40.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.390+0000 7f4b21400700 1 --2- 172.21.15.123:0/1088941478 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f4b000692b0 0x7f4b0006b760 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:40.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.390+0000 7f4b21400700 1 --2- 172.21.15.123:0/1088941478 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b1c102c40 0x7f4b1c1138e0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:40.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.390+0000 7f4b21400700 1 --2- 172.21.15.123:0/1088941478 >> [v2:172.21.15.123:6810/1040012134,v1:172.21.15.123:6811/1040012134] conn(0x7f4b0006ef80 0x7f4b00071390 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:40.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.390+0000 7f4b21400700 1 --2- 172.21.15.123:0/1088941478 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4b1c1035e0 0x7f4b1c113e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:40.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.390+0000 7f4b21400700 1 -- 172.21.15.123:0/1088941478 >> 172.21.15.123:0/1088941478 conn(0x7f4b1c0f8830 msgr2=0x7f4b1c0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:23:40.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.390+0000 7f4b21400700 1 -- 172.21.15.123:0/1088941478 shutdown_connections 2024-03-30T04:23:40.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:40.390+0000 7f4b21400700 1 -- 172.21.15.123:0/1088941478 wait complete. 2024-03-30T04:23:41.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.283+0000 7f22f7800700 1 -- 172.21.15.123:0/210051429 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f22f00708a0 msgr2=0x7f22f0070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:41.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.283+0000 7f22f7800700 1 --2- 172.21.15.123:0/210051429 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f22f00708a0 0x7f22f0070d10 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f22dc004970 tx=0x7f22dc009e30 comp rx=0 tx=0).stop 2024-03-30T04:23:41.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.283+0000 7f22f7800700 1 -- 172.21.15.123:0/210051429 shutdown_connections 2024-03-30T04:23:41.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.283+0000 7f22f7800700 1 --2- 172.21.15.123:0/210051429 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f22f00708a0 0x7f22f0070d10 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:41.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.283+0000 7f22f7800700 1 --2- 172.21.15.123:0/210051429 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22f0071f90 0x7f22f0070360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:41.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.283+0000 7f22f7800700 1 -- 172.21.15.123:0/210051429 >> 172.21.15.123:0/210051429 conn(0x7f22f00f89b0 msgr2=0x7f22f00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:23:41.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.283+0000 7f22f7800700 1 -- 172.21.15.123:0/210051429 shutdown_connections 2024-03-30T04:23:41.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.283+0000 7f22f7800700 1 -- 172.21.15.123:0/210051429 wait complete. 2024-03-30T04:23:41.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.284+0000 7f22f7800700 1 Processor -- start 2024-03-30T04:23:41.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.284+0000 7f22f7800700 1 -- start start 2024-03-30T04:23:41.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.285+0000 7f22f7800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22f00708a0 0x7f22f011a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:41.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.285+0000 7f22f7800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f22f0071f90 0x7f22f011a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:41.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.285+0000 7f22f7800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f011ac60 con 0x7f22f0071f90 2024-03-30T04:23:41.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.285+0000 7f22f7800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f0114030 con 0x7f22f00708a0 2024-03-30T04:23:41.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.285+0000 7f22efe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f22f0071f90 0x7f22f011a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:41.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.285+0000 7f22efe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f22f0071f90 0x7f22f011a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44698/0 (socket says 172.21.15.123:44698) 2024-03-30T04:23:41.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.285+0000 7f22f4e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22f00708a0 0x7f22f011a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:41.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.285+0000 7f22efe00700 1 -- 172.21.15.123:0/4147793423 learned_addr learned my addr 172.21.15.123:0/4147793423 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:23:41.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.285+0000 7f22efe00700 1 -- 172.21.15.123:0/4147793423 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22f00708a0 msgr2=0x7f22f011a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:41.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.285+0000 7f22efe00700 1 --2- 172.21.15.123:0/4147793423 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22f00708a0 0x7f22f011a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:41.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.285+0000 7f22efe00700 1 -- 172.21.15.123:0/4147793423 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22dc005040 con 0x7f22f0071f90 2024-03-30T04:23:41.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.286+0000 7f22efe00700 1 --2- 172.21.15.123:0/4147793423 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f22f0071f90 0x7f22f011a610 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f22dc005010 tx=0x7f22dc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:23:41.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.286+0000 7f22ecc00700 1 -- 172.21.15.123:0/4147793423 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f22dc00e040 con 0x7f22f0071f90 2024-03-30T04:23:41.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.286+0000 7f22ecc00700 1 -- 172.21.15.123:0/4147793423 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f22dc004290 con 0x7f22f0071f90 2024-03-30T04:23:41.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.286+0000 7f22ecc00700 1 -- 172.21.15.123:0/4147793423 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f22dc013d40 con 0x7f22f0071f90 2024-03-30T04:23:41.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.286+0000 7f22f7800700 1 -- 172.21.15.123:0/4147793423 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22f01142b0 con 0x7f22f0071f90 2024-03-30T04:23:41.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.286+0000 7f22f7800700 1 -- 172.21.15.123:0/4147793423 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22f01147f0 con 0x7f22f0071f90 2024-03-30T04:23:41.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.287+0000 7f22f7800700 1 -- 172.21.15.123:0/4147793423 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f22f0114d90 con 0x7f22f0071f90 2024-03-30T04:23:41.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.288+0000 7f22ecc00700 1 -- 172.21.15.123:0/4147793423 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f22dc013520 con 0x7f22f0071f90 2024-03-30T04:23:41.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.289+0000 7f22ecc00700 1 --2- 172.21.15.123:0/4147793423 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f22d80692b0 0x7f22d806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:41.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.289+0000 7f22ecc00700 1 -- 172.21.15.123:0/4147793423 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f22dc088ae0 con 0x7f22f0071f90 2024-03-30T04:23:41.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.289+0000 7f22f4e00700 1 --2- 172.21.15.123:0/4147793423 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f22d80692b0 0x7f22d806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:41.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.289+0000 7f22ecc00700 1 --2- 172.21.15.123:0/4147793423 >> [v2:172.21.15.135:6816/1609344282,v1:172.21.15.135:6817/1609344282] conn(0x7f22d806ef80 0x7f22d8071390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:41.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.289+0000 7f22ecc00700 1 -- 172.21.15.123:0/4147793423 --> [v2:172.21.15.135:6816/1609344282,v1:172.21.15.135:6817/1609344282] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f22d8071a40 con 0x7f22d806ef80 2024-03-30T04:23:41.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.289+0000 7f22ecc00700 1 -- 172.21.15.123:0/4147793423 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f22dc088ec0 con 0x7f22f0071f90 2024-03-30T04:23:41.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.290+0000 7f22f5800700 1 --2- 172.21.15.123:0/4147793423 >> [v2:172.21.15.135:6816/1609344282,v1:172.21.15.135:6817/1609344282] conn(0x7f22d806ef80 0x7f22d8071390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:41.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.290+0000 7f22f4e00700 1 --2- 172.21.15.123:0/4147793423 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f22d80692b0 0x7f22d806b760 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f22e4009b80 tx=0x7f22e400b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:23:41.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.290+0000 7f22f5800700 1 --2- 172.21.15.123:0/4147793423 >> [v2:172.21.15.135:6816/1609344282,v1:172.21.15.135:6817/1609344282] conn(0x7f22d806ef80 0x7f22d8071390 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:23:41.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.291+0000 7f22ecc00700 1 -- 172.21.15.123:0/4147793423 <== osd.5 v2:172.21.15.135:6816/1609344282 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f22d8071a40 con 0x7f22d806ef80 2024-03-30T04:23:41.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.307+0000 7f22f7800700 1 -- 172.21.15.123:0/4147793423 --> [v2:172.21.15.135:6816/1609344282,v1:172.21.15.135:6817/1609344282] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f22f00fa520 con 0x7f22d806ef80 2024-03-30T04:23:41.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.307+0000 7f22ecc00700 1 -- 172.21.15.123:0/4147793423 <== osd.5 v2:172.21.15.135:6816/1609344282 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f22f00fa520 con 0x7f22d806ef80 2024-03-30T04:23:41.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.308+0000 7f22f7800700 1 -- 172.21.15.123:0/4147793423 >> [v2:172.21.15.135:6816/1609344282,v1:172.21.15.135:6817/1609344282] conn(0x7f22d806ef80 msgr2=0x7f22d8071390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:41.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.308+0000 7f22f7800700 1 --2- 172.21.15.123:0/4147793423 >> [v2:172.21.15.135:6816/1609344282,v1:172.21.15.135:6817/1609344282] conn(0x7f22d806ef80 0x7f22d8071390 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:41.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.308+0000 7f22f7800700 1 -- 172.21.15.123:0/4147793423 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f22d80692b0 msgr2=0x7f22d806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:41.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.308+0000 7f22f7800700 1 --2- 172.21.15.123:0/4147793423 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f22d80692b0 0x7f22d806b760 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f22e4009b80 tx=0x7f22e400b040 comp rx=0 tx=0).stop 2024-03-30T04:23:41.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.308+0000 7f22f7800700 1 -- 172.21.15.123:0/4147793423 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f22f0071f90 msgr2=0x7f22f011a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:41.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.308+0000 7f22f7800700 1 --2- 172.21.15.123:0/4147793423 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f22f0071f90 0x7f22f011a610 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f22dc005010 tx=0x7f22dc009e30 comp rx=0 tx=0).stop 2024-03-30T04:23:41.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.308+0000 7f22f7800700 1 -- 172.21.15.123:0/4147793423 shutdown_connections 2024-03-30T04:23:41.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.308+0000 7f22f7800700 1 --2- 172.21.15.123:0/4147793423 >> [v2:172.21.15.135:6816/1609344282,v1:172.21.15.135:6817/1609344282] conn(0x7f22d806ef80 0x7f22d8071390 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:41.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.308+0000 7f22f7800700 1 --2- 172.21.15.123:0/4147793423 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f22d80692b0 0x7f22d806b760 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:41.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.308+0000 7f22f7800700 1 --2- 172.21.15.123:0/4147793423 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22f00708a0 0x7f22f011a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:41.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.308+0000 7f22f7800700 1 --2- 172.21.15.123:0/4147793423 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f22f0071f90 0x7f22f011a610 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:41.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.308+0000 7f22f7800700 1 -- 172.21.15.123:0/4147793423 >> 172.21.15.123:0/4147793423 conn(0x7f22f00f89b0 msgr2=0x7f22f00f9b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:23:41.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.308+0000 7f22f7800700 1 -- 172.21.15.123:0/4147793423 shutdown_connections 2024-03-30T04:23:41.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:39.308+0000 7f22f7800700 1 -- 172.21.15.123:0/4147793423 wait complete. 2024-03-30T04:23:41.345 INFO:teuthology.orchestra.run.smithi123.stdout:60129542178 2024-03-30T04:23:41.345 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.1 2024-03-30T04:23:42.309 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:42 smithi123 ceph-mon[35245]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:23:42.309 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:42 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:23:42.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:42 smithi135 ceph-mon[44364]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:23:42.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:42 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:23:42.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.136+0000 7f3417200700 1 -- 172.21.15.123:0/465404640 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3410071f90 msgr2=0x7f3410070360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:42.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.136+0000 7f3417200700 1 --2- 172.21.15.123:0/465404640 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3410071f90 0x7f3410070360 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3400004970 tx=0x7f3400009e30 comp rx=0 tx=0).stop 2024-03-30T04:23:42.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.136+0000 7f3417200700 1 -- 172.21.15.123:0/465404640 shutdown_connections 2024-03-30T04:23:42.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.136+0000 7f3417200700 1 --2- 172.21.15.123:0/465404640 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f34100708a0 0x7f3410070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:42.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.136+0000 7f3417200700 1 --2- 172.21.15.123:0/465404640 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3410071f90 0x7f3410070360 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:42.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.136+0000 7f3417200700 1 -- 172.21.15.123:0/465404640 >> 172.21.15.123:0/465404640 conn(0x7f34100f89b0 msgr2=0x7f34100fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:23:42.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.136+0000 7f3417200700 1 -- 172.21.15.123:0/465404640 shutdown_connections 2024-03-30T04:23:42.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.136+0000 7f3417200700 1 -- 172.21.15.123:0/465404640 wait complete. 2024-03-30T04:23:42.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.137+0000 7f3417200700 1 Processor -- start 2024-03-30T04:23:42.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.137+0000 7f3417200700 1 -- start start 2024-03-30T04:23:42.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.137+0000 7f3417200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34100708a0 0x7f341011a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:42.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.137+0000 7f3417200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3410071f90 0x7f341011a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:42.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.137+0000 7f3417200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f341011ac50 con 0x7f34100708a0 2024-03-30T04:23:42.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.137+0000 7f3417200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3410114020 con 0x7f3410071f90 2024-03-30T04:23:42.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.138+0000 7f340fe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34100708a0 0x7f341011a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:42.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.138+0000 7f340fe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34100708a0 0x7f341011a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44758/0 (socket says 172.21.15.123:44758) 2024-03-30T04:23:42.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.138+0000 7f340fe00700 1 -- 172.21.15.123:0/2582688666 learned_addr learned my addr 172.21.15.123:0/2582688666 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:23:42.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.138+0000 7f340f400700 1 --2- 172.21.15.123:0/2582688666 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3410071f90 0x7f341011a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:42.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.138+0000 7f340fe00700 1 -- 172.21.15.123:0/2582688666 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3410071f90 msgr2=0x7f341011a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:42.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.138+0000 7f340fe00700 1 --2- 172.21.15.123:0/2582688666 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3410071f90 0x7f341011a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:42.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.138+0000 7f340fe00700 1 -- 172.21.15.123:0/2582688666 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3400005040 con 0x7f34100708a0 2024-03-30T04:23:42.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.138+0000 7f340fe00700 1 --2- 172.21.15.123:0/2582688666 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34100708a0 0x7f341011a0c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3400004f50 tx=0x7f3400009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:23:42.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.139+0000 7f3407e00700 1 -- 172.21.15.123:0/2582688666 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f340000e040 con 0x7f34100708a0 2024-03-30T04:23:42.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.139+0000 7f3407e00700 1 -- 172.21.15.123:0/2582688666 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3400013420 con 0x7f34100708a0 2024-03-30T04:23:42.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.139+0000 7f3417200700 1 -- 172.21.15.123:0/2582688666 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34101142a0 con 0x7f34100708a0 2024-03-30T04:23:42.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.139+0000 7f3407e00700 1 -- 172.21.15.123:0/2582688666 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f340001c630 con 0x7f34100708a0 2024-03-30T04:23:42.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.139+0000 7f3417200700 1 -- 172.21.15.123:0/2582688666 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34101147e0 con 0x7f34100708a0 2024-03-30T04:23:42.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.140+0000 7f3417200700 1 -- 172.21.15.123:0/2582688666 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3410114dd0 con 0x7f34100708a0 2024-03-30T04:23:42.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.141+0000 7f3407e00700 1 -- 172.21.15.123:0/2582688666 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f3400013590 con 0x7f34100708a0 2024-03-30T04:23:42.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.142+0000 7f3407e00700 1 --2- 172.21.15.123:0/2582688666 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f33f40692b0 0x7f33f406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:42.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.142+0000 7f3407e00700 1 -- 172.21.15.123:0/2582688666 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f3400087fd0 con 0x7f34100708a0 2024-03-30T04:23:42.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.142+0000 7f340f400700 1 --2- 172.21.15.123:0/2582688666 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f33f40692b0 0x7f33f406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:42.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.142+0000 7f3407e00700 1 --2- 172.21.15.123:0/2582688666 >> [v2:172.21.15.123:6818/1260172805,v1:172.21.15.123:6819/1260172805] conn(0x7f33f406ef80 0x7f33f4071390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:42.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.142+0000 7f3407e00700 1 -- 172.21.15.123:0/2582688666 --> [v2:172.21.15.123:6818/1260172805,v1:172.21.15.123:6819/1260172805] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f33f4071a40 con 0x7f33f406ef80 2024-03-30T04:23:42.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.142+0000 7f3407e00700 1 -- 172.21.15.123:0/2582688666 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f340008a580 con 0x7f34100708a0 2024-03-30T04:23:42.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.143+0000 7f3415200700 1 --2- 172.21.15.123:0/2582688666 >> [v2:172.21.15.123:6818/1260172805,v1:172.21.15.123:6819/1260172805] conn(0x7f33f406ef80 0x7f33f4071390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:42.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.143+0000 7f340f400700 1 --2- 172.21.15.123:0/2582688666 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f33f40692b0 0x7f33f406b760 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3410115960 tx=0x7f33fc00b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:23:42.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.143+0000 7f3415200700 1 --2- 172.21.15.123:0/2582688666 >> [v2:172.21.15.123:6818/1260172805,v1:172.21.15.123:6819/1260172805] conn(0x7f33f406ef80 0x7f33f4071390 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:23:42.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.144+0000 7f3407e00700 1 -- 172.21.15.123:0/2582688666 <== osd.2 v2:172.21.15.123:6818/1260172805 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f33f4071a40 con 0x7f33f406ef80 2024-03-30T04:23:42.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.160+0000 7f3417200700 1 -- 172.21.15.123:0/2582688666 --> [v2:172.21.15.123:6818/1260172805,v1:172.21.15.123:6819/1260172805] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3410114430 con 0x7f33f406ef80 2024-03-30T04:23:42.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.161+0000 7f3407e00700 1 -- 172.21.15.123:0/2582688666 <== osd.2 v2:172.21.15.123:6818/1260172805 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3410114430 con 0x7f33f406ef80 2024-03-30T04:23:42.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.161+0000 7f3417200700 1 -- 172.21.15.123:0/2582688666 >> [v2:172.21.15.123:6818/1260172805,v1:172.21.15.123:6819/1260172805] conn(0x7f33f406ef80 msgr2=0x7f33f4071390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:42.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.161+0000 7f3417200700 1 --2- 172.21.15.123:0/2582688666 >> [v2:172.21.15.123:6818/1260172805,v1:172.21.15.123:6819/1260172805] conn(0x7f33f406ef80 0x7f33f4071390 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:42.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.161+0000 7f3417200700 1 -- 172.21.15.123:0/2582688666 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f33f40692b0 msgr2=0x7f33f406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:42.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.161+0000 7f3417200700 1 --2- 172.21.15.123:0/2582688666 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f33f40692b0 0x7f33f406b760 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3410115960 tx=0x7f33fc00b040 comp rx=0 tx=0).stop 2024-03-30T04:23:42.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.161+0000 7f3417200700 1 -- 172.21.15.123:0/2582688666 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34100708a0 msgr2=0x7f341011a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:42.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.161+0000 7f3417200700 1 --2- 172.21.15.123:0/2582688666 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34100708a0 0x7f341011a0c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3400004f50 tx=0x7f3400009e30 comp rx=0 tx=0).stop 2024-03-30T04:23:42.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.161+0000 7f3417200700 1 -- 172.21.15.123:0/2582688666 shutdown_connections 2024-03-30T04:23:42.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.161+0000 7f3417200700 1 --2- 172.21.15.123:0/2582688666 >> [v2:172.21.15.123:6818/1260172805,v1:172.21.15.123:6819/1260172805] conn(0x7f33f406ef80 0x7f33f4071390 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:42.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.161+0000 7f3417200700 1 --2- 172.21.15.123:0/2582688666 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f33f40692b0 0x7f33f406b760 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:42.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.161+0000 7f3417200700 1 --2- 172.21.15.123:0/2582688666 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34100708a0 0x7f341011a0c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:42.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.161+0000 7f3417200700 1 --2- 172.21.15.123:0/2582688666 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3410071f90 0x7f341011a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:42.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.161+0000 7f3417200700 1 -- 172.21.15.123:0/2582688666 >> 172.21.15.123:0/2582688666 conn(0x7f34100f89b0 msgr2=0x7f34100f9b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:23:42.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.161+0000 7f3417200700 1 -- 172.21.15.123:0/2582688666 shutdown_connections 2024-03-30T04:23:42.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.161+0000 7f3417200700 1 -- 172.21.15.123:0/2582688666 wait complete. 2024-03-30T04:23:42.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.029+0000 7f930ec00700 1 -- 172.21.15.123:0/2835854873 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9308070c70 msgr2=0x7f9308108f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:42.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.029+0000 7f930ec00700 1 --2- 172.21.15.123:0/2835854873 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9308070c70 0x7f9308108f50 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f92f4004970 tx=0x7f92f4009e30 comp rx=0 tx=0).stop 2024-03-30T04:23:42.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.029+0000 7f930ec00700 1 -- 172.21.15.123:0/2835854873 shutdown_connections 2024-03-30T04:23:42.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.029+0000 7f930ec00700 1 --2- 172.21.15.123:0/2835854873 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9308070c70 0x7f9308108f50 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:42.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.029+0000 7f930ec00700 1 --2- 172.21.15.123:0/2835854873 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9308070360 0x7f9308070730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:42.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.029+0000 7f930ec00700 1 -- 172.21.15.123:0/2835854873 >> 172.21.15.123:0/2835854873 conn(0x7f93080f8830 msgr2=0x7f93080fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:23:42.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.030+0000 7f930ec00700 1 -- 172.21.15.123:0/2835854873 shutdown_connections 2024-03-30T04:23:42.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.030+0000 7f930ec00700 1 -- 172.21.15.123:0/2835854873 wait complete. 2024-03-30T04:23:42.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.030+0000 7f930ec00700 1 Processor -- start 2024-03-30T04:23:42.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.031+0000 7f930ec00700 1 -- start start 2024-03-30T04:23:42.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.031+0000 7f930ec00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9308070360 0x7f9308115ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:42.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.031+0000 7f930ec00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9308070c70 0x7f9308116000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:42.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.031+0000 7f930ec00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9308116650 con 0x7f9308070c70 2024-03-30T04:23:42.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.031+0000 7f930ec00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f930810fa40 con 0x7f9308070360 2024-03-30T04:23:42.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.031+0000 7f930d800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9308070360 0x7f9308115ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:42.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.032+0000 7f930d800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9308070360 0x7f9308115ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.123:47202/0 (socket says 172.21.15.123:47202) 2024-03-30T04:23:42.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.032+0000 7f930d800700 1 -- 172.21.15.123:0/1026256884 learned_addr learned my addr 172.21.15.123:0/1026256884 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:23:42.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.032+0000 7f930ce00700 1 --2- 172.21.15.123:0/1026256884 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9308070c70 0x7f9308116000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:42.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.032+0000 7f930d800700 1 -- 172.21.15.123:0/1026256884 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9308070c70 msgr2=0x7f9308116000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:42.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.032+0000 7f930d800700 1 --2- 172.21.15.123:0/1026256884 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9308070c70 0x7f9308116000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:42.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.032+0000 7f930d800700 1 -- 172.21.15.123:0/1026256884 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92f4005040 con 0x7f9308070360 2024-03-30T04:23:42.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.032+0000 7f930d800700 1 --2- 172.21.15.123:0/1026256884 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9308070360 0x7f9308115ac0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f92fc002a50 tx=0x7f92fc0046b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:23:42.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.034+0000 7f9305600700 1 -- 172.21.15.123:0/1026256884 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f92fc007cc0 con 0x7f9308070360 2024-03-30T04:23:42.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.034+0000 7f9305600700 1 -- 172.21.15.123:0/1026256884 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f92fc00f040 con 0x7f9308070360 2024-03-30T04:23:42.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.034+0000 7f930ec00700 1 -- 172.21.15.123:0/1026256884 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f930810fd20 con 0x7f9308070360 2024-03-30T04:23:42.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.034+0000 7f9305600700 1 -- 172.21.15.123:0/1026256884 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f92fc00d9c0 con 0x7f9308070360 2024-03-30T04:23:42.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.035+0000 7f930ec00700 1 -- 172.21.15.123:0/1026256884 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9308110230 con 0x7f9308070360 2024-03-30T04:23:42.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.035+0000 7f930ec00700 1 -- 172.21.15.123:0/1026256884 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9308110720 con 0x7f9308070360 2024-03-30T04:23:42.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.036+0000 7f9305600700 1 -- 172.21.15.123:0/1026256884 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f92fc009070 con 0x7f9308070360 2024-03-30T04:23:42.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.037+0000 7f9305600700 1 --2- 172.21.15.123:0/1026256884 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f92f00692b0 0x7f92f006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:42.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.037+0000 7f9305600700 1 -- 172.21.15.123:0/1026256884 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f92fc086c70 con 0x7f9308070360 2024-03-30T04:23:42.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.037+0000 7f9305600700 1 --2- 172.21.15.123:0/1026256884 >> [v2:172.21.15.135:6808/1699881600,v1:172.21.15.135:6809/1699881600] conn(0x7f92f006ef80 0x7f92f0071390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:42.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.038+0000 7f9305600700 1 -- 172.21.15.123:0/1026256884 --> [v2:172.21.15.135:6808/1699881600,v1:172.21.15.135:6809/1699881600] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f92f0071a40 con 0x7f92f006ef80 2024-03-30T04:23:42.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.038+0000 7f930ce00700 1 --2- 172.21.15.123:0/1026256884 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f92f00692b0 0x7f92f006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:42.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.038+0000 7f9305600700 1 -- 172.21.15.123:0/1026256884 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f92fc089220 con 0x7f9308070360 2024-03-30T04:23:42.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.038+0000 7f930e200700 1 --2- 172.21.15.123:0/1026256884 >> [v2:172.21.15.135:6808/1699881600,v1:172.21.15.135:6809/1699881600] conn(0x7f92f006ef80 0x7f92f0071390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:42.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.038+0000 7f930ce00700 1 --2- 172.21.15.123:0/1026256884 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f92f00692b0 0x7f92f006b760 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f92f400b370 tx=0x7f92f4009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:23:42.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.039+0000 7f930e200700 1 --2- 172.21.15.123:0/1026256884 >> [v2:172.21.15.135:6808/1699881600,v1:172.21.15.135:6809/1699881600] conn(0x7f92f006ef80 0x7f92f0071390 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:23:42.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.040+0000 7f9305600700 1 -- 172.21.15.123:0/1026256884 <== osd.4 v2:172.21.15.135:6808/1699881600 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f92f0071a40 con 0x7f92f006ef80 2024-03-30T04:23:42.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.055+0000 7f930ec00700 1 -- 172.21.15.123:0/1026256884 --> [v2:172.21.15.135:6808/1699881600,v1:172.21.15.135:6809/1699881600] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f93080fa740 con 0x7f92f006ef80 2024-03-30T04:23:42.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.055+0000 7f9305600700 1 -- 172.21.15.123:0/1026256884 <== osd.4 v2:172.21.15.135:6808/1699881600 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f93080fa740 con 0x7f92f006ef80 2024-03-30T04:23:42.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.056+0000 7f930ec00700 1 -- 172.21.15.123:0/1026256884 >> [v2:172.21.15.135:6808/1699881600,v1:172.21.15.135:6809/1699881600] conn(0x7f92f006ef80 msgr2=0x7f92f0071390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:42.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.056+0000 7f930ec00700 1 --2- 172.21.15.123:0/1026256884 >> [v2:172.21.15.135:6808/1699881600,v1:172.21.15.135:6809/1699881600] conn(0x7f92f006ef80 0x7f92f0071390 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:42.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.056+0000 7f930ec00700 1 -- 172.21.15.123:0/1026256884 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f92f00692b0 msgr2=0x7f92f006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:42.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.056+0000 7f930ec00700 1 --2- 172.21.15.123:0/1026256884 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f92f00692b0 0x7f92f006b760 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f92f400b370 tx=0x7f92f4009e30 comp rx=0 tx=0).stop 2024-03-30T04:23:42.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.056+0000 7f930ec00700 1 -- 172.21.15.123:0/1026256884 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9308070360 msgr2=0x7f9308115ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:42.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.056+0000 7f930ec00700 1 --2- 172.21.15.123:0/1026256884 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9308070360 0x7f9308115ac0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f92fc002a50 tx=0x7f92fc0046b0 comp rx=0 tx=0).stop 2024-03-30T04:23:42.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.056+0000 7f930ec00700 1 -- 172.21.15.123:0/1026256884 shutdown_connections 2024-03-30T04:23:42.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.056+0000 7f930ec00700 1 --2- 172.21.15.123:0/1026256884 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f92f00692b0 0x7f92f006b760 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:42.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.056+0000 7f930ec00700 1 --2- 172.21.15.123:0/1026256884 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9308070360 0x7f9308115ac0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:42.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.056+0000 7f930ec00700 1 --2- 172.21.15.123:0/1026256884 >> [v2:172.21.15.135:6808/1699881600,v1:172.21.15.135:6809/1699881600] conn(0x7f92f006ef80 0x7f92f0071390 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:42.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.056+0000 7f930ec00700 1 --2- 172.21.15.123:0/1026256884 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9308070c70 0x7f9308116000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:42.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.056+0000 7f930ec00700 1 -- 172.21.15.123:0/1026256884 >> 172.21.15.123:0/1026256884 conn(0x7f93080f8830 msgr2=0x7f93080f9d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:23:42.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.056+0000 7f930ec00700 1 -- 172.21.15.123:0/1026256884 shutdown_connections 2024-03-30T04:23:42.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:42.056+0000 7f930ec00700 1 -- 172.21.15.123:0/1026256884 wait complete. 2024-03-30T04:23:43.468 INFO:teuthology.orchestra.run.smithi123.stdout:81604378653 2024-03-30T04:23:43.468 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.2 2024-03-30T04:23:43.800 INFO:teuthology.orchestra.run.smithi123.stdout:167503724553 2024-03-30T04:23:43.801 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.5 2024-03-30T04:23:44.026 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:23:44.031 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:23:44.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:44 smithi135 ceph-mon[44364]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:23:44.393 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:44 smithi123 ceph-mon[35245]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:23:44.694 INFO:teuthology.orchestra.run.smithi123.stdout:146028888079 2024-03-30T04:23:44.694 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.4 2024-03-30T04:23:45.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.350+0000 7f3170a00700 1 -- 172.21.15.123:0/3831482205 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f316c070360 msgr2=0x7f316c070730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:45.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.350+0000 7f3170a00700 1 --2- 172.21.15.123:0/3831482205 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f316c070360 0x7f316c070730 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f315c004970 tx=0x7f315c009e30 comp rx=0 tx=0).stop 2024-03-30T04:23:45.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.350+0000 7f3170a00700 1 -- 172.21.15.123:0/3831482205 shutdown_connections 2024-03-30T04:23:45.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.350+0000 7f3170a00700 1 --2- 172.21.15.123:0/3831482205 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f316c070c70 0x7f316c109110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:45.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.350+0000 7f3170a00700 1 --2- 172.21.15.123:0/3831482205 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f316c070360 0x7f316c070730 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:45.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.350+0000 7f3170a00700 1 -- 172.21.15.123:0/3831482205 >> 172.21.15.123:0/3831482205 conn(0x7f316c0f89b0 msgr2=0x7f316c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:23:45.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.350+0000 7f3170a00700 1 -- 172.21.15.123:0/3831482205 shutdown_connections 2024-03-30T04:23:45.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.350+0000 7f3170a00700 1 -- 172.21.15.123:0/3831482205 wait complete. 2024-03-30T04:23:45.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.351+0000 7f3170a00700 1 Processor -- start 2024-03-30T04:23:45.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.351+0000 7f3170a00700 1 -- start start 2024-03-30T04:23:45.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.352+0000 7f3170a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f316c070360 0x7f316c115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:45.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.352+0000 7f3170a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f316c070c70 0x7f316c1161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:45.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.352+0000 7f3170a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f316c116810 con 0x7f316c070c70 2024-03-30T04:23:45.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.352+0000 7f3170a00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f316c10fbe0 con 0x7f316c070360 2024-03-30T04:23:45.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.352+0000 7f3169400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f316c070c70 0x7f316c1161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:45.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.352+0000 7f3169400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f316c070c70 0x7f316c1161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37220/0 (socket says 172.21.15.123:37220) 2024-03-30T04:23:45.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.352+0000 7f3169400700 1 -- 172.21.15.123:0/1560673840 learned_addr learned my addr 172.21.15.123:0/1560673840 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:23:45.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.352+0000 7f3169e00700 1 --2- 172.21.15.123:0/1560673840 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f316c070360 0x7f316c115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:45.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.352+0000 7f3169400700 1 -- 172.21.15.123:0/1560673840 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f316c070360 msgr2=0x7f316c115c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:45.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.352+0000 7f3169400700 1 --2- 172.21.15.123:0/1560673840 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f316c070360 0x7f316c115c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:45.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.352+0000 7f3169400700 1 -- 172.21.15.123:0/1560673840 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f315c005040 con 0x7f316c070c70 2024-03-30T04:23:45.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.353+0000 7f3169400700 1 --2- 172.21.15.123:0/1560673840 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f316c070c70 0x7f316c1161c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3158002a50 tx=0x7f31580046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:23:45.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.353+0000 7f3162000700 1 -- 172.21.15.123:0/1560673840 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3158007cc0 con 0x7f316c070c70 2024-03-30T04:23:45.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.353+0000 7f3162000700 1 -- 172.21.15.123:0/1560673840 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f315800f040 con 0x7f316c070c70 2024-03-30T04:23:45.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.353+0000 7f3162000700 1 -- 172.21.15.123:0/1560673840 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3158009540 con 0x7f316c070c70 2024-03-30T04:23:45.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.353+0000 7f3170a00700 1 -- 172.21.15.123:0/1560673840 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f316c10fec0 con 0x7f316c070c70 2024-03-30T04:23:45.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.353+0000 7f3170a00700 1 -- 172.21.15.123:0/1560673840 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f316c110400 con 0x7f316c070c70 2024-03-30T04:23:45.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.354+0000 7f3170a00700 1 -- 172.21.15.123:0/1560673840 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f316c110930 con 0x7f316c070c70 2024-03-30T04:23:45.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.355+0000 7f3162000700 1 -- 172.21.15.123:0/1560673840 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f31580096a0 con 0x7f316c070c70 2024-03-30T04:23:45.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.356+0000 7f3162000700 1 --2- 172.21.15.123:0/1560673840 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f31500692b0 0x7f315006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:45.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.356+0000 7f3162000700 1 -- 172.21.15.123:0/1560673840 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f3158086c30 con 0x7f316c070c70 2024-03-30T04:23:45.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.356+0000 7f3169e00700 1 --2- 172.21.15.123:0/1560673840 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f31500692b0 0x7f315006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:45.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.356+0000 7f3162000700 1 --2- 172.21.15.123:0/1560673840 >> [v2:172.21.15.135:6800/2906534885,v1:172.21.15.135:6801/2906534885] conn(0x7f315006ef80 0x7f3150071390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:45.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.356+0000 7f3162000700 1 -- 172.21.15.123:0/1560673840 --> [v2:172.21.15.135:6800/2906534885,v1:172.21.15.135:6801/2906534885] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3150071a40 con 0x7f315006ef80 2024-03-30T04:23:45.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.356+0000 7f3162000700 1 -- 172.21.15.123:0/1560673840 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f315808b220 con 0x7f316c070c70 2024-03-30T04:23:45.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.356+0000 7f316a800700 1 --2- 172.21.15.123:0/1560673840 >> [v2:172.21.15.135:6800/2906534885,v1:172.21.15.135:6801/2906534885] conn(0x7f315006ef80 0x7f3150071390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:45.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.356+0000 7f3169e00700 1 --2- 172.21.15.123:0/1560673840 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f31500692b0 0x7f315006b760 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f315c009fd0 tx=0x7f315c009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:23:45.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.357+0000 7f316a800700 1 --2- 172.21.15.123:0/1560673840 >> [v2:172.21.15.135:6800/2906534885,v1:172.21.15.135:6801/2906534885] conn(0x7f315006ef80 0x7f3150071390 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:23:45.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.358+0000 7f3162000700 1 -- 172.21.15.123:0/1560673840 <== osd.3 v2:172.21.15.135:6800/2906534885 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f3150071a40 con 0x7f315006ef80 2024-03-30T04:23:45.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.373+0000 7f3170a00700 1 -- 172.21.15.123:0/1560673840 --> [v2:172.21.15.135:6800/2906534885,v1:172.21.15.135:6801/2906534885] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f316c05e680 con 0x7f315006ef80 2024-03-30T04:23:45.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.374+0000 7f3162000700 1 -- 172.21.15.123:0/1560673840 <== osd.3 v2:172.21.15.135:6800/2906534885 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f316c05e680 con 0x7f315006ef80 2024-03-30T04:23:45.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.375+0000 7f3170a00700 1 -- 172.21.15.123:0/1560673840 >> [v2:172.21.15.135:6800/2906534885,v1:172.21.15.135:6801/2906534885] conn(0x7f315006ef80 msgr2=0x7f3150071390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:45.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.375+0000 7f3170a00700 1 --2- 172.21.15.123:0/1560673840 >> [v2:172.21.15.135:6800/2906534885,v1:172.21.15.135:6801/2906534885] conn(0x7f315006ef80 0x7f3150071390 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:45.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.375+0000 7f3170a00700 1 -- 172.21.15.123:0/1560673840 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f31500692b0 msgr2=0x7f315006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:45.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.375+0000 7f3170a00700 1 --2- 172.21.15.123:0/1560673840 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f31500692b0 0x7f315006b760 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f315c009fd0 tx=0x7f315c009e30 comp rx=0 tx=0).stop 2024-03-30T04:23:45.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.375+0000 7f3170a00700 1 -- 172.21.15.123:0/1560673840 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f316c070c70 msgr2=0x7f316c1161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:45.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.375+0000 7f3170a00700 1 --2- 172.21.15.123:0/1560673840 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f316c070c70 0x7f316c1161c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3158002a50 tx=0x7f31580046b0 comp rx=0 tx=0).stop 2024-03-30T04:23:45.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.375+0000 7f3170a00700 1 -- 172.21.15.123:0/1560673840 shutdown_connections 2024-03-30T04:23:45.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.375+0000 7f3170a00700 1 --2- 172.21.15.123:0/1560673840 >> [v2:172.21.15.135:6800/2906534885,v1:172.21.15.135:6801/2906534885] conn(0x7f315006ef80 0x7f3150071390 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:45.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.375+0000 7f3170a00700 1 --2- 172.21.15.123:0/1560673840 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f31500692b0 0x7f315006b760 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:45.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.375+0000 7f3170a00700 1 --2- 172.21.15.123:0/1560673840 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f316c070360 0x7f316c115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:45.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.375+0000 7f3170a00700 1 --2- 172.21.15.123:0/1560673840 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f316c070c70 0x7f316c1161c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:45.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.375+0000 7f3170a00700 1 -- 172.21.15.123:0/1560673840 >> 172.21.15.123:0/1560673840 conn(0x7f316c0f89b0 msgr2=0x7f316c1015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:23:45.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.375+0000 7f3170a00700 1 -- 172.21.15.123:0/1560673840 shutdown_connections 2024-03-30T04:23:45.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:44.375+0000 7f3170a00700 1 -- 172.21.15.123:0/1560673840 wait complete. 2024-03-30T04:23:46.343 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:46 smithi123 ceph-mon[35245]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:23:46.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:46 smithi135 ceph-mon[44364]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:23:46.706 INFO:teuthology.orchestra.run.smithi123.stdout:111669149719 2024-03-30T04:23:46.706 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.3 2024-03-30T04:23:47.429 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:23:47.429 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:23:48.265 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:48 smithi123 ceph-mon[35245]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:23:48.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:48 smithi135 ceph-mon[44364]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:23:49.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:49.046+0000 7fd592600700 1 -- 172.21.15.123:0/3424657916 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd58c102e00 msgr2=0x7fd58c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:49.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:49.046+0000 7fd592600700 1 --2- 172.21.15.123:0/3424657916 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd58c102e00 0x7fd58c1031d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd57c004940 tx=0x7fd57c009e30 comp rx=0 tx=0).stop 2024-03-30T04:23:49.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:49.046+0000 7fd592600700 1 -- 172.21.15.123:0/3424657916 shutdown_connections 2024-03-30T04:23:49.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:49.046+0000 7fd592600700 1 --2- 172.21.15.123:0/3424657916 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd58c1037a0 0x7fd58c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:49.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:49.046+0000 7fd592600700 1 --2- 172.21.15.123:0/3424657916 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd58c102e00 0x7fd58c1031d0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:49.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:49.046+0000 7fd592600700 1 -- 172.21.15.123:0/3424657916 >> 172.21.15.123:0/3424657916 conn(0x7fd58c0f89b0 msgr2=0x7fd58c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:23:49.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:49.046+0000 7fd592600700 1 -- 172.21.15.123:0/3424657916 shutdown_connections 2024-03-30T04:23:49.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:49.046+0000 7fd592600700 1 -- 172.21.15.123:0/3424657916 wait complete. 2024-03-30T04:23:49.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:49.047+0000 7fd592600700 1 Processor -- start 2024-03-30T04:23:49.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:49.047+0000 7fd592600700 1 -- start start 2024-03-30T04:23:49.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:49.048+0000 7fd592600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd58c102e00 0x7fd58c115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:49.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:49.048+0000 7fd592600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd58c1037a0 0x7fd58c1161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:49.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:49.048+0000 7fd592600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd58c116810 con 0x7fd58c1037a0 2024-03-30T04:23:49.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:49.048+0000 7fd592600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd58c10fbe0 con 0x7fd58c102e00 2024-03-30T04:23:49.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:49.048+0000 7fd58aa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd58c1037a0 0x7fd58c1161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:49.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:49.048+0000 7fd58aa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd58c1037a0 0x7fd58c1161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37228/0 (socket says 172.21.15.123:37228) 2024-03-30T04:23:49.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:49.048+0000 7fd58aa00700 1 -- 172.21.15.123:0/2157211215 learned_addr learned my addr 172.21.15.123:0/2157211215 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:23:49.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:49.048+0000 7fd58b400700 1 --2- 172.21.15.123:0/2157211215 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd58c102e00 0x7fd58c115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:49.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:49.048+0000 7fd58aa00700 1 -- 172.21.15.123:0/2157211215 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd58c102e00 msgr2=0x7fd58c115c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:49.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:49.048+0000 7fd58aa00700 1 --2- 172.21.15.123:0/2157211215 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd58c102e00 0x7fd58c115c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:49.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:49.048+0000 7fd58aa00700 1 -- 172.21.15.123:0/2157211215 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd57c005040 con 0x7fd58c1037a0 2024-03-30T04:23:49.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:49.048+0000 7fd58b400700 1 --2- 172.21.15.123:0/2157211215 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd58c102e00 0x7fd58c115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:23:49.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:49.049+0000 7fd58aa00700 1 --2- 172.21.15.123:0/2157211215 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd58c1037a0 0x7fd58c1161c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd57800eb40 tx=0x7fd57800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:23:49.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:49.049+0000 7fd583400700 1 -- 172.21.15.123:0/2157211215 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd578007cc0 con 0x7fd58c1037a0 2024-03-30T04:23:49.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:49.049+0000 7fd583400700 1 -- 172.21.15.123:0/2157211215 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd57800f040 con 0x7fd58c1037a0 2024-03-30T04:23:49.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:49.049+0000 7fd592600700 1 -- 172.21.15.123:0/2157211215 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd58c10fe60 con 0x7fd58c1037a0 2024-03-30T04:23:49.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:49.049+0000 7fd583400700 1 -- 172.21.15.123:0/2157211215 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd578009570 con 0x7fd58c1037a0 2024-03-30T04:23:49.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:49.049+0000 7fd592600700 1 -- 172.21.15.123:0/2157211215 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd58c110420 con 0x7fd58c1037a0 2024-03-30T04:23:49.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:49.050+0000 7fd592600700 1 -- 172.21.15.123:0/2157211215 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd58c0fe550 con 0x7fd58c1037a0 2024-03-30T04:23:49.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:49.051+0000 7fd583400700 1 -- 172.21.15.123:0/2157211215 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fd5780096d0 con 0x7fd58c1037a0 2024-03-30T04:23:49.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:49.052+0000 7fd583400700 1 --2- 172.21.15.123:0/2157211215 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fd5700692b0 0x7fd57006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:49.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:49.052+0000 7fd583400700 1 -- 172.21.15.123:0/2157211215 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd578086a10 con 0x7fd58c1037a0 2024-03-30T04:23:49.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:49.052+0000 7fd58b400700 1 --2- 172.21.15.123:0/2157211215 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fd5700692b0 0x7fd57006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:49.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:49.052+0000 7fd58b400700 1 --2- 172.21.15.123:0/2157211215 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fd5700692b0 0x7fd57006b760 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd57c00b0d0 tx=0x7fd57c009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:23:49.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:49.056+0000 7fd583400700 1 -- 172.21.15.123:0/2157211215 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fd57804e190 con 0x7fd58c1037a0 2024-03-30T04:23:49.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:49.253+0000 7fd592600700 1 -- 172.21.15.123:0/2157211215 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fd58c04bed0 con 0x7fd58c1037a0 2024-03-30T04:23:49.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:49.253+0000 7fd583400700 1 -- 172.21.15.123:0/2157211215 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd578056aa0 con 0x7fd58c1037a0 2024-03-30T04:23:49.252 INFO:teuthology.orchestra.run.smithi123.stdout:60129542180 2024-03-30T04:23:49.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:49.256+0000 7fd592600700 1 -- 172.21.15.123:0/2157211215 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fd5700692b0 msgr2=0x7fd57006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:49.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:49.256+0000 7fd592600700 1 --2- 172.21.15.123:0/2157211215 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fd5700692b0 0x7fd57006b760 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd57c00b0d0 tx=0x7fd57c009e30 comp rx=0 tx=0).stop 2024-03-30T04:23:49.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:49.256+0000 7fd592600700 1 -- 172.21.15.123:0/2157211215 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd58c1037a0 msgr2=0x7fd58c1161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:49.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:49.256+0000 7fd592600700 1 --2- 172.21.15.123:0/2157211215 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd58c1037a0 0x7fd58c1161c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd57800eb40 tx=0x7fd57800b040 comp rx=0 tx=0).stop 2024-03-30T04:23:49.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:49.256+0000 7fd592600700 1 -- 172.21.15.123:0/2157211215 shutdown_connections 2024-03-30T04:23:49.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:49.256+0000 7fd592600700 1 --2- 172.21.15.123:0/2157211215 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fd5700692b0 0x7fd57006b760 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:49.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:49.256+0000 7fd592600700 1 --2- 172.21.15.123:0/2157211215 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd58c102e00 0x7fd58c115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:49.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:49.256+0000 7fd592600700 1 --2- 172.21.15.123:0/2157211215 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd58c1037a0 0x7fd58c1161c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:49.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:49.256+0000 7fd592600700 1 -- 172.21.15.123:0/2157211215 >> 172.21.15.123:0/2157211215 conn(0x7fd58c0f89b0 msgr2=0x7fd58c0f9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:23:49.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:49.256+0000 7fd592600700 1 -- 172.21.15.123:0/2157211215 shutdown_connections 2024-03-30T04:23:49.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:49.256+0000 7fd592600700 1 -- 172.21.15.123:0/2157211215 wait complete. 2024-03-30T04:23:50.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:50 smithi135 ceph-mon[44364]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:23:50.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:50 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/2157211215' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-30T04:23:50.390 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:50 smithi123 ceph-mon[35245]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:23:50.390 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:50 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/2157211215' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-30T04:23:50.413 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542178 got 60129542180 for osd.1 2024-03-30T04:23:50.413 DEBUG:teuthology.parallel:result is None 2024-03-30T04:23:51.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.378+0000 7f446ac00700 1 -- 172.21.15.123:0/1921077591 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44641035f0 msgr2=0x7f44640fcc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:51.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.378+0000 7f446ac00700 1 --2- 172.21.15.123:0/1921077591 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44641035f0 0x7f44640fcc60 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4450004940 tx=0x7f4450009e30 comp rx=0 tx=0).stop 2024-03-30T04:23:51.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.378+0000 7f446ac00700 1 -- 172.21.15.123:0/1921077591 shutdown_connections 2024-03-30T04:23:51.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.378+0000 7f446ac00700 1 --2- 172.21.15.123:0/1921077591 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44641035f0 0x7f44640fcc60 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:51.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.378+0000 7f446ac00700 1 --2- 172.21.15.123:0/1921077591 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4464102c50 0x7f4464103020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:51.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.378+0000 7f446ac00700 1 -- 172.21.15.123:0/1921077591 >> 172.21.15.123:0/1921077591 conn(0x7f44640f87e0 msgr2=0x7f44640fabf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:23:51.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.378+0000 7f446ac00700 1 -- 172.21.15.123:0/1921077591 shutdown_connections 2024-03-30T04:23:51.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.378+0000 7f446ac00700 1 -- 172.21.15.123:0/1921077591 wait complete. 2024-03-30T04:23:51.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.379+0000 7f446ac00700 1 Processor -- start 2024-03-30T04:23:51.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.379+0000 7f446ac00700 1 -- start start 2024-03-30T04:23:51.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.380+0000 7f446ac00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4464102c50 0x7f4464113940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:51.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.380+0000 7f446ac00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f44641035f0 0x7f4464113e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:51.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.380+0000 7f446ac00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44641144d0 con 0x7f4464102c50 2024-03-30T04:23:51.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.380+0000 7f446ac00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f446410d8c0 con 0x7f44641035f0 2024-03-30T04:23:51.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.380+0000 7f4469800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4464102c50 0x7f4464113940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:51.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.380+0000 7f4469800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4464102c50 0x7f4464113940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37246/0 (socket says 172.21.15.123:37246) 2024-03-30T04:23:51.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.380+0000 7f4469800700 1 -- 172.21.15.123:0/1284200853 learned_addr learned my addr 172.21.15.123:0/1284200853 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:23:51.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.380+0000 7f4468e00700 1 --2- 172.21.15.123:0/1284200853 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f44641035f0 0x7f4464113e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:51.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.380+0000 7f4469800700 1 -- 172.21.15.123:0/1284200853 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f44641035f0 msgr2=0x7f4464113e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:51.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.380+0000 7f4469800700 1 --2- 172.21.15.123:0/1284200853 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f44641035f0 0x7f4464113e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:51.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.380+0000 7f4469800700 1 -- 172.21.15.123:0/1284200853 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4450005040 con 0x7f4464102c50 2024-03-30T04:23:51.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.381+0000 7f4469800700 1 --2- 172.21.15.123:0/1284200853 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4464102c50 0x7f4464113940 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4458002a50 tx=0x7f44580046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:23:51.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.381+0000 7f4461600700 1 -- 172.21.15.123:0/1284200853 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4458007cc0 con 0x7f4464102c50 2024-03-30T04:23:51.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.381+0000 7f4461600700 1 -- 172.21.15.123:0/1284200853 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f445800f040 con 0x7f4464102c50 2024-03-30T04:23:51.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.381+0000 7f446ac00700 1 -- 172.21.15.123:0/1284200853 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f446410dba0 con 0x7f4464102c50 2024-03-30T04:23:51.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.381+0000 7f4461600700 1 -- 172.21.15.123:0/1284200853 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f445800d9c0 con 0x7f4464102c50 2024-03-30T04:23:51.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.381+0000 7f446ac00700 1 -- 172.21.15.123:0/1284200853 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f446410e130 con 0x7f4464102c50 2024-03-30T04:23:51.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.382+0000 7f446ac00700 1 -- 172.21.15.123:0/1284200853 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44640fe3a0 con 0x7f4464102c50 2024-03-30T04:23:51.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.383+0000 7f4461600700 1 -- 172.21.15.123:0/1284200853 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f445800db60 con 0x7f4464102c50 2024-03-30T04:23:51.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.384+0000 7f4461600700 1 --2- 172.21.15.123:0/1284200853 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f444c0692b0 0x7f444c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:51.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.384+0000 7f4461600700 1 -- 172.21.15.123:0/1284200853 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4458086490 con 0x7f4464102c50 2024-03-30T04:23:51.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.384+0000 7f4468e00700 1 --2- 172.21.15.123:0/1284200853 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f444c0692b0 0x7f444c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:51.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.385+0000 7f4468e00700 1 --2- 172.21.15.123:0/1284200853 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f444c0692b0 0x7f444c06b760 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f446410eba0 tx=0x7f4450009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:23:51.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.391+0000 7f4461600700 1 -- 172.21.15.123:0/1284200853 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f445804d190 con 0x7f4464102c50 2024-03-30T04:23:51.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.591+0000 7f446ac00700 1 -- 172.21.15.123:0/1284200853 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f4464109c50 con 0x7f4464102c50 2024-03-30T04:23:51.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.591+0000 7f4461600700 1 -- 172.21.15.123:0/1284200853 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f44580564f0 con 0x7f4464102c50 2024-03-30T04:23:51.591 INFO:teuthology.orchestra.run.smithi123.stdout:38654705708 2024-03-30T04:23:51.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.594+0000 7f446ac00700 1 -- 172.21.15.123:0/1284200853 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f444c0692b0 msgr2=0x7f444c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:51.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.594+0000 7f446ac00700 1 --2- 172.21.15.123:0/1284200853 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f444c0692b0 0x7f444c06b760 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f446410eba0 tx=0x7f4450009e30 comp rx=0 tx=0).stop 2024-03-30T04:23:51.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.594+0000 7f446ac00700 1 -- 172.21.15.123:0/1284200853 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4464102c50 msgr2=0x7f4464113940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:51.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.594+0000 7f446ac00700 1 --2- 172.21.15.123:0/1284200853 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4464102c50 0x7f4464113940 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4458002a50 tx=0x7f44580046b0 comp rx=0 tx=0).stop 2024-03-30T04:23:51.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.594+0000 7f446ac00700 1 -- 172.21.15.123:0/1284200853 shutdown_connections 2024-03-30T04:23:51.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.594+0000 7f446ac00700 1 --2- 172.21.15.123:0/1284200853 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f444c0692b0 0x7f444c06b760 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:51.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.594+0000 7f446ac00700 1 --2- 172.21.15.123:0/1284200853 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4464102c50 0x7f4464113940 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:51.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.594+0000 7f446ac00700 1 --2- 172.21.15.123:0/1284200853 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f44641035f0 0x7f4464113e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:51.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.594+0000 7f446ac00700 1 -- 172.21.15.123:0/1284200853 >> 172.21.15.123:0/1284200853 conn(0x7f44640f87e0 msgr2=0x7f44640f98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:23:51.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.594+0000 7f446ac00700 1 -- 172.21.15.123:0/1284200853 shutdown_connections 2024-03-30T04:23:51.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.594+0000 7f446ac00700 1 -- 172.21.15.123:0/1284200853 wait complete. 2024-03-30T04:23:52.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.867+0000 7f89b9000700 1 -- 172.21.15.123:0/2111493872 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f89b4102c40 msgr2=0x7f89b4103010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:52.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.867+0000 7f89b9000700 1 --2- 172.21.15.123:0/2111493872 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f89b4102c40 0x7f89b4103010 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f89a4004940 tx=0x7f89a4009e30 comp rx=0 tx=0).stop 2024-03-30T04:23:52.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.867+0000 7f89b9000700 1 -- 172.21.15.123:0/2111493872 shutdown_connections 2024-03-30T04:23:52.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.867+0000 7f89b9000700 1 --2- 172.21.15.123:0/2111493872 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f89b41035e0 0x7f89b40fcc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:52.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.867+0000 7f89b9000700 1 --2- 172.21.15.123:0/2111493872 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f89b4102c40 0x7f89b4103010 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:52.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.867+0000 7f89b9000700 1 -- 172.21.15.123:0/2111493872 >> 172.21.15.123:0/2111493872 conn(0x7f89b40f8830 msgr2=0x7f89b40fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:23:52.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.867+0000 7f89b9000700 1 -- 172.21.15.123:0/2111493872 shutdown_connections 2024-03-30T04:23:52.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.867+0000 7f89b9000700 1 -- 172.21.15.123:0/2111493872 wait complete. 2024-03-30T04:23:52.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.868+0000 7f89b9000700 1 Processor -- start 2024-03-30T04:23:52.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.868+0000 7f89b9000700 1 -- start start 2024-03-30T04:23:52.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.868+0000 7f89b9000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f89b4102c40 0x7f89b410ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:52.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.868+0000 7f89b9000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f89b41035e0 0x7f89b410b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:52.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.868+0000 7f89b9000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b410ef00 con 0x7f89b41035e0 2024-03-30T04:23:52.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.868+0000 7f89b9000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b410b7f0 con 0x7f89b4102c40 2024-03-30T04:23:52.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.869+0000 7f89b2a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f89b41035e0 0x7f89b410b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:52.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.869+0000 7f89b2a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f89b41035e0 0x7f89b410b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37262/0 (socket says 172.21.15.123:37262) 2024-03-30T04:23:52.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.869+0000 7f89b2a00700 1 -- 172.21.15.123:0/4052137881 learned_addr learned my addr 172.21.15.123:0/4052137881 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:23:52.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.869+0000 7f89b3400700 1 --2- 172.21.15.123:0/4052137881 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f89b4102c40 0x7f89b410ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:52.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.869+0000 7f89b2a00700 1 -- 172.21.15.123:0/4052137881 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f89b4102c40 msgr2=0x7f89b410ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:52.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.869+0000 7f89b2a00700 1 --2- 172.21.15.123:0/4052137881 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f89b4102c40 0x7f89b410ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:52.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.869+0000 7f89b2a00700 1 -- 172.21.15.123:0/4052137881 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89a4005040 con 0x7f89b41035e0 2024-03-30T04:23:52.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.869+0000 7f89b2a00700 1 --2- 172.21.15.123:0/4052137881 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f89b41035e0 0x7f89b410b2b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f89a0002a50 tx=0x7f89a00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:23:52.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.869+0000 7f89ab400700 1 -- 172.21.15.123:0/4052137881 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f89a0007cc0 con 0x7f89b41035e0 2024-03-30T04:23:52.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.869+0000 7f89ab400700 1 -- 172.21.15.123:0/4052137881 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f89a000f040 con 0x7f89b41035e0 2024-03-30T04:23:52.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.869+0000 7f89b9000700 1 -- 172.21.15.123:0/4052137881 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89b410bad0 con 0x7f89b41035e0 2024-03-30T04:23:52.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.869+0000 7f89ab400700 1 -- 172.21.15.123:0/4052137881 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f89a00095d0 con 0x7f89b41035e0 2024-03-30T04:23:52.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.870+0000 7f89b9000700 1 -- 172.21.15.123:0/4052137881 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89b411e450 con 0x7f89b41035e0 2024-03-30T04:23:52.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.871+0000 7f89b9000700 1 -- 172.21.15.123:0/4052137881 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89b40fe390 con 0x7f89b41035e0 2024-03-30T04:23:52.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.871+0000 7f89ab400700 1 -- 172.21.15.123:0/4052137881 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f89a0009730 con 0x7f89b41035e0 2024-03-30T04:23:52.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.872+0000 7f89ab400700 1 --2- 172.21.15.123:0/4052137881 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f89980692b0 0x7f899806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:52.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.872+0000 7f89ab400700 1 -- 172.21.15.123:0/4052137881 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f89a0086b30 con 0x7f89b41035e0 2024-03-30T04:23:52.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.872+0000 7f89b3400700 1 --2- 172.21.15.123:0/4052137881 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f89980692b0 0x7f899806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:52.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.873+0000 7f89b3400700 1 --2- 172.21.15.123:0/4052137881 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f89980692b0 0x7f899806b760 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f89a400b3b0 tx=0x7f89a4009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:23:52.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:51.876+0000 7f89ab400700 1 -- 172.21.15.123:0/4052137881 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f89a0053100 con 0x7f89b41035e0 2024-03-30T04:23:52.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.076+0000 7f89b9000700 1 -- 172.21.15.123:0/4052137881 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f89b405f5f0 con 0x7f89b41035e0 2024-03-30T04:23:52.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.076+0000 7f89ab400700 1 -- 172.21.15.123:0/4052137881 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f89a0056720 con 0x7f89b41035e0 2024-03-30T04:23:52.075 INFO:teuthology.orchestra.run.smithi123.stdout:81604378655 2024-03-30T04:23:52.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.078+0000 7f89b9000700 1 -- 172.21.15.123:0/4052137881 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f89980692b0 msgr2=0x7f899806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:52.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.078+0000 7f89b9000700 1 --2- 172.21.15.123:0/4052137881 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f89980692b0 0x7f899806b760 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f89a400b3b0 tx=0x7f89a4009e30 comp rx=0 tx=0).stop 2024-03-30T04:23:52.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.078+0000 7f89b9000700 1 -- 172.21.15.123:0/4052137881 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f89b41035e0 msgr2=0x7f89b410b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:52.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.078+0000 7f89b9000700 1 --2- 172.21.15.123:0/4052137881 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f89b41035e0 0x7f89b410b2b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f89a0002a50 tx=0x7f89a00046b0 comp rx=0 tx=0).stop 2024-03-30T04:23:52.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.078+0000 7f89b9000700 1 -- 172.21.15.123:0/4052137881 shutdown_connections 2024-03-30T04:23:52.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.078+0000 7f89b9000700 1 --2- 172.21.15.123:0/4052137881 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f89980692b0 0x7f899806b760 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:52.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.078+0000 7f89b9000700 1 --2- 172.21.15.123:0/4052137881 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f89b4102c40 0x7f89b410ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:52.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.078+0000 7f89b9000700 1 --2- 172.21.15.123:0/4052137881 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f89b41035e0 0x7f89b410b2b0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:52.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.078+0000 7f89b9000700 1 -- 172.21.15.123:0/4052137881 >> 172.21.15.123:0/4052137881 conn(0x7f89b40f8830 msgr2=0x7f89b40f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:23:52.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.079+0000 7f89b9000700 1 -- 172.21.15.123:0/4052137881 shutdown_connections 2024-03-30T04:23:52.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.079+0000 7f89b9000700 1 -- 172.21.15.123:0/4052137881 wait complete. 2024-03-30T04:23:52.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:52 smithi135 ceph-mon[44364]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:23:52.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:52 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/1284200853' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-30T04:23:52.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.330+0000 7fe3bbc00700 1 -- 172.21.15.123:0/3232934355 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3b4102e00 msgr2=0x7fe3b41031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:52.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.330+0000 7fe3bbc00700 1 --2- 172.21.15.123:0/3232934355 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3b4102e00 0x7fe3b41031d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe3a8004940 tx=0x7fe3a8009e30 comp rx=0 tx=0).stop 2024-03-30T04:23:52.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.330+0000 7fe3bbc00700 1 -- 172.21.15.123:0/3232934355 shutdown_connections 2024-03-30T04:23:52.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.330+0000 7fe3bbc00700 1 --2- 172.21.15.123:0/3232934355 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe3b41037a0 0x7fe3b40fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:52.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.330+0000 7fe3bbc00700 1 --2- 172.21.15.123:0/3232934355 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3b4102e00 0x7fe3b41031d0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:52.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.330+0000 7fe3bbc00700 1 -- 172.21.15.123:0/3232934355 >> 172.21.15.123:0/3232934355 conn(0x7fe3b40f89b0 msgr2=0x7fe3b40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:23:52.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.330+0000 7fe3bbc00700 1 -- 172.21.15.123:0/3232934355 shutdown_connections 2024-03-30T04:23:52.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.330+0000 7fe3bbc00700 1 -- 172.21.15.123:0/3232934355 wait complete. 2024-03-30T04:23:52.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.331+0000 7fe3bbc00700 1 Processor -- start 2024-03-30T04:23:52.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.331+0000 7fe3bbc00700 1 -- start start 2024-03-30T04:23:52.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.331+0000 7fe3bbc00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe3b4102e00 0x7fe3b4115d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:52.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.331+0000 7fe3bbc00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3b41037a0 0x7fe3b4116250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:52.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.331+0000 7fe3bbc00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b41168a0 con 0x7fe3b41037a0 2024-03-30T04:23:52.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.331+0000 7fe3bbc00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b410fc70 con 0x7fe3b4102e00 2024-03-30T04:23:52.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.332+0000 7fe3b3e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3b41037a0 0x7fe3b4116250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:52.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.332+0000 7fe3b3e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3b41037a0 0x7fe3b4116250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37278/0 (socket says 172.21.15.123:37278) 2024-03-30T04:23:52.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.332+0000 7fe3b3e00700 1 -- 172.21.15.123:0/3556081901 learned_addr learned my addr 172.21.15.123:0/3556081901 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:23:52.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.332+0000 7fe3b9200700 1 --2- 172.21.15.123:0/3556081901 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe3b4102e00 0x7fe3b4115d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:52.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.332+0000 7fe3b3e00700 1 -- 172.21.15.123:0/3556081901 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe3b4102e00 msgr2=0x7fe3b4115d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:52.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.332+0000 7fe3b3e00700 1 --2- 172.21.15.123:0/3556081901 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe3b4102e00 0x7fe3b4115d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:52.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.332+0000 7fe3b3e00700 1 -- 172.21.15.123:0/3556081901 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a8005040 con 0x7fe3b41037a0 2024-03-30T04:23:52.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.332+0000 7fe3b9200700 1 --2- 172.21.15.123:0/3556081901 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe3b4102e00 0x7fe3b4115d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:23:52.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.332+0000 7fe3b3e00700 1 --2- 172.21.15.123:0/3556081901 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3b41037a0 0x7fe3b4116250 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0002a50 tx=0x7fe3a0004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:23:52.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.332+0000 7fe3b0c00700 1 -- 172.21.15.123:0/3556081901 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3a0007cc0 con 0x7fe3b41037a0 2024-03-30T04:23:52.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.332+0000 7fe3b0c00700 1 -- 172.21.15.123:0/3556081901 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe3a000f040 con 0x7fe3b41037a0 2024-03-30T04:23:52.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.333+0000 7fe3bbc00700 1 -- 172.21.15.123:0/3556081901 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3b410ff50 con 0x7fe3b41037a0 2024-03-30T04:23:52.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.333+0000 7fe3b0c00700 1 -- 172.21.15.123:0/3556081901 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3a000d9a0 con 0x7fe3b41037a0 2024-03-30T04:23:52.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.333+0000 7fe3bbc00700 1 -- 172.21.15.123:0/3556081901 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3b4110510 con 0x7fe3b41037a0 2024-03-30T04:23:52.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.334+0000 7fe3bbc00700 1 -- 172.21.15.123:0/3556081901 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3b40fe550 con 0x7fe3b41037a0 2024-03-30T04:23:52.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.334+0000 7fe3b0c00700 1 -- 172.21.15.123:0/3556081901 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fe3a000dc60 con 0x7fe3b41037a0 2024-03-30T04:23:52.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.336+0000 7fe3b0c00700 1 --2- 172.21.15.123:0/3556081901 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fe39c0692b0 0x7fe39c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:52.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.336+0000 7fe3b0c00700 1 -- 172.21.15.123:0/3556081901 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fe3a0086d10 con 0x7fe3b41037a0 2024-03-30T04:23:52.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.336+0000 7fe3b9200700 1 --2- 172.21.15.123:0/3556081901 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fe39c0692b0 0x7fe39c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:52.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.336+0000 7fe3b9200700 1 --2- 172.21.15.123:0/3556081901 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fe39c0692b0 0x7fe39c06b760 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe3a800b840 tx=0x7fe3a8009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:23:52.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.340+0000 7fe3b0c00700 1 -- 172.21.15.123:0/3556081901 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fe3a0056cf0 con 0x7fe3b41037a0 2024-03-30T04:23:52.453 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:23:52.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:52 smithi123 ceph-mon[35245]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:23:52.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:52 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/1284200853' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-30T04:23:52.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.567+0000 7fe3bbc00700 1 -- 172.21.15.123:0/3556081901 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fe3b404bed0 con 0x7fe3b41037a0 2024-03-30T04:23:52.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.567+0000 7fe3b0c00700 1 -- 172.21.15.123:0/3556081901 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fe3a0056880 con 0x7fe3b41037a0 2024-03-30T04:23:52.566 INFO:teuthology.orchestra.run.smithi123.stdout:167503724555 2024-03-30T04:23:52.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.569+0000 7fe3bbc00700 1 -- 172.21.15.123:0/3556081901 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fe39c0692b0 msgr2=0x7fe39c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:52.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.569+0000 7fe3bbc00700 1 --2- 172.21.15.123:0/3556081901 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fe39c0692b0 0x7fe39c06b760 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe3a800b840 tx=0x7fe3a8009e30 comp rx=0 tx=0).stop 2024-03-30T04:23:52.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.569+0000 7fe3bbc00700 1 -- 172.21.15.123:0/3556081901 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3b41037a0 msgr2=0x7fe3b4116250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:52.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.569+0000 7fe3bbc00700 1 --2- 172.21.15.123:0/3556081901 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3b41037a0 0x7fe3b4116250 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0002a50 tx=0x7fe3a0004600 comp rx=0 tx=0).stop 2024-03-30T04:23:52.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.569+0000 7fe3bbc00700 1 -- 172.21.15.123:0/3556081901 shutdown_connections 2024-03-30T04:23:52.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.569+0000 7fe3bbc00700 1 --2- 172.21.15.123:0/3556081901 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fe39c0692b0 0x7fe39c06b760 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:52.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.569+0000 7fe3bbc00700 1 --2- 172.21.15.123:0/3556081901 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe3b4102e00 0x7fe3b4115d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:52.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.569+0000 7fe3bbc00700 1 --2- 172.21.15.123:0/3556081901 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3b41037a0 0x7fe3b4116250 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:52.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.569+0000 7fe3bbc00700 1 -- 172.21.15.123:0/3556081901 >> 172.21.15.123:0/3556081901 conn(0x7fe3b40f89b0 msgr2=0x7fe3b40704f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:23:52.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.570+0000 7fe3bbc00700 1 -- 172.21.15.123:0/3556081901 shutdown_connections 2024-03-30T04:23:52.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:52.570+0000 7fe3bbc00700 1 -- 172.21.15.123:0/3556081901 wait complete. 2024-03-30T04:23:52.945 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378653 got 81604378655 for osd.2 2024-03-30T04:23:52.946 DEBUG:teuthology.parallel:result is None 2024-03-30T04:23:53.212 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:53 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/4052137881' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-30T04:23:53.212 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:53 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/3556081901' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-30T04:23:53.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:53 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/4052137881' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-30T04:23:53.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:53 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/3556081901' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-30T04:23:54.074 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:23:54.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:54 smithi135 ceph-mon[44364]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:23:54.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:54 smithi123 ceph-mon[35245]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:23:54.501 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705706 got 38654705708 for osd.0 2024-03-30T04:23:54.501 DEBUG:teuthology.parallel:result is None 2024-03-30T04:23:55.174 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724553 got 167503724555 for osd.5 2024-03-30T04:23:55.175 DEBUG:teuthology.parallel:result is None 2024-03-30T04:23:55.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:55.414+0000 7feb7b200700 1 -- 172.21.15.123:0/1055867629 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb741037a0 msgr2=0x7feb740fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:55.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:55.414+0000 7feb7b200700 1 --2- 172.21.15.123:0/1055867629 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb741037a0 0x7feb740fce10 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7feb60004940 tx=0x7feb60009e30 comp rx=0 tx=0).stop 2024-03-30T04:23:55.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:55.414+0000 7feb7b200700 1 -- 172.21.15.123:0/1055867629 shutdown_connections 2024-03-30T04:23:55.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:55.414+0000 7feb7b200700 1 --2- 172.21.15.123:0/1055867629 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb741037a0 0x7feb740fce10 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:55.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:55.414+0000 7feb7b200700 1 --2- 172.21.15.123:0/1055867629 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feb74102e00 0x7feb741031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:55.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:55.414+0000 7feb7b200700 1 -- 172.21.15.123:0/1055867629 >> 172.21.15.123:0/1055867629 conn(0x7feb740f89b0 msgr2=0x7feb740fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:23:55.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:55.414+0000 7feb7b200700 1 -- 172.21.15.123:0/1055867629 shutdown_connections 2024-03-30T04:23:55.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:55.414+0000 7feb7b200700 1 -- 172.21.15.123:0/1055867629 wait complete. 2024-03-30T04:23:55.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:55.415+0000 7feb7b200700 1 Processor -- start 2024-03-30T04:23:55.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:55.415+0000 7feb7b200700 1 -- start start 2024-03-30T04:23:55.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:55.415+0000 7feb7b200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feb74102e00 0x7feb740702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:55.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:55.415+0000 7feb7b200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb741037a0 0x7feb74070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:55.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:55.415+0000 7feb7b200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb74070e20 con 0x7feb741037a0 2024-03-30T04:23:55.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:55.415+0000 7feb7b200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb74070f60 con 0x7feb74102e00 2024-03-30T04:23:55.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:55.415+0000 7feb73400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb741037a0 0x7feb74070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:55.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:55.415+0000 7feb73400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb741037a0 0x7feb74070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37664/0 (socket says 172.21.15.123:37664) 2024-03-30T04:23:55.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:55.415+0000 7feb73400700 1 -- 172.21.15.123:0/1062231679 learned_addr learned my addr 172.21.15.123:0/1062231679 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:23:55.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:55.416+0000 7feb73400700 1 -- 172.21.15.123:0/1062231679 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feb74102e00 msgr2=0x7feb740702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:55.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:55.416+0000 7feb73400700 1 --2- 172.21.15.123:0/1062231679 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feb74102e00 0x7feb740702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:55.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:55.416+0000 7feb73400700 1 -- 172.21.15.123:0/1062231679 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb60005040 con 0x7feb741037a0 2024-03-30T04:23:55.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:55.416+0000 7feb73400700 1 --2- 172.21.15.123:0/1062231679 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb741037a0 0x7feb74070800 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7feb60009380 tx=0x7feb60009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:23:55.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:55.416+0000 7feb6be00700 1 -- 172.21.15.123:0/1062231679 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb6000e040 con 0x7feb741037a0 2024-03-30T04:23:55.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:55.416+0000 7feb6be00700 1 -- 172.21.15.123:0/1062231679 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feb60013420 con 0x7feb741037a0 2024-03-30T04:23:55.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:55.416+0000 7feb7b200700 1 -- 172.21.15.123:0/1062231679 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb74119de0 con 0x7feb741037a0 2024-03-30T04:23:55.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:55.416+0000 7feb6be00700 1 -- 172.21.15.123:0/1062231679 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb6001c630 con 0x7feb741037a0 2024-03-30T04:23:55.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:55.416+0000 7feb7b200700 1 -- 172.21.15.123:0/1062231679 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb7411a2c0 con 0x7feb741037a0 2024-03-30T04:23:55.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:55.417+0000 7feb7b200700 1 -- 172.21.15.123:0/1062231679 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb740fe550 con 0x7feb741037a0 2024-03-30T04:23:55.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:55.418+0000 7feb6be00700 1 -- 172.21.15.123:0/1062231679 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7feb600139e0 con 0x7feb741037a0 2024-03-30T04:23:55.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:55.419+0000 7feb6be00700 1 --2- 172.21.15.123:0/1062231679 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7feb580692a0 0x7feb5806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:55.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:55.419+0000 7feb6be00700 1 -- 172.21.15.123:0/1062231679 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7feb60087f20 con 0x7feb741037a0 2024-03-30T04:23:55.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:55.419+0000 7feb73e00700 1 --2- 172.21.15.123:0/1062231679 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7feb580692a0 0x7feb5806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:55.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:55.420+0000 7feb73e00700 1 --2- 172.21.15.123:0/1062231679 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7feb580692a0 0x7feb5806b750 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7feb64006fd0 tx=0x7feb64006db0 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:23:55.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:55.424+0000 7feb6be00700 1 -- 172.21.15.123:0/1062231679 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7feb600ba190 con 0x7feb741037a0 2024-03-30T04:23:55.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:55.635+0000 7feb7b200700 1 -- 172.21.15.123:0/1062231679 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7feb7404bed0 con 0x7feb741037a0 2024-03-30T04:23:55.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:55.635+0000 7feb6be00700 1 -- 172.21.15.123:0/1062231679 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7feb60054630 con 0x7feb741037a0 2024-03-30T04:23:55.634 INFO:teuthology.orchestra.run.smithi123.stdout:146028888081 2024-03-30T04:23:55.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:55.637+0000 7feb7b200700 1 -- 172.21.15.123:0/1062231679 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7feb580692a0 msgr2=0x7feb5806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:55.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:55.637+0000 7feb7b200700 1 --2- 172.21.15.123:0/1062231679 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7feb580692a0 0x7feb5806b750 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7feb64006fd0 tx=0x7feb64006db0 comp rx=0 tx=0).stop 2024-03-30T04:23:55.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:55.637+0000 7feb7b200700 1 -- 172.21.15.123:0/1062231679 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb741037a0 msgr2=0x7feb74070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:55.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:55.637+0000 7feb7b200700 1 --2- 172.21.15.123:0/1062231679 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb741037a0 0x7feb74070800 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7feb60009380 tx=0x7feb60009e30 comp rx=0 tx=0).stop 2024-03-30T04:23:55.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:55.638+0000 7feb7b200700 1 -- 172.21.15.123:0/1062231679 shutdown_connections 2024-03-30T04:23:55.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:55.638+0000 7feb7b200700 1 --2- 172.21.15.123:0/1062231679 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7feb580692a0 0x7feb5806b750 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:55.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:55.638+0000 7feb7b200700 1 --2- 172.21.15.123:0/1062231679 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feb74102e00 0x7feb740702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:55.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:55.638+0000 7feb7b200700 1 --2- 172.21.15.123:0/1062231679 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb741037a0 0x7feb74070800 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:55.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:55.638+0000 7feb7b200700 1 -- 172.21.15.123:0/1062231679 >> 172.21.15.123:0/1062231679 conn(0x7feb740f89b0 msgr2=0x7feb740f99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:23:55.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:55.638+0000 7feb7b200700 1 -- 172.21.15.123:0/1062231679 shutdown_connections 2024-03-30T04:23:55.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:55.638+0000 7feb7b200700 1 -- 172.21.15.123:0/1062231679 wait complete. 2024-03-30T04:23:56.344 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:56 smithi123 ceph-mon[35245]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:23:56.344 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:56 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/1062231679' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-30T04:23:56.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:56 smithi135 ceph-mon[44364]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:23:56.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:56 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/1062231679' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-30T04:23:56.647 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888079 got 146028888081 for osd.4 2024-03-30T04:23:56.648 DEBUG:teuthology.parallel:result is None 2024-03-30T04:23:57.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:57 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:23:57.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:57 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:23:57.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:57.649+0000 7fe4a7e00700 1 -- 172.21.15.123:0/3410616107 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4a0102e00 msgr2=0x7fe4a01031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:58.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:57.649+0000 7fe4a7e00700 1 --2- 172.21.15.123:0/3410616107 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4a0102e00 0x7fe4a01031d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe494004940 tx=0x7fe494009e30 comp rx=0 tx=0).stop 2024-03-30T04:23:58.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:57.649+0000 7fe4a7e00700 1 -- 172.21.15.123:0/3410616107 shutdown_connections 2024-03-30T04:23:58.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:57.649+0000 7fe4a7e00700 1 --2- 172.21.15.123:0/3410616107 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe4a01037a0 0x7fe4a00fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:58.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:57.649+0000 7fe4a7e00700 1 --2- 172.21.15.123:0/3410616107 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4a0102e00 0x7fe4a01031d0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:58.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:57.649+0000 7fe4a7e00700 1 -- 172.21.15.123:0/3410616107 >> 172.21.15.123:0/3410616107 conn(0x7fe4a00f89b0 msgr2=0x7fe4a00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:23:58.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:57.649+0000 7fe4a7e00700 1 -- 172.21.15.123:0/3410616107 shutdown_connections 2024-03-30T04:23:58.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:57.649+0000 7fe4a7e00700 1 -- 172.21.15.123:0/3410616107 wait complete. 2024-03-30T04:23:58.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:57.650+0000 7fe4a7e00700 1 Processor -- start 2024-03-30T04:23:58.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:57.650+0000 7fe4a7e00700 1 -- start start 2024-03-30T04:23:58.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:57.651+0000 7fe4a7e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe4a0102e00 0x7fe4a01138b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:58.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:57.651+0000 7fe4a7e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4a01037a0 0x7fe4a0113df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:58.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:57.651+0000 7fe4a7e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a0114440 con 0x7fe4a01037a0 2024-03-30T04:23:58.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:57.651+0000 7fe4a7e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a010da40 con 0x7fe4a0102e00 2024-03-30T04:23:58.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:57.651+0000 7fe4a4a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4a01037a0 0x7fe4a0113df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:58.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:57.651+0000 7fe4a4a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4a01037a0 0x7fe4a0113df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37690/0 (socket says 172.21.15.123:37690) 2024-03-30T04:23:58.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:57.651+0000 7fe4a4a00700 1 -- 172.21.15.123:0/2842451802 learned_addr learned my addr 172.21.15.123:0/2842451802 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:23:58.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:57.651+0000 7fe4a5400700 1 --2- 172.21.15.123:0/2842451802 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe4a0102e00 0x7fe4a01138b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:58.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:57.651+0000 7fe4a4a00700 1 -- 172.21.15.123:0/2842451802 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe4a0102e00 msgr2=0x7fe4a01138b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:58.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:57.651+0000 7fe4a4a00700 1 --2- 172.21.15.123:0/2842451802 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe4a0102e00 0x7fe4a01138b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:58.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:57.651+0000 7fe4a4a00700 1 -- 172.21.15.123:0/2842451802 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe494005040 con 0x7fe4a01037a0 2024-03-30T04:23:58.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:57.652+0000 7fe4a4a00700 1 --2- 172.21.15.123:0/2842451802 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4a01037a0 0x7fe4a0113df0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe48c00c890 tx=0x7fe48c0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:23:58.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:57.652+0000 7fe49d600700 1 -- 172.21.15.123:0/2842451802 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe48c00a700 con 0x7fe4a01037a0 2024-03-30T04:23:58.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:57.652+0000 7fe49d600700 1 -- 172.21.15.123:0/2842451802 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe48c00ad10 con 0x7fe4a01037a0 2024-03-30T04:23:58.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:57.652+0000 7fe4a7e00700 1 -- 172.21.15.123:0/2842451802 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4a010dd20 con 0x7fe4a01037a0 2024-03-30T04:23:58.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:57.652+0000 7fe49d600700 1 -- 172.21.15.123:0/2842451802 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe48c012e70 con 0x7fe4a01037a0 2024-03-30T04:23:58.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:57.652+0000 7fe4a7e00700 1 -- 172.21.15.123:0/2842451802 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4a010e2e0 con 0x7fe4a01037a0 2024-03-30T04:23:58.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:57.653+0000 7fe4a7e00700 1 -- 172.21.15.123:0/2842451802 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4a00fe550 con 0x7fe4a01037a0 2024-03-30T04:23:58.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:57.654+0000 7fe49d600700 1 -- 172.21.15.123:0/2842451802 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fe48c00a860 con 0x7fe4a01037a0 2024-03-30T04:23:58.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:57.655+0000 7fe49d600700 1 --2- 172.21.15.123:0/2842451802 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fe4880692a0 0x7fe48806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:23:58.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:57.655+0000 7fe49d600700 1 -- 172.21.15.123:0/2842451802 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fe48c086e90 con 0x7fe4a01037a0 2024-03-30T04:23:58.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:57.655+0000 7fe4a5400700 1 --2- 172.21.15.123:0/2842451802 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fe4880692a0 0x7fe48806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:23:58.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:57.656+0000 7fe4a5400700 1 --2- 172.21.15.123:0/2842451802 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fe4880692a0 0x7fe48806b750 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe49400b840 tx=0x7fe494009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:23:58.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:57.659+0000 7fe49d600700 1 -- 172.21.15.123:0/2842451802 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fe48c056f70 con 0x7fe4a01037a0 2024-03-30T04:23:58.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:57.854+0000 7fe4a7e00700 1 -- 172.21.15.123:0/2842451802 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fe4a0109dc0 con 0x7fe4a01037a0 2024-03-30T04:23:58.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:57.854+0000 7fe49d600700 1 -- 172.21.15.123:0/2842451802 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fe48c056b00 con 0x7fe4a01037a0 2024-03-30T04:23:58.037 INFO:teuthology.orchestra.run.smithi123.stdout:111669149722 2024-03-30T04:23:58.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:57.857+0000 7fe4a7e00700 1 -- 172.21.15.123:0/2842451802 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fe4880692a0 msgr2=0x7fe48806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:58.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:57.857+0000 7fe4a7e00700 1 --2- 172.21.15.123:0/2842451802 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fe4880692a0 0x7fe48806b750 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe49400b840 tx=0x7fe494009e30 comp rx=0 tx=0).stop 2024-03-30T04:23:58.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:57.857+0000 7fe4a7e00700 1 -- 172.21.15.123:0/2842451802 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4a01037a0 msgr2=0x7fe4a0113df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:23:58.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:57.857+0000 7fe4a7e00700 1 --2- 172.21.15.123:0/2842451802 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4a01037a0 0x7fe4a0113df0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe48c00c890 tx=0x7fe48c0044c0 comp rx=0 tx=0).stop 2024-03-30T04:23:58.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:57.857+0000 7fe4a7e00700 1 -- 172.21.15.123:0/2842451802 shutdown_connections 2024-03-30T04:23:58.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:57.857+0000 7fe4a7e00700 1 --2- 172.21.15.123:0/2842451802 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fe4880692a0 0x7fe48806b750 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:58.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:57.857+0000 7fe4a7e00700 1 --2- 172.21.15.123:0/2842451802 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe4a0102e00 0x7fe4a01138b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:58.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:57.857+0000 7fe4a7e00700 1 --2- 172.21.15.123:0/2842451802 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4a01037a0 0x7fe4a0113df0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:23:58.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:57.857+0000 7fe4a7e00700 1 -- 172.21.15.123:0/2842451802 >> 172.21.15.123:0/2842451802 conn(0x7fe4a00f89b0 msgr2=0x7fe4a00f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:23:58.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:57.857+0000 7fe4a7e00700 1 -- 172.21.15.123:0/2842451802 shutdown_connections 2024-03-30T04:23:58.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:23:57.857+0000 7fe4a7e00700 1 -- 172.21.15.123:0/2842451802 wait complete. 2024-03-30T04:23:58.166 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:58 smithi123 ceph-mon[35245]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:23:58.166 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:23:58 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/2842451802' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-30T04:23:58.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:58 smithi135 ceph-mon[44364]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:23:58.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:23:58 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/2842451802' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-30T04:23:58.492 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149719 got 111669149722 for osd.3 2024-03-30T04:23:58.492 DEBUG:teuthology.parallel:result is None 2024-03-30T04:23:58.492 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-30T04:23:58.492 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-03-30T04:23:58.920 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:24:00.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:00 smithi135 ceph-mon[44364]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:24:00.435 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:00 smithi123 ceph-mon[35245]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:24:00.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:00.712+0000 7fef4c000700 1 -- 172.21.15.123:0/2491715116 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef44070360 msgr2=0x7fef44070730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:00.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:00.712+0000 7fef4c000700 1 --2- 172.21.15.123:0/2491715116 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef44070360 0x7fef44070730 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fef38004970 tx=0x7fef38009e30 comp rx=0 tx=0).stop 2024-03-30T04:24:00.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:00.712+0000 7fef4c000700 1 -- 172.21.15.123:0/2491715116 shutdown_connections 2024-03-30T04:24:00.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:00.712+0000 7fef4c000700 1 --2- 172.21.15.123:0/2491715116 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fef44070c70 0x7fef44109110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:00.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:00.712+0000 7fef4c000700 1 --2- 172.21.15.123:0/2491715116 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef44070360 0x7fef44070730 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:00.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:00.712+0000 7fef4c000700 1 -- 172.21.15.123:0/2491715116 >> 172.21.15.123:0/2491715116 conn(0x7fef440f89b0 msgr2=0x7fef440fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:24:00.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:00.712+0000 7fef4c000700 1 -- 172.21.15.123:0/2491715116 shutdown_connections 2024-03-30T04:24:00.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:00.712+0000 7fef4c000700 1 -- 172.21.15.123:0/2491715116 wait complete. 2024-03-30T04:24:00.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:00.713+0000 7fef4c000700 1 Processor -- start 2024-03-30T04:24:00.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:00.713+0000 7fef4c000700 1 -- start start 2024-03-30T04:24:00.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:00.714+0000 7fef4c000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef44070360 0x7fef44115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:24:00.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:00.714+0000 7fef4c000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fef44070c70 0x7fef441161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:24:00.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:00.714+0000 7fef4c000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef44116810 con 0x7fef44070360 2024-03-30T04:24:00.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:00.714+0000 7fef4c000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef4410fbe0 con 0x7fef44070c70 2024-03-30T04:24:00.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:00.714+0000 7fef49600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef44070360 0x7fef44115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:24:00.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:00.714+0000 7fef49600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef44070360 0x7fef44115c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37720/0 (socket says 172.21.15.123:37720) 2024-03-30T04:24:00.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:00.714+0000 7fef49600700 1 -- 172.21.15.123:0/1667732852 learned_addr learned my addr 172.21.15.123:0/1667732852 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:24:00.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:00.714+0000 7fef48c00700 1 --2- 172.21.15.123:0/1667732852 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fef44070c70 0x7fef441161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:24:00.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:00.714+0000 7fef49600700 1 -- 172.21.15.123:0/1667732852 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fef44070c70 msgr2=0x7fef441161c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:00.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:00.714+0000 7fef49600700 1 --2- 172.21.15.123:0/1667732852 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fef44070c70 0x7fef441161c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:00.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:00.714+0000 7fef49600700 1 -- 172.21.15.123:0/1667732852 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef38005040 con 0x7fef44070360 2024-03-30T04:24:00.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:00.714+0000 7fef48c00700 1 --2- 172.21.15.123:0/1667732852 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fef44070c70 0x7fef441161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:24:00.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:00.715+0000 7fef49600700 1 --2- 172.21.15.123:0/1667732852 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef44070360 0x7fef44115c80 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fef38005010 tx=0x7fef38009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:24:00.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:00.715+0000 7fef41600700 1 -- 172.21.15.123:0/1667732852 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef3800e040 con 0x7fef44070360 2024-03-30T04:24:00.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:00.715+0000 7fef4c000700 1 -- 172.21.15.123:0/1667732852 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef4410fe60 con 0x7fef44070360 2024-03-30T04:24:00.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:00.715+0000 7fef41600700 1 -- 172.21.15.123:0/1667732852 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fef38009930 con 0x7fef44070360 2024-03-30T04:24:00.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:00.715+0000 7fef41600700 1 -- 172.21.15.123:0/1667732852 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef3801c3f0 con 0x7fef44070360 2024-03-30T04:24:00.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:00.715+0000 7fef4c000700 1 -- 172.21.15.123:0/1667732852 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef441103a0 con 0x7fef44070360 2024-03-30T04:24:00.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:00.716+0000 7fef4c000700 1 -- 172.21.15.123:0/1667732852 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef44103b90 con 0x7fef44070360 2024-03-30T04:24:00.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:00.717+0000 7fef41600700 1 -- 172.21.15.123:0/1667732852 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fef380136b0 con 0x7fef44070360 2024-03-30T04:24:00.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:00.718+0000 7fef41600700 1 --2- 172.21.15.123:0/1667732852 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fef2c0692b0 0x7fef2c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:24:00.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:00.718+0000 7fef41600700 1 -- 172.21.15.123:0/1667732852 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fef380880d0 con 0x7fef44070360 2024-03-30T04:24:00.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:00.718+0000 7fef48c00700 1 --2- 172.21.15.123:0/1667732852 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fef2c0692b0 0x7fef2c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:24:00.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:00.718+0000 7fef48c00700 1 --2- 172.21.15.123:0/1667732852 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fef2c0692b0 0x7fef2c06b760 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fef30004600 tx=0x7fef3000b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:24:00.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:00.722+0000 7fef41600700 1 -- 172.21.15.123:0/1667732852 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fef38058130 con 0x7fef44070360 2024-03-30T04:24:00.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:00.910+0000 7fef4c000700 1 -- 172.21.15.123:0/1667732852 --> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fef4410c020 con 0x7fef2c0692b0 2024-03-30T04:24:00.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:00.912+0000 7fef41600700 1 -- 172.21.15.123:0/1667732852 <== mgr.14206 v2:172.21.15.123:6800/1564935593 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+27815 (secure 0 0 0) 0x7fef4410c020 con 0x7fef2c0692b0 2024-03-30T04:24:00.911 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:24:00.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:00.914+0000 7fef4c000700 1 -- 172.21.15.123:0/1667732852 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fef2c0692b0 msgr2=0x7fef2c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:00.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:00.914+0000 7fef4c000700 1 --2- 172.21.15.123:0/1667732852 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fef2c0692b0 0x7fef2c06b760 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fef30004600 tx=0x7fef3000b040 comp rx=0 tx=0).stop 2024-03-30T04:24:00.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:00.914+0000 7fef4c000700 1 -- 172.21.15.123:0/1667732852 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef44070360 msgr2=0x7fef44115c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:00.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:00.914+0000 7fef4c000700 1 --2- 172.21.15.123:0/1667732852 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef44070360 0x7fef44115c80 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fef38005010 tx=0x7fef38009e30 comp rx=0 tx=0).stop 2024-03-30T04:24:00.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:00.914+0000 7fef4c000700 1 -- 172.21.15.123:0/1667732852 shutdown_connections 2024-03-30T04:24:00.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:00.914+0000 7fef4c000700 1 --2- 172.21.15.123:0/1667732852 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fef2c0692b0 0x7fef2c06b760 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:00.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:00.914+0000 7fef4c000700 1 --2- 172.21.15.123:0/1667732852 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef44070360 0x7fef44115c80 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:00.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:00.914+0000 7fef4c000700 1 --2- 172.21.15.123:0/1667732852 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fef44070c70 0x7fef441161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:00.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:00.914+0000 7fef4c000700 1 -- 172.21.15.123:0/1667732852 >> 172.21.15.123:0/1667732852 conn(0x7fef440f89b0 msgr2=0x7fef441015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:24:00.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:00.915+0000 7fef4c000700 1 -- 172.21.15.123:0/1667732852 shutdown_connections 2024-03-30T04:24:00.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:00.915+0000 7fef4c000700 1 -- 172.21.15.123:0/1667732852 wait complete. 2024-03-30T04:24:00.916 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-03-30T04:24:01.595 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":180,"stamp":"2024-03-30T04:24:00.293622+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1788020,"kb_used_data":3872,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":560682892,"statfs":{"total":575970213888,"available":574139281408,"internally_reserved":0,"allocated":3964928,"data_stored":2605601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002376"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":189,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-30T04:23:06.208357+0000","last_change":"2024-03-30T04:22:34.878326+0000","last_active":"2024-03-30T04:23:06.208357+0000","last_peered":"2024-03-30T04:23:06.208357+0000","last_clean":"2024-03-30T04:23:06.208357+0000","last_became_active":"2024-03-30T04:22:34.877928+0000","last_became_peered":"2024-03-30T04:22:34.877928+0000","last_unstale":"2024-03-30T04:23:06.208357+0000","last_undegraded":"2024-03-30T04:23:06.208357+0000","last_fullsized":"2024-03-30T04:23:06.208357+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T04:21:35.176029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T04:21:35.176029+0000","last_clean_scrub_stamp":"2024-03-30T04:21:35.176029+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:10:10.472729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724557,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297712,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447440,"statfs":{"total":95995035648,"available":95690178560,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]}]},{"osd":4,"up_from":34,"seq":146028888083,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297716,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447436,"statfs":{"total":95995035648,"available":95690174464,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:23:36 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.522,"5min":1.522,"15min":1.522},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.53600000000000003}]},{"osd":1,"last update":"Sat Mar 30 04:23:36 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.5529999999999999,"5min":1.5529999999999999,"15min":1.5529999999999999},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.66900000000000004}]},{"osd":2,"last update":"Sat Mar 30 04:23:36 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.50900000000000001}]},{"osd":3,"last update":"Sat Mar 30 04:23:36 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.56000000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]}]},{"osd":3,"up_from":26,"seq":111669149723,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298300,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446852,"statfs":{"total":95995035648,"available":95689576448,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:23:08 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.503},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.65900000000000003}]},{"osd":1,"last update":"Sat Mar 30 04:23:08 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.51300000000000001}]},{"osd":2,"last update":"Sat Mar 30 04:23:08 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.159,"5min":1.159,"15min":1.159},"last":0.45200000000000001}]},{"osd":4,"last update":"Sat Mar 30 04:23:37 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.56399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]}]},{"osd":2,"up_from":19,"seq":81604378657,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297708,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447444,"statfs":{"total":95995035648,"available":95690182656,"internally_reserved":0,"allocated":360448,"data_stored":138532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:23:34 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.41299999999999998,"15min":0.41099999999999998},"min":{"1min":0.34200000000000003,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.34899999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.58599999999999997,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.32900000000000001}]},{"osd":1,"last update":"Sat Mar 30 04:23:34 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.45300000000000001,"15min":0.45500000000000002},"min":{"1min":0.33800000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.56399999999999995,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.49099999999999999,"15min":0.495},"min":{"1min":0.38200000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.622,"5min":1.081,"15min":1.081},"last":0.38600000000000001}]},{"osd":3,"last update":"Sat Mar 30 04:23:04 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.94999999999999996},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.89000000000000001}]},{"osd":4,"last update":"Sat Mar 30 04:23:39 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.91400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82699999999999996}]}]},{"osd":0,"up_from":9,"seq":38654705710,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298292,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446860,"statfs":{"total":95995035648,"available":95689584640,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 30 04:23:06 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"min":{"1min":0.28599999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.66300000000000003,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.439,"5min":0.41499999999999998,"15min":0.41099999999999998},"min":{"1min":0.33600000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.73999999999999999,"5min":0.88,"15min":0.88},"last":0.437}]},{"osd":2,"last update":"Sat Mar 30 04:23:38 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.46400000000000002,"15min":0.46600000000000003},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.63,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.373},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.47099999999999997,"15min":0.46999999999999997},"min":{"1min":0.372,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.60899999999999999,"5min":0.753,"15min":0.753},"last":0.48599999999999999}]},{"osd":3,"last update":"Sat Mar 30 04:23:06 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.76000000000000001}]},{"osd":4,"last update":"Sat Mar 30 04:23:38 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.66400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]}]},{"osd":1,"up_from":14,"seq":60129542182,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298292,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446860,"statfs":{"total":95995035648,"available":95689584640,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:23:06 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.38400000000000001,"15min":0.378},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.39800000000000002,"15min":0.38900000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.40799999999999997}]},{"osd":2,"last update":"Sat Mar 30 04:23:37 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.46700000000000003,"15min":0.46300000000000002},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.331,"5min":1.331,"15min":1.331},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.47099999999999997,"15min":0.46800000000000003},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.47199999999999998}]},{"osd":3,"last update":"Sat Mar 30 04:23:06 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.53000000000000003}]},{"osd":4,"last update":"Sat Mar 30 04:23:37 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.499},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.403,"5min":1.403,"15min":1.403},"last":0.51500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T04:24:01.596 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-03-30T04:24:02.014 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:24:02.328 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:02 smithi123 ceph-mon[35245]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:24:02.328 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:02 smithi123 ceph-mon[35245]: from='client.14432 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T04:24:02.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:02 smithi135 ceph-mon[44364]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:24:02.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:02 smithi135 ceph-mon[44364]: from='client.14432 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T04:24:03.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:03.731+0000 7fccc2200700 1 -- 172.21.15.123:0/2206928845 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccbc1037a0 msgr2=0x7fccbc0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:03.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:03.731+0000 7fccc2200700 1 --2- 172.21.15.123:0/2206928845 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccbc1037a0 0x7fccbc0fce10 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fcca8004940 tx=0x7fcca8009e30 comp rx=0 tx=0).stop 2024-03-30T04:24:03.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:03.731+0000 7fccc2200700 1 -- 172.21.15.123:0/2206928845 shutdown_connections 2024-03-30T04:24:03.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:03.731+0000 7fccc2200700 1 --2- 172.21.15.123:0/2206928845 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccbc1037a0 0x7fccbc0fce10 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:03.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:03.731+0000 7fccc2200700 1 --2- 172.21.15.123:0/2206928845 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fccbc102e00 0x7fccbc1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:03.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:03.731+0000 7fccc2200700 1 -- 172.21.15.123:0/2206928845 >> 172.21.15.123:0/2206928845 conn(0x7fccbc0f89b0 msgr2=0x7fccbc0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:24:03.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:03.731+0000 7fccc2200700 1 -- 172.21.15.123:0/2206928845 shutdown_connections 2024-03-30T04:24:03.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:03.731+0000 7fccc2200700 1 -- 172.21.15.123:0/2206928845 wait complete. 2024-03-30T04:24:03.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:03.732+0000 7fccc2200700 1 Processor -- start 2024-03-30T04:24:03.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:03.732+0000 7fccc2200700 1 -- start start 2024-03-30T04:24:03.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:03.732+0000 7fccc2200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fccbc102e00 0x7fccbc0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:24:03.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:03.732+0000 7fccc2200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccbc1037a0 0x7fccbc070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:24:03.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:03.732+0000 7fccc2200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccbc070d90 con 0x7fccbc1037a0 2024-03-30T04:24:03.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:03.732+0000 7fccc2200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccbc070ed0 con 0x7fccbc102e00 2024-03-30T04:24:03.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:03.733+0000 7fccbaa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccbc1037a0 0x7fccbc070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:24:03.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:03.733+0000 7fccbaa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccbc1037a0 0x7fccbc070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37726/0 (socket says 172.21.15.123:37726) 2024-03-30T04:24:03.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:03.733+0000 7fccbaa00700 1 -- 172.21.15.123:0/1451868608 learned_addr learned my addr 172.21.15.123:0/1451868608 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:24:03.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:03.733+0000 7fccbb400700 1 --2- 172.21.15.123:0/1451868608 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fccbc102e00 0x7fccbc0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:24:03.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:03.733+0000 7fccbaa00700 1 -- 172.21.15.123:0/1451868608 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fccbc102e00 msgr2=0x7fccbc0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:03.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:03.733+0000 7fccbaa00700 1 --2- 172.21.15.123:0/1451868608 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fccbc102e00 0x7fccbc0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:03.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:03.733+0000 7fccbaa00700 1 -- 172.21.15.123:0/1451868608 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcca8005040 con 0x7fccbc1037a0 2024-03-30T04:24:03.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:03.733+0000 7fccbb400700 1 --2- 172.21.15.123:0/1451868608 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fccbc102e00 0x7fccbc0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:24:03.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:03.733+0000 7fccbaa00700 1 --2- 172.21.15.123:0/1451868608 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccbc1037a0 0x7fccbc070800 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fcca8009390 tx=0x7fcca8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:24:03.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:03.733+0000 7fccb3400700 1 -- 172.21.15.123:0/1451868608 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcca800e040 con 0x7fccbc1037a0 2024-03-30T04:24:03.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:03.734+0000 7fccb3400700 1 -- 172.21.15.123:0/1451868608 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcca8013420 con 0x7fccbc1037a0 2024-03-30T04:24:03.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:03.734+0000 7fccc2200700 1 -- 172.21.15.123:0/1451868608 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccbc071120 con 0x7fccbc1037a0 2024-03-30T04:24:03.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:03.734+0000 7fccb3400700 1 -- 172.21.15.123:0/1451868608 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcca801c630 con 0x7fccbc1037a0 2024-03-30T04:24:03.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:03.734+0000 7fccc2200700 1 -- 172.21.15.123:0/1451868608 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccbc11a230 con 0x7fccbc1037a0 2024-03-30T04:24:03.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:03.735+0000 7fccc2200700 1 -- 172.21.15.123:0/1451868608 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccbc0fe550 con 0x7fccbc1037a0 2024-03-30T04:24:03.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:03.735+0000 7fccb3400700 1 -- 172.21.15.123:0/1451868608 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fcca8013a00 con 0x7fccbc1037a0 2024-03-30T04:24:03.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:03.736+0000 7fccb3400700 1 --2- 172.21.15.123:0/1451868608 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fcca00692b0 0x7fcca006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:24:03.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:03.736+0000 7fccb3400700 1 -- 172.21.15.123:0/1451868608 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fcca8088160 con 0x7fccbc1037a0 2024-03-30T04:24:03.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:03.737+0000 7fccbb400700 1 --2- 172.21.15.123:0/1451868608 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fcca00692b0 0x7fcca006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:24:03.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:03.737+0000 7fccbb400700 1 --2- 172.21.15.123:0/1451868608 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fcca00692b0 0x7fcca006b760 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fccac0041d0 tx=0x7fccac009040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:24:03.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:03.741+0000 7fccb3400700 1 -- 172.21.15.123:0/1451868608 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fcca8054730 con 0x7fccbc1037a0 2024-03-30T04:24:03.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:03.930+0000 7fccc2200700 1 -- 172.21.15.123:0/1451868608 --> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fccbc109e40 con 0x7fcca00692b0 2024-03-30T04:24:03.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:03.931+0000 7fccb3400700 1 -- 172.21.15.123:0/1451868608 <== mgr.14206 v2:172.21.15.123:6800/1564935593 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+27787 (secure 0 0 0) 0x7fccac004dc0 con 0x7fcca00692b0 2024-03-30T04:24:03.931 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:24:03.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:03.934+0000 7fccc2200700 1 -- 172.21.15.123:0/1451868608 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fcca00692b0 msgr2=0x7fcca006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:03.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:03.934+0000 7fccc2200700 1 --2- 172.21.15.123:0/1451868608 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fcca00692b0 0x7fcca006b760 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fccac0041d0 tx=0x7fccac009040 comp rx=0 tx=0).stop 2024-03-30T04:24:03.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:03.934+0000 7fccc2200700 1 -- 172.21.15.123:0/1451868608 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccbc1037a0 msgr2=0x7fccbc070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:03.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:03.934+0000 7fccc2200700 1 --2- 172.21.15.123:0/1451868608 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccbc1037a0 0x7fccbc070800 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fcca8009390 tx=0x7fcca8009e30 comp rx=0 tx=0).stop 2024-03-30T04:24:03.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:03.934+0000 7fccc2200700 1 -- 172.21.15.123:0/1451868608 shutdown_connections 2024-03-30T04:24:03.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:03.934+0000 7fccc2200700 1 --2- 172.21.15.123:0/1451868608 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fcca00692b0 0x7fcca006b760 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:03.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:03.934+0000 7fccc2200700 1 --2- 172.21.15.123:0/1451868608 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fccbc102e00 0x7fccbc0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:03.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:03.934+0000 7fccc2200700 1 --2- 172.21.15.123:0/1451868608 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccbc1037a0 0x7fccbc070800 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:03.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:03.934+0000 7fccc2200700 1 -- 172.21.15.123:0/1451868608 >> 172.21.15.123:0/1451868608 conn(0x7fccbc0f89b0 msgr2=0x7fccbc0f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:24:03.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:03.934+0000 7fccc2200700 1 -- 172.21.15.123:0/1451868608 shutdown_connections 2024-03-30T04:24:03.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:03.934+0000 7fccc2200700 1 -- 172.21.15.123:0/1451868608 wait complete. 2024-03-30T04:24:03.934 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-03-30T04:24:04.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:04 smithi123 ceph-mon[35245]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:24:04.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:04 smithi135 ceph-mon[44364]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:24:04.691 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":181,"stamp":"2024-03-30T04:24:02.294160+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1788020,"kb_used_data":3872,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":560682892,"statfs":{"total":575970213888,"available":574139281408,"internally_reserved":0,"allocated":3964928,"data_stored":2605601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002440"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":189,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-30T04:23:06.208357+0000","last_change":"2024-03-30T04:22:34.878326+0000","last_active":"2024-03-30T04:23:06.208357+0000","last_peered":"2024-03-30T04:23:06.208357+0000","last_clean":"2024-03-30T04:23:06.208357+0000","last_became_active":"2024-03-30T04:22:34.877928+0000","last_became_peered":"2024-03-30T04:22:34.877928+0000","last_unstale":"2024-03-30T04:23:06.208357+0000","last_undegraded":"2024-03-30T04:23:06.208357+0000","last_fullsized":"2024-03-30T04:23:06.208357+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T04:21:35.176029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T04:21:35.176029+0000","last_clean_scrub_stamp":"2024-03-30T04:21:35.176029+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:10:10.472729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724558,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297712,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447440,"statfs":{"total":95995035648,"available":95690178560,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.748}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78600000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]}]},{"osd":4,"up_from":34,"seq":146028888083,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297716,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447436,"statfs":{"total":95995035648,"available":95690174464,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:23:36 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.522,"5min":1.522,"15min":1.522},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.53600000000000003}]},{"osd":1,"last update":"Sat Mar 30 04:23:36 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.5529999999999999,"5min":1.5529999999999999,"15min":1.5529999999999999},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.66900000000000004}]},{"osd":2,"last update":"Sat Mar 30 04:23:36 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.50900000000000001}]},{"osd":3,"last update":"Sat Mar 30 04:23:36 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.56000000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]}]},{"osd":3,"up_from":26,"seq":111669149723,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298300,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446852,"statfs":{"total":95995035648,"available":95689576448,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:23:08 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.503},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.65900000000000003}]},{"osd":1,"last update":"Sat Mar 30 04:23:08 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.51300000000000001}]},{"osd":2,"last update":"Sat Mar 30 04:23:08 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.159,"5min":1.159,"15min":1.159},"last":0.45200000000000001}]},{"osd":4,"last update":"Sat Mar 30 04:23:37 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.56399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]}]},{"osd":2,"up_from":19,"seq":81604378657,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297708,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447444,"statfs":{"total":95995035648,"available":95690182656,"internally_reserved":0,"allocated":360448,"data_stored":138532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:23:34 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.41299999999999998,"15min":0.41099999999999998},"min":{"1min":0.34200000000000003,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.34899999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.58599999999999997,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.32900000000000001}]},{"osd":1,"last update":"Sat Mar 30 04:23:34 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.45300000000000001,"15min":0.45500000000000002},"min":{"1min":0.33800000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.56399999999999995,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.49099999999999999,"15min":0.495},"min":{"1min":0.38200000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.622,"5min":1.081,"15min":1.081},"last":0.38600000000000001}]},{"osd":3,"last update":"Sat Mar 30 04:23:04 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.94999999999999996},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.89000000000000001}]},{"osd":4,"last update":"Sat Mar 30 04:23:39 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.91400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82699999999999996}]}]},{"osd":0,"up_from":9,"seq":38654705710,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298292,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446860,"statfs":{"total":95995035648,"available":95689584640,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 30 04:23:06 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"min":{"1min":0.28599999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.66300000000000003,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.439,"5min":0.41499999999999998,"15min":0.41099999999999998},"min":{"1min":0.33600000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.73999999999999999,"5min":0.88,"15min":0.88},"last":0.437}]},{"osd":2,"last update":"Sat Mar 30 04:23:38 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.46400000000000002,"15min":0.46600000000000003},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.63,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.373},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.47099999999999997,"15min":0.46999999999999997},"min":{"1min":0.372,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.60899999999999999,"5min":0.753,"15min":0.753},"last":0.48599999999999999}]},{"osd":3,"last update":"Sat Mar 30 04:23:06 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.76000000000000001}]},{"osd":4,"last update":"Sat Mar 30 04:23:38 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.66400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]}]},{"osd":1,"up_from":14,"seq":60129542183,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298292,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446860,"statfs":{"total":95995035648,"available":95689584640,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:23:06 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.38400000000000001,"15min":0.378},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.39800000000000002,"15min":0.38900000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.54900000000000004}]},{"osd":2,"last update":"Sat Mar 30 04:23:37 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.46700000000000003,"15min":0.46300000000000002},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.331,"5min":1.331,"15min":1.331},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.47099999999999997,"15min":0.46800000000000003},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.501}]},{"osd":3,"last update":"Sat Mar 30 04:23:06 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.73099999999999998}]},{"osd":4,"last update":"Sat Mar 30 04:23:37 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.403,"5min":1.403,"15min":1.403},"last":0.65200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69599999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T04:24:04.692 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-30T04:24:04.692 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-30T04:24:04.692 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-30T04:24:04.693 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph health --format=json 2024-03-30T04:24:05.111 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:24:05.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:05 smithi135 ceph-mon[44364]: from='client.14436 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T04:24:05.362 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:05 smithi123 ceph-mon[35245]: from='client.14436 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T04:24:06.343 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:06 smithi123 ceph-mon[35245]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:24:06.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:06 smithi135 ceph-mon[44364]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:24:06.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:06.918+0000 7f359fc00700 1 -- 172.21.15.123:0/3835976609 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35981037a0 msgr2=0x7f35980fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:06.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:06.918+0000 7f359fc00700 1 --2- 172.21.15.123:0/3835976609 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35981037a0 0x7f35980fce10 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3584004940 tx=0x7f3584009e30 comp rx=0 tx=0).stop 2024-03-30T04:24:06.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:06.919+0000 7f359fc00700 1 -- 172.21.15.123:0/3835976609 shutdown_connections 2024-03-30T04:24:06.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:06.919+0000 7f359fc00700 1 --2- 172.21.15.123:0/3835976609 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35981037a0 0x7f35980fce10 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:06.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:06.919+0000 7f359fc00700 1 --2- 172.21.15.123:0/3835976609 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3598102e00 0x7f35981031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:06.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:06.919+0000 7f359fc00700 1 -- 172.21.15.123:0/3835976609 >> 172.21.15.123:0/3835976609 conn(0x7f35980f89b0 msgr2=0x7f35980fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:24:06.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:06.919+0000 7f359fc00700 1 -- 172.21.15.123:0/3835976609 shutdown_connections 2024-03-30T04:24:06.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:06.919+0000 7f359fc00700 1 -- 172.21.15.123:0/3835976609 wait complete. 2024-03-30T04:24:06.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:06.919+0000 7f359fc00700 1 Processor -- start 2024-03-30T04:24:06.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:06.920+0000 7f359fc00700 1 -- start start 2024-03-30T04:24:06.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:06.920+0000 7f359fc00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3598102e00 0x7f3598115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:24:06.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:06.920+0000 7f359fc00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35981037a0 0x7f35981161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:24:06.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:06.920+0000 7f359fc00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35981167f0 con 0x7f35981037a0 2024-03-30T04:24:06.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:06.920+0000 7f359fc00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f359810fbc0 con 0x7f3598102e00 2024-03-30T04:24:06.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:06.920+0000 7f3597e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35981037a0 0x7f35981161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:24:06.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:06.920+0000 7f3597e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35981037a0 0x7f35981161a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53702/0 (socket says 172.21.15.123:53702) 2024-03-30T04:24:06.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:06.920+0000 7f3597e00700 1 -- 172.21.15.123:0/519639857 learned_addr learned my addr 172.21.15.123:0/519639857 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:24:06.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:06.920+0000 7f359d200700 1 --2- 172.21.15.123:0/519639857 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3598102e00 0x7f3598115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:24:06.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:06.921+0000 7f3597e00700 1 -- 172.21.15.123:0/519639857 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3598102e00 msgr2=0x7f3598115c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:06.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:06.921+0000 7f3597e00700 1 --2- 172.21.15.123:0/519639857 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3598102e00 0x7f3598115c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:06.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:06.921+0000 7f3597e00700 1 -- 172.21.15.123:0/519639857 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3584005040 con 0x7f35981037a0 2024-03-30T04:24:06.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:06.921+0000 7f3597e00700 1 --2- 172.21.15.123:0/519639857 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35981037a0 0x7f35981161a0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3584009390 tx=0x7f3584009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:24:06.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:06.921+0000 7f3594c00700 1 -- 172.21.15.123:0/519639857 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f358400e040 con 0x7f35981037a0 2024-03-30T04:24:06.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:06.921+0000 7f3594c00700 1 -- 172.21.15.123:0/519639857 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3584004290 con 0x7f35981037a0 2024-03-30T04:24:06.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:06.921+0000 7f359fc00700 1 -- 172.21.15.123:0/519639857 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f359810fe40 con 0x7f35981037a0 2024-03-30T04:24:06.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:06.921+0000 7f3594c00700 1 -- 172.21.15.123:0/519639857 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3584013e70 con 0x7f35981037a0 2024-03-30T04:24:06.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:06.921+0000 7f359fc00700 1 -- 172.21.15.123:0/519639857 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3598110400 con 0x7f35981037a0 2024-03-30T04:24:06.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:06.922+0000 7f359fc00700 1 -- 172.21.15.123:0/519639857 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35980fe550 con 0x7f35981037a0 2024-03-30T04:24:06.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:06.923+0000 7f3594c00700 1 -- 172.21.15.123:0/519639857 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f3584013690 con 0x7f35981037a0 2024-03-30T04:24:06.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:06.924+0000 7f3594c00700 1 --2- 172.21.15.123:0/519639857 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f35800692b0 0x7f358006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:24:06.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:06.924+0000 7f3594c00700 1 -- 172.21.15.123:0/519639857 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f3584087fa0 con 0x7f35981037a0 2024-03-30T04:24:06.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:06.924+0000 7f359d200700 1 --2- 172.21.15.123:0/519639857 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f35800692b0 0x7f358006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:24:06.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:06.925+0000 7f359d200700 1 --2- 172.21.15.123:0/519639857 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f35800692b0 0x7f358006b760 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f358c009fd0 tx=0x7f358c00b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:24:06.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:06.928+0000 7f3594c00700 1 -- 172.21.15.123:0/519639857 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f3584058130 con 0x7f35981037a0 2024-03-30T04:24:07.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:07.168+0000 7f359fc00700 1 -- 172.21.15.123:0/519639857 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f359805f5f0 con 0x7f35981037a0 2024-03-30T04:24:07.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:07.168+0000 7f3594c00700 1 -- 172.21.15.123:0/519639857 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f3584013420 con 0x7f35981037a0 2024-03-30T04:24:07.167 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:24:07.167 INFO:teuthology.orchestra.run.smithi123.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-30T04:24:07.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:07.171+0000 7f359fc00700 1 -- 172.21.15.123:0/519639857 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f35800692b0 msgr2=0x7f358006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:07.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:07.171+0000 7f359fc00700 1 --2- 172.21.15.123:0/519639857 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f35800692b0 0x7f358006b760 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f358c009fd0 tx=0x7f358c00b040 comp rx=0 tx=0).stop 2024-03-30T04:24:07.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:07.171+0000 7f359fc00700 1 -- 172.21.15.123:0/519639857 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35981037a0 msgr2=0x7f35981161a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:07.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:07.171+0000 7f359fc00700 1 --2- 172.21.15.123:0/519639857 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35981037a0 0x7f35981161a0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3584009390 tx=0x7f3584009e30 comp rx=0 tx=0).stop 2024-03-30T04:24:07.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:07.171+0000 7f359fc00700 1 -- 172.21.15.123:0/519639857 shutdown_connections 2024-03-30T04:24:07.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:07.171+0000 7f359fc00700 1 --2- 172.21.15.123:0/519639857 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f35800692b0 0x7f358006b760 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:07.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:07.171+0000 7f359fc00700 1 --2- 172.21.15.123:0/519639857 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3598102e00 0x7f3598115c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:07.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:07.171+0000 7f359fc00700 1 --2- 172.21.15.123:0/519639857 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35981037a0 0x7f35981161a0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:07.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:07.171+0000 7f359fc00700 1 -- 172.21.15.123:0/519639857 >> 172.21.15.123:0/519639857 conn(0x7f35980f89b0 msgr2=0x7f35980f99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:24:07.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:07.171+0000 7f359fc00700 1 -- 172.21.15.123:0/519639857 shutdown_connections 2024-03-30T04:24:07.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:07.171+0000 7f359fc00700 1 -- 172.21.15.123:0/519639857 wait complete. 2024-03-30T04:24:07.710 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-30T04:24:07.710 INFO:tasks.cephadm:Setup complete, yielding 2024-03-30T04:24:07.710 INFO:teuthology.run_tasks:Running task print... 2024-03-30T04:24:07.718 INFO:teuthology.task.print:**** done end installing quincy cephadm ... 2024-03-30T04:24:07.718 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-30T04:24:07.727 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi123.front.sepia.ceph.com 2024-03-30T04:24:07.728 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2024-03-30T04:24:08.151 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:24:08.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:08 smithi135 ceph-mon[44364]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:24:08.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:08 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/519639857' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-30T04:24:08.412 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:08 smithi123 ceph-mon[35245]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:24:08.412 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:08 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/519639857' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-30T04:24:09.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:09.927+0000 7faac0c00700 1 -- 172.21.15.123:0/2106663387 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faabc0fcee0 msgr2=0x7faabc0fd310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:09.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:09.927+0000 7faac0c00700 1 --2- 172.21.15.123:0/2106663387 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faabc0fcee0 0x7faabc0fd310 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7faaac004940 tx=0x7faaac009e30 comp rx=0 tx=0).stop 2024-03-30T04:24:09.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:09.927+0000 7faac0c00700 1 -- 172.21.15.123:0/2106663387 shutdown_connections 2024-03-30T04:24:09.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:09.927+0000 7faac0c00700 1 --2- 172.21.15.123:0/2106663387 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faabc0fe0d0 0x7faabc0fe540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:09.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:09.927+0000 7faac0c00700 1 --2- 172.21.15.123:0/2106663387 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faabc0fcee0 0x7faabc0fd310 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:09.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:09.927+0000 7faac0c00700 1 -- 172.21.15.123:0/2106663387 >> 172.21.15.123:0/2106663387 conn(0x7faabc0f8560 msgr2=0x7faabc0fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:24:09.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:09.927+0000 7faac0c00700 1 -- 172.21.15.123:0/2106663387 shutdown_connections 2024-03-30T04:24:09.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:09.927+0000 7faac0c00700 1 -- 172.21.15.123:0/2106663387 wait complete. 2024-03-30T04:24:09.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:09.928+0000 7faac0c00700 1 Processor -- start 2024-03-30T04:24:09.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:09.928+0000 7faac0c00700 1 -- start start 2024-03-30T04:24:09.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:09.929+0000 7faac0c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faabc0fcee0 0x7faabc10fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:24:09.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:09.929+0000 7faac0c00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faabc0fe0d0 0x7faabc110080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:24:09.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:09.929+0000 7faac0c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faabc110680 con 0x7faabc0fcee0 2024-03-30T04:24:09.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:09.929+0000 7faac0c00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faabc110ed0 con 0x7faabc0fe0d0 2024-03-30T04:24:09.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:09.929+0000 7faab9e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faabc0fcee0 0x7faabc10fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:24:09.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:09.929+0000 7faab9e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faabc0fcee0 0x7faabc10fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53732/0 (socket says 172.21.15.123:53732) 2024-03-30T04:24:09.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:09.929+0000 7faab9e00700 1 -- 172.21.15.123:0/2102292936 learned_addr learned my addr 172.21.15.123:0/2102292936 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:24:09.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:09.929+0000 7faab9400700 1 --2- 172.21.15.123:0/2102292936 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faabc0fe0d0 0x7faabc110080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:24:09.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:09.929+0000 7faab9e00700 1 -- 172.21.15.123:0/2102292936 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faabc0fe0d0 msgr2=0x7faabc110080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:09.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:09.929+0000 7faab9e00700 1 --2- 172.21.15.123:0/2102292936 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faabc0fe0d0 0x7faabc110080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:09.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:09.929+0000 7faab9e00700 1 -- 172.21.15.123:0/2102292936 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaac005040 con 0x7faabc0fcee0 2024-03-30T04:24:09.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:09.930+0000 7faab9e00700 1 --2- 172.21.15.123:0/2102292936 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faabc0fcee0 0x7faabc10fb40 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7faaac000c00 tx=0x7faaac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:24:09.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:09.930+0000 7faab2000700 1 -- 172.21.15.123:0/2102292936 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faaac00e070 con 0x7faabc0fcee0 2024-03-30T04:24:09.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:09.930+0000 7faab2000700 1 -- 172.21.15.123:0/2102292936 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faaac0093b0 con 0x7faabc0fcee0 2024-03-30T04:24:09.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:09.930+0000 7faab2000700 1 -- 172.21.15.123:0/2102292936 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faaac01cd80 con 0x7faabc0fcee0 2024-03-30T04:24:09.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:09.930+0000 7faac0c00700 1 -- 172.21.15.123:0/2102292936 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faabc111150 con 0x7faabc0fcee0 2024-03-30T04:24:09.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:09.930+0000 7faac0c00700 1 -- 172.21.15.123:0/2102292936 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faabc111630 con 0x7faabc0fcee0 2024-03-30T04:24:09.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:09.931+0000 7faac0c00700 1 -- 172.21.15.123:0/2102292936 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faabc101a60 con 0x7faabc0fcee0 2024-03-30T04:24:09.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:09.932+0000 7faab2000700 1 -- 172.21.15.123:0/2102292936 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7faaac013a30 con 0x7faabc0fcee0 2024-03-30T04:24:09.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:09.933+0000 7faab2000700 1 --2- 172.21.15.123:0/2102292936 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7faaa00692b0 0x7faaa006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:24:09.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:09.933+0000 7faab2000700 1 -- 172.21.15.123:0/2102292936 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7faaac088370 con 0x7faabc0fcee0 2024-03-30T04:24:09.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:09.933+0000 7faab9400700 1 --2- 172.21.15.123:0/2102292936 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7faaa00692b0 0x7faaa006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:24:09.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:09.934+0000 7faab9400700 1 --2- 172.21.15.123:0/2102292936 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7faaa00692b0 0x7faaa006b760 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7faaa8009b80 tx=0x7faaa800b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:24:09.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:09.938+0000 7faab2000700 1 -- 172.21.15.123:0/2102292936 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7faaac058510 con 0x7faabc0fcee0 2024-03-30T04:24:10.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:10.141+0000 7faac0c00700 1 -- 172.21.15.123:0/2102292936 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) v1 -- 0x7faabc05f5f0 con 0x7faabc0fcee0 2024-03-30T04:24:10.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:10.146+0000 7faab2000700 1 -- 172.21.15.123:0/2102292936 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v20)=0 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7faaac0580a0 con 0x7faabc0fcee0 2024-03-30T04:24:10.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:10.149+0000 7faac0c00700 1 -- 172.21.15.123:0/2102292936 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7faaa00692b0 msgr2=0x7faaa006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:10.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:10.149+0000 7faac0c00700 1 --2- 172.21.15.123:0/2102292936 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7faaa00692b0 0x7faaa006b760 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7faaa8009b80 tx=0x7faaa800b040 comp rx=0 tx=0).stop 2024-03-30T04:24:10.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:10.149+0000 7faac0c00700 1 -- 172.21.15.123:0/2102292936 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faabc0fcee0 msgr2=0x7faabc10fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:10.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:10.149+0000 7faac0c00700 1 --2- 172.21.15.123:0/2102292936 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faabc0fcee0 0x7faabc10fb40 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7faaac000c00 tx=0x7faaac009e30 comp rx=0 tx=0).stop 2024-03-30T04:24:10.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:10.149+0000 7faac0c00700 1 -- 172.21.15.123:0/2102292936 shutdown_connections 2024-03-30T04:24:10.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:10.149+0000 7faac0c00700 1 --2- 172.21.15.123:0/2102292936 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7faaa00692b0 0x7faaa006b760 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:10.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:10.149+0000 7faac0c00700 1 --2- 172.21.15.123:0/2102292936 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faabc0fcee0 0x7faabc10fb40 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:10.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:10.149+0000 7faac0c00700 1 --2- 172.21.15.123:0/2102292936 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faabc0fe0d0 0x7faabc110080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:10.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:10.149+0000 7faac0c00700 1 -- 172.21.15.123:0/2102292936 >> 172.21.15.123:0/2102292936 conn(0x7faabc0f8560 msgr2=0x7faabc0f9d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:24:10.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:10.149+0000 7faac0c00700 1 -- 172.21.15.123:0/2102292936 shutdown_connections 2024-03-30T04:24:10.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:10.149+0000 7faac0c00700 1 -- 172.21.15.123:0/2102292936 wait complete. 2024-03-30T04:24:10.315 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:10 smithi123 ceph-mon[35245]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:24:10.315 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:10 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:24:10.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:10 smithi135 ceph-mon[44364]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:24:10.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:10 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:24:10.866 INFO:teuthology.run_tasks:Running task print... 2024-03-30T04:24:10.875 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2024-03-30T04:24:10.875 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-30T04:24:10.883 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi123.front.sepia.ceph.com 2024-03-30T04:24:10.884 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- bash -c 'ceph orch status' 2024-03-30T04:24:11.302 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:24:11.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:11 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/2102292936' entity='client.admin' 2024-03-30T04:24:11.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:11 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:24:11.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:11 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:24:11.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:11 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:11.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:11 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:24:11.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:11 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:24:11.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:11 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:24:11.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:11 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:11.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:11 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/2102292936' entity='client.admin' 2024-03-30T04:24:11.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:11 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:24:11.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:11 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:24:11.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:11 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:11.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:11 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:24:11.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:11 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:24:11.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:11 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:24:11.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:11 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:12.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:12 smithi135 ceph-mon[44364]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:24:12.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:12 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:24:12.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:12 smithi123 ceph-mon[35245]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:24:12.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:12 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:24:13.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:13.153+0000 7fda8dc00700 1 -- 172.21.15.123:0/54436878 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda880fe0d0 msgr2=0x7fda880fe540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:13.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:13.153+0000 7fda8dc00700 1 --2- 172.21.15.123:0/54436878 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda880fe0d0 0x7fda880fe540 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fda70004940 tx=0x7fda70009e30 comp rx=0 tx=0).stop 2024-03-30T04:24:13.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:13.153+0000 7fda8dc00700 1 -- 172.21.15.123:0/54436878 shutdown_connections 2024-03-30T04:24:13.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:13.153+0000 7fda8dc00700 1 --2- 172.21.15.123:0/54436878 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda880fe0d0 0x7fda880fe540 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:13.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:13.153+0000 7fda8dc00700 1 --2- 172.21.15.123:0/54436878 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fda880fcee0 0x7fda880fd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:13.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:13.153+0000 7fda8dc00700 1 -- 172.21.15.123:0/54436878 >> 172.21.15.123:0/54436878 conn(0x7fda880f8560 msgr2=0x7fda880fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:24:13.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:13.153+0000 7fda8dc00700 1 -- 172.21.15.123:0/54436878 shutdown_connections 2024-03-30T04:24:13.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:13.153+0000 7fda8dc00700 1 -- 172.21.15.123:0/54436878 wait complete. 2024-03-30T04:24:13.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:13.154+0000 7fda8dc00700 1 Processor -- start 2024-03-30T04:24:13.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:13.154+0000 7fda8dc00700 1 -- start start 2024-03-30T04:24:13.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:13.154+0000 7fda8dc00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fda880fcee0 0x7fda8810fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:24:13.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:13.154+0000 7fda8dc00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda880fe0d0 0x7fda881100c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:24:13.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:13.154+0000 7fda8dc00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda881106c0 con 0x7fda880fe0d0 2024-03-30T04:24:13.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:13.154+0000 7fda8dc00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda88110f10 con 0x7fda880fcee0 2024-03-30T04:24:13.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:13.155+0000 7fda7fe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda880fe0d0 0x7fda881100c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:24:13.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:13.155+0000 7fda7fe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda880fe0d0 0x7fda881100c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53742/0 (socket says 172.21.15.123:53742) 2024-03-30T04:24:13.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:13.155+0000 7fda7fe00700 1 -- 172.21.15.123:0/2495391485 learned_addr learned my addr 172.21.15.123:0/2495391485 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:24:13.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:13.155+0000 7fda86a00700 1 --2- 172.21.15.123:0/2495391485 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fda880fcee0 0x7fda8810fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:24:13.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:13.155+0000 7fda7fe00700 1 -- 172.21.15.123:0/2495391485 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fda880fcee0 msgr2=0x7fda8810fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:13.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:13.155+0000 7fda7fe00700 1 --2- 172.21.15.123:0/2495391485 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fda880fcee0 0x7fda8810fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:13.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:13.155+0000 7fda7fe00700 1 -- 172.21.15.123:0/2495391485 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda70005040 con 0x7fda880fe0d0 2024-03-30T04:24:13.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:13.155+0000 7fda7fe00700 1 --2- 172.21.15.123:0/2495391485 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda880fe0d0 0x7fda881100c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fda70009380 tx=0x7fda70009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:24:13.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:13.155+0000 7fda7ea00700 1 -- 172.21.15.123:0/2495391485 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda7000e040 con 0x7fda880fe0d0 2024-03-30T04:24:13.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:13.155+0000 7fda7ea00700 1 -- 172.21.15.123:0/2495391485 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fda70004290 con 0x7fda880fe0d0 2024-03-30T04:24:13.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:13.155+0000 7fda8dc00700 1 -- 172.21.15.123:0/2495391485 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda88111190 con 0x7fda880fe0d0 2024-03-30T04:24:13.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:13.155+0000 7fda7ea00700 1 -- 172.21.15.123:0/2495391485 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda7001c3f0 con 0x7fda880fe0d0 2024-03-30T04:24:13.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:13.155+0000 7fda8dc00700 1 -- 172.21.15.123:0/2495391485 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda881116d0 con 0x7fda880fe0d0 2024-03-30T04:24:13.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:13.157+0000 7fda8dc00700 1 -- 172.21.15.123:0/2495391485 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda8805f5f0 con 0x7fda880fe0d0 2024-03-30T04:24:13.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:13.157+0000 7fda7ea00700 1 -- 172.21.15.123:0/2495391485 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fda70013420 con 0x7fda880fe0d0 2024-03-30T04:24:13.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:13.158+0000 7fda7ea00700 1 --2- 172.21.15.123:0/2495391485 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fda6c0692b0 0x7fda6c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:24:13.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:13.158+0000 7fda7ea00700 1 -- 172.21.15.123:0/2495391485 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fda70088240 con 0x7fda880fe0d0 2024-03-30T04:24:13.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:13.158+0000 7fda86a00700 1 --2- 172.21.15.123:0/2495391485 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fda6c0692b0 0x7fda6c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:24:13.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:13.159+0000 7fda86a00700 1 --2- 172.21.15.123:0/2495391485 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fda6c0692b0 0x7fda6c06b760 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fda78009fd0 tx=0x7fda78009f00 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:24:13.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:13.163+0000 7fda7ea00700 1 -- 172.21.15.123:0/2495391485 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fda700583d0 con 0x7fda880fe0d0 2024-03-30T04:24:13.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:13.372+0000 7fda8dc00700 1 -- 172.21.15.123:0/2495391485 --> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7fda88111a00 con 0x7fda6c0692b0 2024-03-30T04:24:13.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:13.373+0000 7fda7ea00700 1 -- 172.21.15.123:0/2495391485 <== mgr.14206 v2:172.21.15.123:6800/1564935593 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fda88111a00 con 0x7fda6c0692b0 2024-03-30T04:24:13.373 INFO:teuthology.orchestra.run.smithi123.stdout:Backend: cephadm 2024-03-30T04:24:13.373 INFO:teuthology.orchestra.run.smithi123.stdout:Available: Yes 2024-03-30T04:24:13.373 INFO:teuthology.orchestra.run.smithi123.stdout:Paused: No 2024-03-30T04:24:13.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:13.375+0000 7fda8dc00700 1 -- 172.21.15.123:0/2495391485 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fda6c0692b0 msgr2=0x7fda6c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:13.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:13.375+0000 7fda8dc00700 1 --2- 172.21.15.123:0/2495391485 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fda6c0692b0 0x7fda6c06b760 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fda78009fd0 tx=0x7fda78009f00 comp rx=0 tx=0).stop 2024-03-30T04:24:13.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:13.375+0000 7fda8dc00700 1 -- 172.21.15.123:0/2495391485 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda880fe0d0 msgr2=0x7fda881100c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:13.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:13.375+0000 7fda8dc00700 1 --2- 172.21.15.123:0/2495391485 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda880fe0d0 0x7fda881100c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fda70009380 tx=0x7fda70009e30 comp rx=0 tx=0).stop 2024-03-30T04:24:13.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:13.376+0000 7fda8dc00700 1 -- 172.21.15.123:0/2495391485 shutdown_connections 2024-03-30T04:24:13.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:13.376+0000 7fda8dc00700 1 --2- 172.21.15.123:0/2495391485 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fda6c0692b0 0x7fda6c06b760 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:13.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:13.376+0000 7fda8dc00700 1 --2- 172.21.15.123:0/2495391485 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fda880fcee0 0x7fda8810fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:13.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:13.376+0000 7fda8dc00700 1 --2- 172.21.15.123:0/2495391485 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda880fe0d0 0x7fda881100c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:13.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:13.376+0000 7fda8dc00700 1 -- 172.21.15.123:0/2495391485 >> 172.21.15.123:0/2495391485 conn(0x7fda880f8560 msgr2=0x7fda880f9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:24:13.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:13.376+0000 7fda8dc00700 1 -- 172.21.15.123:0/2495391485 shutdown_connections 2024-03-30T04:24:13.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:13.376+0000 7fda8dc00700 1 -- 172.21.15.123:0/2495391485 wait complete. 2024-03-30T04:24:14.042 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- bash -c 'ceph orch ps' 2024-03-30T04:24:14.461 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:24:14.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:14 smithi123 ceph-mon[35245]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:24:14.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:14 smithi135 ceph-mon[44364]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:24:15.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:15 smithi123 ceph-mon[35245]: from='client.14448 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:24:15.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:15 smithi135 ceph-mon[44364]: from='client.14448 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:24:16.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:16.199+0000 7f8271000700 1 -- 172.21.15.123:0/4078313625 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f826c0fe0d0 msgr2=0x7f826c0fe540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:16.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:16.199+0000 7f8271000700 1 --2- 172.21.15.123:0/4078313625 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f826c0fe0d0 0x7f826c0fe540 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8258004940 tx=0x7f8258009e30 comp rx=0 tx=0).stop 2024-03-30T04:24:16.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:16.200+0000 7f8271000700 1 -- 172.21.15.123:0/4078313625 shutdown_connections 2024-03-30T04:24:16.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:16.200+0000 7f8271000700 1 --2- 172.21.15.123:0/4078313625 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f826c0fe0d0 0x7f826c0fe540 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:16.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:16.200+0000 7f8271000700 1 --2- 172.21.15.123:0/4078313625 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f826c0fcee0 0x7f826c0fd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:16.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:16.200+0000 7f8271000700 1 -- 172.21.15.123:0/4078313625 >> 172.21.15.123:0/4078313625 conn(0x7f826c0f8560 msgr2=0x7f826c0fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:24:16.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:16.200+0000 7f8271000700 1 -- 172.21.15.123:0/4078313625 shutdown_connections 2024-03-30T04:24:16.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:16.200+0000 7f8271000700 1 -- 172.21.15.123:0/4078313625 wait complete. 2024-03-30T04:24:16.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:16.200+0000 7f8271000700 1 Processor -- start 2024-03-30T04:24:16.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:16.201+0000 7f8271000700 1 -- start start 2024-03-30T04:24:16.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:16.201+0000 7f8271000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f826c0fcee0 0x7f826c071dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:24:16.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:16.201+0000 7f8271000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f826c0fe0d0 0x7f826c070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:24:16.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:16.201+0000 7f8271000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f826c0708a0 con 0x7f826c0fcee0 2024-03-30T04:24:16.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:16.201+0000 7f8271000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f826c0709e0 con 0x7f826c0fe0d0 2024-03-30T04:24:16.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:16.201+0000 7f8269e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f826c0fcee0 0x7f826c071dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:24:16.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:16.201+0000 7f8269e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f826c0fcee0 0x7f826c071dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35098/0 (socket says 172.21.15.123:35098) 2024-03-30T04:24:16.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:16.201+0000 7f8269e00700 1 -- 172.21.15.123:0/3357064662 learned_addr learned my addr 172.21.15.123:0/3357064662 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:24:16.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:16.201+0000 7f8269400700 1 --2- 172.21.15.123:0/3357064662 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f826c0fe0d0 0x7f826c070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:24:16.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:16.201+0000 7f8269e00700 1 -- 172.21.15.123:0/3357064662 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f826c0fe0d0 msgr2=0x7f826c070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:16.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:16.201+0000 7f8269e00700 1 --2- 172.21.15.123:0/3357064662 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f826c0fe0d0 0x7f826c070310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:16.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:16.201+0000 7f8269e00700 1 -- 172.21.15.123:0/3357064662 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8258005040 con 0x7f826c0fcee0 2024-03-30T04:24:16.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:16.202+0000 7f8269e00700 1 --2- 172.21.15.123:0/3357064662 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f826c0fcee0 0x7f826c071dd0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f825c002a50 tx=0x7f825c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:24:16.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:16.202+0000 7f8262000700 1 -- 172.21.15.123:0/3357064662 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f825c007cc0 con 0x7f826c0fcee0 2024-03-30T04:24:16.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:16.202+0000 7f8262000700 1 -- 172.21.15.123:0/3357064662 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f825c00f040 con 0x7f826c0fcee0 2024-03-30T04:24:16.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:16.202+0000 7f8271000700 1 -- 172.21.15.123:0/3357064662 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f826c070c90 con 0x7f826c0fcee0 2024-03-30T04:24:16.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:16.202+0000 7f8262000700 1 -- 172.21.15.123:0/3357064662 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f825c00d990 con 0x7f826c0fcee0 2024-03-30T04:24:16.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:16.202+0000 7f8271000700 1 -- 172.21.15.123:0/3357064662 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f826c115a20 con 0x7f826c0fcee0 2024-03-30T04:24:16.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:16.203+0000 7f8271000700 1 -- 172.21.15.123:0/3357064662 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f826c05f5f0 con 0x7f826c0fcee0 2024-03-30T04:24:16.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:16.204+0000 7f8262000700 1 -- 172.21.15.123:0/3357064662 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f825c009710 con 0x7f826c0fcee0 2024-03-30T04:24:16.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:16.205+0000 7f8262000700 1 --2- 172.21.15.123:0/3357064662 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f82500692b0 0x7f825006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:24:16.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:16.205+0000 7f8262000700 1 -- 172.21.15.123:0/3357064662 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f825c086b60 con 0x7f826c0fcee0 2024-03-30T04:24:16.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:16.205+0000 7f8269400700 1 --2- 172.21.15.123:0/3357064662 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f82500692b0 0x7f825006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:24:16.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:16.205+0000 7f8269400700 1 --2- 172.21.15.123:0/3357064662 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f82500692b0 0x7f825006b760 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8258009fd0 tx=0x7f8258009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:24:16.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:16.209+0000 7f8262000700 1 -- 172.21.15.123:0/3357064662 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f825c053130 con 0x7f826c0fcee0 2024-03-30T04:24:16.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:16 smithi123 ceph-mon[35245]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:24:16.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:16.416+0000 7f8271000700 1 -- 172.21.15.123:0/3357064662 --> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f826c102580 con 0x7f82500692b0 2024-03-30T04:24:16.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:16.424+0000 7f8262000700 1 -- 172.21.15.123:0/3357064662 <== mgr.14206 v2:172.21.15.123:6800/1564935593 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+2718 (secure 0 0 0) 0x7f826c102580 con 0x7f82500692b0 2024-03-30T04:24:16.422 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:24:16.422 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (4m) 2m ago 5m 13.7M - 0.25.0 c8568f914cd2 0d2c80043347 2024-03-30T04:24:16.422 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (6m) 2m ago 6m 6647k - 17.2.7-797-g5c024335 f77085f6aab7 bceb8b8c4837 2024-03-30T04:24:16.423 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (5m) 71s ago 5m 6651k - 17.2.7-797-g5c024335 f77085f6aab7 cd0fe67b2d34 2024-03-30T04:24:16.423 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (4m) 2m ago 5m 74.1M - 9.4.7 954c08fa6188 b7d7eaffc084 2024-03-30T04:24:16.423 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:9283 running (7m) 2m ago 7m 473M - 17.2.7-797-g5c024335 f77085f6aab7 7b98aee9a3b6 2024-03-30T04:24:16.423 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283 running (4m) 71s ago 4m 430M - 17.2.7-797-g5c024335 f77085f6aab7 37fc80673f76 2024-03-30T04:24:16.423 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (7m) 2m ago 7m 41.5M 2048M 17.2.7-797-g5c024335 f77085f6aab7 51d18587a7df 2024-03-30T04:24:16.423 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (4m) 71s ago 4m 37.1M 2048M 17.2.7-797-g5c024335 f77085f6aab7 f81052b09dd8 2024-03-30T04:24:16.423 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (6m) 2m ago 6m 7268k - 1.5.0 0da6a335fe13 9848cad1e995 2024-03-30T04:24:16.423 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (4m) 71s ago 4m 7768k - 1.5.0 0da6a335fe13 3c2c8e799cc2 2024-03-30T04:24:16.423 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (3m) 2m ago 3m 37.6M 4348M 17.2.7-797-g5c024335 f77085f6aab7 57a7ba3fd75e 2024-03-30T04:24:16.423 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (3m) 2m ago 3m 40.9M 4348M 17.2.7-797-g5c024335 f77085f6aab7 e1a6c1818a0c 2024-03-30T04:24:16.423 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (2m) 2m ago 2m 36.9M 4348M 17.2.7-797-g5c024335 f77085f6aab7 4c6a727567bd 2024-03-30T04:24:16.423 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (2m) 71s ago 2m 37.9M 5372M 17.2.7-797-g5c024335 f77085f6aab7 6fbdd1d46a9d 2024-03-30T04:24:16.423 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (108s) 71s ago 107s 35.7M 5372M 17.2.7-797-g5c024335 f77085f6aab7 b74988e947e5 2024-03-30T04:24:16.423 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (77s) 71s ago 76s 10.4M 5372M 17.2.7-797-g5c024335 f77085f6aab7 1fd3185f5484 2024-03-30T04:24:16.424 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (4m) 2m ago 5m 35.6M - 2.43.0 a07b618ecd1d ba3c351b14cf 2024-03-30T04:24:16.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:16.427+0000 7f8271000700 1 -- 172.21.15.123:0/3357064662 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f82500692b0 msgr2=0x7f825006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:16.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:16.427+0000 7f8271000700 1 --2- 172.21.15.123:0/3357064662 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f82500692b0 0x7f825006b760 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8258009fd0 tx=0x7f8258009e30 comp rx=0 tx=0).stop 2024-03-30T04:24:16.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:16.427+0000 7f8271000700 1 -- 172.21.15.123:0/3357064662 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f826c0fcee0 msgr2=0x7f826c071dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:16.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:16.427+0000 7f8271000700 1 --2- 172.21.15.123:0/3357064662 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f826c0fcee0 0x7f826c071dd0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f825c002a50 tx=0x7f825c004600 comp rx=0 tx=0).stop 2024-03-30T04:24:16.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:16.427+0000 7f8271000700 1 -- 172.21.15.123:0/3357064662 shutdown_connections 2024-03-30T04:24:16.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:16.427+0000 7f8271000700 1 --2- 172.21.15.123:0/3357064662 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f82500692b0 0x7f825006b760 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:16.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:16.427+0000 7f8271000700 1 --2- 172.21.15.123:0/3357064662 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f826c0fcee0 0x7f826c071dd0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:16.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:16.427+0000 7f8271000700 1 --2- 172.21.15.123:0/3357064662 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f826c0fe0d0 0x7f826c070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:16.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:16.427+0000 7f8271000700 1 -- 172.21.15.123:0/3357064662 >> 172.21.15.123:0/3357064662 conn(0x7f826c0f8560 msgr2=0x7f826c0f9da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:24:16.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:16.427+0000 7f8271000700 1 -- 172.21.15.123:0/3357064662 shutdown_connections 2024-03-30T04:24:16.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:16.427+0000 7f8271000700 1 -- 172.21.15.123:0/3357064662 wait complete. 2024-03-30T04:24:16.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:16 smithi135 ceph-mon[44364]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:24:17.118 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- bash -c 'ceph orch ls' 2024-03-30T04:24:17.535 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:24:18.410 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:18 smithi123 ceph-mon[35245]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:24:18.410 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:18 smithi123 ceph-mon[35245]: from='client.14452 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:24:18.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:18 smithi135 ceph-mon[44364]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:24:18.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:18 smithi135 ceph-mon[44364]: from='client.14452 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:24:19.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:19.311+0000 7f0507200700 1 -- 172.21.15.123:0/3879733209 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05080fdec0 msgr2=0x7f05080fe330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:19.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:19.311+0000 7f0507200700 1 --2- 172.21.15.123:0/3879733209 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05080fdec0 0x7f05080fe330 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f04f4004940 tx=0x7f04f4009e30 comp rx=0 tx=0).stop 2024-03-30T04:24:19.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:19.311+0000 7f0507200700 1 -- 172.21.15.123:0/3879733209 shutdown_connections 2024-03-30T04:24:19.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:19.311+0000 7f0507200700 1 --2- 172.21.15.123:0/3879733209 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05080fdec0 0x7f05080fe330 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:19.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:19.311+0000 7f0507200700 1 --2- 172.21.15.123:0/3879733209 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f05080fcc10 0x7f05080fd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:19.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:19.311+0000 7f0507200700 1 -- 172.21.15.123:0/3879733209 >> 172.21.15.123:0/3879733209 conn(0x7f05080f83d0 msgr2=0x7f05080fa7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:24:19.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:19.311+0000 7f0507200700 1 -- 172.21.15.123:0/3879733209 shutdown_connections 2024-03-30T04:24:19.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:19.312+0000 7f0507200700 1 -- 172.21.15.123:0/3879733209 wait complete. 2024-03-30T04:24:19.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:19.312+0000 7f0507200700 1 Processor -- start 2024-03-30T04:24:19.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:19.313+0000 7f0507200700 1 -- start start 2024-03-30T04:24:19.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:19.313+0000 7f0507200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f05080fcc10 0x7f0508071e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:24:19.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:19.313+0000 7f0507200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05080fdec0 0x7f0508070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:24:19.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:19.313+0000 7f0507200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05080708c0 con 0x7f05080fdec0 2024-03-30T04:24:19.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:19.313+0000 7f0507200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0508070a30 con 0x7f05080fcc10 2024-03-30T04:24:19.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:19.313+0000 7f0505400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05080fdec0 0x7f0508070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:24:19.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:19.313+0000 7f0505400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05080fdec0 0x7f0508070310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35118/0 (socket says 172.21.15.123:35118) 2024-03-30T04:24:19.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:19.313+0000 7f0505400700 1 -- 172.21.15.123:0/322979867 learned_addr learned my addr 172.21.15.123:0/322979867 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:24:19.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:19.313+0000 7f0505e00700 1 --2- 172.21.15.123:0/322979867 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f05080fcc10 0x7f0508071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:24:19.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:19.313+0000 7f0505400700 1 -- 172.21.15.123:0/322979867 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f05080fcc10 msgr2=0x7f0508071e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:19.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:19.313+0000 7f0505400700 1 --2- 172.21.15.123:0/322979867 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f05080fcc10 0x7f0508071e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:19.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:19.314+0000 7f0505400700 1 -- 172.21.15.123:0/322979867 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04f4005040 con 0x7f05080fdec0 2024-03-30T04:24:19.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:19.314+0000 7f0505400700 1 --2- 172.21.15.123:0/322979867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05080fdec0 0x7f0508070310 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f04f400b3b0 tx=0x7f04f4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:24:19.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:19.314+0000 7f04fe000700 1 -- 172.21.15.123:0/322979867 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f04f400e040 con 0x7f05080fdec0 2024-03-30T04:24:19.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:19.314+0000 7f04fe000700 1 -- 172.21.15.123:0/322979867 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f04f4004290 con 0x7f05080fdec0 2024-03-30T04:24:19.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:19.314+0000 7f0507200700 1 -- 172.21.15.123:0/322979867 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0508070cb0 con 0x7f05080fdec0 2024-03-30T04:24:19.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:19.314+0000 7f04fe000700 1 -- 172.21.15.123:0/322979867 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f04f401c3f0 con 0x7f05080fdec0 2024-03-30T04:24:19.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:19.314+0000 7f0507200700 1 -- 172.21.15.123:0/322979867 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05081157a0 con 0x7f05080fdec0 2024-03-30T04:24:19.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:19.315+0000 7f0507200700 1 -- 172.21.15.123:0/322979867 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f050805f5f0 con 0x7f05080fdec0 2024-03-30T04:24:19.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:19.316+0000 7f04fe000700 1 -- 172.21.15.123:0/322979867 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f04f4013420 con 0x7f05080fdec0 2024-03-30T04:24:19.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:19.317+0000 7f04fe000700 1 --2- 172.21.15.123:0/322979867 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f04ec0692b0 0x7f04ec06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:24:19.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:19.317+0000 7f04fe000700 1 -- 172.21.15.123:0/322979867 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f04f40880b0 con 0x7f05080fdec0 2024-03-30T04:24:19.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:19.317+0000 7f0505e00700 1 --2- 172.21.15.123:0/322979867 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f04ec0692b0 0x7f04ec06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:24:19.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:19.318+0000 7f0505e00700 1 --2- 172.21.15.123:0/322979867 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f04ec0692b0 0x7f04ec06b760 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0508071bd0 tx=0x7f04f8009040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:24:19.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:19.321+0000 7f04fe000700 1 -- 172.21.15.123:0/322979867 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f04f40547b0 con 0x7f05080fdec0 2024-03-30T04:24:19.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:19.525+0000 7f0507200700 1 -- 172.21.15.123:0/322979867 --> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f0508102320 con 0x7f04ec0692b0 2024-03-30T04:24:19.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:19.529+0000 7f04fe000700 1 -- 172.21.15.123:0/322979867 <== mgr.14206 v2:172.21.15.123:6800/1564935593 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1179 (secure 0 0 0) 0x7f0508102320 con 0x7f04ec0692b0 2024-03-30T04:24:19.528 INFO:teuthology.orchestra.run.smithi123.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-30T04:24:19.528 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager ?:9093,9094 1/1 2m ago 6m count:1 2024-03-30T04:24:19.528 INFO:teuthology.orchestra.run.smithi123.stdout:crash 2/2 2m ago 6m * 2024-03-30T04:24:19.528 INFO:teuthology.orchestra.run.smithi123.stdout:grafana ?:3000 1/1 2m ago 6m count:1 2024-03-30T04:24:19.528 INFO:teuthology.orchestra.run.smithi123.stdout:mgr 2/2 2m ago 6m count:2 2024-03-30T04:24:19.528 INFO:teuthology.orchestra.run.smithi123.stdout:mon 2/2 2m ago 5m smithi123:172.21.15.123=smithi123;smithi135:172.21.15.135=smithi135;count:2 2024-03-30T04:24:19.529 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter ?:9100 2/2 2m ago 6m * 2024-03-30T04:24:19.529 INFO:teuthology.orchestra.run.smithi123.stdout:osd 6 2m ago - 2024-03-30T04:24:19.529 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus ?:9095 1/1 2m ago 6m count:1 2024-03-30T04:24:19.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:19.531+0000 7f0507200700 1 -- 172.21.15.123:0/322979867 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f04ec0692b0 msgr2=0x7f04ec06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:19.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:19.531+0000 7f0507200700 1 --2- 172.21.15.123:0/322979867 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f04ec0692b0 0x7f04ec06b760 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0508071bd0 tx=0x7f04f8009040 comp rx=0 tx=0).stop 2024-03-30T04:24:19.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:19.531+0000 7f0507200700 1 -- 172.21.15.123:0/322979867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05080fdec0 msgr2=0x7f0508070310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:19.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:19.531+0000 7f0507200700 1 --2- 172.21.15.123:0/322979867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05080fdec0 0x7f0508070310 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f04f400b3b0 tx=0x7f04f4009e30 comp rx=0 tx=0).stop 2024-03-30T04:24:19.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:19.531+0000 7f0507200700 1 -- 172.21.15.123:0/322979867 shutdown_connections 2024-03-30T04:24:19.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:19.531+0000 7f0507200700 1 --2- 172.21.15.123:0/322979867 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f04ec0692b0 0x7f04ec06b760 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:19.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:19.531+0000 7f0507200700 1 --2- 172.21.15.123:0/322979867 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f05080fcc10 0x7f0508071e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:19.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:19.531+0000 7f0507200700 1 --2- 172.21.15.123:0/322979867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05080fdec0 0x7f0508070310 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:19.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:19.531+0000 7f0507200700 1 -- 172.21.15.123:0/322979867 >> 172.21.15.123:0/322979867 conn(0x7f05080f83d0 msgr2=0x7f05080f9b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:24:19.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:19.532+0000 7f0507200700 1 -- 172.21.15.123:0/322979867 shutdown_connections 2024-03-30T04:24:19.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:19.532+0000 7f0507200700 1 -- 172.21.15.123:0/322979867 wait complete. 2024-03-30T04:24:20.092 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- bash -c 'ceph orch host ls' 2024-03-30T04:24:20.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:20 smithi123 ceph-mon[35245]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:24:20.516 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:24:20.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:20 smithi135 ceph-mon[44364]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:24:21.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:21 smithi123 ceph-mon[35245]: from='client.14456 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:24:21.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:21 smithi135 ceph-mon[44364]: from='client.14456 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:24:22.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:22.341+0000 7f2ee4e00700 1 -- 172.21.15.123:0/3946044484 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ee0071f50 msgr2=0x7f2ee0070460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:22.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:22.341+0000 7f2ee4e00700 1 --2- 172.21.15.123:0/3946044484 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ee0071f50 0x7f2ee0070460 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f2ed0004940 tx=0x7f2ed0009e30 comp rx=0 tx=0).stop 2024-03-30T04:24:22.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:22.341+0000 7f2ee4e00700 1 -- 172.21.15.123:0/3946044484 shutdown_connections 2024-03-30T04:24:22.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:22.341+0000 7f2ee4e00700 1 --2- 172.21.15.123:0/3946044484 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2ee00709a0 0x7f2ee0070e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:22.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:22.341+0000 7f2ee4e00700 1 --2- 172.21.15.123:0/3946044484 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ee0071f50 0x7f2ee0070460 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:22.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:22.341+0000 7f2ee4e00700 1 -- 172.21.15.123:0/3946044484 >> 172.21.15.123:0/3946044484 conn(0x7f2ee00f8560 msgr2=0x7f2ee00fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:24:22.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:22.341+0000 7f2ee4e00700 1 -- 172.21.15.123:0/3946044484 shutdown_connections 2024-03-30T04:24:22.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:22.341+0000 7f2ee4e00700 1 -- 172.21.15.123:0/3946044484 wait complete. 2024-03-30T04:24:22.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:22.341+0000 7f2ee4e00700 1 Processor -- start 2024-03-30T04:24:22.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:22.342+0000 7f2ee4e00700 1 -- start start 2024-03-30T04:24:22.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:22.342+0000 7f2ee4e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2ee00709a0 0x7f2ee010fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:24:22.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:22.342+0000 7f2ee4e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ee0071f50 0x7f2ee0110090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:24:22.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:22.342+0000 7f2ee4e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee0110690 con 0x7f2ee0071f50 2024-03-30T04:24:22.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:22.342+0000 7f2ee4e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee0110ee0 con 0x7f2ee00709a0 2024-03-30T04:24:22.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:22.342+0000 7f2edd400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ee0071f50 0x7f2ee0110090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:24:22.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:22.342+0000 7f2edd400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ee0071f50 0x7f2ee0110090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35130/0 (socket says 172.21.15.123:35130) 2024-03-30T04:24:22.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:22.342+0000 7f2edd400700 1 -- 172.21.15.123:0/3091337895 learned_addr learned my addr 172.21.15.123:0/3091337895 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:24:22.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:22.342+0000 7f2edde00700 1 --2- 172.21.15.123:0/3091337895 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2ee00709a0 0x7f2ee010fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:24:22.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:22.343+0000 7f2edd400700 1 -- 172.21.15.123:0/3091337895 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2ee00709a0 msgr2=0x7f2ee010fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:22.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:22.343+0000 7f2edd400700 1 --2- 172.21.15.123:0/3091337895 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2ee00709a0 0x7f2ee010fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:22.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:22.343+0000 7f2edd400700 1 -- 172.21.15.123:0/3091337895 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ed0005040 con 0x7f2ee0071f50 2024-03-30T04:24:22.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:22.343+0000 7f2edd400700 1 --2- 172.21.15.123:0/3091337895 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ee0071f50 0x7f2ee0110090 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc00c8c0 tx=0x7f2ecc0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:24:22.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:22.343+0000 7f2ed6000700 1 -- 172.21.15.123:0/3091337895 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ecc00a760 con 0x7f2ee0071f50 2024-03-30T04:24:22.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:22.343+0000 7f2ed6000700 1 -- 172.21.15.123:0/3091337895 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2ecc00ad70 con 0x7f2ee0071f50 2024-03-30T04:24:22.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:22.343+0000 7f2ee4e00700 1 -- 172.21.15.123:0/3091337895 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ee01111c0 con 0x7f2ee0071f50 2024-03-30T04:24:22.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:22.343+0000 7f2ed6000700 1 -- 172.21.15.123:0/3091337895 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ecc0113a0 con 0x7f2ee0071f50 2024-03-30T04:24:22.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:22.343+0000 7f2ee4e00700 1 -- 172.21.15.123:0/3091337895 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ee0111700 con 0x7f2ee0071f50 2024-03-30T04:24:22.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:22.344+0000 7f2ee4e00700 1 -- 172.21.15.123:0/3091337895 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ee005f5f0 con 0x7f2ee0071f50 2024-03-30T04:24:22.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:22.345+0000 7f2ed6000700 1 -- 172.21.15.123:0/3091337895 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f2ecc012400 con 0x7f2ee0071f50 2024-03-30T04:24:22.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:22.347+0000 7f2ed6000700 1 --2- 172.21.15.123:0/3091337895 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f2ec40692b0 0x7f2ec406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:24:22.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:22.347+0000 7f2ed6000700 1 -- 172.21.15.123:0/3091337895 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f2ecc086f50 con 0x7f2ee0071f50 2024-03-30T04:24:22.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:22.347+0000 7f2edde00700 1 --2- 172.21.15.123:0/3091337895 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f2ec40692b0 0x7f2ec406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:24:22.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:22.347+0000 7f2edde00700 1 --2- 172.21.15.123:0/3091337895 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f2ec40692b0 0x7f2ec406b760 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2ed000b3b0 tx=0x7f2ed0009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:24:22.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:22.351+0000 7f2ed6000700 1 -- 172.21.15.123:0/3091337895 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f2ecc056fb0 con 0x7f2ee0071f50 2024-03-30T04:24:22.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:22 smithi135 ceph-mon[44364]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:24:22.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:22 smithi123 ceph-mon[35245]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:24:22.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:22.551+0000 7f2ee4e00700 1 -- 172.21.15.123:0/3091337895 --> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f2ee0111af0 con 0x7f2ec40692b0 2024-03-30T04:24:22.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:22.553+0000 7f2ed6000700 1 -- 172.21.15.123:0/3091337895 <== mgr.14206 v2:172.21.15.123:6800/1564935593 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+148 (secure 0 0 0) 0x7f2ee0111af0 con 0x7f2ec40692b0 2024-03-30T04:24:22.552 INFO:teuthology.orchestra.run.smithi123.stdout:HOST ADDR LABELS STATUS 2024-03-30T04:24:22.552 INFO:teuthology.orchestra.run.smithi123.stdout:smithi123 172.21.15.123 2024-03-30T04:24:22.552 INFO:teuthology.orchestra.run.smithi123.stdout:smithi135 172.21.15.135 2024-03-30T04:24:22.552 INFO:teuthology.orchestra.run.smithi123.stdout:2 hosts in cluster 2024-03-30T04:24:22.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:22.555+0000 7f2ee4e00700 1 -- 172.21.15.123:0/3091337895 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f2ec40692b0 msgr2=0x7f2ec406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:22.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:22.555+0000 7f2ee4e00700 1 --2- 172.21.15.123:0/3091337895 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f2ec40692b0 0x7f2ec406b760 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2ed000b3b0 tx=0x7f2ed0009e30 comp rx=0 tx=0).stop 2024-03-30T04:24:22.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:22.555+0000 7f2ee4e00700 1 -- 172.21.15.123:0/3091337895 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ee0071f50 msgr2=0x7f2ee0110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:22.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:22.555+0000 7f2ee4e00700 1 --2- 172.21.15.123:0/3091337895 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ee0071f50 0x7f2ee0110090 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc00c8c0 tx=0x7f2ecc0044c0 comp rx=0 tx=0).stop 2024-03-30T04:24:22.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:22.556+0000 7f2ee4e00700 1 -- 172.21.15.123:0/3091337895 shutdown_connections 2024-03-30T04:24:22.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:22.556+0000 7f2ee4e00700 1 --2- 172.21.15.123:0/3091337895 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f2ec40692b0 0x7f2ec406b760 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:22.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:22.556+0000 7f2ee4e00700 1 --2- 172.21.15.123:0/3091337895 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2ee00709a0 0x7f2ee010fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:22.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:22.556+0000 7f2ee4e00700 1 --2- 172.21.15.123:0/3091337895 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ee0071f50 0x7f2ee0110090 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:22.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:22.556+0000 7f2ee4e00700 1 -- 172.21.15.123:0/3091337895 >> 172.21.15.123:0/3091337895 conn(0x7f2ee00f8560 msgr2=0x7f2ee00f9d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:24:22.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:22.556+0000 7f2ee4e00700 1 -- 172.21.15.123:0/3091337895 shutdown_connections 2024-03-30T04:24:22.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:22.556+0000 7f2ee4e00700 1 -- 172.21.15.123:0/3091337895 wait complete. 2024-03-30T04:24:23.236 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- bash -c 'ceph orch device ls' 2024-03-30T04:24:23.656 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:24:24.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:24 smithi123 ceph-mon[35245]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:24:24.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:24 smithi123 ceph-mon[35245]: from='client.14460 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:24:24.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:24 smithi135 ceph-mon[44364]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:24:24.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:24 smithi135 ceph-mon[44364]: from='client.14460 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:24:25.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:25.468+0000 7f1896200700 1 -- 172.21.15.123:0/437444929 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18900fcee0 msgr2=0x7f18900fd310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:25.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:25.468+0000 7f1896200700 1 --2- 172.21.15.123:0/437444929 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18900fcee0 0x7f18900fd310 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1880004940 tx=0x7f1880009e30 comp rx=0 tx=0).stop 2024-03-30T04:24:25.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:25.468+0000 7f1896200700 1 -- 172.21.15.123:0/437444929 shutdown_connections 2024-03-30T04:24:25.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:25.468+0000 7f1896200700 1 --2- 172.21.15.123:0/437444929 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f18900fe0d0 0x7f18900fe540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:25.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:25.468+0000 7f1896200700 1 --2- 172.21.15.123:0/437444929 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18900fcee0 0x7f18900fd310 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:25.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:25.468+0000 7f1896200700 1 -- 172.21.15.123:0/437444929 >> 172.21.15.123:0/437444929 conn(0x7f18900f8560 msgr2=0x7f18900fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:24:25.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:25.468+0000 7f1896200700 1 -- 172.21.15.123:0/437444929 shutdown_connections 2024-03-30T04:24:25.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:25.468+0000 7f1896200700 1 -- 172.21.15.123:0/437444929 wait complete. 2024-03-30T04:24:25.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:25.469+0000 7f1896200700 1 Processor -- start 2024-03-30T04:24:25.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:25.469+0000 7f1896200700 1 -- start start 2024-03-30T04:24:25.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:25.469+0000 7f1896200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18900fcee0 0x7f1890071dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:24:25.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:25.469+0000 7f1896200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f18900fe0d0 0x7f1890070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:24:25.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:25.469+0000 7f1896200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18900708a0 con 0x7f18900fcee0 2024-03-30T04:24:25.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:25.469+0000 7f1896200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18900709e0 con 0x7f18900fe0d0 2024-03-30T04:24:25.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:25.470+0000 7f188f400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18900fcee0 0x7f1890071dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:24:25.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:25.470+0000 7f188f400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18900fcee0 0x7f1890071dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49260/0 (socket says 172.21.15.123:49260) 2024-03-30T04:24:25.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:25.470+0000 7f188f400700 1 -- 172.21.15.123:0/3643762740 learned_addr learned my addr 172.21.15.123:0/3643762740 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:24:25.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:25.470+0000 7f188ea00700 1 --2- 172.21.15.123:0/3643762740 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f18900fe0d0 0x7f1890070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:24:25.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:25.470+0000 7f188f400700 1 -- 172.21.15.123:0/3643762740 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f18900fe0d0 msgr2=0x7f1890070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:25.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:25.470+0000 7f188f400700 1 --2- 172.21.15.123:0/3643762740 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f18900fe0d0 0x7f1890070310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:25.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:25.470+0000 7f188f400700 1 -- 172.21.15.123:0/3643762740 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1880005040 con 0x7f18900fcee0 2024-03-30T04:24:25.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:25.470+0000 7f188f400700 1 --2- 172.21.15.123:0/3643762740 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18900fcee0 0x7f1890071dd0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f188000b900 tx=0x7f1880009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:24:25.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:25.470+0000 7f1887400700 1 -- 172.21.15.123:0/3643762740 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f188000e040 con 0x7f18900fcee0 2024-03-30T04:24:25.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:25.471+0000 7f1887400700 1 -- 172.21.15.123:0/3643762740 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1880004290 con 0x7f18900fcee0 2024-03-30T04:24:25.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:25.471+0000 7f1896200700 1 -- 172.21.15.123:0/3643762740 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1890070c30 con 0x7f18900fcee0 2024-03-30T04:24:25.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:25.471+0000 7f1887400700 1 -- 172.21.15.123:0/3643762740 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f188001c3f0 con 0x7f18900fcee0 2024-03-30T04:24:25.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:25.471+0000 7f1896200700 1 -- 172.21.15.123:0/3643762740 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1890115a00 con 0x7f18900fcee0 2024-03-30T04:24:25.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:25.472+0000 7f1896200700 1 -- 172.21.15.123:0/3643762740 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f189005f5f0 con 0x7f18900fcee0 2024-03-30T04:24:25.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:25.472+0000 7f1887400700 1 -- 172.21.15.123:0/3643762740 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f1880013420 con 0x7f18900fcee0 2024-03-30T04:24:25.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:25.473+0000 7f1887400700 1 --2- 172.21.15.123:0/3643762740 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f18740692b0 0x7f187406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:24:25.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:25.473+0000 7f1887400700 1 -- 172.21.15.123:0/3643762740 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1880088160 con 0x7f18900fcee0 2024-03-30T04:24:25.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:25.474+0000 7f188ea00700 1 --2- 172.21.15.123:0/3643762740 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f18740692b0 0x7f187406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:24:25.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:25.474+0000 7f188ea00700 1 --2- 172.21.15.123:0/3643762740 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f18740692b0 0x7f187406b760 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f187c004f40 tx=0x7f187c00b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:24:25.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:25.478+0000 7f1887400700 1 -- 172.21.15.123:0/3643762740 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f1880054860 con 0x7f18900fcee0 2024-03-30T04:24:25.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:25.681+0000 7f1896200700 1 -- 172.21.15.123:0/3643762740 --> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f1890102580 con 0x7f18740692b0 2024-03-30T04:24:25.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:25.682+0000 7f1887400700 1 -- 172.21.15.123:0/3643762740 <== mgr.14206 v2:172.21.15.123:6800/1564935593 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+399 (secure 0 0 0) 0x7f1890102580 con 0x7f18740692b0 2024-03-30T04:24:25.681 INFO:teuthology.orchestra.run.smithi123.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-30T04:24:25.681 INFO:teuthology.orchestra.run.smithi123.stdout:smithi123 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400YK400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-30T04:24:25.681 INFO:teuthology.orchestra.run.smithi123.stdout:smithi135 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620401BA400BGN 372G No 76s ago Has a FileSystem, LVM detected 2024-03-30T04:24:25.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:25.685+0000 7f1896200700 1 -- 172.21.15.123:0/3643762740 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f18740692b0 msgr2=0x7f187406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:25.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:25.685+0000 7f1896200700 1 --2- 172.21.15.123:0/3643762740 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f18740692b0 0x7f187406b760 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f187c004f40 tx=0x7f187c00b040 comp rx=0 tx=0).stop 2024-03-30T04:24:25.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:25.685+0000 7f1896200700 1 -- 172.21.15.123:0/3643762740 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18900fcee0 msgr2=0x7f1890071dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:25.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:25.685+0000 7f1896200700 1 --2- 172.21.15.123:0/3643762740 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18900fcee0 0x7f1890071dd0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f188000b900 tx=0x7f1880009e30 comp rx=0 tx=0).stop 2024-03-30T04:24:25.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:25.685+0000 7f1896200700 1 -- 172.21.15.123:0/3643762740 shutdown_connections 2024-03-30T04:24:25.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:25.685+0000 7f1896200700 1 --2- 172.21.15.123:0/3643762740 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f18740692b0 0x7f187406b760 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:25.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:25.685+0000 7f1896200700 1 --2- 172.21.15.123:0/3643762740 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18900fcee0 0x7f1890071dd0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:25.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:25.685+0000 7f1896200700 1 --2- 172.21.15.123:0/3643762740 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f18900fe0d0 0x7f1890070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:25.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:25.685+0000 7f1896200700 1 -- 172.21.15.123:0/3643762740 >> 172.21.15.123:0/3643762740 conn(0x7f18900f8560 msgr2=0x7f18900f9da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:24:25.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:25.685+0000 7f1896200700 1 -- 172.21.15.123:0/3643762740 shutdown_connections 2024-03-30T04:24:25.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:25.685+0000 7f1896200700 1 -- 172.21.15.123:0/3643762740 wait complete. 2024-03-30T04:24:26.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:26 smithi123 ceph-mon[35245]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:24:26.243 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-30T04:24:26.252 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi123.front.sepia.ceph.com 2024-03-30T04:24:26.252 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- bash -c 'ceph fs volume create cephfs --placement=4' 2024-03-30T04:24:26.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:26 smithi135 ceph-mon[44364]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:24:26.677 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:24:27.449 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:27 smithi123 ceph-mon[35245]: from='client.14464 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:24:27.449 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:27 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:24:27.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:27 smithi135 ceph-mon[44364]: from='client.14464 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:24:27.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:27 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:24:28.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:28.315+0000 7fd300e00700 1 -- 172.21.15.123:0/1426994223 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2fc0708f0 msgr2=0x7fd2fc070d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:28.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:28.315+0000 7fd300e00700 1 --2- 172.21.15.123:0/1426994223 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2fc0708f0 0x7fd2fc070d60 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd2e8004940 tx=0x7fd2e8009e30 comp rx=0 tx=0).stop 2024-03-30T04:24:28.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:28.316+0000 7fd300e00700 1 -- 172.21.15.123:0/1426994223 shutdown_connections 2024-03-30T04:24:28.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:28.316+0000 7fd300e00700 1 --2- 172.21.15.123:0/1426994223 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2fc0708f0 0x7fd2fc070d60 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:28.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:28.316+0000 7fd300e00700 1 --2- 172.21.15.123:0/1426994223 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2fc071ea0 0x7fd2fc0703b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:28.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:28.316+0000 7fd300e00700 1 -- 172.21.15.123:0/1426994223 >> 172.21.15.123:0/1426994223 conn(0x7fd2fc0f83d0 msgr2=0x7fd2fc0fa7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:24:28.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:28.316+0000 7fd300e00700 1 -- 172.21.15.123:0/1426994223 shutdown_connections 2024-03-30T04:24:28.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:28.316+0000 7fd300e00700 1 -- 172.21.15.123:0/1426994223 wait complete. 2024-03-30T04:24:28.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:28.316+0000 7fd300e00700 1 Processor -- start 2024-03-30T04:24:28.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:28.317+0000 7fd300e00700 1 -- start start 2024-03-30T04:24:28.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:28.317+0000 7fd300e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2fc0708f0 0x7fd2fc113de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:24:28.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:28.317+0000 7fd300e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2fc071ea0 0x7fd2fc114320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:24:28.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:28.317+0000 7fd300e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2fc1150c0 con 0x7fd2fc071ea0 2024-03-30T04:24:28.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:28.317+0000 7fd300e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2fc115230 con 0x7fd2fc0708f0 2024-03-30T04:24:28.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:28.317+0000 7fd2faa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2fc071ea0 0x7fd2fc114320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:24:28.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:28.317+0000 7fd2faa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2fc071ea0 0x7fd2fc114320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49286/0 (socket says 172.21.15.123:49286) 2024-03-30T04:24:28.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:28.317+0000 7fd2faa00700 1 -- 172.21.15.123:0/802192798 learned_addr learned my addr 172.21.15.123:0/802192798 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:24:28.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:28.317+0000 7fd2fb400700 1 --2- 172.21.15.123:0/802192798 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2fc0708f0 0x7fd2fc113de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:24:28.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:28.318+0000 7fd2faa00700 1 -- 172.21.15.123:0/802192798 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2fc0708f0 msgr2=0x7fd2fc113de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:28.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:28.318+0000 7fd2faa00700 1 --2- 172.21.15.123:0/802192798 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2fc0708f0 0x7fd2fc113de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:28.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:28.318+0000 7fd2faa00700 1 -- 172.21.15.123:0/802192798 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e8005040 con 0x7fd2fc071ea0 2024-03-30T04:24:28.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:28.318+0000 7fd2faa00700 1 --2- 172.21.15.123:0/802192798 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2fc071ea0 0x7fd2fc114320 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fd2e800b3b0 tx=0x7fd2e8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:24:28.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:28.318+0000 7fd2f3400700 1 -- 172.21.15.123:0/802192798 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd2e800e040 con 0x7fd2fc071ea0 2024-03-30T04:24:28.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:28.318+0000 7fd2f3400700 1 -- 172.21.15.123:0/802192798 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd2e8004290 con 0x7fd2fc071ea0 2024-03-30T04:24:28.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:28.318+0000 7fd300e00700 1 -- 172.21.15.123:0/802192798 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2fc1154b0 con 0x7fd2fc071ea0 2024-03-30T04:24:28.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:28.318+0000 7fd2f3400700 1 -- 172.21.15.123:0/802192798 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd2e801c3f0 con 0x7fd2fc071ea0 2024-03-30T04:24:28.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:28.318+0000 7fd300e00700 1 -- 172.21.15.123:0/802192798 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2fc115a70 con 0x7fd2fc071ea0 2024-03-30T04:24:28.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:28.320+0000 7fd300e00700 1 -- 172.21.15.123:0/802192798 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2fc10c3e0 con 0x7fd2fc071ea0 2024-03-30T04:24:28.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:28.320+0000 7fd2f3400700 1 -- 172.21.15.123:0/802192798 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fd2e8013420 con 0x7fd2fc071ea0 2024-03-30T04:24:28.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:28.321+0000 7fd2f3400700 1 --2- 172.21.15.123:0/802192798 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fd2e00692b0 0x7fd2e006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:24:28.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:28.321+0000 7fd2f3400700 1 -- 172.21.15.123:0/802192798 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd2e8087ff0 con 0x7fd2fc071ea0 2024-03-30T04:24:28.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:28.322+0000 7fd2fb400700 1 --2- 172.21.15.123:0/802192798 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fd2e00692b0 0x7fd2e006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:24:28.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:28.322+0000 7fd2fb400700 1 --2- 172.21.15.123:0/802192798 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fd2e00692b0 0x7fd2e006b760 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc0fc790 tx=0x7fd2ec00b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:24:28.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:28.326+0000 7fd2f3400700 1 -- 172.21.15.123:0/802192798 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fd2e80547b0 con 0x7fd2fc071ea0 2024-03-30T04:24:28.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:28 smithi123 ceph-mon[35245]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:24:28.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:28.549+0000 7fd300e00700 1 -- 172.21.15.123:0/802192798 --> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7fd2fc115ee0 con 0x7fd2e00692b0 2024-03-30T04:24:28.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:28 smithi135 ceph-mon[44364]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:24:29.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:29 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-03-30T04:24:29.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:29 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-03-30T04:24:30.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:30.233+0000 7fd2f3400700 1 -- 172.21.15.123:0/802192798 <== mgr.14206 v2:172.21.15.123:6800/1564935593 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fd2fc115ee0 con 0x7fd2e00692b0 2024-03-30T04:24:30.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:30.237+0000 7fd300e00700 1 -- 172.21.15.123:0/802192798 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fd2e00692b0 msgr2=0x7fd2e006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:30.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:30.237+0000 7fd300e00700 1 --2- 172.21.15.123:0/802192798 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fd2e00692b0 0x7fd2e006b760 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc0fc790 tx=0x7fd2ec00b040 comp rx=0 tx=0).stop 2024-03-30T04:24:30.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:30.237+0000 7fd300e00700 1 -- 172.21.15.123:0/802192798 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2fc071ea0 msgr2=0x7fd2fc114320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:30.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:30.237+0000 7fd300e00700 1 --2- 172.21.15.123:0/802192798 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2fc071ea0 0x7fd2fc114320 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fd2e800b3b0 tx=0x7fd2e8009e30 comp rx=0 tx=0).stop 2024-03-30T04:24:30.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:30.237+0000 7fd300e00700 1 -- 172.21.15.123:0/802192798 shutdown_connections 2024-03-30T04:24:30.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:30.237+0000 7fd300e00700 1 --2- 172.21.15.123:0/802192798 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fd2e00692b0 0x7fd2e006b760 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:30.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:30.237+0000 7fd300e00700 1 --2- 172.21.15.123:0/802192798 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2fc0708f0 0x7fd2fc113de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:30.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:30.237+0000 7fd300e00700 1 --2- 172.21.15.123:0/802192798 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2fc071ea0 0x7fd2fc114320 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:30.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:30.237+0000 7fd300e00700 1 -- 172.21.15.123:0/802192798 >> 172.21.15.123:0/802192798 conn(0x7fd2fc0f83d0 msgr2=0x7fd2fc0f9cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:24:30.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:30.238+0000 7fd300e00700 1 -- 172.21.15.123:0/802192798 shutdown_connections 2024-03-30T04:24:30.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:30.239+0000 7fd300e00700 1 -- 172.21.15.123:0/802192798 wait complete. 2024-03-30T04:24:30.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:30 smithi123 ceph-mon[35245]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:24:30.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:30 smithi123 ceph-mon[35245]: from='client.14468 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:24:30.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:30 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-03-30T04:24:30.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:30 smithi123 ceph-mon[35245]: osdmap e41: 6 total, 6 up, 6 in 2024-03-30T04:24:30.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:30 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-03-30T04:24:30.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:30 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-mon-smithi123[35221]: 2024-03-30T04:24:30.210+0000 7f4ee5200700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-30T04:24:30.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:30 smithi135 ceph-mon[44364]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:24:30.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:30 smithi135 ceph-mon[44364]: from='client.14468 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:24:30.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:30 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-03-30T04:24:30.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:30 smithi135 ceph-mon[44364]: osdmap e41: 6 total, 6 up, 6 in 2024-03-30T04:24:30.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:30 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-03-30T04:24:31.001 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- bash -c 'ceph fs dump' 2024-03-30T04:24:31.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:31 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-03-30T04:24:31.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:31 smithi123 ceph-mon[35245]: osdmap e42: 6 total, 6 up, 6 in 2024-03-30T04:24:31.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:31 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-03-30T04:24:31.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:31 smithi123 ceph-mon[35245]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-30T04:24:31.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:31 smithi123 ceph-mon[35245]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-30T04:24:31.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:31 smithi123 ceph-mon[35245]: osdmap e43: 6 total, 6 up, 6 in 2024-03-30T04:24:31.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:31 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-03-30T04:24:31.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:31 smithi123 ceph-mon[35245]: fsmap cephfs:0 2024-03-30T04:24:31.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:31 smithi123 ceph-mon[35245]: Saving service mds.cephfs spec with placement count:4 2024-03-30T04:24:31.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:31 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:31.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:31 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:24:31.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:31 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:24:31.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:31 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:24:31.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:31 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:31.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:31 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi123.tqxvzr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T04:24:31.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:31 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi123.tqxvzr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T04:24:31.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:31 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:24:31.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:31 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-03-30T04:24:31.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:31 smithi135 ceph-mon[44364]: osdmap e42: 6 total, 6 up, 6 in 2024-03-30T04:24:31.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:31 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-03-30T04:24:31.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:31 smithi135 ceph-mon[44364]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-30T04:24:31.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:31 smithi135 ceph-mon[44364]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-30T04:24:31.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:31 smithi135 ceph-mon[44364]: osdmap e43: 6 total, 6 up, 6 in 2024-03-30T04:24:31.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:31 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-03-30T04:24:31.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:31 smithi135 ceph-mon[44364]: fsmap cephfs:0 2024-03-30T04:24:31.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:31 smithi135 ceph-mon[44364]: Saving service mds.cephfs spec with placement count:4 2024-03-30T04:24:31.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:31 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:31.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:31 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:24:31.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:31 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:24:31.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:31 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:24:31.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:31 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:31.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:31 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi123.tqxvzr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T04:24:31.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:31 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi123.tqxvzr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T04:24:31.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:31 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:24:32.318 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:32 smithi123 ceph-mon[35245]: Deploying daemon mds.cephfs.smithi123.tqxvzr on smithi123 2024-03-30T04:24:32.318 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:32 smithi123 ceph-mon[35245]: pgmap v198: 65 pgs: 17 creating+peering, 47 unknown, 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:24:32.318 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:32 smithi123 ceph-mon[35245]: osdmap e44: 6 total, 6 up, 6 in 2024-03-30T04:24:32.324 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:24:32.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:32 smithi135 ceph-mon[44364]: Deploying daemon mds.cephfs.smithi123.tqxvzr on smithi123 2024-03-30T04:24:32.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:32 smithi135 ceph-mon[44364]: pgmap v198: 65 pgs: 17 creating+peering, 47 unknown, 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:24:32.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:32 smithi135 ceph-mon[44364]: osdmap e44: 6 total, 6 up, 6 in 2024-03-30T04:24:33.555 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:33 smithi123 ceph-mon[35245]: osdmap e45: 6 total, 6 up, 6 in 2024-03-30T04:24:33.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:33 smithi135 ceph-mon[44364]: osdmap e45: 6 total, 6 up, 6 in 2024-03-30T04:24:34.560 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:34 smithi123 ceph-mon[35245]: pgmap v201: 65 pgs: 8 creating+activating, 22 creating+peering, 22 unknown, 13 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:24:34.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:34 smithi135 ceph-mon[44364]: pgmap v201: 65 pgs: 8 creating+activating, 22 creating+peering, 22 unknown, 13 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:24:34.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:34.614+0000 7f9a00600700 1 -- 172.21.15.123:0/4154255718 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99f80fe0d0 msgr2=0x7f99f80fe540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:34.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:34.614+0000 7f9a00600700 1 --2- 172.21.15.123:0/4154255718 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99f80fe0d0 0x7f99f80fe540 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f99e4004940 tx=0x7f99e4009e30 comp rx=0 tx=0).stop 2024-03-30T04:24:34.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:34.614+0000 7f9a00600700 1 -- 172.21.15.123:0/4154255718 shutdown_connections 2024-03-30T04:24:34.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:34.614+0000 7f9a00600700 1 --2- 172.21.15.123:0/4154255718 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99f80fe0d0 0x7f99f80fe540 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:34.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:34.614+0000 7f9a00600700 1 --2- 172.21.15.123:0/4154255718 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f99f80fcee0 0x7f99f80fd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:34.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:34.614+0000 7f9a00600700 1 -- 172.21.15.123:0/4154255718 >> 172.21.15.123:0/4154255718 conn(0x7f99f80f8560 msgr2=0x7f99f80fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:24:34.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:34.614+0000 7f9a00600700 1 -- 172.21.15.123:0/4154255718 shutdown_connections 2024-03-30T04:24:34.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:34.614+0000 7f9a00600700 1 -- 172.21.15.123:0/4154255718 wait complete. 2024-03-30T04:24:34.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:34.615+0000 7f9a00600700 1 Processor -- start 2024-03-30T04:24:34.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:34.615+0000 7f9a00600700 1 -- start start 2024-03-30T04:24:34.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:34.615+0000 7f9a00600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99f80fcee0 0x7f99f810d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:24:34.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:34.615+0000 7f9a00600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f99f80fe0d0 0x7f99f810deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:24:34.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:34.615+0000 7f9a00600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f810e4b0 con 0x7f99f80fcee0 2024-03-30T04:24:34.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:34.615+0000 7f9a00600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f810ed00 con 0x7f99f80fe0d0 2024-03-30T04:24:34.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:34.615+0000 7f99fdc00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99f80fcee0 0x7f99f810d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:24:34.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:34.615+0000 7f99fdc00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99f80fcee0 0x7f99f810d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57494/0 (socket says 172.21.15.123:57494) 2024-03-30T04:24:34.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:34.615+0000 7f99fdc00700 1 -- 172.21.15.123:0/408588705 learned_addr learned my addr 172.21.15.123:0/408588705 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:24:34.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:34.615+0000 7f99fd200700 1 --2- 172.21.15.123:0/408588705 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f99f80fe0d0 0x7f99f810deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:24:34.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:34.616+0000 7f99fdc00700 1 -- 172.21.15.123:0/408588705 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f99f80fe0d0 msgr2=0x7f99f810deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:34.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:34.616+0000 7f99fdc00700 1 --2- 172.21.15.123:0/408588705 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f99f80fe0d0 0x7f99f810deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:34.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:34.616+0000 7f99fdc00700 1 -- 172.21.15.123:0/408588705 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99e4005040 con 0x7f99f80fcee0 2024-03-30T04:24:34.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:34.616+0000 7f99fdc00700 1 --2- 172.21.15.123:0/408588705 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99f80fcee0 0x7f99f810d970 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f99ec002a80 tx=0x7f99ec004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:24:34.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:34.616+0000 7f99f5600700 1 -- 172.21.15.123:0/408588705 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99ec007cc0 con 0x7f99f80fcee0 2024-03-30T04:24:34.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:34.616+0000 7f99f5600700 1 -- 172.21.15.123:0/408588705 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f99ec00f040 con 0x7f99f80fcee0 2024-03-30T04:24:34.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:34.616+0000 7f9a00600700 1 -- 172.21.15.123:0/408588705 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99f810efe0 con 0x7f99f80fcee0 2024-03-30T04:24:34.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:34.616+0000 7f99f5600700 1 -- 172.21.15.123:0/408588705 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99ec00d9c0 con 0x7f99f80fcee0 2024-03-30T04:24:34.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:34.617+0000 7f9a00600700 1 -- 172.21.15.123:0/408588705 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99f810f520 con 0x7f99f80fcee0 2024-03-30T04:24:34.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:34.618+0000 7f9a00600700 1 -- 172.21.15.123:0/408588705 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99f805f5f0 con 0x7f99f80fcee0 2024-03-30T04:24:34.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:34.619+0000 7f99f5600700 1 -- 172.21.15.123:0/408588705 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f99ec00db20 con 0x7f99f80fcee0 2024-03-30T04:24:34.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:34.620+0000 7f99f5600700 1 --2- 172.21.15.123:0/408588705 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f99e00692b0 0x7f99e006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:24:34.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:34.620+0000 7f99f5600700 1 -- 172.21.15.123:0/408588705 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f99ec086f60 con 0x7f99f80fcee0 2024-03-30T04:24:34.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:34.620+0000 7f99fd200700 1 --2- 172.21.15.123:0/408588705 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f99e00692b0 0x7f99e006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:24:34.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:34.621+0000 7f99fd200700 1 --2- 172.21.15.123:0/408588705 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f99e00692b0 0x7f99e006b760 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f99e4009fd0 tx=0x7f99e4009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:24:34.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:34.629+0000 7f99f5600700 1 -- 172.21.15.123:0/408588705 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f99ec056c00 con 0x7f99f80fcee0 2024-03-30T04:24:34.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:34.872+0000 7f9a00600700 1 -- 172.21.15.123:0/408588705 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f99f810f890 con 0x7f99f80fcee0 2024-03-30T04:24:34.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:34.873+0000 7f99f5600700 1 -- 172.21.15.123:0/408588705 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 2 v2) v1 ==== 75+0+1076 (secure 0 0 0) 0x7f99ec056790 con 0x7f99f80fcee0 2024-03-30T04:24:34.871 INFO:teuthology.orchestra.run.smithi123.stdout:e2 2024-03-30T04:24:34.872 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:24:34.872 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:24:34.872 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:24:34.872 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:24:34.872 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:24:34.872 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:24:34.872 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 2 2024-03-30T04:24:34.872 INFO:teuthology.orchestra.run.smithi123.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-30T04:24:34.872 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:24:34.872 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:24:30.211833+0000 2024-03-30T04:24:34.872 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:24:34.872 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:24:34.872 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:24:34.872 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:24:34.873 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:24:34.873 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:24:34.873 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:24:34.873 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 0 2024-03-30T04:24:34.873 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:24:34.873 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 1 2024-03-30T04:24:34.873 INFO:teuthology.orchestra.run.smithi123.stdout:in 2024-03-30T04:24:34.873 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T04:24:34.873 INFO:teuthology.orchestra.run.smithi123.stdout:failed 2024-03-30T04:24:34.873 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:24:34.873 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:24:34.873 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:24:34.873 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:24:34.873 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:24:34.873 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:24:34.874 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 0 2024-03-30T04:24:34.874 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:24:34.874 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:24:34.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:34.875+0000 7f9a00600700 1 -- 172.21.15.123:0/408588705 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f99e00692b0 msgr2=0x7f99e006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:34.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:34.875+0000 7f9a00600700 1 --2- 172.21.15.123:0/408588705 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f99e00692b0 0x7f99e006b760 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f99e4009fd0 tx=0x7f99e4009e30 comp rx=0 tx=0).stop 2024-03-30T04:24:34.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:34.875+0000 7f9a00600700 1 -- 172.21.15.123:0/408588705 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99f80fcee0 msgr2=0x7f99f810d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:34.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:34.875+0000 7f9a00600700 1 --2- 172.21.15.123:0/408588705 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99f80fcee0 0x7f99f810d970 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f99ec002a80 tx=0x7f99ec004600 comp rx=0 tx=0).stop 2024-03-30T04:24:34.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:34.875+0000 7f9a00600700 1 -- 172.21.15.123:0/408588705 shutdown_connections 2024-03-30T04:24:34.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:34.875+0000 7f9a00600700 1 --2- 172.21.15.123:0/408588705 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f99e00692b0 0x7f99e006b760 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:34.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:34.875+0000 7f9a00600700 1 --2- 172.21.15.123:0/408588705 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99f80fcee0 0x7f99f810d970 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:34.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:34.875+0000 7f9a00600700 1 --2- 172.21.15.123:0/408588705 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f99f80fe0d0 0x7f99f810deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:34.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:34.875+0000 7f9a00600700 1 -- 172.21.15.123:0/408588705 >> 172.21.15.123:0/408588705 conn(0x7f99f80f8560 msgr2=0x7f99f80f9d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:24:34.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:34.875+0000 7f9a00600700 1 -- 172.21.15.123:0/408588705 shutdown_connections 2024-03-30T04:24:34.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:34.875+0000 7f9a00600700 1 -- 172.21.15.123:0/408588705 wait complete. 2024-03-30T04:24:34.875 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 2 2024-03-30T04:24:35.490 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-30T04:24:35.499 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi123.front.sepia.ceph.com 2024-03-30T04:24:35.499 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- bash -c 'ceph fs set cephfs max_mds 2' 2024-03-30T04:24:35.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:35 smithi123 ceph-mon[35245]: pgmap v202: 65 pgs: 8 creating+activating, 22 creating+peering, 13 unknown, 22 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:24:35.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:35 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:35.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:35 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:35.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:35 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:35.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:35 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi135.zvnlap", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T04:24:35.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:35 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi135.zvnlap", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T04:24:35.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:35 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:24:35.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:35 smithi123 ceph-mon[35245]: Deploying daemon mds.cephfs.smithi135.zvnlap on smithi135 2024-03-30T04:24:35.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:35 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/408588705' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:24:35.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:35 smithi123 ceph-mon[35245]: mds.? [v2:172.21.15.123:6826/2393193575,v1:172.21.15.123:6827/2393193575] up:boot 2024-03-30T04:24:35.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:35 smithi123 ceph-mon[35245]: daemon mds.cephfs.smithi123.tqxvzr assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-03-30T04:24:35.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:35 smithi123 ceph-mon[35245]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-30T04:24:35.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:35 smithi123 ceph-mon[35245]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-30T04:24:35.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:35 smithi123 ceph-mon[35245]: Cluster is now healthy 2024-03-30T04:24:35.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:35 smithi123 ceph-mon[35245]: fsmap cephfs:0 1 up:standby 2024-03-30T04:24:35.718 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:35 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi123.tqxvzr"}]: dispatch 2024-03-30T04:24:35.718 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:35 smithi123 ceph-mon[35245]: fsmap cephfs:1 {0=cephfs.smithi123.tqxvzr=up:creating} 2024-03-30T04:24:35.718 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:35 smithi123 ceph-mon[35245]: daemon mds.cephfs.smithi123.tqxvzr is now active in filesystem cephfs as rank 0 2024-03-30T04:24:35.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:35 smithi135 ceph-mon[44364]: pgmap v202: 65 pgs: 8 creating+activating, 22 creating+peering, 13 unknown, 22 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:24:35.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:35 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:35.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:35 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:35.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:35 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:35.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:35 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi135.zvnlap", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T04:24:35.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:35 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi135.zvnlap", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T04:24:35.852 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:35 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:24:35.852 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:35 smithi135 ceph-mon[44364]: Deploying daemon mds.cephfs.smithi135.zvnlap on smithi135 2024-03-30T04:24:35.852 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:35 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/408588705' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:24:35.852 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:35 smithi135 ceph-mon[44364]: mds.? [v2:172.21.15.123:6826/2393193575,v1:172.21.15.123:6827/2393193575] up:boot 2024-03-30T04:24:35.852 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:35 smithi135 ceph-mon[44364]: daemon mds.cephfs.smithi123.tqxvzr assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-03-30T04:24:35.852 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:35 smithi135 ceph-mon[44364]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-30T04:24:35.852 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:35 smithi135 ceph-mon[44364]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-30T04:24:35.852 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:35 smithi135 ceph-mon[44364]: Cluster is now healthy 2024-03-30T04:24:35.852 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:35 smithi135 ceph-mon[44364]: fsmap cephfs:0 1 up:standby 2024-03-30T04:24:35.852 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:35 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi123.tqxvzr"}]: dispatch 2024-03-30T04:24:35.852 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:35 smithi135 ceph-mon[44364]: fsmap cephfs:1 {0=cephfs.smithi123.tqxvzr=up:creating} 2024-03-30T04:24:35.852 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:35 smithi135 ceph-mon[44364]: daemon mds.cephfs.smithi123.tqxvzr is now active in filesystem cephfs as rank 0 2024-03-30T04:24:35.923 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:24:37.389 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:37 smithi123 ceph-mon[35245]: mds.? [v2:172.21.15.123:6826/2393193575,v1:172.21.15.123:6827/2393193575] up:active 2024-03-30T04:24:37.389 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:37 smithi123 ceph-mon[35245]: fsmap cephfs:1 {0=cephfs.smithi123.tqxvzr=up:active} 2024-03-30T04:24:37.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:37 smithi135 ceph-mon[44364]: mds.? [v2:172.21.15.123:6826/2393193575,v1:172.21.15.123:6827/2393193575] up:active 2024-03-30T04:24:37.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:37 smithi135 ceph-mon[44364]: fsmap cephfs:1 {0=cephfs.smithi123.tqxvzr=up:active} 2024-03-30T04:24:37.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:37.700+0000 7f3b95c00700 1 -- 172.21.15.123:0/1314311113 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b900fe0d0 msgr2=0x7f3b900fe540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:37.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:37.700+0000 7f3b95c00700 1 --2- 172.21.15.123:0/1314311113 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b900fe0d0 0x7f3b900fe540 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c004940 tx=0x7f3b7c009e30 comp rx=0 tx=0).stop 2024-03-30T04:24:37.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:37.700+0000 7f3b95c00700 1 -- 172.21.15.123:0/1314311113 shutdown_connections 2024-03-30T04:24:37.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:37.700+0000 7f3b95c00700 1 --2- 172.21.15.123:0/1314311113 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b900fe0d0 0x7f3b900fe540 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:37.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:37.700+0000 7f3b95c00700 1 --2- 172.21.15.123:0/1314311113 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b900fcee0 0x7f3b900fd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:37.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:37.700+0000 7f3b95c00700 1 -- 172.21.15.123:0/1314311113 >> 172.21.15.123:0/1314311113 conn(0x7f3b900f8560 msgr2=0x7f3b900fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:24:37.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:37.700+0000 7f3b95c00700 1 -- 172.21.15.123:0/1314311113 shutdown_connections 2024-03-30T04:24:37.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:37.700+0000 7f3b95c00700 1 -- 172.21.15.123:0/1314311113 wait complete. 2024-03-30T04:24:37.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:37.701+0000 7f3b95c00700 1 Processor -- start 2024-03-30T04:24:37.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:37.701+0000 7f3b95c00700 1 -- start start 2024-03-30T04:24:37.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:37.702+0000 7f3b95c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b900fcee0 0x7f3b9010faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:24:37.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:37.702+0000 7f3b95c00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b900fe0d0 0x7f3b90110030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:24:37.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:37.702+0000 7f3b95c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b90110630 con 0x7f3b900fcee0 2024-03-30T04:24:37.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:37.702+0000 7f3b95c00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b90110e80 con 0x7f3b900fe0d0 2024-03-30T04:24:37.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:37.702+0000 7f3b8ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b900fcee0 0x7f3b9010faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:24:37.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:37.702+0000 7f3b8ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b900fcee0 0x7f3b9010faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57516/0 (socket says 172.21.15.123:57516) 2024-03-30T04:24:37.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:37.702+0000 7f3b8ea00700 1 -- 172.21.15.123:0/1532531242 learned_addr learned my addr 172.21.15.123:0/1532531242 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:24:37.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:37.702+0000 7f3b8e000700 1 --2- 172.21.15.123:0/1532531242 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b900fe0d0 0x7f3b90110030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:24:37.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:37.702+0000 7f3b8ea00700 1 -- 172.21.15.123:0/1532531242 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b900fe0d0 msgr2=0x7f3b90110030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:37.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:37.702+0000 7f3b8ea00700 1 --2- 172.21.15.123:0/1532531242 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b900fe0d0 0x7f3b90110030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:37.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:37.702+0000 7f3b8ea00700 1 -- 172.21.15.123:0/1532531242 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b7c005040 con 0x7f3b900fcee0 2024-03-30T04:24:37.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:37.703+0000 7f3b8ea00700 1 --2- 172.21.15.123:0/1532531242 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b900fcee0 0x7f3b9010faf0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3b8000eb70 tx=0x7f3b8000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:24:37.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:37.703+0000 7f3b86a00700 1 -- 172.21.15.123:0/1532531242 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b80007cc0 con 0x7f3b900fcee0 2024-03-30T04:24:37.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:37.703+0000 7f3b86a00700 1 -- 172.21.15.123:0/1532531242 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3b8000f040 con 0x7f3b900fcee0 2024-03-30T04:24:37.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:37.703+0000 7f3b95c00700 1 -- 172.21.15.123:0/1532531242 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b90111100 con 0x7f3b900fcee0 2024-03-30T04:24:37.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:37.703+0000 7f3b86a00700 1 -- 172.21.15.123:0/1532531242 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b8000c9d0 con 0x7f3b900fcee0 2024-03-30T04:24:37.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:37.703+0000 7f3b95c00700 1 -- 172.21.15.123:0/1532531242 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b90111640 con 0x7f3b900fcee0 2024-03-30T04:24:37.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:37.704+0000 7f3b95c00700 1 -- 172.21.15.123:0/1532531242 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b90101a60 con 0x7f3b900fcee0 2024-03-30T04:24:37.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:37.705+0000 7f3b86a00700 1 -- 172.21.15.123:0/1532531242 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f3b8000cb30 con 0x7f3b900fcee0 2024-03-30T04:24:37.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:37.706+0000 7f3b86a00700 1 --2- 172.21.15.123:0/1532531242 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f3b740692a0 0x7f3b7406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:24:37.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:37.706+0000 7f3b86a00700 1 -- 172.21.15.123:0/1532531242 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f3b80086c30 con 0x7f3b900fcee0 2024-03-30T04:24:37.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:37.706+0000 7f3b8e000700 1 --2- 172.21.15.123:0/1532531242 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f3b740692a0 0x7f3b7406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:24:37.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:37.707+0000 7f3b8e000700 1 --2- 172.21.15.123:0/1532531242 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f3b740692a0 0x7f3b7406b750 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c004890 tx=0x7f3b7c009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:24:37.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:37.711+0000 7f3b86a00700 1 -- 172.21.15.123:0/1532531242 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f3b8004e190 con 0x7f3b900fcee0 2024-03-30T04:24:37.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:37.954+0000 7f3b95c00700 1 -- 172.21.15.123:0/1532531242 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7f3b9004bed0 con 0x7f3b900fcee0 2024-03-30T04:24:38.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:38.257+0000 7f3b86a00700 1 -- 172.21.15.123:0/1532531242 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v6) v1 ==== 105+0+0 (secure 0 0 0) 0x7f3b80056a00 con 0x7f3b900fcee0 2024-03-30T04:24:38.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:38.260+0000 7f3b95c00700 1 -- 172.21.15.123:0/1532531242 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f3b740692a0 msgr2=0x7f3b7406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:38.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:38.260+0000 7f3b95c00700 1 --2- 172.21.15.123:0/1532531242 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f3b740692a0 0x7f3b7406b750 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c004890 tx=0x7f3b7c009e30 comp rx=0 tx=0).stop 2024-03-30T04:24:38.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:38.260+0000 7f3b95c00700 1 -- 172.21.15.123:0/1532531242 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b900fcee0 msgr2=0x7f3b9010faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:38.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:38.260+0000 7f3b95c00700 1 --2- 172.21.15.123:0/1532531242 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b900fcee0 0x7f3b9010faf0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3b8000eb70 tx=0x7f3b8000b040 comp rx=0 tx=0).stop 2024-03-30T04:24:38.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:38.260+0000 7f3b95c00700 1 -- 172.21.15.123:0/1532531242 shutdown_connections 2024-03-30T04:24:38.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:38.260+0000 7f3b95c00700 1 --2- 172.21.15.123:0/1532531242 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f3b740692a0 0x7f3b7406b750 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:38.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:38.260+0000 7f3b95c00700 1 --2- 172.21.15.123:0/1532531242 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b900fcee0 0x7f3b9010faf0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:38.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:38.260+0000 7f3b95c00700 1 --2- 172.21.15.123:0/1532531242 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b900fe0d0 0x7f3b90110030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:38.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:38.260+0000 7f3b95c00700 1 -- 172.21.15.123:0/1532531242 >> 172.21.15.123:0/1532531242 conn(0x7f3b900f8560 msgr2=0x7f3b900f9c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:24:38.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:38.260+0000 7f3b95c00700 1 -- 172.21.15.123:0/1532531242 shutdown_connections 2024-03-30T04:24:38.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:38.260+0000 7f3b95c00700 1 -- 172.21.15.123:0/1532531242 wait complete. 2024-03-30T04:24:38.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:38 smithi135 ceph-mon[44364]: pgmap v203: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 336 B/s wr, 0 op/s 2024-03-30T04:24:38.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:38 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/1532531242' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-03-30T04:24:38.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:38 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:38.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:38 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:38.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:38 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:38.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:38 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi123.dvtind", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T04:24:38.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:38 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi123.dvtind", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T04:24:38.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:38 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:24:38.651 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:38 smithi123 ceph-mon[35245]: pgmap v203: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 336 B/s wr, 0 op/s 2024-03-30T04:24:38.651 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:38 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/1532531242' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-03-30T04:24:38.651 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:38 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:38.651 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:38 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:38.651 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:38 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:38.651 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:38 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi123.dvtind", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T04:24:38.651 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:38 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi123.dvtind", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T04:24:38.651 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:38 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:24:39.111 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-30T04:24:39.129 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi123.front.sepia.ceph.com 2024-03-30T04:24:39.129 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- bash -c 'ceph fs set cephfs allow_standby_replay true' 2024-03-30T04:24:39.397 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:39 smithi123 ceph-mon[35245]: Deploying daemon mds.cephfs.smithi123.dvtind on smithi123 2024-03-30T04:24:39.397 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:39 smithi123 ceph-mon[35245]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-30T04:24:39.397 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:39 smithi123 ceph-mon[35245]: mds.? [v2:172.21.15.135:6824/4050541062,v1:172.21.15.135:6825/4050541062] up:boot 2024-03-30T04:24:39.397 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:39 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/1532531242' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-03-30T04:24:39.397 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:39 smithi123 ceph-mon[35245]: daemon mds.cephfs.smithi135.zvnlap assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-30T04:24:39.398 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:39 smithi123 ceph-mon[35245]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-03-30T04:24:39.398 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:39 smithi123 ceph-mon[35245]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-30T04:24:39.398 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:39 smithi123 ceph-mon[35245]: fsmap cephfs:1 {0=cephfs.smithi123.tqxvzr=up:active} 1 up:standby 2024-03-30T04:24:39.398 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:39 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi135.zvnlap"}]: dispatch 2024-03-30T04:24:39.398 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:39 smithi123 ceph-mon[35245]: fsmap cephfs:2 {0=cephfs.smithi123.tqxvzr=up:active,1=cephfs.smithi135.zvnlap=up:creating} 2024-03-30T04:24:39.398 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:39 smithi123 ceph-mon[35245]: daemon mds.cephfs.smithi135.zvnlap is now active in filesystem cephfs as rank 1 2024-03-30T04:24:39.605 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:39 smithi135 ceph-mon[44364]: Deploying daemon mds.cephfs.smithi123.dvtind on smithi123 2024-03-30T04:24:39.605 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:39 smithi135 ceph-mon[44364]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-30T04:24:39.605 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:39 smithi135 ceph-mon[44364]: mds.? [v2:172.21.15.135:6824/4050541062,v1:172.21.15.135:6825/4050541062] up:boot 2024-03-30T04:24:39.605 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:39 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/1532531242' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-03-30T04:24:39.605 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:39 smithi135 ceph-mon[44364]: daemon mds.cephfs.smithi135.zvnlap assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-30T04:24:39.605 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:39 smithi135 ceph-mon[44364]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-03-30T04:24:39.605 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:39 smithi135 ceph-mon[44364]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-30T04:24:39.605 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:39 smithi135 ceph-mon[44364]: fsmap cephfs:1 {0=cephfs.smithi123.tqxvzr=up:active} 1 up:standby 2024-03-30T04:24:39.606 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:39 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi135.zvnlap"}]: dispatch 2024-03-30T04:24:39.606 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:39 smithi135 ceph-mon[44364]: fsmap cephfs:2 {0=cephfs.smithi123.tqxvzr=up:active,1=cephfs.smithi135.zvnlap=up:creating} 2024-03-30T04:24:39.606 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:39 smithi135 ceph-mon[44364]: daemon mds.cephfs.smithi135.zvnlap is now active in filesystem cephfs as rank 1 2024-03-30T04:24:40.448 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:24:40.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:40 smithi135 ceph-mon[44364]: pgmap v204: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s wr, 1 op/s 2024-03-30T04:24:40.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:40 smithi135 ceph-mon[44364]: mds.? [v2:172.21.15.135:6824/4050541062,v1:172.21.15.135:6825/4050541062] up:active 2024-03-30T04:24:40.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:40 smithi135 ceph-mon[44364]: fsmap cephfs:2 {0=cephfs.smithi123.tqxvzr=up:active,1=cephfs.smithi135.zvnlap=up:active} 2024-03-30T04:24:40.695 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:40 smithi123 ceph-mon[35245]: pgmap v204: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s wr, 1 op/s 2024-03-30T04:24:40.695 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:40 smithi123 ceph-mon[35245]: mds.? [v2:172.21.15.135:6824/4050541062,v1:172.21.15.135:6825/4050541062] up:active 2024-03-30T04:24:40.695 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:40 smithi123 ceph-mon[35245]: fsmap cephfs:2 {0=cephfs.smithi123.tqxvzr=up:active,1=cephfs.smithi135.zvnlap=up:active} 2024-03-30T04:24:41.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:41 smithi123 ceph-mon[35245]: pgmap v205: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s wr, 6 op/s 2024-03-30T04:24:41.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:41 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:41.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:41 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:24:41.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:41 smithi135 ceph-mon[44364]: pgmap v205: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s wr, 6 op/s 2024-03-30T04:24:41.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:41 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:41.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:41 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:24:42.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:42 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:42.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:42 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:42.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:42.731+0000 7fb941a00700 1 -- 172.21.15.123:0/943427427 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb93c0fe0d0 msgr2=0x7fb93c0fe540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:42.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:42.731+0000 7fb941a00700 1 --2- 172.21.15.123:0/943427427 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb93c0fe0d0 0x7fb93c0fe540 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb928004940 tx=0x7fb928009e30 comp rx=0 tx=0).stop 2024-03-30T04:24:42.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:42.731+0000 7fb941a00700 1 -- 172.21.15.123:0/943427427 shutdown_connections 2024-03-30T04:24:42.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:42.731+0000 7fb941a00700 1 --2- 172.21.15.123:0/943427427 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb93c0fe0d0 0x7fb93c0fe540 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:42.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:42.731+0000 7fb941a00700 1 --2- 172.21.15.123:0/943427427 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb93c0fcee0 0x7fb93c0fd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:42.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:42.731+0000 7fb941a00700 1 -- 172.21.15.123:0/943427427 >> 172.21.15.123:0/943427427 conn(0x7fb93c0f8560 msgr2=0x7fb93c0fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:24:42.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:42.731+0000 7fb941a00700 1 -- 172.21.15.123:0/943427427 shutdown_connections 2024-03-30T04:24:42.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:42.731+0000 7fb941a00700 1 -- 172.21.15.123:0/943427427 wait complete. 2024-03-30T04:24:42.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:42.732+0000 7fb941a00700 1 Processor -- start 2024-03-30T04:24:42.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:42.732+0000 7fb941a00700 1 -- start start 2024-03-30T04:24:42.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:42.733+0000 7fb941a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb93c0fcee0 0x7fb93c10d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:24:42.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:42.733+0000 7fb941a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb93c0fe0d0 0x7fb93c10de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:24:42.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:42.733+0000 7fb941a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb93c10e490 con 0x7fb93c0fe0d0 2024-03-30T04:24:42.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:42.733+0000 7fb941a00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb93c10ec70 con 0x7fb93c0fcee0 2024-03-30T04:24:42.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:42.733+0000 7fb93a000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb93c0fe0d0 0x7fb93c10de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:24:42.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:42.733+0000 7fb93a000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb93c0fe0d0 0x7fb93c10de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57540/0 (socket says 172.21.15.123:57540) 2024-03-30T04:24:42.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:42.733+0000 7fb93a000700 1 -- 172.21.15.123:0/837331723 learned_addr learned my addr 172.21.15.123:0/837331723 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:24:42.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:42.733+0000 7fb93aa00700 1 --2- 172.21.15.123:0/837331723 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb93c0fcee0 0x7fb93c10d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:24:42.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:42.734+0000 7fb93a000700 1 -- 172.21.15.123:0/837331723 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb93c0fcee0 msgr2=0x7fb93c10d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:42.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:42.734+0000 7fb93a000700 1 --2- 172.21.15.123:0/837331723 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb93c0fcee0 0x7fb93c10d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:42.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:42.734+0000 7fb93a000700 1 -- 172.21.15.123:0/837331723 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb928005040 con 0x7fb93c0fe0d0 2024-03-30T04:24:42.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:42.734+0000 7fb93a000700 1 --2- 172.21.15.123:0/837331723 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb93c0fe0d0 0x7fb93c10de90 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb9280137b0 tx=0x7fb928009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:24:42.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:42.734+0000 7fb932a00700 1 -- 172.21.15.123:0/837331723 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb92800e070 con 0x7fb93c0fe0d0 2024-03-30T04:24:42.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:42.734+0000 7fb941a00700 1 -- 172.21.15.123:0/837331723 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb93c10eef0 con 0x7fb93c0fe0d0 2024-03-30T04:24:42.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:42.734+0000 7fb932a00700 1 -- 172.21.15.123:0/837331723 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb92801c400 con 0x7fb93c0fe0d0 2024-03-30T04:24:42.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:42.734+0000 7fb932a00700 1 -- 172.21.15.123:0/837331723 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb92800b650 con 0x7fb93c0fe0d0 2024-03-30T04:24:42.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:42.734+0000 7fb941a00700 1 -- 172.21.15.123:0/837331723 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb93c10f3d0 con 0x7fb93c0fe0d0 2024-03-30T04:24:42.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:42.735+0000 7fb941a00700 1 -- 172.21.15.123:0/837331723 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb93c05f5f0 con 0x7fb93c0fe0d0 2024-03-30T04:24:42.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:42.736+0000 7fb932a00700 1 -- 172.21.15.123:0/837331723 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fb92801c570 con 0x7fb93c0fe0d0 2024-03-30T04:24:42.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:42.737+0000 7fb932a00700 1 --2- 172.21.15.123:0/837331723 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fb9200692b0 0x7fb92006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:24:42.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:42.737+0000 7fb932a00700 1 -- 172.21.15.123:0/837331723 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fb9280887d0 con 0x7fb93c0fe0d0 2024-03-30T04:24:42.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:42.737+0000 7fb93aa00700 1 --2- 172.21.15.123:0/837331723 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fb9200692b0 0x7fb92006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:24:42.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:42.738+0000 7fb93aa00700 1 --2- 172.21.15.123:0/837331723 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fb9200692b0 0x7fb92006b760 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb92c009b80 tx=0x7fb92c00b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:24:42.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:42.742+0000 7fb932a00700 1 -- 172.21.15.123:0/837331723 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fb928058530 con 0x7fb93c0fe0d0 2024-03-30T04:24:42.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:42.988+0000 7fb941a00700 1 -- 172.21.15.123:0/837331723 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7fb93c04bed0 con 0x7fb93c0fe0d0 2024-03-30T04:24:43.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:43.348+0000 7fb932a00700 1 -- 172.21.15.123:0/837331723 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v9) v1 ==== 121+0+0 (secure 0 0 0) 0x7fb9280580c0 con 0x7fb93c0fe0d0 2024-03-30T04:24:43.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:43.351+0000 7fb941a00700 1 -- 172.21.15.123:0/837331723 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fb9200692b0 msgr2=0x7fb92006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:43.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:43.351+0000 7fb941a00700 1 --2- 172.21.15.123:0/837331723 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fb9200692b0 0x7fb92006b760 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb92c009b80 tx=0x7fb92c00b040 comp rx=0 tx=0).stop 2024-03-30T04:24:43.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:43.351+0000 7fb941a00700 1 -- 172.21.15.123:0/837331723 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb93c0fe0d0 msgr2=0x7fb93c10de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:43.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:43.351+0000 7fb941a00700 1 --2- 172.21.15.123:0/837331723 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb93c0fe0d0 0x7fb93c10de90 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb9280137b0 tx=0x7fb928009e30 comp rx=0 tx=0).stop 2024-03-30T04:24:43.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:43.351+0000 7fb941a00700 1 -- 172.21.15.123:0/837331723 shutdown_connections 2024-03-30T04:24:43.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:43.351+0000 7fb941a00700 1 --2- 172.21.15.123:0/837331723 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fb9200692b0 0x7fb92006b760 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:43.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:43.351+0000 7fb941a00700 1 --2- 172.21.15.123:0/837331723 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb93c0fcee0 0x7fb93c10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:43.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:43.351+0000 7fb941a00700 1 --2- 172.21.15.123:0/837331723 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb93c0fe0d0 0x7fb93c10de90 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:43.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:43.351+0000 7fb941a00700 1 -- 172.21.15.123:0/837331723 >> 172.21.15.123:0/837331723 conn(0x7fb93c0f8560 msgr2=0x7fb93c0f9d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:24:43.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:43.351+0000 7fb941a00700 1 -- 172.21.15.123:0/837331723 shutdown_connections 2024-03-30T04:24:43.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:43.351+0000 7fb941a00700 1 -- 172.21.15.123:0/837331723 wait complete. 2024-03-30T04:24:43.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:43 smithi123 ceph-mon[35245]: pgmap v206: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s wr, 8 op/s 2024-03-30T04:24:43.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:43 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:43.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:43 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:43.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:43 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:43.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:43 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi135.xwgjvb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T04:24:43.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:43 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi135.xwgjvb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T04:24:43.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:43 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:24:43.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:43 smithi123 ceph-mon[35245]: Deploying daemon mds.cephfs.smithi135.xwgjvb on smithi135 2024-03-30T04:24:43.968 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:43 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/837331723' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-30T04:24:43.968 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:43 smithi123 ceph-mon[35245]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-03-30T04:24:43.968 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:43 smithi123 ceph-mon[35245]: Cluster is now healthy 2024-03-30T04:24:43.968 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:43 smithi123 ceph-mon[35245]: mds.? [v2:172.21.15.123:6828/3037849622,v1:172.21.15.123:6829/3037849622] up:boot 2024-03-30T04:24:43.968 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:43 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/837331723' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-30T04:24:43.968 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:43 smithi123 ceph-mon[35245]: fsmap cephfs:2 {0=cephfs.smithi123.tqxvzr=up:active,1=cephfs.smithi135.zvnlap=up:active} 1 up:standby 2024-03-30T04:24:43.968 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:43 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi123.dvtind"}]: dispatch 2024-03-30T04:24:43.968 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:43 smithi123 ceph-mon[35245]: fsmap cephfs:2 {0=cephfs.smithi123.tqxvzr=up:active,1=cephfs.smithi135.zvnlap=up:active} 1 up:standby-replay 2024-03-30T04:24:44.024 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-30T04:24:44.035 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi123.front.sepia.ceph.com 2024-03-30T04:24:44.035 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- bash -c 'ceph fs set cephfs inline_data false' 2024-03-30T04:24:44.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:43 smithi135 ceph-mon[44364]: pgmap v206: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s wr, 8 op/s 2024-03-30T04:24:44.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:43 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:44.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:43 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:44.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:43 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:44.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:43 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi135.xwgjvb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T04:24:44.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:43 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi135.xwgjvb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T04:24:44.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:43 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:24:44.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:43 smithi135 ceph-mon[44364]: Deploying daemon mds.cephfs.smithi135.xwgjvb on smithi135 2024-03-30T04:24:44.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:43 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/837331723' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-30T04:24:44.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:43 smithi135 ceph-mon[44364]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-03-30T04:24:44.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:43 smithi135 ceph-mon[44364]: Cluster is now healthy 2024-03-30T04:24:44.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:43 smithi135 ceph-mon[44364]: mds.? [v2:172.21.15.123:6828/3037849622,v1:172.21.15.123:6829/3037849622] up:boot 2024-03-30T04:24:44.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:43 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/837331723' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-30T04:24:44.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:43 smithi135 ceph-mon[44364]: fsmap cephfs:2 {0=cephfs.smithi123.tqxvzr=up:active,1=cephfs.smithi135.zvnlap=up:active} 1 up:standby 2024-03-30T04:24:44.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:43 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi123.dvtind"}]: dispatch 2024-03-30T04:24:44.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:43 smithi135 ceph-mon[44364]: fsmap cephfs:2 {0=cephfs.smithi123.tqxvzr=up:active,1=cephfs.smithi135.zvnlap=up:active} 1 up:standby-replay 2024-03-30T04:24:44.462 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:24:46.109 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:45 smithi123 ceph-mon[35245]: pgmap v207: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s wr, 6 op/s 2024-03-30T04:24:46.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:45 smithi135 ceph-mon[44364]: pgmap v207: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s wr, 6 op/s 2024-03-30T04:24:46.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:46.416+0000 7f661da00700 1 -- 172.21.15.123:0/2298244527 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66180ffe70 msgr2=0x7f6618102290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:46.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:46.416+0000 7f661da00700 1 --2- 172.21.15.123:0/2298244527 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66180ffe70 0x7f6618102290 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f6604004940 tx=0x7f6604009e30 comp rx=0 tx=0).stop 2024-03-30T04:24:46.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:46.416+0000 7f661da00700 1 -- 172.21.15.123:0/2298244527 shutdown_connections 2024-03-30T04:24:46.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:46.416+0000 7f661da00700 1 --2- 172.21.15.123:0/2298244527 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66180ffe70 0x7f6618102290 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:46.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:46.416+0000 7f661da00700 1 --2- 172.21.15.123:0/2298244527 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f66180662e0 0x7f66180ff930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:46.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:46.416+0000 7f661da00700 1 -- 172.21.15.123:0/2298244527 >> 172.21.15.123:0/2298244527 conn(0x7f66180f73c0 msgr2=0x7f66180f97d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:24:46.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:46.416+0000 7f661da00700 1 -- 172.21.15.123:0/2298244527 shutdown_connections 2024-03-30T04:24:46.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:46.416+0000 7f661da00700 1 -- 172.21.15.123:0/2298244527 wait complete. 2024-03-30T04:24:46.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:46.417+0000 7f661da00700 1 Processor -- start 2024-03-30T04:24:46.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:46.417+0000 7f661da00700 1 -- start start 2024-03-30T04:24:46.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:46.417+0000 7f661da00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f66180662e0 0x7f661810fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:24:46.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:46.417+0000 7f661da00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66180ffe70 0x7f6618110000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:24:46.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:46.417+0000 7f661da00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6618110600 con 0x7f66180ffe70 2024-03-30T04:24:46.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:46.417+0000 7f661da00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6618110e50 con 0x7f66180662e0 2024-03-30T04:24:46.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:46.418+0000 7f6616000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66180ffe70 0x7f6618110000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:24:46.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:46.418+0000 7f6616000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66180ffe70 0x7f6618110000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39820/0 (socket says 172.21.15.123:39820) 2024-03-30T04:24:46.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:46.418+0000 7f6616000700 1 -- 172.21.15.123:0/3341310608 learned_addr learned my addr 172.21.15.123:0/3341310608 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:24:46.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:46.418+0000 7f6616a00700 1 --2- 172.21.15.123:0/3341310608 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f66180662e0 0x7f661810fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:24:46.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:46.418+0000 7f6616000700 1 -- 172.21.15.123:0/3341310608 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f66180662e0 msgr2=0x7f661810fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:46.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:46.418+0000 7f6616000700 1 --2- 172.21.15.123:0/3341310608 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f66180662e0 0x7f661810fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:46.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:46.418+0000 7f6616000700 1 -- 172.21.15.123:0/3341310608 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6604005040 con 0x7f66180ffe70 2024-03-30T04:24:46.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:46.418+0000 7f6616a00700 1 --2- 172.21.15.123:0/3341310608 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f66180662e0 0x7f661810fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:24:46.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:46.418+0000 7f6616000700 1 --2- 172.21.15.123:0/3341310608 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66180ffe70 0x7f6618110000 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f660400b0d0 tx=0x7f6604009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:24:46.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:46.419+0000 7f660ea00700 1 -- 172.21.15.123:0/3341310608 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f660400e070 con 0x7f66180ffe70 2024-03-30T04:24:46.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:46.419+0000 7f660ea00700 1 -- 172.21.15.123:0/3341310608 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f660400b410 con 0x7f66180ffe70 2024-03-30T04:24:46.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:46.419+0000 7f661da00700 1 -- 172.21.15.123:0/3341310608 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66181110d0 con 0x7f66180ffe70 2024-03-30T04:24:46.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:46.419+0000 7f660ea00700 1 -- 172.21.15.123:0/3341310608 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6604012600 con 0x7f66180ffe70 2024-03-30T04:24:46.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:46.419+0000 7f661da00700 1 -- 172.21.15.123:0/3341310608 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6618111610 con 0x7f66180ffe70 2024-03-30T04:24:46.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:46.420+0000 7f661da00700 1 -- 172.21.15.123:0/3341310608 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66180f8ef0 con 0x7f66180ffe70 2024-03-30T04:24:46.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:46.421+0000 7f660ea00700 1 -- 172.21.15.123:0/3341310608 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f660400b9f0 con 0x7f66180ffe70 2024-03-30T04:24:46.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:46.422+0000 7f660ea00700 1 --2- 172.21.15.123:0/3341310608 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f65fc0692b0 0x7f65fc06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:24:46.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:46.422+0000 7f660ea00700 1 -- 172.21.15.123:0/3341310608 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f6604087d60 con 0x7f66180ffe70 2024-03-30T04:24:46.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:46.422+0000 7f6616a00700 1 --2- 172.21.15.123:0/3341310608 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f65fc0692b0 0x7f65fc06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:24:46.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:46.423+0000 7f6616a00700 1 --2- 172.21.15.123:0/3341310608 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f65fc0692b0 0x7f65fc06b760 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6608004600 tx=0x7f660800b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:24:46.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:46.426+0000 7f660ea00700 1 -- 172.21.15.123:0/3341310608 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f6604057c70 con 0x7f66180ffe70 2024-03-30T04:24:46.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:46.682+0000 7f661da00700 1 -- 172.21.15.123:0/3341310608 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"} v 0) v1 -- 0x7f6618105890 con 0x7f66180ffe70 2024-03-30T04:24:47.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:47.074+0000 7f660ea00700 1 -- 172.21.15.123:0/3341310608 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]=0 inline data disabled v13) v1 ==== 133+0+0 (secure 0 0 0) 0x7f6618105890 con 0x7f66180ffe70 2024-03-30T04:24:47.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:47.077+0000 7f661da00700 1 -- 172.21.15.123:0/3341310608 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f65fc0692b0 msgr2=0x7f65fc06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:47.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:47.077+0000 7f661da00700 1 --2- 172.21.15.123:0/3341310608 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f65fc0692b0 0x7f65fc06b760 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6608004600 tx=0x7f660800b040 comp rx=0 tx=0).stop 2024-03-30T04:24:47.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:47.077+0000 7f661da00700 1 -- 172.21.15.123:0/3341310608 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66180ffe70 msgr2=0x7f6618110000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:47.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:47.077+0000 7f661da00700 1 --2- 172.21.15.123:0/3341310608 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66180ffe70 0x7f6618110000 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f660400b0d0 tx=0x7f6604009e30 comp rx=0 tx=0).stop 2024-03-30T04:24:47.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:47.077+0000 7f661da00700 1 -- 172.21.15.123:0/3341310608 shutdown_connections 2024-03-30T04:24:47.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:47.077+0000 7f661da00700 1 --2- 172.21.15.123:0/3341310608 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f65fc0692b0 0x7f65fc06b760 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:47.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:47.077+0000 7f661da00700 1 --2- 172.21.15.123:0/3341310608 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f66180662e0 0x7f661810fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:47.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:47.077+0000 7f661da00700 1 --2- 172.21.15.123:0/3341310608 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66180ffe70 0x7f6618110000 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:47.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:47.077+0000 7f661da00700 1 -- 172.21.15.123:0/3341310608 >> 172.21.15.123:0/3341310608 conn(0x7f66180f73c0 msgr2=0x7f66180fbad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:24:47.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:47.077+0000 7f661da00700 1 -- 172.21.15.123:0/3341310608 shutdown_connections 2024-03-30T04:24:47.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:47.077+0000 7f661da00700 1 -- 172.21.15.123:0/3341310608 wait complete. 2024-03-30T04:24:47.077 INFO:teuthology.orchestra.run.smithi123.stderr:inline data disabled 2024-03-30T04:24:47.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:46 smithi123 ceph-mon[35245]: mds.? [v2:172.21.15.135:6826/1096986122,v1:172.21.15.135:6827/1096986122] up:boot 2024-03-30T04:24:47.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:46 smithi123 ceph-mon[35245]: fsmap cephfs:2 {0=cephfs.smithi123.tqxvzr=up:active,1=cephfs.smithi135.zvnlap=up:active} 1 up:standby-replay 1 up:standby 2024-03-30T04:24:47.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:46 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi135.xwgjvb"}]: dispatch 2024-03-30T04:24:47.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:46 smithi123 ceph-mon[35245]: fsmap cephfs:2 {0=cephfs.smithi123.tqxvzr=up:active,1=cephfs.smithi135.zvnlap=up:active} 2 up:standby-replay 2024-03-30T04:24:47.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:46 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:47.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:46 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:47.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:46 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:47.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:46 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:47.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:46 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:47.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:46 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:24:47.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:46 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:47.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:46 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/3341310608' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]: dispatch 2024-03-30T04:24:47.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:46 smithi135 ceph-mon[44364]: mds.? [v2:172.21.15.135:6826/1096986122,v1:172.21.15.135:6827/1096986122] up:boot 2024-03-30T04:24:47.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:46 smithi135 ceph-mon[44364]: fsmap cephfs:2 {0=cephfs.smithi123.tqxvzr=up:active,1=cephfs.smithi135.zvnlap=up:active} 1 up:standby-replay 1 up:standby 2024-03-30T04:24:47.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:46 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi135.xwgjvb"}]: dispatch 2024-03-30T04:24:47.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:46 smithi135 ceph-mon[44364]: fsmap cephfs:2 {0=cephfs.smithi123.tqxvzr=up:active,1=cephfs.smithi135.zvnlap=up:active} 2 up:standby-replay 2024-03-30T04:24:47.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:46 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:47.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:46 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:47.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:46 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:47.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:46 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:47.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:46 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:47.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:46 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:24:47.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:46 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:47.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:46 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/3341310608' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]: dispatch 2024-03-30T04:24:47.958 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-30T04:24:47.968 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi123.front.sepia.ceph.com 2024-03-30T04:24:47.968 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- bash -c 'ceph fs dump' 2024-03-30T04:24:48.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:48 smithi123 ceph-mon[35245]: pgmap v208: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-03-30T04:24:48.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:48 smithi123 ceph-mon[35245]: mds.? [v2:172.21.15.135:6824/4050541062,v1:172.21.15.135:6825/4050541062] up:active 2024-03-30T04:24:48.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:48 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/3341310608' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]': finished 2024-03-30T04:24:48.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:48 smithi123 ceph-mon[35245]: mds.? [v2:172.21.15.123:6828/3037849622,v1:172.21.15.123:6829/3037849622] up:standby-replay 2024-03-30T04:24:48.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:48 smithi123 ceph-mon[35245]: Dropping low affinity active daemon mds.cephfs.smithi123.tqxvzr in favor of higher affinity standby. 2024-03-30T04:24:48.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:48 smithi123 ceph-mon[35245]: Replacing daemon mds.cephfs.smithi123.tqxvzr as rank 0 with standby daemon mds.cephfs.smithi123.dvtind 2024-03-30T04:24:48.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:48 smithi123 ceph-mon[35245]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-30T04:24:48.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:48 smithi123 ceph-mon[35245]: fsmap cephfs:2 {0=cephfs.smithi123.tqxvzr=up:active,1=cephfs.smithi135.zvnlap=up:active} 2 up:standby-replay 2024-03-30T04:24:48.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:48 smithi123 ceph-mon[35245]: osdmap e46: 6 total, 6 up, 6 in 2024-03-30T04:24:48.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:48 smithi123 ceph-mon[35245]: fsmap cephfs:2/2 {0=cephfs.smithi123.dvtind=up:replay,1=cephfs.smithi135.zvnlap=up:active} 1 up:standby-replay 2024-03-30T04:24:48.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:48 smithi135 ceph-mon[44364]: pgmap v208: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-03-30T04:24:48.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:48 smithi135 ceph-mon[44364]: mds.? [v2:172.21.15.135:6824/4050541062,v1:172.21.15.135:6825/4050541062] up:active 2024-03-30T04:24:48.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:48 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/3341310608' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]': finished 2024-03-30T04:24:48.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:48 smithi135 ceph-mon[44364]: mds.? [v2:172.21.15.123:6828/3037849622,v1:172.21.15.123:6829/3037849622] up:standby-replay 2024-03-30T04:24:48.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:48 smithi135 ceph-mon[44364]: Dropping low affinity active daemon mds.cephfs.smithi123.tqxvzr in favor of higher affinity standby. 2024-03-30T04:24:48.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:48 smithi135 ceph-mon[44364]: Replacing daemon mds.cephfs.smithi123.tqxvzr as rank 0 with standby daemon mds.cephfs.smithi123.dvtind 2024-03-30T04:24:48.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:48 smithi135 ceph-mon[44364]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-30T04:24:48.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:48 smithi135 ceph-mon[44364]: fsmap cephfs:2 {0=cephfs.smithi123.tqxvzr=up:active,1=cephfs.smithi135.zvnlap=up:active} 2 up:standby-replay 2024-03-30T04:24:48.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:48 smithi135 ceph-mon[44364]: osdmap e46: 6 total, 6 up, 6 in 2024-03-30T04:24:48.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:48 smithi135 ceph-mon[44364]: fsmap cephfs:2/2 {0=cephfs.smithi123.dvtind=up:replay,1=cephfs.smithi135.zvnlap=up:active} 1 up:standby-replay 2024-03-30T04:24:48.546 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:24:49.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:49 smithi135 ceph-mon[44364]: mds.? [v2:172.21.15.123:6826/2433675995,v1:172.21.15.123:6827/2433675995] up:boot 2024-03-30T04:24:49.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:49 smithi135 ceph-mon[44364]: mds.? [v2:172.21.15.123:6828/3037849622,v1:172.21.15.123:6829/3037849622] up:resolve 2024-03-30T04:24:49.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:49 smithi135 ceph-mon[44364]: fsmap cephfs:2/2 {0=cephfs.smithi123.dvtind=up:resolve,1=cephfs.smithi135.zvnlap=up:active} 1 up:standby-replay 1 up:standby 2024-03-30T04:24:49.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:49 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi123.tqxvzr"}]: dispatch 2024-03-30T04:24:49.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:49 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:49.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:49 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:49.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:49 smithi123 ceph-mon[35245]: mds.? [v2:172.21.15.123:6826/2433675995,v1:172.21.15.123:6827/2433675995] up:boot 2024-03-30T04:24:49.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:49 smithi123 ceph-mon[35245]: mds.? [v2:172.21.15.123:6828/3037849622,v1:172.21.15.123:6829/3037849622] up:resolve 2024-03-30T04:24:49.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:49 smithi123 ceph-mon[35245]: fsmap cephfs:2/2 {0=cephfs.smithi123.dvtind=up:resolve,1=cephfs.smithi135.zvnlap=up:active} 1 up:standby-replay 1 up:standby 2024-03-30T04:24:49.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:49 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi123.tqxvzr"}]: dispatch 2024-03-30T04:24:49.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:49 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:49.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:49 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:50.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:50 smithi135 ceph-mon[44364]: pgmap v210: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-03-30T04:24:50.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:50 smithi135 ceph-mon[44364]: mds.? [v2:172.21.15.123:6828/3037849622,v1:172.21.15.123:6829/3037849622] up:reconnect 2024-03-30T04:24:50.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:50 smithi135 ceph-mon[44364]: fsmap cephfs:2/2 {0=cephfs.smithi123.dvtind=up:reconnect,1=cephfs.smithi135.zvnlap=up:active} 1 up:standby-replay 1 up:standby 2024-03-30T04:24:50.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:50 smithi123 ceph-mon[35245]: pgmap v210: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-03-30T04:24:50.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:50 smithi123 ceph-mon[35245]: mds.? [v2:172.21.15.123:6828/3037849622,v1:172.21.15.123:6829/3037849622] up:reconnect 2024-03-30T04:24:50.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:50 smithi123 ceph-mon[35245]: fsmap cephfs:2/2 {0=cephfs.smithi123.dvtind=up:reconnect,1=cephfs.smithi135.zvnlap=up:active} 1 up:standby-replay 1 up:standby 2024-03-30T04:24:51.104 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:51.105+0000 7f604fe00700 1 -- 172.21.15.123:0/3723163600 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60480fcee0 msgr2=0x7f60480fd310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:51.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:51.105+0000 7f604fe00700 1 --2- 172.21.15.123:0/3723163600 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60480fcee0 0x7f60480fd310 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f603c004940 tx=0x7f603c009e30 comp rx=0 tx=0).stop 2024-03-30T04:24:51.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:51.105+0000 7f604fe00700 1 -- 172.21.15.123:0/3723163600 shutdown_connections 2024-03-30T04:24:51.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:51.105+0000 7f604fe00700 1 --2- 172.21.15.123:0/3723163600 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f60480fe0d0 0x7f60480fe540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:51.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:51.105+0000 7f604fe00700 1 --2- 172.21.15.123:0/3723163600 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60480fcee0 0x7f60480fd310 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:51.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:51.105+0000 7f604fe00700 1 -- 172.21.15.123:0/3723163600 >> 172.21.15.123:0/3723163600 conn(0x7f60480f8560 msgr2=0x7f60480fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:24:51.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:51.106+0000 7f604fe00700 1 -- 172.21.15.123:0/3723163600 shutdown_connections 2024-03-30T04:24:51.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:51.106+0000 7f604fe00700 1 -- 172.21.15.123:0/3723163600 wait complete. 2024-03-30T04:24:51.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:51.107+0000 7f604fe00700 1 Processor -- start 2024-03-30T04:24:51.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:51.107+0000 7f604fe00700 1 -- start start 2024-03-30T04:24:51.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:51.107+0000 7f604fe00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f60480fcee0 0x7f604810fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:24:51.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:51.107+0000 7f604fe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60480fe0d0 0x7f60481100a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:24:51.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:51.107+0000 7f604fe00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60481106a0 con 0x7f60480fe0d0 2024-03-30T04:24:51.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:51.107+0000 7f604fe00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6048110ef0 con 0x7f60480fcee0 2024-03-30T04:24:51.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:51.108+0000 7f604d400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f60480fcee0 0x7f604810fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:24:51.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:51.108+0000 7f604ca00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60480fe0d0 0x7f60481100a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:24:51.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:51.108+0000 7f604d400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f60480fcee0 0x7f604810fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.123:60728/0 (socket says 172.21.15.123:60728) 2024-03-30T04:24:51.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:51.108+0000 7f604d400700 1 -- 172.21.15.123:0/2783147081 learned_addr learned my addr 172.21.15.123:0/2783147081 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:24:51.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:51.108+0000 7f604ca00700 1 -- 172.21.15.123:0/2783147081 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f60480fcee0 msgr2=0x7f604810fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:51.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:51.108+0000 7f604ca00700 1 --2- 172.21.15.123:0/2783147081 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f60480fcee0 0x7f604810fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:51.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:51.108+0000 7f604ca00700 1 -- 172.21.15.123:0/2783147081 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f603c005040 con 0x7f60480fe0d0 2024-03-30T04:24:51.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:51.108+0000 7f604d400700 1 --2- 172.21.15.123:0/2783147081 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f60480fcee0 0x7f604810fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:24:51.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:51.109+0000 7f604ca00700 1 --2- 172.21.15.123:0/2783147081 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60480fe0d0 0x7f60481100a0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6034009fd0 tx=0x7f603400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:24:51.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:51.109+0000 7f6045600700 1 -- 172.21.15.123:0/2783147081 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6034004e90 con 0x7f60480fe0d0 2024-03-30T04:24:51.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:51.109+0000 7f6045600700 1 -- 172.21.15.123:0/2783147081 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f603400f040 con 0x7f60480fe0d0 2024-03-30T04:24:51.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:51.109+0000 7f604fe00700 1 -- 172.21.15.123:0/2783147081 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6048111170 con 0x7f60480fe0d0 2024-03-30T04:24:51.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:51.109+0000 7f6045600700 1 -- 172.21.15.123:0/2783147081 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f603400c9b0 con 0x7f60480fe0d0 2024-03-30T04:24:51.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:51.109+0000 7f604fe00700 1 -- 172.21.15.123:0/2783147081 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60481116b0 con 0x7f60480fe0d0 2024-03-30T04:24:51.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:51.110+0000 7f604fe00700 1 -- 172.21.15.123:0/2783147081 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f604805f5f0 con 0x7f60480fe0d0 2024-03-30T04:24:51.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:51.111+0000 7f6045600700 1 -- 172.21.15.123:0/2783147081 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f6034007cc0 con 0x7f60480fe0d0 2024-03-30T04:24:51.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:51.113+0000 7f6045600700 1 --2- 172.21.15.123:0/2783147081 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f60300692b0 0x7f603006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:24:51.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:51.113+0000 7f6045600700 1 -- 172.21.15.123:0/2783147081 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f6034086c40 con 0x7f60480fe0d0 2024-03-30T04:24:51.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:51.113+0000 7f604d400700 1 --2- 172.21.15.123:0/2783147081 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f60300692b0 0x7f603006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:24:51.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:51.114+0000 7f604d400700 1 --2- 172.21.15.123:0/2783147081 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f60300692b0 0x7f603006b760 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f603c00b0d0 tx=0x7f603c009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:24:51.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:51.118+0000 7f6045600700 1 -- 172.21.15.123:0/2783147081 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f603404e190 con 0x7f60480fe0d0 2024-03-30T04:24:51.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:51 smithi135 ceph-mon[44364]: mds.? [v2:172.21.15.123:6828/3037849622,v1:172.21.15.123:6829/3037849622] up:rejoin 2024-03-30T04:24:51.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:51 smithi135 ceph-mon[44364]: mds.? [v2:172.21.15.135:6826/1096986122,v1:172.21.15.135:6827/1096986122] up:standby-replay 2024-03-30T04:24:51.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:51 smithi135 ceph-mon[44364]: fsmap cephfs:2/2 {0=cephfs.smithi123.dvtind=up:rejoin,1=cephfs.smithi135.zvnlap=up:active} 1 up:standby-replay 1 up:standby 2024-03-30T04:24:51.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:51 smithi135 ceph-mon[44364]: daemon mds.cephfs.smithi123.dvtind is now active in filesystem cephfs as rank 0 2024-03-30T04:24:51.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:51.363+0000 7f604fe00700 1 -- 172.21.15.123:0/2783147081 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f6048111a70 con 0x7f60480fe0d0 2024-03-30T04:24:51.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:51.363+0000 7f6045600700 1 -- 172.21.15.123:0/2783147081 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 19 v19) v1 ==== 76+0+1838 (secure 0 0 0) 0x7f6034019740 con 0x7f60480fe0d0 2024-03-30T04:24:51.362 INFO:teuthology.orchestra.run.smithi123.stdout:e19 2024-03-30T04:24:51.362 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:24:51.362 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:24:51.362 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:24:51.363 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:24:51.363 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:24:51.363 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:24:51.363 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 19 2024-03-30T04:24:51.363 INFO:teuthology.orchestra.run.smithi123.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-30T04:24:51.363 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:24:51.363 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:24:51.100702+0000 2024-03-30T04:24:51.363 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:24:51.363 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:24:51.363 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:24:51.363 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:24:51.363 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:24:51.363 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:24:51.363 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:24:51.363 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 46 2024-03-30T04:24:51.364 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:24:51.364 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T04:24:51.364 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T04:24:51.364 INFO:teuthology.orchestra.run.smithi123.stdout:up {0=14488,1=24309} 2024-03-30T04:24:51.364 INFO:teuthology.orchestra.run.smithi123.stdout:failed 2024-03-30T04:24:51.364 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:24:51.364 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:24:51.364 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:24:51.364 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:24:51.364 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:24:51.364 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:24:51.364 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T04:24:51.364 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{0:14488} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.123:6828/3037849622,v1:172.21.15.123:6829/3037849622] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:24:51.364 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{0:14500} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/2433675995,v1:172.21.15.123:6827/2433675995] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:24:51.364 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{1:24309} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.135:6824/4050541062,v1:172.21.15.135:6825/4050541062] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:24:51.365 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{1:24319} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.135:6826/1096986122,v1:172.21.15.135:6827/1096986122] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:24:51.365 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:24:51.365 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:24:51.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:51.366+0000 7f604fe00700 1 -- 172.21.15.123:0/2783147081 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f60300692b0 msgr2=0x7f603006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:51.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:51.366+0000 7f604fe00700 1 --2- 172.21.15.123:0/2783147081 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f60300692b0 0x7f603006b760 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f603c00b0d0 tx=0x7f603c009e30 comp rx=0 tx=0).stop 2024-03-30T04:24:51.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:51.366+0000 7f604fe00700 1 -- 172.21.15.123:0/2783147081 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60480fe0d0 msgr2=0x7f60481100a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:51.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:51.366+0000 7f604fe00700 1 --2- 172.21.15.123:0/2783147081 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60480fe0d0 0x7f60481100a0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6034009fd0 tx=0x7f603400b040 comp rx=0 tx=0).stop 2024-03-30T04:24:51.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:51.366+0000 7f604fe00700 1 -- 172.21.15.123:0/2783147081 shutdown_connections 2024-03-30T04:24:51.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:51.366+0000 7f604fe00700 1 --2- 172.21.15.123:0/2783147081 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f60300692b0 0x7f603006b760 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:51.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:51.366+0000 7f604fe00700 1 --2- 172.21.15.123:0/2783147081 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f60480fcee0 0x7f604810fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:51.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:51.366+0000 7f604fe00700 1 --2- 172.21.15.123:0/2783147081 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60480fe0d0 0x7f60481100a0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:51.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:51.366+0000 7f604fe00700 1 -- 172.21.15.123:0/2783147081 >> 172.21.15.123:0/2783147081 conn(0x7f60480f8560 msgr2=0x7f60480f9cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:24:51.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:51.366+0000 7f604fe00700 1 -- 172.21.15.123:0/2783147081 shutdown_connections 2024-03-30T04:24:51.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:51.366+0000 7f604fe00700 1 -- 172.21.15.123:0/2783147081 wait complete. 2024-03-30T04:24:51.366 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 19 2024-03-30T04:24:51.386 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:51 smithi123 ceph-mon[35245]: mds.? [v2:172.21.15.123:6828/3037849622,v1:172.21.15.123:6829/3037849622] up:rejoin 2024-03-30T04:24:51.386 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:51 smithi123 ceph-mon[35245]: mds.? [v2:172.21.15.135:6826/1096986122,v1:172.21.15.135:6827/1096986122] up:standby-replay 2024-03-30T04:24:51.386 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:51 smithi123 ceph-mon[35245]: fsmap cephfs:2/2 {0=cephfs.smithi123.dvtind=up:rejoin,1=cephfs.smithi135.zvnlap=up:active} 1 up:standby-replay 1 up:standby 2024-03-30T04:24:51.387 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:51 smithi123 ceph-mon[35245]: daemon mds.cephfs.smithi123.dvtind is now active in filesystem cephfs as rank 0 2024-03-30T04:24:52.236 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2024-03-30T04:24:52.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:52 smithi135 ceph-mon[44364]: pgmap v211: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-30T04:24:52.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:52 smithi135 ceph-mon[44364]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-30T04:24:52.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:52 smithi135 ceph-mon[44364]: Cluster is now healthy 2024-03-30T04:24:52.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:52 smithi135 ceph-mon[44364]: mds.? [v2:172.21.15.123:6828/3037849622,v1:172.21.15.123:6829/3037849622] up:active 2024-03-30T04:24:52.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:52 smithi135 ceph-mon[44364]: fsmap cephfs:2 {0=cephfs.smithi123.dvtind=up:active,1=cephfs.smithi135.zvnlap=up:active} 1 up:standby-replay 1 up:standby 2024-03-30T04:24:52.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:52 smithi135 ceph-mon[44364]: fsmap cephfs:2 {0=cephfs.smithi123.dvtind=up:active,1=cephfs.smithi135.zvnlap=up:active} 2 up:standby-replay 2024-03-30T04:24:52.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:52 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/2783147081' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:24:52.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:52 smithi123 ceph-mon[35245]: pgmap v211: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-30T04:24:52.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:52 smithi123 ceph-mon[35245]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-30T04:24:52.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:52 smithi123 ceph-mon[35245]: Cluster is now healthy 2024-03-30T04:24:52.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:52 smithi123 ceph-mon[35245]: mds.? [v2:172.21.15.123:6828/3037849622,v1:172.21.15.123:6829/3037849622] up:active 2024-03-30T04:24:52.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:52 smithi123 ceph-mon[35245]: fsmap cephfs:2 {0=cephfs.smithi123.dvtind=up:active,1=cephfs.smithi135.zvnlap=up:active} 1 up:standby-replay 1 up:standby 2024-03-30T04:24:52.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:52 smithi123 ceph-mon[35245]: fsmap cephfs:2 {0=cephfs.smithi123.dvtind=up:active,1=cephfs.smithi135.zvnlap=up:active} 2 up:standby-replay 2024-03-30T04:24:52.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:52 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/2783147081' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:24:53.088 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:24:54.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:54 smithi135 ceph-mon[44364]: pgmap v212: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.9 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-30T04:24:54.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:54 smithi123 ceph-mon[35245]: pgmap v212: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.9 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-30T04:24:55.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:55.715+0000 7f2c76400700 1 -- 172.21.15.123:0/2212617093 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c70066280 msgr2=0x7f2c700ffa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:55.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:55.715+0000 7f2c76400700 1 --2- 172.21.15.123:0/2212617093 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c70066280 0x7f2c700ffa20 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f2c60004940 tx=0x7f2c60009e30 comp rx=0 tx=0).stop 2024-03-30T04:24:55.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:55.716+0000 7f2c76400700 1 -- 172.21.15.123:0/2212617093 shutdown_connections 2024-03-30T04:24:55.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:55.716+0000 7f2c76400700 1 --2- 172.21.15.123:0/2212617093 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2c700fff60 0x7f2c70102340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:55.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:55.716+0000 7f2c76400700 1 --2- 172.21.15.123:0/2212617093 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c70066280 0x7f2c700ffa20 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:55.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:55.716+0000 7f2c76400700 1 -- 172.21.15.123:0/2212617093 >> 172.21.15.123:0/2212617093 conn(0x7f2c700f7430 msgr2=0x7f2c700f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:24:55.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:55.716+0000 7f2c76400700 1 -- 172.21.15.123:0/2212617093 shutdown_connections 2024-03-30T04:24:55.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:55.716+0000 7f2c76400700 1 -- 172.21.15.123:0/2212617093 wait complete. 2024-03-30T04:24:55.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:55.716+0000 7f2c76400700 1 Processor -- start 2024-03-30T04:24:55.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:55.717+0000 7f2c76400700 1 -- start start 2024-03-30T04:24:55.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:55.717+0000 7f2c76400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c70066280 0x7f2c7010fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:24:55.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:55.717+0000 7f2c76400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2c700fff60 0x7f2c701100c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:24:55.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:55.717+0000 7f2c76400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c701106c0 con 0x7f2c70066280 2024-03-30T04:24:55.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:55.717+0000 7f2c76400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c70110f10 con 0x7f2c700fff60 2024-03-30T04:24:55.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:55.717+0000 7f2c6f400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c70066280 0x7f2c7010fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:24:55.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:55.717+0000 7f2c6f400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c70066280 0x7f2c7010fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54776/0 (socket says 172.21.15.123:54776) 2024-03-30T04:24:55.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:55.717+0000 7f2c6f400700 1 -- 172.21.15.123:0/250016423 learned_addr learned my addr 172.21.15.123:0/250016423 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:24:55.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:55.717+0000 7f2c6ea00700 1 --2- 172.21.15.123:0/250016423 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2c700fff60 0x7f2c701100c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:24:55.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:55.718+0000 7f2c6f400700 1 -- 172.21.15.123:0/250016423 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2c700fff60 msgr2=0x7f2c701100c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:55.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:55.718+0000 7f2c6f400700 1 --2- 172.21.15.123:0/250016423 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2c700fff60 0x7f2c701100c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:55.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:55.718+0000 7f2c6f400700 1 -- 172.21.15.123:0/250016423 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c60005040 con 0x7f2c70066280 2024-03-30T04:24:55.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:55.718+0000 7f2c6f400700 1 --2- 172.21.15.123:0/250016423 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c70066280 0x7f2c7010fb80 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2c60000c00 tx=0x7f2c60009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:24:55.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:55.718+0000 7f2c67400700 1 -- 172.21.15.123:0/250016423 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c6000e040 con 0x7f2c70066280 2024-03-30T04:24:55.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:55.718+0000 7f2c67400700 1 -- 172.21.15.123:0/250016423 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2c6000bc50 con 0x7f2c70066280 2024-03-30T04:24:55.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:55.718+0000 7f2c67400700 1 -- 172.21.15.123:0/250016423 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c6001c630 con 0x7f2c70066280 2024-03-30T04:24:55.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:55.718+0000 7f2c76400700 1 -- 172.21.15.123:0/250016423 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c70111190 con 0x7f2c70066280 2024-03-30T04:24:55.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:55.719+0000 7f2c76400700 1 -- 172.21.15.123:0/250016423 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c70111680 con 0x7f2c70066280 2024-03-30T04:24:55.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:55.720+0000 7f2c76400700 1 -- 172.21.15.123:0/250016423 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c7005f5f0 con 0x7f2c70066280 2024-03-30T04:24:55.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:55.721+0000 7f2c67400700 1 -- 172.21.15.123:0/250016423 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f2c60013420 con 0x7f2c70066280 2024-03-30T04:24:55.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:55.721+0000 7f2c67400700 1 --2- 172.21.15.123:0/250016423 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f2c540692a0 0x7f2c5406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:24:55.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:55.721+0000 7f2c67400700 1 -- 172.21.15.123:0/250016423 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f2c60088260 con 0x7f2c70066280 2024-03-30T04:24:55.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:55.722+0000 7f2c6ea00700 1 --2- 172.21.15.123:0/250016423 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f2c540692a0 0x7f2c5406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:24:55.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:55.722+0000 7f2c6ea00700 1 --2- 172.21.15.123:0/250016423 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f2c540692a0 0x7f2c5406b750 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c004f40 tx=0x7f2c5c00b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:24:55.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:55.727+0000 7f2c67400700 1 -- 172.21.15.123:0/250016423 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f2c60058080 con 0x7f2c70066280 2024-03-30T04:24:55.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:55.974+0000 7f2c76400700 1 -- 172.21.15.123:0/250016423 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c7004bed0 con 0x7f2c70066280 2024-03-30T04:24:55.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:55.974+0000 7f2c67400700 1 -- 172.21.15.123:0/250016423 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+4731 (secure 0 0 0) 0x7f2c6000bdc0 con 0x7f2c70066280 2024-03-30T04:24:55.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:55.977+0000 7f2c76400700 1 -- 172.21.15.123:0/250016423 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f2c540692a0 msgr2=0x7f2c5406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:55.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:55.977+0000 7f2c76400700 1 --2- 172.21.15.123:0/250016423 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f2c540692a0 0x7f2c5406b750 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c004f40 tx=0x7f2c5c00b040 comp rx=0 tx=0).stop 2024-03-30T04:24:55.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:55.977+0000 7f2c76400700 1 -- 172.21.15.123:0/250016423 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c70066280 msgr2=0x7f2c7010fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:24:55.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:55.977+0000 7f2c76400700 1 --2- 172.21.15.123:0/250016423 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c70066280 0x7f2c7010fb80 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2c60000c00 tx=0x7f2c60009e30 comp rx=0 tx=0).stop 2024-03-30T04:24:55.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:55.977+0000 7f2c76400700 1 -- 172.21.15.123:0/250016423 shutdown_connections 2024-03-30T04:24:55.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:55.977+0000 7f2c76400700 1 --2- 172.21.15.123:0/250016423 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f2c540692a0 0x7f2c5406b750 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:55.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:55.977+0000 7f2c76400700 1 --2- 172.21.15.123:0/250016423 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c70066280 0x7f2c7010fb80 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:55.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:55.977+0000 7f2c76400700 1 --2- 172.21.15.123:0/250016423 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2c700fff60 0x7f2c701100c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:24:55.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:55.977+0000 7f2c76400700 1 -- 172.21.15.123:0/250016423 >> 172.21.15.123:0/250016423 conn(0x7f2c700f7430 msgr2=0x7f2c700fbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:24:55.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:55.977+0000 7f2c76400700 1 -- 172.21.15.123:0/250016423 shutdown_connections 2024-03-30T04:24:55.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:24:55.977+0000 7f2c76400700 1 -- 172.21.15.123:0/250016423 wait complete. 2024-03-30T04:24:55.977 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 19 2024-03-30T04:24:55.991 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:24:56.343 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:56 smithi123 ceph-mon[35245]: pgmap v213: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.0 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-30T04:24:56.343 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:56 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:56.343 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:56 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:56.343 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:56 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:24:56.343 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:56 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:24:56.343 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:56 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:56.343 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:56 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:24:56.343 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:56 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/250016423' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:24:56.354 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:56 smithi135 ceph-mon[44364]: pgmap v213: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.0 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-30T04:24:56.354 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:56 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:56.354 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:56 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:56.355 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:56 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:24:56.355 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:56 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:24:56.355 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:56 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:56.355 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:56 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:24:56.355 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:56 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/250016423' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:24:56.802 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2024-03-30T04:24:57.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:57 smithi135 ceph-mon[44364]: pgmap v214: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 7.1 KiB/s rd, 0 B/s wr, 8 op/s 2024-03-30T04:24:57.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:57 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:57.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:57 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:24:57.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:57 smithi123 ceph-mon[35245]: pgmap v214: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 7.1 KiB/s rd, 0 B/s wr, 8 op/s 2024-03-30T04:24:57.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:57 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:57.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:57 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:24:57.763 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:24:59.864 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:59 smithi123 ceph-mon[35245]: pgmap v215: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.3 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-30T04:24:59.865 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:59 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:24:59.865 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:24:59 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:25:00.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:59 smithi135 ceph-mon[44364]: pgmap v215: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.3 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-30T04:25:00.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:59 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:25:00.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:24:59 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:25:00.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:00.384+0000 7f7b4e200700 1 -- 172.21.15.123:0/2208761108 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b480fff00 msgr2=0x7f7b48102320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:25:00.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:00.384+0000 7f7b4e200700 1 --2- 172.21.15.123:0/2208761108 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b480fff00 0x7f7b48102320 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7b34004940 tx=0x7f7b34009e30 comp rx=0 tx=0).stop 2024-03-30T04:25:00.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:00.384+0000 7f7b4e200700 1 -- 172.21.15.123:0/2208761108 shutdown_connections 2024-03-30T04:25:00.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:00.384+0000 7f7b4e200700 1 --2- 172.21.15.123:0/2208761108 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b480fff00 0x7f7b48102320 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:00.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:00.384+0000 7f7b4e200700 1 --2- 172.21.15.123:0/2208761108 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b480662e0 0x7f7b480ff9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:00.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:00.384+0000 7f7b4e200700 1 -- 172.21.15.123:0/2208761108 >> 172.21.15.123:0/2208761108 conn(0x7f7b480f7430 msgr2=0x7f7b480f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:25:00.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:00.384+0000 7f7b4e200700 1 -- 172.21.15.123:0/2208761108 shutdown_connections 2024-03-30T04:25:00.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:00.384+0000 7f7b4e200700 1 -- 172.21.15.123:0/2208761108 wait complete. 2024-03-30T04:25:00.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:00.385+0000 7f7b4e200700 1 Processor -- start 2024-03-30T04:25:00.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:00.385+0000 7f7b4e200700 1 -- start start 2024-03-30T04:25:00.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:00.385+0000 7f7b4e200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b480662e0 0x7f7b4810fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:25:00.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:00.385+0000 7f7b4e200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b480fff00 0x7f7b48110110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:25:00.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:00.385+0000 7f7b4e200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b48110710 con 0x7f7b480662e0 2024-03-30T04:25:00.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:00.385+0000 7f7b4e200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b48110f10 con 0x7f7b480fff00 2024-03-30T04:25:00.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:00.386+0000 7f7b47400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b480662e0 0x7f7b4810fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:25:00.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:00.386+0000 7f7b47400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b480662e0 0x7f7b4810fbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54800/0 (socket says 172.21.15.123:54800) 2024-03-30T04:25:00.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:00.386+0000 7f7b47400700 1 -- 172.21.15.123:0/800928353 learned_addr learned my addr 172.21.15.123:0/800928353 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:25:00.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:00.386+0000 7f7b46a00700 1 --2- 172.21.15.123:0/800928353 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b480fff00 0x7f7b48110110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:25:00.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:00.386+0000 7f7b47400700 1 -- 172.21.15.123:0/800928353 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b480fff00 msgr2=0x7f7b48110110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:25:00.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:00.386+0000 7f7b47400700 1 --2- 172.21.15.123:0/800928353 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b480fff00 0x7f7b48110110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:00.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:00.386+0000 7f7b47400700 1 -- 172.21.15.123:0/800928353 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b34005040 con 0x7f7b480662e0 2024-03-30T04:25:00.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:00.386+0000 7f7b47400700 1 --2- 172.21.15.123:0/800928353 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b480662e0 0x7f7b4810fbd0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f7b3800c800 tx=0x7f7b38004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:25:00.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:00.387+0000 7f7b3f400700 1 -- 172.21.15.123:0/800928353 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b380106f0 con 0x7f7b480662e0 2024-03-30T04:25:00.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:00.387+0000 7f7b3f400700 1 -- 172.21.15.123:0/800928353 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7b38010d00 con 0x7f7b480662e0 2024-03-30T04:25:00.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:00.387+0000 7f7b4e200700 1 -- 172.21.15.123:0/800928353 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b481111f0 con 0x7f7b480662e0 2024-03-30T04:25:00.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:00.387+0000 7f7b3f400700 1 -- 172.21.15.123:0/800928353 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b380113a0 con 0x7f7b480662e0 2024-03-30T04:25:00.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:00.387+0000 7f7b4e200700 1 -- 172.21.15.123:0/800928353 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b48111730 con 0x7f7b480662e0 2024-03-30T04:25:00.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:00.388+0000 7f7b4e200700 1 -- 172.21.15.123:0/800928353 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b480f8fb0 con 0x7f7b480662e0 2024-03-30T04:25:00.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:00.389+0000 7f7b3f400700 1 -- 172.21.15.123:0/800928353 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f7b38012400 con 0x7f7b480662e0 2024-03-30T04:25:00.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:00.389+0000 7f7b3f400700 1 --2- 172.21.15.123:0/800928353 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f7b2c0692b0 0x7f7b2c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:25:00.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:00.390+0000 7f7b3f400700 1 -- 172.21.15.123:0/800928353 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f7b38087160 con 0x7f7b480662e0 2024-03-30T04:25:00.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:00.390+0000 7f7b46a00700 1 --2- 172.21.15.123:0/800928353 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f7b2c0692b0 0x7f7b2c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:25:00.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:00.390+0000 7f7b46a00700 1 --2- 172.21.15.123:0/800928353 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f7b2c0692b0 0x7f7b2c06b760 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7b34004f90 tx=0x7f7b34009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:25:00.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:00.394+0000 7f7b3f400700 1 -- 172.21.15.123:0/800928353 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f7b38056eb0 con 0x7f7b480662e0 2024-03-30T04:25:00.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:00.672+0000 7f7b4e200700 1 -- 172.21.15.123:0/800928353 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7f7b481058b0 con 0x7f7b480662e0 2024-03-30T04:25:00.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:00.673+0000 7f7b3f400700 1 -- 172.21.15.123:0/800928353 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v19) v1 ==== 78+0+99 (secure 0 0 0) 0x7f7b38056a40 con 0x7f7b480662e0 2024-03-30T04:25:00.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:00.675+0000 7f7b4e200700 1 -- 172.21.15.123:0/800928353 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f7b2c0692b0 msgr2=0x7f7b2c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:25:00.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:00.675+0000 7f7b4e200700 1 --2- 172.21.15.123:0/800928353 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f7b2c0692b0 0x7f7b2c06b760 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7b34004f90 tx=0x7f7b34009e30 comp rx=0 tx=0).stop 2024-03-30T04:25:00.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:00.675+0000 7f7b4e200700 1 -- 172.21.15.123:0/800928353 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b480662e0 msgr2=0x7f7b4810fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:25:00.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:00.675+0000 7f7b4e200700 1 --2- 172.21.15.123:0/800928353 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b480662e0 0x7f7b4810fbd0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f7b3800c800 tx=0x7f7b38004470 comp rx=0 tx=0).stop 2024-03-30T04:25:00.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:00.675+0000 7f7b4e200700 1 -- 172.21.15.123:0/800928353 shutdown_connections 2024-03-30T04:25:00.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:00.675+0000 7f7b4e200700 1 --2- 172.21.15.123:0/800928353 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f7b2c0692b0 0x7f7b2c06b760 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:00.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:00.675+0000 7f7b4e200700 1 --2- 172.21.15.123:0/800928353 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b480662e0 0x7f7b4810fbd0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:00.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:00.675+0000 7f7b4e200700 1 --2- 172.21.15.123:0/800928353 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b480fff00 0x7f7b48110110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:00.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:00.675+0000 7f7b4e200700 1 -- 172.21.15.123:0/800928353 >> 172.21.15.123:0/800928353 conn(0x7f7b480f7430 msgr2=0x7f7b480fc030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:25:00.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:00.675+0000 7f7b4e200700 1 -- 172.21.15.123:0/800928353 shutdown_connections 2024-03-30T04:25:00.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:00.676+0000 7f7b4e200700 1 -- 172.21.15.123:0/800928353 wait complete. 2024-03-30T04:25:00.690 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:25:01.354 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2024-03-30T04:25:01.367 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T04:25:01.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:01.638+0000 7f3279200700 1 -- 172.21.15.123:0/19259740 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32740fdf80 msgr2=0x7f32740fe3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:25:01.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:01.638+0000 7f3279200700 1 --2- 172.21.15.123:0/19259740 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32740fdf80 0x7f32740fe3f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3260004940 tx=0x7f3260009e30 comp rx=0 tx=0).stop 2024-03-30T04:25:01.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:01.638+0000 7f3279200700 1 -- 172.21.15.123:0/19259740 shutdown_connections 2024-03-30T04:25:01.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:01.638+0000 7f3279200700 1 --2- 172.21.15.123:0/19259740 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32740fdf80 0x7f32740fe3f0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:01.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:01.638+0000 7f3279200700 1 --2- 172.21.15.123:0/19259740 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f32740fccd0 0x7f32740fd0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:01.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:01.638+0000 7f3279200700 1 -- 172.21.15.123:0/19259740 >> 172.21.15.123:0/19259740 conn(0x7f32740f8430 msgr2=0x7f32740fa860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:25:01.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:01.638+0000 7f3279200700 1 -- 172.21.15.123:0/19259740 shutdown_connections 2024-03-30T04:25:01.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:01.638+0000 7f3279200700 1 -- 172.21.15.123:0/19259740 wait complete. 2024-03-30T04:25:01.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:01.639+0000 7f3279200700 1 Processor -- start 2024-03-30T04:25:01.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:01.639+0000 7f3279200700 1 -- start start 2024-03-30T04:25:01.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:01.640+0000 7f3279200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32740fccd0 0x7f327410f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:25:01.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:01.640+0000 7f3279200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f32740fdf80 0x7f327410fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:25:01.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:01.640+0000 7f3279200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32741104e0 con 0x7f32740fccd0 2024-03-30T04:25:01.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:01.640+0000 7f3279200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3274110650 con 0x7f32740fdf80 2024-03-30T04:25:01.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:01.640+0000 7f3271e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32740fccd0 0x7f327410f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:25:01.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:01.640+0000 7f3271e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32740fccd0 0x7f327410f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54824/0 (socket says 172.21.15.123:54824) 2024-03-30T04:25:01.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:01.640+0000 7f3271e00700 1 -- 172.21.15.123:0/1008497111 learned_addr learned my addr 172.21.15.123:0/1008497111 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:25:01.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:01.640+0000 7f3271400700 1 --2- 172.21.15.123:0/1008497111 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f32740fdf80 0x7f327410fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:25:01.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:01.640+0000 7f3271e00700 1 -- 172.21.15.123:0/1008497111 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f32740fdf80 msgr2=0x7f327410fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:25:01.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:01.640+0000 7f3271e00700 1 --2- 172.21.15.123:0/1008497111 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f32740fdf80 0x7f327410fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:01.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:01.640+0000 7f3271e00700 1 -- 172.21.15.123:0/1008497111 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3260005040 con 0x7f32740fccd0 2024-03-30T04:25:01.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:01.641+0000 7f3271e00700 1 --2- 172.21.15.123:0/1008497111 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32740fccd0 0x7f327410f9a0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f3264002a50 tx=0x7f32640046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:25:01.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:01.641+0000 7f326a000700 1 -- 172.21.15.123:0/1008497111 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3264007cc0 con 0x7f32740fccd0 2024-03-30T04:25:01.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:01.641+0000 7f326a000700 1 -- 172.21.15.123:0/1008497111 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f326400f040 con 0x7f32740fccd0 2024-03-30T04:25:01.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:01.641+0000 7f3279200700 1 -- 172.21.15.123:0/1008497111 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32741150b0 con 0x7f32740fccd0 2024-03-30T04:25:01.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:01.641+0000 7f326a000700 1 -- 172.21.15.123:0/1008497111 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f326400d9c0 con 0x7f32740fccd0 2024-03-30T04:25:01.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:01.641+0000 7f3279200700 1 -- 172.21.15.123:0/1008497111 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3274115670 con 0x7f32740fccd0 2024-03-30T04:25:01.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:01.642+0000 7f3279200700 1 -- 172.21.15.123:0/1008497111 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3274101910 con 0x7f32740fccd0 2024-03-30T04:25:01.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:01.643+0000 7f326a000700 1 -- 172.21.15.123:0/1008497111 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f326400db20 con 0x7f32740fccd0 2024-03-30T04:25:01.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:01.644+0000 7f326a000700 1 --2- 172.21.15.123:0/1008497111 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f32580692a0 0x7f325806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:25:01.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:01.644+0000 7f326a000700 1 -- 172.21.15.123:0/1008497111 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f3264086e30 con 0x7f32740fccd0 2024-03-30T04:25:01.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:01.644+0000 7f3271400700 1 --2- 172.21.15.123:0/1008497111 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f32580692a0 0x7f325806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:25:01.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:01.644+0000 7f3271400700 1 --2- 172.21.15.123:0/1008497111 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f32580692a0 0x7f325806b750 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3260009fd0 tx=0x7f3260009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:25:01.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:01.648+0000 7f326a000700 1 -- 172.21.15.123:0/1008497111 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f3264056ba0 con 0x7f32740fccd0 2024-03-30T04:25:01.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:01.860+0000 7f3279200700 1 -- 172.21.15.123:0/1008497111 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3274064970 con 0x7f32740fccd0 2024-03-30T04:25:01.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:01.860+0000 7f326a000700 1 -- 172.21.15.123:0/1008497111 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+14458 (secure 0 0 0) 0x7f3264056730 con 0x7f32740fccd0 2024-03-30T04:25:01.859 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:25:01.859 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":46,"fsid":"2e77a2be-ee4c-11ee-b647-cb9ed24678a4","created":"2024-03-30T04:16:30.949442+0000","modified":"2024-03-30T04:24:47.075777+0000","last_up_change":"2024-03-30T04:23:06.203543+0000","last_in_change":"2024-03-30T04:22:44.504478+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T04:21:34.842970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-03-30T04:24:29.050323+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-03-30T04:24:29.704111+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3968446b-8696-4eec-b640-c97973ca054d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":1768008525},{"type":"v1","addr":"172.21.15.123:6803","nonce":1768008525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":1768008525},{"type":"v1","addr":"172.21.15.123:6805","nonce":1768008525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1768008525},{"type":"v1","addr":"172.21.15.123:6809","nonce":1768008525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":1768008525},{"type":"v1","addr":"172.21.15.123:6807","nonce":1768008525}]},"public_addr":"172.21.15.123:6803/1768008525","cluster_addr":"172.21.15.123:6805/1768008525","heartbeat_back_addr":"172.21.15.123:6809/1768008525","heartbeat_front_addr":"172.21.15.123:6807/1768008525","state":["exists","up"]},{"osd":1,"uuid":"336be34b-bb3c-46ab-8439-af04a1366db5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1040012134},{"type":"v1","addr":"172.21.15.123:6811","nonce":1040012134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1040012134},{"type":"v1","addr":"172.21.15.123:6813","nonce":1040012134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1040012134},{"type":"v1","addr":"172.21.15.123:6817","nonce":1040012134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1040012134},{"type":"v1","addr":"172.21.15.123:6815","nonce":1040012134}]},"public_addr":"172.21.15.123:6811/1040012134","cluster_addr":"172.21.15.123:6813/1040012134","heartbeat_back_addr":"172.21.15.123:6817/1040012134","heartbeat_front_addr":"172.21.15.123:6815/1040012134","state":["exists","up"]},{"osd":2,"uuid":"4386fc8a-e9ed-413e-bd3a-bb84a0e62827","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1260172805},{"type":"v1","addr":"172.21.15.123:6819","nonce":1260172805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1260172805},{"type":"v1","addr":"172.21.15.123:6821","nonce":1260172805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1260172805},{"type":"v1","addr":"172.21.15.123:6825","nonce":1260172805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1260172805},{"type":"v1","addr":"172.21.15.123:6823","nonce":1260172805}]},"public_addr":"172.21.15.123:6819/1260172805","cluster_addr":"172.21.15.123:6821/1260172805","heartbeat_back_addr":"172.21.15.123:6825/1260172805","heartbeat_front_addr":"172.21.15.123:6823/1260172805","state":["exists","up"]},{"osd":3,"uuid":"086d86d4-ff1a-47dd-8416-447ccf9064ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":2906534885},{"type":"v1","addr":"172.21.15.135:6801","nonce":2906534885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":2906534885},{"type":"v1","addr":"172.21.15.135:6803","nonce":2906534885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":2906534885},{"type":"v1","addr":"172.21.15.135:6807","nonce":2906534885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":2906534885},{"type":"v1","addr":"172.21.15.135:6805","nonce":2906534885}]},"public_addr":"172.21.15.135:6801/2906534885","cluster_addr":"172.21.15.135:6803/2906534885","heartbeat_back_addr":"172.21.15.135:6807/2906534885","heartbeat_front_addr":"172.21.15.135:6805/2906534885","state":["exists","up"]},{"osd":4,"uuid":"31868cec-dd4d-44dd-a150-7ebdbd61ea4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":1699881600},{"type":"v1","addr":"172.21.15.135:6809","nonce":1699881600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":1699881600},{"type":"v1","addr":"172.21.15.135:6811","nonce":1699881600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":1699881600},{"type":"v1","addr":"172.21.15.135:6815","nonce":1699881600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":1699881600},{"type":"v1","addr":"172.21.15.135:6813","nonce":1699881600}]},"public_addr":"172.21.15.135:6809/1699881600","cluster_addr":"172.21.15.135:6811/1699881600","heartbeat_back_addr":"172.21.15.135:6815/1699881600","heartbeat_front_addr":"172.21.15.135:6813/1699881600","state":["exists","up"]},{"osd":5,"uuid":"dc6e11b9-4729-4504-8917-51a3bb5e2b7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":1609344282},{"type":"v1","addr":"172.21.15.135:6817","nonce":1609344282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":1609344282},{"type":"v1","addr":"172.21.15.135:6819","nonce":1609344282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":1609344282},{"type":"v1","addr":"172.21.15.135:6823","nonce":1609344282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":1609344282},{"type":"v1","addr":"172.21.15.135:6821","nonce":1609344282}]},"public_addr":"172.21.15.135:6817/1609344282","cluster_addr":"172.21.15.135:6819/1609344282","heartbeat_back_addr":"172.21.15.135:6823/1609344282","heartbeat_front_addr":"172.21.15.135:6821/1609344282","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T04:20:25.138939+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T04:20:58.657719+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T04:21:32.397235+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T04:22:02.245820+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T04:22:33.842250+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T04:23:04.503576+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.123:6827/2393193575":"2024-03-31T04:24:47.075751+0000","172.21.15.123:0/3037148222":"2024-03-31T04:17:41.459573+0000","172.21.15.123:6826/2393193575":"2024-03-31T04:24:47.075751+0000","172.21.15.123:0/250978570":"2024-03-31T04:17:05.107043+0000","172.21.15.123:0/2358069559":"2024-03-31T04:17:41.459573+0000","172.21.15.123:0/518566709":"2024-03-31T04:18:56.235750+0000","172.21.15.123:6800/1291592055":"2024-03-31T04:17:05.107043+0000","172.21.15.123:0/2366611723":"2024-03-31T04:17:05.107043+0000","172.21.15.123:6801/1291592055":"2024-03-31T04:17:05.107043+0000","172.21.15.123:0/133647565":"2024-03-31T04:17:41.459573+0000","172.21.15.123:6800/1776532690":"2024-03-31T04:17:41.459573+0000","172.21.15.123:0/2756071515":"2024-03-31T04:17:05.107043+0000","172.21.15.123:0/1275706223":"2024-03-31T04:18:56.235750+0000","172.21.15.123:6801/1776532690":"2024-03-31T04:17:41.459573+0000","172.21.15.123:0/3593359364":"2024-03-31T04:18:56.235750+0000","172.21.15.123:6801/70490328":"2024-03-31T04:18:56.235750+0000","172.21.15.123:6800/70490328":"2024-03-31T04:18:56.235750+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T04:25:01.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:01.863+0000 7f3279200700 1 -- 172.21.15.123:0/1008497111 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f32580692a0 msgr2=0x7f325806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:25:01.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:01.863+0000 7f3279200700 1 --2- 172.21.15.123:0/1008497111 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f32580692a0 0x7f325806b750 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3260009fd0 tx=0x7f3260009e30 comp rx=0 tx=0).stop 2024-03-30T04:25:01.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:01.863+0000 7f3279200700 1 -- 172.21.15.123:0/1008497111 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32740fccd0 msgr2=0x7f327410f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:25:01.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:01.863+0000 7f3279200700 1 --2- 172.21.15.123:0/1008497111 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32740fccd0 0x7f327410f9a0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f3264002a50 tx=0x7f32640046b0 comp rx=0 tx=0).stop 2024-03-30T04:25:01.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:01.863+0000 7f3279200700 1 -- 172.21.15.123:0/1008497111 shutdown_connections 2024-03-30T04:25:01.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:01.863+0000 7f3279200700 1 --2- 172.21.15.123:0/1008497111 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f32580692a0 0x7f325806b750 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:01.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:01.863+0000 7f3279200700 1 --2- 172.21.15.123:0/1008497111 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32740fccd0 0x7f327410f9a0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:01.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:01.863+0000 7f3279200700 1 --2- 172.21.15.123:0/1008497111 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f32740fdf80 0x7f327410fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:01.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:01.863+0000 7f3279200700 1 -- 172.21.15.123:0/1008497111 >> 172.21.15.123:0/1008497111 conn(0x7f32740f8430 msgr2=0x7f32740f9eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:25:01.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:01.863+0000 7f3279200700 1 -- 172.21.15.123:0/1008497111 shutdown_connections 2024-03-30T04:25:01.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:01.863+0000 7f3279200700 1 -- 172.21.15.123:0/1008497111 wait complete. 2024-03-30T04:25:01.879 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:25:01 smithi123 ceph-mon[35245]: pgmap v216: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-30T04:25:01.879 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:25:01 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/800928353' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-03-30T04:25:01.882 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T04:21:34.842970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-03-30T04:24:29.050323+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-03-30T04:24:29.704111+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-30T04:25:01.883 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-30T04:25:02.058 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:25:01 smithi135 ceph-mon[44364]: pgmap v216: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-30T04:25:02.059 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:25:01 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/800928353' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-03-30T04:25:02.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.083+0000 7feb96c00700 1 -- 172.21.15.123:0/3034893710 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb900fccb0 msgr2=0x7feb900fd0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:25:02.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.083+0000 7feb96c00700 1 --2- 172.21.15.123:0/3034893710 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb900fccb0 0x7feb900fd0c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7feb80004940 tx=0x7feb80009e30 comp rx=0 tx=0).stop 2024-03-30T04:25:02.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.084+0000 7feb96c00700 1 -- 172.21.15.123:0/3034893710 shutdown_connections 2024-03-30T04:25:02.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.084+0000 7feb96c00700 1 --2- 172.21.15.123:0/3034893710 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feb900fdf60 0x7feb900fe3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:02.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.084+0000 7feb96c00700 1 --2- 172.21.15.123:0/3034893710 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb900fccb0 0x7feb900fd0c0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:02.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.084+0000 7feb96c00700 1 -- 172.21.15.123:0/3034893710 >> 172.21.15.123:0/3034893710 conn(0x7feb900f8430 msgr2=0x7feb900fa840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:25:02.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.084+0000 7feb96c00700 1 -- 172.21.15.123:0/3034893710 shutdown_connections 2024-03-30T04:25:02.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.084+0000 7feb96c00700 1 -- 172.21.15.123:0/3034893710 wait complete. 2024-03-30T04:25:02.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.084+0000 7feb96c00700 1 Processor -- start 2024-03-30T04:25:02.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.085+0000 7feb96c00700 1 -- start start 2024-03-30T04:25:02.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.085+0000 7feb96c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb900fccb0 0x7feb9010d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:25:02.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.085+0000 7feb96c00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feb900fdf60 0x7feb9010dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:25:02.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.085+0000 7feb96c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb9010e290 con 0x7feb900fccb0 2024-03-30T04:25:02.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.085+0000 7feb96c00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb9010e400 con 0x7feb900fdf60 2024-03-30T04:25:02.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.085+0000 7feb8fe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb900fccb0 0x7feb9010d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:25:02.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.085+0000 7feb8fe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb900fccb0 0x7feb9010d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54842/0 (socket says 172.21.15.123:54842) 2024-03-30T04:25:02.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.085+0000 7feb8fe00700 1 -- 172.21.15.123:0/607721731 learned_addr learned my addr 172.21.15.123:0/607721731 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:25:02.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.085+0000 7feb8f400700 1 --2- 172.21.15.123:0/607721731 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feb900fdf60 0x7feb9010dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:25:02.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.085+0000 7feb8fe00700 1 -- 172.21.15.123:0/607721731 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feb900fdf60 msgr2=0x7feb9010dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:25:02.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.085+0000 7feb8fe00700 1 --2- 172.21.15.123:0/607721731 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feb900fdf60 0x7feb9010dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:02.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.085+0000 7feb8fe00700 1 -- 172.21.15.123:0/607721731 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb80005040 con 0x7feb900fccb0 2024-03-30T04:25:02.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.086+0000 7feb8fe00700 1 --2- 172.21.15.123:0/607721731 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb900fccb0 0x7feb9010d750 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7feb8000b3b0 tx=0x7feb80009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:25:02.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.086+0000 7feb87e00700 1 -- 172.21.15.123:0/607721731 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb8000e040 con 0x7feb900fccb0 2024-03-30T04:25:02.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.086+0000 7feb87e00700 1 -- 172.21.15.123:0/607721731 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feb80004290 con 0x7feb900fccb0 2024-03-30T04:25:02.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.086+0000 7feb96c00700 1 -- 172.21.15.123:0/607721731 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb90112db0 con 0x7feb900fccb0 2024-03-30T04:25:02.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.086+0000 7feb87e00700 1 -- 172.21.15.123:0/607721731 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb8001c3f0 con 0x7feb900fccb0 2024-03-30T04:25:02.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.086+0000 7feb96c00700 1 -- 172.21.15.123:0/607721731 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb901132e0 con 0x7feb900fccb0 2024-03-30T04:25:02.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.087+0000 7feb96c00700 1 -- 172.21.15.123:0/607721731 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb90064970 con 0x7feb900fccb0 2024-03-30T04:25:02.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.088+0000 7feb87e00700 1 -- 172.21.15.123:0/607721731 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7feb80013420 con 0x7feb900fccb0 2024-03-30T04:25:02.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.089+0000 7feb87e00700 1 --2- 172.21.15.123:0/607721731 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7feb740692a0 0x7feb7406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:25:02.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.089+0000 7feb87e00700 1 -- 172.21.15.123:0/607721731 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7feb800884e0 con 0x7feb900fccb0 2024-03-30T04:25:02.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.089+0000 7feb8f400700 1 --2- 172.21.15.123:0/607721731 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7feb740692a0 0x7feb7406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:25:02.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.089+0000 7feb8f400700 1 --2- 172.21.15.123:0/607721731 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7feb740692a0 0x7feb7406b750 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7feb7c004f40 tx=0x7feb7c00b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:25:02.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.093+0000 7feb87e00700 1 -- 172.21.15.123:0/607721731 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7feb80058250 con 0x7feb900fccb0 2024-03-30T04:25:02.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.295+0000 7feb96c00700 1 -- 172.21.15.123:0/607721731 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7feb90113500 con 0x7feb900fccb0 2024-03-30T04:25:02.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.296+0000 7feb87e00700 1 -- 172.21.15.123:0/607721731 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v46) v1 ==== 93+0+10 (secure 0 0 0) 0x7feb80057de0 con 0x7feb900fccb0 2024-03-30T04:25:02.295 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 1 2024-03-30T04:25:02.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.299+0000 7feb96c00700 1 -- 172.21.15.123:0/607721731 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7feb740692a0 msgr2=0x7feb7406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:25:02.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.299+0000 7feb96c00700 1 --2- 172.21.15.123:0/607721731 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7feb740692a0 0x7feb7406b750 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7feb7c004f40 tx=0x7feb7c00b040 comp rx=0 tx=0).stop 2024-03-30T04:25:02.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.299+0000 7feb96c00700 1 -- 172.21.15.123:0/607721731 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb900fccb0 msgr2=0x7feb9010d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:25:02.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.299+0000 7feb96c00700 1 --2- 172.21.15.123:0/607721731 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb900fccb0 0x7feb9010d750 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7feb8000b3b0 tx=0x7feb80009e30 comp rx=0 tx=0).stop 2024-03-30T04:25:02.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.299+0000 7feb96c00700 1 -- 172.21.15.123:0/607721731 shutdown_connections 2024-03-30T04:25:02.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.299+0000 7feb96c00700 1 --2- 172.21.15.123:0/607721731 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7feb740692a0 0x7feb7406b750 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:02.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.299+0000 7feb96c00700 1 --2- 172.21.15.123:0/607721731 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb900fccb0 0x7feb9010d750 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:02.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.299+0000 7feb96c00700 1 --2- 172.21.15.123:0/607721731 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feb900fdf60 0x7feb9010dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:02.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.299+0000 7feb96c00700 1 -- 172.21.15.123:0/607721731 >> 172.21.15.123:0/607721731 conn(0x7feb900f8430 msgr2=0x7feb900f9bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:25:02.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.300+0000 7feb96c00700 1 -- 172.21.15.123:0/607721731 shutdown_connections 2024-03-30T04:25:02.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.300+0000 7feb96c00700 1 -- 172.21.15.123:0/607721731 wait complete. 2024-03-30T04:25:02.321 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-03-30T04:25:02.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.523+0000 7fc158200700 1 -- 172.21.15.123:0/3335296607 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc1500fccc0 msgr2=0x7fc1500fd0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:25:02.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.523+0000 7fc158200700 1 --2- 172.21.15.123:0/3335296607 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc1500fccc0 0x7fc1500fd0d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fc144004940 tx=0x7fc144009e30 comp rx=0 tx=0).stop 2024-03-30T04:25:02.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.524+0000 7fc158200700 1 -- 172.21.15.123:0/3335296607 shutdown_connections 2024-03-30T04:25:02.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.524+0000 7fc158200700 1 --2- 172.21.15.123:0/3335296607 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc1500fdf70 0x7fc1500fe3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:02.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.524+0000 7fc158200700 1 --2- 172.21.15.123:0/3335296607 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc1500fccc0 0x7fc1500fd0d0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:02.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.524+0000 7fc158200700 1 -- 172.21.15.123:0/3335296607 >> 172.21.15.123:0/3335296607 conn(0x7fc1500f8440 msgr2=0x7fc1500fa850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:25:02.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.524+0000 7fc158200700 1 -- 172.21.15.123:0/3335296607 shutdown_connections 2024-03-30T04:25:02.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.524+0000 7fc158200700 1 -- 172.21.15.123:0/3335296607 wait complete. 2024-03-30T04:25:02.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.524+0000 7fc158200700 1 Processor -- start 2024-03-30T04:25:02.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.525+0000 7fc158200700 1 -- start start 2024-03-30T04:25:02.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.525+0000 7fc158200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc1500fccc0 0x7fc150071c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:25:02.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.525+0000 7fc158200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc1500fdf70 0x7fc150070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:25:02.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.525+0000 7fc158200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc150072210 con 0x7fc1500fdf70 2024-03-30T04:25:02.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.525+0000 7fc158200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc150070850 con 0x7fc1500fccc0 2024-03-30T04:25:02.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.525+0000 7fc154e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc1500fdf70 0x7fc150070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:25:02.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.525+0000 7fc154e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc1500fdf70 0x7fc150070310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54862/0 (socket says 172.21.15.123:54862) 2024-03-30T04:25:02.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.525+0000 7fc154e00700 1 -- 172.21.15.123:0/30400390 learned_addr learned my addr 172.21.15.123:0/30400390 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:25:02.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.525+0000 7fc155800700 1 --2- 172.21.15.123:0/30400390 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc1500fccc0 0x7fc150071c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:25:02.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.525+0000 7fc154e00700 1 -- 172.21.15.123:0/30400390 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc1500fccc0 msgr2=0x7fc150071c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:25:02.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.525+0000 7fc154e00700 1 --2- 172.21.15.123:0/30400390 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc1500fccc0 0x7fc150071c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:02.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.525+0000 7fc154e00700 1 -- 172.21.15.123:0/30400390 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc144005040 con 0x7fc1500fdf70 2024-03-30T04:25:02.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.525+0000 7fc155800700 1 --2- 172.21.15.123:0/30400390 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc1500fccc0 0x7fc150071c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:25:02.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.526+0000 7fc154e00700 1 --2- 172.21.15.123:0/30400390 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc1500fdf70 0x7fc150070310 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fc13c00eb40 tx=0x7fc13c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:25:02.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.526+0000 7fc14d600700 1 -- 172.21.15.123:0/30400390 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc13c007cc0 con 0x7fc1500fdf70 2024-03-30T04:25:02.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.526+0000 7fc14d600700 1 -- 172.21.15.123:0/30400390 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc13c00f040 con 0x7fc1500fdf70 2024-03-30T04:25:02.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.526+0000 7fc158200700 1 -- 172.21.15.123:0/30400390 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc150070ad0 con 0x7fc1500fdf70 2024-03-30T04:25:02.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.526+0000 7fc14d600700 1 -- 172.21.15.123:0/30400390 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc13c009570 con 0x7fc1500fdf70 2024-03-30T04:25:02.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.526+0000 7fc158200700 1 -- 172.21.15.123:0/30400390 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc150071010 con 0x7fc1500fdf70 2024-03-30T04:25:02.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.527+0000 7fc158200700 1 -- 172.21.15.123:0/30400390 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc150064970 con 0x7fc1500fdf70 2024-03-30T04:25:02.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.528+0000 7fc14d600700 1 -- 172.21.15.123:0/30400390 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fc13c0096d0 con 0x7fc1500fdf70 2024-03-30T04:25:02.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.529+0000 7fc14d600700 1 --2- 172.21.15.123:0/30400390 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fc1380692a0 0x7fc13806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:25:02.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.529+0000 7fc14d600700 1 -- 172.21.15.123:0/30400390 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fc13c086de0 con 0x7fc1500fdf70 2024-03-30T04:25:02.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.529+0000 7fc155800700 1 --2- 172.21.15.123:0/30400390 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fc1380692a0 0x7fc13806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:25:02.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.530+0000 7fc155800700 1 --2- 172.21.15.123:0/30400390 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fc1380692a0 0x7fc13806b750 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc144009fd0 tx=0x7fc144009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:25:02.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.533+0000 7fc14d600700 1 -- 172.21.15.123:0/30400390 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fc13c04e190 con 0x7fc1500fdf70 2024-03-30T04:25:02.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.743+0000 7fc158200700 1 -- 172.21.15.123:0/30400390 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7fc150105670 con 0x7fc1500fdf70 2024-03-30T04:25:02.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.743+0000 7fc14d600700 1 -- 172.21.15.123:0/30400390 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v46) v1 ==== 107+0+11 (secure 0 0 0) 0x7fc13c056c00 con 0x7fc1500fdf70 2024-03-30T04:25:02.742 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 32 2024-03-30T04:25:02.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.745+0000 7fc158200700 1 -- 172.21.15.123:0/30400390 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fc1380692a0 msgr2=0x7fc13806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:25:02.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.745+0000 7fc158200700 1 --2- 172.21.15.123:0/30400390 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fc1380692a0 0x7fc13806b750 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc144009fd0 tx=0x7fc144009e30 comp rx=0 tx=0).stop 2024-03-30T04:25:02.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.745+0000 7fc158200700 1 -- 172.21.15.123:0/30400390 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc1500fdf70 msgr2=0x7fc150070310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:25:02.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.746+0000 7fc158200700 1 --2- 172.21.15.123:0/30400390 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc1500fdf70 0x7fc150070310 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fc13c00eb40 tx=0x7fc13c00b040 comp rx=0 tx=0).stop 2024-03-30T04:25:02.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.746+0000 7fc158200700 1 -- 172.21.15.123:0/30400390 shutdown_connections 2024-03-30T04:25:02.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.746+0000 7fc158200700 1 --2- 172.21.15.123:0/30400390 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fc1380692a0 0x7fc13806b750 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:02.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.746+0000 7fc158200700 1 --2- 172.21.15.123:0/30400390 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc1500fccc0 0x7fc150071c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:02.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.746+0000 7fc158200700 1 --2- 172.21.15.123:0/30400390 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc1500fdf70 0x7fc150070310 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:02.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.746+0000 7fc158200700 1 -- 172.21.15.123:0/30400390 >> 172.21.15.123:0/30400390 conn(0x7fc1500f8440 msgr2=0x7fc1500f9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:25:02.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.746+0000 7fc158200700 1 -- 172.21.15.123:0/30400390 shutdown_connections 2024-03-30T04:25:02.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.746+0000 7fc158200700 1 -- 172.21.15.123:0/30400390 wait complete. 2024-03-30T04:25:02.763 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-03-30T04:25:02.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.957+0000 7f2b9c800700 1 -- 172.21.15.123:0/3784319774 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b94065aa0 msgr2=0x7f2b940ff890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:25:02.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.957+0000 7f2b9c800700 1 --2- 172.21.15.123:0/3784319774 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b94065aa0 0x7f2b940ff890 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f2b88004940 tx=0x7f2b88009e30 comp rx=0 tx=0).stop 2024-03-30T04:25:02.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.957+0000 7f2b9c800700 1 -- 172.21.15.123:0/3784319774 shutdown_connections 2024-03-30T04:25:02.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.957+0000 7f2b9c800700 1 --2- 172.21.15.123:0/3784319774 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2b940ffdd0 0x7f2b941021f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:02.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.957+0000 7f2b9c800700 1 --2- 172.21.15.123:0/3784319774 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b94065aa0 0x7f2b940ff890 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:02.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.957+0000 7f2b9c800700 1 -- 172.21.15.123:0/3784319774 >> 172.21.15.123:0/3784319774 conn(0x7f2b940f7300 msgr2=0x7f2b940f9730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:25:02.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.957+0000 7f2b9c800700 1 -- 172.21.15.123:0/3784319774 shutdown_connections 2024-03-30T04:25:02.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.957+0000 7f2b9c800700 1 -- 172.21.15.123:0/3784319774 wait complete. 2024-03-30T04:25:02.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.958+0000 7f2b9c800700 1 Processor -- start 2024-03-30T04:25:02.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.958+0000 7f2b9c800700 1 -- start start 2024-03-30T04:25:02.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.959+0000 7f2b9c800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2b94065aa0 0x7f2b9410d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:25:02.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.959+0000 7f2b9c800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b940ffdd0 0x7f2b9410dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:25:02.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.959+0000 7f2b9c800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9410e300 con 0x7f2b940ffdd0 2024-03-30T04:25:02.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.959+0000 7f2b9c800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b94112ca0 con 0x7f2b94065aa0 2024-03-30T04:25:02.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.959+0000 7f2b99400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b940ffdd0 0x7f2b9410dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:25:02.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.959+0000 7f2b99400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b940ffdd0 0x7f2b9410dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54878/0 (socket says 172.21.15.123:54878) 2024-03-30T04:25:02.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.959+0000 7f2b99400700 1 -- 172.21.15.123:0/3936483710 learned_addr learned my addr 172.21.15.123:0/3936483710 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:25:02.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.959+0000 7f2b99e00700 1 --2- 172.21.15.123:0/3936483710 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2b94065aa0 0x7f2b9410d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:25:02.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.959+0000 7f2b99400700 1 -- 172.21.15.123:0/3936483710 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2b94065aa0 msgr2=0x7f2b9410d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:25:02.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.959+0000 7f2b99400700 1 --2- 172.21.15.123:0/3936483710 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2b94065aa0 0x7f2b9410d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:02.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.959+0000 7f2b99400700 1 -- 172.21.15.123:0/3936483710 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b88005040 con 0x7f2b940ffdd0 2024-03-30T04:25:02.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.960+0000 7f2b99400700 1 --2- 172.21.15.123:0/3936483710 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b940ffdd0 0x7f2b9410dc70 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f2b8400b790 tx=0x7f2b84009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:25:02.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.960+0000 7f2b8e000700 1 -- 172.21.15.123:0/3936483710 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b8400f6f0 con 0x7f2b940ffdd0 2024-03-30T04:25:02.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.960+0000 7f2b8e000700 1 -- 172.21.15.123:0/3936483710 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2b8400fd00 con 0x7f2b940ffdd0 2024-03-30T04:25:02.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.960+0000 7f2b8e000700 1 -- 172.21.15.123:0/3936483710 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b840103a0 con 0x7f2b940ffdd0 2024-03-30T04:25:02.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.960+0000 7f2b9c800700 1 -- 172.21.15.123:0/3936483710 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b94112f00 con 0x7f2b940ffdd0 2024-03-30T04:25:02.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.960+0000 7f2b9c800700 1 -- 172.21.15.123:0/3936483710 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b94113440 con 0x7f2b940ffdd0 2024-03-30T04:25:02.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.961+0000 7f2b9c800700 1 -- 172.21.15.123:0/3936483710 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b94064970 con 0x7f2b940ffdd0 2024-03-30T04:25:02.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.962+0000 7f2b8e000700 1 -- 172.21.15.123:0/3936483710 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f2b8400f850 con 0x7f2b940ffdd0 2024-03-30T04:25:02.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.963+0000 7f2b8e000700 1 --2- 172.21.15.123:0/3936483710 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f2b7c0692a0 0x7f2b7c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:25:02.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.963+0000 7f2b8e000700 1 -- 172.21.15.123:0/3936483710 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f2b84086f60 con 0x7f2b940ffdd0 2024-03-30T04:25:02.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.963+0000 7f2b99e00700 1 --2- 172.21.15.123:0/3936483710 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f2b7c0692a0 0x7f2b7c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:25:02.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.964+0000 7f2b99e00700 1 --2- 172.21.15.123:0/3936483710 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f2b7c0692a0 0x7f2b7c06b750 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2b8800b7d0 tx=0x7f2b88009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:25:02.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:02.967+0000 7f2b8e000700 1 -- 172.21.15.123:0/3936483710 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f2b84056cf0 con 0x7f2b940ffdd0 2024-03-30T04:25:02.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:25:02 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/1008497111' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:25:02.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:25:02 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/607721731' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T04:25:03.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:25:02 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/1008497111' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:25:03.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:25:02 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/607721731' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T04:25:03.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.178+0000 7f2b9c800700 1 -- 172.21.15.123:0/3936483710 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f2b941056c0 con 0x7f2b940ffdd0 2024-03-30T04:25:03.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.178+0000 7f2b8e000700 1 -- 172.21.15.123:0/3936483710 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v46) v1 ==== 107+0+11 (secure 0 0 0) 0x7f2b84056880 con 0x7f2b940ffdd0 2024-03-30T04:25:03.177 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 32 2024-03-30T04:25:03.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.181+0000 7f2b9c800700 1 -- 172.21.15.123:0/3936483710 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f2b7c0692a0 msgr2=0x7f2b7c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:25:03.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.181+0000 7f2b9c800700 1 --2- 172.21.15.123:0/3936483710 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f2b7c0692a0 0x7f2b7c06b750 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2b8800b7d0 tx=0x7f2b88009e30 comp rx=0 tx=0).stop 2024-03-30T04:25:03.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.181+0000 7f2b9c800700 1 -- 172.21.15.123:0/3936483710 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b940ffdd0 msgr2=0x7f2b9410dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:25:03.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.181+0000 7f2b9c800700 1 --2- 172.21.15.123:0/3936483710 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b940ffdd0 0x7f2b9410dc70 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f2b8400b790 tx=0x7f2b84009250 comp rx=0 tx=0).stop 2024-03-30T04:25:03.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.181+0000 7f2b9c800700 1 -- 172.21.15.123:0/3936483710 shutdown_connections 2024-03-30T04:25:03.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.181+0000 7f2b9c800700 1 --2- 172.21.15.123:0/3936483710 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f2b7c0692a0 0x7f2b7c06b750 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:03.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.181+0000 7f2b9c800700 1 --2- 172.21.15.123:0/3936483710 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2b94065aa0 0x7f2b9410d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:03.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.181+0000 7f2b9c800700 1 --2- 172.21.15.123:0/3936483710 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b940ffdd0 0x7f2b9410dc70 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:03.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.181+0000 7f2b9c800700 1 -- 172.21.15.123:0/3936483710 >> 172.21.15.123:0/3936483710 conn(0x7f2b940f7300 msgr2=0x7f2b940f7720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:25:03.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.181+0000 7f2b9c800700 1 -- 172.21.15.123:0/3936483710 shutdown_connections 2024-03-30T04:25:03.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.181+0000 7f2b9c800700 1 -- 172.21.15.123:0/3936483710 wait complete. 2024-03-30T04:25:03.198 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T04:25:03.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.409+0000 7f47e3c00700 1 -- 172.21.15.123:0/640897208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f47dc071e40 msgr2=0x7f47dc070350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:25:03.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.409+0000 7f47e3c00700 1 --2- 172.21.15.123:0/640897208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f47dc071e40 0x7f47dc070350 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f47d0004940 tx=0x7f47d0009e30 comp rx=0 tx=0).stop 2024-03-30T04:25:03.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.409+0000 7f47e3c00700 1 -- 172.21.15.123:0/640897208 shutdown_connections 2024-03-30T04:25:03.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.409+0000 7f47e3c00700 1 --2- 172.21.15.123:0/640897208 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f47dc070890 0x7f47dc070d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:03.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.409+0000 7f47e3c00700 1 --2- 172.21.15.123:0/640897208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f47dc071e40 0x7f47dc070350 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:03.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.409+0000 7f47e3c00700 1 -- 172.21.15.123:0/640897208 >> 172.21.15.123:0/640897208 conn(0x7f47dc0f8410 msgr2=0x7f47dc0fa840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:25:03.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.409+0000 7f47e3c00700 1 -- 172.21.15.123:0/640897208 shutdown_connections 2024-03-30T04:25:03.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.409+0000 7f47e3c00700 1 -- 172.21.15.123:0/640897208 wait complete. 2024-03-30T04:25:03.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.410+0000 7f47e3c00700 1 Processor -- start 2024-03-30T04:25:03.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.410+0000 7f47e3c00700 1 -- start start 2024-03-30T04:25:03.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.411+0000 7f47e3c00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f47dc070890 0x7f47dc10f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:25:03.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.411+0000 7f47e3c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f47dc071e40 0x7f47dc10fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:25:03.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.411+0000 7f47e3c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47dc1104c0 con 0x7f47dc071e40 2024-03-30T04:25:03.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.411+0000 7f47e3c00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47dc110630 con 0x7f47dc070890 2024-03-30T04:25:03.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.411+0000 7f47dbe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f47dc071e40 0x7f47dc10fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:25:03.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.411+0000 7f47dbe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f47dc071e40 0x7f47dc10fec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54892/0 (socket says 172.21.15.123:54892) 2024-03-30T04:25:03.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.411+0000 7f47dbe00700 1 -- 172.21.15.123:0/1645536662 learned_addr learned my addr 172.21.15.123:0/1645536662 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:25:03.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.411+0000 7f47e1200700 1 --2- 172.21.15.123:0/1645536662 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f47dc070890 0x7f47dc10f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:25:03.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.411+0000 7f47dbe00700 1 -- 172.21.15.123:0/1645536662 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f47dc070890 msgr2=0x7f47dc10f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:25:03.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.411+0000 7f47dbe00700 1 --2- 172.21.15.123:0/1645536662 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f47dc070890 0x7f47dc10f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:03.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.411+0000 7f47dbe00700 1 -- 172.21.15.123:0/1645536662 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47d0005040 con 0x7f47dc071e40 2024-03-30T04:25:03.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.412+0000 7f47dbe00700 1 --2- 172.21.15.123:0/1645536662 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f47dc071e40 0x7f47dc10fec0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f47c8002a50 tx=0x7f47c80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:25:03.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.412+0000 7f47d8c00700 1 -- 172.21.15.123:0/1645536662 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f47c8007cc0 con 0x7f47dc071e40 2024-03-30T04:25:03.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.412+0000 7f47e3c00700 1 -- 172.21.15.123:0/1645536662 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47dc115090 con 0x7f47dc071e40 2024-03-30T04:25:03.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.412+0000 7f47d8c00700 1 -- 172.21.15.123:0/1645536662 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f47c800f040 con 0x7f47dc071e40 2024-03-30T04:25:03.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.412+0000 7f47d8c00700 1 -- 172.21.15.123:0/1645536662 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f47c800d9c0 con 0x7f47dc071e40 2024-03-30T04:25:03.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.412+0000 7f47e3c00700 1 -- 172.21.15.123:0/1645536662 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47dc115650 con 0x7f47dc071e40 2024-03-30T04:25:03.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.413+0000 7f47e3c00700 1 -- 172.21.15.123:0/1645536662 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47dc064dc0 con 0x7f47dc071e40 2024-03-30T04:25:03.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.414+0000 7f47d8c00700 1 -- 172.21.15.123:0/1645536662 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f47c800db20 con 0x7f47dc071e40 2024-03-30T04:25:03.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.415+0000 7f47d8c00700 1 --2- 172.21.15.123:0/1645536662 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f47c40692a0 0x7f47c406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:25:03.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.415+0000 7f47d8c00700 1 -- 172.21.15.123:0/1645536662 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f47c8086ed0 con 0x7f47dc071e40 2024-03-30T04:25:03.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.415+0000 7f47e1200700 1 --2- 172.21.15.123:0/1645536662 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f47c40692a0 0x7f47c406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:25:03.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.416+0000 7f47e1200700 1 --2- 172.21.15.123:0/1645536662 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f47c40692a0 0x7f47c406b750 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f47d0000c00 tx=0x7f47d0009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:25:03.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.420+0000 7f47d8c00700 1 -- 172.21.15.123:0/1645536662 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f47c8056ba0 con 0x7f47dc071e40 2024-03-30T04:25:03.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.675+0000 7f47e3c00700 1 -- 172.21.15.123:0/1645536662 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47dc1158c0 con 0x7f47dc071e40 2024-03-30T04:25:03.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.675+0000 7f47d8c00700 1 -- 172.21.15.123:0/1645536662 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+4731 (secure 0 0 0) 0x7f47dc1158c0 con 0x7f47dc071e40 2024-03-30T04:25:03.674 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:25:03.674 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T04:24:30.211769+0000","modified":"2024-03-30T04:24:51.100702+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14488,"mds_1":24309},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14488":{"gid":14488,"name":"cephfs.smithi123.dvtind","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.123:6829/3037849622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3037849622},{"type":"v1","addr":"172.21.15.123:6829","nonce":3037849622}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14500":{"gid":14500,"name":"cephfs.smithi123.tqxvzr","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6827/2433675995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":2433675995},{"type":"v1","addr":"172.21.15.123:6827","nonce":2433675995}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24309":{"gid":24309,"name":"cephfs.smithi135.zvnlap","rank":1,"incarnation":7,"state":"up:active","state_seq":4,"addr":"172.21.15.135:6825/4050541062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":4050541062},{"type":"v1","addr":"172.21.15.135:6825","nonce":4050541062}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24319":{"gid":24319,"name":"cephfs.smithi135.xwgjvb","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.135:6827/1096986122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":1096986122},{"type":"v1","addr":"172.21.15.135:6827","nonce":1096986122}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2024-03-30T04:25:03.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.677+0000 7f47e3c00700 1 -- 172.21.15.123:0/1645536662 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f47c40692a0 msgr2=0x7f47c406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:25:03.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.677+0000 7f47e3c00700 1 --2- 172.21.15.123:0/1645536662 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f47c40692a0 0x7f47c406b750 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f47d0000c00 tx=0x7f47d0009e30 comp rx=0 tx=0).stop 2024-03-30T04:25:03.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.677+0000 7f47e3c00700 1 -- 172.21.15.123:0/1645536662 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f47dc071e40 msgr2=0x7f47dc10fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:25:03.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.677+0000 7f47e3c00700 1 --2- 172.21.15.123:0/1645536662 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f47dc071e40 0x7f47dc10fec0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f47c8002a50 tx=0x7f47c80046b0 comp rx=0 tx=0).stop 2024-03-30T04:25:03.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.678+0000 7f47e3c00700 1 -- 172.21.15.123:0/1645536662 shutdown_connections 2024-03-30T04:25:03.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.678+0000 7f47e3c00700 1 --2- 172.21.15.123:0/1645536662 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f47c40692a0 0x7f47c406b750 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:03.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.678+0000 7f47e3c00700 1 --2- 172.21.15.123:0/1645536662 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f47dc070890 0x7f47dc10f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:03.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.678+0000 7f47e3c00700 1 --2- 172.21.15.123:0/1645536662 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f47dc071e40 0x7f47dc10fec0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:03.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.678+0000 7f47e3c00700 1 -- 172.21.15.123:0/1645536662 >> 172.21.15.123:0/1645536662 conn(0x7f47dc0f8410 msgr2=0x7f47dc0f9c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:25:03.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.678+0000 7f47e3c00700 1 -- 172.21.15.123:0/1645536662 shutdown_connections 2024-03-30T04:25:03.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:03.678+0000 7f47e3c00700 1 -- 172.21.15.123:0/1645536662 wait complete. 2024-03-30T04:25:03.677 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 19 2024-03-30T04:25:03.695 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 19, 'max_mds': 2, 'flags': 50} 2024-03-30T04:25:03.695 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-03-30T04:25:03.721 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-03-30T04:25:03.722 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.1': {}} 2024-03-30T04:25:03.722 INFO:tasks.ceph_fuse:client.0 config is {} 2024-03-30T04:25:03.722 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-30T04:25:03.722 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-30T04:25:03.722 INFO:tasks.ceph_fuse:client.1 config is {} 2024-03-30T04:25:03.722 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-30T04:25:03.722 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-30T04:25:03.722 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:25:03.723 DEBUG:teuthology.orchestra.run.smithi135:> ip netns list 2024-03-30T04:25:03.771 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:25:03.771 DEBUG:teuthology.orchestra.run.smithi135:> sudo ip link delete ceph-brx 2024-03-30T04:25:03.844 INFO:teuthology.orchestra.run.smithi135.stderr:Cannot find device "ceph-brx" 2024-03-30T04:25:03.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T04:25:03.847 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:25:03.847 DEBUG:teuthology.orchestra.run.smithi123:> ip netns list 2024-03-30T04:25:03.893 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:25:03.893 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link delete ceph-brx 2024-03-30T04:25:03.951 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:25:03 smithi123 ceph-mon[35245]: pgmap v217: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-30T04:25:03.952 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:25:03 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/30400390' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-30T04:25:03.952 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:25:03 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/3936483710' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-30T04:25:03.952 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:25:03 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:25:03.952 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:25:03 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:25:03.952 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:25:03 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:25:03.952 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:25:03 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:25:03.952 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:25:03 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:25:03.964 INFO:teuthology.orchestra.run.smithi123.stderr:Cannot find device "ceph-brx" 2024-03-30T04:25:03.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T04:25:03.967 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-03-30T04:25:03.967 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph fs ls 2024-03-30T04:25:04.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:25:03 smithi135 ceph-mon[44364]: pgmap v217: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-30T04:25:04.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:25:03 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/30400390' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-30T04:25:04.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:25:03 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/3936483710' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-30T04:25:04.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:25:03 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:25:04.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:25:03 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:25:04.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:25:03 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:25:04.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:25:03 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:25:04.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:25:03 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:25:04.384 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:25:04.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:25:04 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/1645536662' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:25:05.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:25:04 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/1645536662' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:25:05.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:25:05 smithi123 ceph-mon[35245]: pgmap v218: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.5 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-30T04:25:06.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:25:05 smithi135 ceph-mon[44364]: pgmap v218: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.5 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-30T04:25:06.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:06.277+0000 7f03a3200700 1 -- 172.21.15.123:0/3666843302 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f039c0fde90 msgr2=0x7f039c0fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:25:06.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:06.277+0000 7f03a3200700 1 --2- 172.21.15.123:0/3666843302 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f039c0fde90 0x7f039c0fe260 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f038c004940 tx=0x7f038c009e30 comp rx=0 tx=0).stop 2024-03-30T04:25:06.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:06.277+0000 7f03a3200700 1 -- 172.21.15.123:0/3666843302 shutdown_connections 2024-03-30T04:25:06.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:06.277+0000 7f03a3200700 1 --2- 172.21.15.123:0/3666843302 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f039c0fe830 0x7f039c106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:06.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:06.277+0000 7f03a3200700 1 --2- 172.21.15.123:0/3666843302 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f039c0fde90 0x7f039c0fe260 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:06.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:06.277+0000 7f03a3200700 1 -- 172.21.15.123:0/3666843302 >> 172.21.15.123:0/3666843302 conn(0x7f039c072bc0 msgr2=0x7f039c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:25:06.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:06.277+0000 7f03a3200700 1 -- 172.21.15.123:0/3666843302 shutdown_connections 2024-03-30T04:25:06.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:06.277+0000 7f03a3200700 1 -- 172.21.15.123:0/3666843302 wait complete. 2024-03-30T04:25:06.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:06.278+0000 7f03a3200700 1 Processor -- start 2024-03-30T04:25:06.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:06.278+0000 7f03a3200700 1 -- start start 2024-03-30T04:25:06.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:06.279+0000 7f03a3200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f039c0fde90 0x7f039c115d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:25:06.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:06.279+0000 7f03a3200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f039c0fe830 0x7f039c116240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:25:06.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:06.279+0000 7f03a3200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f039c116920 con 0x7f039c0fde90 2024-03-30T04:25:06.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:06.279+0000 7f03a3200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f039c10fc60 con 0x7f039c0fe830 2024-03-30T04:25:06.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:06.279+0000 7f039be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f039c0fde90 0x7f039c115d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:25:06.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:06.279+0000 7f039be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f039c0fde90 0x7f039c115d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47368/0 (socket says 172.21.15.123:47368) 2024-03-30T04:25:06.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:06.279+0000 7f039be00700 1 -- 172.21.15.123:0/3655412633 learned_addr learned my addr 172.21.15.123:0/3655412633 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:25:06.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:06.279+0000 7f039b400700 1 --2- 172.21.15.123:0/3655412633 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f039c0fe830 0x7f039c116240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:25:06.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:06.279+0000 7f039be00700 1 -- 172.21.15.123:0/3655412633 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f039c0fe830 msgr2=0x7f039c116240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:25:06.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:06.279+0000 7f039be00700 1 --2- 172.21.15.123:0/3655412633 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f039c0fe830 0x7f039c116240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:06.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:06.279+0000 7f039be00700 1 -- 172.21.15.123:0/3655412633 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f038c005040 con 0x7f039c0fde90 2024-03-30T04:25:06.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:06.280+0000 7f039be00700 1 --2- 172.21.15.123:0/3655412633 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f039c0fde90 0x7f039c115d00 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f038c000c00 tx=0x7f038c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:25:06.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:06.280+0000 7f0393e00700 1 -- 172.21.15.123:0/3655412633 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f038c00e040 con 0x7f039c0fde90 2024-03-30T04:25:06.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:06.280+0000 7f03a3200700 1 -- 172.21.15.123:0/3655412633 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f039c10fe80 con 0x7f039c0fde90 2024-03-30T04:25:06.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:06.280+0000 7f0393e00700 1 -- 172.21.15.123:0/3655412633 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f038c00bed0 con 0x7f039c0fde90 2024-03-30T04:25:06.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:06.280+0000 7f0393e00700 1 -- 172.21.15.123:0/3655412633 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f038c01c3f0 con 0x7f039c0fde90 2024-03-30T04:25:06.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:06.280+0000 7f03a3200700 1 -- 172.21.15.123:0/3655412633 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f039c1103c0 con 0x7f039c0fde90 2024-03-30T04:25:06.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:06.281+0000 7f03a3200700 1 -- 172.21.15.123:0/3655412633 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f039c106a20 con 0x7f039c0fde90 2024-03-30T04:25:06.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:06.282+0000 7f0393e00700 1 -- 172.21.15.123:0/3655412633 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f038c013420 con 0x7f039c0fde90 2024-03-30T04:25:06.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:06.283+0000 7f0393e00700 1 --2- 172.21.15.123:0/3655412633 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f03800692b0 0x7f038006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:25:06.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:06.283+0000 7f0393e00700 1 -- 172.21.15.123:0/3655412633 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f038c088670 con 0x7f039c0fde90 2024-03-30T04:25:06.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:06.283+0000 7f039b400700 1 --2- 172.21.15.123:0/3655412633 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f03800692b0 0x7f038006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:25:06.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:06.283+0000 7f039b400700 1 --2- 172.21.15.123:0/3655412633 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f03800692b0 0x7f038006b760 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f039c1115a0 tx=0x7f03880091a0 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:25:06.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:06.287+0000 7f0393e00700 1 -- 172.21.15.123:0/3655412633 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f038c058340 con 0x7f039c0fde90 2024-03-30T04:25:06.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:06.530+0000 7f03a3200700 1 -- 172.21.15.123:0/3655412633 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f039c109e70 con 0x7f039c0fde90 2024-03-30T04:25:06.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:06.530+0000 7f0393e00700 1 -- 172.21.15.123:0/3655412633 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v19) v1 ==== 53+0+83 (secure 0 0 0) 0x7f038c057ed0 con 0x7f039c0fde90 2024-03-30T04:25:06.529 INFO:teuthology.orchestra.run.smithi123.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-03-30T04:25:06.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:06.533+0000 7f03a3200700 1 -- 172.21.15.123:0/3655412633 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f03800692b0 msgr2=0x7f038006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:25:06.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:06.533+0000 7f03a3200700 1 --2- 172.21.15.123:0/3655412633 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f03800692b0 0x7f038006b760 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f039c1115a0 tx=0x7f03880091a0 comp rx=0 tx=0).stop 2024-03-30T04:25:06.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:06.533+0000 7f03a3200700 1 -- 172.21.15.123:0/3655412633 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f039c0fde90 msgr2=0x7f039c115d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:25:06.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:06.533+0000 7f03a3200700 1 --2- 172.21.15.123:0/3655412633 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f039c0fde90 0x7f039c115d00 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f038c000c00 tx=0x7f038c009e30 comp rx=0 tx=0).stop 2024-03-30T04:25:06.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:06.533+0000 7f03a3200700 1 -- 172.21.15.123:0/3655412633 shutdown_connections 2024-03-30T04:25:06.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:06.533+0000 7f03a3200700 1 --2- 172.21.15.123:0/3655412633 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f03800692b0 0x7f038006b760 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:06.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:06.533+0000 7f03a3200700 1 --2- 172.21.15.123:0/3655412633 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f039c0fde90 0x7f039c115d00 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:06.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:06.533+0000 7f03a3200700 1 --2- 172.21.15.123:0/3655412633 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f039c0fe830 0x7f039c116240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:06.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:06.533+0000 7f03a3200700 1 -- 172.21.15.123:0/3655412633 >> 172.21.15.123:0/3655412633 conn(0x7f039c072bc0 msgr2=0x7f039c0fb7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:25:06.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:06.533+0000 7f03a3200700 1 -- 172.21.15.123:0/3655412633 shutdown_connections 2024-03-30T04:25:06.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:06.533+0000 7f03a3200700 1 -- 172.21.15.123:0/3655412633 wait complete. 2024-03-30T04:25:06.925 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:25:06 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/3655412633' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-30T04:25:07.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:25:06 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/3655412633' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-30T04:25:07.334 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-30T04:25:07.334 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-30T04:25:07.334 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi123.front.sepia.ceph.com 2024-03-30T04:25:07.334 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-30T04:25:07.334 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-30T04:25:07.334 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-30T04:25:07.334 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-30T04:25:07.334 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-30T04:25:07.335 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-30T04:25:07.335 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:25:07.335 DEBUG:teuthology.orchestra.run.smithi123:> ip addr 2024-03-30T04:25:07.382 INFO:teuthology.orchestra.run.smithi123.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-30T04:25:07.382 INFO:teuthology.orchestra.run.smithi123.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-30T04:25:07.382 INFO:teuthology.orchestra.run.smithi123.stdout: inet 127.0.0.1/8 scope host lo 2024-03-30T04:25:07.382 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-03-30T04:25:07.382 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 ::1/128 scope host 2024-03-30T04:25:07.383 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-03-30T04:25:07.383 INFO:teuthology.orchestra.run.smithi123.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-30T04:25:07.383 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:d8:da:6c brd ff:ff:ff:ff:ff:ff 2024-03-30T04:25:07.383 INFO:teuthology.orchestra.run.smithi123.stdout: altname enp7s0f0 2024-03-30T04:25:07.383 INFO:teuthology.orchestra.run.smithi123.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-30T04:25:07.383 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:d8:da:6d brd ff:ff:ff:ff:ff:ff 2024-03-30T04:25:07.383 INFO:teuthology.orchestra.run.smithi123.stdout: altname enp7s0f1 2024-03-30T04:25:07.383 INFO:teuthology.orchestra.run.smithi123.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-30T04:25:07.383 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:8f:d3:f0 brd ff:ff:ff:ff:ff:ff 2024-03-30T04:25:07.383 INFO:teuthology.orchestra.run.smithi123.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-30T04:25:07.383 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:8f:d3:f1 brd ff:ff:ff:ff:ff:ff 2024-03-30T04:25:07.383 INFO:teuthology.orchestra.run.smithi123.stdout: inet 172.21.15.123/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-30T04:25:07.383 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft 41963sec preferred_lft 41963sec 2024-03-30T04:25:07.383 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::ec4:7aff:fe8f:d3f1/64 scope link 2024-03-30T04:25:07.384 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-03-30T04:25:07.384 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-30T04:25:07.384 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:25:07.384 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec bash -c ' 2024-03-30T04:25:07.384 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2024-03-30T04:25:07.384 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link add name ceph-brx type bridge 2024-03-30T04:25:07.384 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip addr flush dev ceph-brx 2024-03-30T04:25:07.385 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link set ceph-brx up 2024-03-30T04:25:07.385 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-30T04:25:07.385 DEBUG:teuthology.orchestra.run.smithi123:> ') 2024-03-30T04:25:07.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:25:07.837 DEBUG:teuthology.orchestra.run.smithi123:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-30T04:25:07.900 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:25:07 smithi123 ceph-mon[35245]: pgmap v219: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.4 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-30T04:25:07.906 INFO:teuthology.orchestra.run.smithi123.stdout:1 2024-03-30T04:25:07.909 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:25:07.909 DEBUG:teuthology.orchestra.run.smithi123:> route 2024-03-30T04:25:07.978 INFO:teuthology.orchestra.run.smithi123.stdout:Kernel IP routing table 2024-03-30T04:25:07.978 INFO:teuthology.orchestra.run.smithi123.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-30T04:25:07.978 INFO:teuthology.orchestra.run.smithi123.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-30T04:25:07.978 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-30T04:25:07.978 INFO:teuthology.orchestra.run.smithi123.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-30T04:25:07.979 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:25:07.979 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec bash -c ' 2024-03-30T04:25:07.979 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2024-03-30T04:25:07.979 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-30T04:25:07.979 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-30T04:25:07.979 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-30T04:25:07.980 DEBUG:teuthology.orchestra.run.smithi123:> ') 2024-03-30T04:25:08.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:25:07 smithi135 ceph-mon[44364]: pgmap v219: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.4 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-30T04:25:08.612 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:25:08.612 DEBUG:teuthology.orchestra.run.smithi123:> ip netns list 2024-03-30T04:25:08.659 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:25:08.660 DEBUG:teuthology.orchestra.run.smithi123:> ip netns list-id 2024-03-30T04:25:08.705 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:25:08.705 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec bash -c ' 2024-03-30T04:25:08.705 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2024-03-30T04:25:08.705 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-30T04:25:08.705 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-30T04:25:08.705 DEBUG:teuthology.orchestra.run.smithi123:> ') 2024-03-30T04:25:08.883 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-30T04:25:08.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:25:08.883 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec bash -c ' 2024-03-30T04:25:08.883 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2024-03-30T04:25:08.883 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-30T04:25:08.884 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-30T04:25:08.884 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-30T04:25:08.884 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-30T04:25:08.884 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-30T04:25:08.884 DEBUG:teuthology.orchestra.run.smithi123:> ') 2024-03-30T04:25:09.147 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:25:09.147 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec bash -c ' 2024-03-30T04:25:09.147 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2024-03-30T04:25:09.147 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link set brx.0 up 2024-03-30T04:25:09.147 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link set dev brx.0 master ceph-brx 2024-03-30T04:25:09.148 DEBUG:teuthology.orchestra.run.smithi123:> ') 2024-03-30T04:25:09.314 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-30T04:25:09.314 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-30T04:25:09.314 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-30T04:25:09.359 INFO:teuthology.orchestra.run.smithi123.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-30T04:25:09.359 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-30T04:25:09.359 DEBUG:teuthology.orchestra.run.smithi123:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-30T04:25:09.444 DEBUG:teuthology.orchestra.run.smithi123:> sudo modprobe fuse 2024-03-30T04:25:09.552 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-30T04:25:09.552 DEBUG:teuthology.orchestra.run.smithi123:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-30T04:25:09.619 INFO:teuthology.orchestra.run.smithi123.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-30T04:25:09.621 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-30T04:25:09.622 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:25:09.622 DEBUG:teuthology.orchestra.run.smithi123:> ls /sys/fs/fuse/connections 2024-03-30T04:25:09.667 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-30T04:25:09.810 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-30T04:25:09.819 DEBUG:teuthology.orchestra.run.smithi123:> sudo modprobe fuse 2024-03-30T04:25:09.874 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:25:09 smithi123 ceph-mon[35245]: pgmap v220: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:25:09.893 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-30T04:25:09.893 DEBUG:teuthology.orchestra.run.smithi123:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-30T04:25:09.961 INFO:teuthology.orchestra.run.smithi123.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-30T04:25:09.963 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-30T04:25:09.964 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:25:09.964 DEBUG:teuthology.orchestra.run.smithi123:> ls /sys/fs/fuse/connections 2024-03-30T04:25:10.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi123.stderr:2024-03-30T04:25:10.021+0000 7f9825af8540 -1 init, newargv = 0x561f95123e60 newargc=15 2024-03-30T04:25:10.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi123.stderr:ceph-fuse[94945]: starting ceph client 2024-03-30T04:25:10.033 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi123.stderr:ceph-fuse[94945]: starting fuse 2024-03-30T04:25:10.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:25:09 smithi135 ceph-mon[44364]: pgmap v220: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:25:11.110 DEBUG:teuthology.orchestra.run.smithi123:> sudo modprobe fuse 2024-03-30T04:25:11.190 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-30T04:25:11.190 DEBUG:teuthology.orchestra.run.smithi123:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-30T04:25:11.261 INFO:teuthology.orchestra.run.smithi123.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-30T04:25:11.263 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-30T04:25:11.264 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:25:11.264 DEBUG:teuthology.orchestra.run.smithi123:> ls /sys/fs/fuse/connections 2024-03-30T04:25:11.311 INFO:teuthology.orchestra.run.smithi123.stdout:147 2024-03-30T04:25:11.311 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [147] 2024-03-30T04:25:11.311 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-03-30T04:25:11.311 DEBUG:teuthology.orchestra.run.smithi123:> import glob 2024-03-30T04:25:11.311 DEBUG:teuthology.orchestra.run.smithi123:> import re 2024-03-30T04:25:11.312 DEBUG:teuthology.orchestra.run.smithi123:> import os 2024-03-30T04:25:11.312 DEBUG:teuthology.orchestra.run.smithi123:> import subprocess 2024-03-30T04:25:11.312 DEBUG:teuthology.orchestra.run.smithi123:> 2024-03-30T04:25:11.312 DEBUG:teuthology.orchestra.run.smithi123:> def _find_admin_socket(client_name): 2024-03-30T04:25:11.312 DEBUG:teuthology.orchestra.run.smithi123:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-30T04:25:11.312 DEBUG:teuthology.orchestra.run.smithi123:> files = glob.glob(asok_path) 2024-03-30T04:25:11.312 DEBUG:teuthology.orchestra.run.smithi123:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-30T04:25:11.312 DEBUG:teuthology.orchestra.run.smithi123:> 2024-03-30T04:25:11.312 DEBUG:teuthology.orchestra.run.smithi123:> # Given a non-glob path, it better be there 2024-03-30T04:25:11.312 DEBUG:teuthology.orchestra.run.smithi123:> if "*" not in asok_path: 2024-03-30T04:25:11.312 DEBUG:teuthology.orchestra.run.smithi123:> assert(len(files) == 1) 2024-03-30T04:25:11.312 DEBUG:teuthology.orchestra.run.smithi123:> return files[0] 2024-03-30T04:25:11.313 DEBUG:teuthology.orchestra.run.smithi123:> 2024-03-30T04:25:11.313 DEBUG:teuthology.orchestra.run.smithi123:> for f in files: 2024-03-30T04:25:11.313 DEBUG:teuthology.orchestra.run.smithi123:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-30T04:25:11.313 DEBUG:teuthology.orchestra.run.smithi123:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-30T04:25:11.313 DEBUG:teuthology.orchestra.run.smithi123:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-30T04:25:11.313 DEBUG:teuthology.orchestra.run.smithi123:> contents = proc_f.read() 2024-03-30T04:25:11.313 DEBUG:teuthology.orchestra.run.smithi123:> if mountpoint in contents: 2024-03-30T04:25:11.313 DEBUG:teuthology.orchestra.run.smithi123:> return f 2024-03-30T04:25:11.313 DEBUG:teuthology.orchestra.run.smithi123:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-30T04:25:11.313 DEBUG:teuthology.orchestra.run.smithi123:> 2024-03-30T04:25:11.313 DEBUG:teuthology.orchestra.run.smithi123:> print(_find_admin_socket("client.0")) 2024-03-30T04:25:11.313 DEBUG:teuthology.orchestra.run.smithi123:> ' 2024-03-30T04:25:11.454 INFO:teuthology.orchestra.run.smithi123.stdout:/var/run/ceph/ceph-client.0.94945.asok 2024-03-30T04:25:11.633 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.94945.asok 2024-03-30T04:25:11.633 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:25:11.633 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.94945.asok status 2024-03-30T04:25:11.850 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:25:11.850 INFO:teuthology.orchestra.run.smithi123.stdout: "metadata": { 2024-03-30T04:25:11.850 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_sha1": "5c02433542ab1af3975314808d208cabd22abc39", 2024-03-30T04:25:11.850 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_version": "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)", 2024-03-30T04:25:11.850 INFO:teuthology.orchestra.run.smithi123.stdout: "entity_id": "0", 2024-03-30T04:25:11.850 INFO:teuthology.orchestra.run.smithi123.stdout: "hostname": "smithi123", 2024-03-30T04:25:11.850 INFO:teuthology.orchestra.run.smithi123.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-30T04:25:11.850 INFO:teuthology.orchestra.run.smithi123.stdout: "pid": "94945", 2024-03-30T04:25:11.851 INFO:teuthology.orchestra.run.smithi123.stdout: "root": "/" 2024-03-30T04:25:11.851 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:25:11.851 INFO:teuthology.orchestra.run.smithi123.stdout: "dentry_count": 0, 2024-03-30T04:25:11.851 INFO:teuthology.orchestra.run.smithi123.stdout: "dentry_pinned_count": 0, 2024-03-30T04:25:11.851 INFO:teuthology.orchestra.run.smithi123.stdout: "id": 14540, 2024-03-30T04:25:11.851 INFO:teuthology.orchestra.run.smithi123.stdout: "inst": { 2024-03-30T04:25:11.851 INFO:teuthology.orchestra.run.smithi123.stdout: "name": { 2024-03-30T04:25:11.851 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "client", 2024-03-30T04:25:11.851 INFO:teuthology.orchestra.run.smithi123.stdout: "num": 14540 2024-03-30T04:25:11.851 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:25:11.851 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": { 2024-03-30T04:25:11.851 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2024-03-30T04:25:11.851 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "192.168.0.1:0", 2024-03-30T04:25:11.851 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 1247659015 2024-03-30T04:25:11.852 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:25:11.852 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:25:11.852 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": { 2024-03-30T04:25:11.852 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2024-03-30T04:25:11.852 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "192.168.0.1:0", 2024-03-30T04:25:11.852 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 1247659015 2024-03-30T04:25:11.852 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:25:11.852 INFO:teuthology.orchestra.run.smithi123.stdout: "inst_str": "client.14540 192.168.0.1:0/1247659015", 2024-03-30T04:25:11.852 INFO:teuthology.orchestra.run.smithi123.stdout: "addr_str": "192.168.0.1:0/1247659015", 2024-03-30T04:25:11.852 INFO:teuthology.orchestra.run.smithi123.stdout: "inode_count": 1, 2024-03-30T04:25:11.852 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_epoch": 19, 2024-03-30T04:25:11.852 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_epoch": 46, 2024-03-30T04:25:11.852 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_epoch_barrier": 0, 2024-03-30T04:25:11.852 INFO:teuthology.orchestra.run.smithi123.stdout: "blocklisted": false, 2024-03-30T04:25:11.853 INFO:teuthology.orchestra.run.smithi123.stdout: "fs_name": "cephfs" 2024-03-30T04:25:11.853 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:25:11.864 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- ceph fs ls 2024-03-30T04:25:11.923 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:25:11 smithi123 ceph-mon[35245]: pgmap v221: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 170 B/s wr, 4 op/s 2024-03-30T04:25:11.923 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:25:11 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:25:12.059 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:25:11 smithi135 ceph-mon[44364]: pgmap v221: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 170 B/s wr, 4 op/s 2024-03-30T04:25:12.059 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:25:11 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:25:12.287 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:25:13.972 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:25:13 smithi123 ceph-mon[35245]: pgmap v222: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 341 B/s wr, 4 op/s 2024-03-30T04:25:14.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:25:13 smithi135 ceph-mon[44364]: pgmap v222: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 341 B/s wr, 4 op/s 2024-03-30T04:25:14.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:14.343+0000 7f6b91c00700 1 -- 172.21.15.123:0/3977254038 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b8c102e20 msgr2=0x7f6b8c1031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:25:14.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:14.343+0000 7f6b91c00700 1 --2- 172.21.15.123:0/3977254038 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b8c102e20 0x7f6b8c1031f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c004940 tx=0x7f6b7c009e30 comp rx=0 tx=0).stop 2024-03-30T04:25:14.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:14.343+0000 7f6b91c00700 1 -- 172.21.15.123:0/3977254038 shutdown_connections 2024-03-30T04:25:14.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:14.343+0000 7f6b91c00700 1 --2- 172.21.15.123:0/3977254038 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b8c1037c0 0x7f6b8c0fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:14.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:14.343+0000 7f6b91c00700 1 --2- 172.21.15.123:0/3977254038 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b8c102e20 0x7f6b8c1031f0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:14.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:14.343+0000 7f6b91c00700 1 -- 172.21.15.123:0/3977254038 >> 172.21.15.123:0/3977254038 conn(0x7f6b8c0f89d0 msgr2=0x7f6b8c0fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:25:14.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:14.343+0000 7f6b91c00700 1 -- 172.21.15.123:0/3977254038 shutdown_connections 2024-03-30T04:25:14.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:14.343+0000 7f6b91c00700 1 -- 172.21.15.123:0/3977254038 wait complete. 2024-03-30T04:25:14.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:14.344+0000 7f6b91c00700 1 Processor -- start 2024-03-30T04:25:14.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:14.344+0000 7f6b91c00700 1 -- start start 2024-03-30T04:25:14.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:14.345+0000 7f6b91c00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b8c102e20 0x7f6b8c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:25:14.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:14.345+0000 7f6b91c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b8c1037c0 0x7f6b8c070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:25:14.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:14.345+0000 7f6b91c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b8c070d90 con 0x7f6b8c1037c0 2024-03-30T04:25:14.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:14.345+0000 7f6b91c00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b8c070ed0 con 0x7f6b8c102e20 2024-03-30T04:25:14.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:14.345+0000 7f6b8a000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b8c1037c0 0x7f6b8c070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:25:14.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:14.345+0000 7f6b8a000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b8c1037c0 0x7f6b8c070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33482/0 (socket says 172.21.15.123:33482) 2024-03-30T04:25:14.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:14.345+0000 7f6b8a000700 1 -- 172.21.15.123:0/1122765406 learned_addr learned my addr 172.21.15.123:0/1122765406 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:25:14.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:14.345+0000 7f6b8aa00700 1 --2- 172.21.15.123:0/1122765406 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b8c102e20 0x7f6b8c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:25:14.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:14.345+0000 7f6b8a000700 1 -- 172.21.15.123:0/1122765406 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b8c102e20 msgr2=0x7f6b8c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:25:14.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:14.345+0000 7f6b8a000700 1 --2- 172.21.15.123:0/1122765406 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b8c102e20 0x7f6b8c0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:14.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:14.345+0000 7f6b8a000700 1 -- 172.21.15.123:0/1122765406 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b7c005040 con 0x7f6b8c1037c0 2024-03-30T04:25:14.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:14.345+0000 7f6b8a000700 1 --2- 172.21.15.123:0/1122765406 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b8c1037c0 0x7f6b8c070800 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f6b78002a80 tx=0x7f6b78004630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:25:14.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:14.346+0000 7f6b82a00700 1 -- 172.21.15.123:0/1122765406 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b78007cc0 con 0x7f6b8c1037c0 2024-03-30T04:25:14.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:14.346+0000 7f6b82a00700 1 -- 172.21.15.123:0/1122765406 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6b7800f040 con 0x7f6b8c1037c0 2024-03-30T04:25:14.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:14.346+0000 7f6b91c00700 1 -- 172.21.15.123:0/1122765406 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b8c119e20 con 0x7f6b8c1037c0 2024-03-30T04:25:14.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:14.346+0000 7f6b82a00700 1 -- 172.21.15.123:0/1122765406 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b7800da40 con 0x7f6b8c1037c0 2024-03-30T04:25:14.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:14.346+0000 7f6b91c00700 1 -- 172.21.15.123:0/1122765406 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b8c11a2a0 con 0x7f6b8c1037c0 2024-03-30T04:25:14.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:14.347+0000 7f6b91c00700 1 -- 172.21.15.123:0/1122765406 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b8c0fe570 con 0x7f6b8c1037c0 2024-03-30T04:25:14.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:14.348+0000 7f6b82a00700 1 -- 172.21.15.123:0/1122765406 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f6b7800dbe0 con 0x7f6b8c1037c0 2024-03-30T04:25:14.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:14.348+0000 7f6b82a00700 1 --2- 172.21.15.123:0/1122765406 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f6b700692b0 0x7f6b7006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:25:14.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:14.349+0000 7f6b82a00700 1 -- 172.21.15.123:0/1122765406 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f6b780868d0 con 0x7f6b8c1037c0 2024-03-30T04:25:14.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:14.349+0000 7f6b8aa00700 1 --2- 172.21.15.123:0/1122765406 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f6b700692b0 0x7f6b7006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:25:14.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:14.349+0000 7f6b8aa00700 1 --2- 172.21.15.123:0/1122765406 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f6b700692b0 0x7f6b7006b760 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c00b020 tx=0x7f6b7c009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:25:14.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:14.353+0000 7f6b82a00700 1 -- 172.21.15.123:0/1122765406 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f6b78052b10 con 0x7f6b8c1037c0 2024-03-30T04:25:14.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:14.594+0000 7f6b91c00700 1 -- 172.21.15.123:0/1122765406 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f6b8c04bed0 con 0x7f6b8c1037c0 2024-03-30T04:25:14.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:14.594+0000 7f6b82a00700 1 -- 172.21.15.123:0/1122765406 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v19) v1 ==== 53+0+83 (secure 0 0 0) 0x7f6b78017070 con 0x7f6b8c1037c0 2024-03-30T04:25:14.593 INFO:teuthology.orchestra.run.smithi123.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-03-30T04:25:14.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:14.596+0000 7f6b91c00700 1 -- 172.21.15.123:0/1122765406 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f6b700692b0 msgr2=0x7f6b7006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:25:14.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:14.596+0000 7f6b91c00700 1 --2- 172.21.15.123:0/1122765406 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f6b700692b0 0x7f6b7006b760 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c00b020 tx=0x7f6b7c009e30 comp rx=0 tx=0).stop 2024-03-30T04:25:14.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:14.596+0000 7f6b91c00700 1 -- 172.21.15.123:0/1122765406 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b8c1037c0 msgr2=0x7f6b8c070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:25:14.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:14.596+0000 7f6b91c00700 1 --2- 172.21.15.123:0/1122765406 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b8c1037c0 0x7f6b8c070800 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f6b78002a80 tx=0x7f6b78004630 comp rx=0 tx=0).stop 2024-03-30T04:25:14.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:14.596+0000 7f6b91c00700 1 -- 172.21.15.123:0/1122765406 shutdown_connections 2024-03-30T04:25:14.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:14.596+0000 7f6b91c00700 1 --2- 172.21.15.123:0/1122765406 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f6b700692b0 0x7f6b7006b760 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:14.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:14.596+0000 7f6b91c00700 1 --2- 172.21.15.123:0/1122765406 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b8c102e20 0x7f6b8c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:14.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:14.596+0000 7f6b91c00700 1 --2- 172.21.15.123:0/1122765406 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b8c1037c0 0x7f6b8c070800 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:14.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:14.597+0000 7f6b91c00700 1 -- 172.21.15.123:0/1122765406 >> 172.21.15.123:0/1122765406 conn(0x7f6b8c0f89d0 msgr2=0x7f6b8c0f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:25:14.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:14.597+0000 7f6b91c00700 1 -- 172.21.15.123:0/1122765406 shutdown_connections 2024-03-30T04:25:14.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:14.597+0000 7f6b91c00700 1 -- 172.21.15.123:0/1122765406 wait complete. 2024-03-30T04:25:14.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:25:14 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/1122765406' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-30T04:25:15.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:25:14 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/1122765406' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-30T04:25:15.403 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-30T04:25:15.403 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-30T04:25:15.403 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi135.front.sepia.ceph.com 2024-03-30T04:25:15.404 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-30T04:25:15.404 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-30T04:25:15.404 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-30T04:25:15.404 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-30T04:25:15.404 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-30T04:25:15.404 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-30T04:25:15.404 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:25:15.404 DEBUG:teuthology.orchestra.run.smithi135:> ip addr 2024-03-30T04:25:15.455 INFO:teuthology.orchestra.run.smithi135.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-30T04:25:15.455 INFO:teuthology.orchestra.run.smithi135.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-30T04:25:15.455 INFO:teuthology.orchestra.run.smithi135.stdout: inet 127.0.0.1/8 scope host lo 2024-03-30T04:25:15.455 INFO:teuthology.orchestra.run.smithi135.stdout: valid_lft forever preferred_lft forever 2024-03-30T04:25:15.455 INFO:teuthology.orchestra.run.smithi135.stdout: inet6 ::1/128 scope host 2024-03-30T04:25:15.456 INFO:teuthology.orchestra.run.smithi135.stdout: valid_lft forever preferred_lft forever 2024-03-30T04:25:15.456 INFO:teuthology.orchestra.run.smithi135.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-30T04:25:15.456 INFO:teuthology.orchestra.run.smithi135.stdout: link/ether 0c:c4:7a:d9:36:76 brd ff:ff:ff:ff:ff:ff 2024-03-30T04:25:15.456 INFO:teuthology.orchestra.run.smithi135.stdout: altname enp7s0f0 2024-03-30T04:25:15.456 INFO:teuthology.orchestra.run.smithi135.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-30T04:25:15.456 INFO:teuthology.orchestra.run.smithi135.stdout: link/ether 0c:c4:7a:d9:36:77 brd ff:ff:ff:ff:ff:ff 2024-03-30T04:25:15.456 INFO:teuthology.orchestra.run.smithi135.stdout: altname enp7s0f1 2024-03-30T04:25:15.456 INFO:teuthology.orchestra.run.smithi135.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-30T04:25:15.456 INFO:teuthology.orchestra.run.smithi135.stdout: link/ether 0c:c4:7a:88:80:82 brd ff:ff:ff:ff:ff:ff 2024-03-30T04:25:15.456 INFO:teuthology.orchestra.run.smithi135.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-30T04:25:15.456 INFO:teuthology.orchestra.run.smithi135.stdout: link/ether 0c:c4:7a:88:80:83 brd ff:ff:ff:ff:ff:ff 2024-03-30T04:25:15.456 INFO:teuthology.orchestra.run.smithi135.stdout: inet 172.21.15.135/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-30T04:25:15.457 INFO:teuthology.orchestra.run.smithi135.stdout: valid_lft 41940sec preferred_lft 41940sec 2024-03-30T04:25:15.457 INFO:teuthology.orchestra.run.smithi135.stdout: inet6 fe80::ec4:7aff:fe88:8083/64 scope link 2024-03-30T04:25:15.457 INFO:teuthology.orchestra.run.smithi135.stdout: valid_lft forever preferred_lft forever 2024-03-30T04:25:15.458 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-30T04:25:15.458 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:25:15.458 DEBUG:teuthology.orchestra.run.smithi135:> (cd / && exec bash -c ' 2024-03-30T04:25:15.458 DEBUG:teuthology.orchestra.run.smithi135:> set -e 2024-03-30T04:25:15.458 DEBUG:teuthology.orchestra.run.smithi135:> sudo ip link add name ceph-brx type bridge 2024-03-30T04:25:15.458 DEBUG:teuthology.orchestra.run.smithi135:> sudo ip addr flush dev ceph-brx 2024-03-30T04:25:15.458 DEBUG:teuthology.orchestra.run.smithi135:> sudo ip link set ceph-brx up 2024-03-30T04:25:15.458 DEBUG:teuthology.orchestra.run.smithi135:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-30T04:25:15.458 DEBUG:teuthology.orchestra.run.smithi135:> ') 2024-03-30T04:25:15.816 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:25:15 smithi135 ceph-mon[44364]: pgmap v223: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 341 B/s wr, 3 op/s 2024-03-30T04:25:15.878 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:25:15.878 DEBUG:teuthology.orchestra.run.smithi135:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-30T04:25:15.945 INFO:teuthology.orchestra.run.smithi135.stdout:1 2024-03-30T04:25:15.948 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:25:15.948 DEBUG:teuthology.orchestra.run.smithi135:> route 2024-03-30T04:25:15.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:25:15 smithi123 ceph-mon[35245]: pgmap v223: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 341 B/s wr, 3 op/s 2024-03-30T04:25:16.017 INFO:teuthology.orchestra.run.smithi135.stdout:Kernel IP routing table 2024-03-30T04:25:16.018 INFO:teuthology.orchestra.run.smithi135.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-30T04:25:16.018 INFO:teuthology.orchestra.run.smithi135.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-30T04:25:16.018 INFO:teuthology.orchestra.run.smithi135.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-30T04:25:16.018 INFO:teuthology.orchestra.run.smithi135.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-30T04:25:16.019 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:25:16.019 DEBUG:teuthology.orchestra.run.smithi135:> (cd / && exec bash -c ' 2024-03-30T04:25:16.019 DEBUG:teuthology.orchestra.run.smithi135:> set -e 2024-03-30T04:25:16.019 DEBUG:teuthology.orchestra.run.smithi135:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-30T04:25:16.019 DEBUG:teuthology.orchestra.run.smithi135:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-30T04:25:16.019 DEBUG:teuthology.orchestra.run.smithi135:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-30T04:25:16.019 DEBUG:teuthology.orchestra.run.smithi135:> ') 2024-03-30T04:25:16.710 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:25:16.710 DEBUG:teuthology.orchestra.run.smithi135:> ip netns list 2024-03-30T04:25:16.756 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:25:16.757 DEBUG:teuthology.orchestra.run.smithi135:> ip netns list-id 2024-03-30T04:25:16.842 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:25:16.842 DEBUG:teuthology.orchestra.run.smithi135:> (cd / && exec bash -c ' 2024-03-30T04:25:16.842 DEBUG:teuthology.orchestra.run.smithi135:> set -e 2024-03-30T04:25:16.842 DEBUG:teuthology.orchestra.run.smithi135:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-30T04:25:16.842 DEBUG:teuthology.orchestra.run.smithi135:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-03-30T04:25:16.842 DEBUG:teuthology.orchestra.run.smithi135:> ') 2024-03-30T04:25:17.021 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-03-30T04:25:17.021 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:25:17.021 DEBUG:teuthology.orchestra.run.smithi135:> (cd / && exec bash -c ' 2024-03-30T04:25:17.021 DEBUG:teuthology.orchestra.run.smithi135:> set -e 2024-03-30T04:25:17.021 DEBUG:teuthology.orchestra.run.smithi135:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-03-30T04:25:17.021 DEBUG:teuthology.orchestra.run.smithi135:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-30T04:25:17.021 DEBUG:teuthology.orchestra.run.smithi135:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-03-30T04:25:17.021 DEBUG:teuthology.orchestra.run.smithi135:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-03-30T04:25:17.022 DEBUG:teuthology.orchestra.run.smithi135:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-03-30T04:25:17.022 DEBUG:teuthology.orchestra.run.smithi135:> ') 2024-03-30T04:25:17.280 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:25:17.280 DEBUG:teuthology.orchestra.run.smithi135:> (cd / && exec bash -c ' 2024-03-30T04:25:17.280 DEBUG:teuthology.orchestra.run.smithi135:> set -e 2024-03-30T04:25:17.280 DEBUG:teuthology.orchestra.run.smithi135:> sudo ip link set brx.0 up 2024-03-30T04:25:17.280 DEBUG:teuthology.orchestra.run.smithi135:> sudo ip link set dev brx.0 master ceph-brx 2024-03-30T04:25:17.280 DEBUG:teuthology.orchestra.run.smithi135:> ') 2024-03-30T04:25:17.445 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-03-30T04:25:17.446 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-30T04:25:17.446 DEBUG:teuthology.orchestra.run.smithi135:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-30T04:25:17.490 INFO:teuthology.orchestra.run.smithi135.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-30T04:25:17.490 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-30T04:25:17.491 DEBUG:teuthology.orchestra.run.smithi135:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-30T04:25:17.574 DEBUG:teuthology.orchestra.run.smithi135:> sudo modprobe fuse 2024-03-30T04:25:17.691 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-30T04:25:18.531 DEBUG:teuthology.orchestra.run.smithi135:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-30T04:25:18.532 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:25:17 smithi123 ceph-mon[35245]: pgmap v224: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-30T04:25:18.532 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:25:17 smithi135 ceph-mon[44364]: pgmap v224: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-30T04:25:18.598 INFO:teuthology.orchestra.run.smithi135.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-30T04:25:18.601 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-30T04:25:18.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:25:18.602 DEBUG:teuthology.orchestra.run.smithi135:> ls /sys/fs/fuse/connections 2024-03-30T04:25:18.647 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-30T04:25:18.647 DEBUG:teuthology.orchestra.run.smithi135:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-03-30T04:25:18.696 DEBUG:teuthology.orchestra.run.smithi135:> sudo modprobe fuse 2024-03-30T04:25:18.770 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-30T04:25:18.770 DEBUG:teuthology.orchestra.run.smithi135:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-30T04:25:18.836 INFO:teuthology.orchestra.run.smithi135.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-30T04:25:18.839 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-30T04:25:18.839 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:25:18.840 DEBUG:teuthology.orchestra.run.smithi135:> ls /sys/fs/fuse/connections 2024-03-30T04:25:18.894 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi135.stderr:2024-03-30T04:25:18.896+0000 7f188023e540 -1 init, newargv = 0x55ee57b3b2d0 newargc=15 2024-03-30T04:25:18.895 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi135.stderr:ceph-fuse[75179]: starting ceph client 2024-03-30T04:25:18.906 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi135.stderr:ceph-fuse[75179]: starting fuse 2024-03-30T04:25:19.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:25:19 smithi123 ceph-mon[35245]: pgmap v225: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 341 B/s wr, 3 op/s 2024-03-30T04:25:19.985 DEBUG:teuthology.orchestra.run.smithi135:> sudo modprobe fuse 2024-03-30T04:25:20.055 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:25:19 smithi135 ceph-mon[44364]: pgmap v225: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 341 B/s wr, 3 op/s 2024-03-30T04:25:20.064 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-30T04:25:20.064 DEBUG:teuthology.orchestra.run.smithi135:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-30T04:25:20.131 INFO:teuthology.orchestra.run.smithi135.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-30T04:25:20.134 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-30T04:25:20.134 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:25:20.135 DEBUG:teuthology.orchestra.run.smithi135:> ls /sys/fs/fuse/connections 2024-03-30T04:25:20.180 INFO:teuthology.orchestra.run.smithi135.stdout:123 2024-03-30T04:25:20.180 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [123] 2024-03-30T04:25:20.181 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-03-30T04:25:20.181 DEBUG:teuthology.orchestra.run.smithi135:> import glob 2024-03-30T04:25:20.181 DEBUG:teuthology.orchestra.run.smithi135:> import re 2024-03-30T04:25:20.181 DEBUG:teuthology.orchestra.run.smithi135:> import os 2024-03-30T04:25:20.181 DEBUG:teuthology.orchestra.run.smithi135:> import subprocess 2024-03-30T04:25:20.181 DEBUG:teuthology.orchestra.run.smithi135:> 2024-03-30T04:25:20.181 DEBUG:teuthology.orchestra.run.smithi135:> def _find_admin_socket(client_name): 2024-03-30T04:25:20.181 DEBUG:teuthology.orchestra.run.smithi135:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-03-30T04:25:20.181 DEBUG:teuthology.orchestra.run.smithi135:> files = glob.glob(asok_path) 2024-03-30T04:25:20.181 DEBUG:teuthology.orchestra.run.smithi135:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-03-30T04:25:20.181 DEBUG:teuthology.orchestra.run.smithi135:> 2024-03-30T04:25:20.181 DEBUG:teuthology.orchestra.run.smithi135:> # Given a non-glob path, it better be there 2024-03-30T04:25:20.181 DEBUG:teuthology.orchestra.run.smithi135:> if "*" not in asok_path: 2024-03-30T04:25:20.182 DEBUG:teuthology.orchestra.run.smithi135:> assert(len(files) == 1) 2024-03-30T04:25:20.182 DEBUG:teuthology.orchestra.run.smithi135:> return files[0] 2024-03-30T04:25:20.182 DEBUG:teuthology.orchestra.run.smithi135:> 2024-03-30T04:25:20.182 DEBUG:teuthology.orchestra.run.smithi135:> for f in files: 2024-03-30T04:25:20.182 DEBUG:teuthology.orchestra.run.smithi135:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-30T04:25:20.182 DEBUG:teuthology.orchestra.run.smithi135:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-30T04:25:20.182 DEBUG:teuthology.orchestra.run.smithi135:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-30T04:25:20.182 DEBUG:teuthology.orchestra.run.smithi135:> contents = proc_f.read() 2024-03-30T04:25:20.182 DEBUG:teuthology.orchestra.run.smithi135:> if mountpoint in contents: 2024-03-30T04:25:20.182 DEBUG:teuthology.orchestra.run.smithi135:> return f 2024-03-30T04:25:20.182 DEBUG:teuthology.orchestra.run.smithi135:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-30T04:25:20.182 DEBUG:teuthology.orchestra.run.smithi135:> 2024-03-30T04:25:20.182 DEBUG:teuthology.orchestra.run.smithi135:> print(_find_admin_socket("client.1")) 2024-03-30T04:25:20.183 DEBUG:teuthology.orchestra.run.smithi135:> ' 2024-03-30T04:25:20.361 INFO:teuthology.orchestra.run.smithi135.stdout:/var/run/ceph/ceph-client.1.75179.asok 2024-03-30T04:25:20.539 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.75179.asok 2024-03-30T04:25:20.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:25:20.539 DEBUG:teuthology.orchestra.run.smithi135:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.75179.asok status 2024-03-30T04:25:20.756 INFO:teuthology.orchestra.run.smithi135.stdout:{ 2024-03-30T04:25:20.756 INFO:teuthology.orchestra.run.smithi135.stdout: "metadata": { 2024-03-30T04:25:20.756 INFO:teuthology.orchestra.run.smithi135.stdout: "ceph_sha1": "5c02433542ab1af3975314808d208cabd22abc39", 2024-03-30T04:25:20.756 INFO:teuthology.orchestra.run.smithi135.stdout: "ceph_version": "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)", 2024-03-30T04:25:20.756 INFO:teuthology.orchestra.run.smithi135.stdout: "entity_id": "1", 2024-03-30T04:25:20.756 INFO:teuthology.orchestra.run.smithi135.stdout: "hostname": "smithi135", 2024-03-30T04:25:20.756 INFO:teuthology.orchestra.run.smithi135.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-03-30T04:25:20.756 INFO:teuthology.orchestra.run.smithi135.stdout: "pid": "75179", 2024-03-30T04:25:20.756 INFO:teuthology.orchestra.run.smithi135.stdout: "root": "/" 2024-03-30T04:25:20.756 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-03-30T04:25:20.756 INFO:teuthology.orchestra.run.smithi135.stdout: "dentry_count": 0, 2024-03-30T04:25:20.757 INFO:teuthology.orchestra.run.smithi135.stdout: "dentry_pinned_count": 0, 2024-03-30T04:25:20.757 INFO:teuthology.orchestra.run.smithi135.stdout: "id": 24369, 2024-03-30T04:25:20.757 INFO:teuthology.orchestra.run.smithi135.stdout: "inst": { 2024-03-30T04:25:20.757 INFO:teuthology.orchestra.run.smithi135.stdout: "name": { 2024-03-30T04:25:20.757 INFO:teuthology.orchestra.run.smithi135.stdout: "type": "client", 2024-03-30T04:25:20.757 INFO:teuthology.orchestra.run.smithi135.stdout: "num": 24369 2024-03-30T04:25:20.757 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-03-30T04:25:20.757 INFO:teuthology.orchestra.run.smithi135.stdout: "addr": { 2024-03-30T04:25:20.757 INFO:teuthology.orchestra.run.smithi135.stdout: "type": "v1", 2024-03-30T04:25:20.757 INFO:teuthology.orchestra.run.smithi135.stdout: "addr": "192.168.0.1:0", 2024-03-30T04:25:20.757 INFO:teuthology.orchestra.run.smithi135.stdout: "nonce": 585251847 2024-03-30T04:25:20.757 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-03-30T04:25:20.757 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-03-30T04:25:20.757 INFO:teuthology.orchestra.run.smithi135.stdout: "addr": { 2024-03-30T04:25:20.757 INFO:teuthology.orchestra.run.smithi135.stdout: "type": "v1", 2024-03-30T04:25:20.758 INFO:teuthology.orchestra.run.smithi135.stdout: "addr": "192.168.0.1:0", 2024-03-30T04:25:20.758 INFO:teuthology.orchestra.run.smithi135.stdout: "nonce": 585251847 2024-03-30T04:25:20.758 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-03-30T04:25:20.758 INFO:teuthology.orchestra.run.smithi135.stdout: "inst_str": "client.24369 192.168.0.1:0/585251847", 2024-03-30T04:25:20.758 INFO:teuthology.orchestra.run.smithi135.stdout: "addr_str": "192.168.0.1:0/585251847", 2024-03-30T04:25:20.758 INFO:teuthology.orchestra.run.smithi135.stdout: "inode_count": 1, 2024-03-30T04:25:20.758 INFO:teuthology.orchestra.run.smithi135.stdout: "mds_epoch": 19, 2024-03-30T04:25:20.758 INFO:teuthology.orchestra.run.smithi135.stdout: "osd_epoch": 46, 2024-03-30T04:25:20.758 INFO:teuthology.orchestra.run.smithi135.stdout: "osd_epoch_barrier": 0, 2024-03-30T04:25:20.758 INFO:teuthology.orchestra.run.smithi135.stdout: "blocklisted": false, 2024-03-30T04:25:20.758 INFO:teuthology.orchestra.run.smithi135.stdout: "fs_name": "cephfs" 2024-03-30T04:25:20.758 INFO:teuthology.orchestra.run.smithi135.stdout:} 2024-03-30T04:25:20.770 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:25:20.770 DEBUG:teuthology.orchestra.run.smithi123:> stat --file-system '--printf=%T 2024-03-30T04:25:20.771 DEBUG:teuthology.orchestra.run.smithi123:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-30T04:25:20.826 INFO:teuthology.orchestra.run.smithi123.stdout:fuseblk 2024-03-30T04:25:20.826 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-30T04:25:20.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:25:20.826 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-30T04:25:20.911 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:25:20.912 DEBUG:teuthology.orchestra.run.smithi135:> stat --file-system '--printf=%T 2024-03-30T04:25:20.912 DEBUG:teuthology.orchestra.run.smithi135:> ' -- /home/ubuntu/cephtest/mnt.1 2024-03-30T04:25:20.957 INFO:teuthology.orchestra.run.smithi135.stdout:fuseblk 2024-03-30T04:25:20.957 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-03-30T04:25:20.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:25:20.957 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-30T04:25:21.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:25:21 smithi123 ceph-mon[35245]: pgmap v226: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s rd, 426 B/s wr, 4 op/s 2024-03-30T04:25:22.060 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:25:21 smithi135 ceph-mon[44364]: pgmap v226: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s rd, 426 B/s wr, 4 op/s 2024-03-30T04:25:22.724 INFO:teuthology.run_tasks:Running task print... 2024-03-30T04:25:22.735 INFO:teuthology.task.print:**** done client 2024-03-30T04:25:22.735 INFO:teuthology.run_tasks:Running task parallel... 2024-03-30T04:25:22.745 INFO:teuthology.task.parallel:starting parallel... 2024-03-30T04:25:22.746 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-03-30T04:25:22.746 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-03-30T04:25:22.747 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi123.front.sepia.ceph.com 2024-03-30T04:25:22.747 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -e sha1=470100a4ccbcf7769052bf09a5413881ccf1b97f -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-03-30T04:25:22.747 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-03-30T04:25:22.747 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-03-30T04:25:22.751 INFO:tasks.workunit:Pulling workunits from ref 470100a4ccbcf7769052bf09a5413881ccf1b97f 2024-03-30T04:25:22.751 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-30T04:25:22.751 INFO:tasks.workunit:timeout=3h 2024-03-30T04:25:22.751 INFO:tasks.workunit:cleanup=True 2024-03-30T04:25:22.751 DEBUG:teuthology.orchestra.run.smithi123:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-30T04:25:22.809 INFO:teuthology.orchestra.run.smithi123.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-30T04:25:22.809 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-03-30T04:25:22.809 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 93h/147d Inode: 1 Links: 2 2024-03-30T04:25:22.810 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T04:25:22.810 INFO:teuthology.orchestra.run.smithi123.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-30T04:25:22.810 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-03-30T04:25:22.810 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2024-03-30 04:24:35.247023979 +0000 2024-03-30T04:25:22.810 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2024-03-30 04:25:20.908427876 +0000 2024-03-30T04:25:22.810 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: - 2024-03-30T04:25:22.810 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-30T04:25:22.810 DEBUG:teuthology.orchestra.run.smithi123:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-30T04:25:22.936 DEBUG:teuthology.orchestra.run.smithi135:> stat -- /home/ubuntu/cephtest/mnt.1 2024-03-30T04:25:22.989 INFO:teuthology.orchestra.run.smithi135.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-03-30T04:25:22.989 INFO:teuthology.orchestra.run.smithi135.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-03-30T04:25:22.989 INFO:teuthology.orchestra.run.smithi135.stdout:Device: 7bh/123d Inode: 1 Links: 3 2024-03-30T04:25:22.989 INFO:teuthology.orchestra.run.smithi135.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T04:25:22.989 INFO:teuthology.orchestra.run.smithi135.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-30T04:25:22.989 INFO:teuthology.orchestra.run.smithi135.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-03-30T04:25:22.989 INFO:teuthology.orchestra.run.smithi135.stdout:Modify: 2024-03-30 04:25:22.929458554 +0000 2024-03-30T04:25:22.989 INFO:teuthology.orchestra.run.smithi135.stdout:Change: 2024-03-30 04:25:22.929458554 +0000 2024-03-30T04:25:22.989 INFO:teuthology.orchestra.run.smithi135.stdout: Birth: - 2024-03-30T04:25:22.989 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-03-30T04:25:22.990 DEBUG:teuthology.orchestra.run.smithi135:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-03-30T04:25:23.067 DEBUG:teuthology.orchestra.run.smithi123:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 470100a4ccbcf7769052bf09a5413881ccf1b97f 2024-03-30T04:25:23.068 DEBUG:teuthology.orchestra.run.smithi135:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout 470100a4ccbcf7769052bf09a5413881ccf1b97f 2024-03-30T04:25:23.114 INFO:tasks.workunit.client.1.smithi135.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-03-30T04:25:23.118 INFO:tasks.workunit.client.0.smithi123.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-30T04:25:23.213 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:25:23.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:25:23 smithi123 ceph-mon[35245]: pgmap v227: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 341 B/s wr, 4 op/s 2024-03-30T04:25:24.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:25:23 smithi135 ceph-mon[44364]: pgmap v227: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 341 B/s wr, 4 op/s 2024-03-30T04:25:25.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:25.244+0000 7f650d200700 1 -- 172.21.15.123:0/1429829122 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65080709a0 msgr2=0x7f6508070e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:25:25.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:25.244+0000 7f650d200700 1 --2- 172.21.15.123:0/1429829122 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65080709a0 0x7f6508070e10 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f64f4004940 tx=0x7f64f4009e30 comp rx=0 tx=0).stop 2024-03-30T04:25:25.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:25.245+0000 7f650d200700 1 -- 172.21.15.123:0/1429829122 shutdown_connections 2024-03-30T04:25:25.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:25.245+0000 7f650d200700 1 --2- 172.21.15.123:0/1429829122 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65080709a0 0x7f6508070e10 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:25.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:25.245+0000 7f650d200700 1 --2- 172.21.15.123:0/1429829122 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6508071f50 0x7f6508070460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:25.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:25.245+0000 7f650d200700 1 -- 172.21.15.123:0/1429829122 >> 172.21.15.123:0/1429829122 conn(0x7f65080f8560 msgr2=0x7f65080fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:25:25.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:25.245+0000 7f650d200700 1 -- 172.21.15.123:0/1429829122 shutdown_connections 2024-03-30T04:25:25.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:25.245+0000 7f650d200700 1 -- 172.21.15.123:0/1429829122 wait complete. 2024-03-30T04:25:25.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:25.245+0000 7f650d200700 1 Processor -- start 2024-03-30T04:25:25.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:25.246+0000 7f650d200700 1 -- start start 2024-03-30T04:25:25.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:25.246+0000 7f650d200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65080709a0 0x7f6508113f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:25:25.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:25.246+0000 7f650d200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6508071f50 0x7f6508114480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:25:25.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:25.246+0000 7f650d200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6508114a80 con 0x7f65080709a0 2024-03-30T04:25:25.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:25.246+0000 7f650d200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65081152d0 con 0x7f6508071f50 2024-03-30T04:25:25.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:25.246+0000 7f6505e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65080709a0 0x7f6508113f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:25:25.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:25.246+0000 7f6505e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65080709a0 0x7f6508113f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56210/0 (socket says 172.21.15.123:56210) 2024-03-30T04:25:25.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:25.246+0000 7f6505e00700 1 -- 172.21.15.123:0/216880671 learned_addr learned my addr 172.21.15.123:0/216880671 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:25:25.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:25.246+0000 7f6505400700 1 --2- 172.21.15.123:0/216880671 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6508071f50 0x7f6508114480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:25:25.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:25.246+0000 7f6505e00700 1 -- 172.21.15.123:0/216880671 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6508071f50 msgr2=0x7f6508114480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:25:25.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:25.247+0000 7f6505e00700 1 --2- 172.21.15.123:0/216880671 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6508071f50 0x7f6508114480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:25.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:25.247+0000 7f6505e00700 1 -- 172.21.15.123:0/216880671 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64f4005040 con 0x7f65080709a0 2024-03-30T04:25:25.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:25.247+0000 7f6505400700 1 --2- 172.21.15.123:0/216880671 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6508071f50 0x7f6508114480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:25:25.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:25.247+0000 7f6505e00700 1 --2- 172.21.15.123:0/216880671 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65080709a0 0x7f6508113f40 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f64f800c8e0 tx=0x7f64f80044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:25:25.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:25.247+0000 7f64fe000700 1 -- 172.21.15.123:0/216880671 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f64f800a940 con 0x7f65080709a0 2024-03-30T04:25:25.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:25.247+0000 7f64fe000700 1 -- 172.21.15.123:0/216880671 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f64f800ce80 con 0x7f65080709a0 2024-03-30T04:25:25.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:25.247+0000 7f64fe000700 1 -- 172.21.15.123:0/216880671 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f64f80113a0 con 0x7f65080709a0 2024-03-30T04:25:25.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:25.247+0000 7f650d200700 1 -- 172.21.15.123:0/216880671 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65081155b0 con 0x7f65080709a0 2024-03-30T04:25:25.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:25.247+0000 7f650d200700 1 -- 172.21.15.123:0/216880671 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6508115af0 con 0x7f65080709a0 2024-03-30T04:25:25.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:25.248+0000 7f650d200700 1 -- 172.21.15.123:0/216880671 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f650805f5f0 con 0x7f65080709a0 2024-03-30T04:25:25.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:25.249+0000 7f64fe000700 1 -- 172.21.15.123:0/216880671 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f64f8012400 con 0x7f65080709a0 2024-03-30T04:25:25.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:25.250+0000 7f64fe000700 1 --2- 172.21.15.123:0/216880671 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f64ec0692b0 0x7f64ec06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:25:25.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:25.250+0000 7f64fe000700 1 -- 172.21.15.123:0/216880671 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f64f80872b0 con 0x7f65080709a0 2024-03-30T04:25:25.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:25.250+0000 7f6505400700 1 --2- 172.21.15.123:0/216880671 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f64ec0692b0 0x7f64ec06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:25:25.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:25.251+0000 7f6505400700 1 --2- 172.21.15.123:0/216880671 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f64ec0692b0 0x7f64ec06b760 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f64f400b3b0 tx=0x7f64f4009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:25:25.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:25.254+0000 7f64fe000700 1 -- 172.21.15.123:0/216880671 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f64f8057000 con 0x7f65080709a0 2024-03-30T04:25:25.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:25.448+0000 7f650d200700 1 -- 172.21.15.123:0/216880671 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7f650804bed0 con 0x7f65080709a0 2024-03-30T04:25:25.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:25.449+0000 7f64fe000700 1 -- 172.21.15.123:0/216880671 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v21)=0 v21) v1 ==== 155+0+0 (secure 0 0 0) 0x7f64f8056b90 con 0x7f65080709a0 2024-03-30T04:25:25.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:25.451+0000 7f650d200700 1 -- 172.21.15.123:0/216880671 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f64ec0692b0 msgr2=0x7f64ec06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:25:25.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:25.451+0000 7f650d200700 1 --2- 172.21.15.123:0/216880671 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f64ec0692b0 0x7f64ec06b760 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f64f400b3b0 tx=0x7f64f4009e30 comp rx=0 tx=0).stop 2024-03-30T04:25:25.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:25.451+0000 7f650d200700 1 -- 172.21.15.123:0/216880671 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65080709a0 msgr2=0x7f6508113f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:25:25.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:25.451+0000 7f650d200700 1 --2- 172.21.15.123:0/216880671 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65080709a0 0x7f6508113f40 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f64f800c8e0 tx=0x7f64f80044c0 comp rx=0 tx=0).stop 2024-03-30T04:25:25.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:25.451+0000 7f650d200700 1 -- 172.21.15.123:0/216880671 shutdown_connections 2024-03-30T04:25:25.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:25.451+0000 7f650d200700 1 --2- 172.21.15.123:0/216880671 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f64ec0692b0 0x7f64ec06b760 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:25.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:25.451+0000 7f650d200700 1 --2- 172.21.15.123:0/216880671 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65080709a0 0x7f6508113f40 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:25.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:25.451+0000 7f650d200700 1 --2- 172.21.15.123:0/216880671 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6508071f50 0x7f6508114480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:25.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:25.451+0000 7f650d200700 1 -- 172.21.15.123:0/216880671 >> 172.21.15.123:0/216880671 conn(0x7f65080f8560 msgr2=0x7f65080f9ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:25:25.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:25.451+0000 7f650d200700 1 -- 172.21.15.123:0/216880671 shutdown_connections 2024-03-30T04:25:25.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:25.451+0000 7f650d200700 1 -- 172.21.15.123:0/216880671 wait complete. 2024-03-30T04:25:25.787 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:25:25 smithi123 ceph-mon[35245]: pgmap v228: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-30T04:25:25.957 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:25:25 smithi135 ceph-mon[44364]: pgmap v228: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 170 B/s wr, 3 op/s 2024-03-30T04:25:26.289 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -e sha1=470100a4ccbcf7769052bf09a5413881ccf1b97f -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-03-30T04:25:26.718 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:25:26.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:25:26 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:25:27.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:25:26 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:25:27.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:25:27 smithi123 ceph-mon[35245]: pgmap v229: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.2 KiB/s rd, 767 B/s wr, 6 op/s 2024-03-30T04:25:28.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:25:27 smithi135 ceph-mon[44364]: pgmap v229: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.2 KiB/s rd, 767 B/s wr, 6 op/s 2024-03-30T04:25:28.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:28.718+0000 7f42e3a00700 1 -- 172.21.15.123:0/1499087720 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42dc0662e0 msgr2=0x7f42dc101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:25:28.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:28.718+0000 7f42e3a00700 1 --2- 172.21.15.123:0/1499087720 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42dc0662e0 0x7f42dc101dc0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f42d0004940 tx=0x7f42d0009e30 comp rx=0 tx=0).stop 2024-03-30T04:25:28.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:28.718+0000 7f42e3a00700 1 -- 172.21.15.123:0/1499087720 shutdown_connections 2024-03-30T04:25:28.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:28.718+0000 7f42e3a00700 1 --2- 172.21.15.123:0/1499087720 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f42dc102300 0x7f42dc1046e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:28.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:28.718+0000 7f42e3a00700 1 --2- 172.21.15.123:0/1499087720 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42dc0662e0 0x7f42dc101dc0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:28.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:28.718+0000 7f42e3a00700 1 -- 172.21.15.123:0/1499087720 >> 172.21.15.123:0/1499087720 conn(0x7f42dc0f7430 msgr2=0x7f42dc0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:25:28.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:28.718+0000 7f42e3a00700 1 -- 172.21.15.123:0/1499087720 shutdown_connections 2024-03-30T04:25:28.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:28.718+0000 7f42e3a00700 1 -- 172.21.15.123:0/1499087720 wait complete. 2024-03-30T04:25:28.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:28.719+0000 7f42e3a00700 1 Processor -- start 2024-03-30T04:25:28.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:28.719+0000 7f42e3a00700 1 -- start start 2024-03-30T04:25:28.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:28.719+0000 7f42e3a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f42dc0662e0 0x7f42dc10fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:25:28.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:28.719+0000 7f42e3a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42dc102300 0x7f42dc1100a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:25:28.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:28.719+0000 7f42e3a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42dc1106a0 con 0x7f42dc102300 2024-03-30T04:25:28.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:28.719+0000 7f42e3a00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42dc110ef0 con 0x7f42dc0662e0 2024-03-30T04:25:28.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:28.719+0000 7f42dbe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42dc102300 0x7f42dc1100a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:25:28.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:28.720+0000 7f42dbe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42dc102300 0x7f42dc1100a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56224/0 (socket says 172.21.15.123:56224) 2024-03-30T04:25:28.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:28.720+0000 7f42dbe00700 1 -- 172.21.15.123:0/767852562 learned_addr learned my addr 172.21.15.123:0/767852562 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:25:28.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:28.720+0000 7f42e1000700 1 --2- 172.21.15.123:0/767852562 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f42dc0662e0 0x7f42dc10fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:25:28.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:28.720+0000 7f42dbe00700 1 -- 172.21.15.123:0/767852562 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f42dc0662e0 msgr2=0x7f42dc10fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:25:28.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:28.720+0000 7f42dbe00700 1 --2- 172.21.15.123:0/767852562 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f42dc0662e0 0x7f42dc10fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:28.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:28.720+0000 7f42dbe00700 1 -- 172.21.15.123:0/767852562 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42d0005040 con 0x7f42dc102300 2024-03-30T04:25:28.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:28.720+0000 7f42dbe00700 1 --2- 172.21.15.123:0/767852562 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42dc102300 0x7f42dc1100a0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f42c8002a50 tx=0x7f42c80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:25:28.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:28.720+0000 7f42d8c00700 1 -- 172.21.15.123:0/767852562 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f42c8007cc0 con 0x7f42dc102300 2024-03-30T04:25:28.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:28.720+0000 7f42e3a00700 1 -- 172.21.15.123:0/767852562 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42dc1111d0 con 0x7f42dc102300 2024-03-30T04:25:28.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:28.720+0000 7f42d8c00700 1 -- 172.21.15.123:0/767852562 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f42c800f040 con 0x7f42dc102300 2024-03-30T04:25:28.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:28.720+0000 7f42d8c00700 1 -- 172.21.15.123:0/767852562 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f42c800d9c0 con 0x7f42dc102300 2024-03-30T04:25:28.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:28.721+0000 7f42e3a00700 1 -- 172.21.15.123:0/767852562 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42dc111710 con 0x7f42dc102300 2024-03-30T04:25:28.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:28.722+0000 7f42e3a00700 1 -- 172.21.15.123:0/767852562 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42dc05f5f0 con 0x7f42dc102300 2024-03-30T04:25:28.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:28.722+0000 7f42d8c00700 1 -- 172.21.15.123:0/767852562 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f42c800db20 con 0x7f42dc102300 2024-03-30T04:25:28.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:28.724+0000 7f42d8c00700 1 --2- 172.21.15.123:0/767852562 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f42c40692a0 0x7f42c406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:25:28.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:28.724+0000 7f42d8c00700 1 -- 172.21.15.123:0/767852562 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f42c8086fa0 con 0x7f42dc102300 2024-03-30T04:25:28.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:28.724+0000 7f42e1000700 1 --2- 172.21.15.123:0/767852562 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f42c40692a0 0x7f42c406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:25:28.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:28.724+0000 7f42e1000700 1 --2- 172.21.15.123:0/767852562 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f42c40692a0 0x7f42c406b750 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f42d000b900 tx=0x7f42d0009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:25:28.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:28.728+0000 7f42d8c00700 1 -- 172.21.15.123:0/767852562 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f42c8056bf0 con 0x7f42dc102300 2024-03-30T04:25:28.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:28.925+0000 7f42e3a00700 1 -- 172.21.15.123:0/767852562 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7f42dc1119d0 con 0x7f42dc102300 2024-03-30T04:25:28.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:28.925+0000 7f42d8c00700 1 -- 172.21.15.123:0/767852562 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v21)=0 v21) v1 ==== 163+0+0 (secure 0 0 0) 0x7f42c8056780 con 0x7f42dc102300 2024-03-30T04:25:28.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:28.927+0000 7f42e3a00700 1 -- 172.21.15.123:0/767852562 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f42c40692a0 msgr2=0x7f42c406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:25:28.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:28.927+0000 7f42e3a00700 1 --2- 172.21.15.123:0/767852562 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f42c40692a0 0x7f42c406b750 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f42d000b900 tx=0x7f42d0009e30 comp rx=0 tx=0).stop 2024-03-30T04:25:28.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:28.927+0000 7f42e3a00700 1 -- 172.21.15.123:0/767852562 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42dc102300 msgr2=0x7f42dc1100a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:25:28.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:28.927+0000 7f42e3a00700 1 --2- 172.21.15.123:0/767852562 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42dc102300 0x7f42dc1100a0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f42c8002a50 tx=0x7f42c80046b0 comp rx=0 tx=0).stop 2024-03-30T04:25:28.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:28.928+0000 7f42e3a00700 1 -- 172.21.15.123:0/767852562 shutdown_connections 2024-03-30T04:25:28.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:28.928+0000 7f42e3a00700 1 --2- 172.21.15.123:0/767852562 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f42c40692a0 0x7f42c406b750 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:28.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:28.928+0000 7f42e3a00700 1 --2- 172.21.15.123:0/767852562 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f42dc0662e0 0x7f42dc10fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:28.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:28.928+0000 7f42e3a00700 1 --2- 172.21.15.123:0/767852562 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42dc102300 0x7f42dc1100a0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:28.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:28.928+0000 7f42e3a00700 1 -- 172.21.15.123:0/767852562 >> 172.21.15.123:0/767852562 conn(0x7f42dc0f7430 msgr2=0x7f42dc0f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:25:28.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:28.928+0000 7f42e3a00700 1 -- 172.21.15.123:0/767852562 shutdown_connections 2024-03-30T04:25:28.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:28.928+0000 7f42e3a00700 1 -- 172.21.15.123:0/767852562 wait complete. 2024-03-30T04:25:29.711 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -e sha1=470100a4ccbcf7769052bf09a5413881ccf1b97f -- bash -c 'ceph config set global log_to_journald false --force' 2024-03-30T04:25:29.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:25:29 smithi123 ceph-mon[35245]: pgmap v230: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 4 op/s 2024-03-30T04:25:30.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:25:29 smithi135 ceph-mon[44364]: pgmap v230: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 4 op/s 2024-03-30T04:25:30.137 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:25:31.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:25:31 smithi123 ceph-mon[35245]: pgmap v231: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 767 B/s wr, 5 op/s 2024-03-30T04:25:32.058 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:25:31 smithi135 ceph-mon[44364]: pgmap v231: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 767 B/s wr, 5 op/s 2024-03-30T04:25:32.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:32.103+0000 7f33fb000700 1 -- 172.21.15.123:0/1304633320 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33f4071e20 msgr2=0x7f33f4070330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:25:32.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:32.103+0000 7f33fb000700 1 --2- 172.21.15.123:0/1304633320 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33f4071e20 0x7f33f4070330 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f33e4004940 tx=0x7f33e4009e30 comp rx=0 tx=0).stop 2024-03-30T04:25:32.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:32.103+0000 7f33fb000700 1 -- 172.21.15.123:0/1304633320 shutdown_connections 2024-03-30T04:25:32.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:32.103+0000 7f33fb000700 1 --2- 172.21.15.123:0/1304633320 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f33f4070870 0x7f33f4070ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:32.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:32.103+0000 7f33fb000700 1 --2- 172.21.15.123:0/1304633320 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33f4071e20 0x7f33f4070330 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:32.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:32.103+0000 7f33fb000700 1 -- 172.21.15.123:0/1304633320 >> 172.21.15.123:0/1304633320 conn(0x7f33f40f8540 msgr2=0x7f33f40fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:25:32.104 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:32.103+0000 7f33fb000700 1 -- 172.21.15.123:0/1304633320 shutdown_connections 2024-03-30T04:25:32.104 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:32.103+0000 7f33fb000700 1 -- 172.21.15.123:0/1304633320 wait complete. 2024-03-30T04:25:32.104 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:32.104+0000 7f33fb000700 1 Processor -- start 2024-03-30T04:25:32.104 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:32.104+0000 7f33fb000700 1 -- start start 2024-03-30T04:25:32.104 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:32.104+0000 7f33fb000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f33f4070870 0x7f33f410fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:25:32.104 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:32.104+0000 7f33fb000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33f4071e20 0x7f33f41100f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:25:32.104 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:32.104+0000 7f33fb000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33f41106f0 con 0x7f33f4071e20 2024-03-30T04:25:32.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:32.104+0000 7f33fb000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33f4110f40 con 0x7f33f4070870 2024-03-30T04:25:32.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:32.105+0000 7f33f3400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33f4071e20 0x7f33f41100f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:25:32.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:32.105+0000 7f33f3400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33f4071e20 0x7f33f41100f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56244/0 (socket says 172.21.15.123:56244) 2024-03-30T04:25:32.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:32.105+0000 7f33f3400700 1 -- 172.21.15.123:0/2801452526 learned_addr learned my addr 172.21.15.123:0/2801452526 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:25:32.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:32.105+0000 7f33f3e00700 1 --2- 172.21.15.123:0/2801452526 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f33f4070870 0x7f33f410fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:25:32.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:32.105+0000 7f33f3400700 1 -- 172.21.15.123:0/2801452526 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f33f4070870 msgr2=0x7f33f410fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:25:32.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:32.105+0000 7f33f3400700 1 --2- 172.21.15.123:0/2801452526 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f33f4070870 0x7f33f410fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:32.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:32.105+0000 7f33f3400700 1 -- 172.21.15.123:0/2801452526 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33e4005040 con 0x7f33f4071e20 2024-03-30T04:25:32.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:32.105+0000 7f33f3400700 1 --2- 172.21.15.123:0/2801452526 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33f4071e20 0x7f33f41100f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f33e000c8c0 tx=0x7f33e00044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:25:32.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:32.105+0000 7f33ebe00700 1 -- 172.21.15.123:0/2801452526 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f33e000a760 con 0x7f33f4071e20 2024-03-30T04:25:32.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:32.106+0000 7f33ebe00700 1 -- 172.21.15.123:0/2801452526 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f33e000ad70 con 0x7f33f4071e20 2024-03-30T04:25:32.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:32.106+0000 7f33fb000700 1 -- 172.21.15.123:0/2801452526 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33f4111220 con 0x7f33f4071e20 2024-03-30T04:25:32.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:32.106+0000 7f33ebe00700 1 -- 172.21.15.123:0/2801452526 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f33e00113a0 con 0x7f33f4071e20 2024-03-30T04:25:32.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:32.106+0000 7f33fb000700 1 -- 172.21.15.123:0/2801452526 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33f4111760 con 0x7f33f4071e20 2024-03-30T04:25:32.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:32.107+0000 7f33fb000700 1 -- 172.21.15.123:0/2801452526 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33f405f5f0 con 0x7f33f4071e20 2024-03-30T04:25:32.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:32.108+0000 7f33ebe00700 1 -- 172.21.15.123:0/2801452526 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f33e0012400 con 0x7f33f4071e20 2024-03-30T04:25:32.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:32.109+0000 7f33ebe00700 1 --2- 172.21.15.123:0/2801452526 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f33d80692b0 0x7f33d806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:25:32.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:32.109+0000 7f33ebe00700 1 -- 172.21.15.123:0/2801452526 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f33e00872e0 con 0x7f33f4071e20 2024-03-30T04:25:32.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:32.109+0000 7f33f3e00700 1 --2- 172.21.15.123:0/2801452526 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f33d80692b0 0x7f33d806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:25:32.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:32.110+0000 7f33f3e00700 1 --2- 172.21.15.123:0/2801452526 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f33d80692b0 0x7f33d806b760 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f33e4000c00 tx=0x7f33e4009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:25:32.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:32.113+0000 7f33ebe00700 1 -- 172.21.15.123:0/2801452526 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f33e0056fb0 con 0x7f33f4071e20 2024-03-30T04:25:32.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:32.309+0000 7f33fb000700 1 -- 172.21.15.123:0/2801452526 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7f33f404bed0 con 0x7f33f4071e20 2024-03-30T04:25:32.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:32.310+0000 7f33ebe00700 1 -- 172.21.15.123:0/2801452526 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v21)=0 v21) v1 ==== 135+0+0 (secure 0 0 0) 0x7f33e005c020 con 0x7f33f4071e20 2024-03-30T04:25:32.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:32.312+0000 7f33fb000700 1 -- 172.21.15.123:0/2801452526 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f33d80692b0 msgr2=0x7f33d806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:25:32.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:32.312+0000 7f33fb000700 1 --2- 172.21.15.123:0/2801452526 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f33d80692b0 0x7f33d806b760 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f33e4000c00 tx=0x7f33e4009e30 comp rx=0 tx=0).stop 2024-03-30T04:25:32.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:32.312+0000 7f33fb000700 1 -- 172.21.15.123:0/2801452526 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33f4071e20 msgr2=0x7f33f41100f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:25:32.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:32.312+0000 7f33fb000700 1 --2- 172.21.15.123:0/2801452526 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33f4071e20 0x7f33f41100f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f33e000c8c0 tx=0x7f33e00044c0 comp rx=0 tx=0).stop 2024-03-30T04:25:32.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:32.312+0000 7f33fb000700 1 -- 172.21.15.123:0/2801452526 shutdown_connections 2024-03-30T04:25:32.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:32.312+0000 7f33fb000700 1 --2- 172.21.15.123:0/2801452526 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f33d80692b0 0x7f33d806b760 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:32.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:32.312+0000 7f33fb000700 1 --2- 172.21.15.123:0/2801452526 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f33f4070870 0x7f33f410fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:32.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:32.312+0000 7f33fb000700 1 --2- 172.21.15.123:0/2801452526 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33f4071e20 0x7f33f41100f0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:32.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:32.312+0000 7f33fb000700 1 -- 172.21.15.123:0/2801452526 >> 172.21.15.123:0/2801452526 conn(0x7f33f40f8540 msgr2=0x7f33f40fa130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:25:32.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:32.312+0000 7f33fb000700 1 -- 172.21.15.123:0/2801452526 shutdown_connections 2024-03-30T04:25:32.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:32.312+0000 7f33fb000700 1 -- 172.21.15.123:0/2801452526 wait complete. 2024-03-30T04:25:33.082 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -e sha1=470100a4ccbcf7769052bf09a5413881ccf1b97f -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr' 2024-03-30T04:25:33.513 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:25:33.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:25:33 smithi123 ceph-mon[35245]: pgmap v232: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.6 KiB/s rd, 767 B/s wr, 5 op/s 2024-03-30T04:25:34.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:25:33 smithi135 ceph-mon[44364]: pgmap v232: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.6 KiB/s rd, 767 B/s wr, 5 op/s 2024-03-30T04:25:35.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:35.322+0000 7f217ae00700 1 -- 172.21.15.123:0/2893916842 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21740fe0d0 msgr2=0x7f21740fe540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:25:35.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:35.322+0000 7f217ae00700 1 --2- 172.21.15.123:0/2893916842 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21740fe0d0 0x7f21740fe540 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f2160004940 tx=0x7f2160009e30 comp rx=0 tx=0).stop 2024-03-30T04:25:35.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:35.322+0000 7f217ae00700 1 -- 172.21.15.123:0/2893916842 shutdown_connections 2024-03-30T04:25:35.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:35.322+0000 7f217ae00700 1 --2- 172.21.15.123:0/2893916842 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21740fe0d0 0x7f21740fe540 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:35.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:35.322+0000 7f217ae00700 1 --2- 172.21.15.123:0/2893916842 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f21740fcee0 0x7f21740fd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:35.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:35.322+0000 7f217ae00700 1 -- 172.21.15.123:0/2893916842 >> 172.21.15.123:0/2893916842 conn(0x7f21740f8560 msgr2=0x7f21740fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:25:35.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:35.322+0000 7f217ae00700 1 -- 172.21.15.123:0/2893916842 shutdown_connections 2024-03-30T04:25:35.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:35.322+0000 7f217ae00700 1 -- 172.21.15.123:0/2893916842 wait complete. 2024-03-30T04:25:35.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:35.323+0000 7f217ae00700 1 Processor -- start 2024-03-30T04:25:35.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:35.323+0000 7f217ae00700 1 -- start start 2024-03-30T04:25:35.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:35.324+0000 7f217ae00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f21740fcee0 0x7f2174071df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:25:35.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:35.324+0000 7f217ae00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21740fe0d0 0x7f2174070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:25:35.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:35.324+0000 7f217ae00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2174070930 con 0x7f21740fe0d0 2024-03-30T04:25:35.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:35.324+0000 7f217ae00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2174070a70 con 0x7f21740fcee0 2024-03-30T04:25:35.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:35.324+0000 7f2173400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21740fe0d0 0x7f2174070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:25:35.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:35.324+0000 7f2173400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21740fe0d0 0x7f2174070310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:51026/0 (socket says 172.21.15.123:51026) 2024-03-30T04:25:35.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:35.324+0000 7f2173400700 1 -- 172.21.15.123:0/1059535199 learned_addr learned my addr 172.21.15.123:0/1059535199 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:25:35.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:35.324+0000 7f2173e00700 1 --2- 172.21.15.123:0/1059535199 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f21740fcee0 0x7f2174071df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:25:35.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:35.324+0000 7f2173400700 1 -- 172.21.15.123:0/1059535199 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f21740fcee0 msgr2=0x7f2174071df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:25:35.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:35.324+0000 7f2173400700 1 --2- 172.21.15.123:0/1059535199 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f21740fcee0 0x7f2174071df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:25:35.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:35.324+0000 7f2173400700 1 -- 172.21.15.123:0/1059535199 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2160005040 con 0x7f21740fe0d0 2024-03-30T04:25:35.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:35.325+0000 7f2173400700 1 --2- 172.21.15.123:0/1059535199 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21740fe0d0 0x7f2174070310 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f2160009380 tx=0x7f2160009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:25:35.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:35.325+0000 7f216be00700 1 -- 172.21.15.123:0/1059535199 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f216000e040 con 0x7f21740fe0d0 2024-03-30T04:25:35.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:35.325+0000 7f216be00700 1 -- 172.21.15.123:0/1059535199 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2160004290 con 0x7f21740fe0d0 2024-03-30T04:25:35.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:35.325+0000 7f217ae00700 1 -- 172.21.15.123:0/1059535199 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2174070cf0 con 0x7f21740fe0d0 2024-03-30T04:25:35.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:35.325+0000 7f216be00700 1 -- 172.21.15.123:0/1059535199 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f216001c3f0 con 0x7f21740fe0d0 2024-03-30T04:25:35.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:35.325+0000 7f217ae00700 1 -- 172.21.15.123:0/1059535199 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21741159f0 con 0x7f21740fe0d0 2024-03-30T04:25:35.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:35.326+0000 7f217ae00700 1 -- 172.21.15.123:0/1059535199 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f217405f5f0 con 0x7f21740fe0d0 2024-03-30T04:25:35.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:35.327+0000 7f216be00700 1 -- 172.21.15.123:0/1059535199 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f2160013420 con 0x7f21740fe0d0 2024-03-30T04:25:35.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:35.328+0000 7f216be00700 1 --2- 172.21.15.123:0/1059535199 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f21580692b0 0x7f215806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:25:35.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:35.328+0000 7f216be00700 1 -- 172.21.15.123:0/1059535199 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f2160088630 con 0x7f21740fe0d0 2024-03-30T04:25:35.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:35.328+0000 7f2173e00700 1 --2- 172.21.15.123:0/1059535199 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f21580692b0 0x7f215806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:25:35.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:35.329+0000 7f2173e00700 1 --2- 172.21.15.123:0/1059535199 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f21580692b0 0x7f215806b760 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2174071b80 tx=0x7f21640091a0 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:25:35.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:35.333+0000 7f216be00700 1 -- 172.21.15.123:0/1059535199 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f2160054870 con 0x7f21740fe0d0 2024-03-30T04:25:35.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:25:35.544+0000 7f217ae00700 1 -- 172.21.15.123:0/1059535199 --> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f", "daemon_types": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f2174102490 con 0x7f21580692b0 2024-03-30T04:25:35.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:25:35 smithi123 ceph-mon[35245]: pgmap v233: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 682 B/s wr, 4 op/s 2024-03-30T04:25:36.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:25:35 smithi135 ceph-mon[44364]: pgmap v233: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 682 B/s wr, 4 op/s 2024-03-30T04:25:36.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:25:36 smithi123 ceph-mon[35245]: from='client.14564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:25:37.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:25:36 smithi135 ceph-mon[44364]: from='client.14564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:25:37.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:25:37 smithi123 ceph-mon[35245]: pgmap v234: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.2 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-30T04:25:38.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:25:37 smithi135 ceph-mon[44364]: pgmap v234: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.2 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-30T04:25:39.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:25:39 smithi123 ceph-mon[35245]: pgmap v235: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 597 B/s wr, 4 op/s 2024-03-30T04:25:40.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:25:39 smithi135 ceph-mon[44364]: pgmap v235: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 597 B/s wr, 4 op/s 2024-03-30T04:25:41.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:25:41 smithi123 ceph-mon[35245]: pgmap v236: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 597 B/s wr, 4 op/s 2024-03-30T04:25:41.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:25:41 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:25:42.058 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:25:41 smithi135 ceph-mon[44364]: pgmap v236: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 597 B/s wr, 4 op/s 2024-03-30T04:25:42.058 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:25:41 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:25:43.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:25:43 smithi123 ceph-mon[35245]: pgmap v237: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 597 B/s wr, 4 op/s 2024-03-30T04:25:44.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:25:43 smithi135 ceph-mon[44364]: pgmap v237: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 597 B/s wr, 4 op/s 2024-03-30T04:25:45.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:25:45 smithi123 ceph-mon[35245]: pgmap v238: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-30T04:25:46.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:25:45 smithi135 ceph-mon[44364]: pgmap v238: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-30T04:25:48.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:25:47 smithi135 ceph-mon[44364]: pgmap v239: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.8 KiB/s rd, 511 B/s wr, 5 op/s 2024-03-30T04:25:48.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:25:47 smithi123 ceph-mon[35245]: pgmap v239: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.8 KiB/s rd, 511 B/s wr, 5 op/s 2024-03-30T04:25:50.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:25:49 smithi135 ceph-mon[44364]: pgmap v240: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:25:50.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:25:49 smithi123 ceph-mon[35245]: pgmap v240: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:25:52.058 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:25:51 smithi135 ceph-mon[44364]: pgmap v241: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-30T04:25:52.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:25:51 smithi123 ceph-mon[35245]: pgmap v241: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-30T04:25:54.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:25:53 smithi135 ceph-mon[44364]: pgmap v242: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-30T04:25:54.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:25:53 smithi123 ceph-mon[35245]: pgmap v242: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-30T04:25:56.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:25:55 smithi135 ceph-mon[44364]: pgmap v243: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:25:56.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:25:55 smithi123 ceph-mon[35245]: pgmap v243: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:25:57.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:25:56 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:25:57.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:25:56 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:25:58.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:25:57 smithi135 ceph-mon[44364]: pgmap v244: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:25:58.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:25:57 smithi123 ceph-mon[35245]: pgmap v244: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:26:00.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:25:59 smithi135 ceph-mon[44364]: pgmap v245: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:26:00.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:25:59 smithi123 ceph-mon[35245]: pgmap v245: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:26:02.058 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:01 smithi135 ceph-mon[44364]: pgmap v246: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-30T04:26:02.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:01 smithi123 ceph-mon[35245]: pgmap v246: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-30T04:26:04.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:03 smithi135 ceph-mon[44364]: pgmap v247: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-30T04:26:04.129 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:03 smithi123 ceph-mon[35245]: pgmap v247: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-30T04:26:04.890 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:04 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:26:04.891 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:04 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:26:04.891 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:04 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:26:04.891 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:04 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:26:05.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:04 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:26:05.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:04 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:26:05.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:04 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:26:05.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:04 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:26:05.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:05.368+0000 7f216be00700 1 -- 172.21.15.123:0/1059535199 <== mgr.14206 v2:172.21.15.123:6800/1564935593 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7f2174102490 con 0x7f21580692b0 2024-03-30T04:26:05.367 INFO:teuthology.orchestra.run.smithi123.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f 2024-03-30T04:26:05.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:05.371+0000 7f217ae00700 1 -- 172.21.15.123:0/1059535199 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f21580692b0 msgr2=0x7f215806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:26:05.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:05.371+0000 7f217ae00700 1 --2- 172.21.15.123:0/1059535199 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f21580692b0 0x7f215806b760 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2174071b80 tx=0x7f21640091a0 comp rx=0 tx=0).stop 2024-03-30T04:26:05.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:05.371+0000 7f217ae00700 1 -- 172.21.15.123:0/1059535199 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21740fe0d0 msgr2=0x7f2174070310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:26:05.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:05.371+0000 7f217ae00700 1 --2- 172.21.15.123:0/1059535199 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21740fe0d0 0x7f2174070310 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f2160009380 tx=0x7f2160009e30 comp rx=0 tx=0).stop 2024-03-30T04:26:05.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:05.371+0000 7f217ae00700 1 -- 172.21.15.123:0/1059535199 shutdown_connections 2024-03-30T04:26:05.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:05.371+0000 7f217ae00700 1 --2- 172.21.15.123:0/1059535199 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f21580692b0 0x7f215806b760 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:05.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:05.371+0000 7f217ae00700 1 --2- 172.21.15.123:0/1059535199 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f21740fcee0 0x7f2174071df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:05.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:05.371+0000 7f217ae00700 1 --2- 172.21.15.123:0/1059535199 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21740fe0d0 0x7f2174070310 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:05.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:05.371+0000 7f217ae00700 1 -- 172.21.15.123:0/1059535199 >> 172.21.15.123:0/1059535199 conn(0x7f21740f8560 msgr2=0x7f21740f9cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:26:05.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:05.371+0000 7f217ae00700 1 -- 172.21.15.123:0/1059535199 shutdown_connections 2024-03-30T04:26:05.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:05.371+0000 7f217ae00700 1 -- 172.21.15.123:0/1059535199 wait complete. 2024-03-30T04:26:05.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:05 smithi123 ceph-mon[35245]: pgmap v248: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:26:05.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:05 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:26:05.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:05 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:26:05.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:05 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:26:05.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:05 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:26:05.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:05 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:26:06.092 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -e sha1=470100a4ccbcf7769052bf09a5413881ccf1b97f -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-03-30T04:26:06.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:05 smithi135 ceph-mon[44364]: pgmap v248: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:26:06.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:05 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:26:06.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:05 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:26:06.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:05 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:26:06.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:05 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:26:06.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:05 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:26:06.628 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:26:07.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:06 smithi135 ceph-mon[44364]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f 2024-03-30T04:26:07.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:06 smithi135 ceph-mon[44364]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f 2024-03-30T04:26:07.179 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:06 smithi123 ceph-mon[35245]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f 2024-03-30T04:26:07.179 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:06 smithi123 ceph-mon[35245]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f 2024-03-30T04:26:08.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:07 smithi135 ceph-mon[44364]: pgmap v249: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:26:08.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:07 smithi123 ceph-mon[35245]: pgmap v249: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:26:09.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:09.776+0000 7f6ad2400700 1 -- 172.21.15.123:0/4135200280 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6acc0fe0d0 msgr2=0x7f6acc0fe540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:26:09.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:09.776+0000 7f6ad2400700 1 --2- 172.21.15.123:0/4135200280 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6acc0fe0d0 0x7f6acc0fe540 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f6ab8004940 tx=0x7f6ab8009e30 comp rx=0 tx=0).stop 2024-03-30T04:26:09.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:09.776+0000 7f6ad2400700 1 -- 172.21.15.123:0/4135200280 shutdown_connections 2024-03-30T04:26:09.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:09.776+0000 7f6ad2400700 1 --2- 172.21.15.123:0/4135200280 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6acc0fe0d0 0x7f6acc0fe540 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:09.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:09.776+0000 7f6ad2400700 1 --2- 172.21.15.123:0/4135200280 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6acc0fcee0 0x7f6acc0fd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:09.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:09.776+0000 7f6ad2400700 1 -- 172.21.15.123:0/4135200280 >> 172.21.15.123:0/4135200280 conn(0x7f6acc0f8560 msgr2=0x7f6acc0fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:26:09.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:09.776+0000 7f6ad2400700 1 -- 172.21.15.123:0/4135200280 shutdown_connections 2024-03-30T04:26:09.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:09.776+0000 7f6ad2400700 1 -- 172.21.15.123:0/4135200280 wait complete. 2024-03-30T04:26:09.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:09.777+0000 7f6ad2400700 1 Processor -- start 2024-03-30T04:26:09.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:09.777+0000 7f6ad2400700 1 -- start start 2024-03-30T04:26:09.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:09.778+0000 7f6ad2400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6acc0fcee0 0x7f6acc10d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:26:09.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:09.778+0000 7f6ad2400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6acc0fe0d0 0x7f6acc10deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:26:09.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:09.778+0000 7f6ad2400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6acc10e4b0 con 0x7f6acc0fe0d0 2024-03-30T04:26:09.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:09.778+0000 7f6ad2400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6acc10ed00 con 0x7f6acc0fcee0 2024-03-30T04:26:09.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:09.778+0000 7f6acaa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6acc0fe0d0 0x7f6acc10deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:26:09.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:09.778+0000 7f6acb400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6acc0fcee0 0x7f6acc10d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:26:09.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:09.778+0000 7f6acaa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6acc0fe0d0 0x7f6acc10deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:59336/0 (socket says 172.21.15.123:59336) 2024-03-30T04:26:09.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:09.778+0000 7f6acb400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6acc0fcee0 0x7f6acc10d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.123:52442/0 (socket says 172.21.15.123:52442) 2024-03-30T04:26:09.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:09.778+0000 7f6acaa00700 1 -- 172.21.15.123:0/3529551839 learned_addr learned my addr 172.21.15.123:0/3529551839 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:26:09.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:09.779+0000 7f6acb400700 1 -- 172.21.15.123:0/3529551839 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6acc0fe0d0 msgr2=0x7f6acc10deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:26:09.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:09.779+0000 7f6acb400700 1 --2- 172.21.15.123:0/3529551839 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6acc0fe0d0 0x7f6acc10deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:09.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:09.779+0000 7f6acb400700 1 -- 172.21.15.123:0/3529551839 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab8005040 con 0x7f6acc0fcee0 2024-03-30T04:26:09.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:09.779+0000 7f6acaa00700 1 --2- 172.21.15.123:0/3529551839 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6acc0fe0d0 0x7f6acc10deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:26:09.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:09.779+0000 7f6acb400700 1 --2- 172.21.15.123:0/3529551839 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6acc0fcee0 0x7f6acc10d970 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6abc002840 tx=0x7f6abc004570 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:26:09.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:09.779+0000 7f6ac3400700 1 -- 172.21.15.123:0/3529551839 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6abc007cc0 con 0x7f6acc0fcee0 2024-03-30T04:26:09.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:09.779+0000 7f6ad2400700 1 -- 172.21.15.123:0/3529551839 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6acc10efe0 con 0x7f6acc0fcee0 2024-03-30T04:26:09.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:09.779+0000 7f6ac3400700 1 -- 172.21.15.123:0/3529551839 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6abc00f040 con 0x7f6acc0fcee0 2024-03-30T04:26:09.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:09.779+0000 7f6ac3400700 1 -- 172.21.15.123:0/3529551839 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6abc013860 con 0x7f6acc0fcee0 2024-03-30T04:26:09.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:09.779+0000 7f6ad2400700 1 -- 172.21.15.123:0/3529551839 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6acc10f520 con 0x7f6acc0fcee0 2024-03-30T04:26:09.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:09.781+0000 7f6ad2400700 1 -- 172.21.15.123:0/3529551839 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6acc05f5f0 con 0x7f6acc0fcee0 2024-03-30T04:26:09.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:09.782+0000 7f6ac3400700 1 -- 172.21.15.123:0/3529551839 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f6abc01c400 con 0x7f6acc0fcee0 2024-03-30T04:26:09.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:09.782+0000 7f6ac3400700 1 --2- 172.21.15.123:0/3529551839 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f6ab00692a0 0x7f6ab006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:26:09.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:09.783+0000 7f6ac3400700 1 -- 172.21.15.123:0/3529551839 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f6abc086ef0 con 0x7f6acc0fcee0 2024-03-30T04:26:09.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:09.783+0000 7f6acaa00700 1 --2- 172.21.15.123:0/3529551839 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f6ab00692a0 0x7f6ab006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:26:09.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:09.783+0000 7f6acaa00700 1 --2- 172.21.15.123:0/3529551839 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f6ab00692a0 0x7f6ab006b750 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6ab800b3b0 tx=0x7f6ab8009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:26:09.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:09.788+0000 7f6ac3400700 1 -- 172.21.15.123:0/3529551839 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f6abc056b40 con 0x7f6acc0fcee0 2024-03-30T04:26:10.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.007+0000 7f6ad2400700 1 -- 172.21.15.123:0/3529551839 --> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6acc10f8a0 con 0x7f6ab00692a0 2024-03-30T04:26:10.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.008+0000 7f6ac3400700 1 -- 172.21.15.123:0/3529551839 <== mgr.14206 v2:172.21.15.123:6800/1564935593 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f6acc10f8a0 con 0x7f6ab00692a0 2024-03-30T04:26:10.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.010+0000 7f6ad2400700 1 -- 172.21.15.123:0/3529551839 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f6ab00692a0 msgr2=0x7f6ab006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:26:10.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.010+0000 7f6ad2400700 1 --2- 172.21.15.123:0/3529551839 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f6ab00692a0 0x7f6ab006b750 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6ab800b3b0 tx=0x7f6ab8009e30 comp rx=0 tx=0).stop 2024-03-30T04:26:10.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.011+0000 7f6ad2400700 1 -- 172.21.15.123:0/3529551839 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6acc0fcee0 msgr2=0x7f6acc10d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:26:10.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.011+0000 7f6ad2400700 1 --2- 172.21.15.123:0/3529551839 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6acc0fcee0 0x7f6acc10d970 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6abc002840 tx=0x7f6abc004570 comp rx=0 tx=0).stop 2024-03-30T04:26:10.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.011+0000 7f6ad2400700 1 -- 172.21.15.123:0/3529551839 shutdown_connections 2024-03-30T04:26:10.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.011+0000 7f6ad2400700 1 --2- 172.21.15.123:0/3529551839 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f6ab00692a0 0x7f6ab006b750 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:10.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.011+0000 7f6ad2400700 1 --2- 172.21.15.123:0/3529551839 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6acc0fcee0 0x7f6acc10d970 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:10.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.011+0000 7f6ad2400700 1 --2- 172.21.15.123:0/3529551839 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6acc0fe0d0 0x7f6acc10deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:10.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.011+0000 7f6ad2400700 1 -- 172.21.15.123:0/3529551839 >> 172.21.15.123:0/3529551839 conn(0x7f6acc0f8560 msgr2=0x7f6acc0f9d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:26:10.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.011+0000 7f6ad2400700 1 -- 172.21.15.123:0/3529551839 shutdown_connections 2024-03-30T04:26:10.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.011+0000 7f6ad2400700 1 -- 172.21.15.123:0/3529551839 wait complete. 2024-03-30T04:26:10.023 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:26:10.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.152+0000 7f8652800700 1 -- 172.21.15.123:0/1976768564 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f864c0fdff0 msgr2=0x7f864c0fe460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:26:10.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.152+0000 7f8652800700 1 --2- 172.21.15.123:0/1976768564 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f864c0fdff0 0x7f864c0fe460 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f8638004940 tx=0x7f8638009e30 comp rx=0 tx=0).stop 2024-03-30T04:26:10.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.152+0000 7f8652800700 1 -- 172.21.15.123:0/1976768564 shutdown_connections 2024-03-30T04:26:10.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.152+0000 7f8652800700 1 --2- 172.21.15.123:0/1976768564 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f864c0fdff0 0x7f864c0fe460 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:10.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.152+0000 7f8652800700 1 --2- 172.21.15.123:0/1976768564 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f864c0fce80 0x7f864c0fd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:10.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.152+0000 7f8652800700 1 -- 172.21.15.123:0/1976768564 >> 172.21.15.123:0/1976768564 conn(0x7f864c0f8560 msgr2=0x7f864c0fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:26:10.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.152+0000 7f8652800700 1 -- 172.21.15.123:0/1976768564 shutdown_connections 2024-03-30T04:26:10.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.152+0000 7f8652800700 1 -- 172.21.15.123:0/1976768564 wait complete. 2024-03-30T04:26:10.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.153+0000 7f8652800700 1 Processor -- start 2024-03-30T04:26:10.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.153+0000 7f8652800700 1 -- start start 2024-03-30T04:26:10.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.154+0000 7f8652800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f864c0fce80 0x7f864c10faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:26:10.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.154+0000 7f8652800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f864c0fdff0 0x7f864c10ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:26:10.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.154+0000 7f8652800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864c1105e0 con 0x7f864c0fdff0 2024-03-30T04:26:10.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.154+0000 7f8652800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864c110e30 con 0x7f864c0fce80 2024-03-30T04:26:10.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.154+0000 7f864aa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f864c0fdff0 0x7f864c10ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:26:10.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.154+0000 7f864aa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f864c0fdff0 0x7f864c10ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:59364/0 (socket says 172.21.15.123:59364) 2024-03-30T04:26:10.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.154+0000 7f864aa00700 1 -- 172.21.15.123:0/830559709 learned_addr learned my addr 172.21.15.123:0/830559709 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:26:10.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.154+0000 7f864b400700 1 --2- 172.21.15.123:0/830559709 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f864c0fce80 0x7f864c10faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:26:10.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.154+0000 7f864aa00700 1 -- 172.21.15.123:0/830559709 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f864c0fce80 msgr2=0x7f864c10faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:26:10.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.154+0000 7f864aa00700 1 --2- 172.21.15.123:0/830559709 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f864c0fce80 0x7f864c10faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:10.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.154+0000 7f864aa00700 1 -- 172.21.15.123:0/830559709 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8638005040 con 0x7f864c0fdff0 2024-03-30T04:26:10.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.155+0000 7f864aa00700 1 --2- 172.21.15.123:0/830559709 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f864c0fdff0 0x7f864c10ffe0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f86380137b0 tx=0x7f8638009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:26:10.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.155+0000 7f8643400700 1 -- 172.21.15.123:0/830559709 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8638021070 con 0x7f864c0fdff0 2024-03-30T04:26:10.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.155+0000 7f8643400700 1 -- 172.21.15.123:0/830559709 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f863801c400 con 0x7f864c0fdff0 2024-03-30T04:26:10.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.155+0000 7f8643400700 1 -- 172.21.15.123:0/830559709 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f863800e070 con 0x7f864c0fdff0 2024-03-30T04:26:10.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.155+0000 7f8652800700 1 -- 172.21.15.123:0/830559709 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f864c1110b0 con 0x7f864c0fdff0 2024-03-30T04:26:10.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.155+0000 7f8652800700 1 -- 172.21.15.123:0/830559709 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f864c111590 con 0x7f864c0fdff0 2024-03-30T04:26:10.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.156+0000 7f8652800700 1 -- 172.21.15.123:0/830559709 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f864c101980 con 0x7f864c0fdff0 2024-03-30T04:26:10.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.157+0000 7f8643400700 1 -- 172.21.15.123:0/830559709 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f863801c570 con 0x7f864c0fdff0 2024-03-30T04:26:10.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.158+0000 7f8643400700 1 --2- 172.21.15.123:0/830559709 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f86300692b0 0x7f863006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:26:10.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.158+0000 7f8643400700 1 -- 172.21.15.123:0/830559709 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f8638088410 con 0x7f864c0fdff0 2024-03-30T04:26:10.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.158+0000 7f864b400700 1 --2- 172.21.15.123:0/830559709 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f86300692b0 0x7f863006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:26:10.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.159+0000 7f864b400700 1 --2- 172.21.15.123:0/830559709 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f86300692b0 0x7f863006b760 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f863c009b80 tx=0x7f863c00b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:26:10.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.163+0000 7f8643400700 1 -- 172.21.15.123:0/830559709 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f86380580e0 con 0x7f864c0fdff0 2024-03-30T04:26:10.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.387+0000 7f8652800700 1 -- 172.21.15.123:0/830559709 --> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f864c105930 con 0x7f86300692b0 2024-03-30T04:26:10.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.389+0000 7f8643400700 1 -- 172.21.15.123:0/830559709 <== mgr.14206 v2:172.21.15.123:6800/1564935593 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f864c105930 con 0x7f86300692b0 2024-03-30T04:26:10.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.391+0000 7f8652800700 1 -- 172.21.15.123:0/830559709 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f86300692b0 msgr2=0x7f863006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:26:10.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.391+0000 7f8652800700 1 --2- 172.21.15.123:0/830559709 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f86300692b0 0x7f863006b760 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f863c009b80 tx=0x7f863c00b040 comp rx=0 tx=0).stop 2024-03-30T04:26:10.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.391+0000 7f8652800700 1 -- 172.21.15.123:0/830559709 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f864c0fdff0 msgr2=0x7f864c10ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:26:10.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.391+0000 7f8652800700 1 --2- 172.21.15.123:0/830559709 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f864c0fdff0 0x7f864c10ffe0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f86380137b0 tx=0x7f8638009e30 comp rx=0 tx=0).stop 2024-03-30T04:26:10.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.391+0000 7f8652800700 1 -- 172.21.15.123:0/830559709 shutdown_connections 2024-03-30T04:26:10.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.391+0000 7f8652800700 1 --2- 172.21.15.123:0/830559709 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f86300692b0 0x7f863006b760 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:10.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.391+0000 7f8652800700 1 --2- 172.21.15.123:0/830559709 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f864c0fce80 0x7f864c10faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:10.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.391+0000 7f8652800700 1 --2- 172.21.15.123:0/830559709 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f864c0fdff0 0x7f864c10ffe0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:10.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.391+0000 7f8652800700 1 -- 172.21.15.123:0/830559709 >> 172.21.15.123:0/830559709 conn(0x7f864c0f8560 msgr2=0x7f864c0f9cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:26:10.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.391+0000 7f8652800700 1 -- 172.21.15.123:0/830559709 shutdown_connections 2024-03-30T04:26:10.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.391+0000 7f8652800700 1 -- 172.21.15.123:0/830559709 wait complete. 2024-03-30T04:26:10.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:10 smithi123 ceph-mon[35245]: pgmap v250: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:26:10.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:10 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:26:10.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:10 smithi123 ceph-mon[35245]: Upgrade: Target is version 18.2.2-1018-g470100a4 (unknown) 2024-03-30T04:26:10.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:10 smithi123 ceph-mon[35245]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35, digests ['quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35'] 2024-03-30T04:26:10.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:10 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:26:10.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:10 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:26:10.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:10 smithi123 ceph-mon[35245]: Upgrade: Need to upgrade myself (mgr.smithi123.xgksqq) 2024-03-30T04:26:10.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.534+0000 7ff517a00700 1 -- 172.21.15.123:0/908704885 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff5100fdcf0 msgr2=0x7ff510100170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:26:10.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.534+0000 7ff517a00700 1 --2- 172.21.15.123:0/908704885 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff5100fdcf0 0x7ff510100170 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc004940 tx=0x7ff4fc009e30 comp rx=0 tx=0).stop 2024-03-30T04:26:10.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.534+0000 7ff517a00700 1 -- 172.21.15.123:0/908704885 shutdown_connections 2024-03-30T04:26:10.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.534+0000 7ff517a00700 1 --2- 172.21.15.123:0/908704885 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff5100fdcf0 0x7ff510100170 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:10.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.534+0000 7ff517a00700 1 --2- 172.21.15.123:0/908704885 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff5100662e0 0x7ff5100fd7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:10.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.534+0000 7ff517a00700 1 -- 172.21.15.123:0/908704885 >> 172.21.15.123:0/908704885 conn(0x7ff5100f7430 msgr2=0x7ff5100f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:26:10.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.534+0000 7ff517a00700 1 -- 172.21.15.123:0/908704885 shutdown_connections 2024-03-30T04:26:10.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.534+0000 7ff517a00700 1 -- 172.21.15.123:0/908704885 wait complete. 2024-03-30T04:26:10.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.535+0000 7ff517a00700 1 Processor -- start 2024-03-30T04:26:10.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.535+0000 7ff517a00700 1 -- start start 2024-03-30T04:26:10.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.536+0000 7ff517a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff5100662e0 0x7ff510113ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:26:10.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.536+0000 7ff517a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff5100fdcf0 0x7ff510114400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:26:10.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.536+0000 7ff517a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff510114a00 con 0x7ff5100662e0 2024-03-30T04:26:10.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.536+0000 7ff517a00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff510115250 con 0x7ff5100fdcf0 2024-03-30T04:26:10.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.536+0000 7ff515000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff5100662e0 0x7ff510113ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:26:10.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.536+0000 7ff515000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff5100662e0 0x7ff510113ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:59376/0 (socket says 172.21.15.123:59376) 2024-03-30T04:26:10.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.536+0000 7ff515000700 1 -- 172.21.15.123:0/3505979677 learned_addr learned my addr 172.21.15.123:0/3505979677 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:26:10.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.536+0000 7ff50fe00700 1 --2- 172.21.15.123:0/3505979677 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff5100fdcf0 0x7ff510114400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:26:10.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.536+0000 7ff515000700 1 -- 172.21.15.123:0/3505979677 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff5100fdcf0 msgr2=0x7ff510114400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:26:10.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.536+0000 7ff515000700 1 --2- 172.21.15.123:0/3505979677 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff5100fdcf0 0x7ff510114400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:10.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.536+0000 7ff515000700 1 -- 172.21.15.123:0/3505979677 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4fc005040 con 0x7ff5100662e0 2024-03-30T04:26:10.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.537+0000 7ff515000700 1 --2- 172.21.15.123:0/3505979677 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff5100662e0 0x7ff510113ec0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff50400eb40 tx=0x7ff50400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:26:10.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.537+0000 7ff50cc00700 1 -- 172.21.15.123:0/3505979677 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff504007cc0 con 0x7ff5100662e0 2024-03-30T04:26:10.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.537+0000 7ff50cc00700 1 -- 172.21.15.123:0/3505979677 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff50400f040 con 0x7ff5100662e0 2024-03-30T04:26:10.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.537+0000 7ff517a00700 1 -- 172.21.15.123:0/3505979677 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5101154d0 con 0x7ff5100662e0 2024-03-30T04:26:10.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.537+0000 7ff50cc00700 1 -- 172.21.15.123:0/3505979677 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff50400c8f0 con 0x7ff5100662e0 2024-03-30T04:26:10.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.537+0000 7ff517a00700 1 -- 172.21.15.123:0/3505979677 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff510115a10 con 0x7ff5100662e0 2024-03-30T04:26:10.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.538+0000 7ff517a00700 1 -- 172.21.15.123:0/3505979677 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff51005f5f0 con 0x7ff5100662e0 2024-03-30T04:26:10.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.539+0000 7ff50cc00700 1 -- 172.21.15.123:0/3505979677 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7ff50400cb60 con 0x7ff5100662e0 2024-03-30T04:26:10.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.541+0000 7ff50cc00700 1 --2- 172.21.15.123:0/3505979677 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7ff4f80692b0 0x7ff4f806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:26:10.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.541+0000 7ff50cc00700 1 -- 172.21.15.123:0/3505979677 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7ff504086c60 con 0x7ff5100662e0 2024-03-30T04:26:10.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.541+0000 7ff50fe00700 1 --2- 172.21.15.123:0/3505979677 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7ff4f80692b0 0x7ff4f806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:26:10.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.542+0000 7ff50fe00700 1 --2- 172.21.15.123:0/3505979677 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7ff4f80692b0 0x7ff4f806b760 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc009fd0 tx=0x7ff4fc009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:26:10.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.544+0000 7ff50cc00700 1 -- 172.21.15.123:0/3505979677 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7ff50404e190 con 0x7ff5100662e0 2024-03-30T04:26:10.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:10 smithi135 ceph-mon[44364]: pgmap v250: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:26:10.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:10 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:26:10.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:10 smithi135 ceph-mon[44364]: Upgrade: Target is version 18.2.2-1018-g470100a4 (unknown) 2024-03-30T04:26:10.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:10 smithi135 ceph-mon[44364]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35, digests ['quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35'] 2024-03-30T04:26:10.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:10 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:26:10.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:10 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:26:10.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:10 smithi135 ceph-mon[44364]: Upgrade: Need to upgrade myself (mgr.smithi123.xgksqq) 2024-03-30T04:26:10.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.777+0000 7ff517a00700 1 -- 172.21.15.123:0/3505979677 --> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff510115d90 con 0x7ff4f80692b0 2024-03-30T04:26:10.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.788+0000 7ff50cc00700 1 -- 172.21.15.123:0/3505979677 <== mgr.14206 v2:172.21.15.123:6800/1564935593 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3366 (secure 0 0 0) 0x7ff510115d90 con 0x7ff4f80692b0 2024-03-30T04:26:10.786 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:26:10.786 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (6m) 67s ago 7m 13.1M - 0.25.0 c8568f914cd2 0d2c80043347 2024-03-30T04:26:10.787 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (8m) 67s ago 8m 6647k - 17.2.7-797-g5c024335 f77085f6aab7 bceb8b8c4837 2024-03-30T04:26:10.787 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (6m) 72s ago 6m 6651k - 17.2.7-797-g5c024335 f77085f6aab7 cd0fe67b2d34 2024-03-30T04:26:10.787 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (6m) 67s ago 7m 74.0M - 9.4.7 954c08fa6188 b7d7eaffc084 2024-03-30T04:26:10.787 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (88s) 67s ago 87s 13.2M - 17.2.7-797-g5c024335 f77085f6aab7 2df066fba7a6 2024-03-30T04:26:10.787 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (96s) 67s ago 96s 11.5M - 17.2.7-797-g5c024335 f77085f6aab7 1293b8520db1 2024-03-30T04:26:10.787 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (85s) 72s ago 84s 11.6M - 17.2.7-797-g5c024335 f77085f6aab7 0dc41326818c 2024-03-30T04:26:10.787 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (93s) 72s ago 92s 13.0M - 17.2.7-797-g5c024335 f77085f6aab7 aef8a0cfc553 2024-03-30T04:26:10.787 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:9283 running (9m) 67s ago 9m 482M - 17.2.7-797-g5c024335 f77085f6aab7 7b98aee9a3b6 2024-03-30T04:26:10.787 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283 running (6m) 72s ago 6m 431M - 17.2.7-797-g5c024335 f77085f6aab7 37fc80673f76 2024-03-30T04:26:10.787 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (9m) 67s ago 9m 45.1M 2048M 17.2.7-797-g5c024335 f77085f6aab7 51d18587a7df 2024-03-30T04:26:10.787 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (6m) 72s ago 6m 32.3M 2048M 17.2.7-797-g5c024335 f77085f6aab7 f81052b09dd8 2024-03-30T04:26:10.787 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (8m) 67s ago 8m 7860k - 1.5.0 0da6a335fe13 9848cad1e995 2024-03-30T04:26:10.787 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (6m) 72s ago 6m 7545k - 1.5.0 0da6a335fe13 3c2c8e799cc2 2024-03-30T04:26:10.788 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (5m) 67s ago 5m 46.9M 4348M 17.2.7-797-g5c024335 f77085f6aab7 57a7ba3fd75e 2024-03-30T04:26:10.788 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (5m) 67s ago 5m 47.3M 4348M 17.2.7-797-g5c024335 f77085f6aab7 e1a6c1818a0c 2024-03-30T04:26:10.788 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (4m) 67s ago 4m 43.2M 4348M 17.2.7-797-g5c024335 f77085f6aab7 4c6a727567bd 2024-03-30T04:26:10.788 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (4m) 72s ago 4m 43.2M 5372M 17.2.7-797-g5c024335 f77085f6aab7 6fbdd1d46a9d 2024-03-30T04:26:10.788 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (3m) 72s ago 3m 40.4M 5372M 17.2.7-797-g5c024335 f77085f6aab7 b74988e947e5 2024-03-30T04:26:10.788 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (3m) 72s ago 3m 40.8M 5372M 17.2.7-797-g5c024335 f77085f6aab7 1fd3185f5484 2024-03-30T04:26:10.788 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (5m) 67s ago 7m 43.4M - 2.43.0 a07b618ecd1d ba3c351b14cf 2024-03-30T04:26:10.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.790+0000 7ff517a00700 1 -- 172.21.15.123:0/3505979677 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7ff4f80692b0 msgr2=0x7ff4f806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:26:10.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.790+0000 7ff517a00700 1 --2- 172.21.15.123:0/3505979677 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7ff4f80692b0 0x7ff4f806b760 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc009fd0 tx=0x7ff4fc009e30 comp rx=0 tx=0).stop 2024-03-30T04:26:10.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.790+0000 7ff517a00700 1 -- 172.21.15.123:0/3505979677 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff5100662e0 msgr2=0x7ff510113ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:26:10.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.790+0000 7ff517a00700 1 --2- 172.21.15.123:0/3505979677 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff5100662e0 0x7ff510113ec0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff50400eb40 tx=0x7ff50400b040 comp rx=0 tx=0).stop 2024-03-30T04:26:10.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.790+0000 7ff517a00700 1 -- 172.21.15.123:0/3505979677 shutdown_connections 2024-03-30T04:26:10.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.790+0000 7ff517a00700 1 --2- 172.21.15.123:0/3505979677 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7ff4f80692b0 0x7ff4f806b760 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:10.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.790+0000 7ff517a00700 1 --2- 172.21.15.123:0/3505979677 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff5100662e0 0x7ff510113ec0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:10.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.790+0000 7ff517a00700 1 --2- 172.21.15.123:0/3505979677 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff5100fdcf0 0x7ff510114400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:10.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.790+0000 7ff517a00700 1 -- 172.21.15.123:0/3505979677 >> 172.21.15.123:0/3505979677 conn(0x7ff5100f7430 msgr2=0x7ff5100f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:26:10.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.790+0000 7ff517a00700 1 -- 172.21.15.123:0/3505979677 shutdown_connections 2024-03-30T04:26:10.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.790+0000 7ff517a00700 1 -- 172.21.15.123:0/3505979677 wait complete. 2024-03-30T04:26:10.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.936+0000 7f382fc00700 1 -- 172.21.15.123:0/2071962194 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3828071e40 msgr2=0x7f3828070350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:26:10.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.936+0000 7f382fc00700 1 --2- 172.21.15.123:0/2071962194 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3828071e40 0x7f3828070350 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f381c004940 tx=0x7f381c009e30 comp rx=0 tx=0).stop 2024-03-30T04:26:10.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.937+0000 7f382fc00700 1 -- 172.21.15.123:0/2071962194 shutdown_connections 2024-03-30T04:26:10.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.937+0000 7f382fc00700 1 --2- 172.21.15.123:0/2071962194 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3828070890 0x7f3828070d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:10.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.937+0000 7f382fc00700 1 --2- 172.21.15.123:0/2071962194 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3828071e40 0x7f3828070350 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:10.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.937+0000 7f382fc00700 1 -- 172.21.15.123:0/2071962194 >> 172.21.15.123:0/2071962194 conn(0x7f38280f8550 msgr2=0x7f38280fa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:26:10.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.937+0000 7f382fc00700 1 -- 172.21.15.123:0/2071962194 shutdown_connections 2024-03-30T04:26:10.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.937+0000 7f382fc00700 1 -- 172.21.15.123:0/2071962194 wait complete. 2024-03-30T04:26:10.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.938+0000 7f382fc00700 1 Processor -- start 2024-03-30T04:26:10.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.938+0000 7f382fc00700 1 -- start start 2024-03-30T04:26:10.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.938+0000 7f382fc00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3828070890 0x7f382810fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:26:10.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.938+0000 7f382fc00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3828071e40 0x7f382810ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:26:10.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.938+0000 7f382fc00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3828110590 con 0x7f3828071e40 2024-03-30T04:26:10.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.938+0000 7f382fc00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3828110700 con 0x7f3828070890 2024-03-30T04:26:10.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.938+0000 7f382d200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3828070890 0x7f382810fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:26:10.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.938+0000 7f382d200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3828070890 0x7f382810fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.123:52488/0 (socket says 172.21.15.123:52488) 2024-03-30T04:26:10.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.938+0000 7f382d200700 1 -- 172.21.15.123:0/1850356518 learned_addr learned my addr 172.21.15.123:0/1850356518 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:26:10.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.939+0000 7f3827e00700 1 --2- 172.21.15.123:0/1850356518 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3828071e40 0x7f382810ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:26:10.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.939+0000 7f382d200700 1 -- 172.21.15.123:0/1850356518 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3828071e40 msgr2=0x7f382810ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:26:10.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.939+0000 7f382d200700 1 --2- 172.21.15.123:0/1850356518 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3828071e40 0x7f382810ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:10.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.939+0000 7f382d200700 1 -- 172.21.15.123:0/1850356518 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f381c005040 con 0x7f3828070890 2024-03-30T04:26:10.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.939+0000 7f382d200700 1 --2- 172.21.15.123:0/1850356518 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3828070890 0x7f382810fa50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f381c009380 tx=0x7f381c009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:26:10.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.940+0000 7f3824c00700 1 -- 172.21.15.123:0/1850356518 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f381c00e040 con 0x7f3828070890 2024-03-30T04:26:10.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.940+0000 7f382fc00700 1 -- 172.21.15.123:0/1850356518 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3828115100 con 0x7f3828070890 2024-03-30T04:26:10.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.940+0000 7f3824c00700 1 -- 172.21.15.123:0/1850356518 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f381c004290 con 0x7f3828070890 2024-03-30T04:26:10.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.940+0000 7f3824c00700 1 -- 172.21.15.123:0/1850356518 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f381c01c3f0 con 0x7f3828070890 2024-03-30T04:26:10.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.940+0000 7f382fc00700 1 -- 172.21.15.123:0/1850356518 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38281156c0 con 0x7f3828070890 2024-03-30T04:26:10.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.941+0000 7f382fc00700 1 -- 172.21.15.123:0/1850356518 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f382804c710 con 0x7f3828070890 2024-03-30T04:26:10.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.942+0000 7f3824c00700 1 -- 172.21.15.123:0/1850356518 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f381c0137b0 con 0x7f3828070890 2024-03-30T04:26:10.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.942+0000 7f3824c00700 1 --2- 172.21.15.123:0/1850356518 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f38100692a0 0x7f381006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:26:10.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.943+0000 7f3824c00700 1 -- 172.21.15.123:0/1850356518 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f381c088590 con 0x7f3828070890 2024-03-30T04:26:10.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.943+0000 7f3827e00700 1 --2- 172.21.15.123:0/1850356518 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f38100692a0 0x7f381006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:26:10.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.943+0000 7f3827e00700 1 --2- 172.21.15.123:0/1850356518 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f38100692a0 0x7f381006b750 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3814004f40 tx=0x7f381400b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:26:10.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:10.949+0000 7f3824c00700 1 -- 172.21.15.123:0/1850356518 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f381c058260 con 0x7f3828070890 2024-03-30T04:26:11.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.257+0000 7f382fc00700 1 -- 172.21.15.123:0/1850356518 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f3828115ac0 con 0x7f3828070890 2024-03-30T04:26:11.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.258+0000 7f3824c00700 1 -- 172.21.15.123:0/1850356518 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7f381c057df0 con 0x7f3828070890 2024-03-30T04:26:11.259 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:26:11.259 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:26:11.259 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 2 2024-03-30T04:26:11.259 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:26:11.259 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:26:11.259 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 2 2024-03-30T04:26:11.259 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:26:11.259 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:26:11.259 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 6 2024-03-30T04:26:11.260 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:26:11.260 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:26:11.260 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 4 2024-03-30T04:26:11.260 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:26:11.260 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:26:11.260 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 14 2024-03-30T04:26:11.260 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:26:11.260 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:26:11.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.260+0000 7f382fc00700 1 -- 172.21.15.123:0/1850356518 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f38100692a0 msgr2=0x7f381006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:26:11.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.260+0000 7f382fc00700 1 --2- 172.21.15.123:0/1850356518 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f38100692a0 0x7f381006b750 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3814004f40 tx=0x7f381400b040 comp rx=0 tx=0).stop 2024-03-30T04:26:11.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.260+0000 7f382fc00700 1 -- 172.21.15.123:0/1850356518 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3828070890 msgr2=0x7f382810fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:26:11.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.260+0000 7f382fc00700 1 --2- 172.21.15.123:0/1850356518 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3828070890 0x7f382810fa50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f381c009380 tx=0x7f381c009e30 comp rx=0 tx=0).stop 2024-03-30T04:26:11.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.260+0000 7f382fc00700 1 -- 172.21.15.123:0/1850356518 shutdown_connections 2024-03-30T04:26:11.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.260+0000 7f382fc00700 1 --2- 172.21.15.123:0/1850356518 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7f38100692a0 0x7f381006b750 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:11.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.260+0000 7f382fc00700 1 --2- 172.21.15.123:0/1850356518 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3828070890 0x7f382810fa50 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:11.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.260+0000 7f382fc00700 1 --2- 172.21.15.123:0/1850356518 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3828071e40 0x7f382810ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:11.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.260+0000 7f382fc00700 1 -- 172.21.15.123:0/1850356518 >> 172.21.15.123:0/1850356518 conn(0x7f38280f8550 msgr2=0x7f38280f9cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:26:11.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.260+0000 7f382fc00700 1 -- 172.21.15.123:0/1850356518 shutdown_connections 2024-03-30T04:26:11.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.260+0000 7f382fc00700 1 -- 172.21.15.123:0/1850356518 wait complete. 2024-03-30T04:26:11.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.412+0000 7fe9d7800700 1 -- 172.21.15.123:0/145578594 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9d00fe0d0 msgr2=0x7fe9d00fe540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:26:11.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.412+0000 7fe9d7800700 1 --2- 172.21.15.123:0/145578594 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9d00fe0d0 0x7fe9d00fe540 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc004940 tx=0x7fe9bc009e30 comp rx=0 tx=0).stop 2024-03-30T04:26:11.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.412+0000 7fe9d7800700 1 -- 172.21.15.123:0/145578594 shutdown_connections 2024-03-30T04:26:11.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.412+0000 7fe9d7800700 1 --2- 172.21.15.123:0/145578594 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9d00fe0d0 0x7fe9d00fe540 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:11.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.412+0000 7fe9d7800700 1 --2- 172.21.15.123:0/145578594 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe9d00fcee0 0x7fe9d00fd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:11.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.412+0000 7fe9d7800700 1 -- 172.21.15.123:0/145578594 >> 172.21.15.123:0/145578594 conn(0x7fe9d00f8560 msgr2=0x7fe9d00fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:26:11.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.412+0000 7fe9d7800700 1 -- 172.21.15.123:0/145578594 shutdown_connections 2024-03-30T04:26:11.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.412+0000 7fe9d7800700 1 -- 172.21.15.123:0/145578594 wait complete. 2024-03-30T04:26:11.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.413+0000 7fe9d7800700 1 Processor -- start 2024-03-30T04:26:11.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.413+0000 7fe9d7800700 1 -- start start 2024-03-30T04:26:11.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.414+0000 7fe9d7800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9d00fcee0 0x7fe9d010d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:26:11.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.414+0000 7fe9d7800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe9d00fe0d0 0x7fe9d010df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:26:11.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.414+0000 7fe9d7800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d010e530 con 0x7fe9d00fcee0 2024-03-30T04:26:11.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.414+0000 7fe9d7800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d010ed10 con 0x7fe9d00fe0d0 2024-03-30T04:26:11.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.414+0000 7fe9d4e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9d00fcee0 0x7fe9d010d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:26:11.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.414+0000 7fe9d4e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9d00fcee0 0x7fe9d010d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:59410/0 (socket says 172.21.15.123:59410) 2024-03-30T04:26:11.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.414+0000 7fe9d4e00700 1 -- 172.21.15.123:0/2199142281 learned_addr learned my addr 172.21.15.123:0/2199142281 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:26:11.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.414+0000 7fe9cfe00700 1 --2- 172.21.15.123:0/2199142281 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe9d00fe0d0 0x7fe9d010df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:26:11.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.414+0000 7fe9d4e00700 1 -- 172.21.15.123:0/2199142281 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe9d00fe0d0 msgr2=0x7fe9d010df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:26:11.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.414+0000 7fe9d4e00700 1 --2- 172.21.15.123:0/2199142281 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe9d00fe0d0 0x7fe9d010df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:11.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.414+0000 7fe9d4e00700 1 -- 172.21.15.123:0/2199142281 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9bc005040 con 0x7fe9d00fcee0 2024-03-30T04:26:11.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.415+0000 7fe9d4e00700 1 --2- 172.21.15.123:0/2199142281 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9d00fcee0 0x7fe9d010d9f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe9c400eb40 tx=0x7fe9c400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:26:11.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.415+0000 7fe9ccc00700 1 -- 172.21.15.123:0/2199142281 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9c4007cc0 con 0x7fe9d00fcee0 2024-03-30T04:26:11.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.415+0000 7fe9ccc00700 1 -- 172.21.15.123:0/2199142281 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe9c400f040 con 0x7fe9d00fcee0 2024-03-30T04:26:11.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.415+0000 7fe9d7800700 1 -- 172.21.15.123:0/2199142281 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9d010ef90 con 0x7fe9d00fcee0 2024-03-30T04:26:11.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.415+0000 7fe9ccc00700 1 -- 172.21.15.123:0/2199142281 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9c400c9f0 con 0x7fe9d00fcee0 2024-03-30T04:26:11.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.415+0000 7fe9d7800700 1 -- 172.21.15.123:0/2199142281 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9d010f4d0 con 0x7fe9d00fcee0 2024-03-30T04:26:11.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.417+0000 7fe9d7800700 1 -- 172.21.15.123:0/2199142281 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9d005f5f0 con 0x7fe9d00fcee0 2024-03-30T04:26:11.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.417+0000 7fe9ccc00700 1 -- 172.21.15.123:0/2199142281 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fe9c400cb50 con 0x7fe9d00fcee0 2024-03-30T04:26:11.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.419+0000 7fe9ccc00700 1 --2- 172.21.15.123:0/2199142281 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fe9b80692b0 0x7fe9b806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:26:11.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.419+0000 7fe9ccc00700 1 -- 172.21.15.123:0/2199142281 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fe9c4086d00 con 0x7fe9d00fcee0 2024-03-30T04:26:11.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.419+0000 7fe9cfe00700 1 --2- 172.21.15.123:0/2199142281 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fe9b80692b0 0x7fe9b806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:26:11.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.419+0000 7fe9cfe00700 1 --2- 172.21.15.123:0/2199142281 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fe9b80692b0 0x7fe9b806b760 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc00b0d0 tx=0x7fe9bc009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:26:11.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.422+0000 7fe9ccc00700 1 -- 172.21.15.123:0/2199142281 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fe9c40569d0 con 0x7fe9d00fcee0 2024-03-30T04:26:11.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:11 smithi135 ceph-mon[44364]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 on smithi135 2024-03-30T04:26:11.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:11 smithi135 ceph-mon[44364]: from='client.24383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:26:11.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.661+0000 7fe9d7800700 1 -- 172.21.15.123:0/2199142281 --> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe9d010f8a0 con 0x7fe9b80692b0 2024-03-30T04:26:11.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.662+0000 7fe9ccc00700 1 -- 172.21.15.123:0/2199142281 <== mgr.14206 v2:172.21.15.123:6800/1564935593 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7fe9d010f8a0 con 0x7fe9b80692b0 2024-03-30T04:26:11.661 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:26:11.661 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:26:11.661 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:26:11.661 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-30T04:26:11.661 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [], 2024-03-30T04:26:11.661 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "0/2 daemons upgraded", 2024-03-30T04:26:11.662 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 image on host smithi135", 2024-03-30T04:26:11.662 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:26:11.662 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:26:11.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.666+0000 7fe9d7800700 1 -- 172.21.15.123:0/2199142281 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fe9b80692b0 msgr2=0x7fe9b806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:26:11.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.666+0000 7fe9d7800700 1 --2- 172.21.15.123:0/2199142281 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fe9b80692b0 0x7fe9b806b760 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc00b0d0 tx=0x7fe9bc009e30 comp rx=0 tx=0).stop 2024-03-30T04:26:11.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.666+0000 7fe9d7800700 1 -- 172.21.15.123:0/2199142281 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9d00fcee0 msgr2=0x7fe9d010d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:26:11.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.666+0000 7fe9d7800700 1 --2- 172.21.15.123:0/2199142281 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9d00fcee0 0x7fe9d010d9f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe9c400eb40 tx=0x7fe9c400b040 comp rx=0 tx=0).stop 2024-03-30T04:26:11.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.666+0000 7fe9d7800700 1 -- 172.21.15.123:0/2199142281 shutdown_connections 2024-03-30T04:26:11.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.666+0000 7fe9d7800700 1 --2- 172.21.15.123:0/2199142281 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fe9b80692b0 0x7fe9b806b760 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:11.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.666+0000 7fe9d7800700 1 --2- 172.21.15.123:0/2199142281 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9d00fcee0 0x7fe9d010d9f0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:11.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.666+0000 7fe9d7800700 1 --2- 172.21.15.123:0/2199142281 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe9d00fe0d0 0x7fe9d010df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:11.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.666+0000 7fe9d7800700 1 -- 172.21.15.123:0/2199142281 >> 172.21.15.123:0/2199142281 conn(0x7fe9d00f8560 msgr2=0x7fe9d00f9da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:26:11.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.666+0000 7fe9d7800700 1 -- 172.21.15.123:0/2199142281 shutdown_connections 2024-03-30T04:26:11.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:11.666+0000 7fe9d7800700 1 -- 172.21.15.123:0/2199142281 wait complete. 2024-03-30T04:26:11.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:11 smithi123 ceph-mon[35245]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 on smithi135 2024-03-30T04:26:11.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:11 smithi123 ceph-mon[35245]: from='client.24383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:26:12.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:12 smithi135 ceph-mon[44364]: pgmap v251: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-30T04:26:12.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:12 smithi135 ceph-mon[44364]: from='client.14572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:26:12.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:12 smithi135 ceph-mon[44364]: from='client.14576 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:26:12.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:12 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/1850356518' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:26:12.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:12 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:26:12.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:12 smithi123 ceph-mon[35245]: pgmap v251: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-30T04:26:12.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:12 smithi123 ceph-mon[35245]: from='client.14572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:26:12.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:12 smithi123 ceph-mon[35245]: from='client.14576 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:26:12.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:12 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/1850356518' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:26:12.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:12 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:26:13.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:13 smithi135 ceph-mon[44364]: from='client.14584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:26:13.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:13 smithi123 ceph-mon[35245]: from='client.14584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:26:14.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:14 smithi135 ceph-mon[44364]: pgmap v252: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-30T04:26:14.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:14 smithi123 ceph-mon[35245]: pgmap v252: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-30T04:26:16.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:16 smithi135 ceph-mon[44364]: pgmap v253: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:26:16.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:16 smithi123 ceph-mon[35245]: pgmap v253: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:26:18.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:18 smithi135 ceph-mon[44364]: pgmap v254: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:26:18.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:18 smithi123 ceph-mon[35245]: pgmap v254: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:26:20.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:20 smithi135 ceph-mon[44364]: pgmap v255: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:26:20.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:20 smithi123 ceph-mon[35245]: pgmap v255: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:26:22.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:22 smithi135 ceph-mon[44364]: pgmap v256: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-30T04:26:22.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:22 smithi123 ceph-mon[35245]: pgmap v256: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-30T04:26:24.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:24 smithi135 ceph-mon[44364]: pgmap v257: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-30T04:26:24.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:24 smithi123 ceph-mon[35245]: pgmap v257: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-30T04:26:26.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:26 smithi135 ceph-mon[44364]: pgmap v258: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:26:26.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:26 smithi123 ceph-mon[35245]: pgmap v258: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:26:27.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:27 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:26:27.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:27 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:26:28.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:28 smithi135 ceph-mon[44364]: pgmap v259: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:26:28.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:28 smithi123 ceph-mon[35245]: pgmap v259: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:26:30.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:30 smithi135 ceph-mon[44364]: pgmap v260: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:26:30.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:30 smithi123 ceph-mon[35245]: pgmap v260: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:26:32.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:32 smithi135 ceph-mon[44364]: pgmap v261: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-30T04:26:32.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:32 smithi123 ceph-mon[35245]: pgmap v261: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-30T04:26:34.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:34 smithi135 ceph-mon[44364]: pgmap v262: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-30T04:26:34.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:34 smithi123 ceph-mon[35245]: pgmap v262: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-30T04:26:36.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:36 smithi135 ceph-mon[44364]: pgmap v263: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:26:36.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:36 smithi123 ceph-mon[35245]: pgmap v263: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:26:38.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:38 smithi135 ceph-mon[44364]: pgmap v264: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:26:38.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:38 smithi123 ceph-mon[35245]: pgmap v264: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:26:40.482 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:40 smithi135 ceph-mon[44364]: pgmap v265: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:26:40.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:40 smithi123 ceph-mon[35245]: pgmap v265: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:26:41.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:41.826+0000 7fdc30600700 1 -- 172.21.15.123:0/566855158 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc280fce70 msgr2=0x7fdc280fd280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:26:41.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:41.826+0000 7fdc30600700 1 --2- 172.21.15.123:0/566855158 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc280fce70 0x7fdc280fd280 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c004940 tx=0x7fdc1c009e30 comp rx=0 tx=0).stop 2024-03-30T04:26:41.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:41.827+0000 7fdc30600700 1 -- 172.21.15.123:0/566855158 shutdown_connections 2024-03-30T04:26:41.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:41.827+0000 7fdc30600700 1 --2- 172.21.15.123:0/566855158 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdc280fdfe0 0x7fdc280fe450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:41.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:41.827+0000 7fdc30600700 1 --2- 172.21.15.123:0/566855158 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc280fce70 0x7fdc280fd280 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:41.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:41.827+0000 7fdc30600700 1 -- 172.21.15.123:0/566855158 >> 172.21.15.123:0/566855158 conn(0x7fdc280f8550 msgr2=0x7fdc280fa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:26:41.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:41.827+0000 7fdc30600700 1 -- 172.21.15.123:0/566855158 shutdown_connections 2024-03-30T04:26:41.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:41.827+0000 7fdc30600700 1 -- 172.21.15.123:0/566855158 wait complete. 2024-03-30T04:26:41.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:41.828+0000 7fdc30600700 1 Processor -- start 2024-03-30T04:26:41.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:41.828+0000 7fdc30600700 1 -- start start 2024-03-30T04:26:41.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:41.828+0000 7fdc30600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdc280fce70 0x7fdc2810d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:26:41.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:41.828+0000 7fdc30600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc280fdfe0 0x7fdc2810dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:26:41.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:41.828+0000 7fdc30600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2810e310 con 0x7fdc280fdfe0 2024-03-30T04:26:41.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:41.828+0000 7fdc30600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2810e480 con 0x7fdc280fce70 2024-03-30T04:26:41.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:41.829+0000 7fdc2d200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc280fdfe0 0x7fdc2810dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:26:41.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:41.829+0000 7fdc2dc00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdc280fce70 0x7fdc2810d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:26:41.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:41.829+0000 7fdc2d200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc280fdfe0 0x7fdc2810dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42032/0 (socket says 172.21.15.123:42032) 2024-03-30T04:26:41.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:41.829+0000 7fdc2dc00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdc280fce70 0x7fdc2810d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.123:45988/0 (socket says 172.21.15.123:45988) 2024-03-30T04:26:41.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:41.829+0000 7fdc2d200700 1 -- 172.21.15.123:0/3741889857 learned_addr learned my addr 172.21.15.123:0/3741889857 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:26:41.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:41.830+0000 7fdc2dc00700 1 -- 172.21.15.123:0/3741889857 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc280fdfe0 msgr2=0x7fdc2810dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:26:41.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:41.830+0000 7fdc2dc00700 1 --2- 172.21.15.123:0/3741889857 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc280fdfe0 0x7fdc2810dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:41.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:41.830+0000 7fdc2dc00700 1 -- 172.21.15.123:0/3741889857 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc1c005040 con 0x7fdc280fce70 2024-03-30T04:26:41.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:41.830+0000 7fdc2d200700 1 --2- 172.21.15.123:0/3741889857 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc280fdfe0 0x7fdc2810dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:26:41.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:41.830+0000 7fdc2dc00700 1 --2- 172.21.15.123:0/3741889857 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdc280fce70 0x7fdc2810d7d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c00b0d0 tx=0x7fdc1c009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:26:41.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:41.830+0000 7fdc25600700 1 -- 172.21.15.123:0/3741889857 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc1c00e070 con 0x7fdc280fce70 2024-03-30T04:26:41.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:41.830+0000 7fdc25600700 1 -- 172.21.15.123:0/3741889857 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdc1c01d460 con 0x7fdc280fce70 2024-03-30T04:26:41.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:41.830+0000 7fdc30600700 1 -- 172.21.15.123:0/3741889857 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc28112e80 con 0x7fdc280fce70 2024-03-30T04:26:41.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:41.830+0000 7fdc25600700 1 -- 172.21.15.123:0/3741889857 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc1c00b670 con 0x7fdc280fce70 2024-03-30T04:26:41.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:41.831+0000 7fdc30600700 1 -- 172.21.15.123:0/3741889857 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc281133b0 con 0x7fdc280fce70 2024-03-30T04:26:41.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:41.832+0000 7fdc30600700 1 -- 172.21.15.123:0/3741889857 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc2804c710 con 0x7fdc280fce70 2024-03-30T04:26:41.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:41.833+0000 7fdc25600700 1 -- 172.21.15.123:0/3741889857 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fdc1c01da60 con 0x7fdc280fce70 2024-03-30T04:26:41.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:41.834+0000 7fdc25600700 1 --2- 172.21.15.123:0/3741889857 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fdc100692a0 0x7fdc1006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:26:41.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:41.834+0000 7fdc25600700 1 -- 172.21.15.123:0/3741889857 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fdc1c088310 con 0x7fdc280fce70 2024-03-30T04:26:41.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:41.834+0000 7fdc2d200700 1 --2- 172.21.15.123:0/3741889857 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fdc100692a0 0x7fdc1006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:26:41.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:41.835+0000 7fdc2d200700 1 --2- 172.21.15.123:0/3741889857 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fdc100692a0 0x7fdc1006b750 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fdc140041d0 tx=0x7fdc1400b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:26:41.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:41.843+0000 7fdc25600700 1 -- 172.21.15.123:0/3741889857 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fdc1c057fe0 con 0x7fdc280fce70 2024-03-30T04:26:42.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:42 smithi135 ceph-mon[44364]: pgmap v266: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-30T04:26:42.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:42 smithi135 ceph-mon[44364]: Upgrade: Updating mgr.smithi135.kyblli 2024-03-30T04:26:42.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:42 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:26:42.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:42 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi135.kyblli", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T04:26:42.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:42 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T04:26:42.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:42 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:26:42.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:42 smithi135 ceph-mon[44364]: Deploying daemon mgr.smithi135.kyblli on smithi135 2024-03-30T04:26:42.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:42 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:26:42.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.167+0000 7fdc30600700 1 -- 172.21.15.123:0/3741889857 --> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdc28102340 con 0x7fdc100692a0 2024-03-30T04:26:42.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.169+0000 7fdc25600700 1 -- 172.21.15.123:0/3741889857 <== mgr.14206 v2:172.21.15.123:6800/1564935593 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7fdc28102340 con 0x7fdc100692a0 2024-03-30T04:26:42.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.171+0000 7fdc30600700 1 -- 172.21.15.123:0/3741889857 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fdc100692a0 msgr2=0x7fdc1006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:26:42.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.171+0000 7fdc30600700 1 --2- 172.21.15.123:0/3741889857 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fdc100692a0 0x7fdc1006b750 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fdc140041d0 tx=0x7fdc1400b040 comp rx=0 tx=0).stop 2024-03-30T04:26:42.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.171+0000 7fdc30600700 1 -- 172.21.15.123:0/3741889857 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdc280fce70 msgr2=0x7fdc2810d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:26:42.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.171+0000 7fdc30600700 1 --2- 172.21.15.123:0/3741889857 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdc280fce70 0x7fdc2810d7d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c00b0d0 tx=0x7fdc1c009e30 comp rx=0 tx=0).stop 2024-03-30T04:26:42.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.171+0000 7fdc30600700 1 -- 172.21.15.123:0/3741889857 shutdown_connections 2024-03-30T04:26:42.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.171+0000 7fdc30600700 1 --2- 172.21.15.123:0/3741889857 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fdc100692a0 0x7fdc1006b750 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:42.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.171+0000 7fdc30600700 1 --2- 172.21.15.123:0/3741889857 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdc280fce70 0x7fdc2810d7d0 secure :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c00b0d0 tx=0x7fdc1c009e30 comp rx=0 tx=0).stop 2024-03-30T04:26:42.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.171+0000 7fdc30600700 1 --2- 172.21.15.123:0/3741889857 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc280fdfe0 0x7fdc2810dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:42.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.171+0000 7fdc30600700 1 -- 172.21.15.123:0/3741889857 >> 172.21.15.123:0/3741889857 conn(0x7fdc280f8550 msgr2=0x7fdc280f9c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:26:42.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.172+0000 7fdc30600700 1 -- 172.21.15.123:0/3741889857 shutdown_connections 2024-03-30T04:26:42.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.172+0000 7fdc30600700 1 -- 172.21.15.123:0/3741889857 wait complete. 2024-03-30T04:26:42.185 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:26:42.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.323+0000 7fba20800700 1 -- 172.21.15.123:0/3496875889 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba18071e40 msgr2=0x7fba18070350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:26:42.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.323+0000 7fba20800700 1 --2- 172.21.15.123:0/3496875889 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba18071e40 0x7fba18070350 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fba0c004940 tx=0x7fba0c009e30 comp rx=0 tx=0).stop 2024-03-30T04:26:42.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.323+0000 7fba20800700 1 -- 172.21.15.123:0/3496875889 shutdown_connections 2024-03-30T04:26:42.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.323+0000 7fba20800700 1 --2- 172.21.15.123:0/3496875889 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fba18070890 0x7fba18070d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:42.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.323+0000 7fba20800700 1 --2- 172.21.15.123:0/3496875889 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba18071e40 0x7fba18070350 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:42.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.323+0000 7fba20800700 1 -- 172.21.15.123:0/3496875889 >> 172.21.15.123:0/3496875889 conn(0x7fba180f8550 msgr2=0x7fba180fa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:26:42.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.323+0000 7fba20800700 1 -- 172.21.15.123:0/3496875889 shutdown_connections 2024-03-30T04:26:42.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.323+0000 7fba20800700 1 -- 172.21.15.123:0/3496875889 wait complete. 2024-03-30T04:26:42.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.324+0000 7fba20800700 1 Processor -- start 2024-03-30T04:26:42.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.325+0000 7fba20800700 1 -- start start 2024-03-30T04:26:42.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.325+0000 7fba20800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fba18070890 0x7fba1810fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:26:42.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.325+0000 7fba20800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba18071e40 0x7fba1810ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:26:42.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.325+0000 7fba20800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba18110540 con 0x7fba18071e40 2024-03-30T04:26:42.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.325+0000 7fba20800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba181106b0 con 0x7fba18070890 2024-03-30T04:26:42.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.326+0000 7fba1de00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fba18070890 0x7fba1810fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:26:42.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.326+0000 7fba1de00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fba18070890 0x7fba1810fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.123:45998/0 (socket says 172.21.15.123:45998) 2024-03-30T04:26:42.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.326+0000 7fba1de00700 1 -- 172.21.15.123:0/1067325107 learned_addr learned my addr 172.21.15.123:0/1067325107 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:26:42.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.326+0000 7fba1d400700 1 --2- 172.21.15.123:0/1067325107 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba18071e40 0x7fba1810ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:26:42.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.326+0000 7fba1de00700 1 -- 172.21.15.123:0/1067325107 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba18071e40 msgr2=0x7fba1810ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:26:42.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.326+0000 7fba1de00700 1 --2- 172.21.15.123:0/1067325107 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba18071e40 0x7fba1810ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:42.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.326+0000 7fba1de00700 1 -- 172.21.15.123:0/1067325107 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba0c005040 con 0x7fba18070890 2024-03-30T04:26:42.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.326+0000 7fba1d400700 1 --2- 172.21.15.123:0/1067325107 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba18071e40 0x7fba1810ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:26:42.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.327+0000 7fba1de00700 1 --2- 172.21.15.123:0/1067325107 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fba18070890 0x7fba1810fa00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fba0c000c00 tx=0x7fba0c009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:26:42.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.327+0000 7fba12000700 1 -- 172.21.15.123:0/1067325107 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba0c00e070 con 0x7fba18070890 2024-03-30T04:26:42.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.328+0000 7fba12000700 1 -- 172.21.15.123:0/1067325107 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fba0c0093b0 con 0x7fba18070890 2024-03-30T04:26:42.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.328+0000 7fba20800700 1 -- 172.21.15.123:0/1067325107 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba181150b0 con 0x7fba18070890 2024-03-30T04:26:42.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.328+0000 7fba12000700 1 -- 172.21.15.123:0/1067325107 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba0c01cd80 con 0x7fba18070890 2024-03-30T04:26:42.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.328+0000 7fba20800700 1 -- 172.21.15.123:0/1067325107 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba18115610 con 0x7fba18070890 2024-03-30T04:26:42.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.329+0000 7fba20800700 1 -- 172.21.15.123:0/1067325107 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba1804c710 con 0x7fba18070890 2024-03-30T04:26:42.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.331+0000 7fba12000700 1 -- 172.21.15.123:0/1067325107 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fba0c013a30 con 0x7fba18070890 2024-03-30T04:26:42.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.332+0000 7fba12000700 1 --2- 172.21.15.123:0/1067325107 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fba00069370 0x7fba0006b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:26:42.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.333+0000 7fba12000700 1 -- 172.21.15.123:0/1067325107 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fba0c0887c0 con 0x7fba18070890 2024-03-30T04:26:42.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.333+0000 7fba1d400700 1 --2- 172.21.15.123:0/1067325107 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fba00069370 0x7fba0006b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:26:42.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.334+0000 7fba1d400700 1 --2- 172.21.15.123:0/1067325107 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fba00069370 0x7fba0006b820 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fba08009b80 tx=0x7fba0800b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:26:42.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.341+0000 7fba12000700 1 -- 172.21.15.123:0/1067325107 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fba0c058510 con 0x7fba18070890 2024-03-30T04:26:42.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:42 smithi123 ceph-mon[35245]: pgmap v266: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-30T04:26:42.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:42 smithi123 ceph-mon[35245]: Upgrade: Updating mgr.smithi135.kyblli 2024-03-30T04:26:42.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:42 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:26:42.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:42 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi135.kyblli", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T04:26:42.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:42 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T04:26:42.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:42 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:26:42.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:42 smithi123 ceph-mon[35245]: Deploying daemon mgr.smithi135.kyblli on smithi135 2024-03-30T04:26:42.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:42 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:26:42.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.605+0000 7fba20800700 1 -- 172.21.15.123:0/1067325107 --> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fba18103ee0 con 0x7fba00069370 2024-03-30T04:26:42.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.606+0000 7fba12000700 1 -- 172.21.15.123:0/1067325107 <== mgr.14206 v2:172.21.15.123:6800/1564935593 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7fba18103ee0 con 0x7fba00069370 2024-03-30T04:26:42.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.608+0000 7fba20800700 1 -- 172.21.15.123:0/1067325107 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fba00069370 msgr2=0x7fba0006b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:26:42.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.608+0000 7fba20800700 1 --2- 172.21.15.123:0/1067325107 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fba00069370 0x7fba0006b820 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fba08009b80 tx=0x7fba0800b040 comp rx=0 tx=0).stop 2024-03-30T04:26:42.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.608+0000 7fba20800700 1 -- 172.21.15.123:0/1067325107 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fba18070890 msgr2=0x7fba1810fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:26:42.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.608+0000 7fba20800700 1 --2- 172.21.15.123:0/1067325107 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fba18070890 0x7fba1810fa00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fba0c000c00 tx=0x7fba0c009e30 comp rx=0 tx=0).stop 2024-03-30T04:26:42.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.609+0000 7fba20800700 1 -- 172.21.15.123:0/1067325107 shutdown_connections 2024-03-30T04:26:42.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.609+0000 7fba20800700 1 --2- 172.21.15.123:0/1067325107 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fba00069370 0x7fba0006b820 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:42.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.609+0000 7fba20800700 1 --2- 172.21.15.123:0/1067325107 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fba18070890 0x7fba1810fa00 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:42.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.609+0000 7fba20800700 1 --2- 172.21.15.123:0/1067325107 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba18071e40 0x7fba1810ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:42.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.609+0000 7fba20800700 1 -- 172.21.15.123:0/1067325107 >> 172.21.15.123:0/1067325107 conn(0x7fba180f8550 msgr2=0x7fba180f9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:26:42.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.609+0000 7fba20800700 1 -- 172.21.15.123:0/1067325107 shutdown_connections 2024-03-30T04:26:42.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.609+0000 7fba20800700 1 -- 172.21.15.123:0/1067325107 wait complete. 2024-03-30T04:26:42.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.745+0000 7fe38be00700 1 -- 172.21.15.123:0/1932667612 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe38c070890 msgr2=0x7fe38c070d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:26:42.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.745+0000 7fe38be00700 1 --2- 172.21.15.123:0/1932667612 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe38c070890 0x7fe38c070d00 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe378004940 tx=0x7fe378009e30 comp rx=0 tx=0).stop 2024-03-30T04:26:42.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.745+0000 7fe38be00700 1 -- 172.21.15.123:0/1932667612 shutdown_connections 2024-03-30T04:26:42.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.745+0000 7fe38be00700 1 --2- 172.21.15.123:0/1932667612 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe38c070890 0x7fe38c070d00 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:42.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.745+0000 7fe38be00700 1 --2- 172.21.15.123:0/1932667612 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe38c071e40 0x7fe38c070350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:42.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.745+0000 7fe38be00700 1 -- 172.21.15.123:0/1932667612 >> 172.21.15.123:0/1932667612 conn(0x7fe38c0f8390 msgr2=0x7fe38c0fa7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:26:42.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.746+0000 7fe38be00700 1 -- 172.21.15.123:0/1932667612 shutdown_connections 2024-03-30T04:26:42.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.746+0000 7fe38be00700 1 -- 172.21.15.123:0/1932667612 wait complete. 2024-03-30T04:26:42.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.747+0000 7fe38be00700 1 Processor -- start 2024-03-30T04:26:42.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.747+0000 7fe38be00700 1 -- start start 2024-03-30T04:26:42.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.747+0000 7fe38be00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe38c070890 0x7fe38c10f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:26:42.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.748+0000 7fe38be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe38c071e40 0x7fe38c10fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:26:42.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.748+0000 7fe38be00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe38c110430 con 0x7fe38c071e40 2024-03-30T04:26:42.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.748+0000 7fe38be00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe38c110c80 con 0x7fe38c070890 2024-03-30T04:26:42.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.748+0000 7fe38aa00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe38c070890 0x7fe38c10f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:26:42.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.748+0000 7fe38a000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe38c071e40 0x7fe38c10fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:26:42.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.748+0000 7fe38a000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe38c071e40 0x7fe38c10fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42076/0 (socket says 172.21.15.123:42076) 2024-03-30T04:26:42.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.748+0000 7fe38a000700 1 -- 172.21.15.123:0/177577474 learned_addr learned my addr 172.21.15.123:0/177577474 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:26:42.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.748+0000 7fe38a000700 1 -- 172.21.15.123:0/177577474 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe38c070890 msgr2=0x7fe38c10f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:26:42.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.749+0000 7fe38a000700 1 --2- 172.21.15.123:0/177577474 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe38c070890 0x7fe38c10f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:42.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.749+0000 7fe38a000700 1 -- 172.21.15.123:0/177577474 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe378005040 con 0x7fe38c071e40 2024-03-30T04:26:42.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.749+0000 7fe38aa00700 1 --2- 172.21.15.123:0/177577474 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe38c070890 0x7fe38c10f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:26:42.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.749+0000 7fe38a000700 1 --2- 172.21.15.123:0/177577474 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe38c071e40 0x7fe38c10fe30 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe3780137b0 tx=0x7fe378009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:26:42.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.749+0000 7fe382a00700 1 -- 172.21.15.123:0/177577474 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe37800e070 con 0x7fe38c071e40 2024-03-30T04:26:42.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.749+0000 7fe38be00700 1 -- 172.21.15.123:0/177577474 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe38c110f00 con 0x7fe38c071e40 2024-03-30T04:26:42.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.750+0000 7fe382a00700 1 -- 172.21.15.123:0/177577474 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe37801c400 con 0x7fe38c071e40 2024-03-30T04:26:42.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.750+0000 7fe382a00700 1 -- 172.21.15.123:0/177577474 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe37800b650 con 0x7fe38c071e40 2024-03-30T04:26:42.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.750+0000 7fe38be00700 1 -- 172.21.15.123:0/177577474 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe38c1113e0 con 0x7fe38c071e40 2024-03-30T04:26:42.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.751+0000 7fe38be00700 1 -- 172.21.15.123:0/177577474 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe38c05f5f0 con 0x7fe38c071e40 2024-03-30T04:26:42.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.753+0000 7fe382a00700 1 -- 172.21.15.123:0/177577474 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fe37801c950 con 0x7fe38c071e40 2024-03-30T04:26:42.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.754+0000 7fe382a00700 1 --2- 172.21.15.123:0/177577474 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fe3700692a0 0x7fe37006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:26:42.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.754+0000 7fe382a00700 1 -- 172.21.15.123:0/177577474 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fe3780887b0 con 0x7fe38c071e40 2024-03-30T04:26:42.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.755+0000 7fe38aa00700 1 --2- 172.21.15.123:0/177577474 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fe3700692a0 0x7fe37006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:26:42.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.756+0000 7fe38aa00700 1 --2- 172.21.15.123:0/177577474 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fe3700692a0 0x7fe37006b750 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe37c009b80 tx=0x7fe37c00b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:26:42.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:42.763+0000 7fe382a00700 1 -- 172.21.15.123:0/177577474 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fe378058500 con 0x7fe38c071e40 2024-03-30T04:26:43.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.031+0000 7fe38be00700 1 -- 172.21.15.123:0/177577474 --> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fe38c111820 con 0x7fe3700692a0 2024-03-30T04:26:43.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.041+0000 7fe382a00700 1 -- 172.21.15.123:0/177577474 <== mgr.14206 v2:172.21.15.123:6800/1564935593 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3410 (secure 0 0 0) 0x7fe38c111820 con 0x7fe3700692a0 2024-03-30T04:26:43.040 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:26:43.040 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (6m) 99s ago 8m 13.1M - 0.25.0 c8568f914cd2 0d2c80043347 2024-03-30T04:26:43.040 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (8m) 99s ago 8m 6647k - 17.2.7-797-g5c024335 f77085f6aab7 bceb8b8c4837 2024-03-30T04:26:43.040 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (7m) 104s ago 7m 6651k - 17.2.7-797-g5c024335 f77085f6aab7 cd0fe67b2d34 2024-03-30T04:26:43.040 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (6m) 99s ago 8m 74.0M - 9.4.7 954c08fa6188 b7d7eaffc084 2024-03-30T04:26:43.040 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (2m) 99s ago 2m 13.2M - 17.2.7-797-g5c024335 f77085f6aab7 2df066fba7a6 2024-03-30T04:26:43.040 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (2m) 99s ago 2m 11.5M - 17.2.7-797-g5c024335 f77085f6aab7 1293b8520db1 2024-03-30T04:26:43.040 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (117s) 104s ago 117s 11.6M - 17.2.7-797-g5c024335 f77085f6aab7 0dc41326818c 2024-03-30T04:26:43.040 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (2m) 104s ago 2m 13.0M - 17.2.7-797-g5c024335 f77085f6aab7 aef8a0cfc553 2024-03-30T04:26:43.040 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:9283 running (10m) 99s ago 10m 482M - 17.2.7-797-g5c024335 f77085f6aab7 7b98aee9a3b6 2024-03-30T04:26:43.040 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283 running (7m) 104s ago 7m 431M - 17.2.7-797-g5c024335 f77085f6aab7 37fc80673f76 2024-03-30T04:26:43.040 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (10m) 99s ago 10m 45.1M 2048M 17.2.7-797-g5c024335 f77085f6aab7 51d18587a7df 2024-03-30T04:26:43.041 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (7m) 104s ago 7m 32.3M 2048M 17.2.7-797-g5c024335 f77085f6aab7 f81052b09dd8 2024-03-30T04:26:43.041 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (8m) 99s ago 8m 7860k - 1.5.0 0da6a335fe13 9848cad1e995 2024-03-30T04:26:43.041 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (7m) 104s ago 7m 7545k - 1.5.0 0da6a335fe13 3c2c8e799cc2 2024-03-30T04:26:43.041 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (6m) 99s ago 6m 46.9M 4348M 17.2.7-797-g5c024335 f77085f6aab7 57a7ba3fd75e 2024-03-30T04:26:43.041 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (5m) 99s ago 5m 47.3M 4348M 17.2.7-797-g5c024335 f77085f6aab7 e1a6c1818a0c 2024-03-30T04:26:43.041 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (5m) 99s ago 5m 43.2M 4348M 17.2.7-797-g5c024335 f77085f6aab7 4c6a727567bd 2024-03-30T04:26:43.041 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (4m) 104s ago 4m 43.2M 5372M 17.2.7-797-g5c024335 f77085f6aab7 6fbdd1d46a9d 2024-03-30T04:26:43.041 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (4m) 104s ago 4m 40.4M 5372M 17.2.7-797-g5c024335 f77085f6aab7 b74988e947e5 2024-03-30T04:26:43.041 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (3m) 104s ago 3m 40.8M 5372M 17.2.7-797-g5c024335 f77085f6aab7 1fd3185f5484 2024-03-30T04:26:43.041 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (6m) 99s ago 7m 43.4M - 2.43.0 a07b618ecd1d ba3c351b14cf 2024-03-30T04:26:43.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.043+0000 7fe38be00700 1 -- 172.21.15.123:0/177577474 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fe3700692a0 msgr2=0x7fe37006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:26:43.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.043+0000 7fe38be00700 1 --2- 172.21.15.123:0/177577474 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fe3700692a0 0x7fe37006b750 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe37c009b80 tx=0x7fe37c00b040 comp rx=0 tx=0).stop 2024-03-30T04:26:43.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.043+0000 7fe38be00700 1 -- 172.21.15.123:0/177577474 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe38c071e40 msgr2=0x7fe38c10fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:26:43.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.043+0000 7fe38be00700 1 --2- 172.21.15.123:0/177577474 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe38c071e40 0x7fe38c10fe30 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe3780137b0 tx=0x7fe378009e30 comp rx=0 tx=0).stop 2024-03-30T04:26:43.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.043+0000 7fe38be00700 1 -- 172.21.15.123:0/177577474 shutdown_connections 2024-03-30T04:26:43.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.043+0000 7fe38be00700 1 --2- 172.21.15.123:0/177577474 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fe3700692a0 0x7fe37006b750 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:43.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.043+0000 7fe38be00700 1 --2- 172.21.15.123:0/177577474 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe38c070890 0x7fe38c10f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:43.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.043+0000 7fe38be00700 1 --2- 172.21.15.123:0/177577474 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe38c071e40 0x7fe38c10fe30 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:43.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.043+0000 7fe38be00700 1 -- 172.21.15.123:0/177577474 >> 172.21.15.123:0/177577474 conn(0x7fe38c0f8390 msgr2=0x7fe38c0f9a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:26:43.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.043+0000 7fe38be00700 1 -- 172.21.15.123:0/177577474 shutdown_connections 2024-03-30T04:26:43.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.044+0000 7fe38be00700 1 -- 172.21.15.123:0/177577474 wait complete. 2024-03-30T04:26:43.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.181+0000 7fc748400700 1 -- 172.21.15.123:0/1496474237 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc740071f50 msgr2=0x7fc740070460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:26:43.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.181+0000 7fc748400700 1 --2- 172.21.15.123:0/1496474237 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc740071f50 0x7fc740070460 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fc734004940 tx=0x7fc734009e30 comp rx=0 tx=0).stop 2024-03-30T04:26:43.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.181+0000 7fc748400700 1 -- 172.21.15.123:0/1496474237 shutdown_connections 2024-03-30T04:26:43.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.181+0000 7fc748400700 1 --2- 172.21.15.123:0/1496474237 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7400709a0 0x7fc740070e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:43.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.181+0000 7fc748400700 1 --2- 172.21.15.123:0/1496474237 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc740071f50 0x7fc740070460 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:43.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.181+0000 7fc748400700 1 -- 172.21.15.123:0/1496474237 >> 172.21.15.123:0/1496474237 conn(0x7fc7400f8560 msgr2=0x7fc7400fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:26:43.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.181+0000 7fc748400700 1 -- 172.21.15.123:0/1496474237 shutdown_connections 2024-03-30T04:26:43.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.181+0000 7fc748400700 1 -- 172.21.15.123:0/1496474237 wait complete. 2024-03-30T04:26:43.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.182+0000 7fc748400700 1 Processor -- start 2024-03-30T04:26:43.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.182+0000 7fc748400700 1 -- start start 2024-03-30T04:26:43.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.182+0000 7fc748400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7400709a0 0x7fc740113f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:26:43.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.182+0000 7fc748400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc740071f50 0x7fc7401144a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:26:43.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.182+0000 7fc748400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc740114aa0 con 0x7fc740071f50 2024-03-30T04:26:43.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.182+0000 7fc748400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7401152f0 con 0x7fc7400709a0 2024-03-30T04:26:43.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.183+0000 7fc745a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7400709a0 0x7fc740113f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:26:43.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.183+0000 7fc745000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc740071f50 0x7fc7401144a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:26:43.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.183+0000 7fc745a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7400709a0 0x7fc740113f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.123:46042/0 (socket says 172.21.15.123:46042) 2024-03-30T04:26:43.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.183+0000 7fc745a00700 1 -- 172.21.15.123:0/1488835318 learned_addr learned my addr 172.21.15.123:0/1488835318 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:26:43.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.183+0000 7fc745a00700 1 -- 172.21.15.123:0/1488835318 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc740071f50 msgr2=0x7fc7401144a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:26:43.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.183+0000 7fc745a00700 1 --2- 172.21.15.123:0/1488835318 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc740071f50 0x7fc7401144a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:43.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.183+0000 7fc745a00700 1 -- 172.21.15.123:0/1488835318 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc734005040 con 0x7fc7400709a0 2024-03-30T04:26:43.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.183+0000 7fc745000700 1 --2- 172.21.15.123:0/1488835318 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc740071f50 0x7fc7401144a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:26:43.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.183+0000 7fc745a00700 1 --2- 172.21.15.123:0/1488835318 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7400709a0 0x7fc740113f60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc73400b900 tx=0x7fc734009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:26:43.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.184+0000 7fc73d600700 1 -- 172.21.15.123:0/1488835318 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc73400e040 con 0x7fc7400709a0 2024-03-30T04:26:43.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.184+0000 7fc73d600700 1 -- 172.21.15.123:0/1488835318 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc734004290 con 0x7fc7400709a0 2024-03-30T04:26:43.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.184+0000 7fc748400700 1 -- 172.21.15.123:0/1488835318 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc740115570 con 0x7fc7400709a0 2024-03-30T04:26:43.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.184+0000 7fc73d600700 1 -- 172.21.15.123:0/1488835318 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc73401c3f0 con 0x7fc7400709a0 2024-03-30T04:26:43.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.184+0000 7fc748400700 1 -- 172.21.15.123:0/1488835318 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc740115ab0 con 0x7fc7400709a0 2024-03-30T04:26:43.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.185+0000 7fc748400700 1 -- 172.21.15.123:0/1488835318 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc74005f5f0 con 0x7fc7400709a0 2024-03-30T04:26:43.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.186+0000 7fc73d600700 1 -- 172.21.15.123:0/1488835318 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fc734013420 con 0x7fc7400709a0 2024-03-30T04:26:43.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.187+0000 7fc73d600700 1 --2- 172.21.15.123:0/1488835318 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fc7280692b0 0x7fc72806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:26:43.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.187+0000 7fc73d600700 1 -- 172.21.15.123:0/1488835318 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fc734088520 con 0x7fc7400709a0 2024-03-30T04:26:43.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.187+0000 7fc745000700 1 --2- 172.21.15.123:0/1488835318 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fc7280692b0 0x7fc72806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:26:43.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.188+0000 7fc745000700 1 --2- 172.21.15.123:0/1488835318 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fc7280692b0 0x7fc72806b760 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc72c010520 tx=0x7fc72c00b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:26:43.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.196+0000 7fc73d600700 1 -- 172.21.15.123:0/1488835318 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fc7340581f0 con 0x7fc7400709a0 2024-03-30T04:26:43.304 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:43 smithi135 ceph-mon[44364]: from='client.24403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:26:43.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:43 smithi123 ceph-mon[35245]: from='client.24403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:26:43.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.484+0000 7fc748400700 1 -- 172.21.15.123:0/1488835318 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc740115ea0 con 0x7fc7400709a0 2024-03-30T04:26:43.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.485+0000 7fc73d600700 1 -- 172.21.15.123:0/1488835318 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7fc734086e60 con 0x7fc7400709a0 2024-03-30T04:26:43.483 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:26:43.483 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:26:43.483 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 2 2024-03-30T04:26:43.484 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:26:43.484 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:26:43.484 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 2 2024-03-30T04:26:43.484 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:26:43.484 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:26:43.484 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 6 2024-03-30T04:26:43.484 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:26:43.484 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:26:43.484 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 4 2024-03-30T04:26:43.484 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:26:43.484 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:26:43.484 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 14 2024-03-30T04:26:43.484 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:26:43.484 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:26:43.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.487+0000 7fc748400700 1 -- 172.21.15.123:0/1488835318 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fc7280692b0 msgr2=0x7fc72806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:26:43.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.487+0000 7fc748400700 1 --2- 172.21.15.123:0/1488835318 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fc7280692b0 0x7fc72806b760 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc72c010520 tx=0x7fc72c00b040 comp rx=0 tx=0).stop 2024-03-30T04:26:43.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.487+0000 7fc748400700 1 -- 172.21.15.123:0/1488835318 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7400709a0 msgr2=0x7fc740113f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:26:43.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.487+0000 7fc748400700 1 --2- 172.21.15.123:0/1488835318 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7400709a0 0x7fc740113f60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc73400b900 tx=0x7fc734009e30 comp rx=0 tx=0).stop 2024-03-30T04:26:43.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.487+0000 7fc748400700 1 -- 172.21.15.123:0/1488835318 shutdown_connections 2024-03-30T04:26:43.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.487+0000 7fc748400700 1 --2- 172.21.15.123:0/1488835318 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fc7280692b0 0x7fc72806b760 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:43.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.487+0000 7fc748400700 1 --2- 172.21.15.123:0/1488835318 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7400709a0 0x7fc740113f60 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:43.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.487+0000 7fc748400700 1 --2- 172.21.15.123:0/1488835318 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc740071f50 0x7fc7401144a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:43.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.487+0000 7fc748400700 1 -- 172.21.15.123:0/1488835318 >> 172.21.15.123:0/1488835318 conn(0x7fc7400f8560 msgr2=0x7fc7400f9d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:26:43.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.487+0000 7fc748400700 1 -- 172.21.15.123:0/1488835318 shutdown_connections 2024-03-30T04:26:43.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.487+0000 7fc748400700 1 -- 172.21.15.123:0/1488835318 wait complete. 2024-03-30T04:26:43.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.628+0000 7fc096e00700 1 -- 172.21.15.123:0/3269555860 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0900fffa0 msgr2=0x7fc090102380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:26:43.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.628+0000 7fc096e00700 1 --2- 172.21.15.123:0/3269555860 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0900fffa0 0x7fc090102380 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc07c004940 tx=0x7fc07c009e30 comp rx=0 tx=0).stop 2024-03-30T04:26:43.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.628+0000 7fc096e00700 1 -- 172.21.15.123:0/3269555860 shutdown_connections 2024-03-30T04:26:43.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.628+0000 7fc096e00700 1 --2- 172.21.15.123:0/3269555860 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0900fffa0 0x7fc090102380 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:43.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.628+0000 7fc096e00700 1 --2- 172.21.15.123:0/3269555860 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc090066ad0 0x7fc0900ffa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:43.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.628+0000 7fc096e00700 1 -- 172.21.15.123:0/3269555860 >> 172.21.15.123:0/3269555860 conn(0x7fc0900f7410 msgr2=0x7fc0900f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:26:43.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.628+0000 7fc096e00700 1 -- 172.21.15.123:0/3269555860 shutdown_connections 2024-03-30T04:26:43.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.628+0000 7fc096e00700 1 -- 172.21.15.123:0/3269555860 wait complete. 2024-03-30T04:26:43.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.629+0000 7fc096e00700 1 Processor -- start 2024-03-30T04:26:43.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.629+0000 7fc096e00700 1 -- start start 2024-03-30T04:26:43.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.629+0000 7fc096e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc090066ad0 0x7fc09010fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:26:43.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.629+0000 7fc096e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc0900fffa0 0x7fc090110020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:26:43.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.629+0000 7fc096e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc090110620 con 0x7fc090066ad0 2024-03-30T04:26:43.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.629+0000 7fc096e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc090110e70 con 0x7fc0900fffa0 2024-03-30T04:26:43.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.630+0000 7fc08fe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc090066ad0 0x7fc09010fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:26:43.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.630+0000 7fc08fe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc090066ad0 0x7fc09010fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42116/0 (socket says 172.21.15.123:42116) 2024-03-30T04:26:43.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.630+0000 7fc08fe00700 1 -- 172.21.15.123:0/1065854164 learned_addr learned my addr 172.21.15.123:0/1065854164 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:26:43.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.630+0000 7fc08f400700 1 --2- 172.21.15.123:0/1065854164 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc0900fffa0 0x7fc090110020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:26:43.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.630+0000 7fc08fe00700 1 -- 172.21.15.123:0/1065854164 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc0900fffa0 msgr2=0x7fc090110020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:26:43.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.630+0000 7fc08fe00700 1 --2- 172.21.15.123:0/1065854164 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc0900fffa0 0x7fc090110020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:43.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.630+0000 7fc08fe00700 1 -- 172.21.15.123:0/1065854164 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc07c005040 con 0x7fc090066ad0 2024-03-30T04:26:43.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.630+0000 7fc08fe00700 1 --2- 172.21.15.123:0/1065854164 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc090066ad0 0x7fc09010fae0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fc080002a80 tx=0x7fc0800046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:26:43.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.631+0000 7fc087e00700 1 -- 172.21.15.123:0/1065854164 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc080007cc0 con 0x7fc090066ad0 2024-03-30T04:26:43.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.631+0000 7fc087e00700 1 -- 172.21.15.123:0/1065854164 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc08000f040 con 0x7fc090066ad0 2024-03-30T04:26:43.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.631+0000 7fc096e00700 1 -- 172.21.15.123:0/1065854164 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc090111150 con 0x7fc090066ad0 2024-03-30T04:26:43.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.631+0000 7fc087e00700 1 -- 172.21.15.123:0/1065854164 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc08000d9e0 con 0x7fc090066ad0 2024-03-30T04:26:43.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.631+0000 7fc096e00700 1 -- 172.21.15.123:0/1065854164 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc090111690 con 0x7fc090066ad0 2024-03-30T04:26:43.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.632+0000 7fc096e00700 1 -- 172.21.15.123:0/1065854164 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0900f8fb0 con 0x7fc090066ad0 2024-03-30T04:26:43.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.633+0000 7fc087e00700 1 -- 172.21.15.123:0/1065854164 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fc08000dc50 con 0x7fc090066ad0 2024-03-30T04:26:43.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.633+0000 7fc087e00700 1 --2- 172.21.15.123:0/1065854164 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fc0740692b0 0x7fc07406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:26:43.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.633+0000 7fc087e00700 1 -- 172.21.15.123:0/1065854164 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fc080086a60 con 0x7fc090066ad0 2024-03-30T04:26:43.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.634+0000 7fc08f400700 1 --2- 172.21.15.123:0/1065854164 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fc0740692b0 0x7fc07406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:26:43.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.634+0000 7fc08f400700 1 --2- 172.21.15.123:0/1065854164 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fc0740692b0 0x7fc07406b760 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc07c004f90 tx=0x7fc07c009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:26:43.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.638+0000 7fc087e00700 1 -- 172.21.15.123:0/1065854164 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fc08004d190 con 0x7fc090066ad0 2024-03-30T04:26:43.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.861+0000 7fc096e00700 1 -- 172.21.15.123:0/1065854164 --> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc0901118b0 con 0x7fc0740692b0 2024-03-30T04:26:43.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.862+0000 7fc087e00700 1 -- 172.21.15.123:0/1065854164 <== mgr.14206 v2:172.21.15.123:6800/1564935593 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7fc0901118b0 con 0x7fc0740692b0 2024-03-30T04:26:43.861 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:26:43.861 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:26:43.861 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:26:43.861 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-30T04:26:43.861 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [], 2024-03-30T04:26:43.861 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "0/2 daemons upgraded", 2024-03-30T04:26:43.861 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mgr daemons", 2024-03-30T04:26:43.861 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:26:43.861 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:26:43.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.864+0000 7fc096e00700 1 -- 172.21.15.123:0/1065854164 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fc0740692b0 msgr2=0x7fc07406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:26:43.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.864+0000 7fc096e00700 1 --2- 172.21.15.123:0/1065854164 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fc0740692b0 0x7fc07406b760 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc07c004f90 tx=0x7fc07c009e30 comp rx=0 tx=0).stop 2024-03-30T04:26:43.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.864+0000 7fc096e00700 1 -- 172.21.15.123:0/1065854164 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc090066ad0 msgr2=0x7fc09010fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:26:43.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.864+0000 7fc096e00700 1 --2- 172.21.15.123:0/1065854164 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc090066ad0 0x7fc09010fae0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fc080002a80 tx=0x7fc0800046b0 comp rx=0 tx=0).stop 2024-03-30T04:26:43.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.864+0000 7fc096e00700 1 -- 172.21.15.123:0/1065854164 shutdown_connections 2024-03-30T04:26:43.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.864+0000 7fc096e00700 1 --2- 172.21.15.123:0/1065854164 >> [v2:172.21.15.123:6800/1564935593,v1:172.21.15.123:6801/1564935593] conn(0x7fc0740692b0 0x7fc07406b760 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:43.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.864+0000 7fc096e00700 1 --2- 172.21.15.123:0/1065854164 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc090066ad0 0x7fc09010fae0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:43.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.864+0000 7fc096e00700 1 --2- 172.21.15.123:0/1065854164 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc0900fffa0 0x7fc090110020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:26:43.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.864+0000 7fc096e00700 1 -- 172.21.15.123:0/1065854164 >> 172.21.15.123:0/1065854164 conn(0x7fc0900f7410 msgr2=0x7fc0900fbba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:26:43.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.865+0000 7fc096e00700 1 -- 172.21.15.123:0/1065854164 shutdown_connections 2024-03-30T04:26:43.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:26:43.865+0000 7fc096e00700 1 -- 172.21.15.123:0/1065854164 wait complete. 2024-03-30T04:26:44.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:44 smithi135 ceph-mon[44364]: pgmap v267: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-30T04:26:44.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:44 smithi135 ceph-mon[44364]: from='client.24407 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:26:44.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:44 smithi135 ceph-mon[44364]: from='client.14596 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:26:44.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:44 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/1488835318' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:26:44.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:44 smithi123 ceph-mon[35245]: pgmap v267: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-30T04:26:44.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:44 smithi123 ceph-mon[35245]: from='client.24407 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:26:44.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:44 smithi123 ceph-mon[35245]: from='client.14596 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:26:44.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:44 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/1488835318' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:26:45.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:45 smithi135 ceph-mon[44364]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:26:45.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:45 smithi123 ceph-mon[35245]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:26:46.320 INFO:tasks.workunit.client.0.smithi123.stderr:Updating files: 64% (7876/12271) Updating files: 65% (7977/12271) Updating files: 66% (8099/12271) Updating files: 67% (8222/12271) Updating files: 68% (8345/12271) Updating files: 69% (8467/12271) Updating files: 70% (8590/12271) Updating files: 71% (8713/12271) Updating files: 72% (8836/12271) Updating files: 73% (8958/12271) Updating files: 74% (9081/12271) Updating files: 75% (9204/12271) Updating files: 76% (9326/12271) Updating files: 77% (9449/12271) Updating files: 78% (9572/12271) Updating files: 79% (9695/12271) Updating files: 80% (9817/12271) Updating files: 81% (9940/12271) Updating files: 82% (10063/12271) Updating files: 83% (10185/12271) Updating files: 84% (10308/12271) Updating files: 85% (10431/12271) Updating files: 86% (10554/12271) Updating files: 87% (10676/12271) Updating files: 88% (10799/12271) Updating files: 89% (10922/12271) Updating files: 90% (11044/12271) Updating files: 91% (11167/12271) Updating files: 92% (11290/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-03-30T04:26:46.343 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:46 smithi123 ceph-mon[35245]: pgmap v268: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:26:46.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:46 smithi135 ceph-mon[44364]: pgmap v268: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:26:47.167 INFO:tasks.workunit.client.0.smithi123.stderr:Note: switching to '470100a4ccbcf7769052bf09a5413881ccf1b97f'. 2024-03-30T04:26:47.167 INFO:tasks.workunit.client.0.smithi123.stderr: 2024-03-30T04:26:47.167 INFO:tasks.workunit.client.0.smithi123.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-30T04:26:47.167 INFO:tasks.workunit.client.0.smithi123.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-30T04:26:47.167 INFO:tasks.workunit.client.0.smithi123.stderr:state without impacting any branches by switching back to a branch. 2024-03-30T04:26:47.167 INFO:tasks.workunit.client.0.smithi123.stderr: 2024-03-30T04:26:47.167 INFO:tasks.workunit.client.0.smithi123.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-30T04:26:47.167 INFO:tasks.workunit.client.0.smithi123.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-30T04:26:47.167 INFO:tasks.workunit.client.0.smithi123.stderr: 2024-03-30T04:26:47.168 INFO:tasks.workunit.client.0.smithi123.stderr: git switch -c 2024-03-30T04:26:47.168 INFO:tasks.workunit.client.0.smithi123.stderr: 2024-03-30T04:26:47.168 INFO:tasks.workunit.client.0.smithi123.stderr:Or undo this operation with: 2024-03-30T04:26:47.168 INFO:tasks.workunit.client.0.smithi123.stderr: 2024-03-30T04:26:47.168 INFO:tasks.workunit.client.0.smithi123.stderr: git switch - 2024-03-30T04:26:47.168 INFO:tasks.workunit.client.0.smithi123.stderr: 2024-03-30T04:26:47.168 INFO:tasks.workunit.client.0.smithi123.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-30T04:26:47.168 INFO:tasks.workunit.client.0.smithi123.stderr: 2024-03-30T04:26:47.168 INFO:tasks.workunit.client.0.smithi123.stderr:HEAD is now at 470100a4ccb DNM: trigger builds again 2024-03-30T04:26:47.182 DEBUG:teuthology.orchestra.run.smithi123:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-30T04:26:47.240 INFO:tasks.workunit.client.0.smithi123.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-30T04:26:47.246 INFO:tasks.workunit.client.0.smithi123.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-30T04:26:47.246 INFO:tasks.workunit.client.0.smithi123.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-30T04:26:47.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:47 smithi135 ceph-mon[44364]: pgmap v269: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:26:47.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:47 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:26:47.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:47 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:26:47.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:47 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:26:47.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:47 smithi123 ceph-mon[35245]: pgmap v269: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:26:47.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:47 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:26:47.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:47 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:26:47.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:47 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:26:47.936 INFO:tasks.workunit.client.0.smithi123.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-30T04:26:48.021 INFO:tasks.workunit.client.0.smithi123.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-30T04:26:48.085 INFO:tasks.workunit.client.0.smithi123.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-30T04:26:48.089 INFO:tasks.workunit.client.0.smithi123.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-30T04:26:48.089 INFO:tasks.workunit.client.0.smithi123.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-30T04:26:48.152 INFO:tasks.workunit.client.0.smithi123.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-30T04:26:48.160 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-30T04:26:48.160 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-30T04:26:48.204 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.0... 2024-03-30T04:26:48.207 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-03-30T04:26:48.207 DEBUG:teuthology.orchestra.run.smithi123:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=470100a4ccbcf7769052bf09a5413881ccf1b97f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2024-03-30T04:26:48.310 INFO:tasks.workunit.client.0.smithi123.stderr:+ mkdir -p fsstress 2024-03-30T04:26:48.314 INFO:tasks.workunit.client.0.smithi123.stderr:+ pushd fsstress 2024-03-30T04:26:48.317 INFO:tasks.workunit.client.0.smithi123.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-03-30T04:26:48.317 INFO:tasks.workunit.client.0.smithi123.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-03-30T04:26:49.135 INFO:tasks.workunit.client.0.smithi123.stderr:+ tar xzf ltp-full.tgz 2024-03-30T04:26:49.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:49 smithi135 ceph-mon[44364]: pgmap v270: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:26:49.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:49 smithi123 ceph-mon[35245]: pgmap v270: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:26:52.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:52 smithi135 ceph-mon[44364]: pgmap v271: 65 pgs: 65 active+clean; 3.3 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.3 KiB/s rd, 234 KiB/s wr, 12 op/s 2024-03-30T04:26:52.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:52 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:26:52.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:52 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:26:52.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:52 smithi123 ceph-mon[35245]: pgmap v271: 65 pgs: 65 active+clean; 3.3 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.3 KiB/s rd, 234 KiB/s wr, 12 op/s 2024-03-30T04:26:52.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:52 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:26:52.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:52 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:26:53.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:53 smithi135 ceph-mon[44364]: pgmap v272: 65 pgs: 65 active+clean; 8.1 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.3 KiB/s rd, 662 KiB/s wr, 30 op/s 2024-03-30T04:26:53.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:53 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:26:53.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:53 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:26:54.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:53 smithi123 ceph-mon[35245]: pgmap v272: 65 pgs: 65 active+clean; 8.1 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.3 KiB/s rd, 662 KiB/s wr, 30 op/s 2024-03-30T04:26:54.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:53 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:26:54.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:53 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:26:56.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:55 smithi135 ceph-mon[44364]: pgmap v273: 65 pgs: 65 active+clean; 13 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.9 KiB/s rd, 1.0 MiB/s wr, 42 op/s 2024-03-30T04:26:56.172 INFO:tasks.workunit.client.1.smithi135.stderr:Updating files: 57% (7102/12271) Updating files: 58% (7118/12271) Updating files: 59% (7240/12271) Updating files: 60% (7363/12271) Updating files: 61% (7486/12271) Updating files: 62% (7609/12271) Updating files: 63% (7731/12271) Updating files: 64% (7854/12271) Updating files: 65% (7977/12271) Updating files: 66% (8099/12271) Updating files: 67% (8222/12271) Updating files: 68% (8345/12271) Updating files: 69% (8467/12271) Updating files: 70% (8590/12271) Updating files: 71% (8713/12271) Updating files: 72% (8836/12271) Updating files: 73% (8958/12271) Updating files: 74% (9081/12271) Updating files: 75% (9204/12271) Updating files: 76% (9326/12271) Updating files: 77% (9449/12271) Updating files: 78% (9572/12271) Updating files: 79% (9695/12271) Updating files: 80% (9817/12271) Updating files: 81% (9940/12271) Updating files: 82% (10063/12271) Updating files: 83% (10185/12271) Updating files: 84% (10308/12271) Updating files: 85% (10431/12271) Updating files: 86% (10554/12271) Updating files: 87% (10676/12271) Updating files: 88% (10799/12271) Updating files: 89% (10922/12271) Updating files: 90% (11044/12271) Updating files: 91% (11167/12271) Updating files: 91% (11229/12271) Updating files: 92% (11290/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-03-30T04:26:56.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:55 smithi123 ceph-mon[35245]: pgmap v273: 65 pgs: 65 active+clean; 13 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.9 KiB/s rd, 1.0 MiB/s wr, 42 op/s 2024-03-30T04:26:57.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:56 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:26:57.113 INFO:tasks.workunit.client.1.smithi135.stderr:Note: switching to '470100a4ccbcf7769052bf09a5413881ccf1b97f'. 2024-03-30T04:26:57.114 INFO:tasks.workunit.client.1.smithi135.stderr: 2024-03-30T04:26:57.114 INFO:tasks.workunit.client.1.smithi135.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-30T04:26:57.114 INFO:tasks.workunit.client.1.smithi135.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-30T04:26:57.114 INFO:tasks.workunit.client.1.smithi135.stderr:state without impacting any branches by switching back to a branch. 2024-03-30T04:26:57.114 INFO:tasks.workunit.client.1.smithi135.stderr: 2024-03-30T04:26:57.114 INFO:tasks.workunit.client.1.smithi135.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-30T04:26:57.114 INFO:tasks.workunit.client.1.smithi135.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-30T04:26:57.114 INFO:tasks.workunit.client.1.smithi135.stderr: 2024-03-30T04:26:57.114 INFO:tasks.workunit.client.1.smithi135.stderr: git switch -c 2024-03-30T04:26:57.114 INFO:tasks.workunit.client.1.smithi135.stderr: 2024-03-30T04:26:57.114 INFO:tasks.workunit.client.1.smithi135.stderr:Or undo this operation with: 2024-03-30T04:26:57.114 INFO:tasks.workunit.client.1.smithi135.stderr: 2024-03-30T04:26:57.115 INFO:tasks.workunit.client.1.smithi135.stderr: git switch - 2024-03-30T04:26:57.115 INFO:tasks.workunit.client.1.smithi135.stderr: 2024-03-30T04:26:57.115 INFO:tasks.workunit.client.1.smithi135.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-30T04:26:57.115 INFO:tasks.workunit.client.1.smithi135.stderr: 2024-03-30T04:26:57.115 INFO:tasks.workunit.client.1.smithi135.stderr:HEAD is now at 470100a4ccb DNM: trigger builds again 2024-03-30T04:26:57.137 DEBUG:teuthology.orchestra.run.smithi135:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-03-30T04:26:57.194 INFO:tasks.workunit.client.1.smithi135.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-30T04:26:57.199 INFO:tasks.workunit.client.1.smithi135.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-03-30T04:26:57.199 INFO:tasks.workunit.client.1.smithi135.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-30T04:26:57.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:56 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:26:58.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:57 smithi135 ceph-mon[44364]: pgmap v274: 65 pgs: 65 active+clean; 18 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 167 KiB/s rd, 1.5 MiB/s wr, 115 op/s 2024-03-30T04:26:58.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:57 smithi135 ceph-mon[44364]: Standby manager daemon smithi135.kyblli restarted 2024-03-30T04:26:58.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:57 smithi135 ceph-mon[44364]: Standby manager daemon smithi135.kyblli started 2024-03-30T04:26:58.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:57 smithi135 ceph-mon[44364]: from='mgr.? 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi135.kyblli/crt"}]: dispatch 2024-03-30T04:26:58.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:57 smithi135 ceph-mon[44364]: from='mgr.? 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T04:26:58.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:57 smithi135 ceph-mon[44364]: from='mgr.? 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi135.kyblli/key"}]: dispatch 2024-03-30T04:26:58.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:57 smithi135 ceph-mon[44364]: from='mgr.? 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T04:26:58.133 INFO:tasks.workunit.client.1.smithi135.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-30T04:26:58.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:57 smithi123 ceph-mon[35245]: pgmap v274: 65 pgs: 65 active+clean; 18 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 167 KiB/s rd, 1.5 MiB/s wr, 115 op/s 2024-03-30T04:26:58.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:57 smithi123 ceph-mon[35245]: Standby manager daemon smithi135.kyblli restarted 2024-03-30T04:26:58.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:57 smithi123 ceph-mon[35245]: Standby manager daemon smithi135.kyblli started 2024-03-30T04:26:58.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:57 smithi123 ceph-mon[35245]: from='mgr.? 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi135.kyblli/crt"}]: dispatch 2024-03-30T04:26:58.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:57 smithi123 ceph-mon[35245]: from='mgr.? 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T04:26:58.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:57 smithi123 ceph-mon[35245]: from='mgr.? 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi135.kyblli/key"}]: dispatch 2024-03-30T04:26:58.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:57 smithi123 ceph-mon[35245]: from='mgr.? 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T04:26:58.274 INFO:tasks.workunit.client.1.smithi135.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-30T04:26:58.342 INFO:tasks.workunit.client.1.smithi135.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-03-30T04:26:58.346 INFO:tasks.workunit.client.1.smithi135.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-03-30T04:26:58.346 INFO:tasks.workunit.client.1.smithi135.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-30T04:26:58.415 INFO:tasks.workunit.client.1.smithi135.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-03-30T04:26:58.422 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-30T04:26:58.422 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-03-30T04:26:58.467 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.1... 2024-03-30T04:26:58.468 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-03-30T04:26:58.468 DEBUG:teuthology.orchestra.run.smithi135:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=470100a4ccbcf7769052bf09a5413881ccf1b97f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh 2024-03-30T04:26:58.576 INFO:tasks.workunit.client.1.smithi135.stderr:+ mkdir -p fsstress 2024-03-30T04:26:58.581 INFO:tasks.workunit.client.1.smithi135.stderr:+ pushd fsstress 2024-03-30T04:26:58.583 INFO:tasks.workunit.client.1.smithi135.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-03-30T04:26:58.583 INFO:tasks.workunit.client.1.smithi135.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-03-30T04:26:59.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:58 smithi135 ceph-mon[44364]: mgrmap e20: smithi123.xgksqq(active, since 8m), standbys: smithi135.kyblli 2024-03-30T04:26:59.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:58 smithi135 ceph-mon[44364]: fsmap cephfs:2 {0=cephfs.smithi123.dvtind=up:active,1=cephfs.smithi135.zvnlap=up:active} 2 up:standby-replay 2024-03-30T04:26:59.199 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:58 smithi123 ceph-mon[35245]: mgrmap e20: smithi123.xgksqq(active, since 8m), standbys: smithi135.kyblli 2024-03-30T04:26:59.199 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:58 smithi123 ceph-mon[35245]: fsmap cephfs:2 {0=cephfs.smithi123.dvtind=up:active,1=cephfs.smithi135.zvnlap=up:active} 2 up:standby-replay 2024-03-30T04:26:59.568 INFO:tasks.workunit.client.1.smithi135.stderr:+ tar xzf ltp-full.tgz 2024-03-30T04:27:00.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:59 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:27:00.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:59 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:27:00.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:59 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:27:00.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:59 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:27:00.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:59 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:27:00.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:59 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:27:00.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:59 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mgr fail", "who": "smithi123.xgksqq"}]: dispatch 2024-03-30T04:27:00.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:59 smithi135 ceph-mon[44364]: osdmap e47: 6 total, 6 up, 6 in 2024-03-30T04:27:00.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:59 smithi135 ceph-mon[44364]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "mgr fail", "who": "smithi123.xgksqq"}]': finished 2024-03-30T04:27:00.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:59 smithi135 ceph-mon[44364]: mgrmap e21: smithi135.kyblli(active, starting, since 0.0118201s) 2024-03-30T04:27:00.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:59 smithi135 ceph-mon[44364]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "mon metadata", "id": "smithi123"}]: dispatch 2024-03-30T04:27:00.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:59 smithi135 ceph-mon[44364]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-03-30T04:27:00.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:59 smithi135 ceph-mon[44364]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi123.dvtind"}]: dispatch 2024-03-30T04:27:00.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:59 smithi135 ceph-mon[44364]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi123.tqxvzr"}]: dispatch 2024-03-30T04:27:00.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:59 smithi135 ceph-mon[44364]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi135.zvnlap"}]: dispatch 2024-03-30T04:27:00.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:59 smithi135 ceph-mon[44364]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi135.xwgjvb"}]: dispatch 2024-03-30T04:27:00.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:59 smithi135 ceph-mon[44364]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "mgr metadata", "who": "smithi135.kyblli", "id": "smithi135.kyblli"}]: dispatch 2024-03-30T04:27:00.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:59 smithi135 ceph-mon[44364]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T04:27:00.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:59 smithi135 ceph-mon[44364]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T04:27:00.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:59 smithi135 ceph-mon[44364]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T04:27:00.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:59 smithi135 ceph-mon[44364]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T04:27:00.103 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:59 smithi135 ceph-mon[44364]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T04:27:00.103 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:59 smithi135 ceph-mon[44364]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T04:27:00.103 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:59 smithi135 ceph-mon[44364]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-30T04:27:00.103 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:59 smithi135 ceph-mon[44364]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-30T04:27:00.103 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:59 smithi135 ceph-mon[44364]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-30T04:27:00.103 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:59 smithi135 ceph-mon[44364]: Manager daemon smithi135.kyblli is now available 2024-03-30T04:27:00.103 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:59 smithi135 ceph-mon[44364]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:27:00.103 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:59 smithi135 ceph-mon[44364]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:27:00.103 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:59 smithi135 ceph-mon[44364]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi135.kyblli/mirror_snapshot_schedule"}]: dispatch 2024-03-30T04:27:00.103 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:59 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi135.kyblli/mirror_snapshot_schedule"}]: dispatch 2024-03-30T04:27:00.103 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:59 smithi135 ceph-mon[44364]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi135.kyblli/trash_purge_schedule"}]: dispatch 2024-03-30T04:27:00.103 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:59 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi135.kyblli/trash_purge_schedule"}]: dispatch 2024-03-30T04:27:00.103 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:59 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:00.103 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:26:59 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:00.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:59 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:27:00.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:59 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:27:00.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:59 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:27:00.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:59 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:27:00.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:59 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' 2024-03-30T04:27:00.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:59 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:27:00.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:59 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mgr fail", "who": "smithi123.xgksqq"}]: dispatch 2024-03-30T04:27:00.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:59 smithi123 ceph-mon[35245]: osdmap e47: 6 total, 6 up, 6 in 2024-03-30T04:27:00.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:59 smithi123 ceph-mon[35245]: from='mgr.14206 172.21.15.123:0/1843388385' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "mgr fail", "who": "smithi123.xgksqq"}]': finished 2024-03-30T04:27:00.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:59 smithi123 ceph-mon[35245]: mgrmap e21: smithi135.kyblli(active, starting, since 0.0118201s) 2024-03-30T04:27:00.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:59 smithi123 ceph-mon[35245]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "mon metadata", "id": "smithi123"}]: dispatch 2024-03-30T04:27:00.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:59 smithi123 ceph-mon[35245]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-03-30T04:27:00.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:59 smithi123 ceph-mon[35245]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi123.dvtind"}]: dispatch 2024-03-30T04:27:00.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:59 smithi123 ceph-mon[35245]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi123.tqxvzr"}]: dispatch 2024-03-30T04:27:00.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:59 smithi123 ceph-mon[35245]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi135.zvnlap"}]: dispatch 2024-03-30T04:27:00.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:59 smithi123 ceph-mon[35245]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi135.xwgjvb"}]: dispatch 2024-03-30T04:27:00.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:59 smithi123 ceph-mon[35245]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "mgr metadata", "who": "smithi135.kyblli", "id": "smithi135.kyblli"}]: dispatch 2024-03-30T04:27:00.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:59 smithi123 ceph-mon[35245]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T04:27:00.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:59 smithi123 ceph-mon[35245]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T04:27:00.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:59 smithi123 ceph-mon[35245]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T04:27:00.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:59 smithi123 ceph-mon[35245]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T04:27:00.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:59 smithi123 ceph-mon[35245]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T04:27:00.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:59 smithi123 ceph-mon[35245]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T04:27:00.219 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:59 smithi123 ceph-mon[35245]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-30T04:27:00.219 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:59 smithi123 ceph-mon[35245]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-30T04:27:00.219 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:59 smithi123 ceph-mon[35245]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-30T04:27:00.219 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:59 smithi123 ceph-mon[35245]: Manager daemon smithi135.kyblli is now available 2024-03-30T04:27:00.219 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:59 smithi123 ceph-mon[35245]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:27:00.219 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:59 smithi123 ceph-mon[35245]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:27:00.219 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:59 smithi123 ceph-mon[35245]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi135.kyblli/mirror_snapshot_schedule"}]: dispatch 2024-03-30T04:27:00.219 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:59 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi135.kyblli/mirror_snapshot_schedule"}]: dispatch 2024-03-30T04:27:00.219 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:59 smithi123 ceph-mon[35245]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi135.kyblli/trash_purge_schedule"}]: dispatch 2024-03-30T04:27:00.219 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:59 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi135.kyblli/trash_purge_schedule"}]: dispatch 2024-03-30T04:27:00.219 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:59 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:00.219 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:26:59 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:01.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:01 smithi123 ceph-mon[35245]: Deploying cephadm binary to smithi135 2024-03-30T04:27:01.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:01 smithi123 ceph-mon[35245]: Deploying cephadm binary to smithi123 2024-03-30T04:27:01.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:01 smithi123 ceph-mon[35245]: [30/Mar/2024:04:27:00] ENGINE Bus STARTING 2024-03-30T04:27:01.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:01 smithi123 ceph-mon[35245]: mgrmap e22: smithi135.kyblli(active, since 1.01513s) 2024-03-30T04:27:01.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:01 smithi123 ceph-mon[35245]: pgmap v3: 65 pgs: 65 active+clean; 38 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:27:01.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:01 smithi135 ceph-mon[44364]: Deploying cephadm binary to smithi135 2024-03-30T04:27:01.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:01 smithi135 ceph-mon[44364]: Deploying cephadm binary to smithi123 2024-03-30T04:27:01.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:01 smithi135 ceph-mon[44364]: [30/Mar/2024:04:27:00] ENGINE Bus STARTING 2024-03-30T04:27:01.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:01 smithi135 ceph-mon[44364]: mgrmap e22: smithi135.kyblli(active, since 1.01513s) 2024-03-30T04:27:01.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:01 smithi135 ceph-mon[44364]: pgmap v3: 65 pgs: 65 active+clean; 38 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:27:02.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:02 smithi135 ceph-mon[44364]: [30/Mar/2024:04:27:00] ENGINE Serving on https://172.21.15.135:7150 2024-03-30T04:27:02.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:02 smithi135 ceph-mon[44364]: [30/Mar/2024:04:27:00] ENGINE Serving on http://172.21.15.135:8765 2024-03-30T04:27:02.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:02 smithi135 ceph-mon[44364]: [30/Mar/2024:04:27:00] ENGINE Bus STARTED 2024-03-30T04:27:02.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:02 smithi135 ceph-mon[44364]: pgmap v4: 65 pgs: 65 active+clean; 38 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:27:02.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:02 smithi123 ceph-mon[35245]: [30/Mar/2024:04:27:00] ENGINE Serving on https://172.21.15.135:7150 2024-03-30T04:27:02.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:02 smithi123 ceph-mon[35245]: [30/Mar/2024:04:27:00] ENGINE Serving on http://172.21.15.135:8765 2024-03-30T04:27:02.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:02 smithi123 ceph-mon[35245]: [30/Mar/2024:04:27:00] ENGINE Bus STARTED 2024-03-30T04:27:02.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:02 smithi123 ceph-mon[35245]: pgmap v4: 65 pgs: 65 active+clean; 38 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:27:03.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:03 smithi123 ceph-mon[35245]: mgrmap e23: smithi135.kyblli(active, since 3s) 2024-03-30T04:27:03.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:03 smithi135 ceph-mon[44364]: mgrmap e23: smithi135.kyblli(active, since 3s) 2024-03-30T04:27:04.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:04 smithi123 ceph-mon[35245]: pgmap v5: 65 pgs: 65 active+clean; 38 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:27:04.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:04 smithi135 ceph-mon[44364]: pgmap v5: 65 pgs: 65 active+clean; 38 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-03-30T04:27:06.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:06 smithi123 ceph-mon[35245]: pgmap v6: 65 pgs: 65 active+clean; 43 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 196 KiB/s rd, 1.1 MiB/s wr, 117 op/s 2024-03-30T04:27:06.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:06 smithi135 ceph-mon[44364]: pgmap v6: 65 pgs: 65 active+clean; 43 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 196 KiB/s rd, 1.1 MiB/s wr, 117 op/s 2024-03-30T04:27:08.450 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:08 smithi123 ceph-mon[35245]: pgmap v7: 65 pgs: 65 active+clean; 50 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 757 KiB/s rd, 1.9 MiB/s wr, 247 op/s 2024-03-30T04:27:08.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:08 smithi135 ceph-mon[44364]: pgmap v7: 65 pgs: 65 active+clean; 50 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 757 KiB/s rd, 1.9 MiB/s wr, 247 op/s 2024-03-30T04:27:10.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:10 smithi123 ceph-mon[35245]: pgmap v8: 65 pgs: 65 active+clean; 50 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 588 KiB/s rd, 1.5 MiB/s wr, 192 op/s 2024-03-30T04:27:10.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:10 smithi135 ceph-mon[44364]: pgmap v8: 65 pgs: 65 active+clean; 50 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 588 KiB/s rd, 1.5 MiB/s wr, 192 op/s 2024-03-30T04:27:11.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:11 smithi123 ceph-mon[35245]: Standby manager daemon smithi123.xgksqq started 2024-03-30T04:27:11.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:11 smithi123 ceph-mon[35245]: from='mgr.? 172.21.15.123:0/4148571342' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi123.xgksqq/crt"}]: dispatch 2024-03-30T04:27:11.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:11 smithi123 ceph-mon[35245]: from='mgr.? 172.21.15.123:0/4148571342' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T04:27:11.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:11 smithi123 ceph-mon[35245]: from='mgr.? 172.21.15.123:0/4148571342' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi123.xgksqq/key"}]: dispatch 2024-03-30T04:27:11.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:11 smithi123 ceph-mon[35245]: from='mgr.? 172.21.15.123:0/4148571342' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T04:27:11.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:11 smithi135 ceph-mon[44364]: Standby manager daemon smithi123.xgksqq started 2024-03-30T04:27:11.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:11 smithi135 ceph-mon[44364]: from='mgr.? 172.21.15.123:0/4148571342' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi123.xgksqq/crt"}]: dispatch 2024-03-30T04:27:11.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:11 smithi135 ceph-mon[44364]: from='mgr.? 172.21.15.123:0/4148571342' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T04:27:11.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:11 smithi135 ceph-mon[44364]: from='mgr.? 172.21.15.123:0/4148571342' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi123.xgksqq/key"}]: dispatch 2024-03-30T04:27:11.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:11 smithi135 ceph-mon[44364]: from='mgr.? 172.21.15.123:0/4148571342' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T04:27:12.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:12 smithi135 ceph-mon[44364]: pgmap v9: 65 pgs: 65 active+clean; 67 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.0 MiB/s rd, 2.8 MiB/s wr, 295 op/s 2024-03-30T04:27:12.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:12 smithi135 ceph-mon[44364]: mgrmap e24: smithi135.kyblli(active, since 12s), standbys: smithi123.xgksqq 2024-03-30T04:27:12.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:12 smithi135 ceph-mon[44364]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "mgr metadata", "who": "smithi123.xgksqq", "id": "smithi123.xgksqq"}]: dispatch 2024-03-30T04:27:12.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:12 smithi123 ceph-mon[35245]: pgmap v9: 65 pgs: 65 active+clean; 67 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.0 MiB/s rd, 2.8 MiB/s wr, 295 op/s 2024-03-30T04:27:12.470 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:12 smithi123 ceph-mon[35245]: mgrmap e24: smithi135.kyblli(active, since 12s), standbys: smithi123.xgksqq 2024-03-30T04:27:12.470 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:12 smithi123 ceph-mon[35245]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "mgr metadata", "who": "smithi123.xgksqq", "id": "smithi123.xgksqq"}]: dispatch 2024-03-30T04:27:13.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:13 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:13.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:13 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:13.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:13 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:13.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:13 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:14.055 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.056+0000 7f9cafe00700 1 -- 172.21.15.123:0/150837000 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ca80fcee0 msgr2=0x7f9ca80fd310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:27:14.055 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.056+0000 7f9cafe00700 1 --2- 172.21.15.123:0/150837000 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ca80fcee0 0x7f9ca80fd310 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c004940 tx=0x7f9c9c009e30 comp rx=0 tx=0).stop 2024-03-30T04:27:14.055 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.056+0000 7f9cafe00700 1 -- 172.21.15.123:0/150837000 shutdown_connections 2024-03-30T04:27:14.055 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.056+0000 7f9cafe00700 1 --2- 172.21.15.123:0/150837000 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9ca80fe0d0 0x7f9ca80fe540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:14.055 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.056+0000 7f9cafe00700 1 --2- 172.21.15.123:0/150837000 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ca80fcee0 0x7f9ca80fd310 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:14.055 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.056+0000 7f9cafe00700 1 -- 172.21.15.123:0/150837000 >> 172.21.15.123:0/150837000 conn(0x7f9ca80f8560 msgr2=0x7f9ca80fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:27:14.055 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.056+0000 7f9cafe00700 1 -- 172.21.15.123:0/150837000 shutdown_connections 2024-03-30T04:27:14.055 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.056+0000 7f9cafe00700 1 -- 172.21.15.123:0/150837000 wait complete. 2024-03-30T04:27:14.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.058+0000 7f9cafe00700 1 Processor -- start 2024-03-30T04:27:14.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.058+0000 7f9cafe00700 1 -- start start 2024-03-30T04:27:14.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.058+0000 7f9cafe00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9ca80fcee0 0x7f9ca8071e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:27:14.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.058+0000 7f9cafe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ca80fe0d0 0x7f9ca8070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:27:14.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.058+0000 7f9cafe00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca8070950 con 0x7f9ca80fe0d0 2024-03-30T04:27:14.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.058+0000 7f9cafe00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca8070ac0 con 0x7f9ca80fcee0 2024-03-30T04:27:14.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.059+0000 7f9cad400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9ca80fcee0 0x7f9ca8071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:27:14.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.059+0000 7f9caca00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ca80fe0d0 0x7f9ca8070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:27:14.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.059+0000 7f9caca00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ca80fe0d0 0x7f9ca8070310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39998/0 (socket says 172.21.15.123:39998) 2024-03-30T04:27:14.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.059+0000 7f9caca00700 1 -- 172.21.15.123:0/2857867045 learned_addr learned my addr 172.21.15.123:0/2857867045 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:27:14.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.059+0000 7f9caca00700 1 -- 172.21.15.123:0/2857867045 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9ca80fcee0 msgr2=0x7f9ca8071e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:27:14.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.059+0000 7f9caca00700 1 --2- 172.21.15.123:0/2857867045 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9ca80fcee0 0x7f9ca8071e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:14.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.059+0000 7f9caca00700 1 -- 172.21.15.123:0/2857867045 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c9c005040 con 0x7f9ca80fe0d0 2024-03-30T04:27:14.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.060+0000 7f9caca00700 1 --2- 172.21.15.123:0/2857867045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ca80fe0d0 0x7f9ca8070310 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f9c9400b790 tx=0x7f9c94009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:27:14.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.060+0000 7f9ca5600700 1 -- 172.21.15.123:0/2857867045 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c9400f6f0 con 0x7f9ca80fe0d0 2024-03-30T04:27:14.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.060+0000 7f9ca5600700 1 -- 172.21.15.123:0/2857867045 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9c9400fd00 con 0x7f9ca80fe0d0 2024-03-30T04:27:14.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.060+0000 7f9ca5600700 1 -- 172.21.15.123:0/2857867045 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c940103a0 con 0x7f9ca80fe0d0 2024-03-30T04:27:14.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.060+0000 7f9cafe00700 1 -- 172.21.15.123:0/2857867045 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ca8070da0 con 0x7f9ca80fe0d0 2024-03-30T04:27:14.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.060+0000 7f9cafe00700 1 -- 172.21.15.123:0/2857867045 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ca8115a10 con 0x7f9ca80fe0d0 2024-03-30T04:27:14.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.062+0000 7f9cafe00700 1 -- 172.21.15.123:0/2857867045 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ca805f5f0 con 0x7f9ca80fe0d0 2024-03-30T04:27:14.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.065+0000 7f9ca5600700 1 -- 172.21.15.123:0/2857867045 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) v1 ==== 88766+0+0 (secure 0 0 0) 0x7f9c9400f850 con 0x7f9ca80fe0d0 2024-03-30T04:27:14.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.067+0000 7f9ca5600700 1 --2- 172.21.15.123:0/2857867045 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f9c9006aec0 0x7f9c9006d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:27:14.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.067+0000 7f9ca5600700 1 -- 172.21.15.123:0/2857867045 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f9c94089890 con 0x7f9ca80fe0d0 2024-03-30T04:27:14.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.068+0000 7f9cad400700 1 --2- 172.21.15.123:0/2857867045 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f9c9006aec0 0x7f9c9006d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:27:14.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.069+0000 7f9cad400700 1 --2- 172.21.15.123:0/2857867045 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f9c9006aec0 0x7f9c9006d370 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c000c00 tx=0x7f9c9c009e30 comp rx=0 tx=0).ready entity=mgr.14608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:27:14.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.074+0000 7f9ca5600700 1 -- 172.21.15.123:0/2857867045 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f9c94053df0 con 0x7f9ca80fe0d0 2024-03-30T04:27:14.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.437+0000 7f9cafe00700 1 -- 172.21.15.123:0/2857867045 --> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ca8102520 con 0x7f9c9006aec0 2024-03-30T04:27:14.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.440+0000 7f9ca5600700 1 -- 172.21.15.123:0/2857867045 <== mgr.14608 v2:172.21.15.135:6828/3601553667 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f9ca8102520 con 0x7f9c9006aec0 2024-03-30T04:27:14.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.443+0000 7f9cafe00700 1 -- 172.21.15.123:0/2857867045 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f9c9006aec0 msgr2=0x7f9c9006d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:27:14.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.443+0000 7f9cafe00700 1 --2- 172.21.15.123:0/2857867045 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f9c9006aec0 0x7f9c9006d370 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c000c00 tx=0x7f9c9c009e30 comp rx=0 tx=0).stop 2024-03-30T04:27:14.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.443+0000 7f9cafe00700 1 -- 172.21.15.123:0/2857867045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ca80fe0d0 msgr2=0x7f9ca8070310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:27:14.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.443+0000 7f9cafe00700 1 --2- 172.21.15.123:0/2857867045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ca80fe0d0 0x7f9ca8070310 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f9c9400b790 tx=0x7f9c94009250 comp rx=0 tx=0).stop 2024-03-30T04:27:14.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.444+0000 7f9cafe00700 1 -- 172.21.15.123:0/2857867045 shutdown_connections 2024-03-30T04:27:14.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.444+0000 7f9cafe00700 1 --2- 172.21.15.123:0/2857867045 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f9c9006aec0 0x7f9c9006d370 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:14.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.444+0000 7f9cafe00700 1 --2- 172.21.15.123:0/2857867045 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9ca80fcee0 0x7f9ca8071e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:14.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.444+0000 7f9cafe00700 1 --2- 172.21.15.123:0/2857867045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ca80fe0d0 0x7f9ca8070310 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:14.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.444+0000 7f9cafe00700 1 -- 172.21.15.123:0/2857867045 >> 172.21.15.123:0/2857867045 conn(0x7f9ca80f8560 msgr2=0x7f9ca80f9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:27:14.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.444+0000 7f9cafe00700 1 -- 172.21.15.123:0/2857867045 shutdown_connections 2024-03-30T04:27:14.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.444+0000 7f9cafe00700 1 -- 172.21.15.123:0/2857867045 wait complete. 2024-03-30T04:27:14.458 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:27:14.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:14 smithi123 ceph-mon[35245]: pgmap v10: 65 pgs: 65 active+clean; 67 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 971 KiB/s rd, 2.6 MiB/s wr, 270 op/s 2024-03-30T04:27:14.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:14 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:14.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:14 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:14.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:14 smithi135 ceph-mon[44364]: pgmap v10: 65 pgs: 65 active+clean; 67 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 971 KiB/s rd, 2.6 MiB/s wr, 270 op/s 2024-03-30T04:27:14.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:14 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:14.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:14 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:14.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.607+0000 7fb6a9400700 1 -- 172.21.15.123:0/1818108314 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6a41002a0 msgr2=0x7fb6a4100710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:27:14.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.607+0000 7fb6a9400700 1 --2- 172.21.15.123:0/1818108314 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6a41002a0 0x7fb6a4100710 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fb690004940 tx=0x7fb690009e30 comp rx=0 tx=0).stop 2024-03-30T04:27:14.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.607+0000 7fb6a9400700 1 -- 172.21.15.123:0/1818108314 shutdown_connections 2024-03-30T04:27:14.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.607+0000 7fb6a9400700 1 --2- 172.21.15.123:0/1818108314 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6a41002a0 0x7fb6a4100710 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:14.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.607+0000 7fb6a9400700 1 --2- 172.21.15.123:0/1818108314 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb6a40feff0 0x7fb6a40ff400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:14.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.607+0000 7fb6a9400700 1 -- 172.21.15.123:0/1818108314 >> 172.21.15.123:0/1818108314 conn(0x7fb6a40fa950 msgr2=0x7fb6a40fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:27:14.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.607+0000 7fb6a9400700 1 -- 172.21.15.123:0/1818108314 shutdown_connections 2024-03-30T04:27:14.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.607+0000 7fb6a9400700 1 -- 172.21.15.123:0/1818108314 wait complete. 2024-03-30T04:27:14.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.608+0000 7fb6a9400700 1 Processor -- start 2024-03-30T04:27:14.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.608+0000 7fb6a9400700 1 -- start start 2024-03-30T04:27:14.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.609+0000 7fb6a9400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6a40feff0 0x7fb6a410fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:27:14.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.609+0000 7fb6a9400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb6a41002a0 0x7fb6a4110090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:27:14.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.609+0000 7fb6a9400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a4110690 con 0x7fb6a40feff0 2024-03-30T04:27:14.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.609+0000 7fb6a9400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a4110ee0 con 0x7fb6a41002a0 2024-03-30T04:27:14.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.609+0000 7fb6a2800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6a40feff0 0x7fb6a410fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:27:14.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.610+0000 7fb6a2800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6a40feff0 0x7fb6a410fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40004/0 (socket says 172.21.15.123:40004) 2024-03-30T04:27:14.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.610+0000 7fb6a2800700 1 -- 172.21.15.123:0/3403741697 learned_addr learned my addr 172.21.15.123:0/3403741697 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:27:14.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.610+0000 7fb6a1e00700 1 --2- 172.21.15.123:0/3403741697 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb6a41002a0 0x7fb6a4110090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:27:14.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.610+0000 7fb6a2800700 1 -- 172.21.15.123:0/3403741697 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb6a41002a0 msgr2=0x7fb6a4110090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:27:14.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.610+0000 7fb6a2800700 1 --2- 172.21.15.123:0/3403741697 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb6a41002a0 0x7fb6a4110090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:14.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.610+0000 7fb6a2800700 1 -- 172.21.15.123:0/3403741697 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb690005040 con 0x7fb6a40feff0 2024-03-30T04:27:14.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.610+0000 7fb6a2800700 1 --2- 172.21.15.123:0/3403741697 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6a40feff0 0x7fb6a410fb50 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb694002a50 tx=0x7fb6940046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:27:14.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.611+0000 7fb69aa00700 1 -- 172.21.15.123:0/3403741697 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb694007cc0 con 0x7fb6a40feff0 2024-03-30T04:27:14.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.611+0000 7fb69aa00700 1 -- 172.21.15.123:0/3403741697 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb69400f040 con 0x7fb6a40feff0 2024-03-30T04:27:14.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.611+0000 7fb69aa00700 1 -- 172.21.15.123:0/3403741697 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb69400d9c0 con 0x7fb6a40feff0 2024-03-30T04:27:14.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.612+0000 7fb6a9400700 1 -- 172.21.15.123:0/3403741697 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6a41111c0 con 0x7fb6a40feff0 2024-03-30T04:27:14.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.612+0000 7fb6a9400700 1 -- 172.21.15.123:0/3403741697 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6a4111700 con 0x7fb6a40feff0 2024-03-30T04:27:14.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.613+0000 7fb6a9400700 1 -- 172.21.15.123:0/3403741697 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6a405f5f0 con 0x7fb6a40feff0 2024-03-30T04:27:14.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.616+0000 7fb69aa00700 1 -- 172.21.15.123:0/3403741697 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) v1 ==== 88766+0+0 (secure 0 0 0) 0x7fb69400db20 con 0x7fb6a40feff0 2024-03-30T04:27:14.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.616+0000 7fb69aa00700 1 --2- 172.21.15.123:0/3403741697 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7fb68806af90 0x7fb68806d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:27:14.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.616+0000 7fb69aa00700 1 -- 172.21.15.123:0/3403741697 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7fb694088f00 con 0x7fb6a40feff0 2024-03-30T04:27:14.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.617+0000 7fb6a1e00700 1 --2- 172.21.15.123:0/3403741697 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7fb68806af90 0x7fb68806d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:27:14.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.618+0000 7fb6a1e00700 1 --2- 172.21.15.123:0/3403741697 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7fb68806af90 0x7fb68806d440 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb690004f90 tx=0x7fb690009e30 comp rx=0 tx=0).ready entity=mgr.14608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:27:14.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.623+0000 7fb69aa00700 1 -- 172.21.15.123:0/3403741697 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fb69404e560 con 0x7fb6a40feff0 2024-03-30T04:27:14.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.962+0000 7fb6a9400700 1 -- 172.21.15.123:0/3403741697 --> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb6a4111af0 con 0x7fb68806af90 2024-03-30T04:27:14.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.966+0000 7fb69aa00700 1 -- 172.21.15.123:0/3403741697 <== mgr.14608 v2:172.21.15.135:6828/3601553667 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7fb6a4111af0 con 0x7fb68806af90 2024-03-30T04:27:14.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.970+0000 7fb6a9400700 1 -- 172.21.15.123:0/3403741697 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7fb68806af90 msgr2=0x7fb68806d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:27:14.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.970+0000 7fb6a9400700 1 --2- 172.21.15.123:0/3403741697 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7fb68806af90 0x7fb68806d440 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb690004f90 tx=0x7fb690009e30 comp rx=0 tx=0).stop 2024-03-30T04:27:14.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.970+0000 7fb6a9400700 1 -- 172.21.15.123:0/3403741697 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6a40feff0 msgr2=0x7fb6a410fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:27:14.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.970+0000 7fb6a9400700 1 --2- 172.21.15.123:0/3403741697 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6a40feff0 0x7fb6a410fb50 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb694002a50 tx=0x7fb6940046b0 comp rx=0 tx=0).stop 2024-03-30T04:27:14.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.971+0000 7fb6a9400700 1 -- 172.21.15.123:0/3403741697 shutdown_connections 2024-03-30T04:27:14.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.971+0000 7fb6a9400700 1 --2- 172.21.15.123:0/3403741697 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7fb68806af90 0x7fb68806d440 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:14.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.971+0000 7fb6a9400700 1 --2- 172.21.15.123:0/3403741697 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6a40feff0 0x7fb6a410fb50 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:14.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.971+0000 7fb6a9400700 1 --2- 172.21.15.123:0/3403741697 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb6a41002a0 0x7fb6a4110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:14.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.971+0000 7fb6a9400700 1 -- 172.21.15.123:0/3403741697 >> 172.21.15.123:0/3403741697 conn(0x7fb6a40fa950 msgr2=0x7fb6a40fd100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:27:14.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.971+0000 7fb6a9400700 1 -- 172.21.15.123:0/3403741697 shutdown_connections 2024-03-30T04:27:14.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:14.971+0000 7fb6a9400700 1 -- 172.21.15.123:0/3403741697 wait complete. 2024-03-30T04:27:15.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.165+0000 7f01d4a00700 1 -- 172.21.15.123:0/3843495384 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01d00662e0 msgr2=0x7f01d0101c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:27:15.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.165+0000 7f01d4a00700 1 --2- 172.21.15.123:0/3843495384 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01d00662e0 0x7f01d0101c00 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f01c0004940 tx=0x7f01c0009e30 comp rx=0 tx=0).stop 2024-03-30T04:27:15.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.165+0000 7f01d4a00700 1 -- 172.21.15.123:0/3843495384 shutdown_connections 2024-03-30T04:27:15.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.165+0000 7f01d4a00700 1 --2- 172.21.15.123:0/3843495384 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f01d0102140 0x7f01d0104520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:15.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.165+0000 7f01d4a00700 1 --2- 172.21.15.123:0/3843495384 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01d00662e0 0x7f01d0101c00 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:15.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.165+0000 7f01d4a00700 1 -- 172.21.15.123:0/3843495384 >> 172.21.15.123:0/3843495384 conn(0x7f01d00f7270 msgr2=0x7f01d00f96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:27:15.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.165+0000 7f01d4a00700 1 -- 172.21.15.123:0/3843495384 shutdown_connections 2024-03-30T04:27:15.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.166+0000 7f01d4a00700 1 -- 172.21.15.123:0/3843495384 wait complete. 2024-03-30T04:27:15.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.167+0000 7f01d4a00700 1 Processor -- start 2024-03-30T04:27:15.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.167+0000 7f01d4a00700 1 -- start start 2024-03-30T04:27:15.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.167+0000 7f01d4a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f01d00662e0 0x7f01d010f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:27:15.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.167+0000 7f01d4a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01d0102140 0x7f01d010fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:27:15.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.167+0000 7f01d4a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d0110460 con 0x7f01d0102140 2024-03-30T04:27:15.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.167+0000 7f01d4a00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d0110cb0 con 0x7f01d00662e0 2024-03-30T04:27:15.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.168+0000 7f01cde00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01d0102140 0x7f01d010fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:27:15.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.168+0000 7f01cde00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01d0102140 0x7f01d010fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40014/0 (socket says 172.21.15.123:40014) 2024-03-30T04:27:15.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.168+0000 7f01cde00700 1 -- 172.21.15.123:0/1392349216 learned_addr learned my addr 172.21.15.123:0/1392349216 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:27:15.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.168+0000 7f01ce800700 1 --2- 172.21.15.123:0/1392349216 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f01d00662e0 0x7f01d010f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:27:15.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.168+0000 7f01ce800700 1 -- 172.21.15.123:0/1392349216 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01d0102140 msgr2=0x7f01d010fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:27:15.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.168+0000 7f01ce800700 1 --2- 172.21.15.123:0/1392349216 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01d0102140 0x7f01d010fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:15.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.168+0000 7f01ce800700 1 -- 172.21.15.123:0/1392349216 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01c0005040 con 0x7f01d00662e0 2024-03-30T04:27:15.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.168+0000 7f01cde00700 1 --2- 172.21.15.123:0/1392349216 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01d0102140 0x7f01d010fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:27:15.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.169+0000 7f01ce800700 1 --2- 172.21.15.123:0/1392349216 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f01d00662e0 0x7f01d010f920 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f01c0000c00 tx=0x7f01c0009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:27:15.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.169+0000 7f01c6a00700 1 -- 172.21.15.123:0/1392349216 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f01c000e070 con 0x7f01d00662e0 2024-03-30T04:27:15.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.169+0000 7f01c6a00700 1 -- 172.21.15.123:0/1392349216 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f01c001d460 con 0x7f01d00662e0 2024-03-30T04:27:15.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.169+0000 7f01c6a00700 1 -- 172.21.15.123:0/1392349216 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f01c000b670 con 0x7f01d00662e0 2024-03-30T04:27:15.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.169+0000 7f01d4a00700 1 -- 172.21.15.123:0/1392349216 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01d0110f30 con 0x7f01d00662e0 2024-03-30T04:27:15.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.169+0000 7f01d4a00700 1 -- 172.21.15.123:0/1392349216 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01d0111440 con 0x7f01d00662e0 2024-03-30T04:27:15.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.170+0000 7f01d4a00700 1 -- 172.21.15.123:0/1392349216 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01d005f5f0 con 0x7f01d00662e0 2024-03-30T04:27:15.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.172+0000 7f01c6a00700 1 -- 172.21.15.123:0/1392349216 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 24) v1 ==== 88766+0+0 (secure 0 0 0) 0x7f01c0013a50 con 0x7f01d00662e0 2024-03-30T04:27:15.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.172+0000 7f01c6a00700 1 --2- 172.21.15.123:0/1392349216 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f01b406aec0 0x7f01b406d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:27:15.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.173+0000 7f01c6a00700 1 -- 172.21.15.123:0/1392349216 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f01c008a740 con 0x7f01d00662e0 2024-03-30T04:27:15.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.173+0000 7f01cde00700 1 --2- 172.21.15.123:0/1392349216 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f01b406aec0 0x7f01b406d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:27:15.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.174+0000 7f01cde00700 1 --2- 172.21.15.123:0/1392349216 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f01b406aec0 0x7f01b406d370 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f01bc0041d0 tx=0x7f01bc00b040 comp rx=0 tx=0).ready entity=mgr.14608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:27:15.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.178+0000 7f01c6a00700 1 -- 172.21.15.123:0/1392349216 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f01c0058800 con 0x7f01d00662e0 2024-03-30T04:27:15.263 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:15 smithi135 ceph-mon[44364]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:27:15.263 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:15 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:15.263 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:15 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:15.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:15 smithi123 ceph-mon[35245]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:27:15.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:15 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:15.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:15 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:15.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.514+0000 7f01d4a00700 1 -- 172.21.15.123:0/1392349216 --> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f01d0111880 con 0x7f01b406aec0 2024-03-30T04:27:15.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.528+0000 7f01c6a00700 1 -- 172.21.15.123:0/1392349216 <== mgr.14608 v2:172.21.15.135:6828/3601553667 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3388 (secure 0 0 0) 0x7f01d0111880 con 0x7f01b406aec0 2024-03-30T04:27:15.527 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:27:15.527 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (7m) 1s ago 8m 13.1M - 0.25.0 c8568f914cd2 0d2c80043347 2024-03-30T04:27:15.527 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (9m) 1s ago 9m 6647k - 17.2.7-797-g5c024335 f77085f6aab7 bceb8b8c4837 2024-03-30T04:27:15.527 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (8m) 3s ago 8m 6651k - 17.2.7-797-g5c024335 f77085f6aab7 cd0fe67b2d34 2024-03-30T04:27:15.527 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (7m) 1s ago 8m 73.6M - 9.4.7 954c08fa6188 b7d7eaffc084 2024-03-30T04:27:15.527 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (2m) 1s ago 2m 84.0M - 17.2.7-797-g5c024335 f77085f6aab7 2df066fba7a6 2024-03-30T04:27:15.527 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (2m) 1s ago 2m 32.5M - 17.2.7-797-g5c024335 f77085f6aab7 1293b8520db1 2024-03-30T04:27:15.527 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (2m) 3s ago 2m 23.5M - 17.2.7-797-g5c024335 f77085f6aab7 0dc41326818c 2024-03-30T04:27:15.527 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (2m) 3s ago 2m 25.7M - 17.2.7-797-g5c024335 f77085f6aab7 aef8a0cfc553 2024-03-30T04:27:15.527 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:9283 running (10m) 1s ago 10m 160M - 17.2.7-797-g5c024335 f77085f6aab7 7b98aee9a3b6 2024-03-30T04:27:15.528 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283 running (29s) 3s ago 7m 479M - 18.2.2-1018-g470100a4 e7c444a08cf7 20e5e0a05742 2024-03-30T04:27:15.528 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (10m) 1s ago 10m 45.2M 2048M 17.2.7-797-g5c024335 f77085f6aab7 51d18587a7df 2024-03-30T04:27:15.528 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (7m) 3s ago 7m 37.6M 2048M 17.2.7-797-g5c024335 f77085f6aab7 f81052b09dd8 2024-03-30T04:27:15.528 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (9m) 1s ago 9m 8002k - 1.5.0 0da6a335fe13 9848cad1e995 2024-03-30T04:27:15.528 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (7m) 3s ago 7m 7541k - 1.5.0 0da6a335fe13 3c2c8e799cc2 2024-03-30T04:27:15.528 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (6m) 1s ago 6m 67.2M 4348M 17.2.7-797-g5c024335 f77085f6aab7 57a7ba3fd75e 2024-03-30T04:27:15.528 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (6m) 1s ago 6m 72.0M 4348M 17.2.7-797-g5c024335 f77085f6aab7 e1a6c1818a0c 2024-03-30T04:27:15.528 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (5m) 1s ago 5m 60.7M 4348M 17.2.7-797-g5c024335 f77085f6aab7 4c6a727567bd 2024-03-30T04:27:15.528 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (5m) 3s ago 5m 65.2M 5372M 17.2.7-797-g5c024335 f77085f6aab7 6fbdd1d46a9d 2024-03-30T04:27:15.528 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (4m) 3s ago 4m 61.2M 5372M 17.2.7-797-g5c024335 f77085f6aab7 b74988e947e5 2024-03-30T04:27:15.528 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (4m) 3s ago 4m 62.7M 5372M 17.2.7-797-g5c024335 f77085f6aab7 1fd3185f5484 2024-03-30T04:27:15.528 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (7m) 1s ago 8m 46.6M - 2.43.0 a07b618ecd1d ba3c351b14cf 2024-03-30T04:27:15.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.530+0000 7f01d4a00700 1 -- 172.21.15.123:0/1392349216 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f01b406aec0 msgr2=0x7f01b406d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:27:15.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.530+0000 7f01d4a00700 1 --2- 172.21.15.123:0/1392349216 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f01b406aec0 0x7f01b406d370 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f01bc0041d0 tx=0x7f01bc00b040 comp rx=0 tx=0).stop 2024-03-30T04:27:15.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.530+0000 7f01d4a00700 1 -- 172.21.15.123:0/1392349216 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f01d00662e0 msgr2=0x7f01d010f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:27:15.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.530+0000 7f01d4a00700 1 --2- 172.21.15.123:0/1392349216 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f01d00662e0 0x7f01d010f920 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f01c0000c00 tx=0x7f01c0009e30 comp rx=0 tx=0).stop 2024-03-30T04:27:15.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.530+0000 7f01d4a00700 1 -- 172.21.15.123:0/1392349216 shutdown_connections 2024-03-30T04:27:15.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.530+0000 7f01d4a00700 1 --2- 172.21.15.123:0/1392349216 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f01b406aec0 0x7f01b406d370 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:15.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.531+0000 7f01d4a00700 1 --2- 172.21.15.123:0/1392349216 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f01d00662e0 0x7f01d010f920 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:15.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.531+0000 7f01d4a00700 1 --2- 172.21.15.123:0/1392349216 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01d0102140 0x7f01d010fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:15.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.531+0000 7f01d4a00700 1 -- 172.21.15.123:0/1392349216 >> 172.21.15.123:0/1392349216 conn(0x7f01d00f7270 msgr2=0x7f01d00f7690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:27:15.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.531+0000 7f01d4a00700 1 -- 172.21.15.123:0/1392349216 shutdown_connections 2024-03-30T04:27:15.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.531+0000 7f01d4a00700 1 -- 172.21.15.123:0/1392349216 wait complete. 2024-03-30T04:27:15.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.719+0000 7f97d3000700 1 -- 172.21.15.123:0/1082444765 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97cc0fcee0 msgr2=0x7f97cc0fd310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:27:15.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.719+0000 7f97d3000700 1 --2- 172.21.15.123:0/1082444765 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97cc0fcee0 0x7f97cc0fd310 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f97c0004940 tx=0x7f97c0009e30 comp rx=0 tx=0).stop 2024-03-30T04:27:15.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.720+0000 7f97d3000700 1 -- 172.21.15.123:0/1082444765 shutdown_connections 2024-03-30T04:27:15.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.720+0000 7f97d3000700 1 --2- 172.21.15.123:0/1082444765 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f97cc0fe0d0 0x7f97cc0fe540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:15.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.720+0000 7f97d3000700 1 --2- 172.21.15.123:0/1082444765 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97cc0fcee0 0x7f97cc0fd310 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:15.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.720+0000 7f97d3000700 1 -- 172.21.15.123:0/1082444765 >> 172.21.15.123:0/1082444765 conn(0x7f97cc0f8560 msgr2=0x7f97cc0fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:27:15.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.720+0000 7f97d3000700 1 -- 172.21.15.123:0/1082444765 shutdown_connections 2024-03-30T04:27:15.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.720+0000 7f97d3000700 1 -- 172.21.15.123:0/1082444765 wait complete. 2024-03-30T04:27:15.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.722+0000 7f97d3000700 1 Processor -- start 2024-03-30T04:27:15.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.722+0000 7f97d3000700 1 -- start start 2024-03-30T04:27:15.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.723+0000 7f97d3000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f97cc0fcee0 0x7f97cc10d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:27:15.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.723+0000 7f97d3000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97cc0fe0d0 0x7f97cc10df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:27:15.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.723+0000 7f97d3000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97cc10e510 con 0x7f97cc0fe0d0 2024-03-30T04:27:15.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.723+0000 7f97d3000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97cc10ecf0 con 0x7f97cc0fcee0 2024-03-30T04:27:15.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.724+0000 7f97cb400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97cc0fe0d0 0x7f97cc10df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:27:15.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.724+0000 7f97cb400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97cc0fe0d0 0x7f97cc10df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40038/0 (socket says 172.21.15.123:40038) 2024-03-30T04:27:15.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.724+0000 7f97cb400700 1 -- 172.21.15.123:0/2420031047 learned_addr learned my addr 172.21.15.123:0/2420031047 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:27:15.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.724+0000 7f97cbe00700 1 --2- 172.21.15.123:0/2420031047 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f97cc0fcee0 0x7f97cc10d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:27:15.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.724+0000 7f97cb400700 1 -- 172.21.15.123:0/2420031047 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f97cc0fcee0 msgr2=0x7f97cc10d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:27:15.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.724+0000 7f97cb400700 1 --2- 172.21.15.123:0/2420031047 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f97cc0fcee0 0x7f97cc10d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:15.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.724+0000 7f97cb400700 1 -- 172.21.15.123:0/2420031047 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97c0005040 con 0x7f97cc0fe0d0 2024-03-30T04:27:15.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.725+0000 7f97cb400700 1 --2- 172.21.15.123:0/2420031047 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97cc0fe0d0 0x7f97cc10df10 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f97b8002a50 tx=0x7f97b80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:27:15.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.725+0000 7f97bfe00700 1 -- 172.21.15.123:0/2420031047 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97b8007cc0 con 0x7f97cc0fe0d0 2024-03-30T04:27:15.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.725+0000 7f97bfe00700 1 -- 172.21.15.123:0/2420031047 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f97b800f040 con 0x7f97cc0fe0d0 2024-03-30T04:27:15.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.725+0000 7f97bfe00700 1 -- 172.21.15.123:0/2420031047 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97b8009540 con 0x7f97cc0fe0d0 2024-03-30T04:27:15.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.725+0000 7f97d3000700 1 -- 172.21.15.123:0/2420031047 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97cc10efd0 con 0x7f97cc0fe0d0 2024-03-30T04:27:15.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.726+0000 7f97d3000700 1 -- 172.21.15.123:0/2420031047 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97cc10f510 con 0x7f97cc0fe0d0 2024-03-30T04:27:15.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.729+0000 7f97bfe00700 1 -- 172.21.15.123:0/2420031047 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) v1 ==== 88766+0+0 (secure 0 0 0) 0x7f97b80096a0 con 0x7f97cc0fe0d0 2024-03-30T04:27:15.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.729+0000 7f97d3000700 1 -- 172.21.15.123:0/2420031047 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97cc05f5f0 con 0x7f97cc0fe0d0 2024-03-30T04:27:15.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.731+0000 7f97bfe00700 1 --2- 172.21.15.123:0/2420031047 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f97b406aec0 0x7f97b406d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:27:15.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.731+0000 7f97bfe00700 1 -- 172.21.15.123:0/2420031047 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f97b8089610 con 0x7f97cc0fe0d0 2024-03-30T04:27:15.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.732+0000 7f97cbe00700 1 --2- 172.21.15.123:0/2420031047 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f97b406aec0 0x7f97b406d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:27:15.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.732+0000 7f97cbe00700 1 --2- 172.21.15.123:0/2420031047 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f97b406aec0 0x7f97b406d370 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f97c000b3b0 tx=0x7f97c0009e30 comp rx=0 tx=0).ready entity=mgr.14608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:27:15.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:15.736+0000 7f97bfe00700 1 -- 172.21.15.123:0/2420031047 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f97b80574f0 con 0x7f97cc0fe0d0 2024-03-30T04:27:16.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.169+0000 7f97d3000700 1 -- 172.21.15.123:0/2420031047 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f97cc10f780 con 0x7f97cc0fe0d0 2024-03-30T04:27:16.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.170+0000 7f97bfe00700 1 -- 172.21.15.123:0/2420031047 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+850 (secure 0 0 0) 0x7f97b8057080 con 0x7f97cc0fe0d0 2024-03-30T04:27:16.168 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:27:16.168 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:27:16.168 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 2 2024-03-30T04:27:16.169 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:27:16.169 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:27:16.169 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:27:16.169 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 1 2024-03-30T04:27:16.169 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:27:16.169 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:27:16.169 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 6 2024-03-30T04:27:16.169 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:27:16.169 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:27:16.169 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 4 2024-03-30T04:27:16.169 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:27:16.169 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:27:16.169 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 13, 2024-03-30T04:27:16.169 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 1 2024-03-30T04:27:16.170 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:27:16.170 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:27:16.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.173+0000 7f97d3000700 1 -- 172.21.15.123:0/2420031047 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f97b406aec0 msgr2=0x7f97b406d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:27:16.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.173+0000 7f97d3000700 1 --2- 172.21.15.123:0/2420031047 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f97b406aec0 0x7f97b406d370 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f97c000b3b0 tx=0x7f97c0009e30 comp rx=0 tx=0).stop 2024-03-30T04:27:16.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.173+0000 7f97d3000700 1 -- 172.21.15.123:0/2420031047 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97cc0fe0d0 msgr2=0x7f97cc10df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:27:16.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.173+0000 7f97d3000700 1 --2- 172.21.15.123:0/2420031047 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97cc0fe0d0 0x7f97cc10df10 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f97b8002a50 tx=0x7f97b80046b0 comp rx=0 tx=0).stop 2024-03-30T04:27:16.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.173+0000 7f97d3000700 1 -- 172.21.15.123:0/2420031047 shutdown_connections 2024-03-30T04:27:16.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.173+0000 7f97d3000700 1 --2- 172.21.15.123:0/2420031047 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f97b406aec0 0x7f97b406d370 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:16.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.173+0000 7f97d3000700 1 --2- 172.21.15.123:0/2420031047 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f97cc0fcee0 0x7f97cc10d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:16.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.173+0000 7f97d3000700 1 --2- 172.21.15.123:0/2420031047 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97cc0fe0d0 0x7f97cc10df10 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:16.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.173+0000 7f97d3000700 1 -- 172.21.15.123:0/2420031047 >> 172.21.15.123:0/2420031047 conn(0x7f97cc0f8560 msgr2=0x7f97cc0f9da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:27:16.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.173+0000 7f97d3000700 1 -- 172.21.15.123:0/2420031047 shutdown_connections 2024-03-30T04:27:16.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.173+0000 7f97d3000700 1 -- 172.21.15.123:0/2420031047 wait complete. 2024-03-30T04:27:16.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.350+0000 7f3a0a600700 1 -- 172.21.15.123:0/2898140047 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3a04066240 msgr2=0x7f3a040ffa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:27:16.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.350+0000 7f3a0a600700 1 --2- 172.21.15.123:0/2898140047 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3a04066240 0x7f3a040ffa60 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f39f4009ce0 tx=0x7f39f4004d10 comp rx=0 tx=0).stop 2024-03-30T04:27:16.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.351+0000 7f3a0a600700 1 -- 172.21.15.123:0/2898140047 shutdown_connections 2024-03-30T04:27:16.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.351+0000 7f3a0a600700 1 --2- 172.21.15.123:0/2898140047 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3a040fffa0 0x7f3a04102380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:16.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.351+0000 7f3a0a600700 1 --2- 172.21.15.123:0/2898140047 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3a04066240 0x7f3a040ffa60 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:16.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.351+0000 7f3a0a600700 1 -- 172.21.15.123:0/2898140047 >> 172.21.15.123:0/2898140047 conn(0x7f3a040f7430 msgr2=0x7f3a040f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:27:16.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.351+0000 7f3a0a600700 1 -- 172.21.15.123:0/2898140047 shutdown_connections 2024-03-30T04:27:16.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.351+0000 7f3a0a600700 1 -- 172.21.15.123:0/2898140047 wait complete. 2024-03-30T04:27:16.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.352+0000 7f3a0a600700 1 Processor -- start 2024-03-30T04:27:16.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.352+0000 7f3a0a600700 1 -- start start 2024-03-30T04:27:16.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.353+0000 7f3a0a600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3a04066240 0x7f3a0410fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:27:16.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.353+0000 7f3a0a600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3a040fffa0 0x7f3a041102a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:27:16.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.353+0000 7f3a0a600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a041108f0 con 0x7f3a04066240 2024-03-30T04:27:16.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.353+0000 7f3a0a600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a0410b730 con 0x7f3a040fffa0 2024-03-30T04:27:16.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.353+0000 7f3a02a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3a040fffa0 0x7f3a041102a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:27:16.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.353+0000 7f3a02a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3a040fffa0 0x7f3a041102a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.123:37144/0 (socket says 172.21.15.123:37144) 2024-03-30T04:27:16.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.353+0000 7f3a02a00700 1 -- 172.21.15.123:0/1386595642 learned_addr learned my addr 172.21.15.123:0/1386595642 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:27:16.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.354+0000 7f3a03400700 1 --2- 172.21.15.123:0/1386595642 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3a04066240 0x7f3a0410fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:27:16.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.354+0000 7f3a02a00700 1 -- 172.21.15.123:0/1386595642 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3a04066240 msgr2=0x7f3a0410fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:27:16.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.354+0000 7f3a02a00700 1 --2- 172.21.15.123:0/1386595642 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3a04066240 0x7f3a0410fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:16.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.354+0000 7f3a02a00700 1 -- 172.21.15.123:0/1386595642 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39f4009940 con 0x7f3a040fffa0 2024-03-30T04:27:16.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.354+0000 7f3a02a00700 1 --2- 172.21.15.123:0/1386595642 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3a040fffa0 0x7f3a041102a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f39f000eb40 tx=0x7f39f000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:27:16.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.355+0000 7f39fb400700 1 -- 172.21.15.123:0/1386595642 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39f0007cc0 con 0x7f3a040fffa0 2024-03-30T04:27:16.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.355+0000 7f39fb400700 1 -- 172.21.15.123:0/1386595642 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f39f000f040 con 0x7f3a040fffa0 2024-03-30T04:27:16.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.355+0000 7f3a0a600700 1 -- 172.21.15.123:0/1386595642 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a0410b980 con 0x7f3a040fffa0 2024-03-30T04:27:16.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.355+0000 7f39fb400700 1 -- 172.21.15.123:0/1386595642 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39f0009570 con 0x7f3a040fffa0 2024-03-30T04:27:16.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.355+0000 7f3a0a600700 1 -- 172.21.15.123:0/1386595642 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a0410bf40 con 0x7f3a040fffa0 2024-03-30T04:27:16.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.356+0000 7f3a0a600700 1 -- 172.21.15.123:0/1386595642 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a0405f5f0 con 0x7f3a040fffa0 2024-03-30T04:27:16.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.357+0000 7f39fb400700 1 -- 172.21.15.123:0/1386595642 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 24) v1 ==== 88766+0+0 (secure 0 0 0) 0x7f39f00097e0 con 0x7f3a040fffa0 2024-03-30T04:27:16.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.359+0000 7f39fb400700 1 --2- 172.21.15.123:0/1386595642 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f39e806aec0 0x7f39e806d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:27:16.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.359+0000 7f39fb400700 1 -- 172.21.15.123:0/1386595642 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f39f0089550 con 0x7f3a040fffa0 2024-03-30T04:27:16.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.359+0000 7f3a03400700 1 --2- 172.21.15.123:0/1386595642 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f39e806aec0 0x7f39e806d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:27:16.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.360+0000 7f3a03400700 1 --2- 172.21.15.123:0/1386595642 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f39e806aec0 0x7f39e806d370 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f39f4004f80 tx=0x7f39f4004d10 comp rx=0 tx=0).ready entity=mgr.14608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:27:16.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.365+0000 7f39fb400700 1 -- 172.21.15.123:0/1386595642 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f39f004f560 con 0x7f3a040fffa0 2024-03-30T04:27:16.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:16 smithi123 ceph-mon[35245]: from='client.14636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:27:16.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:16 smithi123 ceph-mon[35245]: from='client.14640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:27:16.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:16 smithi123 ceph-mon[35245]: pgmap v11: 65 pgs: 65 active+clean; 70 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.0 MiB/s rd, 2.8 MiB/s wr, 315 op/s 2024-03-30T04:27:16.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:16 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:16.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:16 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:16.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:16 smithi123 ceph-mon[35245]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:27:16.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:16 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:27:16.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:16 smithi123 ceph-mon[35245]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:27:16.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:16 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:27:16.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:16 smithi123 ceph-mon[35245]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:27:16.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:16 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:27:16.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:16 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:16.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:16 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/2420031047' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:27:16.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:16 smithi135 ceph-mon[44364]: from='client.14636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:27:16.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:16 smithi135 ceph-mon[44364]: from='client.14640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:27:16.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:16 smithi135 ceph-mon[44364]: pgmap v11: 65 pgs: 65 active+clean; 70 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.0 MiB/s rd, 2.8 MiB/s wr, 315 op/s 2024-03-30T04:27:16.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:16 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:16.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:16 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:16.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:16 smithi135 ceph-mon[44364]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:27:16.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:16 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:27:16.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:16 smithi135 ceph-mon[44364]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:27:16.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:16 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:27:16.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:16 smithi135 ceph-mon[44364]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:27:16.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:16 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:27:16.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:16 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:16.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:16 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/2420031047' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:27:16.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.703+0000 7f3a0a600700 1 -- 172.21.15.123:0/1386595642 --> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3a0410c2e0 con 0x7f39e806aec0 2024-03-30T04:27:16.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.705+0000 7f39fb400700 1 -- 172.21.15.123:0/1386595642 <== mgr.14608 v2:172.21.15.135:6828/3601553667 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f3a0410c2e0 con 0x7f39e806aec0 2024-03-30T04:27:16.704 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:27:16.704 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:27:16.704 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:27:16.704 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-30T04:27:16.704 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [], 2024-03-30T04:27:16.704 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "1/2 daemons upgraded", 2024-03-30T04:27:16.704 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "", 2024-03-30T04:27:16.704 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:27:16.704 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:27:16.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.707+0000 7f3a0a600700 1 -- 172.21.15.123:0/1386595642 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f39e806aec0 msgr2=0x7f39e806d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:27:16.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.707+0000 7f3a0a600700 1 --2- 172.21.15.123:0/1386595642 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f39e806aec0 0x7f39e806d370 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f39f4004f80 tx=0x7f39f4004d10 comp rx=0 tx=0).stop 2024-03-30T04:27:16.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.707+0000 7f3a0a600700 1 -- 172.21.15.123:0/1386595642 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3a040fffa0 msgr2=0x7f3a041102a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:27:16.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.707+0000 7f3a0a600700 1 --2- 172.21.15.123:0/1386595642 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3a040fffa0 0x7f3a041102a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f39f000eb40 tx=0x7f39f000b040 comp rx=0 tx=0).stop 2024-03-30T04:27:16.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.707+0000 7f3a0a600700 1 -- 172.21.15.123:0/1386595642 shutdown_connections 2024-03-30T04:27:16.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.707+0000 7f3a0a600700 1 --2- 172.21.15.123:0/1386595642 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f39e806aec0 0x7f39e806d370 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:16.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.707+0000 7f3a0a600700 1 --2- 172.21.15.123:0/1386595642 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3a04066240 0x7f3a0410fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:16.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.707+0000 7f3a0a600700 1 --2- 172.21.15.123:0/1386595642 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3a040fffa0 0x7f3a041102a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:16.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.707+0000 7f3a0a600700 1 -- 172.21.15.123:0/1386595642 >> 172.21.15.123:0/1386595642 conn(0x7f3a040f7430 msgr2=0x7f3a040f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:27:16.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.708+0000 7f3a0a600700 1 -- 172.21.15.123:0/1386595642 shutdown_connections 2024-03-30T04:27:16.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:16.708+0000 7f3a0a600700 1 -- 172.21.15.123:0/1386595642 wait complete. 2024-03-30T04:27:17.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:17 smithi123 ceph-mon[35245]: from='client.24459 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:27:17.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:17 smithi123 ceph-mon[35245]: Adjusting osd_memory_target on smithi123 to 1618M 2024-03-30T04:27:17.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:17 smithi135 ceph-mon[44364]: from='client.24459 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:27:17.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:17 smithi135 ceph-mon[44364]: Adjusting osd_memory_target on smithi123 to 1618M 2024-03-30T04:27:18.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:18 smithi123 ceph-mon[35245]: from='client.24467 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:27:18.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:18 smithi123 ceph-mon[35245]: pgmap v12: 65 pgs: 65 active+clean; 78 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 3.1 MiB/s wr, 359 op/s 2024-03-30T04:27:18.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:18 smithi135 ceph-mon[44364]: from='client.24467 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:27:18.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:18 smithi135 ceph-mon[44364]: pgmap v12: 65 pgs: 65 active+clean; 78 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 3.1 MiB/s wr, 359 op/s 2024-03-30T04:27:20.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:20 smithi123 ceph-mon[35245]: pgmap v13: 65 pgs: 65 active+clean; 78 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 264 op/s 2024-03-30T04:27:20.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:20 smithi135 ceph-mon[44364]: pgmap v13: 65 pgs: 65 active+clean; 78 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 264 op/s 2024-03-30T04:27:22.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:22 smithi135 ceph-mon[44364]: pgmap v14: 65 pgs: 65 active+clean; 95 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 4.0 MiB/s wr, 451 op/s 2024-03-30T04:27:22.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:22 smithi123 ceph-mon[35245]: pgmap v14: 65 pgs: 65 active+clean; 95 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 4.0 MiB/s wr, 451 op/s 2024-03-30T04:27:24.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:23 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:24.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:23 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:24.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:23 smithi135 ceph-mon[44364]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:27:24.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:23 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:27:24.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:23 smithi135 ceph-mon[44364]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:27:24.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:23 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:27:24.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:23 smithi135 ceph-mon[44364]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:27:24.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:23 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:27:24.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:23 smithi135 ceph-mon[44364]: Adjusting osd_memory_target on smithi135 to 2642M 2024-03-30T04:27:24.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:23 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:24.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:23 smithi135 ceph-mon[44364]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:27:24.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:23 smithi135 ceph-mon[44364]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:27:24.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:23 smithi135 ceph-mon[44364]: Updating smithi123:/etc/ceph/ceph.conf 2024-03-30T04:27:24.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:23 smithi135 ceph-mon[44364]: Updating smithi135:/etc/ceph/ceph.conf 2024-03-30T04:27:24.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:23 smithi135 ceph-mon[44364]: pgmap v15: 65 pgs: 65 active+clean; 95 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 325 op/s 2024-03-30T04:27:24.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:23 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:24.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:23 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:24.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:23 smithi123 ceph-mon[35245]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:27:24.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:23 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:27:24.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:23 smithi123 ceph-mon[35245]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:27:24.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:23 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:27:24.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:23 smithi123 ceph-mon[35245]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:27:24.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:23 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:27:24.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:23 smithi123 ceph-mon[35245]: Adjusting osd_memory_target on smithi135 to 2642M 2024-03-30T04:27:24.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:23 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:24.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:23 smithi123 ceph-mon[35245]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:27:24.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:23 smithi123 ceph-mon[35245]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:27:24.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:23 smithi123 ceph-mon[35245]: Updating smithi123:/etc/ceph/ceph.conf 2024-03-30T04:27:24.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:23 smithi123 ceph-mon[35245]: Updating smithi135:/etc/ceph/ceph.conf 2024-03-30T04:27:24.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:23 smithi123 ceph-mon[35245]: pgmap v15: 65 pgs: 65 active+clean; 95 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 325 op/s 2024-03-30T04:27:25.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:25 smithi135 ceph-mon[44364]: Updating smithi123:/var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T04:27:25.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:25 smithi135 ceph-mon[44364]: Updating smithi135:/var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T04:27:25.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:25 smithi123 ceph-mon[35245]: Updating smithi123:/var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T04:27:25.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:25 smithi123 ceph-mon[35245]: Updating smithi135:/var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T04:27:26.342 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:26 smithi123 ceph-mon[35245]: Updating smithi123:/etc/ceph/ceph.client.admin.keyring 2024-03-30T04:27:26.342 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:26 smithi123 ceph-mon[35245]: Updating smithi135:/etc/ceph/ceph.client.admin.keyring 2024-03-30T04:27:26.342 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:26 smithi123 ceph-mon[35245]: Updating smithi123:/var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T04:27:26.342 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:26 smithi123 ceph-mon[35245]: pgmap v16: 65 pgs: 65 active+clean; 102 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 383 op/s 2024-03-30T04:27:26.342 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:26 smithi123 ceph-mon[35245]: Updating smithi135:/var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T04:27:26.342 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:26 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:26.342 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:26 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:26.342 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:26 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:26.343 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:26 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:26.343 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:26 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:26.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:26 smithi135 ceph-mon[44364]: Updating smithi123:/etc/ceph/ceph.client.admin.keyring 2024-03-30T04:27:26.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:26 smithi135 ceph-mon[44364]: Updating smithi135:/etc/ceph/ceph.client.admin.keyring 2024-03-30T04:27:26.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:26 smithi135 ceph-mon[44364]: Updating smithi123:/var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T04:27:26.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:26 smithi135 ceph-mon[44364]: pgmap v16: 65 pgs: 65 active+clean; 102 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 383 op/s 2024-03-30T04:27:26.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:26 smithi135 ceph-mon[44364]: Updating smithi135:/var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T04:27:26.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:26 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:26.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:26 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:26.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:26 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:26.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:26 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:26.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:26 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:27.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:27 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:27.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:27 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:27.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:27 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:27.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:27 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:27.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:27 smithi135 ceph-mon[44364]: Reconfiguring node-exporter.smithi123 (dependencies changed)... 2024-03-30T04:27:27.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:27 smithi135 ceph-mon[44364]: Reconfiguring daemon node-exporter.smithi123 on smithi123 2024-03-30T04:27:27.440 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:27 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:27.440 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:27 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:27.440 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:27 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:27.440 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:27 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:27.440 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:27 smithi123 ceph-mon[35245]: Reconfiguring node-exporter.smithi123 (dependencies changed)... 2024-03-30T04:27:27.440 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:27 smithi123 ceph-mon[35245]: Reconfiguring daemon node-exporter.smithi123 on smithi123 2024-03-30T04:27:28.215 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:28 smithi123 ceph-mon[35245]: pgmap v17: 65 pgs: 65 active+clean; 111 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 3.7 MiB/s wr, 442 op/s 2024-03-30T04:27:28.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:28 smithi135 ceph-mon[44364]: pgmap v17: 65 pgs: 65 active+clean; 111 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 3.7 MiB/s wr, 442 op/s 2024-03-30T04:27:30.461 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:30 smithi123 ceph-mon[35245]: pgmap v18: 65 pgs: 65 active+clean; 111 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 349 op/s 2024-03-30T04:27:30.461 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:30 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:30.461 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:30 smithi123 ceph-mon[35245]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:27:30.461 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:30 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:30.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:30 smithi135 ceph-mon[44364]: pgmap v18: 65 pgs: 65 active+clean; 111 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 349 op/s 2024-03-30T04:27:30.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:30 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:30.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:30 smithi135 ceph-mon[44364]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:27:30.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:30 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:31.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:31 smithi123 ceph-mon[35245]: Reconfiguring alertmanager.smithi123 (dependencies changed)... 2024-03-30T04:27:31.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:31 smithi123 ceph-mon[35245]: Reconfiguring daemon alertmanager.smithi123 on smithi123 2024-03-30T04:27:31.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:31 smithi135 ceph-mon[44364]: Reconfiguring alertmanager.smithi123 (dependencies changed)... 2024-03-30T04:27:31.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:31 smithi135 ceph-mon[44364]: Reconfiguring daemon alertmanager.smithi123 on smithi123 2024-03-30T04:27:32.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:32 smithi135 ceph-mon[44364]: pgmap v19: 65 pgs: 65 active+clean; 129 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 2.1 MiB/s rd, 4.5 MiB/s wr, 500 op/s 2024-03-30T04:27:32.555 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:32 smithi123 ceph-mon[35245]: pgmap v19: 65 pgs: 65 active+clean; 129 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 2.1 MiB/s rd, 4.5 MiB/s wr, 500 op/s 2024-03-30T04:27:34.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:34 smithi123 ceph-mon[35245]: pgmap v20: 65 pgs: 65 active+clean; 129 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 3.0 MiB/s wr, 312 op/s 2024-03-30T04:27:34.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:34 smithi135 ceph-mon[44364]: pgmap v20: 65 pgs: 65 active+clean; 129 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 3.0 MiB/s wr, 312 op/s 2024-03-30T04:27:36.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:36 smithi123 ceph-mon[35245]: pgmap v21: 65 pgs: 65 active+clean; 136 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.8 MiB/s rd, 3.6 MiB/s wr, 382 op/s 2024-03-30T04:27:36.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:36 smithi135 ceph-mon[44364]: pgmap v21: 65 pgs: 65 active+clean; 136 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.8 MiB/s rd, 3.6 MiB/s wr, 382 op/s 2024-03-30T04:27:38.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:38 smithi123 ceph-mon[35245]: pgmap v22: 65 pgs: 65 active+clean; 143 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.8 MiB/s rd, 3.7 MiB/s wr, 403 op/s 2024-03-30T04:27:38.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:38 smithi135 ceph-mon[44364]: pgmap v22: 65 pgs: 65 active+clean; 143 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.8 MiB/s rd, 3.7 MiB/s wr, 403 op/s 2024-03-30T04:27:40.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:40 smithi123 ceph-mon[35245]: pgmap v23: 65 pgs: 65 active+clean; 143 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 299 op/s 2024-03-30T04:27:40.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:40 smithi135 ceph-mon[44364]: pgmap v23: 65 pgs: 65 active+clean; 143 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 299 op/s 2024-03-30T04:27:41.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:41 smithi123 ceph-mon[35245]: pgmap v24: 65 pgs: 65 active+clean; 154 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.9 MiB/s rd, 3.9 MiB/s wr, 444 op/s 2024-03-30T04:27:41.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:41 smithi135 ceph-mon[44364]: pgmap v24: 65 pgs: 65 active+clean; 154 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.9 MiB/s rd, 3.9 MiB/s wr, 444 op/s 2024-03-30T04:27:44.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:44 smithi123 ceph-mon[35245]: pgmap v25: 65 pgs: 65 active+clean; 154 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.0 MiB/s rd, 2.3 MiB/s wr, 294 op/s 2024-03-30T04:27:44.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:44 smithi135 ceph-mon[44364]: pgmap v25: 65 pgs: 65 active+clean; 154 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.0 MiB/s rd, 2.3 MiB/s wr, 294 op/s 2024-03-30T04:27:45.336 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:45 smithi123 ceph-mon[35245]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:27:45.336 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:45 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:45.336 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:45 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:45.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:45 smithi135 ceph-mon[44364]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:27:45.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:45 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:45.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:45 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:46.431 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:46 smithi123 ceph-mon[35245]: Reconfiguring grafana.smithi123 (dependencies changed)... 2024-03-30T04:27:46.431 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:46 smithi123 ceph-mon[35245]: Reconfiguring daemon grafana.smithi123 on smithi123 2024-03-30T04:27:46.431 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:46 smithi123 ceph-mon[35245]: pgmap v26: 65 pgs: 65 active+clean; 167 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 3.4 MiB/s wr, 384 op/s 2024-03-30T04:27:46.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:46 smithi135 ceph-mon[44364]: Reconfiguring grafana.smithi123 (dependencies changed)... 2024-03-30T04:27:46.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:46 smithi135 ceph-mon[44364]: Reconfiguring daemon grafana.smithi123 on smithi123 2024-03-30T04:27:46.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:46 smithi135 ceph-mon[44364]: pgmap v26: 65 pgs: 65 active+clean; 167 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 3.4 MiB/s wr, 384 op/s 2024-03-30T04:27:46.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:46.863+0000 7f6606e00700 1 -- 172.21.15.123:0/2687858402 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66000ff010 msgr2=0x7f66000ff420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:27:46.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:46.863+0000 7f6606e00700 1 --2- 172.21.15.123:0/2687858402 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66000ff010 0x7f66000ff420 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f65f0004940 tx=0x7f65f0009e30 comp rx=0 tx=0).stop 2024-03-30T04:27:46.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:46.863+0000 7f6606e00700 1 -- 172.21.15.123:0/2687858402 shutdown_connections 2024-03-30T04:27:46.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:46.863+0000 7f6606e00700 1 --2- 172.21.15.123:0/2687858402 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f66001002c0 0x7f6600100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:46.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:46.863+0000 7f6606e00700 1 --2- 172.21.15.123:0/2687858402 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66000ff010 0x7f66000ff420 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:46.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:46.863+0000 7f6606e00700 1 -- 172.21.15.123:0/2687858402 >> 172.21.15.123:0/2687858402 conn(0x7f66000fa790 msgr2=0x7f66000fcba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:27:46.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:46.863+0000 7f6606e00700 1 -- 172.21.15.123:0/2687858402 shutdown_connections 2024-03-30T04:27:46.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:46.863+0000 7f6606e00700 1 -- 172.21.15.123:0/2687858402 wait complete. 2024-03-30T04:27:46.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:46.864+0000 7f6606e00700 1 Processor -- start 2024-03-30T04:27:46.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:46.864+0000 7f6606e00700 1 -- start start 2024-03-30T04:27:46.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:46.865+0000 7f6606e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66000ff010 0x7f660010fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:27:46.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:46.865+0000 7f6606e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f66001002c0 0x7f660010ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:27:46.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:46.865+0000 7f6606e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6600110550 con 0x7f66000ff010 2024-03-30T04:27:46.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:46.865+0000 7f6606e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66001106c0 con 0x7f66001002c0 2024-03-30T04:27:46.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:46.865+0000 7f65ffe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66000ff010 0x7f660010fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:27:46.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:46.865+0000 7f65ffe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66000ff010 0x7f660010fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37242/0 (socket says 172.21.15.123:37242) 2024-03-30T04:27:46.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:46.865+0000 7f65ffe00700 1 -- 172.21.15.123:0/3893376192 learned_addr learned my addr 172.21.15.123:0/3893376192 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:27:46.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:46.866+0000 7f65ffe00700 1 -- 172.21.15.123:0/3893376192 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f66001002c0 msgr2=0x7f660010ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:27:46.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:46.866+0000 7f65ffe00700 1 --2- 172.21.15.123:0/3893376192 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f66001002c0 0x7f660010ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:46.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:46.866+0000 7f65ffe00700 1 -- 172.21.15.123:0/3893376192 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65f0005040 con 0x7f66000ff010 2024-03-30T04:27:46.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:46.866+0000 7f65ffe00700 1 --2- 172.21.15.123:0/3893376192 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66000ff010 0x7f660010fa10 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f65f000b0d0 tx=0x7f65f0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:27:46.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:46.866+0000 7f65f7e00700 1 -- 172.21.15.123:0/3893376192 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f65f000e070 con 0x7f66000ff010 2024-03-30T04:27:46.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:46.866+0000 7f65f7e00700 1 -- 172.21.15.123:0/3893376192 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f65f000b410 con 0x7f66000ff010 2024-03-30T04:27:46.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:46.866+0000 7f6606e00700 1 -- 172.21.15.123:0/3893376192 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6600115070 con 0x7f66000ff010 2024-03-30T04:27:46.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:46.866+0000 7f65f7e00700 1 -- 172.21.15.123:0/3893376192 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f65f0012600 con 0x7f66000ff010 2024-03-30T04:27:46.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:46.866+0000 7f6606e00700 1 -- 172.21.15.123:0/3893376192 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6600115630 con 0x7f66000ff010 2024-03-30T04:27:46.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:46.868+0000 7f6606e00700 1 -- 172.21.15.123:0/3893376192 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f660004c710 con 0x7f66000ff010 2024-03-30T04:27:46.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:46.868+0000 7f65f7e00700 1 -- 172.21.15.123:0/3893376192 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) v1 ==== 88766+0+0 (secure 0 0 0) 0x7f65f000b9f0 con 0x7f66000ff010 2024-03-30T04:27:46.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:46.870+0000 7f65f7e00700 1 --2- 172.21.15.123:0/3893376192 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f65e406aed0 0x7f65e406d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:27:46.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:46.870+0000 7f65f7e00700 1 -- 172.21.15.123:0/3893376192 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f65f008a500 con 0x7f66000ff010 2024-03-30T04:27:46.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:46.870+0000 7f65ff400700 1 --2- 172.21.15.123:0/3893376192 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f65e406aed0 0x7f65e406d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:27:46.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:46.871+0000 7f65ff400700 1 --2- 172.21.15.123:0/3893376192 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f65e406aed0 0x7f65e406d380 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f65ec009fd0 tx=0x7f65ec00b040 comp rx=0 tx=0).ready entity=mgr.14608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:27:46.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:46.875+0000 7f65f7e00700 1 -- 172.21.15.123:0/3893376192 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f65f0058490 con 0x7f66000ff010 2024-03-30T04:27:47.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.164+0000 7f6606e00700 1 -- 172.21.15.123:0/3893376192 --> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6600104630 con 0x7f65e406aed0 2024-03-30T04:27:47.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.166+0000 7f65f7e00700 1 -- 172.21.15.123:0/3893376192 <== mgr.14608 v2:172.21.15.135:6828/3601553667 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f6600104630 con 0x7f65e406aed0 2024-03-30T04:27:47.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.169+0000 7f6606e00700 1 -- 172.21.15.123:0/3893376192 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f65e406aed0 msgr2=0x7f65e406d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:27:47.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.169+0000 7f6606e00700 1 --2- 172.21.15.123:0/3893376192 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f65e406aed0 0x7f65e406d380 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f65ec009fd0 tx=0x7f65ec00b040 comp rx=0 tx=0).stop 2024-03-30T04:27:47.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.169+0000 7f6606e00700 1 -- 172.21.15.123:0/3893376192 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66000ff010 msgr2=0x7f660010fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:27:47.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.169+0000 7f6606e00700 1 --2- 172.21.15.123:0/3893376192 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66000ff010 0x7f660010fa10 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f65f000b0d0 tx=0x7f65f0009e30 comp rx=0 tx=0).stop 2024-03-30T04:27:47.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.169+0000 7f6606e00700 1 -- 172.21.15.123:0/3893376192 shutdown_connections 2024-03-30T04:27:47.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.169+0000 7f6606e00700 1 --2- 172.21.15.123:0/3893376192 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f65e406aed0 0x7f65e406d380 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:47.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.169+0000 7f6606e00700 1 --2- 172.21.15.123:0/3893376192 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66000ff010 0x7f660010fa10 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:47.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.169+0000 7f6606e00700 1 --2- 172.21.15.123:0/3893376192 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f66001002c0 0x7f660010ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:47.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.169+0000 7f6606e00700 1 -- 172.21.15.123:0/3893376192 >> 172.21.15.123:0/3893376192 conn(0x7f66000fa790 msgr2=0x7f66000fbe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:27:47.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.169+0000 7f6606e00700 1 -- 172.21.15.123:0/3893376192 shutdown_connections 2024-03-30T04:27:47.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.169+0000 7f6606e00700 1 -- 172.21.15.123:0/3893376192 wait complete. 2024-03-30T04:27:47.186 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:27:47.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.319+0000 7f87a0c00700 1 -- 172.21.15.123:0/1126596511 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f879c071e20 msgr2=0x7f879c070330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:27:47.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.319+0000 7f87a0c00700 1 --2- 172.21.15.123:0/1126596511 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f879c071e20 0x7f879c070330 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f878c004940 tx=0x7f878c009e30 comp rx=0 tx=0).stop 2024-03-30T04:27:47.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.319+0000 7f87a0c00700 1 -- 172.21.15.123:0/1126596511 shutdown_connections 2024-03-30T04:27:47.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.319+0000 7f87a0c00700 1 --2- 172.21.15.123:0/1126596511 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f879c070870 0x7f879c070ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:47.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.319+0000 7f87a0c00700 1 --2- 172.21.15.123:0/1126596511 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f879c071e20 0x7f879c070330 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:47.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.319+0000 7f87a0c00700 1 -- 172.21.15.123:0/1126596511 >> 172.21.15.123:0/1126596511 conn(0x7f879c0f85a0 msgr2=0x7f879c0fa9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:27:47.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.319+0000 7f87a0c00700 1 -- 172.21.15.123:0/1126596511 shutdown_connections 2024-03-30T04:27:47.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.319+0000 7f87a0c00700 1 -- 172.21.15.123:0/1126596511 wait complete. 2024-03-30T04:27:47.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.320+0000 7f87a0c00700 1 Processor -- start 2024-03-30T04:27:47.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.320+0000 7f87a0c00700 1 -- start start 2024-03-30T04:27:47.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.321+0000 7f87a0c00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f879c070870 0x7f879c10fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:27:47.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.321+0000 7f87a0c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f879c071e20 0x7f879c1100d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:27:47.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.321+0000 7f87a0c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f879c1106d0 con 0x7f879c071e20 2024-03-30T04:27:47.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.321+0000 7f87a0c00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f879c110f20 con 0x7f879c070870 2024-03-30T04:27:47.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.321+0000 7f8799400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f879c071e20 0x7f879c1100d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:27:47.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.321+0000 7f8799e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f879c070870 0x7f879c10fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:27:47.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.321+0000 7f8799400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f879c071e20 0x7f879c1100d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37260/0 (socket says 172.21.15.123:37260) 2024-03-30T04:27:47.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.321+0000 7f8799400700 1 -- 172.21.15.123:0/4080065791 learned_addr learned my addr 172.21.15.123:0/4080065791 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:27:47.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.321+0000 7f8799400700 1 -- 172.21.15.123:0/4080065791 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f879c070870 msgr2=0x7f879c10fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:27:47.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.321+0000 7f8799400700 1 --2- 172.21.15.123:0/4080065791 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f879c070870 0x7f879c10fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:47.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.321+0000 7f8799400700 1 -- 172.21.15.123:0/4080065791 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f878c005040 con 0x7f879c071e20 2024-03-30T04:27:47.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.322+0000 7f8799400700 1 --2- 172.21.15.123:0/4080065791 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f879c071e20 0x7f879c1100d0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f878800eb70 tx=0x7f878800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:27:47.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.322+0000 7f8792000700 1 -- 172.21.15.123:0/4080065791 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8788007cc0 con 0x7f879c071e20 2024-03-30T04:27:47.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.322+0000 7f8792000700 1 -- 172.21.15.123:0/4080065791 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f878800f040 con 0x7f879c071e20 2024-03-30T04:27:47.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.322+0000 7f87a0c00700 1 -- 172.21.15.123:0/4080065791 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f879c1111a0 con 0x7f879c071e20 2024-03-30T04:27:47.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.322+0000 7f8792000700 1 -- 172.21.15.123:0/4080065791 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f878800c9f0 con 0x7f879c071e20 2024-03-30T04:27:47.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.322+0000 7f87a0c00700 1 -- 172.21.15.123:0/4080065791 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f879c1116e0 con 0x7f879c071e20 2024-03-30T04:27:47.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.323+0000 7f87a0c00700 1 -- 172.21.15.123:0/4080065791 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f879c05f5f0 con 0x7f879c071e20 2024-03-30T04:27:47.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.324+0000 7f8792000700 1 -- 172.21.15.123:0/4080065791 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) v1 ==== 88766+0+0 (secure 0 0 0) 0x7f87880090d0 con 0x7f879c071e20 2024-03-30T04:27:47.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.325+0000 7f8792000700 1 --2- 172.21.15.123:0/4080065791 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f878006aec0 0x7f878006d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:27:47.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.325+0000 7f8792000700 1 -- 172.21.15.123:0/4080065791 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f8788089490 con 0x7f879c071e20 2024-03-30T04:27:47.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.325+0000 7f8799e00700 1 --2- 172.21.15.123:0/4080065791 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f878006aec0 0x7f878006d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:27:47.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.326+0000 7f8799e00700 1 --2- 172.21.15.123:0/4080065791 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f878006aec0 0x7f878006d370 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f878c013870 tx=0x7f878c009e30 comp rx=0 tx=0).ready entity=mgr.14608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:27:47.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.330+0000 7f8792000700 1 -- 172.21.15.123:0/4080065791 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f87880572f0 con 0x7f879c071e20 2024-03-30T04:27:47.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.631+0000 7f87a0c00700 1 -- 172.21.15.123:0/4080065791 --> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f879c111a10 con 0x7f878006aec0 2024-03-30T04:27:47.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.633+0000 7f8792000700 1 -- 172.21.15.123:0/4080065791 <== mgr.14608 v2:172.21.15.135:6828/3601553667 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f879c111a10 con 0x7f878006aec0 2024-03-30T04:27:47.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.636+0000 7f87a0c00700 1 -- 172.21.15.123:0/4080065791 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f878006aec0 msgr2=0x7f878006d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:27:47.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.636+0000 7f87a0c00700 1 --2- 172.21.15.123:0/4080065791 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f878006aec0 0x7f878006d370 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f878c013870 tx=0x7f878c009e30 comp rx=0 tx=0).stop 2024-03-30T04:27:47.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.636+0000 7f87a0c00700 1 -- 172.21.15.123:0/4080065791 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f879c071e20 msgr2=0x7f879c1100d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:27:47.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.636+0000 7f87a0c00700 1 --2- 172.21.15.123:0/4080065791 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f879c071e20 0x7f879c1100d0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f878800eb70 tx=0x7f878800b040 comp rx=0 tx=0).stop 2024-03-30T04:27:47.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.636+0000 7f87a0c00700 1 -- 172.21.15.123:0/4080065791 shutdown_connections 2024-03-30T04:27:47.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.636+0000 7f87a0c00700 1 --2- 172.21.15.123:0/4080065791 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f878006aec0 0x7f878006d370 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:47.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.636+0000 7f87a0c00700 1 --2- 172.21.15.123:0/4080065791 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f879c070870 0x7f879c10fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:47.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.636+0000 7f87a0c00700 1 --2- 172.21.15.123:0/4080065791 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f879c071e20 0x7f879c1100d0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:47.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.636+0000 7f87a0c00700 1 -- 172.21.15.123:0/4080065791 >> 172.21.15.123:0/4080065791 conn(0x7f879c0f85a0 msgr2=0x7f879c0f9da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:27:47.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.637+0000 7f87a0c00700 1 -- 172.21.15.123:0/4080065791 shutdown_connections 2024-03-30T04:27:47.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.637+0000 7f87a0c00700 1 -- 172.21.15.123:0/4080065791 wait complete. 2024-03-30T04:27:47.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.815+0000 7fc12c200700 1 -- 172.21.15.123:0/3854207097 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc1241033f0 msgr2=0x7fc124103800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:27:47.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.815+0000 7fc12c200700 1 --2- 172.21.15.123:0/3854207097 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc1241033f0 0x7fc124103800 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc118004940 tx=0x7fc118009e30 comp rx=0 tx=0).stop 2024-03-30T04:27:47.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.816+0000 7fc12c200700 1 -- 172.21.15.123:0/3854207097 shutdown_connections 2024-03-30T04:27:47.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.816+0000 7fc12c200700 1 --2- 172.21.15.123:0/3854207097 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc1241046a0 0x7fc124104b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:47.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.816+0000 7fc12c200700 1 --2- 172.21.15.123:0/3854207097 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc1241033f0 0x7fc124103800 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:47.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.816+0000 7fc12c200700 1 -- 172.21.15.123:0/3854207097 >> 172.21.15.123:0/3854207097 conn(0x7fc124066d00 msgr2=0x7fc124067120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:27:47.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.816+0000 7fc12c200700 1 -- 172.21.15.123:0/3854207097 shutdown_connections 2024-03-30T04:27:47.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.816+0000 7fc12c200700 1 -- 172.21.15.123:0/3854207097 wait complete. 2024-03-30T04:27:47.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.817+0000 7fc12c200700 1 Processor -- start 2024-03-30T04:27:47.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.817+0000 7fc12c200700 1 -- start start 2024-03-30T04:27:47.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.817+0000 7fc12c200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc1241033f0 0x7fc124113f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:27:47.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.817+0000 7fc12c200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc1241046a0 0x7fc1241144a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:27:47.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.818+0000 7fc12c200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc124114aa0 con 0x7fc1241046a0 2024-03-30T04:27:47.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.818+0000 7fc12c200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1241152f0 con 0x7fc1241033f0 2024-03-30T04:27:47.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.818+0000 7fc129800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc1241033f0 0x7fc124113f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:27:47.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.818+0000 7fc129800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc1241033f0 0x7fc124113f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.123:48102/0 (socket says 172.21.15.123:48102) 2024-03-30T04:27:47.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.818+0000 7fc129800700 1 -- 172.21.15.123:0/3868555964 learned_addr learned my addr 172.21.15.123:0/3868555964 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:27:47.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.818+0000 7fc128e00700 1 --2- 172.21.15.123:0/3868555964 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc1241046a0 0x7fc1241144a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:27:47.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.819+0000 7fc129800700 1 -- 172.21.15.123:0/3868555964 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc1241046a0 msgr2=0x7fc1241144a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:27:47.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.819+0000 7fc129800700 1 --2- 172.21.15.123:0/3868555964 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc1241046a0 0x7fc1241144a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:47.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.819+0000 7fc129800700 1 -- 172.21.15.123:0/3868555964 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc118005040 con 0x7fc1241033f0 2024-03-30T04:27:47.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.819+0000 7fc129800700 1 --2- 172.21.15.123:0/3868555964 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc1241033f0 0x7fc124113f60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc11800b3b0 tx=0x7fc118009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:27:47.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.820+0000 7fc121600700 1 -- 172.21.15.123:0/3868555964 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc11800e040 con 0x7fc1241033f0 2024-03-30T04:27:47.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.820+0000 7fc121600700 1 -- 172.21.15.123:0/3868555964 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc118004290 con 0x7fc1241033f0 2024-03-30T04:27:47.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.820+0000 7fc12c200700 1 -- 172.21.15.123:0/3868555964 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc124115570 con 0x7fc1241033f0 2024-03-30T04:27:47.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.820+0000 7fc121600700 1 -- 172.21.15.123:0/3868555964 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc11801c3f0 con 0x7fc1241033f0 2024-03-30T04:27:47.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.820+0000 7fc12c200700 1 -- 172.21.15.123:0/3868555964 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc124115ab0 con 0x7fc1241033f0 2024-03-30T04:27:47.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.821+0000 7fc12c200700 1 -- 172.21.15.123:0/3868555964 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc12405f5f0 con 0x7fc1241033f0 2024-03-30T04:27:47.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.822+0000 7fc121600700 1 -- 172.21.15.123:0/3868555964 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 24) v1 ==== 88766+0+0 (secure 0 0 0) 0x7fc11800bb80 con 0x7fc1241033f0 2024-03-30T04:27:47.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.823+0000 7fc121600700 1 --2- 172.21.15.123:0/3868555964 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7fc10c06aec0 0x7fc10c06d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:27:47.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.823+0000 7fc121600700 1 -- 172.21.15.123:0/3868555964 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7fc11808ab90 con 0x7fc1241033f0 2024-03-30T04:27:47.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.824+0000 7fc128e00700 1 --2- 172.21.15.123:0/3868555964 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7fc10c06aec0 0x7fc10c06d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:27:47.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.825+0000 7fc128e00700 1 --2- 172.21.15.123:0/3868555964 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7fc10c06aec0 0x7fc10c06d370 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc110004f40 tx=0x7fc11000b040 comp rx=0 tx=0).ready entity=mgr.14608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:27:47.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:47.831+0000 7fc121600700 1 -- 172.21.15.123:0/3868555964 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fc118058a70 con 0x7fc1241033f0 2024-03-30T04:27:48.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.136+0000 7fc12c200700 1 -- 172.21.15.123:0/3868555964 --> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc124115e80 con 0x7fc10c06aec0 2024-03-30T04:27:48.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.149+0000 7fc121600700 1 -- 172.21.15.123:0/3868555964 <== mgr.14608 v2:172.21.15.135:6828/3601553667 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3388 (secure 0 0 0) 0x7fc124115e80 con 0x7fc10c06aec0 2024-03-30T04:27:48.148 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:27:48.148 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 starting - - - - 2024-03-30T04:27:48.148 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (9m) 33s ago 9m 6647k - 17.2.7-797-g5c024335 f77085f6aab7 bceb8b8c4837 2024-03-30T04:27:48.148 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (8m) 35s ago 8m 6651k - 17.2.7-797-g5c024335 f77085f6aab7 cd0fe67b2d34 2024-03-30T04:27:48.149 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (7m) 33s ago 9m 73.6M - 9.4.7 954c08fa6188 b7d7eaffc084 2024-03-30T04:27:48.149 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (3m) 33s ago 3m 84.0M - 17.2.7-797-g5c024335 f77085f6aab7 2df066fba7a6 2024-03-30T04:27:48.149 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (3m) 33s ago 3m 32.5M - 17.2.7-797-g5c024335 f77085f6aab7 1293b8520db1 2024-03-30T04:27:48.149 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (3m) 35s ago 3m 23.5M - 17.2.7-797-g5c024335 f77085f6aab7 0dc41326818c 2024-03-30T04:27:48.149 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (3m) 35s ago 3m 25.7M - 17.2.7-797-g5c024335 f77085f6aab7 aef8a0cfc553 2024-03-30T04:27:48.149 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:9283 running (11m) 33s ago 11m 160M - 17.2.7-797-g5c024335 f77085f6aab7 7b98aee9a3b6 2024-03-30T04:27:48.149 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283 running (62s) 35s ago 8m 479M - 18.2.2-1018-g470100a4 e7c444a08cf7 20e5e0a05742 2024-03-30T04:27:48.149 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (11m) 33s ago 11m 45.2M 2048M 17.2.7-797-g5c024335 f77085f6aab7 51d18587a7df 2024-03-30T04:27:48.149 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (8m) 35s ago 8m 37.6M 2048M 17.2.7-797-g5c024335 f77085f6aab7 f81052b09dd8 2024-03-30T04:27:48.149 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 starting - - - - 2024-03-30T04:27:48.149 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (8m) 35s ago 8m 7541k - 1.5.0 0da6a335fe13 3c2c8e799cc2 2024-03-30T04:27:48.149 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (7m) 33s ago 7m 67.2M 1618M 17.2.7-797-g5c024335 f77085f6aab7 57a7ba3fd75e 2024-03-30T04:27:48.150 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (6m) 33s ago 6m 72.0M 1618M 17.2.7-797-g5c024335 f77085f6aab7 e1a6c1818a0c 2024-03-30T04:27:48.150 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (6m) 33s ago 6m 60.7M 1618M 17.2.7-797-g5c024335 f77085f6aab7 4c6a727567bd 2024-03-30T04:27:48.150 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (5m) 35s ago 5m 65.2M 2642M 17.2.7-797-g5c024335 f77085f6aab7 6fbdd1d46a9d 2024-03-30T04:27:48.150 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (5m) 35s ago 5m 61.2M 2642M 17.2.7-797-g5c024335 f77085f6aab7 b74988e947e5 2024-03-30T04:27:48.150 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (4m) 35s ago 4m 62.7M 2642M 17.2.7-797-g5c024335 f77085f6aab7 1fd3185f5484 2024-03-30T04:27:48.150 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (7m) 33s ago 9m 46.6M - 2.43.0 a07b618ecd1d ba3c351b14cf 2024-03-30T04:27:48.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.152+0000 7fc12c200700 1 -- 172.21.15.123:0/3868555964 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7fc10c06aec0 msgr2=0x7fc10c06d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:27:48.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.152+0000 7fc12c200700 1 --2- 172.21.15.123:0/3868555964 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7fc10c06aec0 0x7fc10c06d370 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc110004f40 tx=0x7fc11000b040 comp rx=0 tx=0).stop 2024-03-30T04:27:48.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.152+0000 7fc12c200700 1 -- 172.21.15.123:0/3868555964 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc1241033f0 msgr2=0x7fc124113f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:27:48.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.152+0000 7fc12c200700 1 --2- 172.21.15.123:0/3868555964 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc1241033f0 0x7fc124113f60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc11800b3b0 tx=0x7fc118009e30 comp rx=0 tx=0).stop 2024-03-30T04:27:48.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.152+0000 7fc12c200700 1 -- 172.21.15.123:0/3868555964 shutdown_connections 2024-03-30T04:27:48.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.152+0000 7fc12c200700 1 --2- 172.21.15.123:0/3868555964 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7fc10c06aec0 0x7fc10c06d370 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:48.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.152+0000 7fc12c200700 1 --2- 172.21.15.123:0/3868555964 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc1241033f0 0x7fc124113f60 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:48.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.152+0000 7fc12c200700 1 --2- 172.21.15.123:0/3868555964 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc1241046a0 0x7fc1241144a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:48.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.152+0000 7fc12c200700 1 -- 172.21.15.123:0/3868555964 >> 172.21.15.123:0/3868555964 conn(0x7fc124066d00 msgr2=0x7fc124075290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:27:48.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.153+0000 7fc12c200700 1 -- 172.21.15.123:0/3868555964 shutdown_connections 2024-03-30T04:27:48.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.153+0000 7fc12c200700 1 -- 172.21.15.123:0/3868555964 wait complete. 2024-03-30T04:27:48.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.339+0000 7fa6aba00700 1 -- 172.21.15.123:0/2291306207 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6a40fcf80 msgr2=0x7fa6a40fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:27:48.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.339+0000 7fa6aba00700 1 --2- 172.21.15.123:0/2291306207 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6a40fcf80 0x7fa6a40fd3f0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fa690004940 tx=0x7fa690009e30 comp rx=0 tx=0).stop 2024-03-30T04:27:48.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.339+0000 7fa6aba00700 1 -- 172.21.15.123:0/2291306207 shutdown_connections 2024-03-30T04:27:48.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.339+0000 7fa6aba00700 1 --2- 172.21.15.123:0/2291306207 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6a40fcf80 0x7fa6a40fd3f0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:48.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.339+0000 7fa6aba00700 1 --2- 172.21.15.123:0/2291306207 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa6a40fbcd0 0x7fa6a40fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:48.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.339+0000 7fa6aba00700 1 -- 172.21.15.123:0/2291306207 >> 172.21.15.123:0/2291306207 conn(0x7fa6a40f7430 msgr2=0x7fa6a40f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:27:48.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.340+0000 7fa6aba00700 1 -- 172.21.15.123:0/2291306207 shutdown_connections 2024-03-30T04:27:48.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.340+0000 7fa6aba00700 1 -- 172.21.15.123:0/2291306207 wait complete. 2024-03-30T04:27:48.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.341+0000 7fa6aba00700 1 Processor -- start 2024-03-30T04:27:48.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.341+0000 7fa6aba00700 1 -- start start 2024-03-30T04:27:48.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.341+0000 7fa6aba00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6a40fbcd0 0x7fa6a406e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:27:48.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.341+0000 7fa6aba00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa6a40fcf80 0x7fa6a406e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:27:48.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.342+0000 7fa6aba00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a4119970 con 0x7fa6a40fbcd0 2024-03-30T04:27:48.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.342+0000 7fa6aba00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a4119ab0 con 0x7fa6a40fcf80 2024-03-30T04:27:48.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.343+0000 7fa6a9000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6a40fbcd0 0x7fa6a406e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:27:48.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.343+0000 7fa6a9000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6a40fbcd0 0x7fa6a406e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37292/0 (socket says 172.21.15.123:37292) 2024-03-30T04:27:48.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.343+0000 7fa6a9000700 1 -- 172.21.15.123:0/3121656683 learned_addr learned my addr 172.21.15.123:0/3121656683 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:27:48.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.343+0000 7fa6a3e00700 1 --2- 172.21.15.123:0/3121656683 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa6a40fcf80 0x7fa6a406e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:27:48.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.343+0000 7fa6a3e00700 1 -- 172.21.15.123:0/3121656683 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6a40fbcd0 msgr2=0x7fa6a406e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:27:48.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.343+0000 7fa6a3e00700 1 --2- 172.21.15.123:0/3121656683 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6a40fbcd0 0x7fa6a406e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:48.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.343+0000 7fa6a3e00700 1 -- 172.21.15.123:0/3121656683 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa690005040 con 0x7fa6a40fcf80 2024-03-30T04:27:48.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.344+0000 7fa6a9000700 1 --2- 172.21.15.123:0/3121656683 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6a40fbcd0 0x7fa6a406e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:27:48.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.344+0000 7fa6a3e00700 1 --2- 172.21.15.123:0/3121656683 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa6a40fcf80 0x7fa6a406e9c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa690009fd0 tx=0x7fa690009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:27:48.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.345+0000 7fa6a0c00700 1 -- 172.21.15.123:0/3121656683 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa690020070 con 0x7fa6a40fcf80 2024-03-30T04:27:48.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.345+0000 7fa6a0c00700 1 -- 172.21.15.123:0/3121656683 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa69001c400 con 0x7fa6a40fcf80 2024-03-30T04:27:48.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.345+0000 7fa6a0c00700 1 -- 172.21.15.123:0/3121656683 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa69000e070 con 0x7fa6a40fcf80 2024-03-30T04:27:48.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.345+0000 7fa6aba00700 1 -- 172.21.15.123:0/3121656683 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6a406ef00 con 0x7fa6a40fcf80 2024-03-30T04:27:48.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.346+0000 7fa6aba00700 1 -- 172.21.15.123:0/3121656683 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6a4122570 con 0x7fa6a40fcf80 2024-03-30T04:27:48.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.350+0000 7fa6aba00700 1 -- 172.21.15.123:0/3121656683 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6a405f5f0 con 0x7fa6a40fcf80 2024-03-30T04:27:48.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.351+0000 7fa6a0c00700 1 -- 172.21.15.123:0/3121656683 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 24) v1 ==== 88766+0+0 (secure 0 0 0) 0x7fa69000b760 con 0x7fa6a40fcf80 2024-03-30T04:27:48.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.352+0000 7fa6a0c00700 1 --2- 172.21.15.123:0/3121656683 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7fa68806af20 0x7fa68806d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:27:48.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.353+0000 7fa6a0c00700 1 -- 172.21.15.123:0/3121656683 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7fa69008a830 con 0x7fa6a40fcf80 2024-03-30T04:27:48.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.353+0000 7fa6a9000700 1 --2- 172.21.15.123:0/3121656683 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7fa68806af20 0x7fa68806d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:27:48.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.354+0000 7fa6a9000700 1 --2- 172.21.15.123:0/3121656683 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7fa68806af20 0x7fa68806d3d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa69c004fd0 tx=0x7fa69c00b040 comp rx=0 tx=0).ready entity=mgr.14608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:27:48.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.358+0000 7fa6a0c00700 1 -- 172.21.15.123:0/3121656683 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fa690055c60 con 0x7fa6a40fcf80 2024-03-30T04:27:48.449 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:48 smithi123 ceph-mon[35245]: pgmap v27: 65 pgs: 65 active+clean; 173 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 3.3 MiB/s wr, 357 op/s 2024-03-30T04:27:48.449 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:48 smithi123 ceph-mon[35245]: from='client.14656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:27:48.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:48 smithi135 ceph-mon[44364]: pgmap v27: 65 pgs: 65 active+clean; 173 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 3.3 MiB/s wr, 357 op/s 2024-03-30T04:27:48.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:48 smithi135 ceph-mon[44364]: from='client.14656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:27:48.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.791+0000 7fa6aba00700 1 -- 172.21.15.123:0/3121656683 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fa6a4103570 con 0x7fa6a40fcf80 2024-03-30T04:27:48.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.794+0000 7fa6a0c00700 1 -- 172.21.15.123:0/3121656683 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+850 (secure 0 0 0) 0x7fa690059280 con 0x7fa6a40fcf80 2024-03-30T04:27:48.792 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:27:48.793 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:27:48.793 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 2 2024-03-30T04:27:48.793 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:27:48.793 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:27:48.793 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:27:48.793 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 1 2024-03-30T04:27:48.793 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:27:48.793 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:27:48.793 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 6 2024-03-30T04:27:48.793 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:27:48.793 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:27:48.793 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 4 2024-03-30T04:27:48.794 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:27:48.794 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:27:48.794 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 13, 2024-03-30T04:27:48.794 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 1 2024-03-30T04:27:48.794 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:27:48.794 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:27:48.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.798+0000 7fa6aba00700 1 -- 172.21.15.123:0/3121656683 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7fa68806af20 msgr2=0x7fa68806d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:27:48.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.798+0000 7fa6aba00700 1 --2- 172.21.15.123:0/3121656683 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7fa68806af20 0x7fa68806d3d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa69c004fd0 tx=0x7fa69c00b040 comp rx=0 tx=0).stop 2024-03-30T04:27:48.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.798+0000 7fa6aba00700 1 -- 172.21.15.123:0/3121656683 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa6a40fcf80 msgr2=0x7fa6a406e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:27:48.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.798+0000 7fa6aba00700 1 --2- 172.21.15.123:0/3121656683 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa6a40fcf80 0x7fa6a406e9c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa690009fd0 tx=0x7fa690009e30 comp rx=0 tx=0).stop 2024-03-30T04:27:48.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.800+0000 7fa6aba00700 1 -- 172.21.15.123:0/3121656683 shutdown_connections 2024-03-30T04:27:48.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.800+0000 7fa6aba00700 1 --2- 172.21.15.123:0/3121656683 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7fa68806af20 0x7fa68806d3d0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:48.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.800+0000 7fa6aba00700 1 --2- 172.21.15.123:0/3121656683 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6a40fbcd0 0x7fa6a406e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:48.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.800+0000 7fa6aba00700 1 --2- 172.21.15.123:0/3121656683 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa6a40fcf80 0x7fa6a406e9c0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:48.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.800+0000 7fa6aba00700 1 -- 172.21.15.123:0/3121656683 >> 172.21.15.123:0/3121656683 conn(0x7fa6a40f7430 msgr2=0x7fa6a40f8c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:27:48.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.800+0000 7fa6aba00700 1 -- 172.21.15.123:0/3121656683 shutdown_connections 2024-03-30T04:27:48.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:48.800+0000 7fa6aba00700 1 -- 172.21.15.123:0/3121656683 wait complete. 2024-03-30T04:27:49.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:49.005+0000 7f3bde600700 1 -- 172.21.15.123:0/2540408848 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bd80fe0d0 msgr2=0x7f3bd80fe540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:27:49.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:49.005+0000 7f3bde600700 1 --2- 172.21.15.123:0/2540408848 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bd80fe0d0 0x7f3bd80fe540 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f3bc4004940 tx=0x7f3bc4009e30 comp rx=0 tx=0).stop 2024-03-30T04:27:49.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:49.005+0000 7f3bde600700 1 -- 172.21.15.123:0/2540408848 shutdown_connections 2024-03-30T04:27:49.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:49.005+0000 7f3bde600700 1 --2- 172.21.15.123:0/2540408848 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bd80fe0d0 0x7f3bd80fe540 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:49.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:49.005+0000 7f3bde600700 1 --2- 172.21.15.123:0/2540408848 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3bd80fcee0 0x7f3bd80fd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:49.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:49.005+0000 7f3bde600700 1 -- 172.21.15.123:0/2540408848 >> 172.21.15.123:0/2540408848 conn(0x7f3bd80f8560 msgr2=0x7f3bd80fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:27:49.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:49.006+0000 7f3bde600700 1 -- 172.21.15.123:0/2540408848 shutdown_connections 2024-03-30T04:27:49.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:49.006+0000 7f3bde600700 1 -- 172.21.15.123:0/2540408848 wait complete. 2024-03-30T04:27:49.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:49.007+0000 7f3bde600700 1 Processor -- start 2024-03-30T04:27:49.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:49.007+0000 7f3bde600700 1 -- start start 2024-03-30T04:27:49.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:49.008+0000 7f3bde600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3bd80fcee0 0x7f3bd81140e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:27:49.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:49.008+0000 7f3bde600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bd8114620 0x7f3bd8115470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:27:49.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:49.008+0000 7f3bde600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd8114ae0 con 0x7f3bd8114620 2024-03-30T04:27:49.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:49.008+0000 7f3bde600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd8114c20 con 0x7f3bd80fcee0 2024-03-30T04:27:49.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:49.009+0000 7f3bd7400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3bd80fcee0 0x7f3bd81140e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:27:49.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:49.009+0000 7f3bd6a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bd8114620 0x7f3bd8115470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:27:49.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:49.009+0000 7f3bd6a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bd8114620 0x7f3bd8115470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37304/0 (socket says 172.21.15.123:37304) 2024-03-30T04:27:49.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:49.009+0000 7f3bd6a00700 1 -- 172.21.15.123:0/3315929438 learned_addr learned my addr 172.21.15.123:0/3315929438 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:27:49.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:49.010+0000 7f3bd6a00700 1 -- 172.21.15.123:0/3315929438 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3bd80fcee0 msgr2=0x7f3bd81140e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:27:49.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:49.010+0000 7f3bd6a00700 1 --2- 172.21.15.123:0/3315929438 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3bd80fcee0 0x7f3bd81140e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:49.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:49.010+0000 7f3bd6a00700 1 -- 172.21.15.123:0/3315929438 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bc4005040 con 0x7f3bd8114620 2024-03-30T04:27:49.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:49.010+0000 7f3bd6a00700 1 --2- 172.21.15.123:0/3315929438 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bd8114620 0x7f3bd8115470 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f3bc400b0d0 tx=0x7f3bc4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:27:49.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:49.011+0000 7f3bcf400700 1 -- 172.21.15.123:0/3315929438 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3bc400e040 con 0x7f3bd8114620 2024-03-30T04:27:49.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:49.011+0000 7f3bcf400700 1 -- 172.21.15.123:0/3315929438 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3bc400b410 con 0x7f3bd8114620 2024-03-30T04:27:49.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:49.011+0000 7f3bcf400700 1 -- 172.21.15.123:0/3315929438 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3bc4012600 con 0x7f3bd8114620 2024-03-30T04:27:49.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:49.011+0000 7f3bde600700 1 -- 172.21.15.123:0/3315929438 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bd8115a70 con 0x7f3bd8114620 2024-03-30T04:27:49.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:49.011+0000 7f3bde600700 1 -- 172.21.15.123:0/3315929438 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bd8116000 con 0x7f3bd8114620 2024-03-30T04:27:49.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:49.014+0000 7f3bde600700 1 -- 172.21.15.123:0/3315929438 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bd8103e40 con 0x7f3bd8114620 2024-03-30T04:27:49.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:49.014+0000 7f3bcf400700 1 -- 172.21.15.123:0/3315929438 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) v1 ==== 88766+0+0 (secure 0 0 0) 0x7f3bc400b9f0 con 0x7f3bd8114620 2024-03-30T04:27:49.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:49.016+0000 7f3bcf400700 1 --2- 172.21.15.123:0/3315929438 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f3bbc06aec0 0x7f3bbc06d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:27:49.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:49.016+0000 7f3bcf400700 1 -- 172.21.15.123:0/3315929438 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f3bc4089f80 con 0x7f3bd8114620 2024-03-30T04:27:49.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:49.016+0000 7f3bd7400700 1 --2- 172.21.15.123:0/3315929438 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f3bbc06aec0 0x7f3bbc06d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:27:49.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:49.017+0000 7f3bd7400700 1 --2- 172.21.15.123:0/3315929438 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f3bbc06aec0 0x7f3bbc06d370 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3bc8009b80 tx=0x7f3bc800b040 comp rx=0 tx=0).ready entity=mgr.14608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:27:49.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:49.022+0000 7f3bcf400700 1 -- 172.21.15.123:0/3315929438 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f3bc4057e60 con 0x7f3bd8114620 2024-03-30T04:27:49.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:49.357+0000 7f3bde600700 1 -- 172.21.15.123:0/3315929438 --> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3bd81037a0 con 0x7f3bbc06aec0 2024-03-30T04:27:49.356 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:27:49.356 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:27:49.356 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:27:49.356 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-30T04:27:49.356 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [], 2024-03-30T04:27:49.356 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "1/2 daemons upgraded", 2024-03-30T04:27:49.356 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "", 2024-03-30T04:27:49.356 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:27:49.357 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:27:49.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:49.358+0000 7f3bcf400700 1 -- 172.21.15.123:0/3315929438 <== mgr.14608 v2:172.21.15.135:6828/3601553667 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f3bd81037a0 con 0x7f3bbc06aec0 2024-03-30T04:27:49.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:49.361+0000 7f3bde600700 1 -- 172.21.15.123:0/3315929438 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f3bbc06aec0 msgr2=0x7f3bbc06d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:27:49.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:49.362+0000 7f3bde600700 1 --2- 172.21.15.123:0/3315929438 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f3bbc06aec0 0x7f3bbc06d370 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3bc8009b80 tx=0x7f3bc800b040 comp rx=0 tx=0).stop 2024-03-30T04:27:49.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:49.362+0000 7f3bde600700 1 -- 172.21.15.123:0/3315929438 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bd8114620 msgr2=0x7f3bd8115470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:27:49.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:49.362+0000 7f3bde600700 1 --2- 172.21.15.123:0/3315929438 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bd8114620 0x7f3bd8115470 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f3bc400b0d0 tx=0x7f3bc4009e30 comp rx=0 tx=0).stop 2024-03-30T04:27:49.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:49.362+0000 7f3bde600700 1 -- 172.21.15.123:0/3315929438 shutdown_connections 2024-03-30T04:27:49.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:49.362+0000 7f3bde600700 1 --2- 172.21.15.123:0/3315929438 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f3bbc06aec0 0x7f3bbc06d370 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:49.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:49.362+0000 7f3bde600700 1 --2- 172.21.15.123:0/3315929438 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3bd80fcee0 0x7f3bd81140e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:49.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:49.362+0000 7f3bde600700 1 --2- 172.21.15.123:0/3315929438 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bd8114620 0x7f3bd8115470 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:27:49.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:49.362+0000 7f3bde600700 1 -- 172.21.15.123:0/3315929438 >> 172.21.15.123:0/3315929438 conn(0x7f3bd80f8560 msgr2=0x7f3bd8106d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:27:49.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:49.362+0000 7f3bde600700 1 -- 172.21.15.123:0/3315929438 shutdown_connections 2024-03-30T04:27:49.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:27:49.362+0000 7f3bde600700 1 -- 172.21.15.123:0/3315929438 wait complete. 2024-03-30T04:27:49.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:49 smithi123 ceph-mon[35245]: from='client.14660 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:27:49.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:49 smithi123 ceph-mon[35245]: from='client.24477 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:27:49.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:49 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/3121656683' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:27:49.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:49 smithi135 ceph-mon[44364]: from='client.14660 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:27:49.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:49 smithi135 ceph-mon[44364]: from='client.24477 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:27:49.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:49 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/3121656683' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:27:50.289 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:50 smithi123 ceph-mon[35245]: pgmap v28: 65 pgs: 65 active+clean; 173 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 278 op/s 2024-03-30T04:27:50.289 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:50 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:50.289 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:50 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:50.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:50 smithi135 ceph-mon[44364]: pgmap v28: 65 pgs: 65 active+clean; 173 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 278 op/s 2024-03-30T04:27:50.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:50 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:50.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:50 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:27:51.450 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:51 smithi123 ceph-mon[35245]: from='client.14672 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:27:51.451 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:51 smithi123 ceph-mon[35245]: Reconfiguring prometheus.smithi123 (dependencies changed)... 2024-03-30T04:27:51.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:51 smithi135 ceph-mon[44364]: from='client.14672 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:27:51.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:51 smithi135 ceph-mon[44364]: Reconfiguring prometheus.smithi123 (dependencies changed)... 2024-03-30T04:27:52.015 INFO:tasks.workunit.client.0.smithi123.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-03-30T04:27:52.033 INFO:tasks.workunit.client.0.smithi123.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-03-30T04:27:52.033 INFO:tasks.workunit.client.0.smithi123.stderr:+ make 2024-03-30T04:27:52.216 INFO:tasks.workunit.client.0.smithi123.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-03-30T04:27:52.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:52 smithi135 ceph-mon[44364]: Reconfiguring daemon prometheus.smithi123 on smithi123 2024-03-30T04:27:52.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:52 smithi135 ceph-mon[44364]: pgmap v29: 65 pgs: 65 active+clean; 191 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 4.2 MiB/s wr, 423 op/s 2024-03-30T04:27:52.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:52 smithi123 ceph-mon[35245]: Reconfiguring daemon prometheus.smithi123 on smithi123 2024-03-30T04:27:52.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:52 smithi123 ceph-mon[35245]: pgmap v29: 65 pgs: 65 active+clean; 191 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 4.2 MiB/s wr, 423 op/s 2024-03-30T04:27:53.005 INFO:tasks.workunit.client.0.smithi123.stderr:++ readlink -f fsstress 2024-03-30T04:27:53.009 INFO:tasks.workunit.client.0.smithi123.stderr:+ BIN=/home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress 2024-03-30T04:27:53.009 INFO:tasks.workunit.client.0.smithi123.stderr:+ popd 2024-03-30T04:27:53.012 INFO:tasks.workunit.client.0.smithi123.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-03-30T04:27:53.012 INFO:tasks.workunit.client.0.smithi123.stderr:+ popd 2024-03-30T04:27:53.013 INFO:tasks.workunit.client.0.smithi123.stdout:~/cephtest/mnt.0/client.0/tmp 2024-03-30T04:27:53.014 INFO:tasks.workunit.client.0.smithi123.stderr:++ mktemp -d -p . 2024-03-30T04:27:53.019 INFO:tasks.workunit.client.0.smithi123.stderr:+ T=./tmp.V4EPUBEJVW 2024-03-30T04:27:53.019 INFO:tasks.workunit.client.0.smithi123.stderr:+ /home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress -d ./tmp.V4EPUBEJVW -l 1 -n 1000 -p 10 -v 2024-03-30T04:27:53.025 INFO:tasks.workunit.client.0.smithi123.stdout:seed = 1711799051 2024-03-30T04:27:53.031 INFO:tasks.workunit.client.0.smithi123.stdout:0/0: rmdir - no directory 2024-03-30T04:27:53.032 INFO:tasks.workunit.client.0.smithi123.stdout:0/1: write - no filename 2024-03-30T04:27:53.035 INFO:tasks.workunit.client.0.smithi123.stdout:0/2: symlink l0 0 2024-03-30T04:27:53.036 INFO:tasks.workunit.client.0.smithi123.stdout:1/0: chown . 106613842 1 2024-03-30T04:27:53.038 INFO:tasks.workunit.client.0.smithi123.stdout:2/0: write - no filename 2024-03-30T04:27:53.040 INFO:tasks.workunit.client.0.smithi123.stdout:0/3: unlink l0 0 2024-03-30T04:27:53.042 INFO:tasks.workunit.client.0.smithi123.stdout:3/0: dread - no filename 2024-03-30T04:27:53.042 INFO:tasks.workunit.client.0.smithi123.stdout:1/1: mknod c0 0 2024-03-30T04:27:53.044 INFO:tasks.workunit.client.0.smithi123.stdout:0/4: creat f1 x:0 0 0 2024-03-30T04:27:53.045 INFO:tasks.workunit.client.0.smithi123.stdout:0/5: stat f1 0 2024-03-30T04:27:53.045 INFO:tasks.workunit.client.0.smithi123.stdout:2/1: mkdir d0 0 2024-03-30T04:27:53.045 INFO:tasks.workunit.client.0.smithi123.stdout:2/2: write - no filename 2024-03-30T04:27:53.045 INFO:tasks.workunit.client.0.smithi123.stdout:2/3: write - no filename 2024-03-30T04:27:53.045 INFO:tasks.workunit.client.0.smithi123.stdout:2/4: dwrite - no filename 2024-03-30T04:27:53.047 INFO:tasks.workunit.client.0.smithi123.stdout:2/5: rename d0 to d0/d1 22 2024-03-30T04:27:53.047 INFO:tasks.workunit.client.0.smithi123.stdout:2/6: dread - no filename 2024-03-30T04:27:53.047 INFO:tasks.workunit.client.0.smithi123.stdout:2/7: read - no filename 2024-03-30T04:27:53.047 INFO:tasks.workunit.client.0.smithi123.stdout:2/8: dread - no filename 2024-03-30T04:27:53.047 INFO:tasks.workunit.client.0.smithi123.stdout:2/9: unlink - no file 2024-03-30T04:27:53.047 INFO:tasks.workunit.client.0.smithi123.stdout:2/10: write - no filename 2024-03-30T04:27:53.048 INFO:tasks.workunit.client.0.smithi123.stdout:4/0: fsync - no filename 2024-03-30T04:27:53.048 INFO:tasks.workunit.client.0.smithi123.stdout:2/11: chown d0 0 1 2024-03-30T04:27:53.048 INFO:tasks.workunit.client.0.smithi123.stdout:2/12: dwrite - no filename 2024-03-30T04:27:53.048 INFO:tasks.workunit.client.0.smithi123.stdout:2/13: dread - no filename 2024-03-30T04:27:53.048 INFO:tasks.workunit.client.0.smithi123.stdout:2/14: truncate - no filename 2024-03-30T04:27:53.048 INFO:tasks.workunit.client.0.smithi123.stdout:2/15: dread - no filename 2024-03-30T04:27:53.050 INFO:tasks.workunit.client.0.smithi123.stdout:2/16: rename d0 to d0/d2 22 2024-03-30T04:27:53.050 INFO:tasks.workunit.client.0.smithi123.stdout:2/17: unlink - no file 2024-03-30T04:27:53.050 INFO:tasks.workunit.client.0.smithi123.stdout:0/6: unlink f1 0 2024-03-30T04:27:53.050 INFO:tasks.workunit.client.0.smithi123.stdout:0/7: readlink - no filename 2024-03-30T04:27:53.050 INFO:tasks.workunit.client.0.smithi123.stdout:0/8: fdatasync - no filename 2024-03-30T04:27:53.050 INFO:tasks.workunit.client.0.smithi123.stdout:0/9: chown . 18 1 2024-03-30T04:27:53.051 INFO:tasks.workunit.client.0.smithi123.stdout:0/10: rmdir - no directory 2024-03-30T04:27:53.051 INFO:tasks.workunit.client.0.smithi123.stdout:0/11: dread - no filename 2024-03-30T04:27:53.051 INFO:tasks.workunit.client.0.smithi123.stdout:0/12: dread - no filename 2024-03-30T04:27:53.051 INFO:tasks.workunit.client.0.smithi123.stdout:0/13: chown . 1 1 2024-03-30T04:27:53.052 INFO:tasks.workunit.client.0.smithi123.stdout:3/1: creat f0 x:0 0 0 2024-03-30T04:27:53.052 INFO:tasks.workunit.client.0.smithi123.stdout:3/2: readlink - no filename 2024-03-30T04:27:53.054 INFO:tasks.workunit.client.0.smithi123.stdout:5/0: dwrite - no filename 2024-03-30T04:27:53.054 INFO:tasks.workunit.client.0.smithi123.stdout:5/1: link - no file 2024-03-30T04:27:53.054 INFO:tasks.workunit.client.0.smithi123.stdout:5/2: write - no filename 2024-03-30T04:27:53.054 INFO:tasks.workunit.client.0.smithi123.stdout:5/3: unlink - no file 2024-03-30T04:27:53.054 INFO:tasks.workunit.client.0.smithi123.stdout:5/4: dread - no filename 2024-03-30T04:27:53.055 INFO:tasks.workunit.client.0.smithi123.stdout:5/5: chown . 164 1 2024-03-30T04:27:53.056 INFO:tasks.workunit.client.0.smithi123.stdout:0/14: symlink l2 0 2024-03-30T04:27:53.057 INFO:tasks.workunit.client.0.smithi123.stdout:1/2: link c0 c1 0 2024-03-30T04:27:53.057 INFO:tasks.workunit.client.0.smithi123.stdout:1/3: dread - no filename 2024-03-30T04:27:53.057 INFO:tasks.workunit.client.0.smithi123.stdout:0/15: chown l2 95312870 1 2024-03-30T04:27:53.058 INFO:tasks.workunit.client.0.smithi123.stdout:3/3: write f0 [242703,82443] 0 2024-03-30T04:27:53.058 INFO:tasks.workunit.client.0.smithi123.stdout:4/1: symlink l0 0 2024-03-30T04:27:53.058 INFO:tasks.workunit.client.0.smithi123.stdout:4/2: truncate - no filename 2024-03-30T04:27:53.059 INFO:tasks.workunit.client.0.smithi123.stdout:6/0: truncate - no filename 2024-03-30T04:27:53.061 INFO:tasks.workunit.client.0.smithi123.stdout:1/4: symlink l2 0 2024-03-30T04:27:53.061 INFO:tasks.workunit.client.0.smithi123.stdout:1/5: dread - no filename 2024-03-30T04:27:53.061 INFO:tasks.workunit.client.0.smithi123.stdout:0/16: chown l2 1988503 1 2024-03-30T04:27:53.062 INFO:tasks.workunit.client.0.smithi123.stdout:3/4: truncate f0 1290161 0 2024-03-30T04:27:53.063 INFO:tasks.workunit.client.0.smithi123.stdout:4/3: mknod c1 0 2024-03-30T04:27:53.064 INFO:tasks.workunit.client.0.smithi123.stdout:4/4: stat c1 0 2024-03-30T04:27:53.064 INFO:tasks.workunit.client.0.smithi123.stdout:5/6: creat f0 x:0 0 0 2024-03-30T04:27:53.064 INFO:tasks.workunit.client.0.smithi123.stdout:0/17: creat f3 x:0 0 0 2024-03-30T04:27:53.066 INFO:tasks.workunit.client.0.smithi123.stdout:6/1: creat f0 x:0 0 0 2024-03-30T04:27:53.067 INFO:tasks.workunit.client.0.smithi123.stdout:3/5: unlink f0 0 2024-03-30T04:27:53.067 INFO:tasks.workunit.client.0.smithi123.stdout:3/6: read - no filename 2024-03-30T04:27:53.067 INFO:tasks.workunit.client.0.smithi123.stdout:3/7: dread - no filename 2024-03-30T04:27:53.067 INFO:tasks.workunit.client.0.smithi123.stdout:5/7: fdatasync f0 0 2024-03-30T04:27:53.068 INFO:tasks.workunit.client.0.smithi123.stdout:8/0: dread - no filename 2024-03-30T04:27:53.069 INFO:tasks.workunit.client.0.smithi123.stdout:8/1: dread - no filename 2024-03-30T04:27:53.069 INFO:tasks.workunit.client.0.smithi123.stdout:8/2: unlink - no file 2024-03-30T04:27:53.069 INFO:tasks.workunit.client.0.smithi123.stdout:8/3: chown . 158327243 1 2024-03-30T04:27:53.070 INFO:tasks.workunit.client.0.smithi123.stdout:4/5: unlink c1 0 2024-03-30T04:27:53.071 INFO:tasks.workunit.client.0.smithi123.stdout:0/18: creat f4 x:0 0 0 2024-03-30T04:27:53.073 INFO:tasks.workunit.client.0.smithi123.stdout:6/2: rename f0 to f1 0 2024-03-30T04:27:53.075 INFO:tasks.workunit.client.0.smithi123.stdout:3/8: symlink l1 0 2024-03-30T04:27:53.076 INFO:tasks.workunit.client.0.smithi123.stdout:3/9: chown l1 53347 1 2024-03-30T04:27:53.078 INFO:tasks.workunit.client.0.smithi123.stdout:5/8: mknod c1 0 2024-03-30T04:27:53.078 INFO:tasks.workunit.client.0.smithi123.stdout:9/0: chown . 705 1 2024-03-30T04:27:53.078 INFO:tasks.workunit.client.0.smithi123.stdout:9/1: dwrite - no filename 2024-03-30T04:27:53.078 INFO:tasks.workunit.client.0.smithi123.stdout:9/2: write - no filename 2024-03-30T04:27:53.078 INFO:tasks.workunit.client.0.smithi123.stdout:9/3: truncate - no filename 2024-03-30T04:27:53.078 INFO:tasks.workunit.client.0.smithi123.stdout:9/4: unlink - no file 2024-03-30T04:27:53.078 INFO:tasks.workunit.client.0.smithi123.stdout:9/5: write - no filename 2024-03-30T04:27:53.078 INFO:tasks.workunit.client.0.smithi123.stdout:9/6: unlink - no file 2024-03-30T04:27:53.079 INFO:tasks.workunit.client.0.smithi123.stdout:5/9: write f0 [948775,109960] 0 2024-03-30T04:27:53.080 INFO:tasks.workunit.client.0.smithi123.stdout:4/6: mkdir d2 0 2024-03-30T04:27:53.081 INFO:tasks.workunit.client.0.smithi123.stdout:5/10: write f0 [752604,38113] 0 2024-03-30T04:27:53.082 INFO:tasks.workunit.client.0.smithi123.stdout:1/6: link l2 l3 0 2024-03-30T04:27:53.082 INFO:tasks.workunit.client.0.smithi123.stdout:1/7: dwrite - no filename 2024-03-30T04:27:53.082 INFO:tasks.workunit.client.0.smithi123.stdout:4/7: rename d2 to d2/d3 22 2024-03-30T04:27:53.084 INFO:tasks.workunit.client.0.smithi123.stdout:3/10: rename l1 to l2 0 2024-03-30T04:27:53.084 INFO:tasks.workunit.client.0.smithi123.stdout:3/11: truncate - no filename 2024-03-30T04:27:53.085 INFO:tasks.workunit.client.0.smithi123.stdout:8/4: creat f0 x:0 0 0 2024-03-30T04:27:53.089 INFO:tasks.workunit.client.0.smithi123.stdout:6/3: link f1 f2 0 2024-03-30T04:27:53.090 INFO:tasks.workunit.client.0.smithi123.stdout:6/4: dread - f2 zero size 2024-03-30T04:27:53.090 INFO:tasks.workunit.client.0.smithi123.stdout:5/11: creat f2 x:0 0 0 2024-03-30T04:27:53.093 INFO:tasks.workunit.client.0.smithi123.stdout:8/5: symlink l1 0 2024-03-30T04:27:53.093 INFO:tasks.workunit.client.0.smithi123.stdout:5/12: write f0 [894693,115789] 0 2024-03-30T04:27:53.095 INFO:tasks.workunit.client.0.smithi123.stdout:9/7: creat f0 x:0 0 0 2024-03-30T04:27:53.096 INFO:tasks.workunit.client.0.smithi123.stdout:1/8: rename l2 to l4 0 2024-03-30T04:27:53.096 INFO:tasks.workunit.client.0.smithi123.stdout:1/9: dwrite - no filename 2024-03-30T04:27:53.096 INFO:tasks.workunit.client.0.smithi123.stdout:5/13: chown c1 1 1 2024-03-30T04:27:53.097 INFO:tasks.workunit.client.0.smithi123.stdout:5/14: chown f2 200 1 2024-03-30T04:27:53.098 INFO:tasks.workunit.client.0.smithi123.stdout:5/15: write f2 [273772,73891] 0 2024-03-30T04:27:53.099 INFO:tasks.workunit.client.0.smithi123.stdout:4/8: mkdir d2/d4 0 2024-03-30T04:27:53.099 INFO:tasks.workunit.client.0.smithi123.stdout:4/9: dwrite - no filename 2024-03-30T04:27:53.100 INFO:tasks.workunit.client.0.smithi123.stdout:6/5: mknod c3 0 2024-03-30T04:27:53.101 INFO:tasks.workunit.client.0.smithi123.stdout:6/6: dread - f2 zero size 2024-03-30T04:27:53.106 INFO:tasks.workunit.client.0.smithi123.stdout:5/16: creat f3 x:0 0 0 2024-03-30T04:27:53.107 INFO:tasks.workunit.client.0.smithi123.stdout:5/17: fdatasync f0 0 2024-03-30T04:27:53.110 INFO:tasks.workunit.client.0.smithi123.stdout:5/18: write f0 [5434,7756] 0 2024-03-30T04:27:53.110 INFO:tasks.workunit.client.0.smithi123.stdout:5/19: rmdir - no directory 2024-03-30T04:27:53.115 INFO:tasks.workunit.client.0.smithi123.stdout:4/10: symlink d2/d4/l5 0 2024-03-30T04:27:53.117 INFO:tasks.workunit.client.0.smithi123.stdout:6/7: fsync f2 0 2024-03-30T04:27:53.121 INFO:tasks.workunit.client.0.smithi123.stdout:3/12: chown l2 2028 1 2024-03-30T04:27:53.122 INFO:tasks.workunit.client.0.smithi123.stdout:3/13: truncate - no filename 2024-03-30T04:27:53.122 INFO:tasks.workunit.client.0.smithi123.stdout:3/14: write - no filename 2024-03-30T04:27:53.123 INFO:tasks.workunit.client.0.smithi123.stdout:4/11: symlink d2/l6 0 2024-03-30T04:27:53.125 INFO:tasks.workunit.client.0.smithi123.stdout:8/6: getdents . 0 2024-03-30T04:27:53.126 INFO:tasks.workunit.client.0.smithi123.stdout:9/8: getdents . 0 2024-03-30T04:27:53.127 INFO:tasks.workunit.client.0.smithi123.stdout:6/8: creat f4 x:0 0 0 2024-03-30T04:27:53.127 INFO:tasks.workunit.client.0.smithi123.stdout:1/10: link c1 c5 0 2024-03-30T04:27:53.128 INFO:tasks.workunit.client.0.smithi123.stdout:3/15: mknod c3 0 2024-03-30T04:27:53.128 INFO:tasks.workunit.client.0.smithi123.stdout:3/16: dread - no filename 2024-03-30T04:27:53.132 INFO:tasks.workunit.client.0.smithi123.stdout:4/12: mkdir d2/d4/d7 0 2024-03-30T04:27:53.132 INFO:tasks.workunit.client.0.smithi123.stdout:8/7: creat f2 x:0 0 0 2024-03-30T04:27:53.135 INFO:tasks.workunit.client.0.smithi123.stdout:4/13: chown l0 0 1 2024-03-30T04:27:53.136 INFO:tasks.workunit.client.0.smithi123.stdout:4/14: write - no filename 2024-03-30T04:27:53.136 INFO:tasks.workunit.client.0.smithi123.stdout:4/15: dwrite - no filename 2024-03-30T04:27:53.136 INFO:tasks.workunit.client.0.smithi123.stdout:3/17: symlink l4 0 2024-03-30T04:27:53.136 INFO:tasks.workunit.client.0.smithi123.stdout:3/18: dwrite - no filename 2024-03-30T04:27:53.136 INFO:tasks.workunit.client.0.smithi123.stdout:3/19: truncate - no filename 2024-03-30T04:27:53.136 INFO:tasks.workunit.client.0.smithi123.stdout:3/20: dwrite - no filename 2024-03-30T04:27:53.138 INFO:tasks.workunit.client.0.smithi123.stdout:8/8: read - f0 zero size 2024-03-30T04:27:53.138 INFO:tasks.workunit.client.0.smithi123.stdout:1/11: rename l3 to l6 0 2024-03-30T04:27:53.138 INFO:tasks.workunit.client.0.smithi123.stdout:3/21: stat c3 0 2024-03-30T04:27:53.138 INFO:tasks.workunit.client.0.smithi123.stdout:3/22: write - no filename 2024-03-30T04:27:53.139 INFO:tasks.workunit.client.0.smithi123.stdout:3/23: dwrite - no filename 2024-03-30T04:27:53.139 INFO:tasks.workunit.client.0.smithi123.stdout:3/24: dread - no filename 2024-03-30T04:27:53.139 INFO:tasks.workunit.client.0.smithi123.stdout:3/25: truncate - no filename 2024-03-30T04:27:53.139 INFO:tasks.workunit.client.0.smithi123.stdout:6/9: write f4 [750603,108514] 0 2024-03-30T04:27:53.140 INFO:tasks.workunit.client.0.smithi123.stdout:4/16: chown d2/d4/d7 3 1 2024-03-30T04:27:53.141 INFO:tasks.workunit.client.0.smithi123.stdout:8/9: chown f2 2044 1 2024-03-30T04:27:53.143 INFO:tasks.workunit.client.0.smithi123.stdout:5/20: dwrite f2 [0,4194304] 0 2024-03-30T04:27:53.146 INFO:tasks.workunit.client.0.smithi123.stdout:6/10: dread - f2 zero size 2024-03-30T04:27:53.146 INFO:tasks.workunit.client.0.smithi123.stdout:4/17: stat d2/d4/d7 0 2024-03-30T04:27:53.146 INFO:tasks.workunit.client.0.smithi123.stdout:4/18: dread - no filename 2024-03-30T04:27:53.147 INFO:tasks.workunit.client.0.smithi123.stdout:5/21: dread - f3 zero size 2024-03-30T04:27:53.150 INFO:tasks.workunit.client.0.smithi123.stdout:1/12: chown c5 138215510 1 2024-03-30T04:27:53.151 INFO:tasks.workunit.client.0.smithi123.stdout:4/19: stat d2/d4/d7 0 2024-03-30T04:27:53.151 INFO:tasks.workunit.client.0.smithi123.stdout:4/20: truncate - no filename 2024-03-30T04:27:53.152 INFO:tasks.workunit.client.0.smithi123.stdout:4/21: fdatasync - no filename 2024-03-30T04:27:53.155 INFO:tasks.workunit.client.0.smithi123.stdout:8/10: unlink l1 0 2024-03-30T04:27:53.156 INFO:tasks.workunit.client.0.smithi123.stdout:6/11: fdatasync f2 0 2024-03-30T04:27:53.158 INFO:tasks.workunit.client.0.smithi123.stdout:3/26: getdents . 0 2024-03-30T04:27:53.159 INFO:tasks.workunit.client.0.smithi123.stdout:3/27: write - no filename 2024-03-30T04:27:53.159 INFO:tasks.workunit.client.0.smithi123.stdout:3/28: dread - no filename 2024-03-30T04:27:53.160 INFO:tasks.workunit.client.0.smithi123.stdout:4/22: creat d2/d4/f8 x:0 0 0 2024-03-30T04:27:53.162 INFO:tasks.workunit.client.0.smithi123.stdout:6/12: symlink l5 0 2024-03-30T04:27:53.163 INFO:tasks.workunit.client.0.smithi123.stdout:3/29: mknod c5 0 2024-03-30T04:27:53.163 INFO:tasks.workunit.client.0.smithi123.stdout:3/30: fsync - no filename 2024-03-30T04:27:53.163 INFO:tasks.workunit.client.0.smithi123.stdout:3/31: dread - no filename 2024-03-30T04:27:53.164 INFO:tasks.workunit.client.0.smithi123.stdout:4/23: unlink l0 0 2024-03-30T04:27:53.168 INFO:tasks.workunit.client.0.smithi123.stdout:6/13: creat f6 x:0 0 0 2024-03-30T04:27:53.169 INFO:tasks.workunit.client.0.smithi123.stdout:4/24: readlink d2/d4/l5 0 2024-03-30T04:27:53.169 INFO:tasks.workunit.client.0.smithi123.stdout:3/32: creat f6 x:0 0 0 2024-03-30T04:27:53.169 INFO:tasks.workunit.client.0.smithi123.stdout:3/33: rmdir - no directory 2024-03-30T04:27:53.175 INFO:tasks.workunit.client.0.smithi123.stdout:9/9: dwrite f0 [0,4194304] 0 2024-03-30T04:27:53.175 INFO:tasks.workunit.client.0.smithi123.stdout:3/34: read - f6 zero size 2024-03-30T04:27:53.177 INFO:tasks.workunit.client.0.smithi123.stdout:4/25: dread - d2/d4/f8 zero size 2024-03-30T04:27:53.181 INFO:tasks.workunit.client.0.smithi123.stdout:9/10: creat f1 x:0 0 0 2024-03-30T04:27:53.181 INFO:tasks.workunit.client.0.smithi123.stdout:9/11: readlink - no filename 2024-03-30T04:27:53.185 INFO:tasks.workunit.client.0.smithi123.stdout:6/14: truncate f2 238877 0 2024-03-30T04:27:53.192 INFO:tasks.workunit.client.0.smithi123.stdout:4/26: truncate d2/d4/f8 664079 0 2024-03-30T04:27:53.197 INFO:tasks.workunit.client.0.smithi123.stdout:8/11: dwrite f0 [0,4194304] 0 2024-03-30T04:27:53.197 INFO:tasks.workunit.client.0.smithi123.stdout:6/15: stat f1 0 2024-03-30T04:27:53.202 INFO:tasks.workunit.client.0.smithi123.stdout:4/27: symlink d2/d4/l9 0 2024-03-30T04:27:53.205 INFO:tasks.workunit.client.0.smithi123.stdout:6/16: mknod c7 0 2024-03-30T04:27:53.205 INFO:tasks.workunit.client.0.smithi123.stdout:5/22: dwrite f2 [4194304,4194304] 0 2024-03-30T04:27:53.205 INFO:tasks.workunit.client.0.smithi123.stdout:8/12: write f2 [990502,102003] 0 2024-03-30T04:27:53.213 INFO:tasks.workunit.client.0.smithi123.stdout:5/23: read f2 [4313553,56170] 0 2024-03-30T04:27:53.214 INFO:tasks.workunit.client.0.smithi123.stdout:6/17: creat f8 x:0 0 0 2024-03-30T04:27:53.214 INFO:tasks.workunit.client.0.smithi123.stdout:4/28: creat d2/d4/fa x:0 0 0 2024-03-30T04:27:53.219 INFO:tasks.workunit.client.0.smithi123.stdout:5/24: dread - f3 zero size 2024-03-30T04:27:53.224 INFO:tasks.workunit.client.0.smithi123.stdout:3/35: dwrite f6 [0,4194304] 0 2024-03-30T04:27:53.224 INFO:tasks.workunit.client.0.smithi123.stdout:6/18: rename f8 to f9 0 2024-03-30T04:27:53.224 INFO:tasks.workunit.client.0.smithi123.stdout:4/29: write d2/d4/fa [303142,18983] 0 2024-03-30T04:27:53.226 INFO:tasks.workunit.client.0.smithi123.stdout:5/25: write f2 [8952696,61495] 0 2024-03-30T04:27:53.229 INFO:tasks.workunit.client.0.smithi123.stdout:6/19: creat fa x:0 0 0 2024-03-30T04:27:53.230 INFO:tasks.workunit.client.0.smithi123.stdout:4/30: write d2/d4/fa [1223109,95364] 0 2024-03-30T04:27:53.232 INFO:tasks.workunit.client.0.smithi123.stdout:4/31: readlink d2/d4/l9 0 2024-03-30T04:27:53.234 INFO:tasks.workunit.client.0.smithi123.stdout:3/36: rename f6 to f7 0 2024-03-30T04:27:53.241 INFO:tasks.workunit.client.0.smithi123.stdout:8/13: truncate f0 568666 0 2024-03-30T04:27:53.241 INFO:tasks.workunit.client.0.smithi123.stdout:3/37: write f7 [4035807,29379] 0 2024-03-30T04:27:53.243 INFO:tasks.workunit.client.0.smithi123.stdout:8/14: truncate f2 2116873 0 2024-03-30T04:27:53.244 INFO:tasks.workunit.client.0.smithi123.stdout:3/38: chown l2 42584 1 2024-03-30T04:27:53.246 INFO:tasks.workunit.client.0.smithi123.stdout:4/32: creat d2/d4/d7/fb x:0 0 0 2024-03-30T04:27:53.251 INFO:tasks.workunit.client.0.smithi123.stdout:5/26: truncate f2 4164462 0 2024-03-30T04:27:53.251 INFO:tasks.workunit.client.0.smithi123.stdout:4/33: write d2/d4/d7/fb [529598,57698] 0 2024-03-30T04:27:53.253 INFO:tasks.workunit.client.0.smithi123.stdout:5/27: write f3 [284062,38928] 0 2024-03-30T04:27:53.262 INFO:tasks.workunit.client.0.smithi123.stdout:6/20: dwrite fa [0,4194304] 0 2024-03-30T04:27:53.262 INFO:tasks.workunit.client.0.smithi123.stdout:6/21: rmdir - no directory 2024-03-30T04:27:53.263 INFO:tasks.workunit.client.0.smithi123.stdout:5/28: write f0 [1687101,1592] 0 2024-03-30T04:27:53.269 INFO:tasks.workunit.client.0.smithi123.stdout:5/29: symlink l4 0 2024-03-30T04:27:53.271 INFO:tasks.workunit.client.0.smithi123.stdout:5/30: stat l4 0 2024-03-30T04:27:53.273 INFO:tasks.workunit.client.0.smithi123.stdout:3/39: dread f7 [0,4194304] 0 2024-03-30T04:27:53.277 INFO:tasks.workunit.client.0.smithi123.stdout:8/15: dwrite f2 [0,4194304] 0 2024-03-30T04:27:53.283 INFO:tasks.workunit.client.0.smithi123.stdout:8/16: stat f2 0 2024-03-30T04:27:53.283 INFO:tasks.workunit.client.0.smithi123.stdout:6/22: write f2 [1149835,14681] 0 2024-03-30T04:27:53.293 INFO:tasks.workunit.client.0.smithi123.stdout:8/17: mknod c3 0 2024-03-30T04:27:53.293 INFO:tasks.workunit.client.0.smithi123.stdout:3/40: link l2 l8 0 2024-03-30T04:27:53.295 INFO:tasks.workunit.client.0.smithi123.stdout:4/34: dwrite d2/d4/f8 [0,4194304] 0 2024-03-30T04:27:53.297 INFO:tasks.workunit.client.0.smithi123.stdout:5/31: dread f0 [0,4194304] 0 2024-03-30T04:27:53.297 INFO:tasks.workunit.client.0.smithi123.stdout:4/35: stat d2/d4 0 2024-03-30T04:27:53.299 INFO:tasks.workunit.client.0.smithi123.stdout:5/32: mknod c5 0 2024-03-30T04:27:53.305 INFO:tasks.workunit.client.0.smithi123.stdout:4/36: creat d2/d4/d7/fc x:0 0 0 2024-03-30T04:27:53.306 INFO:tasks.workunit.client.0.smithi123.stdout:5/33: mknod c6 0 2024-03-30T04:27:53.314 INFO:tasks.workunit.client.0.smithi123.stdout:5/34: rename f3 to f7 0 2024-03-30T04:27:53.316 INFO:tasks.workunit.client.0.smithi123.stdout:8/18: fdatasync f2 0 2024-03-30T04:27:53.321 INFO:tasks.workunit.client.0.smithi123.stdout:8/19: fdatasync f0 0 2024-03-30T04:27:53.321 INFO:tasks.workunit.client.0.smithi123.stdout:8/20: readlink - no filename 2024-03-30T04:27:53.325 INFO:tasks.workunit.client.0.smithi123.stdout:5/35: dread f7 [0,4194304] 0 2024-03-30T04:27:53.326 INFO:tasks.workunit.client.0.smithi123.stdout:3/41: dwrite f7 [4194304,4194304] 0 2024-03-30T04:27:53.328 INFO:tasks.workunit.client.0.smithi123.stdout:8/21: symlink l4 0 2024-03-30T04:27:53.329 INFO:tasks.workunit.client.0.smithi123.stdout:6/23: dwrite f6 [0,4194304] 0 2024-03-30T04:27:53.329 INFO:tasks.workunit.client.0.smithi123.stdout:8/22: fsync f2 0 2024-03-30T04:27:53.330 INFO:tasks.workunit.client.0.smithi123.stdout:6/24: dread - f9 zero size 2024-03-30T04:27:53.334 INFO:tasks.workunit.client.0.smithi123.stdout:4/37: dread d2/d4/f8 [0,4194304] 0 2024-03-30T04:27:53.334 INFO:tasks.workunit.client.0.smithi123.stdout:5/36: dread f7 [0,4194304] 0 2024-03-30T04:27:53.337 INFO:tasks.workunit.client.0.smithi123.stdout:5/37: symlink l8 0 2024-03-30T04:27:53.339 INFO:tasks.workunit.client.0.smithi123.stdout:5/38: creat f9 x:0 0 0 2024-03-30T04:27:53.341 INFO:tasks.workunit.client.0.smithi123.stdout:4/38: rename d2/l6 to d2/d4/d7/ld 0 2024-03-30T04:27:53.343 INFO:tasks.workunit.client.0.smithi123.stdout:8/23: dread f2 [0,4194304] 0 2024-03-30T04:27:53.343 INFO:tasks.workunit.client.0.smithi123.stdout:5/39: mknod ca 0 2024-03-30T04:27:53.344 INFO:tasks.workunit.client.0.smithi123.stdout:8/24: chown l4 2 1 2024-03-30T04:27:53.345 INFO:tasks.workunit.client.0.smithi123.stdout:5/40: creat fb x:0 0 0 2024-03-30T04:27:53.346 INFO:tasks.workunit.client.0.smithi123.stdout:8/25: truncate f0 676887 0 2024-03-30T04:27:53.346 INFO:tasks.workunit.client.0.smithi123.stdout:5/41: chown c5 131 1 2024-03-30T04:27:53.347 INFO:tasks.workunit.client.0.smithi123.stdout:8/26: chown l4 0 1 2024-03-30T04:27:53.354 INFO:tasks.workunit.client.0.smithi123.stdout:6/25: dread f4 [0,4194304] 0 2024-03-30T04:27:53.360 INFO:tasks.workunit.client.0.smithi123.stdout:6/26: link f9 fb 0 2024-03-30T04:27:53.367 INFO:tasks.workunit.client.0.smithi123.stdout:6/27: symlink lc 0 2024-03-30T04:27:53.372 INFO:tasks.workunit.client.0.smithi123.stdout:6/28: dread - fb zero size 2024-03-30T04:27:53.381 INFO:tasks.workunit.client.0.smithi123.stdout:4/39: dwrite d2/d4/fa [0,4194304] 0 2024-03-30T04:27:53.388 INFO:tasks.workunit.client.0.smithi123.stdout:6/29: rename lc to ld 0 2024-03-30T04:27:53.388 INFO:tasks.workunit.client.0.smithi123.stdout:8/27: dwrite f0 [0,4194304] 0 2024-03-30T04:27:53.389 INFO:tasks.workunit.client.0.smithi123.stdout:5/42: dwrite f7 [0,4194304] 0 2024-03-30T04:27:53.395 INFO:tasks.workunit.client.0.smithi123.stdout:4/40: creat d2/d4/d7/fe x:0 0 0 2024-03-30T04:27:53.395 INFO:tasks.workunit.client.0.smithi123.stdout:5/43: truncate f9 559411 0 2024-03-30T04:27:53.396 INFO:tasks.workunit.client.0.smithi123.stdout:5/44: symlink lc 0 2024-03-30T04:27:53.398 INFO:tasks.workunit.client.0.smithi123.stdout:5/45: read f9 [539495,111266] 0 2024-03-30T04:27:53.400 INFO:tasks.workunit.client.0.smithi123.stdout:4/41: link d2/d4/d7/fe d2/d4/ff 0 2024-03-30T04:27:53.403 INFO:tasks.workunit.client.0.smithi123.stdout:5/46: write f0 [2481253,120580] 0 2024-03-30T04:27:53.404 INFO:tasks.workunit.client.0.smithi123.stdout:4/42: creat d2/d4/f10 x:0 0 0 2024-03-30T04:27:53.405 INFO:tasks.workunit.client.0.smithi123.stdout:5/47: dread - fb zero size 2024-03-30T04:27:53.408 INFO:tasks.workunit.client.0.smithi123.stdout:4/43: mkdir d2/d4/d11 0 2024-03-30T04:27:53.409 INFO:tasks.workunit.client.0.smithi123.stdout:5/48: unlink c5 0 2024-03-30T04:27:53.411 INFO:tasks.workunit.client.0.smithi123.stdout:6/30: dwrite f6 [0,4194304] 0 2024-03-30T04:27:53.412 INFO:tasks.workunit.client.0.smithi123.stdout:5/49: mkdir dd 0 2024-03-30T04:27:53.412 INFO:tasks.workunit.client.0.smithi123.stdout:6/31: write f6 [3711080,83391] 0 2024-03-30T04:27:53.414 INFO:tasks.workunit.client.0.smithi123.stdout:4/44: symlink d2/d4/l12 0 2024-03-30T04:27:53.415 INFO:tasks.workunit.client.0.smithi123.stdout:4/45: rename d2 to d2/d13 22 2024-03-30T04:27:53.416 INFO:tasks.workunit.client.0.smithi123.stdout:6/32: creat fe x:0 0 0 2024-03-30T04:27:53.417 INFO:tasks.workunit.client.0.smithi123.stdout:5/50: link f7 dd/fe 0 2024-03-30T04:27:53.419 INFO:tasks.workunit.client.0.smithi123.stdout:6/33: mknod cf 0 2024-03-30T04:27:53.420 INFO:tasks.workunit.client.0.smithi123.stdout:6/34: write fa [1610105,22060] 0 2024-03-30T04:27:53.422 INFO:tasks.workunit.client.0.smithi123.stdout:8/28: fsync f0 0 2024-03-30T04:27:53.423 INFO:tasks.workunit.client.0.smithi123.stdout:5/51: symlink dd/lf 0 2024-03-30T04:27:53.425 INFO:tasks.workunit.client.0.smithi123.stdout:4/46: rmdir d2/d4/d11 0 2024-03-30T04:27:53.433 INFO:tasks.workunit.client.0.smithi123.stdout:4/47: dread d2/d4/d7/fb [0,4194304] 0 2024-03-30T04:27:53.436 INFO:tasks.workunit.client.0.smithi123.stdout:8/29: truncate f0 2659847 0 2024-03-30T04:27:53.438 INFO:tasks.workunit.client.0.smithi123.stdout:4/48: mknod d2/c14 0 2024-03-30T04:27:53.441 INFO:tasks.workunit.client.0.smithi123.stdout:6/35: dwrite f6 [0,4194304] 0 2024-03-30T04:27:53.446 INFO:tasks.workunit.client.0.smithi123.stdout:8/30: getdents . 0 2024-03-30T04:27:53.448 INFO:tasks.workunit.client.0.smithi123.stdout:4/49: symlink d2/d4/l15 0 2024-03-30T04:27:53.449 INFO:tasks.workunit.client.0.smithi123.stdout:5/52: dwrite f0 [0,4194304] 0 2024-03-30T04:27:53.450 INFO:tasks.workunit.client.0.smithi123.stdout:5/53: readlink l4 0 2024-03-30T04:27:53.452 INFO:tasks.workunit.client.0.smithi123.stdout:5/54: read - fb zero size 2024-03-30T04:27:53.453 INFO:tasks.workunit.client.0.smithi123.stdout:4/50: creat d2/d4/f16 x:0 0 0 2024-03-30T04:27:53.456 INFO:tasks.workunit.client.0.smithi123.stdout:6/36: dread f6 [0,4194304] 0 2024-03-30T04:27:53.457 INFO:tasks.workunit.client.0.smithi123.stdout:8/31: rename f0 to f5 0 2024-03-30T04:27:53.458 INFO:tasks.workunit.client.0.smithi123.stdout:5/55: truncate f9 341588 0 2024-03-30T04:27:53.459 INFO:tasks.workunit.client.0.smithi123.stdout:5/56: readlink l4 0 2024-03-30T04:27:53.460 INFO:tasks.workunit.client.0.smithi123.stdout:5/57: write f0 [3488559,54930] 0 2024-03-30T04:27:53.461 INFO:tasks.workunit.client.0.smithi123.stdout:6/37: mkdir d10 0 2024-03-30T04:27:53.463 INFO:tasks.workunit.client.0.smithi123.stdout:8/32: read f2 [324629,20311] 0 2024-03-30T04:27:53.463 INFO:tasks.workunit.client.0.smithi123.stdout:8/33: rmdir - no directory 2024-03-30T04:27:53.465 INFO:tasks.workunit.client.0.smithi123.stdout:4/51: link d2/d4/l12 d2/d4/d7/l17 0 2024-03-30T04:27:53.466 INFO:tasks.workunit.client.0.smithi123.stdout:6/38: creat d10/f11 x:0 0 0 2024-03-30T04:27:53.467 INFO:tasks.workunit.client.0.smithi123.stdout:4/52: readlink d2/d4/l5 0 2024-03-30T04:27:53.469 INFO:tasks.workunit.client.0.smithi123.stdout:8/34: link f2 f6 0 2024-03-30T04:27:53.470 INFO:tasks.workunit.client.0.smithi123.stdout:6/39: symlink d10/l12 0 2024-03-30T04:27:53.470 INFO:tasks.workunit.client.0.smithi123.stdout:6/40: stat c3 0 2024-03-30T04:27:53.471 INFO:tasks.workunit.client.0.smithi123.stdout:4/53: symlink d2/d4/d7/l18 0 2024-03-30T04:27:53.475 INFO:tasks.workunit.client.0.smithi123.stdout:6/41: mkdir d10/d13 0 2024-03-30T04:27:53.477 INFO:tasks.workunit.client.0.smithi123.stdout:8/35: symlink l7 0 2024-03-30T04:27:53.479 INFO:tasks.workunit.client.0.smithi123.stdout:6/42: unlink d10/l12 0 2024-03-30T04:27:53.482 INFO:tasks.workunit.client.0.smithi123.stdout:5/58: dwrite dd/fe [0,4194304] 0 2024-03-30T04:27:53.482 INFO:tasks.workunit.client.0.smithi123.stdout:4/54: link d2/d4/f16 d2/d4/d7/f19 0 2024-03-30T04:27:53.484 INFO:tasks.workunit.client.0.smithi123.stdout:6/43: symlink d10/l14 0 2024-03-30T04:27:53.484 INFO:tasks.workunit.client.0.smithi123.stdout:4/55: dread - d2/d4/f16 zero size 2024-03-30T04:27:53.487 INFO:tasks.workunit.client.0.smithi123.stdout:8/36: creat f8 x:0 0 0 2024-03-30T04:27:53.487 INFO:tasks.workunit.client.0.smithi123.stdout:8/37: readlink l7 0 2024-03-30T04:27:53.488 INFO:tasks.workunit.client.0.smithi123.stdout:5/59: mknod dd/c10 0 2024-03-30T04:27:53.491 INFO:tasks.workunit.client.0.smithi123.stdout:4/56: mkdir d2/d4/d7/d1a 0 2024-03-30T04:27:53.494 INFO:tasks.workunit.client.0.smithi123.stdout:5/60: rename dd/c10 to dd/c11 0 2024-03-30T04:27:53.495 INFO:tasks.workunit.client.0.smithi123.stdout:5/61: readlink dd/lf 0 2024-03-30T04:27:53.500 INFO:tasks.workunit.client.0.smithi123.stdout:8/38: dread f5 [0,4194304] 0 2024-03-30T04:27:53.500 INFO:tasks.workunit.client.0.smithi123.stdout:8/39: stat f6 0 2024-03-30T04:27:53.502 INFO:tasks.workunit.client.0.smithi123.stdout:6/44: getdents d10 0 2024-03-30T04:27:53.510 INFO:tasks.workunit.client.0.smithi123.stdout:5/62: truncate f9 1299753 0 2024-03-30T04:27:53.512 INFO:tasks.workunit.client.0.smithi123.stdout:4/57: dwrite d2/d4/f8 [0,4194304] 0 2024-03-30T04:27:53.514 INFO:tasks.workunit.client.0.smithi123.stdout:8/40: dread f2 [0,4194304] 0 2024-03-30T04:27:53.514 INFO:tasks.workunit.client.0.smithi123.stdout:4/58: rename d2/d4 to d2/d4/d7/d1b 22 2024-03-30T04:27:53.517 INFO:tasks.workunit.client.0.smithi123.stdout:4/59: write d2/d4/d7/fc [132777,35661] 0 2024-03-30T04:27:53.519 INFO:tasks.workunit.client.0.smithi123.stdout:8/41: write f6 [727106,29506] 0 2024-03-30T04:27:53.523 INFO:tasks.workunit.client.0.smithi123.stdout:6/45: dwrite d10/f11 [0,4194304] 0 2024-03-30T04:27:53.526 INFO:tasks.workunit.client.0.smithi123.stdout:6/46: read f2 [1018950,119213] 0 2024-03-30T04:27:53.527 INFO:tasks.workunit.client.0.smithi123.stdout:6/47: chown fb 2148036 1 2024-03-30T04:27:53.527 INFO:tasks.workunit.client.0.smithi123.stdout:8/42: read f2 [1123495,69017] 0 2024-03-30T04:27:53.532 INFO:tasks.workunit.client.0.smithi123.stdout:8/43: readlink l4 0 2024-03-30T04:27:53.533 INFO:tasks.workunit.client.0.smithi123.stdout:6/48: dread f4 [0,4194304] 0 2024-03-30T04:27:53.537 INFO:tasks.workunit.client.0.smithi123.stdout:6/49: mknod d10/c15 0 2024-03-30T04:27:53.540 INFO:tasks.workunit.client.0.smithi123.stdout:8/44: rename f2 to f9 0 2024-03-30T04:27:53.541 INFO:tasks.workunit.client.0.smithi123.stdout:6/50: unlink c3 0 2024-03-30T04:27:53.545 INFO:tasks.workunit.client.0.smithi123.stdout:8/45: mkdir da 0 2024-03-30T04:27:53.546 INFO:tasks.workunit.client.0.smithi123.stdout:6/51: truncate f4 1617031 0 2024-03-30T04:27:53.550 INFO:tasks.workunit.client.0.smithi123.stdout:4/60: dwrite d2/d4/d7/f19 [0,4194304] 0 2024-03-30T04:27:53.552 INFO:tasks.workunit.client.0.smithi123.stdout:8/46: creat da/fb x:0 0 0 2024-03-30T04:27:53.554 INFO:tasks.workunit.client.0.smithi123.stdout:4/61: creat d2/f1c x:0 0 0 2024-03-30T04:27:53.556 INFO:tasks.workunit.client.0.smithi123.stdout:6/52: write fb [676082,118368] 0 2024-03-30T04:27:53.557 INFO:tasks.workunit.client.0.smithi123.stdout:8/47: mkdir da/dc 0 2024-03-30T04:27:53.558 INFO:tasks.workunit.client.0.smithi123.stdout:4/62: read d2/d4/f8 [888177,24116] 0 2024-03-30T04:27:53.562 INFO:tasks.workunit.client.0.smithi123.stdout:4/63: chown d2/d4/d7/fc 1026467795 1 2024-03-30T04:27:53.565 INFO:tasks.workunit.client.0.smithi123.stdout:4/64: write d2/d4/f16 [1094950,45051] 0 2024-03-30T04:27:53.567 INFO:tasks.workunit.client.0.smithi123.stdout:6/53: dread f2 [0,4194304] 0 2024-03-30T04:27:53.572 INFO:tasks.workunit.client.0.smithi123.stdout:8/48: truncate f9 1677542 0 2024-03-30T04:27:53.574 INFO:tasks.workunit.client.0.smithi123.stdout:6/54: mknod d10/d13/c16 0 2024-03-30T04:27:53.582 INFO:tasks.workunit.client.0.smithi123.stdout:4/65: dwrite d2/d4/f16 [0,4194304] 0 2024-03-30T04:27:53.584 INFO:tasks.workunit.client.0.smithi123.stdout:4/66: stat d2/d4/fa 0 2024-03-30T04:27:53.589 INFO:tasks.workunit.client.0.smithi123.stdout:4/67: symlink d2/d4/d7/l1d 0 2024-03-30T04:27:53.590 INFO:tasks.workunit.client.0.smithi123.stdout:6/55: dread f6 [0,4194304] 0 2024-03-30T04:27:53.594 INFO:tasks.workunit.client.0.smithi123.stdout:8/49: dwrite f8 [0,4194304] 0 2024-03-30T04:27:53.595 INFO:tasks.workunit.client.0.smithi123.stdout:4/68: symlink d2/l1e 0 2024-03-30T04:27:53.600 INFO:tasks.workunit.client.0.smithi123.stdout:4/69: dread d2/d4/d7/fb [0,4194304] 0 2024-03-30T04:27:53.603 INFO:tasks.workunit.client.0.smithi123.stdout:4/70: chown d2/d4/l5 7187307 1 2024-03-30T04:27:53.605 INFO:tasks.workunit.client.0.smithi123.stdout:4/71: truncate d2/d4/f10 601877 0 2024-03-30T04:27:53.608 INFO:tasks.workunit.client.0.smithi123.stdout:6/56: dwrite fa [0,4194304] 0 2024-03-30T04:27:53.610 INFO:tasks.workunit.client.0.smithi123.stdout:6/57: chown f6 949353423 1 2024-03-30T04:27:53.610 INFO:tasks.workunit.client.0.smithi123.stdout:6/58: readlink l5 0 2024-03-30T04:27:53.620 INFO:tasks.workunit.client.0.smithi123.stdout:8/50: dwrite da/fb [0,4194304] 0 2024-03-30T04:27:53.622 INFO:tasks.workunit.client.0.smithi123.stdout:8/51: readlink l7 0 2024-03-30T04:27:53.630 INFO:tasks.workunit.client.0.smithi123.stdout:4/72: dwrite d2/d4/fa [0,4194304] 0 2024-03-30T04:27:53.639 INFO:tasks.workunit.client.0.smithi123.stdout:4/73: creat d2/d4/d7/d1a/f1f x:0 0 0 2024-03-30T04:27:53.640 INFO:tasks.workunit.client.0.smithi123.stdout:6/59: dwrite f4 [0,4194304] 0 2024-03-30T04:27:53.643 INFO:tasks.workunit.client.0.smithi123.stdout:6/60: creat d10/d13/f17 x:0 0 0 2024-03-30T04:27:53.646 INFO:tasks.workunit.client.0.smithi123.stdout:6/61: dread - d10/d13/f17 zero size 2024-03-30T04:27:53.646 INFO:tasks.workunit.client.0.smithi123.stdout:4/74: link d2/d4/d7/d1a/f1f d2/f20 0 2024-03-30T04:27:53.651 INFO:tasks.workunit.client.0.smithi123.stdout:8/52: write f5 [1197502,120183] 0 2024-03-30T04:27:53.655 INFO:tasks.workunit.client.0.smithi123.stdout:6/62: fsync fa 0 2024-03-30T04:27:53.655 INFO:tasks.workunit.client.0.smithi123.stdout:8/53: rename c3 to da/cd 0 2024-03-30T04:27:53.660 INFO:tasks.workunit.client.0.smithi123.stdout:6/63: unlink d10/c15 0 2024-03-30T04:27:53.662 INFO:tasks.workunit.client.0.smithi123.stdout:4/75: write d2/d4/f8 [2390318,27018] 0 2024-03-30T04:27:53.664 INFO:tasks.workunit.client.0.smithi123.stdout:8/54: dread f6 [0,4194304] 0 2024-03-30T04:27:53.664 INFO:tasks.workunit.client.0.smithi123.stdout:4/76: chown d2/d4/d7/l18 16092 1 2024-03-30T04:27:53.665 INFO:tasks.workunit.client.0.smithi123.stdout:6/64: link d10/l14 d10/l18 0 2024-03-30T04:27:53.666 INFO:tasks.workunit.client.0.smithi123.stdout:8/55: getdents da/dc 0 2024-03-30T04:27:53.671 INFO:tasks.workunit.client.0.smithi123.stdout:8/56: dread f6 [0,4194304] 0 2024-03-30T04:27:53.673 INFO:tasks.workunit.client.0.smithi123.stdout:4/77: write d2/d4/ff [509987,38492] 0 2024-03-30T04:27:53.674 INFO:tasks.workunit.client.0.smithi123.stdout:8/57: creat da/fe x:0 0 0 2024-03-30T04:27:53.674 INFO:tasks.workunit.client.0.smithi123.stdout:4/78: readlink d2/d4/l5 0 2024-03-30T04:27:53.674 INFO:tasks.workunit.client.0.smithi123.stdout:8/58: chown f5 4645650 1 2024-03-30T04:27:53.675 INFO:tasks.workunit.client.0.smithi123.stdout:8/59: readlink l7 0 2024-03-30T04:27:53.675 INFO:tasks.workunit.client.0.smithi123.stdout:8/60: readlink l4 0 2024-03-30T04:27:53.677 INFO:tasks.workunit.client.0.smithi123.stdout:4/79: creat d2/d4/d7/d1a/f21 x:0 0 0 2024-03-30T04:27:53.677 INFO:tasks.workunit.client.0.smithi123.stdout:8/61: creat da/ff x:0 0 0 2024-03-30T04:27:53.678 INFO:tasks.workunit.client.0.smithi123.stdout:8/62: dread - da/fe zero size 2024-03-30T04:27:53.682 INFO:tasks.workunit.client.0.smithi123.stdout:4/80: link d2/d4/d7/fb d2/d4/f22 0 2024-03-30T04:27:53.683 INFO:tasks.workunit.client.0.smithi123.stdout:8/63: creat da/dc/f10 x:0 0 0 2024-03-30T04:27:53.684 INFO:tasks.workunit.client.0.smithi123.stdout:4/81: symlink d2/l23 0 2024-03-30T04:27:53.685 INFO:tasks.workunit.client.0.smithi123.stdout:8/64: creat da/f11 x:0 0 0 2024-03-30T04:27:53.687 INFO:tasks.workunit.client.0.smithi123.stdout:8/65: creat da/f12 x:0 0 0 2024-03-30T04:27:53.687 INFO:tasks.workunit.client.0.smithi123.stdout:8/66: dread - da/ff zero size 2024-03-30T04:27:53.689 INFO:tasks.workunit.client.0.smithi123.stdout:8/67: creat da/dc/f13 x:0 0 0 2024-03-30T04:27:53.698 INFO:tasks.workunit.client.0.smithi123.stdout:4/82: dwrite d2/d4/d7/f19 [0,4194304] 0 2024-03-30T04:27:53.701 INFO:tasks.workunit.client.0.smithi123.stdout:4/83: mknod d2/d4/c24 0 2024-03-30T04:27:53.707 INFO:tasks.workunit.client.0.smithi123.stdout:4/84: link d2/d4/l9 d2/l25 0 2024-03-30T04:27:53.712 INFO:tasks.workunit.client.0.smithi123.stdout:4/85: dread d2/d4/f10 [0,4194304] 0 2024-03-30T04:27:53.715 INFO:tasks.workunit.client.0.smithi123.stdout:4/86: mkdir d2/d26 0 2024-03-30T04:27:53.716 INFO:tasks.workunit.client.0.smithi123.stdout:4/87: readlink d2/d4/d7/l1d 0 2024-03-30T04:27:53.733 INFO:tasks.workunit.client.0.smithi123.stdout:4/88: dwrite d2/f20 [0,4194304] 0 2024-03-30T04:27:53.736 INFO:tasks.workunit.client.0.smithi123.stdout:4/89: mkdir d2/d26/d27 0 2024-03-30T04:27:53.738 INFO:tasks.workunit.client.0.smithi123.stdout:4/90: symlink d2/l28 0 2024-03-30T04:27:53.743 INFO:tasks.workunit.client.0.smithi123.stdout:4/91: link d2/l25 d2/l29 0 2024-03-30T04:27:53.745 INFO:tasks.workunit.client.0.smithi123.stdout:4/92: truncate d2/d4/d7/fe 1291647 0 2024-03-30T04:27:54.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:54 smithi123 ceph-mon[35245]: pgmap v30: 65 pgs: 65 active+clean; 191 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 1.2 MiB/s rd, 3.2 MiB/s wr, 278 op/s 2024-03-30T04:27:54.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:54 smithi123 ceph-mon[35245]: fsmap cephfs:2 {0=cephfs.smithi123.dvtind=up:active,1=cephfs.smithi135.zvnlap=up:active} 2 up:standby-replay 2024-03-30T04:27:54.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:54 smithi135 ceph-mon[44364]: pgmap v30: 65 pgs: 65 active+clean; 191 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 1.2 MiB/s rd, 3.2 MiB/s wr, 278 op/s 2024-03-30T04:27:54.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:54 smithi135 ceph-mon[44364]: fsmap cephfs:2 {0=cephfs.smithi123.dvtind=up:active,1=cephfs.smithi135.zvnlap=up:active} 2 up:standby-replay 2024-03-30T04:27:56.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:56 smithi123 ceph-mon[35245]: pgmap v31: 65 pgs: 65 active+clean; 255 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 2.0 MiB/s rd, 11 MiB/s wr, 351 op/s 2024-03-30T04:27:56.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:56 smithi135 ceph-mon[44364]: pgmap v31: 65 pgs: 65 active+clean; 255 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 2.0 MiB/s rd, 11 MiB/s wr, 351 op/s 2024-03-30T04:27:56.786 INFO:tasks.workunit.client.0.smithi123.stdout:2/18: sync 2024-03-30T04:27:56.786 INFO:tasks.workunit.client.0.smithi123.stdout:2/19: write - no filename 2024-03-30T04:27:56.786 INFO:tasks.workunit.client.0.smithi123.stdout:2/20: dread - no filename 2024-03-30T04:27:56.786 INFO:tasks.workunit.client.0.smithi123.stdout:2/21: dread - no filename 2024-03-30T04:27:56.786 INFO:tasks.workunit.client.0.smithi123.stdout:2/22: write - no filename 2024-03-30T04:27:56.786 INFO:tasks.workunit.client.0.smithi123.stdout:2/23: dwrite - no filename 2024-03-30T04:27:56.788 INFO:tasks.workunit.client.0.smithi123.stdout:2/24: mknod d0/c3 0 2024-03-30T04:27:56.789 INFO:tasks.workunit.client.0.smithi123.stdout:2/25: read - no filename 2024-03-30T04:27:56.789 INFO:tasks.workunit.client.0.smithi123.stdout:2/26: dread - no filename 2024-03-30T04:27:56.789 INFO:tasks.workunit.client.0.smithi123.stdout:2/27: dread - no filename 2024-03-30T04:27:56.789 INFO:tasks.workunit.client.0.smithi123.stdout:2/28: truncate - no filename 2024-03-30T04:27:56.790 INFO:tasks.workunit.client.0.smithi123.stdout:2/29: symlink d0/l4 0 2024-03-30T04:27:56.790 INFO:tasks.workunit.client.0.smithi123.stdout:2/30: write - no filename 2024-03-30T04:27:56.791 INFO:tasks.workunit.client.0.smithi123.stdout:2/31: chown d0/l4 41404391 1 2024-03-30T04:27:56.791 INFO:tasks.workunit.client.0.smithi123.stdout:2/32: write - no filename 2024-03-30T04:27:56.791 INFO:tasks.workunit.client.0.smithi123.stdout:2/33: dwrite - no filename 2024-03-30T04:27:56.792 INFO:tasks.workunit.client.0.smithi123.stdout:2/34: dwrite - no filename 2024-03-30T04:27:56.793 INFO:tasks.workunit.client.0.smithi123.stdout:2/35: creat d0/f5 x:0 0 0 2024-03-30T04:27:56.795 INFO:tasks.workunit.client.0.smithi123.stdout:2/36: creat d0/f6 x:0 0 0 2024-03-30T04:27:56.796 INFO:tasks.workunit.client.0.smithi123.stdout:2/37: write d0/f6 [989127,48521] 0 2024-03-30T04:27:56.797 INFO:tasks.workunit.client.0.smithi123.stdout:2/38: readlink d0/l4 0 2024-03-30T04:27:56.801 INFO:tasks.workunit.client.0.smithi123.stdout:2/39: mknod d0/c7 0 2024-03-30T04:27:56.801 INFO:tasks.workunit.client.0.smithi123.stdout:2/40: fdatasync d0/f5 0 2024-03-30T04:27:56.805 INFO:tasks.workunit.client.0.smithi123.stdout:2/41: mkdir d0/d8 0 2024-03-30T04:27:56.808 INFO:tasks.workunit.client.0.smithi123.stdout:2/42: rename d0/c7 to d0/c9 0 2024-03-30T04:27:56.825 INFO:tasks.workunit.client.0.smithi123.stdout:2/43: dwrite d0/f5 [0,4194304] 0 2024-03-30T04:27:56.828 INFO:tasks.workunit.client.0.smithi123.stdout:2/44: unlink d0/l4 0 2024-03-30T04:27:56.831 INFO:tasks.workunit.client.0.smithi123.stdout:7/0: sync 2024-03-30T04:27:56.831 INFO:tasks.workunit.client.0.smithi123.stdout:7/1: chown . 4521 1 2024-03-30T04:27:56.831 INFO:tasks.workunit.client.0.smithi123.stdout:7/2: write - no filename 2024-03-30T04:27:56.831 INFO:tasks.workunit.client.0.smithi123.stdout:7/3: link - no file 2024-03-30T04:27:56.836 INFO:tasks.workunit.client.0.smithi123.stdout:7/4: creat f0 x:0 0 0 2024-03-30T04:27:56.838 INFO:tasks.workunit.client.0.smithi123.stdout:7/5: creat f1 x:0 0 0 2024-03-30T04:27:56.841 INFO:tasks.workunit.client.0.smithi123.stdout:7/6: mkdir d2 0 2024-03-30T04:27:56.845 INFO:tasks.workunit.client.0.smithi123.stdout:7/7: mknod d2/c3 0 2024-03-30T04:27:56.852 INFO:tasks.workunit.client.0.smithi123.stdout:2/45: fdatasync d0/f5 0 2024-03-30T04:27:56.861 INFO:tasks.workunit.client.0.smithi123.stdout:2/46: creat d0/d8/fa x:0 0 0 2024-03-30T04:27:56.866 INFO:tasks.workunit.client.0.smithi123.stdout:7/8: dwrite f1 [0,4194304] 0 2024-03-30T04:27:56.870 INFO:tasks.workunit.client.0.smithi123.stdout:2/47: dread d0/f6 [0,4194304] 0 2024-03-30T04:27:56.872 INFO:tasks.workunit.client.0.smithi123.stdout:7/9: symlink d2/l4 0 2024-03-30T04:27:56.873 INFO:tasks.workunit.client.0.smithi123.stdout:7/10: stat d2/l4 0 2024-03-30T04:27:56.883 INFO:tasks.workunit.client.0.smithi123.stdout:2/48: dread d0/f5 [0,4194304] 0 2024-03-30T04:27:56.884 INFO:tasks.workunit.client.0.smithi123.stdout:2/49: stat d0 0 2024-03-30T04:27:56.905 INFO:tasks.workunit.client.0.smithi123.stdout:2/50: dwrite d0/f6 [0,4194304] 0 2024-03-30T04:27:56.937 INFO:tasks.workunit.client.0.smithi123.stdout:0/19: sync 2024-03-30T04:27:56.937 INFO:tasks.workunit.client.0.smithi123.stdout:0/20: rmdir - no directory 2024-03-30T04:27:56.940 INFO:tasks.workunit.client.0.smithi123.stdout:0/21: creat f5 x:0 0 0 2024-03-30T04:27:56.941 INFO:tasks.workunit.client.0.smithi123.stdout:0/22: dread - f4 zero size 2024-03-30T04:27:56.944 INFO:tasks.workunit.client.0.smithi123.stdout:0/23: mkdir d6 0 2024-03-30T04:27:56.945 INFO:tasks.workunit.client.0.smithi123.stdout:0/24: dread - f5 zero size 2024-03-30T04:27:56.946 INFO:tasks.workunit.client.0.smithi123.stdout:0/25: fsync f4 0 2024-03-30T04:27:56.963 INFO:tasks.workunit.client.0.smithi123.stdout:0/26: dwrite f3 [0,4194304] 0 2024-03-30T04:27:56.964 INFO:tasks.workunit.client.0.smithi123.stdout:0/27: dread - f4 zero size 2024-03-30T04:27:56.972 INFO:tasks.workunit.client.0.smithi123.stdout:1/13: sync 2024-03-30T04:27:56.972 INFO:tasks.workunit.client.0.smithi123.stdout:8/68: sync 2024-03-30T04:27:56.972 INFO:tasks.workunit.client.0.smithi123.stdout:9/12: sync 2024-03-30T04:27:56.972 INFO:tasks.workunit.client.0.smithi123.stdout:5/63: sync 2024-03-30T04:27:56.972 INFO:tasks.workunit.client.0.smithi123.stdout:6/65: sync 2024-03-30T04:27:56.972 INFO:tasks.workunit.client.0.smithi123.stdout:3/42: sync 2024-03-30T04:27:56.972 INFO:tasks.workunit.client.0.smithi123.stdout:4/93: sync 2024-03-30T04:27:56.974 INFO:tasks.workunit.client.0.smithi123.stdout:6/66: fdatasync fa 0 2024-03-30T04:27:56.975 INFO:tasks.workunit.client.0.smithi123.stdout:9/13: dread - f1 zero size 2024-03-30T04:27:56.976 INFO:tasks.workunit.client.0.smithi123.stdout:8/69: read f5 [2201811,59638] 0 2024-03-30T04:27:56.977 INFO:tasks.workunit.client.0.smithi123.stdout:5/64: write dd/fe [3810281,15586] 0 2024-03-30T04:27:56.978 INFO:tasks.workunit.client.0.smithi123.stdout:8/70: chown da 787 1 2024-03-30T04:27:56.984 INFO:tasks.workunit.client.0.smithi123.stdout:4/94: unlink d2/l28 0 2024-03-30T04:27:56.987 INFO:tasks.workunit.client.0.smithi123.stdout:3/43: mknod c9 0 2024-03-30T04:27:56.989 INFO:tasks.workunit.client.0.smithi123.stdout:3/44: chown c9 1 1 2024-03-30T04:27:56.991 INFO:tasks.workunit.client.0.smithi123.stdout:4/95: dread d2/d4/f22 [0,4194304] 0 2024-03-30T04:27:56.991 INFO:tasks.workunit.client.0.smithi123.stdout:0/28: dwrite f5 [0,4194304] 0 2024-03-30T04:27:56.992 INFO:tasks.workunit.client.0.smithi123.stdout:1/14: rename c5 to c7 0 2024-03-30T04:27:56.992 INFO:tasks.workunit.client.0.smithi123.stdout:1/15: dwrite - no filename 2024-03-30T04:27:56.992 INFO:tasks.workunit.client.0.smithi123.stdout:1/16: dread - no filename 2024-03-30T04:27:56.992 INFO:tasks.workunit.client.0.smithi123.stdout:1/17: truncate - no filename 2024-03-30T04:27:56.994 INFO:tasks.workunit.client.0.smithi123.stdout:4/96: write d2/d4/d7/f19 [3587044,114251] 0 2024-03-30T04:27:57.006 INFO:tasks.workunit.client.0.smithi123.stdout:9/14: creat f2 x:0 0 0 2024-03-30T04:27:57.014 INFO:tasks.workunit.client.0.smithi123.stdout:4/97: dread d2/d4/ff [0,4194304] 0 2024-03-30T04:27:57.016 INFO:tasks.workunit.client.0.smithi123.stdout:0/29: dread f3 [0,4194304] 0 2024-03-30T04:27:57.018 INFO:tasks.workunit.client.0.smithi123.stdout:3/45: dwrite f7 [0,4194304] 0 2024-03-30T04:27:57.025 INFO:tasks.workunit.client.0.smithi123.stdout:9/15: mknod c3 0 2024-03-30T04:27:57.028 INFO:tasks.workunit.client.0.smithi123.stdout:1/18: stat c0 0 2024-03-30T04:27:57.037 INFO:tasks.workunit.client.0.smithi123.stdout:3/46: rename c3 to ca 0 2024-03-30T04:27:57.037 INFO:tasks.workunit.client.0.smithi123.stdout:3/47: rmdir - no directory 2024-03-30T04:27:57.040 INFO:tasks.workunit.client.0.smithi123.stdout:3/48: write f7 [6055306,4649] 0 2024-03-30T04:27:57.040 INFO:tasks.workunit.client.0.smithi123.stdout:0/30: dwrite f4 [0,4194304] 0 2024-03-30T04:27:57.043 INFO:tasks.workunit.client.0.smithi123.stdout:9/16: creat f4 x:0 0 0 2024-03-30T04:27:57.059 INFO:tasks.workunit.client.0.smithi123.stdout:4/98: rmdir d2/d26/d27 0 2024-03-30T04:27:57.060 INFO:tasks.workunit.client.0.smithi123.stdout:0/31: mknod d6/c7 0 2024-03-30T04:27:57.069 INFO:tasks.workunit.client.0.smithi123.stdout:4/99: rmdir d2/d4/d7/d1a 39 2024-03-30T04:27:57.071 INFO:tasks.workunit.client.0.smithi123.stdout:0/32: dread f3 [0,4194304] 0 2024-03-30T04:27:57.075 INFO:tasks.workunit.client.0.smithi123.stdout:0/33: unlink f3 0 2024-03-30T04:27:57.189 INFO:tasks.workunit.client.0.smithi123.stdout:6/67: sync 2024-03-30T04:27:57.392 INFO:tasks.workunit.client.0.smithi123.stdout:6/68: stat d10/l18 0 2024-03-30T04:27:57.394 INFO:tasks.workunit.client.0.smithi123.stdout:6/69: creat d10/f19 x:0 0 0 2024-03-30T04:27:57.396 INFO:tasks.workunit.client.0.smithi123.stdout:6/70: mknod d10/d13/c1a 0 2024-03-30T04:27:57.412 INFO:tasks.workunit.client.0.smithi123.stdout:6/71: dwrite fb [0,4194304] 0 2024-03-30T04:27:57.443 INFO:tasks.workunit.client.0.smithi123.stdout:7/11: truncate f1 1686995 0 2024-03-30T04:27:57.465 INFO:tasks.workunit.client.0.smithi123.stdout:7/12: dwrite f0 [0,4194304] 0 2024-03-30T04:27:57.466 INFO:tasks.workunit.client.0.smithi123.stdout:2/51: dwrite d0/f6 [4194304,4194304] 0 2024-03-30T04:27:57.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:27:57 smithi123 ceph-mon[35245]: pgmap v32: 65 pgs: 65 active+clean; 282 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 2.0 MiB/s rd, 13 MiB/s wr, 303 op/s 2024-03-30T04:27:57.469 INFO:tasks.workunit.client.0.smithi123.stdout:7/13: symlink d2/l5 0 2024-03-30T04:27:57.471 INFO:tasks.workunit.client.0.smithi123.stdout:7/14: symlink d2/l6 0 2024-03-30T04:27:57.474 INFO:tasks.workunit.client.0.smithi123.stdout:7/15: rename d2/l4 to d2/l7 0 2024-03-30T04:27:57.487 INFO:tasks.workunit.client.0.smithi123.stdout:1/19: chown c7 26349702 1 2024-03-30T04:27:57.487 INFO:tasks.workunit.client.0.smithi123.stdout:1/20: truncate - no filename 2024-03-30T04:27:57.488 INFO:tasks.workunit.client.0.smithi123.stdout:1/21: stat c7 0 2024-03-30T04:27:57.488 INFO:tasks.workunit.client.0.smithi123.stdout:1/22: dwrite - no filename 2024-03-30T04:27:57.488 INFO:tasks.workunit.client.0.smithi123.stdout:1/23: dwrite - no filename 2024-03-30T04:27:57.490 INFO:tasks.workunit.client.0.smithi123.stdout:2/52: dwrite d0/d8/fa [0,4194304] 0 2024-03-30T04:27:57.495 INFO:tasks.workunit.client.0.smithi123.stdout:5/65: write f2 [1761853,34372] 0 2024-03-30T04:27:57.496 INFO:tasks.workunit.client.0.smithi123.stdout:1/24: creat f8 x:0 0 0 2024-03-30T04:27:57.497 INFO:tasks.workunit.client.0.smithi123.stdout:2/53: symlink d0/lb 0 2024-03-30T04:27:57.498 INFO:tasks.workunit.client.0.smithi123.stdout:5/66: symlink dd/l12 0 2024-03-30T04:27:57.500 INFO:tasks.workunit.client.0.smithi123.stdout:1/25: symlink l9 0 2024-03-30T04:27:57.501 INFO:tasks.workunit.client.0.smithi123.stdout:2/54: write d0/f6 [8749786,27342] 0 2024-03-30T04:27:57.504 INFO:tasks.workunit.client.0.smithi123.stdout:8/71: dwrite f9 [0,4194304] 0 2024-03-30T04:27:57.504 INFO:tasks.workunit.client.0.smithi123.stdout:1/26: truncate f8 500759 0 2024-03-30T04:27:57.507 INFO:tasks.workunit.client.0.smithi123.stdout:8/72: unlink l4 0 2024-03-30T04:27:57.508 INFO:tasks.workunit.client.0.smithi123.stdout:9/17: getdents . 0 2024-03-30T04:27:57.510 INFO:tasks.workunit.client.0.smithi123.stdout:8/73: dread - da/dc/f13 zero size 2024-03-30T04:27:57.511 INFO:tasks.workunit.client.0.smithi123.stdout:1/27: rename c1 to ca 0 2024-03-30T04:27:57.512 INFO:tasks.workunit.client.0.smithi123.stdout:5/67: truncate f9 481988 0 2024-03-30T04:27:57.513 INFO:tasks.workunit.client.0.smithi123.stdout:8/74: read f6 [72799,38859] 0 2024-03-30T04:27:57.515 INFO:tasks.workunit.client.0.smithi123.stdout:9/18: symlink l5 0 2024-03-30T04:27:57.516 INFO:tasks.workunit.client.0.smithi123.stdout:8/75: write da/dc/f13 [59474,56135] 0 2024-03-30T04:27:57.518 INFO:tasks.workunit.client.0.smithi123.stdout:9/19: chown f1 194429090 1 2024-03-30T04:27:57.519 INFO:tasks.workunit.client.0.smithi123.stdout:1/28: mkdir db 0 2024-03-30T04:27:57.519 INFO:tasks.workunit.client.0.smithi123.stdout:9/20: dread - f4 zero size 2024-03-30T04:27:57.524 INFO:tasks.workunit.client.0.smithi123.stdout:5/68: symlink dd/l13 0 2024-03-30T04:27:57.525 INFO:tasks.workunit.client.0.smithi123.stdout:0/34: getdents d6 0 2024-03-30T04:27:57.525 INFO:tasks.workunit.client.0.smithi123.stdout:5/69: fsync f7 0 2024-03-30T04:27:57.528 INFO:tasks.workunit.client.0.smithi123.stdout:3/49: truncate f7 7950048 0 2024-03-30T04:27:57.529 INFO:tasks.workunit.client.0.smithi123.stdout:9/21: mkdir d6 0 2024-03-30T04:27:57.532 INFO:tasks.workunit.client.0.smithi123.stdout:2/55: getdents d0/d8 0 2024-03-30T04:27:57.536 INFO:tasks.workunit.client.0.smithi123.stdout:1/29: symlink db/lc 0 2024-03-30T04:27:57.542 INFO:tasks.workunit.client.0.smithi123.stdout:0/35: dread f5 [0,4194304] 0 2024-03-30T04:27:57.542 INFO:tasks.workunit.client.0.smithi123.stdout:5/70: dread dd/fe [0,4194304] 0 2024-03-30T04:27:57.543 INFO:tasks.workunit.client.0.smithi123.stdout:9/22: creat d6/f7 x:0 0 0 2024-03-30T04:27:57.545 INFO:tasks.workunit.client.0.smithi123.stdout:1/30: mkdir db/dd 0 2024-03-30T04:27:57.547 INFO:tasks.workunit.client.0.smithi123.stdout:8/76: getdents da 0 2024-03-30T04:27:57.549 INFO:tasks.workunit.client.0.smithi123.stdout:1/31: truncate f8 1355143 0 2024-03-30T04:27:57.550 INFO:tasks.workunit.client.0.smithi123.stdout:8/77: dread - da/f11 zero size 2024-03-30T04:27:57.550 INFO:tasks.workunit.client.0.smithi123.stdout:0/36: mknod d6/c8 0 2024-03-30T04:27:57.550 INFO:tasks.workunit.client.0.smithi123.stdout:4/100: dwrite d2/d4/f22 [0,4194304] 0 2024-03-30T04:27:57.552 INFO:tasks.workunit.client.0.smithi123.stdout:4/101: write d2/d4/d7/fb [2737544,12821] 0 2024-03-30T04:27:57.554 INFO:tasks.workunit.client.0.smithi123.stdout:8/78: read - da/dc/f10 zero size 2024-03-30T04:27:57.554 INFO:tasks.workunit.client.0.smithi123.stdout:9/23: unlink f4 0 2024-03-30T04:27:57.555 INFO:tasks.workunit.client.0.smithi123.stdout:8/79: readlink l7 0 2024-03-30T04:27:57.558 INFO:tasks.workunit.client.0.smithi123.stdout:8/80: write da/f11 [492415,72819] 0 2024-03-30T04:27:57.562 INFO:tasks.workunit.client.0.smithi123.stdout:0/37: write f5 [3668794,101430] 0 2024-03-30T04:27:57.562 INFO:tasks.workunit.client.0.smithi123.stdout:1/32: chown l6 141766499 1 2024-03-30T04:27:57.566 INFO:tasks.workunit.client.0.smithi123.stdout:0/38: write f4 [2241807,13506] 0 2024-03-30T04:27:57.566 INFO:tasks.workunit.client.0.smithi123.stdout:9/24: mkdir d6/d8 0 2024-03-30T04:27:57.568 INFO:tasks.workunit.client.0.smithi123.stdout:8/81: mkdir da/dc/d14 0 2024-03-30T04:27:57.571 INFO:tasks.workunit.client.0.smithi123.stdout:8/82: stat da/ff 0 2024-03-30T04:27:57.571 INFO:tasks.workunit.client.0.smithi123.stdout:9/25: dread - f1 zero size 2024-03-30T04:27:57.575 INFO:tasks.workunit.client.0.smithi123.stdout:1/33: dread f8 [0,4194304] 0 2024-03-30T04:27:57.576 INFO:tasks.workunit.client.0.smithi123.stdout:8/83: truncate da/dc/f10 95893 0 2024-03-30T04:27:57.577 INFO:tasks.workunit.client.0.smithi123.stdout:4/102: getdents d2/d26 0 2024-03-30T04:27:57.578 INFO:tasks.workunit.client.0.smithi123.stdout:8/84: fsync da/dc/f10 0 2024-03-30T04:27:57.580 INFO:tasks.workunit.client.0.smithi123.stdout:1/34: creat db/fe x:0 0 0 2024-03-30T04:27:57.582 INFO:tasks.workunit.client.0.smithi123.stdout:9/26: getdents d6/d8 0 2024-03-30T04:27:57.585 INFO:tasks.workunit.client.0.smithi123.stdout:9/27: chown f2 0 1 2024-03-30T04:27:57.589 INFO:tasks.workunit.client.0.smithi123.stdout:0/39: dwrite f5 [0,4194304] 0 2024-03-30T04:27:57.590 INFO:tasks.workunit.client.0.smithi123.stdout:4/103: rename d2/d4/d7/l17 to d2/d26/l2a 0 2024-03-30T04:27:57.591 INFO:tasks.workunit.client.0.smithi123.stdout:1/35: getdents db/dd 0 2024-03-30T04:27:57.592 INFO:tasks.workunit.client.0.smithi123.stdout:9/28: creat d6/f9 x:0 0 0 2024-03-30T04:27:57.595 INFO:tasks.workunit.client.0.smithi123.stdout:8/85: getdents da 0 2024-03-30T04:27:57.596 INFO:tasks.workunit.client.0.smithi123.stdout:8/86: chown da/fe 133847473 1 2024-03-30T04:27:57.596 INFO:tasks.workunit.client.0.smithi123.stdout:0/40: creat d6/f9 x:0 0 0 2024-03-30T04:27:57.598 INFO:tasks.workunit.client.0.smithi123.stdout:4/104: creat d2/d4/f2b x:0 0 0 2024-03-30T04:27:57.598 INFO:tasks.workunit.client.0.smithi123.stdout:8/87: chown da/fb 156 1 2024-03-30T04:27:57.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:27:57 smithi135 ceph-mon[44364]: pgmap v32: 65 pgs: 65 active+clean; 282 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 2.0 MiB/s rd, 13 MiB/s wr, 303 op/s 2024-03-30T04:27:57.601 INFO:tasks.workunit.client.0.smithi123.stdout:8/88: write f8 [4051414,123788] 0 2024-03-30T04:27:57.602 INFO:tasks.workunit.client.0.smithi123.stdout:0/41: creat d6/fa x:0 0 0 2024-03-30T04:27:57.607 INFO:tasks.workunit.client.0.smithi123.stdout:1/36: dread f8 [0,4194304] 0 2024-03-30T04:27:57.607 INFO:tasks.workunit.client.0.smithi123.stdout:4/105: mkdir d2/d4/d2c 0 2024-03-30T04:27:57.607 INFO:tasks.workunit.client.0.smithi123.stdout:0/42: dread - d6/fa zero size 2024-03-30T04:27:57.609 INFO:tasks.workunit.client.0.smithi123.stdout:4/106: write d2/d4/f2b [346764,54712] 0 2024-03-30T04:27:57.614 INFO:tasks.workunit.client.0.smithi123.stdout:0/43: rename d6/c7 to d6/cb 0 2024-03-30T04:27:57.621 INFO:tasks.workunit.client.0.smithi123.stdout:0/44: mknod d6/cc 0 2024-03-30T04:27:57.625 INFO:tasks.workunit.client.0.smithi123.stdout:0/45: fsync f5 0 2024-03-30T04:27:57.627 INFO:tasks.workunit.client.0.smithi123.stdout:8/89: dwrite da/fb [0,4194304] 0 2024-03-30T04:27:57.630 INFO:tasks.workunit.client.0.smithi123.stdout:0/46: mkdir d6/dd 0 2024-03-30T04:27:57.635 INFO:tasks.workunit.client.0.smithi123.stdout:1/37: dwrite db/fe [0,4194304] 0 2024-03-30T04:27:57.637 INFO:tasks.workunit.client.0.smithi123.stdout:8/90: truncate da/dc/f10 848574 0 2024-03-30T04:27:57.639 INFO:tasks.workunit.client.0.smithi123.stdout:8/91: truncate f9 5175512 0 2024-03-30T04:27:57.643 INFO:tasks.workunit.client.0.smithi123.stdout:1/38: rename l9 to db/lf 0 2024-03-30T04:27:57.644 INFO:tasks.workunit.client.0.smithi123.stdout:8/92: dread - da/fe zero size 2024-03-30T04:27:57.648 INFO:tasks.workunit.client.0.smithi123.stdout:8/93: write f8 [1743279,107946] 0 2024-03-30T04:27:57.655 INFO:tasks.workunit.client.0.smithi123.stdout:8/94: creat da/dc/d14/f15 x:0 0 0 2024-03-30T04:27:57.659 INFO:tasks.workunit.client.0.smithi123.stdout:8/95: getdents da 0 2024-03-30T04:27:57.710 INFO:tasks.workunit.client.0.smithi123.stdout:6/72: write fb [4393217,5364] 0 2024-03-30T04:27:57.710 INFO:tasks.workunit.client.0.smithi123.stdout:7/16: fsync f1 0 2024-03-30T04:27:57.713 INFO:tasks.workunit.client.0.smithi123.stdout:6/73: write fb [4448896,124261] 0 2024-03-30T04:27:57.718 INFO:tasks.workunit.client.0.smithi123.stdout:2/56: getdents d0 0 2024-03-30T04:27:57.721 INFO:tasks.workunit.client.0.smithi123.stdout:7/17: link f1 d2/f8 0 2024-03-30T04:27:57.728 INFO:tasks.workunit.client.0.smithi123.stdout:2/57: mkdir d0/dc 0 2024-03-30T04:27:57.733 INFO:tasks.workunit.client.0.smithi123.stdout:6/74: dwrite fa [0,4194304] 0 2024-03-30T04:27:57.754 INFO:tasks.workunit.client.0.smithi123.stdout:7/18: write f1 [1354745,102926] 0 2024-03-30T04:27:57.762 INFO:tasks.workunit.client.0.smithi123.stdout:6/75: creat d10/f1b x:0 0 0 2024-03-30T04:27:57.767 INFO:tasks.workunit.client.0.smithi123.stdout:7/19: creat d2/f9 x:0 0 0 2024-03-30T04:27:57.769 INFO:tasks.workunit.client.0.smithi123.stdout:6/76: mkdir d10/d13/d1c 0 2024-03-30T04:27:57.771 INFO:tasks.workunit.client.0.smithi123.stdout:3/50: dread f7 [0,4194304] 0 2024-03-30T04:27:57.772 INFO:tasks.workunit.client.0.smithi123.stdout:2/58: rmdir d0/dc 0 2024-03-30T04:27:57.774 INFO:tasks.workunit.client.0.smithi123.stdout:7/20: mkdir d2/da 0 2024-03-30T04:27:57.775 INFO:tasks.workunit.client.0.smithi123.stdout:7/21: readlink d2/l5 0 2024-03-30T04:27:57.775 INFO:tasks.workunit.client.0.smithi123.stdout:6/77: creat d10/f1d x:0 0 0 2024-03-30T04:27:57.776 INFO:tasks.workunit.client.0.smithi123.stdout:7/22: chown d2/da 581 1 2024-03-30T04:27:57.784 INFO:tasks.workunit.client.0.smithi123.stdout:7/23: rename d2/f9 to d2/da/fb 0 2024-03-30T04:27:57.786 INFO:tasks.workunit.client.0.smithi123.stdout:2/59: dread d0/d8/fa [0,4194304] 0 2024-03-30T04:27:57.793 INFO:tasks.workunit.client.0.smithi123.stdout:7/24: dread f1 [0,4194304] 0 2024-03-30T04:27:57.794 INFO:tasks.workunit.client.0.smithi123.stdout:9/29: rmdir d6 39 2024-03-30T04:27:57.796 INFO:tasks.workunit.client.0.smithi123.stdout:7/25: mkdir d2/dc 0 2024-03-30T04:27:57.797 INFO:tasks.workunit.client.0.smithi123.stdout:6/78: dwrite d10/f19 [0,4194304] 0 2024-03-30T04:27:57.799 INFO:tasks.workunit.client.0.smithi123.stdout:6/79: stat c7 0 2024-03-30T04:27:57.800 INFO:tasks.workunit.client.0.smithi123.stdout:7/26: mknod d2/cd 0 2024-03-30T04:27:57.802 INFO:tasks.workunit.client.0.smithi123.stdout:9/30: rmdir d6 39 2024-03-30T04:27:57.811 INFO:tasks.workunit.client.0.smithi123.stdout:9/31: truncate d6/f9 108463 0 2024-03-30T04:27:57.811 INFO:tasks.workunit.client.0.smithi123.stdout:0/47: truncate f5 4054778 0 2024-03-30T04:27:57.817 INFO:tasks.workunit.client.0.smithi123.stdout:9/32: fsync f0 0 2024-03-30T04:27:57.822 INFO:tasks.workunit.client.0.smithi123.stdout:4/107: dwrite d2/d4/f10 [0,4194304] 0 2024-03-30T04:27:57.826 INFO:tasks.workunit.client.0.smithi123.stdout:0/48: creat d6/fe x:0 0 0 2024-03-30T04:27:57.826 INFO:tasks.workunit.client.0.smithi123.stdout:1/39: truncate db/fe 1286181 0 2024-03-30T04:27:57.828 INFO:tasks.workunit.client.0.smithi123.stdout:0/49: dread - d6/f9 zero size 2024-03-30T04:27:57.830 INFO:tasks.workunit.client.0.smithi123.stdout:0/50: write d6/f9 [1022781,114101] 0 2024-03-30T04:27:57.838 INFO:tasks.workunit.client.0.smithi123.stdout:4/108: creat d2/d4/f2d x:0 0 0 2024-03-30T04:27:57.841 INFO:tasks.workunit.client.0.smithi123.stdout:7/27: dwrite d2/da/fb [0,4194304] 0 2024-03-30T04:27:57.841 INFO:tasks.workunit.client.0.smithi123.stdout:1/40: dread f8 [0,4194304] 0 2024-03-30T04:27:57.845 INFO:tasks.workunit.client.0.smithi123.stdout:9/33: rmdir d6/d8 0 2024-03-30T04:27:57.848 INFO:tasks.workunit.client.0.smithi123.stdout:4/109: mkdir d2/d2e 0 2024-03-30T04:27:57.848 INFO:tasks.workunit.client.0.smithi123.stdout:9/34: write f2 [159848,52368] 0 2024-03-30T04:27:57.849 INFO:tasks.workunit.client.0.smithi123.stdout:8/96: dwrite f5 [0,4194304] 0 2024-03-30T04:27:57.851 INFO:tasks.workunit.client.0.smithi123.stdout:8/97: dread - da/ff zero size 2024-03-30T04:27:57.855 INFO:tasks.workunit.client.0.smithi123.stdout:1/41: creat db/dd/f10 x:0 0 0 2024-03-30T04:27:57.865 INFO:tasks.workunit.client.0.smithi123.stdout:9/35: symlink d6/la 0 2024-03-30T04:27:57.869 INFO:tasks.workunit.client.0.smithi123.stdout:7/28: mknod d2/dc/ce 0 2024-03-30T04:27:57.870 INFO:tasks.workunit.client.0.smithi123.stdout:8/98: creat da/dc/f16 x:0 0 0 2024-03-30T04:27:57.873 INFO:tasks.workunit.client.0.smithi123.stdout:8/99: write da/dc/f10 [241312,3172] 0 2024-03-30T04:27:57.873 INFO:tasks.workunit.client.0.smithi123.stdout:8/100: stat da/f11 0 2024-03-30T04:27:57.875 INFO:tasks.workunit.client.0.smithi123.stdout:4/110: dread d2/d4/d7/d1a/f1f [0,4194304] 0 2024-03-30T04:27:57.880 INFO:tasks.workunit.client.0.smithi123.stdout:9/36: creat d6/fb x:0 0 0 2024-03-30T04:27:57.881 INFO:tasks.workunit.client.0.smithi123.stdout:7/29: creat d2/dc/ff x:0 0 0 2024-03-30T04:27:57.888 INFO:tasks.workunit.client.0.smithi123.stdout:1/42: rename c7 to db/c11 0 2024-03-30T04:27:57.895 INFO:tasks.workunit.client.0.smithi123.stdout:8/101: symlink da/l17 0 2024-03-30T04:27:57.900 INFO:tasks.workunit.client.0.smithi123.stdout:4/111: dwrite d2/d4/f10 [0,4194304] 0 2024-03-30T04:27:57.904 INFO:tasks.workunit.client.0.smithi123.stdout:7/30: rename d2/dc/ce to d2/dc/c10 0 2024-03-30T04:27:57.905 INFO:tasks.workunit.client.0.smithi123.stdout:1/43: mkdir db/d12 0 2024-03-30T04:27:57.909 INFO:tasks.workunit.client.0.smithi123.stdout:8/102: mkdir da/dc/d14/d18 0 2024-03-30T04:27:57.911 INFO:tasks.workunit.client.0.smithi123.stdout:4/112: read d2/d4/d7/fb [2374038,74335] 0 2024-03-30T04:27:57.917 INFO:tasks.workunit.client.0.smithi123.stdout:9/37: dwrite f2 [0,4194304] 0 2024-03-30T04:27:57.919 INFO:tasks.workunit.client.0.smithi123.stdout:5/71: dwrite f9 [0,4194304] 0 2024-03-30T04:27:57.920 INFO:tasks.workunit.client.0.smithi123.stdout:1/44: creat db/dd/f13 x:0 0 0 2024-03-30T04:27:57.927 INFO:tasks.workunit.client.0.smithi123.stdout:5/72: chown dd/l13 260594 1 2024-03-30T04:27:57.928 INFO:tasks.workunit.client.0.smithi123.stdout:8/103: symlink da/dc/l19 0 2024-03-30T04:27:57.932 INFO:tasks.workunit.client.0.smithi123.stdout:6/80: truncate f9 1568130 0 2024-03-30T04:27:57.934 INFO:tasks.workunit.client.0.smithi123.stdout:3/51: dwrite f7 [0,4194304] 0 2024-03-30T04:27:57.951 INFO:tasks.workunit.client.0.smithi123.stdout:7/31: dwrite f1 [0,4194304] 0 2024-03-30T04:27:57.955 INFO:tasks.workunit.client.0.smithi123.stdout:7/32: chown d2/da/fb 1 1 2024-03-30T04:27:57.958 INFO:tasks.workunit.client.0.smithi123.stdout:7/33: dread - d2/dc/ff zero size 2024-03-30T04:27:57.960 INFO:tasks.workunit.client.0.smithi123.stdout:9/38: link f1 d6/fc 0 2024-03-30T04:27:57.960 INFO:tasks.workunit.client.0.smithi123.stdout:5/73: dwrite f2 [0,4194304] 0 2024-03-30T04:27:57.962 INFO:tasks.workunit.client.0.smithi123.stdout:5/74: readlink l4 0 2024-03-30T04:27:57.962 INFO:tasks.workunit.client.0.smithi123.stdout:9/39: chown f2 412561 1 2024-03-30T04:27:57.964 INFO:tasks.workunit.client.0.smithi123.stdout:5/75: write f2 [1383152,52884] 0 2024-03-30T04:27:57.966 INFO:tasks.workunit.client.0.smithi123.stdout:5/76: stat f7 0 2024-03-30T04:27:57.966 INFO:tasks.workunit.client.0.smithi123.stdout:3/52: chown l8 10366 1 2024-03-30T04:27:57.967 INFO:tasks.workunit.client.0.smithi123.stdout:8/104: creat da/dc/f1a x:0 0 0 2024-03-30T04:27:57.967 INFO:tasks.workunit.client.0.smithi123.stdout:3/53: fdatasync f7 0 2024-03-30T04:27:57.969 INFO:tasks.workunit.client.0.smithi123.stdout:5/77: chown f0 114 1 2024-03-30T04:27:57.970 INFO:tasks.workunit.client.0.smithi123.stdout:5/78: readlink lc 0 2024-03-30T04:27:57.974 INFO:tasks.workunit.client.0.smithi123.stdout:8/105: dread f9 [4194304,4194304] 0 2024-03-30T04:27:57.976 INFO:tasks.workunit.client.0.smithi123.stdout:8/106: truncate da/fe 883541 0 2024-03-30T04:27:57.977 INFO:tasks.workunit.client.0.smithi123.stdout:7/34: mkdir d2/da/d11 0 2024-03-30T04:27:57.991 INFO:tasks.workunit.client.0.smithi123.stdout:3/54: mknod cb 0 2024-03-30T04:27:57.994 INFO:tasks.workunit.client.0.smithi123.stdout:5/79: symlink dd/l14 0 2024-03-30T04:27:57.995 INFO:tasks.workunit.client.0.smithi123.stdout:7/35: mknod d2/c12 0 2024-03-30T04:27:57.996 INFO:tasks.workunit.client.0.smithi123.stdout:3/55: write f7 [5390899,80350] 0 2024-03-30T04:27:57.998 INFO:tasks.workunit.client.0.smithi123.stdout:1/45: link l6 db/d12/l14 0 2024-03-30T04:27:58.006 INFO:tasks.workunit.client.0.smithi123.stdout:9/40: dwrite d6/fc [0,4194304] 0 2024-03-30T04:27:58.007 INFO:tasks.workunit.client.0.smithi123.stdout:3/56: symlink lc 0 2024-03-30T04:27:58.018 INFO:tasks.workunit.client.0.smithi123.stdout:7/36: dwrite d2/f8 [0,4194304] 0 2024-03-30T04:27:58.019 INFO:tasks.workunit.client.0.smithi123.stdout:3/57: creat fd x:0 0 0 2024-03-30T04:27:58.021 INFO:tasks.workunit.client.0.smithi123.stdout:7/37: write f1 [1494885,63277] 0 2024-03-30T04:27:58.022 INFO:tasks.workunit.client.0.smithi123.stdout:1/46: rename db/c11 to db/d12/c15 0 2024-03-30T04:27:58.022 INFO:tasks.workunit.client.0.smithi123.stdout:7/38: write f0 [2467622,7120] 0 2024-03-30T04:27:58.029 INFO:tasks.workunit.client.0.smithi123.stdout:9/41: link d6/f7 d6/fd 0 2024-03-30T04:27:58.030 INFO:tasks.workunit.client.0.smithi123.stdout:3/58: rename f7 to fe 0 2024-03-30T04:27:58.042 INFO:tasks.workunit.client.0.smithi123.stdout:9/42: creat d6/fe x:0 0 0 2024-03-30T04:27:58.051 INFO:tasks.workunit.client.0.smithi123.stdout:4/113: truncate d2/d4/fa 328545 0 2024-03-30T04:27:58.052 INFO:tasks.workunit.client.0.smithi123.stdout:7/39: dwrite d2/dc/ff [0,4194304] 0 2024-03-30T04:27:58.065 INFO:tasks.workunit.client.0.smithi123.stdout:9/43: mknod d6/cf 0 2024-03-30T04:27:58.074 INFO:tasks.workunit.client.0.smithi123.stdout:7/40: creat d2/f13 x:0 0 0 2024-03-30T04:27:58.083 INFO:tasks.workunit.client.0.smithi123.stdout:4/114: dwrite d2/d4/f10 [4194304,4194304] 0 2024-03-30T04:27:58.086 INFO:tasks.workunit.client.0.smithi123.stdout:7/41: rename f1 to d2/da/d11/f14 0 2024-03-30T04:27:58.088 INFO:tasks.workunit.client.0.smithi123.stdout:7/42: rename d2/da/d11 to d2/da/d11/d15 22 2024-03-30T04:27:58.098 INFO:tasks.workunit.client.0.smithi123.stdout:7/43: symlink d2/da/l16 0 2024-03-30T04:27:58.106 INFO:tasks.workunit.client.0.smithi123.stdout:5/80: truncate f0 1187046 0 2024-03-30T04:27:58.106 INFO:tasks.workunit.client.0.smithi123.stdout:5/81: chown f2 19273432 1 2024-03-30T04:27:58.108 INFO:tasks.workunit.client.0.smithi123.stdout:5/82: read f2 [3407304,31474] 0 2024-03-30T04:27:58.113 INFO:tasks.workunit.client.0.smithi123.stdout:5/83: creat dd/f15 x:0 0 0 2024-03-30T04:27:58.125 INFO:tasks.workunit.client.0.smithi123.stdout:5/84: dread f7 [0,4194304] 0 2024-03-30T04:27:58.126 INFO:tasks.workunit.client.0.smithi123.stdout:5/85: write f9 [314173,98011] 0 2024-03-30T04:27:58.128 INFO:tasks.workunit.client.0.smithi123.stdout:5/86: fdatasync f7 0 2024-03-30T04:27:58.130 INFO:tasks.workunit.client.0.smithi123.stdout:3/59: truncate fe 7509301 0 2024-03-30T04:27:58.131 INFO:tasks.workunit.client.0.smithi123.stdout:5/87: write fb [998738,11389] 0 2024-03-30T04:27:58.132 INFO:tasks.workunit.client.0.smithi123.stdout:6/81: link f9 d10/d13/d1c/f1e 0 2024-03-30T04:27:58.135 INFO:tasks.workunit.client.0.smithi123.stdout:3/60: creat ff x:0 0 0 2024-03-30T04:27:58.137 INFO:tasks.workunit.client.0.smithi123.stdout:1/47: dwrite f8 [0,4194304] 0 2024-03-30T04:27:58.143 INFO:tasks.workunit.client.0.smithi123.stdout:5/88: creat dd/f16 x:0 0 0 2024-03-30T04:27:58.144 INFO:tasks.workunit.client.0.smithi123.stdout:9/44: rmdir d6 39 2024-03-30T04:27:58.149 INFO:tasks.workunit.client.0.smithi123.stdout:3/61: mkdir d10 0 2024-03-30T04:27:58.150 INFO:tasks.workunit.client.0.smithi123.stdout:3/62: dread - ff zero size 2024-03-30T04:27:58.150 INFO:tasks.workunit.client.0.smithi123.stdout:3/63: chown cb 10374 1 2024-03-30T04:27:58.152 INFO:tasks.workunit.client.0.smithi123.stdout:1/48: unlink db/dd/f10 0 2024-03-30T04:27:58.155 INFO:tasks.workunit.client.0.smithi123.stdout:1/49: write db/dd/f13 [571266,8603] 0 2024-03-30T04:27:58.160 INFO:tasks.workunit.client.0.smithi123.stdout:6/82: link f6 d10/d13/d1c/f1f 0 2024-03-30T04:27:58.171 INFO:tasks.workunit.client.0.smithi123.stdout:9/45: creat d6/f10 x:0 0 0 2024-03-30T04:27:58.172 INFO:tasks.workunit.client.0.smithi123.stdout:9/46: dread - d6/fd zero size 2024-03-30T04:27:58.173 INFO:tasks.workunit.client.0.smithi123.stdout:9/47: readlink d6/la 0 2024-03-30T04:27:58.174 INFO:tasks.workunit.client.0.smithi123.stdout:6/83: unlink cf 0 2024-03-30T04:27:58.175 INFO:tasks.workunit.client.0.smithi123.stdout:4/115: truncate d2/d4/d7/d1a/f1f 3153529 0 2024-03-30T04:27:58.185 INFO:tasks.workunit.client.0.smithi123.stdout:4/116: mknod d2/d4/c2f 0 2024-03-30T04:27:58.194 INFO:tasks.workunit.client.0.smithi123.stdout:9/48: dwrite d6/fe [0,4194304] 0 2024-03-30T04:27:58.219 INFO:tasks.workunit.client.0.smithi123.stdout:3/64: dwrite fe [0,4194304] 0 2024-03-30T04:27:58.229 INFO:tasks.workunit.client.0.smithi123.stdout:6/84: truncate d10/d13/d1c/f1e 305903 0 2024-03-30T04:27:58.243 INFO:tasks.workunit.client.0.smithi123.stdout:3/65: rmdir d10 0 2024-03-30T04:27:58.244 INFO:tasks.workunit.client.0.smithi123.stdout:3/66: fsync fe 0 2024-03-30T04:27:58.252 INFO:tasks.workunit.client.0.smithi123.stdout:3/67: link fe f11 0 2024-03-30T04:27:58.253 INFO:tasks.workunit.client.0.smithi123.stdout:9/49: getdents d6 0 2024-03-30T04:27:58.256 INFO:tasks.workunit.client.0.smithi123.stdout:4/117: rmdir d2/d4/d7 39 2024-03-30T04:27:58.260 INFO:tasks.workunit.client.0.smithi123.stdout:1/50: truncate db/fe 438575 0 2024-03-30T04:27:58.265 INFO:tasks.workunit.client.0.smithi123.stdout:1/51: mknod db/c16 0 2024-03-30T04:27:58.269 INFO:tasks.workunit.client.0.smithi123.stdout:4/118: rmdir d2/d4 39 2024-03-30T04:27:58.272 INFO:tasks.workunit.client.0.smithi123.stdout:5/89: write f0 [1461628,89691] 0 2024-03-30T04:27:58.272 INFO:tasks.workunit.client.0.smithi123.stdout:9/50: dwrite d6/f10 [0,4194304] 0 2024-03-30T04:27:58.274 INFO:tasks.workunit.client.0.smithi123.stdout:9/51: dread - d6/f7 zero size 2024-03-30T04:27:58.275 INFO:tasks.workunit.client.0.smithi123.stdout:5/90: read dd/fe [3151430,57583] 0 2024-03-30T04:27:58.276 INFO:tasks.workunit.client.0.smithi123.stdout:3/68: dwrite f11 [0,4194304] 0 2024-03-30T04:27:58.278 INFO:tasks.workunit.client.0.smithi123.stdout:3/69: chown fe 1286 1 2024-03-30T04:27:58.280 INFO:tasks.workunit.client.0.smithi123.stdout:4/119: fdatasync d2/d4/f2b 0 2024-03-30T04:27:58.282 INFO:tasks.workunit.client.0.smithi123.stdout:9/52: rmdir d6 39 2024-03-30T04:27:58.289 INFO:tasks.workunit.client.0.smithi123.stdout:3/70: creat f12 x:0 0 0 2024-03-30T04:27:58.290 INFO:tasks.workunit.client.0.smithi123.stdout:3/71: dread - f12 zero size 2024-03-30T04:27:58.292 INFO:tasks.workunit.client.0.smithi123.stdout:5/91: dread f9 [0,4194304] 0 2024-03-30T04:27:58.296 INFO:tasks.workunit.client.0.smithi123.stdout:9/53: dread - d6/f7 zero size 2024-03-30T04:27:58.303 INFO:tasks.workunit.client.0.smithi123.stdout:5/92: mknod dd/c17 0 2024-03-30T04:27:58.304 INFO:tasks.workunit.client.0.smithi123.stdout:9/54: symlink d6/l11 0 2024-03-30T04:27:58.309 INFO:tasks.workunit.client.0.smithi123.stdout:5/93: mknod dd/c18 0 2024-03-30T04:27:58.316 INFO:tasks.workunit.client.0.smithi123.stdout:5/94: mkdir dd/d19 0 2024-03-30T04:27:58.319 INFO:tasks.workunit.client.0.smithi123.stdout:6/85: truncate d10/d13/d1c/f1f 3741900 0 2024-03-30T04:27:58.324 INFO:tasks.workunit.client.0.smithi123.stdout:6/86: mkdir d10/d20 0 2024-03-30T04:27:58.327 INFO:tasks.workunit.client.0.smithi123.stdout:6/87: write d10/f1d [1008218,61860] 0 2024-03-30T04:27:58.339 INFO:tasks.workunit.client.0.smithi123.stdout:5/95: dwrite dd/f16 [0,4194304] 0 2024-03-30T04:27:58.341 INFO:tasks.workunit.client.0.smithi123.stdout:6/88: creat d10/d13/d1c/f21 x:0 0 0 2024-03-30T04:27:58.343 INFO:tasks.workunit.client.0.smithi123.stdout:1/52: read db/fe [219498,84230] 0 2024-03-30T04:27:58.344 INFO:tasks.workunit.client.0.smithi123.stdout:6/89: read - fe zero size 2024-03-30T04:27:58.350 INFO:tasks.workunit.client.0.smithi123.stdout:4/120: fsync d2/d4/d7/d1a/f1f 0 2024-03-30T04:27:58.355 INFO:tasks.workunit.client.0.smithi123.stdout:5/96: dread f0 [0,4194304] 0 2024-03-30T04:27:58.359 INFO:tasks.workunit.client.0.smithi123.stdout:6/90: rename d10/d13/d1c/f21 to d10/d20/f22 0 2024-03-30T04:27:58.363 INFO:tasks.workunit.client.0.smithi123.stdout:1/53: unlink l6 0 2024-03-30T04:27:58.365 INFO:tasks.workunit.client.0.smithi123.stdout:3/72: getdents . 0 2024-03-30T04:27:58.368 INFO:tasks.workunit.client.0.smithi123.stdout:5/97: mknod dd/c1a 0 2024-03-30T04:27:58.375 INFO:tasks.workunit.client.0.smithi123.stdout:4/121: link d2/d4/f10 d2/d4/d7/d1a/f30 0 2024-03-30T04:27:58.377 INFO:tasks.workunit.client.0.smithi123.stdout:1/54: dread f8 [0,4194304] 0 2024-03-30T04:27:58.377 INFO:tasks.workunit.client.0.smithi123.stdout:4/122: chown d2/d4/l5 51800546 1 2024-03-30T04:27:58.382 INFO:tasks.workunit.client.0.smithi123.stdout:5/98: symlink dd/l1b 0 2024-03-30T04:27:58.384 INFO:tasks.workunit.client.0.smithi123.stdout:5/99: chown dd/c17 4 1 2024-03-30T04:27:58.388 INFO:tasks.workunit.client.0.smithi123.stdout:5/100: write dd/f16 [3071413,79547] 0 2024-03-30T04:27:58.393 INFO:tasks.workunit.client.0.smithi123.stdout:4/123: unlink d2/d4/f10 0 2024-03-30T04:27:58.399 INFO:tasks.workunit.client.0.smithi123.stdout:9/55: dwrite f2 [4194304,4194304] 0 2024-03-30T04:27:58.411 INFO:tasks.workunit.client.0.smithi123.stdout:1/55: rename ca to db/d12/c17 0 2024-03-30T04:27:58.415 INFO:tasks.workunit.client.0.smithi123.stdout:5/101: creat dd/f1c x:0 0 0 2024-03-30T04:27:58.415 INFO:tasks.workunit.client.0.smithi123.stdout:1/56: dread db/fe [0,4194304] 0 2024-03-30T04:27:58.418 INFO:tasks.workunit.client.0.smithi123.stdout:9/56: rmdir d6 39 2024-03-30T04:27:58.422 INFO:tasks.workunit.client.0.smithi123.stdout:6/91: getdents d10 0 2024-03-30T04:27:58.423 INFO:tasks.workunit.client.0.smithi123.stdout:5/102: creat dd/f1d x:0 0 0 2024-03-30T04:27:58.423 INFO:tasks.workunit.client.0.smithi123.stdout:6/92: dread - fe zero size 2024-03-30T04:27:58.423 INFO:tasks.workunit.client.0.smithi123.stdout:5/103: read - dd/f1c zero size 2024-03-30T04:27:58.425 INFO:tasks.workunit.client.0.smithi123.stdout:1/57: symlink db/l18 0 2024-03-30T04:27:58.433 INFO:tasks.workunit.client.0.smithi123.stdout:5/104: mknod dd/d19/c1e 0 2024-03-30T04:27:58.434 INFO:tasks.workunit.client.0.smithi123.stdout:5/105: chown f2 93 1 2024-03-30T04:27:58.438 INFO:tasks.workunit.client.0.smithi123.stdout:5/106: mkdir dd/d19/d1f 0 2024-03-30T04:27:58.450 INFO:tasks.workunit.client.0.smithi123.stdout:6/93: dwrite d10/d20/f22 [0,4194304] 0 2024-03-30T04:27:58.451 INFO:tasks.workunit.client.0.smithi123.stdout:6/94: readlink d10/l18 0 2024-03-30T04:27:58.455 INFO:tasks.workunit.client.0.smithi123.stdout:3/73: truncate f11 6325717 0 2024-03-30T04:27:58.456 INFO:tasks.workunit.client.0.smithi123.stdout:6/95: dread f2 [0,4194304] 0 2024-03-30T04:27:58.460 INFO:tasks.workunit.client.0.smithi123.stdout:3/74: mkdir d13 0 2024-03-30T04:27:58.461 INFO:tasks.workunit.client.0.smithi123.stdout:3/75: dread - f12 zero size 2024-03-30T04:27:58.462 INFO:tasks.workunit.client.0.smithi123.stdout:3/76: read - fd zero size 2024-03-30T04:27:58.463 INFO:tasks.workunit.client.0.smithi123.stdout:3/77: chown ff 20 1 2024-03-30T04:27:58.463 INFO:tasks.workunit.client.0.smithi123.stdout:6/96: unlink d10/d13/c1a 0 2024-03-30T04:27:58.464 INFO:tasks.workunit.client.0.smithi123.stdout:3/78: chown l4 178985042 1 2024-03-30T04:27:58.465 INFO:tasks.workunit.client.0.smithi123.stdout:6/97: stat f6 0 2024-03-30T04:27:58.468 INFO:tasks.workunit.client.0.smithi123.stdout:6/98: unlink fa 0 2024-03-30T04:27:58.471 INFO:tasks.workunit.client.0.smithi123.stdout:1/58: rename db/d12 to db/dd/d19 0 2024-03-30T04:27:58.471 INFO:tasks.workunit.client.0.smithi123.stdout:9/57: rename d6 to d6/d12 22 2024-03-30T04:27:58.476 INFO:tasks.workunit.client.0.smithi123.stdout:9/58: write f2 [8641393,60338] 0 2024-03-30T04:27:58.479 INFO:tasks.workunit.client.0.smithi123.stdout:1/59: creat db/dd/d19/f1a x:0 0 0 2024-03-30T04:27:58.483 INFO:tasks.workunit.client.0.smithi123.stdout:9/59: symlink d6/l13 0 2024-03-30T04:27:58.486 INFO:tasks.workunit.client.0.smithi123.stdout:6/99: link d10/l14 d10/d13/l23 0 2024-03-30T04:27:58.488 INFO:tasks.workunit.client.0.smithi123.stdout:5/107: truncate f0 213007 0 2024-03-30T04:27:58.492 INFO:tasks.workunit.client.0.smithi123.stdout:5/108: truncate dd/f1d 527957 0 2024-03-30T04:27:58.494 INFO:tasks.workunit.client.0.smithi123.stdout:4/124: dwrite d2/d4/fa [0,4194304] 0 2024-03-30T04:27:58.501 INFO:tasks.workunit.client.0.smithi123.stdout:4/125: readlink d2/d4/l12 0 2024-03-30T04:27:58.509 INFO:tasks.workunit.client.0.smithi123.stdout:4/126: write d2/d4/d7/fc [827257,36237] 0 2024-03-30T04:27:58.510 INFO:tasks.workunit.client.0.smithi123.stdout:1/60: rename db/dd/d19 to db/d1b 0 2024-03-30T04:27:58.514 INFO:tasks.workunit.client.0.smithi123.stdout:1/61: write f8 [244202,86246] 0 2024-03-30T04:27:58.520 INFO:tasks.workunit.client.0.smithi123.stdout:3/79: write fe [1348981,120569] 0 2024-03-30T04:27:58.526 INFO:tasks.workunit.client.0.smithi123.stdout:3/80: write f12 [182608,26544] 0 2024-03-30T04:27:58.532 INFO:tasks.workunit.client.0.smithi123.stdout:3/81: read f12 [57742,68405] 0 2024-03-30T04:27:58.541 INFO:tasks.workunit.client.0.smithi123.stdout:3/82: creat d13/f14 x:0 0 0 2024-03-30T04:27:58.550 INFO:tasks.workunit.client.0.smithi123.stdout:6/100: dwrite f2 [0,4194304] 0 2024-03-30T04:27:58.551 INFO:tasks.workunit.client.0.smithi123.stdout:4/127: dwrite d2/d4/d7/fe [0,4194304] 0 2024-03-30T04:27:58.555 INFO:tasks.workunit.client.0.smithi123.stdout:1/62: dwrite db/dd/f13 [0,4194304] 0 2024-03-30T04:27:58.558 INFO:tasks.workunit.client.0.smithi123.stdout:6/101: rmdir d10 39 2024-03-30T04:27:58.562 INFO:tasks.workunit.client.0.smithi123.stdout:1/63: mkdir db/d1b/d1c 0 2024-03-30T04:27:58.566 INFO:tasks.workunit.client.0.smithi123.stdout:1/64: write f8 [3289315,45793] 0 2024-03-30T04:27:58.570 INFO:tasks.workunit.client.0.smithi123.stdout:4/128: symlink d2/d4/d7/l31 0 2024-03-30T04:27:58.576 INFO:tasks.workunit.client.0.smithi123.stdout:3/83: dwrite fe [0,4194304] 0 2024-03-30T04:27:58.576 INFO:tasks.workunit.client.0.smithi123.stdout:4/129: write d2/d4/f2d [936692,125944] 0 2024-03-30T04:27:58.578 INFO:tasks.workunit.client.0.smithi123.stdout:1/65: link f8 db/d1b/f1d 0 2024-03-30T04:27:58.579 INFO:tasks.workunit.client.0.smithi123.stdout:6/102: rename d10/l14 to d10/d20/l24 0 2024-03-30T04:27:58.581 INFO:tasks.workunit.client.0.smithi123.stdout:6/103: chown d10/d13/c16 843833470 1 2024-03-30T04:27:58.584 INFO:tasks.workunit.client.0.smithi123.stdout:6/104: creat d10/d13/f25 x:0 0 0 2024-03-30T04:27:58.586 INFO:tasks.workunit.client.0.smithi123.stdout:4/130: mknod d2/d4/d7/d1a/c32 0 2024-03-30T04:27:58.588 INFO:tasks.workunit.client.0.smithi123.stdout:6/105: creat d10/f26 x:0 0 0 2024-03-30T04:27:58.593 INFO:tasks.workunit.client.0.smithi123.stdout:6/106: unlink d10/d13/c16 0 2024-03-30T04:27:58.595 INFO:tasks.workunit.client.0.smithi123.stdout:6/107: dread - d10/d13/f25 zero size 2024-03-30T04:27:58.597 INFO:tasks.workunit.client.0.smithi123.stdout:1/66: dread f8 [0,4194304] 0 2024-03-30T04:27:58.597 INFO:tasks.workunit.client.0.smithi123.stdout:6/108: write d10/f26 [119791,11409] 0 2024-03-30T04:27:58.599 INFO:tasks.workunit.client.0.smithi123.stdout:6/109: read - d10/d13/f25 zero size 2024-03-30T04:27:58.601 INFO:tasks.workunit.client.0.smithi123.stdout:1/67: truncate db/fe 649667 0 2024-03-30T04:27:58.604 INFO:tasks.workunit.client.0.smithi123.stdout:6/110: rename d10/f19 to d10/f27 0 2024-03-30T04:27:58.605 INFO:tasks.workunit.client.0.smithi123.stdout:6/111: stat d10/f26 0 2024-03-30T04:27:58.609 INFO:tasks.workunit.client.0.smithi123.stdout:6/112: rmdir d10/d13/d1c 39 2024-03-30T04:27:58.611 INFO:tasks.workunit.client.0.smithi123.stdout:1/68: chown db/fe 67037739 1 2024-03-30T04:27:58.615 INFO:tasks.workunit.client.0.smithi123.stdout:6/113: read fb [281573,77380] 0 2024-03-30T04:27:58.616 INFO:tasks.workunit.client.0.smithi123.stdout:4/131: truncate d2/d4/f8 2734176 0 2024-03-30T04:27:58.617 INFO:tasks.workunit.client.0.smithi123.stdout:1/69: write db/dd/f13 [5057239,40380] 0 2024-03-30T04:27:58.623 INFO:tasks.workunit.client.0.smithi123.stdout:1/70: creat db/d1b/f1e x:0 0 0 2024-03-30T04:27:58.626 INFO:tasks.workunit.client.0.smithi123.stdout:6/114: write f9 [846929,74969] 0 2024-03-30T04:27:58.628 INFO:tasks.workunit.client.0.smithi123.stdout:6/115: dread - d10/d13/f25 zero size 2024-03-30T04:27:58.630 INFO:tasks.workunit.client.0.smithi123.stdout:1/71: getdents db/d1b 0 2024-03-30T04:27:58.634 INFO:tasks.workunit.client.0.smithi123.stdout:1/72: symlink db/l1f 0 2024-03-30T04:27:58.634 INFO:tasks.workunit.client.0.smithi123.stdout:6/116: mknod d10/d13/c28 0 2024-03-30T04:27:58.637 INFO:tasks.workunit.client.0.smithi123.stdout:1/73: creat db/f20 x:0 0 0 2024-03-30T04:27:58.637 INFO:tasks.workunit.client.0.smithi123.stdout:6/117: unlink fb 0 2024-03-30T04:27:58.639 INFO:tasks.workunit.client.0.smithi123.stdout:1/74: creat db/dd/f21 x:0 0 0 2024-03-30T04:27:58.640 INFO:tasks.workunit.client.0.smithi123.stdout:6/118: creat d10/f29 x:0 0 0 2024-03-30T04:27:58.643 INFO:tasks.workunit.client.0.smithi123.stdout:1/75: fsync db/fe 0 2024-03-30T04:27:58.644 INFO:tasks.workunit.client.0.smithi123.stdout:1/76: dread - db/d1b/f1e zero size 2024-03-30T04:27:58.646 INFO:tasks.workunit.client.0.smithi123.stdout:6/119: creat d10/d13/d1c/f2a x:0 0 0 2024-03-30T04:27:58.649 INFO:tasks.workunit.client.0.smithi123.stdout:6/120: creat d10/d20/f2b x:0 0 0 2024-03-30T04:27:58.655 INFO:tasks.workunit.client.0.smithi123.stdout:1/77: link f8 db/f22 0 2024-03-30T04:27:58.656 INFO:tasks.workunit.client.0.smithi123.stdout:1/78: chown db/l1f 178037651 1 2024-03-30T04:27:58.658 INFO:tasks.workunit.client.0.smithi123.stdout:1/79: chown l4 8 1 2024-03-30T04:27:58.676 INFO:tasks.workunit.client.0.smithi123.stdout:6/121: dwrite f9 [0,4194304] 0 2024-03-30T04:27:58.680 INFO:tasks.workunit.client.0.smithi123.stdout:1/80: dwrite f8 [0,4194304] 0 2024-03-30T04:27:58.686 INFO:tasks.workunit.client.0.smithi123.stdout:1/81: rmdir db 39 2024-03-30T04:27:58.702 INFO:tasks.workunit.client.0.smithi123.stdout:6/122: dwrite d10/d13/d1c/f2a [0,4194304] 0 2024-03-30T04:27:58.711 INFO:tasks.workunit.client.0.smithi123.stdout:6/123: creat d10/d13/d1c/f2c x:0 0 0 2024-03-30T04:27:58.714 INFO:tasks.workunit.client.0.smithi123.stdout:6/124: fdatasync d10/d13/f17 0 2024-03-30T04:27:58.716 INFO:tasks.workunit.client.0.smithi123.stdout:6/125: fsync d10/d13/f17 0 2024-03-30T04:27:58.716 INFO:tasks.workunit.client.0.smithi123.stdout:1/82: dwrite db/dd/f21 [0,4194304] 0 2024-03-30T04:27:58.717 INFO:tasks.workunit.client.0.smithi123.stdout:1/83: rename db to db/d23 22 2024-03-30T04:27:58.719 INFO:tasks.workunit.client.0.smithi123.stdout:1/84: write db/f20 [400060,86673] 0 2024-03-30T04:27:58.720 INFO:tasks.workunit.client.0.smithi123.stdout:6/126: symlink d10/d13/d1c/l2d 0 2024-03-30T04:27:58.722 INFO:tasks.workunit.client.0.smithi123.stdout:1/85: write db/d1b/f1e [997055,60406] 0 2024-03-30T04:27:58.724 INFO:tasks.workunit.client.0.smithi123.stdout:6/127: chown d10/d13/d1c/l2d 30 1 2024-03-30T04:27:58.727 INFO:tasks.workunit.client.0.smithi123.stdout:6/128: chown d10/d20/l24 592 1 2024-03-30T04:27:58.729 INFO:tasks.workunit.client.0.smithi123.stdout:6/129: chown d10/d13/d1c/f1f 153650 1 2024-03-30T04:27:58.731 INFO:tasks.workunit.client.0.smithi123.stdout:1/86: link db/dd/f13 db/f24 0 2024-03-30T04:27:58.735 INFO:tasks.workunit.client.0.smithi123.stdout:1/87: unlink db/f22 0 2024-03-30T04:27:58.743 INFO:tasks.workunit.client.0.smithi123.stdout:6/130: dread f1 [0,4194304] 0 2024-03-30T04:27:58.748 INFO:tasks.workunit.client.0.smithi123.stdout:6/131: creat d10/d13/f2e x:0 0 0 2024-03-30T04:27:58.750 INFO:tasks.workunit.client.0.smithi123.stdout:6/132: dread - d10/f1b zero size 2024-03-30T04:27:58.759 INFO:tasks.workunit.client.0.smithi123.stdout:1/88: dwrite db/d1b/f1e [0,4194304] 0 2024-03-30T04:27:58.764 INFO:tasks.workunit.client.0.smithi123.stdout:1/89: symlink db/dd/l25 0 2024-03-30T04:27:58.771 INFO:tasks.workunit.client.0.smithi123.stdout:6/133: dwrite d10/d13/d1c/f1e [0,4194304] 0 2024-03-30T04:27:58.775 INFO:tasks.workunit.client.0.smithi123.stdout:6/134: mknod d10/d13/d1c/c2f 0 2024-03-30T04:27:58.778 INFO:tasks.workunit.client.0.smithi123.stdout:6/135: mknod d10/d20/c30 0 2024-03-30T04:27:58.785 INFO:tasks.workunit.client.0.smithi123.stdout:6/136: rmdir d10/d13/d1c 39 2024-03-30T04:27:58.789 INFO:tasks.workunit.client.0.smithi123.stdout:6/137: symlink d10/d13/d1c/l31 0 2024-03-30T04:27:58.793 INFO:tasks.workunit.client.0.smithi123.stdout:6/138: link d10/d13/l23 d10/d20/l32 0 2024-03-30T04:27:58.793 INFO:tasks.workunit.client.0.smithi123.stdout:6/139: chown d10 237089694 1 2024-03-30T04:27:58.795 INFO:tasks.workunit.client.0.smithi123.stdout:6/140: write d10/d20/f22 [4323596,26657] 0 2024-03-30T04:27:58.798 INFO:tasks.workunit.client.0.smithi123.stdout:6/141: symlink d10/d20/l33 0 2024-03-30T04:27:58.801 INFO:tasks.workunit.client.0.smithi123.stdout:6/142: chown d10/d13/l23 652 1 2024-03-30T04:27:58.802 INFO:tasks.workunit.client.0.smithi123.stdout:6/143: dread - d10/d13/f25 zero size 2024-03-30T04:27:58.804 INFO:tasks.workunit.client.0.smithi123.stdout:6/144: unlink d10/d13/f2e 0 2024-03-30T04:27:58.805 INFO:tasks.workunit.client.0.smithi123.stdout:6/145: fsync f9 0 2024-03-30T04:27:58.814 INFO:tasks.workunit.client.0.smithi123.stdout:6/146: dread d10/d13/d1c/f1f [0,4194304] 0 2024-03-30T04:27:58.816 INFO:tasks.workunit.client.0.smithi123.stdout:6/147: unlink f1 0 2024-03-30T04:27:58.821 INFO:tasks.workunit.client.0.smithi123.stdout:2/60: sync 2024-03-30T04:27:58.825 INFO:tasks.workunit.client.0.smithi123.stdout:2/61: unlink d0/f5 0 2024-03-30T04:27:58.835 INFO:tasks.workunit.client.0.smithi123.stdout:6/148: truncate f2 3314154 0 2024-03-30T04:27:58.840 INFO:tasks.workunit.client.0.smithi123.stdout:6/149: write d10/f27 [2759565,70731] 0 2024-03-30T04:27:58.852 INFO:tasks.workunit.client.0.smithi123.stdout:6/150: symlink d10/d13/l34 0 2024-03-30T04:27:58.862 INFO:tasks.workunit.client.0.smithi123.stdout:6/151: dread d10/f11 [0,4194304] 0 2024-03-30T04:27:58.865 INFO:tasks.workunit.client.0.smithi123.stdout:6/152: mknod d10/d20/c35 0 2024-03-30T04:27:58.866 INFO:tasks.workunit.client.0.smithi123.stdout:6/153: chown d10/f29 126877845 1 2024-03-30T04:27:58.867 INFO:tasks.workunit.client.0.smithi123.stdout:6/154: readlink d10/d20/l33 0 2024-03-30T04:27:58.881 INFO:tasks.workunit.client.0.smithi123.stdout:6/155: dwrite d10/d13/d1c/f2a [0,4194304] 0 2024-03-30T04:27:58.883 INFO:tasks.workunit.client.0.smithi123.stdout:6/156: write d10/d13/f17 [169960,47420] 0 2024-03-30T04:27:58.886 INFO:tasks.workunit.client.0.smithi123.stdout:6/157: chown d10/d13/d1c/f2c 3117 1 2024-03-30T04:27:58.890 INFO:tasks.workunit.client.0.smithi123.stdout:6/158: chown d10/d13/f17 14854777 1 2024-03-30T04:27:58.895 INFO:tasks.workunit.client.0.smithi123.stdout:6/159: link d10/f26 d10/d20/f36 0 2024-03-30T04:27:58.898 INFO:tasks.workunit.client.0.smithi123.stdout:6/160: mknod d10/d13/d1c/c37 0 2024-03-30T04:27:58.925 INFO:tasks.workunit.client.0.smithi123.stdout:6/161: dwrite d10/d13/d1c/f1f [0,4194304] 0 2024-03-30T04:27:58.928 INFO:tasks.workunit.client.0.smithi123.stdout:6/162: creat d10/f38 x:0 0 0 2024-03-30T04:27:58.930 INFO:tasks.workunit.client.0.smithi123.stdout:6/163: read f9 [3231232,54773] 0 2024-03-30T04:27:58.934 INFO:tasks.workunit.client.0.smithi123.stdout:6/164: creat d10/d13/f39 x:0 0 0 2024-03-30T04:27:58.935 INFO:tasks.workunit.client.0.smithi123.stdout:6/165: chown d10/d13/d1c/l31 0 1 2024-03-30T04:27:58.937 INFO:tasks.workunit.client.0.smithi123.stdout:6/166: write d10/d20/f22 [905997,115557] 0 2024-03-30T04:27:58.941 INFO:tasks.workunit.client.0.smithi123.stdout:6/167: link f9 d10/f3a 0 2024-03-30T04:27:58.943 INFO:tasks.workunit.client.0.smithi123.stdout:6/168: rename f9 to d10/d20/f3b 0 2024-03-30T04:27:58.944 INFO:tasks.workunit.client.0.smithi123.stdout:6/169: dread - d10/f29 zero size 2024-03-30T04:27:58.946 INFO:tasks.workunit.client.0.smithi123.stdout:6/170: mknod d10/d20/c3c 0 2024-03-30T04:27:58.948 INFO:tasks.workunit.client.0.smithi123.stdout:6/171: symlink d10/l3d 0 2024-03-30T04:27:58.950 INFO:tasks.workunit.client.0.smithi123.stdout:6/172: mknod d10/d13/c3e 0 2024-03-30T04:27:58.953 INFO:tasks.workunit.client.0.smithi123.stdout:6/173: unlink f4 0 2024-03-30T04:27:58.965 INFO:tasks.workunit.client.0.smithi123.stdout:6/174: dwrite d10/d13/d1c/f2a [0,4194304] 0 2024-03-30T04:27:59.064 INFO:tasks.workunit.client.0.smithi123.stdout:0/51: sync 2024-03-30T04:27:59.067 INFO:tasks.workunit.client.0.smithi123.stdout:0/52: truncate f5 2666993 0 2024-03-30T04:27:59.069 INFO:tasks.workunit.client.0.smithi123.stdout:0/53: mkdir d6/dd/df 0 2024-03-30T04:27:59.072 INFO:tasks.workunit.client.0.smithi123.stdout:0/54: rmdir d6/dd/df 0 2024-03-30T04:27:59.089 INFO:tasks.workunit.client.0.smithi123.stdout:0/55: dread f5 [0,4194304] 0 2024-03-30T04:27:59.091 INFO:tasks.workunit.client.0.smithi123.stdout:0/56: chown d6/fa 8981710 1 2024-03-30T04:27:59.092 INFO:tasks.workunit.client.0.smithi123.stdout:0/57: write d6/fa [55409,12566] 0 2024-03-30T04:27:59.096 INFO:tasks.workunit.client.0.smithi123.stdout:0/58: symlink d6/dd/l10 0 2024-03-30T04:27:59.098 INFO:tasks.workunit.client.0.smithi123.stdout:0/59: symlink d6/dd/l11 0 2024-03-30T04:27:59.100 INFO:tasks.workunit.client.0.smithi123.stdout:0/60: write d6/f9 [2136724,103419] 0 2024-03-30T04:27:59.103 INFO:tasks.workunit.client.0.smithi123.stdout:0/61: rename f4 to d6/f12 0 2024-03-30T04:27:59.104 INFO:tasks.workunit.client.0.smithi123.stdout:0/62: mknod d6/c13 0 2024-03-30T04:27:59.106 INFO:tasks.workunit.client.0.smithi123.stdout:0/63: creat d6/dd/f14 x:0 0 0 2024-03-30T04:27:59.107 INFO:tasks.workunit.client.0.smithi123.stdout:0/64: chown d6/dd/l10 349560721 1 2024-03-30T04:27:59.114 INFO:tasks.workunit.client.0.smithi123.stdout:0/65: dread f5 [0,4194304] 0 2024-03-30T04:27:59.116 INFO:tasks.workunit.client.0.smithi123.stdout:0/66: mkdir d6/dd/d15 0 2024-03-30T04:27:59.117 INFO:tasks.workunit.client.0.smithi123.stdout:0/67: mkdir d6/dd/d16 0 2024-03-30T04:27:59.122 INFO:tasks.workunit.client.0.smithi123.stdout:0/68: unlink d6/cc 0 2024-03-30T04:27:59.122 INFO:tasks.workunit.client.0.smithi123.stdout:0/69: chown d6/c8 28666 1 2024-03-30T04:27:59.131 INFO:tasks.workunit.client.0.smithi123.stdout:0/70: dread d6/f9 [0,4194304] 0 2024-03-30T04:27:59.131 INFO:tasks.workunit.client.0.smithi123.stdout:0/71: chown d6/dd 49733895 1 2024-03-30T04:27:59.140 INFO:tasks.workunit.client.0.smithi123.stdout:0/72: dread d6/f12 [0,4194304] 0 2024-03-30T04:27:59.141 INFO:tasks.workunit.client.0.smithi123.stdout:0/73: read - d6/fe zero size 2024-03-30T04:27:59.241 INFO:tasks.workunit.client.0.smithi123.stdout:8/107: sync 2024-03-30T04:27:59.243 INFO:tasks.workunit.client.0.smithi123.stdout:8/108: symlink da/dc/d14/l1b 0 2024-03-30T04:27:59.246 INFO:tasks.workunit.client.0.smithi123.stdout:8/109: symlink da/l1c 0 2024-03-30T04:27:59.259 INFO:tasks.workunit.client.0.smithi123.stdout:8/110: dwrite f5 [4194304,4194304] 0 2024-03-30T04:27:59.262 INFO:tasks.workunit.client.0.smithi123.stdout:8/111: fsync da/f12 0 2024-03-30T04:27:59.264 INFO:tasks.workunit.client.0.smithi123.stdout:8/112: chown f8 212878 1 2024-03-30T04:27:59.264 INFO:tasks.workunit.client.0.smithi123.stdout:8/113: fdatasync da/dc/f10 0 2024-03-30T04:27:59.270 INFO:tasks.workunit.client.0.smithi123.stdout:8/114: link da/dc/f13 da/dc/d14/f1d 0 2024-03-30T04:27:59.270 INFO:tasks.workunit.client.0.smithi123.stdout:6/175: sync 2024-03-30T04:27:59.270 INFO:tasks.workunit.client.0.smithi123.stdout:7/44: sync 2024-03-30T04:27:59.270 INFO:tasks.workunit.client.0.smithi123.stdout:0/74: sync 2024-03-30T04:27:59.271 INFO:tasks.workunit.client.0.smithi123.stdout:1/90: sync 2024-03-30T04:27:59.271 INFO:tasks.workunit.client.0.smithi123.stdout:3/84: sync 2024-03-30T04:27:59.271 INFO:tasks.workunit.client.0.smithi123.stdout:5/109: sync 2024-03-30T04:27:59.271 INFO:tasks.workunit.client.0.smithi123.stdout:2/62: sync 2024-03-30T04:27:59.271 INFO:tasks.workunit.client.0.smithi123.stdout:9/60: sync 2024-03-30T04:27:59.271 INFO:tasks.workunit.client.0.smithi123.stdout:4/132: sync 2024-03-30T04:27:59.271 INFO:tasks.workunit.client.0.smithi123.stdout:7/45: readlink d2/l6 0 2024-03-30T04:27:59.273 INFO:tasks.workunit.client.0.smithi123.stdout:5/110: symlink dd/l20 0 2024-03-30T04:27:59.274 INFO:tasks.workunit.client.0.smithi123.stdout:9/61: rmdir d6 39 2024-03-30T04:27:59.275 INFO:tasks.workunit.client.0.smithi123.stdout:3/85: mknod d13/c15 0 2024-03-30T04:27:59.275 INFO:tasks.workunit.client.0.smithi123.stdout:6/176: read f6 [1386795,96273] 0 2024-03-30T04:27:59.276 INFO:tasks.workunit.client.0.smithi123.stdout:8/115: write da/dc/f1a [961793,17429] 0 2024-03-30T04:27:59.277 INFO:tasks.workunit.client.0.smithi123.stdout:9/62: fdatasync f1 0 2024-03-30T04:27:59.278 INFO:tasks.workunit.client.0.smithi123.stdout:0/75: write d6/fe [237213,100957] 0 2024-03-30T04:27:59.279 INFO:tasks.workunit.client.0.smithi123.stdout:2/63: read d0/d8/fa [2423588,16698] 0 2024-03-30T04:27:59.280 INFO:tasks.workunit.client.0.smithi123.stdout:8/116: readlink l7 0 2024-03-30T04:27:59.284 INFO:tasks.workunit.client.0.smithi123.stdout:7/46: creat d2/dc/f17 x:0 0 0 2024-03-30T04:27:59.284 INFO:tasks.workunit.client.0.smithi123.stdout:6/177: rmdir d10/d20 39 2024-03-30T04:27:59.285 INFO:tasks.workunit.client.0.smithi123.stdout:3/86: write fe [5456081,111077] 0 2024-03-30T04:27:59.285 INFO:tasks.workunit.client.0.smithi123.stdout:7/47: dread - d2/f13 zero size 2024-03-30T04:27:59.285 INFO:tasks.workunit.client.0.smithi123.stdout:0/76: chown d6/fe 2162538 1 2024-03-30T04:27:59.287 INFO:tasks.workunit.client.0.smithi123.stdout:4/133: symlink d2/d4/l33 0 2024-03-30T04:27:59.287 INFO:tasks.workunit.client.0.smithi123.stdout:6/178: dread - d10/f29 zero size 2024-03-30T04:27:59.289 INFO:tasks.workunit.client.0.smithi123.stdout:9/63: read d6/fe [3553500,127377] 0 2024-03-30T04:27:59.295 INFO:tasks.workunit.client.0.smithi123.stdout:5/111: getdents dd/d19 0 2024-03-30T04:27:59.297 INFO:tasks.workunit.client.0.smithi123.stdout:7/48: mknod d2/da/c18 0 2024-03-30T04:27:59.298 INFO:tasks.workunit.client.0.smithi123.stdout:5/112: write f7 [2334355,67786] 0 2024-03-30T04:27:59.298 INFO:tasks.workunit.client.0.smithi123.stdout:7/49: chown d2/c12 23129325 1 2024-03-30T04:27:59.298 INFO:tasks.workunit.client.0.smithi123.stdout:0/77: creat d6/dd/d16/f17 x:0 0 0 2024-03-30T04:27:59.299 INFO:tasks.workunit.client.0.smithi123.stdout:0/78: stat f5 0 2024-03-30T04:27:59.299 INFO:tasks.workunit.client.0.smithi123.stdout:9/64: mknod d6/c14 0 2024-03-30T04:27:59.304 INFO:tasks.workunit.client.0.smithi123.stdout:3/87: link ca d13/c16 0 2024-03-30T04:27:59.305 INFO:tasks.workunit.client.0.smithi123.stdout:5/113: symlink dd/l21 0 2024-03-30T04:27:59.306 INFO:tasks.workunit.client.0.smithi123.stdout:9/65: creat d6/f15 x:0 0 0 2024-03-30T04:27:59.306 INFO:tasks.workunit.client.0.smithi123.stdout:3/88: fdatasync fd 0 2024-03-30T04:27:59.308 INFO:tasks.workunit.client.0.smithi123.stdout:5/114: chown c1 400 1 2024-03-30T04:27:59.308 INFO:tasks.workunit.client.0.smithi123.stdout:8/117: dread f6 [0,4194304] 0 2024-03-30T04:27:59.308 INFO:tasks.workunit.client.0.smithi123.stdout:0/79: mkdir d6/dd/d18 0 2024-03-30T04:27:59.311 INFO:tasks.workunit.client.0.smithi123.stdout:5/115: fsync dd/f16 0 2024-03-30T04:27:59.313 INFO:tasks.workunit.client.0.smithi123.stdout:1/91: truncate db/dd/f21 1653958 0 2024-03-30T04:27:59.314 INFO:tasks.workunit.client.0.smithi123.stdout:3/89: rename l8 to d13/l17 0 2024-03-30T04:27:59.314 INFO:tasks.workunit.client.0.smithi123.stdout:8/118: write da/fe [1211171,7828] 0 2024-03-30T04:27:59.315 INFO:tasks.workunit.client.0.smithi123.stdout:3/90: read - d13/f14 zero size 2024-03-30T04:27:59.316 INFO:tasks.workunit.client.0.smithi123.stdout:8/119: chown da/dc/d14 1985 1 2024-03-30T04:27:59.319 INFO:tasks.workunit.client.0.smithi123.stdout:2/64: dwrite d0/f6 [4194304,4194304] 0 2024-03-30T04:27:59.319 INFO:tasks.workunit.client.0.smithi123.stdout:0/80: write d6/dd/d16/f17 [647131,39419] 0 2024-03-30T04:27:59.319 INFO:tasks.workunit.client.0.smithi123.stdout:6/179: getdents d10/d13/d1c 0 2024-03-30T04:27:59.322 INFO:tasks.workunit.client.0.smithi123.stdout:7/50: truncate f0 1226641 0 2024-03-30T04:27:59.322 INFO:tasks.workunit.client.0.smithi123.stdout:3/91: mknod d13/c18 0 2024-03-30T04:27:59.327 INFO:tasks.workunit.client.0.smithi123.stdout:8/120: truncate da/dc/f13 472712 0 2024-03-30T04:27:59.328 INFO:tasks.workunit.client.0.smithi123.stdout:2/65: write d0/f6 [613001,11552] 0 2024-03-30T04:27:59.334 INFO:tasks.workunit.client.0.smithi123.stdout:6/180: mkdir d10/d20/d3f 0 2024-03-30T04:27:59.334 INFO:tasks.workunit.client.0.smithi123.stdout:3/92: readlink d13/l17 0 2024-03-30T04:27:59.336 INFO:tasks.workunit.client.0.smithi123.stdout:9/66: truncate f2 697451 0 2024-03-30T04:27:59.337 INFO:tasks.workunit.client.0.smithi123.stdout:8/121: rmdir da/dc 39 2024-03-30T04:27:59.337 INFO:tasks.workunit.client.0.smithi123.stdout:7/51: creat d2/da/d11/f19 x:0 0 0 2024-03-30T04:27:59.341 INFO:tasks.workunit.client.0.smithi123.stdout:3/93: symlink d13/l19 0 2024-03-30T04:27:59.342 INFO:tasks.workunit.client.0.smithi123.stdout:8/122: chown f9 209055296 1 2024-03-30T04:27:59.342 INFO:tasks.workunit.client.0.smithi123.stdout:9/67: rename d6/c14 to d6/c16 0 2024-03-30T04:27:59.343 INFO:tasks.workunit.client.0.smithi123.stdout:4/134: dwrite d2/d4/d7/d1a/f21 [0,4194304] 0 2024-03-30T04:27:59.343 INFO:tasks.workunit.client.0.smithi123.stdout:9/68: stat c3 0 2024-03-30T04:27:59.345 INFO:tasks.workunit.client.0.smithi123.stdout:2/66: link d0/c9 d0/d8/cd 0 2024-03-30T04:27:59.347 INFO:tasks.workunit.client.0.smithi123.stdout:1/92: dread db/d1b/f1d [0,4194304] 0 2024-03-30T04:27:59.348 INFO:tasks.workunit.client.0.smithi123.stdout:3/94: symlink d13/l1a 0 2024-03-30T04:27:59.354 INFO:tasks.workunit.client.0.smithi123.stdout:2/67: unlink d0/f6 0 2024-03-30T04:27:59.361 INFO:tasks.workunit.client.0.smithi123.stdout:3/95: creat d13/f1b x:0 0 0 2024-03-30T04:27:59.362 INFO:tasks.workunit.client.0.smithi123.stdout:4/135: mkdir d2/d34 0 2024-03-30T04:27:59.365 INFO:tasks.workunit.client.0.smithi123.stdout:6/181: truncate d10/d13/d1c/f1e 3696722 0 2024-03-30T04:27:59.368 INFO:tasks.workunit.client.0.smithi123.stdout:2/68: dread d0/d8/fa [0,4194304] 0 2024-03-30T04:27:59.368 INFO:tasks.workunit.client.0.smithi123.stdout:6/182: write d10/d13/f25 [645066,106821] 0 2024-03-30T04:27:59.369 INFO:tasks.workunit.client.0.smithi123.stdout:9/69: link d6/fe d6/f17 0 2024-03-30T04:27:59.371 INFO:tasks.workunit.client.0.smithi123.stdout:3/96: write f12 [660416,39321] 0 2024-03-30T04:27:59.372 INFO:tasks.workunit.client.0.smithi123.stdout:8/123: creat da/dc/d14/d18/f1e x:0 0 0 2024-03-30T04:27:59.373 INFO:tasks.workunit.client.0.smithi123.stdout:2/69: symlink d0/le 0 2024-03-30T04:27:59.375 INFO:tasks.workunit.client.0.smithi123.stdout:8/124: read - da/dc/d14/f15 zero size 2024-03-30T04:27:59.376 INFO:tasks.workunit.client.0.smithi123.stdout:4/136: rmdir d2 39 2024-03-30T04:27:59.378 INFO:tasks.workunit.client.0.smithi123.stdout:8/125: fsync f5 0 2024-03-30T04:27:59.385 INFO:tasks.workunit.client.0.smithi123.stdout:2/70: rename d0/c9 to d0/cf 0 2024-03-30T04:27:59.395 INFO:tasks.workunit.client.0.smithi123.stdout:2/71: write d0/d8/fa [3527529,82882] 0 2024-03-30T04:27:59.395 INFO:tasks.workunit.client.0.smithi123.stdout:4/137: truncate d2/d4/f8 2152896 0 2024-03-30T04:27:59.399 INFO:tasks.workunit.client.0.smithi123.stdout:9/70: dwrite f0 [0,4194304] 0 2024-03-30T04:27:59.400 INFO:tasks.workunit.client.0.smithi123.stdout:1/93: dwrite db/dd/f21 [0,4194304] 0 2024-03-30T04:27:59.407 INFO:tasks.workunit.client.0.smithi123.stdout:2/72: creat d0/f10 x:0 0 0 2024-03-30T04:27:59.407 INFO:tasks.workunit.client.0.smithi123.stdout:4/138: write d2/d4/d7/d1a/f21 [470085,27167] 0 2024-03-30T04:27:59.407 INFO:tasks.workunit.client.0.smithi123.stdout:3/97: dwrite d13/f1b [0,4194304] 0 2024-03-30T04:27:59.408 INFO:tasks.workunit.client.0.smithi123.stdout:9/71: creat d6/f18 x:0 0 0 2024-03-30T04:27:59.408 INFO:tasks.workunit.client.0.smithi123.stdout:8/126: dwrite f8 [0,4194304] 0 2024-03-30T04:27:59.410 INFO:tasks.workunit.client.0.smithi123.stdout:2/73: dread - d0/f10 zero size 2024-03-30T04:27:59.413 INFO:tasks.workunit.client.0.smithi123.stdout:9/72: dread - d6/fd zero size 2024-03-30T04:27:59.417 INFO:tasks.workunit.client.0.smithi123.stdout:2/74: write d0/d8/fa [3437258,22309] 0 2024-03-30T04:27:59.418 INFO:tasks.workunit.client.0.smithi123.stdout:3/98: rename d13/l1a to d13/l1c 0 2024-03-30T04:27:59.418 INFO:tasks.workunit.client.0.smithi123.stdout:9/73: mknod d6/c19 0 2024-03-30T04:27:59.419 INFO:tasks.workunit.client.0.smithi123.stdout:4/139: write d2/d4/d7/d1a/f21 [2604750,93718] 0 2024-03-30T04:27:59.422 INFO:tasks.workunit.client.0.smithi123.stdout:2/75: truncate d0/f10 356264 0 2024-03-30T04:27:59.422 INFO:tasks.workunit.client.0.smithi123.stdout:4/140: stat d2/d4/d7/d1a/f21 0 2024-03-30T04:27:59.433 INFO:tasks.workunit.client.0.smithi123.stdout:3/99: creat d13/f1d x:0 0 0 2024-03-30T04:27:59.442 INFO:tasks.workunit.client.0.smithi123.stdout:4/141: dread d2/d4/f2d [0,4194304] 0 2024-03-30T04:27:59.443 INFO:tasks.workunit.client.0.smithi123.stdout:9/74: link d6/l13 d6/l1a 0 2024-03-30T04:27:59.449 INFO:tasks.workunit.client.0.smithi123.stdout:1/94: dwrite db/f20 [0,4194304] 0 2024-03-30T04:27:59.450 INFO:tasks.workunit.client.0.smithi123.stdout:8/127: dwrite f6 [4194304,4194304] 0 2024-03-30T04:27:59.456 INFO:tasks.workunit.client.0.smithi123.stdout:1/95: fsync db/d1b/f1e 0 2024-03-30T04:27:59.458 INFO:tasks.workunit.client.0.smithi123.stdout:4/142: mkdir d2/d2e/d35 0 2024-03-30T04:27:59.458 INFO:tasks.workunit.client.0.smithi123.stdout:1/96: chown db/dd 178980 1 2024-03-30T04:27:59.463 INFO:tasks.workunit.client.0.smithi123.stdout:2/76: dread d0/d8/fa [0,4194304] 0 2024-03-30T04:27:59.471 INFO:tasks.workunit.client.0.smithi123.stdout:3/100: dwrite f12 [0,4194304] 0 2024-03-30T04:27:59.477 INFO:tasks.workunit.client.0.smithi123.stdout:2/77: chown d0/f10 4375 1 2024-03-30T04:27:59.477 INFO:tasks.workunit.client.0.smithi123.stdout:2/78: stat d0/f10 0 2024-03-30T04:27:59.480 INFO:tasks.workunit.client.0.smithi123.stdout:3/101: unlink lc 0 2024-03-30T04:27:59.482 INFO:tasks.workunit.client.0.smithi123.stdout:1/97: dread db/fe [0,4194304] 0 2024-03-30T04:27:59.486 INFO:tasks.workunit.client.0.smithi123.stdout:1/98: rename db to db/dd/d26 22 2024-03-30T04:27:59.486 INFO:tasks.workunit.client.0.smithi123.stdout:3/102: rename d13/l19 to d13/l1e 0 2024-03-30T04:27:59.493 INFO:tasks.workunit.client.0.smithi123.stdout:3/103: write ff [949398,55529] 0 2024-03-30T04:27:59.494 INFO:tasks.workunit.client.0.smithi123.stdout:1/99: mknod db/d1b/c27 0 2024-03-30T04:27:59.495 INFO:tasks.workunit.client.0.smithi123.stdout:9/75: dread f0 [0,4194304] 0 2024-03-30T04:27:59.496 INFO:tasks.workunit.client.0.smithi123.stdout:8/128: dread f8 [0,4194304] 0 2024-03-30T04:27:59.499 INFO:tasks.workunit.client.0.smithi123.stdout:3/104: fdatasync fe 0 2024-03-30T04:27:59.504 INFO:tasks.workunit.client.0.smithi123.stdout:1/100: creat db/d1b/f28 x:0 0 0 2024-03-30T04:27:59.504 INFO:tasks.workunit.client.0.smithi123.stdout:4/143: dwrite d2/d4/d7/d1a/f1f [0,4194304] 0 2024-03-30T04:27:59.505 INFO:tasks.workunit.client.0.smithi123.stdout:1/101: fsync db/f20 0 2024-03-30T04:27:59.507 INFO:tasks.workunit.client.0.smithi123.stdout:8/129: read da/dc/f10 [473452,105582] 0 2024-03-30T04:27:59.508 INFO:tasks.workunit.client.0.smithi123.stdout:1/102: truncate db/f24 6059445 0 2024-03-30T04:27:59.508 INFO:tasks.workunit.client.0.smithi123.stdout:3/105: mknod d13/c1f 0 2024-03-30T04:27:59.509 INFO:tasks.workunit.client.0.smithi123.stdout:2/79: dread d0/d8/fa [0,4194304] 0 2024-03-30T04:27:59.510 INFO:tasks.workunit.client.0.smithi123.stdout:8/130: readlink da/dc/l19 0 2024-03-30T04:27:59.514 INFO:tasks.workunit.client.0.smithi123.stdout:8/131: write da/ff [190944,55774] 0 2024-03-30T04:27:59.519 INFO:tasks.workunit.client.0.smithi123.stdout:2/80: write d0/d8/fa [2301902,7362] 0 2024-03-30T04:27:59.526 INFO:tasks.workunit.client.0.smithi123.stdout:9/76: dwrite d6/fb [0,4194304] 0 2024-03-30T04:27:59.530 INFO:tasks.workunit.client.0.smithi123.stdout:1/103: dread db/dd/f21 [0,4194304] 0 2024-03-30T04:27:59.534 INFO:tasks.workunit.client.0.smithi123.stdout:1/104: creat db/d1b/d1c/f29 x:0 0 0 2024-03-30T04:27:59.535 INFO:tasks.workunit.client.0.smithi123.stdout:1/105: creat db/d1b/f2a x:0 0 0 2024-03-30T04:27:59.536 INFO:tasks.workunit.client.0.smithi123.stdout:1/106: fdatasync db/f20 0 2024-03-30T04:27:59.538 INFO:tasks.workunit.client.0.smithi123.stdout:1/107: fdatasync db/d1b/f1a 0 2024-03-30T04:27:59.539 INFO:tasks.workunit.client.0.smithi123.stdout:9/77: write d6/f7 [987031,103859] 0 2024-03-30T04:27:59.539 INFO:tasks.workunit.client.0.smithi123.stdout:1/108: stat db/d1b/f28 0 2024-03-30T04:27:59.540 INFO:tasks.workunit.client.0.smithi123.stdout:9/78: chown d6/f15 3 1 2024-03-30T04:27:59.542 INFO:tasks.workunit.client.0.smithi123.stdout:1/109: creat db/d1b/d1c/f2b x:0 0 0 2024-03-30T04:27:59.543 INFO:tasks.workunit.client.0.smithi123.stdout:9/79: readlink d6/l11 0 2024-03-30T04:27:59.545 INFO:tasks.workunit.client.0.smithi123.stdout:1/110: mknod db/dd/c2c 0 2024-03-30T04:27:59.547 INFO:tasks.workunit.client.0.smithi123.stdout:9/80: rename d6/f18 to d6/f1b 0 2024-03-30T04:27:59.548 INFO:tasks.workunit.client.0.smithi123.stdout:1/111: symlink db/l2d 0 2024-03-30T04:27:59.550 INFO:tasks.workunit.client.0.smithi123.stdout:1/112: chown db/dd/l25 31 1 2024-03-30T04:27:59.550 INFO:tasks.workunit.client.0.smithi123.stdout:9/81: unlink d6/l1a 0 2024-03-30T04:27:59.551 INFO:tasks.workunit.client.0.smithi123.stdout:1/113: rmdir db 39 2024-03-30T04:27:59.552 INFO:tasks.workunit.client.0.smithi123.stdout:9/82: mknod d6/c1c 0 2024-03-30T04:27:59.554 INFO:tasks.workunit.client.0.smithi123.stdout:9/83: symlink d6/l1d 0 2024-03-30T04:27:59.555 INFO:tasks.workunit.client.0.smithi123.stdout:9/84: chown d6/f10 90417805 1 2024-03-30T04:27:59.557 INFO:tasks.workunit.client.0.smithi123.stdout:9/85: symlink d6/l1e 0 2024-03-30T04:27:59.561 INFO:tasks.workunit.client.0.smithi123.stdout:1/114: write db/d1b/f1d [4121996,3916] 0 2024-03-30T04:27:59.562 INFO:tasks.workunit.client.0.smithi123.stdout:1/115: read db/d1b/f1d [2272155,19132] 0 2024-03-30T04:27:59.565 INFO:tasks.workunit.client.0.smithi123.stdout:1/116: fsync db/fe 0 2024-03-30T04:27:59.568 INFO:tasks.workunit.client.0.smithi123.stdout:1/117: truncate db/f24 6543840 0 2024-03-30T04:27:59.575 INFO:tasks.workunit.client.0.smithi123.stdout:9/86: dwrite d6/f15 [0,4194304] 0 2024-03-30T04:27:59.577 INFO:tasks.workunit.client.0.smithi123.stdout:9/87: write f1 [925937,42703] 0 2024-03-30T04:27:59.589 INFO:tasks.workunit.client.0.smithi123.stdout:1/118: dwrite db/fe [0,4194304] 0 2024-03-30T04:27:59.596 INFO:tasks.workunit.client.0.smithi123.stdout:9/88: dwrite f0 [0,4194304] 0 2024-03-30T04:27:59.605 INFO:tasks.workunit.client.0.smithi123.stdout:9/89: getdents d6 0 2024-03-30T04:27:59.606 INFO:tasks.workunit.client.0.smithi123.stdout:1/119: dread db/d1b/f1d [0,4194304] 0 2024-03-30T04:27:59.607 INFO:tasks.workunit.client.0.smithi123.stdout:9/90: dread d6/f9 [0,4194304] 0 2024-03-30T04:27:59.610 INFO:tasks.workunit.client.0.smithi123.stdout:1/120: creat db/d1b/d1c/f2e x:0 0 0 2024-03-30T04:27:59.610 INFO:tasks.workunit.client.0.smithi123.stdout:9/91: mkdir d6/d1f 0 2024-03-30T04:27:59.611 INFO:tasks.workunit.client.0.smithi123.stdout:1/121: dread - db/d1b/f28 zero size 2024-03-30T04:27:59.612 INFO:tasks.workunit.client.0.smithi123.stdout:9/92: mknod d6/c20 0 2024-03-30T04:27:59.616 INFO:tasks.workunit.client.0.smithi123.stdout:9/93: creat d6/f21 x:0 0 0 2024-03-30T04:27:59.622 INFO:tasks.workunit.client.0.smithi123.stdout:9/94: rename d6/f17 to d6/d1f/f22 0 2024-03-30T04:27:59.628 INFO:tasks.workunit.client.0.smithi123.stdout:1/122: dread db/fe [0,4194304] 0 2024-03-30T04:27:59.631 INFO:tasks.workunit.client.0.smithi123.stdout:9/95: rmdir d6 39 2024-03-30T04:27:59.637 INFO:tasks.workunit.client.0.smithi123.stdout:9/96: unlink f2 0 2024-03-30T04:27:59.639 INFO:tasks.workunit.client.0.smithi123.stdout:9/97: write d6/f10 [2914430,37700] 0 2024-03-30T04:27:59.642 INFO:tasks.workunit.client.0.smithi123.stdout:1/123: dread f8 [0,4194304] 0 2024-03-30T04:27:59.643 INFO:tasks.workunit.client.0.smithi123.stdout:9/98: mkdir d6/d23 0 2024-03-30T04:27:59.644 INFO:tasks.workunit.client.0.smithi123.stdout:9/99: chown d6/la 3586 1 2024-03-30T04:27:59.645 INFO:tasks.workunit.client.0.smithi123.stdout:1/124: rmdir db/dd 39 2024-03-30T04:27:59.648 INFO:tasks.workunit.client.0.smithi123.stdout:1/125: chown db/dd 47516698 1 2024-03-30T04:27:59.649 INFO:tasks.workunit.client.0.smithi123.stdout:1/126: dread - db/d1b/f28 zero size 2024-03-30T04:27:59.653 INFO:tasks.workunit.client.0.smithi123.stdout:9/100: getdents d6 0 2024-03-30T04:27:59.657 INFO:tasks.workunit.client.0.smithi123.stdout:9/101: write d6/fe [3412098,69034] 0 2024-03-30T04:27:59.662 INFO:tasks.workunit.client.0.smithi123.stdout:9/102: rename d6/c1c to d6/d1f/c24 0 2024-03-30T04:27:59.662 INFO:tasks.workunit.client.0.smithi123.stdout:1/127: dread db/f20 [0,4194304] 0 2024-03-30T04:27:59.664 INFO:tasks.workunit.client.0.smithi123.stdout:9/103: mknod d6/d1f/c25 0 2024-03-30T04:27:59.667 INFO:tasks.workunit.client.0.smithi123.stdout:9/104: rename d6/fe to d6/f26 0 2024-03-30T04:27:59.669 INFO:tasks.workunit.client.0.smithi123.stdout:9/105: mknod d6/d1f/c27 0 2024-03-30T04:27:59.678 INFO:tasks.workunit.client.0.smithi123.stdout:9/106: truncate d6/fb 1020338 0 2024-03-30T04:27:59.678 INFO:tasks.workunit.client.0.smithi123.stdout:1/128: dwrite db/dd/f21 [0,4194304] 0 2024-03-30T04:27:59.680 INFO:tasks.workunit.client.0.smithi123.stdout:9/107: rmdir d6/d1f 39 2024-03-30T04:27:59.682 INFO:tasks.workunit.client.0.smithi123.stdout:9/108: write d6/f7 [1752409,31647] 0 2024-03-30T04:27:59.682 INFO:tasks.workunit.client.0.smithi123.stdout:1/129: rename db/d1b/f1e to db/f2f 0 2024-03-30T04:27:59.685 INFO:tasks.workunit.client.0.smithi123.stdout:9/109: chown d6/d1f/f22 1994 1 2024-03-30T04:27:59.689 INFO:tasks.workunit.client.0.smithi123.stdout:9/110: creat d6/d1f/f28 x:0 0 0 2024-03-30T04:27:59.693 INFO:tasks.workunit.client.0.smithi123.stdout:9/111: rename d6/f7 to d6/d1f/f29 0 2024-03-30T04:27:59.699 INFO:tasks.workunit.client.0.smithi123.stdout:9/112: creat d6/d23/f2a x:0 0 0 2024-03-30T04:27:59.699 INFO:tasks.workunit.client.0.smithi123.stdout:1/130: write db/f24 [560430,84849] 0 2024-03-30T04:27:59.704 INFO:tasks.workunit.client.0.smithi123.stdout:1/131: chown db/d1b/c17 976365 1 2024-03-30T04:27:59.706 INFO:tasks.workunit.client.0.smithi123.stdout:1/132: symlink db/d1b/l30 0 2024-03-30T04:27:59.707 INFO:tasks.workunit.client.0.smithi123.stdout:1/133: truncate db/dd/f13 7023676 0 2024-03-30T04:27:59.709 INFO:tasks.workunit.client.0.smithi123.stdout:9/113: dread d6/d1f/f29 [0,4194304] 0 2024-03-30T04:27:59.712 INFO:tasks.workunit.client.0.smithi123.stdout:9/114: unlink d6/d1f/f22 0 2024-03-30T04:27:59.713 INFO:tasks.workunit.client.0.smithi123.stdout:9/115: creat d6/d23/f2b x:0 0 0 2024-03-30T04:27:59.718 INFO:tasks.workunit.client.0.smithi123.stdout:9/116: rename d6/c16 to d6/d1f/c2c 0 2024-03-30T04:27:59.773 INFO:tasks.workunit.client.0.smithi123.stdout:9/117: creat d6/d1f/f2d x:0 0 0 2024-03-30T04:27:59.775 INFO:tasks.workunit.client.0.smithi123.stdout:1/134: dwrite db/dd/f13 [4194304,4194304] 0 2024-03-30T04:27:59.775 INFO:tasks.workunit.client.0.smithi123.stdout:9/118: creat d6/d1f/f2e x:0 0 0 2024-03-30T04:27:59.777 INFO:tasks.workunit.client.0.smithi123.stdout:1/135: fsync db/d1b/d1c/f29 0 2024-03-30T04:27:59.777 INFO:tasks.workunit.client.0.smithi123.stdout:9/119: chown d6/d1f/f2d 39 1 2024-03-30T04:27:59.779 INFO:tasks.workunit.client.0.smithi123.stdout:9/120: write d6/f10 [4871783,110328] 0 2024-03-30T04:27:59.780 INFO:tasks.workunit.client.0.smithi123.stdout:1/136: mknod db/c31 0 2024-03-30T04:27:59.783 INFO:tasks.workunit.client.0.smithi123.stdout:1/137: rmdir db/dd 39 2024-03-30T04:27:59.785 INFO:tasks.workunit.client.0.smithi123.stdout:9/121: link d6/l1d d6/l2f 0 2024-03-30T04:27:59.787 INFO:tasks.workunit.client.0.smithi123.stdout:9/122: dread - d6/d1f/f28 zero size 2024-03-30T04:27:59.788 INFO:tasks.workunit.client.0.smithi123.stdout:1/138: mknod db/d1b/c32 0 2024-03-30T04:27:59.791 INFO:tasks.workunit.client.0.smithi123.stdout:9/123: mknod d6/c30 0 2024-03-30T04:27:59.793 INFO:tasks.workunit.client.0.smithi123.stdout:1/139: mkdir db/dd/d33 0 2024-03-30T04:27:59.794 INFO:tasks.workunit.client.0.smithi123.stdout:9/124: truncate d6/d1f/f28 156643 0 2024-03-30T04:27:59.795 INFO:tasks.workunit.client.0.smithi123.stdout:1/140: stat db/d1b/f1d 0 2024-03-30T04:27:59.799 INFO:tasks.workunit.client.0.smithi123.stdout:1/141: fsync db/f24 0 2024-03-30T04:27:59.807 INFO:tasks.workunit.client.0.smithi123.stdout:1/142: getdents db 0 2024-03-30T04:27:59.810 INFO:tasks.workunit.client.0.smithi123.stdout:1/143: symlink db/dd/l34 0 2024-03-30T04:27:59.812 INFO:tasks.workunit.client.0.smithi123.stdout:9/125: dwrite d6/f15 [0,4194304] 0 2024-03-30T04:27:59.815 INFO:tasks.workunit.client.0.smithi123.stdout:9/126: symlink d6/l31 0 2024-03-30T04:27:59.816 INFO:tasks.workunit.client.0.smithi123.stdout:9/127: fsync d6/d1f/f2d 0 2024-03-30T04:27:59.820 INFO:tasks.workunit.client.0.smithi123.stdout:9/128: rename d6/cf to d6/c32 0 2024-03-30T04:27:59.821 INFO:tasks.workunit.client.0.smithi123.stdout:1/144: dread db/f2f [0,4194304] 0 2024-03-30T04:27:59.823 INFO:tasks.workunit.client.0.smithi123.stdout:1/145: chown db/d1b/f28 6 1 2024-03-30T04:27:59.825 INFO:tasks.workunit.client.0.smithi123.stdout:9/129: link d6/d23/f2b d6/d23/f33 0 2024-03-30T04:27:59.826 INFO:tasks.workunit.client.0.smithi123.stdout:1/146: symlink db/dd/d33/l35 0 2024-03-30T04:27:59.828 INFO:tasks.workunit.client.0.smithi123.stdout:1/147: write db/dd/f13 [786881,11584] 0 2024-03-30T04:27:59.828 INFO:tasks.workunit.client.0.smithi123.stdout:9/130: unlink d6/d1f/c24 0 2024-03-30T04:27:59.849 INFO:tasks.workunit.client.0.smithi123.stdout:1/148: dwrite db/d1b/d1c/f29 [0,4194304] 0 2024-03-30T04:27:59.863 INFO:tasks.workunit.client.0.smithi123.stdout:1/149: dread db/dd/f13 [0,4194304] 0 2024-03-30T04:27:59.864 INFO:tasks.workunit.client.0.smithi123.stdout:1/150: write db/d1b/f2a [56248,76215] 0 2024-03-30T04:27:59.878 INFO:tasks.workunit.client.0.smithi123.stdout:1/151: dwrite f8 [0,4194304] 0 2024-03-30T04:27:59.902 INFO:tasks.workunit.client.0.smithi123.stdout:1/152: fsync db/d1b/f1d 0 2024-03-30T04:27:59.903 INFO:tasks.workunit.client.0.smithi123.stdout:1/153: fsync db/d1b/d1c/f2e 0 2024-03-30T04:27:59.904 INFO:tasks.workunit.client.0.smithi123.stdout:1/154: chown db/d1b/f28 325 1 2024-03-30T04:27:59.906 INFO:tasks.workunit.client.0.smithi123.stdout:1/155: mknod db/dd/d33/c36 0 2024-03-30T04:27:59.908 INFO:tasks.workunit.client.0.smithi123.stdout:1/156: creat db/d1b/f37 x:0 0 0 2024-03-30T04:27:59.910 INFO:tasks.workunit.client.0.smithi123.stdout:1/157: write db/d1b/d1c/f29 [4135593,35659] 0 2024-03-30T04:27:59.919 INFO:tasks.workunit.client.0.smithi123.stdout:5/116: sync 2024-03-30T04:27:59.919 INFO:tasks.workunit.client.0.smithi123.stdout:7/52: sync 2024-03-30T04:27:59.919 INFO:tasks.workunit.client.0.smithi123.stdout:0/81: sync 2024-03-30T04:27:59.920 INFO:tasks.workunit.client.0.smithi123.stdout:7/53: stat d2/da/l16 0 2024-03-30T04:27:59.922 INFO:tasks.workunit.client.0.smithi123.stdout:5/117: truncate dd/fe 4379184 0 2024-03-30T04:27:59.925 INFO:tasks.workunit.client.0.smithi123.stdout:5/118: dread - dd/f15 zero size 2024-03-30T04:27:59.926 INFO:tasks.workunit.client.0.smithi123.stdout:1/158: dwrite db/f20 [0,4194304] 0 2024-03-30T04:27:59.932 INFO:tasks.workunit.client.0.smithi123.stdout:5/119: getdents dd/d19 0 2024-03-30T04:27:59.938 INFO:tasks.workunit.client.0.smithi123.stdout:7/54: dread d2/dc/ff [0,4194304] 0 2024-03-30T04:27:59.942 INFO:tasks.workunit.client.0.smithi123.stdout:0/82: dwrite d6/f9 [0,4194304] 0 2024-03-30T04:27:59.942 INFO:tasks.workunit.client.0.smithi123.stdout:1/159: dread db/dd/f21 [0,4194304] 0 2024-03-30T04:27:59.948 INFO:tasks.workunit.client.0.smithi123.stdout:0/83: unlink d6/dd/d16/f17 0 2024-03-30T04:27:59.953 INFO:tasks.workunit.client.0.smithi123.stdout:7/55: dread d2/da/fb [0,4194304] 0 2024-03-30T04:27:59.964 INFO:tasks.workunit.client.0.smithi123.stdout:1/160: dwrite db/f2f [0,4194304] 0 2024-03-30T04:27:59.966 INFO:tasks.workunit.client.0.smithi123.stdout:0/84: fdatasync d6/f9 0 2024-03-30T04:27:59.966 INFO:tasks.workunit.client.0.smithi123.stdout:1/161: mknod db/d1b/c38 0 2024-03-30T04:27:59.970 INFO:tasks.workunit.client.0.smithi123.stdout:1/162: mkdir db/d1b/d1c/d39 0 2024-03-30T04:27:59.972 INFO:tasks.workunit.client.0.smithi123.stdout:0/85: link d6/f9 d6/dd/d15/f19 0 2024-03-30T04:27:59.973 INFO:tasks.workunit.client.0.smithi123.stdout:0/86: readlink d6/dd/l11 0 2024-03-30T04:27:59.973 INFO:tasks.workunit.client.0.smithi123.stdout:1/163: truncate db/d1b/d1c/f2b 475239 0 2024-03-30T04:27:59.976 INFO:tasks.workunit.client.0.smithi123.stdout:0/87: mknod d6/dd/d16/c1a 0 2024-03-30T04:27:59.978 INFO:tasks.workunit.client.0.smithi123.stdout:0/88: truncate d6/dd/d15/f19 4388900 0 2024-03-30T04:27:59.979 INFO:tasks.workunit.client.0.smithi123.stdout:0/89: chown d6 16406781 1 2024-03-30T04:27:59.981 INFO:tasks.workunit.client.0.smithi123.stdout:0/90: getdents d6/dd/d18 0 2024-03-30T04:27:59.983 INFO:tasks.workunit.client.0.smithi123.stdout:0/91: creat d6/f1b x:0 0 0 2024-03-30T04:27:59.984 INFO:tasks.workunit.client.0.smithi123.stdout:0/92: chown d6/c8 16905 1 2024-03-30T04:27:59.985 INFO:tasks.workunit.client.0.smithi123.stdout:1/164: dread f8 [0,4194304] 0 2024-03-30T04:27:59.988 INFO:tasks.workunit.client.0.smithi123.stdout:1/165: symlink db/d1b/d1c/l3a 0 2024-03-30T04:27:59.991 INFO:tasks.workunit.client.0.smithi123.stdout:1/166: symlink db/dd/l3b 0 2024-03-30T04:27:59.993 INFO:tasks.workunit.client.0.smithi123.stdout:0/93: fsync d6/dd/d15/f19 0 2024-03-30T04:27:59.995 INFO:tasks.workunit.client.0.smithi123.stdout:1/167: rename db/lc to db/dd/l3c 0 2024-03-30T04:27:59.996 INFO:tasks.workunit.client.0.smithi123.stdout:0/94: mknod d6/dd/d18/c1c 0 2024-03-30T04:27:59.999 INFO:tasks.workunit.client.0.smithi123.stdout:0/95: chown d6/cb 22586 1 2024-03-30T04:28:00.005 INFO:tasks.workunit.client.0.smithi123.stdout:1/168: symlink db/dd/d33/l3d 0 2024-03-30T04:28:00.010 INFO:tasks.workunit.client.0.smithi123.stdout:1/169: rename db/f2f to db/d1b/f3e 0 2024-03-30T04:28:00.018 INFO:tasks.workunit.client.0.smithi123.stdout:1/170: fdatasync db/fe 0 2024-03-30T04:28:00.018 INFO:tasks.workunit.client.0.smithi123.stdout:0/96: dwrite f5 [0,4194304] 0 2024-03-30T04:28:00.027 INFO:tasks.workunit.client.0.smithi123.stdout:1/171: truncate db/f24 1372989 0 2024-03-30T04:28:00.029 INFO:tasks.workunit.client.0.smithi123.stdout:1/172: chown db/d1b/f3e 199722 1 2024-03-30T04:28:00.032 INFO:tasks.workunit.client.0.smithi123.stdout:1/173: mknod db/d1b/d1c/d39/c3f 0 2024-03-30T04:28:00.035 INFO:tasks.workunit.client.0.smithi123.stdout:1/174: fdatasync db/f24 0 2024-03-30T04:28:00.038 INFO:tasks.workunit.client.0.smithi123.stdout:3/106: sync 2024-03-30T04:28:00.038 INFO:tasks.workunit.client.0.smithi123.stdout:4/144: sync 2024-03-30T04:28:00.038 INFO:tasks.workunit.client.0.smithi123.stdout:6/183: sync 2024-03-30T04:28:00.039 INFO:tasks.workunit.client.0.smithi123.stdout:8/132: sync 2024-03-30T04:28:00.040 INFO:tasks.workunit.client.0.smithi123.stdout:3/107: dread - fd zero size 2024-03-30T04:28:00.040 INFO:tasks.workunit.client.0.smithi123.stdout:1/175: symlink db/l40 0 2024-03-30T04:28:00.041 INFO:tasks.workunit.client.0.smithi123.stdout:6/184: stat d10/f1d 0 2024-03-30T04:28:00.042 INFO:tasks.workunit.client.0.smithi123.stdout:3/108: chown d13/l17 2 1 2024-03-30T04:28:00.043 INFO:tasks.workunit.client.0.smithi123.stdout:3/109: fsync ff 0 2024-03-30T04:28:00.044 INFO:tasks.workunit.client.0.smithi123.stdout:8/133: unlink da/cd 0 2024-03-30T04:28:00.044 INFO:tasks.workunit.client.0.smithi123.stdout:3/110: read fe [3729549,25866] 0 2024-03-30T04:28:00.047 INFO:tasks.workunit.client.0.smithi123.stdout:4/145: truncate d2/f1c 326371 0 2024-03-30T04:28:00.048 INFO:tasks.workunit.client.0.smithi123.stdout:1/176: creat db/f41 x:0 0 0 2024-03-30T04:28:00.050 INFO:tasks.workunit.client.0.smithi123.stdout:4/146: write d2/d4/d7/fe [984039,86330] 0 2024-03-30T04:28:00.056 INFO:tasks.workunit.client.0.smithi123.stdout:4/147: fsync d2/f1c 0 2024-03-30T04:28:00.059 INFO:tasks.workunit.client.0.smithi123.stdout:1/177: getdents db/dd 0 2024-03-30T04:28:00.065 INFO:tasks.workunit.client.0.smithi123.stdout:4/148: rename d2/d4/l9 to d2/d4/d7/l36 0 2024-03-30T04:28:00.067 INFO:tasks.workunit.client.0.smithi123.stdout:1/178: rename db/dd/d33 to db/dd/d42 0 2024-03-30T04:28:00.069 INFO:tasks.workunit.client.0.smithi123.stdout:6/185: dwrite d10/d13/d1c/f2a [4194304,4194304] 0 2024-03-30T04:28:00.070 INFO:tasks.workunit.client.0.smithi123.stdout:3/111: dwrite ff [0,4194304] 0 2024-03-30T04:28:00.074 INFO:tasks.workunit.client.0.smithi123.stdout:1/179: chown db/d1b/c32 161 1 2024-03-30T04:28:00.074 INFO:tasks.workunit.client.0.smithi123.stdout:4/149: mknod d2/d4/d2c/c37 0 2024-03-30T04:28:00.076 INFO:tasks.workunit.client.0.smithi123.stdout:1/180: dread - db/f41 zero size 2024-03-30T04:28:00.076 INFO:tasks.workunit.client.0.smithi123.stdout:3/112: mknod d13/c20 0 2024-03-30T04:28:00.078 INFO:tasks.workunit.client.0.smithi123.stdout:3/113: chown f11 0 1 2024-03-30T04:28:00.078 INFO:tasks.workunit.client.0.smithi123.stdout:6/186: creat d10/d20/f40 x:0 0 0 2024-03-30T04:28:00.079 INFO:tasks.workunit.client.0.smithi123.stdout:4/150: creat d2/d2e/f38 x:0 0 0 2024-03-30T04:28:00.080 INFO:tasks.workunit.client.0.smithi123.stdout:6/187: readlink d10/d20/l24 0 2024-03-30T04:28:00.082 INFO:tasks.workunit.client.0.smithi123.stdout:4/151: rename d2 to d2/d34/d39 22 2024-03-30T04:28:00.083 INFO:tasks.workunit.client.0.smithi123.stdout:6/188: dread - d10/f1b zero size 2024-03-30T04:28:00.085 INFO:tasks.workunit.client.0.smithi123.stdout:1/181: mknod db/c43 0 2024-03-30T04:28:00.088 INFO:tasks.workunit.client.0.smithi123.stdout:4/152: creat d2/d2e/f3a x:0 0 0 2024-03-30T04:28:00.090 INFO:tasks.workunit.client.0.smithi123.stdout:1/182: rename db/dd/l3c to db/d1b/d1c/l44 0 2024-03-30T04:28:00.097 INFO:tasks.workunit.client.0.smithi123.stdout:4/153: getdents d2/d4/d2c 0 2024-03-30T04:28:00.099 INFO:tasks.workunit.client.0.smithi123.stdout:3/114: rmdir d13 39 2024-03-30T04:28:00.100 INFO:tasks.workunit.client.0.smithi123.stdout:3/115: dread - fd zero size 2024-03-30T04:28:00.102 INFO:tasks.workunit.client.0.smithi123.stdout:3/116: readlink l2 0 2024-03-30T04:28:00.105 INFO:tasks.workunit.client.0.smithi123.stdout:4/154: creat d2/d4/f3b x:0 0 0 2024-03-30T04:28:00.107 INFO:tasks.workunit.client.0.smithi123.stdout:6/189: dwrite d10/f11 [0,4194304] 0 2024-03-30T04:28:00.107 INFO:tasks.workunit.client.0.smithi123.stdout:3/117: dread - d13/f14 zero size 2024-03-30T04:28:00.107 INFO:tasks.workunit.client.0.smithi123.stdout:4/155: stat d2/d26 0 2024-03-30T04:28:00.110 INFO:tasks.workunit.client.0.smithi123.stdout:4/156: chown d2/d2e/f38 8 1 2024-03-30T04:28:00.110 INFO:tasks.workunit.client.0.smithi123.stdout:1/183: dread db/d1b/d1c/f29 [0,4194304] 0 2024-03-30T04:28:00.113 INFO:tasks.workunit.client.0.smithi123.stdout:6/190: mkdir d10/d20/d41 0 2024-03-30T04:28:00.114 INFO:tasks.workunit.client.0.smithi123.stdout:1/184: readlink db/dd/l25 0 2024-03-30T04:28:00.116 INFO:tasks.workunit.client.0.smithi123.stdout:6/191: chown d10/l18 20 1 2024-03-30T04:28:00.116 INFO:tasks.workunit.client.0.smithi123.stdout:3/118: unlink d13/c16 0 2024-03-30T04:28:00.117 INFO:tasks.workunit.client.0.smithi123.stdout:1/185: rmdir db/dd 39 2024-03-30T04:28:00.120 INFO:tasks.workunit.client.0.smithi123.stdout:6/192: mkdir d10/d13/d1c/d42 0 2024-03-30T04:28:00.121 INFO:tasks.workunit.client.0.smithi123.stdout:6/193: read d10/f11 [3471041,39622] 0 2024-03-30T04:28:00.127 INFO:tasks.workunit.client.0.smithi123.stdout:1/186: creat db/f45 x:0 0 0 2024-03-30T04:28:00.130 INFO:tasks.workunit.client.0.smithi123.stdout:6/194: rename d10/d13/l23 to d10/d13/l43 0 2024-03-30T04:28:00.132 INFO:tasks.workunit.client.0.smithi123.stdout:1/187: stat db/dd/f13 0 2024-03-30T04:28:00.135 INFO:tasks.workunit.client.0.smithi123.stdout:6/195: mknod d10/d20/c44 0 2024-03-30T04:28:00.145 INFO:tasks.workunit.client.0.smithi123.stdout:4/157: dwrite d2/f1c [0,4194304] 0 2024-03-30T04:28:00.145 INFO:tasks.workunit.client.0.smithi123.stdout:3/119: dwrite d13/f14 [0,4194304] 0 2024-03-30T04:28:00.161 INFO:tasks.workunit.client.0.smithi123.stdout:4/158: creat d2/d4/f3c x:0 0 0 2024-03-30T04:28:00.161 INFO:tasks.workunit.client.0.smithi123.stdout:6/196: dread f2 [0,4194304] 0 2024-03-30T04:28:00.167 INFO:tasks.workunit.client.0.smithi123.stdout:1/188: dwrite db/d1b/f37 [0,4194304] 0 2024-03-30T04:28:00.170 INFO:tasks.workunit.client.0.smithi123.stdout:6/197: mkdir d10/d13/d45 0 2024-03-30T04:28:00.174 INFO:tasks.workunit.client.0.smithi123.stdout:6/198: dread - d10/d13/d1c/f2c zero size 2024-03-30T04:28:00.175 INFO:tasks.workunit.client.0.smithi123.stdout:3/120: dwrite ff [0,4194304] 0 2024-03-30T04:28:00.184 INFO:tasks.workunit.client.0.smithi123.stdout:3/121: stat d13/l1c 0 2024-03-30T04:28:00.185 INFO:tasks.workunit.client.0.smithi123.stdout:6/199: mkdir d10/d13/d45/d46 0 2024-03-30T04:28:00.185 INFO:tasks.workunit.client.0.smithi123.stdout:3/122: stat f12 0 2024-03-30T04:28:00.185 INFO:tasks.workunit.client.0.smithi123.stdout:3/123: mkdir d13/d21 0 2024-03-30T04:28:00.186 INFO:tasks.workunit.client.0.smithi123.stdout:4/159: dread d2/d4/d7/d1a/f30 [4194304,4194304] 0 2024-03-30T04:28:00.186 INFO:tasks.workunit.client.0.smithi123.stdout:2/81: sync 2024-03-30T04:28:00.190 INFO:tasks.workunit.client.0.smithi123.stdout:6/200: dread d10/d13/f17 [0,4194304] 0 2024-03-30T04:28:00.192 INFO:tasks.workunit.client.0.smithi123.stdout:2/82: mkdir d0/d11 0 2024-03-30T04:28:00.192 INFO:tasks.workunit.client.0.smithi123.stdout:6/201: chown f6 403 1 2024-03-30T04:28:00.192 INFO:tasks.workunit.client.0.smithi123.stdout:4/160: symlink d2/d4/l3d 0 2024-03-30T04:28:00.196 INFO:tasks.workunit.client.0.smithi123.stdout:2/83: write d0/f10 [557602,69419] 0 2024-03-30T04:28:00.214 INFO:tasks.workunit.client.0.smithi123.stdout:3/124: dread d13/f14 [0,4194304] 0 2024-03-30T04:28:00.214 INFO:tasks.workunit.client.0.smithi123.stdout:2/84: rename d0/cf to d0/d8/c12 0 2024-03-30T04:28:00.216 INFO:tasks.workunit.client.0.smithi123.stdout:6/202: write d10/d13/d1c/f1e [1539244,74554] 0 2024-03-30T04:28:00.217 INFO:tasks.workunit.client.0.smithi123.stdout:3/125: creat d13/d21/f22 x:0 0 0 2024-03-30T04:28:00.219 INFO:tasks.workunit.client.0.smithi123.stdout:3/126: read fe [2474750,94127] 0 2024-03-30T04:28:00.219 INFO:tasks.workunit.client.0.smithi123.stdout:6/203: dread - d10/d13/f39 zero size 2024-03-30T04:28:00.220 INFO:tasks.workunit.client.0.smithi123.stdout:4/161: dread d2/d4/fa [0,4194304] 0 2024-03-30T04:28:00.221 INFO:tasks.workunit.client.0.smithi123.stdout:3/127: write f12 [4708727,44538] 0 2024-03-30T04:28:00.224 INFO:tasks.workunit.client.0.smithi123.stdout:4/162: creat d2/d2e/f3e x:0 0 0 2024-03-30T04:28:00.224 INFO:tasks.workunit.client.0.smithi123.stdout:6/204: truncate d10/d20/f2b 721253 0 2024-03-30T04:28:00.224 INFO:tasks.workunit.client.0.smithi123.stdout:3/128: truncate d13/d21/f22 266438 0 2024-03-30T04:28:00.225 INFO:tasks.workunit.client.0.smithi123.stdout:2/85: getdents d0/d8 0 2024-03-30T04:28:00.225 INFO:tasks.workunit.client.0.smithi123.stdout:6/205: stat d10/d20/f22 0 2024-03-30T04:28:00.227 INFO:tasks.workunit.client.0.smithi123.stdout:4/163: write d2/d4/f2b [305842,64497] 0 2024-03-30T04:28:00.229 INFO:tasks.workunit.client.0.smithi123.stdout:2/86: mknod d0/d8/c13 0 2024-03-30T04:28:00.230 INFO:tasks.workunit.client.0.smithi123.stdout:6/206: creat d10/d13/d1c/d42/f47 x:0 0 0 2024-03-30T04:28:00.232 INFO:tasks.workunit.client.0.smithi123.stdout:3/129: getdents d13 0 2024-03-30T04:28:00.233 INFO:tasks.workunit.client.0.smithi123.stdout:2/87: creat d0/f14 x:0 0 0 2024-03-30T04:28:00.234 INFO:tasks.workunit.client.0.smithi123.stdout:4/164: creat d2/d34/f3f x:0 0 0 2024-03-30T04:28:00.235 INFO:tasks.workunit.client.0.smithi123.stdout:3/130: write fe [3414822,123279] 0 2024-03-30T04:28:00.236 INFO:tasks.workunit.client.0.smithi123.stdout:2/88: symlink d0/l15 0 2024-03-30T04:28:00.237 INFO:tasks.workunit.client.0.smithi123.stdout:6/207: read d10/f26 [88556,43574] 0 2024-03-30T04:28:00.240 INFO:tasks.workunit.client.0.smithi123.stdout:6/208: write d10/f38 [776716,4353] 0 2024-03-30T04:28:00.243 INFO:tasks.workunit.client.0.smithi123.stdout:6/209: read d10/d20/f3b [2583951,17934] 0 2024-03-30T04:28:00.244 INFO:tasks.workunit.client.0.smithi123.stdout:6/210: fdatasync d10/f1b 0 2024-03-30T04:28:00.248 INFO:tasks.workunit.client.0.smithi123.stdout:2/89: rmdir d0/d11 0 2024-03-30T04:28:00.252 INFO:tasks.workunit.client.0.smithi123.stdout:2/90: mkdir d0/d8/d16 0 2024-03-30T04:28:00.257 INFO:tasks.workunit.client.0.smithi123.stdout:2/91: creat d0/d8/f17 x:0 0 0 2024-03-30T04:28:00.257 INFO:tasks.workunit.client.0.smithi123.stdout:4/165: dwrite d2/d4/d7/fb [0,4194304] 0 2024-03-30T04:28:00.258 INFO:tasks.workunit.client.0.smithi123.stdout:3/131: dread f11 [4194304,4194304] 0 2024-03-30T04:28:00.260 INFO:tasks.workunit.client.0.smithi123.stdout:2/92: write d0/f14 [179418,64265] 0 2024-03-30T04:28:00.262 INFO:tasks.workunit.client.0.smithi123.stdout:4/166: mknod d2/c40 0 2024-03-30T04:28:00.263 INFO:tasks.workunit.client.0.smithi123.stdout:3/132: unlink d13/l1c 0 2024-03-30T04:28:00.267 INFO:tasks.workunit.client.0.smithi123.stdout:3/133: mkdir d13/d21/d23 0 2024-03-30T04:28:00.269 INFO:tasks.workunit.client.0.smithi123.stdout:4/167: mknod d2/d4/d2c/c41 0 2024-03-30T04:28:00.269 INFO:tasks.workunit.client.0.smithi123.stdout:6/211: dwrite d10/d13/d1c/f1e [0,4194304] 0 2024-03-30T04:28:00.270 INFO:tasks.workunit.client.0.smithi123.stdout:3/134: mkdir d13/d24 0 2024-03-30T04:28:00.271 INFO:tasks.workunit.client.0.smithi123.stdout:3/135: readlink d13/l17 0 2024-03-30T04:28:00.272 INFO:tasks.workunit.client.0.smithi123.stdout:6/212: truncate f2 1397521 0 2024-03-30T04:28:00.275 INFO:tasks.workunit.client.0.smithi123.stdout:4/168: rename d2/d2e to d2/d4/d2c/d42 0 2024-03-30T04:28:00.277 INFO:tasks.workunit.client.0.smithi123.stdout:3/136: creat d13/d21/d23/f25 x:0 0 0 2024-03-30T04:28:00.277 INFO:tasks.workunit.client.0.smithi123.stdout:6/213: truncate d10/d20/f36 694271 0 2024-03-30T04:28:00.278 INFO:tasks.workunit.client.0.smithi123.stdout:4/169: chown d2/d4/f3b 61 1 2024-03-30T04:28:00.287 INFO:tasks.workunit.client.0.smithi123.stdout:2/93: dwrite d0/f14 [0,4194304] 0 2024-03-30T04:28:00.297 INFO:tasks.workunit.client.0.smithi123.stdout:3/137: dread f11 [0,4194304] 0 2024-03-30T04:28:00.297 INFO:tasks.workunit.client.0.smithi123.stdout:2/94: creat d0/d8/d16/f18 x:0 0 0 2024-03-30T04:28:00.303 INFO:tasks.workunit.client.0.smithi123.stdout:2/95: dread - d0/d8/d16/f18 zero size 2024-03-30T04:28:00.305 INFO:tasks.workunit.client.0.smithi123.stdout:3/138: symlink d13/d21/d23/l26 0 2024-03-30T04:28:00.309 INFO:tasks.workunit.client.0.smithi123.stdout:3/139: readlink d13/d21/d23/l26 0 2024-03-30T04:28:00.309 INFO:tasks.workunit.client.0.smithi123.stdout:6/214: dwrite d10/d13/d1c/d42/f47 [0,4194304] 0 2024-03-30T04:28:00.310 INFO:tasks.workunit.client.0.smithi123.stdout:2/96: mknod d0/d8/d16/c19 0 2024-03-30T04:28:00.312 INFO:tasks.workunit.client.0.smithi123.stdout:3/140: rename d13 to d13/d24/d27 22 2024-03-30T04:28:00.315 INFO:tasks.workunit.client.0.smithi123.stdout:4/170: dwrite d2/d4/f16 [4194304,4194304] 0 2024-03-30T04:28:00.318 INFO:tasks.workunit.client.0.smithi123.stdout:2/97: write d0/d8/fa [1328007,15104] 0 2024-03-30T04:28:00.326 INFO:tasks.workunit.client.0.smithi123.stdout:2/98: mkdir d0/d8/d1a 0 2024-03-30T04:28:00.330 INFO:tasks.workunit.client.0.smithi123.stdout:2/99: creat d0/d8/f1b x:0 0 0 2024-03-30T04:28:00.332 INFO:tasks.workunit.client.0.smithi123.stdout:6/215: truncate d10/d13/d1c/f2a 4568118 0 2024-03-30T04:28:00.332 INFO:tasks.workunit.client.0.smithi123.stdout:9/131: sync 2024-03-30T04:28:00.334 INFO:tasks.workunit.client.0.smithi123.stdout:2/100: chown d0/d8/f17 284 1 2024-03-30T04:28:00.336 INFO:tasks.workunit.client.0.smithi123.stdout:9/132: unlink d6/f1b 0 2024-03-30T04:28:00.338 INFO:tasks.workunit.client.0.smithi123.stdout:4/171: dwrite d2/d4/d7/fc [0,4194304] 0 2024-03-30T04:28:00.342 INFO:tasks.workunit.client.0.smithi123.stdout:6/216: creat d10/f48 x:0 0 0 2024-03-30T04:28:00.348 INFO:tasks.workunit.client.0.smithi123.stdout:3/141: dwrite d13/f14 [0,4194304] 0 2024-03-30T04:28:00.349 INFO:tasks.workunit.client.0.smithi123.stdout:4/172: unlink d2/l29 0 2024-03-30T04:28:00.350 INFO:tasks.workunit.client.0.smithi123.stdout:3/142: readlink l2 0 2024-03-30T04:28:00.352 INFO:tasks.workunit.client.0.smithi123.stdout:3/143: write d13/f1b [1386141,116612] 0 2024-03-30T04:28:00.354 INFO:tasks.workunit.client.0.smithi123.stdout:3/144: stat f12 0 2024-03-30T04:28:00.358 INFO:tasks.workunit.client.0.smithi123.stdout:4/173: write d2/d4/d7/d1a/f1f [5154550,89245] 0 2024-03-30T04:28:00.359 INFO:tasks.workunit.client.0.smithi123.stdout:3/145: mknod d13/d24/c28 0 2024-03-30T04:28:00.363 INFO:tasks.workunit.client.0.smithi123.stdout:3/146: write d13/f14 [999607,102358] 0 2024-03-30T04:28:00.367 INFO:tasks.workunit.client.0.smithi123.stdout:4/174: unlink d2/d4/d7/l18 0 2024-03-30T04:28:00.376 INFO:tasks.workunit.client.0.smithi123.stdout:2/101: dwrite d0/d8/f17 [0,4194304] 0 2024-03-30T04:28:00.378 INFO:tasks.workunit.client.0.smithi123.stdout:2/102: fdatasync d0/d8/f1b 0 2024-03-30T04:28:00.379 INFO:tasks.workunit.client.0.smithi123.stdout:4/175: mknod d2/d4/d2c/d42/d35/c43 0 2024-03-30T04:28:00.382 INFO:tasks.workunit.client.0.smithi123.stdout:9/133: dwrite d6/d1f/f29 [0,4194304] 0 2024-03-30T04:28:00.384 INFO:tasks.workunit.client.0.smithi123.stdout:2/103: dread - d0/d8/f1b zero size 2024-03-30T04:28:00.388 INFO:tasks.workunit.client.0.smithi123.stdout:2/104: stat d0/lb 0 2024-03-30T04:28:00.389 INFO:tasks.workunit.client.0.smithi123.stdout:9/134: creat d6/d1f/f34 x:0 0 0 2024-03-30T04:28:00.392 INFO:tasks.workunit.client.0.smithi123.stdout:4/176: write d2/d4/f2d [1564719,113842] 0 2024-03-30T04:28:00.400 INFO:tasks.workunit.client.0.smithi123.stdout:3/147: dwrite f11 [4194304,4194304] 0 2024-03-30T04:28:00.401 INFO:tasks.workunit.client.0.smithi123.stdout:3/148: stat d13/d21/d23/f25 0 2024-03-30T04:28:00.405 INFO:tasks.workunit.client.0.smithi123.stdout:5/120: sync 2024-03-30T04:28:00.405 INFO:tasks.workunit.client.0.smithi123.stdout:1/189: sync 2024-03-30T04:28:00.405 INFO:tasks.workunit.client.0.smithi123.stdout:7/56: sync 2024-03-30T04:28:00.405 INFO:tasks.workunit.client.0.smithi123.stdout:0/97: sync 2024-03-30T04:28:00.405 INFO:tasks.workunit.client.0.smithi123.stdout:8/134: sync 2024-03-30T04:28:00.405 INFO:tasks.workunit.client.0.smithi123.stdout:2/105: rename d0/d8/d1a to d0/d8/d1c 0 2024-03-30T04:28:00.407 INFO:tasks.workunit.client.0.smithi123.stdout:5/121: fsync fb 0 2024-03-30T04:28:00.410 INFO:tasks.workunit.client.0.smithi123.stdout:3/149: mknod d13/d21/d23/c29 0 2024-03-30T04:28:00.420 INFO:tasks.workunit.client.0.smithi123.stdout:5/122: write fb [2028617,7887] 0 2024-03-30T04:28:00.424 INFO:tasks.workunit.client.0.smithi123.stdout:2/106: write d0/f14 [1237832,51974] 0 2024-03-30T04:28:00.425 INFO:tasks.workunit.client.0.smithi123.stdout:8/135: rename l7 to da/dc/l1f 0 2024-03-30T04:28:00.426 INFO:tasks.workunit.client.0.smithi123.stdout:7/57: chown d2/dc/c10 46684 1 2024-03-30T04:28:00.435 INFO:tasks.workunit.client.0.smithi123.stdout:9/135: dwrite f1 [0,4194304] 0 2024-03-30T04:28:00.435 INFO:tasks.workunit.client.0.smithi123.stdout:7/58: read - d2/f13 zero size 2024-03-30T04:28:00.440 INFO:tasks.workunit.client.0.smithi123.stdout:1/190: rename db/d1b/d1c to db/dd/d42/d46 0 2024-03-30T04:28:00.445 INFO:tasks.workunit.client.0.smithi123.stdout:9/136: fsync d6/fd 0 2024-03-30T04:28:00.445 INFO:tasks.workunit.client.0.smithi123.stdout:0/98: dwrite f5 [0,4194304] 0 2024-03-30T04:28:00.447 INFO:tasks.workunit.client.0.smithi123.stdout:6/217: dread d10/f27 [0,4194304] 0 2024-03-30T04:28:00.447 INFO:tasks.workunit.client.0.smithi123.stdout:2/107: write d0/d8/d16/f18 [534269,29644] 0 2024-03-30T04:28:00.456 INFO:tasks.workunit.client.0.smithi123.stdout:8/136: rename f9 to da/dc/f20 0 2024-03-30T04:28:00.461 INFO:tasks.workunit.client.0.smithi123.stdout:0/99: chown d6/fe 88804 1 2024-03-30T04:28:00.462 INFO:tasks.workunit.client.0.smithi123.stdout:4/177: dwrite d2/f1c [0,4194304] 0 2024-03-30T04:28:00.465 INFO:tasks.workunit.client.0.smithi123.stdout:1/191: rmdir db/dd/d42 39 2024-03-30T04:28:00.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:00 smithi123 ceph-mon[35245]: pgmap v33: 65 pgs: 65 active+clean; 282 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.8 MiB/s rd, 13 MiB/s wr, 260 op/s 2024-03-30T04:28:00.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:00 smithi123 ceph-mon[35245]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:28:00.467 INFO:tasks.workunit.client.0.smithi123.stdout:3/150: dwrite d13/f1d [0,4194304] 0 2024-03-30T04:28:00.467 INFO:tasks.workunit.client.0.smithi123.stdout:6/218: read d10/d13/d1c/f1e [1806859,1112] 0 2024-03-30T04:28:00.469 INFO:tasks.workunit.client.0.smithi123.stdout:0/100: chown d6/fe 3609763 1 2024-03-30T04:28:00.471 INFO:tasks.workunit.client.0.smithi123.stdout:3/151: fsync f11 0 2024-03-30T04:28:00.478 INFO:tasks.workunit.client.0.smithi123.stdout:0/101: dread - d6/dd/f14 zero size 2024-03-30T04:28:00.484 INFO:tasks.workunit.client.0.smithi123.stdout:6/219: dread - d10/d20/f40 zero size 2024-03-30T04:28:00.490 INFO:tasks.workunit.client.0.smithi123.stdout:8/137: dread da/f11 [0,4194304] 0 2024-03-30T04:28:00.503 INFO:tasks.workunit.client.0.smithi123.stdout:7/59: dwrite d2/f13 [0,4194304] 0 2024-03-30T04:28:00.504 INFO:tasks.workunit.client.0.smithi123.stdout:1/192: read db/dd/d42/d46/f2b [309163,25849] 0 2024-03-30T04:28:00.506 INFO:tasks.workunit.client.0.smithi123.stdout:6/220: truncate d10/f3a 4263551 0 2024-03-30T04:28:00.513 INFO:tasks.workunit.client.0.smithi123.stdout:5/123: dwrite f2 [4194304,4194304] 0 2024-03-30T04:28:00.521 INFO:tasks.workunit.client.0.smithi123.stdout:8/138: truncate da/dc/d14/f1d 1154030 0 2024-03-30T04:28:00.522 INFO:tasks.workunit.client.0.smithi123.stdout:7/60: unlink d2/da/c18 0 2024-03-30T04:28:00.523 INFO:tasks.workunit.client.0.smithi123.stdout:2/108: dwrite d0/d8/d16/f18 [0,4194304] 0 2024-03-30T04:28:00.525 INFO:tasks.workunit.client.0.smithi123.stdout:1/193: dread db/d1b/f2a [0,4194304] 0 2024-03-30T04:28:00.528 INFO:tasks.workunit.client.0.smithi123.stdout:5/124: creat dd/d19/f22 x:0 0 0 2024-03-30T04:28:00.532 INFO:tasks.workunit.client.0.smithi123.stdout:4/178: dwrite d2/d4/d7/fc [0,4194304] 0 2024-03-30T04:28:00.533 INFO:tasks.workunit.client.0.smithi123.stdout:7/61: symlink d2/l1a 0 2024-03-30T04:28:00.536 INFO:tasks.workunit.client.0.smithi123.stdout:1/194: write db/d1b/f1d [2746199,63371] 0 2024-03-30T04:28:00.539 INFO:tasks.workunit.client.0.smithi123.stdout:3/152: dread d13/f1b [0,4194304] 0 2024-03-30T04:28:00.541 INFO:tasks.workunit.client.0.smithi123.stdout:3/153: stat d13 0 2024-03-30T04:28:00.541 INFO:tasks.workunit.client.0.smithi123.stdout:4/179: write d2/f20 [3167757,103437] 0 2024-03-30T04:28:00.541 INFO:tasks.workunit.client.0.smithi123.stdout:5/125: symlink dd/l23 0 2024-03-30T04:28:00.548 INFO:tasks.workunit.client.0.smithi123.stdout:7/62: creat d2/f1b x:0 0 0 2024-03-30T04:28:00.549 INFO:tasks.workunit.client.0.smithi123.stdout:8/139: dread f6 [0,4194304] 0 2024-03-30T04:28:00.552 INFO:tasks.workunit.client.0.smithi123.stdout:2/109: link d0/f10 d0/f1d 0 2024-03-30T04:28:00.552 INFO:tasks.workunit.client.0.smithi123.stdout:1/195: stat db/fe 0 2024-03-30T04:28:00.561 INFO:tasks.workunit.client.0.smithi123.stdout:3/154: creat d13/d21/f2a x:0 0 0 2024-03-30T04:28:00.561 INFO:tasks.workunit.client.0.smithi123.stdout:8/140: rmdir da 39 2024-03-30T04:28:00.569 INFO:tasks.workunit.client.0.smithi123.stdout:8/141: dread - da/dc/d14/f15 zero size 2024-03-30T04:28:00.580 INFO:tasks.workunit.client.0.smithi123.stdout:6/221: write d10/d20/f3b [1732771,90465] 0 2024-03-30T04:28:00.584 INFO:tasks.workunit.client.0.smithi123.stdout:3/155: dread fe [0,4194304] 0 2024-03-30T04:28:00.584 INFO:tasks.workunit.client.0.smithi123.stdout:2/110: truncate d0/f1d 91337 0 2024-03-30T04:28:00.587 INFO:tasks.workunit.client.0.smithi123.stdout:8/142: dread da/fe [0,4194304] 0 2024-03-30T04:28:00.587 INFO:tasks.workunit.client.0.smithi123.stdout:1/196: dwrite db/d1b/f37 [0,4194304] 0 2024-03-30T04:28:00.591 INFO:tasks.workunit.client.0.smithi123.stdout:5/126: dread f2 [0,4194304] 0 2024-03-30T04:28:00.593 INFO:tasks.workunit.client.0.smithi123.stdout:7/63: dwrite d2/f1b [0,4194304] 0 2024-03-30T04:28:00.594 INFO:tasks.workunit.client.0.smithi123.stdout:2/111: creat d0/d8/d16/f1e x:0 0 0 2024-03-30T04:28:00.595 INFO:tasks.workunit.client.0.smithi123.stdout:6/222: mknod d10/d13/d1c/d42/c49 0 2024-03-30T04:28:00.598 INFO:tasks.workunit.client.0.smithi123.stdout:1/197: creat db/dd/d42/d46/f47 x:0 0 0 2024-03-30T04:28:00.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:00 smithi135 ceph-mon[44364]: pgmap v33: 65 pgs: 65 active+clean; 282 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.8 MiB/s rd, 13 MiB/s wr, 260 op/s 2024-03-30T04:28:00.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:00 smithi135 ceph-mon[44364]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:28:00.601 INFO:tasks.workunit.client.0.smithi123.stdout:8/143: write da/f11 [751958,63379] 0 2024-03-30T04:28:00.602 INFO:tasks.workunit.client.0.smithi123.stdout:3/156: rename d13/d21/f22 to d13/d21/d23/f2b 0 2024-03-30T04:28:00.604 INFO:tasks.workunit.client.0.smithi123.stdout:1/198: chown db/dd/d42/d46/d39/c3f 40778176 1 2024-03-30T04:28:00.606 INFO:tasks.workunit.client.0.smithi123.stdout:5/127: mknod dd/c24 0 2024-03-30T04:28:00.608 INFO:tasks.workunit.client.0.smithi123.stdout:4/180: dwrite d2/d4/f8 [0,4194304] 0 2024-03-30T04:28:00.610 INFO:tasks.workunit.client.0.smithi123.stdout:4/181: read - d2/d34/f3f zero size 2024-03-30T04:28:00.614 INFO:tasks.workunit.client.0.smithi123.stdout:3/157: symlink d13/l2c 0 2024-03-30T04:28:00.615 INFO:tasks.workunit.client.0.smithi123.stdout:8/144: mkdir da/dc/d14/d18/d21 0 2024-03-30T04:28:00.617 INFO:tasks.workunit.client.0.smithi123.stdout:5/128: rename dd/l13 to dd/d19/l25 0 2024-03-30T04:28:00.620 INFO:tasks.workunit.client.0.smithi123.stdout:8/145: write da/fb [3078998,2311] 0 2024-03-30T04:28:00.621 INFO:tasks.workunit.client.0.smithi123.stdout:4/182: write d2/d4/fa [2479693,63542] 0 2024-03-30T04:28:00.622 INFO:tasks.workunit.client.0.smithi123.stdout:1/199: creat db/dd/f48 x:0 0 0 2024-03-30T04:28:00.622 INFO:tasks.workunit.client.0.smithi123.stdout:5/129: creat dd/f26 x:0 0 0 2024-03-30T04:28:00.629 INFO:tasks.workunit.client.0.smithi123.stdout:3/158: link d13/c15 d13/d21/d23/c2d 0 2024-03-30T04:28:00.632 INFO:tasks.workunit.client.0.smithi123.stdout:8/146: mknod da/c22 0 2024-03-30T04:28:00.633 INFO:tasks.workunit.client.0.smithi123.stdout:5/130: write dd/f16 [1268351,112837] 0 2024-03-30T04:28:00.634 INFO:tasks.workunit.client.0.smithi123.stdout:4/183: symlink d2/d4/d2c/d42/l44 0 2024-03-30T04:28:00.634 INFO:tasks.workunit.client.0.smithi123.stdout:6/223: dwrite d10/f1b [0,4194304] 0 2024-03-30T04:28:00.637 INFO:tasks.workunit.client.0.smithi123.stdout:8/147: read da/dc/f20 [7470864,85778] 0 2024-03-30T04:28:00.638 INFO:tasks.workunit.client.0.smithi123.stdout:5/131: dread - dd/f15 zero size 2024-03-30T04:28:00.641 INFO:tasks.workunit.client.0.smithi123.stdout:4/184: chown d2/d4/d7/d1a/f30 6803535 1 2024-03-30T04:28:00.643 INFO:tasks.workunit.client.0.smithi123.stdout:7/64: dwrite d2/dc/ff [4194304,4194304] 0 2024-03-30T04:28:00.644 INFO:tasks.workunit.client.0.smithi123.stdout:8/148: mknod da/dc/c23 0 2024-03-30T04:28:00.649 INFO:tasks.workunit.client.0.smithi123.stdout:7/65: mkdir d2/dc/d1c 0 2024-03-30T04:28:00.651 INFO:tasks.workunit.client.0.smithi123.stdout:8/149: mkdir da/dc/d14/d24 0 2024-03-30T04:28:00.652 INFO:tasks.workunit.client.0.smithi123.stdout:4/185: creat d2/f45 x:0 0 0 2024-03-30T04:28:00.655 INFO:tasks.workunit.client.0.smithi123.stdout:7/66: write d2/dc/ff [8990113,70959] 0 2024-03-30T04:28:00.659 INFO:tasks.workunit.client.0.smithi123.stdout:1/200: dwrite db/dd/d42/d46/f29 [0,4194304] 0 2024-03-30T04:28:00.659 INFO:tasks.workunit.client.0.smithi123.stdout:4/186: mknod d2/c46 0 2024-03-30T04:28:00.662 INFO:tasks.workunit.client.0.smithi123.stdout:8/150: dread da/fe [0,4194304] 0 2024-03-30T04:28:00.663 INFO:tasks.workunit.client.0.smithi123.stdout:7/67: link d2/f1b d2/dc/d1c/f1d 0 2024-03-30T04:28:00.667 INFO:tasks.workunit.client.0.smithi123.stdout:4/187: write d2/d4/fa [3773410,107773] 0 2024-03-30T04:28:00.667 INFO:tasks.workunit.client.0.smithi123.stdout:5/132: dwrite dd/f16 [0,4194304] 0 2024-03-30T04:28:00.673 INFO:tasks.workunit.client.0.smithi123.stdout:1/201: rename db/d1b/l30 to db/dd/l49 0 2024-03-30T04:28:00.674 INFO:tasks.workunit.client.0.smithi123.stdout:4/188: rename d2 to d2/d4/d7/d47 22 2024-03-30T04:28:00.675 INFO:tasks.workunit.client.0.smithi123.stdout:7/68: creat d2/da/d11/f1e x:0 0 0 2024-03-30T04:28:00.676 INFO:tasks.workunit.client.0.smithi123.stdout:4/189: chown d2/f45 15168017 1 2024-03-30T04:28:00.677 INFO:tasks.workunit.client.0.smithi123.stdout:7/69: write d2/dc/f17 [744173,83073] 0 2024-03-30T04:28:00.681 INFO:tasks.workunit.client.0.smithi123.stdout:1/202: link db/d1b/c27 db/dd/d42/d46/d39/c4a 0 2024-03-30T04:28:00.681 INFO:tasks.workunit.client.0.smithi123.stdout:7/70: creat d2/da/d11/f1f x:0 0 0 2024-03-30T04:28:00.683 INFO:tasks.workunit.client.0.smithi123.stdout:4/190: symlink d2/d4/d7/d1a/l48 0 2024-03-30T04:28:00.683 INFO:tasks.workunit.client.0.smithi123.stdout:1/203: fdatasync f8 0 2024-03-30T04:28:00.688 INFO:tasks.workunit.client.0.smithi123.stdout:8/151: dwrite f8 [0,4194304] 0 2024-03-30T04:28:00.690 INFO:tasks.workunit.client.0.smithi123.stdout:4/191: link d2/d4/f8 d2/d4/f49 0 2024-03-30T04:28:00.692 INFO:tasks.workunit.client.0.smithi123.stdout:8/152: dread - da/dc/d14/f15 zero size 2024-03-30T04:28:00.704 INFO:tasks.workunit.client.0.smithi123.stdout:7/71: dwrite d2/dc/f17 [0,4194304] 0 2024-03-30T04:28:00.710 INFO:tasks.workunit.client.0.smithi123.stdout:7/72: chown d2/da/d11 12469 1 2024-03-30T04:28:00.711 INFO:tasks.workunit.client.0.smithi123.stdout:1/204: dwrite db/f24 [0,4194304] 0 2024-03-30T04:28:00.713 INFO:tasks.workunit.client.0.smithi123.stdout:1/205: write f8 [4423009,2760] 0 2024-03-30T04:28:00.714 INFO:tasks.workunit.client.0.smithi123.stdout:4/192: dwrite d2/d4/f3b [0,4194304] 0 2024-03-30T04:28:00.720 INFO:tasks.workunit.client.0.smithi123.stdout:1/206: getdents db 0 2024-03-30T04:28:00.723 INFO:tasks.workunit.client.0.smithi123.stdout:1/207: creat db/d1b/f4b x:0 0 0 2024-03-30T04:28:00.734 INFO:tasks.workunit.client.0.smithi123.stdout:4/193: dwrite d2/d4/d7/d1a/f1f [4194304,4194304] 0 2024-03-30T04:28:00.738 INFO:tasks.workunit.client.0.smithi123.stdout:4/194: creat d2/d4/d2c/d42/f4a x:0 0 0 2024-03-30T04:28:00.740 INFO:tasks.workunit.client.0.smithi123.stdout:4/195: chown d2/d4/d7/l1d 14073 1 2024-03-30T04:28:00.742 INFO:tasks.workunit.client.0.smithi123.stdout:1/208: dwrite db/d1b/f4b [0,4194304] 0 2024-03-30T04:28:00.748 INFO:tasks.workunit.client.0.smithi123.stdout:4/196: rename d2/d4/d7/f19 to d2/d4/d7/f4b 0 2024-03-30T04:28:00.750 INFO:tasks.workunit.client.0.smithi123.stdout:4/197: read - d2/d4/d2c/d42/f38 zero size 2024-03-30T04:28:00.753 INFO:tasks.workunit.client.0.smithi123.stdout:4/198: write d2/d4/f16 [6123340,15219] 0 2024-03-30T04:28:00.755 INFO:tasks.workunit.client.0.smithi123.stdout:4/199: creat d2/d34/f4c x:0 0 0 2024-03-30T04:28:00.757 INFO:tasks.workunit.client.0.smithi123.stdout:4/200: mknod d2/d34/c4d 0 2024-03-30T04:28:00.763 INFO:tasks.workunit.client.0.smithi123.stdout:4/201: rename d2/d4/f3b to d2/d4/d7/f4e 0 2024-03-30T04:28:00.765 INFO:tasks.workunit.client.0.smithi123.stdout:4/202: write d2/d4/f22 [176627,83511] 0 2024-03-30T04:28:00.790 INFO:tasks.workunit.client.0.smithi123.stdout:9/137: sync 2024-03-30T04:28:00.790 INFO:tasks.workunit.client.0.smithi123.stdout:0/102: sync 2024-03-30T04:28:00.791 INFO:tasks.workunit.client.0.smithi123.stdout:9/138: write d6/d1f/f2d [604630,21172] 0 2024-03-30T04:28:00.793 INFO:tasks.workunit.client.0.smithi123.stdout:9/139: unlink d6/f9 0 2024-03-30T04:28:00.802 INFO:tasks.workunit.client.0.smithi123.stdout:9/140: dread d6/f15 [0,4194304] 0 2024-03-30T04:28:00.804 INFO:tasks.workunit.client.0.smithi123.stdout:9/141: mknod d6/d23/c35 0 2024-03-30T04:28:00.909 INFO:tasks.workunit.client.0.smithi123.stdout:2/112: rename d0/d8 to d0/d1f 0 2024-03-30T04:28:00.912 INFO:tasks.workunit.client.0.smithi123.stdout:3/159: getdents d13/d21 0 2024-03-30T04:28:00.914 INFO:tasks.workunit.client.0.smithi123.stdout:3/160: chown d13/c1f 0 1 2024-03-30T04:28:00.923 INFO:tasks.workunit.client.0.smithi123.stdout:3/161: creat d13/d21/d23/f2e x:0 0 0 2024-03-30T04:28:00.929 INFO:tasks.workunit.client.0.smithi123.stdout:3/162: mkdir d13/d21/d2f 0 2024-03-30T04:28:00.931 INFO:tasks.workunit.client.0.smithi123.stdout:5/133: dread fb [0,4194304] 0 2024-03-30T04:28:00.939 INFO:tasks.workunit.client.0.smithi123.stdout:5/134: creat dd/d19/d1f/f27 x:0 0 0 2024-03-30T04:28:00.945 INFO:tasks.workunit.client.0.smithi123.stdout:5/135: unlink lc 0 2024-03-30T04:28:00.947 INFO:tasks.workunit.client.0.smithi123.stdout:6/224: truncate d10/d13/d1c/d42/f47 178202 0 2024-03-30T04:28:00.951 INFO:tasks.workunit.client.0.smithi123.stdout:3/163: dwrite ff [0,4194304] 0 2024-03-30T04:28:00.955 INFO:tasks.workunit.client.0.smithi123.stdout:4/203: getdents d2 0 2024-03-30T04:28:00.957 INFO:tasks.workunit.client.0.smithi123.stdout:4/204: truncate d2/f45 606300 0 2024-03-30T04:28:00.959 INFO:tasks.workunit.client.0.smithi123.stdout:5/136: link dd/f15 dd/d19/f28 0 2024-03-30T04:28:00.959 INFO:tasks.workunit.client.0.smithi123.stdout:4/205: readlink d2/d4/d7/l1d 0 2024-03-30T04:28:00.961 INFO:tasks.workunit.client.0.smithi123.stdout:5/137: chown dd/c18 2 1 2024-03-30T04:28:00.962 INFO:tasks.workunit.client.0.smithi123.stdout:5/138: dread - dd/f1c zero size 2024-03-30T04:28:00.964 INFO:tasks.workunit.client.0.smithi123.stdout:6/225: symlink d10/d13/l4a 0 2024-03-30T04:28:00.965 INFO:tasks.workunit.client.0.smithi123.stdout:1/209: rmdir db/dd 39 2024-03-30T04:28:00.966 INFO:tasks.workunit.client.0.smithi123.stdout:7/73: dread d2/f8 [0,4194304] 0 2024-03-30T04:28:00.967 INFO:tasks.workunit.client.0.smithi123.stdout:6/226: dread - d10/d13/d1c/f2c zero size 2024-03-30T04:28:00.969 INFO:tasks.workunit.client.0.smithi123.stdout:7/74: read d2/f13 [532010,91104] 0 2024-03-30T04:28:00.972 INFO:tasks.workunit.client.0.smithi123.stdout:4/206: creat d2/d26/f4f x:0 0 0 2024-03-30T04:28:00.972 INFO:tasks.workunit.client.0.smithi123.stdout:8/153: dread da/f11 [0,4194304] 0 2024-03-30T04:28:00.973 INFO:tasks.workunit.client.0.smithi123.stdout:5/139: symlink dd/l29 0 2024-03-30T04:28:00.975 INFO:tasks.workunit.client.0.smithi123.stdout:8/154: dread - da/dc/f16 zero size 2024-03-30T04:28:00.979 INFO:tasks.workunit.client.0.smithi123.stdout:8/155: chown da/l1c 2 1 2024-03-30T04:28:00.982 INFO:tasks.workunit.client.0.smithi123.stdout:8/156: chown da/dc/d14/d18/f1e 57 1 2024-03-30T04:28:00.985 INFO:tasks.workunit.client.0.smithi123.stdout:4/207: mknod d2/c50 0 2024-03-30T04:28:00.986 INFO:tasks.workunit.client.0.smithi123.stdout:3/164: dwrite d13/f1b [4194304,4194304] 0 2024-03-30T04:28:00.988 INFO:tasks.workunit.client.0.smithi123.stdout:1/210: truncate db/d1b/f2a 900988 0 2024-03-30T04:28:00.992 INFO:tasks.workunit.client.0.smithi123.stdout:9/142: fdatasync d6/d1f/f2d 0 2024-03-30T04:28:01.000 INFO:tasks.workunit.client.0.smithi123.stdout:8/157: mknod da/dc/c25 0 2024-03-30T04:28:01.015 INFO:tasks.workunit.client.0.smithi123.stdout:6/227: rmdir d10/d20/d41 0 2024-03-30T04:28:01.015 INFO:tasks.workunit.client.0.smithi123.stdout:1/211: write db/dd/f13 [15364,17334] 0 2024-03-30T04:28:01.017 INFO:tasks.workunit.client.0.smithi123.stdout:5/140: dwrite dd/d19/f28 [0,4194304] 0 2024-03-30T04:28:01.019 INFO:tasks.workunit.client.0.smithi123.stdout:5/141: stat dd/d19/f22 0 2024-03-30T04:28:01.022 INFO:tasks.workunit.client.0.smithi123.stdout:3/165: truncate f11 3838806 0 2024-03-30T04:28:01.027 INFO:tasks.workunit.client.0.smithi123.stdout:5/142: read f7 [602264,5012] 0 2024-03-30T04:28:01.034 INFO:tasks.workunit.client.0.smithi123.stdout:9/143: unlink d6/l1d 0 2024-03-30T04:28:01.038 INFO:tasks.workunit.client.0.smithi123.stdout:2/113: truncate d0/d1f/fa 3792050 0 2024-03-30T04:28:01.039 INFO:tasks.workunit.client.0.smithi123.stdout:2/114: fsync d0/f14 0 2024-03-30T04:28:01.042 INFO:tasks.workunit.client.0.smithi123.stdout:3/166: rmdir d13/d21 39 2024-03-30T04:28:01.044 INFO:tasks.workunit.client.0.smithi123.stdout:3/167: chown d13/c1f 7 1 2024-03-30T04:28:01.049 INFO:tasks.workunit.client.0.smithi123.stdout:5/143: creat dd/d19/d1f/f2a x:0 0 0 2024-03-30T04:28:01.053 INFO:tasks.workunit.client.0.smithi123.stdout:2/115: dread d0/f14 [0,4194304] 0 2024-03-30T04:28:01.054 INFO:tasks.workunit.client.0.smithi123.stdout:6/228: dread d10/d13/d1c/f1f [0,4194304] 0 2024-03-30T04:28:01.054 INFO:tasks.workunit.client.0.smithi123.stdout:2/116: dread - d0/d1f/f1b zero size 2024-03-30T04:28:01.056 INFO:tasks.workunit.client.0.smithi123.stdout:6/229: fsync d10/d20/f2b 0 2024-03-30T04:28:01.061 INFO:tasks.workunit.client.0.smithi123.stdout:3/168: fdatasync d13/d21/d23/f25 0 2024-03-30T04:28:01.063 INFO:tasks.workunit.client.0.smithi123.stdout:3/169: chown d13/l2c 22349 1 2024-03-30T04:28:01.065 INFO:tasks.workunit.client.0.smithi123.stdout:5/144: creat dd/d19/f2b x:0 0 0 2024-03-30T04:28:01.069 INFO:tasks.workunit.client.0.smithi123.stdout:1/212: creat db/dd/f4c x:0 0 0 2024-03-30T04:28:01.076 INFO:tasks.workunit.client.0.smithi123.stdout:5/145: dread dd/f15 [0,4194304] 0 2024-03-30T04:28:01.076 INFO:tasks.workunit.client.0.smithi123.stdout:6/230: unlink d10/d20/c30 0 2024-03-30T04:28:01.084 INFO:tasks.workunit.client.0.smithi123.stdout:1/213: readlink db/lf 0 2024-03-30T04:28:01.085 INFO:tasks.workunit.client.0.smithi123.stdout:1/214: dread - db/dd/d42/d46/f2e zero size 2024-03-30T04:28:01.089 INFO:tasks.workunit.client.0.smithi123.stdout:6/231: chown f2 201010647 1 2024-03-30T04:28:01.089 INFO:tasks.workunit.client.0.smithi123.stdout:8/158: getdents da/dc 0 2024-03-30T04:28:01.091 INFO:tasks.workunit.client.0.smithi123.stdout:9/144: getdents d6/d23 0 2024-03-30T04:28:01.094 INFO:tasks.workunit.client.0.smithi123.stdout:5/146: rename dd/d19/d1f/f2a to dd/d19/f2c 0 2024-03-30T04:28:01.097 INFO:tasks.workunit.client.0.smithi123.stdout:8/159: dread da/dc/f13 [0,4194304] 0 2024-03-30T04:28:01.100 INFO:tasks.workunit.client.0.smithi123.stdout:1/215: rmdir db/d1b 39 2024-03-30T04:28:01.103 INFO:tasks.workunit.client.0.smithi123.stdout:6/232: mknod d10/d13/d1c/c4b 0 2024-03-30T04:28:01.104 INFO:tasks.workunit.client.0.smithi123.stdout:8/160: dread da/dc/f10 [0,4194304] 0 2024-03-30T04:28:01.107 INFO:tasks.workunit.client.0.smithi123.stdout:6/233: chown d10/d13/d1c/l2d 125 1 2024-03-30T04:28:01.110 INFO:tasks.workunit.client.0.smithi123.stdout:6/234: write d10/f48 [223467,53137] 0 2024-03-30T04:28:01.114 INFO:tasks.workunit.client.0.smithi123.stdout:5/147: dwrite dd/f16 [0,4194304] 0 2024-03-30T04:28:01.116 INFO:tasks.workunit.client.0.smithi123.stdout:5/148: truncate f9 5122440 0 2024-03-30T04:28:01.120 INFO:tasks.workunit.client.0.smithi123.stdout:4/208: truncate d2/d4/d7/f4e 294062 0 2024-03-30T04:28:01.127 INFO:tasks.workunit.client.0.smithi123.stdout:8/161: symlink da/dc/l26 0 2024-03-30T04:28:01.130 INFO:tasks.workunit.client.0.smithi123.stdout:6/235: mkdir d10/d13/d1c/d4c 0 2024-03-30T04:28:01.131 INFO:tasks.workunit.client.0.smithi123.stdout:6/236: chown d10/d13/d1c/f1f 9670 1 2024-03-30T04:28:01.132 INFO:tasks.workunit.client.0.smithi123.stdout:3/170: getdents d13/d21 0 2024-03-30T04:28:01.133 INFO:tasks.workunit.client.0.smithi123.stdout:6/237: readlink d10/d13/d1c/l31 0 2024-03-30T04:28:01.136 INFO:tasks.workunit.client.0.smithi123.stdout:5/149: mknod dd/c2d 0 2024-03-30T04:28:01.144 INFO:tasks.workunit.client.0.smithi123.stdout:4/209: dread - d2/d34/f4c zero size 2024-03-30T04:28:01.147 INFO:tasks.workunit.client.0.smithi123.stdout:8/162: write da/f11 [1552729,9562] 0 2024-03-30T04:28:01.153 INFO:tasks.workunit.client.0.smithi123.stdout:3/171: mkdir d13/d21/d30 0 2024-03-30T04:28:01.159 INFO:tasks.workunit.client.0.smithi123.stdout:1/216: creat db/dd/f4d x:0 0 0 2024-03-30T04:28:01.161 INFO:tasks.workunit.client.0.smithi123.stdout:1/217: read - db/dd/f4d zero size 2024-03-30T04:28:01.165 INFO:tasks.workunit.client.0.smithi123.stdout:4/210: write d2/f1c [3327982,128683] 0 2024-03-30T04:28:01.166 INFO:tasks.workunit.client.0.smithi123.stdout:9/145: link d6/d1f/c2c d6/d23/c36 0 2024-03-30T04:28:01.170 INFO:tasks.workunit.client.0.smithi123.stdout:6/238: creat d10/d20/d3f/f4d x:0 0 0 2024-03-30T04:28:01.172 INFO:tasks.workunit.client.0.smithi123.stdout:6/239: read - d10/f29 zero size 2024-03-30T04:28:01.173 INFO:tasks.workunit.client.0.smithi123.stdout:0/103: sync 2024-03-30T04:28:01.174 INFO:tasks.workunit.client.0.smithi123.stdout:3/172: dwrite d13/d21/d23/f25 [0,4194304] 0 2024-03-30T04:28:01.178 INFO:tasks.workunit.client.0.smithi123.stdout:6/240: fdatasync d10/f3a 0 2024-03-30T04:28:01.178 INFO:tasks.workunit.client.0.smithi123.stdout:0/104: fsync d6/dd/f14 0 2024-03-30T04:28:01.179 INFO:tasks.workunit.client.0.smithi123.stdout:8/163: mkdir da/d27 0 2024-03-30T04:28:01.179 INFO:tasks.workunit.client.0.smithi123.stdout:5/150: dwrite dd/f15 [0,4194304] 0 2024-03-30T04:28:01.192 INFO:tasks.workunit.client.0.smithi123.stdout:2/117: dwrite d0/d1f/fa [0,4194304] 0 2024-03-30T04:28:01.203 INFO:tasks.workunit.client.0.smithi123.stdout:1/218: rmdir db/d1b 39 2024-03-30T04:28:01.211 INFO:tasks.workunit.client.0.smithi123.stdout:6/241: dwrite d10/d20/f22 [4194304,4194304] 0 2024-03-30T04:28:01.212 INFO:tasks.workunit.client.0.smithi123.stdout:0/105: dread d6/f9 [0,4194304] 0 2024-03-30T04:28:01.216 INFO:tasks.workunit.client.0.smithi123.stdout:0/106: read - d6/dd/f14 zero size 2024-03-30T04:28:01.216 INFO:tasks.workunit.client.0.smithi123.stdout:4/211: symlink d2/d4/d7/l51 0 2024-03-30T04:28:01.217 INFO:tasks.workunit.client.0.smithi123.stdout:5/151: dwrite dd/f1d [0,4194304] 0 2024-03-30T04:28:01.221 INFO:tasks.workunit.client.0.smithi123.stdout:2/118: mkdir d0/d1f/d1c/d20 0 2024-03-30T04:28:01.228 INFO:tasks.workunit.client.0.smithi123.stdout:9/146: write d6/f10 [1634779,129482] 0 2024-03-30T04:28:01.235 INFO:tasks.workunit.client.0.smithi123.stdout:3/173: unlink d13/d21/d23/c2d 0 2024-03-30T04:28:01.240 INFO:tasks.workunit.client.0.smithi123.stdout:8/164: creat da/dc/d14/d18/d21/f28 x:0 0 0 2024-03-30T04:28:01.240 INFO:tasks.workunit.client.0.smithi123.stdout:4/212: rmdir d2 39 2024-03-30T04:28:01.243 INFO:tasks.workunit.client.0.smithi123.stdout:0/107: dwrite d6/dd/d15/f19 [4194304,4194304] 0 2024-03-30T04:28:01.243 INFO:tasks.workunit.client.0.smithi123.stdout:5/152: mknod dd/d19/c2e 0 2024-03-30T04:28:01.248 INFO:tasks.workunit.client.0.smithi123.stdout:2/119: dwrite d0/d1f/f17 [0,4194304] 0 2024-03-30T04:28:01.251 INFO:tasks.workunit.client.0.smithi123.stdout:2/120: fsync d0/f14 0 2024-03-30T04:28:01.252 INFO:tasks.workunit.client.0.smithi123.stdout:8/165: dread da/f11 [0,4194304] 0 2024-03-30T04:28:01.257 INFO:tasks.workunit.client.0.smithi123.stdout:8/166: dread - da/dc/d14/d18/f1e zero size 2024-03-30T04:28:01.268 INFO:tasks.workunit.client.0.smithi123.stdout:4/213: read d2/d4/ff [1844566,87069] 0 2024-03-30T04:28:01.269 INFO:tasks.workunit.client.0.smithi123.stdout:2/121: dread d0/d1f/fa [0,4194304] 0 2024-03-30T04:28:01.270 INFO:tasks.workunit.client.0.smithi123.stdout:5/153: creat dd/f2f x:0 0 0 2024-03-30T04:28:01.272 INFO:tasks.workunit.client.0.smithi123.stdout:3/174: rename d13/d21 to d13/d24/d31 0 2024-03-30T04:28:01.275 INFO:tasks.workunit.client.0.smithi123.stdout:3/175: write d13/f1b [535724,127605] 0 2024-03-30T04:28:01.278 INFO:tasks.workunit.client.0.smithi123.stdout:3/176: write f12 [4791377,71017] 0 2024-03-30T04:28:01.280 INFO:tasks.workunit.client.0.smithi123.stdout:3/177: write ff [558132,11299] 0 2024-03-30T04:28:01.281 INFO:tasks.workunit.client.0.smithi123.stdout:8/167: dread f6 [0,4194304] 0 2024-03-30T04:28:01.290 INFO:tasks.workunit.client.0.smithi123.stdout:4/214: chown d2/l25 413 1 2024-03-30T04:28:01.294 INFO:tasks.workunit.client.0.smithi123.stdout:2/122: rename d0/d1f/d16/c19 to d0/d1f/c21 0 2024-03-30T04:28:01.299 INFO:tasks.workunit.client.0.smithi123.stdout:5/154: rmdir dd 39 2024-03-30T04:28:01.305 INFO:tasks.workunit.client.0.smithi123.stdout:9/147: creat d6/d1f/f37 x:0 0 0 2024-03-30T04:28:01.308 INFO:tasks.workunit.client.0.smithi123.stdout:2/123: creat d0/d1f/d1c/f22 x:0 0 0 2024-03-30T04:28:01.311 INFO:tasks.workunit.client.0.smithi123.stdout:5/155: dread - dd/f26 zero size 2024-03-30T04:28:01.314 INFO:tasks.workunit.client.0.smithi123.stdout:1/219: getdents db/d1b 0 2024-03-30T04:28:01.314 INFO:tasks.workunit.client.0.smithi123.stdout:8/168: symlink da/d27/l29 0 2024-03-30T04:28:01.314 INFO:tasks.workunit.client.0.smithi123.stdout:8/169: fsync f5 0 2024-03-30T04:28:01.316 INFO:tasks.workunit.client.0.smithi123.stdout:4/215: rmdir d2/d26 39 2024-03-30T04:28:01.317 INFO:tasks.workunit.client.0.smithi123.stdout:0/108: getdents d6/dd/d15 0 2024-03-30T04:28:01.324 INFO:tasks.workunit.client.0.smithi123.stdout:0/109: symlink d6/dd/l1d 0 2024-03-30T04:28:01.329 INFO:tasks.workunit.client.0.smithi123.stdout:9/148: creat d6/f38 x:0 0 0 2024-03-30T04:28:01.332 INFO:tasks.workunit.client.0.smithi123.stdout:4/216: rmdir d2/d4/d2c/d42 39 2024-03-30T04:28:01.338 INFO:tasks.workunit.client.0.smithi123.stdout:1/220: rename db/dd/d42/l35 to db/l4e 0 2024-03-30T04:28:01.346 INFO:tasks.workunit.client.0.smithi123.stdout:5/156: dwrite dd/f15 [0,4194304] 0 2024-03-30T04:28:01.347 INFO:tasks.workunit.client.0.smithi123.stdout:2/124: dwrite d0/d1f/f1b [0,4194304] 0 2024-03-30T04:28:01.349 INFO:tasks.workunit.client.0.smithi123.stdout:4/217: unlink d2/d4/f3c 0 2024-03-30T04:28:01.351 INFO:tasks.workunit.client.0.smithi123.stdout:1/221: write db/f41 [634271,8817] 0 2024-03-30T04:28:01.358 INFO:tasks.workunit.client.0.smithi123.stdout:9/149: getdents d6 0 2024-03-30T04:28:01.360 INFO:tasks.workunit.client.0.smithi123.stdout:3/178: dwrite fe [0,4194304] 0 2024-03-30T04:28:01.360 INFO:tasks.workunit.client.0.smithi123.stdout:5/157: chown dd/d19/f2c 4127072 1 2024-03-30T04:28:01.360 INFO:tasks.workunit.client.0.smithi123.stdout:1/222: fdatasync db/d1b/f28 0 2024-03-30T04:28:01.361 INFO:tasks.workunit.client.0.smithi123.stdout:2/125: dread d0/f1d [0,4194304] 0 2024-03-30T04:28:01.364 INFO:tasks.workunit.client.0.smithi123.stdout:4/218: mknod d2/d34/c52 0 2024-03-30T04:28:01.364 INFO:tasks.workunit.client.0.smithi123.stdout:7/75: sync 2024-03-30T04:28:01.369 INFO:tasks.workunit.client.0.smithi123.stdout:1/223: creat db/dd/d42/f4f x:0 0 0 2024-03-30T04:28:01.369 INFO:tasks.workunit.client.0.smithi123.stdout:0/110: dread d6/f9 [4194304,4194304] 0 2024-03-30T04:28:01.371 INFO:tasks.workunit.client.0.smithi123.stdout:8/170: dwrite da/f12 [0,4194304] 0 2024-03-30T04:28:01.373 INFO:tasks.workunit.client.0.smithi123.stdout:5/158: rename c6 to dd/d19/c30 0 2024-03-30T04:28:01.376 INFO:tasks.workunit.client.0.smithi123.stdout:4/219: symlink d2/d34/l53 0 2024-03-30T04:28:01.377 INFO:tasks.workunit.client.0.smithi123.stdout:2/126: creat d0/d1f/d16/f23 x:0 0 0 2024-03-30T04:28:01.379 INFO:tasks.workunit.client.0.smithi123.stdout:8/171: mknod da/dc/c2a 0 2024-03-30T04:28:01.379 INFO:tasks.workunit.client.0.smithi123.stdout:1/224: mknod db/d1b/c50 0 2024-03-30T04:28:01.383 INFO:tasks.workunit.client.0.smithi123.stdout:5/159: symlink dd/d19/l31 0 2024-03-30T04:28:01.388 INFO:tasks.workunit.client.0.smithi123.stdout:2/127: creat d0/d1f/f24 x:0 0 0 2024-03-30T04:28:01.388 INFO:tasks.workunit.client.0.smithi123.stdout:1/225: truncate db/d1b/f2a 1096022 0 2024-03-30T04:28:01.389 INFO:tasks.workunit.client.0.smithi123.stdout:4/220: chown d2/d4/d7/ld 14 1 2024-03-30T04:28:01.391 INFO:tasks.workunit.client.0.smithi123.stdout:6/242: sync 2024-03-30T04:28:01.391 INFO:tasks.workunit.client.0.smithi123.stdout:3/179: dwrite fd [0,4194304] 0 2024-03-30T04:28:01.392 INFO:tasks.workunit.client.0.smithi123.stdout:2/128: fdatasync d0/d1f/f17 0 2024-03-30T04:28:01.393 INFO:tasks.workunit.client.0.smithi123.stdout:1/226: dread - db/dd/d42/d46/f47 zero size 2024-03-30T04:28:01.395 INFO:tasks.workunit.client.0.smithi123.stdout:0/111: creat d6/dd/d16/f1e x:0 0 0 2024-03-30T04:28:01.396 INFO:tasks.workunit.client.0.smithi123.stdout:5/160: creat dd/f32 x:0 0 0 2024-03-30T04:28:01.398 INFO:tasks.workunit.client.0.smithi123.stdout:2/129: write d0/d1f/d16/f18 [2115083,84827] 0 2024-03-30T04:28:01.402 INFO:tasks.workunit.client.0.smithi123.stdout:0/112: chown d6/fe 19 1 2024-03-30T04:28:01.405 INFO:tasks.workunit.client.0.smithi123.stdout:3/180: write d13/d24/d31/d23/f2e [634226,12169] 0 2024-03-30T04:28:01.405 INFO:tasks.workunit.client.0.smithi123.stdout:8/172: mknod da/dc/d14/d24/c2b 0 2024-03-30T04:28:01.406 INFO:tasks.workunit.client.0.smithi123.stdout:5/161: mknod dd/d19/c33 0 2024-03-30T04:28:01.406 INFO:tasks.workunit.client.0.smithi123.stdout:0/113: chown d6/cb 9491 1 2024-03-30T04:28:01.406 INFO:tasks.workunit.client.0.smithi123.stdout:1/227: chown db/l40 769 1 2024-03-30T04:28:01.407 INFO:tasks.workunit.client.0.smithi123.stdout:8/173: dread - da/dc/f16 zero size 2024-03-30T04:28:01.410 INFO:tasks.workunit.client.0.smithi123.stdout:2/130: mkdir d0/d1f/d25 0 2024-03-30T04:28:01.412 INFO:tasks.workunit.client.0.smithi123.stdout:5/162: write dd/f26 [569322,15280] 0 2024-03-30T04:28:01.412 INFO:tasks.workunit.client.0.smithi123.stdout:6/243: symlink d10/d13/l4e 0 2024-03-30T04:28:01.412 INFO:tasks.workunit.client.0.smithi123.stdout:8/174: write da/dc/f1a [635849,66340] 0 2024-03-30T04:28:01.417 INFO:tasks.workunit.client.0.smithi123.stdout:5/163: fsync dd/d19/f28 0 2024-03-30T04:28:01.418 INFO:tasks.workunit.client.0.smithi123.stdout:9/150: truncate d6/f15 1212743 0 2024-03-30T04:28:01.419 INFO:tasks.workunit.client.0.smithi123.stdout:7/76: truncate d2/f1b 772551 0 2024-03-30T04:28:01.422 INFO:tasks.workunit.client.0.smithi123.stdout:5/164: dread - dd/f32 zero size 2024-03-30T04:28:01.423 INFO:tasks.workunit.client.0.smithi123.stdout:1/228: mkdir db/dd/d42/d46/d51 0 2024-03-30T04:28:01.424 INFO:tasks.workunit.client.0.smithi123.stdout:0/114: write d6/f12 [404881,2781] 0 2024-03-30T04:28:01.427 INFO:tasks.workunit.client.0.smithi123.stdout:4/221: dwrite d2/f45 [0,4194304] 0 2024-03-30T04:28:01.430 INFO:tasks.workunit.client.0.smithi123.stdout:1/229: dread - db/dd/d42/d46/f2e zero size 2024-03-30T04:28:01.432 INFO:tasks.workunit.client.0.smithi123.stdout:6/244: mknod d10/d13/d1c/d42/c4f 0 2024-03-30T04:28:01.434 INFO:tasks.workunit.client.0.smithi123.stdout:9/151: mknod d6/d1f/c39 0 2024-03-30T04:28:01.438 INFO:tasks.workunit.client.0.smithi123.stdout:0/115: mkdir d6/dd/d18/d1f 0 2024-03-30T04:28:01.438 INFO:tasks.workunit.client.0.smithi123.stdout:4/222: fdatasync d2/d4/d7/d1a/f30 0 2024-03-30T04:28:01.444 INFO:tasks.workunit.client.0.smithi123.stdout:1/230: chown db/dd/d42/c36 111299783 1 2024-03-30T04:28:01.446 INFO:tasks.workunit.client.0.smithi123.stdout:4/223: write d2/d4/f22 [517770,2920] 0 2024-03-30T04:28:01.456 INFO:tasks.workunit.client.0.smithi123.stdout:6/245: fsync d10/d13/d1c/d42/f47 0 2024-03-30T04:28:01.458 INFO:tasks.workunit.client.0.smithi123.stdout:1/231: truncate db/d1b/f1d 5451925 0 2024-03-30T04:28:01.461 INFO:tasks.workunit.client.0.smithi123.stdout:2/131: dwrite d0/d1f/d16/f18 [0,4194304] 0 2024-03-30T04:28:01.462 INFO:tasks.workunit.client.0.smithi123.stdout:3/181: dwrite d13/d24/d31/d23/f2b [0,4194304] 0 2024-03-30T04:28:01.470 INFO:tasks.workunit.client.0.smithi123.stdout:9/152: getdents d6 0 2024-03-30T04:28:01.479 INFO:tasks.workunit.client.0.smithi123.stdout:4/224: symlink d2/d26/l54 0 2024-03-30T04:28:01.479 INFO:tasks.workunit.client.0.smithi123.stdout:1/232: write db/d1b/f37 [3062267,129047] 0 2024-03-30T04:28:01.484 INFO:tasks.workunit.client.0.smithi123.stdout:2/132: write d0/d1f/d16/f1e [140783,29696] 0 2024-03-30T04:28:01.487 INFO:tasks.workunit.client.0.smithi123.stdout:8/175: dwrite da/fe [0,4194304] 0 2024-03-30T04:28:01.492 INFO:tasks.workunit.client.0.smithi123.stdout:9/153: unlink d6/f21 0 2024-03-30T04:28:01.497 INFO:tasks.workunit.client.0.smithi123.stdout:5/165: dwrite f9 [4194304,4194304] 0 2024-03-30T04:28:01.500 INFO:tasks.workunit.client.0.smithi123.stdout:3/182: mknod d13/d24/d31/c32 0 2024-03-30T04:28:01.502 INFO:tasks.workunit.client.0.smithi123.stdout:1/233: write db/dd/f48 [521541,111708] 0 2024-03-30T04:28:01.502 INFO:tasks.workunit.client.0.smithi123.stdout:2/133: rmdir d0/d1f 39 2024-03-30T04:28:01.506 INFO:tasks.workunit.client.0.smithi123.stdout:7/77: dwrite d2/dc/ff [8388608,4194304] 0 2024-03-30T04:28:01.506 INFO:tasks.workunit.client.0.smithi123.stdout:8/176: dread - da/dc/d14/d18/d21/f28 zero size 2024-03-30T04:28:01.509 INFO:tasks.workunit.client.0.smithi123.stdout:0/116: dwrite d6/f9 [4194304,4194304] 0 2024-03-30T04:28:01.509 INFO:tasks.workunit.client.0.smithi123.stdout:9/154: rmdir d6/d1f 39 2024-03-30T04:28:01.512 INFO:tasks.workunit.client.0.smithi123.stdout:7/78: fdatasync d2/da/d11/f1f 0 2024-03-30T04:28:01.517 INFO:tasks.workunit.client.0.smithi123.stdout:6/246: rename ld to d10/l50 0 2024-03-30T04:28:01.517 INFO:tasks.workunit.client.0.smithi123.stdout:7/79: chown d2/da/l16 723 1 2024-03-30T04:28:01.520 INFO:tasks.workunit.client.0.smithi123.stdout:5/166: readlink dd/d19/l25 0 2024-03-30T04:28:01.522 INFO:tasks.workunit.client.0.smithi123.stdout:0/117: mknod d6/dd/c20 0 2024-03-30T04:28:01.523 INFO:tasks.workunit.client.0.smithi123.stdout:5/167: chown dd/f26 208952 1 2024-03-30T04:28:01.524 INFO:tasks.workunit.client.0.smithi123.stdout:8/177: unlink da/dc/d14/f15 0 2024-03-30T04:28:01.526 INFO:tasks.workunit.client.0.smithi123.stdout:3/183: symlink d13/l33 0 2024-03-30T04:28:01.530 INFO:tasks.workunit.client.0.smithi123.stdout:2/134: creat d0/d1f/d16/f26 x:0 0 0 2024-03-30T04:28:01.532 INFO:tasks.workunit.client.0.smithi123.stdout:6/247: creat d10/d13/d1c/d42/f51 x:0 0 0 2024-03-30T04:28:01.533 INFO:tasks.workunit.client.0.smithi123.stdout:0/118: write d6/dd/d15/f19 [385129,116160] 0 2024-03-30T04:28:01.533 INFO:tasks.workunit.client.0.smithi123.stdout:9/155: read - d6/d1f/f34 zero size 2024-03-30T04:28:01.535 INFO:tasks.workunit.client.0.smithi123.stdout:8/178: mknod da/d27/c2c 0 2024-03-30T04:28:01.536 INFO:tasks.workunit.client.0.smithi123.stdout:8/179: readlink da/d27/l29 0 2024-03-30T04:28:01.538 INFO:tasks.workunit.client.0.smithi123.stdout:3/184: unlink d13/l17 0 2024-03-30T04:28:01.539 INFO:tasks.workunit.client.0.smithi123.stdout:0/119: read d6/fa [9311,30658] 0 2024-03-30T04:28:01.540 INFO:tasks.workunit.client.0.smithi123.stdout:4/225: rename d2/l25 to d2/d4/d7/d1a/l55 0 2024-03-30T04:28:01.541 INFO:tasks.workunit.client.0.smithi123.stdout:2/135: mkdir d0/d1f/d27 0 2024-03-30T04:28:01.545 INFO:tasks.workunit.client.0.smithi123.stdout:9/156: rename d6/d1f to d6/d1f/d3a 22 2024-03-30T04:28:01.548 INFO:tasks.workunit.client.0.smithi123.stdout:6/248: fsync d10/f26 0 2024-03-30T04:28:01.548 INFO:tasks.workunit.client.0.smithi123.stdout:0/120: mknod d6/c21 0 2024-03-30T04:28:01.554 INFO:tasks.workunit.client.0.smithi123.stdout:2/136: chown d0/d1f/f24 65002 1 2024-03-30T04:28:01.554 INFO:tasks.workunit.client.0.smithi123.stdout:7/80: dread d2/f8 [0,4194304] 0 2024-03-30T04:28:01.554 INFO:tasks.workunit.client.0.smithi123.stdout:9/157: creat d6/d23/f3b x:0 0 0 2024-03-30T04:28:01.556 INFO:tasks.workunit.client.0.smithi123.stdout:3/185: read - d13/d24/d31/f2a zero size 2024-03-30T04:28:01.558 INFO:tasks.workunit.client.0.smithi123.stdout:0/121: rename d6/dd to d6/dd/d15/d22 22 2024-03-30T04:28:01.562 INFO:tasks.workunit.client.0.smithi123.stdout:6/249: write d10/d13/f25 [877562,115406] 0 2024-03-30T04:28:01.566 INFO:tasks.workunit.client.0.smithi123.stdout:7/81: read d2/da/d11/f14 [1104019,62912] 0 2024-03-30T04:28:01.569 INFO:tasks.workunit.client.0.smithi123.stdout:5/168: dwrite dd/f32 [0,4194304] 0 2024-03-30T04:28:01.574 INFO:tasks.workunit.client.0.smithi123.stdout:0/122: dread - d6/dd/d16/f1e zero size 2024-03-30T04:28:01.579 INFO:tasks.workunit.client.0.smithi123.stdout:7/82: creat d2/da/d11/f20 x:0 0 0 2024-03-30T04:28:01.579 INFO:tasks.workunit.client.0.smithi123.stdout:5/169: mknod dd/d19/d1f/c34 0 2024-03-30T04:28:01.580 INFO:tasks.workunit.client.0.smithi123.stdout:6/250: write d10/f1b [5073676,63242] 0 2024-03-30T04:28:01.580 INFO:tasks.workunit.client.0.smithi123.stdout:3/186: unlink d13/l2c 0 2024-03-30T04:28:01.584 INFO:tasks.workunit.client.0.smithi123.stdout:9/158: symlink d6/l3c 0 2024-03-30T04:28:01.584 INFO:tasks.workunit.client.0.smithi123.stdout:0/123: rename d6/dd/d18 to d6/dd/d18/d23 22 2024-03-30T04:28:01.584 INFO:tasks.workunit.client.0.smithi123.stdout:4/226: dwrite d2/d4/f16 [0,4194304] 0 2024-03-30T04:28:01.590 INFO:tasks.workunit.client.0.smithi123.stdout:5/170: mkdir dd/d19/d1f/d35 0 2024-03-30T04:28:01.592 INFO:tasks.workunit.client.0.smithi123.stdout:7/83: mknod d2/da/c21 0 2024-03-30T04:28:01.596 INFO:tasks.workunit.client.0.smithi123.stdout:5/171: dread - dd/d19/f22 zero size 2024-03-30T04:28:01.599 INFO:tasks.workunit.client.0.smithi123.stdout:0/124: creat d6/f24 x:0 0 0 2024-03-30T04:28:01.604 INFO:tasks.workunit.client.0.smithi123.stdout:4/227: mknod d2/c56 0 2024-03-30T04:28:01.606 INFO:tasks.workunit.client.0.smithi123.stdout:6/251: mknod d10/d13/c52 0 2024-03-30T04:28:01.607 INFO:tasks.workunit.client.0.smithi123.stdout:2/137: dwrite d0/d1f/f1b [0,4194304] 0 2024-03-30T04:28:01.618 INFO:tasks.workunit.client.0.smithi123.stdout:5/172: rename dd/d19/c2e to dd/c36 0 2024-03-30T04:28:01.621 INFO:tasks.workunit.client.0.smithi123.stdout:0/125: creat d6/dd/d15/f25 x:0 0 0 2024-03-30T04:28:01.622 INFO:tasks.workunit.client.0.smithi123.stdout:9/159: link d6/c20 d6/d1f/c3d 0 2024-03-30T04:28:01.623 INFO:tasks.workunit.client.0.smithi123.stdout:3/187: getdents d13 0 2024-03-30T04:28:01.627 INFO:tasks.workunit.client.0.smithi123.stdout:3/188: fsync d13/d24/d31/d23/f25 0 2024-03-30T04:28:01.630 INFO:tasks.workunit.client.0.smithi123.stdout:2/138: symlink d0/d1f/l28 0 2024-03-30T04:28:01.631 INFO:tasks.workunit.client.0.smithi123.stdout:7/84: dread f0 [0,4194304] 0 2024-03-30T04:28:01.634 INFO:tasks.workunit.client.0.smithi123.stdout:4/228: mknod d2/d4/d7/c57 0 2024-03-30T04:28:01.636 INFO:tasks.workunit.client.0.smithi123.stdout:2/139: write d0/d1f/f24 [838712,112477] 0 2024-03-30T04:28:01.640 INFO:tasks.workunit.client.0.smithi123.stdout:3/189: mknod d13/d24/c34 0 2024-03-30T04:28:01.643 INFO:tasks.workunit.client.0.smithi123.stdout:9/160: mknod d6/d23/c3e 0 2024-03-30T04:28:01.647 INFO:tasks.workunit.client.0.smithi123.stdout:9/161: write d6/d23/f3b [382653,125385] 0 2024-03-30T04:28:01.648 INFO:tasks.workunit.client.0.smithi123.stdout:7/85: unlink d2/c3 0 2024-03-30T04:28:01.655 INFO:tasks.workunit.client.0.smithi123.stdout:4/229: chown d2/d4/d7/l36 3536920 1 2024-03-30T04:28:01.657 INFO:tasks.workunit.client.0.smithi123.stdout:7/86: chown d2/cd 27076017 1 2024-03-30T04:28:01.658 INFO:tasks.workunit.client.0.smithi123.stdout:5/173: dwrite dd/d19/d1f/f27 [0,4194304] 0 2024-03-30T04:28:01.659 INFO:tasks.workunit.client.0.smithi123.stdout:6/252: rename d10/d13/d1c/d42/c4f to d10/d13/c53 0 2024-03-30T04:28:01.660 INFO:tasks.workunit.client.0.smithi123.stdout:0/126: dwrite d6/f1b [0,4194304] 0 2024-03-30T04:28:01.664 INFO:tasks.workunit.client.0.smithi123.stdout:3/190: creat d13/d24/d31/f35 x:0 0 0 2024-03-30T04:28:01.664 INFO:tasks.workunit.client.0.smithi123.stdout:4/230: rename d2/d34 to d2/d34/d58 22 2024-03-30T04:28:01.669 INFO:tasks.workunit.client.0.smithi123.stdout:3/191: chown d13/f1b 58142 1 2024-03-30T04:28:01.669 INFO:tasks.workunit.client.0.smithi123.stdout:9/162: creat d6/d1f/f3f x:0 0 0 2024-03-30T04:28:01.670 INFO:tasks.workunit.client.0.smithi123.stdout:2/140: creat d0/d1f/d27/f29 x:0 0 0 2024-03-30T04:28:01.676 INFO:tasks.workunit.client.0.smithi123.stdout:9/163: write d6/d23/f2a [95080,81548] 0 2024-03-30T04:28:01.677 INFO:tasks.workunit.client.0.smithi123.stdout:7/87: write d2/f8 [4806182,99288] 0 2024-03-30T04:28:01.684 INFO:tasks.workunit.client.0.smithi123.stdout:0/127: symlink d6/dd/d18/d1f/l26 0 2024-03-30T04:28:01.685 INFO:tasks.workunit.client.0.smithi123.stdout:4/231: rmdir d2/d4/d2c/d42 39 2024-03-30T04:28:01.689 INFO:tasks.workunit.client.0.smithi123.stdout:4/232: chown d2/f1c 6 1 2024-03-30T04:28:01.691 INFO:tasks.workunit.client.0.smithi123.stdout:0/128: rmdir d6/dd/d16 39 2024-03-30T04:28:01.694 INFO:tasks.workunit.client.0.smithi123.stdout:4/233: write d2/d4/d7/d1a/f21 [4779592,61099] 0 2024-03-30T04:28:01.704 INFO:tasks.workunit.client.0.smithi123.stdout:7/88: dread d2/f8 [0,4194304] 0 2024-03-30T04:28:01.704 INFO:tasks.workunit.client.0.smithi123.stdout:6/253: getdents d10/d13/d1c/d42 0 2024-03-30T04:28:01.708 INFO:tasks.workunit.client.0.smithi123.stdout:0/129: link d6/dd/f14 d6/dd/d18/d1f/f27 0 2024-03-30T04:28:01.711 INFO:tasks.workunit.client.0.smithi123.stdout:4/234: symlink d2/d4/d2c/l59 0 2024-03-30T04:28:01.713 INFO:tasks.workunit.client.0.smithi123.stdout:3/192: dwrite d13/d24/d31/d23/f2e [0,4194304] 0 2024-03-30T04:28:01.716 INFO:tasks.workunit.client.0.smithi123.stdout:7/89: write d2/da/fb [1782463,74737] 0 2024-03-30T04:28:01.718 INFO:tasks.workunit.client.0.smithi123.stdout:0/130: unlink d6/dd/c20 0 2024-03-30T04:28:01.724 INFO:tasks.workunit.client.0.smithi123.stdout:0/131: truncate d6/fe 1142371 0 2024-03-30T04:28:01.725 INFO:tasks.workunit.client.0.smithi123.stdout:7/90: mkdir d2/da/d22 0 2024-03-30T04:28:01.727 INFO:tasks.workunit.client.0.smithi123.stdout:6/254: rmdir d10/d13/d45/d46 0 2024-03-30T04:28:01.731 INFO:tasks.workunit.client.0.smithi123.stdout:3/193: truncate d13/d24/d31/d23/f25 2332474 0 2024-03-30T04:28:01.732 INFO:tasks.workunit.client.0.smithi123.stdout:0/132: fdatasync d6/dd/d18/d1f/f27 0 2024-03-30T04:28:01.733 INFO:tasks.workunit.client.0.smithi123.stdout:5/174: dread f9 [0,4194304] 0 2024-03-30T04:28:01.735 INFO:tasks.workunit.client.0.smithi123.stdout:4/235: rename d2/d34/l53 to d2/l5a 0 2024-03-30T04:28:01.735 INFO:tasks.workunit.client.0.smithi123.stdout:3/194: chown d13/d24 495272 1 2024-03-30T04:28:01.736 INFO:tasks.workunit.client.0.smithi123.stdout:5/175: fsync dd/d19/f28 0 2024-03-30T04:28:01.736 INFO:tasks.workunit.client.0.smithi123.stdout:6/255: symlink d10/d13/d45/l54 0 2024-03-30T04:28:01.739 INFO:tasks.workunit.client.0.smithi123.stdout:0/133: chown d6/dd/d16 26823 1 2024-03-30T04:28:01.739 INFO:tasks.workunit.client.0.smithi123.stdout:5/176: read f9 [7228982,44368] 0 2024-03-30T04:28:01.739 INFO:tasks.workunit.client.0.smithi123.stdout:0/134: fsync d6/f1b 0 2024-03-30T04:28:01.740 INFO:tasks.workunit.client.0.smithi123.stdout:6/256: write d10/d20/d3f/f4d [872098,127527] 0 2024-03-30T04:28:01.748 INFO:tasks.workunit.client.0.smithi123.stdout:6/257: mknod d10/d13/d45/c55 0 2024-03-30T04:28:01.750 INFO:tasks.workunit.client.0.smithi123.stdout:0/135: mkdir d6/dd/d15/d28 0 2024-03-30T04:28:01.751 INFO:tasks.workunit.client.0.smithi123.stdout:6/258: fsync d10/d20/f22 0 2024-03-30T04:28:01.754 INFO:tasks.workunit.client.0.smithi123.stdout:0/136: read f5 [3646057,61821] 0 2024-03-30T04:28:01.757 INFO:tasks.workunit.client.0.smithi123.stdout:6/259: creat d10/d20/f56 x:0 0 0 2024-03-30T04:28:01.757 INFO:tasks.workunit.client.0.smithi123.stdout:4/236: getdents d2/d4/d2c 0 2024-03-30T04:28:01.758 INFO:tasks.workunit.client.0.smithi123.stdout:5/177: link dd/l29 dd/d19/d1f/d35/l37 0 2024-03-30T04:28:01.760 INFO:tasks.workunit.client.0.smithi123.stdout:0/137: creat d6/dd/d15/f29 x:0 0 0 2024-03-30T04:28:01.764 INFO:tasks.workunit.client.0.smithi123.stdout:5/178: creat dd/d19/d1f/d35/f38 x:0 0 0 2024-03-30T04:28:01.765 INFO:tasks.workunit.client.0.smithi123.stdout:3/195: getdents d13/d24 0 2024-03-30T04:28:01.774 INFO:tasks.workunit.client.0.smithi123.stdout:5/179: truncate dd/d19/f28 3760591 0 2024-03-30T04:28:01.775 INFO:tasks.workunit.client.0.smithi123.stdout:5/180: readlink l8 0 2024-03-30T04:28:01.776 INFO:tasks.workunit.client.0.smithi123.stdout:7/91: dread d2/da/fb [0,4194304] 0 2024-03-30T04:28:01.777 INFO:tasks.workunit.client.0.smithi123.stdout:3/196: rmdir d13/d24/d31/d2f 0 2024-03-30T04:28:01.783 INFO:tasks.workunit.client.0.smithi123.stdout:1/234: sync 2024-03-30T04:28:01.787 INFO:tasks.workunit.client.0.smithi123.stdout:3/197: symlink d13/d24/d31/d23/l36 0 2024-03-30T04:28:01.787 INFO:tasks.workunit.client.0.smithi123.stdout:4/237: dwrite d2/d4/f8 [0,4194304] 0 2024-03-30T04:28:01.790 INFO:tasks.workunit.client.0.smithi123.stdout:0/138: dwrite d6/f1b [0,4194304] 0 2024-03-30T04:28:01.792 INFO:tasks.workunit.client.0.smithi123.stdout:7/92: creat d2/dc/d1c/f23 x:0 0 0 2024-03-30T04:28:01.794 INFO:tasks.workunit.client.0.smithi123.stdout:1/235: chown db/c43 65 1 2024-03-30T04:28:01.796 INFO:tasks.workunit.client.0.smithi123.stdout:6/260: dread d10/f38 [0,4194304] 0 2024-03-30T04:28:01.796 INFO:tasks.workunit.client.0.smithi123.stdout:1/236: read - db/dd/f4d zero size 2024-03-30T04:28:01.800 INFO:tasks.workunit.client.0.smithi123.stdout:3/198: creat d13/d24/d31/f37 x:0 0 0 2024-03-30T04:28:01.803 INFO:tasks.workunit.client.0.smithi123.stdout:4/238: mknod d2/d4/d2c/d42/c5b 0 2024-03-30T04:28:01.804 INFO:tasks.workunit.client.0.smithi123.stdout:7/93: symlink d2/dc/d1c/l24 0 2024-03-30T04:28:01.806 INFO:tasks.workunit.client.0.smithi123.stdout:1/237: rmdir db 39 2024-03-30T04:28:01.806 INFO:tasks.workunit.client.0.smithi123.stdout:3/199: truncate d13/d24/d31/f2a 991688 0 2024-03-30T04:28:01.809 INFO:tasks.workunit.client.0.smithi123.stdout:7/94: symlink d2/dc/l25 0 2024-03-30T04:28:01.820 INFO:tasks.workunit.client.0.smithi123.stdout:0/139: dread d6/f1b [0,4194304] 0 2024-03-30T04:28:01.820 INFO:tasks.workunit.client.0.smithi123.stdout:3/200: rename f12 to d13/d24/d31/d23/f38 0 2024-03-30T04:28:01.821 INFO:tasks.workunit.client.0.smithi123.stdout:1/238: write db/fe [1724110,19832] 0 2024-03-30T04:28:01.824 INFO:tasks.workunit.client.0.smithi123.stdout:3/201: readlink d13/d24/d31/d23/l26 0 2024-03-30T04:28:01.826 INFO:tasks.workunit.client.0.smithi123.stdout:0/140: mknod d6/dd/d18/d1f/c2a 0 2024-03-30T04:28:01.831 INFO:tasks.workunit.client.0.smithi123.stdout:1/239: creat db/dd/d42/d46/d39/f52 x:0 0 0 2024-03-30T04:28:01.839 INFO:tasks.workunit.client.0.smithi123.stdout:7/95: dwrite d2/da/d11/f20 [0,4194304] 0 2024-03-30T04:28:01.839 INFO:tasks.workunit.client.0.smithi123.stdout:3/202: mkdir d13/d24/d39 0 2024-03-30T04:28:01.841 INFO:tasks.workunit.client.0.smithi123.stdout:7/96: fdatasync d2/dc/d1c/f23 0 2024-03-30T04:28:01.843 INFO:tasks.workunit.client.0.smithi123.stdout:0/141: unlink d6/dd/d15/f19 0 2024-03-30T04:28:01.845 INFO:tasks.workunit.client.0.smithi123.stdout:7/97: write d2/da/d11/f1e [168401,78379] 0 2024-03-30T04:28:01.846 INFO:tasks.workunit.client.0.smithi123.stdout:6/261: dread d10/f27 [0,4194304] 0 2024-03-30T04:28:01.849 INFO:tasks.workunit.client.0.smithi123.stdout:1/240: unlink db/dd/f21 0 2024-03-30T04:28:01.856 INFO:tasks.workunit.client.0.smithi123.stdout:3/203: mkdir d13/d24/d31/d23/d3a 0 2024-03-30T04:28:01.857 INFO:tasks.workunit.client.0.smithi123.stdout:5/181: dread dd/d19/f28 [0,4194304] 0 2024-03-30T04:28:01.858 INFO:tasks.workunit.client.0.smithi123.stdout:0/142: write d6/f1b [5161004,2603] 0 2024-03-30T04:28:01.859 INFO:tasks.workunit.client.0.smithi123.stdout:7/98: fsync f0 0 2024-03-30T04:28:01.863 INFO:tasks.workunit.client.0.smithi123.stdout:4/239: rmdir d2/d4 39 2024-03-30T04:28:01.864 INFO:tasks.workunit.client.0.smithi123.stdout:0/143: dread - d6/dd/d16/f1e zero size 2024-03-30T04:28:01.865 INFO:tasks.workunit.client.0.smithi123.stdout:1/241: creat db/dd/d42/d46/f53 x:0 0 0 2024-03-30T04:28:01.867 INFO:tasks.workunit.client.0.smithi123.stdout:5/182: symlink dd/l39 0 2024-03-30T04:28:01.869 INFO:tasks.workunit.client.0.smithi123.stdout:3/204: mknod d13/d24/d31/d23/c3b 0 2024-03-30T04:28:01.870 INFO:tasks.workunit.client.0.smithi123.stdout:5/183: write dd/f26 [106832,48430] 0 2024-03-30T04:28:01.872 INFO:tasks.workunit.client.0.smithi123.stdout:7/99: mknod d2/dc/d1c/c26 0 2024-03-30T04:28:01.877 INFO:tasks.workunit.client.0.smithi123.stdout:0/144: creat d6/dd/d15/f2b x:0 0 0 2024-03-30T04:28:01.883 INFO:tasks.workunit.client.0.smithi123.stdout:5/184: mkdir dd/d19/d1f/d35/d3a 0 2024-03-30T04:28:01.884 INFO:tasks.workunit.client.0.smithi123.stdout:5/185: chown ca 3 1 2024-03-30T04:28:01.886 INFO:tasks.workunit.client.0.smithi123.stdout:0/145: mknod d6/c2c 0 2024-03-30T04:28:01.886 INFO:tasks.workunit.client.0.smithi123.stdout:6/262: write d10/d13/d1c/d42/f47 [228855,10403] 0 2024-03-30T04:28:01.887 INFO:tasks.workunit.client.0.smithi123.stdout:3/205: mkdir d13/d24/d31/d23/d3a/d3c 0 2024-03-30T04:28:01.890 INFO:tasks.workunit.client.0.smithi123.stdout:1/242: creat db/dd/f54 x:0 0 0 2024-03-30T04:28:01.897 INFO:tasks.workunit.client.0.smithi123.stdout:3/206: mkdir d13/d24/d31/d23/d3a/d3c/d3d 0 2024-03-30T04:28:01.904 INFO:tasks.workunit.client.0.smithi123.stdout:0/146: link l2 d6/dd/l2d 0 2024-03-30T04:28:01.908 INFO:tasks.workunit.client.0.smithi123.stdout:5/186: rmdir dd/d19/d1f/d35 39 2024-03-30T04:28:01.914 INFO:tasks.workunit.client.0.smithi123.stdout:4/240: truncate d2/d4/d7/d1a/f21 2033235 0 2024-03-30T04:28:01.915 INFO:tasks.workunit.client.0.smithi123.stdout:0/147: symlink d6/dd/d16/l2e 0 2024-03-30T04:28:01.923 INFO:tasks.workunit.client.0.smithi123.stdout:6/263: dwrite d10/d13/f25 [0,4194304] 0 2024-03-30T04:28:01.923 INFO:tasks.workunit.client.0.smithi123.stdout:1/243: dwrite db/f41 [0,4194304] 0 2024-03-30T04:28:01.925 INFO:tasks.workunit.client.0.smithi123.stdout:4/241: write d2/d4/d2c/d42/f3e [569936,52415] 0 2024-03-30T04:28:01.926 INFO:tasks.workunit.client.0.smithi123.stdout:3/207: dread f11 [0,4194304] 0 2024-03-30T04:28:01.930 INFO:tasks.workunit.client.0.smithi123.stdout:0/148: truncate d6/dd/f14 70575 0 2024-03-30T04:28:01.930 INFO:tasks.workunit.client.0.smithi123.stdout:5/187: write dd/f15 [358837,81871] 0 2024-03-30T04:28:01.935 INFO:tasks.workunit.client.0.smithi123.stdout:6/264: fdatasync d10/d13/d1c/f1e 0 2024-03-30T04:28:01.938 INFO:tasks.workunit.client.0.smithi123.stdout:3/208: chown d13/d24/d31/d23/f38 0 1 2024-03-30T04:28:01.940 INFO:tasks.workunit.client.0.smithi123.stdout:7/100: dwrite d2/f1b [0,4194304] 0 2024-03-30T04:28:01.947 INFO:tasks.workunit.client.0.smithi123.stdout:6/265: truncate d10/d13/d1c/d42/f51 673336 0 2024-03-30T04:28:01.951 INFO:tasks.workunit.client.0.smithi123.stdout:6/266: read - d10/d20/f40 zero size 2024-03-30T04:28:01.955 INFO:tasks.workunit.client.0.smithi123.stdout:1/244: dwrite db/dd/f13 [0,4194304] 0 2024-03-30T04:28:01.962 INFO:tasks.workunit.client.0.smithi123.stdout:7/101: fdatasync d2/da/d11/f14 0 2024-03-30T04:28:01.965 INFO:tasks.workunit.client.0.smithi123.stdout:7/102: write d2/dc/d1c/f23 [220123,116929] 0 2024-03-30T04:28:01.967 INFO:tasks.workunit.client.0.smithi123.stdout:3/209: creat d13/d24/d31/d23/d3a/f3e x:0 0 0 2024-03-30T04:28:01.968 INFO:tasks.workunit.client.0.smithi123.stdout:5/188: chown dd/d19/d1f/d35/l37 414 1 2024-03-30T04:28:01.974 INFO:tasks.workunit.client.0.smithi123.stdout:7/103: creat d2/dc/f27 x:0 0 0 2024-03-30T04:28:01.977 INFO:tasks.workunit.client.0.smithi123.stdout:1/245: link db/dd/f4d db/dd/d42/d46/d51/f55 0 2024-03-30T04:28:01.981 INFO:tasks.workunit.client.0.smithi123.stdout:3/210: mknod d13/d24/d31/d23/d3a/d3c/c3f 0 2024-03-30T04:28:01.984 INFO:tasks.workunit.client.0.smithi123.stdout:6/267: dwrite d10/f1d [0,4194304] 0 2024-03-30T04:28:01.984 INFO:tasks.workunit.client.0.smithi123.stdout:1/246: write db/dd/f48 [903683,12588] 0 2024-03-30T04:28:01.986 INFO:tasks.workunit.client.0.smithi123.stdout:5/189: rename c1 to dd/d19/d1f/d35/c3b 0 2024-03-30T04:28:01.988 INFO:tasks.workunit.client.0.smithi123.stdout:3/211: chown d13/c15 508 1 2024-03-30T04:28:01.989 INFO:tasks.workunit.client.0.smithi123.stdout:5/190: chown dd/l29 309230685 1 2024-03-30T04:28:01.994 INFO:tasks.workunit.client.0.smithi123.stdout:1/247: mkdir db/dd/d42/d46/d56 0 2024-03-30T04:28:02.004 INFO:tasks.workunit.client.0.smithi123.stdout:3/212: rename d13/d24/d31/f37 to d13/d24/d31/d23/d3a/f40 0 2024-03-30T04:28:02.005 INFO:tasks.workunit.client.0.smithi123.stdout:3/213: chown l4 364 1 2024-03-30T04:28:02.007 INFO:tasks.workunit.client.0.smithi123.stdout:3/214: chown c5 1022778454 1 2024-03-30T04:28:02.008 INFO:tasks.workunit.client.0.smithi123.stdout:6/268: symlink d10/d13/d1c/d4c/l57 0 2024-03-30T04:28:02.008 INFO:tasks.workunit.client.0.smithi123.stdout:4/242: dread d2/d4/d7/d1a/f30 [4194304,4194304] 0 2024-03-30T04:28:02.013 INFO:tasks.workunit.client.0.smithi123.stdout:1/248: rmdir db/dd/d42/d46/d51 39 2024-03-30T04:28:02.014 INFO:tasks.workunit.client.0.smithi123.stdout:1/249: fdatasync db/d1b/f1d 0 2024-03-30T04:28:02.024 INFO:tasks.workunit.client.0.smithi123.stdout:6/269: write d10/f38 [1526901,84278] 0 2024-03-30T04:28:02.026 INFO:tasks.workunit.client.0.smithi123.stdout:6/270: chown d10/d20/c35 926780 1 2024-03-30T04:28:02.027 INFO:tasks.workunit.client.0.smithi123.stdout:3/215: rename d13/d24/d31/f2a to d13/d24/d39/f41 0 2024-03-30T04:28:02.029 INFO:tasks.workunit.client.0.smithi123.stdout:6/271: read d10/f11 [3644563,125468] 0 2024-03-30T04:28:02.043 INFO:tasks.workunit.client.0.smithi123.stdout:7/104: truncate d2/dc/d1c/f1d 2789971 0 2024-03-30T04:28:02.045 INFO:tasks.workunit.client.0.smithi123.stdout:7/105: chown d2/f8 2421 1 2024-03-30T04:28:02.046 INFO:tasks.workunit.client.0.smithi123.stdout:3/216: write f11 [1935697,66987] 0 2024-03-30T04:28:02.046 INFO:tasks.workunit.client.0.smithi123.stdout:7/106: chown d2/dc/l25 0 1 2024-03-30T04:28:02.048 INFO:tasks.workunit.client.0.smithi123.stdout:7/107: stat d2/dc/l25 0 2024-03-30T04:28:02.049 INFO:tasks.workunit.client.0.smithi123.stdout:7/108: fsync d2/dc/f27 0 2024-03-30T04:28:02.050 INFO:tasks.workunit.client.0.smithi123.stdout:6/272: creat d10/d20/f58 x:0 0 0 2024-03-30T04:28:02.052 INFO:tasks.workunit.client.0.smithi123.stdout:7/109: read d2/f8 [3660737,60883] 0 2024-03-30T04:28:02.060 INFO:tasks.workunit.client.0.smithi123.stdout:5/191: getdents dd/d19/d1f 0 2024-03-30T04:28:02.064 INFO:tasks.workunit.client.0.smithi123.stdout:3/217: rename d13/d24/d31/d23/l36 to d13/d24/d31/d23/d3a/l42 0 2024-03-30T04:28:02.067 INFO:tasks.workunit.client.0.smithi123.stdout:7/110: mkdir d2/d28 0 2024-03-30T04:28:02.068 INFO:tasks.workunit.client.0.smithi123.stdout:7/111: dread - d2/da/d11/f19 zero size 2024-03-30T04:28:02.070 INFO:tasks.workunit.client.0.smithi123.stdout:7/112: chown d2/f13 7437229 1 2024-03-30T04:28:02.072 INFO:tasks.workunit.client.0.smithi123.stdout:5/192: mkdir dd/d19/d1f/d3c 0 2024-03-30T04:28:02.072 INFO:tasks.workunit.client.0.smithi123.stdout:7/113: fdatasync d2/dc/d1c/f23 0 2024-03-30T04:28:02.077 INFO:tasks.workunit.client.0.smithi123.stdout:8/180: sync 2024-03-30T04:28:02.080 INFO:tasks.workunit.client.0.smithi123.stdout:6/273: creat d10/f59 x:0 0 0 2024-03-30T04:28:02.080 INFO:tasks.workunit.client.0.smithi123.stdout:8/181: write da/dc/f10 [1431189,50743] 0 2024-03-30T04:28:02.081 INFO:tasks.workunit.client.0.smithi123.stdout:4/243: truncate d2/d4/f22 984580 0 2024-03-30T04:28:02.082 INFO:tasks.workunit.client.0.smithi123.stdout:8/182: stat da/fe 0 2024-03-30T04:28:02.084 INFO:tasks.workunit.client.0.smithi123.stdout:8/183: fsync da/f12 0 2024-03-30T04:28:02.090 INFO:tasks.workunit.client.0.smithi123.stdout:5/193: dread dd/f32 [0,4194304] 0 2024-03-30T04:28:02.091 INFO:tasks.workunit.client.0.smithi123.stdout:7/114: symlink d2/dc/d1c/l29 0 2024-03-30T04:28:02.094 INFO:tasks.workunit.client.0.smithi123.stdout:5/194: chown dd/f26 3352522 1 2024-03-30T04:28:02.096 INFO:tasks.workunit.client.0.smithi123.stdout:7/115: chown d2/da/d11/f1f 165 1 2024-03-30T04:28:02.104 INFO:tasks.workunit.client.0.smithi123.stdout:1/250: dwrite db/dd/d42/d46/d51/f55 [0,4194304] 0 2024-03-30T04:28:02.104 INFO:tasks.workunit.client.0.smithi123.stdout:3/218: readlink d13/d24/d31/d23/d3a/l42 0 2024-03-30T04:28:02.108 INFO:tasks.workunit.client.0.smithi123.stdout:3/219: write ff [718965,116133] 0 2024-03-30T04:28:02.114 INFO:tasks.workunit.client.0.smithi123.stdout:3/220: read ff [53568,127633] 0 2024-03-30T04:28:02.116 INFO:tasks.workunit.client.0.smithi123.stdout:8/184: dwrite f8 [0,4194304] 0 2024-03-30T04:28:02.117 INFO:tasks.workunit.client.0.smithi123.stdout:3/221: readlink d13/l33 0 2024-03-30T04:28:02.125 INFO:tasks.workunit.client.0.smithi123.stdout:6/274: dwrite d10/d20/f3b [4194304,4194304] 0 2024-03-30T04:28:02.127 INFO:tasks.workunit.client.0.smithi123.stdout:8/185: write da/dc/d14/d18/d21/f28 [235822,30780] 0 2024-03-30T04:28:02.135 INFO:tasks.workunit.client.0.smithi123.stdout:8/186: dread da/dc/f13 [0,4194304] 0 2024-03-30T04:28:02.145 INFO:tasks.workunit.client.0.smithi123.stdout:1/251: creat db/dd/d42/d46/d39/f57 x:0 0 0 2024-03-30T04:28:02.151 INFO:tasks.workunit.client.0.smithi123.stdout:3/222: dwrite d13/d24/d31/f35 [0,4194304] 0 2024-03-30T04:28:02.155 INFO:tasks.workunit.client.0.smithi123.stdout:1/252: read db/d1b/f3e [2502547,102361] 0 2024-03-30T04:28:02.156 INFO:tasks.workunit.client.0.smithi123.stdout:6/275: dwrite d10/d13/f25 [0,4194304] 0 2024-03-30T04:28:02.157 INFO:tasks.workunit.client.0.smithi123.stdout:8/187: dread da/dc/d14/d18/d21/f28 [0,4194304] 0 2024-03-30T04:28:02.159 INFO:tasks.workunit.client.0.smithi123.stdout:6/276: fdatasync d10/f38 0 2024-03-30T04:28:02.161 INFO:tasks.workunit.client.0.smithi123.stdout:8/188: fsync da/fb 0 2024-03-30T04:28:02.161 INFO:tasks.workunit.client.0.smithi123.stdout:7/116: dread d2/dc/ff [4194304,4194304] 0 2024-03-30T04:28:02.165 INFO:tasks.workunit.client.0.smithi123.stdout:5/195: mknod dd/d19/d1f/c3d 0 2024-03-30T04:28:02.166 INFO:tasks.workunit.client.0.smithi123.stdout:1/253: truncate db/dd/d42/d46/d39/f57 271847 0 2024-03-30T04:28:02.169 INFO:tasks.workunit.client.0.smithi123.stdout:1/254: chown db/d1b/f37 6 1 2024-03-30T04:28:02.178 INFO:tasks.workunit.client.0.smithi123.stdout:3/223: mknod d13/d24/d31/d23/d3a/d3c/c43 0 2024-03-30T04:28:02.180 INFO:tasks.workunit.client.0.smithi123.stdout:3/224: write d13/f14 [4832949,56134] 0 2024-03-30T04:28:02.183 INFO:tasks.workunit.client.0.smithi123.stdout:8/189: mkdir da/dc/d14/d24/d2d 0 2024-03-30T04:28:02.184 INFO:tasks.workunit.client.0.smithi123.stdout:5/196: dread dd/f15 [0,4194304] 0 2024-03-30T04:28:02.186 INFO:tasks.workunit.client.0.smithi123.stdout:7/117: rename d2/da/d11/f14 to d2/d28/f2a 0 2024-03-30T04:28:02.191 INFO:tasks.workunit.client.0.smithi123.stdout:1/255: mknod db/dd/d42/d46/d51/c58 0 2024-03-30T04:28:02.197 INFO:tasks.workunit.client.0.smithi123.stdout:3/225: write d13/d24/d31/d23/f38 [1748151,4110] 0 2024-03-30T04:28:02.203 INFO:tasks.workunit.client.0.smithi123.stdout:5/197: read dd/fe [2165009,109591] 0 2024-03-30T04:28:02.206 INFO:tasks.workunit.client.0.smithi123.stdout:1/256: dread db/dd/f13 [0,4194304] 0 2024-03-30T04:28:02.207 INFO:tasks.workunit.client.0.smithi123.stdout:6/277: link d10/f3a d10/f5a 0 2024-03-30T04:28:02.211 INFO:tasks.workunit.client.0.smithi123.stdout:8/190: truncate f5 6060974 0 2024-03-30T04:28:02.213 INFO:tasks.workunit.client.0.smithi123.stdout:3/226: rename d13/c15 to d13/d24/c44 0 2024-03-30T04:28:02.220 INFO:tasks.workunit.client.0.smithi123.stdout:1/257: stat db/l18 0 2024-03-30T04:28:02.223 INFO:tasks.workunit.client.0.smithi123.stdout:6/278: mkdir d10/d13/d1c/d5b 0 2024-03-30T04:28:02.226 INFO:tasks.workunit.client.0.smithi123.stdout:3/227: dread d13/d24/d31/f35 [0,4194304] 0 2024-03-30T04:28:02.232 INFO:tasks.workunit.client.0.smithi123.stdout:2/141: sync 2024-03-30T04:28:02.234 INFO:tasks.workunit.client.0.smithi123.stdout:5/198: dread dd/fe [0,4194304] 0 2024-03-30T04:28:02.242 INFO:tasks.workunit.client.0.smithi123.stdout:2/142: mkdir d0/d1f/d1c/d2a 0 2024-03-30T04:28:02.246 INFO:tasks.workunit.client.0.smithi123.stdout:1/258: creat db/f59 x:0 0 0 2024-03-30T04:28:02.251 INFO:tasks.workunit.client.0.smithi123.stdout:3/228: dwrite d13/d24/d31/d23/d3a/f40 [0,4194304] 0 2024-03-30T04:28:02.251 INFO:tasks.workunit.client.0.smithi123.stdout:5/199: write dd/d19/d1f/d35/f38 [771674,30990] 0 2024-03-30T04:28:02.256 INFO:tasks.workunit.client.0.smithi123.stdout:2/143: symlink d0/d1f/d1c/d20/l2b 0 2024-03-30T04:28:02.256 INFO:tasks.workunit.client.0.smithi123.stdout:3/229: write d13/f1b [7678109,108733] 0 2024-03-30T04:28:02.256 INFO:tasks.workunit.client.0.smithi123.stdout:5/200: fdatasync dd/d19/d1f/d35/f38 0 2024-03-30T04:28:02.262 INFO:tasks.workunit.client.0.smithi123.stdout:5/201: dread - dd/d19/f22 zero size 2024-03-30T04:28:02.262 INFO:tasks.workunit.client.0.smithi123.stdout:2/144: write d0/d1f/f17 [893788,80953] 0 2024-03-30T04:28:02.264 INFO:tasks.workunit.client.0.smithi123.stdout:2/145: read - d0/d1f/d1c/f22 zero size 2024-03-30T04:28:02.270 INFO:tasks.workunit.client.0.smithi123.stdout:1/259: rename db/l1f to db/dd/d42/d46/d39/l5a 0 2024-03-30T04:28:02.274 INFO:tasks.workunit.client.0.smithi123.stdout:1/260: truncate db/dd/d42/d46/f2e 193395 0 2024-03-30T04:28:02.275 INFO:tasks.workunit.client.0.smithi123.stdout:5/202: mkdir dd/d19/d3e 0 2024-03-30T04:28:02.277 INFO:tasks.workunit.client.0.smithi123.stdout:7/118: dwrite d2/f1b [0,4194304] 0 2024-03-30T04:28:02.280 INFO:tasks.workunit.client.0.smithi123.stdout:3/230: creat d13/d24/d31/d30/f45 x:0 0 0 2024-03-30T04:28:02.286 INFO:tasks.workunit.client.0.smithi123.stdout:5/203: rename ca to dd/c3f 0 2024-03-30T04:28:02.289 INFO:tasks.workunit.client.0.smithi123.stdout:7/119: symlink d2/da/d11/l2b 0 2024-03-30T04:28:02.294 INFO:tasks.workunit.client.0.smithi123.stdout:3/231: symlink d13/d24/d39/l46 0 2024-03-30T04:28:02.300 INFO:tasks.workunit.client.0.smithi123.stdout:8/191: dread f5 [0,4194304] 0 2024-03-30T04:28:02.301 INFO:tasks.workunit.client.0.smithi123.stdout:1/261: rmdir db/dd/d42 39 2024-03-30T04:28:02.303 INFO:tasks.workunit.client.0.smithi123.stdout:5/204: dread dd/f32 [0,4194304] 0 2024-03-30T04:28:02.309 INFO:tasks.workunit.client.0.smithi123.stdout:3/232: rename d13/d24/c44 to d13/d24/d31/d23/c47 0 2024-03-30T04:28:02.310 INFO:tasks.workunit.client.0.smithi123.stdout:3/233: fdatasync fe 0 2024-03-30T04:28:02.312 INFO:tasks.workunit.client.0.smithi123.stdout:8/192: creat da/dc/d14/d24/d2d/f2e x:0 0 0 2024-03-30T04:28:02.313 INFO:tasks.workunit.client.0.smithi123.stdout:8/193: fsync f8 0 2024-03-30T04:28:02.314 INFO:tasks.workunit.client.0.smithi123.stdout:7/120: rmdir d2/da/d11 39 2024-03-30T04:28:02.320 INFO:tasks.workunit.client.0.smithi123.stdout:5/205: getdents dd/d19/d1f/d35/d3a 0 2024-03-30T04:28:02.327 INFO:tasks.workunit.client.0.smithi123.stdout:1/262: rename db/d1b/c27 to db/dd/d42/c5b 0 2024-03-30T04:28:02.333 INFO:tasks.workunit.client.0.smithi123.stdout:7/121: mknod d2/da/d11/c2c 0 2024-03-30T04:28:02.337 INFO:tasks.workunit.client.0.smithi123.stdout:3/234: rmdir d13/d24/d31/d23/d3a/d3c/d3d 0 2024-03-30T04:28:02.339 INFO:tasks.workunit.client.0.smithi123.stdout:5/206: link dd/f1d dd/d19/d1f/d35/f40 0 2024-03-30T04:28:02.341 INFO:tasks.workunit.client.0.smithi123.stdout:5/207: fsync dd/f2f 0 2024-03-30T04:28:02.343 INFO:tasks.workunit.client.0.smithi123.stdout:3/235: unlink d13/d24/d31/d23/d3a/f3e 0 2024-03-30T04:28:02.345 INFO:tasks.workunit.client.0.smithi123.stdout:5/208: fdatasync f9 0 2024-03-30T04:28:02.347 INFO:tasks.workunit.client.0.smithi123.stdout:3/236: write d13/d24/d31/d23/d3a/f40 [2423706,31197] 0 2024-03-30T04:28:02.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:02 smithi135 ceph-mon[44364]: pgmap v34: 65 pgs: 65 active+clean; 454 MiB data, 3.2 GiB used, 533 GiB / 536 GiB avail; 2.7 MiB/s rd, 40 MiB/s wr, 358 op/s 2024-03-30T04:28:02.356 INFO:tasks.workunit.client.0.smithi123.stdout:3/237: truncate d13/d24/d31/d30/f45 736971 0 2024-03-30T04:28:02.360 INFO:tasks.workunit.client.0.smithi123.stdout:8/194: dwrite da/dc/d14/f1d [0,4194304] 0 2024-03-30T04:28:02.360 INFO:tasks.workunit.client.0.smithi123.stdout:7/122: dwrite d2/da/d11/f19 [0,4194304] 0 2024-03-30T04:28:02.363 INFO:tasks.workunit.client.0.smithi123.stdout:1/263: dread db/d1b/f37 [0,4194304] 0 2024-03-30T04:28:02.364 INFO:tasks.workunit.client.0.smithi123.stdout:3/238: read d13/d24/d31/d23/f2b [2352840,27009] 0 2024-03-30T04:28:02.369 INFO:tasks.workunit.client.0.smithi123.stdout:8/195: write da/ff [986989,89402] 0 2024-03-30T04:28:02.376 INFO:tasks.workunit.client.0.smithi123.stdout:7/123: rename d2/d28 to d2/da/d2d 0 2024-03-30T04:28:02.376 INFO:tasks.workunit.client.0.smithi123.stdout:5/209: dwrite fb [0,4194304] 0 2024-03-30T04:28:02.376 INFO:tasks.workunit.client.0.smithi123.stdout:3/239: write d13/d24/d31/d23/f38 [3815807,93479] 0 2024-03-30T04:28:02.378 INFO:tasks.workunit.client.0.smithi123.stdout:7/124: chown d2/da 239299 1 2024-03-30T04:28:02.378 INFO:tasks.workunit.client.0.smithi123.stdout:5/210: chown dd/f16 195 1 2024-03-30T04:28:02.379 INFO:tasks.workunit.client.0.smithi123.stdout:3/240: readlink d13/d24/d31/d23/l26 0 2024-03-30T04:28:02.379 INFO:tasks.workunit.client.0.smithi123.stdout:1/264: mkdir db/dd/d42/d46/d56/d5c 0 2024-03-30T04:28:02.383 INFO:tasks.workunit.client.0.smithi123.stdout:8/196: symlink da/l2f 0 2024-03-30T04:28:02.383 INFO:tasks.workunit.client.0.smithi123.stdout:7/125: write d2/da/fb [3785720,46635] 0 2024-03-30T04:28:02.389 INFO:tasks.workunit.client.0.smithi123.stdout:7/126: unlink d2/f1b 0 2024-03-30T04:28:02.393 INFO:tasks.workunit.client.0.smithi123.stdout:3/241: link d13/d24/d31/d23/c3b d13/c48 0 2024-03-30T04:28:02.398 INFO:tasks.workunit.client.0.smithi123.stdout:7/127: dread d2/da/d11/f1e [0,4194304] 0 2024-03-30T04:28:02.399 INFO:tasks.workunit.client.0.smithi123.stdout:1/265: fsync db/d1b/f2a 0 2024-03-30T04:28:02.405 INFO:tasks.workunit.client.0.smithi123.stdout:1/266: write db/dd/d42/d46/f47 [522319,70404] 0 2024-03-30T04:28:02.408 INFO:tasks.workunit.client.0.smithi123.stdout:8/197: dwrite da/dc/d14/f1d [0,4194304] 0 2024-03-30T04:28:02.411 INFO:tasks.workunit.client.0.smithi123.stdout:1/267: mkdir db/dd/d42/d46/d5d 0 2024-03-30T04:28:02.412 INFO:tasks.workunit.client.0.smithi123.stdout:8/198: stat da/dc/f1a 0 2024-03-30T04:28:02.417 INFO:tasks.workunit.client.0.smithi123.stdout:1/268: truncate db/dd/d42/d46/d39/f57 655847 0 2024-03-30T04:28:02.417 INFO:tasks.workunit.client.0.smithi123.stdout:3/242: truncate d13/d24/d31/d23/f38 2503847 0 2024-03-30T04:28:02.422 INFO:tasks.workunit.client.0.smithi123.stdout:1/269: mkdir db/dd/d42/d46/d56/d5e 0 2024-03-30T04:28:02.424 INFO:tasks.workunit.client.0.smithi123.stdout:1/270: chown db/dd/f48 72727 1 2024-03-30T04:28:02.431 INFO:tasks.workunit.client.0.smithi123.stdout:7/128: dread d2/dc/f17 [0,4194304] 0 2024-03-30T04:28:02.436 INFO:tasks.workunit.client.0.smithi123.stdout:7/129: chown d2/l1a 4 1 2024-03-30T04:28:02.438 INFO:tasks.workunit.client.0.smithi123.stdout:8/199: dread da/dc/f1a [0,4194304] 0 2024-03-30T04:28:02.442 INFO:tasks.workunit.client.0.smithi123.stdout:8/200: rmdir da 39 2024-03-30T04:28:02.442 INFO:tasks.workunit.client.0.smithi123.stdout:7/130: write d2/da/d2d/f2a [5407476,124] 0 2024-03-30T04:28:02.447 INFO:tasks.workunit.client.0.smithi123.stdout:3/243: dwrite d13/d24/d39/f41 [0,4194304] 0 2024-03-30T04:28:02.447 INFO:tasks.workunit.client.0.smithi123.stdout:0/149: sync 2024-03-30T04:28:02.447 INFO:tasks.workunit.client.0.smithi123.stdout:9/164: sync 2024-03-30T04:28:02.451 INFO:tasks.workunit.client.0.smithi123.stdout:1/271: dwrite db/d1b/f1a [0,4194304] 0 2024-03-30T04:28:02.453 INFO:tasks.workunit.client.0.smithi123.stdout:0/150: mkdir d6/dd/d18/d2f 0 2024-03-30T04:28:02.454 INFO:tasks.workunit.client.0.smithi123.stdout:1/272: stat db/dd/d42/d46/d56/d5e 0 2024-03-30T04:28:02.454 INFO:tasks.workunit.client.0.smithi123.stdout:3/244: rmdir d13/d24/d39 39 2024-03-30T04:28:02.455 INFO:tasks.workunit.client.0.smithi123.stdout:0/151: chown d6/dd/d18/c1c 0 1 2024-03-30T04:28:02.456 INFO:tasks.workunit.client.0.smithi123.stdout:9/165: truncate d6/f26 4696165 0 2024-03-30T04:28:02.458 INFO:tasks.workunit.client.0.smithi123.stdout:8/201: link da/f11 da/dc/d14/d18/f30 0 2024-03-30T04:28:02.458 INFO:tasks.workunit.client.0.smithi123.stdout:0/152: chown d6/dd/d18/d2f 4070 1 2024-03-30T04:28:02.460 INFO:tasks.workunit.client.0.smithi123.stdout:0/153: fdatasync d6/f12 0 2024-03-30T04:28:02.465 INFO:tasks.workunit.client.0.smithi123.stdout:1/273: mkdir db/dd/d5f 0 2024-03-30T04:28:02.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:02 smithi123 ceph-mon[35245]: pgmap v34: 65 pgs: 65 active+clean; 454 MiB data, 3.2 GiB used, 533 GiB / 536 GiB avail; 2.7 MiB/s rd, 40 MiB/s wr, 358 op/s 2024-03-30T04:28:02.468 INFO:tasks.workunit.client.0.smithi123.stdout:0/154: dread d6/dd/d18/d1f/f27 [0,4194304] 0 2024-03-30T04:28:02.478 INFO:tasks.workunit.client.0.smithi123.stdout:9/166: getdents d6/d1f 0 2024-03-30T04:28:02.480 INFO:tasks.workunit.client.0.smithi123.stdout:0/155: symlink d6/dd/d18/l30 0 2024-03-30T04:28:02.481 INFO:tasks.workunit.client.0.smithi123.stdout:7/131: dwrite d2/da/d2d/f2a [0,4194304] 0 2024-03-30T04:28:02.485 INFO:tasks.workunit.client.0.smithi123.stdout:7/132: chown d2/da/d11/f1e 53 1 2024-03-30T04:28:02.486 INFO:tasks.workunit.client.0.smithi123.stdout:8/202: dread da/dc/d14/f1d [0,4194304] 0 2024-03-30T04:28:02.493 INFO:tasks.workunit.client.0.smithi123.stdout:1/274: dwrite db/d1b/f2a [0,4194304] 0 2024-03-30T04:28:02.500 INFO:tasks.workunit.client.0.smithi123.stdout:3/245: dread d13/f14 [0,4194304] 0 2024-03-30T04:28:02.503 INFO:tasks.workunit.client.0.smithi123.stdout:9/167: dwrite d6/f26 [0,4194304] 0 2024-03-30T04:28:02.511 INFO:tasks.workunit.client.0.smithi123.stdout:8/203: truncate da/ff 269295 0 2024-03-30T04:28:02.513 INFO:tasks.workunit.client.0.smithi123.stdout:3/246: link d13/d24/c34 d13/d24/c49 0 2024-03-30T04:28:02.520 INFO:tasks.workunit.client.0.smithi123.stdout:3/247: link d13/f1d d13/f4a 0 2024-03-30T04:28:02.522 INFO:tasks.workunit.client.0.smithi123.stdout:1/275: dwrite db/dd/d42/d46/d39/f52 [0,4194304] 0 2024-03-30T04:28:02.525 INFO:tasks.workunit.client.0.smithi123.stdout:1/276: unlink c0 0 2024-03-30T04:28:02.528 INFO:tasks.workunit.client.0.smithi123.stdout:3/248: fsync d13/d24/d31/d23/f25 0 2024-03-30T04:28:02.528 INFO:tasks.workunit.client.0.smithi123.stdout:9/168: dwrite d6/f26 [0,4194304] 0 2024-03-30T04:28:02.533 INFO:tasks.workunit.client.0.smithi123.stdout:3/249: creat d13/d24/d31/f4b x:0 0 0 2024-03-30T04:28:02.537 INFO:tasks.workunit.client.0.smithi123.stdout:1/277: creat db/f60 x:0 0 0 2024-03-30T04:28:02.544 INFO:tasks.workunit.client.0.smithi123.stdout:9/169: dread d6/d23/f3b [0,4194304] 0 2024-03-30T04:28:02.544 INFO:tasks.workunit.client.0.smithi123.stdout:3/250: rename d13/l33 to d13/d24/l4c 0 2024-03-30T04:28:02.546 INFO:tasks.workunit.client.0.smithi123.stdout:9/170: dread - d6/d1f/f2e zero size 2024-03-30T04:28:02.547 INFO:tasks.workunit.client.0.smithi123.stdout:3/251: rmdir d13 39 2024-03-30T04:28:02.550 INFO:tasks.workunit.client.0.smithi123.stdout:9/171: read f0 [3425192,126265] 0 2024-03-30T04:28:02.554 INFO:tasks.workunit.client.0.smithi123.stdout:3/252: readlink d13/d24/l4c 0 2024-03-30T04:28:02.556 INFO:tasks.workunit.client.0.smithi123.stdout:3/253: read d13/d24/d31/d30/f45 [302203,79124] 0 2024-03-30T04:28:02.558 INFO:tasks.workunit.client.0.smithi123.stdout:1/278: truncate db/dd/d42/d46/d39/f52 1603532 0 2024-03-30T04:28:02.562 INFO:tasks.workunit.client.0.smithi123.stdout:3/254: creat d13/d24/f4d x:0 0 0 2024-03-30T04:28:02.566 INFO:tasks.workunit.client.0.smithi123.stdout:1/279: rename db/d1b/f1a to db/dd/d42/f61 0 2024-03-30T04:28:02.570 INFO:tasks.workunit.client.0.smithi123.stdout:1/280: mkdir db/dd/d42/d46/d5d/d62 0 2024-03-30T04:28:02.572 INFO:tasks.workunit.client.0.smithi123.stdout:3/255: write ff [1233760,93232] 0 2024-03-30T04:28:02.575 INFO:tasks.workunit.client.0.smithi123.stdout:1/281: unlink db/dd/d42/c36 0 2024-03-30T04:28:02.577 INFO:tasks.workunit.client.0.smithi123.stdout:3/256: write d13/d24/d31/d23/f2b [2943168,121798] 0 2024-03-30T04:28:02.581 INFO:tasks.workunit.client.0.smithi123.stdout:3/257: write f11 [1494232,124823] 0 2024-03-30T04:28:02.595 INFO:tasks.workunit.client.0.smithi123.stdout:1/282: dwrite db/d1b/f4b [0,4194304] 0 2024-03-30T04:28:02.598 INFO:tasks.workunit.client.0.smithi123.stdout:1/283: mknod db/d1b/c63 0 2024-03-30T04:28:02.602 INFO:tasks.workunit.client.0.smithi123.stdout:3/258: dwrite d13/d24/d31/d23/f2b [0,4194304] 0 2024-03-30T04:28:02.606 INFO:tasks.workunit.client.0.smithi123.stdout:1/284: creat db/dd/d42/d46/d56/d5c/f64 x:0 0 0 2024-03-30T04:28:02.606 INFO:tasks.workunit.client.0.smithi123.stdout:3/259: mkdir d13/d24/d31/d23/d3a/d3c/d4e 0 2024-03-30T04:28:02.608 INFO:tasks.workunit.client.0.smithi123.stdout:1/285: mknod db/dd/c65 0 2024-03-30T04:28:02.612 INFO:tasks.workunit.client.0.smithi123.stdout:3/260: symlink d13/d24/d39/l4f 0 2024-03-30T04:28:02.618 INFO:tasks.workunit.client.0.smithi123.stdout:3/261: rename d13/d24/d31/f35 to d13/d24/f50 0 2024-03-30T04:28:02.620 INFO:tasks.workunit.client.0.smithi123.stdout:3/262: write fd [3140794,13257] 0 2024-03-30T04:28:02.642 INFO:tasks.workunit.client.0.smithi123.stdout:1/286: dread db/fe [0,4194304] 0 2024-03-30T04:28:02.645 INFO:tasks.workunit.client.0.smithi123.stdout:3/263: dread d13/d24/d31/d23/f38 [0,4194304] 0 2024-03-30T04:28:02.647 INFO:tasks.workunit.client.0.smithi123.stdout:3/264: rename d13/d24 to d13/d24/d31/d23/d3a/d51 22 2024-03-30T04:28:02.651 INFO:tasks.workunit.client.0.smithi123.stdout:3/265: mknod d13/d24/d31/d23/d3a/d3c/d4e/c52 0 2024-03-30T04:28:02.659 INFO:tasks.workunit.client.0.smithi123.stdout:1/287: dwrite db/dd/d42/d46/d39/f52 [0,4194304] 0 2024-03-30T04:28:02.665 INFO:tasks.workunit.client.0.smithi123.stdout:1/288: mknod db/dd/c66 0 2024-03-30T04:28:02.668 INFO:tasks.workunit.client.0.smithi123.stdout:1/289: getdents db/dd/d5f 0 2024-03-30T04:28:02.672 INFO:tasks.workunit.client.0.smithi123.stdout:1/290: symlink db/dd/d42/d46/d56/l67 0 2024-03-30T04:28:02.676 INFO:tasks.workunit.client.0.smithi123.stdout:1/291: creat db/dd/d42/f68 x:0 0 0 2024-03-30T04:28:02.680 INFO:tasks.workunit.client.0.smithi123.stdout:1/292: dread - db/f45 zero size 2024-03-30T04:28:02.691 INFO:tasks.workunit.client.0.smithi123.stdout:3/266: dread d13/f1b [0,4194304] 0 2024-03-30T04:28:02.693 INFO:tasks.workunit.client.0.smithi123.stdout:3/267: stat d13/d24/d39 0 2024-03-30T04:28:02.699 INFO:tasks.workunit.client.0.smithi123.stdout:3/268: creat d13/d24/f53 x:0 0 0 2024-03-30T04:28:02.702 INFO:tasks.workunit.client.0.smithi123.stdout:1/293: dwrite db/fe [0,4194304] 0 2024-03-30T04:28:02.702 INFO:tasks.workunit.client.0.smithi123.stdout:3/269: creat d13/d24/d31/d23/d3a/f54 x:0 0 0 2024-03-30T04:28:02.704 INFO:tasks.workunit.client.0.smithi123.stdout:3/270: chown d13/d24/d31/d23/f25 1 1 2024-03-30T04:28:02.706 INFO:tasks.workunit.client.0.smithi123.stdout:3/271: chown d13/d24/d31/d23/d3a/d3c/d4e 0 1 2024-03-30T04:28:02.707 INFO:tasks.workunit.client.0.smithi123.stdout:1/294: rename db/dd/c65 to db/dd/d42/d46/d56/d5c/c69 0 2024-03-30T04:28:02.711 INFO:tasks.workunit.client.0.smithi123.stdout:1/295: write db/dd/d42/d46/d39/f57 [702758,79752] 0 2024-03-30T04:28:02.711 INFO:tasks.workunit.client.0.smithi123.stdout:3/272: truncate d13/d24/d31/d23/d3a/f40 4990817 0 2024-03-30T04:28:02.719 INFO:tasks.workunit.client.0.smithi123.stdout:1/296: rename db/dd/d42/d46/d56/l67 to db/dd/d42/d46/l6a 0 2024-03-30T04:28:02.732 INFO:tasks.workunit.client.0.smithi123.stdout:3/273: dread d13/d24/d31/d23/f2e [0,4194304] 0 2024-03-30T04:28:02.745 INFO:tasks.workunit.client.0.smithi123.stdout:3/274: mknod d13/d24/d31/d23/d3a/d3c/d4e/c55 0 2024-03-30T04:28:02.747 INFO:tasks.workunit.client.0.smithi123.stdout:3/275: chown d13/f1d 12028497 1 2024-03-30T04:28:02.750 INFO:tasks.workunit.client.0.smithi123.stdout:1/297: creat db/dd/d42/d46/d5d/d62/f6b x:0 0 0 2024-03-30T04:28:02.752 INFO:tasks.workunit.client.0.smithi123.stdout:1/298: fsync db/dd/d42/d46/f2e 0 2024-03-30T04:28:02.762 INFO:tasks.workunit.client.0.smithi123.stdout:3/276: dread f11 [0,4194304] 0 2024-03-30T04:28:02.763 INFO:tasks.workunit.client.0.smithi123.stdout:3/277: chown d13/d24 33880117 1 2024-03-30T04:28:02.778 INFO:tasks.workunit.client.0.smithi123.stdout:3/278: dwrite fd [0,4194304] 0 2024-03-30T04:28:02.784 INFO:tasks.workunit.client.0.smithi123.stdout:3/279: creat d13/d24/d31/d30/f56 x:0 0 0 2024-03-30T04:28:02.785 INFO:tasks.workunit.client.0.smithi123.stdout:3/280: chown d13/f1b 5 1 2024-03-30T04:28:02.787 INFO:tasks.workunit.client.0.smithi123.stdout:3/281: chown d13/d24/d31/d30/f56 23617 1 2024-03-30T04:28:02.787 INFO:tasks.workunit.client.0.smithi123.stdout:1/299: dread f8 [0,4194304] 0 2024-03-30T04:28:02.789 INFO:tasks.workunit.client.0.smithi123.stdout:1/300: chown db/dd/d42/l3d 5 1 2024-03-30T04:28:02.792 INFO:tasks.workunit.client.0.smithi123.stdout:3/282: link d13/d24/f50 d13/d24/f57 0 2024-03-30T04:28:02.793 INFO:tasks.workunit.client.0.smithi123.stdout:3/283: dread - d13/d24/f4d zero size 2024-03-30T04:28:02.797 INFO:tasks.workunit.client.0.smithi123.stdout:3/284: unlink d13/d24/d31/d23/f38 0 2024-03-30T04:28:02.801 INFO:tasks.workunit.client.0.smithi123.stdout:1/301: dwrite f8 [0,4194304] 0 2024-03-30T04:28:02.804 INFO:tasks.workunit.client.0.smithi123.stdout:1/302: chown db/dd/d42/c5b 0 1 2024-03-30T04:28:02.807 INFO:tasks.workunit.client.0.smithi123.stdout:1/303: write db/f20 [3135123,52139] 0 2024-03-30T04:28:02.810 INFO:tasks.workunit.client.0.smithi123.stdout:1/304: creat db/dd/d42/f6c x:0 0 0 2024-03-30T04:28:02.812 INFO:tasks.workunit.client.0.smithi123.stdout:1/305: write db/dd/d42/f68 [152781,22095] 0 2024-03-30T04:28:02.813 INFO:tasks.workunit.client.0.smithi123.stdout:1/306: write db/fe [362322,44971] 0 2024-03-30T04:28:02.816 INFO:tasks.workunit.client.0.smithi123.stdout:1/307: mkdir db/dd/d5f/d6d 0 2024-03-30T04:28:02.817 INFO:tasks.workunit.client.0.smithi123.stdout:1/308: dread - db/dd/f54 zero size 2024-03-30T04:28:02.818 INFO:tasks.workunit.client.0.smithi123.stdout:1/309: dread - db/f60 zero size 2024-03-30T04:28:02.820 INFO:tasks.workunit.client.0.smithi123.stdout:1/310: chown db/d1b/c15 34583108 1 2024-03-30T04:28:02.821 INFO:tasks.workunit.client.0.smithi123.stdout:1/311: fsync db/dd/d42/d46/d39/f57 0 2024-03-30T04:28:02.822 INFO:tasks.workunit.client.0.smithi123.stdout:4/244: sync 2024-03-30T04:28:02.822 INFO:tasks.workunit.client.0.smithi123.stdout:6/279: sync 2024-03-30T04:28:02.822 INFO:tasks.workunit.client.0.smithi123.stdout:1/312: dread - db/dd/d42/d46/f53 zero size 2024-03-30T04:28:02.825 INFO:tasks.workunit.client.0.smithi123.stdout:4/245: write d2/d4/d2c/d42/f3a [677688,77144] 0 2024-03-30T04:28:02.825 INFO:tasks.workunit.client.0.smithi123.stdout:1/313: creat db/f6e x:0 0 0 2024-03-30T04:28:02.828 INFO:tasks.workunit.client.0.smithi123.stdout:1/314: rmdir db/dd/d5f 39 2024-03-30T04:28:02.828 INFO:tasks.workunit.client.0.smithi123.stdout:6/280: link d10/d13/d1c/c4b d10/d20/d3f/c5c 0 2024-03-30T04:28:02.830 INFO:tasks.workunit.client.0.smithi123.stdout:1/315: chown db/dd/d42/d46/d56/d5c/f64 0 1 2024-03-30T04:28:02.832 INFO:tasks.workunit.client.0.smithi123.stdout:4/246: rename d2/d4/f2b to d2/d4/d2c/d42/d35/f5c 0 2024-03-30T04:28:02.835 INFO:tasks.workunit.client.0.smithi123.stdout:6/281: creat d10/d13/d1c/f5d x:0 0 0 2024-03-30T04:28:02.836 INFO:tasks.workunit.client.0.smithi123.stdout:1/316: creat db/dd/d42/f6f x:0 0 0 2024-03-30T04:28:02.838 INFO:tasks.workunit.client.0.smithi123.stdout:4/247: unlink d2/d4/f2d 0 2024-03-30T04:28:02.840 INFO:tasks.workunit.client.0.smithi123.stdout:4/248: stat d2/d4/d2c 0 2024-03-30T04:28:02.840 INFO:tasks.workunit.client.0.smithi123.stdout:6/282: symlink d10/d20/d3f/l5e 0 2024-03-30T04:28:02.841 INFO:tasks.workunit.client.0.smithi123.stdout:4/249: truncate d2/f1c 4268097 0 2024-03-30T04:28:02.843 INFO:tasks.workunit.client.0.smithi123.stdout:6/283: unlink d10/d20/f22 0 2024-03-30T04:28:02.846 INFO:tasks.workunit.client.0.smithi123.stdout:4/250: mkdir d2/d4/d2c/d42/d35/d5d 0 2024-03-30T04:28:02.870 INFO:tasks.workunit.client.0.smithi123.stdout:1/317: dread db/d1b/f3e [0,4194304] 0 2024-03-30T04:28:02.870 INFO:tasks.workunit.client.0.smithi123.stdout:6/284: dwrite d10/f3a [4194304,4194304] 0 2024-03-30T04:28:02.874 INFO:tasks.workunit.client.0.smithi123.stdout:6/285: creat d10/d13/d1c/d42/f5f x:0 0 0 2024-03-30T04:28:02.875 INFO:tasks.workunit.client.0.smithi123.stdout:6/286: fdatasync d10/d20/f2b 0 2024-03-30T04:28:02.877 INFO:tasks.workunit.client.0.smithi123.stdout:6/287: readlink d10/d20/d3f/l5e 0 2024-03-30T04:28:02.881 INFO:tasks.workunit.client.0.smithi123.stdout:6/288: write d10/d13/d1c/f1f [632358,103073] 0 2024-03-30T04:28:02.883 INFO:tasks.workunit.client.0.smithi123.stdout:4/251: dread d2/d4/fa [0,4194304] 0 2024-03-30T04:28:02.885 INFO:tasks.workunit.client.0.smithi123.stdout:6/289: mkdir d10/d60 0 2024-03-30T04:28:02.885 INFO:tasks.workunit.client.0.smithi123.stdout:4/252: fsync d2/d34/f4c 0 2024-03-30T04:28:02.886 INFO:tasks.workunit.client.0.smithi123.stdout:1/318: dread db/d1b/f4b [0,4194304] 0 2024-03-30T04:28:02.889 INFO:tasks.workunit.client.0.smithi123.stdout:4/253: chown d2/d4/d7/c57 61623 1 2024-03-30T04:28:02.895 INFO:tasks.workunit.client.0.smithi123.stdout:6/290: rename d10/d13/l43 to d10/l61 0 2024-03-30T04:28:02.896 INFO:tasks.workunit.client.0.smithi123.stdout:4/254: read d2/d4/f16 [6963690,41997] 0 2024-03-30T04:28:02.900 INFO:tasks.workunit.client.0.smithi123.stdout:1/319: rename db/dd/d42/d46/f2b to db/dd/d42/d46/d56/f70 0 2024-03-30T04:28:02.901 INFO:tasks.workunit.client.0.smithi123.stdout:6/291: creat d10/d20/d3f/f62 x:0 0 0 2024-03-30T04:28:02.902 INFO:tasks.workunit.client.0.smithi123.stdout:1/320: rmdir db/dd/d42/d46/d5d 39 2024-03-30T04:28:02.904 INFO:tasks.workunit.client.0.smithi123.stdout:4/255: creat d2/d4/d7/d1a/f5e x:0 0 0 2024-03-30T04:28:02.905 INFO:tasks.workunit.client.0.smithi123.stdout:1/321: write db/dd/d42/f6c [559657,79521] 0 2024-03-30T04:28:02.906 INFO:tasks.workunit.client.0.smithi123.stdout:6/292: rename l5 to d10/d13/d45/l63 0 2024-03-30T04:28:02.906 INFO:tasks.workunit.client.0.smithi123.stdout:4/256: chown d2/d4/l12 0 1 2024-03-30T04:28:02.912 INFO:tasks.workunit.client.0.smithi123.stdout:4/257: creat d2/d4/d7/d1a/f5f x:0 0 0 2024-03-30T04:28:02.914 INFO:tasks.workunit.client.0.smithi123.stdout:1/322: rename db/dd/d42/d46/l44 to db/dd/d42/d46/d5d/l71 0 2024-03-30T04:28:02.914 INFO:tasks.workunit.client.0.smithi123.stdout:6/293: link d10/d13/d1c/c2f d10/d13/d1c/d42/c64 0 2024-03-30T04:28:02.916 INFO:tasks.workunit.client.0.smithi123.stdout:4/258: creat d2/d26/f60 x:0 0 0 2024-03-30T04:28:02.918 INFO:tasks.workunit.client.0.smithi123.stdout:6/294: chown d10/l18 1697 1 2024-03-30T04:28:02.918 INFO:tasks.workunit.client.0.smithi123.stdout:6/295: fsync d10/f3a 0 2024-03-30T04:28:02.920 INFO:tasks.workunit.client.0.smithi123.stdout:6/296: readlink d10/d20/l24 0 2024-03-30T04:28:02.923 INFO:tasks.workunit.client.0.smithi123.stdout:6/297: write d10/d13/d1c/d42/f51 [957046,73522] 0 2024-03-30T04:28:02.927 INFO:tasks.workunit.client.0.smithi123.stdout:6/298: creat d10/d20/d3f/f65 x:0 0 0 2024-03-30T04:28:02.932 INFO:tasks.workunit.client.0.smithi123.stdout:1/323: dwrite db/f60 [0,4194304] 0 2024-03-30T04:28:02.933 INFO:tasks.workunit.client.0.smithi123.stdout:6/299: creat d10/d13/d1c/d4c/f66 x:0 0 0 2024-03-30T04:28:02.937 INFO:tasks.workunit.client.0.smithi123.stdout:4/259: dwrite d2/d4/d2c/d42/f38 [0,4194304] 0 2024-03-30T04:28:02.939 INFO:tasks.workunit.client.0.smithi123.stdout:6/300: creat d10/d13/d45/f67 x:0 0 0 2024-03-30T04:28:02.940 INFO:tasks.workunit.client.0.smithi123.stdout:1/324: rmdir db/dd/d5f 39 2024-03-30T04:28:02.952 INFO:tasks.workunit.client.0.smithi123.stdout:4/260: rename d2/l1e to d2/d4/d2c/d42/d35/d5d/l61 0 2024-03-30T04:28:02.955 INFO:tasks.workunit.client.0.smithi123.stdout:1/325: dread db/dd/d42/f6c [0,4194304] 0 2024-03-30T04:28:02.956 INFO:tasks.workunit.client.0.smithi123.stdout:6/301: creat d10/f68 x:0 0 0 2024-03-30T04:28:02.958 INFO:tasks.workunit.client.0.smithi123.stdout:6/302: fsync d10/d20/f3b 0 2024-03-30T04:28:02.960 INFO:tasks.workunit.client.0.smithi123.stdout:4/261: symlink d2/l62 0 2024-03-30T04:28:02.960 INFO:tasks.workunit.client.0.smithi123.stdout:2/146: sync 2024-03-30T04:28:02.960 INFO:tasks.workunit.client.0.smithi123.stdout:5/211: sync 2024-03-30T04:28:02.962 INFO:tasks.workunit.client.0.smithi123.stdout:5/212: dread - dd/d19/f22 zero size 2024-03-30T04:28:02.962 INFO:tasks.workunit.client.0.smithi123.stdout:5/213: readlink l4 0 2024-03-30T04:28:02.963 INFO:tasks.workunit.client.0.smithi123.stdout:6/303: mknod d10/d60/c69 0 2024-03-30T04:28:02.964 INFO:tasks.workunit.client.0.smithi123.stdout:5/214: chown dd/l39 5 1 2024-03-30T04:28:02.967 INFO:tasks.workunit.client.0.smithi123.stdout:4/262: mkdir d2/d4/d7/d63 0 2024-03-30T04:28:02.967 INFO:tasks.workunit.client.0.smithi123.stdout:5/215: read - dd/d19/f2b zero size 2024-03-30T04:28:02.969 INFO:tasks.workunit.client.0.smithi123.stdout:2/147: mknod d0/d1f/d1c/d2a/c2c 0 2024-03-30T04:28:02.969 INFO:tasks.workunit.client.0.smithi123.stdout:4/263: write d2/d4/f8 [1119850,21262] 0 2024-03-30T04:28:02.972 INFO:tasks.workunit.client.0.smithi123.stdout:2/148: dread - d0/d1f/d16/f23 zero size 2024-03-30T04:28:02.972 INFO:tasks.workunit.client.0.smithi123.stdout:5/216: mkdir dd/d41 0 2024-03-30T04:28:02.975 INFO:tasks.workunit.client.0.smithi123.stdout:1/326: dread db/d1b/f37 [0,4194304] 0 2024-03-30T04:28:02.977 INFO:tasks.workunit.client.0.smithi123.stdout:2/149: creat d0/d1f/f2d x:0 0 0 2024-03-30T04:28:02.978 INFO:tasks.workunit.client.0.smithi123.stdout:1/327: write db/dd/f4d [1062829,123430] 0 2024-03-30T04:28:02.979 INFO:tasks.workunit.client.0.smithi123.stdout:2/150: dread - d0/d1f/f2d zero size 2024-03-30T04:28:02.981 INFO:tasks.workunit.client.0.smithi123.stdout:4/264: rename d2/d4/d7/d1a/c32 to d2/d4/d2c/d42/d35/c64 0 2024-03-30T04:28:02.983 INFO:tasks.workunit.client.0.smithi123.stdout:2/151: fdatasync d0/f14 0 2024-03-30T04:28:02.985 INFO:tasks.workunit.client.0.smithi123.stdout:1/328: mkdir db/dd/d42/d46/d51/d72 0 2024-03-30T04:28:02.990 INFO:tasks.workunit.client.0.smithi123.stdout:6/304: dwrite d10/f3a [4194304,4194304] 0 2024-03-30T04:28:02.995 INFO:tasks.workunit.client.0.smithi123.stdout:1/329: fdatasync db/dd/d42/f6c 0 2024-03-30T04:28:03.004 INFO:tasks.workunit.client.0.smithi123.stdout:7/133: sync 2024-03-30T04:28:03.004 INFO:tasks.workunit.client.0.smithi123.stdout:0/156: sync 2024-03-30T04:28:03.004 INFO:tasks.workunit.client.0.smithi123.stdout:9/172: sync 2024-03-30T04:28:03.004 INFO:tasks.workunit.client.0.smithi123.stdout:3/285: sync 2024-03-30T04:28:03.004 INFO:tasks.workunit.client.0.smithi123.stdout:8/204: sync 2024-03-30T04:28:03.011 INFO:tasks.workunit.client.0.smithi123.stdout:0/157: write d6/fe [1518951,36708] 0 2024-03-30T04:28:03.011 INFO:tasks.workunit.client.0.smithi123.stdout:3/286: stat d13/d24/d31/f4b 0 2024-03-30T04:28:03.012 INFO:tasks.workunit.client.0.smithi123.stdout:1/330: rename db/dd/f4d to db/dd/d42/d46/d56/d5e/f73 0 2024-03-30T04:28:03.012 INFO:tasks.workunit.client.0.smithi123.stdout:2/152: dwrite d0/d1f/d27/f29 [0,4194304] 0 2024-03-30T04:28:03.017 INFO:tasks.workunit.client.0.smithi123.stdout:3/287: chown cb 538 1 2024-03-30T04:28:03.020 INFO:tasks.workunit.client.0.smithi123.stdout:6/305: dwrite d10/f29 [0,4194304] 0 2024-03-30T04:28:03.021 INFO:tasks.workunit.client.0.smithi123.stdout:3/288: read - d13/d24/d31/d23/d3a/f54 zero size 2024-03-30T04:28:03.030 INFO:tasks.workunit.client.0.smithi123.stdout:7/134: mknod d2/da/d2d/c2e 0 2024-03-30T04:28:03.035 INFO:tasks.workunit.client.0.smithi123.stdout:0/158: fdatasync f5 0 2024-03-30T04:28:03.041 INFO:tasks.workunit.client.0.smithi123.stdout:0/159: chown d6/c21 8 1 2024-03-30T04:28:03.042 INFO:tasks.workunit.client.0.smithi123.stdout:4/265: link d2/d4/d2c/d42/d35/d5d/l61 d2/l65 0 2024-03-30T04:28:03.043 INFO:tasks.workunit.client.0.smithi123.stdout:9/173: link d6/d1f/f29 d6/d23/f40 0 2024-03-30T04:28:03.056 INFO:tasks.workunit.client.0.smithi123.stdout:4/266: symlink d2/d34/l66 0 2024-03-30T04:28:03.056 INFO:tasks.workunit.client.0.smithi123.stdout:3/289: dwrite d13/d24/d31/f4b [0,4194304] 0 2024-03-30T04:28:03.061 INFO:tasks.workunit.client.0.smithi123.stdout:8/205: dread f6 [4194304,4194304] 0 2024-03-30T04:28:03.063 INFO:tasks.workunit.client.0.smithi123.stdout:6/306: rename d10/d60 to d10/d13/d6a 0 2024-03-30T04:28:03.066 INFO:tasks.workunit.client.0.smithi123.stdout:7/135: write d2/dc/d1c/f23 [28712,39749] 0 2024-03-30T04:28:03.066 INFO:tasks.workunit.client.0.smithi123.stdout:2/153: link d0/le d0/l2e 0 2024-03-30T04:28:03.066 INFO:tasks.workunit.client.0.smithi123.stdout:5/217: dwrite f0 [0,4194304] 0 2024-03-30T04:28:03.077 INFO:tasks.workunit.client.0.smithi123.stdout:3/290: mknod d13/d24/d31/d30/c58 0 2024-03-30T04:28:03.079 INFO:tasks.workunit.client.0.smithi123.stdout:6/307: write d10/f59 [727196,32902] 0 2024-03-30T04:28:03.082 INFO:tasks.workunit.client.0.smithi123.stdout:1/331: dwrite db/d1b/f28 [0,4194304] 0 2024-03-30T04:28:03.087 INFO:tasks.workunit.client.0.smithi123.stdout:0/160: dwrite f5 [0,4194304] 0 2024-03-30T04:28:03.092 INFO:tasks.workunit.client.0.smithi123.stdout:4/267: unlink d2/l23 0 2024-03-30T04:28:03.095 INFO:tasks.workunit.client.0.smithi123.stdout:4/268: chown d2/f45 138 1 2024-03-30T04:28:03.101 INFO:tasks.workunit.client.0.smithi123.stdout:3/291: mknod d13/d24/d31/d23/d3a/d3c/d4e/c59 0 2024-03-30T04:28:03.104 INFO:tasks.workunit.client.0.smithi123.stdout:7/136: creat d2/da/d22/f2f x:0 0 0 2024-03-30T04:28:03.104 INFO:tasks.workunit.client.0.smithi123.stdout:8/206: dwrite f8 [0,4194304] 0 2024-03-30T04:28:03.108 INFO:tasks.workunit.client.0.smithi123.stdout:5/218: dread f0 [0,4194304] 0 2024-03-30T04:28:03.108 INFO:tasks.workunit.client.0.smithi123.stdout:7/137: read d2/da/d11/f19 [2676014,42488] 0 2024-03-30T04:28:03.109 INFO:tasks.workunit.client.0.smithi123.stdout:2/154: rmdir d0/d1f/d25 0 2024-03-30T04:28:03.113 INFO:tasks.workunit.client.0.smithi123.stdout:2/155: readlink d0/d1f/l28 0 2024-03-30T04:28:03.114 INFO:tasks.workunit.client.0.smithi123.stdout:4/269: rename d2/d4/d7/fc to d2/d4/d2c/f67 0 2024-03-30T04:28:03.119 INFO:tasks.workunit.client.0.smithi123.stdout:4/270: chown d2/d4/f16 1479673785 1 2024-03-30T04:28:03.119 INFO:tasks.workunit.client.0.smithi123.stdout:0/161: mkdir d6/dd/d31 0 2024-03-30T04:28:03.121 INFO:tasks.workunit.client.0.smithi123.stdout:9/174: dwrite d6/d23/f33 [0,4194304] 0 2024-03-30T04:28:03.122 INFO:tasks.workunit.client.0.smithi123.stdout:1/332: link db/f59 db/dd/d42/d46/d56/d5c/f74 0 2024-03-30T04:28:03.125 INFO:tasks.workunit.client.0.smithi123.stdout:1/333: write db/f6e [962344,17553] 0 2024-03-30T04:28:03.129 INFO:tasks.workunit.client.0.smithi123.stdout:9/175: truncate d6/d1f/f2e 563015 0 2024-03-30T04:28:03.130 INFO:tasks.workunit.client.0.smithi123.stdout:8/207: mkdir da/d27/d31 0 2024-03-30T04:28:03.135 INFO:tasks.workunit.client.0.smithi123.stdout:6/308: getdents d10/d13/d6a 0 2024-03-30T04:28:03.138 INFO:tasks.workunit.client.0.smithi123.stdout:5/219: mknod dd/d19/d3e/c42 0 2024-03-30T04:28:03.148 INFO:tasks.workunit.client.0.smithi123.stdout:9/176: rename d6/fc to d6/d1f/f41 0 2024-03-30T04:28:03.151 INFO:tasks.workunit.client.0.smithi123.stdout:5/220: write dd/d19/d1f/d35/f38 [908366,10072] 0 2024-03-30T04:28:03.151 INFO:tasks.workunit.client.0.smithi123.stdout:3/292: dread d13/d24/f57 [0,4194304] 0 2024-03-30T04:28:03.153 INFO:tasks.workunit.client.0.smithi123.stdout:2/156: dwrite d0/d1f/d16/f1e [0,4194304] 0 2024-03-30T04:28:03.154 INFO:tasks.workunit.client.0.smithi123.stdout:0/162: dwrite d6/f24 [0,4194304] 0 2024-03-30T04:28:03.155 INFO:tasks.workunit.client.0.smithi123.stdout:8/208: rmdir da/d27/d31 0 2024-03-30T04:28:03.158 INFO:tasks.workunit.client.0.smithi123.stdout:1/334: dread db/f60 [0,4194304] 0 2024-03-30T04:28:03.159 INFO:tasks.workunit.client.0.smithi123.stdout:6/309: symlink d10/d13/d1c/d5b/l6b 0 2024-03-30T04:28:03.160 INFO:tasks.workunit.client.0.smithi123.stdout:9/177: mknod d6/d23/c42 0 2024-03-30T04:28:03.161 INFO:tasks.workunit.client.0.smithi123.stdout:3/293: write d13/d24/d31/d23/f2e [3327142,103321] 0 2024-03-30T04:28:03.163 INFO:tasks.workunit.client.0.smithi123.stdout:2/157: symlink d0/d1f/d1c/d2a/l2f 0 2024-03-30T04:28:03.163 INFO:tasks.workunit.client.0.smithi123.stdout:6/310: truncate d10/d13/f17 749817 0 2024-03-30T04:28:03.165 INFO:tasks.workunit.client.0.smithi123.stdout:9/178: chown d6/fb 1472 1 2024-03-30T04:28:03.169 INFO:tasks.workunit.client.0.smithi123.stdout:4/271: rename d2/d4/d2c/c41 to d2/d34/c68 0 2024-03-30T04:28:03.172 INFO:tasks.workunit.client.0.smithi123.stdout:8/209: creat da/dc/f32 x:0 0 0 2024-03-30T04:28:03.173 INFO:tasks.workunit.client.0.smithi123.stdout:5/221: creat dd/d41/f43 x:0 0 0 2024-03-30T04:28:03.178 INFO:tasks.workunit.client.0.smithi123.stdout:6/311: creat d10/d13/d1c/f6c x:0 0 0 2024-03-30T04:28:03.178 INFO:tasks.workunit.client.0.smithi123.stdout:5/222: read - dd/f2f zero size 2024-03-30T04:28:03.179 INFO:tasks.workunit.client.0.smithi123.stdout:3/294: fsync d13/f4a 0 2024-03-30T04:28:03.181 INFO:tasks.workunit.client.0.smithi123.stdout:7/138: dwrite d2/dc/d1c/f1d [0,4194304] 0 2024-03-30T04:28:03.184 INFO:tasks.workunit.client.0.smithi123.stdout:7/139: fsync d2/da/d11/f1f 0 2024-03-30T04:28:03.187 INFO:tasks.workunit.client.0.smithi123.stdout:7/140: truncate d2/dc/f27 387184 0 2024-03-30T04:28:03.189 INFO:tasks.workunit.client.0.smithi123.stdout:4/272: dwrite d2/d34/f3f [0,4194304] 0 2024-03-30T04:28:03.232 INFO:tasks.workunit.client.0.smithi123.stdout:2/158: mkdir d0/d1f/d30 0 2024-03-30T04:28:03.265 INFO:tasks.workunit.client.0.smithi123.stdout:1/335: symlink db/dd/l75 0 2024-03-30T04:28:03.265 INFO:tasks.workunit.client.0.smithi123.stdout:9/179: mknod d6/d23/c43 0 2024-03-30T04:28:03.265 INFO:tasks.workunit.client.0.smithi123.stdout:9/180: fdatasync d6/d1f/f2d 0 2024-03-30T04:28:03.266 INFO:tasks.workunit.client.0.smithi123.stdout:5/223: symlink dd/d19/d3e/l44 0 2024-03-30T04:28:03.266 INFO:tasks.workunit.client.0.smithi123.stdout:6/312: mknod d10/d13/d1c/d5b/c6d 0 2024-03-30T04:28:03.266 INFO:tasks.workunit.client.0.smithi123.stdout:6/313: mknod d10/d13/d6a/c6e 0 2024-03-30T04:28:03.266 INFO:tasks.workunit.client.0.smithi123.stdout:7/141: symlink d2/da/l30 0 2024-03-30T04:28:03.266 INFO:tasks.workunit.client.0.smithi123.stdout:4/273: fdatasync d2/d4/f22 0 2024-03-30T04:28:03.266 INFO:tasks.workunit.client.0.smithi123.stdout:2/159: mknod d0/d1f/d30/c31 0 2024-03-30T04:28:03.266 INFO:tasks.workunit.client.0.smithi123.stdout:8/210: link da/dc/d14/l1b da/dc/l33 0 2024-03-30T04:28:03.266 INFO:tasks.workunit.client.0.smithi123.stdout:8/211: stat da/dc/d14/d18/d21 0 2024-03-30T04:28:03.266 INFO:tasks.workunit.client.0.smithi123.stdout:9/181: mknod d6/c44 0 2024-03-30T04:28:03.266 INFO:tasks.workunit.client.0.smithi123.stdout:5/224: creat dd/d19/d1f/d35/d3a/f45 x:0 0 0 2024-03-30T04:28:03.266 INFO:tasks.workunit.client.0.smithi123.stdout:6/314: unlink d10/d20/d3f/f62 0 2024-03-30T04:28:03.266 INFO:tasks.workunit.client.0.smithi123.stdout:4/274: readlink d2/d4/d2c/d42/l44 0 2024-03-30T04:28:03.266 INFO:tasks.workunit.client.0.smithi123.stdout:9/182: rmdir d6/d23 39 2024-03-30T04:28:03.266 INFO:tasks.workunit.client.0.smithi123.stdout:6/315: stat d10/d13/l34 0 2024-03-30T04:28:03.267 INFO:tasks.workunit.client.0.smithi123.stdout:7/142: symlink d2/dc/d1c/l31 0 2024-03-30T04:28:03.267 INFO:tasks.workunit.client.0.smithi123.stdout:7/143: chown d2/dc/l25 723306611 1 2024-03-30T04:28:03.268 INFO:tasks.workunit.client.0.smithi123.stdout:5/225: mknod dd/d19/d1f/d35/c46 0 2024-03-30T04:28:03.269 INFO:tasks.workunit.client.0.smithi123.stdout:1/336: truncate db/d1b/f4b 984674 0 2024-03-30T04:28:03.271 INFO:tasks.workunit.client.0.smithi123.stdout:9/183: unlink d6/f38 0 2024-03-30T04:28:03.275 INFO:tasks.workunit.client.0.smithi123.stdout:6/316: rename f2 to d10/d13/d1c/f6f 0 2024-03-30T04:28:03.276 INFO:tasks.workunit.client.0.smithi123.stdout:1/337: read db/f6e [621896,92322] 0 2024-03-30T04:28:03.278 INFO:tasks.workunit.client.0.smithi123.stdout:1/338: chown db/dd/d42/d46/d56 46 1 2024-03-30T04:28:03.280 INFO:tasks.workunit.client.0.smithi123.stdout:8/212: dwrite f6 [0,4194304] 0 2024-03-30T04:28:03.280 INFO:tasks.workunit.client.0.smithi123.stdout:5/226: unlink dd/l21 0 2024-03-30T04:28:03.281 INFO:tasks.workunit.client.0.smithi123.stdout:2/160: getdents d0/d1f/d1c/d20 0 2024-03-30T04:28:03.283 INFO:tasks.workunit.client.0.smithi123.stdout:9/184: fdatasync d6/d1f/f41 0 2024-03-30T04:28:03.286 INFO:tasks.workunit.client.0.smithi123.stdout:2/161: write d0/d1f/d16/f23 [907988,100356] 0 2024-03-30T04:28:03.287 INFO:tasks.workunit.client.0.smithi123.stdout:6/317: mknod d10/d13/d1c/d5b/c70 0 2024-03-30T04:28:03.290 INFO:tasks.workunit.client.0.smithi123.stdout:5/227: fsync f7 0 2024-03-30T04:28:03.291 INFO:tasks.workunit.client.0.smithi123.stdout:9/185: write d6/d1f/f2e [1483929,120113] 0 2024-03-30T04:28:03.292 INFO:tasks.workunit.client.0.smithi123.stdout:8/213: rmdir da/d27 39 2024-03-30T04:28:03.294 INFO:tasks.workunit.client.0.smithi123.stdout:2/162: truncate d0/d1f/f24 1488059 0 2024-03-30T04:28:03.294 INFO:tasks.workunit.client.0.smithi123.stdout:6/318: creat d10/d13/d1c/f71 x:0 0 0 2024-03-30T04:28:03.295 INFO:tasks.workunit.client.0.smithi123.stdout:5/228: fsync dd/fe 0 2024-03-30T04:28:03.305 INFO:tasks.workunit.client.0.smithi123.stdout:6/319: symlink d10/d13/d1c/d5b/l72 0 2024-03-30T04:28:03.306 INFO:tasks.workunit.client.0.smithi123.stdout:5/229: rename dd/d19/f2c to dd/d41/f47 0 2024-03-30T04:28:03.310 INFO:tasks.workunit.client.0.smithi123.stdout:6/320: mknod d10/d20/c73 0 2024-03-30T04:28:03.312 INFO:tasks.workunit.client.0.smithi123.stdout:7/144: write d2/f13 [3871542,36014] 0 2024-03-30T04:28:03.313 INFO:tasks.workunit.client.0.smithi123.stdout:5/230: mknod dd/d41/c48 0 2024-03-30T04:28:03.314 INFO:tasks.workunit.client.0.smithi123.stdout:1/339: dwrite db/f60 [0,4194304] 0 2024-03-30T04:28:03.317 INFO:tasks.workunit.client.0.smithi123.stdout:6/321: write d10/d13/d1c/f1f [5006589,4263] 0 2024-03-30T04:28:03.321 INFO:tasks.workunit.client.0.smithi123.stdout:1/340: chown db/d1b/c38 12820136 1 2024-03-30T04:28:03.328 INFO:tasks.workunit.client.0.smithi123.stdout:2/163: write d0/f1d [237061,93138] 0 2024-03-30T04:28:03.328 INFO:tasks.workunit.client.0.smithi123.stdout:6/322: read d10/d13/d1c/f1f [4592555,68048] 0 2024-03-30T04:28:03.335 INFO:tasks.workunit.client.0.smithi123.stdout:2/164: stat d0/d1f/d30/c31 0 2024-03-30T04:28:03.336 INFO:tasks.workunit.client.0.smithi123.stdout:1/341: fsync db/dd/d42/d46/d56/d5e/f73 0 2024-03-30T04:28:03.337 INFO:tasks.workunit.client.0.smithi123.stdout:6/323: mkdir d10/d13/d1c/d74 0 2024-03-30T04:28:03.337 INFO:tasks.workunit.client.0.smithi123.stdout:4/275: dwrite d2/d4/d7/d1a/f21 [0,4194304] 0 2024-03-30T04:28:03.340 INFO:tasks.workunit.client.0.smithi123.stdout:8/214: dread da/f12 [0,4194304] 0 2024-03-30T04:28:03.349 INFO:tasks.workunit.client.0.smithi123.stdout:9/186: dwrite d6/d1f/f29 [4194304,4194304] 0 2024-03-30T04:28:03.349 INFO:tasks.workunit.client.0.smithi123.stdout:6/324: mknod d10/d13/c75 0 2024-03-30T04:28:03.356 INFO:tasks.workunit.client.0.smithi123.stdout:7/145: dwrite d2/dc/f27 [0,4194304] 0 2024-03-30T04:28:03.358 INFO:tasks.workunit.client.0.smithi123.stdout:8/215: symlink da/dc/l34 0 2024-03-30T04:28:03.359 INFO:tasks.workunit.client.0.smithi123.stdout:9/187: write d6/d1f/f2d [794906,344] 0 2024-03-30T04:28:03.359 INFO:tasks.workunit.client.0.smithi123.stdout:6/325: write d10/d13/d1c/f5d [407222,36033] 0 2024-03-30T04:28:03.365 INFO:tasks.workunit.client.0.smithi123.stdout:4/276: write d2/d4/d7/fb [1502332,62371] 0 2024-03-30T04:28:03.365 INFO:tasks.workunit.client.0.smithi123.stdout:8/216: truncate da/f12 4673023 0 2024-03-30T04:28:03.371 INFO:tasks.workunit.client.0.smithi123.stdout:9/188: mkdir d6/d1f/d45 0 2024-03-30T04:28:03.371 INFO:tasks.workunit.client.0.smithi123.stdout:7/146: mknod d2/da/d11/c32 0 2024-03-30T04:28:03.371 INFO:tasks.workunit.client.0.smithi123.stdout:6/326: symlink d10/d13/d45/l76 0 2024-03-30T04:28:03.374 INFO:tasks.workunit.client.0.smithi123.stdout:2/165: dwrite d0/d1f/f17 [0,4194304] 0 2024-03-30T04:28:03.376 INFO:tasks.workunit.client.1.smithi135.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-03-30T04:28:03.384 INFO:tasks.workunit.client.0.smithi123.stdout:1/342: dwrite db/d1b/f37 [0,4194304] 0 2024-03-30T04:28:03.384 INFO:tasks.workunit.client.0.smithi123.stdout:9/189: creat d6/d23/f46 x:0 0 0 2024-03-30T04:28:03.386 INFO:tasks.workunit.client.0.smithi123.stdout:9/190: readlink d6/la 0 2024-03-30T04:28:03.397 INFO:tasks.workunit.client.0.smithi123.stdout:2/166: rename d0/d1f/c12 to d0/d1f/d16/c32 0 2024-03-30T04:28:03.398 INFO:tasks.workunit.client.0.smithi123.stdout:1/343: fdatasync db/dd/f13 0 2024-03-30T04:28:03.400 INFO:tasks.workunit.client.0.smithi123.stdout:7/147: creat d2/da/f33 x:0 0 0 2024-03-30T04:28:03.401 INFO:tasks.workunit.client.0.smithi123.stdout:2/167: stat d0/l15 0 2024-03-30T04:28:03.406 INFO:tasks.workunit.client.0.smithi123.stdout:8/217: dwrite f8 [4194304,4194304] 0 2024-03-30T04:28:03.411 INFO:tasks.workunit.client.0.smithi123.stdout:6/327: dwrite d10/d20/d3f/f65 [0,4194304] 0 2024-03-30T04:28:03.411 INFO:tasks.workunit.client.0.smithi123.stdout:2/168: write d0/f1d [623299,51079] 0 2024-03-30T04:28:03.412 INFO:tasks.workunit.client.1.smithi135.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-03-30T04:28:03.412 INFO:tasks.workunit.client.1.smithi135.stderr:+ make 2024-03-30T04:28:03.415 INFO:tasks.workunit.client.0.smithi123.stdout:4/277: dwrite d2/d4/d2c/d42/f3e [0,4194304] 0 2024-03-30T04:28:03.419 INFO:tasks.workunit.client.0.smithi123.stdout:8/218: rename da/l17 to da/d27/l35 0 2024-03-30T04:28:03.419 INFO:tasks.workunit.client.0.smithi123.stdout:6/328: truncate d10/d20/f40 937851 0 2024-03-30T04:28:03.426 INFO:tasks.workunit.client.0.smithi123.stdout:7/148: dread d2/da/d2d/f2a [4194304,4194304] 0 2024-03-30T04:28:03.431 INFO:tasks.workunit.client.0.smithi123.stdout:2/169: dread d0/d1f/d16/f23 [0,4194304] 0 2024-03-30T04:28:03.432 INFO:tasks.workunit.client.0.smithi123.stdout:8/219: rmdir da/dc/d14/d18/d21 39 2024-03-30T04:28:03.435 INFO:tasks.workunit.client.0.smithi123.stdout:1/344: dwrite db/dd/d42/d46/d56/d5e/f73 [0,4194304] 0 2024-03-30T04:28:03.436 INFO:tasks.workunit.client.0.smithi123.stdout:1/345: read - db/dd/f4c zero size 2024-03-30T04:28:03.438 INFO:tasks.workunit.client.0.smithi123.stdout:6/329: mkdir d10/d13/d45/d77 0 2024-03-30T04:28:03.439 INFO:tasks.workunit.client.0.smithi123.stdout:1/346: stat db/c43 0 2024-03-30T04:28:03.443 INFO:tasks.workunit.client.0.smithi123.stdout:8/220: creat da/dc/d14/f36 x:0 0 0 2024-03-30T04:28:03.447 INFO:tasks.workunit.client.0.smithi123.stdout:1/347: truncate db/dd/d42/d46/d56/d5c/f74 380493 0 2024-03-30T04:28:03.449 INFO:tasks.workunit.client.0.smithi123.stdout:6/330: unlink d10/d13/d45/c55 0 2024-03-30T04:28:03.450 INFO:tasks.workunit.client.0.smithi123.stdout:7/149: dwrite d2/da/fb [0,4194304] 0 2024-03-30T04:28:03.452 INFO:tasks.workunit.client.0.smithi123.stdout:4/278: rename d2/d4/d7/d1a/l55 to d2/d4/d7/d1a/l69 0 2024-03-30T04:28:03.457 INFO:tasks.workunit.client.0.smithi123.stdout:2/170: getdents d0 0 2024-03-30T04:28:03.457 INFO:tasks.workunit.client.0.smithi123.stdout:6/331: dread - d10/d13/d1c/f6c zero size 2024-03-30T04:28:03.462 INFO:tasks.workunit.client.0.smithi123.stdout:7/150: write d2/dc/d1c/f23 [1072956,47666] 0 2024-03-30T04:28:03.463 INFO:tasks.workunit.client.0.smithi123.stdout:1/348: creat db/dd/d42/d46/d56/d5c/f76 x:0 0 0 2024-03-30T04:28:03.468 INFO:tasks.workunit.client.0.smithi123.stdout:7/151: truncate d2/f8 5499489 0 2024-03-30T04:28:03.474 INFO:tasks.workunit.client.0.smithi123.stdout:1/349: truncate db/dd/d42/d46/d56/d5c/f74 1329918 0 2024-03-30T04:28:03.476 INFO:tasks.workunit.client.0.smithi123.stdout:6/332: link d10/d13/d1c/f2a d10/d20/f78 0 2024-03-30T04:28:03.477 INFO:tasks.workunit.client.0.smithi123.stdout:1/350: write db/f45 [542686,61463] 0 2024-03-30T04:28:03.479 INFO:tasks.workunit.client.0.smithi123.stdout:6/333: write d10/f68 [329823,119261] 0 2024-03-30T04:28:03.488 INFO:tasks.workunit.client.0.smithi123.stdout:2/171: dwrite d0/f1d [0,4194304] 0 2024-03-30T04:28:03.489 INFO:tasks.workunit.client.0.smithi123.stdout:6/334: mknod d10/d13/c79 0 2024-03-30T04:28:03.492 INFO:tasks.workunit.client.0.smithi123.stdout:7/152: dread d2/dc/d1c/f23 [0,4194304] 0 2024-03-30T04:28:03.494 INFO:tasks.workunit.client.0.smithi123.stdout:2/172: creat d0/d1f/d16/f33 x:0 0 0 2024-03-30T04:28:03.495 INFO:tasks.workunit.client.0.smithi123.stdout:8/221: dwrite da/dc/d14/d18/f30 [0,4194304] 0 2024-03-30T04:28:03.498 INFO:tasks.workunit.client.0.smithi123.stdout:7/153: mkdir d2/da/d11/d34 0 2024-03-30T04:28:03.502 INFO:tasks.workunit.client.0.smithi123.stdout:2/173: readlink d0/l2e 0 2024-03-30T04:28:03.503 INFO:tasks.workunit.client.0.smithi123.stdout:4/279: dread d2/d4/ff [0,4194304] 0 2024-03-30T04:28:03.506 INFO:tasks.workunit.client.0.smithi123.stdout:8/222: mknod da/dc/d14/c37 0 2024-03-30T04:28:03.506 INFO:tasks.workunit.client.0.smithi123.stdout:1/351: dwrite db/d1b/f3e [0,4194304] 0 2024-03-30T04:28:03.506 INFO:tasks.workunit.client.0.smithi123.stdout:7/154: symlink d2/l35 0 2024-03-30T04:28:03.508 INFO:tasks.workunit.client.0.smithi123.stdout:6/335: write d10/d13/d1c/f6f [329386,71563] 0 2024-03-30T04:28:03.509 INFO:tasks.workunit.client.0.smithi123.stdout:1/352: stat db/dd/l34 0 2024-03-30T04:28:03.509 INFO:tasks.workunit.client.0.smithi123.stdout:4/280: write d2/d34/f4c [457183,16209] 0 2024-03-30T04:28:03.509 INFO:tasks.workunit.client.0.smithi123.stdout:7/155: readlink d2/dc/l25 0 2024-03-30T04:28:03.511 INFO:tasks.workunit.client.1.smithi135.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-03-30T04:28:03.511 INFO:tasks.workunit.client.0.smithi123.stdout:6/336: readlink d10/d13/l4e 0 2024-03-30T04:28:03.513 INFO:tasks.workunit.client.0.smithi123.stdout:4/281: chown d2/d4/d7/l1d 980720 1 2024-03-30T04:28:03.514 INFO:tasks.workunit.client.0.smithi123.stdout:8/223: write da/dc/f1a [363016,85924] 0 2024-03-30T04:28:03.516 INFO:tasks.workunit.client.0.smithi123.stdout:4/282: write d2/d34/f3f [4361045,35379] 0 2024-03-30T04:28:03.519 INFO:tasks.workunit.client.0.smithi123.stdout:1/353: truncate db/dd/d42/f68 477186 0 2024-03-30T04:28:03.522 INFO:tasks.workunit.client.0.smithi123.stdout:2/174: write d0/f10 [4946711,75138] 0 2024-03-30T04:28:03.526 INFO:tasks.workunit.client.0.smithi123.stdout:1/354: read db/dd/d42/f68 [80202,113317] 0 2024-03-30T04:28:03.528 INFO:tasks.workunit.client.0.smithi123.stdout:8/224: symlink da/dc/d14/d18/l38 0 2024-03-30T04:28:03.531 INFO:tasks.workunit.client.0.smithi123.stdout:1/355: mknod db/dd/d42/d46/d56/c77 0 2024-03-30T04:28:03.537 INFO:tasks.workunit.client.0.smithi123.stdout:1/356: write db/dd/f13 [4330903,104360] 0 2024-03-30T04:28:03.537 INFO:tasks.workunit.client.0.smithi123.stdout:4/283: dread d2/d4/fa [0,4194304] 0 2024-03-30T04:28:03.546 INFO:tasks.workunit.client.0.smithi123.stdout:4/284: creat d2/d4/d2c/d42/f6a x:0 0 0 2024-03-30T04:28:03.546 INFO:tasks.workunit.client.0.smithi123.stdout:7/156: dwrite d2/dc/ff [8388608,4194304] 0 2024-03-30T04:28:03.546 INFO:tasks.workunit.client.0.smithi123.stdout:6/337: dwrite d10/f48 [0,4194304] 0 2024-03-30T04:28:03.547 INFO:tasks.workunit.client.0.smithi123.stdout:1/357: mkdir db/dd/d42/d46/d56/d5e/d78 0 2024-03-30T04:28:03.552 INFO:tasks.workunit.client.0.smithi123.stdout:2/175: dwrite d0/d1f/d1c/f22 [0,4194304] 0 2024-03-30T04:28:03.553 INFO:tasks.workunit.client.0.smithi123.stdout:4/285: write d2/d4/f8 [349549,34791] 0 2024-03-30T04:28:03.556 INFO:tasks.workunit.client.0.smithi123.stdout:1/358: fsync db/dd/d42/d46/f53 0 2024-03-30T04:28:03.560 INFO:tasks.workunit.client.0.smithi123.stdout:6/338: dread d10/f26 [0,4194304] 0 2024-03-30T04:28:03.561 INFO:tasks.workunit.client.0.smithi123.stdout:8/225: dwrite da/dc/f20 [0,4194304] 0 2024-03-30T04:28:03.567 INFO:tasks.workunit.client.0.smithi123.stdout:4/286: rename d2/d4/d7/ld to d2/d4/d2c/d42/l6b 0 2024-03-30T04:28:03.567 INFO:tasks.workunit.client.0.smithi123.stdout:8/226: write da/dc/f16 [844246,64506] 0 2024-03-30T04:28:03.568 INFO:tasks.workunit.client.0.smithi123.stdout:6/339: write d10/f11 [3463426,96590] 0 2024-03-30T04:28:03.569 INFO:tasks.workunit.client.0.smithi123.stdout:7/157: getdents d2/da/d2d 0 2024-03-30T04:28:03.579 INFO:tasks.workunit.client.0.smithi123.stdout:2/176: dread d0/d1f/f17 [0,4194304] 0 2024-03-30T04:28:03.579 INFO:tasks.workunit.client.0.smithi123.stdout:7/158: creat d2/da/d2d/f36 x:0 0 0 2024-03-30T04:28:03.580 INFO:tasks.workunit.client.0.smithi123.stdout:8/227: rmdir da/dc/d14/d18/d21 39 2024-03-30T04:28:03.580 INFO:tasks.workunit.client.0.smithi123.stdout:6/340: rename d10/d13/d1c/d74 to d10/d13/d6a/d7a 0 2024-03-30T04:28:03.581 INFO:tasks.workunit.client.0.smithi123.stdout:4/287: unlink d2/d4/d2c/d42/d35/c64 0 2024-03-30T04:28:03.587 INFO:tasks.workunit.client.0.smithi123.stdout:6/341: rmdir d10/d20 39 2024-03-30T04:28:03.589 INFO:tasks.workunit.client.0.smithi123.stdout:7/159: mknod d2/da/d11/d34/c37 0 2024-03-30T04:28:03.591 INFO:tasks.workunit.client.0.smithi123.stdout:8/228: mknod da/dc/d14/d24/c39 0 2024-03-30T04:28:03.592 INFO:tasks.workunit.client.0.smithi123.stdout:1/359: dwrite db/d1b/f2a [0,4194304] 0 2024-03-30T04:28:03.596 INFO:tasks.workunit.client.0.smithi123.stdout:8/229: symlink da/d27/l3a 0 2024-03-30T04:28:03.598 INFO:tasks.workunit.client.0.smithi123.stdout:6/342: write d10/d13/f17 [446469,49536] 0 2024-03-30T04:28:03.599 INFO:tasks.workunit.client.0.smithi123.stdout:7/160: truncate d2/da/d11/f19 527868 0 2024-03-30T04:28:03.602 INFO:tasks.workunit.client.0.smithi123.stdout:8/230: unlink da/dc/d14/f1d 0 2024-03-30T04:28:03.603 INFO:tasks.workunit.client.0.smithi123.stdout:7/161: read d2/dc/f27 [3149960,26868] 0 2024-03-30T04:28:03.607 INFO:tasks.workunit.client.0.smithi123.stdout:1/360: creat db/dd/d42/d46/f79 x:0 0 0 2024-03-30T04:28:03.607 INFO:tasks.workunit.client.0.smithi123.stdout:6/343: getdents d10/d13/d45/d77 0 2024-03-30T04:28:03.608 INFO:tasks.workunit.client.0.smithi123.stdout:2/177: dwrite d0/d1f/d16/f26 [0,4194304] 0 2024-03-30T04:28:03.614 INFO:tasks.workunit.client.0.smithi123.stdout:7/162: rmdir d2/da/d2d 39 2024-03-30T04:28:03.617 INFO:tasks.workunit.client.0.smithi123.stdout:2/178: unlink d0/l2e 0 2024-03-30T04:28:03.619 INFO:tasks.workunit.client.0.smithi123.stdout:7/163: read d2/dc/f17 [1515354,19227] 0 2024-03-30T04:28:03.620 INFO:tasks.workunit.client.0.smithi123.stdout:4/288: dwrite d2/d4/d2c/f67 [0,4194304] 0 2024-03-30T04:28:03.620 INFO:tasks.workunit.client.0.smithi123.stdout:2/179: fsync d0/d1f/f1b 0 2024-03-30T04:28:03.621 INFO:tasks.workunit.client.0.smithi123.stdout:8/231: dread da/dc/f1a [0,4194304] 0 2024-03-30T04:28:03.623 INFO:tasks.workunit.client.0.smithi123.stdout:4/289: fdatasync d2/d26/f4f 0 2024-03-30T04:28:03.625 INFO:tasks.workunit.client.0.smithi123.stdout:6/344: rename d10/d13/d1c/d5b/c6d to d10/c7b 0 2024-03-30T04:28:03.631 INFO:tasks.workunit.client.0.smithi123.stdout:1/361: getdents db/dd/d42/d46/d56 0 2024-03-30T04:28:03.631 INFO:tasks.workunit.client.0.smithi123.stdout:8/232: creat da/d27/f3b x:0 0 0 2024-03-30T04:28:03.633 INFO:tasks.workunit.client.0.smithi123.stdout:1/362: chown db/f41 15 1 2024-03-30T04:28:03.635 INFO:tasks.workunit.client.0.smithi123.stdout:6/345: creat d10/d13/d45/d77/f7c x:0 0 0 2024-03-30T04:28:03.638 INFO:tasks.workunit.client.0.smithi123.stdout:8/233: chown da/dc/d14/d24/d2d/f2e 618 1 2024-03-30T04:28:03.639 INFO:tasks.workunit.client.0.smithi123.stdout:4/290: creat d2/d4/d7/f6c x:0 0 0 2024-03-30T04:28:03.642 INFO:tasks.workunit.client.0.smithi123.stdout:2/180: truncate d0/d1f/f1b 627817 0 2024-03-30T04:28:03.643 INFO:tasks.workunit.client.0.smithi123.stdout:8/234: rmdir da/dc 39 2024-03-30T04:28:03.644 INFO:tasks.workunit.client.0.smithi123.stdout:7/164: dread d2/dc/f27 [0,4194304] 0 2024-03-30T04:28:03.646 INFO:tasks.workunit.client.0.smithi123.stdout:6/346: mknod d10/d13/d1c/d42/c7d 0 2024-03-30T04:28:03.648 INFO:tasks.workunit.client.0.smithi123.stdout:4/291: mknod d2/d26/c6d 0 2024-03-30T04:28:03.650 INFO:tasks.workunit.client.0.smithi123.stdout:2/181: rmdir d0/d1f/d1c/d2a 39 2024-03-30T04:28:03.650 INFO:tasks.workunit.client.0.smithi123.stdout:8/235: dread - da/dc/d14/f36 zero size 2024-03-30T04:28:03.650 INFO:tasks.workunit.client.0.smithi123.stdout:4/292: chown d2/d4/f8 947 1 2024-03-30T04:28:03.651 INFO:tasks.workunit.client.0.smithi123.stdout:8/236: chown da/dc 570 1 2024-03-30T04:28:03.651 INFO:tasks.workunit.client.0.smithi123.stdout:1/363: truncate f8 4206857 0 2024-03-30T04:28:03.652 INFO:tasks.workunit.client.0.smithi123.stdout:4/293: fsync d2/d4/f49 0 2024-03-30T04:28:03.655 INFO:tasks.workunit.client.0.smithi123.stdout:6/347: mknod d10/d13/d6a/d7a/c7e 0 2024-03-30T04:28:03.656 INFO:tasks.workunit.client.0.smithi123.stdout:7/165: write d2/dc/d1c/f23 [1952834,120964] 0 2024-03-30T04:28:03.658 INFO:tasks.workunit.client.0.smithi123.stdout:2/182: fsync d0/d1f/fa 0 2024-03-30T04:28:03.660 INFO:tasks.workunit.client.0.smithi123.stdout:8/237: write da/dc/f16 [1249111,96414] 0 2024-03-30T04:28:03.663 INFO:tasks.workunit.client.0.smithi123.stdout:2/183: read d0/f1d [1342854,82718] 0 2024-03-30T04:28:03.664 INFO:tasks.workunit.client.0.smithi123.stdout:4/294: symlink d2/d4/l6e 0 2024-03-30T04:28:03.665 INFO:tasks.workunit.client.0.smithi123.stdout:2/184: fsync d0/f14 0 2024-03-30T04:28:03.669 INFO:tasks.workunit.client.0.smithi123.stdout:7/166: mknod d2/da/c38 0 2024-03-30T04:28:03.671 INFO:tasks.workunit.client.0.smithi123.stdout:8/238: rename da/dc/d14/d24 to da/d27/d3c 0 2024-03-30T04:28:03.673 INFO:tasks.workunit.client.0.smithi123.stdout:8/239: write f6 [6443589,67341] 0 2024-03-30T04:28:03.675 INFO:tasks.workunit.client.0.smithi123.stdout:4/295: unlink d2/d26/l54 0 2024-03-30T04:28:03.676 INFO:tasks.workunit.client.0.smithi123.stdout:1/364: dwrite db/f41 [0,4194304] 0 2024-03-30T04:28:03.677 INFO:tasks.workunit.client.0.smithi123.stdout:6/348: rename d10/d20/f78 to d10/d20/f7f 0 2024-03-30T04:28:03.681 INFO:tasks.workunit.client.0.smithi123.stdout:2/185: unlink d0/c3 0 2024-03-30T04:28:03.684 INFO:tasks.workunit.client.0.smithi123.stdout:1/365: readlink db/l4e 0 2024-03-30T04:28:03.685 INFO:tasks.workunit.client.0.smithi123.stdout:4/296: mkdir d2/d4/d2c/d42/d35/d5d/d6f 0 2024-03-30T04:28:03.687 INFO:tasks.workunit.client.0.smithi123.stdout:6/349: creat d10/d13/d1c/d42/f80 x:0 0 0 2024-03-30T04:28:03.689 INFO:tasks.workunit.client.0.smithi123.stdout:7/167: creat d2/da/d2d/f39 x:0 0 0 2024-03-30T04:28:03.690 INFO:tasks.workunit.client.0.smithi123.stdout:1/366: mkdir db/dd/d42/d46/d5d/d7a 0 2024-03-30T04:28:03.691 INFO:tasks.workunit.client.0.smithi123.stdout:6/350: readlink d10/l61 0 2024-03-30T04:28:03.691 INFO:tasks.workunit.client.0.smithi123.stdout:4/297: read d2/d4/f8 [3061130,119626] 0 2024-03-30T04:28:03.692 INFO:tasks.workunit.client.0.smithi123.stdout:2/186: mknod d0/d1f/d1c/d2a/c34 0 2024-03-30T04:28:03.694 INFO:tasks.workunit.client.0.smithi123.stdout:8/240: creat da/f3d x:0 0 0 2024-03-30T04:28:03.695 INFO:tasks.workunit.client.0.smithi123.stdout:6/351: mknod d10/d13/d45/c81 0 2024-03-30T04:28:03.696 INFO:tasks.workunit.client.0.smithi123.stdout:2/187: truncate d0/d1f/f24 2135148 0 2024-03-30T04:28:03.697 INFO:tasks.workunit.client.0.smithi123.stdout:6/352: stat d10/d13/d1c/f2c 0 2024-03-30T04:28:03.699 INFO:tasks.workunit.client.0.smithi123.stdout:1/367: symlink db/dd/d42/d46/l7b 0 2024-03-30T04:28:03.699 INFO:tasks.workunit.client.0.smithi123.stdout:6/353: stat fe 0 2024-03-30T04:28:03.699 INFO:tasks.workunit.client.0.smithi123.stdout:2/188: stat d0/d1f/d1c/d2a/c2c 0 2024-03-30T04:28:03.702 INFO:tasks.workunit.client.0.smithi123.stdout:8/241: rename da/dc/l1f to da/d27/d3c/d2d/l3e 0 2024-03-30T04:28:03.704 INFO:tasks.workunit.client.0.smithi123.stdout:2/189: unlink d0/d1f/c13 0 2024-03-30T04:28:03.705 INFO:tasks.workunit.client.0.smithi123.stdout:1/368: write db/dd/d42/d46/d56/d5c/f76 [680984,104870] 0 2024-03-30T04:28:03.712 INFO:tasks.workunit.client.0.smithi123.stdout:6/354: rename d10/d13/d1c/d4c to d10/d82 0 2024-03-30T04:28:03.717 INFO:tasks.workunit.client.0.smithi123.stdout:8/242: rename da/d27/l3a to da/d27/l3f 0 2024-03-30T04:28:03.723 INFO:tasks.workunit.client.0.smithi123.stdout:8/243: fdatasync da/f3d 0 2024-03-30T04:28:03.724 INFO:tasks.workunit.client.0.smithi123.stdout:4/298: dwrite d2/d4/d7/fb [0,4194304] 0 2024-03-30T04:28:03.731 INFO:tasks.workunit.client.0.smithi123.stdout:7/168: dwrite d2/da/d11/f1e [0,4194304] 0 2024-03-30T04:28:03.735 INFO:tasks.workunit.client.0.smithi123.stdout:6/355: read d10/d13/d1c/f5d [396036,100432] 0 2024-03-30T04:28:03.736 INFO:tasks.workunit.client.0.smithi123.stdout:4/299: creat d2/d26/f70 x:0 0 0 2024-03-30T04:28:03.739 INFO:tasks.workunit.client.0.smithi123.stdout:7/169: mkdir d2/dc/d3a 0 2024-03-30T04:28:03.741 INFO:tasks.workunit.client.0.smithi123.stdout:1/369: dread db/d1b/f28 [0,4194304] 0 2024-03-30T04:28:03.743 INFO:tasks.workunit.client.0.smithi123.stdout:2/190: dread d0/f1d [0,4194304] 0 2024-03-30T04:28:03.743 INFO:tasks.workunit.client.0.smithi123.stdout:1/370: chown db/f60 8 1 2024-03-30T04:28:03.744 INFO:tasks.workunit.client.0.smithi123.stdout:7/170: mkdir d2/da/d22/d3b 0 2024-03-30T04:28:03.749 INFO:tasks.workunit.client.0.smithi123.stdout:6/356: getdents d10/d13/d45 0 2024-03-30T04:28:03.749 INFO:tasks.workunit.client.0.smithi123.stdout:2/191: mknod d0/d1f/c35 0 2024-03-30T04:28:03.751 INFO:tasks.workunit.client.0.smithi123.stdout:7/171: creat d2/da/d11/f3c x:0 0 0 2024-03-30T04:28:03.752 INFO:tasks.workunit.client.0.smithi123.stdout:8/244: dread da/dc/d14/d18/d21/f28 [0,4194304] 0 2024-03-30T04:28:03.758 INFO:tasks.workunit.client.0.smithi123.stdout:1/371: unlink db/dd/l49 0 2024-03-30T04:28:03.758 INFO:tasks.workunit.client.0.smithi123.stdout:4/300: dread d2/d34/f3f [0,4194304] 0 2024-03-30T04:28:03.758 INFO:tasks.workunit.client.0.smithi123.stdout:6/357: mknod d10/d13/d1c/d5b/c83 0 2024-03-30T04:28:03.759 INFO:tasks.workunit.client.0.smithi123.stdout:7/172: creat d2/da/d2d/f3d x:0 0 0 2024-03-30T04:28:03.760 INFO:tasks.workunit.client.0.smithi123.stdout:8/245: read da/dc/d14/d18/d21/f28 [222981,121785] 0 2024-03-30T04:28:03.763 INFO:tasks.workunit.client.0.smithi123.stdout:1/372: mknod db/dd/d42/c7c 0 2024-03-30T04:28:03.763 INFO:tasks.workunit.client.0.smithi123.stdout:6/358: creat d10/d82/f84 x:0 0 0 2024-03-30T04:28:03.764 INFO:tasks.workunit.client.0.smithi123.stdout:4/301: read d2/d4/d7/d1a/f21 [2099208,1987] 0 2024-03-30T04:28:03.768 INFO:tasks.workunit.client.0.smithi123.stdout:7/173: truncate d2/da/d11/f1e 4718763 0 2024-03-30T04:28:03.768 INFO:tasks.workunit.client.0.smithi123.stdout:0/163: sync 2024-03-30T04:28:03.768 INFO:tasks.workunit.client.0.smithi123.stdout:6/359: write d10/d82/f66 [208226,18196] 0 2024-03-30T04:28:03.770 INFO:tasks.workunit.client.0.smithi123.stdout:2/192: dread d0/d1f/f24 [0,4194304] 0 2024-03-30T04:28:03.771 INFO:tasks.workunit.client.0.smithi123.stdout:0/164: read - d6/dd/d16/f1e zero size 2024-03-30T04:28:03.776 INFO:tasks.workunit.client.0.smithi123.stdout:4/302: getdents d2/d4 0 2024-03-30T04:28:03.776 INFO:tasks.workunit.client.0.smithi123.stdout:7/174: mknod d2/da/d22/d3b/c3e 0 2024-03-30T04:28:03.776 INFO:tasks.workunit.client.0.smithi123.stdout:2/193: mkdir d0/d1f/d1c/d36 0 2024-03-30T04:28:03.781 INFO:tasks.workunit.client.0.smithi123.stdout:1/373: truncate db/dd/d42/d46/d39/f52 3355529 0 2024-03-30T04:28:03.782 INFO:tasks.workunit.client.0.smithi123.stdout:8/246: dwrite da/f12 [4194304,4194304] 0 2024-03-30T04:28:03.782 INFO:tasks.workunit.client.0.smithi123.stdout:2/194: write d0/d1f/d16/f23 [726237,78746] 0 2024-03-30T04:28:03.787 INFO:tasks.workunit.client.0.smithi123.stdout:2/195: mkdir d0/d37 0 2024-03-30T04:28:03.795 INFO:tasks.workunit.client.0.smithi123.stdout:8/247: link da/f3d da/dc/d14/d18/d21/f40 0 2024-03-30T04:28:03.797 INFO:tasks.workunit.client.0.smithi123.stdout:1/374: dread db/dd/d42/d46/f47 [0,4194304] 0 2024-03-30T04:28:03.798 INFO:tasks.workunit.client.0.smithi123.stdout:7/175: dwrite d2/da/d11/f1e [0,4194304] 0 2024-03-30T04:28:03.798 INFO:tasks.workunit.client.0.smithi123.stdout:0/165: dread d6/f12 [0,4194304] 0 2024-03-30T04:28:03.799 INFO:tasks.workunit.client.0.smithi123.stdout:2/196: write d0/f10 [2481195,57876] 0 2024-03-30T04:28:03.799 INFO:tasks.workunit.client.0.smithi123.stdout:1/375: stat db/dd 0 2024-03-30T04:28:03.803 INFO:tasks.workunit.client.0.smithi123.stdout:2/197: write d0/d1f/d16/f1e [2849468,63512] 0 2024-03-30T04:28:03.805 INFO:tasks.workunit.client.0.smithi123.stdout:1/376: unlink db/d1b/f4b 0 2024-03-30T04:28:03.807 INFO:tasks.workunit.client.0.smithi123.stdout:7/176: rename d2/dc/d1c/l29 to d2/da/d11/d34/l3f 0 2024-03-30T04:28:03.810 INFO:tasks.workunit.client.0.smithi123.stdout:7/177: dread - d2/da/d2d/f36 zero size 2024-03-30T04:28:03.814 INFO:tasks.workunit.client.0.smithi123.stdout:1/377: mknod db/dd/d5f/d6d/c7d 0 2024-03-30T04:28:03.814 INFO:tasks.workunit.client.0.smithi123.stdout:2/198: rename d0/d1f/c21 to d0/d1f/d30/c38 0 2024-03-30T04:28:03.816 INFO:tasks.workunit.client.0.smithi123.stdout:1/378: chown db/l40 211456 1 2024-03-30T04:28:03.817 INFO:tasks.workunit.client.0.smithi123.stdout:2/199: chown d0/d1f/d1c/d2a/c2c 3602 1 2024-03-30T04:28:03.818 INFO:tasks.workunit.client.0.smithi123.stdout:8/248: dwrite da/dc/d14/f36 [0,4194304] 0 2024-03-30T04:28:03.820 INFO:tasks.workunit.client.0.smithi123.stdout:2/200: write d0/d1f/d16/f26 [4776198,71676] 0 2024-03-30T04:28:03.821 INFO:tasks.workunit.client.0.smithi123.stdout:8/249: write da/fe [2854573,7075] 0 2024-03-30T04:28:03.822 INFO:tasks.workunit.client.0.smithi123.stdout:1/379: mkdir db/dd/d5f/d6d/d7e 0 2024-03-30T04:28:03.824 INFO:tasks.workunit.client.0.smithi123.stdout:7/178: rmdir d2/dc 39 2024-03-30T04:28:03.825 INFO:tasks.workunit.client.0.smithi123.stdout:1/380: dread - db/dd/d42/d46/d5d/d62/f6b zero size 2024-03-30T04:28:03.827 INFO:tasks.workunit.client.0.smithi123.stdout:1/381: readlink db/l4e 0 2024-03-30T04:28:03.829 INFO:tasks.workunit.client.0.smithi123.stdout:7/179: creat d2/da/d2d/f40 x:0 0 0 2024-03-30T04:28:03.829 INFO:tasks.workunit.client.0.smithi123.stdout:7/180: chown d2/da 5203590 1 2024-03-30T04:28:03.830 INFO:tasks.workunit.client.0.smithi123.stdout:2/201: mkdir d0/d1f/d1c/d39 0 2024-03-30T04:28:03.834 INFO:tasks.workunit.client.0.smithi123.stdout:1/382: unlink db/f41 0 2024-03-30T04:28:03.835 INFO:tasks.workunit.client.0.smithi123.stdout:2/202: readlink d0/le 0 2024-03-30T04:28:03.838 INFO:tasks.workunit.client.0.smithi123.stdout:1/383: creat db/dd/d42/d46/d56/f7f x:0 0 0 2024-03-30T04:28:03.840 INFO:tasks.workunit.client.0.smithi123.stdout:2/203: symlink d0/d1f/d1c/d20/l3a 0 2024-03-30T04:28:03.842 INFO:tasks.workunit.client.0.smithi123.stdout:1/384: truncate db/dd/d42/f68 1032235 0 2024-03-30T04:28:03.845 INFO:tasks.workunit.client.0.smithi123.stdout:1/385: mkdir db/dd/d42/d46/d51/d72/d80 0 2024-03-30T04:28:03.850 INFO:tasks.workunit.client.0.smithi123.stdout:1/386: rmdir db/dd/d42/d46/d51/d72/d80 0 2024-03-30T04:28:03.852 INFO:tasks.workunit.client.0.smithi123.stdout:7/181: dwrite f0 [0,4194304] 0 2024-03-30T04:28:03.853 INFO:tasks.workunit.client.0.smithi123.stdout:2/204: dread d0/d1f/f17 [0,4194304] 0 2024-03-30T04:28:03.859 INFO:tasks.workunit.client.0.smithi123.stdout:2/205: mknod d0/d1f/d1c/c3b 0 2024-03-30T04:28:03.860 INFO:tasks.workunit.client.0.smithi123.stdout:7/182: creat d2/f41 x:0 0 0 2024-03-30T04:28:03.862 INFO:tasks.workunit.client.0.smithi123.stdout:2/206: creat d0/d1f/d16/f3c x:0 0 0 2024-03-30T04:28:03.862 INFO:tasks.workunit.client.0.smithi123.stdout:7/183: write d2/da/fb [1881784,29213] 0 2024-03-30T04:28:03.863 INFO:tasks.workunit.client.0.smithi123.stdout:2/207: fsync d0/d1f/f2d 0 2024-03-30T04:28:03.866 INFO:tasks.workunit.client.0.smithi123.stdout:7/184: write d2/dc/d1c/f23 [806947,34565] 0 2024-03-30T04:28:03.868 INFO:tasks.workunit.client.0.smithi123.stdout:7/185: chown d2/f13 19 1 2024-03-30T04:28:03.869 INFO:tasks.workunit.client.0.smithi123.stdout:2/208: link d0/f10 d0/d1f/d16/f3d 0 2024-03-30T04:28:03.870 INFO:tasks.workunit.client.0.smithi123.stdout:7/186: write d2/da/f33 [641655,127517] 0 2024-03-30T04:28:03.871 INFO:tasks.workunit.client.0.smithi123.stdout:2/209: rename d0/d1f to d0/d1f/d1c/d39/d3e 22 2024-03-30T04:28:03.871 INFO:tasks.workunit.client.0.smithi123.stdout:2/210: fdatasync d0/f14 0 2024-03-30T04:28:03.873 INFO:tasks.workunit.client.0.smithi123.stdout:7/187: rmdir d2/da 39 2024-03-30T04:28:03.876 INFO:tasks.workunit.client.0.smithi123.stdout:2/211: creat d0/d1f/d1c/f3f x:0 0 0 2024-03-30T04:28:03.877 INFO:tasks.workunit.client.0.smithi123.stdout:2/212: readlink d0/le 0 2024-03-30T04:28:03.879 INFO:tasks.workunit.client.0.smithi123.stdout:7/188: rename d2/da/d11/f1f to d2/da/d2d/f42 0 2024-03-30T04:28:03.880 INFO:tasks.workunit.client.0.smithi123.stdout:1/387: dread db/dd/f13 [0,4194304] 0 2024-03-30T04:28:03.882 INFO:tasks.workunit.client.0.smithi123.stdout:2/213: mkdir d0/d1f/d1c/d40 0 2024-03-30T04:28:03.884 INFO:tasks.workunit.client.0.smithi123.stdout:7/189: rename d2/da/d2d/f36 to d2/da/d2d/f43 0 2024-03-30T04:28:03.887 INFO:tasks.workunit.client.0.smithi123.stdout:2/214: write d0/d1f/f24 [2004214,57693] 0 2024-03-30T04:28:03.887 INFO:tasks.workunit.client.0.smithi123.stdout:1/388: dread db/d1b/f1d [4194304,4194304] 0 2024-03-30T04:28:03.887 INFO:tasks.workunit.client.0.smithi123.stdout:7/190: symlink d2/da/l44 0 2024-03-30T04:28:03.888 INFO:tasks.workunit.client.0.smithi123.stdout:2/215: stat d0/d1f/f17 0 2024-03-30T04:28:03.890 INFO:tasks.workunit.client.0.smithi123.stdout:1/389: creat db/dd/d5f/d6d/f81 x:0 0 0 2024-03-30T04:28:03.891 INFO:tasks.workunit.client.0.smithi123.stdout:2/216: rmdir d0/d1f/d1c/d20 39 2024-03-30T04:28:03.893 INFO:tasks.workunit.client.0.smithi123.stdout:1/390: creat db/dd/f82 x:0 0 0 2024-03-30T04:28:03.895 INFO:tasks.workunit.client.0.smithi123.stdout:2/217: creat d0/d1f/d1c/f41 x:0 0 0 2024-03-30T04:28:03.895 INFO:tasks.workunit.client.0.smithi123.stdout:1/391: chown db/dd/d42/d46/d56/f70 5 1 2024-03-30T04:28:03.900 INFO:tasks.workunit.client.0.smithi123.stdout:2/218: rename d0/f14 to d0/d1f/d1c/d39/f42 0 2024-03-30T04:28:03.904 INFO:tasks.workunit.client.0.smithi123.stdout:2/219: creat d0/d1f/d1c/d40/f43 x:0 0 0 2024-03-30T04:28:03.905 INFO:tasks.workunit.client.0.smithi123.stdout:7/191: dwrite d2/da/d2d/f2a [0,4194304] 0 2024-03-30T04:28:03.906 INFO:tasks.workunit.client.0.smithi123.stdout:1/392: write f8 [2091918,93555] 0 2024-03-30T04:28:03.908 INFO:tasks.workunit.client.0.smithi123.stdout:2/220: mkdir d0/d1f/d1c/d39/d44 0 2024-03-30T04:28:03.910 INFO:tasks.workunit.client.0.smithi123.stdout:7/192: rename d2/dc/ff to d2/da/d11/f45 0 2024-03-30T04:28:03.917 INFO:tasks.workunit.client.0.smithi123.stdout:2/221: rmdir d0/d1f/d1c/d40 39 2024-03-30T04:28:03.919 INFO:tasks.workunit.client.0.smithi123.stdout:2/222: chown d0/d1f/d1c/d2a/l2f 900413116 1 2024-03-30T04:28:03.921 INFO:tasks.workunit.client.0.smithi123.stdout:2/223: write d0/d1f/d1c/f41 [675903,32216] 0 2024-03-30T04:28:03.928 INFO:tasks.workunit.client.0.smithi123.stdout:2/224: link d0/lb d0/d1f/d1c/l45 0 2024-03-30T04:28:03.929 INFO:tasks.workunit.client.0.smithi123.stdout:7/193: dread d2/dc/f17 [0,4194304] 0 2024-03-30T04:28:03.929 INFO:tasks.workunit.client.0.smithi123.stdout:2/225: chown d0/d1f/d1c/f3f 158251629 1 2024-03-30T04:28:03.931 INFO:tasks.workunit.client.0.smithi123.stdout:7/194: write d2/da/d11/f20 [762084,21994] 0 2024-03-30T04:28:03.933 INFO:tasks.workunit.client.0.smithi123.stdout:7/195: write d2/da/d22/f2f [176868,89323] 0 2024-03-30T04:28:03.935 INFO:tasks.workunit.client.0.smithi123.stdout:7/196: stat f0 0 2024-03-30T04:28:03.936 INFO:tasks.workunit.client.0.smithi123.stdout:7/197: write f0 [4348831,100377] 0 2024-03-30T04:28:03.938 INFO:tasks.workunit.client.0.smithi123.stdout:7/198: dread - d2/da/d2d/f43 zero size 2024-03-30T04:28:03.948 INFO:tasks.workunit.client.0.smithi123.stdout:2/226: dwrite d0/d1f/d1c/d40/f43 [0,4194304] 0 2024-03-30T04:28:03.954 INFO:tasks.workunit.client.0.smithi123.stdout:7/199: dwrite d2/dc/d1c/f1d [0,4194304] 0 2024-03-30T04:28:03.958 INFO:tasks.workunit.client.0.smithi123.stdout:2/227: creat d0/d1f/d1c/d20/f46 x:0 0 0 2024-03-30T04:28:03.959 INFO:tasks.workunit.client.0.smithi123.stdout:2/228: read d0/f10 [2764849,124636] 0 2024-03-30T04:28:03.971 INFO:tasks.workunit.client.0.smithi123.stdout:7/200: link d2/l7 d2/da/d11/d34/l46 0 2024-03-30T04:28:03.978 INFO:tasks.workunit.client.0.smithi123.stdout:2/229: dwrite d0/d1f/d16/f33 [0,4194304] 0 2024-03-30T04:28:03.980 INFO:tasks.workunit.client.0.smithi123.stdout:7/201: creat d2/dc/d1c/f47 x:0 0 0 2024-03-30T04:28:03.984 INFO:tasks.workunit.client.0.smithi123.stdout:7/202: write d2/f8 [359955,88191] 0 2024-03-30T04:28:03.985 INFO:tasks.workunit.client.0.smithi123.stdout:7/203: stat d2/dc/d1c/l31 0 2024-03-30T04:28:03.991 INFO:tasks.workunit.client.0.smithi123.stdout:7/204: creat d2/da/d11/f48 x:0 0 0 2024-03-30T04:28:03.993 INFO:tasks.workunit.client.0.smithi123.stdout:2/230: getdents d0/d1f/d1c/d2a 0 2024-03-30T04:28:03.994 INFO:tasks.workunit.client.0.smithi123.stdout:7/205: mkdir d2/dc/d49 0 2024-03-30T04:28:03.996 INFO:tasks.workunit.client.0.smithi123.stdout:2/231: truncate d0/d1f/d16/f23 1172458 0 2024-03-30T04:28:03.997 INFO:tasks.workunit.client.0.smithi123.stdout:5/231: sync 2024-03-30T04:28:03.997 INFO:tasks.workunit.client.0.smithi123.stdout:9/191: sync 2024-03-30T04:28:03.997 INFO:tasks.workunit.client.0.smithi123.stdout:3/295: sync 2024-03-30T04:28:03.997 INFO:tasks.workunit.client.0.smithi123.stdout:7/206: readlink d2/dc/d1c/l24 0 2024-03-30T04:28:04.000 INFO:tasks.workunit.client.0.smithi123.stdout:3/296: write d13/d24/d31/d23/d3a/f40 [162550,3012] 0 2024-03-30T04:28:04.006 INFO:tasks.workunit.client.0.smithi123.stdout:7/207: rename d2/dc/l25 to d2/da/d22/l4a 0 2024-03-30T04:28:04.012 INFO:tasks.workunit.client.0.smithi123.stdout:7/208: chown d2/da/d11/f45 9053 1 2024-03-30T04:28:04.014 INFO:tasks.workunit.client.0.smithi123.stdout:7/209: dread - d2/dc/d1c/f47 zero size 2024-03-30T04:28:04.020 INFO:tasks.workunit.client.0.smithi123.stdout:7/210: fdatasync d2/da/d11/f20 0 2024-03-30T04:28:04.021 INFO:tasks.workunit.client.0.smithi123.stdout:9/192: dwrite d6/d1f/f34 [0,4194304] 0 2024-03-30T04:28:04.023 INFO:tasks.workunit.client.0.smithi123.stdout:5/232: dread f7 [0,4194304] 0 2024-03-30T04:28:04.024 INFO:tasks.workunit.client.0.smithi123.stdout:5/233: readlink l4 0 2024-03-30T04:28:04.025 INFO:tasks.workunit.client.0.smithi123.stdout:5/234: dread - dd/d41/f43 zero size 2024-03-30T04:28:04.027 INFO:tasks.workunit.client.0.smithi123.stdout:9/193: rmdir d6/d1f 39 2024-03-30T04:28:04.027 INFO:tasks.workunit.client.0.smithi123.stdout:9/194: stat l5 0 2024-03-30T04:28:04.028 INFO:tasks.workunit.client.0.smithi123.stdout:3/297: dread d13/d24/d31/d23/f2b [0,4194304] 0 2024-03-30T04:28:04.029 INFO:tasks.workunit.client.0.smithi123.stdout:3/298: chown fd 3 1 2024-03-30T04:28:04.030 INFO:tasks.workunit.client.0.smithi123.stdout:5/235: symlink dd/d19/l49 0 2024-03-30T04:28:04.032 INFO:tasks.workunit.client.0.smithi123.stdout:9/195: rename d6/c30 to d6/d23/c47 0 2024-03-30T04:28:04.039 INFO:tasks.workunit.client.0.smithi123.stdout:2/232: truncate d0/d1f/d16/f18 2233820 0 2024-03-30T04:28:04.040 INFO:tasks.workunit.client.0.smithi123.stdout:7/211: dwrite d2/da/d11/f3c [0,4194304] 0 2024-03-30T04:28:04.044 INFO:tasks.workunit.client.0.smithi123.stdout:7/212: fdatasync d2/dc/d1c/f1d 0 2024-03-30T04:28:04.048 INFO:tasks.workunit.client.0.smithi123.stdout:7/213: read d2/da/fb [440654,62827] 0 2024-03-30T04:28:04.054 INFO:tasks.workunit.client.0.smithi123.stdout:9/196: dwrite d6/d23/f2b [4194304,4194304] 0 2024-03-30T04:28:04.057 INFO:tasks.workunit.client.0.smithi123.stdout:3/299: dwrite f11 [0,4194304] 0 2024-03-30T04:28:04.061 INFO:tasks.workunit.client.0.smithi123.stdout:3/300: chown d13/c48 47095 1 2024-03-30T04:28:04.062 INFO:tasks.workunit.client.0.smithi123.stdout:7/214: link d2/da/d11/f1e d2/da/d22/d3b/f4b 0 2024-03-30T04:28:04.063 INFO:tasks.workunit.client.0.smithi123.stdout:9/197: symlink d6/l48 0 2024-03-30T04:28:04.065 INFO:tasks.workunit.client.0.smithi123.stdout:3/301: chown d13/d24/d31/d23/d3a/d3c/d4e/c55 54 1 2024-03-30T04:28:04.069 INFO:tasks.workunit.client.0.smithi123.stdout:2/233: rmdir d0/d1f/d16 39 2024-03-30T04:28:04.073 INFO:tasks.workunit.client.0.smithi123.stdout:2/234: readlink d0/d1f/d1c/d20/l2b 0 2024-03-30T04:28:04.075 INFO:tasks.workunit.client.0.smithi123.stdout:5/236: write dd/d19/f28 [1484336,46540] 0 2024-03-30T04:28:04.078 INFO:tasks.workunit.client.0.smithi123.stdout:9/198: dread d6/d1f/f2e [0,4194304] 0 2024-03-30T04:28:04.078 INFO:tasks.workunit.client.0.smithi123.stdout:2/235: stat d0/d1f/d1c/d2a/c2c 0 2024-03-30T04:28:04.084 INFO:tasks.workunit.client.0.smithi123.stdout:5/237: rename dd/c1a to dd/d41/c4a 0 2024-03-30T04:28:04.085 INFO:tasks.workunit.client.0.smithi123.stdout:9/199: mkdir d6/d49 0 2024-03-30T04:28:04.087 INFO:tasks.workunit.client.0.smithi123.stdout:5/238: creat dd/d19/d1f/d35/f4b x:0 0 0 2024-03-30T04:28:04.088 INFO:tasks.workunit.client.0.smithi123.stdout:2/236: mknod d0/d1f/d1c/d36/c47 0 2024-03-30T04:28:04.089 INFO:tasks.workunit.client.0.smithi123.stdout:5/239: dread - dd/d41/f47 zero size 2024-03-30T04:28:04.090 INFO:tasks.workunit.client.0.smithi123.stdout:2/237: dread - d0/d1f/d1c/d20/f46 zero size 2024-03-30T04:28:04.092 INFO:tasks.workunit.client.0.smithi123.stdout:2/238: stat d0/d1f/d27/f29 0 2024-03-30T04:28:04.097 INFO:tasks.workunit.client.0.smithi123.stdout:5/240: getdents dd/d19/d3e 0 2024-03-30T04:28:04.098 INFO:tasks.workunit.client.0.smithi123.stdout:2/239: mknod d0/d1f/d16/c48 0 2024-03-30T04:28:04.098 INFO:tasks.workunit.client.0.smithi123.stdout:5/241: fdatasync fb 0 2024-03-30T04:28:04.102 INFO:tasks.workunit.client.0.smithi123.stdout:5/242: write f2 [1985659,34026] 0 2024-03-30T04:28:04.102 INFO:tasks.workunit.client.0.smithi123.stdout:2/240: rmdir d0/d1f/d1c/d36 39 2024-03-30T04:28:04.104 INFO:tasks.workunit.client.0.smithi123.stdout:9/200: dwrite d6/d23/f2a [0,4194304] 0 2024-03-30T04:28:04.106 INFO:tasks.workunit.client.0.smithi123.stdout:3/302: dwrite d13/d24/f57 [0,4194304] 0 2024-03-30T04:28:04.113 INFO:tasks.workunit.client.0.smithi123.stdout:2/241: creat d0/d1f/d1c/d39/d44/f49 x:0 0 0 2024-03-30T04:28:04.113 INFO:tasks.workunit.client.0.smithi123.stdout:3/303: creat d13/d24/d31/d30/f5a x:0 0 0 2024-03-30T04:28:04.119 INFO:tasks.workunit.client.0.smithi123.stdout:2/242: mkdir d0/d1f/d1c/d2a/d4a 0 2024-03-30T04:28:04.122 INFO:tasks.workunit.client.0.smithi123.stdout:9/201: dread d6/d1f/f34 [0,4194304] 0 2024-03-30T04:28:04.133 INFO:tasks.workunit.client.0.smithi123.stdout:3/304: dwrite d13/d24/d31/d30/f45 [0,4194304] 0 2024-03-30T04:28:04.137 INFO:tasks.workunit.client.0.smithi123.stdout:3/305: write d13/f14 [500532,67271] 0 2024-03-30T04:28:04.139 INFO:tasks.workunit.client.0.smithi123.stdout:9/202: dwrite f0 [0,4194304] 0 2024-03-30T04:28:04.149 INFO:tasks.workunit.client.0.smithi123.stdout:9/203: rename d6/l48 to d6/d1f/l4a 0 2024-03-30T04:28:04.152 INFO:tasks.workunit.client.0.smithi123.stdout:9/204: symlink d6/l4b 0 2024-03-30T04:28:04.152 INFO:tasks.workunit.client.0.smithi123.stdout:2/243: dread d0/d1f/fa [0,4194304] 0 2024-03-30T04:28:04.158 INFO:tasks.workunit.client.0.smithi123.stdout:2/244: getdents d0/d1f/d1c/d20 0 2024-03-30T04:28:04.160 INFO:tasks.workunit.client.0.smithi123.stdout:2/245: readlink d0/lb 0 2024-03-30T04:28:04.163 INFO:tasks.workunit.client.0.smithi123.stdout:2/246: chown d0/d1f/d1c/d39/d44/f49 3758 1 2024-03-30T04:28:04.165 INFO:tasks.workunit.client.0.smithi123.stdout:2/247: mkdir d0/d1f/d1c/d39/d4b 0 2024-03-30T04:28:04.166 INFO:tasks.workunit.client.0.smithi123.stdout:3/306: dread d13/f14 [4194304,4194304] 0 2024-03-30T04:28:04.168 INFO:tasks.workunit.client.0.smithi123.stdout:2/248: truncate d0/f10 4506994 0 2024-03-30T04:28:04.169 INFO:tasks.workunit.client.0.smithi123.stdout:3/307: symlink d13/d24/d39/l5b 0 2024-03-30T04:28:04.170 INFO:tasks.workunit.client.0.smithi123.stdout:3/308: fdatasync d13/d24/d31/f4b 0 2024-03-30T04:28:04.173 INFO:tasks.workunit.client.0.smithi123.stdout:2/249: mkdir d0/d1f/d1c/d39/d4b/d4c 0 2024-03-30T04:28:04.174 INFO:tasks.workunit.client.0.smithi123.stdout:3/309: mknod d13/d24/d31/d23/d3a/d3c/d4e/c5c 0 2024-03-30T04:28:04.179 INFO:tasks.workunit.client.0.smithi123.stdout:3/310: write d13/d24/d31/d23/f2b [4699552,32929] 0 2024-03-30T04:28:04.183 INFO:tasks.workunit.client.0.smithi123.stdout:2/250: link d0/d1f/d16/f26 d0/d1f/d1c/f4d 0 2024-03-30T04:28:04.184 INFO:tasks.workunit.client.0.smithi123.stdout:3/311: mkdir d13/d24/d31/d23/d3a/d3c/d4e/d5d 0 2024-03-30T04:28:04.185 INFO:tasks.workunit.client.0.smithi123.stdout:2/251: dread - d0/d1f/d1c/d20/f46 zero size 2024-03-30T04:28:04.185 INFO:tasks.workunit.client.0.smithi123.stdout:9/205: dread d6/d23/f33 [0,4194304] 0 2024-03-30T04:28:04.186 INFO:tasks.workunit.client.0.smithi123.stdout:3/312: write d13/d24/d31/d23/f2b [3151834,82364] 0 2024-03-30T04:28:04.189 INFO:tasks.workunit.client.0.smithi123.stdout:3/313: creat d13/f5e x:0 0 0 2024-03-30T04:28:04.201 INFO:tasks.workunit.client.0.smithi123.stdout:2/252: dwrite d0/d1f/f17 [0,4194304] 0 2024-03-30T04:28:04.202 INFO:tasks.workunit.client.0.smithi123.stdout:9/206: dwrite d6/d23/f3b [0,4194304] 0 2024-03-30T04:28:04.215 INFO:tasks.workunit.client.0.smithi123.stdout:9/207: mkdir d6/d1f/d45/d4c 0 2024-03-30T04:28:04.215 INFO:tasks.workunit.client.0.smithi123.stdout:2/253: write d0/d1f/d16/f26 [388696,55647] 0 2024-03-30T04:28:04.219 INFO:tasks.workunit.client.0.smithi123.stdout:3/314: dread d13/f1b [0,4194304] 0 2024-03-30T04:28:04.221 INFO:tasks.workunit.client.0.smithi123.stdout:2/254: read d0/d1f/d27/f29 [2452906,102855] 0 2024-03-30T04:28:04.223 INFO:tasks.workunit.client.0.smithi123.stdout:2/255: mknod d0/d37/c4e 0 2024-03-30T04:28:04.227 INFO:tasks.workunit.client.0.smithi123.stdout:9/208: dread f0 [0,4194304] 0 2024-03-30T04:28:04.229 INFO:tasks.workunit.client.0.smithi123.stdout:2/256: getdents d0/d1f/d1c/d39/d44 0 2024-03-30T04:28:04.234 INFO:tasks.workunit.client.0.smithi123.stdout:9/209: getdents d6 0 2024-03-30T04:28:04.234 INFO:tasks.workunit.client.0.smithi123.stdout:2/257: creat d0/d1f/d1c/d2a/d4a/f4f x:0 0 0 2024-03-30T04:28:04.236 INFO:tasks.workunit.client.0.smithi123.stdout:9/210: mkdir d6/d1f/d4d 0 2024-03-30T04:28:04.239 INFO:tasks.workunit.client.0.smithi123.stdout:9/211: creat d6/d1f/d45/d4c/f4e x:0 0 0 2024-03-30T04:28:04.239 INFO:tasks.workunit.client.0.smithi123.stdout:2/258: fsync d0/d1f/d1c/d2a/d4a/f4f 0 2024-03-30T04:28:04.242 INFO:tasks.workunit.client.0.smithi123.stdout:9/212: fsync d6/f10 0 2024-03-30T04:28:04.242 INFO:tasks.workunit.client.0.smithi123.stdout:9/213: write d6/f26 [3157564,4822] 0 2024-03-30T04:28:04.242 INFO:tasks.workunit.client.0.smithi123.stdout:2/259: symlink d0/d1f/d16/l50 0 2024-03-30T04:28:04.243 INFO:tasks.workunit.client.0.smithi123.stdout:2/260: dread - d0/d1f/f2d zero size 2024-03-30T04:28:04.244 INFO:tasks.workunit.client.0.smithi123.stdout:9/214: write d6/d23/f40 [1426332,10312] 0 2024-03-30T04:28:04.246 INFO:tasks.workunit.client.0.smithi123.stdout:2/261: read d0/d1f/d16/f23 [967479,113164] 0 2024-03-30T04:28:04.249 INFO:tasks.workunit.client.0.smithi123.stdout:2/262: dread d0/d1f/d16/f23 [0,4194304] 0 2024-03-30T04:28:04.251 INFO:tasks.workunit.client.0.smithi123.stdout:2/263: write d0/d1f/f2d [29202,23210] 0 2024-03-30T04:28:04.253 INFO:tasks.workunit.client.0.smithi123.stdout:2/264: chown d0/d1f/d16/f3d 4451439 1 2024-03-30T04:28:04.255 INFO:tasks.workunit.client.0.smithi123.stdout:2/265: rmdir d0/d1f/d1c 39 2024-03-30T04:28:04.312 INFO:tasks.workunit.client.0.smithi123.stdout:2/266: creat d0/d1f/d1c/f51 x:0 0 0 2024-03-30T04:28:04.314 INFO:tasks.workunit.client.0.smithi123.stdout:2/267: mknod d0/d1f/d27/c52 0 2024-03-30T04:28:04.317 INFO:tasks.workunit.client.0.smithi123.stdout:2/268: getdents d0 0 2024-03-30T04:28:04.319 INFO:tasks.workunit.client.0.smithi123.stdout:2/269: write d0/d1f/d1c/f3f [367197,77573] 0 2024-03-30T04:28:04.323 INFO:tasks.workunit.client.0.smithi123.stdout:2/270: rename d0/d1f/d1c/d2a/l2f to d0/d1f/d1c/d40/l53 0 2024-03-30T04:28:04.324 INFO:tasks.workunit.client.0.smithi123.stdout:2/271: chown d0/d1f/d30/c31 1 1 2024-03-30T04:28:04.328 INFO:tasks.workunit.client.0.smithi123.stdout:2/272: dread d0/d1f/d16/f23 [0,4194304] 0 2024-03-30T04:28:04.336 INFO:tasks.workunit.client.0.smithi123.stdout:2/273: dread d0/d1f/fa [0,4194304] 0 2024-03-30T04:28:04.345 INFO:tasks.workunit.client.0.smithi123.stdout:2/274: dread d0/d1f/d16/f3d [0,4194304] 0 2024-03-30T04:28:04.347 INFO:tasks.workunit.client.0.smithi123.stdout:2/275: creat d0/d1f/d1c/d20/f54 x:0 0 0 2024-03-30T04:28:04.349 INFO:tasks.workunit.client.0.smithi123.stdout:2/276: write d0/d1f/d1c/d20/f46 [273188,35521] 0 2024-03-30T04:28:04.351 INFO:tasks.workunit.client.0.smithi123.stdout:2/277: write d0/d1f/d16/f33 [252470,74357] 0 2024-03-30T04:28:04.354 INFO:tasks.workunit.client.0.smithi123.stdout:2/278: creat d0/d1f/d1c/d39/d44/f55 x:0 0 0 2024-03-30T04:28:04.356 INFO:tasks.workunit.client.0.smithi123.stdout:2/279: dread - d0/d1f/d1c/d39/d44/f49 zero size 2024-03-30T04:28:04.358 INFO:tasks.workunit.client.0.smithi123.stdout:2/280: creat d0/d1f/d1c/d20/f56 x:0 0 0 2024-03-30T04:28:04.360 INFO:tasks.workunit.client.0.smithi123.stdout:2/281: read d0/d1f/d1c/d40/f43 [4120009,123086] 0 2024-03-30T04:28:04.362 INFO:tasks.workunit.client.0.smithi123.stdout:2/282: mkdir d0/d37/d57 0 2024-03-30T04:28:04.363 INFO:tasks.workunit.client.0.smithi123.stdout:2/283: read - d0/d1f/d16/f3c zero size 2024-03-30T04:28:04.366 INFO:tasks.workunit.client.0.smithi123.stdout:2/284: symlink d0/d1f/d1c/d36/l58 0 2024-03-30T04:28:04.377 INFO:tasks.workunit.client.0.smithi123.stdout:2/285: dwrite d0/d1f/f24 [0,4194304] 0 2024-03-30T04:28:04.379 INFO:tasks.workunit.client.0.smithi123.stdout:2/286: truncate d0/d1f/f17 4444406 0 2024-03-30T04:28:04.382 INFO:tasks.workunit.client.0.smithi123.stdout:2/287: unlink d0/d1f/d1c/d20/f54 0 2024-03-30T04:28:04.385 INFO:tasks.workunit.client.0.smithi123.stdout:2/288: symlink d0/d1f/d1c/d2a/l59 0 2024-03-30T04:28:04.387 INFO:tasks.workunit.client.0.smithi123.stdout:2/289: stat d0/d1f/d16/c48 0 2024-03-30T04:28:04.388 INFO:tasks.workunit.client.0.smithi123.stdout:2/290: fdatasync d0/d1f/f17 0 2024-03-30T04:28:04.401 INFO:tasks.workunit.client.0.smithi123.stdout:2/291: dwrite d0/d1f/d1c/f51 [0,4194304] 0 2024-03-30T04:28:04.411 INFO:tasks.workunit.client.0.smithi123.stdout:2/292: dread d0/d1f/fa [0,4194304] 0 2024-03-30T04:28:04.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:04 smithi123 ceph-mon[35245]: pgmap v35: 65 pgs: 65 active+clean; 454 MiB data, 3.2 GiB used, 533 GiB / 536 GiB avail; 2.2 MiB/s rd, 39 MiB/s wr, 212 op/s 2024-03-30T04:28:04.483 INFO:tasks.workunit.client.0.smithi123.stdout:6/360: sync 2024-03-30T04:28:04.486 INFO:tasks.workunit.client.0.smithi123.stdout:6/361: creat d10/d13/d1c/d5b/f85 x:0 0 0 2024-03-30T04:28:04.491 INFO:tasks.workunit.client.0.smithi123.stdout:6/362: rename d10/d13/d1c/d42/c64 to d10/d13/d6a/c86 0 2024-03-30T04:28:04.493 INFO:tasks.workunit.client.0.smithi123.stdout:6/363: creat d10/d13/d1c/d42/f87 x:0 0 0 2024-03-30T04:28:04.495 INFO:tasks.workunit.client.0.smithi123.stdout:6/364: symlink d10/d82/l88 0 2024-03-30T04:28:04.497 INFO:tasks.workunit.client.0.smithi123.stdout:6/365: creat d10/d13/d1c/d5b/f89 x:0 0 0 2024-03-30T04:28:04.498 INFO:tasks.workunit.client.0.smithi123.stdout:6/366: dread - d10/d13/d45/f67 zero size 2024-03-30T04:28:04.498 INFO:tasks.workunit.client.0.smithi123.stdout:6/367: stat d10/d13 0 2024-03-30T04:28:04.519 INFO:tasks.workunit.client.0.smithi123.stdout:6/368: dread d10/d13/d1c/f1e [4194304,4194304] 0 2024-03-30T04:28:04.521 INFO:tasks.workunit.client.0.smithi123.stdout:6/369: dread - d10/d13/d1c/f2c zero size 2024-03-30T04:28:04.568 INFO:tasks.workunit.client.0.smithi123.stdout:0/166: sync 2024-03-30T04:28:04.568 INFO:tasks.workunit.client.0.smithi123.stdout:8/250: sync 2024-03-30T04:28:04.568 INFO:tasks.workunit.client.0.smithi123.stdout:1/393: sync 2024-03-30T04:28:04.568 INFO:tasks.workunit.client.0.smithi123.stdout:7/215: sync 2024-03-30T04:28:04.568 INFO:tasks.workunit.client.0.smithi123.stdout:4/303: sync 2024-03-30T04:28:04.570 INFO:tasks.workunit.client.0.smithi123.stdout:8/251: mknod da/d27/d3c/c41 0 2024-03-30T04:28:04.572 INFO:tasks.workunit.client.0.smithi123.stdout:0/167: mkdir d6/dd/d16/d32 0 2024-03-30T04:28:04.574 INFO:tasks.workunit.client.0.smithi123.stdout:1/394: rename db/dd/f82 to db/dd/d5f/d6d/f83 0 2024-03-30T04:28:04.575 INFO:tasks.workunit.client.0.smithi123.stdout:4/304: mkdir d2/d4/d7/d1a/d71 0 2024-03-30T04:28:04.576 INFO:tasks.workunit.client.0.smithi123.stdout:1/395: chown db/dd/d42/d46/d5d/d62 0 1 2024-03-30T04:28:04.577 INFO:tasks.workunit.client.0.smithi123.stdout:0/168: getdents d6/dd/d15/d28 0 2024-03-30T04:28:04.578 INFO:tasks.workunit.client.0.smithi123.stdout:8/252: write da/dc/f13 [763200,15607] 0 2024-03-30T04:28:04.580 INFO:tasks.workunit.client.0.smithi123.stdout:0/169: chown d6/dd/d16/l2e 17979806 1 2024-03-30T04:28:04.581 INFO:tasks.workunit.client.0.smithi123.stdout:1/396: write db/d1b/f3e [986164,69502] 0 2024-03-30T04:28:04.589 INFO:tasks.workunit.client.0.smithi123.stdout:4/305: link d2/d4/d7/l36 d2/d4/d2c/l72 0 2024-03-30T04:28:04.590 INFO:tasks.workunit.client.0.smithi123.stdout:1/397: symlink db/dd/d42/d46/d5d/d7a/l84 0 2024-03-30T04:28:04.590 INFO:tasks.workunit.client.0.smithi123.stdout:7/216: dwrite d2/dc/d1c/f23 [0,4194304] 0 2024-03-30T04:28:04.593 INFO:tasks.workunit.client.0.smithi123.stdout:8/253: dread da/f12 [4194304,4194304] 0 2024-03-30T04:28:04.594 INFO:tasks.workunit.client.0.smithi123.stdout:1/398: write db/dd/d42/d46/f29 [3538663,59756] 0 2024-03-30T04:28:04.598 INFO:tasks.workunit.client.0.smithi123.stdout:8/254: creat da/dc/f42 x:0 0 0 2024-03-30T04:28:04.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:04 smithi135 ceph-mon[44364]: pgmap v35: 65 pgs: 65 active+clean; 454 MiB data, 3.2 GiB used, 533 GiB / 536 GiB avail; 2.2 MiB/s rd, 39 MiB/s wr, 212 op/s 2024-03-30T04:28:04.601 INFO:tasks.workunit.client.0.smithi123.stdout:4/306: creat d2/d4/d2c/d42/d35/d5d/d6f/f73 x:0 0 0 2024-03-30T04:28:04.607 INFO:tasks.workunit.client.0.smithi123.stdout:4/307: write d2/d4/d7/d1a/f1f [7269270,95907] 0 2024-03-30T04:28:04.610 INFO:tasks.workunit.client.0.smithi123.stdout:4/308: read - d2/d26/f60 zero size 2024-03-30T04:28:04.614 INFO:tasks.workunit.client.0.smithi123.stdout:7/217: dwrite d2/da/fb [0,4194304] 0 2024-03-30T04:28:04.621 INFO:tasks.workunit.client.0.smithi123.stdout:8/255: dwrite f5 [4194304,4194304] 0 2024-03-30T04:28:04.624 INFO:tasks.workunit.client.0.smithi123.stdout:7/218: unlink d2/da/d11/d34/c37 0 2024-03-30T04:28:04.627 INFO:tasks.workunit.client.0.smithi123.stdout:8/256: symlink da/d27/l43 0 2024-03-30T04:28:04.630 INFO:tasks.workunit.client.0.smithi123.stdout:1/399: dread db/f20 [0,4194304] 0 2024-03-30T04:28:04.631 INFO:tasks.workunit.client.0.smithi123.stdout:8/257: creat da/dc/d14/f44 x:0 0 0 2024-03-30T04:28:04.633 INFO:tasks.workunit.client.0.smithi123.stdout:4/309: rename d2/d4/d7/d1a/l69 to d2/d4/d2c/l74 0 2024-03-30T04:28:04.636 INFO:tasks.workunit.client.0.smithi123.stdout:7/219: creat d2/da/f4c x:0 0 0 2024-03-30T04:28:04.636 INFO:tasks.workunit.client.0.smithi123.stdout:1/400: mkdir db/dd/d5f/d6d/d85 0 2024-03-30T04:28:04.638 INFO:tasks.workunit.client.0.smithi123.stdout:7/220: fsync d2/da/d22/f2f 0 2024-03-30T04:28:04.641 INFO:tasks.workunit.client.0.smithi123.stdout:4/310: unlink d2/d4/d7/d1a/f5f 0 2024-03-30T04:28:04.641 INFO:tasks.workunit.client.0.smithi123.stdout:8/258: rename da/f12 to da/dc/d14/d18/d21/f45 0 2024-03-30T04:28:04.643 INFO:tasks.workunit.client.0.smithi123.stdout:8/259: chown da/d27/l29 8330652 1 2024-03-30T04:28:04.644 INFO:tasks.workunit.client.0.smithi123.stdout:1/401: mknod db/d1b/c86 0 2024-03-30T04:28:04.644 INFO:tasks.workunit.client.0.smithi123.stdout:7/221: mkdir d2/d4d 0 2024-03-30T04:28:04.646 INFO:tasks.workunit.client.0.smithi123.stdout:7/222: stat d2/dc/c10 0 2024-03-30T04:28:04.646 INFO:tasks.workunit.client.0.smithi123.stdout:1/402: chown db/dd/d42/d46/d51/c58 1812608 1 2024-03-30T04:28:04.646 INFO:tasks.workunit.client.0.smithi123.stdout:4/311: rmdir d2/d4/d2c/d42/d35/d5d/d6f 39 2024-03-30T04:28:04.648 INFO:tasks.workunit.client.0.smithi123.stdout:1/403: fsync db/d1b/f1d 0 2024-03-30T04:28:04.649 INFO:tasks.workunit.client.0.smithi123.stdout:7/223: truncate d2/da/f4c 1004707 0 2024-03-30T04:28:04.654 INFO:tasks.workunit.client.0.smithi123.stdout:8/260: link da/f11 da/f46 0 2024-03-30T04:28:04.655 INFO:tasks.workunit.client.0.smithi123.stdout:1/404: creat db/dd/d5f/f87 x:0 0 0 2024-03-30T04:28:04.657 INFO:tasks.workunit.client.0.smithi123.stdout:7/224: creat d2/da/d22/d3b/f4e x:0 0 0 2024-03-30T04:28:04.657 INFO:tasks.workunit.client.0.smithi123.stdout:1/405: chown db/dd/d42/d46/d39 0 1 2024-03-30T04:28:04.658 INFO:tasks.workunit.client.0.smithi123.stdout:7/225: chown d2/l1a 2903 1 2024-03-30T04:28:04.659 INFO:tasks.workunit.client.0.smithi123.stdout:8/261: write da/f46 [2896137,60862] 0 2024-03-30T04:28:04.662 INFO:tasks.workunit.client.0.smithi123.stdout:7/226: mknod d2/dc/c4f 0 2024-03-30T04:28:04.664 INFO:tasks.workunit.client.0.smithi123.stdout:8/262: fdatasync da/f3d 0 2024-03-30T04:28:04.665 INFO:tasks.workunit.client.0.smithi123.stdout:1/406: rename db/dd/c66 to db/dd/d42/d46/d51/d72/c88 0 2024-03-30T04:28:04.666 INFO:tasks.workunit.client.0.smithi123.stdout:7/227: rmdir d2/dc/d1c 39 2024-03-30T04:28:04.680 INFO:tasks.workunit.client.0.smithi123.stdout:7/228: dwrite d2/da/d2d/f39 [0,4194304] 0 2024-03-30T04:28:04.682 INFO:tasks.workunit.client.0.smithi123.stdout:7/229: stat d2/dc/d1c/l24 0 2024-03-30T04:28:04.683 INFO:tasks.workunit.client.0.smithi123.stdout:7/230: write d2/da/d11/f20 [1451998,26078] 0 2024-03-30T04:28:04.698 INFO:tasks.workunit.client.0.smithi123.stdout:7/231: dwrite d2/dc/d1c/f1d [0,4194304] 0 2024-03-30T04:28:04.699 INFO:tasks.workunit.client.0.smithi123.stdout:3/315: sync 2024-03-30T04:28:04.699 INFO:tasks.workunit.client.0.smithi123.stdout:2/293: sync 2024-03-30T04:28:04.699 INFO:tasks.workunit.client.0.smithi123.stdout:6/370: sync 2024-03-30T04:28:04.699 INFO:tasks.workunit.client.0.smithi123.stdout:5/243: sync 2024-03-30T04:28:04.699 INFO:tasks.workunit.client.0.smithi123.stdout:9/215: sync 2024-03-30T04:28:04.702 INFO:tasks.workunit.client.0.smithi123.stdout:7/232: creat d2/d4d/f50 x:0 0 0 2024-03-30T04:28:04.703 INFO:tasks.workunit.client.0.smithi123.stdout:7/233: fdatasync d2/f13 0 2024-03-30T04:28:04.708 INFO:tasks.workunit.client.0.smithi123.stdout:2/294: fdatasync d0/d1f/f1b 0 2024-03-30T04:28:04.710 INFO:tasks.workunit.client.0.smithi123.stdout:3/316: rename d13/d24/c34 to d13/d24/d39/c5f 0 2024-03-30T04:28:04.711 INFO:tasks.workunit.client.0.smithi123.stdout:7/234: write d2/da/d11/f20 [2684131,16220] 0 2024-03-30T04:28:04.713 INFO:tasks.workunit.client.0.smithi123.stdout:2/295: creat d0/d1f/d1c/d40/f5a x:0 0 0 2024-03-30T04:28:04.714 INFO:tasks.workunit.client.0.smithi123.stdout:7/235: chown d2/dc/d1c 31081 1 2024-03-30T04:28:04.714 INFO:tasks.workunit.client.0.smithi123.stdout:3/317: fdatasync d13/d24/d31/d30/f5a 0 2024-03-30T04:28:04.714 INFO:tasks.workunit.client.0.smithi123.stdout:6/371: truncate f6 1044812 0 2024-03-30T04:28:04.718 INFO:tasks.workunit.client.0.smithi123.stdout:7/236: write d2/dc/d1c/f1d [982445,108836] 0 2024-03-30T04:28:04.718 INFO:tasks.workunit.client.0.smithi123.stdout:9/216: dwrite d6/fd [4194304,4194304] 0 2024-03-30T04:28:04.722 INFO:tasks.workunit.client.0.smithi123.stdout:3/318: read fd [2941198,110625] 0 2024-03-30T04:28:04.724 INFO:tasks.workunit.client.0.smithi123.stdout:9/217: creat d6/d1f/f4f x:0 0 0 2024-03-30T04:28:04.735 INFO:tasks.workunit.client.0.smithi123.stdout:3/319: link d13/d24/d31/d23/c3b d13/d24/d31/d23/c60 0 2024-03-30T04:28:04.738 INFO:tasks.workunit.client.0.smithi123.stdout:3/320: dread - d13/d24/d31/d23/d3a/f54 zero size 2024-03-30T04:28:04.742 INFO:tasks.workunit.client.0.smithi123.stdout:6/372: dwrite d10/d20/f7f [0,4194304] 0 2024-03-30T04:28:04.745 INFO:tasks.workunit.client.0.smithi123.stdout:3/321: chown d13/d24/d31/d23/d3a/d3c/d4e/c55 63902872 1 2024-03-30T04:28:04.745 INFO:tasks.workunit.client.0.smithi123.stdout:6/373: dread - d10/d82/f84 zero size 2024-03-30T04:28:04.746 INFO:tasks.workunit.client.0.smithi123.stdout:7/237: dwrite d2/dc/d1c/f1d [4194304,4194304] 0 2024-03-30T04:28:04.750 INFO:tasks.workunit.client.0.smithi123.stdout:9/218: dwrite d6/d1f/f2e [0,4194304] 0 2024-03-30T04:28:04.752 INFO:tasks.workunit.client.0.smithi123.stdout:6/374: write d10/d13/d1c/d5b/f85 [20898,76645] 0 2024-03-30T04:28:04.758 INFO:tasks.workunit.client.0.smithi123.stdout:6/375: creat d10/d82/f8a x:0 0 0 2024-03-30T04:28:04.764 INFO:tasks.workunit.client.0.smithi123.stdout:7/238: dread d2/da/d11/f1e [0,4194304] 0 2024-03-30T04:28:04.765 INFO:tasks.workunit.client.0.smithi123.stdout:6/376: getdents d10/d13/d6a 0 2024-03-30T04:28:04.766 INFO:tasks.workunit.client.0.smithi123.stdout:7/239: chown d2/dc/c10 5820 1 2024-03-30T04:28:04.769 INFO:tasks.workunit.client.0.smithi123.stdout:7/240: write d2/f8 [2482131,101387] 0 2024-03-30T04:28:04.770 INFO:tasks.workunit.client.0.smithi123.stdout:6/377: creat d10/d13/f8b x:0 0 0 2024-03-30T04:28:04.773 INFO:tasks.workunit.client.0.smithi123.stdout:9/219: dwrite d6/d23/f33 [4194304,4194304] 0 2024-03-30T04:28:04.778 INFO:tasks.workunit.client.0.smithi123.stdout:7/241: link d2/da/f33 d2/da/d2d/f51 0 2024-03-30T04:28:04.788 INFO:tasks.workunit.client.0.smithi123.stdout:6/378: truncate d10/f5a 2386600 0 2024-03-30T04:28:04.791 INFO:tasks.workunit.client.0.smithi123.stdout:6/379: readlink d10/l50 0 2024-03-30T04:28:04.792 INFO:tasks.workunit.client.0.smithi123.stdout:9/220: dread d6/d1f/f2e [0,4194304] 0 2024-03-30T04:28:04.794 INFO:tasks.workunit.client.0.smithi123.stdout:6/380: mknod d10/d13/d6a/d7a/c8c 0 2024-03-30T04:28:04.795 INFO:tasks.workunit.client.0.smithi123.stdout:9/221: creat d6/d1f/d45/f50 x:0 0 0 2024-03-30T04:28:04.795 INFO:tasks.workunit.client.0.smithi123.stdout:7/242: dread d2/da/fb [0,4194304] 0 2024-03-30T04:28:04.797 INFO:tasks.workunit.client.0.smithi123.stdout:9/222: write d6/f10 [2407215,107417] 0 2024-03-30T04:28:04.798 INFO:tasks.workunit.client.0.smithi123.stdout:6/381: write d10/d13/d45/d77/f7c [702016,84089] 0 2024-03-30T04:28:04.801 INFO:tasks.workunit.client.0.smithi123.stdout:7/243: creat d2/dc/f52 x:0 0 0 2024-03-30T04:28:04.803 INFO:tasks.workunit.client.0.smithi123.stdout:6/382: write d10/d13/d1c/f2a [2551502,18928] 0 2024-03-30T04:28:04.804 INFO:tasks.workunit.client.0.smithi123.stdout:9/223: creat d6/f51 x:0 0 0 2024-03-30T04:28:04.805 INFO:tasks.workunit.client.0.smithi123.stdout:7/244: readlink d2/l7 0 2024-03-30T04:28:04.808 INFO:tasks.workunit.client.0.smithi123.stdout:6/383: write d10/d20/f58 [929429,93782] 0 2024-03-30T04:28:04.811 INFO:tasks.workunit.client.0.smithi123.stdout:7/245: mknod d2/dc/c53 0 2024-03-30T04:28:04.812 INFO:tasks.workunit.client.0.smithi123.stdout:7/246: readlink d2/da/d11/d34/l46 0 2024-03-30T04:28:04.813 INFO:tasks.workunit.client.0.smithi123.stdout:7/247: fdatasync d2/f41 0 2024-03-30T04:28:04.814 INFO:tasks.workunit.client.0.smithi123.stdout:6/384: write d10/d13/d1c/d42/f47 [642824,52332] 0 2024-03-30T04:28:04.816 INFO:tasks.workunit.client.0.smithi123.stdout:6/385: chown d10/d13/f8b 2176717 1 2024-03-30T04:28:04.817 INFO:tasks.workunit.client.0.smithi123.stdout:7/248: truncate d2/f8 5655560 0 2024-03-30T04:28:04.821 INFO:tasks.workunit.client.0.smithi123.stdout:7/249: creat d2/da/d22/d3b/f54 x:0 0 0 2024-03-30T04:28:04.822 INFO:tasks.workunit.client.0.smithi123.stdout:7/250: write f0 [3150053,113962] 0 2024-03-30T04:28:04.825 INFO:tasks.workunit.client.0.smithi123.stdout:7/251: mkdir d2/dc/d3a/d55 0 2024-03-30T04:28:04.828 INFO:tasks.workunit.client.0.smithi123.stdout:7/252: mkdir d2/dc/d56 0 2024-03-30T04:28:04.829 INFO:tasks.workunit.client.0.smithi123.stdout:9/224: dwrite d6/d23/f2b [0,4194304] 0 2024-03-30T04:28:04.839 INFO:tasks.workunit.client.0.smithi123.stdout:6/386: dread d10/d13/d1c/f1e [0,4194304] 0 2024-03-30T04:28:04.839 INFO:tasks.workunit.client.0.smithi123.stdout:9/225: link d6/d23/c47 d6/d1f/d45/c52 0 2024-03-30T04:28:04.840 INFO:tasks.workunit.client.0.smithi123.stdout:9/226: fdatasync d6/d1f/f3f 0 2024-03-30T04:28:04.841 INFO:tasks.workunit.client.0.smithi123.stdout:9/227: stat d6/d23/f46 0 2024-03-30T04:28:04.842 INFO:tasks.workunit.client.0.smithi123.stdout:6/387: creat d10/d13/d1c/d5b/f8d x:0 0 0 2024-03-30T04:28:04.843 INFO:tasks.workunit.client.0.smithi123.stdout:9/228: dread - d6/d23/f46 zero size 2024-03-30T04:28:04.845 INFO:tasks.workunit.client.0.smithi123.stdout:6/388: chown d10/d13/d1c/d5b/f8d 24679031 1 2024-03-30T04:28:04.845 INFO:tasks.workunit.client.0.smithi123.stdout:6/389: chown d10/f59 4 1 2024-03-30T04:28:04.846 INFO:tasks.workunit.client.0.smithi123.stdout:7/253: dwrite d2/dc/f27 [0,4194304] 0 2024-03-30T04:28:04.849 INFO:tasks.workunit.client.0.smithi123.stdout:6/390: mkdir d10/d82/d8e 0 2024-03-30T04:28:04.850 INFO:tasks.workunit.client.0.smithi123.stdout:7/254: creat d2/dc/d3a/f57 x:0 0 0 2024-03-30T04:28:04.851 INFO:tasks.workunit.client.0.smithi123.stdout:6/391: stat d10/d13/d1c/d5b/f8d 0 2024-03-30T04:28:04.859 INFO:tasks.workunit.client.0.smithi123.stdout:6/392: mknod d10/d13/d1c/d5b/c8f 0 2024-03-30T04:28:04.861 INFO:tasks.workunit.client.0.smithi123.stdout:9/229: dwrite d6/d1f/f4f [0,4194304] 0 2024-03-30T04:28:04.868 INFO:tasks.workunit.client.0.smithi123.stdout:7/255: dread d2/da/d11/f3c [0,4194304] 0 2024-03-30T04:28:04.869 INFO:tasks.workunit.client.0.smithi123.stdout:6/393: dread d10/f1b [4194304,4194304] 0 2024-03-30T04:28:04.871 INFO:tasks.workunit.client.0.smithi123.stdout:6/394: read - fe zero size 2024-03-30T04:28:04.875 INFO:tasks.workunit.client.0.smithi123.stdout:6/395: unlink d10/d13/d1c/d42/c49 0 2024-03-30T04:28:04.878 INFO:tasks.workunit.client.0.smithi123.stdout:6/396: dread d10/d20/f36 [0,4194304] 0 2024-03-30T04:28:04.880 INFO:tasks.workunit.client.0.smithi123.stdout:7/256: truncate d2/da/d2d/f39 2386793 0 2024-03-30T04:28:04.881 INFO:tasks.workunit.client.0.smithi123.stdout:6/397: creat d10/d13/d6a/f90 x:0 0 0 2024-03-30T04:28:04.885 INFO:tasks.workunit.client.0.smithi123.stdout:6/398: rename d10/d20/f56 to d10/d20/d3f/f91 0 2024-03-30T04:28:04.887 INFO:tasks.workunit.client.0.smithi123.stdout:6/399: write d10/d13/d45/d77/f7c [762703,105731] 0 2024-03-30T04:28:04.894 INFO:tasks.workunit.client.0.smithi123.stdout:6/400: dread d10/d20/d3f/f4d [0,4194304] 0 2024-03-30T04:28:04.894 INFO:tasks.workunit.client.0.smithi123.stdout:7/257: dwrite d2/da/d11/f1e [0,4194304] 0 2024-03-30T04:28:04.896 INFO:tasks.workunit.client.0.smithi123.stdout:6/401: fdatasync d10/d13/d45/d77/f7c 0 2024-03-30T04:28:04.900 INFO:tasks.workunit.client.0.smithi123.stdout:7/258: creat d2/da/d11/f58 x:0 0 0 2024-03-30T04:28:04.902 INFO:tasks.workunit.client.0.smithi123.stdout:7/259: mkdir d2/dc/d3a/d55/d59 0 2024-03-30T04:28:04.906 INFO:tasks.workunit.client.0.smithi123.stdout:7/260: rename d2/l1a to d2/dc/d3a/d55/l5a 0 2024-03-30T04:28:04.910 INFO:tasks.workunit.client.0.smithi123.stdout:7/261: rename d2/cd to d2/da/d2d/c5b 0 2024-03-30T04:28:04.912 INFO:tasks.workunit.client.0.smithi123.stdout:6/402: dwrite d10/d13/d1c/f2c [0,4194304] 0 2024-03-30T04:28:04.914 INFO:tasks.workunit.client.0.smithi123.stdout:7/262: rename d2/l6 to d2/da/d2d/l5c 0 2024-03-30T04:28:04.915 INFO:tasks.workunit.client.0.smithi123.stdout:6/403: truncate d10/d82/f84 944974 0 2024-03-30T04:28:04.932 INFO:tasks.workunit.client.0.smithi123.stdout:7/263: dwrite d2/da/d11/f20 [0,4194304] 0 2024-03-30T04:28:04.933 INFO:tasks.workunit.client.0.smithi123.stdout:6/404: dwrite d10/d82/f8a [0,4194304] 0 2024-03-30T04:28:04.935 INFO:tasks.workunit.client.0.smithi123.stdout:6/405: fsync fe 0 2024-03-30T04:28:04.956 INFO:tasks.workunit.client.0.smithi123.stdout:7/264: dread d2/da/d11/f45 [8388608,4194304] 0 2024-03-30T04:28:04.972 INFO:tasks.workunit.client.0.smithi123.stdout:6/406: dread d10/f1b [0,4194304] 0 2024-03-30T04:28:04.976 INFO:tasks.workunit.client.0.smithi123.stdout:7/265: dwrite d2/da/d2d/f51 [0,4194304] 0 2024-03-30T04:28:04.980 INFO:tasks.workunit.client.0.smithi123.stdout:7/266: symlink d2/dc/d49/l5d 0 2024-03-30T04:28:04.982 INFO:tasks.workunit.client.0.smithi123.stdout:7/267: readlink d2/da/d22/l4a 0 2024-03-30T04:28:04.985 INFO:tasks.workunit.client.0.smithi123.stdout:7/268: mknod d2/dc/d3a/d55/d59/c5e 0 2024-03-30T04:28:04.990 INFO:tasks.workunit.client.0.smithi123.stdout:7/269: link d2/da/d2d/c2e d2/d4d/c5f 0 2024-03-30T04:28:04.999 INFO:tasks.workunit.client.0.smithi123.stdout:7/270: truncate d2/da/d2d/f2a 243086 0 2024-03-30T04:28:05.000 INFO:tasks.workunit.client.0.smithi123.stdout:7/271: write d2/da/f33 [981320,42016] 0 2024-03-30T04:28:05.002 INFO:tasks.workunit.client.0.smithi123.stdout:7/272: chown d2/da/d11/c32 58 1 2024-03-30T04:28:05.006 INFO:tasks.workunit.client.0.smithi123.stdout:7/273: creat d2/dc/d56/f60 x:0 0 0 2024-03-30T04:28:05.009 INFO:tasks.workunit.client.0.smithi123.stdout:7/274: rename d2/dc/d3a to d2/da/d11/d61 0 2024-03-30T04:28:05.011 INFO:tasks.workunit.client.0.smithi123.stdout:7/275: write d2/da/d11/f1e [4624244,67985] 0 2024-03-30T04:28:05.013 INFO:tasks.workunit.client.0.smithi123.stdout:0/170: sync 2024-03-30T04:28:05.014 INFO:tasks.workunit.client.0.smithi123.stdout:7/276: unlink d2/da/d11/d34/l46 0 2024-03-30T04:28:05.016 INFO:tasks.workunit.client.0.smithi123.stdout:7/277: write d2/dc/d1c/f1d [5091452,122502] 0 2024-03-30T04:28:05.016 INFO:tasks.workunit.client.0.smithi123.stdout:0/171: creat d6/dd/d15/d28/f33 x:0 0 0 2024-03-30T04:28:05.019 INFO:tasks.workunit.client.0.smithi123.stdout:0/172: truncate d6/dd/f14 259646 0 2024-03-30T04:28:05.020 INFO:tasks.workunit.client.0.smithi123.stdout:0/173: dread - d6/dd/d15/f2b zero size 2024-03-30T04:28:05.023 INFO:tasks.workunit.client.0.smithi123.stdout:0/174: getdents d6 0 2024-03-30T04:28:05.025 INFO:tasks.workunit.client.0.smithi123.stdout:0/175: mknod d6/dd/d16/c34 0 2024-03-30T04:28:05.028 INFO:tasks.workunit.client.0.smithi123.stdout:0/176: mkdir d6/dd/d31/d35 0 2024-03-30T04:28:05.029 INFO:tasks.workunit.client.0.smithi123.stdout:0/177: chown d6/f1b 260 1 2024-03-30T04:28:05.030 INFO:tasks.workunit.client.0.smithi123.stdout:0/178: chown d6/f12 394 1 2024-03-30T04:28:05.033 INFO:tasks.workunit.client.0.smithi123.stdout:0/179: creat d6/dd/f36 x:0 0 0 2024-03-30T04:28:05.034 INFO:tasks.workunit.client.0.smithi123.stdout:0/180: stat d6/dd/d18/d1f 0 2024-03-30T04:28:05.045 INFO:tasks.workunit.client.0.smithi123.stdout:0/181: dwrite f5 [0,4194304] 0 2024-03-30T04:28:05.050 INFO:tasks.workunit.client.0.smithi123.stdout:0/182: dread d6/dd/f14 [0,4194304] 0 2024-03-30T04:28:05.052 INFO:tasks.workunit.client.0.smithi123.stdout:0/183: write d6/dd/d15/d28/f33 [297479,101835] 0 2024-03-30T04:28:05.053 INFO:tasks.workunit.client.0.smithi123.stdout:0/184: fdatasync d6/dd/f36 0 2024-03-30T04:28:05.056 INFO:tasks.workunit.client.0.smithi123.stdout:0/185: mknod d6/dd/d15/d28/c37 0 2024-03-30T04:28:05.056 INFO:tasks.workunit.client.0.smithi123.stdout:0/186: stat d6/dd/d15 0 2024-03-30T04:28:05.062 INFO:tasks.workunit.client.0.smithi123.stdout:0/187: symlink d6/dd/d18/l38 0 2024-03-30T04:28:05.062 INFO:tasks.workunit.client.0.smithi123.stdout:0/188: unlink d6/dd/d18/l38 0 2024-03-30T04:28:05.068 INFO:tasks.workunit.client.0.smithi123.stdout:0/189: getdents d6/dd/d15/d28 0 2024-03-30T04:28:05.154 INFO:tasks.workunit.client.0.smithi123.stdout:8/263: sync 2024-03-30T04:28:05.154 INFO:tasks.workunit.client.0.smithi123.stdout:4/312: sync 2024-03-30T04:28:05.154 INFO:tasks.workunit.client.0.smithi123.stdout:1/407: sync 2024-03-30T04:28:05.157 INFO:tasks.workunit.client.0.smithi123.stdout:4/313: fsync d2/d4/d7/fb 0 2024-03-30T04:28:05.158 INFO:tasks.workunit.client.0.smithi123.stdout:4/314: chown d2 44 1 2024-03-30T04:28:05.158 INFO:tasks.workunit.client.0.smithi123.stdout:1/408: mkdir db/dd/d5f/d6d/d85/d89 0 2024-03-30T04:28:05.160 INFO:tasks.workunit.client.0.smithi123.stdout:4/315: write d2/d4/d2c/d42/f38 [2685134,71268] 0 2024-03-30T04:28:05.162 INFO:tasks.workunit.client.0.smithi123.stdout:4/316: readlink d2/d4/l15 0 2024-03-30T04:28:05.164 INFO:tasks.workunit.client.0.smithi123.stdout:1/409: rename db/dd/d42/f6c to db/dd/d42/d46/d39/f8a 0 2024-03-30T04:28:05.166 INFO:tasks.workunit.client.0.smithi123.stdout:4/317: fdatasync d2/d4/d2c/d42/d35/d5d/d6f/f73 0 2024-03-30T04:28:05.167 INFO:tasks.workunit.client.0.smithi123.stdout:1/410: mknod db/dd/d5f/c8b 0 2024-03-30T04:28:05.169 INFO:tasks.workunit.client.0.smithi123.stdout:1/411: dread - db/dd/d42/d46/d5d/d62/f6b zero size 2024-03-30T04:28:05.194 INFO:tasks.workunit.client.0.smithi123.stdout:4/318: dread d2/d34/f4c [0,4194304] 0 2024-03-30T04:28:05.195 INFO:tasks.workunit.client.0.smithi123.stdout:1/412: dread db/dd/f48 [0,4194304] 0 2024-03-30T04:28:05.197 INFO:tasks.workunit.client.0.smithi123.stdout:1/413: dread - db/dd/d5f/f87 zero size 2024-03-30T04:28:05.197 INFO:tasks.workunit.client.0.smithi123.stdout:4/319: unlink d2/d4/c24 0 2024-03-30T04:28:05.198 INFO:tasks.workunit.client.0.smithi123.stdout:4/320: stat d2/d26 0 2024-03-30T04:28:05.200 INFO:tasks.workunit.client.0.smithi123.stdout:1/414: mknod db/dd/d42/d46/d5d/d62/c8c 0 2024-03-30T04:28:05.201 INFO:tasks.workunit.client.0.smithi123.stdout:1/415: fdatasync db/f45 0 2024-03-30T04:28:05.202 INFO:tasks.workunit.client.0.smithi123.stdout:1/416: chown db/dd/l3b 1 1 2024-03-30T04:28:05.206 INFO:tasks.workunit.client.0.smithi123.stdout:1/417: creat db/dd/d42/d46/d51/f8d x:0 0 0 2024-03-30T04:28:05.212 INFO:tasks.workunit.client.0.smithi123.stdout:4/321: rmdir d2/d4/d7/d1a/d71 0 2024-03-30T04:28:05.215 INFO:tasks.workunit.client.0.smithi123.stdout:4/322: chown d2/d4/d2c/d42/c5b 10458 1 2024-03-30T04:28:05.217 INFO:tasks.workunit.client.0.smithi123.stdout:4/323: fdatasync d2/d4/d7/fe 0 2024-03-30T04:28:05.221 INFO:tasks.workunit.client.0.smithi123.stdout:4/324: rename d2/d4/d7 to d2/d4/d7/d75 22 2024-03-30T04:28:05.225 INFO:tasks.workunit.client.0.smithi123.stdout:4/325: rename d2/l62 to d2/d4/l76 0 2024-03-30T04:28:05.228 INFO:tasks.workunit.client.0.smithi123.stdout:1/418: dread db/d1b/f28 [0,4194304] 0 2024-03-30T04:28:05.230 INFO:tasks.workunit.client.0.smithi123.stdout:1/419: rmdir db/dd/d42/d46/d51/d72 39 2024-03-30T04:28:05.232 INFO:tasks.workunit.client.0.smithi123.stdout:4/326: link d2/d4/d7/f4e d2/d34/f77 0 2024-03-30T04:28:05.234 INFO:tasks.workunit.client.0.smithi123.stdout:1/420: creat db/dd/f8e x:0 0 0 2024-03-30T04:28:05.236 INFO:tasks.workunit.client.0.smithi123.stdout:1/421: write db/dd/d42/d46/d56/d5c/f76 [1561865,93771] 0 2024-03-30T04:28:05.238 INFO:tasks.workunit.client.0.smithi123.stdout:4/327: mkdir d2/d4/d2c/d42/d35/d5d/d78 0 2024-03-30T04:28:05.239 INFO:tasks.workunit.client.0.smithi123.stdout:1/422: readlink db/l2d 0 2024-03-30T04:28:05.239 INFO:tasks.workunit.client.0.smithi123.stdout:4/328: dread - d2/d26/f4f zero size 2024-03-30T04:28:05.241 INFO:tasks.workunit.client.0.smithi123.stdout:1/423: fdatasync db/dd/d42/d46/d56/f7f 0 2024-03-30T04:28:05.243 INFO:tasks.workunit.client.0.smithi123.stdout:4/329: symlink d2/d4/d7/l79 0 2024-03-30T04:28:05.244 INFO:tasks.workunit.client.0.smithi123.stdout:1/424: write db/f24 [2362426,18941] 0 2024-03-30T04:28:05.246 INFO:tasks.workunit.client.0.smithi123.stdout:4/330: symlink d2/d4/d7/d63/l7a 0 2024-03-30T04:28:05.247 INFO:tasks.workunit.client.0.smithi123.stdout:1/425: dread - db/dd/d5f/d6d/f83 zero size 2024-03-30T04:28:05.249 INFO:tasks.workunit.client.0.smithi123.stdout:1/426: write db/dd/d5f/f87 [437491,99480] 0 2024-03-30T04:28:05.254 INFO:tasks.workunit.client.0.smithi123.stdout:1/427: rename db/dd/d42/d46/d56/d5e to db/dd/d42/d8f 0 2024-03-30T04:28:05.260 INFO:tasks.workunit.client.0.smithi123.stdout:4/331: dwrite d2/d4/d7/d1a/f21 [0,4194304] 0 2024-03-30T04:28:05.264 INFO:tasks.workunit.client.0.smithi123.stdout:5/244: sync 2024-03-30T04:28:05.264 INFO:tasks.workunit.client.0.smithi123.stdout:4/332: write d2/d4/d7/f4b [8944279,37991] 0 2024-03-30T04:28:05.269 INFO:tasks.workunit.client.0.smithi123.stdout:1/428: truncate db/dd/d42/d46/f29 1405439 0 2024-03-30T04:28:05.270 INFO:tasks.workunit.client.0.smithi123.stdout:5/245: mknod dd/d19/d3e/c4c 0 2024-03-30T04:28:05.271 INFO:tasks.workunit.client.0.smithi123.stdout:4/333: mknod d2/c7b 0 2024-03-30T04:28:05.272 INFO:tasks.workunit.client.0.smithi123.stdout:5/246: write dd/d41/f47 [157334,27929] 0 2024-03-30T04:28:05.273 INFO:tasks.workunit.client.0.smithi123.stdout:4/334: fdatasync d2/d4/f22 0 2024-03-30T04:28:05.274 INFO:tasks.workunit.client.0.smithi123.stdout:5/247: chown dd/d19/d1f 35513865 1 2024-03-30T04:28:05.277 INFO:tasks.workunit.client.0.smithi123.stdout:5/248: symlink dd/d19/d1f/d35/d3a/l4d 0 2024-03-30T04:28:05.279 INFO:tasks.workunit.client.0.smithi123.stdout:5/249: truncate dd/d41/f43 471083 0 2024-03-30T04:28:05.279 INFO:tasks.workunit.client.0.smithi123.stdout:5/250: stat dd/f1c 0 2024-03-30T04:28:05.281 INFO:tasks.workunit.client.0.smithi123.stdout:5/251: creat dd/f4e x:0 0 0 2024-03-30T04:28:05.283 INFO:tasks.workunit.client.0.smithi123.stdout:5/252: rename dd/d19/d1f to dd/d19/d1f/d4f 22 2024-03-30T04:28:05.286 INFO:tasks.workunit.client.0.smithi123.stdout:5/253: symlink dd/d19/d1f/l50 0 2024-03-30T04:28:05.287 INFO:tasks.workunit.client.0.smithi123.stdout:4/335: dwrite d2/d4/f8 [0,4194304] 0 2024-03-30T04:28:05.289 INFO:tasks.workunit.client.0.smithi123.stdout:5/254: creat dd/d41/f51 x:0 0 0 2024-03-30T04:28:05.290 INFO:tasks.workunit.client.0.smithi123.stdout:4/336: creat d2/d34/f7c x:0 0 0 2024-03-30T04:28:05.290 INFO:tasks.workunit.client.0.smithi123.stdout:5/255: chown dd/fe 1 1 2024-03-30T04:28:05.293 INFO:tasks.workunit.client.0.smithi123.stdout:4/337: getdents d2/d4/d2c/d42/d35/d5d/d78 0 2024-03-30T04:28:05.295 INFO:tasks.workunit.client.0.smithi123.stdout:5/256: symlink dd/d19/d1f/d3c/l52 0 2024-03-30T04:28:05.297 INFO:tasks.workunit.client.0.smithi123.stdout:5/257: symlink dd/d19/d3e/l53 0 2024-03-30T04:28:05.300 INFO:tasks.workunit.client.0.smithi123.stdout:5/258: symlink dd/d19/d1f/d35/l54 0 2024-03-30T04:28:05.301 INFO:tasks.workunit.client.0.smithi123.stdout:5/259: chown dd/f15 4127422 1 2024-03-30T04:28:05.306 INFO:tasks.workunit.client.0.smithi123.stdout:5/260: rename dd/l20 to dd/d19/d1f/l55 0 2024-03-30T04:28:05.307 INFO:tasks.workunit.client.0.smithi123.stdout:5/261: fsync dd/d19/d1f/f27 0 2024-03-30T04:28:05.310 INFO:tasks.workunit.client.0.smithi123.stdout:5/262: creat dd/d19/d1f/d3c/f56 x:0 0 0 2024-03-30T04:28:05.313 INFO:tasks.workunit.client.0.smithi123.stdout:5/263: mkdir dd/d19/d1f/d35/d3a/d57 0 2024-03-30T04:28:05.318 INFO:tasks.workunit.client.0.smithi123.stdout:5/264: mknod dd/d19/d1f/d35/d3a/d57/c58 0 2024-03-30T04:28:05.319 INFO:tasks.workunit.client.0.smithi123.stdout:5/265: chown dd/d41/f43 9707606 1 2024-03-30T04:28:05.322 INFO:tasks.workunit.client.0.smithi123.stdout:5/266: creat dd/d19/f59 x:0 0 0 2024-03-30T04:28:05.326 INFO:tasks.workunit.client.0.smithi123.stdout:5/267: rmdir dd/d19/d1f/d35 39 2024-03-30T04:28:05.330 INFO:tasks.workunit.client.0.smithi123.stdout:5/268: symlink dd/d19/d1f/d35/d3a/l5a 0 2024-03-30T04:28:05.346 INFO:tasks.workunit.client.0.smithi123.stdout:5/269: dread fb [0,4194304] 0 2024-03-30T04:28:05.350 INFO:tasks.workunit.client.0.smithi123.stdout:5/270: unlink f7 0 2024-03-30T04:28:05.350 INFO:tasks.workunit.client.0.smithi123.stdout:5/271: chown dd/c18 10506761 1 2024-03-30T04:28:05.350 INFO:tasks.workunit.client.0.smithi123.stdout:5/272: read - dd/f2f zero size 2024-03-30T04:28:05.352 INFO:tasks.workunit.client.0.smithi123.stdout:5/273: rmdir dd/d19/d1f/d35/d3a 39 2024-03-30T04:28:05.354 INFO:tasks.workunit.client.0.smithi123.stdout:5/274: mkdir dd/d19/d3e/d5b 0 2024-03-30T04:28:05.360 INFO:tasks.workunit.client.0.smithi123.stdout:5/275: write dd/fe [5161180,25677] 0 2024-03-30T04:28:05.362 INFO:tasks.workunit.client.0.smithi123.stdout:5/276: write f9 [2683666,42437] 0 2024-03-30T04:28:05.369 INFO:tasks.workunit.client.0.smithi123.stdout:5/277: stat dd/f1d 0 2024-03-30T04:28:05.373 INFO:tasks.workunit.client.0.smithi123.stdout:5/278: read dd/f32 [1840935,61555] 0 2024-03-30T04:28:05.377 INFO:tasks.workunit.client.0.smithi123.stdout:9/230: sync 2024-03-30T04:28:05.377 INFO:tasks.workunit.client.0.smithi123.stdout:3/322: sync 2024-03-30T04:28:05.377 INFO:tasks.workunit.client.0.smithi123.stdout:2/296: sync 2024-03-30T04:28:05.377 INFO:tasks.workunit.client.0.smithi123.stdout:5/279: getdents dd/d41 0 2024-03-30T04:28:05.379 INFO:tasks.workunit.client.0.smithi123.stdout:3/323: dread - d13/d24/d31/d30/f5a zero size 2024-03-30T04:28:05.382 INFO:tasks.workunit.client.0.smithi123.stdout:5/280: mknod dd/d19/d1f/d35/d3a/c5c 0 2024-03-30T04:28:05.385 INFO:tasks.workunit.client.0.smithi123.stdout:3/324: getdents d13/d24/d31 0 2024-03-30T04:28:05.385 INFO:tasks.workunit.client.0.smithi123.stdout:3/325: chown ff 69433 1 2024-03-30T04:28:05.388 INFO:tasks.workunit.client.0.smithi123.stdout:5/281: link dd/f4e dd/d19/f5d 0 2024-03-30T04:28:05.389 INFO:tasks.workunit.client.0.smithi123.stdout:9/231: dread d6/d23/f3b [0,4194304] 0 2024-03-30T04:28:05.392 INFO:tasks.workunit.client.0.smithi123.stdout:5/282: creat dd/d19/d3e/d5b/f5e x:0 0 0 2024-03-30T04:28:05.393 INFO:tasks.workunit.client.0.smithi123.stdout:9/232: mknod d6/d49/c53 0 2024-03-30T04:28:05.395 INFO:tasks.workunit.client.0.smithi123.stdout:3/326: truncate d13/d24/f57 645453 0 2024-03-30T04:28:05.396 INFO:tasks.workunit.client.0.smithi123.stdout:9/233: creat d6/d1f/d45/d4c/f54 x:0 0 0 2024-03-30T04:28:05.397 INFO:tasks.workunit.client.0.smithi123.stdout:3/327: stat d13/d24/d31/d23/f25 0 2024-03-30T04:28:05.402 INFO:tasks.workunit.client.0.smithi123.stdout:9/234: getdents d6/d49 0 2024-03-30T04:28:05.405 INFO:tasks.workunit.client.0.smithi123.stdout:5/283: truncate dd/f15 3256640 0 2024-03-30T04:28:05.407 INFO:tasks.workunit.client.0.smithi123.stdout:5/284: mkdir dd/d19/d5f 0 2024-03-30T04:28:05.410 INFO:tasks.workunit.client.0.smithi123.stdout:9/235: link d6/l1e d6/d23/l55 0 2024-03-30T04:28:05.411 INFO:tasks.workunit.client.0.smithi123.stdout:5/285: write dd/f2f [445251,14884] 0 2024-03-30T04:28:05.413 INFO:tasks.workunit.client.0.smithi123.stdout:9/236: mkdir d6/d23/d56 0 2024-03-30T04:28:05.414 INFO:tasks.workunit.client.0.smithi123.stdout:5/286: mknod dd/d19/d1f/d3c/c60 0 2024-03-30T04:28:05.416 INFO:tasks.workunit.client.0.smithi123.stdout:5/287: symlink dd/d19/d5f/l61 0 2024-03-30T04:28:05.418 INFO:tasks.workunit.client.0.smithi123.stdout:5/288: dread - dd/d41/f51 zero size 2024-03-30T04:28:05.422 INFO:tasks.workunit.client.0.smithi123.stdout:5/289: mkdir dd/d19/d3e/d5b/d62 0 2024-03-30T04:28:05.425 INFO:tasks.workunit.client.0.smithi123.stdout:9/237: dread d6/d1f/f4f [0,4194304] 0 2024-03-30T04:28:05.426 INFO:tasks.workunit.client.0.smithi123.stdout:5/290: dread dd/d41/f43 [0,4194304] 0 2024-03-30T04:28:05.427 INFO:tasks.workunit.client.0.smithi123.stdout:3/328: dread d13/d24/d31/d23/d3a/f40 [0,4194304] 0 2024-03-30T04:28:05.429 INFO:tasks.workunit.client.0.smithi123.stdout:9/238: truncate f1 2304964 0 2024-03-30T04:28:05.430 INFO:tasks.workunit.client.0.smithi123.stdout:3/329: creat d13/d24/d31/f61 x:0 0 0 2024-03-30T04:28:05.432 INFO:tasks.workunit.client.0.smithi123.stdout:9/239: symlink d6/d1f/d45/l57 0 2024-03-30T04:28:05.434 INFO:tasks.workunit.client.0.smithi123.stdout:3/330: symlink d13/d24/d31/d30/l62 0 2024-03-30T04:28:05.437 INFO:tasks.workunit.client.0.smithi123.stdout:9/240: rename d6/d23/d56 to d6/d58 0 2024-03-30T04:28:05.440 INFO:tasks.workunit.client.0.smithi123.stdout:9/241: readlink d6/d1f/d45/l57 0 2024-03-30T04:28:05.440 INFO:tasks.workunit.client.0.smithi123.stdout:5/291: dwrite f0 [0,4194304] 0 2024-03-30T04:28:05.441 INFO:tasks.workunit.client.0.smithi123.stdout:9/242: dread - d6/d1f/f37 zero size 2024-03-30T04:28:05.445 INFO:tasks.workunit.client.0.smithi123.stdout:9/243: mknod d6/d49/c59 0 2024-03-30T04:28:05.448 INFO:tasks.workunit.client.0.smithi123.stdout:9/244: creat d6/d58/f5a x:0 0 0 2024-03-30T04:28:05.449 INFO:tasks.workunit.client.0.smithi123.stdout:3/331: truncate d13/d24/d31/d23/d3a/f40 4850078 0 2024-03-30T04:28:05.451 INFO:tasks.workunit.client.0.smithi123.stdout:3/332: read d13/f14 [3244166,54783] 0 2024-03-30T04:28:05.452 INFO:tasks.workunit.client.0.smithi123.stdout:9/245: creat d6/d1f/d4d/f5b x:0 0 0 2024-03-30T04:28:05.453 INFO:tasks.workunit.client.0.smithi123.stdout:9/246: stat d6/d1f/f3f 0 2024-03-30T04:28:05.453 INFO:tasks.workunit.client.0.smithi123.stdout:3/333: chown d13/c1f 7 1 2024-03-30T04:28:05.455 INFO:tasks.workunit.client.0.smithi123.stdout:9/247: creat d6/d58/f5c x:0 0 0 2024-03-30T04:28:05.456 INFO:tasks.workunit.client.0.smithi123.stdout:3/334: mkdir d13/d24/d31/d23/d63 0 2024-03-30T04:28:05.457 INFO:tasks.workunit.client.0.smithi123.stdout:5/292: dwrite dd/d19/f2b [0,4194304] 0 2024-03-30T04:28:05.460 INFO:tasks.workunit.client.0.smithi123.stdout:9/248: creat d6/d23/f5d x:0 0 0 2024-03-30T04:28:05.463 INFO:tasks.workunit.client.0.smithi123.stdout:3/335: unlink d13/d24/d39/c5f 0 2024-03-30T04:28:05.468 INFO:tasks.workunit.client.0.smithi123.stdout:5/293: rename dd/d19/f5d to dd/d19/d3e/f63 0 2024-03-30T04:28:05.470 INFO:tasks.workunit.client.0.smithi123.stdout:9/249: dread d6/f10 [4194304,4194304] 0 2024-03-30T04:28:05.473 INFO:tasks.workunit.client.0.smithi123.stdout:9/250: symlink d6/d23/l5e 0 2024-03-30T04:28:05.475 INFO:tasks.workunit.client.0.smithi123.stdout:9/251: readlink d6/d1f/l4a 0 2024-03-30T04:28:05.477 INFO:tasks.workunit.client.0.smithi123.stdout:9/252: stat d6/fd 0 2024-03-30T04:28:05.479 INFO:tasks.workunit.client.0.smithi123.stdout:9/253: readlink d6/l3c 0 2024-03-30T04:28:05.481 INFO:tasks.workunit.client.0.smithi123.stdout:9/254: chown d6/d23/c3e 0 1 2024-03-30T04:28:05.481 INFO:tasks.workunit.client.0.smithi123.stdout:3/336: dwrite d13/d24/f53 [0,4194304] 0 2024-03-30T04:28:05.482 INFO:tasks.workunit.client.0.smithi123.stdout:9/255: stat d6/d1f/c25 0 2024-03-30T04:28:05.483 INFO:tasks.workunit.client.0.smithi123.stdout:9/256: chown c3 84 1 2024-03-30T04:28:05.484 INFO:tasks.workunit.client.0.smithi123.stdout:9/257: dread - d6/f51 zero size 2024-03-30T04:28:05.485 INFO:tasks.workunit.client.0.smithi123.stdout:3/337: write d13/d24/d31/f4b [4007747,90284] 0 2024-03-30T04:28:05.491 INFO:tasks.workunit.client.0.smithi123.stdout:9/258: mknod d6/d1f/d45/c5f 0 2024-03-30T04:28:05.496 INFO:tasks.workunit.client.0.smithi123.stdout:9/259: mknod d6/c60 0 2024-03-30T04:28:05.499 INFO:tasks.workunit.client.0.smithi123.stdout:5/294: dread dd/f15 [0,4194304] 0 2024-03-30T04:28:05.501 INFO:tasks.workunit.client.0.smithi123.stdout:3/338: dread f11 [0,4194304] 0 2024-03-30T04:28:05.501 INFO:tasks.workunit.client.0.smithi123.stdout:5/295: write dd/d41/f47 [966945,27378] 0 2024-03-30T04:28:05.503 INFO:tasks.workunit.client.0.smithi123.stdout:5/296: fdatasync dd/d19/d1f/f27 0 2024-03-30T04:28:05.508 INFO:tasks.workunit.client.0.smithi123.stdout:9/260: dread f0 [0,4194304] 0 2024-03-30T04:28:05.510 INFO:tasks.workunit.client.0.smithi123.stdout:5/297: link dd/c18 dd/c64 0 2024-03-30T04:28:05.511 INFO:tasks.workunit.client.0.smithi123.stdout:9/261: symlink d6/d58/l61 0 2024-03-30T04:28:05.513 INFO:tasks.workunit.client.0.smithi123.stdout:9/262: readlink d6/l31 0 2024-03-30T04:28:05.517 INFO:tasks.workunit.client.0.smithi123.stdout:5/298: link dd/fe dd/d19/d1f/d35/d3a/d57/f65 0 2024-03-30T04:28:05.518 INFO:tasks.workunit.client.0.smithi123.stdout:3/339: dwrite fd [4194304,4194304] 0 2024-03-30T04:28:05.520 INFO:tasks.workunit.client.0.smithi123.stdout:5/299: mknod dd/d19/d3e/c66 0 2024-03-30T04:28:05.529 INFO:tasks.workunit.client.0.smithi123.stdout:5/300: creat dd/d19/d1f/f67 x:0 0 0 2024-03-30T04:28:05.531 INFO:tasks.workunit.client.0.smithi123.stdout:5/301: chown dd/d19/d1f/d3c/l52 2460 1 2024-03-30T04:28:05.531 INFO:tasks.workunit.client.0.smithi123.stdout:3/340: link d13/f1d d13/d24/d31/f64 0 2024-03-30T04:28:05.531 INFO:tasks.workunit.client.0.smithi123.stdout:9/263: dwrite d6/d23/f5d [0,4194304] 0 2024-03-30T04:28:05.535 INFO:tasks.workunit.client.0.smithi123.stdout:9/264: mknod d6/d23/c62 0 2024-03-30T04:28:05.538 INFO:tasks.workunit.client.0.smithi123.stdout:3/341: link d13/c20 d13/d24/d31/d30/c65 0 2024-03-30T04:28:05.540 INFO:tasks.workunit.client.0.smithi123.stdout:5/302: write dd/f15 [2288227,71283] 0 2024-03-30T04:28:05.541 INFO:tasks.workunit.client.0.smithi123.stdout:9/265: symlink d6/d1f/d4d/l63 0 2024-03-30T04:28:05.542 INFO:tasks.workunit.client.0.smithi123.stdout:3/342: mkdir d13/d24/d31/d23/d3a/d3c/d66 0 2024-03-30T04:28:05.544 INFO:tasks.workunit.client.0.smithi123.stdout:5/303: mkdir dd/d19/d1f/d68 0 2024-03-30T04:28:05.546 INFO:tasks.workunit.client.0.smithi123.stdout:8/264: sync 2024-03-30T04:28:05.546 INFO:tasks.workunit.client.0.smithi123.stdout:6/407: sync 2024-03-30T04:28:05.546 INFO:tasks.workunit.client.0.smithi123.stdout:7/278: sync 2024-03-30T04:28:05.546 INFO:tasks.workunit.client.0.smithi123.stdout:0/190: sync 2024-03-30T04:28:05.550 INFO:tasks.workunit.client.0.smithi123.stdout:6/408: chown d10/d20/c44 6 1 2024-03-30T04:28:05.550 INFO:tasks.workunit.client.0.smithi123.stdout:7/279: rmdir d2/dc/d49 39 2024-03-30T04:28:05.551 INFO:tasks.workunit.client.0.smithi123.stdout:8/265: symlink da/d27/l47 0 2024-03-30T04:28:05.553 INFO:tasks.workunit.client.0.smithi123.stdout:0/191: mknod d6/dd/d16/d32/c39 0 2024-03-30T04:28:05.556 INFO:tasks.workunit.client.0.smithi123.stdout:5/304: link dd/d19/d1f/d3c/f56 dd/d19/d3e/d5b/f69 0 2024-03-30T04:28:05.560 INFO:tasks.workunit.client.0.smithi123.stdout:7/280: write d2/dc/f17 [2070317,61891] 0 2024-03-30T04:28:05.561 INFO:tasks.workunit.client.0.smithi123.stdout:8/266: rename da/d27/d3c/c2b to da/dc/d14/c48 0 2024-03-30T04:28:05.562 INFO:tasks.workunit.client.0.smithi123.stdout:9/266: dread d6/d1f/f4f [0,4194304] 0 2024-03-30T04:28:05.563 INFO:tasks.workunit.client.0.smithi123.stdout:5/305: creat dd/d19/d1f/d35/d3a/d57/f6a x:0 0 0 2024-03-30T04:28:05.566 INFO:tasks.workunit.client.0.smithi123.stdout:7/281: dread - d2/da/d22/d3b/f4e zero size 2024-03-30T04:28:05.566 INFO:tasks.workunit.client.0.smithi123.stdout:3/343: dread d13/d24/d31/d23/f25 [0,4194304] 0 2024-03-30T04:28:05.569 INFO:tasks.workunit.client.0.smithi123.stdout:9/267: write d6/d1f/d4d/f5b [658284,46160] 0 2024-03-30T04:28:05.569 INFO:tasks.workunit.client.0.smithi123.stdout:5/306: write dd/d19/d1f/d35/d3a/f45 [585851,25204] 0 2024-03-30T04:28:05.573 INFO:tasks.workunit.client.0.smithi123.stdout:9/268: rmdir d6/d23 39 2024-03-30T04:28:05.575 INFO:tasks.workunit.client.0.smithi123.stdout:7/282: creat d2/da/d11/d61/f62 x:0 0 0 2024-03-30T04:28:05.579 INFO:tasks.workunit.client.0.smithi123.stdout:9/269: read d6/f26 [1992974,73244] 0 2024-03-30T04:28:05.581 INFO:tasks.workunit.client.0.smithi123.stdout:0/192: dwrite d6/dd/d15/d28/f33 [0,4194304] 0 2024-03-30T04:28:05.588 INFO:tasks.workunit.client.0.smithi123.stdout:7/283: rmdir d2 39 2024-03-30T04:28:05.588 INFO:tasks.workunit.client.0.smithi123.stdout:6/409: dwrite d10/f3a [0,4194304] 0 2024-03-30T04:28:05.588 INFO:tasks.workunit.client.0.smithi123.stdout:9/270: truncate d6/d1f/f4f 4861356 0 2024-03-30T04:28:05.592 INFO:tasks.workunit.client.0.smithi123.stdout:3/344: link d13/d24/c49 d13/c67 0 2024-03-30T04:28:05.594 INFO:tasks.workunit.client.0.smithi123.stdout:0/193: mkdir d6/dd/d31/d35/d3a 0 2024-03-30T04:28:05.596 INFO:tasks.workunit.client.0.smithi123.stdout:9/271: write d6/d1f/f4f [590962,4678] 0 2024-03-30T04:28:05.599 INFO:tasks.workunit.client.0.smithi123.stdout:6/410: creat d10/d20/d3f/f92 x:0 0 0 2024-03-30T04:28:05.600 INFO:tasks.workunit.client.0.smithi123.stdout:0/194: mknod d6/dd/d18/d1f/c3b 0 2024-03-30T04:28:05.600 INFO:tasks.workunit.client.0.smithi123.stdout:5/307: dwrite dd/d19/f2b [0,4194304] 0 2024-03-30T04:28:05.604 INFO:tasks.workunit.client.0.smithi123.stdout:9/272: truncate d6/f15 264703 0 2024-03-30T04:28:05.604 INFO:tasks.workunit.client.0.smithi123.stdout:6/411: write d10/d13/d1c/f2a [4784033,13767] 0 2024-03-30T04:28:05.605 INFO:tasks.workunit.client.0.smithi123.stdout:5/308: chown dd/f26 609 1 2024-03-30T04:28:05.607 INFO:tasks.workunit.client.0.smithi123.stdout:8/267: dread da/dc/d14/d18/d21/f45 [0,4194304] 0 2024-03-30T04:28:05.607 INFO:tasks.workunit.client.0.smithi123.stdout:3/345: creat d13/d24/d31/d23/d3a/d3c/d4e/d5d/f68 x:0 0 0 2024-03-30T04:28:05.610 INFO:tasks.workunit.client.0.smithi123.stdout:3/346: stat fe 0 2024-03-30T04:28:05.611 INFO:tasks.workunit.client.0.smithi123.stdout:6/412: write d10/d82/f84 [678626,95839] 0 2024-03-30T04:28:05.612 INFO:tasks.workunit.client.0.smithi123.stdout:5/309: truncate dd/f15 3861150 0 2024-03-30T04:28:05.613 INFO:tasks.workunit.client.0.smithi123.stdout:9/273: read d6/d23/f40 [4861876,25528] 0 2024-03-30T04:28:05.614 INFO:tasks.workunit.client.0.smithi123.stdout:8/268: read - da/dc/f42 zero size 2024-03-30T04:28:05.617 INFO:tasks.workunit.client.0.smithi123.stdout:4/338: sync 2024-03-30T04:28:05.618 INFO:tasks.workunit.client.0.smithi123.stdout:1/429: sync 2024-03-30T04:28:05.619 INFO:tasks.workunit.client.0.smithi123.stdout:8/269: stat da/d27/d3c/d2d/f2e 0 2024-03-30T04:28:05.622 INFO:tasks.workunit.client.0.smithi123.stdout:3/347: symlink d13/d24/d31/d23/d3a/d3c/l69 0 2024-03-30T04:28:05.623 INFO:tasks.workunit.client.0.smithi123.stdout:9/274: rmdir d6/d1f/d45/d4c 39 2024-03-30T04:28:05.624 INFO:tasks.workunit.client.0.smithi123.stdout:7/284: dwrite d2/da/d2d/f42 [0,4194304] 0 2024-03-30T04:28:05.628 INFO:tasks.workunit.client.0.smithi123.stdout:9/275: mknod d6/d49/c64 0 2024-03-30T04:28:05.629 INFO:tasks.workunit.client.0.smithi123.stdout:1/430: dread db/dd/d5f/f87 [0,4194304] 0 2024-03-30T04:28:05.630 INFO:tasks.workunit.client.0.smithi123.stdout:8/270: rename f5 to da/dc/d14/d18/f49 0 2024-03-30T04:28:05.631 INFO:tasks.workunit.client.0.smithi123.stdout:7/285: truncate d2/dc/d1c/f1d 9429736 0 2024-03-30T04:28:05.634 INFO:tasks.workunit.client.0.smithi123.stdout:0/195: getdents d6/dd/d18/d1f 0 2024-03-30T04:28:05.637 INFO:tasks.workunit.client.0.smithi123.stdout:9/276: creat d6/d49/f65 x:0 0 0 2024-03-30T04:28:05.639 INFO:tasks.workunit.client.0.smithi123.stdout:6/413: dwrite d10/f68 [0,4194304] 0 2024-03-30T04:28:05.645 INFO:tasks.workunit.client.0.smithi123.stdout:8/271: rename da/dc/l33 to da/d27/d3c/d2d/l4a 0 2024-03-30T04:28:05.648 INFO:tasks.workunit.client.0.smithi123.stdout:3/348: getdents d13/d24/d31 0 2024-03-30T04:28:05.649 INFO:tasks.workunit.client.0.smithi123.stdout:1/431: mkdir db/dd/d5f/d6d/d85/d89/d90 0 2024-03-30T04:28:05.651 INFO:tasks.workunit.client.0.smithi123.stdout:0/196: symlink d6/dd/d31/d35/d3a/l3c 0 2024-03-30T04:28:05.652 INFO:tasks.workunit.client.0.smithi123.stdout:5/310: write dd/f1d [3713974,64142] 0 2024-03-30T04:28:05.657 INFO:tasks.workunit.client.0.smithi123.stdout:5/311: dread - dd/d19/f22 zero size 2024-03-30T04:28:05.658 INFO:tasks.workunit.client.0.smithi123.stdout:9/277: rename d6/d23/f2a to d6/f66 0 2024-03-30T04:28:05.664 INFO:tasks.workunit.client.0.smithi123.stdout:9/278: chown d6/f51 1004 1 2024-03-30T04:28:05.666 INFO:tasks.workunit.client.0.smithi123.stdout:1/432: creat db/dd/d42/d8f/d78/f91 x:0 0 0 2024-03-30T04:28:05.674 INFO:tasks.workunit.client.0.smithi123.stdout:6/414: dwrite d10/d13/d1c/d5b/f85 [0,4194304] 0 2024-03-30T04:28:05.674 INFO:tasks.workunit.client.0.smithi123.stdout:1/433: chown db/dd/d42/d46/d56/f70 672 1 2024-03-30T04:28:05.674 INFO:tasks.workunit.client.0.smithi123.stdout:5/312: creat dd/d19/d1f/d68/f6b x:0 0 0 2024-03-30T04:28:05.674 INFO:tasks.workunit.client.0.smithi123.stdout:8/272: dread da/dc/d14/d18/d21/f45 [0,4194304] 0 2024-03-30T04:28:05.674 INFO:tasks.workunit.client.0.smithi123.stdout:9/279: mkdir d6/d49/d67 0 2024-03-30T04:28:05.675 INFO:tasks.workunit.client.0.smithi123.stdout:6/415: dread - fe zero size 2024-03-30T04:28:05.678 INFO:tasks.workunit.client.0.smithi123.stdout:9/280: mknod d6/d1f/c68 0 2024-03-30T04:28:05.680 INFO:tasks.workunit.client.0.smithi123.stdout:6/416: chown d10/d20/f40 713 1 2024-03-30T04:28:05.683 INFO:tasks.workunit.client.0.smithi123.stdout:6/417: read d10/d20/f7f [2282257,93220] 0 2024-03-30T04:28:05.683 INFO:tasks.workunit.client.0.smithi123.stdout:5/313: creat dd/d19/d1f/d35/d3a/f6c x:0 0 0 2024-03-30T04:28:05.689 INFO:tasks.workunit.client.0.smithi123.stdout:0/197: truncate d6/fe 236541 0 2024-03-30T04:28:05.691 INFO:tasks.workunit.client.0.smithi123.stdout:5/314: symlink dd/d19/l6d 0 2024-03-30T04:28:05.692 INFO:tasks.workunit.client.0.smithi123.stdout:6/418: symlink d10/d13/d1c/d5b/l93 0 2024-03-30T04:28:05.693 INFO:tasks.workunit.client.0.smithi123.stdout:9/281: mkdir d6/d49/d67/d69 0 2024-03-30T04:28:05.693 INFO:tasks.workunit.client.0.smithi123.stdout:0/198: chown d6/dd/d16/l2e 172078345 1 2024-03-30T04:28:05.694 INFO:tasks.workunit.client.0.smithi123.stdout:3/349: dread d13/f1d [0,4194304] 0 2024-03-30T04:28:05.697 INFO:tasks.workunit.client.0.smithi123.stdout:6/419: read d10/d13/f25 [928613,4201] 0 2024-03-30T04:28:05.697 INFO:tasks.workunit.client.0.smithi123.stdout:5/315: write dd/f1c [714722,5888] 0 2024-03-30T04:28:05.698 INFO:tasks.workunit.client.0.smithi123.stdout:0/199: dread - d6/dd/d16/f1e zero size 2024-03-30T04:28:05.699 INFO:tasks.workunit.client.0.smithi123.stdout:9/282: creat d6/d58/f6a x:0 0 0 2024-03-30T04:28:05.702 INFO:tasks.workunit.client.0.smithi123.stdout:6/420: readlink d10/d82/l57 0 2024-03-30T04:28:05.703 INFO:tasks.workunit.client.0.smithi123.stdout:5/316: fdatasync dd/d19/f2b 0 2024-03-30T04:28:05.707 INFO:tasks.workunit.client.0.smithi123.stdout:0/200: creat d6/dd/d15/d28/f3d x:0 0 0 2024-03-30T04:28:05.707 INFO:tasks.workunit.client.0.smithi123.stdout:9/283: truncate f1 1594728 0 2024-03-30T04:28:05.712 INFO:tasks.workunit.client.0.smithi123.stdout:6/421: write d10/d13/f25 [3933709,76098] 0 2024-03-30T04:28:05.712 INFO:tasks.workunit.client.0.smithi123.stdout:5/317: fdatasync dd/d19/d1f/d35/d3a/f45 0 2024-03-30T04:28:05.714 INFO:tasks.workunit.client.0.smithi123.stdout:8/273: dwrite da/dc/d14/d18/d21/f28 [0,4194304] 0 2024-03-30T04:28:05.716 INFO:tasks.workunit.client.0.smithi123.stdout:3/350: truncate d13/d24/d31/d23/d3a/d3c/d4e/d5d/f68 961508 0 2024-03-30T04:28:05.717 INFO:tasks.workunit.client.0.smithi123.stdout:9/284: symlink d6/d49/l6b 0 2024-03-30T04:28:05.720 INFO:tasks.workunit.client.0.smithi123.stdout:9/285: dread - d6/d58/f5c zero size 2024-03-30T04:28:05.720 INFO:tasks.workunit.client.0.smithi123.stdout:1/434: dwrite db/dd/d42/f68 [0,4194304] 0 2024-03-30T04:28:05.722 INFO:tasks.workunit.client.0.smithi123.stdout:5/318: symlink dd/d41/l6e 0 2024-03-30T04:28:05.724 INFO:tasks.workunit.client.0.smithi123.stdout:5/319: readlink dd/d19/d1f/l50 0 2024-03-30T04:28:05.725 INFO:tasks.workunit.client.0.smithi123.stdout:6/422: mknod d10/d13/c94 0 2024-03-30T04:28:05.728 INFO:tasks.workunit.client.0.smithi123.stdout:8/274: creat da/d27/d3c/d2d/f4b x:0 0 0 2024-03-30T04:28:05.732 INFO:tasks.workunit.client.0.smithi123.stdout:9/286: write d6/d23/f40 [4241493,27619] 0 2024-03-30T04:28:05.734 INFO:tasks.workunit.client.0.smithi123.stdout:1/435: creat db/dd/d42/d46/d39/f92 x:0 0 0 2024-03-30T04:28:05.737 INFO:tasks.workunit.client.0.smithi123.stdout:1/436: chown db/dd/f54 12681907 1 2024-03-30T04:28:05.741 INFO:tasks.workunit.client.0.smithi123.stdout:5/320: rmdir dd/d19/d1f/d68 39 2024-03-30T04:28:05.741 INFO:tasks.workunit.client.0.smithi123.stdout:3/351: symlink d13/d24/d31/d23/d3a/d3c/d66/l6a 0 2024-03-30T04:28:05.743 INFO:tasks.workunit.client.0.smithi123.stdout:0/201: truncate f5 3028635 0 2024-03-30T04:28:05.746 INFO:tasks.workunit.client.0.smithi123.stdout:5/321: stat dd/l1b 0 2024-03-30T04:28:05.747 INFO:tasks.workunit.client.0.smithi123.stdout:0/202: write d6/f24 [1885639,109430] 0 2024-03-30T04:28:05.751 INFO:tasks.workunit.client.0.smithi123.stdout:6/423: creat d10/d13/f95 x:0 0 0 2024-03-30T04:28:05.760 INFO:tasks.workunit.client.0.smithi123.stdout:0/203: rename d6/dd/d18/l30 to d6/dd/d16/l3e 0 2024-03-30T04:28:05.761 INFO:tasks.workunit.client.0.smithi123.stdout:9/287: dwrite d6/d1f/f37 [0,4194304] 0 2024-03-30T04:28:05.763 INFO:tasks.workunit.client.0.smithi123.stdout:1/437: dwrite db/dd/d42/d46/d56/d5c/f64 [0,4194304] 0 2024-03-30T04:28:05.767 INFO:tasks.workunit.client.0.smithi123.stdout:6/424: creat d10/f96 x:0 0 0 2024-03-30T04:28:05.770 INFO:tasks.workunit.client.0.smithi123.stdout:6/425: read d10/d82/f8a [1736775,100054] 0 2024-03-30T04:28:05.773 INFO:tasks.workunit.client.0.smithi123.stdout:8/275: dread da/f11 [0,4194304] 0 2024-03-30T04:28:05.775 INFO:tasks.workunit.client.0.smithi123.stdout:5/322: dwrite dd/d19/d1f/d35/f40 [0,4194304] 0 2024-03-30T04:28:05.779 INFO:tasks.workunit.client.0.smithi123.stdout:3/352: dread fd [0,4194304] 0 2024-03-30T04:28:05.779 INFO:tasks.workunit.client.0.smithi123.stdout:0/204: dread d6/fe [0,4194304] 0 2024-03-30T04:28:05.782 INFO:tasks.workunit.client.0.smithi123.stdout:9/288: chown d6/d1f/d45/d4c/f54 324742 1 2024-03-30T04:28:05.782 INFO:tasks.workunit.client.0.smithi123.stdout:3/353: stat d13/d24/d31/d23/d3a/d3c/d4e/c5c 0 2024-03-30T04:28:05.782 INFO:tasks.workunit.client.0.smithi123.stdout:8/276: rename da/dc/f1a to da/d27/d3c/d2d/f4c 0 2024-03-30T04:28:05.783 INFO:tasks.workunit.client.0.smithi123.stdout:0/205: write d6/dd/d15/f29 [865603,69290] 0 2024-03-30T04:28:05.784 INFO:tasks.workunit.client.0.smithi123.stdout:9/289: dread - d6/d49/f65 zero size 2024-03-30T04:28:05.788 INFO:tasks.workunit.client.0.smithi123.stdout:0/206: mknod d6/dd/d16/c3f 0 2024-03-30T04:28:05.792 INFO:tasks.workunit.client.0.smithi123.stdout:6/426: creat d10/f97 x:0 0 0 2024-03-30T04:28:05.793 INFO:tasks.workunit.client.0.smithi123.stdout:0/207: rename d6/fe to d6/f40 0 2024-03-30T04:28:05.794 INFO:tasks.workunit.client.0.smithi123.stdout:9/290: creat d6/d49/d67/d69/f6c x:0 0 0 2024-03-30T04:28:05.796 INFO:tasks.workunit.client.0.smithi123.stdout:5/323: rename dd/d19/d3e/c66 to dd/c6f 0 2024-03-30T04:28:05.802 INFO:tasks.workunit.client.0.smithi123.stdout:0/208: write d6/dd/d15/d28/f33 [909329,9185] 0 2024-03-30T04:28:05.802 INFO:tasks.workunit.client.0.smithi123.stdout:6/427: rmdir d10/d13/d1c/d5b 39 2024-03-30T04:28:05.802 INFO:tasks.workunit.client.0.smithi123.stdout:3/354: getdents d13/d24/d31/d23/d3a/d3c/d4e 0 2024-03-30T04:28:05.803 INFO:tasks.workunit.client.0.smithi123.stdout:9/291: creat d6/d49/f6d x:0 0 0 2024-03-30T04:28:05.805 INFO:tasks.workunit.client.0.smithi123.stdout:1/438: dwrite db/dd/d42/d46/d56/d5c/f64 [0,4194304] 0 2024-03-30T04:28:05.806 INFO:tasks.workunit.client.0.smithi123.stdout:5/324: mknod dd/d19/d3e/d5b/c70 0 2024-03-30T04:28:05.808 INFO:tasks.workunit.client.0.smithi123.stdout:0/209: symlink d6/l41 0 2024-03-30T04:28:05.809 INFO:tasks.workunit.client.0.smithi123.stdout:6/428: write d10/d13/d1c/d42/f51 [272352,91718] 0 2024-03-30T04:28:05.811 INFO:tasks.workunit.client.0.smithi123.stdout:3/355: mknod d13/d24/d31/d23/c6b 0 2024-03-30T04:28:05.816 INFO:tasks.workunit.client.0.smithi123.stdout:2/297: sync 2024-03-30T04:28:05.818 INFO:tasks.workunit.client.0.smithi123.stdout:9/292: rename d6/l13 to d6/l6e 0 2024-03-30T04:28:05.819 INFO:tasks.workunit.client.0.smithi123.stdout:6/429: chown d10/d13/d1c/c4b 170 1 2024-03-30T04:28:05.820 INFO:tasks.workunit.client.0.smithi123.stdout:1/439: dread db/f59 [0,4194304] 0 2024-03-30T04:28:05.822 INFO:tasks.workunit.client.0.smithi123.stdout:8/277: dwrite da/dc/d14/d18/f30 [4194304,4194304] 0 2024-03-30T04:28:05.826 INFO:tasks.workunit.client.0.smithi123.stdout:5/325: creat dd/d19/d3e/d5b/d62/f71 x:0 0 0 2024-03-30T04:28:05.830 INFO:tasks.workunit.client.0.smithi123.stdout:3/356: dread d13/d24/d31/d23/d3a/d3c/d4e/d5d/f68 [0,4194304] 0 2024-03-30T04:28:05.832 INFO:tasks.workunit.client.0.smithi123.stdout:6/430: dread d10/f26 [0,4194304] 0 2024-03-30T04:28:05.833 INFO:tasks.workunit.client.0.smithi123.stdout:8/278: rmdir da/dc 39 2024-03-30T04:28:05.833 INFO:tasks.workunit.client.0.smithi123.stdout:0/210: link d6/dd/d15/d28/f33 d6/dd/d31/d35/f42 0 2024-03-30T04:28:05.837 INFO:tasks.workunit.client.0.smithi123.stdout:5/326: chown dd/lf 215211 1 2024-03-30T04:28:05.840 INFO:tasks.workunit.client.0.smithi123.stdout:2/298: creat d0/d1f/d1c/f5b x:0 0 0 2024-03-30T04:28:05.842 INFO:tasks.workunit.client.0.smithi123.stdout:6/431: symlink d10/d82/d8e/l98 0 2024-03-30T04:28:05.847 INFO:tasks.workunit.client.0.smithi123.stdout:0/211: creat d6/dd/d31/d35/f43 x:0 0 0 2024-03-30T04:28:05.851 INFO:tasks.workunit.client.0.smithi123.stdout:6/432: mkdir d10/d13/d6a/d7a/d99 0 2024-03-30T04:28:05.853 INFO:tasks.workunit.client.0.smithi123.stdout:5/327: rmdir dd/d19/d1f/d3c 39 2024-03-30T04:28:05.855 INFO:tasks.workunit.client.0.smithi123.stdout:0/212: chown d6/dd/d31/d35/d3a 7425 1 2024-03-30T04:28:05.857 INFO:tasks.workunit.client.0.smithi123.stdout:2/299: write d0/d1f/fa [261288,82988] 0 2024-03-30T04:28:05.859 INFO:tasks.workunit.client.0.smithi123.stdout:1/440: dwrite db/dd/d42/d46/f79 [0,4194304] 0 2024-03-30T04:28:05.863 INFO:tasks.workunit.client.0.smithi123.stdout:0/213: symlink d6/l44 0 2024-03-30T04:28:05.865 INFO:tasks.workunit.client.0.smithi123.stdout:6/433: mkdir d10/d82/d9a 0 2024-03-30T04:28:05.869 INFO:tasks.workunit.client.0.smithi123.stdout:9/293: dwrite d6/f10 [0,4194304] 0 2024-03-30T04:28:05.869 INFO:tasks.workunit.client.0.smithi123.stdout:5/328: unlink dd/d19/d1f/d35/d3a/f6c 0 2024-03-30T04:28:05.869 INFO:tasks.workunit.client.0.smithi123.stdout:6/434: dread - fe zero size 2024-03-30T04:28:05.879 INFO:tasks.workunit.client.0.smithi123.stdout:0/214: write d6/dd/d31/d35/f42 [586332,19799] 0 2024-03-30T04:28:05.880 INFO:tasks.workunit.client.0.smithi123.stdout:3/357: dwrite d13/f14 [0,4194304] 0 2024-03-30T04:28:05.882 INFO:tasks.workunit.client.0.smithi123.stdout:8/279: dwrite da/d27/d3c/d2d/f2e [0,4194304] 0 2024-03-30T04:28:05.888 INFO:tasks.workunit.client.0.smithi123.stdout:3/358: readlink d13/d24/d31/d23/d3a/d3c/d66/l6a 0 2024-03-30T04:28:05.890 INFO:tasks.workunit.client.0.smithi123.stdout:2/300: link d0/d1f/f1b d0/d1f/d1c/d39/d44/f5c 0 2024-03-30T04:28:05.890 INFO:tasks.workunit.client.0.smithi123.stdout:8/280: fsync da/d27/f3b 0 2024-03-30T04:28:05.890 INFO:tasks.workunit.client.0.smithi123.stdout:0/215: creat d6/dd/d31/d35/f45 x:0 0 0 2024-03-30T04:28:05.897 INFO:tasks.workunit.client.0.smithi123.stdout:6/435: dread f6 [0,4194304] 0 2024-03-30T04:28:05.898 INFO:tasks.workunit.client.0.smithi123.stdout:0/216: chown d6/dd/d18/d1f/c2a 61183769 1 2024-03-30T04:28:05.902 INFO:tasks.workunit.client.0.smithi123.stdout:2/301: chown d0/d37/c4e 0 1 2024-03-30T04:28:05.907 INFO:tasks.workunit.client.0.smithi123.stdout:1/441: dwrite db/dd/d42/d46/d56/d5c/f74 [0,4194304] 0 2024-03-30T04:28:05.908 INFO:tasks.workunit.client.0.smithi123.stdout:6/436: fdatasync d10/d13/d1c/d42/f5f 0 2024-03-30T04:28:05.909 INFO:tasks.workunit.client.0.smithi123.stdout:9/294: dwrite d6/d58/f6a [0,4194304] 0 2024-03-30T04:28:05.913 INFO:tasks.workunit.client.0.smithi123.stdout:2/302: creat d0/d1f/d1c/d20/f5d x:0 0 0 2024-03-30T04:28:05.920 INFO:tasks.workunit.client.0.smithi123.stdout:2/303: unlink d0/d1f/d16/f3d 0 2024-03-30T04:28:05.922 INFO:tasks.workunit.client.0.smithi123.stdout:3/359: dread d13/f1d [0,4194304] 0 2024-03-30T04:28:05.926 INFO:tasks.workunit.client.0.smithi123.stdout:9/295: mknod d6/d1f/d4d/c6f 0 2024-03-30T04:28:05.926 INFO:tasks.workunit.client.0.smithi123.stdout:2/304: readlink d0/d1f/d1c/d36/l58 0 2024-03-30T04:28:05.928 INFO:tasks.workunit.client.0.smithi123.stdout:2/305: chown d0/d37/d57 37999161 1 2024-03-30T04:28:05.929 INFO:tasks.workunit.client.0.smithi123.stdout:0/217: getdents d6/dd/d15 0 2024-03-30T04:28:05.930 INFO:tasks.workunit.client.0.smithi123.stdout:9/296: chown d6/d1f/f2d 3858359 1 2024-03-30T04:28:05.934 INFO:tasks.workunit.client.0.smithi123.stdout:6/437: dwrite d10/d13/d1c/d42/f51 [0,4194304] 0 2024-03-30T04:28:05.936 INFO:tasks.workunit.client.0.smithi123.stdout:9/297: write d6/d1f/d45/d4c/f4e [46522,128216] 0 2024-03-30T04:28:05.936 INFO:tasks.workunit.client.0.smithi123.stdout:4/339: sync 2024-03-30T04:28:05.936 INFO:tasks.workunit.client.0.smithi123.stdout:0/218: symlink d6/dd/d15/l46 0 2024-03-30T04:28:05.938 INFO:tasks.workunit.client.0.smithi123.stdout:6/438: dread - d10/d13/d1c/f6c zero size 2024-03-30T04:28:05.941 INFO:tasks.workunit.client.0.smithi123.stdout:2/306: rename d0/d1f/d1c/d2a/d4a to d0/d1f/d5e 0 2024-03-30T04:28:05.942 INFO:tasks.workunit.client.0.smithi123.stdout:1/442: dread db/f59 [0,4194304] 0 2024-03-30T04:28:05.942 INFO:tasks.workunit.client.0.smithi123.stdout:6/439: stat d10/f27 0 2024-03-30T04:28:05.943 INFO:tasks.workunit.client.0.smithi123.stdout:4/340: fsync d2/d34/f77 0 2024-03-30T04:28:05.945 INFO:tasks.workunit.client.0.smithi123.stdout:0/219: dread d6/f40 [0,4194304] 0 2024-03-30T04:28:05.945 INFO:tasks.workunit.client.0.smithi123.stdout:5/329: dread f9 [0,4194304] 0 2024-03-30T04:28:05.946 INFO:tasks.workunit.client.0.smithi123.stdout:9/298: rename d6/f51 to d6/d1f/f70 0 2024-03-30T04:28:05.950 INFO:tasks.workunit.client.0.smithi123.stdout:1/443: creat db/dd/d42/d46/d39/f93 x:0 0 0 2024-03-30T04:28:05.952 INFO:tasks.workunit.client.0.smithi123.stdout:6/440: symlink d10/d13/d6a/d7a/l9b 0 2024-03-30T04:28:05.952 INFO:tasks.workunit.client.0.smithi123.stdout:2/307: mkdir d0/d1f/d1c/d39/d4b/d4c/d5f 0 2024-03-30T04:28:05.957 INFO:tasks.workunit.client.0.smithi123.stdout:5/330: mkdir dd/d19/d1f/d3c/d72 0 2024-03-30T04:28:05.958 INFO:tasks.workunit.client.0.smithi123.stdout:7/286: sync 2024-03-30T04:28:05.958 INFO:tasks.workunit.client.0.smithi123.stdout:2/308: mknod d0/d1f/d16/c60 0 2024-03-30T04:28:05.959 INFO:tasks.workunit.client.0.smithi123.stdout:5/331: dread - dd/d19/d1f/d3c/f56 zero size 2024-03-30T04:28:05.959 INFO:tasks.workunit.client.0.smithi123.stdout:6/441: chown d10/d13/d1c/d5b/f89 225 1 2024-03-30T04:28:05.963 INFO:tasks.workunit.client.0.smithi123.stdout:6/442: chown d10/d13/d1c/d5b/f85 9 1 2024-03-30T04:28:05.964 INFO:tasks.workunit.client.0.smithi123.stdout:2/309: mknod d0/d1f/d1c/d20/c61 0 2024-03-30T04:28:05.968 INFO:tasks.workunit.client.0.smithi123.stdout:7/287: rename d2/dc/d1c/f47 to d2/dc/d49/f63 0 2024-03-30T04:28:05.973 INFO:tasks.workunit.client.0.smithi123.stdout:7/288: mknod d2/d4d/c64 0 2024-03-30T04:28:05.974 INFO:tasks.workunit.client.0.smithi123.stdout:2/310: creat d0/d37/d57/f62 x:0 0 0 2024-03-30T04:28:05.975 INFO:tasks.workunit.client.0.smithi123.stdout:1/444: dwrite db/dd/d42/d46/d56/d5c/f64 [4194304,4194304] 0 2024-03-30T04:28:05.976 INFO:tasks.workunit.client.0.smithi123.stdout:7/289: fdatasync d2/dc/d1c/f1d 0 2024-03-30T04:28:05.978 INFO:tasks.workunit.client.0.smithi123.stdout:2/311: mknod d0/d1f/d16/c63 0 2024-03-30T04:28:05.980 INFO:tasks.workunit.client.0.smithi123.stdout:1/445: stat db/dd/d42/d46/f29 0 2024-03-30T04:28:05.980 INFO:tasks.workunit.client.0.smithi123.stdout:7/290: fdatasync d2/dc/f17 0 2024-03-30T04:28:05.986 INFO:tasks.workunit.client.0.smithi123.stdout:6/443: dwrite d10/d20/f40 [0,4194304] 0 2024-03-30T04:28:05.987 INFO:tasks.workunit.client.0.smithi123.stdout:4/341: dread d2/d4/fa [0,4194304] 0 2024-03-30T04:28:05.989 INFO:tasks.workunit.client.0.smithi123.stdout:4/342: chown d2/d34/f3f 14 1 2024-03-30T04:28:05.989 INFO:tasks.workunit.client.0.smithi123.stdout:1/446: mkdir db/dd/d5f/d6d/d85/d89/d90/d94 0 2024-03-30T04:28:05.991 INFO:tasks.workunit.client.0.smithi123.stdout:6/444: creat d10/d13/d45/f9c x:0 0 0 2024-03-30T04:28:05.993 INFO:tasks.workunit.client.0.smithi123.stdout:4/343: truncate d2/d4/d7/d1a/f5e 569542 0 2024-03-30T04:28:05.993 INFO:tasks.workunit.client.0.smithi123.stdout:6/445: write d10/d13/d45/f67 [919677,117073] 0 2024-03-30T04:28:05.997 INFO:tasks.workunit.client.0.smithi123.stdout:6/446: truncate d10/d13/d45/f9c 600070 0 2024-03-30T04:28:05.998 INFO:tasks.workunit.client.0.smithi123.stdout:7/291: dread d2/da/d2d/f51 [0,4194304] 0 2024-03-30T04:28:05.999 INFO:tasks.workunit.client.0.smithi123.stdout:6/447: chown d10/d13/d6a/d7a/c7e 0 1 2024-03-30T04:28:06.000 INFO:tasks.workunit.client.0.smithi123.stdout:4/344: rename d2/d34/l66 to d2/d4/d2c/l7d 0 2024-03-30T04:28:06.002 INFO:tasks.workunit.client.0.smithi123.stdout:7/292: fsync d2/f8 0 2024-03-30T04:28:06.002 INFO:tasks.workunit.client.0.smithi123.stdout:6/448: write d10/d13/d1c/f2c [1343728,17439] 0 2024-03-30T04:28:06.004 INFO:tasks.workunit.client.0.smithi123.stdout:4/345: write d2/d34/f3f [2993659,55319] 0 2024-03-30T04:28:06.008 INFO:tasks.workunit.client.0.smithi123.stdout:6/449: getdents d10/d82/d8e 0 2024-03-30T04:28:06.009 INFO:tasks.workunit.client.0.smithi123.stdout:4/346: mknod d2/d4/d7/d1a/c7e 0 2024-03-30T04:28:06.011 INFO:tasks.workunit.client.0.smithi123.stdout:6/450: dread d10/d20/f2b [0,4194304] 0 2024-03-30T04:28:06.012 INFO:tasks.workunit.client.0.smithi123.stdout:6/451: fdatasync d10/d13/d1c/f1e 0 2024-03-30T04:28:06.015 INFO:tasks.workunit.client.0.smithi123.stdout:7/293: dread d2/da/d2d/f51 [0,4194304] 0 2024-03-30T04:28:06.017 INFO:tasks.workunit.client.0.smithi123.stdout:7/294: write d2/da/d11/d61/f62 [578839,65627] 0 2024-03-30T04:28:06.018 INFO:tasks.workunit.client.0.smithi123.stdout:7/295: dread - d2/da/d2d/f3d zero size 2024-03-30T04:28:06.020 INFO:tasks.workunit.client.0.smithi123.stdout:7/296: chown d2/da/d2d/f3d 0 1 2024-03-30T04:28:06.025 INFO:tasks.workunit.client.0.smithi123.stdout:6/452: dwrite d10/f48 [0,4194304] 0 2024-03-30T04:28:06.029 INFO:tasks.workunit.client.0.smithi123.stdout:6/453: link d10/d13/d1c/d5b/c8f d10/d13/d6a/d7a/c9d 0 2024-03-30T04:28:06.240 INFO:tasks.workunit.client.0.smithi123.stdout:8/281: dread da/dc/d14/f36 [0,4194304] 0 2024-03-30T04:28:06.243 INFO:tasks.workunit.client.0.smithi123.stdout:8/282: creat da/dc/d14/d18/d21/f4d x:0 0 0 2024-03-30T04:28:06.245 INFO:tasks.workunit.client.0.smithi123.stdout:1/447: sync 2024-03-30T04:28:06.245 INFO:tasks.workunit.client.0.smithi123.stdout:4/347: sync 2024-03-30T04:28:06.254 INFO:tasks.workunit.client.0.smithi123.stdout:8/283: dread da/ff [0,4194304] 0 2024-03-30T04:28:06.263 INFO:tasks.workunit.client.0.smithi123.stdout:1/448: dwrite db/dd/d42/d46/d39/f57 [0,4194304] 0 2024-03-30T04:28:06.267 INFO:tasks.workunit.client.0.smithi123.stdout:1/449: symlink db/dd/d42/d46/l95 0 2024-03-30T04:28:06.281 INFO:tasks.workunit.client.0.smithi123.stdout:4/348: dread d2/d4/d7/fb [0,4194304] 0 2024-03-30T04:28:06.294 INFO:tasks.workunit.client.0.smithi123.stdout:9/299: truncate d6/d1f/f34 1128389 0 2024-03-30T04:28:06.297 INFO:tasks.workunit.client.0.smithi123.stdout:9/300: creat d6/d1f/f71 x:0 0 0 2024-03-30T04:28:06.297 INFO:tasks.workunit.client.0.smithi123.stdout:5/332: write dd/d19/d1f/d35/d3a/d57/f65 [5693128,74413] 0 2024-03-30T04:28:06.302 INFO:tasks.workunit.client.0.smithi123.stdout:0/220: dwrite d6/fa [0,4194304] 0 2024-03-30T04:28:06.302 INFO:tasks.workunit.client.0.smithi123.stdout:5/333: dread - dd/f4e zero size 2024-03-30T04:28:06.302 INFO:tasks.workunit.client.0.smithi123.stdout:9/301: mkdir d6/d1f/d72 0 2024-03-30T04:28:06.304 INFO:tasks.workunit.client.0.smithi123.stdout:9/302: chown d6/d1f/c68 559 1 2024-03-30T04:28:06.304 INFO:tasks.workunit.client.0.smithi123.stdout:9/303: stat d6/d1f/d45/l57 0 2024-03-30T04:28:06.305 INFO:tasks.workunit.client.0.smithi123.stdout:7/297: write d2/f8 [535580,903] 0 2024-03-30T04:28:06.306 INFO:tasks.workunit.client.0.smithi123.stdout:0/221: creat d6/dd/d18/f47 x:0 0 0 2024-03-30T04:28:06.309 INFO:tasks.workunit.client.0.smithi123.stdout:4/349: rmdir d2 39 2024-03-30T04:28:06.318 INFO:tasks.workunit.client.0.smithi123.stdout:2/312: dread d0/d1f/d16/f26 [0,4194304] 0 2024-03-30T04:28:06.318 INFO:tasks.workunit.client.0.smithi123.stdout:5/334: link dd/d41/f47 dd/d19/d1f/d68/f73 0 2024-03-30T04:28:06.320 INFO:tasks.workunit.client.0.smithi123.stdout:0/222: mknod d6/dd/d18/d2f/c48 0 2024-03-30T04:28:06.320 INFO:tasks.workunit.client.0.smithi123.stdout:7/298: dread d2/da/d11/f19 [0,4194304] 0 2024-03-30T04:28:06.320 INFO:tasks.workunit.client.0.smithi123.stdout:5/335: chown fb 133634269 1 2024-03-30T04:28:06.325 INFO:tasks.workunit.client.0.smithi123.stdout:3/360: dread d13/f1b [4194304,4194304] 0 2024-03-30T04:28:06.326 INFO:tasks.workunit.client.0.smithi123.stdout:8/284: write da/dc/d14/d18/d21/f40 [311876,80023] 0 2024-03-30T04:28:06.332 INFO:tasks.workunit.client.0.smithi123.stdout:3/361: dread - d13/d24/d31/f61 zero size 2024-03-30T04:28:06.333 INFO:tasks.workunit.client.0.smithi123.stdout:2/313: truncate d0/d1f/d1c/d20/f5d 137072 0 2024-03-30T04:28:06.336 INFO:tasks.workunit.client.0.smithi123.stdout:9/304: dwrite d6/fd [0,4194304] 0 2024-03-30T04:28:06.339 INFO:tasks.workunit.client.0.smithi123.stdout:4/350: truncate d2/d4/d7/fb 1329709 0 2024-03-30T04:28:06.339 INFO:tasks.workunit.client.0.smithi123.stdout:5/336: write dd/f32 [4392310,24743] 0 2024-03-30T04:28:06.344 INFO:tasks.workunit.client.0.smithi123.stdout:5/337: chown dd/d19/f59 15718 1 2024-03-30T04:28:06.344 INFO:tasks.workunit.client.0.smithi123.stdout:0/223: truncate d6/f12 262494 0 2024-03-30T04:28:06.347 INFO:tasks.workunit.client.0.smithi123.stdout:9/305: chown c3 2 1 2024-03-30T04:28:06.349 INFO:tasks.workunit.client.0.smithi123.stdout:4/351: fsync d2/d4/d7/d1a/f1f 0 2024-03-30T04:28:06.349 INFO:tasks.workunit.client.0.smithi123.stdout:6/454: dwrite d10/f26 [0,4194304] 0 2024-03-30T04:28:06.349 INFO:tasks.workunit.client.0.smithi123.stdout:3/362: creat d13/d24/d31/d23/f6c x:0 0 0 2024-03-30T04:28:06.350 INFO:tasks.workunit.client.0.smithi123.stdout:2/314: unlink d0/d1f/d1c/d20/l3a 0 2024-03-30T04:28:06.357 INFO:tasks.workunit.client.0.smithi123.stdout:0/224: symlink d6/dd/d18/l49 0 2024-03-30T04:28:06.359 INFO:tasks.workunit.client.0.smithi123.stdout:9/306: unlink d6/d1f/f28 0 2024-03-30T04:28:06.361 INFO:tasks.workunit.client.0.smithi123.stdout:4/352: chown d2/d4/d7/l31 0 1 2024-03-30T04:28:06.363 INFO:tasks.workunit.client.0.smithi123.stdout:5/338: write dd/d19/d1f/d35/d3a/d57/f65 [3465954,48138] 0 2024-03-30T04:28:06.364 INFO:tasks.workunit.client.0.smithi123.stdout:3/363: symlink d13/d24/d31/d23/d3a/d3c/d4e/l6d 0 2024-03-30T04:28:06.367 INFO:tasks.workunit.client.0.smithi123.stdout:6/455: dread d10/d13/d1c/f5d [0,4194304] 0 2024-03-30T04:28:06.368 INFO:tasks.workunit.client.0.smithi123.stdout:2/315: creat d0/d1f/d1c/d39/d4b/d4c/f64 x:0 0 0 2024-03-30T04:28:06.368 INFO:tasks.workunit.client.0.smithi123.stdout:8/285: dwrite da/d27/f3b [0,4194304] 0 2024-03-30T04:28:06.368 INFO:tasks.workunit.client.0.smithi123.stdout:4/353: write d2/d4/d2c/d42/f6a [811819,45562] 0 2024-03-30T04:28:06.370 INFO:tasks.workunit.client.0.smithi123.stdout:7/299: dwrite d2/da/d2d/f42 [0,4194304] 0 2024-03-30T04:28:06.370 INFO:tasks.workunit.client.0.smithi123.stdout:3/364: chown d13/d24/d31/d23/c6b 132 1 2024-03-30T04:28:06.376 INFO:tasks.workunit.client.0.smithi123.stdout:6/456: readlink d10/d13/d1c/d5b/l72 0 2024-03-30T04:28:06.381 INFO:tasks.workunit.client.0.smithi123.stdout:6/457: chown d10/d82/d8e 0 1 2024-03-30T04:28:06.382 INFO:tasks.workunit.client.0.smithi123.stdout:0/225: creat d6/dd/f4a x:0 0 0 2024-03-30T04:28:06.387 INFO:tasks.workunit.client.0.smithi123.stdout:2/316: creat d0/d1f/d5e/f65 x:0 0 0 2024-03-30T04:28:06.388 INFO:tasks.workunit.client.0.smithi123.stdout:7/300: symlink d2/da/d2d/l65 0 2024-03-30T04:28:06.391 INFO:tasks.workunit.client.0.smithi123.stdout:9/307: dwrite d6/d58/f6a [4194304,4194304] 0 2024-03-30T04:28:06.391 INFO:tasks.workunit.client.0.smithi123.stdout:5/339: write dd/d19/d1f/d3c/f56 [3635,74516] 0 2024-03-30T04:28:06.396 INFO:tasks.workunit.client.0.smithi123.stdout:7/301: write d2/da/d2d/f42 [5048133,88871] 0 2024-03-30T04:28:06.396 INFO:tasks.workunit.client.0.smithi123.stdout:8/286: creat da/f4e x:0 0 0 2024-03-30T04:28:06.398 INFO:tasks.workunit.client.0.smithi123.stdout:3/365: link c5 d13/d24/d31/d23/d3a/c6e 0 2024-03-30T04:28:06.403 INFO:tasks.workunit.client.0.smithi123.stdout:8/287: chown da/d27/d3c/d2d/f2e 15922 1 2024-03-30T04:28:06.404 INFO:tasks.workunit.client.0.smithi123.stdout:6/458: write d10/d20/f7f [5444665,13261] 0 2024-03-30T04:28:06.404 INFO:tasks.workunit.client.0.smithi123.stdout:5/340: truncate dd/d19/d3e/f63 157612 0 2024-03-30T04:28:06.404 INFO:tasks.workunit.client.0.smithi123.stdout:3/366: mknod d13/d24/d31/c6f 0 2024-03-30T04:28:06.409 INFO:tasks.workunit.client.0.smithi123.stdout:8/288: dread da/ff [0,4194304] 0 2024-03-30T04:28:06.409 INFO:tasks.workunit.client.0.smithi123.stdout:4/354: dwrite d2/d4/f16 [0,4194304] 0 2024-03-30T04:28:06.411 INFO:tasks.workunit.client.0.smithi123.stdout:1/450: sync 2024-03-30T04:28:06.412 INFO:tasks.workunit.client.0.smithi123.stdout:6/459: dread d10/d20/f2b [0,4194304] 0 2024-03-30T04:28:06.413 INFO:tasks.workunit.client.0.smithi123.stdout:0/226: dread d6/f12 [0,4194304] 0 2024-03-30T04:28:06.417 INFO:tasks.workunit.client.0.smithi123.stdout:8/289: truncate da/dc/f32 117158 0 2024-03-30T04:28:06.417 INFO:tasks.workunit.client.0.smithi123.stdout:3/367: unlink d13/d24/d31/d23/d3a/d3c/d4e/c52 0 2024-03-30T04:28:06.422 INFO:tasks.workunit.client.0.smithi123.stdout:0/227: mkdir d6/dd/d15/d28/d4b 0 2024-03-30T04:28:06.423 INFO:tasks.workunit.client.0.smithi123.stdout:8/290: chown da/f3d 91374449 1 2024-03-30T04:28:06.423 INFO:tasks.workunit.client.0.smithi123.stdout:4/355: write d2/d4/d7/d1a/f5e [270814,73941] 0 2024-03-30T04:28:06.429 INFO:tasks.workunit.client.0.smithi123.stdout:9/308: dwrite d6/d23/f3b [0,4194304] 0 2024-03-30T04:28:06.430 INFO:tasks.workunit.client.0.smithi123.stdout:6/460: creat d10/f9e x:0 0 0 2024-03-30T04:28:06.433 INFO:tasks.workunit.client.0.smithi123.stdout:2/317: dwrite d0/f10 [0,4194304] 0 2024-03-30T04:28:06.438 INFO:tasks.workunit.client.0.smithi123.stdout:8/291: write da/dc/d14/f44 [486633,34668] 0 2024-03-30T04:28:06.443 INFO:tasks.workunit.client.0.smithi123.stdout:0/228: truncate d6/dd/d31/d35/f45 180062 0 2024-03-30T04:28:06.443 INFO:tasks.workunit.client.0.smithi123.stdout:9/309: creat d6/d49/f73 x:0 0 0 2024-03-30T04:28:06.444 INFO:tasks.workunit.client.0.smithi123.stdout:1/451: link db/d1b/c63 db/dd/d42/d46/d51/d72/c96 0 2024-03-30T04:28:06.444 INFO:tasks.workunit.client.0.smithi123.stdout:6/461: mkdir d10/d13/d6a/d7a/d9f 0 2024-03-30T04:28:06.446 INFO:tasks.workunit.client.0.smithi123.stdout:8/292: write da/d27/f3b [2622017,130829] 0 2024-03-30T04:28:06.457 INFO:tasks.workunit.client.0.smithi123.stdout:2/318: creat d0/d37/d57/f66 x:0 0 0 2024-03-30T04:28:06.457 INFO:tasks.workunit.client.0.smithi123.stdout:0/229: creat d6/dd/d18/d1f/f4c x:0 0 0 2024-03-30T04:28:06.458 INFO:tasks.workunit.client.0.smithi123.stdout:3/368: dread d13/f1b [4194304,4194304] 0 2024-03-30T04:28:06.460 INFO:tasks.workunit.client.0.smithi123.stdout:5/341: dwrite dd/f4e [0,4194304] 0 2024-03-30T04:28:06.465 INFO:tasks.workunit.client.0.smithi123.stdout:5/342: fsync dd/d19/f28 0 2024-03-30T04:28:06.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:06 smithi123 ceph-mon[35245]: pgmap v36: 65 pgs: 65 active+clean; 649 MiB data, 3.8 GiB used, 533 GiB / 536 GiB avail; 8.4 MiB/s rd, 76 MiB/s wr, 309 op/s 2024-03-30T04:28:06.470 INFO:tasks.workunit.client.0.smithi123.stdout:5/343: chown dd/d19/d1f/d35/f40 14450 1 2024-03-30T04:28:06.473 INFO:tasks.workunit.client.0.smithi123.stdout:4/356: dwrite d2/d4/d7/f6c [0,4194304] 0 2024-03-30T04:28:06.474 INFO:tasks.workunit.client.0.smithi123.stdout:4/357: stat d2/f45 0 2024-03-30T04:28:06.476 INFO:tasks.workunit.client.0.smithi123.stdout:9/310: dread d6/d23/f3b [0,4194304] 0 2024-03-30T04:28:06.477 INFO:tasks.workunit.client.0.smithi123.stdout:3/369: creat d13/d24/f70 x:0 0 0 2024-03-30T04:28:06.478 INFO:tasks.workunit.client.0.smithi123.stdout:2/319: creat d0/d1f/d1c/d39/d4b/d4c/d5f/f67 x:0 0 0 2024-03-30T04:28:06.480 INFO:tasks.workunit.client.0.smithi123.stdout:1/452: rename db/dd/d5f/d6d/f83 to db/dd/d42/d46/d56/d5c/f97 0 2024-03-30T04:28:06.482 INFO:tasks.workunit.client.0.smithi123.stdout:8/293: rename da/dc/d14/d18/f49 to da/dc/d14/d18/f4f 0 2024-03-30T04:28:06.483 INFO:tasks.workunit.client.0.smithi123.stdout:0/230: link d6/dd/d16/c3f d6/dd/c4d 0 2024-03-30T04:28:06.483 INFO:tasks.workunit.client.0.smithi123.stdout:2/320: mkdir d0/d1f/d27/d68 0 2024-03-30T04:28:06.484 INFO:tasks.workunit.client.0.smithi123.stdout:4/358: stat d2/d4/d2c/l72 0 2024-03-30T04:28:06.485 INFO:tasks.workunit.client.0.smithi123.stdout:1/453: read db/d1b/f28 [1386118,71061] 0 2024-03-30T04:28:06.487 INFO:tasks.workunit.client.0.smithi123.stdout:9/311: creat d6/d1f/d45/d4c/f74 x:0 0 0 2024-03-30T04:28:06.488 INFO:tasks.workunit.client.0.smithi123.stdout:8/294: chown da/d27/c2c 951 1 2024-03-30T04:28:06.488 INFO:tasks.workunit.client.0.smithi123.stdout:3/370: write d13/d24/d31/d23/f2e [752251,96399] 0 2024-03-30T04:28:06.492 INFO:tasks.workunit.client.0.smithi123.stdout:0/231: mkdir d6/dd/d16/d32/d4e 0 2024-03-30T04:28:06.493 INFO:tasks.workunit.client.0.smithi123.stdout:9/312: stat d6/l31 0 2024-03-30T04:28:06.497 INFO:tasks.workunit.client.0.smithi123.stdout:2/321: unlink d0/d37/d57/f66 0 2024-03-30T04:28:06.498 INFO:tasks.workunit.client.0.smithi123.stdout:6/462: dwrite d10/d13/f25 [0,4194304] 0 2024-03-30T04:28:06.502 INFO:tasks.workunit.client.0.smithi123.stdout:8/295: symlink da/l50 0 2024-03-30T04:28:06.502 INFO:tasks.workunit.client.0.smithi123.stdout:0/232: write d6/dd/d15/d28/f3d [489100,63862] 0 2024-03-30T04:28:06.507 INFO:tasks.workunit.client.0.smithi123.stdout:9/313: creat d6/d49/f75 x:0 0 0 2024-03-30T04:28:06.507 INFO:tasks.workunit.client.0.smithi123.stdout:6/463: stat d10/d82/f8a 0 2024-03-30T04:28:06.507 INFO:tasks.workunit.client.0.smithi123.stdout:5/344: dwrite dd/fe [0,4194304] 0 2024-03-30T04:28:06.507 INFO:tasks.workunit.client.0.smithi123.stdout:5/345: chown dd/d19/d1f/f27 194 1 2024-03-30T04:28:06.508 INFO:tasks.workunit.client.0.smithi123.stdout:1/454: write db/dd/d42/d46/d56/d5c/f97 [720487,4369] 0 2024-03-30T04:28:06.508 INFO:tasks.workunit.client.0.smithi123.stdout:6/464: truncate d10/f11 5181833 0 2024-03-30T04:28:06.510 INFO:tasks.workunit.client.0.smithi123.stdout:3/371: creat d13/d24/d31/d23/d63/f71 x:0 0 0 2024-03-30T04:28:06.512 INFO:tasks.workunit.client.0.smithi123.stdout:4/359: getdents d2/d4/d2c/d42/d35 0 2024-03-30T04:28:06.512 INFO:tasks.workunit.client.0.smithi123.stdout:8/296: creat da/d27/d3c/f51 x:0 0 0 2024-03-30T04:28:06.513 INFO:tasks.workunit.client.0.smithi123.stdout:9/314: symlink d6/l76 0 2024-03-30T04:28:06.515 INFO:tasks.workunit.client.0.smithi123.stdout:8/297: read da/f46 [1521735,62274] 0 2024-03-30T04:28:06.515 INFO:tasks.workunit.client.0.smithi123.stdout:1/455: fdatasync db/dd/d42/d46/d56/d5c/f76 0 2024-03-30T04:28:06.518 INFO:tasks.workunit.client.0.smithi123.stdout:9/315: write d6/d1f/d45/f50 [965603,75676] 0 2024-03-30T04:28:06.520 INFO:tasks.workunit.client.0.smithi123.stdout:3/372: creat d13/d24/d31/d23/d3a/d3c/d4e/f72 x:0 0 0 2024-03-30T04:28:06.521 INFO:tasks.workunit.client.0.smithi123.stdout:8/298: write da/fb [1435739,111424] 0 2024-03-30T04:28:06.523 INFO:tasks.workunit.client.0.smithi123.stdout:4/360: chown d2/d4/d7/d1a/f30 1 1 2024-03-30T04:28:06.525 INFO:tasks.workunit.client.0.smithi123.stdout:9/316: mknod d6/d58/c77 0 2024-03-30T04:28:06.526 INFO:tasks.workunit.client.0.smithi123.stdout:1/456: creat db/dd/d42/f98 x:0 0 0 2024-03-30T04:28:06.527 INFO:tasks.workunit.client.0.smithi123.stdout:3/373: dread - d13/d24/d31/d23/d63/f71 zero size 2024-03-30T04:28:06.531 INFO:tasks.workunit.client.0.smithi123.stdout:9/317: stat d6/d58/f5a 0 2024-03-30T04:28:06.533 INFO:tasks.workunit.client.0.smithi123.stdout:8/299: write da/dc/d14/d18/f4f [2647746,33098] 0 2024-03-30T04:28:06.534 INFO:tasks.workunit.client.0.smithi123.stdout:2/322: dwrite d0/d1f/d1c/d39/d4b/d4c/f64 [0,4194304] 0 2024-03-30T04:28:06.538 INFO:tasks.workunit.client.0.smithi123.stdout:2/323: dread - d0/d1f/d1c/d39/d44/f55 zero size 2024-03-30T04:28:06.539 INFO:tasks.workunit.client.0.smithi123.stdout:6/465: dwrite d10/f3a [0,4194304] 0 2024-03-30T04:28:06.540 INFO:tasks.workunit.client.0.smithi123.stdout:3/374: creat d13/d24/d31/d23/d3a/d3c/f73 x:0 0 0 2024-03-30T04:28:06.542 INFO:tasks.workunit.client.0.smithi123.stdout:9/318: fsync d6/d1f/f2e 0 2024-03-30T04:28:06.543 INFO:tasks.workunit.client.0.smithi123.stdout:4/361: getdents d2/d4/d2c 0 2024-03-30T04:28:06.547 INFO:tasks.workunit.client.0.smithi123.stdout:8/300: dread da/dc/d14/f44 [0,4194304] 0 2024-03-30T04:28:06.549 INFO:tasks.workunit.client.0.smithi123.stdout:3/375: stat d13/d24/d39/f41 0 2024-03-30T04:28:06.550 INFO:tasks.workunit.client.0.smithi123.stdout:2/324: rename d0/d1f/d1c to d0/d1f/d1c/d69 22 2024-03-30T04:28:06.552 INFO:tasks.workunit.client.0.smithi123.stdout:9/319: mknod d6/d49/d67/d69/c78 0 2024-03-30T04:28:06.553 INFO:tasks.workunit.client.0.smithi123.stdout:8/301: write f8 [4439321,25799] 0 2024-03-30T04:28:06.555 INFO:tasks.workunit.client.0.smithi123.stdout:9/320: write d6/fd [7400272,6371] 0 2024-03-30T04:28:06.556 INFO:tasks.workunit.client.0.smithi123.stdout:4/362: write d2/d4/d2c/d42/d35/f5c [1218021,38777] 0 2024-03-30T04:28:06.562 INFO:tasks.workunit.client.0.smithi123.stdout:1/457: getdents db/dd/d5f/d6d/d85/d89/d90 0 2024-03-30T04:28:06.563 INFO:tasks.workunit.client.0.smithi123.stdout:8/302: creat da/dc/f52 x:0 0 0 2024-03-30T04:28:06.565 INFO:tasks.workunit.client.0.smithi123.stdout:8/303: fdatasync da/f3d 0 2024-03-30T04:28:06.565 INFO:tasks.workunit.client.0.smithi123.stdout:1/458: read db/f45 [403277,50651] 0 2024-03-30T04:28:06.565 INFO:tasks.workunit.client.0.smithi123.stdout:3/376: fsync d13/d24/f57 0 2024-03-30T04:28:06.568 INFO:tasks.workunit.client.0.smithi123.stdout:9/321: fsync d6/d23/f40 0 2024-03-30T04:28:06.568 INFO:tasks.workunit.client.0.smithi123.stdout:4/363: symlink d2/d4/d2c/d42/l7f 0 2024-03-30T04:28:06.570 INFO:tasks.workunit.client.0.smithi123.stdout:6/466: truncate d10/d13/d1c/d42/f51 2584082 0 2024-03-30T04:28:06.572 INFO:tasks.workunit.client.0.smithi123.stdout:3/377: truncate d13/d24/f70 1025441 0 2024-03-30T04:28:06.573 INFO:tasks.workunit.client.0.smithi123.stdout:8/304: creat da/d27/f53 x:0 0 0 2024-03-30T04:28:06.573 INFO:tasks.workunit.client.0.smithi123.stdout:9/322: chown l5 475867343 1 2024-03-30T04:28:06.576 INFO:tasks.workunit.client.0.smithi123.stdout:3/378: fdatasync d13/d24/d31/d30/f45 0 2024-03-30T04:28:06.577 INFO:tasks.workunit.client.0.smithi123.stdout:9/323: chown d6/d1f/f3f 2362 1 2024-03-30T04:28:06.579 INFO:tasks.workunit.client.0.smithi123.stdout:4/364: mkdir d2/d4/d2c/d42/d80 0 2024-03-30T04:28:06.579 INFO:tasks.workunit.client.0.smithi123.stdout:8/305: mknod da/d27/c54 0 2024-03-30T04:28:06.580 INFO:tasks.workunit.client.0.smithi123.stdout:3/379: read d13/d24/d31/f64 [3034196,17632] 0 2024-03-30T04:28:06.582 INFO:tasks.workunit.client.0.smithi123.stdout:8/306: write f8 [3815219,12273] 0 2024-03-30T04:28:06.587 INFO:tasks.workunit.client.0.smithi123.stdout:6/467: dread d10/d13/d1c/f6f [0,4194304] 0 2024-03-30T04:28:06.589 INFO:tasks.workunit.client.0.smithi123.stdout:4/365: mknod d2/d4/d7/d63/c81 0 2024-03-30T04:28:06.592 INFO:tasks.workunit.client.0.smithi123.stdout:3/380: mknod d13/d24/d31/d23/d3a/d3c/c74 0 2024-03-30T04:28:06.598 INFO:tasks.workunit.client.0.smithi123.stdout:4/366: creat d2/d4/d7/d63/f82 x:0 0 0 2024-03-30T04:28:06.598 INFO:tasks.workunit.client.0.smithi123.stdout:9/324: dwrite d6/d1f/d45/f50 [0,4194304] 0 2024-03-30T04:28:06.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:06 smithi135 ceph-mon[44364]: pgmap v36: 65 pgs: 65 active+clean; 649 MiB data, 3.8 GiB used, 533 GiB / 536 GiB avail; 8.4 MiB/s rd, 76 MiB/s wr, 309 op/s 2024-03-30T04:28:06.601 INFO:tasks.workunit.client.0.smithi123.stdout:6/468: creat d10/fa0 x:0 0 0 2024-03-30T04:28:06.603 INFO:tasks.workunit.client.0.smithi123.stdout:8/307: truncate f8 5279208 0 2024-03-30T04:28:06.605 INFO:tasks.workunit.client.0.smithi123.stdout:8/308: dread - da/dc/f52 zero size 2024-03-30T04:28:06.608 INFO:tasks.workunit.client.0.smithi123.stdout:9/325: symlink d6/d49/d67/l79 0 2024-03-30T04:28:06.610 INFO:tasks.workunit.client.0.smithi123.stdout:1/459: dwrite db/dd/d42/d46/d39/f52 [0,4194304] 0 2024-03-30T04:28:06.615 INFO:tasks.workunit.client.0.smithi123.stdout:8/309: creat da/f55 x:0 0 0 2024-03-30T04:28:06.617 INFO:tasks.workunit.client.0.smithi123.stdout:1/460: mknod db/dd/d42/d8f/c99 0 2024-03-30T04:28:06.683 INFO:tasks.workunit.client.0.smithi123.stdout:7/302: sync 2024-03-30T04:28:06.684 INFO:tasks.workunit.client.0.smithi123.stdout:1/461: fsync db/f45 0 2024-03-30T04:28:06.685 INFO:tasks.workunit.client.0.smithi123.stdout:8/310: write da/d27/d3c/d2d/f4b [167348,90521] 0 2024-03-30T04:28:06.690 INFO:tasks.workunit.client.0.smithi123.stdout:7/303: creat d2/da/d2d/f66 x:0 0 0 2024-03-30T04:28:06.695 INFO:tasks.workunit.client.0.smithi123.stdout:3/381: dwrite d13/d24/d31/d23/d3a/f54 [0,4194304] 0 2024-03-30T04:28:06.695 INFO:tasks.workunit.client.0.smithi123.stdout:6/469: dwrite d10/f26 [4194304,4194304] 0 2024-03-30T04:28:06.698 INFO:tasks.workunit.client.0.smithi123.stdout:6/470: stat d10/d13/d1c/d5b/c70 0 2024-03-30T04:28:06.700 INFO:tasks.workunit.client.0.smithi123.stdout:9/326: dwrite d6/d1f/f71 [0,4194304] 0 2024-03-30T04:28:06.704 INFO:tasks.workunit.client.0.smithi123.stdout:7/304: write d2/da/fb [4597605,19348] 0 2024-03-30T04:28:06.710 INFO:tasks.workunit.client.0.smithi123.stdout:6/471: readlink d10/d13/l34 0 2024-03-30T04:28:06.712 INFO:tasks.workunit.client.0.smithi123.stdout:8/311: truncate da/fb 3355652 0 2024-03-30T04:28:06.713 INFO:tasks.workunit.client.0.smithi123.stdout:7/305: creat d2/dc/d56/f67 x:0 0 0 2024-03-30T04:28:06.716 INFO:tasks.workunit.client.0.smithi123.stdout:9/327: mknod d6/d1f/d4d/c7a 0 2024-03-30T04:28:06.717 INFO:tasks.workunit.client.0.smithi123.stdout:1/462: getdents db/dd/d42/d46/d5d/d7a 0 2024-03-30T04:28:06.718 INFO:tasks.workunit.client.0.smithi123.stdout:8/312: mkdir da/d27/d56 0 2024-03-30T04:28:06.719 INFO:tasks.workunit.client.0.smithi123.stdout:7/306: mkdir d2/dc/d49/d68 0 2024-03-30T04:28:06.719 INFO:tasks.workunit.client.0.smithi123.stdout:9/328: chown d6/d23/c42 6468307 1 2024-03-30T04:28:06.727 INFO:tasks.workunit.client.0.smithi123.stdout:8/313: truncate da/f46 8237265 0 2024-03-30T04:28:06.728 INFO:tasks.workunit.client.0.smithi123.stdout:3/382: dwrite d13/f4a [0,4194304] 0 2024-03-30T04:28:06.732 INFO:tasks.workunit.client.0.smithi123.stdout:7/307: mknod d2/da/d11/d61/c69 0 2024-03-30T04:28:06.734 INFO:tasks.workunit.client.0.smithi123.stdout:3/383: creat d13/d24/d31/f75 x:0 0 0 2024-03-30T04:28:06.734 INFO:tasks.workunit.client.0.smithi123.stdout:9/329: link d6/d49/c64 d6/d1f/c7b 0 2024-03-30T04:28:06.739 INFO:tasks.workunit.client.0.smithi123.stdout:7/308: rename d2/da/d11/f3c to d2/d4d/f6a 0 2024-03-30T04:28:06.741 INFO:tasks.workunit.client.0.smithi123.stdout:3/384: creat d13/d24/d31/d23/f76 x:0 0 0 2024-03-30T04:28:06.741 INFO:tasks.workunit.client.0.smithi123.stdout:7/309: stat d2/dc/d1c/f23 0 2024-03-30T04:28:06.741 INFO:tasks.workunit.client.0.smithi123.stdout:1/463: write db/dd/d42/d46/d56/f70 [340558,109284] 0 2024-03-30T04:28:06.742 INFO:tasks.workunit.client.0.smithi123.stdout:9/330: mkdir d6/d7c 0 2024-03-30T04:28:06.747 INFO:tasks.workunit.client.0.smithi123.stdout:1/464: rmdir db 39 2024-03-30T04:28:06.749 INFO:tasks.workunit.client.0.smithi123.stdout:7/310: mknod d2/da/c6b 0 2024-03-30T04:28:06.750 INFO:tasks.workunit.client.0.smithi123.stdout:7/311: chown d2/f8 435 1 2024-03-30T04:28:06.752 INFO:tasks.workunit.client.0.smithi123.stdout:7/312: chown d2/dc/c4f 404 1 2024-03-30T04:28:06.753 INFO:tasks.workunit.client.0.smithi123.stdout:8/314: truncate da/dc/f16 72436 0 2024-03-30T04:28:06.754 INFO:tasks.workunit.client.0.smithi123.stdout:7/313: chown d2/dc/d49/l5d 41 1 2024-03-30T04:28:06.755 INFO:tasks.workunit.client.0.smithi123.stdout:1/465: chown db/dd/d42/d46/d5d/d7a 761 1 2024-03-30T04:28:06.755 INFO:tasks.workunit.client.0.smithi123.stdout:9/331: mkdir d6/d1f/d72/d7d 0 2024-03-30T04:28:06.758 INFO:tasks.workunit.client.0.smithi123.stdout:9/332: write d6/d1f/f37 [1790772,98677] 0 2024-03-30T04:28:06.761 INFO:tasks.workunit.client.0.smithi123.stdout:7/314: creat d2/dc/d49/f6c x:0 0 0 2024-03-30T04:28:06.763 INFO:tasks.workunit.client.0.smithi123.stdout:1/466: rmdir db/dd/d5f/d6d/d85 39 2024-03-30T04:28:06.769 INFO:tasks.workunit.client.0.smithi123.stdout:7/315: creat d2/da/d22/d3b/f6d x:0 0 0 2024-03-30T04:28:06.770 INFO:tasks.workunit.client.0.smithi123.stdout:1/467: chown db/dd/d5f/d6d/d85 152 1 2024-03-30T04:28:06.772 INFO:tasks.workunit.client.0.smithi123.stdout:1/468: readlink db/lf 0 2024-03-30T04:28:06.774 INFO:tasks.workunit.client.0.smithi123.stdout:9/333: creat d6/f7e x:0 0 0 2024-03-30T04:28:06.774 INFO:tasks.workunit.client.0.smithi123.stdout:7/316: fdatasync d2/d4d/f6a 0 2024-03-30T04:28:06.776 INFO:tasks.workunit.client.0.smithi123.stdout:1/469: fsync db/f6e 0 2024-03-30T04:28:06.778 INFO:tasks.workunit.client.0.smithi123.stdout:3/385: dwrite d13/d24/d31/d23/d3a/f40 [0,4194304] 0 2024-03-30T04:28:06.780 INFO:tasks.workunit.client.0.smithi123.stdout:8/315: dread da/dc/f20 [0,4194304] 0 2024-03-30T04:28:06.781 INFO:tasks.workunit.client.0.smithi123.stdout:3/386: chown l4 125 1 2024-03-30T04:28:06.784 INFO:tasks.workunit.client.0.smithi123.stdout:7/317: symlink d2/da/d11/d34/l6e 0 2024-03-30T04:28:06.784 INFO:tasks.workunit.client.0.smithi123.stdout:8/316: write da/dc/d14/d18/d21/f40 [588225,56815] 0 2024-03-30T04:28:06.787 INFO:tasks.workunit.client.0.smithi123.stdout:3/387: symlink d13/d24/d31/d23/d3a/d3c/d4e/l77 0 2024-03-30T04:28:06.791 INFO:tasks.workunit.client.0.smithi123.stdout:7/318: mknod d2/da/d22/d3b/c6f 0 2024-03-30T04:28:06.793 INFO:tasks.workunit.client.0.smithi123.stdout:8/317: dread da/dc/f16 [0,4194304] 0 2024-03-30T04:28:06.793 INFO:tasks.workunit.client.0.smithi123.stdout:3/388: unlink d13/d24/d31/d23/d3a/d3c/d66/l6a 0 2024-03-30T04:28:06.793 INFO:tasks.workunit.client.0.smithi123.stdout:7/319: fdatasync d2/dc/d1c/f1d 0 2024-03-30T04:28:06.794 INFO:tasks.workunit.client.0.smithi123.stdout:9/334: link d6/d23/c43 d6/d1f/d72/d7d/c7f 0 2024-03-30T04:28:06.796 INFO:tasks.workunit.client.0.smithi123.stdout:7/320: fsync d2/da/d2d/f42 0 2024-03-30T04:28:06.799 INFO:tasks.workunit.client.0.smithi123.stdout:7/321: write d2/da/d11/d61/f57 [521383,15346] 0 2024-03-30T04:28:06.801 INFO:tasks.workunit.client.0.smithi123.stdout:3/389: link f11 d13/d24/d31/d23/d3a/d3c/f78 0 2024-03-30T04:28:06.801 INFO:tasks.workunit.client.0.smithi123.stdout:1/470: dread db/dd/d42/d46/d39/f52 [0,4194304] 0 2024-03-30T04:28:06.801 INFO:tasks.workunit.client.0.smithi123.stdout:7/322: chown d2/da/d22/f2f 92 1 2024-03-30T04:28:06.802 INFO:tasks.workunit.client.0.smithi123.stdout:7/323: fdatasync f0 0 2024-03-30T04:28:06.804 INFO:tasks.workunit.client.0.smithi123.stdout:3/390: mkdir d13/d24/d39/d79 0 2024-03-30T04:28:06.804 INFO:tasks.workunit.client.0.smithi123.stdout:7/324: write d2/da/d2d/f42 [3802380,128638] 0 2024-03-30T04:28:06.808 INFO:tasks.workunit.client.0.smithi123.stdout:7/325: rename d2/da/fb to d2/da/d11/d34/f70 0 2024-03-30T04:28:06.810 INFO:tasks.workunit.client.0.smithi123.stdout:0/233: sync 2024-03-30T04:28:06.814 INFO:tasks.workunit.client.0.smithi123.stdout:0/234: readlink d6/dd/d16/l3e 0 2024-03-30T04:28:06.818 INFO:tasks.workunit.client.0.smithi123.stdout:1/471: dwrite db/dd/f54 [0,4194304] 0 2024-03-30T04:28:06.818 INFO:tasks.workunit.client.0.smithi123.stdout:9/335: dread d6/d1f/f41 [0,4194304] 0 2024-03-30T04:28:06.821 INFO:tasks.workunit.client.0.smithi123.stdout:0/235: truncate f5 2212432 0 2024-03-30T04:28:06.824 INFO:tasks.workunit.client.0.smithi123.stdout:9/336: dread - d6/d1f/f3f zero size 2024-03-30T04:28:06.826 INFO:tasks.workunit.client.0.smithi123.stdout:0/236: chown d6/c8 9123 1 2024-03-30T04:28:06.827 INFO:tasks.workunit.client.0.smithi123.stdout:1/472: fsync db/dd/d42/d46/f79 0 2024-03-30T04:28:06.829 INFO:tasks.workunit.client.0.smithi123.stdout:3/391: dwrite d13/d24/d31/d23/d3a/f40 [0,4194304] 0 2024-03-30T04:28:06.833 INFO:tasks.workunit.client.0.smithi123.stdout:0/237: truncate d6/dd/d18/f47 730995 0 2024-03-30T04:28:06.834 INFO:tasks.workunit.client.0.smithi123.stdout:3/392: truncate d13/f5e 694932 0 2024-03-30T04:28:06.835 INFO:tasks.workunit.client.0.smithi123.stdout:7/326: dwrite d2/dc/d56/f60 [0,4194304] 0 2024-03-30T04:28:06.839 INFO:tasks.workunit.client.0.smithi123.stdout:0/238: write d6/dd/f36 [154061,52169] 0 2024-03-30T04:28:06.843 INFO:tasks.workunit.client.0.smithi123.stdout:7/327: write d2/da/f33 [2975007,25422] 0 2024-03-30T04:28:06.847 INFO:tasks.workunit.client.0.smithi123.stdout:0/239: stat d6/f9 0 2024-03-30T04:28:06.849 INFO:tasks.workunit.client.0.smithi123.stdout:3/393: link ff d13/d24/d39/f7a 0 2024-03-30T04:28:06.852 INFO:tasks.workunit.client.0.smithi123.stdout:3/394: mkdir d13/d7b 0 2024-03-30T04:28:06.855 INFO:tasks.workunit.client.0.smithi123.stdout:0/240: mknod d6/dd/d31/d35/c4f 0 2024-03-30T04:28:06.855 INFO:tasks.workunit.client.0.smithi123.stdout:9/337: dwrite d6/d1f/d45/d4c/f4e [0,4194304] 0 2024-03-30T04:28:06.867 INFO:tasks.workunit.client.0.smithi123.stdout:9/338: getdents d6/d1f/d45/d4c 0 2024-03-30T04:28:06.868 INFO:tasks.workunit.client.0.smithi123.stdout:0/241: rmdir d6/dd/d31/d35 39 2024-03-30T04:28:06.873 INFO:tasks.workunit.client.0.smithi123.stdout:0/242: write d6/dd/d15/d28/f33 [1552222,14299] 0 2024-03-30T04:28:06.873 INFO:tasks.workunit.client.0.smithi123.stdout:3/395: dwrite d13/d24/d31/d23/f2e [0,4194304] 0 2024-03-30T04:28:06.873 INFO:tasks.workunit.client.0.smithi123.stdout:7/328: dread d2/dc/f17 [0,4194304] 0 2024-03-30T04:28:06.875 INFO:tasks.workunit.client.0.smithi123.stdout:9/339: getdents d6/d1f 0 2024-03-30T04:28:06.877 INFO:tasks.workunit.client.0.smithi123.stdout:3/396: unlink d13/c1f 0 2024-03-30T04:28:06.881 INFO:tasks.workunit.client.0.smithi123.stdout:3/397: fdatasync ff 0 2024-03-30T04:28:06.882 INFO:tasks.workunit.client.0.smithi123.stdout:7/329: creat d2/da/f71 x:0 0 0 2024-03-30T04:28:06.885 INFO:tasks.workunit.client.0.smithi123.stdout:7/330: write d2/dc/d49/f6c [809790,102156] 0 2024-03-30T04:28:06.886 INFO:tasks.workunit.client.0.smithi123.stdout:0/243: write d6/dd/d18/d1f/f27 [1246608,47182] 0 2024-03-30T04:28:06.891 INFO:tasks.workunit.client.0.smithi123.stdout:7/331: chown d2/da/d22/d3b/f6d 173761 1 2024-03-30T04:28:06.892 INFO:tasks.workunit.client.0.smithi123.stdout:3/398: truncate d13/d24/d31/d23/d3a/d3c/d4e/f72 635378 0 2024-03-30T04:28:06.895 INFO:tasks.workunit.client.0.smithi123.stdout:3/399: creat d13/f7c x:0 0 0 2024-03-30T04:28:06.898 INFO:tasks.workunit.client.0.smithi123.stdout:0/244: rename d6/dd/c4d to d6/dd/d16/c50 0 2024-03-30T04:28:06.900 INFO:tasks.workunit.client.0.smithi123.stdout:9/340: dwrite d6/d23/f2b [4194304,4194304] 0 2024-03-30T04:28:06.900 INFO:tasks.workunit.client.0.smithi123.stdout:7/332: creat d2/da/d11/d61/f72 x:0 0 0 2024-03-30T04:28:06.903 INFO:tasks.workunit.client.0.smithi123.stdout:3/400: mknod d13/d24/d31/d23/d3a/d3c/c7d 0 2024-03-30T04:28:06.904 INFO:tasks.workunit.client.0.smithi123.stdout:9/341: mknod d6/d1f/c80 0 2024-03-30T04:28:06.908 INFO:tasks.workunit.client.0.smithi123.stdout:3/401: mknod d13/d24/d31/d23/c7e 0 2024-03-30T04:28:06.912 INFO:tasks.workunit.client.0.smithi123.stdout:0/245: rename d6/dd/f14 to d6/dd/f51 0 2024-03-30T04:28:06.913 INFO:tasks.workunit.client.0.smithi123.stdout:3/402: write d13/d24/d39/f41 [3944677,16330] 0 2024-03-30T04:28:06.919 INFO:tasks.workunit.client.0.smithi123.stdout:9/342: rename c3 to d6/c81 0 2024-03-30T04:28:06.920 INFO:tasks.workunit.client.0.smithi123.stdout:0/246: mkdir d6/d52 0 2024-03-30T04:28:06.924 INFO:tasks.workunit.client.0.smithi123.stdout:3/403: unlink d13/d24/f53 0 2024-03-30T04:28:06.926 INFO:tasks.workunit.client.0.smithi123.stdout:9/343: mkdir d6/d7c/d82 0 2024-03-30T04:28:06.930 INFO:tasks.workunit.client.0.smithi123.stdout:0/247: symlink d6/dd/d16/d32/d4e/l53 0 2024-03-30T04:28:06.932 INFO:tasks.workunit.client.0.smithi123.stdout:3/404: rename d13/d24/d39/f41 to d13/d24/d31/d23/d3a/d3c/d66/f7f 0 2024-03-30T04:28:06.934 INFO:tasks.workunit.client.0.smithi123.stdout:0/248: chown d6/dd/d31/d35/d3a 748526609 1 2024-03-30T04:28:06.937 INFO:tasks.workunit.client.0.smithi123.stdout:3/405: creat d13/f80 x:0 0 0 2024-03-30T04:28:06.937 INFO:tasks.workunit.client.0.smithi123.stdout:9/344: link d6/d1f/l4a d6/d49/d67/l83 0 2024-03-30T04:28:06.938 INFO:tasks.workunit.client.0.smithi123.stdout:7/333: dread d2/da/d2d/f39 [0,4194304] 0 2024-03-30T04:28:06.938 INFO:tasks.workunit.client.0.smithi123.stdout:0/249: mknod d6/dd/d15/d28/d4b/c54 0 2024-03-30T04:28:06.941 INFO:tasks.workunit.client.0.smithi123.stdout:3/406: chown d13/d24/d31/d23/d3a/d3c/d4e/c5c 28897575 1 2024-03-30T04:28:06.944 INFO:tasks.workunit.client.0.smithi123.stdout:9/345: rename d6/l11 to d6/d1f/d4d/l84 0 2024-03-30T04:28:06.946 INFO:tasks.workunit.client.0.smithi123.stdout:9/346: dread - d6/d1f/d45/d4c/f54 zero size 2024-03-30T04:28:06.946 INFO:tasks.workunit.client.0.smithi123.stdout:3/407: unlink d13/d24/d31/d23/d3a/d3c/c7d 0 2024-03-30T04:28:06.949 INFO:tasks.workunit.client.0.smithi123.stdout:3/408: write d13/d24/d31/d23/f2b [3035523,19449] 0 2024-03-30T04:28:06.956 INFO:tasks.workunit.client.0.smithi123.stdout:3/409: creat d13/d24/d31/d23/d3a/d3c/d4e/f81 x:0 0 0 2024-03-30T04:28:06.957 INFO:tasks.workunit.client.0.smithi123.stdout:0/250: dwrite d6/dd/d15/d28/f3d [0,4194304] 0 2024-03-30T04:28:06.958 INFO:tasks.workunit.client.0.smithi123.stdout:3/410: fsync d13/d24/d31/d23/f6c 0 2024-03-30T04:28:06.960 INFO:tasks.workunit.client.0.smithi123.stdout:9/347: dread d6/d23/f3b [0,4194304] 0 2024-03-30T04:28:06.961 INFO:tasks.workunit.client.0.smithi123.stdout:7/334: dread d2/da/d22/d3b/f4b [0,4194304] 0 2024-03-30T04:28:06.961 INFO:tasks.workunit.client.0.smithi123.stdout:0/251: symlink d6/dd/d15/d28/d4b/l55 0 2024-03-30T04:28:06.963 INFO:tasks.workunit.client.0.smithi123.stdout:9/348: fdatasync d6/d1f/d45/d4c/f4e 0 2024-03-30T04:28:06.964 INFO:tasks.workunit.client.0.smithi123.stdout:3/411: creat d13/d24/d31/d23/d3a/d3c/d4e/d5d/f82 x:0 0 0 2024-03-30T04:28:06.966 INFO:tasks.workunit.client.0.smithi123.stdout:0/252: mkdir d6/dd/d31/d56 0 2024-03-30T04:28:06.968 INFO:tasks.workunit.client.0.smithi123.stdout:3/412: fsync d13/d24/d39/f7a 0 2024-03-30T04:28:06.971 INFO:tasks.workunit.client.0.smithi123.stdout:7/335: link d2/dc/d49/f63 d2/da/d22/f73 0 2024-03-30T04:28:06.972 INFO:tasks.workunit.client.0.smithi123.stdout:5/346: sync 2024-03-30T04:28:06.973 INFO:tasks.workunit.client.0.smithi123.stdout:2/325: sync 2024-03-30T04:28:06.975 INFO:tasks.workunit.client.0.smithi123.stdout:0/253: dread d6/dd/d15/f29 [0,4194304] 0 2024-03-30T04:28:06.976 INFO:tasks.workunit.client.0.smithi123.stdout:9/349: mknod d6/d7c/d82/c85 0 2024-03-30T04:28:06.986 INFO:tasks.workunit.client.0.smithi123.stdout:5/347: creat dd/f74 x:0 0 0 2024-03-30T04:28:06.988 INFO:tasks.workunit.client.0.smithi123.stdout:9/350: truncate d6/fb 442263 0 2024-03-30T04:28:06.989 INFO:tasks.workunit.client.0.smithi123.stdout:3/413: dwrite d13/d24/d31/d23/f2e [0,4194304] 0 2024-03-30T04:28:06.990 INFO:tasks.workunit.client.0.smithi123.stdout:2/326: dread d0/d1f/d1c/f4d [0,4194304] 0 2024-03-30T04:28:06.991 INFO:tasks.workunit.client.0.smithi123.stdout:9/351: dread - d6/d23/f46 zero size 2024-03-30T04:28:06.991 INFO:tasks.workunit.client.0.smithi123.stdout:7/336: dwrite d2/dc/d49/f6c [0,4194304] 0 2024-03-30T04:28:06.992 INFO:tasks.workunit.client.0.smithi123.stdout:5/348: creat dd/d19/d3e/f75 x:0 0 0 2024-03-30T04:28:06.993 INFO:tasks.workunit.client.0.smithi123.stdout:2/327: stat d0/f10 0 2024-03-30T04:28:06.993 INFO:tasks.workunit.client.0.smithi123.stdout:9/352: chown d6/d49 990 1 2024-03-30T04:28:06.994 INFO:tasks.workunit.client.0.smithi123.stdout:3/414: stat d13/d24/d39/f7a 0 2024-03-30T04:28:06.997 INFO:tasks.workunit.client.0.smithi123.stdout:2/328: chown d0/d1f/d5e/f4f 63 1 2024-03-30T04:28:06.997 INFO:tasks.workunit.client.0.smithi123.stdout:7/337: write d2/dc/d56/f60 [1160009,19933] 0 2024-03-30T04:28:06.998 INFO:tasks.workunit.client.0.smithi123.stdout:9/353: chown d6/d23/c36 26318607 1 2024-03-30T04:28:06.999 INFO:tasks.workunit.client.0.smithi123.stdout:5/349: dread dd/d41/f43 [0,4194304] 0 2024-03-30T04:28:07.000 INFO:tasks.workunit.client.0.smithi123.stdout:2/329: write d0/d1f/d5e/f65 [36650,49438] 0 2024-03-30T04:28:07.004 INFO:tasks.workunit.client.0.smithi123.stdout:5/350: chown dd/d19/d3e/d5b/d62/f71 10 1 2024-03-30T04:28:07.009 INFO:tasks.workunit.client.0.smithi123.stdout:7/338: rename d2/da/d2d/f42 to d2/da/d22/d3b/f74 0 2024-03-30T04:28:07.012 INFO:tasks.workunit.client.0.smithi123.stdout:7/339: chown d2/da/d22/d3b/c3e 495229 1 2024-03-30T04:28:07.015 INFO:tasks.workunit.client.0.smithi123.stdout:7/340: write d2/da/d22/d3b/f6d [163034,35381] 0 2024-03-30T04:28:07.019 INFO:tasks.workunit.client.0.smithi123.stdout:7/341: symlink d2/da/d11/l75 0 2024-03-30T04:28:07.022 INFO:tasks.workunit.client.0.smithi123.stdout:5/351: truncate dd/f32 2743069 0 2024-03-30T04:28:07.024 INFO:tasks.workunit.client.0.smithi123.stdout:7/342: write d2/da/d22/d3b/f6d [733334,64491] 0 2024-03-30T04:28:07.025 INFO:tasks.workunit.client.0.smithi123.stdout:5/352: fsync dd/f2f 0 2024-03-30T04:28:07.025 INFO:tasks.workunit.client.0.smithi123.stdout:9/354: dwrite f0 [0,4194304] 0 2024-03-30T04:28:07.027 INFO:tasks.workunit.client.0.smithi123.stdout:2/330: dread d0/d1f/d1c/d39/d44/f5c [0,4194304] 0 2024-03-30T04:28:07.034 INFO:tasks.workunit.client.0.smithi123.stdout:7/343: mkdir d2/da/d11/d34/d76 0 2024-03-30T04:28:07.035 INFO:tasks.workunit.client.0.smithi123.stdout:5/353: mknod dd/d19/d1f/d35/c76 0 2024-03-30T04:28:07.040 INFO:tasks.workunit.client.0.smithi123.stdout:5/354: chown dd/c17 13652 1 2024-03-30T04:28:07.042 INFO:tasks.workunit.client.0.smithi123.stdout:3/415: dread fd [4194304,4194304] 0 2024-03-30T04:28:07.043 INFO:tasks.workunit.client.0.smithi123.stdout:7/344: link d2/dc/c4f d2/da/d11/d34/d76/c77 0 2024-03-30T04:28:07.043 INFO:tasks.workunit.client.0.smithi123.stdout:2/331: write d0/d1f/f1b [94563,44636] 0 2024-03-30T04:28:07.048 INFO:tasks.workunit.client.0.smithi123.stdout:2/332: truncate d0/f10 4958501 0 2024-03-30T04:28:07.049 INFO:tasks.workunit.client.0.smithi123.stdout:9/355: dwrite d6/d1f/f4f [0,4194304] 0 2024-03-30T04:28:07.050 INFO:tasks.workunit.client.0.smithi123.stdout:7/345: creat d2/dc/d49/d68/f78 x:0 0 0 2024-03-30T04:28:07.054 INFO:tasks.workunit.client.0.smithi123.stdout:3/416: rename d13/d24/d31/d23/d3a/c6e to d13/d24/d31/d30/c83 0 2024-03-30T04:28:07.057 INFO:tasks.workunit.client.0.smithi123.stdout:9/356: chown d6/d23/c43 58988517 1 2024-03-30T04:28:07.058 INFO:tasks.workunit.client.0.smithi123.stdout:3/417: mknod d13/d24/d31/c84 0 2024-03-30T04:28:07.060 INFO:tasks.workunit.client.0.smithi123.stdout:7/346: write d2/da/d11/d34/f70 [1525175,14188] 0 2024-03-30T04:28:07.063 INFO:tasks.workunit.client.0.smithi123.stdout:3/418: chown d13/d24/d31/d23/d3a/d3c/d4e/c5c 19386 1 2024-03-30T04:28:07.064 INFO:tasks.workunit.client.0.smithi123.stdout:9/357: rename d6/d23/c47 to d6/d1f/c86 0 2024-03-30T04:28:07.064 INFO:tasks.workunit.client.0.smithi123.stdout:5/355: dwrite dd/d19/d1f/f67 [0,4194304] 0 2024-03-30T04:28:07.066 INFO:tasks.workunit.client.0.smithi123.stdout:7/347: chown d2/da/d11/d61/f72 166962 1 2024-03-30T04:28:07.068 INFO:tasks.workunit.client.0.smithi123.stdout:3/419: chown l4 983 1 2024-03-30T04:28:07.069 INFO:tasks.workunit.client.0.smithi123.stdout:7/348: stat d2/da/d2d/f40 0 2024-03-30T04:28:07.073 INFO:tasks.workunit.client.0.smithi123.stdout:3/420: mknod d13/d24/d31/c85 0 2024-03-30T04:28:07.079 INFO:tasks.workunit.client.0.smithi123.stdout:7/349: getdents d2/da/d11/d34 0 2024-03-30T04:28:07.081 INFO:tasks.workunit.client.0.smithi123.stdout:3/421: fsync d13/d24/d31/d23/d3a/d3c/d4e/d5d/f82 0 2024-03-30T04:28:07.081 INFO:tasks.workunit.client.0.smithi123.stdout:5/356: dread dd/d41/f47 [0,4194304] 0 2024-03-30T04:28:07.081 INFO:tasks.workunit.client.0.smithi123.stdout:2/333: dwrite d0/d1f/d1c/f3f [0,4194304] 0 2024-03-30T04:28:07.081 INFO:tasks.workunit.client.0.smithi123.stdout:7/350: chown d2/l35 361234248 1 2024-03-30T04:28:07.084 INFO:tasks.workunit.client.0.smithi123.stdout:7/351: write d2/da/d22/f2f [871741,86862] 0 2024-03-30T04:28:07.086 INFO:tasks.workunit.client.0.smithi123.stdout:3/422: symlink d13/l86 0 2024-03-30T04:28:07.092 INFO:tasks.workunit.client.0.smithi123.stdout:5/357: creat dd/d19/d1f/d3c/f77 x:0 0 0 2024-03-30T04:28:07.095 INFO:tasks.workunit.client.0.smithi123.stdout:5/358: fdatasync dd/d19/d3e/d5b/f69 0 2024-03-30T04:28:07.096 INFO:tasks.workunit.client.0.smithi123.stdout:5/359: read dd/d19/d3e/f63 [4087099,10158] 0 2024-03-30T04:28:07.099 INFO:tasks.workunit.client.0.smithi123.stdout:7/352: dread d2/da/d2d/f39 [0,4194304] 0 2024-03-30T04:28:07.101 INFO:tasks.workunit.client.0.smithi123.stdout:5/360: symlink dd/d19/d1f/d68/l78 0 2024-03-30T04:28:07.101 INFO:tasks.workunit.client.0.smithi123.stdout:2/334: truncate d0/d1f/d1c/d39/f42 376923 0 2024-03-30T04:28:07.105 INFO:tasks.workunit.client.0.smithi123.stdout:9/358: dread d6/d23/f5d [0,4194304] 0 2024-03-30T04:28:07.108 INFO:tasks.workunit.client.0.smithi123.stdout:9/359: symlink d6/d49/d67/d69/l87 0 2024-03-30T04:28:07.114 INFO:tasks.workunit.client.0.smithi123.stdout:9/360: readlink d6/d1f/d45/l57 0 2024-03-30T04:28:07.114 INFO:tasks.workunit.client.0.smithi123.stdout:9/361: fdatasync d6/d1f/f2e 0 2024-03-30T04:28:07.121 INFO:tasks.workunit.client.0.smithi123.stdout:7/353: dwrite d2/da/d2d/f3d [0,4194304] 0 2024-03-30T04:28:07.124 INFO:tasks.workunit.client.0.smithi123.stdout:9/362: mknod d6/d1f/d72/d7d/c88 0 2024-03-30T04:28:07.126 INFO:tasks.workunit.client.0.smithi123.stdout:2/335: dwrite d0/d1f/d1c/d20/f46 [0,4194304] 0 2024-03-30T04:28:07.126 INFO:tasks.workunit.client.0.smithi123.stdout:5/361: dwrite dd/d19/d1f/d35/d3a/d57/f6a [0,4194304] 0 2024-03-30T04:28:07.126 INFO:tasks.workunit.client.0.smithi123.stdout:7/354: mknod d2/da/d22/d3b/c79 0 2024-03-30T04:28:07.136 INFO:tasks.workunit.client.0.smithi123.stdout:5/362: fsync f9 0 2024-03-30T04:28:07.136 INFO:tasks.workunit.client.0.smithi123.stdout:7/355: unlink d2/dc/f27 0 2024-03-30T04:28:07.139 INFO:tasks.workunit.client.0.smithi123.stdout:7/356: fsync d2/da/d22/f73 0 2024-03-30T04:28:07.143 INFO:tasks.workunit.client.0.smithi123.stdout:7/357: write d2/da/d11/f48 [548492,37048] 0 2024-03-30T04:28:07.144 INFO:tasks.workunit.client.0.smithi123.stdout:5/363: getdents dd/d19/d1f/d35/d3a/d57 0 2024-03-30T04:28:07.146 INFO:tasks.workunit.client.0.smithi123.stdout:7/358: fsync d2/dc/f52 0 2024-03-30T04:28:07.147 INFO:tasks.workunit.client.0.smithi123.stdout:7/359: fsync d2/dc/f52 0 2024-03-30T04:28:07.148 INFO:tasks.workunit.client.0.smithi123.stdout:9/363: dread d6/d1f/d45/f50 [0,4194304] 0 2024-03-30T04:28:07.149 INFO:tasks.workunit.client.0.smithi123.stdout:2/336: dread d0/d1f/d1c/f4d [0,4194304] 0 2024-03-30T04:28:07.152 INFO:tasks.workunit.client.0.smithi123.stdout:9/364: creat d6/d7c/f89 x:0 0 0 2024-03-30T04:28:07.153 INFO:tasks.workunit.client.0.smithi123.stdout:2/337: truncate d0/d37/d57/f62 536722 0 2024-03-30T04:28:07.161 INFO:tasks.workunit.client.0.smithi123.stdout:7/360: dread d2/da/d2d/f3d [0,4194304] 0 2024-03-30T04:28:07.162 INFO:tasks.workunit.client.0.smithi123.stdout:2/338: link d0/d1f/d1c/d20/f46 d0/d1f/d1c/d39/d4b/d4c/f6a 0 2024-03-30T04:28:07.164 INFO:tasks.workunit.client.0.smithi123.stdout:2/339: write d0/f1d [75650,120650] 0 2024-03-30T04:28:07.165 INFO:tasks.workunit.client.0.smithi123.stdout:7/361: write d2/da/d22/d3b/f54 [733815,22739] 0 2024-03-30T04:28:07.167 INFO:tasks.workunit.client.0.smithi123.stdout:7/362: stat d2/da/d22 0 2024-03-30T04:28:07.170 INFO:tasks.workunit.client.0.smithi123.stdout:9/365: dwrite d6/d1f/f2e [0,4194304] 0 2024-03-30T04:28:07.172 INFO:tasks.workunit.client.0.smithi123.stdout:5/364: dread dd/d19/d1f/f27 [0,4194304] 0 2024-03-30T04:28:07.175 INFO:tasks.workunit.client.0.smithi123.stdout:9/366: unlink d6/d1f/c27 0 2024-03-30T04:28:07.176 INFO:tasks.workunit.client.0.smithi123.stdout:5/365: mkdir dd/d19/d1f/d35/d3a/d79 0 2024-03-30T04:28:07.177 INFO:tasks.workunit.client.0.smithi123.stdout:8/318: sync 2024-03-30T04:28:07.177 INFO:tasks.workunit.client.0.smithi123.stdout:1/473: sync 2024-03-30T04:28:07.178 INFO:tasks.workunit.client.0.smithi123.stdout:6/472: sync 2024-03-30T04:28:07.178 INFO:tasks.workunit.client.0.smithi123.stdout:4/367: sync 2024-03-30T04:28:07.178 INFO:tasks.workunit.client.0.smithi123.stdout:5/366: write f0 [2954150,37838] 0 2024-03-30T04:28:07.181 INFO:tasks.workunit.client.0.smithi123.stdout:9/367: symlink d6/d1f/l8a 0 2024-03-30T04:28:07.183 INFO:tasks.workunit.client.0.smithi123.stdout:2/340: truncate d0/d1f/d1c/f22 2799521 0 2024-03-30T04:28:07.183 INFO:tasks.workunit.client.0.smithi123.stdout:6/473: fdatasync d10/f26 0 2024-03-30T04:28:07.187 INFO:tasks.workunit.client.0.smithi123.stdout:5/367: write dd/d19/d3e/d5b/f5e [542122,87420] 0 2024-03-30T04:28:07.188 INFO:tasks.workunit.client.0.smithi123.stdout:4/368: write d2/d4/d7/d63/f82 [986656,40139] 0 2024-03-30T04:28:07.188 INFO:tasks.workunit.client.0.smithi123.stdout:1/474: truncate db/dd/d42/d46/f2e 425059 0 2024-03-30T04:28:07.188 INFO:tasks.workunit.client.0.smithi123.stdout:9/368: chown d6/d1f/d45/l57 39895 1 2024-03-30T04:28:07.190 INFO:tasks.workunit.client.0.smithi123.stdout:2/341: fsync d0/d1f/d16/f1e 0 2024-03-30T04:28:07.201 INFO:tasks.workunit.client.0.smithi123.stdout:2/342: write d0/d1f/fa [367657,74029] 0 2024-03-30T04:28:07.206 INFO:tasks.workunit.client.0.smithi123.stdout:8/319: dread da/dc/f10 [0,4194304] 0 2024-03-30T04:28:07.206 INFO:tasks.workunit.client.0.smithi123.stdout:2/343: fsync d0/d1f/d1c/d39/d44/f5c 0 2024-03-30T04:28:07.207 INFO:tasks.workunit.client.0.smithi123.stdout:9/369: getdents d6/d49/d67 0 2024-03-30T04:28:07.207 INFO:tasks.workunit.client.0.smithi123.stdout:7/363: dwrite d2/da/d11/f19 [0,4194304] 0 2024-03-30T04:28:07.212 INFO:tasks.workunit.client.0.smithi123.stdout:1/475: creat db/dd/d42/f9a x:0 0 0 2024-03-30T04:28:07.219 INFO:tasks.workunit.client.0.smithi123.stdout:2/344: write d0/d1f/d1c/f5b [899182,127086] 0 2024-03-30T04:28:07.231 INFO:tasks.workunit.client.0.smithi123.stdout:2/345: truncate d0/d1f/fa 4929385 0 2024-03-30T04:28:07.237 INFO:tasks.workunit.client.0.smithi123.stdout:5/368: dwrite dd/f1c [0,4194304] 0 2024-03-30T04:28:07.237 INFO:tasks.workunit.client.0.smithi123.stdout:7/364: symlink d2/da/d11/d34/d76/l7a 0 2024-03-30T04:28:07.237 INFO:tasks.workunit.client.0.smithi123.stdout:6/474: dwrite d10/d13/d1c/f5d [0,4194304] 0 2024-03-30T04:28:07.244 INFO:tasks.workunit.client.0.smithi123.stdout:6/475: chown d10/d13/d6a 534702021 1 2024-03-30T04:28:07.244 INFO:tasks.workunit.client.0.smithi123.stdout:4/369: dwrite d2/d4/d7/fb [0,4194304] 0 2024-03-30T04:28:07.246 INFO:tasks.workunit.client.0.smithi123.stdout:1/476: link db/dd/d42/d46/f47 db/dd/d42/d8f/f9b 0 2024-03-30T04:28:07.250 INFO:tasks.workunit.client.0.smithi123.stdout:2/346: chown d0/d1f/d1c/f5b 14146 1 2024-03-30T04:28:07.252 INFO:tasks.workunit.client.0.smithi123.stdout:8/320: dwrite da/dc/f52 [0,4194304] 0 2024-03-30T04:28:07.254 INFO:tasks.workunit.client.0.smithi123.stdout:6/476: chown d10/d20/c35 81 1 2024-03-30T04:28:07.260 INFO:tasks.workunit.client.0.smithi123.stdout:9/370: dwrite d6/f66 [4194304,4194304] 0 2024-03-30T04:28:07.262 INFO:tasks.workunit.client.0.smithi123.stdout:6/477: chown d10/d13/d1c/f2c 477176342 1 2024-03-30T04:28:07.266 INFO:tasks.workunit.client.0.smithi123.stdout:6/478: readlink d10/d13/d6a/d7a/l9b 0 2024-03-30T04:28:07.266 INFO:tasks.workunit.client.0.smithi123.stdout:2/347: rename d0/d1f/d1c/d20/c61 to d0/d1f/d30/c6b 0 2024-03-30T04:28:07.269 INFO:tasks.workunit.client.0.smithi123.stdout:1/477: mkdir db/dd/d5f/d6d/d85/d89/d90/d94/d9c 0 2024-03-30T04:28:07.280 INFO:tasks.workunit.client.0.smithi123.stdout:5/369: dwrite dd/d19/d1f/d35/f4b [0,4194304] 0 2024-03-30T04:28:07.283 INFO:tasks.workunit.client.0.smithi123.stdout:7/365: dread d2/da/d2d/f3d [0,4194304] 0 2024-03-30T04:28:07.288 INFO:tasks.workunit.client.0.smithi123.stdout:2/348: getdents d0/d1f/d1c/d39/d4b/d4c/d5f 0 2024-03-30T04:28:07.296 INFO:tasks.workunit.client.0.smithi123.stdout:7/366: symlink d2/dc/d49/d68/l7b 0 2024-03-30T04:28:07.300 INFO:tasks.workunit.client.0.smithi123.stdout:9/371: dwrite d6/d23/f46 [0,4194304] 0 2024-03-30T04:28:07.301 INFO:tasks.workunit.client.0.smithi123.stdout:2/349: symlink d0/d1f/d1c/d39/d4b/l6c 0 2024-03-30T04:28:07.301 INFO:tasks.workunit.client.0.smithi123.stdout:5/370: write dd/d19/d1f/f67 [789566,61387] 0 2024-03-30T04:28:07.308 INFO:tasks.workunit.client.0.smithi123.stdout:8/321: dwrite da/dc/f20 [8388608,4194304] 0 2024-03-30T04:28:07.308 INFO:tasks.workunit.client.0.smithi123.stdout:6/479: dwrite d10/d13/d1c/d42/f80 [0,4194304] 0 2024-03-30T04:28:07.309 INFO:tasks.workunit.client.0.smithi123.stdout:5/371: fsync dd/d19/d1f/d35/f38 0 2024-03-30T04:28:07.311 INFO:tasks.workunit.client.0.smithi123.stdout:2/350: creat d0/f6d x:0 0 0 2024-03-30T04:28:07.316 INFO:tasks.workunit.client.0.smithi123.stdout:1/478: dread db/dd/d42/d46/d56/d5c/f76 [0,4194304] 0 2024-03-30T04:28:07.317 INFO:tasks.workunit.client.0.smithi123.stdout:8/322: creat da/d27/d3c/d2d/f57 x:0 0 0 2024-03-30T04:28:07.318 INFO:tasks.workunit.client.0.smithi123.stdout:5/372: creat dd/d19/d3e/d5b/d62/f7a x:0 0 0 2024-03-30T04:28:07.320 INFO:tasks.workunit.client.0.smithi123.stdout:9/372: unlink d6/d1f/d45/c52 0 2024-03-30T04:28:07.320 INFO:tasks.workunit.client.0.smithi123.stdout:6/480: write d10/d13/d1c/d42/f5f [47012,43443] 0 2024-03-30T04:28:07.324 INFO:tasks.workunit.client.0.smithi123.stdout:4/370: dread d2/d4/ff [0,4194304] 0 2024-03-30T04:28:07.328 INFO:tasks.workunit.client.0.smithi123.stdout:5/373: write dd/d19/d1f/d68/f73 [766862,5583] 0 2024-03-30T04:28:07.330 INFO:tasks.workunit.client.0.smithi123.stdout:6/481: unlink d10/d13/d1c/d42/f5f 0 2024-03-30T04:28:07.331 INFO:tasks.workunit.client.0.smithi123.stdout:9/373: link d6/d23/f2b d6/d1f/d72/f8b 0 2024-03-30T04:28:07.339 INFO:tasks.workunit.client.0.smithi123.stdout:9/374: symlink d6/d7c/d82/l8c 0 2024-03-30T04:28:07.339 INFO:tasks.workunit.client.0.smithi123.stdout:2/351: truncate d0/d1f/d1c/f51 4134370 0 2024-03-30T04:28:07.343 INFO:tasks.workunit.client.0.smithi123.stdout:1/479: dwrite db/f59 [0,4194304] 0 2024-03-30T04:28:07.347 INFO:tasks.workunit.client.0.smithi123.stdout:9/375: truncate d6/d7c/f89 379007 0 2024-03-30T04:28:07.352 INFO:tasks.workunit.client.0.smithi123.stdout:9/376: chown d6/d1f/d4d/c6f 57794886 1 2024-03-30T04:28:07.352 INFO:tasks.workunit.client.0.smithi123.stdout:2/352: dread d0/d1f/f2d [0,4194304] 0 2024-03-30T04:28:07.354 INFO:tasks.workunit.client.0.smithi123.stdout:1/480: mknod db/dd/d42/d46/d51/c9d 0 2024-03-30T04:28:07.356 INFO:tasks.workunit.client.0.smithi123.stdout:4/371: dwrite d2/d4/d2c/d42/f4a [0,4194304] 0 2024-03-30T04:28:07.361 INFO:tasks.workunit.client.0.smithi123.stdout:5/374: truncate dd/d19/d1f/d35/d3a/d57/f65 3206644 0 2024-03-30T04:28:07.362 INFO:tasks.workunit.client.0.smithi123.stdout:6/482: dread d10/f1b [0,4194304] 0 2024-03-30T04:28:07.367 INFO:tasks.workunit.client.0.smithi123.stdout:8/323: dwrite da/dc/d14/d18/d21/f45 [0,4194304] 0 2024-03-30T04:28:07.368 INFO:tasks.workunit.client.0.smithi123.stdout:5/375: symlink dd/d19/l7b 0 2024-03-30T04:28:07.370 INFO:tasks.workunit.client.0.smithi123.stdout:6/483: creat d10/d13/d6a/d7a/fa1 x:0 0 0 2024-03-30T04:28:07.371 INFO:tasks.workunit.client.0.smithi123.stdout:9/377: link d6/f15 d6/d23/f8d 0 2024-03-30T04:28:07.374 INFO:tasks.workunit.client.0.smithi123.stdout:8/324: symlink da/dc/d14/d18/d21/l58 0 2024-03-30T04:28:07.377 INFO:tasks.workunit.client.0.smithi123.stdout:2/353: rename d0/d1f/d1c/d39/d44/f55 to d0/d1f/f6e 0 2024-03-30T04:28:07.377 INFO:tasks.workunit.client.0.smithi123.stdout:9/378: write d6/d1f/d4d/f5b [329699,95225] 0 2024-03-30T04:28:07.379 INFO:tasks.workunit.client.0.smithi123.stdout:6/484: truncate d10/d13/d6a/d7a/fa1 595010 0 2024-03-30T04:28:07.381 INFO:tasks.workunit.client.0.smithi123.stdout:2/354: fsync d0/d1f/d16/f1e 0 2024-03-30T04:28:07.381 INFO:tasks.workunit.client.0.smithi123.stdout:4/372: link d2/d4/d7/f4b d2/d4/d2c/d42/d35/d5d/d78/f83 0 2024-03-30T04:28:07.382 INFO:tasks.workunit.client.0.smithi123.stdout:6/485: chown d10/d13/f8b 44593 1 2024-03-30T04:28:07.382 INFO:tasks.workunit.client.0.smithi123.stdout:8/325: dread da/dc/f32 [0,4194304] 0 2024-03-30T04:28:07.385 INFO:tasks.workunit.client.0.smithi123.stdout:9/379: mknod d6/d49/c8e 0 2024-03-30T04:28:07.385 INFO:tasks.workunit.client.0.smithi123.stdout:1/481: dwrite db/dd/d42/d46/d5d/d62/f6b [0,4194304] 0 2024-03-30T04:28:07.388 INFO:tasks.workunit.client.0.smithi123.stdout:6/486: write d10/d13/f8b [300477,4542] 0 2024-03-30T04:28:07.388 INFO:tasks.workunit.client.0.smithi123.stdout:8/326: chown da/d27/f3b 1769445 1 2024-03-30T04:28:07.391 INFO:tasks.workunit.client.0.smithi123.stdout:2/355: rename d0/f6d to d0/d1f/d5e/f6f 0 2024-03-30T04:28:07.393 INFO:tasks.workunit.client.0.smithi123.stdout:6/487: mkdir d10/d20/da2 0 2024-03-30T04:28:07.396 INFO:tasks.workunit.client.0.smithi123.stdout:1/482: mkdir db/dd/d42/d46/d5d/d62/d9e 0 2024-03-30T04:28:07.398 INFO:tasks.workunit.client.0.smithi123.stdout:8/327: rename da/d27/c2c to da/dc/c59 0 2024-03-30T04:28:07.400 INFO:tasks.workunit.client.0.smithi123.stdout:6/488: mkdir d10/d20/d3f/da3 0 2024-03-30T04:28:07.401 INFO:tasks.workunit.client.0.smithi123.stdout:1/483: readlink db/dd/d42/d46/l95 0 2024-03-30T04:28:07.405 INFO:tasks.workunit.client.0.smithi123.stdout:9/380: dwrite d6/f7e [0,4194304] 0 2024-03-30T04:28:07.406 INFO:tasks.workunit.client.0.smithi123.stdout:6/489: write d10/d13/d6a/d7a/fa1 [1283760,127714] 0 2024-03-30T04:28:07.407 INFO:tasks.workunit.client.0.smithi123.stdout:4/373: rename d2/d4/d7/d1a/f21 to d2/d4/d2c/f84 0 2024-03-30T04:28:07.410 INFO:tasks.workunit.client.0.smithi123.stdout:1/484: unlink db/dd/d42/f9a 0 2024-03-30T04:28:07.412 INFO:tasks.workunit.client.0.smithi123.stdout:2/356: getdents d0/d1f/d1c/d20 0 2024-03-30T04:28:07.416 INFO:tasks.workunit.client.0.smithi123.stdout:1/485: readlink db/dd/d42/d46/l6a 0 2024-03-30T04:28:07.417 INFO:tasks.workunit.client.0.smithi123.stdout:8/328: dwrite da/d27/f3b [0,4194304] 0 2024-03-30T04:28:07.423 INFO:tasks.workunit.client.0.smithi123.stdout:6/490: creat d10/d13/d1c/fa4 x:0 0 0 2024-03-30T04:28:07.429 INFO:tasks.workunit.client.0.smithi123.stdout:6/491: dread d10/d20/f2b [0,4194304] 0 2024-03-30T04:28:07.434 INFO:tasks.workunit.client.0.smithi123.stdout:6/492: fsync d10/d20/d3f/f4d 0 2024-03-30T04:28:07.438 INFO:tasks.workunit.client.0.smithi123.stdout:6/493: unlink d10/d13/f25 0 2024-03-30T04:28:07.442 INFO:tasks.workunit.client.0.smithi123.stdout:1/486: dwrite db/dd/d5f/f87 [0,4194304] 0 2024-03-30T04:28:07.442 INFO:tasks.workunit.client.0.smithi123.stdout:6/494: read d10/d20/d3f/f65 [1890589,93507] 0 2024-03-30T04:28:07.445 INFO:tasks.workunit.client.0.smithi123.stdout:1/487: read - db/dd/d42/d46/d39/f93 zero size 2024-03-30T04:28:07.447 INFO:tasks.workunit.client.0.smithi123.stdout:4/374: creat d2/d4/d2c/d42/d35/d5d/d78/f85 x:0 0 0 2024-03-30T04:28:07.448 INFO:tasks.workunit.client.0.smithi123.stdout:6/495: symlink d10/d13/d6a/la5 0 2024-03-30T04:28:07.461 INFO:tasks.workunit.client.0.smithi123.stdout:6/496: link d10/d13/d6a/c86 d10/d82/d9a/ca6 0 2024-03-30T04:28:07.462 INFO:tasks.workunit.client.0.smithi123.stdout:8/329: dwrite da/ff [0,4194304] 0 2024-03-30T04:28:07.465 INFO:tasks.workunit.client.0.smithi123.stdout:8/330: fsync da/dc/f13 0 2024-03-30T04:28:07.466 INFO:tasks.workunit.client.0.smithi123.stdout:2/357: dwrite d0/d1f/d16/f18 [0,4194304] 0 2024-03-30T04:28:07.469 INFO:tasks.workunit.client.0.smithi123.stdout:6/497: write d10/f11 [1387125,8775] 0 2024-03-30T04:28:07.469 INFO:tasks.workunit.client.0.smithi123.stdout:1/488: dwrite db/dd/d42/f98 [0,4194304] 0 2024-03-30T04:28:07.472 INFO:tasks.workunit.client.0.smithi123.stdout:2/358: stat d0/d1f/f1b 0 2024-03-30T04:28:07.476 INFO:tasks.workunit.client.0.smithi123.stdout:8/331: chown da/d27/d3c/d2d/l4a 78 1 2024-03-30T04:28:07.476 INFO:tasks.workunit.client.0.smithi123.stdout:4/375: dwrite d2/d4/d2c/d42/f3a [0,4194304] 0 2024-03-30T04:28:07.477 INFO:tasks.workunit.client.0.smithi123.stdout:6/498: creat d10/d82/fa7 x:0 0 0 2024-03-30T04:28:07.479 INFO:tasks.workunit.client.0.smithi123.stdout:2/359: mkdir d0/d1f/d1c/d39/d4b/d70 0 2024-03-30T04:28:07.481 INFO:tasks.workunit.client.0.smithi123.stdout:2/360: dread - d0/d1f/f6e zero size 2024-03-30T04:28:07.484 INFO:tasks.workunit.client.0.smithi123.stdout:8/332: symlink da/d27/d3c/d2d/l5a 0 2024-03-30T04:28:07.485 INFO:tasks.workunit.client.0.smithi123.stdout:2/361: mkdir d0/d1f/d5e/d71 0 2024-03-30T04:28:07.486 INFO:tasks.workunit.client.0.smithi123.stdout:1/489: getdents db/dd/d5f/d6d 0 2024-03-30T04:28:07.487 INFO:tasks.workunit.client.0.smithi123.stdout:4/376: write d2/d4/fa [4209126,71464] 0 2024-03-30T04:28:07.489 INFO:tasks.workunit.client.0.smithi123.stdout:2/362: read - d0/d1f/d16/f3c zero size 2024-03-30T04:28:07.490 INFO:tasks.workunit.client.0.smithi123.stdout:8/333: write da/dc/d14/d18/d21/f40 [37674,75926] 0 2024-03-30T04:28:07.493 INFO:tasks.workunit.client.0.smithi123.stdout:4/377: creat d2/f86 x:0 0 0 2024-03-30T04:28:07.494 INFO:tasks.workunit.client.0.smithi123.stdout:8/334: symlink da/dc/l5b 0 2024-03-30T04:28:07.494 INFO:tasks.workunit.client.0.smithi123.stdout:4/378: chown d2/d4 23430 1 2024-03-30T04:28:07.495 INFO:tasks.workunit.client.0.smithi123.stdout:6/499: dread d10/f1b [0,4194304] 0 2024-03-30T04:28:07.495 INFO:tasks.workunit.client.0.smithi123.stdout:8/335: read da/dc/f52 [1959813,4583] 0 2024-03-30T04:28:07.498 INFO:tasks.workunit.client.0.smithi123.stdout:0/254: sync 2024-03-30T04:28:07.500 INFO:tasks.workunit.client.0.smithi123.stdout:4/379: fsync d2/d4/f8 0 2024-03-30T04:28:07.505 INFO:tasks.workunit.client.0.smithi123.stdout:6/500: creat d10/d13/d1c/d5b/fa8 x:0 0 0 2024-03-30T04:28:07.505 INFO:tasks.workunit.client.0.smithi123.stdout:8/336: mkdir da/d27/d3c/d2d/d5c 0 2024-03-30T04:28:07.507 INFO:tasks.workunit.client.0.smithi123.stdout:2/363: rename d0/d1f/d1c/d39/f42 to d0/d1f/d30/f72 0 2024-03-30T04:28:07.512 INFO:tasks.workunit.client.0.smithi123.stdout:2/364: stat d0/d1f/f17 0 2024-03-30T04:28:07.513 INFO:tasks.workunit.client.0.smithi123.stdout:4/380: mkdir d2/d4/d2c/d42/d80/d87 0 2024-03-30T04:28:07.513 INFO:tasks.workunit.client.0.smithi123.stdout:0/255: truncate d6/dd/d18/d1f/f27 1549466 0 2024-03-30T04:28:07.515 INFO:tasks.workunit.client.0.smithi123.stdout:1/490: dwrite db/d1b/f1d [4194304,4194304] 0 2024-03-30T04:28:07.522 INFO:tasks.workunit.client.0.smithi123.stdout:1/491: mknod db/dd/d5f/d6d/c9f 0 2024-03-30T04:28:07.523 INFO:tasks.workunit.client.0.smithi123.stdout:2/365: symlink d0/d1f/d1c/d39/d4b/d4c/d5f/l73 0 2024-03-30T04:28:07.529 INFO:tasks.workunit.client.0.smithi123.stdout:4/381: dread d2/d4/d7/d1a/f5e [0,4194304] 0 2024-03-30T04:28:07.529 INFO:tasks.workunit.client.0.smithi123.stdout:2/366: creat d0/d37/f74 x:0 0 0 2024-03-30T04:28:07.530 INFO:tasks.workunit.client.0.smithi123.stdout:0/256: dread d6/dd/f51 [0,4194304] 0 2024-03-30T04:28:07.532 INFO:tasks.workunit.client.0.smithi123.stdout:1/492: link db/f60 db/dd/d42/d8f/d78/fa0 0 2024-03-30T04:28:07.534 INFO:tasks.workunit.client.0.smithi123.stdout:4/382: stat d2/d4/d7/fe 0 2024-03-30T04:28:07.535 INFO:tasks.workunit.client.0.smithi123.stdout:8/337: dwrite da/f4e [0,4194304] 0 2024-03-30T04:28:07.536 INFO:tasks.workunit.client.0.smithi123.stdout:2/367: creat d0/d1f/d1c/d36/f75 x:0 0 0 2024-03-30T04:28:07.543 INFO:tasks.workunit.client.0.smithi123.stdout:0/257: link d6/dd/d18/d1f/c3b d6/dd/d31/d56/c57 0 2024-03-30T04:28:07.544 INFO:tasks.workunit.client.0.smithi123.stdout:2/368: getdents d0/d1f/d27/d68 0 2024-03-30T04:28:07.545 INFO:tasks.workunit.client.0.smithi123.stdout:4/383: dread d2/d4/d7/d63/f82 [0,4194304] 0 2024-03-30T04:28:07.545 INFO:tasks.workunit.client.0.smithi123.stdout:0/258: stat d6/dd/d18/d2f 0 2024-03-30T04:28:07.548 INFO:tasks.workunit.client.0.smithi123.stdout:2/369: write d0/d1f/d1c/f5b [1186379,46929] 0 2024-03-30T04:28:07.548 INFO:tasks.workunit.client.0.smithi123.stdout:4/384: creat d2/d34/f88 x:0 0 0 2024-03-30T04:28:07.549 INFO:tasks.workunit.client.0.smithi123.stdout:0/259: read - d6/dd/d31/d35/f43 zero size 2024-03-30T04:28:07.553 INFO:tasks.workunit.client.0.smithi123.stdout:4/385: mkdir d2/d4/d2c/d42/d89 0 2024-03-30T04:28:07.553 INFO:tasks.workunit.client.0.smithi123.stdout:6/501: dread d10/f68 [0,4194304] 0 2024-03-30T04:28:07.555 INFO:tasks.workunit.client.0.smithi123.stdout:0/260: unlink d6/dd/d18/d1f/f4c 0 2024-03-30T04:28:07.555 INFO:tasks.workunit.client.0.smithi123.stdout:8/338: dread da/dc/f20 [8388608,4194304] 0 2024-03-30T04:28:07.555 INFO:tasks.workunit.client.0.smithi123.stdout:1/493: truncate db/dd/d42/d46/d39/f52 2878253 0 2024-03-30T04:28:07.558 INFO:tasks.workunit.client.0.smithi123.stdout:8/339: chown da/d27/d3c/d2d/d5c 22 1 2024-03-30T04:28:07.559 INFO:tasks.workunit.client.0.smithi123.stdout:0/261: write d6/dd/d31/d35/f45 [12854,64820] 0 2024-03-30T04:28:07.559 INFO:tasks.workunit.client.0.smithi123.stdout:4/386: creat d2/d4/d2c/d42/d35/d5d/d6f/f8a x:0 0 0 2024-03-30T04:28:07.560 INFO:tasks.workunit.client.0.smithi123.stdout:1/494: rmdir db/dd/d42/d46/d39 39 2024-03-30T04:28:07.566 INFO:tasks.workunit.client.0.smithi123.stdout:0/262: unlink d6/dd/d18/d1f/f27 0 2024-03-30T04:28:07.568 INFO:tasks.workunit.client.0.smithi123.stdout:8/340: dread da/dc/d14/d18/d21/f40 [0,4194304] 0 2024-03-30T04:28:07.571 INFO:tasks.workunit.client.0.smithi123.stdout:0/263: creat d6/dd/d18/d2f/f58 x:0 0 0 2024-03-30T04:28:07.572 INFO:tasks.workunit.client.0.smithi123.stdout:1/495: getdents db/dd/d42/d46/d5d/d7a 0 2024-03-30T04:28:07.574 INFO:tasks.workunit.client.0.smithi123.stdout:6/502: dwrite d10/d13/d1c/d42/f80 [0,4194304] 0 2024-03-30T04:28:07.575 INFO:tasks.workunit.client.0.smithi123.stdout:0/264: write d6/dd/d18/d2f/f58 [431774,25595] 0 2024-03-30T04:28:07.579 INFO:tasks.workunit.client.0.smithi123.stdout:6/503: truncate d10/f1d 5057791 0 2024-03-30T04:28:07.581 INFO:tasks.workunit.client.0.smithi123.stdout:8/341: getdents da 0 2024-03-30T04:28:07.584 INFO:tasks.workunit.client.0.smithi123.stdout:0/265: rename d6/dd/d31/d56 to d6/dd/d15/d28/d59 0 2024-03-30T04:28:07.584 INFO:tasks.workunit.client.0.smithi123.stdout:8/342: fsync da/d27/f53 0 2024-03-30T04:28:07.590 INFO:tasks.workunit.client.0.smithi123.stdout:6/504: rename d10/d13/d6a/d7a/c7e to d10/d13/d6a/ca9 0 2024-03-30T04:28:07.590 INFO:tasks.workunit.client.0.smithi123.stdout:4/387: dwrite d2/d4/d7/d1a/f30 [8388608,4194304] 0 2024-03-30T04:28:07.594 INFO:tasks.workunit.client.0.smithi123.stdout:0/266: rename d6/dd/d15/f29 to d6/dd/d15/d28/f5a 0 2024-03-30T04:28:07.596 INFO:tasks.workunit.client.0.smithi123.stdout:4/388: write d2/d34/f4c [162717,47601] 0 2024-03-30T04:28:07.605 INFO:tasks.workunit.client.0.smithi123.stdout:4/389: unlink d2/d4/d2c/l74 0 2024-03-30T04:28:07.610 INFO:tasks.workunit.client.0.smithi123.stdout:6/505: dwrite d10/d13/d45/f67 [0,4194304] 0 2024-03-30T04:28:07.615 INFO:tasks.workunit.client.0.smithi123.stdout:0/267: truncate d6/dd/f36 105430 0 2024-03-30T04:28:07.616 INFO:tasks.workunit.client.0.smithi123.stdout:4/390: creat d2/d4/d2c/d42/f8b x:0 0 0 2024-03-30T04:28:07.619 INFO:tasks.workunit.client.0.smithi123.stdout:0/268: dread - d6/dd/d15/f25 zero size 2024-03-30T04:28:07.622 INFO:tasks.workunit.client.0.smithi123.stdout:4/391: symlink d2/d4/d2c/d42/l8c 0 2024-03-30T04:28:07.623 INFO:tasks.workunit.client.0.smithi123.stdout:1/496: dread db/dd/d42/d8f/f73 [0,4194304] 0 2024-03-30T04:28:07.623 INFO:tasks.workunit.client.0.smithi123.stdout:0/269: chown d6/dd/d18/d2f/c48 71408500 1 2024-03-30T04:28:07.624 INFO:tasks.workunit.client.0.smithi123.stdout:8/343: dread da/dc/d14/d18/f4f [4194304,4194304] 0 2024-03-30T04:28:07.624 INFO:tasks.workunit.client.0.smithi123.stdout:1/497: readlink l4 0 2024-03-30T04:28:07.632 INFO:tasks.workunit.client.0.smithi123.stdout:8/344: rename da/dc/l5b to da/d27/l5d 0 2024-03-30T04:28:07.634 INFO:tasks.workunit.client.0.smithi123.stdout:1/498: mknod db/dd/d5f/d6d/d7e/ca1 0 2024-03-30T04:28:07.634 INFO:tasks.workunit.client.0.smithi123.stdout:8/345: stat da/dc/d14 0 2024-03-30T04:28:07.635 INFO:tasks.workunit.client.0.smithi123.stdout:6/506: dwrite d10/d13/d1c/d42/f80 [4194304,4194304] 0 2024-03-30T04:28:07.635 INFO:tasks.workunit.client.0.smithi123.stdout:0/270: rename d6/dd/d15/f25 to d6/dd/d31/d35/f5b 0 2024-03-30T04:28:07.638 INFO:tasks.workunit.client.0.smithi123.stdout:6/507: read - d10/d13/d1c/f71 zero size 2024-03-30T04:28:07.642 INFO:tasks.workunit.client.0.smithi123.stdout:1/499: mkdir db/dd/d42/d46/d56/da2 0 2024-03-30T04:28:07.642 INFO:tasks.workunit.client.0.smithi123.stdout:8/346: creat da/d27/d3c/f5e x:0 0 0 2024-03-30T04:28:07.643 INFO:tasks.workunit.client.0.smithi123.stdout:0/271: creat d6/dd/d15/d28/f5c x:0 0 0 2024-03-30T04:28:07.646 INFO:tasks.workunit.client.0.smithi123.stdout:8/347: chown da/d27/d3c/f5e 92627528 1 2024-03-30T04:28:07.652 INFO:tasks.workunit.client.0.smithi123.stdout:6/508: link d10/d13/d1c/d42/f80 d10/d13/d45/faa 0 2024-03-30T04:28:07.656 INFO:tasks.workunit.client.0.smithi123.stdout:1/500: creat db/dd/d5f/d6d/fa3 x:0 0 0 2024-03-30T04:28:07.658 INFO:tasks.workunit.client.0.smithi123.stdout:4/392: dwrite d2/d4/d7/fe [4194304,4194304] 0 2024-03-30T04:28:07.658 INFO:tasks.workunit.client.0.smithi123.stdout:3/423: sync 2024-03-30T04:28:07.658 INFO:tasks.workunit.client.0.smithi123.stdout:7/367: sync 2024-03-30T04:28:07.666 INFO:tasks.workunit.client.0.smithi123.stdout:1/501: write db/dd/d42/d46/f79 [4735179,30947] 0 2024-03-30T04:28:07.671 INFO:tasks.workunit.client.0.smithi123.stdout:4/393: write d2/d4/d7/d1a/f1f [478091,14764] 0 2024-03-30T04:28:07.672 INFO:tasks.workunit.client.0.smithi123.stdout:3/424: getdents d13/d24/d31/d23/d3a/d3c 0 2024-03-30T04:28:07.674 INFO:tasks.workunit.client.0.smithi123.stdout:1/502: mknod db/dd/d5f/d6d/d7e/ca4 0 2024-03-30T04:28:07.678 INFO:tasks.workunit.client.0.smithi123.stdout:0/272: dwrite d6/dd/d15/d28/f3d [0,4194304] 0 2024-03-30T04:28:07.681 INFO:tasks.workunit.client.0.smithi123.stdout:1/503: write db/d1b/f1d [3432661,44541] 0 2024-03-30T04:28:07.688 INFO:tasks.workunit.client.0.smithi123.stdout:6/509: dwrite d10/d13/d1c/d5b/f85 [0,4194304] 0 2024-03-30T04:28:07.689 INFO:tasks.workunit.client.0.smithi123.stdout:4/394: creat d2/d4/d2c/d42/d89/f8d x:0 0 0 2024-03-30T04:28:07.690 INFO:tasks.workunit.client.0.smithi123.stdout:0/273: mkdir d6/dd/d16/d32/d5d 0 2024-03-30T04:28:07.693 INFO:tasks.workunit.client.0.smithi123.stdout:8/348: dwrite da/d27/f3b [0,4194304] 0 2024-03-30T04:28:07.696 INFO:tasks.workunit.client.0.smithi123.stdout:0/274: write d6/f24 [3082758,42620] 0 2024-03-30T04:28:07.708 INFO:tasks.workunit.client.0.smithi123.stdout:4/395: symlink d2/d4/d2c/d42/d35/d5d/l8e 0 2024-03-30T04:28:07.709 INFO:tasks.workunit.client.0.smithi123.stdout:8/349: dread da/dc/f10 [0,4194304] 0 2024-03-30T04:28:07.711 INFO:tasks.workunit.client.0.smithi123.stdout:4/396: write d2/d4/ff [3702912,69578] 0 2024-03-30T04:28:07.711 INFO:tasks.workunit.client.0.smithi123.stdout:1/504: link db/dd/d42/d8f/f73 db/dd/d5f/d6d/d85/d89/fa5 0 2024-03-30T04:28:07.718 INFO:tasks.workunit.client.0.smithi123.stdout:7/368: dread d2/dc/d1c/f1d [0,4194304] 0 2024-03-30T04:28:07.721 INFO:tasks.workunit.client.0.smithi123.stdout:4/397: mknod d2/d4/d2c/d42/d35/d5d/c8f 0 2024-03-30T04:28:07.722 INFO:tasks.workunit.client.0.smithi123.stdout:0/275: dread d6/dd/d15/d28/f3d [0,4194304] 0 2024-03-30T04:28:07.722 INFO:tasks.workunit.client.0.smithi123.stdout:7/369: write d2/f8 [518331,75465] 0 2024-03-30T04:28:07.723 INFO:tasks.workunit.client.0.smithi123.stdout:3/425: dread d13/d24/d31/d23/d3a/f40 [0,4194304] 0 2024-03-30T04:28:07.730 INFO:tasks.workunit.client.0.smithi123.stdout:0/276: fsync d6/dd/d31/d35/f42 0 2024-03-30T04:28:07.730 INFO:tasks.workunit.client.0.smithi123.stdout:1/505: read db/dd/d42/d46/d56/d5c/f97 [313851,82455] 0 2024-03-30T04:28:07.734 INFO:tasks.workunit.client.0.smithi123.stdout:1/506: chown db/fe 1491712 1 2024-03-30T04:28:07.744 INFO:tasks.workunit.client.0.smithi123.stdout:6/510: dread d10/d13/d1c/f2a [0,4194304] 0 2024-03-30T04:28:07.747 INFO:tasks.workunit.client.0.smithi123.stdout:5/376: sync 2024-03-30T04:28:07.749 INFO:tasks.workunit.client.0.smithi123.stdout:7/370: unlink d2/da/d22/d3b/f74 0 2024-03-30T04:28:07.755 INFO:tasks.workunit.client.0.smithi123.stdout:8/350: truncate da/dc/f10 811334 0 2024-03-30T04:28:07.757 INFO:tasks.workunit.client.0.smithi123.stdout:1/507: creat db/dd/d42/d46/d51/d72/fa6 x:0 0 0 2024-03-30T04:28:07.759 INFO:tasks.workunit.client.0.smithi123.stdout:7/371: rmdir d2/da/d11/d34 39 2024-03-30T04:28:07.761 INFO:tasks.workunit.client.0.smithi123.stdout:0/277: mknod d6/d52/c5e 0 2024-03-30T04:28:07.763 INFO:tasks.workunit.client.0.smithi123.stdout:8/351: mknod da/c5f 0 2024-03-30T04:28:07.764 INFO:tasks.workunit.client.0.smithi123.stdout:6/511: creat d10/d20/da2/fab x:0 0 0 2024-03-30T04:28:07.766 INFO:tasks.workunit.client.0.smithi123.stdout:6/512: dread - d10/d13/d1c/fa4 zero size 2024-03-30T04:28:07.770 INFO:tasks.workunit.client.0.smithi123.stdout:4/398: dread d2/d4/d2c/f84 [0,4194304] 0 2024-03-30T04:28:07.772 INFO:tasks.workunit.client.0.smithi123.stdout:3/426: dwrite d13/d24/d31/d23/d3a/d3c/d66/f7f [4194304,4194304] 0 2024-03-30T04:28:07.784 INFO:tasks.workunit.client.0.smithi123.stdout:7/372: creat d2/dc/d1c/f7c x:0 0 0 2024-03-30T04:28:07.795 INFO:tasks.workunit.client.0.smithi123.stdout:1/508: dwrite db/dd/d5f/d6d/f81 [0,4194304] 0 2024-03-30T04:28:07.797 INFO:tasks.workunit.client.0.smithi123.stdout:0/278: dwrite d6/dd/d15/d28/f3d [0,4194304] 0 2024-03-30T04:28:07.800 INFO:tasks.workunit.client.0.smithi123.stdout:9/381: sync 2024-03-30T04:28:07.801 INFO:tasks.workunit.client.0.smithi123.stdout:2/370: sync 2024-03-30T04:28:07.808 INFO:tasks.workunit.client.0.smithi123.stdout:4/399: dwrite d2/d4/fa [0,4194304] 0 2024-03-30T04:28:07.808 INFO:tasks.workunit.client.0.smithi123.stdout:5/377: creat dd/f7c x:0 0 0 2024-03-30T04:28:07.814 INFO:tasks.workunit.client.0.smithi123.stdout:0/279: truncate d6/dd/d18/d2f/f58 1349896 0 2024-03-30T04:28:07.814 INFO:tasks.workunit.client.0.smithi123.stdout:6/513: rename d10/d13/d1c to d10/d20/dac 0 2024-03-30T04:28:07.817 INFO:tasks.workunit.client.0.smithi123.stdout:4/400: creat d2/d34/f90 x:0 0 0 2024-03-30T04:28:07.820 INFO:tasks.workunit.client.0.smithi123.stdout:7/373: truncate d2/da/d11/f20 3905426 0 2024-03-30T04:28:07.820 INFO:tasks.workunit.client.0.smithi123.stdout:1/509: mkdir db/dd/d5f/d6d/d85/d89/da7 0 2024-03-30T04:28:07.824 INFO:tasks.workunit.client.0.smithi123.stdout:8/352: stat da/dc/d14/d18/f1e 0 2024-03-30T04:28:07.824 INFO:tasks.workunit.client.0.smithi123.stdout:9/382: creat d6/d1f/d72/f8f x:0 0 0 2024-03-30T04:28:07.826 INFO:tasks.workunit.client.0.smithi123.stdout:5/378: write dd/d19/d3e/d5b/f69 [167551,32080] 0 2024-03-30T04:28:07.826 INFO:tasks.workunit.client.0.smithi123.stdout:4/401: write d2/d34/f4c [1399284,13835] 0 2024-03-30T04:28:07.826 INFO:tasks.workunit.client.0.smithi123.stdout:2/371: symlink d0/d1f/d5e/l76 0 2024-03-30T04:28:07.828 INFO:tasks.workunit.client.0.smithi123.stdout:7/374: dread - d2/da/d11/d61/f72 zero size 2024-03-30T04:28:07.830 INFO:tasks.workunit.client.0.smithi123.stdout:2/372: dread - d0/d1f/d16/f3c zero size 2024-03-30T04:28:07.832 INFO:tasks.workunit.client.0.smithi123.stdout:2/373: fdatasync d0/d1f/f24 0 2024-03-30T04:28:07.833 INFO:tasks.workunit.client.0.smithi123.stdout:6/514: creat d10/d13/d45/d77/fad x:0 0 0 2024-03-30T04:28:07.836 INFO:tasks.workunit.client.0.smithi123.stdout:4/402: fdatasync d2/d4/d7/d1a/f5e 0 2024-03-30T04:28:07.839 INFO:tasks.workunit.client.0.smithi123.stdout:3/427: dread d13/d24/d31/f4b [0,4194304] 0 2024-03-30T04:28:07.840 INFO:tasks.workunit.client.0.smithi123.stdout:8/353: mknod da/d27/d56/c60 0 2024-03-30T04:28:07.843 INFO:tasks.workunit.client.0.smithi123.stdout:0/280: getdents d6/dd/d18/d1f 0 2024-03-30T04:28:07.843 INFO:tasks.workunit.client.0.smithi123.stdout:9/383: unlink d6/d1f/c3d 0 2024-03-30T04:28:07.845 INFO:tasks.workunit.client.0.smithi123.stdout:7/375: creat d2/da/d11/d34/f7d x:0 0 0 2024-03-30T04:28:07.845 INFO:tasks.workunit.client.0.smithi123.stdout:6/515: creat d10/d13/d6a/fae x:0 0 0 2024-03-30T04:28:07.847 INFO:tasks.workunit.client.0.smithi123.stdout:8/354: mknod da/dc/d14/d18/d21/c61 0 2024-03-30T04:28:07.849 INFO:tasks.workunit.client.0.smithi123.stdout:5/379: dread dd/d19/d1f/f27 [0,4194304] 0 2024-03-30T04:28:07.850 INFO:tasks.workunit.client.0.smithi123.stdout:9/384: stat d6/d7c/d82/l8c 0 2024-03-30T04:28:07.852 INFO:tasks.workunit.client.0.smithi123.stdout:6/516: read - d10/d20/dac/d5b/fa8 zero size 2024-03-30T04:28:07.856 INFO:tasks.workunit.client.0.smithi123.stdout:4/403: dread d2/d4/d2c/d42/d35/f5c [0,4194304] 0 2024-03-30T04:28:07.856 INFO:tasks.workunit.client.0.smithi123.stdout:3/428: link d13/d24/d31/d23/f76 d13/d24/d31/d23/d63/f87 0 2024-03-30T04:28:07.859 INFO:tasks.workunit.client.0.smithi123.stdout:2/374: dwrite d0/d1f/d1c/d40/f5a [0,4194304] 0 2024-03-30T04:28:07.865 INFO:tasks.workunit.client.0.smithi123.stdout:8/355: mkdir da/dc/d14/d18/d62 0 2024-03-30T04:28:07.865 INFO:tasks.workunit.client.0.smithi123.stdout:3/429: rmdir d13/d24/d31/d30 39 2024-03-30T04:28:07.867 INFO:tasks.workunit.client.0.smithi123.stdout:5/380: creat dd/f7d x:0 0 0 2024-03-30T04:28:07.870 INFO:tasks.workunit.client.0.smithi123.stdout:3/430: truncate d13/d24/d31/d23/d3a/d3c/f73 270318 0 2024-03-30T04:28:07.871 INFO:tasks.workunit.client.0.smithi123.stdout:2/375: fdatasync d0/d1f/d1c/d40/f43 0 2024-03-30T04:28:07.871 INFO:tasks.workunit.client.0.smithi123.stdout:0/281: link d6/cb d6/dd/d16/d32/c5f 0 2024-03-30T04:28:07.875 INFO:tasks.workunit.client.0.smithi123.stdout:2/376: chown d0/d1f/d5e 204364 1 2024-03-30T04:28:07.876 INFO:tasks.workunit.client.0.smithi123.stdout:9/385: link d6/d1f/f2d d6/d1f/d45/f90 0 2024-03-30T04:28:07.877 INFO:tasks.workunit.client.0.smithi123.stdout:9/386: dread - d6/d49/f75 zero size 2024-03-30T04:28:07.879 INFO:tasks.workunit.client.0.smithi123.stdout:2/377: chown d0/d1f/d1c/d39/d4b/d4c/f6a 399 1 2024-03-30T04:28:07.884 INFO:tasks.workunit.client.0.smithi123.stdout:9/387: truncate d6/d49/f6d 684087 0 2024-03-30T04:28:07.886 INFO:tasks.workunit.client.0.smithi123.stdout:0/282: fsync d6/f40 0 2024-03-30T04:28:07.886 INFO:tasks.workunit.client.0.smithi123.stdout:8/356: symlink da/dc/d14/d18/l63 0 2024-03-30T04:28:07.891 INFO:tasks.workunit.client.0.smithi123.stdout:6/517: getdents d10/d20/dac/d5b 0 2024-03-30T04:28:07.892 INFO:tasks.workunit.client.0.smithi123.stdout:4/404: getdents d2/d4/d2c/d42/d80 0 2024-03-30T04:28:07.895 INFO:tasks.workunit.client.0.smithi123.stdout:9/388: symlink d6/d49/l91 0 2024-03-30T04:28:07.896 INFO:tasks.workunit.client.0.smithi123.stdout:3/431: dread d13/d24/d31/d23/d3a/f40 [0,4194304] 0 2024-03-30T04:28:07.897 INFO:tasks.workunit.client.0.smithi123.stdout:2/378: symlink d0/d1f/d1c/l77 0 2024-03-30T04:28:07.900 INFO:tasks.workunit.client.0.smithi123.stdout:7/376: dwrite d2/da/d11/f20 [4194304,4194304] 0 2024-03-30T04:28:07.900 INFO:tasks.workunit.client.0.smithi123.stdout:9/389: write d6/d49/f65 [266861,33450] 0 2024-03-30T04:28:07.900 INFO:tasks.workunit.client.0.smithi123.stdout:0/283: truncate d6/f12 440883 0 2024-03-30T04:28:07.901 INFO:tasks.workunit.client.0.smithi123.stdout:6/518: mknod d10/d13/d6a/d7a/caf 0 2024-03-30T04:28:07.903 INFO:tasks.workunit.client.0.smithi123.stdout:4/405: mkdir d2/d4/d2c/d42/d80/d87/d91 0 2024-03-30T04:28:07.906 INFO:tasks.workunit.client.0.smithi123.stdout:9/390: mkdir d6/d1f/d92 0 2024-03-30T04:28:07.907 INFO:tasks.workunit.client.0.smithi123.stdout:7/377: write d2/da/d11/f19 [1107364,120291] 0 2024-03-30T04:28:07.908 INFO:tasks.workunit.client.0.smithi123.stdout:6/519: stat d10/d20/dac/d42/f80 0 2024-03-30T04:28:07.911 INFO:tasks.workunit.client.0.smithi123.stdout:7/378: write f0 [3265472,43857] 0 2024-03-30T04:28:07.912 INFO:tasks.workunit.client.0.smithi123.stdout:4/406: rmdir d2/d4/d2c/d42/d35/d5d/d78 39 2024-03-30T04:28:07.916 INFO:tasks.workunit.client.0.smithi123.stdout:7/379: write d2/dc/d1c/f1d [8443631,104735] 0 2024-03-30T04:28:07.918 INFO:tasks.workunit.client.0.smithi123.stdout:7/380: chown d2/da/d11/d61/f72 52160 1 2024-03-30T04:28:07.918 INFO:tasks.workunit.client.0.smithi123.stdout:0/284: rename d6/dd/d16/d32/c39 to d6/dd/d18/d2f/c60 0 2024-03-30T04:28:07.921 INFO:tasks.workunit.client.0.smithi123.stdout:3/432: write d13/d24/d31/d23/d3a/d3c/f78 [899979,26339] 0 2024-03-30T04:28:07.922 INFO:tasks.workunit.client.0.smithi123.stdout:2/379: truncate d0/d1f/d16/f18 3124657 0 2024-03-30T04:28:07.924 INFO:tasks.workunit.client.0.smithi123.stdout:4/407: creat d2/d4/d2c/d42/d35/d5d/f92 x:0 0 0 2024-03-30T04:28:07.925 INFO:tasks.workunit.client.0.smithi123.stdout:7/381: read d2/dc/d56/f60 [800843,42375] 0 2024-03-30T04:28:07.927 INFO:tasks.workunit.client.0.smithi123.stdout:3/433: chown d13/d24/d31/d23/c3b 241 1 2024-03-30T04:28:07.928 INFO:tasks.workunit.client.0.smithi123.stdout:2/380: mkdir d0/d1f/d78 0 2024-03-30T04:28:07.930 INFO:tasks.workunit.client.0.smithi123.stdout:0/285: rename d6/f12 to d6/dd/d15/d28/d59/f61 0 2024-03-30T04:28:07.934 INFO:tasks.workunit.client.0.smithi123.stdout:2/381: write d0/d1f/d1c/d39/d44/f49 [97333,70519] 0 2024-03-30T04:28:07.934 INFO:tasks.workunit.client.0.smithi123.stdout:7/382: mknod d2/da/d11/d61/c7e 0 2024-03-30T04:28:07.945 INFO:tasks.workunit.client.0.smithi123.stdout:7/383: mkdir d2/dc/d7f 0 2024-03-30T04:28:07.946 INFO:tasks.workunit.client.0.smithi123.stdout:0/286: link d6/dd/d16/d32/c5f d6/c62 0 2024-03-30T04:28:07.949 INFO:tasks.workunit.client.0.smithi123.stdout:7/384: chown d2/dc/c53 5 1 2024-03-30T04:28:07.949 INFO:tasks.workunit.client.0.smithi123.stdout:6/520: dread d10/d20/f3b [0,4194304] 0 2024-03-30T04:28:07.954 INFO:tasks.workunit.client.0.smithi123.stdout:4/408: write d2/d4/d2c/d42/d35/d5d/d78/f83 [1443893,107782] 0 2024-03-30T04:28:07.954 INFO:tasks.workunit.client.0.smithi123.stdout:7/385: stat d2/dc/d1c/l31 0 2024-03-30T04:28:07.956 INFO:tasks.workunit.client.0.smithi123.stdout:3/434: dwrite d13/f7c [0,4194304] 0 2024-03-30T04:28:07.958 INFO:tasks.workunit.client.0.smithi123.stdout:4/409: chown d2/d26/l2a 24 1 2024-03-30T04:28:07.963 INFO:tasks.workunit.client.0.smithi123.stdout:0/287: symlink d6/dd/d16/d32/d4e/l63 0 2024-03-30T04:28:07.964 INFO:tasks.workunit.client.0.smithi123.stdout:6/521: symlink d10/lb0 0 2024-03-30T04:28:07.967 INFO:tasks.workunit.client.0.smithi123.stdout:3/435: creat d13/d24/d31/d23/f88 x:0 0 0 2024-03-30T04:28:07.967 INFO:tasks.workunit.client.0.smithi123.stdout:2/382: dwrite d0/d1f/d16/f33 [0,4194304] 0 2024-03-30T04:28:07.970 INFO:tasks.workunit.client.0.smithi123.stdout:0/288: symlink d6/dd/d16/d32/d4e/l64 0 2024-03-30T04:28:07.978 INFO:tasks.workunit.client.0.smithi123.stdout:6/522: rename d10/d20/dac/d5b/f89 to d10/d20/da2/fb1 0 2024-03-30T04:28:07.980 INFO:tasks.workunit.client.0.smithi123.stdout:2/383: mknod d0/d1f/d1c/d39/d4b/d70/c79 0 2024-03-30T04:28:07.980 INFO:tasks.workunit.client.0.smithi123.stdout:0/289: symlink d6/dd/d31/l65 0 2024-03-30T04:28:07.981 INFO:tasks.workunit.client.0.smithi123.stdout:3/436: mknod d13/d24/d31/d30/c89 0 2024-03-30T04:28:07.982 INFO:tasks.workunit.client.0.smithi123.stdout:3/437: fsync d13/f1d 0 2024-03-30T04:28:07.984 INFO:tasks.workunit.client.0.smithi123.stdout:0/290: dread - d6/dd/d15/f2b zero size 2024-03-30T04:28:07.984 INFO:tasks.workunit.client.0.smithi123.stdout:6/523: creat d10/fb2 x:0 0 0 2024-03-30T04:28:07.985 INFO:tasks.workunit.client.0.smithi123.stdout:3/438: dread - d13/f80 zero size 2024-03-30T04:28:07.989 INFO:tasks.workunit.client.0.smithi123.stdout:2/384: rename d0/d1f/d1c/f51 to d0/d1f/d5e/f7a 0 2024-03-30T04:28:07.989 INFO:tasks.workunit.client.0.smithi123.stdout:0/291: rename d6/dd to d6/dd/d16/d66 22 2024-03-30T04:28:07.990 INFO:tasks.workunit.client.0.smithi123.stdout:4/410: dwrite d2/f20 [4194304,4194304] 0 2024-03-30T04:28:07.992 INFO:tasks.workunit.client.0.smithi123.stdout:0/292: rename d6/dd/d15 to d6/dd/d15/d28/d67 22 2024-03-30T04:28:07.993 INFO:tasks.workunit.client.0.smithi123.stdout:2/385: mknod d0/d1f/d1c/d40/c7b 0 2024-03-30T04:28:07.995 INFO:tasks.workunit.client.0.smithi123.stdout:0/293: write d6/dd/d18/f47 [1675213,117827] 0 2024-03-30T04:28:08.003 INFO:tasks.workunit.client.0.smithi123.stdout:0/294: rename d6/dd/d15/d28/d59/f61 to d6/dd/d16/d32/d4e/f68 0 2024-03-30T04:28:08.003 INFO:tasks.workunit.client.0.smithi123.stdout:3/439: rmdir d13/d24/d31/d23 39 2024-03-30T04:28:08.012 INFO:tasks.workunit.client.0.smithi123.stdout:4/411: truncate d2/d4/d7/f4b 6698153 0 2024-03-30T04:28:08.012 INFO:tasks.workunit.client.0.smithi123.stdout:3/440: fsync d13/d24/d31/d23/f25 0 2024-03-30T04:28:08.014 INFO:tasks.workunit.client.0.smithi123.stdout:3/441: chown fe 0 1 2024-03-30T04:28:08.021 INFO:tasks.workunit.client.0.smithi123.stdout:3/442: chown d13/d24/d31/d23/f6c 192747 1 2024-03-30T04:28:08.023 INFO:tasks.workunit.client.0.smithi123.stdout:0/295: dwrite d6/dd/d16/f1e [0,4194304] 0 2024-03-30T04:28:08.024 INFO:tasks.workunit.client.0.smithi123.stdout:3/443: chown d13/d24/d31/c6f 2336112 1 2024-03-30T04:28:08.025 INFO:tasks.workunit.client.0.smithi123.stdout:0/296: readlink d6/dd/l2d 0 2024-03-30T04:28:08.026 INFO:tasks.workunit.client.0.smithi123.stdout:2/386: dread d0/d1f/d1c/d40/f43 [0,4194304] 0 2024-03-30T04:28:08.029 INFO:tasks.workunit.client.0.smithi123.stdout:3/444: read d13/d24/d31/d23/d3a/d3c/d66/f7f [3243740,8330] 0 2024-03-30T04:28:08.031 INFO:tasks.workunit.client.0.smithi123.stdout:0/297: mkdir d6/dd/d16/d32/d4e/d69 0 2024-03-30T04:28:08.031 INFO:tasks.workunit.client.0.smithi123.stdout:2/387: rmdir d0/d1f/d1c/d20 39 2024-03-30T04:28:08.033 INFO:tasks.workunit.client.0.smithi123.stdout:3/445: creat d13/f8a x:0 0 0 2024-03-30T04:28:08.034 INFO:tasks.workunit.client.0.smithi123.stdout:0/298: symlink d6/d52/l6a 0 2024-03-30T04:28:08.035 INFO:tasks.workunit.client.0.smithi123.stdout:3/446: write f11 [4966484,118835] 0 2024-03-30T04:28:08.036 INFO:tasks.workunit.client.0.smithi123.stdout:2/388: creat d0/d1f/d1c/d2a/f7c x:0 0 0 2024-03-30T04:28:08.047 INFO:tasks.workunit.client.0.smithi123.stdout:2/389: getdents d0/d1f/d1c/d2a 0 2024-03-30T04:28:08.050 INFO:tasks.workunit.client.0.smithi123.stdout:2/390: fsync d0/d37/f74 0 2024-03-30T04:28:08.053 INFO:tasks.workunit.client.0.smithi123.stdout:2/391: creat d0/d1f/d1c/d36/f7d x:0 0 0 2024-03-30T04:28:08.055 INFO:tasks.workunit.client.0.smithi123.stdout:3/447: dwrite d13/f14 [4194304,4194304] 0 2024-03-30T04:28:08.061 INFO:tasks.workunit.client.0.smithi123.stdout:4/412: dread d2/d4/d2c/d42/d35/d5d/d78/f83 [0,4194304] 0 2024-03-30T04:28:08.064 INFO:tasks.workunit.client.0.smithi123.stdout:4/413: truncate d2/d4/d7/d63/f82 1076880 0 2024-03-30T04:28:08.065 INFO:tasks.workunit.client.0.smithi123.stdout:0/299: dread d6/fa [0,4194304] 0 2024-03-30T04:28:08.067 INFO:tasks.workunit.client.0.smithi123.stdout:4/414: stat d2/d4/f49 0 2024-03-30T04:28:08.067 INFO:tasks.workunit.client.0.smithi123.stdout:0/300: stat d6/dd/d16/f1e 0 2024-03-30T04:28:08.068 INFO:tasks.workunit.client.0.smithi123.stdout:0/301: fsync d6/dd/f4a 0 2024-03-30T04:28:08.069 INFO:tasks.workunit.client.0.smithi123.stdout:4/415: creat d2/d34/f93 x:0 0 0 2024-03-30T04:28:08.071 INFO:tasks.workunit.client.0.smithi123.stdout:4/416: write d2/d4/d7/d1a/f30 [8906296,106036] 0 2024-03-30T04:28:08.074 INFO:tasks.workunit.client.0.smithi123.stdout:3/448: truncate d13/d24/d31/d23/d3a/d3c/f78 2128188 0 2024-03-30T04:28:08.076 INFO:tasks.workunit.client.0.smithi123.stdout:4/417: creat d2/d4/d2c/d42/d89/f94 x:0 0 0 2024-03-30T04:28:08.076 INFO:tasks.workunit.client.0.smithi123.stdout:0/302: write d6/f9 [7891144,47534] 0 2024-03-30T04:28:08.078 INFO:tasks.workunit.client.0.smithi123.stdout:0/303: chown d6/c8 1063608185 1 2024-03-30T04:28:08.079 INFO:tasks.workunit.client.0.smithi123.stdout:3/449: getdents d13/d24 0 2024-03-30T04:28:08.081 INFO:tasks.workunit.client.0.smithi123.stdout:0/304: symlink d6/dd/d31/d35/l6b 0 2024-03-30T04:28:08.082 INFO:tasks.workunit.client.0.smithi123.stdout:3/450: write d13/f1b [4975362,25698] 0 2024-03-30T04:28:08.082 INFO:tasks.workunit.client.0.smithi123.stdout:4/418: truncate d2/d4/d2c/d42/d35/d5d/d78/f85 743404 0 2024-03-30T04:28:08.083 INFO:tasks.workunit.client.0.smithi123.stdout:0/305: fsync d6/dd/d18/f47 0 2024-03-30T04:28:08.085 INFO:tasks.workunit.client.0.smithi123.stdout:4/419: write d2/d4/f22 [2545697,84792] 0 2024-03-30T04:28:08.087 INFO:tasks.workunit.client.0.smithi123.stdout:0/306: symlink d6/dd/l6c 0 2024-03-30T04:28:08.103 INFO:tasks.workunit.client.0.smithi123.stdout:3/451: dwrite d13/d24/d31/d23/d3a/d3c/d66/f7f [0,4194304] 0 2024-03-30T04:28:08.103 INFO:tasks.workunit.client.0.smithi123.stdout:4/420: dwrite d2/d34/f7c [0,4194304] 0 2024-03-30T04:28:08.110 INFO:tasks.workunit.client.0.smithi123.stdout:4/421: symlink d2/d4/d2c/d42/d35/d5d/l95 0 2024-03-30T04:28:08.111 INFO:tasks.workunit.client.0.smithi123.stdout:3/452: write d13/d24/d31/d23/d3a/d3c/d4e/f81 [521251,112905] 0 2024-03-30T04:28:08.113 INFO:tasks.workunit.client.0.smithi123.stdout:3/453: stat d13/d24/d31/d23/d3a/d3c/c43 0 2024-03-30T04:28:08.115 INFO:tasks.workunit.client.0.smithi123.stdout:3/454: readlink d13/d24/d31/d30/l62 0 2024-03-30T04:28:08.117 INFO:tasks.workunit.client.0.smithi123.stdout:0/307: dread d6/dd/d31/d35/f42 [0,4194304] 0 2024-03-30T04:28:08.121 INFO:tasks.workunit.client.0.smithi123.stdout:3/455: creat d13/d24/d31/d23/d3a/d3c/f8b x:0 0 0 2024-03-30T04:28:08.123 INFO:tasks.workunit.client.0.smithi123.stdout:0/308: symlink d6/dd/l6d 0 2024-03-30T04:28:08.124 INFO:tasks.workunit.client.0.smithi123.stdout:3/456: symlink d13/l8c 0 2024-03-30T04:28:08.130 INFO:tasks.workunit.client.0.smithi123.stdout:3/457: rename d13/d24/d39/l4f to d13/d24/d31/d23/d3a/d3c/d4e/d5d/l8d 0 2024-03-30T04:28:08.134 INFO:tasks.workunit.client.0.smithi123.stdout:0/309: write d6/dd/f36 [525477,72076] 0 2024-03-30T04:28:08.135 INFO:tasks.workunit.client.0.smithi123.stdout:0/310: write d6/dd/d18/d2f/f58 [1817008,32870] 0 2024-03-30T04:28:08.138 INFO:tasks.workunit.client.0.smithi123.stdout:0/311: write d6/dd/d15/d28/f33 [1598800,95324] 0 2024-03-30T04:28:08.145 INFO:tasks.workunit.client.0.smithi123.stdout:3/458: dwrite d13/d24/d31/f64 [4194304,4194304] 0 2024-03-30T04:28:08.150 INFO:tasks.workunit.client.0.smithi123.stdout:3/459: creat d13/d24/d31/d23/d3a/d3c/d4e/d5d/f8e x:0 0 0 2024-03-30T04:28:08.150 INFO:tasks.workunit.client.0.smithi123.stdout:0/312: dread d6/dd/d31/d35/f42 [0,4194304] 0 2024-03-30T04:28:08.151 INFO:tasks.workunit.client.0.smithi123.stdout:0/313: read d6/dd/d18/f47 [1677567,121791] 0 2024-03-30T04:28:08.155 INFO:tasks.workunit.client.0.smithi123.stdout:0/314: mkdir d6/dd/d16/d32/d4e/d69/d6e 0 2024-03-30T04:28:08.160 INFO:tasks.workunit.client.0.smithi123.stdout:0/315: rename d6/dd/d31/d35/d3a/l3c to d6/d52/l6f 0 2024-03-30T04:28:08.161 INFO:tasks.workunit.client.0.smithi123.stdout:0/316: rmdir d6/dd/d15/d28/d4b 39 2024-03-30T04:28:08.162 INFO:tasks.workunit.client.0.smithi123.stdout:0/317: fsync d6/dd/d15/f2b 0 2024-03-30T04:28:08.168 INFO:tasks.workunit.client.0.smithi123.stdout:0/318: getdents d6/dd/d31/d35/d3a 0 2024-03-30T04:28:08.171 INFO:tasks.workunit.client.0.smithi123.stdout:0/319: creat d6/dd/d31/d35/d3a/f70 x:0 0 0 2024-03-30T04:28:08.173 INFO:tasks.workunit.client.0.smithi123.stdout:0/320: mknod d6/dd/d16/d32/c71 0 2024-03-30T04:28:08.174 INFO:tasks.workunit.client.0.smithi123.stdout:0/321: chown d6/dd/d15/f2b 20 1 2024-03-30T04:28:08.176 INFO:tasks.workunit.client.0.smithi123.stdout:0/322: read f5 [128430,99110] 0 2024-03-30T04:28:08.181 INFO:tasks.workunit.client.0.smithi123.stdout:0/323: dread d6/dd/f51 [0,4194304] 0 2024-03-30T04:28:08.183 INFO:tasks.workunit.client.0.smithi123.stdout:0/324: creat d6/f72 x:0 0 0 2024-03-30T04:28:08.184 INFO:tasks.workunit.client.0.smithi123.stdout:0/325: write d6/dd/d16/f1e [3965545,87894] 0 2024-03-30T04:28:08.187 INFO:tasks.workunit.client.0.smithi123.stdout:0/326: creat d6/d52/f73 x:0 0 0 2024-03-30T04:28:08.191 INFO:tasks.workunit.client.0.smithi123.stdout:0/327: getdents d6/dd/d15/d28/d4b 0 2024-03-30T04:28:08.192 INFO:tasks.workunit.client.0.smithi123.stdout:0/328: dread - d6/d52/f73 zero size 2024-03-30T04:28:08.195 INFO:tasks.workunit.client.0.smithi123.stdout:0/329: fsync d6/dd/d31/d35/f5b 0 2024-03-30T04:28:08.207 INFO:tasks.workunit.client.0.smithi123.stdout:0/330: dwrite d6/dd/d15/f2b [0,4194304] 0 2024-03-30T04:28:08.208 INFO:tasks.workunit.client.0.smithi123.stdout:0/331: stat d6/dd/d18/d1f 0 2024-03-30T04:28:08.225 INFO:tasks.workunit.client.0.smithi123.stdout:0/332: dread f5 [0,4194304] 0 2024-03-30T04:28:08.233 INFO:tasks.workunit.client.0.smithi123.stdout:0/333: truncate d6/dd/d16/f1e 3657667 0 2024-03-30T04:28:08.245 INFO:tasks.workunit.client.0.smithi123.stdout:0/334: dwrite d6/dd/d31/d35/f43 [0,4194304] 0 2024-03-30T04:28:08.246 INFO:tasks.workunit.client.0.smithi123.stdout:0/335: chown d6/l44 0 1 2024-03-30T04:28:08.252 INFO:tasks.workunit.client.0.smithi123.stdout:0/336: dread f5 [0,4194304] 0 2024-03-30T04:28:08.255 INFO:tasks.workunit.client.0.smithi123.stdout:0/337: creat d6/dd/d16/d32/d4e/f74 x:0 0 0 2024-03-30T04:28:08.258 INFO:tasks.workunit.client.0.smithi123.stdout:0/338: truncate d6/f40 448553 0 2024-03-30T04:28:08.260 INFO:tasks.workunit.client.0.smithi123.stdout:0/339: symlink d6/l75 0 2024-03-30T04:28:08.262 INFO:tasks.workunit.client.0.smithi123.stdout:0/340: write d6/dd/d15/f2b [2022266,12921] 0 2024-03-30T04:28:08.264 INFO:tasks.workunit.client.0.smithi123.stdout:0/341: truncate f5 2725936 0 2024-03-30T04:28:08.267 INFO:tasks.workunit.client.0.smithi123.stdout:0/342: symlink d6/l76 0 2024-03-30T04:28:08.269 INFO:tasks.workunit.client.0.smithi123.stdout:0/343: read d6/dd/d31/d35/f42 [1616612,56791] 0 2024-03-30T04:28:08.281 INFO:tasks.workunit.client.0.smithi123.stdout:0/344: dwrite d6/dd/d15/d28/f5c [0,4194304] 0 2024-03-30T04:28:08.285 INFO:tasks.workunit.client.0.smithi123.stdout:0/345: rmdir d6/dd/d18/d2f 39 2024-03-30T04:28:08.297 INFO:tasks.workunit.client.0.smithi123.stdout:0/346: dwrite d6/dd/d31/d35/f45 [0,4194304] 0 2024-03-30T04:28:08.301 INFO:tasks.workunit.client.0.smithi123.stdout:0/347: creat d6/dd/d15/d28/d4b/f77 x:0 0 0 2024-03-30T04:28:08.303 INFO:tasks.workunit.client.0.smithi123.stdout:0/348: unlink d6/d52/c5e 0 2024-03-30T04:28:08.305 INFO:tasks.workunit.client.0.smithi123.stdout:0/349: chown d6/dd/d18/l49 0 1 2024-03-30T04:28:08.309 INFO:tasks.workunit.client.0.smithi123.stdout:0/350: creat d6/dd/d15/d28/f78 x:0 0 0 2024-03-30T04:28:08.312 INFO:tasks.workunit.client.0.smithi123.stdout:0/351: truncate d6/dd/d15/d28/f5a 1168261 0 2024-03-30T04:28:08.319 INFO:tasks.workunit.client.0.smithi123.stdout:0/352: dread d6/dd/d18/d2f/f58 [0,4194304] 0 2024-03-30T04:28:08.321 INFO:tasks.workunit.client.0.smithi123.stdout:0/353: mknod d6/c79 0 2024-03-30T04:28:08.323 INFO:tasks.workunit.client.0.smithi123.stdout:0/354: creat d6/dd/d16/d32/d4e/d69/f7a x:0 0 0 2024-03-30T04:28:08.326 INFO:tasks.workunit.client.0.smithi123.stdout:0/355: creat d6/dd/d31/d35/f7b x:0 0 0 2024-03-30T04:28:08.328 INFO:tasks.workunit.client.0.smithi123.stdout:0/356: rmdir d6/dd/d31/d35/d3a 39 2024-03-30T04:28:08.329 INFO:tasks.workunit.client.0.smithi123.stdout:0/357: chown d6/f24 221765007 1 2024-03-30T04:28:08.334 INFO:tasks.workunit.client.0.smithi123.stdout:0/358: creat d6/dd/d16/d32/d4e/d69/d6e/f7c x:0 0 0 2024-03-30T04:28:08.337 INFO:tasks.workunit.client.0.smithi123.stdout:0/359: truncate d6/dd/d31/d35/f5b 725432 0 2024-03-30T04:28:08.338 INFO:tasks.workunit.client.0.smithi123.stdout:0/360: chown d6/dd/d16/d32/c71 289599 1 2024-03-30T04:28:08.340 INFO:tasks.workunit.client.0.smithi123.stdout:0/361: dread - d6/dd/d16/d32/d4e/f74 zero size 2024-03-30T04:28:08.346 INFO:tasks.workunit.client.0.smithi123.stdout:0/362: truncate d6/dd/f36 165516 0 2024-03-30T04:28:08.348 INFO:tasks.workunit.client.0.smithi123.stdout:0/363: write d6/dd/d15/d28/f5c [1430008,29129] 0 2024-03-30T04:28:08.355 INFO:tasks.workunit.client.0.smithi123.stdout:0/364: rename d6/dd/d15/d28/d59 to d6/dd/d31/d35/d3a/d7d 0 2024-03-30T04:28:08.361 INFO:tasks.workunit.client.0.smithi123.stdout:0/365: dread d6/dd/f36 [0,4194304] 0 2024-03-30T04:28:08.364 INFO:tasks.workunit.client.0.smithi123.stdout:0/366: creat d6/dd/d31/d35/f7e x:0 0 0 2024-03-30T04:28:08.367 INFO:tasks.workunit.client.0.smithi123.stdout:0/367: truncate d6/dd/d16/d32/d4e/d69/d6e/f7c 439463 0 2024-03-30T04:28:08.369 INFO:tasks.workunit.client.0.smithi123.stdout:0/368: dread - d6/dd/d15/d28/d4b/f77 zero size 2024-03-30T04:28:08.371 INFO:tasks.workunit.client.0.smithi123.stdout:0/369: mkdir d6/dd/d16/d32/d5d/d7f 0 2024-03-30T04:28:08.373 INFO:tasks.workunit.client.0.smithi123.stdout:0/370: mknod d6/dd/d18/d2f/c80 0 2024-03-30T04:28:08.378 INFO:tasks.workunit.client.0.smithi123.stdout:0/371: link d6/dd/d16/d32/d4e/d69/f7a d6/dd/d16/d32/d4e/d69/f81 0 2024-03-30T04:28:08.380 INFO:tasks.workunit.client.0.smithi123.stdout:0/372: unlink f5 0 2024-03-30T04:28:08.381 INFO:tasks.workunit.client.0.smithi123.stdout:0/373: write d6/dd/f4a [195802,70277] 0 2024-03-30T04:28:08.384 INFO:tasks.workunit.client.0.smithi123.stdout:0/374: creat d6/dd/d15/d28/d4b/f82 x:0 0 0 2024-03-30T04:28:08.386 INFO:tasks.workunit.client.0.smithi123.stdout:0/375: creat d6/dd/d16/d32/d5d/f83 x:0 0 0 2024-03-30T04:28:08.398 INFO:tasks.workunit.client.0.smithi123.stdout:0/376: dwrite d6/dd/d15/f2b [0,4194304] 0 2024-03-30T04:28:08.401 INFO:tasks.workunit.client.0.smithi123.stdout:0/377: write d6/dd/d15/d28/d4b/f77 [793449,97277] 0 2024-03-30T04:28:08.404 INFO:tasks.workunit.client.0.smithi123.stdout:0/378: unlink d6/l41 0 2024-03-30T04:28:08.409 INFO:tasks.workunit.client.0.smithi123.stdout:0/379: truncate d6/dd/d16/d32/d4e/d69/d6e/f7c 1315797 0 2024-03-30T04:28:08.412 INFO:tasks.workunit.client.0.smithi123.stdout:0/380: unlink d6/dd/d31/d35/d3a/d7d/c57 0 2024-03-30T04:28:08.414 INFO:tasks.workunit.client.0.smithi123.stdout:0/381: write d6/dd/d15/d28/f33 [970016,89690] 0 2024-03-30T04:28:08.417 INFO:tasks.workunit.client.0.smithi123.stdout:0/382: truncate d6/dd/d31/d35/f7e 633836 0 2024-03-30T04:28:08.418 INFO:tasks.workunit.client.0.smithi123.stdout:0/383: chown d6/dd/d31/d35/d3a 27487 1 2024-03-30T04:28:08.419 INFO:tasks.workunit.client.0.smithi123.stdout:0/384: chown d6/dd/d18/d1f 171887 1 2024-03-30T04:28:08.422 INFO:tasks.workunit.client.0.smithi123.stdout:0/385: link d6/dd/d16/l3e d6/l84 0 2024-03-30T04:28:08.425 INFO:tasks.workunit.client.0.smithi123.stdout:0/386: creat d6/dd/d18/d2f/f85 x:0 0 0 2024-03-30T04:28:08.436 INFO:tasks.workunit.client.0.smithi123.stdout:0/387: dwrite d6/f72 [0,4194304] 0 2024-03-30T04:28:08.449 INFO:tasks.workunit.client.0.smithi123.stdout:5/381: sync 2024-03-30T04:28:08.450 INFO:tasks.workunit.client.0.smithi123.stdout:1/510: sync 2024-03-30T04:28:08.451 INFO:tasks.workunit.client.0.smithi123.stdout:5/382: dread - dd/d19/f59 zero size 2024-03-30T04:28:08.453 INFO:tasks.workunit.client.0.smithi123.stdout:0/388: truncate d6/f24 234024 0 2024-03-30T04:28:08.455 INFO:tasks.workunit.client.0.smithi123.stdout:5/383: mknod dd/c7e 0 2024-03-30T04:28:08.456 INFO:tasks.workunit.client.0.smithi123.stdout:0/389: rmdir d6/dd/d15/d28 39 2024-03-30T04:28:08.457 INFO:tasks.workunit.client.0.smithi123.stdout:5/384: rmdir dd/d41 39 2024-03-30T04:28:08.459 INFO:tasks.workunit.client.0.smithi123.stdout:0/390: write d6/dd/f4a [1161283,258] 0 2024-03-30T04:28:08.464 INFO:tasks.workunit.client.0.smithi123.stdout:5/385: rename dd/f2f to dd/d41/f7f 0 2024-03-30T04:28:08.465 INFO:tasks.workunit.client.0.smithi123.stdout:1/511: dwrite db/dd/f13 [4194304,4194304] 0 2024-03-30T04:28:08.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:08 smithi123 ceph-mon[35245]: pgmap v37: 65 pgs: 65 active+clean; 789 MiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 16 MiB/s rd, 92 MiB/s wr, 295 op/s 2024-03-30T04:28:08.469 INFO:tasks.workunit.client.0.smithi123.stdout:5/386: rmdir dd/d19/d3e/d5b/d62 39 2024-03-30T04:28:08.470 INFO:tasks.workunit.client.0.smithi123.stdout:5/387: fsync dd/d41/f51 0 2024-03-30T04:28:08.473 INFO:tasks.workunit.client.0.smithi123.stdout:0/391: rename d6/dd/d31/d35/d3a/f70 to d6/dd/d31/f86 0 2024-03-30T04:28:08.475 INFO:tasks.workunit.client.0.smithi123.stdout:5/388: rmdir dd/d41 39 2024-03-30T04:28:08.477 INFO:tasks.workunit.client.0.smithi123.stdout:1/512: rename db/dd/d42/d46/d51/c9d to db/dd/d42/ca8 0 2024-03-30T04:28:08.478 INFO:tasks.workunit.client.0.smithi123.stdout:5/389: rmdir dd/d19 39 2024-03-30T04:28:08.480 INFO:tasks.workunit.client.0.smithi123.stdout:0/392: rename d6/dd/d31/d35/f7e to d6/dd/d16/d32/d4e/d69/d6e/f87 0 2024-03-30T04:28:08.481 INFO:tasks.workunit.client.0.smithi123.stdout:1/513: truncate db/dd/d42/d46/d56/f7f 610668 0 2024-03-30T04:28:08.484 INFO:tasks.workunit.client.0.smithi123.stdout:5/390: truncate dd/d19/f59 889004 0 2024-03-30T04:28:08.488 INFO:tasks.workunit.client.0.smithi123.stdout:1/514: dread db/dd/d42/d46/d56/f70 [0,4194304] 0 2024-03-30T04:28:08.488 INFO:tasks.workunit.client.0.smithi123.stdout:5/391: creat dd/d19/d1f/d35/d3a/f80 x:0 0 0 2024-03-30T04:28:08.491 INFO:tasks.workunit.client.0.smithi123.stdout:0/393: rmdir d6/dd/d31/d35/d3a/d7d 0 2024-03-30T04:28:08.492 INFO:tasks.workunit.client.0.smithi123.stdout:0/394: read d6/dd/d31/d35/f43 [2824388,122307] 0 2024-03-30T04:28:08.493 INFO:tasks.workunit.client.0.smithi123.stdout:1/515: mknod db/dd/d42/d8f/ca9 0 2024-03-30T04:28:08.495 INFO:tasks.workunit.client.0.smithi123.stdout:1/516: fsync db/dd/f4c 0 2024-03-30T04:28:08.495 INFO:tasks.workunit.client.0.smithi123.stdout:0/395: creat d6/dd/d16/f88 x:0 0 0 2024-03-30T04:28:08.497 INFO:tasks.workunit.client.0.smithi123.stdout:5/392: creat dd/f81 x:0 0 0 2024-03-30T04:28:08.499 INFO:tasks.workunit.client.0.smithi123.stdout:1/517: rmdir db/dd/d42/d46/d39 39 2024-03-30T04:28:08.500 INFO:tasks.workunit.client.0.smithi123.stdout:1/518: chown db/d1b/c38 103 1 2024-03-30T04:28:08.501 INFO:tasks.workunit.client.0.smithi123.stdout:0/396: mknod d6/dd/d16/d32/d4e/d69/d6e/c89 0 2024-03-30T04:28:08.507 INFO:tasks.workunit.client.0.smithi123.stdout:0/397: mkdir d6/dd/d15/d28/d4b/d8a 0 2024-03-30T04:28:08.509 INFO:tasks.workunit.client.0.smithi123.stdout:0/398: write d6/dd/d15/d28/d4b/f77 [1189911,66291] 0 2024-03-30T04:28:08.510 INFO:tasks.workunit.client.0.smithi123.stdout:1/519: link db/dd/d42/d46/f2e db/dd/d5f/d6d/faa 0 2024-03-30T04:28:08.514 INFO:tasks.workunit.client.0.smithi123.stdout:1/520: read db/dd/d42/d46/d5d/d62/f6b [390733,57107] 0 2024-03-30T04:28:08.521 INFO:tasks.workunit.client.0.smithi123.stdout:5/393: dread dd/d19/d1f/d35/f4b [0,4194304] 0 2024-03-30T04:28:08.521 INFO:tasks.workunit.client.0.smithi123.stdout:0/399: dread d6/dd/d18/f47 [0,4194304] 0 2024-03-30T04:28:08.522 INFO:tasks.workunit.client.0.smithi123.stdout:5/394: fsync dd/d19/d1f/d35/f38 0 2024-03-30T04:28:08.527 INFO:tasks.workunit.client.0.smithi123.stdout:0/400: dread d6/dd/d18/d2f/f58 [0,4194304] 0 2024-03-30T04:28:08.528 INFO:tasks.workunit.client.0.smithi123.stdout:5/395: link dd/d19/d1f/d35/d3a/d57/f6a dd/d19/d1f/f82 0 2024-03-30T04:28:08.529 INFO:tasks.workunit.client.0.smithi123.stdout:0/401: write d6/dd/d31/d35/f42 [3908665,11855] 0 2024-03-30T04:28:08.531 INFO:tasks.workunit.client.0.smithi123.stdout:5/396: creat dd/d19/d1f/d35/d3a/f83 x:0 0 0 2024-03-30T04:28:08.533 INFO:tasks.workunit.client.0.smithi123.stdout:5/397: dread - dd/d19/d3e/d5b/d62/f7a zero size 2024-03-30T04:28:08.534 INFO:tasks.workunit.client.0.smithi123.stdout:0/402: rename d6/d52/l6f to d6/dd/d16/d32/d4e/l8b 0 2024-03-30T04:28:08.535 INFO:tasks.workunit.client.0.smithi123.stdout:5/398: write dd/f1c [1953517,113797] 0 2024-03-30T04:28:08.538 INFO:tasks.workunit.client.0.smithi123.stdout:5/399: write dd/d19/d1f/f27 [3450385,73766] 0 2024-03-30T04:28:08.539 INFO:tasks.workunit.client.0.smithi123.stdout:0/403: creat d6/dd/f8c x:0 0 0 2024-03-30T04:28:08.542 INFO:tasks.workunit.client.0.smithi123.stdout:0/404: chown d6/dd/d31/d35/f5b 1154005371 1 2024-03-30T04:28:08.543 INFO:tasks.workunit.client.0.smithi123.stdout:0/405: dread - d6/dd/d16/d32/d5d/f83 zero size 2024-03-30T04:28:08.546 INFO:tasks.workunit.client.0.smithi123.stdout:0/406: symlink d6/l8d 0 2024-03-30T04:28:08.546 INFO:tasks.workunit.client.0.smithi123.stdout:5/400: link dd/d19/d1f/d35/d3a/d57/f65 dd/d19/d3e/d5b/d62/f84 0 2024-03-30T04:28:08.547 INFO:tasks.workunit.client.0.smithi123.stdout:0/407: fsync d6/dd/d18/d2f/f85 0 2024-03-30T04:28:08.548 INFO:tasks.workunit.client.0.smithi123.stdout:0/408: chown d6/f1b 1395561305 1 2024-03-30T04:28:08.551 INFO:tasks.workunit.client.0.smithi123.stdout:8/357: sync 2024-03-30T04:28:08.551 INFO:tasks.workunit.client.0.smithi123.stdout:9/391: sync 2024-03-30T04:28:08.551 INFO:tasks.workunit.client.0.smithi123.stdout:0/409: write d6/dd/d15/d28/f3d [1667883,43130] 0 2024-03-30T04:28:08.554 INFO:tasks.workunit.client.0.smithi123.stdout:8/358: mkdir da/d27/d3c/d64 0 2024-03-30T04:28:08.556 INFO:tasks.workunit.client.1.smithi135.stderr:++ readlink -f fsstress 2024-03-30T04:28:08.556 INFO:tasks.workunit.client.0.smithi123.stdout:8/359: stat da/dc/c2a 0 2024-03-30T04:28:08.557 INFO:tasks.workunit.client.0.smithi123.stdout:0/410: dread d6/dd/d31/d35/f5b [0,4194304] 0 2024-03-30T04:28:08.557 INFO:tasks.workunit.client.0.smithi123.stdout:9/392: rename d6/f66 to d6/d1f/d72/f93 0 2024-03-30T04:28:08.559 INFO:tasks.workunit.client.1.smithi135.stderr:+ BIN=/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress 2024-03-30T04:28:08.559 INFO:tasks.workunit.client.1.smithi135.stderr:+ popd 2024-03-30T04:28:08.560 INFO:tasks.workunit.client.0.smithi123.stdout:8/360: creat da/d27/d56/f65 x:0 0 0 2024-03-30T04:28:08.561 INFO:tasks.workunit.client.0.smithi123.stdout:8/361: readlink da/d27/l43 0 2024-03-30T04:28:08.562 INFO:tasks.workunit.client.1.smithi135.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-03-30T04:28:08.562 INFO:tasks.workunit.client.1.smithi135.stderr:+ popd 2024-03-30T04:28:08.563 INFO:tasks.workunit.client.0.smithi123.stdout:0/411: creat d6/dd/d15/f8e x:0 0 0 2024-03-30T04:28:08.563 INFO:tasks.workunit.client.1.smithi135.stdout:~/cephtest/mnt.1/client.1/tmp 2024-03-30T04:28:08.564 INFO:tasks.workunit.client.1.smithi135.stderr:++ mktemp -d -p . 2024-03-30T04:28:08.564 INFO:tasks.workunit.client.0.smithi123.stdout:5/401: dwrite dd/d19/d1f/f82 [0,4194304] 0 2024-03-30T04:28:08.566 INFO:tasks.workunit.client.0.smithi123.stdout:0/412: unlink d6/dd/f36 0 2024-03-30T04:28:08.568 INFO:tasks.workunit.client.0.smithi123.stdout:9/393: write d6/d1f/f2d [946711,111622] 0 2024-03-30T04:28:08.569 INFO:tasks.workunit.client.1.smithi135.stderr:+ T=./tmp.e0cy5c8jtQ 2024-03-30T04:28:08.569 INFO:tasks.workunit.client.1.smithi135.stderr:+ /home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress -d ./tmp.e0cy5c8jtQ -l 1 -n 1000 -p 10 -v 2024-03-30T04:28:08.571 INFO:tasks.workunit.client.0.smithi123.stdout:9/394: write d6/d1f/d72/f8f [856574,83737] 0 2024-03-30T04:28:08.572 INFO:tasks.workunit.client.0.smithi123.stdout:0/413: creat d6/dd/d31/d35/d3a/f8f x:0 0 0 2024-03-30T04:28:08.575 INFO:tasks.workunit.client.1.smithi135.stdout:seed = 1712277461 2024-03-30T04:28:08.583 INFO:tasks.workunit.client.1.smithi135.stdout:0/0: fdatasync - no filename 2024-03-30T04:28:08.583 INFO:tasks.workunit.client.1.smithi135.stdout:0/1: write - no filename 2024-03-30T04:28:08.583 INFO:tasks.workunit.client.1.smithi135.stdout:0/2: write - no filename 2024-03-30T04:28:08.583 INFO:tasks.workunit.client.1.smithi135.stdout:0/3: write - no filename 2024-03-30T04:28:08.583 INFO:tasks.workunit.client.1.smithi135.stdout:0/4: dwrite - no filename 2024-03-30T04:28:08.583 INFO:tasks.workunit.client.0.smithi123.stdout:5/402: dwrite dd/d19/d1f/f82 [0,4194304] 0 2024-03-30T04:28:08.585 INFO:tasks.workunit.client.1.smithi135.stdout:1/0: link - no file 2024-03-30T04:28:08.586 INFO:tasks.workunit.client.1.smithi135.stdout:0/5: mkdir d0 0 2024-03-30T04:28:08.586 INFO:tasks.workunit.client.1.smithi135.stdout:0/6: dwrite - no filename 2024-03-30T04:28:08.587 INFO:tasks.workunit.client.1.smithi135.stdout:0/7: chown d0 2318222 1 2024-03-30T04:28:08.587 INFO:tasks.workunit.client.1.smithi135.stdout:0/8: unlink - no file 2024-03-30T04:28:08.588 INFO:tasks.workunit.client.0.smithi123.stdout:0/414: truncate d6/dd/d15/d28/f3d 141257 0 2024-03-30T04:28:08.589 INFO:tasks.workunit.client.1.smithi135.stdout:1/1: mknod c0 0 2024-03-30T04:28:08.589 INFO:tasks.workunit.client.1.smithi135.stdout:1/2: write - no filename 2024-03-30T04:28:08.589 INFO:tasks.workunit.client.1.smithi135.stdout:1/3: read - no filename 2024-03-30T04:28:08.589 INFO:tasks.workunit.client.1.smithi135.stdout:1/4: write - no filename 2024-03-30T04:28:08.589 INFO:tasks.workunit.client.1.smithi135.stdout:1/5: dwrite - no filename 2024-03-30T04:28:08.590 INFO:tasks.workunit.client.0.smithi123.stdout:5/403: creat dd/d19/f85 x:0 0 0 2024-03-30T04:28:08.590 INFO:tasks.workunit.client.1.smithi135.stdout:2/0: write - no filename 2024-03-30T04:28:08.590 INFO:tasks.workunit.client.1.smithi135.stdout:2/1: read - no filename 2024-03-30T04:28:08.591 INFO:tasks.workunit.client.1.smithi135.stdout:0/9: mkdir d0/d1 0 2024-03-30T04:28:08.591 INFO:tasks.workunit.client.0.smithi123.stdout:0/415: symlink d6/l90 0 2024-03-30T04:28:08.593 INFO:tasks.workunit.client.0.smithi123.stdout:9/395: dwrite d6/d1f/d4d/f5b [0,4194304] 0 2024-03-30T04:28:08.595 INFO:tasks.workunit.client.1.smithi135.stdout:3/0: fdatasync - no filename 2024-03-30T04:28:08.595 INFO:tasks.workunit.client.1.smithi135.stdout:3/1: dwrite - no filename 2024-03-30T04:28:08.595 INFO:tasks.workunit.client.1.smithi135.stdout:3/2: write - no filename 2024-03-30T04:28:08.595 INFO:tasks.workunit.client.1.smithi135.stdout:3/3: rmdir - no directory 2024-03-30T04:28:08.595 INFO:tasks.workunit.client.1.smithi135.stdout:3/4: rename - no filename 2024-03-30T04:28:08.595 INFO:tasks.workunit.client.1.smithi135.stdout:3/5: dread - no filename 2024-03-30T04:28:08.595 INFO:tasks.workunit.client.1.smithi135.stdout:3/6: chown . 5687 1 2024-03-30T04:28:08.595 INFO:tasks.workunit.client.1.smithi135.stdout:3/7: rename - no filename 2024-03-30T04:28:08.595 INFO:tasks.workunit.client.0.smithi123.stdout:8/362: dread da/f46 [4194304,4194304] 0 2024-03-30T04:28:08.598 INFO:tasks.workunit.client.0.smithi123.stdout:8/363: rmdir da/dc 39 2024-03-30T04:28:08.601 INFO:tasks.workunit.client.1.smithi135.stdout:1/6: unlink c0 0 2024-03-30T04:28:08.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:08 smithi135 ceph-mon[44364]: pgmap v37: 65 pgs: 65 active+clean; 789 MiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 16 MiB/s rd, 92 MiB/s wr, 295 op/s 2024-03-30T04:28:08.602 INFO:tasks.workunit.client.0.smithi123.stdout:8/364: fdatasync da/dc/f16 0 2024-03-30T04:28:08.603 INFO:tasks.workunit.client.1.smithi135.stdout:2/2: mknod c0 0 2024-03-30T04:28:08.604 INFO:tasks.workunit.client.0.smithi123.stdout:8/365: stat da/dc/d14/d18/f30 0 2024-03-30T04:28:08.604 INFO:tasks.workunit.client.1.smithi135.stdout:2/3: write - no filename 2024-03-30T04:28:08.606 INFO:tasks.workunit.client.1.smithi135.stdout:2/4: chown c0 2564 1 2024-03-30T04:28:08.606 INFO:tasks.workunit.client.0.smithi123.stdout:8/366: unlink da/dc/d14/d18/f1e 0 2024-03-30T04:28:08.607 INFO:tasks.workunit.client.1.smithi135.stdout:2/5: write - no filename 2024-03-30T04:28:08.607 INFO:tasks.workunit.client.1.smithi135.stdout:2/6: dread - no filename 2024-03-30T04:28:08.607 INFO:tasks.workunit.client.1.smithi135.stdout:4/0: read - no filename 2024-03-30T04:28:08.608 INFO:tasks.workunit.client.1.smithi135.stdout:0/10: creat d0/f2 x:0 0 0 2024-03-30T04:28:08.609 INFO:tasks.workunit.client.0.smithi123.stdout:8/367: chown da/d27/d3c/f5e 1166 1 2024-03-30T04:28:08.613 INFO:tasks.workunit.client.1.smithi135.stdout:3/8: creat f0 x:0 0 0 2024-03-30T04:28:08.613 INFO:tasks.workunit.client.0.smithi123.stdout:8/368: getdents da/d27 0 2024-03-30T04:28:08.614 INFO:tasks.workunit.client.1.smithi135.stdout:3/9: dread - f0 zero size 2024-03-30T04:28:08.615 INFO:tasks.workunit.client.1.smithi135.stdout:2/7: symlink l1 0 2024-03-30T04:28:08.616 INFO:tasks.workunit.client.1.smithi135.stdout:6/0: rmdir - no directory 2024-03-30T04:28:08.617 INFO:tasks.workunit.client.1.smithi135.stdout:0/11: unlink d0/f2 0 2024-03-30T04:28:08.617 INFO:tasks.workunit.client.1.smithi135.stdout:0/12: fdatasync - no filename 2024-03-30T04:28:08.618 INFO:tasks.workunit.client.0.smithi123.stdout:8/369: unlink da/l50 0 2024-03-30T04:28:08.619 INFO:tasks.workunit.client.1.smithi135.stdout:0/13: chown d0/d1 886 1 2024-03-30T04:28:08.619 INFO:tasks.workunit.client.1.smithi135.stdout:4/1: symlink l0 0 2024-03-30T04:28:08.621 INFO:tasks.workunit.client.1.smithi135.stdout:4/2: stat l0 0 2024-03-30T04:28:08.621 INFO:tasks.workunit.client.1.smithi135.stdout:4/3: write - no filename 2024-03-30T04:28:08.622 INFO:tasks.workunit.client.1.smithi135.stdout:0/14: creat d0/f3 x:0 0 0 2024-03-30T04:28:08.623 INFO:tasks.workunit.client.0.smithi123.stdout:9/396: dread d6/d1f/f41 [0,4194304] 0 2024-03-30T04:28:08.625 INFO:tasks.workunit.client.1.smithi135.stdout:6/1: mknod c0 0 2024-03-30T04:28:08.625 INFO:tasks.workunit.client.1.smithi135.stdout:7/0: write - no filename 2024-03-30T04:28:08.625 INFO:tasks.workunit.client.1.smithi135.stdout:6/2: truncate - no filename 2024-03-30T04:28:08.625 INFO:tasks.workunit.client.1.smithi135.stdout:6/3: dread - no filename 2024-03-30T04:28:08.625 INFO:tasks.workunit.client.1.smithi135.stdout:6/4: write - no filename 2024-03-30T04:28:08.627 INFO:tasks.workunit.client.0.smithi123.stdout:9/397: write d6/d7c/f89 [84015,121061] 0 2024-03-30T04:28:08.628 INFO:tasks.workunit.client.1.smithi135.stdout:8/0: dwrite - no filename 2024-03-30T04:28:08.629 INFO:tasks.workunit.client.1.smithi135.stdout:8/1: link - no file 2024-03-30T04:28:08.629 INFO:tasks.workunit.client.1.smithi135.stdout:6/5: chown c0 23 1 2024-03-30T04:28:08.629 INFO:tasks.workunit.client.1.smithi135.stdout:6/6: dread - no filename 2024-03-30T04:28:08.629 INFO:tasks.workunit.client.1.smithi135.stdout:0/15: read - d0/f3 zero size 2024-03-30T04:28:08.629 INFO:tasks.workunit.client.1.smithi135.stdout:4/4: rename l0 to l1 0 2024-03-30T04:28:08.629 INFO:tasks.workunit.client.1.smithi135.stdout:4/5: fsync - no filename 2024-03-30T04:28:08.632 INFO:tasks.workunit.client.0.smithi123.stdout:9/398: unlink d6/d1f/d45/d4c/f4e 0 2024-03-30T04:28:08.633 INFO:tasks.workunit.client.1.smithi135.stdout:9/0: readlink - no filename 2024-03-30T04:28:08.634 INFO:tasks.workunit.client.0.smithi123.stdout:9/399: write d6/d49/f6d [95899,7898] 0 2024-03-30T04:28:08.634 INFO:tasks.workunit.client.1.smithi135.stdout:0/16: rename d0 to d0/d4 22 2024-03-30T04:28:08.635 INFO:tasks.workunit.client.1.smithi135.stdout:6/7: creat f1 x:0 0 0 2024-03-30T04:28:08.636 INFO:tasks.workunit.client.0.smithi123.stdout:5/404: dread fb [0,4194304] 0 2024-03-30T04:28:08.637 INFO:tasks.workunit.client.1.smithi135.stdout:7/1: creat f0 x:0 0 0 2024-03-30T04:28:08.637 INFO:tasks.workunit.client.1.smithi135.stdout:8/2: creat f0 x:0 0 0 2024-03-30T04:28:08.638 INFO:tasks.workunit.client.0.smithi123.stdout:9/400: truncate d6/d49/f65 741111 0 2024-03-30T04:28:08.639 INFO:tasks.workunit.client.0.smithi123.stdout:0/416: link d6/dd/d15/d28/f3d d6/dd/d15/d28/d4b/f91 0 2024-03-30T04:28:08.639 INFO:tasks.workunit.client.1.smithi135.stdout:5/0: fsync - no filename 2024-03-30T04:28:08.639 INFO:tasks.workunit.client.1.smithi135.stdout:5/1: unlink - no file 2024-03-30T04:28:08.640 INFO:tasks.workunit.client.1.smithi135.stdout:8/3: write f0 [154583,128210] 0 2024-03-30T04:28:08.642 INFO:tasks.workunit.client.1.smithi135.stdout:8/4: write f0 [1270775,61822] 0 2024-03-30T04:28:08.642 INFO:tasks.workunit.client.1.smithi135.stdout:7/2: creat f1 x:0 0 0 2024-03-30T04:28:08.642 INFO:tasks.workunit.client.1.smithi135.stdout:7/3: rmdir - no directory 2024-03-30T04:28:08.644 INFO:tasks.workunit.client.0.smithi123.stdout:9/401: chown d6/d1f/c86 11 1 2024-03-30T04:28:08.645 INFO:tasks.workunit.client.1.smithi135.stdout:9/1: creat f0 x:0 0 0 2024-03-30T04:28:08.646 INFO:tasks.workunit.client.0.smithi123.stdout:9/402: dread - d6/d1f/f70 zero size 2024-03-30T04:28:08.647 INFO:tasks.workunit.client.0.smithi123.stdout:5/405: symlink dd/d19/d1f/d35/d3a/l86 0 2024-03-30T04:28:08.648 INFO:tasks.workunit.client.0.smithi123.stdout:8/370: dread da/fb [0,4194304] 0 2024-03-30T04:28:08.649 INFO:tasks.workunit.client.1.smithi135.stdout:9/2: write f0 [28904,7226] 0 2024-03-30T04:28:08.649 INFO:tasks.workunit.client.0.smithi123.stdout:9/403: read - d6/d1f/d45/d4c/f54 zero size 2024-03-30T04:28:08.650 INFO:tasks.workunit.client.1.smithi135.stdout:7/4: creat f2 x:0 0 0 2024-03-30T04:28:08.651 INFO:tasks.workunit.client.1.smithi135.stdout:5/2: mknod c0 0 2024-03-30T04:28:08.651 INFO:tasks.workunit.client.0.smithi123.stdout:5/406: rename dd/f4e to dd/d19/f87 0 2024-03-30T04:28:08.651 INFO:tasks.workunit.client.0.smithi123.stdout:9/404: fdatasync d6/d1f/f71 0 2024-03-30T04:28:08.653 INFO:tasks.workunit.client.0.smithi123.stdout:9/405: fsync d6/d1f/f71 0 2024-03-30T04:28:08.653 INFO:tasks.workunit.client.0.smithi123.stdout:8/371: write da/ff [1875441,54226] 0 2024-03-30T04:28:08.656 INFO:tasks.workunit.client.1.smithi135.stdout:5/3: mkdir d1 0 2024-03-30T04:28:08.656 INFO:tasks.workunit.client.1.smithi135.stdout:5/4: dwrite - no filename 2024-03-30T04:28:08.656 INFO:tasks.workunit.client.1.smithi135.stdout:5/5: dread - no filename 2024-03-30T04:28:08.657 INFO:tasks.workunit.client.0.smithi123.stdout:5/407: mknod dd/d19/d1f/d3c/c88 0 2024-03-30T04:28:08.657 INFO:tasks.workunit.client.1.smithi135.stdout:7/5: truncate f0 1037039 0 2024-03-30T04:28:08.657 INFO:tasks.workunit.client.1.smithi135.stdout:7/6: readlink - no filename 2024-03-30T04:28:08.658 INFO:tasks.workunit.client.0.smithi123.stdout:5/408: stat dd/d19/d3e/c4c 0 2024-03-30T04:28:08.664 INFO:tasks.workunit.client.0.smithi123.stdout:9/406: truncate d6/d23/f8d 1114601 0 2024-03-30T04:28:08.664 INFO:tasks.workunit.client.1.smithi135.stdout:5/6: chown d1 96 1 2024-03-30T04:28:08.664 INFO:tasks.workunit.client.1.smithi135.stdout:5/7: dwrite - no filename 2024-03-30T04:28:08.665 INFO:tasks.workunit.client.0.smithi123.stdout:9/407: readlink d6/l3c 0 2024-03-30T04:28:08.666 INFO:tasks.workunit.client.0.smithi123.stdout:0/417: write d6/dd/d15/d28/d4b/f91 [724424,40928] 0 2024-03-30T04:28:08.668 INFO:tasks.workunit.client.1.smithi135.stdout:5/8: stat d1 0 2024-03-30T04:28:08.669 INFO:tasks.workunit.client.0.smithi123.stdout:9/408: mknod d6/d1f/d45/c94 0 2024-03-30T04:28:08.671 INFO:tasks.workunit.client.1.smithi135.stdout:7/7: write f1 [655330,70704] 0 2024-03-30T04:28:08.672 INFO:tasks.workunit.client.0.smithi123.stdout:9/409: creat d6/d1f/d72/f95 x:0 0 0 2024-03-30T04:28:08.672 INFO:tasks.workunit.client.0.smithi123.stdout:5/409: write dd/d19/d3e/d5b/d62/f84 [1870138,80619] 0 2024-03-30T04:28:08.675 INFO:tasks.workunit.client.0.smithi123.stdout:9/410: creat d6/d58/f96 x:0 0 0 2024-03-30T04:28:08.675 INFO:tasks.workunit.client.0.smithi123.stdout:7/386: sync 2024-03-30T04:28:08.677 INFO:tasks.workunit.client.0.smithi123.stdout:5/410: getdents dd/d19/d1f/d3c/d72 0 2024-03-30T04:28:08.677 INFO:tasks.workunit.client.0.smithi123.stdout:9/411: fdatasync d6/d1f/d72/f93 0 2024-03-30T04:28:08.678 INFO:tasks.workunit.client.0.smithi123.stdout:7/387: chown d2/da/d11/c2c 229060764 1 2024-03-30T04:28:08.679 INFO:tasks.workunit.client.0.smithi123.stdout:5/411: dread - dd/d19/d3e/f75 zero size 2024-03-30T04:28:08.680 INFO:tasks.workunit.client.0.smithi123.stdout:0/418: dread d6/fa [0,4194304] 0 2024-03-30T04:28:08.681 INFO:tasks.workunit.client.0.smithi123.stdout:7/388: mkdir d2/d4d/d80 0 2024-03-30T04:28:08.682 INFO:tasks.workunit.client.0.smithi123.stdout:0/419: readlink d6/dd/l6c 0 2024-03-30T04:28:08.683 INFO:tasks.workunit.client.0.smithi123.stdout:7/389: dread - d2/da/d11/d61/f72 zero size 2024-03-30T04:28:08.684 INFO:tasks.workunit.client.0.smithi123.stdout:0/420: dread - d6/dd/d31/d35/d3a/f8f zero size 2024-03-30T04:28:08.685 INFO:tasks.workunit.client.0.smithi123.stdout:7/390: fsync d2/dc/d56/f67 0 2024-03-30T04:28:08.686 INFO:tasks.workunit.client.1.smithi135.stdout:6/8: dwrite f1 [0,4194304] 0 2024-03-30T04:28:08.692 INFO:tasks.workunit.client.1.smithi135.stdout:5/9: rename c0 to d1/c2 0 2024-03-30T04:28:08.692 INFO:tasks.workunit.client.1.smithi135.stdout:5/10: dread - no filename 2024-03-30T04:28:08.692 INFO:tasks.workunit.client.1.smithi135.stdout:5/11: truncate - no filename 2024-03-30T04:28:08.692 INFO:tasks.workunit.client.0.smithi123.stdout:9/412: dread d6/d1f/d4d/f5b [0,4194304] 0 2024-03-30T04:28:08.693 INFO:tasks.workunit.client.0.smithi123.stdout:7/391: link d2/da/d2d/f66 d2/dc/f81 0 2024-03-30T04:28:08.697 INFO:tasks.workunit.client.0.smithi123.stdout:5/412: dwrite dd/d19/f59 [0,4194304] 0 2024-03-30T04:28:08.702 INFO:tasks.workunit.client.0.smithi123.stdout:5/413: mknod dd/d19/d5f/c89 0 2024-03-30T04:28:08.705 INFO:tasks.workunit.client.1.smithi135.stdout:6/9: mkdir d2 0 2024-03-30T04:28:08.707 INFO:tasks.workunit.client.0.smithi123.stdout:9/413: rename d6/d49/d67 to d6/d1f/d45/d97 0 2024-03-30T04:28:08.707 INFO:tasks.workunit.client.1.smithi135.stdout:8/5: dwrite f0 [0,4194304] 0 2024-03-30T04:28:08.707 INFO:tasks.workunit.client.1.smithi135.stdout:6/10: chown d2 56286 1 2024-03-30T04:28:08.709 INFO:tasks.workunit.client.0.smithi123.stdout:9/414: truncate d6/d49/f65 1593487 0 2024-03-30T04:28:08.710 INFO:tasks.workunit.client.1.smithi135.stdout:5/12: mkdir d1/d3 0 2024-03-30T04:28:08.710 INFO:tasks.workunit.client.1.smithi135.stdout:5/13: write - no filename 2024-03-30T04:28:08.711 INFO:tasks.workunit.client.1.smithi135.stdout:5/14: dwrite - no filename 2024-03-30T04:28:08.713 INFO:tasks.workunit.client.1.smithi135.stdout:8/6: chown f0 49257 1 2024-03-30T04:28:08.720 INFO:tasks.workunit.client.1.smithi135.stdout:9/3: dwrite f0 [0,4194304] 0 2024-03-30T04:28:08.726 INFO:tasks.workunit.client.1.smithi135.stdout:8/7: symlink l1 0 2024-03-30T04:28:08.729 INFO:tasks.workunit.client.1.smithi135.stdout:5/15: creat d1/f4 x:0 0 0 2024-03-30T04:28:08.729 INFO:tasks.workunit.client.0.smithi123.stdout:0/421: dread d6/f9 [0,4194304] 0 2024-03-30T04:28:08.730 INFO:tasks.workunit.client.1.smithi135.stdout:7/8: dwrite f2 [0,4194304] 0 2024-03-30T04:28:08.733 INFO:tasks.workunit.client.1.smithi135.stdout:8/8: symlink l2 0 2024-03-30T04:28:08.733 INFO:tasks.workunit.client.0.smithi123.stdout:9/415: dwrite d6/d1f/f41 [0,4194304] 0 2024-03-30T04:28:08.740 INFO:tasks.workunit.client.1.smithi135.stdout:7/9: creat f3 x:0 0 0 2024-03-30T04:28:08.740 INFO:tasks.workunit.client.0.smithi123.stdout:9/416: mknod d6/d1f/d45/d4c/c98 0 2024-03-30T04:28:08.745 INFO:tasks.workunit.client.1.smithi135.stdout:5/16: symlink d1/d3/l5 0 2024-03-30T04:28:08.745 INFO:tasks.workunit.client.0.smithi123.stdout:9/417: mknod d6/d49/c99 0 2024-03-30T04:28:08.751 INFO:tasks.workunit.client.1.smithi135.stdout:3/10: getdents . 0 2024-03-30T04:28:08.754 INFO:tasks.workunit.client.1.smithi135.stdout:9/4: dwrite f0 [0,4194304] 0 2024-03-30T04:28:08.756 INFO:tasks.workunit.client.1.smithi135.stdout:9/5: write f0 [321979,32660] 0 2024-03-30T04:28:08.756 INFO:tasks.workunit.client.1.smithi135.stdout:3/11: symlink l1 0 2024-03-30T04:28:08.758 INFO:tasks.workunit.client.0.smithi123.stdout:9/418: rename d6/d49/l91 to d6/d1f/d72/d7d/l9a 0 2024-03-30T04:28:08.760 INFO:tasks.workunit.client.1.smithi135.stdout:9/6: rename f0 to f1 0 2024-03-30T04:28:08.760 INFO:tasks.workunit.client.1.smithi135.stdout:0/17: rmdir d0 39 2024-03-30T04:28:08.766 INFO:tasks.workunit.client.1.smithi135.stdout:0/18: dread - d0/f3 zero size 2024-03-30T04:28:08.767 INFO:tasks.workunit.client.1.smithi135.stdout:4/6: chown l1 51184 1 2024-03-30T04:28:08.767 INFO:tasks.workunit.client.1.smithi135.stdout:0/19: dread - d0/f3 zero size 2024-03-30T04:28:08.770 INFO:tasks.workunit.client.0.smithi123.stdout:9/419: truncate d6/d1f/d72/f8f 435710 0 2024-03-30T04:28:08.770 INFO:tasks.workunit.client.1.smithi135.stdout:0/20: fsync d0/f3 0 2024-03-30T04:28:08.771 INFO:tasks.workunit.client.1.smithi135.stdout:0/21: readlink - no filename 2024-03-30T04:28:08.772 INFO:tasks.workunit.client.1.smithi135.stdout:0/22: rmdir d0 39 2024-03-30T04:28:08.774 INFO:tasks.workunit.client.0.smithi123.stdout:9/420: write d6/d1f/f3f [716750,22451] 0 2024-03-30T04:28:08.776 INFO:tasks.workunit.client.1.smithi135.stdout:0/23: mkdir d0/d5 0 2024-03-30T04:28:08.777 INFO:tasks.workunit.client.1.smithi135.stdout:5/17: fsync d1/f4 0 2024-03-30T04:28:08.779 INFO:tasks.workunit.client.0.smithi123.stdout:9/421: symlink d6/d1f/d45/d97/l9b 0 2024-03-30T04:28:08.780 INFO:tasks.workunit.client.1.smithi135.stdout:8/9: truncate f0 3152178 0 2024-03-30T04:28:08.780 INFO:tasks.workunit.client.1.smithi135.stdout:3/12: dwrite f0 [0,4194304] 0 2024-03-30T04:28:08.781 INFO:tasks.workunit.client.1.smithi135.stdout:3/13: chown l1 83559666 1 2024-03-30T04:28:08.782 INFO:tasks.workunit.client.1.smithi135.stdout:0/24: truncate d0/f3 72185 0 2024-03-30T04:28:08.782 INFO:tasks.workunit.client.0.smithi123.stdout:9/422: creat d6/d1f/d72/d7d/f9c x:0 0 0 2024-03-30T04:28:08.782 INFO:tasks.workunit.client.1.smithi135.stdout:3/14: chown f0 936 1 2024-03-30T04:28:08.783 INFO:tasks.workunit.client.1.smithi135.stdout:5/18: symlink d1/l6 0 2024-03-30T04:28:08.784 INFO:tasks.workunit.client.0.smithi123.stdout:9/423: chown d6/d1f/d45/d4c/f54 1107128 1 2024-03-30T04:28:08.785 INFO:tasks.workunit.client.1.smithi135.stdout:3/15: creat f2 x:0 0 0 2024-03-30T04:28:08.786 INFO:tasks.workunit.client.0.smithi123.stdout:9/424: fdatasync d6/d1f/d4d/f5b 0 2024-03-30T04:28:08.787 INFO:tasks.workunit.client.1.smithi135.stdout:0/25: symlink d0/d5/l6 0 2024-03-30T04:28:08.788 INFO:tasks.workunit.client.0.smithi123.stdout:9/425: fdatasync d6/d1f/f37 0 2024-03-30T04:28:08.788 INFO:tasks.workunit.client.1.smithi135.stdout:7/10: truncate f1 318586 0 2024-03-30T04:28:08.789 INFO:tasks.workunit.client.1.smithi135.stdout:5/19: mkdir d1/d7 0 2024-03-30T04:28:08.791 INFO:tasks.workunit.client.0.smithi123.stdout:9/426: mknod d6/d1f/d72/d7d/c9d 0 2024-03-30T04:28:08.791 INFO:tasks.workunit.client.1.smithi135.stdout:3/16: mkdir d3 0 2024-03-30T04:28:08.792 INFO:tasks.workunit.client.1.smithi135.stdout:7/11: mkdir d4 0 2024-03-30T04:28:08.794 INFO:tasks.workunit.client.0.smithi123.stdout:9/427: fsync d6/d23/f8d 0 2024-03-30T04:28:08.794 INFO:tasks.workunit.client.1.smithi135.stdout:9/7: truncate f1 2314948 0 2024-03-30T04:28:08.796 INFO:tasks.workunit.client.1.smithi135.stdout:0/26: mknod d0/d1/c7 0 2024-03-30T04:28:08.798 INFO:tasks.workunit.client.1.smithi135.stdout:0/27: truncate d0/f3 878162 0 2024-03-30T04:28:08.800 INFO:tasks.workunit.client.1.smithi135.stdout:5/20: creat d1/d7/f8 x:0 0 0 2024-03-30T04:28:08.801 INFO:tasks.workunit.client.1.smithi135.stdout:3/17: creat d3/f4 x:0 0 0 2024-03-30T04:28:08.802 INFO:tasks.workunit.client.1.smithi135.stdout:9/8: creat f2 x:0 0 0 2024-03-30T04:28:08.803 INFO:tasks.workunit.client.1.smithi135.stdout:7/12: symlink d4/l5 0 2024-03-30T04:28:08.805 INFO:tasks.workunit.client.1.smithi135.stdout:9/9: dread - f2 zero size 2024-03-30T04:28:08.806 INFO:tasks.workunit.client.0.smithi123.stdout:6/524: sync 2024-03-30T04:28:08.806 INFO:tasks.workunit.client.0.smithi123.stdout:2/392: sync 2024-03-30T04:28:08.806 INFO:tasks.workunit.client.0.smithi123.stdout:1/521: sync 2024-03-30T04:28:08.806 INFO:tasks.workunit.client.0.smithi123.stdout:3/460: sync 2024-03-30T04:28:08.806 INFO:tasks.workunit.client.0.smithi123.stdout:4/422: sync 2024-03-30T04:28:08.806 INFO:tasks.workunit.client.1.smithi135.stdout:3/18: creat d3/f5 x:0 0 0 2024-03-30T04:28:08.808 INFO:tasks.workunit.client.1.smithi135.stdout:5/21: rename d1/l6 to d1/d3/l9 0 2024-03-30T04:28:08.808 INFO:tasks.workunit.client.1.smithi135.stdout:0/28: dread d0/f3 [0,4194304] 0 2024-03-30T04:28:08.809 INFO:tasks.workunit.client.0.smithi123.stdout:2/393: write d0/d1f/f6e [93017,113481] 0 2024-03-30T04:28:08.809 INFO:tasks.workunit.client.1.smithi135.stdout:9/10: write f2 [970832,9145] 0 2024-03-30T04:28:08.809 INFO:tasks.workunit.client.1.smithi135.stdout:7/13: mkdir d4/d6 0 2024-03-30T04:28:08.812 INFO:tasks.workunit.client.1.smithi135.stdout:3/19: mkdir d3/d6 0 2024-03-30T04:28:08.812 INFO:tasks.workunit.client.0.smithi123.stdout:1/522: dread db/dd/d42/d46/d56/f70 [0,4194304] 0 2024-03-30T04:28:08.812 INFO:tasks.workunit.client.0.smithi123.stdout:6/525: mkdir d10/d82/d9a/db3 0 2024-03-30T04:28:08.812 INFO:tasks.workunit.client.0.smithi123.stdout:2/394: chown d0/d1f/d5e/f6f 911418979 1 2024-03-30T04:28:08.815 INFO:tasks.workunit.client.1.smithi135.stdout:8/10: dwrite f0 [0,4194304] 0 2024-03-30T04:28:08.816 INFO:tasks.workunit.client.1.smithi135.stdout:0/29: dread d0/f3 [0,4194304] 0 2024-03-30T04:28:08.818 INFO:tasks.workunit.client.0.smithi123.stdout:2/395: mknod d0/d1f/d1c/c7e 0 2024-03-30T04:28:08.818 INFO:tasks.workunit.client.1.smithi135.stdout:7/14: symlink d4/l7 0 2024-03-30T04:28:08.818 INFO:tasks.workunit.client.1.smithi135.stdout:0/30: chown d0 4 1 2024-03-30T04:28:08.819 INFO:tasks.workunit.client.1.smithi135.stdout:7/15: readlink d4/l5 0 2024-03-30T04:28:08.822 INFO:tasks.workunit.client.0.smithi123.stdout:6/526: getdents d10/d20/dac/d42 0 2024-03-30T04:28:08.822 INFO:tasks.workunit.client.1.smithi135.stdout:8/11: rename l2 to l3 0 2024-03-30T04:28:08.823 INFO:tasks.workunit.client.0.smithi123.stdout:2/396: read d0/d1f/d1c/d40/f43 [2274144,13363] 0 2024-03-30T04:28:08.825 INFO:tasks.workunit.client.1.smithi135.stdout:3/20: link f2 d3/d6/f7 0 2024-03-30T04:28:08.826 INFO:tasks.workunit.client.1.smithi135.stdout:3/21: readlink l1 0 2024-03-30T04:28:08.830 INFO:tasks.workunit.client.1.smithi135.stdout:0/31: creat d0/d5/f8 x:0 0 0 2024-03-30T04:28:08.832 INFO:tasks.workunit.client.1.smithi135.stdout:7/16: creat d4/f8 x:0 0 0 2024-03-30T04:28:08.834 INFO:tasks.workunit.client.0.smithi123.stdout:4/423: dwrite d2/d4/d2c/d42/d35/d5d/d6f/f8a [0,4194304] 0 2024-03-30T04:28:08.836 INFO:tasks.workunit.client.1.smithi135.stdout:3/22: symlink d3/l8 0 2024-03-30T04:28:08.837 INFO:tasks.workunit.client.1.smithi135.stdout:5/22: getdents d1/d7 0 2024-03-30T04:28:08.839 INFO:tasks.workunit.client.1.smithi135.stdout:0/32: mkdir d0/d5/d9 0 2024-03-30T04:28:08.843 INFO:tasks.workunit.client.1.smithi135.stdout:5/23: mkdir d1/da 0 2024-03-30T04:28:08.844 INFO:tasks.workunit.client.0.smithi123.stdout:4/424: mkdir d2/d4/d2c/d42/d80/d96 0 2024-03-30T04:28:08.849 INFO:tasks.workunit.client.0.smithi123.stdout:1/523: dwrite db/f24 [0,4194304] 0 2024-03-30T04:28:08.849 INFO:tasks.workunit.client.0.smithi123.stdout:3/461: dwrite d13/d24/d31/f61 [0,4194304] 0 2024-03-30T04:28:08.849 INFO:tasks.workunit.client.1.smithi135.stdout:3/23: creat d3/f9 x:0 0 0 2024-03-30T04:28:08.850 INFO:tasks.workunit.client.1.smithi135.stdout:0/33: creat d0/d1/fa x:0 0 0 2024-03-30T04:28:08.852 INFO:tasks.workunit.client.1.smithi135.stdout:9/11: dwrite f1 [0,4194304] 0 2024-03-30T04:28:08.853 INFO:tasks.workunit.client.0.smithi123.stdout:3/462: dread - d13/f80 zero size 2024-03-30T04:28:08.856 INFO:tasks.workunit.client.1.smithi135.stdout:3/24: write d3/f5 [409953,29790] 0 2024-03-30T04:28:08.857 INFO:tasks.workunit.client.0.smithi123.stdout:2/397: dwrite d0/d1f/f24 [0,4194304] 0 2024-03-30T04:28:08.860 INFO:tasks.workunit.client.1.smithi135.stdout:8/12: dwrite f0 [0,4194304] 0 2024-03-30T04:28:08.860 INFO:tasks.workunit.client.1.smithi135.stdout:5/24: mknod d1/cb 0 2024-03-30T04:28:08.861 INFO:tasks.workunit.client.0.smithi123.stdout:6/527: dwrite d10/fb2 [0,4194304] 0 2024-03-30T04:28:08.861 INFO:tasks.workunit.client.1.smithi135.stdout:7/17: fsync d4/f8 0 2024-03-30T04:28:08.863 INFO:tasks.workunit.client.0.smithi123.stdout:1/524: mknod db/dd/d42/d46/d56/d5c/cab 0 2024-03-30T04:28:08.867 INFO:tasks.workunit.client.1.smithi135.stdout:8/13: readlink l3 0 2024-03-30T04:28:08.869 INFO:tasks.workunit.client.1.smithi135.stdout:0/34: dread d0/f3 [0,4194304] 0 2024-03-30T04:28:08.871 INFO:tasks.workunit.client.1.smithi135.stdout:5/25: link d1/cb d1/d3/cc 0 2024-03-30T04:28:08.876 INFO:tasks.workunit.client.0.smithi123.stdout:4/425: rename d2/d4/l12 to d2/d4/d2c/d42/d80/l97 0 2024-03-30T04:28:08.876 INFO:tasks.workunit.client.1.smithi135.stdout:8/14: link f0 f4 0 2024-03-30T04:28:08.880 INFO:tasks.workunit.client.1.smithi135.stdout:0/35: write d0/d5/f8 [308946,95199] 0 2024-03-30T04:28:08.887 INFO:tasks.workunit.client.0.smithi123.stdout:8/372: sync 2024-03-30T04:28:08.888 INFO:tasks.workunit.client.0.smithi123.stdout:2/398: mknod d0/d1f/d5e/d71/c7f 0 2024-03-30T04:28:08.890 INFO:tasks.workunit.client.1.smithi135.stdout:0/36: truncate d0/d1/fa 996820 0 2024-03-30T04:28:08.892 INFO:tasks.workunit.client.0.smithi123.stdout:6/528: dwrite d10/d13/f8b [0,4194304] 0 2024-03-30T04:28:08.896 INFO:tasks.workunit.client.0.smithi123.stdout:3/463: dwrite d13/d24/d31/d23/f88 [0,4194304] 0 2024-03-30T04:28:08.899 INFO:tasks.workunit.client.1.smithi135.stdout:7/18: dwrite d4/f8 [0,4194304] 0 2024-03-30T04:28:08.903 INFO:tasks.workunit.client.0.smithi123.stdout:8/373: mknod da/d27/d3c/c66 0 2024-03-30T04:28:08.908 INFO:tasks.workunit.client.1.smithi135.stdout:7/19: getdents d4/d6 0 2024-03-30T04:28:08.908 INFO:tasks.workunit.client.1.smithi135.stdout:5/26: dwrite d1/f4 [0,4194304] 0 2024-03-30T04:28:08.909 INFO:tasks.workunit.client.0.smithi123.stdout:1/525: dwrite db/dd/f54 [0,4194304] 0 2024-03-30T04:28:08.913 INFO:tasks.workunit.client.1.smithi135.stdout:9/12: dread f2 [0,4194304] 0 2024-03-30T04:28:08.917 INFO:tasks.workunit.client.0.smithi123.stdout:4/426: link d2/d4/d7/l79 d2/d34/l98 0 2024-03-30T04:28:08.922 INFO:tasks.workunit.client.0.smithi123.stdout:4/427: write d2/d4/d2c/d42/d35/d5d/d6f/f73 [963153,67276] 0 2024-03-30T04:28:08.923 INFO:tasks.workunit.client.0.smithi123.stdout:3/464: mknod d13/d24/d31/d30/c8f 0 2024-03-30T04:28:08.924 INFO:tasks.workunit.client.0.smithi123.stdout:8/374: mknod da/dc/d14/d18/d21/c67 0 2024-03-30T04:28:08.927 INFO:tasks.workunit.client.1.smithi135.stdout:9/13: creat f3 x:0 0 0 2024-03-30T04:28:08.927 INFO:tasks.workunit.client.1.smithi135.stdout:8/15: dwrite f4 [4194304,4194304] 0 2024-03-30T04:28:08.928 INFO:tasks.workunit.client.0.smithi123.stdout:8/375: stat da/dc/d14/d18 0 2024-03-30T04:28:08.929 INFO:tasks.workunit.client.0.smithi123.stdout:1/526: rename db/dd/d42/f68 to db/dd/d42/d46/d5d/d62/fac 0 2024-03-30T04:28:08.939 INFO:tasks.workunit.client.1.smithi135.stdout:0/37: dwrite d0/d5/f8 [0,4194304] 0 2024-03-30T04:28:08.939 INFO:tasks.workunit.client.1.smithi135.stdout:9/14: dread f2 [0,4194304] 0 2024-03-30T04:28:08.941 INFO:tasks.workunit.client.0.smithi123.stdout:2/399: unlink d0/d1f/d30/c6b 0 2024-03-30T04:28:08.945 INFO:tasks.workunit.client.0.smithi123.stdout:8/376: dread - da/f55 zero size 2024-03-30T04:28:08.950 INFO:tasks.workunit.client.0.smithi123.stdout:3/465: mkdir d13/d24/d39/d79/d90 0 2024-03-30T04:28:08.950 INFO:tasks.workunit.client.0.smithi123.stdout:4/428: mknod d2/d4/d7/d1a/c99 0 2024-03-30T04:28:08.950 INFO:tasks.workunit.client.1.smithi135.stdout:9/15: dread f2 [0,4194304] 0 2024-03-30T04:28:08.951 INFO:tasks.workunit.client.1.smithi135.stdout:9/16: chown f1 416316476 1 2024-03-30T04:28:08.951 INFO:tasks.workunit.client.1.smithi135.stdout:9/17: dread - f3 zero size 2024-03-30T04:28:08.952 INFO:tasks.workunit.client.0.smithi123.stdout:4/429: chown d2/d4/d2c/d42/d80/d87 209502 1 2024-03-30T04:28:08.952 INFO:tasks.workunit.client.0.smithi123.stdout:6/529: getdents d10/d82 0 2024-03-30T04:28:08.955 INFO:tasks.workunit.client.1.smithi135.stdout:8/16: dread f4 [0,4194304] 0 2024-03-30T04:28:08.955 INFO:tasks.workunit.client.0.smithi123.stdout:3/466: creat d13/d24/d31/d23/f91 x:0 0 0 2024-03-30T04:28:08.955 INFO:tasks.workunit.client.1.smithi135.stdout:0/38: dread d0/d1/fa [0,4194304] 0 2024-03-30T04:28:08.956 INFO:tasks.workunit.client.0.smithi123.stdout:8/377: symlink da/d27/d3c/d64/l68 0 2024-03-30T04:28:08.958 INFO:tasks.workunit.client.1.smithi135.stdout:8/17: write f0 [9434496,94702] 0 2024-03-30T04:28:08.960 INFO:tasks.workunit.client.0.smithi123.stdout:6/530: write d10/f1b [1976286,45853] 0 2024-03-30T04:28:08.963 INFO:tasks.workunit.client.1.smithi135.stdout:0/39: dread d0/f3 [0,4194304] 0 2024-03-30T04:28:08.965 INFO:tasks.workunit.client.0.smithi123.stdout:1/527: dread db/f6e [0,4194304] 0 2024-03-30T04:28:08.966 INFO:tasks.workunit.client.1.smithi135.stdout:0/40: readlink d0/d5/l6 0 2024-03-30T04:28:08.967 INFO:tasks.workunit.client.0.smithi123.stdout:3/467: symlink d13/d24/d31/d23/d3a/d3c/l92 0 2024-03-30T04:28:08.968 INFO:tasks.workunit.client.0.smithi123.stdout:4/430: mknod d2/d4/d2c/d42/d80/d87/c9a 0 2024-03-30T04:28:08.969 INFO:tasks.workunit.client.1.smithi135.stdout:7/20: truncate f1 784870 0 2024-03-30T04:28:08.969 INFO:tasks.workunit.client.0.smithi123.stdout:1/528: readlink db/lf 0 2024-03-30T04:28:08.972 INFO:tasks.workunit.client.0.smithi123.stdout:3/468: write d13/f8a [504396,108342] 0 2024-03-30T04:28:08.974 INFO:tasks.workunit.client.0.smithi123.stdout:3/469: dread - d13/f80 zero size 2024-03-30T04:28:08.974 INFO:tasks.workunit.client.0.smithi123.stdout:2/400: rename d0/d1f/d1c/d39/d44 to d0/d1f/d1c/d80 0 2024-03-30T04:28:08.977 INFO:tasks.workunit.client.0.smithi123.stdout:4/431: mkdir d2/d4/d2c/d42/d35/d9b 0 2024-03-30T04:28:08.978 INFO:tasks.workunit.client.1.smithi135.stdout:8/18: dread f0 [0,4194304] 0 2024-03-30T04:28:08.980 INFO:tasks.workunit.client.1.smithi135.stdout:8/19: creat f5 x:0 0 0 2024-03-30T04:28:08.981 INFO:tasks.workunit.client.0.smithi123.stdout:8/378: symlink da/d27/d3c/d2d/d5c/l69 0 2024-03-30T04:28:08.982 INFO:tasks.workunit.client.1.smithi135.stdout:8/20: truncate f5 613718 0 2024-03-30T04:28:08.986 INFO:tasks.workunit.client.0.smithi123.stdout:2/401: mkdir d0/d1f/d16/d81 0 2024-03-30T04:28:08.986 INFO:tasks.workunit.client.0.smithi123.stdout:3/470: mknod d13/d24/d39/d79/c93 0 2024-03-30T04:28:08.986 INFO:tasks.workunit.client.0.smithi123.stdout:8/379: chown da/d27/l47 1890610 1 2024-03-30T04:28:08.986 INFO:tasks.workunit.client.0.smithi123.stdout:4/432: fdatasync d2/d26/f60 0 2024-03-30T04:28:08.987 INFO:tasks.workunit.client.0.smithi123.stdout:6/531: dread d10/d20/dac/d42/f47 [0,4194304] 0 2024-03-30T04:28:08.987 INFO:tasks.workunit.client.0.smithi123.stdout:1/529: creat db/dd/d42/d46/d39/fad x:0 0 0 2024-03-30T04:28:08.989 INFO:tasks.workunit.client.1.smithi135.stdout:5/27: fdatasync d1/f4 0 2024-03-30T04:28:08.989 INFO:tasks.workunit.client.1.smithi135.stdout:9/18: truncate f1 3442250 0 2024-03-30T04:28:08.991 INFO:tasks.workunit.client.1.smithi135.stdout:5/28: chown d1/d7/f8 2696 1 2024-03-30T04:28:08.992 INFO:tasks.workunit.client.0.smithi123.stdout:2/402: mkdir d0/d1f/d82 0 2024-03-30T04:28:08.995 INFO:tasks.workunit.client.1.smithi135.stdout:9/19: getdents . 0 2024-03-30T04:28:08.995 INFO:tasks.workunit.client.0.smithi123.stdout:4/433: symlink d2/d4/d2c/d42/l9c 0 2024-03-30T04:28:08.996 INFO:tasks.workunit.client.0.smithi123.stdout:6/532: mknod d10/d13/d45/d77/cb4 0 2024-03-30T04:28:08.998 INFO:tasks.workunit.client.0.smithi123.stdout:1/530: dread - db/dd/d42/d46/d51/d72/fa6 zero size 2024-03-30T04:28:09.001 INFO:tasks.workunit.client.0.smithi123.stdout:8/380: dread da/dc/f10 [0,4194304] 0 2024-03-30T04:28:09.002 INFO:tasks.workunit.client.0.smithi123.stdout:1/531: chown db/dd/d42/d8f/ca9 2 1 2024-03-30T04:28:09.005 INFO:tasks.workunit.client.1.smithi135.stdout:0/41: dwrite d0/f3 [0,4194304] 0 2024-03-30T04:28:09.005 INFO:tasks.workunit.client.1.smithi135.stdout:8/21: fdatasync f0 0 2024-03-30T04:28:09.007 INFO:tasks.workunit.client.0.smithi123.stdout:5/414: sync 2024-03-30T04:28:09.007 INFO:tasks.workunit.client.0.smithi123.stdout:7/392: sync 2024-03-30T04:28:09.016 INFO:tasks.workunit.client.1.smithi135.stdout:0/42: readlink d0/d5/l6 0 2024-03-30T04:28:09.019 INFO:tasks.workunit.client.0.smithi123.stdout:6/533: write d10/f27 [3221269,38437] 0 2024-03-30T04:28:09.021 INFO:tasks.workunit.client.0.smithi123.stdout:1/532: dread - db/dd/d42/f6f zero size 2024-03-30T04:28:09.027 INFO:tasks.workunit.client.0.smithi123.stdout:7/393: write d2/da/d11/d61/f72 [388609,69377] 0 2024-03-30T04:28:09.031 INFO:tasks.workunit.client.0.smithi123.stdout:5/415: creat dd/d19/d3e/d5b/f8a x:0 0 0 2024-03-30T04:28:09.032 INFO:tasks.workunit.client.1.smithi135.stdout:5/29: dwrite d1/d7/f8 [0,4194304] 0 2024-03-30T04:28:09.035 INFO:tasks.workunit.client.0.smithi123.stdout:1/533: unlink db/dd/d42/c5b 0 2024-03-30T04:28:09.038 INFO:tasks.workunit.client.0.smithi123.stdout:3/471: dwrite d13/d24/f70 [0,4194304] 0 2024-03-30T04:28:09.042 INFO:tasks.workunit.client.0.smithi123.stdout:0/422: sync 2024-03-30T04:28:09.042 INFO:tasks.workunit.client.0.smithi123.stdout:9/428: sync 2024-03-30T04:28:09.044 INFO:tasks.workunit.client.0.smithi123.stdout:5/416: rmdir dd/d19/d3e 39 2024-03-30T04:28:09.045 INFO:tasks.workunit.client.0.smithi123.stdout:9/429: stat d6/f15 0 2024-03-30T04:28:09.048 INFO:tasks.workunit.client.0.smithi123.stdout:2/403: dwrite d0/f10 [4194304,4194304] 0 2024-03-30T04:28:09.052 INFO:tasks.workunit.client.0.smithi123.stdout:4/434: dwrite d2/d26/f4f [0,4194304] 0 2024-03-30T04:28:09.052 INFO:tasks.workunit.client.1.smithi135.stdout:9/20: dwrite f2 [0,4194304] 0 2024-03-30T04:28:09.058 INFO:tasks.workunit.client.0.smithi123.stdout:8/381: dwrite da/d27/d3c/d2d/f57 [0,4194304] 0 2024-03-30T04:28:09.062 INFO:tasks.workunit.client.0.smithi123.stdout:2/404: symlink d0/d37/l83 0 2024-03-30T04:28:09.063 INFO:tasks.workunit.client.1.smithi135.stdout:5/30: mknod d1/da/cd 0 2024-03-30T04:28:09.064 INFO:tasks.workunit.client.1.smithi135.stdout:7/21: dwrite f1 [0,4194304] 0 2024-03-30T04:28:09.068 INFO:tasks.workunit.client.0.smithi123.stdout:0/423: rename d6/dd/d18/d2f/c80 to d6/d52/c92 0 2024-03-30T04:28:09.070 INFO:tasks.workunit.client.0.smithi123.stdout:5/417: chown dd/d19/d3e/d5b/f5e 8268585 1 2024-03-30T04:28:09.071 INFO:tasks.workunit.client.0.smithi123.stdout:8/382: symlink da/d27/d3c/d2d/l6a 0 2024-03-30T04:28:09.072 INFO:tasks.workunit.client.1.smithi135.stdout:9/21: chown f3 2588938 1 2024-03-30T04:28:09.080 INFO:tasks.workunit.client.1.smithi135.stdout:7/22: mkdir d4/d9 0 2024-03-30T04:28:09.080 INFO:tasks.workunit.client.0.smithi123.stdout:4/435: mkdir d2/d4/d2c/d42/d80/d87/d91/d9d 0 2024-03-30T04:28:09.080 INFO:tasks.workunit.client.0.smithi123.stdout:5/418: truncate dd/f74 400370 0 2024-03-30T04:28:09.080 INFO:tasks.workunit.client.0.smithi123.stdout:6/534: dread d10/fb2 [0,4194304] 0 2024-03-30T04:28:09.080 INFO:tasks.workunit.client.0.smithi123.stdout:8/383: chown da/dc/f20 782 1 2024-03-30T04:28:09.082 INFO:tasks.workunit.client.1.smithi135.stdout:8/22: dwrite f5 [0,4194304] 0 2024-03-30T04:28:09.084 INFO:tasks.workunit.client.1.smithi135.stdout:7/23: dread - f3 zero size 2024-03-30T04:28:09.084 INFO:tasks.workunit.client.0.smithi123.stdout:4/436: write d2/f20 [4012929,35655] 0 2024-03-30T04:28:09.090 INFO:tasks.workunit.client.0.smithi123.stdout:9/430: dwrite d6/d23/f5d [0,4194304] 0 2024-03-30T04:28:09.092 INFO:tasks.workunit.client.1.smithi135.stdout:7/24: chown d4/l5 475 1 2024-03-30T04:28:09.092 INFO:tasks.workunit.client.1.smithi135.stdout:8/23: readlink l3 0 2024-03-30T04:28:09.095 INFO:tasks.workunit.client.0.smithi123.stdout:8/384: symlink da/dc/l6b 0 2024-03-30T04:28:09.098 INFO:tasks.workunit.client.0.smithi123.stdout:5/419: creat dd/d19/d3e/d5b/d62/f8b x:0 0 0 2024-03-30T04:28:09.099 INFO:tasks.workunit.client.0.smithi123.stdout:6/535: fsync d10/d20/dac/f1f 0 2024-03-30T04:28:09.101 INFO:tasks.workunit.client.1.smithi135.stdout:8/24: mkdir d6 0 2024-03-30T04:28:09.102 INFO:tasks.workunit.client.1.smithi135.stdout:7/25: read f0 [573142,53694] 0 2024-03-30T04:28:09.105 INFO:tasks.workunit.client.1.smithi135.stdout:5/31: dread d1/f4 [0,4194304] 0 2024-03-30T04:28:09.105 INFO:tasks.workunit.client.0.smithi123.stdout:0/424: creat d6/dd/d15/d28/d4b/d8a/f93 x:0 0 0 2024-03-30T04:28:09.108 INFO:tasks.workunit.client.1.smithi135.stdout:8/25: readlink l1 0 2024-03-30T04:28:09.112 INFO:tasks.workunit.client.1.smithi135.stdout:0/43: dwrite d0/d1/fa [0,4194304] 0 2024-03-30T04:28:09.115 INFO:tasks.workunit.client.0.smithi123.stdout:3/472: dwrite d13/d24/d31/d23/d3a/d3c/f8b [0,4194304] 0 2024-03-30T04:28:09.121 INFO:tasks.workunit.client.0.smithi123.stdout:9/431: truncate d6/d1f/d4d/f5b 2550893 0 2024-03-30T04:28:09.123 INFO:tasks.workunit.client.0.smithi123.stdout:8/385: write da/f3d [284556,102652] 0 2024-03-30T04:28:09.124 INFO:tasks.workunit.client.0.smithi123.stdout:6/536: creat d10/d13/d6a/fb5 x:0 0 0 2024-03-30T04:28:09.126 INFO:tasks.workunit.client.1.smithi135.stdout:9/22: dwrite f3 [0,4194304] 0 2024-03-30T04:28:09.132 INFO:tasks.workunit.client.0.smithi123.stdout:5/420: mkdir dd/d19/d1f/d3c/d72/d8c 0 2024-03-30T04:28:09.132 INFO:tasks.workunit.client.0.smithi123.stdout:2/405: dwrite d0/d1f/d1c/d39/d4b/d4c/d5f/f67 [0,4194304] 0 2024-03-30T04:28:09.133 INFO:tasks.workunit.client.0.smithi123.stdout:3/473: fsync d13/d24/d31/d23/f2b 0 2024-03-30T04:28:09.139 INFO:tasks.workunit.client.1.smithi135.stdout:9/23: rename f3 to f4 0 2024-03-30T04:28:09.140 INFO:tasks.workunit.client.0.smithi123.stdout:7/394: dread d2/da/d11/f1e [0,4194304] 0 2024-03-30T04:28:09.142 INFO:tasks.workunit.client.1.smithi135.stdout:8/26: dread f5 [0,4194304] 0 2024-03-30T04:28:09.142 INFO:tasks.workunit.client.0.smithi123.stdout:1/534: dread db/dd/d42/d46/d5d/d62/f6b [0,4194304] 0 2024-03-30T04:28:09.143 INFO:tasks.workunit.client.1.smithi135.stdout:9/24: symlink l5 0 2024-03-30T04:28:09.144 INFO:tasks.workunit.client.0.smithi123.stdout:6/537: fsync d10/d20/dac/f2c 0 2024-03-30T04:28:09.144 INFO:tasks.workunit.client.0.smithi123.stdout:9/432: rmdir d6/d1f/d72 39 2024-03-30T04:28:09.148 INFO:tasks.workunit.client.0.smithi123.stdout:8/386: mkdir da/d27/d56/d6c 0 2024-03-30T04:28:09.148 INFO:tasks.workunit.client.1.smithi135.stdout:0/44: dread d0/f3 [0,4194304] 0 2024-03-30T04:28:09.149 INFO:tasks.workunit.client.1.smithi135.stdout:5/32: dread d1/d7/f8 [0,4194304] 0 2024-03-30T04:28:09.150 INFO:tasks.workunit.client.1.smithi135.stdout:7/26: dwrite f1 [0,4194304] 0 2024-03-30T04:28:09.152 INFO:tasks.workunit.client.0.smithi123.stdout:4/437: dwrite d2/d4/d2c/d42/f8b [0,4194304] 0 2024-03-30T04:28:09.154 INFO:tasks.workunit.client.1.smithi135.stdout:7/27: fdatasync d4/f8 0 2024-03-30T04:28:09.154 INFO:tasks.workunit.client.1.smithi135.stdout:5/33: symlink d1/le 0 2024-03-30T04:28:09.155 INFO:tasks.workunit.client.0.smithi123.stdout:0/425: rename d6/dd/d16/d32/c5f to d6/dd/d18/c94 0 2024-03-30T04:28:09.156 INFO:tasks.workunit.client.0.smithi123.stdout:3/474: mknod d13/d24/d31/d30/c94 0 2024-03-30T04:28:09.156 INFO:tasks.workunit.client.1.smithi135.stdout:7/28: stat d4 0 2024-03-30T04:28:09.158 INFO:tasks.workunit.client.1.smithi135.stdout:5/34: creat d1/d7/ff x:0 0 0 2024-03-30T04:28:09.160 INFO:tasks.workunit.client.1.smithi135.stdout:5/35: chown d1 678 1 2024-03-30T04:28:09.160 INFO:tasks.workunit.client.0.smithi123.stdout:7/395: write d2/da/d11/d34/f70 [3349370,108427] 0 2024-03-30T04:28:09.160 INFO:tasks.workunit.client.0.smithi123.stdout:5/421: read dd/d19/d1f/d35/f40 [1287978,50877] 0 2024-03-30T04:28:09.161 INFO:tasks.workunit.client.0.smithi123.stdout:0/426: readlink d6/dd/d16/l2e 0 2024-03-30T04:28:09.161 INFO:tasks.workunit.client.1.smithi135.stdout:7/29: symlink d4/d9/la 0 2024-03-30T04:28:09.163 INFO:tasks.workunit.client.0.smithi123.stdout:9/433: unlink d6/c60 0 2024-03-30T04:28:09.164 INFO:tasks.workunit.client.0.smithi123.stdout:7/396: readlink d2/da/d11/l2b 0 2024-03-30T04:28:09.164 INFO:tasks.workunit.client.1.smithi135.stdout:7/30: chown f1 10934300 1 2024-03-30T04:28:09.164 INFO:tasks.workunit.client.1.smithi135.stdout:5/36: read d1/f4 [445657,84417] 0 2024-03-30T04:28:09.167 INFO:tasks.workunit.client.1.smithi135.stdout:5/37: fsync d1/f4 0 2024-03-30T04:28:09.169 INFO:tasks.workunit.client.0.smithi123.stdout:7/397: chown d2/da/d11/f48 360472 1 2024-03-30T04:28:09.170 INFO:tasks.workunit.client.1.smithi135.stdout:8/27: dread f0 [0,4194304] 0 2024-03-30T04:28:09.172 INFO:tasks.workunit.client.1.smithi135.stdout:8/28: write f4 [7605357,16406] 0 2024-03-30T04:28:09.173 INFO:tasks.workunit.client.1.smithi135.stdout:5/38: chown d1/c2 508597579 1 2024-03-30T04:28:09.177 INFO:tasks.workunit.client.1.smithi135.stdout:0/45: dwrite d0/d1/fa [0,4194304] 0 2024-03-30T04:28:09.185 INFO:tasks.workunit.client.0.smithi123.stdout:1/535: dread - db/dd/d42/d46/d39/f93 zero size 2024-03-30T04:28:09.185 INFO:tasks.workunit.client.1.smithi135.stdout:8/29: getdents d6 0 2024-03-30T04:28:09.185 INFO:tasks.workunit.client.1.smithi135.stdout:5/39: creat d1/d7/f10 x:0 0 0 2024-03-30T04:28:09.187 INFO:tasks.workunit.client.0.smithi123.stdout:2/406: dwrite d0/d1f/d1c/d2a/f7c [0,4194304] 0 2024-03-30T04:28:09.187 INFO:tasks.workunit.client.0.smithi123.stdout:7/398: mknod d2/da/d22/c82 0 2024-03-30T04:28:09.189 INFO:tasks.workunit.client.0.smithi123.stdout:6/538: mknod d10/d20/d3f/da3/cb6 0 2024-03-30T04:28:09.189 INFO:tasks.workunit.client.0.smithi123.stdout:8/387: mkdir da/d27/d56/d6c/d6d 0 2024-03-30T04:28:09.192 INFO:tasks.workunit.client.1.smithi135.stdout:5/40: mkdir d1/d7/d11 0 2024-03-30T04:28:09.192 INFO:tasks.workunit.client.1.smithi135.stdout:7/31: dwrite f2 [0,4194304] 0 2024-03-30T04:28:09.192 INFO:tasks.workunit.client.0.smithi123.stdout:7/399: write f0 [579828,30416] 0 2024-03-30T04:28:09.194 INFO:tasks.workunit.client.0.smithi123.stdout:0/427: link d6/dd/d16/f88 d6/dd/d31/d35/d3a/f95 0 2024-03-30T04:28:09.196 INFO:tasks.workunit.client.0.smithi123.stdout:1/536: mkdir db/dd/d5f/dae 0 2024-03-30T04:28:09.197 INFO:tasks.workunit.client.1.smithi135.stdout:5/41: chown d1/cb 287165857 1 2024-03-30T04:28:09.198 INFO:tasks.workunit.client.0.smithi123.stdout:0/428: chown d6/dd/d15/d28 12 1 2024-03-30T04:28:09.204 INFO:tasks.workunit.client.0.smithi123.stdout:3/475: dwrite d13/d24/d31/d23/f88 [0,4194304] 0 2024-03-30T04:28:09.205 INFO:tasks.workunit.client.1.smithi135.stdout:7/32: rename d4/d9 to d4/db 0 2024-03-30T04:28:09.206 INFO:tasks.workunit.client.1.smithi135.stdout:8/30: dread f0 [4194304,4194304] 0 2024-03-30T04:28:09.210 INFO:tasks.workunit.client.0.smithi123.stdout:8/388: creat da/dc/d14/f6e x:0 0 0 2024-03-30T04:28:09.213 INFO:tasks.workunit.client.0.smithi123.stdout:4/438: dread d2/d4/d2c/d42/f4a [0,4194304] 0 2024-03-30T04:28:09.217 INFO:tasks.workunit.client.1.smithi135.stdout:0/46: dwrite d0/d5/f8 [0,4194304] 0 2024-03-30T04:28:09.219 INFO:tasks.workunit.client.0.smithi123.stdout:5/422: rename dd/d41/f51 to dd/d19/d3e/d5b/f8d 0 2024-03-30T04:28:09.221 INFO:tasks.workunit.client.1.smithi135.stdout:5/42: dread d1/f4 [0,4194304] 0 2024-03-30T04:28:09.221 INFO:tasks.workunit.client.0.smithi123.stdout:1/537: mkdir db/dd/d42/d46/d56/daf 0 2024-03-30T04:28:09.222 INFO:tasks.workunit.client.1.smithi135.stdout:5/43: chown d1/d3 62 1 2024-03-30T04:28:09.227 INFO:tasks.workunit.client.1.smithi135.stdout:0/47: unlink d0/d5/f8 0 2024-03-30T04:28:09.229 INFO:tasks.workunit.client.1.smithi135.stdout:7/33: dread f2 [0,4194304] 0 2024-03-30T04:28:09.230 INFO:tasks.workunit.client.0.smithi123.stdout:7/400: mkdir d2/dc/d83 0 2024-03-30T04:28:09.232 INFO:tasks.workunit.client.1.smithi135.stdout:8/31: dread f4 [0,4194304] 0 2024-03-30T04:28:09.232 INFO:tasks.workunit.client.1.smithi135.stdout:7/34: write f1 [3743134,73658] 0 2024-03-30T04:28:09.232 INFO:tasks.workunit.client.0.smithi123.stdout:3/476: creat d13/d24/d31/d23/d63/f95 x:0 0 0 2024-03-30T04:28:09.233 INFO:tasks.workunit.client.0.smithi123.stdout:1/538: dread - db/dd/d42/d46/d51/d72/fa6 zero size 2024-03-30T04:28:09.236 INFO:tasks.workunit.client.0.smithi123.stdout:7/401: chown d2/d4d/f50 1 1 2024-03-30T04:28:09.236 INFO:tasks.workunit.client.1.smithi135.stdout:7/35: write f1 [3299609,15702] 0 2024-03-30T04:28:09.238 INFO:tasks.workunit.client.0.smithi123.stdout:3/477: write d13/d24/d31/f75 [10402,81496] 0 2024-03-30T04:28:09.241 INFO:tasks.workunit.client.1.smithi135.stdout:7/36: chown d4/l7 58 1 2024-03-30T04:28:09.242 INFO:tasks.workunit.client.1.smithi135.stdout:8/32: creat d6/f7 x:0 0 0 2024-03-30T04:28:09.242 INFO:tasks.workunit.client.0.smithi123.stdout:0/429: write d6/f1b [2101050,84991] 0 2024-03-30T04:28:09.243 INFO:tasks.workunit.client.1.smithi135.stdout:0/48: truncate d0/f3 3032408 0 2024-03-30T04:28:09.247 INFO:tasks.workunit.client.1.smithi135.stdout:8/33: mkdir d6/d8 0 2024-03-30T04:28:09.248 INFO:tasks.workunit.client.1.smithi135.stdout:5/44: dwrite d1/f4 [0,4194304] 0 2024-03-30T04:28:09.249 INFO:tasks.workunit.client.0.smithi123.stdout:2/407: write d0/d1f/d16/f18 [2437860,88300] 0 2024-03-30T04:28:09.252 INFO:tasks.workunit.client.1.smithi135.stdout:5/45: write d1/d7/f8 [3827296,88070] 0 2024-03-30T04:28:09.252 INFO:tasks.workunit.client.0.smithi123.stdout:2/408: stat d0/d1f/d1c/d39/d4b/d4c/d5f 0 2024-03-30T04:28:09.254 INFO:tasks.workunit.client.0.smithi123.stdout:6/539: link d10/d13/d6a/la5 d10/d20/d3f/da3/lb7 0 2024-03-30T04:28:09.258 INFO:tasks.workunit.client.0.smithi123.stdout:6/540: write d10/d20/dac/d5b/fa8 [161511,100167] 0 2024-03-30T04:28:09.259 INFO:tasks.workunit.client.0.smithi123.stdout:7/402: mknod d2/dc/d56/c84 0 2024-03-30T04:28:09.259 INFO:tasks.workunit.client.0.smithi123.stdout:0/430: creat d6/d52/f96 x:0 0 0 2024-03-30T04:28:09.262 INFO:tasks.workunit.client.0.smithi123.stdout:3/478: rename d13/d24/d31/c85 to d13/d24/d31/d23/d3a/d3c/d4e/d5d/c96 0 2024-03-30T04:28:09.263 INFO:tasks.workunit.client.0.smithi123.stdout:2/409: mknod d0/d1f/d1c/d39/d4b/c84 0 2024-03-30T04:28:09.265 INFO:tasks.workunit.client.0.smithi123.stdout:7/403: write d2/da/d11/f19 [320493,30945] 0 2024-03-30T04:28:09.268 INFO:tasks.workunit.client.0.smithi123.stdout:1/539: creat db/dd/d42/d46/d56/da2/fb0 x:0 0 0 2024-03-30T04:28:09.271 INFO:tasks.workunit.client.0.smithi123.stdout:1/540: readlink db/l40 0 2024-03-30T04:28:09.272 INFO:tasks.workunit.client.1.smithi135.stdout:0/49: dread d0/d1/fa [0,4194304] 0 2024-03-30T04:28:09.272 INFO:tasks.workunit.client.0.smithi123.stdout:2/410: truncate d0/d1f/d5e/f4f 963550 0 2024-03-30T04:28:09.273 INFO:tasks.workunit.client.0.smithi123.stdout:0/431: dread - d6/dd/d16/d32/d4e/d69/f81 zero size 2024-03-30T04:28:09.276 INFO:tasks.workunit.client.0.smithi123.stdout:7/404: mkdir d2/da/d11/d34/d85 0 2024-03-30T04:28:09.277 INFO:tasks.workunit.client.1.smithi135.stdout:8/34: dwrite f0 [4194304,4194304] 0 2024-03-30T04:28:09.279 INFO:tasks.workunit.client.1.smithi135.stdout:7/37: dwrite f0 [0,4194304] 0 2024-03-30T04:28:09.281 INFO:tasks.workunit.client.0.smithi123.stdout:1/541: rmdir db/dd/d42/d46/d56/d5c 39 2024-03-30T04:28:09.281 INFO:tasks.workunit.client.0.smithi123.stdout:3/479: write d13/d24/d31/d23/d3a/f40 [254909,47208] 0 2024-03-30T04:28:09.282 INFO:tasks.workunit.client.0.smithi123.stdout:8/389: truncate da/dc/d14/d18/d21/f45 6744042 0 2024-03-30T04:28:09.282 INFO:tasks.workunit.client.1.smithi135.stdout:7/38: truncate f3 646736 0 2024-03-30T04:28:09.285 INFO:tasks.workunit.client.1.smithi135.stdout:7/39: write f1 [4006736,25232] 0 2024-03-30T04:28:09.290 INFO:tasks.workunit.client.0.smithi123.stdout:2/411: write d0/d1f/d1c/d40/f43 [2648148,86580] 0 2024-03-30T04:28:09.291 INFO:tasks.workunit.client.1.smithi135.stdout:5/46: dwrite d1/d7/ff [0,4194304] 0 2024-03-30T04:28:09.291 INFO:tasks.workunit.client.0.smithi123.stdout:7/405: creat d2/da/d22/f86 x:0 0 0 2024-03-30T04:28:09.295 INFO:tasks.workunit.client.1.smithi135.stdout:7/40: mknod d4/cc 0 2024-03-30T04:28:09.295 INFO:tasks.workunit.client.1.smithi135.stdout:8/35: creat d6/d8/f9 x:0 0 0 2024-03-30T04:28:09.296 INFO:tasks.workunit.client.1.smithi135.stdout:5/47: write d1/d7/ff [3702019,15160] 0 2024-03-30T04:28:09.297 INFO:tasks.workunit.client.0.smithi123.stdout:4/439: dread d2/f1c [0,4194304] 0 2024-03-30T04:28:09.299 INFO:tasks.workunit.client.1.smithi135.stdout:7/41: write d4/f8 [118479,25548] 0 2024-03-30T04:28:09.300 INFO:tasks.workunit.client.0.smithi123.stdout:2/412: read d0/d1f/d1c/d2a/f7c [3255834,97952] 0 2024-03-30T04:28:09.302 INFO:tasks.workunit.client.0.smithi123.stdout:1/542: creat db/dd/d42/d46/d51/fb1 x:0 0 0 2024-03-30T04:28:09.305 INFO:tasks.workunit.client.0.smithi123.stdout:6/541: dwrite d10/d20/dac/f2c [0,4194304] 0 2024-03-30T04:28:09.310 INFO:tasks.workunit.client.0.smithi123.stdout:7/406: creat d2/f87 x:0 0 0 2024-03-30T04:28:09.313 INFO:tasks.workunit.client.0.smithi123.stdout:4/440: unlink d2/d4/d2c/d42/d35/d5d/f92 0 2024-03-30T04:28:09.317 INFO:tasks.workunit.client.0.smithi123.stdout:1/543: unlink db/dd/l25 0 2024-03-30T04:28:09.317 INFO:tasks.workunit.client.1.smithi135.stdout:0/50: dwrite d0/d1/fa [0,4194304] 0 2024-03-30T04:28:09.318 INFO:tasks.workunit.client.0.smithi123.stdout:2/413: dread d0/d37/d57/f62 [0,4194304] 0 2024-03-30T04:28:09.319 INFO:tasks.workunit.client.0.smithi123.stdout:0/432: dread d6/fa [0,4194304] 0 2024-03-30T04:28:09.321 INFO:tasks.workunit.client.0.smithi123.stdout:4/441: readlink d2/d4/d2c/d42/d35/d5d/l8e 0 2024-03-30T04:28:09.322 INFO:tasks.workunit.client.1.smithi135.stdout:0/51: stat d0/d1/c7 0 2024-03-30T04:28:09.323 INFO:tasks.workunit.client.0.smithi123.stdout:6/542: creat d10/d20/dac/d42/fb8 x:0 0 0 2024-03-30T04:28:09.326 INFO:tasks.workunit.client.1.smithi135.stdout:7/42: dread d4/f8 [0,4194304] 0 2024-03-30T04:28:09.326 INFO:tasks.workunit.client.1.smithi135.stdout:0/52: write d0/d1/fa [4712005,127595] 0 2024-03-30T04:28:09.327 INFO:tasks.workunit.client.1.smithi135.stdout:8/36: rmdir d6/d8 39 2024-03-30T04:28:09.329 INFO:tasks.workunit.client.0.smithi123.stdout:7/407: creat d2/da/d11/d61/d55/d59/f88 x:0 0 0 2024-03-30T04:28:09.332 INFO:tasks.workunit.client.1.smithi135.stdout:5/48: dread d1/d7/f8 [0,4194304] 0 2024-03-30T04:28:09.333 INFO:tasks.workunit.client.1.smithi135.stdout:0/53: creat d0/d5/d9/fb x:0 0 0 2024-03-30T04:28:09.333 INFO:tasks.workunit.client.0.smithi123.stdout:6/543: creat d10/d13/d45/fb9 x:0 0 0 2024-03-30T04:28:09.335 INFO:tasks.workunit.client.1.smithi135.stdout:7/43: rename d4/db/la to d4/ld 0 2024-03-30T04:28:09.335 INFO:tasks.workunit.client.0.smithi123.stdout:2/414: write d0/d1f/d1c/d20/f5d [956465,101557] 0 2024-03-30T04:28:09.335 INFO:tasks.workunit.client.0.smithi123.stdout:8/390: dwrite da/dc/f42 [0,4194304] 0 2024-03-30T04:28:09.335 INFO:tasks.workunit.client.0.smithi123.stdout:4/442: read d2/d4/d2c/d42/d35/d5d/d78/f85 [205216,84219] 0 2024-03-30T04:28:09.336 INFO:tasks.workunit.client.1.smithi135.stdout:8/37: creat d6/fa x:0 0 0 2024-03-30T04:28:09.338 INFO:tasks.workunit.client.0.smithi123.stdout:0/433: creat d6/dd/d15/d28/f97 x:0 0 0 2024-03-30T04:28:09.338 INFO:tasks.workunit.client.0.smithi123.stdout:3/480: dread d13/d24/f57 [0,4194304] 0 2024-03-30T04:28:09.338 INFO:tasks.workunit.client.1.smithi135.stdout:8/38: fsync f0 0 2024-03-30T04:28:09.339 INFO:tasks.workunit.client.1.smithi135.stdout:0/54: creat d0/fc x:0 0 0 2024-03-30T04:28:09.339 INFO:tasks.workunit.client.1.smithi135.stdout:5/49: creat d1/d3/f12 x:0 0 0 2024-03-30T04:28:09.340 INFO:tasks.workunit.client.1.smithi135.stdout:8/39: truncate d6/fa 593833 0 2024-03-30T04:28:09.341 INFO:tasks.workunit.client.0.smithi123.stdout:0/434: chown d6/dd/d16/d32/d5d 41113 1 2024-03-30T04:28:09.341 INFO:tasks.workunit.client.0.smithi123.stdout:2/415: symlink d0/d1f/d1c/d39/l85 0 2024-03-30T04:28:09.343 INFO:tasks.workunit.client.0.smithi123.stdout:8/391: chown da/dc/d14/l1b 433 1 2024-03-30T04:28:09.343 INFO:tasks.workunit.client.0.smithi123.stdout:6/544: write d10/d20/dac/d5b/f85 [3367203,122795] 0 2024-03-30T04:28:09.345 INFO:tasks.workunit.client.1.smithi135.stdout:5/50: fdatasync d1/d7/f8 0 2024-03-30T04:28:09.348 INFO:tasks.workunit.client.0.smithi123.stdout:8/392: dread - da/d27/f53 zero size 2024-03-30T04:28:09.348 INFO:tasks.workunit.client.1.smithi135.stdout:5/51: write d1/d7/f10 [663957,129683] 0 2024-03-30T04:28:09.351 INFO:tasks.workunit.client.0.smithi123.stdout:3/481: write d13/d24/d31/d23/f2e [2154904,4958] 0 2024-03-30T04:28:09.358 INFO:tasks.workunit.client.0.smithi123.stdout:1/544: truncate f8 6772281 0 2024-03-30T04:28:09.361 INFO:tasks.workunit.client.0.smithi123.stdout:8/393: creat da/d27/d3c/d2d/d5c/f6f x:0 0 0 2024-03-30T04:28:09.361 INFO:tasks.workunit.client.0.smithi123.stdout:3/482: rename d13/d24/d31/d30/c94 to d13/d24/d39/d79/d90/c97 0 2024-03-30T04:28:09.362 INFO:tasks.workunit.client.1.smithi135.stdout:7/44: rmdir d4 39 2024-03-30T04:28:09.362 INFO:tasks.workunit.client.0.smithi123.stdout:6/545: mkdir d10/d13/d6a/d7a/d99/dba 0 2024-03-30T04:28:09.363 INFO:tasks.workunit.client.1.smithi135.stdout:8/40: getdents d6 0 2024-03-30T04:28:09.365 INFO:tasks.workunit.client.0.smithi123.stdout:0/435: getdents d6/dd 0 2024-03-30T04:28:09.367 INFO:tasks.workunit.client.1.smithi135.stdout:0/55: write d0/f3 [2538630,123991] 0 2024-03-30T04:28:09.368 INFO:tasks.workunit.client.0.smithi123.stdout:7/408: truncate d2/da/d11/f45 2005729 0 2024-03-30T04:28:09.371 INFO:tasks.workunit.client.0.smithi123.stdout:0/436: chown d6/dd/d15/d28/d4b/d8a/f93 1 1 2024-03-30T04:28:09.375 INFO:tasks.workunit.client.1.smithi135.stdout:7/45: rename d4/d6 to d4/db/de 0 2024-03-30T04:28:09.376 INFO:tasks.workunit.client.1.smithi135.stdout:5/52: dwrite d1/d3/f12 [0,4194304] 0 2024-03-30T04:28:09.377 INFO:tasks.workunit.client.0.smithi123.stdout:7/409: symlink d2/da/d11/d61/d55/d59/l89 0 2024-03-30T04:28:09.381 INFO:tasks.workunit.client.0.smithi123.stdout:7/410: chown d2/da/d2d 37396780 1 2024-03-30T04:28:09.383 INFO:tasks.workunit.client.0.smithi123.stdout:8/394: dread da/fb [0,4194304] 0 2024-03-30T04:28:09.384 INFO:tasks.workunit.client.1.smithi135.stdout:7/46: symlink d4/db/lf 0 2024-03-30T04:28:09.387 INFO:tasks.workunit.client.1.smithi135.stdout:8/41: dread f4 [4194304,4194304] 0 2024-03-30T04:28:09.388 INFO:tasks.workunit.client.0.smithi123.stdout:0/437: rename d6/dd/d16/d32 to d6/dd/d15/d98 0 2024-03-30T04:28:09.389 INFO:tasks.workunit.client.1.smithi135.stdout:0/56: fdatasync d0/d1/fa 0 2024-03-30T04:28:09.393 INFO:tasks.workunit.client.0.smithi123.stdout:9/434: sync 2024-03-30T04:28:09.393 INFO:tasks.workunit.client.1.smithi135.stdout:0/57: creat d0/d5/d9/fd x:0 0 0 2024-03-30T04:28:09.395 INFO:tasks.workunit.client.1.smithi135.stdout:8/42: dread d6/fa [0,4194304] 0 2024-03-30T04:28:09.395 INFO:tasks.workunit.client.1.smithi135.stdout:5/53: dread d1/f4 [0,4194304] 0 2024-03-30T04:28:09.397 INFO:tasks.workunit.client.1.smithi135.stdout:8/43: dread - d6/f7 zero size 2024-03-30T04:28:09.398 INFO:tasks.workunit.client.0.smithi123.stdout:4/443: truncate d2/d4/d7/fb 3067701 0 2024-03-30T04:28:09.398 INFO:tasks.workunit.client.0.smithi123.stdout:6/546: dwrite d10/d20/d3f/f92 [0,4194304] 0 2024-03-30T04:28:09.401 INFO:tasks.workunit.client.1.smithi135.stdout:0/58: read - d0/d5/d9/fb zero size 2024-03-30T04:28:09.402 INFO:tasks.workunit.client.1.smithi135.stdout:8/44: truncate d6/f7 471779 0 2024-03-30T04:28:09.403 INFO:tasks.workunit.client.0.smithi123.stdout:9/435: readlink d6/d58/l61 0 2024-03-30T04:28:09.407 INFO:tasks.workunit.client.1.smithi135.stdout:8/45: mkdir d6/d8/db 0 2024-03-30T04:28:09.407 INFO:tasks.workunit.client.0.smithi123.stdout:1/545: dwrite db/dd/d42/d46/d39/f8a [0,4194304] 0 2024-03-30T04:28:09.408 INFO:tasks.workunit.client.1.smithi135.stdout:8/46: rename d6 to d6/dc 22 2024-03-30T04:28:09.408 INFO:tasks.workunit.client.1.smithi135.stdout:7/47: dwrite d4/f8 [0,4194304] 0 2024-03-30T04:28:09.410 INFO:tasks.workunit.client.0.smithi123.stdout:0/438: chown d6/dd/d15/f2b 26379883 1 2024-03-30T04:28:09.411 INFO:tasks.workunit.client.1.smithi135.stdout:8/47: symlink d6/ld 0 2024-03-30T04:28:09.413 INFO:tasks.workunit.client.0.smithi123.stdout:2/416: dread d0/d1f/d1c/d39/d4b/d4c/f6a [0,4194304] 0 2024-03-30T04:28:09.417 INFO:tasks.workunit.client.0.smithi123.stdout:3/483: dwrite d13/f80 [0,4194304] 0 2024-03-30T04:28:09.420 INFO:tasks.workunit.client.0.smithi123.stdout:9/436: write d6/d23/f5d [2928540,119838] 0 2024-03-30T04:28:09.421 INFO:tasks.workunit.client.0.smithi123.stdout:6/547: symlink d10/d13/d45/lbb 0 2024-03-30T04:28:09.422 INFO:tasks.workunit.client.1.smithi135.stdout:5/54: dwrite d1/d7/f8 [0,4194304] 0 2024-03-30T04:28:09.426 INFO:tasks.workunit.client.0.smithi123.stdout:0/439: write d6/d52/f96 [865367,16887] 0 2024-03-30T04:28:09.432 INFO:tasks.workunit.client.1.smithi135.stdout:5/55: getdents d1/d3 0 2024-03-30T04:28:09.433 INFO:tasks.workunit.client.0.smithi123.stdout:3/484: rmdir d13/d24/d31/d23 39 2024-03-30T04:28:09.434 INFO:tasks.workunit.client.1.smithi135.stdout:8/48: dread f0 [4194304,4194304] 0 2024-03-30T04:28:09.438 INFO:tasks.workunit.client.0.smithi123.stdout:6/548: chown d10/d13/d6a/d7a/c8c 988 1 2024-03-30T04:28:09.440 INFO:tasks.workunit.client.0.smithi123.stdout:8/395: dwrite da/f55 [0,4194304] 0 2024-03-30T04:28:09.441 INFO:tasks.workunit.client.0.smithi123.stdout:0/440: write d6/dd/f8c [872736,17036] 0 2024-03-30T04:28:09.443 INFO:tasks.workunit.client.1.smithi135.stdout:7/48: truncate d4/f8 2548044 0 2024-03-30T04:28:09.444 INFO:tasks.workunit.client.0.smithi123.stdout:9/437: stat d6/d1f/d45/d97/d69/f6c 0 2024-03-30T04:28:09.450 INFO:tasks.workunit.client.0.smithi123.stdout:6/549: truncate d10/d20/dac/d5b/fa8 711606 0 2024-03-30T04:28:09.454 INFO:tasks.workunit.client.0.smithi123.stdout:8/396: mknod da/d27/d3c/c70 0 2024-03-30T04:28:09.455 INFO:tasks.workunit.client.1.smithi135.stdout:5/56: dwrite d1/d7/f8 [0,4194304] 0 2024-03-30T04:28:09.456 INFO:tasks.workunit.client.0.smithi123.stdout:9/438: symlink d6/d1f/d45/d97/l9e 0 2024-03-30T04:28:09.456 INFO:tasks.workunit.client.0.smithi123.stdout:0/441: rmdir d6/dd/d15/d98 39 2024-03-30T04:28:09.458 INFO:tasks.workunit.client.1.smithi135.stdout:5/57: creat d1/d7/f13 x:0 0 0 2024-03-30T04:28:09.462 INFO:tasks.workunit.client.0.smithi123.stdout:1/546: link db/dd/d42/d46/d39/fad db/dd/d42/d46/d5d/d7a/fb2 0 2024-03-30T04:28:09.462 INFO:tasks.workunit.client.1.smithi135.stdout:7/49: truncate d4/f8 3228278 0 2024-03-30T04:28:09.463 INFO:tasks.workunit.client.0.smithi123.stdout:3/485: creat d13/d24/d31/f98 x:0 0 0 2024-03-30T04:28:09.463 INFO:tasks.workunit.client.1.smithi135.stdout:8/49: truncate d6/fa 1613140 0 2024-03-30T04:28:09.464 INFO:tasks.workunit.client.0.smithi123.stdout:9/439: mknod d6/d23/c9f 0 2024-03-30T04:28:09.464 INFO:tasks.workunit.client.0.smithi123.stdout:5/423: sync 2024-03-30T04:28:09.467 INFO:tasks.workunit.client.1.smithi135.stdout:7/50: creat d4/db/f10 x:0 0 0 2024-03-30T04:28:09.467 INFO:tasks.workunit.client.0.smithi123.stdout:2/417: dwrite d0/d1f/d1c/d36/f7d [0,4194304] 0 2024-03-30T04:28:09.469 INFO:tasks.workunit.client.1.smithi135.stdout:7/51: dread - d4/db/f10 zero size 2024-03-30T04:28:09.469 INFO:tasks.workunit.client.0.smithi123.stdout:5/424: read - dd/d19/f22 zero size 2024-03-30T04:28:09.470 INFO:tasks.workunit.client.1.smithi135.stdout:8/50: stat d6/fa 0 2024-03-30T04:28:09.472 INFO:tasks.workunit.client.0.smithi123.stdout:5/425: readlink dd/d19/l31 0 2024-03-30T04:28:09.474 INFO:tasks.workunit.client.1.smithi135.stdout:5/58: dread d1/d7/f8 [0,4194304] 0 2024-03-30T04:28:09.476 INFO:tasks.workunit.client.1.smithi135.stdout:7/52: creat d4/f11 x:0 0 0 2024-03-30T04:28:09.477 INFO:tasks.workunit.client.1.smithi135.stdout:7/53: write f3 [508098,84565] 0 2024-03-30T04:28:09.478 INFO:tasks.workunit.client.0.smithi123.stdout:2/418: mknod d0/d37/d57/c86 0 2024-03-30T04:28:09.479 INFO:tasks.workunit.client.1.smithi135.stdout:7/54: fdatasync f0 0 2024-03-30T04:28:09.479 INFO:tasks.workunit.client.1.smithi135.stdout:5/59: write d1/f4 [2865286,71407] 0 2024-03-30T04:28:09.481 INFO:tasks.workunit.client.1.smithi135.stdout:8/51: rename d6/d8/f9 to d6/d8/db/fe 0 2024-03-30T04:28:09.483 INFO:tasks.workunit.client.1.smithi135.stdout:5/60: mknod d1/c14 0 2024-03-30T04:28:09.484 INFO:tasks.workunit.client.1.smithi135.stdout:8/52: rmdir d6/d8/db 39 2024-03-30T04:28:09.485 INFO:tasks.workunit.client.0.smithi123.stdout:4/444: truncate d2/d4/f22 1493725 0 2024-03-30T04:28:09.485 INFO:tasks.workunit.client.1.smithi135.stdout:7/55: getdents d4/db/de 0 2024-03-30T04:28:09.487 INFO:tasks.workunit.client.0.smithi123.stdout:3/486: symlink d13/d24/d31/d23/d63/l99 0 2024-03-30T04:28:09.487 INFO:tasks.workunit.client.1.smithi135.stdout:5/61: unlink d1/d3/l9 0 2024-03-30T04:28:09.489 INFO:tasks.workunit.client.1.smithi135.stdout:8/53: mknod d6/d8/db/cf 0 2024-03-30T04:28:09.489 INFO:tasks.workunit.client.1.smithi135.stdout:5/62: readlink d1/d3/l5 0 2024-03-30T04:28:09.491 INFO:tasks.workunit.client.1.smithi135.stdout:7/56: rename d4/db/lf to d4/db/de/l12 0 2024-03-30T04:28:09.492 INFO:tasks.workunit.client.0.smithi123.stdout:5/426: chown dd/d19/d1f/d35/c3b 74661588 1 2024-03-30T04:28:09.493 INFO:tasks.workunit.client.0.smithi123.stdout:4/445: write d2/d4/d7/d1a/f1f [6294744,90142] 0 2024-03-30T04:28:09.494 INFO:tasks.workunit.client.1.smithi135.stdout:8/54: symlink d6/d8/l10 0 2024-03-30T04:28:09.494 INFO:tasks.workunit.client.0.smithi123.stdout:0/442: truncate d6/dd/f8c 876588 0 2024-03-30T04:28:09.497 INFO:tasks.workunit.client.1.smithi135.stdout:7/57: rmdir d4/db/de 39 2024-03-30T04:28:09.499 INFO:tasks.workunit.client.1.smithi135.stdout:7/58: truncate d4/f8 3842782 0 2024-03-30T04:28:09.503 INFO:tasks.workunit.client.1.smithi135.stdout:7/59: mknod d4/c13 0 2024-03-30T04:28:09.506 INFO:tasks.workunit.client.1.smithi135.stdout:5/63: truncate d1/d3/f12 365715 0 2024-03-30T04:28:09.507 INFO:tasks.workunit.client.0.smithi123.stdout:2/419: dwrite d0/d1f/d1c/d20/f5d [0,4194304] 0 2024-03-30T04:28:09.509 INFO:tasks.workunit.client.1.smithi135.stdout:5/64: creat d1/d3/f15 x:0 0 0 2024-03-30T04:28:09.509 INFO:tasks.workunit.client.0.smithi123.stdout:9/440: truncate d6/d1f/f2e 2731448 0 2024-03-30T04:28:09.511 INFO:tasks.workunit.client.1.smithi135.stdout:5/65: symlink d1/d7/l16 0 2024-03-30T04:28:09.515 INFO:tasks.workunit.client.1.smithi135.stdout:5/66: unlink d1/d7/f8 0 2024-03-30T04:28:09.515 INFO:tasks.workunit.client.1.smithi135.stdout:7/60: dread f0 [0,4194304] 0 2024-03-30T04:28:09.516 INFO:tasks.workunit.client.0.smithi123.stdout:2/420: read d0/d1f/f17 [422715,121885] 0 2024-03-30T04:28:09.518 INFO:tasks.workunit.client.0.smithi123.stdout:5/427: dwrite dd/fe [0,4194304] 0 2024-03-30T04:28:09.518 INFO:tasks.workunit.client.1.smithi135.stdout:5/67: unlink d1/da/cd 0 2024-03-30T04:28:09.520 INFO:tasks.workunit.client.1.smithi135.stdout:5/68: fdatasync d1/d3/f15 0 2024-03-30T04:28:09.521 INFO:tasks.workunit.client.1.smithi135.stdout:7/61: creat d4/db/f14 x:0 0 0 2024-03-30T04:28:09.522 INFO:tasks.workunit.client.0.smithi123.stdout:1/547: dread db/dd/d5f/d6d/f81 [0,4194304] 0 2024-03-30T04:28:09.522 INFO:tasks.workunit.client.0.smithi123.stdout:0/443: dread d6/dd/d15/d28/f5c [0,4194304] 0 2024-03-30T04:28:09.525 INFO:tasks.workunit.client.1.smithi135.stdout:5/69: read d1/d7/f10 [542068,114156] 0 2024-03-30T04:28:09.528 INFO:tasks.workunit.client.0.smithi123.stdout:2/421: creat d0/d1f/d16/f87 x:0 0 0 2024-03-30T04:28:09.528 INFO:tasks.workunit.client.1.smithi135.stdout:5/70: mkdir d1/da/d17 0 2024-03-30T04:28:09.530 INFO:tasks.workunit.client.1.smithi135.stdout:5/71: read d1/f4 [1736838,57415] 0 2024-03-30T04:28:09.531 INFO:tasks.workunit.client.0.smithi123.stdout:5/428: creat dd/d19/d1f/d35/d3a/f8e x:0 0 0 2024-03-30T04:28:09.532 INFO:tasks.workunit.client.0.smithi123.stdout:2/422: truncate d0/d1f/f2d 1055377 0 2024-03-30T04:28:09.533 INFO:tasks.workunit.client.0.smithi123.stdout:5/429: readlink dd/d19/l25 0 2024-03-30T04:28:09.534 INFO:tasks.workunit.client.0.smithi123.stdout:0/444: mkdir d6/dd/d15/d98/d4e/d69/d6e/d99 0 2024-03-30T04:28:09.534 INFO:tasks.workunit.client.0.smithi123.stdout:3/487: dread d13/f14 [0,4194304] 0 2024-03-30T04:28:09.538 INFO:tasks.workunit.client.1.smithi135.stdout:5/72: creat d1/da/f18 x:0 0 0 2024-03-30T04:28:09.539 INFO:tasks.workunit.client.0.smithi123.stdout:9/441: creat d6/d1f/d45/fa0 x:0 0 0 2024-03-30T04:28:09.540 INFO:tasks.workunit.client.1.smithi135.stdout:7/62: dwrite f3 [0,4194304] 0 2024-03-30T04:28:09.544 INFO:tasks.workunit.client.0.smithi123.stdout:0/445: mkdir d6/dd/d15/d98/d4e/d9a 0 2024-03-30T04:28:09.545 INFO:tasks.workunit.client.1.smithi135.stdout:7/63: write f1 [2520036,116586] 0 2024-03-30T04:28:09.546 INFO:tasks.workunit.client.1.smithi135.stdout:5/73: fdatasync d1/d3/f12 0 2024-03-30T04:28:09.547 INFO:tasks.workunit.client.0.smithi123.stdout:1/548: link db/dd/d42/d46/l6a db/dd/d5f/d6d/d85/d89/d90/d94/d9c/lb3 0 2024-03-30T04:28:09.548 INFO:tasks.workunit.client.0.smithi123.stdout:5/430: dread dd/d41/f47 [0,4194304] 0 2024-03-30T04:28:09.551 INFO:tasks.workunit.client.0.smithi123.stdout:3/488: creat d13/d24/d31/d30/f9a x:0 0 0 2024-03-30T04:28:09.551 INFO:tasks.workunit.client.0.smithi123.stdout:4/446: link d2/c14 d2/d4/c9e 0 2024-03-30T04:28:09.551 INFO:tasks.workunit.client.0.smithi123.stdout:9/442: truncate d6/d49/f75 1019020 0 2024-03-30T04:28:09.556 INFO:tasks.workunit.client.0.smithi123.stdout:1/549: truncate db/dd/d42/d8f/d78/f91 97073 0 2024-03-30T04:28:09.557 INFO:tasks.workunit.client.0.smithi123.stdout:0/446: rename d6/c21 to d6/dd/d15/d98/c9b 0 2024-03-30T04:28:09.559 INFO:tasks.workunit.client.1.smithi135.stdout:7/64: dread f1 [0,4194304] 0 2024-03-30T04:28:09.559 INFO:tasks.workunit.client.0.smithi123.stdout:5/431: readlink dd/l29 0 2024-03-30T04:28:09.564 INFO:tasks.workunit.client.1.smithi135.stdout:7/65: mkdir d4/db/de/d15 0 2024-03-30T04:28:09.565 INFO:tasks.workunit.client.1.smithi135.stdout:5/74: rmdir d1/d7 39 2024-03-30T04:28:09.566 INFO:tasks.workunit.client.0.smithi123.stdout:9/443: chown d6/d1f/d72/d7d 16692664 1 2024-03-30T04:28:09.567 INFO:tasks.workunit.client.0.smithi123.stdout:1/550: readlink db/dd/d42/d46/d5d/l71 0 2024-03-30T04:28:09.568 INFO:tasks.workunit.client.1.smithi135.stdout:7/66: creat d4/f16 x:0 0 0 2024-03-30T04:28:09.570 INFO:tasks.workunit.client.0.smithi123.stdout:5/432: rename dd/d19/d3e to dd/d19/d3e/d5b/d8f 22 2024-03-30T04:28:09.570 INFO:tasks.workunit.client.0.smithi123.stdout:4/447: mkdir d2/d4/d2c/d42/d35/d5d/d9f 0 2024-03-30T04:28:09.572 INFO:tasks.workunit.client.1.smithi135.stdout:5/75: creat d1/d7/d11/f19 x:0 0 0 2024-03-30T04:28:09.573 INFO:tasks.workunit.client.0.smithi123.stdout:2/423: dwrite d0/d1f/d16/f23 [0,4194304] 0 2024-03-30T04:28:09.574 INFO:tasks.workunit.client.0.smithi123.stdout:0/447: symlink d6/dd/d15/d98/d4e/d9a/l9c 0 2024-03-30T04:28:09.574 INFO:tasks.workunit.client.1.smithi135.stdout:7/67: rename d4/ld to d4/db/de/l17 0 2024-03-30T04:28:09.576 INFO:tasks.workunit.client.1.smithi135.stdout:5/76: unlink d1/d7/d11/f19 0 2024-03-30T04:28:09.577 INFO:tasks.workunit.client.0.smithi123.stdout:5/433: write dd/d19/d1f/d35/f38 [1575044,77793] 0 2024-03-30T04:28:09.578 INFO:tasks.workunit.client.0.smithi123.stdout:1/551: creat db/dd/d42/d46/d5d/d62/fb4 x:0 0 0 2024-03-30T04:28:09.579 INFO:tasks.workunit.client.0.smithi123.stdout:5/434: stat dd/d19/d1f/d35 0 2024-03-30T04:28:09.581 INFO:tasks.workunit.client.1.smithi135.stdout:5/77: mkdir d1/d7/d1a 0 2024-03-30T04:28:09.582 INFO:tasks.workunit.client.0.smithi123.stdout:2/424: mknod d0/d1f/d1c/d2a/c88 0 2024-03-30T04:28:09.584 INFO:tasks.workunit.client.1.smithi135.stdout:5/78: fsync d1/f4 0 2024-03-30T04:28:09.585 INFO:tasks.workunit.client.0.smithi123.stdout:5/435: dread - dd/d19/d3e/d5b/d62/f8b zero size 2024-03-30T04:28:09.587 INFO:tasks.workunit.client.0.smithi123.stdout:2/425: creat d0/d1f/d1c/d36/f89 x:0 0 0 2024-03-30T04:28:09.588 INFO:tasks.workunit.client.1.smithi135.stdout:5/79: creat d1/da/f1b x:0 0 0 2024-03-30T04:28:09.590 INFO:tasks.workunit.client.0.smithi123.stdout:5/436: chown dd/d19/d1f/d3c/c60 1960547 1 2024-03-30T04:28:09.591 INFO:tasks.workunit.client.1.smithi135.stdout:7/68: dread f3 [0,4194304] 0 2024-03-30T04:28:09.592 INFO:tasks.workunit.client.0.smithi123.stdout:1/552: creat db/dd/d42/d46/d56/daf/fb5 x:0 0 0 2024-03-30T04:28:09.594 INFO:tasks.workunit.client.0.smithi123.stdout:2/426: creat d0/d37/f8a x:0 0 0 2024-03-30T04:28:09.594 INFO:tasks.workunit.client.1.smithi135.stdout:7/69: symlink d4/db/l18 0 2024-03-30T04:28:09.596 INFO:tasks.workunit.client.0.smithi123.stdout:3/489: dwrite d13/d24/d31/d23/f76 [0,4194304] 0 2024-03-30T04:28:09.597 INFO:tasks.workunit.client.0.smithi123.stdout:1/553: stat db/dd/d42/d46/l95 0 2024-03-30T04:28:09.597 INFO:tasks.workunit.client.1.smithi135.stdout:7/70: creat d4/db/f19 x:0 0 0 2024-03-30T04:28:09.598 INFO:tasks.workunit.client.0.smithi123.stdout:0/448: getdents d6/dd/d15/d98/d4e 0 2024-03-30T04:28:09.598 INFO:tasks.workunit.client.1.smithi135.stdout:7/71: dread - d4/f16 zero size 2024-03-30T04:28:09.599 INFO:tasks.workunit.client.1.smithi135.stdout:7/72: readlink d4/l5 0 2024-03-30T04:28:09.602 INFO:tasks.workunit.client.0.smithi123.stdout:2/427: write d0/d1f/d1c/d39/d4b/d4c/d5f/f67 [4802918,17028] 0 2024-03-30T04:28:09.603 INFO:tasks.workunit.client.1.smithi135.stdout:7/73: mkdir d4/db/d1a 0 2024-03-30T04:28:09.604 INFO:tasks.workunit.client.0.smithi123.stdout:3/490: mknod d13/d24/d39/d79/c9b 0 2024-03-30T04:28:09.605 INFO:tasks.workunit.client.0.smithi123.stdout:5/437: truncate dd/d19/f28 1258611 0 2024-03-30T04:28:09.607 INFO:tasks.workunit.client.1.smithi135.stdout:7/74: write d4/f8 [388206,54938] 0 2024-03-30T04:28:09.607 INFO:tasks.workunit.client.0.smithi123.stdout:1/554: truncate db/dd/d42/d46/d39/f92 141272 0 2024-03-30T04:28:09.609 INFO:tasks.workunit.client.1.smithi135.stdout:7/75: rename d4/db/de to d4/db/de/d1b 22 2024-03-30T04:28:09.612 INFO:tasks.workunit.client.0.smithi123.stdout:3/491: write d13/d24/d31/d23/d63/f95 [700090,98543] 0 2024-03-30T04:28:09.612 INFO:tasks.workunit.client.1.smithi135.stdout:7/76: unlink d4/db/f19 0 2024-03-30T04:28:09.615 INFO:tasks.workunit.client.0.smithi123.stdout:2/428: dread d0/d1f/f2d [0,4194304] 0 2024-03-30T04:28:09.616 INFO:tasks.workunit.client.0.smithi123.stdout:5/438: rename dd/d19/d3e/d5b/f69 to dd/d41/f90 0 2024-03-30T04:28:09.616 INFO:tasks.workunit.client.0.smithi123.stdout:3/492: chown d13/d24/d31/d30 1778052 1 2024-03-30T04:28:09.618 INFO:tasks.workunit.client.0.smithi123.stdout:0/449: link d6/dd/d16/c34 d6/dd/d15/c9d 0 2024-03-30T04:28:09.619 INFO:tasks.workunit.client.0.smithi123.stdout:5/439: chown dd/d19/d1f/d35/c46 1838280 1 2024-03-30T04:28:09.620 INFO:tasks.workunit.client.0.smithi123.stdout:2/429: write d0/d1f/f17 [1765229,95778] 0 2024-03-30T04:28:09.623 INFO:tasks.workunit.client.0.smithi123.stdout:5/440: dread - dd/d19/d3e/d5b/d62/f7a zero size 2024-03-30T04:28:09.624 INFO:tasks.workunit.client.0.smithi123.stdout:3/493: symlink d13/l9c 0 2024-03-30T04:28:09.626 INFO:tasks.workunit.client.0.smithi123.stdout:5/441: dread - dd/d19/d1f/d35/d3a/f80 zero size 2024-03-30T04:28:09.627 INFO:tasks.workunit.client.1.smithi135.stdout:7/77: dwrite d4/db/f10 [0,4194304] 0 2024-03-30T04:28:09.627 INFO:tasks.workunit.client.1.smithi135.stdout:7/78: stat d4/f11 0 2024-03-30T04:28:09.628 INFO:tasks.workunit.client.0.smithi123.stdout:0/450: creat d6/dd/d18/d1f/f9e x:0 0 0 2024-03-30T04:28:09.628 INFO:tasks.workunit.client.0.smithi123.stdout:2/430: creat d0/d1f/d27/f8b x:0 0 0 2024-03-30T04:28:09.629 INFO:tasks.workunit.client.0.smithi123.stdout:3/494: read d13/d24/d39/f7a [3330052,120322] 0 2024-03-30T04:28:09.632 INFO:tasks.workunit.client.1.smithi135.stdout:7/79: rename f0 to d4/db/f1c 0 2024-03-30T04:28:09.634 INFO:tasks.workunit.client.0.smithi123.stdout:1/555: rename db/dd/d42/d46/d56/f70 to db/dd/d42/d46/d56/fb6 0 2024-03-30T04:28:09.634 INFO:tasks.workunit.client.0.smithi123.stdout:3/495: chown d13/d24/d31/d23/d3a/d3c/d4e/d5d/f82 103689643 1 2024-03-30T04:28:09.635 INFO:tasks.workunit.client.1.smithi135.stdout:7/80: creat d4/db/de/f1d x:0 0 0 2024-03-30T04:28:09.635 INFO:tasks.workunit.client.1.smithi135.stdout:7/81: stat d4/db/l18 0 2024-03-30T04:28:09.638 INFO:tasks.workunit.client.0.smithi123.stdout:0/451: creat d6/dd/d15/d98/d4e/d69/d6e/f9f x:0 0 0 2024-03-30T04:28:09.638 INFO:tasks.workunit.client.0.smithi123.stdout:2/431: mkdir d0/d1f/d1c/d2a/d8c 0 2024-03-30T04:28:09.640 INFO:tasks.workunit.client.0.smithi123.stdout:3/496: rmdir d13/d24/d31/d23/d3a/d3c/d4e 39 2024-03-30T04:28:09.641 INFO:tasks.workunit.client.0.smithi123.stdout:0/452: write d6/dd/d18/d1f/f9e [484680,1980] 0 2024-03-30T04:28:09.644 INFO:tasks.workunit.client.0.smithi123.stdout:2/432: mknod d0/d1f/d1c/d39/d4b/c8d 0 2024-03-30T04:28:09.645 INFO:tasks.workunit.client.0.smithi123.stdout:3/497: write d13/f1b [5532106,12434] 0 2024-03-30T04:28:09.645 INFO:tasks.workunit.client.1.smithi135.stdout:7/82: dread f1 [0,4194304] 0 2024-03-30T04:28:09.646 INFO:tasks.workunit.client.0.smithi123.stdout:1/556: read db/dd/d42/d46/d56/d5c/f97 [692780,83230] 0 2024-03-30T04:28:09.648 INFO:tasks.workunit.client.1.smithi135.stdout:7/83: unlink d4/db/f10 0 2024-03-30T04:28:09.650 INFO:tasks.workunit.client.0.smithi123.stdout:2/433: readlink d0/d1f/d1c/d36/l58 0 2024-03-30T04:28:09.651 INFO:tasks.workunit.client.1.smithi135.stdout:7/84: symlink d4/db/de/d15/l1e 0 2024-03-30T04:28:09.652 INFO:tasks.workunit.client.1.smithi135.stdout:7/85: write d4/db/de/f1d [121193,11665] 0 2024-03-30T04:28:09.653 INFO:tasks.workunit.client.1.smithi135.stdout:7/86: dread - d4/f16 zero size 2024-03-30T04:28:09.657 INFO:tasks.workunit.client.0.smithi123.stdout:3/498: rename d13/d24/d31/d30/c89 to d13/d24/d39/d79/c9d 0 2024-03-30T04:28:09.660 INFO:tasks.workunit.client.0.smithi123.stdout:2/434: truncate d0/d1f/d16/f87 975979 0 2024-03-30T04:28:09.660 INFO:tasks.workunit.client.0.smithi123.stdout:5/442: dwrite dd/d41/f90 [0,4194304] 0 2024-03-30T04:28:09.660 INFO:tasks.workunit.client.0.smithi123.stdout:3/499: readlink d13/d24/d39/l5b 0 2024-03-30T04:28:09.663 INFO:tasks.workunit.client.0.smithi123.stdout:5/443: stat dd/c36 0 2024-03-30T04:28:09.667 INFO:tasks.workunit.client.0.smithi123.stdout:5/444: fdatasync dd/d19/d1f/d68/f6b 0 2024-03-30T04:28:09.667 INFO:tasks.workunit.client.0.smithi123.stdout:2/435: write d0/d1f/d1c/f4d [788012,128600] 0 2024-03-30T04:28:09.667 INFO:tasks.workunit.client.1.smithi135.stdout:7/87: dwrite d4/f8 [0,4194304] 0 2024-03-30T04:28:09.669 INFO:tasks.workunit.client.0.smithi123.stdout:3/500: creat d13/d24/d39/f9e x:0 0 0 2024-03-30T04:28:09.672 INFO:tasks.workunit.client.0.smithi123.stdout:1/557: link db/dd/d42/d46/d39/f8a db/dd/fb7 0 2024-03-30T04:28:09.672 INFO:tasks.workunit.client.0.smithi123.stdout:1/558: readlink db/l2d 0 2024-03-30T04:28:09.674 INFO:tasks.workunit.client.1.smithi135.stdout:7/88: link d4/db/f14 d4/db/f1f 0 2024-03-30T04:28:09.677 INFO:tasks.workunit.client.1.smithi135.stdout:7/89: write d4/db/de/f1d [210295,60238] 0 2024-03-30T04:28:09.678 INFO:tasks.workunit.client.0.smithi123.stdout:5/445: mkdir dd/d19/d91 0 2024-03-30T04:28:09.680 INFO:tasks.workunit.client.0.smithi123.stdout:5/446: readlink dd/d19/d1f/d35/d3a/l86 0 2024-03-30T04:28:09.680 INFO:tasks.workunit.client.1.smithi135.stdout:7/90: mknod d4/c20 0 2024-03-30T04:28:09.681 INFO:tasks.workunit.client.0.smithi123.stdout:1/559: rmdir db/dd/d5f 39 2024-03-30T04:28:09.683 INFO:tasks.workunit.client.1.smithi135.stdout:7/91: symlink d4/db/de/d15/l21 0 2024-03-30T04:28:09.683 INFO:tasks.workunit.client.0.smithi123.stdout:2/436: creat d0/d1f/d82/f8e x:0 0 0 2024-03-30T04:28:09.685 INFO:tasks.workunit.client.0.smithi123.stdout:0/453: write d6/dd/d31/d35/f43 [2986815,2798] 0 2024-03-30T04:28:09.686 INFO:tasks.workunit.client.1.smithi135.stdout:7/92: mknod d4/db/c22 0 2024-03-30T04:28:09.687 INFO:tasks.workunit.client.1.smithi135.stdout:7/93: fdatasync d4/f11 0 2024-03-30T04:28:09.688 INFO:tasks.workunit.client.0.smithi123.stdout:2/437: unlink d0/d37/c4e 0 2024-03-30T04:28:09.688 INFO:tasks.workunit.client.0.smithi123.stdout:1/560: symlink db/lb8 0 2024-03-30T04:28:09.689 INFO:tasks.workunit.client.1.smithi135.stdout:7/94: stat d4/db/de/l12 0 2024-03-30T04:28:09.690 INFO:tasks.workunit.client.0.smithi123.stdout:0/454: read d6/f1b [3054819,109716] 0 2024-03-30T04:28:09.692 INFO:tasks.workunit.client.0.smithi123.stdout:5/447: getdents dd/d19/d1f 0 2024-03-30T04:28:09.692 INFO:tasks.workunit.client.1.smithi135.stdout:7/95: write d4/db/f14 [62655,117660] 0 2024-03-30T04:28:09.693 INFO:tasks.workunit.client.0.smithi123.stdout:1/561: write db/dd/fb7 [1867240,18851] 0 2024-03-30T04:28:09.694 INFO:tasks.workunit.client.0.smithi123.stdout:3/501: link d13/d24/d31/d23/d3a/d3c/d4e/d5d/l8d d13/d24/d31/d23/d63/l9f 0 2024-03-30T04:28:09.695 INFO:tasks.workunit.client.1.smithi135.stdout:7/96: symlink d4/l23 0 2024-03-30T04:28:09.696 INFO:tasks.workunit.client.0.smithi123.stdout:0/455: creat d6/dd/d16/fa0 x:0 0 0 2024-03-30T04:28:09.696 INFO:tasks.workunit.client.0.smithi123.stdout:5/448: read dd/d41/f43 [93858,124267] 0 2024-03-30T04:28:09.696 INFO:tasks.workunit.client.0.smithi123.stdout:2/438: creat d0/d1f/f8f x:0 0 0 2024-03-30T04:28:09.698 INFO:tasks.workunit.client.0.smithi123.stdout:2/439: dread - d0/d1f/d1c/d36/f89 zero size 2024-03-30T04:28:09.698 INFO:tasks.workunit.client.1.smithi135.stdout:7/97: rename f1 to d4/db/d1a/f24 0 2024-03-30T04:28:09.699 INFO:tasks.workunit.client.0.smithi123.stdout:1/562: fsync db/dd/d42/f4f 0 2024-03-30T04:28:09.700 INFO:tasks.workunit.client.1.smithi135.stdout:7/98: write d4/db/f14 [809602,19816] 0 2024-03-30T04:28:09.702 INFO:tasks.workunit.client.0.smithi123.stdout:2/440: write d0/d1f/d1c/d39/d4b/d4c/f64 [86907,83593] 0 2024-03-30T04:28:09.707 INFO:tasks.workunit.client.0.smithi123.stdout:1/563: mknod db/dd/d5f/cb9 0 2024-03-30T04:28:09.707 INFO:tasks.workunit.client.0.smithi123.stdout:2/441: chown d0/d1f/d1c/d39/d4b/c8d 25103918 1 2024-03-30T04:28:09.707 INFO:tasks.workunit.client.0.smithi123.stdout:3/502: getdents d13/d24/d31/d23/d3a/d3c/d4e/d5d 0 2024-03-30T04:28:09.711 INFO:tasks.workunit.client.0.smithi123.stdout:3/503: stat d13/d24/d31/d23/c60 0 2024-03-30T04:28:09.712 INFO:tasks.workunit.client.0.smithi123.stdout:1/564: fdatasync db/d1b/f1d 0 2024-03-30T04:28:09.718 INFO:tasks.workunit.client.0.smithi123.stdout:0/456: dwrite d6/dd/f51 [0,4194304] 0 2024-03-30T04:28:09.718 INFO:tasks.workunit.client.0.smithi123.stdout:1/565: symlink db/dd/d5f/d6d/d85/d89/lba 0 2024-03-30T04:28:09.719 INFO:tasks.workunit.client.0.smithi123.stdout:0/457: readlink d6/dd/l6c 0 2024-03-30T04:28:09.724 INFO:tasks.workunit.client.0.smithi123.stdout:3/504: link d13/c20 d13/d24/d31/d23/d3a/d3c/d4e/ca0 0 2024-03-30T04:28:09.729 INFO:tasks.workunit.client.0.smithi123.stdout:3/505: chown c5 232070 1 2024-03-30T04:28:09.730 INFO:tasks.workunit.client.0.smithi123.stdout:0/458: symlink d6/dd/d31/d35/la1 0 2024-03-30T04:28:09.732 INFO:tasks.workunit.client.0.smithi123.stdout:2/442: dwrite d0/d37/f74 [0,4194304] 0 2024-03-30T04:28:09.736 INFO:tasks.workunit.client.0.smithi123.stdout:3/506: creat d13/fa1 x:0 0 0 2024-03-30T04:28:09.736 INFO:tasks.workunit.client.0.smithi123.stdout:1/566: link db/dd/d5f/c8b db/dd/d42/d46/d5d/d62/d9e/cbb 0 2024-03-30T04:28:09.739 INFO:tasks.workunit.client.0.smithi123.stdout:0/459: read d6/dd/d31/d35/f43 [2643741,23686] 0 2024-03-30T04:28:09.742 INFO:tasks.workunit.client.0.smithi123.stdout:2/443: rmdir d0/d1f/d1c/d40 39 2024-03-30T04:28:09.745 INFO:tasks.workunit.client.0.smithi123.stdout:2/444: write d0/d1f/d1c/d36/f89 [532195,122055] 0 2024-03-30T04:28:09.746 INFO:tasks.workunit.client.0.smithi123.stdout:3/507: rmdir d13/d7b 0 2024-03-30T04:28:09.747 INFO:tasks.workunit.client.0.smithi123.stdout:3/508: fdatasync d13/f7c 0 2024-03-30T04:28:09.750 INFO:tasks.workunit.client.0.smithi123.stdout:3/509: truncate d13/d24/f4d 247836 0 2024-03-30T04:28:09.762 INFO:tasks.workunit.client.0.smithi123.stdout:1/567: dwrite db/dd/d5f/d6d/fa3 [0,4194304] 0 2024-03-30T04:28:09.765 INFO:tasks.workunit.client.0.smithi123.stdout:2/445: truncate d0/d1f/d1c/d39/d4b/d4c/f6a 463841 0 2024-03-30T04:28:09.766 INFO:tasks.workunit.client.0.smithi123.stdout:1/568: chown db/dd/f8e 90539755 1 2024-03-30T04:28:09.771 INFO:tasks.workunit.client.0.smithi123.stdout:0/460: dwrite d6/dd/d15/d98/d4e/d69/f7a [0,4194304] 0 2024-03-30T04:28:09.774 INFO:tasks.workunit.client.0.smithi123.stdout:1/569: fdatasync db/dd/d42/f4f 0 2024-03-30T04:28:09.774 INFO:tasks.workunit.client.0.smithi123.stdout:2/446: creat d0/d1f/d1c/d2a/d8c/f90 x:0 0 0 2024-03-30T04:28:09.777 INFO:tasks.workunit.client.0.smithi123.stdout:2/447: readlink d0/d1f/d1c/l45 0 2024-03-30T04:28:09.781 INFO:tasks.workunit.client.0.smithi123.stdout:2/448: stat d0/d1f/d1c/l77 0 2024-03-30T04:28:09.782 INFO:tasks.workunit.client.0.smithi123.stdout:3/510: dwrite d13/d24/d31/d23/d3a/d3c/d4e/d5d/f68 [0,4194304] 0 2024-03-30T04:28:09.784 INFO:tasks.workunit.client.0.smithi123.stdout:2/449: read d0/f1d [6665198,43610] 0 2024-03-30T04:28:09.784 INFO:tasks.workunit.client.0.smithi123.stdout:1/570: rename db/dd/d42/d46/d56/f7f to db/dd/d42/d8f/fbc 0 2024-03-30T04:28:09.787 INFO:tasks.workunit.client.0.smithi123.stdout:3/511: chown d13/d24/f50 59 1 2024-03-30T04:28:09.792 INFO:tasks.workunit.client.0.smithi123.stdout:1/571: mknod db/dd/d42/d46/d5d/d62/cbd 0 2024-03-30T04:28:09.793 INFO:tasks.workunit.client.0.smithi123.stdout:2/450: truncate d0/d1f/d5e/f7a 4855610 0 2024-03-30T04:28:09.794 INFO:tasks.workunit.client.0.smithi123.stdout:0/461: dread d6/dd/f51 [0,4194304] 0 2024-03-30T04:28:09.794 INFO:tasks.workunit.client.0.smithi123.stdout:3/512: symlink d13/d24/d31/d23/d3a/d3c/d4e/la2 0 2024-03-30T04:28:09.797 INFO:tasks.workunit.client.0.smithi123.stdout:2/451: fsync d0/d1f/d1c/d39/d4b/d4c/d5f/f67 0 2024-03-30T04:28:09.805 INFO:tasks.workunit.client.0.smithi123.stdout:1/572: dread db/dd/d42/d46/f29 [0,4194304] 0 2024-03-30T04:28:09.809 INFO:tasks.workunit.client.0.smithi123.stdout:3/513: dread d13/d24/d31/d23/d3a/d3c/f8b [0,4194304] 0 2024-03-30T04:28:09.810 INFO:tasks.workunit.client.0.smithi123.stdout:1/573: symlink db/dd/d42/d46/d56/lbe 0 2024-03-30T04:28:09.813 INFO:tasks.workunit.client.0.smithi123.stdout:2/452: dread d0/f10 [4194304,4194304] 0 2024-03-30T04:28:09.815 INFO:tasks.workunit.client.0.smithi123.stdout:2/453: symlink d0/d1f/d27/l91 0 2024-03-30T04:28:09.818 INFO:tasks.workunit.client.0.smithi123.stdout:1/574: write db/f6e [1359950,101061] 0 2024-03-30T04:28:09.818 INFO:tasks.workunit.client.0.smithi123.stdout:2/454: write d0/d1f/d27/f8b [334222,1331] 0 2024-03-30T04:28:09.821 INFO:tasks.workunit.client.0.smithi123.stdout:0/462: dread d6/f1b [0,4194304] 0 2024-03-30T04:28:09.821 INFO:tasks.workunit.client.0.smithi123.stdout:1/575: readlink db/dd/d42/d46/d5d/l71 0 2024-03-30T04:28:09.823 INFO:tasks.workunit.client.0.smithi123.stdout:2/455: unlink d0/d1f/d27/f29 0 2024-03-30T04:28:09.825 INFO:tasks.workunit.client.0.smithi123.stdout:0/463: symlink d6/dd/d15/d98/d4e/d9a/la2 0 2024-03-30T04:28:09.827 INFO:tasks.workunit.client.0.smithi123.stdout:2/456: getdents d0/d1f/d1c/d39/d4b 0 2024-03-30T04:28:09.828 INFO:tasks.workunit.client.0.smithi123.stdout:0/464: write d6/dd/d15/f2b [1690835,82064] 0 2024-03-30T04:28:09.828 INFO:tasks.workunit.client.0.smithi123.stdout:3/514: dwrite ff [0,4194304] 0 2024-03-30T04:28:09.834 INFO:tasks.workunit.client.0.smithi123.stdout:0/465: mkdir d6/dd/d18/d2f/da3 0 2024-03-30T04:28:09.835 INFO:tasks.workunit.client.0.smithi123.stdout:3/515: mknod d13/d24/d39/d79/ca3 0 2024-03-30T04:28:09.836 INFO:tasks.workunit.client.0.smithi123.stdout:0/466: fdatasync d6/dd/d18/f47 0 2024-03-30T04:28:09.836 INFO:tasks.workunit.client.0.smithi123.stdout:2/457: getdents d0/d1f/d1c/d39 0 2024-03-30T04:28:09.838 INFO:tasks.workunit.client.0.smithi123.stdout:0/467: read d6/dd/d15/d98/d4e/d69/f7a [642961,20478] 0 2024-03-30T04:28:09.839 INFO:tasks.workunit.client.0.smithi123.stdout:2/458: creat d0/d1f/d82/f92 x:0 0 0 2024-03-30T04:28:09.840 INFO:tasks.workunit.client.0.smithi123.stdout:3/516: chown d13/d24/d31/d23/d3a/l42 91 1 2024-03-30T04:28:09.843 INFO:tasks.workunit.client.0.smithi123.stdout:0/468: symlink d6/dd/d15/d98/la4 0 2024-03-30T04:28:09.844 INFO:tasks.workunit.client.0.smithi123.stdout:2/459: creat d0/d1f/d1c/d80/f93 x:0 0 0 2024-03-30T04:28:09.845 INFO:tasks.workunit.client.0.smithi123.stdout:3/517: read d13/d24/d31/d23/d3a/d3c/d4e/d5d/f68 [537698,120401] 0 2024-03-30T04:28:09.852 INFO:tasks.workunit.client.0.smithi123.stdout:0/469: getdents d6/dd/d15/d98/d4e/d9a 0 2024-03-30T04:28:09.859 INFO:tasks.workunit.client.0.smithi123.stdout:3/518: dread d13/d24/d39/f7a [0,4194304] 0 2024-03-30T04:28:09.860 INFO:tasks.workunit.client.0.smithi123.stdout:0/470: truncate d6/dd/d31/d35/f42 1446372 0 2024-03-30T04:28:09.864 INFO:tasks.workunit.client.0.smithi123.stdout:3/519: creat d13/d24/d31/d23/d3a/fa4 x:0 0 0 2024-03-30T04:28:09.864 INFO:tasks.workunit.client.0.smithi123.stdout:0/471: symlink d6/dd/d16/la5 0 2024-03-30T04:28:09.867 INFO:tasks.workunit.client.0.smithi123.stdout:3/520: unlink l4 0 2024-03-30T04:28:09.869 INFO:tasks.workunit.client.0.smithi123.stdout:0/472: getdents d6/dd/d15 0 2024-03-30T04:28:09.874 INFO:tasks.workunit.client.0.smithi123.stdout:0/473: chown d6/l84 1687 1 2024-03-30T04:28:09.874 INFO:tasks.workunit.client.0.smithi123.stdout:2/460: dread d0/d1f/d1c/d40/f5a [0,4194304] 0 2024-03-30T04:28:09.880 INFO:tasks.workunit.client.0.smithi123.stdout:2/461: rename d0/d1f/d1c/f22 to d0/d1f/d30/f94 0 2024-03-30T04:28:09.883 INFO:tasks.workunit.client.0.smithi123.stdout:2/462: mknod d0/d1f/d82/c95 0 2024-03-30T04:28:09.884 INFO:tasks.workunit.client.0.smithi123.stdout:3/521: dwrite d13/d24/d31/d23/f25 [0,4194304] 0 2024-03-30T04:28:09.891 INFO:tasks.workunit.client.0.smithi123.stdout:7/411: sync 2024-03-30T04:28:09.891 INFO:tasks.workunit.client.0.smithi123.stdout:6/550: sync 2024-03-30T04:28:09.892 INFO:tasks.workunit.client.0.smithi123.stdout:0/474: dwrite d6/dd/d15/d98/d4e/d69/d6e/f87 [0,4194304] 0 2024-03-30T04:28:09.894 INFO:tasks.workunit.client.0.smithi123.stdout:2/463: rmdir d0/d1f/d30 39 2024-03-30T04:28:09.895 INFO:tasks.workunit.client.0.smithi123.stdout:7/412: stat d2/da/f4c 0 2024-03-30T04:28:09.899 INFO:tasks.workunit.client.0.smithi123.stdout:6/551: symlink d10/d13/d45/lbc 0 2024-03-30T04:28:09.901 INFO:tasks.workunit.client.0.smithi123.stdout:2/464: creat d0/d1f/d1c/d39/f96 x:0 0 0 2024-03-30T04:28:09.903 INFO:tasks.workunit.client.0.smithi123.stdout:0/475: rename d6/dd/d15/d28/f5a to d6/dd/d15/d28/d4b/d8a/fa6 0 2024-03-30T04:28:09.904 INFO:tasks.workunit.client.0.smithi123.stdout:2/465: fdatasync d0/d37/f8a 0 2024-03-30T04:28:09.906 INFO:tasks.workunit.client.0.smithi123.stdout:7/413: link d2/da/d11/d61/f57 d2/dc/d7f/f8a 0 2024-03-30T04:28:09.910 INFO:tasks.workunit.client.0.smithi123.stdout:6/552: creat d10/d13/d6a/d7a/d99/dba/fbd x:0 0 0 2024-03-30T04:28:09.911 INFO:tasks.workunit.client.0.smithi123.stdout:7/414: creat d2/da/d11/d61/d55/d59/f8b x:0 0 0 2024-03-30T04:28:09.911 INFO:tasks.workunit.client.0.smithi123.stdout:3/522: dread d13/d24/d31/f4b [0,4194304] 0 2024-03-30T04:28:09.912 INFO:tasks.workunit.client.0.smithi123.stdout:6/553: mkdir d10/d13/d6a/d7a/dbe 0 2024-03-30T04:28:09.914 INFO:tasks.workunit.client.0.smithi123.stdout:3/523: fdatasync d13/d24/f50 0 2024-03-30T04:28:09.915 INFO:tasks.workunit.client.0.smithi123.stdout:7/415: symlink d2/dc/d49/l8c 0 2024-03-30T04:28:09.916 INFO:tasks.workunit.client.0.smithi123.stdout:6/554: symlink d10/d82/d8e/lbf 0 2024-03-30T04:28:09.917 INFO:tasks.workunit.client.0.smithi123.stdout:3/524: stat d13/d24/d31/d23/d3a/d3c/c74 0 2024-03-30T04:28:09.918 INFO:tasks.workunit.client.0.smithi123.stdout:6/555: fdatasync d10/d20/f2b 0 2024-03-30T04:28:09.919 INFO:tasks.workunit.client.0.smithi123.stdout:7/416: write d2/da/d11/f48 [361259,59688] 0 2024-03-30T04:28:09.920 INFO:tasks.workunit.client.0.smithi123.stdout:3/525: write d13/f8a [1475339,115129] 0 2024-03-30T04:28:09.922 INFO:tasks.workunit.client.0.smithi123.stdout:0/476: write d6/dd/d31/d35/f5b [460361,84026] 0 2024-03-30T04:28:09.925 INFO:tasks.workunit.client.0.smithi123.stdout:7/417: mknod d2/dc/d56/c8d 0 2024-03-30T04:28:09.927 INFO:tasks.workunit.client.0.smithi123.stdout:6/556: rmdir d10/d13/d6a/d7a/dbe 0 2024-03-30T04:28:09.928 INFO:tasks.workunit.client.0.smithi123.stdout:3/526: write d13/d24/d31/d23/d3a/fa4 [705023,25199] 0 2024-03-30T04:28:09.930 INFO:tasks.workunit.client.0.smithi123.stdout:0/477: write d6/f1b [3735394,85376] 0 2024-03-30T04:28:09.933 INFO:tasks.workunit.client.0.smithi123.stdout:2/466: dwrite d0/d1f/d1c/d80/f93 [0,4194304] 0 2024-03-30T04:28:09.933 INFO:tasks.workunit.client.0.smithi123.stdout:0/478: readlink d6/dd/d15/l46 0 2024-03-30T04:28:09.935 INFO:tasks.workunit.client.0.smithi123.stdout:7/418: mkdir d2/da/d11/d34/d85/d8e 0 2024-03-30T04:28:09.936 INFO:tasks.workunit.client.0.smithi123.stdout:6/557: symlink d10/d20/dac/d5b/lc0 0 2024-03-30T04:28:09.937 INFO:tasks.workunit.client.0.smithi123.stdout:2/467: chown d0/d1f/d1c/d36/f7d 352623 1 2024-03-30T04:28:09.937 INFO:tasks.workunit.client.0.smithi123.stdout:3/527: write d13/d24/d31/d23/d3a/d3c/f8b [2230668,48222] 0 2024-03-30T04:28:09.938 INFO:tasks.workunit.client.0.smithi123.stdout:6/558: write d10/f96 [687949,118568] 0 2024-03-30T04:28:09.939 INFO:tasks.workunit.client.0.smithi123.stdout:0/479: mkdir d6/dd/d15/d98/d4e/d69/d6e/d99/da7 0 2024-03-30T04:28:09.943 INFO:tasks.workunit.client.0.smithi123.stdout:2/468: mkdir d0/d1f/d1c/d39/d4b/d70/d97 0 2024-03-30T04:28:09.945 INFO:tasks.workunit.client.0.smithi123.stdout:3/528: mkdir d13/d24/d31/d23/d3a/d3c/d4e/d5d/da5 0 2024-03-30T04:28:09.949 INFO:tasks.workunit.client.0.smithi123.stdout:6/559: write d10/d13/d45/faa [2041218,130541] 0 2024-03-30T04:28:09.957 INFO:tasks.workunit.client.0.smithi123.stdout:7/419: dwrite d2/da/d11/d61/f72 [0,4194304] 0 2024-03-30T04:28:09.958 INFO:tasks.workunit.client.0.smithi123.stdout:6/560: rename d10/d13 to d10/d82/dc1 0 2024-03-30T04:28:09.959 INFO:tasks.workunit.client.0.smithi123.stdout:7/420: fdatasync d2/da/d22/d3b/f6d 0 2024-03-30T04:28:09.960 INFO:tasks.workunit.client.0.smithi123.stdout:2/469: getdents d0/d1f/d1c/d39 0 2024-03-30T04:28:09.965 INFO:tasks.workunit.client.0.smithi123.stdout:6/561: truncate d10/f38 1000102 0 2024-03-30T04:28:09.966 INFO:tasks.workunit.client.0.smithi123.stdout:3/529: write f11 [2910081,6554] 0 2024-03-30T04:28:09.968 INFO:tasks.workunit.client.1.smithi135.stdout:1/7: sync 2024-03-30T04:28:09.968 INFO:tasks.workunit.client.1.smithi135.stdout:1/8: read - no filename 2024-03-30T04:28:09.968 INFO:tasks.workunit.client.1.smithi135.stdout:1/9: dread - no filename 2024-03-30T04:28:09.970 INFO:tasks.workunit.client.0.smithi123.stdout:3/530: write d13/f14 [4471509,116363] 0 2024-03-30T04:28:09.970 INFO:tasks.workunit.client.0.smithi123.stdout:6/562: read - d10/d20/dac/d42/fb8 zero size 2024-03-30T04:28:09.970 INFO:tasks.workunit.client.1.smithi135.stdout:1/10: mknod c1 0 2024-03-30T04:28:09.970 INFO:tasks.workunit.client.1.smithi135.stdout:1/11: read - no filename 2024-03-30T04:28:09.973 INFO:tasks.workunit.client.1.smithi135.stdout:1/12: mknod c2 0 2024-03-30T04:28:09.973 INFO:tasks.workunit.client.1.smithi135.stdout:1/13: chown c1 7 1 2024-03-30T04:28:09.978 INFO:tasks.workunit.client.1.smithi135.stdout:1/14: creat f3 x:0 0 0 2024-03-30T04:28:09.978 INFO:tasks.workunit.client.1.smithi135.stdout:1/15: dread - f3 zero size 2024-03-30T04:28:09.978 INFO:tasks.workunit.client.0.smithi123.stdout:6/563: readlink d10/d82/dc1/d45/lbc 0 2024-03-30T04:28:09.982 INFO:tasks.workunit.client.0.smithi123.stdout:0/480: dwrite d6/dd/d31/d35/f42 [0,4194304] 0 2024-03-30T04:28:09.985 INFO:tasks.workunit.client.0.smithi123.stdout:3/531: rename d13/d24/d31/d30/c65 to d13/ca6 0 2024-03-30T04:28:09.985 INFO:tasks.workunit.client.1.smithi135.stdout:1/16: fsync f3 0 2024-03-30T04:28:09.986 INFO:tasks.workunit.client.0.smithi123.stdout:2/470: dwrite d0/d1f/f1b [0,4194304] 0 2024-03-30T04:28:09.987 INFO:tasks.workunit.client.0.smithi123.stdout:6/564: creat d10/d82/dc1/d6a/d7a/d99/fc2 x:0 0 0 2024-03-30T04:28:09.989 INFO:tasks.workunit.client.1.smithi135.stdout:1/17: getdents . 0 2024-03-30T04:28:09.990 INFO:tasks.workunit.client.1.smithi135.stdout:1/18: chown f3 0 1 2024-03-30T04:28:09.991 INFO:tasks.workunit.client.1.smithi135.stdout:1/19: mknod c4 0 2024-03-30T04:28:09.991 INFO:tasks.workunit.client.1.smithi135.stdout:1/20: dread - f3 zero size 2024-03-30T04:28:09.992 INFO:tasks.workunit.client.0.smithi123.stdout:2/471: dread - d0/d1f/d1c/d20/f56 zero size 2024-03-30T04:28:09.993 INFO:tasks.workunit.client.0.smithi123.stdout:7/421: dwrite d2/da/d22/d3b/f6d [0,4194304] 0 2024-03-30T04:28:09.993 INFO:tasks.workunit.client.1.smithi135.stdout:1/21: creat f5 x:0 0 0 2024-03-30T04:28:09.994 INFO:tasks.workunit.client.1.smithi135.stdout:1/22: chown c4 42 1 2024-03-30T04:28:09.996 INFO:tasks.workunit.client.1.smithi135.stdout:1/23: write f3 [126397,117750] 0 2024-03-30T04:28:09.998 INFO:tasks.workunit.client.1.smithi135.stdout:1/24: creat f6 x:0 0 0 2024-03-30T04:28:09.998 INFO:tasks.workunit.client.0.smithi123.stdout:0/481: link d6/dd/d15/d28/f33 d6/d52/fa8 0 2024-03-30T04:28:09.999 INFO:tasks.workunit.client.0.smithi123.stdout:6/565: creat d10/d20/da2/fc3 x:0 0 0 2024-03-30T04:28:10.001 INFO:tasks.workunit.client.1.smithi135.stdout:1/25: unlink c4 0 2024-03-30T04:28:10.003 INFO:tasks.workunit.client.1.smithi135.stdout:1/26: write f5 [249433,99920] 0 2024-03-30T04:28:10.003 INFO:tasks.workunit.client.0.smithi123.stdout:7/422: creat d2/da/d22/d3b/f8f x:0 0 0 2024-03-30T04:28:10.008 INFO:tasks.workunit.client.1.smithi135.stdout:1/27: link f6 f7 0 2024-03-30T04:28:10.009 INFO:tasks.workunit.client.0.smithi123.stdout:0/482: creat d6/dd/d31/fa9 x:0 0 0 2024-03-30T04:28:10.014 INFO:tasks.workunit.client.1.smithi135.stdout:1/28: dread f5 [0,4194304] 0 2024-03-30T04:28:10.015 INFO:tasks.workunit.client.0.smithi123.stdout:6/566: rmdir d10 39 2024-03-30T04:28:10.016 INFO:tasks.workunit.client.0.smithi123.stdout:0/483: unlink d6/dd/d15/d98/c71 0 2024-03-30T04:28:10.017 INFO:tasks.workunit.client.1.smithi135.stdout:1/29: write f7 [110471,8181] 0 2024-03-30T04:28:10.018 INFO:tasks.workunit.client.1.smithi135.stdout:1/30: stat f7 0 2024-03-30T04:28:10.019 INFO:tasks.workunit.client.0.smithi123.stdout:2/472: dwrite d0/d1f/f17 [0,4194304] 0 2024-03-30T04:28:10.021 INFO:tasks.workunit.client.0.smithi123.stdout:0/484: unlink d6/dd/d31/l65 0 2024-03-30T04:28:10.024 INFO:tasks.workunit.client.1.smithi135.stdout:1/31: dread f6 [0,4194304] 0 2024-03-30T04:28:10.025 INFO:tasks.workunit.client.1.smithi135.stdout:1/32: dread f7 [0,4194304] 0 2024-03-30T04:28:10.029 INFO:tasks.workunit.client.0.smithi123.stdout:0/485: symlink d6/dd/d18/d2f/laa 0 2024-03-30T04:28:10.029 INFO:tasks.workunit.client.0.smithi123.stdout:2/473: getdents d0/d1f/d1c/d20 0 2024-03-30T04:28:10.035 INFO:tasks.workunit.client.0.smithi123.stdout:7/423: dwrite d2/da/d2d/f39 [0,4194304] 0 2024-03-30T04:28:10.037 INFO:tasks.workunit.client.0.smithi123.stdout:2/474: creat d0/d1f/d1c/d40/f98 x:0 0 0 2024-03-30T04:28:10.040 INFO:tasks.workunit.client.0.smithi123.stdout:0/486: getdents d6/dd/d15/d98 0 2024-03-30T04:28:10.042 INFO:tasks.workunit.client.1.smithi135.stdout:1/33: dwrite f7 [0,4194304] 0 2024-03-30T04:28:10.045 INFO:tasks.workunit.client.0.smithi123.stdout:0/487: truncate d6/f1b 5847794 0 2024-03-30T04:28:10.045 INFO:tasks.workunit.client.0.smithi123.stdout:6/567: dwrite d10/d82/f66 [0,4194304] 0 2024-03-30T04:28:10.047 INFO:tasks.workunit.client.0.smithi123.stdout:2/475: mkdir d0/d1f/d1c/d2a/d99 0 2024-03-30T04:28:10.052 INFO:tasks.workunit.client.1.smithi135.stdout:1/34: dread f7 [0,4194304] 0 2024-03-30T04:28:10.053 INFO:tasks.workunit.client.0.smithi123.stdout:2/476: rmdir d0/d1f/d5e 39 2024-03-30T04:28:10.061 INFO:tasks.workunit.client.0.smithi123.stdout:0/488: unlink d6/dd/d18/d1f/c3b 0 2024-03-30T04:28:10.064 INFO:tasks.workunit.client.0.smithi123.stdout:2/477: rename d0/d1f/d16/l50 to d0/d37/l9a 0 2024-03-30T04:28:10.065 INFO:tasks.workunit.client.1.smithi135.stdout:0/59: sync 2024-03-30T04:28:10.065 INFO:tasks.workunit.client.1.smithi135.stdout:4/7: sync 2024-03-30T04:28:10.065 INFO:tasks.workunit.client.1.smithi135.stdout:4/8: dread - no filename 2024-03-30T04:28:10.065 INFO:tasks.workunit.client.1.smithi135.stdout:4/9: dwrite - no filename 2024-03-30T04:28:10.066 INFO:tasks.workunit.client.1.smithi135.stdout:4/10: read - no filename 2024-03-30T04:28:10.066 INFO:tasks.workunit.client.1.smithi135.stdout:3/25: sync 2024-03-30T04:28:10.066 INFO:tasks.workunit.client.1.smithi135.stdout:5/80: sync 2024-03-30T04:28:10.066 INFO:tasks.workunit.client.1.smithi135.stdout:6/11: sync 2024-03-30T04:28:10.066 INFO:tasks.workunit.client.1.smithi135.stdout:2/8: sync 2024-03-30T04:28:10.066 INFO:tasks.workunit.client.1.smithi135.stdout:2/9: rmdir - no directory 2024-03-30T04:28:10.066 INFO:tasks.workunit.client.1.smithi135.stdout:2/10: write - no filename 2024-03-30T04:28:10.066 INFO:tasks.workunit.client.1.smithi135.stdout:2/11: truncate - no filename 2024-03-30T04:28:10.066 INFO:tasks.workunit.client.1.smithi135.stdout:2/12: rmdir - no directory 2024-03-30T04:28:10.066 INFO:tasks.workunit.client.1.smithi135.stdout:8/55: sync 2024-03-30T04:28:10.066 INFO:tasks.workunit.client.1.smithi135.stdout:7/99: sync 2024-03-30T04:28:10.066 INFO:tasks.workunit.client.1.smithi135.stdout:9/25: sync 2024-03-30T04:28:10.067 INFO:tasks.workunit.client.0.smithi123.stdout:0/489: read d6/dd/d15/d98/d4e/d69/d6e/f7c [1206544,125980] 0 2024-03-30T04:28:10.067 INFO:tasks.workunit.client.0.smithi123.stdout:7/424: link d2/da/d2d/c2e d2/dc/d49/d68/c90 0 2024-03-30T04:28:10.069 INFO:tasks.workunit.client.1.smithi135.stdout:2/13: stat l1 0 2024-03-30T04:28:10.069 INFO:tasks.workunit.client.1.smithi135.stdout:2/14: write - no filename 2024-03-30T04:28:10.069 INFO:tasks.workunit.client.1.smithi135.stdout:2/15: fdatasync - no filename 2024-03-30T04:28:10.069 INFO:tasks.workunit.client.0.smithi123.stdout:2/478: stat d0/d1f/d1c/d39/d4b/d4c/d5f 0 2024-03-30T04:28:10.071 INFO:tasks.workunit.client.1.smithi135.stdout:3/26: chown f0 113201128 1 2024-03-30T04:28:10.071 INFO:tasks.workunit.client.1.smithi135.stdout:0/60: read d0/d1/fa [1126566,90136] 0 2024-03-30T04:28:10.072 INFO:tasks.workunit.client.0.smithi123.stdout:6/568: getdents d10/d82/dc1/d45 0 2024-03-30T04:28:10.072 INFO:tasks.workunit.client.1.smithi135.stdout:6/12: rename c0 to d2/c3 0 2024-03-30T04:28:10.073 INFO:tasks.workunit.client.1.smithi135.stdout:4/11: rename l1 to l2 0 2024-03-30T04:28:10.073 INFO:tasks.workunit.client.1.smithi135.stdout:4/12: dread - no filename 2024-03-30T04:28:10.073 INFO:tasks.workunit.client.1.smithi135.stdout:4/13: dwrite - no filename 2024-03-30T04:28:10.074 INFO:tasks.workunit.client.1.smithi135.stdout:9/26: rename f4 to f6 0 2024-03-30T04:28:10.074 INFO:tasks.workunit.client.0.smithi123.stdout:2/479: mkdir d0/d37/d9b 0 2024-03-30T04:28:10.075 INFO:tasks.workunit.client.0.smithi123.stdout:7/425: getdents d2/d4d/d80 0 2024-03-30T04:28:10.077 INFO:tasks.workunit.client.0.smithi123.stdout:2/480: chown d0/d1f/d1c/d36/c47 295374995 1 2024-03-30T04:28:10.077 INFO:tasks.workunit.client.1.smithi135.stdout:2/16: symlink l2 0 2024-03-30T04:28:10.077 INFO:tasks.workunit.client.1.smithi135.stdout:2/17: dread - no filename 2024-03-30T04:28:10.078 INFO:tasks.workunit.client.0.smithi123.stdout:7/426: dread - d2/dc/d1c/f7c zero size 2024-03-30T04:28:10.079 INFO:tasks.workunit.client.0.smithi123.stdout:0/490: getdents d6/dd/d15/d98/d4e/d69/d6e 0 2024-03-30T04:28:10.081 INFO:tasks.workunit.client.1.smithi135.stdout:7/100: rename f2 to d4/db/de/f25 0 2024-03-30T04:28:10.082 INFO:tasks.workunit.client.1.smithi135.stdout:0/61: symlink d0/d1/le 0 2024-03-30T04:28:10.084 INFO:tasks.workunit.client.0.smithi123.stdout:2/481: write d0/d1f/f17 [226131,124481] 0 2024-03-30T04:28:10.084 INFO:tasks.workunit.client.1.smithi135.stdout:9/27: mkdir d7 0 2024-03-30T04:28:10.086 INFO:tasks.workunit.client.1.smithi135.stdout:7/101: stat f3 0 2024-03-30T04:28:10.086 INFO:tasks.workunit.client.1.smithi135.stdout:2/18: creat f3 x:0 0 0 2024-03-30T04:28:10.090 INFO:tasks.workunit.client.0.smithi123.stdout:2/482: creat d0/d1f/d5e/d71/f9c x:0 0 0 2024-03-30T04:28:10.092 INFO:tasks.workunit.client.0.smithi123.stdout:2/483: read - d0/d1f/d16/f3c zero size 2024-03-30T04:28:10.093 INFO:tasks.workunit.client.1.smithi135.stdout:9/28: write f2 [1075566,98365] 0 2024-03-30T04:28:10.094 INFO:tasks.workunit.client.0.smithi123.stdout:2/484: readlink d0/d1f/d1c/d36/l58 0 2024-03-30T04:28:10.095 INFO:tasks.workunit.client.1.smithi135.stdout:1/35: dwrite f3 [0,4194304] 0 2024-03-30T04:28:10.099 INFO:tasks.workunit.client.1.smithi135.stdout:2/19: mkdir d4 0 2024-03-30T04:28:10.099 INFO:tasks.workunit.client.1.smithi135.stdout:7/102: write d4/db/f1f [127760,2866] 0 2024-03-30T04:28:10.101 INFO:tasks.workunit.client.0.smithi123.stdout:0/491: dread d6/dd/d15/d98/d4e/d69/f7a [0,4194304] 0 2024-03-30T04:28:10.103 INFO:tasks.workunit.client.0.smithi123.stdout:0/492: chown d6/c2c 2955849 1 2024-03-30T04:28:10.106 INFO:tasks.workunit.client.0.smithi123.stdout:0/493: write d6/dd/f51 [2259966,47197] 0 2024-03-30T04:28:10.108 INFO:tasks.workunit.client.1.smithi135.stdout:3/27: write d3/d6/f7 [157898,69872] 0 2024-03-30T04:28:10.109 INFO:tasks.workunit.client.0.smithi123.stdout:2/485: dread d0/d1f/d16/f23 [0,4194304] 0 2024-03-30T04:28:10.112 INFO:tasks.workunit.client.1.smithi135.stdout:4/14: chown l2 168265390 1 2024-03-30T04:28:10.112 INFO:tasks.workunit.client.0.smithi123.stdout:2/486: mknod d0/d1f/d1c/c9d 0 2024-03-30T04:28:10.112 INFO:tasks.workunit.client.1.smithi135.stdout:6/13: truncate f1 3714075 0 2024-03-30T04:28:10.114 INFO:tasks.workunit.client.1.smithi135.stdout:4/15: readlink l2 0 2024-03-30T04:28:10.114 INFO:tasks.workunit.client.1.smithi135.stdout:4/16: dread - no filename 2024-03-30T04:28:10.114 INFO:tasks.workunit.client.1.smithi135.stdout:7/103: write d4/db/de/f1d [1084841,99587] 0 2024-03-30T04:28:10.115 INFO:tasks.workunit.client.1.smithi135.stdout:5/81: dwrite d1/d3/f15 [0,4194304] 0 2024-03-30T04:28:10.115 INFO:tasks.workunit.client.0.smithi123.stdout:2/487: truncate d0/d1f/d5e/d71/f9c 459116 0 2024-03-30T04:28:10.117 INFO:tasks.workunit.client.0.smithi123.stdout:2/488: write d0/d1f/d1c/d80/f5c [1588196,106483] 0 2024-03-30T04:28:10.117 INFO:tasks.workunit.client.1.smithi135.stdout:3/28: write f2 [542925,46025] 0 2024-03-30T04:28:10.121 INFO:tasks.workunit.client.1.smithi135.stdout:6/14: creat d2/f4 x:0 0 0 2024-03-30T04:28:10.121 INFO:tasks.workunit.client.1.smithi135.stdout:2/20: creat d4/f5 x:0 0 0 2024-03-30T04:28:10.123 INFO:tasks.workunit.client.0.smithi123.stdout:2/489: link d0/d1f/d27/f8b d0/d1f/d1c/d39/d4b/d4c/d5f/f9e 0 2024-03-30T04:28:10.126 INFO:tasks.workunit.client.1.smithi135.stdout:4/17: creat f3 x:0 0 0 2024-03-30T04:28:10.129 INFO:tasks.workunit.client.1.smithi135.stdout:3/29: chown f0 389987515 1 2024-03-30T04:28:10.131 INFO:tasks.workunit.client.0.smithi123.stdout:2/490: link d0/d1f/d1c/d20/f46 d0/d1f/d1c/d20/f9f 0 2024-03-30T04:28:10.132 INFO:tasks.workunit.client.1.smithi135.stdout:7/104: chown d4/db/de/d15/l21 157 1 2024-03-30T04:28:10.132 INFO:tasks.workunit.client.1.smithi135.stdout:0/62: getdents d0/d5/d9 0 2024-03-30T04:28:10.140 INFO:tasks.workunit.client.1.smithi135.stdout:8/56: dwrite d6/fa [0,4194304] 0 2024-03-30T04:28:10.142 INFO:tasks.workunit.client.1.smithi135.stdout:2/21: unlink l2 0 2024-03-30T04:28:10.142 INFO:tasks.workunit.client.0.smithi123.stdout:0/494: dread d6/f72 [0,4194304] 0 2024-03-30T04:28:10.148 INFO:tasks.workunit.client.1.smithi135.stdout:0/63: creat d0/d5/d9/ff x:0 0 0 2024-03-30T04:28:10.148 INFO:tasks.workunit.client.1.smithi135.stdout:8/57: chown l1 132228 1 2024-03-30T04:28:10.148 INFO:tasks.workunit.client.0.smithi123.stdout:2/491: dwrite d0/d1f/d1c/d39/d4b/d4c/d5f/f9e [0,4194304] 0 2024-03-30T04:28:10.149 INFO:tasks.workunit.client.1.smithi135.stdout:4/18: fsync f3 0 2024-03-30T04:28:10.149 INFO:tasks.workunit.client.0.smithi123.stdout:2/492: stat d0/d1f/d1c 0 2024-03-30T04:28:10.153 INFO:tasks.workunit.client.0.smithi123.stdout:2/493: creat d0/d1f/d1c/d39/d4b/fa0 x:0 0 0 2024-03-30T04:28:10.156 INFO:tasks.workunit.client.1.smithi135.stdout:2/22: unlink c0 0 2024-03-30T04:28:10.162 INFO:tasks.workunit.client.1.smithi135.stdout:1/36: dwrite f7 [4194304,4194304] 0 2024-03-30T04:28:10.163 INFO:tasks.workunit.client.1.smithi135.stdout:0/64: mkdir d0/d5/d10 0 2024-03-30T04:28:10.164 INFO:tasks.workunit.client.0.smithi123.stdout:2/494: dread d0/d1f/d1c/d2a/f7c [0,4194304] 0 2024-03-30T04:28:10.166 INFO:tasks.workunit.client.1.smithi135.stdout:4/19: link f3 f4 0 2024-03-30T04:28:10.167 INFO:tasks.workunit.client.1.smithi135.stdout:7/105: creat d4/db/f26 x:0 0 0 2024-03-30T04:28:10.167 INFO:tasks.workunit.client.0.smithi123.stdout:2/495: mkdir d0/d37/d57/da1 0 2024-03-30T04:28:10.170 INFO:tasks.workunit.client.0.smithi123.stdout:2/496: read d0/d1f/d27/f8b [3165549,128861] 0 2024-03-30T04:28:10.174 INFO:tasks.workunit.client.0.smithi123.stdout:2/497: creat d0/d1f/d1c/d20/fa2 x:0 0 0 2024-03-30T04:28:10.176 INFO:tasks.workunit.client.0.smithi123.stdout:2/498: write d0/d1f/d16/f87 [1200274,108983] 0 2024-03-30T04:28:10.178 INFO:tasks.workunit.client.0.smithi123.stdout:2/499: write d0/d1f/d1c/d20/fa2 [444444,39005] 0 2024-03-30T04:28:10.181 INFO:tasks.workunit.client.1.smithi135.stdout:3/30: dread f0 [0,4194304] 0 2024-03-30T04:28:10.182 INFO:tasks.workunit.client.1.smithi135.stdout:2/23: getdents d4 0 2024-03-30T04:28:10.184 INFO:tasks.workunit.client.0.smithi123.stdout:2/500: dread d0/d1f/d1c/d36/f89 [0,4194304] 0 2024-03-30T04:28:10.184 INFO:tasks.workunit.client.1.smithi135.stdout:6/15: write f1 [3041064,26301] 0 2024-03-30T04:28:10.184 INFO:tasks.workunit.client.1.smithi135.stdout:0/65: unlink d0/d5/l6 0 2024-03-30T04:28:10.184 INFO:tasks.workunit.client.1.smithi135.stdout:5/82: dwrite d1/d7/f10 [0,4194304] 0 2024-03-30T04:28:10.187 INFO:tasks.workunit.client.0.smithi123.stdout:2/501: symlink d0/d1f/d27/d68/la3 0 2024-03-30T04:28:10.187 INFO:tasks.workunit.client.1.smithi135.stdout:6/16: fsync d2/f4 0 2024-03-30T04:28:10.187 INFO:tasks.workunit.client.1.smithi135.stdout:4/20: write f3 [310590,91180] 0 2024-03-30T04:28:10.187 INFO:tasks.workunit.client.1.smithi135.stdout:1/37: fdatasync f6 0 2024-03-30T04:28:10.188 INFO:tasks.workunit.client.0.smithi123.stdout:2/502: fsync d0/d1f/d16/f18 0 2024-03-30T04:28:10.190 INFO:tasks.workunit.client.1.smithi135.stdout:1/38: stat f5 0 2024-03-30T04:28:10.193 INFO:tasks.workunit.client.1.smithi135.stdout:6/17: fdatasync d2/f4 0 2024-03-30T04:28:10.195 INFO:tasks.workunit.client.0.smithi123.stdout:2/503: truncate d0/d1f/d1c/d39/d4b/d4c/d5f/f67 2696221 0 2024-03-30T04:28:10.196 INFO:tasks.workunit.client.1.smithi135.stdout:4/21: creat f5 x:0 0 0 2024-03-30T04:28:10.197 INFO:tasks.workunit.client.1.smithi135.stdout:3/31: link d3/f9 d3/d6/fa 0 2024-03-30T04:28:10.197 INFO:tasks.workunit.client.1.smithi135.stdout:1/39: fdatasync f7 0 2024-03-30T04:28:10.198 INFO:tasks.workunit.client.1.smithi135.stdout:5/83: mknod d1/da/d17/c1c 0 2024-03-30T04:28:10.199 INFO:tasks.workunit.client.1.smithi135.stdout:8/58: dread f4 [4194304,4194304] 0 2024-03-30T04:28:10.199 INFO:tasks.workunit.client.1.smithi135.stdout:6/18: dread - d2/f4 zero size 2024-03-30T04:28:10.203 INFO:tasks.workunit.client.1.smithi135.stdout:4/22: mknod c6 0 2024-03-30T04:28:10.206 INFO:tasks.workunit.client.0.smithi123.stdout:2/504: dread d0/d1f/f17 [0,4194304] 0 2024-03-30T04:28:10.208 INFO:tasks.workunit.client.1.smithi135.stdout:1/40: creat f8 x:0 0 0 2024-03-30T04:28:10.208 INFO:tasks.workunit.client.1.smithi135.stdout:6/19: dread - d2/f4 zero size 2024-03-30T04:28:10.208 INFO:tasks.workunit.client.0.smithi123.stdout:2/505: write d0/d1f/d16/f33 [3194035,98553] 0 2024-03-30T04:28:10.209 INFO:tasks.workunit.client.1.smithi135.stdout:7/106: dread d4/db/d1a/f24 [0,4194304] 0 2024-03-30T04:28:10.209 INFO:tasks.workunit.client.1.smithi135.stdout:4/23: symlink l7 0 2024-03-30T04:28:10.211 INFO:tasks.workunit.client.0.smithi123.stdout:2/506: mknod d0/d1f/d1c/d36/ca4 0 2024-03-30T04:28:10.213 INFO:tasks.workunit.client.0.smithi123.stdout:2/507: truncate d0/f10 416248 0 2024-03-30T04:28:10.215 INFO:tasks.workunit.client.1.smithi135.stdout:3/32: getdents d3 0 2024-03-30T04:28:10.215 INFO:tasks.workunit.client.1.smithi135.stdout:4/24: chown c6 11 1 2024-03-30T04:28:10.218 INFO:tasks.workunit.client.0.smithi123.stdout:2/508: getdents d0/d1f/d1c/d39/d4b/d4c/d5f 0 2024-03-30T04:28:10.218 INFO:tasks.workunit.client.1.smithi135.stdout:6/20: chown d2/f4 1854 1 2024-03-30T04:28:10.218 INFO:tasks.workunit.client.0.smithi123.stdout:2/509: stat d0/d1f/f8f 0 2024-03-30T04:28:10.220 INFO:tasks.workunit.client.0.smithi123.stdout:2/510: fsync d0/d1f/d1c/d40/f98 0 2024-03-30T04:28:10.222 INFO:tasks.workunit.client.1.smithi135.stdout:3/33: dread - d3/f9 zero size 2024-03-30T04:28:10.222 INFO:tasks.workunit.client.1.smithi135.stdout:4/25: truncate f5 83388 0 2024-03-30T04:28:10.225 INFO:tasks.workunit.client.1.smithi135.stdout:2/24: dwrite d4/f5 [0,4194304] 0 2024-03-30T04:28:10.228 INFO:tasks.workunit.client.1.smithi135.stdout:3/34: fdatasync d3/f4 0 2024-03-30T04:28:10.228 INFO:tasks.workunit.client.1.smithi135.stdout:7/107: rmdir d4/db/de/d15 39 2024-03-30T04:28:10.229 INFO:tasks.workunit.client.0.smithi123.stdout:2/511: dread d0/d1f/d1c/d2a/f7c [0,4194304] 0 2024-03-30T04:28:10.231 INFO:tasks.workunit.client.1.smithi135.stdout:6/21: unlink f1 0 2024-03-30T04:28:10.231 INFO:tasks.workunit.client.1.smithi135.stdout:0/66: dwrite d0/f3 [0,4194304] 0 2024-03-30T04:28:10.234 INFO:tasks.workunit.client.1.smithi135.stdout:4/26: mknod c8 0 2024-03-30T04:28:10.234 INFO:tasks.workunit.client.1.smithi135.stdout:5/84: link d1/d3/l5 d1/d7/d1a/l1d 0 2024-03-30T04:28:10.238 INFO:tasks.workunit.client.1.smithi135.stdout:6/22: rmdir d2 39 2024-03-30T04:28:10.238 INFO:tasks.workunit.client.1.smithi135.stdout:7/108: symlink d4/l27 0 2024-03-30T04:28:10.241 INFO:tasks.workunit.client.1.smithi135.stdout:5/85: read d1/d7/ff [2713719,45062] 0 2024-03-30T04:28:10.243 INFO:tasks.workunit.client.0.smithi123.stdout:2/512: symlink d0/d1f/d1c/d2a/d8c/la5 0 2024-03-30T04:28:10.246 INFO:tasks.workunit.client.0.smithi123.stdout:2/513: chown d0/d1f/d30/c38 128470 1 2024-03-30T04:28:10.248 INFO:tasks.workunit.client.1.smithi135.stdout:8/59: dwrite f0 [0,4194304] 0 2024-03-30T04:28:10.250 INFO:tasks.workunit.client.1.smithi135.stdout:4/27: mknod c9 0 2024-03-30T04:28:10.250 INFO:tasks.workunit.client.1.smithi135.stdout:1/41: dwrite f6 [0,4194304] 0 2024-03-30T04:28:10.253 INFO:tasks.workunit.client.1.smithi135.stdout:3/35: getdents d3/d6 0 2024-03-30T04:28:10.258 INFO:tasks.workunit.client.1.smithi135.stdout:6/23: truncate d2/f4 135399 0 2024-03-30T04:28:10.258 INFO:tasks.workunit.client.1.smithi135.stdout:0/67: link d0/d1/fa d0/d5/d10/f11 0 2024-03-30T04:28:10.259 INFO:tasks.workunit.client.1.smithi135.stdout:1/42: mknod c9 0 2024-03-30T04:28:10.262 INFO:tasks.workunit.client.1.smithi135.stdout:4/28: fdatasync f5 0 2024-03-30T04:28:10.263 INFO:tasks.workunit.client.1.smithi135.stdout:3/36: mkdir d3/db 0 2024-03-30T04:28:10.267 INFO:tasks.workunit.client.1.smithi135.stdout:9/29: sync 2024-03-30T04:28:10.272 INFO:tasks.workunit.client.1.smithi135.stdout:7/109: creat d4/db/de/f28 x:0 0 0 2024-03-30T04:28:10.272 INFO:tasks.workunit.client.1.smithi135.stdout:0/68: creat d0/f12 x:0 0 0 2024-03-30T04:28:10.273 INFO:tasks.workunit.client.1.smithi135.stdout:8/60: truncate d6/d8/db/fe 1001033 0 2024-03-30T04:28:10.275 INFO:tasks.workunit.client.1.smithi135.stdout:1/43: unlink f8 0 2024-03-30T04:28:10.277 INFO:tasks.workunit.client.1.smithi135.stdout:8/61: fdatasync d6/fa 0 2024-03-30T04:28:10.277 INFO:tasks.workunit.client.1.smithi135.stdout:4/29: write f4 [818662,123465] 0 2024-03-30T04:28:10.277 INFO:tasks.workunit.client.0.smithi123.stdout:2/514: dwrite d0/d1f/d30/f94 [0,4194304] 0 2024-03-30T04:28:10.278 INFO:tasks.workunit.client.1.smithi135.stdout:2/25: dwrite d4/f5 [4194304,4194304] 0 2024-03-30T04:28:10.278 INFO:tasks.workunit.client.1.smithi135.stdout:0/69: dread - d0/f12 zero size 2024-03-30T04:28:10.279 INFO:tasks.workunit.client.1.smithi135.stdout:1/44: write f7 [2407043,43371] 0 2024-03-30T04:28:10.279 INFO:tasks.workunit.client.0.smithi123.stdout:2/515: chown d0/d1f/d1c/d39/d4b/d70/d97 85738 1 2024-03-30T04:28:10.280 INFO:tasks.workunit.client.1.smithi135.stdout:6/24: rename d2/c3 to d2/c5 0 2024-03-30T04:28:10.281 INFO:tasks.workunit.client.1.smithi135.stdout:3/37: mknod d3/cc 0 2024-03-30T04:28:10.282 INFO:tasks.workunit.client.1.smithi135.stdout:2/26: write f3 [775748,20787] 0 2024-03-30T04:28:10.283 INFO:tasks.workunit.client.1.smithi135.stdout:8/62: write f4 [10502766,91879] 0 2024-03-30T04:28:10.285 INFO:tasks.workunit.client.0.smithi123.stdout:2/516: rename d0/d1f/d1c/c9d to d0/d1f/d1c/d39/d4b/d70/d97/ca6 0 2024-03-30T04:28:10.286 INFO:tasks.workunit.client.1.smithi135.stdout:6/25: fsync d2/f4 0 2024-03-30T04:28:10.287 INFO:tasks.workunit.client.0.smithi123.stdout:2/517: dread - d0/d1f/d1c/d20/f56 zero size 2024-03-30T04:28:10.287 INFO:tasks.workunit.client.1.smithi135.stdout:6/26: chown d2 1 1 2024-03-30T04:28:10.289 INFO:tasks.workunit.client.1.smithi135.stdout:4/30: creat fa x:0 0 0 2024-03-30T04:28:10.289 INFO:tasks.workunit.client.1.smithi135.stdout:0/70: chown d0/d5/d9/fb 1 1 2024-03-30T04:28:10.289 INFO:tasks.workunit.client.0.smithi123.stdout:2/518: creat d0/d1f/d1c/d39/d4b/fa7 x:0 0 0 2024-03-30T04:28:10.290 INFO:tasks.workunit.client.1.smithi135.stdout:8/63: chown f0 1853 1 2024-03-30T04:28:10.290 INFO:tasks.workunit.client.1.smithi135.stdout:1/45: write f3 [1761934,76245] 0 2024-03-30T04:28:10.290 INFO:tasks.workunit.client.1.smithi135.stdout:5/86: dread d1/f4 [0,4194304] 0 2024-03-30T04:28:10.292 INFO:tasks.workunit.client.0.smithi123.stdout:2/519: write d0/d1f/d16/f23 [658363,21746] 0 2024-03-30T04:28:10.292 INFO:tasks.workunit.client.1.smithi135.stdout:8/64: fsync d6/fa 0 2024-03-30T04:28:10.294 INFO:tasks.workunit.client.0.smithi123.stdout:2/520: mkdir d0/d1f/d1c/d80/da8 0 2024-03-30T04:28:10.296 INFO:tasks.workunit.client.0.smithi123.stdout:2/521: chown d0/d1f/d1c/d39/d4b/d70/c79 10 1 2024-03-30T04:28:10.299 INFO:tasks.workunit.client.0.smithi123.stdout:2/522: symlink d0/d1f/d1c/d2a/la9 0 2024-03-30T04:28:10.301 INFO:tasks.workunit.client.0.smithi123.stdout:2/523: read d0/d1f/d27/f8b [610794,126446] 0 2024-03-30T04:28:10.303 INFO:tasks.workunit.client.1.smithi135.stdout:3/38: mknod d3/d6/cd 0 2024-03-30T04:28:10.305 INFO:tasks.workunit.client.0.smithi123.stdout:2/524: symlink d0/d1f/d1c/laa 0 2024-03-30T04:28:10.306 INFO:tasks.workunit.client.1.smithi135.stdout:4/31: creat fb x:0 0 0 2024-03-30T04:28:10.308 INFO:tasks.workunit.client.0.smithi123.stdout:2/525: creat d0/d1f/d1c/d36/fab x:0 0 0 2024-03-30T04:28:10.308 INFO:tasks.workunit.client.0.smithi123.stdout:2/526: fdatasync d0/d1f/d16/f1e 0 2024-03-30T04:28:10.311 INFO:tasks.workunit.client.0.smithi123.stdout:2/527: getdents d0/d1f/d16/d81 0 2024-03-30T04:28:10.312 INFO:tasks.workunit.client.1.smithi135.stdout:1/46: creat fa x:0 0 0 2024-03-30T04:28:10.313 INFO:tasks.workunit.client.1.smithi135.stdout:0/71: unlink d0/d1/le 0 2024-03-30T04:28:10.313 INFO:tasks.workunit.client.1.smithi135.stdout:2/27: getdents d4 0 2024-03-30T04:28:10.313 INFO:tasks.workunit.client.0.smithi123.stdout:2/528: creat d0/d1f/d1c/d40/fac x:0 0 0 2024-03-30T04:28:10.314 INFO:tasks.workunit.client.0.smithi123.stdout:2/529: chown d0/d1f/d1c/d40 16 1 2024-03-30T04:28:10.315 INFO:tasks.workunit.client.0.smithi123.stdout:2/530: readlink d0/le 0 2024-03-30T04:28:10.316 INFO:tasks.workunit.client.0.smithi123.stdout:2/531: mknod d0/cad 0 2024-03-30T04:28:10.317 INFO:tasks.workunit.client.1.smithi135.stdout:4/32: mknod cc 0 2024-03-30T04:28:10.318 INFO:tasks.workunit.client.0.smithi123.stdout:8/397: sync 2024-03-30T04:28:10.318 INFO:tasks.workunit.client.0.smithi123.stdout:2/532: write d0/d1f/f17 [66805,52646] 0 2024-03-30T04:28:10.319 INFO:tasks.workunit.client.0.smithi123.stdout:8/398: chown da/d27/d3c/d2d 237017283 1 2024-03-30T04:28:10.321 INFO:tasks.workunit.client.0.smithi123.stdout:2/533: creat d0/d1f/d1c/d80/da8/fae x:0 0 0 2024-03-30T04:28:10.322 INFO:tasks.workunit.client.1.smithi135.stdout:7/110: dwrite d4/f11 [0,4194304] 0 2024-03-30T04:28:10.322 INFO:tasks.workunit.client.0.smithi123.stdout:2/534: readlink d0/d1f/d1c/d39/l85 0 2024-03-30T04:28:10.322 INFO:tasks.workunit.client.0.smithi123.stdout:8/399: getdents da/d27/d3c/d64 0 2024-03-30T04:28:10.326 INFO:tasks.workunit.client.0.smithi123.stdout:2/535: rename d0/d1f/d27/l91 to d0/d1f/d1c/d36/laf 0 2024-03-30T04:28:10.329 INFO:tasks.workunit.client.0.smithi123.stdout:8/400: read da/d27/d3c/d2d/f57 [503735,121964] 0 2024-03-30T04:28:10.329 INFO:tasks.workunit.client.0.smithi123.stdout:2/536: creat d0/d1f/d27/fb0 x:0 0 0 2024-03-30T04:28:10.331 INFO:tasks.workunit.client.0.smithi123.stdout:8/401: write da/d27/d3c/f51 [745624,46296] 0 2024-03-30T04:28:10.332 INFO:tasks.workunit.client.1.smithi135.stdout:0/72: mkdir d0/d13 0 2024-03-30T04:28:10.334 INFO:tasks.workunit.client.0.smithi123.stdout:8/402: creat da/dc/d14/f71 x:0 0 0 2024-03-30T04:28:10.335 INFO:tasks.workunit.client.1.smithi135.stdout:6/27: dwrite d2/f4 [0,4194304] 0 2024-03-30T04:28:10.336 INFO:tasks.workunit.client.1.smithi135.stdout:3/39: creat d3/db/fe x:0 0 0 2024-03-30T04:28:10.336 INFO:tasks.workunit.client.0.smithi123.stdout:2/537: rename d0/d1f/d1c/d39/d4b/d4c/f6a to d0/d1f/d1c/d39/d4b/fb1 0 2024-03-30T04:28:10.339 INFO:tasks.workunit.client.1.smithi135.stdout:5/87: link d1/cb d1/d7/d1a/c1e 0 2024-03-30T04:28:10.339 INFO:tasks.workunit.client.0.smithi123.stdout:8/403: creat da/d27/d56/d6c/d6d/f72 x:0 0 0 2024-03-30T04:28:10.340 INFO:tasks.workunit.client.0.smithi123.stdout:2/538: symlink d0/d1f/d1c/d2a/d8c/lb2 0 2024-03-30T04:28:10.340 INFO:tasks.workunit.client.1.smithi135.stdout:7/111: creat d4/f29 x:0 0 0 2024-03-30T04:28:10.341 INFO:tasks.workunit.client.1.smithi135.stdout:9/30: dwrite f1 [0,4194304] 0 2024-03-30T04:28:10.342 INFO:tasks.workunit.client.1.smithi135.stdout:0/73: fdatasync d0/d5/d9/fd 0 2024-03-30T04:28:10.346 INFO:tasks.workunit.client.1.smithi135.stdout:9/31: write f1 [4364465,25614] 0 2024-03-30T04:28:10.346 INFO:tasks.workunit.client.1.smithi135.stdout:5/88: dread - d1/da/f1b zero size 2024-03-30T04:28:10.347 INFO:tasks.workunit.client.0.smithi123.stdout:2/539: creat d0/d1f/d1c/fb3 x:0 0 0 2024-03-30T04:28:10.348 INFO:tasks.workunit.client.1.smithi135.stdout:8/65: dwrite d6/fa [0,4194304] 0 2024-03-30T04:28:10.349 INFO:tasks.workunit.client.1.smithi135.stdout:0/74: write d0/f12 [500098,129716] 0 2024-03-30T04:28:10.352 INFO:tasks.workunit.client.0.smithi123.stdout:2/540: creat d0/fb4 x:0 0 0 2024-03-30T04:28:10.354 INFO:tasks.workunit.client.0.smithi123.stdout:2/541: chown d0/f1d 935914 1 2024-03-30T04:28:10.355 INFO:tasks.workunit.client.1.smithi135.stdout:5/89: stat d1/d3/f15 0 2024-03-30T04:28:10.355 INFO:tasks.workunit.client.1.smithi135.stdout:6/28: symlink d2/l6 0 2024-03-30T04:28:10.356 INFO:tasks.workunit.client.1.smithi135.stdout:3/40: truncate d3/f9 128950 0 2024-03-30T04:28:10.358 INFO:tasks.workunit.client.1.smithi135.stdout:5/90: stat d1/d7/d1a 0 2024-03-30T04:28:10.358 INFO:tasks.workunit.client.1.smithi135.stdout:9/32: unlink l5 0 2024-03-30T04:28:10.359 INFO:tasks.workunit.client.0.smithi123.stdout:2/542: mkdir d0/d1f/d1c/d39/d4b/d70/d97/db5 0 2024-03-30T04:28:10.360 INFO:tasks.workunit.client.1.smithi135.stdout:2/28: dwrite f3 [0,4194304] 0 2024-03-30T04:28:10.363 INFO:tasks.workunit.client.1.smithi135.stdout:0/75: symlink d0/l14 0 2024-03-30T04:28:10.363 INFO:tasks.workunit.client.0.smithi123.stdout:8/404: truncate da/d27/f3b 1514790 0 2024-03-30T04:28:10.364 INFO:tasks.workunit.client.1.smithi135.stdout:6/29: rmdir d2 39 2024-03-30T04:28:10.367 INFO:tasks.workunit.client.1.smithi135.stdout:5/91: fdatasync d1/d7/f13 0 2024-03-30T04:28:10.368 INFO:tasks.workunit.client.0.smithi123.stdout:8/405: mknod da/c73 0 2024-03-30T04:28:10.368 INFO:tasks.workunit.client.0.smithi123.stdout:2/543: link d0/d1f/d1c/d2a/c88 d0/d1f/d1c/d39/d4b/d70/d97/cb6 0 2024-03-30T04:28:10.372 INFO:tasks.workunit.client.0.smithi123.stdout:2/544: mknod d0/d1f/d1c/d39/d4b/d4c/cb7 0 2024-03-30T04:28:10.373 INFO:tasks.workunit.client.0.smithi123.stdout:2/545: stat d0/d1f/c35 0 2024-03-30T04:28:10.376 INFO:tasks.workunit.client.1.smithi135.stdout:0/76: creat d0/d5/d9/f15 x:0 0 0 2024-03-30T04:28:10.376 INFO:tasks.workunit.client.1.smithi135.stdout:8/66: link f4 d6/f11 0 2024-03-30T04:28:10.378 INFO:tasks.workunit.client.0.smithi123.stdout:2/546: link d0/d1f/d1c/d39/d4b/d4c/f64 d0/d1f/d1c/d20/fb8 0 2024-03-30T04:28:10.381 INFO:tasks.workunit.client.0.smithi123.stdout:2/547: creat d0/d1f/d5e/fb9 x:0 0 0 2024-03-30T04:28:10.382 INFO:tasks.workunit.client.1.smithi135.stdout:5/92: truncate d1/d7/ff 198582 0 2024-03-30T04:28:10.382 INFO:tasks.workunit.client.1.smithi135.stdout:6/30: write d2/f4 [63101,39552] 0 2024-03-30T04:28:10.383 INFO:tasks.workunit.client.0.smithi123.stdout:2/548: dread - d0/d1f/d82/f92 zero size 2024-03-30T04:28:10.385 INFO:tasks.workunit.client.0.smithi123.stdout:2/549: readlink d0/d1f/d1c/l45 0 2024-03-30T04:28:10.386 INFO:tasks.workunit.client.0.smithi123.stdout:8/406: truncate da/dc/f20 2436233 0 2024-03-30T04:28:10.386 INFO:tasks.workunit.client.0.smithi123.stdout:2/550: dread - d0/d1f/f8f zero size 2024-03-30T04:28:10.387 INFO:tasks.workunit.client.1.smithi135.stdout:6/31: chown d2/f4 1 1 2024-03-30T04:28:10.388 INFO:tasks.workunit.client.0.smithi123.stdout:2/551: readlink d0/d1f/d27/d68/la3 0 2024-03-30T04:28:10.388 INFO:tasks.workunit.client.1.smithi135.stdout:0/77: mkdir d0/d1/d16 0 2024-03-30T04:28:10.388 INFO:tasks.workunit.client.0.smithi123.stdout:8/407: mkdir da/d27/d3c/d2d/d74 0 2024-03-30T04:28:10.390 INFO:tasks.workunit.client.1.smithi135.stdout:5/93: fsync d1/d7/f13 0 2024-03-30T04:28:10.391 INFO:tasks.workunit.client.0.smithi123.stdout:8/408: symlink da/d27/d56/d6c/d6d/l75 0 2024-03-30T04:28:10.394 INFO:tasks.workunit.client.0.smithi123.stdout:8/409: write da/f55 [3975244,40707] 0 2024-03-30T04:28:10.394 INFO:tasks.workunit.client.1.smithi135.stdout:3/41: write d3/f9 [873361,21166] 0 2024-03-30T04:28:10.395 INFO:tasks.workunit.client.0.smithi123.stdout:2/552: getdents d0/d1f/d27 0 2024-03-30T04:28:10.396 INFO:tasks.workunit.client.1.smithi135.stdout:6/32: write d2/f4 [554924,9026] 0 2024-03-30T04:28:10.396 INFO:tasks.workunit.client.1.smithi135.stdout:5/94: chown d1/f4 264998 1 2024-03-30T04:28:10.399 INFO:tasks.workunit.client.0.smithi123.stdout:8/410: rename da/d27/c54 to da/d27/d56/d6c/c76 0 2024-03-30T04:28:10.402 INFO:tasks.workunit.client.1.smithi135.stdout:2/29: dwrite f3 [0,4194304] 0 2024-03-30T04:28:10.402 INFO:tasks.workunit.client.0.smithi123.stdout:9/444: sync 2024-03-30T04:28:10.403 INFO:tasks.workunit.client.1.smithi135.stdout:6/33: fdatasync d2/f4 0 2024-03-30T04:28:10.404 INFO:tasks.workunit.client.0.smithi123.stdout:8/411: read f8 [1604282,21450] 0 2024-03-30T04:28:10.405 INFO:tasks.workunit.client.1.smithi135.stdout:0/78: dread d0/f12 [0,4194304] 0 2024-03-30T04:28:10.405 INFO:tasks.workunit.client.1.smithi135.stdout:7/112: dwrite d4/db/d1a/f24 [0,4194304] 0 2024-03-30T04:28:10.407 INFO:tasks.workunit.client.0.smithi123.stdout:9/445: truncate d6/d1f/f41 4501075 0 2024-03-30T04:28:10.408 INFO:tasks.workunit.client.1.smithi135.stdout:2/30: truncate f3 4651072 0 2024-03-30T04:28:10.409 INFO:tasks.workunit.client.0.smithi123.stdout:8/412: mknod da/d27/d56/d6c/c77 0 2024-03-30T04:28:10.413 INFO:tasks.workunit.client.0.smithi123.stdout:9/446: mknod d6/d1f/d72/ca1 0 2024-03-30T04:28:10.414 INFO:tasks.workunit.client.1.smithi135.stdout:2/31: mkdir d4/d6 0 2024-03-30T04:28:10.416 INFO:tasks.workunit.client.1.smithi135.stdout:8/67: dread f4 [4194304,4194304] 0 2024-03-30T04:28:10.416 INFO:tasks.workunit.client.0.smithi123.stdout:8/413: mkdir da/d27/d3c/d64/d78 0 2024-03-30T04:28:10.416 INFO:tasks.workunit.client.1.smithi135.stdout:7/113: write d4/f16 [159835,27159] 0 2024-03-30T04:28:10.417 INFO:tasks.workunit.client.0.smithi123.stdout:2/553: dwrite d0/d1f/f6e [0,4194304] 0 2024-03-30T04:28:10.418 INFO:tasks.workunit.client.1.smithi135.stdout:0/79: creat d0/d5/d9/f17 x:0 0 0 2024-03-30T04:28:10.421 INFO:tasks.workunit.client.0.smithi123.stdout:9/447: chown d6/l1e 33 1 2024-03-30T04:28:10.422 INFO:tasks.workunit.client.1.smithi135.stdout:8/68: chown d6/ld 53810979 1 2024-03-30T04:28:10.423 INFO:tasks.workunit.client.1.smithi135.stdout:2/32: creat d4/f7 x:0 0 0 2024-03-30T04:28:10.426 INFO:tasks.workunit.client.1.smithi135.stdout:7/114: mkdir d4/db/d1a/d2a 0 2024-03-30T04:28:10.434 INFO:tasks.workunit.client.1.smithi135.stdout:5/95: dread d1/f4 [0,4194304] 0 2024-03-30T04:28:10.435 INFO:tasks.workunit.client.1.smithi135.stdout:0/80: rename d0/l14 to d0/d5/d10/l18 0 2024-03-30T04:28:10.438 INFO:tasks.workunit.client.0.smithi123.stdout:2/554: dread d0/d1f/d1c/f41 [0,4194304] 0 2024-03-30T04:28:10.441 INFO:tasks.workunit.client.0.smithi123.stdout:2/555: unlink d0/d1f/d16/c63 0 2024-03-30T04:28:10.441 INFO:tasks.workunit.client.1.smithi135.stdout:3/42: dwrite d3/f4 [0,4194304] 0 2024-03-30T04:28:10.442 INFO:tasks.workunit.client.0.smithi123.stdout:8/414: write f6 [1311107,128481] 0 2024-03-30T04:28:10.444 INFO:tasks.workunit.client.1.smithi135.stdout:3/43: chown d3/d6/f7 0 1 2024-03-30T04:28:10.445 INFO:tasks.workunit.client.0.smithi123.stdout:2/556: rmdir d0/d1f/d1c/d39/d4b/d70/d97 39 2024-03-30T04:28:10.446 INFO:tasks.workunit.client.1.smithi135.stdout:3/44: fdatasync d3/d6/f7 0 2024-03-30T04:28:10.447 INFO:tasks.workunit.client.0.smithi123.stdout:8/415: creat da/d27/d56/d6c/d6d/f79 x:0 0 0 2024-03-30T04:28:10.449 INFO:tasks.workunit.client.0.smithi123.stdout:2/557: symlink d0/d1f/d82/lba 0 2024-03-30T04:28:10.449 INFO:tasks.workunit.client.1.smithi135.stdout:8/69: unlink d6/f7 0 2024-03-30T04:28:10.451 INFO:tasks.workunit.client.0.smithi123.stdout:8/416: mknod da/d27/d3c/d2d/d5c/c7a 0 2024-03-30T04:28:10.451 INFO:tasks.workunit.client.1.smithi135.stdout:3/45: chown d3/d6/cd 0 1 2024-03-30T04:28:10.453 INFO:tasks.workunit.client.0.smithi123.stdout:9/448: dread d6/d1f/f2d [0,4194304] 0 2024-03-30T04:28:10.454 INFO:tasks.workunit.client.0.smithi123.stdout:8/417: mknod da/d27/d3c/d2d/c7b 0 2024-03-30T04:28:10.455 INFO:tasks.workunit.client.0.smithi123.stdout:9/449: stat d6/c81 0 2024-03-30T04:28:10.456 INFO:tasks.workunit.client.1.smithi135.stdout:0/81: rmdir d0/d5/d10 39 2024-03-30T04:28:10.457 INFO:tasks.workunit.client.1.smithi135.stdout:3/46: read d3/d6/fa [700175,122850] 0 2024-03-30T04:28:10.458 INFO:tasks.workunit.client.0.smithi123.stdout:8/418: unlink da/d27/d3c/d2d/d5c/c7a 0 2024-03-30T04:28:10.458 INFO:tasks.workunit.client.1.smithi135.stdout:2/33: dread d4/f5 [4194304,4194304] 0 2024-03-30T04:28:10.461 INFO:tasks.workunit.client.1.smithi135.stdout:5/96: dread d1/d7/f10 [0,4194304] 0 2024-03-30T04:28:10.463 INFO:tasks.workunit.client.0.smithi123.stdout:8/419: rename da/l1c to da/d27/d3c/d2d/l7c 0 2024-03-30T04:28:10.463 INFO:tasks.workunit.client.1.smithi135.stdout:5/97: stat d1/d3 0 2024-03-30T04:28:10.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:10 smithi123 ceph-mon[35245]: pgmap v38: 65 pgs: 65 active+clean; 789 MiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 16 MiB/s rd, 89 MiB/s wr, 253 op/s 2024-03-30T04:28:10.467 INFO:tasks.workunit.client.0.smithi123.stdout:8/420: rename da/d27/l43 to da/d27/d3c/l7d 0 2024-03-30T04:28:10.469 INFO:tasks.workunit.client.0.smithi123.stdout:9/450: dwrite d6/f10 [0,4194304] 0 2024-03-30T04:28:10.469 INFO:tasks.workunit.client.1.smithi135.stdout:8/70: dread f4 [0,4194304] 0 2024-03-30T04:28:10.472 INFO:tasks.workunit.client.0.smithi123.stdout:9/451: read d6/d1f/f71 [4070507,93476] 0 2024-03-30T04:28:10.472 INFO:tasks.workunit.client.1.smithi135.stdout:3/47: mknod d3/cf 0 2024-03-30T04:28:10.474 INFO:tasks.workunit.client.1.smithi135.stdout:3/48: fdatasync d3/f5 0 2024-03-30T04:28:10.474 INFO:tasks.workunit.client.1.smithi135.stdout:5/98: unlink d1/d7/d1a/l1d 0 2024-03-30T04:28:10.475 INFO:tasks.workunit.client.0.smithi123.stdout:9/452: write d6/d1f/f29 [8793849,55918] 0 2024-03-30T04:28:10.477 INFO:tasks.workunit.client.1.smithi135.stdout:7/115: creat d4/db/de/f2b x:0 0 0 2024-03-30T04:28:10.478 INFO:tasks.workunit.client.0.smithi123.stdout:9/453: symlink d6/d58/la2 0 2024-03-30T04:28:10.478 INFO:tasks.workunit.client.1.smithi135.stdout:2/34: dwrite f3 [0,4194304] 0 2024-03-30T04:28:10.480 INFO:tasks.workunit.client.0.smithi123.stdout:9/454: chown d6/d1f/d45/d97/d69/f6c 1638065839 1 2024-03-30T04:28:10.482 INFO:tasks.workunit.client.0.smithi123.stdout:5/449: sync 2024-03-30T04:28:10.482 INFO:tasks.workunit.client.0.smithi123.stdout:7/427: sync 2024-03-30T04:28:10.482 INFO:tasks.workunit.client.0.smithi123.stdout:3/532: sync 2024-03-30T04:28:10.482 INFO:tasks.workunit.client.1.smithi135.stdout:3/49: mkdir d3/d10 0 2024-03-30T04:28:10.483 INFO:tasks.workunit.client.0.smithi123.stdout:9/455: chown d6/d1f/d72/d7d/c9d 18823535 1 2024-03-30T04:28:10.484 INFO:tasks.workunit.client.0.smithi123.stdout:6/569: sync 2024-03-30T04:28:10.484 INFO:tasks.workunit.client.0.smithi123.stdout:4/448: sync 2024-03-30T04:28:10.484 INFO:tasks.workunit.client.0.smithi123.stdout:0/495: sync 2024-03-30T04:28:10.485 INFO:tasks.workunit.client.0.smithi123.stdout:1/576: sync 2024-03-30T04:28:10.485 INFO:tasks.workunit.client.0.smithi123.stdout:8/421: dwrite da/dc/d14/f36 [0,4194304] 0 2024-03-30T04:28:10.486 INFO:tasks.workunit.client.1.smithi135.stdout:7/116: mkdir d4/db/d1a/d2a/d2c 0 2024-03-30T04:28:10.487 INFO:tasks.workunit.client.1.smithi135.stdout:2/35: symlink d4/d6/l8 0 2024-03-30T04:28:10.489 INFO:tasks.workunit.client.1.smithi135.stdout:0/82: dwrite d0/f12 [0,4194304] 0 2024-03-30T04:28:10.490 INFO:tasks.workunit.client.1.smithi135.stdout:7/117: write d4/db/de/f1d [2046433,84341] 0 2024-03-30T04:28:10.491 INFO:tasks.workunit.client.0.smithi123.stdout:9/456: unlink d6/d23/f3b 0 2024-03-30T04:28:10.491 INFO:tasks.workunit.client.0.smithi123.stdout:0/496: readlink d6/dd/d15/d98/d4e/l63 0 2024-03-30T04:28:10.491 INFO:tasks.workunit.client.0.smithi123.stdout:7/428: mkdir d2/da/d11/d34/d76/d91 0 2024-03-30T04:28:10.492 INFO:tasks.workunit.client.1.smithi135.stdout:0/83: read - d0/d5/d9/f15 zero size 2024-03-30T04:28:10.492 INFO:tasks.workunit.client.0.smithi123.stdout:3/533: creat d13/d24/d31/d23/d63/fa7 x:0 0 0 2024-03-30T04:28:10.493 INFO:tasks.workunit.client.0.smithi123.stdout:5/450: write dd/d19/d1f/d35/f4b [4576681,104228] 0 2024-03-30T04:28:10.494 INFO:tasks.workunit.client.0.smithi123.stdout:9/457: write d6/f10 [4257680,25452] 0 2024-03-30T04:28:10.496 INFO:tasks.workunit.client.0.smithi123.stdout:0/497: dread - d6/dd/d15/f8e zero size 2024-03-30T04:28:10.496 INFO:tasks.workunit.client.0.smithi123.stdout:7/429: chown d2/da/d22/f2f 3 1 2024-03-30T04:28:10.498 INFO:tasks.workunit.client.0.smithi123.stdout:3/534: fsync d13/d24/d31/d23/d3a/f40 0 2024-03-30T04:28:10.500 INFO:tasks.workunit.client.1.smithi135.stdout:0/84: mkdir d0/d19 0 2024-03-30T04:28:10.500 INFO:tasks.workunit.client.0.smithi123.stdout:8/422: link da/dc/f42 da/d27/f7e 0 2024-03-30T04:28:10.501 INFO:tasks.workunit.client.1.smithi135.stdout:7/118: creat d4/db/d1a/d2a/d2c/f2d x:0 0 0 2024-03-30T04:28:10.502 INFO:tasks.workunit.client.0.smithi123.stdout:4/449: mknod d2/d4/d2c/ca0 0 2024-03-30T04:28:10.503 INFO:tasks.workunit.client.1.smithi135.stdout:3/50: read d3/f9 [166789,11956] 0 2024-03-30T04:28:10.504 INFO:tasks.workunit.client.0.smithi123.stdout:8/423: stat da/d27/d56/c60 0 2024-03-30T04:28:10.505 INFO:tasks.workunit.client.0.smithi123.stdout:5/451: creat dd/d19/d1f/d3c/f92 x:0 0 0 2024-03-30T04:28:10.505 INFO:tasks.workunit.client.0.smithi123.stdout:6/570: link d10/d20/c35 d10/d20/da2/cc4 0 2024-03-30T04:28:10.505 INFO:tasks.workunit.client.0.smithi123.stdout:7/430: mknod d2/da/d11/d61/c92 0 2024-03-30T04:28:10.506 INFO:tasks.workunit.client.0.smithi123.stdout:1/577: creat db/dd/d5f/d6d/fbf x:0 0 0 2024-03-30T04:28:10.507 INFO:tasks.workunit.client.0.smithi123.stdout:0/498: write d6/fa [420778,79591] 0 2024-03-30T04:28:10.508 INFO:tasks.workunit.client.1.smithi135.stdout:0/85: write d0/f12 [1810844,95337] 0 2024-03-30T04:28:10.511 INFO:tasks.workunit.client.1.smithi135.stdout:2/36: truncate f3 1100599 0 2024-03-30T04:28:10.511 INFO:tasks.workunit.client.0.smithi123.stdout:9/458: mkdir d6/d1f/da3 0 2024-03-30T04:28:10.511 INFO:tasks.workunit.client.0.smithi123.stdout:0/499: symlink d6/dd/lab 0 2024-03-30T04:28:10.512 INFO:tasks.workunit.client.1.smithi135.stdout:7/119: mkdir d4/db/d1a/d2a/d2c/d2e 0 2024-03-30T04:28:10.513 INFO:tasks.workunit.client.0.smithi123.stdout:9/459: chown d6/d23/f5d 280 1 2024-03-30T04:28:10.513 INFO:tasks.workunit.client.0.smithi123.stdout:7/431: fdatasync d2/da/d11/d61/f72 0 2024-03-30T04:28:10.513 INFO:tasks.workunit.client.1.smithi135.stdout:3/51: creat d3/f11 x:0 0 0 2024-03-30T04:28:10.513 INFO:tasks.workunit.client.1.smithi135.stdout:0/86: read d0/f12 [1758231,64298] 0 2024-03-30T04:28:10.516 INFO:tasks.workunit.client.1.smithi135.stdout:8/71: dwrite f5 [0,4194304] 0 2024-03-30T04:28:10.516 INFO:tasks.workunit.client.0.smithi123.stdout:8/424: creat da/d27/d3c/d2d/d74/f7f x:0 0 0 2024-03-30T04:28:10.519 INFO:tasks.workunit.client.0.smithi123.stdout:1/578: mknod db/dd/d5f/d6d/d85/cc0 0 2024-03-30T04:28:10.520 INFO:tasks.workunit.client.0.smithi123.stdout:7/432: unlink d2/c12 0 2024-03-30T04:28:10.520 INFO:tasks.workunit.client.0.smithi123.stdout:0/500: creat d6/dd/d18/d2f/fac x:0 0 0 2024-03-30T04:28:10.521 INFO:tasks.workunit.client.1.smithi135.stdout:3/52: creat d3/d6/f12 x:0 0 0 2024-03-30T04:28:10.523 INFO:tasks.workunit.client.0.smithi123.stdout:4/450: rename d2/d4/d7/l36 to d2/d4/la1 0 2024-03-30T04:28:10.524 INFO:tasks.workunit.client.1.smithi135.stdout:3/53: fsync d3/d6/f7 0 2024-03-30T04:28:10.524 INFO:tasks.workunit.client.1.smithi135.stdout:0/87: symlink d0/l1a 0 2024-03-30T04:28:10.525 INFO:tasks.workunit.client.0.smithi123.stdout:1/579: readlink db/dd/d42/d46/d56/lbe 0 2024-03-30T04:28:10.525 INFO:tasks.workunit.client.0.smithi123.stdout:0/501: readlink d6/dd/lab 0 2024-03-30T04:28:10.526 INFO:tasks.workunit.client.1.smithi135.stdout:7/120: creat d4/db/d1a/d2a/d2c/d2e/f2f x:0 0 0 2024-03-30T04:28:10.529 INFO:tasks.workunit.client.1.smithi135.stdout:0/88: write d0/d5/d9/f17 [880543,69470] 0 2024-03-30T04:28:10.529 INFO:tasks.workunit.client.0.smithi123.stdout:0/502: readlink d6/l76 0 2024-03-30T04:28:10.529 INFO:tasks.workunit.client.0.smithi123.stdout:5/452: getdents dd/d19/d1f/d3c/d72 0 2024-03-30T04:28:10.529 INFO:tasks.workunit.client.0.smithi123.stdout:9/460: link d6/d1f/d72/f8b d6/d58/fa4 0 2024-03-30T04:28:10.529 INFO:tasks.workunit.client.1.smithi135.stdout:3/54: truncate d3/f9 1850864 0 2024-03-30T04:28:10.532 INFO:tasks.workunit.client.0.smithi123.stdout:8/425: rename da/d27/d3c/d2d/d5c/l69 to da/d27/d56/l80 0 2024-03-30T04:28:10.533 INFO:tasks.workunit.client.1.smithi135.stdout:3/55: chown d3/d10 12026931 1 2024-03-30T04:28:10.535 INFO:tasks.workunit.client.0.smithi123.stdout:4/451: truncate d2/d4/f16 979617 0 2024-03-30T04:28:10.537 INFO:tasks.workunit.client.1.smithi135.stdout:0/89: creat d0/d1/d16/f1b x:0 0 0 2024-03-30T04:28:10.537 INFO:tasks.workunit.client.0.smithi123.stdout:6/571: dwrite d10/d20/d3f/f4d [0,4194304] 0 2024-03-30T04:28:10.538 INFO:tasks.workunit.client.0.smithi123.stdout:0/503: mkdir d6/dd/d15/d28/d4b/dad 0 2024-03-30T04:28:10.539 INFO:tasks.workunit.client.1.smithi135.stdout:8/72: dread f4 [0,4194304] 0 2024-03-30T04:28:10.540 INFO:tasks.workunit.client.0.smithi123.stdout:8/426: truncate da/d27/d3c/f5e 324059 0 2024-03-30T04:28:10.543 INFO:tasks.workunit.client.0.smithi123.stdout:6/572: truncate d10/d82/dc1/d45/d77/fad 1023592 0 2024-03-30T04:28:10.545 INFO:tasks.workunit.client.0.smithi123.stdout:0/504: mknod d6/dd/cae 0 2024-03-30T04:28:10.545 INFO:tasks.workunit.client.0.smithi123.stdout:7/433: link d2/da/d11/d34/d76/l7a d2/dc/d49/l93 0 2024-03-30T04:28:10.547 INFO:tasks.workunit.client.1.smithi135.stdout:2/37: dwrite d4/f5 [4194304,4194304] 0 2024-03-30T04:28:10.548 INFO:tasks.workunit.client.0.smithi123.stdout:7/434: dread - d2/dc/d56/f67 zero size 2024-03-30T04:28:10.549 INFO:tasks.workunit.client.0.smithi123.stdout:1/580: rmdir db/dd/d5f/d6d/d85/d89/da7 0 2024-03-30T04:28:10.551 INFO:tasks.workunit.client.0.smithi123.stdout:6/573: symlink d10/d20/lc5 0 2024-03-30T04:28:10.551 INFO:tasks.workunit.client.1.smithi135.stdout:7/121: dread f3 [0,4194304] 0 2024-03-30T04:28:10.551 INFO:tasks.workunit.client.1.smithi135.stdout:2/38: rmdir d4/d6 39 2024-03-30T04:28:10.552 INFO:tasks.workunit.client.0.smithi123.stdout:8/427: truncate da/fe 1981924 0 2024-03-30T04:28:10.553 INFO:tasks.workunit.client.0.smithi123.stdout:0/505: rename d6/dd/d15/d28/d4b/f77 to d6/dd/d18/d1f/faf 0 2024-03-30T04:28:10.554 INFO:tasks.workunit.client.1.smithi135.stdout:2/39: truncate d4/f7 134512 0 2024-03-30T04:28:10.554 INFO:tasks.workunit.client.1.smithi135.stdout:7/122: dread - d4/db/de/f28 zero size 2024-03-30T04:28:10.555 INFO:tasks.workunit.client.1.smithi135.stdout:2/40: chown f3 1981 1 2024-03-30T04:28:10.555 INFO:tasks.workunit.client.0.smithi123.stdout:8/428: read - da/d27/d3c/d2d/d5c/f6f zero size 2024-03-30T04:28:10.555 INFO:tasks.workunit.client.0.smithi123.stdout:7/435: mkdir d2/da/d11/d61/d55/d94 0 2024-03-30T04:28:10.555 INFO:tasks.workunit.client.0.smithi123.stdout:0/506: dread - d6/dd/d15/d98/d5d/f83 zero size 2024-03-30T04:28:10.557 INFO:tasks.workunit.client.0.smithi123.stdout:6/574: unlink d10/d82/dc1/f39 0 2024-03-30T04:28:10.557 INFO:tasks.workunit.client.1.smithi135.stdout:2/41: stat d4/d6/l8 0 2024-03-30T04:28:10.559 INFO:tasks.workunit.client.0.smithi123.stdout:0/507: chown d6/dd/d16/c1a 28185 1 2024-03-30T04:28:10.560 INFO:tasks.workunit.client.0.smithi123.stdout:7/436: mkdir d2/da/d11/d34/d76/d95 0 2024-03-30T04:28:10.563 INFO:tasks.workunit.client.0.smithi123.stdout:6/575: unlink d10/d20/c44 0 2024-03-30T04:28:10.563 INFO:tasks.workunit.client.0.smithi123.stdout:8/429: dread da/f3d [0,4194304] 0 2024-03-30T04:28:10.564 INFO:tasks.workunit.client.0.smithi123.stdout:7/437: write d2/da/d22/d3b/f4e [442747,43997] 0 2024-03-30T04:28:10.565 INFO:tasks.workunit.client.0.smithi123.stdout:1/581: rename db/dd/d42/f98 to db/dd/d5f/d6d/fc1 0 2024-03-30T04:28:10.567 INFO:tasks.workunit.client.0.smithi123.stdout:6/576: truncate d10/d20/dac/f6f 1942585 0 2024-03-30T04:28:10.568 INFO:tasks.workunit.client.1.smithi135.stdout:2/42: dread d4/f5 [4194304,4194304] 0 2024-03-30T04:28:10.569 INFO:tasks.workunit.client.0.smithi123.stdout:8/430: chown da/f46 118469985 1 2024-03-30T04:28:10.569 INFO:tasks.workunit.client.0.smithi123.stdout:7/438: dread - d2/da/d2d/f43 zero size 2024-03-30T04:28:10.570 INFO:tasks.workunit.client.1.smithi135.stdout:4/33: sync 2024-03-30T04:28:10.570 INFO:tasks.workunit.client.1.smithi135.stdout:7/123: dwrite d4/db/d1a/f24 [0,4194304] 0 2024-03-30T04:28:10.570 INFO:tasks.workunit.client.0.smithi123.stdout:1/582: read db/dd/d42/f61 [3755300,78755] 0 2024-03-30T04:28:10.571 INFO:tasks.workunit.client.1.smithi135.stdout:4/34: stat c8 0 2024-03-30T04:28:10.571 INFO:tasks.workunit.client.0.smithi123.stdout:6/577: dread - d10/d20/dac/fa4 zero size 2024-03-30T04:28:10.573 INFO:tasks.workunit.client.0.smithi123.stdout:0/508: dread d6/dd/d15/d28/f3d [0,4194304] 0 2024-03-30T04:28:10.573 INFO:tasks.workunit.client.1.smithi135.stdout:2/43: unlink l1 0 2024-03-30T04:28:10.575 INFO:tasks.workunit.client.1.smithi135.stdout:4/35: creat fd x:0 0 0 2024-03-30T04:28:10.576 INFO:tasks.workunit.client.1.smithi135.stdout:7/124: write d4/db/de/f25 [2460163,95115] 0 2024-03-30T04:28:10.580 INFO:tasks.workunit.client.1.smithi135.stdout:2/44: mknod d4/c9 0 2024-03-30T04:28:10.580 INFO:tasks.workunit.client.0.smithi123.stdout:7/439: mknod d2/c96 0 2024-03-30T04:28:10.581 INFO:tasks.workunit.client.0.smithi123.stdout:0/509: creat d6/dd/d15/d28/d4b/d8a/fb0 x:0 0 0 2024-03-30T04:28:10.585 INFO:tasks.workunit.client.0.smithi123.stdout:6/578: creat d10/d82/dc1/d6a/d7a/d9f/fc6 x:0 0 0 2024-03-30T04:28:10.585 INFO:tasks.workunit.client.0.smithi123.stdout:7/440: write d2/da/d22/d3b/f6d [737424,111981] 0 2024-03-30T04:28:10.590 INFO:tasks.workunit.client.1.smithi135.stdout:7/125: creat d4/db/de/f30 x:0 0 0 2024-03-30T04:28:10.590 INFO:tasks.workunit.client.0.smithi123.stdout:1/583: creat db/dd/d42/d46/d51/fc2 x:0 0 0 2024-03-30T04:28:10.595 INFO:tasks.workunit.client.1.smithi135.stdout:7/126: fsync d4/db/d1a/d2a/d2c/f2d 0 2024-03-30T04:28:10.595 INFO:tasks.workunit.client.0.smithi123.stdout:8/431: dwrite da/f4e [0,4194304] 0 2024-03-30T04:28:10.598 INFO:tasks.workunit.client.0.smithi123.stdout:6/579: rename d10/d20/dac to d10/d82/d8e/dc7 0 2024-03-30T04:28:10.600 INFO:tasks.workunit.client.0.smithi123.stdout:8/432: truncate da/d27/f53 783991 0 2024-03-30T04:28:10.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:10 smithi135 ceph-mon[44364]: pgmap v38: 65 pgs: 65 active+clean; 789 MiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 16 MiB/s rd, 89 MiB/s wr, 253 op/s 2024-03-30T04:28:10.604 INFO:tasks.workunit.client.0.smithi123.stdout:0/510: dwrite d6/dd/d31/d35/f7b [0,4194304] 0 2024-03-30T04:28:10.607 INFO:tasks.workunit.client.0.smithi123.stdout:1/584: creat db/dd/d42/d46/d5d/d62/d9e/fc3 x:0 0 0 2024-03-30T04:28:10.607 INFO:tasks.workunit.client.0.smithi123.stdout:6/580: dread - d10/d82/d8e/dc7/d42/f87 zero size 2024-03-30T04:28:10.609 INFO:tasks.workunit.client.1.smithi135.stdout:7/127: creat d4/db/de/d15/f31 x:0 0 0 2024-03-30T04:28:10.610 INFO:tasks.workunit.client.0.smithi123.stdout:0/511: readlink d6/dd/d15/d98/d4e/l53 0 2024-03-30T04:28:10.611 INFO:tasks.workunit.client.0.smithi123.stdout:6/581: mknod d10/d82/dc1/d6a/d7a/d99/cc8 0 2024-03-30T04:28:10.612 INFO:tasks.workunit.client.1.smithi135.stdout:2/45: dread d4/f5 [4194304,4194304] 0 2024-03-30T04:28:10.613 INFO:tasks.workunit.client.1.smithi135.stdout:4/36: dwrite f5 [0,4194304] 0 2024-03-30T04:28:10.615 INFO:tasks.workunit.client.0.smithi123.stdout:6/582: chown d10/d82/d8e/dc7/d5b/l72 485 1 2024-03-30T04:28:10.616 INFO:tasks.workunit.client.0.smithi123.stdout:7/441: dread d2/da/d2d/f2a [0,4194304] 0 2024-03-30T04:28:10.621 INFO:tasks.workunit.client.1.smithi135.stdout:8/73: dwrite f0 [0,4194304] 0 2024-03-30T04:28:10.623 INFO:tasks.workunit.client.0.smithi123.stdout:3/535: dread d13/d24/d31/f64 [4194304,4194304] 0 2024-03-30T04:28:10.623 INFO:tasks.workunit.client.0.smithi123.stdout:7/442: write d2/da/d22/d3b/f54 [1416497,22504] 0 2024-03-30T04:28:10.624 INFO:tasks.workunit.client.1.smithi135.stdout:0/90: dwrite d0/d1/fa [0,4194304] 0 2024-03-30T04:28:10.626 INFO:tasks.workunit.client.0.smithi123.stdout:1/585: getdents db/dd/d5f/d6d/d85 0 2024-03-30T04:28:10.630 INFO:tasks.workunit.client.1.smithi135.stdout:7/128: creat d4/db/d1a/d2a/f32 x:0 0 0 2024-03-30T04:28:10.631 INFO:tasks.workunit.client.1.smithi135.stdout:7/129: chown d4 7 1 2024-03-30T04:28:10.631 INFO:tasks.workunit.client.1.smithi135.stdout:4/37: dread f3 [0,4194304] 0 2024-03-30T04:28:10.633 INFO:tasks.workunit.client.0.smithi123.stdout:4/452: write d2/d26/f70 [1011749,26031] 0 2024-03-30T04:28:10.633 INFO:tasks.workunit.client.0.smithi123.stdout:5/453: write dd/f32 [1424827,131047] 0 2024-03-30T04:28:10.635 INFO:tasks.workunit.client.1.smithi135.stdout:0/91: symlink d0/d5/d9/l1c 0 2024-03-30T04:28:10.636 INFO:tasks.workunit.client.1.smithi135.stdout:4/38: creat fe x:0 0 0 2024-03-30T04:28:10.639 INFO:tasks.workunit.client.1.smithi135.stdout:8/74: creat d6/f12 x:0 0 0 2024-03-30T04:28:10.640 INFO:tasks.workunit.client.0.smithi123.stdout:0/512: dread d6/f72 [0,4194304] 0 2024-03-30T04:28:10.642 INFO:tasks.workunit.client.1.smithi135.stdout:8/75: write f4 [5104799,99700] 0 2024-03-30T04:28:10.643 INFO:tasks.workunit.client.1.smithi135.stdout:4/39: rename fe to ff 0 2024-03-30T04:28:10.646 INFO:tasks.workunit.client.0.smithi123.stdout:0/513: chown d6/dd/d15/d98/d4e/d69/f7a 1672137 1 2024-03-30T04:28:10.647 INFO:tasks.workunit.client.1.smithi135.stdout:4/40: mkdir d10 0 2024-03-30T04:28:10.650 INFO:tasks.workunit.client.0.smithi123.stdout:0/514: chown d6/dd/d15/d98/d4e/d69 1129416 1 2024-03-30T04:28:10.650 INFO:tasks.workunit.client.1.smithi135.stdout:2/46: truncate d4/f5 3226908 0 2024-03-30T04:28:10.653 INFO:tasks.workunit.client.1.smithi135.stdout:4/41: mknod d10/c11 0 2024-03-30T04:28:10.656 INFO:tasks.workunit.client.0.smithi123.stdout:3/536: creat d13/d24/d31/d23/d3a/d3c/fa8 x:0 0 0 2024-03-30T04:28:10.657 INFO:tasks.workunit.client.1.smithi135.stdout:2/47: mkdir d4/da 0 2024-03-30T04:28:10.658 INFO:tasks.workunit.client.0.smithi123.stdout:0/515: truncate d6/dd/d31/d35/d3a/f8f 1004414 0 2024-03-30T04:28:10.659 INFO:tasks.workunit.client.1.smithi135.stdout:7/130: dread d4/db/de/f25 [0,4194304] 0 2024-03-30T04:28:10.659 INFO:tasks.workunit.client.1.smithi135.stdout:0/92: dread d0/f12 [0,4194304] 0 2024-03-30T04:28:10.659 INFO:tasks.workunit.client.0.smithi123.stdout:3/537: stat d13/d24/d31/d23/d3a/d3c/d66/f7f 0 2024-03-30T04:28:10.660 INFO:tasks.workunit.client.0.smithi123.stdout:5/454: dwrite dd/d19/d3e/d5b/d62/f84 [0,4194304] 0 2024-03-30T04:28:10.661 INFO:tasks.workunit.client.1.smithi135.stdout:8/76: dread d6/fa [0,4194304] 0 2024-03-30T04:28:10.667 INFO:tasks.workunit.client.0.smithi123.stdout:5/455: dread - dd/f7c zero size 2024-03-30T04:28:10.667 INFO:tasks.workunit.client.0.smithi123.stdout:0/516: read d6/dd/f8c [767836,97203] 0 2024-03-30T04:28:10.669 INFO:tasks.workunit.client.0.smithi123.stdout:5/456: readlink dd/d19/d1f/d35/d3a/l5a 0 2024-03-30T04:28:10.669 INFO:tasks.workunit.client.1.smithi135.stdout:7/131: rename d4/l7 to d4/db/d1a/d2a/l33 0 2024-03-30T04:28:10.671 INFO:tasks.workunit.client.0.smithi123.stdout:6/583: truncate d10/d20/f40 2851581 0 2024-03-30T04:28:10.673 INFO:tasks.workunit.client.0.smithi123.stdout:1/586: truncate db/dd/d42/d46/d39/f92 22101 0 2024-03-30T04:28:10.673 INFO:tasks.workunit.client.0.smithi123.stdout:7/443: write d2/da/d11/d61/f57 [452684,21602] 0 2024-03-30T04:28:10.676 INFO:tasks.workunit.client.0.smithi123.stdout:5/457: creat dd/d19/d1f/d68/f93 x:0 0 0 2024-03-30T04:28:10.678 INFO:tasks.workunit.client.0.smithi123.stdout:0/517: chown d6/dd/d18/d1f/faf 7 1 2024-03-30T04:28:10.679 INFO:tasks.workunit.client.0.smithi123.stdout:1/587: readlink db/l4e 0 2024-03-30T04:28:10.682 INFO:tasks.workunit.client.1.smithi135.stdout:8/77: dread f5 [0,4194304] 0 2024-03-30T04:28:10.684 INFO:tasks.workunit.client.1.smithi135.stdout:7/132: rename d4/cc to d4/db/d1a/d2a/d2c/d2e/c34 0 2024-03-30T04:28:10.689 INFO:tasks.workunit.client.1.smithi135.stdout:0/93: dwrite d0/f3 [0,4194304] 0 2024-03-30T04:28:10.689 INFO:tasks.workunit.client.0.smithi123.stdout:7/444: fdatasync d2/dc/f81 0 2024-03-30T04:28:10.691 INFO:tasks.workunit.client.0.smithi123.stdout:3/538: dwrite d13/fa1 [0,4194304] 0 2024-03-30T04:28:10.693 INFO:tasks.workunit.client.0.smithi123.stdout:7/445: fdatasync d2/da/d11/f20 0 2024-03-30T04:28:10.693 INFO:tasks.workunit.client.1.smithi135.stdout:0/94: mkdir d0/d13/d1d 0 2024-03-30T04:28:10.696 INFO:tasks.workunit.client.0.smithi123.stdout:6/584: creat d10/d82/d9a/db3/fc9 x:0 0 0 2024-03-30T04:28:10.696 INFO:tasks.workunit.client.0.smithi123.stdout:3/539: write d13/d24/d31/d30/f9a [614317,92365] 0 2024-03-30T04:28:10.697 INFO:tasks.workunit.client.1.smithi135.stdout:0/95: mkdir d0/d13/d1e 0 2024-03-30T04:28:10.702 INFO:tasks.workunit.client.1.smithi135.stdout:0/96: dread - d0/d5/d9/fb zero size 2024-03-30T04:28:10.703 INFO:tasks.workunit.client.0.smithi123.stdout:1/588: symlink db/dd/d5f/dae/lc4 0 2024-03-30T04:28:10.705 INFO:tasks.workunit.client.1.smithi135.stdout:8/78: dread d6/fa [0,4194304] 0 2024-03-30T04:28:10.709 INFO:tasks.workunit.client.0.smithi123.stdout:7/446: rename d2/l35 to d2/da/d11/d34/d85/l97 0 2024-03-30T04:28:10.712 INFO:tasks.workunit.client.1.smithi135.stdout:0/97: write d0/d5/d9/fd [171664,108034] 0 2024-03-30T04:28:10.712 INFO:tasks.workunit.client.1.smithi135.stdout:8/79: write d6/f12 [625249,22089] 0 2024-03-30T04:28:10.714 INFO:tasks.workunit.client.1.smithi135.stdout:8/80: chown d6 0 1 2024-03-30T04:28:10.716 INFO:tasks.workunit.client.1.smithi135.stdout:7/133: dwrite d4/f29 [0,4194304] 0 2024-03-30T04:28:10.716 INFO:tasks.workunit.client.0.smithi123.stdout:7/447: readlink d2/da/l44 0 2024-03-30T04:28:10.718 INFO:tasks.workunit.client.0.smithi123.stdout:3/540: mkdir d13/d24/d31/d23/d3a/d3c/d66/da9 0 2024-03-30T04:28:10.721 INFO:tasks.workunit.client.0.smithi123.stdout:6/585: rename d10/d82/dc1/d6a/f90 to d10/d82/dc1/d6a/fca 0 2024-03-30T04:28:10.721 INFO:tasks.workunit.client.1.smithi135.stdout:8/81: mknod d6/d8/db/c13 0 2024-03-30T04:28:10.722 INFO:tasks.workunit.client.0.smithi123.stdout:4/453: dread d2/d4/d2c/d42/f38 [0,4194304] 0 2024-03-30T04:28:10.723 INFO:tasks.workunit.client.0.smithi123.stdout:1/589: mknod db/d1b/cc5 0 2024-03-30T04:28:10.724 INFO:tasks.workunit.client.1.smithi135.stdout:8/82: chown d6/d8/l10 796 1 2024-03-30T04:28:10.724 INFO:tasks.workunit.client.1.smithi135.stdout:7/134: write d4/db/de/d15/f31 [374141,103269] 0 2024-03-30T04:28:10.725 INFO:tasks.workunit.client.0.smithi123.stdout:6/586: write d10/f9e [379535,55228] 0 2024-03-30T04:28:10.727 INFO:tasks.workunit.client.1.smithi135.stdout:7/135: write d4/db/de/f2b [589474,73152] 0 2024-03-30T04:28:10.727 INFO:tasks.workunit.client.0.smithi123.stdout:0/518: dwrite d6/dd/d15/d28/d4b/d8a/f93 [0,4194304] 0 2024-03-30T04:28:10.728 INFO:tasks.workunit.client.1.smithi135.stdout:0/98: link d0/d5/d9/ff d0/d13/d1d/f1f 0 2024-03-30T04:28:10.728 INFO:tasks.workunit.client.0.smithi123.stdout:5/458: dwrite dd/d19/f85 [0,4194304] 0 2024-03-30T04:28:10.729 INFO:tasks.workunit.client.0.smithi123.stdout:3/541: truncate d13/d24/d39/f9e 688713 0 2024-03-30T04:28:10.730 INFO:tasks.workunit.client.1.smithi135.stdout:2/48: dwrite d4/f5 [0,4194304] 0 2024-03-30T04:28:10.731 INFO:tasks.workunit.client.1.smithi135.stdout:0/99: readlink d0/l1a 0 2024-03-30T04:28:10.735 INFO:tasks.workunit.client.0.smithi123.stdout:3/542: chown d13/d24/l4c 226792576 1 2024-03-30T04:28:10.736 INFO:tasks.workunit.client.1.smithi135.stdout:7/136: symlink d4/db/d1a/d2a/d2c/l35 0 2024-03-30T04:28:10.737 INFO:tasks.workunit.client.0.smithi123.stdout:0/519: read d6/fa [1332045,118277] 0 2024-03-30T04:28:10.743 INFO:tasks.workunit.client.0.smithi123.stdout:7/448: mkdir d2/d4d/d80/d98 0 2024-03-30T04:28:10.747 INFO:tasks.workunit.client.0.smithi123.stdout:0/520: fdatasync d6/d52/fa8 0 2024-03-30T04:28:10.758 INFO:tasks.workunit.client.0.smithi123.stdout:7/449: dread d2/da/d22/d3b/f4e [0,4194304] 0 2024-03-30T04:28:10.758 INFO:tasks.workunit.client.0.smithi123.stdout:1/590: rename db/dd/d42/d46/d56/lbe to db/dd/d42/d46/d56/lc6 0 2024-03-30T04:28:10.761 INFO:tasks.workunit.client.1.smithi135.stdout:8/83: dwrite d6/f12 [0,4194304] 0 2024-03-30T04:28:10.763 INFO:tasks.workunit.client.1.smithi135.stdout:2/49: dread d4/f5 [0,4194304] 0 2024-03-30T04:28:10.764 INFO:tasks.workunit.client.0.smithi123.stdout:3/543: link d13/d24/d31/d23/d63/f71 d13/d24/d31/d30/faa 0 2024-03-30T04:28:10.767 INFO:tasks.workunit.client.0.smithi123.stdout:5/459: dwrite dd/d19/f22 [0,4194304] 0 2024-03-30T04:28:10.769 INFO:tasks.workunit.client.1.smithi135.stdout:2/50: write f3 [66843,75937] 0 2024-03-30T04:28:10.770 INFO:tasks.workunit.client.0.smithi123.stdout:4/454: write d2/d4/f16 [772064,127901] 0 2024-03-30T04:28:10.772 INFO:tasks.workunit.client.0.smithi123.stdout:4/455: readlink d2/d4/d2c/d42/l8c 0 2024-03-30T04:28:10.774 INFO:tasks.workunit.client.1.smithi135.stdout:0/100: dwrite d0/f12 [0,4194304] 0 2024-03-30T04:28:10.774 INFO:tasks.workunit.client.1.smithi135.stdout:9/33: sync 2024-03-30T04:28:10.775 INFO:tasks.workunit.client.1.smithi135.stdout:8/84: rename d6/d8/db to d6/d8/d14 0 2024-03-30T04:28:10.775 INFO:tasks.workunit.client.1.smithi135.stdout:1/47: sync 2024-03-30T04:28:10.776 INFO:tasks.workunit.client.0.smithi123.stdout:1/591: symlink db/dd/d42/d46/d51/d72/lc7 0 2024-03-30T04:28:10.781 INFO:tasks.workunit.client.0.smithi123.stdout:7/450: creat d2/da/d11/d61/d55/d94/f99 x:0 0 0 2024-03-30T04:28:10.782 INFO:tasks.workunit.client.1.smithi135.stdout:1/48: creat fb x:0 0 0 2024-03-30T04:28:10.783 INFO:tasks.workunit.client.1.smithi135.stdout:7/137: dwrite d4/db/de/d15/f31 [0,4194304] 0 2024-03-30T04:28:10.783 INFO:tasks.workunit.client.1.smithi135.stdout:9/34: write f2 [3457004,21315] 0 2024-03-30T04:28:10.786 INFO:tasks.workunit.client.0.smithi123.stdout:7/451: fdatasync d2/da/d11/d61/d55/d59/f88 0 2024-03-30T04:28:10.788 INFO:tasks.workunit.client.1.smithi135.stdout:2/51: creat d4/da/fb x:0 0 0 2024-03-30T04:28:10.788 INFO:tasks.workunit.client.0.smithi123.stdout:0/521: link d6/f40 d6/dd/d18/d2f/fb1 0 2024-03-30T04:28:10.788 INFO:tasks.workunit.client.0.smithi123.stdout:5/460: dread dd/d19/d1f/d35/f38 [0,4194304] 0 2024-03-30T04:28:10.789 INFO:tasks.workunit.client.0.smithi123.stdout:7/452: write d2/da/d11/f20 [363337,5871] 0 2024-03-30T04:28:10.790 INFO:tasks.workunit.client.0.smithi123.stdout:1/592: creat db/dd/d42/d46/d51/d72/fc8 x:0 0 0 2024-03-30T04:28:10.790 INFO:tasks.workunit.client.0.smithi123.stdout:4/456: mkdir d2/d4/d2c/d42/d35/d9b/da2 0 2024-03-30T04:28:10.794 INFO:tasks.workunit.client.1.smithi135.stdout:9/35: unlink f2 0 2024-03-30T04:28:10.798 INFO:tasks.workunit.client.0.smithi123.stdout:7/453: readlink d2/da/d11/d61/d55/l5a 0 2024-03-30T04:28:10.798 INFO:tasks.workunit.client.1.smithi135.stdout:2/52: write d4/f5 [3668240,44233] 0 2024-03-30T04:28:10.804 INFO:tasks.workunit.client.1.smithi135.stdout:1/49: dread f7 [0,4194304] 0 2024-03-30T04:28:10.805 INFO:tasks.workunit.client.1.smithi135.stdout:0/101: dread d0/f12 [0,4194304] 0 2024-03-30T04:28:10.807 INFO:tasks.workunit.client.1.smithi135.stdout:1/50: mknod cc 0 2024-03-30T04:28:10.811 INFO:tasks.workunit.client.0.smithi123.stdout:0/522: dread d6/dd/f51 [0,4194304] 0 2024-03-30T04:28:10.812 INFO:tasks.workunit.client.1.smithi135.stdout:9/36: dread f6 [0,4194304] 0 2024-03-30T04:28:10.812 INFO:tasks.workunit.client.1.smithi135.stdout:8/85: rmdir d6 39 2024-03-30T04:28:10.812 INFO:tasks.workunit.client.1.smithi135.stdout:0/102: write d0/f12 [2575687,49911] 0 2024-03-30T04:28:10.813 INFO:tasks.workunit.client.1.smithi135.stdout:1/51: creat fd x:0 0 0 2024-03-30T04:28:10.813 INFO:tasks.workunit.client.1.smithi135.stdout:8/86: chown f4 8006968 1 2024-03-30T04:28:10.814 INFO:tasks.workunit.client.0.smithi123.stdout:5/461: dwrite dd/d19/d3e/d5b/f8a [0,4194304] 0 2024-03-30T04:28:10.815 INFO:tasks.workunit.client.0.smithi123.stdout:4/457: truncate d2/d4/d2c/f84 134851 0 2024-03-30T04:28:10.815 INFO:tasks.workunit.client.1.smithi135.stdout:1/52: stat c1 0 2024-03-30T04:28:10.816 INFO:tasks.workunit.client.1.smithi135.stdout:7/138: dread d4/db/de/f1d [0,4194304] 0 2024-03-30T04:28:10.816 INFO:tasks.workunit.client.1.smithi135.stdout:1/53: fdatasync f3 0 2024-03-30T04:28:10.816 INFO:tasks.workunit.client.1.smithi135.stdout:0/103: write d0/f3 [3412317,103738] 0 2024-03-30T04:28:10.817 INFO:tasks.workunit.client.0.smithi123.stdout:1/593: write db/dd/d42/d46/d39/f52 [368509,85083] 0 2024-03-30T04:28:10.817 INFO:tasks.workunit.client.0.smithi123.stdout:0/523: mkdir d6/dd/d15/d98/d4e/d9a/db2 0 2024-03-30T04:28:10.818 INFO:tasks.workunit.client.1.smithi135.stdout:1/54: chown fa 8346827 1 2024-03-30T04:28:10.820 INFO:tasks.workunit.client.0.smithi123.stdout:1/594: write db/dd/f54 [322464,85045] 0 2024-03-30T04:28:10.821 INFO:tasks.workunit.client.1.smithi135.stdout:1/55: mknod ce 0 2024-03-30T04:28:10.822 INFO:tasks.workunit.client.0.smithi123.stdout:7/454: write d2/f13 [337055,118856] 0 2024-03-30T04:28:10.822 INFO:tasks.workunit.client.1.smithi135.stdout:8/87: write d6/d8/d14/fe [1799071,52805] 0 2024-03-30T04:28:10.823 INFO:tasks.workunit.client.1.smithi135.stdout:1/56: write fa [814573,4687] 0 2024-03-30T04:28:10.823 INFO:tasks.workunit.client.1.smithi135.stdout:2/53: fsync d4/f5 0 2024-03-30T04:28:10.823 INFO:tasks.workunit.client.0.smithi123.stdout:4/458: mkdir d2/d4/d2c/d42/d35/da3 0 2024-03-30T04:28:10.825 INFO:tasks.workunit.client.0.smithi123.stdout:7/455: chown d2/da/d11/d34/l6e 0 1 2024-03-30T04:28:10.826 INFO:tasks.workunit.client.1.smithi135.stdout:0/104: symlink d0/d13/d1d/l20 0 2024-03-30T04:28:10.826 INFO:tasks.workunit.client.0.smithi123.stdout:4/459: write d2/d4/d2c/d42/f3a [3623261,47746] 0 2024-03-30T04:28:10.828 INFO:tasks.workunit.client.1.smithi135.stdout:1/57: chown c2 5343261 1 2024-03-30T04:28:10.828 INFO:tasks.workunit.client.0.smithi123.stdout:4/460: chown d2/d4/d7/d1a/c99 22415 1 2024-03-30T04:28:10.830 INFO:tasks.workunit.client.0.smithi123.stdout:5/462: truncate dd/f15 969407 0 2024-03-30T04:28:10.832 INFO:tasks.workunit.client.1.smithi135.stdout:8/88: write f4 [9037758,82952] 0 2024-03-30T04:28:10.832 INFO:tasks.workunit.client.0.smithi123.stdout:1/595: creat db/dd/d5f/d6d/d85/d89/fc9 x:0 0 0 2024-03-30T04:28:10.834 INFO:tasks.workunit.client.1.smithi135.stdout:0/105: creat d0/f21 x:0 0 0 2024-03-30T04:28:10.835 INFO:tasks.workunit.client.0.smithi123.stdout:0/524: link d6/dd/d18/d1f/faf d6/dd/d15/fb3 0 2024-03-30T04:28:10.835 INFO:tasks.workunit.client.1.smithi135.stdout:8/89: stat f4 0 2024-03-30T04:28:10.837 INFO:tasks.workunit.client.1.smithi135.stdout:1/58: symlink lf 0 2024-03-30T04:28:10.837 INFO:tasks.workunit.client.1.smithi135.stdout:2/54: rmdir d4 39 2024-03-30T04:28:10.838 INFO:tasks.workunit.client.0.smithi123.stdout:5/463: mknod dd/d19/c94 0 2024-03-30T04:28:10.840 INFO:tasks.workunit.client.1.smithi135.stdout:0/106: stat d0/d13/d1d/f1f 0 2024-03-30T04:28:10.848 INFO:tasks.workunit.client.0.smithi123.stdout:7/456: write d2/dc/d1c/f23 [1080663,85325] 0 2024-03-30T04:28:10.849 INFO:tasks.workunit.client.0.smithi123.stdout:4/461: dwrite d2/d34/f3f [0,4194304] 0 2024-03-30T04:28:10.851 INFO:tasks.workunit.client.1.smithi135.stdout:1/59: rename lf to l10 0 2024-03-30T04:28:10.851 INFO:tasks.workunit.client.0.smithi123.stdout:2/558: sync 2024-03-30T04:28:10.852 INFO:tasks.workunit.client.1.smithi135.stdout:8/90: mkdir d6/d8/d15 0 2024-03-30T04:28:10.853 INFO:tasks.workunit.client.1.smithi135.stdout:0/107: symlink d0/d5/d9/l22 0 2024-03-30T04:28:10.855 INFO:tasks.workunit.client.1.smithi135.stdout:9/37: dwrite f1 [0,4194304] 0 2024-03-30T04:28:10.856 INFO:tasks.workunit.client.1.smithi135.stdout:1/60: symlink l11 0 2024-03-30T04:28:10.858 INFO:tasks.workunit.client.0.smithi123.stdout:4/462: fdatasync d2/d4/d2c/d42/d35/f5c 0 2024-03-30T04:28:10.858 INFO:tasks.workunit.client.0.smithi123.stdout:7/457: creat d2/da/d11/d34/d76/d91/f9a x:0 0 0 2024-03-30T04:28:10.858 INFO:tasks.workunit.client.1.smithi135.stdout:7/139: dwrite d4/db/d1a/d2a/f32 [0,4194304] 0 2024-03-30T04:28:10.862 INFO:tasks.workunit.client.1.smithi135.stdout:9/38: mkdir d7/d8 0 2024-03-30T04:28:10.863 INFO:tasks.workunit.client.1.smithi135.stdout:0/108: chown d0/d5/d9/fb 0 1 2024-03-30T04:28:10.864 INFO:tasks.workunit.client.0.smithi123.stdout:1/596: dread db/d1b/f1d [4194304,4194304] 0 2024-03-30T04:28:10.866 INFO:tasks.workunit.client.1.smithi135.stdout:7/140: chown d4/db/f1f 222924850 1 2024-03-30T04:28:10.866 INFO:tasks.workunit.client.0.smithi123.stdout:2/559: rename d0/d1f/d1c/d39/d4b/d70 to d0/d1f/d1c/d80/dbb 0 2024-03-30T04:28:10.868 INFO:tasks.workunit.client.1.smithi135.stdout:8/91: getdents d6/d8/d14 0 2024-03-30T04:28:10.869 INFO:tasks.workunit.client.0.smithi123.stdout:5/464: dread dd/f15 [0,4194304] 0 2024-03-30T04:28:10.869 INFO:tasks.workunit.client.0.smithi123.stdout:2/560: readlink d0/d1f/d1c/d20/l2b 0 2024-03-30T04:28:10.869 INFO:tasks.workunit.client.0.smithi123.stdout:0/525: dwrite d6/dd/d18/d1f/faf [0,4194304] 0 2024-03-30T04:28:10.873 INFO:tasks.workunit.client.1.smithi135.stdout:9/39: write f6 [4644612,105718] 0 2024-03-30T04:28:10.873 INFO:tasks.workunit.client.1.smithi135.stdout:0/109: mkdir d0/d5/d9/d23 0 2024-03-30T04:28:10.876 INFO:tasks.workunit.client.0.smithi123.stdout:7/458: getdents d2/dc/d49/d68 0 2024-03-30T04:28:10.877 INFO:tasks.workunit.client.0.smithi123.stdout:2/561: creat d0/d1f/d5e/fbc x:0 0 0 2024-03-30T04:28:10.878 INFO:tasks.workunit.client.1.smithi135.stdout:2/55: dwrite d4/da/fb [0,4194304] 0 2024-03-30T04:28:10.878 INFO:tasks.workunit.client.1.smithi135.stdout:7/141: getdents d4/db/d1a 0 2024-03-30T04:28:10.878 INFO:tasks.workunit.client.1.smithi135.stdout:0/110: write d0/f12 [2746417,111764] 0 2024-03-30T04:28:10.880 INFO:tasks.workunit.client.1.smithi135.stdout:7/142: fdatasync d4/f16 0 2024-03-30T04:28:10.880 INFO:tasks.workunit.client.0.smithi123.stdout:1/597: read db/dd/d42/d8f/f73 [493637,122669] 0 2024-03-30T04:28:10.880 INFO:tasks.workunit.client.0.smithi123.stdout:7/459: write d2/da/d11/d61/f72 [3841653,27172] 0 2024-03-30T04:28:10.880 INFO:tasks.workunit.client.1.smithi135.stdout:0/111: fdatasync d0/f3 0 2024-03-30T04:28:10.881 INFO:tasks.workunit.client.0.smithi123.stdout:0/526: mknod d6/dd/d15/d98/d4e/d9a/cb4 0 2024-03-30T04:28:10.882 INFO:tasks.workunit.client.1.smithi135.stdout:2/56: rename f3 to d4/da/fc 0 2024-03-30T04:28:10.883 INFO:tasks.workunit.client.1.smithi135.stdout:0/112: chown d0/f3 1 1 2024-03-30T04:28:10.883 INFO:tasks.workunit.client.1.smithi135.stdout:9/40: symlink d7/d8/l9 0 2024-03-30T04:28:10.886 INFO:tasks.workunit.client.0.smithi123.stdout:7/460: chown d2/dc/d1c/f23 2 1 2024-03-30T04:28:10.887 INFO:tasks.workunit.client.1.smithi135.stdout:1/61: truncate f7 6483 0 2024-03-30T04:28:10.888 INFO:tasks.workunit.client.1.smithi135.stdout:0/113: fdatasync d0/f21 0 2024-03-30T04:28:10.892 INFO:tasks.workunit.client.1.smithi135.stdout:0/114: write d0/f21 [536620,94028] 0 2024-03-30T04:28:10.893 INFO:tasks.workunit.client.1.smithi135.stdout:8/92: dread d6/f11 [4194304,4194304] 0 2024-03-30T04:28:10.896 INFO:tasks.workunit.client.1.smithi135.stdout:7/143: symlink d4/db/d1a/d2a/l36 0 2024-03-30T04:28:10.896 INFO:tasks.workunit.client.0.smithi123.stdout:5/465: dwrite dd/d19/f2b [4194304,4194304] 0 2024-03-30T04:28:10.898 INFO:tasks.workunit.client.1.smithi135.stdout:1/62: rename c1 to c12 0 2024-03-30T04:28:10.899 INFO:tasks.workunit.client.0.smithi123.stdout:0/527: dread d6/dd/d31/d35/d3a/f8f [0,4194304] 0 2024-03-30T04:28:10.905 INFO:tasks.workunit.client.1.smithi135.stdout:6/34: sync 2024-03-30T04:28:10.906 INFO:tasks.workunit.client.0.smithi123.stdout:1/598: symlink db/dd/d42/d46/d56/daf/lca 0 2024-03-30T04:28:10.906 INFO:tasks.workunit.client.0.smithi123.stdout:4/463: link d2/d4/d7/d63/l7a d2/d4/d2c/d42/d35/d9b/da2/la4 0 2024-03-30T04:28:10.909 INFO:tasks.workunit.client.0.smithi123.stdout:2/562: unlink d0/d37/l9a 0 2024-03-30T04:28:10.912 INFO:tasks.workunit.client.0.smithi123.stdout:5/466: unlink dd/l29 0 2024-03-30T04:28:10.916 INFO:tasks.workunit.client.0.smithi123.stdout:7/461: dwrite d2/dc/d1c/f7c [0,4194304] 0 2024-03-30T04:28:10.916 INFO:tasks.workunit.client.0.smithi123.stdout:4/464: read d2/f20 [5202793,74702] 0 2024-03-30T04:28:10.919 INFO:tasks.workunit.client.1.smithi135.stdout:9/41: rename d7/d8 to d7/da 0 2024-03-30T04:28:10.919 INFO:tasks.workunit.client.0.smithi123.stdout:5/467: readlink dd/l14 0 2024-03-30T04:28:10.919 INFO:tasks.workunit.client.0.smithi123.stdout:4/465: chown d2/c50 0 1 2024-03-30T04:28:10.922 INFO:tasks.workunit.client.0.smithi123.stdout:0/528: creat d6/dd/d18/fb5 x:0 0 0 2024-03-30T04:28:10.922 INFO:tasks.workunit.client.1.smithi135.stdout:8/93: dread d6/f11 [4194304,4194304] 0 2024-03-30T04:28:10.923 INFO:tasks.workunit.client.0.smithi123.stdout:4/466: chown d2/d4/d2c/d42/c5b 1 1 2024-03-30T04:28:10.925 INFO:tasks.workunit.client.1.smithi135.stdout:2/57: dwrite d4/f5 [4194304,4194304] 0 2024-03-30T04:28:10.925 INFO:tasks.workunit.client.0.smithi123.stdout:0/529: chown d6/dd/d15/d28/c37 590551385 1 2024-03-30T04:28:10.925 INFO:tasks.workunit.client.1.smithi135.stdout:0/115: dread d0/f3 [0,4194304] 0 2024-03-30T04:28:10.928 INFO:tasks.workunit.client.1.smithi135.stdout:8/94: read f4 [5561439,76767] 0 2024-03-30T04:28:10.928 INFO:tasks.workunit.client.0.smithi123.stdout:1/599: creat db/dd/d5f/d6d/fcb x:0 0 0 2024-03-30T04:28:10.929 INFO:tasks.workunit.client.0.smithi123.stdout:2/563: rmdir d0/d1f/d16/d81 0 2024-03-30T04:28:10.931 INFO:tasks.workunit.client.1.smithi135.stdout:1/63: getdents . 0 2024-03-30T04:28:10.931 INFO:tasks.workunit.client.0.smithi123.stdout:4/467: truncate d2/d4/d7/f6c 4443626 0 2024-03-30T04:28:10.933 INFO:tasks.workunit.client.0.smithi123.stdout:7/462: fdatasync d2/da/d22/d3b/f4e 0 2024-03-30T04:28:10.933 INFO:tasks.workunit.client.1.smithi135.stdout:9/42: symlink d7/lb 0 2024-03-30T04:28:10.934 INFO:tasks.workunit.client.1.smithi135.stdout:3/56: sync 2024-03-30T04:28:10.934 INFO:tasks.workunit.client.1.smithi135.stdout:4/42: sync 2024-03-30T04:28:10.934 INFO:tasks.workunit.client.1.smithi135.stdout:0/116: chown d0/d13 107756 1 2024-03-30T04:28:10.934 INFO:tasks.workunit.client.1.smithi135.stdout:5/99: sync 2024-03-30T04:28:10.936 INFO:tasks.workunit.client.0.smithi123.stdout:2/564: write d0/d1f/d16/f18 [3250824,12334] 0 2024-03-30T04:28:10.937 INFO:tasks.workunit.client.0.smithi123.stdout:1/600: write db/dd/d42/d46/d5d/d62/fac [1259356,79598] 0 2024-03-30T04:28:10.939 INFO:tasks.workunit.client.0.smithi123.stdout:7/463: chown d2/dc/d49/f6c 840049940 1 2024-03-30T04:28:10.940 INFO:tasks.workunit.client.1.smithi135.stdout:3/57: readlink d3/l8 0 2024-03-30T04:28:10.940 INFO:tasks.workunit.client.1.smithi135.stdout:4/43: truncate fa 135084 0 2024-03-30T04:28:10.942 INFO:tasks.workunit.client.1.smithi135.stdout:1/64: mknod c13 0 2024-03-30T04:28:10.942 INFO:tasks.workunit.client.0.smithi123.stdout:7/464: stat d2/d4d/f6a 0 2024-03-30T04:28:10.945 INFO:tasks.workunit.client.1.smithi135.stdout:7/144: dwrite d4/db/f1f [0,4194304] 0 2024-03-30T04:28:10.947 INFO:tasks.workunit.client.1.smithi135.stdout:2/58: dread d4/da/fc [0,4194304] 0 2024-03-30T04:28:10.950 INFO:tasks.workunit.client.0.smithi123.stdout:5/468: dwrite dd/d19/d1f/d35/f4b [0,4194304] 0 2024-03-30T04:28:10.950 INFO:tasks.workunit.client.0.smithi123.stdout:7/465: creat d2/dc/d49/d68/f9b x:0 0 0 2024-03-30T04:28:10.953 INFO:tasks.workunit.client.1.smithi135.stdout:6/35: dwrite d2/f4 [0,4194304] 0 2024-03-30T04:28:10.953 INFO:tasks.workunit.client.1.smithi135.stdout:9/43: write f1 [3706,24666] 0 2024-03-30T04:28:10.953 INFO:tasks.workunit.client.1.smithi135.stdout:0/117: rename d0/d5 to d0/d5/d10/d24 22 2024-03-30T04:28:10.953 INFO:tasks.workunit.client.1.smithi135.stdout:1/65: readlink l10 0 2024-03-30T04:28:10.955 INFO:tasks.workunit.client.1.smithi135.stdout:0/118: fsync d0/f21 0 2024-03-30T04:28:10.956 INFO:tasks.workunit.client.0.smithi123.stdout:7/466: read - d2/da/d22/d3b/f8f zero size 2024-03-30T04:28:10.956 INFO:tasks.workunit.client.1.smithi135.stdout:6/36: readlink d2/l6 0 2024-03-30T04:28:10.956 INFO:tasks.workunit.client.1.smithi135.stdout:4/44: chown cc 0 1 2024-03-30T04:28:10.956 INFO:tasks.workunit.client.0.smithi123.stdout:2/565: fsync d0/d1f/d1c/d39/d4b/d4c/d5f/f67 0 2024-03-30T04:28:10.958 INFO:tasks.workunit.client.1.smithi135.stdout:7/145: fdatasync d4/db/d1a/d2a/d2c/f2d 0 2024-03-30T04:28:10.959 INFO:tasks.workunit.client.1.smithi135.stdout:9/44: write f6 [640156,104561] 0 2024-03-30T04:28:10.959 INFO:tasks.workunit.client.1.smithi135.stdout:0/119: readlink d0/d5/d9/l22 0 2024-03-30T04:28:10.960 INFO:tasks.workunit.client.1.smithi135.stdout:4/45: dread - ff zero size 2024-03-30T04:28:10.960 INFO:tasks.workunit.client.0.smithi123.stdout:1/601: creat db/fcc x:0 0 0 2024-03-30T04:28:10.960 INFO:tasks.workunit.client.0.smithi123.stdout:4/468: dwrite d2/d4/d7/d1a/f30 [8388608,4194304] 0 2024-03-30T04:28:10.961 INFO:tasks.workunit.client.1.smithi135.stdout:5/100: getdents d1/da 0 2024-03-30T04:28:10.965 INFO:tasks.workunit.client.0.smithi123.stdout:7/467: mknod d2/da/d11/d34/d76/c9c 0 2024-03-30T04:28:10.969 INFO:tasks.workunit.client.1.smithi135.stdout:5/101: truncate d1/d3/f12 1258700 0 2024-03-30T04:28:10.969 INFO:tasks.workunit.client.1.smithi135.stdout:1/66: creat f14 x:0 0 0 2024-03-30T04:28:10.969 INFO:tasks.workunit.client.1.smithi135.stdout:1/67: rmdir - no directory 2024-03-30T04:28:10.970 INFO:tasks.workunit.client.0.smithi123.stdout:0/530: write d6/dd/d18/d2f/fb1 [331123,73144] 0 2024-03-30T04:28:10.971 INFO:tasks.workunit.client.0.smithi123.stdout:2/566: mknod d0/d1f/d5e/d71/cbd 0 2024-03-30T04:28:10.971 INFO:tasks.workunit.client.0.smithi123.stdout:5/469: link dd/d19/l49 dd/d19/d1f/d3c/l95 0 2024-03-30T04:28:10.972 INFO:tasks.workunit.client.0.smithi123.stdout:7/468: creat d2/dc/d49/d68/f9d x:0 0 0 2024-03-30T04:28:10.974 INFO:tasks.workunit.client.1.smithi135.stdout:1/68: truncate fd 1025128 0 2024-03-30T04:28:10.974 INFO:tasks.workunit.client.1.smithi135.stdout:3/58: dread f0 [0,4194304] 0 2024-03-30T04:28:10.974 INFO:tasks.workunit.client.1.smithi135.stdout:6/37: mknod d2/c7 0 2024-03-30T04:28:10.975 INFO:tasks.workunit.client.0.smithi123.stdout:2/567: dread - d0/d1f/d1c/d40/fac zero size 2024-03-30T04:28:10.975 INFO:tasks.workunit.client.0.smithi123.stdout:4/469: truncate d2/d4/d2c/d42/f38 4983234 0 2024-03-30T04:28:10.975 INFO:tasks.workunit.client.0.smithi123.stdout:5/470: dread - dd/f7d zero size 2024-03-30T04:28:10.975 INFO:tasks.workunit.client.1.smithi135.stdout:2/59: mkdir d4/da/dd 0 2024-03-30T04:28:10.975 INFO:tasks.workunit.client.0.smithi123.stdout:0/531: chown d6/dd/d18/fb5 3 1 2024-03-30T04:28:10.979 INFO:tasks.workunit.client.0.smithi123.stdout:2/568: fsync d0/d1f/d1c/d80/f49 0 2024-03-30T04:28:10.979 INFO:tasks.workunit.client.0.smithi123.stdout:7/469: mkdir d2/da/d11/d61/d55/d9e 0 2024-03-30T04:28:10.980 INFO:tasks.workunit.client.0.smithi123.stdout:9/461: sync 2024-03-30T04:28:10.980 INFO:tasks.workunit.client.0.smithi123.stdout:7/470: fsync d2/f87 0 2024-03-30T04:28:10.982 INFO:tasks.workunit.client.1.smithi135.stdout:8/95: creat d6/d8/f16 x:0 0 0 2024-03-30T04:28:10.982 INFO:tasks.workunit.client.1.smithi135.stdout:7/146: creat d4/db/d1a/d2a/f37 x:0 0 0 2024-03-30T04:28:10.983 INFO:tasks.workunit.client.0.smithi123.stdout:0/532: truncate d6/dd/d18/d2f/f58 2524827 0 2024-03-30T04:28:10.984 INFO:tasks.workunit.client.0.smithi123.stdout:2/569: unlink d0/le 0 2024-03-30T04:28:10.985 INFO:tasks.workunit.client.1.smithi135.stdout:5/102: write d1/d7/f10 [128537,34161] 0 2024-03-30T04:28:10.985 INFO:tasks.workunit.client.1.smithi135.stdout:8/96: readlink d6/d8/l10 0 2024-03-30T04:28:10.985 INFO:tasks.workunit.client.1.smithi135.stdout:9/45: dread f6 [0,4194304] 0 2024-03-30T04:28:10.986 INFO:tasks.workunit.client.0.smithi123.stdout:4/470: mkdir d2/d4/d2c/d42/d80/da5 0 2024-03-30T04:28:10.987 INFO:tasks.workunit.client.1.smithi135.stdout:1/69: dread f7 [0,4194304] 0 2024-03-30T04:28:10.988 INFO:tasks.workunit.client.0.smithi123.stdout:0/533: dread - d6/dd/d18/d2f/f85 zero size 2024-03-30T04:28:10.988 INFO:tasks.workunit.client.1.smithi135.stdout:2/60: rename d4/d6/l8 to d4/da/dd/le 0 2024-03-30T04:28:10.990 INFO:tasks.workunit.client.1.smithi135.stdout:3/59: getdents d3/d10 0 2024-03-30T04:28:10.991 INFO:tasks.workunit.client.1.smithi135.stdout:2/61: chown d4 0 1 2024-03-30T04:28:10.991 INFO:tasks.workunit.client.1.smithi135.stdout:7/147: fsync d4/db/d1a/d2a/f32 0 2024-03-30T04:28:10.992 INFO:tasks.workunit.client.0.smithi123.stdout:9/462: rename d6/d23/c42 to d6/d1f/d45/ca5 0 2024-03-30T04:28:10.993 INFO:tasks.workunit.client.1.smithi135.stdout:1/70: mknod c15 0 2024-03-30T04:28:10.996 INFO:tasks.workunit.client.0.smithi123.stdout:4/471: unlink d2/d4/d7/d1a/f1f 0 2024-03-30T04:28:10.997 INFO:tasks.workunit.client.0.smithi123.stdout:0/534: rmdir d6/dd/d18 39 2024-03-30T04:28:11.000 INFO:tasks.workunit.client.0.smithi123.stdout:7/471: link d2/d4d/f50 d2/da/d11/d34/d76/d91/f9f 0 2024-03-30T04:28:11.001 INFO:tasks.workunit.client.0.smithi123.stdout:7/472: readlink d2/da/l30 0 2024-03-30T04:28:11.003 INFO:tasks.workunit.client.0.smithi123.stdout:5/471: dwrite dd/d19/f59 [4194304,4194304] 0 2024-03-30T04:28:11.004 INFO:tasks.workunit.client.1.smithi135.stdout:4/46: dwrite fb [0,4194304] 0 2024-03-30T04:28:11.004 INFO:tasks.workunit.client.1.smithi135.stdout:7/148: mkdir d4/db/de/d15/d38 0 2024-03-30T04:28:11.005 INFO:tasks.workunit.client.1.smithi135.stdout:8/97: creat d6/d8/f17 x:0 0 0 2024-03-30T04:28:11.007 INFO:tasks.workunit.client.0.smithi123.stdout:1/602: truncate db/dd/d42/d46/d56/d5c/f64 4412692 0 2024-03-30T04:28:11.008 INFO:tasks.workunit.client.1.smithi135.stdout:7/149: fdatasync f3 0 2024-03-30T04:28:11.010 INFO:tasks.workunit.client.0.smithi123.stdout:1/603: chown db/dd/d42/d46/d56/da2 8 1 2024-03-30T04:28:11.013 INFO:tasks.workunit.client.0.smithi123.stdout:7/473: fsync d2/da/f4c 0 2024-03-30T04:28:11.013 INFO:tasks.workunit.client.0.smithi123.stdout:0/535: chown d6/dd/d18/c94 5747 1 2024-03-30T04:28:11.013 INFO:tasks.workunit.client.1.smithi135.stdout:9/46: dread f6 [0,4194304] 0 2024-03-30T04:28:11.013 INFO:tasks.workunit.client.1.smithi135.stdout:4/47: write f3 [1650348,33252] 0 2024-03-30T04:28:11.015 INFO:tasks.workunit.client.0.smithi123.stdout:2/570: rename d0/d1f/d16/c60 to d0/d1f/d1c/d80/dbb/cbe 0 2024-03-30T04:28:11.017 INFO:tasks.workunit.client.1.smithi135.stdout:4/48: fsync ff 0 2024-03-30T04:28:11.021 INFO:tasks.workunit.client.1.smithi135.stdout:8/98: stat d6/d8/f17 0 2024-03-30T04:28:11.021 INFO:tasks.workunit.client.0.smithi123.stdout:7/474: write d2/da/d2d/f39 [2670651,37529] 0 2024-03-30T04:28:11.022 INFO:tasks.workunit.client.1.smithi135.stdout:2/62: dread d4/da/fb [0,4194304] 0 2024-03-30T04:28:11.025 INFO:tasks.workunit.client.0.smithi123.stdout:9/463: write d6/d1f/d72/f8f [1048229,101161] 0 2024-03-30T04:28:11.026 INFO:tasks.workunit.client.1.smithi135.stdout:7/150: link d4/db/d1a/d2a/f32 d4/db/d1a/d2a/d2c/f39 0 2024-03-30T04:28:11.028 INFO:tasks.workunit.client.1.smithi135.stdout:4/49: creat d10/f12 x:0 0 0 2024-03-30T04:28:11.031 INFO:tasks.workunit.client.1.smithi135.stdout:8/99: rmdir d6/d8 39 2024-03-30T04:28:11.034 INFO:tasks.workunit.client.0.smithi123.stdout:5/472: truncate dd/f1d 1602401 0 2024-03-30T04:28:11.036 INFO:tasks.workunit.client.1.smithi135.stdout:4/50: write ff [391601,111368] 0 2024-03-30T04:28:11.037 INFO:tasks.workunit.client.0.smithi123.stdout:4/472: dwrite d2/f1c [0,4194304] 0 2024-03-30T04:28:11.039 INFO:tasks.workunit.client.0.smithi123.stdout:6/587: sync 2024-03-30T04:28:11.040 INFO:tasks.workunit.client.0.smithi123.stdout:3/544: sync 2024-03-30T04:28:11.040 INFO:tasks.workunit.client.0.smithi123.stdout:8/433: sync 2024-03-30T04:28:11.041 INFO:tasks.workunit.client.0.smithi123.stdout:0/536: rename d6/dd/d15/d98/d4e/d69/f7a to d6/dd/d15/d28/fb6 0 2024-03-30T04:28:11.045 INFO:tasks.workunit.client.1.smithi135.stdout:5/103: dwrite d1/d7/ff [0,4194304] 0 2024-03-30T04:28:11.045 INFO:tasks.workunit.client.1.smithi135.stdout:7/151: write d4/db/d1a/d2a/d2c/f39 [486808,335] 0 2024-03-30T04:28:11.046 INFO:tasks.workunit.client.0.smithi123.stdout:2/571: link d0/d1f/d30/c31 d0/d37/d57/cbf 0 2024-03-30T04:28:11.047 INFO:tasks.workunit.client.0.smithi123.stdout:1/604: getdents db/dd/d42/d46/d56/da2 0 2024-03-30T04:28:11.051 INFO:tasks.workunit.client.0.smithi123.stdout:0/537: write d6/dd/d15/d28/d4b/f82 [180115,57720] 0 2024-03-30T04:28:11.053 INFO:tasks.workunit.client.1.smithi135.stdout:3/60: dwrite d3/d6/fa [0,4194304] 0 2024-03-30T04:28:11.054 INFO:tasks.workunit.client.1.smithi135.stdout:7/152: write d4/db/d1a/d2a/f32 [666450,44566] 0 2024-03-30T04:28:11.056 INFO:tasks.workunit.client.1.smithi135.stdout:9/47: dread f1 [0,4194304] 0 2024-03-30T04:28:11.060 INFO:tasks.workunit.client.1.smithi135.stdout:4/51: link c9 d10/c13 0 2024-03-30T04:28:11.061 INFO:tasks.workunit.client.0.smithi123.stdout:8/434: fsync da/dc/d14/d18/d21/f40 0 2024-03-30T04:28:11.062 INFO:tasks.workunit.client.1.smithi135.stdout:3/61: chown d3/d6/fa 263366 1 2024-03-30T04:28:11.065 INFO:tasks.workunit.client.0.smithi123.stdout:7/475: rename d2/d4d/c5f to d2/dc/d49/d68/ca0 0 2024-03-30T04:28:11.065 INFO:tasks.workunit.client.0.smithi123.stdout:8/435: chown da/f46 7729840 1 2024-03-30T04:28:11.065 INFO:tasks.workunit.client.0.smithi123.stdout:4/473: stat d2/f86 0 2024-03-30T04:28:11.065 INFO:tasks.workunit.client.1.smithi135.stdout:9/48: creat d7/da/fc x:0 0 0 2024-03-30T04:28:11.066 INFO:tasks.workunit.client.1.smithi135.stdout:8/100: dread f4 [0,4194304] 0 2024-03-30T04:28:11.069 INFO:tasks.workunit.client.0.smithi123.stdout:6/588: dwrite d10/f97 [0,4194304] 0 2024-03-30T04:28:11.071 INFO:tasks.workunit.client.1.smithi135.stdout:3/62: dread d3/d6/f7 [0,4194304] 0 2024-03-30T04:28:11.072 INFO:tasks.workunit.client.1.smithi135.stdout:4/52: rmdir d10 39 2024-03-30T04:28:11.073 INFO:tasks.workunit.client.1.smithi135.stdout:5/104: dread d1/d3/f15 [0,4194304] 0 2024-03-30T04:28:11.074 INFO:tasks.workunit.client.1.smithi135.stdout:3/63: write d3/f4 [1463829,69502] 0 2024-03-30T04:28:11.075 INFO:tasks.workunit.client.0.smithi123.stdout:5/473: unlink dd/d19/d1f/d35/f4b 0 2024-03-30T04:28:11.075 INFO:tasks.workunit.client.0.smithi123.stdout:3/545: mkdir d13/d24/d31/d23/d3a/d3c/dab 0 2024-03-30T04:28:11.075 INFO:tasks.workunit.client.0.smithi123.stdout:2/572: symlink d0/d1f/d1c/d80/dbb/lc0 0 2024-03-30T04:28:11.075 INFO:tasks.workunit.client.1.smithi135.stdout:4/53: fdatasync ff 0 2024-03-30T04:28:11.076 INFO:tasks.workunit.client.1.smithi135.stdout:8/101: creat d6/f18 x:0 0 0 2024-03-30T04:28:11.078 INFO:tasks.workunit.client.0.smithi123.stdout:3/546: readlink d13/d24/d31/d23/l26 0 2024-03-30T04:28:11.080 INFO:tasks.workunit.client.1.smithi135.stdout:3/64: chown l1 1 1 2024-03-30T04:28:11.081 INFO:tasks.workunit.client.0.smithi123.stdout:9/464: dread d6/d1f/d72/f93 [4194304,4194304] 0 2024-03-30T04:28:11.083 INFO:tasks.workunit.client.1.smithi135.stdout:5/105: mkdir d1/d7/d1a/d1f 0 2024-03-30T04:28:11.083 INFO:tasks.workunit.client.0.smithi123.stdout:0/538: mkdir d6/dd/d18/d2f/da3/db7 0 2024-03-30T04:28:11.084 INFO:tasks.workunit.client.0.smithi123.stdout:1/605: creat db/fcd x:0 0 0 2024-03-30T04:28:11.086 INFO:tasks.workunit.client.1.smithi135.stdout:3/65: write d3/db/fe [482320,103786] 0 2024-03-30T04:28:11.088 INFO:tasks.workunit.client.0.smithi123.stdout:8/436: rename da/fe to da/d27/d3c/d64/f81 0 2024-03-30T04:28:11.089 INFO:tasks.workunit.client.0.smithi123.stdout:9/465: symlink d6/d1f/d4d/la6 0 2024-03-30T04:28:11.090 INFO:tasks.workunit.client.1.smithi135.stdout:8/102: read d6/d8/d14/fe [494813,34702] 0 2024-03-30T04:28:11.092 INFO:tasks.workunit.client.1.smithi135.stdout:5/106: mknod d1/c20 0 2024-03-30T04:28:11.093 INFO:tasks.workunit.client.0.smithi123.stdout:4/474: creat d2/d4/d2c/d42/d80/d87/d91/d9d/fa6 x:0 0 0 2024-03-30T04:28:11.094 INFO:tasks.workunit.client.1.smithi135.stdout:7/153: dwrite d4/db/f14 [0,4194304] 0 2024-03-30T04:28:11.094 INFO:tasks.workunit.client.0.smithi123.stdout:8/437: dread - da/dc/d14/d18/d21/f4d zero size 2024-03-30T04:28:11.094 INFO:tasks.workunit.client.1.smithi135.stdout:9/49: dread f1 [0,4194304] 0 2024-03-30T04:28:11.096 INFO:tasks.workunit.client.0.smithi123.stdout:2/573: rename d0/d1f/d1c/d20/fb8 to d0/d1f/d1c/d80/dbb/d97/fc1 0 2024-03-30T04:28:11.098 INFO:tasks.workunit.client.0.smithi123.stdout:8/438: fdatasync da/dc/d14/d18/d21/f28 0 2024-03-30T04:28:11.098 INFO:tasks.workunit.client.1.smithi135.stdout:3/66: truncate d3/d6/f12 137838 0 2024-03-30T04:28:11.098 INFO:tasks.workunit.client.1.smithi135.stdout:8/103: fdatasync d6/f11 0 2024-03-30T04:28:11.100 INFO:tasks.workunit.client.0.smithi123.stdout:8/439: readlink da/d27/d3c/d2d/l5a 0 2024-03-30T04:28:11.100 INFO:tasks.workunit.client.0.smithi123.stdout:0/539: unlink d6/dd/d16/c50 0 2024-03-30T04:28:11.104 INFO:tasks.workunit.client.0.smithi123.stdout:9/466: chown d6/d1f/d4d/f5b 25808763 1 2024-03-30T04:28:11.105 INFO:tasks.workunit.client.1.smithi135.stdout:9/50: write d7/da/fc [910691,83762] 0 2024-03-30T04:28:11.106 INFO:tasks.workunit.client.0.smithi123.stdout:1/606: mkdir db/dd/d42/d46/d56/dce 0 2024-03-30T04:28:11.106 INFO:tasks.workunit.client.1.smithi135.stdout:5/107: symlink d1/da/l21 0 2024-03-30T04:28:11.110 INFO:tasks.workunit.client.1.smithi135.stdout:3/67: rename f0 to d3/d10/f13 0 2024-03-30T04:28:11.110 INFO:tasks.workunit.client.1.smithi135.stdout:7/154: mkdir d4/db/de/d15/d38/d3a 0 2024-03-30T04:28:11.112 INFO:tasks.workunit.client.0.smithi123.stdout:6/589: getdents d10/d20 0 2024-03-30T04:28:11.114 INFO:tasks.workunit.client.0.smithi123.stdout:3/547: rename d13/d24/d31/d23/d3a/d3c/d4e/f81 to d13/d24/d39/d79/d90/fac 0 2024-03-30T04:28:11.114 INFO:tasks.workunit.client.1.smithi135.stdout:4/54: dwrite fa [0,4194304] 0 2024-03-30T04:28:11.115 INFO:tasks.workunit.client.1.smithi135.stdout:9/51: symlink d7/da/ld 0 2024-03-30T04:28:11.115 INFO:tasks.workunit.client.1.smithi135.stdout:7/155: read - d4/db/de/f28 zero size 2024-03-30T04:28:11.116 INFO:tasks.workunit.client.0.smithi123.stdout:8/440: symlink da/d27/l82 0 2024-03-30T04:28:11.116 INFO:tasks.workunit.client.1.smithi135.stdout:8/104: creat d6/d8/f19 x:0 0 0 2024-03-30T04:28:11.117 INFO:tasks.workunit.client.1.smithi135.stdout:8/105: stat f5 0 2024-03-30T04:28:11.118 INFO:tasks.workunit.client.1.smithi135.stdout:8/106: chown d6/d8/d14 0 1 2024-03-30T04:28:11.119 INFO:tasks.workunit.client.1.smithi135.stdout:5/108: rename d1/c20 to d1/da/c22 0 2024-03-30T04:28:11.122 INFO:tasks.workunit.client.0.smithi123.stdout:2/574: dread d0/d1f/d1c/d39/d4b/d4c/d5f/f9e [0,4194304] 0 2024-03-30T04:28:11.125 INFO:tasks.workunit.client.1.smithi135.stdout:4/55: chown d10/c11 257 1 2024-03-30T04:28:11.125 INFO:tasks.workunit.client.1.smithi135.stdout:8/107: symlink d6/d8/d15/l1a 0 2024-03-30T04:28:11.130 INFO:tasks.workunit.client.0.smithi123.stdout:3/548: read - d13/d24/d31/d23/d63/fa7 zero size 2024-03-30T04:28:11.132 INFO:tasks.workunit.client.1.smithi135.stdout:3/68: dread d3/d6/fa [0,4194304] 0 2024-03-30T04:28:11.135 INFO:tasks.workunit.client.1.smithi135.stdout:9/52: truncate f6 55832 0 2024-03-30T04:28:11.136 INFO:tasks.workunit.client.0.smithi123.stdout:7/476: dwrite d2/da/d22/d3b/f4e [0,4194304] 0 2024-03-30T04:28:11.137 INFO:tasks.workunit.client.1.smithi135.stdout:3/69: link d3/f4 d3/d10/f14 0 2024-03-30T04:28:11.138 INFO:tasks.workunit.client.1.smithi135.stdout:3/70: dread - d3/f11 zero size 2024-03-30T04:28:11.139 INFO:tasks.workunit.client.1.smithi135.stdout:9/53: creat d7/fe x:0 0 0 2024-03-30T04:28:11.140 INFO:tasks.workunit.client.1.smithi135.stdout:3/71: write d3/d6/f7 [443718,34914] 0 2024-03-30T04:28:11.141 INFO:tasks.workunit.client.1.smithi135.stdout:9/54: creat d7/ff x:0 0 0 2024-03-30T04:28:11.141 INFO:tasks.workunit.client.0.smithi123.stdout:0/540: dwrite d6/dd/d15/d98/d4e/d69/d6e/f9f [0,4194304] 0 2024-03-30T04:28:11.143 INFO:tasks.workunit.client.1.smithi135.stdout:5/109: truncate d1/d3/f12 882062 0 2024-03-30T04:28:11.143 INFO:tasks.workunit.client.0.smithi123.stdout:2/575: mknod d0/d1f/d1c/d2a/d99/cc2 0 2024-03-30T04:28:11.144 INFO:tasks.workunit.client.1.smithi135.stdout:8/108: truncate d6/fa 737107 0 2024-03-30T04:28:11.144 INFO:tasks.workunit.client.0.smithi123.stdout:1/607: truncate db/dd/fb7 2116802 0 2024-03-30T04:28:11.145 INFO:tasks.workunit.client.0.smithi123.stdout:4/475: write d2/d4/d2c/f67 [4513174,103409] 0 2024-03-30T04:28:11.148 INFO:tasks.workunit.client.0.smithi123.stdout:7/477: write d2/dc/d7f/f8a [4781,64771] 0 2024-03-30T04:28:11.148 INFO:tasks.workunit.client.1.smithi135.stdout:9/55: read d7/da/fc [183428,80628] 0 2024-03-30T04:28:11.149 INFO:tasks.workunit.client.1.smithi135.stdout:8/109: symlink d6/d8/d14/l1b 0 2024-03-30T04:28:11.150 INFO:tasks.workunit.client.0.smithi123.stdout:0/541: readlink d6/l84 0 2024-03-30T04:28:11.154 INFO:tasks.workunit.client.0.smithi123.stdout:0/542: chown d6/fa 2 1 2024-03-30T04:28:11.155 INFO:tasks.workunit.client.0.smithi123.stdout:1/608: dread - db/dd/d42/d46/d56/da2/fb0 zero size 2024-03-30T04:28:11.155 INFO:tasks.workunit.client.1.smithi135.stdout:8/110: creat d6/d8/f1c x:0 0 0 2024-03-30T04:28:11.158 INFO:tasks.workunit.client.0.smithi123.stdout:2/576: dread - d0/d1f/d1c/d39/d4b/fa7 zero size 2024-03-30T04:28:11.160 INFO:tasks.workunit.client.1.smithi135.stdout:5/110: dread d1/d7/f10 [0,4194304] 0 2024-03-30T04:28:11.164 INFO:tasks.workunit.client.0.smithi123.stdout:6/590: dwrite d10/d20/d3f/f91 [0,4194304] 0 2024-03-30T04:28:11.165 INFO:tasks.workunit.client.1.smithi135.stdout:9/56: dread f1 [0,4194304] 0 2024-03-30T04:28:11.166 INFO:tasks.workunit.client.1.smithi135.stdout:5/111: dread - d1/d7/f13 zero size 2024-03-30T04:28:11.166 INFO:tasks.workunit.client.1.smithi135.stdout:0/120: sync 2024-03-30T04:28:11.169 INFO:tasks.workunit.client.0.smithi123.stdout:3/549: creat d13/d24/d31/d23/d3a/d3c/d4e/d5d/da5/fad x:0 0 0 2024-03-30T04:28:11.169 INFO:tasks.workunit.client.1.smithi135.stdout:3/72: truncate d3/d6/fa 936999 0 2024-03-30T04:28:11.171 INFO:tasks.workunit.client.1.smithi135.stdout:9/57: rmdir d7/da 39 2024-03-30T04:28:11.174 INFO:tasks.workunit.client.1.smithi135.stdout:0/121: mkdir d0/d5/d9/d25 0 2024-03-30T04:28:11.176 INFO:tasks.workunit.client.0.smithi123.stdout:0/543: mkdir d6/dd/d31/db8 0 2024-03-30T04:28:11.176 INFO:tasks.workunit.client.0.smithi123.stdout:2/577: dread d0/d1f/d1c/f41 [0,4194304] 0 2024-03-30T04:28:11.176 INFO:tasks.workunit.client.0.smithi123.stdout:7/478: rename d2/da/d2d/l65 to d2/da/d11/d61/d55/d9e/la1 0 2024-03-30T04:28:11.177 INFO:tasks.workunit.client.1.smithi135.stdout:5/112: mknod d1/da/c23 0 2024-03-30T04:28:11.178 INFO:tasks.workunit.client.0.smithi123.stdout:3/550: fdatasync d13/d24/f70 0 2024-03-30T04:28:11.182 INFO:tasks.workunit.client.1.smithi135.stdout:0/122: symlink d0/d5/l26 0 2024-03-30T04:28:11.183 INFO:tasks.workunit.client.0.smithi123.stdout:8/441: dread da/fb [0,4194304] 0 2024-03-30T04:28:11.184 INFO:tasks.workunit.client.0.smithi123.stdout:4/476: getdents d2/d4/d7/d1a 0 2024-03-30T04:28:11.184 INFO:tasks.workunit.client.0.smithi123.stdout:1/609: creat db/dd/d5f/d6d/fcf x:0 0 0 2024-03-30T04:28:11.185 INFO:tasks.workunit.client.1.smithi135.stdout:8/111: dwrite d6/f11 [4194304,4194304] 0 2024-03-30T04:28:11.185 INFO:tasks.workunit.client.0.smithi123.stdout:6/591: rename d10/d82/d8e/dc7/d5b/l93 to d10/d20/d3f/da3/lcb 0 2024-03-30T04:28:11.186 INFO:tasks.workunit.client.0.smithi123.stdout:8/442: dread - da/dc/d14/f6e zero size 2024-03-30T04:28:11.186 INFO:tasks.workunit.client.1.smithi135.stdout:9/58: symlink d7/l10 0 2024-03-30T04:28:11.187 INFO:tasks.workunit.client.1.smithi135.stdout:5/113: symlink d1/da/d17/l24 0 2024-03-30T04:28:11.187 INFO:tasks.workunit.client.0.smithi123.stdout:1/610: write db/f24 [8617063,31711] 0 2024-03-30T04:28:11.188 INFO:tasks.workunit.client.1.smithi135.stdout:0/123: symlink d0/d13/l27 0 2024-03-30T04:28:11.191 INFO:tasks.workunit.client.0.smithi123.stdout:7/479: rename d2/dc/d7f/f8a to d2/dc/d7f/fa2 0 2024-03-30T04:28:11.193 INFO:tasks.workunit.client.1.smithi135.stdout:8/112: fdatasync f5 0 2024-03-30T04:28:11.195 INFO:tasks.workunit.client.1.smithi135.stdout:3/73: dread d3/f9 [0,4194304] 0 2024-03-30T04:28:11.196 INFO:tasks.workunit.client.1.smithi135.stdout:3/74: chown f2 18 1 2024-03-30T04:28:11.196 INFO:tasks.workunit.client.1.smithi135.stdout:5/114: stat d1/cb 0 2024-03-30T04:28:11.197 INFO:tasks.workunit.client.0.smithi123.stdout:4/477: truncate d2/d4/d2c/d42/d35/d5d/d6f/f73 1596140 0 2024-03-30T04:28:11.197 INFO:tasks.workunit.client.1.smithi135.stdout:5/115: stat d1/d3/f15 0 2024-03-30T04:28:11.198 INFO:tasks.workunit.client.0.smithi123.stdout:2/578: dread d0/d1f/d30/f72 [0,4194304] 0 2024-03-30T04:28:11.201 INFO:tasks.workunit.client.0.smithi123.stdout:1/611: mkdir db/dd/d42/d46/d56/d5c/dd0 0 2024-03-30T04:28:11.204 INFO:tasks.workunit.client.1.smithi135.stdout:8/113: rmdir d6 39 2024-03-30T04:28:11.205 INFO:tasks.workunit.client.0.smithi123.stdout:7/480: creat d2/dc/d7f/fa3 x:0 0 0 2024-03-30T04:28:11.206 INFO:tasks.workunit.client.0.smithi123.stdout:4/478: truncate d2/d4/d2c/d42/d35/d5d/d6f/f73 1850241 0 2024-03-30T04:28:11.206 INFO:tasks.workunit.client.1.smithi135.stdout:3/75: rmdir d3/d10 39 2024-03-30T04:28:11.207 INFO:tasks.workunit.client.0.smithi123.stdout:7/481: chown d2/da/d11 0 1 2024-03-30T04:28:11.207 INFO:tasks.workunit.client.0.smithi123.stdout:0/544: rmdir d6/dd/d18/d2f/da3/db7 0 2024-03-30T04:28:11.210 INFO:tasks.workunit.client.0.smithi123.stdout:3/551: creat d13/d24/d31/fae x:0 0 0 2024-03-30T04:28:11.212 INFO:tasks.workunit.client.0.smithi123.stdout:6/592: link d10/d20/d3f/f4d d10/d82/d8e/dc7/d5b/fcc 0 2024-03-30T04:28:11.213 INFO:tasks.workunit.client.1.smithi135.stdout:5/116: symlink d1/l25 0 2024-03-30T04:28:11.213 INFO:tasks.workunit.client.0.smithi123.stdout:2/579: mknod d0/d1f/d1c/d80/da8/cc3 0 2024-03-30T04:28:11.214 INFO:tasks.workunit.client.1.smithi135.stdout:8/114: dread f0 [8388608,4194304] 0 2024-03-30T04:28:11.214 INFO:tasks.workunit.client.1.smithi135.stdout:9/59: getdents d7 0 2024-03-30T04:28:11.216 INFO:tasks.workunit.client.1.smithi135.stdout:5/117: chown d1/d7/d11 1285279 1 2024-03-30T04:28:11.216 INFO:tasks.workunit.client.1.smithi135.stdout:0/124: getdents d0/d1/d16 0 2024-03-30T04:28:11.218 INFO:tasks.workunit.client.0.smithi123.stdout:0/545: truncate d6/fa 1158230 0 2024-03-30T04:28:11.219 INFO:tasks.workunit.client.1.smithi135.stdout:5/118: unlink d1/c2 0 2024-03-30T04:28:11.220 INFO:tasks.workunit.client.0.smithi123.stdout:0/546: chown d6/f72 1043718846 1 2024-03-30T04:28:11.221 INFO:tasks.workunit.client.0.smithi123.stdout:6/593: symlink d10/d82/dc1/d45/d77/lcd 0 2024-03-30T04:28:11.221 INFO:tasks.workunit.client.0.smithi123.stdout:2/580: chown d0/d1f/d1c/d20/f5d 3416 1 2024-03-30T04:28:11.222 INFO:tasks.workunit.client.1.smithi135.stdout:5/119: chown d1/da 1487342 1 2024-03-30T04:28:11.223 INFO:tasks.workunit.client.0.smithi123.stdout:7/482: link d2/da/c38 d2/d4d/ca4 0 2024-03-30T04:28:11.224 INFO:tasks.workunit.client.1.smithi135.stdout:8/115: link d6/f18 d6/d8/f1d 0 2024-03-30T04:28:11.225 INFO:tasks.workunit.client.0.smithi123.stdout:0/547: fsync d6/dd/d31/d35/d3a/f95 0 2024-03-30T04:28:11.225 INFO:tasks.workunit.client.0.smithi123.stdout:8/443: dread da/dc/f42 [0,4194304] 0 2024-03-30T04:28:11.226 INFO:tasks.workunit.client.1.smithi135.stdout:5/120: read d1/f4 [249105,71489] 0 2024-03-30T04:28:11.226 INFO:tasks.workunit.client.1.smithi135.stdout:9/60: dread d7/da/fc [0,4194304] 0 2024-03-30T04:28:11.227 INFO:tasks.workunit.client.1.smithi135.stdout:8/116: read - d6/d8/f1c zero size 2024-03-30T04:28:11.227 INFO:tasks.workunit.client.0.smithi123.stdout:6/594: creat d10/d20/fce x:0 0 0 2024-03-30T04:28:11.228 INFO:tasks.workunit.client.0.smithi123.stdout:0/548: rmdir d6/dd/d18/d1f 39 2024-03-30T04:28:11.229 INFO:tasks.workunit.client.1.smithi135.stdout:9/61: chown d7/lb 2 1 2024-03-30T04:28:11.230 INFO:tasks.workunit.client.0.smithi123.stdout:6/595: rmdir d10/d82/d8e 39 2024-03-30T04:28:11.231 INFO:tasks.workunit.client.1.smithi135.stdout:5/121: rmdir d1/d7/d1a 39 2024-03-30T04:28:11.232 INFO:tasks.workunit.client.0.smithi123.stdout:0/549: unlink d6/f40 0 2024-03-30T04:28:11.233 INFO:tasks.workunit.client.1.smithi135.stdout:9/62: creat d7/f11 x:0 0 0 2024-03-30T04:28:11.233 INFO:tasks.workunit.client.1.smithi135.stdout:8/117: truncate d6/d8/f1d 373899 0 2024-03-30T04:28:11.233 INFO:tasks.workunit.client.1.smithi135.stdout:3/76: dwrite d3/d6/f12 [0,4194304] 0 2024-03-30T04:28:11.233 INFO:tasks.workunit.client.0.smithi123.stdout:2/581: link d0/d1f/d1c/d39/d4b/d4c/d5f/f67 d0/d37/d9b/fc4 0 2024-03-30T04:28:11.234 INFO:tasks.workunit.client.0.smithi123.stdout:7/483: link d2/da/d11/d61/c7e d2/d4d/d80/d98/ca5 0 2024-03-30T04:28:11.236 INFO:tasks.workunit.client.1.smithi135.stdout:3/77: chown d3/d6/cd 1105 1 2024-03-30T04:28:11.236 INFO:tasks.workunit.client.0.smithi123.stdout:8/444: mkdir da/d27/d3c/d64/d78/d83 0 2024-03-30T04:28:11.241 INFO:tasks.workunit.client.1.smithi135.stdout:0/125: dwrite d0/f3 [0,4194304] 0 2024-03-30T04:28:11.241 INFO:tasks.workunit.client.0.smithi123.stdout:2/582: unlink d0/d1f/d82/f8e 0 2024-03-30T04:28:11.241 INFO:tasks.workunit.client.0.smithi123.stdout:7/484: creat d2/da/d2d/fa6 x:0 0 0 2024-03-30T04:28:11.243 INFO:tasks.workunit.client.0.smithi123.stdout:2/583: dread - d0/d1f/d5e/f6f zero size 2024-03-30T04:28:11.243 INFO:tasks.workunit.client.1.smithi135.stdout:3/78: mkdir d3/d10/d15 0 2024-03-30T04:28:11.244 INFO:tasks.workunit.client.1.smithi135.stdout:3/79: fsync d3/f5 0 2024-03-30T04:28:11.245 INFO:tasks.workunit.client.0.smithi123.stdout:8/445: truncate da/ff 4229076 0 2024-03-30T04:28:11.247 INFO:tasks.workunit.client.0.smithi123.stdout:7/485: dread - d2/da/d2d/f40 zero size 2024-03-30T04:28:11.247 INFO:tasks.workunit.client.1.smithi135.stdout:0/126: truncate d0/d5/d9/fb 122052 0 2024-03-30T04:28:11.248 INFO:tasks.workunit.client.0.smithi123.stdout:2/584: creat d0/d1f/d1c/d80/da8/fc5 x:0 0 0 2024-03-30T04:28:11.249 INFO:tasks.workunit.client.1.smithi135.stdout:5/122: dread d1/d7/ff [0,4194304] 0 2024-03-30T04:28:11.249 INFO:tasks.workunit.client.0.smithi123.stdout:8/446: creat da/d27/d56/d6c/d6d/f84 x:0 0 0 2024-03-30T04:28:11.249 INFO:tasks.workunit.client.0.smithi123.stdout:7/486: dread - d2/da/d2d/f40 zero size 2024-03-30T04:28:11.252 INFO:tasks.workunit.client.0.smithi123.stdout:8/447: fsync da/d27/f3b 0 2024-03-30T04:28:11.253 INFO:tasks.workunit.client.0.smithi123.stdout:0/550: dread d6/dd/d15/d28/d4b/f82 [0,4194304] 0 2024-03-30T04:28:11.254 INFO:tasks.workunit.client.1.smithi135.stdout:3/80: truncate d3/f5 343558 0 2024-03-30T04:28:11.254 INFO:tasks.workunit.client.1.smithi135.stdout:0/127: creat d0/d1/d16/f28 x:0 0 0 2024-03-30T04:28:11.258 INFO:tasks.workunit.client.1.smithi135.stdout:3/81: mkdir d3/d10/d16 0 2024-03-30T04:28:11.259 INFO:tasks.workunit.client.1.smithi135.stdout:0/128: truncate d0/d1/d16/f28 311178 0 2024-03-30T04:28:11.259 INFO:tasks.workunit.client.0.smithi123.stdout:7/487: creat d2/da/d11/d34/d76/fa7 x:0 0 0 2024-03-30T04:28:11.262 INFO:tasks.workunit.client.0.smithi123.stdout:2/585: getdents d0/d1f/d1c/d36 0 2024-03-30T04:28:11.263 INFO:tasks.workunit.client.1.smithi135.stdout:5/123: symlink d1/d7/d1a/d1f/l26 0 2024-03-30T04:28:11.264 INFO:tasks.workunit.client.0.smithi123.stdout:7/488: symlink d2/dc/d49/la8 0 2024-03-30T04:28:11.264 INFO:tasks.workunit.client.1.smithi135.stdout:5/124: readlink d1/l25 0 2024-03-30T04:28:11.266 INFO:tasks.workunit.client.0.smithi123.stdout:0/551: fsync d6/f24 0 2024-03-30T04:28:11.267 INFO:tasks.workunit.client.0.smithi123.stdout:2/586: mkdir d0/d1f/d1c/d20/dc6 0 2024-03-30T04:28:11.271 INFO:tasks.workunit.client.0.smithi123.stdout:7/489: chown d2/da/d11/d61/d55/d94 1393156 1 2024-03-30T04:28:11.275 INFO:tasks.workunit.client.0.smithi123.stdout:1/612: write db/dd/d42/d46/d56/d5c/f64 [3132947,35671] 0 2024-03-30T04:28:11.276 INFO:tasks.workunit.client.1.smithi135.stdout:5/125: truncate d1/d7/f13 61930 0 2024-03-30T04:28:11.279 INFO:tasks.workunit.client.0.smithi123.stdout:7/490: creat d2/da/d11/d34/d85/fa9 x:0 0 0 2024-03-30T04:28:11.280 INFO:tasks.workunit.client.1.smithi135.stdout:0/129: dread d0/f3 [0,4194304] 0 2024-03-30T04:28:11.280 INFO:tasks.workunit.client.0.smithi123.stdout:6/596: write fe [573972,126159] 0 2024-03-30T04:28:11.280 INFO:tasks.workunit.client.0.smithi123.stdout:8/448: dwrite da/d27/f53 [0,4194304] 0 2024-03-30T04:28:11.282 INFO:tasks.workunit.client.0.smithi123.stdout:3/552: dread d13/f1b [0,4194304] 0 2024-03-30T04:28:11.285 INFO:tasks.workunit.client.0.smithi123.stdout:1/613: fsync db/dd/d42/d46/d39/f52 0 2024-03-30T04:28:11.285 INFO:tasks.workunit.client.1.smithi135.stdout:0/130: readlink d0/d5/l26 0 2024-03-30T04:28:11.290 INFO:tasks.workunit.client.1.smithi135.stdout:8/118: dwrite d6/fa [0,4194304] 0 2024-03-30T04:28:11.290 INFO:tasks.workunit.client.0.smithi123.stdout:8/449: stat da/d27/d3c/d2d/d5c/f6f 0 2024-03-30T04:28:11.290 INFO:tasks.workunit.client.0.smithi123.stdout:7/491: mknod d2/dc/d7f/caa 0 2024-03-30T04:28:11.290 INFO:tasks.workunit.client.0.smithi123.stdout:6/597: fsync d10/d82/d9a/db3/fc9 0 2024-03-30T04:28:11.291 INFO:tasks.workunit.client.1.smithi135.stdout:3/82: dwrite f2 [0,4194304] 0 2024-03-30T04:28:11.294 INFO:tasks.workunit.client.0.smithi123.stdout:8/450: write da/ff [4085179,104307] 0 2024-03-30T04:28:11.294 INFO:tasks.workunit.client.1.smithi135.stdout:8/119: fsync d6/d8/f19 0 2024-03-30T04:28:11.296 INFO:tasks.workunit.client.1.smithi135.stdout:0/131: truncate d0/d5/d9/fd 1019199 0 2024-03-30T04:28:11.299 INFO:tasks.workunit.client.0.smithi123.stdout:2/587: dwrite d0/fb4 [0,4194304] 0 2024-03-30T04:28:11.299 INFO:tasks.workunit.client.0.smithi123.stdout:6/598: readlink d10/d82/dc1/l4a 0 2024-03-30T04:28:11.300 INFO:tasks.workunit.client.1.smithi135.stdout:9/63: dwrite d7/ff [0,4194304] 0 2024-03-30T04:28:11.300 INFO:tasks.workunit.client.0.smithi123.stdout:8/451: truncate da/f3d 835300 0 2024-03-30T04:28:11.301 INFO:tasks.workunit.client.1.smithi135.stdout:5/126: link d1/cb d1/d3/c27 0 2024-03-30T04:28:11.302 INFO:tasks.workunit.client.0.smithi123.stdout:8/452: chown da/d27/d56/c60 1 1 2024-03-30T04:28:11.305 INFO:tasks.workunit.client.1.smithi135.stdout:0/132: truncate d0/d1/d16/f28 1329899 0 2024-03-30T04:28:11.306 INFO:tasks.workunit.client.0.smithi123.stdout:7/492: mkdir d2/da/d2d/dab 0 2024-03-30T04:28:11.306 INFO:tasks.workunit.client.0.smithi123.stdout:1/614: rmdir db/dd/d42/d46/d5d 39 2024-03-30T04:28:11.307 INFO:tasks.workunit.client.1.smithi135.stdout:5/127: write d1/da/f18 [341007,61038] 0 2024-03-30T04:28:11.309 INFO:tasks.workunit.client.1.smithi135.stdout:3/83: mkdir d3/d17 0 2024-03-30T04:28:11.309 INFO:tasks.workunit.client.1.smithi135.stdout:5/128: stat d1/d7/d1a/d1f/l26 0 2024-03-30T04:28:11.310 INFO:tasks.workunit.client.1.smithi135.stdout:9/64: chown d7/da/fc 102 1 2024-03-30T04:28:11.311 INFO:tasks.workunit.client.1.smithi135.stdout:9/65: chown d7/da 282384 1 2024-03-30T04:28:11.311 INFO:tasks.workunit.client.1.smithi135.stdout:0/133: creat d0/d1/f29 x:0 0 0 2024-03-30T04:28:11.312 INFO:tasks.workunit.client.0.smithi123.stdout:0/552: dwrite d6/dd/d15/d28/fb6 [0,4194304] 0 2024-03-30T04:28:11.314 INFO:tasks.workunit.client.1.smithi135.stdout:9/66: write d7/f11 [188525,45157] 0 2024-03-30T04:28:11.314 INFO:tasks.workunit.client.0.smithi123.stdout:0/553: readlink l2 0 2024-03-30T04:28:11.315 INFO:tasks.workunit.client.1.smithi135.stdout:5/129: write d1/da/f1b [132188,13864] 0 2024-03-30T04:28:11.315 INFO:tasks.workunit.client.1.smithi135.stdout:9/67: chown d7/da 0 1 2024-03-30T04:28:11.316 INFO:tasks.workunit.client.0.smithi123.stdout:0/554: dread - d6/dd/d31/d35/d3a/f95 zero size 2024-03-30T04:28:11.317 INFO:tasks.workunit.client.1.smithi135.stdout:3/84: write d3/d10/f13 [3107937,79185] 0 2024-03-30T04:28:11.318 INFO:tasks.workunit.client.0.smithi123.stdout:6/599: creat d10/d82/d9a/db3/fcf x:0 0 0 2024-03-30T04:28:11.318 INFO:tasks.workunit.client.0.smithi123.stdout:2/588: chown d0/d1f/cd 15 1 2024-03-30T04:28:11.322 INFO:tasks.workunit.client.0.smithi123.stdout:8/453: rename da/d27/l3f to da/d27/d56/l85 0 2024-03-30T04:28:11.322 INFO:tasks.workunit.client.0.smithi123.stdout:6/600: stat d10/d82/dc1/d6a/d7a/d99/dba 0 2024-03-30T04:28:11.323 INFO:tasks.workunit.client.0.smithi123.stdout:7/493: creat d2/da/d11/d34/d85/fac x:0 0 0 2024-03-30T04:28:11.324 INFO:tasks.workunit.client.0.smithi123.stdout:1/615: rmdir db/dd/d5f/d6d/d85/d89/d90/d94 39 2024-03-30T04:28:11.328 INFO:tasks.workunit.client.0.smithi123.stdout:2/589: unlink d0/d1f/d1c/d36/f7d 0 2024-03-30T04:28:11.332 INFO:tasks.workunit.client.0.smithi123.stdout:7/494: write d2/da/d11/f48 [1380949,14553] 0 2024-03-30T04:28:11.333 INFO:tasks.workunit.client.0.smithi123.stdout:6/601: creat d10/d82/dc1/d6a/d7a/d99/dba/fd0 x:0 0 0 2024-03-30T04:28:11.333 INFO:tasks.workunit.client.0.smithi123.stdout:2/590: stat d0/d1f/d5e/d71/c7f 0 2024-03-30T04:28:11.336 INFO:tasks.workunit.client.1.smithi135.stdout:5/130: getdents d1/da 0 2024-03-30T04:28:11.339 INFO:tasks.workunit.client.0.smithi123.stdout:1/616: symlink db/dd/d42/d46/d5d/d62/d9e/ld1 0 2024-03-30T04:28:11.342 INFO:tasks.workunit.client.0.smithi123.stdout:7/495: rename d2/da/f33 to d2/da/d11/d61/d55/d9e/fad 0 2024-03-30T04:28:11.342 INFO:tasks.workunit.client.1.smithi135.stdout:0/134: dwrite d0/d5/d10/f11 [0,4194304] 0 2024-03-30T04:28:11.342 INFO:tasks.workunit.client.1.smithi135.stdout:3/85: dread d3/d6/f7 [0,4194304] 0 2024-03-30T04:28:11.344 INFO:tasks.workunit.client.1.smithi135.stdout:0/135: read d0/f21 [580985,54190] 0 2024-03-30T04:28:11.346 INFO:tasks.workunit.client.0.smithi123.stdout:1/617: dread - db/dd/d42/d46/f53 zero size 2024-03-30T04:28:11.348 INFO:tasks.workunit.client.0.smithi123.stdout:1/618: chown db/dd/d42/d46/d39/f52 87051 1 2024-03-30T04:28:11.350 INFO:tasks.workunit.client.0.smithi123.stdout:1/619: readlink db/dd/d42/d46/l3a 0 2024-03-30T04:28:11.350 INFO:tasks.workunit.client.0.smithi123.stdout:8/454: dread da/d27/f3b [0,4194304] 0 2024-03-30T04:28:11.351 INFO:tasks.workunit.client.1.smithi135.stdout:0/136: truncate d0/d1/d16/f1b 866640 0 2024-03-30T04:28:11.355 INFO:tasks.workunit.client.0.smithi123.stdout:8/455: unlink da/dc/c2a 0 2024-03-30T04:28:11.355 INFO:tasks.workunit.client.1.smithi135.stdout:3/86: write d3/f5 [888576,91399] 0 2024-03-30T04:28:11.356 INFO:tasks.workunit.client.0.smithi123.stdout:8/456: chown da/f3d 4 1 2024-03-30T04:28:11.357 INFO:tasks.workunit.client.0.smithi123.stdout:1/620: creat db/dd/d42/d46/d56/da2/fd2 x:0 0 0 2024-03-30T04:28:11.357 INFO:tasks.workunit.client.1.smithi135.stdout:5/131: dread d1/d7/f10 [0,4194304] 0 2024-03-30T04:28:11.360 INFO:tasks.workunit.client.1.smithi135.stdout:5/132: rmdir d1/d7/d1a 39 2024-03-30T04:28:11.360 INFO:tasks.workunit.client.1.smithi135.stdout:0/137: dread d0/d1/d16/f1b [0,4194304] 0 2024-03-30T04:28:11.361 INFO:tasks.workunit.client.0.smithi123.stdout:3/553: truncate d13/d24/d31/d23/d3a/d3c/d4e/f72 430751 0 2024-03-30T04:28:11.362 INFO:tasks.workunit.client.1.smithi135.stdout:0/138: fsync d0/d5/d9/fb 0 2024-03-30T04:28:11.363 INFO:tasks.workunit.client.0.smithi123.stdout:3/554: stat d13/d24/d31/d23/d63/fa7 0 2024-03-30T04:28:11.363 INFO:tasks.workunit.client.1.smithi135.stdout:0/139: fsync d0/f21 0 2024-03-30T04:28:11.363 INFO:tasks.workunit.client.0.smithi123.stdout:1/621: read db/dd/d42/d46/d39/f8a [502330,22344] 0 2024-03-30T04:28:11.364 INFO:tasks.workunit.client.1.smithi135.stdout:0/140: stat d0/d5/d9/l1c 0 2024-03-30T04:28:11.364 INFO:tasks.workunit.client.0.smithi123.stdout:1/622: readlink db/l40 0 2024-03-30T04:28:11.365 INFO:tasks.workunit.client.1.smithi135.stdout:5/133: link d1/da/c22 d1/c28 0 2024-03-30T04:28:11.366 INFO:tasks.workunit.client.1.smithi135.stdout:0/141: read d0/d1/fa [3737098,120990] 0 2024-03-30T04:28:11.366 INFO:tasks.workunit.client.1.smithi135.stdout:5/134: chown d1 837066 1 2024-03-30T04:28:11.369 INFO:tasks.workunit.client.0.smithi123.stdout:1/623: unlink db/dd/d42/d46/f47 0 2024-03-30T04:28:11.371 INFO:tasks.workunit.client.1.smithi135.stdout:0/142: rename d0/d5/d9/d25 to d0/d5/d9/d23/d2a 0 2024-03-30T04:28:11.372 INFO:tasks.workunit.client.1.smithi135.stdout:0/143: dread - d0/d13/d1d/f1f zero size 2024-03-30T04:28:11.374 INFO:tasks.workunit.client.1.smithi135.stdout:0/144: chown d0/d5/d9/fb 851 1 2024-03-30T04:28:11.374 INFO:tasks.workunit.client.0.smithi123.stdout:1/624: unlink db/fcc 0 2024-03-30T04:28:11.375 INFO:tasks.workunit.client.1.smithi135.stdout:0/145: chown d0/d1/d16/f28 116 1 2024-03-30T04:28:11.376 INFO:tasks.workunit.client.0.smithi123.stdout:2/591: dwrite d0/d37/d9b/fc4 [0,4194304] 0 2024-03-30T04:28:11.376 INFO:tasks.workunit.client.0.smithi123.stdout:8/457: dwrite da/dc/f13 [4194304,4194304] 0 2024-03-30T04:28:11.377 INFO:tasks.workunit.client.0.smithi123.stdout:2/592: chown d0 38 1 2024-03-30T04:28:11.378 INFO:tasks.workunit.client.1.smithi135.stdout:5/135: write d1/d3/f12 [473076,91116] 0 2024-03-30T04:28:11.380 INFO:tasks.workunit.client.1.smithi135.stdout:0/146: creat d0/d5/d10/f2b x:0 0 0 2024-03-30T04:28:11.381 INFO:tasks.workunit.client.0.smithi123.stdout:1/625: symlink db/dd/d5f/dae/ld3 0 2024-03-30T04:28:11.382 INFO:tasks.workunit.client.1.smithi135.stdout:5/136: chown d1/l25 4696 1 2024-03-30T04:28:11.385 INFO:tasks.workunit.client.1.smithi135.stdout:5/137: truncate d1/d3/f12 983371 0 2024-03-30T04:28:11.386 INFO:tasks.workunit.client.1.smithi135.stdout:0/147: creat d0/d5/d10/f2c x:0 0 0 2024-03-30T04:28:11.388 INFO:tasks.workunit.client.0.smithi123.stdout:2/593: dread d0/d1f/d1c/d36/f89 [0,4194304] 0 2024-03-30T04:28:11.389 INFO:tasks.workunit.client.0.smithi123.stdout:3/555: dwrite d13/d24/d31/d23/f88 [4194304,4194304] 0 2024-03-30T04:28:11.394 INFO:tasks.workunit.client.1.smithi135.stdout:5/138: chown d1/d7/d1a/c1e 352992 1 2024-03-30T04:28:11.394 INFO:tasks.workunit.client.1.smithi135.stdout:0/148: mkdir d0/d19/d2d 0 2024-03-30T04:28:11.394 INFO:tasks.workunit.client.1.smithi135.stdout:5/139: stat d1/da/d17/c1c 0 2024-03-30T04:28:11.397 INFO:tasks.workunit.client.0.smithi123.stdout:2/594: dread d0/d1f/d1c/f41 [0,4194304] 0 2024-03-30T04:28:11.399 INFO:tasks.workunit.client.1.smithi135.stdout:5/140: getdents d1/d7/d11 0 2024-03-30T04:28:11.400 INFO:tasks.workunit.client.1.smithi135.stdout:0/149: dread d0/f21 [0,4194304] 0 2024-03-30T04:28:11.401 INFO:tasks.workunit.client.0.smithi123.stdout:8/458: dwrite da/d27/f53 [0,4194304] 0 2024-03-30T04:28:11.402 INFO:tasks.workunit.client.1.smithi135.stdout:0/150: fdatasync d0/f21 0 2024-03-30T04:28:11.402 INFO:tasks.workunit.client.1.smithi135.stdout:5/141: write d1/d7/f13 [713332,56176] 0 2024-03-30T04:28:11.402 INFO:tasks.workunit.client.0.smithi123.stdout:2/595: write d0/d1f/d27/f8b [4338312,110893] 0 2024-03-30T04:28:11.413 INFO:tasks.workunit.client.0.smithi123.stdout:8/459: creat da/dc/d14/d18/f86 x:0 0 0 2024-03-30T04:28:11.419 INFO:tasks.workunit.client.0.smithi123.stdout:3/556: dwrite d13/d24/d31/fae [0,4194304] 0 2024-03-30T04:28:11.421 INFO:tasks.workunit.client.1.smithi135.stdout:0/151: dwrite d0/f3 [0,4194304] 0 2024-03-30T04:28:11.422 INFO:tasks.workunit.client.0.smithi123.stdout:2/596: link d0/d1f/d1c/d36/l58 d0/d1f/d1c/d2a/d8c/lc7 0 2024-03-30T04:28:11.424 INFO:tasks.workunit.client.1.smithi135.stdout:0/152: truncate d0/d1/d16/f28 1873546 0 2024-03-30T04:28:11.425 INFO:tasks.workunit.client.0.smithi123.stdout:8/460: rename da/d27/l29 to da/dc/d14/d18/l87 0 2024-03-30T04:28:11.425 INFO:tasks.workunit.client.0.smithi123.stdout:3/557: symlink d13/d24/d31/d23/d3a/d3c/d4e/laf 0 2024-03-30T04:28:11.427 INFO:tasks.workunit.client.1.smithi135.stdout:0/153: creat d0/d5/d9/f2e x:0 0 0 2024-03-30T04:28:11.432 INFO:tasks.workunit.client.1.smithi135.stdout:5/142: write d1/f4 [21900,126878] 0 2024-03-30T04:28:11.441 INFO:tasks.workunit.client.0.smithi123.stdout:8/461: dread da/d27/d3c/d2d/f4b [0,4194304] 0 2024-03-30T04:28:11.446 INFO:tasks.workunit.client.0.smithi123.stdout:8/462: mknod da/dc/c88 0 2024-03-30T04:28:11.447 INFO:tasks.workunit.client.1.smithi135.stdout:5/143: dread d1/d3/f15 [0,4194304] 0 2024-03-30T04:28:11.448 INFO:tasks.workunit.client.1.smithi135.stdout:0/154: dwrite d0/d5/d10/f2c [0,4194304] 0 2024-03-30T04:28:11.452 INFO:tasks.workunit.client.1.smithi135.stdout:0/155: getdents d0/d19/d2d 0 2024-03-30T04:28:11.454 INFO:tasks.workunit.client.0.smithi123.stdout:2/597: dwrite d0/d1f/d1c/d39/d4b/fb1 [0,4194304] 0 2024-03-30T04:28:11.455 INFO:tasks.workunit.client.1.smithi135.stdout:0/156: symlink d0/d1/d16/l2f 0 2024-03-30T04:28:11.455 INFO:tasks.workunit.client.1.smithi135.stdout:0/157: fsync d0/f21 0 2024-03-30T04:28:11.456 INFO:tasks.workunit.client.0.smithi123.stdout:1/626: dread db/fe [0,4194304] 0 2024-03-30T04:28:11.456 INFO:tasks.workunit.client.0.smithi123.stdout:8/463: link da/c5f da/dc/d14/d18/d62/c89 0 2024-03-30T04:28:11.458 INFO:tasks.workunit.client.1.smithi135.stdout:0/158: read d0/d1/d16/f28 [351763,125965] 0 2024-03-30T04:28:11.458 INFO:tasks.workunit.client.1.smithi135.stdout:0/159: chown d0/d1 7826 1 2024-03-30T04:28:11.460 INFO:tasks.workunit.client.1.smithi135.stdout:5/144: dread d1/d7/f10 [0,4194304] 0 2024-03-30T04:28:11.460 INFO:tasks.workunit.client.1.smithi135.stdout:0/160: stat d0/d5/d9/l22 0 2024-03-30T04:28:11.461 INFO:tasks.workunit.client.0.smithi123.stdout:1/627: chown db/dd/d42/d46/d56/fb6 1792 1 2024-03-30T04:28:11.462 INFO:tasks.workunit.client.1.smithi135.stdout:5/145: readlink d1/d3/l5 0 2024-03-30T04:28:11.463 INFO:tasks.workunit.client.1.smithi135.stdout:0/161: rmdir d0/d5/d10 39 2024-03-30T04:28:11.465 INFO:tasks.workunit.client.0.smithi123.stdout:1/628: dread - db/dd/d42/d46/d51/d72/fa6 zero size 2024-03-30T04:28:11.466 INFO:tasks.workunit.client.1.smithi135.stdout:0/162: unlink d0/l1a 0 2024-03-30T04:28:11.470 INFO:tasks.workunit.client.1.smithi135.stdout:0/163: truncate d0/d1/d16/f1b 1666433 0 2024-03-30T04:28:11.472 INFO:tasks.workunit.client.0.smithi123.stdout:2/598: symlink d0/d1f/d82/lc8 0 2024-03-30T04:28:11.472 INFO:tasks.workunit.client.1.smithi135.stdout:0/164: write d0/d5/d9/fb [151504,95290] 0 2024-03-30T04:28:11.475 INFO:tasks.workunit.client.0.smithi123.stdout:1/629: rmdir db/dd/d5f/d6d/d85/d89/d90/d94/d9c 39 2024-03-30T04:28:11.476 INFO:tasks.workunit.client.1.smithi135.stdout:0/165: write d0/d5/d9/fb [138031,42863] 0 2024-03-30T04:28:11.477 INFO:tasks.workunit.client.0.smithi123.stdout:2/599: symlink d0/d1f/d30/lc9 0 2024-03-30T04:28:11.477 INFO:tasks.workunit.client.0.smithi123.stdout:2/600: readlink d0/l15 0 2024-03-30T04:28:11.478 INFO:tasks.workunit.client.1.smithi135.stdout:0/166: dread - d0/d5/d9/ff zero size 2024-03-30T04:28:11.480 INFO:tasks.workunit.client.1.smithi135.stdout:0/167: truncate d0/fc 708413 0 2024-03-30T04:28:11.481 INFO:tasks.workunit.client.1.smithi135.stdout:6/38: sync 2024-03-30T04:28:11.481 INFO:tasks.workunit.client.1.smithi135.stdout:1/71: sync 2024-03-30T04:28:11.482 INFO:tasks.workunit.client.0.smithi123.stdout:2/601: truncate d0/d1f/d1c/f41 1693487 0 2024-03-30T04:28:11.483 INFO:tasks.workunit.client.1.smithi135.stdout:5/146: dwrite d1/d3/f15 [4194304,4194304] 0 2024-03-30T04:28:11.486 INFO:tasks.workunit.client.1.smithi135.stdout:1/72: mkdir d16 0 2024-03-30T04:28:11.487 INFO:tasks.workunit.client.1.smithi135.stdout:0/168: mknod d0/d5/d9/d23/d2a/c30 0 2024-03-30T04:28:11.490 INFO:tasks.workunit.client.0.smithi123.stdout:3/558: dread d13/f7c [0,4194304] 0 2024-03-30T04:28:11.490 INFO:tasks.workunit.client.0.smithi123.stdout:2/602: creat d0/d1f/d5e/d71/fca x:0 0 0 2024-03-30T04:28:11.497 INFO:tasks.workunit.client.1.smithi135.stdout:1/73: dread fd [0,4194304] 0 2024-03-30T04:28:11.501 INFO:tasks.workunit.client.0.smithi123.stdout:2/603: write d0/d1f/d1c/d40/f5a [1170788,1811] 0 2024-03-30T04:28:11.501 INFO:tasks.workunit.client.0.smithi123.stdout:3/559: symlink d13/d24/d31/d23/d3a/d3c/lb0 0 2024-03-30T04:28:11.504 INFO:tasks.workunit.client.0.smithi123.stdout:2/604: fsync d0/d1f/d1c/d80/f93 0 2024-03-30T04:28:11.506 INFO:tasks.workunit.client.1.smithi135.stdout:0/169: dread d0/d1/d16/f1b [0,4194304] 0 2024-03-30T04:28:11.506 INFO:tasks.workunit.client.0.smithi123.stdout:3/560: creat d13/d24/d31/d23/d3a/fb1 x:0 0 0 2024-03-30T04:28:11.507 INFO:tasks.workunit.client.0.smithi123.stdout:8/464: dwrite da/fb [0,4194304] 0 2024-03-30T04:28:11.508 INFO:tasks.workunit.client.0.smithi123.stdout:2/605: creat d0/d1f/d16/fcb x:0 0 0 2024-03-30T04:28:11.509 INFO:tasks.workunit.client.1.smithi135.stdout:1/74: dread fa [0,4194304] 0 2024-03-30T04:28:11.509 INFO:tasks.workunit.client.0.smithi123.stdout:3/561: chown d13/d24/d31/d23/d3a/d3c/d4e/d5d/da5 1833954838 1 2024-03-30T04:28:11.509 INFO:tasks.workunit.client.0.smithi123.stdout:8/465: stat da/d27/d3c/f5e 0 2024-03-30T04:28:11.511 INFO:tasks.workunit.client.0.smithi123.stdout:3/562: readlink d13/d24/d39/l46 0 2024-03-30T04:28:11.511 INFO:tasks.workunit.client.1.smithi135.stdout:1/75: truncate fb 807117 0 2024-03-30T04:28:11.512 INFO:tasks.workunit.client.1.smithi135.stdout:0/170: creat d0/d5/d10/f31 x:0 0 0 2024-03-30T04:28:11.513 INFO:tasks.workunit.client.1.smithi135.stdout:5/147: dwrite d1/da/f1b [0,4194304] 0 2024-03-30T04:28:11.515 INFO:tasks.workunit.client.1.smithi135.stdout:0/171: write d0/d5/d9/f2e [814366,113486] 0 2024-03-30T04:28:11.515 INFO:tasks.workunit.client.1.smithi135.stdout:1/76: rename cc to d16/c17 0 2024-03-30T04:28:11.517 INFO:tasks.workunit.client.1.smithi135.stdout:0/172: read - d0/d1/f29 zero size 2024-03-30T04:28:11.519 INFO:tasks.workunit.client.1.smithi135.stdout:5/148: rename d1/da/l21 to d1/d3/l29 0 2024-03-30T04:28:11.523 INFO:tasks.workunit.client.1.smithi135.stdout:1/77: creat d16/f18 x:0 0 0 2024-03-30T04:28:11.526 INFO:tasks.workunit.client.1.smithi135.stdout:5/149: mknod d1/d7/d1a/c2a 0 2024-03-30T04:28:11.526 INFO:tasks.workunit.client.1.smithi135.stdout:1/78: mkdir d16/d19 0 2024-03-30T04:28:11.530 INFO:tasks.workunit.client.0.smithi123.stdout:2/606: dwrite d0/d1f/d1c/d39/d4b/fa7 [0,4194304] 0 2024-03-30T04:28:11.530 INFO:tasks.workunit.client.1.smithi135.stdout:1/79: unlink f6 0 2024-03-30T04:28:11.533 INFO:tasks.workunit.client.0.smithi123.stdout:2/607: creat d0/d1f/d16/fcc x:0 0 0 2024-03-30T04:28:11.536 INFO:tasks.workunit.client.0.smithi123.stdout:2/608: write d0/d1f/d1c/d20/f9f [2052338,64514] 0 2024-03-30T04:28:11.536 INFO:tasks.workunit.client.1.smithi135.stdout:1/80: mkdir d16/d1a 0 2024-03-30T04:28:11.539 INFO:tasks.workunit.client.0.smithi123.stdout:2/609: write d0/d37/d9b/fc4 [1384903,34001] 0 2024-03-30T04:28:11.540 INFO:tasks.workunit.client.0.smithi123.stdout:3/563: getdents d13/d24/d31/d23/d3a 0 2024-03-30T04:28:11.542 INFO:tasks.workunit.client.1.smithi135.stdout:0/173: dwrite d0/d5/d10/f2c [0,4194304] 0 2024-03-30T04:28:11.544 INFO:tasks.workunit.client.1.smithi135.stdout:1/81: mknod d16/d19/c1b 0 2024-03-30T04:28:11.544 INFO:tasks.workunit.client.0.smithi123.stdout:8/466: dread da/dc/f20 [0,4194304] 0 2024-03-30T04:28:11.544 INFO:tasks.workunit.client.0.smithi123.stdout:3/564: write d13/d24/f70 [649214,115680] 0 2024-03-30T04:28:11.549 INFO:tasks.workunit.client.1.smithi135.stdout:0/174: creat d0/d1/d16/f32 x:0 0 0 2024-03-30T04:28:11.549 INFO:tasks.workunit.client.1.smithi135.stdout:5/150: dread d1/d3/f15 [4194304,4194304] 0 2024-03-30T04:28:11.552 INFO:tasks.workunit.client.0.smithi123.stdout:3/565: chown d13/d24/d31/f61 301180 1 2024-03-30T04:28:11.554 INFO:tasks.workunit.client.1.smithi135.stdout:5/151: mkdir d1/d3/d2b 0 2024-03-30T04:28:11.557 INFO:tasks.workunit.client.1.smithi135.stdout:1/82: link fa d16/f1c 0 2024-03-30T04:28:11.559 INFO:tasks.workunit.client.0.smithi123.stdout:3/566: mkdir d13/d24/d31/d23/d3a/d3c/dab/db2 0 2024-03-30T04:28:11.560 INFO:tasks.workunit.client.1.smithi135.stdout:1/83: creat d16/f1d x:0 0 0 2024-03-30T04:28:11.561 INFO:tasks.workunit.client.1.smithi135.stdout:5/152: getdents d1 0 2024-03-30T04:28:11.562 INFO:tasks.workunit.client.1.smithi135.stdout:1/84: readlink l11 0 2024-03-30T04:28:11.564 INFO:tasks.workunit.client.1.smithi135.stdout:1/85: chown d16/f18 0 1 2024-03-30T04:28:11.565 INFO:tasks.workunit.client.1.smithi135.stdout:1/86: write f14 [404090,27045] 0 2024-03-30T04:28:11.566 INFO:tasks.workunit.client.0.smithi123.stdout:2/610: dwrite d0/d1f/d1c/d80/f49 [0,4194304] 0 2024-03-30T04:28:11.567 INFO:tasks.workunit.client.1.smithi135.stdout:0/175: dread d0/d5/d10/f2c [0,4194304] 0 2024-03-30T04:28:11.567 INFO:tasks.workunit.client.1.smithi135.stdout:1/87: dread - d16/f1d zero size 2024-03-30T04:28:11.567 INFO:tasks.workunit.client.1.smithi135.stdout:5/153: creat d1/d7/f2c x:0 0 0 2024-03-30T04:28:11.568 INFO:tasks.workunit.client.0.smithi123.stdout:8/467: dwrite da/ff [4194304,4194304] 0 2024-03-30T04:28:11.569 INFO:tasks.workunit.client.0.smithi123.stdout:8/468: chown da 30957237 1 2024-03-30T04:28:11.569 INFO:tasks.workunit.client.0.smithi123.stdout:2/611: symlink d0/d1f/d16/lcd 0 2024-03-30T04:28:11.569 INFO:tasks.workunit.client.1.smithi135.stdout:0/176: readlink d0/d1/d16/l2f 0 2024-03-30T04:28:11.571 INFO:tasks.workunit.client.1.smithi135.stdout:1/88: symlink d16/d19/l1e 0 2024-03-30T04:28:11.576 INFO:tasks.workunit.client.1.smithi135.stdout:1/89: unlink c2 0 2024-03-30T04:28:11.578 INFO:tasks.workunit.client.0.smithi123.stdout:2/612: unlink d0/d1f/d1c/d80/da8/fc5 0 2024-03-30T04:28:11.581 INFO:tasks.workunit.client.0.smithi123.stdout:3/567: dwrite f11 [0,4194304] 0 2024-03-30T04:28:11.584 INFO:tasks.workunit.client.1.smithi135.stdout:0/177: link d0/d1/fa d0/d5/d9/f33 0 2024-03-30T04:28:11.585 INFO:tasks.workunit.client.1.smithi135.stdout:1/90: chown d16/f18 4289132 1 2024-03-30T04:28:11.585 INFO:tasks.workunit.client.1.smithi135.stdout:0/178: fdatasync d0/d5/d9/f15 0 2024-03-30T04:28:11.585 INFO:tasks.workunit.client.0.smithi123.stdout:3/568: symlink d13/d24/d31/d23/d3a/d3c/d66/lb3 0 2024-03-30T04:28:11.585 INFO:tasks.workunit.client.0.smithi123.stdout:3/569: chown fe 54 1 2024-03-30T04:28:11.586 INFO:tasks.workunit.client.1.smithi135.stdout:0/179: mkdir d0/d34 0 2024-03-30T04:28:11.591 INFO:tasks.workunit.client.0.smithi123.stdout:3/570: rename d13/d24/d31/d30/c8f to d13/d24/d31/d30/cb4 0 2024-03-30T04:28:11.592 INFO:tasks.workunit.client.1.smithi135.stdout:5/154: dwrite d1/f4 [4194304,4194304] 0 2024-03-30T04:28:11.597 INFO:tasks.workunit.client.1.smithi135.stdout:7/156: sync 2024-03-30T04:28:11.597 INFO:tasks.workunit.client.1.smithi135.stdout:2/63: sync 2024-03-30T04:28:11.597 INFO:tasks.workunit.client.1.smithi135.stdout:8/120: sync 2024-03-30T04:28:11.597 INFO:tasks.workunit.client.1.smithi135.stdout:4/56: sync 2024-03-30T04:28:11.599 INFO:tasks.workunit.client.0.smithi123.stdout:2/613: truncate d0/d1f/d1c/d40/f43 3464629 0 2024-03-30T04:28:11.601 INFO:tasks.workunit.client.0.smithi123.stdout:2/614: stat d0/d1f/d5e/d71 0 2024-03-30T04:28:11.605 INFO:tasks.workunit.client.1.smithi135.stdout:4/57: creat d10/f14 x:0 0 0 2024-03-30T04:28:11.607 INFO:tasks.workunit.client.0.smithi123.stdout:8/469: dwrite da/f11 [4194304,4194304] 0 2024-03-30T04:28:11.609 INFO:tasks.workunit.client.1.smithi135.stdout:8/121: truncate d6/d8/d14/fe 2727652 0 2024-03-30T04:28:11.614 INFO:tasks.workunit.client.1.smithi135.stdout:7/157: truncate f3 2159089 0 2024-03-30T04:28:11.615 INFO:tasks.workunit.client.0.smithi123.stdout:3/571: dwrite d13/d24/d31/d23/f2e [0,4194304] 0 2024-03-30T04:28:11.615 INFO:tasks.workunit.client.1.smithi135.stdout:1/91: getdents d16/d1a 0 2024-03-30T04:28:11.617 INFO:tasks.workunit.client.1.smithi135.stdout:8/122: mknod d6/c1e 0 2024-03-30T04:28:11.617 INFO:tasks.workunit.client.0.smithi123.stdout:8/470: rename da/dc/d14/d18/d21/f28 to da/d27/f8a 0 2024-03-30T04:28:11.619 INFO:tasks.workunit.client.0.smithi123.stdout:8/471: readlink da/d27/l5d 0 2024-03-30T04:28:11.625 INFO:tasks.workunit.client.1.smithi135.stdout:1/92: write d16/f1d [530337,116402] 0 2024-03-30T04:28:11.625 INFO:tasks.workunit.client.1.smithi135.stdout:7/158: chown d4/db/de/l17 337 1 2024-03-30T04:28:11.627 INFO:tasks.workunit.client.0.smithi123.stdout:2/615: dwrite d0/d37/f74 [0,4194304] 0 2024-03-30T04:28:11.628 INFO:tasks.workunit.client.1.smithi135.stdout:8/123: mkdir d6/d8/d15/d1f 0 2024-03-30T04:28:11.628 INFO:tasks.workunit.client.1.smithi135.stdout:2/64: dread d4/f5 [0,4194304] 0 2024-03-30T04:28:11.630 INFO:tasks.workunit.client.0.smithi123.stdout:2/616: stat d0/d37/d9b 0 2024-03-30T04:28:11.635 INFO:tasks.workunit.client.1.smithi135.stdout:1/93: creat d16/d19/f1f x:0 0 0 2024-03-30T04:28:11.638 INFO:tasks.workunit.client.1.smithi135.stdout:5/155: dwrite d1/da/f1b [0,4194304] 0 2024-03-30T04:28:11.638 INFO:tasks.workunit.client.0.smithi123.stdout:8/472: dwrite da/d27/d56/f65 [0,4194304] 0 2024-03-30T04:28:11.642 INFO:tasks.workunit.client.0.smithi123.stdout:8/473: chown da/dc/d14/f71 112247 1 2024-03-30T04:28:11.643 INFO:tasks.workunit.client.1.smithi135.stdout:0/180: dwrite d0/d5/d9/f33 [4194304,4194304] 0 2024-03-30T04:28:11.644 INFO:tasks.workunit.client.0.smithi123.stdout:3/572: dwrite d13/d24/d31/d23/d3a/d3c/d4e/d5d/f68 [0,4194304] 0 2024-03-30T04:28:11.648 INFO:tasks.workunit.client.1.smithi135.stdout:4/58: dwrite fd [0,4194304] 0 2024-03-30T04:28:11.648 INFO:tasks.workunit.client.1.smithi135.stdout:2/65: write d4/f5 [9322641,6070] 0 2024-03-30T04:28:11.651 INFO:tasks.workunit.client.1.smithi135.stdout:1/94: mkdir d16/d19/d20 0 2024-03-30T04:28:11.656 INFO:tasks.workunit.client.1.smithi135.stdout:8/124: getdents d6/d8 0 2024-03-30T04:28:11.660 INFO:tasks.workunit.client.1.smithi135.stdout:1/95: fdatasync d16/f1d 0 2024-03-30T04:28:11.664 INFO:tasks.workunit.client.1.smithi135.stdout:4/59: dread - d10/f14 zero size 2024-03-30T04:28:11.666 INFO:tasks.workunit.client.1.smithi135.stdout:5/156: rename d1/d7/l16 to d1/d7/l2d 0 2024-03-30T04:28:11.667 INFO:tasks.workunit.client.0.smithi123.stdout:3/573: dwrite d13/d24/d31/d30/f45 [0,4194304] 0 2024-03-30T04:28:11.670 INFO:tasks.workunit.client.0.smithi123.stdout:8/474: dwrite da/d27/f8a [0,4194304] 0 2024-03-30T04:28:11.673 INFO:tasks.workunit.client.1.smithi135.stdout:8/125: creat d6/d8/f20 x:0 0 0 2024-03-30T04:28:11.677 INFO:tasks.workunit.client.0.smithi123.stdout:8/475: mknod da/dc/d14/c8b 0 2024-03-30T04:28:11.678 INFO:tasks.workunit.client.1.smithi135.stdout:1/96: unlink d16/d19/l1e 0 2024-03-30T04:28:11.683 INFO:tasks.workunit.client.0.smithi123.stdout:8/476: mknod da/d27/d3c/d2d/c8c 0 2024-03-30T04:28:11.683 INFO:tasks.workunit.client.1.smithi135.stdout:5/157: symlink d1/d7/d1a/d1f/l2e 0 2024-03-30T04:28:11.692 INFO:tasks.workunit.client.0.smithi123.stdout:3/574: dwrite d13/d24/d31/d23/f6c [0,4194304] 0 2024-03-30T04:28:11.692 INFO:tasks.workunit.client.1.smithi135.stdout:7/159: dwrite d4/db/d1a/d2a/f32 [4194304,4194304] 0 2024-03-30T04:28:11.693 INFO:tasks.workunit.client.0.smithi123.stdout:8/477: rename da/l2f to da/dc/l8d 0 2024-03-30T04:28:11.696 INFO:tasks.workunit.client.0.smithi123.stdout:8/478: write da/ff [6538181,74336] 0 2024-03-30T04:28:11.702 INFO:tasks.workunit.client.1.smithi135.stdout:7/160: symlink d4/db/d1a/d2a/d2c/l3b 0 2024-03-30T04:28:11.703 INFO:tasks.workunit.client.0.smithi123.stdout:8/479: creat da/d27/d3c/d64/f8e x:0 0 0 2024-03-30T04:28:11.703 INFO:tasks.workunit.client.1.smithi135.stdout:4/60: dread f4 [0,4194304] 0 2024-03-30T04:28:11.707 INFO:tasks.workunit.client.0.smithi123.stdout:3/575: link d13/c20 d13/d24/d39/d79/d90/cb5 0 2024-03-30T04:28:11.708 INFO:tasks.workunit.client.1.smithi135.stdout:2/66: dwrite d4/da/fb [0,4194304] 0 2024-03-30T04:28:11.710 INFO:tasks.workunit.client.1.smithi135.stdout:4/61: symlink d10/l15 0 2024-03-30T04:28:11.710 INFO:tasks.workunit.client.0.smithi123.stdout:3/576: unlink d13/d24/d31/d23/d63/fa7 0 2024-03-30T04:28:11.711 INFO:tasks.workunit.client.1.smithi135.stdout:4/62: chown d10/f14 1 1 2024-03-30T04:28:11.712 INFO:tasks.workunit.client.1.smithi135.stdout:7/161: mknod d4/db/d1a/d2a/c3c 0 2024-03-30T04:28:11.714 INFO:tasks.workunit.client.1.smithi135.stdout:4/63: readlink d10/l15 0 2024-03-30T04:28:11.715 INFO:tasks.workunit.client.1.smithi135.stdout:2/67: creat d4/d6/ff x:0 0 0 2024-03-30T04:28:11.716 INFO:tasks.workunit.client.0.smithi123.stdout:3/577: read d13/d24/d31/f61 [335300,64064] 0 2024-03-30T04:28:11.717 INFO:tasks.workunit.client.1.smithi135.stdout:4/64: mknod d10/c16 0 2024-03-30T04:28:11.719 INFO:tasks.workunit.client.1.smithi135.stdout:7/162: mkdir d4/db/d1a/d3d 0 2024-03-30T04:28:11.725 INFO:tasks.workunit.client.1.smithi135.stdout:8/126: stat d6/d8/d14/fe 0 2024-03-30T04:28:11.726 INFO:tasks.workunit.client.1.smithi135.stdout:4/65: rename f5 to d10/f17 0 2024-03-30T04:28:11.726 INFO:tasks.workunit.client.1.smithi135.stdout:1/97: write f7 [379274,104674] 0 2024-03-30T04:28:11.727 INFO:tasks.workunit.client.1.smithi135.stdout:7/163: write d4/db/de/d15/f31 [3280012,57443] 0 2024-03-30T04:28:11.728 INFO:tasks.workunit.client.1.smithi135.stdout:2/68: mknod d4/c10 0 2024-03-30T04:28:11.732 INFO:tasks.workunit.client.1.smithi135.stdout:4/66: creat d10/f18 x:0 0 0 2024-03-30T04:28:11.734 INFO:tasks.workunit.client.1.smithi135.stdout:2/69: creat d4/d6/f11 x:0 0 0 2024-03-30T04:28:11.734 INFO:tasks.workunit.client.1.smithi135.stdout:8/127: chown d6/d8/d15/l1a 10533751 1 2024-03-30T04:28:11.735 INFO:tasks.workunit.client.0.smithi123.stdout:3/578: dwrite fd [0,4194304] 0 2024-03-30T04:28:11.738 INFO:tasks.workunit.client.1.smithi135.stdout:1/98: rename f14 to d16/d1a/f21 0 2024-03-30T04:28:11.740 INFO:tasks.workunit.client.1.smithi135.stdout:4/67: chown d10/c13 129339347 1 2024-03-30T04:28:11.741 INFO:tasks.workunit.client.1.smithi135.stdout:2/70: mkdir d4/da/dd/d12 0 2024-03-30T04:28:11.742 INFO:tasks.workunit.client.0.smithi123.stdout:3/579: creat d13/d24/d31/d23/d3a/d3c/fb6 x:0 0 0 2024-03-30T04:28:11.745 INFO:tasks.workunit.client.1.smithi135.stdout:4/68: creat d10/f19 x:0 0 0 2024-03-30T04:28:11.747 INFO:tasks.workunit.client.0.smithi123.stdout:3/580: creat d13/d24/d31/d23/d3a/d3c/d4e/d5d/fb7 x:0 0 0 2024-03-30T04:28:11.749 INFO:tasks.workunit.client.0.smithi123.stdout:3/581: stat d13/d24/d39/d79/d90/fac 0 2024-03-30T04:28:11.749 INFO:tasks.workunit.client.1.smithi135.stdout:2/71: creat d4/da/dd/f13 x:0 0 0 2024-03-30T04:28:11.752 INFO:tasks.workunit.client.1.smithi135.stdout:8/128: rename d6/d8/f19 to d6/d8/d14/f21 0 2024-03-30T04:28:11.756 INFO:tasks.workunit.client.1.smithi135.stdout:4/69: mkdir d10/d1a 0 2024-03-30T04:28:11.758 INFO:tasks.workunit.client.1.smithi135.stdout:2/72: unlink d4/da/fc 0 2024-03-30T04:28:11.759 INFO:tasks.workunit.client.1.smithi135.stdout:2/73: read - d4/d6/f11 zero size 2024-03-30T04:28:11.760 INFO:tasks.workunit.client.0.smithi123.stdout:9/467: sync 2024-03-30T04:28:11.760 INFO:tasks.workunit.client.0.smithi123.stdout:0/555: sync 2024-03-30T04:28:11.760 INFO:tasks.workunit.client.0.smithi123.stdout:5/474: sync 2024-03-30T04:28:11.760 INFO:tasks.workunit.client.0.smithi123.stdout:4/479: sync 2024-03-30T04:28:11.761 INFO:tasks.workunit.client.1.smithi135.stdout:4/70: unlink d10/f12 0 2024-03-30T04:28:11.762 INFO:tasks.workunit.client.0.smithi123.stdout:9/468: rmdir d6/d1f/d4d 39 2024-03-30T04:28:11.766 INFO:tasks.workunit.client.1.smithi135.stdout:2/74: mknod d4/c14 0 2024-03-30T04:28:11.766 INFO:tasks.workunit.client.1.smithi135.stdout:4/71: write f3 [2207125,27806] 0 2024-03-30T04:28:11.766 INFO:tasks.workunit.client.0.smithi123.stdout:4/480: creat d2/d4/d2c/fa7 x:0 0 0 2024-03-30T04:28:11.767 INFO:tasks.workunit.client.1.smithi135.stdout:7/164: truncate d4/db/d1a/f24 2198570 0 2024-03-30T04:28:11.767 INFO:tasks.workunit.client.0.smithi123.stdout:4/481: readlink d2/d4/d7/l51 0 2024-03-30T04:28:11.768 INFO:tasks.workunit.client.0.smithi123.stdout:5/475: creat dd/d19/d1f/d35/d3a/d79/f96 x:0 0 0 2024-03-30T04:28:11.769 INFO:tasks.workunit.client.0.smithi123.stdout:0/556: getdents d6/dd/d15/d28/d4b 0 2024-03-30T04:28:11.770 INFO:tasks.workunit.client.1.smithi135.stdout:1/99: truncate d16/d1a/f21 86122 0 2024-03-30T04:28:11.771 INFO:tasks.workunit.client.0.smithi123.stdout:0/557: rmdir d6/dd/d15/d98 39 2024-03-30T04:28:11.772 INFO:tasks.workunit.client.1.smithi135.stdout:8/129: chown d6/d8/d15/d1f 118921342 1 2024-03-30T04:28:11.773 INFO:tasks.workunit.client.1.smithi135.stdout:4/72: symlink d10/d1a/l1b 0 2024-03-30T04:28:11.773 INFO:tasks.workunit.client.0.smithi123.stdout:4/482: rename d2/d4/d7/l51 to d2/d4/d2c/d42/d35/d5d/d78/la8 0 2024-03-30T04:28:11.774 INFO:tasks.workunit.client.0.smithi123.stdout:0/558: mkdir d6/dd/d31/d35/d3a/db9 0 2024-03-30T04:28:11.774 INFO:tasks.workunit.client.1.smithi135.stdout:1/100: creat d16/d1a/f22 x:0 0 0 2024-03-30T04:28:11.775 INFO:tasks.workunit.client.1.smithi135.stdout:7/165: symlink d4/db/de/l3e 0 2024-03-30T04:28:11.775 INFO:tasks.workunit.client.1.smithi135.stdout:1/101: chown d16/d19/d20 2575 1 2024-03-30T04:28:11.776 INFO:tasks.workunit.client.1.smithi135.stdout:7/166: stat d4/db/f26 0 2024-03-30T04:28:11.776 INFO:tasks.workunit.client.1.smithi135.stdout:2/75: getdents d4/d6 0 2024-03-30T04:28:11.778 INFO:tasks.workunit.client.1.smithi135.stdout:1/102: creat d16/d1a/f23 x:0 0 0 2024-03-30T04:28:11.781 INFO:tasks.workunit.client.0.smithi123.stdout:4/483: link d2/d4/d7/d1a/c99 d2/d4/d2c/d42/ca9 0 2024-03-30T04:28:11.781 INFO:tasks.workunit.client.1.smithi135.stdout:7/167: chown d4/db/c22 3796330 1 2024-03-30T04:28:11.783 INFO:tasks.workunit.client.1.smithi135.stdout:1/103: unlink fd 0 2024-03-30T04:28:11.783 INFO:tasks.workunit.client.1.smithi135.stdout:2/76: link d4/c10 d4/d6/c15 0 2024-03-30T04:28:11.784 INFO:tasks.workunit.client.1.smithi135.stdout:2/77: fsync d4/f5 0 2024-03-30T04:28:11.787 INFO:tasks.workunit.client.0.smithi123.stdout:5/476: dwrite dd/d19/d3e/f63 [4194304,4194304] 0 2024-03-30T04:28:11.787 INFO:tasks.workunit.client.1.smithi135.stdout:2/78: write d4/f7 [14627,3679] 0 2024-03-30T04:28:11.788 INFO:tasks.workunit.client.1.smithi135.stdout:7/168: rmdir d4/db/d1a/d2a/d2c 39 2024-03-30T04:28:11.791 INFO:tasks.workunit.client.0.smithi123.stdout:0/559: dread d6/dd/d15/d98/d4e/d69/f81 [0,4194304] 0 2024-03-30T04:28:11.792 INFO:tasks.workunit.client.0.smithi123.stdout:5/477: creat dd/d19/d1f/d3c/f97 x:0 0 0 2024-03-30T04:28:11.793 INFO:tasks.workunit.client.1.smithi135.stdout:7/169: truncate d4/db/de/f2b 784140 0 2024-03-30T04:28:11.794 INFO:tasks.workunit.client.0.smithi123.stdout:5/478: dread - dd/d19/d1f/d68/f6b zero size 2024-03-30T04:28:11.794 INFO:tasks.workunit.client.0.smithi123.stdout:0/560: rmdir d6/dd/d15/d98/d4e/d69/d6e 39 2024-03-30T04:28:11.798 INFO:tasks.workunit.client.0.smithi123.stdout:0/561: write d6/dd/d18/d2f/f85 [991556,101183] 0 2024-03-30T04:28:11.799 INFO:tasks.workunit.client.0.smithi123.stdout:5/479: creat dd/d19/d1f/d3c/d72/d8c/f98 x:0 0 0 2024-03-30T04:28:11.800 INFO:tasks.workunit.client.1.smithi135.stdout:7/170: mknod d4/db/c3f 0 2024-03-30T04:28:11.800 INFO:tasks.workunit.client.1.smithi135.stdout:2/79: truncate d4/f5 8784411 0 2024-03-30T04:28:11.805 INFO:tasks.workunit.client.0.smithi123.stdout:0/562: getdents d6/dd/d15/d28/d4b/d8a 0 2024-03-30T04:28:11.806 INFO:tasks.workunit.client.1.smithi135.stdout:2/80: rmdir d4/d6 39 2024-03-30T04:28:11.809 INFO:tasks.workunit.client.0.smithi123.stdout:0/563: rename d6/cb to d6/d52/cba 0 2024-03-30T04:28:11.810 INFO:tasks.workunit.client.0.smithi123.stdout:4/484: dread d2/d4/d2c/d42/d35/d5d/d6f/f8a [0,4194304] 0 2024-03-30T04:28:11.811 INFO:tasks.workunit.client.1.smithi135.stdout:1/104: dwrite d16/d1a/f23 [0,4194304] 0 2024-03-30T04:28:11.812 INFO:tasks.workunit.client.1.smithi135.stdout:3/87: sync 2024-03-30T04:28:11.812 INFO:tasks.workunit.client.1.smithi135.stdout:9/68: sync 2024-03-30T04:28:11.813 INFO:tasks.workunit.client.0.smithi123.stdout:0/564: chown d6/dd/d18/d2f/f58 1576 1 2024-03-30T04:28:11.819 INFO:tasks.workunit.client.1.smithi135.stdout:2/81: readlink d4/da/dd/le 0 2024-03-30T04:28:11.819 INFO:tasks.workunit.client.1.smithi135.stdout:8/130: dwrite d6/d8/f1d [0,4194304] 0 2024-03-30T04:28:11.823 INFO:tasks.workunit.client.1.smithi135.stdout:8/131: fsync f5 0 2024-03-30T04:28:11.823 INFO:tasks.workunit.client.1.smithi135.stdout:7/171: truncate d4/f11 3355849 0 2024-03-30T04:28:11.824 INFO:tasks.workunit.client.1.smithi135.stdout:3/88: rmdir d3/d10/d15 0 2024-03-30T04:28:11.828 INFO:tasks.workunit.client.1.smithi135.stdout:8/132: truncate d6/d8/f20 150974 0 2024-03-30T04:28:11.829 INFO:tasks.workunit.client.0.smithi123.stdout:5/480: dread dd/d19/f87 [0,4194304] 0 2024-03-30T04:28:11.832 INFO:tasks.workunit.client.1.smithi135.stdout:1/105: getdents d16/d19 0 2024-03-30T04:28:11.833 INFO:tasks.workunit.client.0.smithi123.stdout:4/485: dwrite d2/d34/f88 [0,4194304] 0 2024-03-30T04:28:11.834 INFO:tasks.workunit.client.0.smithi123.stdout:5/481: mkdir dd/d19/d5f/d99 0 2024-03-30T04:28:11.839 INFO:tasks.workunit.client.1.smithi135.stdout:2/82: write d4/d6/f11 [418011,15393] 0 2024-03-30T04:28:11.839 INFO:tasks.workunit.client.1.smithi135.stdout:3/89: creat d3/d10/f18 x:0 0 0 2024-03-30T04:28:11.842 INFO:tasks.workunit.client.0.smithi123.stdout:3/582: getdents d13/d24/d31/d23/d3a/d3c 0 2024-03-30T04:28:11.845 INFO:tasks.workunit.client.0.smithi123.stdout:4/486: write d2/f20 [1989798,121502] 0 2024-03-30T04:28:11.846 INFO:tasks.workunit.client.1.smithi135.stdout:8/133: creat d6/d8/d15/f22 x:0 0 0 2024-03-30T04:28:11.847 INFO:tasks.workunit.client.0.smithi123.stdout:3/583: mkdir d13/d24/d31/d23/d3a/d3c/db8 0 2024-03-30T04:28:11.848 INFO:tasks.workunit.client.0.smithi123.stdout:4/487: readlink d2/l5a 0 2024-03-30T04:28:11.849 INFO:tasks.workunit.client.1.smithi135.stdout:1/106: symlink d16/d19/l24 0 2024-03-30T04:28:11.849 INFO:tasks.workunit.client.1.smithi135.stdout:8/134: stat d6/fa 0 2024-03-30T04:28:11.851 INFO:tasks.workunit.client.1.smithi135.stdout:2/83: rename d4/d6/f11 to d4/d6/f16 0 2024-03-30T04:28:11.854 INFO:tasks.workunit.client.1.smithi135.stdout:9/69: dread f1 [0,4194304] 0 2024-03-30T04:28:11.854 INFO:tasks.workunit.client.1.smithi135.stdout:8/135: write f5 [2737397,129498] 0 2024-03-30T04:28:11.855 INFO:tasks.workunit.client.1.smithi135.stdout:3/90: creat d3/db/f19 x:0 0 0 2024-03-30T04:28:11.856 INFO:tasks.workunit.client.0.smithi123.stdout:4/488: rename d2/d4/d7/f4b to d2/d4/d2c/d42/d35/faa 0 2024-03-30T04:28:11.859 INFO:tasks.workunit.client.1.smithi135.stdout:1/107: creat d16/f25 x:0 0 0 2024-03-30T04:28:11.860 INFO:tasks.workunit.client.0.smithi123.stdout:4/489: mkdir d2/dab 0 2024-03-30T04:28:11.861 INFO:tasks.workunit.client.1.smithi135.stdout:9/70: fdatasync f6 0 2024-03-30T04:28:11.863 INFO:tasks.workunit.client.1.smithi135.stdout:8/136: mkdir d6/d23 0 2024-03-30T04:28:11.864 INFO:tasks.workunit.client.0.smithi123.stdout:5/482: rmdir dd 39 2024-03-30T04:28:11.868 INFO:tasks.workunit.client.0.smithi123.stdout:4/490: getdents d2/d26 0 2024-03-30T04:28:11.869 INFO:tasks.workunit.client.1.smithi135.stdout:1/108: unlink c12 0 2024-03-30T04:28:11.869 INFO:tasks.workunit.client.1.smithi135.stdout:7/172: dwrite d4/db/f14 [4194304,4194304] 0 2024-03-30T04:28:11.869 INFO:tasks.workunit.client.0.smithi123.stdout:5/483: fsync dd/d19/f28 0 2024-03-30T04:28:11.870 INFO:tasks.workunit.client.1.smithi135.stdout:2/84: dread d4/d6/f16 [0,4194304] 0 2024-03-30T04:28:11.872 INFO:tasks.workunit.client.1.smithi135.stdout:3/91: link d3/d6/f12 d3/db/f1a 0 2024-03-30T04:28:11.873 INFO:tasks.workunit.client.1.smithi135.stdout:7/173: chown d4/l27 1989694 1 2024-03-30T04:28:11.874 INFO:tasks.workunit.client.0.smithi123.stdout:4/491: mknod d2/d4/d2c/d42/d80/da5/cac 0 2024-03-30T04:28:11.874 INFO:tasks.workunit.client.1.smithi135.stdout:8/137: dread d6/d8/f20 [0,4194304] 0 2024-03-30T04:28:11.874 INFO:tasks.workunit.client.1.smithi135.stdout:3/92: stat d3/d6 0 2024-03-30T04:28:11.875 INFO:tasks.workunit.client.0.smithi123.stdout:5/484: creat dd/d19/d1f/d3c/d72/d8c/f9a x:0 0 0 2024-03-30T04:28:11.876 INFO:tasks.workunit.client.1.smithi135.stdout:3/93: readlink l1 0 2024-03-30T04:28:11.876 INFO:tasks.workunit.client.0.smithi123.stdout:0/565: dwrite d6/f24 [0,4194304] 0 2024-03-30T04:28:11.877 INFO:tasks.workunit.client.1.smithi135.stdout:9/71: creat d7/f12 x:0 0 0 2024-03-30T04:28:11.879 INFO:tasks.workunit.client.0.smithi123.stdout:9/469: dread d6/d58/f6a [4194304,4194304] 0 2024-03-30T04:28:11.880 INFO:tasks.workunit.client.1.smithi135.stdout:3/94: chown d3/d10 103608 1 2024-03-30T04:28:11.880 INFO:tasks.workunit.client.0.smithi123.stdout:9/470: read d6/d58/f6a [7602644,30887] 0 2024-03-30T04:28:11.880 INFO:tasks.workunit.client.0.smithi123.stdout:0/566: dread - d6/dd/d15/d98/d5d/f83 zero size 2024-03-30T04:28:11.881 INFO:tasks.workunit.client.1.smithi135.stdout:9/72: truncate d7/f11 388010 0 2024-03-30T04:28:11.881 INFO:tasks.workunit.client.0.smithi123.stdout:5/485: rmdir dd/d19/d1f/d35/d3a 39 2024-03-30T04:28:11.882 INFO:tasks.workunit.client.0.smithi123.stdout:9/471: chown d6/d7c/d82/l8c 80 1 2024-03-30T04:28:11.884 INFO:tasks.workunit.client.0.smithi123.stdout:0/567: mknod d6/dd/d18/cbb 0 2024-03-30T04:28:11.886 INFO:tasks.workunit.client.0.smithi123.stdout:9/472: symlink d6/d1f/d72/la7 0 2024-03-30T04:28:11.888 INFO:tasks.workunit.client.0.smithi123.stdout:0/568: read d6/dd/d18/d2f/f58 [1952472,54042] 0 2024-03-30T04:28:11.888 INFO:tasks.workunit.client.1.smithi135.stdout:8/138: write d6/d8/f1c [73413,26375] 0 2024-03-30T04:28:11.890 INFO:tasks.workunit.client.1.smithi135.stdout:1/109: mkdir d16/d19/d20/d26 0 2024-03-30T04:28:11.892 INFO:tasks.workunit.client.1.smithi135.stdout:7/174: rename d4/db/de/d15/d38 to d4/db/de/d15/d40 0 2024-03-30T04:28:11.893 INFO:tasks.workunit.client.1.smithi135.stdout:2/85: getdents d4/da/dd/d12 0 2024-03-30T04:28:11.893 INFO:tasks.workunit.client.0.smithi123.stdout:0/569: unlink d6/dd/d31/d35/l6b 0 2024-03-30T04:28:11.895 INFO:tasks.workunit.client.1.smithi135.stdout:2/86: read - d4/d6/ff zero size 2024-03-30T04:28:11.898 INFO:tasks.workunit.client.0.smithi123.stdout:3/584: dread d13/d24/f70 [0,4194304] 0 2024-03-30T04:28:11.900 INFO:tasks.workunit.client.1.smithi135.stdout:2/87: write d4/d6/f16 [552584,15425] 0 2024-03-30T04:28:11.900 INFO:tasks.workunit.client.1.smithi135.stdout:8/139: dread f4 [8388608,4194304] 0 2024-03-30T04:28:11.901 INFO:tasks.workunit.client.1.smithi135.stdout:1/110: rename l11 to d16/d19/d20/l27 0 2024-03-30T04:28:11.901 INFO:tasks.workunit.client.0.smithi123.stdout:0/570: creat d6/dd/d15/d98/d4e/d69/fbc x:0 0 0 2024-03-30T04:28:11.902 INFO:tasks.workunit.client.0.smithi123.stdout:3/585: mkdir d13/d24/d39/d79/db9 0 2024-03-30T04:28:11.902 INFO:tasks.workunit.client.0.smithi123.stdout:5/486: dwrite dd/d19/d1f/d35/d3a/d79/f96 [0,4194304] 0 2024-03-30T04:28:11.903 INFO:tasks.workunit.client.0.smithi123.stdout:0/571: fdatasync d6/d52/f73 0 2024-03-30T04:28:11.905 INFO:tasks.workunit.client.0.smithi123.stdout:5/487: rmdir dd/d19/d5f 39 2024-03-30T04:28:11.906 INFO:tasks.workunit.client.0.smithi123.stdout:3/586: truncate d13/d24/d31/f75 1101329 0 2024-03-30T04:28:11.906 INFO:tasks.workunit.client.1.smithi135.stdout:7/175: rename d4/db/d1a/d2a/c3c to d4/db/d1a/c41 0 2024-03-30T04:28:11.908 INFO:tasks.workunit.client.1.smithi135.stdout:1/111: symlink d16/l28 0 2024-03-30T04:28:11.908 INFO:tasks.workunit.client.1.smithi135.stdout:1/112: stat f7 0 2024-03-30T04:28:11.909 INFO:tasks.workunit.client.0.smithi123.stdout:0/572: mknod d6/dd/d18/d2f/cbd 0 2024-03-30T04:28:11.909 INFO:tasks.workunit.client.1.smithi135.stdout:2/88: link d4/c9 d4/da/c17 0 2024-03-30T04:28:11.910 INFO:tasks.workunit.client.1.smithi135.stdout:1/113: chown d16/d19/d20/d26 234068 1 2024-03-30T04:28:11.911 INFO:tasks.workunit.client.0.smithi123.stdout:3/587: mknod d13/d24/d31/d23/cba 0 2024-03-30T04:28:11.913 INFO:tasks.workunit.client.1.smithi135.stdout:2/89: write d4/da/dd/f13 [301635,24904] 0 2024-03-30T04:28:11.917 INFO:tasks.workunit.client.1.smithi135.stdout:1/114: mkdir d16/d29 0 2024-03-30T04:28:11.917 INFO:tasks.workunit.client.0.smithi123.stdout:9/473: write d6/d1f/f34 [42551,63466] 0 2024-03-30T04:28:11.918 INFO:tasks.workunit.client.0.smithi123.stdout:0/573: mknod d6/dd/d15/d98/d5d/d7f/cbe 0 2024-03-30T04:28:11.919 INFO:tasks.workunit.client.0.smithi123.stdout:9/474: write d6/d1f/f41 [3379206,105888] 0 2024-03-30T04:28:11.919 INFO:tasks.workunit.client.1.smithi135.stdout:9/73: getdents d7 0 2024-03-30T04:28:11.921 INFO:tasks.workunit.client.1.smithi135.stdout:2/90: write d4/d6/f16 [46582,90946] 0 2024-03-30T04:28:11.923 INFO:tasks.workunit.client.0.smithi123.stdout:7/496: sync 2024-03-30T04:28:11.923 INFO:tasks.workunit.client.0.smithi123.stdout:6/602: sync 2024-03-30T04:28:11.923 INFO:tasks.workunit.client.0.smithi123.stdout:1/630: sync 2024-03-30T04:28:11.924 INFO:tasks.workunit.client.1.smithi135.stdout:7/176: getdents d4/db 0 2024-03-30T04:28:11.924 INFO:tasks.workunit.client.1.smithi135.stdout:3/95: truncate d3/d6/f7 1113285 0 2024-03-30T04:28:11.925 INFO:tasks.workunit.client.0.smithi123.stdout:7/497: readlink d2/da/d11/d34/l3f 0 2024-03-30T04:28:11.925 INFO:tasks.workunit.client.1.smithi135.stdout:1/115: mknod d16/c2a 0 2024-03-30T04:28:11.926 INFO:tasks.workunit.client.0.smithi123.stdout:3/588: rmdir d13 39 2024-03-30T04:28:11.927 INFO:tasks.workunit.client.0.smithi123.stdout:5/488: truncate dd/d41/f90 1438334 0 2024-03-30T04:28:11.929 INFO:tasks.workunit.client.0.smithi123.stdout:7/498: chown d2/da/d11/d61/d55/d94 3 1 2024-03-30T04:28:11.931 INFO:tasks.workunit.client.1.smithi135.stdout:2/91: rmdir d4/d6 39 2024-03-30T04:28:11.932 INFO:tasks.workunit.client.0.smithi123.stdout:5/489: chown dd/d19/f2b 144049637 1 2024-03-30T04:28:11.932 INFO:tasks.workunit.client.0.smithi123.stdout:9/475: symlink d6/d1f/da3/la8 0 2024-03-30T04:28:11.935 INFO:tasks.workunit.client.0.smithi123.stdout:6/603: unlink d10/d82/d8e/dc7/f6f 0 2024-03-30T04:28:11.935 INFO:tasks.workunit.client.1.smithi135.stdout:7/177: truncate d4/db/d1a/d2a/f37 366099 0 2024-03-30T04:28:11.937 INFO:tasks.workunit.client.1.smithi135.stdout:2/92: write d4/f7 [389750,5016] 0 2024-03-30T04:28:11.937 INFO:tasks.workunit.client.1.smithi135.stdout:8/140: dwrite d6/d8/f16 [0,4194304] 0 2024-03-30T04:28:11.937 INFO:tasks.workunit.client.0.smithi123.stdout:7/499: write d2/da/d22/f2f [667683,45060] 0 2024-03-30T04:28:11.939 INFO:tasks.workunit.client.1.smithi135.stdout:2/93: readlink d4/da/dd/le 0 2024-03-30T04:28:11.940 INFO:tasks.workunit.client.1.smithi135.stdout:3/96: rename d3/d6/f7 to d3/d10/d16/f1b 0 2024-03-30T04:28:11.942 INFO:tasks.workunit.client.1.smithi135.stdout:8/141: rename d6 to d6/d23/d24 22 2024-03-30T04:28:11.943 INFO:tasks.workunit.client.0.smithi123.stdout:6/604: unlink d10/d82/dc1/f95 0 2024-03-30T04:28:11.944 INFO:tasks.workunit.client.1.smithi135.stdout:1/116: creat d16/d19/f2b x:0 0 0 2024-03-30T04:28:11.945 INFO:tasks.workunit.client.0.smithi123.stdout:0/574: dwrite d6/dd/d15/d28/f78 [0,4194304] 0 2024-03-30T04:28:11.947 INFO:tasks.workunit.client.1.smithi135.stdout:2/94: rename d4/da to d4/da/d18 22 2024-03-30T04:28:11.949 INFO:tasks.workunit.client.1.smithi135.stdout:3/97: read d3/d6/f12 [432090,130772] 0 2024-03-30T04:28:11.951 INFO:tasks.workunit.client.0.smithi123.stdout:6/605: readlink d10/d82/dc1/d45/l54 0 2024-03-30T04:28:11.952 INFO:tasks.workunit.client.1.smithi135.stdout:1/117: chown d16/d1a/f22 2 1 2024-03-30T04:28:11.954 INFO:tasks.workunit.client.1.smithi135.stdout:8/142: write d6/d8/d15/f22 [509113,58233] 0 2024-03-30T04:28:11.955 INFO:tasks.workunit.client.1.smithi135.stdout:7/178: dread d4/db/d1a/d2a/f37 [0,4194304] 0 2024-03-30T04:28:11.955 INFO:tasks.workunit.client.1.smithi135.stdout:3/98: chown l1 1587111745 1 2024-03-30T04:28:11.956 INFO:tasks.workunit.client.1.smithi135.stdout:2/95: write d4/d6/f16 [871517,20340] 0 2024-03-30T04:28:11.957 INFO:tasks.workunit.client.1.smithi135.stdout:9/74: dwrite d7/ff [0,4194304] 0 2024-03-30T04:28:11.958 INFO:tasks.workunit.client.0.smithi123.stdout:6/606: write d10/d20/da2/fc3 [336467,116809] 0 2024-03-30T04:28:11.958 INFO:tasks.workunit.client.0.smithi123.stdout:7/500: creat d2/da/d11/d34/d85/fae x:0 0 0 2024-03-30T04:28:11.959 INFO:tasks.workunit.client.1.smithi135.stdout:3/99: dread - d3/f11 zero size 2024-03-30T04:28:11.962 INFO:tasks.workunit.client.1.smithi135.stdout:1/118: mkdir d16/d29/d2c 0 2024-03-30T04:28:11.965 INFO:tasks.workunit.client.0.smithi123.stdout:9/476: dwrite d6/f10 [4194304,4194304] 0 2024-03-30T04:28:11.966 INFO:tasks.workunit.client.0.smithi123.stdout:5/490: rename dd/d19/d3e/d5b/d62/f71 to dd/d19/d1f/d35/d3a/d79/f9b 0 2024-03-30T04:28:11.966 INFO:tasks.workunit.client.1.smithi135.stdout:7/179: unlink d4/db/d1a/d2a/f32 0 2024-03-30T04:28:11.969 INFO:tasks.workunit.client.1.smithi135.stdout:3/100: dread d3/d6/fa [0,4194304] 0 2024-03-30T04:28:11.970 INFO:tasks.workunit.client.1.smithi135.stdout:1/119: symlink d16/d29/l2d 0 2024-03-30T04:28:11.970 INFO:tasks.workunit.client.0.smithi123.stdout:3/589: dwrite d13/d24/d31/d23/f6c [0,4194304] 0 2024-03-30T04:28:11.974 INFO:tasks.workunit.client.1.smithi135.stdout:7/180: readlink d4/db/de/d15/l1e 0 2024-03-30T04:28:11.975 INFO:tasks.workunit.client.1.smithi135.stdout:8/143: link d6/d8/f1c d6/f25 0 2024-03-30T04:28:11.976 INFO:tasks.workunit.client.0.smithi123.stdout:5/491: truncate f2 2228042 0 2024-03-30T04:28:11.976 INFO:tasks.workunit.client.1.smithi135.stdout:8/144: stat f4 0 2024-03-30T04:28:11.978 INFO:tasks.workunit.client.0.smithi123.stdout:0/575: dread d6/dd/d18/d2f/f85 [0,4194304] 0 2024-03-30T04:28:11.979 INFO:tasks.workunit.client.1.smithi135.stdout:2/96: rename d4/c9 to d4/c19 0 2024-03-30T04:28:11.982 INFO:tasks.workunit.client.1.smithi135.stdout:6/39: sync 2024-03-30T04:28:11.987 INFO:tasks.workunit.client.1.smithi135.stdout:8/145: mknod d6/d8/c26 0 2024-03-30T04:28:11.987 INFO:tasks.workunit.client.0.smithi123.stdout:5/492: rmdir dd/d19/d1f/d3c/d72/d8c 39 2024-03-30T04:28:11.988 INFO:tasks.workunit.client.1.smithi135.stdout:7/181: mkdir d4/db/d1a/d2a/d2c/d2e/d42 0 2024-03-30T04:28:11.990 INFO:tasks.workunit.client.0.smithi123.stdout:1/631: dwrite db/dd/d42/d46/d39/f92 [0,4194304] 0 2024-03-30T04:28:11.991 INFO:tasks.workunit.client.1.smithi135.stdout:9/75: write f6 [326158,111108] 0 2024-03-30T04:28:11.993 INFO:tasks.workunit.client.0.smithi123.stdout:5/493: chown dd/c7e 1989389800 1 2024-03-30T04:28:11.994 INFO:tasks.workunit.client.1.smithi135.stdout:7/182: fdatasync d4/db/de/f28 0 2024-03-30T04:28:11.995 INFO:tasks.workunit.client.1.smithi135.stdout:8/146: chown d6/f12 409266273 1 2024-03-30T04:28:11.996 INFO:tasks.workunit.client.0.smithi123.stdout:0/576: chown d6/dd/d18/d2f/c60 1959486448 1 2024-03-30T04:28:11.998 INFO:tasks.workunit.client.0.smithi123.stdout:3/590: rename cb to d13/d24/d31/d23/d3a/d3c/d4e/d5d/cbb 0 2024-03-30T04:28:12.000 INFO:tasks.workunit.client.1.smithi135.stdout:7/183: dread - d4/db/de/f28 zero size 2024-03-30T04:28:12.000 INFO:tasks.workunit.client.1.smithi135.stdout:9/76: write d7/f12 [721073,62296] 0 2024-03-30T04:28:12.002 INFO:tasks.workunit.client.1.smithi135.stdout:3/101: dread d3/d10/f13 [0,4194304] 0 2024-03-30T04:28:12.006 INFO:tasks.workunit.client.0.smithi123.stdout:5/494: chown dd/d19/d3e/d5b/d62/f7a 125327785 1 2024-03-30T04:28:12.007 INFO:tasks.workunit.client.1.smithi135.stdout:1/120: dwrite d16/f25 [0,4194304] 0 2024-03-30T04:28:12.012 INFO:tasks.workunit.client.0.smithi123.stdout:7/501: dwrite d2/da/d11/d34/f70 [0,4194304] 0 2024-03-30T04:28:12.012 INFO:tasks.workunit.client.0.smithi123.stdout:9/477: dwrite d6/d1f/d45/d4c/f54 [0,4194304] 0 2024-03-30T04:28:12.015 INFO:tasks.workunit.client.1.smithi135.stdout:3/102: creat d3/d17/f1c x:0 0 0 2024-03-30T04:28:12.016 INFO:tasks.workunit.client.0.smithi123.stdout:3/591: symlink d13/d24/d31/d23/d63/lbc 0 2024-03-30T04:28:12.020 INFO:tasks.workunit.client.0.smithi123.stdout:1/632: write db/dd/f8e [738698,113639] 0 2024-03-30T04:28:12.021 INFO:tasks.workunit.client.1.smithi135.stdout:6/40: dread d2/f4 [0,4194304] 0 2024-03-30T04:28:12.030 INFO:tasks.workunit.client.0.smithi123.stdout:3/592: creat d13/d24/fbd x:0 0 0 2024-03-30T04:28:12.031 INFO:tasks.workunit.client.1.smithi135.stdout:2/97: dwrite d4/d6/f16 [0,4194304] 0 2024-03-30T04:28:12.031 INFO:tasks.workunit.client.1.smithi135.stdout:1/121: chown d16/c17 78274698 1 2024-03-30T04:28:12.031 INFO:tasks.workunit.client.0.smithi123.stdout:5/495: mkdir dd/d19/d1f/d3c/d72/d8c/d9c 0 2024-03-30T04:28:12.032 INFO:tasks.workunit.client.0.smithi123.stdout:1/633: truncate db/f20 3378474 0 2024-03-30T04:28:12.033 INFO:tasks.workunit.client.1.smithi135.stdout:7/184: chown d4/db/d1a/d2a/d2c/d2e/d42 243013 1 2024-03-30T04:28:12.035 INFO:tasks.workunit.client.0.smithi123.stdout:5/496: fsync dd/d19/d3e/d5b/d62/f7a 0 2024-03-30T04:28:12.035 INFO:tasks.workunit.client.0.smithi123.stdout:0/577: getdents d6/dd/d15/d98/d4e 0 2024-03-30T04:28:12.040 INFO:tasks.workunit.client.0.smithi123.stdout:1/634: unlink db/dd/d42/d46/d5d/d7a/fb2 0 2024-03-30T04:28:12.041 INFO:tasks.workunit.client.0.smithi123.stdout:5/497: rmdir dd/d19/d1f/d35/d3a/d79 39 2024-03-30T04:28:12.045 INFO:tasks.workunit.client.1.smithi135.stdout:8/147: dwrite d6/d8/f16 [4194304,4194304] 0 2024-03-30T04:28:12.045 INFO:tasks.workunit.client.0.smithi123.stdout:1/635: mkdir db/dd/d5f/d6d/d85/dd4 0 2024-03-30T04:28:12.046 INFO:tasks.workunit.client.1.smithi135.stdout:3/103: getdents d3/d17 0 2024-03-30T04:28:12.050 INFO:tasks.workunit.client.1.smithi135.stdout:9/77: dread f1 [0,4194304] 0 2024-03-30T04:28:12.051 INFO:tasks.workunit.client.0.smithi123.stdout:1/636: rmdir db/dd/d42/d46/d56/d5c 39 2024-03-30T04:28:12.053 INFO:tasks.workunit.client.0.smithi123.stdout:1/637: chown db/d1b/c38 2144227 1 2024-03-30T04:28:12.053 INFO:tasks.workunit.client.1.smithi135.stdout:7/185: creat d4/db/d1a/d2a/f43 x:0 0 0 2024-03-30T04:28:12.054 INFO:tasks.workunit.client.0.smithi123.stdout:1/638: readlink l4 0 2024-03-30T04:28:12.055 INFO:tasks.workunit.client.0.smithi123.stdout:1/639: chown db/dd/d42 313 1 2024-03-30T04:28:12.056 INFO:tasks.workunit.client.1.smithi135.stdout:8/148: link f0 d6/d23/f27 0 2024-03-30T04:28:12.057 INFO:tasks.workunit.client.0.smithi123.stdout:6/607: dread d10/d82/dc1/f8b [0,4194304] 0 2024-03-30T04:28:12.066 INFO:tasks.workunit.client.1.smithi135.stdout:8/149: symlink d6/d8/d15/l28 0 2024-03-30T04:28:12.066 INFO:tasks.workunit.client.0.smithi123.stdout:6/608: creat d10/d82/d8e/fd1 x:0 0 0 2024-03-30T04:28:12.069 INFO:tasks.workunit.client.0.smithi123.stdout:6/609: chown d10/d82/dc1/d6a/d7a/d99/dba 1370739 1 2024-03-30T04:28:12.073 INFO:tasks.workunit.client.1.smithi135.stdout:7/186: fsync d4/db/d1a/d2a/d2c/f39 0 2024-03-30T04:28:12.074 INFO:tasks.workunit.client.0.smithi123.stdout:3/593: truncate d13/d24/d31/d23/d3a/d3c/fa8 880675 0 2024-03-30T04:28:12.074 INFO:tasks.workunit.client.0.smithi123.stdout:7/502: dread d2/f13 [0,4194304] 0 2024-03-30T04:28:12.076 INFO:tasks.workunit.client.0.smithi123.stdout:6/610: mkdir d10/d82/dc1/dd2 0 2024-03-30T04:28:12.083 INFO:tasks.workunit.client.1.smithi135.stdout:6/41: dwrite d2/f4 [4194304,4194304] 0 2024-03-30T04:28:12.084 INFO:tasks.workunit.client.0.smithi123.stdout:6/611: creat d10/d82/dc1/d45/fd3 x:0 0 0 2024-03-30T04:28:12.085 INFO:tasks.workunit.client.1.smithi135.stdout:7/187: chown d4/l23 55 1 2024-03-30T04:28:12.085 INFO:tasks.workunit.client.1.smithi135.stdout:2/98: dwrite d4/da/dd/f13 [0,4194304] 0 2024-03-30T04:28:12.086 INFO:tasks.workunit.client.1.smithi135.stdout:8/150: creat d6/d8/d14/f29 x:0 0 0 2024-03-30T04:28:12.087 INFO:tasks.workunit.client.1.smithi135.stdout:6/42: stat d2/f4 0 2024-03-30T04:28:12.093 INFO:tasks.workunit.client.0.smithi123.stdout:1/640: dwrite db/dd/d5f/d6d/d85/d89/fa5 [0,4194304] 0 2024-03-30T04:28:12.095 INFO:tasks.workunit.client.0.smithi123.stdout:5/498: dread fb [0,4194304] 0 2024-03-30T04:28:12.097 INFO:tasks.workunit.client.0.smithi123.stdout:6/612: getdents d10/d82/dc1/d6a 0 2024-03-30T04:28:12.099 INFO:tasks.workunit.client.1.smithi135.stdout:6/43: creat d2/f8 x:0 0 0 2024-03-30T04:28:12.101 INFO:tasks.workunit.client.0.smithi123.stdout:5/499: chown dd/d19/d5f/d99 90766049 1 2024-03-30T04:28:12.102 INFO:tasks.workunit.client.0.smithi123.stdout:5/500: readlink l4 0 2024-03-30T04:28:12.102 INFO:tasks.workunit.client.1.smithi135.stdout:3/104: dwrite d3/f11 [0,4194304] 0 2024-03-30T04:28:12.107 INFO:tasks.workunit.client.0.smithi123.stdout:0/578: dread d6/dd/d15/fb3 [0,4194304] 0 2024-03-30T04:28:12.107 INFO:tasks.workunit.client.0.smithi123.stdout:1/641: creat db/dd/d5f/d6d/fd5 x:0 0 0 2024-03-30T04:28:12.113 INFO:tasks.workunit.client.1.smithi135.stdout:6/44: creat d2/f9 x:0 0 0 2024-03-30T04:28:12.114 INFO:tasks.workunit.client.1.smithi135.stdout:9/78: dwrite d7/f11 [0,4194304] 0 2024-03-30T04:28:12.114 INFO:tasks.workunit.client.0.smithi123.stdout:6/613: truncate d10/d82/dc1/d45/f9c 1181388 0 2024-03-30T04:28:12.114 INFO:tasks.workunit.client.0.smithi123.stdout:1/642: dread - db/dd/d42/d46/d51/d72/fc8 zero size 2024-03-30T04:28:12.116 INFO:tasks.workunit.client.1.smithi135.stdout:3/105: truncate d3/f9 359270 0 2024-03-30T04:28:12.117 INFO:tasks.workunit.client.1.smithi135.stdout:9/79: fdatasync f6 0 2024-03-30T04:28:12.118 INFO:tasks.workunit.client.1.smithi135.stdout:3/106: readlink l1 0 2024-03-30T04:28:12.120 INFO:tasks.workunit.client.0.smithi123.stdout:3/594: dread d13/fa1 [0,4194304] 0 2024-03-30T04:28:12.121 INFO:tasks.workunit.client.1.smithi135.stdout:3/107: chown d3/d10 775773 1 2024-03-30T04:28:12.125 INFO:tasks.workunit.client.0.smithi123.stdout:5/501: dread dd/f15 [0,4194304] 0 2024-03-30T04:28:12.125 INFO:tasks.workunit.client.0.smithi123.stdout:6/614: creat d10/d82/dc1/d6a/d7a/d9f/fd4 x:0 0 0 2024-03-30T04:28:12.126 INFO:tasks.workunit.client.1.smithi135.stdout:3/108: write d3/d10/f18 [605095,16731] 0 2024-03-30T04:28:12.127 INFO:tasks.workunit.client.1.smithi135.stdout:7/188: dread d4/db/de/d15/f31 [0,4194304] 0 2024-03-30T04:28:12.128 INFO:tasks.workunit.client.0.smithi123.stdout:5/502: fsync dd/fe 0 2024-03-30T04:28:12.129 INFO:tasks.workunit.client.0.smithi123.stdout:6/615: chown d10/fb2 96952 1 2024-03-30T04:28:12.130 INFO:tasks.workunit.client.1.smithi135.stdout:2/99: dwrite d4/d6/f16 [0,4194304] 0 2024-03-30T04:28:12.132 INFO:tasks.workunit.client.1.smithi135.stdout:8/151: dwrite d6/d8/d14/fe [0,4194304] 0 2024-03-30T04:28:12.132 INFO:tasks.workunit.client.1.smithi135.stdout:2/100: chown d4/da/dd 0 1 2024-03-30T04:28:12.133 INFO:tasks.workunit.client.0.smithi123.stdout:2/617: sync 2024-03-30T04:28:12.133 INFO:tasks.workunit.client.1.smithi135.stdout:8/152: readlink l3 0 2024-03-30T04:28:12.134 INFO:tasks.workunit.client.0.smithi123.stdout:0/579: dread d6/dd/d15/fb3 [0,4194304] 0 2024-03-30T04:28:12.136 INFO:tasks.workunit.client.1.smithi135.stdout:3/109: rename d3/d10/f13 to d3/f1d 0 2024-03-30T04:28:12.137 INFO:tasks.workunit.client.1.smithi135.stdout:2/101: rmdir d4/da/dd 39 2024-03-30T04:28:12.137 INFO:tasks.workunit.client.1.smithi135.stdout:3/110: fsync d3/db/fe 0 2024-03-30T04:28:12.138 INFO:tasks.workunit.client.1.smithi135.stdout:7/189: getdents d4/db/d1a/d3d 0 2024-03-30T04:28:12.138 INFO:tasks.workunit.client.0.smithi123.stdout:7/503: dread d2/da/d11/d61/f72 [0,4194304] 0 2024-03-30T04:28:12.139 INFO:tasks.workunit.client.1.smithi135.stdout:8/153: mkdir d6/d8/d15/d2a 0 2024-03-30T04:28:12.140 INFO:tasks.workunit.client.1.smithi135.stdout:7/190: chown d4/c13 0 1 2024-03-30T04:28:12.141 INFO:tasks.workunit.client.1.smithi135.stdout:2/102: creat d4/d6/f1a x:0 0 0 2024-03-30T04:28:12.141 INFO:tasks.workunit.client.1.smithi135.stdout:3/111: creat d3/d10/d16/f1e x:0 0 0 2024-03-30T04:28:12.142 INFO:tasks.workunit.client.0.smithi123.stdout:2/618: mknod d0/d37/cce 0 2024-03-30T04:28:12.143 INFO:tasks.workunit.client.0.smithi123.stdout:1/643: rename db/dd/d5f/dae/ld3 to db/dd/d42/d8f/ld6 0 2024-03-30T04:28:12.143 INFO:tasks.workunit.client.0.smithi123.stdout:6/616: dread - d10/d82/d8e/dc7/f6c zero size 2024-03-30T04:28:12.145 INFO:tasks.workunit.client.1.smithi135.stdout:3/112: creat d3/d6/f1f x:0 0 0 2024-03-30T04:28:12.147 INFO:tasks.workunit.client.0.smithi123.stdout:7/504: mknod d2/da/d11/d34/d76/caf 0 2024-03-30T04:28:12.154 INFO:tasks.workunit.client.0.smithi123.stdout:7/505: write d2/f87 [1040017,76152] 0 2024-03-30T04:28:12.156 INFO:tasks.workunit.client.0.smithi123.stdout:0/580: symlink d6/dd/d15/lbf 0 2024-03-30T04:28:12.157 INFO:tasks.workunit.client.0.smithi123.stdout:7/506: chown d2/dc/f81 164410848 1 2024-03-30T04:28:12.158 INFO:tasks.workunit.client.1.smithi135.stdout:6/45: fsync d2/f9 0 2024-03-30T04:28:12.159 INFO:tasks.workunit.client.0.smithi123.stdout:7/507: dread - d2/da/d22/f73 zero size 2024-03-30T04:28:12.161 INFO:tasks.workunit.client.0.smithi123.stdout:1/644: mkdir db/dd/d42/d46/d51/dd7 0 2024-03-30T04:28:12.164 INFO:tasks.workunit.client.0.smithi123.stdout:1/645: chown db/dd/f48 1071261539 1 2024-03-30T04:28:12.165 INFO:tasks.workunit.client.1.smithi135.stdout:8/154: truncate d6/d8/f16 4854918 0 2024-03-30T04:28:12.166 INFO:tasks.workunit.client.1.smithi135.stdout:2/103: dwrite d4/d6/ff [0,4194304] 0 2024-03-30T04:28:12.167 INFO:tasks.workunit.client.0.smithi123.stdout:3/595: truncate d13/f14 2973169 0 2024-03-30T04:28:12.167 INFO:tasks.workunit.client.0.smithi123.stdout:5/503: write dd/f26 [657365,4293] 0 2024-03-30T04:28:12.168 INFO:tasks.workunit.client.0.smithi123.stdout:7/508: rmdir d2/d4d/d80/d98 39 2024-03-30T04:28:12.168 INFO:tasks.workunit.client.0.smithi123.stdout:1/646: chown db/dd/d42/f6f 472062 1 2024-03-30T04:28:12.170 INFO:tasks.workunit.client.1.smithi135.stdout:7/191: truncate d4/db/de/d15/f31 583010 0 2024-03-30T04:28:12.171 INFO:tasks.workunit.client.1.smithi135.stdout:8/155: creat d6/d8/d15/f2b x:0 0 0 2024-03-30T04:28:12.172 INFO:tasks.workunit.client.1.smithi135.stdout:3/113: dwrite d3/d10/d16/f1e [0,4194304] 0 2024-03-30T04:28:12.175 INFO:tasks.workunit.client.0.smithi123.stdout:7/509: write d2/dc/d1c/f7c [2581031,106572] 0 2024-03-30T04:28:12.176 INFO:tasks.workunit.client.1.smithi135.stdout:7/192: write d4/db/de/f30 [49464,2359] 0 2024-03-30T04:28:12.180 INFO:tasks.workunit.client.0.smithi123.stdout:3/596: mkdir d13/d24/d31/d23/d3a/d3c/d4e/dbe 0 2024-03-30T04:28:12.182 INFO:tasks.workunit.client.0.smithi123.stdout:5/504: mknod dd/d19/c9d 0 2024-03-30T04:28:12.182 INFO:tasks.workunit.client.1.smithi135.stdout:3/114: symlink d3/l20 0 2024-03-30T04:28:12.184 INFO:tasks.workunit.client.0.smithi123.stdout:0/581: dwrite d6/dd/d15/d98/d4e/d69/fbc [0,4194304] 0 2024-03-30T04:28:12.186 INFO:tasks.workunit.client.0.smithi123.stdout:1/647: creat db/dd/d42/d46/d51/dd7/fd8 x:0 0 0 2024-03-30T04:28:12.188 INFO:tasks.workunit.client.1.smithi135.stdout:6/46: dread d2/f4 [0,4194304] 0 2024-03-30T04:28:12.188 INFO:tasks.workunit.client.1.smithi135.stdout:7/193: unlink d4/db/d1a/d2a/d2c/d2e/c34 0 2024-03-30T04:28:12.189 INFO:tasks.workunit.client.0.smithi123.stdout:1/648: read - db/dd/d42/d46/d5d/d62/fb4 zero size 2024-03-30T04:28:12.194 INFO:tasks.workunit.client.1.smithi135.stdout:5/158: sync 2024-03-30T04:28:12.194 INFO:tasks.workunit.client.1.smithi135.stdout:0/181: sync 2024-03-30T04:28:12.194 INFO:tasks.workunit.client.1.smithi135.stdout:3/115: creat d3/d17/f21 x:0 0 0 2024-03-30T04:28:12.195 INFO:tasks.workunit.client.1.smithi135.stdout:2/104: dwrite d4/d6/f1a [0,4194304] 0 2024-03-30T04:28:12.197 INFO:tasks.workunit.client.1.smithi135.stdout:5/159: fsync d1/d3/f12 0 2024-03-30T04:28:12.200 INFO:tasks.workunit.client.1.smithi135.stdout:6/47: creat d2/fa x:0 0 0 2024-03-30T04:28:12.201 INFO:tasks.workunit.client.1.smithi135.stdout:8/156: dread d6/d8/d14/fe [0,4194304] 0 2024-03-30T04:28:12.203 INFO:tasks.workunit.client.0.smithi123.stdout:7/510: rmdir d2/da/d11/d34/d76/d95 0 2024-03-30T04:28:12.204 INFO:tasks.workunit.client.1.smithi135.stdout:3/116: mkdir d3/d6/d22 0 2024-03-30T04:28:12.205 INFO:tasks.workunit.client.1.smithi135.stdout:0/182: mknod d0/d13/c35 0 2024-03-30T04:28:12.206 INFO:tasks.workunit.client.1.smithi135.stdout:6/48: creat d2/fb x:0 0 0 2024-03-30T04:28:12.209 INFO:tasks.workunit.client.1.smithi135.stdout:5/160: creat d1/d3/d2b/f2f x:0 0 0 2024-03-30T04:28:12.209 INFO:tasks.workunit.client.0.smithi123.stdout:2/619: dread d0/d1f/d16/f33 [0,4194304] 0 2024-03-30T04:28:12.210 INFO:tasks.workunit.client.0.smithi123.stdout:5/505: mkdir dd/d19/d1f/d3c/d9e 0 2024-03-30T04:28:12.210 INFO:tasks.workunit.client.0.smithi123.stdout:0/582: dread d6/d52/f96 [0,4194304] 0 2024-03-30T04:28:12.212 INFO:tasks.workunit.client.0.smithi123.stdout:3/597: dwrite d13/d24/d31/f64 [4194304,4194304] 0 2024-03-30T04:28:12.215 INFO:tasks.workunit.client.1.smithi135.stdout:8/157: mknod d6/d8/d15/c2c 0 2024-03-30T04:28:12.219 INFO:tasks.workunit.client.0.smithi123.stdout:2/620: truncate d0/d1f/d27/fb0 515512 0 2024-03-30T04:28:12.219 INFO:tasks.workunit.client.0.smithi123.stdout:3/598: write d13/d24/d31/d23/f2b [5397684,62985] 0 2024-03-30T04:28:12.222 INFO:tasks.workunit.client.1.smithi135.stdout:6/49: creat d2/fc x:0 0 0 2024-03-30T04:28:12.225 INFO:tasks.workunit.client.1.smithi135.stdout:3/117: dread d3/d6/fa [0,4194304] 0 2024-03-30T04:28:12.226 INFO:tasks.workunit.client.1.smithi135.stdout:5/161: creat d1/d7/d11/f30 x:0 0 0 2024-03-30T04:28:12.229 INFO:tasks.workunit.client.1.smithi135.stdout:7/194: dwrite d4/db/f1c [0,4194304] 0 2024-03-30T04:28:12.229 INFO:tasks.workunit.client.1.smithi135.stdout:2/105: dread d4/f5 [0,4194304] 0 2024-03-30T04:28:12.230 INFO:tasks.workunit.client.1.smithi135.stdout:7/195: stat d4/db/de/d15/l1e 0 2024-03-30T04:28:12.231 INFO:tasks.workunit.client.1.smithi135.stdout:3/118: truncate d3/d10/f18 1594120 0 2024-03-30T04:28:12.232 INFO:tasks.workunit.client.1.smithi135.stdout:7/196: fdatasync d4/db/d1a/d2a/d2c/d2e/f2f 0 2024-03-30T04:28:12.236 INFO:tasks.workunit.client.0.smithi123.stdout:2/621: symlink d0/d1f/d27/d68/lcf 0 2024-03-30T04:28:12.239 INFO:tasks.workunit.client.1.smithi135.stdout:5/162: fdatasync d1/d3/f15 0 2024-03-30T04:28:12.240 INFO:tasks.workunit.client.1.smithi135.stdout:2/106: chown d4/da/dd/d12 1090 1 2024-03-30T04:28:12.241 INFO:tasks.workunit.client.1.smithi135.stdout:2/107: chown d4/c14 40749 1 2024-03-30T04:28:12.242 INFO:tasks.workunit.client.0.smithi123.stdout:3/599: dwrite d13/d24/d31/d23/d3a/d3c/d4e/d5d/da5/fad [0,4194304] 0 2024-03-30T04:28:12.243 INFO:tasks.workunit.client.1.smithi135.stdout:3/119: mkdir d3/d10/d16/d23 0 2024-03-30T04:28:12.247 INFO:tasks.workunit.client.1.smithi135.stdout:7/197: fsync d4/db/d1a/d2a/d2c/f39 0 2024-03-30T04:28:12.247 INFO:tasks.workunit.client.1.smithi135.stdout:8/158: creat d6/d8/f2d x:0 0 0 2024-03-30T04:28:12.249 INFO:tasks.workunit.client.1.smithi135.stdout:2/108: rmdir d4/d6 39 2024-03-30T04:28:12.250 INFO:tasks.workunit.client.0.smithi123.stdout:0/583: rename d6/dd/d15/d98/d4e/d69/d6e/f7c to d6/dd/d15/d98/fc0 0 2024-03-30T04:28:12.253 INFO:tasks.workunit.client.1.smithi135.stdout:5/163: symlink d1/d7/l31 0 2024-03-30T04:28:12.254 INFO:tasks.workunit.client.1.smithi135.stdout:7/198: rmdir d4/db/d1a/d2a/d2c 39 2024-03-30T04:28:12.255 INFO:tasks.workunit.client.1.smithi135.stdout:5/164: read d1/f4 [3880058,97698] 0 2024-03-30T04:28:12.255 INFO:tasks.workunit.client.1.smithi135.stdout:5/165: chown d1 253634212 1 2024-03-30T04:28:12.255 INFO:tasks.workunit.client.1.smithi135.stdout:8/159: creat d6/d8/d14/f2e x:0 0 0 2024-03-30T04:28:12.260 INFO:tasks.workunit.client.1.smithi135.stdout:2/109: symlink d4/d6/l1b 0 2024-03-30T04:28:12.261 INFO:tasks.workunit.client.1.smithi135.stdout:6/50: rmdir d2 39 2024-03-30T04:28:12.261 INFO:tasks.workunit.client.0.smithi123.stdout:2/622: dwrite d0/d1f/d16/f1e [0,4194304] 0 2024-03-30T04:28:12.264 INFO:tasks.workunit.client.0.smithi123.stdout:1/649: truncate db/dd/d42/d46/d56/d5c/f74 1946161 0 2024-03-30T04:28:12.264 INFO:tasks.workunit.client.1.smithi135.stdout:0/183: truncate d0/d5/d10/f2c 1917369 0 2024-03-30T04:28:12.265 INFO:tasks.workunit.client.0.smithi123.stdout:1/650: fsync db/dd/f8e 0 2024-03-30T04:28:12.265 INFO:tasks.workunit.client.0.smithi123.stdout:7/511: dwrite d2/da/d11/d61/f62 [0,4194304] 0 2024-03-30T04:28:12.266 INFO:tasks.workunit.client.1.smithi135.stdout:0/184: chown d0/d1/d16 18899 1 2024-03-30T04:28:12.268 INFO:tasks.workunit.client.0.smithi123.stdout:8/480: sync 2024-03-30T04:28:12.268 INFO:tasks.workunit.client.1.smithi135.stdout:7/199: truncate d4/f11 12984 0 2024-03-30T04:28:12.271 INFO:tasks.workunit.client.1.smithi135.stdout:8/160: symlink d6/d8/l2f 0 2024-03-30T04:28:12.273 INFO:tasks.workunit.client.1.smithi135.stdout:3/120: link d3/d6/f12 d3/f24 0 2024-03-30T04:28:12.276 INFO:tasks.workunit.client.0.smithi123.stdout:5/506: creat dd/d19/f9f x:0 0 0 2024-03-30T04:28:12.277 INFO:tasks.workunit.client.1.smithi135.stdout:8/161: readlink d6/d8/d15/l1a 0 2024-03-30T04:28:12.277 INFO:tasks.workunit.client.1.smithi135.stdout:6/51: readlink d2/l6 0 2024-03-30T04:28:12.278 INFO:tasks.workunit.client.0.smithi123.stdout:5/507: write dd/fe [673239,107756] 0 2024-03-30T04:28:12.280 INFO:tasks.workunit.client.1.smithi135.stdout:2/110: creat d4/f1c x:0 0 0 2024-03-30T04:28:12.282 INFO:tasks.workunit.client.0.smithi123.stdout:1/651: rmdir db/dd/d42/d46/d39 39 2024-03-30T04:28:12.282 INFO:tasks.workunit.client.1.smithi135.stdout:6/52: mkdir d2/dd 0 2024-03-30T04:28:12.283 INFO:tasks.workunit.client.0.smithi123.stdout:8/481: symlink da/d27/l8f 0 2024-03-30T04:28:12.285 INFO:tasks.workunit.client.1.smithi135.stdout:6/53: dread - d2/fb zero size 2024-03-30T04:28:12.286 INFO:tasks.workunit.client.1.smithi135.stdout:2/111: creat d4/d6/f1d x:0 0 0 2024-03-30T04:28:12.286 INFO:tasks.workunit.client.0.smithi123.stdout:2/623: fsync d0/d1f/d1c/f41 0 2024-03-30T04:28:12.288 INFO:tasks.workunit.client.0.smithi123.stdout:0/584: dread d6/dd/d18/d1f/f9e [0,4194304] 0 2024-03-30T04:28:12.289 INFO:tasks.workunit.client.1.smithi135.stdout:5/166: dwrite d1/d7/f10 [0,4194304] 0 2024-03-30T04:28:12.290 INFO:tasks.workunit.client.1.smithi135.stdout:6/54: mknod d2/ce 0 2024-03-30T04:28:12.293 INFO:tasks.workunit.client.0.smithi123.stdout:0/585: truncate d6/dd/d15/d28/f3d 1375298 0 2024-03-30T04:28:12.294 INFO:tasks.workunit.client.1.smithi135.stdout:5/167: unlink d1/d3/l5 0 2024-03-30T04:28:12.294 INFO:tasks.workunit.client.0.smithi123.stdout:5/508: getdents dd/d19/d1f/d35/d3a/d57 0 2024-03-30T04:28:12.296 INFO:tasks.workunit.client.1.smithi135.stdout:8/162: dread f5 [0,4194304] 0 2024-03-30T04:28:12.296 INFO:tasks.workunit.client.1.smithi135.stdout:6/55: write d2/f4 [7382515,77438] 0 2024-03-30T04:28:12.297 INFO:tasks.workunit.client.0.smithi123.stdout:1/652: symlink db/ld9 0 2024-03-30T04:28:12.297 INFO:tasks.workunit.client.1.smithi135.stdout:5/168: write d1/d7/d11/f30 [539465,12163] 0 2024-03-30T04:28:12.298 INFO:tasks.workunit.client.1.smithi135.stdout:6/56: readlink d2/l6 0 2024-03-30T04:28:12.299 INFO:tasks.workunit.client.1.smithi135.stdout:2/112: rename d4/d6/f1d to d4/da/f1e 0 2024-03-30T04:28:12.299 INFO:tasks.workunit.client.1.smithi135.stdout:0/185: truncate d0/d5/d10/f2c 1960006 0 2024-03-30T04:28:12.300 INFO:tasks.workunit.client.1.smithi135.stdout:5/169: fsync d1/d3/d2b/f2f 0 2024-03-30T04:28:12.300 INFO:tasks.workunit.client.0.smithi123.stdout:5/509: fsync dd/f81 0 2024-03-30T04:28:12.301 INFO:tasks.workunit.client.1.smithi135.stdout:8/163: creat d6/d8/d14/f30 x:0 0 0 2024-03-30T04:28:12.302 INFO:tasks.workunit.client.0.smithi123.stdout:3/600: link d13/d24/d31/d23/d3a/d3c/d4e/d5d/l8d d13/d24/d31/d23/d3a/lbf 0 2024-03-30T04:28:12.304 INFO:tasks.workunit.client.0.smithi123.stdout:2/624: rename d0/d1f/d1c/d36/l58 to d0/d1f/d1c/d2a/ld0 0 2024-03-30T04:28:12.305 INFO:tasks.workunit.client.1.smithi135.stdout:8/164: read f4 [9355409,100329] 0 2024-03-30T04:28:12.309 INFO:tasks.workunit.client.0.smithi123.stdout:1/653: fdatasync db/dd/d42/f61 0 2024-03-30T04:28:12.310 INFO:tasks.workunit.client.1.smithi135.stdout:2/113: creat d4/d6/f1f x:0 0 0 2024-03-30T04:28:12.311 INFO:tasks.workunit.client.0.smithi123.stdout:5/510: dread - dd/d19/d1f/d68/f6b zero size 2024-03-30T04:28:12.312 INFO:tasks.workunit.client.1.smithi135.stdout:6/57: symlink d2/dd/lf 0 2024-03-30T04:28:12.312 INFO:tasks.workunit.client.0.smithi123.stdout:1/654: fsync db/f6e 0 2024-03-30T04:28:12.314 INFO:tasks.workunit.client.0.smithi123.stdout:7/512: dread d2/da/d22/d3b/f4e [0,4194304] 0 2024-03-30T04:28:12.316 INFO:tasks.workunit.client.1.smithi135.stdout:8/165: write f5 [97505,26856] 0 2024-03-30T04:28:12.318 INFO:tasks.workunit.client.0.smithi123.stdout:3/601: mknod d13/d24/d31/d23/d3a/d3c/db8/cc0 0 2024-03-30T04:28:12.319 INFO:tasks.workunit.client.1.smithi135.stdout:8/166: chown d6/fa 889561 1 2024-03-30T04:28:12.322 INFO:tasks.workunit.client.0.smithi123.stdout:8/482: dwrite da/dc/f20 [0,4194304] 0 2024-03-30T04:28:12.323 INFO:tasks.workunit.client.1.smithi135.stdout:6/58: rename d2/c5 to d2/c10 0 2024-03-30T04:28:12.324 INFO:tasks.workunit.client.0.smithi123.stdout:4/492: sync 2024-03-30T04:28:12.324 INFO:tasks.workunit.client.0.smithi123.stdout:9/478: sync 2024-03-30T04:28:12.325 INFO:tasks.workunit.client.0.smithi123.stdout:7/513: mkdir d2/da/d11/d34/db0 0 2024-03-30T04:28:12.326 INFO:tasks.workunit.client.1.smithi135.stdout:0/186: dwrite d0/d1/fa [4194304,4194304] 0 2024-03-30T04:28:12.328 INFO:tasks.workunit.client.1.smithi135.stdout:6/59: write d2/f9 [176989,56889] 0 2024-03-30T04:28:12.331 INFO:tasks.workunit.client.1.smithi135.stdout:5/170: dread d1/d7/ff [0,4194304] 0 2024-03-30T04:28:12.332 INFO:tasks.workunit.client.1.smithi135.stdout:6/60: write d2/f4 [8462236,55383] 0 2024-03-30T04:28:12.333 INFO:tasks.workunit.client.0.smithi123.stdout:9/479: chown d6/d1f/da3/la8 409801420 1 2024-03-30T04:28:12.334 INFO:tasks.workunit.client.1.smithi135.stdout:0/187: mknod d0/d5/c36 0 2024-03-30T04:28:12.334 INFO:tasks.workunit.client.1.smithi135.stdout:5/171: read - d1/d3/d2b/f2f zero size 2024-03-30T04:28:12.335 INFO:tasks.workunit.client.0.smithi123.stdout:0/586: truncate d6/f9 7819696 0 2024-03-30T04:28:12.336 INFO:tasks.workunit.client.1.smithi135.stdout:6/61: write d2/f4 [7896277,18948] 0 2024-03-30T04:28:12.336 INFO:tasks.workunit.client.1.smithi135.stdout:2/114: dwrite d4/f1c [0,4194304] 0 2024-03-30T04:28:12.336 INFO:tasks.workunit.client.0.smithi123.stdout:3/602: symlink d13/d24/d39/d79/d90/lc1 0 2024-03-30T04:28:12.336 INFO:tasks.workunit.client.0.smithi123.stdout:9/480: write d6/fd [6222961,24133] 0 2024-03-30T04:28:12.337 INFO:tasks.workunit.client.1.smithi135.stdout:0/188: symlink d0/d1/l37 0 2024-03-30T04:28:12.340 INFO:tasks.workunit.client.1.smithi135.stdout:2/115: mkdir d4/d6/d20 0 2024-03-30T04:28:12.340 INFO:tasks.workunit.client.0.smithi123.stdout:1/655: creat db/dd/d42/d46/d51/fda x:0 0 0 2024-03-30T04:28:12.342 INFO:tasks.workunit.client.1.smithi135.stdout:5/172: rename d1/d3/cc to d1/d3/c32 0 2024-03-30T04:28:12.342 INFO:tasks.workunit.client.1.smithi135.stdout:2/116: readlink d4/da/dd/le 0 2024-03-30T04:28:12.344 INFO:tasks.workunit.client.1.smithi135.stdout:0/189: truncate d0/d5/d9/ff 606842 0 2024-03-30T04:28:12.346 INFO:tasks.workunit.client.0.smithi123.stdout:5/511: dread dd/d19/f87 [0,4194304] 0 2024-03-30T04:28:12.347 INFO:tasks.workunit.client.1.smithi135.stdout:5/173: stat d1/d7/l2d 0 2024-03-30T04:28:12.349 INFO:tasks.workunit.client.0.smithi123.stdout:0/587: rename d6/l76 to d6/lc1 0 2024-03-30T04:28:12.352 INFO:tasks.workunit.client.1.smithi135.stdout:2/117: creat d4/da/f21 x:0 0 0 2024-03-30T04:28:12.355 INFO:tasks.workunit.client.1.smithi135.stdout:0/190: rename d0/f21 to d0/d19/d2d/f38 0 2024-03-30T04:28:12.357 INFO:tasks.workunit.client.1.smithi135.stdout:0/191: dread - d0/d5/d9/f15 zero size 2024-03-30T04:28:12.357 INFO:tasks.workunit.client.0.smithi123.stdout:5/512: write dd/d19/d1f/f82 [5100867,68758] 0 2024-03-30T04:28:12.360 INFO:tasks.workunit.client.1.smithi135.stdout:2/118: symlink d4/da/l22 0 2024-03-30T04:28:12.360 INFO:tasks.workunit.client.0.smithi123.stdout:8/483: dwrite da/d27/d3c/d64/f8e [0,4194304] 0 2024-03-30T04:28:12.361 INFO:tasks.workunit.client.0.smithi123.stdout:5/513: stat f9 0 2024-03-30T04:28:12.362 INFO:tasks.workunit.client.1.smithi135.stdout:8/167: write d6/d8/f20 [462481,120148] 0 2024-03-30T04:28:12.363 INFO:tasks.workunit.client.1.smithi135.stdout:6/62: dwrite d2/f4 [4194304,4194304] 0 2024-03-30T04:28:12.363 INFO:tasks.workunit.client.1.smithi135.stdout:8/168: fdatasync d6/f18 0 2024-03-30T04:28:12.365 INFO:tasks.workunit.client.0.smithi123.stdout:1/656: dread - db/dd/d42/d46/d39/fad zero size 2024-03-30T04:28:12.366 INFO:tasks.workunit.client.0.smithi123.stdout:9/481: link d6/d1f/f3f d6/fa9 0 2024-03-30T04:28:12.367 INFO:tasks.workunit.client.1.smithi135.stdout:6/63: write d2/fb [598002,126040] 0 2024-03-30T04:28:12.371 INFO:tasks.workunit.client.1.smithi135.stdout:5/174: getdents d1/d7 0 2024-03-30T04:28:12.376 INFO:tasks.workunit.client.1.smithi135.stdout:5/175: write d1/d7/d11/f30 [984869,54305] 0 2024-03-30T04:28:12.377 INFO:tasks.workunit.client.0.smithi123.stdout:2/625: dread d0/d1f/d1c/d80/dbb/d97/fc1 [0,4194304] 0 2024-03-30T04:28:12.378 INFO:tasks.workunit.client.1.smithi135.stdout:6/64: link d2/f9 d2/f11 0 2024-03-30T04:28:12.378 INFO:tasks.workunit.client.1.smithi135.stdout:5/176: readlink d1/l25 0 2024-03-30T04:28:12.382 INFO:tasks.workunit.client.0.smithi123.stdout:8/484: write da/d27/d3c/d2d/d5c/f6f [595014,106670] 0 2024-03-30T04:28:12.383 INFO:tasks.workunit.client.1.smithi135.stdout:8/169: dread d6/fa [0,4194304] 0 2024-03-30T04:28:12.384 INFO:tasks.workunit.client.1.smithi135.stdout:6/65: creat d2/f12 x:0 0 0 2024-03-30T04:28:12.386 INFO:tasks.workunit.client.1.smithi135.stdout:4/73: sync 2024-03-30T04:28:12.389 INFO:tasks.workunit.client.1.smithi135.stdout:0/192: dread d0/d13/d1d/f1f [0,4194304] 0 2024-03-30T04:28:12.390 INFO:tasks.workunit.client.0.smithi123.stdout:2/626: fsync d0/d1f/d1c/d20/f5d 0 2024-03-30T04:28:12.391 INFO:tasks.workunit.client.1.smithi135.stdout:2/119: fdatasync d4/f1c 0 2024-03-30T04:28:12.392 INFO:tasks.workunit.client.1.smithi135.stdout:6/66: creat d2/f13 x:0 0 0 2024-03-30T04:28:12.393 INFO:tasks.workunit.client.1.smithi135.stdout:0/193: dread - d0/d5/d9/f15 zero size 2024-03-30T04:28:12.393 INFO:tasks.workunit.client.0.smithi123.stdout:6/617: sync 2024-03-30T04:28:12.394 INFO:tasks.workunit.client.0.smithi123.stdout:4/493: dwrite d2/f86 [0,4194304] 0 2024-03-30T04:28:12.396 INFO:tasks.workunit.client.1.smithi135.stdout:2/120: write d4/f7 [696982,123645] 0 2024-03-30T04:28:12.399 INFO:tasks.workunit.client.1.smithi135.stdout:4/74: creat d10/d1a/f1c x:0 0 0 2024-03-30T04:28:12.399 INFO:tasks.workunit.client.0.smithi123.stdout:0/588: dwrite d6/dd/d15/f8e [0,4194304] 0 2024-03-30T04:28:12.400 INFO:tasks.workunit.client.1.smithi135.stdout:6/67: dread d2/f11 [0,4194304] 0 2024-03-30T04:28:12.403 INFO:tasks.workunit.client.0.smithi123.stdout:8/485: readlink da/d27/d3c/d2d/l3e 0 2024-03-30T04:28:12.404 INFO:tasks.workunit.client.0.smithi123.stdout:6/618: write d10/f1b [5599494,85777] 0 2024-03-30T04:28:12.404 INFO:tasks.workunit.client.1.smithi135.stdout:2/121: creat d4/da/dd/f23 x:0 0 0 2024-03-30T04:28:12.405 INFO:tasks.workunit.client.0.smithi123.stdout:3/603: dwrite d13/d24/d31/d23/d3a/d3c/d4e/d5d/fb7 [0,4194304] 0 2024-03-30T04:28:12.405 INFO:tasks.workunit.client.0.smithi123.stdout:1/657: creat db/dd/fdb x:0 0 0 2024-03-30T04:28:12.407 INFO:tasks.workunit.client.0.smithi123.stdout:5/514: truncate dd/d19/f2b 7124598 0 2024-03-30T04:28:12.407 INFO:tasks.workunit.client.0.smithi123.stdout:2/627: creat d0/d1f/d1c/d39/fd1 x:0 0 0 2024-03-30T04:28:12.407 INFO:tasks.workunit.client.0.smithi123.stdout:4/494: write d2/d4/f49 [550428,79725] 0 2024-03-30T04:28:12.408 INFO:tasks.workunit.client.0.smithi123.stdout:9/482: write d6/d1f/f4f [4971257,9825] 0 2024-03-30T04:28:12.409 INFO:tasks.workunit.client.0.smithi123.stdout:7/514: dread d2/da/d2d/f3d [0,4194304] 0 2024-03-30T04:28:12.410 INFO:tasks.workunit.client.1.smithi135.stdout:5/177: dwrite d1/da/f1b [0,4194304] 0 2024-03-30T04:28:12.414 INFO:tasks.workunit.client.0.smithi123.stdout:2/628: stat d0/d1f/d30/c38 0 2024-03-30T04:28:12.417 INFO:tasks.workunit.client.0.smithi123.stdout:0/589: symlink d6/dd/d31/lc2 0 2024-03-30T04:28:12.417 INFO:tasks.workunit.client.1.smithi135.stdout:2/122: mknod d4/da/dd/d12/c24 0 2024-03-30T04:28:12.418 INFO:tasks.workunit.client.0.smithi123.stdout:8/486: creat da/d27/d56/d6c/d6d/f90 x:0 0 0 2024-03-30T04:28:12.418 INFO:tasks.workunit.client.0.smithi123.stdout:9/483: mkdir d6/d1f/d45/d97/d69/daa 0 2024-03-30T04:28:12.418 INFO:tasks.workunit.client.1.smithi135.stdout:5/178: fdatasync d1/d7/ff 0 2024-03-30T04:28:12.426 INFO:tasks.workunit.client.1.smithi135.stdout:4/75: dread f4 [0,4194304] 0 2024-03-30T04:28:12.427 INFO:tasks.workunit.client.1.smithi135.stdout:8/170: dwrite d6/d8/f1d [4194304,4194304] 0 2024-03-30T04:28:12.427 INFO:tasks.workunit.client.0.smithi123.stdout:7/515: write d2/da/d11/f48 [1193318,101951] 0 2024-03-30T04:28:12.427 INFO:tasks.workunit.client.0.smithi123.stdout:1/658: mkdir db/dd/d42/d46/d51/dd7/ddc 0 2024-03-30T04:28:12.428 INFO:tasks.workunit.client.0.smithi123.stdout:5/515: dread - dd/d19/d1f/d68/f93 zero size 2024-03-30T04:28:12.428 INFO:tasks.workunit.client.0.smithi123.stdout:8/487: write da/d27/d3c/f51 [41772,31976] 0 2024-03-30T04:28:12.429 INFO:tasks.workunit.client.1.smithi135.stdout:5/179: fdatasync d1/d7/f10 0 2024-03-30T04:28:12.434 INFO:tasks.workunit.client.0.smithi123.stdout:7/516: fsync d2/da/d11/f20 0 2024-03-30T04:28:12.435 INFO:tasks.workunit.client.1.smithi135.stdout:4/76: symlink d10/d1a/l1d 0 2024-03-30T04:28:12.439 INFO:tasks.workunit.client.1.smithi135.stdout:2/123: rename d4/d6/ff to d4/f25 0 2024-03-30T04:28:12.444 INFO:tasks.workunit.client.1.smithi135.stdout:4/77: rename d10/c13 to d10/c1e 0 2024-03-30T04:28:12.445 INFO:tasks.workunit.client.0.smithi123.stdout:9/484: rename d6/d1f/f41 to d6/d1f/d45/d97/d69/fab 0 2024-03-30T04:28:12.446 INFO:tasks.workunit.client.0.smithi123.stdout:1/659: creat db/dd/d5f/d6d/d85/d89/fdd x:0 0 0 2024-03-30T04:28:12.450 INFO:tasks.workunit.client.1.smithi135.stdout:6/68: dwrite d2/f11 [0,4194304] 0 2024-03-30T04:28:12.450 INFO:tasks.workunit.client.0.smithi123.stdout:3/604: dwrite d13/d24/d31/d30/faa [0,4194304] 0 2024-03-30T04:28:12.451 INFO:tasks.workunit.client.0.smithi123.stdout:0/590: creat d6/dd/d15/fc3 x:0 0 0 2024-03-30T04:28:12.451 INFO:tasks.workunit.client.1.smithi135.stdout:6/69: dread - d2/f13 zero size 2024-03-30T04:28:12.454 INFO:tasks.workunit.client.0.smithi123.stdout:1/660: read - db/dd/d42/d46/d56/da2/fd2 zero size 2024-03-30T04:28:12.456 INFO:tasks.workunit.client.1.smithi135.stdout:8/171: rename f4 to d6/d8/f31 0 2024-03-30T04:28:12.460 INFO:tasks.workunit.client.1.smithi135.stdout:6/70: getdents d2/dd 0 2024-03-30T04:28:12.461 INFO:tasks.workunit.client.1.smithi135.stdout:2/124: dread d4/da/fb [0,4194304] 0 2024-03-30T04:28:12.461 INFO:tasks.workunit.client.0.smithi123.stdout:4/495: dread d2/d4/d2c/d42/d35/d5d/d6f/f8a [0,4194304] 0 2024-03-30T04:28:12.461 INFO:tasks.workunit.client.0.smithi123.stdout:5/516: rename dd/c6f to dd/d19/d1f/ca0 0 2024-03-30T04:28:12.462 INFO:tasks.workunit.client.1.smithi135.stdout:8/172: dread d6/d8/f20 [0,4194304] 0 2024-03-30T04:28:12.463 INFO:tasks.workunit.client.0.smithi123.stdout:3/605: chown d13/d24/d31/d23/f76 216 1 2024-03-30T04:28:12.464 INFO:tasks.workunit.client.1.smithi135.stdout:6/71: mkdir d2/dd/d14 0 2024-03-30T04:28:12.464 INFO:tasks.workunit.client.1.smithi135.stdout:4/78: dwrite fb [0,4194304] 0 2024-03-30T04:28:12.465 INFO:tasks.workunit.client.1.smithi135.stdout:6/72: dread - d2/fc zero size 2024-03-30T04:28:12.466 INFO:tasks.workunit.client.0.smithi123.stdout:5/517: chown dd/d41 97 1 2024-03-30T04:28:12.467 INFO:tasks.workunit.client.0.smithi123.stdout:5/518: readlink dd/l1b 0 2024-03-30T04:28:12.468 INFO:tasks.workunit.client.1.smithi135.stdout:4/79: creat d10/d1a/f1f x:0 0 0 2024-03-30T04:28:12.468 INFO:tasks.workunit.client.1.smithi135.stdout:6/73: truncate d2/f11 4579428 0 2024-03-30T04:28:12.469 INFO:tasks.workunit.client.1.smithi135.stdout:4/80: dread - d10/f18 zero size 2024-03-30T04:28:12.469 INFO:tasks.workunit.client.1.smithi135.stdout:6/74: dread - d2/fc zero size 2024-03-30T04:28:12.470 INFO:tasks.workunit.client.1.smithi135.stdout:6/75: chown d2/f11 12 1 2024-03-30T04:28:12.476 INFO:tasks.workunit.client.0.smithi123.stdout:4/496: rename d2/d4/d2c/d42/d80/d87 to d2/d4/d2c/d42/dad 0 2024-03-30T04:28:12.479 INFO:tasks.workunit.client.0.smithi123.stdout:8/488: dwrite da/dc/d14/d18/d21/f40 [0,4194304] 0 2024-03-30T04:28:12.480 INFO:tasks.workunit.client.0.smithi123.stdout:4/497: chown d2/d4/d2c/d42/l8c 22174 1 2024-03-30T04:28:12.482 INFO:tasks.workunit.client.0.smithi123.stdout:8/489: stat da/d27/l35 0 2024-03-30T04:28:12.482 INFO:tasks.workunit.client.0.smithi123.stdout:3/606: unlink d13/d24/d31/d23/d3a/d3c/l69 0 2024-03-30T04:28:12.482 INFO:tasks.workunit.client.0.smithi123.stdout:2/629: dwrite d0/d1f/d5e/f7a [0,4194304] 0 2024-03-30T04:28:12.485 INFO:tasks.workunit.client.0.smithi123.stdout:5/519: mknod dd/d19/d1f/d35/d3a/d79/ca1 0 2024-03-30T04:28:12.487 INFO:tasks.workunit.client.0.smithi123.stdout:2/630: chown d0/d37 2595 1 2024-03-30T04:28:12.489 INFO:tasks.workunit.client.1.smithi135.stdout:2/125: dwrite d4/da/f21 [0,4194304] 0 2024-03-30T04:28:12.489 INFO:tasks.workunit.client.0.smithi123.stdout:5/520: stat dd/c11 0 2024-03-30T04:28:12.495 INFO:tasks.workunit.client.0.smithi123.stdout:4/498: creat d2/d4/d2c/d42/d35/d5d/fae x:0 0 0 2024-03-30T04:28:12.498 INFO:tasks.workunit.client.0.smithi123.stdout:5/521: creat dd/d19/d3e/d5b/fa2 x:0 0 0 2024-03-30T04:28:12.498 INFO:tasks.workunit.client.1.smithi135.stdout:4/81: dwrite f4 [0,4194304] 0 2024-03-30T04:28:12.502 INFO:tasks.workunit.client.0.smithi123.stdout:0/591: dwrite d6/dd/d15/d98/d4e/d69/f81 [4194304,4194304] 0 2024-03-30T04:28:12.502 INFO:tasks.workunit.client.1.smithi135.stdout:6/76: rmdir d2 39 2024-03-30T04:28:12.502 INFO:tasks.workunit.client.1.smithi135.stdout:2/126: creat d4/da/dd/f26 x:0 0 0 2024-03-30T04:28:12.504 INFO:tasks.workunit.client.1.smithi135.stdout:4/82: chown c8 101785 1 2024-03-30T04:28:12.505 INFO:tasks.workunit.client.0.smithi123.stdout:4/499: write d2/d4/d2c/d42/d35/faa [2011877,60812] 0 2024-03-30T04:28:12.508 INFO:tasks.workunit.client.1.smithi135.stdout:2/127: symlink d4/d6/l27 0 2024-03-30T04:28:12.509 INFO:tasks.workunit.client.1.smithi135.stdout:4/83: chown d10/f14 53396 1 2024-03-30T04:28:12.511 INFO:tasks.workunit.client.0.smithi123.stdout:0/592: stat d6/dd/d15/d28/f33 0 2024-03-30T04:28:12.511 INFO:tasks.workunit.client.1.smithi135.stdout:6/77: write d2/f8 [108316,10375] 0 2024-03-30T04:28:12.517 INFO:tasks.workunit.client.1.smithi135.stdout:4/84: mknod d10/c20 0 2024-03-30T04:28:12.522 INFO:tasks.workunit.client.1.smithi135.stdout:2/128: rename d4/da/c17 to d4/c28 0 2024-03-30T04:28:12.524 INFO:tasks.workunit.client.0.smithi123.stdout:4/500: mknod d2/d4/d2c/d42/d80/caf 0 2024-03-30T04:28:12.525 INFO:tasks.workunit.client.0.smithi123.stdout:8/490: dwrite da/dc/d14/d18/f4f [0,4194304] 0 2024-03-30T04:28:12.527 INFO:tasks.workunit.client.0.smithi123.stdout:8/491: stat da/d27/f53 0 2024-03-30T04:28:12.529 INFO:tasks.workunit.client.1.smithi135.stdout:4/85: rename d10/f18 to d10/d1a/f21 0 2024-03-30T04:28:12.532 INFO:tasks.workunit.client.0.smithi123.stdout:4/501: creat d2/d4/d2c/d42/dad/d91/d9d/fb0 x:0 0 0 2024-03-30T04:28:12.533 INFO:tasks.workunit.client.1.smithi135.stdout:4/86: write fd [2954891,100338] 0 2024-03-30T04:28:12.535 INFO:tasks.workunit.client.0.smithi123.stdout:4/502: readlink d2/d4/d2c/d42/l8c 0 2024-03-30T04:28:12.535 INFO:tasks.workunit.client.0.smithi123.stdout:5/522: dread dd/d19/d3e/f63 [0,4194304] 0 2024-03-30T04:28:12.536 INFO:tasks.workunit.client.0.smithi123.stdout:3/607: dread d13/f80 [0,4194304] 0 2024-03-30T04:28:12.539 INFO:tasks.workunit.client.0.smithi123.stdout:4/503: chown d2/d34/f88 443750664 1 2024-03-30T04:28:12.540 INFO:tasks.workunit.client.1.smithi135.stdout:4/87: rename d10/f19 to d10/d1a/f22 0 2024-03-30T04:28:12.550 INFO:tasks.workunit.client.1.smithi135.stdout:7/200: sync 2024-03-30T04:28:12.550 INFO:tasks.workunit.client.1.smithi135.stdout:9/80: sync 2024-03-30T04:28:12.550 INFO:tasks.workunit.client.1.smithi135.stdout:1/122: sync 2024-03-30T04:28:12.550 INFO:tasks.workunit.client.0.smithi123.stdout:1/661: dread db/f24 [4194304,4194304] 0 2024-03-30T04:28:12.552 INFO:tasks.workunit.client.1.smithi135.stdout:6/78: fsync d2/f8 0 2024-03-30T04:28:12.553 INFO:tasks.workunit.client.0.smithi123.stdout:2/631: dread d0/d1f/d1c/d2a/f7c [0,4194304] 0 2024-03-30T04:28:12.554 INFO:tasks.workunit.client.0.smithi123.stdout:2/632: chown d0/d1f/d1c 400 1 2024-03-30T04:28:12.555 INFO:tasks.workunit.client.1.smithi135.stdout:9/81: mknod d7/da/c13 0 2024-03-30T04:28:12.555 INFO:tasks.workunit.client.0.smithi123.stdout:1/662: truncate db/d1b/f2a 3804075 0 2024-03-30T04:28:12.558 INFO:tasks.workunit.client.0.smithi123.stdout:2/633: truncate d0/d1f/d16/f26 5319341 0 2024-03-30T04:28:12.558 INFO:tasks.workunit.client.1.smithi135.stdout:6/79: chown d2/fa 520842921 1 2024-03-30T04:28:12.558 INFO:tasks.workunit.client.1.smithi135.stdout:9/82: fdatasync f6 0 2024-03-30T04:28:12.558 INFO:tasks.workunit.client.1.smithi135.stdout:1/123: truncate d16/f18 831062 0 2024-03-30T04:28:12.559 INFO:tasks.workunit.client.1.smithi135.stdout:7/201: symlink d4/l44 0 2024-03-30T04:28:12.559 INFO:tasks.workunit.client.0.smithi123.stdout:1/663: mkdir db/dd/d5f/d6d/dde 0 2024-03-30T04:28:12.563 INFO:tasks.workunit.client.1.smithi135.stdout:4/88: dwrite fb [0,4194304] 0 2024-03-30T04:28:12.564 INFO:tasks.workunit.client.1.smithi135.stdout:6/80: write d2/fc [795009,35578] 0 2024-03-30T04:28:12.565 INFO:tasks.workunit.client.1.smithi135.stdout:7/202: fsync d4/db/f26 0 2024-03-30T04:28:12.567 INFO:tasks.workunit.client.0.smithi123.stdout:2/634: creat d0/d1f/d1c/d80/dbb/d97/db5/fd2 x:0 0 0 2024-03-30T04:28:12.569 INFO:tasks.workunit.client.1.smithi135.stdout:1/124: link d16/f18 d16/d29/d2c/f2e 0 2024-03-30T04:28:12.570 INFO:tasks.workunit.client.1.smithi135.stdout:1/125: chown d16/d29/d2c 938 1 2024-03-30T04:28:12.572 INFO:tasks.workunit.client.1.smithi135.stdout:1/126: chown d16/d29/d2c 2939168 1 2024-03-30T04:28:12.577 INFO:tasks.workunit.client.1.smithi135.stdout:7/203: rename d4/db/d1a/d2a/d2c/d2e/f2f to d4/db/de/d15/f45 0 2024-03-30T04:28:12.584 INFO:tasks.workunit.client.0.smithi123.stdout:0/593: dwrite d6/fa [0,4194304] 0 2024-03-30T04:28:12.584 INFO:tasks.workunit.client.0.smithi123.stdout:4/504: dwrite d2/f45 [0,4194304] 0 2024-03-30T04:28:12.584 INFO:tasks.workunit.client.0.smithi123.stdout:8/492: dwrite da/dc/d14/d18/d21/f45 [0,4194304] 0 2024-03-30T04:28:12.584 INFO:tasks.workunit.client.0.smithi123.stdout:1/664: dread db/dd/d42/d46/d56/d5c/f97 [0,4194304] 0 2024-03-30T04:28:12.586 INFO:tasks.workunit.client.1.smithi135.stdout:1/127: rename fa to d16/f2f 0 2024-03-30T04:28:12.589 INFO:tasks.workunit.client.1.smithi135.stdout:1/128: write d16/f1d [725335,70304] 0 2024-03-30T04:28:12.589 INFO:tasks.workunit.client.1.smithi135.stdout:7/204: read d4/f11 [11652,110280] 0 2024-03-30T04:28:12.590 INFO:tasks.workunit.client.0.smithi123.stdout:8/493: symlink da/d27/d3c/d2d/d5c/l91 0 2024-03-30T04:28:12.592 INFO:tasks.workunit.client.1.smithi135.stdout:6/81: dwrite d2/f13 [0,4194304] 0 2024-03-30T04:28:12.594 INFO:tasks.workunit.client.1.smithi135.stdout:7/205: truncate d4/db/f26 1043155 0 2024-03-30T04:28:12.595 INFO:tasks.workunit.client.0.smithi123.stdout:0/594: unlink d6/dd/d15/d98/fc0 0 2024-03-30T04:28:12.598 INFO:tasks.workunit.client.0.smithi123.stdout:2/635: dwrite d0/d1f/d1c/d80/f5c [0,4194304] 0 2024-03-30T04:28:12.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:12 smithi135 ceph-mon[44364]: pgmap v39: 65 pgs: 65 active+clean; 1.3 GiB data, 5.4 GiB used, 531 GiB / 536 GiB avail; 39 MiB/s rd, 161 MiB/s wr, 397 op/s 2024-03-30T04:28:12.602 INFO:tasks.workunit.client.1.smithi135.stdout:7/206: truncate d4/db/de/f1d 2292119 0 2024-03-30T04:28:12.603 INFO:tasks.workunit.client.0.smithi123.stdout:4/505: creat d2/d4/d2c/d42/dad/d91/fb1 x:0 0 0 2024-03-30T04:28:12.606 INFO:tasks.workunit.client.0.smithi123.stdout:2/636: dread - d0/d1f/d16/fcb zero size 2024-03-30T04:28:12.606 INFO:tasks.workunit.client.0.smithi123.stdout:4/506: dread - d2/d4/d2c/d42/d89/f8d zero size 2024-03-30T04:28:12.608 INFO:tasks.workunit.client.1.smithi135.stdout:4/89: getdents d10 0 2024-03-30T04:28:12.609 INFO:tasks.workunit.client.1.smithi135.stdout:7/207: fdatasync d4/db/de/f1d 0 2024-03-30T04:28:12.610 INFO:tasks.workunit.client.1.smithi135.stdout:9/83: truncate f1 726161 0 2024-03-30T04:28:12.612 INFO:tasks.workunit.client.0.smithi123.stdout:1/665: getdents db/dd/d5f 0 2024-03-30T04:28:12.613 INFO:tasks.workunit.client.1.smithi135.stdout:4/90: read - d10/d1a/f1c zero size 2024-03-30T04:28:12.614 INFO:tasks.workunit.client.0.smithi123.stdout:8/494: creat da/f92 x:0 0 0 2024-03-30T04:28:12.614 INFO:tasks.workunit.client.0.smithi123.stdout:4/507: symlink d2/d26/lb2 0 2024-03-30T04:28:12.615 INFO:tasks.workunit.client.1.smithi135.stdout:9/84: write f6 [1445937,91910] 0 2024-03-30T04:28:12.617 INFO:tasks.workunit.client.0.smithi123.stdout:1/666: creat db/dd/d42/d46/d51/d72/fdf x:0 0 0 2024-03-30T04:28:12.619 INFO:tasks.workunit.client.0.smithi123.stdout:4/508: fsync d2/d4/d2c/d42/dad/d91/d9d/fa6 0 2024-03-30T04:28:12.619 INFO:tasks.workunit.client.0.smithi123.stdout:2/637: creat d0/d1f/d1c/d20/dc6/fd3 x:0 0 0 2024-03-30T04:28:12.620 INFO:tasks.workunit.client.0.smithi123.stdout:8/495: mkdir da/d27/d56/d93 0 2024-03-30T04:28:12.624 INFO:tasks.workunit.client.1.smithi135.stdout:9/85: chown d7/ff 438 1 2024-03-30T04:28:12.625 INFO:tasks.workunit.client.0.smithi123.stdout:8/496: write da/dc/d14/d18/d21/f4d [1022630,116382] 0 2024-03-30T04:28:12.626 INFO:tasks.workunit.client.1.smithi135.stdout:1/129: dwrite d16/d19/f1f [0,4194304] 0 2024-03-30T04:28:12.630 INFO:tasks.workunit.client.1.smithi135.stdout:9/86: fsync d7/f11 0 2024-03-30T04:28:12.630 INFO:tasks.workunit.client.1.smithi135.stdout:6/82: dwrite d2/f4 [4194304,4194304] 0 2024-03-30T04:28:12.630 INFO:tasks.workunit.client.1.smithi135.stdout:9/87: readlink d7/da/l9 0 2024-03-30T04:28:12.635 INFO:tasks.workunit.client.1.smithi135.stdout:1/130: mkdir d16/d29/d2c/d30 0 2024-03-30T04:28:12.636 INFO:tasks.workunit.client.0.smithi123.stdout:2/638: creat d0/d1f/d1c/d39/d4b/fd4 x:0 0 0 2024-03-30T04:28:12.636 INFO:tasks.workunit.client.1.smithi135.stdout:6/83: creat d2/dd/d14/f15 x:0 0 0 2024-03-30T04:28:12.636 INFO:tasks.workunit.client.0.smithi123.stdout:4/509: mkdir d2/d4/db3 0 2024-03-30T04:28:12.637 INFO:tasks.workunit.client.1.smithi135.stdout:1/131: fsync f3 0 2024-03-30T04:28:12.638 INFO:tasks.workunit.client.1.smithi135.stdout:6/84: chown d2/f12 24773 1 2024-03-30T04:28:12.639 INFO:tasks.workunit.client.0.smithi123.stdout:1/667: rmdir db/dd/d5f/d6d/dde 0 2024-03-30T04:28:12.639 INFO:tasks.workunit.client.1.smithi135.stdout:6/85: truncate d2/f12 533008 0 2024-03-30T04:28:12.640 INFO:tasks.workunit.client.0.smithi123.stdout:2/639: chown d0/d1f/d1c/d39/d4b/d4c 99 1 2024-03-30T04:28:12.640 INFO:tasks.workunit.client.1.smithi135.stdout:1/132: fdatasync d16/f1c 0 2024-03-30T04:28:12.641 INFO:tasks.workunit.client.1.smithi135.stdout:6/86: chown d2/fa 1744 1 2024-03-30T04:28:12.643 INFO:tasks.workunit.client.1.smithi135.stdout:9/88: dread d7/f11 [0,4194304] 0 2024-03-30T04:28:12.645 INFO:tasks.workunit.client.0.smithi123.stdout:2/640: stat d0/d1f/d1c/d39/l85 0 2024-03-30T04:28:12.647 INFO:tasks.workunit.client.1.smithi135.stdout:9/89: symlink d7/da/l14 0 2024-03-30T04:28:12.648 INFO:tasks.workunit.client.1.smithi135.stdout:1/133: truncate d16/d1a/f21 1060118 0 2024-03-30T04:28:12.649 INFO:tasks.workunit.client.1.smithi135.stdout:6/87: link d2/ce d2/dd/c16 0 2024-03-30T04:28:12.651 INFO:tasks.workunit.client.1.smithi135.stdout:9/90: mknod d7/da/c15 0 2024-03-30T04:28:12.651 INFO:tasks.workunit.client.1.smithi135.stdout:6/88: fdatasync d2/fa 0 2024-03-30T04:28:12.654 INFO:tasks.workunit.client.0.smithi123.stdout:8/497: dwrite da/d27/d3c/f5e [0,4194304] 0 2024-03-30T04:28:12.657 INFO:tasks.workunit.client.0.smithi123.stdout:1/668: truncate db/dd/d42/d8f/d78/fa0 5156467 0 2024-03-30T04:28:12.660 INFO:tasks.workunit.client.1.smithi135.stdout:9/91: dread d7/f12 [0,4194304] 0 2024-03-30T04:28:12.661 INFO:tasks.workunit.client.0.smithi123.stdout:1/669: read - db/dd/d42/f4f zero size 2024-03-30T04:28:12.662 INFO:tasks.workunit.client.1.smithi135.stdout:4/91: truncate fd 2931209 0 2024-03-30T04:28:12.663 INFO:tasks.workunit.client.0.smithi123.stdout:8/498: mknod da/d27/d56/d6c/c94 0 2024-03-30T04:28:12.663 INFO:tasks.workunit.client.1.smithi135.stdout:7/208: write d4/db/d1a/d2a/d2c/f39 [6416417,98722] 0 2024-03-30T04:28:12.668 INFO:tasks.workunit.client.0.smithi123.stdout:2/641: rename d0/d1f/d5e/d71 to d0/d1f/d1c/d80/dbb/dd5 0 2024-03-30T04:28:12.671 INFO:tasks.workunit.client.1.smithi135.stdout:6/89: dread d2/f4 [0,4194304] 0 2024-03-30T04:28:12.672 INFO:tasks.workunit.client.1.smithi135.stdout:9/92: rename d7/da/fc to d7/da/f16 0 2024-03-30T04:28:12.672 INFO:tasks.workunit.client.1.smithi135.stdout:7/209: chown d4/db/d1a/d2a/d2c/d2e/d42 218796 1 2024-03-30T04:28:12.672 INFO:tasks.workunit.client.0.smithi123.stdout:2/642: chown d0/d1f/d1c/d80/dbb/d97 21879854 1 2024-03-30T04:28:12.676 INFO:tasks.workunit.client.1.smithi135.stdout:1/134: dwrite fb [0,4194304] 0 2024-03-30T04:28:12.677 INFO:tasks.workunit.client.1.smithi135.stdout:7/210: write d4/f29 [2074918,78185] 0 2024-03-30T04:28:12.678 INFO:tasks.workunit.client.1.smithi135.stdout:6/90: creat d2/dd/f17 x:0 0 0 2024-03-30T04:28:12.683 INFO:tasks.workunit.client.0.smithi123.stdout:2/643: unlink d0/d1f/d1c/d2a/d8c/lb2 0 2024-03-30T04:28:12.692 INFO:tasks.workunit.client.1.smithi135.stdout:6/91: truncate d2/dd/f17 1012476 0 2024-03-30T04:28:12.696 INFO:tasks.workunit.client.0.smithi123.stdout:1/670: dwrite db/dd/d42/d46/d5d/d62/f6b [4194304,4194304] 0 2024-03-30T04:28:12.696 INFO:tasks.workunit.client.0.smithi123.stdout:8/499: dwrite da/d27/d3c/d2d/d5c/f6f [0,4194304] 0 2024-03-30T04:28:12.698 INFO:tasks.workunit.client.0.smithi123.stdout:6/619: sync 2024-03-30T04:28:12.700 INFO:tasks.workunit.client.1.smithi135.stdout:4/92: dwrite d10/f17 [0,4194304] 0 2024-03-30T04:28:12.701 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:12 smithi123 ceph-mon[35245]: pgmap v39: 65 pgs: 65 active+clean; 1.3 GiB data, 5.4 GiB used, 531 GiB / 536 GiB avail; 39 MiB/s rd, 161 MiB/s wr, 397 op/s 2024-03-30T04:28:12.704 INFO:tasks.workunit.client.1.smithi135.stdout:6/92: mkdir d2/dd/d18 0 2024-03-30T04:28:12.708 INFO:tasks.workunit.client.0.smithi123.stdout:8/500: mkdir da/d27/d95 0 2024-03-30T04:28:12.708 INFO:tasks.workunit.client.0.smithi123.stdout:6/620: mkdir d10/d82/dc1/d45/d77/dd5 0 2024-03-30T04:28:12.709 INFO:tasks.workunit.client.0.smithi123.stdout:8/501: chown da/ff 1793030294 1 2024-03-30T04:28:12.712 INFO:tasks.workunit.client.1.smithi135.stdout:6/93: mknod d2/dd/c19 0 2024-03-30T04:28:12.713 INFO:tasks.workunit.client.0.smithi123.stdout:6/621: write d10/d82/d8e/dc7/f71 [335837,7407] 0 2024-03-30T04:28:12.713 INFO:tasks.workunit.client.0.smithi123.stdout:2/644: getdents d0/d1f/d5e 0 2024-03-30T04:28:12.715 INFO:tasks.workunit.client.0.smithi123.stdout:4/510: dread d2/f20 [0,4194304] 0 2024-03-30T04:28:12.715 INFO:tasks.workunit.client.0.smithi123.stdout:8/502: chown da/d27/d56/c60 1407900925 1 2024-03-30T04:28:12.717 INFO:tasks.workunit.client.1.smithi135.stdout:9/93: dwrite d7/fe [0,4194304] 0 2024-03-30T04:28:12.720 INFO:tasks.workunit.client.1.smithi135.stdout:4/93: getdents d10 0 2024-03-30T04:28:12.724 INFO:tasks.workunit.client.0.smithi123.stdout:4/511: readlink d2/d4/d2c/l59 0 2024-03-30T04:28:12.725 INFO:tasks.workunit.client.1.smithi135.stdout:6/94: write d2/fc [456661,41343] 0 2024-03-30T04:28:12.727 INFO:tasks.workunit.client.1.smithi135.stdout:1/135: dwrite fb [0,4194304] 0 2024-03-30T04:28:12.729 INFO:tasks.workunit.client.1.smithi135.stdout:6/95: creat d2/dd/f1a x:0 0 0 2024-03-30T04:28:12.730 INFO:tasks.workunit.client.1.smithi135.stdout:7/211: dwrite d4/f11 [0,4194304] 0 2024-03-30T04:28:12.731 INFO:tasks.workunit.client.1.smithi135.stdout:1/136: symlink d16/d19/d20/l31 0 2024-03-30T04:28:12.734 INFO:tasks.workunit.client.1.smithi135.stdout:6/96: symlink d2/dd/l1b 0 2024-03-30T04:28:12.735 INFO:tasks.workunit.client.1.smithi135.stdout:7/212: chown d4/db/d1a/d2a/f37 290377 1 2024-03-30T04:28:12.741 INFO:tasks.workunit.client.1.smithi135.stdout:1/137: unlink c9 0 2024-03-30T04:28:12.746 INFO:tasks.workunit.client.0.smithi123.stdout:8/503: dwrite da/dc/d14/d18/f86 [0,4194304] 0 2024-03-30T04:28:12.746 INFO:tasks.workunit.client.1.smithi135.stdout:3/121: sync 2024-03-30T04:28:12.748 INFO:tasks.workunit.client.1.smithi135.stdout:3/122: fsync d3/d17/f1c 0 2024-03-30T04:28:12.751 INFO:tasks.workunit.client.0.smithi123.stdout:8/504: chown da/dc/d14/d18/l38 0 1 2024-03-30T04:28:12.752 INFO:tasks.workunit.client.0.smithi123.stdout:6/622: dread d10/f9e [0,4194304] 0 2024-03-30T04:28:12.757 INFO:tasks.workunit.client.1.smithi135.stdout:3/123: creat d3/d6/f25 x:0 0 0 2024-03-30T04:28:12.759 INFO:tasks.workunit.client.0.smithi123.stdout:2/645: dwrite d0/d1f/d5e/f6f [0,4194304] 0 2024-03-30T04:28:12.766 INFO:tasks.workunit.client.1.smithi135.stdout:9/94: dwrite d7/da/f16 [0,4194304] 0 2024-03-30T04:28:12.771 INFO:tasks.workunit.client.1.smithi135.stdout:1/138: dread f7 [0,4194304] 0 2024-03-30T04:28:12.772 INFO:tasks.workunit.client.1.smithi135.stdout:9/95: chown d7/f11 610239205 1 2024-03-30T04:28:12.772 INFO:tasks.workunit.client.0.smithi123.stdout:4/512: dread d2/d4/d2c/d42/f3a [0,4194304] 0 2024-03-30T04:28:12.774 INFO:tasks.workunit.client.1.smithi135.stdout:7/213: dwrite d4/f11 [0,4194304] 0 2024-03-30T04:28:12.778 INFO:tasks.workunit.client.1.smithi135.stdout:7/214: write d4/f11 [2060261,35952] 0 2024-03-30T04:28:12.780 INFO:tasks.workunit.client.1.smithi135.stdout:1/139: mkdir d16/d29/d32 0 2024-03-30T04:28:12.780 INFO:tasks.workunit.client.0.smithi123.stdout:8/505: dwrite da/dc/d14/d18/f30 [0,4194304] 0 2024-03-30T04:28:12.781 INFO:tasks.workunit.client.1.smithi135.stdout:1/140: chown d16/d1a 1747 1 2024-03-30T04:28:12.783 INFO:tasks.workunit.client.1.smithi135.stdout:3/124: dread d3/d10/d16/f1e [0,4194304] 0 2024-03-30T04:28:12.784 INFO:tasks.workunit.client.1.smithi135.stdout:9/96: symlink d7/da/l17 0 2024-03-30T04:28:12.790 INFO:tasks.workunit.client.1.smithi135.stdout:7/215: dread - d4/db/d1a/d2a/d2c/f2d zero size 2024-03-30T04:28:12.797 INFO:tasks.workunit.client.1.smithi135.stdout:4/94: truncate d10/f17 4191215 0 2024-03-30T04:28:12.799 INFO:tasks.workunit.client.1.smithi135.stdout:9/97: mknod d7/c18 0 2024-03-30T04:28:12.801 INFO:tasks.workunit.client.1.smithi135.stdout:4/95: dread - d10/f14 zero size 2024-03-30T04:28:12.805 INFO:tasks.workunit.client.1.smithi135.stdout:4/96: chown d10/d1a/f21 5673 1 2024-03-30T04:28:12.807 INFO:tasks.workunit.client.1.smithi135.stdout:6/97: dread d2/fc [0,4194304] 0 2024-03-30T04:28:12.808 INFO:tasks.workunit.client.1.smithi135.stdout:1/141: mkdir d16/d19/d20/d26/d33 0 2024-03-30T04:28:12.813 INFO:tasks.workunit.client.1.smithi135.stdout:9/98: chown d7/l10 28 1 2024-03-30T04:28:12.816 INFO:tasks.workunit.client.1.smithi135.stdout:6/98: mknod d2/c1c 0 2024-03-30T04:28:12.818 INFO:tasks.workunit.client.1.smithi135.stdout:1/142: stat d16/d19/d20/l27 0 2024-03-30T04:28:12.821 INFO:tasks.workunit.client.1.smithi135.stdout:5/180: sync 2024-03-30T04:28:12.821 INFO:tasks.workunit.client.1.smithi135.stdout:0/194: sync 2024-03-30T04:28:12.822 INFO:tasks.workunit.client.0.smithi123.stdout:8/506: mkdir da/d27/d56/d6c/d96 0 2024-03-30T04:28:12.824 INFO:tasks.workunit.client.1.smithi135.stdout:6/99: symlink d2/dd/d18/l1d 0 2024-03-30T04:28:12.830 INFO:tasks.workunit.client.1.smithi135.stdout:0/195: mknod d0/d1/c39 0 2024-03-30T04:28:12.831 INFO:tasks.workunit.client.0.smithi123.stdout:8/507: creat da/d27/d3c/d2d/f97 x:0 0 0 2024-03-30T04:28:12.835 INFO:tasks.workunit.client.0.smithi123.stdout:8/508: creat da/d27/d3c/f98 x:0 0 0 2024-03-30T04:28:12.836 INFO:tasks.workunit.client.1.smithi135.stdout:0/196: truncate d0/d5/d9/f17 1747953 0 2024-03-30T04:28:12.839 INFO:tasks.workunit.client.1.smithi135.stdout:9/99: rename d7/lb to d7/da/l19 0 2024-03-30T04:28:12.842 INFO:tasks.workunit.client.1.smithi135.stdout:3/125: write d3/f24 [3139081,27064] 0 2024-03-30T04:28:12.847 INFO:tasks.workunit.client.0.smithi123.stdout:8/509: dread da/d27/d3c/f51 [0,4194304] 0 2024-03-30T04:28:12.848 INFO:tasks.workunit.client.1.smithi135.stdout:0/197: mkdir d0/d13/d1e/d3a 0 2024-03-30T04:28:12.850 INFO:tasks.workunit.client.0.smithi123.stdout:8/510: chown da/dc/d14/d18/d21/f40 33953 1 2024-03-30T04:28:12.853 INFO:tasks.workunit.client.1.smithi135.stdout:1/143: rename l10 to d16/d19/d20/d26/d33/l34 0 2024-03-30T04:28:12.853 INFO:tasks.workunit.client.1.smithi135.stdout:3/126: write d3/f24 [919647,4401] 0 2024-03-30T04:28:12.857 INFO:tasks.workunit.client.1.smithi135.stdout:6/100: dwrite d2/dd/f1a [0,4194304] 0 2024-03-30T04:28:12.860 INFO:tasks.workunit.client.1.smithi135.stdout:6/101: fsync d2/dd/f17 0 2024-03-30T04:28:12.861 INFO:tasks.workunit.client.1.smithi135.stdout:9/100: rename d7 to d7/d1a 22 2024-03-30T04:28:12.863 INFO:tasks.workunit.client.1.smithi135.stdout:7/216: dread d4/f29 [0,4194304] 0 2024-03-30T04:28:12.863 INFO:tasks.workunit.client.1.smithi135.stdout:4/97: dread d10/f17 [0,4194304] 0 2024-03-30T04:28:12.864 INFO:tasks.workunit.client.1.smithi135.stdout:6/102: dread - d2/dd/d14/f15 zero size 2024-03-30T04:28:12.864 INFO:tasks.workunit.client.1.smithi135.stdout:4/98: readlink l2 0 2024-03-30T04:28:12.867 INFO:tasks.workunit.client.1.smithi135.stdout:3/127: mknod d3/d6/d22/c26 0 2024-03-30T04:28:12.868 INFO:tasks.workunit.client.1.smithi135.stdout:4/99: creat d10/d1a/f23 x:0 0 0 2024-03-30T04:28:12.870 INFO:tasks.workunit.client.1.smithi135.stdout:4/100: dread - d10/f14 zero size 2024-03-30T04:28:12.873 INFO:tasks.workunit.client.1.smithi135.stdout:4/101: read fb [1260216,24483] 0 2024-03-30T04:28:12.874 INFO:tasks.workunit.client.1.smithi135.stdout:4/102: stat l2 0 2024-03-30T04:28:12.876 INFO:tasks.workunit.client.1.smithi135.stdout:7/217: mkdir d4/db/d1a/d2a/d2c/d2e/d46 0 2024-03-30T04:28:12.883 INFO:tasks.workunit.client.1.smithi135.stdout:4/103: unlink ff 0 2024-03-30T04:28:12.886 INFO:tasks.workunit.client.1.smithi135.stdout:7/218: write d4/db/de/d15/f45 [209993,70104] 0 2024-03-30T04:28:12.889 INFO:tasks.workunit.client.0.smithi123.stdout:9/485: sync 2024-03-30T04:28:12.889 INFO:tasks.workunit.client.0.smithi123.stdout:7/517: sync 2024-03-30T04:28:12.890 INFO:tasks.workunit.client.1.smithi135.stdout:0/198: write d0/d1/d16/f28 [428627,87057] 0 2024-03-30T04:28:12.891 INFO:tasks.workunit.client.1.smithi135.stdout:4/104: mknod d10/c24 0 2024-03-30T04:28:12.891 INFO:tasks.workunit.client.1.smithi135.stdout:3/128: read d3/d10/d16/f1b [419800,25032] 0 2024-03-30T04:28:12.891 INFO:tasks.workunit.client.1.smithi135.stdout:9/101: dwrite d7/da/f16 [0,4194304] 0 2024-03-30T04:28:12.895 INFO:tasks.workunit.client.0.smithi123.stdout:9/486: mkdir d6/d1f/d45/d97/d69/daa/dac 0 2024-03-30T04:28:12.897 INFO:tasks.workunit.client.1.smithi135.stdout:9/102: readlink d7/da/ld 0 2024-03-30T04:28:12.899 INFO:tasks.workunit.client.1.smithi135.stdout:4/105: fdatasync d10/d1a/f21 0 2024-03-30T04:28:12.901 INFO:tasks.workunit.client.0.smithi123.stdout:9/487: mkdir d6/dad 0 2024-03-30T04:28:12.901 INFO:tasks.workunit.client.1.smithi135.stdout:0/199: mkdir d0/d13/d1d/d3b 0 2024-03-30T04:28:12.901 INFO:tasks.workunit.client.1.smithi135.stdout:4/106: dread - d10/d1a/f21 zero size 2024-03-30T04:28:12.901 INFO:tasks.workunit.client.1.smithi135.stdout:6/103: dwrite d2/f12 [0,4194304] 0 2024-03-30T04:28:12.902 INFO:tasks.workunit.client.0.smithi123.stdout:9/488: chown d6/d23/c35 1058980 1 2024-03-30T04:28:12.903 INFO:tasks.workunit.client.1.smithi135.stdout:4/107: stat fa 0 2024-03-30T04:28:12.906 INFO:tasks.workunit.client.0.smithi123.stdout:9/489: truncate d6/f26 1611400 0 2024-03-30T04:28:12.910 INFO:tasks.workunit.client.1.smithi135.stdout:3/129: truncate d3/f4 1387950 0 2024-03-30T04:28:12.912 INFO:tasks.workunit.client.1.smithi135.stdout:3/130: fsync d3/db/fe 0 2024-03-30T04:28:12.914 INFO:tasks.workunit.client.1.smithi135.stdout:3/131: stat d3/d10/d16 0 2024-03-30T04:28:12.915 INFO:tasks.workunit.client.0.smithi123.stdout:4/513: truncate d2/f20 899900 0 2024-03-30T04:28:12.916 INFO:tasks.workunit.client.1.smithi135.stdout:3/132: fdatasync d3/db/fe 0 2024-03-30T04:28:12.917 INFO:tasks.workunit.client.1.smithi135.stdout:3/133: dread - d3/db/f19 zero size 2024-03-30T04:28:12.918 INFO:tasks.workunit.client.0.smithi123.stdout:4/514: readlink d2/d4/l3d 0 2024-03-30T04:28:12.919 INFO:tasks.workunit.client.0.smithi123.stdout:4/515: stat d2/d4/d2c/d42/l8c 0 2024-03-30T04:28:12.920 INFO:tasks.workunit.client.0.smithi123.stdout:7/518: rename d2/da/d22/f2f to d2/da/d11/fb1 0 2024-03-30T04:28:12.921 INFO:tasks.workunit.client.0.smithi123.stdout:4/516: write d2/d34/f90 [953822,31131] 0 2024-03-30T04:28:12.922 INFO:tasks.workunit.client.1.smithi135.stdout:7/219: rmdir d4/db/d1a/d2a/d2c/d2e/d42 0 2024-03-30T04:28:12.924 INFO:tasks.workunit.client.1.smithi135.stdout:0/200: getdents d0/d13/d1e/d3a 0 2024-03-30T04:28:12.926 INFO:tasks.workunit.client.1.smithi135.stdout:7/220: dread d4/db/de/f30 [0,4194304] 0 2024-03-30T04:28:12.926 INFO:tasks.workunit.client.1.smithi135.stdout:6/104: link d2/fa d2/dd/d14/f1e 0 2024-03-30T04:28:12.927 INFO:tasks.workunit.client.1.smithi135.stdout:0/201: stat d0/d5/d9/f2e 0 2024-03-30T04:28:12.928 INFO:tasks.workunit.client.1.smithi135.stdout:9/103: rename d7/da to d7/d1b 0 2024-03-30T04:28:12.934 INFO:tasks.workunit.client.1.smithi135.stdout:4/108: getdents d10 0 2024-03-30T04:28:12.939 INFO:tasks.workunit.client.0.smithi123.stdout:7/519: dwrite d2/dc/d1c/f23 [0,4194304] 0 2024-03-30T04:28:12.939 INFO:tasks.workunit.client.1.smithi135.stdout:0/202: symlink d0/d1/l3c 0 2024-03-30T04:28:12.940 INFO:tasks.workunit.client.1.smithi135.stdout:7/221: symlink d4/db/d1a/l47 0 2024-03-30T04:28:12.941 INFO:tasks.workunit.client.1.smithi135.stdout:3/134: dread f2 [0,4194304] 0 2024-03-30T04:28:12.942 INFO:tasks.workunit.client.1.smithi135.stdout:9/104: creat d7/d1b/f1c x:0 0 0 2024-03-30T04:28:12.942 INFO:tasks.workunit.client.0.smithi123.stdout:6/623: mkdir d10/d82/dd6 0 2024-03-30T04:28:12.943 INFO:tasks.workunit.client.1.smithi135.stdout:4/109: symlink d10/d1a/l25 0 2024-03-30T04:28:12.944 INFO:tasks.workunit.client.0.smithi123.stdout:6/624: stat d10/lb0 0 2024-03-30T04:28:12.945 INFO:tasks.workunit.client.1.smithi135.stdout:0/203: write d0/d5/d9/f2e [1333429,114595] 0 2024-03-30T04:28:12.947 INFO:tasks.workunit.client.0.smithi123.stdout:6/625: chown d10/d20/da2/fb1 3 1 2024-03-30T04:28:12.948 INFO:tasks.workunit.client.1.smithi135.stdout:6/105: dread d2/f11 [0,4194304] 0 2024-03-30T04:28:12.949 INFO:tasks.workunit.client.0.smithi123.stdout:9/490: dread d6/f7e [0,4194304] 0 2024-03-30T04:28:12.951 INFO:tasks.workunit.client.1.smithi135.stdout:7/222: symlink d4/db/d1a/d2a/d2c/d2e/l48 0 2024-03-30T04:28:12.952 INFO:tasks.workunit.client.0.smithi123.stdout:9/491: write d6/d49/f65 [1655868,104394] 0 2024-03-30T04:28:12.953 INFO:tasks.workunit.client.1.smithi135.stdout:9/105: symlink d7/d1b/l1d 0 2024-03-30T04:28:12.956 INFO:tasks.workunit.client.1.smithi135.stdout:4/110: write d10/d1a/f22 [531021,123716] 0 2024-03-30T04:28:12.956 INFO:tasks.workunit.client.1.smithi135.stdout:3/135: creat d3/d10/d16/d23/f27 x:0 0 0 2024-03-30T04:28:12.958 INFO:tasks.workunit.client.1.smithi135.stdout:6/106: symlink d2/dd/l1f 0 2024-03-30T04:28:12.958 INFO:tasks.workunit.client.1.smithi135.stdout:3/136: stat f2 0 2024-03-30T04:28:12.959 INFO:tasks.workunit.client.1.smithi135.stdout:4/111: fsync d10/d1a/f23 0 2024-03-30T04:28:12.959 INFO:tasks.workunit.client.1.smithi135.stdout:7/223: truncate d4/db/de/f30 641604 0 2024-03-30T04:28:12.960 INFO:tasks.workunit.client.1.smithi135.stdout:6/107: write d2/f8 [572224,68049] 0 2024-03-30T04:28:12.961 INFO:tasks.workunit.client.1.smithi135.stdout:3/137: symlink d3/db/l28 0 2024-03-30T04:28:12.964 INFO:tasks.workunit.client.1.smithi135.stdout:4/112: mknod d10/c26 0 2024-03-30T04:28:12.965 INFO:tasks.workunit.client.1.smithi135.stdout:6/108: mknod d2/dd/c20 0 2024-03-30T04:28:12.965 INFO:tasks.workunit.client.1.smithi135.stdout:7/224: rename d4/db/d1a to d4/db/d1a/d2a/d2c/d49 22 2024-03-30T04:28:12.966 INFO:tasks.workunit.client.1.smithi135.stdout:4/113: truncate d10/d1a/f22 1690029 0 2024-03-30T04:28:12.967 INFO:tasks.workunit.client.1.smithi135.stdout:6/109: read d2/dd/f1a [1788238,52599] 0 2024-03-30T04:28:12.967 INFO:tasks.workunit.client.1.smithi135.stdout:7/225: readlink d4/db/de/d15/l1e 0 2024-03-30T04:28:12.970 INFO:tasks.workunit.client.1.smithi135.stdout:6/110: write d2/f13 [2020471,37829] 0 2024-03-30T04:28:12.970 INFO:tasks.workunit.client.1.smithi135.stdout:4/114: creat d10/d1a/f27 x:0 0 0 2024-03-30T04:28:12.970 INFO:tasks.workunit.client.1.smithi135.stdout:7/226: write d4/f8 [5103867,104551] 0 2024-03-30T04:28:12.972 INFO:tasks.workunit.client.1.smithi135.stdout:4/115: write f3 [2075320,21115] 0 2024-03-30T04:28:12.975 INFO:tasks.workunit.client.1.smithi135.stdout:6/111: write d2/dd/f17 [821053,93524] 0 2024-03-30T04:28:12.978 INFO:tasks.workunit.client.1.smithi135.stdout:7/227: symlink d4/db/de/d15/l4a 0 2024-03-30T04:28:12.978 INFO:tasks.workunit.client.1.smithi135.stdout:8/173: sync 2024-03-30T04:28:12.978 INFO:tasks.workunit.client.1.smithi135.stdout:2/129: sync 2024-03-30T04:28:12.978 INFO:tasks.workunit.client.1.smithi135.stdout:4/116: write d10/f17 [619142,62553] 0 2024-03-30T04:28:12.979 INFO:tasks.workunit.client.1.smithi135.stdout:6/112: mkdir d2/dd/d18/d21 0 2024-03-30T04:28:12.981 INFO:tasks.workunit.client.1.smithi135.stdout:8/174: write f5 [3338276,84899] 0 2024-03-30T04:28:12.982 INFO:tasks.workunit.client.1.smithi135.stdout:4/117: mknod d10/d1a/c28 0 2024-03-30T04:28:12.983 INFO:tasks.workunit.client.1.smithi135.stdout:2/130: symlink d4/da/dd/d12/l29 0 2024-03-30T04:28:12.984 INFO:tasks.workunit.client.1.smithi135.stdout:3/138: dwrite d3/db/fe [0,4194304] 0 2024-03-30T04:28:12.985 INFO:tasks.workunit.client.1.smithi135.stdout:8/175: stat d6/f18 0 2024-03-30T04:28:12.987 INFO:tasks.workunit.client.1.smithi135.stdout:2/131: fsync d4/da/f21 0 2024-03-30T04:28:12.992 INFO:tasks.workunit.client.1.smithi135.stdout:3/139: chown d3/f11 744974 1 2024-03-30T04:28:12.996 INFO:tasks.workunit.client.1.smithi135.stdout:4/118: symlink d10/d1a/l29 0 2024-03-30T04:28:12.997 INFO:tasks.workunit.client.1.smithi135.stdout:4/119: chown fb 105 1 2024-03-30T04:28:12.998 INFO:tasks.workunit.client.1.smithi135.stdout:4/120: readlink d10/d1a/l1d 0 2024-03-30T04:28:12.999 INFO:tasks.workunit.client.1.smithi135.stdout:4/121: stat d10/f14 0 2024-03-30T04:28:12.999 INFO:tasks.workunit.client.1.smithi135.stdout:7/228: rename d4/db/de/d15 to d4/db/d1a/d4b 0 2024-03-30T04:28:13.003 INFO:tasks.workunit.client.1.smithi135.stdout:7/229: write d4/db/de/f28 [732053,53949] 0 2024-03-30T04:28:13.004 INFO:tasks.workunit.client.1.smithi135.stdout:6/113: truncate d2/f12 1220581 0 2024-03-30T04:28:13.006 INFO:tasks.workunit.client.1.smithi135.stdout:7/230: fsync d4/db/f26 0 2024-03-30T04:28:13.006 INFO:tasks.workunit.client.1.smithi135.stdout:6/114: read d2/fc [262073,51911] 0 2024-03-30T04:28:13.009 INFO:tasks.workunit.client.1.smithi135.stdout:6/115: fdatasync d2/f4 0 2024-03-30T04:28:13.009 INFO:tasks.workunit.client.1.smithi135.stdout:8/176: dwrite d6/d8/f1d [4194304,4194304] 0 2024-03-30T04:28:13.010 INFO:tasks.workunit.client.1.smithi135.stdout:8/177: chown f5 2032768070 1 2024-03-30T04:28:13.012 INFO:tasks.workunit.client.1.smithi135.stdout:7/231: mknod d4/db/d1a/d2a/d2c/d2e/c4c 0 2024-03-30T04:28:13.013 INFO:tasks.workunit.client.1.smithi135.stdout:6/116: creat d2/dd/d18/f22 x:0 0 0 2024-03-30T04:28:13.017 INFO:tasks.workunit.client.1.smithi135.stdout:4/122: dread d10/f17 [0,4194304] 0 2024-03-30T04:28:13.018 INFO:tasks.workunit.client.1.smithi135.stdout:6/117: symlink d2/l23 0 2024-03-30T04:28:13.019 INFO:tasks.workunit.client.1.smithi135.stdout:4/123: readlink d10/l15 0 2024-03-30T04:28:13.020 INFO:tasks.workunit.client.1.smithi135.stdout:6/118: readlink d2/dd/l1f 0 2024-03-30T04:28:13.021 INFO:tasks.workunit.client.0.smithi123.stdout:7/520: creat d2/dc/d1c/fb2 x:0 0 0 2024-03-30T04:28:13.022 INFO:tasks.workunit.client.0.smithi123.stdout:6/626: creat d10/d82/d8e/dc7/fd7 x:0 0 0 2024-03-30T04:28:13.024 INFO:tasks.workunit.client.0.smithi123.stdout:4/517: dwrite d2/d4/f22 [0,4194304] 0 2024-03-30T04:28:13.024 INFO:tasks.workunit.client.0.smithi123.stdout:9/492: rename d6/d1f/l8a to d6/lae 0 2024-03-30T04:28:13.025 INFO:tasks.workunit.client.1.smithi135.stdout:6/119: fsync d2/f13 0 2024-03-30T04:28:13.027 INFO:tasks.workunit.client.0.smithi123.stdout:6/627: dread - d10/d82/dc1/d45/fd3 zero size 2024-03-30T04:28:13.028 INFO:tasks.workunit.client.0.smithi123.stdout:7/521: unlink d2/da/d11/d61/f57 0 2024-03-30T04:28:13.029 INFO:tasks.workunit.client.0.smithi123.stdout:4/518: creat d2/d34/fb4 x:0 0 0 2024-03-30T04:28:13.029 INFO:tasks.workunit.client.0.smithi123.stdout:6/628: chown d10/fb2 214242534 1 2024-03-30T04:28:13.032 INFO:tasks.workunit.client.1.smithi135.stdout:8/178: dread f5 [0,4194304] 0 2024-03-30T04:28:13.033 INFO:tasks.workunit.client.0.smithi123.stdout:7/522: mknod d2/da/d11/d34/cb3 0 2024-03-30T04:28:13.034 INFO:tasks.workunit.client.0.smithi123.stdout:4/519: mkdir d2/d4/d7/d63/db5 0 2024-03-30T04:28:13.035 INFO:tasks.workunit.client.0.smithi123.stdout:7/523: fdatasync d2/f8 0 2024-03-30T04:28:13.036 INFO:tasks.workunit.client.0.smithi123.stdout:4/520: readlink d2/d4/d7/l31 0 2024-03-30T04:28:13.042 INFO:tasks.workunit.client.1.smithi135.stdout:4/124: dwrite d10/d1a/f1f [0,4194304] 0 2024-03-30T04:28:13.043 INFO:tasks.workunit.client.1.smithi135.stdout:4/125: readlink l7 0 2024-03-30T04:28:13.045 INFO:tasks.workunit.client.1.smithi135.stdout:6/120: dwrite d2/f13 [0,4194304] 0 2024-03-30T04:28:13.047 INFO:tasks.workunit.client.0.smithi123.stdout:9/493: dwrite d6/d1f/d45/fa0 [0,4194304] 0 2024-03-30T04:28:13.056 INFO:tasks.workunit.client.0.smithi123.stdout:6/629: dwrite d10/f96 [0,4194304] 0 2024-03-30T04:28:13.068 INFO:tasks.workunit.client.0.smithi123.stdout:4/521: dread d2/d4/f16 [0,4194304] 0 2024-03-30T04:28:13.068 INFO:tasks.workunit.client.0.smithi123.stdout:7/524: dwrite d2/da/d22/f86 [0,4194304] 0 2024-03-30T04:28:13.073 INFO:tasks.workunit.client.0.smithi123.stdout:5/523: sync 2024-03-30T04:28:13.078 INFO:tasks.workunit.client.0.smithi123.stdout:7/525: creat d2/da/d11/fb4 x:0 0 0 2024-03-30T04:28:13.085 INFO:tasks.workunit.client.0.smithi123.stdout:6/630: dwrite d10/d82/d8e/dc7/f1e [4194304,4194304] 0 2024-03-30T04:28:13.090 INFO:tasks.workunit.client.0.smithi123.stdout:7/526: getdents d2/dc/d83 0 2024-03-30T04:28:13.094 INFO:tasks.workunit.client.0.smithi123.stdout:9/494: write d6/d1f/d45/d4c/f74 [1041909,56047] 0 2024-03-30T04:28:13.094 INFO:tasks.workunit.client.0.smithi123.stdout:5/524: getdents dd/d19/d1f/d3c/d72/d8c 0 2024-03-30T04:28:13.097 INFO:tasks.workunit.client.0.smithi123.stdout:6/631: creat d10/d82/dd6/fd8 x:0 0 0 2024-03-30T04:28:13.100 INFO:tasks.workunit.client.0.smithi123.stdout:7/527: rename d2/da/d2d to d2/da/d11/d61/d55/d59/db5 0 2024-03-30T04:28:13.103 INFO:tasks.workunit.client.0.smithi123.stdout:9/495: unlink d6/d7c/d82/c85 0 2024-03-30T04:28:13.109 INFO:tasks.workunit.client.0.smithi123.stdout:7/528: rename d2/da/d11/d34/d76/l7a to d2/da/d11/d61/d55/lb6 0 2024-03-30T04:28:13.110 INFO:tasks.workunit.client.0.smithi123.stdout:6/632: getdents d10/d82/d9a/db3 0 2024-03-30T04:28:13.115 INFO:tasks.workunit.client.0.smithi123.stdout:4/522: dread d2/d4/d2c/d42/f8b [0,4194304] 0 2024-03-30T04:28:13.120 INFO:tasks.workunit.client.0.smithi123.stdout:4/523: creat d2/d4/d2c/d42/d35/d9b/fb6 x:0 0 0 2024-03-30T04:28:13.124 INFO:tasks.workunit.client.0.smithi123.stdout:4/524: mkdir d2/d4/d2c/d42/d35/d9b/db7 0 2024-03-30T04:28:13.126 INFO:tasks.workunit.client.0.smithi123.stdout:4/525: dread - d2/d34/f93 zero size 2024-03-30T04:28:13.127 INFO:tasks.workunit.client.0.smithi123.stdout:7/529: dwrite d2/f87 [0,4194304] 0 2024-03-30T04:28:13.130 INFO:tasks.workunit.client.0.smithi123.stdout:4/526: chown d2/d34/f90 2 1 2024-03-30T04:28:13.132 INFO:tasks.workunit.client.0.smithi123.stdout:6/633: dwrite d10/d82/d8e/dc7/fa4 [0,4194304] 0 2024-03-30T04:28:13.133 INFO:tasks.workunit.client.0.smithi123.stdout:9/496: dread d6/d1f/f29 [4194304,4194304] 0 2024-03-30T04:28:13.134 INFO:tasks.workunit.client.0.smithi123.stdout:7/530: fsync d2/dc/d56/f67 0 2024-03-30T04:28:13.136 INFO:tasks.workunit.client.0.smithi123.stdout:7/531: write f0 [824969,51016] 0 2024-03-30T04:28:13.138 INFO:tasks.workunit.client.0.smithi123.stdout:4/527: rename d2/d4/d2c/d42/d35/d5d to d2/d4/d2c/d42/d35/d5d/db8 22 2024-03-30T04:28:13.139 INFO:tasks.workunit.client.0.smithi123.stdout:7/532: symlink d2/lb7 0 2024-03-30T04:28:13.140 INFO:tasks.workunit.client.0.smithi123.stdout:4/528: chown d2/d4/ff 254287 1 2024-03-30T04:28:13.144 INFO:tasks.workunit.client.0.smithi123.stdout:4/529: write d2/d4/d2c/d42/dad/d91/d9d/fa6 [456844,128886] 0 2024-03-30T04:28:13.149 INFO:tasks.workunit.client.0.smithi123.stdout:7/533: truncate d2/da/d11/f45 1451397 0 2024-03-30T04:28:13.154 INFO:tasks.workunit.client.0.smithi123.stdout:4/530: getdents d2/d4/d2c 0 2024-03-30T04:28:13.157 INFO:tasks.workunit.client.0.smithi123.stdout:6/634: dwrite d10/d20/f58 [0,4194304] 0 2024-03-30T04:28:13.160 INFO:tasks.workunit.client.0.smithi123.stdout:9/497: dwrite d6/fd [4194304,4194304] 0 2024-03-30T04:28:13.161 INFO:tasks.workunit.client.0.smithi123.stdout:7/534: rename d2/da/d11/d61/d55/d59/db5/f39 to d2/da/fb8 0 2024-03-30T04:28:13.166 INFO:tasks.workunit.client.0.smithi123.stdout:4/531: mknod d2/d4/d2c/d42/cb9 0 2024-03-30T04:28:13.166 INFO:tasks.workunit.client.0.smithi123.stdout:6/635: creat d10/d82/dc1/dd2/fd9 x:0 0 0 2024-03-30T04:28:13.167 INFO:tasks.workunit.client.1.smithi135.stdout:8/179: mkdir d6/d8/d14/d32 0 2024-03-30T04:28:13.168 INFO:tasks.workunit.client.0.smithi123.stdout:4/532: write d2/d34/f93 [455205,20753] 0 2024-03-30T04:28:13.169 INFO:tasks.workunit.client.0.smithi123.stdout:6/636: dread - d10/d82/dc1/d6a/fae zero size 2024-03-30T04:28:13.170 INFO:tasks.workunit.client.0.smithi123.stdout:9/498: creat d6/d1f/d45/d4c/faf x:0 0 0 2024-03-30T04:28:13.171 INFO:tasks.workunit.client.1.smithi135.stdout:4/126: rename d10/c11 to d10/c2a 0 2024-03-30T04:28:13.174 INFO:tasks.workunit.client.0.smithi123.stdout:4/533: write d2/d4/d2c/d42/d35/d5d/fae [888806,126372] 0 2024-03-30T04:28:13.176 INFO:tasks.workunit.client.1.smithi135.stdout:4/127: write d10/f17 [1286689,85904] 0 2024-03-30T04:28:13.178 INFO:tasks.workunit.client.1.smithi135.stdout:7/232: truncate d4/db/f1c 3329097 0 2024-03-30T04:28:13.179 INFO:tasks.workunit.client.1.smithi135.stdout:8/180: creat d6/d8/d15/d2a/f33 x:0 0 0 2024-03-30T04:28:13.179 INFO:tasks.workunit.client.1.smithi135.stdout:6/121: link d2/f11 d2/dd/d18/f24 0 2024-03-30T04:28:13.181 INFO:tasks.workunit.client.1.smithi135.stdout:4/128: mkdir d10/d1a/d2b 0 2024-03-30T04:28:13.182 INFO:tasks.workunit.client.0.smithi123.stdout:6/637: dread - d10/d82/d9a/db3/fcf zero size 2024-03-30T04:28:13.184 INFO:tasks.workunit.client.1.smithi135.stdout:8/181: write f5 [5045645,61666] 0 2024-03-30T04:28:13.184 INFO:tasks.workunit.client.1.smithi135.stdout:6/122: creat d2/dd/d18/f25 x:0 0 0 2024-03-30T04:28:13.185 INFO:tasks.workunit.client.1.smithi135.stdout:4/129: write d10/d1a/f23 [1040880,116367] 0 2024-03-30T04:28:13.188 INFO:tasks.workunit.client.0.smithi123.stdout:9/499: dread d6/d1f/d45/d4c/f74 [0,4194304] 0 2024-03-30T04:28:13.189 INFO:tasks.workunit.client.0.smithi123.stdout:6/638: mkdir d10/d82/dc1/dd2/dda 0 2024-03-30T04:28:13.191 INFO:tasks.workunit.client.0.smithi123.stdout:9/500: rmdir d6/d1f/d72 39 2024-03-30T04:28:13.193 INFO:tasks.workunit.client.0.smithi123.stdout:7/535: dwrite d2/da/d11/d34/f70 [0,4194304] 0 2024-03-30T04:28:13.193 INFO:tasks.workunit.client.0.smithi123.stdout:9/501: chown d6/d1f/f29 2978 1 2024-03-30T04:28:13.197 INFO:tasks.workunit.client.0.smithi123.stdout:9/502: mknod d6/d1f/d45/d97/d69/daa/cb0 0 2024-03-30T04:28:13.199 INFO:tasks.workunit.client.0.smithi123.stdout:9/503: readlink d6/d1f/d72/la7 0 2024-03-30T04:28:13.202 INFO:tasks.workunit.client.1.smithi135.stdout:6/123: dread d2/dd/d18/f24 [0,4194304] 0 2024-03-30T04:28:13.203 INFO:tasks.workunit.client.0.smithi123.stdout:9/504: rmdir d6/d1f/d72/d7d 39 2024-03-30T04:28:13.205 INFO:tasks.workunit.client.1.smithi135.stdout:6/124: symlink d2/dd/d14/l26 0 2024-03-30T04:28:13.206 INFO:tasks.workunit.client.1.smithi135.stdout:6/125: chown d2/dd/d18/f25 16834 1 2024-03-30T04:28:13.206 INFO:tasks.workunit.client.1.smithi135.stdout:6/126: stat d2/dd/d18/l1d 0 2024-03-30T04:28:13.207 INFO:tasks.workunit.client.1.smithi135.stdout:5/181: sync 2024-03-30T04:28:13.214 INFO:tasks.workunit.client.1.smithi135.stdout:5/182: rename d1/d7/d1a/d1f/l26 to d1/d3/d2b/l33 0 2024-03-30T04:28:13.214 INFO:tasks.workunit.client.1.smithi135.stdout:4/130: dread d10/d1a/f23 [0,4194304] 0 2024-03-30T04:28:13.216 INFO:tasks.workunit.client.0.smithi123.stdout:7/536: dwrite d2/dc/d56/f67 [0,4194304] 0 2024-03-30T04:28:13.219 INFO:tasks.workunit.client.0.smithi123.stdout:4/534: dread d2/d34/f7c [0,4194304] 0 2024-03-30T04:28:13.220 INFO:tasks.workunit.client.0.smithi123.stdout:7/537: creat d2/da/d11/d61/d55/d59/db5/dab/fb9 x:0 0 0 2024-03-30T04:28:13.225 INFO:tasks.workunit.client.0.smithi123.stdout:7/538: mknod d2/da/d22/cba 0 2024-03-30T04:28:13.230 INFO:tasks.workunit.client.0.smithi123.stdout:7/539: creat d2/da/d11/d34/d85/d8e/fbb x:0 0 0 2024-03-30T04:28:13.232 INFO:tasks.workunit.client.0.smithi123.stdout:6/639: dread d10/d20/d3f/f65 [0,4194304] 0 2024-03-30T04:28:13.233 INFO:tasks.workunit.client.0.smithi123.stdout:4/535: dread d2/d4/d2c/d42/d35/d5d/d78/f83 [0,4194304] 0 2024-03-30T04:28:13.236 INFO:tasks.workunit.client.0.smithi123.stdout:3/608: sync 2024-03-30T04:28:13.240 INFO:tasks.workunit.client.0.smithi123.stdout:7/540: write d2/da/d11/f45 [1817598,11037] 0 2024-03-30T04:28:13.241 INFO:tasks.workunit.client.0.smithi123.stdout:4/536: mkdir d2/d4/d2c/d42/d35/d9b/dba 0 2024-03-30T04:28:13.242 INFO:tasks.workunit.client.0.smithi123.stdout:9/505: dwrite d6/d23/f8d [0,4194304] 0 2024-03-30T04:28:13.246 INFO:tasks.workunit.client.0.smithi123.stdout:3/609: creat d13/d24/d39/d79/d90/fc2 x:0 0 0 2024-03-30T04:28:13.246 INFO:tasks.workunit.client.0.smithi123.stdout:7/541: mkdir d2/dc/d49/dbc 0 2024-03-30T04:28:13.256 INFO:tasks.workunit.client.1.smithi135.stdout:8/182: write d6/f25 [12841,115906] 0 2024-03-30T04:28:13.256 INFO:tasks.workunit.client.1.smithi135.stdout:7/233: write d4/db/d1a/d4b/f31 [1433749,23532] 0 2024-03-30T04:28:13.259 INFO:tasks.workunit.client.1.smithi135.stdout:7/234: write d4/db/de/f1d [2262524,79569] 0 2024-03-30T04:28:13.260 INFO:tasks.workunit.client.0.smithi123.stdout:6/640: dwrite d10/d20/f3b [4194304,4194304] 0 2024-03-30T04:28:13.261 INFO:tasks.workunit.client.1.smithi135.stdout:7/235: readlink d4/db/l18 0 2024-03-30T04:28:13.262 INFO:tasks.workunit.client.1.smithi135.stdout:8/183: creat d6/d8/d15/d1f/f34 x:0 0 0 2024-03-30T04:28:13.265 INFO:tasks.workunit.client.1.smithi135.stdout:8/184: write d6/fa [5217794,7565] 0 2024-03-30T04:28:13.269 INFO:tasks.workunit.client.1.smithi135.stdout:8/185: readlink d6/d8/d15/l1a 0 2024-03-30T04:28:13.269 INFO:tasks.workunit.client.1.smithi135.stdout:8/186: chown d6 213499 1 2024-03-30T04:28:13.272 INFO:tasks.workunit.client.1.smithi135.stdout:7/236: rename d4/db/d1a/d2a to d4/d4d 0 2024-03-30T04:28:13.296 INFO:tasks.workunit.client.0.smithi123.stdout:9/506: dread d6/d1f/f29 [0,4194304] 0 2024-03-30T04:28:13.303 INFO:tasks.workunit.client.0.smithi123.stdout:0/595: sync 2024-03-30T04:28:13.303 INFO:tasks.workunit.client.0.smithi123.stdout:8/511: sync 2024-03-30T04:28:13.303 INFO:tasks.workunit.client.0.smithi123.stdout:1/671: sync 2024-03-30T04:28:13.303 INFO:tasks.workunit.client.0.smithi123.stdout:2/646: sync 2024-03-30T04:28:13.303 INFO:tasks.workunit.client.0.smithi123.stdout:5/525: sync 2024-03-30T04:28:13.307 INFO:tasks.workunit.client.0.smithi123.stdout:2/647: write d0/d1f/d16/f26 [2196239,24243] 0 2024-03-30T04:28:13.336 INFO:tasks.workunit.client.0.smithi123.stdout:2/648: dread d0/d1f/d1c/d40/f5a [0,4194304] 0 2024-03-30T04:28:13.337 INFO:tasks.workunit.client.1.smithi135.stdout:1/144: sync 2024-03-30T04:28:13.565 INFO:tasks.workunit.client.1.smithi135.stdout:2/132: sync 2024-03-30T04:28:13.565 INFO:tasks.workunit.client.1.smithi135.stdout:0/204: sync 2024-03-30T04:28:13.565 INFO:tasks.workunit.client.1.smithi135.stdout:9/106: sync 2024-03-30T04:28:13.568 INFO:tasks.workunit.client.1.smithi135.stdout:9/107: chown d7/d1b/f16 11549114 1 2024-03-30T04:28:13.577 INFO:tasks.workunit.client.0.smithi123.stdout:3/610: symlink d13/d24/d31/d23/d3a/lc3 0 2024-03-30T04:28:13.577 INFO:tasks.workunit.client.0.smithi123.stdout:4/537: link d2/d26/c6d d2/cbb 0 2024-03-30T04:28:13.580 INFO:tasks.workunit.client.1.smithi135.stdout:2/133: dwrite d4/d6/f16 [0,4194304] 0 2024-03-30T04:28:13.580 INFO:tasks.workunit.client.0.smithi123.stdout:6/641: rename d10/d20/d3f/da3/lb7 to d10/d82/dd6/ldb 0 2024-03-30T04:28:13.584 INFO:tasks.workunit.client.0.smithi123.stdout:0/596: mknod d6/dd/d18/d1f/cc4 0 2024-03-30T04:28:13.585 INFO:tasks.workunit.client.0.smithi123.stdout:5/526: rmdir dd/d19/d5f 39 2024-03-30T04:28:13.588 INFO:tasks.workunit.client.0.smithi123.stdout:8/512: chown da/dc/d14/d18/d62/c89 11342 1 2024-03-30T04:28:13.588 INFO:tasks.workunit.client.0.smithi123.stdout:1/672: mkdir db/dd/d42/d46/d56/da2/de0 0 2024-03-30T04:28:13.591 INFO:tasks.workunit.client.0.smithi123.stdout:2/649: write d0/d1f/d1c/d80/dbb/d97/fc1 [1326122,50556] 0 2024-03-30T04:28:13.592 INFO:tasks.workunit.client.0.smithi123.stdout:4/538: stat d2/d34/c68 0 2024-03-30T04:28:13.595 INFO:tasks.workunit.client.0.smithi123.stdout:4/539: chown d2/d4/d2c/d42/l9c 44989 1 2024-03-30T04:28:13.595 INFO:tasks.workunit.client.0.smithi123.stdout:6/642: chown d10/f38 215 1 2024-03-30T04:28:13.596 INFO:tasks.workunit.client.0.smithi123.stdout:5/527: mknod dd/d19/d3e/d5b/d62/ca3 0 2024-03-30T04:28:13.600 INFO:tasks.workunit.client.0.smithi123.stdout:2/650: truncate d0/d1f/d1c/d39/d4b/fa0 885264 0 2024-03-30T04:28:13.600 INFO:tasks.workunit.client.0.smithi123.stdout:5/528: write dd/d19/d1f/f82 [1719938,67326] 0 2024-03-30T04:28:13.601 INFO:tasks.workunit.client.0.smithi123.stdout:6/643: truncate d10/d20/d3f/f91 4525480 0 2024-03-30T04:28:13.601 INFO:tasks.workunit.client.0.smithi123.stdout:8/513: dread da/dc/f32 [0,4194304] 0 2024-03-30T04:28:13.603 INFO:tasks.workunit.client.0.smithi123.stdout:7/542: link d2/da/d11/f58 d2/fbd 0 2024-03-30T04:28:13.604 INFO:tasks.workunit.client.0.smithi123.stdout:0/597: creat d6/dd/d18/d2f/fc5 x:0 0 0 2024-03-30T04:28:13.604 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:13 smithi135 ceph-mon[44364]: pgmap v40: 65 pgs: 65 active+clean; 1.3 GiB data, 5.4 GiB used, 531 GiB / 536 GiB avail; 38 MiB/s rd, 133 MiB/s wr, 300 op/s 2024-03-30T04:28:13.605 INFO:tasks.workunit.client.0.smithi123.stdout:4/540: creat d2/dab/fbc x:0 0 0 2024-03-30T04:28:13.607 INFO:tasks.workunit.client.0.smithi123.stdout:1/673: creat db/fe1 x:0 0 0 2024-03-30T04:28:13.609 INFO:tasks.workunit.client.0.smithi123.stdout:5/529: creat dd/d19/d3e/d5b/d62/fa4 x:0 0 0 2024-03-30T04:28:13.611 INFO:tasks.workunit.client.0.smithi123.stdout:6/644: rmdir d10/d82/dc1/d6a/d7a/d9f 39 2024-03-30T04:28:13.614 INFO:tasks.workunit.client.0.smithi123.stdout:2/651: rename d0/d1f/d1c/d2a to d0/d1f/d27/d68/dd6 0 2024-03-30T04:28:13.615 INFO:tasks.workunit.client.1.smithi135.stdout:4/131: sync 2024-03-30T04:28:13.615 INFO:tasks.workunit.client.1.smithi135.stdout:5/183: sync 2024-03-30T04:28:13.616 INFO:tasks.workunit.client.1.smithi135.stdout:3/140: sync 2024-03-30T04:28:13.618 INFO:tasks.workunit.client.0.smithi123.stdout:0/598: unlink d6/dd/d18/l49 0 2024-03-30T04:28:13.621 INFO:tasks.workunit.client.0.smithi123.stdout:4/541: creat d2/d26/fbd x:0 0 0 2024-03-30T04:28:13.621 INFO:tasks.workunit.client.0.smithi123.stdout:0/599: fdatasync d6/dd/d18/d2f/fc5 0 2024-03-30T04:28:13.624 INFO:tasks.workunit.client.0.smithi123.stdout:8/514: mkdir da/d99 0 2024-03-30T04:28:13.624 INFO:tasks.workunit.client.0.smithi123.stdout:0/600: readlink d6/dd/d16/l3e 0 2024-03-30T04:28:13.626 INFO:tasks.workunit.client.0.smithi123.stdout:5/530: creat dd/d19/d1f/fa5 x:0 0 0 2024-03-30T04:28:13.626 INFO:tasks.workunit.client.0.smithi123.stdout:8/515: write f6 [666465,66560] 0 2024-03-30T04:28:13.626 INFO:tasks.workunit.client.0.smithi123.stdout:3/611: dread d13/d24/d31/d23/d3a/d3c/f73 [0,4194304] 0 2024-03-30T04:28:13.627 INFO:tasks.workunit.client.1.smithi135.stdout:8/187: mkdir d6/d8/d15/d2a/d35 0 2024-03-30T04:28:13.627 INFO:tasks.workunit.client.0.smithi123.stdout:1/674: dwrite db/dd/d42/d46/d56/da2/fb0 [0,4194304] 0 2024-03-30T04:28:13.631 INFO:tasks.workunit.client.0.smithi123.stdout:0/601: read d6/dd/d31/d35/f42 [1347469,111015] 0 2024-03-30T04:28:13.633 INFO:tasks.workunit.client.0.smithi123.stdout:0/602: fsync d6/dd/d31/d35/f7b 0 2024-03-30T04:28:13.635 INFO:tasks.workunit.client.0.smithi123.stdout:4/542: write d2/d4/d2c/d42/f8b [5045503,28594] 0 2024-03-30T04:28:13.636 INFO:tasks.workunit.client.0.smithi123.stdout:9/507: truncate d6/d49/f75 694178 0 2024-03-30T04:28:13.638 INFO:tasks.workunit.client.0.smithi123.stdout:6/645: fdatasync d10/d82/dc1/d45/f9c 0 2024-03-30T04:28:13.638 INFO:tasks.workunit.client.0.smithi123.stdout:0/603: write d6/dd/d31/d35/f5b [481116,98367] 0 2024-03-30T04:28:13.638 INFO:tasks.workunit.client.1.smithi135.stdout:3/141: dwrite d3/db/fe [0,4194304] 0 2024-03-30T04:28:13.639 INFO:tasks.workunit.client.0.smithi123.stdout:7/543: mknod d2/dc/d7f/cbe 0 2024-03-30T04:28:13.640 INFO:tasks.workunit.client.1.smithi135.stdout:3/142: write d3/f11 [2620681,67422] 0 2024-03-30T04:28:13.643 INFO:tasks.workunit.client.1.smithi135.stdout:3/143: readlink d3/l8 0 2024-03-30T04:28:13.645 INFO:tasks.workunit.client.1.smithi135.stdout:3/144: chown d3/d6/f12 0 1 2024-03-30T04:28:13.646 INFO:tasks.workunit.client.0.smithi123.stdout:0/604: dread - d6/dd/d16/f88 zero size 2024-03-30T04:28:13.654 INFO:tasks.workunit.client.0.smithi123.stdout:7/544: dread - d2/da/d11/d61/d55/d59/db5/dab/fb9 zero size 2024-03-30T04:28:13.655 INFO:tasks.workunit.client.0.smithi123.stdout:9/508: mknod d6/d1f/d45/d97/d69/daa/cb1 0 2024-03-30T04:28:13.655 INFO:tasks.workunit.client.0.smithi123.stdout:4/543: creat d2/d4/d2c/d42/d35/d5d/d78/fbe x:0 0 0 2024-03-30T04:28:13.655 INFO:tasks.workunit.client.0.smithi123.stdout:1/675: creat db/dd/d42/d8f/fe2 x:0 0 0 2024-03-30T04:28:13.656 INFO:tasks.workunit.client.0.smithi123.stdout:5/531: creat dd/d19/d1f/d35/d3a/fa6 x:0 0 0 2024-03-30T04:28:13.656 INFO:tasks.workunit.client.0.smithi123.stdout:8/516: dwrite da/d27/d56/d6c/d6d/f79 [0,4194304] 0 2024-03-30T04:28:13.659 INFO:tasks.workunit.client.0.smithi123.stdout:8/517: fsync da/d27/d56/f65 0 2024-03-30T04:28:13.661 INFO:tasks.workunit.client.0.smithi123.stdout:0/605: fsync d6/dd/d15/d28/d4b/f82 0 2024-03-30T04:28:13.664 INFO:tasks.workunit.client.0.smithi123.stdout:9/509: readlink d6/d1f/l4a 0 2024-03-30T04:28:13.667 INFO:tasks.workunit.client.0.smithi123.stdout:7/545: dread - d2/f41 zero size 2024-03-30T04:28:13.670 INFO:tasks.workunit.client.0.smithi123.stdout:6/646: getdents d10/d82/d9a/db3 0 2024-03-30T04:28:13.670 INFO:tasks.workunit.client.0.smithi123.stdout:1/676: unlink db/dd/d42/d46/d5d/d62/fac 0 2024-03-30T04:28:13.673 INFO:tasks.workunit.client.0.smithi123.stdout:0/606: mkdir d6/dd/d15/d98/dc6 0 2024-03-30T04:28:13.674 INFO:tasks.workunit.client.0.smithi123.stdout:9/510: chown d6/d1f/d72/d7d/f9c 1747894 1 2024-03-30T04:28:13.675 INFO:tasks.workunit.client.0.smithi123.stdout:7/546: symlink d2/da/d22/lbf 0 2024-03-30T04:28:13.676 INFO:tasks.workunit.client.0.smithi123.stdout:4/544: symlink d2/d4/d7/d63/db5/lbf 0 2024-03-30T04:28:13.680 INFO:tasks.workunit.client.0.smithi123.stdout:1/677: mknod db/dd/d42/d46/d56/d5c/ce3 0 2024-03-30T04:28:13.681 INFO:tasks.workunit.client.0.smithi123.stdout:9/511: symlink d6/d1f/d45/d4c/lb2 0 2024-03-30T04:28:13.681 INFO:tasks.workunit.client.0.smithi123.stdout:6/647: mknod d10/d82/dc1/d6a/d7a/cdc 0 2024-03-30T04:28:13.683 INFO:tasks.workunit.client.0.smithi123.stdout:5/532: rename dd/f7d to dd/d19/d1f/d35/d3a/fa7 0 2024-03-30T04:28:13.688 INFO:tasks.workunit.client.0.smithi123.stdout:6/648: symlink d10/d82/d8e/dc7/d5b/ldd 0 2024-03-30T04:28:13.691 INFO:tasks.workunit.client.0.smithi123.stdout:4/545: mkdir d2/d4/d2c/d42/d35/d5d/dc0 0 2024-03-30T04:28:13.699 INFO:tasks.workunit.client.0.smithi123.stdout:6/649: read d10/d82/dc1/d6a/d7a/fa1 [993453,130957] 0 2024-03-30T04:28:13.699 INFO:tasks.workunit.client.0.smithi123.stdout:8/518: truncate da/dc/d14/d18/d21/f45 4933821 0 2024-03-30T04:28:13.700 INFO:tasks.workunit.client.0.smithi123.stdout:4/546: write d2/d4/d2c/d42/d35/d5d/d78/f83 [2570759,47119] 0 2024-03-30T04:28:13.701 INFO:tasks.workunit.client.0.smithi123.stdout:5/533: mkdir dd/d19/d5f/d99/da8 0 2024-03-30T04:28:13.702 INFO:tasks.workunit.client.0.smithi123.stdout:9/512: dwrite d6/d1f/d45/d4c/faf [0,4194304] 0 2024-03-30T04:28:13.703 INFO:tasks.workunit.client.0.smithi123.stdout:7/547: dread d2/da/d22/d3b/f4e [0,4194304] 0 2024-03-30T04:28:13.705 INFO:tasks.workunit.client.0.smithi123.stdout:8/519: chown da/d27/d3c/d2d/f57 23 1 2024-03-30T04:28:13.708 INFO:tasks.workunit.client.0.smithi123.stdout:4/547: readlink d2/d34/l98 0 2024-03-30T04:28:13.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:13 smithi123 ceph-mon[35245]: pgmap v40: 65 pgs: 65 active+clean; 1.3 GiB data, 5.4 GiB used, 531 GiB / 536 GiB avail; 38 MiB/s rd, 133 MiB/s wr, 300 op/s 2024-03-30T04:28:13.716 INFO:tasks.workunit.client.0.smithi123.stdout:7/548: mknod d2/da/d22/d3b/cc0 0 2024-03-30T04:28:13.719 INFO:tasks.workunit.client.0.smithi123.stdout:4/548: rmdir d2 39 2024-03-30T04:28:13.722 INFO:tasks.workunit.client.0.smithi123.stdout:7/549: write d2/da/d11/f45 [1385027,57954] 0 2024-03-30T04:28:13.724 INFO:tasks.workunit.client.0.smithi123.stdout:4/549: chown d2/d4/d7/d63 10000802 1 2024-03-30T04:28:13.725 INFO:tasks.workunit.client.0.smithi123.stdout:5/534: creat dd/d19/d1f/d35/fa9 x:0 0 0 2024-03-30T04:28:13.726 INFO:tasks.workunit.client.0.smithi123.stdout:8/520: rename da/dc/d14/d18/d21/f40 to da/f9a 0 2024-03-30T04:28:13.727 INFO:tasks.workunit.client.0.smithi123.stdout:9/513: dwrite d6/d23/f8d [0,4194304] 0 2024-03-30T04:28:13.729 INFO:tasks.workunit.client.0.smithi123.stdout:5/535: write dd/f7c [829012,41580] 0 2024-03-30T04:28:13.729 INFO:tasks.workunit.client.0.smithi123.stdout:8/521: read da/f3d [3302709,45780] 0 2024-03-30T04:28:13.732 INFO:tasks.workunit.client.0.smithi123.stdout:4/550: stat d2/d4/d7/d63/l7a 0 2024-03-30T04:28:13.735 INFO:tasks.workunit.client.0.smithi123.stdout:9/514: truncate d6/d1f/f4f 5678247 0 2024-03-30T04:28:13.736 INFO:tasks.workunit.client.0.smithi123.stdout:6/650: dread d10/d20/f58 [0,4194304] 0 2024-03-30T04:28:13.736 INFO:tasks.workunit.client.1.smithi135.stdout:1/145: creat d16/d29/f35 x:0 0 0 2024-03-30T04:28:13.737 INFO:tasks.workunit.client.0.smithi123.stdout:5/536: unlink dd/d19/d1f/d35/d3a/c5c 0 2024-03-30T04:28:13.739 INFO:tasks.workunit.client.1.smithi135.stdout:0/205: symlink d0/d5/d9/l3d 0 2024-03-30T04:28:13.740 INFO:tasks.workunit.client.1.smithi135.stdout:2/134: creat d4/da/dd/d12/f2a x:0 0 0 2024-03-30T04:28:13.744 INFO:tasks.workunit.client.0.smithi123.stdout:5/537: read f9 [701843,130074] 0 2024-03-30T04:28:13.750 INFO:tasks.workunit.client.0.smithi123.stdout:6/651: dread d10/d82/d8e/dc7/f2a [4194304,4194304] 0 2024-03-30T04:28:13.750 INFO:tasks.workunit.client.1.smithi135.stdout:4/132: mknod d10/c2c 0 2024-03-30T04:28:13.750 INFO:tasks.workunit.client.1.smithi135.stdout:3/145: mknod d3/d10/c29 0 2024-03-30T04:28:13.750 INFO:tasks.workunit.client.1.smithi135.stdout:2/135: write d4/da/dd/f26 [386473,16994] 0 2024-03-30T04:28:13.751 INFO:tasks.workunit.client.1.smithi135.stdout:1/146: symlink d16/d29/d2c/l36 0 2024-03-30T04:28:13.754 INFO:tasks.workunit.client.0.smithi123.stdout:4/551: dwrite d2/d4/d7/fb [0,4194304] 0 2024-03-30T04:28:13.754 INFO:tasks.workunit.client.1.smithi135.stdout:7/237: creat d4/d4d/f4e x:0 0 0 2024-03-30T04:28:13.756 INFO:tasks.workunit.client.1.smithi135.stdout:3/146: fdatasync f2 0 2024-03-30T04:28:13.756 INFO:tasks.workunit.client.0.smithi123.stdout:9/515: dwrite d6/f7e [0,4194304] 0 2024-03-30T04:28:13.758 INFO:tasks.workunit.client.0.smithi123.stdout:6/652: creat d10/d82/dc1/d6a/d7a/d99/dba/fde x:0 0 0 2024-03-30T04:28:13.758 INFO:tasks.workunit.client.1.smithi135.stdout:1/147: read d16/d19/f1f [3932736,26148] 0 2024-03-30T04:28:13.759 INFO:tasks.workunit.client.0.smithi123.stdout:4/552: write d2/d4/f22 [3034458,60450] 0 2024-03-30T04:28:13.762 INFO:tasks.workunit.client.0.smithi123.stdout:9/516: symlink d6/d1f/d72/lb3 0 2024-03-30T04:28:13.762 INFO:tasks.workunit.client.0.smithi123.stdout:4/553: chown d2/dab/fbc 723 1 2024-03-30T04:28:13.763 INFO:tasks.workunit.client.1.smithi135.stdout:0/206: dread d0/d19/d2d/f38 [0,4194304] 0 2024-03-30T04:28:13.764 INFO:tasks.workunit.client.0.smithi123.stdout:9/517: creat d6/d7c/fb4 x:0 0 0 2024-03-30T04:28:13.765 INFO:tasks.workunit.client.1.smithi135.stdout:2/136: fdatasync d4/da/fb 0 2024-03-30T04:28:13.765 INFO:tasks.workunit.client.1.smithi135.stdout:0/207: chown d0/d13 150654052 1 2024-03-30T04:28:13.765 INFO:tasks.workunit.client.0.smithi123.stdout:4/554: write d2/d4/d2c/d42/d35/d5d/d78/f83 [2764734,4030] 0 2024-03-30T04:28:13.767 INFO:tasks.workunit.client.0.smithi123.stdout:9/518: chown d6/l6e 616071220 1 2024-03-30T04:28:13.770 INFO:tasks.workunit.client.1.smithi135.stdout:0/208: truncate d0/d19/d2d/f38 1314477 0 2024-03-30T04:28:13.770 INFO:tasks.workunit.client.0.smithi123.stdout:4/555: mkdir d2/d4/d7/dc1 0 2024-03-30T04:28:13.771 INFO:tasks.workunit.client.1.smithi135.stdout:7/238: mkdir d4/db/d1a/d4b/d4f 0 2024-03-30T04:28:13.772 INFO:tasks.workunit.client.0.smithi123.stdout:4/556: dread - d2/dab/fbc zero size 2024-03-30T04:28:13.775 INFO:tasks.workunit.client.1.smithi135.stdout:7/239: write d4/d4d/f4e [503031,113132] 0 2024-03-30T04:28:13.780 INFO:tasks.workunit.client.0.smithi123.stdout:7/550: dread d2/d4d/f6a [0,4194304] 0 2024-03-30T04:28:13.783 INFO:tasks.workunit.client.0.smithi123.stdout:7/551: read - d2/da/d11/d61/d55/d59/db5/fa6 zero size 2024-03-30T04:28:13.783 INFO:tasks.workunit.client.1.smithi135.stdout:7/240: dread d4/db/d1a/d4b/f45 [0,4194304] 0 2024-03-30T04:28:13.786 INFO:tasks.workunit.client.1.smithi135.stdout:1/148: stat d16/d1a/f21 0 2024-03-30T04:28:13.790 INFO:tasks.workunit.client.1.smithi135.stdout:9/108: getdents d7 0 2024-03-30T04:28:13.791 INFO:tasks.workunit.client.1.smithi135.stdout:9/109: readlink d7/d1b/l17 0 2024-03-30T04:28:13.797 INFO:tasks.workunit.client.1.smithi135.stdout:2/137: mknod d4/d6/c2b 0 2024-03-30T04:28:13.797 INFO:tasks.workunit.client.1.smithi135.stdout:3/147: rmdir d3/d10 39 2024-03-30T04:28:13.798 INFO:tasks.workunit.client.1.smithi135.stdout:0/209: mkdir d0/d5/d9/d23/d3e 0 2024-03-30T04:28:13.798 INFO:tasks.workunit.client.0.smithi123.stdout:9/519: write d6/d58/f5c [307032,92877] 0 2024-03-30T04:28:13.798 INFO:tasks.workunit.client.0.smithi123.stdout:4/557: write d2/d34/f4c [1227303,51360] 0 2024-03-30T04:28:13.801 INFO:tasks.workunit.client.1.smithi135.stdout:7/241: unlink d4/db/c3f 0 2024-03-30T04:28:13.801 INFO:tasks.workunit.client.0.smithi123.stdout:4/558: chown d2/d4/d2c/d42/d35/d5d/d9f 8337 1 2024-03-30T04:28:13.803 INFO:tasks.workunit.client.1.smithi135.stdout:4/133: link d10/d1a/l1d d10/d1a/l2d 0 2024-03-30T04:28:13.803 INFO:tasks.workunit.client.0.smithi123.stdout:9/520: mknod d6/d1f/d72/cb5 0 2024-03-30T04:28:13.805 INFO:tasks.workunit.client.0.smithi123.stdout:9/521: chown d6/d1f/d45/d97 377260705 1 2024-03-30T04:28:13.805 INFO:tasks.workunit.client.0.smithi123.stdout:7/552: dread d2/da/d11/d61/f72 [0,4194304] 0 2024-03-30T04:28:13.807 INFO:tasks.workunit.client.0.smithi123.stdout:9/522: read d6/d1f/f3f [520596,86817] 0 2024-03-30T04:28:13.808 INFO:tasks.workunit.client.0.smithi123.stdout:6/653: dwrite d10/d20/d3f/f4d [0,4194304] 0 2024-03-30T04:28:13.810 INFO:tasks.workunit.client.1.smithi135.stdout:8/188: dread f0 [0,4194304] 0 2024-03-30T04:28:13.810 INFO:tasks.workunit.client.0.smithi123.stdout:4/559: rename d2/d4/d2c/d42/dad/c9a to d2/d4/d7/d63/cc2 0 2024-03-30T04:28:13.811 INFO:tasks.workunit.client.0.smithi123.stdout:6/654: fsync d10/f96 0 2024-03-30T04:28:13.812 INFO:tasks.workunit.client.0.smithi123.stdout:4/560: stat d2/d34/l98 0 2024-03-30T04:28:13.813 INFO:tasks.workunit.client.1.smithi135.stdout:2/138: creat d4/d6/d20/f2c x:0 0 0 2024-03-30T04:28:13.814 INFO:tasks.workunit.client.1.smithi135.stdout:7/242: rename d4/l5 to d4/l50 0 2024-03-30T04:28:13.815 INFO:tasks.workunit.client.0.smithi123.stdout:6/655: write d10/d82/d8e/dc7/d5b/fcc [3869431,40237] 0 2024-03-30T04:28:13.817 INFO:tasks.workunit.client.0.smithi123.stdout:9/523: mkdir d6/d1f/d72/db6 0 2024-03-30T04:28:13.818 INFO:tasks.workunit.client.1.smithi135.stdout:9/110: getdents d7/d1b 0 2024-03-30T04:28:13.818 INFO:tasks.workunit.client.1.smithi135.stdout:2/139: rename d4 to d4/da/d2d 22 2024-03-30T04:28:13.819 INFO:tasks.workunit.client.0.smithi123.stdout:4/561: creat d2/d4/d2c/d42/fc3 x:0 0 0 2024-03-30T04:28:13.820 INFO:tasks.workunit.client.0.smithi123.stdout:6/656: creat d10/d82/d9a/db3/fdf x:0 0 0 2024-03-30T04:28:13.821 INFO:tasks.workunit.client.0.smithi123.stdout:4/562: stat d2/d4/d7/d1a/l48 0 2024-03-30T04:28:13.821 INFO:tasks.workunit.client.1.smithi135.stdout:8/189: mknod d6/d8/d15/d2a/d35/c36 0 2024-03-30T04:28:13.823 INFO:tasks.workunit.client.1.smithi135.stdout:2/140: readlink d4/da/l22 0 2024-03-30T04:28:13.823 INFO:tasks.workunit.client.1.smithi135.stdout:7/243: mkdir d4/d4d/d2c/d51 0 2024-03-30T04:28:13.823 INFO:tasks.workunit.client.1.smithi135.stdout:1/149: dwrite d16/d1a/f23 [0,4194304] 0 2024-03-30T04:28:13.826 INFO:tasks.workunit.client.0.smithi123.stdout:7/553: dwrite d2/f13 [0,4194304] 0 2024-03-30T04:28:13.826 INFO:tasks.workunit.client.0.smithi123.stdout:6/657: symlink d10/d82/d8e/dc7/d5b/le0 0 2024-03-30T04:28:13.826 INFO:tasks.workunit.client.0.smithi123.stdout:4/563: write d2/d4/d2c/d42/fc3 [62581,75989] 0 2024-03-30T04:28:13.827 INFO:tasks.workunit.client.1.smithi135.stdout:8/190: creat d6/d8/d15/d2a/f37 x:0 0 0 2024-03-30T04:28:13.832 INFO:tasks.workunit.client.1.smithi135.stdout:0/210: rmdir d0/d5 39 2024-03-30T04:28:13.832 INFO:tasks.workunit.client.1.smithi135.stdout:0/211: stat d0/d13/d1d/l20 0 2024-03-30T04:28:13.833 INFO:tasks.workunit.client.0.smithi123.stdout:7/554: mknod d2/da/d11/d61/d55/d94/cc1 0 2024-03-30T04:28:13.835 INFO:tasks.workunit.client.0.smithi123.stdout:4/564: creat d2/d4/d7/fc4 x:0 0 0 2024-03-30T04:28:13.835 INFO:tasks.workunit.client.0.smithi123.stdout:8/522: dread da/dc/f52 [0,4194304] 0 2024-03-30T04:28:13.838 INFO:tasks.workunit.client.0.smithi123.stdout:7/555: truncate d2/f41 753196 0 2024-03-30T04:28:13.838 INFO:tasks.workunit.client.1.smithi135.stdout:1/150: dread d16/f18 [0,4194304] 0 2024-03-30T04:28:13.839 INFO:tasks.workunit.client.0.smithi123.stdout:9/524: dwrite d6/d1f/f37 [4194304,4194304] 0 2024-03-30T04:28:13.842 INFO:tasks.workunit.client.1.smithi135.stdout:2/141: truncate d4/da/fb 1253285 0 2024-03-30T04:28:13.845 INFO:tasks.workunit.client.0.smithi123.stdout:4/565: dread - d2/d4/d2c/d42/d89/f94 zero size 2024-03-30T04:28:13.845 INFO:tasks.workunit.client.0.smithi123.stdout:9/525: rename d6/d1f/d4d to d6/d1f/d4d/db7 22 2024-03-30T04:28:13.846 INFO:tasks.workunit.client.0.smithi123.stdout:7/556: write d2/f8 [1018931,31149] 0 2024-03-30T04:28:13.851 INFO:tasks.workunit.client.1.smithi135.stdout:8/191: creat d6/d8/f38 x:0 0 0 2024-03-30T04:28:13.851 INFO:tasks.workunit.client.1.smithi135.stdout:0/212: write d0/d5/d10/f11 [9039416,45205] 0 2024-03-30T04:28:13.852 INFO:tasks.workunit.client.0.smithi123.stdout:9/526: chown d6/d7c/d82/l8c 7125 1 2024-03-30T04:28:13.852 INFO:tasks.workunit.client.0.smithi123.stdout:4/566: dread - d2/d4/d2c/d42/d35/d9b/fb6 zero size 2024-03-30T04:28:13.858 INFO:tasks.workunit.client.1.smithi135.stdout:2/142: mkdir d4/da/dd/d2e 0 2024-03-30T04:28:13.858 INFO:tasks.workunit.client.0.smithi123.stdout:4/567: mknod d2/d4/d2c/d42/d35/cc5 0 2024-03-30T04:28:13.858 INFO:tasks.workunit.client.0.smithi123.stdout:9/527: mkdir d6/d1f/d92/db8 0 2024-03-30T04:28:13.859 INFO:tasks.workunit.client.1.smithi135.stdout:1/151: mkdir d16/d29/d2c/d30/d37 0 2024-03-30T04:28:13.859 INFO:tasks.workunit.client.1.smithi135.stdout:9/111: dread d7/f11 [0,4194304] 0 2024-03-30T04:28:13.859 INFO:tasks.workunit.client.1.smithi135.stdout:8/192: read d6/d8/f20 [513400,41918] 0 2024-03-30T04:28:13.860 INFO:tasks.workunit.client.0.smithi123.stdout:9/528: creat d6/d1f/d4d/fb9 x:0 0 0 2024-03-30T04:28:13.861 INFO:tasks.workunit.client.1.smithi135.stdout:9/112: chown d7 254474897 1 2024-03-30T04:28:13.862 INFO:tasks.workunit.client.1.smithi135.stdout:0/213: dread - d0/d5/d9/f15 zero size 2024-03-30T04:28:13.863 INFO:tasks.workunit.client.1.smithi135.stdout:2/143: readlink d4/d6/l27 0 2024-03-30T04:28:13.865 INFO:tasks.workunit.client.1.smithi135.stdout:2/144: fsync d4/da/f21 0 2024-03-30T04:28:13.865 INFO:tasks.workunit.client.1.smithi135.stdout:7/244: write d4/db/de/f25 [2401000,122038] 0 2024-03-30T04:28:13.866 INFO:tasks.workunit.client.0.smithi123.stdout:9/529: readlink d6/d1f/d45/d97/l9b 0 2024-03-30T04:28:13.866 INFO:tasks.workunit.client.0.smithi123.stdout:9/530: chown d6/d1f/c86 46 1 2024-03-30T04:28:13.872 INFO:tasks.workunit.client.0.smithi123.stdout:4/568: dread d2/d4/d2c/d42/d35/d5d/d6f/f8a [0,4194304] 0 2024-03-30T04:28:13.872 INFO:tasks.workunit.client.1.smithi135.stdout:2/145: read - d4/da/f1e zero size 2024-03-30T04:28:13.872 INFO:tasks.workunit.client.1.smithi135.stdout:7/245: chown d4/l23 913325 1 2024-03-30T04:28:13.874 INFO:tasks.workunit.client.0.smithi123.stdout:8/523: write da/d27/d3c/d64/f81 [2546034,124175] 0 2024-03-30T04:28:13.875 INFO:tasks.workunit.client.1.smithi135.stdout:3/148: dwrite f2 [0,4194304] 0 2024-03-30T04:28:13.877 INFO:tasks.workunit.client.0.smithi123.stdout:8/524: chown da/d27/d56 2962751 1 2024-03-30T04:28:13.880 INFO:tasks.workunit.client.1.smithi135.stdout:2/146: dread - d4/da/dd/d12/f2a zero size 2024-03-30T04:28:13.882 INFO:tasks.workunit.client.0.smithi123.stdout:8/525: mknod da/d27/d3c/d64/d78/d83/c9b 0 2024-03-30T04:28:13.883 INFO:tasks.workunit.client.0.smithi123.stdout:9/531: dwrite d6/d1f/d45/d4c/f54 [4194304,4194304] 0 2024-03-30T04:28:13.885 INFO:tasks.workunit.client.1.smithi135.stdout:6/127: sync 2024-03-30T04:28:13.885 INFO:tasks.workunit.client.0.smithi123.stdout:7/557: truncate d2/da/d11/d61/d55/d59/db5/f51 3257960 0 2024-03-30T04:28:13.887 INFO:tasks.workunit.client.0.smithi123.stdout:7/558: chown d2/f8 1672 1 2024-03-30T04:28:13.887 INFO:tasks.workunit.client.0.smithi123.stdout:8/526: fdatasync da/dc/d14/d18/f30 0 2024-03-30T04:28:13.890 INFO:tasks.workunit.client.0.smithi123.stdout:8/527: chown da/dc/d14/f36 1 1 2024-03-30T04:28:13.891 INFO:tasks.workunit.client.1.smithi135.stdout:1/152: dread d16/f25 [0,4194304] 0 2024-03-30T04:28:13.893 INFO:tasks.workunit.client.0.smithi123.stdout:4/569: dwrite d2/f1c [0,4194304] 0 2024-03-30T04:28:13.894 INFO:tasks.workunit.client.1.smithi135.stdout:9/113: link d7/d1b/l19 d7/l1e 0 2024-03-30T04:28:13.895 INFO:tasks.workunit.client.0.smithi123.stdout:8/528: mkdir da/dc/d14/d9c 0 2024-03-30T04:28:13.898 INFO:tasks.workunit.client.0.smithi123.stdout:7/559: getdents d2/dc 0 2024-03-30T04:28:13.898 INFO:tasks.workunit.client.1.smithi135.stdout:0/214: dwrite d0/d1/d16/f32 [0,4194304] 0 2024-03-30T04:28:13.898 INFO:tasks.workunit.client.1.smithi135.stdout:7/246: symlink d4/d4d/d2c/d51/l52 0 2024-03-30T04:28:13.899 INFO:tasks.workunit.client.0.smithi123.stdout:6/658: dread d10/d82/d8e/dc7/f5d [0,4194304] 0 2024-03-30T04:28:13.902 INFO:tasks.workunit.client.0.smithi123.stdout:4/570: fdatasync d2/d4/d2c/d42/d35/d5d/d78/f85 0 2024-03-30T04:28:13.904 INFO:tasks.workunit.client.0.smithi123.stdout:6/659: fdatasync d10/d82/d8e/dc7/d42/f47 0 2024-03-30T04:28:13.904 INFO:tasks.workunit.client.0.smithi123.stdout:8/529: getdents da/d99 0 2024-03-30T04:28:13.906 INFO:tasks.workunit.client.1.smithi135.stdout:6/128: rename d2/dd/d18/l1d to d2/l27 0 2024-03-30T04:28:13.907 INFO:tasks.workunit.client.1.smithi135.stdout:3/149: creat d3/f2a x:0 0 0 2024-03-30T04:28:13.911 INFO:tasks.workunit.client.1.smithi135.stdout:9/114: write d7/f11 [3518173,32073] 0 2024-03-30T04:28:13.911 INFO:tasks.workunit.client.0.smithi123.stdout:8/530: mkdir da/dc/d14/d9d 0 2024-03-30T04:28:13.912 INFO:tasks.workunit.client.0.smithi123.stdout:6/660: creat d10/d82/dc1/d6a/d7a/d99/dba/fe1 x:0 0 0 2024-03-30T04:28:13.914 INFO:tasks.workunit.client.1.smithi135.stdout:1/153: rmdir d16/d29/d2c/d30/d37 0 2024-03-30T04:28:13.915 INFO:tasks.workunit.client.0.smithi123.stdout:4/571: creat d2/fc6 x:0 0 0 2024-03-30T04:28:13.916 INFO:tasks.workunit.client.0.smithi123.stdout:8/531: mknod da/d27/d3c/d2d/c9e 0 2024-03-30T04:28:13.916 INFO:tasks.workunit.client.1.smithi135.stdout:0/215: read d0/d5/d10/f11 [542821,41835] 0 2024-03-30T04:28:13.917 INFO:tasks.workunit.client.0.smithi123.stdout:4/572: write d2/d4/d7/f6c [3651263,57504] 0 2024-03-30T04:28:13.924 INFO:tasks.workunit.client.1.smithi135.stdout:9/115: fdatasync f1 0 2024-03-30T04:28:13.925 INFO:tasks.workunit.client.0.smithi123.stdout:8/532: getdents da/d27/d3c 0 2024-03-30T04:28:13.925 INFO:tasks.workunit.client.1.smithi135.stdout:8/193: dwrite d6/d8/f31 [0,4194304] 0 2024-03-30T04:28:13.925 INFO:tasks.workunit.client.1.smithi135.stdout:1/154: unlink d16/l28 0 2024-03-30T04:28:13.926 INFO:tasks.workunit.client.1.smithi135.stdout:7/247: dread d4/db/de/f28 [0,4194304] 0 2024-03-30T04:28:13.928 INFO:tasks.workunit.client.1.smithi135.stdout:0/216: fsync d0/d5/d10/f11 0 2024-03-30T04:28:13.930 INFO:tasks.workunit.client.1.smithi135.stdout:2/147: dwrite d4/d6/f1a [0,4194304] 0 2024-03-30T04:28:13.930 INFO:tasks.workunit.client.0.smithi123.stdout:4/573: link d2/d4/d2c/d42/dad/d91/d9d/fa6 d2/d4/d2c/d42/d35/d9b/db7/fc7 0 2024-03-30T04:28:13.935 INFO:tasks.workunit.client.1.smithi135.stdout:1/155: write f7 [161767,25755] 0 2024-03-30T04:28:13.937 INFO:tasks.workunit.client.0.smithi123.stdout:8/533: rename da/d27/d3c/d64/f8e to da/dc/d14/d18/d62/f9f 0 2024-03-30T04:28:13.938 INFO:tasks.workunit.client.1.smithi135.stdout:1/156: chown d16/d19 1 1 2024-03-30T04:28:13.938 INFO:tasks.workunit.client.1.smithi135.stdout:1/157: chown d16 3871863 1 2024-03-30T04:28:13.942 INFO:tasks.workunit.client.0.smithi123.stdout:8/534: creat da/d27/d3c/d2d/fa0 x:0 0 0 2024-03-30T04:28:13.943 INFO:tasks.workunit.client.1.smithi135.stdout:8/194: unlink d6/d8/d15/l28 0 2024-03-30T04:28:13.945 INFO:tasks.workunit.client.1.smithi135.stdout:9/116: rename d7/d1b/c13 to d7/c1f 0 2024-03-30T04:28:13.945 INFO:tasks.workunit.client.1.smithi135.stdout:0/217: rmdir d0/d5/d9/d23 39 2024-03-30T04:28:13.948 INFO:tasks.workunit.client.1.smithi135.stdout:8/195: chown d6/d8/d15/l1a 42127502 1 2024-03-30T04:28:13.948 INFO:tasks.workunit.client.1.smithi135.stdout:3/150: link d3/cc d3/d10/d16/d23/c2b 0 2024-03-30T04:28:13.949 INFO:tasks.workunit.client.1.smithi135.stdout:9/117: chown d7/fe 6 1 2024-03-30T04:28:13.949 INFO:tasks.workunit.client.1.smithi135.stdout:7/248: symlink d4/db/l53 0 2024-03-30T04:28:13.953 INFO:tasks.workunit.client.1.smithi135.stdout:2/148: mkdir d4/da/dd/d2e/d2f 0 2024-03-30T04:28:13.953 INFO:tasks.workunit.client.1.smithi135.stdout:3/151: write d3/f5 [1249223,98914] 0 2024-03-30T04:28:13.953 INFO:tasks.workunit.client.1.smithi135.stdout:7/249: read - d4/d4d/f43 zero size 2024-03-30T04:28:13.955 INFO:tasks.workunit.client.1.smithi135.stdout:1/158: link d16/d29/f35 d16/d29/d32/f38 0 2024-03-30T04:28:13.956 INFO:tasks.workunit.client.1.smithi135.stdout:0/218: mkdir d0/d34/d3f 0 2024-03-30T04:28:13.962 INFO:tasks.workunit.client.0.smithi123.stdout:6/661: dread d10/d82/f8a [0,4194304] 0 2024-03-30T04:28:13.962 INFO:tasks.workunit.client.0.smithi123.stdout:8/535: dwrite da/d27/d56/d6c/d6d/f72 [0,4194304] 0 2024-03-30T04:28:13.967 INFO:tasks.workunit.client.1.smithi135.stdout:2/149: creat d4/da/dd/d2e/d2f/f30 x:0 0 0 2024-03-30T04:28:13.967 INFO:tasks.workunit.client.1.smithi135.stdout:8/196: rename d6/d8/d15/d1f/f34 to d6/d8/f39 0 2024-03-30T04:28:13.967 INFO:tasks.workunit.client.1.smithi135.stdout:7/250: write d4/db/d1a/d4b/f31 [816502,111847] 0 2024-03-30T04:28:13.967 INFO:tasks.workunit.client.0.smithi123.stdout:7/560: write d2/da/d11/d61/d55/d9e/fad [937864,3197] 0 2024-03-30T04:28:13.974 INFO:tasks.workunit.client.0.smithi123.stdout:8/536: getdents da/dc/d14/d18 0 2024-03-30T04:28:13.977 INFO:tasks.workunit.client.1.smithi135.stdout:2/150: creat d4/da/dd/d12/f31 x:0 0 0 2024-03-30T04:28:13.978 INFO:tasks.workunit.client.1.smithi135.stdout:8/197: unlink d6/d8/d15/f22 0 2024-03-30T04:28:13.979 INFO:tasks.workunit.client.0.smithi123.stdout:7/561: rename d2/da/d22/d3b/f54 to d2/da/d11/d61/d55/d9e/fc2 0 2024-03-30T04:28:13.982 INFO:tasks.workunit.client.0.smithi123.stdout:7/562: write d2/dc/d7f/fa3 [739438,21859] 0 2024-03-30T04:28:13.984 INFO:tasks.workunit.client.1.smithi135.stdout:8/198: dread - d6/d8/d14/f30 zero size 2024-03-30T04:28:13.985 INFO:tasks.workunit.client.1.smithi135.stdout:0/219: dwrite d0/f3 [0,4194304] 0 2024-03-30T04:28:13.986 INFO:tasks.workunit.client.0.smithi123.stdout:4/574: dread d2/d26/f4f [0,4194304] 0 2024-03-30T04:28:13.987 INFO:tasks.workunit.client.1.smithi135.stdout:2/151: readlink d4/da/dd/d12/l29 0 2024-03-30T04:28:13.988 INFO:tasks.workunit.client.0.smithi123.stdout:7/563: getdents d2/d4d 0 2024-03-30T04:28:13.992 INFO:tasks.workunit.client.0.smithi123.stdout:8/537: write da/dc/d14/f44 [1067376,41373] 0 2024-03-30T04:28:13.994 INFO:tasks.workunit.client.0.smithi123.stdout:4/575: mknod d2/d4/d2c/d42/d35/d9b/db7/cc8 0 2024-03-30T04:28:13.994 INFO:tasks.workunit.client.0.smithi123.stdout:7/564: creat d2/dc/d49/dbc/fc3 x:0 0 0 2024-03-30T04:28:13.995 INFO:tasks.workunit.client.1.smithi135.stdout:2/152: chown d4/d6/l1b 3151 1 2024-03-30T04:28:13.995 INFO:tasks.workunit.client.1.smithi135.stdout:3/152: dwrite d3/d6/f1f [0,4194304] 0 2024-03-30T04:28:13.996 INFO:tasks.workunit.client.0.smithi123.stdout:8/538: creat da/d99/fa1 x:0 0 0 2024-03-30T04:28:13.998 INFO:tasks.workunit.client.0.smithi123.stdout:4/576: truncate d2/d4/d7/fc4 640325 0 2024-03-30T04:28:14.001 INFO:tasks.workunit.client.1.smithi135.stdout:9/118: getdents d7/d1b 0 2024-03-30T04:28:14.002 INFO:tasks.workunit.client.0.smithi123.stdout:8/539: mkdir da/dc/d14/d18/d21/da2 0 2024-03-30T04:28:14.002 INFO:tasks.workunit.client.0.smithi123.stdout:7/565: truncate d2/dc/d49/f63 532053 0 2024-03-30T04:28:14.003 INFO:tasks.workunit.client.0.smithi123.stdout:4/577: mknod d2/cc9 0 2024-03-30T04:28:14.005 INFO:tasks.workunit.client.1.smithi135.stdout:0/220: write d0/d19/d2d/f38 [2113179,71169] 0 2024-03-30T04:28:14.007 INFO:tasks.workunit.client.0.smithi123.stdout:7/566: mknod d2/da/d11/d34/d76/cc4 0 2024-03-30T04:28:14.007 INFO:tasks.workunit.client.0.smithi123.stdout:6/662: dread d10/f97 [0,4194304] 0 2024-03-30T04:28:14.007 INFO:tasks.workunit.client.1.smithi135.stdout:2/153: truncate d4/f5 7779132 0 2024-03-30T04:28:14.008 INFO:tasks.workunit.client.0.smithi123.stdout:8/540: write da/dc/d14/f36 [671926,45866] 0 2024-03-30T04:28:14.008 INFO:tasks.workunit.client.1.smithi135.stdout:3/153: creat d3/d10/d16/d23/f2c x:0 0 0 2024-03-30T04:28:14.008 INFO:tasks.workunit.client.1.smithi135.stdout:9/119: rmdir d7 39 2024-03-30T04:28:14.013 INFO:tasks.workunit.client.0.smithi123.stdout:7/567: creat d2/dc/d49/dbc/fc5 x:0 0 0 2024-03-30T04:28:14.015 INFO:tasks.workunit.client.0.smithi123.stdout:7/568: fsync d2/dc/d49/d68/f9b 0 2024-03-30T04:28:14.016 INFO:tasks.workunit.client.1.smithi135.stdout:7/251: dread f3 [0,4194304] 0 2024-03-30T04:28:14.018 INFO:tasks.workunit.client.0.smithi123.stdout:7/569: fsync d2/f8 0 2024-03-30T04:28:14.020 INFO:tasks.workunit.client.1.smithi135.stdout:3/154: creat d3/d10/f2d x:0 0 0 2024-03-30T04:28:14.021 INFO:tasks.workunit.client.1.smithi135.stdout:9/120: chown d7/c1f 5 1 2024-03-30T04:28:14.022 INFO:tasks.workunit.client.0.smithi123.stdout:4/578: dread d2/d4/d2c/d42/d35/faa [0,4194304] 0 2024-03-30T04:28:14.022 INFO:tasks.workunit.client.1.smithi135.stdout:2/154: rmdir d4/da/dd/d2e/d2f 39 2024-03-30T04:28:14.024 INFO:tasks.workunit.client.1.smithi135.stdout:8/199: rmdir d6/d8 39 2024-03-30T04:28:14.027 INFO:tasks.workunit.client.0.smithi123.stdout:6/663: rename d10/d20/d3f/da3 to d10/d82/dc1/d6a/d7a/de2 0 2024-03-30T04:28:14.027 INFO:tasks.workunit.client.1.smithi135.stdout:7/252: rename d4/d4d/d2c/f2d to d4/f54 0 2024-03-30T04:28:14.028 INFO:tasks.workunit.client.0.smithi123.stdout:7/570: creat d2/da/d11/fc6 x:0 0 0 2024-03-30T04:28:14.029 INFO:tasks.workunit.client.1.smithi135.stdout:9/121: creat d7/f20 x:0 0 0 2024-03-30T04:28:14.030 INFO:tasks.workunit.client.0.smithi123.stdout:6/664: chown d10/d82/d8e/dc7/d5b/le0 5716 1 2024-03-30T04:28:14.031 INFO:tasks.workunit.client.0.smithi123.stdout:4/579: creat d2/d4/fca x:0 0 0 2024-03-30T04:28:14.031 INFO:tasks.workunit.client.1.smithi135.stdout:8/200: dread - d6/d8/d14/f29 zero size 2024-03-30T04:28:14.035 INFO:tasks.workunit.client.0.smithi123.stdout:6/665: fdatasync d10/d82/dc1/d6a/fca 0 2024-03-30T04:28:14.036 INFO:tasks.workunit.client.1.smithi135.stdout:3/155: rename d3/f2a to d3/d6/d22/f2e 0 2024-03-30T04:28:14.037 INFO:tasks.workunit.client.1.smithi135.stdout:0/221: truncate d0/f12 168467 0 2024-03-30T04:28:14.037 INFO:tasks.workunit.client.0.smithi123.stdout:8/541: dwrite da/dc/f32 [0,4194304] 0 2024-03-30T04:28:14.037 INFO:tasks.workunit.client.0.smithi123.stdout:7/571: truncate d2/da/d11/fb1 648602 0 2024-03-30T04:28:14.040 INFO:tasks.workunit.client.1.smithi135.stdout:9/122: mknod d7/d1b/c21 0 2024-03-30T04:28:14.040 INFO:tasks.workunit.client.0.smithi123.stdout:8/542: write da/dc/d14/d18/f4f [4391023,21270] 0 2024-03-30T04:28:14.042 INFO:tasks.workunit.client.0.smithi123.stdout:7/572: mknod d2/da/d11/d61/d55/cc7 0 2024-03-30T04:28:14.044 INFO:tasks.workunit.client.1.smithi135.stdout:8/201: mknod d6/d8/d15/d2a/d35/c3a 0 2024-03-30T04:28:14.047 INFO:tasks.workunit.client.0.smithi123.stdout:8/543: unlink da/d27/d56/d6c/d6d/l75 0 2024-03-30T04:28:14.048 INFO:tasks.workunit.client.1.smithi135.stdout:3/156: symlink d3/d6/d22/l2f 0 2024-03-30T04:28:14.048 INFO:tasks.workunit.client.1.smithi135.stdout:2/155: dwrite d4/d6/f1f [0,4194304] 0 2024-03-30T04:28:14.049 INFO:tasks.workunit.client.0.smithi123.stdout:7/573: truncate d2/fbd 333976 0 2024-03-30T04:28:14.051 INFO:tasks.workunit.client.0.smithi123.stdout:6/666: write d10/d82/d8e/dc7/f1f [1020448,120946] 0 2024-03-30T04:28:14.052 INFO:tasks.workunit.client.0.smithi123.stdout:8/544: write da/dc/d14/d18/d62/f9f [1853515,120290] 0 2024-03-30T04:28:14.057 INFO:tasks.workunit.client.1.smithi135.stdout:9/123: rename d7/fe to d7/f22 0 2024-03-30T04:28:14.058 INFO:tasks.workunit.client.0.smithi123.stdout:6/667: creat d10/d82/fe3 x:0 0 0 2024-03-30T04:28:14.059 INFO:tasks.workunit.client.1.smithi135.stdout:9/124: truncate d7/f12 1604094 0 2024-03-30T04:28:14.066 INFO:tasks.workunit.client.1.smithi135.stdout:7/253: symlink d4/db/de/l55 0 2024-03-30T04:28:14.067 INFO:tasks.workunit.client.1.smithi135.stdout:9/125: rename d7/c18 to d7/c23 0 2024-03-30T04:28:14.067 INFO:tasks.workunit.client.0.smithi123.stdout:6/668: creat d10/d20/da2/fe4 x:0 0 0 2024-03-30T04:28:14.067 INFO:tasks.workunit.client.0.smithi123.stdout:8/545: getdents da/d27/d3c/d2d 0 2024-03-30T04:28:14.067 INFO:tasks.workunit.client.0.smithi123.stdout:6/669: stat d10/d82/dc1/d6a/d7a/d99/dba 0 2024-03-30T04:28:14.068 INFO:tasks.workunit.client.1.smithi135.stdout:9/126: chown d7/d1b/l14 172572189 1 2024-03-30T04:28:14.069 INFO:tasks.workunit.client.0.smithi123.stdout:8/546: chown da/dc/d14/c37 34273 1 2024-03-30T04:28:14.070 INFO:tasks.workunit.client.1.smithi135.stdout:0/222: symlink d0/d5/d9/d23/d3e/l40 0 2024-03-30T04:28:14.073 INFO:tasks.workunit.client.1.smithi135.stdout:8/202: creat d6/d8/f3b x:0 0 0 2024-03-30T04:28:14.073 INFO:tasks.workunit.client.0.smithi123.stdout:8/547: write da/fb [4811436,112609] 0 2024-03-30T04:28:14.073 INFO:tasks.workunit.client.0.smithi123.stdout:6/670: truncate d10/d82/f84 1438855 0 2024-03-30T04:28:14.073 INFO:tasks.workunit.client.1.smithi135.stdout:3/157: rmdir d3 39 2024-03-30T04:28:14.074 INFO:tasks.workunit.client.1.smithi135.stdout:0/223: dread - d0/d1/f29 zero size 2024-03-30T04:28:14.075 INFO:tasks.workunit.client.0.smithi123.stdout:7/574: dwrite d2/da/d11/d61/d55/d59/db5/dab/fb9 [0,4194304] 0 2024-03-30T04:28:14.077 INFO:tasks.workunit.client.1.smithi135.stdout:2/156: rename d4/da/dd to d4/d32 0 2024-03-30T04:28:14.078 INFO:tasks.workunit.client.0.smithi123.stdout:7/575: chown d2/da/d11/d34/l3f 3596751 1 2024-03-30T04:28:14.088 INFO:tasks.workunit.client.0.smithi123.stdout:8/548: truncate da/d27/d3c/d2d/f57 268384 0 2024-03-30T04:28:14.089 INFO:tasks.workunit.client.0.smithi123.stdout:8/549: chown da/dc/d14/f36 361 1 2024-03-30T04:28:14.090 INFO:tasks.workunit.client.1.smithi135.stdout:0/224: mknod d0/d19/d2d/c41 0 2024-03-30T04:28:14.091 INFO:tasks.workunit.client.0.smithi123.stdout:8/550: write da/f4e [1268404,29585] 0 2024-03-30T04:28:14.093 INFO:tasks.workunit.client.1.smithi135.stdout:8/203: dread d6/d23/f27 [4194304,4194304] 0 2024-03-30T04:28:14.093 INFO:tasks.workunit.client.1.smithi135.stdout:2/157: rename d4/f25 to d4/d32/f33 0 2024-03-30T04:28:14.095 INFO:tasks.workunit.client.0.smithi123.stdout:8/551: write da/dc/f42 [845990,3050] 0 2024-03-30T04:28:14.095 INFO:tasks.workunit.client.1.smithi135.stdout:7/254: dread d4/db/f1c [0,4194304] 0 2024-03-30T04:28:14.098 INFO:tasks.workunit.client.0.smithi123.stdout:8/552: truncate da/dc/f52 3575564 0 2024-03-30T04:28:14.099 INFO:tasks.workunit.client.1.smithi135.stdout:3/158: creat d3/d10/d16/d23/f30 x:0 0 0 2024-03-30T04:28:14.100 INFO:tasks.workunit.client.1.smithi135.stdout:8/204: chown d6/d8/d15/d2a/d35 23672 1 2024-03-30T04:28:14.100 INFO:tasks.workunit.client.1.smithi135.stdout:2/158: mknod d4/d6/c34 0 2024-03-30T04:28:14.101 INFO:tasks.workunit.client.0.smithi123.stdout:8/553: chown da/d27/d56/d6c/d6d/f90 112071918 1 2024-03-30T04:28:14.103 INFO:tasks.workunit.client.1.smithi135.stdout:3/159: truncate d3/db/f19 735548 0 2024-03-30T04:28:14.104 INFO:tasks.workunit.client.0.smithi123.stdout:8/554: mknod da/dc/d14/d18/d21/ca3 0 2024-03-30T04:28:14.104 INFO:tasks.workunit.client.0.smithi123.stdout:7/576: dread d2/dc/d49/f6c [0,4194304] 0 2024-03-30T04:28:14.105 INFO:tasks.workunit.client.1.smithi135.stdout:2/159: creat d4/d6/f35 x:0 0 0 2024-03-30T04:28:14.108 INFO:tasks.workunit.client.1.smithi135.stdout:3/160: stat d3/d10/d16/f1e 0 2024-03-30T04:28:14.110 INFO:tasks.workunit.client.1.smithi135.stdout:2/160: write d4/da/f21 [3457766,94332] 0 2024-03-30T04:28:14.113 INFO:tasks.workunit.client.1.smithi135.stdout:9/127: fsync d7/f12 0 2024-03-30T04:28:14.115 INFO:tasks.workunit.client.1.smithi135.stdout:8/205: truncate f0 7604958 0 2024-03-30T04:28:14.115 INFO:tasks.workunit.client.1.smithi135.stdout:3/161: creat d3/d10/f31 x:0 0 0 2024-03-30T04:28:14.117 INFO:tasks.workunit.client.1.smithi135.stdout:2/161: creat d4/d6/f36 x:0 0 0 2024-03-30T04:28:14.118 INFO:tasks.workunit.client.0.smithi123.stdout:7/577: dwrite d2/da/d11/d34/d76/d91/f9a [0,4194304] 0 2024-03-30T04:28:14.120 INFO:tasks.workunit.client.1.smithi135.stdout:0/225: dwrite d0/d1/d16/f32 [0,4194304] 0 2024-03-30T04:28:14.122 INFO:tasks.workunit.client.1.smithi135.stdout:0/226: chown d0/d13 15197 1 2024-03-30T04:28:14.126 INFO:tasks.workunit.client.1.smithi135.stdout:7/255: dwrite d4/f16 [0,4194304] 0 2024-03-30T04:28:14.132 INFO:tasks.workunit.client.0.smithi123.stdout:7/578: truncate d2/da/d11/d61/d55/d9e/fad 2992426 0 2024-03-30T04:28:14.133 INFO:tasks.workunit.client.1.smithi135.stdout:3/162: symlink d3/d10/l32 0 2024-03-30T04:28:14.137 INFO:tasks.workunit.client.0.smithi123.stdout:7/579: rmdir d2/da/d11/d34/d85/d8e 39 2024-03-30T04:28:14.138 INFO:tasks.workunit.client.1.smithi135.stdout:3/163: chown d3/f5 1281 1 2024-03-30T04:28:14.139 INFO:tasks.workunit.client.1.smithi135.stdout:7/256: chown d4/db/c22 5 1 2024-03-30T04:28:14.143 INFO:tasks.workunit.client.1.smithi135.stdout:0/227: mkdir d0/d5/d42 0 2024-03-30T04:28:14.145 INFO:tasks.workunit.client.0.smithi123.stdout:7/580: truncate d2/da/d11/d34/d76/fa7 741525 0 2024-03-30T04:28:14.145 INFO:tasks.workunit.client.1.smithi135.stdout:7/257: readlink d4/d4d/d2c/l35 0 2024-03-30T04:28:14.146 INFO:tasks.workunit.client.0.smithi123.stdout:7/581: dread - d2/da/d11/fc6 zero size 2024-03-30T04:28:14.149 INFO:tasks.workunit.client.1.smithi135.stdout:3/164: read d3/f1d [2943431,62068] 0 2024-03-30T04:28:14.149 INFO:tasks.workunit.client.1.smithi135.stdout:0/228: creat d0/d1/f43 x:0 0 0 2024-03-30T04:28:14.151 INFO:tasks.workunit.client.0.smithi123.stdout:7/582: rename d2/da/d22/d3b/c3e to d2/da/d11/d34/d85/cc8 0 2024-03-30T04:28:14.152 INFO:tasks.workunit.client.1.smithi135.stdout:3/165: stat d3/d10/d16/d23 0 2024-03-30T04:28:14.153 INFO:tasks.workunit.client.1.smithi135.stdout:0/229: creat d0/d1/f44 x:0 0 0 2024-03-30T04:28:14.153 INFO:tasks.workunit.client.1.smithi135.stdout:7/258: write d4/db/d1a/d4b/f31 [2448196,93520] 0 2024-03-30T04:28:14.155 INFO:tasks.workunit.client.1.smithi135.stdout:3/166: rmdir d3/d6 39 2024-03-30T04:28:14.156 INFO:tasks.workunit.client.0.smithi123.stdout:7/583: unlink d2/da/d11/d61/d55/d59/db5/f66 0 2024-03-30T04:28:14.157 INFO:tasks.workunit.client.1.smithi135.stdout:0/230: mkdir d0/d19/d2d/d45 0 2024-03-30T04:28:14.159 INFO:tasks.workunit.client.0.smithi123.stdout:7/584: readlink d2/da/d11/d61/d55/d9e/la1 0 2024-03-30T04:28:14.160 INFO:tasks.workunit.client.1.smithi135.stdout:3/167: readlink d3/d6/d22/l2f 0 2024-03-30T04:28:14.161 INFO:tasks.workunit.client.1.smithi135.stdout:3/168: readlink l1 0 2024-03-30T04:28:14.161 INFO:tasks.workunit.client.0.smithi123.stdout:7/585: write d2/dc/d1c/f7c [4048861,50012] 0 2024-03-30T04:28:14.162 INFO:tasks.workunit.client.1.smithi135.stdout:9/128: truncate d7/f22 1027834 0 2024-03-30T04:28:14.163 INFO:tasks.workunit.client.1.smithi135.stdout:5/184: sync 2024-03-30T04:28:14.164 INFO:tasks.workunit.client.0.smithi123.stdout:7/586: symlink d2/da/d11/d61/d55/d59/db5/dab/lc9 0 2024-03-30T04:28:14.166 INFO:tasks.workunit.client.1.smithi135.stdout:0/231: creat d0/d13/d1e/f46 x:0 0 0 2024-03-30T04:28:14.167 INFO:tasks.workunit.client.0.smithi123.stdout:2/652: sync 2024-03-30T04:28:14.168 INFO:tasks.workunit.client.0.smithi123.stdout:7/587: mkdir d2/da/d11/d61/d55/d59/db5/dab/dca 0 2024-03-30T04:28:14.170 INFO:tasks.workunit.client.0.smithi123.stdout:2/653: write d0/d1f/d27/f8b [3534580,59015] 0 2024-03-30T04:28:14.171 INFO:tasks.workunit.client.1.smithi135.stdout:3/169: mkdir d3/d6/d22/d33 0 2024-03-30T04:28:14.172 INFO:tasks.workunit.client.0.smithi123.stdout:2/654: write d0/fb4 [1395963,20200] 0 2024-03-30T04:28:14.172 INFO:tasks.workunit.client.1.smithi135.stdout:5/185: mknod d1/c34 0 2024-03-30T04:28:14.173 INFO:tasks.workunit.client.0.smithi123.stdout:2/655: chown d0/d1f/d5e 3490 1 2024-03-30T04:28:14.174 INFO:tasks.workunit.client.1.smithi135.stdout:9/129: fsync d7/f22 0 2024-03-30T04:28:14.177 INFO:tasks.workunit.client.1.smithi135.stdout:7/259: truncate d4/db/de/f25 3654675 0 2024-03-30T04:28:14.177 INFO:tasks.workunit.client.1.smithi135.stdout:2/162: dwrite d4/da/fb [0,4194304] 0 2024-03-30T04:28:14.178 INFO:tasks.workunit.client.1.smithi135.stdout:5/186: write d1/d3/d2b/f2f [171041,115310] 0 2024-03-30T04:28:14.183 INFO:tasks.workunit.client.1.smithi135.stdout:3/170: creat d3/db/f34 x:0 0 0 2024-03-30T04:28:14.184 INFO:tasks.workunit.client.0.smithi123.stdout:2/656: fdatasync d0/d1f/d27/f8b 0 2024-03-30T04:28:14.187 INFO:tasks.workunit.client.1.smithi135.stdout:9/130: unlink d7/l10 0 2024-03-30T04:28:14.188 INFO:tasks.workunit.client.0.smithi123.stdout:2/657: mknod d0/d1f/d78/cd7 0 2024-03-30T04:28:14.191 INFO:tasks.workunit.client.0.smithi123.stdout:7/588: write d2/da/d11/d61/d55/d59/db5/f40 [755580,12515] 0 2024-03-30T04:28:14.194 INFO:tasks.workunit.client.1.smithi135.stdout:3/171: chown d3/db/l28 1031864 1 2024-03-30T04:28:14.198 INFO:tasks.workunit.client.0.smithi123.stdout:7/589: creat d2/dc/d49/fcb x:0 0 0 2024-03-30T04:28:14.200 INFO:tasks.workunit.client.0.smithi123.stdout:7/590: fdatasync d2/dc/d7f/fa3 0 2024-03-30T04:28:14.202 INFO:tasks.workunit.client.1.smithi135.stdout:8/206: dwrite d6/d23/f27 [4194304,4194304] 0 2024-03-30T04:28:14.202 INFO:tasks.workunit.client.0.smithi123.stdout:2/658: dread d0/f1d [0,4194304] 0 2024-03-30T04:28:14.205 INFO:tasks.workunit.client.0.smithi123.stdout:2/659: creat d0/d1f/d1c/d80/dbb/d97/fd8 x:0 0 0 2024-03-30T04:28:14.207 INFO:tasks.workunit.client.1.smithi135.stdout:5/187: getdents d1 0 2024-03-30T04:28:14.218 INFO:tasks.workunit.client.0.smithi123.stdout:7/591: dwrite d2/da/d11/d61/d55/d59/f88 [0,4194304] 0 2024-03-30T04:28:14.220 INFO:tasks.workunit.client.1.smithi135.stdout:0/232: dwrite d0/d5/d9/f33 [4194304,4194304] 0 2024-03-30T04:28:14.221 INFO:tasks.workunit.client.1.smithi135.stdout:3/172: creat d3/d10/d16/d23/f35 x:0 0 0 2024-03-30T04:28:14.223 INFO:tasks.workunit.client.0.smithi123.stdout:7/592: fdatasync d2/da/d11/f58 0 2024-03-30T04:28:14.224 INFO:tasks.workunit.client.1.smithi135.stdout:5/188: mknod d1/d3/c35 0 2024-03-30T04:28:14.226 INFO:tasks.workunit.client.0.smithi123.stdout:7/593: write d2/dc/d49/dbc/fc5 [874929,76404] 0 2024-03-30T04:28:14.230 INFO:tasks.workunit.client.1.smithi135.stdout:8/207: link d6/d8/d15/d2a/f37 d6/d8/d15/d2a/f3c 0 2024-03-30T04:28:14.231 INFO:tasks.workunit.client.1.smithi135.stdout:3/173: chown d3/d6/cd 64079 1 2024-03-30T04:28:14.232 INFO:tasks.workunit.client.1.smithi135.stdout:7/260: dwrite d4/f16 [0,4194304] 0 2024-03-30T04:28:14.232 INFO:tasks.workunit.client.0.smithi123.stdout:7/594: fsync d2/dc/f81 0 2024-03-30T04:28:14.233 INFO:tasks.workunit.client.1.smithi135.stdout:0/233: truncate d0/d1/f29 495338 0 2024-03-30T04:28:14.241 INFO:tasks.workunit.client.1.smithi135.stdout:2/163: dwrite d4/d6/f35 [0,4194304] 0 2024-03-30T04:28:14.242 INFO:tasks.workunit.client.0.smithi123.stdout:7/595: truncate d2/da/d11/d34/d85/d8e/fbb 432508 0 2024-03-30T04:28:14.245 INFO:tasks.workunit.client.0.smithi123.stdout:7/596: dread - d2/dc/d49/d68/f9d zero size 2024-03-30T04:28:14.259 INFO:tasks.workunit.client.1.smithi135.stdout:8/208: rmdir d6/d8/d15 39 2024-03-30T04:28:14.262 INFO:tasks.workunit.client.1.smithi135.stdout:7/261: mkdir d4/d4d/d2c/d51/d56 0 2024-03-30T04:28:14.263 INFO:tasks.workunit.client.1.smithi135.stdout:2/164: creat d4/d32/d12/f37 x:0 0 0 2024-03-30T04:28:14.264 INFO:tasks.workunit.client.1.smithi135.stdout:9/131: dwrite d7/ff [0,4194304] 0 2024-03-30T04:28:14.266 INFO:tasks.workunit.client.0.smithi123.stdout:2/660: dread d0/d1f/d16/f26 [0,4194304] 0 2024-03-30T04:28:14.278 INFO:tasks.workunit.client.1.smithi135.stdout:7/262: dread d4/db/f26 [0,4194304] 0 2024-03-30T04:28:14.290 INFO:tasks.workunit.client.1.smithi135.stdout:2/165: dread d4/da/fb [0,4194304] 0 2024-03-30T04:28:14.291 INFO:tasks.workunit.client.1.smithi135.stdout:5/189: dread d1/d7/ff [0,4194304] 0 2024-03-30T04:28:14.294 INFO:tasks.workunit.client.1.smithi135.stdout:5/190: fdatasync d1/d7/f2c 0 2024-03-30T04:28:14.294 INFO:tasks.workunit.client.1.smithi135.stdout:3/174: dwrite d3/d6/fa [0,4194304] 0 2024-03-30T04:28:14.301 INFO:tasks.workunit.client.1.smithi135.stdout:5/191: creat d1/f36 x:0 0 0 2024-03-30T04:28:14.302 INFO:tasks.workunit.client.1.smithi135.stdout:2/166: write d4/f5 [1214977,66735] 0 2024-03-30T04:28:14.303 INFO:tasks.workunit.client.1.smithi135.stdout:3/175: truncate d3/d17/f1c 572107 0 2024-03-30T04:28:14.306 INFO:tasks.workunit.client.1.smithi135.stdout:3/176: readlink l1 0 2024-03-30T04:28:14.306 INFO:tasks.workunit.client.1.smithi135.stdout:9/132: dread f6 [0,4194304] 0 2024-03-30T04:28:14.308 INFO:tasks.workunit.client.1.smithi135.stdout:5/192: write d1/d3/f15 [1616273,44748] 0 2024-03-30T04:28:14.310 INFO:tasks.workunit.client.1.smithi135.stdout:9/133: symlink d7/l24 0 2024-03-30T04:28:14.312 INFO:tasks.workunit.client.1.smithi135.stdout:9/134: fdatasync d7/f12 0 2024-03-30T04:28:14.314 INFO:tasks.workunit.client.1.smithi135.stdout:7/263: dwrite d4/db/f1f [0,4194304] 0 2024-03-30T04:28:14.318 INFO:tasks.workunit.client.1.smithi135.stdout:7/264: chown d4/db/d1a/d4b/f31 1 1 2024-03-30T04:28:14.324 INFO:tasks.workunit.client.1.smithi135.stdout:9/135: creat d7/d1b/f25 x:0 0 0 2024-03-30T04:28:14.329 INFO:tasks.workunit.client.1.smithi135.stdout:9/136: creat d7/f26 x:0 0 0 2024-03-30T04:28:14.332 INFO:tasks.workunit.client.1.smithi135.stdout:2/167: dwrite d4/d6/d20/f2c [0,4194304] 0 2024-03-30T04:28:14.334 INFO:tasks.workunit.client.1.smithi135.stdout:2/168: chown d4/d6 0 1 2024-03-30T04:28:14.341 INFO:tasks.workunit.client.1.smithi135.stdout:2/169: read d4/d6/f1a [2205863,5019] 0 2024-03-30T04:28:14.341 INFO:tasks.workunit.client.1.smithi135.stdout:9/137: truncate d7/f12 1127281 0 2024-03-30T04:28:14.341 INFO:tasks.workunit.client.1.smithi135.stdout:8/209: dread d6/f12 [0,4194304] 0 2024-03-30T04:28:14.341 INFO:tasks.workunit.client.1.smithi135.stdout:5/193: dwrite d1/da/f18 [0,4194304] 0 2024-03-30T04:28:14.355 INFO:tasks.workunit.client.1.smithi135.stdout:2/170: write d4/da/fb [924828,59481] 0 2024-03-30T04:28:14.358 INFO:tasks.workunit.client.1.smithi135.stdout:9/138: dread f1 [0,4194304] 0 2024-03-30T04:28:14.358 INFO:tasks.workunit.client.1.smithi135.stdout:5/194: creat d1/f37 x:0 0 0 2024-03-30T04:28:14.363 INFO:tasks.workunit.client.1.smithi135.stdout:5/195: unlink d1/f4 0 2024-03-30T04:28:14.364 INFO:tasks.workunit.client.1.smithi135.stdout:7/265: dwrite d4/db/f1c [0,4194304] 0 2024-03-30T04:28:14.366 INFO:tasks.workunit.client.1.smithi135.stdout:9/139: rename d7/d1b/c15 to d7/c27 0 2024-03-30T04:28:14.370 INFO:tasks.workunit.client.1.smithi135.stdout:7/266: rename d4/d4d to d4/d4d/d2c/d57 22 2024-03-30T04:28:14.379 INFO:tasks.workunit.client.1.smithi135.stdout:8/210: dwrite d6/f12 [0,4194304] 0 2024-03-30T04:28:14.379 INFO:tasks.workunit.client.0.smithi123.stdout:3/612: sync 2024-03-30T04:28:14.383 INFO:tasks.workunit.client.1.smithi135.stdout:2/171: truncate d4/d32/f33 3449127 0 2024-03-30T04:28:14.384 INFO:tasks.workunit.client.1.smithi135.stdout:9/140: write f1 [310176,114276] 0 2024-03-30T04:28:14.387 INFO:tasks.workunit.client.1.smithi135.stdout:7/267: write d4/db/d1a/f24 [2024463,94842] 0 2024-03-30T04:28:14.388 INFO:tasks.workunit.client.1.smithi135.stdout:6/129: sync 2024-03-30T04:28:14.389 INFO:tasks.workunit.client.1.smithi135.stdout:8/211: symlink d6/d8/d15/l3d 0 2024-03-30T04:28:14.390 INFO:tasks.workunit.client.1.smithi135.stdout:4/134: sync 2024-03-30T04:28:14.392 INFO:tasks.workunit.client.1.smithi135.stdout:5/196: dwrite d1/d7/f10 [0,4194304] 0 2024-03-30T04:28:14.398 INFO:tasks.workunit.client.1.smithi135.stdout:9/141: creat d7/d1b/f28 x:0 0 0 2024-03-30T04:28:14.400 INFO:tasks.workunit.client.0.smithi123.stdout:3/613: write d13/d24/d31/d23/d3a/d3c/d4e/d5d/f8e [148878,38068] 0 2024-03-30T04:28:14.401 INFO:tasks.workunit.client.1.smithi135.stdout:8/212: symlink d6/d8/d15/d2a/d35/l3e 0 2024-03-30T04:28:14.402 INFO:tasks.workunit.client.1.smithi135.stdout:4/135: creat d10/d1a/f2e x:0 0 0 2024-03-30T04:28:14.402 INFO:tasks.workunit.client.1.smithi135.stdout:5/197: rename d1/d3/f12 to d1/d3/f38 0 2024-03-30T04:28:14.404 INFO:tasks.workunit.client.0.smithi123.stdout:3/614: unlink l2 0 2024-03-30T04:28:14.406 INFO:tasks.workunit.client.1.smithi135.stdout:9/142: mkdir d7/d29 0 2024-03-30T04:28:14.406 INFO:tasks.workunit.client.1.smithi135.stdout:8/213: dread - d6/d8/d15/d2a/f37 zero size 2024-03-30T04:28:14.407 INFO:tasks.workunit.client.1.smithi135.stdout:2/172: rename d4/d32/d12/f37 to d4/d32/d2e/d2f/f38 0 2024-03-30T04:28:14.408 INFO:tasks.workunit.client.1.smithi135.stdout:5/198: readlink d1/d7/l31 0 2024-03-30T04:28:14.408 INFO:tasks.workunit.client.0.smithi123.stdout:3/615: mknod d13/d24/d31/d23/d3a/d3c/d4e/cc4 0 2024-03-30T04:28:14.409 INFO:tasks.workunit.client.1.smithi135.stdout:8/214: fdatasync d6/d23/f27 0 2024-03-30T04:28:14.410 INFO:tasks.workunit.client.0.smithi123.stdout:3/616: read d13/f80 [1549725,30438] 0 2024-03-30T04:28:14.414 INFO:tasks.workunit.client.1.smithi135.stdout:5/199: chown d1/c34 2187829 1 2024-03-30T04:28:14.415 INFO:tasks.workunit.client.1.smithi135.stdout:4/136: creat d10/d1a/d2b/f2f x:0 0 0 2024-03-30T04:28:14.415 INFO:tasks.workunit.client.0.smithi123.stdout:3/617: mknod d13/d24/d31/d23/d3a/d3c/dab/cc5 0 2024-03-30T04:28:14.418 INFO:tasks.workunit.client.1.smithi135.stdout:8/215: write d6/f12 [1635216,65184] 0 2024-03-30T04:28:14.419 INFO:tasks.workunit.client.0.smithi123.stdout:3/618: mknod d13/d24/d31/d23/d3a/d3c/d4e/cc6 0 2024-03-30T04:28:14.422 INFO:tasks.workunit.client.1.smithi135.stdout:6/130: dwrite d2/f4 [0,4194304] 0 2024-03-30T04:28:14.423 INFO:tasks.workunit.client.1.smithi135.stdout:5/200: write d1/d3/f15 [6146987,51039] 0 2024-03-30T04:28:14.426 INFO:tasks.workunit.client.1.smithi135.stdout:4/137: dread - d10/d1a/f27 zero size 2024-03-30T04:28:14.437 INFO:tasks.workunit.client.1.smithi135.stdout:7/268: dwrite d4/d4d/d2c/f39 [4194304,4194304] 0 2024-03-30T04:28:14.437 INFO:tasks.workunit.client.1.smithi135.stdout:5/201: write d1/f36 [698215,56335] 0 2024-03-30T04:28:14.440 INFO:tasks.workunit.client.0.smithi123.stdout:3/619: truncate d13/d24/d31/fae 1743040 0 2024-03-30T04:28:14.447 INFO:tasks.workunit.client.1.smithi135.stdout:6/131: symlink d2/dd/d18/l28 0 2024-03-30T04:28:14.449 INFO:tasks.workunit.client.1.smithi135.stdout:5/202: write d1/d7/f13 [1479809,103792] 0 2024-03-30T04:28:14.452 INFO:tasks.workunit.client.1.smithi135.stdout:9/143: dwrite d7/f22 [0,4194304] 0 2024-03-30T04:28:14.456 INFO:tasks.workunit.client.1.smithi135.stdout:2/173: dwrite d4/f1c [4194304,4194304] 0 2024-03-30T04:28:14.464 INFO:tasks.workunit.client.1.smithi135.stdout:8/216: getdents d6 0 2024-03-30T04:28:14.474 INFO:tasks.workunit.client.1.smithi135.stdout:5/203: creat d1/f39 x:0 0 0 2024-03-30T04:28:14.475 INFO:tasks.workunit.client.1.smithi135.stdout:7/269: rename d4/d4d/f37 to d4/db/d1a/d4b/d40/d3a/f58 0 2024-03-30T04:28:14.477 INFO:tasks.workunit.client.1.smithi135.stdout:5/204: write d1/d7/f2c [85850,12845] 0 2024-03-30T04:28:14.482 INFO:tasks.workunit.client.1.smithi135.stdout:9/144: fsync f6 0 2024-03-30T04:28:14.485 INFO:tasks.workunit.client.1.smithi135.stdout:9/145: chown d7/d1b/f25 100102991 1 2024-03-30T04:28:14.486 INFO:tasks.workunit.client.1.smithi135.stdout:9/146: read f6 [154286,87718] 0 2024-03-30T04:28:14.501 INFO:tasks.workunit.client.1.smithi135.stdout:4/138: dwrite fd [0,4194304] 0 2024-03-30T04:28:14.503 INFO:tasks.workunit.client.1.smithi135.stdout:1/159: sync 2024-03-30T04:28:14.504 INFO:tasks.workunit.client.1.smithi135.stdout:5/205: dwrite d1/d3/f15 [0,4194304] 0 2024-03-30T04:28:14.505 INFO:tasks.workunit.client.1.smithi135.stdout:8/217: getdents d6/d8/d15/d2a/d35 0 2024-03-30T04:28:14.505 INFO:tasks.workunit.client.1.smithi135.stdout:7/270: dread d4/db/d1a/d4b/d40/d3a/f58 [0,4194304] 0 2024-03-30T04:28:14.507 INFO:tasks.workunit.client.1.smithi135.stdout:6/132: write d2/f12 [2029810,102512] 0 2024-03-30T04:28:14.507 INFO:tasks.workunit.client.1.smithi135.stdout:2/174: getdents d4/d32 0 2024-03-30T04:28:14.507 INFO:tasks.workunit.client.1.smithi135.stdout:4/139: mkdir d10/d1a/d30 0 2024-03-30T04:28:14.508 INFO:tasks.workunit.client.1.smithi135.stdout:9/147: dread d7/f11 [0,4194304] 0 2024-03-30T04:28:14.511 INFO:tasks.workunit.client.1.smithi135.stdout:5/206: mkdir d1/d7/d11/d3a 0 2024-03-30T04:28:14.513 INFO:tasks.workunit.client.1.smithi135.stdout:7/271: mkdir d4/d4d/d59 0 2024-03-30T04:28:14.515 INFO:tasks.workunit.client.1.smithi135.stdout:4/140: creat d10/d1a/d2b/f31 x:0 0 0 2024-03-30T04:28:14.519 INFO:tasks.workunit.client.1.smithi135.stdout:9/148: rename d7/f20 to d7/d29/f2a 0 2024-03-30T04:28:14.519 INFO:tasks.workunit.client.1.smithi135.stdout:6/133: symlink d2/dd/d14/l29 0 2024-03-30T04:28:14.520 INFO:tasks.workunit.client.1.smithi135.stdout:5/207: creat d1/d7/d1a/f3b x:0 0 0 2024-03-30T04:28:14.522 INFO:tasks.workunit.client.1.smithi135.stdout:2/175: link d4/d32/d2e/d2f/f38 d4/d6/d20/f39 0 2024-03-30T04:28:14.525 INFO:tasks.workunit.client.1.smithi135.stdout:2/176: dread - d4/d32/d2e/d2f/f30 zero size 2024-03-30T04:28:14.525 INFO:tasks.workunit.client.1.smithi135.stdout:4/141: mkdir d10/d1a/d32 0 2024-03-30T04:28:14.527 INFO:tasks.workunit.client.1.smithi135.stdout:5/208: mkdir d1/d7/d11/d3c 0 2024-03-30T04:28:14.528 INFO:tasks.workunit.client.1.smithi135.stdout:9/149: mkdir d7/d1b/d2b 0 2024-03-30T04:28:14.529 INFO:tasks.workunit.client.1.smithi135.stdout:4/142: chown d10/d1a/d2b/f31 317929 1 2024-03-30T04:28:14.530 INFO:tasks.workunit.client.1.smithi135.stdout:2/177: write d4/d32/d12/f2a [134877,107672] 0 2024-03-30T04:28:14.533 INFO:tasks.workunit.client.1.smithi135.stdout:4/143: chown d10/d1a/f1c 12581 1 2024-03-30T04:28:14.535 INFO:tasks.workunit.client.1.smithi135.stdout:5/209: creat d1/da/d17/f3d x:0 0 0 2024-03-30T04:28:14.538 INFO:tasks.workunit.client.1.smithi135.stdout:9/150: creat d7/d29/f2c x:0 0 0 2024-03-30T04:28:14.538 INFO:tasks.workunit.client.1.smithi135.stdout:2/178: symlink d4/d32/l3a 0 2024-03-30T04:28:14.541 INFO:tasks.workunit.client.1.smithi135.stdout:4/144: mknod d10/c33 0 2024-03-30T04:28:14.542 INFO:tasks.workunit.client.1.smithi135.stdout:4/145: dread - d10/f14 zero size 2024-03-30T04:28:14.542 INFO:tasks.workunit.client.1.smithi135.stdout:9/151: truncate d7/d29/f2c 968054 0 2024-03-30T04:28:14.543 INFO:tasks.workunit.client.1.smithi135.stdout:5/210: creat d1/f3e x:0 0 0 2024-03-30T04:28:14.544 INFO:tasks.workunit.client.1.smithi135.stdout:4/146: chown d10/f17 2 1 2024-03-30T04:28:14.546 INFO:tasks.workunit.client.1.smithi135.stdout:2/179: creat d4/d32/d2e/d2f/f3b x:0 0 0 2024-03-30T04:28:14.546 INFO:tasks.workunit.client.1.smithi135.stdout:4/147: write fd [2173268,110182] 0 2024-03-30T04:28:14.548 INFO:tasks.workunit.client.1.smithi135.stdout:2/180: dread - d4/d32/d2e/d2f/f3b zero size 2024-03-30T04:28:14.554 INFO:tasks.workunit.client.1.smithi135.stdout:4/148: truncate d10/d1a/d2b/f31 523300 0 2024-03-30T04:28:14.555 INFO:tasks.workunit.client.1.smithi135.stdout:5/211: rename d1/da/c22 to d1/d7/c3f 0 2024-03-30T04:28:14.555 INFO:tasks.workunit.client.0.smithi123.stdout:5/538: sync 2024-03-30T04:28:14.555 INFO:tasks.workunit.client.0.smithi123.stdout:9/532: sync 2024-03-30T04:28:14.555 INFO:tasks.workunit.client.0.smithi123.stdout:8/555: sync 2024-03-30T04:28:14.555 INFO:tasks.workunit.client.0.smithi123.stdout:2/661: sync 2024-03-30T04:28:14.556 INFO:tasks.workunit.client.0.smithi123.stdout:7/597: sync 2024-03-30T04:28:14.556 INFO:tasks.workunit.client.0.smithi123.stdout:1/678: sync 2024-03-30T04:28:14.556 INFO:tasks.workunit.client.0.smithi123.stdout:6/671: sync 2024-03-30T04:28:14.556 INFO:tasks.workunit.client.0.smithi123.stdout:4/580: sync 2024-03-30T04:28:14.556 INFO:tasks.workunit.client.0.smithi123.stdout:0/607: sync 2024-03-30T04:28:14.556 INFO:tasks.workunit.client.1.smithi135.stdout:2/181: mkdir d4/d6/d20/d3c 0 2024-03-30T04:28:14.559 INFO:tasks.workunit.client.1.smithi135.stdout:6/134: getdents d2/dd/d14 0 2024-03-30T04:28:14.560 INFO:tasks.workunit.client.0.smithi123.stdout:1/679: stat db/dd/d42/d8f/d78/fa0 0 2024-03-30T04:28:14.560 INFO:tasks.workunit.client.1.smithi135.stdout:4/149: write d10/d1a/d2b/f2f [493723,94216] 0 2024-03-30T04:28:14.561 INFO:tasks.workunit.client.1.smithi135.stdout:2/182: dread - d4/d32/d2e/d2f/f3b zero size 2024-03-30T04:28:14.562 INFO:tasks.workunit.client.0.smithi123.stdout:8/556: write da/dc/d14/f44 [744950,22196] 0 2024-03-30T04:28:14.562 INFO:tasks.workunit.client.0.smithi123.stdout:5/539: creat dd/d19/d3e/d5b/d62/faa x:0 0 0 2024-03-30T04:28:14.563 INFO:tasks.workunit.client.0.smithi123.stdout:4/581: creat d2/fcb x:0 0 0 2024-03-30T04:28:14.565 INFO:tasks.workunit.client.1.smithi135.stdout:2/183: chown d4/d32/f13 26104 1 2024-03-30T04:28:14.565 INFO:tasks.workunit.client.0.smithi123.stdout:9/533: rename d6/d1f/d45/d4c/c98 to d6/d1f/d45/d4c/cba 0 2024-03-30T04:28:14.566 INFO:tasks.workunit.client.1.smithi135.stdout:6/135: write d2/dd/d18/f22 [291859,126741] 0 2024-03-30T04:28:14.569 INFO:tasks.workunit.client.1.smithi135.stdout:9/152: dwrite d7/d1b/f25 [0,4194304] 0 2024-03-30T04:28:14.571 INFO:tasks.workunit.client.0.smithi123.stdout:9/534: stat d6/f10 0 2024-03-30T04:28:14.573 INFO:tasks.workunit.client.0.smithi123.stdout:0/608: dread d6/dd/f4a [0,4194304] 0 2024-03-30T04:28:14.574 INFO:tasks.workunit.client.0.smithi123.stdout:7/598: rename d2/da/d22/f73 to d2/dc/d1c/fcc 0 2024-03-30T04:28:14.575 INFO:tasks.workunit.client.0.smithi123.stdout:9/535: mknod d6/d7c/cbb 0 2024-03-30T04:28:14.577 INFO:tasks.workunit.client.1.smithi135.stdout:9/153: mknod d7/d29/c2d 0 2024-03-30T04:28:14.578 INFO:tasks.workunit.client.0.smithi123.stdout:0/609: read d6/dd/d15/f2b [2135722,35893] 0 2024-03-30T04:28:14.579 INFO:tasks.workunit.client.0.smithi123.stdout:6/672: rename d10/d82/d8e/dc7/f71 to d10/d82/dc1/d45/fe5 0 2024-03-30T04:28:14.580 INFO:tasks.workunit.client.1.smithi135.stdout:6/136: dread d2/dd/d18/f22 [0,4194304] 0 2024-03-30T04:28:14.580 INFO:tasks.workunit.client.1.smithi135.stdout:5/212: truncate d1/d7/f13 468827 0 2024-03-30T04:28:14.583 INFO:tasks.workunit.client.1.smithi135.stdout:6/137: readlink d2/dd/lf 0 2024-03-30T04:28:14.584 INFO:tasks.workunit.client.0.smithi123.stdout:9/536: fdatasync d6/d1f/f3f 0 2024-03-30T04:28:14.584 INFO:tasks.workunit.client.1.smithi135.stdout:9/154: mknod d7/d1b/c2e 0 2024-03-30T04:28:14.585 INFO:tasks.workunit.client.0.smithi123.stdout:7/599: truncate d2/dc/d56/f67 5018974 0 2024-03-30T04:28:14.586 INFO:tasks.workunit.client.0.smithi123.stdout:5/540: dwrite dd/d19/f9f [0,4194304] 0 2024-03-30T04:28:14.587 INFO:tasks.workunit.client.1.smithi135.stdout:5/213: truncate d1/d3/f38 1661118 0 2024-03-30T04:28:14.587 INFO:tasks.workunit.client.1.smithi135.stdout:9/155: chown d7/d1b/d2b 15283794 1 2024-03-30T04:28:14.588 INFO:tasks.workunit.client.1.smithi135.stdout:2/184: creat d4/d32/d2e/f3d x:0 0 0 2024-03-30T04:28:14.590 INFO:tasks.workunit.client.0.smithi123.stdout:6/673: mknod d10/d82/d9a/db3/ce6 0 2024-03-30T04:28:14.591 INFO:tasks.workunit.client.0.smithi123.stdout:8/557: write da/dc/f10 [511402,45563] 0 2024-03-30T04:28:14.591 INFO:tasks.workunit.client.1.smithi135.stdout:5/214: write d1/d3/f15 [7746659,25535] 0 2024-03-30T04:28:14.592 INFO:tasks.workunit.client.1.smithi135.stdout:9/156: readlink d7/d1b/l9 0 2024-03-30T04:28:14.593 INFO:tasks.workunit.client.1.smithi135.stdout:2/185: chown d4/da/fb 8296116 1 2024-03-30T04:28:14.593 INFO:tasks.workunit.client.1.smithi135.stdout:4/150: dread fb [0,4194304] 0 2024-03-30T04:28:14.596 INFO:tasks.workunit.client.0.smithi123.stdout:7/600: rmdir d2/da/d11/d34/d76/d91 39 2024-03-30T04:28:14.596 INFO:tasks.workunit.client.0.smithi123.stdout:8/558: stat da/d27/d3c/d64/f81 0 2024-03-30T04:28:14.596 INFO:tasks.workunit.client.0.smithi123.stdout:6/674: readlink d10/d82/d8e/dc7/d5b/lc0 0 2024-03-30T04:28:14.596 INFO:tasks.workunit.client.0.smithi123.stdout:8/559: stat da/dc/d14/d18/f4f 0 2024-03-30T04:28:14.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:14 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:28:14.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:14 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:28:14.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:14 smithi135 ceph-mon[44364]: Reconfiguring node-exporter.smithi135 (dependencies changed)... 2024-03-30T04:28:14.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:14 smithi135 ceph-mon[44364]: Reconfiguring daemon node-exporter.smithi135 on smithi135 2024-03-30T04:28:14.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:14 smithi135 ceph-mon[44364]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:28:14.602 INFO:tasks.workunit.client.0.smithi123.stdout:2/662: truncate d0/d1f/d16/f26 2633239 0 2024-03-30T04:28:14.602 INFO:tasks.workunit.client.1.smithi135.stdout:2/186: symlink d4/d32/d12/l3e 0 2024-03-30T04:28:14.602 INFO:tasks.workunit.client.0.smithi123.stdout:5/541: rename f2 to dd/d19/d1f/d3c/fab 0 2024-03-30T04:28:14.603 INFO:tasks.workunit.client.1.smithi135.stdout:6/138: truncate d2/f11 4749112 0 2024-03-30T04:28:14.605 INFO:tasks.workunit.client.0.smithi123.stdout:4/582: dread d2/d34/f4c [0,4194304] 0 2024-03-30T04:28:14.608 INFO:tasks.workunit.client.0.smithi123.stdout:2/663: write d0/d1f/d1c/d39/d4b/d4c/f64 [910909,16391] 0 2024-03-30T04:28:14.608 INFO:tasks.workunit.client.0.smithi123.stdout:4/583: chown d2/d4/d2c/d42/dad/d91 84 1 2024-03-30T04:28:14.608 INFO:tasks.workunit.client.1.smithi135.stdout:5/215: creat d1/d7/d1a/d1f/f40 x:0 0 0 2024-03-30T04:28:14.609 INFO:tasks.workunit.client.0.smithi123.stdout:7/601: symlink d2/dc/d1c/lcd 0 2024-03-30T04:28:14.611 INFO:tasks.workunit.client.1.smithi135.stdout:6/139: unlink d2/dd/l1f 0 2024-03-30T04:28:14.612 INFO:tasks.workunit.client.1.smithi135.stdout:6/140: read - d2/fa zero size 2024-03-30T04:28:14.613 INFO:tasks.workunit.client.0.smithi123.stdout:9/537: write d6/d1f/d45/d97/d69/f6c [124171,120640] 0 2024-03-30T04:28:14.613 INFO:tasks.workunit.client.1.smithi135.stdout:4/151: link d10/f17 d10/f34 0 2024-03-30T04:28:14.615 INFO:tasks.workunit.client.0.smithi123.stdout:8/560: dread da/dc/d14/d18/d21/f4d [0,4194304] 0 2024-03-30T04:28:14.617 INFO:tasks.workunit.client.1.smithi135.stdout:9/157: dwrite f1 [0,4194304] 0 2024-03-30T04:28:14.617 INFO:tasks.workunit.client.1.smithi135.stdout:2/187: rename d4/d32/f23 to d4/d32/d12/f3f 0 2024-03-30T04:28:14.618 INFO:tasks.workunit.client.0.smithi123.stdout:6/675: link d10/d82/d8e/dc7/d42/f80 d10/d82/d8e/dc7/d42/fe7 0 2024-03-30T04:28:14.619 INFO:tasks.workunit.client.0.smithi123.stdout:4/584: symlink d2/d4/d2c/d42/d80/d96/lcc 0 2024-03-30T04:28:14.620 INFO:tasks.workunit.client.1.smithi135.stdout:2/188: readlink d4/d32/l3a 0 2024-03-30T04:28:14.620 INFO:tasks.workunit.client.1.smithi135.stdout:5/216: symlink d1/d7/d11/l41 0 2024-03-30T04:28:14.621 INFO:tasks.workunit.client.1.smithi135.stdout:5/217: chown d1/d3 1 1 2024-03-30T04:28:14.622 INFO:tasks.workunit.client.1.smithi135.stdout:6/141: fdatasync d2/dd/d14/f1e 0 2024-03-30T04:28:14.624 INFO:tasks.workunit.client.0.smithi123.stdout:4/585: chown d2/d4/d2c/f84 13711310 1 2024-03-30T04:28:14.625 INFO:tasks.workunit.client.1.smithi135.stdout:4/152: symlink d10/l35 0 2024-03-30T04:28:14.628 INFO:tasks.workunit.client.0.smithi123.stdout:3/620: sync 2024-03-30T04:28:14.630 INFO:tasks.workunit.client.1.smithi135.stdout:5/218: symlink d1/l42 0 2024-03-30T04:28:14.632 INFO:tasks.workunit.client.1.smithi135.stdout:5/219: truncate d1/f3e 177542 0 2024-03-30T04:28:14.633 INFO:tasks.workunit.client.1.smithi135.stdout:5/220: dread - d1/f37 zero size 2024-03-30T04:28:14.634 INFO:tasks.workunit.client.0.smithi123.stdout:5/542: dwrite dd/f7c [0,4194304] 0 2024-03-30T04:28:14.634 INFO:tasks.workunit.client.0.smithi123.stdout:4/586: rename d2/d26/f4f to d2/d4/d2c/fcd 0 2024-03-30T04:28:14.637 INFO:tasks.workunit.client.1.smithi135.stdout:9/158: rmdir d7/d1b/d2b 0 2024-03-30T04:28:14.642 INFO:tasks.workunit.client.1.smithi135.stdout:2/189: rename d4/d32/d2e/d2f/f3b to d4/d32/d2e/f40 0 2024-03-30T04:28:14.643 INFO:tasks.workunit.client.0.smithi123.stdout:2/664: dread d0/d1f/d1c/d40/f5a [0,4194304] 0 2024-03-30T04:28:14.646 INFO:tasks.workunit.client.1.smithi135.stdout:5/221: creat d1/d7/f43 x:0 0 0 2024-03-30T04:28:14.646 INFO:tasks.workunit.client.0.smithi123.stdout:9/538: dread d6/d1f/d45/d4c/f54 [4194304,4194304] 0 2024-03-30T04:28:14.647 INFO:tasks.workunit.client.0.smithi123.stdout:6/676: dread f6 [0,4194304] 0 2024-03-30T04:28:14.647 INFO:tasks.workunit.client.0.smithi123.stdout:3/621: mkdir d13/d24/d31/d23/d3a/d3c/d4e/d5d/dc7 0 2024-03-30T04:28:14.649 INFO:tasks.workunit.client.1.smithi135.stdout:4/153: rmdir d10/d1a/d30 0 2024-03-30T04:28:14.649 INFO:tasks.workunit.client.0.smithi123.stdout:5/543: unlink dd/d19/d1f/ca0 0 2024-03-30T04:28:14.653 INFO:tasks.workunit.client.0.smithi123.stdout:3/622: chown d13/c67 622835 1 2024-03-30T04:28:14.653 INFO:tasks.workunit.client.0.smithi123.stdout:2/665: unlink d0/d1f/d1c/d80/dbb/d97/fc1 0 2024-03-30T04:28:14.656 INFO:tasks.workunit.client.0.smithi123.stdout:9/539: creat d6/d1f/d92/db8/fbc x:0 0 0 2024-03-30T04:28:14.657 INFO:tasks.workunit.client.1.smithi135.stdout:9/159: dwrite d7/d1b/f1c [0,4194304] 0 2024-03-30T04:28:14.657 INFO:tasks.workunit.client.1.smithi135.stdout:6/142: dread d2/f9 [4194304,4194304] 0 2024-03-30T04:28:14.660 INFO:tasks.workunit.client.1.smithi135.stdout:6/143: write d2/f13 [5109653,130382] 0 2024-03-30T04:28:14.662 INFO:tasks.workunit.client.0.smithi123.stdout:7/602: dwrite d2/dc/d49/f63 [0,4194304] 0 2024-03-30T04:28:14.662 INFO:tasks.workunit.client.0.smithi123.stdout:5/544: creat dd/d19/d1f/d3c/d72/d8c/fac x:0 0 0 2024-03-30T04:28:14.662 INFO:tasks.workunit.client.1.smithi135.stdout:9/160: chown d7/c23 8874858 1 2024-03-30T04:28:14.664 INFO:tasks.workunit.client.1.smithi135.stdout:2/190: rename d4/d6/c15 to d4/d32/d12/c41 0 2024-03-30T04:28:14.667 INFO:tasks.workunit.client.1.smithi135.stdout:2/191: chown d4/da/l22 41 1 2024-03-30T04:28:14.668 INFO:tasks.workunit.client.1.smithi135.stdout:4/154: truncate d10/f17 2411042 0 2024-03-30T04:28:14.668 INFO:tasks.workunit.client.0.smithi123.stdout:2/666: creat d0/d1f/d1c/d39/fd9 x:0 0 0 2024-03-30T04:28:14.669 INFO:tasks.workunit.client.0.smithi123.stdout:6/677: truncate d10/d82/dc1/d6a/d7a/d9f/fc6 847048 0 2024-03-30T04:28:14.669 INFO:tasks.workunit.client.1.smithi135.stdout:2/192: chown d4/d6/f35 46955 1 2024-03-30T04:28:14.670 INFO:tasks.workunit.client.1.smithi135.stdout:4/155: dread - d10/d1a/f1c zero size 2024-03-30T04:28:14.670 INFO:tasks.workunit.client.0.smithi123.stdout:9/540: symlink d6/d1f/lbd 0 2024-03-30T04:28:14.671 INFO:tasks.workunit.client.1.smithi135.stdout:9/161: creat d7/f2f x:0 0 0 2024-03-30T04:28:14.673 INFO:tasks.workunit.client.0.smithi123.stdout:2/667: chown d0/d1f/d16/lcd 10 1 2024-03-30T04:28:14.674 INFO:tasks.workunit.client.1.smithi135.stdout:4/156: symlink d10/l36 0 2024-03-30T04:28:14.678 INFO:tasks.workunit.client.0.smithi123.stdout:2/668: stat d0/d1f/d5e 0 2024-03-30T04:28:14.679 INFO:tasks.workunit.client.1.smithi135.stdout:5/222: getdents d1/da 0 2024-03-30T04:28:14.679 INFO:tasks.workunit.client.1.smithi135.stdout:4/157: readlink d10/d1a/l25 0 2024-03-30T04:28:14.679 INFO:tasks.workunit.client.1.smithi135.stdout:9/162: symlink d7/l30 0 2024-03-30T04:28:14.679 INFO:tasks.workunit.client.1.smithi135.stdout:0/234: sync 2024-03-30T04:28:14.679 INFO:tasks.workunit.client.1.smithi135.stdout:3/177: sync 2024-03-30T04:28:14.679 INFO:tasks.workunit.client.0.smithi123.stdout:7/603: dread - d2/dc/f52 zero size 2024-03-30T04:28:14.680 INFO:tasks.workunit.client.1.smithi135.stdout:5/223: write d1/f3e [164601,116691] 0 2024-03-30T04:28:14.681 INFO:tasks.workunit.client.1.smithi135.stdout:4/158: mknod d10/d1a/d32/c37 0 2024-03-30T04:28:14.681 INFO:tasks.workunit.client.1.smithi135.stdout:3/178: creat d3/d6/f36 x:0 0 0 2024-03-30T04:28:14.683 INFO:tasks.workunit.client.1.smithi135.stdout:0/235: mknod d0/d34/d3f/c47 0 2024-03-30T04:28:14.683 INFO:tasks.workunit.client.0.smithi123.stdout:8/561: dread da/d27/d3c/f5e [0,4194304] 0 2024-03-30T04:28:14.684 INFO:tasks.workunit.client.1.smithi135.stdout:4/159: chown d10/d1a/f1f 2137588855 1 2024-03-30T04:28:14.685 INFO:tasks.workunit.client.0.smithi123.stdout:2/669: creat d0/d37/d57/fda x:0 0 0 2024-03-30T04:28:14.688 INFO:tasks.workunit.client.1.smithi135.stdout:5/224: creat d1/d7/d11/d3c/f44 x:0 0 0 2024-03-30T04:28:14.689 INFO:tasks.workunit.client.0.smithi123.stdout:6/678: mkdir d10/d82/dc1/d6a/d7a/de8 0 2024-03-30T04:28:14.691 INFO:tasks.workunit.client.1.smithi135.stdout:5/225: fdatasync d1/d3/f15 0 2024-03-30T04:28:14.691 INFO:tasks.workunit.client.0.smithi123.stdout:7/604: rename d2/da/d11/d34/d76/caf to d2/da/d11/d61/d55/d59/db5/cce 0 2024-03-30T04:28:14.694 INFO:tasks.workunit.client.0.smithi123.stdout:9/541: truncate d6/d23/f33 1549158 0 2024-03-30T04:28:14.694 INFO:tasks.workunit.client.1.smithi135.stdout:6/144: dwrite d2/dd/d18/f22 [0,4194304] 0 2024-03-30T04:28:14.695 INFO:tasks.workunit.client.0.smithi123.stdout:4/587: creat d2/d4/d2c/d42/d35/d5d/dc0/fce x:0 0 0 2024-03-30T04:28:14.695 INFO:tasks.workunit.client.0.smithi123.stdout:8/562: mknod da/d27/d56/d6c/ca4 0 2024-03-30T04:28:14.696 INFO:tasks.workunit.client.1.smithi135.stdout:5/226: write d1/f39 [599602,39598] 0 2024-03-30T04:28:14.697 INFO:tasks.workunit.client.1.smithi135.stdout:3/179: symlink d3/db/l37 0 2024-03-30T04:28:14.698 INFO:tasks.workunit.client.1.smithi135.stdout:4/160: mkdir d10/d38 0 2024-03-30T04:28:14.698 INFO:tasks.workunit.client.1.smithi135.stdout:0/236: truncate d0/f12 773410 0 2024-03-30T04:28:14.698 INFO:tasks.workunit.client.0.smithi123.stdout:2/670: symlink d0/d1f/d1c/d20/dc6/ldb 0 2024-03-30T04:28:14.699 INFO:tasks.workunit.client.0.smithi123.stdout:7/605: stat d2/da/d11/d61/f62 0 2024-03-30T04:28:14.700 INFO:tasks.workunit.client.1.smithi135.stdout:6/145: readlink d2/dd/d18/l28 0 2024-03-30T04:28:14.700 INFO:tasks.workunit.client.1.smithi135.stdout:0/237: readlink d0/d1/l37 0 2024-03-30T04:28:14.702 INFO:tasks.workunit.client.0.smithi123.stdout:8/563: symlink da/dc/d14/la5 0 2024-03-30T04:28:14.704 INFO:tasks.workunit.client.1.smithi135.stdout:5/227: mkdir d1/d45 0 2024-03-30T04:28:14.704 INFO:tasks.workunit.client.0.smithi123.stdout:4/588: mkdir d2/d4/d7/d1a/dcf 0 2024-03-30T04:28:14.704 INFO:tasks.workunit.client.0.smithi123.stdout:5/545: dwrite dd/d41/f47 [0,4194304] 0 2024-03-30T04:28:14.705 INFO:tasks.workunit.client.1.smithi135.stdout:4/161: write d10/d1a/f21 [829722,16367] 0 2024-03-30T04:28:14.707 INFO:tasks.workunit.client.1.smithi135.stdout:6/146: chown d2/dd/d14/f1e 29798 1 2024-03-30T04:28:14.709 INFO:tasks.workunit.client.1.smithi135.stdout:0/238: creat d0/d5/d9/d23/d3e/f48 x:0 0 0 2024-03-30T04:28:14.712 INFO:tasks.workunit.client.1.smithi135.stdout:6/147: readlink d2/l23 0 2024-03-30T04:28:14.712 INFO:tasks.workunit.client.1.smithi135.stdout:9/163: dwrite d7/f11 [0,4194304] 0 2024-03-30T04:28:14.714 INFO:tasks.workunit.client.1.smithi135.stdout:0/239: chown d0/d13/d1d/d3b 14 1 2024-03-30T04:28:14.715 INFO:tasks.workunit.client.1.smithi135.stdout:9/164: fsync d7/d29/f2c 0 2024-03-30T04:28:14.718 INFO:tasks.workunit.client.0.smithi123.stdout:5/546: write dd/d19/d3e/d5b/d62/faa [354562,75969] 0 2024-03-30T04:28:14.718 INFO:tasks.workunit.client.0.smithi123.stdout:8/564: mknod da/d27/d56/d93/ca6 0 2024-03-30T04:28:14.718 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:14 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:28:14.718 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:14 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:28:14.718 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:14 smithi123 ceph-mon[35245]: Reconfiguring node-exporter.smithi135 (dependencies changed)... 2024-03-30T04:28:14.718 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:14 smithi123 ceph-mon[35245]: Reconfiguring daemon node-exporter.smithi135 on smithi135 2024-03-30T04:28:14.719 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:14 smithi123 ceph-mon[35245]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:28:14.719 INFO:tasks.workunit.client.1.smithi135.stdout:0/240: write d0/d5/d9/f15 [593700,127394] 0 2024-03-30T04:28:14.719 INFO:tasks.workunit.client.1.smithi135.stdout:5/228: mkdir d1/d7/d1a/d46 0 2024-03-30T04:28:14.719 INFO:tasks.workunit.client.1.smithi135.stdout:9/165: truncate d7/f26 999621 0 2024-03-30T04:28:14.723 INFO:tasks.workunit.client.0.smithi123.stdout:7/606: rename d2/da/d22/d3b/f6d to d2/da/d11/d34/d76/d91/fcf 0 2024-03-30T04:28:14.723 INFO:tasks.workunit.client.0.smithi123.stdout:3/623: dread d13/d24/d31/d23/f6c [0,4194304] 0 2024-03-30T04:28:14.724 INFO:tasks.workunit.client.0.smithi123.stdout:4/589: rmdir d2/d4/d2c/d42/d35/d9b/dba 0 2024-03-30T04:28:14.724 INFO:tasks.workunit.client.1.smithi135.stdout:3/180: dread f2 [0,4194304] 0 2024-03-30T04:28:14.726 INFO:tasks.workunit.client.1.smithi135.stdout:3/181: readlink d3/d10/l32 0 2024-03-30T04:28:14.726 INFO:tasks.workunit.client.1.smithi135.stdout:0/241: creat d0/d13/d1e/f49 x:0 0 0 2024-03-30T04:28:14.728 INFO:tasks.workunit.client.0.smithi123.stdout:5/547: mknod dd/d19/d1f/d3c/cad 0 2024-03-30T04:28:14.729 INFO:tasks.workunit.client.1.smithi135.stdout:9/166: symlink d7/d29/l31 0 2024-03-30T04:28:14.729 INFO:tasks.workunit.client.0.smithi123.stdout:8/565: creat da/dc/d14/d18/d62/fa7 x:0 0 0 2024-03-30T04:28:14.734 INFO:tasks.workunit.client.0.smithi123.stdout:4/590: symlink d2/d4/d2c/d42/d35/d5d/dc0/ld0 0 2024-03-30T04:28:14.734 INFO:tasks.workunit.client.0.smithi123.stdout:2/671: dread d0/d1f/d27/fb0 [0,4194304] 0 2024-03-30T04:28:14.736 INFO:tasks.workunit.client.1.smithi135.stdout:3/182: symlink d3/d6/l38 0 2024-03-30T04:28:14.745 INFO:tasks.workunit.client.0.smithi123.stdout:5/548: mknod dd/d19/d1f/d35/d3a/d79/cae 0 2024-03-30T04:28:14.748 INFO:tasks.workunit.client.1.smithi135.stdout:6/148: dwrite d2/f12 [0,4194304] 0 2024-03-30T04:28:14.749 INFO:tasks.workunit.client.1.smithi135.stdout:5/229: dwrite d1/da/d17/f3d [0,4194304] 0 2024-03-30T04:28:14.749 INFO:tasks.workunit.client.1.smithi135.stdout:9/167: dread d7/f22 [0,4194304] 0 2024-03-30T04:28:14.751 INFO:tasks.workunit.client.0.smithi123.stdout:3/624: dread d13/f8a [0,4194304] 0 2024-03-30T04:28:14.756 INFO:tasks.workunit.client.0.smithi123.stdout:5/549: unlink dd/d19/d1f/d3c/f77 0 2024-03-30T04:28:14.756 INFO:tasks.workunit.client.1.smithi135.stdout:9/168: write f1 [2833655,96616] 0 2024-03-30T04:28:14.760 INFO:tasks.workunit.client.1.smithi135.stdout:9/169: write d7/d29/f2c [1581980,120099] 0 2024-03-30T04:28:14.761 INFO:tasks.workunit.client.0.smithi123.stdout:8/566: dwrite da/f4e [0,4194304] 0 2024-03-30T04:28:14.761 INFO:tasks.workunit.client.1.smithi135.stdout:4/162: dread f4 [0,4194304] 0 2024-03-30T04:28:14.761 INFO:tasks.workunit.client.0.smithi123.stdout:3/625: dread - d13/d24/d39/d79/d90/fc2 zero size 2024-03-30T04:28:14.763 INFO:tasks.workunit.client.0.smithi123.stdout:6/679: write d10/d82/d8e/dc7/d42/f80 [3636192,29266] 0 2024-03-30T04:28:14.764 INFO:tasks.workunit.client.1.smithi135.stdout:4/163: chown fd 20 1 2024-03-30T04:28:14.764 INFO:tasks.workunit.client.1.smithi135.stdout:0/242: dwrite d0/d1/f44 [0,4194304] 0 2024-03-30T04:28:14.767 INFO:tasks.workunit.client.0.smithi123.stdout:2/672: dwrite d0/d1f/d1c/d20/f9f [0,4194304] 0 2024-03-30T04:28:14.767 INFO:tasks.workunit.client.1.smithi135.stdout:3/183: dwrite d3/f24 [4194304,4194304] 0 2024-03-30T04:28:14.776 INFO:tasks.workunit.client.0.smithi123.stdout:8/567: truncate da/f11 9230097 0 2024-03-30T04:28:14.781 INFO:tasks.workunit.client.1.smithi135.stdout:6/149: write d2/fa [573752,45046] 0 2024-03-30T04:28:14.783 INFO:tasks.workunit.client.0.smithi123.stdout:8/568: stat da/dc/d14/f36 0 2024-03-30T04:28:14.784 INFO:tasks.workunit.client.0.smithi123.stdout:2/673: write d0/d1f/d1c/d40/fac [658164,44561] 0 2024-03-30T04:28:14.786 INFO:tasks.workunit.client.1.smithi135.stdout:6/150: read d2/f9 [1131481,95051] 0 2024-03-30T04:28:14.788 INFO:tasks.workunit.client.0.smithi123.stdout:8/569: stat da/d27/d3c/d2d/f4c 0 2024-03-30T04:28:14.790 INFO:tasks.workunit.client.0.smithi123.stdout:1/680: sync 2024-03-30T04:28:14.790 INFO:tasks.workunit.client.0.smithi123.stdout:0/610: sync 2024-03-30T04:28:14.795 INFO:tasks.workunit.client.0.smithi123.stdout:9/542: dwrite d6/f26 [0,4194304] 0 2024-03-30T04:28:14.795 INFO:tasks.workunit.client.1.smithi135.stdout:6/151: rename d2/c1c to d2/dd/c2a 0 2024-03-30T04:28:14.799 INFO:tasks.workunit.client.1.smithi135.stdout:4/164: dread d10/f34 [0,4194304] 0 2024-03-30T04:28:14.801 INFO:tasks.workunit.client.1.smithi135.stdout:3/184: dwrite d3/d6/f12 [0,4194304] 0 2024-03-30T04:28:14.801 INFO:tasks.workunit.client.0.smithi123.stdout:0/611: dread - d6/dd/d15/d28/f97 zero size 2024-03-30T04:28:14.803 INFO:tasks.workunit.client.0.smithi123.stdout:6/680: mknod d10/d82/dc1/dd2/dda/ce9 0 2024-03-30T04:28:14.804 INFO:tasks.workunit.client.1.smithi135.stdout:4/165: mkdir d10/d1a/d2b/d39 0 2024-03-30T04:28:14.805 INFO:tasks.workunit.client.1.smithi135.stdout:5/230: dread d1/f36 [0,4194304] 0 2024-03-30T04:28:14.806 INFO:tasks.workunit.client.0.smithi123.stdout:2/674: creat d0/d1f/d16/fdc x:0 0 0 2024-03-30T04:28:14.806 INFO:tasks.workunit.client.1.smithi135.stdout:0/243: write d0/f12 [210344,67635] 0 2024-03-30T04:28:14.808 INFO:tasks.workunit.client.0.smithi123.stdout:7/607: truncate d2/da/d11/d34/d76/d91/f9a 1252039 0 2024-03-30T04:28:14.812 INFO:tasks.workunit.client.1.smithi135.stdout:0/244: fdatasync d0/d5/d10/f2b 0 2024-03-30T04:28:14.815 INFO:tasks.workunit.client.1.smithi135.stdout:0/245: chown d0/d1/l3c 46 1 2024-03-30T04:28:14.816 INFO:tasks.workunit.client.0.smithi123.stdout:3/626: link d13/d24/d31/d23/d3a/d3c/fa8 d13/d24/d31/d23/fc8 0 2024-03-30T04:28:14.821 INFO:tasks.workunit.client.0.smithi123.stdout:4/591: rmdir d2/d4/d7/d1a/dcf 0 2024-03-30T04:28:14.822 INFO:tasks.workunit.client.0.smithi123.stdout:4/592: dread - d2/d4/fca zero size 2024-03-30T04:28:14.825 INFO:tasks.workunit.client.0.smithi123.stdout:1/681: mkdir db/dd/d42/d46/d56/daf/de4 0 2024-03-30T04:28:14.826 INFO:tasks.workunit.client.1.smithi135.stdout:3/185: link d3/cc d3/d17/c39 0 2024-03-30T04:28:14.829 INFO:tasks.workunit.client.1.smithi135.stdout:9/170: dwrite d7/f12 [0,4194304] 0 2024-03-30T04:28:14.830 INFO:tasks.workunit.client.0.smithi123.stdout:6/681: creat d10/d20/fea x:0 0 0 2024-03-30T04:28:14.830 INFO:tasks.workunit.client.1.smithi135.stdout:0/246: read d0/d5/d9/f33 [8527181,11087] 0 2024-03-30T04:28:14.831 INFO:tasks.workunit.client.0.smithi123.stdout:6/682: truncate f6 1618054 0 2024-03-30T04:28:14.833 INFO:tasks.workunit.client.0.smithi123.stdout:8/570: symlink da/d27/d56/d6c/d96/la8 0 2024-03-30T04:28:14.835 INFO:tasks.workunit.client.1.smithi135.stdout:3/186: chown d3/d10/d16 42729722 1 2024-03-30T04:28:14.836 INFO:tasks.workunit.client.0.smithi123.stdout:3/627: stat d13/d24/d31/d23/d3a/d3c/c43 0 2024-03-30T04:28:14.840 INFO:tasks.workunit.client.1.smithi135.stdout:0/247: write d0/f12 [1660966,4673] 0 2024-03-30T04:28:14.841 INFO:tasks.workunit.client.0.smithi123.stdout:7/608: dwrite d2/dc/d49/d68/f9d [0,4194304] 0 2024-03-30T04:28:14.841 INFO:tasks.workunit.client.1.smithi135.stdout:9/171: symlink d7/l32 0 2024-03-30T04:28:14.847 INFO:tasks.workunit.client.1.smithi135.stdout:9/172: symlink d7/d1b/l33 0 2024-03-30T04:28:14.848 INFO:tasks.workunit.client.0.smithi123.stdout:5/550: dread dd/d19/d1f/d35/d3a/d57/f65 [0,4194304] 0 2024-03-30T04:28:14.850 INFO:tasks.workunit.client.1.smithi135.stdout:5/231: dwrite d1/f3e [0,4194304] 0 2024-03-30T04:28:14.850 INFO:tasks.workunit.client.1.smithi135.stdout:4/166: dwrite fd [0,4194304] 0 2024-03-30T04:28:14.854 INFO:tasks.workunit.client.0.smithi123.stdout:3/628: symlink d13/d24/d39/d79/d90/lc9 0 2024-03-30T04:28:14.854 INFO:tasks.workunit.client.1.smithi135.stdout:0/248: write d0/d5/d10/f11 [8405126,68940] 0 2024-03-30T04:28:14.855 INFO:tasks.workunit.client.1.smithi135.stdout:4/167: chown f4 2057 1 2024-03-30T04:28:14.855 INFO:tasks.workunit.client.1.smithi135.stdout:1/160: sync 2024-03-30T04:28:14.855 INFO:tasks.workunit.client.1.smithi135.stdout:5/232: stat d1/da/d17/c1c 0 2024-03-30T04:28:14.859 INFO:tasks.workunit.client.0.smithi123.stdout:7/609: mkdir d2/d4d/dd0 0 2024-03-30T04:28:14.861 INFO:tasks.workunit.client.0.smithi123.stdout:2/675: creat d0/d1f/fdd x:0 0 0 2024-03-30T04:28:14.862 INFO:tasks.workunit.client.0.smithi123.stdout:6/683: symlink d10/d82/leb 0 2024-03-30T04:28:14.863 INFO:tasks.workunit.client.1.smithi135.stdout:5/233: write d1/f37 [789458,23180] 0 2024-03-30T04:28:14.867 INFO:tasks.workunit.client.0.smithi123.stdout:3/629: creat d13/d24/d31/d23/d3a/d3c/db8/fca x:0 0 0 2024-03-30T04:28:14.868 INFO:tasks.workunit.client.1.smithi135.stdout:3/187: creat d3/d6/d22/d33/f3a x:0 0 0 2024-03-30T04:28:14.870 INFO:tasks.workunit.client.0.smithi123.stdout:6/684: chown d10/d82/dc1/d45/c81 541 1 2024-03-30T04:28:14.871 INFO:tasks.workunit.client.0.smithi123.stdout:0/612: getdents d6 0 2024-03-30T04:28:14.874 INFO:tasks.workunit.client.0.smithi123.stdout:5/551: creat dd/d19/d1f/d35/faf x:0 0 0 2024-03-30T04:28:14.874 INFO:tasks.workunit.client.0.smithi123.stdout:6/685: dread - d10/d20/da2/fb1 zero size 2024-03-30T04:28:14.878 INFO:tasks.workunit.client.1.smithi135.stdout:6/152: write d2/dd/f1a [4822601,34134] 0 2024-03-30T04:28:14.878 INFO:tasks.workunit.client.0.smithi123.stdout:2/676: unlink d0/d1f/d1c/d40/l53 0 2024-03-30T04:28:14.878 INFO:tasks.workunit.client.0.smithi123.stdout:6/686: truncate d10/d20/f2b 1341218 0 2024-03-30T04:28:14.881 INFO:tasks.workunit.client.1.smithi135.stdout:1/161: mknod d16/d19/d20/c39 0 2024-03-30T04:28:14.882 INFO:tasks.workunit.client.0.smithi123.stdout:0/613: symlink d6/dd/d31/d35/d3a/db9/lc7 0 2024-03-30T04:28:14.886 INFO:tasks.workunit.client.1.smithi135.stdout:1/162: write fb [4185420,7585] 0 2024-03-30T04:28:14.887 INFO:tasks.workunit.client.1.smithi135.stdout:0/249: dwrite d0/d5/d9/f2e [0,4194304] 0 2024-03-30T04:28:14.887 INFO:tasks.workunit.client.0.smithi123.stdout:5/552: creat dd/d19/d1f/d35/d3a/d79/fb0 x:0 0 0 2024-03-30T04:28:14.887 INFO:tasks.workunit.client.0.smithi123.stdout:3/630: link c5 d13/d24/d31/d23/ccb 0 2024-03-30T04:28:14.887 INFO:tasks.workunit.client.0.smithi123.stdout:2/677: mkdir d0/d1f/d27/d68/dd6/dde 0 2024-03-30T04:28:14.890 INFO:tasks.workunit.client.1.smithi135.stdout:0/250: fsync d0/d5/d10/f11 0 2024-03-30T04:28:14.892 INFO:tasks.workunit.client.0.smithi123.stdout:0/614: creat d6/dd/d15/fc8 x:0 0 0 2024-03-30T04:28:14.892 INFO:tasks.workunit.client.0.smithi123.stdout:3/631: chown d13/d24/d31/d23/d63/f95 16 1 2024-03-30T04:28:14.894 INFO:tasks.workunit.client.1.smithi135.stdout:3/188: fdatasync f2 0 2024-03-30T04:28:14.894 INFO:tasks.workunit.client.0.smithi123.stdout:5/553: dread dd/d41/f43 [0,4194304] 0 2024-03-30T04:28:14.896 INFO:tasks.workunit.client.0.smithi123.stdout:1/682: dread db/dd/f54 [0,4194304] 0 2024-03-30T04:28:14.897 INFO:tasks.workunit.client.1.smithi135.stdout:6/153: dread d2/f4 [0,4194304] 0 2024-03-30T04:28:14.898 INFO:tasks.workunit.client.1.smithi135.stdout:3/189: readlink d3/db/l28 0 2024-03-30T04:28:14.899 INFO:tasks.workunit.client.1.smithi135.stdout:5/234: getdents d1/d7/d1a/d46 0 2024-03-30T04:28:14.899 INFO:tasks.workunit.client.0.smithi123.stdout:0/615: unlink d6/lc1 0 2024-03-30T04:28:14.902 INFO:tasks.workunit.client.0.smithi123.stdout:8/571: dread f6 [0,4194304] 0 2024-03-30T04:28:14.902 INFO:tasks.workunit.client.1.smithi135.stdout:0/251: creat d0/d13/d1d/f4a x:0 0 0 2024-03-30T04:28:14.904 INFO:tasks.workunit.client.1.smithi135.stdout:6/154: mkdir d2/dd/d14/d2b 0 2024-03-30T04:28:14.907 INFO:tasks.workunit.client.0.smithi123.stdout:1/683: write db/dd/d42/d46/d51/d72/fc8 [361068,117628] 0 2024-03-30T04:28:14.908 INFO:tasks.workunit.client.1.smithi135.stdout:1/163: dwrite d16/d1a/f23 [0,4194304] 0 2024-03-30T04:28:14.913 INFO:tasks.workunit.client.0.smithi123.stdout:4/593: truncate d2/f1c 3099450 0 2024-03-30T04:28:14.913 INFO:tasks.workunit.client.1.smithi135.stdout:3/190: mknod d3/d10/d16/d23/c3b 0 2024-03-30T04:28:14.914 INFO:tasks.workunit.client.1.smithi135.stdout:5/235: mknod d1/da/d17/c47 0 2024-03-30T04:28:14.916 INFO:tasks.workunit.client.0.smithi123.stdout:9/543: dwrite d6/d1f/d45/f50 [0,4194304] 0 2024-03-30T04:28:14.917 INFO:tasks.workunit.client.0.smithi123.stdout:4/594: write d2/d4/d7/f6c [1140770,95480] 0 2024-03-30T04:28:14.917 INFO:tasks.workunit.client.1.smithi135.stdout:4/168: getdents d10 0 2024-03-30T04:28:14.919 INFO:tasks.workunit.client.1.smithi135.stdout:9/173: truncate d7/d1b/f1c 2728157 0 2024-03-30T04:28:14.922 INFO:tasks.workunit.client.1.smithi135.stdout:4/169: write d10/d1a/f21 [375974,100552] 0 2024-03-30T04:28:14.922 INFO:tasks.workunit.client.1.smithi135.stdout:1/164: rmdir d16/d19/d20/d26/d33 39 2024-03-30T04:28:14.924 INFO:tasks.workunit.client.0.smithi123.stdout:8/572: rmdir da/d27/d56/d93 39 2024-03-30T04:28:14.925 INFO:tasks.workunit.client.0.smithi123.stdout:7/610: dread d2/da/d11/f19 [0,4194304] 0 2024-03-30T04:28:14.928 INFO:tasks.workunit.client.1.smithi135.stdout:4/170: chown d10/c33 2261 1 2024-03-30T04:28:14.929 INFO:tasks.workunit.client.0.smithi123.stdout:3/632: rename d13/f7c to d13/d24/fcc 0 2024-03-30T04:28:14.932 INFO:tasks.workunit.client.1.smithi135.stdout:5/236: creat d1/d7/f48 x:0 0 0 2024-03-30T04:28:14.934 INFO:tasks.workunit.client.1.smithi135.stdout:1/165: mkdir d16/d29/d2c/d30/d3a 0 2024-03-30T04:28:14.939 INFO:tasks.workunit.client.1.smithi135.stdout:0/252: dwrite d0/d5/d10/f2b [0,4194304] 0 2024-03-30T04:28:14.939 INFO:tasks.workunit.client.1.smithi135.stdout:9/174: link d7/f22 d7/d1b/f34 0 2024-03-30T04:28:14.940 INFO:tasks.workunit.client.1.smithi135.stdout:5/237: fdatasync d1/d7/d1a/d1f/f40 0 2024-03-30T04:28:14.940 INFO:tasks.workunit.client.1.smithi135.stdout:4/171: creat d10/d1a/f3a x:0 0 0 2024-03-30T04:28:14.942 INFO:tasks.workunit.client.0.smithi123.stdout:3/633: chown d13/d24/d31/d30/c58 4840 1 2024-03-30T04:28:14.943 INFO:tasks.workunit.client.0.smithi123.stdout:3/634: chown d13/d24 119 1 2024-03-30T04:28:14.943 INFO:tasks.workunit.client.0.smithi123.stdout:4/595: dread d2/d4/d2c/d42/d35/d5d/fae [0,4194304] 0 2024-03-30T04:28:14.944 INFO:tasks.workunit.client.1.smithi135.stdout:1/166: fdatasync d16/f18 0 2024-03-30T04:28:14.946 INFO:tasks.workunit.client.1.smithi135.stdout:9/175: creat d7/d1b/f35 x:0 0 0 2024-03-30T04:28:14.946 INFO:tasks.workunit.client.0.smithi123.stdout:0/616: getdents d6/dd/d15/d28/d4b 0 2024-03-30T04:28:14.950 INFO:tasks.workunit.client.0.smithi123.stdout:2/678: truncate d0/d1f/d1c/d39/d4b/fb1 2799048 0 2024-03-30T04:28:14.951 INFO:tasks.workunit.client.1.smithi135.stdout:9/176: write f6 [406661,127450] 0 2024-03-30T04:28:14.951 INFO:tasks.workunit.client.1.smithi135.stdout:4/172: symlink d10/d1a/d32/l3b 0 2024-03-30T04:28:14.951 INFO:tasks.workunit.client.1.smithi135.stdout:5/238: readlink d1/d7/d1a/d1f/l2e 0 2024-03-30T04:28:14.954 INFO:tasks.workunit.client.1.smithi135.stdout:1/167: fsync d16/d29/f35 0 2024-03-30T04:28:14.954 INFO:tasks.workunit.client.0.smithi123.stdout:2/679: write d0/d1f/d1c/fb3 [455032,75682] 0 2024-03-30T04:28:14.955 INFO:tasks.workunit.client.0.smithi123.stdout:9/544: dread d6/d49/f65 [0,4194304] 0 2024-03-30T04:28:14.956 INFO:tasks.workunit.client.0.smithi123.stdout:4/596: chown d2/d4/d2c/l59 0 1 2024-03-30T04:28:14.958 INFO:tasks.workunit.client.0.smithi123.stdout:3/635: rmdir d13/d24/d31/d23/d3a/d3c/d4e/d5d 39 2024-03-30T04:28:14.959 INFO:tasks.workunit.client.1.smithi135.stdout:0/253: write d0/d5/d10/f2c [725407,84718] 0 2024-03-30T04:28:14.960 INFO:tasks.workunit.client.0.smithi123.stdout:7/611: creat d2/dc/d83/fd1 x:0 0 0 2024-03-30T04:28:14.960 INFO:tasks.workunit.client.1.smithi135.stdout:3/191: dwrite d3/db/f34 [0,4194304] 0 2024-03-30T04:28:14.962 INFO:tasks.workunit.client.0.smithi123.stdout:9/545: chown d6/d1f/d45/d4c/lb2 65169177 1 2024-03-30T04:28:14.963 INFO:tasks.workunit.client.1.smithi135.stdout:5/239: read - d1/d7/d1a/f3b zero size 2024-03-30T04:28:14.963 INFO:tasks.workunit.client.1.smithi135.stdout:0/254: write d0/f12 [1613935,11477] 0 2024-03-30T04:28:14.964 INFO:tasks.workunit.client.0.smithi123.stdout:2/680: mkdir d0/d1f/d82/ddf 0 2024-03-30T04:28:14.965 INFO:tasks.workunit.client.1.smithi135.stdout:6/155: dwrite d2/f9 [0,4194304] 0 2024-03-30T04:28:14.970 INFO:tasks.workunit.client.1.smithi135.stdout:5/240: readlink d1/d7/l31 0 2024-03-30T04:28:14.970 INFO:tasks.workunit.client.1.smithi135.stdout:4/173: rename d10/d1a/c28 to d10/d1a/d32/c3c 0 2024-03-30T04:28:14.970 INFO:tasks.workunit.client.1.smithi135.stdout:8/218: sync 2024-03-30T04:28:14.971 INFO:tasks.workunit.client.0.smithi123.stdout:0/617: getdents d6/dd/d31/d35/d3a 0 2024-03-30T04:28:14.972 INFO:tasks.workunit.client.1.smithi135.stdout:0/255: creat d0/d1/d16/f4b x:0 0 0 2024-03-30T04:28:14.973 INFO:tasks.workunit.client.0.smithi123.stdout:5/554: dwrite dd/d19/d1f/d35/f40 [0,4194304] 0 2024-03-30T04:28:14.973 INFO:tasks.workunit.client.1.smithi135.stdout:5/241: write d1/d7/f2c [523065,113423] 0 2024-03-30T04:28:14.973 INFO:tasks.workunit.client.1.smithi135.stdout:8/219: write f5 [243008,84461] 0 2024-03-30T04:28:14.976 INFO:tasks.workunit.client.0.smithi123.stdout:0/618: fsync d6/dd/d31/fa9 0 2024-03-30T04:28:14.979 INFO:tasks.workunit.client.1.smithi135.stdout:6/156: mknod d2/dd/d18/c2c 0 2024-03-30T04:28:14.979 INFO:tasks.workunit.client.0.smithi123.stdout:1/684: dread db/fe [0,4194304] 0 2024-03-30T04:28:14.980 INFO:tasks.workunit.client.0.smithi123.stdout:3/636: getdents d13/d24/d31/d23/d3a/d3c/dab 0 2024-03-30T04:28:14.981 INFO:tasks.workunit.client.1.smithi135.stdout:1/168: chown d16/d19/d20/d26/d33/l34 251973043 1 2024-03-30T04:28:14.981 INFO:tasks.workunit.client.1.smithi135.stdout:9/177: link d7/d29/c2d d7/d29/c36 0 2024-03-30T04:28:14.984 INFO:tasks.workunit.client.1.smithi135.stdout:6/157: read d2/f11 [4402801,118833] 0 2024-03-30T04:28:14.985 INFO:tasks.workunit.client.0.smithi123.stdout:0/619: stat d6/dd/d15/c9d 0 2024-03-30T04:28:14.986 INFO:tasks.workunit.client.1.smithi135.stdout:1/169: dread - d16/d1a/f22 zero size 2024-03-30T04:28:14.986 INFO:tasks.workunit.client.1.smithi135.stdout:0/256: unlink d0/d1/c39 0 2024-03-30T04:28:14.991 INFO:tasks.workunit.client.0.smithi123.stdout:3/637: unlink d13/d24/fcc 0 2024-03-30T04:28:14.993 INFO:tasks.workunit.client.0.smithi123.stdout:1/685: creat db/dd/d5f/d6d/d85/fe5 x:0 0 0 2024-03-30T04:28:14.997 INFO:tasks.workunit.client.1.smithi135.stdout:0/257: mknod d0/d13/d1e/c4c 0 2024-03-30T04:28:15.000 INFO:tasks.workunit.client.0.smithi123.stdout:4/597: write d2/d4/ff [7743893,67249] 0 2024-03-30T04:28:15.000 INFO:tasks.workunit.client.0.smithi123.stdout:3/638: read f11 [2587350,88766] 0 2024-03-30T04:28:15.000 INFO:tasks.workunit.client.1.smithi135.stdout:6/158: mknod d2/c2d 0 2024-03-30T04:28:15.001 INFO:tasks.workunit.client.0.smithi123.stdout:0/620: creat d6/dd/d15/d98/d4e/d9a/db2/fc9 x:0 0 0 2024-03-30T04:28:15.002 INFO:tasks.workunit.client.0.smithi123.stdout:1/686: readlink db/dd/d42/d46/l6a 0 2024-03-30T04:28:15.002 INFO:tasks.workunit.client.1.smithi135.stdout:9/178: dread f6 [0,4194304] 0 2024-03-30T04:28:15.004 INFO:tasks.workunit.client.1.smithi135.stdout:5/242: link d1/d7/f48 d1/d3/f49 0 2024-03-30T04:28:15.007 INFO:tasks.workunit.client.1.smithi135.stdout:9/179: chown d7 25834928 1 2024-03-30T04:28:15.008 INFO:tasks.workunit.client.1.smithi135.stdout:3/192: dwrite d3/d10/f31 [0,4194304] 0 2024-03-30T04:28:15.008 INFO:tasks.workunit.client.0.smithi123.stdout:5/555: dwrite dd/f32 [0,4194304] 0 2024-03-30T04:28:15.008 INFO:tasks.workunit.client.0.smithi123.stdout:1/687: rmdir db/dd/d42/d46/d39 39 2024-03-30T04:28:15.009 INFO:tasks.workunit.client.1.smithi135.stdout:6/159: mkdir d2/d2e 0 2024-03-30T04:28:15.011 INFO:tasks.workunit.client.1.smithi135.stdout:1/170: dread d16/f1c [0,4194304] 0 2024-03-30T04:28:15.013 INFO:tasks.workunit.client.0.smithi123.stdout:0/621: creat d6/dd/d15/d98/d5d/d7f/fca x:0 0 0 2024-03-30T04:28:15.014 INFO:tasks.workunit.client.0.smithi123.stdout:4/598: symlink d2/d4/d2c/d42/d35/d9b/db7/ld1 0 2024-03-30T04:28:15.017 INFO:tasks.workunit.client.1.smithi135.stdout:4/174: dwrite f3 [0,4194304] 0 2024-03-30T04:28:15.020 INFO:tasks.workunit.client.1.smithi135.stdout:7/272: sync 2024-03-30T04:28:15.020 INFO:tasks.workunit.client.1.smithi135.stdout:2/193: sync 2024-03-30T04:28:15.021 INFO:tasks.workunit.client.0.smithi123.stdout:0/622: truncate d6/d52/f73 321665 0 2024-03-30T04:28:15.022 INFO:tasks.workunit.client.1.smithi135.stdout:9/180: mkdir d7/d1b/d37 0 2024-03-30T04:28:15.024 INFO:tasks.workunit.client.0.smithi123.stdout:7/612: dwrite d2/dc/d49/f6c [0,4194304] 0 2024-03-30T04:28:15.024 INFO:tasks.workunit.client.0.smithi123.stdout:3/639: mkdir d13/d24/d31/d23/d3a/d3c/d4e/d5d/da5/dcd 0 2024-03-30T04:28:15.030 INFO:tasks.workunit.client.0.smithi123.stdout:6/687: sync 2024-03-30T04:28:15.030 INFO:tasks.workunit.client.0.smithi123.stdout:8/573: sync 2024-03-30T04:28:15.031 INFO:tasks.workunit.client.0.smithi123.stdout:4/599: mkdir d2/d34/dd2 0 2024-03-30T04:28:15.031 INFO:tasks.workunit.client.1.smithi135.stdout:8/220: dwrite d6/d8/d14/f21 [0,4194304] 0 2024-03-30T04:28:15.031 INFO:tasks.workunit.client.1.smithi135.stdout:4/175: mkdir d10/d1a/d3d 0 2024-03-30T04:28:15.032 INFO:tasks.workunit.client.0.smithi123.stdout:7/613: read - d2/dc/d49/d68/f9b zero size 2024-03-30T04:28:15.032 INFO:tasks.workunit.client.0.smithi123.stdout:2/681: dwrite d0/d1f/d1c/d39/d4b/d4c/f64 [4194304,4194304] 0 2024-03-30T04:28:15.036 INFO:tasks.workunit.client.1.smithi135.stdout:0/258: rmdir d0/d13/d1e 39 2024-03-30T04:28:15.037 INFO:tasks.workunit.client.0.smithi123.stdout:1/688: creat db/dd/d5f/d6d/d85/dd4/fe6 x:0 0 0 2024-03-30T04:28:15.037 INFO:tasks.workunit.client.0.smithi123.stdout:9/546: dread d6/d1f/d45/d97/d69/fab [0,4194304] 0 2024-03-30T04:28:15.038 INFO:tasks.workunit.client.0.smithi123.stdout:4/600: stat d2/d4/d2c/l59 0 2024-03-30T04:28:15.039 INFO:tasks.workunit.client.1.smithi135.stdout:8/221: read - d6/d8/f3b zero size 2024-03-30T04:28:15.039 INFO:tasks.workunit.client.0.smithi123.stdout:0/623: unlink d6/l75 0 2024-03-30T04:28:15.041 INFO:tasks.workunit.client.0.smithi123.stdout:8/574: mkdir da/dc/da9 0 2024-03-30T04:28:15.041 INFO:tasks.workunit.client.0.smithi123.stdout:6/688: dread - d10/d82/d8e/dc7/fd7 zero size 2024-03-30T04:28:15.044 INFO:tasks.workunit.client.0.smithi123.stdout:3/640: read d13/d24/d31/d23/d3a/d3c/d4e/d5d/f8e [146885,69305] 0 2024-03-30T04:28:15.046 INFO:tasks.workunit.client.0.smithi123.stdout:9/547: chown d6/fd 2 1 2024-03-30T04:28:15.046 INFO:tasks.workunit.client.0.smithi123.stdout:6/689: fsync d10/d82/d8e/dc7/d42/f47 0 2024-03-30T04:28:15.046 INFO:tasks.workunit.client.1.smithi135.stdout:6/160: symlink d2/d2e/l2f 0 2024-03-30T04:28:15.048 INFO:tasks.workunit.client.0.smithi123.stdout:2/682: chown d0/d1f/d1c/d80/dbb/dd5/f9c 3500 1 2024-03-30T04:28:15.049 INFO:tasks.workunit.client.1.smithi135.stdout:1/171: rename d16/d1a/f22 to d16/d19/d20/d26/d33/f3b 0 2024-03-30T04:28:15.050 INFO:tasks.workunit.client.1.smithi135.stdout:6/161: stat d2/dd 0 2024-03-30T04:28:15.050 INFO:tasks.workunit.client.1.smithi135.stdout:9/181: dread d7/f26 [0,4194304] 0 2024-03-30T04:28:15.051 INFO:tasks.workunit.client.0.smithi123.stdout:2/683: dread - d0/d1f/d1c/d20/f56 zero size 2024-03-30T04:28:15.053 INFO:tasks.workunit.client.0.smithi123.stdout:1/689: rename db/dd/l3b to db/dd/d42/d46/d56/d5c/dd0/le7 0 2024-03-30T04:28:15.054 INFO:tasks.workunit.client.1.smithi135.stdout:0/259: truncate d0/d13/d1d/f1f 915324 0 2024-03-30T04:28:15.054 INFO:tasks.workunit.client.1.smithi135.stdout:2/194: dread d4/d32/f26 [0,4194304] 0 2024-03-30T04:28:15.054 INFO:tasks.workunit.client.0.smithi123.stdout:0/624: mknod d6/dd/d15/d98/d4e/d9a/db2/ccb 0 2024-03-30T04:28:15.058 INFO:tasks.workunit.client.0.smithi123.stdout:0/625: chown d6/dd/d15/d28/d4b/c54 8929901 1 2024-03-30T04:28:15.060 INFO:tasks.workunit.client.1.smithi135.stdout:3/193: link d3/d6/f36 d3/f3c 0 2024-03-30T04:28:15.060 INFO:tasks.workunit.client.1.smithi135.stdout:7/273: mkdir d4/db/de/d5a 0 2024-03-30T04:28:15.061 INFO:tasks.workunit.client.0.smithi123.stdout:0/626: readlink d6/dd/d15/d98/la4 0 2024-03-30T04:28:15.062 INFO:tasks.workunit.client.0.smithi123.stdout:5/556: write dd/d19/d1f/d35/d3a/d79/f9b [289947,31374] 0 2024-03-30T04:28:15.063 INFO:tasks.workunit.client.1.smithi135.stdout:5/243: link d1/d3/l29 d1/da/l4a 0 2024-03-30T04:28:15.065 INFO:tasks.workunit.client.0.smithi123.stdout:1/690: symlink db/dd/d42/le8 0 2024-03-30T04:28:15.068 INFO:tasks.workunit.client.0.smithi123.stdout:7/614: getdents d2/dc 0 2024-03-30T04:28:15.071 INFO:tasks.workunit.client.0.smithi123.stdout:6/690: truncate d10/f1d 3092572 0 2024-03-30T04:28:15.072 INFO:tasks.workunit.client.0.smithi123.stdout:4/601: fsync d2/d4/d7/d1a/f5e 0 2024-03-30T04:28:15.072 INFO:tasks.workunit.client.0.smithi123.stdout:8/575: symlink da/dc/d14/d9d/laa 0 2024-03-30T04:28:15.073 INFO:tasks.workunit.client.0.smithi123.stdout:5/557: truncate dd/d19/f28 1469734 0 2024-03-30T04:28:15.075 INFO:tasks.workunit.client.1.smithi135.stdout:4/176: dwrite d10/f14 [0,4194304] 0 2024-03-30T04:28:15.075 INFO:tasks.workunit.client.1.smithi135.stdout:2/195: unlink d4/f7 0 2024-03-30T04:28:15.075 INFO:tasks.workunit.client.0.smithi123.stdout:2/684: link d0/d1f/d1c/d36/f89 d0/d1f/d27/d68/dd6/fe0 0 2024-03-30T04:28:15.081 INFO:tasks.workunit.client.0.smithi123.stdout:6/691: read d10/d82/dc1/d6a/d7a/d9f/fc6 [675305,54855] 0 2024-03-30T04:28:15.083 INFO:tasks.workunit.client.0.smithi123.stdout:2/685: read - d0/d1f/d1c/d80/dbb/d97/fd8 zero size 2024-03-30T04:28:15.084 INFO:tasks.workunit.client.1.smithi135.stdout:5/244: mkdir d1/d7/d11/d3c/d4b 0 2024-03-30T04:28:15.085 INFO:tasks.workunit.client.0.smithi123.stdout:9/548: write d6/d49/f73 [471090,121414] 0 2024-03-30T04:28:15.088 INFO:tasks.workunit.client.0.smithi123.stdout:8/576: creat da/d27/d95/fab x:0 0 0 2024-03-30T04:28:15.088 INFO:tasks.workunit.client.0.smithi123.stdout:3/641: write d13/d24/d31/d23/d3a/d3c/f73 [121995,86907] 0 2024-03-30T04:28:15.089 INFO:tasks.workunit.client.1.smithi135.stdout:5/245: chown d1/d7/d1a/d1f/f40 1 1 2024-03-30T04:28:15.090 INFO:tasks.workunit.client.1.smithi135.stdout:3/194: dread d3/f9 [0,4194304] 0 2024-03-30T04:28:15.094 INFO:tasks.workunit.client.0.smithi123.stdout:7/615: dread d2/dc/d49/f63 [0,4194304] 0 2024-03-30T04:28:15.094 INFO:tasks.workunit.client.0.smithi123.stdout:8/577: chown da/dc/l19 27 1 2024-03-30T04:28:15.094 INFO:tasks.workunit.client.0.smithi123.stdout:2/686: dread - d0/d1f/d1c/d80/dbb/d97/db5/fd2 zero size 2024-03-30T04:28:15.095 INFO:tasks.workunit.client.0.smithi123.stdout:8/578: readlink da/dc/l34 0 2024-03-30T04:28:15.095 INFO:tasks.workunit.client.0.smithi123.stdout:9/549: creat d6/d1f/d72/d7d/fbe x:0 0 0 2024-03-30T04:28:15.098 INFO:tasks.workunit.client.1.smithi135.stdout:9/182: dread d7/ff [0,4194304] 0 2024-03-30T04:28:15.099 INFO:tasks.workunit.client.0.smithi123.stdout:7/616: chown d2/da/d11/d61 56022 1 2024-03-30T04:28:15.101 INFO:tasks.workunit.client.1.smithi135.stdout:6/162: dread d2/dd/d18/f22 [0,4194304] 0 2024-03-30T04:28:15.101 INFO:tasks.workunit.client.1.smithi135.stdout:4/177: creat d10/d1a/d32/f3e x:0 0 0 2024-03-30T04:28:15.102 INFO:tasks.workunit.client.1.smithi135.stdout:5/246: symlink d1/da/d17/l4c 0 2024-03-30T04:28:15.106 INFO:tasks.workunit.client.1.smithi135.stdout:3/195: unlink d3/d10/f18 0 2024-03-30T04:28:15.109 INFO:tasks.workunit.client.0.smithi123.stdout:2/687: link d0/d1f/d1c/d80/da8/cc3 d0/d1f/d1c/d39/ce1 0 2024-03-30T04:28:15.110 INFO:tasks.workunit.client.0.smithi123.stdout:0/627: dread d6/f72 [0,4194304] 0 2024-03-30T04:28:15.113 INFO:tasks.workunit.client.1.smithi135.stdout:8/222: dread d6/d8/d14/fe [0,4194304] 0 2024-03-30T04:28:15.115 INFO:tasks.workunit.client.1.smithi135.stdout:8/223: readlink l3 0 2024-03-30T04:28:15.116 INFO:tasks.workunit.client.1.smithi135.stdout:6/163: mkdir d2/dd/d14/d30 0 2024-03-30T04:28:15.117 INFO:tasks.workunit.client.1.smithi135.stdout:0/260: dread d0/d1/fa [0,4194304] 0 2024-03-30T04:28:15.120 INFO:tasks.workunit.client.1.smithi135.stdout:3/196: creat d3/d6/d22/f3d x:0 0 0 2024-03-30T04:28:15.121 INFO:tasks.workunit.client.0.smithi123.stdout:2/688: getdents d0/d1f/d1c/d80/dbb/dd5 0 2024-03-30T04:28:15.122 INFO:tasks.workunit.client.1.smithi135.stdout:8/224: dread - d6/d8/f38 zero size 2024-03-30T04:28:15.123 INFO:tasks.workunit.client.0.smithi123.stdout:0/628: truncate d6/dd/d15/d28/f97 377275 0 2024-03-30T04:28:15.125 INFO:tasks.workunit.client.0.smithi123.stdout:7/617: dwrite d2/da/d11/d34/d85/fa9 [0,4194304] 0 2024-03-30T04:28:15.127 INFO:tasks.workunit.client.0.smithi123.stdout:9/550: dwrite d6/d1f/d45/f90 [0,4194304] 0 2024-03-30T04:28:15.127 INFO:tasks.workunit.client.0.smithi123.stdout:4/602: write d2/d4/d2c/d42/d35/d5d/d78/f85 [837432,72593] 0 2024-03-30T04:28:15.129 INFO:tasks.workunit.client.1.smithi135.stdout:4/178: getdents d10/d38 0 2024-03-30T04:28:15.129 INFO:tasks.workunit.client.1.smithi135.stdout:9/183: creat d7/d1b/d37/f38 x:0 0 0 2024-03-30T04:28:15.130 INFO:tasks.workunit.client.1.smithi135.stdout:2/196: getdents d4/d32/d2e/d2f 0 2024-03-30T04:28:15.131 INFO:tasks.workunit.client.0.smithi123.stdout:2/689: symlink d0/d1f/le2 0 2024-03-30T04:28:15.132 INFO:tasks.workunit.client.0.smithi123.stdout:1/691: dread db/f24 [0,4194304] 0 2024-03-30T04:28:15.134 INFO:tasks.workunit.client.0.smithi123.stdout:3/642: truncate fd 388006 0 2024-03-30T04:28:15.136 INFO:tasks.workunit.client.0.smithi123.stdout:2/690: dread - d0/d1f/d1c/d39/fd9 zero size 2024-03-30T04:28:15.141 INFO:tasks.workunit.client.0.smithi123.stdout:8/579: write da/d27/f3b [2042599,123117] 0 2024-03-30T04:28:15.144 INFO:tasks.workunit.client.0.smithi123.stdout:6/692: dread d10/d82/d8e/dc7/d42/f51 [0,4194304] 0 2024-03-30T04:28:15.144 INFO:tasks.workunit.client.1.smithi135.stdout:9/184: fsync d7/f12 0 2024-03-30T04:28:15.144 INFO:tasks.workunit.client.1.smithi135.stdout:1/172: dwrite d16/d19/f1f [0,4194304] 0 2024-03-30T04:28:15.145 INFO:tasks.workunit.client.1.smithi135.stdout:5/247: dwrite d1/d7/f43 [0,4194304] 0 2024-03-30T04:28:15.152 INFO:tasks.workunit.client.1.smithi135.stdout:1/173: stat fb 0 2024-03-30T04:28:15.153 INFO:tasks.workunit.client.0.smithi123.stdout:1/692: truncate db/dd/d42/d46/d51/dd7/fd8 305624 0 2024-03-30T04:28:15.153 INFO:tasks.workunit.client.0.smithi123.stdout:4/603: creat d2/d4/d2c/d42/d35/d9b/db7/fd3 x:0 0 0 2024-03-30T04:28:15.154 INFO:tasks.workunit.client.1.smithi135.stdout:3/197: rename d3/d17 to d3/d10/d16/d3e 0 2024-03-30T04:28:15.164 INFO:tasks.workunit.client.1.smithi135.stdout:2/197: write d4/da/f1e [899171,120204] 0 2024-03-30T04:28:15.165 INFO:tasks.workunit.client.0.smithi123.stdout:5/558: dwrite dd/d19/d1f/d35/d3a/f83 [0,4194304] 0 2024-03-30T04:28:15.167 INFO:tasks.workunit.client.0.smithi123.stdout:0/629: rename d6/dd/d16/f1e to d6/dd/d15/d28/fcc 0 2024-03-30T04:28:15.169 INFO:tasks.workunit.client.1.smithi135.stdout:5/248: chown d1/d3/f49 165442001 1 2024-03-30T04:28:15.169 INFO:tasks.workunit.client.1.smithi135.stdout:1/174: chown d16/f18 7582640 1 2024-03-30T04:28:15.169 INFO:tasks.workunit.client.0.smithi123.stdout:8/580: fdatasync da/dc/d14/d18/d21/f45 0 2024-03-30T04:28:15.171 INFO:tasks.workunit.client.0.smithi123.stdout:5/559: write dd/d19/d1f/d3c/f97 [344477,75304] 0 2024-03-30T04:28:15.172 INFO:tasks.workunit.client.0.smithi123.stdout:3/643: creat d13/d24/d31/d23/d3a/d3c/fce x:0 0 0 2024-03-30T04:28:15.172 INFO:tasks.workunit.client.0.smithi123.stdout:4/604: symlink d2/d4/d2c/d42/d35/d5d/d6f/ld4 0 2024-03-30T04:28:15.173 INFO:tasks.workunit.client.1.smithi135.stdout:3/198: chown d3/d10 297 1 2024-03-30T04:28:15.175 INFO:tasks.workunit.client.0.smithi123.stdout:8/581: write da/dc/f10 [1367075,77104] 0 2024-03-30T04:28:15.178 INFO:tasks.workunit.client.0.smithi123.stdout:8/582: readlink da/dc/l34 0 2024-03-30T04:28:15.179 INFO:tasks.workunit.client.0.smithi123.stdout:1/693: creat db/dd/d5f/d6d/d85/d89/d90/d94/fe9 x:0 0 0 2024-03-30T04:28:15.183 INFO:tasks.workunit.client.1.smithi135.stdout:0/261: dwrite d0/d1/f43 [0,4194304] 0 2024-03-30T04:28:15.184 INFO:tasks.workunit.client.1.smithi135.stdout:9/185: creat d7/d1b/d37/f39 x:0 0 0 2024-03-30T04:28:15.184 INFO:tasks.workunit.client.1.smithi135.stdout:4/179: dread d10/d1a/f21 [0,4194304] 0 2024-03-30T04:28:15.185 INFO:tasks.workunit.client.0.smithi123.stdout:5/560: truncate dd/d19/d1f/d68/f73 4250633 0 2024-03-30T04:28:15.185 INFO:tasks.workunit.client.0.smithi123.stdout:3/644: creat d13/d24/d39/d79/d90/fcf x:0 0 0 2024-03-30T04:28:15.188 INFO:tasks.workunit.client.1.smithi135.stdout:7/274: dwrite d4/db/de/f30 [0,4194304] 0 2024-03-30T04:28:15.188 INFO:tasks.workunit.client.1.smithi135.stdout:1/175: dread d16/f1c [0,4194304] 0 2024-03-30T04:28:15.189 INFO:tasks.workunit.client.0.smithi123.stdout:1/694: truncate db/dd/d42/d46/d51/fda 623409 0 2024-03-30T04:28:15.189 INFO:tasks.workunit.client.0.smithi123.stdout:4/605: dread d2/d4/d7/f4e [0,4194304] 0 2024-03-30T04:28:15.192 INFO:tasks.workunit.client.0.smithi123.stdout:3/645: write d13/f1b [4313256,91026] 0 2024-03-30T04:28:15.192 INFO:tasks.workunit.client.1.smithi135.stdout:8/225: dwrite f5 [0,4194304] 0 2024-03-30T04:28:15.195 INFO:tasks.workunit.client.1.smithi135.stdout:0/262: write d0/f3 [4919291,97604] 0 2024-03-30T04:28:15.197 INFO:tasks.workunit.client.1.smithi135.stdout:2/198: write d4/d32/f26 [1254445,23446] 0 2024-03-30T04:28:15.198 INFO:tasks.workunit.client.0.smithi123.stdout:7/618: dread d2/f87 [0,4194304] 0 2024-03-30T04:28:15.199 INFO:tasks.workunit.client.0.smithi123.stdout:5/561: rename dd/f81 to dd/d19/d1f/d3c/d72/d8c/fb1 0 2024-03-30T04:28:15.200 INFO:tasks.workunit.client.1.smithi135.stdout:2/199: chown d4/d32/f26 281565 1 2024-03-30T04:28:15.202 INFO:tasks.workunit.client.1.smithi135.stdout:8/226: chown d6/d8/d15/d2a/d35/l3e 5 1 2024-03-30T04:28:15.210 INFO:tasks.workunit.client.0.smithi123.stdout:1/695: rename db/dd/d5f/d6d/f81 to db/dd/d5f/d6d/d85/dd4/fea 0 2024-03-30T04:28:15.210 INFO:tasks.workunit.client.1.smithi135.stdout:4/180: mknod d10/d1a/c3f 0 2024-03-30T04:28:15.211 INFO:tasks.workunit.client.0.smithi123.stdout:7/619: stat d2/da/d11/d61/d55/d9e/fad 0 2024-03-30T04:28:15.214 INFO:tasks.workunit.client.0.smithi123.stdout:2/691: write d0/d1f/d27/fb0 [54078,28826] 0 2024-03-30T04:28:15.215 INFO:tasks.workunit.client.0.smithi123.stdout:5/562: write dd/d19/d3e/d5b/f8d [603276,23365] 0 2024-03-30T04:28:15.218 INFO:tasks.workunit.client.1.smithi135.stdout:8/227: symlink d6/d8/d15/d2a/l3f 0 2024-03-30T04:28:15.219 INFO:tasks.workunit.client.0.smithi123.stdout:4/606: getdents d2/d4/d7/d1a 0 2024-03-30T04:28:15.221 INFO:tasks.workunit.client.0.smithi123.stdout:4/607: readlink d2/d4/d7/l1d 0 2024-03-30T04:28:15.224 INFO:tasks.workunit.client.1.smithi135.stdout:1/176: rename d16/d19/d20/d26/d33 to d16/d29/d2c/d3c 0 2024-03-30T04:28:15.226 INFO:tasks.workunit.client.0.smithi123.stdout:8/583: dread f8 [0,4194304] 0 2024-03-30T04:28:15.227 INFO:tasks.workunit.client.1.smithi135.stdout:0/263: dread d0/d5/d9/f15 [0,4194304] 0 2024-03-30T04:28:15.227 INFO:tasks.workunit.client.1.smithi135.stdout:3/199: dwrite d3/d6/fa [0,4194304] 0 2024-03-30T04:28:15.228 INFO:tasks.workunit.client.1.smithi135.stdout:6/164: dread d2/dd/f1a [0,4194304] 0 2024-03-30T04:28:15.229 INFO:tasks.workunit.client.1.smithi135.stdout:4/181: mkdir d10/d1a/d32/d40 0 2024-03-30T04:28:15.229 INFO:tasks.workunit.client.0.smithi123.stdout:1/696: link db/dd/d42/d46/d51/d72/fc8 db/dd/d5f/feb 0 2024-03-30T04:28:15.232 INFO:tasks.workunit.client.1.smithi135.stdout:1/177: dread - d16/d29/d2c/d3c/f3b zero size 2024-03-30T04:28:15.234 INFO:tasks.workunit.client.1.smithi135.stdout:5/249: getdents d1/d3 0 2024-03-30T04:28:15.234 INFO:tasks.workunit.client.1.smithi135.stdout:7/275: creat d4/db/d1a/d3d/f5b x:0 0 0 2024-03-30T04:28:15.235 INFO:tasks.workunit.client.0.smithi123.stdout:9/551: dread d6/d1f/f71 [0,4194304] 0 2024-03-30T04:28:15.238 INFO:tasks.workunit.client.0.smithi123.stdout:8/584: unlink da/dc/d14/d18/l63 0 2024-03-30T04:28:15.240 INFO:tasks.workunit.client.1.smithi135.stdout:6/165: rmdir d2/d2e 39 2024-03-30T04:28:15.242 INFO:tasks.workunit.client.1.smithi135.stdout:2/200: dwrite d4/d32/f13 [0,4194304] 0 2024-03-30T04:28:15.243 INFO:tasks.workunit.client.0.smithi123.stdout:6/693: dread d10/d20/f3b [0,4194304] 0 2024-03-30T04:28:15.243 INFO:tasks.workunit.client.0.smithi123.stdout:9/552: chown d6/d1f/d72/d7d/c88 1817265 1 2024-03-30T04:28:15.245 INFO:tasks.workunit.client.0.smithi123.stdout:0/630: truncate d6/dd/f51 1855398 0 2024-03-30T04:28:15.247 INFO:tasks.workunit.client.1.smithi135.stdout:7/276: chown d4/db/d1a/d4b/l4a 16131 1 2024-03-30T04:28:15.248 INFO:tasks.workunit.client.0.smithi123.stdout:1/697: creat db/dd/d5f/dae/fec x:0 0 0 2024-03-30T04:28:15.249 INFO:tasks.workunit.client.1.smithi135.stdout:4/182: rename f3 to d10/d1a/d3d/f41 0 2024-03-30T04:28:15.252 INFO:tasks.workunit.client.1.smithi135.stdout:0/264: dread d0/f3 [4194304,4194304] 0 2024-03-30T04:28:15.253 INFO:tasks.workunit.client.0.smithi123.stdout:4/608: rmdir d2/d4/d2c/d42/d35/d5d/d9f 0 2024-03-30T04:28:15.253 INFO:tasks.workunit.client.0.smithi123.stdout:3/646: rmdir d13/d24 39 2024-03-30T04:28:15.254 INFO:tasks.workunit.client.1.smithi135.stdout:3/200: mknod d3/d6/c3f 0 2024-03-30T04:28:15.256 INFO:tasks.workunit.client.0.smithi123.stdout:9/553: dread d6/d58/f5c [0,4194304] 0 2024-03-30T04:28:15.257 INFO:tasks.workunit.client.1.smithi135.stdout:0/265: chown d0/d1/d16/f28 232 1 2024-03-30T04:28:15.266 INFO:tasks.workunit.client.0.smithi123.stdout:6/694: mknod d10/d82/dc1/d6a/d7a/cec 0 2024-03-30T04:28:15.266 INFO:tasks.workunit.client.0.smithi123.stdout:2/692: write d0/d1f/d1c/d40/f5a [4598121,89019] 0 2024-03-30T04:28:15.267 INFO:tasks.workunit.client.0.smithi123.stdout:3/647: stat d13/d24/d39 0 2024-03-30T04:28:15.267 INFO:tasks.workunit.client.1.smithi135.stdout:1/178: mkdir d16/d29/d2c/d30/d3d 0 2024-03-30T04:28:15.268 INFO:tasks.workunit.client.1.smithi135.stdout:9/186: dwrite d7/f26 [0,4194304] 0 2024-03-30T04:28:15.273 INFO:tasks.workunit.client.0.smithi123.stdout:4/609: truncate d2/d4/d7/d1a/f5e 1027545 0 2024-03-30T04:28:15.277 INFO:tasks.workunit.client.1.smithi135.stdout:7/277: unlink d4/d4d/d2c/d51/l52 0 2024-03-30T04:28:15.280 INFO:tasks.workunit.client.1.smithi135.stdout:4/183: dwrite d10/d1a/f1c [0,4194304] 0 2024-03-30T04:28:15.282 INFO:tasks.workunit.client.0.smithi123.stdout:1/698: write db/d1b/f28 [494644,104784] 0 2024-03-30T04:28:15.286 INFO:tasks.workunit.client.1.smithi135.stdout:6/166: creat d2/d2e/f31 x:0 0 0 2024-03-30T04:28:15.286 INFO:tasks.workunit.client.1.smithi135.stdout:9/187: creat d7/d1b/d37/f3a x:0 0 0 2024-03-30T04:28:15.292 INFO:tasks.workunit.client.1.smithi135.stdout:4/184: rmdir d10/d1a/d32 39 2024-03-30T04:28:15.299 INFO:tasks.workunit.client.1.smithi135.stdout:0/266: link d0/d5/d10/f31 d0/d5/f4d 0 2024-03-30T04:28:15.299 INFO:tasks.workunit.client.0.smithi123.stdout:2/693: dwrite d0/d1f/d1c/d80/f49 [0,4194304] 0 2024-03-30T04:28:15.300 INFO:tasks.workunit.client.1.smithi135.stdout:2/201: write d4/d6/f1a [4723877,42177] 0 2024-03-30T04:28:15.300 INFO:tasks.workunit.client.1.smithi135.stdout:6/167: mknod d2/dd/c32 0 2024-03-30T04:28:15.303 INFO:tasks.workunit.client.0.smithi123.stdout:9/554: dwrite d6/fa9 [0,4194304] 0 2024-03-30T04:28:15.305 INFO:tasks.workunit.client.1.smithi135.stdout:4/185: mkdir d10/d1a/d42 0 2024-03-30T04:28:15.308 INFO:tasks.workunit.client.1.smithi135.stdout:0/267: truncate d0/fc 1555332 0 2024-03-30T04:28:15.311 INFO:tasks.workunit.client.1.smithi135.stdout:9/188: read d7/d29/f2c [1201014,103043] 0 2024-03-30T04:28:15.313 INFO:tasks.workunit.client.1.smithi135.stdout:2/202: write d4/f5 [3770304,58726] 0 2024-03-30T04:28:15.316 INFO:tasks.workunit.client.1.smithi135.stdout:1/179: dwrite d16/d1a/f23 [0,4194304] 0 2024-03-30T04:28:15.316 INFO:tasks.workunit.client.1.smithi135.stdout:5/250: dwrite d1/d3/f49 [0,4194304] 0 2024-03-30T04:28:15.317 INFO:tasks.workunit.client.0.smithi123.stdout:1/699: rmdir db/dd/d42/d46/d56/daf 39 2024-03-30T04:28:15.319 INFO:tasks.workunit.client.0.smithi123.stdout:2/694: mknod d0/d1f/d78/ce3 0 2024-03-30T04:28:15.322 INFO:tasks.workunit.client.0.smithi123.stdout:1/700: mknod db/dd/d5f/ced 0 2024-03-30T04:28:15.325 INFO:tasks.workunit.client.1.smithi135.stdout:0/268: unlink d0/fc 0 2024-03-30T04:28:15.325 INFO:tasks.workunit.client.1.smithi135.stdout:3/201: dread d3/d10/d16/f1e [0,4194304] 0 2024-03-30T04:28:15.328 INFO:tasks.workunit.client.0.smithi123.stdout:6/695: link d10/d82/fe3 d10/d82/d8e/dc7/fed 0 2024-03-30T04:28:15.328 INFO:tasks.workunit.client.0.smithi123.stdout:8/585: creat da/dc/d14/d18/fac x:0 0 0 2024-03-30T04:28:15.328 INFO:tasks.workunit.client.0.smithi123.stdout:1/701: chown db/dd/d42/d46/d51/dd7/ddc 0 1 2024-03-30T04:28:15.328 INFO:tasks.workunit.client.0.smithi123.stdout:2/695: mkdir d0/d1f/d1c/d39/de4 0 2024-03-30T04:28:15.329 INFO:tasks.workunit.client.1.smithi135.stdout:9/189: creat d7/d1b/f3b x:0 0 0 2024-03-30T04:28:15.334 INFO:tasks.workunit.client.0.smithi123.stdout:2/696: mknod d0/d1f/d82/ce5 0 2024-03-30T04:28:15.334 INFO:tasks.workunit.client.1.smithi135.stdout:5/251: mkdir d1/da/d4d 0 2024-03-30T04:28:15.334 INFO:tasks.workunit.client.1.smithi135.stdout:2/203: fdatasync d4/d32/f33 0 2024-03-30T04:28:15.335 INFO:tasks.workunit.client.1.smithi135.stdout:0/269: unlink d0/d5/f4d 0 2024-03-30T04:28:15.335 INFO:tasks.workunit.client.1.smithi135.stdout:7/278: dread d4/db/d1a/f24 [0,4194304] 0 2024-03-30T04:28:15.335 INFO:tasks.workunit.client.1.smithi135.stdout:3/202: symlink d3/d6/l40 0 2024-03-30T04:28:15.336 INFO:tasks.workunit.client.0.smithi123.stdout:2/697: write d0/d1f/fdd [227469,8018] 0 2024-03-30T04:28:15.337 INFO:tasks.workunit.client.1.smithi135.stdout:9/190: creat d7/d29/f3c x:0 0 0 2024-03-30T04:28:15.339 INFO:tasks.workunit.client.1.smithi135.stdout:3/203: chown d3/d10/c29 78235 1 2024-03-30T04:28:15.339 INFO:tasks.workunit.client.1.smithi135.stdout:2/204: stat d4/d32/d12/c24 0 2024-03-30T04:28:15.340 INFO:tasks.workunit.client.1.smithi135.stdout:7/279: write d4/f8 [347526,85156] 0 2024-03-30T04:28:15.340 INFO:tasks.workunit.client.1.smithi135.stdout:3/204: read - d3/f3c zero size 2024-03-30T04:28:15.340 INFO:tasks.workunit.client.1.smithi135.stdout:6/168: dread d2/f9 [0,4194304] 0 2024-03-30T04:28:15.341 INFO:tasks.workunit.client.0.smithi123.stdout:2/698: dread - d0/d1f/d1c/d20/dc6/fd3 zero size 2024-03-30T04:28:15.343 INFO:tasks.workunit.client.0.smithi123.stdout:1/702: getdents db/dd/d5f/dae 0 2024-03-30T04:28:15.343 INFO:tasks.workunit.client.0.smithi123.stdout:2/699: fsync d0/d1f/d16/fdc 0 2024-03-30T04:28:15.343 INFO:tasks.workunit.client.1.smithi135.stdout:5/252: dread d1/f36 [0,4194304] 0 2024-03-30T04:28:15.343 INFO:tasks.workunit.client.1.smithi135.stdout:2/205: write d4/d6/f1f [4414208,7532] 0 2024-03-30T04:28:15.345 INFO:tasks.workunit.client.0.smithi123.stdout:1/703: stat db/dd/f4c 0 2024-03-30T04:28:15.345 INFO:tasks.workunit.client.0.smithi123.stdout:2/700: readlink d0/d1f/d1c/l45 0 2024-03-30T04:28:15.347 INFO:tasks.workunit.client.1.smithi135.stdout:2/206: readlink d4/d6/l27 0 2024-03-30T04:28:15.348 INFO:tasks.workunit.client.1.smithi135.stdout:0/270: truncate d0/d13/d1e/f49 886479 0 2024-03-30T04:28:15.349 INFO:tasks.workunit.client.0.smithi123.stdout:2/701: mkdir d0/d1f/d1c/d80/de6 0 2024-03-30T04:28:15.351 INFO:tasks.workunit.client.1.smithi135.stdout:4/186: write fb [2659030,104691] 0 2024-03-30T04:28:15.351 INFO:tasks.workunit.client.1.smithi135.stdout:0/271: chown d0/d1/d16/f1b 5271587 1 2024-03-30T04:28:15.353 INFO:tasks.workunit.client.0.smithi123.stdout:1/704: rename db/dd/d5f/d6d/d7e/ca4 to db/dd/d42/d46/d5d/d7a/cee 0 2024-03-30T04:28:15.353 INFO:tasks.workunit.client.1.smithi135.stdout:4/187: write fa [2442592,129249] 0 2024-03-30T04:28:15.354 INFO:tasks.workunit.client.0.smithi123.stdout:1/705: stat db/lb8 0 2024-03-30T04:28:15.355 INFO:tasks.workunit.client.0.smithi123.stdout:2/702: creat d0/d1f/d27/d68/dd6/d8c/fe7 x:0 0 0 2024-03-30T04:28:15.356 INFO:tasks.workunit.client.0.smithi123.stdout:2/703: readlink d0/lb 0 2024-03-30T04:28:15.359 INFO:tasks.workunit.client.0.smithi123.stdout:4/610: write d2/d4/d2c/d42/dad/d91/d9d/fa6 [1070578,8991] 0 2024-03-30T04:28:15.359 INFO:tasks.workunit.client.1.smithi135.stdout:7/280: rename d4/d4d to d4/db/d1a/d5c 0 2024-03-30T04:28:15.359 INFO:tasks.workunit.client.1.smithi135.stdout:1/180: write d16/f2f [930798,51951] 0 2024-03-30T04:28:15.362 INFO:tasks.workunit.client.1.smithi135.stdout:6/169: dread d2/dd/d18/f22 [0,4194304] 0 2024-03-30T04:28:15.362 INFO:tasks.workunit.client.0.smithi123.stdout:2/704: write d0/d1f/d1c/d39/d4b/fa0 [885294,53235] 0 2024-03-30T04:28:15.364 INFO:tasks.workunit.client.1.smithi135.stdout:9/191: link d7/f12 d7/d1b/f3d 0 2024-03-30T04:28:15.365 INFO:tasks.workunit.client.0.smithi123.stdout:9/555: truncate d6/d1f/d45/d97/d69/f6c 94735 0 2024-03-30T04:28:15.366 INFO:tasks.workunit.client.1.smithi135.stdout:3/205: dread d3/f4 [0,4194304] 0 2024-03-30T04:28:15.367 INFO:tasks.workunit.client.1.smithi135.stdout:0/272: symlink d0/d19/l4e 0 2024-03-30T04:28:15.368 INFO:tasks.workunit.client.0.smithi123.stdout:6/696: write d10/d82/f8a [3970751,7100] 0 2024-03-30T04:28:15.370 INFO:tasks.workunit.client.1.smithi135.stdout:0/273: write d0/d5/d10/f2b [1417102,39202] 0 2024-03-30T04:28:15.370 INFO:tasks.workunit.client.1.smithi135.stdout:2/207: mknod d4/c42 0 2024-03-30T04:28:15.371 INFO:tasks.workunit.client.0.smithi123.stdout:3/648: dwrite d13/d24/d31/d23/f91 [0,4194304] 0 2024-03-30T04:28:15.373 INFO:tasks.workunit.client.0.smithi123.stdout:1/706: getdents db/dd/d5f/d6d/d7e 0 2024-03-30T04:28:15.374 INFO:tasks.workunit.client.1.smithi135.stdout:5/253: creat d1/d7/d11/d3a/f4e x:0 0 0 2024-03-30T04:28:15.374 INFO:tasks.workunit.client.1.smithi135.stdout:1/181: symlink d16/d1a/l3e 0 2024-03-30T04:28:15.375 INFO:tasks.workunit.client.0.smithi123.stdout:9/556: fdatasync d6/d49/f65 0 2024-03-30T04:28:15.375 INFO:tasks.workunit.client.0.smithi123.stdout:8/586: write da/d27/d3c/d2d/f2e [1065971,32977] 0 2024-03-30T04:28:15.376 INFO:tasks.workunit.client.0.smithi123.stdout:2/705: getdents d0/d1f/d1c/d80/de6 0 2024-03-30T04:28:15.380 INFO:tasks.workunit.client.1.smithi135.stdout:3/206: mkdir d3/d10/d16/d41 0 2024-03-30T04:28:15.380 INFO:tasks.workunit.client.1.smithi135.stdout:6/170: write d2/f11 [1679928,27755] 0 2024-03-30T04:28:15.380 INFO:tasks.workunit.client.1.smithi135.stdout:9/192: write f6 [166838,98654] 0 2024-03-30T04:28:15.381 INFO:tasks.workunit.client.0.smithi123.stdout:6/697: mkdir d10/d82/dc1/d6a/d7a/de8/dee 0 2024-03-30T04:28:15.383 INFO:tasks.workunit.client.1.smithi135.stdout:4/188: rename d10/c1e to d10/d1a/d2b/d39/c43 0 2024-03-30T04:28:15.384 INFO:tasks.workunit.client.1.smithi135.stdout:6/171: readlink d2/dd/d14/l29 0 2024-03-30T04:28:15.385 INFO:tasks.workunit.client.0.smithi123.stdout:4/611: truncate d2/d34/f93 102545 0 2024-03-30T04:28:15.387 INFO:tasks.workunit.client.1.smithi135.stdout:0/274: creat d0/d13/d1e/f4f x:0 0 0 2024-03-30T04:28:15.387 INFO:tasks.workunit.client.1.smithi135.stdout:6/172: write d2/f12 [46155,49510] 0 2024-03-30T04:28:15.387 INFO:tasks.workunit.client.0.smithi123.stdout:3/649: symlink d13/d24/d31/d23/d3a/d3c/d4e/d5d/ld0 0 2024-03-30T04:28:15.387 INFO:tasks.workunit.client.0.smithi123.stdout:9/557: dread - d6/d1f/d72/f95 zero size 2024-03-30T04:28:15.388 INFO:tasks.workunit.client.1.smithi135.stdout:7/281: mkdir d4/db/d1a/d5c/d5d 0 2024-03-30T04:28:15.391 INFO:tasks.workunit.client.1.smithi135.stdout:7/282: readlink d4/db/d1a/d5c/d2c/l35 0 2024-03-30T04:28:15.391 INFO:tasks.workunit.client.0.smithi123.stdout:6/698: creat d10/d82/d8e/dc7/d5b/fef x:0 0 0 2024-03-30T04:28:15.391 INFO:tasks.workunit.client.1.smithi135.stdout:9/193: chown d7/d29/c2d 125516 1 2024-03-30T04:28:15.392 INFO:tasks.workunit.client.0.smithi123.stdout:8/587: creat da/dc/da9/fad x:0 0 0 2024-03-30T04:28:15.394 INFO:tasks.workunit.client.0.smithi123.stdout:9/558: mknod d6/d1f/d92/db8/cbf 0 2024-03-30T04:28:15.395 INFO:tasks.workunit.client.0.smithi123.stdout:3/650: truncate d13/d24/d31/f98 295025 0 2024-03-30T04:28:15.395 INFO:tasks.workunit.client.1.smithi135.stdout:2/208: rmdir d4 39 2024-03-30T04:28:15.396 INFO:tasks.workunit.client.1.smithi135.stdout:4/189: truncate d10/d1a/d32/f3e 1004005 0 2024-03-30T04:28:15.398 INFO:tasks.workunit.client.1.smithi135.stdout:5/254: link d1/da/f1b d1/f4f 0 2024-03-30T04:28:15.398 INFO:tasks.workunit.client.1.smithi135.stdout:4/190: write d10/d1a/f3a [159774,35797] 0 2024-03-30T04:28:15.399 INFO:tasks.workunit.client.1.smithi135.stdout:4/191: stat cc 0 2024-03-30T04:28:15.400 INFO:tasks.workunit.client.0.smithi123.stdout:4/612: symlink d2/ld5 0 2024-03-30T04:28:15.400 INFO:tasks.workunit.client.0.smithi123.stdout:8/588: mkdir da/d27/d3c/d2d/dae 0 2024-03-30T04:28:15.400 INFO:tasks.workunit.client.1.smithi135.stdout:6/173: mknod d2/dd/d14/c33 0 2024-03-30T04:28:15.401 INFO:tasks.workunit.client.0.smithi123.stdout:9/559: mknod d6/d7c/d82/cc0 0 2024-03-30T04:28:15.402 INFO:tasks.workunit.client.1.smithi135.stdout:5/255: write d1/da/d17/f3d [2560602,18395] 0 2024-03-30T04:28:15.403 INFO:tasks.workunit.client.1.smithi135.stdout:0/275: unlink d0/d1/d16/l2f 0 2024-03-30T04:28:15.405 INFO:tasks.workunit.client.1.smithi135.stdout:9/194: creat d7/d29/f3e x:0 0 0 2024-03-30T04:28:15.409 INFO:tasks.workunit.client.0.smithi123.stdout:1/707: write db/dd/d42/d46/d56/daf/fb5 [1030217,121265] 0 2024-03-30T04:28:15.409 INFO:tasks.workunit.client.0.smithi123.stdout:9/560: write f1 [1531278,4910] 0 2024-03-30T04:28:15.410 INFO:tasks.workunit.client.1.smithi135.stdout:6/174: unlink d2/dd/d18/c2c 0 2024-03-30T04:28:15.411 INFO:tasks.workunit.client.1.smithi135.stdout:6/175: stat d2/dd 0 2024-03-30T04:28:15.414 INFO:tasks.workunit.client.1.smithi135.stdout:5/256: write d1/f4f [3709328,23694] 0 2024-03-30T04:28:15.416 INFO:tasks.workunit.client.0.smithi123.stdout:1/708: fsync db/dd/d42/d46/d5d/d62/f6b 0 2024-03-30T04:28:15.417 INFO:tasks.workunit.client.1.smithi135.stdout:7/283: mknod d4/db/d1a/d5c/d2c/d51/d56/c5e 0 2024-03-30T04:28:15.418 INFO:tasks.workunit.client.1.smithi135.stdout:2/209: mkdir d4/d32/d43 0 2024-03-30T04:28:15.420 INFO:tasks.workunit.client.1.smithi135.stdout:2/210: readlink d4/d6/l1b 0 2024-03-30T04:28:15.426 INFO:tasks.workunit.client.0.smithi123.stdout:9/561: getdents d6/d1f/d45 0 2024-03-30T04:28:15.427 INFO:tasks.workunit.client.1.smithi135.stdout:3/207: getdents d3/d6 0 2024-03-30T04:28:15.427 INFO:tasks.workunit.client.1.smithi135.stdout:0/276: mknod d0/d13/c50 0 2024-03-30T04:28:15.427 INFO:tasks.workunit.client.1.smithi135.stdout:3/208: truncate d3/db/fe 4658400 0 2024-03-30T04:28:15.427 INFO:tasks.workunit.client.0.smithi123.stdout:2/706: dwrite d0/d1f/d1c/d36/f89 [0,4194304] 0 2024-03-30T04:28:15.428 INFO:tasks.workunit.client.0.smithi123.stdout:1/709: creat db/dd/d42/d46/d56/dce/fef x:0 0 0 2024-03-30T04:28:15.428 INFO:tasks.workunit.client.1.smithi135.stdout:6/176: dread d2/f13 [4194304,4194304] 0 2024-03-30T04:28:15.429 INFO:tasks.workunit.client.1.smithi135.stdout:1/182: dread fb [0,4194304] 0 2024-03-30T04:28:15.429 INFO:tasks.workunit.client.1.smithi135.stdout:5/257: rename d1/d7/l2d to d1/d3/d2b/l50 0 2024-03-30T04:28:15.430 INFO:tasks.workunit.client.1.smithi135.stdout:4/192: rmdir d10/d38 0 2024-03-30T04:28:15.430 INFO:tasks.workunit.client.1.smithi135.stdout:5/258: readlink d1/le 0 2024-03-30T04:28:15.431 INFO:tasks.workunit.client.1.smithi135.stdout:1/183: write d16/f2f [937747,29010] 0 2024-03-30T04:28:15.433 INFO:tasks.workunit.client.0.smithi123.stdout:6/699: dwrite d10/f38 [0,4194304] 0 2024-03-30T04:28:15.433 INFO:tasks.workunit.client.0.smithi123.stdout:9/562: fdatasync d6/d23/f33 0 2024-03-30T04:28:15.436 INFO:tasks.workunit.client.1.smithi135.stdout:5/259: fsync d1/d3/d2b/f2f 0 2024-03-30T04:28:15.437 INFO:tasks.workunit.client.1.smithi135.stdout:0/277: creat d0/d5/d9/d23/d2a/f51 x:0 0 0 2024-03-30T04:28:15.437 INFO:tasks.workunit.client.0.smithi123.stdout:3/651: dwrite d13/d24/d31/d23/d3a/d3c/d4e/d5d/f82 [0,4194304] 0 2024-03-30T04:28:15.440 INFO:tasks.workunit.client.0.smithi123.stdout:2/707: creat d0/d1f/d82/fe8 x:0 0 0 2024-03-30T04:28:15.442 INFO:tasks.workunit.client.0.smithi123.stdout:2/708: readlink d0/d1f/d1c/d36/laf 0 2024-03-30T04:28:15.445 INFO:tasks.workunit.client.1.smithi135.stdout:9/195: getdents d7/d1b 0 2024-03-30T04:28:15.445 INFO:tasks.workunit.client.0.smithi123.stdout:6/700: mkdir d10/d82/d8e/df0 0 2024-03-30T04:28:15.447 INFO:tasks.workunit.client.0.smithi123.stdout:1/710: rmdir db/dd/d42 39 2024-03-30T04:28:15.448 INFO:tasks.workunit.client.0.smithi123.stdout:8/589: dread da/f9a [0,4194304] 0 2024-03-30T04:28:15.448 INFO:tasks.workunit.client.1.smithi135.stdout:1/184: dread d16/f2f [0,4194304] 0 2024-03-30T04:28:15.449 INFO:tasks.workunit.client.1.smithi135.stdout:1/185: dread - d16/d29/d32/f38 zero size 2024-03-30T04:28:15.449 INFO:tasks.workunit.client.1.smithi135.stdout:9/196: truncate d7/d1b/f3d 5085899 0 2024-03-30T04:28:15.452 INFO:tasks.workunit.client.1.smithi135.stdout:7/284: link d4/db/d1a/d5c/d2c/d51/d56/c5e d4/db/d1a/d3d/c5f 0 2024-03-30T04:28:15.453 INFO:tasks.workunit.client.0.smithi123.stdout:1/711: creat db/dd/d5f/d6d/d7e/ff0 x:0 0 0 2024-03-30T04:28:15.454 INFO:tasks.workunit.client.0.smithi123.stdout:8/590: mkdir da/d27/daf 0 2024-03-30T04:28:15.455 INFO:tasks.workunit.client.1.smithi135.stdout:2/211: creat d4/da/f44 x:0 0 0 2024-03-30T04:28:15.457 INFO:tasks.workunit.client.1.smithi135.stdout:6/177: creat d2/f34 x:0 0 0 2024-03-30T04:28:15.458 INFO:tasks.workunit.client.0.smithi123.stdout:8/591: dread - da/d27/d3c/f98 zero size 2024-03-30T04:28:15.459 INFO:tasks.workunit.client.0.smithi123.stdout:1/712: unlink db/fe 0 2024-03-30T04:28:15.461 INFO:tasks.workunit.client.1.smithi135.stdout:3/209: dwrite d3/f5 [0,4194304] 0 2024-03-30T04:28:15.461 INFO:tasks.workunit.client.0.smithi123.stdout:8/592: chown da/dc/d14/d18 81626 1 2024-03-30T04:28:15.463 INFO:tasks.workunit.client.1.smithi135.stdout:3/210: chown d3/db 58385 1 2024-03-30T04:28:15.467 INFO:tasks.workunit.client.1.smithi135.stdout:0/278: mkdir d0/d13/d1d/d3b/d52 0 2024-03-30T04:28:15.467 INFO:tasks.workunit.client.1.smithi135.stdout:7/285: read d4/db/f1c [2206165,70282] 0 2024-03-30T04:28:15.469 INFO:tasks.workunit.client.0.smithi123.stdout:8/593: write da/d27/d3c/d2d/d5c/f6f [3612057,74952] 0 2024-03-30T04:28:15.471 INFO:tasks.workunit.client.0.smithi123.stdout:3/652: dwrite d13/d24/d39/d79/d90/fac [0,4194304] 0 2024-03-30T04:28:15.472 INFO:tasks.workunit.client.0.smithi123.stdout:2/709: dwrite d0/d1f/d1c/d20/dc6/fd3 [0,4194304] 0 2024-03-30T04:28:15.474 INFO:tasks.workunit.client.1.smithi135.stdout:2/212: rename d4/d6/d20 to d4/d32/d12/d45 0 2024-03-30T04:28:15.481 INFO:tasks.workunit.client.1.smithi135.stdout:2/213: write d4/d6/f1a [1230324,55965] 0 2024-03-30T04:28:15.483 INFO:tasks.workunit.client.1.smithi135.stdout:4/193: dwrite d10/d1a/f23 [0,4194304] 0 2024-03-30T04:28:15.488 INFO:tasks.workunit.client.0.smithi123.stdout:8/594: creat da/dc/d14/d18/fb0 x:0 0 0 2024-03-30T04:28:15.489 INFO:tasks.workunit.client.1.smithi135.stdout:7/286: symlink d4/db/d1a/d5c/d2c/d2e/d46/l60 0 2024-03-30T04:28:15.490 INFO:tasks.workunit.client.0.smithi123.stdout:6/701: write d10/fa0 [730685,6157] 0 2024-03-30T04:28:15.490 INFO:tasks.workunit.client.0.smithi123.stdout:3/653: dread d13/d24/d31/d23/d3a/d3c/fa8 [0,4194304] 0 2024-03-30T04:28:15.492 INFO:tasks.workunit.client.1.smithi135.stdout:4/194: symlink d10/d1a/d32/l44 0 2024-03-30T04:28:15.493 INFO:tasks.workunit.client.0.smithi123.stdout:6/702: fdatasync d10/d82/d8e/dc7/d5b/fef 0 2024-03-30T04:28:15.493 INFO:tasks.workunit.client.0.smithi123.stdout:2/710: getdents d0/d1f/d27/d68/dd6/d99 0 2024-03-30T04:28:15.494 INFO:tasks.workunit.client.1.smithi135.stdout:2/214: rmdir d4/d32/d2e/d2f 39 2024-03-30T04:28:15.495 INFO:tasks.workunit.client.0.smithi123.stdout:6/703: stat d10/f27 0 2024-03-30T04:28:15.496 INFO:tasks.workunit.client.0.smithi123.stdout:8/595: creat da/dc/d14/d9d/fb1 x:0 0 0 2024-03-30T04:28:15.500 INFO:tasks.workunit.client.1.smithi135.stdout:6/178: getdents d2/dd/d14 0 2024-03-30T04:28:15.500 INFO:tasks.workunit.client.0.smithi123.stdout:3/654: creat d13/d24/d31/d23/d3a/fd1 x:0 0 0 2024-03-30T04:28:15.501 INFO:tasks.workunit.client.0.smithi123.stdout:2/711: rename d0/d1f/d27/fb0 to d0/d37/d9b/fe9 0 2024-03-30T04:28:15.503 INFO:tasks.workunit.client.1.smithi135.stdout:3/211: dwrite d3/d10/d16/d23/f30 [0,4194304] 0 2024-03-30T04:28:15.503 INFO:tasks.workunit.client.1.smithi135.stdout:2/215: write d4/d6/f1f [567414,19284] 0 2024-03-30T04:28:15.504 INFO:tasks.workunit.client.0.smithi123.stdout:3/655: dread - d13/d24/d39/d79/d90/fcf zero size 2024-03-30T04:28:15.505 INFO:tasks.workunit.client.0.smithi123.stdout:6/704: rmdir d10/d82/d9a/db3 39 2024-03-30T04:28:15.505 INFO:tasks.workunit.client.0.smithi123.stdout:8/596: mknod da/d27/d56/d6c/cb2 0 2024-03-30T04:28:15.506 INFO:tasks.workunit.client.1.smithi135.stdout:4/195: creat d10/d1a/d2b/d39/f45 x:0 0 0 2024-03-30T04:28:15.506 INFO:tasks.workunit.client.1.smithi135.stdout:0/279: dwrite d0/d5/d9/f17 [0,4194304] 0 2024-03-30T04:28:15.510 INFO:tasks.workunit.client.1.smithi135.stdout:2/216: truncate d4/da/f44 816044 0 2024-03-30T04:28:15.512 INFO:tasks.workunit.client.1.smithi135.stdout:6/179: symlink d2/dd/d14/l35 0 2024-03-30T04:28:15.513 INFO:tasks.workunit.client.1.smithi135.stdout:3/212: write d3/d10/f2d [457484,49069] 0 2024-03-30T04:28:15.516 INFO:tasks.workunit.client.1.smithi135.stdout:7/287: mkdir d4/db/d1a/d5c/d5d/d61 0 2024-03-30T04:28:15.517 INFO:tasks.workunit.client.0.smithi123.stdout:7/620: sync 2024-03-30T04:28:15.525 INFO:tasks.workunit.client.1.smithi135.stdout:2/217: mkdir d4/d32/d46 0 2024-03-30T04:28:15.525 INFO:tasks.workunit.client.0.smithi123.stdout:7/621: symlink d2/dc/d7f/ld2 0 2024-03-30T04:28:15.528 INFO:tasks.workunit.client.1.smithi135.stdout:3/213: creat d3/d10/d16/d23/f42 x:0 0 0 2024-03-30T04:28:15.528 INFO:tasks.workunit.client.0.smithi123.stdout:8/597: dread da/dc/f32 [0,4194304] 0 2024-03-30T04:28:15.529 INFO:tasks.workunit.client.1.smithi135.stdout:0/280: mkdir d0/d13/d53 0 2024-03-30T04:28:15.531 INFO:tasks.workunit.client.1.smithi135.stdout:7/288: truncate d4/db/d1a/d5c/f4e 823871 0 2024-03-30T04:28:15.532 INFO:tasks.workunit.client.1.smithi135.stdout:1/186: dwrite d16/d1a/f21 [0,4194304] 0 2024-03-30T04:28:15.533 INFO:tasks.workunit.client.1.smithi135.stdout:5/260: dwrite d1/d7/f13 [0,4194304] 0 2024-03-30T04:28:15.533 INFO:tasks.workunit.client.1.smithi135.stdout:3/214: fdatasync d3/d6/d22/d33/f3a 0 2024-03-30T04:28:15.535 INFO:tasks.workunit.client.0.smithi123.stdout:3/656: dwrite d13/d24/d31/d23/d3a/d3c/d4e/d5d/fb7 [0,4194304] 0 2024-03-30T04:28:15.536 INFO:tasks.workunit.client.1.smithi135.stdout:2/218: mkdir d4/d6/d47 0 2024-03-30T04:28:15.537 INFO:tasks.workunit.client.0.smithi123.stdout:8/598: unlink da/d27/d3c/d2d/fa0 0 2024-03-30T04:28:15.544 INFO:tasks.workunit.client.1.smithi135.stdout:0/281: rmdir d0/d19/d2d 39 2024-03-30T04:28:15.544 INFO:tasks.workunit.client.0.smithi123.stdout:6/705: truncate d10/d82/dc1/d45/fe5 233481 0 2024-03-30T04:28:15.545 INFO:tasks.workunit.client.1.smithi135.stdout:3/215: creat d3/d10/d16/f43 x:0 0 0 2024-03-30T04:28:15.546 INFO:tasks.workunit.client.1.smithi135.stdout:6/180: rmdir d2 39 2024-03-30T04:28:15.548 INFO:tasks.workunit.client.1.smithi135.stdout:4/196: rename d10/d1a/d2b/d39 to d10/d1a/d42/d46 0 2024-03-30T04:28:15.549 INFO:tasks.workunit.client.0.smithi123.stdout:7/622: link d2/da/d11/d61/d55/d59/db5/cce d2/dc/d7f/cd3 0 2024-03-30T04:28:15.549 INFO:tasks.workunit.client.0.smithi123.stdout:2/712: dwrite d0/d1f/d16/f3c [0,4194304] 0 2024-03-30T04:28:15.550 INFO:tasks.workunit.client.1.smithi135.stdout:4/197: chown d10/d1a/f1c 27440 1 2024-03-30T04:28:15.551 INFO:tasks.workunit.client.0.smithi123.stdout:6/706: mkdir d10/d82/dc1/d6a/d7a/de2/df1 0 2024-03-30T04:28:15.557 INFO:tasks.workunit.client.1.smithi135.stdout:7/289: dwrite d4/db/de/f1d [0,4194304] 0 2024-03-30T04:28:15.561 INFO:tasks.workunit.client.0.smithi123.stdout:8/599: dwrite da/dc/d14/d18/f86 [0,4194304] 0 2024-03-30T04:28:15.561 INFO:tasks.workunit.client.1.smithi135.stdout:2/219: creat d4/d32/d2e/f48 x:0 0 0 2024-03-30T04:28:15.563 INFO:tasks.workunit.client.1.smithi135.stdout:2/220: fsync d4/f5 0 2024-03-30T04:28:15.566 INFO:tasks.workunit.client.0.smithi123.stdout:8/600: dread - da/dc/d14/f6e zero size 2024-03-30T04:28:15.566 INFO:tasks.workunit.client.1.smithi135.stdout:6/181: unlink d2/fa 0 2024-03-30T04:28:15.567 INFO:tasks.workunit.client.1.smithi135.stdout:0/282: mkdir d0/d13/d53/d54 0 2024-03-30T04:28:15.567 INFO:tasks.workunit.client.0.smithi123.stdout:8/601: stat da/d27/d3c/d2d/f2e 0 2024-03-30T04:28:15.568 INFO:tasks.workunit.client.1.smithi135.stdout:7/290: symlink d4/db/d1a/d5c/d2c/d2e/l62 0 2024-03-30T04:28:15.571 INFO:tasks.workunit.client.1.smithi135.stdout:6/182: readlink d2/dd/d14/l29 0 2024-03-30T04:28:15.575 INFO:tasks.workunit.client.1.smithi135.stdout:0/283: chown d0/d5/d9/d23/d2a/f51 0 1 2024-03-30T04:28:15.576 INFO:tasks.workunit.client.0.smithi123.stdout:8/602: link da/dc/d14/d18/fac da/d27/fb3 0 2024-03-30T04:28:15.582 INFO:tasks.workunit.client.0.smithi123.stdout:7/623: write d2/da/f71 [993574,54646] 0 2024-03-30T04:28:15.583 INFO:tasks.workunit.client.1.smithi135.stdout:4/198: dwrite d10/d1a/f22 [0,4194304] 0 2024-03-30T04:28:15.584 INFO:tasks.workunit.client.0.smithi123.stdout:2/713: write d0/d1f/d5e/f4f [1805534,10026] 0 2024-03-30T04:28:15.585 INFO:tasks.workunit.client.1.smithi135.stdout:7/291: write d4/db/f1c [952896,77485] 0 2024-03-30T04:28:15.586 INFO:tasks.workunit.client.1.smithi135.stdout:6/183: mkdir d2/d36 0 2024-03-30T04:28:15.590 INFO:tasks.workunit.client.1.smithi135.stdout:0/284: mkdir d0/d13/d1d/d55 0 2024-03-30T04:28:15.591 INFO:tasks.workunit.client.1.smithi135.stdout:6/184: fsync d2/f9 0 2024-03-30T04:28:15.591 INFO:tasks.workunit.client.1.smithi135.stdout:3/216: dwrite d3/d6/fa [0,4194304] 0 2024-03-30T04:28:15.592 INFO:tasks.workunit.client.0.smithi123.stdout:2/714: mknod d0/d1f/d1c/d80/dbb/d97/cea 0 2024-03-30T04:28:15.592 INFO:tasks.workunit.client.1.smithi135.stdout:4/199: stat d10/d1a/d2b/f31 0 2024-03-30T04:28:15.594 INFO:tasks.workunit.client.1.smithi135.stdout:3/217: fdatasync d3/db/f19 0 2024-03-30T04:28:15.599 INFO:tasks.workunit.client.0.smithi123.stdout:6/707: dwrite d10/d82/d8e/dc7/d42/f87 [0,4194304] 0 2024-03-30T04:28:15.599 INFO:tasks.workunit.client.1.smithi135.stdout:6/185: creat d2/d2e/f37 x:0 0 0 2024-03-30T04:28:15.600 INFO:tasks.workunit.client.0.smithi123.stdout:2/715: rename d0/d1f/d1c/d39/d4b/fa0 to d0/d1f/d78/feb 0 2024-03-30T04:28:15.602 INFO:tasks.workunit.client.0.smithi123.stdout:2/716: write d0/d1f/fdd [392860,50567] 0 2024-03-30T04:28:15.603 INFO:tasks.workunit.client.1.smithi135.stdout:4/200: mknod d10/d1a/d42/d46/c47 0 2024-03-30T04:28:15.604 INFO:tasks.workunit.client.1.smithi135.stdout:0/285: link d0/d13/d1e/f46 d0/d5/f56 0 2024-03-30T04:28:15.604 INFO:tasks.workunit.client.1.smithi135.stdout:3/218: write d3/d10/d16/d3e/f21 [852386,80427] 0 2024-03-30T04:28:15.606 INFO:tasks.workunit.client.1.smithi135.stdout:6/186: write d2/d2e/f31 [720240,26406] 0 2024-03-30T04:28:15.608 INFO:tasks.workunit.client.0.smithi123.stdout:2/717: write d0/d1f/d27/d68/dd6/fe0 [1276486,96470] 0 2024-03-30T04:28:15.611 INFO:tasks.workunit.client.0.smithi123.stdout:2/718: readlink d0/d1f/d1c/d20/dc6/ldb 0 2024-03-30T04:28:15.614 INFO:tasks.workunit.client.1.smithi135.stdout:2/221: dwrite d4/d6/f16 [4194304,4194304] 0 2024-03-30T04:28:15.617 INFO:tasks.workunit.client.0.smithi123.stdout:8/603: truncate da/d27/d3c/d2d/d5c/f6f 2577406 0 2024-03-30T04:28:15.619 INFO:tasks.workunit.client.0.smithi123.stdout:8/604: chown da/dc/d14/f44 3242413 1 2024-03-30T04:28:15.619 INFO:tasks.workunit.client.1.smithi135.stdout:6/187: write d2/dd/d18/f22 [2549812,33061] 0 2024-03-30T04:28:15.620 INFO:tasks.workunit.client.0.smithi123.stdout:2/719: fsync d0/d1f/d78/feb 0 2024-03-30T04:28:15.622 INFO:tasks.workunit.client.0.smithi123.stdout:8/605: mknod da/d27/d56/d6c/cb4 0 2024-03-30T04:28:15.624 INFO:tasks.workunit.client.0.smithi123.stdout:2/720: readlink d0/d1f/d1c/d39/d4b/d4c/d5f/l73 0 2024-03-30T04:28:15.626 INFO:tasks.workunit.client.1.smithi135.stdout:0/286: symlink d0/d5/d9/d23/d2a/l57 0 2024-03-30T04:28:15.632 INFO:tasks.workunit.client.0.smithi123.stdout:7/624: dread d2/da/d11/f45 [0,4194304] 0 2024-03-30T04:28:15.634 INFO:tasks.workunit.client.0.smithi123.stdout:7/625: chown d2/dc/d49/d68/f9d 9046 1 2024-03-30T04:28:15.635 INFO:tasks.workunit.client.1.smithi135.stdout:6/188: mkdir d2/dd/d14/d38 0 2024-03-30T04:28:15.639 INFO:tasks.workunit.client.1.smithi135.stdout:3/219: getdents d3/d10/d16/d23 0 2024-03-30T04:28:15.641 INFO:tasks.workunit.client.0.smithi123.stdout:3/657: dread d13/d24/d31/d23/f76 [0,4194304] 0 2024-03-30T04:28:15.642 INFO:tasks.workunit.client.1.smithi135.stdout:0/287: rename d0/d13/d1d to d0/d5/d9/d23/d58 0 2024-03-30T04:28:15.642 INFO:tasks.workunit.client.1.smithi135.stdout:2/222: dwrite d4/d6/f1f [4194304,4194304] 0 2024-03-30T04:28:15.643 INFO:tasks.workunit.client.0.smithi123.stdout:2/721: dwrite d0/d1f/d16/fcc [0,4194304] 0 2024-03-30T04:28:15.645 INFO:tasks.workunit.client.1.smithi135.stdout:3/220: creat d3/d10/f44 x:0 0 0 2024-03-30T04:28:15.646 INFO:tasks.workunit.client.0.smithi123.stdout:2/722: fsync d0/d1f/d5e/fb9 0 2024-03-30T04:28:15.646 INFO:tasks.workunit.client.1.smithi135.stdout:2/223: write d4/d32/d12/f31 [788355,41346] 0 2024-03-30T04:28:15.648 INFO:tasks.workunit.client.0.smithi123.stdout:6/708: fsync d10/d82/d8e/dc7/d42/f87 0 2024-03-30T04:28:15.650 INFO:tasks.workunit.client.0.smithi123.stdout:7/626: dread d2/f87 [0,4194304] 0 2024-03-30T04:28:15.650 INFO:tasks.workunit.client.0.smithi123.stdout:6/709: fdatasync d10/d82/dc1/d45/faa 0 2024-03-30T04:28:15.651 INFO:tasks.workunit.client.1.smithi135.stdout:0/288: rename d0/d5/d10/f2b to d0/d5/d42/f59 0 2024-03-30T04:28:15.653 INFO:tasks.workunit.client.1.smithi135.stdout:6/189: creat d2/f39 x:0 0 0 2024-03-30T04:28:15.660 INFO:tasks.workunit.client.0.smithi123.stdout:7/627: read d2/da/d11/f1e [3075959,50340] 0 2024-03-30T04:28:15.661 INFO:tasks.workunit.client.0.smithi123.stdout:3/658: truncate d13/d24/d31/f64 427772 0 2024-03-30T04:28:15.665 INFO:tasks.workunit.client.1.smithi135.stdout:3/221: creat d3/f45 x:0 0 0 2024-03-30T04:28:15.666 INFO:tasks.workunit.client.0.smithi123.stdout:3/659: write d13/f80 [1442541,60696] 0 2024-03-30T04:28:15.666 INFO:tasks.workunit.client.0.smithi123.stdout:0/631: sync 2024-03-30T04:28:15.666 INFO:tasks.workunit.client.0.smithi123.stdout:5/563: sync 2024-03-30T04:28:15.667 INFO:tasks.workunit.client.1.smithi135.stdout:0/289: creat d0/d13/d1e/d3a/f5a x:0 0 0 2024-03-30T04:28:15.667 INFO:tasks.workunit.client.0.smithi123.stdout:7/628: creat d2/da/d11/d61/d55/fd4 x:0 0 0 2024-03-30T04:28:15.667 INFO:tasks.workunit.client.1.smithi135.stdout:4/201: dwrite d10/d1a/f22 [4194304,4194304] 0 2024-03-30T04:28:15.670 INFO:tasks.workunit.client.1.smithi135.stdout:4/202: fsync d10/d1a/f3a 0 2024-03-30T04:28:15.672 INFO:tasks.workunit.client.1.smithi135.stdout:0/290: read d0/d13/d1e/f49 [236955,100540] 0 2024-03-30T04:28:15.673 INFO:tasks.workunit.client.1.smithi135.stdout:4/203: write d10/d1a/f3a [426850,118225] 0 2024-03-30T04:28:15.675 INFO:tasks.workunit.client.1.smithi135.stdout:4/204: read fd [1126884,7458] 0 2024-03-30T04:28:15.677 INFO:tasks.workunit.client.1.smithi135.stdout:6/190: creat d2/dd/d14/f3a x:0 0 0 2024-03-30T04:28:15.679 INFO:tasks.workunit.client.1.smithi135.stdout:3/222: rename d3/d10/d16/d3e to d3/d6/d46 0 2024-03-30T04:28:15.679 INFO:tasks.workunit.client.1.smithi135.stdout:4/205: chown d10/d1a/f27 649 1 2024-03-30T04:28:15.679 INFO:tasks.workunit.client.0.smithi123.stdout:6/710: write d10/d82/d8e/dc7/d5b/fa8 [462753,93497] 0 2024-03-30T04:28:15.680 INFO:tasks.workunit.client.1.smithi135.stdout:6/191: write d2/dd/f17 [204885,70515] 0 2024-03-30T04:28:15.683 INFO:tasks.workunit.client.1.smithi135.stdout:4/206: fdatasync d10/d1a/f23 0 2024-03-30T04:28:15.683 INFO:tasks.workunit.client.1.smithi135.stdout:0/291: readlink d0/d5/d10/l18 0 2024-03-30T04:28:15.683 INFO:tasks.workunit.client.1.smithi135.stdout:3/223: dread - d3/d6/d22/f3d zero size 2024-03-30T04:28:15.684 INFO:tasks.workunit.client.0.smithi123.stdout:2/723: dwrite d0/d1f/d1c/f41 [0,4194304] 0 2024-03-30T04:28:15.686 INFO:tasks.workunit.client.0.smithi123.stdout:8/606: dread da/dc/f52 [0,4194304] 0 2024-03-30T04:28:15.689 INFO:tasks.workunit.client.0.smithi123.stdout:5/564: rename dd/d19/d1f/d35/d3a/fa7 to dd/d19/d1f/d35/d3a/d57/fb2 0 2024-03-30T04:28:15.690 INFO:tasks.workunit.client.1.smithi135.stdout:4/207: fdatasync d10/d1a/f1f 0 2024-03-30T04:28:15.693 INFO:tasks.workunit.client.1.smithi135.stdout:4/208: read f4 [637675,109054] 0 2024-03-30T04:28:15.694 INFO:tasks.workunit.client.0.smithi123.stdout:6/711: write f6 [991168,23445] 0 2024-03-30T04:28:15.694 INFO:tasks.workunit.client.1.smithi135.stdout:0/292: write d0/d5/d9/f15 [684873,31408] 0 2024-03-30T04:28:15.695 INFO:tasks.workunit.client.0.smithi123.stdout:8/607: mknod da/d27/d3c/cb5 0 2024-03-30T04:28:15.698 INFO:tasks.workunit.client.1.smithi135.stdout:6/192: symlink d2/l3b 0 2024-03-30T04:28:15.699 INFO:tasks.workunit.client.0.smithi123.stdout:6/712: fsync d10/f96 0 2024-03-30T04:28:15.700 INFO:tasks.workunit.client.1.smithi135.stdout:0/293: read - d0/d13/d1e/f4f zero size 2024-03-30T04:28:15.700 INFO:tasks.workunit.client.0.smithi123.stdout:7/629: dwrite d2/dc/d49/d68/f9d [0,4194304] 0 2024-03-30T04:28:15.701 INFO:tasks.workunit.client.1.smithi135.stdout:6/193: write d2/dd/d14/f3a [94785,38195] 0 2024-03-30T04:28:15.707 INFO:tasks.workunit.client.1.smithi135.stdout:4/209: dread d10/d1a/d2b/f31 [0,4194304] 0 2024-03-30T04:28:15.708 INFO:tasks.workunit.client.1.smithi135.stdout:4/210: stat d10 0 2024-03-30T04:28:15.708 INFO:tasks.workunit.client.0.smithi123.stdout:2/724: rmdir d0/d1f/d1c/d80/dbb/d97 39 2024-03-30T04:28:15.708 INFO:tasks.workunit.client.0.smithi123.stdout:5/565: mkdir dd/d19/d1f/d35/d3a/d57/db3 0 2024-03-30T04:28:15.708 INFO:tasks.workunit.client.0.smithi123.stdout:7/630: dread - d2/da/d22/d3b/f8f zero size 2024-03-30T04:28:15.709 INFO:tasks.workunit.client.0.smithi123.stdout:0/632: dread d6/dd/d18/d2f/f85 [0,4194304] 0 2024-03-30T04:28:15.712 INFO:tasks.workunit.client.1.smithi135.stdout:4/211: getdents d10/d1a/d32/d40 0 2024-03-30T04:28:15.714 INFO:tasks.workunit.client.0.smithi123.stdout:8/608: rename da/f11 to da/d99/fb6 0 2024-03-30T04:28:15.716 INFO:tasks.workunit.client.0.smithi123.stdout:6/713: mknod d10/d20/da2/cf2 0 2024-03-30T04:28:15.718 INFO:tasks.workunit.client.1.smithi135.stdout:4/212: symlink d10/d1a/d42/d46/l48 0 2024-03-30T04:28:15.719 INFO:tasks.workunit.client.0.smithi123.stdout:2/725: mkdir d0/d37/dec 0 2024-03-30T04:28:15.719 INFO:tasks.workunit.client.0.smithi123.stdout:0/633: creat d6/fcd x:0 0 0 2024-03-30T04:28:15.722 INFO:tasks.workunit.client.1.smithi135.stdout:4/213: mkdir d10/d1a/d32/d49 0 2024-03-30T04:28:15.723 INFO:tasks.workunit.client.0.smithi123.stdout:8/609: mkdir da/dc/d14/db7 0 2024-03-30T04:28:15.723 INFO:tasks.workunit.client.0.smithi123.stdout:2/726: stat d0/d1f/fdd 0 2024-03-30T04:28:15.723 INFO:tasks.workunit.client.0.smithi123.stdout:0/634: fsync d6/dd/d15/f2b 0 2024-03-30T04:28:15.725 INFO:tasks.workunit.client.1.smithi135.stdout:0/294: dwrite d0/d5/d9/f15 [0,4194304] 0 2024-03-30T04:28:15.728 INFO:tasks.workunit.client.0.smithi123.stdout:3/660: dread d13/fa1 [0,4194304] 0 2024-03-30T04:28:15.728 INFO:tasks.workunit.client.0.smithi123.stdout:2/727: creat d0/d37/d57/fed x:0 0 0 2024-03-30T04:28:15.729 INFO:tasks.workunit.client.1.smithi135.stdout:4/214: symlink d10/d1a/d42/d46/l4a 0 2024-03-30T04:28:15.730 INFO:tasks.workunit.client.1.smithi135.stdout:6/194: dwrite d2/f13 [0,4194304] 0 2024-03-30T04:28:15.731 INFO:tasks.workunit.client.0.smithi123.stdout:0/635: read d6/f9 [2430274,115160] 0 2024-03-30T04:28:15.734 INFO:tasks.workunit.client.1.smithi135.stdout:4/215: write d10/d1a/f2e [465167,91433] 0 2024-03-30T04:28:15.738 INFO:tasks.workunit.client.1.smithi135.stdout:0/295: rename d0/d1/f44 to d0/d13/d53/d54/f5b 0 2024-03-30T04:28:15.739 INFO:tasks.workunit.client.1.smithi135.stdout:4/216: chown d10/d1a/d32/l44 1 1 2024-03-30T04:28:15.741 INFO:tasks.workunit.client.1.smithi135.stdout:3/224: dread d3/d10/d16/f1b [0,4194304] 0 2024-03-30T04:28:15.741 INFO:tasks.workunit.client.0.smithi123.stdout:5/566: dwrite dd/d19/d1f/d3c/d72/d8c/fac [0,4194304] 0 2024-03-30T04:28:15.742 INFO:tasks.workunit.client.0.smithi123.stdout:3/661: chown d13/d24/d31/d23/d3a/d3c/c74 16159 1 2024-03-30T04:28:15.742 INFO:tasks.workunit.client.1.smithi135.stdout:6/195: creat d2/dd/d18/d21/f3c x:0 0 0 2024-03-30T04:28:15.744 INFO:tasks.workunit.client.1.smithi135.stdout:3/225: chown d3/l20 901628609 1 2024-03-30T04:28:15.745 INFO:tasks.workunit.client.1.smithi135.stdout:4/217: truncate d10/d1a/d42/d46/f45 332929 0 2024-03-30T04:28:15.746 INFO:tasks.workunit.client.0.smithi123.stdout:4/613: sync 2024-03-30T04:28:15.747 INFO:tasks.workunit.client.1.smithi135.stdout:4/218: fdatasync fd 0 2024-03-30T04:28:15.748 INFO:tasks.workunit.client.0.smithi123.stdout:6/714: dwrite d10/f9e [0,4194304] 0 2024-03-30T04:28:15.748 INFO:tasks.workunit.client.0.smithi123.stdout:0/636: symlink d6/dd/d15/d28/d4b/d8a/lce 0 2024-03-30T04:28:15.749 INFO:tasks.workunit.client.1.smithi135.stdout:3/226: write d3/d6/d22/f2e [631382,89961] 0 2024-03-30T04:28:15.753 INFO:tasks.workunit.client.1.smithi135.stdout:6/196: creat d2/d36/f3d x:0 0 0 2024-03-30T04:28:15.753 INFO:tasks.workunit.client.1.smithi135.stdout:0/296: read d0/d1/f29 [61573,89453] 0 2024-03-30T04:28:15.754 INFO:tasks.workunit.client.1.smithi135.stdout:3/227: read d3/d6/d46/f21 [144281,10773] 0 2024-03-30T04:28:15.756 INFO:tasks.workunit.client.1.smithi135.stdout:4/219: rmdir d10/d1a/d32/d49 0 2024-03-30T04:28:15.758 INFO:tasks.workunit.client.0.smithi123.stdout:3/662: fsync d13/f4a 0 2024-03-30T04:28:15.759 INFO:tasks.workunit.client.1.smithi135.stdout:3/228: rmdir d3/db 39 2024-03-30T04:28:15.760 INFO:tasks.workunit.client.1.smithi135.stdout:6/197: creat d2/dd/d14/d38/f3e x:0 0 0 2024-03-30T04:28:15.760 INFO:tasks.workunit.client.0.smithi123.stdout:7/631: truncate f0 3162543 0 2024-03-30T04:28:15.761 INFO:tasks.workunit.client.0.smithi123.stdout:5/567: read dd/d19/d1f/f67 [1341982,94090] 0 2024-03-30T04:28:15.761 INFO:tasks.workunit.client.0.smithi123.stdout:8/610: dread da/dc/f20 [0,4194304] 0 2024-03-30T04:28:15.763 INFO:tasks.workunit.client.0.smithi123.stdout:3/663: chown d13/d24/d39/d79/d90/lc1 7 1 2024-03-30T04:28:15.767 INFO:tasks.workunit.client.1.smithi135.stdout:3/229: mknod d3/db/c47 0 2024-03-30T04:28:15.767 INFO:tasks.workunit.client.1.smithi135.stdout:4/220: truncate d10/d1a/d32/f3e 1977608 0 2024-03-30T04:28:15.769 INFO:tasks.workunit.client.1.smithi135.stdout:4/221: fsync fb 0 2024-03-30T04:28:15.769 INFO:tasks.workunit.client.0.smithi123.stdout:7/632: mkdir d2/d4d/d80/d98/dd5 0 2024-03-30T04:28:15.769 INFO:tasks.workunit.client.0.smithi123.stdout:0/637: dread d6/f72 [0,4194304] 0 2024-03-30T04:28:15.770 INFO:tasks.workunit.client.0.smithi123.stdout:5/568: fdatasync dd/f15 0 2024-03-30T04:28:15.773 INFO:tasks.workunit.client.0.smithi123.stdout:3/664: chown d13/d24/d31/d23/c47 1 1 2024-03-30T04:28:15.773 INFO:tasks.workunit.client.0.smithi123.stdout:7/633: chown d2/da/d11/d34/d76 1 1 2024-03-30T04:28:15.774 INFO:tasks.workunit.client.1.smithi135.stdout:3/230: rmdir d3/d6 39 2024-03-30T04:28:15.774 INFO:tasks.workunit.client.0.smithi123.stdout:8/611: chown da/d27/d3c/d2d/f97 40008613 1 2024-03-30T04:28:15.776 INFO:tasks.workunit.client.0.smithi123.stdout:7/634: stat d2/dc/c4f 0 2024-03-30T04:28:15.777 INFO:tasks.workunit.client.1.smithi135.stdout:6/198: creat d2/dd/d14/f3f x:0 0 0 2024-03-30T04:28:15.779 INFO:tasks.workunit.client.1.smithi135.stdout:0/297: dwrite d0/d5/d10/f11 [0,4194304] 0 2024-03-30T04:28:15.779 INFO:tasks.workunit.client.0.smithi123.stdout:0/638: mkdir d6/d52/dcf 0 2024-03-30T04:28:15.781 INFO:tasks.workunit.client.1.smithi135.stdout:6/199: dread - d2/dd/d18/f25 zero size 2024-03-30T04:28:15.782 INFO:tasks.workunit.client.1.smithi135.stdout:4/222: creat d10/d1a/d32/d40/f4b x:0 0 0 2024-03-30T04:28:15.782 INFO:tasks.workunit.client.0.smithi123.stdout:5/569: mknod dd/d19/d1f/d35/d3a/cb4 0 2024-03-30T04:28:15.784 INFO:tasks.workunit.client.0.smithi123.stdout:5/570: stat dd/d19/d1f/d35 0 2024-03-30T04:28:15.785 INFO:tasks.workunit.client.1.smithi135.stdout:4/223: chown d10/d1a/d42/d46/l48 0 1 2024-03-30T04:28:15.785 INFO:tasks.workunit.client.1.smithi135.stdout:6/200: truncate d2/d36/f3d 789596 0 2024-03-30T04:28:15.786 INFO:tasks.workunit.client.0.smithi123.stdout:3/665: rename d13/d24/d31/f64 to d13/d24/d39/d79/db9/fd2 0 2024-03-30T04:28:15.788 INFO:tasks.workunit.client.1.smithi135.stdout:4/224: write d10/d1a/f2e [249646,107491] 0 2024-03-30T04:28:15.789 INFO:tasks.workunit.client.0.smithi123.stdout:7/635: mknod d2/d4d/d80/d98/dd5/cd6 0 2024-03-30T04:28:15.789 INFO:tasks.workunit.client.0.smithi123.stdout:6/715: dwrite d10/d82/dc1/f8b [0,4194304] 0 2024-03-30T04:28:15.791 INFO:tasks.workunit.client.1.smithi135.stdout:0/298: chown d0/d5/d10/f31 25035 1 2024-03-30T04:28:15.793 INFO:tasks.workunit.client.0.smithi123.stdout:0/639: rename d6/dd/d31/d35/d3a/db9 to d6/dd/d15/d98/dd0 0 2024-03-30T04:28:15.793 INFO:tasks.workunit.client.1.smithi135.stdout:0/299: chown d0/d5/d9/d23/d58/f4a 832 1 2024-03-30T04:28:15.794 INFO:tasks.workunit.client.1.smithi135.stdout:4/225: mknod d10/d1a/d3d/c4c 0 2024-03-30T04:28:15.797 INFO:tasks.workunit.client.1.smithi135.stdout:4/226: mkdir d10/d4d 0 2024-03-30T04:28:15.798 INFO:tasks.workunit.client.1.smithi135.stdout:4/227: fdatasync fa 0 2024-03-30T04:28:15.801 INFO:tasks.workunit.client.1.smithi135.stdout:4/228: creat d10/d4d/f4e x:0 0 0 2024-03-30T04:28:15.803 INFO:tasks.workunit.client.0.smithi123.stdout:3/666: link f11 d13/d24/d31/d23/d3a/d3c/db8/fd3 0 2024-03-30T04:28:15.803 INFO:tasks.workunit.client.0.smithi123.stdout:8/612: dwrite da/dc/d14/d9d/fb1 [0,4194304] 0 2024-03-30T04:28:15.805 INFO:tasks.workunit.client.1.smithi135.stdout:4/229: unlink d10/d1a/d42/d46/f45 0 2024-03-30T04:28:15.810 INFO:tasks.workunit.client.0.smithi123.stdout:0/640: rename d6/dd/l11 to d6/dd/d15/d98/d4e/d69/d6e/ld1 0 2024-03-30T04:28:15.812 INFO:tasks.workunit.client.0.smithi123.stdout:8/613: mkdir da/d27/d3c/d64/d78/db8 0 2024-03-30T04:28:15.814 INFO:tasks.workunit.client.1.smithi135.stdout:3/231: truncate d3/f11 4015739 0 2024-03-30T04:28:15.815 INFO:tasks.workunit.client.1.smithi135.stdout:6/201: write d2/dd/d14/f1e [1200010,39367] 0 2024-03-30T04:28:15.818 INFO:tasks.workunit.client.1.smithi135.stdout:0/300: write d0/d13/d1e/f49 [311178,88644] 0 2024-03-30T04:28:15.819 INFO:tasks.workunit.client.1.smithi135.stdout:8/228: sync 2024-03-30T04:28:15.819 INFO:tasks.workunit.client.1.smithi135.stdout:6/202: mkdir d2/dd/d40 0 2024-03-30T04:28:15.823 INFO:tasks.workunit.client.0.smithi123.stdout:6/716: dwrite d10/d82/d8e/fd1 [0,4194304] 0 2024-03-30T04:28:15.823 INFO:tasks.workunit.client.0.smithi123.stdout:5/571: dwrite dd/d19/d1f/d3c/d72/d8c/f9a [0,4194304] 0 2024-03-30T04:28:15.823 INFO:tasks.workunit.client.1.smithi135.stdout:8/229: chown l3 892473 1 2024-03-30T04:28:15.824 INFO:tasks.workunit.client.1.smithi135.stdout:3/232: dread d3/d6/d46/f1c [0,4194304] 0 2024-03-30T04:28:15.824 INFO:tasks.workunit.client.0.smithi123.stdout:7/636: dread d2/dc/d49/f6c [0,4194304] 0 2024-03-30T04:28:15.824 INFO:tasks.workunit.client.0.smithi123.stdout:3/667: getdents d13/d24/d31/d23/d3a/d3c/dab 0 2024-03-30T04:28:15.825 INFO:tasks.workunit.client.1.smithi135.stdout:0/301: fdatasync d0/d5/d9/f15 0 2024-03-30T04:28:15.825 INFO:tasks.workunit.client.1.smithi135.stdout:6/203: chown d2/dd/d18/d21/f3c 8 1 2024-03-30T04:28:15.826 INFO:tasks.workunit.client.0.smithi123.stdout:8/614: rename da/d27/d56/d6c/d6d/f72 to da/d27/d56/d6c/d96/fb9 0 2024-03-30T04:28:15.827 INFO:tasks.workunit.client.1.smithi135.stdout:3/233: stat d3/db/f34 0 2024-03-30T04:28:15.827 INFO:tasks.workunit.client.0.smithi123.stdout:7/637: chown d2/da/l16 8071 1 2024-03-30T04:28:15.827 INFO:tasks.workunit.client.0.smithi123.stdout:6/717: write d10/d82/d8e/dc7/d42/fe7 [4014249,17125] 0 2024-03-30T04:28:15.829 INFO:tasks.workunit.client.0.smithi123.stdout:5/572: readlink dd/d19/l49 0 2024-03-30T04:28:15.830 INFO:tasks.workunit.client.0.smithi123.stdout:5/573: chown dd 0 1 2024-03-30T04:28:15.830 INFO:tasks.workunit.client.1.smithi135.stdout:6/204: write d2/dd/d18/f22 [4017537,88951] 0 2024-03-30T04:28:15.831 INFO:tasks.workunit.client.1.smithi135.stdout:4/230: dwrite fb [0,4194304] 0 2024-03-30T04:28:15.832 INFO:tasks.workunit.client.1.smithi135.stdout:8/230: rename l1 to d6/l40 0 2024-03-30T04:28:15.833 INFO:tasks.workunit.client.1.smithi135.stdout:0/302: rmdir d0/d5/d10 39 2024-03-30T04:28:15.834 INFO:tasks.workunit.client.0.smithi123.stdout:6/718: mknod d10/d82/dc1/d6a/cf3 0 2024-03-30T04:28:15.835 INFO:tasks.workunit.client.1.smithi135.stdout:3/234: symlink d3/d6/l48 0 2024-03-30T04:28:15.838 INFO:tasks.workunit.client.0.smithi123.stdout:8/615: mknod da/d27/d3c/d2d/dae/cba 0 2024-03-30T04:28:15.839 INFO:tasks.workunit.client.1.smithi135.stdout:6/205: fdatasync d2/f4 0 2024-03-30T04:28:15.840 INFO:tasks.workunit.client.1.smithi135.stdout:4/231: creat d10/f4f x:0 0 0 2024-03-30T04:28:15.841 INFO:tasks.workunit.client.0.smithi123.stdout:5/574: unlink dd/d19/d1f/d68/l78 0 2024-03-30T04:28:15.843 INFO:tasks.workunit.client.1.smithi135.stdout:4/232: read d10/d1a/f21 [195815,11967] 0 2024-03-30T04:28:15.844 INFO:tasks.workunit.client.0.smithi123.stdout:6/719: creat d10/d82/dc1/d6a/d7a/de8/ff4 x:0 0 0 2024-03-30T04:28:15.844 INFO:tasks.workunit.client.0.smithi123.stdout:5/575: write dd/d19/d3e/d5b/d62/faa [1267996,55337] 0 2024-03-30T04:28:15.848 INFO:tasks.workunit.client.1.smithi135.stdout:6/206: rename d2/dd/d14/d30 to d2/dd/d14/d30/d41 22 2024-03-30T04:28:15.848 INFO:tasks.workunit.client.1.smithi135.stdout:3/235: symlink d3/l49 0 2024-03-30T04:28:15.849 INFO:tasks.workunit.client.0.smithi123.stdout:7/638: creat d2/da/d11/d61/d55/d59/db5/dab/dca/fd7 x:0 0 0 2024-03-30T04:28:15.849 INFO:tasks.workunit.client.1.smithi135.stdout:4/233: mknod d10/d4d/c50 0 2024-03-30T04:28:15.852 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:15 smithi135 ceph-mon[44364]: pgmap v41: 65 pgs: 65 active+clean; 1.7 GiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 56 MiB/s rd, 208 MiB/s wr, 434 op/s 2024-03-30T04:28:15.852 INFO:tasks.workunit.client.1.smithi135.stdout:3/236: mknod d3/c4a 0 2024-03-30T04:28:15.853 INFO:tasks.workunit.client.0.smithi123.stdout:5/576: mkdir dd/d19/d1f/d35/d3a/d57/db5 0 2024-03-30T04:28:15.853 INFO:tasks.workunit.client.0.smithi123.stdout:6/720: mkdir d10/d82/d8e/df5 0 2024-03-30T04:28:15.855 INFO:tasks.workunit.client.1.smithi135.stdout:4/234: rmdir d10/d1a 39 2024-03-30T04:28:15.857 INFO:tasks.workunit.client.1.smithi135.stdout:6/207: symlink d2/dd/d14/d30/l42 0 2024-03-30T04:28:15.857 INFO:tasks.workunit.client.0.smithi123.stdout:3/668: write d13/d24/f50 [1513451,40158] 0 2024-03-30T04:28:15.858 INFO:tasks.workunit.client.1.smithi135.stdout:6/208: fdatasync d2/f39 0 2024-03-30T04:28:15.859 INFO:tasks.workunit.client.0.smithi123.stdout:7/639: creat d2/da/d11/d61/d55/d94/fd8 x:0 0 0 2024-03-30T04:28:15.860 INFO:tasks.workunit.client.0.smithi123.stdout:5/577: symlink dd/d19/d1f/d35/d3a/d79/lb6 0 2024-03-30T04:28:15.861 INFO:tasks.workunit.client.1.smithi135.stdout:6/209: read - d2/dd/d14/d38/f3e zero size 2024-03-30T04:28:15.861 INFO:tasks.workunit.client.0.smithi123.stdout:8/616: write da/dc/f16 [292871,96893] 0 2024-03-30T04:28:15.863 INFO:tasks.workunit.client.0.smithi123.stdout:3/669: unlink d13/d24/d31/d23/d3a/d3c/d4e/l6d 0 2024-03-30T04:28:15.863 INFO:tasks.workunit.client.0.smithi123.stdout:8/617: readlink da/d27/l47 0 2024-03-30T04:28:15.864 INFO:tasks.workunit.client.0.smithi123.stdout:5/578: fsync dd/d19/d1f/d35/d3a/d79/f9b 0 2024-03-30T04:28:15.865 INFO:tasks.workunit.client.1.smithi135.stdout:4/235: symlink d10/d1a/d42/d46/l51 0 2024-03-30T04:28:15.866 INFO:tasks.workunit.client.1.smithi135.stdout:8/231: dread - d6/d8/d14/f2e zero size 2024-03-30T04:28:15.869 INFO:tasks.workunit.client.1.smithi135.stdout:8/232: write d6/d8/f31 [4654955,65570] 0 2024-03-30T04:28:15.869 INFO:tasks.workunit.client.0.smithi123.stdout:3/670: creat d13/d24/d39/d79/d90/fd4 x:0 0 0 2024-03-30T04:28:15.871 INFO:tasks.workunit.client.1.smithi135.stdout:0/303: truncate d0/f3 921196 0 2024-03-30T04:28:15.875 INFO:tasks.workunit.client.0.smithi123.stdout:3/671: read d13/d24/d31/d30/f9a [403809,54960] 0 2024-03-30T04:28:15.875 INFO:tasks.workunit.client.1.smithi135.stdout:8/233: mknod d6/d8/d15/d2a/c41 0 2024-03-30T04:28:15.877 INFO:tasks.workunit.client.1.smithi135.stdout:3/237: dwrite d3/db/fe [0,4194304] 0 2024-03-30T04:28:15.878 INFO:tasks.workunit.client.0.smithi123.stdout:5/579: rmdir dd/d19/d1f/d3c/d72/d8c/d9c 0 2024-03-30T04:28:15.881 INFO:tasks.workunit.client.1.smithi135.stdout:0/304: rename d0/d5/d42/f59 to d0/d19/f5c 0 2024-03-30T04:28:15.882 INFO:tasks.workunit.client.1.smithi135.stdout:6/210: dread d2/f4 [4194304,4194304] 0 2024-03-30T04:28:15.884 INFO:tasks.workunit.client.1.smithi135.stdout:8/234: unlink d6/f18 0 2024-03-30T04:28:15.888 INFO:tasks.workunit.client.0.smithi123.stdout:5/580: link dd/d19/d1f/d3c/f97 dd/d19/d1f/d3c/d72/fb7 0 2024-03-30T04:28:15.888 INFO:tasks.workunit.client.1.smithi135.stdout:3/238: write d3/d10/d16/f1e [4147267,59953] 0 2024-03-30T04:28:15.888 INFO:tasks.workunit.client.1.smithi135.stdout:0/305: mkdir d0/d5/d9/d23/d5d 0 2024-03-30T04:28:15.895 INFO:tasks.workunit.client.1.smithi135.stdout:0/306: creat d0/d1/f5e x:0 0 0 2024-03-30T04:28:15.895 INFO:tasks.workunit.client.0.smithi123.stdout:5/581: mkdir dd/d19/d1f/d35/db8 0 2024-03-30T04:28:15.895 INFO:tasks.workunit.client.0.smithi123.stdout:8/618: dwrite da/d27/d3c/d2d/f57 [0,4194304] 0 2024-03-30T04:28:15.897 INFO:tasks.workunit.client.1.smithi135.stdout:6/211: getdents d2 0 2024-03-30T04:28:15.899 INFO:tasks.workunit.client.1.smithi135.stdout:6/212: fsync d2/fb 0 2024-03-30T04:28:15.899 INFO:tasks.workunit.client.0.smithi123.stdout:7/640: dread d2/da/d22/d3b/f4b [0,4194304] 0 2024-03-30T04:28:15.900 INFO:tasks.workunit.client.1.smithi135.stdout:0/307: chown d0/d5/d9/d23/d5d 1 1 2024-03-30T04:28:15.900 INFO:tasks.workunit.client.1.smithi135.stdout:6/213: fdatasync d2/f11 0 2024-03-30T04:28:15.901 INFO:tasks.workunit.client.0.smithi123.stdout:5/582: symlink dd/d19/d1f/d35/d3a/d57/db5/lb9 0 2024-03-30T04:28:15.904 INFO:tasks.workunit.client.0.smithi123.stdout:1/713: sync 2024-03-30T04:28:15.904 INFO:tasks.workunit.client.0.smithi123.stdout:9/563: sync 2024-03-30T04:28:15.904 INFO:tasks.workunit.client.1.smithi135.stdout:6/214: write d2/dd/d18/f24 [1522165,20620] 0 2024-03-30T04:28:15.904 INFO:tasks.workunit.client.0.smithi123.stdout:5/583: fdatasync dd/d19/d3e/d5b/f8d 0 2024-03-30T04:28:15.905 INFO:tasks.workunit.client.1.smithi135.stdout:4/236: dread fa [0,4194304] 0 2024-03-30T04:28:15.911 INFO:tasks.workunit.client.0.smithi123.stdout:7/641: creat d2/fd9 x:0 0 0 2024-03-30T04:28:15.911 INFO:tasks.workunit.client.0.smithi123.stdout:3/672: write d13/d24/d31/d23/d3a/d3c/d66/f7f [3380285,107853] 0 2024-03-30T04:28:15.911 INFO:tasks.workunit.client.1.smithi135.stdout:6/215: read d2/f11 [2700911,112519] 0 2024-03-30T04:28:15.911 INFO:tasks.workunit.client.0.smithi123.stdout:9/564: chown d6/f7e 101 1 2024-03-30T04:28:15.914 INFO:tasks.workunit.client.0.smithi123.stdout:9/565: readlink d6/l3c 0 2024-03-30T04:28:15.914 INFO:tasks.workunit.client.1.smithi135.stdout:6/216: truncate d2/dd/d14/d38/f3e 1043829 0 2024-03-30T04:28:15.917 INFO:tasks.workunit.client.0.smithi123.stdout:1/714: write db/dd/d42/d46/d51/f55 [4213362,10762] 0 2024-03-30T04:28:15.917 INFO:tasks.workunit.client.0.smithi123.stdout:9/566: fdatasync d6/d1f/f4f 0 2024-03-30T04:28:15.918 INFO:tasks.workunit.client.0.smithi123.stdout:5/584: rename dd/c3f to dd/d19/d1f/cba 0 2024-03-30T04:28:15.923 INFO:tasks.workunit.client.0.smithi123.stdout:7/642: rename d2/da/d11/d34/d76/c9c to d2/dc/d49/d68/cda 0 2024-03-30T04:28:15.923 INFO:tasks.workunit.client.0.smithi123.stdout:1/715: stat db/dd/d42/d46/d5d/d62/d9e 0 2024-03-30T04:28:15.924 INFO:tasks.workunit.client.0.smithi123.stdout:9/567: symlink d6/d1f/d45/d97/lc1 0 2024-03-30T04:28:15.926 INFO:tasks.workunit.client.0.smithi123.stdout:8/619: dwrite da/d27/d56/f65 [4194304,4194304] 0 2024-03-30T04:28:15.927 INFO:tasks.workunit.client.1.smithi135.stdout:8/235: truncate d6/f12 606440 0 2024-03-30T04:28:15.929 INFO:tasks.workunit.client.1.smithi135.stdout:3/239: truncate d3/d6/d22/f2e 713922 0 2024-03-30T04:28:15.929 INFO:tasks.workunit.client.0.smithi123.stdout:5/585: creat dd/d19/d91/fbb x:0 0 0 2024-03-30T04:28:15.930 INFO:tasks.workunit.client.0.smithi123.stdout:7/643: rmdir d2/dc/d56 39 2024-03-30T04:28:15.934 INFO:tasks.workunit.client.1.smithi135.stdout:4/237: dwrite d10/f14 [0,4194304] 0 2024-03-30T04:28:15.935 INFO:tasks.workunit.client.0.smithi123.stdout:3/673: dread d13/d24/d31/d30/f9a [0,4194304] 0 2024-03-30T04:28:15.935 INFO:tasks.workunit.client.0.smithi123.stdout:1/716: truncate db/dd/d5f/d6d/d7e/ff0 345409 0 2024-03-30T04:28:15.935 INFO:tasks.workunit.client.0.smithi123.stdout:9/568: rename d6/d1f/d45/f90 to d6/d58/fc2 0 2024-03-30T04:28:15.936 INFO:tasks.workunit.client.1.smithi135.stdout:3/240: dread - d3/d6/f36 zero size 2024-03-30T04:28:15.937 INFO:tasks.workunit.client.1.smithi135.stdout:6/217: dread d2/dd/d14/f1e [0,4194304] 0 2024-03-30T04:28:15.939 INFO:tasks.workunit.client.1.smithi135.stdout:4/238: dread - d10/d1a/d32/d40/f4b zero size 2024-03-30T04:28:15.940 INFO:tasks.workunit.client.1.smithi135.stdout:6/218: fdatasync d2/dd/d14/f3a 0 2024-03-30T04:28:15.940 INFO:tasks.workunit.client.0.smithi123.stdout:8/620: mknod da/dc/d14/cbb 0 2024-03-30T04:28:15.941 INFO:tasks.workunit.client.1.smithi135.stdout:6/219: chown d2/dd/d14 228586890 1 2024-03-30T04:28:15.942 INFO:tasks.workunit.client.1.smithi135.stdout:3/241: unlink d3/db/l28 0 2024-03-30T04:28:15.942 INFO:tasks.workunit.client.1.smithi135.stdout:0/308: dread d0/d19/d2d/f38 [0,4194304] 0 2024-03-30T04:28:15.945 INFO:tasks.workunit.client.1.smithi135.stdout:0/309: chown d0/d1/d16/f28 627237673 1 2024-03-30T04:28:15.945 INFO:tasks.workunit.client.1.smithi135.stdout:6/220: chown d2/dd/c2a 43268711 1 2024-03-30T04:28:15.947 INFO:tasks.workunit.client.1.smithi135.stdout:3/242: write d3/d6/d22/d33/f3a [16491,66095] 0 2024-03-30T04:28:15.949 INFO:tasks.workunit.client.0.smithi123.stdout:3/674: creat d13/d24/d31/d23/d63/fd5 x:0 0 0 2024-03-30T04:28:15.949 INFO:tasks.workunit.client.1.smithi135.stdout:3/243: chown d3/d6/cd 19 1 2024-03-30T04:28:15.950 INFO:tasks.workunit.client.0.smithi123.stdout:1/717: mkdir db/dd/d42/d46/d56/da2/df1 0 2024-03-30T04:28:15.950 INFO:tasks.workunit.client.0.smithi123.stdout:9/569: truncate d6/d1f/d72/d7d/fbe 671466 0 2024-03-30T04:28:15.951 INFO:tasks.workunit.client.1.smithi135.stdout:4/239: creat d10/d1a/f52 x:0 0 0 2024-03-30T04:28:15.953 INFO:tasks.workunit.client.0.smithi123.stdout:8/621: fsync da/dc/f52 0 2024-03-30T04:28:15.954 INFO:tasks.workunit.client.1.smithi135.stdout:0/310: rmdir d0/d5/d10 39 2024-03-30T04:28:15.956 INFO:tasks.workunit.client.0.smithi123.stdout:3/675: chown d13/d24/d31/d30/c58 12 1 2024-03-30T04:28:15.956 INFO:tasks.workunit.client.1.smithi135.stdout:6/221: dread d2/d36/f3d [0,4194304] 0 2024-03-30T04:28:15.958 INFO:tasks.workunit.client.0.smithi123.stdout:7/644: dwrite d2/da/f71 [0,4194304] 0 2024-03-30T04:28:15.961 INFO:tasks.workunit.client.0.smithi123.stdout:5/586: dwrite dd/d19/d1f/d3c/f92 [0,4194304] 0 2024-03-30T04:28:15.964 INFO:tasks.workunit.client.0.smithi123.stdout:5/587: write dd/f1d [3525202,32643] 0 2024-03-30T04:28:15.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:15 smithi123 ceph-mon[35245]: pgmap v41: 65 pgs: 65 active+clean; 1.7 GiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 56 MiB/s rd, 208 MiB/s wr, 434 op/s 2024-03-30T04:28:15.968 INFO:tasks.workunit.client.0.smithi123.stdout:7/645: mkdir d2/da/d11/d34/d76/d91/ddb 0 2024-03-30T04:28:15.968 INFO:tasks.workunit.client.1.smithi135.stdout:4/240: symlink d10/d1a/d42/l53 0 2024-03-30T04:28:15.968 INFO:tasks.workunit.client.1.smithi135.stdout:3/244: getdents d3/d10/d16 0 2024-03-30T04:28:15.972 INFO:tasks.workunit.client.0.smithi123.stdout:9/570: stat d6/d1f/d45/d4c/lb2 0 2024-03-30T04:28:15.973 INFO:tasks.workunit.client.1.smithi135.stdout:4/241: write f4 [4681153,112910] 0 2024-03-30T04:28:15.974 INFO:tasks.workunit.client.0.smithi123.stdout:3/676: creat d13/d24/fd6 x:0 0 0 2024-03-30T04:28:15.974 INFO:tasks.workunit.client.1.smithi135.stdout:3/245: creat d3/d10/d16/d23/f4b x:0 0 0 2024-03-30T04:28:15.978 INFO:tasks.workunit.client.1.smithi135.stdout:3/246: truncate d3/f45 863547 0 2024-03-30T04:28:15.981 INFO:tasks.workunit.client.1.smithi135.stdout:4/242: truncate d10/d1a/f22 9277762 0 2024-03-30T04:28:15.984 INFO:tasks.workunit.client.1.smithi135.stdout:3/247: mkdir d3/d6/d46/d4c 0 2024-03-30T04:28:15.987 INFO:tasks.workunit.client.1.smithi135.stdout:3/248: chown l1 115832174 1 2024-03-30T04:28:15.988 INFO:tasks.workunit.client.1.smithi135.stdout:6/222: dwrite d2/f39 [0,4194304] 0 2024-03-30T04:28:15.988 INFO:tasks.workunit.client.1.smithi135.stdout:0/311: write d0/d13/d1e/f46 [828346,70684] 0 2024-03-30T04:28:15.988 INFO:tasks.workunit.client.0.smithi123.stdout:5/588: link dd/d19/d1f/d68/f73 dd/d19/fbc 0 2024-03-30T04:28:15.989 INFO:tasks.workunit.client.1.smithi135.stdout:8/236: dread d6/f11 [0,4194304] 0 2024-03-30T04:28:15.989 INFO:tasks.workunit.client.1.smithi135.stdout:4/243: write d10/d1a/d32/f3e [1406410,82304] 0 2024-03-30T04:28:15.989 INFO:tasks.workunit.client.0.smithi123.stdout:3/677: truncate fe 411731 0 2024-03-30T04:28:15.991 INFO:tasks.workunit.client.0.smithi123.stdout:8/622: dwrite da/dc/da9/fad [0,4194304] 0 2024-03-30T04:28:15.991 INFO:tasks.workunit.client.0.smithi123.stdout:9/571: creat d6/d1f/d72/fc3 x:0 0 0 2024-03-30T04:28:15.994 INFO:tasks.workunit.client.0.smithi123.stdout:1/718: dwrite db/dd/d42/d46/d51/d72/fa6 [0,4194304] 0 2024-03-30T04:28:15.997 INFO:tasks.workunit.client.1.smithi135.stdout:3/249: mknod d3/d10/d16/d23/c4d 0 2024-03-30T04:28:15.997 INFO:tasks.workunit.client.1.smithi135.stdout:6/223: mkdir d2/dd/d14/d30/d43 0 2024-03-30T04:28:15.998 INFO:tasks.workunit.client.0.smithi123.stdout:3/678: mkdir d13/d24/d31/d30/dd7 0 2024-03-30T04:28:15.999 INFO:tasks.workunit.client.0.smithi123.stdout:8/623: read - da/d27/d95/fab zero size 2024-03-30T04:28:16.000 INFO:tasks.workunit.client.0.smithi123.stdout:7/646: write f0 [377683,67514] 0 2024-03-30T04:28:16.000 INFO:tasks.workunit.client.0.smithi123.stdout:5/589: creat dd/d19/d1f/d3c/fbd x:0 0 0 2024-03-30T04:28:16.002 INFO:tasks.workunit.client.1.smithi135.stdout:8/237: chown d6/d8/d15/d2a/d35 158634 1 2024-03-30T04:28:16.005 INFO:tasks.workunit.client.0.smithi123.stdout:7/647: fdatasync d2/dc/d49/fcb 0 2024-03-30T04:28:16.006 INFO:tasks.workunit.client.0.smithi123.stdout:9/572: fsync d6/d1f/f2d 0 2024-03-30T04:28:16.007 INFO:tasks.workunit.client.1.smithi135.stdout:0/312: creat d0/d5/d9/d23/d58/d55/f5f x:0 0 0 2024-03-30T04:28:16.007 INFO:tasks.workunit.client.1.smithi135.stdout:6/224: fsync d2/f11 0 2024-03-30T04:28:16.008 INFO:tasks.workunit.client.0.smithi123.stdout:1/719: symlink db/dd/d5f/dae/lf2 0 2024-03-30T04:28:16.010 INFO:tasks.workunit.client.1.smithi135.stdout:0/313: stat d0/d19/d2d/d45 0 2024-03-30T04:28:16.010 INFO:tasks.workunit.client.0.smithi123.stdout:9/573: chown d6/d1f/d45/d4c/lb2 18719153 1 2024-03-30T04:28:16.014 INFO:tasks.workunit.client.1.smithi135.stdout:0/314: chown d0/d13/d53/d54/f5b 34 1 2024-03-30T04:28:16.014 INFO:tasks.workunit.client.1.smithi135.stdout:6/225: creat d2/d36/f44 x:0 0 0 2024-03-30T04:28:16.015 INFO:tasks.workunit.client.0.smithi123.stdout:8/624: creat da/d27/d3c/d2d/d74/fbc x:0 0 0 2024-03-30T04:28:16.015 INFO:tasks.workunit.client.1.smithi135.stdout:8/238: creat d6/d8/f42 x:0 0 0 2024-03-30T04:28:16.017 INFO:tasks.workunit.client.1.smithi135.stdout:6/226: readlink d2/dd/d14/d30/l42 0 2024-03-30T04:28:16.017 INFO:tasks.workunit.client.0.smithi123.stdout:8/625: stat da/fb 0 2024-03-30T04:28:16.022 INFO:tasks.workunit.client.0.smithi123.stdout:8/626: unlink da/d27/l35 0 2024-03-30T04:28:16.023 INFO:tasks.workunit.client.0.smithi123.stdout:1/720: read db/dd/d42/d46/d39/f92 [3273222,43936] 0 2024-03-30T04:28:16.024 INFO:tasks.workunit.client.1.smithi135.stdout:0/315: rename d0/d5/d9/d23/d3e to d0/d5/d9/d23/d2a/d60 0 2024-03-30T04:28:16.026 INFO:tasks.workunit.client.1.smithi135.stdout:3/250: rmdir d3/d10 39 2024-03-30T04:28:16.028 INFO:tasks.workunit.client.0.smithi123.stdout:8/627: write da/d27/d3c/d2d/f2e [3653752,31455] 0 2024-03-30T04:28:16.029 INFO:tasks.workunit.client.0.smithi123.stdout:7/648: getdents d2/dc/d7f 0 2024-03-30T04:28:16.029 INFO:tasks.workunit.client.1.smithi135.stdout:6/227: dread d2/f8 [0,4194304] 0 2024-03-30T04:28:16.036 INFO:tasks.workunit.client.0.smithi123.stdout:7/649: unlink d2/da/d11/d61/d55/d59/db5/dab/fb9 0 2024-03-30T04:28:16.038 INFO:tasks.workunit.client.0.smithi123.stdout:7/650: chown d2/dc/d49/d68 229 1 2024-03-30T04:28:16.039 INFO:tasks.workunit.client.0.smithi123.stdout:8/628: getdents da 0 2024-03-30T04:28:16.040 INFO:tasks.workunit.client.1.smithi135.stdout:9/197: sync 2024-03-30T04:28:16.042 INFO:tasks.workunit.client.0.smithi123.stdout:7/651: dread - d2/da/d11/fb4 zero size 2024-03-30T04:28:16.043 INFO:tasks.workunit.client.0.smithi123.stdout:8/629: symlink da/d27/d3c/d64/lbd 0 2024-03-30T04:28:16.043 INFO:tasks.workunit.client.0.smithi123.stdout:3/679: dwrite d13/d24/d31/f4b [0,4194304] 0 2024-03-30T04:28:16.045 INFO:tasks.workunit.client.1.smithi135.stdout:8/239: dwrite d6/d8/f20 [0,4194304] 0 2024-03-30T04:28:16.047 INFO:tasks.workunit.client.0.smithi123.stdout:8/630: chown da/dc/c25 0 1 2024-03-30T04:28:16.050 INFO:tasks.workunit.client.0.smithi123.stdout:5/590: dread dd/d19/d3e/f63 [4194304,4194304] 0 2024-03-30T04:28:16.052 INFO:tasks.workunit.client.0.smithi123.stdout:7/652: read d2/da/d11/d34/d76/d91/f9a [92445,113611] 0 2024-03-30T04:28:16.052 INFO:tasks.workunit.client.0.smithi123.stdout:8/631: dread - da/d27/fb3 zero size 2024-03-30T04:28:16.054 INFO:tasks.workunit.client.0.smithi123.stdout:1/721: dwrite db/dd/d5f/d6d/fc1 [0,4194304] 0 2024-03-30T04:28:16.054 INFO:tasks.workunit.client.0.smithi123.stdout:3/680: read d13/d24/d31/d23/d3a/d3c/d4e/d5d/f8e [136132,100347] 0 2024-03-30T04:28:16.056 INFO:tasks.workunit.client.1.smithi135.stdout:4/244: dwrite d10/d1a/d2b/f31 [0,4194304] 0 2024-03-30T04:28:16.058 INFO:tasks.workunit.client.0.smithi123.stdout:5/591: creat dd/d19/d5f/d99/fbe x:0 0 0 2024-03-30T04:28:16.059 INFO:tasks.workunit.client.0.smithi123.stdout:1/722: dread - db/dd/d42/d46/d51/d72/fdf zero size 2024-03-30T04:28:16.059 INFO:tasks.workunit.client.0.smithi123.stdout:3/681: symlink d13/d24/d31/d23/d3a/ld8 0 2024-03-30T04:28:16.061 INFO:tasks.workunit.client.1.smithi135.stdout:6/228: truncate d2/dd/d14/d38/f3e 766414 0 2024-03-30T04:28:16.063 INFO:tasks.workunit.client.1.smithi135.stdout:8/240: symlink d6/d8/d15/l43 0 2024-03-30T04:28:16.063 INFO:tasks.workunit.client.1.smithi135.stdout:0/316: dwrite d0/d1/f5e [0,4194304] 0 2024-03-30T04:28:16.063 INFO:tasks.workunit.client.1.smithi135.stdout:9/198: link d7/d1b/l9 d7/l3f 0 2024-03-30T04:28:16.065 INFO:tasks.workunit.client.0.smithi123.stdout:3/682: write d13/f80 [3615790,118638] 0 2024-03-30T04:28:16.065 INFO:tasks.workunit.client.0.smithi123.stdout:7/653: mknod d2/da/d11/d61/d55/d59/db5/dab/dca/cdc 0 2024-03-30T04:28:16.066 INFO:tasks.workunit.client.1.smithi135.stdout:3/251: dwrite d3/d10/f14 [0,4194304] 0 2024-03-30T04:28:16.066 INFO:tasks.workunit.client.1.smithi135.stdout:6/229: write d2/d36/f44 [953598,30496] 0 2024-03-30T04:28:16.068 INFO:tasks.workunit.client.0.smithi123.stdout:5/592: symlink dd/d19/d1f/d35/d3a/d57/lbf 0 2024-03-30T04:28:16.068 INFO:tasks.workunit.client.0.smithi123.stdout:7/654: fsync d2/dc/d49/dbc/fc5 0 2024-03-30T04:28:16.070 INFO:tasks.workunit.client.1.smithi135.stdout:3/252: readlink d3/d6/d22/l2f 0 2024-03-30T04:28:16.074 INFO:tasks.workunit.client.1.smithi135.stdout:9/199: creat d7/d1b/f40 x:0 0 0 2024-03-30T04:28:16.077 INFO:tasks.workunit.client.1.smithi135.stdout:8/241: write d6/d8/f31 [1079076,109813] 0 2024-03-30T04:28:16.080 INFO:tasks.workunit.client.1.smithi135.stdout:9/200: mkdir d7/d41 0 2024-03-30T04:28:16.083 INFO:tasks.workunit.client.0.smithi123.stdout:7/655: unlink d2/dc/d56/f60 0 2024-03-30T04:28:16.083 INFO:tasks.workunit.client.0.smithi123.stdout:1/723: rename db/d1b/c50 to db/dd/d42/d46/d39/cf3 0 2024-03-30T04:28:16.085 INFO:tasks.workunit.client.1.smithi135.stdout:3/253: mknod d3/d10/d16/d41/c4e 0 2024-03-30T04:28:16.088 INFO:tasks.workunit.client.1.smithi135.stdout:0/317: link d0/d5/d9/d23/d2a/d60/f48 d0/d13/d1e/d3a/f61 0 2024-03-30T04:28:16.088 INFO:tasks.workunit.client.1.smithi135.stdout:3/254: dread - d3/d10/d16/d23/f27 zero size 2024-03-30T04:28:16.088 INFO:tasks.workunit.client.1.smithi135.stdout:8/242: fdatasync d6/d8/f39 0 2024-03-30T04:28:16.089 INFO:tasks.workunit.client.1.smithi135.stdout:9/201: symlink d7/d41/l42 0 2024-03-30T04:28:16.092 INFO:tasks.workunit.client.1.smithi135.stdout:9/202: chown d7/l24 514 1 2024-03-30T04:28:16.093 INFO:tasks.workunit.client.0.smithi123.stdout:3/683: rename d13/d24/d31/d23/c47 to d13/d24/d39/cd9 0 2024-03-30T04:28:16.094 INFO:tasks.workunit.client.1.smithi135.stdout:8/243: write f0 [301109,77420] 0 2024-03-30T04:28:16.094 INFO:tasks.workunit.client.1.smithi135.stdout:0/318: fsync d0/d5/d10/f11 0 2024-03-30T04:28:16.096 INFO:tasks.workunit.client.1.smithi135.stdout:3/255: truncate d3/d6/d22/f3d 81122 0 2024-03-30T04:28:16.098 INFO:tasks.workunit.client.1.smithi135.stdout:9/203: creat d7/d41/f43 x:0 0 0 2024-03-30T04:28:16.099 INFO:tasks.workunit.client.1.smithi135.stdout:0/319: creat d0/d1/d16/f62 x:0 0 0 2024-03-30T04:28:16.100 INFO:tasks.workunit.client.1.smithi135.stdout:9/204: dread - d7/d1b/d37/f39 zero size 2024-03-30T04:28:16.101 INFO:tasks.workunit.client.1.smithi135.stdout:8/244: truncate d6/d23/f27 9073474 0 2024-03-30T04:28:16.101 INFO:tasks.workunit.client.1.smithi135.stdout:3/256: symlink d3/d10/d16/l4f 0 2024-03-30T04:28:16.101 INFO:tasks.workunit.client.0.smithi123.stdout:3/684: write d13/d24/d31/d23/f88 [2753447,75454] 0 2024-03-30T04:28:16.102 INFO:tasks.workunit.client.1.smithi135.stdout:3/257: stat d3/f3c 0 2024-03-30T04:28:16.103 INFO:tasks.workunit.client.1.smithi135.stdout:9/205: creat d7/f44 x:0 0 0 2024-03-30T04:28:16.104 INFO:tasks.workunit.client.1.smithi135.stdout:3/258: chown d3/d10/l32 25912965 1 2024-03-30T04:28:16.105 INFO:tasks.workunit.client.1.smithi135.stdout:0/320: mkdir d0/d5/d9/d23/d58/d63 0 2024-03-30T04:28:16.106 INFO:tasks.workunit.client.1.smithi135.stdout:8/245: write d6/d8/f17 [412470,80907] 0 2024-03-30T04:28:16.106 INFO:tasks.workunit.client.1.smithi135.stdout:9/206: write d7/d29/f3e [775571,15180] 0 2024-03-30T04:28:16.109 INFO:tasks.workunit.client.0.smithi123.stdout:3/685: write d13/fa1 [1273579,119316] 0 2024-03-30T04:28:16.110 INFO:tasks.workunit.client.1.smithi135.stdout:9/207: chown d7/d41/f43 9184176 1 2024-03-30T04:28:16.114 INFO:tasks.workunit.client.1.smithi135.stdout:8/246: stat d6/d8/f1c 0 2024-03-30T04:28:16.114 INFO:tasks.workunit.client.1.smithi135.stdout:6/230: write d2/dd/d18/f24 [4194055,14976] 0 2024-03-30T04:28:16.115 INFO:tasks.workunit.client.1.smithi135.stdout:3/259: rename d3/d6 to d3/d50 0 2024-03-30T04:28:16.115 INFO:tasks.workunit.client.0.smithi123.stdout:8/632: dread da/fb [0,4194304] 0 2024-03-30T04:28:16.115 INFO:tasks.workunit.client.0.smithi123.stdout:1/724: dwrite db/dd/d42/d46/d5d/d62/d9e/fc3 [0,4194304] 0 2024-03-30T04:28:16.116 INFO:tasks.workunit.client.0.smithi123.stdout:5/593: dwrite dd/d19/d1f/d35/d3a/d57/f65 [4194304,4194304] 0 2024-03-30T04:28:16.116 INFO:tasks.workunit.client.1.smithi135.stdout:3/260: write d3/f9 [3658405,31119] 0 2024-03-30T04:28:16.118 INFO:tasks.workunit.client.1.smithi135.stdout:9/208: mknod d7/d41/c45 0 2024-03-30T04:28:16.119 INFO:tasks.workunit.client.1.smithi135.stdout:6/231: mkdir d2/d45 0 2024-03-30T04:28:16.120 INFO:tasks.workunit.client.0.smithi123.stdout:8/633: unlink da/d27/d3c/d2d/f97 0 2024-03-30T04:28:16.121 INFO:tasks.workunit.client.0.smithi123.stdout:3/686: symlink d13/d24/d31/d23/d3a/d3c/d4e/d5d/da5/lda 0 2024-03-30T04:28:16.124 INFO:tasks.workunit.client.0.smithi123.stdout:5/594: symlink dd/d19/d1f/d68/lc0 0 2024-03-30T04:28:16.126 INFO:tasks.workunit.client.1.smithi135.stdout:0/321: dread d0/d1/fa [0,4194304] 0 2024-03-30T04:28:16.126 INFO:tasks.workunit.client.1.smithi135.stdout:9/209: creat d7/d29/f46 x:0 0 0 2024-03-30T04:28:16.128 INFO:tasks.workunit.client.0.smithi123.stdout:8/634: write da/d99/fb6 [6991502,112744] 0 2024-03-30T04:28:16.128 INFO:tasks.workunit.client.0.smithi123.stdout:1/725: truncate db/dd/d5f/d6d/d85/d89/d90/d94/fe9 923305 0 2024-03-30T04:28:16.129 INFO:tasks.workunit.client.0.smithi123.stdout:2/728: sync 2024-03-30T04:28:16.129 INFO:tasks.workunit.client.0.smithi123.stdout:4/614: sync 2024-03-30T04:28:16.130 INFO:tasks.workunit.client.1.smithi135.stdout:9/210: symlink d7/d1b/l47 0 2024-03-30T04:28:16.130 INFO:tasks.workunit.client.0.smithi123.stdout:5/595: creat dd/d19/d1f/d35/db8/fc1 x:0 0 0 2024-03-30T04:28:16.131 INFO:tasks.workunit.client.1.smithi135.stdout:0/322: mknod d0/d5/d9/d23/d58/d3b/c64 0 2024-03-30T04:28:16.134 INFO:tasks.workunit.client.0.smithi123.stdout:8/635: rmdir da/d27/d3c/d64 39 2024-03-30T04:28:16.134 INFO:tasks.workunit.client.0.smithi123.stdout:4/615: fdatasync d2/d26/fbd 0 2024-03-30T04:28:16.135 INFO:tasks.workunit.client.0.smithi123.stdout:1/726: write db/dd/d42/d46/d56/da2/fb0 [2259788,20039] 0 2024-03-30T04:28:16.139 INFO:tasks.workunit.client.0.smithi123.stdout:3/687: fdatasync d13/d24/d31/d23/f88 0 2024-03-30T04:28:16.140 INFO:tasks.workunit.client.1.smithi135.stdout:0/323: creat d0/d5/d9/d23/d2a/f65 x:0 0 0 2024-03-30T04:28:16.140 INFO:tasks.workunit.client.1.smithi135.stdout:3/261: dwrite d3/f24 [4194304,4194304] 0 2024-03-30T04:28:16.141 INFO:tasks.workunit.client.0.smithi123.stdout:5/596: mkdir dd/d19/d5f/dc2 0 2024-03-30T04:28:16.143 INFO:tasks.workunit.client.0.smithi123.stdout:2/729: mknod d0/d1f/d1c/d40/cee 0 2024-03-30T04:28:16.143 INFO:tasks.workunit.client.1.smithi135.stdout:0/324: unlink d0/d1/d16/f28 0 2024-03-30T04:28:16.145 INFO:tasks.workunit.client.1.smithi135.stdout:9/211: dread d7/d29/f3e [0,4194304] 0 2024-03-30T04:28:16.146 INFO:tasks.workunit.client.0.smithi123.stdout:4/616: creat d2/d4/d7/d63/db5/fd6 x:0 0 0 2024-03-30T04:28:16.146 INFO:tasks.workunit.client.1.smithi135.stdout:0/325: fdatasync d0/d5/d9/d23/d58/f4a 0 2024-03-30T04:28:16.147 INFO:tasks.workunit.client.0.smithi123.stdout:2/730: chown d0/d1f/d1c/d80/da8/fae 20853 1 2024-03-30T04:28:16.148 INFO:tasks.workunit.client.0.smithi123.stdout:8/636: mknod da/d27/d56/d6c/d6d/cbe 0 2024-03-30T04:28:16.152 INFO:tasks.workunit.client.0.smithi123.stdout:8/637: write da/dc/f13 [8317941,65231] 0 2024-03-30T04:28:16.152 INFO:tasks.workunit.client.0.smithi123.stdout:5/597: mknod dd/d19/d5f/d99/cc3 0 2024-03-30T04:28:16.152 INFO:tasks.workunit.client.0.smithi123.stdout:7/656: write d2/dc/f17 [1975025,119046] 0 2024-03-30T04:28:16.152 INFO:tasks.workunit.client.0.smithi123.stdout:1/727: mknod db/dd/d5f/d6d/d85/d89/d90/cf4 0 2024-03-30T04:28:16.155 INFO:tasks.workunit.client.0.smithi123.stdout:3/688: read d13/f4a [257995,55361] 0 2024-03-30T04:28:16.156 INFO:tasks.workunit.client.0.smithi123.stdout:4/617: rmdir d2/d34 39 2024-03-30T04:28:16.156 INFO:tasks.workunit.client.1.smithi135.stdout:4/245: dread d10/d1a/f1f [0,4194304] 0 2024-03-30T04:28:16.157 INFO:tasks.workunit.client.1.smithi135.stdout:0/326: getdents d0/d5 0 2024-03-30T04:28:16.158 INFO:tasks.workunit.client.1.smithi135.stdout:0/327: chown d0/d5 2243237 1 2024-03-30T04:28:16.159 INFO:tasks.workunit.client.0.smithi123.stdout:1/728: readlink db/dd/d42/d46/l6a 0 2024-03-30T04:28:16.161 INFO:tasks.workunit.client.0.smithi123.stdout:7/657: creat d2/dc/d56/fdd x:0 0 0 2024-03-30T04:28:16.162 INFO:tasks.workunit.client.1.smithi135.stdout:8/247: write d6/d8/f1d [692250,30311] 0 2024-03-30T04:28:16.163 INFO:tasks.workunit.client.1.smithi135.stdout:4/246: creat d10/d1a/d42/f54 x:0 0 0 2024-03-30T04:28:16.164 INFO:tasks.workunit.client.1.smithi135.stdout:0/328: creat d0/d5/d9/d23/f66 x:0 0 0 2024-03-30T04:28:16.165 INFO:tasks.workunit.client.0.smithi123.stdout:2/731: unlink d0/d1f/d1c/d40/f43 0 2024-03-30T04:28:16.167 INFO:tasks.workunit.client.1.smithi135.stdout:8/248: write d6/d8/f1c [1063772,122168] 0 2024-03-30T04:28:16.167 INFO:tasks.workunit.client.0.smithi123.stdout:3/689: fsync d13/d24/d39/d79/db9/fd2 0 2024-03-30T04:28:16.168 INFO:tasks.workunit.client.1.smithi135.stdout:6/232: creat d2/dd/d14/d2b/f46 x:0 0 0 2024-03-30T04:28:16.169 INFO:tasks.workunit.client.1.smithi135.stdout:4/247: mknod d10/d1a/c55 0 2024-03-30T04:28:16.171 INFO:tasks.workunit.client.0.smithi123.stdout:8/638: truncate da/d27/d3c/d2d/d5c/f6f 1281305 0 2024-03-30T04:28:16.173 INFO:tasks.workunit.client.1.smithi135.stdout:6/233: chown d2/f39 4002232 1 2024-03-30T04:28:16.174 INFO:tasks.workunit.client.1.smithi135.stdout:8/249: unlink d6/d8/l2f 0 2024-03-30T04:28:16.175 INFO:tasks.workunit.client.0.smithi123.stdout:2/732: mknod d0/d37/cef 0 2024-03-30T04:28:16.175 INFO:tasks.workunit.client.1.smithi135.stdout:0/329: mkdir d0/d5/d9/d23/d58/d63/d67 0 2024-03-30T04:28:16.175 INFO:tasks.workunit.client.1.smithi135.stdout:9/212: dwrite d7/f2f [0,4194304] 0 2024-03-30T04:28:16.175 INFO:tasks.workunit.client.0.smithi123.stdout:7/658: dread d2/dc/d7f/fa2 [0,4194304] 0 2024-03-30T04:28:16.177 INFO:tasks.workunit.client.1.smithi135.stdout:9/213: dread - d7/f44 zero size 2024-03-30T04:28:16.177 INFO:tasks.workunit.client.0.smithi123.stdout:5/598: rename dd/d19/d1f/d35/f40 to dd/d19/d1f/d35/fc4 0 2024-03-30T04:28:16.178 INFO:tasks.workunit.client.1.smithi135.stdout:4/248: symlink d10/d1a/d2b/l56 0 2024-03-30T04:28:16.181 INFO:tasks.workunit.client.0.smithi123.stdout:3/690: creat d13/d24/d31/d23/d3a/d3c/d4e/d5d/da5/fdb x:0 0 0 2024-03-30T04:28:16.182 INFO:tasks.workunit.client.0.smithi123.stdout:8/639: symlink da/d27/d56/d6c/d6d/lbf 0 2024-03-30T04:28:16.184 INFO:tasks.workunit.client.0.smithi123.stdout:2/733: unlink d0/d1f/d16/f3c 0 2024-03-30T04:28:16.186 INFO:tasks.workunit.client.0.smithi123.stdout:7/659: write d2/da/d22/d3b/f4e [3442162,33705] 0 2024-03-30T04:28:16.186 INFO:tasks.workunit.client.0.smithi123.stdout:5/599: symlink dd/d19/d5f/lc5 0 2024-03-30T04:28:16.189 INFO:tasks.workunit.client.0.smithi123.stdout:2/734: write d0/d1f/d16/fcb [63349,21203] 0 2024-03-30T04:28:16.190 INFO:tasks.workunit.client.0.smithi123.stdout:8/640: unlink da/d99/fa1 0 2024-03-30T04:28:16.191 INFO:tasks.workunit.client.1.smithi135.stdout:6/234: link d2/d36/f44 d2/dd/d14/d2b/f47 0 2024-03-30T04:28:16.194 INFO:tasks.workunit.client.1.smithi135.stdout:3/262: dread d3/f11 [0,4194304] 0 2024-03-30T04:28:16.196 INFO:tasks.workunit.client.1.smithi135.stdout:8/250: write d6/d8/d15/d2a/f37 [735615,46992] 0 2024-03-30T04:28:16.198 INFO:tasks.workunit.client.0.smithi123.stdout:8/641: mknod da/d27/d56/cc0 0 2024-03-30T04:28:16.199 INFO:tasks.workunit.client.1.smithi135.stdout:4/249: rename d10/d1a/f1c to d10/f57 0 2024-03-30T04:28:16.199 INFO:tasks.workunit.client.1.smithi135.stdout:0/330: link d0/d5/d9/d23/d58/d3b/c64 d0/d34/c68 0 2024-03-30T04:28:16.200 INFO:tasks.workunit.client.1.smithi135.stdout:6/235: rename d2/dd/d14/d30/d43 to d2/dd/d14/d30/d43/d48 22 2024-03-30T04:28:16.201 INFO:tasks.workunit.client.0.smithi123.stdout:7/660: rename d2/da/d11/d61/d55/d59/db5/fa6 to d2/da/d11/d61/d55/fde 0 2024-03-30T04:28:16.202 INFO:tasks.workunit.client.1.smithi135.stdout:8/251: rmdir d6/d8 39 2024-03-30T04:28:16.204 INFO:tasks.workunit.client.0.smithi123.stdout:8/642: fdatasync da/dc/f52 0 2024-03-30T04:28:16.205 INFO:tasks.workunit.client.1.smithi135.stdout:9/214: dread f6 [0,4194304] 0 2024-03-30T04:28:16.206 INFO:tasks.workunit.client.1.smithi135.stdout:5/261: sync 2024-03-30T04:28:16.206 INFO:tasks.workunit.client.1.smithi135.stdout:2/224: sync 2024-03-30T04:28:16.206 INFO:tasks.workunit.client.1.smithi135.stdout:1/187: sync 2024-03-30T04:28:16.206 INFO:tasks.workunit.client.1.smithi135.stdout:7/292: sync 2024-03-30T04:28:16.206 INFO:tasks.workunit.client.1.smithi135.stdout:3/263: creat d3/f51 x:0 0 0 2024-03-30T04:28:16.209 INFO:tasks.workunit.client.0.smithi123.stdout:5/600: dwrite dd/d19/d1f/d35/d3a/f83 [0,4194304] 0 2024-03-30T04:28:16.210 INFO:tasks.workunit.client.0.smithi123.stdout:8/643: mknod da/cc1 0 2024-03-30T04:28:16.210 INFO:tasks.workunit.client.1.smithi135.stdout:4/250: mknod d10/d1a/c58 0 2024-03-30T04:28:16.211 INFO:tasks.workunit.client.1.smithi135.stdout:2/225: rmdir d4 39 2024-03-30T04:28:16.213 INFO:tasks.workunit.client.1.smithi135.stdout:0/331: creat d0/d5/d9/d23/d5d/f69 x:0 0 0 2024-03-30T04:28:16.213 INFO:tasks.workunit.client.1.smithi135.stdout:3/264: write d3/d10/d16/d23/f30 [4712108,87022] 0 2024-03-30T04:28:16.213 INFO:tasks.workunit.client.1.smithi135.stdout:7/293: chown d4/f54 0 1 2024-03-30T04:28:16.214 INFO:tasks.workunit.client.1.smithi135.stdout:9/215: rename d7/d29/f2a to d7/d29/f48 0 2024-03-30T04:28:16.215 INFO:tasks.workunit.client.0.smithi123.stdout:3/691: write d13/d24/f4d [1077822,4087] 0 2024-03-30T04:28:16.215 INFO:tasks.workunit.client.0.smithi123.stdout:2/735: write d0/d1f/d1c/d80/dbb/dd5/f9c [1362168,97605] 0 2024-03-30T04:28:16.216 INFO:tasks.workunit.client.1.smithi135.stdout:9/216: fdatasync d7/f44 0 2024-03-30T04:28:16.216 INFO:tasks.workunit.client.1.smithi135.stdout:5/262: fdatasync d1/f36 0 2024-03-30T04:28:16.217 INFO:tasks.workunit.client.0.smithi123.stdout:7/661: write d2/da/d11/f58 [1280809,106996] 0 2024-03-30T04:28:16.218 INFO:tasks.workunit.client.1.smithi135.stdout:1/188: creat d16/f3f x:0 0 0 2024-03-30T04:28:16.220 INFO:tasks.workunit.client.0.smithi123.stdout:5/601: truncate dd/d19/d1f/d35/d3a/f80 1001645 0 2024-03-30T04:28:16.223 INFO:tasks.workunit.client.1.smithi135.stdout:2/226: fdatasync d4/da/f1e 0 2024-03-30T04:28:16.223 INFO:tasks.workunit.client.1.smithi135.stdout:8/252: creat d6/d8/d15/d1f/f44 x:0 0 0 2024-03-30T04:28:16.223 INFO:tasks.workunit.client.0.smithi123.stdout:5/602: chown dd/d19/d1f/d3c/d72/d8c/f9a 853 1 2024-03-30T04:28:16.224 INFO:tasks.workunit.client.1.smithi135.stdout:5/263: write d1/da/f1b [982301,121647] 0 2024-03-30T04:28:16.224 INFO:tasks.workunit.client.1.smithi135.stdout:6/236: getdents d2/d36 0 2024-03-30T04:28:16.227 INFO:tasks.workunit.client.0.smithi123.stdout:8/644: mkdir da/d27/d56/d6c/d96/dc2 0 2024-03-30T04:28:16.230 INFO:tasks.workunit.client.0.smithi123.stdout:2/736: link d0/d1f/d1c/d39/ce1 d0/d1f/d1c/d80/dbb/d97/cf0 0 2024-03-30T04:28:16.231 INFO:tasks.workunit.client.1.smithi135.stdout:3/265: symlink d3/d10/d16/d41/l52 0 2024-03-30T04:28:16.232 INFO:tasks.workunit.client.1.smithi135.stdout:9/217: mknod d7/c49 0 2024-03-30T04:28:16.233 INFO:tasks.workunit.client.0.smithi123.stdout:8/645: rmdir da/dc/d14/d9d 39 2024-03-30T04:28:16.234 INFO:tasks.workunit.client.0.smithi123.stdout:2/737: dread - d0/d37/f8a zero size 2024-03-30T04:28:16.234 INFO:tasks.workunit.client.1.smithi135.stdout:0/332: symlink d0/l6a 0 2024-03-30T04:28:16.235 INFO:tasks.workunit.client.1.smithi135.stdout:6/237: dread - d2/dd/d14/f3f zero size 2024-03-30T04:28:16.236 INFO:tasks.workunit.client.0.smithi123.stdout:8/646: stat da/d27/d3c/f98 0 2024-03-30T04:28:16.237 INFO:tasks.workunit.client.1.smithi135.stdout:4/251: creat d10/f59 x:0 0 0 2024-03-30T04:28:16.238 INFO:tasks.workunit.client.0.smithi123.stdout:0/641: sync 2024-03-30T04:28:16.239 INFO:tasks.workunit.client.1.smithi135.stdout:3/266: fsync d3/d50/f1f 0 2024-03-30T04:28:16.239 INFO:tasks.workunit.client.0.smithi123.stdout:6/721: sync 2024-03-30T04:28:16.240 INFO:tasks.workunit.client.0.smithi123.stdout:8/647: write da/dc/d14/f6e [253993,46613] 0 2024-03-30T04:28:16.241 INFO:tasks.workunit.client.0.smithi123.stdout:7/662: dwrite d2/dc/d56/fdd [0,4194304] 0 2024-03-30T04:28:16.241 INFO:tasks.workunit.client.0.smithi123.stdout:0/642: fsync d6/dd/d31/d35/f42 0 2024-03-30T04:28:16.242 INFO:tasks.workunit.client.0.smithi123.stdout:2/738: truncate d0/d1f/d1c/d40/fac 850960 0 2024-03-30T04:28:16.243 INFO:tasks.workunit.client.1.smithi135.stdout:0/333: chown d0/d5/d9/d23/d58/d3b 7 1 2024-03-30T04:28:16.244 INFO:tasks.workunit.client.1.smithi135.stdout:9/218: creat d7/d1b/d37/f4a x:0 0 0 2024-03-30T04:28:16.244 INFO:tasks.workunit.client.0.smithi123.stdout:6/722: symlink d10/d82/d8e/dc7/d42/lf6 0 2024-03-30T04:28:16.246 INFO:tasks.workunit.client.0.smithi123.stdout:7/663: mknod d2/dc/d49/d68/cdf 0 2024-03-30T04:28:16.246 INFO:tasks.workunit.client.1.smithi135.stdout:1/189: write d16/d29/f35 [622939,36887] 0 2024-03-30T04:28:16.246 INFO:tasks.workunit.client.1.smithi135.stdout:3/267: chown d3/d10/d16/f1b 127 1 2024-03-30T04:28:16.251 INFO:tasks.workunit.client.0.smithi123.stdout:0/643: write d6/dd/d15/d28/d4b/f91 [1310531,28636] 0 2024-03-30T04:28:16.251 INFO:tasks.workunit.client.1.smithi135.stdout:7/294: truncate d4/f29 3569578 0 2024-03-30T04:28:16.253 INFO:tasks.workunit.client.1.smithi135.stdout:0/334: chown d0/d5/l26 5199555 1 2024-03-30T04:28:16.254 INFO:tasks.workunit.client.1.smithi135.stdout:3/268: stat d3/d50/cd 0 2024-03-30T04:28:16.255 INFO:tasks.workunit.client.0.smithi123.stdout:7/664: fdatasync d2/da/d11/d61/d55/d59/db5/f43 0 2024-03-30T04:28:16.256 INFO:tasks.workunit.client.1.smithi135.stdout:1/190: readlink d16/d29/l2d 0 2024-03-30T04:28:16.256 INFO:tasks.workunit.client.1.smithi135.stdout:5/264: truncate d1/d7/f2c 425917 0 2024-03-30T04:28:16.260 INFO:tasks.workunit.client.0.smithi123.stdout:6/723: truncate d10/d82/d8e/dc7/d42/fb8 12667 0 2024-03-30T04:28:16.263 INFO:tasks.workunit.client.0.smithi123.stdout:2/739: getdents d0/d1f/d27 0 2024-03-30T04:28:16.263 INFO:tasks.workunit.client.1.smithi135.stdout:2/227: dwrite d4/d32/d12/f2a [0,4194304] 0 2024-03-30T04:28:16.264 INFO:tasks.workunit.client.0.smithi123.stdout:7/665: write d2/da/d11/d61/d55/d59/db5/f2a [235026,95431] 0 2024-03-30T04:28:16.266 INFO:tasks.workunit.client.1.smithi135.stdout:5/265: fsync d1/d7/ff 0 2024-03-30T04:28:16.266 INFO:tasks.workunit.client.0.smithi123.stdout:3/692: dread d13/d24/d31/d23/f88 [0,4194304] 0 2024-03-30T04:28:16.270 INFO:tasks.workunit.client.1.smithi135.stdout:8/253: dwrite d6/f11 [0,4194304] 0 2024-03-30T04:28:16.274 INFO:tasks.workunit.client.1.smithi135.stdout:3/269: creat d3/d10/d16/d41/f53 x:0 0 0 2024-03-30T04:28:16.275 INFO:tasks.workunit.client.1.smithi135.stdout:1/191: symlink d16/d29/d2c/d30/d3a/l40 0 2024-03-30T04:28:16.278 INFO:tasks.workunit.client.1.smithi135.stdout:5/266: fsync d1/d3/d2b/f2f 0 2024-03-30T04:28:16.279 INFO:tasks.workunit.client.0.smithi123.stdout:8/648: dwrite da/d27/d3c/d2d/f57 [4194304,4194304] 0 2024-03-30T04:28:16.281 INFO:tasks.workunit.client.1.smithi135.stdout:2/228: symlink d4/d32/d12/d45/d3c/l49 0 2024-03-30T04:28:16.281 INFO:tasks.workunit.client.1.smithi135.stdout:1/192: fdatasync d16/d19/f2b 0 2024-03-30T04:28:16.282 INFO:tasks.workunit.client.1.smithi135.stdout:0/335: link d0/d5/d9/ff d0/d5/d9/d23/d58/d55/f6b 0 2024-03-30T04:28:16.286 INFO:tasks.workunit.client.1.smithi135.stdout:8/254: write d6/d8/d14/f2e [834846,125132] 0 2024-03-30T04:28:16.290 INFO:tasks.workunit.client.1.smithi135.stdout:5/267: symlink d1/da/l51 0 2024-03-30T04:28:16.290 INFO:tasks.workunit.client.1.smithi135.stdout:7/295: dread d4/db/d1a/d5c/f4e [0,4194304] 0 2024-03-30T04:28:16.293 INFO:tasks.workunit.client.1.smithi135.stdout:9/219: dwrite d7/d29/f3c [0,4194304] 0 2024-03-30T04:28:16.299 INFO:tasks.workunit.client.1.smithi135.stdout:1/193: unlink ce 0 2024-03-30T04:28:16.301 INFO:tasks.workunit.client.0.smithi123.stdout:7/666: dwrite d2/dc/d49/dbc/fc3 [0,4194304] 0 2024-03-30T04:28:16.304 INFO:tasks.workunit.client.1.smithi135.stdout:8/255: chown d6/d8 0 1 2024-03-30T04:28:16.304 INFO:tasks.workunit.client.0.smithi123.stdout:2/740: fdatasync d0/d1f/d1c/f3f 0 2024-03-30T04:28:16.305 INFO:tasks.workunit.client.1.smithi135.stdout:5/268: readlink d1/d7/d1a/d1f/l2e 0 2024-03-30T04:28:16.305 INFO:tasks.workunit.client.1.smithi135.stdout:2/229: write d4/d32/d12/f3f [486700,102202] 0 2024-03-30T04:28:16.307 INFO:tasks.workunit.client.0.smithi123.stdout:0/644: dread d6/d52/fa8 [0,4194304] 0 2024-03-30T04:28:16.311 INFO:tasks.workunit.client.0.smithi123.stdout:7/667: creat d2/da/d22/fe0 x:0 0 0 2024-03-30T04:28:16.311 INFO:tasks.workunit.client.1.smithi135.stdout:1/194: mknod d16/d1a/c41 0 2024-03-30T04:28:16.312 INFO:tasks.workunit.client.1.smithi135.stdout:8/256: fsync d6/d8/d15/f2b 0 2024-03-30T04:28:16.312 INFO:tasks.workunit.client.1.smithi135.stdout:0/336: truncate d0/d5/d9/d23/d58/d55/f5f 466355 0 2024-03-30T04:28:16.312 INFO:tasks.workunit.client.1.smithi135.stdout:9/220: rmdir d7/d41 39 2024-03-30T04:28:16.314 INFO:tasks.workunit.client.0.smithi123.stdout:6/724: getdents d10/d82/d8e/dc7 0 2024-03-30T04:28:16.316 INFO:tasks.workunit.client.1.smithi135.stdout:5/269: mkdir d1/d45/d52 0 2024-03-30T04:28:16.316 INFO:tasks.workunit.client.0.smithi123.stdout:0/645: write d6/dd/d15/d98/d4e/d69/f81 [1185097,16888] 0 2024-03-30T04:28:16.318 INFO:tasks.workunit.client.1.smithi135.stdout:8/257: chown l3 89432887 1 2024-03-30T04:28:16.322 INFO:tasks.workunit.client.0.smithi123.stdout:8/649: creat da/d27/d3c/d64/d78/db8/fc3 x:0 0 0 2024-03-30T04:28:16.322 INFO:tasks.workunit.client.0.smithi123.stdout:7/668: creat d2/d4d/d80/d98/dd5/fe1 x:0 0 0 2024-03-30T04:28:16.322 INFO:tasks.workunit.client.1.smithi135.stdout:0/337: write d0/d1/d16/f4b [502034,65721] 0 2024-03-30T04:28:16.323 INFO:tasks.workunit.client.1.smithi135.stdout:3/270: dread d3/f4 [0,4194304] 0 2024-03-30T04:28:16.323 INFO:tasks.workunit.client.1.smithi135.stdout:9/221: dread - d7/d1b/f40 zero size 2024-03-30T04:28:16.323 INFO:tasks.workunit.client.1.smithi135.stdout:7/296: write d4/db/d1a/d4b/f45 [847208,89857] 0 2024-03-30T04:28:16.323 INFO:tasks.workunit.client.1.smithi135.stdout:5/270: write d1/d7/f48 [2264858,52908] 0 2024-03-30T04:28:16.324 INFO:tasks.workunit.client.0.smithi123.stdout:6/725: creat d10/d82/d8e/df5/ff7 x:0 0 0 2024-03-30T04:28:16.325 INFO:tasks.workunit.client.0.smithi123.stdout:8/650: readlink da/dc/d14/d9d/laa 0 2024-03-30T04:28:16.325 INFO:tasks.workunit.client.1.smithi135.stdout:1/195: unlink d16/d19/c1b 0 2024-03-30T04:28:16.331 INFO:tasks.workunit.client.0.smithi123.stdout:0/646: symlink d6/dd/d15/d98/d5d/d7f/ld2 0 2024-03-30T04:28:16.331 INFO:tasks.workunit.client.1.smithi135.stdout:1/196: chown d16/d19/d20/l27 3399746 1 2024-03-30T04:28:16.331 INFO:tasks.workunit.client.0.smithi123.stdout:6/726: creat d10/d82/dc1/d6a/d7a/d99/dba/ff8 x:0 0 0 2024-03-30T04:28:16.333 INFO:tasks.workunit.client.1.smithi135.stdout:2/230: link d4/da/f21 d4/d32/d2e/f4a 0 2024-03-30T04:28:16.333 INFO:tasks.workunit.client.0.smithi123.stdout:2/741: dread d0/d1f/d27/d68/dd6/fe0 [0,4194304] 0 2024-03-30T04:28:16.334 INFO:tasks.workunit.client.0.smithi123.stdout:0/647: readlink d6/dd/d16/la5 0 2024-03-30T04:28:16.335 INFO:tasks.workunit.client.0.smithi123.stdout:8/651: truncate da/d27/fb3 676555 0 2024-03-30T04:28:16.337 INFO:tasks.workunit.client.1.smithi135.stdout:2/231: readlink d4/d6/l1b 0 2024-03-30T04:28:16.338 INFO:tasks.workunit.client.0.smithi123.stdout:2/742: read d0/d1f/d1c/d80/f93 [4073531,54278] 0 2024-03-30T04:28:16.339 INFO:tasks.workunit.client.1.smithi135.stdout:8/258: creat d6/d8/d15/d2a/d35/f45 x:0 0 0 2024-03-30T04:28:16.340 INFO:tasks.workunit.client.1.smithi135.stdout:8/259: readlink d6/d8/d15/l3d 0 2024-03-30T04:28:16.341 INFO:tasks.workunit.client.1.smithi135.stdout:3/271: mkdir d3/d54 0 2024-03-30T04:28:16.344 INFO:tasks.workunit.client.0.smithi123.stdout:6/727: getdents d10 0 2024-03-30T04:28:16.344 INFO:tasks.workunit.client.0.smithi123.stdout:2/743: creat d0/ff1 x:0 0 0 2024-03-30T04:28:16.347 INFO:tasks.workunit.client.0.smithi123.stdout:0/648: dread d6/dd/d31/d35/f5b [0,4194304] 0 2024-03-30T04:28:16.350 INFO:tasks.workunit.client.0.smithi123.stdout:7/669: dwrite d2/da/d22/f86 [0,4194304] 0 2024-03-30T04:28:16.351 INFO:tasks.workunit.client.1.smithi135.stdout:3/272: truncate d3/d50/f25 26210 0 2024-03-30T04:28:16.352 INFO:tasks.workunit.client.0.smithi123.stdout:6/728: mkdir d10/d82/df9 0 2024-03-30T04:28:16.354 INFO:tasks.workunit.client.1.smithi135.stdout:9/222: rename d7/l3f to d7/l4b 0 2024-03-30T04:28:16.358 INFO:tasks.workunit.client.0.smithi123.stdout:2/744: write d0/d1f/d1c/d36/f89 [19091,70160] 0 2024-03-30T04:28:16.358 INFO:tasks.workunit.client.0.smithi123.stdout:7/670: write d2/da/d11/f48 [1901260,111498] 0 2024-03-30T04:28:16.360 INFO:tasks.workunit.client.0.smithi123.stdout:0/649: mkdir d6/dd/d15/d28/dd3 0 2024-03-30T04:28:16.362 INFO:tasks.workunit.client.0.smithi123.stdout:8/652: dwrite da/f46 [4194304,4194304] 0 2024-03-30T04:28:16.362 INFO:tasks.workunit.client.1.smithi135.stdout:9/223: chown d7/f26 482767 1 2024-03-30T04:28:16.362 INFO:tasks.workunit.client.1.smithi135.stdout:2/232: creat d4/d32/d2e/f4b x:0 0 0 2024-03-30T04:28:16.363 INFO:tasks.workunit.client.0.smithi123.stdout:3/693: write d13/d24/d31/d23/f25 [4596251,6198] 0 2024-03-30T04:28:16.364 INFO:tasks.workunit.client.1.smithi135.stdout:7/297: dwrite d4/db/de/f1d [0,4194304] 0 2024-03-30T04:28:16.367 INFO:tasks.workunit.client.0.smithi123.stdout:6/729: creat d10/d82/dc1/d6a/d7a/d99/dba/ffa x:0 0 0 2024-03-30T04:28:16.367 INFO:tasks.workunit.client.0.smithi123.stdout:2/745: mkdir d0/d1f/d1c/d39/df2 0 2024-03-30T04:28:16.368 INFO:tasks.workunit.client.1.smithi135.stdout:2/233: fsync d4/f5 0 2024-03-30T04:28:16.370 INFO:tasks.workunit.client.0.smithi123.stdout:7/671: unlink d2/dc/d7f/cd3 0 2024-03-30T04:28:16.370 INFO:tasks.workunit.client.1.smithi135.stdout:3/273: chown d3/d10/f31 381814884 1 2024-03-30T04:28:16.370 INFO:tasks.workunit.client.1.smithi135.stdout:0/338: dwrite d0/d5/d9/d23/f66 [0,4194304] 0 2024-03-30T04:28:16.370 INFO:tasks.workunit.client.0.smithi123.stdout:6/730: creat d10/d82/dc1/d6a/d7a/de8/ffb x:0 0 0 2024-03-30T04:28:16.372 INFO:tasks.workunit.client.1.smithi135.stdout:5/271: dwrite d1/d7/d11/d3c/f44 [0,4194304] 0 2024-03-30T04:28:16.373 INFO:tasks.workunit.client.1.smithi135.stdout:8/260: write d6/f12 [502953,100573] 0 2024-03-30T04:28:16.373 INFO:tasks.workunit.client.1.smithi135.stdout:9/224: write d7/d41/f43 [392832,39869] 0 2024-03-30T04:28:16.374 INFO:tasks.workunit.client.0.smithi123.stdout:6/731: chown d10/d20/fea 4332514 1 2024-03-30T04:28:16.374 INFO:tasks.workunit.client.0.smithi123.stdout:0/650: creat d6/dd/d15/d98/d4e/d69/d6e/fd4 x:0 0 0 2024-03-30T04:28:16.376 INFO:tasks.workunit.client.0.smithi123.stdout:8/653: mkdir da/dc/da9/dc4 0 2024-03-30T04:28:16.377 INFO:tasks.workunit.client.0.smithi123.stdout:2/746: mkdir d0/d1f/d16/df3 0 2024-03-30T04:28:16.379 INFO:tasks.workunit.client.0.smithi123.stdout:8/654: chown da/d27/d3c/cb5 89513 1 2024-03-30T04:28:16.381 INFO:tasks.workunit.client.0.smithi123.stdout:3/694: symlink d13/d24/d31/d23/d3a/d3c/d4e/d5d/da5/dcd/ldc 0 2024-03-30T04:28:16.384 INFO:tasks.workunit.client.1.smithi135.stdout:7/298: stat d4/db/d1a/d3d/c5f 0 2024-03-30T04:28:16.384 INFO:tasks.workunit.client.1.smithi135.stdout:8/261: write d6/d8/d14/f30 [525874,67021] 0 2024-03-30T04:28:16.384 INFO:tasks.workunit.client.0.smithi123.stdout:0/651: getdents d6/dd/d15/d98/d5d/d7f 0 2024-03-30T04:28:16.384 INFO:tasks.workunit.client.0.smithi123.stdout:2/747: readlink d0/d1f/d27/d68/lcf 0 2024-03-30T04:28:16.385 INFO:tasks.workunit.client.1.smithi135.stdout:0/339: truncate d0/d1/d16/f62 732616 0 2024-03-30T04:28:16.385 INFO:tasks.workunit.client.1.smithi135.stdout:3/274: write d3/d50/d22/f3d [519207,69399] 0 2024-03-30T04:28:16.385 INFO:tasks.workunit.client.1.smithi135.stdout:2/234: unlink d4/d32/d2e/f3d 0 2024-03-30T04:28:16.385 INFO:tasks.workunit.client.0.smithi123.stdout:9/574: sync 2024-03-30T04:28:16.386 INFO:tasks.workunit.client.0.smithi123.stdout:0/652: fsync d6/d52/fa8 0 2024-03-30T04:28:16.388 INFO:tasks.workunit.client.0.smithi123.stdout:3/695: creat d13/d24/d31/d30/fdd x:0 0 0 2024-03-30T04:28:16.388 INFO:tasks.workunit.client.1.smithi135.stdout:1/197: dread d16/f25 [0,4194304] 0 2024-03-30T04:28:16.388 INFO:tasks.workunit.client.1.smithi135.stdout:9/225: rename d7/c1f to d7/d1b/d37/c4c 0 2024-03-30T04:28:16.392 INFO:tasks.workunit.client.1.smithi135.stdout:7/299: rmdir d4/db/de 39 2024-03-30T04:28:16.393 INFO:tasks.workunit.client.0.smithi123.stdout:8/655: rename da/dc/d14/f36 to da/dc/d14/d18/d21/da2/fc5 0 2024-03-30T04:28:16.395 INFO:tasks.workunit.client.0.smithi123.stdout:2/748: rmdir d0/d1f/d1c/d80/dbb/d97 39 2024-03-30T04:28:16.396 INFO:tasks.workunit.client.0.smithi123.stdout:6/732: read d10/d82/dc1/d45/d77/f7c [316279,38847] 0 2024-03-30T04:28:16.397 INFO:tasks.workunit.client.0.smithi123.stdout:7/672: dwrite d2/dc/d7f/fa3 [0,4194304] 0 2024-03-30T04:28:16.398 INFO:tasks.workunit.client.1.smithi135.stdout:9/226: symlink d7/d1b/d37/l4d 0 2024-03-30T04:28:16.399 INFO:tasks.workunit.client.1.smithi135.stdout:1/198: creat d16/f42 x:0 0 0 2024-03-30T04:28:16.401 INFO:tasks.workunit.client.0.smithi123.stdout:0/653: symlink d6/dd/d15/d98/d4e/d69/ld5 0 2024-03-30T04:28:16.402 INFO:tasks.workunit.client.1.smithi135.stdout:0/340: symlink d0/d5/d9/d23/d58/d55/l6c 0 2024-03-30T04:28:16.403 INFO:tasks.workunit.client.1.smithi135.stdout:2/235: mknod d4/d32/d2e/d2f/c4c 0 2024-03-30T04:28:16.403 INFO:tasks.workunit.client.0.smithi123.stdout:0/654: chown d6/d52/dcf 539 1 2024-03-30T04:28:16.403 INFO:tasks.workunit.client.0.smithi123.stdout:6/733: unlink d10/d20/lc5 0 2024-03-30T04:28:16.404 INFO:tasks.workunit.client.0.smithi123.stdout:8/656: symlink da/dc/d14/d9c/lc6 0 2024-03-30T04:28:16.404 INFO:tasks.workunit.client.0.smithi123.stdout:7/673: write d2/da/d11/fc6 [1030343,84047] 0 2024-03-30T04:28:16.405 INFO:tasks.workunit.client.1.smithi135.stdout:8/262: symlink d6/d8/d14/d32/l46 0 2024-03-30T04:28:16.406 INFO:tasks.workunit.client.0.smithi123.stdout:3/696: mkdir d13/d24/d31/d23/d3a/d3c/d4e/dde 0 2024-03-30T04:28:16.408 INFO:tasks.workunit.client.1.smithi135.stdout:1/199: mkdir d16/d29/d2c/d43 0 2024-03-30T04:28:16.408 INFO:tasks.workunit.client.0.smithi123.stdout:8/657: write da/dc/d14/f6e [433239,122465] 0 2024-03-30T04:28:16.411 INFO:tasks.workunit.client.0.smithi123.stdout:3/697: fsync d13/f5e 0 2024-03-30T04:28:16.412 INFO:tasks.workunit.client.1.smithi135.stdout:3/275: creat d3/f55 x:0 0 0 2024-03-30T04:28:16.412 INFO:tasks.workunit.client.0.smithi123.stdout:6/734: write d10/d82/dc1/d6a/d7a/d99/dba/fe1 [979155,90792] 0 2024-03-30T04:28:16.413 INFO:tasks.workunit.client.0.smithi123.stdout:0/655: creat d6/dd/d15/d28/dd3/fd6 x:0 0 0 2024-03-30T04:28:16.415 INFO:tasks.workunit.client.0.smithi123.stdout:7/674: mknod d2/dc/d1c/ce2 0 2024-03-30T04:28:16.416 INFO:tasks.workunit.client.1.smithi135.stdout:8/263: dread - d6/d8/d14/f29 zero size 2024-03-30T04:28:16.416 INFO:tasks.workunit.client.0.smithi123.stdout:0/656: readlink d6/dd/d15/d98/d4e/l53 0 2024-03-30T04:28:16.417 INFO:tasks.workunit.client.1.smithi135.stdout:9/227: creat d7/f4e x:0 0 0 2024-03-30T04:28:16.419 INFO:tasks.workunit.client.0.smithi123.stdout:0/657: write d6/dd/d15/f8e [1568126,49414] 0 2024-03-30T04:28:16.419 INFO:tasks.workunit.client.0.smithi123.stdout:8/658: read da/d27/d3c/f51 [156127,130323] 0 2024-03-30T04:28:16.421 INFO:tasks.workunit.client.0.smithi123.stdout:7/675: mkdir d2/da/d11/d61/d55/d59/db5/dab/de3 0 2024-03-30T04:28:16.422 INFO:tasks.workunit.client.0.smithi123.stdout:2/749: dread d0/d1f/d1c/d36/f89 [0,4194304] 0 2024-03-30T04:28:16.423 INFO:tasks.workunit.client.0.smithi123.stdout:0/658: readlink d6/dd/d31/lc2 0 2024-03-30T04:28:16.424 INFO:tasks.workunit.client.1.smithi135.stdout:3/276: creat d3/d50/d46/f56 x:0 0 0 2024-03-30T04:28:16.425 INFO:tasks.workunit.client.1.smithi135.stdout:2/236: symlink d4/d32/d2e/l4d 0 2024-03-30T04:28:16.426 INFO:tasks.workunit.client.0.smithi123.stdout:8/659: write da/dc/f20 [424300,20357] 0 2024-03-30T04:28:16.427 INFO:tasks.workunit.client.0.smithi123.stdout:7/676: chown d2/dc/d49/dbc/fc3 32261874 1 2024-03-30T04:28:16.428 INFO:tasks.workunit.client.0.smithi123.stdout:2/750: write d0/d1f/d16/fdc [763541,53540] 0 2024-03-30T04:28:16.431 INFO:tasks.workunit.client.0.smithi123.stdout:7/677: chown d2/dc/c10 8 1 2024-03-30T04:28:16.433 INFO:tasks.workunit.client.1.smithi135.stdout:2/237: unlink d4/c42 0 2024-03-30T04:28:16.436 INFO:tasks.workunit.client.0.smithi123.stdout:8/660: dread - da/d27/d56/d6c/d6d/f90 zero size 2024-03-30T04:28:16.438 INFO:tasks.workunit.client.0.smithi123.stdout:0/659: rename d6/dd/d15/d98/d4e/l53 to d6/dd/d15/d98/d5d/ld7 0 2024-03-30T04:28:16.439 INFO:tasks.workunit.client.0.smithi123.stdout:7/678: truncate d2/dc/f52 632622 0 2024-03-30T04:28:16.441 INFO:tasks.workunit.client.1.smithi135.stdout:3/277: rename d3/f55 to d3/d50/d46/f57 0 2024-03-30T04:28:16.442 INFO:tasks.workunit.client.1.smithi135.stdout:2/238: write d4/d32/d12/d45/f2c [3175244,38333] 0 2024-03-30T04:28:16.442 INFO:tasks.workunit.client.1.smithi135.stdout:1/200: dwrite d16/f42 [0,4194304] 0 2024-03-30T04:28:16.445 INFO:tasks.workunit.client.0.smithi123.stdout:9/575: write d6/d1f/d72/f8f [1744841,66381] 0 2024-03-30T04:28:16.445 INFO:tasks.workunit.client.0.smithi123.stdout:3/698: dread d13/d24/d31/d23/d3a/d3c/d4e/d5d/f82 [0,4194304] 0 2024-03-30T04:28:16.446 INFO:tasks.workunit.client.0.smithi123.stdout:0/660: mkdir d6/dd/d15/d98/d4e/d69/d6e/dd8 0 2024-03-30T04:28:16.449 INFO:tasks.workunit.client.0.smithi123.stdout:8/661: link da/f46 da/d27/d3c/d2d/d5c/fc7 0 2024-03-30T04:28:16.450 INFO:tasks.workunit.client.1.smithi135.stdout:1/201: chown d16/d29/d2c/d30/d3a/l40 50922351 1 2024-03-30T04:28:16.451 INFO:tasks.workunit.client.1.smithi135.stdout:3/278: creat d3/d50/d46/f58 x:0 0 0 2024-03-30T04:28:16.453 INFO:tasks.workunit.client.1.smithi135.stdout:2/239: write d4/d32/d2e/f40 [200354,15582] 0 2024-03-30T04:28:16.455 INFO:tasks.workunit.client.0.smithi123.stdout:8/662: readlink da/d27/d56/d6c/d96/la8 0 2024-03-30T04:28:16.456 INFO:tasks.workunit.client.1.smithi135.stdout:8/264: dread d6/d8/f31 [0,4194304] 0 2024-03-30T04:28:16.456 INFO:tasks.workunit.client.1.smithi135.stdout:7/300: dread d4/db/de/f25 [0,4194304] 0 2024-03-30T04:28:16.456 INFO:tasks.workunit.client.1.smithi135.stdout:1/202: write d16/f1c [408170,82362] 0 2024-03-30T04:28:16.457 INFO:tasks.workunit.client.0.smithi123.stdout:9/576: fdatasync d6/d1f/f2e 0 2024-03-30T04:28:16.457 INFO:tasks.workunit.client.0.smithi123.stdout:3/699: mknod d13/d24/d31/d23/d3a/d3c/d66/da9/cdf 0 2024-03-30T04:28:16.460 INFO:tasks.workunit.client.1.smithi135.stdout:2/240: readlink d4/d32/d2e/l4d 0 2024-03-30T04:28:16.462 INFO:tasks.workunit.client.0.smithi123.stdout:8/663: rmdir da/dc/d14/d18/d21/da2 39 2024-03-30T04:28:16.463 INFO:tasks.workunit.client.1.smithi135.stdout:1/203: chown d16/c17 1 1 2024-03-30T04:28:16.463 INFO:tasks.workunit.client.1.smithi135.stdout:3/279: dread d3/d10/f2d [0,4194304] 0 2024-03-30T04:28:16.466 INFO:tasks.workunit.client.1.smithi135.stdout:8/265: unlink d6/d8/d14/l1b 0 2024-03-30T04:28:16.468 INFO:tasks.workunit.client.0.smithi123.stdout:2/751: dwrite d0/d1f/d16/f23 [0,4194304] 0 2024-03-30T04:28:16.469 INFO:tasks.workunit.client.0.smithi123.stdout:3/700: mkdir d13/d24/d31/de0 0 2024-03-30T04:28:16.469 INFO:tasks.workunit.client.0.smithi123.stdout:7/679: dwrite d2/da/d11/f20 [0,4194304] 0 2024-03-30T04:28:16.471 INFO:tasks.workunit.client.1.smithi135.stdout:5/272: dread d1/da/f18 [0,4194304] 0 2024-03-30T04:28:16.471 INFO:tasks.workunit.client.1.smithi135.stdout:0/341: dwrite d0/d13/d1e/d3a/f61 [0,4194304] 0 2024-03-30T04:28:16.476 INFO:tasks.workunit.client.1.smithi135.stdout:2/241: mknod d4/d32/d2e/c4e 0 2024-03-30T04:28:16.477 INFO:tasks.workunit.client.0.smithi123.stdout:2/752: symlink d0/d1f/d1c/d80/lf4 0 2024-03-30T04:28:16.478 INFO:tasks.workunit.client.1.smithi135.stdout:5/273: rmdir d1/d45 39 2024-03-30T04:28:16.481 INFO:tasks.workunit.client.0.smithi123.stdout:3/701: creat d13/fe1 x:0 0 0 2024-03-30T04:28:16.482 INFO:tasks.workunit.client.1.smithi135.stdout:2/242: readlink d4/da/l22 0 2024-03-30T04:28:16.484 INFO:tasks.workunit.client.1.smithi135.stdout:7/301: creat d4/db/d1a/d5c/d2c/f63 x:0 0 0 2024-03-30T04:28:16.485 INFO:tasks.workunit.client.0.smithi123.stdout:8/664: creat da/d27/d56/fc8 x:0 0 0 2024-03-30T04:28:16.485 INFO:tasks.workunit.client.0.smithi123.stdout:7/680: symlink d2/da/d11/le4 0 2024-03-30T04:28:16.486 INFO:tasks.workunit.client.1.smithi135.stdout:5/274: rename d1/d3/f15 to d1/d3/d2b/f53 0 2024-03-30T04:28:16.487 INFO:tasks.workunit.client.1.smithi135.stdout:2/243: stat d4/d6/f1a 0 2024-03-30T04:28:16.487 INFO:tasks.workunit.client.1.smithi135.stdout:9/228: dread d7/f12 [0,4194304] 0 2024-03-30T04:28:16.487 INFO:tasks.workunit.client.0.smithi123.stdout:9/577: dwrite d6/d1f/f34 [0,4194304] 0 2024-03-30T04:28:16.489 INFO:tasks.workunit.client.1.smithi135.stdout:1/204: link d16/d19/d20/c39 d16/d29/d2c/d30/d3a/c44 0 2024-03-30T04:28:16.492 INFO:tasks.workunit.client.0.smithi123.stdout:3/702: symlink d13/d24/d31/d30/dd7/le2 0 2024-03-30T04:28:16.493 INFO:tasks.workunit.client.1.smithi135.stdout:1/205: fdatasync d16/d1a/f23 0 2024-03-30T04:28:16.493 INFO:tasks.workunit.client.1.smithi135.stdout:5/275: chown d1/f36 38208105 1 2024-03-30T04:28:16.493 INFO:tasks.workunit.client.1.smithi135.stdout:2/244: mknod d4/d6/c4f 0 2024-03-30T04:28:16.499 INFO:tasks.workunit.client.1.smithi135.stdout:9/229: mkdir d7/d4f 0 2024-03-30T04:28:16.500 INFO:tasks.workunit.client.0.smithi123.stdout:9/578: rename d6/d1f/d45 to d6/d1f/d45/d97/d69/daa/dc4 22 2024-03-30T04:28:16.504 INFO:tasks.workunit.client.0.smithi123.stdout:0/661: dwrite d6/dd/d15/d28/d4b/d8a/f93 [0,4194304] 0 2024-03-30T04:28:16.504 INFO:tasks.workunit.client.1.smithi135.stdout:9/230: fdatasync d7/f4e 0 2024-03-30T04:28:16.505 INFO:tasks.workunit.client.1.smithi135.stdout:3/280: dwrite d3/d50/fa [0,4194304] 0 2024-03-30T04:28:16.506 INFO:tasks.workunit.client.0.smithi123.stdout:3/703: symlink d13/d24/d31/d23/d3a/d3c/d4e/dbe/le3 0 2024-03-30T04:28:16.506 INFO:tasks.workunit.client.1.smithi135.stdout:9/231: readlink d7/d41/l42 0 2024-03-30T04:28:16.511 INFO:tasks.workunit.client.1.smithi135.stdout:1/206: creat d16/f45 x:0 0 0 2024-03-30T04:28:16.511 INFO:tasks.workunit.client.1.smithi135.stdout:2/245: mkdir d4/d32/d50 0 2024-03-30T04:28:16.512 INFO:tasks.workunit.client.1.smithi135.stdout:5/276: chown d1/da/l4a 25821371 1 2024-03-30T04:28:16.513 INFO:tasks.workunit.client.0.smithi123.stdout:2/753: dwrite d0/d1f/d16/f23 [0,4194304] 0 2024-03-30T04:28:16.515 INFO:tasks.workunit.client.0.smithi123.stdout:7/681: dread d2/da/d11/f1e [0,4194304] 0 2024-03-30T04:28:16.519 INFO:tasks.workunit.client.1.smithi135.stdout:5/277: fdatasync d1/f3e 0 2024-03-30T04:28:16.519 INFO:tasks.workunit.client.1.smithi135.stdout:8/266: dwrite d6/d8/f1d [4194304,4194304] 0 2024-03-30T04:28:16.521 INFO:tasks.workunit.client.0.smithi123.stdout:2/754: chown d0/d1f/d1c/c7e 104 1 2024-03-30T04:28:16.521 INFO:tasks.workunit.client.0.smithi123.stdout:7/682: chown d2/da/d11/d61 2158 1 2024-03-30T04:28:16.522 INFO:tasks.workunit.client.1.smithi135.stdout:2/246: rmdir d4/d32/d2e/d2f 39 2024-03-30T04:28:16.522 INFO:tasks.workunit.client.1.smithi135.stdout:1/207: creat d16/d29/d32/f46 x:0 0 0 2024-03-30T04:28:16.524 INFO:tasks.workunit.client.0.smithi123.stdout:0/662: rename d6/dd/d31/d35/f7b to d6/dd/d15/d28/dd3/fd9 0 2024-03-30T04:28:16.524 INFO:tasks.workunit.client.0.smithi123.stdout:8/665: dwrite da/f46 [4194304,4194304] 0 2024-03-30T04:28:16.525 INFO:tasks.workunit.client.0.smithi123.stdout:3/704: symlink d13/d24/d39/d79/d90/le4 0 2024-03-30T04:28:16.526 INFO:tasks.workunit.client.0.smithi123.stdout:2/755: chown d0/d1f/d27/d68/dd6 821237912 1 2024-03-30T04:28:16.534 INFO:tasks.workunit.client.1.smithi135.stdout:1/208: symlink d16/d29/d2c/d30/d3a/l47 0 2024-03-30T04:28:16.536 INFO:tasks.workunit.client.1.smithi135.stdout:9/232: link d7/d1b/l19 d7/d4f/l50 0 2024-03-30T04:28:16.537 INFO:tasks.workunit.client.0.smithi123.stdout:8/666: symlink da/dc/d14/db7/lc9 0 2024-03-30T04:28:16.539 INFO:tasks.workunit.client.0.smithi123.stdout:3/705: getdents d13/d24/d31/de0 0 2024-03-30T04:28:16.541 INFO:tasks.workunit.client.1.smithi135.stdout:5/278: getdents d1/d45/d52 0 2024-03-30T04:28:16.541 INFO:tasks.workunit.client.0.smithi123.stdout:0/663: creat d6/dd/d15/d28/fda x:0 0 0 2024-03-30T04:28:16.541 INFO:tasks.workunit.client.0.smithi123.stdout:8/667: chown da/d27/d3c/d64/d78 40 1 2024-03-30T04:28:16.542 INFO:tasks.workunit.client.0.smithi123.stdout:2/756: read d0/d1f/d1c/d39/d4b/fa7 [1962693,129885] 0 2024-03-30T04:28:16.543 INFO:tasks.workunit.client.1.smithi135.stdout:1/209: symlink d16/l48 0 2024-03-30T04:28:16.543 INFO:tasks.workunit.client.1.smithi135.stdout:2/247: write d4/d32/d2e/d2f/f30 [523916,65304] 0 2024-03-30T04:28:16.547 INFO:tasks.workunit.client.1.smithi135.stdout:5/279: readlink d1/da/l51 0 2024-03-30T04:28:16.550 INFO:tasks.workunit.client.0.smithi123.stdout:0/664: creat d6/dd/d31/d35/d3a/fdb x:0 0 0 2024-03-30T04:28:16.550 INFO:tasks.workunit.client.1.smithi135.stdout:7/302: dwrite d4/db/d1a/d5c/d2c/f39 [4194304,4194304] 0 2024-03-30T04:28:16.551 INFO:tasks.workunit.client.0.smithi123.stdout:3/706: symlink d13/d24/d31/d23/d63/le5 0 2024-03-30T04:28:16.551 INFO:tasks.workunit.client.0.smithi123.stdout:7/683: dwrite d2/da/d11/d61/f62 [0,4194304] 0 2024-03-30T04:28:16.551 INFO:tasks.workunit.client.0.smithi123.stdout:3/707: readlink d13/l8c 0 2024-03-30T04:28:16.555 INFO:tasks.workunit.client.0.smithi123.stdout:2/757: fsync d0/d1f/d16/f18 0 2024-03-30T04:28:16.555 INFO:tasks.workunit.client.1.smithi135.stdout:5/280: write d1/f4f [4455550,84837] 0 2024-03-30T04:28:16.558 INFO:tasks.workunit.client.0.smithi123.stdout:4/618: sync 2024-03-30T04:28:16.558 INFO:tasks.workunit.client.0.smithi123.stdout:1/729: sync 2024-03-30T04:28:16.559 INFO:tasks.workunit.client.0.smithi123.stdout:3/708: mknod d13/d24/d31/d23/d3a/ce6 0 2024-03-30T04:28:16.560 INFO:tasks.workunit.client.1.smithi135.stdout:9/233: link d7/d1b/f25 d7/d29/f51 0 2024-03-30T04:28:16.560 INFO:tasks.workunit.client.1.smithi135.stdout:8/267: dwrite f5 [4194304,4194304] 0 2024-03-30T04:28:16.561 INFO:tasks.workunit.client.1.smithi135.stdout:1/210: unlink d16/f1d 0 2024-03-30T04:28:16.561 INFO:tasks.workunit.client.0.smithi123.stdout:7/684: mknod d2/da/d11/d34/d76/ce5 0 2024-03-30T04:28:16.562 INFO:tasks.workunit.client.0.smithi123.stdout:1/730: read - db/dd/d42/f6f zero size 2024-03-30T04:28:16.563 INFO:tasks.workunit.client.0.smithi123.stdout:2/758: symlink d0/d1f/d16/lf5 0 2024-03-30T04:28:16.564 INFO:tasks.workunit.client.0.smithi123.stdout:8/668: dread da/dc/da9/fad [0,4194304] 0 2024-03-30T04:28:16.564 INFO:tasks.workunit.client.1.smithi135.stdout:3/281: write d3/d50/d46/f21 [1258113,53218] 0 2024-03-30T04:28:16.566 INFO:tasks.workunit.client.0.smithi123.stdout:3/709: mkdir d13/d24/d39/d79/db9/de7 0 2024-03-30T04:28:16.566 INFO:tasks.workunit.client.0.smithi123.stdout:9/579: dread d6/d1f/d45/fa0 [0,4194304] 0 2024-03-30T04:28:16.567 INFO:tasks.workunit.client.1.smithi135.stdout:5/281: mkdir d1/d3/d2b/d54 0 2024-03-30T04:28:16.569 INFO:tasks.workunit.client.0.smithi123.stdout:4/619: stat d2/d4/d2c/d42/d35/d5d/d78/la8 0 2024-03-30T04:28:16.572 INFO:tasks.workunit.client.1.smithi135.stdout:1/211: readlink d16/d29/d2c/d30/d3a/l40 0 2024-03-30T04:28:16.572 INFO:tasks.workunit.client.1.smithi135.stdout:2/248: symlink d4/d32/d50/l51 0 2024-03-30T04:28:16.573 INFO:tasks.workunit.client.0.smithi123.stdout:4/620: mknod d2/d4/db3/cd7 0 2024-03-30T04:28:16.573 INFO:tasks.workunit.client.1.smithi135.stdout:9/234: creat d7/d41/f52 x:0 0 0 2024-03-30T04:28:16.574 INFO:tasks.workunit.client.1.smithi135.stdout:8/268: mkdir d6/d8/d15/d2a/d47 0 2024-03-30T04:28:16.576 INFO:tasks.workunit.client.0.smithi123.stdout:4/621: write d2/d4/fca [369581,24656] 0 2024-03-30T04:28:16.578 INFO:tasks.workunit.client.1.smithi135.stdout:9/235: fsync d7/d1b/d37/f38 0 2024-03-30T04:28:16.581 INFO:tasks.workunit.client.1.smithi135.stdout:2/249: rmdir d4/d32/d2e 39 2024-03-30T04:28:16.581 INFO:tasks.workunit.client.1.smithi135.stdout:1/212: mknod d16/d29/d2c/c49 0 2024-03-30T04:28:16.582 INFO:tasks.workunit.client.1.smithi135.stdout:9/236: write d7/d1b/f40 [203451,53862] 0 2024-03-30T04:28:16.582 INFO:tasks.workunit.client.1.smithi135.stdout:1/213: read fb [762767,8734] 0 2024-03-30T04:28:16.583 INFO:tasks.workunit.client.0.smithi123.stdout:9/580: dread d6/d49/f6d [0,4194304] 0 2024-03-30T04:28:16.586 INFO:tasks.workunit.client.1.smithi135.stdout:5/282: mkdir d1/d55 0 2024-03-30T04:28:16.590 INFO:tasks.workunit.client.1.smithi135.stdout:1/214: mknod d16/d19/d20/d26/c4a 0 2024-03-30T04:28:16.591 INFO:tasks.workunit.client.1.smithi135.stdout:5/283: creat d1/d7/d1a/d1f/f56 x:0 0 0 2024-03-30T04:28:16.591 INFO:tasks.workunit.client.0.smithi123.stdout:0/665: write d6/f1b [1193352,8136] 0 2024-03-30T04:28:16.592 INFO:tasks.workunit.client.1.smithi135.stdout:3/282: dread d3/d50/d22/f2e [0,4194304] 0 2024-03-30T04:28:16.592 INFO:tasks.workunit.client.1.smithi135.stdout:5/284: stat d1/c34 0 2024-03-30T04:28:16.594 INFO:tasks.workunit.client.1.smithi135.stdout:9/237: dread f6 [0,4194304] 0 2024-03-30T04:28:16.594 INFO:tasks.workunit.client.1.smithi135.stdout:1/215: mknod d16/d29/d2c/d3c/c4b 0 2024-03-30T04:28:16.596 INFO:tasks.workunit.client.1.smithi135.stdout:3/283: creat d3/d50/d46/f59 x:0 0 0 2024-03-30T04:28:16.597 INFO:tasks.workunit.client.1.smithi135.stdout:9/238: chown d7/f11 28729 1 2024-03-30T04:28:16.598 INFO:tasks.workunit.client.1.smithi135.stdout:5/285: creat d1/d55/f57 x:0 0 0 2024-03-30T04:28:16.598 INFO:tasks.workunit.client.1.smithi135.stdout:1/216: truncate d16/f3f 1021228 0 2024-03-30T04:28:16.599 INFO:tasks.workunit.client.0.smithi123.stdout:8/669: write da/dc/f52 [3843990,101081] 0 2024-03-30T04:28:16.600 INFO:tasks.workunit.client.0.smithi123.stdout:3/710: dwrite d13/d24/d31/d23/d3a/d3c/d4e/d5d/f82 [0,4194304] 0 2024-03-30T04:28:16.601 INFO:tasks.workunit.client.1.smithi135.stdout:1/217: fdatasync d16/d29/f35 0 2024-03-30T04:28:16.601 INFO:tasks.workunit.client.0.smithi123.stdout:9/581: creat d6/fc5 x:0 0 0 2024-03-30T04:28:16.602 INFO:tasks.workunit.client.1.smithi135.stdout:3/284: symlink d3/d50/d22/d33/l5a 0 2024-03-30T04:28:16.604 INFO:tasks.workunit.client.0.smithi123.stdout:4/622: dwrite d2/d4/f22 [0,4194304] 0 2024-03-30T04:28:16.604 INFO:tasks.workunit.client.1.smithi135.stdout:1/218: readlink d16/d19/d20/l31 0 2024-03-30T04:28:16.605 INFO:tasks.workunit.client.1.smithi135.stdout:9/239: dread d7/f12 [4194304,4194304] 0 2024-03-30T04:28:16.608 INFO:tasks.workunit.client.0.smithi123.stdout:0/666: symlink d6/dd/d15/d28/ldc 0 2024-03-30T04:28:16.609 INFO:tasks.workunit.client.1.smithi135.stdout:3/285: write d3/f51 [301718,51548] 0 2024-03-30T04:28:16.612 INFO:tasks.workunit.client.0.smithi123.stdout:4/623: write d2/d4/d2c/d42/fc3 [283716,73336] 0 2024-03-30T04:28:16.615 INFO:tasks.workunit.client.0.smithi123.stdout:8/670: mknod da/d27/d3c/d64/d78/d83/cca 0 2024-03-30T04:28:16.616 INFO:tasks.workunit.client.0.smithi123.stdout:3/711: truncate d13/d24/d39/d79/d90/fc2 15275 0 2024-03-30T04:28:16.616 INFO:tasks.workunit.client.1.smithi135.stdout:1/219: write d16/d29/f35 [1639028,75851] 0 2024-03-30T04:28:16.616 INFO:tasks.workunit.client.1.smithi135.stdout:2/250: dwrite d4/d32/d12/f3f [0,4194304] 0 2024-03-30T04:28:16.616 INFO:tasks.workunit.client.1.smithi135.stdout:1/220: write f3 [285745,25783] 0 2024-03-30T04:28:16.618 INFO:tasks.workunit.client.1.smithi135.stdout:9/240: rmdir d7/d1b/d37 39 2024-03-30T04:28:16.618 INFO:tasks.workunit.client.1.smithi135.stdout:8/269: rmdir d6 39 2024-03-30T04:28:16.621 INFO:tasks.workunit.client.1.smithi135.stdout:9/241: read - d7/d1b/f3b zero size 2024-03-30T04:28:16.624 INFO:tasks.workunit.client.0.smithi123.stdout:4/624: write d2/d4/d2c/fa7 [55289,32365] 0 2024-03-30T04:28:16.624 INFO:tasks.workunit.client.0.smithi123.stdout:0/667: rename d6/dd/d16 to d6/dd/d15/d28/dd3/ddd 0 2024-03-30T04:28:16.625 INFO:tasks.workunit.client.0.smithi123.stdout:8/671: unlink da/d27/d56/d6c/d96/fb9 0 2024-03-30T04:28:16.627 INFO:tasks.workunit.client.1.smithi135.stdout:8/270: write d6/d8/f42 [904269,97570] 0 2024-03-30T04:28:16.633 INFO:tasks.workunit.client.1.smithi135.stdout:9/242: dread - d7/d29/f48 zero size 2024-03-30T04:28:16.635 INFO:tasks.workunit.client.1.smithi135.stdout:8/271: chown d6/f12 0 1 2024-03-30T04:28:16.636 INFO:tasks.workunit.client.1.smithi135.stdout:5/286: dwrite d1/d7/f10 [0,4194304] 0 2024-03-30T04:28:16.637 INFO:tasks.workunit.client.0.smithi123.stdout:0/668: creat d6/dd/d15/fde x:0 0 0 2024-03-30T04:28:16.637 INFO:tasks.workunit.client.1.smithi135.stdout:7/303: dwrite d4/db/de/f25 [0,4194304] 0 2024-03-30T04:28:16.643 INFO:tasks.workunit.client.0.smithi123.stdout:3/712: dwrite d13/d24/d31/d23/d63/fd5 [0,4194304] 0 2024-03-30T04:28:16.644 INFO:tasks.workunit.client.0.smithi123.stdout:4/625: dread d2/d4/d2c/d42/f6a [0,4194304] 0 2024-03-30T04:28:16.645 INFO:tasks.workunit.client.1.smithi135.stdout:5/287: stat d1/d7/d1a/d1f/f56 0 2024-03-30T04:28:16.648 INFO:tasks.workunit.client.1.smithi135.stdout:8/272: write d6/fa [693699,85532] 0 2024-03-30T04:28:16.653 INFO:tasks.workunit.client.0.smithi123.stdout:8/672: dwrite da/d27/d3c/d2d/f57 [4194304,4194304] 0 2024-03-30T04:28:16.658 INFO:tasks.workunit.client.0.smithi123.stdout:9/582: dwrite d6/d7c/f89 [0,4194304] 0 2024-03-30T04:28:16.660 INFO:tasks.workunit.client.0.smithi123.stdout:4/626: creat d2/d4/d2c/d42/d35/d5d/d6f/fd8 x:0 0 0 2024-03-30T04:28:16.661 INFO:tasks.workunit.client.1.smithi135.stdout:7/304: mknod d4/db/de/c64 0 2024-03-30T04:28:16.664 INFO:tasks.workunit.client.0.smithi123.stdout:0/669: dread d6/dd/d15/d98/d4e/f68 [0,4194304] 0 2024-03-30T04:28:16.666 INFO:tasks.workunit.client.1.smithi135.stdout:1/221: dread d16/f25 [0,4194304] 0 2024-03-30T04:28:16.668 INFO:tasks.workunit.client.1.smithi135.stdout:2/251: dwrite d4/d32/d2e/f40 [0,4194304] 0 2024-03-30T04:28:16.671 INFO:tasks.workunit.client.1.smithi135.stdout:8/273: symlink d6/d23/l48 0 2024-03-30T04:28:16.672 INFO:tasks.workunit.client.0.smithi123.stdout:0/670: truncate d6/dd/d31/d35/f42 4688698 0 2024-03-30T04:28:16.673 INFO:tasks.workunit.client.1.smithi135.stdout:7/305: mknod d4/db/c65 0 2024-03-30T04:28:16.675 INFO:tasks.workunit.client.0.smithi123.stdout:0/671: stat d6/dd/d31/d35/la1 0 2024-03-30T04:28:16.681 INFO:tasks.workunit.client.1.smithi135.stdout:1/222: unlink d16/f3f 0 2024-03-30T04:28:16.682 INFO:tasks.workunit.client.1.smithi135.stdout:5/288: dread d1/d7/f2c [0,4194304] 0 2024-03-30T04:28:16.683 INFO:tasks.workunit.client.1.smithi135.stdout:7/306: rmdir d4/db/d1a/d5c 39 2024-03-30T04:28:16.684 INFO:tasks.workunit.client.1.smithi135.stdout:5/289: chown d1/da 21 1 2024-03-30T04:28:16.684 INFO:tasks.workunit.client.1.smithi135.stdout:9/243: dwrite d7/d29/f3c [4194304,4194304] 0 2024-03-30T04:28:16.688 INFO:tasks.workunit.client.0.smithi123.stdout:8/673: creat da/dc/d14/d18/fcb x:0 0 0 2024-03-30T04:28:16.688 INFO:tasks.workunit.client.1.smithi135.stdout:1/223: link f3 d16/d1a/f4c 0 2024-03-30T04:28:16.690 INFO:tasks.workunit.client.1.smithi135.stdout:8/274: truncate d6/d8/d15/d2a/f33 247131 0 2024-03-30T04:28:16.690 INFO:tasks.workunit.client.1.smithi135.stdout:1/224: readlink d16/d29/d2c/d30/d3a/l47 0 2024-03-30T04:28:16.692 INFO:tasks.workunit.client.1.smithi135.stdout:5/290: dread - d1/d7/d1a/f3b zero size 2024-03-30T04:28:16.694 INFO:tasks.workunit.client.1.smithi135.stdout:1/225: rmdir d16/d29/d2c/d30 39 2024-03-30T04:28:16.697 INFO:tasks.workunit.client.1.smithi135.stdout:9/244: rename d7/d41/f52 to d7/d1b/f53 0 2024-03-30T04:28:16.698 INFO:tasks.workunit.client.1.smithi135.stdout:8/275: mknod d6/d8/d15/d2a/c49 0 2024-03-30T04:28:16.702 INFO:tasks.workunit.client.0.smithi123.stdout:4/627: dwrite d2/d34/f7c [0,4194304] 0 2024-03-30T04:28:16.703 INFO:tasks.workunit.client.0.smithi123.stdout:5/603: sync 2024-03-30T04:28:16.705 INFO:tasks.workunit.client.1.smithi135.stdout:5/291: rename d1/da/d17/c47 to d1/d7/d11/c58 0 2024-03-30T04:28:16.707 INFO:tasks.workunit.client.1.smithi135.stdout:9/245: creat d7/d1b/d37/f54 x:0 0 0 2024-03-30T04:28:16.708 INFO:tasks.workunit.client.0.smithi123.stdout:0/672: dwrite d6/dd/d31/d35/f42 [0,4194304] 0 2024-03-30T04:28:16.710 INFO:tasks.workunit.client.0.smithi123.stdout:4/628: symlink d2/ld9 0 2024-03-30T04:28:16.711 INFO:tasks.workunit.client.1.smithi135.stdout:1/226: dread d16/f2f [0,4194304] 0 2024-03-30T04:28:16.712 INFO:tasks.workunit.client.0.smithi123.stdout:3/713: dwrite d13/d24/d31/d23/f76 [0,4194304] 0 2024-03-30T04:28:16.717 INFO:tasks.workunit.client.1.smithi135.stdout:1/227: rmdir d16/d29/d2c 39 2024-03-30T04:28:16.718 INFO:tasks.workunit.client.0.smithi123.stdout:9/583: dwrite d6/fb [0,4194304] 0 2024-03-30T04:28:16.719 INFO:tasks.workunit.client.0.smithi123.stdout:3/714: chown d13/d24/d31/d23/d3a/d3c/d4e/d5d/da5 7423311 1 2024-03-30T04:28:16.720 INFO:tasks.workunit.client.0.smithi123.stdout:0/673: dread - d6/dd/d15/d98/d4e/d69/d6e/fd4 zero size 2024-03-30T04:28:16.721 INFO:tasks.workunit.client.1.smithi135.stdout:1/228: truncate d16/f45 283933 0 2024-03-30T04:28:16.726 INFO:tasks.workunit.client.0.smithi123.stdout:5/604: rename dd/fe to dd/d19/d1f/d68/fc6 0 2024-03-30T04:28:16.727 INFO:tasks.workunit.client.1.smithi135.stdout:7/307: dwrite d4/db/d1a/d4b/f45 [0,4194304] 0 2024-03-30T04:28:16.729 INFO:tasks.workunit.client.1.smithi135.stdout:7/308: readlink d4/l27 0 2024-03-30T04:28:16.733 INFO:tasks.workunit.client.1.smithi135.stdout:6/238: sync 2024-03-30T04:28:16.733 INFO:tasks.workunit.client.1.smithi135.stdout:4/252: sync 2024-03-30T04:28:16.733 INFO:tasks.workunit.client.1.smithi135.stdout:0/342: sync 2024-03-30T04:28:16.733 INFO:tasks.workunit.client.1.smithi135.stdout:8/276: dwrite d6/d8/f38 [0,4194304] 0 2024-03-30T04:28:16.734 INFO:tasks.workunit.client.0.smithi123.stdout:3/715: mkdir d13/d24/d31/d23/d3a/d3c/dab/db2/de8 0 2024-03-30T04:28:16.734 INFO:tasks.workunit.client.0.smithi123.stdout:5/605: creat dd/d19/d5f/d99/fc7 x:0 0 0 2024-03-30T04:28:16.735 INFO:tasks.workunit.client.0.smithi123.stdout:0/674: mknod d6/dd/d15/d98/d5d/cdf 0 2024-03-30T04:28:16.741 INFO:tasks.workunit.client.0.smithi123.stdout:5/606: fdatasync dd/d19/d3e/d5b/d62/f7a 0 2024-03-30T04:28:16.744 INFO:tasks.workunit.client.1.smithi135.stdout:2/252: dread d4/d32/d12/d45/f2c [0,4194304] 0 2024-03-30T04:28:16.745 INFO:tasks.workunit.client.0.smithi123.stdout:0/675: mkdir d6/d52/de0 0 2024-03-30T04:28:16.746 INFO:tasks.workunit.client.1.smithi135.stdout:4/253: rmdir d10 39 2024-03-30T04:28:16.748 INFO:tasks.workunit.client.1.smithi135.stdout:8/277: unlink f5 0 2024-03-30T04:28:16.750 INFO:tasks.workunit.client.0.smithi123.stdout:5/607: mknod dd/d41/cc8 0 2024-03-30T04:28:16.751 INFO:tasks.workunit.client.1.smithi135.stdout:6/239: unlink d2/dd/d18/f25 0 2024-03-30T04:28:16.751 INFO:tasks.workunit.client.0.smithi123.stdout:0/676: write d6/d52/f73 [1083401,61774] 0 2024-03-30T04:28:16.753 INFO:tasks.workunit.client.0.smithi123.stdout:3/716: creat d13/fe9 x:0 0 0 2024-03-30T04:28:16.755 INFO:tasks.workunit.client.0.smithi123.stdout:9/584: dwrite d6/d23/f8d [0,4194304] 0 2024-03-30T04:28:16.757 INFO:tasks.workunit.client.1.smithi135.stdout:0/343: symlink d0/d19/d2d/d45/l6d 0 2024-03-30T04:28:16.760 INFO:tasks.workunit.client.1.smithi135.stdout:0/344: readlink d0/d13/l27 0 2024-03-30T04:28:16.760 INFO:tasks.workunit.client.1.smithi135.stdout:7/309: rename d4/db/de/f25 to d4/db/d1a/f66 0 2024-03-30T04:28:16.760 INFO:tasks.workunit.client.0.smithi123.stdout:8/674: dread da/dc/d14/d18/f86 [0,4194304] 0 2024-03-30T04:28:16.763 INFO:tasks.workunit.client.1.smithi135.stdout:6/240: mknod d2/dd/d14/d38/c49 0 2024-03-30T04:28:16.764 INFO:tasks.workunit.client.0.smithi123.stdout:0/677: mknod d6/d52/ce1 0 2024-03-30T04:28:16.766 INFO:tasks.workunit.client.0.smithi123.stdout:3/717: mkdir d13/d24/d39/d79/dea 0 2024-03-30T04:28:16.766 INFO:tasks.workunit.client.1.smithi135.stdout:8/278: getdents d6/d8/d15/d2a/d47 0 2024-03-30T04:28:16.768 INFO:tasks.workunit.client.1.smithi135.stdout:0/345: write d0/d5/d9/f2e [1166736,109707] 0 2024-03-30T04:28:16.769 INFO:tasks.workunit.client.0.smithi123.stdout:8/675: chown da/dc/d14/d18/d21/ca3 1101728 1 2024-03-30T04:28:16.769 INFO:tasks.workunit.client.1.smithi135.stdout:1/229: dread f5 [0,4194304] 0 2024-03-30T04:28:16.770 INFO:tasks.workunit.client.0.smithi123.stdout:0/678: stat d6/dd/d15/d28/d4b/d8a/lce 0 2024-03-30T04:28:16.770 INFO:tasks.workunit.client.0.smithi123.stdout:9/585: symlink d6/d1f/da3/lc6 0 2024-03-30T04:28:16.771 INFO:tasks.workunit.client.1.smithi135.stdout:7/310: stat d4/f29 0 2024-03-30T04:28:16.773 INFO:tasks.workunit.client.1.smithi135.stdout:4/254: rename d10/d1a/d32/f3e to d10/d1a/d3d/f5a 0 2024-03-30T04:28:16.776 INFO:tasks.workunit.client.0.smithi123.stdout:4/629: write d2/d4/d2c/d42/d89/f94 [52397,89901] 0 2024-03-30T04:28:16.776 INFO:tasks.workunit.client.0.smithi123.stdout:3/718: creat d13/d24/d31/d23/feb x:0 0 0 2024-03-30T04:28:16.778 INFO:tasks.workunit.client.0.smithi123.stdout:8/676: symlink da/dc/da9/lcc 0 2024-03-30T04:28:16.781 INFO:tasks.workunit.client.1.smithi135.stdout:5/292: dread d1/d7/f48 [0,4194304] 0 2024-03-30T04:28:16.782 INFO:tasks.workunit.client.1.smithi135.stdout:6/241: creat d2/dd/d14/d38/f4a x:0 0 0 2024-03-30T04:28:16.783 INFO:tasks.workunit.client.1.smithi135.stdout:0/346: unlink d0/d1/l3c 0 2024-03-30T04:28:16.784 INFO:tasks.workunit.client.1.smithi135.stdout:0/347: chown d0/d5/d42 385 1 2024-03-30T04:28:16.785 INFO:tasks.workunit.client.1.smithi135.stdout:9/246: dread d7/d29/f51 [0,4194304] 0 2024-03-30T04:28:16.785 INFO:tasks.workunit.client.0.smithi123.stdout:8/677: rename da/dc/da9/lcc to da/dc/lcd 0 2024-03-30T04:28:16.787 INFO:tasks.workunit.client.0.smithi123.stdout:5/608: link dd/l23 dd/d19/d1f/d35/d3a/d79/lc9 0 2024-03-30T04:28:16.787 INFO:tasks.workunit.client.1.smithi135.stdout:1/230: creat d16/d1a/f4d x:0 0 0 2024-03-30T04:28:16.789 INFO:tasks.workunit.client.1.smithi135.stdout:4/255: write d10/d1a/f21 [226806,81846] 0 2024-03-30T04:28:16.789 INFO:tasks.workunit.client.0.smithi123.stdout:0/679: creat d6/dd/d15/d98/d4e/d69/fe2 x:0 0 0 2024-03-30T04:28:16.790 INFO:tasks.workunit.client.1.smithi135.stdout:7/311: symlink d4/db/d1a/d4b/d40/l67 0 2024-03-30T04:28:16.791 INFO:tasks.workunit.client.0.smithi123.stdout:4/630: dread d2/d4/d2c/f84 [0,4194304] 0 2024-03-30T04:28:16.793 INFO:tasks.workunit.client.1.smithi135.stdout:5/293: creat d1/d7/f59 x:0 0 0 2024-03-30T04:28:16.795 INFO:tasks.workunit.client.0.smithi123.stdout:5/609: creat dd/d19/d5f/fca x:0 0 0 2024-03-30T04:28:16.795 INFO:tasks.workunit.client.1.smithi135.stdout:8/279: creat d6/d8/d15/d2a/d47/f4a x:0 0 0 2024-03-30T04:28:16.796 INFO:tasks.workunit.client.1.smithi135.stdout:5/294: write d1/d7/d1a/d1f/f56 [884833,47517] 0 2024-03-30T04:28:16.801 INFO:tasks.workunit.client.1.smithi135.stdout:3/286: sync 2024-03-30T04:28:16.804 INFO:tasks.workunit.client.0.smithi123.stdout:6/735: sync 2024-03-30T04:28:16.806 INFO:tasks.workunit.client.1.smithi135.stdout:9/247: mkdir d7/d55 0 2024-03-30T04:28:16.807 INFO:tasks.workunit.client.1.smithi135.stdout:2/253: truncate d4/d32/d12/f31 704546 0 2024-03-30T04:28:16.810 INFO:tasks.workunit.client.1.smithi135.stdout:5/295: mknod d1/d3/d2b/c5a 0 2024-03-30T04:28:16.812 INFO:tasks.workunit.client.0.smithi123.stdout:4/631: write d2/d4/d2c/f67 [4298849,12663] 0 2024-03-30T04:28:16.812 INFO:tasks.workunit.client.1.smithi135.stdout:4/256: dwrite d10/d1a/f21 [0,4194304] 0 2024-03-30T04:28:16.814 INFO:tasks.workunit.client.0.smithi123.stdout:6/736: creat d10/d82/d8e/dc7/d5b/ffc x:0 0 0 2024-03-30T04:28:16.814 INFO:tasks.workunit.client.1.smithi135.stdout:3/287: creat d3/d50/f5b x:0 0 0 2024-03-30T04:28:16.814 INFO:tasks.workunit.client.1.smithi135.stdout:7/312: truncate d4/db/d1a/d4b/d40/d3a/f58 1144162 0 2024-03-30T04:28:16.816 INFO:tasks.workunit.client.0.smithi123.stdout:9/586: dread d6/d23/f40 [4194304,4194304] 0 2024-03-30T04:28:16.817 INFO:tasks.workunit.client.1.smithi135.stdout:7/313: write d4/db/de/f2b [24757,45287] 0 2024-03-30T04:28:16.819 INFO:tasks.workunit.client.1.smithi135.stdout:7/314: chown d4/db/d1a/d4b/d40/l67 126912 1 2024-03-30T04:28:16.820 INFO:tasks.workunit.client.0.smithi123.stdout:0/680: dwrite d6/dd/d15/d28/fda [0,4194304] 0 2024-03-30T04:28:16.821 INFO:tasks.workunit.client.0.smithi123.stdout:4/632: write d2/d4/d2c/fa7 [294081,10160] 0 2024-03-30T04:28:16.821 INFO:tasks.workunit.client.1.smithi135.stdout:7/315: chown d4/db/d1a/d4b/l21 0 1 2024-03-30T04:28:16.822 INFO:tasks.workunit.client.1.smithi135.stdout:5/296: creat d1/d3/d2b/f5b x:0 0 0 2024-03-30T04:28:16.823 INFO:tasks.workunit.client.1.smithi135.stdout:6/242: write d2/d36/f44 [1702050,77515] 0 2024-03-30T04:28:16.825 INFO:tasks.workunit.client.1.smithi135.stdout:4/257: fsync d10/f17 0 2024-03-30T04:28:16.826 INFO:tasks.workunit.client.0.smithi123.stdout:3/719: dwrite d13/d24/d31/d23/fc8 [0,4194304] 0 2024-03-30T04:28:16.827 INFO:tasks.workunit.client.1.smithi135.stdout:9/248: mknod d7/d1b/c56 0 2024-03-30T04:28:16.828 INFO:tasks.workunit.client.0.smithi123.stdout:6/737: rename d10/d82/d8e/dc7/d5b/f8d to d10/d82/dc1/dd2/ffd 0 2024-03-30T04:28:16.830 INFO:tasks.workunit.client.1.smithi135.stdout:0/348: getdents d0/d5/d9/d23/d58/d55 0 2024-03-30T04:28:16.830 INFO:tasks.workunit.client.1.smithi135.stdout:1/231: getdents d16/d19/d20/d26 0 2024-03-30T04:28:16.830 INFO:tasks.workunit.client.0.smithi123.stdout:9/587: unlink d6/d1f/d45/d97/d69/l87 0 2024-03-30T04:28:16.831 INFO:tasks.workunit.client.0.smithi123.stdout:4/633: read - d2/d4/d2c/d42/dad/d91/fb1 zero size 2024-03-30T04:28:16.831 INFO:tasks.workunit.client.0.smithi123.stdout:0/681: dread - d6/dd/d15/fc8 zero size 2024-03-30T04:28:16.835 INFO:tasks.workunit.client.1.smithi135.stdout:3/288: mkdir d3/d54/d5c 0 2024-03-30T04:28:16.835 INFO:tasks.workunit.client.0.smithi123.stdout:6/738: creat d10/d82/d8e/ffe x:0 0 0 2024-03-30T04:28:16.836 INFO:tasks.workunit.client.1.smithi135.stdout:5/297: rename d1/d7/d11/d3a to d1/d7/d1a/d46/d5c 0 2024-03-30T04:28:16.837 INFO:tasks.workunit.client.0.smithi123.stdout:3/720: chown d13/d24/d31/d23/d3a/d3c/d4e/cc6 30461957 1 2024-03-30T04:28:16.840 INFO:tasks.workunit.client.0.smithi123.stdout:6/739: fsync d10/d82/d8e/dc7/fd7 0 2024-03-30T04:28:16.841 INFO:tasks.workunit.client.1.smithi135.stdout:6/243: dread - d2/dd/d14/f15 zero size 2024-03-30T04:28:16.843 INFO:tasks.workunit.client.0.smithi123.stdout:9/588: creat d6/d7c/d82/fc7 x:0 0 0 2024-03-30T04:28:16.844 INFO:tasks.workunit.client.1.smithi135.stdout:4/258: symlink d10/l5b 0 2024-03-30T04:28:16.845 INFO:tasks.workunit.client.0.smithi123.stdout:2/759: sync 2024-03-30T04:28:16.845 INFO:tasks.workunit.client.0.smithi123.stdout:7/685: sync 2024-03-30T04:28:16.845 INFO:tasks.workunit.client.0.smithi123.stdout:1/731: sync 2024-03-30T04:28:16.845 INFO:tasks.workunit.client.0.smithi123.stdout:6/740: chown d10/d82/l57 573 1 2024-03-30T04:28:16.846 INFO:tasks.workunit.client.1.smithi135.stdout:0/349: rename d0/d34/d3f/c47 to d0/d19/d2d/d45/c6e 0 2024-03-30T04:28:16.847 INFO:tasks.workunit.client.1.smithi135.stdout:5/298: mknod d1/d3/c5d 0 2024-03-30T04:28:16.847 INFO:tasks.workunit.client.1.smithi135.stdout:7/316: dread d4/db/d1a/d5c/f4e [0,4194304] 0 2024-03-30T04:28:16.848 INFO:tasks.workunit.client.0.smithi123.stdout:8/678: dread f8 [0,4194304] 0 2024-03-30T04:28:16.849 INFO:tasks.workunit.client.1.smithi135.stdout:0/350: fdatasync d0/d1/d16/f4b 0 2024-03-30T04:28:16.850 INFO:tasks.workunit.client.0.smithi123.stdout:4/634: symlink d2/d4/d7/lda 0 2024-03-30T04:28:16.851 INFO:tasks.workunit.client.0.smithi123.stdout:1/732: write db/dd/d5f/d6d/d85/dd4/fe6 [1046543,118901] 0 2024-03-30T04:28:16.852 INFO:tasks.workunit.client.0.smithi123.stdout:2/760: symlink d0/d1f/d82/lf6 0 2024-03-30T04:28:16.855 INFO:tasks.workunit.client.1.smithi135.stdout:6/244: creat d2/dd/d18/f4b x:0 0 0 2024-03-30T04:28:16.858 INFO:tasks.workunit.client.0.smithi123.stdout:0/682: rename d6/dd/d18/d2f/laa to d6/dd/d18/d2f/da3/le3 0 2024-03-30T04:28:16.859 INFO:tasks.workunit.client.0.smithi123.stdout:2/761: mkdir d0/d1f/d27/df7 0 2024-03-30T04:28:16.859 INFO:tasks.workunit.client.0.smithi123.stdout:4/635: dread - d2/d26/f60 zero size 2024-03-30T04:28:16.860 INFO:tasks.workunit.client.1.smithi135.stdout:1/232: creat d16/d29/d2c/d30/f4e x:0 0 0 2024-03-30T04:28:16.862 INFO:tasks.workunit.client.1.smithi135.stdout:8/280: truncate d6/d8/f1d 5656464 0 2024-03-30T04:28:16.863 INFO:tasks.workunit.client.1.smithi135.stdout:7/317: write d4/f54 [279136,53902] 0 2024-03-30T04:28:16.865 INFO:tasks.workunit.client.1.smithi135.stdout:3/289: rename d3/d10/d16/d23/c2b to d3/d50/d22/d33/c5d 0 2024-03-30T04:28:16.865 INFO:tasks.workunit.client.0.smithi123.stdout:1/733: fdatasync db/dd/d42/d46/d39/f57 0 2024-03-30T04:28:16.867 INFO:tasks.workunit.client.1.smithi135.stdout:0/351: unlink d0/d5/d9/f15 0 2024-03-30T04:28:16.868 INFO:tasks.workunit.client.0.smithi123.stdout:2/762: unlink d0/d1f/d1c/d20/fa2 0 2024-03-30T04:28:16.871 INFO:tasks.workunit.client.1.smithi135.stdout:3/290: chown d3/f4 1046695315 1 2024-03-30T04:28:16.872 INFO:tasks.workunit.client.1.smithi135.stdout:5/299: dwrite d1/d7/ff [4194304,4194304] 0 2024-03-30T04:28:16.873 INFO:tasks.workunit.client.1.smithi135.stdout:0/352: chown d0/d5/d10/f11 491575426 1 2024-03-30T04:28:16.874 INFO:tasks.workunit.client.0.smithi123.stdout:6/741: fdatasync d10/d82/dc1/dd2/ffd 0 2024-03-30T04:28:16.874 INFO:tasks.workunit.client.1.smithi135.stdout:1/233: fdatasync d16/f2f 0 2024-03-30T04:28:16.877 INFO:tasks.workunit.client.0.smithi123.stdout:2/763: symlink d0/d1f/d5e/lf8 0 2024-03-30T04:28:16.877 INFO:tasks.workunit.client.1.smithi135.stdout:4/259: creat d10/d1a/f5c x:0 0 0 2024-03-30T04:28:16.882 INFO:tasks.workunit.client.1.smithi135.stdout:6/245: creat d2/d2e/f4c x:0 0 0 2024-03-30T04:28:16.882 INFO:tasks.workunit.client.1.smithi135.stdout:3/291: creat d3/d10/f5e x:0 0 0 2024-03-30T04:28:16.884 INFO:tasks.workunit.client.1.smithi135.stdout:1/234: rmdir d16/d1a 39 2024-03-30T04:28:16.885 INFO:tasks.workunit.client.0.smithi123.stdout:9/589: dwrite d6/d1f/d45/d4c/f54 [4194304,4194304] 0 2024-03-30T04:28:16.886 INFO:tasks.workunit.client.0.smithi123.stdout:1/734: creat db/dd/d42/d46/ff5 x:0 0 0 2024-03-30T04:28:16.889 INFO:tasks.workunit.client.0.smithi123.stdout:0/683: dread d6/dd/d18/d2f/f58 [0,4194304] 0 2024-03-30T04:28:16.891 INFO:tasks.workunit.client.1.smithi135.stdout:4/260: symlink d10/d1a/d42/l5d 0 2024-03-30T04:28:16.891 INFO:tasks.workunit.client.1.smithi135.stdout:7/318: creat d4/db/de/d5a/f68 x:0 0 0 2024-03-30T04:28:16.892 INFO:tasks.workunit.client.1.smithi135.stdout:1/235: fsync d16/f25 0 2024-03-30T04:28:16.895 INFO:tasks.workunit.client.1.smithi135.stdout:0/353: creat d0/d5/d9/f6f x:0 0 0 2024-03-30T04:28:16.896 INFO:tasks.workunit.client.0.smithi123.stdout:2/764: fsync d0/d1f/d1c/f5b 0 2024-03-30T04:28:16.897 INFO:tasks.workunit.client.1.smithi135.stdout:6/246: write d2/dd/d14/f3f [182818,102896] 0 2024-03-30T04:28:16.897 INFO:tasks.workunit.client.1.smithi135.stdout:3/292: mknod d3/d50/d46/d4c/c5f 0 2024-03-30T04:28:16.898 INFO:tasks.workunit.client.0.smithi123.stdout:7/686: dwrite d2/dc/d49/d68/f78 [0,4194304] 0 2024-03-30T04:28:16.898 INFO:tasks.workunit.client.0.smithi123.stdout:1/735: chown db/dd/d5f/d6d/d85/dd4 157065330 1 2024-03-30T04:28:16.899 INFO:tasks.workunit.client.0.smithi123.stdout:3/721: dread d13/f1b [0,4194304] 0 2024-03-30T04:28:16.900 INFO:tasks.workunit.client.1.smithi135.stdout:7/319: chown d4/db/d1a/d4b/d40/l67 628646 1 2024-03-30T04:28:16.901 INFO:tasks.workunit.client.1.smithi135.stdout:2/254: dread d4/d6/f35 [0,4194304] 0 2024-03-30T04:28:16.901 INFO:tasks.workunit.client.0.smithi123.stdout:2/765: readlink d0/lb 0 2024-03-30T04:28:16.906 INFO:tasks.workunit.client.0.smithi123.stdout:9/590: rename d6/lae to d6/d1f/d72/d7d/lc8 0 2024-03-30T04:28:16.907 INFO:tasks.workunit.client.0.smithi123.stdout:7/687: fsync d2/dc/d1c/f1d 0 2024-03-30T04:28:16.909 INFO:tasks.workunit.client.1.smithi135.stdout:6/247: chown d2/dd/d14/l35 7764 1 2024-03-30T04:28:16.910 INFO:tasks.workunit.client.0.smithi123.stdout:8/679: dread da/d27/f53 [0,4194304] 0 2024-03-30T04:28:16.911 INFO:tasks.workunit.client.0.smithi123.stdout:3/722: mknod d13/d24/d39/d79/db9/cec 0 2024-03-30T04:28:16.912 INFO:tasks.workunit.client.1.smithi135.stdout:8/281: dwrite d6/d8/d15/d2a/f37 [0,4194304] 0 2024-03-30T04:28:16.914 INFO:tasks.workunit.client.0.smithi123.stdout:1/736: unlink db/dd/d42/d46/d56/dce/fef 0 2024-03-30T04:28:16.915 INFO:tasks.workunit.client.1.smithi135.stdout:6/248: read d2/d2e/f31 [732103,35456] 0 2024-03-30T04:28:16.916 INFO:tasks.workunit.client.0.smithi123.stdout:8/680: fsync da/d27/d3c/d64/f81 0 2024-03-30T04:28:16.917 INFO:tasks.workunit.client.0.smithi123.stdout:6/742: dwrite d10/d82/d8e/dc7/fa4 [0,4194304] 0 2024-03-30T04:28:16.917 INFO:tasks.workunit.client.0.smithi123.stdout:4/636: dread d2/d34/f3f [0,4194304] 0 2024-03-30T04:28:16.917 INFO:tasks.workunit.client.1.smithi135.stdout:3/293: creat d3/d50/d46/f60 x:0 0 0 2024-03-30T04:28:16.920 INFO:tasks.workunit.client.0.smithi123.stdout:6/743: chown d10/d82/d8e/dc7/d5b 2692 1 2024-03-30T04:28:16.925 INFO:tasks.workunit.client.0.smithi123.stdout:9/591: mkdir d6/dc9 0 2024-03-30T04:28:16.925 INFO:tasks.workunit.client.1.smithi135.stdout:1/236: rename d16/d1a/l3e to d16/d29/d2c/d30/d3a/l4f 0 2024-03-30T04:28:16.928 INFO:tasks.workunit.client.0.smithi123.stdout:6/744: truncate d10/d82/d8e/dc7/d5b/fef 388735 0 2024-03-30T04:28:16.928 INFO:tasks.workunit.client.0.smithi123.stdout:3/723: rmdir d13/d24/d31/d23/d3a/d3c/d66 39 2024-03-30T04:28:16.930 INFO:tasks.workunit.client.0.smithi123.stdout:3/724: chown c9 20209 1 2024-03-30T04:28:16.931 INFO:tasks.workunit.client.1.smithi135.stdout:5/300: write d1/d7/f2c [269813,3877] 0 2024-03-30T04:28:16.931 INFO:tasks.workunit.client.1.smithi135.stdout:7/320: creat d4/db/f69 x:0 0 0 2024-03-30T04:28:16.932 INFO:tasks.workunit.client.0.smithi123.stdout:8/681: unlink da/dc/f32 0 2024-03-30T04:28:16.932 INFO:tasks.workunit.client.0.smithi123.stdout:3/725: write d13/d24/d31/d23/d3a/d3c/f73 [751916,67443] 0 2024-03-30T04:28:16.934 INFO:tasks.workunit.client.0.smithi123.stdout:8/682: fsync da/dc/f20 0 2024-03-30T04:28:16.934 INFO:tasks.workunit.client.1.smithi135.stdout:1/237: readlink d16/d29/d2c/d3c/l34 0 2024-03-30T04:28:16.935 INFO:tasks.workunit.client.1.smithi135.stdout:3/294: rename d3/f3c to d3/d50/d22/d33/f61 0 2024-03-30T04:28:16.941 INFO:tasks.workunit.client.0.smithi123.stdout:1/737: truncate db/dd/d42/d46/d39/fad 553529 0 2024-03-30T04:28:16.942 INFO:tasks.workunit.client.1.smithi135.stdout:3/295: mkdir d3/d50/d22/d33/d62 0 2024-03-30T04:28:16.943 INFO:tasks.workunit.client.1.smithi135.stdout:6/249: dwrite d2/d2e/f37 [0,4194304] 0 2024-03-30T04:28:16.944 INFO:tasks.workunit.client.0.smithi123.stdout:7/688: dwrite d2/da/d11/d61/f72 [0,4194304] 0 2024-03-30T04:28:16.945 INFO:tasks.workunit.client.1.smithi135.stdout:5/301: rename d1/d7/d1a/d46 to d1/d3/d2b/d54/d5e 0 2024-03-30T04:28:16.947 INFO:tasks.workunit.client.0.smithi123.stdout:8/683: rmdir da/d27/d3c/d64/d78 39 2024-03-30T04:28:16.947 INFO:tasks.workunit.client.0.smithi123.stdout:0/684: link d6/dd/d15/d98/d4e/d9a/cb4 d6/dd/d31/d35/d3a/ce4 0 2024-03-30T04:28:16.948 INFO:tasks.workunit.client.0.smithi123.stdout:2/766: getdents d0/d37/d57 0 2024-03-30T04:28:16.951 INFO:tasks.workunit.client.1.smithi135.stdout:3/296: mknod d3/c63 0 2024-03-30T04:28:16.953 INFO:tasks.workunit.client.0.smithi123.stdout:9/592: mknod d6/d1f/d72/cca 0 2024-03-30T04:28:16.956 INFO:tasks.workunit.client.0.smithi123.stdout:0/685: write d6/dd/d15/d98/d5d/f83 [646785,64663] 0 2024-03-30T04:28:16.956 INFO:tasks.workunit.client.1.smithi135.stdout:3/297: fdatasync d3/f5 0 2024-03-30T04:28:16.957 INFO:tasks.workunit.client.0.smithi123.stdout:0/686: chown d6/dd 889139 1 2024-03-30T04:28:16.959 INFO:tasks.workunit.client.1.smithi135.stdout:6/250: rmdir d2/d45 0 2024-03-30T04:28:16.959 INFO:tasks.workunit.client.0.smithi123.stdout:1/738: truncate db/dd/d42/d46/d39/f93 216889 0 2024-03-30T04:28:16.961 INFO:tasks.workunit.client.1.smithi135.stdout:4/261: dwrite d10/d1a/f21 [4194304,4194304] 0 2024-03-30T04:28:16.964 INFO:tasks.workunit.client.0.smithi123.stdout:7/689: rename d2/da/l30 to d2/d4d/dd0/le6 0 2024-03-30T04:28:16.967 INFO:tasks.workunit.client.0.smithi123.stdout:7/690: fsync d2/dc/d49/dbc/fc5 0 2024-03-30T04:28:16.967 INFO:tasks.workunit.client.0.smithi123.stdout:3/726: creat d13/d24/fed x:0 0 0 2024-03-30T04:28:16.967 INFO:tasks.workunit.client.1.smithi135.stdout:3/298: dread - d3/d50/d46/f60 zero size 2024-03-30T04:28:16.970 INFO:tasks.workunit.client.1.smithi135.stdout:1/238: dwrite d16/d29/d2c/d30/f4e [0,4194304] 0 2024-03-30T04:28:16.970 INFO:tasks.workunit.client.0.smithi123.stdout:7/691: dread - d2/da/d11/d61/d55/d94/fd8 zero size 2024-03-30T04:28:16.971 INFO:tasks.workunit.client.0.smithi123.stdout:4/637: truncate d2/d34/f4c 919594 0 2024-03-30T04:28:16.973 INFO:tasks.workunit.client.0.smithi123.stdout:6/745: getdents d10/d82/d8e/dc7/d42 0 2024-03-30T04:28:16.974 INFO:tasks.workunit.client.1.smithi135.stdout:4/262: rename d10/d1a/f2e to d10/d1a/d42/f5e 0 2024-03-30T04:28:16.975 INFO:tasks.workunit.client.0.smithi123.stdout:0/687: mknod d6/dd/d18/ce5 0 2024-03-30T04:28:16.976 INFO:tasks.workunit.client.1.smithi135.stdout:7/321: dread f3 [0,4194304] 0 2024-03-30T04:28:16.980 INFO:tasks.workunit.client.0.smithi123.stdout:0/688: chown d6/dd/d15/d28/d4b/f82 782667 1 2024-03-30T04:28:16.982 INFO:tasks.workunit.client.0.smithi123.stdout:0/689: fsync d6/f1b 0 2024-03-30T04:28:16.982 INFO:tasks.workunit.client.1.smithi135.stdout:1/239: mkdir d16/d29/d2c/d43/d50 0 2024-03-30T04:28:16.987 INFO:tasks.workunit.client.1.smithi135.stdout:7/322: mknod d4/db/d1a/d5c/d5d/c6a 0 2024-03-30T04:28:16.988 INFO:tasks.workunit.client.0.smithi123.stdout:4/638: chown d2/d4/d2c/l72 58210 1 2024-03-30T04:28:16.988 INFO:tasks.workunit.client.1.smithi135.stdout:1/240: read d16/f25 [4121005,102965] 0 2024-03-30T04:28:16.989 INFO:tasks.workunit.client.1.smithi135.stdout:8/282: dread d6/d8/d14/fe [0,4194304] 0 2024-03-30T04:28:16.990 INFO:tasks.workunit.client.1.smithi135.stdout:5/302: dwrite d1/f3e [0,4194304] 0 2024-03-30T04:28:16.991 INFO:tasks.workunit.client.0.smithi123.stdout:9/593: dwrite d6/fd [4194304,4194304] 0 2024-03-30T04:28:16.992 INFO:tasks.workunit.client.0.smithi123.stdout:9/594: stat d6/d23 0 2024-03-30T04:28:16.994 INFO:tasks.workunit.client.1.smithi135.stdout:7/323: fsync d4/db/d1a/d5c/d2c/f39 0 2024-03-30T04:28:16.995 INFO:tasks.workunit.client.0.smithi123.stdout:2/767: rename d0/d1f/d1c/d39/d4b/fb1 to d0/d1f/d1c/d39/de4/ff9 0 2024-03-30T04:28:16.995 INFO:tasks.workunit.client.0.smithi123.stdout:6/746: dwrite d10/d20/f2b [0,4194304] 0 2024-03-30T04:28:17.002 INFO:tasks.workunit.client.0.smithi123.stdout:8/684: truncate da/dc/f13 8376029 0 2024-03-30T04:28:17.005 INFO:tasks.workunit.client.0.smithi123.stdout:1/739: write db/dd/d5f/d6d/fcb [996889,32010] 0 2024-03-30T04:28:17.007 INFO:tasks.workunit.client.0.smithi123.stdout:0/690: mknod d6/dd/d15/d28/d4b/d8a/ce6 0 2024-03-30T04:28:17.007 INFO:tasks.workunit.client.0.smithi123.stdout:4/639: stat d2/d34/f93 0 2024-03-30T04:28:17.008 INFO:tasks.workunit.client.0.smithi123.stdout:2/768: write d0/d1f/d27/f8b [3658534,81391] 0 2024-03-30T04:28:17.009 INFO:tasks.workunit.client.0.smithi123.stdout:5/610: sync 2024-03-30T04:28:17.011 INFO:tasks.workunit.client.0.smithi123.stdout:6/747: symlink d10/d82/dd6/lff 0 2024-03-30T04:28:17.012 INFO:tasks.workunit.client.0.smithi123.stdout:4/640: fsync d2/d4/d2c/d42/dad/d91/fb1 0 2024-03-30T04:28:17.013 INFO:tasks.workunit.client.0.smithi123.stdout:8/685: creat da/d27/d95/fce x:0 0 0 2024-03-30T04:28:17.013 INFO:tasks.workunit.client.1.smithi135.stdout:4/263: dwrite fb [0,4194304] 0 2024-03-30T04:28:17.014 INFO:tasks.workunit.client.1.smithi135.stdout:8/283: getdents d6/d8/d15/d2a/d35 0 2024-03-30T04:28:17.016 INFO:tasks.workunit.client.0.smithi123.stdout:5/611: stat dd/d19/d1f/fa5 0 2024-03-30T04:28:17.016 INFO:tasks.workunit.client.0.smithi123.stdout:2/769: creat d0/d1f/d16/ffa x:0 0 0 2024-03-30T04:28:17.016 INFO:tasks.workunit.client.0.smithi123.stdout:9/595: rename d6/d49/f6d to d6/d1f/fcb 0 2024-03-30T04:28:17.017 INFO:tasks.workunit.client.0.smithi123.stdout:3/727: getdents d13/d24/d31/d23/d3a/d3c/d66/da9 0 2024-03-30T04:28:17.019 INFO:tasks.workunit.client.0.smithi123.stdout:0/691: creat d6/dd/d15/d98/d4e/d69/d6e/dd8/fe7 x:0 0 0 2024-03-30T04:28:17.019 INFO:tasks.workunit.client.1.smithi135.stdout:5/303: dread d1/f37 [0,4194304] 0 2024-03-30T04:28:17.022 INFO:tasks.workunit.client.1.smithi135.stdout:8/284: link d6/d8/f1c d6/d8/d15/d2a/f4b 0 2024-03-30T04:28:17.023 INFO:tasks.workunit.client.0.smithi123.stdout:7/692: write d2/da/d11/f1e [2889527,6819] 0 2024-03-30T04:28:17.026 INFO:tasks.workunit.client.0.smithi123.stdout:6/748: write d10/d82/dc1/dd2/fd9 [745606,130063] 0 2024-03-30T04:28:17.028 INFO:tasks.workunit.client.0.smithi123.stdout:8/686: creat da/d27/d3c/d2d/fcf x:0 0 0 2024-03-30T04:28:17.028 INFO:tasks.workunit.client.0.smithi123.stdout:9/596: truncate d6/f15 4607791 0 2024-03-30T04:28:17.028 INFO:tasks.workunit.client.1.smithi135.stdout:8/285: creat d6/d8/d15/d1f/f4c x:0 0 0 2024-03-30T04:28:17.030 INFO:tasks.workunit.client.0.smithi123.stdout:5/612: mknod dd/d41/ccb 0 2024-03-30T04:28:17.030 INFO:tasks.workunit.client.1.smithi135.stdout:7/324: dwrite d4/db/d1a/f66 [0,4194304] 0 2024-03-30T04:28:17.034 INFO:tasks.workunit.client.1.smithi135.stdout:8/286: write d6/d8/f3b [256989,94416] 0 2024-03-30T04:28:17.039 INFO:tasks.workunit.client.0.smithi123.stdout:2/770: symlink d0/d1f/d1c/d39/df2/lfb 0 2024-03-30T04:28:17.040 INFO:tasks.workunit.client.1.smithi135.stdout:8/287: chown d6/d8/d15/f2b 8808 1 2024-03-30T04:28:17.041 INFO:tasks.workunit.client.0.smithi123.stdout:9/597: unlink d6/d1f/d72/d7d/c7f 0 2024-03-30T04:28:17.042 INFO:tasks.workunit.client.1.smithi135.stdout:5/304: dwrite d1/d7/ff [4194304,4194304] 0 2024-03-30T04:28:17.043 INFO:tasks.workunit.client.1.smithi135.stdout:6/251: dread d2/dd/d18/f24 [0,4194304] 0 2024-03-30T04:28:17.047 INFO:tasks.workunit.client.1.smithi135.stdout:6/252: write d2/dd/d18/f22 [546455,67972] 0 2024-03-30T04:28:17.048 INFO:tasks.workunit.client.0.smithi123.stdout:6/749: rmdir d10/d82/dc1/d6a/d7a/d99/dba 39 2024-03-30T04:28:17.049 INFO:tasks.workunit.client.0.smithi123.stdout:8/687: dread da/dc/d14/d18/fac [0,4194304] 0 2024-03-30T04:28:17.051 INFO:tasks.workunit.client.1.smithi135.stdout:6/253: truncate d2/dd/f17 1691529 0 2024-03-30T04:28:17.052 INFO:tasks.workunit.client.0.smithi123.stdout:5/613: truncate f9 7816457 0 2024-03-30T04:28:17.054 INFO:tasks.workunit.client.0.smithi123.stdout:0/692: mkdir d6/dd/d15/d28/d4b/dad/de8 0 2024-03-30T04:28:17.055 INFO:tasks.workunit.client.0.smithi123.stdout:2/771: mknod d0/d1f/d1c/d39/df2/cfc 0 2024-03-30T04:28:17.055 INFO:tasks.workunit.client.0.smithi123.stdout:3/728: dwrite d13/d24/d39/d79/d90/fd4 [0,4194304] 0 2024-03-30T04:28:17.062 INFO:tasks.workunit.client.0.smithi123.stdout:5/614: mkdir dd/d19/d1f/d68/dcc 0 2024-03-30T04:28:17.062 INFO:tasks.workunit.client.0.smithi123.stdout:7/693: rmdir d2/da/d11/d61/d55/d59/db5/dab/de3 0 2024-03-30T04:28:17.062 INFO:tasks.workunit.client.1.smithi135.stdout:5/305: dwrite d1/d7/d1a/f3b [0,4194304] 0 2024-03-30T04:28:17.064 INFO:tasks.workunit.client.0.smithi123.stdout:2/772: readlink d0/d1f/d1c/d39/d4b/d4c/d5f/l73 0 2024-03-30T04:28:17.065 INFO:tasks.workunit.client.0.smithi123.stdout:3/729: mkdir d13/d24/d39/d79/dea/dee 0 2024-03-30T04:28:17.070 INFO:tasks.workunit.client.1.smithi135.stdout:6/254: link d2/dd/d18/f4b d2/f4d 0 2024-03-30T04:28:17.071 INFO:tasks.workunit.client.0.smithi123.stdout:7/694: stat d2/da/d11/d61/d55/d59/db5/dab/dca/cdc 0 2024-03-30T04:28:17.071 INFO:tasks.workunit.client.0.smithi123.stdout:8/688: write da/dc/d14/d18/d21/da2/fc5 [3713471,103804] 0 2024-03-30T04:28:17.078 INFO:tasks.workunit.client.1.smithi135.stdout:7/325: dwrite d4/db/d1a/d4b/d40/d3a/f58 [0,4194304] 0 2024-03-30T04:28:17.078 INFO:tasks.workunit.client.1.smithi135.stdout:6/255: truncate d2/d36/f44 1880167 0 2024-03-30T04:28:17.079 INFO:tasks.workunit.client.0.smithi123.stdout:4/641: truncate d2/d4/d7/fe 5020218 0 2024-03-30T04:28:17.082 INFO:tasks.workunit.client.0.smithi123.stdout:7/695: readlink d2/d4d/dd0/le6 0 2024-03-30T04:28:17.088 INFO:tasks.workunit.client.0.smithi123.stdout:2/773: creat d0/d1f/d1c/d80/de6/ffd x:0 0 0 2024-03-30T04:28:17.089 INFO:tasks.workunit.client.0.smithi123.stdout:3/730: mknod d13/d24/d31/d23/d3a/d3c/cef 0 2024-03-30T04:28:17.089 INFO:tasks.workunit.client.0.smithi123.stdout:4/642: readlink d2/d4/l33 0 2024-03-30T04:28:17.090 INFO:tasks.workunit.client.0.smithi123.stdout:9/598: write d6/d1f/d72/f95 [301490,91364] 0 2024-03-30T04:28:17.091 INFO:tasks.workunit.client.1.smithi135.stdout:6/256: getdents d2/d2e 0 2024-03-30T04:28:17.091 INFO:tasks.workunit.client.0.smithi123.stdout:8/689: dread da/dc/d14/d18/f86 [0,4194304] 0 2024-03-30T04:28:17.092 INFO:tasks.workunit.client.1.smithi135.stdout:5/306: dwrite d1/f3e [0,4194304] 0 2024-03-30T04:28:17.094 INFO:tasks.workunit.client.1.smithi135.stdout:4/264: dread d10/d1a/d3d/f41 [0,4194304] 0 2024-03-30T04:28:17.100 INFO:tasks.workunit.client.0.smithi123.stdout:8/690: fsync da/d99/fb6 0 2024-03-30T04:28:17.100 INFO:tasks.workunit.client.0.smithi123.stdout:3/731: mknod d13/d24/cf0 0 2024-03-30T04:28:17.101 INFO:tasks.workunit.client.0.smithi123.stdout:9/599: creat d6/d1f/d92/fcc x:0 0 0 2024-03-30T04:28:17.102 INFO:tasks.workunit.client.0.smithi123.stdout:8/691: stat da/dc/d14/f71 0 2024-03-30T04:28:17.102 INFO:tasks.workunit.client.1.smithi135.stdout:6/257: rename d2/dd/d14/l29 to d2/d36/l4e 0 2024-03-30T04:28:17.104 INFO:tasks.workunit.client.1.smithi135.stdout:7/326: dwrite d4/f54 [0,4194304] 0 2024-03-30T04:28:17.104 INFO:tasks.workunit.client.0.smithi123.stdout:7/696: dwrite d2/da/d22/f86 [0,4194304] 0 2024-03-30T04:28:17.106 INFO:tasks.workunit.client.0.smithi123.stdout:9/600: truncate d6/d1f/d72/fc3 259108 0 2024-03-30T04:28:17.106 INFO:tasks.workunit.client.0.smithi123.stdout:3/732: mknod d13/d24/d31/d30/cf1 0 2024-03-30T04:28:17.108 INFO:tasks.workunit.client.0.smithi123.stdout:3/733: chown d13/d24/d31/d23/d63/f71 3161 1 2024-03-30T04:28:17.109 INFO:tasks.workunit.client.1.smithi135.stdout:7/327: write d4/db/d1a/d3d/f5b [754536,73545] 0 2024-03-30T04:28:17.110 INFO:tasks.workunit.client.1.smithi135.stdout:4/265: rename d10/c2c to d10/d1a/d32/d40/c5f 0 2024-03-30T04:28:17.112 INFO:tasks.workunit.client.0.smithi123.stdout:0/693: truncate d6/dd/d31/d35/f5b 626700 0 2024-03-30T04:28:17.114 INFO:tasks.workunit.client.1.smithi135.stdout:4/266: symlink d10/d1a/d32/l60 0 2024-03-30T04:28:17.116 INFO:tasks.workunit.client.0.smithi123.stdout:9/601: write d6/d1f/d45/fa0 [2126792,31089] 0 2024-03-30T04:28:17.116 INFO:tasks.workunit.client.1.smithi135.stdout:5/307: dwrite d1/d7/ff [4194304,4194304] 0 2024-03-30T04:28:17.116 INFO:tasks.workunit.client.1.smithi135.stdout:7/328: dread f3 [0,4194304] 0 2024-03-30T04:28:17.118 INFO:tasks.workunit.client.1.smithi135.stdout:6/258: link d2/ce d2/c4f 0 2024-03-30T04:28:17.119 INFO:tasks.workunit.client.1.smithi135.stdout:5/308: stat d1/l42 0 2024-03-30T04:28:17.122 INFO:tasks.workunit.client.1.smithi135.stdout:4/267: mknod d10/d1a/c61 0 2024-03-30T04:28:17.125 INFO:tasks.workunit.client.0.smithi123.stdout:9/602: mknod d6/d1f/da3/ccd 0 2024-03-30T04:28:17.125 INFO:tasks.workunit.client.0.smithi123.stdout:2/774: write d0/d1f/f8f [350657,71323] 0 2024-03-30T04:28:17.125 INFO:tasks.workunit.client.0.smithi123.stdout:3/734: link d13/d24/d39/d79/d90/lc9 d13/d24/d31/d30/lf2 0 2024-03-30T04:28:17.131 INFO:tasks.workunit.client.0.smithi123.stdout:4/643: dwrite d2/d34/f90 [0,4194304] 0 2024-03-30T04:28:17.131 INFO:tasks.workunit.client.0.smithi123.stdout:9/603: creat d6/dc9/fce x:0 0 0 2024-03-30T04:28:17.132 INFO:tasks.workunit.client.1.smithi135.stdout:6/259: unlink d2/l27 0 2024-03-30T04:28:17.134 INFO:tasks.workunit.client.1.smithi135.stdout:7/329: mkdir d4/db/d1a/d5c/d6b 0 2024-03-30T04:28:17.135 INFO:tasks.workunit.client.0.smithi123.stdout:2/775: symlink d0/d1f/d82/ddf/lfe 0 2024-03-30T04:28:17.138 INFO:tasks.workunit.client.0.smithi123.stdout:4/644: fsync d2/d4/d2c/d42/f4a 0 2024-03-30T04:28:17.139 INFO:tasks.workunit.client.1.smithi135.stdout:6/260: write d2/f11 [4659361,9591] 0 2024-03-30T04:28:17.139 INFO:tasks.workunit.client.1.smithi135.stdout:7/330: creat d4/db/de/d5a/f6c x:0 0 0 2024-03-30T04:28:17.141 INFO:tasks.workunit.client.0.smithi123.stdout:4/645: stat d2/d4/d2c/l59 0 2024-03-30T04:28:17.143 INFO:tasks.workunit.client.0.smithi123.stdout:4/646: write d2/d4/d2c/d42/fc3 [1357633,40603] 0 2024-03-30T04:28:17.145 INFO:tasks.workunit.client.1.smithi135.stdout:6/261: creat d2/dd/d14/d2b/f50 x:0 0 0 2024-03-30T04:28:17.148 INFO:tasks.workunit.client.0.smithi123.stdout:2/776: creat d0/d1f/d1c/d39/d4b/fff x:0 0 0 2024-03-30T04:28:17.148 INFO:tasks.workunit.client.0.smithi123.stdout:3/735: symlink d13/d24/d31/d23/d3a/d3c/d4e/dde/lf3 0 2024-03-30T04:28:17.149 INFO:tasks.workunit.client.1.smithi135.stdout:7/331: symlink d4/db/d1a/d5c/l6d 0 2024-03-30T04:28:17.151 INFO:tasks.workunit.client.0.smithi123.stdout:9/604: creat d6/d1f/d72/fcf x:0 0 0 2024-03-30T04:28:17.154 INFO:tasks.workunit.client.1.smithi135.stdout:4/268: dwrite d10/d1a/f21 [0,4194304] 0 2024-03-30T04:28:17.154 INFO:tasks.workunit.client.0.smithi123.stdout:3/736: mkdir d13/d24/d31/d23/d3a/d3c/db8/df4 0 2024-03-30T04:28:17.155 INFO:tasks.workunit.client.1.smithi135.stdout:5/309: dwrite d1/d3/d2b/f5b [0,4194304] 0 2024-03-30T04:28:17.155 INFO:tasks.workunit.client.1.smithi135.stdout:6/262: dread d2/dd/d14/d38/f3e [0,4194304] 0 2024-03-30T04:28:17.157 INFO:tasks.workunit.client.0.smithi123.stdout:3/737: dread - d13/d24/d31/d23/d3a/d3c/d4e/d5d/da5/fdb zero size 2024-03-30T04:28:17.157 INFO:tasks.workunit.client.0.smithi123.stdout:9/605: creat d6/d1f/d45/d97/d69/daa/fd0 x:0 0 0 2024-03-30T04:28:17.158 INFO:tasks.workunit.client.1.smithi135.stdout:7/332: rmdir d4/db/d1a/d5c/d6b 0 2024-03-30T04:28:17.158 INFO:tasks.workunit.client.1.smithi135.stdout:4/269: write d10/d1a/f52 [163270,10551] 0 2024-03-30T04:28:17.158 INFO:tasks.workunit.client.0.smithi123.stdout:2/777: truncate d0/d1f/d27/d68/dd6/d8c/f90 328773 0 2024-03-30T04:28:17.158 INFO:tasks.workunit.client.0.smithi123.stdout:3/738: fsync d13/f80 0 2024-03-30T04:28:17.161 INFO:tasks.workunit.client.0.smithi123.stdout:3/739: chown d13/ca6 427 1 2024-03-30T04:28:17.161 INFO:tasks.workunit.client.0.smithi123.stdout:2/778: chown d0/d1f/d1c/d80/dbb/d97 7 1 2024-03-30T04:28:17.162 INFO:tasks.workunit.client.1.smithi135.stdout:5/310: mknod d1/d7/d11/d3c/c5f 0 2024-03-30T04:28:17.163 INFO:tasks.workunit.client.1.smithi135.stdout:4/270: chown d10/d1a/l1b 461067223 1 2024-03-30T04:28:17.164 INFO:tasks.workunit.client.0.smithi123.stdout:9/606: rename d6/d1f/d4d/c7a to d6/d1f/d45/d4c/cd1 0 2024-03-30T04:28:17.166 INFO:tasks.workunit.client.0.smithi123.stdout:9/607: chown d6/d7c/cbb 74180556 1 2024-03-30T04:28:17.167 INFO:tasks.workunit.client.1.smithi135.stdout:5/311: chown d1/d7/f13 33641190 1 2024-03-30T04:28:17.168 INFO:tasks.workunit.client.0.smithi123.stdout:2/779: rename d0/d37/l83 to d0/d1f/d1c/d80/dbb/dd5/l100 0 2024-03-30T04:28:17.170 INFO:tasks.workunit.client.1.smithi135.stdout:4/271: creat d10/d1a/d42/f62 x:0 0 0 2024-03-30T04:28:17.170 INFO:tasks.workunit.client.0.smithi123.stdout:9/608: read d6/fd [2880690,3188] 0 2024-03-30T04:28:17.171 INFO:tasks.workunit.client.1.smithi135.stdout:7/333: truncate d4/db/d1a/d4b/f31 3289370 0 2024-03-30T04:28:17.172 INFO:tasks.workunit.client.0.smithi123.stdout:2/780: rmdir d0/d1f/d27 39 2024-03-30T04:28:17.174 INFO:tasks.workunit.client.0.smithi123.stdout:2/781: chown d0/d1f/d82/fe8 14618524 1 2024-03-30T04:28:17.175 INFO:tasks.workunit.client.1.smithi135.stdout:5/312: rmdir d1/da 39 2024-03-30T04:28:17.176 INFO:tasks.workunit.client.0.smithi123.stdout:2/782: write d0/d1f/d5e/f4f [939535,119902] 0 2024-03-30T04:28:17.177 INFO:tasks.workunit.client.0.smithi123.stdout:3/740: rename d13/d24/d31/d30/c83 to d13/d24/d31/d23/d3a/d3c/cf5 0 2024-03-30T04:28:17.179 INFO:tasks.workunit.client.1.smithi135.stdout:4/272: creat d10/d1a/d42/f63 x:0 0 0 2024-03-30T04:28:17.179 INFO:tasks.workunit.client.1.smithi135.stdout:6/263: dwrite d2/dd/f17 [0,4194304] 0 2024-03-30T04:28:17.180 INFO:tasks.workunit.client.0.smithi123.stdout:3/741: write d13/d24/d39/d79/d90/fc2 [1032025,72325] 0 2024-03-30T04:28:17.181 INFO:tasks.workunit.client.1.smithi135.stdout:7/334: dread d4/db/de/f2b [0,4194304] 0 2024-03-30T04:28:17.182 INFO:tasks.workunit.client.1.smithi135.stdout:9/249: sync 2024-03-30T04:28:17.184 INFO:tasks.workunit.client.0.smithi123.stdout:2/783: rename d0/d1f/d82/ddf/lfe to d0/d1f/d82/ddf/l101 0 2024-03-30T04:28:17.184 INFO:tasks.workunit.client.0.smithi123.stdout:9/609: link d6/d1f/da3/la8 d6/d1f/d45/d97/d69/ld2 0 2024-03-30T04:28:17.185 INFO:tasks.workunit.client.1.smithi135.stdout:4/273: link d10/c16 d10/d1a/d32/d40/c64 0 2024-03-30T04:28:17.186 INFO:tasks.workunit.client.0.smithi123.stdout:3/742: symlink d13/d24/d31/d23/d3a/d3c/db8/lf6 0 2024-03-30T04:28:17.188 INFO:tasks.workunit.client.0.smithi123.stdout:9/610: write d6/d49/f73 [875274,125263] 0 2024-03-30T04:28:17.191 INFO:tasks.workunit.client.0.smithi123.stdout:4/647: dwrite d2/d4/d7/d1a/f30 [0,4194304] 0 2024-03-30T04:28:17.191 INFO:tasks.workunit.client.1.smithi135.stdout:4/274: rmdir d10/d1a/d3d 39 2024-03-30T04:28:17.191 INFO:tasks.workunit.client.1.smithi135.stdout:9/250: mknod d7/d1b/c57 0 2024-03-30T04:28:17.195 INFO:tasks.workunit.client.0.smithi123.stdout:4/648: chown d2/d4/d7/d1a/f5e 1219228 1 2024-03-30T04:28:17.197 INFO:tasks.workunit.client.0.smithi123.stdout:2/784: mknod d0/d37/d57/da1/c102 0 2024-03-30T04:28:17.198 INFO:tasks.workunit.client.1.smithi135.stdout:9/251: chown d7/l1e 194 1 2024-03-30T04:28:17.199 INFO:tasks.workunit.client.1.smithi135.stdout:6/264: read d2/fc [552124,115824] 0 2024-03-30T04:28:17.199 INFO:tasks.workunit.client.1.smithi135.stdout:9/252: write d7/f2f [2612761,15185] 0 2024-03-30T04:28:17.199 INFO:tasks.workunit.client.0.smithi123.stdout:3/743: mkdir d13/d24/d31/df7 0 2024-03-30T04:28:17.199 INFO:tasks.workunit.client.1.smithi135.stdout:4/275: truncate d10/d1a/d42/f63 994813 0 2024-03-30T04:28:17.201 INFO:tasks.workunit.client.1.smithi135.stdout:9/253: truncate f6 2510136 0 2024-03-30T04:28:17.203 INFO:tasks.workunit.client.0.smithi123.stdout:9/611: creat d6/fd3 x:0 0 0 2024-03-30T04:28:17.203 INFO:tasks.workunit.client.0.smithi123.stdout:3/744: truncate d13/d24/d31/d23/feb 253349 0 2024-03-30T04:28:17.203 INFO:tasks.workunit.client.1.smithi135.stdout:4/276: chown d10/d1a/l25 1566 1 2024-03-30T04:28:17.203 INFO:tasks.workunit.client.1.smithi135.stdout:6/265: mknod d2/dd/c51 0 2024-03-30T04:28:17.204 INFO:tasks.workunit.client.0.smithi123.stdout:9/612: chown d6/d1f/d45/l57 1 1 2024-03-30T04:28:17.206 INFO:tasks.workunit.client.0.smithi123.stdout:4/649: read d2/d4/d7/d63/f82 [694378,89619] 0 2024-03-30T04:28:17.207 INFO:tasks.workunit.client.0.smithi123.stdout:9/613: unlink d6/fd 0 2024-03-30T04:28:17.209 INFO:tasks.workunit.client.1.smithi135.stdout:7/335: dwrite d4/f11 [0,4194304] 0 2024-03-30T04:28:17.209 INFO:tasks.workunit.client.0.smithi123.stdout:3/745: mknod d13/d24/d31/d23/d3a/d3c/d4e/d5d/da5/dcd/cf8 0 2024-03-30T04:28:17.209 INFO:tasks.workunit.client.0.smithi123.stdout:4/650: truncate d2/fc6 921523 0 2024-03-30T04:28:17.210 INFO:tasks.workunit.client.1.smithi135.stdout:9/254: symlink d7/d1b/l58 0 2024-03-30T04:28:17.214 INFO:tasks.workunit.client.1.smithi135.stdout:4/277: rename l7 to d10/l65 0 2024-03-30T04:28:17.214 INFO:tasks.workunit.client.0.smithi123.stdout:4/651: symlink d2/d4/d2c/d42/dad/d91/d9d/ldb 0 2024-03-30T04:28:17.216 INFO:tasks.workunit.client.0.smithi123.stdout:3/746: rename d13/c18 to d13/d24/d31/de0/cf9 0 2024-03-30T04:28:17.218 INFO:tasks.workunit.client.1.smithi135.stdout:7/336: mkdir d4/db/d1a/d4b/d40/d6e 0 2024-03-30T04:28:17.219 INFO:tasks.workunit.client.0.smithi123.stdout:9/614: getdents d6/d1f/d92/db8 0 2024-03-30T04:28:17.219 INFO:tasks.workunit.client.1.smithi135.stdout:9/255: unlink d7/d1b/d37/c4c 0 2024-03-30T04:28:17.220 INFO:tasks.workunit.client.0.smithi123.stdout:4/652: mknod d2/d4/d2c/d42/dad/d91/d9d/cdc 0 2024-03-30T04:28:17.221 INFO:tasks.workunit.client.1.smithi135.stdout:9/256: write d7/d29/f46 [323757,85931] 0 2024-03-30T04:28:17.221 INFO:tasks.workunit.client.0.smithi123.stdout:9/615: chown d6/l4b 5 1 2024-03-30T04:28:17.222 INFO:tasks.workunit.client.0.smithi123.stdout:4/653: dread - d2/dab/fbc zero size 2024-03-30T04:28:17.227 INFO:tasks.workunit.client.0.smithi123.stdout:1/740: sync 2024-03-30T04:28:17.228 INFO:tasks.workunit.client.1.smithi135.stdout:9/257: getdents d7 0 2024-03-30T04:28:17.228 INFO:tasks.workunit.client.1.smithi135.stdout:7/337: dread f3 [0,4194304] 0 2024-03-30T04:28:17.229 INFO:tasks.workunit.client.0.smithi123.stdout:9/616: truncate d6/d1f/d45/d97/d69/daa/fd0 317980 0 2024-03-30T04:28:17.229 INFO:tasks.workunit.client.0.smithi123.stdout:4/654: rename d2/c7b to d2/d4/d2c/d42/d80/d96/cdd 0 2024-03-30T04:28:17.234 INFO:tasks.workunit.client.1.smithi135.stdout:4/278: dwrite d10/d1a/f21 [0,4194304] 0 2024-03-30T04:28:17.234 INFO:tasks.workunit.client.0.smithi123.stdout:4/655: creat d2/d4/d7/dc1/fde x:0 0 0 2024-03-30T04:28:17.234 INFO:tasks.workunit.client.0.smithi123.stdout:2/785: dwrite d0/f1d [0,4194304] 0 2024-03-30T04:28:17.235 INFO:tasks.workunit.client.0.smithi123.stdout:1/741: mkdir db/dd/d42/d46/d56/df6 0 2024-03-30T04:28:17.236 INFO:tasks.workunit.client.1.smithi135.stdout:7/338: rename d4/db/d1a/d5c/l36 to d4/db/l6f 0 2024-03-30T04:28:17.245 INFO:tasks.workunit.client.0.smithi123.stdout:2/786: dread d0/d1f/d27/d68/dd6/d8c/f90 [0,4194304] 0 2024-03-30T04:28:17.245 INFO:tasks.workunit.client.0.smithi123.stdout:3/747: dread d13/d24/d31/f61 [0,4194304] 0 2024-03-30T04:28:17.246 INFO:tasks.workunit.client.1.smithi135.stdout:9/258: dwrite d7/f11 [0,4194304] 0 2024-03-30T04:28:17.249 INFO:tasks.workunit.client.1.smithi135.stdout:9/259: creat d7/d41/f59 x:0 0 0 2024-03-30T04:28:17.250 INFO:tasks.workunit.client.0.smithi123.stdout:2/787: dread - d0/d1f/d1c/d39/fd9 zero size 2024-03-30T04:28:17.252 INFO:tasks.workunit.client.1.smithi135.stdout:9/260: chown d7/d1b/f35 6509 1 2024-03-30T04:28:17.255 INFO:tasks.workunit.client.1.smithi135.stdout:9/261: creat d7/f5a x:0 0 0 2024-03-30T04:28:17.263 INFO:tasks.workunit.client.0.smithi123.stdout:2/788: mknod d0/d1f/d27/d68/dd6/d99/c103 0 2024-03-30T04:28:17.264 INFO:tasks.workunit.client.1.smithi135.stdout:6/266: write d2/dd/d18/f4b [202784,42031] 0 2024-03-30T04:28:17.266 INFO:tasks.workunit.client.0.smithi123.stdout:1/742: dwrite db/dd/d5f/d6d/d85/dd4/fe6 [0,4194304] 0 2024-03-30T04:28:17.266 INFO:tasks.workunit.client.0.smithi123.stdout:4/656: dread d2/f1c [0,4194304] 0 2024-03-30T04:28:17.278 INFO:tasks.workunit.client.0.smithi123.stdout:3/748: dwrite d13/fe1 [0,4194304] 0 2024-03-30T04:28:17.279 INFO:tasks.workunit.client.1.smithi135.stdout:4/279: write d10/f17 [298868,118345] 0 2024-03-30T04:28:17.282 INFO:tasks.workunit.client.1.smithi135.stdout:4/280: mkdir d10/d1a/d42/d66 0 2024-03-30T04:28:17.285 INFO:tasks.workunit.client.1.smithi135.stdout:4/281: creat d10/d1a/d42/f67 x:0 0 0 2024-03-30T04:28:17.286 INFO:tasks.workunit.client.0.smithi123.stdout:3/749: dread d13/f5e [0,4194304] 0 2024-03-30T04:28:17.294 INFO:tasks.workunit.client.1.smithi135.stdout:4/282: getdents d10/d1a/d42/d46 0 2024-03-30T04:28:17.296 INFO:tasks.workunit.client.1.smithi135.stdout:4/283: write d10/d1a/f52 [859693,116396] 0 2024-03-30T04:28:17.303 INFO:tasks.workunit.client.1.smithi135.stdout:4/284: chown d10/d1a/d3d/f41 40563880 1 2024-03-30T04:28:17.304 INFO:tasks.workunit.client.1.smithi135.stdout:9/262: dread d7/f11 [0,4194304] 0 2024-03-30T04:28:17.304 INFO:tasks.workunit.client.0.smithi123.stdout:2/789: creat d0/d1f/d27/df7/f104 x:0 0 0 2024-03-30T04:28:17.308 INFO:tasks.workunit.client.1.smithi135.stdout:4/285: creat d10/d1a/f68 x:0 0 0 2024-03-30T04:28:17.309 INFO:tasks.workunit.client.0.smithi123.stdout:1/743: dread db/dd/d42/d46/d56/fb6 [0,4194304] 0 2024-03-30T04:28:17.309 INFO:tasks.workunit.client.1.smithi135.stdout:0/354: sync 2024-03-30T04:28:17.310 INFO:tasks.workunit.client.1.smithi135.stdout:4/286: write d10/d1a/d42/f62 [725501,111754] 0 2024-03-30T04:28:17.311 INFO:tasks.workunit.client.0.smithi123.stdout:4/657: rename d2/d4/d2c/d42/f3a to d2/d4/d2c/d42/d35/d5d/fdf 0 2024-03-30T04:28:17.311 INFO:tasks.workunit.client.0.smithi123.stdout:2/790: mknod d0/d37/dec/c105 0 2024-03-30T04:28:17.312 INFO:tasks.workunit.client.0.smithi123.stdout:3/750: link d13/d24/d31/d23/l26 d13/d24/d39/lfa 0 2024-03-30T04:28:17.312 INFO:tasks.workunit.client.1.smithi135.stdout:4/287: write d10/d1a/f3a [48821,122267] 0 2024-03-30T04:28:17.313 INFO:tasks.workunit.client.1.smithi135.stdout:9/263: write d7/f22 [2120509,71403] 0 2024-03-30T04:28:17.316 INFO:tasks.workunit.client.0.smithi123.stdout:2/791: dread - d0/d1f/d27/d68/dd6/d8c/fe7 zero size 2024-03-30T04:28:17.316 INFO:tasks.workunit.client.1.smithi135.stdout:0/355: rename d0/d19/d2d/d45/l6d to d0/d1/d16/l70 0 2024-03-30T04:28:17.317 INFO:tasks.workunit.client.0.smithi123.stdout:1/744: rename db/dd/d42/d46/d39/f93 to db/dd/d5f/d6d/d7e/ff7 0 2024-03-30T04:28:17.317 INFO:tasks.workunit.client.0.smithi123.stdout:2/792: readlink d0/d1f/d16/lf5 0 2024-03-30T04:28:17.318 INFO:tasks.workunit.client.1.smithi135.stdout:6/267: dread d2/f39 [0,4194304] 0 2024-03-30T04:28:17.321 INFO:tasks.workunit.client.0.smithi123.stdout:2/793: mkdir d0/d1f/d1c/d36/d106 0 2024-03-30T04:28:17.321 INFO:tasks.workunit.client.1.smithi135.stdout:9/264: creat d7/d55/f5b x:0 0 0 2024-03-30T04:28:17.324 INFO:tasks.workunit.client.0.smithi123.stdout:1/745: rename db/d1b/c63 to db/dd/d42/d46/d56/d5c/dd0/cf8 0 2024-03-30T04:28:17.325 INFO:tasks.workunit.client.0.smithi123.stdout:2/794: write d0/d1f/d27/d68/dd6/fe0 [1882137,2896] 0 2024-03-30T04:28:17.327 INFO:tasks.workunit.client.1.smithi135.stdout:9/265: mkdir d7/d4f/d5c 0 2024-03-30T04:28:17.329 INFO:tasks.workunit.client.0.smithi123.stdout:2/795: creat d0/d1f/d1c/d80/de6/f107 x:0 0 0 2024-03-30T04:28:17.332 INFO:tasks.workunit.client.0.smithi123.stdout:2/796: mkdir d0/d1f/d1c/d80/da8/d108 0 2024-03-30T04:28:17.335 INFO:tasks.workunit.client.0.smithi123.stdout:2/797: mkdir d0/d1f/d27/d68/dd6/d109 0 2024-03-30T04:28:17.335 INFO:tasks.workunit.client.1.smithi135.stdout:6/268: dread d2/dd/d18/f24 [0,4194304] 0 2024-03-30T04:28:17.336 INFO:tasks.workunit.client.1.smithi135.stdout:0/356: dwrite d0/d13/d1e/f49 [0,4194304] 0 2024-03-30T04:28:17.338 INFO:tasks.workunit.client.0.smithi123.stdout:2/798: mkdir d0/d1f/d1c/d39/d4b/d10a 0 2024-03-30T04:28:17.340 INFO:tasks.workunit.client.0.smithi123.stdout:2/799: creat d0/d1f/d1c/d39/de4/f10b x:0 0 0 2024-03-30T04:28:17.342 INFO:tasks.workunit.client.1.smithi135.stdout:9/266: write d7/d1b/f1c [368426,54144] 0 2024-03-30T04:28:17.342 INFO:tasks.workunit.client.1.smithi135.stdout:4/288: dread fd [0,4194304] 0 2024-03-30T04:28:17.343 INFO:tasks.workunit.client.0.smithi123.stdout:2/800: stat d0/d1f/d27/d68/dd6/c2c 0 2024-03-30T04:28:17.344 INFO:tasks.workunit.client.0.smithi123.stdout:4/658: getdents d2/d4/d2c/d42 0 2024-03-30T04:28:17.346 INFO:tasks.workunit.client.1.smithi135.stdout:9/267: rename d7/d1b/f3b to d7/d55/f5d 0 2024-03-30T04:28:17.347 INFO:tasks.workunit.client.1.smithi135.stdout:0/357: dread d0/d5/d9/d23/d58/f1f [0,4194304] 0 2024-03-30T04:28:17.347 INFO:tasks.workunit.client.0.smithi123.stdout:3/751: write d13/d24/d31/d23/d3a/f40 [1276432,46782] 0 2024-03-30T04:28:17.348 INFO:tasks.workunit.client.1.smithi135.stdout:6/269: truncate d2/dd/f1a 3427216 0 2024-03-30T04:28:17.349 INFO:tasks.workunit.client.0.smithi123.stdout:1/746: write db/f59 [73916,72664] 0 2024-03-30T04:28:17.349 INFO:tasks.workunit.client.1.smithi135.stdout:4/289: symlink d10/d1a/l69 0 2024-03-30T04:28:17.351 INFO:tasks.workunit.client.1.smithi135.stdout:4/290: dread - d10/d1a/d42/f54 zero size 2024-03-30T04:28:17.351 INFO:tasks.workunit.client.0.smithi123.stdout:3/752: symlink d13/d24/d31/d23/d3a/d3c/d4e/d5d/da5/lfb 0 2024-03-30T04:28:17.352 INFO:tasks.workunit.client.0.smithi123.stdout:3/753: chown d13/d24/d31 48 1 2024-03-30T04:28:17.353 INFO:tasks.workunit.client.0.smithi123.stdout:1/747: unlink db/dd/d42/d46/d5d/d62/cbd 0 2024-03-30T04:28:17.355 INFO:tasks.workunit.client.1.smithi135.stdout:0/358: symlink d0/d5/d9/l71 0 2024-03-30T04:28:17.356 INFO:tasks.workunit.client.1.smithi135.stdout:0/359: fsync d0/d1/f5e 0 2024-03-30T04:28:17.356 INFO:tasks.workunit.client.1.smithi135.stdout:4/291: mknod d10/d1a/d3d/c6a 0 2024-03-30T04:28:17.357 INFO:tasks.workunit.client.1.smithi135.stdout:6/270: mkdir d2/dd/d40/d52 0 2024-03-30T04:28:17.358 INFO:tasks.workunit.client.0.smithi123.stdout:1/748: creat db/dd/d42/d46/d51/d72/ff9 x:0 0 0 2024-03-30T04:28:17.359 INFO:tasks.workunit.client.0.smithi123.stdout:3/754: rename d13/c48 to d13/d24/d31/d23/d3a/d3c/dab/db2/cfc 0 2024-03-30T04:28:17.360 INFO:tasks.workunit.client.1.smithi135.stdout:4/292: truncate d10/d1a/f68 705173 0 2024-03-30T04:28:17.363 INFO:tasks.workunit.client.0.smithi123.stdout:2/801: dwrite d0/d1f/d5e/fb9 [0,4194304] 0 2024-03-30T04:28:17.366 INFO:tasks.workunit.client.0.smithi123.stdout:3/755: write d13/d24/d31/d23/d3a/d3c/fa8 [70351,101910] 0 2024-03-30T04:28:17.368 INFO:tasks.workunit.client.1.smithi135.stdout:9/268: dwrite d7/f2f [0,4194304] 0 2024-03-30T04:28:17.368 INFO:tasks.workunit.client.1.smithi135.stdout:6/271: truncate d2/f4 1269278 0 2024-03-30T04:28:17.369 INFO:tasks.workunit.client.0.smithi123.stdout:2/802: chown d0/d1f/d1c/d80/dbb/dd5/fca 1376059 1 2024-03-30T04:28:17.370 INFO:tasks.workunit.client.0.smithi123.stdout:3/756: fdatasync d13/f5e 0 2024-03-30T04:28:17.372 INFO:tasks.workunit.client.0.smithi123.stdout:2/803: write d0/d1f/d78/feb [1938739,13907] 0 2024-03-30T04:28:17.373 INFO:tasks.workunit.client.0.smithi123.stdout:3/757: chown d13/f5e 138 1 2024-03-30T04:28:17.375 INFO:tasks.workunit.client.0.smithi123.stdout:1/749: link db/dd/d5f/d6d/fa3 db/dd/d42/d46/d5d/ffa 0 2024-03-30T04:28:17.377 INFO:tasks.workunit.client.0.smithi123.stdout:2/804: dread - d0/d1f/d1c/d80/dbb/d97/db5/fd2 zero size 2024-03-30T04:28:17.378 INFO:tasks.workunit.client.1.smithi135.stdout:9/269: creat d7/d4f/d5c/f5e x:0 0 0 2024-03-30T04:28:17.380 INFO:tasks.workunit.client.1.smithi135.stdout:0/360: write d0/d5/d10/f31 [607206,63983] 0 2024-03-30T04:28:17.381 INFO:tasks.workunit.client.1.smithi135.stdout:6/272: mkdir d2/dd/d14/d53 0 2024-03-30T04:28:17.382 INFO:tasks.workunit.client.0.smithi123.stdout:2/805: mknod d0/d1f/d27/d68/dd6/d8c/c10c 0 2024-03-30T04:28:17.384 INFO:tasks.workunit.client.0.smithi123.stdout:2/806: write d0/d1f/d1c/f41 [2301529,88879] 0 2024-03-30T04:28:17.385 INFO:tasks.workunit.client.1.smithi135.stdout:6/273: write d2/dd/d14/f3a [598647,51064] 0 2024-03-30T04:28:17.389 INFO:tasks.workunit.client.1.smithi135.stdout:0/361: creat d0/d13/d1e/d3a/f72 x:0 0 0 2024-03-30T04:28:17.391 INFO:tasks.workunit.client.1.smithi135.stdout:6/274: fdatasync d2/d2e/f31 0 2024-03-30T04:28:17.393 INFO:tasks.workunit.client.0.smithi123.stdout:6/750: sync 2024-03-30T04:28:17.393 INFO:tasks.workunit.client.0.smithi123.stdout:5/615: sync 2024-03-30T04:28:17.393 INFO:tasks.workunit.client.0.smithi123.stdout:2/807: read d0/d1f/d27/d68/dd6/f7c [651675,67] 0 2024-03-30T04:28:17.394 INFO:tasks.workunit.client.1.smithi135.stdout:4/293: dwrite fb [0,4194304] 0 2024-03-30T04:28:17.394 INFO:tasks.workunit.client.1.smithi135.stdout:6/275: chown d2/dd/c20 0 1 2024-03-30T04:28:17.394 INFO:tasks.workunit.client.1.smithi135.stdout:0/362: symlink d0/d5/d9/d23/d5d/l73 0 2024-03-30T04:28:17.399 INFO:tasks.workunit.client.1.smithi135.stdout:4/294: fsync d10/d1a/f3a 0 2024-03-30T04:28:17.400 INFO:tasks.workunit.client.1.smithi135.stdout:6/276: truncate d2/f8 1466844 0 2024-03-30T04:28:17.401 INFO:tasks.workunit.client.0.smithi123.stdout:1/750: dwrite db/dd/d5f/feb [0,4194304] 0 2024-03-30T04:28:17.402 INFO:tasks.workunit.client.1.smithi135.stdout:4/295: chown d10/c24 856886 1 2024-03-30T04:28:17.402 INFO:tasks.workunit.client.0.smithi123.stdout:2/808: write d0/d1f/d1c/d80/dbb/d97/fd8 [295624,31024] 0 2024-03-30T04:28:17.404 INFO:tasks.workunit.client.0.smithi123.stdout:6/751: write d10/d82/dc1/d6a/d7a/d99/dba/fde [201130,81566] 0 2024-03-30T04:28:17.407 INFO:tasks.workunit.client.1.smithi135.stdout:4/296: chown c8 2045832 1 2024-03-30T04:28:17.408 INFO:tasks.workunit.client.1.smithi135.stdout:6/277: chown d2/dd/d18/f4b 2 1 2024-03-30T04:28:17.410 INFO:tasks.workunit.client.1.smithi135.stdout:9/270: dwrite d7/d1b/d37/f4a [0,4194304] 0 2024-03-30T04:28:17.411 INFO:tasks.workunit.client.0.smithi123.stdout:5/616: truncate dd/d19/d1f/d35/d3a/d79/f96 3858444 0 2024-03-30T04:28:17.412 INFO:tasks.workunit.client.0.smithi123.stdout:1/751: creat db/dd/d42/d46/d51/dd7/ffb x:0 0 0 2024-03-30T04:28:17.413 INFO:tasks.workunit.client.0.smithi123.stdout:2/809: truncate d0/d1f/d16/f18 3402129 0 2024-03-30T04:28:17.416 INFO:tasks.workunit.client.0.smithi123.stdout:5/617: read - dd/d19/d5f/fca zero size 2024-03-30T04:28:17.417 INFO:tasks.workunit.client.1.smithi135.stdout:4/297: creat d10/d1a/d42/d46/f6b x:0 0 0 2024-03-30T04:28:17.417 INFO:tasks.workunit.client.1.smithi135.stdout:0/363: link d0/d5/d9/d23/d58/d55/f6b d0/d13/f74 0 2024-03-30T04:28:17.418 INFO:tasks.workunit.client.0.smithi123.stdout:2/810: unlink d0/d1f/d1c/d36/ca4 0 2024-03-30T04:28:17.419 INFO:tasks.workunit.client.0.smithi123.stdout:5/618: chown dd/d19/d1f/d3c/l95 9 1 2024-03-30T04:28:17.419 INFO:tasks.workunit.client.0.smithi123.stdout:6/752: rename d10/d20/da2/fb1 to d10/d82/dc1/d6a/d7a/f100 0 2024-03-30T04:28:17.420 INFO:tasks.workunit.client.0.smithi123.stdout:2/811: dread - d0/ff1 zero size 2024-03-30T04:28:17.420 INFO:tasks.workunit.client.1.smithi135.stdout:9/271: dread d7/d1b/f40 [0,4194304] 0 2024-03-30T04:28:17.425 INFO:tasks.workunit.client.0.smithi123.stdout:2/812: mkdir d0/d1f/d1c/d10d 0 2024-03-30T04:28:17.426 INFO:tasks.workunit.client.1.smithi135.stdout:6/278: symlink d2/dd/d40/d52/l54 0 2024-03-30T04:28:17.427 INFO:tasks.workunit.client.1.smithi135.stdout:0/364: truncate d0/d5/d9/fd 1958866 0 2024-03-30T04:28:17.428 INFO:tasks.workunit.client.0.smithi123.stdout:2/813: write d0/d1f/d5e/f6f [1912703,9978] 0 2024-03-30T04:28:17.430 INFO:tasks.workunit.client.0.smithi123.stdout:2/814: stat d0/d1f/d16/lcd 0 2024-03-30T04:28:17.435 INFO:tasks.workunit.client.0.smithi123.stdout:6/753: dread d10/d82/d8e/dc7/d5b/fa8 [0,4194304] 0 2024-03-30T04:28:17.436 INFO:tasks.workunit.client.0.smithi123.stdout:3/758: dread d13/d24/f70 [0,4194304] 0 2024-03-30T04:28:17.437 INFO:tasks.workunit.client.1.smithi135.stdout:0/365: symlink d0/d13/l75 0 2024-03-30T04:28:17.442 INFO:tasks.workunit.client.0.smithi123.stdout:6/754: chown d10/d82/dc1/d6a/d7a/c8c 248358750 1 2024-03-30T04:28:17.442 INFO:tasks.workunit.client.0.smithi123.stdout:1/752: write db/dd/d42/d46/d56/d5c/f76 [1423664,111231] 0 2024-03-30T04:28:17.444 INFO:tasks.workunit.client.0.smithi123.stdout:3/759: stat d13/d24/d31/d23/ccb 0 2024-03-30T04:28:17.445 INFO:tasks.workunit.client.0.smithi123.stdout:2/815: link d0/d1f/d82/ce5 d0/d1f/d30/c10e 0 2024-03-30T04:28:17.445 INFO:tasks.workunit.client.1.smithi135.stdout:6/279: truncate d2/f39 3424270 0 2024-03-30T04:28:17.446 INFO:tasks.workunit.client.1.smithi135.stdout:4/298: dwrite d10/f17 [0,4194304] 0 2024-03-30T04:28:17.446 INFO:tasks.workunit.client.0.smithi123.stdout:5/619: dwrite dd/d41/f7f [0,4194304] 0 2024-03-30T04:28:17.448 INFO:tasks.workunit.client.1.smithi135.stdout:9/272: dwrite d7/d1b/f28 [0,4194304] 0 2024-03-30T04:28:17.449 INFO:tasks.workunit.client.0.smithi123.stdout:2/816: write d0/d1f/d16/fcb [355210,93329] 0 2024-03-30T04:28:17.449 INFO:tasks.workunit.client.1.smithi135.stdout:4/299: write d10/d1a/d2b/f31 [4990070,59661] 0 2024-03-30T04:28:17.452 INFO:tasks.workunit.client.0.smithi123.stdout:1/753: truncate db/dd/d42/d46/d56/da2/fb0 4891082 0 2024-03-30T04:28:17.453 INFO:tasks.workunit.client.0.smithi123.stdout:1/754: chown db 1007793495 1 2024-03-30T04:28:17.457 INFO:tasks.workunit.client.0.smithi123.stdout:6/755: rename d10/d82/d8e/df5 to d10/d82/dc1/d6a/d7a/de8/d101 0 2024-03-30T04:28:17.457 INFO:tasks.workunit.client.0.smithi123.stdout:1/755: chown db/dd/d42/d46/d56/df6 0 1 2024-03-30T04:28:17.458 INFO:tasks.workunit.client.1.smithi135.stdout:9/273: creat d7/d41/f5f x:0 0 0 2024-03-30T04:28:17.460 INFO:tasks.workunit.client.1.smithi135.stdout:2/255: sync 2024-03-30T04:28:17.464 INFO:tasks.workunit.client.1.smithi135.stdout:9/274: readlink d7/d1b/l9 0 2024-03-30T04:28:17.467 INFO:tasks.workunit.client.0.smithi123.stdout:1/756: mkdir db/dd/d5f/d6d/d85/d89/d90/d94/d9c/dfc 0 2024-03-30T04:28:17.470 INFO:tasks.workunit.client.1.smithi135.stdout:2/256: getdents d4/d32/d43 0 2024-03-30T04:28:17.472 INFO:tasks.workunit.client.0.smithi123.stdout:1/757: chown db/dd/d42/d46/d56/daf/de4 6311895 1 2024-03-30T04:28:17.475 INFO:tasks.workunit.client.0.smithi123.stdout:5/620: dwrite dd/d19/d1f/d35/faf [0,4194304] 0 2024-03-30T04:28:17.475 INFO:tasks.workunit.client.1.smithi135.stdout:2/257: rename d4/d6/c34 to d4/d6/d47/c52 0 2024-03-30T04:28:17.476 INFO:tasks.workunit.client.1.smithi135.stdout:6/280: dwrite d2/d2e/f37 [0,4194304] 0 2024-03-30T04:28:17.476 INFO:tasks.workunit.client.1.smithi135.stdout:2/258: stat d4/d32 0 2024-03-30T04:28:17.478 INFO:tasks.workunit.client.0.smithi123.stdout:2/817: dread d0/d1f/d5e/f4f [0,4194304] 0 2024-03-30T04:28:17.480 INFO:tasks.workunit.client.1.smithi135.stdout:6/281: mkdir d2/dd/d18/d21/d55 0 2024-03-30T04:28:17.481 INFO:tasks.workunit.client.0.smithi123.stdout:5/621: rmdir dd/d19/d1f 39 2024-03-30T04:28:17.482 INFO:tasks.workunit.client.1.smithi135.stdout:6/282: truncate d2/d2e/f4c 886512 0 2024-03-30T04:28:17.486 INFO:tasks.workunit.client.0.smithi123.stdout:2/818: rename d0/d1f/d1c/d39/d4b/l6c to d0/d1f/d1c/d39/d4b/d4c/d5f/l10f 0 2024-03-30T04:28:17.488 INFO:tasks.workunit.client.0.smithi123.stdout:2/819: fdatasync d0/d1f/d1c/d39/fd1 0 2024-03-30T04:28:17.488 INFO:tasks.workunit.client.1.smithi135.stdout:6/283: mknod d2/dd/d14/d2b/c56 0 2024-03-30T04:28:17.491 INFO:tasks.workunit.client.0.smithi123.stdout:3/760: truncate d13/f1b 3408974 0 2024-03-30T04:28:17.493 INFO:tasks.workunit.client.0.smithi123.stdout:1/758: write db/f45 [1127263,77045] 0 2024-03-30T04:28:17.494 INFO:tasks.workunit.client.1.smithi135.stdout:6/284: symlink d2/d36/l57 0 2024-03-30T04:28:17.498 INFO:tasks.workunit.client.0.smithi123.stdout:2/820: getdents d0/d37/d57/da1 0 2024-03-30T04:28:17.499 INFO:tasks.workunit.client.0.smithi123.stdout:1/759: creat db/dd/d5f/dae/ffd x:0 0 0 2024-03-30T04:28:17.500 INFO:tasks.workunit.client.1.smithi135.stdout:2/259: link d4/d32/d2e/f4a d4/d32/d2e/f53 0 2024-03-30T04:28:17.500 INFO:tasks.workunit.client.0.smithi123.stdout:2/821: readlink d0/d1f/d1c/d36/laf 0 2024-03-30T04:28:17.504 INFO:tasks.workunit.client.1.smithi135.stdout:9/275: rmdir d7/d41 39 2024-03-30T04:28:17.504 INFO:tasks.workunit.client.0.smithi123.stdout:1/760: write db/dd/d5f/d6d/d85/dd4/fe6 [19234,37224] 0 2024-03-30T04:28:17.504 INFO:tasks.workunit.client.0.smithi123.stdout:2/822: fdatasync d0/d37/f8a 0 2024-03-30T04:28:17.505 INFO:tasks.workunit.client.1.smithi135.stdout:0/366: dread d0/d5/d10/f11 [4194304,4194304] 0 2024-03-30T04:28:17.507 INFO:tasks.workunit.client.0.smithi123.stdout:3/761: dread d13/d24/d39/d79/d90/fc2 [0,4194304] 0 2024-03-30T04:28:17.508 INFO:tasks.workunit.client.1.smithi135.stdout:0/367: readlink d0/d5/d9/d23/d2a/l57 0 2024-03-30T04:28:17.509 INFO:tasks.workunit.client.1.smithi135.stdout:4/300: dwrite fd [0,4194304] 0 2024-03-30T04:28:17.511 INFO:tasks.workunit.client.0.smithi123.stdout:6/756: dread d10/f3a [0,4194304] 0 2024-03-30T04:28:17.511 INFO:tasks.workunit.client.1.smithi135.stdout:4/301: read d10/f17 [1358331,106463] 0 2024-03-30T04:28:17.517 INFO:tasks.workunit.client.0.smithi123.stdout:3/762: creat d13/d24/d31/d23/d63/ffd x:0 0 0 2024-03-30T04:28:17.517 INFO:tasks.workunit.client.1.smithi135.stdout:0/368: unlink d0/d5/d9/l1c 0 2024-03-30T04:28:17.518 INFO:tasks.workunit.client.0.smithi123.stdout:1/761: link db/dd/d42/d46/d56/da2/fb0 db/dd/ffe 0 2024-03-30T04:28:17.520 INFO:tasks.workunit.client.1.smithi135.stdout:9/276: symlink d7/d1b/d37/l60 0 2024-03-30T04:28:17.521 INFO:tasks.workunit.client.0.smithi123.stdout:3/763: symlink d13/d24/d31/d23/d3a/d3c/d4e/d5d/da5/lfe 0 2024-03-30T04:28:17.523 INFO:tasks.workunit.client.1.smithi135.stdout:6/285: dwrite d2/f13 [0,4194304] 0 2024-03-30T04:28:17.524 INFO:tasks.workunit.client.0.smithi123.stdout:1/762: write db/dd/d42/d46/d56/da2/fb0 [527077,24165] 0 2024-03-30T04:28:17.525 INFO:tasks.workunit.client.0.smithi123.stdout:3/764: read d13/d24/d39/d79/d90/fd4 [972454,78161] 0 2024-03-30T04:28:17.531 INFO:tasks.workunit.client.1.smithi135.stdout:6/286: fdatasync d2/dd/f1a 0 2024-03-30T04:28:17.532 INFO:tasks.workunit.client.0.smithi123.stdout:3/765: mkdir d13/d24/dff 0 2024-03-30T04:28:17.534 INFO:tasks.workunit.client.0.smithi123.stdout:8/692: sync 2024-03-30T04:28:17.534 INFO:tasks.workunit.client.0.smithi123.stdout:3/766: write d13/d24/d31/d30/fdd [741173,80464] 0 2024-03-30T04:28:17.538 INFO:tasks.workunit.client.0.smithi123.stdout:8/693: readlink da/dc/d14/d9d/laa 0 2024-03-30T04:28:17.539 INFO:tasks.workunit.client.0.smithi123.stdout:3/767: write d13/d24/fd6 [127552,102077] 0 2024-03-30T04:28:17.541 INFO:tasks.workunit.client.0.smithi123.stdout:2/823: write d0/d1f/d27/d68/dd6/d8c/f90 [1217343,35011] 0 2024-03-30T04:28:17.544 INFO:tasks.workunit.client.1.smithi135.stdout:4/302: dwrite d10/d1a/d3d/f5a [0,4194304] 0 2024-03-30T04:28:17.550 INFO:tasks.workunit.client.0.smithi123.stdout:5/622: dread dd/d19/f87 [4194304,4194304] 0 2024-03-30T04:28:17.550 INFO:tasks.workunit.client.0.smithi123.stdout:8/694: link da/dc/d14/d18/d21/ca3 da/dc/d14/d9c/cd0 0 2024-03-30T04:28:17.551 INFO:tasks.workunit.client.1.smithi135.stdout:9/277: truncate d7/d29/f3c 3373575 0 2024-03-30T04:28:17.552 INFO:tasks.workunit.client.0.smithi123.stdout:2/824: truncate d0/d1f/d1c/d40/f98 718819 0 2024-03-30T04:28:17.554 INFO:tasks.workunit.client.0.smithi123.stdout:5/623: truncate fb 919395 0 2024-03-30T04:28:17.555 INFO:tasks.workunit.client.1.smithi135.stdout:6/287: dwrite d2/d2e/f4c [0,4194304] 0 2024-03-30T04:28:17.557 INFO:tasks.workunit.client.0.smithi123.stdout:8/695: chown da/d27/d3c/d2d/l5a 668 1 2024-03-30T04:28:17.559 INFO:tasks.workunit.client.0.smithi123.stdout:2/825: rmdir d0/d1f/d1c/d39/d4b/d4c 39 2024-03-30T04:28:17.560 INFO:tasks.workunit.client.0.smithi123.stdout:1/763: dwrite db/dd/d42/d46/d51/fb1 [0,4194304] 0 2024-03-30T04:28:17.564 INFO:tasks.workunit.client.1.smithi135.stdout:9/278: creat d7/d41/f61 x:0 0 0 2024-03-30T04:28:17.566 INFO:tasks.workunit.client.1.smithi135.stdout:0/369: dwrite d0/d5/d9/d23/d58/d55/f6b [0,4194304] 0 2024-03-30T04:28:17.567 INFO:tasks.workunit.client.0.smithi123.stdout:5/624: rename dd/d19/d1f/d35/f38 to dd/d19/d1f/d3c/fcd 0 2024-03-30T04:28:17.567 INFO:tasks.workunit.client.1.smithi135.stdout:6/288: symlink d2/dd/d14/d30/d43/l58 0 2024-03-30T04:28:17.570 INFO:tasks.workunit.client.1.smithi135.stdout:9/279: stat d7/d4f/d5c 0 2024-03-30T04:28:17.572 INFO:tasks.workunit.client.1.smithi135.stdout:0/370: stat d0/d5/d9/d23/d58/l20 0 2024-03-30T04:28:17.575 INFO:tasks.workunit.client.0.smithi123.stdout:1/764: write db/dd/d42/d46/d5d/d62/f6b [8708601,13463] 0 2024-03-30T04:28:17.576 INFO:tasks.workunit.client.1.smithi135.stdout:9/280: creat d7/d29/f62 x:0 0 0 2024-03-30T04:28:17.576 INFO:tasks.workunit.client.0.smithi123.stdout:7/697: sync 2024-03-30T04:28:17.576 INFO:tasks.workunit.client.0.smithi123.stdout:9/617: sync 2024-03-30T04:28:17.577 INFO:tasks.workunit.client.0.smithi123.stdout:0/694: sync 2024-03-30T04:28:17.577 INFO:tasks.workunit.client.0.smithi123.stdout:4/659: sync 2024-03-30T04:28:17.578 INFO:tasks.workunit.client.1.smithi135.stdout:9/281: write d7/d1b/f16 [204851,69605] 0 2024-03-30T04:28:17.584 INFO:tasks.workunit.client.1.smithi135.stdout:6/289: symlink d2/dd/d18/d21/d55/l59 0 2024-03-30T04:28:17.584 INFO:tasks.workunit.client.1.smithi135.stdout:0/371: creat d0/d19/d2d/d45/f76 x:0 0 0 2024-03-30T04:28:17.584 INFO:tasks.workunit.client.0.smithi123.stdout:6/757: dread d10/d20/d3f/f91 [0,4194304] 0 2024-03-30T04:28:17.587 INFO:tasks.workunit.client.0.smithi123.stdout:8/696: link da/d27/d3c/d64/d78/d83/c9b da/dc/d14/d18/d21/da2/cd1 0 2024-03-30T04:28:17.589 INFO:tasks.workunit.client.0.smithi123.stdout:9/618: dread - d6/d7c/fb4 zero size 2024-03-30T04:28:17.590 INFO:tasks.workunit.client.0.smithi123.stdout:7/698: readlink d2/dc/d1c/l24 0 2024-03-30T04:28:17.590 INFO:tasks.workunit.client.0.smithi123.stdout:3/768: dwrite f11 [0,4194304] 0 2024-03-30T04:28:17.592 INFO:tasks.workunit.client.1.smithi135.stdout:0/372: rmdir d0/d5 39 2024-03-30T04:28:17.593 INFO:tasks.workunit.client.0.smithi123.stdout:5/625: read dd/d19/d1f/d3c/f56 [1392212,68983] 0 2024-03-30T04:28:17.593 INFO:tasks.workunit.client.1.smithi135.stdout:4/303: fdatasync d10/d1a/d3d/f5a 0 2024-03-30T04:28:17.596 INFO:tasks.workunit.client.1.smithi135.stdout:0/373: chown d0/d5 250585277 1 2024-03-30T04:28:17.597 INFO:tasks.workunit.client.0.smithi123.stdout:2/826: dread d0/d1f/d16/f87 [0,4194304] 0 2024-03-30T04:28:17.600 INFO:tasks.workunit.client.0.smithi123.stdout:0/695: symlink d6/dd/d15/le9 0 2024-03-30T04:28:17.601 INFO:tasks.workunit.client.1.smithi135.stdout:9/282: dread d7/f2f [0,4194304] 0 2024-03-30T04:28:17.602 INFO:tasks.workunit.client.0.smithi123.stdout:4/660: truncate d2/d4/d2c/d42/dad/d91/d9d/fb0 592709 0 2024-03-30T04:28:17.603 INFO:tasks.workunit.client.1.smithi135.stdout:4/304: rename d10/c20 to d10/d1a/d32/c6c 0 2024-03-30T04:28:17.603 INFO:tasks.workunit.client.1.smithi135.stdout:0/374: stat d0/d5/d9/d23/d2a/l57 0 2024-03-30T04:28:17.603 INFO:tasks.workunit.client.1.smithi135.stdout:9/283: stat d7/f22 0 2024-03-30T04:28:17.604 INFO:tasks.workunit.client.1.smithi135.stdout:6/290: dread d2/d2e/f37 [0,4194304] 0 2024-03-30T04:28:17.604 INFO:tasks.workunit.client.1.smithi135.stdout:4/305: fsync d10/d1a/d42/f54 0 2024-03-30T04:28:17.606 INFO:tasks.workunit.client.0.smithi123.stdout:8/697: creat da/dc/d14/d18/d21/da2/fd2 x:0 0 0 2024-03-30T04:28:17.606 INFO:tasks.workunit.client.1.smithi135.stdout:0/375: write d0/d5/d10/f2c [1306502,121156] 0 2024-03-30T04:28:17.607 INFO:tasks.workunit.client.0.smithi123.stdout:9/619: mknod d6/d1f/d45/cd4 0 2024-03-30T04:28:17.607 INFO:tasks.workunit.client.0.smithi123.stdout:1/765: creat db/dd/d42/d46/d5d/d62/d9e/fff x:0 0 0 2024-03-30T04:28:17.611 INFO:tasks.workunit.client.1.smithi135.stdout:4/306: mknod d10/c6d 0 2024-03-30T04:28:17.614 INFO:tasks.workunit.client.1.smithi135.stdout:4/307: truncate d10/d1a/d32/d40/f4b 38099 0 2024-03-30T04:28:17.615 INFO:tasks.workunit.client.0.smithi123.stdout:2/827: readlink d0/d1f/d82/ddf/l101 0 2024-03-30T04:28:17.615 INFO:tasks.workunit.client.1.smithi135.stdout:4/308: chown d10/d1a 0 1 2024-03-30T04:28:17.622 INFO:tasks.workunit.client.0.smithi123.stdout:6/758: dread d10/d20/f3b [4194304,4194304] 0 2024-03-30T04:28:17.622 INFO:tasks.workunit.client.0.smithi123.stdout:4/661: mkdir d2/d4/de0 0 2024-03-30T04:28:17.625 INFO:tasks.workunit.client.0.smithi123.stdout:1/766: mknod db/dd/d5f/d6d/d85/d89/d90/d94/d9c/c100 0 2024-03-30T04:28:17.626 INFO:tasks.workunit.client.1.smithi135.stdout:0/376: write d0/f3 [689986,118623] 0 2024-03-30T04:28:17.626 INFO:tasks.workunit.client.0.smithi123.stdout:3/769: dread d13/f1b [0,4194304] 0 2024-03-30T04:28:17.626 INFO:tasks.workunit.client.0.smithi123.stdout:8/698: rmdir da/dc/d14/d18/d62 39 2024-03-30T04:28:17.626 INFO:tasks.workunit.client.0.smithi123.stdout:6/759: fsync d10/d82/dc1/f8b 0 2024-03-30T04:28:17.629 INFO:tasks.workunit.client.0.smithi123.stdout:2/828: mknod d0/d1f/d1c/d39/de4/c110 0 2024-03-30T04:28:17.630 INFO:tasks.workunit.client.1.smithi135.stdout:0/377: mkdir d0/d5/d9/d23/d58/d63/d77 0 2024-03-30T04:28:17.633 INFO:tasks.workunit.client.1.smithi135.stdout:6/291: dwrite d2/d2e/f31 [0,4194304] 0 2024-03-30T04:28:17.633 INFO:tasks.workunit.client.0.smithi123.stdout:8/699: fsync da/dc/f20 0 2024-03-30T04:28:17.634 INFO:tasks.workunit.client.1.smithi135.stdout:0/378: write d0/d5/d9/d23/d2a/d60/f48 [642001,92637] 0 2024-03-30T04:28:17.634 INFO:tasks.workunit.client.0.smithi123.stdout:2/829: readlink d0/d1f/d1c/l45 0 2024-03-30T04:28:17.635 INFO:tasks.workunit.client.0.smithi123.stdout:3/770: write d13/d24/d39/d79/d90/fcf [174350,58391] 0 2024-03-30T04:28:17.637 INFO:tasks.workunit.client.1.smithi135.stdout:1/241: sync 2024-03-30T04:28:17.637 INFO:tasks.workunit.client.1.smithi135.stdout:3/299: sync 2024-03-30T04:28:17.638 INFO:tasks.workunit.client.1.smithi135.stdout:0/379: chown d0/d5/d9/d23/d58/l20 47 1 2024-03-30T04:28:17.644 INFO:tasks.workunit.client.1.smithi135.stdout:0/380: creat d0/d13/d1e/f78 x:0 0 0 2024-03-30T04:28:17.644 INFO:tasks.workunit.client.0.smithi123.stdout:7/699: dread d2/da/d11/d61/d55/d9e/fad [0,4194304] 0 2024-03-30T04:28:17.646 INFO:tasks.workunit.client.1.smithi135.stdout:3/300: creat d3/d54/d5c/f64 x:0 0 0 2024-03-30T04:28:17.646 INFO:tasks.workunit.client.1.smithi135.stdout:1/242: mkdir d16/d29/d2c/d30/d3d/d51 0 2024-03-30T04:28:17.649 INFO:tasks.workunit.client.0.smithi123.stdout:2/830: fdatasync d0/d1f/d1c/d36/fab 0 2024-03-30T04:28:17.649 INFO:tasks.workunit.client.0.smithi123.stdout:4/662: unlink d2/d4/d2c/d42/d80/d96/cdd 0 2024-03-30T04:28:17.649 INFO:tasks.workunit.client.1.smithi135.stdout:3/301: creat d3/d50/d46/f65 x:0 0 0 2024-03-30T04:28:17.650 INFO:tasks.workunit.client.1.smithi135.stdout:6/292: dread d2/dd/d14/d2b/f47 [0,4194304] 0 2024-03-30T04:28:17.651 INFO:tasks.workunit.client.1.smithi135.stdout:1/243: creat d16/d29/d2c/d30/d3a/f52 x:0 0 0 2024-03-30T04:28:17.652 INFO:tasks.workunit.client.1.smithi135.stdout:4/309: dread d10/d1a/f1f [0,4194304] 0 2024-03-30T04:28:17.654 INFO:tasks.workunit.client.1.smithi135.stdout:1/244: chown d16/d29/d2c/d3c 83 1 2024-03-30T04:28:17.655 INFO:tasks.workunit.client.0.smithi123.stdout:8/700: truncate da/d27/d3c/d2d/f4b 687258 0 2024-03-30T04:28:17.658 INFO:tasks.workunit.client.1.smithi135.stdout:6/293: rename d2/d36/f3d to d2/dd/d40/d52/f5a 0 2024-03-30T04:28:17.658 INFO:tasks.workunit.client.1.smithi135.stdout:4/310: rmdir d10/d1a/d3d 39 2024-03-30T04:28:17.659 INFO:tasks.workunit.client.0.smithi123.stdout:1/767: rename db/dd/d42/d46/d56/d5c/c69 to db/dd/d42/d46/d56/d5c/c101 0 2024-03-30T04:28:17.660 INFO:tasks.workunit.client.1.smithi135.stdout:1/245: creat d16/d19/d20/d26/f53 x:0 0 0 2024-03-30T04:28:17.661 INFO:tasks.workunit.client.0.smithi123.stdout:3/771: symlink d13/d24/d31/d23/d3a/d3c/d66/l100 0 2024-03-30T04:28:17.665 INFO:tasks.workunit.client.0.smithi123.stdout:2/831: mkdir d0/d1f/d5e/d111 0 2024-03-30T04:28:17.666 INFO:tasks.workunit.client.1.smithi135.stdout:6/294: symlink d2/dd/d14/d53/l5b 0 2024-03-30T04:28:17.666 INFO:tasks.workunit.client.0.smithi123.stdout:5/626: dwrite dd/d19/d1f/d3c/fab [0,4194304] 0 2024-03-30T04:28:17.670 INFO:tasks.workunit.client.1.smithi135.stdout:0/381: dwrite d0/d5/d9/d23/f66 [0,4194304] 0 2024-03-30T04:28:17.671 INFO:tasks.workunit.client.0.smithi123.stdout:9/620: dwrite d6/d1f/d72/f8b [0,4194304] 0 2024-03-30T04:28:17.671 INFO:tasks.workunit.client.1.smithi135.stdout:6/295: readlink d2/dd/d14/d53/l5b 0 2024-03-30T04:28:17.671 INFO:tasks.workunit.client.0.smithi123.stdout:1/768: dread - db/dd/d5f/dae/ffd zero size 2024-03-30T04:28:17.672 INFO:tasks.workunit.client.0.smithi123.stdout:0/696: dread d6/dd/d15/fb3 [0,4194304] 0 2024-03-30T04:28:17.673 INFO:tasks.workunit.client.1.smithi135.stdout:1/246: link d16/d1a/c41 d16/d29/d2c/d43/c54 0 2024-03-30T04:28:17.674 INFO:tasks.workunit.client.1.smithi135.stdout:0/382: chown d0/d19/d2d/d45/f76 219 1 2024-03-30T04:28:17.675 INFO:tasks.workunit.client.1.smithi135.stdout:4/311: write d10/f17 [4988001,61999] 0 2024-03-30T04:28:17.678 INFO:tasks.workunit.client.1.smithi135.stdout:6/296: symlink d2/dd/d14/d53/l5c 0 2024-03-30T04:28:17.680 INFO:tasks.workunit.client.0.smithi123.stdout:1/769: chown db/d1b/f3e 7448002 1 2024-03-30T04:28:17.681 INFO:tasks.workunit.client.1.smithi135.stdout:3/302: dwrite d3/d10/d16/f43 [0,4194304] 0 2024-03-30T04:28:17.685 INFO:tasks.workunit.client.0.smithi123.stdout:2/832: unlink d0/d1f/d16/fcb 0 2024-03-30T04:28:17.685 INFO:tasks.workunit.client.0.smithi123.stdout:6/760: dwrite d10/d82/dc1/d6a/fca [0,4194304] 0 2024-03-30T04:28:17.686 INFO:tasks.workunit.client.1.smithi135.stdout:4/312: creat d10/d1a/d42/f6e x:0 0 0 2024-03-30T04:28:17.689 INFO:tasks.workunit.client.0.smithi123.stdout:2/833: truncate d0/d1f/d1c/fb3 531798 0 2024-03-30T04:28:17.690 INFO:tasks.workunit.client.1.smithi135.stdout:0/383: mkdir d0/d19/d2d/d79 0 2024-03-30T04:28:17.691 INFO:tasks.workunit.client.0.smithi123.stdout:2/834: readlink d0/d1f/d1c/l45 0 2024-03-30T04:28:17.692 INFO:tasks.workunit.client.0.smithi123.stdout:8/701: mknod da/cd3 0 2024-03-30T04:28:17.692 INFO:tasks.workunit.client.0.smithi123.stdout:3/772: creat d13/d24/d39/d79/dea/dee/f101 x:0 0 0 2024-03-30T04:28:17.692 INFO:tasks.workunit.client.1.smithi135.stdout:3/303: rename d3/f5 to d3/d50/d22/d33/d62/f66 0 2024-03-30T04:28:17.694 INFO:tasks.workunit.client.0.smithi123.stdout:5/627: rmdir dd/d19/d1f/d3c/d72 39 2024-03-30T04:28:17.701 INFO:tasks.workunit.client.0.smithi123.stdout:4/663: link d2/d4/d2c/d42/d80/l97 d2/d26/le1 0 2024-03-30T04:28:17.702 INFO:tasks.workunit.client.1.smithi135.stdout:3/304: symlink d3/d54/l67 0 2024-03-30T04:28:17.707 INFO:tasks.workunit.client.1.smithi135.stdout:3/305: mkdir d3/d10/d16/d23/d68 0 2024-03-30T04:28:17.707 INFO:tasks.workunit.client.1.smithi135.stdout:1/247: dwrite d16/d1a/f23 [0,4194304] 0 2024-03-30T04:28:17.709 INFO:tasks.workunit.client.1.smithi135.stdout:3/306: chown d3/d10/f5e 64892 1 2024-03-30T04:28:17.710 INFO:tasks.workunit.client.0.smithi123.stdout:0/697: mknod d6/dd/d31/db8/cea 0 2024-03-30T04:28:17.712 INFO:tasks.workunit.client.1.smithi135.stdout:1/248: mkdir d16/d29/d32/d55 0 2024-03-30T04:28:17.712 INFO:tasks.workunit.client.1.smithi135.stdout:3/307: fdatasync d3/d50/d46/f58 0 2024-03-30T04:28:17.713 INFO:tasks.workunit.client.0.smithi123.stdout:5/628: symlink dd/d19/d1f/d35/d3a/d79/lce 0 2024-03-30T04:28:17.714 INFO:tasks.workunit.client.0.smithi123.stdout:3/773: stat d13/d24/d31/d23/d3a/d3c/d4e/d5d/dc7 0 2024-03-30T04:28:17.715 INFO:tasks.workunit.client.0.smithi123.stdout:9/621: dread d6/d1f/d72/f8b [0,4194304] 0 2024-03-30T04:28:17.718 INFO:tasks.workunit.client.1.smithi135.stdout:1/249: truncate d16/d29/d2c/d3c/f3b 314864 0 2024-03-30T04:28:17.718 INFO:tasks.workunit.client.0.smithi123.stdout:8/702: mkdir da/d27/d56/d6c/dd4 0 2024-03-30T04:28:17.719 INFO:tasks.workunit.client.0.smithi123.stdout:4/664: stat d2/d4/d2c/d42/d35/d5d 0 2024-03-30T04:28:17.720 INFO:tasks.workunit.client.1.smithi135.stdout:1/250: chown d16/d29/d2c/d30/d3a 1051203 1 2024-03-30T04:28:17.722 INFO:tasks.workunit.client.0.smithi123.stdout:6/761: dwrite d10/d82/d8e/ffe [0,4194304] 0 2024-03-30T04:28:17.725 INFO:tasks.workunit.client.1.smithi135.stdout:6/297: truncate d2/d36/f44 2628468 0 2024-03-30T04:28:17.725 INFO:tasks.workunit.client.0.smithi123.stdout:5/629: fdatasync dd/d19/d1f/d35/d3a/d57/f6a 0 2024-03-30T04:28:17.726 INFO:tasks.workunit.client.0.smithi123.stdout:7/700: dread d2/da/d11/d61/d55/d59/f88 [0,4194304] 0 2024-03-30T04:28:17.728 INFO:tasks.workunit.client.1.smithi135.stdout:1/251: creat d16/d19/f56 x:0 0 0 2024-03-30T04:28:17.728 INFO:tasks.workunit.client.1.smithi135.stdout:6/298: truncate d2/d2e/f37 4954803 0 2024-03-30T04:28:17.732 INFO:tasks.workunit.client.0.smithi123.stdout:1/770: dread db/dd/f48 [0,4194304] 0 2024-03-30T04:28:17.732 INFO:tasks.workunit.client.1.smithi135.stdout:6/299: fsync d2/dd/d40/d52/f5a 0 2024-03-30T04:28:17.735 INFO:tasks.workunit.client.1.smithi135.stdout:1/252: creat d16/f57 x:0 0 0 2024-03-30T04:28:17.735 INFO:tasks.workunit.client.1.smithi135.stdout:3/308: fsync d3/d10/d16/f43 0 2024-03-30T04:28:17.739 INFO:tasks.workunit.client.1.smithi135.stdout:6/300: mknod d2/d36/c5d 0 2024-03-30T04:28:17.746 INFO:tasks.workunit.client.1.smithi135.stdout:1/253: unlink d16/d29/d2c/f2e 0 2024-03-30T04:28:17.749 INFO:tasks.workunit.client.1.smithi135.stdout:0/384: write d0/d1/d16/f1b [291109,106241] 0 2024-03-30T04:28:17.752 INFO:tasks.workunit.client.1.smithi135.stdout:4/313: truncate d10/f57 2721323 0 2024-03-30T04:28:17.753 INFO:tasks.workunit.client.1.smithi135.stdout:0/385: dread - d0/d13/d1e/d3a/f5a zero size 2024-03-30T04:28:17.754 INFO:tasks.workunit.client.1.smithi135.stdout:1/254: rename f5 to d16/d29/d32/f58 0 2024-03-30T04:28:17.756 INFO:tasks.workunit.client.1.smithi135.stdout:3/309: link d3/l8 d3/d10/l69 0 2024-03-30T04:28:17.758 INFO:tasks.workunit.client.1.smithi135.stdout:4/314: unlink c8 0 2024-03-30T04:28:17.759 INFO:tasks.workunit.client.1.smithi135.stdout:0/386: write d0/d5/d9/d23/d58/d55/f5f [422156,26457] 0 2024-03-30T04:28:17.766 INFO:tasks.workunit.client.1.smithi135.stdout:0/387: mknod d0/d19/d2d/d45/c7a 0 2024-03-30T04:28:17.767 INFO:tasks.workunit.client.1.smithi135.stdout:4/315: mknod d10/d1a/c6f 0 2024-03-30T04:28:17.769 INFO:tasks.workunit.client.1.smithi135.stdout:5/313: sync 2024-03-30T04:28:17.769 INFO:tasks.workunit.client.1.smithi135.stdout:8/288: sync 2024-03-30T04:28:17.769 INFO:tasks.workunit.client.1.smithi135.stdout:7/339: sync 2024-03-30T04:28:17.772 INFO:tasks.workunit.client.1.smithi135.stdout:5/314: chown d1/f39 7 1 2024-03-30T04:28:17.775 INFO:tasks.workunit.client.1.smithi135.stdout:5/315: fdatasync d1/d7/d11/f30 0 2024-03-30T04:28:17.775 INFO:tasks.workunit.client.1.smithi135.stdout:0/388: symlink d0/d13/d1e/d3a/l7b 0 2024-03-30T04:28:17.778 INFO:tasks.workunit.client.1.smithi135.stdout:1/255: chown d16/d29/d32/f58 103 1 2024-03-30T04:28:17.782 INFO:tasks.workunit.client.1.smithi135.stdout:8/289: dread f0 [8388608,4194304] 0 2024-03-30T04:28:17.782 INFO:tasks.workunit.client.1.smithi135.stdout:5/316: read d1/d7/f43 [1236907,77852] 0 2024-03-30T04:28:17.785 INFO:tasks.workunit.client.1.smithi135.stdout:1/256: write d16/d29/f35 [1390222,20669] 0 2024-03-30T04:28:17.785 INFO:tasks.workunit.client.1.smithi135.stdout:0/389: rmdir d0/d13/d53/d54 39 2024-03-30T04:28:17.788 INFO:tasks.workunit.client.0.smithi123.stdout:9/622: creat d6/d1f/d45/d97/d69/daa/fd5 x:0 0 0 2024-03-30T04:28:17.789 INFO:tasks.workunit.client.1.smithi135.stdout:4/316: link d10/d1a/d42/f63 d10/d1a/d32/f70 0 2024-03-30T04:28:17.791 INFO:tasks.workunit.client.1.smithi135.stdout:6/301: dwrite d2/dd/f1a [0,4194304] 0 2024-03-30T04:28:17.791 INFO:tasks.workunit.client.1.smithi135.stdout:4/317: fdatasync d10/f14 0 2024-03-30T04:28:17.793 INFO:tasks.workunit.client.1.smithi135.stdout:8/290: mknod d6/d8/d15/d2a/d35/c4d 0 2024-03-30T04:28:17.793 INFO:tasks.workunit.client.1.smithi135.stdout:1/257: write d16/d29/d2c/d30/d3a/f52 [447276,90051] 0 2024-03-30T04:28:17.794 INFO:tasks.workunit.client.0.smithi123.stdout:6/762: creat d10/d82/dc1/d6a/d7a/de8/f102 x:0 0 0 2024-03-30T04:28:17.795 INFO:tasks.workunit.client.0.smithi123.stdout:1/771: creat db/dd/d5f/d6d/d85/d89/d90/d94/d9c/f102 x:0 0 0 2024-03-30T04:28:17.797 INFO:tasks.workunit.client.0.smithi123.stdout:1/772: chown db/dd/d5f 24716 1 2024-03-30T04:28:17.798 INFO:tasks.workunit.client.0.smithi123.stdout:4/665: creat d2/d4/d2c/d42/d35/d9b/da2/fe2 x:0 0 0 2024-03-30T04:28:17.801 INFO:tasks.workunit.client.1.smithi135.stdout:3/310: dwrite d3/d50/d22/f2e [0,4194304] 0 2024-03-30T04:28:17.801 INFO:tasks.workunit.client.0.smithi123.stdout:9/623: symlink d6/d1f/da3/ld6 0 2024-03-30T04:28:17.801 INFO:tasks.workunit.client.0.smithi123.stdout:3/774: mknod d13/d24/d31/d23/d3a/d3c/db8/df4/c102 0 2024-03-30T04:28:17.803 INFO:tasks.workunit.client.0.smithi123.stdout:8/703: mknod da/dc/d14/d18/d62/cd5 0 2024-03-30T04:28:17.807 INFO:tasks.workunit.client.0.smithi123.stdout:3/775: chown d13/d24/d39/l46 2 1 2024-03-30T04:28:17.807 INFO:tasks.workunit.client.0.smithi123.stdout:1/773: creat db/dd/d5f/dae/f103 x:0 0 0 2024-03-30T04:28:17.808 INFO:tasks.workunit.client.1.smithi135.stdout:0/390: mknod d0/d5/d9/d23/d58/d63/d77/c7c 0 2024-03-30T04:28:17.810 INFO:tasks.workunit.client.1.smithi135.stdout:8/291: write d6/d8/d14/f21 [593130,67463] 0 2024-03-30T04:28:17.810 INFO:tasks.workunit.client.0.smithi123.stdout:8/704: write da/d27/d56/f65 [5631533,27398] 0 2024-03-30T04:28:17.811 INFO:tasks.workunit.client.1.smithi135.stdout:1/258: rmdir d16/d29/d2c/d43 39 2024-03-30T04:28:17.812 INFO:tasks.workunit.client.1.smithi135.stdout:6/302: chown d2/dd/d14/d53/l5c 4 1 2024-03-30T04:28:17.812 INFO:tasks.workunit.client.1.smithi135.stdout:4/318: mknod d10/d1a/d3d/c71 0 2024-03-30T04:28:17.816 INFO:tasks.workunit.client.0.smithi123.stdout:4/666: write d2/d4/d2c/d42/d35/d5d/fae [628681,116050] 0 2024-03-30T04:28:17.817 INFO:tasks.workunit.client.1.smithi135.stdout:0/391: stat d0/d5/d9/ff 0 2024-03-30T04:28:17.821 INFO:tasks.workunit.client.1.smithi135.stdout:8/292: stat d6/d8/d15/d2a/d35/c4d 0 2024-03-30T04:28:17.821 INFO:tasks.workunit.client.0.smithi123.stdout:2/835: dwrite d0/d1f/d16/f18 [0,4194304] 0 2024-03-30T04:28:17.823 INFO:tasks.workunit.client.1.smithi135.stdout:3/311: mkdir d3/d10/d16/d23/d6a 0 2024-03-30T04:28:17.824 INFO:tasks.workunit.client.0.smithi123.stdout:2/836: write d0/d1f/d27/df7/f104 [861665,31032] 0 2024-03-30T04:28:17.830 INFO:tasks.workunit.client.1.smithi135.stdout:7/340: dwrite f3 [0,4194304] 0 2024-03-30T04:28:17.833 INFO:tasks.workunit.client.0.smithi123.stdout:8/705: symlink da/dc/d14/d18/d21/da2/ld6 0 2024-03-30T04:28:17.837 INFO:tasks.workunit.client.1.smithi135.stdout:5/317: dwrite d1/d7/f48 [0,4194304] 0 2024-03-30T04:28:17.837 INFO:tasks.workunit.client.1.smithi135.stdout:8/293: fdatasync d6/d8/d15/d2a/f33 0 2024-03-30T04:28:17.838 INFO:tasks.workunit.client.1.smithi135.stdout:1/259: rmdir d16/d19/d20/d26 39 2024-03-30T04:28:17.838 INFO:tasks.workunit.client.0.smithi123.stdout:2/837: fsync d0/d1f/d16/f87 0 2024-03-30T04:28:17.841 INFO:tasks.workunit.client.1.smithi135.stdout:5/318: dread - d1/d55/f57 zero size 2024-03-30T04:28:17.842 INFO:tasks.workunit.client.1.smithi135.stdout:7/341: chown d4/db/c65 0 1 2024-03-30T04:28:17.845 INFO:tasks.workunit.client.0.smithi123.stdout:8/706: truncate da/f92 888367 0 2024-03-30T04:28:17.845 INFO:tasks.workunit.client.0.smithi123.stdout:2/838: symlink d0/d1f/d1c/d80/de6/l112 0 2024-03-30T04:28:17.846 INFO:tasks.workunit.client.0.smithi123.stdout:4/667: creat d2/d4/fe3 x:0 0 0 2024-03-30T04:28:17.847 INFO:tasks.workunit.client.1.smithi135.stdout:0/392: symlink d0/d19/d2d/d79/l7d 0 2024-03-30T04:28:17.852 INFO:tasks.workunit.client.1.smithi135.stdout:2/260: sync 2024-03-30T04:28:17.852 INFO:tasks.workunit.client.1.smithi135.stdout:9/284: sync 2024-03-30T04:28:17.858 INFO:tasks.workunit.client.0.smithi123.stdout:4/668: creat d2/d26/fe4 x:0 0 0 2024-03-30T04:28:17.868 INFO:tasks.workunit.client.1.smithi135.stdout:1/260: mknod d16/d29/c59 0 2024-03-30T04:28:17.870 INFO:tasks.workunit.client.1.smithi135.stdout:1/261: rmdir d16/d29/d2c/d3c 39 2024-03-30T04:28:17.870 INFO:tasks.workunit.client.0.smithi123.stdout:4/669: write d2/d4/d2c/d42/d35/d5d/d78/f85 [1677178,32972] 0 2024-03-30T04:28:17.871 INFO:tasks.workunit.client.1.smithi135.stdout:0/393: getdents d0/d5/d42 0 2024-03-30T04:28:17.871 INFO:tasks.workunit.client.1.smithi135.stdout:6/303: dwrite d2/dd/d14/d38/f3e [0,4194304] 0 2024-03-30T04:28:17.876 INFO:tasks.workunit.client.0.smithi123.stdout:2/839: dwrite d0/d1f/f8f [0,4194304] 0 2024-03-30T04:28:17.878 INFO:tasks.workunit.client.1.smithi135.stdout:4/319: dread fa [0,4194304] 0 2024-03-30T04:28:17.887 INFO:tasks.workunit.client.0.smithi123.stdout:4/670: creat d2/d4/d7/fe5 x:0 0 0 2024-03-30T04:28:17.889 INFO:tasks.workunit.client.1.smithi135.stdout:0/394: write d0/d19/d2d/d45/f76 [373582,5225] 0 2024-03-30T04:28:17.894 INFO:tasks.workunit.client.0.smithi123.stdout:2/840: dread d0/d1f/f2d [0,4194304] 0 2024-03-30T04:28:17.894 INFO:tasks.workunit.client.0.smithi123.stdout:4/671: mkdir d2/d4/d2c/d42/dad/de6 0 2024-03-30T04:28:17.895 INFO:tasks.workunit.client.0.smithi123.stdout:0/698: write d6/dd/d31/d35/f45 [3358481,96622] 0 2024-03-30T04:28:17.899 INFO:tasks.workunit.client.1.smithi135.stdout:3/312: dread d3/db/f34 [0,4194304] 0 2024-03-30T04:28:17.900 INFO:tasks.workunit.client.0.smithi123.stdout:2/841: symlink d0/d37/dec/l113 0 2024-03-30T04:28:17.902 INFO:tasks.workunit.client.0.smithi123.stdout:9/624: dread d6/d23/f5d [0,4194304] 0 2024-03-30T04:28:17.904 INFO:tasks.workunit.client.0.smithi123.stdout:9/625: stat d6/d1f/lbd 0 2024-03-30T04:28:17.909 INFO:tasks.workunit.client.1.smithi135.stdout:8/294: dwrite d6/f25 [0,4194304] 0 2024-03-30T04:28:17.912 INFO:tasks.workunit.client.1.smithi135.stdout:7/342: dread d4/db/de/f1d [0,4194304] 0 2024-03-30T04:28:17.913 INFO:tasks.workunit.client.0.smithi123.stdout:9/626: creat d6/dad/fd7 x:0 0 0 2024-03-30T04:28:17.913 INFO:tasks.workunit.client.1.smithi135.stdout:2/261: dwrite d4/da/f1e [0,4194304] 0 2024-03-30T04:28:17.914 INFO:tasks.workunit.client.0.smithi123.stdout:5/630: truncate dd/d19/d1f/d3c/d72/d8c/fac 67197 0 2024-03-30T04:28:17.914 INFO:tasks.workunit.client.0.smithi123.stdout:9/627: fsync d6/f7e 0 2024-03-30T04:28:17.915 INFO:tasks.workunit.client.1.smithi135.stdout:5/319: dread d1/d7/ff [0,4194304] 0 2024-03-30T04:28:17.915 INFO:tasks.workunit.client.1.smithi135.stdout:8/295: truncate d6/d8/d14/f30 1337007 0 2024-03-30T04:28:17.918 INFO:tasks.workunit.client.1.smithi135.stdout:5/320: chown d1/d3/c35 830986152 1 2024-03-30T04:28:17.918 INFO:tasks.workunit.client.1.smithi135.stdout:1/262: dread d16/d29/d2c/d30/f4e [0,4194304] 0 2024-03-30T04:28:17.919 INFO:tasks.workunit.client.1.smithi135.stdout:7/343: chown d4/db/d1a/d4b/f31 29287 1 2024-03-30T04:28:17.920 INFO:tasks.workunit.client.1.smithi135.stdout:8/296: write d6/f25 [480481,16973] 0 2024-03-30T04:28:17.920 INFO:tasks.workunit.client.0.smithi123.stdout:9/628: symlink d6/d1f/d45/d97/d69/ld8 0 2024-03-30T04:28:17.923 INFO:tasks.workunit.client.1.smithi135.stdout:8/297: dread - d6/d8/d15/f2b zero size 2024-03-30T04:28:17.933 INFO:tasks.workunit.client.1.smithi135.stdout:5/321: dread d1/d3/d2b/f5b [0,4194304] 0 2024-03-30T04:28:17.937 INFO:tasks.workunit.client.0.smithi123.stdout:5/631: rename dd/d19/d1f/d35/d3a/f83 to dd/d19/d1f/d3c/fcf 0 2024-03-30T04:28:17.938 INFO:tasks.workunit.client.1.smithi135.stdout:5/322: write d1/f3e [2899592,109099] 0 2024-03-30T04:28:17.939 INFO:tasks.workunit.client.0.smithi123.stdout:4/672: fsync d2/d26/fe4 0 2024-03-30T04:28:17.940 INFO:tasks.workunit.client.0.smithi123.stdout:1/774: dwrite db/dd/d42/d46/f2e [0,4194304] 0 2024-03-30T04:28:17.941 INFO:tasks.workunit.client.0.smithi123.stdout:3/776: dwrite d13/d24/d31/d30/f45 [4194304,4194304] 0 2024-03-30T04:28:17.945 INFO:tasks.workunit.client.0.smithi123.stdout:7/701: mknod d2/d4d/ce7 0 2024-03-30T04:28:17.947 INFO:tasks.workunit.client.1.smithi135.stdout:6/304: symlink d2/dd/d18/d21/l5e 0 2024-03-30T04:28:17.952 INFO:tasks.workunit.client.1.smithi135.stdout:4/320: fsync d10/d1a/d42/f5e 0 2024-03-30T04:28:17.954 INFO:tasks.workunit.client.0.smithi123.stdout:1/775: symlink db/dd/d5f/d6d/d85/l104 0 2024-03-30T04:28:17.958 INFO:tasks.workunit.client.1.smithi135.stdout:0/395: unlink d0/d5/d9/d23/d2a/l57 0 2024-03-30T04:28:17.963 INFO:tasks.workunit.client.1.smithi135.stdout:3/313: write d3/d10/f14 [4601191,9714] 0 2024-03-30T04:28:17.964 INFO:tasks.workunit.client.0.smithi123.stdout:0/699: dread d6/dd/d15/d28/fcc [0,4194304] 0 2024-03-30T04:28:17.969 INFO:tasks.workunit.client.0.smithi123.stdout:0/700: chown d6/dd/d18/d2f/da3 11998 1 2024-03-30T04:28:17.970 INFO:tasks.workunit.client.0.smithi123.stdout:8/707: dwrite da/d27/d3c/d2d/f4b [0,4194304] 0 2024-03-30T04:28:17.970 INFO:tasks.workunit.client.0.smithi123.stdout:1/776: rmdir db/dd/d42/d46/d39 39 2024-03-30T04:28:17.977 INFO:tasks.workunit.client.0.smithi123.stdout:2/842: dwrite d0/d1f/d27/d68/dd6/f7c [0,4194304] 0 2024-03-30T04:28:17.977 INFO:tasks.workunit.client.0.smithi123.stdout:7/702: creat d2/dc/fe8 x:0 0 0 2024-03-30T04:28:17.979 INFO:tasks.workunit.client.1.smithi135.stdout:2/262: mknod d4/d32/d12/d45/d3c/c54 0 2024-03-30T04:28:17.979 INFO:tasks.workunit.client.0.smithi123.stdout:7/703: readlink d2/da/l44 0 2024-03-30T04:28:17.980 INFO:tasks.workunit.client.0.smithi123.stdout:8/708: unlink da/dc/c23 0 2024-03-30T04:28:17.983 INFO:tasks.workunit.client.1.smithi135.stdout:7/344: fsync d4/db/d1a/f24 0 2024-03-30T04:28:17.984 INFO:tasks.workunit.client.0.smithi123.stdout:8/709: readlink da/d27/l82 0 2024-03-30T04:28:17.985 INFO:tasks.workunit.client.1.smithi135.stdout:8/298: creat d6/d8/d14/f4e x:0 0 0 2024-03-30T04:28:17.985 INFO:tasks.workunit.client.0.smithi123.stdout:0/701: read d6/dd/d15/d98/d4e/d69/d6e/f9f [859693,49389] 0 2024-03-30T04:28:17.988 INFO:tasks.workunit.client.1.smithi135.stdout:8/299: write d6/d8/d15/d1f/f44 [225342,54037] 0 2024-03-30T04:28:17.990 INFO:tasks.workunit.client.1.smithi135.stdout:5/323: mknod d1/d55/c60 0 2024-03-30T04:28:17.993 INFO:tasks.workunit.client.0.smithi123.stdout:2/843: write d0/d1f/d1c/d80/de6/f107 [847601,111379] 0 2024-03-30T04:28:17.994 INFO:tasks.workunit.client.1.smithi135.stdout:5/324: write d1/d7/f10 [3991849,13123] 0 2024-03-30T04:28:17.994 INFO:tasks.workunit.client.1.smithi135.stdout:6/305: truncate d2/dd/d40/d52/f5a 1732437 0 2024-03-30T04:28:17.998 INFO:tasks.workunit.client.1.smithi135.stdout:4/321: rmdir d10/d1a/d42 39 2024-03-30T04:28:17.998 INFO:tasks.workunit.client.0.smithi123.stdout:0/702: write d6/dd/d15/d28/d4b/f91 [875588,64518] 0 2024-03-30T04:28:18.000 INFO:tasks.workunit.client.1.smithi135.stdout:2/263: dwrite d4/d32/d2e/f40 [0,4194304] 0 2024-03-30T04:28:18.002 INFO:tasks.workunit.client.1.smithi135.stdout:0/396: mkdir d0/d1/d16/d7e 0 2024-03-30T04:28:18.002 INFO:tasks.workunit.client.0.smithi123.stdout:9/629: dwrite d6/d1f/d72/f93 [4194304,4194304] 0 2024-03-30T04:28:18.003 INFO:tasks.workunit.client.1.smithi135.stdout:2/264: readlink d4/d6/l27 0 2024-03-30T04:28:18.004 INFO:tasks.workunit.client.1.smithi135.stdout:2/265: stat d4/f1c 0 2024-03-30T04:28:18.005 INFO:tasks.workunit.client.1.smithi135.stdout:2/266: stat d4/d32/d2e 0 2024-03-30T04:28:18.008 INFO:tasks.workunit.client.0.smithi123.stdout:5/632: dread dd/f32 [0,4194304] 0 2024-03-30T04:28:18.010 INFO:tasks.workunit.client.1.smithi135.stdout:2/267: fdatasync d4/d6/f1a 0 2024-03-30T04:28:18.010 INFO:tasks.workunit.client.0.smithi123.stdout:8/710: creat da/dc/d14/d18/fd7 x:0 0 0 2024-03-30T04:28:18.011 INFO:tasks.workunit.client.1.smithi135.stdout:2/268: chown d4/d32/f13 424 1 2024-03-30T04:28:18.014 INFO:tasks.workunit.client.0.smithi123.stdout:2/844: mknod d0/d1f/d1c/d80/dbb/dd5/c114 0 2024-03-30T04:28:18.015 INFO:tasks.workunit.client.1.smithi135.stdout:8/300: mkdir d6/d8/d15/d2a/d47/d4f 0 2024-03-30T04:28:18.015 INFO:tasks.workunit.client.0.smithi123.stdout:7/704: truncate d2/f13 323780 0 2024-03-30T04:28:18.019 INFO:tasks.workunit.client.0.smithi123.stdout:9/630: creat d6/d1f/d4d/fd9 x:0 0 0 2024-03-30T04:28:18.020 INFO:tasks.workunit.client.1.smithi135.stdout:5/325: fsync d1/da/d17/f3d 0 2024-03-30T04:28:18.020 INFO:tasks.workunit.client.0.smithi123.stdout:5/633: fdatasync dd/d19/d1f/d35/d3a/d57/f65 0 2024-03-30T04:28:18.022 INFO:tasks.workunit.client.0.smithi123.stdout:4/673: dwrite d2/d4/d2c/d42/d35/f5c [0,4194304] 0 2024-03-30T04:28:18.024 INFO:tasks.workunit.client.0.smithi123.stdout:7/705: dread - d2/da/d11/d34/d85/fac zero size 2024-03-30T04:28:18.026 INFO:tasks.workunit.client.0.smithi123.stdout:9/631: dread - d6/dc9/fce zero size 2024-03-30T04:28:18.028 INFO:tasks.workunit.client.0.smithi123.stdout:3/777: dread d13/d24/d31/f4b [0,4194304] 0 2024-03-30T04:28:18.029 INFO:tasks.workunit.client.1.smithi135.stdout:0/397: dwrite d0/d1/d16/f4b [0,4194304] 0 2024-03-30T04:28:18.032 INFO:tasks.workunit.client.0.smithi123.stdout:5/634: unlink dd/f7c 0 2024-03-30T04:28:18.033 INFO:tasks.workunit.client.0.smithi123.stdout:1/777: dwrite db/dd/d5f/d6d/d85/dd4/fea [0,4194304] 0 2024-03-30T04:28:18.035 INFO:tasks.workunit.client.0.smithi123.stdout:9/632: write f1 [79778,16276] 0 2024-03-30T04:28:18.036 INFO:tasks.workunit.client.1.smithi135.stdout:8/301: rmdir d6/d8/d14/d32 39 2024-03-30T04:28:18.041 INFO:tasks.workunit.client.0.smithi123.stdout:7/706: write d2/da/d11/d61/d55/d59/db5/dab/dca/fd7 [706886,64189] 0 2024-03-30T04:28:18.042 INFO:tasks.workunit.client.0.smithi123.stdout:9/633: chown d6/d49/c53 876 1 2024-03-30T04:28:18.045 INFO:tasks.workunit.client.0.smithi123.stdout:5/635: rmdir dd/d19/d3e/d5b/d62 39 2024-03-30T04:28:18.050 INFO:tasks.workunit.client.0.smithi123.stdout:1/778: creat db/dd/d42/d46/d56/d5c/dd0/f105 x:0 0 0 2024-03-30T04:28:18.052 INFO:tasks.workunit.client.0.smithi123.stdout:5/636: dread - dd/d19/d1f/d3c/fbd zero size 2024-03-30T04:28:18.053 INFO:tasks.workunit.client.0.smithi123.stdout:8/711: dwrite da/d27/d3c/d64/f81 [0,4194304] 0 2024-03-30T04:28:18.053 INFO:tasks.workunit.client.1.smithi135.stdout:4/322: mkdir d10/d72 0 2024-03-30T04:28:18.054 INFO:tasks.workunit.client.0.smithi123.stdout:4/674: link d2/d4/d7/f4e d2/d4/fe7 0 2024-03-30T04:28:18.055 INFO:tasks.workunit.client.1.smithi135.stdout:4/323: write d10/d1a/f1f [4188263,13976] 0 2024-03-30T04:28:18.057 INFO:tasks.workunit.client.0.smithi123.stdout:1/779: chown db/dd/d42/d46/d51/fb1 11411779 1 2024-03-30T04:28:18.059 INFO:tasks.workunit.client.0.smithi123.stdout:7/707: mknod d2/da/d11/d34/db0/ce9 0 2024-03-30T04:28:18.062 INFO:tasks.workunit.client.0.smithi123.stdout:6/763: sync 2024-03-30T04:28:18.066 INFO:tasks.workunit.client.0.smithi123.stdout:8/712: write da/dc/d14/d18/f30 [2212361,63659] 0 2024-03-30T04:28:18.066 INFO:tasks.workunit.client.0.smithi123.stdout:6/764: chown d10/d20/da2 14938 1 2024-03-30T04:28:18.067 INFO:tasks.workunit.client.1.smithi135.stdout:5/326: mknod d1/da/d4d/c61 0 2024-03-30T04:28:18.067 INFO:tasks.workunit.client.0.smithi123.stdout:1/780: rmdir db/d1b 39 2024-03-30T04:28:18.071 INFO:tasks.workunit.client.1.smithi135.stdout:5/327: chown d1/d7/d11/l41 7405256 1 2024-03-30T04:28:18.072 INFO:tasks.workunit.client.1.smithi135.stdout:5/328: chown d1/da/d4d/c61 636 1 2024-03-30T04:28:18.074 INFO:tasks.workunit.client.1.smithi135.stdout:4/324: creat d10/f73 x:0 0 0 2024-03-30T04:28:18.075 INFO:tasks.workunit.client.0.smithi123.stdout:2/845: write d0/d1f/d16/f26 [1446940,8742] 0 2024-03-30T04:28:18.076 INFO:tasks.workunit.client.0.smithi123.stdout:8/713: creat da/d27/fd8 x:0 0 0 2024-03-30T04:28:18.080 INFO:tasks.workunit.client.1.smithi135.stdout:4/325: write d10/d1a/f27 [221878,81593] 0 2024-03-30T04:28:18.080 INFO:tasks.workunit.client.1.smithi135.stdout:5/329: unlink d1/d7/d11/d3c/c5f 0 2024-03-30T04:28:18.080 INFO:tasks.workunit.client.0.smithi123.stdout:1/781: fdatasync db/dd/d42/d46/f53 0 2024-03-30T04:28:18.084 INFO:tasks.workunit.client.0.smithi123.stdout:7/708: getdents d2/da/d22/d3b 0 2024-03-30T04:28:18.085 INFO:tasks.workunit.client.1.smithi135.stdout:8/302: creat d6/f50 x:0 0 0 2024-03-30T04:28:18.086 INFO:tasks.workunit.client.1.smithi135.stdout:0/398: dread d0/d5/d9/fd [0,4194304] 0 2024-03-30T04:28:18.090 INFO:tasks.workunit.client.0.smithi123.stdout:8/714: mkdir da/d27/d3c/d2d/dae/dd9 0 2024-03-30T04:28:18.091 INFO:tasks.workunit.client.1.smithi135.stdout:7/345: symlink d4/db/d1a/l70 0 2024-03-30T04:28:18.093 INFO:tasks.workunit.client.0.smithi123.stdout:2/846: fdatasync d0/d1f/d1c/d39/f96 0 2024-03-30T04:28:18.093 INFO:tasks.workunit.client.0.smithi123.stdout:7/709: write d2/da/d22/d3b/f4b [1550906,83391] 0 2024-03-30T04:28:18.093 INFO:tasks.workunit.client.0.smithi123.stdout:4/675: dwrite d2/d4/d7/fc4 [0,4194304] 0 2024-03-30T04:28:18.095 INFO:tasks.workunit.client.1.smithi135.stdout:0/399: write d0/d5/d10/f31 [1582855,45897] 0 2024-03-30T04:28:18.101 INFO:tasks.workunit.client.0.smithi123.stdout:2/847: write d0/d1f/d1c/d39/fd1 [492349,40245] 0 2024-03-30T04:28:18.102 INFO:tasks.workunit.client.1.smithi135.stdout:1/263: dwrite f3 [0,4194304] 0 2024-03-30T04:28:18.102 INFO:tasks.workunit.client.1.smithi135.stdout:5/330: rmdir d1/da 39 2024-03-30T04:28:18.112 INFO:tasks.workunit.client.1.smithi135.stdout:8/303: symlink d6/d8/l51 0 2024-03-30T04:28:18.113 INFO:tasks.workunit.client.1.smithi135.stdout:4/326: dwrite d10/d4d/f4e [0,4194304] 0 2024-03-30T04:28:18.115 INFO:tasks.workunit.client.0.smithi123.stdout:9/634: dwrite d6/d49/f75 [0,4194304] 0 2024-03-30T04:28:18.118 INFO:tasks.workunit.client.0.smithi123.stdout:4/676: symlink d2/d34/dd2/le8 0 2024-03-30T04:28:18.120 INFO:tasks.workunit.client.1.smithi135.stdout:2/269: mkdir d4/d55 0 2024-03-30T04:28:18.123 INFO:tasks.workunit.client.0.smithi123.stdout:2/848: mknod d0/d1f/d1c/d80/da8/d108/c115 0 2024-03-30T04:28:18.125 INFO:tasks.workunit.client.0.smithi123.stdout:7/710: dread d2/dc/d49/d68/f78 [0,4194304] 0 2024-03-30T04:28:18.125 INFO:tasks.workunit.client.0.smithi123.stdout:5/637: dwrite dd/d19/d1f/d3c/fcf [0,4194304] 0 2024-03-30T04:28:18.128 INFO:tasks.workunit.client.1.smithi135.stdout:8/304: rmdir d6/d23 39 2024-03-30T04:28:18.128 INFO:tasks.workunit.client.1.smithi135.stdout:0/400: mkdir d0/d13/d53/d7f 0 2024-03-30T04:28:18.134 INFO:tasks.workunit.client.0.smithi123.stdout:6/765: dwrite d10/d82/dc1/d6a/d7a/d99/dba/fbd [0,4194304] 0 2024-03-30T04:28:18.136 INFO:tasks.workunit.client.1.smithi135.stdout:5/331: mknod d1/da/d4d/c62 0 2024-03-30T04:28:18.136 INFO:tasks.workunit.client.1.smithi135.stdout:4/327: chown d10/d1a/d32/c3c 1 1 2024-03-30T04:28:18.137 INFO:tasks.workunit.client.1.smithi135.stdout:2/270: creat d4/d32/d2e/d2f/f56 x:0 0 0 2024-03-30T04:28:18.137 INFO:tasks.workunit.client.0.smithi123.stdout:5/638: fdatasync dd/f16 0 2024-03-30T04:28:18.138 INFO:tasks.workunit.client.0.smithi123.stdout:8/715: dwrite da/dc/d14/d18/d21/da2/fd2 [0,4194304] 0 2024-03-30T04:28:18.143 INFO:tasks.workunit.client.0.smithi123.stdout:9/635: link d6/l4b d6/d1f/da3/lda 0 2024-03-30T04:28:18.145 INFO:tasks.workunit.client.0.smithi123.stdout:6/766: rmdir d10/d82/dc1/d6a/d7a/de2 39 2024-03-30T04:28:18.145 INFO:tasks.workunit.client.0.smithi123.stdout:0/703: dread d6/dd/d15/d28/fda [0,4194304] 0 2024-03-30T04:28:18.146 INFO:tasks.workunit.client.0.smithi123.stdout:9/636: write d6/d1f/d72/f8f [2143777,7318] 0 2024-03-30T04:28:18.146 INFO:tasks.workunit.client.0.smithi123.stdout:6/767: chown d10/d82/d9a 3 1 2024-03-30T04:28:18.147 INFO:tasks.workunit.client.0.smithi123.stdout:4/677: rename d2/d4/d2c/l7d to d2/d4/d2c/d42/d35/da3/le9 0 2024-03-30T04:28:18.148 INFO:tasks.workunit.client.1.smithi135.stdout:1/264: dwrite d16/d19/f1f [0,4194304] 0 2024-03-30T04:28:18.148 INFO:tasks.workunit.client.0.smithi123.stdout:5/639: unlink dd/d19/d91/fbb 0 2024-03-30T04:28:18.151 INFO:tasks.workunit.client.1.smithi135.stdout:8/305: mknod d6/d23/c52 0 2024-03-30T04:28:18.152 INFO:tasks.workunit.client.1.smithi135.stdout:4/328: rmdir d10/d1a/d2b 39 2024-03-30T04:28:18.154 INFO:tasks.workunit.client.1.smithi135.stdout:6/306: dwrite d2/dd/d14/d2b/f47 [0,4194304] 0 2024-03-30T04:28:18.155 INFO:tasks.workunit.client.0.smithi123.stdout:0/704: truncate d6/dd/d31/d35/d3a/fdb 675262 0 2024-03-30T04:28:18.155 INFO:tasks.workunit.client.1.smithi135.stdout:4/329: chown d10/f34 22 1 2024-03-30T04:28:18.156 INFO:tasks.workunit.client.1.smithi135.stdout:4/330: fdatasync d10/f59 0 2024-03-30T04:28:18.158 INFO:tasks.workunit.client.0.smithi123.stdout:6/768: fsync d10/f1d 0 2024-03-30T04:28:18.162 INFO:tasks.workunit.client.0.smithi123.stdout:8/716: fsync da/d27/d3c/d2d/d5c/f6f 0 2024-03-30T04:28:18.162 INFO:tasks.workunit.client.0.smithi123.stdout:9/637: getdents d6/d1f/d72/db6 0 2024-03-30T04:28:18.164 INFO:tasks.workunit.client.0.smithi123.stdout:0/705: creat d6/dd/d15/d98/d4e/d9a/db2/feb x:0 0 0 2024-03-30T04:28:18.170 INFO:tasks.workunit.client.1.smithi135.stdout:1/265: fdatasync fb 0 2024-03-30T04:28:18.171 INFO:tasks.workunit.client.0.smithi123.stdout:5/640: dread - dd/d19/d3e/d5b/d62/f8b zero size 2024-03-30T04:28:18.176 INFO:tasks.workunit.client.0.smithi123.stdout:5/641: fsync dd/d19/d1f/d35/d3a/d79/f9b 0 2024-03-30T04:28:18.176 INFO:tasks.workunit.client.0.smithi123.stdout:6/769: mkdir d10/d82/d8e/dc7/d5b/d103 0 2024-03-30T04:28:18.176 INFO:tasks.workunit.client.1.smithi135.stdout:2/271: mknod d4/da/c57 0 2024-03-30T04:28:18.176 INFO:tasks.workunit.client.1.smithi135.stdout:6/307: creat d2/d2e/f5f x:0 0 0 2024-03-30T04:28:18.180 INFO:tasks.workunit.client.1.smithi135.stdout:0/401: rename d0/d5/d9/d23/d2a/f65 to d0/d13/d53/f80 0 2024-03-30T04:28:18.181 INFO:tasks.workunit.client.0.smithi123.stdout:8/717: rename da/dc/d14/d18/d21/f4d to da/d27/d3c/d64/d78/db8/fda 0 2024-03-30T04:28:18.182 INFO:tasks.workunit.client.0.smithi123.stdout:9/638: chown d6/d1f/d4d/l84 2075437400 1 2024-03-30T04:28:18.185 INFO:tasks.workunit.client.0.smithi123.stdout:8/718: readlink da/d27/d3c/l7d 0 2024-03-30T04:28:18.187 INFO:tasks.workunit.client.1.smithi135.stdout:6/308: stat d2/dd/d14/d38/f3e 0 2024-03-30T04:28:18.187 INFO:tasks.workunit.client.0.smithi123.stdout:3/778: dread d13/d24/d31/d23/f91 [0,4194304] 0 2024-03-30T04:28:18.189 INFO:tasks.workunit.client.1.smithi135.stdout:4/331: creat d10/f74 x:0 0 0 2024-03-30T04:28:18.191 INFO:tasks.workunit.client.0.smithi123.stdout:5/642: link dd/d19/d5f/c89 dd/d19/d1f/d35/db8/cd0 0 2024-03-30T04:28:18.193 INFO:tasks.workunit.client.0.smithi123.stdout:8/719: getdents da/dc/da9/dc4 0 2024-03-30T04:28:18.193 INFO:tasks.workunit.client.1.smithi135.stdout:1/266: chown d16/d29/d2c/d43/c54 639357 1 2024-03-30T04:28:18.195 INFO:tasks.workunit.client.1.smithi135.stdout:8/306: dwrite d6/d8/f17 [0,4194304] 0 2024-03-30T04:28:18.197 INFO:tasks.workunit.client.1.smithi135.stdout:8/307: chown d6/d8 2965 1 2024-03-30T04:28:18.198 INFO:tasks.workunit.client.0.smithi123.stdout:8/720: fdatasync da/f92 0 2024-03-30T04:28:18.199 INFO:tasks.workunit.client.0.smithi123.stdout:0/706: dwrite d6/dd/d31/d35/d3a/fdb [0,4194304] 0 2024-03-30T04:28:18.200 INFO:tasks.workunit.client.1.smithi135.stdout:0/402: creat d0/d19/d2d/d79/f81 x:0 0 0 2024-03-30T04:28:18.202 INFO:tasks.workunit.client.0.smithi123.stdout:3/779: symlink d13/d24/d31/d23/d3a/d3c/d4e/d5d/da5/dcd/l103 0 2024-03-30T04:28:18.203 INFO:tasks.workunit.client.0.smithi123.stdout:4/678: truncate d2/d34/f7c 1556479 0 2024-03-30T04:28:18.206 INFO:tasks.workunit.client.0.smithi123.stdout:3/780: dread - d13/d24/d31/d23/d3a/d3c/db8/fca zero size 2024-03-30T04:28:18.206 INFO:tasks.workunit.client.1.smithi135.stdout:0/403: truncate d0/d5/d9/f6f 450478 0 2024-03-30T04:28:18.207 INFO:tasks.workunit.client.1.smithi135.stdout:6/309: creat d2/dd/d14/d30/f60 x:0 0 0 2024-03-30T04:28:18.208 INFO:tasks.workunit.client.0.smithi123.stdout:5/643: mknod dd/d19/d1f/d3c/d72/cd1 0 2024-03-30T04:28:18.209 INFO:tasks.workunit.client.1.smithi135.stdout:8/308: rmdir d6/d8/d15/d2a 39 2024-03-30T04:28:18.210 INFO:tasks.workunit.client.0.smithi123.stdout:3/781: stat d13/d24/d31/d23/d3a/d3c/c74 0 2024-03-30T04:28:18.211 INFO:tasks.workunit.client.1.smithi135.stdout:4/332: symlink d10/d1a/d42/d46/l75 0 2024-03-30T04:28:18.211 INFO:tasks.workunit.client.1.smithi135.stdout:2/272: dread d4/da/f1e [0,4194304] 0 2024-03-30T04:28:18.212 INFO:tasks.workunit.client.1.smithi135.stdout:5/332: truncate d1/d3/d2b/f53 4834803 0 2024-03-30T04:28:18.212 INFO:tasks.workunit.client.1.smithi135.stdout:4/333: dread - d10/f4f zero size 2024-03-30T04:28:18.213 INFO:tasks.workunit.client.0.smithi123.stdout:6/770: dread d10/d82/dc1/d6a/d7a/fa1 [0,4194304] 0 2024-03-30T04:28:18.213 INFO:tasks.workunit.client.0.smithi123.stdout:8/721: unlink da/dc/f52 0 2024-03-30T04:28:18.213 INFO:tasks.workunit.client.0.smithi123.stdout:2/849: dread d0/d1f/d1c/f4d [0,4194304] 0 2024-03-30T04:28:18.213 INFO:tasks.workunit.client.1.smithi135.stdout:1/267: mkdir d16/d5a 0 2024-03-30T04:28:18.216 INFO:tasks.workunit.client.1.smithi135.stdout:0/404: mknod d0/d5/d42/c82 0 2024-03-30T04:28:18.217 INFO:tasks.workunit.client.0.smithi123.stdout:4/679: symlink d2/d4/d7/lea 0 2024-03-30T04:28:18.217 INFO:tasks.workunit.client.1.smithi135.stdout:1/268: read - d16/d19/f56 zero size 2024-03-30T04:28:18.220 INFO:tasks.workunit.client.1.smithi135.stdout:6/310: symlink d2/d2e/l61 0 2024-03-30T04:28:18.222 INFO:tasks.workunit.client.0.smithi123.stdout:8/722: creat da/d27/d3c/d2d/d5c/fdb x:0 0 0 2024-03-30T04:28:18.223 INFO:tasks.workunit.client.1.smithi135.stdout:2/273: truncate d4/d32/d2e/d2f/f38 186550 0 2024-03-30T04:28:18.224 INFO:tasks.workunit.client.0.smithi123.stdout:6/771: fdatasync d10/d82/dc1/d45/d77/fad 0 2024-03-30T04:28:18.224 INFO:tasks.workunit.client.0.smithi123.stdout:9/639: dwrite d6/d23/f46 [0,4194304] 0 2024-03-30T04:28:18.225 INFO:tasks.workunit.client.0.smithi123.stdout:4/680: truncate d2/d4/d7/fb 4838030 0 2024-03-30T04:28:18.228 INFO:tasks.workunit.client.0.smithi123.stdout:8/723: read da/d27/d3c/f5e [2534763,44091] 0 2024-03-30T04:28:18.228 INFO:tasks.workunit.client.1.smithi135.stdout:0/405: mkdir d0/d13/d53/d7f/d83 0 2024-03-30T04:28:18.232 INFO:tasks.workunit.client.1.smithi135.stdout:2/274: write d4/da/f44 [101470,42312] 0 2024-03-30T04:28:18.233 INFO:tasks.workunit.client.0.smithi123.stdout:8/724: dread - da/d27/d56/fc8 zero size 2024-03-30T04:28:18.236 INFO:tasks.workunit.client.1.smithi135.stdout:8/309: truncate d6/d8/f17 3431410 0 2024-03-30T04:28:18.237 INFO:tasks.workunit.client.0.smithi123.stdout:4/681: dread - d2/d4/d2c/d42/d35/d5d/dc0/fce zero size 2024-03-30T04:28:18.237 INFO:tasks.workunit.client.0.smithi123.stdout:9/640: rename d6/d1f/c7b to d6/d1f/d92/cdb 0 2024-03-30T04:28:18.242 INFO:tasks.workunit.client.1.smithi135.stdout:0/406: write d0/d13/f74 [3689262,72075] 0 2024-03-30T04:28:18.244 INFO:tasks.workunit.client.0.smithi123.stdout:8/725: creat da/d27/d56/d6c/d96/fdc x:0 0 0 2024-03-30T04:28:18.244 INFO:tasks.workunit.client.0.smithi123.stdout:4/682: stat d2/d4/d2c/d42/d35/d9b/db7/cc8 0 2024-03-30T04:28:18.247 INFO:tasks.workunit.client.1.smithi135.stdout:2/275: write d4/d6/f1f [5728042,88298] 0 2024-03-30T04:28:18.247 INFO:tasks.workunit.client.0.smithi123.stdout:5/644: dwrite dd/d19/d3e/d5b/d62/f84 [4194304,4194304] 0 2024-03-30T04:28:18.247 INFO:tasks.workunit.client.0.smithi123.stdout:4/683: write d2/d34/fb4 [384358,52279] 0 2024-03-30T04:28:18.251 INFO:tasks.workunit.client.0.smithi123.stdout:3/782: truncate d13/d24/d31/d23/d63/f87 1238121 0 2024-03-30T04:28:18.252 INFO:tasks.workunit.client.1.smithi135.stdout:8/310: creat d6/d8/d15/d1f/f53 x:0 0 0 2024-03-30T04:28:18.256 INFO:tasks.workunit.client.1.smithi135.stdout:0/407: chown d0/l6a 116404 1 2024-03-30T04:28:18.257 INFO:tasks.workunit.client.0.smithi123.stdout:9/641: rename d6/d58/l61 to d6/d1f/d45/d97/d69/daa/ldc 0 2024-03-30T04:28:18.260 INFO:tasks.workunit.client.0.smithi123.stdout:9/642: read - d6/d7c/d82/fc7 zero size 2024-03-30T04:28:18.261 INFO:tasks.workunit.client.1.smithi135.stdout:8/311: stat d6/d8/c26 0 2024-03-30T04:28:18.262 INFO:tasks.workunit.client.0.smithi123.stdout:6/772: truncate d10/d82/dc1/d45/faa 1181119 0 2024-03-30T04:28:18.262 INFO:tasks.workunit.client.0.smithi123.stdout:0/707: dread d6/dd/d15/d98/d4e/d69/d6e/f9f [0,4194304] 0 2024-03-30T04:28:18.263 INFO:tasks.workunit.client.0.smithi123.stdout:4/684: mkdir d2/d4/d2c/d42/d35/d9b/da2/deb 0 2024-03-30T04:28:18.265 INFO:tasks.workunit.client.0.smithi123.stdout:8/726: mknod da/d27/d3c/d64/d78/d83/cdd 0 2024-03-30T04:28:18.266 INFO:tasks.workunit.client.0.smithi123.stdout:2/850: dwrite d0/d37/f8a [0,4194304] 0 2024-03-30T04:28:18.267 INFO:tasks.workunit.client.1.smithi135.stdout:4/334: dwrite d10/f74 [0,4194304] 0 2024-03-30T04:28:18.267 INFO:tasks.workunit.client.1.smithi135.stdout:2/276: read d4/d32/d2e/f4a [3626776,1236] 0 2024-03-30T04:28:18.267 INFO:tasks.workunit.client.1.smithi135.stdout:1/269: dwrite d16/d1a/f21 [0,4194304] 0 2024-03-30T04:28:18.267 INFO:tasks.workunit.client.0.smithi123.stdout:5/645: symlink dd/ld2 0 2024-03-30T04:28:18.268 INFO:tasks.workunit.client.0.smithi123.stdout:9/643: chown d6/d1f/d92/db8/fbc 14 1 2024-03-30T04:28:18.269 INFO:tasks.workunit.client.0.smithi123.stdout:0/708: rmdir d6/dd/d15/d98/d4e/d69 39 2024-03-30T04:28:18.271 INFO:tasks.workunit.client.0.smithi123.stdout:3/783: write d13/d24/d31/d23/d3a/d3c/d4e/d5d/f8e [221332,84400] 0 2024-03-30T04:28:18.271 INFO:tasks.workunit.client.0.smithi123.stdout:2/851: stat d0/d1f/f24 0 2024-03-30T04:28:18.272 INFO:tasks.workunit.client.1.smithi135.stdout:2/277: rmdir d4/d6/d47 39 2024-03-30T04:28:18.274 INFO:tasks.workunit.client.0.smithi123.stdout:9/644: write d6/d1f/f34 [3095538,46944] 0 2024-03-30T04:28:18.275 INFO:tasks.workunit.client.0.smithi123.stdout:4/685: write d2/d4/d2c/d42/d35/faa [709070,49888] 0 2024-03-30T04:28:18.277 INFO:tasks.workunit.client.0.smithi123.stdout:5/646: mkdir dd/d19/d3e/d5b/d62/dd3 0 2024-03-30T04:28:18.282 INFO:tasks.workunit.client.1.smithi135.stdout:0/408: write d0/d5/d9/fb [148420,123431] 0 2024-03-30T04:28:18.283 INFO:tasks.workunit.client.1.smithi135.stdout:1/270: rename d16/f18 to d16/d29/d2c/d43/d50/f5b 0 2024-03-30T04:28:18.285 INFO:tasks.workunit.client.1.smithi135.stdout:5/333: dread d1/d7/f43 [0,4194304] 0 2024-03-30T04:28:18.287 INFO:tasks.workunit.client.1.smithi135.stdout:6/311: dread d2/dd/d18/f22 [0,4194304] 0 2024-03-30T04:28:18.290 INFO:tasks.workunit.client.0.smithi123.stdout:3/784: fdatasync d13/d24/f57 0 2024-03-30T04:28:18.290 INFO:tasks.workunit.client.0.smithi123.stdout:0/709: creat d6/dd/d15/d28/dd3/ddd/fec x:0 0 0 2024-03-30T04:28:18.291 INFO:tasks.workunit.client.0.smithi123.stdout:4/686: fsync d2/d4/d2c/d42/d35/d5d/dc0/fce 0 2024-03-30T04:28:18.292 INFO:tasks.workunit.client.0.smithi123.stdout:2/852: mkdir d0/d1f/d1c/d80/dbb/d97/d116 0 2024-03-30T04:28:18.295 INFO:tasks.workunit.client.1.smithi135.stdout:1/271: symlink d16/d29/d2c/l5c 0 2024-03-30T04:28:18.297 INFO:tasks.workunit.client.0.smithi123.stdout:9/645: creat d6/d7c/fdd x:0 0 0 2024-03-30T04:28:18.297 INFO:tasks.workunit.client.0.smithi123.stdout:4/687: write d2/d4/d7/d1a/f30 [13008127,11906] 0 2024-03-30T04:28:18.298 INFO:tasks.workunit.client.1.smithi135.stdout:8/312: rename d6/d8/d15/d2a/d47 to d6/d8/d15/d54 0 2024-03-30T04:28:18.299 INFO:tasks.workunit.client.1.smithi135.stdout:4/335: dread fd [0,4194304] 0 2024-03-30T04:28:18.304 INFO:tasks.workunit.client.1.smithi135.stdout:0/409: rename d0/d5/d42/c82 to d0/d5/d9/d23/d58/d3b/d52/c84 0 2024-03-30T04:28:18.306 INFO:tasks.workunit.client.1.smithi135.stdout:9/285: sync 2024-03-30T04:28:18.307 INFO:tasks.workunit.client.0.smithi123.stdout:8/727: dwrite da/d27/d3c/d64/d78/db8/fda [0,4194304] 0 2024-03-30T04:28:18.310 INFO:tasks.workunit.client.1.smithi135.stdout:8/313: getdents d6/d8/d14/d32 0 2024-03-30T04:28:18.312 INFO:tasks.workunit.client.1.smithi135.stdout:9/286: getdents d7/d4f 0 2024-03-30T04:28:18.314 INFO:tasks.workunit.client.1.smithi135.stdout:9/287: fsync d7/d1b/d37/f54 0 2024-03-30T04:28:18.315 INFO:tasks.workunit.client.1.smithi135.stdout:8/314: mknod d6/d8/d15/d54/d4f/c55 0 2024-03-30T04:28:18.315 INFO:tasks.workunit.client.1.smithi135.stdout:5/334: dwrite d1/d7/ff [8388608,4194304] 0 2024-03-30T04:28:18.318 INFO:tasks.workunit.client.1.smithi135.stdout:8/315: fdatasync d6/d23/f27 0 2024-03-30T04:28:18.322 INFO:tasks.workunit.client.1.smithi135.stdout:5/335: rmdir d1/d7/d1a/d1f 39 2024-03-30T04:28:18.323 INFO:tasks.workunit.client.0.smithi123.stdout:5/647: chown dd/d19/f2b 743307377 1 2024-03-30T04:28:18.324 INFO:tasks.workunit.client.0.smithi123.stdout:0/710: unlink d6/c13 0 2024-03-30T04:28:18.324 INFO:tasks.workunit.client.1.smithi135.stdout:9/288: mkdir d7/d4f/d5c/d63 0 2024-03-30T04:28:18.326 INFO:tasks.workunit.client.0.smithi123.stdout:4/688: creat d2/d4/d2c/d42/d89/fec x:0 0 0 2024-03-30T04:28:18.326 INFO:tasks.workunit.client.1.smithi135.stdout:0/410: dwrite d0/d5/d9/d23/f66 [0,4194304] 0 2024-03-30T04:28:18.327 INFO:tasks.workunit.client.0.smithi123.stdout:4/689: chown d2/d26 867 1 2024-03-30T04:28:18.331 INFO:tasks.workunit.client.0.smithi123.stdout:3/785: dread d13/d24/d31/d23/fc8 [0,4194304] 0 2024-03-30T04:28:18.332 INFO:tasks.workunit.client.1.smithi135.stdout:2/278: dread d4/da/fb [0,4194304] 0 2024-03-30T04:28:18.334 INFO:tasks.workunit.client.0.smithi123.stdout:9/646: creat d6/d1f/d72/db6/fde x:0 0 0 2024-03-30T04:28:18.336 INFO:tasks.workunit.client.0.smithi123.stdout:8/728: mkdir da/d27/d56/dde 0 2024-03-30T04:28:18.337 INFO:tasks.workunit.client.0.smithi123.stdout:0/711: stat d6/dd/d18/d2f/c48 0 2024-03-30T04:28:18.337 INFO:tasks.workunit.client.0.smithi123.stdout:3/786: unlink d13/f1d 0 2024-03-30T04:28:18.339 INFO:tasks.workunit.client.0.smithi123.stdout:5/648: symlink dd/d19/d1f/d35/ld4 0 2024-03-30T04:28:18.340 INFO:tasks.workunit.client.0.smithi123.stdout:8/729: unlink da/dc/d14/db7/lc9 0 2024-03-30T04:28:18.340 INFO:tasks.workunit.client.0.smithi123.stdout:0/712: write d6/dd/d15/d28/dd3/fd6 [1001787,46063] 0 2024-03-30T04:28:18.343 INFO:tasks.workunit.client.0.smithi123.stdout:9/647: rename d6/d1f/da3/la8 to d6/d1f/d4d/ldf 0 2024-03-30T04:28:18.345 INFO:tasks.workunit.client.0.smithi123.stdout:0/713: unlink d6/dd/d15/le9 0 2024-03-30T04:28:18.345 INFO:tasks.workunit.client.0.smithi123.stdout:9/648: write d6/dc9/fce [60756,115729] 0 2024-03-30T04:28:18.346 INFO:tasks.workunit.client.0.smithi123.stdout:5/649: truncate dd/d19/d3e/d5b/d62/f7a 801909 0 2024-03-30T04:28:18.346 INFO:tasks.workunit.client.0.smithi123.stdout:8/730: symlink da/d27/d3c/d2d/d74/ldf 0 2024-03-30T04:28:18.348 INFO:tasks.workunit.client.0.smithi123.stdout:9/649: readlink d6/l4b 0 2024-03-30T04:28:18.349 INFO:tasks.workunit.client.0.smithi123.stdout:0/714: fsync d6/dd/d15/fb3 0 2024-03-30T04:28:18.351 INFO:tasks.workunit.client.0.smithi123.stdout:9/650: write d6/d49/f65 [691081,87069] 0 2024-03-30T04:28:18.351 INFO:tasks.workunit.client.0.smithi123.stdout:5/650: symlink dd/d19/d1f/d68/dcc/ld5 0 2024-03-30T04:28:18.353 INFO:tasks.workunit.client.0.smithi123.stdout:0/715: chown d6/d52/f73 26408 1 2024-03-30T04:28:18.356 INFO:tasks.workunit.client.0.smithi123.stdout:9/651: mknod d6/ce0 0 2024-03-30T04:28:18.358 INFO:tasks.workunit.client.0.smithi123.stdout:5/651: creat dd/d19/d1f/d35/d3a/d57/db5/fd6 x:0 0 0 2024-03-30T04:28:18.359 INFO:tasks.workunit.client.0.smithi123.stdout:1/782: sync 2024-03-30T04:28:18.359 INFO:tasks.workunit.client.0.smithi123.stdout:0/716: truncate d6/dd/d15/d98/d4e/d9a/db2/fc9 927590 0 2024-03-30T04:28:18.359 INFO:tasks.workunit.client.0.smithi123.stdout:9/652: creat d6/d1f/d92/fe1 x:0 0 0 2024-03-30T04:28:18.361 INFO:tasks.workunit.client.0.smithi123.stdout:5/652: rmdir dd/d19/d1f/d3c 39 2024-03-30T04:28:18.367 INFO:tasks.workunit.client.1.smithi135.stdout:2/279: dread d4/f5 [4194304,4194304] 0 2024-03-30T04:28:18.367 INFO:tasks.workunit.client.0.smithi123.stdout:9/653: creat d6/d1f/d72/d7d/fe2 x:0 0 0 2024-03-30T04:28:18.370 INFO:tasks.workunit.client.0.smithi123.stdout:1/783: symlink db/dd/d42/d46/d51/dd7/l106 0 2024-03-30T04:28:18.371 INFO:tasks.workunit.client.0.smithi123.stdout:9/654: creat d6/d1f/d92/fe3 x:0 0 0 2024-03-30T04:28:18.374 INFO:tasks.workunit.client.0.smithi123.stdout:1/784: mknod db/dd/d42/c107 0 2024-03-30T04:28:18.376 INFO:tasks.workunit.client.0.smithi123.stdout:9/655: creat d6/d1f/d72/db6/fe4 x:0 0 0 2024-03-30T04:28:18.378 INFO:tasks.workunit.client.1.smithi135.stdout:2/280: dwrite d4/d32/f13 [0,4194304] 0 2024-03-30T04:28:18.379 INFO:tasks.workunit.client.1.smithi135.stdout:2/281: chown d4/d32/d46 11 1 2024-03-30T04:28:18.379 INFO:tasks.workunit.client.0.smithi123.stdout:2/853: truncate d0/fb4 2416254 0 2024-03-30T04:28:18.380 INFO:tasks.workunit.client.0.smithi123.stdout:9/656: fsync d6/d58/f5a 0 2024-03-30T04:28:18.381 INFO:tasks.workunit.client.0.smithi123.stdout:2/854: stat d0/d1f/d1c/d39/d4b 0 2024-03-30T04:28:18.383 INFO:tasks.workunit.client.0.smithi123.stdout:8/731: dread da/dc/d14/d18/f4f [4194304,4194304] 0 2024-03-30T04:28:18.384 INFO:tasks.workunit.client.0.smithi123.stdout:9/657: chown d6/d58/f5c 39743 1 2024-03-30T04:28:18.390 INFO:tasks.workunit.client.0.smithi123.stdout:3/787: dread d13/d24/d31/d23/d63/f87 [0,4194304] 0 2024-03-30T04:28:18.391 INFO:tasks.workunit.client.0.smithi123.stdout:2/855: creat d0/d1f/d30/f117 x:0 0 0 2024-03-30T04:28:18.393 INFO:tasks.workunit.client.0.smithi123.stdout:2/856: dread - d0/d1f/d30/f117 zero size 2024-03-30T04:28:18.397 INFO:tasks.workunit.client.0.smithi123.stdout:8/732: mknod da/dc/d14/ce0 0 2024-03-30T04:28:18.400 INFO:tasks.workunit.client.0.smithi123.stdout:0/717: write d6/dd/d18/fb5 [515765,81744] 0 2024-03-30T04:28:18.402 INFO:tasks.workunit.client.0.smithi123.stdout:0/718: stat d6/f1b 0 2024-03-30T04:28:18.403 INFO:tasks.workunit.client.0.smithi123.stdout:9/658: getdents d6/d7c 0 2024-03-30T04:28:18.405 INFO:tasks.workunit.client.0.smithi123.stdout:8/733: truncate da/d27/d3c/d2d/f4c 516608 0 2024-03-30T04:28:18.405 INFO:tasks.workunit.client.0.smithi123.stdout:9/659: chown d6/d1f/c25 1487 1 2024-03-30T04:28:18.410 INFO:tasks.workunit.client.0.smithi123.stdout:1/785: write db/dd/d42/d46/d39/f57 [2491460,80002] 0 2024-03-30T04:28:18.413 INFO:tasks.workunit.client.0.smithi123.stdout:9/660: rename d6/d1f/c25 to d6/d1f/d92/ce5 0 2024-03-30T04:28:18.413 INFO:tasks.workunit.client.0.smithi123.stdout:5/653: dwrite dd/d19/f22 [0,4194304] 0 2024-03-30T04:28:18.415 INFO:tasks.workunit.client.0.smithi123.stdout:0/719: creat d6/dd/d31/d35/fed x:0 0 0 2024-03-30T04:28:18.419 INFO:tasks.workunit.client.0.smithi123.stdout:2/857: truncate d0/fb4 2025214 0 2024-03-30T04:28:18.423 INFO:tasks.workunit.client.0.smithi123.stdout:8/734: rename da/d27/d3c/c70 to da/dc/d14/d9c/ce1 0 2024-03-30T04:28:18.425 INFO:tasks.workunit.client.0.smithi123.stdout:5/654: chown dd/d19/d1f/d3c/f97 13 1 2024-03-30T04:28:18.427 INFO:tasks.workunit.client.0.smithi123.stdout:0/720: unlink d6/dd/d15/lbf 0 2024-03-30T04:28:18.430 INFO:tasks.workunit.client.0.smithi123.stdout:3/788: mkdir d13/d24/d31/d23/d3a/d3c/d4e/d5d/da5/d104 0 2024-03-30T04:28:18.434 INFO:tasks.workunit.client.0.smithi123.stdout:2/858: dread d0/d1f/d30/f72 [0,4194304] 0 2024-03-30T04:28:18.435 INFO:tasks.workunit.client.0.smithi123.stdout:8/735: link da/d27/d3c/d2d/f2e da/d27/d3c/d2d/dae/dd9/fe2 0 2024-03-30T04:28:18.437 INFO:tasks.workunit.client.0.smithi123.stdout:0/721: mknod d6/dd/d15/d28/d4b/d8a/cee 0 2024-03-30T04:28:18.439 INFO:tasks.workunit.client.0.smithi123.stdout:3/789: creat d13/d24/d31/de0/f105 x:0 0 0 2024-03-30T04:28:18.440 INFO:tasks.workunit.client.0.smithi123.stdout:1/786: dwrite db/dd/d42/d46/d5d/d62/d9e/fff [0,4194304] 0 2024-03-30T04:28:18.443 INFO:tasks.workunit.client.0.smithi123.stdout:9/661: write d6/d1f/d4d/f5b [1006741,112503] 0 2024-03-30T04:28:18.446 INFO:tasks.workunit.client.0.smithi123.stdout:5/655: dread dd/d19/d1f/d35/faf [0,4194304] 0 2024-03-30T04:28:18.446 INFO:tasks.workunit.client.0.smithi123.stdout:9/662: fdatasync d6/d1f/d45/d97/d69/daa/fd0 0 2024-03-30T04:28:18.448 INFO:tasks.workunit.client.0.smithi123.stdout:8/736: symlink da/d27/d3c/le3 0 2024-03-30T04:28:18.451 INFO:tasks.workunit.client.1.smithi135.stdout:9/289: rmdir d7/d1b 39 2024-03-30T04:28:18.454 INFO:tasks.workunit.client.1.smithi135.stdout:5/336: unlink d1/d3/c27 0 2024-03-30T04:28:18.454 INFO:tasks.workunit.client.0.smithi123.stdout:1/787: mkdir db/dd/d42/d46/d51/d72/d108 0 2024-03-30T04:28:18.457 INFO:tasks.workunit.client.0.smithi123.stdout:9/663: rmdir d6/d7c 39 2024-03-30T04:28:18.457 INFO:tasks.workunit.client.1.smithi135.stdout:8/316: rename d6/d23/c52 to d6/d8/d14/c56 0 2024-03-30T04:28:18.459 INFO:tasks.workunit.client.0.smithi123.stdout:9/664: chown d6/l3c 163649 1 2024-03-30T04:28:18.460 INFO:tasks.workunit.client.1.smithi135.stdout:5/337: symlink d1/d3/d2b/d54/l63 0 2024-03-30T04:28:18.461 INFO:tasks.workunit.client.0.smithi123.stdout:9/665: readlink d6/d23/l55 0 2024-03-30T04:28:18.461 INFO:tasks.workunit.client.0.smithi123.stdout:1/788: dread db/dd/d42/d46/d51/f55 [4194304,4194304] 0 2024-03-30T04:28:18.462 INFO:tasks.workunit.client.1.smithi135.stdout:8/317: write d6/d8/d15/d54/f4a [508762,102241] 0 2024-03-30T04:28:18.462 INFO:tasks.workunit.client.1.smithi135.stdout:2/282: rename d4/d32/d2e/d2f/f56 to d4/d32/d50/f58 0 2024-03-30T04:28:18.462 INFO:tasks.workunit.client.0.smithi123.stdout:5/656: mkdir dd/d19/d3e/d5b/dd7 0 2024-03-30T04:28:18.464 INFO:tasks.workunit.client.1.smithi135.stdout:9/290: dread d7/d29/f3e [0,4194304] 0 2024-03-30T04:28:18.465 INFO:tasks.workunit.client.0.smithi123.stdout:9/666: chown d6/d23 18699894 1 2024-03-30T04:28:18.465 INFO:tasks.workunit.client.0.smithi123.stdout:3/790: dread d13/d24/d39/d79/d90/fd4 [0,4194304] 0 2024-03-30T04:28:18.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:18 smithi123 ceph-mon[35245]: pgmap v42: 65 pgs: 65 active+clean; 1.8 GiB data, 6.8 GiB used, 530 GiB / 536 GiB avail; 58 MiB/s rd, 194 MiB/s wr, 387 op/s 2024-03-30T04:28:18.467 INFO:tasks.workunit.client.1.smithi135.stdout:5/338: symlink d1/d7/d1a/d1f/l64 0 2024-03-30T04:28:18.468 INFO:tasks.workunit.client.0.smithi123.stdout:9/667: dread - d6/fc5 zero size 2024-03-30T04:28:18.471 INFO:tasks.workunit.client.0.smithi123.stdout:1/789: write db/dd/d5f/dae/ffd [48093,102073] 0 2024-03-30T04:28:18.472 INFO:tasks.workunit.client.0.smithi123.stdout:9/668: stat d6/dad/fd7 0 2024-03-30T04:28:18.472 INFO:tasks.workunit.client.0.smithi123.stdout:8/737: dwrite da/dc/d14/d18/d62/f9f [4194304,4194304] 0 2024-03-30T04:28:18.474 INFO:tasks.workunit.client.0.smithi123.stdout:0/722: dwrite d6/dd/d15/fc3 [0,4194304] 0 2024-03-30T04:28:18.475 INFO:tasks.workunit.client.1.smithi135.stdout:9/291: dread d7/d41/f43 [0,4194304] 0 2024-03-30T04:28:18.476 INFO:tasks.workunit.client.0.smithi123.stdout:9/669: write d6/d1f/d72/fc3 [71564,127260] 0 2024-03-30T04:28:18.477 INFO:tasks.workunit.client.0.smithi123.stdout:9/670: dread - d6/d1f/d92/fcc zero size 2024-03-30T04:28:18.482 INFO:tasks.workunit.client.1.smithi135.stdout:0/411: dwrite d0/d1/f29 [0,4194304] 0 2024-03-30T04:28:18.483 INFO:tasks.workunit.client.0.smithi123.stdout:8/738: write da/f3d [1438592,47628] 0 2024-03-30T04:28:18.483 INFO:tasks.workunit.client.0.smithi123.stdout:1/790: read db/dd/d42/d46/d56/daf/fb5 [697672,130746] 0 2024-03-30T04:28:18.483 INFO:tasks.workunit.client.0.smithi123.stdout:3/791: unlink d13/d24/d31/d23/d3a/d3c/d4e/d5d/cbb 0 2024-03-30T04:28:18.483 INFO:tasks.workunit.client.0.smithi123.stdout:9/671: symlink d6/d49/le6 0 2024-03-30T04:28:18.485 INFO:tasks.workunit.client.0.smithi123.stdout:0/723: mkdir d6/dd/def 0 2024-03-30T04:28:18.485 INFO:tasks.workunit.client.0.smithi123.stdout:1/791: chown db/dd/d42/d46/d56 5187 1 2024-03-30T04:28:18.488 INFO:tasks.workunit.client.0.smithi123.stdout:1/792: fsync db/dd/d42/d46/d56/da2/fb0 0 2024-03-30T04:28:18.489 INFO:tasks.workunit.client.1.smithi135.stdout:8/318: write d6/d8/f1d [5008191,6927] 0 2024-03-30T04:28:18.491 INFO:tasks.workunit.client.0.smithi123.stdout:9/672: mkdir d6/d7c/de7 0 2024-03-30T04:28:18.494 INFO:tasks.workunit.client.0.smithi123.stdout:3/792: rename d13/d24/d39/d79/c93 to d13/d24/d31/d23/d3a/d3c/d4e/d5d/da5/c106 0 2024-03-30T04:28:18.495 INFO:tasks.workunit.client.0.smithi123.stdout:9/673: creat d6/d1f/d92/db8/fe8 x:0 0 0 2024-03-30T04:28:18.497 INFO:tasks.workunit.client.0.smithi123.stdout:3/793: fsync d13/d24/d31/d23/d3a/d3c/d66/f7f 0 2024-03-30T04:28:18.497 INFO:tasks.workunit.client.1.smithi135.stdout:9/292: rename d7/d1b/f40 to d7/d4f/d5c/d63/f64 0 2024-03-30T04:28:18.497 INFO:tasks.workunit.client.1.smithi135.stdout:0/412: creat d0/d1/d16/f85 x:0 0 0 2024-03-30T04:28:18.499 INFO:tasks.workunit.client.0.smithi123.stdout:5/657: dread dd/d19/d1f/d35/d3a/f45 [0,4194304] 0 2024-03-30T04:28:18.500 INFO:tasks.workunit.client.1.smithi135.stdout:2/283: dwrite d4/d32/d12/f3f [0,4194304] 0 2024-03-30T04:28:18.501 INFO:tasks.workunit.client.0.smithi123.stdout:9/674: creat d6/d1f/d72/fe9 x:0 0 0 2024-03-30T04:28:18.502 INFO:tasks.workunit.client.0.smithi123.stdout:3/794: mkdir d13/d24/d39/d79/dea/d107 0 2024-03-30T04:28:18.504 INFO:tasks.workunit.client.1.smithi135.stdout:5/339: dwrite d1/d7/f10 [4194304,4194304] 0 2024-03-30T04:28:18.505 INFO:tasks.workunit.client.0.smithi123.stdout:5/658: truncate dd/f1c 4544148 0 2024-03-30T04:28:18.508 INFO:tasks.workunit.client.1.smithi135.stdout:9/293: rmdir d7/d55 39 2024-03-30T04:28:18.511 INFO:tasks.workunit.client.0.smithi123.stdout:5/659: truncate dd/d19/d1f/d35/d3a/fa6 864618 0 2024-03-30T04:28:18.512 INFO:tasks.workunit.client.0.smithi123.stdout:2/859: dread d0/d1f/d1c/d39/d4b/d4c/d5f/f67 [0,4194304] 0 2024-03-30T04:28:18.514 INFO:tasks.workunit.client.0.smithi123.stdout:5/660: truncate dd/d19/d5f/d99/fc7 741441 0 2024-03-30T04:28:18.515 INFO:tasks.workunit.client.1.smithi135.stdout:0/413: mkdir d0/d5/d9/d23/d58/d63/d77/d86 0 2024-03-30T04:28:18.517 INFO:tasks.workunit.client.0.smithi123.stdout:5/661: chown dd/d19/d1f/d3c/fab 1725 1 2024-03-30T04:28:18.517 INFO:tasks.workunit.client.0.smithi123.stdout:9/675: rename d6/d7c/d82/fc7 to d6/d1f/d45/d97/d69/fea 0 2024-03-30T04:28:18.519 INFO:tasks.workunit.client.0.smithi123.stdout:2/860: creat d0/f118 x:0 0 0 2024-03-30T04:28:18.520 INFO:tasks.workunit.client.0.smithi123.stdout:9/676: read - d6/d1f/d72/fcf zero size 2024-03-30T04:28:18.520 INFO:tasks.workunit.client.1.smithi135.stdout:8/319: rename d6/d8/d15/d2a/c41 to d6/d8/d14/c57 0 2024-03-30T04:28:18.523 INFO:tasks.workunit.client.0.smithi123.stdout:9/677: fsync d6/d23/f8d 0 2024-03-30T04:28:18.524 INFO:tasks.workunit.client.1.smithi135.stdout:2/284: link d4/da/fb d4/da/f59 0 2024-03-30T04:28:18.526 INFO:tasks.workunit.client.1.smithi135.stdout:0/414: getdents d0/d5/d9/d23/d58/d3b 0 2024-03-30T04:28:18.527 INFO:tasks.workunit.client.0.smithi123.stdout:9/678: symlink d6/d7c/d82/leb 0 2024-03-30T04:28:18.527 INFO:tasks.workunit.client.1.smithi135.stdout:3/314: sync 2024-03-30T04:28:18.528 INFO:tasks.workunit.client.1.smithi135.stdout:7/346: sync 2024-03-30T04:28:18.531 INFO:tasks.workunit.client.1.smithi135.stdout:2/285: creat d4/d6/d47/f5a x:0 0 0 2024-03-30T04:28:18.534 INFO:tasks.workunit.client.0.smithi123.stdout:2/861: dwrite d0/d1f/f8f [0,4194304] 0 2024-03-30T04:28:18.535 INFO:tasks.workunit.client.1.smithi135.stdout:0/415: mkdir d0/d5/d42/d87 0 2024-03-30T04:28:18.536 INFO:tasks.workunit.client.1.smithi135.stdout:2/286: write d4/da/f44 [1566893,52994] 0 2024-03-30T04:28:18.536 INFO:tasks.workunit.client.1.smithi135.stdout:3/315: mkdir d3/d10/d16/d6b 0 2024-03-30T04:28:18.537 INFO:tasks.workunit.client.0.smithi123.stdout:2/862: stat d0/d1f/d16/ffa 0 2024-03-30T04:28:18.538 INFO:tasks.workunit.client.1.smithi135.stdout:9/294: dwrite d7/d1b/f34 [0,4194304] 0 2024-03-30T04:28:18.540 INFO:tasks.workunit.client.0.smithi123.stdout:3/795: write d13/d24/fbd [131386,54072] 0 2024-03-30T04:28:18.546 INFO:tasks.workunit.client.1.smithi135.stdout:3/316: unlink d3/l20 0 2024-03-30T04:28:18.546 INFO:tasks.workunit.client.0.smithi123.stdout:2/863: rename d0/d1f/d1c/d80 to d0/d1f/d30/d119 0 2024-03-30T04:28:18.550 INFO:tasks.workunit.client.1.smithi135.stdout:8/320: dwrite d6/f50 [0,4194304] 0 2024-03-30T04:28:18.551 INFO:tasks.workunit.client.0.smithi123.stdout:2/864: mkdir d0/d37/d11a 0 2024-03-30T04:28:18.554 INFO:tasks.workunit.client.1.smithi135.stdout:7/347: symlink d4/db/d1a/d5c/d2c/l71 0 2024-03-30T04:28:18.556 INFO:tasks.workunit.client.1.smithi135.stdout:5/340: write d1/d3/d2b/f5b [114017,34911] 0 2024-03-30T04:28:18.557 INFO:tasks.workunit.client.1.smithi135.stdout:9/295: symlink d7/d4f/d5c/d63/l65 0 2024-03-30T04:28:18.559 INFO:tasks.workunit.client.0.smithi123.stdout:8/739: dread da/dc/da9/fad [0,4194304] 0 2024-03-30T04:28:18.559 INFO:tasks.workunit.client.0.smithi123.stdout:2/865: write d0/d1f/d16/ffa [161112,47899] 0 2024-03-30T04:28:18.559 INFO:tasks.workunit.client.1.smithi135.stdout:8/321: chown d6/d8/d15/d2a/f33 3 1 2024-03-30T04:28:18.560 INFO:tasks.workunit.client.0.smithi123.stdout:5/662: dwrite dd/d19/d3e/d5b/f8a [0,4194304] 0 2024-03-30T04:28:18.562 INFO:tasks.workunit.client.0.smithi123.stdout:8/740: mknod da/d27/d56/ce4 0 2024-03-30T04:28:18.563 INFO:tasks.workunit.client.0.smithi123.stdout:5/663: stat dd/d19/d1f/d3c/f92 0 2024-03-30T04:28:18.564 INFO:tasks.workunit.client.0.smithi123.stdout:2/866: getdents d0/d1f/d1c/d36/d106 0 2024-03-30T04:28:18.566 INFO:tasks.workunit.client.1.smithi135.stdout:9/296: dread - d7/f44 zero size 2024-03-30T04:28:18.567 INFO:tasks.workunit.client.1.smithi135.stdout:3/317: link d3/db/c47 d3/d10/c6c 0 2024-03-30T04:28:18.568 INFO:tasks.workunit.client.1.smithi135.stdout:8/322: creat d6/d8/d15/d54/f58 x:0 0 0 2024-03-30T04:28:18.571 INFO:tasks.workunit.client.0.smithi123.stdout:3/796: dwrite d13/d24/d31/d23/f2b [0,4194304] 0 2024-03-30T04:28:18.575 INFO:tasks.workunit.client.1.smithi135.stdout:9/297: write d7/f4e [1014115,42572] 0 2024-03-30T04:28:18.575 INFO:tasks.workunit.client.1.smithi135.stdout:0/416: dread d0/f3 [0,4194304] 0 2024-03-30T04:28:18.576 INFO:tasks.workunit.client.0.smithi123.stdout:9/679: dread d6/d1f/f2e [0,4194304] 0 2024-03-30T04:28:18.576 INFO:tasks.workunit.client.0.smithi123.stdout:8/741: symlink da/d27/d56/d6c/d96/dc2/le5 0 2024-03-30T04:28:18.577 INFO:tasks.workunit.client.1.smithi135.stdout:8/323: creat d6/d8/d15/d54/f59 x:0 0 0 2024-03-30T04:28:18.578 INFO:tasks.workunit.client.0.smithi123.stdout:3/797: mknod d13/d24/d39/d79/d90/c108 0 2024-03-30T04:28:18.578 INFO:tasks.workunit.client.1.smithi135.stdout:3/318: read d3/f24 [2562274,96761] 0 2024-03-30T04:28:18.579 INFO:tasks.workunit.client.0.smithi123.stdout:5/664: getdents dd/d19/d3e 0 2024-03-30T04:28:18.581 INFO:tasks.workunit.client.0.smithi123.stdout:9/680: symlink d6/d7c/lec 0 2024-03-30T04:28:18.582 INFO:tasks.workunit.client.1.smithi135.stdout:9/298: chown d7/d55/f5d 132366247 1 2024-03-30T04:28:18.583 INFO:tasks.workunit.client.1.smithi135.stdout:7/348: dwrite d4/db/d1a/d5c/f43 [0,4194304] 0 2024-03-30T04:28:18.583 INFO:tasks.workunit.client.1.smithi135.stdout:3/319: fdatasync d3/d50/d46/f58 0 2024-03-30T04:28:18.583 INFO:tasks.workunit.client.1.smithi135.stdout:0/417: mknod d0/d5/d42/c88 0 2024-03-30T04:28:18.584 INFO:tasks.workunit.client.1.smithi135.stdout:8/324: rmdir d6/d8/d15/d54/d4f 39 2024-03-30T04:28:18.590 INFO:tasks.workunit.client.1.smithi135.stdout:0/418: read - d0/d13/d1e/f78 zero size 2024-03-30T04:28:18.593 INFO:tasks.workunit.client.0.smithi123.stdout:9/681: truncate d6/d1f/d72/d7d/f9c 635949 0 2024-03-30T04:28:18.593 INFO:tasks.workunit.client.1.smithi135.stdout:7/349: symlink d4/db/d1a/d4b/l72 0 2024-03-30T04:28:18.594 INFO:tasks.workunit.client.1.smithi135.stdout:8/325: write d6/d8/f31 [8245435,78010] 0 2024-03-30T04:28:18.594 INFO:tasks.workunit.client.0.smithi123.stdout:5/665: mknod dd/d19/d3e/d5b/d62/dd3/cd8 0 2024-03-30T04:28:18.595 INFO:tasks.workunit.client.1.smithi135.stdout:9/299: symlink d7/d55/l66 0 2024-03-30T04:28:18.598 INFO:tasks.workunit.client.1.smithi135.stdout:0/419: write d0/d5/d10/f11 [8112732,80317] 0 2024-03-30T04:28:18.598 INFO:tasks.workunit.client.1.smithi135.stdout:5/341: dwrite d1/d3/f38 [0,4194304] 0 2024-03-30T04:28:18.598 INFO:tasks.workunit.client.0.smithi123.stdout:2/867: dwrite d0/d1f/d27/df7/f104 [0,4194304] 0 2024-03-30T04:28:18.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:18 smithi135 ceph-mon[44364]: pgmap v42: 65 pgs: 65 active+clean; 1.8 GiB data, 6.8 GiB used, 530 GiB / 536 GiB avail; 58 MiB/s rd, 194 MiB/s wr, 387 op/s 2024-03-30T04:28:18.603 INFO:tasks.workunit.client.1.smithi135.stdout:9/300: creat d7/d4f/f67 x:0 0 0 2024-03-30T04:28:18.608 INFO:tasks.workunit.client.0.smithi123.stdout:8/742: dwrite da/dc/d14/d18/f86 [0,4194304] 0 2024-03-30T04:28:18.609 INFO:tasks.workunit.client.1.smithi135.stdout:7/350: mkdir d4/db/d1a/d73 0 2024-03-30T04:28:18.616 INFO:tasks.workunit.client.0.smithi123.stdout:8/743: fsync da/dc/d14/d18/d21/da2/fc5 0 2024-03-30T04:28:18.619 INFO:tasks.workunit.client.0.smithi123.stdout:3/798: dwrite d13/d24/d31/d23/fc8 [0,4194304] 0 2024-03-30T04:28:18.619 INFO:tasks.workunit.client.1.smithi135.stdout:2/287: dwrite d4/da/f21 [4194304,4194304] 0 2024-03-30T04:28:18.620 INFO:tasks.workunit.client.1.smithi135.stdout:9/301: mknod d7/d4f/d5c/c68 0 2024-03-30T04:28:18.621 INFO:tasks.workunit.client.0.smithi123.stdout:9/682: dwrite d6/d1f/d92/db8/fe8 [0,4194304] 0 2024-03-30T04:28:18.621 INFO:tasks.workunit.client.1.smithi135.stdout:0/420: creat d0/d5/d9/d23/d58/d63/f89 x:0 0 0 2024-03-30T04:28:18.621 INFO:tasks.workunit.client.1.smithi135.stdout:5/342: creat d1/d3/d2b/d54/d5e/d5c/f65 x:0 0 0 2024-03-30T04:28:18.621 INFO:tasks.workunit.client.1.smithi135.stdout:6/312: sync 2024-03-30T04:28:18.622 INFO:tasks.workunit.client.0.smithi123.stdout:8/744: symlink da/d27/d3c/d2d/d74/le6 0 2024-03-30T04:28:18.624 INFO:tasks.workunit.client.1.smithi135.stdout:7/351: mknod d4/db/d1a/d5c/d5d/c74 0 2024-03-30T04:28:18.626 INFO:tasks.workunit.client.1.smithi135.stdout:3/320: dwrite d3/f4 [0,4194304] 0 2024-03-30T04:28:18.626 INFO:tasks.workunit.client.1.smithi135.stdout:9/302: dread - d7/d1b/f35 zero size 2024-03-30T04:28:18.627 INFO:tasks.workunit.client.0.smithi123.stdout:8/745: mkdir da/dc/de7 0 2024-03-30T04:28:18.630 INFO:tasks.workunit.client.1.smithi135.stdout:8/326: dwrite d6/d8/d15/d2a/f37 [0,4194304] 0 2024-03-30T04:28:18.633 INFO:tasks.workunit.client.0.smithi123.stdout:9/683: symlink d6/d1f/d92/db8/led 0 2024-03-30T04:28:18.633 INFO:tasks.workunit.client.0.smithi123.stdout:8/746: chown da/dc/d14/d18/d21/f45 30347 1 2024-03-30T04:28:18.636 INFO:tasks.workunit.client.0.smithi123.stdout:9/684: chown d6/fb 25082 1 2024-03-30T04:28:18.637 INFO:tasks.workunit.client.0.smithi123.stdout:3/799: creat d13/d24/d31/d23/d3a/d3c/d4e/d5d/da5/dcd/f109 x:0 0 0 2024-03-30T04:28:18.638 INFO:tasks.workunit.client.1.smithi135.stdout:2/288: dread d4/d32/d2e/d2f/f38 [0,4194304] 0 2024-03-30T04:28:18.641 INFO:tasks.workunit.client.1.smithi135.stdout:0/421: fsync d0/f3 0 2024-03-30T04:28:18.644 INFO:tasks.workunit.client.1.smithi135.stdout:5/343: rename d1/d7/d1a/d1f/f40 to d1/d7/d1a/d1f/f66 0 2024-03-30T04:28:18.646 INFO:tasks.workunit.client.0.smithi123.stdout:8/747: truncate da/f55 3724778 0 2024-03-30T04:28:18.648 INFO:tasks.workunit.client.1.smithi135.stdout:8/327: unlink d6/d23/l48 0 2024-03-30T04:28:18.648 INFO:tasks.workunit.client.1.smithi135.stdout:3/321: creat d3/d10/d16/d41/f6d x:0 0 0 2024-03-30T04:28:18.649 INFO:tasks.workunit.client.0.smithi123.stdout:5/666: write dd/d41/f47 [4591918,61774] 0 2024-03-30T04:28:18.650 INFO:tasks.workunit.client.0.smithi123.stdout:8/748: chown da/d27/d95/fce 347214175 1 2024-03-30T04:28:18.652 INFO:tasks.workunit.client.1.smithi135.stdout:8/328: chown d6/d8/f1c 4 1 2024-03-30T04:28:18.652 INFO:tasks.workunit.client.1.smithi135.stdout:7/352: mkdir d4/db/d1a/d5c/d2c/d2e/d46/d75 0 2024-03-30T04:28:18.653 INFO:tasks.workunit.client.0.smithi123.stdout:4/690: sync 2024-03-30T04:28:18.653 INFO:tasks.workunit.client.0.smithi123.stdout:7/711: sync 2024-03-30T04:28:18.653 INFO:tasks.workunit.client.0.smithi123.stdout:6/773: sync 2024-03-30T04:28:18.653 INFO:tasks.workunit.client.1.smithi135.stdout:2/289: fdatasync d4/d6/f35 0 2024-03-30T04:28:18.654 INFO:tasks.workunit.client.1.smithi135.stdout:0/422: mkdir d0/d5/d9/d23/d8a 0 2024-03-30T04:28:18.659 INFO:tasks.workunit.client.0.smithi123.stdout:5/667: symlink dd/d19/d5f/dc2/ld9 0 2024-03-30T04:28:18.662 INFO:tasks.workunit.client.0.smithi123.stdout:8/749: mknod da/d27/d3c/d2d/ce8 0 2024-03-30T04:28:18.663 INFO:tasks.workunit.client.1.smithi135.stdout:2/290: read d4/d6/f16 [6851754,126574] 0 2024-03-30T04:28:18.666 INFO:tasks.workunit.client.1.smithi135.stdout:5/344: mknod d1/c67 0 2024-03-30T04:28:18.667 INFO:tasks.workunit.client.0.smithi123.stdout:7/712: unlink d2/da/fb8 0 2024-03-30T04:28:18.669 INFO:tasks.workunit.client.1.smithi135.stdout:3/322: mkdir d3/d6e 0 2024-03-30T04:28:18.669 INFO:tasks.workunit.client.1.smithi135.stdout:0/423: mkdir d0/d5/d9/d23/d58/d63/d77/d8b 0 2024-03-30T04:28:18.670 INFO:tasks.workunit.client.0.smithi123.stdout:5/668: readlink dd/l23 0 2024-03-30T04:28:18.671 INFO:tasks.workunit.client.1.smithi135.stdout:6/313: getdents d2/dd/d14/d53 0 2024-03-30T04:28:18.673 INFO:tasks.workunit.client.0.smithi123.stdout:2/868: dwrite d0/d1f/d16/f1e [0,4194304] 0 2024-03-30T04:28:18.673 INFO:tasks.workunit.client.0.smithi123.stdout:4/691: write d2/d4/d2c/d42/d35/d5d/d6f/f8a [3406193,71975] 0 2024-03-30T04:28:18.673 INFO:tasks.workunit.client.1.smithi135.stdout:2/291: rmdir d4 39 2024-03-30T04:28:18.674 INFO:tasks.workunit.client.0.smithi123.stdout:6/774: rename d10/c7b to d10/d82/d9a/c104 0 2024-03-30T04:28:18.674 INFO:tasks.workunit.client.0.smithi123.stdout:5/669: symlink dd/d41/lda 0 2024-03-30T04:28:18.675 INFO:tasks.workunit.client.1.smithi135.stdout:3/323: write d3/d10/d16/d23/f2c [187732,76847] 0 2024-03-30T04:28:18.677 INFO:tasks.workunit.client.1.smithi135.stdout:8/329: dread d6/f50 [0,4194304] 0 2024-03-30T04:28:18.678 INFO:tasks.workunit.client.1.smithi135.stdout:9/303: rename d7/d29/f3c to d7/d1b/f69 0 2024-03-30T04:28:18.680 INFO:tasks.workunit.client.0.smithi123.stdout:2/869: symlink d0/d37/l11b 0 2024-03-30T04:28:18.681 INFO:tasks.workunit.client.0.smithi123.stdout:7/713: mknod d2/da/d11/d61/d55/d59/db5/dab/cea 0 2024-03-30T04:28:18.685 INFO:tasks.workunit.client.1.smithi135.stdout:0/424: chown d0/d1/d16/l70 2696099 1 2024-03-30T04:28:18.685 INFO:tasks.workunit.client.0.smithi123.stdout:2/870: fdatasync d0/d1f/d1c/d39/fd9 0 2024-03-30T04:28:18.687 INFO:tasks.workunit.client.1.smithi135.stdout:7/353: rmdir d4/db/d1a/d5c/d5d/d61 0 2024-03-30T04:28:18.690 INFO:tasks.workunit.client.0.smithi123.stdout:5/670: creat dd/d19/d1f/d35/d3a/d57/db3/fdb x:0 0 0 2024-03-30T04:28:18.690 INFO:tasks.workunit.client.1.smithi135.stdout:8/330: mknod d6/d23/c5a 0 2024-03-30T04:28:18.693 INFO:tasks.workunit.client.1.smithi135.stdout:9/304: rename d7/d1b/f34 to d7/d1b/d37/f6a 0 2024-03-30T04:28:18.694 INFO:tasks.workunit.client.1.smithi135.stdout:7/354: dread d4/db/f26 [0,4194304] 0 2024-03-30T04:28:18.695 INFO:tasks.workunit.client.1.smithi135.stdout:9/305: write d7/d1b/f28 [2986445,97788] 0 2024-03-30T04:28:18.699 INFO:tasks.workunit.client.0.smithi123.stdout:9/685: dread d6/d1f/d4d/f5b [0,4194304] 0 2024-03-30T04:28:18.700 INFO:tasks.workunit.client.0.smithi123.stdout:3/800: dread d13/d24/d31/f61 [0,4194304] 0 2024-03-30T04:28:18.702 INFO:tasks.workunit.client.1.smithi135.stdout:5/345: creat d1/d7/d1a/f68 x:0 0 0 2024-03-30T04:28:18.703 INFO:tasks.workunit.client.0.smithi123.stdout:2/871: mkdir d0/d1f/d30/d119/dbb/d97/d116/d11c 0 2024-03-30T04:28:18.704 INFO:tasks.workunit.client.1.smithi135.stdout:4/336: sync 2024-03-30T04:28:18.704 INFO:tasks.workunit.client.1.smithi135.stdout:1/272: sync 2024-03-30T04:28:18.705 INFO:tasks.workunit.client.1.smithi135.stdout:2/292: mknod d4/d6/c5b 0 2024-03-30T04:28:18.706 INFO:tasks.workunit.client.1.smithi135.stdout:9/306: mknod d7/d41/c6b 0 2024-03-30T04:28:18.707 INFO:tasks.workunit.client.0.smithi123.stdout:4/692: dwrite d2/d4/d2c/f67 [0,4194304] 0 2024-03-30T04:28:18.709 INFO:tasks.workunit.client.1.smithi135.stdout:4/337: write d10/d1a/d42/f54 [674539,20536] 0 2024-03-30T04:28:18.710 INFO:tasks.workunit.client.0.smithi123.stdout:9/686: creat d6/d1f/d72/d7d/fee x:0 0 0 2024-03-30T04:28:18.713 INFO:tasks.workunit.client.1.smithi135.stdout:5/346: symlink d1/d3/d2b/d54/d5e/d5c/l69 0 2024-03-30T04:28:18.714 INFO:tasks.workunit.client.0.smithi123.stdout:5/671: creat dd/d19/d1f/d35/d3a/fdc x:0 0 0 2024-03-30T04:28:18.714 INFO:tasks.workunit.client.0.smithi123.stdout:4/693: dread - d2/d26/fbd zero size 2024-03-30T04:28:18.715 INFO:tasks.workunit.client.0.smithi123.stdout:3/801: truncate d13/d24/d31/d23/d3a/f54 1623100 0 2024-03-30T04:28:18.715 INFO:tasks.workunit.client.0.smithi123.stdout:7/714: dread d2/da/d11/f45 [0,4194304] 0 2024-03-30T04:28:18.716 INFO:tasks.workunit.client.1.smithi135.stdout:6/314: truncate d2/dd/d14/f1e 82712 0 2024-03-30T04:28:18.718 INFO:tasks.workunit.client.1.smithi135.stdout:3/324: getdents d3/d50/d22/d33 0 2024-03-30T04:28:18.719 INFO:tasks.workunit.client.0.smithi123.stdout:2/872: rename d0/d1f/d5e/fbc to d0/d1f/d27/d68/f11d 0 2024-03-30T04:28:18.720 INFO:tasks.workunit.client.1.smithi135.stdout:1/273: mknod d16/d29/d2c/c5d 0 2024-03-30T04:28:18.721 INFO:tasks.workunit.client.1.smithi135.stdout:8/331: creat d6/f5b x:0 0 0 2024-03-30T04:28:18.725 INFO:tasks.workunit.client.1.smithi135.stdout:5/347: write d1/d3/d2b/d54/d5e/d5c/f4e [405905,45833] 0 2024-03-30T04:28:18.727 INFO:tasks.workunit.client.1.smithi135.stdout:3/325: rmdir d3/d54/d5c 39 2024-03-30T04:28:18.727 INFO:tasks.workunit.client.1.smithi135.stdout:7/355: dwrite f3 [0,4194304] 0 2024-03-30T04:28:18.727 INFO:tasks.workunit.client.0.smithi123.stdout:4/694: readlink d2/d4/d2c/d42/d35/d5d/l8e 0 2024-03-30T04:28:18.727 INFO:tasks.workunit.client.0.smithi123.stdout:3/802: unlink d13/d24/f4d 0 2024-03-30T04:28:18.727 INFO:tasks.workunit.client.0.smithi123.stdout:9/687: link d6/d1f/d4d/fb9 d6/d1f/d92/fef 0 2024-03-30T04:28:18.728 INFO:tasks.workunit.client.1.smithi135.stdout:6/315: symlink d2/dd/d40/l62 0 2024-03-30T04:28:18.730 INFO:tasks.workunit.client.0.smithi123.stdout:6/775: chown d10/d82/d8e/dc7/d5b/c8f 4836 1 2024-03-30T04:28:18.730 INFO:tasks.workunit.client.0.smithi123.stdout:2/873: mknod d0/d1f/d27/df7/c11e 0 2024-03-30T04:28:18.730 INFO:tasks.workunit.client.0.smithi123.stdout:4/695: chown d2/d4/d2c/d42/dad/d91/d9d 7457 1 2024-03-30T04:28:18.731 INFO:tasks.workunit.client.1.smithi135.stdout:9/307: link d7/d41/f5f d7/d55/f6c 0 2024-03-30T04:28:18.732 INFO:tasks.workunit.client.0.smithi123.stdout:7/715: rename d2/f87 to d2/dc/d56/feb 0 2024-03-30T04:28:18.735 INFO:tasks.workunit.client.0.smithi123.stdout:2/874: write d0/d1f/d30/d119/dbb/d97/fd8 [212046,84700] 0 2024-03-30T04:28:18.735 INFO:tasks.workunit.client.1.smithi135.stdout:8/332: creat d6/d23/f5c x:0 0 0 2024-03-30T04:28:18.736 INFO:tasks.workunit.client.0.smithi123.stdout:5/672: symlink dd/d19/d3e/d5b/ldd 0 2024-03-30T04:28:18.737 INFO:tasks.workunit.client.0.smithi123.stdout:9/688: creat d6/d58/ff0 x:0 0 0 2024-03-30T04:28:18.737 INFO:tasks.workunit.client.0.smithi123.stdout:4/696: mknod d2/d4/db3/ced 0 2024-03-30T04:28:18.739 INFO:tasks.workunit.client.1.smithi135.stdout:5/348: fsync d1/d7/d1a/d1f/f66 0 2024-03-30T04:28:18.739 INFO:tasks.workunit.client.0.smithi123.stdout:4/697: chown d2/dab 111820 1 2024-03-30T04:28:18.740 INFO:tasks.workunit.client.1.smithi135.stdout:2/293: dwrite d4/f1c [4194304,4194304] 0 2024-03-30T04:28:18.740 INFO:tasks.workunit.client.0.smithi123.stdout:2/875: write d0/d1f/d27/d68/dd6/d8c/f90 [1290272,44337] 0 2024-03-30T04:28:18.744 INFO:tasks.workunit.client.1.smithi135.stdout:8/333: chown d6/d8/f17 808 1 2024-03-30T04:28:18.744 INFO:tasks.workunit.client.0.smithi123.stdout:9/689: chown d6/d7c/lec 17 1 2024-03-30T04:28:18.745 INFO:tasks.workunit.client.0.smithi123.stdout:7/716: creat d2/fec x:0 0 0 2024-03-30T04:28:18.746 INFO:tasks.workunit.client.0.smithi123.stdout:2/876: fsync d0/d1f/d82/fe8 0 2024-03-30T04:28:18.747 INFO:tasks.workunit.client.0.smithi123.stdout:9/690: chown d6/f7e 290730420 1 2024-03-30T04:28:18.748 INFO:tasks.workunit.client.0.smithi123.stdout:5/673: creat dd/d19/d1f/d35/d3a/d57/db3/fde x:0 0 0 2024-03-30T04:28:18.749 INFO:tasks.workunit.client.1.smithi135.stdout:9/308: creat d7/d4f/d5c/f6d x:0 0 0 2024-03-30T04:28:18.749 INFO:tasks.workunit.client.1.smithi135.stdout:1/274: dread d16/d29/d2c/d3c/f3b [0,4194304] 0 2024-03-30T04:28:18.750 INFO:tasks.workunit.client.1.smithi135.stdout:6/316: symlink d2/dd/d18/d21/d55/l63 0 2024-03-30T04:28:18.750 INFO:tasks.workunit.client.0.smithi123.stdout:7/717: truncate d2/dc/d49/fcb 861726 0 2024-03-30T04:28:18.752 INFO:tasks.workunit.client.1.smithi135.stdout:9/309: stat d7/d1b/l1d 0 2024-03-30T04:28:18.753 INFO:tasks.workunit.client.1.smithi135.stdout:0/425: dread d0/d5/d9/f17 [0,4194304] 0 2024-03-30T04:28:18.753 INFO:tasks.workunit.client.1.smithi135.stdout:1/275: write d16/f42 [2015722,70779] 0 2024-03-30T04:28:18.754 INFO:tasks.workunit.client.0.smithi123.stdout:7/718: write d2/da/d11/f1e [4704331,22402] 0 2024-03-30T04:28:18.757 INFO:tasks.workunit.client.1.smithi135.stdout:9/310: fsync d7/d1b/d37/f39 0 2024-03-30T04:28:18.758 INFO:tasks.workunit.client.1.smithi135.stdout:6/317: chown d2/dd/d40/d52 881 1 2024-03-30T04:28:18.758 INFO:tasks.workunit.client.0.smithi123.stdout:9/691: mkdir d6/d7c/de7/df1 0 2024-03-30T04:28:18.760 INFO:tasks.workunit.client.0.smithi123.stdout:0/724: sync 2024-03-30T04:28:18.760 INFO:tasks.workunit.client.1.smithi135.stdout:1/276: chown d16/d19/f2b 1048030 1 2024-03-30T04:28:18.761 INFO:tasks.workunit.client.0.smithi123.stdout:2/877: mkdir d0/d1f/d1c/d10d/d11f 0 2024-03-30T04:28:18.762 INFO:tasks.workunit.client.1.smithi135.stdout:4/338: dwrite d10/d1a/d2b/f31 [0,4194304] 0 2024-03-30T04:28:18.762 INFO:tasks.workunit.client.0.smithi123.stdout:6/776: dwrite d10/fa0 [0,4194304] 0 2024-03-30T04:28:18.763 INFO:tasks.workunit.client.0.smithi123.stdout:7/719: mkdir d2/da/d11/d34/ded 0 2024-03-30T04:28:18.765 INFO:tasks.workunit.client.0.smithi123.stdout:9/692: chown d6/la 554155 1 2024-03-30T04:28:18.767 INFO:tasks.workunit.client.0.smithi123.stdout:0/725: mknod d6/dd/d15/d98/cf0 0 2024-03-30T04:28:18.769 INFO:tasks.workunit.client.0.smithi123.stdout:2/878: mknod d0/d1f/d27/d68/dd6/d99/c120 0 2024-03-30T04:28:18.775 INFO:tasks.workunit.client.1.smithi135.stdout:5/349: chown d1/d3/d2b/l33 26 1 2024-03-30T04:28:18.775 INFO:tasks.workunit.client.1.smithi135.stdout:6/318: rmdir d2/d36 39 2024-03-30T04:28:18.775 INFO:tasks.workunit.client.1.smithi135.stdout:7/356: mkdir d4/db/d1a/d5c/d59/d76 0 2024-03-30T04:28:18.775 INFO:tasks.workunit.client.1.smithi135.stdout:3/326: rmdir d3/d10/d16/d23/d68 0 2024-03-30T04:28:18.776 INFO:tasks.workunit.client.0.smithi123.stdout:9/693: chown d6/d1f/d45/c94 352455 1 2024-03-30T04:28:18.776 INFO:tasks.workunit.client.0.smithi123.stdout:0/726: creat d6/ff1 x:0 0 0 2024-03-30T04:28:18.776 INFO:tasks.workunit.client.0.smithi123.stdout:6/777: symlink d10/d82/d8e/dc7/d42/l105 0 2024-03-30T04:28:18.776 INFO:tasks.workunit.client.0.smithi123.stdout:2/879: truncate d0/d37/f74 3107343 0 2024-03-30T04:28:18.776 INFO:tasks.workunit.client.0.smithi123.stdout:7/720: creat d2/d4d/d80/fee x:0 0 0 2024-03-30T04:28:18.778 INFO:tasks.workunit.client.0.smithi123.stdout:9/694: write d6/d1f/d72/f8b [4673867,91435] 0 2024-03-30T04:28:18.778 INFO:tasks.workunit.client.0.smithi123.stdout:0/727: fdatasync d6/dd/d18/d1f/faf 0 2024-03-30T04:28:18.782 INFO:tasks.workunit.client.0.smithi123.stdout:7/721: write d2/dc/d7f/fa3 [1249651,82023] 0 2024-03-30T04:28:18.783 INFO:tasks.workunit.client.1.smithi135.stdout:3/327: dread - d3/d10/f5e zero size 2024-03-30T04:28:18.785 INFO:tasks.workunit.client.1.smithi135.stdout:2/294: creat d4/d32/d2e/f5c x:0 0 0 2024-03-30T04:28:18.787 INFO:tasks.workunit.client.1.smithi135.stdout:0/426: dwrite d0/d1/fa [4194304,4194304] 0 2024-03-30T04:28:18.792 INFO:tasks.workunit.client.0.smithi123.stdout:0/728: dread d6/dd/d15/d28/f97 [0,4194304] 0 2024-03-30T04:28:18.792 INFO:tasks.workunit.client.1.smithi135.stdout:8/334: creat d6/d8/d15/d54/f5d x:0 0 0 2024-03-30T04:28:18.794 INFO:tasks.workunit.client.1.smithi135.stdout:5/350: mknod d1/d45/d52/c6a 0 2024-03-30T04:28:18.797 INFO:tasks.workunit.client.1.smithi135.stdout:8/335: chown d6/f50 238275567 1 2024-03-30T04:28:18.798 INFO:tasks.workunit.client.1.smithi135.stdout:1/277: dread f3 [0,4194304] 0 2024-03-30T04:28:18.798 INFO:tasks.workunit.client.1.smithi135.stdout:8/336: readlink d6/d8/l51 0 2024-03-30T04:28:18.799 INFO:tasks.workunit.client.0.smithi123.stdout:2/880: dread d0/d1f/d16/f87 [0,4194304] 0 2024-03-30T04:28:18.800 INFO:tasks.workunit.client.0.smithi123.stdout:0/729: rename d6/dd/def to d6/dd/d15/d28/d4b/dad/df2 0 2024-03-30T04:28:18.801 INFO:tasks.workunit.client.0.smithi123.stdout:2/881: fsync d0/d1f/d1c/fb3 0 2024-03-30T04:28:18.802 INFO:tasks.workunit.client.1.smithi135.stdout:5/351: chown d1/d7/f10 297377 1 2024-03-30T04:28:18.803 INFO:tasks.workunit.client.0.smithi123.stdout:6/778: getdents d10/d82/dc1/d45/d77 0 2024-03-30T04:28:18.803 INFO:tasks.workunit.client.1.smithi135.stdout:4/339: dwrite d10/d1a/d42/f62 [0,4194304] 0 2024-03-30T04:28:18.807 INFO:tasks.workunit.client.0.smithi123.stdout:9/695: dread d6/d1f/d72/f8f [0,4194304] 0 2024-03-30T04:28:18.808 INFO:tasks.workunit.client.1.smithi135.stdout:2/295: symlink d4/d6/d47/l5d 0 2024-03-30T04:28:18.810 INFO:tasks.workunit.client.0.smithi123.stdout:0/730: fdatasync d6/dd/d15/fc3 0 2024-03-30T04:28:18.811 INFO:tasks.workunit.client.1.smithi135.stdout:0/427: rename d0/d13/c50 to d0/d5/d42/c8c 0 2024-03-30T04:28:18.813 INFO:tasks.workunit.client.0.smithi123.stdout:6/779: dread - d10/d82/dc1/dd2/ffd zero size 2024-03-30T04:28:18.816 INFO:tasks.workunit.client.1.smithi135.stdout:6/319: getdents d2/dd/d14/d30/d43 0 2024-03-30T04:28:18.816 INFO:tasks.workunit.client.0.smithi123.stdout:0/731: readlink d6/dd/d15/d28/dd3/ddd/l3e 0 2024-03-30T04:28:18.819 INFO:tasks.workunit.client.1.smithi135.stdout:0/428: stat d0/d5/d9/d23/f66 0 2024-03-30T04:28:18.824 INFO:tasks.workunit.client.0.smithi123.stdout:9/696: symlink d6/dad/lf2 0 2024-03-30T04:28:18.824 INFO:tasks.workunit.client.0.smithi123.stdout:5/674: dwrite f0 [0,4194304] 0 2024-03-30T04:28:18.824 INFO:tasks.workunit.client.0.smithi123.stdout:4/698: dwrite d2/d34/f3f [0,4194304] 0 2024-03-30T04:28:18.825 INFO:tasks.workunit.client.1.smithi135.stdout:1/278: fsync d16/d29/d32/f58 0 2024-03-30T04:28:18.826 INFO:tasks.workunit.client.1.smithi135.stdout:7/357: link d4/db/d1a/l70 d4/db/l77 0 2024-03-30T04:28:18.827 INFO:tasks.workunit.client.1.smithi135.stdout:3/328: dwrite d3/d10/d16/f43 [0,4194304] 0 2024-03-30T04:28:18.827 INFO:tasks.workunit.client.0.smithi123.stdout:6/780: readlink d10/d20/d3f/l5e 0 2024-03-30T04:28:18.830 INFO:tasks.workunit.client.1.smithi135.stdout:1/279: dread d16/d29/d2c/d3c/f3b [0,4194304] 0 2024-03-30T04:28:18.833 INFO:tasks.workunit.client.1.smithi135.stdout:3/329: truncate d3/d50/d22/f3d 819246 0 2024-03-30T04:28:18.834 INFO:tasks.workunit.client.1.smithi135.stdout:3/330: write d3/d50/d46/f58 [112557,60568] 0 2024-03-30T04:28:18.835 INFO:tasks.workunit.client.0.smithi123.stdout:0/732: mkdir d6/dd/d15/d98/d5d/d7f/df3 0 2024-03-30T04:28:18.835 INFO:tasks.workunit.client.0.smithi123.stdout:0/733: dread - d6/ff1 zero size 2024-03-30T04:28:18.838 INFO:tasks.workunit.client.0.smithi123.stdout:9/697: symlink d6/d58/lf3 0 2024-03-30T04:28:18.843 INFO:tasks.workunit.client.1.smithi135.stdout:5/352: rename d1/f3e to d1/d45/f6b 0 2024-03-30T04:28:18.843 INFO:tasks.workunit.client.1.smithi135.stdout:9/311: rmdir d7/d4f 39 2024-03-30T04:28:18.843 INFO:tasks.workunit.client.0.smithi123.stdout:1/793: sync 2024-03-30T04:28:18.845 INFO:tasks.workunit.client.1.smithi135.stdout:5/353: chown d1/d7/d1a/d1f 5587750 1 2024-03-30T04:28:18.846 INFO:tasks.workunit.client.0.smithi123.stdout:1/794: chown db/dd/d42/d46/d51/f55 158 1 2024-03-30T04:28:18.846 INFO:tasks.workunit.client.0.smithi123.stdout:1/795: readlink db/l18 0 2024-03-30T04:28:18.853 INFO:tasks.workunit.client.0.smithi123.stdout:6/781: rename d10/d82/f84 to d10/d82/dc1/dd2/f106 0 2024-03-30T04:28:18.855 INFO:tasks.workunit.client.0.smithi123.stdout:1/796: chown db/c31 97766 1 2024-03-30T04:28:18.857 INFO:tasks.workunit.client.0.smithi123.stdout:7/722: write d2/da/d11/d61/d55/d59/f8b [297014,84211] 0 2024-03-30T04:28:18.860 INFO:tasks.workunit.client.1.smithi135.stdout:2/296: symlink d4/d32/d46/l5e 0 2024-03-30T04:28:18.862 INFO:tasks.workunit.client.0.smithi123.stdout:8/750: sync 2024-03-30T04:28:18.862 INFO:tasks.workunit.client.0.smithi123.stdout:3/803: sync 2024-03-30T04:28:18.864 INFO:tasks.workunit.client.1.smithi135.stdout:9/312: stat d7/f26 0 2024-03-30T04:28:18.867 INFO:tasks.workunit.client.0.smithi123.stdout:2/882: write d0/fb4 [2177302,107319] 0 2024-03-30T04:28:18.867 INFO:tasks.workunit.client.1.smithi135.stdout:4/340: rename d10/d1a/d2b/f2f to d10/d1a/d32/d40/f76 0 2024-03-30T04:28:18.868 INFO:tasks.workunit.client.1.smithi135.stdout:5/354: rmdir d1 39 2024-03-30T04:28:18.871 INFO:tasks.workunit.client.1.smithi135.stdout:6/320: creat d2/d36/f64 x:0 0 0 2024-03-30T04:28:18.871 INFO:tasks.workunit.client.1.smithi135.stdout:8/337: creat d6/d8/d14/f5e x:0 0 0 2024-03-30T04:28:18.873 INFO:tasks.workunit.client.0.smithi123.stdout:9/698: dwrite d6/d1f/d92/fcc [0,4194304] 0 2024-03-30T04:28:18.873 INFO:tasks.workunit.client.1.smithi135.stdout:8/338: fdatasync d6/f5b 0 2024-03-30T04:28:18.874 INFO:tasks.workunit.client.0.smithi123.stdout:7/723: creat d2/dc/d83/fef x:0 0 0 2024-03-30T04:28:18.875 INFO:tasks.workunit.client.1.smithi135.stdout:8/339: chown d6/d8/d15/d1f/f44 5989520 1 2024-03-30T04:28:18.875 INFO:tasks.workunit.client.0.smithi123.stdout:3/804: fsync d13/f80 0 2024-03-30T04:28:18.879 INFO:tasks.workunit.client.1.smithi135.stdout:1/280: truncate fb 1437983 0 2024-03-30T04:28:18.881 INFO:tasks.workunit.client.0.smithi123.stdout:7/724: chown d2/da/d11/d61/d55/l5a 59778048 1 2024-03-30T04:28:18.884 INFO:tasks.workunit.client.0.smithi123.stdout:9/699: mkdir d6/d1f/d92/df4 0 2024-03-30T04:28:18.887 INFO:tasks.workunit.client.0.smithi123.stdout:6/782: creat d10/d82/dc1/d6a/d7a/de8/f107 x:0 0 0 2024-03-30T04:28:18.889 INFO:tasks.workunit.client.1.smithi135.stdout:0/429: link d0/d19/d2d/d79/l7d d0/d1/d16/l8d 0 2024-03-30T04:28:18.891 INFO:tasks.workunit.client.1.smithi135.stdout:3/331: dread d3/d10/d16/d23/f30 [0,4194304] 0 2024-03-30T04:28:18.893 INFO:tasks.workunit.client.0.smithi123.stdout:5/675: truncate dd/d19/d1f/d3c/f92 3247371 0 2024-03-30T04:28:18.894 INFO:tasks.workunit.client.1.smithi135.stdout:2/297: creat d4/d55/f5f x:0 0 0 2024-03-30T04:28:18.894 INFO:tasks.workunit.client.0.smithi123.stdout:0/734: dwrite d6/dd/d15/d28/d4b/d8a/fa6 [0,4194304] 0 2024-03-30T04:28:18.895 INFO:tasks.workunit.client.1.smithi135.stdout:5/355: write d1/d3/f49 [3758870,91846] 0 2024-03-30T04:28:18.897 INFO:tasks.workunit.client.1.smithi135.stdout:9/313: mkdir d7/d4f/d6e 0 2024-03-30T04:28:18.899 INFO:tasks.workunit.client.1.smithi135.stdout:5/356: chown d1/f37 0 1 2024-03-30T04:28:18.899 INFO:tasks.workunit.client.0.smithi123.stdout:9/700: unlink d6/f10 0 2024-03-30T04:28:18.900 INFO:tasks.workunit.client.0.smithi123.stdout:4/699: dread d2/d4/d2c/d42/d35/d5d/fdf [0,4194304] 0 2024-03-30T04:28:18.902 INFO:tasks.workunit.client.1.smithi135.stdout:9/314: readlink d7/d1b/l14 0 2024-03-30T04:28:18.903 INFO:tasks.workunit.client.1.smithi135.stdout:4/341: dwrite d10/f73 [0,4194304] 0 2024-03-30T04:28:18.904 INFO:tasks.workunit.client.0.smithi123.stdout:4/700: write d2/d4/d7/fb [3086986,50461] 0 2024-03-30T04:28:18.904 INFO:tasks.workunit.client.1.smithi135.stdout:5/357: dread - d1/d7/f59 zero size 2024-03-30T04:28:18.906 INFO:tasks.workunit.client.1.smithi135.stdout:8/340: rename d6/f12 to d6/d8/f5f 0 2024-03-30T04:28:18.907 INFO:tasks.workunit.client.0.smithi123.stdout:7/725: rename d2/da/d11/d34/l3f to d2/da/d11/d34/d85/d8e/lf0 0 2024-03-30T04:28:18.907 INFO:tasks.workunit.client.0.smithi123.stdout:6/783: rmdir d10/d82/dc1/d6a/d7a/de2 39 2024-03-30T04:28:18.911 INFO:tasks.workunit.client.0.smithi123.stdout:0/735: rename d6/l90 to d6/dd/d15/d28/d4b/dad/de8/lf4 0 2024-03-30T04:28:18.911 INFO:tasks.workunit.client.0.smithi123.stdout:6/784: readlink d10/d82/dc1/d45/l63 0 2024-03-30T04:28:18.912 INFO:tasks.workunit.client.1.smithi135.stdout:0/430: mknod d0/d5/d9/d23/d58/d63/d77/c8e 0 2024-03-30T04:28:18.915 INFO:tasks.workunit.client.0.smithi123.stdout:3/805: dwrite d13/d24/d31/d23/d3a/f40 [4194304,4194304] 0 2024-03-30T04:28:18.924 INFO:tasks.workunit.client.1.smithi135.stdout:4/342: symlink d10/d1a/d42/l77 0 2024-03-30T04:28:18.926 INFO:tasks.workunit.client.1.smithi135.stdout:5/358: creat d1/d55/f6c x:0 0 0 2024-03-30T04:28:18.927 INFO:tasks.workunit.client.1.smithi135.stdout:2/298: mknod d4/d32/d43/c60 0 2024-03-30T04:28:18.927 INFO:tasks.workunit.client.1.smithi135.stdout:1/281: getdents d16/d19/d20 0 2024-03-30T04:28:18.928 INFO:tasks.workunit.client.0.smithi123.stdout:6/785: chown d10/d82/d8e/dc7/d5b/ffc 1 1 2024-03-30T04:28:18.928 INFO:tasks.workunit.client.0.smithi123.stdout:4/701: read d2/d4/d2c/d42/d89/f94 [126414,35991] 0 2024-03-30T04:28:18.930 INFO:tasks.workunit.client.1.smithi135.stdout:2/299: chown d4/d32/d12/f3f 3018948 1 2024-03-30T04:28:18.931 INFO:tasks.workunit.client.1.smithi135.stdout:1/282: chown d16/d29/d2c/d30/d3a/l47 14 1 2024-03-30T04:28:18.932 INFO:tasks.workunit.client.1.smithi135.stdout:0/431: mknod d0/d5/d9/d23/d2a/d60/c8f 0 2024-03-30T04:28:18.932 INFO:tasks.workunit.client.1.smithi135.stdout:6/321: link d2/l6 d2/dd/d14/d2b/l65 0 2024-03-30T04:28:18.934 INFO:tasks.workunit.client.1.smithi135.stdout:4/343: symlink d10/d4d/l78 0 2024-03-30T04:28:18.935 INFO:tasks.workunit.client.1.smithi135.stdout:3/332: link d3/db/c47 d3/d10/c6f 0 2024-03-30T04:28:18.935 INFO:tasks.workunit.client.1.smithi135.stdout:1/283: readlink d16/d29/d2c/d30/d3a/l47 0 2024-03-30T04:28:18.935 INFO:tasks.workunit.client.1.smithi135.stdout:9/315: link d7/d1b/c2e d7/d55/c6f 0 2024-03-30T04:28:18.941 INFO:tasks.workunit.client.1.smithi135.stdout:1/284: dread - d16/d19/f2b zero size 2024-03-30T04:28:18.941 INFO:tasks.workunit.client.1.smithi135.stdout:4/344: dread - d10/d1a/d42/f6e zero size 2024-03-30T04:28:18.942 INFO:tasks.workunit.client.1.smithi135.stdout:4/345: fsync d10/f73 0 2024-03-30T04:28:18.944 INFO:tasks.workunit.client.1.smithi135.stdout:4/346: fsync d10/d1a/f68 0 2024-03-30T04:28:18.945 INFO:tasks.workunit.client.0.smithi123.stdout:6/786: chown d10/f48 70 1 2024-03-30T04:28:18.948 INFO:tasks.workunit.client.1.smithi135.stdout:1/285: creat d16/d19/d20/f5e x:0 0 0 2024-03-30T04:28:18.948 INFO:tasks.workunit.client.0.smithi123.stdout:9/701: dwrite d6/d1f/d4d/f5b [0,4194304] 0 2024-03-30T04:28:18.950 INFO:tasks.workunit.client.0.smithi123.stdout:5/676: dwrite dd/d19/d1f/f27 [0,4194304] 0 2024-03-30T04:28:18.951 INFO:tasks.workunit.client.0.smithi123.stdout:1/797: dread db/dd/d42/d46/d56/da2/fb0 [0,4194304] 0 2024-03-30T04:28:18.951 INFO:tasks.workunit.client.1.smithi135.stdout:3/333: getdents d3/d10/d16 0 2024-03-30T04:28:18.953 INFO:tasks.workunit.client.1.smithi135.stdout:4/347: stat d10/d1a/d42/d46/c43 0 2024-03-30T04:28:18.953 INFO:tasks.workunit.client.0.smithi123.stdout:4/702: creat d2/d4/d2c/d42/d35/d9b/da2/fee x:0 0 0 2024-03-30T04:28:18.954 INFO:tasks.workunit.client.0.smithi123.stdout:3/806: write d13/d24/d39/d79/db9/fd2 [1097312,42920] 0 2024-03-30T04:28:18.955 INFO:tasks.workunit.client.1.smithi135.stdout:1/286: chown d16/d29/d2c/d30/f4e 1961895 1 2024-03-30T04:28:18.955 INFO:tasks.workunit.client.0.smithi123.stdout:2/883: dread d0/d1f/d5e/fb9 [0,4194304] 0 2024-03-30T04:28:18.955 INFO:tasks.workunit.client.0.smithi123.stdout:5/677: readlink dd/d19/l49 0 2024-03-30T04:28:18.957 INFO:tasks.workunit.client.0.smithi123.stdout:7/726: dwrite d2/da/d22/f86 [0,4194304] 0 2024-03-30T04:28:18.959 INFO:tasks.workunit.client.0.smithi123.stdout:6/787: creat d10/d82/d8e/dc7/d5b/f108 x:0 0 0 2024-03-30T04:28:18.962 INFO:tasks.workunit.client.1.smithi135.stdout:5/359: truncate d1/d7/f48 116938 0 2024-03-30T04:28:18.964 INFO:tasks.workunit.client.1.smithi135.stdout:3/334: symlink d3/d6e/l70 0 2024-03-30T04:28:18.967 INFO:tasks.workunit.client.0.smithi123.stdout:4/703: fdatasync d2/d4/d7/d63/f82 0 2024-03-30T04:28:18.969 INFO:tasks.workunit.client.0.smithi123.stdout:0/736: dwrite d6/dd/d15/d28/dd3/fd9 [0,4194304] 0 2024-03-30T04:28:18.970 INFO:tasks.workunit.client.1.smithi135.stdout:6/322: truncate d2/f11 1342473 0 2024-03-30T04:28:18.971 INFO:tasks.workunit.client.0.smithi123.stdout:7/727: mknod d2/da/d11/d61/d55/d94/cf1 0 2024-03-30T04:28:18.977 INFO:tasks.workunit.client.0.smithi123.stdout:2/884: symlink d0/d1f/d27/d68/dd6/d8c/l121 0 2024-03-30T04:28:18.978 INFO:tasks.workunit.client.0.smithi123.stdout:6/788: rmdir d10/d82/d9a/db3 39 2024-03-30T04:28:18.979 INFO:tasks.workunit.client.1.smithi135.stdout:4/348: symlink d10/l79 0 2024-03-30T04:28:18.983 INFO:tasks.workunit.client.0.smithi123.stdout:4/704: truncate d2/d4/d2c/d42/d35/d5d/fdf 298404 0 2024-03-30T04:28:18.984 INFO:tasks.workunit.client.1.smithi135.stdout:8/341: dread f0 [0,4194304] 0 2024-03-30T04:28:18.985 INFO:tasks.workunit.client.1.smithi135.stdout:9/316: dwrite d7/d4f/f67 [0,4194304] 0 2024-03-30T04:28:18.986 INFO:tasks.workunit.client.0.smithi123.stdout:2/885: mkdir d0/d1f/d5e/d122 0 2024-03-30T04:28:18.986 INFO:tasks.workunit.client.0.smithi123.stdout:7/728: symlink d2/da/d11/d34/ded/lf2 0 2024-03-30T04:28:18.987 INFO:tasks.workunit.client.1.smithi135.stdout:6/323: mknod d2/dd/d14/c66 0 2024-03-30T04:28:18.988 INFO:tasks.workunit.client.0.smithi123.stdout:6/789: creat d10/d82/d8e/dc7/f109 x:0 0 0 2024-03-30T04:28:18.989 INFO:tasks.workunit.client.1.smithi135.stdout:4/349: chown d10/d1a/l1d 617890197 1 2024-03-30T04:28:18.991 INFO:tasks.workunit.client.0.smithi123.stdout:4/705: unlink d2/d4/f8 0 2024-03-30T04:28:18.993 INFO:tasks.workunit.client.0.smithi123.stdout:2/886: dread - d0/d1f/d30/d119/de6/ffd zero size 2024-03-30T04:28:18.995 INFO:tasks.workunit.client.0.smithi123.stdout:5/678: dwrite dd/d19/d1f/d35/d3a/fa6 [0,4194304] 0 2024-03-30T04:28:18.996 INFO:tasks.workunit.client.1.smithi135.stdout:3/335: link d3/d50/d22/f2e d3/d50/d22/f71 0 2024-03-30T04:28:18.999 INFO:tasks.workunit.client.1.smithi135.stdout:0/432: dread d0/d5/d9/f2e [0,4194304] 0 2024-03-30T04:28:18.999 INFO:tasks.workunit.client.1.smithi135.stdout:1/287: dread d16/d29/d2c/d30/d3a/f52 [0,4194304] 0 2024-03-30T04:28:19.001 INFO:tasks.workunit.client.0.smithi123.stdout:6/790: rename d10/d82/dc1/c3e to d10/d82/dd6/c10a 0 2024-03-30T04:28:19.001 INFO:tasks.workunit.client.0.smithi123.stdout:3/807: write d13/d24/d31/d30/f5a [357758,116122] 0 2024-03-30T04:28:19.002 INFO:tasks.workunit.client.0.smithi123.stdout:4/706: fsync d2/d4/d2c/d42/d35/d5d/d6f/f8a 0 2024-03-30T04:28:19.008 INFO:tasks.workunit.client.0.smithi123.stdout:2/887: mkdir d0/d1f/d1c/d10d/d11f/d123 0 2024-03-30T04:28:19.009 INFO:tasks.workunit.client.1.smithi135.stdout:4/350: rename d10/l79 to d10/d1a/l7a 0 2024-03-30T04:28:19.012 INFO:tasks.workunit.client.0.smithi123.stdout:0/737: write d6/dd/d18/d1f/f9e [253034,113456] 0 2024-03-30T04:28:19.012 INFO:tasks.workunit.client.0.smithi123.stdout:4/707: chown d2/d4/d2c/d42/d35/cc5 262 1 2024-03-30T04:28:19.015 INFO:tasks.workunit.client.1.smithi135.stdout:0/433: mkdir d0/d5/d9/d23/d5d/d90 0 2024-03-30T04:28:19.015 INFO:tasks.workunit.client.0.smithi123.stdout:3/808: mknod d13/d24/dff/c10a 0 2024-03-30T04:28:19.016 INFO:tasks.workunit.client.0.smithi123.stdout:4/708: fdatasync d2/d4/d2c/d42/d35/d5d/d78/fbe 0 2024-03-30T04:28:19.018 INFO:tasks.workunit.client.0.smithi123.stdout:5/679: symlink dd/d19/d3e/d5b/dd7/ldf 0 2024-03-30T04:28:19.019 INFO:tasks.workunit.client.1.smithi135.stdout:4/351: unlink d10/d1a/l1b 0 2024-03-30T04:28:19.020 INFO:tasks.workunit.client.0.smithi123.stdout:3/809: rmdir d13/d24/d39/d79/d90 39 2024-03-30T04:28:19.022 INFO:tasks.workunit.client.1.smithi135.stdout:1/288: rename d16/d29/d2c/d30 to d16/d5a/d5f 0 2024-03-30T04:28:19.031 INFO:tasks.workunit.client.0.smithi123.stdout:4/709: creat d2/d4/d2c/d42/d35/d5d/d78/fef x:0 0 0 2024-03-30T04:28:19.031 INFO:tasks.workunit.client.0.smithi123.stdout:0/738: read d6/dd/d15/d98/d4e/d69/f81 [648018,99767] 0 2024-03-30T04:28:19.031 INFO:tasks.workunit.client.1.smithi135.stdout:6/324: dwrite d2/dd/d18/f22 [0,4194304] 0 2024-03-30T04:28:19.032 INFO:tasks.workunit.client.0.smithi123.stdout:3/810: mkdir d13/d24/d31/d23/d3a/d3c/db8/d10b 0 2024-03-30T04:28:19.032 INFO:tasks.workunit.client.1.smithi135.stdout:3/336: dwrite d3/d50/f25 [0,4194304] 0 2024-03-30T04:28:19.032 INFO:tasks.workunit.client.0.smithi123.stdout:9/702: dread d6/f15 [0,4194304] 0 2024-03-30T04:28:19.033 INFO:tasks.workunit.client.0.smithi123.stdout:5/680: read dd/d19/d1f/d3c/f56 [15207,20831] 0 2024-03-30T04:28:19.034 INFO:tasks.workunit.client.0.smithi123.stdout:0/739: stat d6/dd/d18/d1f/l26 0 2024-03-30T04:28:19.035 INFO:tasks.workunit.client.1.smithi135.stdout:4/352: truncate d10/d1a/d3d/f5a 2215866 0 2024-03-30T04:28:19.036 INFO:tasks.workunit.client.0.smithi123.stdout:7/729: truncate d2/da/d11/d61/d55/d9e/fc2 1238567 0 2024-03-30T04:28:19.042 INFO:tasks.workunit.client.0.smithi123.stdout:5/681: dread dd/d19/d3e/d5b/f8d [0,4194304] 0 2024-03-30T04:28:19.043 INFO:tasks.workunit.client.0.smithi123.stdout:5/682: chown dd 1641412 1 2024-03-30T04:28:19.044 INFO:tasks.workunit.client.1.smithi135.stdout:6/325: creat d2/dd/d14/d30/f67 x:0 0 0 2024-03-30T04:28:19.045 INFO:tasks.workunit.client.1.smithi135.stdout:3/337: creat d3/d50/d22/f72 x:0 0 0 2024-03-30T04:28:19.046 INFO:tasks.workunit.client.1.smithi135.stdout:4/353: rename d10/d4d/l78 to d10/d1a/d2b/l7b 0 2024-03-30T04:28:19.048 INFO:tasks.workunit.client.0.smithi123.stdout:8/751: sync 2024-03-30T04:28:19.049 INFO:tasks.workunit.client.0.smithi123.stdout:3/811: mknod d13/d24/d31/d23/d3a/d3c/db8/d10b/c10c 0 2024-03-30T04:28:19.051 INFO:tasks.workunit.client.0.smithi123.stdout:9/703: mknod d6/d1f/d72/cf5 0 2024-03-30T04:28:19.051 INFO:tasks.workunit.client.1.smithi135.stdout:6/326: chown d2/dd/lf 39967 1 2024-03-30T04:28:19.051 INFO:tasks.workunit.client.1.smithi135.stdout:3/338: readlink d3/d50/d22/l2f 0 2024-03-30T04:28:19.052 INFO:tasks.workunit.client.0.smithi123.stdout:5/683: unlink dd/c36 0 2024-03-30T04:28:19.052 INFO:tasks.workunit.client.0.smithi123.stdout:7/730: mkdir d2/d4d/d80/df3 0 2024-03-30T04:28:19.053 INFO:tasks.workunit.client.1.smithi135.stdout:9/317: dread d7/d1b/f25 [0,4194304] 0 2024-03-30T04:28:19.054 INFO:tasks.workunit.client.1.smithi135.stdout:1/289: dwrite d16/d19/f1f [4194304,4194304] 0 2024-03-30T04:28:19.056 INFO:tasks.workunit.client.1.smithi135.stdout:1/290: stat d16/d19 0 2024-03-30T04:28:19.060 INFO:tasks.workunit.client.1.smithi135.stdout:3/339: read d3/d50/d46/f21 [315159,105973] 0 2024-03-30T04:28:19.061 INFO:tasks.workunit.client.0.smithi123.stdout:6/791: dread d10/d82/d8e/dc7/d5b/f85 [0,4194304] 0 2024-03-30T04:28:19.061 INFO:tasks.workunit.client.0.smithi123.stdout:1/798: dread db/dd/d5f/d6d/fc1 [0,4194304] 0 2024-03-30T04:28:19.063 INFO:tasks.workunit.client.0.smithi123.stdout:7/731: readlink d2/dc/d49/l93 0 2024-03-30T04:28:19.063 INFO:tasks.workunit.client.1.smithi135.stdout:4/354: mkdir d10/d72/d7c 0 2024-03-30T04:28:19.063 INFO:tasks.workunit.client.1.smithi135.stdout:6/327: mkdir d2/d68 0 2024-03-30T04:28:19.065 INFO:tasks.workunit.client.0.smithi123.stdout:5/684: mkdir dd/d19/d1f/d35/d3a/d57/de0 0 2024-03-30T04:28:19.065 INFO:tasks.workunit.client.0.smithi123.stdout:8/752: symlink da/dc/d14/le9 0 2024-03-30T04:28:19.067 INFO:tasks.workunit.client.1.smithi135.stdout:4/355: chown d10/d1a/d3d 19180 1 2024-03-30T04:28:19.069 INFO:tasks.workunit.client.1.smithi135.stdout:6/328: write d2/dd/d14/d2b/f46 [509156,53697] 0 2024-03-30T04:28:19.069 INFO:tasks.workunit.client.1.smithi135.stdout:1/291: mknod d16/d1a/c60 0 2024-03-30T04:28:19.070 INFO:tasks.workunit.client.0.smithi123.stdout:2/888: dwrite d0/d1f/d27/d68/f11d [0,4194304] 0 2024-03-30T04:28:19.072 INFO:tasks.workunit.client.0.smithi123.stdout:0/740: truncate d6/f72 176411 0 2024-03-30T04:28:19.074 INFO:tasks.workunit.client.1.smithi135.stdout:9/318: symlink d7/l70 0 2024-03-30T04:28:19.075 INFO:tasks.workunit.client.0.smithi123.stdout:4/710: mkdir d2/d4/d2c/d42/d35/d5d/df0 0 2024-03-30T04:28:19.076 INFO:tasks.workunit.client.0.smithi123.stdout:7/732: fdatasync d2/da/d11/d61/d55/d59/db5/f2a 0 2024-03-30T04:28:19.077 INFO:tasks.workunit.client.1.smithi135.stdout:4/356: rename d10/d1a/f68 to d10/d72/f7d 0 2024-03-30T04:28:19.078 INFO:tasks.workunit.client.0.smithi123.stdout:5/685: stat dd/d19/d1f 0 2024-03-30T04:28:19.079 INFO:tasks.workunit.client.1.smithi135.stdout:3/340: truncate d3/d54/d5c/f64 633804 0 2024-03-30T04:28:19.084 INFO:tasks.workunit.client.0.smithi123.stdout:0/741: read d6/dd/d15/d28/d4b/d8a/f93 [1791603,106336] 0 2024-03-30T04:28:19.087 INFO:tasks.workunit.client.0.smithi123.stdout:3/812: write d13/d24/f70 [3893568,6010] 0 2024-03-30T04:28:19.087 INFO:tasks.workunit.client.0.smithi123.stdout:4/711: chown d2/d4/ff 4268 1 2024-03-30T04:28:19.088 INFO:tasks.workunit.client.1.smithi135.stdout:3/341: rename d3/d10/d16/d23/f2c to d3/d50/d46/f73 0 2024-03-30T04:28:19.091 INFO:tasks.workunit.client.1.smithi135.stdout:9/319: getdents d7/d4f 0 2024-03-30T04:28:19.093 INFO:tasks.workunit.client.0.smithi123.stdout:5/686: unlink dd/d19/d1f/d35/d3a/l86 0 2024-03-30T04:28:19.094 INFO:tasks.workunit.client.0.smithi123.stdout:7/733: mknod d2/dc/d49/dbc/cf4 0 2024-03-30T04:28:19.095 INFO:tasks.workunit.client.1.smithi135.stdout:3/342: mkdir d3/d50/d22/d74 0 2024-03-30T04:28:19.096 INFO:tasks.workunit.client.1.smithi135.stdout:1/292: dwrite d16/d1a/f21 [4194304,4194304] 0 2024-03-30T04:28:19.097 INFO:tasks.workunit.client.0.smithi123.stdout:0/742: creat d6/dd/d31/db8/ff5 x:0 0 0 2024-03-30T04:28:19.098 INFO:tasks.workunit.client.1.smithi135.stdout:9/320: rmdir d7/d4f 39 2024-03-30T04:28:19.099 INFO:tasks.workunit.client.1.smithi135.stdout:1/293: fsync d16/d29/f35 0 2024-03-30T04:28:19.102 INFO:tasks.workunit.client.1.smithi135.stdout:4/357: getdents d10/d72 0 2024-03-30T04:28:19.103 INFO:tasks.workunit.client.0.smithi123.stdout:8/753: link da/dc/l26 da/d27/d56/d6c/d96/lea 0 2024-03-30T04:28:19.103 INFO:tasks.workunit.client.1.smithi135.stdout:9/321: write d7/d29/f48 [1027045,35391] 0 2024-03-30T04:28:19.106 INFO:tasks.workunit.client.1.smithi135.stdout:3/343: link d3/d10/d16/d23/c4d d3/d50/d22/c75 0 2024-03-30T04:28:19.107 INFO:tasks.workunit.client.0.smithi123.stdout:3/813: chown d13/d24/d31/d30/lf2 58777987 1 2024-03-30T04:28:19.107 INFO:tasks.workunit.client.1.smithi135.stdout:9/322: write d7/d29/f46 [999286,80947] 0 2024-03-30T04:28:19.109 INFO:tasks.workunit.client.0.smithi123.stdout:2/889: symlink d0/d1f/d30/d119/dbb/d97/d116/d11c/l124 0 2024-03-30T04:28:19.110 INFO:tasks.workunit.client.0.smithi123.stdout:7/734: creat d2/da/d11/d61/ff5 x:0 0 0 2024-03-30T04:28:19.111 INFO:tasks.workunit.client.1.smithi135.stdout:1/294: rename d16/d5a/d5f/d3d/d51 to d16/d29/d61 0 2024-03-30T04:28:19.113 INFO:tasks.workunit.client.0.smithi123.stdout:0/743: symlink d6/dd/d15/d98/d4e/d69/d6e/lf6 0 2024-03-30T04:28:19.115 INFO:tasks.workunit.client.1.smithi135.stdout:9/323: unlink d7/d1b/f69 0 2024-03-30T04:28:19.116 INFO:tasks.workunit.client.1.smithi135.stdout:9/324: readlink d7/d1b/d37/l60 0 2024-03-30T04:28:19.120 INFO:tasks.workunit.client.1.smithi135.stdout:6/329: dread d2/f11 [0,4194304] 0 2024-03-30T04:28:19.121 INFO:tasks.workunit.client.0.smithi123.stdout:9/704: dread d6/d1f/d45/d97/d69/fab [0,4194304] 0 2024-03-30T04:28:19.121 INFO:tasks.workunit.client.0.smithi123.stdout:3/814: write d13/d24/fd6 [210312,89704] 0 2024-03-30T04:28:19.124 INFO:tasks.workunit.client.0.smithi123.stdout:6/792: dwrite d10/d82/f66 [4194304,4194304] 0 2024-03-30T04:28:19.124 INFO:tasks.workunit.client.0.smithi123.stdout:1/799: dread db/dd/d42/d46/d39/f92 [0,4194304] 0 2024-03-30T04:28:19.125 INFO:tasks.workunit.client.0.smithi123.stdout:7/735: truncate d2/da/d22/d3b/f4b 5273215 0 2024-03-30T04:28:19.127 INFO:tasks.workunit.client.0.smithi123.stdout:0/744: chown d6/dd/d31/d35/d3a/ce4 32468568 1 2024-03-30T04:28:19.128 INFO:tasks.workunit.client.1.smithi135.stdout:3/344: dwrite d3/db/fe [0,4194304] 0 2024-03-30T04:28:19.129 INFO:tasks.workunit.client.1.smithi135.stdout:6/330: creat d2/dd/d14/d30/d43/f69 x:0 0 0 2024-03-30T04:28:19.130 INFO:tasks.workunit.client.0.smithi123.stdout:8/754: unlink da/d27/d56/d6c/d96/lea 0 2024-03-30T04:28:19.132 INFO:tasks.workunit.client.1.smithi135.stdout:9/325: rename d7/d1b/c21 to d7/d4f/d5c/c71 0 2024-03-30T04:28:19.138 INFO:tasks.workunit.client.0.smithi123.stdout:2/890: write d0/d1f/d16/f33 [1216143,50149] 0 2024-03-30T04:28:19.138 INFO:tasks.workunit.client.0.smithi123.stdout:3/815: write d13/d24/d31/d23/f88 [1039631,40120] 0 2024-03-30T04:28:19.139 INFO:tasks.workunit.client.0.smithi123.stdout:8/755: mknod da/dc/d14/d9d/ceb 0 2024-03-30T04:28:19.139 INFO:tasks.workunit.client.1.smithi135.stdout:6/331: truncate d2/dd/d14/d30/f60 434332 0 2024-03-30T04:28:19.139 INFO:tasks.workunit.client.1.smithi135.stdout:3/345: getdents d3/d10/d16/d6b 0 2024-03-30T04:28:19.140 INFO:tasks.workunit.client.0.smithi123.stdout:6/793: chown d10/d82/d8e/dc7/fd7 2 1 2024-03-30T04:28:19.143 INFO:tasks.workunit.client.0.smithi123.stdout:0/745: read d6/dd/d15/d28/fb6 [259528,83800] 0 2024-03-30T04:28:19.147 INFO:tasks.workunit.client.0.smithi123.stdout:4/712: dwrite d2/d4/d2c/fa7 [0,4194304] 0 2024-03-30T04:28:19.150 INFO:tasks.workunit.client.0.smithi123.stdout:1/800: truncate db/dd/d42/d46/d51/dd7/fd8 471250 0 2024-03-30T04:28:19.150 INFO:tasks.workunit.client.1.smithi135.stdout:3/346: link d3/d50/d22/d33/f3a d3/f76 0 2024-03-30T04:28:19.151 INFO:tasks.workunit.client.1.smithi135.stdout:1/295: dwrite d16/d29/d2c/d43/d50/f5b [0,4194304] 0 2024-03-30T04:28:19.154 INFO:tasks.workunit.client.1.smithi135.stdout:1/296: write d16/f25 [2013603,1411] 0 2024-03-30T04:28:19.154 INFO:tasks.workunit.client.1.smithi135.stdout:6/332: link d2/l6 d2/dd/d14/d38/l6a 0 2024-03-30T04:28:19.154 INFO:tasks.workunit.client.0.smithi123.stdout:5/687: dwrite dd/d19/f85 [0,4194304] 0 2024-03-30T04:28:19.156 INFO:tasks.workunit.client.1.smithi135.stdout:1/297: write d16/f1c [1493592,7755] 0 2024-03-30T04:28:19.158 INFO:tasks.workunit.client.0.smithi123.stdout:8/756: rmdir da/d27/d3c/d64/d78 39 2024-03-30T04:28:19.161 INFO:tasks.workunit.client.0.smithi123.stdout:4/713: write d2/fc6 [49121,45048] 0 2024-03-30T04:28:19.162 INFO:tasks.workunit.client.1.smithi135.stdout:1/298: mknod d16/d29/c62 0 2024-03-30T04:28:19.165 INFO:tasks.workunit.client.0.smithi123.stdout:5/688: unlink dd/d41/lda 0 2024-03-30T04:28:19.166 INFO:tasks.workunit.client.0.smithi123.stdout:7/736: dread d2/da/d11/fc6 [0,4194304] 0 2024-03-30T04:28:19.168 INFO:tasks.workunit.client.0.smithi123.stdout:9/705: dwrite d6/d23/f46 [0,4194304] 0 2024-03-30T04:28:19.169 INFO:tasks.workunit.client.1.smithi135.stdout:6/333: rmdir d2/dd/d14 39 2024-03-30T04:28:19.170 INFO:tasks.workunit.client.1.smithi135.stdout:3/347: dread d3/d10/d16/d23/f30 [0,4194304] 0 2024-03-30T04:28:19.171 INFO:tasks.workunit.client.0.smithi123.stdout:3/816: mknod d13/d24/d31/d23/d3a/d3c/d66/c10d 0 2024-03-30T04:28:19.172 INFO:tasks.workunit.client.1.smithi135.stdout:1/299: symlink d16/d19/d20/l63 0 2024-03-30T04:28:19.173 INFO:tasks.workunit.client.0.smithi123.stdout:5/689: chown dd/d19/d5f/d99 9292 1 2024-03-30T04:28:19.173 INFO:tasks.workunit.client.1.smithi135.stdout:6/334: rmdir d2/d2e 39 2024-03-30T04:28:19.174 INFO:tasks.workunit.client.0.smithi123.stdout:4/714: creat d2/d4/d2c/d42/d80/ff1 x:0 0 0 2024-03-30T04:28:19.177 INFO:tasks.workunit.client.1.smithi135.stdout:3/348: dread d3/d50/d46/f58 [0,4194304] 0 2024-03-30T04:28:19.177 INFO:tasks.workunit.client.0.smithi123.stdout:4/715: chown d2/l5a 56855 1 2024-03-30T04:28:19.177 INFO:tasks.workunit.client.0.smithi123.stdout:8/757: write da/f46 [4104693,3668] 0 2024-03-30T04:28:19.178 INFO:tasks.workunit.client.0.smithi123.stdout:1/801: truncate db/dd/d42/d46/d56/d5c/dd0/f105 474091 0 2024-03-30T04:28:19.179 INFO:tasks.workunit.client.0.smithi123.stdout:3/817: chown d13/d24/d39/d79/d90 12582 1 2024-03-30T04:28:19.180 INFO:tasks.workunit.client.1.smithi135.stdout:6/335: creat d2/dd/d18/f6b x:0 0 0 2024-03-30T04:28:19.182 INFO:tasks.workunit.client.0.smithi123.stdout:4/716: dread - d2/d4/d2c/d42/d35/d9b/da2/fee zero size 2024-03-30T04:28:19.187 INFO:tasks.workunit.client.0.smithi123.stdout:6/794: creat d10/d82/dc1/d6a/d7a/de2/df1/f10b x:0 0 0 2024-03-30T04:28:19.188 INFO:tasks.workunit.client.1.smithi135.stdout:1/300: link d16/d1a/c60 d16/d29/d61/c64 0 2024-03-30T04:28:19.189 INFO:tasks.workunit.client.0.smithi123.stdout:5/690: mknod dd/d19/d1f/d3c/d9e/ce1 0 2024-03-30T04:28:19.190 INFO:tasks.workunit.client.0.smithi123.stdout:3/818: fdatasync d13/d24/d39/f7a 0 2024-03-30T04:28:19.190 INFO:tasks.workunit.client.0.smithi123.stdout:1/802: chown db/dd/d5f/d6d/d85/d89/fc9 42609 1 2024-03-30T04:28:19.190 INFO:tasks.workunit.client.0.smithi123.stdout:4/717: chown d2/d34/l98 1779 1 2024-03-30T04:28:19.191 INFO:tasks.workunit.client.1.smithi135.stdout:1/301: creat d16/d19/d20/f65 x:0 0 0 2024-03-30T04:28:19.192 INFO:tasks.workunit.client.0.smithi123.stdout:9/706: creat d6/d1f/ff6 x:0 0 0 2024-03-30T04:28:19.192 INFO:tasks.workunit.client.0.smithi123.stdout:0/746: link d6/dd/d15/d28/dd3/ddd/l2e d6/dd/d31/d35/lf7 0 2024-03-30T04:28:19.193 INFO:tasks.workunit.client.0.smithi123.stdout:6/795: symlink d10/d20/da2/l10c 0 2024-03-30T04:28:19.193 INFO:tasks.workunit.client.0.smithi123.stdout:8/758: dread da/d27/d3c/d2d/d5c/fc7 [8388608,4194304] 0 2024-03-30T04:28:19.194 INFO:tasks.workunit.client.1.smithi135.stdout:6/336: rename d2/f34 to d2/dd/d14/f6c 0 2024-03-30T04:28:19.194 INFO:tasks.workunit.client.1.smithi135.stdout:1/302: creat d16/d5a/d5f/f66 x:0 0 0 2024-03-30T04:28:19.197 INFO:tasks.workunit.client.1.smithi135.stdout:3/349: dwrite d3/d10/d16/f1b [0,4194304] 0 2024-03-30T04:28:19.198 INFO:tasks.workunit.client.0.smithi123.stdout:5/691: write dd/d19/d1f/d3c/fcf [1002272,30805] 0 2024-03-30T04:28:19.198 INFO:tasks.workunit.client.0.smithi123.stdout:3/819: write d13/d24/d31/d23/f2b [578236,7207] 0 2024-03-30T04:28:19.200 INFO:tasks.workunit.client.0.smithi123.stdout:6/796: mknod d10/d82/dc1/d6a/c10d 0 2024-03-30T04:28:19.202 INFO:tasks.workunit.client.0.smithi123.stdout:8/759: write da/dc/d14/d9d/fb1 [163533,96480] 0 2024-03-30T04:28:19.205 INFO:tasks.workunit.client.0.smithi123.stdout:7/737: dwrite d2/da/d11/f48 [0,4194304] 0 2024-03-30T04:28:19.206 INFO:tasks.workunit.client.0.smithi123.stdout:5/692: fsync dd/d19/d1f/d35/d3a/d57/fb2 0 2024-03-30T04:28:19.208 INFO:tasks.workunit.client.0.smithi123.stdout:6/797: read d10/d82/d8e/dc7/d42/f47 [427040,43370] 0 2024-03-30T04:28:19.208 INFO:tasks.workunit.client.0.smithi123.stdout:4/718: truncate d2/d4/d2c/d42/d35/d5d/dc0/fce 525895 0 2024-03-30T04:28:19.208 INFO:tasks.workunit.client.0.smithi123.stdout:8/760: write da/d27/f7e [4353249,63196] 0 2024-03-30T04:28:19.209 INFO:tasks.workunit.client.0.smithi123.stdout:1/803: dread db/dd/d42/d46/d56/fb6 [0,4194304] 0 2024-03-30T04:28:19.213 INFO:tasks.workunit.client.1.smithi135.stdout:6/337: dwrite d2/dd/d14/d38/f4a [0,4194304] 0 2024-03-30T04:28:19.216 INFO:tasks.workunit.client.0.smithi123.stdout:3/820: rename d13/d24/d31/d23/feb to d13/d24/d39/d79/db9/de7/f10e 0 2024-03-30T04:28:19.218 INFO:tasks.workunit.client.0.smithi123.stdout:6/798: symlink d10/d82/dc1/d6a/d7a/de2/l10e 0 2024-03-30T04:28:19.220 INFO:tasks.workunit.client.0.smithi123.stdout:8/761: unlink da/d27/d56/d6c/c76 0 2024-03-30T04:28:19.222 INFO:tasks.workunit.client.0.smithi123.stdout:3/821: chown d13/d24/d31/d30/cf1 13 1 2024-03-30T04:28:19.223 INFO:tasks.workunit.client.1.smithi135.stdout:6/338: rmdir d2/dd/d14/d2b 39 2024-03-30T04:28:19.229 INFO:tasks.workunit.client.0.smithi123.stdout:9/707: dwrite d6/d1f/f3f [0,4194304] 0 2024-03-30T04:28:19.232 INFO:tasks.workunit.client.1.smithi135.stdout:3/350: truncate d3/d10/f14 1616574 0 2024-03-30T04:28:19.235 INFO:tasks.workunit.client.0.smithi123.stdout:4/719: read d2/d4/d2c/d42/d35/d5d/d6f/f8a [399560,57804] 0 2024-03-30T04:28:19.235 INFO:tasks.workunit.client.0.smithi123.stdout:3/822: dread - d13/d24/fed zero size 2024-03-30T04:28:19.235 INFO:tasks.workunit.client.0.smithi123.stdout:1/804: rename db/dd/d5f/d6d/c7d to db/dd/d42/d8f/d78/c109 0 2024-03-30T04:28:19.238 INFO:tasks.workunit.client.0.smithi123.stdout:5/693: dwrite dd/d19/d1f/f27 [0,4194304] 0 2024-03-30T04:28:19.239 INFO:tasks.workunit.client.1.smithi135.stdout:3/351: link d3/d50/d22/d33/f3a d3/db/f77 0 2024-03-30T04:28:19.241 INFO:tasks.workunit.client.0.smithi123.stdout:7/738: dread d2/f8 [0,4194304] 0 2024-03-30T04:28:19.243 INFO:tasks.workunit.client.0.smithi123.stdout:6/799: creat d10/d82/d9a/f10f x:0 0 0 2024-03-30T04:28:19.244 INFO:tasks.workunit.client.1.smithi135.stdout:3/352: creat d3/d54/d5c/f78 x:0 0 0 2024-03-30T04:28:19.246 INFO:tasks.workunit.client.0.smithi123.stdout:4/720: symlink d2/d4/d2c/d42/dad/d91/lf2 0 2024-03-30T04:28:19.249 INFO:tasks.workunit.client.0.smithi123.stdout:1/805: creat db/dd/d5f/f10a x:0 0 0 2024-03-30T04:28:19.249 INFO:tasks.workunit.client.0.smithi123.stdout:9/708: dread d6/d1f/d72/fc3 [0,4194304] 0 2024-03-30T04:28:19.249 INFO:tasks.workunit.client.0.smithi123.stdout:3/823: rmdir d13/d24/d31/d23/d3a/d3c/d4e/dde 39 2024-03-30T04:28:19.249 INFO:tasks.workunit.client.0.smithi123.stdout:6/800: readlink d10/d82/d8e/dc7/d42/l105 0 2024-03-30T04:28:19.250 INFO:tasks.workunit.client.0.smithi123.stdout:5/694: creat dd/d19/d3e/d5b/d62/dd3/fe2 x:0 0 0 2024-03-30T04:28:19.250 INFO:tasks.workunit.client.1.smithi135.stdout:3/353: mknod d3/c79 0 2024-03-30T04:28:19.250 INFO:tasks.workunit.client.1.smithi135.stdout:3/354: chown l1 948954640 1 2024-03-30T04:28:19.255 INFO:tasks.workunit.client.0.smithi123.stdout:4/721: symlink d2/d34/lf3 0 2024-03-30T04:28:19.258 INFO:tasks.workunit.client.0.smithi123.stdout:1/806: fsync db/dd/d42/d46/d51/fb1 0 2024-03-30T04:28:19.259 INFO:tasks.workunit.client.0.smithi123.stdout:8/762: dwrite da/dc/d14/d18/fb0 [0,4194304] 0 2024-03-30T04:28:19.260 INFO:tasks.workunit.client.0.smithi123.stdout:6/801: mkdir d10/d82/dc1/d45/d77/d110 0 2024-03-30T04:28:19.261 INFO:tasks.workunit.client.0.smithi123.stdout:9/709: dread d6/d1f/d45/d97/d69/daa/fd0 [0,4194304] 0 2024-03-30T04:28:19.264 INFO:tasks.workunit.client.0.smithi123.stdout:3/824: rmdir d13/d24/d31/d23/d3a/d3c 39 2024-03-30T04:28:19.266 INFO:tasks.workunit.client.1.smithi135.stdout:6/339: dread d2/f12 [0,4194304] 0 2024-03-30T04:28:19.268 INFO:tasks.workunit.client.1.smithi135.stdout:3/355: dwrite d3/d50/fa [0,4194304] 0 2024-03-30T04:28:19.269 INFO:tasks.workunit.client.0.smithi123.stdout:6/802: mkdir d10/d82/dc1/d45/d111 0 2024-03-30T04:28:19.270 INFO:tasks.workunit.client.0.smithi123.stdout:3/825: read - d13/d24/fed zero size 2024-03-30T04:28:19.270 INFO:tasks.workunit.client.0.smithi123.stdout:9/710: write d6/d1f/d72/fe9 [284251,51105] 0 2024-03-30T04:28:19.271 INFO:tasks.workunit.client.1.smithi135.stdout:6/340: truncate d2/dd/d14/f15 857780 0 2024-03-30T04:28:19.272 INFO:tasks.workunit.client.1.smithi135.stdout:3/356: mknod d3/d10/d16/d23/c7a 0 2024-03-30T04:28:19.275 INFO:tasks.workunit.client.0.smithi123.stdout:8/763: link da/d27/d56/fc8 da/dc/da9/dc4/fec 0 2024-03-30T04:28:19.277 INFO:tasks.workunit.client.1.smithi135.stdout:6/341: unlink d2/dd/d14/l26 0 2024-03-30T04:28:19.278 INFO:tasks.workunit.client.1.smithi135.stdout:3/357: mkdir d3/d50/d22/d33/d7b 0 2024-03-30T04:28:19.279 INFO:tasks.workunit.client.0.smithi123.stdout:3/826: write d13/d24/d31/d23/d3a/d3c/fce [920998,98068] 0 2024-03-30T04:28:19.280 INFO:tasks.workunit.client.1.smithi135.stdout:6/342: truncate d2/dd/d14/f3a 1497924 0 2024-03-30T04:28:19.282 INFO:tasks.workunit.client.0.smithi123.stdout:7/739: write d2/da/d11/d34/d76/d91/f9f [338326,94323] 0 2024-03-30T04:28:19.282 INFO:tasks.workunit.client.0.smithi123.stdout:6/803: mkdir d10/d82/dc1/d6a/d7a/de2/df1/d112 0 2024-03-30T04:28:19.283 INFO:tasks.workunit.client.0.smithi123.stdout:5/695: write dd/d19/d3e/d5b/f5e [66082,86499] 0 2024-03-30T04:28:19.285 INFO:tasks.workunit.client.1.smithi135.stdout:3/358: link d3/d50/d22/f3d d3/d50/d22/d74/f7c 0 2024-03-30T04:28:19.286 INFO:tasks.workunit.client.1.smithi135.stdout:6/343: symlink d2/dd/d14/l6d 0 2024-03-30T04:28:19.286 INFO:tasks.workunit.client.0.smithi123.stdout:4/722: creat d2/d4/d7/ff4 x:0 0 0 2024-03-30T04:28:19.286 INFO:tasks.workunit.client.0.smithi123.stdout:9/711: creat d6/d1f/ff7 x:0 0 0 2024-03-30T04:28:19.287 INFO:tasks.workunit.client.1.smithi135.stdout:3/359: rename d3/d10/d16 to d3/d10/d16/d23/d7d 22 2024-03-30T04:28:19.288 INFO:tasks.workunit.client.1.smithi135.stdout:3/360: chown d3/d50/c3f 1690452 1 2024-03-30T04:28:19.289 INFO:tasks.workunit.client.1.smithi135.stdout:3/361: stat d3/d50/cd 0 2024-03-30T04:28:19.290 INFO:tasks.workunit.client.1.smithi135.stdout:3/362: readlink d3/d50/d22/d33/l5a 0 2024-03-30T04:28:19.291 INFO:tasks.workunit.client.1.smithi135.stdout:3/363: stat d3/d10/d16/d23/c3b 0 2024-03-30T04:28:19.292 INFO:tasks.workunit.client.1.smithi135.stdout:3/364: readlink d3/d10/d16/d41/l52 0 2024-03-30T04:28:19.294 INFO:tasks.workunit.client.0.smithi123.stdout:3/827: creat d13/d24/d31/d23/d3a/d3c/d66/f10f x:0 0 0 2024-03-30T04:28:19.298 INFO:tasks.workunit.client.0.smithi123.stdout:7/740: write d2/da/d11/d34/d76/d91/f9f [767351,69254] 0 2024-03-30T04:28:19.298 INFO:tasks.workunit.client.1.smithi135.stdout:6/344: dwrite d2/dd/d18/f4b [0,4194304] 0 2024-03-30T04:28:19.302 INFO:tasks.workunit.client.1.smithi135.stdout:6/345: write d2/dd/f1a [1321187,45064] 0 2024-03-30T04:28:19.302 INFO:tasks.workunit.client.0.smithi123.stdout:6/804: symlink d10/d82/dc1/dd2/dda/l113 0 2024-03-30T04:28:19.303 INFO:tasks.workunit.client.0.smithi123.stdout:8/764: dread da/dc/da9/fad [0,4194304] 0 2024-03-30T04:28:19.304 INFO:tasks.workunit.client.0.smithi123.stdout:4/723: mknod d2/d4/d2c/d42/d35/d9b/db7/cf5 0 2024-03-30T04:28:19.306 INFO:tasks.workunit.client.0.smithi123.stdout:7/741: rename d2/dc/d83/fd1 to d2/da/d22/d3b/ff6 0 2024-03-30T04:28:19.307 INFO:tasks.workunit.client.0.smithi123.stdout:5/696: symlink dd/d19/d5f/d99/da8/le3 0 2024-03-30T04:28:19.308 INFO:tasks.workunit.client.0.smithi123.stdout:8/765: mkdir da/d27/d56/d6c/d6d/ded 0 2024-03-30T04:28:19.309 INFO:tasks.workunit.client.0.smithi123.stdout:4/724: dread - d2/dab/fbc zero size 2024-03-30T04:28:19.312 INFO:tasks.workunit.client.0.smithi123.stdout:9/712: rename d6/d1f/c2c to d6/d1f/d45/d4c/cf8 0 2024-03-30T04:28:19.312 INFO:tasks.workunit.client.0.smithi123.stdout:6/805: truncate d10/d82/d8e/dc7/d42/f51 1318576 0 2024-03-30T04:28:19.314 INFO:tasks.workunit.client.0.smithi123.stdout:3/828: creat d13/d24/d31/d23/d3a/d3c/f110 x:0 0 0 2024-03-30T04:28:19.315 INFO:tasks.workunit.client.0.smithi123.stdout:4/725: creat d2/d26/ff6 x:0 0 0 2024-03-30T04:28:19.317 INFO:tasks.workunit.client.0.smithi123.stdout:5/697: rename dd/d19/l31 to dd/d19/d3e/d5b/d62/le4 0 2024-03-30T04:28:19.318 INFO:tasks.workunit.client.0.smithi123.stdout:8/766: truncate da/dc/d14/f71 115682 0 2024-03-30T04:28:19.319 INFO:tasks.workunit.client.0.smithi123.stdout:5/698: stat dd/d19/f59 0 2024-03-30T04:28:19.321 INFO:tasks.workunit.client.0.smithi123.stdout:9/713: mkdir d6/d49/df9 0 2024-03-30T04:28:19.325 INFO:tasks.workunit.client.0.smithi123.stdout:3/829: read d13/fa1 [2280855,66240] 0 2024-03-30T04:28:19.325 INFO:tasks.workunit.client.1.smithi135.stdout:3/365: dread d3/f11 [0,4194304] 0 2024-03-30T04:28:19.326 INFO:tasks.workunit.client.0.smithi123.stdout:8/767: unlink da/f3d 0 2024-03-30T04:28:19.326 INFO:tasks.workunit.client.0.smithi123.stdout:5/699: fdatasync dd/d19/d1f/d3c/fcf 0 2024-03-30T04:28:19.327 INFO:tasks.workunit.client.1.smithi135.stdout:3/366: chown d3/cc 1596356 1 2024-03-30T04:28:19.331 INFO:tasks.workunit.client.0.smithi123.stdout:7/742: dwrite d2/da/d11/d34/d85/fa9 [4194304,4194304] 0 2024-03-30T04:28:19.333 INFO:tasks.workunit.client.0.smithi123.stdout:9/714: rename d6/d1f/da3 to d6/d1f/d72/d7d/dfa 0 2024-03-30T04:28:19.336 INFO:tasks.workunit.client.0.smithi123.stdout:8/768: readlink da/dc/l8d 0 2024-03-30T04:28:19.336 INFO:tasks.workunit.client.0.smithi123.stdout:9/715: fsync d6/d1f/d92/db8/fe8 0 2024-03-30T04:28:19.340 INFO:tasks.workunit.client.0.smithi123.stdout:9/716: write d6/dad/fd7 [486139,128866] 0 2024-03-30T04:28:19.340 INFO:tasks.workunit.client.1.smithi135.stdout:3/367: dwrite d3/d10/d16/d41/f53 [0,4194304] 0 2024-03-30T04:28:19.342 INFO:tasks.workunit.client.0.smithi123.stdout:3/830: dread d13/d24/d31/f75 [0,4194304] 0 2024-03-30T04:28:19.344 INFO:tasks.workunit.client.0.smithi123.stdout:8/769: readlink da/dc/l26 0 2024-03-30T04:28:19.345 INFO:tasks.workunit.client.1.smithi135.stdout:3/368: creat d3/d10/d16/d23/d6a/f7e x:0 0 0 2024-03-30T04:28:19.348 INFO:tasks.workunit.client.0.smithi123.stdout:5/700: rename f9 to dd/d19/d5f/d99/fe5 0 2024-03-30T04:28:19.348 INFO:tasks.workunit.client.1.smithi135.stdout:3/369: truncate d3/d50/d22/f72 701869 0 2024-03-30T04:28:19.351 INFO:tasks.workunit.client.1.smithi135.stdout:3/370: creat d3/d10/d16/d23/f7f x:0 0 0 2024-03-30T04:28:19.353 INFO:tasks.workunit.client.0.smithi123.stdout:8/770: getdents da/d27/d56/d6c/dd4 0 2024-03-30T04:28:19.357 INFO:tasks.workunit.client.0.smithi123.stdout:5/701: creat dd/d19/d1f/d35/d3a/d57/de0/fe6 x:0 0 0 2024-03-30T04:28:19.357 INFO:tasks.workunit.client.1.smithi135.stdout:3/371: link d3/d50/fa d3/d6e/f80 0 2024-03-30T04:28:19.359 INFO:tasks.workunit.client.0.smithi123.stdout:8/771: creat da/d27/d3c/d2d/dae/fee x:0 0 0 2024-03-30T04:28:19.359 INFO:tasks.workunit.client.0.smithi123.stdout:5/702: readlink dd/d19/d3e/l53 0 2024-03-30T04:28:19.365 INFO:tasks.workunit.client.0.smithi123.stdout:8/772: mknod da/dc/d14/d18/d21/da2/cef 0 2024-03-30T04:28:19.367 INFO:tasks.workunit.client.0.smithi123.stdout:8/773: fdatasync da/d27/d95/fce 0 2024-03-30T04:28:19.368 INFO:tasks.workunit.client.0.smithi123.stdout:5/703: rename dd/d19/f9f to dd/d19/d1f/d3c/d72/d8c/fe7 0 2024-03-30T04:28:19.371 INFO:tasks.workunit.client.0.smithi123.stdout:8/774: fsync f8 0 2024-03-30T04:28:19.374 INFO:tasks.workunit.client.0.smithi123.stdout:9/717: dread d6/d1f/f2d [0,4194304] 0 2024-03-30T04:28:19.376 INFO:tasks.workunit.client.0.smithi123.stdout:9/718: stat d6/d7c 0 2024-03-30T04:28:19.380 INFO:tasks.workunit.client.0.smithi123.stdout:8/775: mknod da/d27/d56/d6c/dd4/cf0 0 2024-03-30T04:28:19.390 INFO:tasks.workunit.client.0.smithi123.stdout:3/831: dread d13/d24/d31/d23/d3a/d3c/d4e/d5d/da5/fad [0,4194304] 0 2024-03-30T04:28:19.392 INFO:tasks.workunit.client.0.smithi123.stdout:7/743: truncate d2/da/d22/d3b/f4b 4611126 0 2024-03-30T04:28:19.392 INFO:tasks.workunit.client.0.smithi123.stdout:6/806: dwrite d10/d82/dc1/d6a/fae [0,4194304] 0 2024-03-30T04:28:19.397 INFO:tasks.workunit.client.0.smithi123.stdout:9/719: rename d6/d23 to d6/d1f/d45/d97/d69/dfb 0 2024-03-30T04:28:19.399 INFO:tasks.workunit.client.0.smithi123.stdout:8/776: symlink da/dc/d14/d18/d21/da2/lf1 0 2024-03-30T04:28:19.401 INFO:tasks.workunit.client.0.smithi123.stdout:4/726: dwrite d2/d34/f93 [0,4194304] 0 2024-03-30T04:28:19.402 INFO:tasks.workunit.client.0.smithi123.stdout:6/807: write d10/d82/d8e/dc7/d5b/fef [875419,61487] 0 2024-03-30T04:28:19.410 INFO:tasks.workunit.client.0.smithi123.stdout:5/704: write dd/d19/d1f/d3c/f92 [3708863,8687] 0 2024-03-30T04:28:19.423 INFO:tasks.workunit.client.0.smithi123.stdout:5/705: rmdir dd/d19/d5f/d99 39 2024-03-30T04:28:19.431 INFO:tasks.workunit.client.0.smithi123.stdout:0/747: sync 2024-03-30T04:28:19.431 INFO:tasks.workunit.client.0.smithi123.stdout:2/891: sync 2024-03-30T04:28:19.432 INFO:tasks.workunit.client.0.smithi123.stdout:9/720: dwrite d6/f26 [4194304,4194304] 0 2024-03-30T04:28:19.433 INFO:tasks.workunit.client.0.smithi123.stdout:3/832: getdents d13/d24/d39/d79/db9 0 2024-03-30T04:28:19.437 INFO:tasks.workunit.client.0.smithi123.stdout:2/892: truncate d0/d1f/fdd 1153987 0 2024-03-30T04:28:19.438 INFO:tasks.workunit.client.0.smithi123.stdout:5/706: mkdir dd/d19/d5f/dc2/de8 0 2024-03-30T04:28:19.442 INFO:tasks.workunit.client.1.smithi135.stdout:7/358: sync 2024-03-30T04:28:19.443 INFO:tasks.workunit.client.0.smithi123.stdout:4/727: dread d2/d4/fe7 [0,4194304] 0 2024-03-30T04:28:19.450 INFO:tasks.workunit.client.0.smithi123.stdout:5/707: creat dd/d19/d1f/d35/d3a/d57/de0/fe9 x:0 0 0 2024-03-30T04:28:19.450 INFO:tasks.workunit.client.0.smithi123.stdout:7/744: dread d2/dc/d1c/f23 [0,4194304] 0 2024-03-30T04:28:19.454 INFO:tasks.workunit.client.0.smithi123.stdout:6/808: dwrite d10/d20/da2/fc3 [0,4194304] 0 2024-03-30T04:28:19.459 INFO:tasks.workunit.client.0.smithi123.stdout:9/721: write d6/d1f/d45/d4c/faf [4940372,56362] 0 2024-03-30T04:28:19.468 INFO:tasks.workunit.client.1.smithi135.stdout:7/359: dread d4/db/d1a/d4b/f31 [0,4194304] 0 2024-03-30T04:28:19.469 INFO:tasks.workunit.client.1.smithi135.stdout:7/360: chown d4/db/d1a/d5c/d59/d76 72 1 2024-03-30T04:28:19.470 INFO:tasks.workunit.client.0.smithi123.stdout:7/745: creat d2/da/d11/d61/ff7 x:0 0 0 2024-03-30T04:28:19.472 INFO:tasks.workunit.client.0.smithi123.stdout:9/722: creat d6/d1f/d45/d97/d69/dfb/ffc x:0 0 0 2024-03-30T04:28:19.472 INFO:tasks.workunit.client.0.smithi123.stdout:4/728: mknod d2/d4/d2c/d42/d35/d9b/da2/deb/cf7 0 2024-03-30T04:28:19.476 INFO:tasks.workunit.client.1.smithi135.stdout:7/361: truncate d4/db/d1a/f66 2246485 0 2024-03-30T04:28:19.478 INFO:tasks.workunit.client.0.smithi123.stdout:5/708: creat dd/fea x:0 0 0 2024-03-30T04:28:19.478 INFO:tasks.workunit.client.0.smithi123.stdout:7/746: chown d2/da/d11/d34/l6e 6308935 1 2024-03-30T04:28:19.480 INFO:tasks.workunit.client.0.smithi123.stdout:0/748: truncate d6/d52/fa8 1092362 0 2024-03-30T04:28:19.481 INFO:tasks.workunit.client.0.smithi123.stdout:8/777: link da/dc/d14/d18/d21/da2/ld6 da/d27/d3c/d64/d78/db8/lf2 0 2024-03-30T04:28:19.489 INFO:tasks.workunit.client.0.smithi123.stdout:6/809: dwrite d10/d82/dc1/d6a/d7a/d99/dba/ff8 [0,4194304] 0 2024-03-30T04:28:19.491 INFO:tasks.workunit.client.0.smithi123.stdout:4/729: mknod d2/d4/d2c/d42/d35/d5d/d6f/cf8 0 2024-03-30T04:28:19.491 INFO:tasks.workunit.client.1.smithi135.stdout:7/362: dwrite d4/db/de/f28 [0,4194304] 0 2024-03-30T04:28:19.495 INFO:tasks.workunit.client.0.smithi123.stdout:0/749: write d6/ff1 [545068,92718] 0 2024-03-30T04:28:19.496 INFO:tasks.workunit.client.0.smithi123.stdout:7/747: write d2/da/d22/fe0 [538576,58740] 0 2024-03-30T04:28:19.497 INFO:tasks.workunit.client.1.smithi135.stdout:7/363: symlink d4/db/d1a/d5c/d2c/l78 0 2024-03-30T04:28:19.502 INFO:tasks.workunit.client.0.smithi123.stdout:5/709: chown dd/d19/d1f/d35/l54 5766 1 2024-03-30T04:28:19.505 INFO:tasks.workunit.client.0.smithi123.stdout:7/748: chown d2/da/d11/d34/d76/cc4 0 1 2024-03-30T04:28:19.509 INFO:tasks.workunit.client.0.smithi123.stdout:2/893: dwrite d0/d1f/d1c/d36/fab [0,4194304] 0 2024-03-30T04:28:19.513 INFO:tasks.workunit.client.0.smithi123.stdout:1/807: sync 2024-03-30T04:28:19.513 INFO:tasks.workunit.client.0.smithi123.stdout:3/833: dwrite fd [0,4194304] 0 2024-03-30T04:28:19.522 INFO:tasks.workunit.client.0.smithi123.stdout:9/723: getdents d6/d1f/d45 0 2024-03-30T04:28:19.526 INFO:tasks.workunit.client.0.smithi123.stdout:2/894: write d0/d37/d9b/fe9 [1436970,15932] 0 2024-03-30T04:28:19.533 INFO:tasks.workunit.client.1.smithi135.stdout:7/364: dread d4/db/f14 [4194304,4194304] 0 2024-03-30T04:28:19.535 INFO:tasks.workunit.client.1.smithi135.stdout:7/365: write d4/db/de/d5a/f68 [1045122,82809] 0 2024-03-30T04:28:19.535 INFO:tasks.workunit.client.0.smithi123.stdout:5/710: dread dd/d19/d1f/f82 [4194304,4194304] 0 2024-03-30T04:28:19.536 INFO:tasks.workunit.client.0.smithi123.stdout:7/749: truncate d2/da/d11/d61/d55/d59/db5/f43 65418 0 2024-03-30T04:28:19.543 INFO:tasks.workunit.client.0.smithi123.stdout:9/724: creat d6/d1f/d72/d7d/dfa/ffd x:0 0 0 2024-03-30T04:28:19.546 INFO:tasks.workunit.client.0.smithi123.stdout:4/730: write d2/d4/d7/d1a/f5e [1109661,53751] 0 2024-03-30T04:28:19.546 INFO:tasks.workunit.client.0.smithi123.stdout:7/750: rmdir d2/dc/d49 39 2024-03-30T04:28:19.550 INFO:tasks.workunit.client.1.smithi135.stdout:7/366: dwrite d4/db/f1c [0,4194304] 0 2024-03-30T04:28:19.551 INFO:tasks.workunit.client.0.smithi123.stdout:2/895: rename d0/d1f/d1c/d40/f5a to d0/d1f/f125 0 2024-03-30T04:28:19.552 INFO:tasks.workunit.client.0.smithi123.stdout:9/725: chown d6/d1f/d4d/c6f 11717 1 2024-03-30T04:28:19.553 INFO:tasks.workunit.client.1.smithi135.stdout:7/367: mkdir d4/db/d79 0 2024-03-30T04:28:19.555 INFO:tasks.workunit.client.1.smithi135.stdout:7/368: symlink d4/db/d1a/d4b/d4f/l7a 0 2024-03-30T04:28:19.559 INFO:tasks.workunit.client.1.smithi135.stdout:7/369: rename d4/db/de/f30 to d4/db/d1a/d5c/d5d/f7b 0 2024-03-30T04:28:19.560 INFO:tasks.workunit.client.0.smithi123.stdout:5/711: mkdir dd/d19/d1f/d3c/d72/d8c/deb 0 2024-03-30T04:28:19.560 INFO:tasks.workunit.client.1.smithi135.stdout:7/370: stat d4/db/d1a/d4b 0 2024-03-30T04:28:19.561 INFO:tasks.workunit.client.0.smithi123.stdout:4/731: read d2/d4/d7/d1a/f30 [10536303,123541] 0 2024-03-30T04:28:19.562 INFO:tasks.workunit.client.0.smithi123.stdout:2/896: stat d0/d1f/d30/d119/dbb/dd5/c7f 0 2024-03-30T04:28:19.562 INFO:tasks.workunit.client.0.smithi123.stdout:5/712: readlink l4 0 2024-03-30T04:28:19.564 INFO:tasks.workunit.client.1.smithi135.stdout:7/371: symlink d4/db/d1a/d3d/l7c 0 2024-03-30T04:28:19.565 INFO:tasks.workunit.client.0.smithi123.stdout:8/778: dread da/d27/d56/f65 [0,4194304] 0 2024-03-30T04:28:19.567 INFO:tasks.workunit.client.0.smithi123.stdout:6/810: dwrite d10/d20/f7f [0,4194304] 0 2024-03-30T04:28:19.569 INFO:tasks.workunit.client.0.smithi123.stdout:9/726: read d6/d1f/d45/d97/d69/daa/fd0 [99259,33456] 0 2024-03-30T04:28:19.575 INFO:tasks.workunit.client.1.smithi135.stdout:7/372: dread f3 [0,4194304] 0 2024-03-30T04:28:19.575 INFO:tasks.workunit.client.0.smithi123.stdout:3/834: dwrite d13/d24/d31/d23/d3a/d3c/db8/fca [0,4194304] 0 2024-03-30T04:28:19.576 INFO:tasks.workunit.client.0.smithi123.stdout:2/897: mkdir d0/d1f/d27/d68/dd6/d126 0 2024-03-30T04:28:19.582 INFO:tasks.workunit.client.0.smithi123.stdout:5/713: rename dd/d19/d5f/d99/cc3 to dd/d19/d5f/d99/da8/cec 0 2024-03-30T04:28:19.584 INFO:tasks.workunit.client.0.smithi123.stdout:2/898: chown d0/d1f/d27/d68/dd6/d99/c120 2363 1 2024-03-30T04:28:19.590 INFO:tasks.workunit.client.1.smithi135.stdout:7/373: dwrite d4/db/de/f28 [0,4194304] 0 2024-03-30T04:28:19.594 INFO:tasks.workunit.client.0.smithi123.stdout:2/899: write d0/d1f/d1c/d20/f56 [801773,108298] 0 2024-03-30T04:28:19.594 INFO:tasks.workunit.client.0.smithi123.stdout:3/835: fdatasync d13/d24/d31/d23/d3a/d3c/d4e/d5d/da5/fad 0 2024-03-30T04:28:19.594 INFO:tasks.workunit.client.0.smithi123.stdout:7/751: link d2/dc/d49/d68/ca0 d2/dc/d7f/cf8 0 2024-03-30T04:28:19.595 INFO:tasks.workunit.client.0.smithi123.stdout:0/750: dread d6/fa [0,4194304] 0 2024-03-30T04:28:19.597 INFO:tasks.workunit.client.1.smithi135.stdout:7/374: rename d4/db/de/l12 to d4/db/d1a/d5c/d2c/d2e/d46/d75/l7d 0 2024-03-30T04:28:19.599 INFO:tasks.workunit.client.1.smithi135.stdout:7/375: stat d4/db/d1a/d3d/l7c 0 2024-03-30T04:28:19.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:19.600+0000 7f72c6e00700 1 -- 172.21.15.123:0/1400443850 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72c0071f50 msgr2=0x7f72c0070460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:28:19.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:19.600+0000 7f72c6e00700 1 --2- 172.21.15.123:0/1400443850 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72c0071f50 0x7f72c0070460 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f72b0004940 tx=0x7f72b0009e30 comp rx=0 tx=0).stop 2024-03-30T04:28:19.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:19.601+0000 7f72c6e00700 1 -- 172.21.15.123:0/1400443850 shutdown_connections 2024-03-30T04:28:19.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:19.601+0000 7f72c6e00700 1 --2- 172.21.15.123:0/1400443850 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f72c00709a0 0x7f72c0070e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:19.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:19.601+0000 7f72c6e00700 1 --2- 172.21.15.123:0/1400443850 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72c0071f50 0x7f72c0070460 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:19.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:19.601+0000 7f72c6e00700 1 -- 172.21.15.123:0/1400443850 >> 172.21.15.123:0/1400443850 conn(0x7f72c00f8560 msgr2=0x7f72c00fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:28:19.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:19.601+0000 7f72c6e00700 1 -- 172.21.15.123:0/1400443850 shutdown_connections 2024-03-30T04:28:19.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:19.601+0000 7f72c6e00700 1 -- 172.21.15.123:0/1400443850 wait complete. 2024-03-30T04:28:19.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:19.602+0000 7f72c6e00700 1 Processor -- start 2024-03-30T04:28:19.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:19.603+0000 7f72c6e00700 1 -- start start 2024-03-30T04:28:19.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:19.604+0000 7f72c6e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f72c00709a0 0x7f72c0113f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:28:19.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:19.604+0000 7f72c6e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72c0071f50 0x7f72c01144a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:28:19.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:19.604+0000 7f72c6e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c0114aa0 con 0x7f72c0071f50 2024-03-30T04:28:19.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:19.604+0000 7f72c6e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c01152f0 con 0x7f72c00709a0 2024-03-30T04:28:19.603 INFO:tasks.workunit.client.1.smithi135.stdout:7/376: creat d4/db/d1a/d5c/d2c/d51/f7e x:0 0 0 2024-03-30T04:28:19.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:19.605+0000 7f72bf400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72c0071f50 0x7f72c01144a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:28:19.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:19.605+0000 7f72bf400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72c0071f50 0x7f72c01144a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58264/0 (socket says 172.21.15.123:58264) 2024-03-30T04:28:19.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:19.605+0000 7f72bf400700 1 -- 172.21.15.123:0/3639673155 learned_addr learned my addr 172.21.15.123:0/3639673155 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:28:19.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:19.606+0000 7f72bfe00700 1 --2- 172.21.15.123:0/3639673155 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f72c00709a0 0x7f72c0113f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:28:19.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:19.606+0000 7f72bf400700 1 -- 172.21.15.123:0/3639673155 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f72c00709a0 msgr2=0x7f72c0113f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:28:19.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:19.606+0000 7f72bf400700 1 --2- 172.21.15.123:0/3639673155 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f72c00709a0 0x7f72c0113f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:19.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:19.606+0000 7f72bf400700 1 -- 172.21.15.123:0/3639673155 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72b0005040 con 0x7f72c0071f50 2024-03-30T04:28:19.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:19.607+0000 7f72bfe00700 1 --2- 172.21.15.123:0/3639673155 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f72c00709a0 0x7f72c0113f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:28:19.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:19.607+0000 7f72bf400700 1 --2- 172.21.15.123:0/3639673155 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72c0071f50 0x7f72c01144a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f72ac002bd0 tx=0x7f72ac0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:28:19.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:19.607+0000 7f72b7e00700 1 -- 172.21.15.123:0/3639673155 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f72ac007ab0 con 0x7f72c0071f50 2024-03-30T04:28:19.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:19.607+0000 7f72c6e00700 1 -- 172.21.15.123:0/3639673155 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72c01155d0 con 0x7f72c0071f50 2024-03-30T04:28:19.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:19.608+0000 7f72b7e00700 1 -- 172.21.15.123:0/3639673155 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f72ac00f040 con 0x7f72c0071f50 2024-03-30T04:28:19.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:19.608+0000 7f72b7e00700 1 -- 172.21.15.123:0/3639673155 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f72ac009600 con 0x7f72c0071f50 2024-03-30T04:28:19.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:19.608+0000 7f72c6e00700 1 -- 172.21.15.123:0/3639673155 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72c0115b10 con 0x7f72c0071f50 2024-03-30T04:28:19.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:19.609+0000 7f72c6e00700 1 -- 172.21.15.123:0/3639673155 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72c005f5f0 con 0x7f72c0071f50 2024-03-30T04:28:19.608 INFO:tasks.workunit.client.1.smithi135.stdout:7/377: creat d4/db/d1a/d73/f7f x:0 0 0 2024-03-30T04:28:19.611 INFO:tasks.workunit.client.0.smithi123.stdout:3/836: write d13/d24/fbd [374981,53018] 0 2024-03-30T04:28:19.611 INFO:tasks.workunit.client.0.smithi123.stdout:2/900: read d0/d1f/d1c/f3f [3065244,67381] 0 2024-03-30T04:28:19.611 INFO:tasks.workunit.client.0.smithi123.stdout:8/779: write da/d27/d56/fc8 [199329,76730] 0 2024-03-30T04:28:19.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:19.613+0000 7f72b7e00700 1 -- 172.21.15.123:0/3639673155 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) v1 ==== 88766+0+0 (secure 0 0 0) 0x7f72ac009760 con 0x7f72c0071f50 2024-03-30T04:28:19.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:19.614+0000 7f72b7e00700 1 --2- 172.21.15.123:0/3639673155 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f72a406aec0 0x7f72a406d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:28:19.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:19.614+0000 7f72b7e00700 1 -- 172.21.15.123:0/3639673155 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f72ac089e70 con 0x7f72c0071f50 2024-03-30T04:28:19.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:19.615+0000 7f72bfe00700 1 --2- 172.21.15.123:0/3639673155 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f72a406aec0 0x7f72a406d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:28:19.614 INFO:tasks.workunit.client.0.smithi123.stdout:7/752: fdatasync d2/da/d22/d3b/ff6 0 2024-03-30T04:28:19.614 INFO:tasks.workunit.client.0.smithi123.stdout:5/714: link dd/d19/d3e/d5b/c70 dd/d19/d1f/ced 0 2024-03-30T04:28:19.615 INFO:tasks.workunit.client.0.smithi123.stdout:0/751: chown d6/dd/d18/d2f/cbd 265178477 1 2024-03-30T04:28:19.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:19.617+0000 7f72bfe00700 1 --2- 172.21.15.123:0/3639673155 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f72a406aec0 0x7f72a406d370 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f72b000b900 tx=0x7f72b0009e30 comp rx=0 tx=0).ready entity=mgr.14608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:28:19.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:19.618+0000 7f72b7e00700 1 -- 172.21.15.123:0/3639673155 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f72ac057cd0 con 0x7f72c0071f50 2024-03-30T04:28:19.619 INFO:tasks.workunit.client.0.smithi123.stdout:3/837: chown d13/d24/d31/d23/f76 7716398 1 2024-03-30T04:28:19.620 INFO:tasks.workunit.client.0.smithi123.stdout:2/901: readlink d0/d1f/d27/d68/dd6/d8c/lc7 0 2024-03-30T04:28:19.621 INFO:tasks.workunit.client.0.smithi123.stdout:8/780: chown da/d27/d3c/d2d/dae/cba 3 1 2024-03-30T04:28:19.623 INFO:tasks.workunit.client.1.smithi135.stdout:8/342: sync 2024-03-30T04:28:19.623 INFO:tasks.workunit.client.1.smithi135.stdout:0/434: sync 2024-03-30T04:28:19.623 INFO:tasks.workunit.client.1.smithi135.stdout:2/300: sync 2024-03-30T04:28:19.623 INFO:tasks.workunit.client.1.smithi135.stdout:5/360: sync 2024-03-30T04:28:19.624 INFO:tasks.workunit.client.1.smithi135.stdout:7/378: dwrite d4/db/de/f1d [0,4194304] 0 2024-03-30T04:28:19.625 INFO:tasks.workunit.client.0.smithi123.stdout:5/715: rename dd/d19 to dd/d19/d1f/d68/dcc/dee 22 2024-03-30T04:28:19.625 INFO:tasks.workunit.client.0.smithi123.stdout:9/727: dwrite d6/d1f/d45/d97/d69/daa/fd5 [0,4194304] 0 2024-03-30T04:28:19.626 INFO:tasks.workunit.client.1.smithi135.stdout:5/361: rmdir d1/d3 39 2024-03-30T04:28:19.629 INFO:tasks.workunit.client.0.smithi123.stdout:9/728: dread - d6/d1f/d92/db8/fbc zero size 2024-03-30T04:28:19.629 INFO:tasks.workunit.client.1.smithi135.stdout:8/343: symlink d6/d8/d15/d1f/l60 0 2024-03-30T04:28:19.630 INFO:tasks.workunit.client.0.smithi123.stdout:5/716: chown dd/d19/d1f/f27 3869 1 2024-03-30T04:28:19.631 INFO:tasks.workunit.client.0.smithi123.stdout:5/717: stat dd/d19/d1f/d68/f73 0 2024-03-30T04:28:19.633 INFO:tasks.workunit.client.0.smithi123.stdout:0/752: rename d6/dd/d15/d98/d4e/l8b to d6/dd/d15/d98/d5d/d7f/df3/lf8 0 2024-03-30T04:28:19.635 INFO:tasks.workunit.client.0.smithi123.stdout:9/729: write d6/d1f/d72/db6/fe4 [545633,3542] 0 2024-03-30T04:28:19.636 INFO:tasks.workunit.client.0.smithi123.stdout:7/753: mknod d2/da/d11/d61/d55/d59/db5/dab/dca/cf9 0 2024-03-30T04:28:19.637 INFO:tasks.workunit.client.0.smithi123.stdout:5/718: read - dd/d19/d1f/d35/fa9 zero size 2024-03-30T04:28:19.639 INFO:tasks.workunit.client.0.smithi123.stdout:0/753: write d6/dd/d31/d35/d3a/fdb [1200720,101911] 0 2024-03-30T04:28:19.639 INFO:tasks.workunit.client.0.smithi123.stdout:5/719: readlink dd/d19/l7b 0 2024-03-30T04:28:19.640 INFO:tasks.workunit.client.1.smithi135.stdout:5/362: mkdir d1/d3/d2b/d54/d6d 0 2024-03-30T04:28:19.641 INFO:tasks.workunit.client.1.smithi135.stdout:7/379: link d4/db/d1a/d5c/l6d d4/db/d1a/d5c/d2c/d2e/l80 0 2024-03-30T04:28:19.642 INFO:tasks.workunit.client.0.smithi123.stdout:9/730: chown d6/d1f/d72/lb3 1 1 2024-03-30T04:28:19.644 INFO:tasks.workunit.client.1.smithi135.stdout:5/363: fsync d1/d7/d1a/d1f/f56 0 2024-03-30T04:28:19.645 INFO:tasks.workunit.client.0.smithi123.stdout:2/902: dread d0/d1f/f2d [0,4194304] 0 2024-03-30T04:28:19.649 INFO:tasks.workunit.client.1.smithi135.stdout:5/364: fdatasync d1/f37 0 2024-03-30T04:28:19.650 INFO:tasks.workunit.client.1.smithi135.stdout:0/435: dwrite d0/d1/f29 [4194304,4194304] 0 2024-03-30T04:28:19.651 INFO:tasks.workunit.client.1.smithi135.stdout:8/344: rename d6/d8/d14/c56 to d6/d8/d15/d54/c61 0 2024-03-30T04:28:19.652 INFO:tasks.workunit.client.0.smithi123.stdout:7/754: rename d2/da/d11/d61/d55/d59/db5/f40 to d2/dc/d83/ffa 0 2024-03-30T04:28:19.654 INFO:tasks.workunit.client.1.smithi135.stdout:2/301: write d4/d32/d12/d45/f39 [771184,67949] 0 2024-03-30T04:28:19.655 INFO:tasks.workunit.client.0.smithi123.stdout:2/903: chown d0/d1f/d1c/d39/d4b/d4c/d5f 74764297 1 2024-03-30T04:28:19.656 INFO:tasks.workunit.client.1.smithi135.stdout:2/302: fdatasync d4/d32/d2e/f4b 0 2024-03-30T04:28:19.657 INFO:tasks.workunit.client.0.smithi123.stdout:9/731: creat d6/d1f/d92/ffe x:0 0 0 2024-03-30T04:28:19.659 INFO:tasks.workunit.client.1.smithi135.stdout:0/436: mkdir d0/d13/d91 0 2024-03-30T04:28:19.662 INFO:tasks.workunit.client.1.smithi135.stdout:2/303: read d4/d32/d12/d45/f2c [444788,70040] 0 2024-03-30T04:28:19.664 INFO:tasks.workunit.client.1.smithi135.stdout:5/365: rename d1/d7/d1a/d1f/f56 to d1/f6e 0 2024-03-30T04:28:19.666 INFO:tasks.workunit.client.0.smithi123.stdout:8/781: dwrite da/dc/d14/d18/f30 [8388608,4194304] 0 2024-03-30T04:28:19.667 INFO:tasks.workunit.client.0.smithi123.stdout:2/904: write d0/d1f/d1c/d39/d4b/fff [83364,109469] 0 2024-03-30T04:28:19.671 INFO:tasks.workunit.client.1.smithi135.stdout:2/304: unlink d4/da/c57 0 2024-03-30T04:28:19.674 INFO:tasks.workunit.client.1.smithi135.stdout:0/437: write d0/d19/f5c [3226974,127480] 0 2024-03-30T04:28:19.675 INFO:tasks.workunit.client.1.smithi135.stdout:3/372: sync 2024-03-30T04:28:19.676 INFO:tasks.workunit.client.1.smithi135.stdout:1/303: sync 2024-03-30T04:28:19.676 INFO:tasks.workunit.client.0.smithi123.stdout:9/732: chown d6/d1f/d45/d97/d69/dfb/f5d 117 1 2024-03-30T04:28:19.676 INFO:tasks.workunit.client.0.smithi123.stdout:8/782: mknod da/dc/da9/cf3 0 2024-03-30T04:28:19.676 INFO:tasks.workunit.client.0.smithi123.stdout:0/754: dwrite d6/dd/d31/fa9 [0,4194304] 0 2024-03-30T04:28:19.677 INFO:tasks.workunit.client.1.smithi135.stdout:6/346: sync 2024-03-30T04:28:19.677 INFO:tasks.workunit.client.1.smithi135.stdout:4/358: sync 2024-03-30T04:28:19.677 INFO:tasks.workunit.client.1.smithi135.stdout:9/326: sync 2024-03-30T04:28:19.678 INFO:tasks.workunit.client.1.smithi135.stdout:8/345: dwrite d6/d8/f31 [0,4194304] 0 2024-03-30T04:28:19.680 INFO:tasks.workunit.client.1.smithi135.stdout:2/305: creat d4/d6/f61 x:0 0 0 2024-03-30T04:28:19.681 INFO:tasks.workunit.client.1.smithi135.stdout:3/373: mkdir d3/d10/d81 0 2024-03-30T04:28:19.682 INFO:tasks.workunit.client.1.smithi135.stdout:4/359: fsync d10/d1a/d42/f54 0 2024-03-30T04:28:19.685 INFO:tasks.workunit.client.1.smithi135.stdout:4/360: chown d10/d72 76553 1 2024-03-30T04:28:19.686 INFO:tasks.workunit.client.1.smithi135.stdout:6/347: write d2/fb [810255,4433] 0 2024-03-30T04:28:19.688 INFO:tasks.workunit.client.1.smithi135.stdout:1/304: mknod d16/d19/d20/c67 0 2024-03-30T04:28:19.688 INFO:tasks.workunit.client.0.smithi123.stdout:8/783: rmdir da 39 2024-03-30T04:28:19.688 INFO:tasks.workunit.client.1.smithi135.stdout:0/438: rename d0/d5/d42/c88 to d0/d5/d9/d23/d8a/c92 0 2024-03-30T04:28:19.689 INFO:tasks.workunit.client.1.smithi135.stdout:9/327: truncate d7/d1b/d37/f38 79851 0 2024-03-30T04:28:19.690 INFO:tasks.workunit.client.0.smithi123.stdout:5/720: dwrite dd/d19/d1f/d3c/f56 [0,4194304] 0 2024-03-30T04:28:19.692 INFO:tasks.workunit.client.1.smithi135.stdout:9/328: write d7/d1b/d37/f3a [854897,128562] 0 2024-03-30T04:28:19.693 INFO:tasks.workunit.client.1.smithi135.stdout:5/366: dwrite d1/d7/f43 [0,4194304] 0 2024-03-30T04:28:19.694 INFO:tasks.workunit.client.1.smithi135.stdout:2/306: symlink d4/d6/l62 0 2024-03-30T04:28:19.695 INFO:tasks.workunit.client.0.smithi123.stdout:3/838: dread d13/d24/d31/d23/d3a/f40 [0,4194304] 0 2024-03-30T04:28:19.696 INFO:tasks.workunit.client.1.smithi135.stdout:9/329: readlink d7/d1b/l19 0 2024-03-30T04:28:19.698 INFO:tasks.workunit.client.1.smithi135.stdout:4/361: unlink d10/d1a/d42/d46/l48 0 2024-03-30T04:28:19.700 INFO:tasks.workunit.client.1.smithi135.stdout:9/330: dread - d7/d1b/d37/f54 zero size 2024-03-30T04:28:19.700 INFO:tasks.workunit.client.0.smithi123.stdout:8/784: write da/dc/d14/d18/d62/fa7 [718997,99118] 0 2024-03-30T04:28:19.702 INFO:tasks.workunit.client.1.smithi135.stdout:6/348: unlink d2/fb 0 2024-03-30T04:28:19.707 INFO:tasks.workunit.client.1.smithi135.stdout:0/439: truncate d0/d13/d53/d54/f5b 4349536 0 2024-03-30T04:28:19.708 INFO:tasks.workunit.client.1.smithi135.stdout:5/367: creat d1/d3/d2b/d54/d5e/f6f x:0 0 0 2024-03-30T04:28:19.709 INFO:tasks.workunit.client.1.smithi135.stdout:4/362: rmdir d10/d1a/d32 39 2024-03-30T04:28:19.709 INFO:tasks.workunit.client.1.smithi135.stdout:8/346: getdents d6/d8 0 2024-03-30T04:28:19.710 INFO:tasks.workunit.client.0.smithi123.stdout:8/785: unlink da/dc/c25 0 2024-03-30T04:28:19.712 INFO:tasks.workunit.client.1.smithi135.stdout:9/331: mkdir d7/d41/d72 0 2024-03-30T04:28:19.713 INFO:tasks.workunit.client.1.smithi135.stdout:6/349: creat d2/d36/f6e x:0 0 0 2024-03-30T04:28:19.714 INFO:tasks.workunit.client.0.smithi123.stdout:5/721: mknod dd/d19/d5f/cef 0 2024-03-30T04:28:19.716 INFO:tasks.workunit.client.0.smithi123.stdout:8/786: write da/d27/d3c/d2d/f57 [6076150,103707] 0 2024-03-30T04:28:19.719 INFO:tasks.workunit.client.0.smithi123.stdout:8/787: fdatasync da/d27/fd8 0 2024-03-30T04:28:19.720 INFO:tasks.workunit.client.1.smithi135.stdout:7/380: dread d4/db/d1a/d4b/d40/d3a/f58 [0,4194304] 0 2024-03-30T04:28:19.720 INFO:tasks.workunit.client.1.smithi135.stdout:9/332: fdatasync d7/d1b/d37/f38 0 2024-03-30T04:28:19.720 INFO:tasks.workunit.client.1.smithi135.stdout:7/381: fsync d4/db/f1c 0 2024-03-30T04:28:19.720 INFO:tasks.workunit.client.0.smithi123.stdout:0/755: dread d6/d52/f73 [0,4194304] 0 2024-03-30T04:28:19.723 INFO:tasks.workunit.client.0.smithi123.stdout:8/788: readlink da/d27/d3c/d2d/d74/le6 0 2024-03-30T04:28:19.727 INFO:tasks.workunit.client.0.smithi123.stdout:2/905: dread d0/fb4 [0,4194304] 0 2024-03-30T04:28:19.732 INFO:tasks.workunit.client.1.smithi135.stdout:8/347: rename d6/d8/d14/f21 to d6/d8/d14/d32/f62 0 2024-03-30T04:28:19.732 INFO:tasks.workunit.client.1.smithi135.stdout:6/350: fdatasync d2/d2e/f37 0 2024-03-30T04:28:19.735 INFO:tasks.workunit.client.1.smithi135.stdout:4/363: creat d10/d1a/d42/d66/f7e x:0 0 0 2024-03-30T04:28:19.735 INFO:tasks.workunit.client.0.smithi123.stdout:8/789: rmdir da/dc/de7 0 2024-03-30T04:28:19.735 INFO:tasks.workunit.client.0.smithi123.stdout:5/722: dread dd/d19/d1f/d35/d3a/f80 [0,4194304] 0 2024-03-30T04:28:19.737 INFO:tasks.workunit.client.0.smithi123.stdout:8/790: chown da/d27/daf 57 1 2024-03-30T04:28:19.738 INFO:tasks.workunit.client.1.smithi135.stdout:8/348: readlink d6/d8/d15/d2a/l3f 0 2024-03-30T04:28:19.740 INFO:tasks.workunit.client.1.smithi135.stdout:7/382: rename d4/db/d1a/d73 to d4/db/d1a/d5c/d2c/d51/d56/d81 0 2024-03-30T04:28:19.740 INFO:tasks.workunit.client.0.smithi123.stdout:2/906: mkdir d0/d1f/d30/d127 0 2024-03-30T04:28:19.740 INFO:tasks.workunit.client.0.smithi123.stdout:9/733: dwrite d6/d1f/f2e [0,4194304] 0 2024-03-30T04:28:19.742 INFO:tasks.workunit.client.0.smithi123.stdout:8/791: fdatasync da/dc/d14/d18/d62/fa7 0 2024-03-30T04:28:19.742 INFO:tasks.workunit.client.0.smithi123.stdout:1/808: sync 2024-03-30T04:28:19.745 INFO:tasks.workunit.client.1.smithi135.stdout:6/351: write d2/dd/d14/d38/f3e [1479716,34250] 0 2024-03-30T04:28:19.746 INFO:tasks.workunit.client.0.smithi123.stdout:5/723: creat dd/d19/d5f/dc2/ff0 x:0 0 0 2024-03-30T04:28:19.753 INFO:tasks.workunit.client.0.smithi123.stdout:0/756: dwrite d6/dd/d15/d98/d4e/d69/d6e/f9f [0,4194304] 0 2024-03-30T04:28:19.754 INFO:tasks.workunit.client.0.smithi123.stdout:8/792: fsync da/dc/d14/f71 0 2024-03-30T04:28:19.755 INFO:tasks.workunit.client.1.smithi135.stdout:7/383: mkdir d4/db/d1a/d5c/d82 0 2024-03-30T04:28:19.755 INFO:tasks.workunit.client.0.smithi123.stdout:5/724: dread - dd/d19/d1f/d68/f6b zero size 2024-03-30T04:28:19.756 INFO:tasks.workunit.client.0.smithi123.stdout:2/907: mkdir d0/d1f/d27/d68/dd6/d109/d128 0 2024-03-30T04:28:19.759 INFO:tasks.workunit.client.0.smithi123.stdout:8/793: readlink da/d27/d3c/d2d/l4a 0 2024-03-30T04:28:19.760 INFO:tasks.workunit.client.1.smithi135.stdout:7/384: chown d4/c20 6424841 1 2024-03-30T04:28:19.761 INFO:tasks.workunit.client.0.smithi123.stdout:1/809: link db/fcd db/dd/d42/d46/d56/dce/f10b 0 2024-03-30T04:28:19.762 INFO:tasks.workunit.client.1.smithi135.stdout:9/333: dwrite d7/f44 [0,4194304] 0 2024-03-30T04:28:19.762 INFO:tasks.workunit.client.0.smithi123.stdout:9/734: getdents d6/d1f/d92 0 2024-03-30T04:28:19.762 INFO:tasks.workunit.client.0.smithi123.stdout:2/908: write d0/d1f/d27/df7/f104 [3057091,37445] 0 2024-03-30T04:28:19.763 INFO:tasks.workunit.client.1.smithi135.stdout:8/349: rename d6/d8/f20 to d6/d8/d14/f63 0 2024-03-30T04:28:19.763 INFO:tasks.workunit.client.1.smithi135.stdout:0/440: dwrite d0/d5/d9/d23/d58/d55/f5f [0,4194304] 0 2024-03-30T04:28:19.764 INFO:tasks.workunit.client.0.smithi123.stdout:9/735: readlink d6/l6e 0 2024-03-30T04:28:19.767 INFO:tasks.workunit.client.0.smithi123.stdout:1/810: chown db/dd/d42/d46/d39/f52 20 1 2024-03-30T04:28:19.769 INFO:tasks.workunit.client.0.smithi123.stdout:5/725: dread dd/d19/d1f/d3c/f97 [0,4194304] 0 2024-03-30T04:28:19.771 INFO:tasks.workunit.client.0.smithi123.stdout:9/736: dread - d6/d1f/d72/d7d/dfa/ffd zero size 2024-03-30T04:28:19.772 INFO:tasks.workunit.client.0.smithi123.stdout:8/794: rename da/dc/f42 to da/d27/d56/d6c/d96/dc2/ff4 0 2024-03-30T04:28:19.773 INFO:tasks.workunit.client.1.smithi135.stdout:9/334: write d7/f4e [1344079,105736] 0 2024-03-30T04:28:19.773 INFO:tasks.workunit.client.0.smithi123.stdout:1/811: read - db/dd/d42/d46/d51/d72/ff9 zero size 2024-03-30T04:28:19.777 INFO:tasks.workunit.client.0.smithi123.stdout:2/909: creat d0/d1f/d30/d119/dbb/d97/db5/f129 x:0 0 0 2024-03-30T04:28:19.778 INFO:tasks.workunit.client.1.smithi135.stdout:6/352: unlink d2/dd/d40/d52/f5a 0 2024-03-30T04:28:19.780 INFO:tasks.workunit.client.0.smithi123.stdout:2/910: chown d0/d1f/d30/d119/f49 64530 1 2024-03-30T04:28:19.783 INFO:tasks.workunit.client.1.smithi135.stdout:9/335: dread - d7/d1b/d37/f39 zero size 2024-03-30T04:28:19.783 INFO:tasks.workunit.client.1.smithi135.stdout:4/364: dwrite d10/d1a/d42/d46/f6b [0,4194304] 0 2024-03-30T04:28:19.783 INFO:tasks.workunit.client.0.smithi123.stdout:1/812: mkdir db/dd/d42/d46/d56/df6/d10c 0 2024-03-30T04:28:19.785 INFO:tasks.workunit.client.1.smithi135.stdout:7/385: mkdir d4/db/d1a/d5c/d2c/d83 0 2024-03-30T04:28:19.786 INFO:tasks.workunit.client.1.smithi135.stdout:6/353: mknod d2/d36/c6f 0 2024-03-30T04:28:19.786 INFO:tasks.workunit.client.1.smithi135.stdout:0/441: mknod d0/c93 0 2024-03-30T04:28:19.789 INFO:tasks.workunit.client.1.smithi135.stdout:4/365: fdatasync d10/f14 0 2024-03-30T04:28:19.791 INFO:tasks.workunit.client.1.smithi135.stdout:7/386: stat d4/db/de/f1d 0 2024-03-30T04:28:19.792 INFO:tasks.workunit.client.1.smithi135.stdout:9/336: mknod d7/d1b/c73 0 2024-03-30T04:28:19.795 INFO:tasks.workunit.client.0.smithi123.stdout:1/813: rename db/dd/d42/d46/d51/fb1 to db/dd/d42/d46/d51/d72/f10d 0 2024-03-30T04:28:19.798 INFO:tasks.workunit.client.1.smithi135.stdout:6/354: creat d2/d2e/f70 x:0 0 0 2024-03-30T04:28:19.798 INFO:tasks.workunit.client.1.smithi135.stdout:5/368: dread d1/d3/d2b/f53 [0,4194304] 0 2024-03-30T04:28:19.802 INFO:tasks.workunit.client.1.smithi135.stdout:7/387: write d4/db/d1a/d5c/d2c/f39 [5541692,117125] 0 2024-03-30T04:28:19.804 INFO:tasks.workunit.client.1.smithi135.stdout:7/388: stat d4/db 0 2024-03-30T04:28:19.806 INFO:tasks.workunit.client.0.smithi123.stdout:3/839: dread d13/d24/d31/d23/d3a/d3c/f73 [0,4194304] 0 2024-03-30T04:28:19.806 INFO:tasks.workunit.client.0.smithi123.stdout:9/737: dread d6/d58/f5c [0,4194304] 0 2024-03-30T04:28:19.807 INFO:tasks.workunit.client.0.smithi123.stdout:8/795: dwrite da/d27/d95/fab [0,4194304] 0 2024-03-30T04:28:19.808 INFO:tasks.workunit.client.1.smithi135.stdout:7/389: fdatasync d4/db/de/d5a/f6c 0 2024-03-30T04:28:19.810 INFO:tasks.workunit.client.1.smithi135.stdout:0/442: symlink d0/d13/d53/d7f/d83/l94 0 2024-03-30T04:28:19.814 INFO:tasks.workunit.client.1.smithi135.stdout:9/337: creat d7/d41/d72/f74 x:0 0 0 2024-03-30T04:28:19.815 INFO:tasks.workunit.client.0.smithi123.stdout:9/738: symlink d6/dad/lff 0 2024-03-30T04:28:19.815 INFO:tasks.workunit.client.0.smithi123.stdout:8/796: readlink da/dc/l26 0 2024-03-30T04:28:19.816 INFO:tasks.workunit.client.1.smithi135.stdout:3/374: write d3/f76 [936900,91330] 0 2024-03-30T04:28:19.816 INFO:tasks.workunit.client.1.smithi135.stdout:4/366: dread d10/d1a/d42/d46/f6b [0,4194304] 0 2024-03-30T04:28:19.820 INFO:tasks.workunit.client.0.smithi123.stdout:3/840: truncate d13/d24/d31/d23/d3a/d3c/d4e/d5d/da5/fdb 890544 0 2024-03-30T04:28:19.821 INFO:tasks.workunit.client.1.smithi135.stdout:0/443: dread - d0/d19/d2d/d79/f81 zero size 2024-03-30T04:28:19.822 INFO:tasks.workunit.client.0.smithi123.stdout:6/811: sync 2024-03-30T04:28:19.822 INFO:tasks.workunit.client.0.smithi123.stdout:4/732: sync 2024-03-30T04:28:19.823 INFO:tasks.workunit.client.0.smithi123.stdout:0/757: truncate d6/ff1 176488 0 2024-03-30T04:28:19.828 INFO:tasks.workunit.client.0.smithi123.stdout:9/739: creat d6/d1f/d45/d97/d69/daa/dac/f100 x:0 0 0 2024-03-30T04:28:19.830 INFO:tasks.workunit.client.1.smithi135.stdout:0/444: creat d0/d5/d9/d23/d58/d63/f95 x:0 0 0 2024-03-30T04:28:19.833 INFO:tasks.workunit.client.1.smithi135.stdout:7/390: dread d4/db/de/f2b [0,4194304] 0 2024-03-30T04:28:19.833 INFO:tasks.workunit.client.1.smithi135.stdout:6/355: rename d2/ce to d2/c71 0 2024-03-30T04:28:19.837 INFO:tasks.workunit.client.0.smithi123.stdout:3/841: mknod d13/d24/d31/d23/d3a/d3c/db8/df4/c111 0 2024-03-30T04:28:19.837 INFO:tasks.workunit.client.1.smithi135.stdout:3/375: mknod d3/d50/d22/d33/d7b/c82 0 2024-03-30T04:28:19.838 INFO:tasks.workunit.client.0.smithi123.stdout:0/758: dread - d6/dd/d18/d2f/fac zero size 2024-03-30T04:28:19.838 INFO:tasks.workunit.client.0.smithi123.stdout:5/726: write dd/d19/d3e/d5b/d62/fa4 [56782,74172] 0 2024-03-30T04:28:19.839 INFO:tasks.workunit.client.1.smithi135.stdout:9/338: dwrite d7/f26 [0,4194304] 0 2024-03-30T04:28:19.839 INFO:tasks.workunit.client.1.smithi135.stdout:4/367: symlink d10/d72/d7c/l7f 0 2024-03-30T04:28:19.844 INFO:tasks.workunit.client.0.smithi123.stdout:0/759: dread - d6/dd/d18/d2f/fc5 zero size 2024-03-30T04:28:19.845 INFO:tasks.workunit.client.0.smithi123.stdout:3/842: write d13/d24/d31/d23/fc8 [480825,49719] 0 2024-03-30T04:28:19.846 INFO:tasks.workunit.client.1.smithi135.stdout:7/391: symlink d4/db/de/d5a/l84 0 2024-03-30T04:28:19.848 INFO:tasks.workunit.client.1.smithi135.stdout:0/445: symlink d0/d13/d53/l96 0 2024-03-30T04:28:19.848 INFO:tasks.workunit.client.0.smithi123.stdout:8/797: dwrite da/dc/f10 [0,4194304] 0 2024-03-30T04:28:19.849 INFO:tasks.workunit.client.1.smithi135.stdout:7/392: write d4/db/f1c [1753828,17058] 0 2024-03-30T04:28:19.850 INFO:tasks.workunit.client.1.smithi135.stdout:2/307: dwrite d4/d32/d12/f31 [0,4194304] 0 2024-03-30T04:28:19.851 INFO:tasks.workunit.client.1.smithi135.stdout:1/305: sync 2024-03-30T04:28:19.852 INFO:tasks.workunit.client.1.smithi135.stdout:6/356: rmdir d2/dd/d14/d2b 39 2024-03-30T04:28:19.855 INFO:tasks.workunit.client.0.smithi123.stdout:2/911: dread d0/d1f/d16/f1e [0,4194304] 0 2024-03-30T04:28:19.858 INFO:tasks.workunit.client.1.smithi135.stdout:3/376: write d3/d50/d46/f73 [508779,58276] 0 2024-03-30T04:28:19.860 INFO:tasks.workunit.client.1.smithi135.stdout:7/393: write d4/db/d1a/d5c/d2c/d51/d56/d81/f7f [839248,93162] 0 2024-03-30T04:28:19.861 INFO:tasks.workunit.client.0.smithi123.stdout:1/814: dread db/f60 [0,4194304] 0 2024-03-30T04:28:19.861 INFO:tasks.workunit.client.1.smithi135.stdout:9/339: symlink d7/d4f/d5c/d63/l75 0 2024-03-30T04:28:19.862 INFO:tasks.workunit.client.1.smithi135.stdout:7/394: stat d4/f11 0 2024-03-30T04:28:19.865 INFO:tasks.workunit.client.0.smithi123.stdout:2/912: chown d0/d1f/d30/d119/dbb/d97/fd8 53090317 1 2024-03-30T04:28:19.867 INFO:tasks.workunit.client.1.smithi135.stdout:1/306: creat d16/d29/d2c/d43/d50/f68 x:0 0 0 2024-03-30T04:28:19.869 INFO:tasks.workunit.client.0.smithi123.stdout:2/913: chown d0/d1f/d27/d68/dd6/d99/cc2 15536 1 2024-03-30T04:28:19.870 INFO:tasks.workunit.client.0.smithi123.stdout:0/760: creat d6/dd/d15/d98/dc6/ff9 x:0 0 0 2024-03-30T04:28:19.871 INFO:tasks.workunit.client.0.smithi123.stdout:2/914: stat d0/d1f/d16/c48 0 2024-03-30T04:28:19.871 INFO:tasks.workunit.client.1.smithi135.stdout:4/368: dread d10/f73 [0,4194304] 0 2024-03-30T04:28:19.871 INFO:tasks.workunit.client.1.smithi135.stdout:2/308: rename d4/d6/f36 to d4/d32/d43/f63 0 2024-03-30T04:28:19.872 INFO:tasks.workunit.client.1.smithi135.stdout:5/369: dread d1/d3/d2b/f5b [0,4194304] 0 2024-03-30T04:28:19.872 INFO:tasks.workunit.client.1.smithi135.stdout:6/357: write d2/dd/d14/d2b/f47 [3159263,49436] 0 2024-03-30T04:28:19.874 INFO:tasks.workunit.client.0.smithi123.stdout:9/740: dread d6/d1f/d45/d97/d69/dfb/f40 [8388608,4194304] 0 2024-03-30T04:28:19.875 INFO:tasks.workunit.client.0.smithi123.stdout:7/755: sync 2024-03-30T04:28:19.875 INFO:tasks.workunit.client.0.smithi123.stdout:5/727: write dd/f26 [177923,59924] 0 2024-03-30T04:28:19.876 INFO:tasks.workunit.client.0.smithi123.stdout:1/815: symlink db/dd/l10e 0 2024-03-30T04:28:19.876 INFO:tasks.workunit.client.1.smithi135.stdout:9/340: fsync d7/d1b/f3d 0 2024-03-30T04:28:19.879 INFO:tasks.workunit.client.1.smithi135.stdout:2/309: stat d4/d32/d12/f3f 0 2024-03-30T04:28:19.881 INFO:tasks.workunit.client.1.smithi135.stdout:1/307: symlink d16/d29/d2c/d3c/l69 0 2024-03-30T04:28:19.881 INFO:tasks.workunit.client.0.smithi123.stdout:0/761: write d6/dd/d15/d28/dd3/fd6 [1688427,30366] 0 2024-03-30T04:28:19.883 INFO:tasks.workunit.client.1.smithi135.stdout:2/310: chown d4/d32/d12/l29 0 1 2024-03-30T04:28:19.884 INFO:tasks.workunit.client.1.smithi135.stdout:2/311: readlink d4/d6/l27 0 2024-03-30T04:28:19.885 INFO:tasks.workunit.client.0.smithi123.stdout:4/733: dwrite d2/d4/d7/fe [4194304,4194304] 0 2024-03-30T04:28:19.887 INFO:tasks.workunit.client.0.smithi123.stdout:2/915: rename d0/d1f/d30/d119/dbb/lc0 to d0/d1f/d27/d68/dd6/l12a 0 2024-03-30T04:28:19.891 INFO:tasks.workunit.client.0.smithi123.stdout:9/741: read d6/d1f/d45/d97/d69/daa/fd5 [3441615,70089] 0 2024-03-30T04:28:19.892 INFO:tasks.workunit.client.1.smithi135.stdout:9/341: rename f6 to d7/d4f/d6e/f76 0 2024-03-30T04:28:19.894 INFO:tasks.workunit.client.1.smithi135.stdout:5/370: mkdir d1/da/d17/d70 0 2024-03-30T04:28:19.895 INFO:tasks.workunit.client.0.smithi123.stdout:5/728: fdatasync dd/d19/d3e/f63 0 2024-03-30T04:28:19.896 INFO:tasks.workunit.client.0.smithi123.stdout:6/812: dread d10/d82/d8e/ffe [0,4194304] 0 2024-03-30T04:28:19.896 INFO:tasks.workunit.client.0.smithi123.stdout:8/798: dwrite da/dc/d14/d18/d21/da2/fc5 [0,4194304] 0 2024-03-30T04:28:19.901 INFO:tasks.workunit.client.1.smithi135.stdout:1/308: mknod d16/d19/c6a 0 2024-03-30T04:28:19.902 INFO:tasks.workunit.client.0.smithi123.stdout:2/916: write d0/d1f/d1c/d40/fac [22166,83324] 0 2024-03-30T04:28:19.902 INFO:tasks.workunit.client.1.smithi135.stdout:5/371: fsync d1/d55/f57 0 2024-03-30T04:28:19.904 INFO:tasks.workunit.client.1.smithi135.stdout:9/342: chown d7/d1b/d37/f4a 355 1 2024-03-30T04:28:19.904 INFO:tasks.workunit.client.0.smithi123.stdout:3/843: write d13/d24/d39/d79/d90/fd4 [3933768,95501] 0 2024-03-30T04:28:19.904 INFO:tasks.workunit.client.0.smithi123.stdout:7/756: rmdir d2/da/d11/d61/d55/d59/db5/dab/dca 39 2024-03-30T04:28:19.907 INFO:tasks.workunit.client.0.smithi123.stdout:4/734: dread - d2/d4/d2c/d42/d35/d9b/fb6 zero size 2024-03-30T04:28:19.908 INFO:tasks.workunit.client.0.smithi123.stdout:9/742: read d6/d49/f75 [1448123,63787] 0 2024-03-30T04:28:19.909 INFO:tasks.workunit.client.0.smithi123.stdout:0/762: write d6/dd/d15/d98/d4e/d69/f81 [2078302,108320] 0 2024-03-30T04:28:19.912 INFO:tasks.workunit.client.0.smithi123.stdout:1/816: rename db/dd/d42/d8f/d78/f91 to db/dd/d42/d46/d56/da2/f10f 0 2024-03-30T04:28:19.913 INFO:tasks.workunit.client.1.smithi135.stdout:9/343: chown d7/d1b/ld 211 1 2024-03-30T04:28:19.913 INFO:tasks.workunit.client.1.smithi135.stdout:6/358: creat d2/dd/d14/f72 x:0 0 0 2024-03-30T04:28:19.913 INFO:tasks.workunit.client.1.smithi135.stdout:8/350: dwrite d6/d8/d15/d54/f58 [0,4194304] 0 2024-03-30T04:28:19.915 INFO:tasks.workunit.client.0.smithi123.stdout:7/757: chown d2/dc/d7f/fa3 119439044 1 2024-03-30T04:28:19.919 INFO:tasks.workunit.client.0.smithi123.stdout:5/729: stat dd/d19/d1f/d35/d3a/d79/lce 0 2024-03-30T04:28:19.921 INFO:tasks.workunit.client.1.smithi135.stdout:6/359: stat d2/dd/c19 0 2024-03-30T04:28:19.922 INFO:tasks.workunit.client.1.smithi135.stdout:7/395: dwrite d4/db/d1a/d5c/d5d/f7b [4194304,4194304] 0 2024-03-30T04:28:19.923 INFO:tasks.workunit.client.0.smithi123.stdout:4/735: mkdir d2/d4/d2c/d42/d35/d5d/d6f/df9 0 2024-03-30T04:28:19.924 INFO:tasks.workunit.client.1.smithi135.stdout:6/360: chown d2/dd/d40 2 1 2024-03-30T04:28:19.928 INFO:tasks.workunit.client.1.smithi135.stdout:5/372: creat d1/d3/d2b/d54/f71 x:0 0 0 2024-03-30T04:28:19.930 INFO:tasks.workunit.client.0.smithi123.stdout:5/730: fdatasync dd/d19/d1f/d35/d3a/d57/db3/fdb 0 2024-03-30T04:28:19.930 INFO:tasks.workunit.client.0.smithi123.stdout:8/799: creat da/d27/d56/d6c/dd4/ff5 x:0 0 0 2024-03-30T04:28:19.935 INFO:tasks.workunit.client.1.smithi135.stdout:2/312: dwrite d4/d32/d12/f31 [0,4194304] 0 2024-03-30T04:28:19.937 INFO:tasks.workunit.client.0.smithi123.stdout:3/844: symlink d13/d24/d31/d23/d3a/d3c/d4e/d5d/da5/dcd/l112 0 2024-03-30T04:28:19.944 INFO:tasks.workunit.client.0.smithi123.stdout:7/758: creat d2/da/d11/d61/d55/d59/ffb x:0 0 0 2024-03-30T04:28:19.944 INFO:tasks.workunit.client.0.smithi123.stdout:5/731: symlink dd/d41/lf1 0 2024-03-30T04:28:19.946 INFO:tasks.workunit.client.1.smithi135.stdout:8/351: unlink d6/d8/d14/f2e 0 2024-03-30T04:28:19.948 INFO:tasks.workunit.client.0.smithi123.stdout:0/763: dwrite d6/dd/d18/d1f/f9e [0,4194304] 0 2024-03-30T04:28:19.948 INFO:tasks.workunit.client.0.smithi123.stdout:1/817: creat db/dd/d5f/d6d/d85/d89/d90/f110 x:0 0 0 2024-03-30T04:28:19.949 INFO:tasks.workunit.client.1.smithi135.stdout:0/446: dread d0/d13/d1e/d3a/f61 [0,4194304] 0 2024-03-30T04:28:19.952 INFO:tasks.workunit.client.1.smithi135.stdout:8/352: fdatasync d6/d23/f27 0 2024-03-30T04:28:19.954 INFO:tasks.workunit.client.0.smithi123.stdout:3/845: creat d13/d24/d31/d23/f113 x:0 0 0 2024-03-30T04:28:19.954 INFO:tasks.workunit.client.0.smithi123.stdout:9/743: creat d6/d1f/d72/f101 x:0 0 0 2024-03-30T04:28:19.956 INFO:tasks.workunit.client.1.smithi135.stdout:1/309: getdents d16/d29/d32 0 2024-03-30T04:28:19.958 INFO:tasks.workunit.client.1.smithi135.stdout:7/396: symlink d4/db/d1a/d5c/d2c/d51/l85 0 2024-03-30T04:28:19.958 INFO:tasks.workunit.client.0.smithi123.stdout:7/759: unlink d2/da/d11/d61/d55/d59/f8b 0 2024-03-30T04:28:19.959 INFO:tasks.workunit.client.0.smithi123.stdout:5/732: dread dd/d19/d1f/d35/d3a/d79/f9b [0,4194304] 0 2024-03-30T04:28:19.961 INFO:tasks.workunit.client.1.smithi135.stdout:6/361: creat d2/dd/d18/f73 x:0 0 0 2024-03-30T04:28:19.964 INFO:tasks.workunit.client.0.smithi123.stdout:7/760: stat d2/da/d11/d34/ded 0 2024-03-30T04:28:19.964 INFO:tasks.workunit.client.1.smithi135.stdout:2/313: rmdir d4/d32/d12/d45/d3c 39 2024-03-30T04:28:19.966 INFO:tasks.workunit.client.1.smithi135.stdout:1/310: chown d16/d19/f56 206 1 2024-03-30T04:28:19.967 INFO:tasks.workunit.client.0.smithi123.stdout:6/813: dread d10/f97 [0,4194304] 0 2024-03-30T04:28:19.968 INFO:tasks.workunit.client.1.smithi135.stdout:3/377: dread d3/d10/f31 [0,4194304] 0 2024-03-30T04:28:19.969 INFO:tasks.workunit.client.0.smithi123.stdout:0/764: unlink d6/dd/d15/d28/d4b/f91 0 2024-03-30T04:28:19.969 INFO:tasks.workunit.client.0.smithi123.stdout:9/744: creat d6/d7c/d82/f102 x:0 0 0 2024-03-30T04:28:19.972 INFO:tasks.workunit.client.0.smithi123.stdout:5/733: symlink dd/d19/d1f/d35/d3a/d57/de0/lf2 0 2024-03-30T04:28:19.973 INFO:tasks.workunit.client.1.smithi135.stdout:8/353: mknod d6/d8/d15/c64 0 2024-03-30T04:28:19.974 INFO:tasks.workunit.client.0.smithi123.stdout:7/761: mkdir d2/da/d11/d34/ded/dfc 0 2024-03-30T04:28:19.977 INFO:tasks.workunit.client.1.smithi135.stdout:2/314: creat d4/d32/d2e/d2f/f64 x:0 0 0 2024-03-30T04:28:19.977 INFO:tasks.workunit.client.1.smithi135.stdout:4/369: dwrite d10/d1a/d3d/f5a [0,4194304] 0 2024-03-30T04:28:19.979 INFO:tasks.workunit.client.1.smithi135.stdout:5/373: dwrite d1/d55/f6c [0,4194304] 0 2024-03-30T04:28:19.985 INFO:tasks.workunit.client.1.smithi135.stdout:8/354: write d6/d8/d14/f4e [627273,7963] 0 2024-03-30T04:28:19.985 INFO:tasks.workunit.client.1.smithi135.stdout:2/315: chown d4/d6/f61 110 1 2024-03-30T04:28:19.986 INFO:tasks.workunit.client.1.smithi135.stdout:1/311: rename d16/d29/d32/d55 to d16/d5a/d6b 0 2024-03-30T04:28:19.990 INFO:tasks.workunit.client.1.smithi135.stdout:5/374: write d1/d7/f2c [770242,58872] 0 2024-03-30T04:28:19.990 INFO:tasks.workunit.client.1.smithi135.stdout:4/370: symlink d10/d1a/d3d/l80 0 2024-03-30T04:28:19.990 INFO:tasks.workunit.client.0.smithi123.stdout:2/917: write d0/d1f/d30/f72 [47021,74059] 0 2024-03-30T04:28:19.992 INFO:tasks.workunit.client.0.smithi123.stdout:0/765: creat d6/dd/d15/d28/d4b/d8a/ffa x:0 0 0 2024-03-30T04:28:19.997 INFO:tasks.workunit.client.0.smithi123.stdout:1/818: getdents db/dd/d42/d46/d5d/d7a 0 2024-03-30T04:28:20.002 INFO:tasks.workunit.client.0.smithi123.stdout:8/800: write da/d27/d56/d6c/d6d/f79 [1558380,55372] 0 2024-03-30T04:28:20.009 INFO:tasks.workunit.client.1.smithi135.stdout:7/397: dread d4/db/d1a/d5c/d2c/d51/d56/d81/f7f [0,4194304] 0 2024-03-30T04:28:20.009 INFO:tasks.workunit.client.1.smithi135.stdout:8/355: rename d6/d8/d14/cf to d6/d23/c65 0 2024-03-30T04:28:20.009 INFO:tasks.workunit.client.0.smithi123.stdout:6/814: rmdir d10/d82/dc1/d6a/d7a/de8/dee 0 2024-03-30T04:28:20.012 INFO:tasks.workunit.client.1.smithi135.stdout:0/447: dwrite d0/d1/d16/f32 [4194304,4194304] 0 2024-03-30T04:28:20.014 INFO:tasks.workunit.client.1.smithi135.stdout:7/398: fdatasync d4/f11 0 2024-03-30T04:28:20.015 INFO:tasks.workunit.client.1.smithi135.stdout:6/362: dread d2/dd/d18/f22 [0,4194304] 0 2024-03-30T04:28:20.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.016+0000 7f72c6e00700 1 -- 172.21.15.123:0/3639673155 --> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f72c0115f00 con 0x7f72a406aec0 2024-03-30T04:28:20.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.020+0000 7f72b7e00700 1 -- 172.21.15.123:0/3639673155 <== mgr.14608 v2:172.21.15.135:6828/3601553667 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f72c0115f00 con 0x7f72a406aec0 2024-03-30T04:28:20.019 INFO:tasks.workunit.client.0.smithi123.stdout:8/801: creat da/d27/ff6 x:0 0 0 2024-03-30T04:28:20.021 INFO:tasks.workunit.client.1.smithi135.stdout:3/378: truncate d3/d50/f12 117431 0 2024-03-30T04:28:20.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.022+0000 7f72c6e00700 1 -- 172.21.15.123:0/3639673155 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f72a406aec0 msgr2=0x7f72a406d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:28:20.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.022+0000 7f72c6e00700 1 --2- 172.21.15.123:0/3639673155 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f72a406aec0 0x7f72a406d370 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f72b000b900 tx=0x7f72b0009e30 comp rx=0 tx=0).stop 2024-03-30T04:28:20.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.022+0000 7f72c6e00700 1 -- 172.21.15.123:0/3639673155 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72c0071f50 msgr2=0x7f72c01144a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:28:20.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.022+0000 7f72c6e00700 1 --2- 172.21.15.123:0/3639673155 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72c0071f50 0x7f72c01144a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f72ac002bd0 tx=0x7f72ac0046b0 comp rx=0 tx=0).stop 2024-03-30T04:28:20.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.023+0000 7f72c6e00700 1 -- 172.21.15.123:0/3639673155 shutdown_connections 2024-03-30T04:28:20.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.023+0000 7f72c6e00700 1 --2- 172.21.15.123:0/3639673155 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f72a406aec0 0x7f72a406d370 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:20.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.023+0000 7f72c6e00700 1 --2- 172.21.15.123:0/3639673155 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f72c00709a0 0x7f72c0113f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:20.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.023+0000 7f72c6e00700 1 --2- 172.21.15.123:0/3639673155 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72c0071f50 0x7f72c01144a0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:20.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.023+0000 7f72c6e00700 1 -- 172.21.15.123:0/3639673155 >> 172.21.15.123:0/3639673155 conn(0x7f72c00f8560 msgr2=0x7f72c00f9d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:28:20.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.023+0000 7f72c6e00700 1 -- 172.21.15.123:0/3639673155 shutdown_connections 2024-03-30T04:28:20.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.023+0000 7f72c6e00700 1 -- 172.21.15.123:0/3639673155 wait complete. 2024-03-30T04:28:20.023 INFO:tasks.workunit.client.0.smithi123.stdout:6/815: chown d10/d82/dd6/ldb 1219608 1 2024-03-30T04:28:20.023 INFO:tasks.workunit.client.0.smithi123.stdout:8/802: readlink da/d27/d56/d6c/d6d/lbf 0 2024-03-30T04:28:20.023 INFO:tasks.workunit.client.0.smithi123.stdout:2/918: unlink d0/d1f/d30/c31 0 2024-03-30T04:28:20.023 INFO:tasks.workunit.client.1.smithi135.stdout:6/363: fsync d2/f12 0 2024-03-30T04:28:20.024 INFO:tasks.workunit.client.1.smithi135.stdout:0/448: write d0/d5/d9/d23/d58/d63/f95 [36422,31064] 0 2024-03-30T04:28:20.028 INFO:tasks.workunit.client.0.smithi123.stdout:3/846: write d13/d24/d31/d23/d3a/f54 [729984,75115] 0 2024-03-30T04:28:20.028 INFO:tasks.workunit.client.1.smithi135.stdout:8/356: mknod d6/d8/c66 0 2024-03-30T04:28:20.028 INFO:tasks.workunit.client.1.smithi135.stdout:9/344: dread d7/d1b/f16 [0,4194304] 0 2024-03-30T04:28:20.028 INFO:tasks.workunit.client.1.smithi135.stdout:2/316: dwrite d4/d32/d2e/f48 [0,4194304] 0 2024-03-30T04:28:20.031 INFO:tasks.workunit.client.1.smithi135.stdout:9/345: write d7/d41/f59 [43458,88469] 0 2024-03-30T04:28:20.033 INFO:tasks.workunit.client.0.smithi123.stdout:0/766: creat d6/dd/d15/d98/d4e/d69/ffb x:0 0 0 2024-03-30T04:28:20.035 INFO:tasks.workunit.client.0.smithi123.stdout:7/762: getdents d2/d4d/dd0 0 2024-03-30T04:28:20.035 INFO:tasks.workunit.client.0.smithi123.stdout:8/803: write da/d99/fb6 [11342629,79682] 0 2024-03-30T04:28:20.038 INFO:tasks.workunit.client.0.smithi123.stdout:0/767: chown d6/dd/d31/d35/d3a 32792 1 2024-03-30T04:28:20.038 INFO:tasks.workunit.client.0.smithi123.stdout:2/919: symlink d0/d1f/d27/l12b 0 2024-03-30T04:28:20.038 INFO:tasks.workunit.client.1.smithi135.stdout:7/399: truncate d4/db/d1a/d4b/f31 4023868 0 2024-03-30T04:28:20.041 INFO:tasks.workunit.client.1.smithi135.stdout:4/371: dread d10/d1a/f3a [0,4194304] 0 2024-03-30T04:28:20.041 INFO:tasks.workunit.client.1.smithi135.stdout:6/364: dread d2/dd/d18/f24 [0,4194304] 0 2024-03-30T04:28:20.042 INFO:tasks.workunit.client.0.smithi123.stdout:3/847: mkdir d13/d24/d31/d23/d3a/d3c/d114 0 2024-03-30T04:28:20.044 INFO:tasks.workunit.client.1.smithi135.stdout:3/379: mkdir d3/d10/d16/d41/d83 0 2024-03-30T04:28:20.045 INFO:tasks.workunit.client.1.smithi135.stdout:2/317: mknod d4/d32/d2e/d2f/c65 0 2024-03-30T04:28:20.046 INFO:tasks.workunit.client.0.smithi123.stdout:8/804: creat da/d99/ff7 x:0 0 0 2024-03-30T04:28:20.046 INFO:tasks.workunit.client.0.smithi123.stdout:0/768: fsync d6/dd/d31/d35/d3a/f95 0 2024-03-30T04:28:20.047 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:28:20.048 INFO:tasks.workunit.client.0.smithi123.stdout:2/920: creat d0/d1f/d30/d119/dbb/d97/d116/d11c/f12c x:0 0 0 2024-03-30T04:28:20.049 INFO:tasks.workunit.client.1.smithi135.stdout:7/400: unlink d4/db/f69 0 2024-03-30T04:28:20.054 INFO:tasks.workunit.client.1.smithi135.stdout:7/401: chown d4/db/c65 16535617 1 2024-03-30T04:28:20.055 INFO:tasks.workunit.client.1.smithi135.stdout:4/372: symlink d10/d1a/l81 0 2024-03-30T04:28:20.056 INFO:tasks.workunit.client.0.smithi123.stdout:9/745: truncate d6/d1f/d72/fc3 112367 0 2024-03-30T04:28:20.057 INFO:tasks.workunit.client.0.smithi123.stdout:3/848: truncate d13/d24/d31/d30/f5a 1439329 0 2024-03-30T04:28:20.058 INFO:tasks.workunit.client.0.smithi123.stdout:3/849: stat fe 0 2024-03-30T04:28:20.058 INFO:tasks.workunit.client.0.smithi123.stdout:5/734: dwrite dd/d19/d3e/f75 [0,4194304] 0 2024-03-30T04:28:20.059 INFO:tasks.workunit.client.1.smithi135.stdout:0/449: rename d0/d5/d9/d23/d58/d55/f6b to d0/d13/d53/f97 0 2024-03-30T04:28:20.060 INFO:tasks.workunit.client.0.smithi123.stdout:3/850: write fd [2685589,5224] 0 2024-03-30T04:28:20.062 INFO:tasks.workunit.client.1.smithi135.stdout:8/357: creat d6/d8/d14/f67 x:0 0 0 2024-03-30T04:28:20.063 INFO:tasks.workunit.client.0.smithi123.stdout:8/805: creat da/d27/d3c/d64/d78/d83/ff8 x:0 0 0 2024-03-30T04:28:20.065 INFO:tasks.workunit.client.1.smithi135.stdout:7/402: truncate d4/db/de/f2b 1086949 0 2024-03-30T04:28:20.070 INFO:tasks.workunit.client.0.smithi123.stdout:5/735: readlink dd/d19/d1f/d35/d3a/l5a 0 2024-03-30T04:28:20.070 INFO:tasks.workunit.client.0.smithi123.stdout:0/769: mknod d6/dd/d15/d98/d4e/cfc 0 2024-03-30T04:28:20.074 INFO:tasks.workunit.client.1.smithi135.stdout:4/373: creat d10/d1a/d42/d46/f82 x:0 0 0 2024-03-30T04:28:20.074 INFO:tasks.workunit.client.1.smithi135.stdout:3/380: symlink d3/d10/d16/d41/d83/l84 0 2024-03-30T04:28:20.076 INFO:tasks.workunit.client.1.smithi135.stdout:2/318: rename d4/d6/c4f to d4/d32/d12/d45/d3c/c66 0 2024-03-30T04:28:20.079 INFO:tasks.workunit.client.1.smithi135.stdout:6/365: dwrite d2/dd/d14/d38/f4a [4194304,4194304] 0 2024-03-30T04:28:20.081 INFO:tasks.workunit.client.0.smithi123.stdout:2/921: getdents d0/d1f/d16/df3 0 2024-03-30T04:28:20.082 INFO:tasks.workunit.client.0.smithi123.stdout:8/806: symlink da/dc/d14/db7/lf9 0 2024-03-30T04:28:20.082 INFO:tasks.workunit.client.1.smithi135.stdout:1/312: dread d16/f42 [0,4194304] 0 2024-03-30T04:28:20.083 INFO:tasks.workunit.client.1.smithi135.stdout:0/450: creat d0/d19/d2d/f98 x:0 0 0 2024-03-30T04:28:20.085 INFO:tasks.workunit.client.0.smithi123.stdout:8/807: stat da/d27/d56/cc0 0 2024-03-30T04:28:20.086 INFO:tasks.workunit.client.1.smithi135.stdout:9/346: dwrite d7/d1b/d37/f4a [0,4194304] 0 2024-03-30T04:28:20.089 INFO:tasks.workunit.client.0.smithi123.stdout:5/736: dread - dd/d19/d1f/d35/d3a/d57/db3/fdb zero size 2024-03-30T04:28:20.089 INFO:tasks.workunit.client.0.smithi123.stdout:3/851: read d13/d24/d31/d23/d3a/fa4 [483076,63121] 0 2024-03-30T04:28:20.090 INFO:tasks.workunit.client.1.smithi135.stdout:9/347: chown d7/d41 2 1 2024-03-30T04:28:20.091 INFO:tasks.workunit.client.1.smithi135.stdout:3/381: mknod d3/d50/d46/d4c/c85 0 2024-03-30T04:28:20.092 INFO:tasks.workunit.client.1.smithi135.stdout:8/358: fdatasync d6/d8/d15/d2a/f3c 0 2024-03-30T04:28:20.092 INFO:tasks.workunit.client.1.smithi135.stdout:2/319: fsync d4/da/f1e 0 2024-03-30T04:28:20.093 INFO:tasks.workunit.client.0.smithi123.stdout:1/819: dwrite db/dd/d42/d46/d39/fad [0,4194304] 0 2024-03-30T04:28:20.096 INFO:tasks.workunit.client.0.smithi123.stdout:7/763: dwrite d2/da/d11/fc6 [0,4194304] 0 2024-03-30T04:28:20.097 INFO:tasks.workunit.client.1.smithi135.stdout:0/451: fsync d0/d5/d9/d23/d58/d63/f95 0 2024-03-30T04:28:20.098 INFO:tasks.workunit.client.1.smithi135.stdout:6/366: truncate d2/dd/d14/f15 1658585 0 2024-03-30T04:28:20.100 INFO:tasks.workunit.client.1.smithi135.stdout:8/359: chown d6/d8/d15/d54/f4a 38005 1 2024-03-30T04:28:20.103 INFO:tasks.workunit.client.1.smithi135.stdout:4/374: link d10/d1a/d42/f62 d10/d72/d7c/f83 0 2024-03-30T04:28:20.105 INFO:tasks.workunit.client.0.smithi123.stdout:6/816: dwrite fe [0,4194304] 0 2024-03-30T04:28:20.106 INFO:tasks.workunit.client.1.smithi135.stdout:8/360: chown d6/fa 6256145 1 2024-03-30T04:28:20.106 INFO:tasks.workunit.client.1.smithi135.stdout:7/403: dread d4/db/de/f28 [0,4194304] 0 2024-03-30T04:28:20.108 INFO:tasks.workunit.client.0.smithi123.stdout:3/852: truncate d13/d24/d31/d23/d3a/d3c/d4e/d5d/f8e 885927 0 2024-03-30T04:28:20.109 INFO:tasks.workunit.client.1.smithi135.stdout:9/348: creat d7/f77 x:0 0 0 2024-03-30T04:28:20.109 INFO:tasks.workunit.client.1.smithi135.stdout:2/320: mkdir d4/d32/d2e/d67 0 2024-03-30T04:28:20.113 INFO:tasks.workunit.client.0.smithi123.stdout:3/853: write d13/f80 [139143,117052] 0 2024-03-30T04:28:20.113 INFO:tasks.workunit.client.0.smithi123.stdout:1/820: rename db/dd/d5f/d6d/c9f to db/dd/d42/d46/d56/d5c/dd0/c111 0 2024-03-30T04:28:20.115 INFO:tasks.workunit.client.0.smithi123.stdout:5/737: mkdir dd/d19/d3e/d5b/d62/dd3/df3 0 2024-03-30T04:28:20.116 INFO:tasks.workunit.client.1.smithi135.stdout:6/367: creat d2/dd/d18/d21/d55/f74 x:0 0 0 2024-03-30T04:28:20.116 INFO:tasks.workunit.client.0.smithi123.stdout:4/736: sync 2024-03-30T04:28:20.116 INFO:tasks.workunit.client.0.smithi123.stdout:6/817: creat d10/d20/da2/f114 x:0 0 0 2024-03-30T04:28:20.124 INFO:tasks.workunit.client.1.smithi135.stdout:2/321: symlink d4/da/l68 0 2024-03-30T04:28:20.126 INFO:tasks.workunit.client.0.smithi123.stdout:1/821: dread - db/dd/d5f/d6d/d85/d89/fdd zero size 2024-03-30T04:28:20.127 INFO:tasks.workunit.client.1.smithi135.stdout:1/313: dwrite d16/d1a/f4c [0,4194304] 0 2024-03-30T04:28:20.128 INFO:tasks.workunit.client.1.smithi135.stdout:2/322: write d4/d6/f35 [529916,91278] 0 2024-03-30T04:28:20.128 INFO:tasks.workunit.client.0.smithi123.stdout:6/818: getdents d10/d82/dc1/d45/d77/dd5 0 2024-03-30T04:28:20.129 INFO:tasks.workunit.client.1.smithi135.stdout:4/375: dread d10/f73 [0,4194304] 0 2024-03-30T04:28:20.132 INFO:tasks.workunit.client.0.smithi123.stdout:6/819: chown d10/d82/dc1/d6a/d7a/caf 8 1 2024-03-30T04:28:20.133 INFO:tasks.workunit.client.0.smithi123.stdout:3/854: rename d13/d24/d31/d23/d3a/d3c/d4e/d5d/da5/lfe to d13/d24/d39/d79/dea/l115 0 2024-03-30T04:28:20.136 INFO:tasks.workunit.client.1.smithi135.stdout:2/323: symlink d4/d32/d12/l69 0 2024-03-30T04:28:20.137 INFO:tasks.workunit.client.1.smithi135.stdout:4/376: creat d10/d1a/d42/d66/f84 x:0 0 0 2024-03-30T04:28:20.141 INFO:tasks.workunit.client.1.smithi135.stdout:1/314: truncate d16/d29/f35 1775319 0 2024-03-30T04:28:20.142 INFO:tasks.workunit.client.0.smithi123.stdout:0/770: dwrite d6/f9 [4194304,4194304] 0 2024-03-30T04:28:20.145 INFO:tasks.workunit.client.0.smithi123.stdout:3/855: chown d13/d24/d31/c84 1630 1 2024-03-30T04:28:20.145 INFO:tasks.workunit.client.0.smithi123.stdout:6/820: mkdir d10/d82/d8e/dc7/d5b/d103/d115 0 2024-03-30T04:28:20.145 INFO:tasks.workunit.client.1.smithi135.stdout:4/377: stat d10/d1a/f3a 0 2024-03-30T04:28:20.146 INFO:tasks.workunit.client.1.smithi135.stdout:1/315: rename d16 to d16/d5a/d5f/d6c 22 2024-03-30T04:28:20.147 INFO:tasks.workunit.client.0.smithi123.stdout:4/737: dwrite d2/d4/d2c/d42/d35/f5c [0,4194304] 0 2024-03-30T04:28:20.149 INFO:tasks.workunit.client.0.smithi123.stdout:0/771: write d6/dd/d31/fa9 [2831465,48214] 0 2024-03-30T04:28:20.156 INFO:tasks.workunit.client.0.smithi123.stdout:6/821: unlink d10/d20/d3f/f4d 0 2024-03-30T04:28:20.160 INFO:tasks.workunit.client.0.smithi123.stdout:0/772: read - d6/dd/d15/d98/d4e/f74 zero size 2024-03-30T04:28:20.167 INFO:tasks.workunit.client.0.smithi123.stdout:0/773: rmdir d6/dd/d15/d28/dd3/ddd 39 2024-03-30T04:28:20.168 INFO:tasks.workunit.client.1.smithi135.stdout:8/361: dwrite d6/d8/f1c [0,4194304] 0 2024-03-30T04:28:20.169 INFO:tasks.workunit.client.0.smithi123.stdout:7/764: dread d2/da/d11/d61/f72 [0,4194304] 0 2024-03-30T04:28:20.170 INFO:tasks.workunit.client.1.smithi135.stdout:6/368: dwrite d2/d36/f44 [0,4194304] 0 2024-03-30T04:28:20.170 INFO:tasks.workunit.client.1.smithi135.stdout:7/404: dwrite d4/db/f14 [4194304,4194304] 0 2024-03-30T04:28:20.173 INFO:tasks.workunit.client.0.smithi123.stdout:4/738: symlink d2/d4/de0/lfa 0 2024-03-30T04:28:20.175 INFO:tasks.workunit.client.1.smithi135.stdout:6/369: write d2/d2e/f70 [1015616,72977] 0 2024-03-30T04:28:20.181 INFO:tasks.workunit.client.0.smithi123.stdout:3/856: link d13/d24/d31/d30/l62 d13/d24/d31/d23/d3a/d3c/d4e/d5d/da5/dcd/l116 0 2024-03-30T04:28:20.182 INFO:tasks.workunit.client.0.smithi123.stdout:0/774: unlink d6/dd/d18/c1c 0 2024-03-30T04:28:20.182 INFO:tasks.workunit.client.1.smithi135.stdout:7/405: creat d4/db/d1a/d5c/d5d/f86 x:0 0 0 2024-03-30T04:28:20.184 INFO:tasks.workunit.client.1.smithi135.stdout:2/324: write d4/da/f59 [1989167,64660] 0 2024-03-30T04:28:20.185 INFO:tasks.workunit.client.1.smithi135.stdout:1/316: link d16/l48 d16/d29/d2c/d3c/l6d 0 2024-03-30T04:28:20.187 INFO:tasks.workunit.client.1.smithi135.stdout:6/370: mknod d2/dd/d40/c75 0 2024-03-30T04:28:20.187 INFO:tasks.workunit.client.1.smithi135.stdout:7/406: write d4/f54 [923317,73331] 0 2024-03-30T04:28:20.190 INFO:tasks.workunit.client.1.smithi135.stdout:1/317: stat d16/d29/d2c/d3c 0 2024-03-30T04:28:20.191 INFO:tasks.workunit.client.0.smithi123.stdout:1/822: dwrite db/f20 [0,4194304] 0 2024-03-30T04:28:20.192 INFO:tasks.workunit.client.1.smithi135.stdout:8/362: rename l3 to d6/d8/d14/l68 0 2024-03-30T04:28:20.193 INFO:tasks.workunit.client.1.smithi135.stdout:4/378: dwrite f4 [0,4194304] 0 2024-03-30T04:28:20.195 INFO:tasks.workunit.client.1.smithi135.stdout:6/371: mkdir d2/d36/d76 0 2024-03-30T04:28:20.197 INFO:tasks.workunit.client.0.smithi123.stdout:3/857: write d13/d24/f50 [1182043,19901] 0 2024-03-30T04:28:20.197 INFO:tasks.workunit.client.1.smithi135.stdout:7/407: dread - d4/db/d1a/d5c/d5d/f86 zero size 2024-03-30T04:28:20.198 INFO:tasks.workunit.client.1.smithi135.stdout:1/318: rmdir d16/d29/d2c/d43 39 2024-03-30T04:28:20.202 INFO:tasks.workunit.client.1.smithi135.stdout:1/319: dread - d16/f57 zero size 2024-03-30T04:28:20.203 INFO:tasks.workunit.client.1.smithi135.stdout:7/408: fdatasync d4/db/d1a/d4b/f45 0 2024-03-30T04:28:20.205 INFO:tasks.workunit.client.1.smithi135.stdout:4/379: dread - d10/d1a/d42/d66/f84 zero size 2024-03-30T04:28:20.205 INFO:tasks.workunit.client.0.smithi123.stdout:4/739: getdents d2/d4 0 2024-03-30T04:28:20.205 INFO:tasks.workunit.client.1.smithi135.stdout:6/372: chown d2/dd/d18/d21/d55/f74 3 1 2024-03-30T04:28:20.210 INFO:tasks.workunit.client.1.smithi135.stdout:6/373: chown d2/dd/d18 849 1 2024-03-30T04:28:20.210 INFO:tasks.workunit.client.0.smithi123.stdout:3/858: fdatasync d13/d24/d31/d23/d3a/d3c/db8/fca 0 2024-03-30T04:28:20.211 INFO:tasks.workunit.client.0.smithi123.stdout:0/775: creat d6/dd/d15/d98/d5d/d7f/df3/ffd x:0 0 0 2024-03-30T04:28:20.212 INFO:tasks.workunit.client.1.smithi135.stdout:8/363: mkdir d6/d8/d15/d54/d69 0 2024-03-30T04:28:20.214 INFO:tasks.workunit.client.0.smithi123.stdout:8/808: dread da/dc/f13 [4194304,4194304] 0 2024-03-30T04:28:20.214 INFO:tasks.workunit.client.1.smithi135.stdout:1/320: mknod d16/d5a/c6e 0 2024-03-30T04:28:20.217 INFO:tasks.workunit.client.0.smithi123.stdout:0/776: read - d6/dd/d15/d98/d4e/d69/ffb zero size 2024-03-30T04:28:20.217 INFO:tasks.workunit.client.0.smithi123.stdout:6/822: dwrite d10/f5a [4194304,4194304] 0 2024-03-30T04:28:20.221 INFO:tasks.workunit.client.1.smithi135.stdout:7/409: unlink d4/l50 0 2024-03-30T04:28:20.222 INFO:tasks.workunit.client.0.smithi123.stdout:6/823: readlink d10/d82/d8e/dc7/l2d 0 2024-03-30T04:28:20.223 INFO:tasks.workunit.client.1.smithi135.stdout:2/325: dread d4/d32/d2e/f48 [0,4194304] 0 2024-03-30T04:28:20.224 INFO:tasks.workunit.client.1.smithi135.stdout:1/321: dread d16/d5a/d5f/d3a/f52 [0,4194304] 0 2024-03-30T04:28:20.224 INFO:tasks.workunit.client.0.smithi123.stdout:8/809: write da/d27/ff6 [952895,101172] 0 2024-03-30T04:28:20.225 INFO:tasks.workunit.client.1.smithi135.stdout:8/364: write d6/d8/f39 [163562,28631] 0 2024-03-30T04:28:20.231 INFO:tasks.workunit.client.1.smithi135.stdout:8/365: read - d6/d8/f2d zero size 2024-03-30T04:28:20.232 INFO:tasks.workunit.client.0.smithi123.stdout:0/777: dread d6/d52/f73 [0,4194304] 0 2024-03-30T04:28:20.232 INFO:tasks.workunit.client.0.smithi123.stdout:1/823: mknod db/dd/d42/d46/d56/c112 0 2024-03-30T04:28:20.232 INFO:tasks.workunit.client.0.smithi123.stdout:7/765: dwrite d2/d4d/f6a [0,4194304] 0 2024-03-30T04:28:20.234 INFO:tasks.workunit.client.0.smithi123.stdout:4/740: mkdir d2/dfb 0 2024-03-30T04:28:20.239 INFO:tasks.workunit.client.0.smithi123.stdout:0/778: write d6/dd/d15/d98/d4e/d69/ffb [858959,297] 0 2024-03-30T04:28:20.240 INFO:tasks.workunit.client.0.smithi123.stdout:7/766: mknod d2/dc/d83/cfd 0 2024-03-30T04:28:20.241 INFO:tasks.workunit.client.0.smithi123.stdout:4/741: symlink d2/d4/d7/d63/lfc 0 2024-03-30T04:28:20.243 INFO:tasks.workunit.client.1.smithi135.stdout:2/326: symlink d4/da/l6a 0 2024-03-30T04:28:20.243 INFO:tasks.workunit.client.0.smithi123.stdout:1/824: getdents db/dd/d42/d46/d51/d72 0 2024-03-30T04:28:20.246 INFO:tasks.workunit.client.0.smithi123.stdout:0/779: mkdir d6/dd/d15/d98/dfe 0 2024-03-30T04:28:20.247 INFO:tasks.workunit.client.0.smithi123.stdout:4/742: stat d2/d4/d2c/d42/d35/d9b/db7 0 2024-03-30T04:28:20.247 INFO:tasks.workunit.client.1.smithi135.stdout:0/452: dread d0/d1/f5e [0,4194304] 0 2024-03-30T04:28:20.251 INFO:tasks.workunit.client.0.smithi123.stdout:1/825: mknod db/dd/d42/d46/d56/c113 0 2024-03-30T04:28:20.251 INFO:tasks.workunit.client.1.smithi135.stdout:6/374: dwrite d2/d36/f6e [0,4194304] 0 2024-03-30T04:28:20.253 INFO:tasks.workunit.client.0.smithi123.stdout:3/859: write d13/d24/d39/f7a [3831132,86821] 0 2024-03-30T04:28:20.253 INFO:tasks.workunit.client.1.smithi135.stdout:8/366: stat d6/d8/d15/d54/c61 0 2024-03-30T04:28:20.254 INFO:tasks.workunit.client.1.smithi135.stdout:2/327: creat d4/d32/d43/f6b x:0 0 0 2024-03-30T04:28:20.256 INFO:tasks.workunit.client.0.smithi123.stdout:4/743: truncate d2/f45 1861566 0 2024-03-30T04:28:20.257 INFO:tasks.workunit.client.0.smithi123.stdout:0/780: read - d6/dd/d15/d28/d4b/d8a/fb0 zero size 2024-03-30T04:28:20.263 INFO:tasks.workunit.client.0.smithi123.stdout:3/860: mkdir d13/d117 0 2024-03-30T04:28:20.263 INFO:tasks.workunit.client.0.smithi123.stdout:1/826: creat db/dd/d42/d46/d5d/d62/f114 x:0 0 0 2024-03-30T04:28:20.264 INFO:tasks.workunit.client.0.smithi123.stdout:0/781: fdatasync d6/dd/d15/d98/d4e/f68 0 2024-03-30T04:28:20.265 INFO:tasks.workunit.client.1.smithi135.stdout:0/453: mkdir d0/d13/d53/d54/d99 0 2024-03-30T04:28:20.265 INFO:tasks.workunit.client.1.smithi135.stdout:1/322: truncate d16/d1a/f4c 3316360 0 2024-03-30T04:28:20.266 INFO:tasks.workunit.client.1.smithi135.stdout:5/375: sync 2024-03-30T04:28:20.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.267+0000 7f21ee200700 1 -- 172.21.15.123:0/1451932583 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f21e8066280 msgr2=0x7f21e80ffa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:28:20.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.267+0000 7f21ee200700 1 --2- 172.21.15.123:0/1451932583 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f21e8066280 0x7f21e80ffa20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f21d8004940 tx=0x7f21d8009e30 comp rx=0 tx=0).stop 2024-03-30T04:28:20.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.267+0000 7f21ee200700 1 -- 172.21.15.123:0/1451932583 shutdown_connections 2024-03-30T04:28:20.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.267+0000 7f21ee200700 1 --2- 172.21.15.123:0/1451932583 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21e80fff60 0x7f21e8102340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:20.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.268+0000 7f21ee200700 1 --2- 172.21.15.123:0/1451932583 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f21e8066280 0x7f21e80ffa20 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:20.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.268+0000 7f21ee200700 1 -- 172.21.15.123:0/1451932583 >> 172.21.15.123:0/1451932583 conn(0x7f21e80f7430 msgr2=0x7f21e80f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:28:20.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.268+0000 7f21ee200700 1 -- 172.21.15.123:0/1451932583 shutdown_connections 2024-03-30T04:28:20.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.268+0000 7f21ee200700 1 -- 172.21.15.123:0/1451932583 wait complete. 2024-03-30T04:28:20.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.269+0000 7f21ee200700 1 Processor -- start 2024-03-30T04:28:20.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.270+0000 7f21ee200700 1 -- start start 2024-03-30T04:28:20.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.271+0000 7f21ee200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f21e8066280 0x7f21e8113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:28:20.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.271+0000 7f21ee200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21e80fff60 0x7f21e81144f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:28:20.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.271+0000 7f21ee200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e8114af0 con 0x7f21e80fff60 2024-03-30T04:28:20.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.271+0000 7f21ee200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e81152f0 con 0x7f21e8066280 2024-03-30T04:28:20.272 INFO:tasks.workunit.client.0.smithi123.stdout:3/861: write d13/d24/d31/d23/d3a/d3c/f78 [2716020,9963] 0 2024-03-30T04:28:20.275 INFO:tasks.workunit.client.1.smithi135.stdout:4/380: dwrite d10/d1a/d42/f63 [0,4194304] 0 2024-03-30T04:28:20.276 INFO:tasks.workunit.client.0.smithi123.stdout:1/827: symlink db/dd/d42/d46/d51/dd7/ddc/l115 0 2024-03-30T04:28:20.278 INFO:tasks.workunit.client.1.smithi135.stdout:7/410: rename d4/db/l77 to d4/db/d1a/d5c/d2c/l87 0 2024-03-30T04:28:20.280 INFO:tasks.workunit.client.0.smithi123.stdout:1/828: fdatasync db/dd/d42/d46/d56/d5c/f76 0 2024-03-30T04:28:20.281 INFO:tasks.workunit.client.1.smithi135.stdout:6/375: dwrite d2/dd/d14/d2b/f50 [0,4194304] 0 2024-03-30T04:28:20.286 INFO:tasks.workunit.client.1.smithi135.stdout:1/323: truncate d16/d29/d2c/d43/d50/f68 13313 0 2024-03-30T04:28:20.286 INFO:tasks.workunit.client.1.smithi135.stdout:4/381: mkdir d10/d72/d85 0 2024-03-30T04:28:20.288 INFO:tasks.workunit.client.0.smithi123.stdout:1/829: symlink db/dd/d5f/d6d/d85/d89/d90/d94/l116 0 2024-03-30T04:28:20.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.272+0000 7f21e6a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21e80fff60 0x7f21e81144f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:28:20.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.272+0000 7f21e7400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f21e8066280 0x7f21e8113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:28:20.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.272+0000 7f21e7400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f21e8066280 0x7f21e8113fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.123:41994/0 (socket says 172.21.15.123:41994) 2024-03-30T04:28:20.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.272+0000 7f21e7400700 1 -- 172.21.15.123:0/1443047125 learned_addr learned my addr 172.21.15.123:0/1443047125 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:28:20.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.272+0000 7f21e7400700 1 -- 172.21.15.123:0/1443047125 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21e80fff60 msgr2=0x7f21e81144f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:28:20.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.272+0000 7f21e7400700 1 --2- 172.21.15.123:0/1443047125 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21e80fff60 0x7f21e81144f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:20.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.272+0000 7f21e7400700 1 -- 172.21.15.123:0/1443047125 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d8005040 con 0x7f21e8066280 2024-03-30T04:28:20.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.273+0000 7f21e7400700 1 --2- 172.21.15.123:0/1443047125 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f21e8066280 0x7f21e8113fb0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f21d800b3b0 tx=0x7f21d8009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:28:20.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.274+0000 7f21df400700 1 -- 172.21.15.123:0/1443047125 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f21d800e040 con 0x7f21e8066280 2024-03-30T04:28:20.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.274+0000 7f21df400700 1 -- 172.21.15.123:0/1443047125 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f21d8004290 con 0x7f21e8066280 2024-03-30T04:28:20.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.274+0000 7f21df400700 1 -- 172.21.15.123:0/1443047125 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f21d801c3f0 con 0x7f21e8066280 2024-03-30T04:28:20.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.274+0000 7f21ee200700 1 -- 172.21.15.123:0/1443047125 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21e8115570 con 0x7f21e8066280 2024-03-30T04:28:20.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.274+0000 7f21ee200700 1 -- 172.21.15.123:0/1443047125 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21e8115a60 con 0x7f21e8066280 2024-03-30T04:28:20.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.276+0000 7f21ee200700 1 -- 172.21.15.123:0/1443047125 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21e805f5f0 con 0x7f21e8066280 2024-03-30T04:28:20.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.278+0000 7f21df400700 1 -- 172.21.15.123:0/1443047125 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 24) v1 ==== 88766+0+0 (secure 0 0 0) 0x7f21d8013820 con 0x7f21e8066280 2024-03-30T04:28:20.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.279+0000 7f21df400700 1 --2- 172.21.15.123:0/1443047125 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f21cc06af10 0x7f21cc06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:28:20.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.279+0000 7f21df400700 1 -- 172.21.15.123:0/1443047125 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f21d808ac40 con 0x7f21e8066280 2024-03-30T04:28:20.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.282+0000 7f21e6a00700 1 --2- 172.21.15.123:0/1443047125 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f21cc06af10 0x7f21cc06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:28:20.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.287+0000 7f21df400700 1 -- 172.21.15.123:0/1443047125 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f21d8058aa0 con 0x7f21e8066280 2024-03-30T04:28:20.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.289+0000 7f21e6a00700 1 --2- 172.21.15.123:0/1443047125 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f21cc06af10 0x7f21cc06d3c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f21d4004f40 tx=0x7f21d400b040 comp rx=0 tx=0).ready entity=mgr.14608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:28:20.312 INFO:tasks.workunit.client.0.smithi123.stdout:2/922: sync 2024-03-30T04:28:20.312 INFO:tasks.workunit.client.0.smithi123.stdout:0/782: dwrite d6/dd/d15/d28/d4b/d8a/fb0 [0,4194304] 0 2024-03-30T04:28:20.315 INFO:tasks.workunit.client.0.smithi123.stdout:3/862: dwrite d13/d24/d31/d23/d3a/d3c/d4e/d5d/f8e [0,4194304] 0 2024-03-30T04:28:20.317 INFO:tasks.workunit.client.1.smithi135.stdout:8/367: rename f0 to d6/f6a 0 2024-03-30T04:28:20.317 INFO:tasks.workunit.client.0.smithi123.stdout:4/744: dread d2/d34/f3f [0,4194304] 0 2024-03-30T04:28:20.318 INFO:tasks.workunit.client.1.smithi135.stdout:6/376: dwrite d2/dd/d14/f15 [0,4194304] 0 2024-03-30T04:28:20.318 INFO:tasks.workunit.client.0.smithi123.stdout:0/783: stat d6/dd/f8c 0 2024-03-30T04:28:20.319 INFO:tasks.workunit.client.0.smithi123.stdout:3/863: stat d13/d24/d31/d23/d3a/d3c/dab/cc5 0 2024-03-30T04:28:20.319 INFO:tasks.workunit.client.1.smithi135.stdout:4/382: creat d10/d1a/f86 x:0 0 0 2024-03-30T04:28:20.320 INFO:tasks.workunit.client.0.smithi123.stdout:6/824: dread d10/fa0 [0,4194304] 0 2024-03-30T04:28:20.321 INFO:tasks.workunit.client.1.smithi135.stdout:8/368: write d6/d8/d15/d2a/f37 [2496780,66508] 0 2024-03-30T04:28:20.324 INFO:tasks.workunit.client.0.smithi123.stdout:0/784: write d6/dd/d31/fa9 [3510393,26121] 0 2024-03-30T04:28:20.326 INFO:tasks.workunit.client.0.smithi123.stdout:6/825: rmdir d10/d82/d8e 39 2024-03-30T04:28:20.334 INFO:tasks.workunit.client.1.smithi135.stdout:0/454: truncate d0/d5/d9/f2e 3893306 0 2024-03-30T04:28:20.336 INFO:tasks.workunit.client.0.smithi123.stdout:3/864: creat d13/d24/d31/d23/d3a/d3c/db8/f118 x:0 0 0 2024-03-30T04:28:20.341 INFO:tasks.workunit.client.0.smithi123.stdout:6/826: creat d10/d82/dc1/d6a/d7a/d9f/f116 x:0 0 0 2024-03-30T04:28:20.341 INFO:tasks.workunit.client.1.smithi135.stdout:1/324: write fb [2286189,51605] 0 2024-03-30T04:28:20.342 INFO:tasks.workunit.client.0.smithi123.stdout:2/923: symlink d0/d37/d57/l12d 0 2024-03-30T04:28:20.342 INFO:tasks.workunit.client.0.smithi123.stdout:6/827: chown d10/d20/da2/f114 12 1 2024-03-30T04:28:20.346 INFO:tasks.workunit.client.1.smithi135.stdout:1/325: write d16/d19/f1f [6310683,15373] 0 2024-03-30T04:28:20.347 INFO:tasks.workunit.client.0.smithi123.stdout:2/924: write d0/d1f/d16/f18 [2580438,101044] 0 2024-03-30T04:28:20.348 INFO:tasks.workunit.client.1.smithi135.stdout:6/377: dwrite d2/dd/d14/d30/f67 [0,4194304] 0 2024-03-30T04:28:20.349 INFO:tasks.workunit.client.1.smithi135.stdout:5/376: rename d1/d3/d2b/d54/d5e/f6f to d1/d3/d2b/f72 0 2024-03-30T04:28:20.352 INFO:tasks.workunit.client.0.smithi123.stdout:2/925: write d0/d1f/d82/fe8 [789937,44308] 0 2024-03-30T04:28:20.352 INFO:tasks.workunit.client.1.smithi135.stdout:6/378: stat d2/dd/d18/d21/d55/l59 0 2024-03-30T04:28:20.353 INFO:tasks.workunit.client.0.smithi123.stdout:0/785: dread d6/dd/d18/f47 [0,4194304] 0 2024-03-30T04:28:20.353 INFO:tasks.workunit.client.1.smithi135.stdout:5/377: readlink d1/d3/d2b/l33 0 2024-03-30T04:28:20.356 INFO:tasks.workunit.client.0.smithi123.stdout:2/926: fsync d0/d1f/d30/f117 0 2024-03-30T04:28:20.356 INFO:tasks.workunit.client.0.smithi123.stdout:0/786: chown d6/dd/d31/d35/f45 32050520 1 2024-03-30T04:28:20.357 INFO:tasks.workunit.client.0.smithi123.stdout:4/745: getdents d2/d4 0 2024-03-30T04:28:20.359 INFO:tasks.workunit.client.1.smithi135.stdout:1/326: mkdir d16/d5a/d5f/d3a/d6f 0 2024-03-30T04:28:20.361 INFO:tasks.workunit.client.1.smithi135.stdout:0/455: mknod d0/d19/c9a 0 2024-03-30T04:28:20.364 INFO:tasks.workunit.client.1.smithi135.stdout:6/379: creat d2/dd/d14/d53/f77 x:0 0 0 2024-03-30T04:28:20.364 INFO:tasks.workunit.client.1.smithi135.stdout:0/456: readlink d0/d5/d9/d23/d58/l20 0 2024-03-30T04:28:20.365 INFO:tasks.workunit.client.0.smithi123.stdout:2/927: fsync d0/d1f/d30/f94 0 2024-03-30T04:28:20.366 INFO:tasks.workunit.client.0.smithi123.stdout:3/865: dread d13/d24/d31/d23/d3a/d3c/db8/fca [0,4194304] 0 2024-03-30T04:28:20.368 INFO:tasks.workunit.client.0.smithi123.stdout:2/928: chown d0/d1f/d27/d68/dd6/d8c/l121 3 1 2024-03-30T04:28:20.372 INFO:tasks.workunit.client.1.smithi135.stdout:7/411: rename d4/f29 to d4/db/d1a/d5c/d2c/f88 0 2024-03-30T04:28:20.372 INFO:tasks.workunit.client.0.smithi123.stdout:0/787: mkdir d6/dd/d15/d28/d4b/d8a/dff 0 2024-03-30T04:28:20.374 INFO:tasks.workunit.client.1.smithi135.stdout:1/327: symlink d16/d29/d2c/d3c/l70 0 2024-03-30T04:28:20.375 INFO:tasks.workunit.client.0.smithi123.stdout:2/929: creat d0/d1f/d16/f12e x:0 0 0 2024-03-30T04:28:20.375 INFO:tasks.workunit.client.1.smithi135.stdout:5/378: mknod d1/d3/d2b/d54/d6d/c73 0 2024-03-30T04:28:20.376 INFO:tasks.workunit.client.0.smithi123.stdout:4/746: getdents d2/d4/d2c/d42/d35/d9b 0 2024-03-30T04:28:20.376 INFO:tasks.workunit.client.0.smithi123.stdout:3/866: mkdir d13/d24/d31/d23/d3a/d3c/d114/d119 0 2024-03-30T04:28:20.377 INFO:tasks.workunit.client.1.smithi135.stdout:8/369: dread d6/d8/d14/fe [0,4194304] 0 2024-03-30T04:28:20.377 INFO:tasks.workunit.client.1.smithi135.stdout:0/457: dread d0/d1/d16/f62 [0,4194304] 0 2024-03-30T04:28:20.378 INFO:tasks.workunit.client.1.smithi135.stdout:5/379: dread - d1/d7/f59 zero size 2024-03-30T04:28:20.379 INFO:tasks.workunit.client.1.smithi135.stdout:6/380: creat d2/dd/d14/d30/d43/f78 x:0 0 0 2024-03-30T04:28:20.382 INFO:tasks.workunit.client.1.smithi135.stdout:8/370: fdatasync d6/d8/d14/f5e 0 2024-03-30T04:28:20.382 INFO:tasks.workunit.client.0.smithi123.stdout:2/930: mkdir d0/d37/d11a/d12f 0 2024-03-30T04:28:20.382 INFO:tasks.workunit.client.0.smithi123.stdout:4/747: read d2/d4/d2c/d42/d35/d5d/dc0/fce [524774,128209] 0 2024-03-30T04:28:20.388 INFO:tasks.workunit.client.1.smithi135.stdout:7/412: creat d4/db/d1a/d5c/d2c/d51/d56/f89 x:0 0 0 2024-03-30T04:28:20.390 INFO:tasks.workunit.client.0.smithi123.stdout:4/748: symlink d2/d4/d2c/d42/d35/d9b/da2/deb/lfd 0 2024-03-30T04:28:20.391 INFO:tasks.workunit.client.1.smithi135.stdout:5/380: unlink d1/d7/l31 0 2024-03-30T04:28:20.391 INFO:tasks.workunit.client.0.smithi123.stdout:3/867: symlink d13/d24/d31/d23/d3a/d3c/d4e/dde/l11a 0 2024-03-30T04:28:20.393 INFO:tasks.workunit.client.1.smithi135.stdout:5/381: chown d1/f4f 0 1 2024-03-30T04:28:20.396 INFO:tasks.workunit.client.1.smithi135.stdout:8/371: mknod d6/d8/d15/d1f/c6b 0 2024-03-30T04:28:20.398 INFO:tasks.workunit.client.0.smithi123.stdout:4/749: creat d2/d4/d2c/d42/d35/d5d/d6f/df9/ffe x:0 0 0 2024-03-30T04:28:20.399 INFO:tasks.workunit.client.1.smithi135.stdout:5/382: unlink d1/d3/d2b/d54/d6d/c73 0 2024-03-30T04:28:20.399 INFO:tasks.workunit.client.0.smithi123.stdout:3/868: fdatasync d13/f5e 0 2024-03-30T04:28:20.400 INFO:tasks.workunit.client.0.smithi123.stdout:0/788: truncate d6/dd/d15/f8e 1465382 0 2024-03-30T04:28:20.401 INFO:tasks.workunit.client.1.smithi135.stdout:5/383: write d1/f36 [1562946,95158] 0 2024-03-30T04:28:20.402 INFO:tasks.workunit.client.0.smithi123.stdout:0/789: stat d6/d52/ce1 0 2024-03-30T04:28:20.404 INFO:tasks.workunit.client.0.smithi123.stdout:6/828: dread d10/f11 [0,4194304] 0 2024-03-30T04:28:20.407 INFO:tasks.workunit.client.1.smithi135.stdout:0/458: dwrite d0/d5/d9/d23/d58/f1f [4194304,4194304] 0 2024-03-30T04:28:20.408 INFO:tasks.workunit.client.0.smithi123.stdout:4/750: symlink d2/d4/d2c/d42/d80/da5/lff 0 2024-03-30T04:28:20.409 INFO:tasks.workunit.client.0.smithi123.stdout:3/869: mknod d13/d24/d31/d23/d3a/d3c/d66/c11b 0 2024-03-30T04:28:20.410 INFO:tasks.workunit.client.1.smithi135.stdout:8/372: creat d6/d8/d15/d54/d4f/f6c x:0 0 0 2024-03-30T04:28:20.414 INFO:tasks.workunit.client.0.smithi123.stdout:2/931: dread d0/d1f/d27/d68/f11d [0,4194304] 0 2024-03-30T04:28:20.414 INFO:tasks.workunit.client.0.smithi123.stdout:0/790: creat d6/dd/d15/d28/d4b/dad/f100 x:0 0 0 2024-03-30T04:28:20.415 INFO:tasks.workunit.client.0.smithi123.stdout:4/751: mkdir d2/dab/d100 0 2024-03-30T04:28:20.416 INFO:tasks.workunit.client.1.smithi135.stdout:7/413: getdents d4/db/d1a/d5c/d2c/d2e/d46/d75 0 2024-03-30T04:28:20.416 INFO:tasks.workunit.client.1.smithi135.stdout:9/349: sync 2024-03-30T04:28:20.416 INFO:tasks.workunit.client.1.smithi135.stdout:3/382: sync 2024-03-30T04:28:20.417 INFO:tasks.workunit.client.0.smithi123.stdout:2/932: write d0/d1f/d30/d119/dbb/d97/db5/f129 [565660,12429] 0 2024-03-30T04:28:20.418 INFO:tasks.workunit.client.0.smithi123.stdout:2/933: readlink d0/d1f/d1c/laa 0 2024-03-30T04:28:20.421 INFO:tasks.workunit.client.0.smithi123.stdout:3/870: rename d13/d24/d31/f75 to d13/d24/d31/d23/d3a/d3c/d66/f11c 0 2024-03-30T04:28:20.426 INFO:tasks.workunit.client.1.smithi135.stdout:0/459: mkdir d0/d5/d9/d23/d2a/d60/d9b 0 2024-03-30T04:28:20.426 INFO:tasks.workunit.client.0.smithi123.stdout:3/871: mknod d13/d24/d31/d23/d3a/d3c/d4e/dde/c11d 0 2024-03-30T04:28:20.429 INFO:tasks.workunit.client.1.smithi135.stdout:5/384: getdents d1/d55 0 2024-03-30T04:28:20.433 INFO:tasks.workunit.client.0.smithi123.stdout:6/829: dwrite d10/d82/d8e/dc7/fd7 [0,4194304] 0 2024-03-30T04:28:20.434 INFO:tasks.workunit.client.0.smithi123.stdout:0/791: write d6/dd/d15/d28/f3d [493253,52361] 0 2024-03-30T04:28:20.437 INFO:tasks.workunit.client.0.smithi123.stdout:0/792: chown d6/dd/d15/l46 1 1 2024-03-30T04:28:20.437 INFO:tasks.workunit.client.1.smithi135.stdout:3/383: mkdir d3/d10/d16/d23/d6a/d86 0 2024-03-30T04:28:20.438 INFO:tasks.workunit.client.1.smithi135.stdout:1/328: dwrite f7 [0,4194304] 0 2024-03-30T04:28:20.441 INFO:tasks.workunit.client.0.smithi123.stdout:0/793: chown d6/dd/d18/d1f 969 1 2024-03-30T04:28:20.446 INFO:tasks.workunit.client.0.smithi123.stdout:6/830: rename d10/d82/dc1/d45/l76 to d10/d20/l117 0 2024-03-30T04:28:20.449 INFO:tasks.workunit.client.0.smithi123.stdout:2/934: dwrite d0/d1f/d16/f87 [0,4194304] 0 2024-03-30T04:28:20.449 INFO:tasks.workunit.client.1.smithi135.stdout:9/350: dwrite d7/f26 [0,4194304] 0 2024-03-30T04:28:20.451 INFO:tasks.workunit.client.0.smithi123.stdout:5/738: sync 2024-03-30T04:28:20.451 INFO:tasks.workunit.client.0.smithi123.stdout:9/746: sync 2024-03-30T04:28:20.452 INFO:tasks.workunit.client.1.smithi135.stdout:0/460: rename d0/d5/d9/d23/d58/d55 to d0/d13/d53/d7f/d83/d9c 0 2024-03-30T04:28:20.452 INFO:tasks.workunit.client.1.smithi135.stdout:8/373: creat d6/d8/d15/d54/f6d x:0 0 0 2024-03-30T04:28:20.455 INFO:tasks.workunit.client.1.smithi135.stdout:3/384: creat d3/d50/d22/f87 x:0 0 0 2024-03-30T04:28:20.456 INFO:tasks.workunit.client.1.smithi135.stdout:5/385: dwrite d1/d7/f10 [0,4194304] 0 2024-03-30T04:28:20.457 INFO:tasks.workunit.client.1.smithi135.stdout:8/374: readlink d6/d8/d14/d32/l46 0 2024-03-30T04:28:20.460 INFO:tasks.workunit.client.0.smithi123.stdout:2/935: creat d0/d1f/d16/f130 x:0 0 0 2024-03-30T04:28:20.460 INFO:tasks.workunit.client.1.smithi135.stdout:3/385: write d3/d50/d46/f56 [496537,2948] 0 2024-03-30T04:28:20.461 INFO:tasks.workunit.client.0.smithi123.stdout:5/739: unlink dd/d19/d1f/d3c/fab 0 2024-03-30T04:28:20.461 INFO:tasks.workunit.client.0.smithi123.stdout:3/872: dread d13/d24/d39/d79/db9/fd2 [0,4194304] 0 2024-03-30T04:28:20.463 INFO:tasks.workunit.client.0.smithi123.stdout:9/747: chown d6/d1f/d45/d97/d69/dfb/ffc 1453 1 2024-03-30T04:28:20.464 INFO:tasks.workunit.client.1.smithi135.stdout:3/386: read - d3/d10/f5e zero size 2024-03-30T04:28:20.464 INFO:tasks.workunit.client.1.smithi135.stdout:7/414: link d4/db/de/l3e d4/db/d1a/d4b/l8a 0 2024-03-30T04:28:20.467 INFO:tasks.workunit.client.1.smithi135.stdout:9/351: mkdir d7/d29/d78 0 2024-03-30T04:28:20.467 INFO:tasks.workunit.client.0.smithi123.stdout:5/740: fdatasync dd/d19/d1f/f27 0 2024-03-30T04:28:20.469 INFO:tasks.workunit.client.0.smithi123.stdout:3/873: fsync d13/d24/d31/d30/f45 0 2024-03-30T04:28:20.469 INFO:tasks.workunit.client.1.smithi135.stdout:0/461: mknod d0/d5/d9/d23/d58/d63/d77/c9d 0 2024-03-30T04:28:20.469 INFO:tasks.workunit.client.1.smithi135.stdout:3/387: stat d3/d10/d81 0 2024-03-30T04:28:20.471 INFO:tasks.workunit.client.0.smithi123.stdout:2/936: fdatasync d0/d1f/d1c/d39/d4b/d4c/d5f/f9e 0 2024-03-30T04:28:20.472 INFO:tasks.workunit.client.1.smithi135.stdout:5/386: symlink d1/d7/d11/d3c/l74 0 2024-03-30T04:28:20.474 INFO:tasks.workunit.client.1.smithi135.stdout:9/352: truncate d7/d41/d72/f74 861380 0 2024-03-30T04:28:20.475 INFO:tasks.workunit.client.1.smithi135.stdout:3/388: creat d3/d10/f88 x:0 0 0 2024-03-30T04:28:20.476 INFO:tasks.workunit.client.1.smithi135.stdout:7/415: mknod d4/db/d1a/d4b/d4f/c8b 0 2024-03-30T04:28:20.477 INFO:tasks.workunit.client.1.smithi135.stdout:8/375: creat d6/f6e x:0 0 0 2024-03-30T04:28:20.479 INFO:tasks.workunit.client.0.smithi123.stdout:3/874: readlink d13/d24/d31/d23/d3a/d3c/db8/lf6 0 2024-03-30T04:28:20.479 INFO:tasks.workunit.client.1.smithi135.stdout:5/387: fsync d1/d7/d11/d3c/f44 0 2024-03-30T04:28:20.481 INFO:tasks.workunit.client.0.smithi123.stdout:2/937: read - d0/d37/d57/fed zero size 2024-03-30T04:28:20.482 INFO:tasks.workunit.client.0.smithi123.stdout:0/794: dwrite d6/dd/d15/d28/d4b/d8a/ffa [0,4194304] 0 2024-03-30T04:28:20.482 INFO:tasks.workunit.client.0.smithi123.stdout:9/748: getdents d6/dad 0 2024-03-30T04:28:20.484 INFO:tasks.workunit.client.0.smithi123.stdout:2/938: chown d0/d1f/le2 452494 1 2024-03-30T04:28:20.485 INFO:tasks.workunit.client.1.smithi135.stdout:5/388: chown d1/d3/d2b/d54/d5e 0 1 2024-03-30T04:28:20.486 INFO:tasks.workunit.client.1.smithi135.stdout:3/389: creat d3/d54/d5c/f89 x:0 0 0 2024-03-30T04:28:20.487 INFO:tasks.workunit.client.0.smithi123.stdout:0/795: chown d6/dd/d15/d98/d4e/d69 62 1 2024-03-30T04:28:20.489 INFO:tasks.workunit.client.1.smithi135.stdout:8/376: mknod d6/d8/d15/d2a/c6f 0 2024-03-30T04:28:20.489 INFO:tasks.workunit.client.1.smithi135.stdout:7/416: fdatasync d4/db/d1a/d5c/d2c/f39 0 2024-03-30T04:28:20.489 INFO:tasks.workunit.client.1.smithi135.stdout:3/390: dread - d3/d50/f5b zero size 2024-03-30T04:28:20.491 INFO:tasks.workunit.client.1.smithi135.stdout:5/389: write d1/d7/f13 [2702105,53550] 0 2024-03-30T04:28:20.495 INFO:tasks.workunit.client.0.smithi123.stdout:3/875: rmdir d13/d24/d31/d23/d3a/d3c/db8/df4 39 2024-03-30T04:28:20.496 INFO:tasks.workunit.client.1.smithi135.stdout:8/377: chown d6/d8/d15/c2c 23 1 2024-03-30T04:28:20.496 INFO:tasks.workunit.client.1.smithi135.stdout:0/462: rmdir d0/d5/d9/d23/d2a/d60/d9b 0 2024-03-30T04:28:20.497 INFO:tasks.workunit.client.1.smithi135.stdout:0/463: chown d0 1796704 1 2024-03-30T04:28:20.500 INFO:tasks.workunit.client.1.smithi135.stdout:7/417: read d4/db/d1a/d3d/f5b [611325,24925] 0 2024-03-30T04:28:20.501 INFO:tasks.workunit.client.0.smithi123.stdout:9/749: link d6/d1f/d72/fe9 d6/d1f/d92/db8/f103 0 2024-03-30T04:28:20.502 INFO:tasks.workunit.client.1.smithi135.stdout:8/378: fsync d6/d8/f1c 0 2024-03-30T04:28:20.504 INFO:tasks.workunit.client.1.smithi135.stdout:3/391: creat d3/d10/d16/d23/d6a/f8a x:0 0 0 2024-03-30T04:28:20.506 INFO:tasks.workunit.client.1.smithi135.stdout:5/390: read d1/f4f [2561187,67188] 0 2024-03-30T04:28:20.509 INFO:tasks.workunit.client.0.smithi123.stdout:9/750: write d6/d1f/d72/fcf [228385,64855] 0 2024-03-30T04:28:20.514 INFO:tasks.workunit.client.0.smithi123.stdout:9/751: symlink d6/d1f/d72/db6/l104 0 2024-03-30T04:28:20.515 INFO:tasks.workunit.client.1.smithi135.stdout:3/392: mkdir d3/d50/d22/d8b 0 2024-03-30T04:28:20.515 INFO:tasks.workunit.client.0.smithi123.stdout:6/831: dread d10/fb2 [0,4194304] 0 2024-03-30T04:28:20.515 INFO:tasks.workunit.client.0.smithi123.stdout:9/752: stat d6/d1f/d72/f8f 0 2024-03-30T04:28:20.516 INFO:tasks.workunit.client.1.smithi135.stdout:9/353: dwrite d7/f2f [0,4194304] 0 2024-03-30T04:28:20.517 INFO:tasks.workunit.client.1.smithi135.stdout:0/464: rename d0/d5/d9/d23/d58/d63/d77/c7c to d0/d5/d10/c9e 0 2024-03-30T04:28:20.518 INFO:tasks.workunit.client.1.smithi135.stdout:8/379: truncate d6/d8/d15/d54/f5d 351285 0 2024-03-30T04:28:20.518 INFO:tasks.workunit.client.0.smithi123.stdout:9/753: creat d6/d1f/d92/f105 x:0 0 0 2024-03-30T04:28:20.519 INFO:tasks.workunit.client.1.smithi135.stdout:5/391: creat d1/d3/d2b/f75 x:0 0 0 2024-03-30T04:28:20.520 INFO:tasks.workunit.client.0.smithi123.stdout:6/832: symlink d10/d82/dc1/d6a/d7a/de8/l118 0 2024-03-30T04:28:20.524 INFO:tasks.workunit.client.1.smithi135.stdout:8/380: read d6/d8/d15/d2a/f3c [2741703,67590] 0 2024-03-30T04:28:20.524 INFO:tasks.workunit.client.0.smithi123.stdout:6/833: rmdir d10/d82/dc1/dd2/dda 39 2024-03-30T04:28:20.527 INFO:tasks.workunit.client.1.smithi135.stdout:9/354: write d7/d1b/f16 [2120183,53336] 0 2024-03-30T04:28:20.527 INFO:tasks.workunit.client.1.smithi135.stdout:0/465: mkdir d0/d5/d42/d9f 0 2024-03-30T04:28:20.527 INFO:tasks.workunit.client.1.smithi135.stdout:5/392: truncate d1/d55/f57 563277 0 2024-03-30T04:28:20.530 INFO:tasks.workunit.client.1.smithi135.stdout:7/418: dread d4/db/f1f [4194304,4194304] 0 2024-03-30T04:28:20.532 INFO:tasks.workunit.client.0.smithi123.stdout:6/834: truncate d10/d20/f36 4641901 0 2024-03-30T04:28:20.535 INFO:tasks.workunit.client.1.smithi135.stdout:9/355: creat d7/d41/f79 x:0 0 0 2024-03-30T04:28:20.536 INFO:tasks.workunit.client.0.smithi123.stdout:5/741: dread dd/d19/d1f/d68/fc6 [4194304,4194304] 0 2024-03-30T04:28:20.537 INFO:tasks.workunit.client.0.smithi123.stdout:2/939: dread d0/d1f/d1c/d20/f46 [0,4194304] 0 2024-03-30T04:28:20.537 INFO:tasks.workunit.client.1.smithi135.stdout:8/381: unlink d6/d8/d14/fe 0 2024-03-30T04:28:20.539 INFO:tasks.workunit.client.0.smithi123.stdout:5/742: fsync dd/d19/d3e/d5b/f8a 0 2024-03-30T04:28:20.539 INFO:tasks.workunit.client.1.smithi135.stdout:7/419: creat d4/f8c x:0 0 0 2024-03-30T04:28:20.540 INFO:tasks.workunit.client.0.smithi123.stdout:6/835: mkdir d10/d82/d8e/dc7/d5b/d103/d115/d119 0 2024-03-30T04:28:20.540 INFO:tasks.workunit.client.1.smithi135.stdout:9/356: chown d7/d41/c6b 26 1 2024-03-30T04:28:20.540 INFO:tasks.workunit.client.1.smithi135.stdout:1/329: dread d16/d5a/d5f/f4e [0,4194304] 0 2024-03-30T04:28:20.541 INFO:tasks.workunit.client.0.smithi123.stdout:5/743: chown dd/d19/d3e/d5b/dd7 1865399846 1 2024-03-30T04:28:20.542 INFO:tasks.workunit.client.1.smithi135.stdout:3/393: getdents d3/d10/d16/d41/d83 0 2024-03-30T04:28:20.543 INFO:tasks.workunit.client.1.smithi135.stdout:7/420: chown d4/db/de/f1d 5002 1 2024-03-30T04:28:20.544 INFO:tasks.workunit.client.0.smithi123.stdout:3/876: truncate d13/d24/d31/d23/d3a/f54 481478 0 2024-03-30T04:28:20.545 INFO:tasks.workunit.client.0.smithi123.stdout:6/836: read d10/d82/d8e/dc7/d5b/fef [155424,47849] 0 2024-03-30T04:28:20.547 INFO:tasks.workunit.client.1.smithi135.stdout:7/421: write d4/f54 [4324264,2566] 0 2024-03-30T04:28:20.548 INFO:tasks.workunit.client.0.smithi123.stdout:5/744: mkdir dd/d19/d1f/d3c/d72/d8c/df4 0 2024-03-30T04:28:20.548 INFO:tasks.workunit.client.1.smithi135.stdout:9/357: truncate d7/d1b/d37/f4a 4572865 0 2024-03-30T04:28:20.550 INFO:tasks.workunit.client.1.smithi135.stdout:1/330: getdents d16 0 2024-03-30T04:28:20.551 INFO:tasks.workunit.client.0.smithi123.stdout:6/837: truncate d10/f97 150631 0 2024-03-30T04:28:20.552 INFO:tasks.workunit.client.0.smithi123.stdout:3/877: rmdir d13/d24/d31/d23/d3a/d3c/d66 39 2024-03-30T04:28:20.553 INFO:tasks.workunit.client.1.smithi135.stdout:9/358: symlink d7/d1b/l7a 0 2024-03-30T04:28:20.556 INFO:tasks.workunit.client.0.smithi123.stdout:2/940: dwrite d0/d1f/d78/feb [0,4194304] 0 2024-03-30T04:28:20.557 INFO:tasks.workunit.client.1.smithi135.stdout:3/394: rename d3/d10/d16/d23/d6a/f8a to d3/d10/d16/d23/d6a/d86/f8c 0 2024-03-30T04:28:20.557 INFO:tasks.workunit.client.0.smithi123.stdout:6/838: mknod d10/d82/dc1/d6a/d7a/de8/d101/c11a 0 2024-03-30T04:28:20.559 INFO:tasks.workunit.client.1.smithi135.stdout:9/359: symlink d7/d55/l7b 0 2024-03-30T04:28:20.565 INFO:tasks.workunit.client.0.smithi123.stdout:3/878: chown d13/d24/d31/d23/d3a/d3c/d4e/cc4 13289498 1 2024-03-30T04:28:20.566 INFO:tasks.workunit.client.1.smithi135.stdout:3/395: creat d3/db/f8d x:0 0 0 2024-03-30T04:28:20.567 INFO:tasks.workunit.client.1.smithi135.stdout:0/466: dwrite d0/d5/d9/f17 [0,4194304] 0 2024-03-30T04:28:20.569 INFO:tasks.workunit.client.0.smithi123.stdout:9/754: dwrite f0 [0,4194304] 0 2024-03-30T04:28:20.569 INFO:tasks.workunit.client.1.smithi135.stdout:3/396: readlink d3/d6e/l70 0 2024-03-30T04:28:20.572 INFO:tasks.workunit.client.0.smithi123.stdout:9/755: chown d6/d1f/d45/d97/d69/dfb/ffc 245333 1 2024-03-30T04:28:20.575 INFO:tasks.workunit.client.0.smithi123.stdout:9/756: read - d6/d1f/d4d/fd9 zero size 2024-03-30T04:28:20.575 INFO:tasks.workunit.client.1.smithi135.stdout:5/393: dwrite d1/d3/d2b/f72 [0,4194304] 0 2024-03-30T04:28:20.579 INFO:tasks.workunit.client.0.smithi123.stdout:3/879: chown d13/d24/d31/d23/d3a/d3c/d66 6228188 1 2024-03-30T04:28:20.579 INFO:tasks.workunit.client.0.smithi123.stdout:6/839: link d10/d82/dc1/d6a/c6e d10/d82/d8e/dc7/d5b/d103/d115/c11b 0 2024-03-30T04:28:20.580 INFO:tasks.workunit.client.1.smithi135.stdout:5/394: write d1/f36 [1309874,67341] 0 2024-03-30T04:28:20.584 INFO:tasks.workunit.client.0.smithi123.stdout:6/840: creat d10/d82/dd6/f11c x:0 0 0 2024-03-30T04:28:20.584 INFO:tasks.workunit.client.1.smithi135.stdout:9/360: creat d7/d1b/f7c x:0 0 0 2024-03-30T04:28:20.586 INFO:tasks.workunit.client.0.smithi123.stdout:3/880: write d13/d24/d31/d23/d3a/d3c/d4e/d5d/f8e [3002213,98351] 0 2024-03-30T04:28:20.587 INFO:tasks.workunit.client.1.smithi135.stdout:1/331: link d16/d5a/d5f/d3a/l4f d16/d5a/l71 0 2024-03-30T04:28:20.587 INFO:tasks.workunit.client.1.smithi135.stdout:9/361: stat d7/d29/f51 0 2024-03-30T04:28:20.589 INFO:tasks.workunit.client.0.smithi123.stdout:3/881: stat d13/d24/d31/d23/f25 0 2024-03-30T04:28:20.594 INFO:tasks.workunit.client.0.smithi123.stdout:8/810: sync 2024-03-30T04:28:20.596 INFO:tasks.workunit.client.0.smithi123.stdout:6/841: mkdir d10/d82/d11d 0 2024-03-30T04:28:20.598 INFO:tasks.workunit.client.0.smithi123.stdout:3/882: symlink d13/d24/d31/l11e 0 2024-03-30T04:28:20.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:20 smithi135 ceph-mon[44364]: pgmap v43: 65 pgs: 65 active+clean; 1.8 GiB data, 6.8 GiB used, 530 GiB / 536 GiB avail; 50 MiB/s rd, 170 MiB/s wr, 328 op/s 2024-03-30T04:28:20.601 INFO:tasks.workunit.client.1.smithi135.stdout:5/395: mkdir d1/d3/d76 0 2024-03-30T04:28:20.602 INFO:tasks.workunit.client.1.smithi135.stdout:0/467: dwrite d0/d5/d9/f17 [0,4194304] 0 2024-03-30T04:28:20.603 INFO:tasks.workunit.client.0.smithi123.stdout:6/842: rmdir d10/d82/d11d 0 2024-03-30T04:28:20.604 INFO:tasks.workunit.client.1.smithi135.stdout:1/332: fsync d16/d5a/d5f/f4e 0 2024-03-30T04:28:20.606 INFO:tasks.workunit.client.1.smithi135.stdout:0/468: fdatasync d0/d5/d10/f31 0 2024-03-30T04:28:20.608 INFO:tasks.workunit.client.0.smithi123.stdout:8/811: rename da/d27/d3c/d2d/fcf to da/d27/d56/d6c/d96/ffa 0 2024-03-30T04:28:20.612 INFO:tasks.workunit.client.1.smithi135.stdout:9/362: creat d7/d41/d72/f7d x:0 0 0 2024-03-30T04:28:20.612 INFO:tasks.workunit.client.0.smithi123.stdout:8/812: write da/d27/d56/d6c/dd4/ff5 [825175,127955] 0 2024-03-30T04:28:20.617 INFO:tasks.workunit.client.0.smithi123.stdout:3/883: rename d13/d24/d31/d23/d3a/d3c/dab/db2/de8 to d13/d24/d39/d79/d90/d11f 0 2024-03-30T04:28:20.618 INFO:tasks.workunit.client.0.smithi123.stdout:8/813: mkdir da/dc/d14/d9d/dfb 0 2024-03-30T04:28:20.622 INFO:tasks.workunit.client.1.smithi135.stdout:7/422: truncate d4/db/de/d5a/f68 860049 0 2024-03-30T04:28:20.623 INFO:tasks.workunit.client.0.smithi123.stdout:6/843: dread d10/d82/d8e/dc7/f2a [0,4194304] 0 2024-03-30T04:28:20.624 INFO:tasks.workunit.client.1.smithi135.stdout:9/363: unlink d7/d41/f61 0 2024-03-30T04:28:20.624 INFO:tasks.workunit.client.0.smithi123.stdout:2/941: dread d0/d1f/d16/f18 [0,4194304] 0 2024-03-30T04:28:20.628 INFO:tasks.workunit.client.0.smithi123.stdout:9/757: truncate d6/d1f/d45/d97/d69/fab 239170 0 2024-03-30T04:28:20.628 INFO:tasks.workunit.client.0.smithi123.stdout:6/844: chown d10/d20/da2/f114 13621065 1 2024-03-30T04:28:20.628 INFO:tasks.workunit.client.1.smithi135.stdout:3/397: rename d3/d50/d22/c75 to d3/d54/c8e 0 2024-03-30T04:28:20.629 INFO:tasks.workunit.client.0.smithi123.stdout:2/942: readlink d0/d1f/d82/lf6 0 2024-03-30T04:28:20.629 INFO:tasks.workunit.client.1.smithi135.stdout:0/469: read d0/d5/d9/f2e [3402744,35137] 0 2024-03-30T04:28:20.630 INFO:tasks.workunit.client.0.smithi123.stdout:8/814: symlink da/dc/d14/d9c/lfc 0 2024-03-30T04:28:20.630 INFO:tasks.workunit.client.0.smithi123.stdout:6/845: readlink d10/d20/d3f/l5e 0 2024-03-30T04:28:20.631 INFO:tasks.workunit.client.1.smithi135.stdout:7/423: creat d4/db/d1a/d5c/d5d/f8d x:0 0 0 2024-03-30T04:28:20.632 INFO:tasks.workunit.client.0.smithi123.stdout:2/943: chown d0/d1f/d1c/d10d 0 1 2024-03-30T04:28:20.632 INFO:tasks.workunit.client.0.smithi123.stdout:9/758: chown d6/d1f/d92/fe3 48355 1 2024-03-30T04:28:20.635 INFO:tasks.workunit.client.1.smithi135.stdout:5/396: rename d1/c67 to d1/da/d4d/c77 0 2024-03-30T04:28:20.637 INFO:tasks.workunit.client.1.smithi135.stdout:9/364: rename d7/d41 to d7/d41/d7e 22 2024-03-30T04:28:20.639 INFO:tasks.workunit.client.0.smithi123.stdout:5/745: dread dd/d19/d1f/d35/d3a/d57/f6a [0,4194304] 0 2024-03-30T04:28:20.639 INFO:tasks.workunit.client.1.smithi135.stdout:7/424: mknod d4/db/d1a/d5c/d2c/d83/c8e 0 2024-03-30T04:28:20.642 INFO:tasks.workunit.client.0.smithi123.stdout:9/759: truncate d6/d1f/d45/d97/d69/dfb/ffc 871801 0 2024-03-30T04:28:20.644 INFO:tasks.workunit.client.0.smithi123.stdout:9/760: fsync d6/d1f/d72/db6/fe4 0 2024-03-30T04:28:20.645 INFO:tasks.workunit.client.1.smithi135.stdout:0/470: rename d0/d13/d53/d7f/d83/d9c/f5f to d0/d5/d9/d23/d2a/fa0 0 2024-03-30T04:28:20.650 INFO:tasks.workunit.client.1.smithi135.stdout:7/425: truncate d4/db/de/f28 4818786 0 2024-03-30T04:28:20.651 INFO:tasks.workunit.client.0.smithi123.stdout:9/761: mknod d6/d7c/de7/df1/c106 0 2024-03-30T04:28:20.654 INFO:tasks.workunit.client.0.smithi123.stdout:6/846: write d10/f68 [2984334,42516] 0 2024-03-30T04:28:20.657 INFO:tasks.workunit.client.0.smithi123.stdout:9/762: mknod d6/d1f/d45/d4c/c107 0 2024-03-30T04:28:20.657 INFO:tasks.workunit.client.1.smithi135.stdout:2/328: sync 2024-03-30T04:28:20.658 INFO:tasks.workunit.client.1.smithi135.stdout:1/333: dwrite d16/d19/d20/d26/f53 [0,4194304] 0 2024-03-30T04:28:20.658 INFO:tasks.workunit.client.0.smithi123.stdout:3/884: dread d13/d24/d31/d23/d3a/d3c/f8b [0,4194304] 0 2024-03-30T04:28:20.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.662+0000 7f21ee200700 1 -- 172.21.15.123:0/1443047125 --> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f21e8115e20 con 0x7f21cc06af10 2024-03-30T04:28:20.663 INFO:tasks.workunit.client.0.smithi123.stdout:6/847: chown d10/d82/d8e/dc7/d5b/l6b 10 1 2024-03-30T04:28:20.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.665+0000 7f21df400700 1 -- 172.21.15.123:0/1443047125 <== mgr.14608 v2:172.21.15.135:6828/3601553667 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f21e8115e20 con 0x7f21cc06af10 2024-03-30T04:28:20.664 INFO:tasks.workunit.client.1.smithi135.stdout:7/426: symlink d4/db/d1a/d4b/d4f/l8f 0 2024-03-30T04:28:20.667 INFO:tasks.workunit.client.0.smithi123.stdout:2/944: dwrite d0/d1f/d30/f94 [0,4194304] 0 2024-03-30T04:28:20.668 INFO:tasks.workunit.client.1.smithi135.stdout:9/365: dwrite d7/f77 [0,4194304] 0 2024-03-30T04:28:20.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.671+0000 7f21ee200700 1 -- 172.21.15.123:0/1443047125 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f21cc06af10 msgr2=0x7f21cc06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:28:20.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.671+0000 7f21ee200700 1 --2- 172.21.15.123:0/1443047125 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f21cc06af10 0x7f21cc06d3c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f21d4004f40 tx=0x7f21d400b040 comp rx=0 tx=0).stop 2024-03-30T04:28:20.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.671+0000 7f21ee200700 1 -- 172.21.15.123:0/1443047125 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f21e8066280 msgr2=0x7f21e8113fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:28:20.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.671+0000 7f21ee200700 1 --2- 172.21.15.123:0/1443047125 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f21e8066280 0x7f21e8113fb0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f21d800b3b0 tx=0x7f21d8009e30 comp rx=0 tx=0).stop 2024-03-30T04:28:20.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.671+0000 7f21ee200700 1 -- 172.21.15.123:0/1443047125 shutdown_connections 2024-03-30T04:28:20.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.671+0000 7f21ee200700 1 --2- 172.21.15.123:0/1443047125 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f21cc06af10 0x7f21cc06d3c0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:20.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.671+0000 7f21ee200700 1 --2- 172.21.15.123:0/1443047125 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f21e8066280 0x7f21e8113fb0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:20.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.671+0000 7f21ee200700 1 --2- 172.21.15.123:0/1443047125 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21e80fff60 0x7f21e81144f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:20.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.672+0000 7f21ee200700 1 -- 172.21.15.123:0/1443047125 >> 172.21.15.123:0/1443047125 conn(0x7f21e80f7430 msgr2=0x7f21e80f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:28:20.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.672+0000 7f21ee200700 1 -- 172.21.15.123:0/1443047125 shutdown_connections 2024-03-30T04:28:20.671 INFO:tasks.workunit.client.0.smithi123.stdout:6/848: creat d10/d82/d8e/dc7/d42/f11e x:0 0 0 2024-03-30T04:28:20.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.672+0000 7f21ee200700 1 -- 172.21.15.123:0/1443047125 wait complete. 2024-03-30T04:28:20.672 INFO:tasks.workunit.client.1.smithi135.stdout:5/397: dwrite d1/d3/d2b/f2f [0,4194304] 0 2024-03-30T04:28:20.677 INFO:tasks.workunit.client.1.smithi135.stdout:0/471: dwrite d0/d13/f74 [4194304,4194304] 0 2024-03-30T04:28:20.679 INFO:tasks.workunit.client.0.smithi123.stdout:6/849: fdatasync d10/d82/dc1/d45/fd3 0 2024-03-30T04:28:20.681 INFO:tasks.workunit.client.0.smithi123.stdout:3/885: rename d13/d24/d31/d23/d3a/lbf to d13/d24/d39/d79/db9/l120 0 2024-03-30T04:28:20.683 INFO:tasks.workunit.client.1.smithi135.stdout:2/329: mknod d4/d32/d43/c6c 0 2024-03-30T04:28:20.684 INFO:tasks.workunit.client.0.smithi123.stdout:2/945: link d0/d1f/cd d0/c131 0 2024-03-30T04:28:20.685 INFO:tasks.workunit.client.1.smithi135.stdout:0/472: readlink d0/d13/d53/d7f/d83/d9c/l6c 0 2024-03-30T04:28:20.688 INFO:tasks.workunit.client.0.smithi123.stdout:3/886: chown d13/d24/d31/d23/c3b 44942 1 2024-03-30T04:28:20.691 INFO:tasks.workunit.client.0.smithi123.stdout:2/946: symlink d0/d1f/d27/l132 0 2024-03-30T04:28:20.691 INFO:tasks.workunit.client.0.smithi123.stdout:8/815: dread da/d27/d56/d6c/d96/dc2/ff4 [0,4194304] 0 2024-03-30T04:28:20.693 INFO:tasks.workunit.client.1.smithi135.stdout:7/427: write d4/db/d1a/d5c/d2c/d51/d56/d81/f7f [686595,104834] 0 2024-03-30T04:28:20.697 INFO:tasks.workunit.client.1.smithi135.stdout:9/366: mknod d7/c7f 0 2024-03-30T04:28:20.702 INFO:tasks.workunit.client.1.smithi135.stdout:1/334: creat d16/d19/f72 x:0 0 0 2024-03-30T04:28:20.704 INFO:tasks.workunit.client.0.smithi123.stdout:3/887: creat d13/d24/d31/d23/d3a/d3c/db8/d10b/f121 x:0 0 0 2024-03-30T04:28:20.705 INFO:tasks.workunit.client.0.smithi123.stdout:8/816: rename da/dc/d14/d18/d21 to da/d27/d3c/d64/d78/db8/dfd 0 2024-03-30T04:28:20.708 INFO:tasks.workunit.client.1.smithi135.stdout:3/398: dwrite d3/d50/f36 [0,4194304] 0 2024-03-30T04:28:20.710 INFO:tasks.workunit.client.0.smithi123.stdout:7/767: sync 2024-03-30T04:28:20.711 INFO:tasks.workunit.client.1.smithi135.stdout:1/335: chown d16/d19/f56 2 1 2024-03-30T04:28:20.711 INFO:tasks.workunit.client.0.smithi123.stdout:4/752: sync 2024-03-30T04:28:20.711 INFO:tasks.workunit.client.0.smithi123.stdout:1/830: sync 2024-03-30T04:28:20.711 INFO:tasks.workunit.client.1.smithi135.stdout:2/330: mkdir d4/d32/d12/d45/d6d 0 2024-03-30T04:28:20.712 INFO:tasks.workunit.client.1.smithi135.stdout:3/399: fdatasync d3/d54/d5c/f78 0 2024-03-30T04:28:20.715 INFO:tasks.workunit.client.0.smithi123.stdout:1/831: readlink db/l2d 0 2024-03-30T04:28:20.715 INFO:tasks.workunit.client.1.smithi135.stdout:1/336: readlink d16/d29/d2c/d3c/l70 0 2024-03-30T04:28:20.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:20 smithi123 ceph-mon[35245]: pgmap v43: 65 pgs: 65 active+clean; 1.8 GiB data, 6.8 GiB used, 530 GiB / 536 GiB avail; 50 MiB/s rd, 170 MiB/s wr, 328 op/s 2024-03-30T04:28:20.720 INFO:tasks.workunit.client.1.smithi135.stdout:1/337: write d16/d1a/f23 [436392,9184] 0 2024-03-30T04:28:20.721 INFO:tasks.workunit.client.0.smithi123.stdout:6/850: dwrite d10/d82/dc1/d6a/d7a/de8/ff4 [0,4194304] 0 2024-03-30T04:28:20.722 INFO:tasks.workunit.client.0.smithi123.stdout:7/768: creat d2/dc/d49/d68/ffe x:0 0 0 2024-03-30T04:28:20.722 INFO:tasks.workunit.client.0.smithi123.stdout:2/947: dread d0/d1f/d1c/d39/fd1 [0,4194304] 0 2024-03-30T04:28:20.724 INFO:tasks.workunit.client.0.smithi123.stdout:7/769: dread - d2/dc/f81 zero size 2024-03-30T04:28:20.727 INFO:tasks.workunit.client.0.smithi123.stdout:2/948: chown d0/d1f/f8f 31337 1 2024-03-30T04:28:20.729 INFO:tasks.workunit.client.0.smithi123.stdout:4/753: creat d2/d4/f101 x:0 0 0 2024-03-30T04:28:20.731 INFO:tasks.workunit.client.1.smithi135.stdout:1/338: dread - d16/d19/d20/f65 zero size 2024-03-30T04:28:20.733 INFO:tasks.workunit.client.0.smithi123.stdout:8/817: symlink da/d27/d3c/d64/lfe 0 2024-03-30T04:28:20.733 INFO:tasks.workunit.client.0.smithi123.stdout:1/832: rmdir db 39 2024-03-30T04:28:20.735 INFO:tasks.workunit.client.1.smithi135.stdout:9/367: dwrite d7/d1b/f3d [0,4194304] 0 2024-03-30T04:28:20.736 INFO:tasks.workunit.client.0.smithi123.stdout:7/770: unlink d2/da/d11/d34/d76/fa7 0 2024-03-30T04:28:20.743 INFO:tasks.workunit.client.0.smithi123.stdout:0/796: sync 2024-03-30T04:28:20.744 INFO:tasks.workunit.client.1.smithi135.stdout:7/428: dwrite d4/db/d1a/f24 [0,4194304] 0 2024-03-30T04:28:20.745 INFO:tasks.workunit.client.0.smithi123.stdout:9/763: truncate d6/d1f/d45/d97/d69/fab 416436 0 2024-03-30T04:28:20.746 INFO:tasks.workunit.client.0.smithi123.stdout:0/797: fsync d6/dd/d15/d28/dd3/fd9 0 2024-03-30T04:28:20.748 INFO:tasks.workunit.client.1.smithi135.stdout:9/368: write d7/d29/f3e [650611,65092] 0 2024-03-30T04:28:20.750 INFO:tasks.workunit.client.0.smithi123.stdout:0/798: write d6/dd/d31/db8/ff5 [313322,48628] 0 2024-03-30T04:28:20.752 INFO:tasks.workunit.client.1.smithi135.stdout:2/331: dwrite d4/d32/d2e/f48 [0,4194304] 0 2024-03-30T04:28:20.754 INFO:tasks.workunit.client.1.smithi135.stdout:7/429: truncate d4/db/d1a/d5c/d2c/f63 678486 0 2024-03-30T04:28:20.758 INFO:tasks.workunit.client.1.smithi135.stdout:2/332: chown d4/d32/d2e/f4a 2632807 1 2024-03-30T04:28:20.759 INFO:tasks.workunit.client.1.smithi135.stdout:0/473: truncate d0/d1/d16/f32 3850308 0 2024-03-30T04:28:20.762 INFO:tasks.workunit.client.0.smithi123.stdout:3/888: write d13/d24/d31/d23/d3a/d3c/f8b [1471783,29699] 0 2024-03-30T04:28:20.765 INFO:tasks.workunit.client.0.smithi123.stdout:1/833: fdatasync db/dd/d42/d46/d51/fda 0 2024-03-30T04:28:20.766 INFO:tasks.workunit.client.1.smithi135.stdout:1/339: getdents d16/d19/d20 0 2024-03-30T04:28:20.770 INFO:tasks.workunit.client.1.smithi135.stdout:2/333: symlink d4/d32/d46/l6e 0 2024-03-30T04:28:20.774 INFO:tasks.workunit.client.0.smithi123.stdout:8/818: dwrite da/d27/d56/f65 [4194304,4194304] 0 2024-03-30T04:28:20.776 INFO:tasks.workunit.client.0.smithi123.stdout:6/851: write d10/d20/f40 [3060585,13111] 0 2024-03-30T04:28:20.777 INFO:tasks.workunit.client.1.smithi135.stdout:0/474: link d0/d5/d9/f33 d0/d5/d10/fa1 0 2024-03-30T04:28:20.780 INFO:tasks.workunit.client.0.smithi123.stdout:7/771: mknod d2/da/d11/d61/d55/d59/db5/dab/dca/cff 0 2024-03-30T04:28:20.781 INFO:tasks.workunit.client.0.smithi123.stdout:8/819: write da/d27/d3c/d2d/d5c/fc7 [1263387,70002] 0 2024-03-30T04:28:20.783 INFO:tasks.workunit.client.1.smithi135.stdout:3/400: dwrite d3/d50/d46/f1c [0,4194304] 0 2024-03-30T04:28:20.783 INFO:tasks.workunit.client.0.smithi123.stdout:7/772: readlink d2/dc/d1c/l31 0 2024-03-30T04:28:20.791 INFO:tasks.workunit.client.1.smithi135.stdout:0/475: symlink d0/d5/d42/d87/la2 0 2024-03-30T04:28:20.793 INFO:tasks.workunit.client.0.smithi123.stdout:1/834: write db/dd/d5f/d6d/d85/d89/fa5 [4035976,64831] 0 2024-03-30T04:28:20.793 INFO:tasks.workunit.client.1.smithi135.stdout:3/401: fdatasync d3/d50/f12 0 2024-03-30T04:28:20.795 INFO:tasks.workunit.client.1.smithi135.stdout:0/476: write d0/d5/d9/d23/d58/d63/f89 [397669,64411] 0 2024-03-30T04:28:20.795 INFO:tasks.workunit.client.0.smithi123.stdout:0/799: dread d6/dd/d31/db8/ff5 [0,4194304] 0 2024-03-30T04:28:20.796 INFO:tasks.workunit.client.0.smithi123.stdout:4/754: link d2/d4/db3/ced d2/d4/d2c/d42/dad/de6/c102 0 2024-03-30T04:28:20.799 INFO:tasks.workunit.client.0.smithi123.stdout:6/852: mknod d10/d82/dc1/d6a/c11f 0 2024-03-30T04:28:20.801 INFO:tasks.workunit.client.0.smithi123.stdout:5/746: sync 2024-03-30T04:28:20.801 INFO:tasks.workunit.client.0.smithi123.stdout:8/820: mkdir da/dc/d14/d9d/dff 0 2024-03-30T04:28:20.803 INFO:tasks.workunit.client.0.smithi123.stdout:3/889: rename d13/f8a to d13/d24/d31/d23/d3a/d3c/d4e/dbe/f122 0 2024-03-30T04:28:20.804 INFO:tasks.workunit.client.1.smithi135.stdout:0/477: creat d0/d13/fa3 x:0 0 0 2024-03-30T04:28:20.806 INFO:tasks.workunit.client.0.smithi123.stdout:1/835: fdatasync db/dd/f4c 0 2024-03-30T04:28:20.807 INFO:tasks.workunit.client.1.smithi135.stdout:9/369: fdatasync d7/d1b/f3d 0 2024-03-30T04:28:20.807 INFO:tasks.workunit.client.1.smithi135.stdout:3/402: read d3/d10/d16/f1b [2604269,92201] 0 2024-03-30T04:28:20.810 INFO:tasks.workunit.client.0.smithi123.stdout:3/890: chown d13/d24/d31/d30/f56 0 1 2024-03-30T04:28:20.811 INFO:tasks.workunit.client.0.smithi123.stdout:0/800: mknod d6/dd/d15/d28/c101 0 2024-03-30T04:28:20.819 INFO:tasks.workunit.client.0.smithi123.stdout:0/801: fsync d6/dd/d15/d98/d4e/d9a/db2/feb 0 2024-03-30T04:28:20.827 INFO:tasks.workunit.client.1.smithi135.stdout:1/340: dread d16/d29/d32/f38 [0,4194304] 0 2024-03-30T04:28:20.827 INFO:tasks.workunit.client.1.smithi135.stdout:3/403: write d3/d6e/f80 [2881706,115197] 0 2024-03-30T04:28:20.827 INFO:tasks.workunit.client.1.smithi135.stdout:1/341: write d16/d19/d20/d26/f53 [3068246,106018] 0 2024-03-30T04:28:20.828 INFO:tasks.workunit.client.0.smithi123.stdout:9/764: truncate d6/d1f/d92/fcc 706190 0 2024-03-30T04:28:20.828 INFO:tasks.workunit.client.0.smithi123.stdout:7/773: dwrite d2/da/d22/d3b/ff6 [0,4194304] 0 2024-03-30T04:28:20.828 INFO:tasks.workunit.client.0.smithi123.stdout:0/802: stat d6/dd/d15/d28/fcc 0 2024-03-30T04:28:20.828 INFO:tasks.workunit.client.0.smithi123.stdout:9/765: stat d6/d7c/de7/df1 0 2024-03-30T04:28:20.832 INFO:tasks.workunit.client.1.smithi135.stdout:1/342: write d16/d1a/f4d [854170,6116] 0 2024-03-30T04:28:20.832 INFO:tasks.workunit.client.1.smithi135.stdout:3/404: fsync d3/d50/d22/f3d 0 2024-03-30T04:28:20.835 INFO:tasks.workunit.client.1.smithi135.stdout:7/430: dread d4/db/de/f1d [0,4194304] 0 2024-03-30T04:28:20.839 INFO:tasks.workunit.client.0.smithi123.stdout:6/853: creat d10/d82/d9a/f120 x:0 0 0 2024-03-30T04:28:20.839 INFO:tasks.workunit.client.0.smithi123.stdout:2/949: dread d0/d1f/d30/d119/f49 [0,4194304] 0 2024-03-30T04:28:20.840 INFO:tasks.workunit.client.1.smithi135.stdout:1/343: creat d16/d5a/d5f/d3a/f73 x:0 0 0 2024-03-30T04:28:20.842 INFO:tasks.workunit.client.0.smithi123.stdout:2/950: write d0/d1f/d16/f23 [852553,105418] 0 2024-03-30T04:28:20.844 INFO:tasks.workunit.client.1.smithi135.stdout:3/405: link d3/db/f8d d3/db/f8f 0 2024-03-30T04:28:20.844 INFO:tasks.workunit.client.1.smithi135.stdout:1/344: dread - d16/d5a/d5f/d3a/f73 zero size 2024-03-30T04:28:20.845 INFO:tasks.workunit.client.0.smithi123.stdout:2/951: dread - d0/d1f/d16/f12e zero size 2024-03-30T04:28:20.845 INFO:tasks.workunit.client.0.smithi123.stdout:9/766: creat d6/d1f/d72/d7d/f108 x:0 0 0 2024-03-30T04:28:20.848 INFO:tasks.workunit.client.0.smithi123.stdout:9/767: dread - d6/d1f/d72/db6/fde zero size 2024-03-30T04:28:20.849 INFO:tasks.workunit.client.1.smithi135.stdout:3/406: write d3/d10/f31 [1679693,94667] 0 2024-03-30T04:28:20.850 INFO:tasks.workunit.client.0.smithi123.stdout:8/821: write da/d27/d3c/d2d/dae/dd9/fe2 [2058174,46232] 0 2024-03-30T04:28:20.853 INFO:tasks.workunit.client.1.smithi135.stdout:3/407: write d3/d50/d22/f3d [1486850,16118] 0 2024-03-30T04:28:20.853 INFO:tasks.workunit.client.1.smithi135.stdout:1/345: link d16/d19/d20/l31 d16/d29/l74 0 2024-03-30T04:28:20.854 INFO:tasks.workunit.client.0.smithi123.stdout:4/755: getdents d2/d4 0 2024-03-30T04:28:20.856 INFO:tasks.workunit.client.0.smithi123.stdout:1/836: write db/dd/d42/d46/f29 [2321005,69825] 0 2024-03-30T04:28:20.856 INFO:tasks.workunit.client.0.smithi123.stdout:8/822: chown da/d27/d56/d6c/d6d/f90 5047787 1 2024-03-30T04:28:20.856 INFO:tasks.workunit.client.1.smithi135.stdout:3/408: creat d3/d6e/f90 x:0 0 0 2024-03-30T04:28:20.862 INFO:tasks.workunit.client.1.smithi135.stdout:3/409: mkdir d3/d50/d22/d8b/d91 0 2024-03-30T04:28:20.864 INFO:tasks.workunit.client.0.smithi123.stdout:9/768: read d6/d1f/f3f [2517220,128759] 0 2024-03-30T04:28:20.866 INFO:tasks.workunit.client.0.smithi123.stdout:3/891: rename d13/d24/d31/d23/d3a/d3c/d4e to d13/d24/d31/d123 0 2024-03-30T04:28:20.868 INFO:tasks.workunit.client.1.smithi135.stdout:0/478: dread d0/d1/f43 [0,4194304] 0 2024-03-30T04:28:20.868 INFO:tasks.workunit.client.0.smithi123.stdout:7/774: link d2/dc/d49/l8c d2/da/d11/d34/ded/dfc/l100 0 2024-03-30T04:28:20.869 INFO:tasks.workunit.client.0.smithi123.stdout:5/747: getdents dd/d19/d1f/d3c 0 2024-03-30T04:28:20.869 INFO:tasks.workunit.client.1.smithi135.stdout:3/410: rename d3/db/f34 to d3/f92 0 2024-03-30T04:28:20.874 INFO:tasks.workunit.client.0.smithi123.stdout:6/854: write d10/d82/d8e/dc7/d5b/fa8 [330972,122316] 0 2024-03-30T04:28:20.874 INFO:tasks.workunit.client.1.smithi135.stdout:0/479: creat d0/d5/d9/d23/fa4 x:0 0 0 2024-03-30T04:28:20.875 INFO:tasks.workunit.client.0.smithi123.stdout:5/748: dread dd/d19/d1f/d3c/d72/fb7 [0,4194304] 0 2024-03-30T04:28:20.879 INFO:tasks.workunit.client.0.smithi123.stdout:3/892: unlink c5 0 2024-03-30T04:28:20.880 INFO:tasks.workunit.client.1.smithi135.stdout:1/346: dwrite d16/d29/d2c/d3c/f3b [0,4194304] 0 2024-03-30T04:28:20.880 INFO:tasks.workunit.client.1.smithi135.stdout:0/480: symlink d0/d13/d53/la5 0 2024-03-30T04:28:20.882 INFO:tasks.workunit.client.0.smithi123.stdout:7/775: creat d2/dc/d7f/f101 x:0 0 0 2024-03-30T04:28:20.882 INFO:tasks.workunit.client.1.smithi135.stdout:7/431: dread d4/db/f1c [0,4194304] 0 2024-03-30T04:28:20.884 INFO:tasks.workunit.client.0.smithi123.stdout:0/803: getdents d6/dd/d15/d28 0 2024-03-30T04:28:20.884 INFO:tasks.workunit.client.1.smithi135.stdout:3/411: dread d3/d50/d22/d74/f7c [0,4194304] 0 2024-03-30T04:28:20.885 INFO:tasks.workunit.client.1.smithi135.stdout:1/347: symlink d16/d1a/l75 0 2024-03-30T04:28:20.885 INFO:tasks.workunit.client.0.smithi123.stdout:9/769: mkdir d6/d1f/d72/d109 0 2024-03-30T04:28:20.886 INFO:tasks.workunit.client.0.smithi123.stdout:4/756: dread d2/d4/d2c/d42/d35/d5d/d6f/f73 [0,4194304] 0 2024-03-30T04:28:20.887 INFO:tasks.workunit.client.1.smithi135.stdout:1/348: readlink d16/d29/d2c/l36 0 2024-03-30T04:28:20.887 INFO:tasks.workunit.client.0.smithi123.stdout:8/823: creat da/d27/d56/f100 x:0 0 0 2024-03-30T04:28:20.888 INFO:tasks.workunit.client.1.smithi135.stdout:7/432: write d4/db/d1a/d5c/d2c/d51/f7e [977581,31040] 0 2024-03-30T04:28:20.888 INFO:tasks.workunit.client.1.smithi135.stdout:1/349: dread - d16/d19/f72 zero size 2024-03-30T04:28:20.890 INFO:tasks.workunit.client.1.smithi135.stdout:1/350: dread - d16/d19/f72 zero size 2024-03-30T04:28:20.891 INFO:tasks.workunit.client.0.smithi123.stdout:2/952: getdents d0/d1f/d27/d68 0 2024-03-30T04:28:20.893 INFO:tasks.workunit.client.1.smithi135.stdout:3/412: mknod d3/d50/d22/d74/c93 0 2024-03-30T04:28:20.893 INFO:tasks.workunit.client.0.smithi123.stdout:6/855: symlink d10/d82/dc1/d6a/d7a/de2/df1/d112/l121 0 2024-03-30T04:28:20.893 INFO:tasks.workunit.client.1.smithi135.stdout:1/351: read f3 [2553169,102949] 0 2024-03-30T04:28:20.894 INFO:tasks.workunit.client.0.smithi123.stdout:1/837: getdents db/dd/d5f/d6d/d85 0 2024-03-30T04:28:20.894 INFO:tasks.workunit.client.1.smithi135.stdout:7/433: mkdir d4/db/d1a/d90 0 2024-03-30T04:28:20.894 INFO:tasks.workunit.client.1.smithi135.stdout:3/413: read - d3/d10/f5e zero size 2024-03-30T04:28:20.895 INFO:tasks.workunit.client.0.smithi123.stdout:9/770: mkdir d6/d1f/d45/d97/d10a 0 2024-03-30T04:28:20.897 INFO:tasks.workunit.client.0.smithi123.stdout:7/776: mknod d2/dc/d56/c102 0 2024-03-30T04:28:20.897 INFO:tasks.workunit.client.1.smithi135.stdout:3/414: chown d3/d50/d22/d74 2402066 1 2024-03-30T04:28:20.899 INFO:tasks.workunit.client.0.smithi123.stdout:3/893: creat d13/d24/d31/d23/d3a/d3c/d114/d119/f124 x:0 0 0 2024-03-30T04:28:20.901 INFO:tasks.workunit.client.1.smithi135.stdout:1/352: creat d16/d29/d2c/f76 x:0 0 0 2024-03-30T04:28:20.901 INFO:tasks.workunit.client.0.smithi123.stdout:4/757: mkdir d2/d4/d7/dc1/d103 0 2024-03-30T04:28:20.902 INFO:tasks.workunit.client.1.smithi135.stdout:7/434: creat d4/db/d1a/d5c/d59/d76/f91 x:0 0 0 2024-03-30T04:28:20.904 INFO:tasks.workunit.client.1.smithi135.stdout:3/415: unlink d3/d10/d16/d41/f53 0 2024-03-30T04:28:20.904 INFO:tasks.workunit.client.0.smithi123.stdout:1/838: unlink db/dd/d42/d46/d51/d72/fc8 0 2024-03-30T04:28:20.908 INFO:tasks.workunit.client.1.smithi135.stdout:1/353: mkdir d16/d5a/d6b/d77 0 2024-03-30T04:28:20.909 INFO:tasks.workunit.client.0.smithi123.stdout:8/824: rmdir da/d27/d56 39 2024-03-30T04:28:20.909 INFO:tasks.workunit.client.1.smithi135.stdout:0/481: dwrite d0/d5/d9/f33 [8388608,4194304] 0 2024-03-30T04:28:20.910 INFO:tasks.workunit.client.0.smithi123.stdout:9/771: rename d6/l1e to d6/d1f/d72/d7d/l10b 0 2024-03-30T04:28:20.912 INFO:tasks.workunit.client.1.smithi135.stdout:3/416: rename d3/d10/d81 to d3/d10/d16/d23/d6a/d94 0 2024-03-30T04:28:20.913 INFO:tasks.workunit.client.0.smithi123.stdout:0/804: truncate d6/dd/d31/d35/d3a/f8f 1917987 0 2024-03-30T04:28:20.913 INFO:tasks.workunit.client.0.smithi123.stdout:6/856: dread d10/fb2 [0,4194304] 0 2024-03-30T04:28:20.915 INFO:tasks.workunit.client.1.smithi135.stdout:3/417: read - d3/d10/d16/d23/d6a/d86/f8c zero size 2024-03-30T04:28:20.916 INFO:tasks.workunit.client.1.smithi135.stdout:1/354: creat d16/d29/d2c/d3c/f78 x:0 0 0 2024-03-30T04:28:20.916 INFO:tasks.workunit.client.0.smithi123.stdout:8/825: write da/d27/d3c/d2d/d5c/fc7 [11969866,72779] 0 2024-03-30T04:28:20.918 INFO:tasks.workunit.client.1.smithi135.stdout:7/435: write d4/f16 [1188502,46769] 0 2024-03-30T04:28:20.918 INFO:tasks.workunit.client.1.smithi135.stdout:0/482: write d0/d5/d9/d23/d2a/fa0 [5199367,88119] 0 2024-03-30T04:28:20.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.919+0000 7fee92a00700 1 -- 172.21.15.123:0/3455316661 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee8c0fdcf0 msgr2=0x7fee8c100170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:28:20.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.919+0000 7fee92a00700 1 --2- 172.21.15.123:0/3455316661 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee8c0fdcf0 0x7fee8c100170 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fee7c0049c0 tx=0x7fee7c009e30 comp rx=0 tx=0).stop 2024-03-30T04:28:20.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.919+0000 7fee92a00700 1 -- 172.21.15.123:0/3455316661 shutdown_connections 2024-03-30T04:28:20.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.919+0000 7fee92a00700 1 --2- 172.21.15.123:0/3455316661 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee8c0fdcf0 0x7fee8c100170 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:20.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.919+0000 7fee92a00700 1 --2- 172.21.15.123:0/3455316661 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fee8c0662e0 0x7fee8c0fd7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:20.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.919+0000 7fee92a00700 1 -- 172.21.15.123:0/3455316661 >> 172.21.15.123:0/3455316661 conn(0x7fee8c0f7430 msgr2=0x7fee8c0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:28:20.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.919+0000 7fee92a00700 1 -- 172.21.15.123:0/3455316661 shutdown_connections 2024-03-30T04:28:20.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.920+0000 7fee92a00700 1 -- 172.21.15.123:0/3455316661 wait complete. 2024-03-30T04:28:20.920 INFO:tasks.workunit.client.0.smithi123.stdout:8/826: chown da/dc/f10 1332188 1 2024-03-30T04:28:20.921 INFO:tasks.workunit.client.1.smithi135.stdout:7/436: write d4/db/f26 [360304,125394] 0 2024-03-30T04:28:20.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.921+0000 7fee92a00700 1 Processor -- start 2024-03-30T04:28:20.923 INFO:tasks.workunit.client.1.smithi135.stdout:3/418: link d3/cc d3/d54/d5c/c95 0 2024-03-30T04:28:20.923 INFO:tasks.workunit.client.1.smithi135.stdout:1/355: read d16/d19/f1f [3479677,40620] 0 2024-03-30T04:28:20.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.921+0000 7fee92a00700 1 -- start start 2024-03-30T04:28:20.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.922+0000 7fee92a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee8c0662e0 0x7fee8c10b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:28:20.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.922+0000 7fee92a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fee8c0fdcf0 0x7fee8c10bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:28:20.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.923+0000 7fee92a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee8c10c1f0 con 0x7fee8c0662e0 2024-03-30T04:28:20.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.923+0000 7fee92a00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee8c10ca40 con 0x7fee8c0fdcf0 2024-03-30T04:28:20.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.923+0000 7fee8be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee8c0662e0 0x7fee8c10b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:28:20.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.923+0000 7fee8be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee8c0662e0 0x7fee8c10b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58304/0 (socket says 172.21.15.123:58304) 2024-03-30T04:28:20.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.923+0000 7fee8b400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fee8c0fdcf0 0x7fee8c10bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:28:20.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.923+0000 7fee8be00700 1 -- 172.21.15.123:0/1356209257 learned_addr learned my addr 172.21.15.123:0/1356209257 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:28:20.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.924+0000 7fee8b400700 1 -- 172.21.15.123:0/1356209257 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee8c0662e0 msgr2=0x7fee8c10b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:28:20.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.924+0000 7fee8b400700 1 --2- 172.21.15.123:0/1356209257 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee8c0662e0 0x7fee8c10b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:20.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.924+0000 7fee8b400700 1 -- 172.21.15.123:0/1356209257 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee7c005040 con 0x7fee8c0fdcf0 2024-03-30T04:28:20.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.925+0000 7fee8b400700 1 --2- 172.21.15.123:0/1356209257 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fee8c0fdcf0 0x7fee8c10bbf0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fee7c004910 tx=0x7fee7c009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:28:20.925 INFO:tasks.workunit.client.1.smithi135.stdout:3/419: dread - d3/d50/f5b zero size 2024-03-30T04:28:20.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.926+0000 7fee83e00700 1 -- 172.21.15.123:0/1356209257 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee7c01c8b0 con 0x7fee8c0fdcf0 2024-03-30T04:28:20.925 INFO:tasks.workunit.client.0.smithi123.stdout:4/758: mkdir d2/d4/d2c/d42/d35/d5d/d78/d104 0 2024-03-30T04:28:20.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.926+0000 7fee92a00700 1 -- 172.21.15.123:0/1356209257 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee8c10ccc0 con 0x7fee8c0fdcf0 2024-03-30T04:28:20.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.927+0000 7fee83e00700 1 -- 172.21.15.123:0/1356209257 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fee7c01cec0 con 0x7fee8c0fdcf0 2024-03-30T04:28:20.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.927+0000 7fee83e00700 1 -- 172.21.15.123:0/1356209257 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee7c024d40 con 0x7fee8c0fdcf0 2024-03-30T04:28:20.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.927+0000 7fee92a00700 1 -- 172.21.15.123:0/1356209257 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee8c10d200 con 0x7fee8c0fdcf0 2024-03-30T04:28:20.926 INFO:tasks.workunit.client.1.smithi135.stdout:7/437: mkdir d4/db/d79/d92 0 2024-03-30T04:28:20.927 INFO:tasks.workunit.client.1.smithi135.stdout:1/356: write d16/d29/d2c/d43/d50/f5b [2213271,29377] 0 2024-03-30T04:28:20.927 INFO:tasks.workunit.client.1.smithi135.stdout:3/420: chown d3/d50/l40 35035 1 2024-03-30T04:28:20.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.929+0000 7fee92a00700 1 -- 172.21.15.123:0/1356209257 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee8c05f5f0 con 0x7fee8c0fdcf0 2024-03-30T04:28:20.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.931+0000 7fee83e00700 1 -- 172.21.15.123:0/1356209257 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 24) v1 ==== 88766+0+0 (secure 0 0 0) 0x7fee7c021030 con 0x7fee8c0fdcf0 2024-03-30T04:28:20.930 INFO:tasks.workunit.client.0.smithi123.stdout:7/777: rename d2/dc/d49/d68/f78 to d2/da/d11/d34/f103 0 2024-03-30T04:28:20.932 INFO:tasks.workunit.client.1.smithi135.stdout:7/438: unlink d4/db/de/l55 0 2024-03-30T04:28:20.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.933+0000 7fee83e00700 1 --2- 172.21.15.123:0/1356209257 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7fee7006af10 0x7fee7006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:28:20.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.933+0000 7fee83e00700 1 -- 172.21.15.123:0/1356209257 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7fee7c09b8b0 con 0x7fee8c0fdcf0 2024-03-30T04:28:20.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.934+0000 7fee8be00700 1 --2- 172.21.15.123:0/1356209257 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7fee7006af10 0x7fee7006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:28:20.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.936+0000 7fee8be00700 1 --2- 172.21.15.123:0/1356209257 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7fee7006af10 0x7fee7006d3c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fee74009fd0 tx=0x7fee7400b040 comp rx=0 tx=0).ready entity=mgr.14608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:28:20.934 INFO:tasks.workunit.client.0.smithi123.stdout:2/953: dread d0/d1f/d5e/fb9 [0,4194304] 0 2024-03-30T04:28:20.935 INFO:tasks.workunit.client.0.smithi123.stdout:4/759: truncate d2/d4/d2c/d42/d35/d5d/d78/f83 3236074 0 2024-03-30T04:28:20.936 INFO:tasks.workunit.client.0.smithi123.stdout:3/894: symlink d13/d24/d31/d123/d5d/da5/d104/l125 0 2024-03-30T04:28:20.939 INFO:tasks.workunit.client.0.smithi123.stdout:0/805: mknod d6/dd/d15/d98/c102 0 2024-03-30T04:28:20.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:20.939+0000 7fee83e00700 1 -- 172.21.15.123:0/1356209257 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fee7c069850 con 0x7fee8c0fdcf0 2024-03-30T04:28:20.940 INFO:tasks.workunit.client.1.smithi135.stdout:3/421: getdents d3/d54 0 2024-03-30T04:28:20.941 INFO:tasks.workunit.client.1.smithi135.stdout:7/439: symlink d4/db/d1a/l93 0 2024-03-30T04:28:20.944 INFO:tasks.workunit.client.1.smithi135.stdout:7/440: chown d4/db/d1a/d5c/d2c/d51/d56/c5e 60220187 1 2024-03-30T04:28:20.945 INFO:tasks.workunit.client.0.smithi123.stdout:0/806: readlink d6/dd/d15/d98/d4e/l64 0 2024-03-30T04:28:20.947 INFO:tasks.workunit.client.1.smithi135.stdout:7/441: write d4/db/d1a/d5c/d5d/f8d [1613,78101] 0 2024-03-30T04:28:20.951 INFO:tasks.workunit.client.1.smithi135.stdout:7/442: getdents d4/db/d1a/d5c/d82 0 2024-03-30T04:28:20.951 INFO:tasks.workunit.client.0.smithi123.stdout:8/827: rename da/d27/d3c/d64/d78/db8/dfd/c61 to da/dc/d14/d9c/c101 0 2024-03-30T04:28:20.951 INFO:tasks.workunit.client.1.smithi135.stdout:1/357: truncate d16/d1a/f21 4380342 0 2024-03-30T04:28:20.952 INFO:tasks.workunit.client.1.smithi135.stdout:3/422: write d3/db/f1a [855968,95206] 0 2024-03-30T04:28:20.954 INFO:tasks.workunit.client.1.smithi135.stdout:7/443: mkdir d4/db/d1a/d94 0 2024-03-30T04:28:20.955 INFO:tasks.workunit.client.0.smithi123.stdout:9/772: mkdir d6/d1f/d10c 0 2024-03-30T04:28:20.959 INFO:tasks.workunit.client.1.smithi135.stdout:3/423: symlink d3/d10/l96 0 2024-03-30T04:28:20.959 INFO:tasks.workunit.client.0.smithi123.stdout:7/778: mkdir d2/dc/d49/d104 0 2024-03-30T04:28:20.959 INFO:tasks.workunit.client.0.smithi123.stdout:1/839: dwrite db/dd/d42/d46/f79 [0,4194304] 0 2024-03-30T04:28:20.959 INFO:tasks.workunit.client.0.smithi123.stdout:8/828: write da/dc/d14/d18/f30 [1915528,35728] 0 2024-03-30T04:28:20.961 INFO:tasks.workunit.client.0.smithi123.stdout:3/895: fdatasync d13/d24/d39/d79/d90/fc2 0 2024-03-30T04:28:20.961 INFO:tasks.workunit.client.1.smithi135.stdout:0/483: dread d0/d13/d53/d54/f5b [0,4194304] 0 2024-03-30T04:28:20.965 INFO:tasks.workunit.client.1.smithi135.stdout:3/424: creat d3/d50/d22/d8b/d91/f97 x:0 0 0 2024-03-30T04:28:20.966 INFO:tasks.workunit.client.0.smithi123.stdout:0/807: fdatasync d6/dd/d15/d98/d4e/f68 0 2024-03-30T04:28:20.968 INFO:tasks.workunit.client.1.smithi135.stdout:0/484: fsync d0/d5/d9/d23/d58/d63/f89 0 2024-03-30T04:28:20.974 INFO:tasks.workunit.client.1.smithi135.stdout:3/425: mkdir d3/d10/d16/d23/d6a/d86/d98 0 2024-03-30T04:28:20.976 INFO:tasks.workunit.client.0.smithi123.stdout:2/954: dwrite d0/d1f/d27/d68/dd6/d8c/f90 [0,4194304] 0 2024-03-30T04:28:20.976 INFO:tasks.workunit.client.0.smithi123.stdout:1/840: read db/dd/d42/d46/d51/f55 [2577033,38764] 0 2024-03-30T04:28:20.976 INFO:tasks.workunit.client.1.smithi135.stdout:0/485: symlink d0/d13/d53/d54/d99/la6 0 2024-03-30T04:28:20.979 INFO:tasks.workunit.client.0.smithi123.stdout:4/760: creat d2/d4/d7/f105 x:0 0 0 2024-03-30T04:28:20.981 INFO:tasks.workunit.client.1.smithi135.stdout:1/358: dwrite d16/d5a/d5f/d3a/f52 [0,4194304] 0 2024-03-30T04:28:20.982 INFO:tasks.workunit.client.0.smithi123.stdout:6/857: dwrite d10/d82/dc1/d6a/d7a/d9f/fd4 [0,4194304] 0 2024-03-30T04:28:20.983 INFO:tasks.workunit.client.0.smithi123.stdout:9/773: mkdir d6/d1f/d72/d10d 0 2024-03-30T04:28:20.985 INFO:tasks.workunit.client.0.smithi123.stdout:3/896: read d13/d24/d31/d23/f76 [1112155,94162] 0 2024-03-30T04:28:20.990 INFO:tasks.workunit.client.0.smithi123.stdout:7/779: rename d2/da/d11/d61/d55/d94/f99 to d2/da/d11/d61/d55/d59/f105 0 2024-03-30T04:28:20.990 INFO:tasks.workunit.client.1.smithi135.stdout:7/444: dwrite d4/db/d1a/d3d/f5b [0,4194304] 0 2024-03-30T04:28:20.991 INFO:tasks.workunit.client.1.smithi135.stdout:1/359: getdents d16/d19/d20/d26 0 2024-03-30T04:28:20.992 INFO:tasks.workunit.client.0.smithi123.stdout:0/808: symlink d6/dd/d15/d98/dfe/l103 0 2024-03-30T04:28:20.995 INFO:tasks.workunit.client.1.smithi135.stdout:3/426: dread d3/db/f77 [0,4194304] 0 2024-03-30T04:28:20.995 INFO:tasks.workunit.client.1.smithi135.stdout:1/360: read f3 [732069,81248] 0 2024-03-30T04:28:20.996 INFO:tasks.workunit.client.0.smithi123.stdout:6/858: unlink d10/d20/f3b 0 2024-03-30T04:28:20.998 INFO:tasks.workunit.client.0.smithi123.stdout:2/955: rename d0/d1f/d1c/d36/c47 to d0/d1f/d1c/d39/d4b/d10a/c133 0 2024-03-30T04:28:21.000 INFO:tasks.workunit.client.0.smithi123.stdout:6/859: fsync fe 0 2024-03-30T04:28:21.001 INFO:tasks.workunit.client.1.smithi135.stdout:7/445: fsync d4/db/d1a/f66 0 2024-03-30T04:28:21.005 INFO:tasks.workunit.client.1.smithi135.stdout:1/361: unlink d16/d29/d2c/f76 0 2024-03-30T04:28:21.009 INFO:tasks.workunit.client.1.smithi135.stdout:3/427: write d3/d50/d46/f21 [1499458,105221] 0 2024-03-30T04:28:21.010 INFO:tasks.workunit.client.0.smithi123.stdout:0/809: stat d6/dd/d31/d35/d3a/fdb 0 2024-03-30T04:28:21.010 INFO:tasks.workunit.client.0.smithi123.stdout:5/749: sync 2024-03-30T04:28:21.010 INFO:tasks.workunit.client.0.smithi123.stdout:7/780: mkdir d2/da/d11/d61/d106 0 2024-03-30T04:28:21.010 INFO:tasks.workunit.client.0.smithi123.stdout:8/829: link da/d27/d56/d6c/d6d/lbf da/dc/d14/d9d/l102 0 2024-03-30T04:28:21.012 INFO:tasks.workunit.client.0.smithi123.stdout:3/897: chown d13/d24/d31/d23/d3a/d3c/fb6 0 1 2024-03-30T04:28:21.012 INFO:tasks.workunit.client.1.smithi135.stdout:3/428: chown d3/d50/f1f 0 1 2024-03-30T04:28:21.013 INFO:tasks.workunit.client.1.smithi135.stdout:0/486: dwrite d0/d19/d2d/f98 [0,4194304] 0 2024-03-30T04:28:21.013 INFO:tasks.workunit.client.1.smithi135.stdout:1/362: symlink d16/d29/d2c/l79 0 2024-03-30T04:28:21.017 INFO:tasks.workunit.client.1.smithi135.stdout:3/429: readlink d3/l8 0 2024-03-30T04:28:21.017 INFO:tasks.workunit.client.1.smithi135.stdout:1/363: fdatasync d16/d29/f35 0 2024-03-30T04:28:21.019 INFO:tasks.workunit.client.0.smithi123.stdout:1/841: rename db/dd/d5f/f87 to db/dd/d5f/d6d/d85/d89/f117 0 2024-03-30T04:28:21.020 INFO:tasks.workunit.client.0.smithi123.stdout:5/750: mkdir dd/d19/d1f/d3c/d9e/df5 0 2024-03-30T04:28:21.022 INFO:tasks.workunit.client.0.smithi123.stdout:1/842: chown db/dd/d5f/d6d/d85/d89/d90/d94/d9c/dfc 17 1 2024-03-30T04:28:21.024 INFO:tasks.workunit.client.1.smithi135.stdout:8/382: sync 2024-03-30T04:28:21.024 INFO:tasks.workunit.client.1.smithi135.stdout:4/383: sync 2024-03-30T04:28:21.024 INFO:tasks.workunit.client.1.smithi135.stdout:6/381: sync 2024-03-30T04:28:21.024 INFO:tasks.workunit.client.1.smithi135.stdout:1/364: fsync d16/d29/d2c/d43/d50/f5b 0 2024-03-30T04:28:21.025 INFO:tasks.workunit.client.0.smithi123.stdout:8/830: symlink da/d27/d3c/d2d/dae/l103 0 2024-03-30T04:28:21.027 INFO:tasks.workunit.client.0.smithi123.stdout:4/761: link d2/d4/d7/d1a/l48 d2/d4/d2c/d42/d35/d9b/da2/deb/l106 0 2024-03-30T04:28:21.028 INFO:tasks.workunit.client.0.smithi123.stdout:3/898: chown d13/d24/d31/d23/ccb 109318211 1 2024-03-30T04:28:21.029 INFO:tasks.workunit.client.1.smithi135.stdout:4/384: chown d10/f57 175776 1 2024-03-30T04:28:21.030 INFO:tasks.workunit.client.1.smithi135.stdout:3/430: mkdir d3/d10/d16/d23/d6a/d86/d98/d99 0 2024-03-30T04:28:21.034 INFO:tasks.workunit.client.1.smithi135.stdout:8/383: unlink d6/d8/d15/d2a/l3f 0 2024-03-30T04:28:21.034 INFO:tasks.workunit.client.0.smithi123.stdout:1/843: truncate db/fe1 40582 0 2024-03-30T04:28:21.035 INFO:tasks.workunit.client.1.smithi135.stdout:1/365: rename d16/d1a/f4d to d16/d5a/d6b/d77/f7a 0 2024-03-30T04:28:21.035 INFO:tasks.workunit.client.0.smithi123.stdout:7/781: rmdir d2/da/d11/d61/d55/d94 39 2024-03-30T04:28:21.036 INFO:tasks.workunit.client.0.smithi123.stdout:8/831: truncate da/d27/d3c/d64/d78/db8/fc3 198196 0 2024-03-30T04:28:21.037 INFO:tasks.workunit.client.1.smithi135.stdout:7/446: truncate d4/db/f14 909319 0 2024-03-30T04:28:21.039 INFO:tasks.workunit.client.1.smithi135.stdout:0/487: dwrite d0/d1/d16/f85 [0,4194304] 0 2024-03-30T04:28:21.039 INFO:tasks.workunit.client.1.smithi135.stdout:8/384: dread - d6/d8/d14/f67 zero size 2024-03-30T04:28:21.040 INFO:tasks.workunit.client.0.smithi123.stdout:2/956: getdents d0/d37/d57 0 2024-03-30T04:28:21.040 INFO:tasks.workunit.client.0.smithi123.stdout:1/844: dread db/dd/d5f/d6d/d7e/ff0 [0,4194304] 0 2024-03-30T04:28:21.042 INFO:tasks.workunit.client.0.smithi123.stdout:4/762: truncate d2/d34/f77 977195 0 2024-03-30T04:28:21.043 INFO:tasks.workunit.client.1.smithi135.stdout:3/431: write d3/d10/f2d [338359,47608] 0 2024-03-30T04:28:21.043 INFO:tasks.workunit.client.0.smithi123.stdout:2/957: chown d0/d1f/f24 23440909 1 2024-03-30T04:28:21.043 INFO:tasks.workunit.client.1.smithi135.stdout:6/382: rename d2/d36/l57 to d2/dd/d18/d21/d55/l79 0 2024-03-30T04:28:21.046 INFO:tasks.workunit.client.1.smithi135.stdout:8/385: mknod d6/d8/d14/d32/c70 0 2024-03-30T04:28:21.046 INFO:tasks.workunit.client.0.smithi123.stdout:4/763: fsync d2/d4/d2c/d42/fc3 0 2024-03-30T04:28:21.047 INFO:tasks.workunit.client.0.smithi123.stdout:0/810: dread d6/d52/fa8 [0,4194304] 0 2024-03-30T04:28:21.047 INFO:tasks.workunit.client.0.smithi123.stdout:3/899: rmdir d13/d24/d31/d123/d5d/da5/dcd 39 2024-03-30T04:28:21.047 INFO:tasks.workunit.client.1.smithi135.stdout:0/488: symlink d0/d5/d9/d23/d5d/la7 0 2024-03-30T04:28:21.050 INFO:tasks.workunit.client.1.smithi135.stdout:1/366: link d16/d29/d2c/d3c/f3b d16/d5a/d6b/f7b 0 2024-03-30T04:28:21.050 INFO:tasks.workunit.client.0.smithi123.stdout:3/900: fdatasync d13/d24/f50 0 2024-03-30T04:28:21.051 INFO:tasks.workunit.client.1.smithi135.stdout:7/447: symlink d4/db/d1a/d5c/l95 0 2024-03-30T04:28:21.057 INFO:tasks.workunit.client.0.smithi123.stdout:9/774: dwrite d6/d58/f5a [0,4194304] 0 2024-03-30T04:28:21.057 INFO:tasks.workunit.client.0.smithi123.stdout:0/811: stat d6/dd/d15/d98/d4e/d69/d6e/c89 0 2024-03-30T04:28:21.058 INFO:tasks.workunit.client.1.smithi135.stdout:3/432: symlink d3/d10/d16/d41/l9a 0 2024-03-30T04:28:21.060 INFO:tasks.workunit.client.0.smithi123.stdout:7/782: write d2/da/d11/d61/d55/d59/f88 [1137452,73372] 0 2024-03-30T04:28:21.063 INFO:tasks.workunit.client.0.smithi123.stdout:0/812: write d6/f9 [6554992,115310] 0 2024-03-30T04:28:21.066 INFO:tasks.workunit.client.1.smithi135.stdout:3/433: mkdir d3/d10/d9b 0 2024-03-30T04:28:21.069 INFO:tasks.workunit.client.1.smithi135.stdout:3/434: fdatasync d3/d50/d46/f73 0 2024-03-30T04:28:21.070 INFO:tasks.workunit.client.1.smithi135.stdout:0/489: dwrite d0/d1/f5e [0,4194304] 0 2024-03-30T04:28:21.073 INFO:tasks.workunit.client.1.smithi135.stdout:6/383: rmdir d2/dd/d18/d21/d55 39 2024-03-30T04:28:21.075 INFO:tasks.workunit.client.0.smithi123.stdout:8/832: creat da/dc/da9/dc4/f104 x:0 0 0 2024-03-30T04:28:21.075 INFO:tasks.workunit.client.0.smithi123.stdout:6/860: dwrite d10/d82/dc1/d6a/d7a/fa1 [0,4194304] 0 2024-03-30T04:28:21.081 INFO:tasks.workunit.client.1.smithi135.stdout:3/435: mkdir d3/d10/d9c 0 2024-03-30T04:28:21.084 INFO:tasks.workunit.client.1.smithi135.stdout:7/448: getdents d4/db/d1a/d5c/d2c/d83 0 2024-03-30T04:28:21.084 INFO:tasks.workunit.client.1.smithi135.stdout:3/436: read - d3/d50/f5b zero size 2024-03-30T04:28:21.087 INFO:tasks.workunit.client.0.smithi123.stdout:1/845: creat db/dd/d5f/dae/f118 x:0 0 0 2024-03-30T04:28:21.087 INFO:tasks.workunit.client.0.smithi123.stdout:5/751: truncate dd/f26 361461 0 2024-03-30T04:28:21.087 INFO:tasks.workunit.client.1.smithi135.stdout:1/367: dwrite d16/d19/f1f [0,4194304] 0 2024-03-30T04:28:21.089 INFO:tasks.workunit.client.1.smithi135.stdout:3/437: dread - d3/d10/d16/d23/f7f zero size 2024-03-30T04:28:21.089 INFO:tasks.workunit.client.1.smithi135.stdout:1/368: dread - d16/d19/f72 zero size 2024-03-30T04:28:21.092 INFO:tasks.workunit.client.1.smithi135.stdout:4/385: dread fa [0,4194304] 0 2024-03-30T04:28:21.094 INFO:tasks.workunit.client.0.smithi123.stdout:1/846: dread - db/dd/d5f/d6d/d85/d89/d90/d94/d9c/f102 zero size 2024-03-30T04:28:21.097 INFO:tasks.workunit.client.1.smithi135.stdout:0/490: dread d0/d5/d9/f17 [0,4194304] 0 2024-03-30T04:28:21.098 INFO:tasks.workunit.client.0.smithi123.stdout:4/764: dread d2/d4/d2c/d42/dad/d91/d9d/fa6 [0,4194304] 0 2024-03-30T04:28:21.099 INFO:tasks.workunit.client.1.smithi135.stdout:8/386: truncate d6/d8/f1d 2037341 0 2024-03-30T04:28:21.100 INFO:tasks.workunit.client.1.smithi135.stdout:0/491: dread - d0/d5/d9/d23/fa4 zero size 2024-03-30T04:28:21.104 INFO:tasks.workunit.client.1.smithi135.stdout:4/386: unlink d10/f4f 0 2024-03-30T04:28:21.107 INFO:tasks.workunit.client.0.smithi123.stdout:0/813: rmdir d6/dd/d15/d28/d4b/dad 39 2024-03-30T04:28:21.107 INFO:tasks.workunit.client.1.smithi135.stdout:8/387: unlink d6/d8/d15/d1f/c6b 0 2024-03-30T04:28:21.107 INFO:tasks.workunit.client.1.smithi135.stdout:3/438: rmdir d3/d10/d16/d6b 0 2024-03-30T04:28:21.111 INFO:tasks.workunit.client.1.smithi135.stdout:0/492: unlink d0/d13/d53/d54/f5b 0 2024-03-30T04:28:21.111 INFO:tasks.workunit.client.0.smithi123.stdout:0/814: readlink d6/dd/d15/d98/d5d/d7f/ld2 0 2024-03-30T04:28:21.115 INFO:tasks.workunit.client.0.smithi123.stdout:9/775: dread d6/d1f/d45/d97/d69/f6c [0,4194304] 0 2024-03-30T04:28:21.116 INFO:tasks.workunit.client.0.smithi123.stdout:7/783: mkdir d2/da/d11/d61/d55/d59/db5/dab/d107 0 2024-03-30T04:28:21.117 INFO:tasks.workunit.client.1.smithi135.stdout:3/439: mkdir d3/d6e/d9d 0 2024-03-30T04:28:21.117 INFO:tasks.workunit.client.1.smithi135.stdout:0/493: mkdir d0/d19/d2d/d45/da8 0 2024-03-30T04:28:21.121 INFO:tasks.workunit.client.1.smithi135.stdout:1/369: dread d16/f2f [0,4194304] 0 2024-03-30T04:28:21.121 INFO:tasks.workunit.client.0.smithi123.stdout:6/861: chown d10/d20/d3f 6 1 2024-03-30T04:28:21.124 INFO:tasks.workunit.client.0.smithi123.stdout:1/847: unlink db/dd/d42/d46/d56/d5c/c101 0 2024-03-30T04:28:21.128 INFO:tasks.workunit.client.1.smithi135.stdout:6/384: dread d2/dd/f17 [0,4194304] 0 2024-03-30T04:28:21.131 INFO:tasks.workunit.client.0.smithi123.stdout:9/776: rmdir d6/dc9 39 2024-03-30T04:28:21.131 INFO:tasks.workunit.client.0.smithi123.stdout:0/815: mkdir d6/dd/d15/d98/d5d/d7f/df3/d104 0 2024-03-30T04:28:21.131 INFO:tasks.workunit.client.1.smithi135.stdout:6/385: write d2/f4d [2022935,89986] 0 2024-03-30T04:28:21.133 INFO:tasks.workunit.client.0.smithi123.stdout:7/784: creat d2/d4d/dd0/f108 x:0 0 0 2024-03-30T04:28:21.134 INFO:tasks.workunit.client.1.smithi135.stdout:1/370: creat d16/d5a/d5f/d3a/d6f/f7c x:0 0 0 2024-03-30T04:28:21.135 INFO:tasks.workunit.client.0.smithi123.stdout:9/777: stat d6/d58/lf3 0 2024-03-30T04:28:21.136 INFO:tasks.workunit.client.0.smithi123.stdout:3/901: dread f11 [0,4194304] 0 2024-03-30T04:28:21.137 INFO:tasks.workunit.client.1.smithi135.stdout:4/387: dwrite d10/d72/f7d [0,4194304] 0 2024-03-30T04:28:21.137 INFO:tasks.workunit.client.0.smithi123.stdout:4/765: dwrite d2/d4/d2c/d42/d35/d5d/d78/fbe [0,4194304] 0 2024-03-30T04:28:21.139 INFO:tasks.workunit.client.1.smithi135.stdout:6/386: fdatasync d2/dd/d14/f15 0 2024-03-30T04:28:21.142 INFO:tasks.workunit.client.0.smithi123.stdout:9/778: dread - d6/d7c/d82/f102 zero size 2024-03-30T04:28:21.142 INFO:tasks.workunit.client.0.smithi123.stdout:5/752: creat dd/d19/d1f/d3c/d72/ff6 x:0 0 0 2024-03-30T04:28:21.143 INFO:tasks.workunit.client.1.smithi135.stdout:0/494: dread d0/d13/d53/f97 [4194304,4194304] 0 2024-03-30T04:28:21.144 INFO:tasks.workunit.client.0.smithi123.stdout:9/779: stat d6/d58/fc2 0 2024-03-30T04:28:21.144 INFO:tasks.workunit.client.1.smithi135.stdout:4/388: creat d10/d4d/f87 x:0 0 0 2024-03-30T04:28:21.145 INFO:tasks.workunit.client.1.smithi135.stdout:1/371: creat d16/d5a/d5f/d3a/d6f/f7d x:0 0 0 2024-03-30T04:28:21.147 INFO:tasks.workunit.client.0.smithi123.stdout:1/848: truncate db/dd/d5f/d6d/d85/d89/d90/d94/d9c/f102 503089 0 2024-03-30T04:28:21.147 INFO:tasks.workunit.client.1.smithi135.stdout:6/387: mknod d2/dd/d14/d38/c7a 0 2024-03-30T04:28:21.150 INFO:tasks.workunit.client.0.smithi123.stdout:7/785: stat d2/da/d11/fb1 0 2024-03-30T04:28:21.151 INFO:tasks.workunit.client.1.smithi135.stdout:4/389: chown d10/d1a/d2b/l56 2061827 1 2024-03-30T04:28:21.151 INFO:tasks.workunit.client.1.smithi135.stdout:8/388: dwrite d6/d8/f2d [0,4194304] 0 2024-03-30T04:28:21.152 INFO:tasks.workunit.client.0.smithi123.stdout:7/786: write d2/fbd [1441406,108922] 0 2024-03-30T04:28:21.153 INFO:tasks.workunit.client.0.smithi123.stdout:2/958: write d0/d1f/d1c/d40/f98 [532250,35715] 0 2024-03-30T04:28:21.155 INFO:tasks.workunit.client.1.smithi135.stdout:4/390: stat d10/f17 0 2024-03-30T04:28:21.155 INFO:tasks.workunit.client.1.smithi135.stdout:1/372: creat d16/d5a/d5f/d3d/f7e x:0 0 0 2024-03-30T04:28:21.159 INFO:tasks.workunit.client.1.smithi135.stdout:1/373: chown d16/d29/d61 72033 1 2024-03-30T04:28:21.161 INFO:tasks.workunit.client.1.smithi135.stdout:0/495: getdents d0/d1/d16 0 2024-03-30T04:28:21.162 INFO:tasks.workunit.client.1.smithi135.stdout:4/391: truncate d10/d4d/f87 958900 0 2024-03-30T04:28:21.163 INFO:tasks.workunit.client.0.smithi123.stdout:5/753: mkdir dd/d19/d1f/d68/dcc/df7 0 2024-03-30T04:28:21.164 INFO:tasks.workunit.client.1.smithi135.stdout:1/374: write f7 [4368540,128065] 0 2024-03-30T04:28:21.166 INFO:tasks.workunit.client.1.smithi135.stdout:1/375: fdatasync d16/f45 0 2024-03-30T04:28:21.167 INFO:tasks.workunit.client.1.smithi135.stdout:4/392: readlink d10/d1a/d32/l44 0 2024-03-30T04:28:21.170 INFO:tasks.workunit.client.1.smithi135.stdout:1/376: write d16/d19/d20/f65 [632588,26793] 0 2024-03-30T04:28:21.179 INFO:tasks.workunit.client.0.smithi123.stdout:1/849: mkdir db/dd/d42/d46/d56/d119 0 2024-03-30T04:28:21.180 INFO:tasks.workunit.client.1.smithi135.stdout:0/496: dread d0/d5/d10/fa1 [8388608,4194304] 0 2024-03-30T04:28:21.181 INFO:tasks.workunit.client.0.smithi123.stdout:8/833: dwrite da/f55 [4194304,4194304] 0 2024-03-30T04:28:21.182 INFO:tasks.workunit.client.0.smithi123.stdout:5/754: mknod dd/d19/d3e/d5b/d62/cf8 0 2024-03-30T04:28:21.183 INFO:tasks.workunit.client.1.smithi135.stdout:0/497: creat d0/d5/d9/d23/d8a/fa9 x:0 0 0 2024-03-30T04:28:21.184 INFO:tasks.workunit.client.1.smithi135.stdout:6/388: dread d2/f39 [0,4194304] 0 2024-03-30T04:28:21.187 INFO:tasks.workunit.client.0.smithi123.stdout:0/816: getdents d6/dd/d15/d28/d4b 0 2024-03-30T04:28:21.191 INFO:tasks.workunit.client.0.smithi123.stdout:9/780: dwrite d6/fa9 [0,4194304] 0 2024-03-30T04:28:21.192 INFO:tasks.workunit.client.0.smithi123.stdout:4/766: dread d2/d4/f16 [0,4194304] 0 2024-03-30T04:28:21.197 INFO:tasks.workunit.client.0.smithi123.stdout:3/902: link d13/l9c d13/d24/d31/d23/l126 0 2024-03-30T04:28:21.197 INFO:tasks.workunit.client.0.smithi123.stdout:5/755: symlink dd/d19/d5f/dc2/lf9 0 2024-03-30T04:28:21.198 INFO:tasks.workunit.client.1.smithi135.stdout:6/389: dread d2/d36/f6e [0,4194304] 0 2024-03-30T04:28:21.201 INFO:tasks.workunit.client.0.smithi123.stdout:6/862: dwrite d10/f3a [0,4194304] 0 2024-03-30T04:28:21.205 INFO:tasks.workunit.client.1.smithi135.stdout:6/390: unlink d2/dd/d14/d2b/f47 0 2024-03-30T04:28:21.206 INFO:tasks.workunit.client.1.smithi135.stdout:0/498: dwrite d0/d13/d1e/f4f [0,4194304] 0 2024-03-30T04:28:21.208 INFO:tasks.workunit.client.0.smithi123.stdout:2/959: getdents d0/d1f/d1c/d39/de4 0 2024-03-30T04:28:21.209 INFO:tasks.workunit.client.0.smithi123.stdout:4/767: fsync d2/d4/f16 0 2024-03-30T04:28:21.210 INFO:tasks.workunit.client.1.smithi135.stdout:0/499: write d0/d1/d16/f4b [392204,73036] 0 2024-03-30T04:28:21.210 INFO:tasks.workunit.client.0.smithi123.stdout:8/834: fsync da/ff 0 2024-03-30T04:28:21.212 INFO:tasks.workunit.client.1.smithi135.stdout:6/391: stat d2/l6 0 2024-03-30T04:28:21.213 INFO:tasks.workunit.client.0.smithi123.stdout:5/756: rmdir dd/d19/d1f/d35/d3a/d57 39 2024-03-30T04:28:21.217 INFO:tasks.workunit.client.0.smithi123.stdout:3/903: unlink d13/d24/d31/d123/d5d/fb7 0 2024-03-30T04:28:21.217 INFO:tasks.workunit.client.0.smithi123.stdout:4/768: write d2/d4/d2c/d42/fc3 [1371793,87660] 0 2024-03-30T04:28:21.218 INFO:tasks.workunit.client.1.smithi135.stdout:4/393: dread d10/f14 [0,4194304] 0 2024-03-30T04:28:21.218 INFO:tasks.workunit.client.1.smithi135.stdout:0/500: rmdir d0/d5/d9/d23/d5d 39 2024-03-30T04:28:21.218 INFO:tasks.workunit.client.0.smithi123.stdout:0/817: unlink d6/dd/d18/d2f/fb1 0 2024-03-30T04:28:21.220 INFO:tasks.workunit.client.0.smithi123.stdout:9/781: symlink d6/d1f/d10c/l10e 0 2024-03-30T04:28:21.222 INFO:tasks.workunit.client.1.smithi135.stdout:0/501: rename d0/d19 to d0/d19/daa 22 2024-03-30T04:28:21.225 INFO:tasks.workunit.client.0.smithi123.stdout:3/904: readlink d13/d24/d31/d123/d5d/ld0 0 2024-03-30T04:28:21.225 INFO:tasks.workunit.client.1.smithi135.stdout:6/392: creat d2/dd/d18/d21/d55/f7b x:0 0 0 2024-03-30T04:28:21.227 INFO:tasks.workunit.client.0.smithi123.stdout:5/757: rename dd/d19/d3e/d5b/ldd to dd/d19/d5f/dc2/lfa 0 2024-03-30T04:28:21.227 INFO:tasks.workunit.client.1.smithi135.stdout:4/394: symlink d10/d1a/l88 0 2024-03-30T04:28:21.228 INFO:tasks.workunit.client.1.smithi135.stdout:4/395: chown fb 71 1 2024-03-30T04:28:21.228 INFO:tasks.workunit.client.0.smithi123.stdout:2/960: creat d0/d37/d11a/d12f/f134 x:0 0 0 2024-03-30T04:28:21.229 INFO:tasks.workunit.client.1.smithi135.stdout:0/502: creat d0/d1/fab x:0 0 0 2024-03-30T04:28:21.229 INFO:tasks.workunit.client.1.smithi135.stdout:6/393: mknod d2/d36/c7c 0 2024-03-30T04:28:21.229 INFO:tasks.workunit.client.1.smithi135.stdout:4/396: fsync d10/d1a/d3d/f5a 0 2024-03-30T04:28:21.230 INFO:tasks.workunit.client.0.smithi123.stdout:3/905: stat d13/d24/d31/d123/d5d/da5/d104/l125 0 2024-03-30T04:28:21.234 INFO:tasks.workunit.client.1.smithi135.stdout:4/397: mkdir d10/d4d/d89 0 2024-03-30T04:28:21.239 INFO:tasks.workunit.client.0.smithi123.stdout:6/863: creat d10/d82/dc1/d45/d111/f122 x:0 0 0 2024-03-30T04:28:21.239 INFO:tasks.workunit.client.0.smithi123.stdout:1/850: write db/d1b/f28 [2546398,5823] 0 2024-03-30T04:28:21.239 INFO:tasks.workunit.client.1.smithi135.stdout:6/394: fdatasync d2/f9 0 2024-03-30T04:28:21.239 INFO:tasks.workunit.client.1.smithi135.stdout:4/398: unlink f4 0 2024-03-30T04:28:21.242 INFO:tasks.workunit.client.0.smithi123.stdout:4/769: creat d2/dfb/f107 x:0 0 0 2024-03-30T04:28:21.243 INFO:tasks.workunit.client.1.smithi135.stdout:4/399: getdents d10/d1a/d32 0 2024-03-30T04:28:21.245 INFO:tasks.workunit.client.1.smithi135.stdout:4/400: chown d10/d1a/d42 1202960 1 2024-03-30T04:28:21.246 INFO:tasks.workunit.client.0.smithi123.stdout:6/864: symlink d10/d82/dc1/d45/l123 0 2024-03-30T04:28:21.248 INFO:tasks.workunit.client.1.smithi135.stdout:4/401: mknod d10/d1a/d32/d40/c8a 0 2024-03-30T04:28:21.250 INFO:tasks.workunit.client.1.smithi135.stdout:0/503: dwrite d0/d13/d1e/f78 [0,4194304] 0 2024-03-30T04:28:21.250 INFO:tasks.workunit.client.0.smithi123.stdout:5/758: creat dd/ffb x:0 0 0 2024-03-30T04:28:21.251 INFO:tasks.workunit.client.1.smithi135.stdout:5/398: sync 2024-03-30T04:28:21.251 INFO:tasks.workunit.client.1.smithi135.stdout:2/334: sync 2024-03-30T04:28:21.252 INFO:tasks.workunit.client.0.smithi123.stdout:4/770: mkdir d2/d4/d7/dc1/d108 0 2024-03-30T04:28:21.252 INFO:tasks.workunit.client.1.smithi135.stdout:0/504: chown d0/d13/d91 1 1 2024-03-30T04:28:21.256 INFO:tasks.workunit.client.1.smithi135.stdout:5/399: symlink d1/d3/d2b/l78 0 2024-03-30T04:28:21.257 INFO:tasks.workunit.client.1.smithi135.stdout:4/402: creat d10/d4d/d89/f8b x:0 0 0 2024-03-30T04:28:21.257 INFO:tasks.workunit.client.1.smithi135.stdout:6/395: dwrite d2/dd/d14/d53/f77 [0,4194304] 0 2024-03-30T04:28:21.260 INFO:tasks.workunit.client.1.smithi135.stdout:5/400: read d1/f6e [70647,85568] 0 2024-03-30T04:28:21.260 INFO:tasks.workunit.client.0.smithi123.stdout:2/961: dwrite d0/d1f/d30/f117 [0,4194304] 0 2024-03-30T04:28:21.260 INFO:tasks.workunit.client.0.smithi123.stdout:1/851: getdents db/dd/d42/d46/d56/da2/df1 0 2024-03-30T04:28:21.261 INFO:tasks.workunit.client.1.smithi135.stdout:2/335: creat d4/d6/f6f x:0 0 0 2024-03-30T04:28:21.261 INFO:tasks.workunit.client.1.smithi135.stdout:4/403: truncate d10/d1a/f27 717277 0 2024-03-30T04:28:21.264 INFO:tasks.workunit.client.1.smithi135.stdout:6/396: write d2/fc [1105885,59772] 0 2024-03-30T04:28:21.266 INFO:tasks.workunit.client.1.smithi135.stdout:5/401: mkdir d1/d3/d2b/d54/d5e/d5c/d79 0 2024-03-30T04:28:21.269 INFO:tasks.workunit.client.1.smithi135.stdout:5/402: chown d1/d7/f43 58586740 1 2024-03-30T04:28:21.272 INFO:tasks.workunit.client.1.smithi135.stdout:6/397: mkdir d2/d7d 0 2024-03-30T04:28:21.272 INFO:tasks.workunit.client.1.smithi135.stdout:4/404: truncate d10/d1a/f23 2706554 0 2024-03-30T04:28:21.273 INFO:tasks.workunit.client.1.smithi135.stdout:5/403: write d1/d3/d2b/f75 [651526,30382] 0 2024-03-30T04:28:21.274 INFO:tasks.workunit.client.0.smithi123.stdout:0/818: write d6/dd/d15/d28/f5c [2528872,48210] 0 2024-03-30T04:28:21.279 INFO:tasks.workunit.client.1.smithi135.stdout:6/398: readlink d2/d36/l4e 0 2024-03-30T04:28:21.281 INFO:tasks.workunit.client.1.smithi135.stdout:0/505: dwrite d0/d5/d9/d23/d2a/f51 [0,4194304] 0 2024-03-30T04:28:21.285 INFO:tasks.workunit.client.0.smithi123.stdout:5/759: dwrite dd/d19/d3e/f63 [0,4194304] 0 2024-03-30T04:28:21.285 INFO:tasks.workunit.client.0.smithi123.stdout:1/852: creat db/dd/d5f/d6d/d85/d89/d90/d94/d9c/dfc/f11a x:0 0 0 2024-03-30T04:28:21.286 INFO:tasks.workunit.client.1.smithi135.stdout:4/405: fdatasync d10/d1a/f23 0 2024-03-30T04:28:21.288 INFO:tasks.workunit.client.1.smithi135.stdout:5/404: chown d1/d7/d1a/d1f/l2e 277567 1 2024-03-30T04:28:21.291 INFO:tasks.workunit.client.0.smithi123.stdout:6/865: write d10/d82/dc1/d45/fe5 [188550,17861] 0 2024-03-30T04:28:21.291 INFO:tasks.workunit.client.0.smithi123.stdout:0/819: symlink d6/dd/d31/db8/l105 0 2024-03-30T04:28:21.292 INFO:tasks.workunit.client.1.smithi135.stdout:6/399: readlink d2/l3b 0 2024-03-30T04:28:21.293 INFO:tasks.workunit.client.0.smithi123.stdout:2/962: write d0/d37/d57/f62 [745878,127704] 0 2024-03-30T04:28:21.295 INFO:tasks.workunit.client.1.smithi135.stdout:0/506: creat d0/d13/d53/d7f/d83/fac x:0 0 0 2024-03-30T04:28:21.295 INFO:tasks.workunit.client.0.smithi123.stdout:9/782: dwrite d6/d1f/d45/d97/d69/fab [0,4194304] 0 2024-03-30T04:28:21.298 INFO:tasks.workunit.client.0.smithi123.stdout:8/835: dwrite da/f9a [0,4194304] 0 2024-03-30T04:28:21.300 INFO:tasks.workunit.client.0.smithi123.stdout:3/906: dwrite d13/d24/d31/d23/d3a/fa4 [0,4194304] 0 2024-03-30T04:28:21.301 INFO:tasks.workunit.client.1.smithi135.stdout:5/405: write d1/f4f [5305435,45394] 0 2024-03-30T04:28:21.304 INFO:tasks.workunit.client.1.smithi135.stdout:0/507: write d0/d13/d53/f97 [2438122,13069] 0 2024-03-30T04:28:21.304 INFO:tasks.workunit.client.1.smithi135.stdout:2/336: dwrite d4/d32/d50/f58 [0,4194304] 0 2024-03-30T04:28:21.311 INFO:tasks.workunit.client.1.smithi135.stdout:6/400: creat d2/dd/d14/d38/f7e x:0 0 0 2024-03-30T04:28:21.312 INFO:tasks.workunit.client.0.smithi123.stdout:5/760: mknod dd/d19/d1f/d35/d3a/d79/cfc 0 2024-03-30T04:28:21.316 INFO:tasks.workunit.client.0.smithi123.stdout:6/866: read d10/f48 [808733,97867] 0 2024-03-30T04:28:21.318 INFO:tasks.workunit.client.1.smithi135.stdout:2/337: symlink d4/d32/d12/d45/l70 0 2024-03-30T04:28:21.322 INFO:tasks.workunit.client.0.smithi123.stdout:6/867: chown d10/d82/dc1/c52 124318978 1 2024-03-30T04:28:21.324 INFO:tasks.workunit.client.0.smithi123.stdout:5/761: read dd/d19/fbc [4111366,4545] 0 2024-03-30T04:28:21.329 INFO:tasks.workunit.client.1.smithi135.stdout:4/406: dwrite d10/d1a/d42/f63 [0,4194304] 0 2024-03-30T04:28:21.342 INFO:tasks.workunit.client.1.smithi135.stdout:4/407: write d10/d1a/d42/f67 [190090,76229] 0 2024-03-30T04:28:21.346 INFO:tasks.workunit.client.1.smithi135.stdout:0/508: dread d0/d5/d9/d23/d2a/fa0 [4194304,4194304] 0 2024-03-30T04:28:21.353 INFO:tasks.workunit.client.0.smithi123.stdout:0/820: creat d6/dd/d15/d28/d4b/dad/de8/f106 x:0 0 0 2024-03-30T04:28:21.355 INFO:tasks.workunit.client.0.smithi123.stdout:9/783: mkdir d6/d1f/d72/d7d/d10f 0 2024-03-30T04:28:21.358 INFO:tasks.workunit.client.0.smithi123.stdout:8/836: creat da/d27/d3c/d64/d78/d83/f105 x:0 0 0 2024-03-30T04:28:21.358 INFO:tasks.workunit.client.1.smithi135.stdout:5/406: dwrite d1/d7/d1a/f68 [0,4194304] 0 2024-03-30T04:28:21.358 INFO:tasks.workunit.client.1.smithi135.stdout:6/401: dwrite d2/dd/d14/d53/f77 [0,4194304] 0 2024-03-30T04:28:21.360 INFO:tasks.workunit.client.0.smithi123.stdout:5/762: dwrite dd/d19/d1f/f27 [0,4194304] 0 2024-03-30T04:28:21.365 INFO:tasks.workunit.client.0.smithi123.stdout:6/868: dwrite d10/d82/dc1/dd2/fd9 [0,4194304] 0 2024-03-30T04:28:21.367 INFO:tasks.workunit.client.1.smithi135.stdout:0/509: dread d0/d5/d9/d23/d58/d63/f95 [0,4194304] 0 2024-03-30T04:28:21.368 INFO:tasks.workunit.client.1.smithi135.stdout:4/408: symlink d10/l8c 0 2024-03-30T04:28:21.371 INFO:tasks.workunit.client.0.smithi123.stdout:7/787: sync 2024-03-30T04:28:21.371 INFO:tasks.workunit.client.1.smithi135.stdout:5/407: mkdir d1/da/d17/d7a 0 2024-03-30T04:28:21.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:21.373+0000 7fee92a00700 1 -- 172.21.15.123:0/1356209257 --> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fee8c10d580 con 0x7fee7006af10 2024-03-30T04:28:21.375 INFO:tasks.workunit.client.0.smithi123.stdout:9/784: chown d6/d1f/d72/d7d/dfa/ccd 5018 1 2024-03-30T04:28:21.376 INFO:tasks.workunit.client.1.smithi135.stdout:0/510: symlink d0/d1/d16/lad 0 2024-03-30T04:28:21.377 INFO:tasks.workunit.client.0.smithi123.stdout:5/763: mknod dd/d19/d3e/d5b/d62/dd3/cfd 0 2024-03-30T04:28:21.379 INFO:tasks.workunit.client.1.smithi135.stdout:4/409: mknod d10/d72/d7c/c8d 0 2024-03-30T04:28:21.383 INFO:tasks.workunit.client.1.smithi135.stdout:5/408: write d1/d3/d2b/d54/d5e/d5c/f65 [549224,72310] 0 2024-03-30T04:28:21.386 INFO:tasks.workunit.client.0.smithi123.stdout:1/853: write db/dd/d42/d46/d56/da2/f10f [303740,1113] 0 2024-03-30T04:28:21.386 INFO:tasks.workunit.client.0.smithi123.stdout:3/907: link d13/d24/d31/d23/d3a/d3c/dab/cc5 d13/d24/d31/d123/d5d/da5/dcd/c127 0 2024-03-30T04:28:21.386 INFO:tasks.workunit.client.0.smithi123.stdout:5/764: symlink dd/d19/d1f/d68/dcc/lfe 0 2024-03-30T04:28:21.387 INFO:tasks.workunit.client.0.smithi123.stdout:8/837: link da/d27/d3c/d64/d78/db8/dfd/da2/ld6 da/d99/l106 0 2024-03-30T04:28:21.387 INFO:tasks.workunit.client.0.smithi123.stdout:9/785: symlink d6/d1f/d45/d97/d69/dfb/l110 0 2024-03-30T04:28:21.387 INFO:tasks.workunit.client.1.smithi135.stdout:5/409: write d1/d7/d1a/f68 [328275,126483] 0 2024-03-30T04:28:21.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:21.392+0000 7fee83e00700 1 -- 172.21.15.123:0/1356209257 <== mgr.14608 v2:172.21.15.135:6828/3601553667 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3388 (secure 0 0 0) 0x7fee8c10d580 con 0x7fee7006af10 2024-03-30T04:28:21.392 INFO:tasks.workunit.client.1.smithi135.stdout:0/511: mknod d0/d1/d16/d7e/cae 0 2024-03-30T04:28:21.392 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:28:21.392 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 starting - - - - 2024-03-30T04:28:21.392 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (10m) 67s ago 10m 6647k - 17.2.7-797-g5c024335 f77085f6aab7 bceb8b8c4837 2024-03-30T04:28:21.392 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (9m) 69s ago 9m 6651k - 17.2.7-797-g5c024335 f77085f6aab7 cd0fe67b2d34 2024-03-30T04:28:21.392 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 starting - - - - 2024-03-30T04:28:21.392 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (3m) 67s ago 3m 84.0M - 17.2.7-797-g5c024335 f77085f6aab7 2df066fba7a6 2024-03-30T04:28:21.392 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (3m) 67s ago 3m 32.5M - 17.2.7-797-g5c024335 f77085f6aab7 1293b8520db1 2024-03-30T04:28:21.392 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (3m) 69s ago 3m 23.5M - 17.2.7-797-g5c024335 f77085f6aab7 0dc41326818c 2024-03-30T04:28:21.392 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (3m) 69s ago 3m 25.7M - 17.2.7-797-g5c024335 f77085f6aab7 aef8a0cfc553 2024-03-30T04:28:21.392 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:9283 running (11m) 67s ago 11m 160M - 17.2.7-797-g5c024335 f77085f6aab7 7b98aee9a3b6 2024-03-30T04:28:21.392 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283 running (95s) 69s ago 8m 479M - 18.2.2-1018-g470100a4 e7c444a08cf7 20e5e0a05742 2024-03-30T04:28:21.392 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (11m) 67s ago 11m 45.2M 2048M 17.2.7-797-g5c024335 f77085f6aab7 51d18587a7df 2024-03-30T04:28:21.393 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (8m) 69s ago 8m 37.6M 2048M 17.2.7-797-g5c024335 f77085f6aab7 f81052b09dd8 2024-03-30T04:28:21.393 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 starting - - - - 2024-03-30T04:28:21.393 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (9m) 69s ago 9m 7541k - 1.5.0 0da6a335fe13 3c2c8e799cc2 2024-03-30T04:28:21.393 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (8m) 67s ago 8m 67.2M 1618M 17.2.7-797-g5c024335 f77085f6aab7 57a7ba3fd75e 2024-03-30T04:28:21.393 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (7m) 67s ago 7m 72.0M 1618M 17.2.7-797-g5c024335 f77085f6aab7 e1a6c1818a0c 2024-03-30T04:28:21.393 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (6m) 67s ago 6m 60.7M 1618M 17.2.7-797-g5c024335 f77085f6aab7 4c6a727567bd 2024-03-30T04:28:21.393 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (6m) 69s ago 6m 65.2M 2642M 17.2.7-797-g5c024335 f77085f6aab7 6fbdd1d46a9d 2024-03-30T04:28:21.393 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (5m) 69s ago 5m 61.2M 2642M 17.2.7-797-g5c024335 f77085f6aab7 b74988e947e5 2024-03-30T04:28:21.393 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (5m) 69s ago 5m 62.7M 2642M 17.2.7-797-g5c024335 f77085f6aab7 1fd3185f5484 2024-03-30T04:28:21.393 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 starting - - - - 2024-03-30T04:28:21.394 INFO:tasks.workunit.client.1.smithi135.stdout:5/410: write d1/d7/d1a/d1f/f66 [673016,59554] 0 2024-03-30T04:28:21.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:21.397+0000 7fee92a00700 1 -- 172.21.15.123:0/1356209257 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7fee7006af10 msgr2=0x7fee7006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:28:21.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:21.397+0000 7fee92a00700 1 --2- 172.21.15.123:0/1356209257 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7fee7006af10 0x7fee7006d3c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fee74009fd0 tx=0x7fee7400b040 comp rx=0 tx=0).stop 2024-03-30T04:28:21.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:21.397+0000 7fee92a00700 1 -- 172.21.15.123:0/1356209257 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fee8c0fdcf0 msgr2=0x7fee8c10bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:28:21.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:21.397+0000 7fee92a00700 1 --2- 172.21.15.123:0/1356209257 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fee8c0fdcf0 0x7fee8c10bbf0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fee7c004910 tx=0x7fee7c009e30 comp rx=0 tx=0).stop 2024-03-30T04:28:21.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:21.397+0000 7fee92a00700 1 -- 172.21.15.123:0/1356209257 shutdown_connections 2024-03-30T04:28:21.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:21.397+0000 7fee92a00700 1 --2- 172.21.15.123:0/1356209257 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7fee7006af10 0x7fee7006d3c0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:21.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:21.397+0000 7fee92a00700 1 --2- 172.21.15.123:0/1356209257 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee8c0662e0 0x7fee8c10b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:21.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:21.397+0000 7fee92a00700 1 --2- 172.21.15.123:0/1356209257 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fee8c0fdcf0 0x7fee8c10bbf0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:21.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:21.397+0000 7fee92a00700 1 -- 172.21.15.123:0/1356209257 >> 172.21.15.123:0/1356209257 conn(0x7fee8c0f7430 msgr2=0x7fee8c0f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:28:21.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:21.398+0000 7fee92a00700 1 -- 172.21.15.123:0/1356209257 shutdown_connections 2024-03-30T04:28:21.398 INFO:tasks.workunit.client.0.smithi123.stdout:1/854: read db/dd/d42/d46/f2e [1583506,35375] 0 2024-03-30T04:28:21.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:21.399+0000 7fee92a00700 1 -- 172.21.15.123:0/1356209257 wait complete. 2024-03-30T04:28:21.399 INFO:tasks.workunit.client.0.smithi123.stdout:5/765: mknod dd/d19/d5f/cff 0 2024-03-30T04:28:21.400 INFO:tasks.workunit.client.1.smithi135.stdout:0/512: creat d0/d5/faf x:0 0 0 2024-03-30T04:28:21.400 INFO:tasks.workunit.client.0.smithi123.stdout:3/908: dread d13/d24/d31/d123/f72 [0,4194304] 0 2024-03-30T04:28:21.400 INFO:tasks.workunit.client.0.smithi123.stdout:8/838: mkdir da/dc/d14/d9d/dff/d107 0 2024-03-30T04:28:21.403 INFO:tasks.workunit.client.0.smithi123.stdout:2/963: dwrite d0/d1f/d1c/d20/f9f [0,4194304] 0 2024-03-30T04:28:21.414 INFO:tasks.workunit.client.0.smithi123.stdout:0/821: dread d6/dd/f4a [0,4194304] 0 2024-03-30T04:28:21.414 INFO:tasks.workunit.client.0.smithi123.stdout:9/786: link d6/d1f/d72/f93 d6/d1f/d10c/f111 0 2024-03-30T04:28:21.415 INFO:tasks.workunit.client.0.smithi123.stdout:2/964: chown d0/d1f/d1c/d39/d4b/d4c/f64 2736 1 2024-03-30T04:28:21.416 INFO:tasks.workunit.client.0.smithi123.stdout:7/788: dwrite d2/d4d/f6a [0,4194304] 0 2024-03-30T04:28:21.418 INFO:tasks.workunit.client.0.smithi123.stdout:9/787: dread - d6/d1f/d92/fe1 zero size 2024-03-30T04:28:21.421 INFO:tasks.workunit.client.1.smithi135.stdout:5/411: dread d1/d3/d2b/f53 [0,4194304] 0 2024-03-30T04:28:21.421 INFO:tasks.workunit.client.1.smithi135.stdout:2/338: dwrite d4/d32/f33 [0,4194304] 0 2024-03-30T04:28:21.421 INFO:tasks.workunit.client.1.smithi135.stdout:6/402: truncate d2/dd/d14/f3a 957319 0 2024-03-30T04:28:21.423 INFO:tasks.workunit.client.0.smithi123.stdout:5/766: mknod dd/c100 0 2024-03-30T04:28:21.424 INFO:tasks.workunit.client.1.smithi135.stdout:2/339: write d4/d32/d2e/f48 [4466297,5861] 0 2024-03-30T04:28:21.426 INFO:tasks.workunit.client.0.smithi123.stdout:2/965: mknod d0/d37/d11a/d12f/c135 0 2024-03-30T04:28:21.435 INFO:tasks.workunit.client.0.smithi123.stdout:9/788: creat d6/d7c/d82/f112 x:0 0 0 2024-03-30T04:28:21.438 INFO:tasks.workunit.client.1.smithi135.stdout:0/513: dread d0/f12 [0,4194304] 0 2024-03-30T04:28:21.439 INFO:tasks.workunit.client.0.smithi123.stdout:0/822: mknod d6/dd/d15/d98/dd0/c107 0 2024-03-30T04:28:21.440 INFO:tasks.workunit.client.1.smithi135.stdout:5/412: mknod d1/d3/d2b/d54/d5e/d5c/d79/c7b 0 2024-03-30T04:28:21.441 INFO:tasks.workunit.client.0.smithi123.stdout:9/789: dread - d6/d7c/d82/f102 zero size 2024-03-30T04:28:21.442 INFO:tasks.workunit.client.1.smithi135.stdout:6/403: getdents d2/dd/d14/d30/d43 0 2024-03-30T04:28:21.442 INFO:tasks.workunit.client.0.smithi123.stdout:4/771: sync 2024-03-30T04:28:21.446 INFO:tasks.workunit.client.0.smithi123.stdout:2/966: creat d0/d1f/d1c/d39/f136 x:0 0 0 2024-03-30T04:28:21.447 INFO:tasks.workunit.client.1.smithi135.stdout:5/413: write d1/d3/d2b/d54/d5e/d5c/f4e [496544,125749] 0 2024-03-30T04:28:21.448 INFO:tasks.workunit.client.0.smithi123.stdout:6/869: dread d10/d82/f66 [4194304,4194304] 0 2024-03-30T04:28:21.449 INFO:tasks.workunit.client.1.smithi135.stdout:5/414: stat d1/da/d17/d70 0 2024-03-30T04:28:21.450 INFO:tasks.workunit.client.0.smithi123.stdout:0/823: truncate d6/dd/d31/d35/fed 764776 0 2024-03-30T04:28:21.450 INFO:tasks.workunit.client.0.smithi123.stdout:1/855: link db/lf db/dd/d42/d8f/l11b 0 2024-03-30T04:28:21.453 INFO:tasks.workunit.client.1.smithi135.stdout:6/404: creat d2/dd/d14/f7f x:0 0 0 2024-03-30T04:28:21.454 INFO:tasks.workunit.client.0.smithi123.stdout:5/767: dwrite dd/d19/d5f/d99/fc7 [0,4194304] 0 2024-03-30T04:28:21.457 INFO:tasks.workunit.client.1.smithi135.stdout:5/415: mknod d1/d3/d2b/d54/d5e/d5c/d79/c7c 0 2024-03-30T04:28:21.459 INFO:tasks.workunit.client.0.smithi123.stdout:9/790: read d6/d1f/d45/f50 [1473927,64754] 0 2024-03-30T04:28:21.459 INFO:tasks.workunit.client.0.smithi123.stdout:7/789: link d2/da/c38 d2/da/d11/d61/d55/d9e/c109 0 2024-03-30T04:28:21.459 INFO:tasks.workunit.client.0.smithi123.stdout:0/824: dread - d6/dd/d15/d98/d4e/d69/d6e/dd8/fe7 zero size 2024-03-30T04:28:21.463 INFO:tasks.workunit.client.0.smithi123.stdout:2/967: write d0/d1f/d1c/d39/d4b/fa7 [5013136,52391] 0 2024-03-30T04:28:21.463 INFO:tasks.workunit.client.0.smithi123.stdout:5/768: fdatasync dd/d19/d1f/f27 0 2024-03-30T04:28:21.464 INFO:tasks.workunit.client.1.smithi135.stdout:6/405: creat d2/dd/d18/d21/f80 x:0 0 0 2024-03-30T04:28:21.465 INFO:tasks.workunit.client.0.smithi123.stdout:6/870: mknod d10/d82/dc1/d45/c124 0 2024-03-30T04:28:21.467 INFO:tasks.workunit.client.0.smithi123.stdout:5/769: truncate dd/d19/d5f/fca 968366 0 2024-03-30T04:28:21.468 INFO:tasks.workunit.client.1.smithi135.stdout:4/410: dread d10/d1a/f21 [4194304,4194304] 0 2024-03-30T04:28:21.471 INFO:tasks.workunit.client.0.smithi123.stdout:1/856: creat db/dd/d42/d8f/d78/f11c x:0 0 0 2024-03-30T04:28:21.474 INFO:tasks.workunit.client.0.smithi123.stdout:1/857: dread - db/dd/d42/f4f zero size 2024-03-30T04:28:21.478 INFO:tasks.workunit.client.0.smithi123.stdout:3/909: dwrite d13/d24/d31/d23/d3a/d3c/db8/fd3 [0,4194304] 0 2024-03-30T04:28:21.479 INFO:tasks.workunit.client.0.smithi123.stdout:9/791: rename d6/d1f/d45/d97/d69 to d6/d49/d113 0 2024-03-30T04:28:21.479 INFO:tasks.workunit.client.0.smithi123.stdout:3/910: stat d13/d24/d31/d23/d63 0 2024-03-30T04:28:21.481 INFO:tasks.workunit.client.0.smithi123.stdout:6/871: symlink d10/d20/l125 0 2024-03-30T04:28:21.483 INFO:tasks.workunit.client.1.smithi135.stdout:4/411: dread d10/f14 [0,4194304] 0 2024-03-30T04:28:21.484 INFO:tasks.workunit.client.0.smithi123.stdout:5/770: rmdir dd/d19/d3e/d5b/dd7 39 2024-03-30T04:28:21.486 INFO:tasks.workunit.client.0.smithi123.stdout:9/792: chown d6/d1f/d72/d7d/fee 4293874 1 2024-03-30T04:28:21.487 INFO:tasks.workunit.client.0.smithi123.stdout:7/790: creat d2/da/d11/d61/d55/f10a x:0 0 0 2024-03-30T04:28:21.487 INFO:tasks.workunit.client.0.smithi123.stdout:8/839: dread da/dc/d14/d9d/fb1 [0,4194304] 0 2024-03-30T04:28:21.489 INFO:tasks.workunit.client.0.smithi123.stdout:1/858: symlink db/dd/d42/d46/d56/da2/de0/l11d 0 2024-03-30T04:28:21.489 INFO:tasks.workunit.client.1.smithi135.stdout:6/406: dwrite d2/f4 [0,4194304] 0 2024-03-30T04:28:21.490 INFO:tasks.workunit.client.0.smithi123.stdout:6/872: truncate d10/d20/da2/fab 1038017 0 2024-03-30T04:28:21.491 INFO:tasks.workunit.client.0.smithi123.stdout:0/825: write d6/fcd [31643,108347] 0 2024-03-30T04:28:21.491 INFO:tasks.workunit.client.1.smithi135.stdout:4/412: write d10/f57 [2158907,62105] 0 2024-03-30T04:28:21.492 INFO:tasks.workunit.client.1.smithi135.stdout:6/407: stat d2/dd/l1b 0 2024-03-30T04:28:21.496 INFO:tasks.workunit.client.0.smithi123.stdout:9/793: readlink d6/d1f/d45/d97/l9e 0 2024-03-30T04:28:21.497 INFO:tasks.workunit.client.0.smithi123.stdout:3/911: dread d13/d24/d31/d123/d5d/da5/fdb [0,4194304] 0 2024-03-30T04:28:21.497 INFO:tasks.workunit.client.1.smithi135.stdout:0/514: dread d0/d5/d9/d23/d2a/d60/f48 [0,4194304] 0 2024-03-30T04:28:21.498 INFO:tasks.workunit.client.0.smithi123.stdout:5/771: dread - dd/d19/d1f/d35/d3a/d57/fb2 zero size 2024-03-30T04:28:21.501 INFO:tasks.workunit.client.0.smithi123.stdout:6/873: mkdir d10/d126 0 2024-03-30T04:28:21.501 INFO:tasks.workunit.client.1.smithi135.stdout:6/408: mkdir d2/d68/d81 0 2024-03-30T04:28:21.502 INFO:tasks.workunit.client.1.smithi135.stdout:0/515: rmdir d0/d5/d9/d23/d58 39 2024-03-30T04:28:21.503 INFO:tasks.workunit.client.0.smithi123.stdout:7/791: truncate d2/dc/d56/f67 1393369 0 2024-03-30T04:28:21.504 INFO:tasks.workunit.client.1.smithi135.stdout:6/409: write d2/dd/d14/d38/f7e [42923,41313] 0 2024-03-30T04:28:21.506 INFO:tasks.workunit.client.1.smithi135.stdout:0/516: truncate d0/d19/d2d/d79/f81 391977 0 2024-03-30T04:28:21.507 INFO:tasks.workunit.client.0.smithi123.stdout:6/874: fdatasync d10/d82/dc1/d6a/d7a/de2/df1/f10b 0 2024-03-30T04:28:21.509 INFO:tasks.workunit.client.1.smithi135.stdout:6/410: creat d2/dd/d40/f82 x:0 0 0 2024-03-30T04:28:21.510 INFO:tasks.workunit.client.1.smithi135.stdout:0/517: fsync d0/d5/d10/f11 0 2024-03-30T04:28:21.511 INFO:tasks.workunit.client.0.smithi123.stdout:3/912: unlink d13/d24/d31/de0/f105 0 2024-03-30T04:28:21.511 INFO:tasks.workunit.client.0.smithi123.stdout:7/792: mknod d2/dc/d7f/c10b 0 2024-03-30T04:28:21.512 INFO:tasks.workunit.client.1.smithi135.stdout:6/411: fdatasync d2/f39 0 2024-03-30T04:28:21.513 INFO:tasks.workunit.client.0.smithi123.stdout:6/875: creat d10/d82/dd6/f127 x:0 0 0 2024-03-30T04:28:21.514 INFO:tasks.workunit.client.0.smithi123.stdout:9/794: mkdir d6/d1f/d72/d7d/d10f/d114 0 2024-03-30T04:28:21.516 INFO:tasks.workunit.client.1.smithi135.stdout:6/412: getdents d2/d7d 0 2024-03-30T04:28:21.517 INFO:tasks.workunit.client.0.smithi123.stdout:7/793: chown d2/da/d22/d3b/cc0 59 1 2024-03-30T04:28:21.517 INFO:tasks.workunit.client.0.smithi123.stdout:5/772: creat dd/d19/d1f/d3c/d72/d8c/df4/f101 x:0 0 0 2024-03-30T04:28:21.517 INFO:tasks.workunit.client.0.smithi123.stdout:6/876: chown d10/d82/dc1/d45/d111 32293670 1 2024-03-30T04:28:21.518 INFO:tasks.workunit.client.1.smithi135.stdout:4/413: dwrite d10/f17 [0,4194304] 0 2024-03-30T04:28:21.521 INFO:tasks.workunit.client.1.smithi135.stdout:6/413: write d2/dd/d14/d30/d43/f69 [506201,121715] 0 2024-03-30T04:28:21.521 INFO:tasks.workunit.client.1.smithi135.stdout:4/414: fdatasync d10/d1a/d32/d40/f4b 0 2024-03-30T04:28:21.525 INFO:tasks.workunit.client.1.smithi135.stdout:4/415: chown d10/d1a/d32/d40/c64 279366 1 2024-03-30T04:28:21.525 INFO:tasks.workunit.client.0.smithi123.stdout:3/913: creat d13/d24/d31/d123/d5d/da5/dcd/f128 x:0 0 0 2024-03-30T04:28:21.526 INFO:tasks.workunit.client.0.smithi123.stdout:7/794: mkdir d2/dc/d7f/d10c 0 2024-03-30T04:28:21.529 INFO:tasks.workunit.client.1.smithi135.stdout:4/416: creat d10/d1a/d32/d40/f8e x:0 0 0 2024-03-30T04:28:21.529 INFO:tasks.workunit.client.1.smithi135.stdout:6/414: write d2/dd/d14/f1e [937262,2290] 0 2024-03-30T04:28:21.532 INFO:tasks.workunit.client.0.smithi123.stdout:9/795: mkdir d6/d1f/d72/d7d/d10f/d114/d115 0 2024-03-30T04:28:21.533 INFO:tasks.workunit.client.1.smithi135.stdout:0/518: dwrite d0/d5/d9/d23/d58/f1f [0,4194304] 0 2024-03-30T04:28:21.535 INFO:tasks.workunit.client.1.smithi135.stdout:6/415: rename d2/d2e/f70 to d2/d68/f83 0 2024-03-30T04:28:21.536 INFO:tasks.workunit.client.0.smithi123.stdout:6/877: getdents d10/d82/dc1/d45/d77/dd5 0 2024-03-30T04:28:21.539 INFO:tasks.workunit.client.0.smithi123.stdout:8/840: dread da/dc/f20 [0,4194304] 0 2024-03-30T04:28:21.541 INFO:tasks.workunit.client.0.smithi123.stdout:3/914: unlink d13/d24/d31/d23/d63/l99 0 2024-03-30T04:28:21.545 INFO:tasks.workunit.client.0.smithi123.stdout:7/795: mknod d2/da/d11/d34/c10d 0 2024-03-30T04:28:21.547 INFO:tasks.workunit.client.1.smithi135.stdout:6/416: read d2/f4d [3817097,98986] 0 2024-03-30T04:28:21.549 INFO:tasks.workunit.client.1.smithi135.stdout:9/370: sync 2024-03-30T04:28:21.551 INFO:tasks.workunit.client.1.smithi135.stdout:0/519: truncate d0/d5/d9/d23/d5d/f69 325058 0 2024-03-30T04:28:21.552 INFO:tasks.workunit.client.0.smithi123.stdout:0/826: dwrite d6/dd/d15/d28/fcc [4194304,4194304] 0 2024-03-30T04:28:21.554 INFO:tasks.workunit.client.1.smithi135.stdout:6/417: symlink d2/dd/d14/d38/l84 0 2024-03-30T04:28:21.555 INFO:tasks.workunit.client.1.smithi135.stdout:0/520: truncate d0/d1/d16/f85 4448950 0 2024-03-30T04:28:21.556 INFO:tasks.workunit.client.0.smithi123.stdout:5/773: creat dd/d19/d3e/d5b/d62/dd3/df3/f102 x:0 0 0 2024-03-30T04:28:21.557 INFO:tasks.workunit.client.1.smithi135.stdout:6/418: readlink d2/dd/d18/d21/d55/l59 0 2024-03-30T04:28:21.559 INFO:tasks.workunit.client.1.smithi135.stdout:6/419: dread - d2/dd/d14/f7f zero size 2024-03-30T04:28:21.560 INFO:tasks.workunit.client.0.smithi123.stdout:5/774: chown dd/d19/d1f/d3c/d72/d8c/deb 2 1 2024-03-30T04:28:21.562 INFO:tasks.workunit.client.0.smithi123.stdout:1/859: dread db/dd/d42/d46/d5d/d62/d9e/fff [0,4194304] 0 2024-03-30T04:28:21.563 INFO:tasks.workunit.client.1.smithi135.stdout:9/371: write d7/d41/f5f [927934,10535] 0 2024-03-30T04:28:21.564 INFO:tasks.workunit.client.0.smithi123.stdout:8/841: symlink da/dc/d14/db7/l108 0 2024-03-30T04:28:21.565 INFO:tasks.workunit.client.1.smithi135.stdout:9/372: chown d7/d4f/d5c/f5e 82481 1 2024-03-30T04:28:21.566 INFO:tasks.workunit.client.1.smithi135.stdout:4/417: dread d10/f74 [0,4194304] 0 2024-03-30T04:28:21.567 INFO:tasks.workunit.client.1.smithi135.stdout:0/521: write d0/d5/d9/fd [27792,66960] 0 2024-03-30T04:28:21.569 INFO:tasks.workunit.client.0.smithi123.stdout:9/796: mknod d6/d1f/d72/d7d/d10f/d114/d115/c116 0 2024-03-30T04:28:21.569 INFO:tasks.workunit.client.0.smithi123.stdout:5/775: chown dd/d19/d1f/d35/d3a/d79/cfc 4294 1 2024-03-30T04:28:21.571 INFO:tasks.workunit.client.0.smithi123.stdout:9/797: stat d6/d1f/c80 0 2024-03-30T04:28:21.571 INFO:tasks.workunit.client.0.smithi123.stdout:8/842: chown da/d27/d3c/d2d/dae/l103 1867776 1 2024-03-30T04:28:21.573 INFO:tasks.workunit.client.1.smithi135.stdout:4/418: rmdir d10/d1a 39 2024-03-30T04:28:21.574 INFO:tasks.workunit.client.1.smithi135.stdout:0/522: rmdir d0/d19/d2d 39 2024-03-30T04:28:21.575 INFO:tasks.workunit.client.1.smithi135.stdout:9/373: dread d7/d41/d72/f74 [0,4194304] 0 2024-03-30T04:28:21.575 INFO:tasks.workunit.client.0.smithi123.stdout:7/796: dwrite d2/da/d11/d34/d76/d91/f9f [0,4194304] 0 2024-03-30T04:28:21.577 INFO:tasks.workunit.client.1.smithi135.stdout:6/420: dwrite d2/f4 [0,4194304] 0 2024-03-30T04:28:21.579 INFO:tasks.workunit.client.1.smithi135.stdout:4/419: fdatasync d10/d1a/d42/f6e 0 2024-03-30T04:28:21.581 INFO:tasks.workunit.client.0.smithi123.stdout:0/827: write d6/dd/d31/f86 [489297,32309] 0 2024-03-30T04:28:21.581 INFO:tasks.workunit.client.0.smithi123.stdout:3/915: unlink d13/d24/d31/d123/d5d/l8d 0 2024-03-30T04:28:21.582 INFO:tasks.workunit.client.0.smithi123.stdout:5/776: creat dd/d19/d3e/f103 x:0 0 0 2024-03-30T04:28:21.583 INFO:tasks.workunit.client.0.smithi123.stdout:8/843: readlink da/dc/d14/d9d/laa 0 2024-03-30T04:28:21.584 INFO:tasks.workunit.client.0.smithi123.stdout:1/860: truncate db/dd/d42/d46/d51/fc2 814693 0 2024-03-30T04:28:21.586 INFO:tasks.workunit.client.0.smithi123.stdout:4/772: sync 2024-03-30T04:28:21.586 INFO:tasks.workunit.client.0.smithi123.stdout:2/968: sync 2024-03-30T04:28:21.588 INFO:tasks.workunit.client.0.smithi123.stdout:4/773: chown d2/d4/ff 8 1 2024-03-30T04:28:21.589 INFO:tasks.workunit.client.0.smithi123.stdout:9/798: creat d6/d1f/d72/d7d/d10f/d114/d115/f117 x:0 0 0 2024-03-30T04:28:21.590 INFO:tasks.workunit.client.1.smithi135.stdout:6/421: creat d2/dd/d14/d2b/f85 x:0 0 0 2024-03-30T04:28:21.591 INFO:tasks.workunit.client.0.smithi123.stdout:5/777: fsync dd/d41/f43 0 2024-03-30T04:28:21.591 INFO:tasks.workunit.client.0.smithi123.stdout:0/828: dread - d6/dd/d15/d98/dc6/ff9 zero size 2024-03-30T04:28:21.592 INFO:tasks.workunit.client.0.smithi123.stdout:9/799: dread - d6/fc5 zero size 2024-03-30T04:28:21.593 INFO:tasks.workunit.client.1.smithi135.stdout:4/420: rename d10/d1a/f52 to d10/d4d/d89/f8f 0 2024-03-30T04:28:21.595 INFO:tasks.workunit.client.1.smithi135.stdout:6/422: chown d2/dd/d14/d2b/f50 4 1 2024-03-30T04:28:21.596 INFO:tasks.workunit.client.0.smithi123.stdout:6/878: truncate d10/d82/f66 5862085 0 2024-03-30T04:28:21.597 INFO:tasks.workunit.client.1.smithi135.stdout:4/421: dread - d10/d1a/f5c zero size 2024-03-30T04:28:21.597 INFO:tasks.workunit.client.0.smithi123.stdout:5/778: chown dd/d19/d1f/d35/d3a/d57/f65 3 1 2024-03-30T04:28:21.598 INFO:tasks.workunit.client.0.smithi123.stdout:3/916: symlink d13/d24/d39/d79/dea/l129 0 2024-03-30T04:28:21.599 INFO:tasks.workunit.client.0.smithi123.stdout:1/861: mkdir db/dd/d42/d46/d56/d5c/d11e 0 2024-03-30T04:28:21.600 INFO:tasks.workunit.client.1.smithi135.stdout:6/423: dread - d2/dd/d14/f7f zero size 2024-03-30T04:28:21.600 INFO:tasks.workunit.client.1.smithi135.stdout:0/523: dwrite d0/d13/d1e/f78 [0,4194304] 0 2024-03-30T04:28:21.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:21 smithi135 ceph-mon[44364]: from='client.14676 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:28:21.602 INFO:tasks.workunit.client.1.smithi135.stdout:4/422: symlink d10/d1a/d42/d46/l90 0 2024-03-30T04:28:21.603 INFO:tasks.workunit.client.0.smithi123.stdout:4/774: mknod d2/d4/d7/d63/c109 0 2024-03-30T04:28:21.603 INFO:tasks.workunit.client.1.smithi135.stdout:9/374: dwrite d7/d1b/f28 [0,4194304] 0 2024-03-30T04:28:21.603 INFO:tasks.workunit.client.1.smithi135.stdout:4/423: stat d10/d4d/c50 0 2024-03-30T04:28:21.605 INFO:tasks.workunit.client.1.smithi135.stdout:9/375: chown d7/d1b/d37/f6a 9 1 2024-03-30T04:28:21.609 INFO:tasks.workunit.client.1.smithi135.stdout:9/376: write d7/d29/f3e [1527790,108687] 0 2024-03-30T04:28:21.611 INFO:tasks.workunit.client.1.smithi135.stdout:4/424: fsync d10/d4d/d89/f8b 0 2024-03-30T04:28:21.612 INFO:tasks.workunit.client.0.smithi123.stdout:5/779: mknod dd/d19/d3e/d5b/d62/dd3/c104 0 2024-03-30T04:28:21.612 INFO:tasks.workunit.client.0.smithi123.stdout:7/797: getdents d2/d4d/d80 0 2024-03-30T04:28:21.614 INFO:tasks.workunit.client.0.smithi123.stdout:1/862: dread db/fe1 [0,4194304] 0 2024-03-30T04:28:21.615 INFO:tasks.workunit.client.1.smithi135.stdout:0/524: dread d0/d5/d9/fb [0,4194304] 0 2024-03-30T04:28:21.616 INFO:tasks.workunit.client.1.smithi135.stdout:6/424: getdents d2/dd/d40 0 2024-03-30T04:28:21.617 INFO:tasks.workunit.client.0.smithi123.stdout:9/800: creat d6/d49/df9/f118 x:0 0 0 2024-03-30T04:28:21.617 INFO:tasks.workunit.client.0.smithi123.stdout:5/780: fsync dd/ffb 0 2024-03-30T04:28:21.618 INFO:tasks.workunit.client.0.smithi123.stdout:6/879: symlink d10/d82/dc1/dd2/l128 0 2024-03-30T04:28:21.621 INFO:tasks.workunit.client.0.smithi123.stdout:2/969: dwrite d0/d1f/d27/f8b [0,4194304] 0 2024-03-30T04:28:21.623 INFO:tasks.workunit.client.1.smithi135.stdout:6/425: rename d2/dd/d18/d21 to d2/dd/d14/d2b/d86 0 2024-03-30T04:28:21.625 INFO:tasks.workunit.client.1.smithi135.stdout:6/426: fsync d2/d2e/f31 0 2024-03-30T04:28:21.627 INFO:tasks.workunit.client.0.smithi123.stdout:7/798: readlink d2/da/d11/d61/d55/lb6 0 2024-03-30T04:28:21.627 INFO:tasks.workunit.client.0.smithi123.stdout:5/781: write dd/d19/d3e/f75 [181157,52952] 0 2024-03-30T04:28:21.629 INFO:tasks.workunit.client.0.smithi123.stdout:1/863: fsync f8 0 2024-03-30T04:28:21.630 INFO:tasks.workunit.client.1.smithi135.stdout:6/427: mkdir d2/dd/d18/d87 0 2024-03-30T04:28:21.632 INFO:tasks.workunit.client.1.smithi135.stdout:6/428: stat d2/f13 0 2024-03-30T04:28:21.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:21.635+0000 7f80b0600700 1 -- 172.21.15.123:0/3048380286 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80a806e9a0 msgr2=0x7f80a806ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:28:21.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:21.635+0000 7f80b0600700 1 --2- 172.21.15.123:0/3048380286 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80a806e9a0 0x7f80a806ee10 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f809c004940 tx=0x7f809c009e30 comp rx=0 tx=0).stop 2024-03-30T04:28:21.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:21.636+0000 7f80b0600700 1 -- 172.21.15.123:0/3048380286 shutdown_connections 2024-03-30T04:28:21.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:21.636+0000 7f80b0600700 1 --2- 172.21.15.123:0/3048380286 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80a806e9a0 0x7f80a806ee10 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:21.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:21.636+0000 7f80b0600700 1 --2- 172.21.15.123:0/3048380286 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f80a806f9d0 0x7f80a806e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:21.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:21.636+0000 7f80b0600700 1 -- 172.21.15.123:0/3048380286 >> 172.21.15.123:0/3048380286 conn(0x7f80a8069de0 msgr2=0x7f80a806c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:28:21.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:21.636+0000 7f80b0600700 1 -- 172.21.15.123:0/3048380286 shutdown_connections 2024-03-30T04:28:21.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:21.636+0000 7f80b0600700 1 -- 172.21.15.123:0/3048380286 wait complete. 2024-03-30T04:28:21.636 INFO:tasks.workunit.client.0.smithi123.stdout:4/775: dwrite d2/d26/fe4 [0,4194304] 0 2024-03-30T04:28:21.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:21.638+0000 7f80b0600700 1 Processor -- start 2024-03-30T04:28:21.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:21.638+0000 7f80b0600700 1 -- start start 2024-03-30T04:28:21.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:21.638+0000 7f80b0600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f80a806e9a0 0x7f80a811c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:28:21.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:21.638+0000 7f80b0600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80a806f9d0 0x7f80a811cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:28:21.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:21.638+0000 7f80b0600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a811d1d0 con 0x7f80a806f9d0 2024-03-30T04:28:21.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:21.638+0000 7f80b0600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a8121b70 con 0x7f80a806e9a0 2024-03-30T04:28:21.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:21.639+0000 7f80ad200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80a806f9d0 0x7f80a811cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:28:21.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:21.639+0000 7f80ad200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80a806f9d0 0x7f80a811cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58326/0 (socket says 172.21.15.123:58326) 2024-03-30T04:28:21.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:21.639+0000 7f80ad200700 1 -- 172.21.15.123:0/329845434 learned_addr learned my addr 172.21.15.123:0/329845434 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:28:21.639 INFO:tasks.workunit.client.0.smithi123.stdout:1/864: chown db/dd/f8e 3502498 1 2024-03-30T04:28:21.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:21.640+0000 7f80ad200700 1 -- 172.21.15.123:0/329845434 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f80a806e9a0 msgr2=0x7f80a811c600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-30T04:28:21.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:21.640+0000 7f80ad200700 1 --2- 172.21.15.123:0/329845434 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f80a806e9a0 0x7f80a811c600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:21.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:21.640+0000 7f80ad200700 1 -- 172.21.15.123:0/329845434 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f809c005040 con 0x7f80a806f9d0 2024-03-30T04:28:21.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:21.640+0000 7f80ad200700 1 --2- 172.21.15.123:0/329845434 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80a806f9d0 0x7f80a811cb40 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f809c00b3b0 tx=0x7f809c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:28:21.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:21.641+0000 7f80a5600700 1 -- 172.21.15.123:0/329845434 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f809c00e040 con 0x7f80a806f9d0 2024-03-30T04:28:21.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:21.642+0000 7f80a5600700 1 -- 172.21.15.123:0/329845434 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f809c004290 con 0x7f80a806f9d0 2024-03-30T04:28:21.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:21.642+0000 7f80a5600700 1 -- 172.21.15.123:0/329845434 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f809c01c3f0 con 0x7f80a806f9d0 2024-03-30T04:28:21.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:21.643+0000 7f80b0600700 1 -- 172.21.15.123:0/329845434 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80a8121d70 con 0x7f80a806f9d0 2024-03-30T04:28:21.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:21.643+0000 7f80b0600700 1 -- 172.21.15.123:0/329845434 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80a81222b0 con 0x7f80a806f9d0 2024-03-30T04:28:21.642 INFO:tasks.workunit.client.1.smithi135.stdout:4/425: dwrite d10/d1a/d2b/f31 [0,4194304] 0 2024-03-30T04:28:21.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:21.645+0000 7f80b0600700 1 -- 172.21.15.123:0/329845434 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80a804c710 con 0x7f80a806f9d0 2024-03-30T04:28:21.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:21.648+0000 7f80a5600700 1 -- 172.21.15.123:0/329845434 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) v1 ==== 88766+0+0 (secure 0 0 0) 0x7f809c00bc50 con 0x7f80a806f9d0 2024-03-30T04:28:21.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:21.649+0000 7f80a5600700 1 --2- 172.21.15.123:0/329845434 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f809006afa0 0x7f809006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:28:21.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:21.650+0000 7f80a5600700 1 -- 172.21.15.123:0/329845434 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f809c08ae00 con 0x7f80a806f9d0 2024-03-30T04:28:21.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:21.650+0000 7f80adc00700 1 --2- 172.21.15.123:0/329845434 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f809006afa0 0x7f809006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:28:21.650 INFO:tasks.workunit.client.1.smithi135.stdout:6/429: rmdir d2/d7d 0 2024-03-30T04:28:21.650 INFO:tasks.workunit.client.1.smithi135.stdout:0/525: dwrite d0/d5/d9/f2e [0,4194304] 0 2024-03-30T04:28:21.650 INFO:tasks.workunit.client.1.smithi135.stdout:6/430: write d2/d2e/f31 [3258234,75161] 0 2024-03-30T04:28:21.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:21.652+0000 7f80adc00700 1 --2- 172.21.15.123:0/329845434 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f809006afa0 0x7f809006d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8094006fd0 tx=0x7f8094006d30 comp rx=0 tx=0).ready entity=mgr.14608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:28:21.651 INFO:tasks.workunit.client.1.smithi135.stdout:0/526: write d0/d5/d10/f11 [12107356,28042] 0 2024-03-30T04:28:21.653 INFO:tasks.workunit.client.1.smithi135.stdout:0/527: chown d0/d13/d1e 34778815 1 2024-03-30T04:28:21.656 INFO:tasks.workunit.client.1.smithi135.stdout:0/528: chown d0/d5/d9/d23/d5d/la7 3385156 1 2024-03-30T04:28:21.659 INFO:tasks.workunit.client.1.smithi135.stdout:0/529: dread - d0/d5/d9/d23/d8a/fa9 zero size 2024-03-30T04:28:21.660 INFO:tasks.workunit.client.0.smithi123.stdout:5/782: rmdir dd/d19/d1f/d35/d3a 39 2024-03-30T04:28:21.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:21.659+0000 7f80a5600700 1 -- 172.21.15.123:0/329845434 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f809c058ce0 con 0x7f80a806f9d0 2024-03-30T04:28:21.665 INFO:tasks.workunit.client.1.smithi135.stdout:0/530: rename d0/d1/d16/l8d to d0/d5/d42/lb0 0 2024-03-30T04:28:21.666 INFO:tasks.workunit.client.0.smithi123.stdout:9/801: dwrite d6/d49/d113/daa/fd0 [0,4194304] 0 2024-03-30T04:28:21.667 INFO:tasks.workunit.client.1.smithi135.stdout:0/531: chown d0/d5/d9/l3d 7689 1 2024-03-30T04:28:21.669 INFO:tasks.workunit.client.1.smithi135.stdout:0/532: fsync d0/d5/d9/d23/f66 0 2024-03-30T04:28:21.670 INFO:tasks.workunit.client.0.smithi123.stdout:8/844: dwrite da/d27/d3c/d2d/d74/f7f [0,4194304] 0 2024-03-30T04:28:21.670 INFO:tasks.workunit.client.0.smithi123.stdout:1/865: dread - db/dd/fdb zero size 2024-03-30T04:28:21.671 INFO:tasks.workunit.client.0.smithi123.stdout:2/970: link d0/d1f/d16/f87 d0/d1f/d27/d68/dd6/f137 0 2024-03-30T04:28:21.672 INFO:tasks.workunit.client.1.smithi135.stdout:0/533: mknod d0/d19/d2d/cb1 0 2024-03-30T04:28:21.676 INFO:tasks.workunit.client.1.smithi135.stdout:0/534: unlink d0/d19/d2d/d79/f81 0 2024-03-30T04:28:21.678 INFO:tasks.workunit.client.0.smithi123.stdout:9/802: truncate d6/d7c/d82/f112 566961 0 2024-03-30T04:28:21.681 INFO:tasks.workunit.client.0.smithi123.stdout:5/783: unlink dd/d19/d1f/d35/fa9 0 2024-03-30T04:28:21.683 INFO:tasks.workunit.client.0.smithi123.stdout:3/917: dread d13/d24/d31/d123/d5d/f68 [0,4194304] 0 2024-03-30T04:28:21.686 INFO:tasks.workunit.client.0.smithi123.stdout:8/845: mknod da/dc/d14/d9c/c109 0 2024-03-30T04:28:21.687 INFO:tasks.workunit.client.0.smithi123.stdout:6/880: link d10/d82/d8e/dc7/c4b d10/d20/c129 0 2024-03-30T04:28:21.688 INFO:tasks.workunit.client.0.smithi123.stdout:2/971: symlink d0/d1f/d27/d68/l138 0 2024-03-30T04:28:21.695 INFO:tasks.workunit.client.1.smithi135.stdout:0/535: dwrite d0/d5/d9/f17 [0,4194304] 0 2024-03-30T04:28:21.697 INFO:tasks.workunit.client.0.smithi123.stdout:8/846: mknod da/d99/c10a 0 2024-03-30T04:28:21.699 INFO:tasks.workunit.client.0.smithi123.stdout:1/866: creat db/dd/d5f/f11f x:0 0 0 2024-03-30T04:28:21.703 INFO:tasks.workunit.client.0.smithi123.stdout:3/918: unlink d13/d24/d31/d123/ca0 0 2024-03-30T04:28:21.704 INFO:tasks.workunit.client.0.smithi123.stdout:5/784: mknod dd/d19/d1f/d3c/d9e/df5/c105 0 2024-03-30T04:28:21.707 INFO:tasks.workunit.client.0.smithi123.stdout:0/829: dread d6/dd/d15/f2b [0,4194304] 0 2024-03-30T04:28:21.709 INFO:tasks.workunit.client.1.smithi135.stdout:0/536: dread d0/d13/d1e/f78 [0,4194304] 0 2024-03-30T04:28:21.711 INFO:tasks.workunit.client.0.smithi123.stdout:2/972: creat d0/d1f/d1c/d39/f139 x:0 0 0 2024-03-30T04:28:21.712 INFO:tasks.workunit.client.1.smithi135.stdout:0/537: write d0/d19/d2d/f98 [2448012,30045] 0 2024-03-30T04:28:21.713 INFO:tasks.workunit.client.0.smithi123.stdout:0/830: dread - d6/dd/d31/d35/d3a/f95 zero size 2024-03-30T04:28:21.713 INFO:tasks.workunit.client.0.smithi123.stdout:6/881: chown d10/d82/d9a/ca6 0 1 2024-03-30T04:28:21.715 INFO:tasks.workunit.client.0.smithi123.stdout:2/973: write d0/d1f/d1c/f41 [392473,106340] 0 2024-03-30T04:28:21.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:21 smithi123 ceph-mon[35245]: from='client.14676 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:28:21.717 INFO:tasks.workunit.client.1.smithi135.stdout:0/538: dread - d0/d13/d53/d7f/d83/fac zero size 2024-03-30T04:28:21.717 INFO:tasks.workunit.client.0.smithi123.stdout:1/867: creat db/dd/d42/d46/d51/d72/f120 x:0 0 0 2024-03-30T04:28:21.718 INFO:tasks.workunit.client.1.smithi135.stdout:0/539: write d0/d5/d9/d23/f66 [4669147,63231] 0 2024-03-30T04:28:21.719 INFO:tasks.workunit.client.1.smithi135.stdout:6/431: dread d2/dd/d14/f15 [0,4194304] 0 2024-03-30T04:28:21.720 INFO:tasks.workunit.client.0.smithi123.stdout:3/919: creat d13/d24/d31/d23/d3a/d3c/db8/d10b/f12a x:0 0 0 2024-03-30T04:28:21.722 INFO:tasks.workunit.client.0.smithi123.stdout:5/785: creat dd/d19/d5f/d99/f106 x:0 0 0 2024-03-30T04:28:21.722 INFO:tasks.workunit.client.1.smithi135.stdout:0/540: creat d0/d13/d1e/fb2 x:0 0 0 2024-03-30T04:28:21.724 INFO:tasks.workunit.client.0.smithi123.stdout:8/847: mkdir da/d27/d3c/d2d/dae/dd9/d10b 0 2024-03-30T04:28:21.724 INFO:tasks.workunit.client.1.smithi135.stdout:6/432: rename d2/f39 to d2/d68/d81/f88 0 2024-03-30T04:28:21.725 INFO:tasks.workunit.client.0.smithi123.stdout:2/974: write d0/d1f/d16/ffa [237861,67916] 0 2024-03-30T04:28:21.728 INFO:tasks.workunit.client.0.smithi123.stdout:5/786: fsync dd/d19/d1f/d3c/d72/ff6 0 2024-03-30T04:28:21.728 INFO:tasks.workunit.client.0.smithi123.stdout:6/882: truncate d10/d82/dc1/d45/f67 4940933 0 2024-03-30T04:28:21.728 INFO:tasks.workunit.client.1.smithi135.stdout:0/541: unlink d0/d13/d53/d7f/d83/l94 0 2024-03-30T04:28:21.732 INFO:tasks.workunit.client.0.smithi123.stdout:3/920: read d13/d24/d31/d30/f45 [2368119,98903] 0 2024-03-30T04:28:21.733 INFO:tasks.workunit.client.0.smithi123.stdout:0/831: symlink d6/dd/l108 0 2024-03-30T04:28:21.734 INFO:tasks.workunit.client.1.smithi135.stdout:6/433: dread d2/f8 [0,4194304] 0 2024-03-30T04:28:21.735 INFO:tasks.workunit.client.1.smithi135.stdout:6/434: chown d2/dd/c32 350 1 2024-03-30T04:28:21.735 INFO:tasks.workunit.client.0.smithi123.stdout:7/799: dread d2/dc/d56/fdd [0,4194304] 0 2024-03-30T04:28:21.735 INFO:tasks.workunit.client.0.smithi123.stdout:6/883: unlink d10/d82/l88 0 2024-03-30T04:28:21.738 INFO:tasks.workunit.client.0.smithi123.stdout:8/848: creat da/d27/d3c/d64/d78/db8/f10c x:0 0 0 2024-03-30T04:28:21.739 INFO:tasks.workunit.client.1.smithi135.stdout:6/435: truncate d2/dd/d14/d53/f77 4218097 0 2024-03-30T04:28:21.741 INFO:tasks.workunit.client.0.smithi123.stdout:6/884: chown d10/d20/fce 40346726 1 2024-03-30T04:28:21.741 INFO:tasks.workunit.client.1.smithi135.stdout:0/542: dread d0/d13/f74 [4194304,4194304] 0 2024-03-30T04:28:21.742 INFO:tasks.workunit.client.0.smithi123.stdout:2/975: truncate d0/d1f/d1c/fb3 541910 0 2024-03-30T04:28:21.743 INFO:tasks.workunit.client.0.smithi123.stdout:1/868: symlink db/dd/d42/d46/d5d/d62/d9e/l121 0 2024-03-30T04:28:21.743 INFO:tasks.workunit.client.1.smithi135.stdout:0/543: fdatasync d0/d1/f43 0 2024-03-30T04:28:21.745 INFO:tasks.workunit.client.1.smithi135.stdout:6/436: read d2/dd/d14/d30/f67 [3579202,9435] 0 2024-03-30T04:28:21.746 INFO:tasks.workunit.client.0.smithi123.stdout:3/921: write d13/d24/d31/d23/d3a/f40 [8582279,109035] 0 2024-03-30T04:28:21.746 INFO:tasks.workunit.client.1.smithi135.stdout:0/544: unlink d0/d5/f56 0 2024-03-30T04:28:21.747 INFO:tasks.workunit.client.0.smithi123.stdout:1/869: chown db/dd/d42/l3d 5 1 2024-03-30T04:28:21.747 INFO:tasks.workunit.client.0.smithi123.stdout:0/832: read d6/dd/d18/d1f/faf [1071539,106339] 0 2024-03-30T04:28:21.750 INFO:tasks.workunit.client.0.smithi123.stdout:6/885: symlink d10/d82/dd6/l12a 0 2024-03-30T04:28:21.751 INFO:tasks.workunit.client.1.smithi135.stdout:0/545: mkdir d0/d1/d16/d7e/db3 0 2024-03-30T04:28:21.752 INFO:tasks.workunit.client.0.smithi123.stdout:9/803: dread d6/f7e [0,4194304] 0 2024-03-30T04:28:21.756 INFO:tasks.workunit.client.0.smithi123.stdout:3/922: truncate d13/d24/d39/d79/d90/fc2 2089166 0 2024-03-30T04:28:21.758 INFO:tasks.workunit.client.1.smithi135.stdout:6/437: write d2/d36/f44 [2130260,106252] 0 2024-03-30T04:28:21.760 INFO:tasks.workunit.client.0.smithi123.stdout:1/870: fsync db/f24 0 2024-03-30T04:28:21.766 INFO:tasks.workunit.client.0.smithi123.stdout:6/886: mknod d10/d82/dc1/d6a/d7a/d99/dba/c12b 0 2024-03-30T04:28:21.766 INFO:tasks.workunit.client.1.smithi135.stdout:6/438: write d2/f9 [1131207,61773] 0 2024-03-30T04:28:21.769 INFO:tasks.workunit.client.0.smithi123.stdout:2/976: mknod d0/d1f/d27/d68/dd6/d109/d128/c13a 0 2024-03-30T04:28:21.769 INFO:tasks.workunit.client.1.smithi135.stdout:6/439: rmdir d2/dd/d14/d2b/d86/d55 39 2024-03-30T04:28:21.769 INFO:tasks.workunit.client.1.smithi135.stdout:0/546: dwrite d0/d1/d16/f85 [0,4194304] 0 2024-03-30T04:28:21.770 INFO:tasks.workunit.client.0.smithi123.stdout:4/776: sync 2024-03-30T04:28:21.770 INFO:tasks.workunit.client.0.smithi123.stdout:1/871: write db/f59 [2827161,127020] 0 2024-03-30T04:28:21.772 INFO:tasks.workunit.client.1.smithi135.stdout:0/547: mknod d0/d34/cb4 0 2024-03-30T04:28:21.773 INFO:tasks.workunit.client.1.smithi135.stdout:6/440: unlink d2/dd/d14/f3f 0 2024-03-30T04:28:21.775 INFO:tasks.workunit.client.0.smithi123.stdout:2/977: chown d0/d1f/d1c/d39/df2/cfc 1024794 1 2024-03-30T04:28:21.775 INFO:tasks.workunit.client.1.smithi135.stdout:0/548: mkdir d0/db5 0 2024-03-30T04:28:21.776 INFO:tasks.workunit.client.1.smithi135.stdout:6/441: symlink d2/l89 0 2024-03-30T04:28:21.776 INFO:tasks.workunit.client.1.smithi135.stdout:7/449: sync 2024-03-30T04:28:21.776 INFO:tasks.workunit.client.1.smithi135.stdout:3/440: sync 2024-03-30T04:28:21.776 INFO:tasks.workunit.client.1.smithi135.stdout:8/389: sync 2024-03-30T04:28:21.777 INFO:tasks.workunit.client.1.smithi135.stdout:1/377: sync 2024-03-30T04:28:21.777 INFO:tasks.workunit.client.0.smithi123.stdout:5/787: rename dd/d19/d1f/d35/d3a/d57/db3/fdb to dd/d19/d1f/f107 0 2024-03-30T04:28:21.779 INFO:tasks.workunit.client.0.smithi123.stdout:6/887: mkdir d10/d82/dc1/d45/d77/d12c 0 2024-03-30T04:28:21.781 INFO:tasks.workunit.client.1.smithi135.stdout:6/442: rename d2/dd to d2/dd/d14/d53/d8a 22 2024-03-30T04:28:21.783 INFO:tasks.workunit.client.0.smithi123.stdout:5/788: dread - dd/d19/d3e/d5b/d62/dd3/fe2 zero size 2024-03-30T04:28:21.785 INFO:tasks.workunit.client.1.smithi135.stdout:1/378: creat d16/d5a/d5f/f7f x:0 0 0 2024-03-30T04:28:21.785 INFO:tasks.workunit.client.1.smithi135.stdout:7/450: mkdir d4/db/d1a/d3d/d96 0 2024-03-30T04:28:21.788 INFO:tasks.workunit.client.1.smithi135.stdout:1/379: dread - d16/d5a/d5f/d3a/d6f/f7c zero size 2024-03-30T04:28:21.788 INFO:tasks.workunit.client.1.smithi135.stdout:6/443: unlink d2/fc 0 2024-03-30T04:28:21.789 INFO:tasks.workunit.client.1.smithi135.stdout:8/390: creat d6/d8/f71 x:0 0 0 2024-03-30T04:28:21.790 INFO:tasks.workunit.client.0.smithi123.stdout:3/923: dwrite f11 [0,4194304] 0 2024-03-30T04:28:21.790 INFO:tasks.workunit.client.1.smithi135.stdout:1/380: readlink d16/d19/d20/l63 0 2024-03-30T04:28:21.792 INFO:tasks.workunit.client.0.smithi123.stdout:6/888: unlink d10/d82/d8e/ffe 0 2024-03-30T04:28:21.793 INFO:tasks.workunit.client.0.smithi123.stdout:1/872: dread db/dd/d5f/dae/ffd [0,4194304] 0 2024-03-30T04:28:21.793 INFO:tasks.workunit.client.1.smithi135.stdout:6/444: creat d2/dd/d14/d38/f8b x:0 0 0 2024-03-30T04:28:21.793 INFO:tasks.workunit.client.1.smithi135.stdout:8/391: write d6/d8/d15/d1f/f53 [96929,80830] 0 2024-03-30T04:28:21.795 INFO:tasks.workunit.client.0.smithi123.stdout:5/789: creat dd/d19/d1f/d3c/d72/d8c/df4/f108 x:0 0 0 2024-03-30T04:28:21.796 INFO:tasks.workunit.client.1.smithi135.stdout:6/445: fsync d2/f11 0 2024-03-30T04:28:21.796 INFO:tasks.workunit.client.1.smithi135.stdout:1/381: fsync d16/d29/d2c/d43/d50/f5b 0 2024-03-30T04:28:21.797 INFO:tasks.workunit.client.1.smithi135.stdout:7/451: rename d4/db/d1a/d5c/l6d to d4/db/d1a/d4b/d4f/l97 0 2024-03-30T04:28:21.798 INFO:tasks.workunit.client.0.smithi123.stdout:0/833: dread d6/dd/d31/fa9 [0,4194304] 0 2024-03-30T04:28:21.799 INFO:tasks.workunit.client.0.smithi123.stdout:2/978: creat d0/d1f/d27/d68/dd6/d126/f13b x:0 0 0 2024-03-30T04:28:21.801 INFO:tasks.workunit.client.1.smithi135.stdout:0/549: write d0/d19/d2d/f38 [424529,51159] 0 2024-03-30T04:28:21.802 INFO:tasks.workunit.client.1.smithi135.stdout:3/441: dread d3/d50/f36 [0,4194304] 0 2024-03-30T04:28:21.803 INFO:tasks.workunit.client.1.smithi135.stdout:6/446: truncate d2/f4d 5109191 0 2024-03-30T04:28:21.804 INFO:tasks.workunit.client.1.smithi135.stdout:8/392: mkdir d6/d8/d15/d54/d69/d72 0 2024-03-30T04:28:21.805 INFO:tasks.workunit.client.1.smithi135.stdout:7/452: mkdir d4/db/d1a/d5c/d2c/d2e/d98 0 2024-03-30T04:28:21.807 INFO:tasks.workunit.client.1.smithi135.stdout:6/447: dread - d2/dd/d14/d38/f8b zero size 2024-03-30T04:28:21.809 INFO:tasks.workunit.client.1.smithi135.stdout:6/448: read d2/f8 [421,19037] 0 2024-03-30T04:28:21.809 INFO:tasks.workunit.client.0.smithi123.stdout:4/777: dwrite d2/d4/d2c/d42/fc3 [0,4194304] 0 2024-03-30T04:28:21.811 INFO:tasks.workunit.client.1.smithi135.stdout:6/449: stat d2/l6 0 2024-03-30T04:28:21.811 INFO:tasks.workunit.client.1.smithi135.stdout:8/393: rmdir d6/d8/d15/d54/d69 39 2024-03-30T04:28:21.811 INFO:tasks.workunit.client.1.smithi135.stdout:7/453: symlink d4/db/d1a/d5c/d59/d76/l99 0 2024-03-30T04:28:21.812 INFO:tasks.workunit.client.0.smithi123.stdout:5/790: write dd/d19/d3e/d5b/d62/f84 [4755925,125744] 0 2024-03-30T04:28:21.813 INFO:tasks.workunit.client.0.smithi123.stdout:0/834: dread d6/dd/d15/d98/d4e/d9a/db2/fc9 [0,4194304] 0 2024-03-30T04:28:21.818 INFO:tasks.workunit.client.1.smithi135.stdout:6/450: symlink d2/d36/d76/l8c 0 2024-03-30T04:28:21.822 INFO:tasks.workunit.client.0.smithi123.stdout:8/849: write da/dc/f16 [1093882,4379] 0 2024-03-30T04:28:21.822 INFO:tasks.workunit.client.0.smithi123.stdout:2/979: dread d0/d1f/f17 [4194304,4194304] 0 2024-03-30T04:28:21.823 INFO:tasks.workunit.client.0.smithi123.stdout:1/873: mkdir db/d122 0 2024-03-30T04:28:21.824 INFO:tasks.workunit.client.0.smithi123.stdout:7/800: write d2/dc/d49/d68/f9b [798668,16051] 0 2024-03-30T04:28:21.826 INFO:tasks.workunit.client.0.smithi123.stdout:8/850: write da/d27/ff6 [1231144,13256] 0 2024-03-30T04:28:21.827 INFO:tasks.workunit.client.1.smithi135.stdout:8/394: mkdir d6/d8/d15/d54/d69/d72/d73 0 2024-03-30T04:28:21.827 INFO:tasks.workunit.client.0.smithi123.stdout:3/924: fdatasync d13/d24/d31/d23/d63/f95 0 2024-03-30T04:28:21.829 INFO:tasks.workunit.client.1.smithi135.stdout:1/382: dwrite d16/d5a/d5f/d3a/d6f/f7d [0,4194304] 0 2024-03-30T04:28:21.831 INFO:tasks.workunit.client.0.smithi123.stdout:5/791: creat dd/d19/d1f/d3c/d72/d8c/df4/f109 x:0 0 0 2024-03-30T04:28:21.832 INFO:tasks.workunit.client.0.smithi123.stdout:3/925: write d13/d24/f70 [281182,99866] 0 2024-03-30T04:28:21.835 INFO:tasks.workunit.client.0.smithi123.stdout:0/835: rmdir d6/dd/d15/d98/d4e/d69 39 2024-03-30T04:28:21.836 INFO:tasks.workunit.client.1.smithi135.stdout:3/442: write d3/d10/f44 [923522,70364] 0 2024-03-30T04:28:21.837 INFO:tasks.workunit.client.1.smithi135.stdout:8/395: symlink d6/d8/d15/d2a/l74 0 2024-03-30T04:28:21.838 INFO:tasks.workunit.client.1.smithi135.stdout:1/383: mknod d16/d5a/d6b/d77/c80 0 2024-03-30T04:28:21.840 INFO:tasks.workunit.client.1.smithi135.stdout:3/443: readlink d3/d10/d16/d41/d83/l84 0 2024-03-30T04:28:21.840 INFO:tasks.workunit.client.1.smithi135.stdout:6/451: link d2/dd/d14/d53/l5c d2/dd/d14/d38/l8d 0 2024-03-30T04:28:21.841 INFO:tasks.workunit.client.0.smithi123.stdout:2/980: mknod d0/d1f/d1c/d36/c13c 0 2024-03-30T04:28:21.842 INFO:tasks.workunit.client.1.smithi135.stdout:3/444: stat d3/d54/d5c/f89 0 2024-03-30T04:28:21.842 INFO:tasks.workunit.client.0.smithi123.stdout:1/874: creat db/dd/d42/d46/d51/dd7/f123 x:0 0 0 2024-03-30T04:28:21.844 INFO:tasks.workunit.client.1.smithi135.stdout:3/445: write d3/d10/d16/d23/f7f [323769,11625] 0 2024-03-30T04:28:21.845 INFO:tasks.workunit.client.0.smithi123.stdout:1/875: chown db/dd/d42 2125730554 1 2024-03-30T04:28:21.847 INFO:tasks.workunit.client.0.smithi123.stdout:6/889: write d10/d82/d8e/dc7/fed [398344,52262] 0 2024-03-30T04:28:21.849 INFO:tasks.workunit.client.1.smithi135.stdout:0/550: dread d0/d5/d10/fa1 [0,4194304] 0 2024-03-30T04:28:21.850 INFO:tasks.workunit.client.0.smithi123.stdout:7/801: creat d2/da/d11/d34/d85/d8e/f10e x:0 0 0 2024-03-30T04:28:21.851 INFO:tasks.workunit.client.1.smithi135.stdout:1/384: unlink d16/d5a/d5f/d3d/f7e 0 2024-03-30T04:28:21.852 INFO:tasks.workunit.client.1.smithi135.stdout:7/454: truncate d4/f54 2175639 0 2024-03-30T04:28:21.854 INFO:tasks.workunit.client.1.smithi135.stdout:7/455: rename d4 to d4/db/d1a/d3d/d9a 22 2024-03-30T04:28:21.856 INFO:tasks.workunit.client.0.smithi123.stdout:6/890: unlink d10/d20/fce 0 2024-03-30T04:28:21.858 INFO:tasks.workunit.client.0.smithi123.stdout:3/926: creat d13/d24/d31/df7/f12b x:0 0 0 2024-03-30T04:28:21.859 INFO:tasks.workunit.client.1.smithi135.stdout:1/385: rmdir d16/d5a 39 2024-03-30T04:28:21.861 INFO:tasks.workunit.client.1.smithi135.stdout:0/551: mknod d0/cb6 0 2024-03-30T04:28:21.862 INFO:tasks.workunit.client.0.smithi123.stdout:5/792: creat dd/d19/d1f/d3c/d72/d8c/deb/f10a x:0 0 0 2024-03-30T04:28:21.863 INFO:tasks.workunit.client.1.smithi135.stdout:8/396: dwrite d6/d8/d15/d1f/f4c [0,4194304] 0 2024-03-30T04:28:21.864 INFO:tasks.workunit.client.0.smithi123.stdout:2/981: dread d0/d1f/d30/d119/de6/f107 [0,4194304] 0 2024-03-30T04:28:21.865 INFO:tasks.workunit.client.1.smithi135.stdout:0/552: readlink d0/d13/d53/d7f/d83/d9c/l6c 0 2024-03-30T04:28:21.865 INFO:tasks.workunit.client.1.smithi135.stdout:8/397: fdatasync d6/d8/d15/d54/f59 0 2024-03-30T04:28:21.866 INFO:tasks.workunit.client.1.smithi135.stdout:0/553: readlink d0/d13/d53/la5 0 2024-03-30T04:28:21.868 INFO:tasks.workunit.client.1.smithi135.stdout:7/456: rename d4/db/de/c64 to d4/db/d1a/d5c/d82/c9b 0 2024-03-30T04:28:21.870 INFO:tasks.workunit.client.0.smithi123.stdout:8/851: write da/dc/f13 [822189,55945] 0 2024-03-30T04:28:21.870 INFO:tasks.workunit.client.0.smithi123.stdout:0/836: creat d6/dd/d18/f109 x:0 0 0 2024-03-30T04:28:21.872 INFO:tasks.workunit.client.1.smithi135.stdout:8/398: mknod d6/d8/d14/c75 0 2024-03-30T04:28:21.872 INFO:tasks.workunit.client.1.smithi135.stdout:7/457: dread - d4/db/d1a/d5c/d5d/f86 zero size 2024-03-30T04:28:21.874 INFO:tasks.workunit.client.0.smithi123.stdout:7/802: getdents d2/d4d/d80/d98 0 2024-03-30T04:28:21.875 INFO:tasks.workunit.client.1.smithi135.stdout:1/386: symlink d16/d1a/l81 0 2024-03-30T04:28:21.876 INFO:tasks.workunit.client.1.smithi135.stdout:0/554: creat d0/d5/d9/d23/d58/d3b/fb7 x:0 0 0 2024-03-30T04:28:21.877 INFO:tasks.workunit.client.0.smithi123.stdout:1/876: getdents db/dd/d42/d46/d56/d5c 0 2024-03-30T04:28:21.878 INFO:tasks.workunit.client.1.smithi135.stdout:8/399: readlink d6/d8/l10 0 2024-03-30T04:28:21.878 INFO:tasks.workunit.client.1.smithi135.stdout:7/458: chown d4/db/d1a/d5c/d2c/d83/c8e 1 1 2024-03-30T04:28:21.878 INFO:tasks.workunit.client.0.smithi123.stdout:2/982: mknod d0/d1f/d5e/c13d 0 2024-03-30T04:28:21.878 INFO:tasks.workunit.client.1.smithi135.stdout:6/452: write d2/f4d [4404438,65547] 0 2024-03-30T04:28:21.882 INFO:tasks.workunit.client.0.smithi123.stdout:5/793: creat dd/d19/d1f/d3c/d72/d8c/f10b x:0 0 0 2024-03-30T04:28:21.883 INFO:tasks.workunit.client.1.smithi135.stdout:1/387: creat d16/d5a/f82 x:0 0 0 2024-03-30T04:28:21.883 INFO:tasks.workunit.client.1.smithi135.stdout:3/446: dread d3/d6e/f80 [0,4194304] 0 2024-03-30T04:28:21.885 INFO:tasks.workunit.client.0.smithi123.stdout:8/852: creat da/d27/d3c/d2d/d5c/f10d x:0 0 0 2024-03-30T04:28:21.886 INFO:tasks.workunit.client.0.smithi123.stdout:2/983: dread - d0/d37/d57/fda zero size 2024-03-30T04:28:21.887 INFO:tasks.workunit.client.1.smithi135.stdout:8/400: symlink d6/d8/d15/d54/d69/d72/l76 0 2024-03-30T04:28:21.889 INFO:tasks.workunit.client.1.smithi135.stdout:3/447: write d3/d54/d5c/f64 [1676408,30658] 0 2024-03-30T04:28:21.890 INFO:tasks.workunit.client.1.smithi135.stdout:1/388: read d16/d29/d2c/d3c/f3b [1712949,35996] 0 2024-03-30T04:28:21.891 INFO:tasks.workunit.client.0.smithi123.stdout:4/778: dwrite d2/d4/d7/f4e [0,4194304] 0 2024-03-30T04:28:21.892 INFO:tasks.workunit.client.0.smithi123.stdout:3/927: write d13/d24/d31/d23/d3a/d3c/db8/d10b/f12a [905159,37917] 0 2024-03-30T04:28:21.892 INFO:tasks.workunit.client.1.smithi135.stdout:6/453: mkdir d2/dd/d14/d2b/d86/d8e 0 2024-03-30T04:28:21.895 INFO:tasks.workunit.client.0.smithi123.stdout:9/804: sync 2024-03-30T04:28:21.897 INFO:tasks.workunit.client.0.smithi123.stdout:7/803: chown d2/da/d11/d61/d55/d59/db5/dab/dca 56565 1 2024-03-30T04:28:21.897 INFO:tasks.workunit.client.1.smithi135.stdout:8/401: rmdir d6/d8/d14 39 2024-03-30T04:28:21.899 INFO:tasks.workunit.client.0.smithi123.stdout:2/984: mknod d0/d1f/d1c/d10d/c13e 0 2024-03-30T04:28:21.899 INFO:tasks.workunit.client.0.smithi123.stdout:9/805: readlink d6/d1f/d45/l57 0 2024-03-30T04:28:21.899 INFO:tasks.workunit.client.1.smithi135.stdout:8/402: stat d6/d8/d15/d2a/d35/c3a 0 2024-03-30T04:28:21.900 INFO:tasks.workunit.client.1.smithi135.stdout:3/448: truncate d3/d10/d16/d23/d6a/d86/f8c 847979 0 2024-03-30T04:28:21.902 INFO:tasks.workunit.client.0.smithi123.stdout:0/837: rename d6/dd/l10 to d6/dd/d18/l10a 0 2024-03-30T04:28:21.904 INFO:tasks.workunit.client.0.smithi123.stdout:5/794: rmdir dd/d19/d3e/d5b/dd7 39 2024-03-30T04:28:21.905 INFO:tasks.workunit.client.1.smithi135.stdout:0/555: dwrite d0/d5/d9/f2e [0,4194304] 0 2024-03-30T04:28:21.908 INFO:tasks.workunit.client.1.smithi135.stdout:3/449: creat d3/d10/d16/d23/d6a/f9e x:0 0 0 2024-03-30T04:28:21.909 INFO:tasks.workunit.client.0.smithi123.stdout:0/838: read d6/dd/d15/fc3 [1985582,114294] 0 2024-03-30T04:28:21.911 INFO:tasks.workunit.client.1.smithi135.stdout:0/556: readlink d0/d1/d16/l70 0 2024-03-30T04:28:21.911 INFO:tasks.workunit.client.1.smithi135.stdout:3/450: chown d3/f11 12 1 2024-03-30T04:28:21.912 INFO:tasks.workunit.client.0.smithi123.stdout:2/985: write d0/d1f/d1c/d20/f56 [86781,73023] 0 2024-03-30T04:28:21.913 INFO:tasks.workunit.client.1.smithi135.stdout:3/451: chown d3/d6e/d9d 0 1 2024-03-30T04:28:21.914 INFO:tasks.workunit.client.0.smithi123.stdout:6/891: dread d10/d20/f2b [0,4194304] 0 2024-03-30T04:28:21.914 INFO:tasks.workunit.client.0.smithi123.stdout:3/928: mknod d13/d24/d31/d23/d3a/d3c/d66/c12c 0 2024-03-30T04:28:21.915 INFO:tasks.workunit.client.0.smithi123.stdout:9/806: dread d6/d1f/d72/fe9 [0,4194304] 0 2024-03-30T04:28:21.915 INFO:tasks.workunit.client.1.smithi135.stdout:7/459: truncate d4/db/d1a/d5c/d2c/d51/f7e 628391 0 2024-03-30T04:28:21.915 INFO:tasks.workunit.client.1.smithi135.stdout:0/557: chown d0/db5 17059019 1 2024-03-30T04:28:21.918 INFO:tasks.workunit.client.0.smithi123.stdout:7/804: chown d2/da/d11/d34/ded/dfc/l100 97318 1 2024-03-30T04:28:21.918 INFO:tasks.workunit.client.1.smithi135.stdout:2/340: sync 2024-03-30T04:28:21.920 INFO:tasks.workunit.client.0.smithi123.stdout:3/929: truncate ff 4316156 0 2024-03-30T04:28:21.921 INFO:tasks.workunit.client.1.smithi135.stdout:1/389: dwrite d16/d5a/f82 [0,4194304] 0 2024-03-30T04:28:21.923 INFO:tasks.workunit.client.1.smithi135.stdout:8/403: getdents d6/d23 0 2024-03-30T04:28:21.926 INFO:tasks.workunit.client.1.smithi135.stdout:2/341: fsync d4/d32/d2e/f5c 0 2024-03-30T04:28:21.927 INFO:tasks.workunit.client.1.smithi135.stdout:0/558: readlink d0/d13/d53/d54/d99/la6 0 2024-03-30T04:28:21.927 INFO:tasks.workunit.client.1.smithi135.stdout:3/452: creat d3/d10/d16/d23/f9f x:0 0 0 2024-03-30T04:28:21.929 INFO:tasks.workunit.client.0.smithi123.stdout:2/986: unlink d0/d1f/fdd 0 2024-03-30T04:28:21.930 INFO:tasks.workunit.client.0.smithi123.stdout:1/877: rename db/d1b/f2a to db/dd/d5f/d6d/d85/d89/d90/d94/f124 0 2024-03-30T04:28:21.932 INFO:tasks.workunit.client.1.smithi135.stdout:1/390: chown d16/d5a/d6b/d77/f7a 3798 1 2024-03-30T04:28:21.933 INFO:tasks.workunit.client.0.smithi123.stdout:8/853: dwrite da/d27/d56/fc8 [0,4194304] 0 2024-03-30T04:28:21.935 INFO:tasks.workunit.client.1.smithi135.stdout:0/559: mkdir d0/db5/db8 0 2024-03-30T04:28:21.935 INFO:tasks.workunit.client.0.smithi123.stdout:3/930: fdatasync d13/fa1 0 2024-03-30T04:28:21.936 INFO:tasks.workunit.client.1.smithi135.stdout:2/342: mknod d4/d32/d46/c71 0 2024-03-30T04:28:21.941 INFO:tasks.workunit.client.0.smithi123.stdout:1/878: mknod db/dd/d5f/dae/c125 0 2024-03-30T04:28:21.942 INFO:tasks.workunit.client.1.smithi135.stdout:0/560: fsync d0/d5/d9/fb 0 2024-03-30T04:28:21.943 INFO:tasks.workunit.client.1.smithi135.stdout:1/391: rename c13 to d16/d1a/c83 0 2024-03-30T04:28:21.943 INFO:tasks.workunit.client.1.smithi135.stdout:6/454: dwrite d2/dd/d14/d2b/d86/d55/f74 [0,4194304] 0 2024-03-30T04:28:21.944 INFO:tasks.workunit.client.0.smithi123.stdout:6/892: dread d10/d82/dc1/d45/f9c [0,4194304] 0 2024-03-30T04:28:21.945 INFO:tasks.workunit.client.0.smithi123.stdout:7/805: mknod d2/da/d11/d61/d106/c10f 0 2024-03-30T04:28:21.945 INFO:tasks.workunit.client.0.smithi123.stdout:2/987: fsync d0/d1f/d1c/d39/fd9 0 2024-03-30T04:28:21.948 INFO:tasks.workunit.client.1.smithi135.stdout:0/561: chown d0/d5/d9/l71 53059 1 2024-03-30T04:28:21.949 INFO:tasks.workunit.client.1.smithi135.stdout:2/343: symlink d4/d32/d12/d45/l72 0 2024-03-30T04:28:21.950 INFO:tasks.workunit.client.1.smithi135.stdout:1/392: dread - d16/d19/f72 zero size 2024-03-30T04:28:21.951 INFO:tasks.workunit.client.0.smithi123.stdout:8/854: mknod da/dc/da9/c10e 0 2024-03-30T04:28:21.952 INFO:tasks.workunit.client.0.smithi123.stdout:4/779: dwrite d2/d4/d7/f105 [0,4194304] 0 2024-03-30T04:28:21.956 INFO:tasks.workunit.client.1.smithi135.stdout:0/562: unlink d0/d5/d9/f6f 0 2024-03-30T04:28:21.957 INFO:tasks.workunit.client.1.smithi135.stdout:1/393: chown d16/d19 353573 1 2024-03-30T04:28:21.961 INFO:tasks.workunit.client.1.smithi135.stdout:5/416: sync 2024-03-30T04:28:21.961 INFO:tasks.workunit.client.1.smithi135.stdout:4/426: sync 2024-03-30T04:28:21.961 INFO:tasks.workunit.client.1.smithi135.stdout:7/460: dwrite d4/db/d1a/d5c/d5d/f8d [0,4194304] 0 2024-03-30T04:28:21.963 INFO:tasks.workunit.client.1.smithi135.stdout:9/377: sync 2024-03-30T04:28:21.965 INFO:tasks.workunit.client.0.smithi123.stdout:4/780: symlink d2/d4/d2c/d42/l10a 0 2024-03-30T04:28:21.969 INFO:tasks.workunit.client.1.smithi135.stdout:5/417: stat d1/d55/c60 0 2024-03-30T04:28:21.969 INFO:tasks.workunit.client.1.smithi135.stdout:4/427: chown d10/d1a/l1d 10754481 1 2024-03-30T04:28:21.970 INFO:tasks.workunit.client.1.smithi135.stdout:9/378: chown d7/d4f/d5c/f6d 46590019 1 2024-03-30T04:28:21.973 INFO:tasks.workunit.client.1.smithi135.stdout:1/394: creat d16/d19/f84 x:0 0 0 2024-03-30T04:28:21.973 INFO:tasks.workunit.client.0.smithi123.stdout:5/795: dwrite dd/d19/d1f/d35/d3a/d57/db3/fde [0,4194304] 0 2024-03-30T04:28:21.974 INFO:tasks.workunit.client.0.smithi123.stdout:3/931: symlink d13/d24/d31/d23/d3a/d3c/db8/d10b/l12d 0 2024-03-30T04:28:21.978 INFO:tasks.workunit.client.0.smithi123.stdout:2/988: mknod d0/d1f/d5e/d111/c13f 0 2024-03-30T04:28:21.980 INFO:tasks.workunit.client.1.smithi135.stdout:3/453: truncate d3/d50/d22/d74/f7c 876470 0 2024-03-30T04:28:21.982 INFO:tasks.workunit.client.1.smithi135.stdout:6/455: dwrite d2/f9 [0,4194304] 0 2024-03-30T04:28:21.983 INFO:tasks.workunit.client.0.smithi123.stdout:6/893: rename d10/d82/d8e/dc7/d5b/ffc to d10/d82/df9/f12d 0 2024-03-30T04:28:21.985 INFO:tasks.workunit.client.1.smithi135.stdout:1/395: truncate d16/d19/f56 145693 0 2024-03-30T04:28:21.985 INFO:tasks.workunit.client.1.smithi135.stdout:4/428: write d10/d1a/d42/f67 [882347,62625] 0 2024-03-30T04:28:21.987 INFO:tasks.workunit.client.0.smithi123.stdout:5/796: dread - dd/d19/d1f/d3c/d72/d8c/deb/f10a zero size 2024-03-30T04:28:21.992 INFO:tasks.workunit.client.1.smithi135.stdout:1/396: mkdir d16/d29/d32/d85 0 2024-03-30T04:28:21.995 INFO:tasks.workunit.client.0.smithi123.stdout:3/932: fdatasync d13/d24/d31/d23/d3a/f54 0 2024-03-30T04:28:21.995 INFO:tasks.workunit.client.1.smithi135.stdout:6/456: write d2/dd/f1a [4984646,76734] 0 2024-03-30T04:28:21.996 INFO:tasks.workunit.client.0.smithi123.stdout:6/894: fdatasync d10/d82/dc1/d6a/fb5 0 2024-03-30T04:28:22.001 INFO:tasks.workunit.client.1.smithi135.stdout:2/344: dwrite d4/d6/f61 [0,4194304] 0 2024-03-30T04:28:22.001 INFO:tasks.workunit.client.1.smithi135.stdout:4/429: creat d10/d1a/d3d/f91 x:0 0 0 2024-03-30T04:28:22.005 INFO:tasks.workunit.client.1.smithi135.stdout:6/457: dread - d2/dd/d14/d2b/d86/f80 zero size 2024-03-30T04:28:22.008 INFO:tasks.workunit.client.0.smithi123.stdout:2/989: getdents d0/d1f/d30/d119/de6 0 2024-03-30T04:28:22.009 INFO:tasks.workunit.client.1.smithi135.stdout:4/430: mkdir d10/d1a/d32/d40/d92 0 2024-03-30T04:28:22.009 INFO:tasks.workunit.client.0.smithi123.stdout:6/895: fdatasync d10/f59 0 2024-03-30T04:28:22.011 INFO:tasks.workunit.client.1.smithi135.stdout:8/404: dread d6/d8/f38 [0,4194304] 0 2024-03-30T04:28:22.015 INFO:tasks.workunit.client.1.smithi135.stdout:2/345: fsync d4/d32/d12/f31 0 2024-03-30T04:28:22.019 INFO:tasks.workunit.client.1.smithi135.stdout:0/563: dwrite d0/d5/d10/f2c [0,4194304] 0 2024-03-30T04:28:22.023 INFO:tasks.workunit.client.0.smithi123.stdout:6/896: dread - d10/d82/dd6/f127 zero size 2024-03-30T04:28:22.023 INFO:tasks.workunit.client.0.smithi123.stdout:0/839: truncate d6/dd/d18/d1f/f9e 3383697 0 2024-03-30T04:28:22.024 INFO:tasks.workunit.client.1.smithi135.stdout:6/458: chown d2/dd/c32 22 1 2024-03-30T04:28:22.027 INFO:tasks.workunit.client.1.smithi135.stdout:8/405: chown d6/d8/d14/c75 165 1 2024-03-30T04:28:22.027 INFO:tasks.workunit.client.1.smithi135.stdout:1/397: link d16/d29/d2c/d43/d50/f5b d16/d5a/d5f/d3d/f86 0 2024-03-30T04:28:22.031 INFO:tasks.workunit.client.1.smithi135.stdout:6/459: creat d2/d36/f8f x:0 0 0 2024-03-30T04:28:22.033 INFO:tasks.workunit.client.1.smithi135.stdout:5/418: dread d1/f36 [0,4194304] 0 2024-03-30T04:28:22.037 INFO:tasks.workunit.client.1.smithi135.stdout:7/461: dwrite d4/db/d1a/d5c/d5d/f7b [0,4194304] 0 2024-03-30T04:28:22.038 INFO:tasks.workunit.client.0.smithi123.stdout:7/806: dwrite d2/da/d11/d34/d76/d91/fcf [0,4194304] 0 2024-03-30T04:28:22.039 INFO:tasks.workunit.client.0.smithi123.stdout:3/933: truncate d13/d24/d31/d23/d3a/d3c/fa8 4884980 0 2024-03-30T04:28:22.041 INFO:tasks.workunit.client.1.smithi135.stdout:3/454: dwrite d3/db/fe [0,4194304] 0 2024-03-30T04:28:22.043 INFO:tasks.workunit.client.0.smithi123.stdout:5/797: dwrite dd/d19/d1f/d3c/fcf [0,4194304] 0 2024-03-30T04:28:22.043 INFO:tasks.workunit.client.0.smithi123.stdout:9/807: dread d6/d1f/d72/f93 [4194304,4194304] 0 2024-03-30T04:28:22.048 INFO:tasks.workunit.client.0.smithi123.stdout:8/855: dread da/dc/f13 [4194304,4194304] 0 2024-03-30T04:28:22.050 INFO:tasks.workunit.client.0.smithi123.stdout:3/934: creat d13/d24/d31/de0/f12e x:0 0 0 2024-03-30T04:28:22.053 INFO:tasks.workunit.client.0.smithi123.stdout:7/807: truncate f0 4158204 0 2024-03-30T04:28:22.057 INFO:tasks.workunit.client.1.smithi135.stdout:1/398: symlink d16/d29/d32/d85/l87 0 2024-03-30T04:28:22.063 INFO:tasks.workunit.client.0.smithi123.stdout:1/879: dread db/dd/d42/d46/d5d/d62/f6b [0,4194304] 0 2024-03-30T04:28:22.063 INFO:tasks.workunit.client.0.smithi123.stdout:9/808: creat d6/d1f/d92/db8/f119 x:0 0 0 2024-03-30T04:28:22.063 INFO:tasks.workunit.client.1.smithi135.stdout:7/462: write d4/db/d1a/d5c/d2c/f39 [3525495,38166] 0 2024-03-30T04:28:22.063 INFO:tasks.workunit.client.1.smithi135.stdout:0/564: getdents d0/d19/d2d/d79 0 2024-03-30T04:28:22.063 INFO:tasks.workunit.client.1.smithi135.stdout:4/431: dread d10/d1a/d42/f54 [0,4194304] 0 2024-03-30T04:28:22.064 INFO:tasks.workunit.client.1.smithi135.stdout:8/406: mknod d6/d8/d14/c77 0 2024-03-30T04:28:22.065 INFO:tasks.workunit.client.1.smithi135.stdout:2/346: rename d4/d6/d47/c52 to d4/d32/d2e/d2f/c73 0 2024-03-30T04:28:22.066 INFO:tasks.workunit.client.1.smithi135.stdout:0/565: chown d0/db5 29439345 1 2024-03-30T04:28:22.068 INFO:tasks.workunit.client.1.smithi135.stdout:1/399: stat d16/d1a/f21 0 2024-03-30T04:28:22.073 INFO:tasks.workunit.client.1.smithi135.stdout:4/432: dread - d10/d1a/d42/d66/f84 zero size 2024-03-30T04:28:22.073 INFO:tasks.workunit.client.0.smithi123.stdout:0/840: rename d6/dd/d18/c94 to d6/dd/d15/d28/dd3/ddd/c10b 0 2024-03-30T04:28:22.074 INFO:tasks.workunit.client.0.smithi123.stdout:8/856: truncate da/f4e 4047901 0 2024-03-30T04:28:22.074 INFO:tasks.workunit.client.0.smithi123.stdout:1/880: write db/dd/d5f/d6d/d85/d89/d90/d94/d9c/f102 [321400,111915] 0 2024-03-30T04:28:22.074 INFO:tasks.workunit.client.0.smithi123.stdout:6/897: dread d10/f59 [0,4194304] 0 2024-03-30T04:28:22.077 INFO:tasks.workunit.client.1.smithi135.stdout:8/407: unlink d6/l40 0 2024-03-30T04:28:22.079 INFO:tasks.workunit.client.1.smithi135.stdout:1/400: write d16/d5a/d5f/f7f [35845,72515] 0 2024-03-30T04:28:22.080 INFO:tasks.workunit.client.1.smithi135.stdout:2/347: symlink d4/d32/d12/l74 0 2024-03-30T04:28:22.080 INFO:tasks.workunit.client.1.smithi135.stdout:3/455: dread d3/f76 [0,4194304] 0 2024-03-30T04:28:22.081 INFO:tasks.workunit.client.0.smithi123.stdout:7/808: creat d2/da/f110 x:0 0 0 2024-03-30T04:28:22.081 INFO:tasks.workunit.client.0.smithi123.stdout:3/935: creat d13/d24/d31/d23/d3a/d3c/dab/db2/f12f x:0 0 0 2024-03-30T04:28:22.083 INFO:tasks.workunit.client.1.smithi135.stdout:4/433: chown c9 1625 1 2024-03-30T04:28:22.086 INFO:tasks.workunit.client.0.smithi123.stdout:9/809: rename d6/d1f/d92/db8/cbf to d6/d1f/d72/d7d/d10f/c11a 0 2024-03-30T04:28:22.086 INFO:tasks.workunit.client.0.smithi123.stdout:7/809: readlink d2/da/d11/d61/d55/d59/l89 0 2024-03-30T04:28:22.087 INFO:tasks.workunit.client.1.smithi135.stdout:6/460: dread d2/dd/d14/d2b/d86/d55/f74 [0,4194304] 0 2024-03-30T04:28:22.088 INFO:tasks.workunit.client.0.smithi123.stdout:5/798: getdents dd/d19/d1f/d35/d3a 0 2024-03-30T04:28:22.090 INFO:tasks.workunit.client.0.smithi123.stdout:3/936: chown d13/d24/d31/d23/d3a/d3c/d66/c11b 3 1 2024-03-30T04:28:22.091 INFO:tasks.workunit.client.0.smithi123.stdout:7/810: chown d2/da/d11/d34/c10d 367 1 2024-03-30T04:28:22.093 INFO:tasks.workunit.client.0.smithi123.stdout:1/881: fdatasync db/dd/d42/d46/d39/f52 0 2024-03-30T04:28:22.095 INFO:tasks.workunit.client.0.smithi123.stdout:2/990: dread d0/d1f/d1c/d39/d4b/fa7 [0,4194304] 0 2024-03-30T04:28:22.095 INFO:tasks.workunit.client.1.smithi135.stdout:7/463: rename d4/db/d1a/d5c/d2c/l71 to d4/db/d1a/d4b/d40/d3a/l9c 0 2024-03-30T04:28:22.100 INFO:tasks.workunit.client.1.smithi135.stdout:3/456: dread d3/f51 [0,4194304] 0 2024-03-30T04:28:22.100 INFO:tasks.workunit.client.1.smithi135.stdout:8/408: write d6/d8/f3b [460546,4414] 0 2024-03-30T04:28:22.101 INFO:tasks.workunit.client.0.smithi123.stdout:0/841: symlink d6/dd/d15/d98/d5d/d7f/df3/d104/l10c 0 2024-03-30T04:28:22.102 INFO:tasks.workunit.client.1.smithi135.stdout:0/566: truncate d0/d19/d2d/d45/f76 1205052 0 2024-03-30T04:28:22.102 INFO:tasks.workunit.client.1.smithi135.stdout:9/379: dread d7/f12 [0,4194304] 0 2024-03-30T04:28:22.103 INFO:tasks.workunit.client.1.smithi135.stdout:8/409: read - d6/d8/d15/f2b zero size 2024-03-30T04:28:22.104 INFO:tasks.workunit.client.0.smithi123.stdout:3/937: mknod d13/d24/d31/d123/d5d/c130 0 2024-03-30T04:28:22.105 INFO:tasks.workunit.client.0.smithi123.stdout:7/811: truncate d2/dc/d83/fef 498217 0 2024-03-30T04:28:22.107 INFO:tasks.workunit.client.0.smithi123.stdout:9/810: mknod d6/d1f/d72/c11b 0 2024-03-30T04:28:22.107 INFO:tasks.workunit.client.1.smithi135.stdout:0/567: fdatasync d0/f3 0 2024-03-30T04:28:22.108 INFO:tasks.workunit.client.1.smithi135.stdout:7/464: creat d4/db/d1a/d5c/d82/f9d x:0 0 0 2024-03-30T04:28:22.109 INFO:tasks.workunit.client.0.smithi123.stdout:3/938: chown d13/d24/d31/d23/f76 355 1 2024-03-30T04:28:22.110 INFO:tasks.workunit.client.0.smithi123.stdout:2/991: symlink d0/d37/d57/da1/l140 0 2024-03-30T04:28:22.114 INFO:tasks.workunit.client.0.smithi123.stdout:7/812: read d2/da/d11/d34/f103 [394111,71305] 0 2024-03-30T04:28:22.115 INFO:tasks.workunit.client.0.smithi123.stdout:8/857: dwrite da/d27/d3c/d64/d78/d83/ff8 [0,4194304] 0 2024-03-30T04:28:22.115 INFO:tasks.workunit.client.1.smithi135.stdout:3/457: symlink d3/d10/d9c/la0 0 2024-03-30T04:28:22.117 INFO:tasks.workunit.client.1.smithi135.stdout:5/419: dread d1/da/d17/f3d [0,4194304] 0 2024-03-30T04:28:22.117 INFO:tasks.workunit.client.1.smithi135.stdout:7/465: creat d4/db/d1a/d5c/d2c/d51/d56/f9e x:0 0 0 2024-03-30T04:28:22.119 INFO:tasks.workunit.client.1.smithi135.stdout:1/401: dwrite d16/d19/f72 [0,4194304] 0 2024-03-30T04:28:22.119 INFO:tasks.workunit.client.0.smithi123.stdout:7/813: chown d2/da/d11/d34/ded 1 1 2024-03-30T04:28:22.120 INFO:tasks.workunit.client.0.smithi123.stdout:8/858: stat da/d27/d56/d6c/d6d 0 2024-03-30T04:28:22.121 INFO:tasks.workunit.client.0.smithi123.stdout:9/811: truncate d6/dad/fd7 1513943 0 2024-03-30T04:28:22.123 INFO:tasks.workunit.client.0.smithi123.stdout:6/898: creat d10/d82/dc1/d6a/d7a/f12e x:0 0 0 2024-03-30T04:28:22.124 INFO:tasks.workunit.client.0.smithi123.stdout:7/814: write d2/d4d/f50 [3731857,70030] 0 2024-03-30T04:28:22.124 INFO:tasks.workunit.client.1.smithi135.stdout:8/410: write d6/d8/f5f [1223625,119743] 0 2024-03-30T04:28:22.125 INFO:tasks.workunit.client.0.smithi123.stdout:8/859: fdatasync da/d27/d95/fab 0 2024-03-30T04:28:22.129 INFO:tasks.workunit.client.1.smithi135.stdout:4/434: dwrite d10/d1a/d42/d46/f6b [0,4194304] 0 2024-03-30T04:28:22.131 INFO:tasks.workunit.client.1.smithi135.stdout:3/458: mkdir d3/d54/d5c/da1 0 2024-03-30T04:28:22.131 INFO:tasks.workunit.client.0.smithi123.stdout:6/899: truncate d10/d82/d8e/dc7/d5b/fa8 1518921 0 2024-03-30T04:28:22.132 INFO:tasks.workunit.client.0.smithi123.stdout:2/992: symlink d0/d1f/d27/l141 0 2024-03-30T04:28:22.135 INFO:tasks.workunit.client.1.smithi135.stdout:2/348: dwrite d4/d32/f13 [0,4194304] 0 2024-03-30T04:28:22.137 INFO:tasks.workunit.client.1.smithi135.stdout:5/420: creat d1/d45/d52/f7d x:0 0 0 2024-03-30T04:28:22.139 INFO:tasks.workunit.client.1.smithi135.stdout:0/568: dread d0/d5/d10/fa1 [0,4194304] 0 2024-03-30T04:28:22.140 INFO:tasks.workunit.client.1.smithi135.stdout:5/421: fdatasync d1/d3/d2b/d54/f71 0 2024-03-30T04:28:22.141 INFO:tasks.workunit.client.0.smithi123.stdout:1/882: rmdir db/dd/d42/d46/d56/d119 0 2024-03-30T04:28:22.142 INFO:tasks.workunit.client.1.smithi135.stdout:7/466: rmdir d4/db/d1a/d5c/d2c/d2e 39 2024-03-30T04:28:22.144 INFO:tasks.workunit.client.0.smithi123.stdout:9/812: rename d6/d1f/d72/d109 to d6/d1f/d72/d10d/d11c 0 2024-03-30T04:28:22.145 INFO:tasks.workunit.client.0.smithi123.stdout:4/781: sync 2024-03-30T04:28:22.146 INFO:tasks.workunit.client.1.smithi135.stdout:4/435: unlink d10/d1a/d42/d46/f6b 0 2024-03-30T04:28:22.149 INFO:tasks.workunit.client.0.smithi123.stdout:0/842: truncate d6/dd/d15/fc3 699639 0 2024-03-30T04:28:22.150 INFO:tasks.workunit.client.1.smithi135.stdout:8/411: rename d6/f6a to d6/d8/d15/d54/d4f/f78 0 2024-03-30T04:28:22.151 INFO:tasks.workunit.client.1.smithi135.stdout:9/380: write d7/d29/f2c [751303,82346] 0 2024-03-30T04:28:22.152 INFO:tasks.workunit.client.0.smithi123.stdout:5/799: link dd/d19/d1f/d3c/c88 dd/d19/d1f/d3c/d72/d8c/deb/c10c 0 2024-03-30T04:28:22.153 INFO:tasks.workunit.client.0.smithi123.stdout:0/843: chown d6/dd/d18/f109 0 1 2024-03-30T04:28:22.153 INFO:tasks.workunit.client.1.smithi135.stdout:3/459: dread d3/f45 [0,4194304] 0 2024-03-30T04:28:22.154 INFO:tasks.workunit.client.1.smithi135.stdout:2/349: unlink d4/d6/f35 0 2024-03-30T04:28:22.158 INFO:tasks.workunit.client.0.smithi123.stdout:3/939: mkdir d13/d24/d31/d23/d3a/d3c/db8/df4/d131 0 2024-03-30T04:28:22.159 INFO:tasks.workunit.client.0.smithi123.stdout:2/993: creat d0/d1f/d1c/d36/f142 x:0 0 0 2024-03-30T04:28:22.160 INFO:tasks.workunit.client.1.smithi135.stdout:8/412: write d6/d8/f5f [106891,25536] 0 2024-03-30T04:28:22.160 INFO:tasks.workunit.client.0.smithi123.stdout:0/844: write d6/dd/d15/d28/dd3/ddd/fec [264173,58244] 0 2024-03-30T04:28:22.162 INFO:tasks.workunit.client.1.smithi135.stdout:5/422: mkdir d1/d3/d76/d7e 0 2024-03-30T04:28:22.162 INFO:tasks.workunit.client.1.smithi135.stdout:7/467: mkdir d4/db/d9f 0 2024-03-30T04:28:22.164 INFO:tasks.workunit.client.1.smithi135.stdout:0/569: link d0/d5/d9/d23/fa4 d0/d5/d42/fb9 0 2024-03-30T04:28:22.165 INFO:tasks.workunit.client.0.smithi123.stdout:1/883: mknod db/dd/d5f/dae/c126 0 2024-03-30T04:28:22.165 INFO:tasks.workunit.client.0.smithi123.stdout:0/845: chown d6/dd/d15/d28/d4b/d8a/cee 11 1 2024-03-30T04:28:22.165 INFO:tasks.workunit.client.1.smithi135.stdout:4/436: creat d10/d72/d85/f93 x:0 0 0 2024-03-30T04:28:22.166 INFO:tasks.workunit.client.1.smithi135.stdout:3/460: rmdir d3/d10/d16/d41/d83 39 2024-03-30T04:28:22.167 INFO:tasks.workunit.client.0.smithi123.stdout:4/782: creat d2/d4/d2c/d42/d35/f10b x:0 0 0 2024-03-30T04:28:22.170 INFO:tasks.workunit.client.1.smithi135.stdout:3/461: stat d3/d10/d16/d23/f35 0 2024-03-30T04:28:22.170 INFO:tasks.workunit.client.1.smithi135.stdout:1/402: dwrite d16/f42 [0,4194304] 0 2024-03-30T04:28:22.170 INFO:tasks.workunit.client.0.smithi123.stdout:8/860: symlink da/d27/d56/l10f 0 2024-03-30T04:28:22.171 INFO:tasks.workunit.client.1.smithi135.stdout:0/570: dread - d0/d13/d53/d7f/d83/fac zero size 2024-03-30T04:28:22.171 INFO:tasks.workunit.client.1.smithi135.stdout:7/468: fdatasync d4/db/d1a/d4b/f45 0 2024-03-30T04:28:22.173 INFO:tasks.workunit.client.0.smithi123.stdout:5/800: truncate dd/f15 1260732 0 2024-03-30T04:28:22.176 INFO:tasks.workunit.client.1.smithi135.stdout:5/423: mkdir d1/d7/d1a/d1f/d7f 0 2024-03-30T04:28:22.177 INFO:tasks.workunit.client.0.smithi123.stdout:7/815: dread d2/da/d11/f19 [0,4194304] 0 2024-03-30T04:28:22.178 INFO:tasks.workunit.client.0.smithi123.stdout:6/900: rename d10/d20/f36 to d10/d20/f12f 0 2024-03-30T04:28:22.179 INFO:tasks.workunit.client.1.smithi135.stdout:7/469: fsync d4/db/d1a/d3d/f5b 0 2024-03-30T04:28:22.179 INFO:tasks.workunit.client.1.smithi135.stdout:5/424: fdatasync d1/d3/f38 0 2024-03-30T04:28:22.181 INFO:tasks.workunit.client.1.smithi135.stdout:2/350: creat d4/d32/f75 x:0 0 0 2024-03-30T04:28:22.182 INFO:tasks.workunit.client.1.smithi135.stdout:3/462: truncate d3/d50/f5b 491213 0 2024-03-30T04:28:22.184 INFO:tasks.workunit.client.1.smithi135.stdout:0/571: creat d0/d13/d53/d7f/fba x:0 0 0 2024-03-30T04:28:22.185 INFO:tasks.workunit.client.0.smithi123.stdout:9/813: unlink d6/d1f/d72/d7d/l9a 0 2024-03-30T04:28:22.189 INFO:tasks.workunit.client.0.smithi123.stdout:0/846: stat d6/dd/f51 0 2024-03-30T04:28:22.191 INFO:tasks.workunit.client.1.smithi135.stdout:1/403: dread d16/f45 [0,4194304] 0 2024-03-30T04:28:22.192 INFO:tasks.workunit.client.1.smithi135.stdout:5/425: mkdir d1/d3/d80 0 2024-03-30T04:28:22.193 INFO:tasks.workunit.client.0.smithi123.stdout:0/847: write d6/f9 [4719957,52054] 0 2024-03-30T04:28:22.193 INFO:tasks.workunit.client.1.smithi135.stdout:9/381: dwrite d7/f44 [0,4194304] 0 2024-03-30T04:28:22.195 INFO:tasks.workunit.client.0.smithi123.stdout:1/884: write db/dd/d5f/dae/ffd [775123,60673] 0 2024-03-30T04:28:22.196 INFO:tasks.workunit.client.1.smithi135.stdout:5/426: read d1/f6e [416549,342] 0 2024-03-30T04:28:22.197 INFO:tasks.workunit.client.0.smithi123.stdout:7/816: dread d2/da/d11/f58 [0,4194304] 0 2024-03-30T04:28:22.197 INFO:tasks.workunit.client.0.smithi123.stdout:4/783: mkdir d2/dfb/d10c 0 2024-03-30T04:28:22.198 INFO:tasks.workunit.client.1.smithi135.stdout:0/572: write d0/d13/f74 [769806,21841] 0 2024-03-30T04:28:22.202 INFO:tasks.workunit.client.1.smithi135.stdout:2/351: write d4/d32/d12/d45/f2c [1506370,75958] 0 2024-03-30T04:28:22.203 INFO:tasks.workunit.client.0.smithi123.stdout:8/861: mknod da/d27/d3c/d64/d78/c110 0 2024-03-30T04:28:22.204 INFO:tasks.workunit.client.0.smithi123.stdout:0/848: chown d6/dd/d15/d98/d4e/d9a/db2/feb 54282 1 2024-03-30T04:28:22.205 INFO:tasks.workunit.client.0.smithi123.stdout:5/801: unlink dd/d19/d1f/d35/ld4 0 2024-03-30T04:28:22.206 INFO:tasks.workunit.client.0.smithi123.stdout:4/784: chown d2/d4/d2c/d42/f3e 82 1 2024-03-30T04:28:22.206 INFO:tasks.workunit.client.1.smithi135.stdout:0/573: dread - d0/d13/d53/f80 zero size 2024-03-30T04:28:22.207 INFO:tasks.workunit.client.0.smithi123.stdout:6/901: dwrite d10/d82/dc1/d6a/d7a/d9f/fd4 [0,4194304] 0 2024-03-30T04:28:22.208 INFO:tasks.workunit.client.1.smithi135.stdout:1/404: mkdir d16/d29/d32/d88 0 2024-03-30T04:28:22.211 INFO:tasks.workunit.client.0.smithi123.stdout:0/849: chown d6/dd/d15/d28/d4b/dad/de8 0 1 2024-03-30T04:28:22.211 INFO:tasks.workunit.client.1.smithi135.stdout:3/463: dread d3/db/f19 [0,4194304] 0 2024-03-30T04:28:22.212 INFO:tasks.workunit.client.0.smithi123.stdout:2/994: creat d0/d1f/d1c/d10d/d11f/d123/f143 x:0 0 0 2024-03-30T04:28:22.212 INFO:tasks.workunit.client.1.smithi135.stdout:1/405: fdatasync d16/d19/f84 0 2024-03-30T04:28:22.213 INFO:tasks.workunit.client.1.smithi135.stdout:4/437: dwrite fa [0,4194304] 0 2024-03-30T04:28:22.213 INFO:tasks.workunit.client.1.smithi135.stdout:0/574: stat d0/d1/c7 0 2024-03-30T04:28:22.214 INFO:tasks.workunit.client.1.smithi135.stdout:5/427: mknod d1/d7/d11/c81 0 2024-03-30T04:28:22.215 INFO:tasks.workunit.client.0.smithi123.stdout:9/814: creat d6/d1f/d45/d4c/f11d x:0 0 0 2024-03-30T04:28:22.216 INFO:tasks.workunit.client.0.smithi123.stdout:1/885: truncate db/dd/d42/d46/d51/f8d 327208 0 2024-03-30T04:28:22.216 INFO:tasks.workunit.client.0.smithi123.stdout:0/850: chown d6/dd/d15/d28/d4b/d8a/ce6 10365336 1 2024-03-30T04:28:22.216 INFO:tasks.workunit.client.0.smithi123.stdout:3/940: creat d13/d24/d31/d23/d3a/d3c/db8/f132 x:0 0 0 2024-03-30T04:28:22.218 INFO:tasks.workunit.client.0.smithi123.stdout:2/995: chown d0/d1f/d27/d68/f11d 780945926 1 2024-03-30T04:28:22.220 INFO:tasks.workunit.client.0.smithi123.stdout:9/815: chown d6/d1f/d45/f50 125398799 1 2024-03-30T04:28:22.221 INFO:tasks.workunit.client.1.smithi135.stdout:0/575: fsync d0/d5/d9/fd 0 2024-03-30T04:28:22.221 INFO:tasks.workunit.client.0.smithi123.stdout:3/941: chown d13/d24/d31/d23/d3a/d3c/db8/d10b/f12a 16019584 1 2024-03-30T04:28:22.222 INFO:tasks.workunit.client.1.smithi135.stdout:7/470: rename d4/db/d1a/d5c/d2c/d51/d56/f89 to d4/db/fa0 0 2024-03-30T04:28:22.222 INFO:tasks.workunit.client.0.smithi123.stdout:1/886: write db/dd/ffe [3742301,126119] 0 2024-03-30T04:28:22.223 INFO:tasks.workunit.client.0.smithi123.stdout:4/785: creat d2/d4/f10d x:0 0 0 2024-03-30T04:28:22.227 INFO:tasks.workunit.client.1.smithi135.stdout:3/464: chown d3/db/f77 5772 1 2024-03-30T04:28:22.229 INFO:tasks.workunit.client.0.smithi123.stdout:6/902: mknod d10/d82/d8e/dc7/d5b/d103/d115/c130 0 2024-03-30T04:28:22.233 INFO:tasks.workunit.client.1.smithi135.stdout:4/438: truncate d10/d1a/f23 3489215 0 2024-03-30T04:28:22.234 INFO:tasks.workunit.client.0.smithi123.stdout:5/802: mknod dd/d19/d3e/d5b/d62/c10d 0 2024-03-30T04:28:22.234 INFO:tasks.workunit.client.1.smithi135.stdout:8/413: dread d6/d8/f17 [0,4194304] 0 2024-03-30T04:28:22.235 INFO:tasks.workunit.client.0.smithi123.stdout:6/903: chown d10/d82/dc1/d6a/d7a/de8/ff4 7294267 1 2024-03-30T04:28:22.236 INFO:tasks.workunit.client.1.smithi135.stdout:7/471: unlink d4/c20 0 2024-03-30T04:28:22.236 INFO:tasks.workunit.client.1.smithi135.stdout:9/382: link d7/d4f/d5c/d63/f64 d7/d1b/f80 0 2024-03-30T04:28:22.238 INFO:tasks.workunit.client.0.smithi123.stdout:7/817: rename d2/da/d11/d34/d76/d91/f9f to d2/da/d11/d61/d55/d59/f111 0 2024-03-30T04:28:22.238 INFO:tasks.workunit.client.1.smithi135.stdout:1/406: rmdir d16/d5a/d5f/d3a/d6f 39 2024-03-30T04:28:22.239 INFO:tasks.workunit.client.1.smithi135.stdout:4/439: fdatasync d10/f74 0 2024-03-30T04:28:22.242 INFO:tasks.workunit.client.1.smithi135.stdout:8/414: write d6/d8/d15/f2b [495000,13855] 0 2024-03-30T04:28:22.243 INFO:tasks.workunit.client.1.smithi135.stdout:7/472: chown d4/f54 1247 1 2024-03-30T04:28:22.243 INFO:tasks.workunit.client.1.smithi135.stdout:3/465: truncate d3/d50/f5b 1196818 0 2024-03-30T04:28:22.243 INFO:tasks.workunit.client.1.smithi135.stdout:1/407: stat d16/d5a/c6e 0 2024-03-30T04:28:22.247 INFO:tasks.workunit.client.1.smithi135.stdout:9/383: write d7/d1b/f3d [5071992,23939] 0 2024-03-30T04:28:22.249 INFO:tasks.workunit.client.0.smithi123.stdout:1/887: symlink db/dd/d42/d46/d5d/d62/l127 0 2024-03-30T04:28:22.249 INFO:tasks.workunit.client.0.smithi123.stdout:9/816: dwrite d6/d7c/fdd [0,4194304] 0 2024-03-30T04:28:22.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.251+0000 7f80b0600700 1 -- 172.21.15.123:0/329845434 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f80a8122890 con 0x7f80a806f9d0 2024-03-30T04:28:22.252 INFO:tasks.workunit.client.1.smithi135.stdout:2/352: dwrite d4/d32/d12/f3f [4194304,4194304] 0 2024-03-30T04:28:22.255 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:28:22.255 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:28:22.255 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 2 2024-03-30T04:28:22.255 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:28:22.255 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:28:22.255 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:28:22.255 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 1 2024-03-30T04:28:22.255 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:28:22.256 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:28:22.256 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 6 2024-03-30T04:28:22.256 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:28:22.256 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:28:22.256 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 4 2024-03-30T04:28:22.256 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:28:22.256 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:28:22.256 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 13, 2024-03-30T04:28:22.256 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 1 2024-03-30T04:28:22.256 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:28:22.256 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:28:22.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.253+0000 7f80a5600700 1 -- 172.21.15.123:0/329845434 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+850 (secure 0 0 0) 0x7f809c058870 con 0x7f80a806f9d0 2024-03-30T04:28:22.257 INFO:tasks.workunit.client.0.smithi123.stdout:8/862: mknod da/d27/d3c/d64/d78/db8/dfd/c111 0 2024-03-30T04:28:22.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.258+0000 7f80b0600700 1 -- 172.21.15.123:0/329845434 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f809006afa0 msgr2=0x7f809006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:28:22.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.258+0000 7f80b0600700 1 --2- 172.21.15.123:0/329845434 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f809006afa0 0x7f809006d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8094006fd0 tx=0x7f8094006d30 comp rx=0 tx=0).stop 2024-03-30T04:28:22.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.258+0000 7f80b0600700 1 -- 172.21.15.123:0/329845434 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80a806f9d0 msgr2=0x7f80a811cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:28:22.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.258+0000 7f80b0600700 1 --2- 172.21.15.123:0/329845434 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80a806f9d0 0x7f80a811cb40 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f809c00b3b0 tx=0x7f809c009e30 comp rx=0 tx=0).stop 2024-03-30T04:28:22.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.258+0000 7f80b0600700 1 -- 172.21.15.123:0/329845434 shutdown_connections 2024-03-30T04:28:22.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.258+0000 7f80b0600700 1 --2- 172.21.15.123:0/329845434 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7f809006afa0 0x7f809006d450 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:22.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.258+0000 7f80b0600700 1 --2- 172.21.15.123:0/329845434 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f80a806e9a0 0x7f80a811c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:22.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.258+0000 7f80b0600700 1 --2- 172.21.15.123:0/329845434 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80a806f9d0 0x7f80a811cb40 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:22.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.258+0000 7f80b0600700 1 -- 172.21.15.123:0/329845434 >> 172.21.15.123:0/329845434 conn(0x7f80a8069de0 msgr2=0x7f80a806b520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:28:22.259 INFO:tasks.workunit.client.1.smithi135.stdout:8/415: mkdir d6/d8/d15/d79 0 2024-03-30T04:28:22.259 INFO:tasks.workunit.client.1.smithi135.stdout:2/353: fdatasync d4/da/f44 0 2024-03-30T04:28:22.259 INFO:tasks.workunit.client.1.smithi135.stdout:9/384: rename d7/d55/f5d to d7/d4f/d5c/d63/f81 0 2024-03-30T04:28:22.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.261+0000 7f80b0600700 1 -- 172.21.15.123:0/329845434 shutdown_connections 2024-03-30T04:28:22.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.261+0000 7f80b0600700 1 -- 172.21.15.123:0/329845434 wait complete. 2024-03-30T04:28:22.260 INFO:tasks.workunit.client.1.smithi135.stdout:8/416: write d6/d23/f5c [664315,99743] 0 2024-03-30T04:28:22.264 INFO:tasks.workunit.client.1.smithi135.stdout:2/354: mkdir d4/d32/d12/d76 0 2024-03-30T04:28:22.264 INFO:tasks.workunit.client.0.smithi123.stdout:2/996: mkdir d0/d1f/d1c/d36/d106/d144 0 2024-03-30T04:28:22.265 INFO:tasks.workunit.client.0.smithi123.stdout:6/904: dwrite d10/d82/d8e/dc7/f109 [0,4194304] 0 2024-03-30T04:28:22.271 INFO:tasks.workunit.client.1.smithi135.stdout:4/440: rename d10/d1a/d42/f6e to d10/d1a/d32/d40/d92/f94 0 2024-03-30T04:28:22.273 INFO:tasks.workunit.client.0.smithi123.stdout:5/803: mknod dd/d19/d1f/d3c/d72/d8c/c10e 0 2024-03-30T04:28:22.273 INFO:tasks.workunit.client.1.smithi135.stdout:4/441: stat d10/d1a/l81 0 2024-03-30T04:28:22.279 INFO:tasks.workunit.client.1.smithi135.stdout:5/428: truncate d1/d3/d2b/d54/d5e/d5c/f4e 577878 0 2024-03-30T04:28:22.282 INFO:tasks.workunit.client.0.smithi123.stdout:1/888: truncate db/dd/f48 377870 0 2024-03-30T04:28:22.285 INFO:tasks.workunit.client.1.smithi135.stdout:0/576: dread d0/d5/d9/d23/d2a/fa0 [0,4194304] 0 2024-03-30T04:28:22.286 INFO:tasks.workunit.client.1.smithi135.stdout:4/442: creat d10/d1a/d3d/f95 x:0 0 0 2024-03-30T04:28:22.286 INFO:tasks.workunit.client.1.smithi135.stdout:8/417: dread d6/d8/d14/f30 [0,4194304] 0 2024-03-30T04:28:22.288 INFO:tasks.workunit.client.1.smithi135.stdout:7/473: truncate d4/db/de/f2b 917019 0 2024-03-30T04:28:22.290 INFO:tasks.workunit.client.1.smithi135.stdout:5/429: mknod d1/d3/d2b/c82 0 2024-03-30T04:28:22.292 INFO:tasks.workunit.client.0.smithi123.stdout:9/817: symlink d6/l11e 0 2024-03-30T04:28:22.292 INFO:tasks.workunit.client.1.smithi135.stdout:5/430: chown d1/d45 2688217 1 2024-03-30T04:28:22.297 INFO:tasks.workunit.client.0.smithi123.stdout:9/818: fdatasync d6/d1f/d92/db8/fbc 0 2024-03-30T04:28:22.297 INFO:tasks.workunit.client.1.smithi135.stdout:9/385: dwrite d7/d1b/f28 [0,4194304] 0 2024-03-30T04:28:22.298 INFO:tasks.workunit.client.1.smithi135.stdout:5/431: creat d1/d45/f83 x:0 0 0 2024-03-30T04:28:22.298 INFO:tasks.workunit.client.0.smithi123.stdout:6/905: link d10/d20/da2/fc3 d10/d82/d8e/f131 0 2024-03-30T04:28:22.300 INFO:tasks.workunit.client.1.smithi135.stdout:8/418: dread d6/fa [4194304,4194304] 0 2024-03-30T04:28:22.300 INFO:tasks.workunit.client.0.smithi123.stdout:1/889: write db/dd/d42/d46/f2e [3843261,94134] 0 2024-03-30T04:28:22.303 INFO:tasks.workunit.client.0.smithi123.stdout:7/818: link d2/da/d11/d61/d55/d59/c5e d2/da/d11/d61/d106/c112 0 2024-03-30T04:28:22.305 INFO:tasks.workunit.client.0.smithi123.stdout:9/819: chown l5 51 1 2024-03-30T04:28:22.307 INFO:tasks.workunit.client.0.smithi123.stdout:4/786: getdents d2/d34 0 2024-03-30T04:28:22.307 INFO:tasks.workunit.client.0.smithi123.stdout:2/997: rmdir d0/d1f/d1c/d36/d106/d144 0 2024-03-30T04:28:22.307 INFO:tasks.workunit.client.1.smithi135.stdout:5/432: creat d1/da/f84 x:0 0 0 2024-03-30T04:28:22.310 INFO:tasks.workunit.client.1.smithi135.stdout:2/355: dwrite d4/d6/f61 [0,4194304] 0 2024-03-30T04:28:22.312 INFO:tasks.workunit.client.0.smithi123.stdout:8/863: dwrite da/d27/f53 [0,4194304] 0 2024-03-30T04:28:22.315 INFO:tasks.workunit.client.1.smithi135.stdout:7/474: dwrite d4/db/d1a/d3d/f5b [0,4194304] 0 2024-03-30T04:28:22.315 INFO:tasks.workunit.client.1.smithi135.stdout:1/408: dread d16/d1a/f23 [0,4194304] 0 2024-03-30T04:28:22.318 INFO:tasks.workunit.client.1.smithi135.stdout:8/419: getdents d6 0 2024-03-30T04:28:22.319 INFO:tasks.workunit.client.0.smithi123.stdout:0/851: dwrite d6/dd/d15/d28/f97 [0,4194304] 0 2024-03-30T04:28:22.321 INFO:tasks.workunit.client.1.smithi135.stdout:1/409: mknod d16/d29/d2c/d43/c89 0 2024-03-30T04:28:22.324 INFO:tasks.workunit.client.1.smithi135.stdout:2/356: rename d4/d32/d50 to d4/d55/d77 0 2024-03-30T04:28:22.330 INFO:tasks.workunit.client.1.smithi135.stdout:1/410: dread - d16/d19/d20/f5e zero size 2024-03-30T04:28:22.331 INFO:tasks.workunit.client.1.smithi135.stdout:8/420: rmdir d6/d8/d15/d54/d69 39 2024-03-30T04:28:22.332 INFO:tasks.workunit.client.1.smithi135.stdout:2/357: stat d4/da/f59 0 2024-03-30T04:28:22.332 INFO:tasks.workunit.client.0.smithi123.stdout:5/804: dread dd/d19/d5f/d99/fe5 [4194304,4194304] 0 2024-03-30T04:28:22.334 INFO:tasks.workunit.client.1.smithi135.stdout:9/386: dwrite d7/d29/f51 [0,4194304] 0 2024-03-30T04:28:22.335 INFO:tasks.workunit.client.1.smithi135.stdout:7/475: rmdir d4/db/d1a/d94 0 2024-03-30T04:28:22.336 INFO:tasks.workunit.client.1.smithi135.stdout:2/358: chown d4/d32/d12/f31 48 1 2024-03-30T04:28:22.338 INFO:tasks.workunit.client.1.smithi135.stdout:7/476: fsync d4/db/f26 0 2024-03-30T04:28:22.338 INFO:tasks.workunit.client.1.smithi135.stdout:8/421: dread d6/d8/d15/d54/f5d [0,4194304] 0 2024-03-30T04:28:22.339 INFO:tasks.workunit.client.0.smithi123.stdout:9/820: dwrite d6/f7e [0,4194304] 0 2024-03-30T04:28:22.340 INFO:tasks.workunit.client.1.smithi135.stdout:9/387: chown d7/d1b/d37/l4d 12149260 1 2024-03-30T04:28:22.340 INFO:tasks.workunit.client.1.smithi135.stdout:1/411: write d16/d5a/d6b/d77/f7a [1701249,81555] 0 2024-03-30T04:28:22.343 INFO:tasks.workunit.client.0.smithi123.stdout:9/821: write d6/d1f/d4d/f5b [2491348,104676] 0 2024-03-30T04:28:22.345 INFO:tasks.workunit.client.1.smithi135.stdout:9/388: fsync d7/d55/f6c 0 2024-03-30T04:28:22.346 INFO:tasks.workunit.client.1.smithi135.stdout:8/422: dread - d6/d8/d14/f67 zero size 2024-03-30T04:28:22.349 INFO:tasks.workunit.client.1.smithi135.stdout:5/433: dwrite d1/d45/d52/f7d [0,4194304] 0 2024-03-30T04:28:22.352 INFO:tasks.workunit.client.1.smithi135.stdout:2/359: creat d4/d32/f78 x:0 0 0 2024-03-30T04:28:22.353 INFO:tasks.workunit.client.1.smithi135.stdout:1/412: mknod d16/d5a/d5f/d3a/c8a 0 2024-03-30T04:28:22.355 INFO:tasks.workunit.client.0.smithi123.stdout:3/942: sync 2024-03-30T04:28:22.356 INFO:tasks.workunit.client.0.smithi123.stdout:5/805: dwrite dd/d19/d5f/d99/f106 [0,4194304] 0 2024-03-30T04:28:22.358 INFO:tasks.workunit.client.1.smithi135.stdout:5/434: mknod d1/d7/d1a/d1f/d7f/c85 0 2024-03-30T04:28:22.358 INFO:tasks.workunit.client.1.smithi135.stdout:2/360: write d4/d32/d12/d45/f2c [4571864,52625] 0 2024-03-30T04:28:22.360 INFO:tasks.workunit.client.0.smithi123.stdout:1/890: mknod db/dd/d5f/d6d/d7e/c128 0 2024-03-30T04:28:22.360 INFO:tasks.workunit.client.1.smithi135.stdout:1/413: unlink d16/d1a/f23 0 2024-03-30T04:28:22.361 INFO:tasks.workunit.client.1.smithi135.stdout:2/361: stat d4/da/f21 0 2024-03-30T04:28:22.362 INFO:tasks.workunit.client.1.smithi135.stdout:8/423: link d6/d8/d15/d2a/l74 d6/d23/l7a 0 2024-03-30T04:28:22.365 INFO:tasks.workunit.client.0.smithi123.stdout:2/998: creat d0/d1f/d82/ddf/f145 x:0 0 0 2024-03-30T04:28:22.366 INFO:tasks.workunit.client.1.smithi135.stdout:2/362: chown d4/d6/f1a 0 1 2024-03-30T04:28:22.366 INFO:tasks.workunit.client.1.smithi135.stdout:1/414: creat d16/d19/d20/f8b x:0 0 0 2024-03-30T04:28:22.367 INFO:tasks.workunit.client.1.smithi135.stdout:8/424: write d6/d8/f71 [215423,16026] 0 2024-03-30T04:28:22.369 INFO:tasks.workunit.client.0.smithi123.stdout:4/787: rename d2/d4/d7/f105 to d2/d4/d2c/d42/d35/d5d/d78/d104/f10e 0 2024-03-30T04:28:22.370 INFO:tasks.workunit.client.0.smithi123.stdout:2/999: stat d0/d1f/d1c/d39/d4b/fff 0 2024-03-30T04:28:22.371 INFO:tasks.workunit.client.1.smithi135.stdout:1/415: fdatasync d16/d19/f72 0 2024-03-30T04:28:22.373 INFO:tasks.workunit.client.0.smithi123.stdout:9/822: rmdir d6/d1f/d45 39 2024-03-30T04:28:22.373 INFO:tasks.workunit.client.1.smithi135.stdout:2/363: symlink d4/d32/d2e/d67/l79 0 2024-03-30T04:28:22.373 INFO:tasks.workunit.client.1.smithi135.stdout:8/425: stat d6/d8/d15/d2a/c49 0 2024-03-30T04:28:22.375 INFO:tasks.workunit.client.1.smithi135.stdout:2/364: chown d4/d6/l62 3 1 2024-03-30T04:28:22.376 INFO:tasks.workunit.client.0.smithi123.stdout:5/806: unlink dd/d19/d1f/f27 0 2024-03-30T04:28:22.377 INFO:tasks.workunit.client.1.smithi135.stdout:8/426: mkdir d6/d23/d7b 0 2024-03-30T04:28:22.380 INFO:tasks.workunit.client.1.smithi135.stdout:2/365: mkdir d4/da/d7a 0 2024-03-30T04:28:22.381 INFO:tasks.workunit.client.1.smithi135.stdout:8/427: truncate d6/d8/f39 838344 0 2024-03-30T04:28:22.386 INFO:tasks.workunit.client.0.smithi123.stdout:4/788: fdatasync d2/d4/d2c/d42/d35/d5d/d78/d104/f10e 0 2024-03-30T04:28:22.388 INFO:tasks.workunit.client.1.smithi135.stdout:1/416: dread d16/d5a/d5f/d3a/d6f/f7d [0,4194304] 0 2024-03-30T04:28:22.389 INFO:tasks.workunit.client.1.smithi135.stdout:9/389: dread d7/f77 [0,4194304] 0 2024-03-30T04:28:22.390 INFO:tasks.workunit.client.0.smithi123.stdout:5/807: creat dd/d19/d1f/d3c/d9e/f10f x:0 0 0 2024-03-30T04:28:22.390 INFO:tasks.workunit.client.1.smithi135.stdout:9/390: chown d7/d29/d78 254 1 2024-03-30T04:28:22.390 INFO:tasks.workunit.client.0.smithi123.stdout:1/891: getdents db/dd/d5f/d6d/d85 0 2024-03-30T04:28:22.392 INFO:tasks.workunit.client.1.smithi135.stdout:1/417: dread d16/d29/d2c/d43/d50/f68 [0,4194304] 0 2024-03-30T04:28:22.396 INFO:tasks.workunit.client.1.smithi135.stdout:9/391: rename d7/f77 to d7/d29/f82 0 2024-03-30T04:28:22.397 INFO:tasks.workunit.client.1.smithi135.stdout:2/366: dwrite d4/d32/d12/d45/f2c [0,4194304] 0 2024-03-30T04:28:22.397 INFO:tasks.workunit.client.0.smithi123.stdout:8/864: dwrite da/d27/d3c/f5e [0,4194304] 0 2024-03-30T04:28:22.398 INFO:tasks.workunit.client.0.smithi123.stdout:1/892: stat db/dd/d5f/dae 0 2024-03-30T04:28:22.399 INFO:tasks.workunit.client.0.smithi123.stdout:5/808: chown dd/d19/d1f/f107 5146717 1 2024-03-30T04:28:22.399 INFO:tasks.workunit.client.1.smithi135.stdout:1/418: getdents d16/d5a/d6b/d77 0 2024-03-30T04:28:22.399 INFO:tasks.workunit.client.1.smithi135.stdout:2/367: truncate d4/d55/f5f 992194 0 2024-03-30T04:28:22.403 INFO:tasks.workunit.client.1.smithi135.stdout:1/419: symlink d16/d5a/d5f/d3d/l8c 0 2024-03-30T04:28:22.403 INFO:tasks.workunit.client.1.smithi135.stdout:2/368: mknod d4/c7b 0 2024-03-30T04:28:22.406 INFO:tasks.workunit.client.0.smithi123.stdout:6/906: write d10/f1d [3819315,75127] 0 2024-03-30T04:28:22.407 INFO:tasks.workunit.client.1.smithi135.stdout:2/369: fdatasync d4/d32/d2e/f40 0 2024-03-30T04:28:22.412 INFO:tasks.workunit.client.1.smithi135.stdout:1/420: rename d16/d29/d32/d88 to d16/d5a/d5f/d3d/d8d 0 2024-03-30T04:28:22.414 INFO:tasks.workunit.client.0.smithi123.stdout:0/852: dwrite d6/dd/d15/fde [0,4194304] 0 2024-03-30T04:28:22.414 INFO:tasks.workunit.client.0.smithi123.stdout:3/943: write d13/d24/d31/d123/d5d/da5/fdb [1873645,103070] 0 2024-03-30T04:28:22.414 INFO:tasks.workunit.client.1.smithi135.stdout:1/421: dread - d16/d19/d20/f8b zero size 2024-03-30T04:28:22.415 INFO:tasks.workunit.client.1.smithi135.stdout:8/428: dread d6/f25 [0,4194304] 0 2024-03-30T04:28:22.416 INFO:tasks.workunit.client.1.smithi135.stdout:2/370: getdents d4/d32/d12/d45/d3c 0 2024-03-30T04:28:22.416 INFO:tasks.workunit.client.1.smithi135.stdout:9/392: dwrite d7/f26 [4194304,4194304] 0 2024-03-30T04:28:22.417 INFO:tasks.workunit.client.0.smithi123.stdout:9/823: dwrite d6/d1f/d92/db8/f119 [0,4194304] 0 2024-03-30T04:28:22.419 INFO:tasks.workunit.client.0.smithi123.stdout:8/865: write da/f92 [406806,49458] 0 2024-03-30T04:28:22.422 INFO:tasks.workunit.client.0.smithi123.stdout:4/789: write d2/d4/d2c/d42/d35/d9b/fb6 [1025921,58073] 0 2024-03-30T04:28:22.424 INFO:tasks.workunit.client.1.smithi135.stdout:8/429: symlink d6/d8/d15/d54/d4f/l7c 0 2024-03-30T04:28:22.424 INFO:tasks.workunit.client.1.smithi135.stdout:9/393: mknod d7/d4f/d6e/c83 0 2024-03-30T04:28:22.424 INFO:tasks.workunit.client.1.smithi135.stdout:8/430: write d6/d8/d15/d1f/f4c [5217433,105634] 0 2024-03-30T04:28:22.424 INFO:tasks.workunit.client.0.smithi123.stdout:1/893: read db/dd/d42/d46/d51/d72/fa6 [2708045,95461] 0 2024-03-30T04:28:22.424 INFO:tasks.workunit.client.0.smithi123.stdout:5/809: mkdir dd/d19/d1f/d35/d3a/d57/db5/d110 0 2024-03-30T04:28:22.424 INFO:tasks.workunit.client.0.smithi123.stdout:3/944: write f11 [4889556,44159] 0 2024-03-30T04:28:22.425 INFO:tasks.workunit.client.1.smithi135.stdout:2/371: unlink d4/d32/d12/d45/l72 0 2024-03-30T04:28:22.425 INFO:tasks.workunit.client.1.smithi135.stdout:8/431: dread - d6/f6e zero size 2024-03-30T04:28:22.427 INFO:tasks.workunit.client.0.smithi123.stdout:7/819: dwrite d2/f13 [0,4194304] 0 2024-03-30T04:28:22.427 INFO:tasks.workunit.client.1.smithi135.stdout:9/394: creat d7/d4f/d5c/f84 x:0 0 0 2024-03-30T04:28:22.427 INFO:tasks.workunit.client.0.smithi123.stdout:9/824: creat d6/d58/f11f x:0 0 0 2024-03-30T04:28:22.433 INFO:tasks.workunit.client.1.smithi135.stdout:8/432: fsync d6/f6e 0 2024-03-30T04:28:22.434 INFO:tasks.workunit.client.0.smithi123.stdout:0/853: creat d6/dd/d15/d98/dd0/f10d x:0 0 0 2024-03-30T04:28:22.438 INFO:tasks.workunit.client.0.smithi123.stdout:6/907: truncate fe 1863844 0 2024-03-30T04:28:22.439 INFO:tasks.workunit.client.0.smithi123.stdout:9/825: mkdir d6/d1f/d92/d120 0 2024-03-30T04:28:22.439 INFO:tasks.workunit.client.0.smithi123.stdout:3/945: write d13/d24/d31/df7/f12b [619733,47894] 0 2024-03-30T04:28:22.440 INFO:tasks.workunit.client.1.smithi135.stdout:2/372: creat d4/da/d7a/f7c x:0 0 0 2024-03-30T04:28:22.440 INFO:tasks.workunit.client.1.smithi135.stdout:8/433: chown d6/d8/d14/d32 0 1 2024-03-30T04:28:22.442 INFO:tasks.workunit.client.0.smithi123.stdout:7/820: fdatasync d2/da/d11/fb1 0 2024-03-30T04:28:22.446 INFO:tasks.workunit.client.0.smithi123.stdout:0/854: creat d6/d52/dcf/f10e x:0 0 0 2024-03-30T04:28:22.447 INFO:tasks.workunit.client.1.smithi135.stdout:8/434: truncate d6/d8/d15/d1f/f44 395419 0 2024-03-30T04:28:22.448 INFO:tasks.workunit.client.0.smithi123.stdout:8/866: truncate da/d27/d95/fab 2552453 0 2024-03-30T04:28:22.450 INFO:tasks.workunit.client.1.smithi135.stdout:2/373: mkdir d4/d32/d12/d45/d7d 0 2024-03-30T04:28:22.455 INFO:tasks.workunit.client.0.smithi123.stdout:8/867: symlink da/d27/d95/l112 0 2024-03-30T04:28:22.459 INFO:tasks.workunit.client.0.smithi123.stdout:7/821: rename d2/da/d11/d34/d76/d91 to d2/da/d11/d61/d55/d9e/d113 0 2024-03-30T04:28:22.464 INFO:tasks.workunit.client.1.smithi135.stdout:9/395: dwrite d7/f22 [4194304,4194304] 0 2024-03-30T04:28:22.472 INFO:tasks.workunit.client.0.smithi123.stdout:0/855: rename d6/d52/de0 to d6/dd/d15/d28/d4b/dad/de8/d10f 0 2024-03-30T04:28:22.473 INFO:tasks.workunit.client.0.smithi123.stdout:8/868: creat da/d27/d56/dde/f113 x:0 0 0 2024-03-30T04:28:22.478 INFO:tasks.workunit.client.0.smithi123.stdout:9/826: rename d6/d49/d113/dfb to d6/d1f/d45/d4c/d121 0 2024-03-30T04:28:22.480 INFO:tasks.workunit.client.0.smithi123.stdout:9/827: dread - d6/d1f/d72/d7d/fee zero size 2024-03-30T04:28:22.484 INFO:tasks.workunit.client.0.smithi123.stdout:4/790: write d2/d4/d2c/d42/d35/d5d/dc0/fce [11154,10736] 0 2024-03-30T04:28:22.491 INFO:tasks.workunit.client.0.smithi123.stdout:1/894: truncate db/dd/d42/d46/d39/fad 1772125 0 2024-03-30T04:28:22.496 INFO:tasks.workunit.client.1.smithi135.stdout:7/477: truncate d4/db/de/f2b 1537168 0 2024-03-30T04:28:22.499 INFO:tasks.workunit.client.0.smithi123.stdout:9/828: mkdir d6/d1f/d122 0 2024-03-30T04:28:22.499 INFO:tasks.workunit.client.0.smithi123.stdout:4/791: mkdir d2/d4/d7/d63/db5/d10f 0 2024-03-30T04:28:22.500 INFO:tasks.workunit.client.1.smithi135.stdout:7/478: creat d4/db/d1a/d4b/d40/d3a/fa1 x:0 0 0 2024-03-30T04:28:22.502 INFO:tasks.workunit.client.1.smithi135.stdout:7/479: fdatasync d4/db/d1a/d5c/d59/d76/f91 0 2024-03-30T04:28:22.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.504+0000 7fad5d600700 1 -- 172.21.15.123:0/1485219832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad580feff0 msgr2=0x7fad580ff400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:28:22.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.504+0000 7fad5d600700 1 --2- 172.21.15.123:0/1485219832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad580feff0 0x7fad580ff400 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fad48004940 tx=0x7fad48009e30 comp rx=0 tx=0).stop 2024-03-30T04:28:22.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.505+0000 7fad5d600700 1 -- 172.21.15.123:0/1485219832 shutdown_connections 2024-03-30T04:28:22.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.505+0000 7fad5d600700 1 --2- 172.21.15.123:0/1485219832 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fad581002a0 0x7fad58100710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:22.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.505+0000 7fad5d600700 1 --2- 172.21.15.123:0/1485219832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad580feff0 0x7fad580ff400 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:22.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.505+0000 7fad5d600700 1 -- 172.21.15.123:0/1485219832 >> 172.21.15.123:0/1485219832 conn(0x7fad580fa950 msgr2=0x7fad580fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:28:22.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.506+0000 7fad5d600700 1 -- 172.21.15.123:0/1485219832 shutdown_connections 2024-03-30T04:28:22.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.506+0000 7fad5d600700 1 -- 172.21.15.123:0/1485219832 wait complete. 2024-03-30T04:28:22.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.507+0000 7fad5d600700 1 Processor -- start 2024-03-30T04:28:22.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.508+0000 7fad5d600700 1 -- start start 2024-03-30T04:28:22.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.508+0000 7fad5d600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad580feff0 0x7fad58113f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:28:22.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.508+0000 7fad5d600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fad581002a0 0x7fad58114450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:28:22.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.509+0000 7fad5d600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad58114a50 con 0x7fad580feff0 2024-03-30T04:28:22.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.509+0000 7fad5d600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad58115250 con 0x7fad581002a0 2024-03-30T04:28:22.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.509+0000 7fad56a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad580feff0 0x7fad58113f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:28:22.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.509+0000 7fad56a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad580feff0 0x7fad58113f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58344/0 (socket says 172.21.15.123:58344) 2024-03-30T04:28:22.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.509+0000 7fad56a00700 1 -- 172.21.15.123:0/4260535560 learned_addr learned my addr 172.21.15.123:0/4260535560 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:28:22.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.510+0000 7fad56000700 1 --2- 172.21.15.123:0/4260535560 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fad581002a0 0x7fad58114450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:28:22.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.510+0000 7fad56a00700 1 -- 172.21.15.123:0/4260535560 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fad581002a0 msgr2=0x7fad58114450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:28:22.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.510+0000 7fad56a00700 1 --2- 172.21.15.123:0/4260535560 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fad581002a0 0x7fad58114450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:22.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.510+0000 7fad56a00700 1 -- 172.21.15.123:0/4260535560 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad48005040 con 0x7fad580feff0 2024-03-30T04:28:22.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.511+0000 7fad56a00700 1 --2- 172.21.15.123:0/4260535560 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad580feff0 0x7fad58113f10 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fad4800b3b0 tx=0x7fad48009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:28:22.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.511+0000 7fad4ea00700 1 -- 172.21.15.123:0/4260535560 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad4800e030 con 0x7fad580feff0 2024-03-30T04:28:22.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.511+0000 7fad5d600700 1 -- 172.21.15.123:0/4260535560 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad581154d0 con 0x7fad580feff0 2024-03-30T04:28:22.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.512+0000 7fad4ea00700 1 -- 172.21.15.123:0/4260535560 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fad48013420 con 0x7fad580feff0 2024-03-30T04:28:22.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.512+0000 7fad4ea00700 1 -- 172.21.15.123:0/4260535560 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad4801c640 con 0x7fad580feff0 2024-03-30T04:28:22.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.512+0000 7fad5d600700 1 -- 172.21.15.123:0/4260535560 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad58115950 con 0x7fad580feff0 2024-03-30T04:28:22.515 INFO:tasks.workunit.client.0.smithi123.stdout:4/792: truncate d2/f45 777151 0 2024-03-30T04:28:22.515 INFO:tasks.workunit.client.0.smithi123.stdout:9/829: creat d6/d49/d113/f123 x:0 0 0 2024-03-30T04:28:22.517 INFO:tasks.workunit.client.0.smithi123.stdout:9/830: fdatasync d6/d49/f65 0 2024-03-30T04:28:22.517 INFO:tasks.workunit.client.1.smithi135.stdout:9/396: truncate d7/d4f/d6e/f76 255481 0 2024-03-30T04:28:22.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.519+0000 7fad5d600700 1 -- 172.21.15.123:0/4260535560 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad58104340 con 0x7fad580feff0 2024-03-30T04:28:22.519 INFO:tasks.workunit.client.1.smithi135.stdout:7/480: dread d4/db/d1a/d5c/d2c/d51/f7e [0,4194304] 0 2024-03-30T04:28:22.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.519+0000 7fad4ea00700 1 -- 172.21.15.123:0/4260535560 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) v1 ==== 88766+0+0 (secure 0 0 0) 0x7fad48013590 con 0x7fad580feff0 2024-03-30T04:28:22.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.522+0000 7fad4ea00700 1 --2- 172.21.15.123:0/4260535560 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7fad3c06afe0 0x7fad3c06d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:28:22.521 INFO:tasks.workunit.client.1.smithi135.stdout:9/397: creat d7/d41/f85 x:0 0 0 2024-03-30T04:28:22.521 INFO:tasks.workunit.client.0.smithi123.stdout:6/908: dwrite d10/d20/da2/fe4 [0,4194304] 0 2024-03-30T04:28:22.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.522+0000 7fad56000700 1 --2- 172.21.15.123:0/4260535560 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7fad3c06afe0 0x7fad3c06d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:28:22.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.523+0000 7fad4ea00700 1 -- 172.21.15.123:0/4260535560 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7fad4808b300 con 0x7fad580feff0 2024-03-30T04:28:22.522 INFO:tasks.workunit.client.0.smithi123.stdout:1/895: creat db/dd/d42/d46/d51/d72/f129 x:0 0 0 2024-03-30T04:28:22.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.524+0000 7fad56000700 1 --2- 172.21.15.123:0/4260535560 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7fad3c06afe0 0x7fad3c06d490 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fad44009fd0 tx=0x7fad44009e50 comp rx=0 tx=0).ready entity=mgr.14608 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:28:22.523 INFO:tasks.workunit.client.0.smithi123.stdout:9/831: stat d6/dad/lff 0 2024-03-30T04:28:22.528 INFO:tasks.workunit.client.0.smithi123.stdout:3/946: dwrite d13/d24/d31/d23/d63/f71 [4194304,4194304] 0 2024-03-30T04:28:22.528 INFO:tasks.workunit.client.0.smithi123.stdout:4/793: rmdir d2/d34/dd2 39 2024-03-30T04:28:22.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.531+0000 7fad4ea00700 1 -- 172.21.15.123:0/4260535560 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fad480591e0 con 0x7fad580feff0 2024-03-30T04:28:22.535 INFO:tasks.workunit.client.0.smithi123.stdout:9/832: mknod d6/d1f/d72/d10d/c124 0 2024-03-30T04:28:22.541 INFO:tasks.workunit.client.0.smithi123.stdout:8/869: truncate da/d27/d56/f65 112032 0 2024-03-30T04:28:22.542 INFO:tasks.workunit.client.1.smithi135.stdout:9/398: dread d7/f44 [0,4194304] 0 2024-03-30T04:28:22.546 INFO:tasks.workunit.client.1.smithi135.stdout:8/435: truncate d6/d8/f17 272076 0 2024-03-30T04:28:22.547 INFO:tasks.workunit.client.0.smithi123.stdout:8/870: dread - da/d27/d56/d6c/d96/fdc zero size 2024-03-30T04:28:22.552 INFO:tasks.workunit.client.0.smithi123.stdout:7/822: dwrite d2/da/d11/d61/d55/d9e/fad [0,4194304] 0 2024-03-30T04:28:22.552 INFO:tasks.workunit.client.1.smithi135.stdout:1/422: dwrite d16/d5a/d6b/f7b [4194304,4194304] 0 2024-03-30T04:28:22.554 INFO:tasks.workunit.client.0.smithi123.stdout:5/810: sync 2024-03-30T04:28:22.555 INFO:tasks.workunit.client.0.smithi123.stdout:6/909: mknod d10/d82/dc1/d6a/d7a/d99/c132 0 2024-03-30T04:28:22.558 INFO:tasks.workunit.client.0.smithi123.stdout:3/947: mkdir d13/d24/d31/d123/d5d/da5/d133 0 2024-03-30T04:28:22.558 INFO:tasks.workunit.client.1.smithi135.stdout:9/399: mkdir d7/d4f/d5c/d63/d86 0 2024-03-30T04:28:22.559 INFO:tasks.workunit.client.0.smithi123.stdout:0/856: dwrite d6/dd/d18/d2f/fac [0,4194304] 0 2024-03-30T04:28:22.561 INFO:tasks.workunit.client.1.smithi135.stdout:7/481: truncate d4/db/de/f1d 2168230 0 2024-03-30T04:28:22.562 INFO:tasks.workunit.client.0.smithi123.stdout:7/823: truncate d2/da/d11/d61/ff7 32689 0 2024-03-30T04:28:22.569 INFO:tasks.workunit.client.1.smithi135.stdout:1/423: symlink d16/d5a/l8e 0 2024-03-30T04:28:22.574 INFO:tasks.workunit.client.1.smithi135.stdout:9/400: unlink d7/l70 0 2024-03-30T04:28:22.575 INFO:tasks.workunit.client.0.smithi123.stdout:0/857: truncate d6/dd/d15/d28/f3d 2031569 0 2024-03-30T04:28:22.578 INFO:tasks.workunit.client.1.smithi135.stdout:8/436: rename d6/d8/d15/d54/d69/d72/l76 to d6/l7d 0 2024-03-30T04:28:22.578 INFO:tasks.workunit.client.0.smithi123.stdout:6/910: creat d10/d82/dc1/d45/d77/dd5/f133 x:0 0 0 2024-03-30T04:28:22.578 INFO:tasks.workunit.client.0.smithi123.stdout:5/811: mknod dd/d19/d1f/d3c/d72/c111 0 2024-03-30T04:28:22.581 INFO:tasks.workunit.client.0.smithi123.stdout:1/896: write db/dd/d42/d46/d5d/ffa [323260,118655] 0 2024-03-30T04:28:22.582 INFO:tasks.workunit.client.0.smithi123.stdout:3/948: creat d13/d24/d31/d23/d3a/d3c/dab/db2/f134 x:0 0 0 2024-03-30T04:28:22.582 INFO:tasks.workunit.client.1.smithi135.stdout:7/482: creat d4/db/d1a/d5c/d82/fa2 x:0 0 0 2024-03-30T04:28:22.584 INFO:tasks.workunit.client.0.smithi123.stdout:9/833: dwrite d6/d1f/d92/ffe [0,4194304] 0 2024-03-30T04:28:22.584 INFO:tasks.workunit.client.1.smithi135.stdout:1/424: symlink d16/d19/d20/d26/l8f 0 2024-03-30T04:28:22.590 INFO:tasks.workunit.client.0.smithi123.stdout:7/824: rename d2/da/d11/d61/d55/fde to d2/d4d/d80/d98/f114 0 2024-03-30T04:28:22.592 INFO:tasks.workunit.client.1.smithi135.stdout:9/401: rename d7/d4f/d6e/f76 to d7/d4f/d6e/f87 0 2024-03-30T04:28:22.592 INFO:tasks.workunit.client.1.smithi135.stdout:7/483: creat d4/db/d1a/d90/fa3 x:0 0 0 2024-03-30T04:28:22.596 INFO:tasks.workunit.client.0.smithi123.stdout:6/911: chown d10/d82/dc1/d6a/d7a/de8/l118 1863831 1 2024-03-30T04:28:22.597 INFO:tasks.workunit.client.0.smithi123.stdout:3/949: mkdir d13/d24/d31/df7/d135 0 2024-03-30T04:28:22.600 INFO:tasks.workunit.client.0.smithi123.stdout:8/871: dwrite da/dc/d14/d18/d62/f9f [8388608,4194304] 0 2024-03-30T04:28:22.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:22 smithi135 ceph-mon[44364]: from='client.24491 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:28:22.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:22 smithi135 ceph-mon[44364]: pgmap v44: 65 pgs: 65 active+clean; 2.4 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 94 MiB/s rd, 278 MiB/s wr, 566 op/s 2024-03-30T04:28:22.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:22 smithi135 ceph-mon[44364]: from='client.24495 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:28:22.605 INFO:tasks.workunit.client.1.smithi135.stdout:7/484: creat d4/db/d1a/d5c/d2c/d51/d56/fa4 x:0 0 0 2024-03-30T04:28:22.605 INFO:tasks.workunit.client.0.smithi123.stdout:8/872: fsync da/d27/d3c/d2d/dae/fee 0 2024-03-30T04:28:22.608 INFO:tasks.workunit.client.1.smithi135.stdout:9/402: rename d7/l30 to d7/d4f/d5c/d63/d86/l88 0 2024-03-30T04:28:22.608 INFO:tasks.workunit.client.0.smithi123.stdout:4/794: dwrite d2/d4/d2c/d42/f38 [0,4194304] 0 2024-03-30T04:28:22.609 INFO:tasks.workunit.client.1.smithi135.stdout:7/485: readlink d4/db/d1a/d5c/d2c/d51/l85 0 2024-03-30T04:28:22.613 INFO:tasks.workunit.client.0.smithi123.stdout:0/858: rename d6/dd/d15/fde to d6/dd/d15/d28/d4b/dad/de8/d10f/f110 0 2024-03-30T04:28:22.614 INFO:tasks.workunit.client.0.smithi123.stdout:7/825: fdatasync d2/f41 0 2024-03-30T04:28:22.614 INFO:tasks.workunit.client.1.smithi135.stdout:9/403: mkdir d7/d4f/d89 0 2024-03-30T04:28:22.619 INFO:tasks.workunit.client.1.smithi135.stdout:9/404: unlink d7/d41/c6b 0 2024-03-30T04:28:22.626 INFO:tasks.workunit.client.0.smithi123.stdout:9/834: symlink d6/d1f/d92/df4/l125 0 2024-03-30T04:28:22.626 INFO:tasks.workunit.client.1.smithi135.stdout:7/486: creat d4/db/d1a/d5c/d2c/d2e/d98/fa5 x:0 0 0 2024-03-30T04:28:22.627 INFO:tasks.workunit.client.0.smithi123.stdout:8/873: write da/d27/d3c/d2d/f4b [96890,64203] 0 2024-03-30T04:28:22.628 INFO:tasks.workunit.client.1.smithi135.stdout:9/405: creat d7/d4f/d89/f8a x:0 0 0 2024-03-30T04:28:22.629 INFO:tasks.workunit.client.0.smithi123.stdout:5/812: getdents dd/d19/d5f/dc2 0 2024-03-30T04:28:22.637 INFO:tasks.workunit.client.0.smithi123.stdout:1/897: link db/dd/d42/d46/d56/daf/fb5 db/dd/d42/d46/d5d/d62/d9e/f12a 0 2024-03-30T04:28:22.637 INFO:tasks.workunit.client.0.smithi123.stdout:9/835: dread - d6/d1f/d92/db8/fbc zero size 2024-03-30T04:28:22.639 INFO:tasks.workunit.client.1.smithi135.stdout:8/437: dread - d6/d8/d15/d54/f6d zero size 2024-03-30T04:28:22.641 INFO:tasks.workunit.client.1.smithi135.stdout:7/487: symlink d4/db/d1a/d5c/la6 0 2024-03-30T04:28:22.645 INFO:tasks.workunit.client.0.smithi123.stdout:5/813: write dd/d19/d3e/d5b/d62/f84 [5245523,26893] 0 2024-03-30T04:28:22.648 INFO:tasks.workunit.client.1.smithi135.stdout:1/425: chown d16/d19/d20/d26/c4a 240793 1 2024-03-30T04:28:22.649 INFO:tasks.workunit.client.0.smithi123.stdout:8/874: mknod da/d27/daf/c114 0 2024-03-30T04:28:22.653 INFO:tasks.workunit.client.0.smithi123.stdout:3/950: dwrite d13/d24/fbd [0,4194304] 0 2024-03-30T04:28:22.654 INFO:tasks.workunit.client.0.smithi123.stdout:7/826: dwrite d2/da/f110 [0,4194304] 0 2024-03-30T04:28:22.658 INFO:tasks.workunit.client.0.smithi123.stdout:9/836: creat d6/d49/d113/daa/dac/f126 x:0 0 0 2024-03-30T04:28:22.658 INFO:tasks.workunit.client.0.smithi123.stdout:6/912: dread d10/fb2 [0,4194304] 0 2024-03-30T04:28:22.662 INFO:tasks.workunit.client.0.smithi123.stdout:4/795: rename d2/d4/d2c/d42/d35/d9b/da2/la4 to d2/d4/d2c/d42/d35/da3/l110 0 2024-03-30T04:28:22.664 INFO:tasks.workunit.client.0.smithi123.stdout:5/814: creat dd/d19/d1f/d35/d3a/d57/db5/f112 x:0 0 0 2024-03-30T04:28:22.666 INFO:tasks.workunit.client.1.smithi135.stdout:9/406: link d7/c27 d7/d29/c8b 0 2024-03-30T04:28:22.667 INFO:tasks.workunit.client.1.smithi135.stdout:1/426: mkdir d16/d19/d20/d26/d90 0 2024-03-30T04:28:22.668 INFO:tasks.workunit.client.0.smithi123.stdout:8/875: stat da/dc/d14/d18/l87 0 2024-03-30T04:28:22.669 INFO:tasks.workunit.client.0.smithi123.stdout:6/913: mknod d10/d82/d8e/c134 0 2024-03-30T04:28:22.669 INFO:tasks.workunit.client.0.smithi123.stdout:3/951: chown d13/d24/d31/d23/d63/f87 1 1 2024-03-30T04:28:22.669 INFO:tasks.workunit.client.0.smithi123.stdout:5/815: stat dd/d19/d1f/d35/d3a/d79/lce 0 2024-03-30T04:28:22.669 INFO:tasks.workunit.client.0.smithi123.stdout:1/898: write db/dd/f8e [314821,63162] 0 2024-03-30T04:28:22.672 INFO:tasks.workunit.client.0.smithi123.stdout:8/876: dread - da/d27/d56/dde/f113 zero size 2024-03-30T04:28:22.673 INFO:tasks.workunit.client.0.smithi123.stdout:5/816: stat dd/d19/f59 0 2024-03-30T04:28:22.674 INFO:tasks.workunit.client.1.smithi135.stdout:7/488: dread d4/db/d1a/d4b/f31 [0,4194304] 0 2024-03-30T04:28:22.674 INFO:tasks.workunit.client.1.smithi135.stdout:1/427: symlink d16/d29/d32/d85/l91 0 2024-03-30T04:28:22.676 INFO:tasks.workunit.client.0.smithi123.stdout:4/796: chown d2/c46 226599 1 2024-03-30T04:28:22.677 INFO:tasks.workunit.client.1.smithi135.stdout:9/407: creat d7/d29/d78/f8c x:0 0 0 2024-03-30T04:28:22.677 INFO:tasks.workunit.client.1.smithi135.stdout:8/438: write d6/d8/d15/d2a/f3c [3670212,96095] 0 2024-03-30T04:28:22.678 INFO:tasks.workunit.client.0.smithi123.stdout:9/837: symlink d6/d1f/d92/d120/l127 0 2024-03-30T04:28:22.679 INFO:tasks.workunit.client.1.smithi135.stdout:1/428: fsync d16/d5a/f82 0 2024-03-30T04:28:22.680 INFO:tasks.workunit.client.0.smithi123.stdout:8/877: creat da/d27/d56/d6c/dd4/f115 x:0 0 0 2024-03-30T04:28:22.682 INFO:tasks.workunit.client.0.smithi123.stdout:7/827: rename d2/dc/d49/d68 to d2/dc/d115 0 2024-03-30T04:28:22.683 INFO:tasks.workunit.client.0.smithi123.stdout:3/952: creat d13/d24/d31/d123/f136 x:0 0 0 2024-03-30T04:28:22.685 INFO:tasks.workunit.client.0.smithi123.stdout:4/797: unlink d2/d4/d7/d63/c109 0 2024-03-30T04:28:22.685 INFO:tasks.workunit.client.0.smithi123.stdout:8/878: dread - da/d27/d3c/d2d/dae/fee zero size 2024-03-30T04:28:22.686 INFO:tasks.workunit.client.1.smithi135.stdout:8/439: mkdir d6/d8/d15/d54/d7e 0 2024-03-30T04:28:22.687 INFO:tasks.workunit.client.1.smithi135.stdout:1/429: creat d16/d5a/f92 x:0 0 0 2024-03-30T04:28:22.689 INFO:tasks.workunit.client.1.smithi135.stdout:9/408: truncate d7/d1b/d37/f54 96447 0 2024-03-30T04:28:22.691 INFO:tasks.workunit.client.1.smithi135.stdout:9/409: chown d7/d1b/f28 55300552 1 2024-03-30T04:28:22.691 INFO:tasks.workunit.client.1.smithi135.stdout:1/430: truncate d16/d29/f35 2448543 0 2024-03-30T04:28:22.692 INFO:tasks.workunit.client.1.smithi135.stdout:8/440: rmdir d6/d8/d15 39 2024-03-30T04:28:22.693 INFO:tasks.workunit.client.1.smithi135.stdout:1/431: fdatasync f7 0 2024-03-30T04:28:22.696 INFO:tasks.workunit.client.1.smithi135.stdout:1/432: chown d16/d29/l2d 7094793 1 2024-03-30T04:28:22.696 INFO:tasks.workunit.client.1.smithi135.stdout:9/410: mkdir d7/d55/d8d 0 2024-03-30T04:28:22.697 INFO:tasks.workunit.client.0.smithi123.stdout:3/953: write d13/d24/d31/d23/d63/ffd [726033,5304] 0 2024-03-30T04:28:22.699 INFO:tasks.workunit.client.0.smithi123.stdout:6/914: write f6 [1426630,98859] 0 2024-03-30T04:28:22.701 INFO:tasks.workunit.client.0.smithi123.stdout:6/915: stat d10/d82/dc1/d6a/la5 0 2024-03-30T04:28:22.701 INFO:tasks.workunit.client.0.smithi123.stdout:4/798: rmdir d2/d34/dd2 39 2024-03-30T04:28:22.701 INFO:tasks.workunit.client.0.smithi123.stdout:3/954: read d13/d24/d31/f4b [3834955,22399] 0 2024-03-30T04:28:22.703 INFO:tasks.workunit.client.0.smithi123.stdout:9/838: link d6/c20 d6/d1f/d45/c128 0 2024-03-30T04:28:22.705 INFO:tasks.workunit.client.0.smithi123.stdout:7/828: write d2/dc/d1c/f23 [3869771,103749] 0 2024-03-30T04:28:22.705 INFO:tasks.workunit.client.0.smithi123.stdout:9/839: chown d6/d1f/c68 317 1 2024-03-30T04:28:22.706 INFO:tasks.workunit.client.0.smithi123.stdout:4/799: chown d2/d4/l33 808632 1 2024-03-30T04:28:22.706 INFO:tasks.workunit.client.1.smithi135.stdout:8/441: mknod d6/d8/d15/d2a/c7f 0 2024-03-30T04:28:22.710 INFO:tasks.workunit.client.0.smithi123.stdout:6/916: chown d10/d20/da2/cc4 1283137 1 2024-03-30T04:28:22.710 INFO:tasks.workunit.client.0.smithi123.stdout:8/879: creat da/dc/d14/d18/f116 x:0 0 0 2024-03-30T04:28:22.715 INFO:tasks.workunit.client.0.smithi123.stdout:9/840: creat d6/d49/df9/f129 x:0 0 0 2024-03-30T04:28:22.716 INFO:tasks.workunit.client.1.smithi135.stdout:1/433: write d16/d5a/d5f/f4e [2500927,56374] 0 2024-03-30T04:28:22.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:22 smithi123 ceph-mon[35245]: from='client.24491 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:28:22.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:22 smithi123 ceph-mon[35245]: pgmap v44: 65 pgs: 65 active+clean; 2.4 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 94 MiB/s rd, 278 MiB/s wr, 566 op/s 2024-03-30T04:28:22.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:22 smithi123 ceph-mon[35245]: from='client.24495 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:28:22.717 INFO:tasks.workunit.client.0.smithi123.stdout:7/829: creat d2/dc/d56/f116 x:0 0 0 2024-03-30T04:28:22.721 INFO:tasks.workunit.client.0.smithi123.stdout:4/800: mkdir d2/d4/d2c/d42/d35/d5d/d78/d111 0 2024-03-30T04:28:22.721 INFO:tasks.workunit.client.0.smithi123.stdout:7/830: readlink d2/da/d11/d61/d55/l5a 0 2024-03-30T04:28:22.721 INFO:tasks.workunit.client.0.smithi123.stdout:6/917: symlink d10/d82/d8e/df0/l135 0 2024-03-30T04:28:22.722 INFO:tasks.workunit.client.1.smithi135.stdout:8/442: rename d6/d8/f2d to d6/d8/d15/d54/d69/d72/d73/f80 0 2024-03-30T04:28:22.723 INFO:tasks.workunit.client.1.smithi135.stdout:7/489: dwrite d4/db/de/f2b [0,4194304] 0 2024-03-30T04:28:22.723 INFO:tasks.workunit.client.0.smithi123.stdout:8/880: creat da/d27/d3c/d2d/dae/dd9/d10b/f117 x:0 0 0 2024-03-30T04:28:22.724 INFO:tasks.workunit.client.1.smithi135.stdout:9/411: dwrite d7/d4f/d89/f8a [0,4194304] 0 2024-03-30T04:28:22.727 INFO:tasks.workunit.client.0.smithi123.stdout:4/801: dread - d2/fcb zero size 2024-03-30T04:28:22.727 INFO:tasks.workunit.client.0.smithi123.stdout:9/841: link d6/d58/f5c d6/d1f/d72/d10d/d11c/f12a 0 2024-03-30T04:28:22.728 INFO:tasks.workunit.client.0.smithi123.stdout:1/899: dread db/dd/d42/d46/d51/f55 [0,4194304] 0 2024-03-30T04:28:22.730 INFO:tasks.workunit.client.1.smithi135.stdout:1/434: creat d16/f93 x:0 0 0 2024-03-30T04:28:22.730 INFO:tasks.workunit.client.0.smithi123.stdout:8/881: dread - da/d27/d3c/d2d/dae/dd9/d10b/f117 zero size 2024-03-30T04:28:22.735 INFO:tasks.workunit.client.1.smithi135.stdout:7/490: creat d4/fa7 x:0 0 0 2024-03-30T04:28:22.736 INFO:tasks.workunit.client.0.smithi123.stdout:9/842: dread - d6/d49/d113/f123 zero size 2024-03-30T04:28:22.738 INFO:tasks.workunit.client.1.smithi135.stdout:9/412: creat d7/d41/d72/f8e x:0 0 0 2024-03-30T04:28:22.739 INFO:tasks.workunit.client.0.smithi123.stdout:4/802: write d2/d4/d7/fb [4326139,78168] 0 2024-03-30T04:28:22.740 INFO:tasks.workunit.client.0.smithi123.stdout:7/831: write d2/da/d11/d61/d55/d59/f111 [4549662,114758] 0 2024-03-30T04:28:22.740 INFO:tasks.workunit.client.1.smithi135.stdout:8/443: fdatasync d6/d8/f31 0 2024-03-30T04:28:22.743 INFO:tasks.workunit.client.0.smithi123.stdout:3/955: dwrite d13/d24/d31/d23/d3a/d3c/dab/db2/f134 [0,4194304] 0 2024-03-30T04:28:22.747 INFO:tasks.workunit.client.1.smithi135.stdout:9/413: mkdir d7/d4f/d8f 0 2024-03-30T04:28:22.747 INFO:tasks.workunit.client.1.smithi135.stdout:1/435: fdatasync d16/f1c 0 2024-03-30T04:28:22.747 INFO:tasks.workunit.client.1.smithi135.stdout:8/444: symlink d6/d8/d15/d54/d69/d72/l81 0 2024-03-30T04:28:22.751 INFO:tasks.workunit.client.1.smithi135.stdout:9/414: mkdir d7/d90 0 2024-03-30T04:28:22.757 INFO:tasks.workunit.client.0.smithi123.stdout:3/956: dread d13/f5e [0,4194304] 0 2024-03-30T04:28:22.758 INFO:tasks.workunit.client.1.smithi135.stdout:1/436: symlink d16/d29/d2c/l94 0 2024-03-30T04:28:22.759 INFO:tasks.workunit.client.1.smithi135.stdout:9/415: write d7/d29/f82 [4119160,34377] 0 2024-03-30T04:28:22.762 INFO:tasks.workunit.client.1.smithi135.stdout:1/437: write fb [173835,6661] 0 2024-03-30T04:28:22.762 INFO:tasks.workunit.client.0.smithi123.stdout:3/957: write d13/d24/d31/d23/d63/ffd [1593589,60980] 0 2024-03-30T04:28:22.765 INFO:tasks.workunit.client.1.smithi135.stdout:9/416: symlink d7/d41/d72/l91 0 2024-03-30T04:28:22.770 INFO:tasks.workunit.client.0.smithi123.stdout:3/958: write d13/d24/d31/d123/d5d/da5/fdb [2763842,44395] 0 2024-03-30T04:28:22.770 INFO:tasks.workunit.client.1.smithi135.stdout:7/491: dwrite d4/db/d1a/d4b/f31 [4194304,4194304] 0 2024-03-30T04:28:22.772 INFO:tasks.workunit.client.0.smithi123.stdout:4/803: dwrite d2/d4/d2c/d42/d35/f5c [0,4194304] 0 2024-03-30T04:28:22.773 INFO:tasks.workunit.client.1.smithi135.stdout:1/438: creat d16/d5a/d5f/d3a/d6f/f95 x:0 0 0 2024-03-30T04:28:22.778 INFO:tasks.workunit.client.1.smithi135.stdout:9/417: readlink d7/l32 0 2024-03-30T04:28:22.779 INFO:tasks.workunit.client.1.smithi135.stdout:9/418: fsync d7/d29/f62 0 2024-03-30T04:28:22.782 INFO:tasks.workunit.client.1.smithi135.stdout:1/439: creat d16/d5a/d5f/d3d/f96 x:0 0 0 2024-03-30T04:28:22.783 INFO:tasks.workunit.client.1.smithi135.stdout:9/419: symlink d7/d90/l92 0 2024-03-30T04:28:22.787 INFO:tasks.workunit.client.1.smithi135.stdout:9/420: write d7/d1b/f7c [818116,96361] 0 2024-03-30T04:28:22.791 INFO:tasks.workunit.client.1.smithi135.stdout:9/421: readlink d7/d41/d72/l91 0 2024-03-30T04:28:22.798 INFO:tasks.workunit.client.1.smithi135.stdout:9/422: rmdir d7/d1b/d37 39 2024-03-30T04:28:22.798 INFO:tasks.workunit.client.0.smithi123.stdout:0/859: sync 2024-03-30T04:28:22.799 INFO:tasks.workunit.client.0.smithi123.stdout:0/860: chown d6/d52/f96 83542 1 2024-03-30T04:28:22.801 INFO:tasks.workunit.client.0.smithi123.stdout:0/861: chown d6/d52 22657010 1 2024-03-30T04:28:22.802 INFO:tasks.workunit.client.1.smithi135.stdout:9/423: symlink d7/d4f/d5c/d63/l93 0 2024-03-30T04:28:22.805 INFO:tasks.workunit.client.1.smithi135.stdout:9/424: chown d7/d1b/l14 52362281 1 2024-03-30T04:28:22.808 INFO:tasks.workunit.client.1.smithi135.stdout:6/461: sync 2024-03-30T04:28:22.809 INFO:tasks.workunit.client.0.smithi123.stdout:1/900: creat db/dd/d5f/d6d/d85/dd4/f12b x:0 0 0 2024-03-30T04:28:22.810 INFO:tasks.workunit.client.1.smithi135.stdout:6/462: stat d2/dd/d18/f6b 0 2024-03-30T04:28:22.819 INFO:tasks.workunit.client.1.smithi135.stdout:7/492: dread d4/f11 [0,4194304] 0 2024-03-30T04:28:22.820 INFO:tasks.workunit.client.0.smithi123.stdout:3/959: unlink d13/d24/d31/d123/d5d/f82 0 2024-03-30T04:28:22.822 INFO:tasks.workunit.client.1.smithi135.stdout:1/440: dread d16/d1a/f4c [0,4194304] 0 2024-03-30T04:28:22.823 INFO:tasks.workunit.client.0.smithi123.stdout:8/882: dwrite da/dc/da9/fad [0,4194304] 0 2024-03-30T04:28:22.825 INFO:tasks.workunit.client.1.smithi135.stdout:1/441: symlink d16/d1a/l97 0 2024-03-30T04:28:22.826 INFO:tasks.workunit.client.0.smithi123.stdout:8/883: write f6 [3046498,41320] 0 2024-03-30T04:28:22.827 INFO:tasks.workunit.client.0.smithi123.stdout:6/918: creat d10/d82/d8e/dc7/d5b/f136 x:0 0 0 2024-03-30T04:28:22.829 INFO:tasks.workunit.client.1.smithi135.stdout:7/493: rename d4/db/d1a/d4b/d4f to d4/db/d1a/d5c/d2c/d51/da8 0 2024-03-30T04:28:22.830 INFO:tasks.workunit.client.1.smithi135.stdout:1/442: truncate d16/d5a/d5f/f7f 971614 0 2024-03-30T04:28:22.832 INFO:tasks.workunit.client.1.smithi135.stdout:7/494: stat d4/db/d1a/d5c/d2c/d2e/d98 0 2024-03-30T04:28:22.835 INFO:tasks.workunit.client.1.smithi135.stdout:1/443: chown d16/d5a/d5f/f66 1395 1 2024-03-30T04:28:22.837 INFO:tasks.workunit.client.1.smithi135.stdout:1/444: chown d16/f42 5569 1 2024-03-30T04:28:22.839 INFO:tasks.workunit.client.1.smithi135.stdout:7/495: rename d4/db/d1a/d4b/d40/d3a to d4/db/d79/da9 0 2024-03-30T04:28:22.843 INFO:tasks.workunit.client.1.smithi135.stdout:6/463: dwrite d2/d36/f44 [0,4194304] 0 2024-03-30T04:28:22.843 INFO:tasks.workunit.client.1.smithi135.stdout:9/425: dread d7/ff [0,4194304] 0 2024-03-30T04:28:22.843 INFO:tasks.workunit.client.1.smithi135.stdout:7/496: chown d4/db/d1a 0 1 2024-03-30T04:28:22.845 INFO:tasks.workunit.client.0.smithi123.stdout:8/884: truncate da/dc/d14/f71 851208 0 2024-03-30T04:28:22.848 INFO:tasks.workunit.client.1.smithi135.stdout:7/497: creat d4/db/d1a/d5c/d2c/d2e/d46/faa x:0 0 0 2024-03-30T04:28:22.849 INFO:tasks.workunit.client.1.smithi135.stdout:6/464: getdents d2/dd/d40 0 2024-03-30T04:28:22.852 INFO:tasks.workunit.client.1.smithi135.stdout:6/465: chown d2/dd/c19 53 1 2024-03-30T04:28:22.852 INFO:tasks.workunit.client.0.smithi123.stdout:4/804: link d2/d26/lb2 d2/d34/l112 0 2024-03-30T04:28:22.854 INFO:tasks.workunit.client.1.smithi135.stdout:7/498: mknod d4/db/d1a/d5c/d2c/d51/d56/cab 0 2024-03-30T04:28:22.856 INFO:tasks.workunit.client.0.smithi123.stdout:9/843: dread d6/d49/d113/daa/fd5 [0,4194304] 0 2024-03-30T04:28:22.856 INFO:tasks.workunit.client.0.smithi123.stdout:8/885: fsync da/d27/d56/d6c/d6d/f84 0 2024-03-30T04:28:22.858 INFO:tasks.workunit.client.1.smithi135.stdout:1/445: dwrite d16/f42 [0,4194304] 0 2024-03-30T04:28:22.860 INFO:tasks.workunit.client.1.smithi135.stdout:6/466: unlink d2/dd/d14/d30/l42 0 2024-03-30T04:28:22.862 INFO:tasks.workunit.client.0.smithi123.stdout:7/832: symlink d2/dc/d49/l117 0 2024-03-30T04:28:22.862 INFO:tasks.workunit.client.1.smithi135.stdout:7/499: write d4/db/d79/da9/f58 [3862196,1064] 0 2024-03-30T04:28:22.865 INFO:tasks.workunit.client.1.smithi135.stdout:6/467: creat d2/dd/d14/d2b/f90 x:0 0 0 2024-03-30T04:28:22.867 INFO:tasks.workunit.client.0.smithi123.stdout:8/886: truncate da/ff 5978253 0 2024-03-30T04:28:22.871 INFO:tasks.workunit.client.0.smithi123.stdout:5/817: sync 2024-03-30T04:28:22.874 INFO:tasks.workunit.client.1.smithi135.stdout:6/468: mknod d2/dd/d18/d87/c91 0 2024-03-30T04:28:22.877 INFO:tasks.workunit.client.0.smithi123.stdout:5/818: chown dd/d19/d3e/d5b/f5e 117031963 1 2024-03-30T04:28:22.879 INFO:tasks.workunit.client.0.smithi123.stdout:8/887: read - da/dc/d14/d18/fcb zero size 2024-03-30T04:28:22.882 INFO:tasks.workunit.client.1.smithi135.stdout:6/469: getdents d2/dd/d14 0 2024-03-30T04:28:22.883 INFO:tasks.workunit.client.0.smithi123.stdout:9/844: dwrite d6/d1f/d72/d7d/d10f/d114/d115/f117 [0,4194304] 0 2024-03-30T04:28:22.893 INFO:tasks.workunit.client.0.smithi123.stdout:9/845: mkdir d6/d1f/d45/d4c/d12b 0 2024-03-30T04:28:22.899 INFO:tasks.workunit.client.0.smithi123.stdout:4/805: dread d2/d4/d2c/d42/d35/d5d/d78/fbe [0,4194304] 0 2024-03-30T04:28:22.910 INFO:tasks.workunit.client.0.smithi123.stdout:4/806: rename d2/d4/d2c/d42/dad/d91/d9d to d2/d4/d7/dc1/d113 0 2024-03-30T04:28:22.923 INFO:tasks.workunit.client.0.smithi123.stdout:1/901: write db/dd/d42/d46/ff5 [659444,64779] 0 2024-03-30T04:28:22.925 INFO:tasks.workunit.client.1.smithi135.stdout:1/446: truncate d16/d5a/d5f/f4e 1346339 0 2024-03-30T04:28:22.926 INFO:tasks.workunit.client.1.smithi135.stdout:7/500: write d4/f8 [2205342,14298] 0 2024-03-30T04:28:22.928 INFO:tasks.workunit.client.0.smithi123.stdout:6/919: write d10/d82/dc1/d6a/fb5 [410246,121449] 0 2024-03-30T04:28:22.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.933+0000 7fad5d600700 1 -- 172.21.15.123:0/4260535560 --> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fad58115bc0 con 0x7fad3c06afe0 2024-03-30T04:28:22.934 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:28:22.934 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:28:22.935 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:28:22.935 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-30T04:28:22.935 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [], 2024-03-30T04:28:22.935 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "1/2 daemons upgraded", 2024-03-30T04:28:22.935 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "", 2024-03-30T04:28:22.935 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:28:22.935 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:28:22.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.936+0000 7fad4ea00700 1 -- 172.21.15.123:0/4260535560 <== mgr.14608 v2:172.21.15.135:6828/3601553667 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7fad58115bc0 con 0x7fad3c06afe0 2024-03-30T04:28:22.937 INFO:tasks.workunit.client.1.smithi135.stdout:6/470: symlink d2/dd/d40/l92 0 2024-03-30T04:28:22.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.939+0000 7fad5d600700 1 -- 172.21.15.123:0/4260535560 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7fad3c06afe0 msgr2=0x7fad3c06d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:28:22.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.939+0000 7fad5d600700 1 --2- 172.21.15.123:0/4260535560 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7fad3c06afe0 0x7fad3c06d490 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fad44009fd0 tx=0x7fad44009e50 comp rx=0 tx=0).stop 2024-03-30T04:28:22.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.939+0000 7fad5d600700 1 -- 172.21.15.123:0/4260535560 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad580feff0 msgr2=0x7fad58113f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:28:22.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.939+0000 7fad5d600700 1 --2- 172.21.15.123:0/4260535560 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad580feff0 0x7fad58113f10 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fad4800b3b0 tx=0x7fad48009e30 comp rx=0 tx=0).stop 2024-03-30T04:28:22.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.940+0000 7fad5d600700 1 -- 172.21.15.123:0/4260535560 shutdown_connections 2024-03-30T04:28:22.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.940+0000 7fad5d600700 1 --2- 172.21.15.123:0/4260535560 >> [v2:172.21.15.135:6828/3601553667,v1:172.21.15.135:6829/3601553667] conn(0x7fad3c06afe0 0x7fad3c06d490 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:22.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.940+0000 7fad5d600700 1 --2- 172.21.15.123:0/4260535560 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad580feff0 0x7fad58113f10 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:22.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.940+0000 7fad5d600700 1 --2- 172.21.15.123:0/4260535560 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fad581002a0 0x7fad58114450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:22.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.940+0000 7fad5d600700 1 -- 172.21.15.123:0/4260535560 >> 172.21.15.123:0/4260535560 conn(0x7fad580fa950 msgr2=0x7fad581014c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:28:22.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.940+0000 7fad5d600700 1 -- 172.21.15.123:0/4260535560 shutdown_connections 2024-03-30T04:28:22.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:22.940+0000 7fad5d600700 1 -- 172.21.15.123:0/4260535560 wait complete. 2024-03-30T04:28:22.940 INFO:tasks.workunit.client.1.smithi135.stdout:6/471: stat d2/d36/c6f 0 2024-03-30T04:28:22.941 INFO:tasks.workunit.client.1.smithi135.stdout:7/501: unlink d4/db/de/f1d 0 2024-03-30T04:28:22.941 INFO:tasks.workunit.client.0.smithi123.stdout:0/862: dwrite d6/dd/f8c [0,4194304] 0 2024-03-30T04:28:22.941 INFO:tasks.workunit.client.0.smithi123.stdout:3/960: dwrite d13/d24/d31/d30/f9a [0,4194304] 0 2024-03-30T04:28:22.942 INFO:tasks.workunit.client.0.smithi123.stdout:1/902: fdatasync db/dd/d5f/d6d/d85/d89/d90/d94/fe9 0 2024-03-30T04:28:22.945 INFO:tasks.workunit.client.1.smithi135.stdout:6/472: write d2/f8 [591380,49603] 0 2024-03-30T04:28:22.945 INFO:tasks.workunit.client.0.smithi123.stdout:0/863: chown d6/dd/d15/l46 158811413 1 2024-03-30T04:28:22.948 INFO:tasks.workunit.client.0.smithi123.stdout:6/920: rmdir d10/d82/dc1/d6a/d7a/de2/df1/d112 39 2024-03-30T04:28:22.948 INFO:tasks.workunit.client.0.smithi123.stdout:7/833: creat d2/da/d11/f118 x:0 0 0 2024-03-30T04:28:22.957 INFO:tasks.workunit.client.0.smithi123.stdout:8/888: unlink da/dc/f16 0 2024-03-30T04:28:22.959 INFO:tasks.workunit.client.0.smithi123.stdout:8/889: stat da/dc/d14/d18/d62/c89 0 2024-03-30T04:28:22.960 INFO:tasks.workunit.client.0.smithi123.stdout:8/890: readlink da/dc/l26 0 2024-03-30T04:28:22.961 INFO:tasks.workunit.client.0.smithi123.stdout:3/961: truncate d13/f5e 724077 0 2024-03-30T04:28:22.965 INFO:tasks.workunit.client.1.smithi135.stdout:1/447: dwrite d16/d5a/d5f/d3d/f86 [0,4194304] 0 2024-03-30T04:28:22.966 INFO:tasks.workunit.client.0.smithi123.stdout:0/864: symlink d6/dd/d15/d28/d4b/d8a/l111 0 2024-03-30T04:28:22.967 INFO:tasks.workunit.client.0.smithi123.stdout:8/891: rmdir da/d27/d56/d6c/dd4 39 2024-03-30T04:28:22.970 INFO:tasks.workunit.client.1.smithi135.stdout:1/448: creat d16/d29/d61/f98 x:0 0 0 2024-03-30T04:28:22.971 INFO:tasks.workunit.client.0.smithi123.stdout:7/834: symlink d2/da/d11/d34/ded/dfc/l119 0 2024-03-30T04:28:22.972 INFO:tasks.workunit.client.1.smithi135.stdout:6/473: dwrite d2/d68/f83 [0,4194304] 0 2024-03-30T04:28:22.973 INFO:tasks.workunit.client.0.smithi123.stdout:3/962: symlink d13/d24/d39/d79/d90/d11f/l137 0 2024-03-30T04:28:22.977 INFO:tasks.workunit.client.0.smithi123.stdout:0/865: fsync d6/dd/d15/fc8 0 2024-03-30T04:28:22.978 INFO:tasks.workunit.client.0.smithi123.stdout:8/892: mknod da/dc/da9/dc4/c118 0 2024-03-30T04:28:22.981 INFO:tasks.workunit.client.0.smithi123.stdout:3/963: stat d13/d24/d31/d23/d3a/d3c/dab/db2/cfc 0 2024-03-30T04:28:22.981 INFO:tasks.workunit.client.1.smithi135.stdout:1/449: rename d16/d1a to d16/d19/d20/d99 0 2024-03-30T04:28:22.983 INFO:tasks.workunit.client.1.smithi135.stdout:6/474: creat d2/dd/d14/d2b/d86/d55/f93 x:0 0 0 2024-03-30T04:28:22.984 INFO:tasks.workunit.client.0.smithi123.stdout:4/807: dwrite d2/d34/f7c [0,4194304] 0 2024-03-30T04:28:22.985 INFO:tasks.workunit.client.0.smithi123.stdout:6/921: creat d10/d82/dc1/d6a/d7a/de2/df1/f137 x:0 0 0 2024-03-30T04:28:22.986 INFO:tasks.workunit.client.1.smithi135.stdout:6/475: chown d2/dd/d18/f73 1098 1 2024-03-30T04:28:22.986 INFO:tasks.workunit.client.1.smithi135.stdout:1/450: mkdir d16/d19/d20/d99/d9a 0 2024-03-30T04:28:22.989 INFO:tasks.workunit.client.1.smithi135.stdout:1/451: truncate d16/d5a/d5f/f66 664860 0 2024-03-30T04:28:22.989 INFO:tasks.workunit.client.1.smithi135.stdout:6/476: mknod d2/dd/d14/d38/c94 0 2024-03-30T04:28:22.992 INFO:tasks.workunit.client.0.smithi123.stdout:0/866: mkdir d6/dd/d15/d28/d4b/dad/df2/d112 0 2024-03-30T04:28:22.995 INFO:tasks.workunit.client.0.smithi123.stdout:3/964: write d13/d24/d31/d23/d3a/d3c/db8/f132 [221989,34997] 0 2024-03-30T04:28:22.998 INFO:tasks.workunit.client.0.smithi123.stdout:0/867: dread - d6/dd/d15/fc8 zero size 2024-03-30T04:28:22.999 INFO:tasks.workunit.client.0.smithi123.stdout:6/922: truncate d10/d82/dc1/d6a/fb5 827661 0 2024-03-30T04:28:23.002 INFO:tasks.workunit.client.0.smithi123.stdout:1/903: dwrite db/dd/d42/d46/d56/da2/fd2 [0,4194304] 0 2024-03-30T04:28:23.004 INFO:tasks.workunit.client.0.smithi123.stdout:7/835: write f0 [4673841,13941] 0 2024-03-30T04:28:23.007 INFO:tasks.workunit.client.0.smithi123.stdout:3/965: mknod d13/d24/d31/d123/dde/c138 0 2024-03-30T04:28:23.011 INFO:tasks.workunit.client.0.smithi123.stdout:6/923: unlink d10/d82/d9a/f120 0 2024-03-30T04:28:23.018 INFO:tasks.workunit.client.0.smithi123.stdout:0/868: mkdir d6/dd/d15/d28/d4b/dad/de8/d10f/d113 0 2024-03-30T04:28:23.018 INFO:tasks.workunit.client.0.smithi123.stdout:4/808: dwrite d2/d4/f10d [0,4194304] 0 2024-03-30T04:28:23.032 INFO:tasks.workunit.client.0.smithi123.stdout:7/836: rename d2/dc/f52 to d2/da/d11/d61/d55/d9e/f11a 0 2024-03-30T04:28:23.032 INFO:tasks.workunit.client.0.smithi123.stdout:3/966: link d13/d24/d39/d79/c9b d13/d24/d31/d23/d3a/d3c/db8/d10b/c139 0 2024-03-30T04:28:23.037 INFO:tasks.workunit.client.0.smithi123.stdout:3/967: creat d13/d117/f13a x:0 0 0 2024-03-30T04:28:23.043 INFO:tasks.workunit.client.0.smithi123.stdout:0/869: rename d6/dd/d15/d98/dfe to d6/dd/d31/d35/d114 0 2024-03-30T04:28:23.043 INFO:tasks.workunit.client.0.smithi123.stdout:3/968: stat d13/d24/d31/d30/f56 0 2024-03-30T04:28:23.050 INFO:tasks.workunit.client.0.smithi123.stdout:7/837: rename d2/da/d11/d61/d55/d9e/d113/fcf to d2/da/d11/d61/d55/d59/db5/f11b 0 2024-03-30T04:28:23.051 INFO:tasks.workunit.client.0.smithi123.stdout:4/809: dwrite d2/d4/d2c/d42/d35/d5d/d78/fef [0,4194304] 0 2024-03-30T04:28:23.051 INFO:tasks.workunit.client.0.smithi123.stdout:0/870: rmdir d6/dd/d18/d2f/da3 39 2024-03-30T04:28:23.053 INFO:tasks.workunit.client.0.smithi123.stdout:3/969: symlink d13/d24/d39/d79/d90/d11f/l13b 0 2024-03-30T04:28:23.067 INFO:tasks.workunit.client.0.smithi123.stdout:6/924: dread d10/f96 [0,4194304] 0 2024-03-30T04:28:23.067 INFO:tasks.workunit.client.0.smithi123.stdout:0/871: creat d6/dd/d15/d98/d5d/f115 x:0 0 0 2024-03-30T04:28:23.068 INFO:tasks.workunit.client.0.smithi123.stdout:8/893: dread da/d27/d56/d6c/d6d/f79 [0,4194304] 0 2024-03-30T04:28:23.069 INFO:tasks.workunit.client.0.smithi123.stdout:1/904: dwrite db/dd/d42/d46/d5d/d62/d9e/f12a [0,4194304] 0 2024-03-30T04:28:23.072 INFO:tasks.workunit.client.0.smithi123.stdout:7/838: truncate d2/dc/d1c/f7c 2470697 0 2024-03-30T04:28:23.081 INFO:tasks.workunit.client.0.smithi123.stdout:9/846: sync 2024-03-30T04:28:23.081 INFO:tasks.workunit.client.0.smithi123.stdout:7/839: stat d2/da/d11/d61/d55/d94 0 2024-03-30T04:28:23.081 INFO:tasks.workunit.client.0.smithi123.stdout:5/819: sync 2024-03-30T04:28:23.081 INFO:tasks.workunit.client.0.smithi123.stdout:4/810: dwrite d2/d26/ff6 [0,4194304] 0 2024-03-30T04:28:23.086 INFO:tasks.workunit.client.0.smithi123.stdout:8/894: unlink da/d27/d56/d6c/dd4/cf0 0 2024-03-30T04:28:23.094 INFO:tasks.workunit.client.0.smithi123.stdout:3/970: truncate d13/d24/d31/d23/d3a/d3c/d66/f7f 5966593 0 2024-03-30T04:28:23.094 INFO:tasks.workunit.client.0.smithi123.stdout:6/925: write d10/d82/dc1/d45/fd3 [65491,22364] 0 2024-03-30T04:28:23.096 INFO:tasks.workunit.client.0.smithi123.stdout:5/820: fsync dd/d19/d1f/d68/f73 0 2024-03-30T04:28:23.099 INFO:tasks.workunit.client.0.smithi123.stdout:0/872: rename d6/d52/cba to d6/dd/d31/c116 0 2024-03-30T04:28:23.103 INFO:tasks.workunit.client.0.smithi123.stdout:1/905: write db/dd/d5f/d6d/d85/d89/d90/d94/fe9 [1666280,9287] 0 2024-03-30T04:28:23.108 INFO:tasks.workunit.client.0.smithi123.stdout:7/840: chown d2/dc/d1c/f7c 30169133 1 2024-03-30T04:28:23.110 INFO:tasks.workunit.client.0.smithi123.stdout:6/926: dread - d10/d82/dc1/d45/fb9 zero size 2024-03-30T04:28:23.112 INFO:tasks.workunit.client.0.smithi123.stdout:9/847: rename d6/d1f/d45/d4c/d121/f5d to d6/d1f/d72/d7d/f12c 0 2024-03-30T04:28:23.114 INFO:tasks.workunit.client.0.smithi123.stdout:8/895: link da/d27/ff6 da/d27/d56/d6c/d96/f119 0 2024-03-30T04:28:23.117 INFO:tasks.workunit.client.0.smithi123.stdout:6/927: mknod d10/d82/dd6/c138 0 2024-03-30T04:28:23.118 INFO:tasks.workunit.client.0.smithi123.stdout:9/848: creat d6/d7c/f12d x:0 0 0 2024-03-30T04:28:23.120 INFO:tasks.workunit.client.0.smithi123.stdout:3/971: creat d13/f13c x:0 0 0 2024-03-30T04:28:23.121 INFO:tasks.workunit.client.0.smithi123.stdout:5/821: dread dd/f26 [0,4194304] 0 2024-03-30T04:28:23.121 INFO:tasks.workunit.client.0.smithi123.stdout:1/906: mkdir db/dd/d5f/d6d/d85/d89/d90/d12c 0 2024-03-30T04:28:23.123 INFO:tasks.workunit.client.0.smithi123.stdout:7/841: dread d2/dc/d7f/fa2 [0,4194304] 0 2024-03-30T04:28:23.123 INFO:tasks.workunit.client.0.smithi123.stdout:8/896: creat da/d27/d56/d6c/d96/f11a x:0 0 0 2024-03-30T04:28:23.127 INFO:tasks.workunit.client.0.smithi123.stdout:9/849: mkdir d6/d1f/d72/d10d/d11c/d12e 0 2024-03-30T04:28:23.129 INFO:tasks.workunit.client.0.smithi123.stdout:5/822: chown dd/d19/d1f/ced 1818 1 2024-03-30T04:28:23.129 INFO:tasks.workunit.client.0.smithi123.stdout:3/972: dread d13/d24/d39/d79/db9/de7/f10e [0,4194304] 0 2024-03-30T04:28:23.129 INFO:tasks.workunit.client.0.smithi123.stdout:7/842: read d2/da/d11/d34/d85/fa9 [4773184,14315] 0 2024-03-30T04:28:23.135 INFO:tasks.workunit.client.0.smithi123.stdout:9/850: fsync d6/d1f/f70 0 2024-03-30T04:28:23.137 INFO:tasks.workunit.client.0.smithi123.stdout:0/873: write d6/dd/d15/fc3 [1186529,118261] 0 2024-03-30T04:28:23.139 INFO:tasks.workunit.client.0.smithi123.stdout:8/897: rename da/dc/d14/d9c/lfc to da/dc/d14/d9c/l11b 0 2024-03-30T04:28:23.139 INFO:tasks.workunit.client.0.smithi123.stdout:5/823: rmdir dd/d19/d1f/d35/d3a 39 2024-03-30T04:28:23.141 INFO:tasks.workunit.client.0.smithi123.stdout:3/973: creat d13/d24/d39/d79/dea/f13d x:0 0 0 2024-03-30T04:28:23.143 INFO:tasks.workunit.client.0.smithi123.stdout:7/843: creat d2/dc/d49/dbc/f11c x:0 0 0 2024-03-30T04:28:23.146 INFO:tasks.workunit.client.0.smithi123.stdout:6/928: write d10/f38 [2451018,101930] 0 2024-03-30T04:28:23.147 INFO:tasks.workunit.client.0.smithi123.stdout:0/874: symlink d6/d52/dcf/l117 0 2024-03-30T04:28:23.150 INFO:tasks.workunit.client.0.smithi123.stdout:8/898: mkdir da/d27/d3c/d2d/d11c 0 2024-03-30T04:28:23.155 INFO:tasks.workunit.client.0.smithi123.stdout:1/907: dread db/dd/d42/d46/d39/f57 [0,4194304] 0 2024-03-30T04:28:23.157 INFO:tasks.workunit.client.0.smithi123.stdout:5/824: dread fb [0,4194304] 0 2024-03-30T04:28:23.161 INFO:tasks.workunit.client.0.smithi123.stdout:6/929: truncate d10/d82/dc1/dd2/ffd 1045447 0 2024-03-30T04:28:23.162 INFO:tasks.workunit.client.0.smithi123.stdout:7/844: write d2/dc/d56/feb [2633515,82227] 0 2024-03-30T04:28:23.163 INFO:tasks.workunit.client.0.smithi123.stdout:9/851: dwrite d6/d49/d113/daa/dac/f100 [0,4194304] 0 2024-03-30T04:28:23.164 INFO:tasks.workunit.client.0.smithi123.stdout:8/899: rename da/d27/d3c/cb5 to da/d27/d56/d93/c11d 0 2024-03-30T04:28:23.167 INFO:tasks.workunit.client.0.smithi123.stdout:1/908: rmdir db/dd/d5f/d6d/d85/d89/d90/d94 39 2024-03-30T04:28:23.169 INFO:tasks.workunit.client.0.smithi123.stdout:9/852: stat d6/d49/d113/daa/dac 0 2024-03-30T04:28:23.170 INFO:tasks.workunit.client.0.smithi123.stdout:5/825: truncate dd/d19/d1f/d3c/d72/d8c/fb1 695975 0 2024-03-30T04:28:23.171 INFO:tasks.workunit.client.0.smithi123.stdout:7/845: dread d2/f41 [0,4194304] 0 2024-03-30T04:28:23.173 INFO:tasks.workunit.client.0.smithi123.stdout:6/930: symlink d10/d82/dc1/d6a/l139 0 2024-03-30T04:28:23.173 INFO:tasks.workunit.client.0.smithi123.stdout:8/900: creat da/dc/f11e x:0 0 0 2024-03-30T04:28:23.178 INFO:tasks.workunit.client.0.smithi123.stdout:5/826: rmdir dd/d19/d5f/d99 39 2024-03-30T04:28:23.180 INFO:tasks.workunit.client.0.smithi123.stdout:6/931: symlink d10/d20/da2/l13a 0 2024-03-30T04:28:23.181 INFO:tasks.workunit.client.0.smithi123.stdout:5/827: read - dd/d19/d3e/f103 zero size 2024-03-30T04:28:23.187 INFO:tasks.workunit.client.0.smithi123.stdout:7/846: symlink d2/da/d11/d34/d85/d8e/l11d 0 2024-03-30T04:28:23.187 INFO:tasks.workunit.client.0.smithi123.stdout:6/932: fdatasync d10/d82/dc1/d45/fe5 0 2024-03-30T04:28:23.187 INFO:tasks.workunit.client.0.smithi123.stdout:5/828: symlink dd/d19/d1f/l113 0 2024-03-30T04:28:23.188 INFO:tasks.workunit.client.0.smithi123.stdout:0/875: dread d6/dd/d15/d28/fb6 [4194304,4194304] 0 2024-03-30T04:28:23.198 INFO:tasks.workunit.client.0.smithi123.stdout:8/901: write da/d27/d3c/d2d/f4c [516009,49764] 0 2024-03-30T04:28:23.201 INFO:tasks.workunit.client.0.smithi123.stdout:7/847: link d2/da/d22/d3b/c79 d2/da/d11/d34/d76/c11e 0 2024-03-30T04:28:23.202 INFO:tasks.workunit.client.0.smithi123.stdout:6/933: write d10/f26 [1248187,112179] 0 2024-03-30T04:28:23.207 INFO:tasks.workunit.client.0.smithi123.stdout:0/876: rename d6/dd/d15/d98/d4e/d69/d6e/f87 to d6/dd/d31/d35/d114/f118 0 2024-03-30T04:28:23.210 INFO:tasks.workunit.client.0.smithi123.stdout:8/902: read da/dc/d14/f6e [514448,44266] 0 2024-03-30T04:28:23.213 INFO:tasks.workunit.client.0.smithi123.stdout:1/909: dwrite db/dd/d5f/d6d/d7e/ff7 [0,4194304] 0 2024-03-30T04:28:23.216 INFO:tasks.workunit.client.0.smithi123.stdout:1/910: stat db/dd/d5f/d6d/d85 0 2024-03-30T04:28:23.217 INFO:tasks.workunit.client.1.smithi135.stdout:3/466: sync 2024-03-30T04:28:23.219 INFO:tasks.workunit.client.0.smithi123.stdout:6/934: truncate d10/d82/d8e/f131 3731753 0 2024-03-30T04:28:23.219 INFO:tasks.workunit.client.1.smithi135.stdout:3/467: truncate d3/d10/f2d 1423663 0 2024-03-30T04:28:23.220 INFO:tasks.workunit.client.0.smithi123.stdout:9/853: dread f1 [0,4194304] 0 2024-03-30T04:28:23.220 INFO:tasks.workunit.client.0.smithi123.stdout:7/848: read d2/da/d22/d3b/f4b [3706867,120710] 0 2024-03-30T04:28:23.223 INFO:tasks.workunit.client.0.smithi123.stdout:5/829: dwrite fb [0,4194304] 0 2024-03-30T04:28:23.226 INFO:tasks.workunit.client.1.smithi135.stdout:3/468: dread d3/d10/f2d [0,4194304] 0 2024-03-30T04:28:23.227 INFO:tasks.workunit.client.0.smithi123.stdout:8/903: creat da/d27/d56/d6c/d6d/f11f x:0 0 0 2024-03-30T04:28:23.229 INFO:tasks.workunit.client.0.smithi123.stdout:1/911: stat db/dd/d42/d46/d5d/d62/d9e/cbb 0 2024-03-30T04:28:23.230 INFO:tasks.workunit.client.1.smithi135.stdout:3/469: rename l1 to d3/d10/la2 0 2024-03-30T04:28:23.232 INFO:tasks.workunit.client.1.smithi135.stdout:3/470: dread - d3/d50/d46/f59 zero size 2024-03-30T04:28:23.234 INFO:tasks.workunit.client.0.smithi123.stdout:5/830: chown dd/d19/d1f/d3c/d72/fb7 193224914 1 2024-03-30T04:28:23.235 INFO:tasks.workunit.client.0.smithi123.stdout:8/904: fdatasync da/ff 0 2024-03-30T04:28:23.236 INFO:tasks.workunit.client.1.smithi135.stdout:3/471: truncate d3/db/f19 1343944 0 2024-03-30T04:28:23.240 INFO:tasks.workunit.client.1.smithi135.stdout:3/472: mknod d3/d10/d9c/ca3 0 2024-03-30T04:28:23.241 INFO:tasks.workunit.client.0.smithi123.stdout:6/935: rename d10/d82/dc1/d6a/d7a/f100 to d10/d82/dc1/d6a/d7a/d99/f13b 0 2024-03-30T04:28:23.242 INFO:tasks.workunit.client.1.smithi135.stdout:3/473: mknod d3/d10/d16/ca4 0 2024-03-30T04:28:23.246 INFO:tasks.workunit.client.1.smithi135.stdout:3/474: creat d3/d10/d16/fa5 x:0 0 0 2024-03-30T04:28:23.247 INFO:tasks.workunit.client.0.smithi123.stdout:1/912: truncate db/dd/d42/d46/d51/d72/fa6 4888477 0 2024-03-30T04:28:23.248 INFO:tasks.workunit.client.0.smithi123.stdout:6/936: dread d10/d82/d8e/dc7/fed [0,4194304] 0 2024-03-30T04:28:23.252 INFO:tasks.workunit.client.0.smithi123.stdout:1/913: chown db/dd/d42/d46/d5d/d62/d9e/cbb 32571914 1 2024-03-30T04:28:23.256 INFO:tasks.workunit.client.0.smithi123.stdout:7/849: dwrite d2/da/d11/d61/ff7 [0,4194304] 0 2024-03-30T04:28:23.257 INFO:tasks.workunit.client.0.smithi123.stdout:6/937: symlink d10/d82/df9/l13c 0 2024-03-30T04:28:23.260 INFO:tasks.workunit.client.1.smithi135.stdout:3/475: dwrite d3/d50/d22/f72 [0,4194304] 0 2024-03-30T04:28:23.266 INFO:tasks.workunit.client.0.smithi123.stdout:0/877: dread d6/dd/d15/fb3 [0,4194304] 0 2024-03-30T04:28:23.269 INFO:tasks.workunit.client.1.smithi135.stdout:3/476: getdents d3/d10 0 2024-03-30T04:28:23.270 INFO:tasks.workunit.client.0.smithi123.stdout:1/914: mkdir db/dd/d42/d46/d56/daf/de4/d12d 0 2024-03-30T04:28:23.274 INFO:tasks.workunit.client.0.smithi123.stdout:5/831: dread dd/f1d [0,4194304] 0 2024-03-30T04:28:23.274 INFO:tasks.workunit.client.1.smithi135.stdout:3/477: getdents d3/d10/d16/d23/d6a/d86 0 2024-03-30T04:28:23.276 INFO:tasks.workunit.client.1.smithi135.stdout:0/577: sync 2024-03-30T04:28:23.277 INFO:tasks.workunit.client.1.smithi135.stdout:0/578: stat d0/d1/f43 0 2024-03-30T04:28:23.281 INFO:tasks.workunit.client.1.smithi135.stdout:0/579: rename d0/d5/d9/l71 to d0/d5/d9/d23/d2a/d60/lbb 0 2024-03-30T04:28:23.282 INFO:tasks.workunit.client.0.smithi123.stdout:0/878: rename d6/dd/d31/d35/f42 to d6/dd/d15/d98/d5d/d7f/df3/d104/f119 0 2024-03-30T04:28:23.283 INFO:tasks.workunit.client.1.smithi135.stdout:0/580: fdatasync d0/f12 0 2024-03-30T04:28:23.289 INFO:tasks.workunit.client.1.smithi135.stdout:0/581: unlink d0/d5/d9/d23/d5d/f69 0 2024-03-30T04:28:23.294 INFO:tasks.workunit.client.0.smithi123.stdout:1/915: mknod db/dd/d42/d46/d56/c12e 0 2024-03-30T04:28:23.295 INFO:tasks.workunit.client.1.smithi135.stdout:0/582: creat d0/d5/d9/d23/d58/d3b/fbc x:0 0 0 2024-03-30T04:28:23.295 INFO:tasks.workunit.client.1.smithi135.stdout:3/478: dread d3/d54/d5c/f64 [0,4194304] 0 2024-03-30T04:28:23.295 INFO:tasks.workunit.client.1.smithi135.stdout:3/479: read d3/db/fe [3029044,14719] 0 2024-03-30T04:28:23.296 INFO:tasks.workunit.client.1.smithi135.stdout:0/583: symlink d0/d13/d53/lbd 0 2024-03-30T04:28:23.297 INFO:tasks.workunit.client.0.smithi123.stdout:6/938: dwrite d10/d82/d8e/dc7/f2a [0,4194304] 0 2024-03-30T04:28:23.298 INFO:tasks.workunit.client.1.smithi135.stdout:0/584: chown d0/d13/d1e/d3a/f5a 40779304 1 2024-03-30T04:28:23.299 INFO:tasks.workunit.client.0.smithi123.stdout:5/832: rmdir dd/d19/d1f/d35/d3a/d57/db5/d110 0 2024-03-30T04:28:23.300 INFO:tasks.workunit.client.1.smithi135.stdout:3/480: rename d3/d10/d16/f1b to d3/fa6 0 2024-03-30T04:28:23.301 INFO:tasks.workunit.client.1.smithi135.stdout:0/585: mknod d0/d5/d9/cbe 0 2024-03-30T04:28:23.304 INFO:tasks.workunit.client.1.smithi135.stdout:0/586: symlink d0/d5/d42/d87/lbf 0 2024-03-30T04:28:23.306 INFO:tasks.workunit.client.1.smithi135.stdout:0/587: write d0/d19/d2d/f98 [2545656,93608] 0 2024-03-30T04:28:23.307 INFO:tasks.workunit.client.0.smithi123.stdout:6/939: mkdir d10/d82/d8e/dc7/d5b/d13d 0 2024-03-30T04:28:23.309 INFO:tasks.workunit.client.0.smithi123.stdout:5/833: mkdir dd/d19/d1f/d35/d114 0 2024-03-30T04:28:23.309 INFO:tasks.workunit.client.1.smithi135.stdout:0/588: chown d0/d5/d9/f17 305892 1 2024-03-30T04:28:23.313 INFO:tasks.workunit.client.0.smithi123.stdout:7/850: dread d2/da/d11/d34/d85/fa9 [0,4194304] 0 2024-03-30T04:28:23.313 INFO:tasks.workunit.client.0.smithi123.stdout:5/834: write dd/d19/d1f/d3c/f92 [4514591,70011] 0 2024-03-30T04:28:23.314 INFO:tasks.workunit.client.0.smithi123.stdout:0/879: dwrite d6/dd/d15/d28/d4b/dad/de8/f106 [0,4194304] 0 2024-03-30T04:28:23.317 INFO:tasks.workunit.client.0.smithi123.stdout:6/940: symlink d10/d82/d8e/dc7/d5b/d103/d115/d119/l13e 0 2024-03-30T04:28:23.319 INFO:tasks.workunit.client.1.smithi135.stdout:1/452: sync 2024-03-30T04:28:23.319 INFO:tasks.workunit.client.1.smithi135.stdout:8/445: sync 2024-03-30T04:28:23.320 INFO:tasks.workunit.client.1.smithi135.stdout:9/426: sync 2024-03-30T04:28:23.320 INFO:tasks.workunit.client.1.smithi135.stdout:4/443: sync 2024-03-30T04:28:23.320 INFO:tasks.workunit.client.1.smithi135.stdout:6/477: sync 2024-03-30T04:28:23.320 INFO:tasks.workunit.client.1.smithi135.stdout:5/435: sync 2024-03-30T04:28:23.320 INFO:tasks.workunit.client.1.smithi135.stdout:2/374: sync 2024-03-30T04:28:23.320 INFO:tasks.workunit.client.1.smithi135.stdout:7/502: sync 2024-03-30T04:28:23.323 INFO:tasks.workunit.client.1.smithi135.stdout:4/444: chown d10/d1a/l29 55 1 2024-03-30T04:28:23.323 INFO:tasks.workunit.client.1.smithi135.stdout:2/375: dread - d4/d6/d47/f5a zero size 2024-03-30T04:28:23.327 INFO:tasks.workunit.client.1.smithi135.stdout:2/376: chown d4/d32/d2e/f4b 29 1 2024-03-30T04:28:23.327 INFO:tasks.workunit.client.1.smithi135.stdout:5/436: read d1/d7/d11/f30 [452772,28248] 0 2024-03-30T04:28:23.328 INFO:tasks.workunit.client.1.smithi135.stdout:1/453: stat d16/d29/d32/f58 0 2024-03-30T04:28:23.328 INFO:tasks.workunit.client.1.smithi135.stdout:9/427: chown d7/d41/d72/f8e 3117 1 2024-03-30T04:28:23.331 INFO:tasks.workunit.client.0.smithi123.stdout:1/916: write db/dd/d42/d8f/f73 [1324894,81334] 0 2024-03-30T04:28:23.332 INFO:tasks.workunit.client.0.smithi123.stdout:0/880: read d6/dd/d15/d28/fb6 [3364338,68206] 0 2024-03-30T04:28:23.333 INFO:tasks.workunit.client.0.smithi123.stdout:1/917: chown db/dd/d42/d46/d5d 6039314 1 2024-03-30T04:28:23.338 INFO:tasks.workunit.client.1.smithi135.stdout:4/445: rmdir d10/d1a/d42/d46 39 2024-03-30T04:28:23.339 INFO:tasks.workunit.client.1.smithi135.stdout:8/446: creat d6/d23/f82 x:0 0 0 2024-03-30T04:28:23.339 INFO:tasks.workunit.client.1.smithi135.stdout:9/428: write d7/d1b/f7c [1837778,38094] 0 2024-03-30T04:28:23.340 INFO:tasks.workunit.client.1.smithi135.stdout:5/437: fsync d1/da/d17/f3d 0 2024-03-30T04:28:23.344 INFO:tasks.workunit.client.1.smithi135.stdout:5/438: fdatasync d1/f4f 0 2024-03-30T04:28:23.350 INFO:tasks.workunit.client.1.smithi135.stdout:3/481: dread d3/d50/d22/d33/f61 [0,4194304] 0 2024-03-30T04:28:23.351 INFO:tasks.workunit.client.0.smithi123.stdout:5/835: creat dd/d19/d5f/dc2/de8/f115 x:0 0 0 2024-03-30T04:28:23.351 INFO:tasks.workunit.client.0.smithi123.stdout:0/881: fsync d6/dd/d15/d98/d4e/d69/f81 0 2024-03-30T04:28:23.352 INFO:tasks.workunit.client.0.smithi123.stdout:1/918: mknod db/dd/d5f/d6d/d85/dd4/c12f 0 2024-03-30T04:28:23.354 INFO:tasks.workunit.client.1.smithi135.stdout:8/447: dread d6/d8/f39 [0,4194304] 0 2024-03-30T04:28:23.354 INFO:tasks.workunit.client.0.smithi123.stdout:5/836: write dd/d19/d1f/d68/fc6 [353815,43448] 0 2024-03-30T04:28:23.354 INFO:tasks.workunit.client.0.smithi123.stdout:0/882: write d6/fcd [622631,38207] 0 2024-03-30T04:28:23.356 INFO:tasks.workunit.client.1.smithi135.stdout:6/478: dwrite d2/d2e/f4c [0,4194304] 0 2024-03-30T04:28:23.357 INFO:tasks.workunit.client.0.smithi123.stdout:6/941: link d10/d82/dc1/d6a/d7a/d9f/fd4 d10/d82/dc1/d45/d111/f13f 0 2024-03-30T04:28:23.358 INFO:tasks.workunit.client.1.smithi135.stdout:6/479: chown d2/d36/c6f 389 1 2024-03-30T04:28:23.359 INFO:tasks.workunit.client.0.smithi123.stdout:5/837: mknod dd/d19/d91/c116 0 2024-03-30T04:28:23.359 INFO:tasks.workunit.client.1.smithi135.stdout:6/480: fsync d2/f4d 0 2024-03-30T04:28:23.360 INFO:tasks.workunit.client.0.smithi123.stdout:0/883: stat d6/dd/d15/d28/dd3/ddd/fec 0 2024-03-30T04:28:23.361 INFO:tasks.workunit.client.1.smithi135.stdout:0/589: dread d0/d5/d9/f17 [0,4194304] 0 2024-03-30T04:28:23.364 INFO:tasks.workunit.client.0.smithi123.stdout:1/919: mknod db/dd/d5f/d6d/d85/d89/d90/d12c/c130 0 2024-03-30T04:28:23.364 INFO:tasks.workunit.client.0.smithi123.stdout:0/884: readlink d6/l44 0 2024-03-30T04:28:23.366 INFO:tasks.workunit.client.1.smithi135.stdout:1/454: creat d16/d5a/d6b/f9b x:0 0 0 2024-03-30T04:28:23.366 INFO:tasks.workunit.client.0.smithi123.stdout:0/885: mknod d6/dd/d18/c11a 0 2024-03-30T04:28:23.367 INFO:tasks.workunit.client.1.smithi135.stdout:5/439: rename d1/d3/d2b/c82 to d1/d3/d76/c86 0 2024-03-30T04:28:23.368 INFO:tasks.workunit.client.0.smithi123.stdout:1/920: mknod db/dd/d42/d46/d56/df6/d10c/c131 0 2024-03-30T04:28:23.369 INFO:tasks.workunit.client.1.smithi135.stdout:3/482: mknod d3/d10/d9c/ca7 0 2024-03-30T04:28:23.369 INFO:tasks.workunit.client.1.smithi135.stdout:0/590: symlink d0/d13/d53/d7f/d83/lc0 0 2024-03-30T04:28:23.371 INFO:tasks.workunit.client.1.smithi135.stdout:4/446: link d10/d4d/f4e d10/d1a/d32/f96 0 2024-03-30T04:28:23.372 INFO:tasks.workunit.client.1.smithi135.stdout:7/503: dread d4/f16 [0,4194304] 0 2024-03-30T04:28:23.375 INFO:tasks.workunit.client.1.smithi135.stdout:4/447: readlink d10/d1a/d2b/l56 0 2024-03-30T04:28:23.378 INFO:tasks.workunit.client.0.smithi123.stdout:5/838: dread dd/f15 [0,4194304] 0 2024-03-30T04:28:23.378 INFO:tasks.workunit.client.1.smithi135.stdout:2/377: getdents d4/da/d7a 0 2024-03-30T04:28:23.379 INFO:tasks.workunit.client.1.smithi135.stdout:7/504: write d4/db/d1a/d5c/d2c/d51/d56/fa4 [344287,79389] 0 2024-03-30T04:28:23.382 INFO:tasks.workunit.client.1.smithi135.stdout:4/448: readlink d10/d1a/d42/d46/l90 0 2024-03-30T04:28:23.382 INFO:tasks.workunit.client.1.smithi135.stdout:3/483: creat d3/d54/d5c/da1/fa8 x:0 0 0 2024-03-30T04:28:23.384 INFO:tasks.workunit.client.1.smithi135.stdout:1/455: rename d16/c2a to d16/d29/d32/d85/c9c 0 2024-03-30T04:28:23.384 INFO:tasks.workunit.client.0.smithi123.stdout:1/921: rename db/dd/d42/d46/d51/d72/fa6 to db/dd/d42/d46/d56/da2/f132 0 2024-03-30T04:28:23.386 INFO:tasks.workunit.client.0.smithi123.stdout:5/839: mkdir dd/d19/d1f/d3c/d9e/d117 0 2024-03-30T04:28:23.387 INFO:tasks.workunit.client.1.smithi135.stdout:9/429: dread d7/d1b/d37/f3a [0,4194304] 0 2024-03-30T04:28:23.389 INFO:tasks.workunit.client.0.smithi123.stdout:1/922: write db/dd/d42/d46/d5d/d62/d9e/fff [4028501,115894] 0 2024-03-30T04:28:23.390 INFO:tasks.workunit.client.0.smithi123.stdout:5/840: creat dd/d19/d1f/d3c/d9e/f118 x:0 0 0 2024-03-30T04:28:23.390 INFO:tasks.workunit.client.0.smithi123.stdout:7/851: truncate d2/da/d11/d61/d55/d59/f111 1647769 0 2024-03-30T04:28:23.392 INFO:tasks.workunit.client.1.smithi135.stdout:6/481: dwrite d2/dd/d14/d30/f67 [4194304,4194304] 0 2024-03-30T04:28:23.393 INFO:tasks.workunit.client.1.smithi135.stdout:1/456: truncate f3 115569 0 2024-03-30T04:28:23.394 INFO:tasks.workunit.client.1.smithi135.stdout:2/378: rename d4/d32/d2e/l4d to d4/d55/l7e 0 2024-03-30T04:28:23.397 INFO:tasks.workunit.client.0.smithi123.stdout:1/923: symlink db/dd/d42/d46/d56/da2/de0/l133 0 2024-03-30T04:28:23.397 INFO:tasks.workunit.client.0.smithi123.stdout:5/841: rename dd/d19/d3e/d5b/d62/dd3/cd8 to dd/d19/d1f/d3c/c119 0 2024-03-30T04:28:23.398 INFO:tasks.workunit.client.1.smithi135.stdout:7/505: mkdir d4/db/d1a/d4b/dac 0 2024-03-30T04:28:23.399 INFO:tasks.workunit.client.1.smithi135.stdout:2/379: fdatasync d4/d32/d2e/f48 0 2024-03-30T04:28:23.399 INFO:tasks.workunit.client.1.smithi135.stdout:1/457: chown d16/d29/d32/d85/l91 9 1 2024-03-30T04:28:23.401 INFO:tasks.workunit.client.0.smithi123.stdout:1/924: symlink db/l134 0 2024-03-30T04:28:23.403 INFO:tasks.workunit.client.0.smithi123.stdout:1/925: chown db/d1b/f1d 11591427 1 2024-03-30T04:28:23.405 INFO:tasks.workunit.client.0.smithi123.stdout:6/942: dwrite d10/d82/fa7 [0,4194304] 0 2024-03-30T04:28:23.406 INFO:tasks.workunit.client.1.smithi135.stdout:9/430: rename d7/d29/f3e to d7/d90/f94 0 2024-03-30T04:28:23.410 INFO:tasks.workunit.client.1.smithi135.stdout:1/458: mkdir d16/d29/d32/d9d 0 2024-03-30T04:28:23.411 INFO:tasks.workunit.client.0.smithi123.stdout:0/886: dwrite d6/dd/d18/d2f/f85 [0,4194304] 0 2024-03-30T04:28:23.411 INFO:tasks.workunit.client.0.smithi123.stdout:1/926: getdents db/dd/d42/d46/d51/d72/d108 0 2024-03-30T04:28:23.416 INFO:tasks.workunit.client.1.smithi135.stdout:0/591: truncate d0/d5/d9/f17 3496874 0 2024-03-30T04:28:23.417 INFO:tasks.workunit.client.1.smithi135.stdout:5/440: dread d1/d7/d1a/f3b [0,4194304] 0 2024-03-30T04:28:23.417 INFO:tasks.workunit.client.0.smithi123.stdout:6/943: chown d10/d82/d8e/dc7/d5b/le0 1618 1 2024-03-30T04:28:23.420 INFO:tasks.workunit.client.1.smithi135.stdout:9/431: creat d7/d4f/d89/f95 x:0 0 0 2024-03-30T04:28:23.422 INFO:tasks.workunit.client.0.smithi123.stdout:5/842: dwrite dd/d19/f85 [0,4194304] 0 2024-03-30T04:28:23.423 INFO:tasks.workunit.client.0.smithi123.stdout:6/944: chown d10/f1b 134456770 1 2024-03-30T04:28:23.423 INFO:tasks.workunit.client.1.smithi135.stdout:9/432: fdatasync d7/d41/f59 0 2024-03-30T04:28:23.424 INFO:tasks.workunit.client.1.smithi135.stdout:7/506: rename d4/db/d1a/d4b/l8a to d4/db/d1a/d5c/d5d/lad 0 2024-03-30T04:28:23.424 INFO:tasks.workunit.client.1.smithi135.stdout:6/482: dread d2/f11 [0,4194304] 0 2024-03-30T04:28:23.425 INFO:tasks.workunit.client.0.smithi123.stdout:0/887: creat d6/dd/d18/d2f/da3/f11b x:0 0 0 2024-03-30T04:28:23.425 INFO:tasks.workunit.client.0.smithi123.stdout:6/945: chown d10/d82/dc1/d45/d77/f7c 46533598 1 2024-03-30T04:28:23.428 INFO:tasks.workunit.client.0.smithi123.stdout:6/946: write d10/d82/dc1/d45/fe5 [511024,130551] 0 2024-03-30T04:28:23.429 INFO:tasks.workunit.client.0.smithi123.stdout:0/888: dread - d6/dd/d15/d28/d4b/dad/f100 zero size 2024-03-30T04:28:23.431 INFO:tasks.workunit.client.0.smithi123.stdout:6/947: readlink d10/d82/d8e/l98 0 2024-03-30T04:28:23.435 INFO:tasks.workunit.client.1.smithi135.stdout:0/592: chown d0/d5/d9/d23/d58/d3b/d52/c84 1 1 2024-03-30T04:28:23.435 INFO:tasks.workunit.client.1.smithi135.stdout:6/483: write d2/d68/f83 [3526381,89177] 0 2024-03-30T04:28:23.436 INFO:tasks.workunit.client.1.smithi135.stdout:2/380: getdents d4/d32/d43 0 2024-03-30T04:28:23.436 INFO:tasks.workunit.client.0.smithi123.stdout:5/843: rename dd/d19/d1f/d3c/d72/d8c/deb to dd/d19/d1f/d35/db8/d11a 0 2024-03-30T04:28:23.438 INFO:tasks.workunit.client.1.smithi135.stdout:7/507: write d4/db/d1a/d5c/d2c/f63 [1417169,84147] 0 2024-03-30T04:28:23.438 INFO:tasks.workunit.client.0.smithi123.stdout:1/927: write db/dd/d42/d46/d39/f8a [707555,116875] 0 2024-03-30T04:28:23.440 INFO:tasks.workunit.client.0.smithi123.stdout:5/844: dread - dd/d19/d1f/d3c/d9e/f118 zero size 2024-03-30T04:28:23.441 INFO:tasks.workunit.client.1.smithi135.stdout:6/484: stat d2/dd/c20 0 2024-03-30T04:28:23.442 INFO:tasks.workunit.client.1.smithi135.stdout:9/433: mknod d7/d90/c96 0 2024-03-30T04:28:23.442 INFO:tasks.workunit.client.1.smithi135.stdout:4/449: dread d10/d1a/d32/f96 [0,4194304] 0 2024-03-30T04:28:23.443 INFO:tasks.workunit.client.0.smithi123.stdout:1/928: creat db/dd/d5f/dae/f135 x:0 0 0 2024-03-30T04:28:23.444 INFO:tasks.workunit.client.0.smithi123.stdout:5/845: mkdir dd/d19/d5f/dc2/d11b 0 2024-03-30T04:28:23.446 INFO:tasks.workunit.client.1.smithi135.stdout:5/441: link d1/d7/f10 d1/d7/d11/f87 0 2024-03-30T04:28:23.446 INFO:tasks.workunit.client.0.smithi123.stdout:6/948: write d10/d82/d8e/dc7/f2c [2283096,60234] 0 2024-03-30T04:28:23.449 INFO:tasks.workunit.client.1.smithi135.stdout:3/484: dwrite d3/d50/d46/f65 [0,4194304] 0 2024-03-30T04:28:23.449 INFO:tasks.workunit.client.1.smithi135.stdout:0/593: dread - d0/d5/d9/d23/d8a/fa9 zero size 2024-03-30T04:28:23.450 INFO:tasks.workunit.client.0.smithi123.stdout:6/949: symlink d10/l140 0 2024-03-30T04:28:23.452 INFO:tasks.workunit.client.0.smithi123.stdout:1/929: getdents db/dd/d42/d46/d56/daf 0 2024-03-30T04:28:23.452 INFO:tasks.workunit.client.1.smithi135.stdout:6/485: mkdir d2/dd/d40/d52/d95 0 2024-03-30T04:28:23.453 INFO:tasks.workunit.client.0.smithi123.stdout:6/950: dread - d10/d82/df9/f12d zero size 2024-03-30T04:28:23.453 INFO:tasks.workunit.client.0.smithi123.stdout:5/846: getdents dd/d19/d1f/d35/d3a/d57/de0 0 2024-03-30T04:28:23.458 INFO:tasks.workunit.client.0.smithi123.stdout:0/889: dread d6/dd/d15/fc3 [0,4194304] 0 2024-03-30T04:28:23.460 INFO:tasks.workunit.client.1.smithi135.stdout:9/434: creat d7/d4f/d5c/d63/d86/f97 x:0 0 0 2024-03-30T04:28:23.460 INFO:tasks.workunit.client.1.smithi135.stdout:0/594: write d0/d5/d10/f31 [1045060,121045] 0 2024-03-30T04:28:23.461 INFO:tasks.workunit.client.1.smithi135.stdout:3/485: mkdir d3/db/da9 0 2024-03-30T04:28:23.461 INFO:tasks.workunit.client.0.smithi123.stdout:1/930: dread - db/dd/d5f/d6d/d85/d89/d90/d94/d9c/dfc/f11a zero size 2024-03-30T04:28:23.462 INFO:tasks.workunit.client.1.smithi135.stdout:6/486: chown d2/dd/d14/c66 20849382 1 2024-03-30T04:28:23.464 INFO:tasks.workunit.client.0.smithi123.stdout:0/890: symlink d6/dd/d15/d98/d4e/d9a/db2/l11c 0 2024-03-30T04:28:23.465 INFO:tasks.workunit.client.1.smithi135.stdout:4/450: creat d10/d4d/f97 x:0 0 0 2024-03-30T04:28:23.466 INFO:tasks.workunit.client.0.smithi123.stdout:1/931: fsync db/dd/d5f/d6d/d85/dd4/fea 0 2024-03-30T04:28:23.468 INFO:tasks.workunit.client.0.smithi123.stdout:5/847: dread dd/d19/d1f/d3c/f97 [0,4194304] 0 2024-03-30T04:28:23.470 INFO:tasks.workunit.client.0.smithi123.stdout:5/848: chown dd/d19/d5f/dc2/ff0 1 1 2024-03-30T04:28:23.471 INFO:tasks.workunit.client.1.smithi135.stdout:2/381: dread d4/d6/f61 [0,4194304] 0 2024-03-30T04:28:23.471 INFO:tasks.workunit.client.1.smithi135.stdout:9/435: mkdir d7/d4f/d6e/d98 0 2024-03-30T04:28:23.471 INFO:tasks.workunit.client.0.smithi123.stdout:1/932: creat db/dd/d42/d46/d56/da2/df1/f136 x:0 0 0 2024-03-30T04:28:23.473 INFO:tasks.workunit.client.0.smithi123.stdout:5/849: chown dd/d19/d1f/d3c/l52 1277 1 2024-03-30T04:28:23.473 INFO:tasks.workunit.client.0.smithi123.stdout:1/933: stat db/dd/d42/d46/d51/dd7/f123 0 2024-03-30T04:28:23.478 INFO:tasks.workunit.client.0.smithi123.stdout:5/850: rmdir dd/d19/d1f/d3c/d72/d8c 39 2024-03-30T04:28:23.479 INFO:tasks.workunit.client.1.smithi135.stdout:4/451: fsync d10/f59 0 2024-03-30T04:28:23.482 INFO:tasks.workunit.client.1.smithi135.stdout:2/382: rmdir d4/d6/d47 39 2024-03-30T04:28:23.482 INFO:tasks.workunit.client.0.smithi123.stdout:5/851: mknod dd/d19/d1f/d3c/d72/d8c/c11c 0 2024-03-30T04:28:23.484 INFO:tasks.workunit.client.0.smithi123.stdout:6/951: truncate d10/d82/d8e/dc7/f1e 3218428 0 2024-03-30T04:28:23.485 INFO:tasks.workunit.client.1.smithi135.stdout:9/436: truncate d7/d55/f5b 635722 0 2024-03-30T04:28:23.486 INFO:tasks.workunit.client.0.smithi123.stdout:0/891: write d6/dd/f4a [204878,724] 0 2024-03-30T04:28:23.486 INFO:tasks.workunit.client.0.smithi123.stdout:1/934: dread db/dd/f8e [0,4194304] 0 2024-03-30T04:28:23.490 INFO:tasks.workunit.client.1.smithi135.stdout:6/487: symlink d2/dd/l96 0 2024-03-30T04:28:23.492 INFO:tasks.workunit.client.0.smithi123.stdout:1/935: creat db/dd/d42/d46/d5d/d62/f137 x:0 0 0 2024-03-30T04:28:23.494 INFO:tasks.workunit.client.0.smithi123.stdout:0/892: mkdir d6/dd/d15/d28/dd3/d11d 0 2024-03-30T04:28:23.494 INFO:tasks.workunit.client.1.smithi135.stdout:7/508: dread d4/db/de/f28 [0,4194304] 0 2024-03-30T04:28:23.495 INFO:tasks.workunit.client.1.smithi135.stdout:4/452: creat d10/d1a/d42/d46/f98 x:0 0 0 2024-03-30T04:28:23.497 INFO:tasks.workunit.client.1.smithi135.stdout:0/595: creat d0/d5/d9/fc1 x:0 0 0 2024-03-30T04:28:23.500 INFO:tasks.workunit.client.0.smithi123.stdout:0/893: mknod d6/dd/d15/d98/d4e/d69/d6e/dd8/c11e 0 2024-03-30T04:28:23.500 INFO:tasks.workunit.client.1.smithi135.stdout:7/509: read d4/db/d1a/d3d/f5b [1732515,25644] 0 2024-03-30T04:28:23.501 INFO:tasks.workunit.client.1.smithi135.stdout:9/437: rename d7/d1b/d37/l4d to d7/d90/l99 0 2024-03-30T04:28:23.501 INFO:tasks.workunit.client.1.smithi135.stdout:3/486: dwrite d3/d50/d46/f65 [4194304,4194304] 0 2024-03-30T04:28:23.503 INFO:tasks.workunit.client.0.smithi123.stdout:1/936: write db/dd/d42/d46/d5d/d62/d9e/fc3 [1332220,40489] 0 2024-03-30T04:28:23.505 INFO:tasks.workunit.client.1.smithi135.stdout:3/487: fdatasync d3/d50/d22/f87 0 2024-03-30T04:28:23.507 INFO:tasks.workunit.client.0.smithi123.stdout:5/852: dread dd/f16 [0,4194304] 0 2024-03-30T04:28:23.509 INFO:tasks.workunit.client.0.smithi123.stdout:6/952: dwrite d10/f29 [0,4194304] 0 2024-03-30T04:28:23.511 INFO:tasks.workunit.client.0.smithi123.stdout:1/937: creat db/dd/d5f/d6d/d85/d89/d90/d94/d9c/f138 x:0 0 0 2024-03-30T04:28:23.511 INFO:tasks.workunit.client.1.smithi135.stdout:2/383: unlink d4/d6/d47/l5d 0 2024-03-30T04:28:23.511 INFO:tasks.workunit.client.0.smithi123.stdout:6/953: fsync f6 0 2024-03-30T04:28:23.512 INFO:tasks.workunit.client.1.smithi135.stdout:3/488: stat d3/d50/fa 0 2024-03-30T04:28:23.512 INFO:tasks.workunit.client.0.smithi123.stdout:6/954: chown d10/d82/dc1/d45 16485491 1 2024-03-30T04:28:23.513 INFO:tasks.workunit.client.0.smithi123.stdout:6/955: chown d10/d20/d3f 8 1 2024-03-30T04:28:23.514 INFO:tasks.workunit.client.0.smithi123.stdout:1/938: write db/dd/d42/d46/d51/f55 [4464756,84588] 0 2024-03-30T04:28:23.516 INFO:tasks.workunit.client.1.smithi135.stdout:3/489: stat d3/d50/d22/d8b/d91/f97 0 2024-03-30T04:28:23.517 INFO:tasks.workunit.client.0.smithi123.stdout:5/853: symlink dd/d19/d1f/d35/d114/l11d 0 2024-03-30T04:28:23.517 INFO:tasks.workunit.client.0.smithi123.stdout:0/894: write d6/dd/d15/f8e [1869861,68667] 0 2024-03-30T04:28:23.518 INFO:tasks.workunit.client.0.smithi123.stdout:0/895: stat d6/c8 0 2024-03-30T04:28:23.518 INFO:tasks.workunit.client.0.smithi123.stdout:5/854: chown dd/d19/d1f/d3c/d72/d8c 480 1 2024-03-30T04:28:23.518 INFO:tasks.workunit.client.1.smithi135.stdout:7/510: mkdir d4/db/de/d5a/dae 0 2024-03-30T04:28:23.522 INFO:tasks.workunit.client.0.smithi123.stdout:6/956: getdents d10/d82/d8e/dc7/d5b/d103/d115 0 2024-03-30T04:28:23.524 INFO:tasks.workunit.client.0.smithi123.stdout:6/957: chown d10/d20/f40 128264 1 2024-03-30T04:28:23.524 INFO:tasks.workunit.client.0.smithi123.stdout:5/855: creat dd/d19/d1f/d3c/d72/d8c/f11e x:0 0 0 2024-03-30T04:28:23.524 INFO:tasks.workunit.client.0.smithi123.stdout:0/896: truncate d6/f24 4948112 0 2024-03-30T04:28:23.526 INFO:tasks.workunit.client.1.smithi135.stdout:5/442: dwrite d1/d3/f49 [0,4194304] 0 2024-03-30T04:28:23.527 INFO:tasks.workunit.client.0.smithi123.stdout:5/856: mkdir dd/d19/d1f/d35/d3a/d11f 0 2024-03-30T04:28:23.528 INFO:tasks.workunit.client.1.smithi135.stdout:6/488: link d2/dd/d14/f3a d2/dd/d14/d2b/d86/d8e/f97 0 2024-03-30T04:28:23.529 INFO:tasks.workunit.client.0.smithi123.stdout:5/857: chown dd/d19/d3e/l44 116 1 2024-03-30T04:28:23.529 INFO:tasks.workunit.client.0.smithi123.stdout:0/897: rename d6/l84 to d6/dd/d15/d28/d4b/l11f 0 2024-03-30T04:28:23.530 INFO:tasks.workunit.client.0.smithi123.stdout:1/939: write db/dd/d5f/d6d/fd5 [648919,96464] 0 2024-03-30T04:28:23.531 INFO:tasks.workunit.client.0.smithi123.stdout:1/940: stat db/dd/d42 0 2024-03-30T04:28:23.532 INFO:tasks.workunit.client.1.smithi135.stdout:2/384: symlink d4/d32/d46/l7f 0 2024-03-30T04:28:23.533 INFO:tasks.workunit.client.1.smithi135.stdout:6/489: read d2/d36/f44 [3789155,108232] 0 2024-03-30T04:28:23.536 INFO:tasks.workunit.client.1.smithi135.stdout:0/596: dwrite d0/d19/d2d/d45/f76 [0,4194304] 0 2024-03-30T04:28:23.536 INFO:tasks.workunit.client.1.smithi135.stdout:2/385: chown d4/d32/d46/l7f 305471 1 2024-03-30T04:28:23.539 INFO:tasks.workunit.client.1.smithi135.stdout:3/490: symlink d3/d54/laa 0 2024-03-30T04:28:23.540 INFO:tasks.workunit.client.1.smithi135.stdout:2/386: dread - d4/d32/d2e/f4b zero size 2024-03-30T04:28:23.540 INFO:tasks.workunit.client.1.smithi135.stdout:4/453: write fd [4180277,83688] 0 2024-03-30T04:28:23.543 INFO:tasks.workunit.client.1.smithi135.stdout:2/387: chown d4/d32/f75 24071767 1 2024-03-30T04:28:23.548 INFO:tasks.workunit.client.1.smithi135.stdout:0/597: mkdir d0/d13/d1e/d3a/dc2 0 2024-03-30T04:28:23.549 INFO:tasks.workunit.client.0.smithi123.stdout:0/898: write d6/dd/d15/d28/d4b/d8a/ffa [944903,110638] 0 2024-03-30T04:28:23.549 INFO:tasks.workunit.client.0.smithi123.stdout:5/858: mkdir dd/d19/d1f/d35/d3a/d57/db3/d120 0 2024-03-30T04:28:23.553 INFO:tasks.workunit.client.0.smithi123.stdout:4/811: sync 2024-03-30T04:28:23.553 INFO:tasks.workunit.client.0.smithi123.stdout:1/941: dwrite db/dd/d42/d46/d56/da2/df1/f136 [0,4194304] 0 2024-03-30T04:28:23.553 INFO:tasks.workunit.client.0.smithi123.stdout:0/899: chown d6/dd/d15/d98/d4e/cfc 26060366 1 2024-03-30T04:28:23.558 INFO:tasks.workunit.client.0.smithi123.stdout:5/859: creat dd/d19/d5f/f121 x:0 0 0 2024-03-30T04:28:23.559 INFO:tasks.workunit.client.1.smithi135.stdout:4/454: rename d10/d1a/d2b/l56 to d10/d72/d85/l99 0 2024-03-30T04:28:23.561 INFO:tasks.workunit.client.0.smithi123.stdout:1/942: fsync db/dd/d42/d46/d56/d5c/f76 0 2024-03-30T04:28:23.561 INFO:tasks.workunit.client.0.smithi123.stdout:4/812: getdents d2/d4/d7/dc1/d108 0 2024-03-30T04:28:23.562 INFO:tasks.workunit.client.1.smithi135.stdout:5/443: rmdir d1/da/d17/d7a 0 2024-03-30T04:28:23.566 INFO:tasks.workunit.client.1.smithi135.stdout:6/490: link d2/dd/l96 d2/d2e/l98 0 2024-03-30T04:28:23.567 INFO:tasks.workunit.client.0.smithi123.stdout:1/943: truncate db/dd/d42/d46/d56/fb6 338526 0 2024-03-30T04:28:23.570 INFO:tasks.workunit.client.1.smithi135.stdout:5/444: write d1/f36 [1915463,46382] 0 2024-03-30T04:28:23.570 INFO:tasks.workunit.client.0.smithi123.stdout:4/813: link d2/d4/d2c/d42/dad/d91/lf2 d2/d4/d7/l114 0 2024-03-30T04:28:23.572 INFO:tasks.workunit.client.1.smithi135.stdout:6/491: write d2/dd/d18/f24 [1253597,117950] 0 2024-03-30T04:28:23.573 INFO:tasks.workunit.client.1.smithi135.stdout:5/445: creat d1/d45/f88 x:0 0 0 2024-03-30T04:28:23.573 INFO:tasks.workunit.client.1.smithi135.stdout:0/598: getdents d0/d13/d53 0 2024-03-30T04:28:23.575 INFO:tasks.workunit.client.1.smithi135.stdout:5/446: chown d1/d7/d1a/c2a 2039479861 1 2024-03-30T04:28:23.576 INFO:tasks.workunit.client.0.smithi123.stdout:1/944: creat db/dd/d42/d46/d56/daf/de4/d12d/f139 x:0 0 0 2024-03-30T04:28:23.576 INFO:tasks.workunit.client.1.smithi135.stdout:0/599: rmdir d0/d34 39 2024-03-30T04:28:23.577 INFO:tasks.workunit.client.1.smithi135.stdout:6/492: readlink d2/dd/l96 0 2024-03-30T04:28:23.577 INFO:tasks.workunit.client.1.smithi135.stdout:7/511: dread d4/db/d1a/d5c/d2c/f88 [0,4194304] 0 2024-03-30T04:28:23.581 INFO:tasks.workunit.client.0.smithi123.stdout:0/900: dwrite d6/dd/d15/d98/d4e/d69/d6e/f9f [4194304,4194304] 0 2024-03-30T04:28:23.582 INFO:tasks.workunit.client.1.smithi135.stdout:5/447: mknod d1/d7/d11/c89 0 2024-03-30T04:28:23.582 INFO:tasks.workunit.client.1.smithi135.stdout:7/512: fsync d4/db/de/f2b 0 2024-03-30T04:28:23.585 INFO:tasks.workunit.client.0.smithi123.stdout:5/860: dwrite dd/d19/d1f/d3c/d72/d8c/f10b [0,4194304] 0 2024-03-30T04:28:23.589 INFO:tasks.workunit.client.0.smithi123.stdout:1/945: truncate db/dd/d5f/d6d/d85/d89/fc9 322705 0 2024-03-30T04:28:23.589 INFO:tasks.workunit.client.0.smithi123.stdout:4/814: dread d2/f20 [0,4194304] 0 2024-03-30T04:28:23.590 INFO:tasks.workunit.client.1.smithi135.stdout:0/600: creat d0/d5/d9/d23/fc3 x:0 0 0 2024-03-30T04:28:23.590 INFO:tasks.workunit.client.0.smithi123.stdout:5/861: fsync dd/d19/d3e/d5b/f5e 0 2024-03-30T04:28:23.594 INFO:tasks.workunit.client.1.smithi135.stdout:4/455: dwrite d10/d1a/d42/d46/f98 [0,4194304] 0 2024-03-30T04:28:23.594 INFO:tasks.workunit.client.0.smithi123.stdout:1/946: fdatasync db/dd/d5f/f10a 0 2024-03-30T04:28:23.595 INFO:tasks.workunit.client.0.smithi123.stdout:0/901: mknod d6/dd/d15/d28/d4b/dad/df2/d112/c120 0 2024-03-30T04:28:23.597 INFO:tasks.workunit.client.0.smithi123.stdout:4/815: creat d2/f115 x:0 0 0 2024-03-30T04:28:23.599 INFO:tasks.workunit.client.0.smithi123.stdout:5/862: mkdir dd/d19/d1f/d68/dcc/df7/d122 0 2024-03-30T04:28:23.600 INFO:tasks.workunit.client.1.smithi135.stdout:5/448: creat d1/d7/d11/f8a x:0 0 0 2024-03-30T04:28:23.600 INFO:tasks.workunit.client.1.smithi135.stdout:7/513: creat d4/db/d79/d92/faf x:0 0 0 2024-03-30T04:28:23.601 INFO:tasks.workunit.client.0.smithi123.stdout:0/902: creat d6/dd/d18/d1f/f121 x:0 0 0 2024-03-30T04:28:23.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:23 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/329845434' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:28:23.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:23 smithi135 ceph-mon[44364]: from='client.14692 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:28:23.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:23 smithi135 ceph-mon[44364]: pgmap v45: 65 pgs: 65 active+clean; 2.4 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 71 MiB/s rd, 206 MiB/s wr, 422 op/s 2024-03-30T04:28:23.602 INFO:tasks.workunit.client.1.smithi135.stdout:7/514: dread - d4/db/fa0 zero size 2024-03-30T04:28:23.602 INFO:tasks.workunit.client.1.smithi135.stdout:5/449: write d1/f4f [5121207,17030] 0 2024-03-30T04:28:23.604 INFO:tasks.workunit.client.0.smithi123.stdout:5/863: unlink dd/d19/f2b 0 2024-03-30T04:28:23.606 INFO:tasks.workunit.client.0.smithi123.stdout:1/947: link db/dd/d42/d46/d56/daf/lca db/dd/d42/d46/d5d/d62/l13a 0 2024-03-30T04:28:23.610 INFO:tasks.workunit.client.1.smithi135.stdout:7/515: mkdir d4/db/de/db0 0 2024-03-30T04:28:23.611 INFO:tasks.workunit.client.0.smithi123.stdout:5/864: link dd/d19/d1f/d3c/d9e/f10f dd/f123 0 2024-03-30T04:28:23.615 INFO:tasks.workunit.client.0.smithi123.stdout:5/865: mkdir dd/d19/d5f/dc2/d124 0 2024-03-30T04:28:23.615 INFO:tasks.workunit.client.1.smithi135.stdout:2/388: dread d4/d32/d2e/f53 [0,4194304] 0 2024-03-30T04:28:23.620 INFO:tasks.workunit.client.0.smithi123.stdout:4/816: write d2/d4/d2c/d42/f8b [3184988,26522] 0 2024-03-30T04:28:23.623 INFO:tasks.workunit.client.0.smithi123.stdout:4/817: creat d2/d4/de0/f116 x:0 0 0 2024-03-30T04:28:23.625 INFO:tasks.workunit.client.0.smithi123.stdout:4/818: read - d2/d4/d7/ff4 zero size 2024-03-30T04:28:23.626 INFO:tasks.workunit.client.0.smithi123.stdout:4/819: chown d2/d34 9342788 1 2024-03-30T04:28:23.629 INFO:tasks.workunit.client.0.smithi123.stdout:5/866: write dd/d41/f43 [363329,94139] 0 2024-03-30T04:28:23.630 INFO:tasks.workunit.client.0.smithi123.stdout:4/820: write d2/d4/d2c/d42/d35/f5c [152509,95933] 0 2024-03-30T04:28:23.634 INFO:tasks.workunit.client.0.smithi123.stdout:4/821: symlink d2/d4/d2c/l117 0 2024-03-30T04:28:23.635 INFO:tasks.workunit.client.1.smithi135.stdout:0/601: dwrite d0/d13/d53/d7f/d83/fac [0,4194304] 0 2024-03-30T04:28:23.636 INFO:tasks.workunit.client.0.smithi123.stdout:5/867: mknod dd/d19/d5f/dc2/d124/c125 0 2024-03-30T04:28:23.637 INFO:tasks.workunit.client.1.smithi135.stdout:6/493: dwrite d2/dd/d14/f3a [0,4194304] 0 2024-03-30T04:28:23.638 INFO:tasks.workunit.client.0.smithi123.stdout:4/822: write d2/d4/d7/dc1/d113/fa6 [284791,99955] 0 2024-03-30T04:28:23.640 INFO:tasks.workunit.client.1.smithi135.stdout:4/456: dwrite d10/d1a/f21 [0,4194304] 0 2024-03-30T04:28:23.644 INFO:tasks.workunit.client.0.smithi123.stdout:4/823: truncate d2/d26/f70 1957171 0 2024-03-30T04:28:23.645 INFO:tasks.workunit.client.0.smithi123.stdout:4/824: chown d2/d34/f93 61108921 1 2024-03-30T04:28:23.649 INFO:tasks.workunit.client.1.smithi135.stdout:2/389: link d4/d32/d2e/d2f/f64 d4/d6/f80 0 2024-03-30T04:28:23.649 INFO:tasks.workunit.client.0.smithi123.stdout:4/825: creat d2/d4/d2c/d42/d35/d5d/d6f/df9/f118 x:0 0 0 2024-03-30T04:28:23.650 INFO:tasks.workunit.client.1.smithi135.stdout:4/457: mknod d10/d1a/d32/c9a 0 2024-03-30T04:28:23.655 INFO:tasks.workunit.client.1.smithi135.stdout:4/458: write d10/d1a/f3a [1478482,126703] 0 2024-03-30T04:28:23.656 INFO:tasks.workunit.client.1.smithi135.stdout:7/516: dwrite d4/db/d1a/d5c/d2c/d51/d56/fa4 [0,4194304] 0 2024-03-30T04:28:23.675 INFO:tasks.workunit.client.0.smithi123.stdout:4/826: write d2/d4/fa [441934,124332] 0 2024-03-30T04:28:23.676 INFO:tasks.workunit.client.1.smithi135.stdout:2/390: mknod d4/d32/d12/d45/d7d/c81 0 2024-03-30T04:28:23.679 INFO:tasks.workunit.client.1.smithi135.stdout:2/391: chown d4/f5 197491 1 2024-03-30T04:28:23.679 INFO:tasks.workunit.client.1.smithi135.stdout:6/494: dwrite d2/dd/d14/d2b/f50 [4194304,4194304] 0 2024-03-30T04:28:23.684 INFO:tasks.workunit.client.1.smithi135.stdout:0/602: dwrite d0/d13/d1e/fb2 [0,4194304] 0 2024-03-30T04:28:23.685 INFO:tasks.workunit.client.0.smithi123.stdout:1/948: dread db/dd/d42/d46/d5d/ffa [0,4194304] 0 2024-03-30T04:28:23.691 INFO:tasks.workunit.client.0.smithi123.stdout:4/827: dwrite d2/d4/fa [0,4194304] 0 2024-03-30T04:28:23.691 INFO:tasks.workunit.client.1.smithi135.stdout:0/603: truncate d0/d13/d1e/d3a/f61 4931042 0 2024-03-30T04:28:23.693 INFO:tasks.workunit.client.0.smithi123.stdout:4/828: write d2/f115 [697464,7559] 0 2024-03-30T04:28:23.694 INFO:tasks.workunit.client.0.smithi123.stdout:1/949: link db/dd/d42/d46/d51/d72/f120 db/dd/d42/d46/d56/daf/de4/d12d/f13b 0 2024-03-30T04:28:23.704 INFO:tasks.workunit.client.0.smithi123.stdout:1/950: mknod db/dd/d42/d46/d56/c13c 0 2024-03-30T04:28:23.714 INFO:tasks.workunit.client.1.smithi135.stdout:7/517: dwrite d4/db/de/d5a/f6c [0,4194304] 0 2024-03-30T04:28:23.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:23 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/329845434' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:28:23.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:23 smithi123 ceph-mon[35245]: from='client.14692 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:28:23.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:23 smithi123 ceph-mon[35245]: pgmap v45: 65 pgs: 65 active+clean; 2.4 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 71 MiB/s rd, 206 MiB/s wr, 422 op/s 2024-03-30T04:28:23.723 INFO:tasks.workunit.client.1.smithi135.stdout:7/518: mknod d4/db/d79/d92/cb1 0 2024-03-30T04:28:23.725 INFO:tasks.workunit.client.1.smithi135.stdout:2/392: dwrite d4/d32/d2e/f5c [0,4194304] 0 2024-03-30T04:28:23.726 INFO:tasks.workunit.client.1.smithi135.stdout:7/519: readlink d4/db/d1a/d5c/l95 0 2024-03-30T04:28:23.729 INFO:tasks.workunit.client.1.smithi135.stdout:2/393: fdatasync d4/d32/d43/f63 0 2024-03-30T04:28:23.731 INFO:tasks.workunit.client.1.smithi135.stdout:7/520: chown d4/db/d1a/d5c/d2c/d51/d56/d81/f7f 203 1 2024-03-30T04:28:23.737 INFO:tasks.workunit.client.1.smithi135.stdout:7/521: write d4/db/d1a/d5c/d2c/d51/d56/f9e [1004364,32594] 0 2024-03-30T04:28:23.741 INFO:tasks.workunit.client.1.smithi135.stdout:4/459: write d10/d1a/d42/f5e [586017,92747] 0 2024-03-30T04:28:23.742 INFO:tasks.workunit.client.0.smithi123.stdout:3/974: sync 2024-03-30T04:28:23.742 INFO:tasks.workunit.client.1.smithi135.stdout:4/460: readlink d10/d1a/l1d 0 2024-03-30T04:28:23.744 INFO:tasks.workunit.client.1.smithi135.stdout:7/522: mknod d4/db/d79/d92/cb2 0 2024-03-30T04:28:23.753 INFO:tasks.workunit.client.1.smithi135.stdout:7/523: symlink d4/db/d79/lb3 0 2024-03-30T04:28:23.755 INFO:tasks.workunit.client.0.smithi123.stdout:4/829: dwrite d2/d4/d2c/d42/f4a [0,4194304] 0 2024-03-30T04:28:23.756 INFO:tasks.workunit.client.0.smithi123.stdout:1/951: dwrite db/dd/d5f/d6d/d85/fe5 [0,4194304] 0 2024-03-30T04:28:23.757 INFO:tasks.workunit.client.1.smithi135.stdout:6/495: dwrite d2/d36/f44 [0,4194304] 0 2024-03-30T04:28:23.762 INFO:tasks.workunit.client.0.smithi123.stdout:4/830: mknod d2/dab/c119 0 2024-03-30T04:28:23.762 INFO:tasks.workunit.client.0.smithi123.stdout:3/975: write d13/f1b [986849,11965] 0 2024-03-30T04:28:23.763 INFO:tasks.workunit.client.0.smithi123.stdout:4/831: readlink d2/d4/d7/lea 0 2024-03-30T04:28:23.768 INFO:tasks.workunit.client.0.smithi123.stdout:3/976: read d13/d24/d39/d79/db9/fd2 [880945,107854] 0 2024-03-30T04:28:23.768 INFO:tasks.workunit.client.0.smithi123.stdout:4/832: chown d2/d4/d7/l114 1 1 2024-03-30T04:28:23.768 INFO:tasks.workunit.client.0.smithi123.stdout:1/952: mknod db/dd/d42/d8f/c13d 0 2024-03-30T04:28:23.774 INFO:tasks.workunit.client.1.smithi135.stdout:6/496: rename d2/d36/f8f to d2/f99 0 2024-03-30T04:28:23.775 INFO:tasks.workunit.client.0.smithi123.stdout:1/953: read - db/dd/d42/d46/f53 zero size 2024-03-30T04:28:23.777 INFO:tasks.workunit.client.0.smithi123.stdout:3/977: mknod d13/d24/d31/d23/d3a/d3c/db8/df4/d131/c13e 0 2024-03-30T04:28:23.777 INFO:tasks.workunit.client.1.smithi135.stdout:6/497: readlink d2/dd/d14/d38/l6a 0 2024-03-30T04:28:23.779 INFO:tasks.workunit.client.1.smithi135.stdout:4/461: dwrite d10/d72/d7c/f83 [0,4194304] 0 2024-03-30T04:28:23.782 INFO:tasks.workunit.client.1.smithi135.stdout:4/462: stat d10/c6d 0 2024-03-30T04:28:23.782 INFO:tasks.workunit.client.1.smithi135.stdout:7/524: dread d4/db/d1a/d4b/f31 [4194304,4194304] 0 2024-03-30T04:28:23.786 INFO:tasks.workunit.client.1.smithi135.stdout:6/498: unlink d2/f8 0 2024-03-30T04:28:23.786 INFO:tasks.workunit.client.0.smithi123.stdout:3/978: read d13/d24/d31/d23/f6c [963932,66609] 0 2024-03-30T04:28:23.790 INFO:tasks.workunit.client.0.smithi123.stdout:3/979: mknod d13/d24/d31/d30/c13f 0 2024-03-30T04:28:23.790 INFO:tasks.workunit.client.1.smithi135.stdout:4/463: creat d10/d72/d7c/f9b x:0 0 0 2024-03-30T04:28:23.790 INFO:tasks.workunit.client.1.smithi135.stdout:7/525: fsync d4/db/d1a/d4b/f31 0 2024-03-30T04:28:23.793 INFO:tasks.workunit.client.0.smithi123.stdout:3/980: rmdir d13/d24/d31/d23/d3a/d3c/db8/df4/d131 39 2024-03-30T04:28:23.794 INFO:tasks.workunit.client.1.smithi135.stdout:7/526: truncate d4/db/fa0 627291 0 2024-03-30T04:28:23.795 INFO:tasks.workunit.client.1.smithi135.stdout:6/499: rename d2/dd/d40/f82 to d2/dd/d14/d38/f9a 0 2024-03-30T04:28:23.798 INFO:tasks.workunit.client.1.smithi135.stdout:6/500: write d2/dd/d18/f73 [109355,62052] 0 2024-03-30T04:28:23.799 INFO:tasks.workunit.client.0.smithi123.stdout:3/981: symlink d13/d24/d31/d23/d3a/d3c/l140 0 2024-03-30T04:28:23.800 INFO:tasks.workunit.client.0.smithi123.stdout:3/982: dread - d13/f13c zero size 2024-03-30T04:28:23.805 INFO:tasks.workunit.client.1.smithi135.stdout:6/501: getdents d2/dd/d14/d30 0 2024-03-30T04:28:23.806 INFO:tasks.workunit.client.0.smithi123.stdout:3/983: rename d13/d24/d31/d123/d5d/da5/lda to d13/d24/d31/d123/dde/l141 0 2024-03-30T04:28:23.807 INFO:tasks.workunit.client.0.smithi123.stdout:4/833: dread d2/d4/d2c/d42/f8b [0,4194304] 0 2024-03-30T04:28:23.809 INFO:tasks.workunit.client.0.smithi123.stdout:4/834: fsync d2/d4/d2c/d42/d35/faa 0 2024-03-30T04:28:23.810 INFO:tasks.workunit.client.0.smithi123.stdout:3/984: creat d13/d24/d39/d79/dea/d107/f142 x:0 0 0 2024-03-30T04:28:23.810 INFO:tasks.workunit.client.0.smithi123.stdout:4/835: fdatasync d2/d4/d7/f4e 0 2024-03-30T04:28:23.810 INFO:tasks.workunit.client.1.smithi135.stdout:7/527: write d4/db/d1a/f66 [1586911,106791] 0 2024-03-30T04:28:23.812 INFO:tasks.workunit.client.0.smithi123.stdout:4/836: write d2/d4/fa [5149937,2644] 0 2024-03-30T04:28:23.814 INFO:tasks.workunit.client.0.smithi123.stdout:3/985: symlink d13/d24/d31/d123/d5d/da5/l143 0 2024-03-30T04:28:23.816 INFO:tasks.workunit.client.0.smithi123.stdout:4/837: creat d2/dfb/f11a x:0 0 0 2024-03-30T04:28:23.816 INFO:tasks.workunit.client.1.smithi135.stdout:4/464: dwrite d10/d1a/d32/d40/f76 [0,4194304] 0 2024-03-30T04:28:23.816 INFO:tasks.workunit.client.1.smithi135.stdout:7/528: mkdir d4/db/de/db0/db4 0 2024-03-30T04:28:23.818 INFO:tasks.workunit.client.0.smithi123.stdout:3/986: symlink d13/d24/d39/d79/d90/l144 0 2024-03-30T04:28:23.819 INFO:tasks.workunit.client.1.smithi135.stdout:7/529: write d4/db/d79/d92/faf [139902,70731] 0 2024-03-30T04:28:23.820 INFO:tasks.workunit.client.0.smithi123.stdout:3/987: write f11 [4142016,105514] 0 2024-03-30T04:28:23.821 INFO:tasks.workunit.client.0.smithi123.stdout:4/838: creat d2/d4/d2c/d42/d80/d96/f11b x:0 0 0 2024-03-30T04:28:23.823 INFO:tasks.workunit.client.1.smithi135.stdout:7/530: dread - d4/db/d1a/d5c/d82/fa2 zero size 2024-03-30T04:28:23.827 INFO:tasks.workunit.client.0.smithi123.stdout:4/839: chown d2/d4/d2c/d42/d35/d9b/da2/deb/l106 861 1 2024-03-30T04:28:23.828 INFO:tasks.workunit.client.1.smithi135.stdout:7/531: getdents d4/db/d1a/d5c 0 2024-03-30T04:28:23.830 INFO:tasks.workunit.client.0.smithi123.stdout:3/988: truncate d13/d24/d31/d23/d3a/d3c/fb6 957843 0 2024-03-30T04:28:23.830 INFO:tasks.workunit.client.0.smithi123.stdout:4/840: read d2/d4/d2c/d42/d35/d5d/d78/fef [497134,62780] 0 2024-03-30T04:28:23.832 INFO:tasks.workunit.client.1.smithi135.stdout:7/532: mkdir d4/db/d1a/d5c/d5d/db5 0 2024-03-30T04:28:23.833 INFO:tasks.workunit.client.0.smithi123.stdout:4/841: chown d2/d4/d2c/d42/d35/d9b/db7 1572 1 2024-03-30T04:28:23.835 INFO:tasks.workunit.client.1.smithi135.stdout:7/533: write d4/db/d1a/d5c/d2c/f63 [57742,82054] 0 2024-03-30T04:28:23.836 INFO:tasks.workunit.client.1.smithi135.stdout:6/502: dread d2/dd/d14/d2b/d86/d55/f74 [0,4194304] 0 2024-03-30T04:28:23.836 INFO:tasks.workunit.client.0.smithi123.stdout:4/842: read - d2/d4/d2c/d42/dad/d91/fb1 zero size 2024-03-30T04:28:23.839 INFO:tasks.workunit.client.0.smithi123.stdout:3/989: rmdir d13/d24/d31/d23 39 2024-03-30T04:28:23.839 INFO:tasks.workunit.client.1.smithi135.stdout:6/503: write d2/dd/d14/d38/f8b [355567,128404] 0 2024-03-30T04:28:23.840 INFO:tasks.workunit.client.1.smithi135.stdout:7/534: getdents d4/db/d1a/d5c/d2c/d83 0 2024-03-30T04:28:23.841 INFO:tasks.workunit.client.1.smithi135.stdout:6/504: readlink d2/dd/d14/d53/l5b 0 2024-03-30T04:28:23.842 INFO:tasks.workunit.client.1.smithi135.stdout:7/535: chown d4/db/d1a/d4b/d40/l67 0 1 2024-03-30T04:28:23.844 INFO:tasks.workunit.client.1.smithi135.stdout:6/505: write d2/d2e/f37 [3002560,44923] 0 2024-03-30T04:28:23.846 INFO:tasks.workunit.client.0.smithi123.stdout:4/843: truncate d2/d4/d7/fe 4982180 0 2024-03-30T04:28:23.847 INFO:tasks.workunit.client.1.smithi135.stdout:6/506: stat d2/dd/c2a 0 2024-03-30T04:28:23.847 INFO:tasks.workunit.client.1.smithi135.stdout:7/536: creat d4/db/d1a/d5c/d59/fb6 x:0 0 0 2024-03-30T04:28:23.847 INFO:tasks.workunit.client.0.smithi123.stdout:4/844: chown d2/d34/f4c 19967767 1 2024-03-30T04:28:23.848 INFO:tasks.workunit.client.1.smithi135.stdout:6/507: dread - d2/d36/f64 zero size 2024-03-30T04:28:23.849 INFO:tasks.workunit.client.0.smithi123.stdout:4/845: readlink d2/d4/d7/d63/l7a 0 2024-03-30T04:28:23.850 INFO:tasks.workunit.client.0.smithi123.stdout:3/990: dread d13/d24/d31/d23/d63/f95 [0,4194304] 0 2024-03-30T04:28:23.850 INFO:tasks.workunit.client.1.smithi135.stdout:6/508: fdatasync d2/d2e/f4c 0 2024-03-30T04:28:23.852 INFO:tasks.workunit.client.1.smithi135.stdout:4/465: dread d10/d1a/f1f [0,4194304] 0 2024-03-30T04:28:23.852 INFO:tasks.workunit.client.1.smithi135.stdout:7/537: creat d4/db/d1a/d5c/d2c/d51/da8/fb7 x:0 0 0 2024-03-30T04:28:23.853 INFO:tasks.workunit.client.0.smithi123.stdout:3/991: creat d13/d24/d39/f145 x:0 0 0 2024-03-30T04:28:23.854 INFO:tasks.workunit.client.1.smithi135.stdout:4/466: mkdir d10/d4d/d9c 0 2024-03-30T04:28:23.855 INFO:tasks.workunit.client.1.smithi135.stdout:4/467: stat d10/f74 0 2024-03-30T04:28:23.856 INFO:tasks.workunit.client.1.smithi135.stdout:7/538: unlink d4/db/c22 0 2024-03-30T04:28:23.858 INFO:tasks.workunit.client.1.smithi135.stdout:7/539: write d4/f11 [2217570,9340] 0 2024-03-30T04:28:23.860 INFO:tasks.workunit.client.1.smithi135.stdout:4/468: link d10/d1a/d3d/f95 d10/d1a/d42/d46/f9d 0 2024-03-30T04:28:23.861 INFO:tasks.workunit.client.0.smithi123.stdout:3/992: write d13/f14 [3763291,112359] 0 2024-03-30T04:28:23.862 INFO:tasks.workunit.client.1.smithi135.stdout:4/469: stat d10/f57 0 2024-03-30T04:28:23.863 INFO:tasks.workunit.client.0.smithi123.stdout:4/846: dwrite d2/d4/d2c/f67 [0,4194304] 0 2024-03-30T04:28:23.865 INFO:tasks.workunit.client.1.smithi135.stdout:7/540: symlink d4/db/d1a/d5c/d2c/d2e/d98/lb8 0 2024-03-30T04:28:23.866 INFO:tasks.workunit.client.1.smithi135.stdout:4/470: mknod d10/d72/c9e 0 2024-03-30T04:28:23.866 INFO:tasks.workunit.client.0.smithi123.stdout:4/847: write d2/dfb/f107 [151567,31729] 0 2024-03-30T04:28:23.867 INFO:tasks.workunit.client.1.smithi135.stdout:4/471: fdatasync d10/d4d/f97 0 2024-03-30T04:28:23.867 INFO:tasks.workunit.client.1.smithi135.stdout:6/509: dwrite d2/f4 [0,4194304] 0 2024-03-30T04:28:23.872 INFO:tasks.workunit.client.1.smithi135.stdout:4/472: mkdir d10/d4d/d89/d9f 0 2024-03-30T04:28:23.874 INFO:tasks.workunit.client.1.smithi135.stdout:8/448: sync 2024-03-30T04:28:23.878 INFO:tasks.workunit.client.1.smithi135.stdout:4/473: mknod d10/d1a/d32/d40/ca0 0 2024-03-30T04:28:23.878 INFO:tasks.workunit.client.1.smithi135.stdout:7/541: rename d4/db/d1a/d5c/d2c/d2e/d46 to d4/db/d1a/d5c/d2c/d2e/db9 0 2024-03-30T04:28:23.878 INFO:tasks.workunit.client.1.smithi135.stdout:8/449: chown d6/d8/d15/d54/d69/d72/d73/f80 125 1 2024-03-30T04:28:23.879 INFO:tasks.workunit.client.0.smithi123.stdout:4/848: write d2/d4/d7/d63/f82 [1497971,118696] 0 2024-03-30T04:28:23.880 INFO:tasks.workunit.client.0.smithi123.stdout:4/849: chown d2/l5a 11 1 2024-03-30T04:28:23.882 INFO:tasks.workunit.client.1.smithi135.stdout:4/474: mknod d10/d1a/ca1 0 2024-03-30T04:28:23.883 INFO:tasks.workunit.client.1.smithi135.stdout:8/450: rmdir d6/d8/d15/d54/d69/d72/d73 39 2024-03-30T04:28:23.884 INFO:tasks.workunit.client.0.smithi123.stdout:9/854: sync 2024-03-30T04:28:23.885 INFO:tasks.workunit.client.1.smithi135.stdout:7/542: creat d4/db/d9f/fba x:0 0 0 2024-03-30T04:28:23.885 INFO:tasks.workunit.client.0.smithi123.stdout:9/855: read - d6/d58/f11f zero size 2024-03-30T04:28:23.886 INFO:tasks.workunit.client.0.smithi123.stdout:9/856: stat d6/d58/la2 0 2024-03-30T04:28:23.889 INFO:tasks.workunit.client.0.smithi123.stdout:9/857: chown d6/d1f/d72/d7d/f12c 534 1 2024-03-30T04:28:23.890 INFO:tasks.workunit.client.0.smithi123.stdout:9/858: dread - d6/d49/d113/fea zero size 2024-03-30T04:28:23.891 INFO:tasks.workunit.client.1.smithi135.stdout:7/543: mknod d4/db/d1a/d4b/d40/cbb 0 2024-03-30T04:28:23.892 INFO:tasks.workunit.client.0.smithi123.stdout:9/859: readlink d6/d49/d113/daa/ldc 0 2024-03-30T04:28:23.894 INFO:tasks.workunit.client.0.smithi123.stdout:9/860: chown d6/d1f/d92/db8/f119 24701 1 2024-03-30T04:28:23.896 INFO:tasks.workunit.client.1.smithi135.stdout:6/510: dwrite d2/dd/d18/f73 [0,4194304] 0 2024-03-30T04:28:23.896 INFO:tasks.workunit.client.0.smithi123.stdout:3/993: dread d13/d24/d39/d79/d90/fd4 [0,4194304] 0 2024-03-30T04:28:23.898 INFO:tasks.workunit.client.1.smithi135.stdout:4/475: getdents d10/d1a/d32/d40 0 2024-03-30T04:28:23.898 INFO:tasks.workunit.client.0.smithi123.stdout:9/861: unlink d6/d1f/d45/d4c/d121/c9f 0 2024-03-30T04:28:23.901 INFO:tasks.workunit.client.1.smithi135.stdout:7/544: rename d4/db/d1a/d5c/d2c/d2e/d98 to d4/db/d79/dbc 0 2024-03-30T04:28:23.901 INFO:tasks.workunit.client.0.smithi123.stdout:3/994: creat d13/f146 x:0 0 0 2024-03-30T04:28:23.903 INFO:tasks.workunit.client.1.smithi135.stdout:7/545: dread - d4/db/d79/dbc/fa5 zero size 2024-03-30T04:28:23.903 INFO:tasks.workunit.client.1.smithi135.stdout:6/511: symlink d2/dd/d14/d38/l9b 0 2024-03-30T04:28:23.904 INFO:tasks.workunit.client.0.smithi123.stdout:9/862: rename d6/d58/f6a to d6/d1f/d72/d10d/d11c/d12e/f12f 0 2024-03-30T04:28:23.904 INFO:tasks.workunit.client.1.smithi135.stdout:4/476: creat d10/d72/d7c/fa2 x:0 0 0 2024-03-30T04:28:23.907 INFO:tasks.workunit.client.1.smithi135.stdout:6/512: rmdir d2/dd/d14/d2b 39 2024-03-30T04:28:23.914 INFO:tasks.workunit.client.1.smithi135.stdout:4/477: mknod d10/d4d/ca3 0 2024-03-30T04:28:23.914 INFO:tasks.workunit.client.1.smithi135.stdout:4/478: symlink d10/d72/d85/la4 0 2024-03-30T04:28:23.919 INFO:tasks.workunit.client.1.smithi135.stdout:8/451: truncate d6/d8/d15/d54/d69/d72/d73/f80 2169985 0 2024-03-30T04:28:23.922 INFO:tasks.workunit.client.0.smithi123.stdout:3/995: dwrite d13/d24/d31/d123/d5d/f68 [0,4194304] 0 2024-03-30T04:28:23.923 INFO:tasks.workunit.client.0.smithi123.stdout:9/863: dwrite d6/fd3 [0,4194304] 0 2024-03-30T04:28:23.926 INFO:tasks.workunit.client.1.smithi135.stdout:8/452: unlink d6/l7d 0 2024-03-30T04:28:23.926 INFO:tasks.workunit.client.0.smithi123.stdout:3/996: write d13/d24/d31/d123/d5d/f8e [669780,99226] 0 2024-03-30T04:28:23.927 INFO:tasks.workunit.client.1.smithi135.stdout:6/513: dwrite d2/d68/f83 [0,4194304] 0 2024-03-30T04:28:23.933 INFO:tasks.workunit.client.0.smithi123.stdout:3/997: creat d13/d24/d31/d123/d5d/da5/d104/f147 x:0 0 0 2024-03-30T04:28:23.935 INFO:tasks.workunit.client.1.smithi135.stdout:6/514: mknod d2/d68/c9c 0 2024-03-30T04:28:23.938 INFO:tasks.workunit.client.1.smithi135.stdout:4/479: dwrite d10/d1a/d32/f96 [0,4194304] 0 2024-03-30T04:28:23.940 INFO:tasks.workunit.client.0.smithi123.stdout:9/864: dwrite d6/d1f/d92/ffe [0,4194304] 0 2024-03-30T04:28:23.945 INFO:tasks.workunit.client.0.smithi123.stdout:9/865: read d6/d1f/d45/d4c/d121/f40 [7256619,116503] 0 2024-03-30T04:28:23.948 INFO:tasks.workunit.client.0.smithi123.stdout:9/866: mknod d6/d1f/d92/df4/c130 0 2024-03-30T04:28:23.948 INFO:tasks.workunit.client.1.smithi135.stdout:8/453: rename d6/d8/d15/d54/f59 to d6/d8/d15/d54/d4f/f83 0 2024-03-30T04:28:23.952 INFO:tasks.workunit.client.1.smithi135.stdout:7/546: dread d4/db/d1a/f24 [0,4194304] 0 2024-03-30T04:28:23.954 INFO:tasks.workunit.client.0.smithi123.stdout:9/867: rename d6/d1f/d72/cca to d6/d1f/d45/d97/d10a/c131 0 2024-03-30T04:28:23.956 INFO:tasks.workunit.client.0.smithi123.stdout:9/868: chown d6/d1f/d45/d4c/d121/f40 15187368 1 2024-03-30T04:28:23.956 INFO:tasks.workunit.client.0.smithi123.stdout:9/869: chown d6/l31 687986 1 2024-03-30T04:28:23.957 INFO:tasks.workunit.client.1.smithi135.stdout:8/454: symlink d6/d8/d14/d32/l84 0 2024-03-30T04:28:23.960 INFO:tasks.workunit.client.1.smithi135.stdout:7/547: unlink d4/db/d1a/d5c/d2c/l35 0 2024-03-30T04:28:23.966 INFO:tasks.workunit.client.1.smithi135.stdout:6/515: dwrite d2/f4d [0,4194304] 0 2024-03-30T04:28:23.967 INFO:tasks.workunit.client.0.smithi123.stdout:3/998: dread d13/d24/d31/d123/d5d/f8e [0,4194304] 0 2024-03-30T04:28:23.968 INFO:tasks.workunit.client.0.smithi123.stdout:9/870: dread d6/d1f/d72/d10d/d11c/f12a [0,4194304] 0 2024-03-30T04:28:23.972 INFO:tasks.workunit.client.0.smithi123.stdout:3/999: mkdir d13/d24/d31/d123/d5d/da5/d148 0 2024-03-30T04:28:23.975 INFO:tasks.workunit.client.0.smithi123.stdout:9/871: dread d6/d1f/d92/db8/f103 [0,4194304] 0 2024-03-30T04:28:23.977 INFO:tasks.workunit.client.1.smithi135.stdout:8/455: rename d6/d8/l51 to d6/d8/d15/d54/d69/l85 0 2024-03-30T04:28:23.979 INFO:tasks.workunit.client.1.smithi135.stdout:6/516: creat d2/dd/d14/d30/d43/f9d x:0 0 0 2024-03-30T04:28:23.981 INFO:tasks.workunit.client.1.smithi135.stdout:4/480: dwrite d10/d1a/d32/d40/d92/f94 [0,4194304] 0 2024-03-30T04:28:23.982 INFO:tasks.workunit.client.1.smithi135.stdout:6/517: dread - d2/d36/f64 zero size 2024-03-30T04:28:23.987 INFO:tasks.workunit.client.1.smithi135.stdout:8/456: creat d6/f86 x:0 0 0 2024-03-30T04:28:23.987 INFO:tasks.workunit.client.1.smithi135.stdout:6/518: read - d2/dd/d14/f6c zero size 2024-03-30T04:28:23.989 INFO:tasks.workunit.client.1.smithi135.stdout:4/481: creat d10/d1a/d32/fa5 x:0 0 0 2024-03-30T04:28:23.993 INFO:tasks.workunit.client.1.smithi135.stdout:4/482: write d10/d1a/d42/f62 [2632323,113532] 0 2024-03-30T04:28:23.993 INFO:tasks.workunit.client.1.smithi135.stdout:7/548: dwrite d4/db/d1a/f24 [0,4194304] 0 2024-03-30T04:28:23.994 INFO:tasks.workunit.client.0.smithi123.stdout:9/872: dread d6/d1f/d72/f8f [0,4194304] 0 2024-03-30T04:28:23.996 INFO:tasks.workunit.client.1.smithi135.stdout:7/549: read - d4/db/d9f/fba zero size 2024-03-30T04:28:23.998 INFO:tasks.workunit.client.1.smithi135.stdout:4/483: creat d10/d72/fa6 x:0 0 0 2024-03-30T04:28:24.000 INFO:tasks.workunit.client.0.smithi123.stdout:6/958: sync 2024-03-30T04:28:24.001 INFO:tasks.workunit.client.0.smithi123.stdout:8/905: sync 2024-03-30T04:28:24.001 INFO:tasks.workunit.client.0.smithi123.stdout:5/868: sync 2024-03-30T04:28:24.001 INFO:tasks.workunit.client.0.smithi123.stdout:0/903: sync 2024-03-30T04:28:24.001 INFO:tasks.workunit.client.0.smithi123.stdout:1/954: sync 2024-03-30T04:28:24.001 INFO:tasks.workunit.client.0.smithi123.stdout:7/852: sync 2024-03-30T04:28:24.002 INFO:tasks.workunit.client.1.smithi135.stdout:7/550: read - d4/db/d1a/d5c/d59/d76/f91 zero size 2024-03-30T04:28:24.006 INFO:tasks.workunit.client.0.smithi123.stdout:5/869: write fb [3393807,2988] 0 2024-03-30T04:28:24.007 INFO:tasks.workunit.client.1.smithi135.stdout:4/484: creat d10/d4d/d89/d9f/fa7 x:0 0 0 2024-03-30T04:28:24.007 INFO:tasks.workunit.client.1.smithi135.stdout:1/459: sync 2024-03-30T04:28:24.013 INFO:tasks.workunit.client.0.smithi123.stdout:6/959: unlink d10/d82/d8e/dc7/l31 0 2024-03-30T04:28:24.013 INFO:tasks.workunit.client.0.smithi123.stdout:7/853: mkdir d2/da/d11/d61/d55/d59/d11f 0 2024-03-30T04:28:24.014 INFO:tasks.workunit.client.1.smithi135.stdout:4/485: symlink d10/d1a/d42/d66/la8 0 2024-03-30T04:28:24.016 INFO:tasks.workunit.client.1.smithi135.stdout:6/519: dwrite d2/dd/d14/f15 [0,4194304] 0 2024-03-30T04:28:24.016 INFO:tasks.workunit.client.0.smithi123.stdout:6/960: readlink d10/d20/d3f/l5e 0 2024-03-30T04:28:24.017 INFO:tasks.workunit.client.0.smithi123.stdout:9/873: dwrite d6/d49/d113/f123 [0,4194304] 0 2024-03-30T04:28:24.018 INFO:tasks.workunit.client.0.smithi123.stdout:5/870: symlink dd/d19/d1f/d3c/d72/d8c/df4/l126 0 2024-03-30T04:28:24.020 INFO:tasks.workunit.client.1.smithi135.stdout:4/486: readlink d10/d1a/l7a 0 2024-03-30T04:28:24.020 INFO:tasks.workunit.client.0.smithi123.stdout:7/854: unlink d2/d4d/d80/d98/f114 0 2024-03-30T04:28:24.023 INFO:tasks.workunit.client.0.smithi123.stdout:5/871: readlink dd/d19/d1f/d68/dcc/ld5 0 2024-03-30T04:28:24.023 INFO:tasks.workunit.client.0.smithi123.stdout:6/961: chown d10/l50 82502315 1 2024-03-30T04:28:24.023 INFO:tasks.workunit.client.1.smithi135.stdout:1/460: getdents d16/d29/d2c/d43 0 2024-03-30T04:28:24.024 INFO:tasks.workunit.client.1.smithi135.stdout:6/520: dread - d2/dd/d14/d2b/d86/f80 zero size 2024-03-30T04:28:24.025 INFO:tasks.workunit.client.0.smithi123.stdout:7/855: dread - d2/dc/d115/ffe zero size 2024-03-30T04:28:24.025 INFO:tasks.workunit.client.0.smithi123.stdout:8/906: dwrite da/dc/da9/dc4/fec [0,4194304] 0 2024-03-30T04:28:24.026 INFO:tasks.workunit.client.0.smithi123.stdout:9/874: mknod d6/d7c/de7/df1/c132 0 2024-03-30T04:28:24.028 INFO:tasks.workunit.client.1.smithi135.stdout:6/521: chown d2/d36/f6e 2613062 1 2024-03-30T04:28:24.030 INFO:tasks.workunit.client.0.smithi123.stdout:1/955: write db/dd/d42/d46/d56/d5c/f97 [1405632,94280] 0 2024-03-30T04:28:24.030 INFO:tasks.workunit.client.0.smithi123.stdout:0/904: getdents d6/dd/d15/d98/d4e/d69/d6e 0 2024-03-30T04:28:24.032 INFO:tasks.workunit.client.0.smithi123.stdout:7/856: write d2/dc/d115/f9b [349055,14359] 0 2024-03-30T04:28:24.033 INFO:tasks.workunit.client.0.smithi123.stdout:8/907: write da/f55 [4331114,59190] 0 2024-03-30T04:28:24.035 INFO:tasks.workunit.client.1.smithi135.stdout:7/551: dwrite d4/f8 [4194304,4194304] 0 2024-03-30T04:28:24.039 INFO:tasks.workunit.client.0.smithi123.stdout:0/905: write d6/dd/d15/d28/d4b/dad/f100 [130948,124979] 0 2024-03-30T04:28:24.039 INFO:tasks.workunit.client.0.smithi123.stdout:7/857: write d2/dc/d56/f116 [580517,100557] 0 2024-03-30T04:28:24.040 INFO:tasks.workunit.client.0.smithi123.stdout:9/875: read d6/d1f/d72/fc3 [96641,17645] 0 2024-03-30T04:28:24.040 INFO:tasks.workunit.client.0.smithi123.stdout:4/850: sync 2024-03-30T04:28:24.043 INFO:tasks.workunit.client.0.smithi123.stdout:5/872: getdents dd/d19/d5f/d99 0 2024-03-30T04:28:24.046 INFO:tasks.workunit.client.1.smithi135.stdout:4/487: dread d10/d1a/f21 [0,4194304] 0 2024-03-30T04:28:24.047 INFO:tasks.workunit.client.0.smithi123.stdout:8/908: write da/dc/da9/dc4/fec [1378973,6942] 0 2024-03-30T04:28:24.048 INFO:tasks.workunit.client.0.smithi123.stdout:4/851: unlink d2/d34/f77 0 2024-03-30T04:28:24.052 INFO:tasks.workunit.client.0.smithi123.stdout:9/876: chown d6/d1f/d92/db8/f119 30 1 2024-03-30T04:28:24.055 INFO:tasks.workunit.client.1.smithi135.stdout:4/488: creat d10/d4d/d89/d9f/fa9 x:0 0 0 2024-03-30T04:28:24.057 INFO:tasks.workunit.client.1.smithi135.stdout:1/461: write d16/d19/d20/d99/f4c [258400,12103] 0 2024-03-30T04:28:24.057 INFO:tasks.workunit.client.0.smithi123.stdout:0/906: chown d6/dd/d15/d28/d4b/f82 61 1 2024-03-30T04:28:24.060 INFO:tasks.workunit.client.1.smithi135.stdout:6/522: dwrite d2/dd/d14/d53/f77 [0,4194304] 0 2024-03-30T04:28:24.061 INFO:tasks.workunit.client.0.smithi123.stdout:5/873: creat dd/d19/d3e/d5b/d62/dd3/df3/f127 x:0 0 0 2024-03-30T04:28:24.062 INFO:tasks.workunit.client.1.smithi135.stdout:1/462: dread d16/d19/f56 [0,4194304] 0 2024-03-30T04:28:24.064 INFO:tasks.workunit.client.1.smithi135.stdout:7/552: dread d4/db/d1a/f24 [0,4194304] 0 2024-03-30T04:28:24.064 INFO:tasks.workunit.client.1.smithi135.stdout:4/489: mkdir d10/d4d/d89/d9f/daa 0 2024-03-30T04:28:24.065 INFO:tasks.workunit.client.1.smithi135.stdout:1/463: fdatasync d16/f1c 0 2024-03-30T04:28:24.067 INFO:tasks.workunit.client.0.smithi123.stdout:0/907: mknod d6/dd/d15/d98/dc6/c122 0 2024-03-30T04:28:24.067 INFO:tasks.workunit.client.1.smithi135.stdout:4/490: read - d10/d1a/f86 zero size 2024-03-30T04:28:24.069 INFO:tasks.workunit.client.0.smithi123.stdout:9/877: creat d6/d1f/d45/d4c/d12b/f133 x:0 0 0 2024-03-30T04:28:24.069 INFO:tasks.workunit.client.1.smithi135.stdout:6/523: rename d2/dd/d14/d53/l5b to d2/d2e/l9e 0 2024-03-30T04:28:24.071 INFO:tasks.workunit.client.0.smithi123.stdout:7/858: dwrite d2/da/d11/d34/d85/d8e/f10e [0,4194304] 0 2024-03-30T04:28:24.071 INFO:tasks.workunit.client.1.smithi135.stdout:4/491: truncate d10/d1a/f1f 4922062 0 2024-03-30T04:28:24.073 INFO:tasks.workunit.client.1.smithi135.stdout:6/524: creat d2/dd/d14/d38/f9f x:0 0 0 2024-03-30T04:28:24.078 INFO:tasks.workunit.client.1.smithi135.stdout:6/525: mknod d2/dd/d14/d2b/d86/d8e/ca0 0 2024-03-30T04:28:24.080 INFO:tasks.workunit.client.0.smithi123.stdout:1/956: truncate db/dd/d42/d46/d5d/d62/d9e/fff 1394565 0 2024-03-30T04:28:24.081 INFO:tasks.workunit.client.0.smithi123.stdout:5/874: read - dd/d19/d1f/d35/d3a/d79/fb0 zero size 2024-03-30T04:28:24.081 INFO:tasks.workunit.client.0.smithi123.stdout:9/878: mkdir d6/d58/d134 0 2024-03-30T04:28:24.082 INFO:tasks.workunit.client.1.smithi135.stdout:6/526: creat d2/d36/d76/fa1 x:0 0 0 2024-03-30T04:28:24.084 INFO:tasks.workunit.client.1.smithi135.stdout:7/553: dread d4/f11 [0,4194304] 0 2024-03-30T04:28:24.085 INFO:tasks.workunit.client.0.smithi123.stdout:0/908: mknod d6/dd/c123 0 2024-03-30T04:28:24.086 INFO:tasks.workunit.client.1.smithi135.stdout:6/527: chown d2/d36/c5d 427022 1 2024-03-30T04:28:24.086 INFO:tasks.workunit.client.0.smithi123.stdout:4/852: dwrite d2/d4/f10d [0,4194304] 0 2024-03-30T04:28:24.087 INFO:tasks.workunit.client.1.smithi135.stdout:4/492: write d10/d1a/d42/d46/f9d [939919,115361] 0 2024-03-30T04:28:24.090 INFO:tasks.workunit.client.0.smithi123.stdout:7/859: rmdir d2/da/d11/d61/d55/d9e/d113/ddb 0 2024-03-30T04:28:24.090 INFO:tasks.workunit.client.0.smithi123.stdout:1/957: read - db/fcd zero size 2024-03-30T04:28:24.091 INFO:tasks.workunit.client.0.smithi123.stdout:6/962: dread d10/d82/d8e/dc7/d42/f87 [0,4194304] 0 2024-03-30T04:28:24.091 INFO:tasks.workunit.client.1.smithi135.stdout:7/554: mkdir d4/db/d1a/d5c/d2c/d51/d56/dbd 0 2024-03-30T04:28:24.093 INFO:tasks.workunit.client.1.smithi135.stdout:4/493: mknod d10/cab 0 2024-03-30T04:28:24.093 INFO:tasks.workunit.client.1.smithi135.stdout:7/555: dread - d4/f8c zero size 2024-03-30T04:28:24.093 INFO:tasks.workunit.client.1.smithi135.stdout:1/464: dwrite d16/d19/d20/d99/f4c [0,4194304] 0 2024-03-30T04:28:24.097 INFO:tasks.workunit.client.0.smithi123.stdout:9/879: unlink d6/d1f/d4d/la6 0 2024-03-30T04:28:24.099 INFO:tasks.workunit.client.0.smithi123.stdout:0/909: dread d6/ff1 [0,4194304] 0 2024-03-30T04:28:24.099 INFO:tasks.workunit.client.0.smithi123.stdout:5/875: write dd/d19/d1f/d35/d3a/d57/de0/fe9 [202024,72672] 0 2024-03-30T04:28:24.101 INFO:tasks.workunit.client.0.smithi123.stdout:6/963: creat d10/d20/f141 x:0 0 0 2024-03-30T04:28:24.102 INFO:tasks.workunit.client.0.smithi123.stdout:4/853: dread - d2/d4/d2c/d42/d35/d5d/d6f/fd8 zero size 2024-03-30T04:28:24.103 INFO:tasks.workunit.client.0.smithi123.stdout:9/880: rmdir d6/d49/d113/daa 39 2024-03-30T04:28:24.105 INFO:tasks.workunit.client.0.smithi123.stdout:7/860: symlink d2/da/d11/d61/d55/d59/db5/l120 0 2024-03-30T04:28:24.105 INFO:tasks.workunit.client.1.smithi135.stdout:7/556: creat d4/db/de/d5a/dae/fbe x:0 0 0 2024-03-30T04:28:24.106 INFO:tasks.workunit.client.0.smithi123.stdout:8/909: dwrite da/f4e [0,4194304] 0 2024-03-30T04:28:24.106 INFO:tasks.workunit.client.1.smithi135.stdout:6/528: link d2/dd/d18/d87/c91 d2/d36/ca2 0 2024-03-30T04:28:24.108 INFO:tasks.workunit.client.1.smithi135.stdout:7/557: chown d4/db/d1a/l47 500 1 2024-03-30T04:28:24.112 INFO:tasks.workunit.client.0.smithi123.stdout:6/964: stat d10/d82/dc1/d6a/d7a/d99/dba/fde 0 2024-03-30T04:28:24.113 INFO:tasks.workunit.client.1.smithi135.stdout:6/529: rename d2/dd/d40/c75 to d2/d68/ca3 0 2024-03-30T04:28:24.116 INFO:tasks.workunit.client.1.smithi135.stdout:6/530: dread - d2/dd/d14/f72 zero size 2024-03-30T04:28:24.117 INFO:tasks.workunit.client.0.smithi123.stdout:0/910: rename d6/dd/d31/d35/f45 to d6/dd/d15/d98/dc6/f124 0 2024-03-30T04:28:24.118 INFO:tasks.workunit.client.1.smithi135.stdout:4/494: dwrite fa [0,4194304] 0 2024-03-30T04:28:24.119 INFO:tasks.workunit.client.0.smithi123.stdout:4/854: mknod d2/d4/d7/dc1/d103/c11c 0 2024-03-30T04:28:24.120 INFO:tasks.workunit.client.0.smithi123.stdout:1/958: dread db/dd/d5f/d6d/d85/d89/d90/d94/d9c/f102 [0,4194304] 0 2024-03-30T04:28:24.121 INFO:tasks.workunit.client.0.smithi123.stdout:4/855: dread - d2/d4/d2c/d42/dad/d91/fb1 zero size 2024-03-30T04:28:24.122 INFO:tasks.workunit.client.0.smithi123.stdout:6/965: mknod d10/d20/d3f/c142 0 2024-03-30T04:28:24.124 INFO:tasks.workunit.client.0.smithi123.stdout:0/911: symlink d6/d52/l125 0 2024-03-30T04:28:24.125 INFO:tasks.workunit.client.1.smithi135.stdout:1/465: dwrite d16/d5a/f82 [0,4194304] 0 2024-03-30T04:28:24.127 INFO:tasks.workunit.client.1.smithi135.stdout:6/531: fdatasync d2/dd/d14/d2b/d86/d55/f74 0 2024-03-30T04:28:24.127 INFO:tasks.workunit.client.1.smithi135.stdout:4/495: stat d10/d72/d7c/c8d 0 2024-03-30T04:28:24.130 INFO:tasks.workunit.client.0.smithi123.stdout:6/966: rmdir d10/d82/d8e/df0 39 2024-03-30T04:28:24.134 INFO:tasks.workunit.client.1.smithi135.stdout:6/532: write d2/f99 [990682,5476] 0 2024-03-30T04:28:24.134 INFO:tasks.workunit.client.0.smithi123.stdout:8/910: read da/d27/d56/d6c/d96/f119 [1207662,35820] 0 2024-03-30T04:28:24.136 INFO:tasks.workunit.client.0.smithi123.stdout:1/959: creat db/dd/d5f/d6d/f13e x:0 0 0 2024-03-30T04:28:24.137 INFO:tasks.workunit.client.1.smithi135.stdout:1/466: rename d16/d19/d20/d26/c4a to d16/d19/d20/d99/c9e 0 2024-03-30T04:28:24.139 INFO:tasks.workunit.client.1.smithi135.stdout:1/467: dread - d16/d19/f2b zero size 2024-03-30T04:28:24.140 INFO:tasks.workunit.client.1.smithi135.stdout:7/558: dread d4/f8 [4194304,4194304] 0 2024-03-30T04:28:24.140 INFO:tasks.workunit.client.1.smithi135.stdout:6/533: creat d2/dd/d14/d2b/d86/d55/fa4 x:0 0 0 2024-03-30T04:28:24.141 INFO:tasks.workunit.client.1.smithi135.stdout:4/496: rename d10/l35 to d10/d1a/d32/d40/d92/lac 0 2024-03-30T04:28:24.144 INFO:tasks.workunit.client.0.smithi123.stdout:9/881: write d6/fb [2931387,121357] 0 2024-03-30T04:28:24.145 INFO:tasks.workunit.client.0.smithi123.stdout:1/960: creat db/dd/d42/d46/d5d/f13f x:0 0 0 2024-03-30T04:28:24.145 INFO:tasks.workunit.client.1.smithi135.stdout:1/468: mkdir d16/d9f 0 2024-03-30T04:28:24.145 INFO:tasks.workunit.client.1.smithi135.stdout:4/497: dread - d10/d4d/d89/d9f/fa9 zero size 2024-03-30T04:28:24.145 INFO:tasks.workunit.client.0.smithi123.stdout:8/911: mknod da/d27/d3c/d64/c120 0 2024-03-30T04:28:24.146 INFO:tasks.workunit.client.1.smithi135.stdout:7/559: mknod d4/db/d1a/d3d/d96/cbf 0 2024-03-30T04:28:24.149 INFO:tasks.workunit.client.1.smithi135.stdout:1/469: write d16/d5a/d5f/d3a/f52 [247678,59125] 0 2024-03-30T04:28:24.150 INFO:tasks.workunit.client.0.smithi123.stdout:6/967: dread d10/d20/da2/fab [0,4194304] 0 2024-03-30T04:28:24.151 INFO:tasks.workunit.client.1.smithi135.stdout:4/498: symlink d10/d1a/d42/lad 0 2024-03-30T04:28:24.152 INFO:tasks.workunit.client.0.smithi123.stdout:9/882: creat d6/dad/f135 x:0 0 0 2024-03-30T04:28:24.153 INFO:tasks.workunit.client.0.smithi123.stdout:1/961: mknod db/dd/d42/d46/d56/df6/c140 0 2024-03-30T04:28:24.154 INFO:tasks.workunit.client.0.smithi123.stdout:6/968: fdatasync d10/d20/fea 0 2024-03-30T04:28:24.156 INFO:tasks.workunit.client.1.smithi135.stdout:4/499: creat d10/d1a/d32/fae x:0 0 0 2024-03-30T04:28:24.158 INFO:tasks.workunit.client.0.smithi123.stdout:8/912: symlink da/d27/d3c/l121 0 2024-03-30T04:28:24.160 INFO:tasks.workunit.client.1.smithi135.stdout:4/500: mkdir d10/d1a/d2b/daf 0 2024-03-30T04:28:24.160 INFO:tasks.workunit.client.1.smithi135.stdout:6/534: dwrite d2/f9 [0,4194304] 0 2024-03-30T04:28:24.162 INFO:tasks.workunit.client.1.smithi135.stdout:6/535: chown d2/dd/d18 125713606 1 2024-03-30T04:28:24.162 INFO:tasks.workunit.client.1.smithi135.stdout:4/501: dread - d10/d72/d7c/f9b zero size 2024-03-30T04:28:24.166 INFO:tasks.workunit.client.0.smithi123.stdout:9/883: creat d6/d1f/d72/f136 x:0 0 0 2024-03-30T04:28:24.167 INFO:tasks.workunit.client.0.smithi123.stdout:8/913: mkdir da/dc/d14/d9d/dff/d122 0 2024-03-30T04:28:24.167 INFO:tasks.workunit.client.0.smithi123.stdout:1/962: write db/dd/d5f/dae/ffd [1407318,125813] 0 2024-03-30T04:28:24.168 INFO:tasks.workunit.client.0.smithi123.stdout:4/856: dread d2/d4/d2c/d42/d35/d5d/d78/fbe [0,4194304] 0 2024-03-30T04:28:24.170 INFO:tasks.workunit.client.1.smithi135.stdout:6/536: getdents d2/dd 0 2024-03-30T04:28:24.172 INFO:tasks.workunit.client.0.smithi123.stdout:0/912: dwrite d6/dd/d15/d28/dd3/ddd/f88 [0,4194304] 0 2024-03-30T04:28:24.173 INFO:tasks.workunit.client.0.smithi123.stdout:6/969: write d10/d82/dc1/d6a/fca [2644694,50553] 0 2024-03-30T04:28:24.174 INFO:tasks.workunit.client.0.smithi123.stdout:9/884: stat d6/d1f/d72/d7d/d10f/d114/d115/c116 0 2024-03-30T04:28:24.175 INFO:tasks.workunit.client.1.smithi135.stdout:6/537: symlink d2/dd/d18/d87/la5 0 2024-03-30T04:28:24.175 INFO:tasks.workunit.client.1.smithi135.stdout:9/438: sync 2024-03-30T04:28:24.175 INFO:tasks.workunit.client.1.smithi135.stdout:2/394: sync 2024-03-30T04:28:24.175 INFO:tasks.workunit.client.1.smithi135.stdout:0/604: sync 2024-03-30T04:28:24.175 INFO:tasks.workunit.client.1.smithi135.stdout:5/450: sync 2024-03-30T04:28:24.175 INFO:tasks.workunit.client.1.smithi135.stdout:3/491: sync 2024-03-30T04:28:24.176 INFO:tasks.workunit.client.0.smithi123.stdout:1/963: chown db/dd/d42/d46/d39/l5a 133963970 1 2024-03-30T04:28:24.176 INFO:tasks.workunit.client.0.smithi123.stdout:9/885: fdatasync d6/f26 0 2024-03-30T04:28:24.177 INFO:tasks.workunit.client.0.smithi123.stdout:0/913: dread - d6/dd/d15/d98/dd0/f10d zero size 2024-03-30T04:28:24.177 INFO:tasks.workunit.client.0.smithi123.stdout:8/914: truncate da/d27/d3c/d2d/d5c/f10d 292931 0 2024-03-30T04:28:24.179 INFO:tasks.workunit.client.1.smithi135.stdout:6/538: dread - d2/dd/d14/d2b/f85 zero size 2024-03-30T04:28:24.180 INFO:tasks.workunit.client.1.smithi135.stdout:9/439: write d7/d1b/d37/f4a [3062731,9973] 0 2024-03-30T04:28:24.181 INFO:tasks.workunit.client.1.smithi135.stdout:5/451: stat d1/d3/d2b/d54/d6d 0 2024-03-30T04:28:24.183 INFO:tasks.workunit.client.1.smithi135.stdout:0/605: mkdir d0/d5/d9/d23/d2a/dc4 0 2024-03-30T04:28:24.185 INFO:tasks.workunit.client.0.smithi123.stdout:4/857: creat d2/d4/d2c/d42/d35/d5d/dc0/f11d x:0 0 0 2024-03-30T04:28:24.185 INFO:tasks.workunit.client.0.smithi123.stdout:6/970: rmdir d10/d82/dc1/dd2 39 2024-03-30T04:28:24.185 INFO:tasks.workunit.client.1.smithi135.stdout:2/395: write d4/d32/f33 [2876991,72319] 0 2024-03-30T04:28:24.186 INFO:tasks.workunit.client.0.smithi123.stdout:9/886: write d6/d1f/f3f [2768117,98593] 0 2024-03-30T04:28:24.191 INFO:tasks.workunit.client.0.smithi123.stdout:8/915: fdatasync da/dc/d14/f6e 0 2024-03-30T04:28:24.191 INFO:tasks.workunit.client.1.smithi135.stdout:7/560: dread f3 [0,4194304] 0 2024-03-30T04:28:24.191 INFO:tasks.workunit.client.1.smithi135.stdout:6/539: write d2/dd/d14/d38/f7e [653254,119150] 0 2024-03-30T04:28:24.191 INFO:tasks.workunit.client.1.smithi135.stdout:5/452: write d1/d3/d2b/d54/d5e/d5c/f65 [1492489,20500] 0 2024-03-30T04:28:24.192 INFO:tasks.workunit.client.1.smithi135.stdout:3/492: mkdir d3/d54/dab 0 2024-03-30T04:28:24.192 INFO:tasks.workunit.client.0.smithi123.stdout:7/861: dread d2/da/d11/d61/d55/d59/f88 [0,4194304] 0 2024-03-30T04:28:24.193 INFO:tasks.workunit.client.0.smithi123.stdout:5/876: dread dd/d19/d1f/d35/d3a/d57/db3/fde [0,4194304] 0 2024-03-30T04:28:24.194 INFO:tasks.workunit.client.1.smithi135.stdout:9/440: creat d7/d4f/d5c/d63/d86/f9a x:0 0 0 2024-03-30T04:28:24.195 INFO:tasks.workunit.client.0.smithi123.stdout:8/916: chown da/d27/d3c/d2d/dae/fee 5 1 2024-03-30T04:28:24.200 INFO:tasks.workunit.client.0.smithi123.stdout:5/877: readlink dd/d19/l7b 0 2024-03-30T04:28:24.200 INFO:tasks.workunit.client.0.smithi123.stdout:0/914: fdatasync d6/dd/d31/d35/d3a/f8f 0 2024-03-30T04:28:24.200 INFO:tasks.workunit.client.1.smithi135.stdout:7/561: write d4/db/d1a/d5c/d2c/d51/d56/d81/f7f [998739,16183] 0 2024-03-30T04:28:24.201 INFO:tasks.workunit.client.0.smithi123.stdout:4/858: mkdir d2/d4/d2c/d42/d35/d5d/d11e 0 2024-03-30T04:28:24.202 INFO:tasks.workunit.client.1.smithi135.stdout:3/493: write d3/d10/d16/f43 [1597365,117333] 0 2024-03-30T04:28:24.203 INFO:tasks.workunit.client.1.smithi135.stdout:6/540: symlink d2/d2e/la6 0 2024-03-30T04:28:24.204 INFO:tasks.workunit.client.0.smithi123.stdout:5/878: readlink dd/d19/d1f/d68/dcc/ld5 0 2024-03-30T04:28:24.206 INFO:tasks.workunit.client.1.smithi135.stdout:5/453: truncate d1/da/f18 4342436 0 2024-03-30T04:28:24.209 INFO:tasks.workunit.client.0.smithi123.stdout:7/862: creat d2/da/d11/d61/d55/d59/db5/dab/f121 x:0 0 0 2024-03-30T04:28:24.213 INFO:tasks.workunit.client.0.smithi123.stdout:1/964: write db/dd/d42/d46/d51/fda [36819,5313] 0 2024-03-30T04:28:24.214 INFO:tasks.workunit.client.1.smithi135.stdout:7/562: dread d4/db/d1a/d5c/d2c/d51/f7e [0,4194304] 0 2024-03-30T04:28:24.216 INFO:tasks.workunit.client.1.smithi135.stdout:4/502: dread d10/d1a/d3d/f41 [0,4194304] 0 2024-03-30T04:28:24.217 INFO:tasks.workunit.client.0.smithi123.stdout:0/915: getdents d6/dd/d15/d98/dd0 0 2024-03-30T04:28:24.217 INFO:tasks.workunit.client.1.smithi135.stdout:2/396: getdents d4/d55/d77 0 2024-03-30T04:28:24.218 INFO:tasks.workunit.client.1.smithi135.stdout:9/441: link d7/d1b/f80 d7/d29/d78/f9b 0 2024-03-30T04:28:24.219 INFO:tasks.workunit.client.1.smithi135.stdout:4/503: readlink d10/d1a/l81 0 2024-03-30T04:28:24.221 INFO:tasks.workunit.client.0.smithi123.stdout:9/887: dwrite d6/dad/fd7 [0,4194304] 0 2024-03-30T04:28:24.223 INFO:tasks.workunit.client.1.smithi135.stdout:7/563: unlink d4/db/d1a/d5c/f43 0 2024-03-30T04:28:24.223 INFO:tasks.workunit.client.1.smithi135.stdout:0/606: dwrite d0/d5/d9/d23/d2a/fa0 [4194304,4194304] 0 2024-03-30T04:28:24.224 INFO:tasks.workunit.client.0.smithi123.stdout:1/965: mkdir db/dd/d42/d46/d56/daf/d141 0 2024-03-30T04:28:24.224 INFO:tasks.workunit.client.1.smithi135.stdout:6/541: link d2/dd/d14/d2b/d86/l5e d2/la7 0 2024-03-30T04:28:24.226 INFO:tasks.workunit.client.1.smithi135.stdout:0/607: write d0/d13/f74 [134673,25923] 0 2024-03-30T04:28:24.227 INFO:tasks.workunit.client.1.smithi135.stdout:9/442: creat d7/d90/f9c x:0 0 0 2024-03-30T04:28:24.230 INFO:tasks.workunit.client.1.smithi135.stdout:0/608: chown d0/d5/d9/l22 509376 1 2024-03-30T04:28:24.234 INFO:tasks.workunit.client.0.smithi123.stdout:7/863: write d2/da/d11/d34/f7d [266569,7366] 0 2024-03-30T04:28:24.235 INFO:tasks.workunit.client.0.smithi123.stdout:9/888: fdatasync d6/d1f/d72/f136 0 2024-03-30T04:28:24.236 INFO:tasks.workunit.client.0.smithi123.stdout:6/971: dwrite d10/d82/dc1/d45/faa [0,4194304] 0 2024-03-30T04:28:24.241 INFO:tasks.workunit.client.0.smithi123.stdout:7/864: chown d2/lb7 12032 1 2024-03-30T04:28:24.242 INFO:tasks.workunit.client.0.smithi123.stdout:1/966: write db/dd/d5f/d6d/d85/d89/d90/d94/d9c/f138 [192503,121370] 0 2024-03-30T04:28:24.244 INFO:tasks.workunit.client.0.smithi123.stdout:9/889: mkdir d6/d1f/d72/db6/d137 0 2024-03-30T04:28:24.249 INFO:tasks.workunit.client.0.smithi123.stdout:0/916: write d6/dd/d31/d35/f5b [505120,121550] 0 2024-03-30T04:28:24.249 INFO:tasks.workunit.client.0.smithi123.stdout:6/972: chown d10/d82/d8e/dc7/f1e 20739051 1 2024-03-30T04:28:24.251 INFO:tasks.workunit.client.0.smithi123.stdout:9/890: write d6/d1f/f3f [2075201,87415] 0 2024-03-30T04:28:24.253 INFO:tasks.workunit.client.0.smithi123.stdout:1/967: dread - db/dd/d42/d8f/d78/f11c zero size 2024-03-30T04:28:24.253 INFO:tasks.workunit.client.0.smithi123.stdout:7/865: creat d2/da/d11/d61/d55/d9e/d113/f122 x:0 0 0 2024-03-30T04:28:24.253 INFO:tasks.workunit.client.0.smithi123.stdout:4/859: dread d2/d4/d2c/fa7 [0,4194304] 0 2024-03-30T04:28:24.257 INFO:tasks.workunit.client.0.smithi123.stdout:8/917: dread da/d27/f8a [0,4194304] 0 2024-03-30T04:28:24.258 INFO:tasks.workunit.client.0.smithi123.stdout:6/973: mkdir d10/d82/dc1/d6a/d143 0 2024-03-30T04:28:24.258 INFO:tasks.workunit.client.0.smithi123.stdout:0/917: symlink d6/dd/d18/l126 0 2024-03-30T04:28:24.259 INFO:tasks.workunit.client.0.smithi123.stdout:9/891: rename d6/d1f/c80 to d6/d7c/c138 0 2024-03-30T04:28:24.262 INFO:tasks.workunit.client.0.smithi123.stdout:0/918: dread - d6/dd/d18/f109 zero size 2024-03-30T04:28:24.263 INFO:tasks.workunit.client.0.smithi123.stdout:4/860: unlink d2/d4/f49 0 2024-03-30T04:28:24.264 INFO:tasks.workunit.client.0.smithi123.stdout:7/866: creat d2/da/d11/d61/d55/d59/db5/dab/dca/f123 x:0 0 0 2024-03-30T04:28:24.266 INFO:tasks.workunit.client.0.smithi123.stdout:8/918: chown da/d27/d3c/d64/d78/db8/dfd/da2/cd1 91850 1 2024-03-30T04:28:24.266 INFO:tasks.workunit.client.0.smithi123.stdout:7/867: stat d2/dc/d115/l7b 0 2024-03-30T04:28:24.266 INFO:tasks.workunit.client.0.smithi123.stdout:1/968: dread db/dd/d5f/d6d/d85/d89/fc9 [0,4194304] 0 2024-03-30T04:28:24.269 INFO:tasks.workunit.client.0.smithi123.stdout:4/861: creat d2/d4/d7/dc1/d103/f11f x:0 0 0 2024-03-30T04:28:24.270 INFO:tasks.workunit.client.0.smithi123.stdout:8/919: chown da/d27/d3c/d2d/d5c/f6f 459 1 2024-03-30T04:28:24.271 INFO:tasks.workunit.client.0.smithi123.stdout:1/969: chown db/dd/d42/d46/d56/c112 2 1 2024-03-30T04:28:24.272 INFO:tasks.workunit.client.0.smithi123.stdout:0/919: rename d6/dd/d15/l46 to d6/dd/d15/d28/dd3/ddd/l127 0 2024-03-30T04:28:24.274 INFO:tasks.workunit.client.0.smithi123.stdout:7/868: mknod d2/da/d11/d34/ded/dfc/c124 0 2024-03-30T04:28:24.275 INFO:tasks.workunit.client.0.smithi123.stdout:9/892: link d6/d1f/d72/d7d/dfa/lda d6/d1f/d45/d4c/d121/l139 0 2024-03-30T04:28:24.278 INFO:tasks.workunit.client.0.smithi123.stdout:9/893: read - d6/d1f/d92/fe3 zero size 2024-03-30T04:28:24.283 INFO:tasks.workunit.client.0.smithi123.stdout:7/869: fsync d2/dc/d1c/f1d 0 2024-03-30T04:28:24.285 INFO:tasks.workunit.client.0.smithi123.stdout:7/870: stat d2/da/d11/d61/d55/d94 0 2024-03-30T04:28:24.286 INFO:tasks.workunit.client.0.smithi123.stdout:6/974: dread d10/d20/f2b [0,4194304] 0 2024-03-30T04:28:24.288 INFO:tasks.workunit.client.0.smithi123.stdout:1/970: getdents db/dd/d42/d46/d56/daf/de4/d12d 0 2024-03-30T04:28:24.291 INFO:tasks.workunit.client.0.smithi123.stdout:7/871: chown d2/da/d11/d61/d55/d9e/c109 69 1 2024-03-30T04:28:24.292 INFO:tasks.workunit.client.0.smithi123.stdout:6/975: dread - d10/d82/dc1/d6a/d7a/d99/fc2 zero size 2024-03-30T04:28:24.294 INFO:tasks.workunit.client.0.smithi123.stdout:1/971: mknod db/dd/d42/d46/d5d/d62/c142 0 2024-03-30T04:28:24.297 INFO:tasks.workunit.client.0.smithi123.stdout:8/920: write da/dc/d14/f71 [744959,19775] 0 2024-03-30T04:28:24.301 INFO:tasks.workunit.client.0.smithi123.stdout:4/862: truncate d2/d4/d7/d1a/f30 4425698 0 2024-03-30T04:28:24.303 INFO:tasks.workunit.client.0.smithi123.stdout:9/894: dwrite d6/d1f/d45/d4c/d121/f8d [4194304,4194304] 0 2024-03-30T04:28:24.304 INFO:tasks.workunit.client.1.smithi135.stdout:6/542: mkdir d2/d68/da8 0 2024-03-30T04:28:24.305 INFO:tasks.workunit.client.1.smithi135.stdout:5/454: write d1/d7/d1a/f3b [502850,127495] 0 2024-03-30T04:28:24.306 INFO:tasks.workunit.client.1.smithi135.stdout:2/397: write d4/d32/d2e/d2f/f64 [1024731,37767] 0 2024-03-30T04:28:24.306 INFO:tasks.workunit.client.1.smithi135.stdout:0/609: rmdir d0/d5/d9/d23/d2a/d60 39 2024-03-30T04:28:24.307 INFO:tasks.workunit.client.0.smithi123.stdout:4/863: chown d2/d4/d7/l31 1668841 1 2024-03-30T04:28:24.309 INFO:tasks.workunit.client.1.smithi135.stdout:3/494: link d3/db/c47 d3/d54/dab/cac 0 2024-03-30T04:28:24.315 INFO:tasks.workunit.client.0.smithi123.stdout:8/921: creat da/d27/f123 x:0 0 0 2024-03-30T04:28:24.315 INFO:tasks.workunit.client.1.smithi135.stdout:6/543: symlink d2/d36/d76/la9 0 2024-03-30T04:28:24.316 INFO:tasks.workunit.client.1.smithi135.stdout:5/455: write d1/d3/f49 [179357,34543] 0 2024-03-30T04:28:24.316 INFO:tasks.workunit.client.1.smithi135.stdout:0/610: write d0/d13/d1e/f4f [883012,59319] 0 2024-03-30T04:28:24.316 INFO:tasks.workunit.client.1.smithi135.stdout:6/544: dread - d2/dd/d14/d38/f9f zero size 2024-03-30T04:28:24.317 INFO:tasks.workunit.client.0.smithi123.stdout:4/864: dread - d2/d4/d7/fe5 zero size 2024-03-30T04:28:24.318 INFO:tasks.workunit.client.1.smithi135.stdout:9/443: getdents d7/d41 0 2024-03-30T04:28:24.320 INFO:tasks.workunit.client.1.smithi135.stdout:6/545: fsync d2/dd/d14/d38/f9f 0 2024-03-30T04:28:24.320 INFO:tasks.workunit.client.1.smithi135.stdout:0/611: chown d0/d13/d53/d7f/d83 189406237 1 2024-03-30T04:28:24.321 INFO:tasks.workunit.client.1.smithi135.stdout:2/398: mknod d4/d6/c82 0 2024-03-30T04:28:24.324 INFO:tasks.workunit.client.1.smithi135.stdout:5/456: unlink d1/d7/d1a/c2a 0 2024-03-30T04:28:24.326 INFO:tasks.workunit.client.1.smithi135.stdout:7/564: dread d4/f16 [0,4194304] 0 2024-03-30T04:28:24.326 INFO:tasks.workunit.client.1.smithi135.stdout:9/444: write d7/d29/f46 [89690,130319] 0 2024-03-30T04:28:24.326 INFO:tasks.workunit.client.0.smithi123.stdout:0/920: dwrite d6/dd/d31/d35/f43 [0,4194304] 0 2024-03-30T04:28:24.327 INFO:tasks.workunit.client.0.smithi123.stdout:9/895: write d6/d58/f5c [1352821,78138] 0 2024-03-30T04:28:24.327 INFO:tasks.workunit.client.0.smithi123.stdout:8/922: unlink da/d27/d3c/d2d/c7b 0 2024-03-30T04:28:24.329 INFO:tasks.workunit.client.1.smithi135.stdout:5/457: stat d1/d3/d2b/d54/d5e/d5c/f65 0 2024-03-30T04:28:24.329 INFO:tasks.workunit.client.1.smithi135.stdout:3/495: mknod d3/d10/d16/d41/d83/cad 0 2024-03-30T04:28:24.334 INFO:tasks.workunit.client.1.smithi135.stdout:5/458: write d1/d7/f43 [2413359,89964] 0 2024-03-30T04:28:24.335 INFO:tasks.workunit.client.1.smithi135.stdout:6/546: unlink d2/d36/f64 0 2024-03-30T04:28:24.336 INFO:tasks.workunit.client.1.smithi135.stdout:2/399: truncate d4/d6/f1f 4795711 0 2024-03-30T04:28:24.343 INFO:tasks.workunit.client.1.smithi135.stdout:9/445: rename d7/d29/f82 to d7/d55/d8d/f9d 0 2024-03-30T04:28:24.346 INFO:tasks.workunit.client.0.smithi123.stdout:6/976: dwrite d10/d82/d8e/dc7/d5b/fef [0,4194304] 0 2024-03-30T04:28:24.347 INFO:tasks.workunit.client.1.smithi135.stdout:7/565: creat d4/db/de/db0/fc0 x:0 0 0 2024-03-30T04:28:24.355 INFO:tasks.workunit.client.0.smithi123.stdout:7/872: dwrite d2/dc/d115/f9d [0,4194304] 0 2024-03-30T04:28:24.356 INFO:tasks.workunit.client.0.smithi123.stdout:1/972: dwrite db/dd/d42/d46/d51/d72/f10d [4194304,4194304] 0 2024-03-30T04:28:24.360 INFO:tasks.workunit.client.0.smithi123.stdout:1/973: write db/d1b/f28 [2171267,50264] 0 2024-03-30T04:28:24.363 INFO:tasks.workunit.client.1.smithi135.stdout:9/446: dread d7/d55/f6c [0,4194304] 0 2024-03-30T04:28:24.366 INFO:tasks.workunit.client.1.smithi135.stdout:0/612: dwrite d0/f12 [0,4194304] 0 2024-03-30T04:28:24.368 INFO:tasks.workunit.client.1.smithi135.stdout:3/496: write d3/db/f8f [587235,66850] 0 2024-03-30T04:28:24.371 INFO:tasks.workunit.client.1.smithi135.stdout:3/497: chown d3/d50/d22/f72 8 1 2024-03-30T04:28:24.377 INFO:tasks.workunit.client.1.smithi135.stdout:5/459: dwrite d1/d3/d2b/d54/f71 [0,4194304] 0 2024-03-30T04:28:24.384 INFO:tasks.workunit.client.0.smithi123.stdout:8/923: creat da/d27/d56/d6c/d96/dc2/f124 x:0 0 0 2024-03-30T04:28:24.387 INFO:tasks.workunit.client.0.smithi123.stdout:6/977: mknod d10/d82/dd6/c144 0 2024-03-30T04:28:24.392 INFO:tasks.workunit.client.0.smithi123.stdout:0/921: symlink d6/dd/d15/d98/d4e/d69/d6e/d99/da7/l128 0 2024-03-30T04:28:24.393 INFO:tasks.workunit.client.0.smithi123.stdout:0/922: chown d6/dd/d31 1655 1 2024-03-30T04:28:24.395 INFO:tasks.workunit.client.0.smithi123.stdout:1/974: symlink db/dd/l143 0 2024-03-30T04:28:24.395 INFO:tasks.workunit.client.1.smithi135.stdout:8/457: sync 2024-03-30T04:28:24.396 INFO:tasks.workunit.client.0.smithi123.stdout:4/865: getdents d2/d4/d7/dc1/d103 0 2024-03-30T04:28:24.396 INFO:tasks.workunit.client.0.smithi123.stdout:9/896: link d6/d1f/d72/ca1 d6/d7c/de7/df1/c13a 0 2024-03-30T04:28:24.399 INFO:tasks.workunit.client.0.smithi123.stdout:6/978: creat d10/d82/dc1/d45/d77/d12c/f145 x:0 0 0 2024-03-30T04:28:24.402 INFO:tasks.workunit.client.0.smithi123.stdout:1/975: unlink db/dd/d42/d8f/fe2 0 2024-03-30T04:28:24.407 INFO:tasks.workunit.client.0.smithi123.stdout:6/979: creat d10/d82/dc1/d45/f146 x:0 0 0 2024-03-30T04:28:24.407 INFO:tasks.workunit.client.0.smithi123.stdout:4/866: read d2/d4/d7/fc4 [2642444,32412] 0 2024-03-30T04:28:24.407 INFO:tasks.workunit.client.0.smithi123.stdout:1/976: write db/dd/d5f/d6d/d85/d89/d90/d94/d9c/dfc/f11a [890241,126189] 0 2024-03-30T04:28:24.411 INFO:tasks.workunit.client.0.smithi123.stdout:1/977: readlink db/dd/d42/d46/d5d/d62/l127 0 2024-03-30T04:28:24.411 INFO:tasks.workunit.client.1.smithi135.stdout:2/400: rename d4/f5 to d4/d32/d46/f83 0 2024-03-30T04:28:24.413 INFO:tasks.workunit.client.1.smithi135.stdout:7/566: creat d4/db/de/db0/fc1 x:0 0 0 2024-03-30T04:28:24.413 INFO:tasks.workunit.client.0.smithi123.stdout:8/924: link da/cd3 da/dc/d14/d18/c125 0 2024-03-30T04:28:24.415 INFO:tasks.workunit.client.0.smithi123.stdout:8/925: dread - da/d27/d56/f100 zero size 2024-03-30T04:28:24.416 INFO:tasks.workunit.client.1.smithi135.stdout:0/613: mkdir d0/d13/d53/d7f/dc5 0 2024-03-30T04:28:24.416 INFO:tasks.workunit.client.0.smithi123.stdout:6/980: rename d10/d82/dc1/d45/fd3 to d10/d82/dd6/f147 0 2024-03-30T04:28:24.417 INFO:tasks.workunit.client.0.smithi123.stdout:0/923: link d6/dd/d15/d28/d4b/dad/de8/lf4 d6/dd/d15/d98/d4e/d69/d6e/d99/da7/l129 0 2024-03-30T04:28:24.418 INFO:tasks.workunit.client.0.smithi123.stdout:9/897: getdents d6/d1f/d92/df4 0 2024-03-30T04:28:24.422 INFO:tasks.workunit.client.1.smithi135.stdout:5/460: rename d1/d7/d1a/d1f/d7f/c85 to d1/da/c8b 0 2024-03-30T04:28:24.422 INFO:tasks.workunit.client.0.smithi123.stdout:1/978: unlink db/dd/d42/d46/d56/dce/f10b 0 2024-03-30T04:28:24.423 INFO:tasks.workunit.client.1.smithi135.stdout:2/401: fsync d4/da/f21 0 2024-03-30T04:28:24.426 INFO:tasks.workunit.client.0.smithi123.stdout:1/979: read db/dd/d42/d46/d56/d5c/f97 [1010470,49251] 0 2024-03-30T04:28:24.427 INFO:tasks.workunit.client.1.smithi135.stdout:8/458: symlink d6/l87 0 2024-03-30T04:28:24.428 INFO:tasks.workunit.client.0.smithi123.stdout:1/980: readlink db/l40 0 2024-03-30T04:28:24.428 INFO:tasks.workunit.client.1.smithi135.stdout:6/547: link d2/d36/c6f d2/dd/d14/d2b/d86/d8e/caa 0 2024-03-30T04:28:24.433 INFO:tasks.workunit.client.1.smithi135.stdout:0/614: creat d0/db5/db8/fc6 x:0 0 0 2024-03-30T04:28:24.433 INFO:tasks.workunit.client.1.smithi135.stdout:8/459: rmdir d6/d8/d15/d2a 39 2024-03-30T04:28:24.435 INFO:tasks.workunit.client.0.smithi123.stdout:6/981: dread d10/d20/f2b [0,4194304] 0 2024-03-30T04:28:24.435 INFO:tasks.workunit.client.0.smithi123.stdout:9/898: getdents d6/d1f/d72/d10d/d11c/d12e 0 2024-03-30T04:28:24.436 INFO:tasks.workunit.client.1.smithi135.stdout:2/402: symlink d4/d32/d12/d45/l84 0 2024-03-30T04:28:24.438 INFO:tasks.workunit.client.0.smithi123.stdout:6/982: chown d10/d82/d8e/dc7/f6c 73489 1 2024-03-30T04:28:24.439 INFO:tasks.workunit.client.1.smithi135.stdout:3/498: write d3/d50/d22/f71 [4677744,104686] 0 2024-03-30T04:28:24.440 INFO:tasks.workunit.client.0.smithi123.stdout:6/983: chown d10/d20/l32 1813526 1 2024-03-30T04:28:24.440 INFO:tasks.workunit.client.0.smithi123.stdout:9/899: creat d6/d49/d113/daa/f13b x:0 0 0 2024-03-30T04:28:24.441 INFO:tasks.workunit.client.1.smithi135.stdout:7/567: creat d4/db/d1a/d5c/d2c/d51/da8/fc2 x:0 0 0 2024-03-30T04:28:24.442 INFO:tasks.workunit.client.1.smithi135.stdout:8/460: truncate d6/d8/d15/d54/d4f/f6c 729436 0 2024-03-30T04:28:24.443 INFO:tasks.workunit.client.0.smithi123.stdout:8/926: dread da/d27/d3c/d2d/d74/f7f [0,4194304] 0 2024-03-30T04:28:24.445 INFO:tasks.workunit.client.0.smithi123.stdout:9/900: creat d6/d1f/d72/d7d/dfa/f13c x:0 0 0 2024-03-30T04:28:24.445 INFO:tasks.workunit.client.1.smithi135.stdout:0/615: mknod d0/d5/d42/d9f/cc7 0 2024-03-30T04:28:24.446 INFO:tasks.workunit.client.0.smithi123.stdout:6/984: getdents d10/d82/d8e/dc7/d5b/d13d 0 2024-03-30T04:28:24.447 INFO:tasks.workunit.client.1.smithi135.stdout:3/499: fsync d3/d10/d16/d23/f4b 0 2024-03-30T04:28:24.448 INFO:tasks.workunit.client.1.smithi135.stdout:0/616: fdatasync d0/d13/f74 0 2024-03-30T04:28:24.449 INFO:tasks.workunit.client.1.smithi135.stdout:7/568: mknod d4/db/d1a/d5c/d2c/d2e/cc3 0 2024-03-30T04:28:24.451 INFO:tasks.workunit.client.0.smithi123.stdout:8/927: getdents da/d27/d3c/d64/d78/db8/dfd/da2 0 2024-03-30T04:28:24.451 INFO:tasks.workunit.client.0.smithi123.stdout:6/985: truncate d10/d82/dc1/d6a/d7a/de2/df1/f10b 480212 0 2024-03-30T04:28:24.453 INFO:tasks.workunit.client.1.smithi135.stdout:5/461: dread d1/d45/f6b [0,4194304] 0 2024-03-30T04:28:24.453 INFO:tasks.workunit.client.1.smithi135.stdout:3/500: write d3/d10/f2d [2010486,43215] 0 2024-03-30T04:28:24.453 INFO:tasks.workunit.client.0.smithi123.stdout:9/901: link d6/d1f/d45/d4c/faf d6/d1f/d10c/f13d 0 2024-03-30T04:28:24.456 INFO:tasks.workunit.client.1.smithi135.stdout:0/617: write d0/d5/d9/f2e [1892625,121747] 0 2024-03-30T04:28:24.457 INFO:tasks.workunit.client.0.smithi123.stdout:8/928: chown da/cd3 1 1 2024-03-30T04:28:24.461 INFO:tasks.workunit.client.1.smithi135.stdout:6/548: dread d2/dd/f1a [0,4194304] 0 2024-03-30T04:28:24.462 INFO:tasks.workunit.client.0.smithi123.stdout:4/867: write d2/d4/fe7 [2551453,81459] 0 2024-03-30T04:28:24.463 INFO:tasks.workunit.client.1.smithi135.stdout:0/618: creat d0/d5/d9/d23/d8a/fc8 x:0 0 0 2024-03-30T04:28:24.465 INFO:tasks.workunit.client.1.smithi135.stdout:6/549: chown d2/d36/d76 10768411 1 2024-03-30T04:28:24.467 INFO:tasks.workunit.client.1.smithi135.stdout:5/462: rmdir d1/d7/d1a/d1f/d7f 0 2024-03-30T04:28:24.469 INFO:tasks.workunit.client.1.smithi135.stdout:6/550: fsync d2/d2e/f4c 0 2024-03-30T04:28:24.473 INFO:tasks.workunit.client.0.smithi123.stdout:6/986: dwrite d10/d82/dc1/d6a/d7a/de2/df1/f137 [0,4194304] 0 2024-03-30T04:28:24.477 INFO:tasks.workunit.client.1.smithi135.stdout:2/403: dread d4/d32/d12/d45/f2c [0,4194304] 0 2024-03-30T04:28:24.477 INFO:tasks.workunit.client.0.smithi123.stdout:6/987: stat d10/d82/dc1/d6a/cf3 0 2024-03-30T04:28:24.478 INFO:tasks.workunit.client.1.smithi135.stdout:5/463: rmdir d1/d3/d2b/d54/d5e 39 2024-03-30T04:28:24.479 INFO:tasks.workunit.client.1.smithi135.stdout:8/461: dwrite d6/d8/d14/f5e [0,4194304] 0 2024-03-30T04:28:24.480 INFO:tasks.workunit.client.1.smithi135.stdout:6/551: stat d2/dd/d14/d38/f4a 0 2024-03-30T04:28:24.480 INFO:tasks.workunit.client.0.smithi123.stdout:4/868: truncate d2/d4/fe7 3871287 0 2024-03-30T04:28:24.483 INFO:tasks.workunit.client.1.smithi135.stdout:7/569: dwrite d4/db/d1a/d5c/d2c/d51/d56/f9e [0,4194304] 0 2024-03-30T04:28:24.484 INFO:tasks.workunit.client.0.smithi123.stdout:1/981: dwrite db/dd/d5f/d6d/d85/d89/fdd [0,4194304] 0 2024-03-30T04:28:24.485 INFO:tasks.workunit.client.0.smithi123.stdout:0/924: dwrite d6/dd/d15/d28/dd3/ddd/fa0 [0,4194304] 0 2024-03-30T04:28:24.486 INFO:tasks.workunit.client.0.smithi123.stdout:6/988: chown d10/d82/dc1/d45/d111/f122 188 1 2024-03-30T04:28:24.486 INFO:tasks.workunit.client.1.smithi135.stdout:5/464: chown d1/c28 1 1 2024-03-30T04:28:24.488 INFO:tasks.workunit.client.0.smithi123.stdout:0/925: fsync d6/dd/d31/d35/fed 0 2024-03-30T04:28:24.488 INFO:tasks.workunit.client.0.smithi123.stdout:1/982: stat db/dd/d42/d8f/d78/fa0 0 2024-03-30T04:28:24.489 INFO:tasks.workunit.client.1.smithi135.stdout:6/552: write d2/dd/d14/d2b/d86/d8e/f97 [1537011,51032] 0 2024-03-30T04:28:24.492 INFO:tasks.workunit.client.1.smithi135.stdout:8/462: rmdir d6/d8/d14 39 2024-03-30T04:28:24.492 INFO:tasks.workunit.client.0.smithi123.stdout:1/983: readlink db/dd/d42/d46/d56/da2/de0/l11d 0 2024-03-30T04:28:24.494 INFO:tasks.workunit.client.1.smithi135.stdout:5/465: rmdir d1 39 2024-03-30T04:28:24.494 INFO:tasks.workunit.client.1.smithi135.stdout:3/501: dwrite d3/db/f77 [0,4194304] 0 2024-03-30T04:28:24.494 INFO:tasks.workunit.client.1.smithi135.stdout:2/404: mknod d4/d32/d2e/d2f/c85 0 2024-03-30T04:28:24.494 INFO:tasks.workunit.client.1.smithi135.stdout:7/570: write d4/db/d79/d92/faf [794492,70998] 0 2024-03-30T04:28:24.495 INFO:tasks.workunit.client.0.smithi123.stdout:0/926: creat d6/dd/d15/d28/dd3/ddd/f12a x:0 0 0 2024-03-30T04:28:24.495 INFO:tasks.workunit.client.0.smithi123.stdout:1/984: chown db/dd/d42/d46/d5d/f13f 968650552 1 2024-03-30T04:28:24.497 INFO:tasks.workunit.client.0.smithi123.stdout:1/985: stat db/dd/d5f/dae/c125 0 2024-03-30T04:28:24.499 INFO:tasks.workunit.client.0.smithi123.stdout:8/929: dread da/dc/d14/d18/f30 [0,4194304] 0 2024-03-30T04:28:24.500 INFO:tasks.workunit.client.1.smithi135.stdout:7/571: chown d4/db/d79/da9/f58 6438482 1 2024-03-30T04:28:24.506 INFO:tasks.workunit.client.0.smithi123.stdout:6/989: link d10/d82/dc1/dd2/ffd d10/f148 0 2024-03-30T04:28:24.507 INFO:tasks.workunit.client.1.smithi135.stdout:3/502: unlink d3/f92 0 2024-03-30T04:28:24.509 INFO:tasks.workunit.client.0.smithi123.stdout:8/930: rename da/dc/d14/cbb to da/d99/c126 0 2024-03-30T04:28:24.511 INFO:tasks.workunit.client.1.smithi135.stdout:2/405: rename d4/d6/f80 to d4/d6/d47/f86 0 2024-03-30T04:28:24.513 INFO:tasks.workunit.client.1.smithi135.stdout:7/572: mkdir d4/db/de/db0/dc4 0 2024-03-30T04:28:24.514 INFO:tasks.workunit.client.0.smithi123.stdout:8/931: getdents da/d27/d3c/d64/d78 0 2024-03-30T04:28:24.518 INFO:tasks.workunit.client.1.smithi135.stdout:8/463: symlink d6/d8/d15/d2a/d35/l88 0 2024-03-30T04:28:24.518 INFO:tasks.workunit.client.0.smithi123.stdout:8/932: rename da/d27/d3c/d64/c120 to da/dc/d14/d9d/dff/d122/c127 0 2024-03-30T04:28:24.519 INFO:tasks.workunit.client.1.smithi135.stdout:5/466: rename d1/d3/d2b/f72 to d1/da/f8c 0 2024-03-30T04:28:24.519 INFO:tasks.workunit.client.1.smithi135.stdout:3/503: symlink d3/d50/d22/d33/d7b/lae 0 2024-03-30T04:28:24.520 INFO:tasks.workunit.client.1.smithi135.stdout:2/406: creat d4/d55/d77/f87 x:0 0 0 2024-03-30T04:28:24.524 INFO:tasks.workunit.client.1.smithi135.stdout:3/504: fsync d3/d6e/f80 0 2024-03-30T04:28:24.530 INFO:tasks.workunit.client.1.smithi135.stdout:7/573: mknod d4/db/d1a/d5c/d82/cc5 0 2024-03-30T04:28:24.531 INFO:tasks.workunit.client.1.smithi135.stdout:6/553: getdents d2/dd/d18/d87 0 2024-03-30T04:28:24.532 INFO:tasks.workunit.client.1.smithi135.stdout:2/407: creat d4/da/d7a/f88 x:0 0 0 2024-03-30T04:28:24.532 INFO:tasks.workunit.client.1.smithi135.stdout:3/505: mkdir d3/d50/d22/d8b/daf 0 2024-03-30T04:28:24.533 INFO:tasks.workunit.client.1.smithi135.stdout:7/574: fsync d4/f16 0 2024-03-30T04:28:24.534 INFO:tasks.workunit.client.1.smithi135.stdout:3/506: fsync d3/d6e/f90 0 2024-03-30T04:28:24.536 INFO:tasks.workunit.client.1.smithi135.stdout:3/507: write d3/f24 [1107770,119600] 0 2024-03-30T04:28:24.543 INFO:tasks.workunit.client.1.smithi135.stdout:2/408: link d4/c7b d4/d55/d77/c89 0 2024-03-30T04:28:24.547 INFO:tasks.workunit.client.1.smithi135.stdout:2/409: chown d4/da/l68 682 1 2024-03-30T04:28:24.551 INFO:tasks.workunit.client.1.smithi135.stdout:8/464: dwrite d6/d8/d15/d2a/f33 [0,4194304] 0 2024-03-30T04:28:24.562 INFO:tasks.workunit.client.1.smithi135.stdout:5/467: dwrite d1/d3/d2b/f53 [4194304,4194304] 0 2024-03-30T04:28:24.562 INFO:tasks.workunit.client.1.smithi135.stdout:6/554: truncate d2/dd/d18/f73 2109595 0 2024-03-30T04:28:24.562 INFO:tasks.workunit.client.1.smithi135.stdout:8/465: write d6/d8/d14/f4e [771641,38653] 0 2024-03-30T04:28:24.563 INFO:tasks.workunit.client.1.smithi135.stdout:2/410: link d4/d6/f1a d4/d32/d12/d76/f8a 0 2024-03-30T04:28:24.569 INFO:tasks.workunit.client.0.smithi123.stdout:1/986: write db/dd/d42/d46/d51/d72/f120 [93303,71530] 0 2024-03-30T04:28:24.570 INFO:tasks.workunit.client.1.smithi135.stdout:3/508: dwrite d3/d50/f12 [0,4194304] 0 2024-03-30T04:28:24.573 INFO:tasks.workunit.client.1.smithi135.stdout:5/468: symlink d1/d3/d2b/d54/d6d/l8d 0 2024-03-30T04:28:24.577 INFO:tasks.workunit.client.1.smithi135.stdout:3/509: symlink d3/d50/d22/d74/lb0 0 2024-03-30T04:28:24.577 INFO:tasks.workunit.client.1.smithi135.stdout:5/469: fdatasync d1/f4f 0 2024-03-30T04:28:24.579 INFO:tasks.workunit.client.0.smithi123.stdout:1/987: creat db/dd/d42/d46/d51/f144 x:0 0 0 2024-03-30T04:28:24.579 INFO:tasks.workunit.client.0.smithi123.stdout:0/927: dwrite d6/dd/d15/d98/d4e/d69/fe2 [0,4194304] 0 2024-03-30T04:28:24.579 INFO:tasks.workunit.client.1.smithi135.stdout:5/470: dread - d1/d45/f88 zero size 2024-03-30T04:28:24.580 INFO:tasks.workunit.client.1.smithi135.stdout:3/510: write d3/d10/d16/d23/f7f [781165,77658] 0 2024-03-30T04:28:24.581 INFO:tasks.workunit.client.1.smithi135.stdout:5/471: stat d1/d3/d2b/d54 0 2024-03-30T04:28:24.581 INFO:tasks.workunit.client.0.smithi123.stdout:1/988: getdents db/d122 0 2024-03-30T04:28:24.582 INFO:tasks.workunit.client.1.smithi135.stdout:6/555: link d2/dd/d14/d38/f7e d2/dd/d14/fab 0 2024-03-30T04:28:24.583 INFO:tasks.workunit.client.1.smithi135.stdout:2/411: truncate d4/da/f44 683819 0 2024-03-30T04:28:24.584 INFO:tasks.workunit.client.0.smithi123.stdout:0/928: mknod d6/dd/d15/d28/d4b/d8a/c12b 0 2024-03-30T04:28:24.587 INFO:tasks.workunit.client.0.smithi123.stdout:1/989: chown db/dd/d42/d46/d5d/d62/c142 1 1 2024-03-30T04:28:24.587 INFO:tasks.workunit.client.1.smithi135.stdout:6/556: write d2/f99 [263657,28461] 0 2024-03-30T04:28:24.587 INFO:tasks.workunit.client.1.smithi135.stdout:3/511: write d3/db/f8d [469226,78666] 0 2024-03-30T04:28:24.587 INFO:tasks.workunit.client.1.smithi135.stdout:1/470: sync 2024-03-30T04:28:24.595 INFO:tasks.workunit.client.0.smithi123.stdout:0/929: dread d6/dd/d15/d28/d4b/f82 [0,4194304] 0 2024-03-30T04:28:24.595 INFO:tasks.workunit.client.1.smithi135.stdout:2/412: write d4/d6/f61 [4674392,6472] 0 2024-03-30T04:28:24.597 INFO:tasks.workunit.client.1.smithi135.stdout:1/471: symlink d16/d5a/d5f/d3d/la0 0 2024-03-30T04:28:24.598 INFO:tasks.workunit.client.0.smithi123.stdout:0/930: rmdir d6/dd/d15/d28/d4b/dad/df2 39 2024-03-30T04:28:24.599 INFO:tasks.workunit.client.0.smithi123.stdout:1/990: write db/dd/d42/d46/d51/f8d [626534,74264] 0 2024-03-30T04:28:24.600 INFO:tasks.workunit.client.0.smithi123.stdout:0/931: chown d6/dd/l108 29656542 1 2024-03-30T04:28:24.604 INFO:tasks.workunit.client.1.smithi135.stdout:5/472: rename d1/d3/d76/d7e to d1/d3/d2b/d54/d5e/d5c/d79/d8e 0 2024-03-30T04:28:24.604 INFO:tasks.workunit.client.0.smithi123.stdout:1/991: mkdir db/dd/d42/d46/d56/da2/de0/d145 0 2024-03-30T04:28:24.606 INFO:tasks.workunit.client.1.smithi135.stdout:5/473: chown d1/da/d4d 2124008675 1 2024-03-30T04:28:24.607 INFO:tasks.workunit.client.1.smithi135.stdout:1/472: dread d16/d29/d32/f58 [0,4194304] 0 2024-03-30T04:28:24.608 INFO:tasks.workunit.client.1.smithi135.stdout:2/413: mknod d4/d32/d2e/d2f/c8b 0 2024-03-30T04:28:24.610 INFO:tasks.workunit.client.1.smithi135.stdout:3/512: link d3/d50/d22/d33/d62/f66 d3/d50/d46/d4c/fb1 0 2024-03-30T04:28:24.614 INFO:tasks.workunit.client.0.smithi123.stdout:1/992: write db/dd/fdb [796432,117669] 0 2024-03-30T04:28:24.614 INFO:tasks.workunit.client.1.smithi135.stdout:3/513: fdatasync d3/d50/d46/f73 0 2024-03-30T04:28:24.615 INFO:tasks.workunit.client.1.smithi135.stdout:5/474: mkdir d1/d45/d52/d8f 0 2024-03-30T04:28:24.616 INFO:tasks.workunit.client.1.smithi135.stdout:1/473: getdents d16/d5a/d6b/d77 0 2024-03-30T04:28:24.617 INFO:tasks.workunit.client.0.smithi123.stdout:0/932: dwrite d6/dd/d15/d98/dd0/f10d [0,4194304] 0 2024-03-30T04:28:24.618 INFO:tasks.workunit.client.1.smithi135.stdout:6/557: dwrite d2/f99 [0,4194304] 0 2024-03-30T04:28:24.620 INFO:tasks.workunit.client.0.smithi123.stdout:0/933: readlink d6/dd/d15/d98/d5d/d7f/df3/d104/l10c 0 2024-03-30T04:28:24.621 INFO:tasks.workunit.client.0.smithi123.stdout:1/993: mknod db/dd/d5f/d6d/d85/d89/d90/d94/d9c/dfc/c146 0 2024-03-30T04:28:24.622 INFO:tasks.workunit.client.1.smithi135.stdout:5/475: fdatasync d1/d7/d11/d3c/f44 0 2024-03-30T04:28:24.626 INFO:tasks.workunit.client.1.smithi135.stdout:3/514: chown d3/d10/d16/d23/f9f 6 1 2024-03-30T04:28:24.626 INFO:tasks.workunit.client.1.smithi135.stdout:8/466: dwrite d6/d8/d14/f29 [0,4194304] 0 2024-03-30T04:28:24.627 INFO:tasks.workunit.client.0.smithi123.stdout:0/934: truncate d6/dd/d15/d28/d4b/d8a/f93 3411640 0 2024-03-30T04:28:24.630 INFO:tasks.workunit.client.0.smithi123.stdout:0/935: creat d6/dd/d15/d98/d5d/d7f/f12c x:0 0 0 2024-03-30T04:28:24.630 INFO:tasks.workunit.client.1.smithi135.stdout:6/558: write d2/d36/f6e [203185,105863] 0 2024-03-30T04:28:24.636 INFO:tasks.workunit.client.1.smithi135.stdout:5/476: creat d1/d45/d52/f90 x:0 0 0 2024-03-30T04:28:24.637 INFO:tasks.workunit.client.1.smithi135.stdout:2/414: rename d4/d55/d77/f87 to d4/d32/d2e/f8c 0 2024-03-30T04:28:24.637 INFO:tasks.workunit.client.1.smithi135.stdout:5/477: chown d1/d3 3545598 1 2024-03-30T04:28:24.646 INFO:tasks.workunit.client.0.smithi123.stdout:1/994: write db/dd/d42/d46/d5d/d62/d9e/fff [649652,39660] 0 2024-03-30T04:28:24.650 INFO:tasks.workunit.client.0.smithi123.stdout:1/995: fdatasync db/dd/d42/d46/d5d/d62/d9e/f12a 0 2024-03-30T04:28:24.653 INFO:tasks.workunit.client.1.smithi135.stdout:1/474: rename d16/d29/l74 to d16/d29/d2c/la1 0 2024-03-30T04:28:24.654 INFO:tasks.workunit.client.0.smithi123.stdout:1/996: read db/dd/d5f/d6d/d7e/ff0 [108658,8767] 0 2024-03-30T04:28:24.655 INFO:tasks.workunit.client.1.smithi135.stdout:1/475: chown d16/d19/d20/d99/l97 7083089 1 2024-03-30T04:28:24.656 INFO:tasks.workunit.client.0.smithi123.stdout:0/936: dwrite d6/dd/d15/d28/f33 [0,4194304] 0 2024-03-30T04:28:24.658 INFO:tasks.workunit.client.1.smithi135.stdout:8/467: rename d6/d8/d14/c75 to d6/d8/d14/d32/c89 0 2024-03-30T04:28:24.659 INFO:tasks.workunit.client.0.smithi123.stdout:1/997: creat db/dd/d42/d46/d56/da2/df1/f147 x:0 0 0 2024-03-30T04:28:24.662 INFO:tasks.workunit.client.0.smithi123.stdout:1/998: write db/dd/d42/d46/d56/d5c/f76 [2097023,53553] 0 2024-03-30T04:28:24.663 INFO:tasks.workunit.client.1.smithi135.stdout:1/476: read d16/d5a/d6b/d77/f7a [919570,37679] 0 2024-03-30T04:28:24.663 INFO:tasks.workunit.client.0.smithi123.stdout:0/937: creat d6/dd/d15/d28/d4b/dad/de8/d10f/d113/f12d x:0 0 0 2024-03-30T04:28:24.666 INFO:tasks.workunit.client.0.smithi123.stdout:1/999: write db/dd/d42/d46/d56/daf/fb5 [4430405,34478] 0 2024-03-30T04:28:24.666 INFO:tasks.workunit.client.0.smithi123.stdout:0/938: dread - d6/dd/d15/d98/d5d/d7f/df3/ffd zero size 2024-03-30T04:28:24.670 INFO:tasks.workunit.client.0.smithi123.stdout:0/939: creat d6/dd/d15/d98/d4e/d69/d6e/f12e x:0 0 0 2024-03-30T04:28:24.673 INFO:tasks.workunit.client.1.smithi135.stdout:6/559: dwrite d2/dd/d18/f22 [0,4194304] 0 2024-03-30T04:28:24.674 INFO:tasks.workunit.client.1.smithi135.stdout:2/415: dwrite d4/d32/d43/f6b [0,4194304] 0 2024-03-30T04:28:24.679 INFO:tasks.workunit.client.1.smithi135.stdout:1/477: creat d16/d29/d2c/d43/fa2 x:0 0 0 2024-03-30T04:28:24.680 INFO:tasks.workunit.client.1.smithi135.stdout:3/515: dwrite d3/d10/d16/d23/f35 [0,4194304] 0 2024-03-30T04:28:24.683 INFO:tasks.workunit.client.0.smithi123.stdout:0/940: dwrite d6/dd/d31/d35/fed [0,4194304] 0 2024-03-30T04:28:24.685 INFO:tasks.workunit.client.1.smithi135.stdout:6/560: symlink d2/dd/lac 0 2024-03-30T04:28:24.685 INFO:tasks.workunit.client.1.smithi135.stdout:3/516: write d3/d50/d46/f21 [781615,97568] 0 2024-03-30T04:28:24.690 INFO:tasks.workunit.client.1.smithi135.stdout:1/478: truncate d16/f2f 2080177 0 2024-03-30T04:28:24.690 INFO:tasks.workunit.client.1.smithi135.stdout:2/416: mkdir d4/da/d8d 0 2024-03-30T04:28:24.690 INFO:tasks.workunit.client.0.smithi123.stdout:5/879: sync 2024-03-30T04:28:24.692 INFO:tasks.workunit.client.0.smithi123.stdout:5/880: stat dd/d19 0 2024-03-30T04:28:24.693 INFO:tasks.workunit.client.0.smithi123.stdout:0/941: getdents d6/d52 0 2024-03-30T04:28:24.694 INFO:tasks.workunit.client.1.smithi135.stdout:3/517: write d3/d10/d16/d23/d6a/f9e [536185,9933] 0 2024-03-30T04:28:24.694 INFO:tasks.workunit.client.1.smithi135.stdout:1/479: stat d16/d5a/d5f/d3a/l47 0 2024-03-30T04:28:24.696 INFO:tasks.workunit.client.0.smithi123.stdout:5/881: chown dd/d19/d1f/d3c/d72/d8c/df4/f108 388329338 1 2024-03-30T04:28:24.698 INFO:tasks.workunit.client.0.smithi123.stdout:0/942: write d6/dd/d15/d98/d5d/d7f/df3/ffd [189185,62608] 0 2024-03-30T04:28:24.699 INFO:tasks.workunit.client.1.smithi135.stdout:2/417: read d4/d32/d2e/f4a [6356143,19550] 0 2024-03-30T04:28:24.700 INFO:tasks.workunit.client.1.smithi135.stdout:1/480: chown d16/d5a/d5f/d3a/d6f/f95 7 1 2024-03-30T04:28:24.702 INFO:tasks.workunit.client.1.smithi135.stdout:3/518: creat d3/d50/d22/d33/d7b/fb2 x:0 0 0 2024-03-30T04:28:24.703 INFO:tasks.workunit.client.0.smithi123.stdout:5/882: chown dd/d19/d1f/d35/d3a/d57/db5/lb9 0 1 2024-03-30T04:28:24.705 INFO:tasks.workunit.client.0.smithi123.stdout:0/943: truncate d6/dd/d31/db8/ff5 728998 0 2024-03-30T04:28:24.705 INFO:tasks.workunit.client.1.smithi135.stdout:8/468: dwrite d6/d8/f38 [0,4194304] 0 2024-03-30T04:28:24.705 INFO:tasks.workunit.client.1.smithi135.stdout:3/519: chown d3/d10/d16/f43 8250 1 2024-03-30T04:28:24.705 INFO:tasks.workunit.client.1.smithi135.stdout:1/481: symlink d16/d29/d2c/la3 0 2024-03-30T04:28:24.709 INFO:tasks.workunit.client.0.smithi123.stdout:5/883: unlink dd/d19/d1f/d35/d3a/d57/lbf 0 2024-03-30T04:28:24.709 INFO:tasks.workunit.client.0.smithi123.stdout:0/944: rename d6/dd/d15/d28 to d6/dd/d15/d28/d4b/d8a/dff/d12f 22 2024-03-30T04:28:24.712 INFO:tasks.workunit.client.1.smithi135.stdout:2/418: read d4/da/f59 [2273652,113228] 0 2024-03-30T04:28:24.714 INFO:tasks.workunit.client.0.smithi123.stdout:5/884: unlink dd/d19/f85 0 2024-03-30T04:28:24.716 INFO:tasks.workunit.client.1.smithi135.stdout:8/469: creat d6/d8/d15/f8a x:0 0 0 2024-03-30T04:28:24.717 INFO:tasks.workunit.client.0.smithi123.stdout:5/885: chown dd/d19/d5f/d99 215750315 1 2024-03-30T04:28:24.722 INFO:tasks.workunit.client.0.smithi123.stdout:0/945: creat d6/dd/d18/d1f/f130 x:0 0 0 2024-03-30T04:28:24.723 INFO:tasks.workunit.client.0.smithi123.stdout:5/886: chown dd/d19/d3e/d5b/d62/cf8 13617823 1 2024-03-30T04:28:24.723 INFO:tasks.workunit.client.1.smithi135.stdout:3/520: dread d3/db/f8f [0,4194304] 0 2024-03-30T04:28:24.724 INFO:tasks.workunit.client.0.smithi123.stdout:5/887: readlink dd/d19/d3e/l44 0 2024-03-30T04:28:24.724 INFO:tasks.workunit.client.1.smithi135.stdout:1/482: truncate d16/d19/f56 746707 0 2024-03-30T04:28:24.727 INFO:tasks.workunit.client.1.smithi135.stdout:8/470: unlink d6/d8/d15/d2a/c49 0 2024-03-30T04:28:24.731 INFO:tasks.workunit.client.0.smithi123.stdout:0/946: rename d6/dd/d31/d35/d114/f118 to d6/dd/d15/d28/dd3/f131 0 2024-03-30T04:28:24.732 INFO:tasks.workunit.client.0.smithi123.stdout:5/888: truncate dd/d19/d5f/d99/fbe 539191 0 2024-03-30T04:28:24.732 INFO:tasks.workunit.client.1.smithi135.stdout:2/419: creat d4/d32/d12/d45/f8e x:0 0 0 2024-03-30T04:28:24.735 INFO:tasks.workunit.client.1.smithi135.stdout:3/521: creat d3/fb3 x:0 0 0 2024-03-30T04:28:24.736 INFO:tasks.workunit.client.0.smithi123.stdout:5/889: creat dd/d19/d3e/d5b/d62/f128 x:0 0 0 2024-03-30T04:28:24.738 INFO:tasks.workunit.client.0.smithi123.stdout:0/947: rename d6/dd/d15/d28/d4b/l11f to d6/dd/d15/d98/d4e/d9a/db2/l132 0 2024-03-30T04:28:24.738 INFO:tasks.workunit.client.1.smithi135.stdout:2/420: write d4/d32/d2e/f4b [806394,101514] 0 2024-03-30T04:28:24.739 INFO:tasks.workunit.client.0.smithi123.stdout:5/890: mkdir dd/d19/d1f/d3c/d9e/df5/d129 0 2024-03-30T04:28:24.742 INFO:tasks.workunit.client.0.smithi123.stdout:0/948: write d6/dd/d15/d28/dd3/ddd/fec [934944,86135] 0 2024-03-30T04:28:24.742 INFO:tasks.workunit.client.1.smithi135.stdout:3/522: creat d3/db/fb4 x:0 0 0 2024-03-30T04:28:24.742 INFO:tasks.workunit.client.0.smithi123.stdout:5/891: chown dd/d19/d1f/l113 1885 1 2024-03-30T04:28:24.743 INFO:tasks.workunit.client.1.smithi135.stdout:1/483: getdents d16/d29 0 2024-03-30T04:28:24.744 INFO:tasks.workunit.client.0.smithi123.stdout:0/949: dread - d6/dd/d18/d2f/fc5 zero size 2024-03-30T04:28:24.746 INFO:tasks.workunit.client.0.smithi123.stdout:0/950: write d6/f9 [6772754,123885] 0 2024-03-30T04:28:24.747 INFO:tasks.workunit.client.0.smithi123.stdout:0/951: fdatasync d6/dd/f8c 0 2024-03-30T04:28:24.748 INFO:tasks.workunit.client.0.smithi123.stdout:5/892: creat dd/d19/d1f/d3c/d72/f12a x:0 0 0 2024-03-30T04:28:24.749 INFO:tasks.workunit.client.1.smithi135.stdout:1/484: write d16/d29/d2c/d43/fa2 [1000608,38484] 0 2024-03-30T04:28:24.749 INFO:tasks.workunit.client.1.smithi135.stdout:3/523: symlink d3/db/lb5 0 2024-03-30T04:28:24.750 INFO:tasks.workunit.client.0.smithi123.stdout:5/893: fdatasync dd/f15 0 2024-03-30T04:28:24.752 INFO:tasks.workunit.client.1.smithi135.stdout:1/485: readlink d16/d29/d2c/d3c/l34 0 2024-03-30T04:28:24.753 INFO:tasks.workunit.client.1.smithi135.stdout:6/561: truncate d2/dd/d14/d38/f4a 5757546 0 2024-03-30T04:28:24.754 INFO:tasks.workunit.client.0.smithi123.stdout:0/952: mknod d6/dd/d15/d98/d5d/d7f/df3/d104/c133 0 2024-03-30T04:28:24.756 INFO:tasks.workunit.client.0.smithi123.stdout:0/953: stat d6/dd/d15/d98/d4e/d9a/db2/fc9 0 2024-03-30T04:28:24.757 INFO:tasks.workunit.client.1.smithi135.stdout:8/471: dwrite d6/f25 [0,4194304] 0 2024-03-30T04:28:24.757 INFO:tasks.workunit.client.0.smithi123.stdout:7/873: sync 2024-03-30T04:28:24.758 INFO:tasks.workunit.client.1.smithi135.stdout:3/524: chown d3/db/c47 2338366 1 2024-03-30T04:28:24.761 INFO:tasks.workunit.client.1.smithi135.stdout:1/486: unlink d16/d29/d32/d85/l91 0 2024-03-30T04:28:24.762 INFO:tasks.workunit.client.0.smithi123.stdout:0/954: mkdir d6/dd/d15/d98/dd0/d134 0 2024-03-30T04:28:24.764 INFO:tasks.workunit.client.1.smithi135.stdout:1/487: readlink d16/d5a/l71 0 2024-03-30T04:28:24.765 INFO:tasks.workunit.client.0.smithi123.stdout:7/874: creat d2/da/d11/d34/ded/f125 x:0 0 0 2024-03-30T04:28:24.765 INFO:tasks.workunit.client.0.smithi123.stdout:0/955: chown d6/dd/d31/d35/d3a/f95 94901 1 2024-03-30T04:28:24.768 INFO:tasks.workunit.client.1.smithi135.stdout:3/525: mkdir d3/d10/d16/d23/db6 0 2024-03-30T04:28:24.769 INFO:tasks.workunit.client.1.smithi135.stdout:8/472: mkdir d6/d8/d15/d54/d69/d72/d8b 0 2024-03-30T04:28:24.779 INFO:tasks.workunit.client.1.smithi135.stdout:3/526: mknod d3/d10/d16/d23/d6a/d86/cb7 0 2024-03-30T04:28:24.783 INFO:tasks.workunit.client.1.smithi135.stdout:8/473: getdents d6/d8 0 2024-03-30T04:28:24.784 INFO:tasks.workunit.client.1.smithi135.stdout:2/421: dread d4/d32/d12/f2a [0,4194304] 0 2024-03-30T04:28:24.785 INFO:tasks.workunit.client.1.smithi135.stdout:6/562: dwrite d2/dd/d14/f15 [0,4194304] 0 2024-03-30T04:28:24.786 INFO:tasks.workunit.client.0.smithi123.stdout:5/894: dwrite dd/d19/f28 [0,4194304] 0 2024-03-30T04:28:24.792 INFO:tasks.workunit.client.0.smithi123.stdout:7/875: dwrite d2/da/d11/f1e [0,4194304] 0 2024-03-30T04:28:24.793 INFO:tasks.workunit.client.1.smithi135.stdout:3/527: fdatasync d3/d50/d46/f58 0 2024-03-30T04:28:24.795 INFO:tasks.workunit.client.0.smithi123.stdout:0/956: dwrite d6/dd/d15/d28/fda [0,4194304] 0 2024-03-30T04:28:24.795 INFO:tasks.workunit.client.1.smithi135.stdout:2/422: symlink d4/d55/d77/l8f 0 2024-03-30T04:28:24.796 INFO:tasks.workunit.client.1.smithi135.stdout:4/504: sync 2024-03-30T04:28:24.796 INFO:tasks.workunit.client.1.smithi135.stdout:1/488: dwrite d16/d5a/d5f/d3a/f52 [4194304,4194304] 0 2024-03-30T04:28:24.798 INFO:tasks.workunit.client.0.smithi123.stdout:5/895: creat dd/d19/d1f/d35/f12b x:0 0 0 2024-03-30T04:28:24.799 INFO:tasks.workunit.client.1.smithi135.stdout:4/505: readlink d10/d1a/d32/l44 0 2024-03-30T04:28:24.799 INFO:tasks.workunit.client.1.smithi135.stdout:8/474: creat d6/f8c x:0 0 0 2024-03-30T04:28:24.800 INFO:tasks.workunit.client.0.smithi123.stdout:9/902: sync 2024-03-30T04:28:24.800 INFO:tasks.workunit.client.0.smithi123.stdout:8/933: sync 2024-03-30T04:28:24.800 INFO:tasks.workunit.client.0.smithi123.stdout:7/876: readlink d2/da/d11/d34/d85/d8e/lf0 0 2024-03-30T04:28:24.800 INFO:tasks.workunit.client.0.smithi123.stdout:6/990: sync 2024-03-30T04:28:24.800 INFO:tasks.workunit.client.0.smithi123.stdout:4/869: sync 2024-03-30T04:28:24.801 INFO:tasks.workunit.client.1.smithi135.stdout:4/506: stat d10/d1a/d32/l60 0 2024-03-30T04:28:24.801 INFO:tasks.workunit.client.1.smithi135.stdout:2/423: stat d4/d32/d2e/f48 0 2024-03-30T04:28:24.802 INFO:tasks.workunit.client.0.smithi123.stdout:9/903: dread - d6/d1f/d72/db6/fde zero size 2024-03-30T04:28:24.805 INFO:tasks.workunit.client.1.smithi135.stdout:1/489: write d16/d29/d2c/d43/fa2 [300303,81223] 0 2024-03-30T04:28:24.810 INFO:tasks.workunit.client.0.smithi123.stdout:4/870: symlink d2/dfb/d10c/l120 0 2024-03-30T04:28:24.811 INFO:tasks.workunit.client.1.smithi135.stdout:6/563: dread d2/f4 [0,4194304] 0 2024-03-30T04:28:24.818 INFO:tasks.workunit.client.0.smithi123.stdout:8/934: rename da/dc/d14/f6e to da/d27/d3c/d64/f128 0 2024-03-30T04:28:24.822 INFO:tasks.workunit.client.1.smithi135.stdout:2/424: chown d4/d32/d12/d45/d3c/c66 21577 1 2024-03-30T04:28:24.823 INFO:tasks.workunit.client.1.smithi135.stdout:3/528: rename d3/d10/d16/d23/f35 to d3/d50/d46/fb8 0 2024-03-30T04:28:24.823 INFO:tasks.workunit.client.0.smithi123.stdout:9/904: rename d6/d1f/f70 to d6/d1f/d72/d7d/dfa/f13e 0 2024-03-30T04:28:24.824 INFO:tasks.workunit.client.0.smithi123.stdout:8/935: unlink da/d27/d56/d6c/d96/ffa 0 2024-03-30T04:28:24.824 INFO:tasks.workunit.client.1.smithi135.stdout:3/529: stat d3/db/c47 0 2024-03-30T04:28:24.825 INFO:tasks.workunit.client.0.smithi123.stdout:6/991: dread d10/d82/d8e/dc7/d42/f87 [0,4194304] 0 2024-03-30T04:28:24.825 INFO:tasks.workunit.client.1.smithi135.stdout:4/507: mkdir d10/d72/d85/db0 0 2024-03-30T04:28:24.827 INFO:tasks.workunit.client.1.smithi135.stdout:8/475: creat d6/d8/d15/d54/d69/d72/d73/f8d x:0 0 0 2024-03-30T04:28:24.828 INFO:tasks.workunit.client.0.smithi123.stdout:6/992: readlink d10/d20/da2/l10c 0 2024-03-30T04:28:24.828 INFO:tasks.workunit.client.1.smithi135.stdout:6/564: readlink d2/dd/d14/d2b/l65 0 2024-03-30T04:28:24.832 INFO:tasks.workunit.client.0.smithi123.stdout:6/993: stat d10/d82/d8e/dc7/d5b/d103 0 2024-03-30T04:28:24.833 INFO:tasks.workunit.client.0.smithi123.stdout:0/957: dread d6/f72 [0,4194304] 0 2024-03-30T04:28:24.835 INFO:tasks.workunit.client.0.smithi123.stdout:4/871: dwrite d2/d34/f93 [0,4194304] 0 2024-03-30T04:28:24.835 INFO:tasks.workunit.client.1.smithi135.stdout:3/530: chown d3/d54/d5c/f78 29478 1 2024-03-30T04:28:24.838 INFO:tasks.workunit.client.0.smithi123.stdout:6/994: creat d10/d82/dc1/d6a/d7a/d9f/f149 x:0 0 0 2024-03-30T04:28:24.839 INFO:tasks.workunit.client.0.smithi123.stdout:4/872: creat d2/d4/d7/dc1/d113/f121 x:0 0 0 2024-03-30T04:28:24.841 INFO:tasks.workunit.client.0.smithi123.stdout:0/958: unlink d6/dd/d15/d28/dd3/fd6 0 2024-03-30T04:28:24.841 INFO:tasks.workunit.client.1.smithi135.stdout:4/508: symlink d10/d4d/d89/d9f/daa/lb1 0 2024-03-30T04:28:24.842 INFO:tasks.workunit.client.1.smithi135.stdout:3/531: creat d3/d10/d16/d23/d6a/d94/fb9 x:0 0 0 2024-03-30T04:28:24.843 INFO:tasks.workunit.client.0.smithi123.stdout:6/995: creat d10/d82/d8e/dc7/d5b/d103/d115/d119/f14a x:0 0 0 2024-03-30T04:28:24.845 INFO:tasks.workunit.client.0.smithi123.stdout:4/873: creat d2/d4/db3/f122 x:0 0 0 2024-03-30T04:28:24.845 INFO:tasks.workunit.client.1.smithi135.stdout:1/490: rename d16/d19/d20 to d16/d19/da4 0 2024-03-30T04:28:24.847 INFO:tasks.workunit.client.1.smithi135.stdout:3/532: truncate d3/d50/f1f 3301467 0 2024-03-30T04:28:24.848 INFO:tasks.workunit.client.1.smithi135.stdout:2/425: getdents d4/d55 0 2024-03-30T04:28:24.850 INFO:tasks.workunit.client.0.smithi123.stdout:5/896: write dd/d19/d1f/d35/d3a/d57/db5/fd6 [109754,18341] 0 2024-03-30T04:28:24.850 INFO:tasks.workunit.client.0.smithi123.stdout:7/877: write d2/da/f4c [1406693,123420] 0 2024-03-30T04:28:24.851 INFO:tasks.workunit.client.0.smithi123.stdout:4/874: truncate d2/d4/d2c/d42/d35/d5d/d78/fef 2423124 0 2024-03-30T04:28:24.851 INFO:tasks.workunit.client.1.smithi135.stdout:6/565: mkdir d2/dd/d40/d52/d95/dad 0 2024-03-30T04:28:24.853 INFO:tasks.workunit.client.1.smithi135.stdout:3/533: creat d3/d10/d16/d23/d6a/d94/fba x:0 0 0 2024-03-30T04:28:24.854 INFO:tasks.workunit.client.0.smithi123.stdout:0/959: getdents d6/dd/d15/d28/d4b 0 2024-03-30T04:28:24.855 INFO:tasks.workunit.client.0.smithi123.stdout:6/996: getdents d10/d82/dc1/d6a/d7a/d99/dba 0 2024-03-30T04:28:24.857 INFO:tasks.workunit.client.1.smithi135.stdout:4/509: link d10/d1a/d42/d46/l4a d10/d1a/d42/d66/lb2 0 2024-03-30T04:28:24.857 INFO:tasks.workunit.client.0.smithi123.stdout:4/875: write d2/d4/d2c/d42/d35/d5d/d78/f83 [2737944,41402] 0 2024-03-30T04:28:24.859 INFO:tasks.workunit.client.0.smithi123.stdout:4/876: chown d2/d26/c6d 65376342 1 2024-03-30T04:28:24.861 INFO:tasks.workunit.client.1.smithi135.stdout:6/566: mkdir d2/dd/d40/d52/d95/dad/dae 0 2024-03-30T04:28:24.863 INFO:tasks.workunit.client.1.smithi135.stdout:4/510: unlink d10/d1a/d42/f67 0 2024-03-30T04:28:24.866 INFO:tasks.workunit.client.1.smithi135.stdout:2/426: getdents d4/d32/d2e 0 2024-03-30T04:28:24.866 INFO:tasks.workunit.client.1.smithi135.stdout:4/511: dread - d10/d72/fa6 zero size 2024-03-30T04:28:24.866 INFO:tasks.workunit.client.0.smithi123.stdout:9/905: write d6/d1f/d72/d7d/fe2 [976364,37171] 0 2024-03-30T04:28:24.867 INFO:tasks.workunit.client.1.smithi135.stdout:1/491: dwrite d16/d5a/d5f/d3a/d6f/f7c [0,4194304] 0 2024-03-30T04:28:24.867 INFO:tasks.workunit.client.0.smithi123.stdout:7/878: rename d2/da/d11/d61/d55/d59/f105 to d2/dc/d49/f126 0 2024-03-30T04:28:24.867 INFO:tasks.workunit.client.0.smithi123.stdout:8/936: truncate da/dc/d14/d9d/fb1 2217077 0 2024-03-30T04:28:24.872 INFO:tasks.workunit.client.0.smithi123.stdout:9/906: rmdir d6/d1f/d92/db8 39 2024-03-30T04:28:24.873 INFO:tasks.workunit.client.1.smithi135.stdout:6/567: mkdir d2/d36/d76/daf 0 2024-03-30T04:28:24.873 INFO:tasks.workunit.client.1.smithi135.stdout:2/427: unlink d4/d32/d12/l69 0 2024-03-30T04:28:24.875 INFO:tasks.workunit.client.0.smithi123.stdout:8/937: symlink da/dc/l129 0 2024-03-30T04:28:24.877 INFO:tasks.workunit.client.1.smithi135.stdout:2/428: stat d4/d32/l3a 0 2024-03-30T04:28:24.879 INFO:tasks.workunit.client.0.smithi123.stdout:0/960: write d6/dd/d18/d2f/fc5 [125209,96467] 0 2024-03-30T04:28:24.879 INFO:tasks.workunit.client.1.smithi135.stdout:8/476: rename d6/d8/d15/d54/d69 to d6/d8/d15/d54/d8e 0 2024-03-30T04:28:24.880 INFO:tasks.workunit.client.1.smithi135.stdout:0/619: sync 2024-03-30T04:28:24.880 INFO:tasks.workunit.client.1.smithi135.stdout:9/447: sync 2024-03-30T04:28:24.882 INFO:tasks.workunit.client.1.smithi135.stdout:6/568: mkdir d2/dd/d18/db0 0 2024-03-30T04:28:24.884 INFO:tasks.workunit.client.1.smithi135.stdout:2/429: symlink d4/da/d8d/l90 0 2024-03-30T04:28:24.890 INFO:tasks.workunit.client.1.smithi135.stdout:0/620: stat d0/d5/d9/d23 0 2024-03-30T04:28:24.891 INFO:tasks.workunit.client.0.smithi123.stdout:4/877: dwrite d2/d4/d2c/f67 [0,4194304] 0 2024-03-30T04:28:24.892 INFO:tasks.workunit.client.0.smithi123.stdout:7/879: write d2/dc/d56/f67 [1440980,6732] 0 2024-03-30T04:28:24.893 INFO:tasks.workunit.client.1.smithi135.stdout:3/534: rmdir d3/d10/d16/d23 39 2024-03-30T04:28:24.897 INFO:tasks.workunit.client.1.smithi135.stdout:9/448: creat d7/d4f/d5c/f9e x:0 0 0 2024-03-30T04:28:24.897 INFO:tasks.workunit.client.0.smithi123.stdout:0/961: mkdir d6/dd/d15/d28/d4b/dad/de8/d10f/d113/d135 0 2024-03-30T04:28:24.898 INFO:tasks.workunit.client.0.smithi123.stdout:6/997: dread d10/d82/d8e/dc7/f1e [0,4194304] 0 2024-03-30T04:28:24.901 INFO:tasks.workunit.client.1.smithi135.stdout:2/430: creat d4/d32/d46/f91 x:0 0 0 2024-03-30T04:28:24.902 INFO:tasks.workunit.client.0.smithi123.stdout:5/897: dwrite dd/d19/d3e/d5b/d62/f7a [0,4194304] 0 2024-03-30T04:28:24.902 INFO:tasks.workunit.client.0.smithi123.stdout:0/962: dread - d6/dd/d15/d28/d4b/dad/de8/d10f/d113/f12d zero size 2024-03-30T04:28:24.903 INFO:tasks.workunit.client.1.smithi135.stdout:9/449: chown d7/d41/f5f 35718 1 2024-03-30T04:28:24.904 INFO:tasks.workunit.client.1.smithi135.stdout:1/492: dwrite d16/f45 [0,4194304] 0 2024-03-30T04:28:24.907 INFO:tasks.workunit.client.0.smithi123.stdout:5/898: chown dd/d19/d3e/d5b/fa2 520401532 1 2024-03-30T04:28:24.907 INFO:tasks.workunit.client.1.smithi135.stdout:6/569: creat d2/dd/d14/d2b/d86/d55/fb1 x:0 0 0 2024-03-30T04:28:24.907 INFO:tasks.workunit.client.1.smithi135.stdout:3/535: rename d3/d10 to d3/d10/d16/dbb 22 2024-03-30T04:28:24.908 INFO:tasks.workunit.client.0.smithi123.stdout:8/938: truncate f6 1689488 0 2024-03-30T04:28:24.912 INFO:tasks.workunit.client.0.smithi123.stdout:5/899: mknod dd/d19/d1f/d35/d3a/d11f/c12c 0 2024-03-30T04:28:24.913 INFO:tasks.workunit.client.1.smithi135.stdout:4/512: dwrite d10/d1a/d42/d66/f84 [0,4194304] 0 2024-03-30T04:28:24.919 INFO:tasks.workunit.client.0.smithi123.stdout:5/900: symlink dd/d19/d3e/d5b/dd7/l12d 0 2024-03-30T04:28:24.928 INFO:tasks.workunit.client.1.smithi135.stdout:6/570: write d2/dd/d14/d2b/d86/d55/f7b [869815,27174] 0 2024-03-30T04:28:24.929 INFO:tasks.workunit.client.0.smithi123.stdout:6/998: dwrite d10/d82/dd6/f147 [0,4194304] 0 2024-03-30T04:28:24.930 INFO:tasks.workunit.client.1.smithi135.stdout:8/477: dwrite d6/f25 [0,4194304] 0 2024-03-30T04:28:24.930 INFO:tasks.workunit.client.1.smithi135.stdout:1/493: symlink d16/d19/da4/la5 0 2024-03-30T04:28:24.930 INFO:tasks.workunit.client.1.smithi135.stdout:9/450: creat d7/d41/d72/f9f x:0 0 0 2024-03-30T04:28:24.931 INFO:tasks.workunit.client.1.smithi135.stdout:7/575: sync 2024-03-30T04:28:24.934 INFO:tasks.workunit.client.1.smithi135.stdout:0/621: write d0/d13/d1e/f46 [902380,119643] 0 2024-03-30T04:28:24.935 INFO:tasks.workunit.client.1.smithi135.stdout:5/478: sync 2024-03-30T04:28:24.935 INFO:tasks.workunit.client.1.smithi135.stdout:4/513: write d10/d1a/d42/d46/f9d [1830423,87206] 0 2024-03-30T04:28:24.938 INFO:tasks.workunit.client.0.smithi123.stdout:8/939: dwrite da/d27/d3c/d2d/d5c/f10d [0,4194304] 0 2024-03-30T04:28:24.939 INFO:tasks.workunit.client.0.smithi123.stdout:7/880: write d2/dc/d83/ffa [442321,95542] 0 2024-03-30T04:28:24.941 INFO:tasks.workunit.client.1.smithi135.stdout:6/571: chown d2/dd/d14/d30/d43/f9d 29 1 2024-03-30T04:28:24.946 INFO:tasks.workunit.client.0.smithi123.stdout:7/881: mkdir d2/d4d/d127 0 2024-03-30T04:28:24.948 INFO:tasks.workunit.client.1.smithi135.stdout:8/478: mkdir d6/d8/d8f 0 2024-03-30T04:28:24.948 INFO:tasks.workunit.client.0.smithi123.stdout:7/882: chown d2/dc/d49/l93 6677 1 2024-03-30T04:28:24.950 INFO:tasks.workunit.client.0.smithi123.stdout:8/940: creat da/d27/d3c/d64/d78/f12a x:0 0 0 2024-03-30T04:28:24.953 INFO:tasks.workunit.client.1.smithi135.stdout:5/479: mknod d1/d45/c91 0 2024-03-30T04:28:24.955 INFO:tasks.workunit.client.0.smithi123.stdout:7/883: unlink d2/dc/d56/f116 0 2024-03-30T04:28:24.955 INFO:tasks.workunit.client.0.smithi123.stdout:0/963: truncate d6/dd/f4a 678410 0 2024-03-30T04:28:24.957 INFO:tasks.workunit.client.1.smithi135.stdout:5/480: chown d1/da/f1b 48105184 1 2024-03-30T04:28:24.961 INFO:tasks.workunit.client.1.smithi135.stdout:7/576: fsync d4/db/de/d5a/f68 0 2024-03-30T04:28:24.964 INFO:tasks.workunit.client.1.smithi135.stdout:7/577: write d4/db/de/f2b [4675143,7771] 0 2024-03-30T04:28:24.965 INFO:tasks.workunit.client.0.smithi123.stdout:8/941: getdents da/d27/d95 0 2024-03-30T04:28:24.969 INFO:tasks.workunit.client.1.smithi135.stdout:1/494: dread d16/d29/d32/f38 [0,4194304] 0 2024-03-30T04:28:24.970 INFO:tasks.workunit.client.0.smithi123.stdout:6/999: write d10/d82/dc1/d6a/d7a/d99/dba/fde [1054623,87709] 0 2024-03-30T04:28:24.972 INFO:tasks.workunit.client.1.smithi135.stdout:9/451: dread d7/d1b/f7c [0,4194304] 0 2024-03-30T04:28:24.972 INFO:tasks.workunit.client.1.smithi135.stdout:1/495: stat d16/d5a/d6b/d77/f7a 0 2024-03-30T04:28:24.973 INFO:tasks.workunit.client.1.smithi135.stdout:6/572: rmdir d2/d36/d76/daf 0 2024-03-30T04:28:24.974 INFO:tasks.workunit.client.0.smithi123.stdout:7/884: dread d2/dc/d115/f9d [0,4194304] 0 2024-03-30T04:28:24.976 INFO:tasks.workunit.client.0.smithi123.stdout:4/878: dread d2/d34/f88 [0,4194304] 0 2024-03-30T04:28:24.978 INFO:tasks.workunit.client.0.smithi123.stdout:5/901: dread dd/d19/d1f/d35/d3a/f45 [0,4194304] 0 2024-03-30T04:28:24.979 INFO:tasks.workunit.client.0.smithi123.stdout:7/885: chown d2/dc/d7f/c10b 3 1 2024-03-30T04:28:24.980 INFO:tasks.workunit.client.0.smithi123.stdout:4/879: write d2/d34/f7c [4752820,40665] 0 2024-03-30T04:28:24.988 INFO:tasks.workunit.client.0.smithi123.stdout:8/942: write da/dc/d14/d9d/fb1 [1866775,49070] 0 2024-03-30T04:28:24.988 INFO:tasks.workunit.client.1.smithi135.stdout:4/514: link d10/d1a/d42/d46/l75 d10/d4d/d9c/lb3 0 2024-03-30T04:28:24.990 INFO:tasks.workunit.client.0.smithi123.stdout:7/886: unlink d2/da/d11/d61/ff7 0 2024-03-30T04:28:24.991 INFO:tasks.workunit.client.1.smithi135.stdout:0/622: getdents d0/d13/d53/d7f/d83 0 2024-03-30T04:28:24.993 INFO:tasks.workunit.client.1.smithi135.stdout:6/573: creat d2/dd/d14/d2b/d86/d55/fb2 x:0 0 0 2024-03-30T04:28:24.994 INFO:tasks.workunit.client.0.smithi123.stdout:0/964: dwrite d6/dd/d15/f2b [0,4194304] 0 2024-03-30T04:28:24.994 INFO:tasks.workunit.client.1.smithi135.stdout:0/623: chown d0/l6a 1411550 1 2024-03-30T04:28:24.996 INFO:tasks.workunit.client.0.smithi123.stdout:7/887: creat d2/da/d11/d61/d55/d59/db5/dab/dca/f128 x:0 0 0 2024-03-30T04:28:24.997 INFO:tasks.workunit.client.1.smithi135.stdout:7/578: creat d4/db/d1a/d4b/d40/d6e/fc6 x:0 0 0 2024-03-30T04:28:24.998 INFO:tasks.workunit.client.1.smithi135.stdout:6/574: chown d2/d2e/f4c 12 1 2024-03-30T04:28:24.999 INFO:tasks.workunit.client.1.smithi135.stdout:2/431: dread d4/d6/f1f [0,4194304] 0 2024-03-30T04:28:24.999 INFO:tasks.workunit.client.1.smithi135.stdout:9/452: dwrite d7/d90/f9c [0,4194304] 0 2024-03-30T04:28:25.000 INFO:tasks.workunit.client.1.smithi135.stdout:7/579: read - d4/f8c zero size 2024-03-30T04:28:25.001 INFO:tasks.workunit.client.0.smithi123.stdout:4/880: link d2/d4/d2c/d42/ca9 d2/d4/d2c/d42/d35/d5d/c123 0 2024-03-30T04:28:25.003 INFO:tasks.workunit.client.1.smithi135.stdout:1/496: mknod d16/d5a/d5f/d3a/ca6 0 2024-03-30T04:28:25.004 INFO:tasks.workunit.client.0.smithi123.stdout:7/888: creat d2/da/d11/d34/d85/d8e/f129 x:0 0 0 2024-03-30T04:28:25.005 INFO:tasks.workunit.client.1.smithi135.stdout:7/580: write d4/db/d1a/d4b/d40/d6e/fc6 [945739,40452] 0 2024-03-30T04:28:25.006 INFO:tasks.workunit.client.0.smithi123.stdout:0/965: getdents d6/dd/d15/d98/d5d/d7f/df3 0 2024-03-30T04:28:25.007 INFO:tasks.workunit.client.1.smithi135.stdout:1/497: write d16/d5a/f92 [456458,79979] 0 2024-03-30T04:28:25.008 INFO:tasks.workunit.client.0.smithi123.stdout:7/889: write d2/f13 [3788969,61470] 0 2024-03-30T04:28:25.011 INFO:tasks.workunit.client.0.smithi123.stdout:5/902: dwrite dd/d19/d5f/d99/fe5 [4194304,4194304] 0 2024-03-30T04:28:25.012 INFO:tasks.workunit.client.1.smithi135.stdout:5/481: getdents d1/d3/d76 0 2024-03-30T04:28:25.012 INFO:tasks.workunit.client.1.smithi135.stdout:6/575: creat d2/dd/d14/d53/fb3 x:0 0 0 2024-03-30T04:28:25.013 INFO:tasks.workunit.client.0.smithi123.stdout:7/890: chown d2/da/d22/d3b/f4b 3302739 1 2024-03-30T04:28:25.013 INFO:tasks.workunit.client.0.smithi123.stdout:5/903: readlink dd/d19/d1f/d68/lc0 0 2024-03-30T04:28:25.016 INFO:tasks.workunit.client.1.smithi135.stdout:1/498: read d16/f2f [272439,44263] 0 2024-03-30T04:28:25.017 INFO:tasks.workunit.client.0.smithi123.stdout:9/907: sync 2024-03-30T04:28:25.017 INFO:tasks.workunit.client.1.smithi135.stdout:2/432: rename d4/d32/d2e/f4a to d4/d32/d43/f92 0 2024-03-30T04:28:25.020 INFO:tasks.workunit.client.1.smithi135.stdout:9/453: chown d7/d55/f5b 52467360 1 2024-03-30T04:28:25.020 INFO:tasks.workunit.client.1.smithi135.stdout:5/482: chown d1/d7/d11/f30 16980198 1 2024-03-30T04:28:25.022 INFO:tasks.workunit.client.1.smithi135.stdout:0/624: link d0/d13/d1e/d3a/f61 d0/d5/d9/d23/d58/d63/d77/d86/fc9 0 2024-03-30T04:28:25.024 INFO:tasks.workunit.client.0.smithi123.stdout:9/908: creat d6/f13f x:0 0 0 2024-03-30T04:28:25.025 INFO:tasks.workunit.client.1.smithi135.stdout:3/536: write d3/d50/f1f [3672055,89650] 0 2024-03-30T04:28:25.027 INFO:tasks.workunit.client.1.smithi135.stdout:0/625: readlink d0/d13/d53/lbd 0 2024-03-30T04:28:25.033 INFO:tasks.workunit.client.0.smithi123.stdout:8/943: dread da/dc/d14/d18/f86 [0,4194304] 0 2024-03-30T04:28:25.034 INFO:tasks.workunit.client.1.smithi135.stdout:6/576: stat d2/dd/c51 0 2024-03-30T04:28:25.034 INFO:tasks.workunit.client.1.smithi135.stdout:5/483: mkdir d1/d3/d76/d92 0 2024-03-30T04:28:25.035 INFO:tasks.workunit.client.0.smithi123.stdout:8/944: stat da/dc/da9/dc4 0 2024-03-30T04:28:25.036 INFO:tasks.workunit.client.1.smithi135.stdout:4/515: dread d10/f57 [0,4194304] 0 2024-03-30T04:28:25.037 INFO:tasks.workunit.client.0.smithi123.stdout:9/909: read d6/fa9 [2563651,14247] 0 2024-03-30T04:28:25.038 INFO:tasks.workunit.client.1.smithi135.stdout:2/433: mknod d4/da/c93 0 2024-03-30T04:28:25.038 INFO:tasks.workunit.client.0.smithi123.stdout:0/966: write d6/dd/d15/d98/d4e/d69/d6e/fd4 [358122,117961] 0 2024-03-30T04:28:25.040 INFO:tasks.workunit.client.1.smithi135.stdout:5/484: chown d1/da/f18 46670902 1 2024-03-30T04:28:25.041 INFO:tasks.workunit.client.0.smithi123.stdout:8/945: creat da/dc/d14/d9d/f12b x:0 0 0 2024-03-30T04:28:25.043 INFO:tasks.workunit.client.0.smithi123.stdout:9/910: fdatasync d6/d1f/d4d/f5b 0 2024-03-30T04:28:25.044 INFO:tasks.workunit.client.0.smithi123.stdout:5/904: dread dd/d19/d3e/d5b/f5e [0,4194304] 0 2024-03-30T04:28:25.044 INFO:tasks.workunit.client.1.smithi135.stdout:5/485: chown d1/d55 3 1 2024-03-30T04:28:25.045 INFO:tasks.workunit.client.1.smithi135.stdout:9/454: dwrite d7/d1b/d37/f4a [0,4194304] 0 2024-03-30T04:28:25.045 INFO:tasks.workunit.client.0.smithi123.stdout:8/946: symlink da/d27/d56/d93/l12c 0 2024-03-30T04:28:25.046 INFO:tasks.workunit.client.1.smithi135.stdout:1/499: rename d16/d5a/d5f/f66 to d16/d5a/d5f/d3a/fa7 0 2024-03-30T04:28:25.047 INFO:tasks.workunit.client.0.smithi123.stdout:0/967: mknod d6/dd/d15/d98/d5d/d7f/df3/d104/c136 0 2024-03-30T04:28:25.049 INFO:tasks.workunit.client.0.smithi123.stdout:4/881: dwrite d2/d34/f88 [0,4194304] 0 2024-03-30T04:28:25.053 INFO:tasks.workunit.client.0.smithi123.stdout:7/891: dwrite d2/dc/d56/fdd [0,4194304] 0 2024-03-30T04:28:25.053 INFO:tasks.workunit.client.1.smithi135.stdout:4/516: creat d10/d1a/d32/d40/d92/fb4 x:0 0 0 2024-03-30T04:28:25.054 INFO:tasks.workunit.client.0.smithi123.stdout:8/947: creat da/dc/d14/db7/f12d x:0 0 0 2024-03-30T04:28:25.056 INFO:tasks.workunit.client.0.smithi123.stdout:0/968: dread - d6/dd/d15/d98/d4e/d69/d6e/dd8/fe7 zero size 2024-03-30T04:28:25.057 INFO:tasks.workunit.client.0.smithi123.stdout:4/882: fsync d2/d4/d7/dc1/fde 0 2024-03-30T04:28:25.061 INFO:tasks.workunit.client.1.smithi135.stdout:5/486: unlink d1/d7/d11/l41 0 2024-03-30T04:28:25.062 INFO:tasks.workunit.client.1.smithi135.stdout:0/626: truncate d0/d5/d9/f2e 2117039 0 2024-03-30T04:28:25.062 INFO:tasks.workunit.client.0.smithi123.stdout:0/969: write d6/dd/d15/d28/f97 [3747814,26721] 0 2024-03-30T04:28:25.063 INFO:tasks.workunit.client.1.smithi135.stdout:1/500: rename d16/d5a/d5f/d3a/d6f/f7d to d16/d5a/d5f/d3a/d6f/fa8 0 2024-03-30T04:28:25.064 INFO:tasks.workunit.client.1.smithi135.stdout:3/537: truncate d3/d50/d22/d33/f61 3091750 0 2024-03-30T04:28:25.065 INFO:tasks.workunit.client.0.smithi123.stdout:7/892: creat d2/da/d11/d61/d55/d59/f12a x:0 0 0 2024-03-30T04:28:25.067 INFO:tasks.workunit.client.1.smithi135.stdout:9/455: link d7/d41/f59 d7/d90/fa0 0 2024-03-30T04:28:25.069 INFO:tasks.workunit.client.0.smithi123.stdout:4/883: mkdir d2/dfb/d124 0 2024-03-30T04:28:25.070 INFO:tasks.workunit.client.1.smithi135.stdout:3/538: stat d3/d50/d22/d33/l5a 0 2024-03-30T04:28:25.072 INFO:tasks.workunit.client.1.smithi135.stdout:5/487: write d1/da/f8c [3895490,63471] 0 2024-03-30T04:28:25.073 INFO:tasks.workunit.client.0.smithi123.stdout:0/970: symlink d6/dd/d15/d98/d5d/l137 0 2024-03-30T04:28:25.074 INFO:tasks.workunit.client.1.smithi135.stdout:1/501: creat d16/d29/fa9 x:0 0 0 2024-03-30T04:28:25.075 INFO:tasks.workunit.client.1.smithi135.stdout:9/456: write d7/d1b/f16 [2790970,111079] 0 2024-03-30T04:28:25.077 INFO:tasks.workunit.client.0.smithi123.stdout:0/971: unlink d6/dd/l6c 0 2024-03-30T04:28:25.080 INFO:tasks.workunit.client.1.smithi135.stdout:7/581: dwrite d4/db/d1a/d5c/f4e [0,4194304] 0 2024-03-30T04:28:25.080 INFO:tasks.workunit.client.1.smithi135.stdout:2/434: symlink d4/d32/d2e/l94 0 2024-03-30T04:28:25.082 INFO:tasks.workunit.client.0.smithi123.stdout:0/972: readlink d6/dd/d15/d28/d4b/d8a/l111 0 2024-03-30T04:28:25.083 INFO:tasks.workunit.client.0.smithi123.stdout:5/905: write dd/d19/d1f/d3c/d72/d8c/f98 [268482,79961] 0 2024-03-30T04:28:25.086 INFO:tasks.workunit.client.1.smithi135.stdout:3/539: symlink d3/d50/d22/d33/d7b/lbc 0 2024-03-30T04:28:25.086 INFO:tasks.workunit.client.0.smithi123.stdout:8/948: dwrite da/d27/d3c/d2d/d5c/fc7 [4194304,4194304] 0 2024-03-30T04:28:25.087 INFO:tasks.workunit.client.1.smithi135.stdout:9/457: creat d7/d41/d72/fa1 x:0 0 0 2024-03-30T04:28:25.091 INFO:tasks.workunit.client.1.smithi135.stdout:9/458: write d7/f12 [5991506,93321] 0 2024-03-30T04:28:25.092 INFO:tasks.workunit.client.0.smithi123.stdout:9/911: dwrite d6/d1f/d72/d7d/f12c [4194304,4194304] 0 2024-03-30T04:28:25.094 INFO:tasks.workunit.client.1.smithi135.stdout:7/582: rmdir d4/db/de 39 2024-03-30T04:28:25.094 INFO:tasks.workunit.client.1.smithi135.stdout:1/502: creat d16/d5a/d5f/d3d/faa x:0 0 0 2024-03-30T04:28:25.095 INFO:tasks.workunit.client.0.smithi123.stdout:0/973: unlink d6/d52/l125 0 2024-03-30T04:28:25.097 INFO:tasks.workunit.client.0.smithi123.stdout:5/906: mknod dd/d19/d1f/d35/d3a/d79/c12e 0 2024-03-30T04:28:25.097 INFO:tasks.workunit.client.1.smithi135.stdout:9/459: chown d7/d4f/d5c/d63/d86/f97 1338089 1 2024-03-30T04:28:25.103 INFO:tasks.workunit.client.0.smithi123.stdout:5/907: fdatasync dd/d19/d1f/d3c/d72/d8c/df4/f109 0 2024-03-30T04:28:25.103 INFO:tasks.workunit.client.1.smithi135.stdout:0/627: dwrite d0/d13/d1e/f46 [0,4194304] 0 2024-03-30T04:28:25.105 INFO:tasks.workunit.client.1.smithi135.stdout:3/540: symlink d3/d50/d46/d4c/lbd 0 2024-03-30T04:28:25.106 INFO:tasks.workunit.client.0.smithi123.stdout:9/912: read d6/d1f/f37 [2732140,93289] 0 2024-03-30T04:28:25.107 INFO:tasks.workunit.client.1.smithi135.stdout:8/479: sync 2024-03-30T04:28:25.111 INFO:tasks.workunit.client.0.smithi123.stdout:9/913: creat d6/d1f/d10c/f140 x:0 0 0 2024-03-30T04:28:25.113 INFO:tasks.workunit.client.0.smithi123.stdout:5/908: mkdir dd/d19/d1f/d68/dcc/d12f 0 2024-03-30T04:28:25.113 INFO:tasks.workunit.client.1.smithi135.stdout:3/541: truncate d3/d54/d5c/da1/fa8 826528 0 2024-03-30T04:28:25.113 INFO:tasks.workunit.client.1.smithi135.stdout:6/577: truncate d2/dd/d14/d30/f67 5652486 0 2024-03-30T04:28:25.114 INFO:tasks.workunit.client.0.smithi123.stdout:5/909: chown dd/l1b 12 1 2024-03-30T04:28:25.116 INFO:tasks.workunit.client.1.smithi135.stdout:7/583: mkdir d4/db/d1a/d90/dc7 0 2024-03-30T04:28:25.116 INFO:tasks.workunit.client.0.smithi123.stdout:9/914: mknod d6/d1f/d72/d7d/dfa/c141 0 2024-03-30T04:28:25.116 INFO:tasks.workunit.client.1.smithi135.stdout:9/460: mknod d7/d29/ca2 0 2024-03-30T04:28:25.118 INFO:tasks.workunit.client.0.smithi123.stdout:0/974: dread d6/dd/d31/d35/fed [0,4194304] 0 2024-03-30T04:28:25.119 INFO:tasks.workunit.client.0.smithi123.stdout:9/915: chown d6/d7c/d82/f112 239 1 2024-03-30T04:28:25.122 INFO:tasks.workunit.client.0.smithi123.stdout:9/916: chown d6/d1f/d72/db6/l104 46278 1 2024-03-30T04:28:25.123 INFO:tasks.workunit.client.0.smithi123.stdout:7/893: write d2/da/d11/d61/d55/d59/db5/f2a [650811,65789] 0 2024-03-30T04:28:25.124 INFO:tasks.workunit.client.1.smithi135.stdout:6/578: write d2/dd/d14/d38/f9f [114236,96994] 0 2024-03-30T04:28:25.131 INFO:tasks.workunit.client.1.smithi135.stdout:4/517: dread d10/f14 [0,4194304] 0 2024-03-30T04:28:25.131 INFO:tasks.workunit.client.1.smithi135.stdout:9/461: dread - d7/f5a zero size 2024-03-30T04:28:25.132 INFO:tasks.workunit.client.0.smithi123.stdout:0/975: dread d6/dd/d15/d28/dd3/ddd/fec [0,4194304] 0 2024-03-30T04:28:25.134 INFO:tasks.workunit.client.0.smithi123.stdout:9/917: getdents d6/d49/d113 0 2024-03-30T04:28:25.134 INFO:tasks.workunit.client.1.smithi135.stdout:4/518: write d10/d4d/d89/d9f/fa9 [219997,48887] 0 2024-03-30T04:28:25.137 INFO:tasks.workunit.client.0.smithi123.stdout:9/918: fsync d6/fa9 0 2024-03-30T04:28:25.138 INFO:tasks.workunit.client.0.smithi123.stdout:0/976: mkdir d6/dd/d15/d98/d5d/d7f/df3/d138 0 2024-03-30T04:28:25.140 INFO:tasks.workunit.client.1.smithi135.stdout:1/503: stat d16/d29/d2c/d43/c89 0 2024-03-30T04:28:25.141 INFO:tasks.workunit.client.0.smithi123.stdout:9/919: write d6/d1f/d10c/f111 [8782049,66346] 0 2024-03-30T04:28:25.142 INFO:tasks.workunit.client.0.smithi123.stdout:5/910: dwrite dd/d19/d1f/d3c/d72/f12a [0,4194304] 0 2024-03-30T04:28:25.145 INFO:tasks.workunit.client.1.smithi135.stdout:6/579: mkdir d2/dd/d40/db4 0 2024-03-30T04:28:25.148 INFO:tasks.workunit.client.1.smithi135.stdout:3/542: dwrite d3/d10/d16/fa5 [0,4194304] 0 2024-03-30T04:28:25.148 INFO:tasks.workunit.client.0.smithi123.stdout:8/949: truncate da/d27/d3c/d2d/dae/dd9/fe2 3800445 0 2024-03-30T04:28:25.149 INFO:tasks.workunit.client.1.smithi135.stdout:5/488: write d1/d3/d2b/d54/d5e/d5c/f4e [584542,39016] 0 2024-03-30T04:28:25.150 INFO:tasks.workunit.client.0.smithi123.stdout:9/920: creat d6/d1f/d45/d4c/d121/f142 x:0 0 0 2024-03-30T04:28:25.151 INFO:tasks.workunit.client.0.smithi123.stdout:5/911: symlink dd/d19/d5f/l130 0 2024-03-30T04:28:25.153 INFO:tasks.workunit.client.0.smithi123.stdout:8/950: truncate da/d27/f8a 4484272 0 2024-03-30T04:28:25.153 INFO:tasks.workunit.client.1.smithi135.stdout:6/580: read d2/f4d [5005887,87249] 0 2024-03-30T04:28:25.154 INFO:tasks.workunit.client.0.smithi123.stdout:9/921: chown d6/d1f/d10c/f111 10 1 2024-03-30T04:28:25.154 INFO:tasks.workunit.client.1.smithi135.stdout:7/584: mkdir d4/db/d1a/d5c/d59/dc8 0 2024-03-30T04:28:25.155 INFO:tasks.workunit.client.1.smithi135.stdout:4/519: truncate d10/d1a/f22 7358992 0 2024-03-30T04:28:25.157 INFO:tasks.workunit.client.1.smithi135.stdout:7/585: chown d4/db/d79/da9 85935735 1 2024-03-30T04:28:25.158 INFO:tasks.workunit.client.0.smithi123.stdout:8/951: creat da/d27/d3c/d64/d78/f12e x:0 0 0 2024-03-30T04:28:25.160 INFO:tasks.workunit.client.1.smithi135.stdout:8/480: creat d6/d8/d14/d32/f90 x:0 0 0 2024-03-30T04:28:25.160 INFO:tasks.workunit.client.1.smithi135.stdout:1/504: symlink d16/d19/da4/d99/lab 0 2024-03-30T04:28:25.160 INFO:tasks.workunit.client.0.smithi123.stdout:4/884: dread d2/d4/d2c/fcd [0,4194304] 0 2024-03-30T04:28:25.165 INFO:tasks.workunit.client.1.smithi135.stdout:0/628: getdents d0/d13/d53/d7f/d83/d9c 0 2024-03-30T04:28:25.165 INFO:tasks.workunit.client.1.smithi135.stdout:2/435: dwrite d4/d6/f1a [0,4194304] 0 2024-03-30T04:28:25.166 INFO:tasks.workunit.client.1.smithi135.stdout:3/543: mkdir d3/d10/d9c/dbe 0 2024-03-30T04:28:25.166 INFO:tasks.workunit.client.0.smithi123.stdout:8/952: write da/d27/d56/d6c/d6d/f79 [27565,38501] 0 2024-03-30T04:28:25.167 INFO:tasks.workunit.client.0.smithi123.stdout:0/977: write d6/dd/d15/d28/dd3/f131 [298525,126246] 0 2024-03-30T04:28:25.174 INFO:tasks.workunit.client.0.smithi123.stdout:4/885: mknod d2/d4/d2c/d42/d35/d5d/d78/d104/c125 0 2024-03-30T04:28:25.178 INFO:tasks.workunit.client.0.smithi123.stdout:7/894: dwrite d2/dc/f17 [0,4194304] 0 2024-03-30T04:28:25.178 INFO:tasks.workunit.client.0.smithi123.stdout:8/953: creat da/d27/d56/d6c/dd4/f12f x:0 0 0 2024-03-30T04:28:25.180 INFO:tasks.workunit.client.1.smithi135.stdout:5/489: creat d1/d3/d80/f93 x:0 0 0 2024-03-30T04:28:25.185 INFO:tasks.workunit.client.1.smithi135.stdout:5/490: chown d1/d7/f59 729619 1 2024-03-30T04:28:25.186 INFO:tasks.workunit.client.1.smithi135.stdout:3/544: creat d3/d54/d5c/da1/fbf x:0 0 0 2024-03-30T04:28:25.188 INFO:tasks.workunit.client.0.smithi123.stdout:0/978: creat d6/dd/d15/d28/d4b/dad/df2/f139 x:0 0 0 2024-03-30T04:28:25.189 INFO:tasks.workunit.client.0.smithi123.stdout:8/954: chown da/d27/d56/d6c/c94 123 1 2024-03-30T04:28:25.190 INFO:tasks.workunit.client.1.smithi135.stdout:5/491: fdatasync d1/d45/d52/f7d 0 2024-03-30T04:28:25.190 INFO:tasks.workunit.client.1.smithi135.stdout:4/520: dread d10/d4d/d89/d9f/fa9 [0,4194304] 0 2024-03-30T04:28:25.192 INFO:tasks.workunit.client.0.smithi123.stdout:5/912: dwrite dd/d19/d3e/d5b/d62/faa [0,4194304] 0 2024-03-30T04:28:25.192 INFO:tasks.workunit.client.1.smithi135.stdout:6/581: dwrite d2/d36/d76/fa1 [0,4194304] 0 2024-03-30T04:28:25.195 INFO:tasks.workunit.client.0.smithi123.stdout:7/895: mknod d2/da/d11/d61/d55/d59/db5/c12b 0 2024-03-30T04:28:25.198 INFO:tasks.workunit.client.1.smithi135.stdout:8/481: dread - d6/d8/d15/d2a/d35/f45 zero size 2024-03-30T04:28:25.199 INFO:tasks.workunit.client.0.smithi123.stdout:9/922: dwrite d6/d49/d113/daa/dac/f126 [0,4194304] 0 2024-03-30T04:28:25.200 INFO:tasks.workunit.client.1.smithi135.stdout:2/436: fsync d4/d32/d2e/d2f/f38 0 2024-03-30T04:28:25.203 INFO:tasks.workunit.client.1.smithi135.stdout:6/582: chown d2/dd/d40/d52 20388297 1 2024-03-30T04:28:25.203 INFO:tasks.workunit.client.0.smithi123.stdout:5/913: dread - dd/d19/d1f/d35/db8/fc1 zero size 2024-03-30T04:28:25.203 INFO:tasks.workunit.client.0.smithi123.stdout:0/979: truncate d6/dd/d15/d98/dc6/ff9 433701 0 2024-03-30T04:28:25.204 INFO:tasks.workunit.client.0.smithi123.stdout:7/896: rmdir d2/da/d11/d61/d55/d59/db5/dab 39 2024-03-30T04:28:25.205 INFO:tasks.workunit.client.0.smithi123.stdout:9/923: mknod d6/d49/c143 0 2024-03-30T04:28:25.206 INFO:tasks.workunit.client.0.smithi123.stdout:4/886: dread d2/d4/d2c/d42/d35/d5d/dc0/fce [0,4194304] 0 2024-03-30T04:28:25.206 INFO:tasks.workunit.client.1.smithi135.stdout:3/545: write d3/fa6 [3857100,118232] 0 2024-03-30T04:28:25.207 INFO:tasks.workunit.client.0.smithi123.stdout:0/980: creat d6/dd/d15/d28/dd3/ddd/f13a x:0 0 0 2024-03-30T04:28:25.208 INFO:tasks.workunit.client.0.smithi123.stdout:5/914: mkdir dd/d19/d1f/d3c/d131 0 2024-03-30T04:28:25.208 INFO:tasks.workunit.client.1.smithi135.stdout:8/482: fdatasync d6/d8/d15/f2b 0 2024-03-30T04:28:25.210 INFO:tasks.workunit.client.1.smithi135.stdout:0/629: unlink d0/d5/d9/d23/d58/d3b/c64 0 2024-03-30T04:28:25.211 INFO:tasks.workunit.client.1.smithi135.stdout:6/583: readlink d2/dd/d14/d38/l9b 0 2024-03-30T04:28:25.213 INFO:tasks.workunit.client.0.smithi123.stdout:4/887: mknod d2/d4/d7/c126 0 2024-03-30T04:28:25.218 INFO:tasks.workunit.client.1.smithi135.stdout:4/521: rmdir d10/d4d/d89/d9f/daa 39 2024-03-30T04:28:25.219 INFO:tasks.workunit.client.1.smithi135.stdout:1/505: dwrite d16/d29/d2c/d3c/f78 [0,4194304] 0 2024-03-30T04:28:25.220 INFO:tasks.workunit.client.0.smithi123.stdout:7/897: unlink d2/da/d11/d61/d55/d59/db5/dab/dca/f128 0 2024-03-30T04:28:25.221 INFO:tasks.workunit.client.1.smithi135.stdout:3/546: chown d3/d10/l32 116279 1 2024-03-30T04:28:25.223 INFO:tasks.workunit.client.0.smithi123.stdout:7/898: dread - d2/d4d/dd0/f108 zero size 2024-03-30T04:28:25.224 INFO:tasks.workunit.client.0.smithi123.stdout:9/924: dread d6/d1f/d45/d4c/d121/f33 [4194304,4194304] 0 2024-03-30T04:28:25.224 INFO:tasks.workunit.client.0.smithi123.stdout:4/888: rename d2/cc9 to d2/d4/d2c/d42/dad/d91/c127 0 2024-03-30T04:28:25.228 INFO:tasks.workunit.client.0.smithi123.stdout:7/899: creat d2/da/d11/d34/ded/f12c x:0 0 0 2024-03-30T04:28:25.228 INFO:tasks.workunit.client.0.smithi123.stdout:9/925: write d6/d49/d113/f123 [1944023,126561] 0 2024-03-30T04:28:25.231 INFO:tasks.workunit.client.0.smithi123.stdout:4/889: dread - d2/d4/d2c/d42/d35/d9b/da2/fe2 zero size 2024-03-30T04:28:25.231 INFO:tasks.workunit.client.1.smithi135.stdout:9/462: dread d7/d4f/f67 [0,4194304] 0 2024-03-30T04:28:25.232 INFO:tasks.workunit.client.0.smithi123.stdout:7/900: read - d2/da/d11/d34/d85/d8e/f129 zero size 2024-03-30T04:28:25.239 INFO:tasks.workunit.client.0.smithi123.stdout:4/890: creat d2/d4/d7/dc1/d108/f128 x:0 0 0 2024-03-30T04:28:25.240 INFO:tasks.workunit.client.1.smithi135.stdout:2/437: creat d4/da/d8d/f95 x:0 0 0 2024-03-30T04:28:25.240 INFO:tasks.workunit.client.0.smithi123.stdout:7/901: mknod d2/dc/d1c/c12d 0 2024-03-30T04:28:25.241 INFO:tasks.workunit.client.0.smithi123.stdout:9/926: link d6/d1f/d45/d4c/d12b/f133 d6/d1f/d45/f144 0 2024-03-30T04:28:25.241 INFO:tasks.workunit.client.0.smithi123.stdout:4/891: stat d2/d4/d2c/d42/cb9 0 2024-03-30T04:28:25.243 INFO:tasks.workunit.client.0.smithi123.stdout:7/902: readlink d2/da/d11/d34/d85/d8e/lf0 0 2024-03-30T04:28:25.246 INFO:tasks.workunit.client.1.smithi135.stdout:4/522: creat d10/d1a/d32/fb5 x:0 0 0 2024-03-30T04:28:25.247 INFO:tasks.workunit.client.0.smithi123.stdout:7/903: fsync d2/da/d11/d61/f72 0 2024-03-30T04:28:25.247 INFO:tasks.workunit.client.0.smithi123.stdout:9/927: fsync d6/d1f/d45/d4c/d121/f2b 0 2024-03-30T04:28:25.248 INFO:tasks.workunit.client.1.smithi135.stdout:9/463: mkdir d7/d4f/d5c/da3 0 2024-03-30T04:28:25.249 INFO:tasks.workunit.client.0.smithi123.stdout:7/904: chown d2/da/d11/f48 14460 1 2024-03-30T04:28:25.250 INFO:tasks.workunit.client.0.smithi123.stdout:9/928: write d6/d1f/f2e [2325618,4010] 0 2024-03-30T04:28:25.251 INFO:tasks.workunit.client.1.smithi135.stdout:6/584: creat d2/d68/da8/fb5 x:0 0 0 2024-03-30T04:28:25.251 INFO:tasks.workunit.client.1.smithi135.stdout:8/483: symlink d6/d8/d15/d54/d7e/l91 0 2024-03-30T04:28:25.252 INFO:tasks.workunit.client.1.smithi135.stdout:2/438: truncate d4/d6/f1f 2906521 0 2024-03-30T04:28:25.252 INFO:tasks.workunit.client.1.smithi135.stdout:9/464: dread - d7/d41/d72/f7d zero size 2024-03-30T04:28:25.252 INFO:tasks.workunit.client.1.smithi135.stdout:8/484: readlink d6/l87 0 2024-03-30T04:28:25.254 INFO:tasks.workunit.client.0.smithi123.stdout:7/905: symlink d2/da/d22/d3b/l12e 0 2024-03-30T04:28:25.255 INFO:tasks.workunit.client.0.smithi123.stdout:9/929: rmdir d6/d49/d113 39 2024-03-30T04:28:25.257 INFO:tasks.workunit.client.0.smithi123.stdout:5/915: dread dd/d19/d1f/d68/fc6 [4194304,4194304] 0 2024-03-30T04:28:25.259 INFO:tasks.workunit.client.0.smithi123.stdout:8/955: write da/dc/d14/d18/fac [587344,105495] 0 2024-03-30T04:28:25.260 INFO:tasks.workunit.client.1.smithi135.stdout:5/492: write d1/d7/d11/f87 [1589958,119363] 0 2024-03-30T04:28:25.262 INFO:tasks.workunit.client.0.smithi123.stdout:9/930: dread - d6/d1f/ff6 zero size 2024-03-30T04:28:25.268 INFO:tasks.workunit.client.1.smithi135.stdout:8/485: creat d6/d8/d15/d54/d7e/f92 x:0 0 0 2024-03-30T04:28:25.270 INFO:tasks.workunit.client.1.smithi135.stdout:7/586: dread d4/db/d1a/d4b/f45 [0,4194304] 0 2024-03-30T04:28:25.270 INFO:tasks.workunit.client.1.smithi135.stdout:2/439: rename d4/d32/d12/d76 to d4/d32/d12/d96 0 2024-03-30T04:28:25.272 INFO:tasks.workunit.client.0.smithi123.stdout:9/931: symlink d6/dc9/l145 0 2024-03-30T04:28:25.273 INFO:tasks.workunit.client.1.smithi135.stdout:8/486: readlink d6/d8/d15/d54/d4f/l7c 0 2024-03-30T04:28:25.276 INFO:tasks.workunit.client.1.smithi135.stdout:0/630: dread d0/d19/f5c [0,4194304] 0 2024-03-30T04:28:25.276 INFO:tasks.workunit.client.0.smithi123.stdout:8/956: link da/d27/d56/d6c/dd4/f12f da/d27/d56/dde/f130 0 2024-03-30T04:28:25.277 INFO:tasks.workunit.client.0.smithi123.stdout:0/981: dread d6/dd/d18/d2f/f85 [0,4194304] 0 2024-03-30T04:28:25.277 INFO:tasks.workunit.client.1.smithi135.stdout:9/465: rename d7/d1b to d7/d29/d78/da4 0 2024-03-30T04:28:25.278 INFO:tasks.workunit.client.0.smithi123.stdout:9/932: read d6/d1f/d10c/f13d [105405,11137] 0 2024-03-30T04:28:25.282 INFO:tasks.workunit.client.0.smithi123.stdout:0/982: creat d6/dd/d31/db8/f13b x:0 0 0 2024-03-30T04:28:25.283 INFO:tasks.workunit.client.0.smithi123.stdout:9/933: readlink d6/d1f/d72/d7d/dfa/ld6 0 2024-03-30T04:28:25.283 INFO:tasks.workunit.client.1.smithi135.stdout:9/466: chown d7/d29/d78/da4/l19 51 1 2024-03-30T04:28:25.284 INFO:tasks.workunit.client.0.smithi123.stdout:8/957: unlink da/d27/d3c/d64/d78/db8/lf2 0 2024-03-30T04:28:25.287 INFO:tasks.workunit.client.1.smithi135.stdout:6/585: dwrite d2/dd/d14/d2b/d86/d55/f74 [0,4194304] 0 2024-03-30T04:28:25.287 INFO:tasks.workunit.client.1.smithi135.stdout:4/523: getdents d10/d1a/d3d 0 2024-03-30T04:28:25.287 INFO:tasks.workunit.client.1.smithi135.stdout:2/440: mknod d4/c97 0 2024-03-30T04:28:25.288 INFO:tasks.workunit.client.0.smithi123.stdout:4/892: dread d2/f86 [0,4194304] 0 2024-03-30T04:28:25.291 INFO:tasks.workunit.client.0.smithi123.stdout:9/934: read d6/d1f/d4d/f5b [1871731,72951] 0 2024-03-30T04:28:25.295 INFO:tasks.workunit.client.0.smithi123.stdout:5/916: dwrite dd/f15 [4194304,4194304] 0 2024-03-30T04:28:25.295 INFO:tasks.workunit.client.1.smithi135.stdout:5/493: mkdir d1/d7/d1a/d94 0 2024-03-30T04:28:25.297 INFO:tasks.workunit.client.1.smithi135.stdout:0/631: creat d0/d5/d10/fca x:0 0 0 2024-03-30T04:28:25.298 INFO:tasks.workunit.client.0.smithi123.stdout:4/893: chown d2/d4/d2c/d42/d35/d9b/db7 983930 1 2024-03-30T04:28:25.298 INFO:tasks.workunit.client.0.smithi123.stdout:0/983: dread d6/fcd [0,4194304] 0 2024-03-30T04:28:25.299 INFO:tasks.workunit.client.1.smithi135.stdout:4/524: truncate d10/d1a/d32/fb5 40638 0 2024-03-30T04:28:25.301 INFO:tasks.workunit.client.0.smithi123.stdout:9/935: mkdir d6/d1f/d10c/d146 0 2024-03-30T04:28:25.304 INFO:tasks.workunit.client.0.smithi123.stdout:7/906: dwrite d2/da/d11/d34/d85/fae [0,4194304] 0 2024-03-30T04:28:25.306 INFO:tasks.workunit.client.0.smithi123.stdout:9/936: chown d6/d1f/d92/ce5 701901400 1 2024-03-30T04:28:25.308 INFO:tasks.workunit.client.0.smithi123.stdout:0/984: chown d6/dd/d15/d28/dd3/ddd/c10b 322 1 2024-03-30T04:28:25.309 INFO:tasks.workunit.client.1.smithi135.stdout:2/441: fsync d4/da/f21 0 2024-03-30T04:28:25.309 INFO:tasks.workunit.client.0.smithi123.stdout:5/917: rename dd/d19/d1f/d3c/d72/d8c/f10b to dd/d19/d1f/d68/dcc/df7/d122/f132 0 2024-03-30T04:28:25.309 INFO:tasks.workunit.client.0.smithi123.stdout:9/937: dread - d6/d1f/d4d/fd9 zero size 2024-03-30T04:28:25.311 INFO:tasks.workunit.client.1.smithi135.stdout:1/506: dread d16/d29/d2c/d3c/f3b [4194304,4194304] 0 2024-03-30T04:28:25.311 INFO:tasks.workunit.client.1.smithi135.stdout:6/586: unlink d2/dd/f1a 0 2024-03-30T04:28:25.312 INFO:tasks.workunit.client.0.smithi123.stdout:0/985: write d6/d52/fa8 [1473933,31262] 0 2024-03-30T04:28:25.313 INFO:tasks.workunit.client.0.smithi123.stdout:4/894: truncate d2/d4/d7/fe 3663197 0 2024-03-30T04:28:25.313 INFO:tasks.workunit.client.0.smithi123.stdout:7/907: symlink d2/d4d/d80/df3/l12f 0 2024-03-30T04:28:25.315 INFO:tasks.workunit.client.0.smithi123.stdout:5/918: rmdir dd/d19/d1f/d35/d3a/d11f 39 2024-03-30T04:28:25.316 INFO:tasks.workunit.client.1.smithi135.stdout:9/467: getdents d7/d55 0 2024-03-30T04:28:25.316 INFO:tasks.workunit.client.1.smithi135.stdout:4/525: write d10/d72/fa6 [201872,19690] 0 2024-03-30T04:28:25.317 INFO:tasks.workunit.client.0.smithi123.stdout:9/938: unlink d6/d1f/d92/db8/f103 0 2024-03-30T04:28:25.318 INFO:tasks.workunit.client.0.smithi123.stdout:5/919: dread - dd/d19/d3e/d5b/d62/dd3/df3/f102 zero size 2024-03-30T04:28:25.318 INFO:tasks.workunit.client.1.smithi135.stdout:0/632: symlink d0/d13/d53/d54/lcb 0 2024-03-30T04:28:25.318 INFO:tasks.workunit.client.1.smithi135.stdout:7/587: dwrite d4/db/d1a/d5c/d2c/d51/d56/f9e [0,4194304] 0 2024-03-30T04:28:25.320 INFO:tasks.workunit.client.1.smithi135.stdout:8/487: dwrite d6/d8/d15/d54/d4f/f6c [0,4194304] 0 2024-03-30T04:28:25.324 INFO:tasks.workunit.client.0.smithi123.stdout:4/895: symlink d2/d4/d7/l129 0 2024-03-30T04:28:25.328 INFO:tasks.workunit.client.1.smithi135.stdout:4/526: fdatasync d10/d1a/f27 0 2024-03-30T04:28:25.328 INFO:tasks.workunit.client.1.smithi135.stdout:9/468: write d7/d41/f79 [115608,91985] 0 2024-03-30T04:28:25.329 INFO:tasks.workunit.client.1.smithi135.stdout:1/507: symlink d16/d5a/d5f/d3a/d6f/lac 0 2024-03-30T04:28:25.329 INFO:tasks.workunit.client.0.smithi123.stdout:5/920: creat dd/d19/d5f/d99/f133 x:0 0 0 2024-03-30T04:28:25.332 INFO:tasks.workunit.client.0.smithi123.stdout:9/939: mkdir d6/d1f/d72/db6/d137/d147 0 2024-03-30T04:28:25.334 INFO:tasks.workunit.client.1.smithi135.stdout:6/587: mknod d2/dd/d14/d53/cb6 0 2024-03-30T04:28:25.335 INFO:tasks.workunit.client.0.smithi123.stdout:7/908: getdents d2/dc/d83 0 2024-03-30T04:28:25.337 INFO:tasks.workunit.client.1.smithi135.stdout:1/508: write fb [251867,62884] 0 2024-03-30T04:28:25.338 INFO:tasks.workunit.client.1.smithi135.stdout:9/469: rmdir d7/d41/d72 39 2024-03-30T04:28:25.338 INFO:tasks.workunit.client.0.smithi123.stdout:4/896: getdents d2/d4/d7/d63/db5/d10f 0 2024-03-30T04:28:25.352 INFO:tasks.workunit.client.1.smithi135.stdout:6/588: stat d2/dd/c16 0 2024-03-30T04:28:25.352 INFO:tasks.workunit.client.1.smithi135.stdout:9/470: unlink d7/d90/c96 0 2024-03-30T04:28:25.355 INFO:tasks.workunit.client.0.smithi123.stdout:8/958: dwrite da/dc/d14/d18/fcb [0,4194304] 0 2024-03-30T04:28:25.364 INFO:tasks.workunit.client.1.smithi135.stdout:0/633: dwrite d0/d5/d9/d23/d58/f1f [0,4194304] 0 2024-03-30T04:28:25.366 INFO:tasks.workunit.client.0.smithi123.stdout:0/986: write d6/dd/d15/d28/f78 [92858,113612] 0 2024-03-30T04:28:25.371 INFO:tasks.workunit.client.1.smithi135.stdout:6/589: dread d2/dd/d14/fab [0,4194304] 0 2024-03-30T04:28:25.372 INFO:tasks.workunit.client.1.smithi135.stdout:8/488: dwrite d6/d8/d15/d1f/f53 [0,4194304] 0 2024-03-30T04:28:25.373 INFO:tasks.workunit.client.1.smithi135.stdout:4/527: creat d10/d4d/d89/d9f/fb6 x:0 0 0 2024-03-30T04:28:25.374 INFO:tasks.workunit.client.1.smithi135.stdout:9/471: dread - d7/d4f/d5c/f6d zero size 2024-03-30T04:28:25.377 INFO:tasks.workunit.client.1.smithi135.stdout:7/588: getdents d4/db/d1a/d5c/d2c/d51/d56 0 2024-03-30T04:28:25.384 INFO:tasks.workunit.client.1.smithi135.stdout:7/589: chown d4/db/d1a/d5c/d82/f9d 489325 1 2024-03-30T04:28:25.393 INFO:tasks.workunit.client.1.smithi135.stdout:4/528: write d10/f73 [4560649,125005] 0 2024-03-30T04:28:25.394 INFO:tasks.workunit.client.1.smithi135.stdout:8/489: fsync d6/d8/d14/f30 0 2024-03-30T04:28:25.398 INFO:tasks.workunit.client.1.smithi135.stdout:6/590: rename d2/dd/d14/d38/f8b to d2/d68/d81/fb7 0 2024-03-30T04:28:25.400 INFO:tasks.workunit.client.1.smithi135.stdout:9/472: symlink d7/d41/d72/la5 0 2024-03-30T04:28:25.404 INFO:tasks.workunit.client.1.smithi135.stdout:0/634: dwrite d0/d13/d53/f80 [0,4194304] 0 2024-03-30T04:28:25.405 INFO:tasks.workunit.client.0.smithi123.stdout:5/921: unlink dd/d19/d1f/d3c/d72/d8c/f98 0 2024-03-30T04:28:25.406 INFO:tasks.workunit.client.1.smithi135.stdout:5/494: dread d1/d3/d2b/f2f [0,4194304] 0 2024-03-30T04:28:25.406 INFO:tasks.workunit.client.1.smithi135.stdout:9/473: readlink d7/d41/d72/la5 0 2024-03-30T04:28:25.406 INFO:tasks.workunit.client.1.smithi135.stdout:6/591: stat d2/dd/d40/d52/l54 0 2024-03-30T04:28:25.407 INFO:tasks.workunit.client.0.smithi123.stdout:4/897: write d2/d4/d2c/fa7 [4747278,7461] 0 2024-03-30T04:28:25.410 INFO:tasks.workunit.client.0.smithi123.stdout:7/909: symlink d2/da/d11/d61/d55/d59/db5/dab/d107/l130 0 2024-03-30T04:28:25.411 INFO:tasks.workunit.client.1.smithi135.stdout:1/509: dread fb [0,4194304] 0 2024-03-30T04:28:25.411 INFO:tasks.workunit.client.0.smithi123.stdout:5/922: readlink dd/d19/d1f/d68/dcc/lfe 0 2024-03-30T04:28:25.411 INFO:tasks.workunit.client.1.smithi135.stdout:8/490: mkdir d6/d8/d15/d2a/d35/d93 0 2024-03-30T04:28:25.411 INFO:tasks.workunit.client.1.smithi135.stdout:7/590: rename d4/db/d79/d92/faf to d4/fc9 0 2024-03-30T04:28:25.413 INFO:tasks.workunit.client.0.smithi123.stdout:7/910: readlink d2/d4d/dd0/le6 0 2024-03-30T04:28:25.416 INFO:tasks.workunit.client.1.smithi135.stdout:7/591: fsync d4/db/d1a/d4b/d40/d6e/fc6 0 2024-03-30T04:28:25.417 INFO:tasks.workunit.client.1.smithi135.stdout:9/474: creat d7/d4f/d89/fa6 x:0 0 0 2024-03-30T04:28:25.417 INFO:tasks.workunit.client.1.smithi135.stdout:7/592: stat d4/db/d79/d92 0 2024-03-30T04:28:25.418 INFO:tasks.workunit.client.1.smithi135.stdout:6/592: dread - d2/dd/d14/d2b/d86/d55/fb2 zero size 2024-03-30T04:28:25.420 INFO:tasks.workunit.client.0.smithi123.stdout:4/898: chown d2/d4/d2c/d42/d35/d5d/c123 6975933 1 2024-03-30T04:28:25.421 INFO:tasks.workunit.client.1.smithi135.stdout:5/495: symlink d1/d45/d52/l95 0 2024-03-30T04:28:25.423 INFO:tasks.workunit.client.1.smithi135.stdout:6/593: dread - d2/dd/d14/d30/d43/f78 zero size 2024-03-30T04:28:25.423 INFO:tasks.workunit.client.0.smithi123.stdout:5/923: symlink dd/d19/d1f/d3c/l134 0 2024-03-30T04:28:25.426 INFO:tasks.workunit.client.0.smithi123.stdout:9/940: dwrite d6/d1f/f29 [4194304,4194304] 0 2024-03-30T04:28:25.427 INFO:tasks.workunit.client.1.smithi135.stdout:1/510: rename d16/d5a/d6b to d16/d5a/dad 0 2024-03-30T04:28:25.428 INFO:tasks.workunit.client.0.smithi123.stdout:4/899: creat d2/d4/d2c/d42/d35/d5d/d6f/f12a x:0 0 0 2024-03-30T04:28:25.430 INFO:tasks.workunit.client.0.smithi123.stdout:5/924: creat dd/d19/d3e/d5b/d62/f135 x:0 0 0 2024-03-30T04:28:25.432 INFO:tasks.workunit.client.0.smithi123.stdout:9/941: fdatasync d6/d49/d113/daa/f13b 0 2024-03-30T04:28:25.432 INFO:tasks.workunit.client.1.smithi135.stdout:4/529: link d10/d4d/ca3 d10/d4d/d89/d9f/cb7 0 2024-03-30T04:28:25.433 INFO:tasks.workunit.client.0.smithi123.stdout:4/900: symlink d2/d26/l12b 0 2024-03-30T04:28:25.433 INFO:tasks.workunit.client.1.smithi135.stdout:9/475: mknod d7/d4f/d5c/da3/ca7 0 2024-03-30T04:28:25.435 INFO:tasks.workunit.client.0.smithi123.stdout:4/901: chown d2/d4/d7/d63/db5/lbf 2 1 2024-03-30T04:28:25.436 INFO:tasks.workunit.client.1.smithi135.stdout:1/511: creat d16/d5a/d5f/d3d/fae x:0 0 0 2024-03-30T04:28:25.436 INFO:tasks.workunit.client.1.smithi135.stdout:0/635: dwrite d0/f12 [0,4194304] 0 2024-03-30T04:28:25.444 INFO:tasks.workunit.client.0.smithi123.stdout:9/942: creat d6/d1f/d45/f148 x:0 0 0 2024-03-30T04:28:25.446 INFO:tasks.workunit.client.1.smithi135.stdout:6/594: dread d2/f4 [0,4194304] 0 2024-03-30T04:28:25.448 INFO:tasks.workunit.client.0.smithi123.stdout:8/959: write da/d27/d3c/d2d/d74/fbc [1042159,87563] 0 2024-03-30T04:28:25.450 INFO:tasks.workunit.client.1.smithi135.stdout:0/636: truncate d0/d5/d9/d23/d8a/fc8 218814 0 2024-03-30T04:28:25.450 INFO:tasks.workunit.client.1.smithi135.stdout:4/530: dread - d10/d1a/d42/d66/f7e zero size 2024-03-30T04:28:25.452 INFO:tasks.workunit.client.1.smithi135.stdout:1/512: creat d16/d19/da4/faf x:0 0 0 2024-03-30T04:28:25.453 INFO:tasks.workunit.client.0.smithi123.stdout:4/902: write d2/fcb [144203,118439] 0 2024-03-30T04:28:25.454 INFO:tasks.workunit.client.0.smithi123.stdout:8/960: chown da/d27/d56/d6c/d6d/f79 277 1 2024-03-30T04:28:25.455 INFO:tasks.workunit.client.1.smithi135.stdout:6/595: symlink d2/dd/d40/d52/d95/lb8 0 2024-03-30T04:28:25.456 INFO:tasks.workunit.client.1.smithi135.stdout:7/593: dwrite d4/db/de/db0/fc1 [0,4194304] 0 2024-03-30T04:28:25.457 INFO:tasks.workunit.client.1.smithi135.stdout:4/531: truncate d10/f17 5416417 0 2024-03-30T04:28:25.460 INFO:tasks.workunit.client.0.smithi123.stdout:4/903: stat d2/d4/d2c/d42/d35/d5d/fdf 0 2024-03-30T04:28:25.460 INFO:tasks.workunit.client.1.smithi135.stdout:4/532: rename d10 to d10/d1a/d32/d40/db8 22 2024-03-30T04:28:25.461 INFO:tasks.workunit.client.1.smithi135.stdout:0/637: unlink d0/d5/d42/fb9 0 2024-03-30T04:28:25.462 INFO:tasks.workunit.client.0.smithi123.stdout:0/987: dwrite d6/dd/d15/d28/d4b/f82 [0,4194304] 0 2024-03-30T04:28:25.463 INFO:tasks.workunit.client.1.smithi135.stdout:8/491: dwrite d6/d8/d14/f30 [0,4194304] 0 2024-03-30T04:28:25.464 INFO:tasks.workunit.client.0.smithi123.stdout:4/904: chown d2/d4/d2c/d42/dad/d91/lf2 391727 1 2024-03-30T04:28:25.465 INFO:tasks.workunit.client.0.smithi123.stdout:0/988: fdatasync d6/dd/d18/d1f/f121 0 2024-03-30T04:28:25.466 INFO:tasks.workunit.client.1.smithi135.stdout:7/594: chown d4/db/d1a/d5c/d2c/d2e/db9/l60 279022480 1 2024-03-30T04:28:25.467 INFO:tasks.workunit.client.1.smithi135.stdout:4/533: stat d10/d4d/d89/f8f 0 2024-03-30T04:28:25.472 INFO:tasks.workunit.client.1.smithi135.stdout:8/492: rename d6 to d6/d8/d94 22 2024-03-30T04:28:25.472 INFO:tasks.workunit.client.1.smithi135.stdout:5/496: write d1/d7/f2c [642031,32099] 0 2024-03-30T04:28:25.474 INFO:tasks.workunit.client.1.smithi135.stdout:9/476: write d7/d29/d78/da4/f35 [338117,104131] 0 2024-03-30T04:28:25.474 INFO:tasks.workunit.client.0.smithi123.stdout:8/961: link da/d27/d56/d93/l12c da/d27/d56/l131 0 2024-03-30T04:28:25.475 INFO:tasks.workunit.client.0.smithi123.stdout:0/989: symlink d6/dd/d15/d98/d4e/l13c 0 2024-03-30T04:28:25.479 INFO:tasks.workunit.client.1.smithi135.stdout:4/534: mkdir d10/d1a/d32/d40/d92/db9 0 2024-03-30T04:28:25.480 INFO:tasks.workunit.client.0.smithi123.stdout:4/905: rename d2/d4/d2c/d42/d35/d5d/d78/d104 to d2/d4/d2c/d42/d80/d96/d12c 0 2024-03-30T04:28:25.480 INFO:tasks.workunit.client.0.smithi123.stdout:8/962: dread - da/d27/d3c/f98 zero size 2024-03-30T04:28:25.480 INFO:tasks.workunit.client.0.smithi123.stdout:0/990: mknod d6/dd/d31/d35/d114/c13d 0 2024-03-30T04:28:25.482 INFO:tasks.workunit.client.1.smithi135.stdout:4/535: readlink d10/d1a/l29 0 2024-03-30T04:28:25.483 INFO:tasks.workunit.client.0.smithi123.stdout:8/963: fsync da/d27/fd8 0 2024-03-30T04:28:25.484 INFO:tasks.workunit.client.0.smithi123.stdout:0/991: write d6/dd/d15/d98/d4e/d69/f81 [5499667,125061] 0 2024-03-30T04:28:25.486 INFO:tasks.workunit.client.1.smithi135.stdout:5/497: unlink d1/d45/d52/f7d 0 2024-03-30T04:28:25.486 INFO:tasks.workunit.client.1.smithi135.stdout:9/477: dread d7/d29/d78/da4/f80 [0,4194304] 0 2024-03-30T04:28:25.487 INFO:tasks.workunit.client.1.smithi135.stdout:0/638: mkdir d0/d5/d9/d23/d58/d63/d77/d8b/dcc 0 2024-03-30T04:28:25.488 INFO:tasks.workunit.client.0.smithi123.stdout:4/906: truncate d2/d4/fe3 139811 0 2024-03-30T04:28:25.489 INFO:tasks.workunit.client.1.smithi135.stdout:0/639: stat d0/d5/d9/d23/d8a 0 2024-03-30T04:28:25.490 INFO:tasks.workunit.client.1.smithi135.stdout:7/595: symlink d4/db/de/db0/dc4/lca 0 2024-03-30T04:28:25.490 INFO:tasks.workunit.client.1.smithi135.stdout:1/513: write d16/d19/da4/d99/f21 [3181286,6683] 0 2024-03-30T04:28:25.491 INFO:tasks.workunit.client.0.smithi123.stdout:5/925: dread dd/d19/d1f/d68/dcc/df7/d122/f132 [0,4194304] 0 2024-03-30T04:28:25.491 INFO:tasks.workunit.client.1.smithi135.stdout:7/596: chown d4/db/d1a/d90/dc7 295 1 2024-03-30T04:28:25.493 INFO:tasks.workunit.client.0.smithi123.stdout:0/992: symlink d6/l13e 0 2024-03-30T04:28:25.493 INFO:tasks.workunit.client.0.smithi123.stdout:4/907: fsync d2/f20 0 2024-03-30T04:28:25.493 INFO:tasks.workunit.client.1.smithi135.stdout:0/640: chown d0/d5/d9/d23/f66 148432546 1 2024-03-30T04:28:25.494 INFO:tasks.workunit.client.1.smithi135.stdout:9/478: dread d7/d41/f59 [0,4194304] 0 2024-03-30T04:28:25.497 INFO:tasks.workunit.client.1.smithi135.stdout:5/498: symlink d1/d3/d2b/l96 0 2024-03-30T04:28:25.498 INFO:tasks.workunit.client.1.smithi135.stdout:1/514: write d16/d19/f1f [2103115,76920] 0 2024-03-30T04:28:25.499 INFO:tasks.workunit.client.1.smithi135.stdout:8/493: creat d6/d8/d15/d54/d8e/d72/f95 x:0 0 0 2024-03-30T04:28:25.500 INFO:tasks.workunit.client.0.smithi123.stdout:9/943: dread d6/d1f/d72/d10d/d11c/d12e/f12f [0,4194304] 0 2024-03-30T04:28:25.500 INFO:tasks.workunit.client.1.smithi135.stdout:7/597: stat d4/db/d1a/d5c/d2c/d51/da8/l7a 0 2024-03-30T04:28:25.500 INFO:tasks.workunit.client.0.smithi123.stdout:0/993: creat d6/dd/d15/d98/dc6/f13f x:0 0 0 2024-03-30T04:28:25.502 INFO:tasks.workunit.client.1.smithi135.stdout:4/536: creat d10/d1a/fba x:0 0 0 2024-03-30T04:28:25.505 INFO:tasks.workunit.client.1.smithi135.stdout:9/479: truncate d7/f5a 991216 0 2024-03-30T04:28:25.505 INFO:tasks.workunit.client.1.smithi135.stdout:8/494: write d6/d8/d15/d1f/f53 [359844,71640] 0 2024-03-30T04:28:25.506 INFO:tasks.workunit.client.1.smithi135.stdout:5/499: symlink d1/d3/d80/l97 0 2024-03-30T04:28:25.511 INFO:tasks.workunit.client.0.smithi123.stdout:0/994: fdatasync d6/dd/d15/d98/d4e/d69/d6e/fd4 0 2024-03-30T04:28:25.511 INFO:tasks.workunit.client.1.smithi135.stdout:4/537: mknod d10/d1a/d2b/cbb 0 2024-03-30T04:28:25.512 INFO:tasks.workunit.client.1.smithi135.stdout:7/598: mknod d4/db/d1a/ccb 0 2024-03-30T04:28:25.515 INFO:tasks.workunit.client.0.smithi123.stdout:0/995: creat d6/f140 x:0 0 0 2024-03-30T04:28:25.517 INFO:tasks.workunit.client.0.smithi123.stdout:8/964: dwrite da/dc/d14/d18/d62/f9f [4194304,4194304] 0 2024-03-30T04:28:25.519 INFO:tasks.workunit.client.1.smithi135.stdout:8/495: dread - d6/d8/d15/d54/d8e/d72/d73/f8d zero size 2024-03-30T04:28:25.522 INFO:tasks.workunit.client.1.smithi135.stdout:1/515: dwrite f3 [0,4194304] 0 2024-03-30T04:28:25.526 INFO:tasks.workunit.client.0.smithi123.stdout:9/944: dwrite d6/d49/d113/fab [0,4194304] 0 2024-03-30T04:28:25.528 INFO:tasks.workunit.client.0.smithi123.stdout:0/996: creat d6/dd/d18/d2f/f141 x:0 0 0 2024-03-30T04:28:25.530 INFO:tasks.workunit.client.0.smithi123.stdout:4/908: dwrite d2/f86 [0,4194304] 0 2024-03-30T04:28:25.533 INFO:tasks.workunit.client.0.smithi123.stdout:5/926: dread dd/d19/d1f/d3c/f92 [0,4194304] 0 2024-03-30T04:28:25.535 INFO:tasks.workunit.client.1.smithi135.stdout:7/599: dread d4/db/d1a/d4b/d40/d6e/fc6 [0,4194304] 0 2024-03-30T04:28:25.537 INFO:tasks.workunit.client.1.smithi135.stdout:1/516: symlink d16/d5a/d5f/d3a/d6f/lb0 0 2024-03-30T04:28:25.538 INFO:tasks.workunit.client.0.smithi123.stdout:0/997: rename d6/dd/d31/f86 to d6/dd/d15/d28/d4b/d8a/f142 0 2024-03-30T04:28:25.538 INFO:tasks.workunit.client.0.smithi123.stdout:9/945: link d6/d49/l6b d6/d1f/d92/d120/l149 0 2024-03-30T04:28:25.540 INFO:tasks.workunit.client.0.smithi123.stdout:0/998: write d6/d52/dcf/f10e [535906,40581] 0 2024-03-30T04:28:25.542 INFO:tasks.workunit.client.0.smithi123.stdout:5/927: mkdir dd/d19/d1f/d3c/d136 0 2024-03-30T04:28:25.547 INFO:tasks.workunit.client.0.smithi123.stdout:9/946: creat d6/d58/f14a x:0 0 0 2024-03-30T04:28:25.548 INFO:tasks.workunit.client.0.smithi123.stdout:4/909: getdents d2/d4/db3 0 2024-03-30T04:28:25.548 INFO:tasks.workunit.client.1.smithi135.stdout:1/517: chown d16/f57 211234 1 2024-03-30T04:28:25.550 INFO:tasks.workunit.client.0.smithi123.stdout:0/999: read d6/dd/d15/d98/d5d/d7f/df3/ffd [240931,28663] 0 2024-03-30T04:28:25.552 INFO:tasks.workunit.client.1.smithi135.stdout:4/538: dwrite d10/d4d/d89/d9f/fa7 [0,4194304] 0 2024-03-30T04:28:25.553 INFO:tasks.workunit.client.1.smithi135.stdout:5/500: dwrite d1/d3/d2b/d54/d5e/d5c/f4e [0,4194304] 0 2024-03-30T04:28:25.553 INFO:tasks.workunit.client.0.smithi123.stdout:9/947: fsync d6/d49/d113/f123 0 2024-03-30T04:28:25.554 INFO:tasks.workunit.client.0.smithi123.stdout:5/928: truncate dd/d19/d3e/d5b/d62/dd3/df3/f127 192474 0 2024-03-30T04:28:25.557 INFO:tasks.workunit.client.0.smithi123.stdout:4/910: readlink d2/d4/d2c/l59 0 2024-03-30T04:28:25.560 INFO:tasks.workunit.client.0.smithi123.stdout:9/948: mknod d6/d1f/d72/db6/c14b 0 2024-03-30T04:28:25.563 INFO:tasks.workunit.client.1.smithi135.stdout:0/641: dwrite d0/d1/d16/f62 [0,4194304] 0 2024-03-30T04:28:25.563 INFO:tasks.workunit.client.0.smithi123.stdout:4/911: readlink d2/l65 0 2024-03-30T04:28:25.564 INFO:tasks.workunit.client.1.smithi135.stdout:5/501: read d1/f39 [222744,123383] 0 2024-03-30T04:28:25.567 INFO:tasks.workunit.client.1.smithi135.stdout:5/502: chown d1/d7/d1a/d1f/f66 26395 1 2024-03-30T04:28:25.569 INFO:tasks.workunit.client.0.smithi123.stdout:4/912: write d2/d4/f101 [582373,73147] 0 2024-03-30T04:28:25.574 INFO:tasks.workunit.client.1.smithi135.stdout:4/539: link d10/c26 d10/d72/d85/cbc 0 2024-03-30T04:28:25.576 INFO:tasks.workunit.client.0.smithi123.stdout:9/949: rename d6/d1f/d45/d4c/cd1 to d6/d7c/d82/c14c 0 2024-03-30T04:28:25.576 INFO:tasks.workunit.client.1.smithi135.stdout:4/540: dread - d10/f59 zero size 2024-03-30T04:28:25.577 INFO:tasks.workunit.client.1.smithi135.stdout:9/480: truncate d7/d29/d78/da4/d37/f4a 3743690 0 2024-03-30T04:28:25.577 INFO:tasks.workunit.client.0.smithi123.stdout:4/913: write d2/d4/d2c/d42/d35/d9b/fb6 [920545,7754] 0 2024-03-30T04:28:25.580 INFO:tasks.workunit.client.1.smithi135.stdout:4/541: stat d10/d4d/d89/d9f/fa9 0 2024-03-30T04:28:25.580 INFO:tasks.workunit.client.1.smithi135.stdout:1/518: rename d16/d5a/d5f/d3d/fae to d16/d5a/d5f/d3a/fb1 0 2024-03-30T04:28:25.584 INFO:tasks.workunit.client.1.smithi135.stdout:7/600: write d4/db/de/d5a/f68 [532154,103770] 0 2024-03-30T04:28:25.588 INFO:tasks.workunit.client.0.smithi123.stdout:9/950: rename d6/d1f/d92/db8 to d6/d1f/d72/db6/d14d 0 2024-03-30T04:28:25.588 INFO:tasks.workunit.client.0.smithi123.stdout:4/914: truncate d2/d4/d2c/d42/d35/d5d/d6f/df9/f118 182571 0 2024-03-30T04:28:25.589 INFO:tasks.workunit.client.1.smithi135.stdout:5/503: getdents d1/d7/d1a/d1f 0 2024-03-30T04:28:25.590 INFO:tasks.workunit.client.1.smithi135.stdout:1/519: unlink d16/d29/d2c/d43/c89 0 2024-03-30T04:28:25.592 INFO:tasks.workunit.client.0.smithi123.stdout:9/951: write d6/d1f/d92/fe1 [846942,78223] 0 2024-03-30T04:28:25.593 INFO:tasks.workunit.client.1.smithi135.stdout:8/496: dwrite d6/d8/f42 [0,4194304] 0 2024-03-30T04:28:25.594 INFO:tasks.workunit.client.0.smithi123.stdout:8/965: dread da/dc/d14/d18/d62/f9f [8388608,4194304] 0 2024-03-30T04:28:25.595 INFO:tasks.workunit.client.0.smithi123.stdout:5/929: truncate dd/d19/d1f/d68/fc6 3016085 0 2024-03-30T04:28:25.595 INFO:tasks.workunit.client.1.smithi135.stdout:5/504: fdatasync d1/d3/d2b/f53 0 2024-03-30T04:28:25.598 INFO:tasks.workunit.client.0.smithi123.stdout:8/966: dread - da/dc/d14/db7/f12d zero size 2024-03-30T04:28:25.598 INFO:tasks.workunit.client.1.smithi135.stdout:4/542: creat d10/d4d/d89/d9f/fbd x:0 0 0 2024-03-30T04:28:25.601 INFO:tasks.workunit.client.0.smithi123.stdout:8/967: chown da/d27/d56/ce4 28092 1 2024-03-30T04:28:25.601 INFO:tasks.workunit.client.0.smithi123.stdout:9/952: dread d6/dc9/fce [0,4194304] 0 2024-03-30T04:28:25.602 INFO:tasks.workunit.client.1.smithi135.stdout:7/601: truncate d4/db/f14 110179 0 2024-03-30T04:28:25.603 INFO:tasks.workunit.client.1.smithi135.stdout:8/497: chown d6/d8/f16 14458439 1 2024-03-30T04:28:25.603 INFO:tasks.workunit.client.0.smithi123.stdout:9/953: fdatasync d6/fc5 0 2024-03-30T04:28:25.603 INFO:tasks.workunit.client.0.smithi123.stdout:5/930: dread dd/d19/d3e/d5b/d62/dd3/df3/f127 [0,4194304] 0 2024-03-30T04:28:25.605 INFO:tasks.workunit.client.0.smithi123.stdout:8/968: symlink da/d27/d56/d6c/dd4/l132 0 2024-03-30T04:28:25.606 INFO:tasks.workunit.client.1.smithi135.stdout:9/481: dread d7/f12 [4194304,4194304] 0 2024-03-30T04:28:25.606 INFO:tasks.workunit.client.1.smithi135.stdout:5/505: rename d1/d7/d1a/f3b to d1/d3/d76/f98 0 2024-03-30T04:28:25.612 INFO:tasks.workunit.client.0.smithi123.stdout:9/954: rmdir d6/d49/df9 39 2024-03-30T04:28:25.614 INFO:tasks.workunit.client.1.smithi135.stdout:1/520: dwrite f3 [0,4194304] 0 2024-03-30T04:28:25.616 INFO:tasks.workunit.client.1.smithi135.stdout:4/543: symlink d10/d72/d85/db0/lbe 0 2024-03-30T04:28:25.619 INFO:tasks.workunit.client.0.smithi123.stdout:9/955: mknod d6/d49/d113/c14e 0 2024-03-30T04:28:25.621 INFO:tasks.workunit.client.1.smithi135.stdout:9/482: symlink d7/d29/la8 0 2024-03-30T04:28:25.622 INFO:tasks.workunit.client.0.smithi123.stdout:4/915: dwrite d2/d4/d7/d63/f82 [0,4194304] 0 2024-03-30T04:28:25.623 INFO:tasks.workunit.client.0.smithi123.stdout:5/931: creat dd/d19/d1f/d35/d3a/d57/de0/f137 x:0 0 0 2024-03-30T04:28:25.624 INFO:tasks.workunit.client.0.smithi123.stdout:9/956: truncate d6/d58/f14a 8380 0 2024-03-30T04:28:25.627 INFO:tasks.workunit.client.0.smithi123.stdout:4/916: rmdir d2/d4/d2c/d42/d35/d5d 39 2024-03-30T04:28:25.629 INFO:tasks.workunit.client.1.smithi135.stdout:9/483: symlink d7/d4f/d5c/da3/la9 0 2024-03-30T04:28:25.630 INFO:tasks.workunit.client.0.smithi123.stdout:9/957: symlink d6/dc9/l14f 0 2024-03-30T04:28:25.635 INFO:tasks.workunit.client.0.smithi123.stdout:5/932: write dd/d19/d1f/d35/d3a/d57/db5/f112 [856329,130327] 0 2024-03-30T04:28:25.637 INFO:tasks.workunit.client.0.smithi123.stdout:8/969: dwrite da/d27/fb3 [0,4194304] 0 2024-03-30T04:28:25.639 INFO:tasks.workunit.client.1.smithi135.stdout:1/521: dwrite d16/d5a/f82 [0,4194304] 0 2024-03-30T04:28:25.644 INFO:tasks.workunit.client.1.smithi135.stdout:9/484: getdents d7/d4f/d5c/d63/d86 0 2024-03-30T04:28:25.649 INFO:tasks.workunit.client.0.smithi123.stdout:8/970: unlink da/dc/d14/d9c/c109 0 2024-03-30T04:28:25.649 INFO:tasks.workunit.client.0.smithi123.stdout:5/933: truncate dd/d19/d1f/d3c/d72/d8c/df4/f108 148406 0 2024-03-30T04:28:25.651 INFO:tasks.workunit.client.1.smithi135.stdout:1/522: mknod d16/d19/da4/d26/cb2 0 2024-03-30T04:28:25.654 INFO:tasks.workunit.client.1.smithi135.stdout:9/485: mknod d7/caa 0 2024-03-30T04:28:25.657 INFO:tasks.workunit.client.0.smithi123.stdout:4/917: dread d2/d4/f16 [0,4194304] 0 2024-03-30T04:28:25.658 INFO:tasks.workunit.client.1.smithi135.stdout:4/544: write d10/d1a/d3d/f41 [5058398,121817] 0 2024-03-30T04:28:25.659 INFO:tasks.workunit.client.1.smithi135.stdout:9/486: fsync d7/d41/f79 0 2024-03-30T04:28:25.659 INFO:tasks.workunit.client.1.smithi135.stdout:1/523: write d16/d5a/d5f/d3d/f96 [276768,31057] 0 2024-03-30T04:28:25.660 INFO:tasks.workunit.client.0.smithi123.stdout:5/934: symlink dd/d19/d1f/d68/dcc/df7/l138 0 2024-03-30T04:28:25.662 INFO:tasks.workunit.client.1.smithi135.stdout:7/602: dread d4/db/d79/da9/f58 [0,4194304] 0 2024-03-30T04:28:25.662 INFO:tasks.workunit.client.0.smithi123.stdout:5/935: fdatasync dd/d19/d1f/d35/db8/d11a/f10a 0 2024-03-30T04:28:25.663 INFO:tasks.workunit.client.0.smithi123.stdout:4/918: stat d2/d4/d7/d63/db5/fd6 0 2024-03-30T04:28:25.664 INFO:tasks.workunit.client.1.smithi135.stdout:8/498: dwrite d6/d8/f17 [0,4194304] 0 2024-03-30T04:28:25.666 INFO:tasks.workunit.client.1.smithi135.stdout:9/487: mknod d7/d4f/cab 0 2024-03-30T04:28:25.666 INFO:tasks.workunit.client.0.smithi123.stdout:4/919: fdatasync d2/d4/f22 0 2024-03-30T04:28:25.667 INFO:tasks.workunit.client.1.smithi135.stdout:4/545: chown d10/d72/fa6 406 1 2024-03-30T04:28:25.671 INFO:tasks.workunit.client.1.smithi135.stdout:7/603: write d4/db/f26 [400849,20424] 0 2024-03-30T04:28:25.672 INFO:tasks.workunit.client.1.smithi135.stdout:8/499: write d6/d23/f5c [1778113,77055] 0 2024-03-30T04:28:25.673 INFO:tasks.workunit.client.1.smithi135.stdout:4/546: chown d10/c24 1848731 1 2024-03-30T04:28:25.673 INFO:tasks.workunit.client.1.smithi135.stdout:9/488: mkdir d7/d4f/dac 0 2024-03-30T04:28:25.674 INFO:tasks.workunit.client.0.smithi123.stdout:9/958: dwrite d6/d1f/d10c/f13d [4194304,4194304] 0 2024-03-30T04:28:25.679 INFO:tasks.workunit.client.1.smithi135.stdout:4/547: unlink d10/d4d/f4e 0 2024-03-30T04:28:25.679 INFO:tasks.workunit.client.1.smithi135.stdout:5/506: dread d1/d7/d11/d3c/f44 [0,4194304] 0 2024-03-30T04:28:25.681 INFO:tasks.workunit.client.1.smithi135.stdout:9/489: unlink d7/d4f/f67 0 2024-03-30T04:28:25.682 INFO:tasks.workunit.client.1.smithi135.stdout:7/604: chown d4/db/d1a/d5c/d5d/c74 16 1 2024-03-30T04:28:25.682 INFO:tasks.workunit.client.0.smithi123.stdout:9/959: symlink d6/d1f/d72/d7d/d10f/d114/l150 0 2024-03-30T04:28:25.683 INFO:tasks.workunit.client.1.smithi135.stdout:8/500: write d6/fa [185557,101629] 0 2024-03-30T04:28:25.689 INFO:tasks.workunit.client.0.smithi123.stdout:4/920: dread d2/d34/f88 [0,4194304] 0 2024-03-30T04:28:25.689 INFO:tasks.workunit.client.0.smithi123.stdout:9/960: rename d6/d1f/d72/d10d to d6/d7c/de7/df1/d151 0 2024-03-30T04:28:25.689 INFO:tasks.workunit.client.1.smithi135.stdout:4/548: creat d10/d72/d85/fbf x:0 0 0 2024-03-30T04:28:25.692 INFO:tasks.workunit.client.0.smithi123.stdout:8/971: dread da/f55 [4194304,4194304] 0 2024-03-30T04:28:25.695 INFO:tasks.workunit.client.1.smithi135.stdout:4/549: fdatasync d10/d1a/d32/fae 0 2024-03-30T04:28:25.695 INFO:tasks.workunit.client.1.smithi135.stdout:1/524: dwrite d16/d19/f2b [0,4194304] 0 2024-03-30T04:28:25.696 INFO:tasks.workunit.client.1.smithi135.stdout:8/501: write d6/d8/d15/d2a/f4b [2501603,95112] 0 2024-03-30T04:28:25.697 INFO:tasks.workunit.client.0.smithi123.stdout:4/921: rename d2/d26/lb2 to d2/d4/de0/l12d 0 2024-03-30T04:28:25.697 INFO:tasks.workunit.client.0.smithi123.stdout:8/972: creat da/dc/d14/d9d/f133 x:0 0 0 2024-03-30T04:28:25.702 INFO:tasks.workunit.client.0.smithi123.stdout:8/973: mkdir da/d27/d3c/d2d/dae/dd9/d134 0 2024-03-30T04:28:25.705 INFO:tasks.workunit.client.1.smithi135.stdout:4/550: dread d10/d72/fa6 [0,4194304] 0 2024-03-30T04:28:25.707 INFO:tasks.workunit.client.1.smithi135.stdout:8/502: mknod d6/d8/d15/d54/c96 0 2024-03-30T04:28:25.710 INFO:tasks.workunit.client.1.smithi135.stdout:7/605: getdents d4/db/d1a/d5c/d2c/d51/d56/d81 0 2024-03-30T04:28:25.710 INFO:tasks.workunit.client.0.smithi123.stdout:8/974: getdents da/d27/d3c/d2d/dae 0 2024-03-30T04:28:25.711 INFO:tasks.workunit.client.0.smithi123.stdout:9/961: dread d6/d1f/d72/db6/fe4 [0,4194304] 0 2024-03-30T04:28:25.715 INFO:tasks.workunit.client.0.smithi123.stdout:8/975: unlink da/d27/l5d 0 2024-03-30T04:28:25.717 INFO:tasks.workunit.client.0.smithi123.stdout:4/922: dwrite d2/d4/d2c/fa7 [0,4194304] 0 2024-03-30T04:28:25.720 INFO:tasks.workunit.client.1.smithi135.stdout:8/503: write d6/d8/f31 [31675,81060] 0 2024-03-30T04:28:25.722 INFO:tasks.workunit.client.0.smithi123.stdout:8/976: chown da/d27/d56/d93/c11d 130259 1 2024-03-30T04:28:25.724 INFO:tasks.workunit.client.1.smithi135.stdout:1/525: dwrite d16/d19/f72 [0,4194304] 0 2024-03-30T04:28:25.733 INFO:tasks.workunit.client.1.smithi135.stdout:1/526: read d16/f42 [1164812,81918] 0 2024-03-30T04:28:25.733 INFO:tasks.workunit.client.1.smithi135.stdout:5/507: dread d1/d7/d1a/d1f/f66 [0,4194304] 0 2024-03-30T04:28:25.737 INFO:tasks.workunit.client.1.smithi135.stdout:4/551: rename d10/d72/d85/cbc to d10/d1a/cc0 0 2024-03-30T04:28:25.737 INFO:tasks.workunit.client.1.smithi135.stdout:3/547: sync 2024-03-30T04:28:25.737 INFO:tasks.workunit.client.1.smithi135.stdout:2/442: sync 2024-03-30T04:28:25.737 INFO:tasks.workunit.client.1.smithi135.stdout:5/508: stat d1/d3/d2b/l96 0 2024-03-30T04:28:25.738 INFO:tasks.workunit.client.1.smithi135.stdout:1/527: read - d16/d29/fa9 zero size 2024-03-30T04:28:25.743 INFO:tasks.workunit.client.1.smithi135.stdout:2/443: fsync d4/d32/d43/f92 0 2024-03-30T04:28:25.751 INFO:tasks.workunit.client.0.smithi123.stdout:4/923: write d2/d4/d7/d1a/f5e [1713648,66565] 0 2024-03-30T04:28:25.751 INFO:tasks.workunit.client.0.smithi123.stdout:4/924: stat d2/d4/d2c/d42/d80/d96/d12c/c125 0 2024-03-30T04:28:25.751 INFO:tasks.workunit.client.1.smithi135.stdout:9/490: dread d7/d29/d78/da4/f25 [0,4194304] 0 2024-03-30T04:28:25.751 INFO:tasks.workunit.client.1.smithi135.stdout:5/509: creat d1/da/d17/d70/f99 x:0 0 0 2024-03-30T04:28:25.751 INFO:tasks.workunit.client.1.smithi135.stdout:1/528: write d16/d5a/d5f/d3a/d6f/fa8 [3548609,39856] 0 2024-03-30T04:28:25.752 INFO:tasks.workunit.client.1.smithi135.stdout:3/548: write d3/d50/d46/f58 [847608,64254] 0 2024-03-30T04:28:25.752 INFO:tasks.workunit.client.0.smithi123.stdout:4/925: dread - d2/d4/db3/f122 zero size 2024-03-30T04:28:25.755 INFO:tasks.workunit.client.1.smithi135.stdout:3/549: readlink d3/d10/d9c/la0 0 2024-03-30T04:28:25.756 INFO:tasks.workunit.client.0.smithi123.stdout:4/926: creat d2/d4/d7/dc1/d113/f12e x:0 0 0 2024-03-30T04:28:25.757 INFO:tasks.workunit.client.1.smithi135.stdout:1/529: write d16/d5a/d5f/d3d/f96 [603098,91399] 0 2024-03-30T04:28:25.758 INFO:tasks.workunit.client.1.smithi135.stdout:3/550: readlink d3/l8 0 2024-03-30T04:28:25.758 INFO:tasks.workunit.client.0.smithi123.stdout:4/927: dread - d2/d26/fbd zero size 2024-03-30T04:28:25.760 INFO:tasks.workunit.client.1.smithi135.stdout:5/510: write d1/d3/d2b/d54/d5e/d5c/f65 [1713859,102442] 0 2024-03-30T04:28:25.765 INFO:tasks.workunit.client.1.smithi135.stdout:7/606: dread f3 [0,4194304] 0 2024-03-30T04:28:25.766 INFO:tasks.workunit.client.0.smithi123.stdout:9/962: dread d6/d7c/f89 [0,4194304] 0 2024-03-30T04:28:25.767 INFO:tasks.workunit.client.0.smithi123.stdout:9/963: dread - d6/fc5 zero size 2024-03-30T04:28:25.768 INFO:tasks.workunit.client.1.smithi135.stdout:1/530: symlink d16/lb3 0 2024-03-30T04:28:25.768 INFO:tasks.workunit.client.0.smithi123.stdout:4/928: write d2/d4/d2c/d42/d89/f94 [29693,81472] 0 2024-03-30T04:28:25.769 INFO:tasks.workunit.client.1.smithi135.stdout:9/491: truncate d7/f44 1795739 0 2024-03-30T04:28:25.769 INFO:tasks.workunit.client.0.smithi123.stdout:9/964: fdatasync d6/d1f/d72/f8b 0 2024-03-30T04:28:25.770 INFO:tasks.workunit.client.1.smithi135.stdout:4/552: dwrite d10/d1a/d2b/f31 [0,4194304] 0 2024-03-30T04:28:25.773 INFO:tasks.workunit.client.0.smithi123.stdout:4/929: chown d2/d4/d2c/d42/d35/d5d/dc0/fce 3 1 2024-03-30T04:28:25.774 INFO:tasks.workunit.client.0.smithi123.stdout:9/965: creat d6/d1f/d92/df4/f152 x:0 0 0 2024-03-30T04:28:25.776 INFO:tasks.workunit.client.1.smithi135.stdout:9/492: mkdir d7/d4f/d6e/dad 0 2024-03-30T04:28:25.779 INFO:tasks.workunit.client.1.smithi135.stdout:1/531: mkdir d16/d5a/d5f/d3d/db4 0 2024-03-30T04:28:25.783 INFO:tasks.workunit.client.0.smithi123.stdout:9/966: symlink d6/d49/l153 0 2024-03-30T04:28:25.783 INFO:tasks.workunit.client.1.smithi135.stdout:2/444: dwrite d4/d6/f16 [4194304,4194304] 0 2024-03-30T04:28:25.783 INFO:tasks.workunit.client.1.smithi135.stdout:4/553: rename d10/d72/d85 to d10/d4d/dc1 0 2024-03-30T04:28:25.788 INFO:tasks.workunit.client.0.smithi123.stdout:9/967: mkdir d6/d154 0 2024-03-30T04:28:25.789 INFO:tasks.workunit.client.1.smithi135.stdout:2/445: chown d4/d32/d12/d45/d3c 145152437 1 2024-03-30T04:28:25.789 INFO:tasks.workunit.client.1.smithi135.stdout:1/532: truncate d16/d19/da4/faf 812057 0 2024-03-30T04:28:25.791 INFO:tasks.workunit.client.0.smithi123.stdout:9/968: readlink d6/d1f/d45/d97/l9b 0 2024-03-30T04:28:25.792 INFO:tasks.workunit.client.0.smithi123.stdout:9/969: stat d6/ce0 0 2024-03-30T04:28:25.793 INFO:tasks.workunit.client.0.smithi123.stdout:4/930: dwrite d2/d4/d2c/d42/d35/d9b/db7/fc7 [0,4194304] 0 2024-03-30T04:28:25.795 INFO:tasks.workunit.client.1.smithi135.stdout:1/533: chown d16/c17 32714227 1 2024-03-30T04:28:25.795 INFO:tasks.workunit.client.0.smithi123.stdout:9/970: chown d6/d49/f75 0 1 2024-03-30T04:28:25.796 INFO:tasks.workunit.client.1.smithi135.stdout:7/607: dwrite d4/db/d9f/fba [0,4194304] 0 2024-03-30T04:28:25.798 INFO:tasks.workunit.client.1.smithi135.stdout:2/446: fdatasync d4/da/d7a/f7c 0 2024-03-30T04:28:25.798 INFO:tasks.workunit.client.0.smithi123.stdout:4/931: creat d2/d4/d7/f12f x:0 0 0 2024-03-30T04:28:25.799 INFO:tasks.workunit.client.1.smithi135.stdout:1/534: fsync d16/d19/f72 0 2024-03-30T04:28:25.801 INFO:tasks.workunit.client.1.smithi135.stdout:2/447: chown d4/d32/f75 3904 1 2024-03-30T04:28:25.802 INFO:tasks.workunit.client.0.smithi123.stdout:9/971: rename d6/d1f/d45/d97/l9e to d6/d1f/l155 0 2024-03-30T04:28:25.803 INFO:tasks.workunit.client.0.smithi123.stdout:4/932: truncate d2/d4/d7/d63/db5/fd6 174372 0 2024-03-30T04:28:25.807 INFO:tasks.workunit.client.0.smithi123.stdout:9/972: creat d6/d1f/d72/db6/d137/f156 x:0 0 0 2024-03-30T04:28:25.808 INFO:tasks.workunit.client.1.smithi135.stdout:9/493: dwrite d7/d55/f6c [0,4194304] 0 2024-03-30T04:28:25.815 INFO:tasks.workunit.client.1.smithi135.stdout:5/511: truncate d1/d3/f49 2434963 0 2024-03-30T04:28:25.816 INFO:tasks.workunit.client.1.smithi135.stdout:1/535: symlink d16/d29/d2c/d3c/lb5 0 2024-03-30T04:28:25.818 INFO:tasks.workunit.client.1.smithi135.stdout:2/448: creat d4/d6/d47/f98 x:0 0 0 2024-03-30T04:28:25.821 INFO:tasks.workunit.client.0.smithi123.stdout:7/911: sync 2024-03-30T04:28:25.822 INFO:tasks.workunit.client.1.smithi135.stdout:5/512: mknod d1/d3/c9a 0 2024-03-30T04:28:25.824 INFO:tasks.workunit.client.0.smithi123.stdout:7/912: write d2/da/d11/d61/d55/d59/db5/f51 [285090,105354] 0 2024-03-30T04:28:25.826 INFO:tasks.workunit.client.1.smithi135.stdout:5/513: truncate d1/d7/d11/f30 2034943 0 2024-03-30T04:28:25.826 INFO:tasks.workunit.client.1.smithi135.stdout:7/608: getdents d4/db/d1a/d5c/d59 0 2024-03-30T04:28:25.829 INFO:tasks.workunit.client.1.smithi135.stdout:2/449: truncate d4/d32/d43/f92 1887277 0 2024-03-30T04:28:25.831 INFO:tasks.workunit.client.1.smithi135.stdout:5/514: truncate d1/d7/f43 4858400 0 2024-03-30T04:28:25.834 INFO:tasks.workunit.client.1.smithi135.stdout:7/609: mkdir d4/db/de/db0/db4/dcc 0 2024-03-30T04:28:25.836 INFO:tasks.workunit.client.1.smithi135.stdout:2/450: mkdir d4/d6/d99 0 2024-03-30T04:28:25.844 INFO:tasks.workunit.client.1.smithi135.stdout:2/451: symlink d4/d6/d47/l9a 0 2024-03-30T04:28:25.845 INFO:tasks.workunit.client.1.smithi135.stdout:1/536: dwrite f3 [4194304,4194304] 0 2024-03-30T04:28:25.850 INFO:tasks.workunit.client.1.smithi135.stdout:9/494: dread d7/d90/f94 [0,4194304] 0 2024-03-30T04:28:25.851 INFO:tasks.workunit.client.0.smithi123.stdout:9/973: write d6/d1f/d45/d4c/f54 [1016510,86603] 0 2024-03-30T04:28:25.851 INFO:tasks.workunit.client.1.smithi135.stdout:1/537: chown d16/d29/d2c/l79 163186777 1 2024-03-30T04:28:25.852 INFO:tasks.workunit.client.1.smithi135.stdout:2/452: symlink d4/d6/d47/l9b 0 2024-03-30T04:28:25.854 INFO:tasks.workunit.client.0.smithi123.stdout:7/913: write d2/da/d11/f48 [4957160,89314] 0 2024-03-30T04:28:25.855 INFO:tasks.workunit.client.0.smithi123.stdout:9/974: unlink d6/d1f/d45/d4c/d121/c3e 0 2024-03-30T04:28:25.865 INFO:tasks.workunit.client.1.smithi135.stdout:7/610: dwrite d4/db/f26 [0,4194304] 0 2024-03-30T04:28:25.874 INFO:tasks.workunit.client.1.smithi135.stdout:7/611: creat d4/db/d1a/d5c/fcd x:0 0 0 2024-03-30T04:28:25.875 INFO:tasks.workunit.client.0.smithi123.stdout:7/914: write d2/dc/d49/dbc/fc3 [2955161,92656] 0 2024-03-30T04:28:25.875 INFO:tasks.workunit.client.1.smithi135.stdout:7/612: write d4/f16 [4137770,12207] 0 2024-03-30T04:28:25.876 INFO:tasks.workunit.client.1.smithi135.stdout:2/453: write d4/f1c [3718255,9904] 0 2024-03-30T04:28:25.879 INFO:tasks.workunit.client.1.smithi135.stdout:7/613: write d4/db/d1a/d4b/d40/d6e/fc6 [410532,84554] 0 2024-03-30T04:28:25.886 INFO:tasks.workunit.client.0.smithi123.stdout:7/915: truncate d2/da/d11/d34/d85/fac 490618 0 2024-03-30T04:28:25.886 INFO:tasks.workunit.client.1.smithi135.stdout:7/614: read - d4/db/d1a/d5c/d2c/d2e/db9/faa zero size 2024-03-30T04:28:25.887 INFO:tasks.workunit.client.1.smithi135.stdout:9/495: dwrite d7/d29/d78/da4/f1c [0,4194304] 0 2024-03-30T04:28:25.890 INFO:tasks.workunit.client.0.smithi123.stdout:9/975: dwrite d6/d1f/f2d [0,4194304] 0 2024-03-30T04:28:25.891 INFO:tasks.workunit.client.0.smithi123.stdout:7/916: mkdir d2/d4d/d80/d98/d131 0 2024-03-30T04:28:25.891 INFO:tasks.workunit.client.1.smithi135.stdout:7/615: fsync d4/db/d1a/d5c/d82/fa2 0 2024-03-30T04:28:25.895 INFO:tasks.workunit.client.1.smithi135.stdout:7/616: write d4/f16 [4330316,34693] 0 2024-03-30T04:28:25.900 INFO:tasks.workunit.client.1.smithi135.stdout:2/454: dwrite d4/d6/f1a [4194304,4194304] 0 2024-03-30T04:28:25.902 INFO:tasks.workunit.client.1.smithi135.stdout:7/617: readlink d4/db/d1a/d5c/l33 0 2024-03-30T04:28:25.905 INFO:tasks.workunit.client.1.smithi135.stdout:9/496: write d7/d29/d78/da4/d37/f38 [557422,127196] 0 2024-03-30T04:28:25.908 INFO:tasks.workunit.client.1.smithi135.stdout:7/618: fsync d4/db/d1a/d5c/d5d/f86 0 2024-03-30T04:28:25.911 INFO:tasks.workunit.client.1.smithi135.stdout:9/497: chown d7/d41/f43 41606062 1 2024-03-30T04:28:25.913 INFO:tasks.workunit.client.1.smithi135.stdout:7/619: chown d4/db/d1a/d5c/f4e 87 1 2024-03-30T04:28:25.915 INFO:tasks.workunit.client.1.smithi135.stdout:9/498: creat d7/d29/fae x:0 0 0 2024-03-30T04:28:25.915 INFO:tasks.workunit.client.0.smithi123.stdout:9/976: symlink d6/d49/d113/daa/dac/l157 0 2024-03-30T04:28:25.919 INFO:tasks.workunit.client.1.smithi135.stdout:7/620: rename f3 to d4/db/de/d5a/dae/fce 0 2024-03-30T04:28:25.920 INFO:tasks.workunit.client.1.smithi135.stdout:1/538: dread d16/f25 [0,4194304] 0 2024-03-30T04:28:25.921 INFO:tasks.workunit.client.1.smithi135.stdout:9/499: creat d7/d55/d8d/faf x:0 0 0 2024-03-30T04:28:25.922 INFO:tasks.workunit.client.1.smithi135.stdout:9/500: chown d7/d4f/cab 1035265 1 2024-03-30T04:28:25.922 INFO:tasks.workunit.client.1.smithi135.stdout:7/621: mkdir d4/db/d79/d92/dcf 0 2024-03-30T04:28:25.923 INFO:tasks.workunit.client.1.smithi135.stdout:2/455: truncate d4/d32/d2e/f5c 1783157 0 2024-03-30T04:28:25.924 INFO:tasks.workunit.client.1.smithi135.stdout:9/501: mknod d7/d29/cb0 0 2024-03-30T04:28:25.926 INFO:tasks.workunit.client.1.smithi135.stdout:9/502: chown d7/d41/f79 39239816 1 2024-03-30T04:28:25.926 INFO:tasks.workunit.client.1.smithi135.stdout:7/622: creat d4/db/d1a/d5c/d2c/d51/fd0 x:0 0 0 2024-03-30T04:28:25.926 INFO:tasks.workunit.client.0.smithi123.stdout:7/917: dwrite d2/da/d11/d34/f70 [0,4194304] 0 2024-03-30T04:28:25.928 INFO:tasks.workunit.client.1.smithi135.stdout:7/623: chown d4/db/d9f/fba 3226186 1 2024-03-30T04:28:25.929 INFO:tasks.workunit.client.1.smithi135.stdout:9/503: write d7/d4f/d5c/d63/d86/f9a [774046,61427] 0 2024-03-30T04:28:25.931 INFO:tasks.workunit.client.0.smithi123.stdout:7/918: mknod d2/dc/c132 0 2024-03-30T04:28:25.933 INFO:tasks.workunit.client.1.smithi135.stdout:9/504: read - d7/d4f/d5c/f5e zero size 2024-03-30T04:28:25.933 INFO:tasks.workunit.client.0.smithi123.stdout:7/919: chown d2/d4d/d80/d98/d131 31000059 1 2024-03-30T04:28:25.934 INFO:tasks.workunit.client.1.smithi135.stdout:6/596: sync 2024-03-30T04:28:25.934 INFO:tasks.workunit.client.1.smithi135.stdout:0/642: sync 2024-03-30T04:28:25.934 INFO:tasks.workunit.client.0.smithi123.stdout:9/977: dwrite d6/d1f/d92/ffe [4194304,4194304] 0 2024-03-30T04:28:25.937 INFO:tasks.workunit.client.1.smithi135.stdout:9/505: dread d7/d4f/d5c/d63/f64 [0,4194304] 0 2024-03-30T04:28:25.937 INFO:tasks.workunit.client.1.smithi135.stdout:7/624: creat d4/db/de/fd1 x:0 0 0 2024-03-30T04:28:25.938 INFO:tasks.workunit.client.1.smithi135.stdout:6/597: creat d2/d68/fb9 x:0 0 0 2024-03-30T04:28:25.940 INFO:tasks.workunit.client.0.smithi123.stdout:7/920: creat d2/da/d11/d61/d55/d94/f133 x:0 0 0 2024-03-30T04:28:25.940 INFO:tasks.workunit.client.1.smithi135.stdout:6/598: readlink d2/dd/d14/d30/d43/l58 0 2024-03-30T04:28:25.943 INFO:tasks.workunit.client.1.smithi135.stdout:2/456: truncate d4/d55/d77/f58 2118439 0 2024-03-30T04:28:25.943 INFO:tasks.workunit.client.1.smithi135.stdout:9/506: chown d7/f44 1119838 1 2024-03-30T04:28:25.944 INFO:tasks.workunit.client.1.smithi135.stdout:6/599: creat d2/d68/fba x:0 0 0 2024-03-30T04:28:25.944 INFO:tasks.workunit.client.0.smithi123.stdout:9/978: getdents d6/d7c/d82 0 2024-03-30T04:28:25.948 INFO:tasks.workunit.client.1.smithi135.stdout:2/457: getdents d4 0 2024-03-30T04:28:25.949 INFO:tasks.workunit.client.1.smithi135.stdout:9/507: mkdir d7/d4f/d6e/d98/db1 0 2024-03-30T04:28:25.955 INFO:tasks.workunit.client.1.smithi135.stdout:6/600: dread d2/dd/d14/d30/d43/f69 [0,4194304] 0 2024-03-30T04:28:25.956 INFO:tasks.workunit.client.1.smithi135.stdout:6/601: fdatasync d2/dd/d14/f72 0 2024-03-30T04:28:25.956 INFO:tasks.workunit.client.1.smithi135.stdout:9/508: creat d7/d29/d78/fb2 x:0 0 0 2024-03-30T04:28:25.956 INFO:tasks.workunit.client.0.smithi123.stdout:7/921: creat d2/dc/d7f/d10c/f134 x:0 0 0 2024-03-30T04:28:25.956 INFO:tasks.workunit.client.0.smithi123.stdout:9/979: fsync d6/d1f/f34 0 2024-03-30T04:28:25.956 INFO:tasks.workunit.client.0.smithi123.stdout:5/936: sync 2024-03-30T04:28:25.956 INFO:tasks.workunit.client.0.smithi123.stdout:9/980: dread - d6/d58/f96 zero size 2024-03-30T04:28:25.956 INFO:tasks.workunit.client.0.smithi123.stdout:7/922: mknod d2/da/d11/d61/d55/d59/d11f/c135 0 2024-03-30T04:28:25.956 INFO:tasks.workunit.client.1.smithi135.stdout:2/458: creat d4/d6/f9c x:0 0 0 2024-03-30T04:28:25.957 INFO:tasks.workunit.client.0.smithi123.stdout:7/923: dread - d2/fd9 zero size 2024-03-30T04:28:25.957 INFO:tasks.workunit.client.1.smithi135.stdout:6/602: truncate d2/dd/d14/f72 443583 0 2024-03-30T04:28:25.959 INFO:tasks.workunit.client.0.smithi123.stdout:9/981: creat d6/d49/f158 x:0 0 0 2024-03-30T04:28:25.959 INFO:tasks.workunit.client.1.smithi135.stdout:2/459: truncate d4/d32/d46/f91 998664 0 2024-03-30T04:28:25.960 INFO:tasks.workunit.client.0.smithi123.stdout:7/924: chown d2/da/d11/d34/d76/cc4 1959 1 2024-03-30T04:28:25.961 INFO:tasks.workunit.client.0.smithi123.stdout:5/937: write dd/d19/d3e/d5b/d62/dd3/df3/f127 [438901,30041] 0 2024-03-30T04:28:25.962 INFO:tasks.workunit.client.1.smithi135.stdout:2/460: write d4/d6/d47/f98 [377858,30191] 0 2024-03-30T04:28:25.967 INFO:tasks.workunit.client.0.smithi123.stdout:7/925: write d2/da/d11/fc6 [55695,102108] 0 2024-03-30T04:28:25.968 INFO:tasks.workunit.client.1.smithi135.stdout:2/461: symlink d4/d32/l9d 0 2024-03-30T04:28:25.969 INFO:tasks.workunit.client.0.smithi123.stdout:9/982: dread d6/d1f/f4f [4194304,4194304] 0 2024-03-30T04:28:25.970 INFO:tasks.workunit.client.0.smithi123.stdout:7/926: mkdir d2/da/d11/d34/db0/d136 0 2024-03-30T04:28:25.976 INFO:tasks.workunit.client.1.smithi135.stdout:2/462: creat d4/d32/f9e x:0 0 0 2024-03-30T04:28:25.979 INFO:tasks.workunit.client.1.smithi135.stdout:2/463: stat d4/d55/d77/l8f 0 2024-03-30T04:28:25.983 INFO:tasks.workunit.client.0.smithi123.stdout:9/983: dwrite d6/f15 [8388608,4194304] 0 2024-03-30T04:28:25.985 INFO:tasks.workunit.client.1.smithi135.stdout:2/464: truncate d4/d32/f26 398039 0 2024-03-30T04:28:25.985 INFO:tasks.workunit.client.1.smithi135.stdout:6/603: dwrite d2/dd/d14/d2b/f46 [0,4194304] 0 2024-03-30T04:28:25.986 INFO:tasks.workunit.client.0.smithi123.stdout:9/984: dread - d6/d49/d113/daa/f13b zero size 2024-03-30T04:28:25.990 INFO:tasks.workunit.client.1.smithi135.stdout:9/509: dread d7/d29/d78/da4/f28 [0,4194304] 0 2024-03-30T04:28:25.990 INFO:tasks.workunit.client.0.smithi123.stdout:9/985: creat d6/d58/f159 x:0 0 0 2024-03-30T04:28:25.996 INFO:tasks.workunit.client.0.smithi123.stdout:7/927: dwrite d2/dc/d49/f126 [0,4194304] 0 2024-03-30T04:28:25.998 INFO:tasks.workunit.client.0.smithi123.stdout:9/986: mkdir d6/d7c/de7/df1/d151/d11c/d12e/d15a 0 2024-03-30T04:28:25.999 INFO:tasks.workunit.client.1.smithi135.stdout:6/604: rename d2/dd/c51 to d2/dd/d14/cbb 0 2024-03-30T04:28:26.002 INFO:tasks.workunit.client.0.smithi123.stdout:7/928: rmdir d2/da/d11/d34/d85/d8e 39 2024-03-30T04:28:26.003 INFO:tasks.workunit.client.1.smithi135.stdout:6/605: creat d2/dd/d14/d30/fbc x:0 0 0 2024-03-30T04:28:26.004 INFO:tasks.workunit.client.0.smithi123.stdout:9/987: mknod d6/d7c/de7/df1/d151/c15b 0 2024-03-30T04:28:26.006 INFO:tasks.workunit.client.1.smithi135.stdout:6/606: mknod d2/d68/da8/cbd 0 2024-03-30T04:28:26.007 INFO:tasks.workunit.client.0.smithi123.stdout:9/988: write d6/d7c/fdd [3430315,48709] 0 2024-03-30T04:28:26.009 INFO:tasks.workunit.client.1.smithi135.stdout:9/510: dwrite d7/d29/f62 [0,4194304] 0 2024-03-30T04:28:26.010 INFO:tasks.workunit.client.0.smithi123.stdout:9/989: dread - d6/d1f/ff6 zero size 2024-03-30T04:28:26.011 INFO:tasks.workunit.client.1.smithi135.stdout:6/607: chown d2/dd/d14/d2b/d86/d55/fb1 7284925 1 2024-03-30T04:28:26.014 INFO:tasks.workunit.client.0.smithi123.stdout:9/990: truncate d6/d1f/d45/fa0 2691411 0 2024-03-30T04:28:26.015 INFO:tasks.workunit.client.0.smithi123.stdout:8/977: sync 2024-03-30T04:28:26.015 INFO:tasks.workunit.client.0.smithi123.stdout:4/933: sync 2024-03-30T04:28:26.019 INFO:tasks.workunit.client.0.smithi123.stdout:9/991: mknod d6/d1f/d72/d7d/c15c 0 2024-03-30T04:28:26.020 INFO:tasks.workunit.client.1.smithi135.stdout:9/511: truncate d7/d41/f79 27555 0 2024-03-30T04:28:26.023 INFO:tasks.workunit.client.1.smithi135.stdout:9/512: write d7/d4f/d5c/f9e [902099,25361] 0 2024-03-30T04:28:26.025 INFO:tasks.workunit.client.0.smithi123.stdout:8/978: fdatasync da/d27/d3c/d2d/d74/f7f 0 2024-03-30T04:28:26.028 INFO:tasks.workunit.client.1.smithi135.stdout:9/513: rename d7/f44 to d7/d4f/d8f/fb3 0 2024-03-30T04:28:26.029 INFO:tasks.workunit.client.0.smithi123.stdout:9/992: rename d6/d1f/d4d/fd9 to d6/dad/f15d 0 2024-03-30T04:28:26.030 INFO:tasks.workunit.client.0.smithi123.stdout:8/979: truncate da/dc/d14/f71 1891512 0 2024-03-30T04:28:26.031 INFO:tasks.workunit.client.0.smithi123.stdout:9/993: read - d6/dad/f135 zero size 2024-03-30T04:28:26.034 INFO:tasks.workunit.client.1.smithi135.stdout:9/514: truncate d7/d29/d78/f9b 684512 0 2024-03-30T04:28:26.037 INFO:tasks.workunit.client.0.smithi123.stdout:9/994: creat d6/d1f/d122/f15e x:0 0 0 2024-03-30T04:28:26.040 INFO:tasks.workunit.client.0.smithi123.stdout:4/934: dwrite d2/d4/d2c/d42/d35/d5d/d6f/df9/ffe [0,4194304] 0 2024-03-30T04:28:26.040 INFO:tasks.workunit.client.1.smithi135.stdout:2/465: dread d4/d32/d12/f2a [0,4194304] 0 2024-03-30T04:28:26.042 INFO:tasks.workunit.client.0.smithi123.stdout:9/995: stat d6/d1f/l155 0 2024-03-30T04:28:26.044 INFO:tasks.workunit.client.1.smithi135.stdout:9/515: truncate d7/d29/d78/da4/d37/f4a 182859 0 2024-03-30T04:28:26.048 INFO:tasks.workunit.client.1.smithi135.stdout:9/516: mkdir d7/d41/db4 0 2024-03-30T04:28:26.048 INFO:tasks.workunit.client.0.smithi123.stdout:9/996: dread - d6/d7c/fb4 zero size 2024-03-30T04:28:26.050 INFO:tasks.workunit.client.0.smithi123.stdout:7/929: dread d2/da/d11/d61/f62 [0,4194304] 0 2024-03-30T04:28:26.051 INFO:tasks.workunit.client.0.smithi123.stdout:4/935: rename d2/d4/d2c/d42/d35/d5d/d6f/f8a to d2/dfb/d10c/f130 0 2024-03-30T04:28:26.053 INFO:tasks.workunit.client.0.smithi123.stdout:8/980: dwrite da/dc/f11e [0,4194304] 0 2024-03-30T04:28:26.054 INFO:tasks.workunit.client.1.smithi135.stdout:6/608: dread d2/dd/d14/d2b/f50 [0,4194304] 0 2024-03-30T04:28:26.054 INFO:tasks.workunit.client.1.smithi135.stdout:9/517: mknod d7/d4f/cb5 0 2024-03-30T04:28:26.057 INFO:tasks.workunit.client.0.smithi123.stdout:7/930: write d2/da/d11/d61/d55/d94/f133 [375693,18582] 0 2024-03-30T04:28:26.058 INFO:tasks.workunit.client.1.smithi135.stdout:9/518: stat d7/d90 0 2024-03-30T04:28:26.059 INFO:tasks.workunit.client.0.smithi123.stdout:8/981: chown da/d27/d3c/d64/lbd 89980282 1 2024-03-30T04:28:26.062 INFO:tasks.workunit.client.0.smithi123.stdout:8/982: fdatasync da/dc/d14/d18/f86 0 2024-03-30T04:28:26.062 INFO:tasks.workunit.client.1.smithi135.stdout:2/466: dwrite d4/d55/f5f [0,4194304] 0 2024-03-30T04:28:26.063 INFO:tasks.workunit.client.1.smithi135.stdout:6/609: mkdir d2/dd/d40/d52/d95/dbe 0 2024-03-30T04:28:26.064 INFO:tasks.workunit.client.0.smithi123.stdout:8/983: readlink da/d27/d3c/d2d/l6a 0 2024-03-30T04:28:26.067 INFO:tasks.workunit.client.0.smithi123.stdout:8/984: chown da/d27/d3c/d64/d78/d83/cca 519920 1 2024-03-30T04:28:26.069 INFO:tasks.workunit.client.0.smithi123.stdout:4/936: dwrite d2/d4/d2c/f67 [0,4194304] 0 2024-03-30T04:28:26.070 INFO:tasks.workunit.client.1.smithi135.stdout:2/467: mkdir d4/d32/d2e/d9f 0 2024-03-30T04:28:26.073 INFO:tasks.workunit.client.0.smithi123.stdout:4/937: fdatasync d2/d4/d2c/d42/f4a 0 2024-03-30T04:28:26.073 INFO:tasks.workunit.client.0.smithi123.stdout:8/985: creat da/dc/d14/d9d/dff/d122/f135 x:0 0 0 2024-03-30T04:28:26.073 INFO:tasks.workunit.client.0.smithi123.stdout:7/931: dwrite d2/da/d11/d61/f72 [0,4194304] 0 2024-03-30T04:28:26.078 INFO:tasks.workunit.client.0.smithi123.stdout:7/932: fsync d2/da/d22/d3b/f4b 0 2024-03-30T04:28:26.079 INFO:tasks.workunit.client.1.smithi135.stdout:2/468: chown d4/d55/d77/f58 113 1 2024-03-30T04:28:26.081 INFO:tasks.workunit.client.1.smithi135.stdout:9/519: dwrite d7/d41/f85 [0,4194304] 0 2024-03-30T04:28:26.083 INFO:tasks.workunit.client.1.smithi135.stdout:9/520: chown d7/d4f/d6e/dad 21 1 2024-03-30T04:28:26.086 INFO:tasks.workunit.client.0.smithi123.stdout:4/938: mknod d2/d4/d2c/d42/d89/c131 0 2024-03-30T04:28:26.087 INFO:tasks.workunit.client.1.smithi135.stdout:2/469: getdents d4/d32/d43 0 2024-03-30T04:28:26.089 INFO:tasks.workunit.client.0.smithi123.stdout:4/939: write d2/f86 [4075984,109952] 0 2024-03-30T04:28:26.090 INFO:tasks.workunit.client.1.smithi135.stdout:2/470: write d4/d6/f61 [2305243,55437] 0 2024-03-30T04:28:26.093 INFO:tasks.workunit.client.0.smithi123.stdout:9/997: dwrite d6/d49/f73 [0,4194304] 0 2024-03-30T04:28:26.094 INFO:tasks.workunit.client.0.smithi123.stdout:7/933: mknod d2/d4d/d80/d98/c137 0 2024-03-30T04:28:26.097 INFO:tasks.workunit.client.0.smithi123.stdout:7/934: chown d2/da/d11/d34/db0/ce9 37188073 1 2024-03-30T04:28:26.113 INFO:tasks.workunit.client.0.smithi123.stdout:4/940: dwrite d2/d4/f22 [0,4194304] 0 2024-03-30T04:28:26.115 INFO:tasks.workunit.client.0.smithi123.stdout:9/998: dwrite d6/d1f/d72/f8f [0,4194304] 0 2024-03-30T04:28:26.116 INFO:tasks.workunit.client.0.smithi123.stdout:4/941: truncate d2/d4/f10d 4335014 0 2024-03-30T04:28:26.116 INFO:tasks.workunit.client.1.smithi135.stdout:2/471: dwrite d4/d32/d2e/f40 [0,4194304] 0 2024-03-30T04:28:26.117 INFO:tasks.workunit.client.0.smithi123.stdout:9/999: readlink d6/d1f/d45/d97/l79 0 2024-03-30T04:28:26.117 INFO:tasks.workunit.client.0.smithi123.stdout:4/942: readlink d2/d26/l12b 0 2024-03-30T04:28:26.118 INFO:tasks.workunit.client.0.smithi123.stdout:8/986: dread da/dc/d14/d9d/fb1 [0,4194304] 0 2024-03-30T04:28:26.122 INFO:tasks.workunit.client.1.smithi135.stdout:2/472: rmdir d4/da/d8d 39 2024-03-30T04:28:26.125 INFO:tasks.workunit.client.0.smithi123.stdout:7/935: write d2/dc/d1c/fb2 [698843,118076] 0 2024-03-30T04:28:26.125 INFO:tasks.workunit.client.1.smithi135.stdout:6/610: dwrite d2/dd/d14/d2b/d86/f3c [0,4194304] 0 2024-03-30T04:28:26.126 INFO:tasks.workunit.client.0.smithi123.stdout:4/943: truncate d2/d34/fb4 232152 0 2024-03-30T04:28:26.130 INFO:tasks.workunit.client.0.smithi123.stdout:4/944: chown d2/d4/d7/fc4 0 1 2024-03-30T04:28:26.130 INFO:tasks.workunit.client.1.smithi135.stdout:6/611: creat d2/d68/fbf x:0 0 0 2024-03-30T04:28:26.131 INFO:tasks.workunit.client.1.smithi135.stdout:2/473: creat d4/d32/d12/d45/fa0 x:0 0 0 2024-03-30T04:28:26.132 INFO:tasks.workunit.client.1.smithi135.stdout:6/612: chown d2/f4 6 1 2024-03-30T04:28:26.134 INFO:tasks.workunit.client.0.smithi123.stdout:7/936: read d2/da/d11/d61/d55/d59/db5/f2a [1048863,91035] 0 2024-03-30T04:28:26.135 INFO:tasks.workunit.client.0.smithi123.stdout:4/945: chown d2/d4/d2c/d42/d35/d9b/db7/cf5 239 1 2024-03-30T04:28:26.135 INFO:tasks.workunit.client.0.smithi123.stdout:8/987: link da/d27/d3c/d64/d78/db8/f10c da/d27/d56/d6c/dd4/f136 0 2024-03-30T04:28:26.136 INFO:tasks.workunit.client.1.smithi135.stdout:9/521: dread d7/ff [0,4194304] 0 2024-03-30T04:28:26.139 INFO:tasks.workunit.client.0.smithi123.stdout:4/946: unlink d2/d4/db3/cd7 0 2024-03-30T04:28:26.140 INFO:tasks.workunit.client.0.smithi123.stdout:8/988: chown da/d27/d56/l85 15573546 1 2024-03-30T04:28:26.144 INFO:tasks.workunit.client.1.smithi135.stdout:2/474: rename d4/d32/d2e/d2f/c4c to d4/d32/d12/d45/ca1 0 2024-03-30T04:28:26.147 INFO:tasks.workunit.client.1.smithi135.stdout:6/613: getdents d2/d36 0 2024-03-30T04:28:26.149 INFO:tasks.workunit.client.0.smithi123.stdout:7/937: dread d2/da/d11/d61/f72 [0,4194304] 0 2024-03-30T04:28:26.151 INFO:tasks.workunit.client.0.smithi123.stdout:7/938: rmdir d2/dc/d7f 39 2024-03-30T04:28:26.153 INFO:tasks.workunit.client.1.smithi135.stdout:6/614: symlink d2/dd/d14/d2b/d86/d8e/lc0 0 2024-03-30T04:28:26.154 INFO:tasks.workunit.client.0.smithi123.stdout:7/939: mknod d2/da/d11/d34/c138 0 2024-03-30T04:28:26.159 INFO:tasks.workunit.client.0.smithi123.stdout:5/938: sync 2024-03-30T04:28:26.159 INFO:tasks.workunit.client.0.smithi123.stdout:8/989: dwrite da/d27/d56/d6c/d96/dc2/ff4 [0,4194304] 0 2024-03-30T04:28:26.160 INFO:tasks.workunit.client.1.smithi135.stdout:6/615: rename d2/dd/d40/l92 to d2/lc1 0 2024-03-30T04:28:26.160 INFO:tasks.workunit.client.0.smithi123.stdout:7/940: dread - d2/da/d11/d61/d55/fd4 zero size 2024-03-30T04:28:26.161 INFO:tasks.workunit.client.0.smithi123.stdout:5/939: unlink dd/d19/d3e/d5b/f5e 0 2024-03-30T04:28:26.163 INFO:tasks.workunit.client.0.smithi123.stdout:7/941: creat d2/da/d11/d34/f139 x:0 0 0 2024-03-30T04:28:26.164 INFO:tasks.workunit.client.1.smithi135.stdout:6/616: truncate d2/dd/d14/d30/d43/f78 868542 0 2024-03-30T04:28:26.164 INFO:tasks.workunit.client.0.smithi123.stdout:8/990: mknod da/d27/d3c/d64/d78/db8/dfd/c137 0 2024-03-30T04:28:26.165 INFO:tasks.workunit.client.0.smithi123.stdout:7/942: chown d2/da/d22/fe0 91693057 1 2024-03-30T04:28:26.166 INFO:tasks.workunit.client.0.smithi123.stdout:8/991: chown da/d27/d3c/d64/d78/db8/dfd 43187 1 2024-03-30T04:28:26.169 INFO:tasks.workunit.client.1.smithi135.stdout:6/617: rename d2/dd/d14/c66 to d2/dd/d14/d2b/cc2 0 2024-03-30T04:28:26.170 INFO:tasks.workunit.client.0.smithi123.stdout:8/992: rmdir da/d27/d3c/d2d/d5c 39 2024-03-30T04:28:26.172 INFO:tasks.workunit.client.0.smithi123.stdout:7/943: unlink d2/da/d11/d61/d55/d59/db5/dab/dca/f123 0 2024-03-30T04:28:26.174 INFO:tasks.workunit.client.0.smithi123.stdout:7/944: stat d2/da/d11/d61/d55/d59/db5/dab 0 2024-03-30T04:28:26.179 INFO:tasks.workunit.client.0.smithi123.stdout:7/945: rename d2/da/d11/f48 to d2/da/d11/d61/d55/d59/db5/dab/d107/f13a 0 2024-03-30T04:28:26.183 INFO:tasks.workunit.client.0.smithi123.stdout:7/946: unlink d2/da/d11/d34/d85/fac 0 2024-03-30T04:28:26.184 INFO:tasks.workunit.client.0.smithi123.stdout:7/947: chown d2/da/d11/d61/d55/d59 181126117 1 2024-03-30T04:28:26.185 INFO:tasks.workunit.client.1.smithi135.stdout:6/618: write d2/f12 [1106015,25668] 0 2024-03-30T04:28:26.187 INFO:tasks.workunit.client.0.smithi123.stdout:7/948: unlink d2/da/c6b 0 2024-03-30T04:28:26.189 INFO:tasks.workunit.client.0.smithi123.stdout:7/949: chown d2/dc/d1c/f1d 118835175 1 2024-03-30T04:28:26.190 INFO:tasks.workunit.client.1.smithi135.stdout:9/522: dwrite d7/d4f/d8f/fb3 [0,4194304] 0 2024-03-30T04:28:26.191 INFO:tasks.workunit.client.0.smithi123.stdout:7/950: stat d2/dc/d1c/ce2 0 2024-03-30T04:28:26.191 INFO:tasks.workunit.client.0.smithi123.stdout:5/940: truncate dd/d19/d1f/d35/d3a/d57/db3/fde 1429696 0 2024-03-30T04:28:26.192 INFO:tasks.workunit.client.1.smithi135.stdout:6/619: getdents d2/dd/d14/d38 0 2024-03-30T04:28:26.194 INFO:tasks.workunit.client.0.smithi123.stdout:8/993: truncate da/d27/d3c/d2d/f4c 390700 0 2024-03-30T04:28:26.195 INFO:tasks.workunit.client.1.smithi135.stdout:9/523: chown d7/l4b 13803 1 2024-03-30T04:28:26.196 INFO:tasks.workunit.client.0.smithi123.stdout:7/951: chown d2/da/d11/d61/d55/d59/db5/dab/d107/f13a 638420637 1 2024-03-30T04:28:26.197 INFO:tasks.workunit.client.0.smithi123.stdout:4/947: dwrite d2/d4/d7/d63/db5/fd6 [0,4194304] 0 2024-03-30T04:28:26.202 INFO:tasks.workunit.client.1.smithi135.stdout:6/620: creat d2/dd/d14/d2b/d86/fc3 x:0 0 0 2024-03-30T04:28:26.203 INFO:tasks.workunit.client.0.smithi123.stdout:8/994: creat da/d27/d56/d6c/d96/dc2/f138 x:0 0 0 2024-03-30T04:28:26.206 INFO:tasks.workunit.client.1.smithi135.stdout:6/621: symlink d2/dd/lc4 0 2024-03-30T04:28:26.207 INFO:tasks.workunit.client.0.smithi123.stdout:7/952: write d2/da/d11/d61/d55/d59/db5/dab/d107/f13a [3019773,7506] 0 2024-03-30T04:28:26.209 INFO:tasks.workunit.client.0.smithi123.stdout:7/953: chown d2/d4d/f6a 49 1 2024-03-30T04:28:26.212 INFO:tasks.workunit.client.1.smithi135.stdout:6/622: mkdir d2/dd/d40/d52/d95/dc5 0 2024-03-30T04:28:26.213 INFO:tasks.workunit.client.0.smithi123.stdout:7/954: mkdir d2/d4d/dd0/d13b 0 2024-03-30T04:28:26.216 INFO:tasks.workunit.client.1.smithi135.stdout:6/623: mknod d2/dd/d40/cc6 0 2024-03-30T04:28:26.217 INFO:tasks.workunit.client.0.smithi123.stdout:5/941: write dd/d19/d1f/d3c/d72/fb7 [222004,65375] 0 2024-03-30T04:28:26.217 INFO:tasks.workunit.client.1.smithi135.stdout:6/624: readlink d2/l89 0 2024-03-30T04:28:26.220 INFO:tasks.workunit.client.1.smithi135.stdout:6/625: creat d2/dd/d14/d30/fc7 x:0 0 0 2024-03-30T04:28:26.222 INFO:tasks.workunit.client.1.smithi135.stdout:9/524: dwrite d7/d55/d8d/f9d [0,4194304] 0 2024-03-30T04:28:26.224 INFO:tasks.workunit.client.1.smithi135.stdout:6/626: creat d2/dd/d14/d2b/fc8 x:0 0 0 2024-03-30T04:28:26.229 INFO:tasks.workunit.client.1.smithi135.stdout:9/525: creat d7/d29/d78/da4/fb6 x:0 0 0 2024-03-30T04:28:26.229 INFO:tasks.workunit.client.0.smithi123.stdout:8/995: dwrite da/d27/d3c/d2d/d5c/fdb [0,4194304] 0 2024-03-30T04:28:26.229 INFO:tasks.workunit.client.0.smithi123.stdout:4/948: dwrite d2/d4/fe7 [4194304,4194304] 0 2024-03-30T04:28:26.235 INFO:tasks.workunit.client.0.smithi123.stdout:4/949: creat d2/dab/f132 x:0 0 0 2024-03-30T04:28:26.235 INFO:tasks.workunit.client.1.smithi135.stdout:9/526: rename d7/d4f/d8f/fb3 to d7/d55/fb7 0 2024-03-30T04:28:26.240 INFO:tasks.workunit.client.1.smithi135.stdout:6/627: link d2/dd/c20 d2/dd/d40/d52/d95/dad/cc9 0 2024-03-30T04:28:26.240 INFO:tasks.workunit.client.1.smithi135.stdout:9/527: read d7/d4f/d5c/d63/d86/f9a [467117,43811] 0 2024-03-30T04:28:26.243 INFO:tasks.workunit.client.1.smithi135.stdout:6/628: rmdir d2/d2e 39 2024-03-30T04:28:26.246 INFO:tasks.workunit.client.1.smithi135.stdout:6/629: dread - d2/dd/d14/d2b/f85 zero size 2024-03-30T04:28:26.248 INFO:tasks.workunit.client.1.smithi135.stdout:9/528: unlink d7/d29/d78/da4/c73 0 2024-03-30T04:28:26.249 INFO:tasks.workunit.client.0.smithi123.stdout:5/942: dread dd/d19/d1f/d35/fc4 [0,4194304] 0 2024-03-30T04:28:26.250 INFO:tasks.workunit.client.1.smithi135.stdout:6/630: write d2/dd/d14/d2b/d86/d55/f74 [435840,11642] 0 2024-03-30T04:28:26.250 INFO:tasks.workunit.client.0.smithi123.stdout:4/950: write d2/d4/d2c/d42/d35/d5d/d78/fef [1953380,96040] 0 2024-03-30T04:28:26.253 INFO:tasks.workunit.client.1.smithi135.stdout:9/529: mknod d7/d4f/d6e/d98/db1/cb8 0 2024-03-30T04:28:26.255 INFO:tasks.workunit.client.0.smithi123.stdout:8/996: dwrite da/dc/d14/d18/d62/f9f [8388608,4194304] 0 2024-03-30T04:28:26.255 INFO:tasks.workunit.client.1.smithi135.stdout:9/530: write d7/d29/d78/da4/d37/f6a [4804815,2996] 0 2024-03-30T04:28:26.259 INFO:tasks.workunit.client.1.smithi135.stdout:9/531: symlink d7/lb9 0 2024-03-30T04:28:26.260 INFO:tasks.workunit.client.0.smithi123.stdout:8/997: symlink da/dc/da9/dc4/l139 0 2024-03-30T04:28:26.262 INFO:tasks.workunit.client.0.smithi123.stdout:5/943: creat dd/d19/d1f/d3c/f139 x:0 0 0 2024-03-30T04:28:26.262 INFO:tasks.workunit.client.1.smithi135.stdout:9/532: mkdir d7/d29/d78/da4/d37/dba 0 2024-03-30T04:28:26.263 INFO:tasks.workunit.client.0.smithi123.stdout:4/951: rename d2/d4/db3/ced to d2/d4/d2c/d42/d35/d9b/db7/c133 0 2024-03-30T04:28:26.264 INFO:tasks.workunit.client.0.smithi123.stdout:5/944: chown dd/d19/d5f/f121 99525902 1 2024-03-30T04:28:26.264 INFO:tasks.workunit.client.1.smithi135.stdout:9/533: dread - d7/d41/d72/f7d zero size 2024-03-30T04:28:26.265 INFO:tasks.workunit.client.0.smithi123.stdout:8/998: creat da/d27/d3c/d2d/d11c/f13a x:0 0 0 2024-03-30T04:28:26.266 INFO:tasks.workunit.client.0.smithi123.stdout:5/945: stat dd/d19/d1f/f67 0 2024-03-30T04:28:26.268 INFO:tasks.workunit.client.0.smithi123.stdout:4/952: truncate d2/d4/d2c/d42/d35/f5c 4651818 0 2024-03-30T04:28:26.270 INFO:tasks.workunit.client.0.smithi123.stdout:4/953: chown d2/d4/d2c/d42/d80/caf 1204810128 1 2024-03-30T04:28:26.270 INFO:tasks.workunit.client.0.smithi123.stdout:5/946: creat dd/d19/d1f/d68/dcc/df7/d122/f13a x:0 0 0 2024-03-30T04:28:26.278 INFO:tasks.workunit.client.1.smithi135.stdout:9/534: truncate d7/d4f/d89/f8a 3399749 0 2024-03-30T04:28:26.279 INFO:tasks.workunit.client.1.smithi135.stdout:9/535: symlink d7/d4f/lbb 0 2024-03-30T04:28:26.281 INFO:tasks.workunit.client.0.smithi123.stdout:4/954: dread d2/d4/d2c/d42/d35/d5d/d78/f83 [0,4194304] 0 2024-03-30T04:28:26.282 INFO:tasks.workunit.client.0.smithi123.stdout:5/947: dread dd/d19/f28 [4194304,4194304] 0 2024-03-30T04:28:26.286 INFO:tasks.workunit.client.0.smithi123.stdout:4/955: creat d2/d4/d2c/d42/d80/d96/d12c/f134 x:0 0 0 2024-03-30T04:28:26.286 INFO:tasks.workunit.client.1.smithi135.stdout:9/536: dread d7/d29/d78/da4/f7c [0,4194304] 0 2024-03-30T04:28:26.287 INFO:tasks.workunit.client.0.smithi123.stdout:8/999: dwrite da/dc/d14/d18/d62/fa7 [0,4194304] 0 2024-03-30T04:28:26.287 INFO:tasks.workunit.client.1.smithi135.stdout:9/537: chown d7/d55/d8d/faf 41307779 1 2024-03-30T04:28:26.289 INFO:tasks.workunit.client.1.smithi135.stdout:9/538: chown d7/d29/f2c 1916 1 2024-03-30T04:28:26.296 INFO:tasks.workunit.client.1.smithi135.stdout:9/539: write d7/ff [1763865,44083] 0 2024-03-30T04:28:26.297 INFO:tasks.workunit.client.1.smithi135.stdout:8/504: sync 2024-03-30T04:28:26.299 INFO:tasks.workunit.client.1.smithi135.stdout:9/540: read d7/d29/f51 [3630565,9485] 0 2024-03-30T04:28:26.301 INFO:tasks.workunit.client.1.smithi135.stdout:8/505: unlink d6/d8/d15/d54/f4a 0 2024-03-30T04:28:26.303 INFO:tasks.workunit.client.1.smithi135.stdout:9/541: chown d7/d90/l99 2045 1 2024-03-30T04:28:26.305 INFO:tasks.workunit.client.0.smithi123.stdout:4/956: dwrite d2/d4/d2c/d42/dad/d91/fb1 [0,4194304] 0 2024-03-30T04:28:26.308 INFO:tasks.workunit.client.0.smithi123.stdout:4/957: symlink d2/d4/d2c/d42/d80/d96/l135 0 2024-03-30T04:28:26.310 INFO:tasks.workunit.client.0.smithi123.stdout:5/948: dread dd/d19/d1f/d68/fc6 [0,4194304] 0 2024-03-30T04:28:26.311 INFO:tasks.workunit.client.0.smithi123.stdout:4/958: rmdir d2/d4/d2c/d42/dad/de6 39 2024-03-30T04:28:26.313 INFO:tasks.workunit.client.0.smithi123.stdout:5/949: mkdir dd/d19/d3e/d5b/d62/dd3/d13b 0 2024-03-30T04:28:26.313 INFO:tasks.workunit.client.0.smithi123.stdout:4/959: read - d2/d4/d2c/d42/d89/f8d zero size 2024-03-30T04:28:26.316 INFO:tasks.workunit.client.0.smithi123.stdout:4/960: unlink d2/d4/d7/l129 0 2024-03-30T04:28:26.318 INFO:tasks.workunit.client.0.smithi123.stdout:5/950: rename dd/d19/d1f/l113 to dd/d19/d5f/dc2/l13c 0 2024-03-30T04:28:26.319 INFO:tasks.workunit.client.1.smithi135.stdout:8/506: dwrite d6/d8/d15/d1f/f44 [0,4194304] 0 2024-03-30T04:28:26.322 INFO:tasks.workunit.client.0.smithi123.stdout:4/961: link d2/d4/d2c/d42/fc3 d2/d4/d2c/d42/d35/d5d/d78/f136 0 2024-03-30T04:28:26.323 INFO:tasks.workunit.client.0.smithi123.stdout:5/951: mkdir dd/d19/d1f/d3c/d9e/df5/d129/d13d 0 2024-03-30T04:28:26.328 INFO:tasks.workunit.client.0.smithi123.stdout:4/962: rename d2/d4/d2c/d42/fc3 to d2/dab/f137 0 2024-03-30T04:28:26.328 INFO:tasks.workunit.client.1.smithi135.stdout:8/507: rename d6/d8/d15/d54/d8e/d72/d73/f80 to d6/d8/f97 0 2024-03-30T04:28:26.333 INFO:tasks.workunit.client.0.smithi123.stdout:4/963: rename d2/d4/d2c/d42/f3e to d2/d4/d2c/d42/d35/d9b/da2/deb/f138 0 2024-03-30T04:28:26.338 INFO:tasks.workunit.client.0.smithi123.stdout:5/952: dwrite dd/d19/d1f/d3c/f97 [0,4194304] 0 2024-03-30T04:28:26.339 INFO:tasks.workunit.client.1.smithi135.stdout:8/508: truncate d6/d23/f27 8632154 0 2024-03-30T04:28:26.341 INFO:tasks.workunit.client.0.smithi123.stdout:5/953: dread - dd/d19/d1f/fa5 zero size 2024-03-30T04:28:26.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:25 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:28:26.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:25 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:28:26.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:25 smithi135 ceph-mon[44364]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-30T04:28:26.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:25 smithi135 ceph-mon[44364]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-30T04:28:26.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:25 smithi135 ceph-mon[44364]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-30T04:28:26.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:25 smithi135 ceph-mon[44364]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-30T04:28:26.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:25 smithi135 ceph-mon[44364]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-30T04:28:26.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:25 smithi135 ceph-mon[44364]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-30T04:28:26.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:25 smithi135 ceph-mon[44364]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:28:26.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:25 smithi135 ceph-mon[44364]: pgmap v46: 65 pgs: 65 active+clean; 2.8 GiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 103 MiB/s rd, 271 MiB/s wr, 551 op/s 2024-03-30T04:28:26.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:25 smithi135 ceph-mon[44364]: Upgrade: Need to upgrade myself (mgr.smithi135.kyblli) 2024-03-30T04:28:26.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:25 smithi135 ceph-mon[44364]: Upgrade: Need to upgrade myself (mgr.smithi135.kyblli) 2024-03-30T04:28:26.359 INFO:tasks.workunit.client.0.smithi123.stdout:4/964: write d2/d4/d7/fe5 [924581,4446] 0 2024-03-30T04:28:26.362 INFO:tasks.workunit.client.1.smithi135.stdout:8/509: dwrite d6/d23/f27 [4194304,4194304] 0 2024-03-30T04:28:26.368 INFO:tasks.workunit.client.0.smithi123.stdout:5/954: dwrite dd/f15 [8388608,4194304] 0 2024-03-30T04:28:26.384 INFO:tasks.workunit.client.1.smithi135.stdout:8/510: write d6/d8/f97 [2600999,79517] 0 2024-03-30T04:28:26.384 INFO:tasks.workunit.client.0.smithi123.stdout:4/965: mkdir d2/d4/d7/d1a/d139 0 2024-03-30T04:28:26.386 INFO:tasks.workunit.client.0.smithi123.stdout:5/955: rmdir dd/d19/d3e/d5b/d62 39 2024-03-30T04:28:26.387 INFO:tasks.workunit.client.1.smithi135.stdout:8/511: chown d6/d8/d14/c77 592483648 1 2024-03-30T04:28:26.387 INFO:tasks.workunit.client.0.smithi123.stdout:4/966: creat d2/d4/db3/f13a x:0 0 0 2024-03-30T04:28:26.388 INFO:tasks.workunit.client.1.smithi135.stdout:8/512: chown d6/d8/d14/f67 1584 1 2024-03-30T04:28:26.388 INFO:tasks.workunit.client.0.smithi123.stdout:5/956: fsync dd/d19/d1f/d68/f93 0 2024-03-30T04:28:26.392 INFO:tasks.workunit.client.0.smithi123.stdout:4/967: creat d2/d4/d2c/d42/d35/d5d/f13b x:0 0 0 2024-03-30T04:28:26.392 INFO:tasks.workunit.client.1.smithi135.stdout:8/513: rmdir d6/d8/d15/d2a/d35/d93 0 2024-03-30T04:28:26.393 INFO:tasks.workunit.client.0.smithi123.stdout:5/957: creat dd/d19/d1f/d68/dcc/d12f/f13e x:0 0 0 2024-03-30T04:28:26.393 INFO:tasks.workunit.client.1.smithi135.stdout:8/514: readlink d6/d8/d15/l3d 0 2024-03-30T04:28:26.396 INFO:tasks.workunit.client.1.smithi135.stdout:8/515: creat d6/d23/d7b/f98 x:0 0 0 2024-03-30T04:28:26.397 INFO:tasks.workunit.client.0.smithi123.stdout:4/968: link d2/d4/fa d2/d4/d7/dc1/d113/f13c 0 2024-03-30T04:28:26.400 INFO:tasks.workunit.client.1.smithi135.stdout:8/516: symlink d6/d8/d15/l99 0 2024-03-30T04:28:26.402 INFO:tasks.workunit.client.0.smithi123.stdout:4/969: link d2/d4/c2f d2/d4/d7/dc1/d103/c13d 0 2024-03-30T04:28:26.403 INFO:tasks.workunit.client.1.smithi135.stdout:8/517: rename d6/d8/d15/d2a/d35/f45 to d6/d8/d14/d32/f9a 0 2024-03-30T04:28:26.404 INFO:tasks.workunit.client.1.smithi135.stdout:8/518: fsync d6/d8/d14/f4e 0 2024-03-30T04:28:26.406 INFO:tasks.workunit.client.0.smithi123.stdout:4/970: symlink d2/d4/d2c/d42/d35/d5d/d78/d111/l13e 0 2024-03-30T04:28:26.407 INFO:tasks.workunit.client.0.smithi123.stdout:4/971: readlink d2/d4/d7/l79 0 2024-03-30T04:28:26.410 INFO:tasks.workunit.client.0.smithi123.stdout:4/972: creat d2/d4/d7/d63/f13f x:0 0 0 2024-03-30T04:28:26.412 INFO:tasks.workunit.client.1.smithi135.stdout:8/519: truncate d6/fa 4816826 0 2024-03-30T04:28:26.414 INFO:tasks.workunit.client.0.smithi123.stdout:4/973: getdents d2/d4/d2c/d42/dad 0 2024-03-30T04:28:26.428 INFO:tasks.workunit.client.0.smithi123.stdout:5/958: dwrite dd/d19/d1f/d35/faf [0,4194304] 0 2024-03-30T04:28:26.432 INFO:tasks.workunit.client.0.smithi123.stdout:4/974: dwrite d2/d4/d2c/d42/f6a [0,4194304] 0 2024-03-30T04:28:26.436 INFO:tasks.workunit.client.0.smithi123.stdout:5/959: symlink dd/d19/d1f/d35/d3a/d57/db3/l13f 0 2024-03-30T04:28:26.438 INFO:tasks.workunit.client.0.smithi123.stdout:7/955: sync 2024-03-30T04:28:26.443 INFO:tasks.workunit.client.0.smithi123.stdout:5/960: creat dd/d19/d1f/d35/db8/d11a/f140 x:0 0 0 2024-03-30T04:28:26.447 INFO:tasks.workunit.client.0.smithi123.stdout:7/956: truncate d2/da/d22/fe0 1467561 0 2024-03-30T04:28:26.456 INFO:tasks.workunit.client.0.smithi123.stdout:7/957: mkdir d2/d4d/d80/df3/d13c 0 2024-03-30T04:28:26.457 INFO:tasks.workunit.client.0.smithi123.stdout:4/975: dread d2/d4/d2c/d42/f6a [0,4194304] 0 2024-03-30T04:28:26.459 INFO:tasks.workunit.client.0.smithi123.stdout:7/958: write d2/dc/d115/f9b [747106,42832] 0 2024-03-30T04:28:26.464 INFO:tasks.workunit.client.0.smithi123.stdout:4/976: read d2/d4/d2c/d42/d35/d5d/d78/fef [1399100,62076] 0 2024-03-30T04:28:26.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:25 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:28:26.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:25 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:28:26.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:25 smithi123 ceph-mon[35245]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-30T04:28:26.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:25 smithi123 ceph-mon[35245]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-30T04:28:26.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:25 smithi123 ceph-mon[35245]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-30T04:28:26.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:25 smithi123 ceph-mon[35245]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-30T04:28:26.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:25 smithi123 ceph-mon[35245]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-30T04:28:26.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:25 smithi123 ceph-mon[35245]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-30T04:28:26.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:25 smithi123 ceph-mon[35245]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:28:26.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:25 smithi123 ceph-mon[35245]: pgmap v46: 65 pgs: 65 active+clean; 2.8 GiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 103 MiB/s rd, 271 MiB/s wr, 551 op/s 2024-03-30T04:28:26.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:25 smithi123 ceph-mon[35245]: Upgrade: Need to upgrade myself (mgr.smithi135.kyblli) 2024-03-30T04:28:26.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:25 smithi123 ceph-mon[35245]: Upgrade: Need to upgrade myself (mgr.smithi135.kyblli) 2024-03-30T04:28:26.467 INFO:tasks.workunit.client.0.smithi123.stdout:4/977: rmdir d2/d4/d2c/d42 39 2024-03-30T04:28:26.477 INFO:tasks.workunit.client.0.smithi123.stdout:4/978: dread d2/d26/f70 [0,4194304] 0 2024-03-30T04:28:26.478 INFO:tasks.workunit.client.0.smithi123.stdout:5/961: dread dd/d19/d3e/d5b/d62/f7a [0,4194304] 0 2024-03-30T04:28:26.482 INFO:tasks.workunit.client.0.smithi123.stdout:5/962: symlink dd/d19/d1f/d35/d3a/l141 0 2024-03-30T04:28:26.483 INFO:tasks.workunit.client.0.smithi123.stdout:7/959: dread d2/dc/d49/f63 [0,4194304] 0 2024-03-30T04:28:26.484 INFO:tasks.workunit.client.0.smithi123.stdout:5/963: chown dd/d19/d5f/d99/f133 1773895186 1 2024-03-30T04:28:26.486 INFO:tasks.workunit.client.0.smithi123.stdout:7/960: symlink d2/da/d11/d61/d55/d59/l13d 0 2024-03-30T04:28:26.493 INFO:tasks.workunit.client.0.smithi123.stdout:7/961: dread d2/dc/d7f/fa2 [0,4194304] 0 2024-03-30T04:28:26.493 INFO:tasks.workunit.client.0.smithi123.stdout:4/979: dwrite d2/d34/f3f [0,4194304] 0 2024-03-30T04:28:26.497 INFO:tasks.workunit.client.1.smithi135.stdout:5/515: sync 2024-03-30T04:28:26.497 INFO:tasks.workunit.client.1.smithi135.stdout:4/554: sync 2024-03-30T04:28:26.497 INFO:tasks.workunit.client.1.smithi135.stdout:7/625: sync 2024-03-30T04:28:26.497 INFO:tasks.workunit.client.1.smithi135.stdout:2/475: sync 2024-03-30T04:28:26.497 INFO:tasks.workunit.client.1.smithi135.stdout:0/643: sync 2024-03-30T04:28:26.497 INFO:tasks.workunit.client.1.smithi135.stdout:3/551: sync 2024-03-30T04:28:26.497 INFO:tasks.workunit.client.1.smithi135.stdout:1/539: sync 2024-03-30T04:28:26.502 INFO:tasks.workunit.client.1.smithi135.stdout:7/626: readlink d4/db/d1a/d5c/d5d/lad 0 2024-03-30T04:28:26.503 INFO:tasks.workunit.client.1.smithi135.stdout:5/516: write d1/d45/f6b [573459,125392] 0 2024-03-30T04:28:26.505 INFO:tasks.workunit.client.1.smithi135.stdout:3/552: creat d3/d50/d22/fc0 x:0 0 0 2024-03-30T04:28:26.507 INFO:tasks.workunit.client.1.smithi135.stdout:4/555: chown d10/d1a/d32/f96 6 1 2024-03-30T04:28:26.509 INFO:tasks.workunit.client.0.smithi123.stdout:7/962: dwrite d2/da/d11/d34/f70 [4194304,4194304] 0 2024-03-30T04:28:26.509 INFO:tasks.workunit.client.1.smithi135.stdout:1/540: getdents d16/d5a/dad 0 2024-03-30T04:28:26.514 INFO:tasks.workunit.client.1.smithi135.stdout:1/541: unlink d16/d5a/d5f/d3d/f86 0 2024-03-30T04:28:26.519 INFO:tasks.workunit.client.1.smithi135.stdout:0/644: dread d0/f12 [0,4194304] 0 2024-03-30T04:28:26.521 INFO:tasks.workunit.client.0.smithi123.stdout:7/963: write d2/da/d11/d61/d55/d59/db5/dab/dca/fd7 [1638134,93704] 0 2024-03-30T04:28:26.523 INFO:tasks.workunit.client.0.smithi123.stdout:7/964: readlink d2/dc/d1c/lcd 0 2024-03-30T04:28:26.530 INFO:tasks.workunit.client.0.smithi123.stdout:4/980: dread d2/d4/fe7 [0,4194304] 0 2024-03-30T04:28:26.532 INFO:tasks.workunit.client.0.smithi123.stdout:4/981: chown d2/d34/dd2 2196651 1 2024-03-30T04:28:26.533 INFO:tasks.workunit.client.1.smithi135.stdout:0/645: fsync d0/d5/d9/d23/d2a/f51 0 2024-03-30T04:28:26.534 INFO:tasks.workunit.client.0.smithi123.stdout:7/965: truncate d2/f13 2634447 0 2024-03-30T04:28:26.534 INFO:tasks.workunit.client.0.smithi123.stdout:4/982: write d2/d4/d7/fb [415659,49919] 0 2024-03-30T04:28:26.535 INFO:tasks.workunit.client.1.smithi135.stdout:4/556: dwrite fa [0,4194304] 0 2024-03-30T04:28:26.538 INFO:tasks.workunit.client.1.smithi135.stdout:5/517: dwrite d1/d7/ff [0,4194304] 0 2024-03-30T04:28:26.539 INFO:tasks.workunit.client.0.smithi123.stdout:7/966: mknod d2/da/d11/d61/d55/d59/db5/dab/c13e 0 2024-03-30T04:28:26.541 INFO:tasks.workunit.client.0.smithi123.stdout:7/967: stat d2/da/d11/d61/d55/f10a 0 2024-03-30T04:28:26.542 INFO:tasks.workunit.client.1.smithi135.stdout:0/646: symlink d0/d5/d42/d87/lcd 0 2024-03-30T04:28:26.543 INFO:tasks.workunit.client.1.smithi135.stdout:7/627: dwrite d4/db/d1a/d5c/d82/fa2 [0,4194304] 0 2024-03-30T04:28:26.543 INFO:tasks.workunit.client.1.smithi135.stdout:1/542: dread d16/f2f [0,4194304] 0 2024-03-30T04:28:26.543 INFO:tasks.workunit.client.1.smithi135.stdout:4/557: mkdir d10/d1a/d42/d66/dc2 0 2024-03-30T04:28:26.544 INFO:tasks.workunit.client.1.smithi135.stdout:5/518: truncate d1/d3/d80/f93 519515 0 2024-03-30T04:28:26.545 INFO:tasks.workunit.client.0.smithi123.stdout:7/968: creat d2/dc/d7f/d10c/f13f x:0 0 0 2024-03-30T04:28:26.548 INFO:tasks.workunit.client.0.smithi123.stdout:7/969: mkdir d2/dc/d49/dbc/d140 0 2024-03-30T04:28:26.549 INFO:tasks.workunit.client.1.smithi135.stdout:4/558: mkdir d10/d4d/dc3 0 2024-03-30T04:28:26.549 INFO:tasks.workunit.client.1.smithi135.stdout:7/628: fsync d4/db/d1a/d5c/d2c/d51/d56/f9e 0 2024-03-30T04:28:26.549 INFO:tasks.workunit.client.1.smithi135.stdout:5/519: fdatasync d1/d7/d11/f87 0 2024-03-30T04:28:26.549 INFO:tasks.workunit.client.1.smithi135.stdout:0/647: symlink d0/d5/d9/d23/d58/d3b/lce 0 2024-03-30T04:28:26.549 INFO:tasks.workunit.client.1.smithi135.stdout:2/476: dread d4/da/fb [0,4194304] 0 2024-03-30T04:28:26.553 INFO:tasks.workunit.client.0.smithi123.stdout:7/970: creat d2/dc/d115/f141 x:0 0 0 2024-03-30T04:28:26.554 INFO:tasks.workunit.client.1.smithi135.stdout:2/477: chown d4/d32/l9d 2582438 1 2024-03-30T04:28:26.556 INFO:tasks.workunit.client.0.smithi123.stdout:4/983: dread d2/f1c [0,4194304] 0 2024-03-30T04:28:26.556 INFO:tasks.workunit.client.1.smithi135.stdout:7/629: write d4/db/d79/da9/f58 [155780,129300] 0 2024-03-30T04:28:26.557 INFO:tasks.workunit.client.0.smithi123.stdout:7/971: mkdir d2/da/d142 0 2024-03-30T04:28:26.557 INFO:tasks.workunit.client.1.smithi135.stdout:0/648: truncate d0/d5/d9/d23/d58/d63/f89 696485 0 2024-03-30T04:28:26.559 INFO:tasks.workunit.client.1.smithi135.stdout:1/543: dread d16/f1c [0,4194304] 0 2024-03-30T04:28:26.559 INFO:tasks.workunit.client.1.smithi135.stdout:5/520: getdents d1/d3/d2b 0 2024-03-30T04:28:26.562 INFO:tasks.workunit.client.0.smithi123.stdout:4/984: stat d2/d4/d2c/d42/d80/da5 0 2024-03-30T04:28:26.562 INFO:tasks.workunit.client.0.smithi123.stdout:7/972: fdatasync d2/da/d11/d61/d55/d59/db5/f11b 0 2024-03-30T04:28:26.562 INFO:tasks.workunit.client.0.smithi123.stdout:4/985: readlink d2/d4/d7/d63/l7a 0 2024-03-30T04:28:26.563 INFO:tasks.workunit.client.1.smithi135.stdout:1/544: readlink d16/d19/da4/l27 0 2024-03-30T04:28:26.563 INFO:tasks.workunit.client.1.smithi135.stdout:0/649: dread - d0/d5/d9/d23/d58/d3b/fbc zero size 2024-03-30T04:28:26.565 INFO:tasks.workunit.client.0.smithi123.stdout:4/986: rmdir d2/d34/dd2 39 2024-03-30T04:28:26.565 INFO:tasks.workunit.client.0.smithi123.stdout:7/973: write d2/da/d11/d34/d85/fae [1520062,8500] 0 2024-03-30T04:28:26.570 INFO:tasks.workunit.client.1.smithi135.stdout:1/545: creat d16/d5a/dad/d77/fb6 x:0 0 0 2024-03-30T04:28:26.571 INFO:tasks.workunit.client.0.smithi123.stdout:4/987: mknod d2/d4/de0/c140 0 2024-03-30T04:28:26.571 INFO:tasks.workunit.client.1.smithi135.stdout:4/559: dwrite d10/d1a/f23 [0,4194304] 0 2024-03-30T04:28:26.572 INFO:tasks.workunit.client.1.smithi135.stdout:0/650: creat d0/d5/d9/d23/d58/d63/fcf x:0 0 0 2024-03-30T04:28:26.574 INFO:tasks.workunit.client.0.smithi123.stdout:4/988: symlink d2/d34/dd2/l141 0 2024-03-30T04:28:26.578 INFO:tasks.workunit.client.1.smithi135.stdout:1/546: rename d16/d5a/d5f/d3a/f52 to d16/d29/d2c/fb7 0 2024-03-30T04:28:26.579 INFO:tasks.workunit.client.0.smithi123.stdout:4/989: mknod d2/d4/d7/d63/c142 0 2024-03-30T04:28:26.585 INFO:tasks.workunit.client.0.smithi123.stdout:4/990: creat d2/d4/d2c/d42/d35/d9b/db7/f143 x:0 0 0 2024-03-30T04:28:26.585 INFO:tasks.workunit.client.1.smithi135.stdout:0/651: mknod d0/d1/d16/d7e/cd0 0 2024-03-30T04:28:26.585 INFO:tasks.workunit.client.1.smithi135.stdout:4/560: fdatasync d10/d4d/d89/f8f 0 2024-03-30T04:28:26.586 INFO:tasks.workunit.client.0.smithi123.stdout:7/974: dwrite d2/da/d11/d34/d85/d8e/f129 [0,4194304] 0 2024-03-30T04:28:26.588 INFO:tasks.workunit.client.0.smithi123.stdout:4/991: creat d2/d34/dd2/f144 x:0 0 0 2024-03-30T04:28:26.590 INFO:tasks.workunit.client.1.smithi135.stdout:2/478: dwrite d4/d32/d12/d45/f39 [0,4194304] 0 2024-03-30T04:28:26.591 INFO:tasks.workunit.client.1.smithi135.stdout:7/630: dwrite d4/db/d1a/d5c/d2c/f63 [0,4194304] 0 2024-03-30T04:28:26.592 INFO:tasks.workunit.client.1.smithi135.stdout:1/547: write d16/d5a/dad/d77/f7a [2013482,56070] 0 2024-03-30T04:28:26.592 INFO:tasks.workunit.client.0.smithi123.stdout:7/975: symlink d2/da/d11/d61/d55/l143 0 2024-03-30T04:28:26.597 INFO:tasks.workunit.client.0.smithi123.stdout:7/976: read d2/f13 [585383,40479] 0 2024-03-30T04:28:26.599 INFO:tasks.workunit.client.1.smithi135.stdout:5/521: write d1/d3/f49 [1342368,74337] 0 2024-03-30T04:28:26.600 INFO:tasks.workunit.client.0.smithi123.stdout:7/977: creat d2/d4d/d80/df3/f144 x:0 0 0 2024-03-30T04:28:26.602 INFO:tasks.workunit.client.0.smithi123.stdout:7/978: symlink d2/da/d11/d61/d55/d9e/l145 0 2024-03-30T04:28:26.603 INFO:tasks.workunit.client.1.smithi135.stdout:4/561: creat d10/d1a/d42/d46/fc4 x:0 0 0 2024-03-30T04:28:26.603 INFO:tasks.workunit.client.0.smithi123.stdout:7/979: chown d2/l5 6 1 2024-03-30T04:28:26.605 INFO:tasks.workunit.client.0.smithi123.stdout:7/980: chown d2/dc/d1c/ce2 7 1 2024-03-30T04:28:26.607 INFO:tasks.workunit.client.0.smithi123.stdout:7/981: stat d2/da/d11/d61/d55/d94/cf1 0 2024-03-30T04:28:26.612 INFO:tasks.workunit.client.1.smithi135.stdout:1/548: symlink d16/d29/d32/d85/lb8 0 2024-03-30T04:28:26.613 INFO:tasks.workunit.client.1.smithi135.stdout:5/522: dread - d1/d7/f59 zero size 2024-03-30T04:28:26.614 INFO:tasks.workunit.client.1.smithi135.stdout:7/631: creat d4/db/d1a/d5c/d5d/db5/fd2 x:0 0 0 2024-03-30T04:28:26.616 INFO:tasks.workunit.client.1.smithi135.stdout:5/523: dread - d1/d45/f88 zero size 2024-03-30T04:28:26.617 INFO:tasks.workunit.client.0.smithi123.stdout:4/992: dwrite d2/d4/d2c/d42/d35/d5d/fdf [0,4194304] 0 2024-03-30T04:28:26.617 INFO:tasks.workunit.client.1.smithi135.stdout:1/549: mknod d16/d29/d32/cb9 0 2024-03-30T04:28:26.617 INFO:tasks.workunit.client.1.smithi135.stdout:2/479: write d4/d32/d2e/f8c [386867,113556] 0 2024-03-30T04:28:26.620 INFO:tasks.workunit.client.0.smithi123.stdout:4/993: symlink d2/d4/d2c/d42/d35/d9b/db7/l145 0 2024-03-30T04:28:26.621 INFO:tasks.workunit.client.1.smithi135.stdout:4/562: read d10/d1a/d32/f70 [800203,102947] 0 2024-03-30T04:28:26.621 INFO:tasks.workunit.client.1.smithi135.stdout:7/632: creat d4/db/de/d5a/dae/fd3 x:0 0 0 2024-03-30T04:28:26.622 INFO:tasks.workunit.client.1.smithi135.stdout:5/524: fdatasync d1/d3/d2b/f5b 0 2024-03-30T04:28:26.624 INFO:tasks.workunit.client.0.smithi123.stdout:4/994: rename d2/dab/c119 to d2/d4/d2c/d42/d35/d5d/d11e/c146 0 2024-03-30T04:28:26.627 INFO:tasks.workunit.client.1.smithi135.stdout:1/550: rename d16/f25 to d16/d29/d32/d85/fba 0 2024-03-30T04:28:26.628 INFO:tasks.workunit.client.1.smithi135.stdout:6/631: sync 2024-03-30T04:28:26.631 INFO:tasks.workunit.client.1.smithi135.stdout:1/551: read d16/d19/f2b [1987594,44526] 0 2024-03-30T04:28:26.632 INFO:tasks.workunit.client.1.smithi135.stdout:4/563: creat d10/d4d/d9c/fc5 x:0 0 0 2024-03-30T04:28:26.634 INFO:tasks.workunit.client.1.smithi135.stdout:7/633: dread d4/db/d1a/d4b/d40/d6e/fc6 [0,4194304] 0 2024-03-30T04:28:26.635 INFO:tasks.workunit.client.1.smithi135.stdout:0/652: truncate d0/f12 3661866 0 2024-03-30T04:28:26.639 INFO:tasks.workunit.client.0.smithi123.stdout:4/995: write d2/d4/d7/fc4 [2616577,109754] 0 2024-03-30T04:28:26.641 INFO:tasks.workunit.client.0.smithi123.stdout:7/982: dwrite d2/da/d22/d3b/f8f [0,4194304] 0 2024-03-30T04:28:26.643 INFO:tasks.workunit.client.0.smithi123.stdout:4/996: rmdir d2/d4/d7/d63/db5 39 2024-03-30T04:28:26.645 INFO:tasks.workunit.client.0.smithi123.stdout:7/983: creat d2/da/d22/f146 x:0 0 0 2024-03-30T04:28:26.648 INFO:tasks.workunit.client.0.smithi123.stdout:7/984: creat d2/da/d11/d34/ded/dfc/f147 x:0 0 0 2024-03-30T04:28:26.650 INFO:tasks.workunit.client.0.smithi123.stdout:7/985: chown d2/dc/d49/dbc/d140 15757 1 2024-03-30T04:28:26.650 INFO:tasks.workunit.client.0.smithi123.stdout:4/997: rename d2/d4/d2c/d42/d35/d9b/da2/deb/l106 to d2/d4/d2c/d42/d80/l147 0 2024-03-30T04:28:26.651 INFO:tasks.workunit.client.1.smithi135.stdout:1/552: creat d16/d29/fbb x:0 0 0 2024-03-30T04:28:26.654 INFO:tasks.workunit.client.0.smithi123.stdout:7/986: mkdir d2/d4d/d80/d148 0 2024-03-30T04:28:26.657 INFO:tasks.workunit.client.0.smithi123.stdout:7/987: creat d2/d4d/d80/d148/f149 x:0 0 0 2024-03-30T04:28:26.657 INFO:tasks.workunit.client.1.smithi135.stdout:6/632: rename d2/dd/d14/d53/f77 to d2/fca 0 2024-03-30T04:28:26.657 INFO:tasks.workunit.client.1.smithi135.stdout:5/525: dwrite d1/d7/f43 [4194304,4194304] 0 2024-03-30T04:28:26.661 INFO:tasks.workunit.client.1.smithi135.stdout:2/480: dwrite d4/d32/d43/f63 [0,4194304] 0 2024-03-30T04:28:26.667 INFO:tasks.workunit.client.1.smithi135.stdout:6/633: unlink d2/f11 0 2024-03-30T04:28:26.669 INFO:tasks.workunit.client.1.smithi135.stdout:1/553: dread d16/d5a/d5f/d3d/f96 [0,4194304] 0 2024-03-30T04:28:26.671 INFO:tasks.workunit.client.0.smithi123.stdout:4/998: dwrite d2/d4/d2c/d42/d35/d5d/dc0/fce [0,4194304] 0 2024-03-30T04:28:26.673 INFO:tasks.workunit.client.1.smithi135.stdout:5/526: read d1/f36 [110134,125460] 0 2024-03-30T04:28:26.673 INFO:tasks.workunit.client.1.smithi135.stdout:2/481: symlink d4/d32/d43/la2 0 2024-03-30T04:28:26.675 INFO:tasks.workunit.client.1.smithi135.stdout:0/653: write d0/d5/d9/d23/d2a/d60/f48 [802977,25999] 0 2024-03-30T04:28:26.677 INFO:tasks.workunit.client.0.smithi123.stdout:7/988: dwrite d2/d4d/f6a [4194304,4194304] 0 2024-03-30T04:28:26.679 INFO:tasks.workunit.client.1.smithi135.stdout:2/482: truncate d4/d6/d47/f98 1075578 0 2024-03-30T04:28:26.682 INFO:tasks.workunit.client.0.smithi123.stdout:4/999: unlink d2/d4/d2c/fcd 0 2024-03-30T04:28:26.683 INFO:tasks.workunit.client.0.smithi123.stdout:7/989: creat d2/da/d11/d61/d55/d9e/f14a x:0 0 0 2024-03-30T04:28:26.683 INFO:tasks.workunit.client.1.smithi135.stdout:7/634: dwrite d4/db/d1a/d3d/f5b [0,4194304] 0 2024-03-30T04:28:26.685 INFO:tasks.workunit.client.0.smithi123.stdout:7/990: rmdir d2/da/d11/d61/d55/d59/db5 39 2024-03-30T04:28:26.685 INFO:tasks.workunit.client.1.smithi135.stdout:1/554: dread - d16/d29/d32/f46 zero size 2024-03-30T04:28:26.687 INFO:tasks.workunit.client.1.smithi135.stdout:2/483: symlink d4/d32/d12/d96/la3 0 2024-03-30T04:28:26.688 INFO:tasks.workunit.client.0.smithi123.stdout:7/991: unlink d2/d4d/d80/d98/ca5 0 2024-03-30T04:28:26.690 INFO:tasks.workunit.client.1.smithi135.stdout:2/484: fdatasync d4/d32/f78 0 2024-03-30T04:28:26.696 INFO:tasks.workunit.client.1.smithi135.stdout:0/654: mkdir d0/d13/d1e/d3a/dc2/dd1 0 2024-03-30T04:28:26.696 INFO:tasks.workunit.client.1.smithi135.stdout:5/527: getdents d1/d45/d52 0 2024-03-30T04:28:26.698 INFO:tasks.workunit.client.1.smithi135.stdout:7/635: rename d4/db/d1a/d5c/f4e to d4/db/d1a/d3d/fd4 0 2024-03-30T04:28:26.701 INFO:tasks.workunit.client.1.smithi135.stdout:0/655: write d0/d5/faf [1019241,32638] 0 2024-03-30T04:28:26.705 INFO:tasks.workunit.client.1.smithi135.stdout:5/528: write d1/d7/d11/f8a [804562,116693] 0 2024-03-30T04:28:26.708 INFO:tasks.workunit.client.1.smithi135.stdout:1/555: mkdir d16/d5a/d5f/d3a/dbc 0 2024-03-30T04:28:26.712 INFO:tasks.workunit.client.1.smithi135.stdout:6/634: dwrite d2/dd/d14/d2b/f50 [0,4194304] 0 2024-03-30T04:28:26.712 INFO:tasks.workunit.client.1.smithi135.stdout:7/636: getdents d4/db/d1a/d5c/d2c/d83 0 2024-03-30T04:28:26.713 INFO:tasks.workunit.client.1.smithi135.stdout:6/635: chown d2/dd/d14/f1e 1118859152 1 2024-03-30T04:28:26.715 INFO:tasks.workunit.client.1.smithi135.stdout:5/529: mkdir d1/d7/d1a/d1f/d9b 0 2024-03-30T04:28:26.716 INFO:tasks.workunit.client.0.smithi123.stdout:5/964: sync 2024-03-30T04:28:26.721 INFO:tasks.workunit.client.0.smithi123.stdout:5/965: symlink dd/d19/d1f/d35/d3a/d57/db5/l142 0 2024-03-30T04:28:26.721 INFO:tasks.workunit.client.1.smithi135.stdout:7/637: mknod d4/db/d1a/d4b/cd5 0 2024-03-30T04:28:26.722 INFO:tasks.workunit.client.1.smithi135.stdout:6/636: creat d2/d68/da8/fcb x:0 0 0 2024-03-30T04:28:26.722 INFO:tasks.workunit.client.0.smithi123.stdout:5/966: dread - dd/d19/d1f/d3c/d9e/f10f zero size 2024-03-30T04:28:26.728 INFO:tasks.workunit.client.0.smithi123.stdout:7/992: truncate d2/da/d11/d34/d85/d8e/f129 2864608 0 2024-03-30T04:28:26.729 INFO:tasks.workunit.client.1.smithi135.stdout:6/637: chown d2/dd/c32 110085 1 2024-03-30T04:28:26.730 INFO:tasks.workunit.client.0.smithi123.stdout:5/967: fdatasync dd/d19/d1f/d3c/f56 0 2024-03-30T04:28:26.730 INFO:tasks.workunit.client.1.smithi135.stdout:7/638: rmdir d4/db/d1a/d3d/d96 39 2024-03-30T04:28:26.734 INFO:tasks.workunit.client.0.smithi123.stdout:5/968: read - dd/d19/d1f/d35/d3a/d57/de0/f137 zero size 2024-03-30T04:28:26.734 INFO:tasks.workunit.client.1.smithi135.stdout:6/638: write d2/dd/d14/f3a [3221658,4653] 0 2024-03-30T04:28:26.736 INFO:tasks.workunit.client.1.smithi135.stdout:5/530: link d1/d3/d76/f98 d1/d7/d1a/d94/f9c 0 2024-03-30T04:28:26.741 INFO:tasks.workunit.client.1.smithi135.stdout:7/639: truncate d4/db/d1a/d5c/d2c/d51/da8/fc2 301299 0 2024-03-30T04:28:26.741 INFO:tasks.workunit.client.0.smithi123.stdout:5/969: creat dd/d19/d1f/d3c/d9e/df5/d129/d13d/f143 x:0 0 0 2024-03-30T04:28:26.745 INFO:tasks.workunit.client.1.smithi135.stdout:7/640: chown d4/db/d1a/d90/fa3 956 1 2024-03-30T04:28:26.750 INFO:tasks.workunit.client.1.smithi135.stdout:1/556: dwrite d16/d29/d2c/fb7 [4194304,4194304] 0 2024-03-30T04:28:26.753 INFO:tasks.workunit.client.1.smithi135.stdout:7/641: symlink d4/db/d1a/d3d/ld6 0 2024-03-30T04:28:26.756 INFO:tasks.workunit.client.1.smithi135.stdout:1/557: symlink d16/d29/d32/d9d/lbd 0 2024-03-30T04:28:26.759 INFO:tasks.workunit.client.1.smithi135.stdout:0/656: dread d0/d13/d1e/f78 [0,4194304] 0 2024-03-30T04:28:26.761 INFO:tasks.workunit.client.0.smithi123.stdout:7/993: dwrite d2/da/d22/fe0 [0,4194304] 0 2024-03-30T04:28:26.764 INFO:tasks.workunit.client.1.smithi135.stdout:0/657: mkdir d0/d13/d91/dd2 0 2024-03-30T04:28:26.765 INFO:tasks.workunit.client.0.smithi123.stdout:7/994: fsync d2/fbd 0 2024-03-30T04:28:26.766 INFO:tasks.workunit.client.1.smithi135.stdout:7/642: getdents d4/db/de/db0/db4 0 2024-03-30T04:28:26.769 INFO:tasks.workunit.client.0.smithi123.stdout:7/995: dread - d2/da/d11/d61/d55/fd4 zero size 2024-03-30T04:28:26.772 INFO:tasks.workunit.client.1.smithi135.stdout:0/658: creat d0/d19/fd3 x:0 0 0 2024-03-30T04:28:26.777 INFO:tasks.workunit.client.0.smithi123.stdout:7/996: link d2/dc/d7f/d10c/f13f d2/da/d11/d61/d55/d59/f14b 0 2024-03-30T04:28:26.778 INFO:tasks.workunit.client.0.smithi123.stdout:5/970: dread dd/d19/f87 [0,4194304] 0 2024-03-30T04:28:26.779 INFO:tasks.workunit.client.1.smithi135.stdout:5/531: truncate d1/d7/d11/f8a 618393 0 2024-03-30T04:28:26.781 INFO:tasks.workunit.client.1.smithi135.stdout:6/639: truncate d2/dd/d14/d2b/d86/d55/f74 537859 0 2024-03-30T04:28:26.782 INFO:tasks.workunit.client.1.smithi135.stdout:7/643: read d4/db/d1a/d5c/d2c/d51/d56/fa4 [444869,123258] 0 2024-03-30T04:28:26.783 INFO:tasks.workunit.client.0.smithi123.stdout:5/971: rename dd/d19/d1f/d3c/d9e/df5/c105 to dd/d19/d1f/d35/d3a/d57/c144 0 2024-03-30T04:28:26.784 INFO:tasks.workunit.client.1.smithi135.stdout:0/659: mknod d0/d13/d1e/d3a/dc2/dd1/cd4 0 2024-03-30T04:28:26.785 INFO:tasks.workunit.client.0.smithi123.stdout:5/972: stat dd/d41/f7f 0 2024-03-30T04:28:26.788 INFO:tasks.workunit.client.1.smithi135.stdout:1/558: dwrite d16/d5a/d5f/d3a/fb1 [0,4194304] 0 2024-03-30T04:28:26.788 INFO:tasks.workunit.client.1.smithi135.stdout:0/660: write d0/d13/d53/f80 [4648389,7763] 0 2024-03-30T04:28:26.789 INFO:tasks.workunit.client.0.smithi123.stdout:5/973: dread dd/f74 [0,4194304] 0 2024-03-30T04:28:26.792 INFO:tasks.workunit.client.0.smithi123.stdout:5/974: symlink dd/d19/d1f/d3c/d9e/d117/l145 0 2024-03-30T04:28:26.794 INFO:tasks.workunit.client.1.smithi135.stdout:1/559: chown d16/d29/d32/d85/lb8 999 1 2024-03-30T04:28:26.797 INFO:tasks.workunit.client.1.smithi135.stdout:5/532: rename d1/d7/d11/c58 to d1/d3/d76/d92/c9d 0 2024-03-30T04:28:26.798 INFO:tasks.workunit.client.1.smithi135.stdout:7/644: getdents d4/db/d1a/d5c/d82 0 2024-03-30T04:28:26.799 INFO:tasks.workunit.client.1.smithi135.stdout:5/533: chown d1/f4f 6221 1 2024-03-30T04:28:26.799 INFO:tasks.workunit.client.1.smithi135.stdout:0/661: mkdir d0/d1/d16/dd5 0 2024-03-30T04:28:26.805 INFO:tasks.workunit.client.1.smithi135.stdout:7/645: write d4/db/d1a/d5c/d5d/db5/fd2 [22859,92081] 0 2024-03-30T04:28:26.807 INFO:tasks.workunit.client.0.smithi123.stdout:7/997: write d2/da/d11/d34/d85/d8e/f129 [858877,12387] 0 2024-03-30T04:28:26.809 INFO:tasks.workunit.client.1.smithi135.stdout:8/520: sync 2024-03-30T04:28:26.809 INFO:tasks.workunit.client.1.smithi135.stdout:9/542: sync 2024-03-30T04:28:26.810 INFO:tasks.workunit.client.0.smithi123.stdout:7/998: fdatasync d2/dc/d49/dbc/fc3 0 2024-03-30T04:28:26.811 INFO:tasks.workunit.client.1.smithi135.stdout:5/534: rename d1/f36 to d1/d45/d52/f9e 0 2024-03-30T04:28:26.811 INFO:tasks.workunit.client.0.smithi123.stdout:5/975: dwrite dd/d19/d1f/d35/d3a/d57/db5/f112 [0,4194304] 0 2024-03-30T04:28:26.812 INFO:tasks.workunit.client.0.smithi123.stdout:7/999: fsync d2/da/d11/d34/d85/fae 0 2024-03-30T04:28:26.814 INFO:tasks.workunit.client.1.smithi135.stdout:8/521: write d6/d8/d14/f30 [434933,36520] 0 2024-03-30T04:28:26.815 INFO:tasks.workunit.client.0.smithi123.stdout:5/976: dread - dd/d19/d1f/d68/f6b zero size 2024-03-30T04:28:26.817 INFO:tasks.workunit.client.0.smithi123.stdout:5/977: fdatasync dd/d19/d3e/d5b/d62/f135 0 2024-03-30T04:28:26.818 INFO:tasks.workunit.client.1.smithi135.stdout:5/535: write d1/d7/ff [10971787,44751] 0 2024-03-30T04:28:26.819 INFO:tasks.workunit.client.1.smithi135.stdout:8/522: chown d6/d8/d15/d1f 8 1 2024-03-30T04:28:26.820 INFO:tasks.workunit.client.1.smithi135.stdout:5/536: readlink d1/le 0 2024-03-30T04:28:26.824 INFO:tasks.workunit.client.1.smithi135.stdout:8/523: stat d6/d8/d15/d54/d8e/d72/d8b 0 2024-03-30T04:28:26.825 INFO:tasks.workunit.client.1.smithi135.stdout:1/560: dwrite d16/d5a/d5f/d3a/d6f/fa8 [0,4194304] 0 2024-03-30T04:28:26.826 INFO:tasks.workunit.client.1.smithi135.stdout:9/543: rename d7/d4f/d5c/d63/d86 to d7/d4f/d6e/d98/db1/dbc 0 2024-03-30T04:28:26.839 INFO:tasks.workunit.client.0.smithi123.stdout:5/978: dwrite dd/d19/d1f/d3c/d72/d8c/f9a [0,4194304] 0 2024-03-30T04:28:26.840 INFO:tasks.workunit.client.1.smithi135.stdout:9/544: creat d7/d29/d78/da4/fbd x:0 0 0 2024-03-30T04:28:26.841 INFO:tasks.workunit.client.1.smithi135.stdout:0/662: dwrite d0/d5/d10/f2c [0,4194304] 0 2024-03-30T04:28:26.843 INFO:tasks.workunit.client.0.smithi123.stdout:5/979: write dd/d19/d1f/d3c/d72/d8c/df4/f108 [857868,40890] 0 2024-03-30T04:28:26.848 INFO:tasks.workunit.client.0.smithi123.stdout:5/980: truncate dd/d19/d3e/d5b/f8a 2368070 0 2024-03-30T04:28:26.849 INFO:tasks.workunit.client.1.smithi135.stdout:7/646: dwrite d4/db/d1a/d5c/d2c/d51/d56/d81/f7f [0,4194304] 0 2024-03-30T04:28:26.850 INFO:tasks.workunit.client.1.smithi135.stdout:8/524: rename d6/d8/d15/d54/d7e/f92 to d6/d8/d15/d54/d8e/d72/d73/f9b 0 2024-03-30T04:28:26.857 INFO:tasks.workunit.client.1.smithi135.stdout:0/663: write d0/d13/d1e/fb2 [4623187,33317] 0 2024-03-30T04:28:26.861 INFO:tasks.workunit.client.1.smithi135.stdout:1/561: link d16/d5a/d5f/d3a/l40 d16/d19/da4/d26/lbe 0 2024-03-30T04:28:26.863 INFO:tasks.workunit.client.1.smithi135.stdout:6/640: dwrite d2/dd/d18/f24 [4194304,4194304] 0 2024-03-30T04:28:26.870 INFO:tasks.workunit.client.1.smithi135.stdout:9/545: dread d7/d90/fa0 [0,4194304] 0 2024-03-30T04:28:26.870 INFO:tasks.workunit.client.1.smithi135.stdout:4/564: sync 2024-03-30T04:28:26.870 INFO:tasks.workunit.client.1.smithi135.stdout:8/525: truncate d6/d23/d7b/f98 543141 0 2024-03-30T04:28:26.870 INFO:tasks.workunit.client.1.smithi135.stdout:3/553: sync 2024-03-30T04:28:26.871 INFO:tasks.workunit.client.1.smithi135.stdout:7/647: dread d4/db/f14 [0,4194304] 0 2024-03-30T04:28:26.873 INFO:tasks.workunit.client.1.smithi135.stdout:0/664: write d0/d13/d53/f97 [8907215,126630] 0 2024-03-30T04:28:26.877 INFO:tasks.workunit.client.1.smithi135.stdout:9/546: unlink d7/d29/d78/da4/ld 0 2024-03-30T04:28:26.882 INFO:tasks.workunit.client.1.smithi135.stdout:6/641: symlink d2/dd/d18/db0/lcc 0 2024-03-30T04:28:26.887 INFO:tasks.workunit.client.1.smithi135.stdout:0/665: write d0/d19/f5c [2793187,121603] 0 2024-03-30T04:28:26.887 INFO:tasks.workunit.client.1.smithi135.stdout:9/547: fsync d7/d41/d72/fa1 0 2024-03-30T04:28:26.888 INFO:tasks.workunit.client.1.smithi135.stdout:6/642: write d2/f12 [4284526,95433] 0 2024-03-30T04:28:26.889 INFO:tasks.workunit.client.1.smithi135.stdout:3/554: truncate d3/d50/d46/f21 1967959 0 2024-03-30T04:28:26.892 INFO:tasks.workunit.client.1.smithi135.stdout:8/526: creat d6/d8/d15/d79/f9c x:0 0 0 2024-03-30T04:28:26.893 INFO:tasks.workunit.client.0.smithi123.stdout:5/981: dwrite dd/f16 [4194304,4194304] 0 2024-03-30T04:28:26.893 INFO:tasks.workunit.client.1.smithi135.stdout:4/565: mkdir d10/d1a/d32/d40/d92/db9/dc6 0 2024-03-30T04:28:26.897 INFO:tasks.workunit.client.1.smithi135.stdout:6/643: chown d2/f9 199 1 2024-03-30T04:28:26.899 INFO:tasks.workunit.client.1.smithi135.stdout:9/548: unlink d7/d29/ca2 0 2024-03-30T04:28:26.899 INFO:tasks.workunit.client.1.smithi135.stdout:1/562: write d16/f1c [504657,36911] 0 2024-03-30T04:28:26.902 INFO:tasks.workunit.client.1.smithi135.stdout:2/485: sync 2024-03-30T04:28:26.902 INFO:tasks.workunit.client.1.smithi135.stdout:6/644: readlink d2/dd/lc4 0 2024-03-30T04:28:26.904 INFO:tasks.workunit.client.0.smithi123.stdout:5/982: dread dd/d19/d1f/d3c/fcd [0,4194304] 0 2024-03-30T04:28:26.908 INFO:tasks.workunit.client.1.smithi135.stdout:4/566: mkdir d10/d1a/d42/d66/dc7 0 2024-03-30T04:28:26.910 INFO:tasks.workunit.client.0.smithi123.stdout:5/983: creat dd/d19/f146 x:0 0 0 2024-03-30T04:28:26.911 INFO:tasks.workunit.client.1.smithi135.stdout:5/537: dread d1/d7/d1a/d94/f9c [0,4194304] 0 2024-03-30T04:28:26.913 INFO:tasks.workunit.client.1.smithi135.stdout:8/527: write d6/d23/d7b/f98 [1266105,45600] 0 2024-03-30T04:28:26.915 INFO:tasks.workunit.client.1.smithi135.stdout:8/528: dread - d6/f8c zero size 2024-03-30T04:28:26.916 INFO:tasks.workunit.client.0.smithi123.stdout:5/984: unlink dd/d19/d3e/d5b/d62/le4 0 2024-03-30T04:28:26.917 INFO:tasks.workunit.client.1.smithi135.stdout:3/555: mkdir d3/d10/d16/d23/db6/dc1 0 2024-03-30T04:28:26.919 INFO:tasks.workunit.client.1.smithi135.stdout:9/549: write d7/d29/d78/da4/f25 [817983,39311] 0 2024-03-30T04:28:26.920 INFO:tasks.workunit.client.0.smithi123.stdout:5/985: stat dd/d19/d1f/d35/d3a/f45 0 2024-03-30T04:28:26.922 INFO:tasks.workunit.client.0.smithi123.stdout:5/986: fsync dd/d41/f90 0 2024-03-30T04:28:26.925 INFO:tasks.workunit.client.0.smithi123.stdout:5/987: truncate dd/d19/d1f/d3c/f139 2477 0 2024-03-30T04:28:26.927 INFO:tasks.workunit.client.0.smithi123.stdout:5/988: mkdir dd/d19/d5f/dc2/d147 0 2024-03-30T04:28:26.927 INFO:tasks.workunit.client.1.smithi135.stdout:4/567: rename d10/d1a/l81 to d10/d72/lc8 0 2024-03-30T04:28:26.931 INFO:tasks.workunit.client.0.smithi123.stdout:5/989: getdents dd 0 2024-03-30T04:28:26.933 INFO:tasks.workunit.client.0.smithi123.stdout:5/990: read dd/d19/d3e/d5b/f8d [485489,66267] 0 2024-03-30T04:28:26.934 INFO:tasks.workunit.client.1.smithi135.stdout:3/556: fdatasync d3/d50/d22/d33/d7b/fb2 0 2024-03-30T04:28:26.935 INFO:tasks.workunit.client.1.smithi135.stdout:7/648: dread d4/f8 [0,4194304] 0 2024-03-30T04:28:26.935 INFO:tasks.workunit.client.1.smithi135.stdout:0/666: dwrite d0/d19/f5c [0,4194304] 0 2024-03-30T04:28:26.941 INFO:tasks.workunit.client.1.smithi135.stdout:3/557: truncate d3/fb3 141613 0 2024-03-30T04:28:26.942 INFO:tasks.workunit.client.0.smithi123.stdout:5/991: write dd/d19/d1f/d35/d3a/d57/f65 [150512,17759] 0 2024-03-30T04:28:26.942 INFO:tasks.workunit.client.1.smithi135.stdout:2/486: rename d4/d6/d47/l9a to d4/d32/d12/d96/la4 0 2024-03-30T04:28:26.943 INFO:tasks.workunit.client.1.smithi135.stdout:7/649: fdatasync d4/db/de/db0/fc1 0 2024-03-30T04:28:26.945 INFO:tasks.workunit.client.1.smithi135.stdout:2/487: chown d4/d6/d99 488814302 1 2024-03-30T04:28:26.948 INFO:tasks.workunit.client.0.smithi123.stdout:5/992: rename dd/d19/d1f/d3c/d9e/d117/l145 to dd/d19/d1f/d3c/d72/d8c/df4/l148 0 2024-03-30T04:28:26.949 INFO:tasks.workunit.client.1.smithi135.stdout:6/645: rename d2/dd/d40/d52/d95 to d2/dd/d14/d30/d43/dcd 0 2024-03-30T04:28:26.951 INFO:tasks.workunit.client.1.smithi135.stdout:1/563: dwrite d16/d5a/d5f/d3a/d6f/f7c [4194304,4194304] 0 2024-03-30T04:28:26.951 INFO:tasks.workunit.client.0.smithi123.stdout:5/993: unlink dd/f123 0 2024-03-30T04:28:26.952 INFO:tasks.workunit.client.1.smithi135.stdout:8/529: rmdir d6/d8/d8f 0 2024-03-30T04:28:26.953 INFO:tasks.workunit.client.1.smithi135.stdout:2/488: dread - d4/da/d7a/f88 zero size 2024-03-30T04:28:26.961 INFO:tasks.workunit.client.1.smithi135.stdout:4/568: mknod d10/d4d/d89/d9f/daa/cc9 0 2024-03-30T04:28:26.964 INFO:tasks.workunit.client.1.smithi135.stdout:5/538: rename d1/d7/d1a/f68 to d1/da/f9f 0 2024-03-30T04:28:26.965 INFO:tasks.workunit.client.1.smithi135.stdout:1/564: write d16/d5a/d5f/d3a/fb1 [527067,89266] 0 2024-03-30T04:28:26.965 INFO:tasks.workunit.client.1.smithi135.stdout:2/489: chown d4/d32/d12/f3f 1962 1 2024-03-30T04:28:26.966 INFO:tasks.workunit.client.1.smithi135.stdout:9/550: link d7/d29/d78/da4/l19 d7/d4f/d6e/dad/lbe 0 2024-03-30T04:28:26.969 INFO:tasks.workunit.client.0.smithi123.stdout:5/994: creat dd/d19/d91/f149 x:0 0 0 2024-03-30T04:28:26.971 INFO:tasks.workunit.client.1.smithi135.stdout:8/530: rmdir d6/d8/d15/d54/d8e/d72/d73 39 2024-03-30T04:28:26.972 INFO:tasks.workunit.client.1.smithi135.stdout:7/650: creat d4/db/d79/d92/dcf/fd7 x:0 0 0 2024-03-30T04:28:26.975 INFO:tasks.workunit.client.1.smithi135.stdout:0/667: write d0/d1/f29 [8761523,102245] 0 2024-03-30T04:28:26.982 INFO:tasks.workunit.client.0.smithi123.stdout:5/995: write dd/d19/d1f/d35/d3a/d57/db3/fde [750804,63497] 0 2024-03-30T04:28:26.983 INFO:tasks.workunit.client.1.smithi135.stdout:3/558: dwrite d3/d50/d46/f21 [0,4194304] 0 2024-03-30T04:28:26.983 INFO:tasks.workunit.client.1.smithi135.stdout:8/531: mkdir d6/d23/d9d 0 2024-03-30T04:28:26.986 INFO:tasks.workunit.client.1.smithi135.stdout:4/569: creat d10/d1a/d42/d66/dc2/fca x:0 0 0 2024-03-30T04:28:26.986 INFO:tasks.workunit.client.1.smithi135.stdout:0/668: fdatasync d0/d5/d10/f31 0 2024-03-30T04:28:26.989 INFO:tasks.workunit.client.1.smithi135.stdout:7/651: chown d4/db/de/l3e 78 1 2024-03-30T04:28:26.990 INFO:tasks.workunit.client.1.smithi135.stdout:9/551: fdatasync d7/d4f/d89/f8a 0 2024-03-30T04:28:26.990 INFO:tasks.workunit.client.1.smithi135.stdout:3/559: write d3/db/fb4 [985457,47794] 0 2024-03-30T04:28:26.996 INFO:tasks.workunit.client.1.smithi135.stdout:9/552: readlink d7/d55/l7b 0 2024-03-30T04:28:26.998 INFO:tasks.workunit.client.0.smithi123.stdout:5/996: dwrite dd/d19/d3e/f63 [4194304,4194304] 0 2024-03-30T04:28:27.000 INFO:tasks.workunit.client.1.smithi135.stdout:1/565: link d16/d19/da4/d99/lab d16/d29/d32/lbf 0 2024-03-30T04:28:27.000 INFO:tasks.workunit.client.1.smithi135.stdout:5/539: truncate d1/d3/d2b/f2f 3437708 0 2024-03-30T04:28:27.004 INFO:tasks.workunit.client.1.smithi135.stdout:2/490: dwrite d4/d32/d2e/f40 [0,4194304] 0 2024-03-30T04:28:27.005 INFO:tasks.workunit.client.1.smithi135.stdout:7/652: chown d4/db/d1a/d5c/d2c/d2e/db9/faa 4332 1 2024-03-30T04:28:27.009 INFO:tasks.workunit.client.1.smithi135.stdout:7/653: readlink d4/db/de/db0/dc4/lca 0 2024-03-30T04:28:27.012 INFO:tasks.workunit.client.1.smithi135.stdout:8/532: dread - d6/d8/d15/d54/d8e/d72/d73/f8d zero size 2024-03-30T04:28:27.012 INFO:tasks.workunit.client.1.smithi135.stdout:3/560: mkdir d3/d50/d22/d33/d7b/dc2 0 2024-03-30T04:28:27.012 INFO:tasks.workunit.client.1.smithi135.stdout:6/646: dwrite d2/dd/d14/d2b/d86/d55/f74 [0,4194304] 0 2024-03-30T04:28:27.013 INFO:tasks.workunit.client.1.smithi135.stdout:9/553: unlink d7/d4f/d5c/c68 0 2024-03-30T04:28:27.015 INFO:tasks.workunit.client.1.smithi135.stdout:1/566: creat d16/d29/d2c/d3c/fc0 x:0 0 0 2024-03-30T04:28:27.020 INFO:tasks.workunit.client.0.smithi123.stdout:5/997: dwrite dd/d19/d1f/d68/fc6 [0,4194304] 0 2024-03-30T04:28:27.021 INFO:tasks.workunit.client.1.smithi135.stdout:6/647: stat d2/c4f 0 2024-03-30T04:28:27.024 INFO:tasks.workunit.client.0.smithi123.stdout:5/998: mknod dd/d19/d1f/d3c/d72/d8c/df4/c14a 0 2024-03-30T04:28:27.026 INFO:tasks.workunit.client.0.smithi123.stdout:5/999: rmdir dd/d19/d5f/d99/da8 39 2024-03-30T04:28:27.028 INFO:tasks.workunit.client.1.smithi135.stdout:7/654: chown d4/db/d1a/d5c/d2c/d2e/db9/d75 1663 1 2024-03-30T04:28:27.030 INFO:tasks.workunit.client.1.smithi135.stdout:9/554: mknod d7/d4f/d6e/d98/cbf 0 2024-03-30T04:28:27.032 INFO:tasks.workunit.client.1.smithi135.stdout:1/567: fdatasync d16/d5a/dad/d77/f7a 0 2024-03-30T04:28:27.035 INFO:tasks.workunit.client.0.smithi123.stderr:+ rm -rf -- ./tmp.V4EPUBEJVW 2024-03-30T04:28:27.046 INFO:tasks.workunit.client.1.smithi135.stdout:4/570: dwrite d10/d1a/d32/d40/d92/f94 [0,4194304] 0 2024-03-30T04:28:27.046 INFO:tasks.workunit.client.1.smithi135.stdout:9/555: readlink d7/d4f/d5c/d63/l65 0 2024-03-30T04:28:27.051 INFO:tasks.workunit.client.1.smithi135.stdout:3/561: dread d3/db/f19 [0,4194304] 0 2024-03-30T04:28:27.052 INFO:tasks.workunit.client.1.smithi135.stdout:7/655: mknod d4/db/d1a/d5c/d2c/cd8 0 2024-03-30T04:28:27.052 INFO:tasks.workunit.client.1.smithi135.stdout:5/540: dwrite d1/d45/d52/f9e [0,4194304] 0 2024-03-30T04:28:27.056 INFO:tasks.workunit.client.1.smithi135.stdout:3/562: truncate d3/f51 805704 0 2024-03-30T04:28:27.058 INFO:tasks.workunit.client.1.smithi135.stdout:1/568: symlink d16/d19/da4/lc1 0 2024-03-30T04:28:27.065 INFO:tasks.workunit.client.1.smithi135.stdout:9/556: fsync d7/d29/d78/da4/f3d 0 2024-03-30T04:28:27.066 INFO:tasks.workunit.client.1.smithi135.stdout:1/569: chown d16/d19/c6a 6 1 2024-03-30T04:28:27.067 INFO:tasks.workunit.client.1.smithi135.stdout:4/571: mkdir d10/d1a/d3d/dcb 0 2024-03-30T04:28:27.070 INFO:tasks.workunit.client.1.smithi135.stdout:3/563: mknod d3/d50/d22/d8b/d91/cc3 0 2024-03-30T04:28:27.072 INFO:tasks.workunit.client.1.smithi135.stdout:4/572: chown d10/f74 43 1 2024-03-30T04:28:27.074 INFO:tasks.workunit.client.1.smithi135.stdout:3/564: readlink d3/d6e/l70 0 2024-03-30T04:28:27.083 INFO:tasks.workunit.client.1.smithi135.stdout:0/669: dread d0/d1/d16/f4b [0,4194304] 0 2024-03-30T04:28:27.084 INFO:tasks.workunit.client.1.smithi135.stdout:7/656: creat d4/db/d1a/d5c/d2c/d51/d56/dbd/fd9 x:0 0 0 2024-03-30T04:28:27.084 INFO:tasks.workunit.client.1.smithi135.stdout:6/648: dwrite d2/dd/d14/d38/f7e [0,4194304] 0 2024-03-30T04:28:27.086 INFO:tasks.workunit.client.1.smithi135.stdout:3/565: rmdir d3/d50/d22/d33/d7b 39 2024-03-30T04:28:27.088 INFO:tasks.workunit.client.1.smithi135.stdout:1/570: chown d16/l48 75 1 2024-03-30T04:28:27.088 INFO:tasks.workunit.client.1.smithi135.stdout:8/533: dwrite d6/d8/f1d [0,4194304] 0 2024-03-30T04:28:27.088 INFO:tasks.workunit.client.1.smithi135.stdout:4/573: creat d10/fcc x:0 0 0 2024-03-30T04:28:27.089 INFO:tasks.workunit.client.1.smithi135.stdout:5/541: dread d1/d7/d11/f30 [0,4194304] 0 2024-03-30T04:28:27.100 INFO:tasks.workunit.client.1.smithi135.stdout:4/574: write d10/d1a/f23 [1609462,69978] 0 2024-03-30T04:28:27.103 INFO:tasks.workunit.client.1.smithi135.stdout:7/657: symlink d4/db/d1a/d4b/lda 0 2024-03-30T04:28:27.109 INFO:tasks.workunit.client.1.smithi135.stdout:1/571: rename d16/d29/d32/f46 to d16/d5a/dad/fc2 0 2024-03-30T04:28:27.113 INFO:tasks.workunit.client.1.smithi135.stdout:0/670: creat d0/d5/d9/d23/d58/d63/d77/d8b/dcc/fd6 x:0 0 0 2024-03-30T04:28:27.115 INFO:tasks.workunit.client.1.smithi135.stdout:6/649: write d2/dd/d14/d2b/d86/d55/f74 [243885,129966] 0 2024-03-30T04:28:27.116 INFO:tasks.workunit.client.1.smithi135.stdout:3/566: rmdir d3/d6e/d9d 0 2024-03-30T04:28:27.118 INFO:tasks.workunit.client.1.smithi135.stdout:7/658: fdatasync d4/db/d1a/d5c/d2c/d51/da8/fc2 0 2024-03-30T04:28:27.120 INFO:tasks.workunit.client.1.smithi135.stdout:1/572: stat d16/d29/d2c/d3c/fc0 0 2024-03-30T04:28:27.120 INFO:tasks.workunit.client.1.smithi135.stdout:9/557: dwrite d7/d29/d78/da4/f25 [4194304,4194304] 0 2024-03-30T04:28:27.124 INFO:tasks.workunit.client.1.smithi135.stdout:9/558: chown d7/caa 6302397 1 2024-03-30T04:28:27.124 INFO:tasks.workunit.client.1.smithi135.stdout:7/659: chown d4/db/d1a/d4b/d40 306 1 2024-03-30T04:28:27.125 INFO:tasks.workunit.client.1.smithi135.stdout:4/575: creat d10/d1a/d3d/dcb/fcd x:0 0 0 2024-03-30T04:28:27.127 INFO:tasks.workunit.client.1.smithi135.stdout:3/567: unlink d3/d50/f1f 0 2024-03-30T04:28:27.130 INFO:tasks.workunit.client.1.smithi135.stdout:8/534: rename d6/d8/d15/d54/d8e/d72/d73/f8d to d6/d8/f9e 0 2024-03-30T04:28:27.135 INFO:tasks.workunit.client.1.smithi135.stdout:5/542: dread d1/d45/d52/f9e [0,4194304] 0 2024-03-30T04:28:27.135 INFO:tasks.workunit.client.1.smithi135.stdout:3/568: symlink d3/d10/lc4 0 2024-03-30T04:28:27.136 INFO:tasks.workunit.client.1.smithi135.stdout:8/535: fsync d6/d8/d15/d2a/f37 0 2024-03-30T04:28:27.141 INFO:tasks.workunit.client.1.smithi135.stdout:6/650: mknod d2/dd/d14/d30/d43/dcd/dc5/cce 0 2024-03-30T04:28:27.146 INFO:tasks.workunit.client.1.smithi135.stdout:4/576: write d10/d1a/f21 [1812885,22587] 0 2024-03-30T04:28:27.148 INFO:tasks.workunit.client.1.smithi135.stdout:5/543: unlink d1/c28 0 2024-03-30T04:28:27.153 INFO:tasks.workunit.client.1.smithi135.stdout:1/573: write d16/d29/d32/d85/fba [162602,20586] 0 2024-03-30T04:28:27.156 INFO:tasks.workunit.client.1.smithi135.stdout:8/536: mkdir d6/d8/d15/d54/d8e/d72/d73/d9f 0 2024-03-30T04:28:27.161 INFO:tasks.workunit.client.1.smithi135.stdout:6/651: symlink d2/dd/d18/d87/lcf 0 2024-03-30T04:28:27.162 INFO:tasks.workunit.client.1.smithi135.stdout:1/574: fdatasync d16/f42 0 2024-03-30T04:28:27.162 INFO:tasks.workunit.client.1.smithi135.stdout:3/569: write d3/d50/d22/d8b/d91/f97 [469361,98967] 0 2024-03-30T04:28:27.165 INFO:tasks.workunit.client.1.smithi135.stdout:5/544: chown d1/d7/c3f 703297914 1 2024-03-30T04:28:27.166 INFO:tasks.workunit.client.1.smithi135.stdout:0/671: dwrite d0/d1/d16/f1b [0,4194304] 0 2024-03-30T04:28:27.167 INFO:tasks.workunit.client.1.smithi135.stdout:1/575: chown d16 261158 1 2024-03-30T04:28:27.174 INFO:tasks.workunit.client.1.smithi135.stdout:8/537: write d6/d8/f16 [5900828,63982] 0 2024-03-30T04:28:27.174 INFO:tasks.workunit.client.1.smithi135.stdout:6/652: symlink d2/dd/d14/d2b/d86/d8e/ld0 0 2024-03-30T04:28:27.177 INFO:tasks.workunit.client.1.smithi135.stdout:0/672: write d0/d13/d1e/f46 [1291218,21917] 0 2024-03-30T04:28:27.181 INFO:tasks.workunit.client.1.smithi135.stdout:7/660: dwrite d4/db/d1a/d3d/fd4 [0,4194304] 0 2024-03-30T04:28:27.181 INFO:tasks.workunit.client.1.smithi135.stdout:3/570: read d3/f45 [836584,105453] 0 2024-03-30T04:28:27.186 INFO:tasks.workunit.client.1.smithi135.stdout:9/559: dread d7/d41/f5f [0,4194304] 0 2024-03-30T04:28:27.187 INFO:tasks.workunit.client.1.smithi135.stdout:0/673: fsync d0/d19/fd3 0 2024-03-30T04:28:27.187 INFO:tasks.workunit.client.1.smithi135.stdout:6/653: write d2/d68/f83 [1590977,19627] 0 2024-03-30T04:28:27.188 INFO:tasks.workunit.client.1.smithi135.stdout:4/577: dwrite d10/d1a/fba [0,4194304] 0 2024-03-30T04:28:27.192 INFO:tasks.workunit.client.1.smithi135.stdout:8/538: mkdir d6/d8/d14/da0 0 2024-03-30T04:28:27.192 INFO:tasks.workunit.client.1.smithi135.stdout:5/545: dread d1/d7/d11/f8a [0,4194304] 0 2024-03-30T04:28:27.192 INFO:tasks.workunit.client.1.smithi135.stdout:7/661: fdatasync d4/db/de/d5a/f6c 0 2024-03-30T04:28:27.199 INFO:tasks.workunit.client.1.smithi135.stdout:4/578: fsync d10/d4d/f97 0 2024-03-30T04:28:27.200 INFO:tasks.workunit.client.1.smithi135.stdout:8/539: readlink d6/d8/d15/l3d 0 2024-03-30T04:28:27.201 INFO:tasks.workunit.client.1.smithi135.stdout:5/546: dread - d1/da/f84 zero size 2024-03-30T04:28:27.206 INFO:tasks.workunit.client.1.smithi135.stdout:3/571: rename d3/d54/l67 to d3/d54/d5c/lc5 0 2024-03-30T04:28:27.206 INFO:tasks.workunit.client.1.smithi135.stdout:1/576: link d16/d29/l2d d16/d5a/d5f/d3a/d6f/lc3 0 2024-03-30T04:28:27.209 INFO:tasks.workunit.client.1.smithi135.stdout:4/579: unlink fb 0 2024-03-30T04:28:27.210 INFO:tasks.workunit.client.1.smithi135.stdout:8/540: mknod d6/d8/d15/d54/d7e/ca1 0 2024-03-30T04:28:27.213 INFO:tasks.workunit.client.1.smithi135.stdout:5/547: rename d1/d7/d1a/d94 to d1/d3/d76/d92/da0 0 2024-03-30T04:28:27.216 INFO:tasks.workunit.client.1.smithi135.stdout:1/577: write d16/d5a/d5f/d3a/fb1 [4456579,20627] 0 2024-03-30T04:28:27.218 INFO:tasks.workunit.client.1.smithi135.stdout:0/674: write d0/d1/fab [862754,41911] 0 2024-03-30T04:28:27.220 INFO:tasks.workunit.client.1.smithi135.stdout:5/548: write d1/d3/d80/f93 [367008,7853] 0 2024-03-30T04:28:27.220 INFO:tasks.workunit.client.1.smithi135.stdout:9/560: dwrite d7/d41/d72/f74 [0,4194304] 0 2024-03-30T04:28:27.225 INFO:tasks.workunit.client.1.smithi135.stdout:8/541: truncate d6/d23/d7b/f98 1965711 0 2024-03-30T04:28:27.227 INFO:tasks.workunit.client.1.smithi135.stdout:9/561: chown d7/d29/d78/da4/c57 110521440 1 2024-03-30T04:28:27.229 INFO:tasks.workunit.client.1.smithi135.stdout:3/572: chown d3/d54/dab 1970285036 1 2024-03-30T04:28:27.229 INFO:tasks.workunit.client.1.smithi135.stdout:3/573: readlink d3/l49 0 2024-03-30T04:28:27.230 INFO:tasks.workunit.client.1.smithi135.stdout:0/675: mkdir d0/d1/dd7 0 2024-03-30T04:28:27.230 INFO:tasks.workunit.client.1.smithi135.stdout:1/578: symlink d16/d29/d2c/d3c/lc4 0 2024-03-30T04:28:27.231 INFO:tasks.workunit.client.1.smithi135.stdout:9/562: write d7/d29/d78/da4/d37/f38 [1505408,796] 0 2024-03-30T04:28:27.237 INFO:tasks.workunit.client.1.smithi135.stdout:4/580: rename d10/d1a/d42/d66/dc2/fca to d10/d1a/d42/d66/dc7/fce 0 2024-03-30T04:28:27.238 INFO:tasks.workunit.client.1.smithi135.stdout:5/549: chown d1/d3/d2b/f2f 37085885 1 2024-03-30T04:28:27.242 INFO:tasks.workunit.client.1.smithi135.stdout:9/563: mknod d7/d41/db4/cc0 0 2024-03-30T04:28:27.242 INFO:tasks.workunit.client.1.smithi135.stdout:5/550: dread - d1/d45/d52/f90 zero size 2024-03-30T04:28:27.243 INFO:tasks.workunit.client.1.smithi135.stdout:3/574: dread d3/f51 [0,4194304] 0 2024-03-30T04:28:27.243 INFO:tasks.workunit.client.1.smithi135.stdout:0/676: mknod d0/d13/cd8 0 2024-03-30T04:28:27.244 INFO:tasks.workunit.client.1.smithi135.stdout:4/581: dread - d10/d1a/d42/d46/f82 zero size 2024-03-30T04:28:27.244 INFO:tasks.workunit.client.1.smithi135.stdout:9/564: chown d7/l32 1735474 1 2024-03-30T04:28:27.246 INFO:tasks.workunit.client.1.smithi135.stdout:6/654: dread d2/d2e/f31 [0,4194304] 0 2024-03-30T04:28:27.246 INFO:tasks.workunit.client.1.smithi135.stdout:5/551: stat d1/d45/d52 0 2024-03-30T04:28:27.246 INFO:tasks.workunit.client.1.smithi135.stdout:0/677: readlink d0/d13/d1e/d3a/l7b 0 2024-03-30T04:28:27.247 INFO:tasks.workunit.client.1.smithi135.stdout:3/575: write d3/d50/fa [805211,102853] 0 2024-03-30T04:28:27.247 INFO:tasks.workunit.client.1.smithi135.stdout:9/565: chown d7/d55/d8d/f9d 39 1 2024-03-30T04:28:27.249 INFO:tasks.workunit.client.1.smithi135.stdout:8/542: write d6/d8/d15/d54/f5d [1163002,64649] 0 2024-03-30T04:28:27.250 INFO:tasks.workunit.client.1.smithi135.stdout:0/678: stat d0/d1/f5e 0 2024-03-30T04:28:27.254 INFO:tasks.workunit.client.1.smithi135.stdout:4/582: mkdir d10/d1a/d32/d40/d92/db9/dcf 0 2024-03-30T04:28:27.255 INFO:tasks.workunit.client.1.smithi135.stdout:3/576: creat d3/db/fc6 x:0 0 0 2024-03-30T04:28:27.256 INFO:tasks.workunit.client.1.smithi135.stdout:4/583: fsync d10/d4d/d9c/fc5 0 2024-03-30T04:28:27.258 INFO:tasks.workunit.client.1.smithi135.stdout:6/655: creat d2/dd/d14/d2b/d86/fd1 x:0 0 0 2024-03-30T04:28:27.260 INFO:tasks.workunit.client.1.smithi135.stdout:4/584: chown d10/c2a 20371 1 2024-03-30T04:28:27.261 INFO:tasks.workunit.client.1.smithi135.stdout:8/543: symlink d6/d8/d15/la2 0 2024-03-30T04:28:27.262 INFO:tasks.workunit.client.1.smithi135.stdout:3/577: mkdir d3/d10/d16/d23/dc7 0 2024-03-30T04:28:27.269 INFO:tasks.workunit.client.1.smithi135.stdout:0/679: dread d0/d5/d10/f31 [0,4194304] 0 2024-03-30T04:28:27.270 INFO:tasks.workunit.client.1.smithi135.stdout:6/656: rmdir d2/dd/d14/d30/d43 39 2024-03-30T04:28:27.271 INFO:tasks.workunit.client.1.smithi135.stdout:3/578: getdents d3/db/da9 0 2024-03-30T04:28:27.275 INFO:tasks.workunit.client.1.smithi135.stdout:0/680: fsync d0/d13/d1e/f4f 0 2024-03-30T04:28:27.276 INFO:tasks.workunit.client.1.smithi135.stdout:9/566: dwrite d7/d29/d78/da4/f3d [4194304,4194304] 0 2024-03-30T04:28:27.277 INFO:tasks.workunit.client.1.smithi135.stdout:8/544: creat d6/d8/d15/d54/d8e/d72/fa3 x:0 0 0 2024-03-30T04:28:27.280 INFO:tasks.workunit.client.1.smithi135.stdout:4/585: write d10/d1a/d3d/f5a [1108015,23409] 0 2024-03-30T04:28:27.282 INFO:tasks.workunit.client.1.smithi135.stdout:5/552: dwrite d1/f6e [0,4194304] 0 2024-03-30T04:28:27.284 INFO:tasks.workunit.client.1.smithi135.stdout:0/681: dread - d0/d13/d1e/d3a/f72 zero size 2024-03-30T04:28:27.286 INFO:tasks.workunit.client.1.smithi135.stdout:8/545: truncate d6/f8c 579142 0 2024-03-30T04:28:27.288 INFO:tasks.workunit.client.1.smithi135.stdout:4/586: fsync d10/d72/d7c/f83 0 2024-03-30T04:28:27.289 INFO:tasks.workunit.client.1.smithi135.stdout:6/657: truncate d2/dd/d14/d2b/d86/d55/fa4 199459 0 2024-03-30T04:28:27.293 INFO:tasks.workunit.client.1.smithi135.stdout:9/567: chown d7/d4f/d6e/dad/lbe 914 1 2024-03-30T04:28:27.294 INFO:tasks.workunit.client.1.smithi135.stdout:5/553: fdatasync d1/d55/f6c 0 2024-03-30T04:28:27.294 INFO:tasks.workunit.client.1.smithi135.stdout:8/546: chown d6/d8/d15/d54/f5d 9418 1 2024-03-30T04:28:27.297 INFO:tasks.workunit.client.1.smithi135.stdout:4/587: read d10/d1a/f1f [1568053,96807] 0 2024-03-30T04:28:27.301 INFO:tasks.workunit.client.1.smithi135.stdout:8/547: creat d6/d8/d15/d2a/fa4 x:0 0 0 2024-03-30T04:28:27.301 INFO:tasks.workunit.client.1.smithi135.stdout:6/658: mkdir d2/dd/d14/d30/d43/dcd/dc5/dd2 0 2024-03-30T04:28:27.302 INFO:tasks.workunit.client.1.smithi135.stdout:4/588: fsync d10/d1a/f86 0 2024-03-30T04:28:27.302 INFO:tasks.workunit.client.1.smithi135.stdout:0/682: truncate d0/d5/d9/d23/fa4 342198 0 2024-03-30T04:28:27.303 INFO:tasks.workunit.client.1.smithi135.stdout:5/554: unlink d1/d3/d2b/f75 0 2024-03-30T04:28:27.305 INFO:tasks.workunit.client.1.smithi135.stdout:9/568: rename d7/d29/f48 to d7/d4f/d5c/fc1 0 2024-03-30T04:28:27.305 INFO:tasks.workunit.client.1.smithi135.stdout:6/659: truncate d2/dd/d14/f3a 4612175 0 2024-03-30T04:28:27.308 INFO:tasks.workunit.client.1.smithi135.stdout:9/569: stat d7/d90 0 2024-03-30T04:28:27.313 INFO:tasks.workunit.client.1.smithi135.stdout:5/555: unlink d1/d3/d2b/d54/d5e/d5c/d79/c7b 0 2024-03-30T04:28:27.316 INFO:tasks.workunit.client.1.smithi135.stdout:5/556: write d1/d7/d11/f87 [2373698,88495] 0 2024-03-30T04:28:27.319 INFO:tasks.workunit.client.1.smithi135.stdout:3/579: dwrite d3/d50/f36 [0,4194304] 0 2024-03-30T04:28:27.319 INFO:tasks.workunit.client.1.smithi135.stdout:9/570: creat d7/d4f/d89/fc2 x:0 0 0 2024-03-30T04:28:27.320 INFO:tasks.workunit.client.1.smithi135.stdout:8/548: creat d6/d8/fa5 x:0 0 0 2024-03-30T04:28:27.324 INFO:tasks.workunit.client.1.smithi135.stdout:4/589: write d10/d1a/f1f [3575152,30774] 0 2024-03-30T04:28:27.327 INFO:tasks.workunit.client.1.smithi135.stdout:0/683: getdents d0/db5/db8 0 2024-03-30T04:28:27.329 INFO:tasks.workunit.client.1.smithi135.stdout:6/660: dwrite d2/f99 [0,4194304] 0 2024-03-30T04:28:27.331 INFO:tasks.workunit.client.1.smithi135.stdout:6/661: write d2/f9 [9342069,60379] 0 2024-03-30T04:28:27.333 INFO:tasks.workunit.client.1.smithi135.stdout:5/557: creat d1/fa1 x:0 0 0 2024-03-30T04:28:27.338 INFO:tasks.workunit.client.1.smithi135.stdout:2/491: sync 2024-03-30T04:28:27.339 INFO:tasks.workunit.client.1.smithi135.stdout:0/684: rename d0/c93 to d0/d19/d2d/d45/cd9 0 2024-03-30T04:28:27.340 INFO:tasks.workunit.client.1.smithi135.stdout:6/662: read - d2/d2e/f5f zero size 2024-03-30T04:28:27.343 INFO:tasks.workunit.client.1.smithi135.stdout:2/492: symlink d4/d55/la5 0 2024-03-30T04:28:27.346 INFO:tasks.workunit.client.1.smithi135.stdout:8/549: dwrite d6/f8c [0,4194304] 0 2024-03-30T04:28:27.347 INFO:tasks.workunit.client.1.smithi135.stdout:0/685: mknod d0/d1/cda 0 2024-03-30T04:28:27.350 INFO:tasks.workunit.client.1.smithi135.stdout:0/686: chown d0/d5/d9/fb 44 1 2024-03-30T04:28:27.352 INFO:tasks.workunit.client.1.smithi135.stdout:5/558: link d1/d3/d2b/d54/f71 d1/d3/fa2 0 2024-03-30T04:28:27.352 INFO:tasks.workunit.client.1.smithi135.stdout:6/663: rename d2/d36/c5d to d2/dd/d40/db4/cd3 0 2024-03-30T04:28:27.356 INFO:tasks.workunit.client.1.smithi135.stdout:9/571: getdents d7/d90 0 2024-03-30T04:28:27.363 INFO:tasks.workunit.client.1.smithi135.stdout:3/580: write d3/d50/f25 [4147526,51392] 0 2024-03-30T04:28:27.364 INFO:tasks.workunit.client.1.smithi135.stdout:8/550: mkdir d6/d8/d15/d54/d7e/da6 0 2024-03-30T04:28:27.365 INFO:tasks.workunit.client.1.smithi135.stdout:0/687: unlink d0/d5/d42/d9f/cc7 0 2024-03-30T04:28:27.368 INFO:tasks.workunit.client.1.smithi135.stdout:9/572: dread d7/d41/f59 [0,4194304] 0 2024-03-30T04:28:27.370 INFO:tasks.workunit.client.1.smithi135.stdout:9/573: chown d7/d29/l31 222 1 2024-03-30T04:28:27.371 INFO:tasks.workunit.client.1.smithi135.stdout:9/574: fsync d7/d4f/d5c/f84 0 2024-03-30T04:28:27.380 INFO:tasks.workunit.client.1.smithi135.stdout:9/575: dread d7/d29/d78/da4/d37/f3a [0,4194304] 0 2024-03-30T04:28:27.382 INFO:tasks.workunit.client.1.smithi135.stdout:9/576: dread - d7/d29/d78/da4/fbd zero size 2024-03-30T04:28:27.383 INFO:tasks.workunit.client.1.smithi135.stdout:6/664: truncate d2/f4d 2162451 0 2024-03-30T04:28:27.385 INFO:tasks.workunit.client.1.smithi135.stdout:3/581: creat d3/d50/fc8 x:0 0 0 2024-03-30T04:28:27.386 INFO:tasks.workunit.client.1.smithi135.stdout:0/688: symlink d0/d5/d42/d87/ldb 0 2024-03-30T04:28:27.391 INFO:tasks.workunit.client.1.smithi135.stdout:3/582: write d3/d50/d46/f73 [874872,35740] 0 2024-03-30T04:28:27.391 INFO:tasks.workunit.client.1.smithi135.stdout:2/493: link d4/d32/d12/d45/fa0 d4/da/d8d/fa6 0 2024-03-30T04:28:27.391 INFO:tasks.workunit.client.1.smithi135.stdout:5/559: mknod d1/d7/d1a/d1f/d9b/ca3 0 2024-03-30T04:28:27.391 INFO:tasks.workunit.client.1.smithi135.stdout:9/577: rmdir d7/d41/db4 39 2024-03-30T04:28:27.391 INFO:tasks.workunit.client.1.smithi135.stdout:8/551: mkdir d6/d8/d15/d54/d7e/da6/da7 0 2024-03-30T04:28:27.395 INFO:tasks.workunit.client.1.smithi135.stdout:5/560: creat d1/d3/fa4 x:0 0 0 2024-03-30T04:28:27.395 INFO:tasks.workunit.client.1.smithi135.stdout:6/665: unlink d2/la7 0 2024-03-30T04:28:27.396 INFO:tasks.workunit.client.1.smithi135.stdout:0/689: link d0/d1/fab d0/d1/dd7/fdc 0 2024-03-30T04:28:27.398 INFO:tasks.workunit.client.1.smithi135.stdout:9/578: rmdir d7/d29/d78/da4/d37 39 2024-03-30T04:28:27.400 INFO:tasks.workunit.client.1.smithi135.stdout:9/579: fsync d7/d29/f51 0 2024-03-30T04:28:27.402 INFO:tasks.workunit.client.1.smithi135.stdout:6/666: dread d2/dd/d14/f72 [0,4194304] 0 2024-03-30T04:28:27.403 INFO:tasks.workunit.client.1.smithi135.stdout:2/494: dread d4/d32/d2e/d2f/f38 [0,4194304] 0 2024-03-30T04:28:27.404 INFO:tasks.workunit.client.1.smithi135.stdout:9/580: write d7/d29/d78/da4/f16 [3068081,120430] 0 2024-03-30T04:28:27.404 INFO:tasks.workunit.client.1.smithi135.stdout:6/667: chown d2/dd/d14/d2b 49220446 1 2024-03-30T04:28:27.405 INFO:tasks.workunit.client.1.smithi135.stdout:2/495: chown d4/da/d8d 460 1 2024-03-30T04:28:27.407 INFO:tasks.workunit.client.1.smithi135.stdout:5/561: truncate d1/d7/f59 967819 0 2024-03-30T04:28:27.407 INFO:tasks.workunit.client.1.smithi135.stdout:6/668: write d2/d68/fba [1021789,122085] 0 2024-03-30T04:28:27.410 INFO:tasks.workunit.client.1.smithi135.stdout:2/496: truncate d4/da/d8d/fa6 396028 0 2024-03-30T04:28:27.411 INFO:tasks.workunit.client.1.smithi135.stdout:8/552: rename d6/d8/d15/d2a/l74 to d6/d8/d15/d54/d8e/d72/d73/d9f/la8 0 2024-03-30T04:28:27.413 INFO:tasks.workunit.client.1.smithi135.stdout:5/562: fdatasync d1/d3/d2b/d54/d5e/d5c/f65 0 2024-03-30T04:28:27.415 INFO:tasks.workunit.client.1.smithi135.stdout:5/563: chown d1/d3/c35 30 1 2024-03-30T04:28:27.415 INFO:tasks.workunit.client.1.smithi135.stdout:0/690: dwrite d0/d13/f74 [4194304,4194304] 0 2024-03-30T04:28:27.420 INFO:tasks.workunit.client.1.smithi135.stdout:6/669: creat d2/d36/fd4 x:0 0 0 2024-03-30T04:28:27.421 INFO:tasks.workunit.client.1.smithi135.stdout:3/583: getdents d3/d10/d16/d23/d6a/d94 0 2024-03-30T04:28:27.423 INFO:tasks.workunit.client.1.smithi135.stdout:2/497: symlink d4/d32/d43/la7 0 2024-03-30T04:28:27.425 INFO:tasks.workunit.client.1.smithi135.stdout:8/553: mkdir d6/d8/d15/d2a/da9 0 2024-03-30T04:28:27.426 INFO:tasks.workunit.client.1.smithi135.stdout:5/564: fsync d1/d3/fa2 0 2024-03-30T04:28:27.431 INFO:tasks.workunit.client.1.smithi135.stdout:6/670: rename d2/fca to d2/d36/d76/fd5 0 2024-03-30T04:28:27.432 INFO:tasks.workunit.client.1.smithi135.stdout:3/584: mknod d3/d10/d16/d41/d83/cc9 0 2024-03-30T04:28:27.436 INFO:tasks.workunit.client.1.smithi135.stdout:2/498: mknod d4/d32/ca8 0 2024-03-30T04:28:27.439 INFO:tasks.workunit.client.1.smithi135.stdout:6/671: creat d2/dd/d40/fd6 x:0 0 0 2024-03-30T04:28:27.440 INFO:tasks.workunit.client.1.smithi135.stdout:2/499: fsync d4/d6/f61 0 2024-03-30T04:28:27.441 INFO:tasks.workunit.client.1.smithi135.stdout:0/691: dwrite d0/d5/d9/d23/d58/d63/d77/d86/fc9 [4194304,4194304] 0 2024-03-30T04:28:27.441 INFO:tasks.workunit.client.1.smithi135.stdout:6/672: chown d2/d36/l4e 18 1 2024-03-30T04:28:27.443 INFO:tasks.workunit.client.1.smithi135.stdout:3/585: rename d3/f76 to d3/d10/fca 0 2024-03-30T04:28:27.443 INFO:tasks.workunit.client.1.smithi135.stdout:6/673: chown d2/d68/c9c 378087 1 2024-03-30T04:28:27.448 INFO:tasks.workunit.client.1.smithi135.stdout:3/586: dread d3/f51 [0,4194304] 0 2024-03-30T04:28:27.449 INFO:tasks.workunit.client.1.smithi135.stdout:0/692: creat d0/d13/d1e/fdd x:0 0 0 2024-03-30T04:28:27.450 INFO:tasks.workunit.client.1.smithi135.stdout:0/693: chown d0/d1/dd7 648195 1 2024-03-30T04:28:27.451 INFO:tasks.workunit.client.1.smithi135.stdout:6/674: symlink d2/dd/d18/d87/ld7 0 2024-03-30T04:28:27.453 INFO:tasks.workunit.client.1.smithi135.stdout:0/694: chown d0/d1/d16/f32 0 1 2024-03-30T04:28:27.453 INFO:tasks.workunit.client.1.smithi135.stdout:8/554: link d6/f5b d6/d8/d15/d54/d8e/faa 0 2024-03-30T04:28:27.456 INFO:tasks.workunit.client.1.smithi135.stdout:3/587: creat d3/d10/d16/d23/fcb x:0 0 0 2024-03-30T04:28:27.464 INFO:tasks.workunit.client.1.smithi135.stdout:2/500: getdents d4/d55/d77 0 2024-03-30T04:28:27.470 INFO:tasks.workunit.client.1.smithi135.stdout:3/588: mkdir d3/db/da9/dcc 0 2024-03-30T04:28:27.473 INFO:tasks.workunit.client.1.smithi135.stdout:0/695: creat d0/d13/fde x:0 0 0 2024-03-30T04:28:27.485 INFO:tasks.workunit.client.1.smithi135.stdout:9/581: write d7/d41/f59 [324376,13663] 0 2024-03-30T04:28:27.486 INFO:tasks.workunit.client.1.smithi135.stdout:6/675: dread d2/dd/d14/d2b/d86/f3c [0,4194304] 0 2024-03-30T04:28:27.487 INFO:tasks.workunit.client.1.smithi135.stdout:5/565: truncate d1/d7/d1a/d1f/f66 87951 0 2024-03-30T04:28:27.489 INFO:tasks.workunit.client.1.smithi135.stdout:9/582: chown d7/d90/f94 17 1 2024-03-30T04:28:27.496 INFO:tasks.workunit.client.1.smithi135.stdout:8/555: dread d6/d8/d14/f5e [0,4194304] 0 2024-03-30T04:28:27.500 INFO:tasks.workunit.client.1.smithi135.stdout:9/583: chown d7/d29/d78/da4/d37/f3a 3 1 2024-03-30T04:28:27.500 INFO:tasks.workunit.client.1.smithi135.stdout:5/566: getdents d1/da/d17/d70 0 2024-03-30T04:28:27.504 INFO:tasks.workunit.client.1.smithi135.stdout:3/589: dwrite d3/d50/d22/f87 [0,4194304] 0 2024-03-30T04:28:27.509 INFO:tasks.workunit.client.1.smithi135.stdout:2/501: dread d4/d32/d12/d45/f2c [0,4194304] 0 2024-03-30T04:28:27.513 INFO:tasks.workunit.client.1.smithi135.stdout:5/567: read d1/d7/d11/f87 [7025817,5408] 0 2024-03-30T04:28:27.514 INFO:tasks.workunit.client.1.smithi135.stdout:8/556: chown d6/d8/d15/d54/d8e/l85 7902604 1 2024-03-30T04:28:27.518 INFO:tasks.workunit.client.1.smithi135.stdout:2/502: creat d4/d32/d43/fa9 x:0 0 0 2024-03-30T04:28:27.518 INFO:tasks.workunit.client.1.smithi135.stdout:9/584: dread - d7/d29/d78/da4/d37/f39 zero size 2024-03-30T04:28:27.518 INFO:tasks.workunit.client.1.smithi135.stdout:0/696: dwrite d0/f3 [0,4194304] 0 2024-03-30T04:28:27.524 INFO:tasks.workunit.client.1.smithi135.stdout:3/590: creat d3/db/fcd x:0 0 0 2024-03-30T04:28:27.524 INFO:tasks.workunit.client.1.smithi135.stdout:5/568: getdents d1/d55 0 2024-03-30T04:28:27.524 INFO:tasks.workunit.client.1.smithi135.stdout:8/557: creat d6/d8/d14/d32/fab x:0 0 0 2024-03-30T04:28:27.524 INFO:tasks.workunit.client.1.smithi135.stdout:9/585: dread - d7/d4f/d5c/f5e zero size 2024-03-30T04:28:27.524 INFO:tasks.workunit.client.1.smithi135.stdout:0/697: write d0/d13/d1e/fdd [872922,6245] 0 2024-03-30T04:28:27.529 INFO:tasks.workunit.client.1.smithi135.stdout:6/676: truncate d2/dd/d14/d2b/d86/d55/f74 729324 0 2024-03-30T04:28:27.533 INFO:tasks.workunit.client.1.smithi135.stdout:2/503: link d4/d55/d77/l51 d4/d55/d77/laa 0 2024-03-30T04:28:27.534 INFO:tasks.workunit.client.1.smithi135.stdout:9/586: symlink d7/d4f/d6e/d98/db1/lc3 0 2024-03-30T04:28:27.534 INFO:tasks.workunit.client.1.smithi135.stdout:3/591: symlink d3/d10/d16/d23/dc7/lce 0 2024-03-30T04:28:27.536 INFO:tasks.workunit.client.1.smithi135.stdout:8/558: mknod d6/d8/d15/d2a/cac 0 2024-03-30T04:28:27.536 INFO:tasks.workunit.client.1.smithi135.stdout:2/504: chown d4/d32/d43/f6b 105011 1 2024-03-30T04:28:27.540 INFO:tasks.workunit.client.1.smithi135.stdout:3/592: symlink d3/d10/d16/d41/d83/lcf 0 2024-03-30T04:28:27.542 INFO:tasks.workunit.client.1.smithi135.stdout:0/698: link d0/d5/d9/d23/d58/f4a d0/d5/d9/d23/d58/d63/d77/fdf 0 2024-03-30T04:28:27.543 INFO:tasks.workunit.client.1.smithi135.stdout:8/559: mkdir d6/d23/d9d/dad 0 2024-03-30T04:28:27.547 INFO:tasks.workunit.client.1.smithi135.stdout:0/699: creat d0/d19/d2d/fe0 x:0 0 0 2024-03-30T04:28:27.553 INFO:tasks.workunit.client.1.smithi135.stdout:2/505: getdents d4/d32/d2e/d67 0 2024-03-30T04:28:27.555 INFO:tasks.workunit.client.1.smithi135.stdout:3/593: rmdir d3/d50/d22/d33/d7b/dc2 0 2024-03-30T04:28:27.561 INFO:tasks.workunit.client.1.smithi135.stdout:2/506: fsync d4/d32/d12/f2a 0 2024-03-30T04:28:27.563 INFO:tasks.workunit.client.1.smithi135.stdout:3/594: write d3/db/f77 [2603581,26503] 0 2024-03-30T04:28:27.570 INFO:tasks.workunit.client.1.smithi135.stdout:8/560: dwrite d6/d8/f17 [4194304,4194304] 0 2024-03-30T04:28:27.570 INFO:tasks.workunit.client.1.smithi135.stdout:3/595: chown d3/d50/d46/f73 6892 1 2024-03-30T04:28:27.577 INFO:tasks.workunit.client.1.smithi135.stdout:2/507: link d4/d6/d47/f86 d4/da/d8d/fab 0 2024-03-30T04:28:27.577 INFO:tasks.workunit.client.1.smithi135.stdout:9/587: dread d7/f22 [4194304,4194304] 0 2024-03-30T04:28:27.577 INFO:tasks.workunit.client.1.smithi135.stdout:3/596: truncate d3/d50/d46/f57 721534 0 2024-03-30T04:28:27.578 INFO:tasks.workunit.client.1.smithi135.stdout:0/700: dwrite d0/d19/d2d/d45/f76 [0,4194304] 0 2024-03-30T04:28:27.580 INFO:tasks.workunit.client.1.smithi135.stdout:3/597: fsync d3/d50/fa 0 2024-03-30T04:28:27.581 INFO:tasks.workunit.client.1.smithi135.stdout:6/677: link d2/dd/d14/d2b/d86/d55/f74 d2/dd/fd8 0 2024-03-30T04:28:27.584 INFO:tasks.workunit.client.1.smithi135.stdout:6/678: dread - d2/dd/d14/d2b/d86/d55/fb2 zero size 2024-03-30T04:28:27.590 INFO:tasks.workunit.client.1.smithi135.stdout:3/598: getdents d3/d50/d22/d8b/daf 0 2024-03-30T04:28:27.590 INFO:tasks.workunit.client.1.smithi135.stdout:7/662: sync 2024-03-30T04:28:27.595 INFO:tasks.workunit.client.1.smithi135.stdout:0/701: link d0/d13/d1e/c4c d0/d1/d16/d7e/ce1 0 2024-03-30T04:28:27.595 INFO:tasks.workunit.client.1.smithi135.stdout:0/702: chown d0/d5/d42 43160121 1 2024-03-30T04:28:27.596 INFO:tasks.workunit.client.1.smithi135.stdout:2/508: mkdir d4/d32/d12/d45/dac 0 2024-03-30T04:28:27.598 INFO:tasks.workunit.client.1.smithi135.stdout:8/561: dwrite d6/d8/d14/f29 [0,4194304] 0 2024-03-30T04:28:27.603 INFO:tasks.workunit.client.1.smithi135.stdout:9/588: getdents d7/d29/d78 0 2024-03-30T04:28:27.609 INFO:tasks.workunit.client.1.smithi135.stdout:7/663: truncate d4/db/d1a/d5c/d2c/f88 353084 0 2024-03-30T04:28:27.611 INFO:tasks.workunit.client.1.smithi135.stdout:6/679: dwrite d2/dd/d14/d2b/fc8 [0,4194304] 0 2024-03-30T04:28:27.619 INFO:tasks.workunit.client.1.smithi135.stdout:6/680: symlink d2/dd/d14/d30/d43/dcd/ld9 0 2024-03-30T04:28:27.619 INFO:tasks.workunit.client.1.smithi135.stdout:9/589: read d7/d41/f79 [23146,115256] 0 2024-03-30T04:28:27.623 INFO:tasks.workunit.client.1.smithi135.stdout:8/562: symlink d6/d8/d15/lae 0 2024-03-30T04:28:27.628 INFO:tasks.workunit.client.1.smithi135.stdout:6/681: dread - d2/dd/d14/d53/fb3 zero size 2024-03-30T04:28:27.632 INFO:tasks.workunit.client.1.smithi135.stdout:2/509: dread d4/d32/d2e/d2f/f38 [0,4194304] 0 2024-03-30T04:28:27.632 INFO:tasks.workunit.client.1.smithi135.stdout:9/590: unlink d7/d29/la8 0 2024-03-30T04:28:27.633 INFO:tasks.workunit.client.1.smithi135.stdout:3/599: dwrite d3/d54/d5c/da1/fa8 [0,4194304] 0 2024-03-30T04:28:27.635 INFO:tasks.workunit.client.1.smithi135.stdout:7/664: getdents d4/db/d1a/d4b/d40/d6e 0 2024-03-30T04:28:27.636 INFO:tasks.workunit.client.1.smithi135.stdout:6/682: write d2/dd/d14/d2b/d86/d55/fb1 [148144,79430] 0 2024-03-30T04:28:27.637 INFO:tasks.workunit.client.1.smithi135.stdout:8/563: write d6/d8/d15/d54/d4f/f78 [4732043,109820] 0 2024-03-30T04:28:27.641 INFO:tasks.workunit.client.1.smithi135.stdout:3/600: chown d3/d10/lc4 122215014 1 2024-03-30T04:28:27.642 INFO:tasks.workunit.client.1.smithi135.stdout:6/683: write d2/d68/fb9 [588218,49427] 0 2024-03-30T04:28:27.650 INFO:tasks.workunit.client.1.smithi135.stdout:8/564: write d6/d8/f1c [1453003,77378] 0 2024-03-30T04:28:27.653 INFO:tasks.workunit.client.1.smithi135.stdout:0/703: dread d0/d5/d9/f17 [0,4194304] 0 2024-03-30T04:28:27.658 INFO:tasks.workunit.client.1.smithi135.stdout:7/665: rename d4/db/d1a/d5c/d82/f9d to d4/db/de/d5a/dae/fdb 0 2024-03-30T04:28:27.659 INFO:tasks.workunit.client.1.smithi135.stdout:6/684: truncate d2/dd/d14/d2b/d86/fd1 33665 0 2024-03-30T04:28:27.659 INFO:tasks.workunit.client.1.smithi135.stdout:8/565: mkdir d6/d8/d15/d54/d7e/daf 0 2024-03-30T04:28:27.661 INFO:tasks.workunit.client.1.smithi135.stdout:0/704: rmdir d0/d5/d9/d23/d2a/d60 39 2024-03-30T04:28:27.667 INFO:tasks.workunit.client.1.smithi135.stdout:6/685: creat d2/dd/d18/d87/fda x:0 0 0 2024-03-30T04:28:27.673 INFO:tasks.workunit.client.1.smithi135.stdout:9/591: dwrite d7/d29/d78/da4/d37/f38 [0,4194304] 0 2024-03-30T04:28:27.673 INFO:tasks.workunit.client.1.smithi135.stdout:0/705: symlink d0/d5/d9/d23/d58/d63/le2 0 2024-03-30T04:28:27.674 INFO:tasks.workunit.client.1.smithi135.stdout:3/601: dwrite d3/d50/d22/fc0 [0,4194304] 0 2024-03-30T04:28:27.681 INFO:tasks.workunit.client.1.smithi135.stdout:0/706: chown d0/d5/d9/d23/d58/d63/d77/d8b/dcc 22 1 2024-03-30T04:28:27.683 INFO:tasks.workunit.client.1.smithi135.stdout:6/686: fsync d2/dd/d14/d38/f9a 0 2024-03-30T04:28:27.688 INFO:tasks.workunit.client.1.smithi135.stdout:8/566: dwrite d6/d8/d14/f30 [0,4194304] 0 2024-03-30T04:28:27.695 INFO:tasks.workunit.client.1.smithi135.stdout:2/510: dwrite d4/da/d8d/fab [0,4194304] 0 2024-03-30T04:28:27.699 INFO:tasks.workunit.client.1.smithi135.stdout:7/666: dread d4/f16 [0,4194304] 0 2024-03-30T04:28:27.701 INFO:tasks.workunit.client.1.smithi135.stdout:9/592: rename d7/d29/d78/da4/d37/f38 to d7/d4f/d5c/d63/fc4 0 2024-03-30T04:28:27.702 INFO:tasks.workunit.client.1.smithi135.stdout:0/707: creat d0/db5/db8/fe3 x:0 0 0 2024-03-30T04:28:27.706 INFO:tasks.workunit.client.1.smithi135.stdout:3/602: creat d3/d54/dab/fd0 x:0 0 0 2024-03-30T04:28:27.708 INFO:tasks.workunit.client.1.smithi135.stdout:2/511: unlink d4/d32/f26 0 2024-03-30T04:28:27.710 INFO:tasks.workunit.client.1.smithi135.stdout:8/567: creat d6/d23/d9d/dad/fb0 x:0 0 0 2024-03-30T04:28:27.710 INFO:tasks.workunit.client.1.smithi135.stdout:9/593: write d7/d29/f46 [77956,55570] 0 2024-03-30T04:28:27.712 INFO:tasks.workunit.client.1.smithi135.stdout:7/667: read - d4/db/d1a/d5c/d2c/d51/da8/fb7 zero size 2024-03-30T04:28:27.718 INFO:tasks.workunit.client.1.smithi135.stdout:0/708: getdents d0/d5/d9/d23 0 2024-03-30T04:28:27.718 INFO:tasks.workunit.client.1.smithi135.stdout:3/603: creat d3/d10/d16/d23/d6a/d86/fd1 x:0 0 0 2024-03-30T04:28:27.719 INFO:tasks.workunit.client.1.smithi135.stdout:9/594: write d7/d29/d78/da4/f16 [976835,53068] 0 2024-03-30T04:28:27.726 INFO:tasks.workunit.client.1.smithi135.stdout:7/668: truncate d4/db/de/d5a/dae/fbe 741665 0 2024-03-30T04:28:27.733 INFO:tasks.workunit.client.1.smithi135.stdout:9/595: mknod d7/d4f/d6e/cc5 0 2024-03-30T04:28:27.734 INFO:tasks.workunit.client.1.smithi135.stdout:2/512: write d4/d32/d12/f31 [3318355,43361] 0 2024-03-30T04:28:27.737 INFO:tasks.workunit.client.1.smithi135.stdout:0/709: creat d0/d13/d53/d7f/dc5/fe4 x:0 0 0 2024-03-30T04:28:27.739 INFO:tasks.workunit.client.1.smithi135.stdout:3/604: rmdir d3/d54/dab 39 2024-03-30T04:28:27.742 INFO:tasks.workunit.client.1.smithi135.stdout:0/710: chown d0/d13/d53/d54 209 1 2024-03-30T04:28:27.747 INFO:tasks.workunit.client.1.smithi135.stdout:3/605: chown d3/d50/d46/d4c/c85 0 1 2024-03-30T04:28:27.747 INFO:tasks.workunit.client.1.smithi135.stdout:8/568: dwrite d6/d8/d14/d32/f62 [4194304,4194304] 0 2024-03-30T04:28:27.757 INFO:tasks.workunit.client.1.smithi135.stdout:9/596: creat d7/d4f/d6e/d98/db1/dbc/fc6 x:0 0 0 2024-03-30T04:28:27.758 INFO:tasks.workunit.client.1.smithi135.stdout:2/513: mkdir d4/d32/d2e/d67/dad 0 2024-03-30T04:28:27.767 INFO:tasks.workunit.client.1.smithi135.stdout:0/711: link d0/d13/d53/f97 d0/d1/d16/d7e/fe5 0 2024-03-30T04:28:27.772 INFO:tasks.workunit.client.1.smithi135.stdout:6/687: dread d2/dd/d14/f15 [0,4194304] 0 2024-03-30T04:28:27.780 INFO:tasks.workunit.client.1.smithi135.stdout:8/569: link d6/f6e d6/d8/d15/d79/fb1 0 2024-03-30T04:28:27.785 INFO:tasks.workunit.client.1.smithi135.stdout:3/606: dwrite d3/d50/d22/d33/f61 [0,4194304] 0 2024-03-30T04:28:27.788 INFO:tasks.workunit.client.1.smithi135.stdout:6/688: rename d2/dd/d14/l35 to d2/dd/d18/db0/ldb 0 2024-03-30T04:28:27.790 INFO:tasks.workunit.client.1.smithi135.stdout:9/597: getdents d7/d55/d8d 0 2024-03-30T04:28:27.791 INFO:tasks.workunit.client.1.smithi135.stdout:7/669: write d4/db/d1a/d5c/d5d/f8d [2106191,53051] 0 2024-03-30T04:28:27.797 INFO:tasks.workunit.client.1.smithi135.stdout:8/570: mkdir d6/d8/d15/d54/d8e/d72/d8b/db2 0 2024-03-30T04:28:27.801 INFO:tasks.workunit.client.1.smithi135.stdout:3/607: write d3/d50/d46/fb8 [45731,69534] 0 2024-03-30T04:28:27.808 INFO:tasks.workunit.client.1.smithi135.stdout:3/608: write d3/d50/f25 [2169780,75963] 0 2024-03-30T04:28:27.810 INFO:tasks.workunit.client.1.smithi135.stdout:8/571: write d6/d8/d15/d54/d4f/f83 [816688,22273] 0 2024-03-30T04:28:27.818 INFO:tasks.workunit.client.1.smithi135.stdout:9/598: dread d7/d29/d78/da4/f80 [0,4194304] 0 2024-03-30T04:28:27.819 INFO:tasks.workunit.client.1.smithi135.stdout:1/579: sync 2024-03-30T04:28:27.823 INFO:tasks.workunit.client.1.smithi135.stdout:7/670: rename d4/db/de to d4/db/d1a/d3d/ddc 0 2024-03-30T04:28:27.824 INFO:tasks.workunit.client.1.smithi135.stdout:2/514: dread d4/d32/d12/f2a [0,4194304] 0 2024-03-30T04:28:27.825 INFO:tasks.workunit.client.1.smithi135.stdout:0/712: dread d0/d13/d53/f97 [0,4194304] 0 2024-03-30T04:28:27.825 INFO:tasks.workunit.client.1.smithi135.stdout:3/609: creat d3/d6e/fd2 x:0 0 0 2024-03-30T04:28:27.827 INFO:tasks.workunit.client.1.smithi135.stdout:8/572: mkdir d6/d23/db3 0 2024-03-30T04:28:27.834 INFO:tasks.workunit.client.1.smithi135.stdout:7/671: chown d4/db/d1a/d5c/d59/d76/l99 1475921 1 2024-03-30T04:28:27.837 INFO:tasks.workunit.client.1.smithi135.stdout:4/590: sync 2024-03-30T04:28:27.837 INFO:tasks.workunit.client.1.smithi135.stdout:5/569: sync 2024-03-30T04:28:27.842 INFO:tasks.workunit.client.1.smithi135.stdout:9/599: mknod d7/d29/d78/da4/d37/dba/cc7 0 2024-03-30T04:28:27.843 INFO:tasks.workunit.client.1.smithi135.stdout:6/689: dread d2/d2e/f37 [0,4194304] 0 2024-03-30T04:28:27.844 INFO:tasks.workunit.client.1.smithi135.stdout:4/591: chown d10/d1a/d3d/dcb/fcd 8447 1 2024-03-30T04:28:27.845 INFO:tasks.workunit.client.1.smithi135.stdout:8/573: fdatasync d6/d8/d14/d32/f9a 0 2024-03-30T04:28:27.846 INFO:tasks.workunit.client.1.smithi135.stdout:5/570: unlink d1/d45/d52/l95 0 2024-03-30T04:28:27.850 INFO:tasks.workunit.client.1.smithi135.stdout:7/672: symlink d4/db/d1a/d5c/d2c/ldd 0 2024-03-30T04:28:27.853 INFO:tasks.workunit.client.1.smithi135.stdout:0/713: dread d0/d5/d9/fb [0,4194304] 0 2024-03-30T04:28:27.854 INFO:tasks.workunit.client.1.smithi135.stdout:4/592: creat d10/d72/fd0 x:0 0 0 2024-03-30T04:28:27.855 INFO:tasks.workunit.client.1.smithi135.stdout:5/571: mkdir d1/d45/d52/da5 0 2024-03-30T04:28:27.858 INFO:tasks.workunit.client.1.smithi135.stdout:9/600: symlink d7/d41/db4/lc8 0 2024-03-30T04:28:27.861 INFO:tasks.workunit.client.1.smithi135.stdout:3/610: dread d3/d54/d5c/f64 [0,4194304] 0 2024-03-30T04:28:27.863 INFO:tasks.workunit.client.1.smithi135.stdout:7/673: dread d4/db/d1a/d5c/d2c/d51/f7e [0,4194304] 0 2024-03-30T04:28:27.865 INFO:tasks.workunit.client.1.smithi135.stdout:6/690: link d2/d2e/la6 d2/dd/d14/d30/d43/dcd/dbe/ldc 0 2024-03-30T04:28:27.866 INFO:tasks.workunit.client.1.smithi135.stdout:0/714: symlink d0/d1/d16/dd5/le6 0 2024-03-30T04:28:27.866 INFO:tasks.workunit.client.1.smithi135.stdout:7/674: fsync d4/fa7 0 2024-03-30T04:28:27.866 INFO:tasks.workunit.client.1.smithi135.stdout:4/593: symlink d10/d4d/d89/d9f/daa/ld1 0 2024-03-30T04:28:27.868 INFO:tasks.workunit.client.1.smithi135.stdout:9/601: creat d7/d29/d78/da4/d37/dba/fc9 x:0 0 0 2024-03-30T04:28:27.873 INFO:tasks.workunit.client.1.smithi135.stdout:0/715: rmdir d0/d5/d9/d23/d58/d3b 39 2024-03-30T04:28:27.875 INFO:tasks.workunit.client.1.smithi135.stdout:6/691: mknod d2/dd/cdd 0 2024-03-30T04:28:27.878 INFO:tasks.workunit.client.1.smithi135.stdout:9/602: mknod d7/d29/d78/da4/d37/cca 0 2024-03-30T04:28:27.878 INFO:tasks.workunit.client.1.smithi135.stdout:4/594: creat d10/d72/fd2 x:0 0 0 2024-03-30T04:28:27.880 INFO:tasks.workunit.client.1.smithi135.stdout:7/675: rename d4/db/d1a/d4b/f45 to d4/db/d1a/d5c/d2c/d51/fde 0 2024-03-30T04:28:27.883 INFO:tasks.workunit.client.1.smithi135.stdout:9/603: rmdir d7/d90 39 2024-03-30T04:28:27.888 INFO:tasks.workunit.client.1.smithi135.stdout:3/611: dread d3/d10/f14 [0,4194304] 0 2024-03-30T04:28:27.891 INFO:tasks.workunit.client.1.smithi135.stdout:3/612: readlink d3/d50/d22/l2f 0 2024-03-30T04:28:27.892 INFO:tasks.workunit.client.1.smithi135.stdout:5/572: dwrite d1/d45/f6b [0,4194304] 0 2024-03-30T04:28:27.907 INFO:tasks.workunit.client.1.smithi135.stdout:6/692: dwrite d2/dd/d14/d2b/d86/d55/fb2 [0,4194304] 0 2024-03-30T04:28:27.919 INFO:tasks.workunit.client.1.smithi135.stdout:7/676: dread d4/fc9 [0,4194304] 0 2024-03-30T04:28:27.919 INFO:tasks.workunit.client.1.smithi135.stdout:7/677: stat d4/db/d1a/ccb 0 2024-03-30T04:28:27.958 INFO:tasks.workunit.client.1.smithi135.stdout:6/693: mkdir d2/dd/d14/d38/dde 0 2024-03-30T04:28:27.960 INFO:tasks.workunit.client.1.smithi135.stdout:7/678: mknod d4/db/d1a/d5c/d2c/d83/cdf 0 2024-03-30T04:28:27.960 INFO:tasks.workunit.client.1.smithi135.stdout:6/694: chown d2/dd/d18/db0/lcc 458 1 2024-03-30T04:28:27.965 INFO:tasks.workunit.client.1.smithi135.stdout:6/695: write d2/dd/d18/f24 [5214165,90356] 0 2024-03-30T04:28:27.965 INFO:tasks.workunit.client.1.smithi135.stdout:0/716: getdents d0/d13 0 2024-03-30T04:28:27.978 INFO:tasks.workunit.client.1.smithi135.stdout:2/515: write d4/d32/d12/f3f [2992367,92404] 0 2024-03-30T04:28:27.985 INFO:tasks.workunit.client.1.smithi135.stdout:0/717: link d0/d5/d9/d23/d5d/l73 d0/d5/d9/d23/d2a/d60/le7 0 2024-03-30T04:28:27.986 INFO:tasks.workunit.client.1.smithi135.stdout:1/580: write d16/d29/f35 [2530796,96936] 0 2024-03-30T04:28:27.989 INFO:tasks.workunit.client.1.smithi135.stdout:1/581: fsync d16/d5a/d5f/d3a/d6f/f95 0 2024-03-30T04:28:28.005 INFO:tasks.workunit.client.1.smithi135.stdout:8/574: write d6/d8/f3b [678134,81709] 0 2024-03-30T04:28:28.008 INFO:tasks.workunit.client.1.smithi135.stdout:8/575: fdatasync d6/d23/f5c 0 2024-03-30T04:28:28.024 INFO:tasks.workunit.client.1.smithi135.stdout:8/576: getdents d6/d23/d9d/dad 0 2024-03-30T04:28:28.035 INFO:tasks.workunit.client.1.smithi135.stdout:8/577: link d6/d8/d15/d2a/f4b d6/d8/d15/fb4 0 2024-03-30T04:28:28.041 INFO:tasks.workunit.client.1.smithi135.stdout:5/573: creat d1/d7/d1a/d1f/fa6 x:0 0 0 2024-03-30T04:28:28.058 INFO:tasks.workunit.client.1.smithi135.stdout:4/595: rmdir d10/d4d/dc1 39 2024-03-30T04:28:28.059 INFO:tasks.workunit.client.1.smithi135.stdout:9/604: write d7/f11 [4789188,98771] 0 2024-03-30T04:28:28.068 INFO:tasks.workunit.client.1.smithi135.stdout:4/596: creat d10/d4d/dc1/fd3 x:0 0 0 2024-03-30T04:28:28.070 INFO:tasks.workunit.client.1.smithi135.stdout:7/679: write d4/db/f14 [265046,97413] 0 2024-03-30T04:28:28.074 INFO:tasks.workunit.client.1.smithi135.stdout:9/605: dread d7/d4f/d6e/f87 [0,4194304] 0 2024-03-30T04:28:28.076 INFO:tasks.workunit.client.1.smithi135.stdout:4/597: mkdir d10/d1a/d2b/daf/dd4 0 2024-03-30T04:28:28.078 INFO:tasks.workunit.client.1.smithi135.stdout:7/680: mknod d4/db/d1a/d5c/d2c/d51/d56/dbd/ce0 0 2024-03-30T04:28:28.080 INFO:tasks.workunit.client.1.smithi135.stdout:9/606: symlink d7/d41/db4/lcb 0 2024-03-30T04:28:28.082 INFO:tasks.workunit.client.1.smithi135.stdout:9/607: dread - d7/d29/fae zero size 2024-03-30T04:28:28.083 INFO:tasks.workunit.client.1.smithi135.stdout:4/598: fsync d10/d1a/d32/d40/f4b 0 2024-03-30T04:28:28.085 INFO:tasks.workunit.client.1.smithi135.stdout:9/608: fsync d7/f11 0 2024-03-30T04:28:28.087 INFO:tasks.workunit.client.1.smithi135.stdout:7/681: write d4/db/d1a/d5c/d2c/d51/fde [3596375,104404] 0 2024-03-30T04:28:28.089 INFO:tasks.workunit.client.1.smithi135.stdout:9/609: mknod d7/d55/ccc 0 2024-03-30T04:28:28.093 INFO:tasks.workunit.client.1.smithi135.stdout:0/718: write d0/d1/d16/f32 [1709782,9268] 0 2024-03-30T04:28:28.103 INFO:tasks.workunit.client.1.smithi135.stdout:9/610: mknod d7/d4f/d6e/ccd 0 2024-03-30T04:28:28.107 INFO:tasks.workunit.client.1.smithi135.stdout:0/719: mkdir d0/d5/d9/d23/d2a/d60/de8 0 2024-03-30T04:28:28.111 INFO:tasks.workunit.client.1.smithi135.stdout:1/582: dwrite d16/d29/d2c/d43/d50/f68 [0,4194304] 0 2024-03-30T04:28:28.113 INFO:tasks.workunit.client.1.smithi135.stdout:4/599: dwrite d10/d1a/f21 [8388608,4194304] 0 2024-03-30T04:28:28.121 INFO:tasks.workunit.client.1.smithi135.stdout:0/720: rename d0/d19/l4e to d0/d5/d9/d23/d2a/dc4/le9 0 2024-03-30T04:28:28.126 INFO:tasks.workunit.client.1.smithi135.stdout:4/600: readlink d10/d72/lc8 0 2024-03-30T04:28:28.134 INFO:tasks.workunit.client.1.smithi135.stdout:2/516: symlink d4/lae 0 2024-03-30T04:28:28.136 INFO:tasks.workunit.client.1.smithi135.stdout:0/721: creat d0/d13/d53/d7f/d83/fea x:0 0 0 2024-03-30T04:28:28.137 INFO:tasks.workunit.client.1.smithi135.stdout:9/611: link d7/d4f/cab d7/d4f/d6e/d98/db1/dbc/cce 0 2024-03-30T04:28:28.139 INFO:tasks.workunit.client.1.smithi135.stdout:0/722: write d0/d5/d9/d23/fc3 [418390,47656] 0 2024-03-30T04:28:28.144 INFO:tasks.workunit.client.1.smithi135.stdout:0/723: chown d0/d13/d1e/d3a/f61 6 1 2024-03-30T04:28:28.146 INFO:tasks.workunit.client.1.smithi135.stdout:4/601: dwrite d10/f14 [4194304,4194304] 0 2024-03-30T04:28:28.152 INFO:tasks.workunit.client.1.smithi135.stdout:5/574: symlink d1/d3/la7 0 2024-03-30T04:28:28.154 INFO:tasks.workunit.client.1.smithi135.stdout:0/724: fdatasync d0/d13/d53/d7f/fba 0 2024-03-30T04:28:28.157 INFO:tasks.workunit.client.1.smithi135.stdout:5/575: write d1/d7/d11/d3c/f44 [4357283,37621] 0 2024-03-30T04:28:28.161 INFO:tasks.workunit.client.1.smithi135.stdout:1/583: symlink d16/d19/lc5 0 2024-03-30T04:28:28.162 INFO:tasks.workunit.client.1.smithi135.stdout:5/576: creat d1/d3/d2b/fa8 x:0 0 0 2024-03-30T04:28:28.165 INFO:tasks.workunit.client.1.smithi135.stdout:1/584: chown d16/d19/da4/d99/l81 13388 1 2024-03-30T04:28:28.168 INFO:tasks.workunit.client.1.smithi135.stdout:0/725: rename d0/d19/d2d/c41 to d0/d34/ceb 0 2024-03-30T04:28:28.170 INFO:tasks.workunit.client.1.smithi135.stdout:1/585: chown d16/d19/da4/l31 169 1 2024-03-30T04:28:28.172 INFO:tasks.workunit.client.1.smithi135.stdout:0/726: creat d0/d5/d10/fec x:0 0 0 2024-03-30T04:28:28.173 INFO:tasks.workunit.client.1.smithi135.stdout:1/586: chown d16/d29/d2c/d3c/lb5 0 1 2024-03-30T04:28:28.175 INFO:tasks.workunit.client.1.smithi135.stdout:0/727: write d0/f3 [2546403,47376] 0 2024-03-30T04:28:28.178 INFO:tasks.workunit.client.1.smithi135.stdout:0/728: rmdir d0/d1/d16 39 2024-03-30T04:28:28.180 INFO:tasks.workunit.client.1.smithi135.stdout:0/729: chown d0/d5/d9/fd 1596326330 1 2024-03-30T04:28:28.181 INFO:tasks.workunit.client.1.smithi135.stdout:1/587: mkdir d16/d5a/d5f/d3a/dc6 0 2024-03-30T04:28:28.183 INFO:tasks.workunit.client.1.smithi135.stdout:0/730: read d0/d1/d16/f4b [2676218,17467] 0 2024-03-30T04:28:28.203 INFO:tasks.workunit.client.1.smithi135.stdout:7/682: dwrite d4/db/d1a/d5c/d2c/f39 [4194304,4194304] 0 2024-03-30T04:28:28.209 INFO:tasks.workunit.client.1.smithi135.stdout:4/602: write d10/d1a/d32/d40/f4b [26158,125827] 0 2024-03-30T04:28:28.212 INFO:tasks.workunit.client.1.smithi135.stdout:5/577: write d1/d7/f10 [6319684,64535] 0 2024-03-30T04:28:28.213 INFO:tasks.workunit.client.1.smithi135.stdout:0/731: dread d0/d5/d9/fd [0,4194304] 0 2024-03-30T04:28:28.214 INFO:tasks.workunit.client.1.smithi135.stdout:5/578: fsync d1/d7/d11/d3c/f44 0 2024-03-30T04:28:28.216 INFO:tasks.workunit.client.1.smithi135.stdout:2/517: chown d4/d32/d12/d45/d3c/c66 16856 1 2024-03-30T04:28:28.221 INFO:tasks.workunit.client.1.smithi135.stdout:5/579: creat d1/d55/fa9 x:0 0 0 2024-03-30T04:28:28.223 INFO:tasks.workunit.client.1.smithi135.stdout:4/603: mknod d10/d4d/cd5 0 2024-03-30T04:28:28.227 INFO:tasks.workunit.client.1.smithi135.stdout:0/732: creat d0/d19/fed x:0 0 0 2024-03-30T04:28:28.227 INFO:tasks.workunit.client.1.smithi135.stdout:9/612: dwrite f1 [0,4194304] 0 2024-03-30T04:28:28.227 INFO:tasks.workunit.client.1.smithi135.stdout:2/518: fdatasync d4/da/d7a/f88 0 2024-03-30T04:28:28.228 INFO:tasks.workunit.client.1.smithi135.stdout:4/604: stat d10/d1a/f27 0 2024-03-30T04:28:28.233 INFO:tasks.workunit.client.1.smithi135.stdout:5/580: mknod d1/d3/d2b/d54/d5e/d5c/d79/caa 0 2024-03-30T04:28:28.237 INFO:tasks.workunit.client.1.smithi135.stdout:4/605: dread - d10/d1a/d42/d46/f82 zero size 2024-03-30T04:28:28.239 INFO:tasks.workunit.client.1.smithi135.stdout:5/581: fdatasync d1/d7/f43 0 2024-03-30T04:28:28.240 INFO:tasks.workunit.client.1.smithi135.stdout:0/733: write d0/d5/d9/d23/fc3 [768950,128981] 0 2024-03-30T04:28:28.242 INFO:tasks.workunit.client.1.smithi135.stdout:1/588: dwrite d16/d5a/d5f/f4e [0,4194304] 0 2024-03-30T04:28:28.242 INFO:tasks.workunit.client.1.smithi135.stdout:9/613: symlink d7/d55/d8d/lcf 0 2024-03-30T04:28:28.242 INFO:tasks.workunit.client.1.smithi135.stdout:7/683: dwrite d4/db/d1a/d3d/ddc/fd1 [0,4194304] 0 2024-03-30T04:28:28.243 INFO:tasks.workunit.client.1.smithi135.stdout:2/519: mkdir d4/d32/daf 0 2024-03-30T04:28:28.243 INFO:tasks.workunit.client.1.smithi135.stdout:4/606: write d10/d72/fd2 [595342,11571] 0 2024-03-30T04:28:28.252 INFO:tasks.workunit.client.1.smithi135.stdout:1/589: creat d16/d5a/d5f/d3a/d6f/fc7 x:0 0 0 2024-03-30T04:28:28.255 INFO:tasks.workunit.client.1.smithi135.stdout:0/734: symlink d0/d34/d3f/lee 0 2024-03-30T04:28:28.258 INFO:tasks.workunit.client.1.smithi135.stdout:7/684: mkdir d4/db/d1a/d3d/ddc/db0/de1 0 2024-03-30T04:28:28.259 INFO:tasks.workunit.client.1.smithi135.stdout:9/614: write d7/d55/f5b [552200,49872] 0 2024-03-30T04:28:28.261 INFO:tasks.workunit.client.1.smithi135.stdout:5/582: link d1/f6e d1/d3/d2b/d54/d6d/fab 0 2024-03-30T04:28:28.261 INFO:tasks.workunit.client.1.smithi135.stdout:4/607: write d10/d72/f7d [1602531,18138] 0 2024-03-30T04:28:28.267 INFO:tasks.workunit.client.1.smithi135.stdout:7/685: mknod d4/db/d1a/d5c/d2c/d51/d56/dbd/ce2 0 2024-03-30T04:28:28.272 INFO:tasks.workunit.client.1.smithi135.stdout:1/590: rename d16/d5a/d5f/d3a/d6f/lb0 to d16/d29/d2c/lc8 0 2024-03-30T04:28:28.273 INFO:tasks.workunit.client.1.smithi135.stdout:5/583: fdatasync d1/f39 0 2024-03-30T04:28:28.276 INFO:tasks.workunit.client.1.smithi135.stdout:2/520: getdents d4/d6/d47 0 2024-03-30T04:28:28.280 INFO:tasks.workunit.client.1.smithi135.stdout:0/735: getdents d0/d5/d42/d9f 0 2024-03-30T04:28:28.280 INFO:tasks.workunit.client.1.smithi135.stdout:1/591: rmdir d16/d5a 39 2024-03-30T04:28:28.285 INFO:tasks.workunit.client.1.smithi135.stdout:2/521: truncate d4/d6/f9c 264105 0 2024-03-30T04:28:28.290 INFO:tasks.workunit.client.1.smithi135.stdout:5/584: dread d1/d7/f2c [0,4194304] 0 2024-03-30T04:28:28.293 INFO:tasks.workunit.client.1.smithi135.stdout:0/736: mknod d0/d19/d2d/d79/cef 0 2024-03-30T04:28:28.293 INFO:tasks.workunit.client.1.smithi135.stdout:4/608: dwrite fa [0,4194304] 0 2024-03-30T04:28:28.294 INFO:tasks.workunit.client.1.smithi135.stdout:7/686: truncate d4/db/d1a/d3d/ddc/d5a/dae/fdb 669708 0 2024-03-30T04:28:28.295 INFO:tasks.workunit.client.1.smithi135.stdout:9/615: dwrite d7/d41/f59 [0,4194304] 0 2024-03-30T04:28:28.296 INFO:tasks.workunit.client.1.smithi135.stdout:5/585: read d1/d45/f6b [3962663,75137] 0 2024-03-30T04:28:28.297 INFO:tasks.workunit.client.1.smithi135.stdout:2/522: unlink d4/d32/d12/d45/d7d/c81 0 2024-03-30T04:28:28.297 INFO:tasks.workunit.client.1.smithi135.stdout:1/592: rename d16/d19 to d16/d5a/d5f/d3d/d8d/dc9 0 2024-03-30T04:28:28.302 INFO:tasks.workunit.client.1.smithi135.stdout:4/609: chown d10/d1a/d32/d40/c5f 1982 1 2024-03-30T04:28:28.303 INFO:tasks.workunit.client.1.smithi135.stdout:9/616: rmdir d7 39 2024-03-30T04:28:28.308 INFO:tasks.workunit.client.1.smithi135.stdout:7/687: creat d4/db/d1a/d5c/d2c/d2e/db9/d75/fe3 x:0 0 0 2024-03-30T04:28:28.309 INFO:tasks.workunit.client.1.smithi135.stdout:0/737: mknod d0/cf0 0 2024-03-30T04:28:28.312 INFO:tasks.workunit.client.1.smithi135.stdout:1/593: creat d16/d5a/d5f/d3d/fca x:0 0 0 2024-03-30T04:28:28.314 INFO:tasks.workunit.client.1.smithi135.stdout:7/688: chown d4/db/d1a/d5c/d2c/d51/fd0 41 1 2024-03-30T04:28:28.318 INFO:tasks.workunit.client.1.smithi135.stdout:7/689: dread - d4/db/d1a/d5c/d59/fb6 zero size 2024-03-30T04:28:28.318 INFO:tasks.workunit.client.1.smithi135.stdout:5/586: mkdir d1/d45/d52/da5/dac 0 2024-03-30T04:28:28.319 INFO:tasks.workunit.client.1.smithi135.stdout:0/738: mkdir d0/d1/d16/dd5/df1 0 2024-03-30T04:28:28.321 INFO:tasks.workunit.client.1.smithi135.stdout:8/578: sync 2024-03-30T04:28:28.322 INFO:tasks.workunit.client.1.smithi135.stdout:3/613: sync 2024-03-30T04:28:28.322 INFO:tasks.workunit.client.1.smithi135.stdout:6/696: sync 2024-03-30T04:28:28.322 INFO:tasks.workunit.client.1.smithi135.stdout:2/523: dwrite d4/da/d8d/fab [0,4194304] 0 2024-03-30T04:28:28.324 INFO:tasks.workunit.client.1.smithi135.stdout:7/690: fdatasync d4/db/d1a/d90/fa3 0 2024-03-30T04:28:28.327 INFO:tasks.workunit.client.1.smithi135.stdout:3/614: write d3/d50/d46/f21 [4043218,99475] 0 2024-03-30T04:28:28.331 INFO:tasks.workunit.client.1.smithi135.stdout:8/579: write d6/d8/d15/d54/d4f/f6c [1168724,68816] 0 2024-03-30T04:28:28.335 INFO:tasks.workunit.client.1.smithi135.stdout:0/739: unlink d0/d5/d10/fa1 0 2024-03-30T04:28:28.337 INFO:tasks.workunit.client.1.smithi135.stdout:9/617: mknod d7/d29/d78/da4/d37/dba/cd0 0 2024-03-30T04:28:28.337 INFO:tasks.workunit.client.1.smithi135.stdout:7/691: write d4/db/d1a/d3d/fd4 [4905100,22646] 0 2024-03-30T04:28:28.342 INFO:tasks.workunit.client.1.smithi135.stdout:5/587: truncate d1/d3/fa2 3452089 0 2024-03-30T04:28:28.342 INFO:tasks.workunit.client.1.smithi135.stdout:0/740: write d0/d13/d53/d7f/d83/fac [3596929,119278] 0 2024-03-30T04:28:28.346 INFO:tasks.workunit.client.1.smithi135.stdout:4/610: dread d10/d1a/f23 [0,4194304] 0 2024-03-30T04:28:28.355 INFO:tasks.workunit.client.1.smithi135.stdout:6/697: link d2/dd/d14/d2b/d86/fc3 d2/d36/fdf 0 2024-03-30T04:28:28.356 INFO:tasks.workunit.client.1.smithi135.stdout:8/580: symlink d6/d8/d15/d54/d8e/d72/d73/d9f/lb5 0 2024-03-30T04:28:28.362 INFO:tasks.workunit.client.1.smithi135.stdout:2/524: creat d4/d6/d99/fb0 x:0 0 0 2024-03-30T04:28:28.363 INFO:tasks.workunit.client.1.smithi135.stdout:0/741: chown d0/d5/d9/d23/d58/f4a 3833241 1 2024-03-30T04:28:28.371 INFO:tasks.workunit.client.1.smithi135.stdout:7/692: rename d4/db/d1a/d5c/d59/dc8 to d4/db/d1a/d5c/d59/d76/de4 0 2024-03-30T04:28:28.373 INFO:tasks.workunit.client.1.smithi135.stdout:4/611: creat d10/d1a/d3d/dcb/fd6 x:0 0 0 2024-03-30T04:28:28.375 INFO:tasks.workunit.client.1.smithi135.stdout:0/742: dread - d0/db5/db8/fe3 zero size 2024-03-30T04:28:28.381 INFO:tasks.workunit.client.1.smithi135.stdout:9/618: dwrite d7/d55/d8d/faf [0,4194304] 0 2024-03-30T04:28:28.381 INFO:tasks.workunit.client.1.smithi135.stdout:2/525: read d4/d32/d2e/f8c [247270,111150] 0 2024-03-30T04:28:28.382 INFO:tasks.workunit.client.1.smithi135.stdout:1/594: dwrite d16/d5a/d5f/d3d/d8d/dc9/f56 [0,4194304] 0 2024-03-30T04:28:28.387 INFO:tasks.workunit.client.1.smithi135.stdout:5/588: dread d1/d45/f6b [0,4194304] 0 2024-03-30T04:28:28.391 INFO:tasks.workunit.client.1.smithi135.stdout:8/581: rename d6/d8/d15/d54/d4f/f6c to d6/d8/d14/da0/fb6 0 2024-03-30T04:28:28.392 INFO:tasks.workunit.client.1.smithi135.stdout:3/615: dwrite d3/d54/d5c/f89 [0,4194304] 0 2024-03-30T04:28:28.395 INFO:tasks.workunit.client.1.smithi135.stdout:9/619: symlink d7/d4f/d5c/da3/ld1 0 2024-03-30T04:28:28.396 INFO:tasks.workunit.client.1.smithi135.stdout:5/589: mkdir d1/d3/d2b/d54/d5e/d5c/dad 0 2024-03-30T04:28:28.397 INFO:tasks.workunit.client.1.smithi135.stdout:9/620: fsync d7/d41/d72/f7d 0 2024-03-30T04:28:28.397 INFO:tasks.workunit.client.1.smithi135.stdout:1/595: creat d16/d5a/d5f/d3d/fcb x:0 0 0 2024-03-30T04:28:28.398 INFO:tasks.workunit.client.1.smithi135.stdout:5/590: readlink d1/d3/d2b/l96 0 2024-03-30T04:28:28.400 INFO:tasks.workunit.client.1.smithi135.stdout:6/698: rename d2/dd/cdd to d2/dd/d14/d30/d43/ce0 0 2024-03-30T04:28:28.402 INFO:tasks.workunit.client.1.smithi135.stdout:1/596: chown d16/d5a/d5f/d3d/f96 140 1 2024-03-30T04:28:28.403 INFO:tasks.workunit.client.1.smithi135.stdout:9/621: dread - d7/d4f/d5c/d63/f81 zero size 2024-03-30T04:28:28.408 INFO:tasks.workunit.client.1.smithi135.stdout:5/591: mknod d1/d3/d76/d92/cae 0 2024-03-30T04:28:28.408 INFO:tasks.workunit.client.1.smithi135.stdout:7/693: rename d4/db/d1a/d3d/ddc/d5a/f68 to d4/db/d1a/d5c/d2c/d2e/fe5 0 2024-03-30T04:28:28.414 INFO:tasks.workunit.client.1.smithi135.stdout:6/699: symlink d2/dd/d14/d38/le1 0 2024-03-30T04:28:28.417 INFO:tasks.workunit.client.1.smithi135.stdout:6/700: chown d2/dd/c32 13703 1 2024-03-30T04:28:28.419 INFO:tasks.workunit.client.1.smithi135.stdout:4/612: dread d10/d4d/d89/f8f [0,4194304] 0 2024-03-30T04:28:28.421 INFO:tasks.workunit.client.1.smithi135.stdout:1/597: rename d16/d5a/d5f/d3a/d6f to d16/d29/d32/dcc 0 2024-03-30T04:28:28.422 INFO:tasks.workunit.client.1.smithi135.stdout:7/694: mknod d4/db/d1a/d5c/d59/ce6 0 2024-03-30T04:28:28.425 INFO:tasks.workunit.client.1.smithi135.stdout:3/616: dwrite d3/f51 [0,4194304] 0 2024-03-30T04:28:28.426 INFO:tasks.workunit.client.1.smithi135.stdout:5/592: mknod d1/d3/d2b/d54/d5e/caf 0 2024-03-30T04:28:28.428 INFO:tasks.workunit.client.1.smithi135.stdout:6/701: readlink d2/dd/d14/d2b/d86/l5e 0 2024-03-30T04:28:28.428 INFO:tasks.workunit.client.1.smithi135.stdout:5/593: dread - d1/d3/fa4 zero size 2024-03-30T04:28:28.429 INFO:tasks.workunit.client.1.smithi135.stdout:1/598: chown d16/d5a/d5f/d3d/d8d/dc9/f56 46 1 2024-03-30T04:28:28.434 INFO:tasks.workunit.client.1.smithi135.stdout:8/582: getdents d6/d8/d14/da0 0 2024-03-30T04:28:28.436 INFO:tasks.workunit.client.1.smithi135.stdout:8/583: dread - d6/d8/d14/d32/fab zero size 2024-03-30T04:28:28.439 INFO:tasks.workunit.client.1.smithi135.stdout:8/584: fsync d6/d23/d9d/dad/fb0 0 2024-03-30T04:28:28.441 INFO:tasks.workunit.client.1.smithi135.stdout:5/594: rename d1/d3/d2b/d54/d6d/fab to d1/d3/d76/d92/fb0 0 2024-03-30T04:28:28.443 INFO:tasks.workunit.client.1.smithi135.stdout:7/695: mknod d4/db/d1a/d3d/d96/ce7 0 2024-03-30T04:28:28.445 INFO:tasks.workunit.client.1.smithi135.stdout:0/743: truncate d0/d13/d1e/fb2 1131427 0 2024-03-30T04:28:28.447 INFO:tasks.workunit.client.1.smithi135.stdout:2/526: truncate d4/da/fb 802536 0 2024-03-30T04:28:28.448 INFO:tasks.workunit.client.1.smithi135.stdout:1/599: creat d16/d5a/d5f/d3d/db4/fcd x:0 0 0 2024-03-30T04:28:28.453 INFO:tasks.workunit.client.1.smithi135.stdout:7/696: rename d4/db/d79/d92/cb2 to d4/db/d1a/d5c/d2c/d51/da8/ce8 0 2024-03-30T04:28:28.464 INFO:tasks.workunit.client.1.smithi135.stdout:1/600: mkdir d16/d29/d2c/dce 0 2024-03-30T04:28:28.466 INFO:tasks.workunit.client.1.smithi135.stdout:7/697: rename d4/db/d1a/d3d/ddc/d5a/dae/fdb to d4/db/d1a/d90/dc7/fe9 0 2024-03-30T04:28:28.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:28 smithi123 ceph-mon[35245]: pgmap v47: 65 pgs: 65 active+clean; 3.0 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 97 MiB/s rd, 228 MiB/s wr, 509 op/s 2024-03-30T04:28:28.469 INFO:tasks.workunit.client.1.smithi135.stdout:3/617: truncate d3/d50/d22/f2e 397920 0 2024-03-30T04:28:28.471 INFO:tasks.workunit.client.1.smithi135.stdout:1/601: creat d16/d29/d2c/dce/fcf x:0 0 0 2024-03-30T04:28:28.477 INFO:tasks.workunit.client.1.smithi135.stdout:0/744: dread d0/d5/d9/d23/f66 [0,4194304] 0 2024-03-30T04:28:28.478 INFO:tasks.workunit.client.1.smithi135.stdout:1/602: write d16/d29/d2c/fb7 [5861593,111578] 0 2024-03-30T04:28:28.485 INFO:tasks.workunit.client.1.smithi135.stdout:2/527: write d4/d32/d12/d45/fa0 [675306,42932] 0 2024-03-30T04:28:28.485 INFO:tasks.workunit.client.1.smithi135.stdout:4/613: dwrite d10/d1a/d32/d40/f8e [0,4194304] 0 2024-03-30T04:28:28.485 INFO:tasks.workunit.client.1.smithi135.stdout:0/745: creat d0/d13/d1e/ff2 x:0 0 0 2024-03-30T04:28:28.490 INFO:tasks.workunit.client.1.smithi135.stdout:1/603: chown d16/d5a/d5f/d3d/d8d/dc9/da4/c67 3329 1 2024-03-30T04:28:28.496 INFO:tasks.workunit.client.1.smithi135.stdout:5/595: dwrite d1/d55/f6c [0,4194304] 0 2024-03-30T04:28:28.496 INFO:tasks.workunit.client.1.smithi135.stdout:7/698: truncate d4/f8 3984569 0 2024-03-30T04:28:28.499 INFO:tasks.workunit.client.1.smithi135.stdout:4/614: dread - d10/d4d/d89/f8b zero size 2024-03-30T04:28:28.500 INFO:tasks.workunit.client.1.smithi135.stdout:3/618: write d3/db/f19 [153118,87924] 0 2024-03-30T04:28:28.509 INFO:tasks.workunit.client.1.smithi135.stdout:8/585: dwrite d6/f50 [4194304,4194304] 0 2024-03-30T04:28:28.512 INFO:tasks.workunit.client.1.smithi135.stdout:7/699: mkdir d4/db/d1a/d90/dea 0 2024-03-30T04:28:28.514 INFO:tasks.workunit.client.1.smithi135.stdout:4/615: read d10/d1a/d32/f96 [638348,28277] 0 2024-03-30T04:28:28.517 INFO:tasks.workunit.client.1.smithi135.stdout:2/528: write d4/d32/d2e/f8c [824091,120835] 0 2024-03-30T04:28:28.517 INFO:tasks.workunit.client.1.smithi135.stdout:1/604: link d16/f93 d16/d5a/d5f/d3d/db4/fd0 0 2024-03-30T04:28:28.520 INFO:tasks.workunit.client.1.smithi135.stdout:3/619: creat d3/d54/dab/fd3 x:0 0 0 2024-03-30T04:28:28.523 INFO:tasks.workunit.client.1.smithi135.stdout:7/700: fsync d4/db/d1a/d5c/d2c/d51/da8/fc2 0 2024-03-30T04:28:28.528 INFO:tasks.workunit.client.1.smithi135.stdout:4/616: mkdir d10/d1a/d42/d66/dd7 0 2024-03-30T04:28:28.531 INFO:tasks.workunit.client.1.smithi135.stdout:0/746: dread d0/d5/d9/d23/d58/f1f [4194304,4194304] 0 2024-03-30T04:28:28.531 INFO:tasks.workunit.client.1.smithi135.stdout:1/605: dread - d16/d5a/d5f/d3d/d8d/dc9/da4/f5e zero size 2024-03-30T04:28:28.535 INFO:tasks.workunit.client.1.smithi135.stdout:2/529: rename d4/lae to d4/d32/d46/lb1 0 2024-03-30T04:28:28.541 INFO:tasks.workunit.client.1.smithi135.stdout:4/617: mknod d10/d4d/cd8 0 2024-03-30T04:28:28.542 INFO:tasks.workunit.client.1.smithi135.stdout:5/596: dwrite d1/d3/d2b/d54/d5e/d5c/f65 [0,4194304] 0 2024-03-30T04:28:28.545 INFO:tasks.workunit.client.1.smithi135.stdout:1/606: dread - d16/d5a/d5f/d3d/db4/fd0 zero size 2024-03-30T04:28:28.550 INFO:tasks.workunit.client.1.smithi135.stdout:5/597: write d1/d7/d11/f87 [7565692,125217] 0 2024-03-30T04:28:28.551 INFO:tasks.workunit.client.1.smithi135.stdout:1/607: chown d16/d5a/d5f/d3d/db4 13 1 2024-03-30T04:28:28.555 INFO:tasks.workunit.client.1.smithi135.stdout:8/586: write d6/d8/f1c [2260887,128882] 0 2024-03-30T04:28:28.558 INFO:tasks.workunit.client.1.smithi135.stdout:1/608: chown d16/d5a/d5f/d3d/db4/fcd 3892000 1 2024-03-30T04:28:28.558 INFO:tasks.workunit.client.1.smithi135.stdout:4/618: truncate d10/d1a/d32/fb5 763697 0 2024-03-30T04:28:28.558 INFO:tasks.workunit.client.1.smithi135.stdout:5/598: chown d1/d3/d76/d92/cae 11641 1 2024-03-30T04:28:28.562 INFO:tasks.workunit.client.1.smithi135.stdout:7/701: dwrite d4/db/d1a/d5c/d59/fb6 [0,4194304] 0 2024-03-30T04:28:28.563 INFO:tasks.workunit.client.1.smithi135.stdout:2/530: dread - d4/d32/d43/fa9 zero size 2024-03-30T04:28:28.569 INFO:tasks.workunit.client.1.smithi135.stdout:0/747: truncate d0/d13/d53/f80 4086654 0 2024-03-30T04:28:28.569 INFO:tasks.workunit.client.1.smithi135.stdout:5/599: truncate d1/da/f84 649184 0 2024-03-30T04:28:28.570 INFO:tasks.workunit.client.1.smithi135.stdout:4/619: unlink d10/d1a/d32/f96 0 2024-03-30T04:28:28.570 INFO:tasks.workunit.client.1.smithi135.stdout:1/609: rmdir d16/d5a/d5f/d3d 39 2024-03-30T04:28:28.572 INFO:tasks.workunit.client.1.smithi135.stdout:7/702: mknod d4/db/d1a/d90/dc7/ceb 0 2024-03-30T04:28:28.579 INFO:tasks.workunit.client.1.smithi135.stdout:4/620: fsync d10/d1a/d32/d40/d92/f94 0 2024-03-30T04:28:28.579 INFO:tasks.workunit.client.1.smithi135.stdout:0/748: creat d0/d13/d1e/d3a/dc2/ff3 x:0 0 0 2024-03-30T04:28:28.580 INFO:tasks.workunit.client.1.smithi135.stdout:8/587: dwrite d6/f8c [0,4194304] 0 2024-03-30T04:28:28.581 INFO:tasks.workunit.client.1.smithi135.stdout:0/749: readlink d0/d5/d9/d23/d58/l20 0 2024-03-30T04:28:28.585 INFO:tasks.workunit.client.1.smithi135.stdout:7/703: creat d4/db/d1a/d5c/d2c/d51/d56/d81/fec x:0 0 0 2024-03-30T04:28:28.586 INFO:tasks.workunit.client.1.smithi135.stdout:8/588: symlink d6/d8/d15/d2a/lb7 0 2024-03-30T04:28:28.589 INFO:tasks.workunit.client.1.smithi135.stdout:7/704: chown d4/db/d1a/d5c/d2c/d83 11581689 1 2024-03-30T04:28:28.591 INFO:tasks.workunit.client.1.smithi135.stdout:0/750: chown d0/d5/d9/f2e 0 1 2024-03-30T04:28:28.594 INFO:tasks.workunit.client.1.smithi135.stdout:7/705: chown d4/db/d1a/d3d/ddc/l3e 2012347737 1 2024-03-30T04:28:28.595 INFO:tasks.workunit.client.1.smithi135.stdout:8/589: mknod d6/d8/d15/d54/d8e/d72/cb8 0 2024-03-30T04:28:28.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:28 smithi135 ceph-mon[44364]: pgmap v47: 65 pgs: 65 active+clean; 3.0 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 97 MiB/s rd, 228 MiB/s wr, 509 op/s 2024-03-30T04:28:28.601 INFO:tasks.workunit.client.1.smithi135.stdout:7/706: chown d4/db/d1a/d3d/ddc/l17 332828 1 2024-03-30T04:28:28.605 INFO:tasks.workunit.client.1.smithi135.stdout:8/590: truncate d6/d8/d14/d32/f90 200032 0 2024-03-30T04:28:28.607 INFO:tasks.workunit.client.1.smithi135.stdout:4/621: dwrite d10/d72/fa6 [0,4194304] 0 2024-03-30T04:28:28.616 INFO:tasks.workunit.client.1.smithi135.stdout:1/610: dwrite d16/d5a/d5f/d3a/f73 [0,4194304] 0 2024-03-30T04:28:28.619 INFO:tasks.workunit.client.1.smithi135.stdout:8/591: link d6/d8/d14/d32/fab d6/d8/d15/d54/d7e/da6/fb9 0 2024-03-30T04:28:28.623 INFO:tasks.workunit.client.1.smithi135.stdout:4/622: symlink d10/d1a/d32/d40/d92/db9/dcf/ld9 0 2024-03-30T04:28:28.623 INFO:tasks.workunit.client.1.smithi135.stdout:8/592: fsync d6/d23/f27 0 2024-03-30T04:28:28.626 INFO:tasks.workunit.client.1.smithi135.stdout:0/751: dwrite d0/d13/d1e/f49 [4194304,4194304] 0 2024-03-30T04:28:28.631 INFO:tasks.workunit.client.1.smithi135.stdout:4/623: write d10/d1a/d2b/f31 [5438205,43317] 0 2024-03-30T04:28:28.638 INFO:tasks.workunit.client.1.smithi135.stdout:0/752: rename d0/d5/d9/d23/d8a to d0/d13/d91/dd2/df4 0 2024-03-30T04:28:28.647 INFO:tasks.workunit.client.1.smithi135.stdout:7/707: dread d4/db/d1a/d3d/fd4 [0,4194304] 0 2024-03-30T04:28:28.647 INFO:tasks.workunit.client.1.smithi135.stdout:1/611: dwrite d16/d5a/d5f/d3d/d8d/dc9/f1f [0,4194304] 0 2024-03-30T04:28:28.648 INFO:tasks.workunit.client.1.smithi135.stdout:8/593: creat d6/d8/d15/d2a/da9/fba x:0 0 0 2024-03-30T04:28:28.651 INFO:tasks.workunit.client.1.smithi135.stdout:4/624: mkdir d10/d1a/d2b/daf/dd4/dda 0 2024-03-30T04:28:28.652 INFO:tasks.workunit.client.1.smithi135.stdout:1/612: truncate d16/f93 236764 0 2024-03-30T04:28:28.656 INFO:tasks.workunit.client.1.smithi135.stdout:8/594: rmdir d6/d23/db3 0 2024-03-30T04:28:28.656 INFO:tasks.workunit.client.1.smithi135.stdout:1/613: truncate d16/d5a/dad/f7b 9425632 0 2024-03-30T04:28:28.659 INFO:tasks.workunit.client.1.smithi135.stdout:7/708: getdents d4/db/d1a/d3d/ddc/d5a 0 2024-03-30T04:28:28.660 INFO:tasks.workunit.client.1.smithi135.stdout:1/614: read d16/d5a/d5f/d3a/f73 [1018981,57694] 0 2024-03-30T04:28:28.664 INFO:tasks.workunit.client.1.smithi135.stdout:0/753: dwrite d0/d13/d1e/d3a/f61 [0,4194304] 0 2024-03-30T04:28:28.664 INFO:tasks.workunit.client.1.smithi135.stdout:1/615: dread - d16/d29/d2c/d3c/fc0 zero size 2024-03-30T04:28:28.666 INFO:tasks.workunit.client.1.smithi135.stdout:8/595: unlink d6/d8/d15/d2a/f37 0 2024-03-30T04:28:28.674 INFO:tasks.workunit.client.1.smithi135.stdout:0/754: mkdir d0/d5/d42/d9f/df5 0 2024-03-30T04:28:28.678 INFO:tasks.workunit.client.1.smithi135.stdout:0/755: symlink d0/d13/d1e/lf6 0 2024-03-30T04:28:28.680 INFO:tasks.workunit.client.1.smithi135.stdout:7/709: dread d4/db/d1a/d5c/d59/fb6 [0,4194304] 0 2024-03-30T04:28:28.688 INFO:tasks.workunit.client.1.smithi135.stdout:1/616: truncate d16/d29/d2c/fb7 2807197 0 2024-03-30T04:28:28.688 INFO:tasks.workunit.client.1.smithi135.stdout:8/596: dwrite d6/f50 [0,4194304] 0 2024-03-30T04:28:28.700 INFO:tasks.workunit.client.1.smithi135.stdout:1/617: truncate d16/d5a/d5f/d3d/db4/fcd 321337 0 2024-03-30T04:28:28.704 INFO:tasks.workunit.client.1.smithi135.stdout:7/710: dwrite d4/db/d1a/d90/fa3 [0,4194304] 0 2024-03-30T04:28:28.706 INFO:tasks.workunit.client.1.smithi135.stdout:8/597: link d6/d8/d15/d2a/fa4 d6/d8/d15/d54/d8e/d72/d73/d9f/fbb 0 2024-03-30T04:28:28.710 INFO:tasks.workunit.client.1.smithi135.stdout:8/598: write d6/d8/d15/f8a [1026515,44255] 0 2024-03-30T04:28:28.713 INFO:tasks.workunit.client.1.smithi135.stdout:7/711: getdents d4/db/d79/d92 0 2024-03-30T04:28:28.714 INFO:tasks.workunit.client.1.smithi135.stdout:1/618: dread d16/d5a/d5f/f7f [0,4194304] 0 2024-03-30T04:28:28.717 INFO:tasks.workunit.client.1.smithi135.stdout:8/599: rename d6/d8/f1d to d6/d8/d15/d1f/fbc 0 2024-03-30T04:28:28.725 INFO:tasks.workunit.client.1.smithi135.stdout:1/619: link c15 d16/d5a/d5f/d3a/dc6/cd1 0 2024-03-30T04:28:28.732 INFO:tasks.workunit.client.1.smithi135.stdout:7/712: write d4/db/d1a/d5c/d2c/d51/f7e [1662173,87432] 0 2024-03-30T04:28:28.739 INFO:tasks.workunit.client.1.smithi135.stdout:7/713: write d4/db/d79/da9/fa1 [962922,22320] 0 2024-03-30T04:28:28.744 INFO:tasks.workunit.client.1.smithi135.stdout:7/714: creat d4/db/fed x:0 0 0 2024-03-30T04:28:28.748 INFO:tasks.workunit.client.1.smithi135.stdout:1/620: dwrite d16/d5a/d5f/d3d/db4/fd0 [0,4194304] 0 2024-03-30T04:28:28.749 INFO:tasks.workunit.client.1.smithi135.stdout:8/600: dread d6/d8/d14/d32/f62 [0,4194304] 0 2024-03-30T04:28:28.752 INFO:tasks.workunit.client.1.smithi135.stdout:8/601: fsync d6/d8/d15/fb4 0 2024-03-30T04:28:28.755 INFO:tasks.workunit.client.1.smithi135.stdout:8/602: readlink d6/ld 0 2024-03-30T04:28:28.758 INFO:tasks.workunit.client.1.smithi135.stdout:8/603: mkdir d6/d8/d15/d54/d8e/dbd 0 2024-03-30T04:28:28.761 INFO:tasks.workunit.client.1.smithi135.stdout:9/622: sync 2024-03-30T04:28:28.761 INFO:tasks.workunit.client.1.smithi135.stdout:8/604: mknod d6/d8/d14/d32/cbe 0 2024-03-30T04:28:28.767 INFO:tasks.workunit.client.1.smithi135.stdout:9/623: rename d7/d4f/d6e/d98 to d7/d4f/d5c/d63/dd2 0 2024-03-30T04:28:28.769 INFO:tasks.workunit.client.1.smithi135.stdout:1/621: dwrite f3 [0,4194304] 0 2024-03-30T04:28:28.772 INFO:tasks.workunit.client.1.smithi135.stdout:1/622: dread - d16/d5a/dad/d77/fb6 zero size 2024-03-30T04:28:28.773 INFO:tasks.workunit.client.1.smithi135.stdout:9/624: rename d7/d29/d78/da4/l58 to d7/d41/d72/ld3 0 2024-03-30T04:28:28.774 INFO:tasks.workunit.client.1.smithi135.stdout:1/623: stat d16/d29/d32/d85/fba 0 2024-03-30T04:28:28.777 INFO:tasks.workunit.client.1.smithi135.stdout:9/625: mkdir d7/d4f/d6e/dad/dd4 0 2024-03-30T04:28:28.779 INFO:tasks.workunit.client.1.smithi135.stdout:1/624: chown d16/d5a/d5f/d3d/d8d/dc9/da4/d99/l81 81001076 1 2024-03-30T04:28:28.786 INFO:tasks.workunit.client.1.smithi135.stdout:9/626: symlink d7/d4f/d5c/ld5 0 2024-03-30T04:28:28.787 INFO:tasks.workunit.client.1.smithi135.stdout:1/625: fdatasync d16/d5a/d5f/d3d/d8d/dc9/f2b 0 2024-03-30T04:28:28.788 INFO:tasks.workunit.client.1.smithi135.stdout:7/715: dread d4/db/d1a/f24 [0,4194304] 0 2024-03-30T04:28:28.794 INFO:tasks.workunit.client.1.smithi135.stdout:7/716: mkdir d4/db/d1a/d5c/d59/d76/de4/dee 0 2024-03-30T04:28:28.799 INFO:tasks.workunit.client.1.smithi135.stdout:9/627: link d7/d4f/d6e/f87 d7/d29/d78/fd6 0 2024-03-30T04:28:28.800 INFO:tasks.workunit.client.1.smithi135.stdout:7/717: write d4/db/d79/dbc/fa5 [1006354,72693] 0 2024-03-30T04:28:28.803 INFO:tasks.workunit.client.1.smithi135.stdout:9/628: fdatasync d7/d29/d78/da4/f80 0 2024-03-30T04:28:28.805 INFO:tasks.workunit.client.1.smithi135.stdout:7/718: chown d4/db/d1a/d5c/d2c/d51/d56/dbd 21 1 2024-03-30T04:28:28.808 INFO:tasks.workunit.client.1.smithi135.stdout:1/626: dwrite d16/d29/d2c/d3c/f3b [4194304,4194304] 0 2024-03-30T04:28:28.809 INFO:tasks.workunit.client.1.smithi135.stdout:7/719: creat d4/db/d1a/d5c/d2c/d51/d56/d81/fef x:0 0 0 2024-03-30T04:28:28.809 INFO:tasks.workunit.client.1.smithi135.stdout:1/627: fdatasync d16/d29/f35 0 2024-03-30T04:28:28.811 INFO:tasks.workunit.client.1.smithi135.stdout:1/628: chown d16/d29/d32/dcc/fa8 1155832 1 2024-03-30T04:28:28.813 INFO:tasks.workunit.client.1.smithi135.stdout:7/720: unlink d4/db/d1a/d90/dc7/ceb 0 2024-03-30T04:28:28.814 INFO:tasks.workunit.client.1.smithi135.stdout:1/629: creat d16/d29/d32/d9d/fd2 x:0 0 0 2024-03-30T04:28:28.815 INFO:tasks.workunit.client.1.smithi135.stdout:7/721: chown d4/db/d1a/d5c/d2c/f63 173255 1 2024-03-30T04:28:28.818 INFO:tasks.workunit.client.1.smithi135.stdout:1/630: symlink d16/d5a/d5f/d3d/db4/ld3 0 2024-03-30T04:28:28.820 INFO:tasks.workunit.client.1.smithi135.stdout:7/722: creat d4/db/d79/dbc/ff0 x:0 0 0 2024-03-30T04:28:28.821 INFO:tasks.workunit.client.1.smithi135.stdout:1/631: chown d16/d29/d32/d9d/fd2 91 1 2024-03-30T04:28:28.821 INFO:tasks.workunit.client.1.smithi135.stdout:7/723: fdatasync d4/db/d1a/d5c/d59/d76/f91 0 2024-03-30T04:28:28.826 INFO:tasks.workunit.client.1.smithi135.stdout:7/724: creat d4/db/d1a/d5c/d2c/d83/ff1 x:0 0 0 2024-03-30T04:28:28.828 INFO:tasks.workunit.client.1.smithi135.stdout:1/632: getdents d16/d5a/d5f/d3d/d8d/dc9/da4 0 2024-03-30T04:28:28.832 INFO:tasks.workunit.client.1.smithi135.stdout:1/633: write d16/d5a/d5f/d3d/d8d/dc9/da4/d99/f21 [3311603,61077] 0 2024-03-30T04:28:28.846 INFO:tasks.workunit.client.1.smithi135.stdout:1/634: dread d16/d29/d2c/d43/d50/f68 [0,4194304] 0 2024-03-30T04:28:28.850 INFO:tasks.workunit.client.1.smithi135.stdout:1/635: creat d16/d29/d32/dcc/fd4 x:0 0 0 2024-03-30T04:28:28.854 INFO:tasks.workunit.client.1.smithi135.stdout:7/725: dread d4/db/d1a/d5c/d2c/f63 [0,4194304] 0 2024-03-30T04:28:28.856 INFO:tasks.workunit.client.1.smithi135.stdout:7/726: rmdir d4/db/d1a/d5c/d2c/d51/d56/dbd 39 2024-03-30T04:28:28.861 INFO:tasks.workunit.client.1.smithi135.stdout:7/727: rmdir d4/db/d1a/d5c/d2c/d51/da8 39 2024-03-30T04:28:28.864 INFO:tasks.workunit.client.1.smithi135.stdout:1/636: dwrite d16/d29/d32/d85/fba [0,4194304] 0 2024-03-30T04:28:28.868 INFO:tasks.workunit.client.1.smithi135.stdout:7/728: truncate d4/f16 4108613 0 2024-03-30T04:28:28.872 INFO:tasks.workunit.client.1.smithi135.stdout:7/729: creat d4/db/d1a/d90/dea/ff2 x:0 0 0 2024-03-30T04:28:28.876 INFO:tasks.workunit.client.1.smithi135.stdout:7/730: symlink d4/db/d1a/d5c/d5d/lf3 0 2024-03-30T04:28:28.881 INFO:tasks.workunit.client.1.smithi135.stdout:7/731: mkdir d4/db/d1a/d5c/d2c/d2e/db9/df4 0 2024-03-30T04:28:28.884 INFO:tasks.workunit.client.1.smithi135.stdout:1/637: dwrite d16/d5a/d5f/d3d/d8d/dc9/da4/faf [0,4194304] 0 2024-03-30T04:28:28.889 INFO:tasks.workunit.client.1.smithi135.stdout:1/638: write d16/d5a/d5f/d3d/d8d/dc9/da4/d99/f21 [2048492,107686] 0 2024-03-30T04:28:28.895 INFO:tasks.workunit.client.1.smithi135.stdout:1/639: truncate d16/f42 4708211 0 2024-03-30T04:28:28.902 INFO:tasks.workunit.client.1.smithi135.stdout:1/640: creat d16/d5a/d5f/d3d/d8d/dc9/fd5 x:0 0 0 2024-03-30T04:28:28.903 INFO:tasks.workunit.client.1.smithi135.stdout:7/732: dwrite d4/db/d79/da9/f58 [4194304,4194304] 0 2024-03-30T04:28:28.905 INFO:tasks.workunit.client.1.smithi135.stdout:1/641: write d16/d5a/d5f/d3d/d8d/dc9/fd5 [760011,101943] 0 2024-03-30T04:28:28.909 INFO:tasks.workunit.client.1.smithi135.stdout:1/642: write d16/d5a/d5f/f4e [2786953,115976] 0 2024-03-30T04:28:28.913 INFO:tasks.workunit.client.1.smithi135.stdout:1/643: write d16/d5a/d5f/d3d/d8d/dc9/f72 [9550,116695] 0 2024-03-30T04:28:28.915 INFO:tasks.workunit.client.1.smithi135.stdout:1/644: dread - d16/d29/fbb zero size 2024-03-30T04:28:28.918 INFO:tasks.workunit.client.1.smithi135.stdout:1/645: symlink d16/d5a/ld6 0 2024-03-30T04:28:28.922 INFO:tasks.workunit.client.1.smithi135.stdout:1/646: stat d16/d29/d2c/fb7 0 2024-03-30T04:28:28.923 INFO:tasks.workunit.client.1.smithi135.stdout:7/733: dwrite d4/db/d1a/d3d/ddc/f2b [0,4194304] 0 2024-03-30T04:28:28.925 INFO:tasks.workunit.client.1.smithi135.stdout:1/647: rmdir d16/d29/d2c/d43/d50 39 2024-03-30T04:28:28.927 INFO:tasks.workunit.client.1.smithi135.stdout:1/648: read - d16/d29/d32/dcc/f95 zero size 2024-03-30T04:28:28.928 INFO:tasks.workunit.client.1.smithi135.stdout:7/734: write d4/db/d1a/d5c/d2c/f63 [3597095,123473] 0 2024-03-30T04:28:28.931 INFO:tasks.workunit.client.1.smithi135.stdout:1/649: write d16/d5a/d5f/d3d/fcb [510778,26079] 0 2024-03-30T04:28:28.931 INFO:tasks.workunit.client.1.smithi135.stdout:7/735: readlink d4/db/d1a/d3d/ddc/l3e 0 2024-03-30T04:28:28.934 INFO:tasks.workunit.client.1.smithi135.stdout:1/650: getdents d16/d9f 0 2024-03-30T04:28:28.937 INFO:tasks.workunit.client.1.smithi135.stdout:7/736: rmdir d4/db/d1a/d4b/dac 0 2024-03-30T04:28:28.940 INFO:tasks.workunit.client.1.smithi135.stdout:7/737: write d4/db/d1a/d3d/ddc/d5a/dae/fd3 [930097,76585] 0 2024-03-30T04:28:28.945 INFO:tasks.workunit.client.1.smithi135.stdout:7/738: read d4/db/d1a/d5c/d2c/d51/d56/d81/f7f [3455542,89012] 0 2024-03-30T04:28:28.951 INFO:tasks.workunit.client.1.smithi135.stdout:1/651: dwrite d16/d29/d32/dcc/fa8 [0,4194304] 0 2024-03-30T04:28:28.956 INFO:tasks.workunit.client.1.smithi135.stdout:1/652: stat d16/d5a/d5f/d3d/d8d/dc9/da4/d99/c83 0 2024-03-30T04:28:28.962 INFO:tasks.workunit.client.1.smithi135.stdout:7/739: dwrite d4/db/fed [0,4194304] 0 2024-03-30T04:28:28.969 INFO:tasks.workunit.client.1.smithi135.stdout:6/702: sync 2024-03-30T04:28:28.970 INFO:tasks.workunit.client.1.smithi135.stdout:7/740: creat d4/db/d1a/d5c/d2c/d51/d56/ff5 x:0 0 0 2024-03-30T04:28:28.973 INFO:tasks.workunit.client.1.smithi135.stdout:1/653: dread d16/d5a/d5f/d3d/d8d/dc9/fd5 [0,4194304] 0 2024-03-30T04:28:28.975 INFO:tasks.workunit.client.1.smithi135.stdout:6/703: creat d2/dd/d14/d2b/d86/d55/fe2 x:0 0 0 2024-03-30T04:28:28.977 INFO:tasks.workunit.client.1.smithi135.stdout:7/741: dread d4/db/d1a/d3d/ddc/d5a/dae/fbe [0,4194304] 0 2024-03-30T04:28:28.978 INFO:tasks.workunit.client.1.smithi135.stdout:1/654: creat d16/d29/d61/fd7 x:0 0 0 2024-03-30T04:28:28.980 INFO:tasks.workunit.client.1.smithi135.stdout:6/704: creat d2/dd/d14/d30/d43/dcd/fe3 x:0 0 0 2024-03-30T04:28:28.981 INFO:tasks.workunit.client.1.smithi135.stdout:7/742: write d4/db/d1a/d5c/d82/fa2 [1751946,121269] 0 2024-03-30T04:28:28.985 INFO:tasks.workunit.client.1.smithi135.stdout:7/743: unlink d4/db/d1a/d5c/d5d/c6a 0 2024-03-30T04:28:28.987 INFO:tasks.workunit.client.1.smithi135.stdout:6/705: mknod d2/dd/d14/d2b/d86/ce4 0 2024-03-30T04:28:28.989 INFO:tasks.workunit.client.1.smithi135.stdout:7/744: creat d4/db/d1a/d3d/d96/ff6 x:0 0 0 2024-03-30T04:28:28.991 INFO:tasks.workunit.client.1.smithi135.stdout:1/655: truncate d16/d29/d2c/d43/d50/f5b 5161977 0 2024-03-30T04:28:28.995 INFO:tasks.workunit.client.1.smithi135.stdout:1/656: mknod d16/d5a/d5f/d3d/db4/cd8 0 2024-03-30T04:28:28.998 INFO:tasks.workunit.client.1.smithi135.stdout:7/745: dread d4/db/d1a/d3d/ddc/d5a/dae/fd3 [0,4194304] 0 2024-03-30T04:28:29.000 INFO:tasks.workunit.client.1.smithi135.stdout:1/657: creat d16/d5a/d5f/d3a/fd9 x:0 0 0 2024-03-30T04:28:29.001 INFO:tasks.workunit.client.1.smithi135.stdout:7/746: fsync d4/db/d1a/d5c/d2c/d51/d56/ff5 0 2024-03-30T04:28:29.002 INFO:tasks.workunit.client.1.smithi135.stdout:1/658: write d16/d29/fbb [663153,125281] 0 2024-03-30T04:28:29.003 INFO:tasks.workunit.client.1.smithi135.stdout:6/706: dwrite d2/f4 [0,4194304] 0 2024-03-30T04:28:29.005 INFO:tasks.workunit.client.1.smithi135.stdout:6/707: fdatasync d2/dd/d14/d2b/d86/d55/fe2 0 2024-03-30T04:28:29.007 INFO:tasks.workunit.client.1.smithi135.stdout:1/659: mknod d16/d29/d2c/dce/cda 0 2024-03-30T04:28:29.009 INFO:tasks.workunit.client.1.smithi135.stdout:1/660: chown d16/d5a/l71 5 1 2024-03-30T04:28:29.009 INFO:tasks.workunit.client.1.smithi135.stdout:6/708: write d2/dd/d14/d38/f9f [721083,112917] 0 2024-03-30T04:28:29.011 INFO:tasks.workunit.client.1.smithi135.stdout:6/709: readlink d2/dd/d18/d87/ld7 0 2024-03-30T04:28:29.012 INFO:tasks.workunit.client.1.smithi135.stdout:6/710: chown d2/c71 5904501 1 2024-03-30T04:28:29.014 INFO:tasks.workunit.client.1.smithi135.stdout:1/661: symlink d16/d5a/dad/ldb 0 2024-03-30T04:28:29.016 INFO:tasks.workunit.client.1.smithi135.stdout:1/662: dread - d16/d29/fa9 zero size 2024-03-30T04:28:29.021 INFO:tasks.workunit.client.1.smithi135.stdout:1/663: creat d16/d29/d61/fdc x:0 0 0 2024-03-30T04:28:29.024 INFO:tasks.workunit.client.1.smithi135.stdout:7/747: dwrite d4/db/d1a/d5c/d2c/d51/d56/d81/fec [0,4194304] 0 2024-03-30T04:28:29.026 INFO:tasks.workunit.client.1.smithi135.stdout:1/664: creat d16/d29/d32/fdd x:0 0 0 2024-03-30T04:28:29.030 INFO:tasks.workunit.client.1.smithi135.stdout:1/665: write d16/d29/d32/dcc/fd4 [567792,55803] 0 2024-03-30T04:28:29.034 INFO:tasks.workunit.client.1.smithi135.stdout:7/748: symlink d4/db/lf7 0 2024-03-30T04:28:29.035 INFO:tasks.workunit.client.1.smithi135.stdout:1/666: fdatasync d16/d5a/dad/d77/fb6 0 2024-03-30T04:28:29.036 INFO:tasks.workunit.client.1.smithi135.stdout:6/711: dwrite d2/d36/fd4 [0,4194304] 0 2024-03-30T04:28:29.037 INFO:tasks.workunit.client.1.smithi135.stdout:7/749: write d4/db/d1a/d3d/ddc/f2b [2742080,65008] 0 2024-03-30T04:28:29.044 INFO:tasks.workunit.client.1.smithi135.stdout:7/750: mknod d4/db/d1a/d3d/ddc/db0/de1/cf8 0 2024-03-30T04:28:29.044 INFO:tasks.workunit.client.1.smithi135.stdout:6/712: creat d2/dd/d14/d2b/d86/fe5 x:0 0 0 2024-03-30T04:28:29.050 INFO:tasks.workunit.client.1.smithi135.stdout:7/751: rename d4/db/d1a/d4b/d40/d6e/fc6 to d4/ff9 0 2024-03-30T04:28:29.052 INFO:tasks.workunit.client.1.smithi135.stdout:1/667: dwrite f3 [0,4194304] 0 2024-03-30T04:28:29.063 INFO:tasks.workunit.client.1.smithi135.stdout:1/668: creat d16/d29/d2c/fde x:0 0 0 2024-03-30T04:28:29.066 INFO:tasks.workunit.client.1.smithi135.stdout:6/713: dwrite d2/dd/d14/d38/f9a [0,4194304] 0 2024-03-30T04:28:29.068 INFO:tasks.workunit.client.1.smithi135.stdout:6/714: chown d2/d36/d76/l8c 1395 1 2024-03-30T04:28:29.071 INFO:tasks.workunit.client.1.smithi135.stdout:1/669: mknod d16/d5a/d5f/d3d/d8d/dc9/da4/d99/d9a/cdf 0 2024-03-30T04:28:29.072 INFO:tasks.workunit.client.1.smithi135.stdout:6/715: readlink d2/d2e/l2f 0 2024-03-30T04:28:29.074 INFO:tasks.workunit.client.1.smithi135.stdout:3/620: sync 2024-03-30T04:28:29.075 INFO:tasks.workunit.client.1.smithi135.stdout:2/531: sync 2024-03-30T04:28:29.080 INFO:tasks.workunit.client.1.smithi135.stdout:1/670: creat d16/d29/d2c/d43/d50/fe0 x:0 0 0 2024-03-30T04:28:29.082 INFO:tasks.workunit.client.1.smithi135.stdout:6/716: getdents d2/dd/d40/db4 0 2024-03-30T04:28:29.083 INFO:tasks.workunit.client.1.smithi135.stdout:1/671: rmdir d16/d5a/d5f 39 2024-03-30T04:28:29.087 INFO:tasks.workunit.client.1.smithi135.stdout:6/717: read d2/dd/d14/d2b/d86/fd1 [27617,67948] 0 2024-03-30T04:28:29.088 INFO:tasks.workunit.client.1.smithi135.stdout:2/532: rename d4/d32/d12/c41 to d4/da/cb2 0 2024-03-30T04:28:29.092 INFO:tasks.workunit.client.1.smithi135.stdout:2/533: mkdir d4/db3 0 2024-03-30T04:28:29.093 INFO:tasks.workunit.client.1.smithi135.stdout:3/621: dwrite d3/d10/d16/d23/f7f [0,4194304] 0 2024-03-30T04:28:29.093 INFO:tasks.workunit.client.1.smithi135.stdout:6/718: mknod d2/dd/d14/d38/dde/ce6 0 2024-03-30T04:28:29.095 INFO:tasks.workunit.client.1.smithi135.stdout:7/752: dread d4/db/f26 [0,4194304] 0 2024-03-30T04:28:29.095 INFO:tasks.workunit.client.1.smithi135.stdout:3/622: chown d3/d50/d46/f73 1944 1 2024-03-30T04:28:29.099 INFO:tasks.workunit.client.1.smithi135.stdout:2/534: mkdir d4/d32/d2e/d67/db4 0 2024-03-30T04:28:29.103 INFO:tasks.workunit.client.1.smithi135.stdout:7/753: mknod d4/db/d1a/d5c/d2c/d2e/db9/d75/cfa 0 2024-03-30T04:28:29.107 INFO:tasks.workunit.client.1.smithi135.stdout:2/535: mkdir d4/d32/d43/db5 0 2024-03-30T04:28:29.108 INFO:tasks.workunit.client.1.smithi135.stdout:7/754: mkdir d4/db/d1a/d90/dea/dfb 0 2024-03-30T04:28:29.114 INFO:tasks.workunit.client.1.smithi135.stdout:2/536: link d4/d32/f13 d4/d32/d12/fb6 0 2024-03-30T04:28:29.120 INFO:tasks.workunit.client.1.smithi135.stdout:2/537: getdents d4/d32/d12/d96 0 2024-03-30T04:28:29.128 INFO:tasks.workunit.client.1.smithi135.stdout:2/538: truncate d4/da/d8d/f95 26710 0 2024-03-30T04:28:29.131 INFO:tasks.workunit.client.1.smithi135.stdout:3/623: dread d3/d50/d22/d33/f61 [0,4194304] 0 2024-03-30T04:28:29.133 INFO:tasks.workunit.client.1.smithi135.stdout:6/719: dread d2/d36/f6e [0,4194304] 0 2024-03-30T04:28:29.135 INFO:tasks.workunit.client.1.smithi135.stdout:2/539: read d4/d6/f1a [4312161,127594] 0 2024-03-30T04:28:29.135 INFO:tasks.workunit.client.1.smithi135.stdout:7/755: dwrite d4/db/d79/dbc/fa5 [0,4194304] 0 2024-03-30T04:28:29.140 INFO:tasks.workunit.client.1.smithi135.stdout:2/540: fsync d4/d32/d43/f6b 0 2024-03-30T04:28:29.143 INFO:tasks.workunit.client.1.smithi135.stdout:3/624: getdents d3/d54/dab 0 2024-03-30T04:28:29.146 INFO:tasks.workunit.client.1.smithi135.stdout:3/625: write d3/d54/dab/fd0 [178728,117973] 0 2024-03-30T04:28:29.150 INFO:tasks.workunit.client.1.smithi135.stdout:3/626: write d3/d50/fa [2727849,30028] 0 2024-03-30T04:28:29.153 INFO:tasks.workunit.client.1.smithi135.stdout:3/627: truncate d3/d50/f12 4253724 0 2024-03-30T04:28:29.154 INFO:tasks.workunit.client.1.smithi135.stdout:6/720: mknod d2/dd/d14/d30/d43/dcd/dc5/dd2/ce7 0 2024-03-30T04:28:29.155 INFO:tasks.workunit.client.1.smithi135.stdout:2/541: getdents d4/d32/d2e/d67/db4 0 2024-03-30T04:28:29.158 INFO:tasks.workunit.client.1.smithi135.stdout:3/628: symlink d3/d50/d22/d8b/d91/ld4 0 2024-03-30T04:28:29.159 INFO:tasks.workunit.client.1.smithi135.stdout:3/629: fdatasync d3/d50/d46/f73 0 2024-03-30T04:28:29.162 INFO:tasks.workunit.client.1.smithi135.stdout:9/629: sync 2024-03-30T04:28:29.162 INFO:tasks.workunit.client.1.smithi135.stdout:8/605: sync 2024-03-30T04:28:29.162 INFO:tasks.workunit.client.1.smithi135.stdout:5/600: sync 2024-03-30T04:28:29.162 INFO:tasks.workunit.client.1.smithi135.stdout:4/625: sync 2024-03-30T04:28:29.162 INFO:tasks.workunit.client.1.smithi135.stdout:2/542: creat d4/d32/d2e/d2f/fb7 x:0 0 0 2024-03-30T04:28:29.165 INFO:tasks.workunit.client.1.smithi135.stdout:0/756: sync 2024-03-30T04:28:29.167 INFO:tasks.workunit.client.1.smithi135.stdout:5/601: read d1/d3/d2b/f2f [3116365,116024] 0 2024-03-30T04:28:29.168 INFO:tasks.workunit.client.1.smithi135.stdout:9/630: rmdir d7/d41/db4 39 2024-03-30T04:28:29.168 INFO:tasks.workunit.client.1.smithi135.stdout:7/756: write d4/db/d1a/d3d/ddc/d5a/dae/fbe [1643124,7803] 0 2024-03-30T04:28:29.170 INFO:tasks.workunit.client.1.smithi135.stdout:8/606: mknod d6/d8/d14/cbf 0 2024-03-30T04:28:29.173 INFO:tasks.workunit.client.1.smithi135.stdout:0/757: readlink d0/d13/d53/d7f/d83/lc0 0 2024-03-30T04:28:29.173 INFO:tasks.workunit.client.1.smithi135.stdout:6/721: getdents d2/dd/d14/d38 0 2024-03-30T04:28:29.176 INFO:tasks.workunit.client.1.smithi135.stdout:0/758: creat d0/db5/ff7 x:0 0 0 2024-03-30T04:28:29.177 INFO:tasks.workunit.client.1.smithi135.stdout:8/607: write d6/d8/d14/f29 [3272949,47955] 0 2024-03-30T04:28:29.179 INFO:tasks.workunit.client.1.smithi135.stdout:3/630: truncate d3/f24 845963 0 2024-03-30T04:28:29.181 INFO:tasks.workunit.client.1.smithi135.stdout:0/759: chown d0/d1/d16/f85 8123023 1 2024-03-30T04:28:29.181 INFO:tasks.workunit.client.1.smithi135.stdout:9/631: symlink d7/d90/ld7 0 2024-03-30T04:28:29.186 INFO:tasks.workunit.client.1.smithi135.stdout:6/722: symlink d2/dd/d14/d30/d43/dcd/dad/dae/le8 0 2024-03-30T04:28:29.186 INFO:tasks.workunit.client.1.smithi135.stdout:9/632: dread - d7/d41/d72/f7d zero size 2024-03-30T04:28:29.190 INFO:tasks.workunit.client.1.smithi135.stdout:2/543: dwrite d4/d32/d12/d45/f8e [0,4194304] 0 2024-03-30T04:28:29.192 INFO:tasks.workunit.client.1.smithi135.stdout:8/608: symlink d6/d8/d15/d54/d8e/d72/d73/d9f/lc0 0 2024-03-30T04:28:29.197 INFO:tasks.workunit.client.1.smithi135.stdout:7/757: truncate d4/db/d1a/d5c/d2c/d51/fde 2925725 0 2024-03-30T04:28:29.199 INFO:tasks.workunit.client.1.smithi135.stdout:6/723: read d2/dd/f17 [1241776,37289] 0 2024-03-30T04:28:29.200 INFO:tasks.workunit.client.1.smithi135.stdout:7/758: write d4/f8c [1036132,19091] 0 2024-03-30T04:28:29.202 INFO:tasks.workunit.client.1.smithi135.stdout:0/760: mknod d0/d5/d9/d23/d2a/dc4/cf8 0 2024-03-30T04:28:29.207 INFO:tasks.workunit.client.1.smithi135.stdout:6/724: write d2/dd/d14/d2b/d86/f3c [3828091,8085] 0 2024-03-30T04:28:29.209 INFO:tasks.workunit.client.1.smithi135.stdout:0/761: mkdir d0/d1/df9 0 2024-03-30T04:28:29.213 INFO:tasks.workunit.client.1.smithi135.stdout:3/631: write d3/d50/f12 [1707364,58395] 0 2024-03-30T04:28:29.214 INFO:tasks.workunit.client.1.smithi135.stdout:2/544: write d4/d32/d46/f83 [1021964,62314] 0 2024-03-30T04:28:29.216 INFO:tasks.workunit.client.1.smithi135.stdout:3/632: readlink d3/d6e/l70 0 2024-03-30T04:28:29.225 INFO:tasks.workunit.client.1.smithi135.stdout:8/609: write d6/d8/d15/d1f/f4c [4364175,129855] 0 2024-03-30T04:28:29.228 INFO:tasks.workunit.client.1.smithi135.stdout:3/633: chown d3/d50/d22/d33/l5a 766 1 2024-03-30T04:28:29.229 INFO:tasks.workunit.client.1.smithi135.stdout:2/545: write d4/d32/d2e/f8c [47851,19230] 0 2024-03-30T04:28:29.231 INFO:tasks.workunit.client.1.smithi135.stdout:3/634: stat d3/d50/d46/f60 0 2024-03-30T04:28:29.231 INFO:tasks.workunit.client.1.smithi135.stdout:9/633: dwrite d7/d4f/d5c/d63/dd2/db1/dbc/f97 [0,4194304] 0 2024-03-30T04:28:29.232 INFO:tasks.workunit.client.1.smithi135.stdout:0/762: getdents d0/db5 0 2024-03-30T04:28:29.238 INFO:tasks.workunit.client.1.smithi135.stdout:2/546: chown d4/d32/d2e/d9f 44065 1 2024-03-30T04:28:29.239 INFO:tasks.workunit.client.1.smithi135.stdout:2/547: chown d4/d32/d43 25725590 1 2024-03-30T04:28:29.239 INFO:tasks.workunit.client.1.smithi135.stdout:3/635: write d3/d50/f36 [4729500,114416] 0 2024-03-30T04:28:29.242 INFO:tasks.workunit.client.1.smithi135.stdout:7/759: dwrite d4/db/d1a/d5c/d2c/d2e/db9/d75/fe3 [0,4194304] 0 2024-03-30T04:28:29.249 INFO:tasks.workunit.client.1.smithi135.stdout:6/725: dwrite d2/dd/d14/d2b/d86/fc3 [0,4194304] 0 2024-03-30T04:28:29.250 INFO:tasks.workunit.client.1.smithi135.stdout:3/636: dread - d3/d10/d16/d41/f6d zero size 2024-03-30T04:28:29.254 INFO:tasks.workunit.client.1.smithi135.stdout:0/763: rename d0/d5/d9/fd to d0/d5/d9/d23/d58/d3b/ffa 0 2024-03-30T04:28:29.256 INFO:tasks.workunit.client.1.smithi135.stdout:6/726: chown d2/dd/d14/d30/d43/dcd 31 1 2024-03-30T04:28:29.257 INFO:tasks.workunit.client.1.smithi135.stdout:7/760: mkdir d4/db/d1a/d5c/d59/d76/de4/dfc 0 2024-03-30T04:28:29.264 INFO:tasks.workunit.client.1.smithi135.stdout:8/610: dwrite d6/d8/d14/f63 [4194304,4194304] 0 2024-03-30T04:28:29.271 INFO:tasks.workunit.client.1.smithi135.stdout:7/761: rename d4/db/d79/da9 to d4/db/d1a/d3d/ddc/db0/dfd 0 2024-03-30T04:28:29.276 INFO:tasks.workunit.client.1.smithi135.stdout:8/611: mkdir d6/d8/d15/d54/d7e/daf/dc1 0 2024-03-30T04:28:29.282 INFO:tasks.workunit.client.1.smithi135.stdout:7/762: unlink d4/db/d1a/d90/fa3 0 2024-03-30T04:28:29.284 INFO:tasks.workunit.client.1.smithi135.stdout:8/612: mknod d6/d8/d15/d1f/cc2 0 2024-03-30T04:28:29.286 INFO:tasks.workunit.client.1.smithi135.stdout:2/548: dread d4/d6/f1a [0,4194304] 0 2024-03-30T04:28:29.288 INFO:tasks.workunit.client.1.smithi135.stdout:8/613: write d6/f50 [518846,88794] 0 2024-03-30T04:28:29.290 INFO:tasks.workunit.client.1.smithi135.stdout:6/727: dwrite d2/d68/f83 [0,4194304] 0 2024-03-30T04:28:29.296 INFO:tasks.workunit.client.1.smithi135.stdout:7/763: mkdir d4/db/d1a/d5c/d2c/d2e/dfe 0 2024-03-30T04:28:29.298 INFO:tasks.workunit.client.1.smithi135.stdout:3/637: truncate d3/d10/d16/f43 1841375 0 2024-03-30T04:28:29.301 INFO:tasks.workunit.client.1.smithi135.stdout:3/638: chown d3/d50/d22/d8b/d91/f97 444 1 2024-03-30T04:28:29.301 INFO:tasks.workunit.client.1.smithi135.stdout:2/549: mknod d4/d32/d43/db5/cb8 0 2024-03-30T04:28:29.304 INFO:tasks.workunit.client.1.smithi135.stdout:6/728: symlink d2/dd/d14/d30/d43/dcd/dbe/le9 0 2024-03-30T04:28:29.306 INFO:tasks.workunit.client.1.smithi135.stdout:8/614: mknod d6/d8/d15/d54/d8e/d72/cc3 0 2024-03-30T04:28:29.307 INFO:tasks.workunit.client.1.smithi135.stdout:7/764: mknod d4/db/d1a/d5c/d2c/cff 0 2024-03-30T04:28:29.312 INFO:tasks.workunit.client.1.smithi135.stdout:3/639: creat d3/d50/d22/d33/d62/fd5 x:0 0 0 2024-03-30T04:28:29.316 INFO:tasks.workunit.client.1.smithi135.stdout:1/672: sync 2024-03-30T04:28:29.321 INFO:tasks.workunit.client.1.smithi135.stdout:3/640: rename d3/d50/d22/d8b to d3/d10/d9c/dbe/dd6 0 2024-03-30T04:28:29.322 INFO:tasks.workunit.client.1.smithi135.stdout:3/641: fdatasync d3/db/fc6 0 2024-03-30T04:28:29.323 INFO:tasks.workunit.client.1.smithi135.stdout:1/673: mknod d16/d29/d2c/d3c/ce1 0 2024-03-30T04:28:29.324 INFO:tasks.workunit.client.1.smithi135.stdout:0/764: truncate d0/d5/d10/f2c 2331947 0 2024-03-30T04:28:29.335 INFO:tasks.workunit.client.1.smithi135.stdout:3/642: mkdir d3/d10/d9c/dbe/dd6/dd7 0 2024-03-30T04:28:29.335 INFO:tasks.workunit.client.1.smithi135.stdout:1/674: unlink d16/d29/d2c/l36 0 2024-03-30T04:28:29.336 INFO:tasks.workunit.client.1.smithi135.stdout:6/729: dwrite d2/dd/d14/d2b/d86/fe5 [0,4194304] 0 2024-03-30T04:28:29.345 INFO:tasks.workunit.client.1.smithi135.stdout:1/675: write d16/d29/d32/dcc/f7c [5866075,83083] 0 2024-03-30T04:28:29.348 INFO:tasks.workunit.client.1.smithi135.stdout:7/765: dread d4/db/d1a/d5c/d2c/f39 [0,4194304] 0 2024-03-30T04:28:29.355 INFO:tasks.workunit.client.1.smithi135.stdout:0/765: link d0/d19/d2d/d45/c7a d0/d5/d9/d23/d58/d63/d77/cfb 0 2024-03-30T04:28:29.355 INFO:tasks.workunit.client.1.smithi135.stdout:8/615: dwrite d6/d8/d15/d1f/f44 [4194304,4194304] 0 2024-03-30T04:28:29.357 INFO:tasks.workunit.client.1.smithi135.stdout:2/550: dwrite d4/d32/d2e/f5c [0,4194304] 0 2024-03-30T04:28:29.358 INFO:tasks.workunit.client.1.smithi135.stdout:8/616: chown d6/d8/d15/d54/d8e 5 1 2024-03-30T04:28:29.366 INFO:tasks.workunit.client.1.smithi135.stdout:3/643: dwrite d3/d10/d16/d23/d6a/d94/fba [0,4194304] 0 2024-03-30T04:28:29.369 INFO:tasks.workunit.client.1.smithi135.stdout:1/676: symlink d16/d29/d32/le2 0 2024-03-30T04:28:29.380 INFO:tasks.workunit.client.1.smithi135.stdout:8/617: rmdir d6/d8/d15/d54/d8e/d72/d73/d9f 39 2024-03-30T04:28:29.381 INFO:tasks.workunit.client.1.smithi135.stdout:6/730: creat d2/fea x:0 0 0 2024-03-30T04:28:29.381 INFO:tasks.workunit.client.1.smithi135.stdout:7/766: mkdir d4/db/d1a/d5c/d2c/d2e/db9/df4/d100 0 2024-03-30T04:28:29.382 INFO:tasks.workunit.client.1.smithi135.stdout:0/766: write d0/d13/d1e/f78 [4525587,49863] 0 2024-03-30T04:28:29.389 INFO:tasks.workunit.client.1.smithi135.stdout:0/767: write d0/d5/d9/d23/d58/d63/fcf [745190,91340] 0 2024-03-30T04:28:29.402 INFO:tasks.workunit.client.1.smithi135.stdout:3/644: symlink d3/d10/d16/d23/d6a/d94/ld8 0 2024-03-30T04:28:29.404 INFO:tasks.workunit.client.1.smithi135.stdout:8/618: rename d6/d8/f42 to d6/d8/d15/d1f/fc4 0 2024-03-30T04:28:29.404 INFO:tasks.workunit.client.1.smithi135.stdout:3/645: chown d3/d10/d16/d23/c7a 1352 1 2024-03-30T04:28:29.406 INFO:tasks.workunit.client.1.smithi135.stdout:6/731: symlink d2/d68/d81/leb 0 2024-03-30T04:28:29.409 INFO:tasks.workunit.client.1.smithi135.stdout:2/551: creat d4/d6/fb9 x:0 0 0 2024-03-30T04:28:29.411 INFO:tasks.workunit.client.1.smithi135.stdout:3/646: unlink d3/d50/c3f 0 2024-03-30T04:28:29.413 INFO:tasks.workunit.client.1.smithi135.stdout:2/552: creat d4/d32/d46/fba x:0 0 0 2024-03-30T04:28:29.414 INFO:tasks.workunit.client.1.smithi135.stdout:3/647: write d3/d6e/fd2 [905204,66245] 0 2024-03-30T04:28:29.415 INFO:tasks.workunit.client.1.smithi135.stdout:6/732: symlink d2/dd/d14/d30/lec 0 2024-03-30T04:28:29.419 INFO:tasks.workunit.client.1.smithi135.stdout:8/619: dread d6/d8/d14/f4e [0,4194304] 0 2024-03-30T04:28:29.421 INFO:tasks.workunit.client.1.smithi135.stdout:0/768: rename d0/d5/d9/d23/d58 to d0/d34/dfc 0 2024-03-30T04:28:29.422 INFO:tasks.workunit.client.1.smithi135.stdout:2/553: symlink d4/d32/d2e/d67/lbb 0 2024-03-30T04:28:29.423 INFO:tasks.workunit.client.1.smithi135.stdout:6/733: fsync d2/d2e/f37 0 2024-03-30T04:28:29.425 INFO:tasks.workunit.client.1.smithi135.stdout:7/767: dread d4/db/d1a/f66 [0,4194304] 0 2024-03-30T04:28:29.429 INFO:tasks.workunit.client.1.smithi135.stdout:7/768: stat d4/db/d1a/d4b/l21 0 2024-03-30T04:28:29.433 INFO:tasks.workunit.client.1.smithi135.stdout:6/734: fdatasync d2/dd/d14/f15 0 2024-03-30T04:28:29.436 INFO:tasks.workunit.client.1.smithi135.stdout:1/677: dread d16/d29/d2c/d43/d50/f5b [0,4194304] 0 2024-03-30T04:28:29.443 INFO:tasks.workunit.client.1.smithi135.stdout:0/769: creat d0/d34/ffd x:0 0 0 2024-03-30T04:28:29.445 INFO:tasks.workunit.client.1.smithi135.stdout:2/554: dwrite d4/d6/f9c [0,4194304] 0 2024-03-30T04:28:29.449 INFO:tasks.workunit.client.1.smithi135.stdout:0/770: creat d0/d1/d16/dd5/ffe x:0 0 0 2024-03-30T04:28:29.451 INFO:tasks.workunit.client.1.smithi135.stdout:0/771: readlink d0/d5/d9/d23/d2a/d60/l40 0 2024-03-30T04:28:29.453 INFO:tasks.workunit.client.1.smithi135.stdout:3/648: truncate d3/d50/d46/f58 857720 0 2024-03-30T04:28:29.454 INFO:tasks.workunit.client.1.smithi135.stdout:8/620: write d6/d8/d15/d54/d8e/faa [340934,63516] 0 2024-03-30T04:28:29.457 INFO:tasks.workunit.client.1.smithi135.stdout:2/555: mkdir d4/d32/d46/dbc 0 2024-03-30T04:28:29.458 INFO:tasks.workunit.client.1.smithi135.stdout:0/772: rmdir d0/d34/dfc/d63/d77/d8b/dcc 39 2024-03-30T04:28:29.462 INFO:tasks.workunit.client.1.smithi135.stdout:7/769: rename d4/db/d1a/d5c/d2c/d51/da8/l97 to d4/db/d1a/d5c/d2c/l101 0 2024-03-30T04:28:29.464 INFO:tasks.workunit.client.1.smithi135.stdout:4/626: sync 2024-03-30T04:28:29.465 INFO:tasks.workunit.client.1.smithi135.stdout:8/621: fdatasync d6/d8/d15/d79/fb1 0 2024-03-30T04:28:29.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:29 smithi123 ceph-mon[35245]: Upgrade: Updating mgr.smithi123.xgksqq 2024-03-30T04:28:29.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:29 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:28:29.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:29 smithi123 ceph-mon[35245]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi123.xgksqq", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T04:28:29.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:29 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi123.xgksqq", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T04:28:29.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:29 smithi123 ceph-mon[35245]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T04:28:29.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:29 smithi123 ceph-mon[35245]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:28:29.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:29 smithi123 ceph-mon[35245]: Deploying daemon mgr.smithi123.xgksqq on smithi123 2024-03-30T04:28:29.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:29 smithi123 ceph-mon[35245]: pgmap v48: 65 pgs: 65 active+clean; 3.0 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 89 MiB/s rd, 206 MiB/s wr, 459 op/s 2024-03-30T04:28:29.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:29 smithi123 ceph-mon[35245]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:28:29.471 INFO:tasks.workunit.client.1.smithi135.stdout:6/735: truncate d2/dd/d14/d2b/d86/d55/f7b 802465 0 2024-03-30T04:28:29.473 INFO:tasks.workunit.client.1.smithi135.stdout:3/649: creat d3/d10/d16/d41/fd9 x:0 0 0 2024-03-30T04:28:29.475 INFO:tasks.workunit.client.1.smithi135.stdout:1/678: dwrite d16/d5a/d5f/d3d/d8d/dc9/da4/d99/f4c [0,4194304] 0 2024-03-30T04:28:29.479 INFO:tasks.workunit.client.1.smithi135.stdout:8/622: symlink d6/d8/d15/d79/lc5 0 2024-03-30T04:28:29.479 INFO:tasks.workunit.client.1.smithi135.stdout:2/556: truncate d4/d32/d43/f92 2253907 0 2024-03-30T04:28:29.480 INFO:tasks.workunit.client.1.smithi135.stdout:4/627: symlink d10/d1a/d42/ldb 0 2024-03-30T04:28:29.480 INFO:tasks.workunit.client.1.smithi135.stdout:3/650: symlink d3/db/lda 0 2024-03-30T04:28:29.484 INFO:tasks.workunit.client.1.smithi135.stdout:1/679: creat d16/d5a/dad/d77/fe3 x:0 0 0 2024-03-30T04:28:29.487 INFO:tasks.workunit.client.1.smithi135.stdout:6/736: rename d2/dd/d14/d30/d43/dcd/dad/cc9 to d2/dd/d14/d38/ced 0 2024-03-30T04:28:29.488 INFO:tasks.workunit.client.1.smithi135.stdout:3/651: chown d3/d50/d22/l2f 6 1 2024-03-30T04:28:29.489 INFO:tasks.workunit.client.1.smithi135.stdout:8/623: chown d6/d8/d15/d54/f5d 993114 1 2024-03-30T04:28:29.493 INFO:tasks.workunit.client.1.smithi135.stdout:2/557: creat d4/d32/d2e/d2f/fbd x:0 0 0 2024-03-30T04:28:29.495 INFO:tasks.workunit.client.1.smithi135.stdout:0/773: dwrite d0/d13/d1e/d3a/dc2/ff3 [0,4194304] 0 2024-03-30T04:28:29.495 INFO:tasks.workunit.client.1.smithi135.stdout:1/680: creat d16/d29/d2c/d43/d50/fe4 x:0 0 0 2024-03-30T04:28:29.497 INFO:tasks.workunit.client.1.smithi135.stdout:8/624: chown d6/d8/d15/d54/d4f/c55 247884 1 2024-03-30T04:28:29.499 INFO:tasks.workunit.client.1.smithi135.stdout:9/634: sync 2024-03-30T04:28:29.500 INFO:tasks.workunit.client.1.smithi135.stdout:5/602: sync 2024-03-30T04:28:29.507 INFO:tasks.workunit.client.1.smithi135.stdout:8/625: fdatasync d6/d23/f82 0 2024-03-30T04:28:29.508 INFO:tasks.workunit.client.1.smithi135.stdout:4/628: write d10/d1a/d42/f63 [2549669,123948] 0 2024-03-30T04:28:29.513 INFO:tasks.workunit.client.1.smithi135.stdout:3/652: write d3/d10/d16/d23/f30 [1130550,24899] 0 2024-03-30T04:28:29.514 INFO:tasks.workunit.client.1.smithi135.stdout:2/558: mkdir d4/d32/d46/dbc/dbe 0 2024-03-30T04:28:29.518 INFO:tasks.workunit.client.1.smithi135.stdout:4/629: chown d10/d1a/d42/d46/f98 1114271 1 2024-03-30T04:28:29.518 INFO:tasks.workunit.client.1.smithi135.stdout:9/635: rename d7/d4f/d5c/da3 to d7/d4f/d5c/d63/dd2/db1/dbc/dd8 0 2024-03-30T04:28:29.522 INFO:tasks.workunit.client.1.smithi135.stdout:9/636: write d7/d4f/d89/fa6 [342258,122737] 0 2024-03-30T04:28:29.524 INFO:tasks.workunit.client.1.smithi135.stdout:5/603: link d1/da/f1b d1/d7/fb1 0 2024-03-30T04:28:29.524 INFO:tasks.workunit.client.1.smithi135.stdout:7/770: dread d4/f11 [0,4194304] 0 2024-03-30T04:28:29.527 INFO:tasks.workunit.client.1.smithi135.stdout:6/737: truncate d2/f99 2399942 0 2024-03-30T04:28:29.528 INFO:tasks.workunit.client.1.smithi135.stdout:3/653: mknod d3/d50/d46/cdb 0 2024-03-30T04:28:29.533 INFO:tasks.workunit.client.1.smithi135.stdout:2/559: symlink d4/d32/daf/lbf 0 2024-03-30T04:28:29.535 INFO:tasks.workunit.client.1.smithi135.stdout:0/774: getdents d0/d13/d1e/d3a/dc2/dd1 0 2024-03-30T04:28:29.538 INFO:tasks.workunit.client.1.smithi135.stdout:3/654: mknod d3/d6e/cdc 0 2024-03-30T04:28:29.542 INFO:tasks.workunit.client.1.smithi135.stdout:1/681: dwrite d16/d5a/dad/fc2 [0,4194304] 0 2024-03-30T04:28:29.547 INFO:tasks.workunit.client.1.smithi135.stdout:5/604: rename d1/d3/d2b/d54/d5e/d5c to d1/da/d17/d70/db2 0 2024-03-30T04:28:29.548 INFO:tasks.workunit.client.1.smithi135.stdout:6/738: creat d2/dd/d40/db4/fee x:0 0 0 2024-03-30T04:28:29.553 INFO:tasks.workunit.client.1.smithi135.stdout:2/560: chown d4/c7b 153374373 1 2024-03-30T04:28:29.555 INFO:tasks.workunit.client.1.smithi135.stdout:7/771: dwrite d4/db/d1a/f24 [0,4194304] 0 2024-03-30T04:28:29.556 INFO:tasks.workunit.client.1.smithi135.stdout:4/630: truncate d10/f57 788986 0 2024-03-30T04:28:29.557 INFO:tasks.workunit.client.1.smithi135.stdout:9/637: truncate d7/d90/fa0 3119446 0 2024-03-30T04:28:29.558 INFO:tasks.workunit.client.1.smithi135.stdout:1/682: readlink d16/d29/l2d 0 2024-03-30T04:28:29.562 INFO:tasks.workunit.client.1.smithi135.stdout:8/626: dwrite d6/d8/d15/d54/f6d [0,4194304] 0 2024-03-30T04:28:29.563 INFO:tasks.workunit.client.1.smithi135.stdout:0/775: creat d0/d5/d42/d9f/df5/fff x:0 0 0 2024-03-30T04:28:29.563 INFO:tasks.workunit.client.1.smithi135.stdout:4/631: readlink d10/d1a/d3d/l80 0 2024-03-30T04:28:29.565 INFO:tasks.workunit.client.1.smithi135.stdout:3/655: rename d3/d10/l96 to d3/d10/d9c/dbe/ldd 0 2024-03-30T04:28:29.575 INFO:tasks.workunit.client.1.smithi135.stdout:1/683: write d16/d29/d2c/d3c/fc0 [90968,88976] 0 2024-03-30T04:28:29.575 INFO:tasks.workunit.client.1.smithi135.stdout:7/772: chown d4/db/d1a/d5c/d2c/ldd 15940 1 2024-03-30T04:28:29.578 INFO:tasks.workunit.client.1.smithi135.stdout:4/632: unlink l2 0 2024-03-30T04:28:29.579 INFO:tasks.workunit.client.1.smithi135.stdout:1/684: stat d16/d5a/d5f/f4e 0 2024-03-30T04:28:29.583 INFO:tasks.workunit.client.1.smithi135.stdout:0/776: link d0/d5/d9/fb d0/d1/d16/dd5/f100 0 2024-03-30T04:28:29.584 INFO:tasks.workunit.client.1.smithi135.stdout:8/627: mkdir d6/d8/d15/d54/d8e/d72/d8b/db2/dc6 0 2024-03-30T04:28:29.587 INFO:tasks.workunit.client.1.smithi135.stdout:5/605: dwrite d1/d3/d80/f93 [0,4194304] 0 2024-03-30T04:28:29.588 INFO:tasks.workunit.client.1.smithi135.stdout:2/561: truncate d4/d32/d43/f92 3223285 0 2024-03-30T04:28:29.594 INFO:tasks.workunit.client.1.smithi135.stdout:4/633: mkdir d10/d1a/d32/d40/d92/db9/ddc 0 2024-03-30T04:28:29.600 INFO:tasks.workunit.client.1.smithi135.stdout:7/773: creat d4/db/d1a/d3d/ddc/db0/de1/f102 x:0 0 0 2024-03-30T04:28:29.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:29 smithi135 ceph-mon[44364]: Upgrade: Updating mgr.smithi123.xgksqq 2024-03-30T04:28:29.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:29 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:28:29.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:29 smithi135 ceph-mon[44364]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi123.xgksqq", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T04:28:29.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:29 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi123.xgksqq", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T04:28:29.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:29 smithi135 ceph-mon[44364]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T04:28:29.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:29 smithi135 ceph-mon[44364]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:28:29.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:29 smithi135 ceph-mon[44364]: Deploying daemon mgr.smithi123.xgksqq on smithi123 2024-03-30T04:28:29.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:29 smithi135 ceph-mon[44364]: pgmap v48: 65 pgs: 65 active+clean; 3.0 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 89 MiB/s rd, 206 MiB/s wr, 459 op/s 2024-03-30T04:28:29.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:29 smithi135 ceph-mon[44364]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:28:29.605 INFO:tasks.workunit.client.1.smithi135.stdout:7/774: read d4/f11 [937514,103228] 0 2024-03-30T04:28:29.606 INFO:tasks.workunit.client.1.smithi135.stdout:9/638: dwrite d7/d29/f51 [4194304,4194304] 0 2024-03-30T04:28:29.609 INFO:tasks.workunit.client.1.smithi135.stdout:5/606: creat d1/d45/d52/fb3 x:0 0 0 2024-03-30T04:28:29.612 INFO:tasks.workunit.client.1.smithi135.stdout:3/656: dwrite d3/d10/d16/d23/f30 [0,4194304] 0 2024-03-30T04:28:29.614 INFO:tasks.workunit.client.1.smithi135.stdout:1/685: symlink d16/d5a/d5f/d3d/d8d/dc9/da4/d99/d9a/le5 0 2024-03-30T04:28:29.623 INFO:tasks.workunit.client.1.smithi135.stdout:4/634: creat d10/d4d/d89/d9f/daa/fdd x:0 0 0 2024-03-30T04:28:29.625 INFO:tasks.workunit.client.1.smithi135.stdout:9/639: creat d7/d4f/d5c/d63/fd9 x:0 0 0 2024-03-30T04:28:29.628 INFO:tasks.workunit.client.1.smithi135.stdout:1/686: chown d16/d5a/dad/fc2 9585650 1 2024-03-30T04:28:29.630 INFO:tasks.workunit.client.1.smithi135.stdout:6/739: write d2/d68/d81/f88 [3244834,130270] 0 2024-03-30T04:28:29.634 INFO:tasks.workunit.client.1.smithi135.stdout:9/640: read d7/d29/d78/da4/f7c [1157868,76509] 0 2024-03-30T04:28:29.635 INFO:tasks.workunit.client.1.smithi135.stdout:0/777: dwrite d0/d13/fde [0,4194304] 0 2024-03-30T04:28:29.647 INFO:tasks.workunit.client.1.smithi135.stdout:3/657: rename d3/d10/d16/f1e to d3/d54/dab/fde 0 2024-03-30T04:28:29.647 INFO:tasks.workunit.client.1.smithi135.stdout:6/740: symlink d2/dd/d18/d87/lef 0 2024-03-30T04:28:29.655 INFO:tasks.workunit.client.1.smithi135.stdout:4/635: dread d10/d1a/d42/f54 [0,4194304] 0 2024-03-30T04:28:29.658 INFO:tasks.workunit.client.1.smithi135.stdout:0/778: unlink d0/d34/dfc/d63/d77/c8e 0 2024-03-30T04:28:29.658 INFO:tasks.workunit.client.1.smithi135.stdout:8/628: dread d6/d8/d15/d54/f58 [0,4194304] 0 2024-03-30T04:28:29.666 INFO:tasks.workunit.client.1.smithi135.stdout:3/658: creat d3/d50/d22/fdf x:0 0 0 2024-03-30T04:28:29.666 INFO:tasks.workunit.client.1.smithi135.stdout:7/775: dwrite d4/f54 [0,4194304] 0 2024-03-30T04:28:29.669 INFO:tasks.workunit.client.1.smithi135.stdout:1/687: creat d16/d5a/d5f/d3a/dc6/fe6 x:0 0 0 2024-03-30T04:28:29.672 INFO:tasks.workunit.client.1.smithi135.stdout:7/776: chown d4/db/d1a/d3d/ddc/db0/db4 6240 1 2024-03-30T04:28:29.672 INFO:tasks.workunit.client.1.smithi135.stdout:1/688: write d16/d5a/d5f/d3d/d8d/dc9/f1f [7140296,12929] 0 2024-03-30T04:28:29.682 INFO:tasks.workunit.client.1.smithi135.stdout:2/562: dread d4/d32/d43/f6b [0,4194304] 0 2024-03-30T04:28:29.682 INFO:tasks.workunit.client.1.smithi135.stdout:1/689: symlink d16/d29/d32/d85/le7 0 2024-03-30T04:28:29.683 INFO:tasks.workunit.client.1.smithi135.stdout:7/777: write d4/ff9 [1810383,13126] 0 2024-03-30T04:28:29.687 INFO:tasks.workunit.client.1.smithi135.stdout:9/641: getdents d7/d4f/d6e/dad 0 2024-03-30T04:28:29.687 INFO:tasks.workunit.client.1.smithi135.stdout:2/563: chown d4/d32/l9d 960887131 1 2024-03-30T04:28:29.690 INFO:tasks.workunit.client.1.smithi135.stdout:7/778: write d4/db/d1a/d5c/d5d/db5/fd2 [207285,83577] 0 2024-03-30T04:28:29.692 INFO:tasks.workunit.client.1.smithi135.stdout:3/659: symlink d3/d10/d16/le0 0 2024-03-30T04:28:29.693 INFO:tasks.workunit.client.1.smithi135.stdout:9/642: chown d7/d29/d78/da4/f25 97835531 1 2024-03-30T04:28:29.693 INFO:tasks.workunit.client.1.smithi135.stdout:1/690: creat d16/d29/d61/fe8 x:0 0 0 2024-03-30T04:28:29.695 INFO:tasks.workunit.client.1.smithi135.stdout:4/636: write d10/d72/d7c/f9b [461883,123396] 0 2024-03-30T04:28:29.700 INFO:tasks.workunit.client.1.smithi135.stdout:2/564: symlink d4/d32/d2e/d67/db4/lc0 0 2024-03-30T04:28:29.703 INFO:tasks.workunit.client.1.smithi135.stdout:0/779: dread d0/d19/d2d/f38 [0,4194304] 0 2024-03-30T04:28:29.705 INFO:tasks.workunit.client.1.smithi135.stdout:1/691: fsync d16/d5a/d5f/f7f 0 2024-03-30T04:28:29.708 INFO:tasks.workunit.client.1.smithi135.stdout:9/643: dread - d7/d41/d72/f8e zero size 2024-03-30T04:28:29.717 INFO:tasks.workunit.client.1.smithi135.stdout:3/660: rename d3/d54/laa to d3/d10/d9c/dbe/dd6/daf/le1 0 2024-03-30T04:28:29.718 INFO:tasks.workunit.client.1.smithi135.stdout:6/741: dread d2/dd/d14/d2b/f50 [4194304,4194304] 0 2024-03-30T04:28:29.723 INFO:tasks.workunit.client.1.smithi135.stdout:1/692: creat d16/d5a/dad/d77/fe9 x:0 0 0 2024-03-30T04:28:29.726 INFO:tasks.workunit.client.1.smithi135.stdout:9/644: unlink f1 0 2024-03-30T04:28:29.729 INFO:tasks.workunit.client.1.smithi135.stdout:8/629: dwrite d6/d8/d15/d79/fb1 [0,4194304] 0 2024-03-30T04:28:29.735 INFO:tasks.workunit.client.1.smithi135.stdout:7/779: dwrite d4/db/d1a/d5c/d5d/f86 [0,4194304] 0 2024-03-30T04:28:29.743 INFO:tasks.workunit.client.1.smithi135.stdout:1/693: write d16/d5a/d5f/d3d/db4/fd0 [2478300,62298] 0 2024-03-30T04:28:29.744 INFO:tasks.workunit.client.1.smithi135.stdout:3/661: creat d3/d10/d16/d23/db6/fe2 x:0 0 0 2024-03-30T04:28:29.745 INFO:tasks.workunit.client.1.smithi135.stdout:8/630: mkdir d6/dc7 0 2024-03-30T04:28:29.758 INFO:tasks.workunit.client.1.smithi135.stdout:3/662: chown d3/d10/d16/d23/d6a/d86/d98 1877390 1 2024-03-30T04:28:29.761 INFO:tasks.workunit.client.1.smithi135.stdout:6/742: dread d2/d36/f6e [0,4194304] 0 2024-03-30T04:28:29.761 INFO:tasks.workunit.client.1.smithi135.stdout:3/663: stat d3/d10/lc4 0 2024-03-30T04:28:29.765 INFO:tasks.workunit.client.1.smithi135.stdout:2/565: dwrite d4/da/f44 [0,4194304] 0 2024-03-30T04:28:29.766 INFO:tasks.workunit.client.1.smithi135.stdout:4/637: dwrite d10/d1a/d42/d46/f9d [0,4194304] 0 2024-03-30T04:28:29.773 INFO:tasks.workunit.client.1.smithi135.stdout:7/780: mknod d4/db/d1a/d5c/d59/d76/de4/c103 0 2024-03-30T04:28:29.776 INFO:tasks.workunit.client.1.smithi135.stdout:0/780: dwrite d0/d5/d10/fec [0,4194304] 0 2024-03-30T04:28:29.779 INFO:tasks.workunit.client.1.smithi135.stdout:4/638: dread - d10/fcc zero size 2024-03-30T04:28:29.786 INFO:tasks.workunit.client.1.smithi135.stdout:6/743: write d2/dd/d14/d2b/d86/d8e/f97 [2034251,101624] 0 2024-03-30T04:28:29.788 INFO:tasks.workunit.client.1.smithi135.stdout:7/781: dread - d4/db/d1a/d3d/d96/ff6 zero size 2024-03-30T04:28:29.791 INFO:tasks.workunit.client.1.smithi135.stdout:2/566: creat d4/d32/d12/d45/d7d/fc1 x:0 0 0 2024-03-30T04:28:29.791 INFO:tasks.workunit.client.1.smithi135.stdout:0/781: mknod d0/d19/d2d/d45/c101 0 2024-03-30T04:28:29.798 INFO:tasks.workunit.client.1.smithi135.stdout:3/664: truncate d3/d10/f31 2418494 0 2024-03-30T04:28:29.801 INFO:tasks.workunit.client.1.smithi135.stdout:6/744: read d2/dd/d14/d2b/d86/f3c [907116,117788] 0 2024-03-30T04:28:29.801 INFO:tasks.workunit.client.1.smithi135.stdout:9/645: dwrite d7/d41/d72/f74 [0,4194304] 0 2024-03-30T04:28:29.809 INFO:tasks.workunit.client.1.smithi135.stdout:1/694: dread d16/d29/d2c/d43/d50/f5b [0,4194304] 0 2024-03-30T04:28:29.810 INFO:tasks.workunit.client.1.smithi135.stdout:8/631: link d6/d8/d14/d32/c89 d6/d8/d15/d54/d4f/cc8 0 2024-03-30T04:28:29.813 INFO:tasks.workunit.client.1.smithi135.stdout:3/665: creat d3/d10/d16/d23/d6a/fe3 x:0 0 0 2024-03-30T04:28:29.814 INFO:tasks.workunit.client.1.smithi135.stdout:2/567: mkdir d4/d32/dc2 0 2024-03-30T04:28:29.815 INFO:tasks.workunit.client.1.smithi135.stdout:6/745: mknod d2/dd/d14/d30/d43/dcd/dc5/dd2/cf0 0 2024-03-30T04:28:29.818 INFO:tasks.workunit.client.1.smithi135.stdout:5/607: sync 2024-03-30T04:28:29.818 INFO:tasks.workunit.client.1.smithi135.stdout:2/568: write d4/d32/d43/f63 [718380,34896] 0 2024-03-30T04:28:29.820 INFO:tasks.workunit.client.1.smithi135.stdout:8/632: mknod d6/d8/d15/d54/d8e/cc9 0 2024-03-30T04:28:29.825 INFO:tasks.workunit.client.1.smithi135.stdout:4/639: getdents d10/d1a/d2b/daf 0 2024-03-30T04:28:29.832 INFO:tasks.workunit.client.1.smithi135.stdout:8/633: mkdir d6/d23/dca 0 2024-03-30T04:28:29.833 INFO:tasks.workunit.client.1.smithi135.stdout:7/782: link d4/db/d1a/d5c/d2c/l101 d4/db/d1a/d90/l104 0 2024-03-30T04:28:29.839 INFO:tasks.workunit.client.1.smithi135.stdout:9/646: getdents d7/d41 0 2024-03-30T04:28:29.841 INFO:tasks.workunit.client.1.smithi135.stdout:4/640: link d10/f74 d10/d72/fde 0 2024-03-30T04:28:29.845 INFO:tasks.workunit.client.1.smithi135.stdout:4/641: read - d10/d4d/d9c/fc5 zero size 2024-03-30T04:28:29.845 INFO:tasks.workunit.client.1.smithi135.stdout:3/666: dwrite d3/d10/d16/d23/d6a/d94/fba [0,4194304] 0 2024-03-30T04:28:29.845 INFO:tasks.workunit.client.1.smithi135.stdout:9/647: write d7/d55/d8d/f9d [272675,65381] 0 2024-03-30T04:28:29.846 INFO:tasks.workunit.client.1.smithi135.stdout:3/667: stat f2 0 2024-03-30T04:28:29.850 INFO:tasks.workunit.client.1.smithi135.stdout:7/783: mkdir d4/db/d1a/d3d/ddc/d5a/d105 0 2024-03-30T04:28:29.850 INFO:tasks.workunit.client.1.smithi135.stdout:3/668: write d3/db/fcd [277109,110205] 0 2024-03-30T04:28:29.862 INFO:tasks.workunit.client.1.smithi135.stdout:8/634: rename d6/f6e to d6/d8/d15/d54/d8e/d72/d73/fcb 0 2024-03-30T04:28:29.865 INFO:tasks.workunit.client.1.smithi135.stdout:8/635: chown d6/d8/d15/d54/d4f/l7c 0 1 2024-03-30T04:28:29.869 INFO:tasks.workunit.client.1.smithi135.stdout:0/782: write d0/d5/d9/fc1 [800632,61077] 0 2024-03-30T04:28:29.870 INFO:tasks.workunit.client.1.smithi135.stdout:7/784: link d4/db/d1a/d5c/d2c/d51/d56/d81/f7f d4/db/d1a/d5c/d82/f106 0 2024-03-30T04:28:29.881 INFO:tasks.workunit.client.1.smithi135.stdout:9/648: dwrite d7/d41/d72/f9f [0,4194304] 0 2024-03-30T04:28:29.882 INFO:tasks.workunit.client.1.smithi135.stdout:1/695: write d16/d5a/dad/f9b [110136,20077] 0 2024-03-30T04:28:29.883 INFO:tasks.workunit.client.1.smithi135.stdout:5/608: write d1/da/f1b [2379492,89628] 0 2024-03-30T04:28:29.884 INFO:tasks.workunit.client.1.smithi135.stdout:0/783: mkdir d0/d5/d9/d23/d5d/d102 0 2024-03-30T04:28:29.889 INFO:tasks.workunit.client.1.smithi135.stdout:5/609: dread - d1/d7/d1a/d1f/fa6 zero size 2024-03-30T04:28:29.893 INFO:tasks.workunit.client.1.smithi135.stdout:4/642: write d10/d1a/d42/d46/f82 [693012,90612] 0 2024-03-30T04:28:29.893 INFO:tasks.workunit.client.1.smithi135.stdout:7/785: link d4/db/d1a/d5c/d5d/f7b d4/db/d1a/d3d/ddc/d5a/f107 0 2024-03-30T04:28:29.897 INFO:tasks.workunit.client.1.smithi135.stdout:7/786: write d4/f54 [2684517,106587] 0 2024-03-30T04:28:29.898 INFO:tasks.workunit.client.1.smithi135.stdout:6/746: dwrite d2/dd/d14/f1e [0,4194304] 0 2024-03-30T04:28:29.906 INFO:tasks.workunit.client.1.smithi135.stdout:5/610: fdatasync d1/d3/d76/f98 0 2024-03-30T04:28:29.911 INFO:tasks.workunit.client.1.smithi135.stdout:9/649: dwrite d7/d41/d72/f9f [0,4194304] 0 2024-03-30T04:28:29.911 INFO:tasks.workunit.client.1.smithi135.stdout:5/611: write d1/da/f1b [4393454,94671] 0 2024-03-30T04:28:29.919 INFO:tasks.workunit.client.1.smithi135.stdout:8/636: dwrite d6/d8/d15/d54/d7e/da6/fb9 [0,4194304] 0 2024-03-30T04:28:29.919 INFO:tasks.workunit.client.1.smithi135.stdout:1/696: mknod d16/d5a/d5f/d3a/dbc/cea 0 2024-03-30T04:28:29.919 INFO:tasks.workunit.client.1.smithi135.stdout:6/747: mknod d2/d68/da8/cf1 0 2024-03-30T04:28:29.922 INFO:tasks.workunit.client.1.smithi135.stdout:0/784: write d0/d5/d10/f11 [5071192,120127] 0 2024-03-30T04:28:29.928 INFO:tasks.workunit.client.1.smithi135.stdout:1/697: chown d16/d5a/dad/f9b 0 1 2024-03-30T04:28:29.928 INFO:tasks.workunit.client.1.smithi135.stdout:8/637: dread - d6/d23/f82 zero size 2024-03-30T04:28:29.930 INFO:tasks.workunit.client.1.smithi135.stdout:8/638: write d6/d8/d15/d1f/f53 [1697543,86170] 0 2024-03-30T04:28:29.932 INFO:tasks.workunit.client.1.smithi135.stdout:8/639: stat d6/d8/d15/d54/d4f/c55 0 2024-03-30T04:28:29.941 INFO:tasks.workunit.client.1.smithi135.stdout:0/785: mkdir d0/d19/d2d/d45/da8/d103 0 2024-03-30T04:28:29.942 INFO:tasks.workunit.client.1.smithi135.stdout:5/612: dread d1/d7/d1a/d1f/f66 [0,4194304] 0 2024-03-30T04:28:29.945 INFO:tasks.workunit.client.1.smithi135.stdout:1/698: rename d16/d29/d2c/fde to d16/d5a/d5f/d3d/d8d/dc9/da4/d26/feb 0 2024-03-30T04:28:29.945 INFO:tasks.workunit.client.1.smithi135.stdout:7/787: dread d4/db/d1a/d5c/d2c/f63 [0,4194304] 0 2024-03-30T04:28:29.946 INFO:tasks.workunit.client.1.smithi135.stdout:9/650: getdents d7/d41 0 2024-03-30T04:28:29.949 INFO:tasks.workunit.client.1.smithi135.stdout:1/699: chown d16/d29/d32/d9d/lbd 80504 1 2024-03-30T04:28:29.951 INFO:tasks.workunit.client.1.smithi135.stdout:9/651: read - d7/d4f/d89/f95 zero size 2024-03-30T04:28:29.952 INFO:tasks.workunit.client.1.smithi135.stdout:0/786: dread d0/d5/d9/fc1 [0,4194304] 0 2024-03-30T04:28:29.953 INFO:tasks.workunit.client.1.smithi135.stdout:1/700: dread - d16/d5a/d5f/d3d/d8d/dc9/da4/d26/feb zero size 2024-03-30T04:28:29.954 INFO:tasks.workunit.client.1.smithi135.stdout:0/787: fsync d0/d13/d53/d7f/d83/fea 0 2024-03-30T04:28:29.955 INFO:tasks.workunit.client.1.smithi135.stdout:8/640: rename d6/d8/d15/d1f/fc4 to d6/fcc 0 2024-03-30T04:28:29.959 INFO:tasks.workunit.client.1.smithi135.stdout:0/788: readlink d0/d34/dfc/l20 0 2024-03-30T04:28:29.960 INFO:tasks.workunit.client.1.smithi135.stdout:9/652: creat d7/d29/d78/da4/d37/dba/fda x:0 0 0 2024-03-30T04:28:29.964 INFO:tasks.workunit.client.1.smithi135.stdout:5/613: rename d1/da/d4d/c77 to d1/d7/d11/cb4 0 2024-03-30T04:28:29.966 INFO:tasks.workunit.client.1.smithi135.stdout:9/653: truncate d7/f4e 36694 0 2024-03-30T04:28:29.969 INFO:tasks.workunit.client.1.smithi135.stdout:0/789: fdatasync d0/d34/dfc/d63/f89 0 2024-03-30T04:28:29.972 INFO:tasks.workunit.client.1.smithi135.stdout:0/790: write d0/d5/d10/fec [379924,49563] 0 2024-03-30T04:28:29.972 INFO:tasks.workunit.client.1.smithi135.stdout:1/701: link d16/d29/d61/fdc d16/d5a/d5f/fec 0 2024-03-30T04:28:29.976 INFO:tasks.workunit.client.1.smithi135.stdout:9/654: unlink d7/d29/d78/da4/d37/f6a 0 2024-03-30T04:28:29.980 INFO:tasks.workunit.client.1.smithi135.stdout:8/641: truncate d6/d8/f97 2410104 0 2024-03-30T04:28:29.980 INFO:tasks.workunit.client.1.smithi135.stdout:4/643: dread d10/d1a/d32/f70 [0,4194304] 0 2024-03-30T04:28:29.980 INFO:tasks.workunit.client.1.smithi135.stdout:5/614: symlink d1/d45/d52/da5/dac/lb5 0 2024-03-30T04:28:29.986 INFO:tasks.workunit.client.1.smithi135.stdout:9/655: fsync d7/d29/d78/da4/d37/f54 0 2024-03-30T04:28:29.986 INFO:tasks.workunit.client.1.smithi135.stdout:7/788: dwrite d4/db/d1a/d5c/d5d/f7b [0,4194304] 0 2024-03-30T04:28:29.996 INFO:tasks.workunit.client.1.smithi135.stdout:5/615: symlink d1/d3/d2b/d54/d6d/lb6 0 2024-03-30T04:28:29.997 INFO:tasks.workunit.client.1.smithi135.stdout:0/791: truncate d0/f12 442766 0 2024-03-30T04:28:29.997 INFO:tasks.workunit.client.1.smithi135.stdout:8/642: creat d6/d23/d7b/fcd x:0 0 0 2024-03-30T04:28:30.008 INFO:tasks.workunit.client.1.smithi135.stdout:0/792: mknod d0/d34/dfc/d63/d67/c104 0 2024-03-30T04:28:30.010 INFO:tasks.workunit.client.1.smithi135.stdout:6/748: dread d2/dd/f17 [0,4194304] 0 2024-03-30T04:28:30.010 INFO:tasks.workunit.client.1.smithi135.stdout:4/644: truncate d10/d4d/d89/d9f/fa9 104780 0 2024-03-30T04:28:30.019 INFO:tasks.workunit.client.1.smithi135.stdout:6/749: dread - d2/dd/d14/d30/fbc zero size 2024-03-30T04:28:30.019 INFO:tasks.workunit.client.1.smithi135.stdout:8/643: getdents d6/d8/d15/d2a/d35 0 2024-03-30T04:28:30.020 INFO:tasks.workunit.client.1.smithi135.stdout:1/702: dwrite d16/d5a/d5f/fec [0,4194304] 0 2024-03-30T04:28:30.023 INFO:tasks.workunit.client.1.smithi135.stdout:0/793: creat d0/d1/dd7/f105 x:0 0 0 2024-03-30T04:28:30.026 INFO:tasks.workunit.client.1.smithi135.stdout:9/656: dwrite d7/d4f/d89/fc2 [0,4194304] 0 2024-03-30T04:28:30.027 INFO:tasks.workunit.client.1.smithi135.stdout:6/750: readlink d2/dd/d14/d30/lec 0 2024-03-30T04:28:30.029 INFO:tasks.workunit.client.1.smithi135.stdout:8/644: read d6/d8/d15/fb4 [1765003,28991] 0 2024-03-30T04:28:30.029 INFO:tasks.workunit.client.1.smithi135.stdout:7/789: dwrite d4/db/d1a/d5c/d2c/d51/fd0 [0,4194304] 0 2024-03-30T04:28:30.029 INFO:tasks.workunit.client.1.smithi135.stdout:4/645: truncate d10/d1a/f5c 337401 0 2024-03-30T04:28:30.038 INFO:tasks.workunit.client.1.smithi135.stdout:1/703: symlink d16/d5a/dad/d77/led 0 2024-03-30T04:28:30.040 INFO:tasks.workunit.client.1.smithi135.stdout:6/751: chown d2/f9 5 1 2024-03-30T04:28:30.045 INFO:tasks.workunit.client.1.smithi135.stdout:5/616: dwrite d1/d45/d52/f9e [0,4194304] 0 2024-03-30T04:28:30.045 INFO:tasks.workunit.client.1.smithi135.stdout:0/794: truncate d0/d5/d9/f2e 2932989 0 2024-03-30T04:28:30.047 INFO:tasks.workunit.client.1.smithi135.stdout:4/646: mkdir d10/d1a/d32/d40/ddf 0 2024-03-30T04:28:30.047 INFO:tasks.workunit.client.1.smithi135.stdout:8/645: creat d6/d8/d15/d54/d8e/fce x:0 0 0 2024-03-30T04:28:30.050 INFO:tasks.workunit.client.1.smithi135.stdout:7/790: creat d4/db/d1a/d5c/d2c/d51/da8/f108 x:0 0 0 2024-03-30T04:28:30.050 INFO:tasks.workunit.client.1.smithi135.stdout:9/657: truncate d7/d29/d78/f8c 352327 0 2024-03-30T04:28:30.054 INFO:tasks.workunit.client.1.smithi135.stdout:5/617: mknod d1/d3/cb7 0 2024-03-30T04:28:30.056 INFO:tasks.workunit.client.1.smithi135.stdout:7/791: stat d4/db/d1a/d5c/d2c/d51/da8/l7a 0 2024-03-30T04:28:30.057 INFO:tasks.workunit.client.1.smithi135.stdout:9/658: rmdir d7/d4f/d6e 39 2024-03-30T04:28:30.061 INFO:tasks.workunit.client.1.smithi135.stdout:1/704: rmdir d16/d5a/d5f/d3d/d8d/dc9/da4/d26/d90 0 2024-03-30T04:28:30.063 INFO:tasks.workunit.client.1.smithi135.stdout:7/792: stat d4/db/f14 0 2024-03-30T04:28:30.065 INFO:tasks.workunit.client.1.smithi135.stdout:5/618: creat d1/d7/d1a/d1f/d9b/fb8 x:0 0 0 2024-03-30T04:28:30.066 INFO:tasks.workunit.client.1.smithi135.stdout:6/752: link d2/l3b d2/dd/d14/d30/d43/dcd/dbe/lf2 0 2024-03-30T04:28:30.069 INFO:tasks.workunit.client.1.smithi135.stdout:7/793: fsync d4/db/d1a/d5c/d2c/d2e/db9/faa 0 2024-03-30T04:28:30.072 INFO:tasks.workunit.client.1.smithi135.stdout:4/647: dread d10/d1a/d32/f70 [0,4194304] 0 2024-03-30T04:28:30.081 INFO:tasks.workunit.client.1.smithi135.stdout:2/569: sync 2024-03-30T04:28:30.086 INFO:tasks.workunit.client.1.smithi135.stdout:2/570: write d4/d32/d12/d45/f8e [4403620,34167] 0 2024-03-30T04:28:30.087 INFO:tasks.workunit.client.1.smithi135.stdout:8/646: dread d6/d8/f16 [4194304,4194304] 0 2024-03-30T04:28:30.088 INFO:tasks.workunit.client.1.smithi135.stdout:5/619: mknod d1/d7/d11/d3c/d4b/cb9 0 2024-03-30T04:28:30.093 INFO:tasks.workunit.client.1.smithi135.stdout:6/753: rename d2/dd/d18/f6b to d2/dd/ff3 0 2024-03-30T04:28:30.093 INFO:tasks.workunit.client.1.smithi135.stdout:1/705: dwrite d16/d5a/d5f/f4e [0,4194304] 0 2024-03-30T04:28:30.095 INFO:tasks.workunit.client.1.smithi135.stdout:5/620: chown d1/da/f1b 368039 1 2024-03-30T04:28:30.098 INFO:tasks.workunit.client.1.smithi135.stdout:7/794: dwrite d4/db/d1a/d5c/d5d/f8d [0,4194304] 0 2024-03-30T04:28:30.098 INFO:tasks.workunit.client.1.smithi135.stdout:4/648: rmdir d10/d1a/d32/d40/d92/db9/ddc 0 2024-03-30T04:28:30.098 INFO:tasks.workunit.client.1.smithi135.stdout:6/754: fsync d2/d68/fb9 0 2024-03-30T04:28:30.099 INFO:tasks.workunit.client.1.smithi135.stdout:8/647: mkdir d6/d8/d15/d1f/dcf 0 2024-03-30T04:28:30.101 INFO:tasks.workunit.client.1.smithi135.stdout:5/621: fsync d1/d45/d52/fb3 0 2024-03-30T04:28:30.107 INFO:tasks.workunit.client.1.smithi135.stdout:2/571: truncate d4/d32/d2e/f53 3118296 0 2024-03-30T04:28:30.109 INFO:tasks.workunit.client.1.smithi135.stdout:7/795: write d4/db/d1a/d5c/d5d/db5/fd2 [926229,130491] 0 2024-03-30T04:28:30.113 INFO:tasks.workunit.client.1.smithi135.stdout:2/572: readlink d4/d32/daf/lbf 0 2024-03-30T04:28:30.115 INFO:tasks.workunit.client.1.smithi135.stdout:0/795: dwrite d0/d1/d16/f4b [4194304,4194304] 0 2024-03-30T04:28:30.116 INFO:tasks.workunit.client.1.smithi135.stdout:8/648: symlink d6/d8/d15/d2a/d35/ld0 0 2024-03-30T04:28:30.120 INFO:tasks.workunit.client.1.smithi135.stdout:7/796: chown d4/db/d1a/d5c/d59/fb6 284213 1 2024-03-30T04:28:30.120 INFO:tasks.workunit.client.1.smithi135.stdout:6/755: creat d2/dd/d18/db0/ff4 x:0 0 0 2024-03-30T04:28:30.120 INFO:tasks.workunit.client.1.smithi135.stdout:5/622: rename d1/d3/c9a to d1/da/d17/d70/cba 0 2024-03-30T04:28:30.122 INFO:tasks.workunit.client.1.smithi135.stdout:4/649: mkdir d10/d1a/d42/de0 0 2024-03-30T04:28:30.124 INFO:tasks.workunit.client.1.smithi135.stdout:6/756: readlink d2/dd/d14/d38/l84 0 2024-03-30T04:28:30.128 INFO:tasks.workunit.client.1.smithi135.stdout:6/757: write d2/dd/d40/fd6 [130341,13379] 0 2024-03-30T04:28:30.129 INFO:tasks.workunit.client.1.smithi135.stdout:7/797: fsync d4/f8 0 2024-03-30T04:28:30.134 INFO:tasks.workunit.client.1.smithi135.stdout:6/758: read d2/d2e/f37 [1169684,30160] 0 2024-03-30T04:28:30.139 INFO:tasks.workunit.client.1.smithi135.stdout:1/706: write d16/d5a/d5f/d3d/d8d/dc9/da4/f5e [593763,64575] 0 2024-03-30T04:28:30.142 INFO:tasks.workunit.client.1.smithi135.stdout:4/650: rename d10/d1a/d32/c37 to d10/d1a/d32/d40/d92/db9/ce1 0 2024-03-30T04:28:30.144 INFO:tasks.workunit.client.1.smithi135.stdout:7/798: mknod d4/db/d1a/d4b/d40/c109 0 2024-03-30T04:28:30.144 INFO:tasks.workunit.client.1.smithi135.stdout:6/759: symlink d2/dd/d14/d30/d43/lf5 0 2024-03-30T04:28:30.148 INFO:tasks.workunit.client.1.smithi135.stdout:8/649: dwrite d6/d8/d15/d54/f6d [0,4194304] 0 2024-03-30T04:28:30.149 INFO:tasks.workunit.client.1.smithi135.stdout:6/760: fdatasync d2/dd/d18/db0/ff4 0 2024-03-30T04:28:30.150 INFO:tasks.workunit.client.1.smithi135.stdout:2/573: truncate d4/d32/d12/f3f 7172490 0 2024-03-30T04:28:30.154 INFO:tasks.workunit.client.1.smithi135.stdout:7/799: mkdir d4/db/d79/dbc/d10a 0 2024-03-30T04:28:30.156 INFO:tasks.workunit.client.1.smithi135.stdout:5/623: getdents d1/d3/d2b/d54 0 2024-03-30T04:28:30.158 INFO:tasks.workunit.client.1.smithi135.stdout:2/574: fdatasync d4/d32/f33 0 2024-03-30T04:28:30.160 INFO:tasks.workunit.client.1.smithi135.stdout:8/650: rename d6/d8/d15/d2a/d35/c36 to d6/d23/dca/cd1 0 2024-03-30T04:28:30.164 INFO:tasks.workunit.client.1.smithi135.stdout:7/800: creat d4/db/d1a/d4b/f10b x:0 0 0 2024-03-30T04:28:30.169 INFO:tasks.workunit.client.1.smithi135.stdout:2/575: creat d4/d32/d46/dbc/fc3 x:0 0 0 2024-03-30T04:28:30.171 INFO:tasks.workunit.client.1.smithi135.stdout:8/651: stat d6/f11 0 2024-03-30T04:28:30.173 INFO:tasks.workunit.client.1.smithi135.stdout:6/761: dwrite d2/d68/fb9 [0,4194304] 0 2024-03-30T04:28:30.176 INFO:tasks.workunit.client.1.smithi135.stdout:4/651: truncate d10/d1a/d32/fb5 449563 0 2024-03-30T04:28:30.178 INFO:tasks.workunit.client.1.smithi135.stdout:7/801: creat d4/db/d1a/d5c/d2c/d2e/f10c x:0 0 0 2024-03-30T04:28:30.179 INFO:tasks.workunit.client.1.smithi135.stdout:6/762: chown d2/dd/d14/f15 2 1 2024-03-30T04:28:30.181 INFO:tasks.workunit.client.1.smithi135.stdout:8/652: creat d6/d23/fd2 x:0 0 0 2024-03-30T04:28:30.186 INFO:tasks.workunit.client.1.smithi135.stdout:2/576: dread d4/d32/d12/d45/f8e [4194304,4194304] 0 2024-03-30T04:28:30.194 INFO:tasks.workunit.client.1.smithi135.stdout:8/653: creat d6/d8/d15/d54/d7e/da6/fd3 x:0 0 0 2024-03-30T04:28:30.194 INFO:tasks.workunit.client.1.smithi135.stdout:1/707: dwrite d16/f57 [0,4194304] 0 2024-03-30T04:28:30.202 INFO:tasks.workunit.client.1.smithi135.stdout:1/708: rmdir d16/d29/d32 39 2024-03-30T04:28:30.203 INFO:tasks.workunit.client.1.smithi135.stdout:5/624: dwrite d1/d7/f13 [0,4194304] 0 2024-03-30T04:28:30.208 INFO:tasks.workunit.client.1.smithi135.stdout:8/654: unlink d6/d8/d15/d2a/d35/c4d 0 2024-03-30T04:28:30.212 INFO:tasks.workunit.client.1.smithi135.stdout:1/709: truncate d16/d5a/dad/d77/fe3 816953 0 2024-03-30T04:28:30.214 INFO:tasks.workunit.client.1.smithi135.stdout:5/625: mknod d1/d45/d52/da5/cbb 0 2024-03-30T04:28:30.214 INFO:tasks.workunit.client.1.smithi135.stdout:6/763: link d2/dd/d14/f72 d2/dd/d14/d2b/d86/d8e/ff6 0 2024-03-30T04:28:30.222 INFO:tasks.workunit.client.1.smithi135.stdout:7/802: dwrite d4/f11 [0,4194304] 0 2024-03-30T04:28:30.225 INFO:tasks.workunit.client.1.smithi135.stdout:6/764: mknod d2/d2e/cf7 0 2024-03-30T04:28:30.226 INFO:tasks.workunit.client.1.smithi135.stdout:3/669: sync 2024-03-30T04:28:30.233 INFO:tasks.workunit.client.1.smithi135.stdout:8/655: creat d6/d8/d15/d54/d7e/daf/dc1/fd4 x:0 0 0 2024-03-30T04:28:30.237 INFO:tasks.workunit.client.1.smithi135.stdout:5/626: dread d1/d7/d11/f8a [0,4194304] 0 2024-03-30T04:28:30.238 INFO:tasks.workunit.client.1.smithi135.stdout:3/670: mkdir d3/d10/d9c/dbe/dd6/daf/de4 0 2024-03-30T04:28:30.241 INFO:tasks.workunit.client.1.smithi135.stdout:4/652: dread d10/d1a/d32/d40/f76 [0,4194304] 0 2024-03-30T04:28:30.241 INFO:tasks.workunit.client.1.smithi135.stdout:6/765: read d2/dd/d14/d2b/fc8 [1728151,68746] 0 2024-03-30T04:28:30.245 INFO:tasks.workunit.client.1.smithi135.stdout:5/627: read d1/d7/f48 [253106,110797] 0 2024-03-30T04:28:30.247 INFO:tasks.workunit.client.1.smithi135.stdout:3/671: readlink d3/d10/d9c/dbe/ldd 0 2024-03-30T04:28:30.253 INFO:tasks.workunit.client.1.smithi135.stdout:6/766: read d2/dd/d14/d2b/d86/d55/fb2 [449822,103633] 0 2024-03-30T04:28:30.254 INFO:tasks.workunit.client.1.smithi135.stdout:1/710: dread d16/d5a/d5f/d3d/d8d/dc9/da4/f65 [0,4194304] 0 2024-03-30T04:28:30.263 INFO:tasks.workunit.client.1.smithi135.stdout:3/672: dread d3/d10/d9c/dbe/dd6/d91/f97 [0,4194304] 0 2024-03-30T04:28:30.265 INFO:tasks.workunit.client.1.smithi135.stdout:3/673: readlink d3/d10/d9c/la0 0 2024-03-30T04:28:30.268 INFO:tasks.workunit.client.1.smithi135.stdout:5/628: rename d1/d7/d11/d3c/d4b/cb9 to d1/d7/d11/cbc 0 2024-03-30T04:28:30.269 INFO:tasks.workunit.client.1.smithi135.stdout:7/803: truncate d4/f11 1796563 0 2024-03-30T04:28:30.272 INFO:tasks.workunit.client.1.smithi135.stdout:1/711: symlink d16/d29/d61/lee 0 2024-03-30T04:28:30.276 INFO:tasks.workunit.client.1.smithi135.stdout:5/629: rmdir d1/d3/d2b/d54 39 2024-03-30T04:28:30.278 INFO:tasks.workunit.client.1.smithi135.stdout:3/674: fsync d3/d50/d22/f71 0 2024-03-30T04:28:30.279 INFO:tasks.workunit.client.1.smithi135.stdout:6/767: dread d2/dd/d14/d2b/d86/d55/f7b [0,4194304] 0 2024-03-30T04:28:30.287 INFO:tasks.workunit.client.1.smithi135.stdout:1/712: creat d16/d29/d2c/fef x:0 0 0 2024-03-30T04:28:30.288 INFO:tasks.workunit.client.1.smithi135.stdout:7/804: write d4/db/fa0 [1235483,78679] 0 2024-03-30T04:28:30.289 INFO:tasks.workunit.client.1.smithi135.stdout:4/653: dwrite d10/d1a/d3d/f91 [0,4194304] 0 2024-03-30T04:28:30.292 INFO:tasks.workunit.client.1.smithi135.stdout:6/768: unlink d2/dd/d18/d87/c91 0 2024-03-30T04:28:30.297 INFO:tasks.workunit.client.1.smithi135.stdout:4/654: chown d10/d1a/d2b/f31 1458 1 2024-03-30T04:28:30.298 INFO:tasks.workunit.client.1.smithi135.stdout:7/805: creat d4/db/d1a/d5c/d59/d76/f10d x:0 0 0 2024-03-30T04:28:30.299 INFO:tasks.workunit.client.1.smithi135.stdout:6/769: mknod d2/dd/d18/cf8 0 2024-03-30T04:28:30.303 INFO:tasks.workunit.client.1.smithi135.stdout:3/675: dwrite d3/d6e/f80 [4194304,4194304] 0 2024-03-30T04:28:30.304 INFO:tasks.workunit.client.1.smithi135.stdout:7/806: rmdir d4/db/d1a/d3d/ddc/db0 39 2024-03-30T04:28:30.308 INFO:tasks.workunit.client.1.smithi135.stdout:3/676: fdatasync d3/d54/dab/fd0 0 2024-03-30T04:28:30.308 INFO:tasks.workunit.client.1.smithi135.stdout:6/770: creat d2/dd/d14/d2b/d86/d55/ff9 x:0 0 0 2024-03-30T04:28:30.309 INFO:tasks.workunit.client.1.smithi135.stdout:5/630: dwrite d1/d3/fa2 [0,4194304] 0 2024-03-30T04:28:30.315 INFO:tasks.workunit.client.1.smithi135.stdout:6/771: dread - d2/d68/da8/fb5 zero size 2024-03-30T04:28:30.316 INFO:tasks.workunit.client.1.smithi135.stdout:7/807: rename d4/db/d1a/f66 to d4/db/d1a/d90/dea/dfb/f10e 0 2024-03-30T04:28:30.316 INFO:tasks.workunit.client.1.smithi135.stdout:1/713: dwrite d16/d5a/d5f/d3d/d8d/dc9/f72 [4194304,4194304] 0 2024-03-30T04:28:30.322 INFO:tasks.workunit.client.1.smithi135.stdout:5/631: truncate d1/d3/d2b/f5b 419143 0 2024-03-30T04:28:30.324 INFO:tasks.workunit.client.1.smithi135.stdout:1/714: write d16/d29/d2c/d3c/f78 [2904403,82340] 0 2024-03-30T04:28:30.331 INFO:tasks.workunit.client.1.smithi135.stdout:4/655: dwrite d10/d72/fa6 [0,4194304] 0 2024-03-30T04:28:30.331 INFO:tasks.workunit.client.1.smithi135.stdout:5/632: mkdir d1/da/d4d/dbd 0 2024-03-30T04:28:30.332 INFO:tasks.workunit.client.1.smithi135.stdout:6/772: creat d2/ffa x:0 0 0 2024-03-30T04:28:30.333 INFO:tasks.workunit.client.1.smithi135.stdout:1/715: read d16/d29/d2c/d43/d50/f68 [3923613,77294] 0 2024-03-30T04:28:30.338 INFO:tasks.workunit.client.1.smithi135.stdout:4/656: chown d10/d1a/d42/d46/l4a 24 1 2024-03-30T04:28:30.340 INFO:tasks.workunit.client.1.smithi135.stdout:6/773: readlink d2/dd/l96 0 2024-03-30T04:28:30.344 INFO:tasks.workunit.client.1.smithi135.stdout:4/657: rmdir d10/d4d/dc1/db0 39 2024-03-30T04:28:30.349 INFO:tasks.workunit.client.1.smithi135.stdout:6/774: fsync d2/dd/d14/d30/f67 0 2024-03-30T04:28:30.351 INFO:tasks.workunit.client.1.smithi135.stdout:4/658: mknod d10/d1a/d2b/daf/ce2 0 2024-03-30T04:28:30.359 INFO:tasks.workunit.client.1.smithi135.stdout:3/677: dwrite d3/d50/d22/d74/f7c [0,4194304] 0 2024-03-30T04:28:30.365 INFO:tasks.workunit.client.1.smithi135.stdout:4/659: rename d10/f73 to d10/d4d/dc1/db0/fe3 0 2024-03-30T04:28:30.367 INFO:tasks.workunit.client.1.smithi135.stdout:6/775: truncate d2/d68/f83 2349461 0 2024-03-30T04:28:30.367 INFO:tasks.workunit.client.1.smithi135.stdout:1/716: dwrite d16/d5a/dad/fc2 [0,4194304] 0 2024-03-30T04:28:30.369 INFO:tasks.workunit.client.1.smithi135.stdout:5/633: dwrite d1/d7/f48 [0,4194304] 0 2024-03-30T04:28:30.375 INFO:tasks.workunit.client.1.smithi135.stdout:5/634: symlink d1/d45/lbe 0 2024-03-30T04:28:30.377 INFO:tasks.workunit.client.1.smithi135.stdout:3/678: symlink d3/d10/d9c/dbe/dd6/dd7/le5 0 2024-03-30T04:28:30.380 INFO:tasks.workunit.client.1.smithi135.stdout:3/679: read d3/db/fcd [340439,6468] 0 2024-03-30T04:28:30.389 INFO:tasks.workunit.client.1.smithi135.stdout:1/717: creat d16/d9f/ff0 x:0 0 0 2024-03-30T04:28:30.389 INFO:tasks.workunit.client.1.smithi135.stdout:5/635: creat d1/d45/fbf x:0 0 0 2024-03-30T04:28:30.392 INFO:tasks.workunit.client.1.smithi135.stdout:1/718: chown d16/d29/d61/f98 97772 1 2024-03-30T04:28:30.393 INFO:tasks.workunit.client.1.smithi135.stdout:3/680: truncate d3/d50/d22/fdf 628112 0 2024-03-30T04:28:30.394 INFO:tasks.workunit.client.1.smithi135.stdout:1/719: write d16/d29/d61/fdc [2418522,87539] 0 2024-03-30T04:28:30.396 INFO:tasks.workunit.client.1.smithi135.stdout:1/720: chown d16/d5a/d5f/f4e 1470015 1 2024-03-30T04:28:30.398 INFO:tasks.workunit.client.1.smithi135.stdout:1/721: read d16/f42 [65427,387] 0 2024-03-30T04:28:30.400 INFO:tasks.workunit.client.1.smithi135.stdout:1/722: write d16/f2f [2271535,30927] 0 2024-03-30T04:28:30.413 INFO:tasks.workunit.client.1.smithi135.stdout:5/636: mknod d1/da/d17/d70/cc0 0 2024-03-30T04:28:30.414 INFO:tasks.workunit.client.1.smithi135.stdout:5/637: chown d1/d55 439040 1 2024-03-30T04:28:30.420 INFO:tasks.workunit.client.1.smithi135.stdout:4/660: dwrite d10/d1a/f22 [4194304,4194304] 0 2024-03-30T04:28:30.423 INFO:tasks.workunit.client.1.smithi135.stdout:4/661: fsync d10/d1a/d42/d46/f9d 0 2024-03-30T04:28:30.423 INFO:tasks.workunit.client.1.smithi135.stdout:5/638: creat d1/d45/d52/da5/fc1 x:0 0 0 2024-03-30T04:28:30.425 INFO:tasks.workunit.client.1.smithi135.stdout:4/662: stat d10/d1a/d42/d46/l90 0 2024-03-30T04:28:30.427 INFO:tasks.workunit.client.1.smithi135.stdout:5/639: creat d1/d45/d52/da5/dac/fc2 x:0 0 0 2024-03-30T04:28:30.432 INFO:tasks.workunit.client.1.smithi135.stdout:5/640: mknod d1/d7/d1a/d1f/d9b/cc3 0 2024-03-30T04:28:30.434 INFO:tasks.workunit.client.1.smithi135.stdout:1/723: dread d16/d29/d32/f38 [0,4194304] 0 2024-03-30T04:28:30.436 INFO:tasks.workunit.client.1.smithi135.stdout:5/641: symlink d1/d3/d76/d92/lc4 0 2024-03-30T04:28:30.436 INFO:tasks.workunit.client.1.smithi135.stdout:1/724: fsync d16/f1c 0 2024-03-30T04:28:30.441 INFO:tasks.workunit.client.1.smithi135.stdout:1/725: unlink d16/d29/d2c/d43/d50/f68 0 2024-03-30T04:28:30.442 INFO:tasks.workunit.client.1.smithi135.stdout:3/681: truncate d3/f24 613993 0 2024-03-30T04:28:30.443 INFO:tasks.workunit.client.1.smithi135.stdout:4/663: dwrite d10/d1a/d32/d40/f8e [0,4194304] 0 2024-03-30T04:28:30.444 INFO:tasks.workunit.client.1.smithi135.stdout:3/682: readlink d3/d50/d22/d74/lb0 0 2024-03-30T04:28:30.446 INFO:tasks.workunit.client.1.smithi135.stdout:1/726: creat d16/d5a/d5f/d3d/db4/ff1 x:0 0 0 2024-03-30T04:28:30.452 INFO:tasks.workunit.client.1.smithi135.stdout:4/664: mknod d10/d1a/ce4 0 2024-03-30T04:28:30.454 INFO:tasks.workunit.client.1.smithi135.stdout:3/683: mknod d3/d10/d9c/dbe/dd6/dd7/ce6 0 2024-03-30T04:28:30.456 INFO:tasks.workunit.client.1.smithi135.stdout:5/642: dwrite d1/d7/d11/f8a [0,4194304] 0 2024-03-30T04:28:30.459 INFO:tasks.workunit.client.1.smithi135.stdout:4/665: creat d10/d1a/d2b/daf/dd4/fe5 x:0 0 0 2024-03-30T04:28:30.461 INFO:tasks.workunit.client.1.smithi135.stdout:3/684: truncate d3/d50/d46/f21 4828802 0 2024-03-30T04:28:30.463 INFO:tasks.workunit.client.1.smithi135.stdout:5/643: symlink d1/d3/d76/lc5 0 2024-03-30T04:28:30.464 INFO:tasks.workunit.client.1.smithi135.stdout:4/666: dread - d10/d4d/d89/d9f/fb6 zero size 2024-03-30T04:28:30.465 INFO:tasks.workunit.client.1.smithi135.stdout:5/644: chown d1/d7/f13 25 1 2024-03-30T04:28:30.465 INFO:tasks.workunit.client.1.smithi135.stdout:3/685: mknod d3/db/da9/dcc/ce7 0 2024-03-30T04:28:30.474 INFO:tasks.workunit.client.1.smithi135.stdout:5/645: dread d1/da/f84 [0,4194304] 0 2024-03-30T04:28:30.474 INFO:tasks.workunit.client.1.smithi135.stdout:3/686: creat d3/fe8 x:0 0 0 2024-03-30T04:28:30.480 INFO:tasks.workunit.client.1.smithi135.stdout:3/687: rename d3/d50/d46/c39 to d3/d50/d22/ce9 0 2024-03-30T04:28:30.485 INFO:tasks.workunit.client.1.smithi135.stdout:4/667: dwrite d10/d72/fd0 [0,4194304] 0 2024-03-30T04:28:30.492 INFO:tasks.workunit.client.1.smithi135.stdout:3/688: getdents d3/d10/d16/d23 0 2024-03-30T04:28:30.494 INFO:tasks.workunit.client.1.smithi135.stdout:3/689: stat d3/d50/d22/fc0 0 2024-03-30T04:28:30.498 INFO:tasks.workunit.client.1.smithi135.stdout:4/668: rmdir d10/d1a/d42/d66/dc2 0 2024-03-30T04:28:30.498 INFO:tasks.workunit.client.1.smithi135.stdout:1/727: dwrite d16/d5a/d5f/d3d/d8d/dc9/da4/f8b [0,4194304] 0 2024-03-30T04:28:30.503 INFO:tasks.workunit.client.1.smithi135.stdout:1/728: write d16/d29/d2c/d3c/f78 [2793738,11100] 0 2024-03-30T04:28:30.503 INFO:tasks.workunit.client.1.smithi135.stdout:3/690: symlink d3/d10/d16/d23/d6a/lea 0 2024-03-30T04:28:30.510 INFO:tasks.workunit.client.1.smithi135.stdout:3/691: write d3/db/f77 [2321662,31924] 0 2024-03-30T04:28:30.512 INFO:tasks.workunit.client.1.smithi135.stdout:4/669: rename d10/d1a/d42/d46/c47 to d10/d1a/d32/d40/ce6 0 2024-03-30T04:28:30.514 INFO:tasks.workunit.client.1.smithi135.stdout:4/670: readlink d10/d1a/d3d/l80 0 2024-03-30T04:28:30.517 INFO:tasks.workunit.client.1.smithi135.stdout:3/692: symlink d3/d10/d9b/leb 0 2024-03-30T04:28:30.517 INFO:tasks.workunit.client.1.smithi135.stdout:1/729: link d16/d29/d2c/d3c/lc4 d16/d5a/d5f/lf2 0 2024-03-30T04:28:30.518 INFO:tasks.workunit.client.1.smithi135.stdout:3/693: dread - d3/db/fc6 zero size 2024-03-30T04:28:30.519 INFO:tasks.workunit.client.1.smithi135.stdout:5/646: dread d1/d7/ff [4194304,4194304] 0 2024-03-30T04:28:30.520 INFO:tasks.workunit.client.1.smithi135.stdout:4/671: rename d10/d1a/d42/d46/f98 to d10/d4d/dc3/fe7 0 2024-03-30T04:28:30.521 INFO:tasks.workunit.client.1.smithi135.stdout:3/694: mkdir d3/d50/dec 0 2024-03-30T04:28:30.524 INFO:tasks.workunit.client.1.smithi135.stdout:3/695: fsync d3/d50/d22/f87 0 2024-03-30T04:28:30.529 INFO:tasks.workunit.client.1.smithi135.stdout:5/647: rename d1/d7/d1a/d1f/d9b/cc3 to d1/da/d4d/cc6 0 2024-03-30T04:28:30.532 INFO:tasks.workunit.client.1.smithi135.stdout:5/648: symlink d1/d45/d52/lc7 0 2024-03-30T04:28:30.534 INFO:tasks.workunit.client.1.smithi135.stdout:5/649: dread - d1/d7/d1a/d1f/d9b/fb8 zero size 2024-03-30T04:28:30.535 INFO:tasks.workunit.client.1.smithi135.stdout:3/696: rename d3/d10/fca to d3/d10/d16/fed 0 2024-03-30T04:28:30.537 INFO:tasks.workunit.client.1.smithi135.stdout:3/697: dread - d3/d10/d16/d23/f27 zero size 2024-03-30T04:28:30.540 INFO:tasks.workunit.client.1.smithi135.stdout:2/577: sync 2024-03-30T04:28:30.540 INFO:tasks.workunit.client.1.smithi135.stdout:9/659: sync 2024-03-30T04:28:30.544 INFO:tasks.workunit.client.1.smithi135.stdout:9/660: fdatasync d7/d41/d72/f8e 0 2024-03-30T04:28:30.547 INFO:tasks.workunit.client.1.smithi135.stdout:2/578: rename d4/d6/f9c to d4/d32/d2e/d67/db4/fc4 0 2024-03-30T04:28:30.551 INFO:tasks.workunit.client.1.smithi135.stdout:9/661: mkdir d7/d55/d8d/ddb 0 2024-03-30T04:28:30.556 INFO:tasks.workunit.client.1.smithi135.stdout:3/698: write d3/d10/d16/d23/d6a/d94/fb9 [1012845,37998] 0 2024-03-30T04:28:30.559 INFO:tasks.workunit.client.1.smithi135.stdout:2/579: getdents d4/d32/d12/d45/d6d 0 2024-03-30T04:28:30.560 INFO:tasks.workunit.client.1.smithi135.stdout:1/730: dwrite d16/d5a/d5f/d3d/d8d/dc9/da4/f65 [0,4194304] 0 2024-03-30T04:28:30.563 INFO:tasks.workunit.client.1.smithi135.stdout:4/672: dwrite d10/d1a/d32/d40/d92/fb4 [0,4194304] 0 2024-03-30T04:28:30.568 INFO:tasks.workunit.client.1.smithi135.stdout:4/673: chown d10/c33 9919746 1 2024-03-30T04:28:30.569 INFO:tasks.workunit.client.1.smithi135.stdout:5/650: dread d1/d3/f38 [0,4194304] 0 2024-03-30T04:28:30.571 INFO:tasks.workunit.client.1.smithi135.stdout:1/731: symlink d16/d5a/d5f/d3a/lf3 0 2024-03-30T04:28:30.577 INFO:tasks.workunit.client.1.smithi135.stdout:1/732: fsync d16/d5a/d5f/d3d/d8d/dc9/da4/d99/f21 0 2024-03-30T04:28:30.578 INFO:tasks.workunit.client.1.smithi135.stdout:5/651: rename d1/d3/cb7 to d1/d7/d1a/cc8 0 2024-03-30T04:28:30.580 INFO:tasks.workunit.client.1.smithi135.stdout:4/674: mkdir d10/d1a/d32/d40/d92/db9/dcf/de8 0 2024-03-30T04:28:30.584 INFO:tasks.workunit.client.1.smithi135.stdout:1/733: unlink d16/d29/d32/le2 0 2024-03-30T04:28:30.588 INFO:tasks.workunit.client.1.smithi135.stdout:3/699: dwrite d3/f51 [0,4194304] 0 2024-03-30T04:28:30.589 INFO:tasks.workunit.client.1.smithi135.stdout:1/734: creat d16/d5a/dad/ff4 x:0 0 0 2024-03-30T04:28:30.591 INFO:tasks.workunit.client.1.smithi135.stdout:9/662: dwrite d7/d90/fa0 [0,4194304] 0 2024-03-30T04:28:30.598 INFO:tasks.workunit.client.1.smithi135.stdout:3/700: write d3/d50/d22/f87 [2677277,123262] 0 2024-03-30T04:28:30.599 INFO:tasks.workunit.client.1.smithi135.stdout:1/735: rename d16/f1c to d16/d9f/ff5 0 2024-03-30T04:28:30.603 INFO:tasks.workunit.client.1.smithi135.stdout:1/736: stat d16/d5a/d5f/d3d 0 2024-03-30T04:28:30.610 INFO:tasks.workunit.client.1.smithi135.stdout:1/737: dread - d16/d29/d61/f98 zero size 2024-03-30T04:28:30.613 INFO:tasks.workunit.client.1.smithi135.stdout:9/663: getdents d7/d4f/d6e/dad/dd4 0 2024-03-30T04:28:30.617 INFO:tasks.workunit.client.1.smithi135.stdout:5/652: dwrite d1/d45/d52/da5/dac/fc2 [0,4194304] 0 2024-03-30T04:28:30.618 INFO:tasks.workunit.client.1.smithi135.stdout:0/796: sync 2024-03-30T04:28:30.618 INFO:tasks.workunit.client.1.smithi135.stdout:8/656: sync 2024-03-30T04:28:30.619 INFO:tasks.workunit.client.1.smithi135.stdout:7/808: sync 2024-03-30T04:28:30.619 INFO:tasks.workunit.client.1.smithi135.stdout:6/776: sync 2024-03-30T04:28:30.620 INFO:tasks.workunit.client.1.smithi135.stdout:2/580: dread d4/d32/d12/f31 [0,4194304] 0 2024-03-30T04:28:30.620 INFO:tasks.workunit.client.1.smithi135.stdout:4/675: dwrite d10/f57 [0,4194304] 0 2024-03-30T04:28:30.621 INFO:tasks.workunit.client.1.smithi135.stdout:3/701: getdents d3/d10/d9c/dbe/dd6/d91 0 2024-03-30T04:28:30.625 INFO:tasks.workunit.client.1.smithi135.stdout:6/777: truncate d2/d68/d81/fb7 968837 0 2024-03-30T04:28:30.629 INFO:tasks.workunit.client.1.smithi135.stdout:2/581: truncate d4/d32/f78 869114 0 2024-03-30T04:28:30.630 INFO:tasks.workunit.client.1.smithi135.stdout:9/664: read d7/d55/f5b [123897,81409] 0 2024-03-30T04:28:30.632 INFO:tasks.workunit.client.1.smithi135.stdout:3/702: read - d3/d54/d5c/f78 zero size 2024-03-30T04:28:30.632 INFO:tasks.workunit.client.1.smithi135.stdout:4/676: chown d10/d1a/d42/ldb 1 1 2024-03-30T04:28:30.632 INFO:tasks.workunit.client.1.smithi135.stdout:8/657: fdatasync d6/d8/f31 0 2024-03-30T04:28:30.633 INFO:tasks.workunit.client.1.smithi135.stdout:7/809: symlink d4/db/d1a/d5c/d82/l10f 0 2024-03-30T04:28:30.637 INFO:tasks.workunit.client.1.smithi135.stdout:0/797: rename d0/d1/d16 to d0/d13/d91/d106 0 2024-03-30T04:28:30.639 INFO:tasks.workunit.client.1.smithi135.stdout:9/665: stat d7/d29/d78/da4/d37/dba 0 2024-03-30T04:28:30.641 INFO:tasks.workunit.client.1.smithi135.stdout:2/582: chown d4/d32/d2e/f4b 0 1 2024-03-30T04:28:30.642 INFO:tasks.workunit.client.1.smithi135.stdout:9/666: readlink d7/d41/l42 0 2024-03-30T04:28:30.644 INFO:tasks.workunit.client.1.smithi135.stdout:6/778: unlink d2/dd/l1b 0 2024-03-30T04:28:30.644 INFO:tasks.workunit.client.1.smithi135.stdout:5/653: creat d1/da/d17/d70/db2/dad/fc9 x:0 0 0 2024-03-30T04:28:30.646 INFO:tasks.workunit.client.1.smithi135.stdout:0/798: chown d0/d5/d9/f33 6 1 2024-03-30T04:28:30.646 INFO:tasks.workunit.client.1.smithi135.stdout:8/658: chown d6/d8/c66 354 1 2024-03-30T04:28:30.646 INFO:tasks.workunit.client.1.smithi135.stdout:3/703: creat d3/d10/d16/d23/d6a/d86/fee x:0 0 0 2024-03-30T04:28:30.647 INFO:tasks.workunit.client.1.smithi135.stdout:1/738: dwrite d16/d5a/dad/ff4 [0,4194304] 0 2024-03-30T04:28:30.652 INFO:tasks.workunit.client.1.smithi135.stdout:9/667: rmdir d7 39 2024-03-30T04:28:30.654 INFO:tasks.workunit.client.1.smithi135.stdout:7/810: creat d4/db/d1a/d5c/d2c/d2e/db9/f110 x:0 0 0 2024-03-30T04:28:30.654 INFO:tasks.workunit.client.1.smithi135.stdout:4/677: creat d10/fe9 x:0 0 0 2024-03-30T04:28:30.661 INFO:tasks.workunit.client.1.smithi135.stdout:2/583: getdents d4/d32/d2e/d9f 0 2024-03-30T04:28:30.661 INFO:tasks.workunit.client.1.smithi135.stdout:0/799: truncate d0/db5/db8/fe3 1010429 0 2024-03-30T04:28:30.661 INFO:tasks.workunit.client.1.smithi135.stdout:5/654: creat d1/d3/d76/d92/fca x:0 0 0 2024-03-30T04:28:30.665 INFO:tasks.workunit.client.1.smithi135.stdout:7/811: read d4/db/d1a/d90/dc7/fe9 [82615,71960] 0 2024-03-30T04:28:30.666 INFO:tasks.workunit.client.1.smithi135.stdout:3/704: mknod d3/d10/d16/d23/d6a/d86/d98/cef 0 2024-03-30T04:28:30.668 INFO:tasks.workunit.client.1.smithi135.stdout:1/739: mknod d16/d5a/d5f/d3d/d8d/dc9/da4/cf6 0 2024-03-30T04:28:30.672 INFO:tasks.workunit.client.1.smithi135.stdout:0/800: symlink d0/d13/d91/l107 0 2024-03-30T04:28:30.673 INFO:tasks.workunit.client.1.smithi135.stdout:5/655: creat d1/d7/fcb x:0 0 0 2024-03-30T04:28:30.673 INFO:tasks.workunit.client.1.smithi135.stdout:7/812: read d4/db/d1a/d90/dea/dfb/f10e [1090394,4439] 0 2024-03-30T04:28:30.677 INFO:tasks.workunit.client.1.smithi135.stdout:3/705: symlink d3/db/da9/dcc/lf0 0 2024-03-30T04:28:30.679 INFO:tasks.workunit.client.1.smithi135.stdout:1/740: mkdir d16/d5a/d5f/d3d/df7 0 2024-03-30T04:28:30.679 INFO:tasks.workunit.client.1.smithi135.stdout:2/584: dread d4/d6/d47/f98 [0,4194304] 0 2024-03-30T04:28:30.680 INFO:tasks.workunit.client.1.smithi135.stdout:9/668: write d7/d41/d72/f8e [410069,128933] 0 2024-03-30T04:28:30.681 INFO:tasks.workunit.client.1.smithi135.stdout:8/659: getdents d6/d8/d15/d1f 0 2024-03-30T04:28:30.682 INFO:tasks.workunit.client.1.smithi135.stdout:1/741: stat d16/d5a/d5f/fec 0 2024-03-30T04:28:30.683 INFO:tasks.workunit.client.1.smithi135.stdout:2/585: stat d4/d32/d12/f31 0 2024-03-30T04:28:30.692 INFO:tasks.workunit.client.1.smithi135.stdout:4/678: dread d10/d72/d7c/f83 [0,4194304] 0 2024-03-30T04:28:30.694 INFO:tasks.workunit.client.1.smithi135.stdout:0/801: unlink d0/d34/c68 0 2024-03-30T04:28:30.695 INFO:tasks.workunit.client.1.smithi135.stdout:6/779: dread d2/f9 [4194304,4194304] 0 2024-03-30T04:28:30.700 INFO:tasks.workunit.client.1.smithi135.stdout:3/706: fdatasync d3/f24 0 2024-03-30T04:28:30.702 INFO:tasks.workunit.client.1.smithi135.stdout:2/586: creat d4/d55/d77/fc5 x:0 0 0 2024-03-30T04:28:30.703 INFO:tasks.workunit.client.1.smithi135.stdout:9/669: read d7/d90/f9c [448468,129821] 0 2024-03-30T04:28:30.703 INFO:tasks.workunit.client.1.smithi135.stdout:7/813: creat d4/db/d1a/d3d/ddc/f111 x:0 0 0 2024-03-30T04:28:30.706 INFO:tasks.workunit.client.1.smithi135.stdout:8/660: creat d6/d8/d15/d54/d8e/d72/d73/d9f/fd5 x:0 0 0 2024-03-30T04:28:30.709 INFO:tasks.workunit.client.1.smithi135.stdout:4/679: rename d10/d72/fa6 to d10/d1a/d2b/daf/dd4/fea 0 2024-03-30T04:28:30.709 INFO:tasks.workunit.client.1.smithi135.stdout:2/587: stat d4/d6/c5b 0 2024-03-30T04:28:30.710 INFO:tasks.workunit.client.1.smithi135.stdout:3/707: mknod d3/d54/d5c/cf1 0 2024-03-30T04:28:30.710 INFO:tasks.workunit.client.1.smithi135.stdout:9/670: fsync d7/d41/d72/f74 0 2024-03-30T04:28:30.711 INFO:tasks.workunit.client.1.smithi135.stdout:5/656: dwrite d1/d7/ff [4194304,4194304] 0 2024-03-30T04:28:30.714 INFO:tasks.workunit.client.1.smithi135.stdout:7/814: dread - d4/db/d1a/d3d/d96/ff6 zero size 2024-03-30T04:28:30.715 INFO:tasks.workunit.client.1.smithi135.stdout:6/780: write d2/dd/d14/d2b/d86/d8e/f97 [4823151,119607] 0 2024-03-30T04:28:30.717 INFO:tasks.workunit.client.1.smithi135.stdout:8/661: symlink d6/d23/dca/ld6 0 2024-03-30T04:28:30.723 INFO:tasks.workunit.client.1.smithi135.stdout:1/742: dwrite d16/d5a/dad/f9b [0,4194304] 0 2024-03-30T04:28:30.728 INFO:tasks.workunit.client.1.smithi135.stdout:5/657: unlink d1/da/l4a 0 2024-03-30T04:28:30.730 INFO:tasks.workunit.client.1.smithi135.stdout:5/658: chown d1 145056 1 2024-03-30T04:28:30.731 INFO:tasks.workunit.client.1.smithi135.stdout:4/680: mknod d10/d1a/ceb 0 2024-03-30T04:28:30.733 INFO:tasks.workunit.client.1.smithi135.stdout:0/802: dwrite d0/d1/fa [4194304,4194304] 0 2024-03-30T04:28:30.735 INFO:tasks.workunit.client.1.smithi135.stdout:3/708: rename d3/db/fcd to d3/d10/d16/d23/d6a/d94/ff2 0 2024-03-30T04:28:30.735 INFO:tasks.workunit.client.1.smithi135.stdout:8/662: write d6/d8/d15/d54/d8e/fce [588885,88440] 0 2024-03-30T04:28:30.737 INFO:tasks.workunit.client.1.smithi135.stdout:6/781: unlink d2/dd/d14/l6d 0 2024-03-30T04:28:30.738 INFO:tasks.workunit.client.1.smithi135.stdout:4/681: chown d10/d1a/d32/d40/d92/fb4 29722471 1 2024-03-30T04:28:30.739 INFO:tasks.workunit.client.1.smithi135.stdout:1/743: mknod d16/d5a/d5f/d3a/cf8 0 2024-03-30T04:28:30.742 INFO:tasks.workunit.client.1.smithi135.stdout:9/671: rename d7/d55/l7b to d7/d4f/d5c/d63/ldc 0 2024-03-30T04:28:30.744 INFO:tasks.workunit.client.1.smithi135.stdout:7/815: creat d4/db/d1a/d3d/ddc/db0/db4/dcc/f112 x:0 0 0 2024-03-30T04:28:30.747 INFO:tasks.workunit.client.1.smithi135.stdout:3/709: mknod d3/d10/d16/d23/d6a/cf3 0 2024-03-30T04:28:30.752 INFO:tasks.workunit.client.1.smithi135.stdout:6/782: creat d2/d68/d81/ffb x:0 0 0 2024-03-30T04:28:30.756 INFO:tasks.workunit.client.1.smithi135.stdout:8/663: symlink d6/d8/d15/ld7 0 2024-03-30T04:28:30.757 INFO:tasks.workunit.client.1.smithi135.stdout:0/803: truncate d0/d5/d9/d23/d2a/f51 4118300 0 2024-03-30T04:28:30.761 INFO:tasks.workunit.client.1.smithi135.stdout:8/664: readlink d6/d8/d15/d2a/lb7 0 2024-03-30T04:28:30.763 INFO:tasks.workunit.client.1.smithi135.stdout:5/659: rename d1/da/d4d/cc6 to d1/d45/d52/da5/dac/ccc 0 2024-03-30T04:28:30.764 INFO:tasks.workunit.client.1.smithi135.stdout:2/588: dwrite d4/d32/d12/d45/d7d/fc1 [0,4194304] 0 2024-03-30T04:28:30.764 INFO:tasks.workunit.client.1.smithi135.stdout:3/710: truncate d3/d10/f14 700112 0 2024-03-30T04:28:30.769 INFO:tasks.workunit.client.1.smithi135.stdout:6/783: truncate d2/dd/d14/d30/f67 489815 0 2024-03-30T04:28:30.769 INFO:tasks.workunit.client.1.smithi135.stdout:2/589: readlink d4/d32/daf/lbf 0 2024-03-30T04:28:30.770 INFO:tasks.workunit.client.1.smithi135.stdout:8/665: creat d6/d8/fd8 x:0 0 0 2024-03-30T04:28:30.774 INFO:tasks.workunit.client.1.smithi135.stdout:8/666: write d6/d8/f1c [4008003,23238] 0 2024-03-30T04:28:30.785 INFO:tasks.workunit.client.1.smithi135.stdout:5/660: creat d1/d7/d1a/fcd x:0 0 0 2024-03-30T04:28:30.788 INFO:tasks.workunit.client.1.smithi135.stdout:3/711: getdents d3/d10/d16 0 2024-03-30T04:28:30.796 INFO:tasks.workunit.client.1.smithi135.stdout:5/661: read - d1/d7/d1a/d1f/d9b/fb8 zero size 2024-03-30T04:28:30.796 INFO:tasks.workunit.client.1.smithi135.stdout:3/712: fsync d3/d10/d16/d23/d6a/d86/fd1 0 2024-03-30T04:28:30.796 INFO:tasks.workunit.client.1.smithi135.stdout:7/816: dwrite d4/db/d1a/d3d/ddc/d5a/dae/fd3 [0,4194304] 0 2024-03-30T04:28:30.804 INFO:tasks.workunit.client.1.smithi135.stdout:0/804: dwrite d0/d13/d53/d7f/d83/fea [0,4194304] 0 2024-03-30T04:28:30.824 INFO:tasks.workunit.client.1.smithi135.stdout:9/672: dwrite d7/d41/f79 [0,4194304] 0 2024-03-30T04:28:30.824 INFO:tasks.workunit.client.1.smithi135.stdout:2/590: dread d4/da/fb [0,4194304] 0 2024-03-30T04:28:30.829 INFO:tasks.workunit.client.1.smithi135.stdout:9/673: chown d7/d29/d78/da4 13872710 1 2024-03-30T04:28:30.831 INFO:tasks.workunit.client.1.smithi135.stdout:6/784: dwrite d2/d36/fd4 [0,4194304] 0 2024-03-30T04:28:30.834 INFO:tasks.workunit.client.1.smithi135.stdout:4/682: dwrite d10/d4d/dc1/f93 [0,4194304] 0 2024-03-30T04:28:30.834 INFO:tasks.workunit.client.1.smithi135.stdout:0/805: read d0/d13/d53/d7f/d83/fac [1902158,74004] 0 2024-03-30T04:28:30.845 INFO:tasks.workunit.client.1.smithi135.stdout:2/591: symlink d4/d32/d12/lc6 0 2024-03-30T04:28:30.850 INFO:tasks.workunit.client.1.smithi135.stdout:7/817: dwrite d4/f8 [0,4194304] 0 2024-03-30T04:28:30.851 INFO:tasks.workunit.client.1.smithi135.stdout:0/806: rmdir d0/d34/dfc/d3b/d52 39 2024-03-30T04:28:30.857 INFO:tasks.workunit.client.1.smithi135.stdout:0/807: truncate d0/d1/dd7/fdc 1743192 0 2024-03-30T04:28:30.859 INFO:tasks.workunit.client.1.smithi135.stdout:0/808: chown d0/d13/d1e/d3a/l7b 134677181 1 2024-03-30T04:28:30.860 INFO:tasks.workunit.client.1.smithi135.stdout:0/809: read - d0/d34/dfc/d3b/fbc zero size 2024-03-30T04:28:30.863 INFO:tasks.workunit.client.1.smithi135.stdout:6/785: rename d2/dd/d14/d30/d43/dcd/fe3 to d2/dd/d18/d87/ffc 0 2024-03-30T04:28:30.865 INFO:tasks.workunit.client.1.smithi135.stdout:6/786: chown d2/dd/d14/c33 3 1 2024-03-30T04:28:30.865 INFO:tasks.workunit.client.1.smithi135.stdout:2/592: creat d4/da/d7a/fc7 x:0 0 0 2024-03-30T04:28:30.867 INFO:tasks.workunit.client.1.smithi135.stdout:7/818: mkdir d4/db/d1a/d5c/d59/d113 0 2024-03-30T04:28:30.867 INFO:tasks.workunit.client.1.smithi135.stdout:9/674: link d7/d29/d78/da4/l14 d7/d29/ldd 0 2024-03-30T04:28:30.872 INFO:tasks.workunit.client.1.smithi135.stdout:3/713: link d3/d10/f14 d3/d10/ff4 0 2024-03-30T04:28:30.873 INFO:tasks.workunit.client.1.smithi135.stdout:3/714: readlink d3/d10/lc4 0 2024-03-30T04:28:30.876 INFO:tasks.workunit.client.1.smithi135.stdout:3/715: fdatasync d3/d10/d16/d23/f30 0 2024-03-30T04:28:30.880 INFO:tasks.workunit.client.1.smithi135.stdout:9/675: creat d7/d41/fde x:0 0 0 2024-03-30T04:28:30.886 INFO:tasks.workunit.client.1.smithi135.stdout:6/787: creat d2/dd/d14/d2b/d86/ffd x:0 0 0 2024-03-30T04:28:30.886 INFO:tasks.workunit.client.1.smithi135.stdout:7/819: fdatasync d4/db/d1a/d5c/d82/f106 0 2024-03-30T04:28:30.893 INFO:tasks.workunit.client.1.smithi135.stdout:9/676: creat d7/d4f/d5c/d63/dd2/db1/dbc/fdf x:0 0 0 2024-03-30T04:28:30.895 INFO:tasks.workunit.client.1.smithi135.stdout:5/662: dread d1/d3/d76/f98 [0,4194304] 0 2024-03-30T04:28:30.896 INFO:tasks.workunit.client.1.smithi135.stdout:2/593: getdents d4/d32/d46/dbc 0 2024-03-30T04:28:30.897 INFO:tasks.workunit.client.1.smithi135.stdout:6/788: write d2/d2e/f31 [2731893,32155] 0 2024-03-30T04:28:30.901 INFO:tasks.workunit.client.1.smithi135.stdout:9/677: chown d7/d4f/d5c/f84 0 1 2024-03-30T04:28:30.904 INFO:tasks.workunit.client.1.smithi135.stdout:5/663: write d1/d3/d76/d92/fca [176480,69486] 0 2024-03-30T04:28:30.910 INFO:tasks.workunit.client.1.smithi135.stdout:2/594: mknod d4/d32/d46/dbc/dbe/cc8 0 2024-03-30T04:28:30.912 INFO:tasks.workunit.client.1.smithi135.stdout:7/820: dread d4/fc9 [0,4194304] 0 2024-03-30T04:28:30.918 INFO:tasks.workunit.client.1.smithi135.stdout:9/678: fdatasync d7/d4f/d6e/f87 0 2024-03-30T04:28:30.922 INFO:tasks.workunit.client.1.smithi135.stdout:2/595: write d4/d32/d12/d45/f8e [475522,128723] 0 2024-03-30T04:28:30.923 INFO:tasks.workunit.client.1.smithi135.stdout:4/683: dread d10/d1a/d42/d46/f9d [0,4194304] 0 2024-03-30T04:28:30.924 INFO:tasks.workunit.client.1.smithi135.stdout:9/679: write d7/f12 [8038531,39049] 0 2024-03-30T04:28:30.925 INFO:tasks.workunit.client.1.smithi135.stdout:7/821: read d4/db/d1a/d5c/d2c/d51/d56/fa4 [3537676,107144] 0 2024-03-30T04:28:30.939 INFO:tasks.workunit.client.1.smithi135.stdout:2/596: creat d4/d6/d99/fc9 x:0 0 0 2024-03-30T04:28:30.940 INFO:tasks.workunit.client.1.smithi135.stdout:4/684: truncate d10/f59 650205 0 2024-03-30T04:28:30.941 INFO:tasks.workunit.client.1.smithi135.stdout:9/680: creat d7/d29/d78/fe0 x:0 0 0 2024-03-30T04:28:30.945 INFO:tasks.workunit.client.1.smithi135.stdout:7/822: link d4/db/d1a/d5c/d2c/d2e/db9/f110 d4/db/d79/d92/dcf/f114 0 2024-03-30T04:28:30.947 INFO:tasks.workunit.client.1.smithi135.stdout:6/789: dread d2/dd/d14/d2b/f46 [0,4194304] 0 2024-03-30T04:28:30.948 INFO:tasks.workunit.client.1.smithi135.stdout:2/597: creat d4/d32/d43/fca x:0 0 0 2024-03-30T04:28:30.950 INFO:tasks.workunit.client.1.smithi135.stdout:7/823: fsync d4/db/d1a/d3d/f5b 0 2024-03-30T04:28:30.952 INFO:tasks.workunit.client.1.smithi135.stdout:5/664: dread d1/da/f18 [0,4194304] 0 2024-03-30T04:28:30.955 INFO:tasks.workunit.client.1.smithi135.stdout:9/681: symlink d7/d55/d8d/ddb/le1 0 2024-03-30T04:28:30.955 INFO:tasks.workunit.client.1.smithi135.stdout:4/685: chown d10/d4d/dc1/l99 415856445 1 2024-03-30T04:28:30.958 INFO:tasks.workunit.client.1.smithi135.stdout:2/598: mkdir d4/d32/d43/db5/dcb 0 2024-03-30T04:28:30.960 INFO:tasks.workunit.client.1.smithi135.stdout:7/824: mknod d4/db/d1a/d5c/d59/c115 0 2024-03-30T04:28:30.962 INFO:tasks.workunit.client.1.smithi135.stdout:6/790: fsync d2/dd/d14/d2b/d86/d8e/ff6 0 2024-03-30T04:28:30.964 INFO:tasks.workunit.client.1.smithi135.stdout:7/825: stat d4/db/d1a/d5c/d2c/f63 0 2024-03-30T04:28:30.964 INFO:tasks.workunit.client.1.smithi135.stdout:4/686: mknod d10/d1a/d32/cec 0 2024-03-30T04:28:30.966 INFO:tasks.workunit.client.1.smithi135.stdout:7/826: readlink d4/l27 0 2024-03-30T04:28:30.972 INFO:tasks.workunit.client.1.smithi135.stdout:2/599: creat d4/d32/d2e/d9f/fcc x:0 0 0 2024-03-30T04:28:30.979 INFO:tasks.workunit.client.1.smithi135.stdout:9/682: getdents d7/d4f/d8f 0 2024-03-30T04:28:30.980 INFO:tasks.workunit.client.1.smithi135.stdout:4/687: unlink d10/d1a/d32/c6c 0 2024-03-30T04:28:30.985 INFO:tasks.workunit.client.1.smithi135.stdout:2/600: truncate d4/d32/d46/dbc/fc3 356537 0 2024-03-30T04:28:30.985 INFO:tasks.workunit.client.1.smithi135.stdout:9/683: chown d7/d29/d78/da4/d37/l60 188846 1 2024-03-30T04:28:30.992 INFO:tasks.workunit.client.1.smithi135.stdout:1/744: sync 2024-03-30T04:28:30.997 INFO:tasks.workunit.client.1.smithi135.stdout:9/684: dread - d7/d4f/d5c/d63/dd2/db1/dbc/fdf zero size 2024-03-30T04:28:30.997 INFO:tasks.workunit.client.1.smithi135.stdout:5/665: dwrite d1/d45/d52/f90 [0,4194304] 0 2024-03-30T04:28:31.003 INFO:tasks.workunit.client.1.smithi135.stdout:2/601: mknod d4/d32/d12/d45/ccd 0 2024-03-30T04:28:31.009 INFO:tasks.workunit.client.1.smithi135.stdout:9/685: fsync d7/d29/f46 0 2024-03-30T04:28:31.015 INFO:tasks.workunit.client.1.smithi135.stdout:7/827: dwrite d4/db/d1a/d5c/d2c/d83/ff1 [0,4194304] 0 2024-03-30T04:28:31.016 INFO:tasks.workunit.client.1.smithi135.stdout:1/745: dread d16/d5a/d5f/d3d/db4/fcd [0,4194304] 0 2024-03-30T04:28:31.017 INFO:tasks.workunit.client.1.smithi135.stdout:2/602: symlink d4/da/d8d/lce 0 2024-03-30T04:28:31.019 INFO:tasks.workunit.client.1.smithi135.stdout:7/828: chown d4/f16 3 1 2024-03-30T04:28:31.020 INFO:tasks.workunit.client.1.smithi135.stdout:6/791: dwrite d2/d2e/f5f [0,4194304] 0 2024-03-30T04:28:31.027 INFO:tasks.workunit.client.1.smithi135.stdout:1/746: dread - d16/d5a/d5f/d3a/dc6/fe6 zero size 2024-03-30T04:28:31.029 INFO:tasks.workunit.client.1.smithi135.stdout:4/688: dwrite d10/d1a/d32/d40/d92/f94 [0,4194304] 0 2024-03-30T04:28:31.033 INFO:tasks.workunit.client.1.smithi135.stdout:1/747: write d16/d5a/d5f/d3d/d8d/dc9/da4/d99/f21 [2451335,4980] 0 2024-03-30T04:28:31.033 INFO:tasks.workunit.client.1.smithi135.stdout:6/792: creat d2/d68/ffe x:0 0 0 2024-03-30T04:28:31.035 INFO:tasks.workunit.client.1.smithi135.stdout:9/686: truncate d7/f11 4249916 0 2024-03-30T04:28:31.035 INFO:tasks.workunit.client.1.smithi135.stdout:7/829: mkdir d4/db/d1a/d5c/d59/d116 0 2024-03-30T04:28:31.040 INFO:tasks.workunit.client.1.smithi135.stdout:5/666: truncate d1/d7/f43 347377 0 2024-03-30T04:28:31.040 INFO:tasks.workunit.client.1.smithi135.stdout:9/687: chown d7/d4f/l50 1655560 1 2024-03-30T04:28:31.041 INFO:tasks.workunit.client.1.smithi135.stdout:1/748: write d16/d5a/f82 [1031766,130607] 0 2024-03-30T04:28:31.044 INFO:tasks.workunit.client.1.smithi135.stdout:8/667: sync 2024-03-30T04:28:31.044 INFO:tasks.workunit.client.1.smithi135.stdout:0/810: sync 2024-03-30T04:28:31.050 INFO:tasks.workunit.client.1.smithi135.stdout:0/811: unlink d0/l6a 0 2024-03-30T04:28:31.050 INFO:tasks.workunit.client.1.smithi135.stdout:9/688: write d7/d41/f5f [1332047,53705] 0 2024-03-30T04:28:31.052 INFO:tasks.workunit.client.1.smithi135.stdout:9/689: readlink d7/l32 0 2024-03-30T04:28:31.053 INFO:tasks.workunit.client.1.smithi135.stdout:1/749: write d16/d5a/d5f/d3d/d8d/dc9/da4/f65 [209476,52960] 0 2024-03-30T04:28:31.057 INFO:tasks.workunit.client.1.smithi135.stdout:0/812: write d0/d1/f29 [2803146,25627] 0 2024-03-30T04:28:31.065 INFO:tasks.workunit.client.1.smithi135.stdout:2/603: dread d4/da/f21 [0,4194304] 0 2024-03-30T04:28:31.066 INFO:tasks.workunit.client.1.smithi135.stdout:8/668: symlink d6/d8/d15/d54/d8e/d72/d8b/db2/ld9 0 2024-03-30T04:28:31.071 INFO:tasks.workunit.client.1.smithi135.stdout:0/813: symlink d0/d34/dfc/d63/l108 0 2024-03-30T04:28:31.073 INFO:tasks.workunit.client.1.smithi135.stdout:9/690: truncate d7/d29/f2c 1430958 0 2024-03-30T04:28:31.073 INFO:tasks.workunit.client.1.smithi135.stdout:5/667: dwrite d1/d7/d1a/fcd [0,4194304] 0 2024-03-30T04:28:31.074 INFO:tasks.workunit.client.1.smithi135.stdout:6/793: truncate d2/d36/fd4 1361765 0 2024-03-30T04:28:31.076 INFO:tasks.workunit.client.1.smithi135.stdout:4/689: write d10/d1a/f5c [62106,87533] 0 2024-03-30T04:28:31.077 INFO:tasks.workunit.client.1.smithi135.stdout:5/668: dread - d1/fa1 zero size 2024-03-30T04:28:31.078 INFO:tasks.workunit.client.1.smithi135.stdout:2/604: rmdir d4/da 39 2024-03-30T04:28:31.080 INFO:tasks.workunit.client.1.smithi135.stdout:4/690: dread - d10/fcc zero size 2024-03-30T04:28:31.082 INFO:tasks.workunit.client.1.smithi135.stdout:5/669: dread - d1/da/d17/d70/f99 zero size 2024-03-30T04:28:31.083 INFO:tasks.workunit.client.1.smithi135.stdout:7/830: truncate d4/db/d1a/d5c/d5d/f8d 3066887 0 2024-03-30T04:28:31.085 INFO:tasks.workunit.client.1.smithi135.stdout:4/691: read d10/d1a/d42/f62 [2745690,71157] 0 2024-03-30T04:28:31.086 INFO:tasks.workunit.client.1.smithi135.stdout:6/794: creat d2/dd/d40/db4/fff x:0 0 0 2024-03-30T04:28:31.088 INFO:tasks.workunit.client.1.smithi135.stdout:7/831: readlink d4/db/d1a/l93 0 2024-03-30T04:28:31.104 INFO:tasks.workunit.client.1.smithi135.stdout:8/669: creat d6/d8/d14/fda x:0 0 0 2024-03-30T04:28:31.104 INFO:tasks.workunit.client.1.smithi135.stdout:1/750: link d16/d29/d32/d85/l87 d16/d5a/d5f/d3d/d8d/dc9/da4/d26/lf9 0 2024-03-30T04:28:31.108 INFO:tasks.workunit.client.1.smithi135.stdout:4/692: dread d10/d1a/d32/fb5 [0,4194304] 0 2024-03-30T04:28:31.108 INFO:tasks.workunit.client.1.smithi135.stdout:9/691: dwrite d7/d55/f6c [4194304,4194304] 0 2024-03-30T04:28:31.108 INFO:tasks.workunit.client.1.smithi135.stdout:2/605: creat d4/d32/d12/d45/d6d/fcf x:0 0 0 2024-03-30T04:28:31.113 INFO:tasks.workunit.client.1.smithi135.stdout:7/832: creat d4/db/d1a/d5c/d2c/d2e/db9/df4/d100/f117 x:0 0 0 2024-03-30T04:28:31.117 INFO:tasks.workunit.client.1.smithi135.stdout:5/670: dwrite d1/da/f1b [4194304,4194304] 0 2024-03-30T04:28:31.136 INFO:tasks.workunit.client.1.smithi135.stdout:9/692: rename d7/d29/d78/da4/f7c to d7/d4f/d5c/d63/fe2 0 2024-03-30T04:28:31.145 INFO:tasks.workunit.client.1.smithi135.stdout:7/833: mkdir d4/db/d1a/d5c/d2c/d83/d118 0 2024-03-30T04:28:31.148 INFO:tasks.workunit.client.1.smithi135.stdout:5/671: creat d1/da/d17/d70/db2/dad/fce x:0 0 0 2024-03-30T04:28:31.149 INFO:tasks.workunit.client.1.smithi135.stdout:2/606: dwrite d4/da/d7a/fc7 [0,4194304] 0 2024-03-30T04:28:31.150 INFO:tasks.workunit.client.1.smithi135.stdout:7/834: write d4/db/d1a/d4b/f10b [675481,32462] 0 2024-03-30T04:28:31.156 INFO:tasks.workunit.client.1.smithi135.stdout:6/795: dread d2/dd/d14/d2b/d86/d8e/f97 [0,4194304] 0 2024-03-30T04:28:31.163 INFO:tasks.workunit.client.1.smithi135.stdout:2/607: mkdir d4/d32/d46/dbc/dd0 0 2024-03-30T04:28:31.172 INFO:tasks.workunit.client.1.smithi135.stdout:1/751: dwrite d16/d5a/d5f/d3d/db4/fcd [0,4194304] 0 2024-03-30T04:28:31.172 INFO:tasks.workunit.client.1.smithi135.stdout:4/693: write d10/f59 [8375,32335] 0 2024-03-30T04:28:31.179 INFO:tasks.workunit.client.1.smithi135.stdout:9/693: rename d7/d29/f51 to d7/fe3 0 2024-03-30T04:28:31.179 INFO:tasks.workunit.client.1.smithi135.stdout:5/672: dwrite d1/d45/d52/f90 [0,4194304] 0 2024-03-30T04:28:31.182 INFO:tasks.workunit.client.1.smithi135.stdout:8/670: truncate d6/d8/d14/d32/fab 1233103 0 2024-03-30T04:28:31.187 INFO:tasks.workunit.client.1.smithi135.stdout:6/796: fdatasync d2/d36/fd4 0 2024-03-30T04:28:31.194 INFO:tasks.workunit.client.1.smithi135.stdout:4/694: mkdir d10/d4d/d89/d9f/daa/ded 0 2024-03-30T04:28:31.200 INFO:tasks.workunit.client.1.smithi135.stdout:3/716: sync 2024-03-30T04:28:31.200 INFO:tasks.workunit.client.1.smithi135.stdout:7/835: rename d4/db/d1a/d5c/d2c/d51/l85 to d4/db/d1a/d5c/d59/l119 0 2024-03-30T04:28:31.202 INFO:tasks.workunit.client.1.smithi135.stdout:2/608: rename d4/d32/d2e to d4/d32/d2e/d9f/dd1 22 2024-03-30T04:28:31.202 INFO:tasks.workunit.client.1.smithi135.stdout:3/717: chown d3/d50/d46/d4c 78481 1 2024-03-30T04:28:31.204 INFO:tasks.workunit.client.1.smithi135.stdout:4/695: dread d10/d72/d7c/f9b [0,4194304] 0 2024-03-30T04:28:31.205 INFO:tasks.workunit.client.1.smithi135.stdout:9/694: fdatasync d7/d29/d78/da4/d37/f39 0 2024-03-30T04:28:31.209 INFO:tasks.workunit.client.1.smithi135.stdout:6/797: mknod d2/dd/d40/db4/c100 0 2024-03-30T04:28:31.211 INFO:tasks.workunit.client.1.smithi135.stdout:7/836: chown d4/db/d1a/d5c/d2c/l3b 11801 1 2024-03-30T04:28:31.211 INFO:tasks.workunit.client.1.smithi135.stdout:9/695: truncate d7/d4f/d5c/d63/fe2 2005343 0 2024-03-30T04:28:31.212 INFO:tasks.workunit.client.1.smithi135.stdout:3/718: mknod d3/d10/d9c/dbe/dd6/daf/cf5 0 2024-03-30T04:28:31.213 INFO:tasks.workunit.client.1.smithi135.stdout:3/719: readlink d3/d50/d22/l2f 0 2024-03-30T04:28:31.216 INFO:tasks.workunit.client.1.smithi135.stdout:7/837: write d4/db/d1a/d5c/d2c/d51/d56/f9e [2127736,51216] 0 2024-03-30T04:28:31.219 INFO:tasks.workunit.client.1.smithi135.stdout:6/798: unlink d2/dd/d14/d2b/d86/f3c 0 2024-03-30T04:28:31.220 INFO:tasks.workunit.client.1.smithi135.stdout:7/838: chown d4/db/d1a/d4b/f31 697 1 2024-03-30T04:28:31.222 INFO:tasks.workunit.client.1.smithi135.stdout:2/609: symlink d4/d32/d46/dbc/dd0/ld2 0 2024-03-30T04:28:31.222 INFO:tasks.workunit.client.1.smithi135.stdout:9/696: dread d7/d29/d78/da4/f80 [0,4194304] 0 2024-03-30T04:28:31.222 INFO:tasks.workunit.client.1.smithi135.stdout:4/696: symlink d10/d1a/d32/d40/ddf/lee 0 2024-03-30T04:28:31.225 INFO:tasks.workunit.client.1.smithi135.stdout:7/839: stat d4/db/d1a/d5c/d2c/d51/da8/fc2 0 2024-03-30T04:28:31.226 INFO:tasks.workunit.client.1.smithi135.stdout:5/673: truncate d1/d7/d1a/fcd 3424533 0 2024-03-30T04:28:31.228 INFO:tasks.workunit.client.1.smithi135.stdout:8/671: getdents d6/d8/d15/d79 0 2024-03-30T04:28:31.228 INFO:tasks.workunit.client.1.smithi135.stdout:9/697: chown d7/d29/d78/da4/d37/dba/cc7 45736054 1 2024-03-30T04:28:31.228 INFO:tasks.workunit.client.1.smithi135.stdout:1/752: dread d16/d5a/d5f/d3d/d8d/dc9/f72 [0,4194304] 0 2024-03-30T04:28:31.230 INFO:tasks.workunit.client.1.smithi135.stdout:9/698: stat d7/d41/d72 0 2024-03-30T04:28:31.233 INFO:tasks.workunit.client.1.smithi135.stdout:8/672: chown d6/d8/d15/d54/d8e/d72/d73/fcb 80270425 1 2024-03-30T04:28:31.245 INFO:tasks.workunit.client.1.smithi135.stdout:3/720: creat d3/d54/ff6 x:0 0 0 2024-03-30T04:28:31.251 INFO:tasks.workunit.client.1.smithi135.stdout:8/673: mkdir d6/d8/d15/d1f/ddb 0 2024-03-30T04:28:31.257 INFO:tasks.workunit.client.1.smithi135.stdout:5/674: dread d1/d7/f59 [0,4194304] 0 2024-03-30T04:28:31.261 INFO:tasks.workunit.client.1.smithi135.stdout:0/814: sync 2024-03-30T04:28:31.279 INFO:tasks.workunit.client.1.smithi135.stdout:1/753: dwrite d16/d5a/d5f/d3d/d8d/dc9/da4/f8b [4194304,4194304] 0 2024-03-30T04:28:31.288 INFO:tasks.workunit.client.1.smithi135.stdout:7/840: dwrite d4/db/d1a/d5c/d2c/d51/fd0 [4194304,4194304] 0 2024-03-30T04:28:31.302 INFO:tasks.workunit.client.1.smithi135.stdout:9/699: dwrite d7/d4f/d6e/f87 [0,4194304] 0 2024-03-30T04:28:31.311 INFO:tasks.workunit.client.1.smithi135.stdout:6/799: dwrite d2/dd/d14/f6c [0,4194304] 0 2024-03-30T04:28:31.312 INFO:tasks.workunit.client.1.smithi135.stdout:4/697: dwrite d10/d1a/f27 [0,4194304] 0 2024-03-30T04:28:31.315 INFO:tasks.workunit.client.1.smithi135.stdout:3/721: dwrite d3/fb3 [0,4194304] 0 2024-03-30T04:28:31.315 INFO:tasks.workunit.client.1.smithi135.stdout:2/610: dwrite d4/d32/d43/f92 [0,4194304] 0 2024-03-30T04:28:31.317 INFO:tasks.workunit.client.1.smithi135.stdout:7/841: write d4/db/fa0 [2339791,60544] 0 2024-03-30T04:28:31.318 INFO:tasks.workunit.client.1.smithi135.stdout:0/815: rename d0/d19/d2d/d45/f76 to d0/d5/d9/d23/d2a/d60/f109 0 2024-03-30T04:28:31.322 INFO:tasks.workunit.client.1.smithi135.stdout:5/675: getdents d1/da/d17/d70 0 2024-03-30T04:28:31.325 INFO:tasks.workunit.client.1.smithi135.stdout:9/700: rmdir d7/d4f/d6e/dad 39 2024-03-30T04:28:31.325 INFO:tasks.workunit.client.1.smithi135.stdout:8/674: dwrite d6/d8/d15/fb4 [0,4194304] 0 2024-03-30T04:28:31.331 INFO:tasks.workunit.client.1.smithi135.stdout:0/816: creat d0/d13/d53/d7f/d83/f10a x:0 0 0 2024-03-30T04:28:31.334 INFO:tasks.workunit.client.1.smithi135.stdout:4/698: write d10/d1a/d32/d40/d92/fb4 [412491,38543] 0 2024-03-30T04:28:31.334 INFO:tasks.workunit.client.1.smithi135.stdout:7/842: write d4/db/fa0 [1333731,90459] 0 2024-03-30T04:28:31.337 INFO:tasks.workunit.client.1.smithi135.stdout:9/701: symlink d7/d41/le4 0 2024-03-30T04:28:31.338 INFO:tasks.workunit.client.1.smithi135.stdout:0/817: chown d0/d13/fde 154936446 1 2024-03-30T04:28:31.340 INFO:tasks.workunit.client.1.smithi135.stdout:5/676: mknod d1/da/d17/d70/ccf 0 2024-03-30T04:28:31.340 INFO:tasks.workunit.client.1.smithi135.stdout:6/800: creat d2/dd/d14/d2b/d86/d55/f101 x:0 0 0 2024-03-30T04:28:31.340 INFO:tasks.workunit.client.1.smithi135.stdout:8/675: symlink d6/d23/d9d/ldc 0 2024-03-30T04:28:31.345 INFO:tasks.workunit.client.1.smithi135.stdout:8/676: chown d6/d23/d9d 867 1 2024-03-30T04:28:31.347 INFO:tasks.workunit.client.1.smithi135.stdout:6/801: chown d2/dd/d14/f6c 693771 1 2024-03-30T04:28:31.347 INFO:tasks.workunit.client.1.smithi135.stdout:9/702: rename d7/d4f to d7/d4f/d6e/dad/de5 22 2024-03-30T04:28:31.348 INFO:tasks.workunit.client.1.smithi135.stdout:3/722: mkdir d3/d10/d16/d23/d6a/d86/d98/d99/df7 0 2024-03-30T04:28:31.351 INFO:tasks.workunit.client.1.smithi135.stdout:0/818: creat d0/d13/d91/d106/dd5/f10b x:0 0 0 2024-03-30T04:28:31.351 INFO:tasks.workunit.client.1.smithi135.stdout:9/703: chown d7/d90/ld7 440955 1 2024-03-30T04:28:31.353 INFO:tasks.workunit.client.1.smithi135.stdout:8/677: write d6/d8/d15/d54/f6d [4873297,29655] 0 2024-03-30T04:28:31.354 INFO:tasks.workunit.client.1.smithi135.stdout:6/802: mknod d2/d2e/c102 0 2024-03-30T04:28:31.355 INFO:tasks.workunit.client.1.smithi135.stdout:3/723: symlink d3/db/da9/dcc/lf8 0 2024-03-30T04:28:31.361 INFO:tasks.workunit.client.1.smithi135.stdout:7/843: getdents d4/db/d1a/d3d/ddc/d5a 0 2024-03-30T04:28:31.361 INFO:tasks.workunit.client.1.smithi135.stdout:0/819: chown d0/d13/d1e/d3a/dc2 112 1 2024-03-30T04:28:31.361 INFO:tasks.workunit.client.1.smithi135.stdout:8/678: readlink d6/d8/d15/lae 0 2024-03-30T04:28:31.361 INFO:tasks.workunit.client.1.smithi135.stdout:9/704: fsync d7/d29/d78/da4/f3d 0 2024-03-30T04:28:31.363 INFO:tasks.workunit.client.1.smithi135.stdout:7/844: dread - d4/db/d1a/d5c/d2c/d2e/db9/df4/d100/f117 zero size 2024-03-30T04:28:31.367 INFO:tasks.workunit.client.1.smithi135.stdout:3/724: write d3/d10/d16/d23/d6a/d94/ff2 [1414822,99156] 0 2024-03-30T04:28:31.367 INFO:tasks.workunit.client.1.smithi135.stdout:0/820: creat d0/d13/d53/d7f/d83/f10c x:0 0 0 2024-03-30T04:28:31.368 INFO:tasks.workunit.client.1.smithi135.stdout:4/699: dread d10/d1a/f3a [0,4194304] 0 2024-03-30T04:28:31.370 INFO:tasks.workunit.client.1.smithi135.stdout:9/705: dread d7/d29/d78/f8c [0,4194304] 0 2024-03-30T04:28:31.371 INFO:tasks.workunit.client.1.smithi135.stdout:7/845: symlink d4/db/d79/d92/l11a 0 2024-03-30T04:28:31.378 INFO:tasks.workunit.client.1.smithi135.stdout:0/821: creat d0/d5/d42/d9f/df5/f10d x:0 0 0 2024-03-30T04:28:31.381 INFO:tasks.workunit.client.1.smithi135.stdout:5/677: dwrite d1/f39 [0,4194304] 0 2024-03-30T04:28:31.381 INFO:tasks.workunit.client.1.smithi135.stdout:7/846: creat d4/db/d1a/d90/f11b x:0 0 0 2024-03-30T04:28:31.386 INFO:tasks.workunit.client.1.smithi135.stdout:2/611: dread d4/d32/d12/f3f [0,4194304] 0 2024-03-30T04:28:31.395 INFO:tasks.workunit.client.1.smithi135.stdout:0/822: mknod d0/d19/d2d/c10e 0 2024-03-30T04:28:31.396 INFO:tasks.workunit.client.1.smithi135.stdout:7/847: mknod d4/db/d1a/d90/dea/c11c 0 2024-03-30T04:28:31.399 INFO:tasks.workunit.client.1.smithi135.stdout:6/803: dwrite d2/d36/d76/fd5 [0,4194304] 0 2024-03-30T04:28:31.411 INFO:tasks.workunit.client.1.smithi135.stdout:3/725: dwrite d3/db/fe [0,4194304] 0 2024-03-30T04:28:31.411 INFO:tasks.workunit.client.1.smithi135.stdout:9/706: dwrite d7/f4e [0,4194304] 0 2024-03-30T04:28:31.418 INFO:tasks.workunit.client.1.smithi135.stdout:5/678: symlink d1/d3/d2b/d54/d5e/ld0 0 2024-03-30T04:28:31.420 INFO:tasks.workunit.client.1.smithi135.stdout:0/823: creat d0/d19/d2d/d45/da8/f10f x:0 0 0 2024-03-30T04:28:31.423 INFO:tasks.workunit.client.1.smithi135.stdout:3/726: mkdir d3/d10/d16/d23/d6a/df9 0 2024-03-30T04:28:31.424 INFO:tasks.workunit.client.1.smithi135.stdout:0/824: write d0/d13/d91/d106/f32 [2763023,12658] 0 2024-03-30T04:28:31.424 INFO:tasks.workunit.client.1.smithi135.stdout:7/848: mkdir d4/db/d1a/d5c/d2c/d51/d11d 0 2024-03-30T04:28:31.426 INFO:tasks.workunit.client.1.smithi135.stdout:9/707: dread - d7/d29/d78/fb2 zero size 2024-03-30T04:28:31.426 INFO:tasks.workunit.client.1.smithi135.stdout:5/679: mkdir d1/da/d17/d70/db2/dd1 0 2024-03-30T04:28:31.429 INFO:tasks.workunit.client.1.smithi135.stdout:2/612: creat d4/d32/d2e/fd3 x:0 0 0 2024-03-30T04:28:31.430 INFO:tasks.workunit.client.1.smithi135.stdout:3/727: mkdir d3/d10/d16/d23/d6a/d94/dfa 0 2024-03-30T04:28:31.431 INFO:tasks.workunit.client.1.smithi135.stdout:0/825: fdatasync d0/d1/fab 0 2024-03-30T04:28:31.436 INFO:tasks.workunit.client.1.smithi135.stdout:5/680: unlink d1/da/d17/d70/db2/f4e 0 2024-03-30T04:28:31.447 INFO:tasks.workunit.client.1.smithi135.stdout:0/826: rename d0/d5/d9/l22 to d0/d5/d9/l110 0 2024-03-30T04:28:31.452 INFO:tasks.workunit.client.1.smithi135.stdout:9/708: dread d7/f4e [0,4194304] 0 2024-03-30T04:28:31.455 INFO:tasks.workunit.client.1.smithi135.stdout:0/827: chown d0/d34/ceb 84 1 2024-03-30T04:28:31.460 INFO:tasks.workunit.client.1.smithi135.stdout:9/709: dread d7/d55/f5b [0,4194304] 0 2024-03-30T04:28:31.467 INFO:tasks.workunit.client.1.smithi135.stdout:0/828: symlink d0/d34/dfc/d63/d77/d8b/l111 0 2024-03-30T04:28:31.470 INFO:tasks.workunit.client.1.smithi135.stdout:6/804: dwrite d2/dd/d14/d30/d43/f69 [0,4194304] 0 2024-03-30T04:28:31.470 INFO:tasks.workunit.client.1.smithi135.stdout:9/710: mkdir d7/d4f/d5c/d63/dd2/db1/dbc/de6 0 2024-03-30T04:28:31.473 INFO:tasks.workunit.client.1.smithi135.stdout:5/681: dread d1/d3/d76/d92/fb0 [0,4194304] 0 2024-03-30T04:28:31.474 INFO:tasks.workunit.client.1.smithi135.stdout:0/829: read - d0/d19/d2d/fe0 zero size 2024-03-30T04:28:31.480 INFO:tasks.workunit.client.1.smithi135.stdout:9/711: mkdir d7/d55/d8d/ddb/de7 0 2024-03-30T04:28:31.480 INFO:tasks.workunit.client.1.smithi135.stdout:6/805: symlink d2/dd/d18/d87/l103 0 2024-03-30T04:28:31.483 INFO:tasks.workunit.client.1.smithi135.stdout:0/830: read d0/d5/d9/ff [1478222,65244] 0 2024-03-30T04:28:31.486 INFO:tasks.workunit.client.1.smithi135.stdout:7/849: dwrite d4/db/d1a/d90/dea/dfb/f10e [0,4194304] 0 2024-03-30T04:28:31.488 INFO:tasks.workunit.client.1.smithi135.stdout:2/613: dwrite d4/d32/f33 [4194304,4194304] 0 2024-03-30T04:28:31.488 INFO:tasks.workunit.client.1.smithi135.stdout:3/728: dread d3/d50/d46/f65 [4194304,4194304] 0 2024-03-30T04:28:31.488 INFO:tasks.workunit.client.1.smithi135.stdout:9/712: write d7/d55/f6c [5840087,117382] 0 2024-03-30T04:28:31.489 INFO:tasks.workunit.client.1.smithi135.stdout:6/806: rmdir d2/dd/d14/d2b/d86/d8e 39 2024-03-30T04:28:31.489 INFO:tasks.workunit.client.1.smithi135.stdout:1/754: sync 2024-03-30T04:28:31.497 INFO:tasks.workunit.client.1.smithi135.stdout:0/831: creat d0/d13/d1e/f112 x:0 0 0 2024-03-30T04:28:31.497 INFO:tasks.workunit.client.1.smithi135.stdout:5/682: truncate d1/d7/f43 106479 0 2024-03-30T04:28:31.499 INFO:tasks.workunit.client.1.smithi135.stdout:1/755: rmdir d16/d5a/d5f/d3d/d8d/dc9/da4/d26 39 2024-03-30T04:28:31.499 INFO:tasks.workunit.client.1.smithi135.stdout:3/729: dread - d3/d10/d16/d41/f6d zero size 2024-03-30T04:28:31.502 INFO:tasks.workunit.client.1.smithi135.stdout:6/807: symlink d2/dd/d14/l104 0 2024-03-30T04:28:31.504 INFO:tasks.workunit.client.1.smithi135.stdout:3/730: dread - d3/d54/dab/fd3 zero size 2024-03-30T04:28:31.510 INFO:tasks.workunit.client.1.smithi135.stdout:9/713: symlink d7/d4f/d5c/d63/dd2/db1/dbc/dd8/le8 0 2024-03-30T04:28:31.513 INFO:tasks.workunit.client.1.smithi135.stdout:6/808: truncate d2/f4d 241331 0 2024-03-30T04:28:31.519 INFO:tasks.workunit.client.1.smithi135.stdout:0/832: creat d0/d5/d9/d23/d5d/d90/f113 x:0 0 0 2024-03-30T04:28:31.521 INFO:tasks.workunit.client.1.smithi135.stdout:3/731: link d3/d10/c29 d3/d50/d22/d33/d7b/cfb 0 2024-03-30T04:28:31.528 INFO:tasks.workunit.client.1.smithi135.stdout:7/850: write d4/db/d1a/d3d/ddc/db0/fc0 [717797,23684] 0 2024-03-30T04:28:31.531 INFO:tasks.workunit.client.1.smithi135.stdout:2/614: dwrite d4/d32/d43/fa9 [0,4194304] 0 2024-03-30T04:28:31.533 INFO:tasks.workunit.client.1.smithi135.stdout:0/833: write d0/d13/d53/d7f/d83/fac [2938704,117546] 0 2024-03-30T04:28:31.534 INFO:tasks.workunit.client.1.smithi135.stdout:2/615: chown d4/d6 11383313 1 2024-03-30T04:28:31.544 INFO:tasks.workunit.client.1.smithi135.stdout:9/714: dwrite d7/d41/d72/f7d [0,4194304] 0 2024-03-30T04:28:31.546 INFO:tasks.workunit.client.1.smithi135.stdout:3/732: write d3/d10/d16/d23/d6a/d86/fee [269320,13370] 0 2024-03-30T04:28:31.546 INFO:tasks.workunit.client.1.smithi135.stdout:0/834: write d0/d5/faf [1955072,122465] 0 2024-03-30T04:28:31.551 INFO:tasks.workunit.client.1.smithi135.stdout:5/683: dread d1/da/f8c [0,4194304] 0 2024-03-30T04:28:31.551 INFO:tasks.workunit.client.1.smithi135.stdout:7/851: write d4/db/d1a/d3d/ddc/db0/db4/dcc/f112 [659409,61946] 0 2024-03-30T04:28:31.551 INFO:tasks.workunit.client.1.smithi135.stdout:0/835: truncate d0/d13/d53/d7f/dc5/fe4 156582 0 2024-03-30T04:28:31.555 INFO:tasks.workunit.client.1.smithi135.stdout:9/715: write d7/d29/d78/da4/f28 [844255,27107] 0 2024-03-30T04:28:31.557 INFO:tasks.workunit.client.1.smithi135.stdout:0/836: symlink d0/d19/d2d/d45/da8/l114 0 2024-03-30T04:28:31.558 INFO:tasks.workunit.client.1.smithi135.stdout:5/684: creat d1/d45/d52/d8f/fd2 x:0 0 0 2024-03-30T04:28:31.561 INFO:tasks.workunit.client.1.smithi135.stdout:1/756: dwrite d16/d29/d2c/fb7 [0,4194304] 0 2024-03-30T04:28:31.567 INFO:tasks.workunit.client.1.smithi135.stdout:0/837: mknod d0/d34/d3f/c115 0 2024-03-30T04:28:31.570 INFO:tasks.workunit.client.1.smithi135.stdout:6/809: dwrite d2/dd/d14/f6c [0,4194304] 0 2024-03-30T04:28:31.571 INFO:tasks.workunit.client.1.smithi135.stdout:9/716: write d7/fe3 [8837947,91033] 0 2024-03-30T04:28:31.572 INFO:tasks.workunit.client.1.smithi135.stdout:7/852: rename d4/db/d1a/d5c/d2c/d83/c8e to d4/db/d1a/d5c/d2c/d51/d11d/c11e 0 2024-03-30T04:28:31.576 INFO:tasks.workunit.client.1.smithi135.stdout:9/717: stat d7/d90 0 2024-03-30T04:28:31.576 INFO:tasks.workunit.client.1.smithi135.stdout:6/810: stat d2/dd/d14/d38/l9b 0 2024-03-30T04:28:31.583 INFO:tasks.workunit.client.1.smithi135.stdout:9/718: stat d7/d55/d8d/lcf 0 2024-03-30T04:28:31.588 INFO:tasks.workunit.client.1.smithi135.stdout:2/616: dwrite d4/d32/d2e/fd3 [0,4194304] 0 2024-03-30T04:28:31.590 INFO:tasks.workunit.client.1.smithi135.stdout:0/838: mknod d0/d5/d9/d23/d2a/c116 0 2024-03-30T04:28:31.593 INFO:tasks.workunit.client.1.smithi135.stdout:9/719: chown d7/d4f/d5c/f84 4853918 1 2024-03-30T04:28:31.594 INFO:tasks.workunit.client.1.smithi135.stdout:1/757: symlink d16/d5a/d5f/d3d/d8d/dc9/da4/d26/lfa 0 2024-03-30T04:28:31.594 INFO:tasks.workunit.client.1.smithi135.stdout:6/811: readlink d2/dd/d14/d2b/d86/d55/l79 0 2024-03-30T04:28:31.594 INFO:tasks.workunit.client.1.smithi135.stdout:0/839: readlink d0/d34/d3f/lee 0 2024-03-30T04:28:31.598 INFO:tasks.workunit.client.1.smithi135.stdout:7/853: truncate d4/db/d1a/d5c/d2c/d51/da8/f108 1027603 0 2024-03-30T04:28:31.600 INFO:tasks.workunit.client.1.smithi135.stdout:5/685: dwrite d1/d45/d52/fb3 [0,4194304] 0 2024-03-30T04:28:31.603 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:31 smithi135 ceph-mon[44364]: mgrmap e25: smithi135.kyblli(active, since 91s), standbys: smithi123.xgksqq 2024-03-30T04:28:31.603 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:31 smithi135 ceph-mon[44364]: pgmap v49: 65 pgs: 65 active+clean; 3.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 117 MiB/s rd, 285 MiB/s wr, 650 op/s 2024-03-30T04:28:31.606 INFO:tasks.workunit.client.1.smithi135.stdout:6/812: truncate d2/dd/d18/d87/fda 164483 0 2024-03-30T04:28:31.610 INFO:tasks.workunit.client.1.smithi135.stdout:1/758: rename d16/d29/d2c/c5d to d16/d5a/dad/d77/cfb 0 2024-03-30T04:28:31.611 INFO:tasks.workunit.client.1.smithi135.stdout:0/840: write d0/d1/dd7/fdc [363474,19979] 0 2024-03-30T04:28:31.611 INFO:tasks.workunit.client.1.smithi135.stdout:9/720: creat d7/d4f/d5c/d63/dd2/db1/dbc/de6/fe9 x:0 0 0 2024-03-30T04:28:31.612 INFO:tasks.workunit.client.1.smithi135.stdout:7/854: symlink d4/db/d1a/d3d/ddc/d5a/l11f 0 2024-03-30T04:28:31.613 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:31 smithi123 ceph-mon[35245]: mgrmap e25: smithi135.kyblli(active, since 91s), standbys: smithi123.xgksqq 2024-03-30T04:28:31.613 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:31 smithi123 ceph-mon[35245]: pgmap v49: 65 pgs: 65 active+clean; 3.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 117 MiB/s rd, 285 MiB/s wr, 650 op/s 2024-03-30T04:28:31.619 INFO:tasks.workunit.client.1.smithi135.stdout:2/617: link d4/d32/d43/la2 d4/d32/d2e/d9f/ld4 0 2024-03-30T04:28:31.619 INFO:tasks.workunit.client.1.smithi135.stdout:9/721: unlink d7/d55/c6f 0 2024-03-30T04:28:31.620 INFO:tasks.workunit.client.1.smithi135.stdout:6/813: chown d2/l3b 316 1 2024-03-30T04:28:31.620 INFO:tasks.workunit.client.1.smithi135.stdout:7/855: mknod d4/db/d1a/d3d/ddc/c120 0 2024-03-30T04:28:31.620 INFO:tasks.workunit.client.1.smithi135.stdout:0/841: fsync d0/d5/d9/d23/d2a/d60/f48 0 2024-03-30T04:28:31.626 INFO:tasks.workunit.client.1.smithi135.stdout:6/814: write d2/dd/d14/f6c [346592,75167] 0 2024-03-30T04:28:31.628 INFO:tasks.workunit.client.1.smithi135.stdout:2/618: rename d4/d6/d47/f98 to d4/d32/d46/dbc/dbe/fd5 0 2024-03-30T04:28:31.632 INFO:tasks.workunit.client.1.smithi135.stdout:7/856: creat d4/db/d1a/d5c/d59/d76/de4/dfc/f121 x:0 0 0 2024-03-30T04:28:31.633 INFO:tasks.workunit.client.1.smithi135.stdout:6/815: mkdir d2/d68/da8/d105 0 2024-03-30T04:28:31.635 INFO:tasks.workunit.client.1.smithi135.stdout:7/857: stat d4/db/d1a/d4b/d40 0 2024-03-30T04:28:31.637 INFO:tasks.workunit.client.1.smithi135.stdout:6/816: write d2/f4 [326320,70638] 0 2024-03-30T04:28:31.637 INFO:tasks.workunit.client.1.smithi135.stdout:1/759: dwrite d16/d29/d2c/fb7 [0,4194304] 0 2024-03-30T04:28:31.641 INFO:tasks.workunit.client.1.smithi135.stdout:0/842: write d0/d5/d10/f2c [2040172,113228] 0 2024-03-30T04:28:31.641 INFO:tasks.workunit.client.1.smithi135.stdout:2/619: rename d4/db3 to d4/da/d7a/dd6 0 2024-03-30T04:28:31.642 INFO:tasks.workunit.client.1.smithi135.stdout:7/858: mkdir d4/d122 0 2024-03-30T04:28:31.645 INFO:tasks.workunit.client.1.smithi135.stdout:1/760: read d16/d29/d2c/d43/d50/f5b [5012174,104176] 0 2024-03-30T04:28:31.645 INFO:tasks.workunit.client.1.smithi135.stdout:2/620: read d4/d32/d2e/f53 [2578154,73754] 0 2024-03-30T04:28:31.649 INFO:tasks.workunit.client.1.smithi135.stdout:6/817: mkdir d2/d106 0 2024-03-30T04:28:31.651 INFO:tasks.workunit.client.1.smithi135.stdout:1/761: dread d16/d5a/d5f/d3a/fa7 [0,4194304] 0 2024-03-30T04:28:31.652 INFO:tasks.workunit.client.1.smithi135.stdout:0/843: symlink d0/l117 0 2024-03-30T04:28:31.652 INFO:tasks.workunit.client.1.smithi135.stdout:2/621: creat d4/d32/d46/dbc/dbe/fd7 x:0 0 0 2024-03-30T04:28:31.655 INFO:tasks.workunit.client.1.smithi135.stdout:0/844: write d0/d1/f29 [5864086,7750] 0 2024-03-30T04:28:31.655 INFO:tasks.workunit.client.1.smithi135.stdout:2/622: fdatasync d4/d32/d2e/f8c 0 2024-03-30T04:28:31.655 INFO:tasks.workunit.client.1.smithi135.stdout:9/722: dread d7/d41/f85 [0,4194304] 0 2024-03-30T04:28:31.658 INFO:tasks.workunit.client.1.smithi135.stdout:1/762: creat d16/d29/d32/d9d/ffc x:0 0 0 2024-03-30T04:28:31.661 INFO:tasks.workunit.client.1.smithi135.stdout:2/623: write d4/d32/d2e/d2f/fb7 [867430,112152] 0 2024-03-30T04:28:31.663 INFO:tasks.workunit.client.1.smithi135.stdout:9/723: unlink d7/c23 0 2024-03-30T04:28:31.665 INFO:tasks.workunit.client.1.smithi135.stdout:2/624: chown d4/d32/d2e/d67/db4 3664 1 2024-03-30T04:28:31.668 INFO:tasks.workunit.client.1.smithi135.stdout:7/859: dwrite d4/db/fa0 [0,4194304] 0 2024-03-30T04:28:31.671 INFO:tasks.workunit.client.1.smithi135.stdout:9/724: mknod d7/d4f/d5c/d63/dd2/db1/cea 0 2024-03-30T04:28:31.671 INFO:tasks.workunit.client.1.smithi135.stdout:2/625: write d4/d6/d47/f86 [4607967,107484] 0 2024-03-30T04:28:31.674 INFO:tasks.workunit.client.1.smithi135.stdout:2/626: stat d4/d6/d47/f86 0 2024-03-30T04:28:31.675 INFO:tasks.workunit.client.1.smithi135.stdout:9/725: write d7/d41/f5f [2599768,32852] 0 2024-03-30T04:28:31.676 INFO:tasks.workunit.client.1.smithi135.stdout:2/627: fdatasync d4/d32/d46/fba 0 2024-03-30T04:28:31.680 INFO:tasks.workunit.client.1.smithi135.stdout:7/860: write d4/db/d1a/d5c/d2c/d51/d56/ff5 [456183,124422] 0 2024-03-30T04:28:31.680 INFO:tasks.workunit.client.1.smithi135.stdout:2/628: fsync d4/da/d8d/f95 0 2024-03-30T04:28:31.681 INFO:tasks.workunit.client.1.smithi135.stdout:0/845: dwrite d0/d5/d9/f33 [8388608,4194304] 0 2024-03-30T04:28:31.687 INFO:tasks.workunit.client.1.smithi135.stdout:9/726: creat d7/d4f/d6e/dad/feb x:0 0 0 2024-03-30T04:28:31.689 INFO:tasks.workunit.client.1.smithi135.stdout:2/629: symlink d4/d55/ld8 0 2024-03-30T04:28:31.695 INFO:tasks.workunit.client.1.smithi135.stdout:9/727: write d7/f22 [5643981,117277] 0 2024-03-30T04:28:31.700 INFO:tasks.workunit.client.1.smithi135.stdout:9/728: chown d7/d29/d78/da4/f28 6106256 1 2024-03-30T04:28:31.701 INFO:tasks.workunit.client.1.smithi135.stdout:2/630: creat d4/da/fd9 x:0 0 0 2024-03-30T04:28:31.705 INFO:tasks.workunit.client.1.smithi135.stdout:7/861: dwrite d4/fc9 [0,4194304] 0 2024-03-30T04:28:31.707 INFO:tasks.workunit.client.1.smithi135.stdout:9/729: mknod d7/d29/d78/da4/d37/dba/cec 0 2024-03-30T04:28:31.707 INFO:tasks.workunit.client.1.smithi135.stdout:2/631: dread - d4/d32/f9e zero size 2024-03-30T04:28:31.709 INFO:tasks.workunit.client.1.smithi135.stdout:7/862: read - d4/db/d1a/d5c/d59/d76/f10d zero size 2024-03-30T04:28:31.712 INFO:tasks.workunit.client.1.smithi135.stdout:9/730: getdents d7/d55/d8d/ddb/de7 0 2024-03-30T04:28:31.717 INFO:tasks.workunit.client.1.smithi135.stdout:7/863: creat d4/db/d1a/d5c/d82/f123 x:0 0 0 2024-03-30T04:28:31.722 INFO:tasks.workunit.client.1.smithi135.stdout:7/864: rename d4/fa7 to d4/db/d1a/d5c/d59/d113/f124 0 2024-03-30T04:28:31.727 INFO:tasks.workunit.client.1.smithi135.stdout:8/679: sync 2024-03-30T04:28:31.728 INFO:tasks.workunit.client.1.smithi135.stdout:2/632: truncate d4/d32/d43/f63 3944105 0 2024-03-30T04:28:31.733 INFO:tasks.workunit.client.1.smithi135.stdout:9/731: dwrite d7/d4f/d5c/d63/dd2/db1/dbc/de6/fe9 [0,4194304] 0 2024-03-30T04:28:31.739 INFO:tasks.workunit.client.1.smithi135.stdout:9/732: symlink d7/d4f/d5c/led 0 2024-03-30T04:28:31.739 INFO:tasks.workunit.client.1.smithi135.stdout:0/846: dwrite d0/d13/d53/f80 [0,4194304] 0 2024-03-30T04:28:31.744 INFO:tasks.workunit.client.1.smithi135.stdout:0/847: dread - d0/d19/d2d/fe0 zero size 2024-03-30T04:28:31.745 INFO:tasks.workunit.client.1.smithi135.stdout:0/848: dread - d0/d34/ffd zero size 2024-03-30T04:28:31.752 INFO:tasks.workunit.client.1.smithi135.stdout:9/733: stat d7/d4f/d5c/d63/dd2/db1/dbc/de6 0 2024-03-30T04:28:31.757 INFO:tasks.workunit.client.1.smithi135.stdout:8/680: dwrite d6/d8/d15/d54/d8e/fce [0,4194304] 0 2024-03-30T04:28:31.757 INFO:tasks.workunit.client.1.smithi135.stdout:7/865: dread d4/db/d1a/d5c/d2c/f39 [0,4194304] 0 2024-03-30T04:28:31.762 INFO:tasks.workunit.client.1.smithi135.stdout:9/734: dread d7/f5a [0,4194304] 0 2024-03-30T04:28:31.775 INFO:tasks.workunit.client.1.smithi135.stdout:9/735: mkdir d7/d4f/dee 0 2024-03-30T04:28:31.850 INFO:tasks.workunit.client.1.smithi135.stdout:8/681: dwrite d6/d8/d15/d54/d7e/daf/dc1/fd4 [0,4194304] 0 2024-03-30T04:28:31.854 INFO:tasks.workunit.client.1.smithi135.stdout:2/633: dread d4/d6/f1a [0,4194304] 0 2024-03-30T04:28:31.854 INFO:tasks.workunit.client.1.smithi135.stdout:9/736: unlink d7/l1e 0 2024-03-30T04:28:31.856 INFO:tasks.workunit.client.1.smithi135.stdout:7/866: dwrite d4/db/d1a/d5c/d2c/d2e/db9/f110 [0,4194304] 0 2024-03-30T04:28:31.857 INFO:tasks.workunit.client.1.smithi135.stdout:9/737: dread - d7/d29/d78/da4/d37/dba/fda zero size 2024-03-30T04:28:31.858 INFO:tasks.workunit.client.1.smithi135.stdout:8/682: fdatasync d6/d8/f16 0 2024-03-30T04:28:31.861 INFO:tasks.workunit.client.1.smithi135.stdout:9/738: mknod d7/d90/cef 0 2024-03-30T04:28:31.863 INFO:tasks.workunit.client.1.smithi135.stdout:9/739: chown d7/d4f/d6e/c83 36529847 1 2024-03-30T04:28:31.863 INFO:tasks.workunit.client.1.smithi135.stdout:7/867: symlink d4/db/d1a/d5c/d2c/d51/l125 0 2024-03-30T04:28:31.871 INFO:tasks.workunit.client.1.smithi135.stdout:8/683: read d6/d8/d14/f30 [1721921,22275] 0 2024-03-30T04:28:31.872 INFO:tasks.workunit.client.1.smithi135.stdout:9/740: dread d7/d4f/d5c/f9e [0,4194304] 0 2024-03-30T04:28:31.874 INFO:tasks.workunit.client.1.smithi135.stdout:7/868: write d4/db/d1a/d3d/ddc/d5a/dae/fce [1733310,19963] 0 2024-03-30T04:28:31.876 INFO:tasks.workunit.client.1.smithi135.stdout:2/634: truncate d4/d32/d12/d96/f8a 2009180 0 2024-03-30T04:28:31.879 INFO:tasks.workunit.client.1.smithi135.stdout:8/684: rename d6/d8/d15/d54/d8e/d72/d73/d9f to d6/d23/d9d/ddd 0 2024-03-30T04:28:31.880 INFO:tasks.workunit.client.1.smithi135.stdout:2/635: chown d4/d32/d12/d45/l84 127777 1 2024-03-30T04:28:31.881 INFO:tasks.workunit.client.1.smithi135.stdout:7/869: truncate d4/db/d1a/d5c/d2c/d51/da8/fc2 1062658 0 2024-03-30T04:28:31.883 INFO:tasks.workunit.client.1.smithi135.stdout:2/636: mkdir d4/d55/dda 0 2024-03-30T04:28:31.884 INFO:tasks.workunit.client.1.smithi135.stdout:7/870: mkdir d4/db/d79/d126 0 2024-03-30T04:28:31.885 INFO:tasks.workunit.client.1.smithi135.stdout:2/637: fsync d4/d32/d2e/d2f/f30 0 2024-03-30T04:28:31.888 INFO:tasks.workunit.client.1.smithi135.stdout:2/638: unlink d4/d6/c82 0 2024-03-30T04:28:31.890 INFO:tasks.workunit.client.1.smithi135.stdout:7/871: symlink d4/db/d1a/d3d/ddc/db0/dfd/l127 0 2024-03-30T04:28:31.892 INFO:tasks.workunit.client.1.smithi135.stdout:2/639: write d4/d6/d99/fb0 [733034,19869] 0 2024-03-30T04:28:31.893 INFO:tasks.workunit.client.1.smithi135.stdout:7/872: chown d4/db/d1a/d5c/d2c/d51/d56/dbd/fd9 20555898 1 2024-03-30T04:28:31.899 INFO:tasks.workunit.client.1.smithi135.stdout:2/640: dread d4/da/fb [0,4194304] 0 2024-03-30T04:28:31.901 INFO:tasks.workunit.client.1.smithi135.stdout:2/641: stat d4/d32/d46/dbc/dd0/ld2 0 2024-03-30T04:28:31.902 INFO:tasks.workunit.client.1.smithi135.stdout:7/873: rmdir d4/db/d1a/d3d/ddc 39 2024-03-30T04:28:31.904 INFO:tasks.workunit.client.1.smithi135.stdout:7/874: readlink d4/db/d1a/d5c/d59/d76/l99 0 2024-03-30T04:28:31.905 INFO:tasks.workunit.client.1.smithi135.stdout:2/642: getdents d4/d55/d77 0 2024-03-30T04:28:31.907 INFO:tasks.workunit.client.1.smithi135.stdout:7/875: symlink d4/db/d79/d92/dcf/l128 0 2024-03-30T04:28:31.909 INFO:tasks.workunit.client.1.smithi135.stdout:2/643: mkdir d4/d32/d2e/d67/db4/ddb 0 2024-03-30T04:28:31.911 INFO:tasks.workunit.client.1.smithi135.stdout:7/876: write d4/db/d1a/d3d/ddc/d5a/dae/fbe [756508,46079] 0 2024-03-30T04:28:31.918 INFO:tasks.workunit.client.1.smithi135.stdout:7/877: link d4/db/l53 d4/db/d1a/d5c/d2c/d51/da8/l129 0 2024-03-30T04:28:31.920 INFO:tasks.workunit.client.1.smithi135.stdout:7/878: write d4/f11 [990740,80166] 0 2024-03-30T04:28:31.925 INFO:tasks.workunit.client.1.smithi135.stdout:2/644: dwrite d4/da/f59 [0,4194304] 0 2024-03-30T04:28:31.934 INFO:tasks.workunit.client.1.smithi135.stdout:2/645: truncate d4/d32/d12/d45/f39 1021137 0 2024-03-30T04:28:31.937 INFO:tasks.workunit.client.1.smithi135.stdout:2/646: dread - d4/d32/d2e/d9f/fcc zero size 2024-03-30T04:28:31.940 INFO:tasks.workunit.client.1.smithi135.stdout:7/879: dwrite d4/db/d1a/d3d/d96/ff6 [0,4194304] 0 2024-03-30T04:28:31.946 INFO:tasks.workunit.client.1.smithi135.stdout:7/880: creat d4/db/d1a/d3d/f12a x:0 0 0 2024-03-30T04:28:31.946 INFO:tasks.workunit.client.1.smithi135.stdout:2/647: getdents d4/d32/d2e/d9f 0 2024-03-30T04:28:31.949 INFO:tasks.workunit.client.1.smithi135.stdout:2/648: write d4/d32/d12/d45/fa0 [589818,127308] 0 2024-03-30T04:28:31.950 INFO:tasks.workunit.client.1.smithi135.stdout:2/649: stat d4/d55/dda 0 2024-03-30T04:28:31.951 INFO:tasks.workunit.client.1.smithi135.stdout:7/881: creat d4/db/d1a/d5c/d2c/d2e/db9/df4/d100/f12b x:0 0 0 2024-03-30T04:28:31.952 INFO:tasks.workunit.client.1.smithi135.stdout:2/650: chown d4/d32/d46/l6e 9 1 2024-03-30T04:28:31.964 INFO:tasks.workunit.client.1.smithi135.stdout:2/651: write d4/d32/d46/dbc/dbe/fd5 [2059524,107635] 0 2024-03-30T04:28:31.967 INFO:tasks.workunit.client.1.smithi135.stdout:7/882: dwrite d4/db/d1a/d5c/d2c/d51/d56/f9e [0,4194304] 0 2024-03-30T04:28:31.967 INFO:tasks.workunit.client.1.smithi135.stdout:2/652: fsync d4/d32/d46/dbc/dbe/fd5 0 2024-03-30T04:28:31.972 INFO:tasks.workunit.client.1.smithi135.stdout:2/653: rename d4/d32/d12/d96 to d4/d55/ddc 0 2024-03-30T04:28:31.973 INFO:tasks.workunit.client.1.smithi135.stdout:7/883: truncate d4/db/d1a/d5c/d82/f106 2329720 0 2024-03-30T04:28:31.977 INFO:tasks.workunit.client.1.smithi135.stdout:2/654: rename d4/da/d7a/fc7 to d4/d6/d99/fdd 0 2024-03-30T04:28:31.978 INFO:tasks.workunit.client.1.smithi135.stdout:7/884: mknod d4/db/d1a/d3d/ddc/db0/c12c 0 2024-03-30T04:28:31.979 INFO:tasks.workunit.client.1.smithi135.stdout:2/655: fsync d4/d32/d12/d45/fa0 0 2024-03-30T04:28:31.981 INFO:tasks.workunit.client.1.smithi135.stdout:2/656: chown d4/d32/d43/f92 405 1 2024-03-30T04:28:31.982 INFO:tasks.workunit.client.1.smithi135.stdout:7/885: symlink d4/db/d1a/d5c/d59/d76/l12d 0 2024-03-30T04:28:31.986 INFO:tasks.workunit.client.1.smithi135.stdout:2/657: mkdir d4/d32/dc2/dde 0 2024-03-30T04:28:31.987 INFO:tasks.workunit.client.1.smithi135.stdout:5/686: sync 2024-03-30T04:28:31.988 INFO:tasks.workunit.client.1.smithi135.stdout:4/700: sync 2024-03-30T04:28:31.988 INFO:tasks.workunit.client.1.smithi135.stdout:3/733: sync 2024-03-30T04:28:31.989 INFO:tasks.workunit.client.1.smithi135.stdout:2/658: write d4/d6/f61 [984648,32844] 0 2024-03-30T04:28:31.990 INFO:tasks.workunit.client.1.smithi135.stdout:7/886: dread d4/db/d1a/d5c/d2c/d51/da8/fc2 [0,4194304] 0 2024-03-30T04:28:31.992 INFO:tasks.workunit.client.1.smithi135.stdout:3/734: write d3/d50/fc8 [1042605,81406] 0 2024-03-30T04:28:31.995 INFO:tasks.workunit.client.1.smithi135.stdout:2/659: dread - d4/da/d7a/f7c zero size 2024-03-30T04:28:31.995 INFO:tasks.workunit.client.1.smithi135.stdout:4/701: read d10/f17 [1616765,70605] 0 2024-03-30T04:28:31.997 INFO:tasks.workunit.client.1.smithi135.stdout:7/887: write d4/db/d1a/d3d/fd4 [2592681,62501] 0 2024-03-30T04:28:31.998 INFO:tasks.workunit.client.1.smithi135.stdout:4/702: chown d10/d1a/d42/d66/dc7 1931165423 1 2024-03-30T04:28:31.998 INFO:tasks.workunit.client.1.smithi135.stdout:3/735: unlink d3/d50/l40 0 2024-03-30T04:28:32.001 INFO:tasks.workunit.client.1.smithi135.stdout:4/703: mkdir d10/d1a/d42/d46/def 0 2024-03-30T04:28:32.004 INFO:tasks.workunit.client.1.smithi135.stdout:2/660: truncate d4/d32/d12/f3f 562280 0 2024-03-30T04:28:32.009 INFO:tasks.workunit.client.1.smithi135.stdout:7/888: rename d4/db/d1a/d5c/d59/d76/de4/dee to d4/db/d1a/d4b/d40/d6e/d12e 0 2024-03-30T04:28:32.011 INFO:tasks.workunit.client.1.smithi135.stdout:2/661: creat d4/da/d7a/dd6/fdf x:0 0 0 2024-03-30T04:28:32.013 INFO:tasks.workunit.client.1.smithi135.stdout:4/704: rename d10/d1a/d32/d40/f76 to d10/d1a/d42/de0/ff0 0 2024-03-30T04:28:32.013 INFO:tasks.workunit.client.1.smithi135.stdout:5/687: dwrite d1/d7/d11/f8a [0,4194304] 0 2024-03-30T04:28:32.015 INFO:tasks.workunit.client.1.smithi135.stdout:4/705: chown d10/d1a/d3d 0 1 2024-03-30T04:28:32.017 INFO:tasks.workunit.client.1.smithi135.stdout:7/889: mknod d4/db/d1a/d3d/ddc/db0/dc4/c12f 0 2024-03-30T04:28:32.023 INFO:tasks.workunit.client.1.smithi135.stdout:7/890: write d4/db/d1a/d5c/d2c/d51/da8/f108 [1000615,3787] 0 2024-03-30T04:28:32.027 INFO:tasks.workunit.client.1.smithi135.stdout:3/736: dread d3/d6e/f80 [4194304,4194304] 0 2024-03-30T04:28:32.031 INFO:tasks.workunit.client.1.smithi135.stdout:7/891: rmdir d4/db/d1a/d5c/d2c/d83/d118 0 2024-03-30T04:28:32.032 INFO:tasks.workunit.client.1.smithi135.stdout:2/662: dwrite d4/d32/d46/dbc/dbe/fd5 [0,4194304] 0 2024-03-30T04:28:32.033 INFO:tasks.workunit.client.1.smithi135.stdout:3/737: mkdir d3/d6e/dfc 0 2024-03-30T04:28:32.037 INFO:tasks.workunit.client.1.smithi135.stdout:7/892: mkdir d4/db/d1a/d3d/ddc/db0/de1/d130 0 2024-03-30T04:28:32.039 INFO:tasks.workunit.client.1.smithi135.stdout:7/893: rmdir d4/db/d1a/d5c 39 2024-03-30T04:28:32.040 INFO:tasks.workunit.client.1.smithi135.stdout:3/738: getdents d3/d50/d22/d33 0 2024-03-30T04:28:32.044 INFO:tasks.workunit.client.1.smithi135.stdout:3/739: fsync d3/d50/d46/f65 0 2024-03-30T04:28:32.050 INFO:tasks.workunit.client.1.smithi135.stdout:7/894: creat d4/db/d1a/d5c/d59/d113/f131 x:0 0 0 2024-03-30T04:28:32.053 INFO:tasks.workunit.client.1.smithi135.stdout:3/740: write d3/d50/d46/f57 [1237747,122754] 0 2024-03-30T04:28:32.055 INFO:tasks.workunit.client.1.smithi135.stdout:5/688: dread d1/d7/f10 [0,4194304] 0 2024-03-30T04:28:32.059 INFO:tasks.workunit.client.1.smithi135.stdout:7/895: rmdir d4/db/d1a/d3d/ddc/d5a 39 2024-03-30T04:28:32.062 INFO:tasks.workunit.client.1.smithi135.stdout:2/663: dread d4/d32/d2e/f48 [0,4194304] 0 2024-03-30T04:28:32.064 INFO:tasks.workunit.client.1.smithi135.stdout:7/896: write d4/db/d1a/d5c/d2c/f39 [5147031,20509] 0 2024-03-30T04:28:32.067 INFO:tasks.workunit.client.1.smithi135.stdout:2/664: symlink d4/d32/d2e/le0 0 2024-03-30T04:28:32.070 INFO:tasks.workunit.client.1.smithi135.stdout:7/897: mknod d4/db/c132 0 2024-03-30T04:28:32.073 INFO:tasks.workunit.client.1.smithi135.stdout:2/665: creat d4/d32/d2e/d67/dad/fe1 x:0 0 0 2024-03-30T04:28:32.076 INFO:tasks.workunit.client.1.smithi135.stdout:5/689: dwrite d1/d3/fa4 [0,4194304] 0 2024-03-30T04:28:32.080 INFO:tasks.workunit.client.1.smithi135.stdout:2/666: creat d4/da/d8d/fe2 x:0 0 0 2024-03-30T04:28:32.081 INFO:tasks.workunit.client.1.smithi135.stdout:5/690: mknod d1/d7/d11/d3c/d4b/cd3 0 2024-03-30T04:28:32.083 INFO:tasks.workunit.client.1.smithi135.stdout:7/898: write d4/db/d1a/d5c/d5d/f8d [3205495,26518] 0 2024-03-30T04:28:32.086 INFO:tasks.workunit.client.1.smithi135.stdout:5/691: chown d1/d7/fcb 329355 1 2024-03-30T04:28:32.088 INFO:tasks.workunit.client.1.smithi135.stdout:3/741: dread d3/d50/d22/fc0 [0,4194304] 0 2024-03-30T04:28:32.088 INFO:tasks.workunit.client.1.smithi135.stdout:5/692: read d1/d7/d11/f8a [595492,104839] 0 2024-03-30T04:28:32.091 INFO:tasks.workunit.client.1.smithi135.stdout:3/742: read d3/d50/d46/f56 [16156,27530] 0 2024-03-30T04:28:32.092 INFO:tasks.workunit.client.1.smithi135.stdout:7/899: rename d4/db/d1a/d5c/d2c/d51/da8/l7a to d4/db/d9f/l133 0 2024-03-30T04:28:32.094 INFO:tasks.workunit.client.1.smithi135.stdout:3/743: fsync d3/d50/d22/f3d 0 2024-03-30T04:28:32.097 INFO:tasks.workunit.client.1.smithi135.stdout:5/693: getdents d1/d3/d80 0 2024-03-30T04:28:32.101 INFO:tasks.workunit.client.1.smithi135.stdout:3/744: creat d3/d10/d16/d23/dc7/ffd x:0 0 0 2024-03-30T04:28:32.102 INFO:tasks.workunit.client.1.smithi135.stdout:2/667: dwrite d4/da/fd9 [0,4194304] 0 2024-03-30T04:28:32.103 INFO:tasks.workunit.client.1.smithi135.stdout:5/694: truncate d1/d7/d1a/d1f/d9b/fb8 522013 0 2024-03-30T04:28:32.106 INFO:tasks.workunit.client.1.smithi135.stdout:2/668: creat d4/d55/ddc/fe3 x:0 0 0 2024-03-30T04:28:32.108 INFO:tasks.workunit.client.1.smithi135.stdout:3/745: link d3/d10/d16/fed d3/d10/ffe 0 2024-03-30T04:28:32.110 INFO:tasks.workunit.client.1.smithi135.stdout:3/746: stat d3/d10/d16/d23/d6a/d94/ff2 0 2024-03-30T04:28:32.115 INFO:tasks.workunit.client.1.smithi135.stdout:3/747: creat d3/db/da9/fff x:0 0 0 2024-03-30T04:28:32.122 INFO:tasks.workunit.client.1.smithi135.stdout:3/748: write d3/f1d [2441441,13181] 0 2024-03-30T04:28:32.124 INFO:tasks.workunit.client.1.smithi135.stdout:5/695: dread d1/d3/d2b/f5b [0,4194304] 0 2024-03-30T04:28:32.126 INFO:tasks.workunit.client.1.smithi135.stdout:2/669: dwrite d4/da/f44 [0,4194304] 0 2024-03-30T04:28:32.129 INFO:tasks.workunit.client.1.smithi135.stdout:2/670: readlink d4/d55/d77/l51 0 2024-03-30T04:28:32.130 INFO:tasks.workunit.client.1.smithi135.stdout:6/818: sync 2024-03-30T04:28:32.130 INFO:tasks.workunit.client.1.smithi135.stdout:1/763: sync 2024-03-30T04:28:32.131 INFO:tasks.workunit.client.1.smithi135.stdout:7/900: dread d4/db/d79/dbc/fa5 [0,4194304] 0 2024-03-30T04:28:32.131 INFO:tasks.workunit.client.1.smithi135.stdout:5/696: read d1/f4f [4958183,72228] 0 2024-03-30T04:28:32.132 INFO:tasks.workunit.client.1.smithi135.stdout:6/819: mkdir d2/d36/d76/d107 0 2024-03-30T04:28:32.133 INFO:tasks.workunit.client.1.smithi135.stdout:3/749: write d3/f11 [3014843,72300] 0 2024-03-30T04:28:32.135 INFO:tasks.workunit.client.1.smithi135.stdout:1/764: mkdir d16/d29/d61/dfd 0 2024-03-30T04:28:32.137 INFO:tasks.workunit.client.1.smithi135.stdout:1/765: fdatasync d16/d5a/dad/f7b 0 2024-03-30T04:28:32.138 INFO:tasks.workunit.client.1.smithi135.stdout:2/671: getdents d4/d32/d46/dbc/dbe 0 2024-03-30T04:28:32.139 INFO:tasks.workunit.client.1.smithi135.stdout:2/672: dread - d4/d6/fb9 zero size 2024-03-30T04:28:32.141 INFO:tasks.workunit.client.1.smithi135.stdout:3/750: getdents d3/d50/d46/d4c 0 2024-03-30T04:28:32.147 INFO:tasks.workunit.client.1.smithi135.stdout:3/751: write d3/d54/ff6 [141237,13646] 0 2024-03-30T04:28:32.149 INFO:tasks.workunit.client.1.smithi135.stdout:2/673: mknod d4/d32/d12/d45/dac/ce4 0 2024-03-30T04:28:32.152 INFO:tasks.workunit.client.1.smithi135.stdout:3/752: stat d3/d50/d22/c26 0 2024-03-30T04:28:32.155 INFO:tasks.workunit.client.1.smithi135.stdout:2/674: mkdir d4/d32/d2e/d67/dad/de5 0 2024-03-30T04:28:32.159 INFO:tasks.workunit.client.1.smithi135.stdout:7/901: dwrite d4/db/d1a/d5c/d2c/d2e/fe5 [0,4194304] 0 2024-03-30T04:28:32.161 INFO:tasks.workunit.client.1.smithi135.stdout:2/675: write d4/da/d8d/fe2 [892645,36172] 0 2024-03-30T04:28:32.166 INFO:tasks.workunit.client.1.smithi135.stdout:7/902: chown d4/fc9 1579242 1 2024-03-30T04:28:32.166 INFO:tasks.workunit.client.1.smithi135.stdout:3/753: rename d3/d10/d16/d23/d6a/d86/fee to d3/d10/d9c/dbe/dd6/dd7/f100 0 2024-03-30T04:28:32.167 INFO:tasks.workunit.client.1.smithi135.stdout:1/766: dwrite d16/d5a/d5f/d3d/d8d/dc9/da4/d26/feb [0,4194304] 0 2024-03-30T04:28:32.169 INFO:tasks.workunit.client.1.smithi135.stdout:5/697: dread d1/d3/f38 [0,4194304] 0 2024-03-30T04:28:32.170 INFO:tasks.workunit.client.1.smithi135.stdout:2/676: mkdir d4/da/d7a/de6 0 2024-03-30T04:28:32.173 INFO:tasks.workunit.client.1.smithi135.stdout:5/698: stat d1/d3/d2b/f5b 0 2024-03-30T04:28:32.178 INFO:tasks.workunit.client.1.smithi135.stdout:3/754: symlink d3/d10/d16/d41/d83/l101 0 2024-03-30T04:28:32.179 INFO:tasks.workunit.client.1.smithi135.stdout:1/767: dread d16/d29/d32/f58 [0,4194304] 0 2024-03-30T04:28:32.185 INFO:tasks.workunit.client.1.smithi135.stdout:2/677: rename d4/d55/ddc to d4/d32/d12/d45/d7d/de7 0 2024-03-30T04:28:32.188 INFO:tasks.workunit.client.1.smithi135.stdout:7/903: link d4/db/d1a/d3d/ddc/db0/dfd/fa1 d4/db/d1a/d3d/ddc/db0/f134 0 2024-03-30T04:28:32.190 INFO:tasks.workunit.client.1.smithi135.stdout:5/699: getdents d1/d45 0 2024-03-30T04:28:32.192 INFO:tasks.workunit.client.1.smithi135.stdout:5/700: write d1/d7/fcb [772579,67850] 0 2024-03-30T04:28:32.194 INFO:tasks.workunit.client.1.smithi135.stdout:7/904: creat d4/db/d1a/d5c/d2c/d51/d56/d81/f135 x:0 0 0 2024-03-30T04:28:32.197 INFO:tasks.workunit.client.1.smithi135.stdout:1/768: dwrite d16/d5a/d5f/d3d/d8d/dc9/da4/f65 [0,4194304] 0 2024-03-30T04:28:32.201 INFO:tasks.workunit.client.1.smithi135.stdout:1/769: stat d16/d29/d61/f98 0 2024-03-30T04:28:32.204 INFO:tasks.workunit.client.1.smithi135.stdout:7/905: symlink d4/db/d1a/d5c/d2c/d2e/db9/df4/l136 0 2024-03-30T04:28:32.210 INFO:tasks.workunit.client.1.smithi135.stdout:2/678: dwrite d4/d32/d2e/f8c [0,4194304] 0 2024-03-30T04:28:32.213 INFO:tasks.workunit.client.1.smithi135.stdout:5/701: dread d1/d3/fa4 [0,4194304] 0 2024-03-30T04:28:32.213 INFO:tasks.workunit.client.1.smithi135.stdout:2/679: dread - d4/da/d7a/f88 zero size 2024-03-30T04:28:32.216 INFO:tasks.workunit.client.1.smithi135.stdout:1/770: rename d16/d5a/d5f/d3d/d8d/dc9/da4/d26/f53 to d16/d5a/ffe 0 2024-03-30T04:28:32.219 INFO:tasks.workunit.client.1.smithi135.stdout:5/702: rename d1/d7/d11/f30 to d1/d7/d11/d3c/fd4 0 2024-03-30T04:28:32.229 INFO:tasks.workunit.client.1.smithi135.stdout:2/680: dread d4/d32/d2e/f8c [0,4194304] 0 2024-03-30T04:28:32.229 INFO:tasks.workunit.client.1.smithi135.stdout:1/771: dread d16/d5a/d5f/d3d/d8d/dc9/fd5 [0,4194304] 0 2024-03-30T04:28:32.232 INFO:tasks.workunit.client.1.smithi135.stdout:2/681: symlink d4/d32/d2e/d67/db4/le8 0 2024-03-30T04:28:32.235 INFO:tasks.workunit.client.1.smithi135.stdout:2/682: write d4/da/d7a/dd6/fdf [564833,40429] 0 2024-03-30T04:28:32.236 INFO:tasks.workunit.client.1.smithi135.stdout:5/703: dwrite d1/d45/d52/f90 [0,4194304] 0 2024-03-30T04:28:32.241 INFO:tasks.workunit.client.1.smithi135.stdout:2/683: fsync d4/d6/d47/f5a 0 2024-03-30T04:28:32.246 INFO:tasks.workunit.client.1.smithi135.stdout:2/684: truncate d4/d32/d2e/d67/dad/fe1 252771 0 2024-03-30T04:28:32.247 INFO:tasks.workunit.client.1.smithi135.stdout:5/704: link d1/d7/fcb d1/d3/d2b/d54/fd5 0 2024-03-30T04:28:32.248 INFO:tasks.workunit.client.1.smithi135.stdout:2/685: chown d4/d32/d12/d45/d3c/c54 655240151 1 2024-03-30T04:28:32.249 INFO:tasks.workunit.client.1.smithi135.stdout:7/906: dread d4/db/d1a/d3d/ddc/db0/fc1 [0,4194304] 0 2024-03-30T04:28:32.252 INFO:tasks.workunit.client.1.smithi135.stdout:2/686: fsync d4/f1c 0 2024-03-30T04:28:32.253 INFO:tasks.workunit.client.1.smithi135.stdout:1/772: write d16/d5a/d5f/d3d/d8d/dc9/f84 [1037751,101563] 0 2024-03-30T04:28:32.255 INFO:tasks.workunit.client.1.smithi135.stdout:7/907: creat d4/d122/f137 x:0 0 0 2024-03-30T04:28:32.256 INFO:tasks.workunit.client.1.smithi135.stdout:1/773: read d16/d29/d2c/d3c/f78 [3632896,33625] 0 2024-03-30T04:28:32.257 INFO:tasks.workunit.client.1.smithi135.stdout:2/687: symlink d4/d32/d43/db5/dcb/le9 0 2024-03-30T04:28:32.262 INFO:tasks.workunit.client.1.smithi135.stdout:1/774: symlink d16/d5a/d5f/d3a/lff 0 2024-03-30T04:28:32.263 INFO:tasks.workunit.client.1.smithi135.stdout:1/775: stat d16/d29/d32 0 2024-03-30T04:28:32.274 INFO:tasks.workunit.client.1.smithi135.stdout:7/908: dwrite d4/db/d1a/d5c/d2c/d51/d56/d81/fef [0,4194304] 0 2024-03-30T04:28:32.276 INFO:tasks.workunit.client.1.smithi135.stdout:2/688: dwrite d4/d55/f5f [4194304,4194304] 0 2024-03-30T04:28:32.283 INFO:tasks.workunit.client.1.smithi135.stdout:2/689: write d4/d6/fb9 [134283,93758] 0 2024-03-30T04:28:32.285 INFO:tasks.workunit.client.1.smithi135.stdout:0/849: sync 2024-03-30T04:28:32.285 INFO:tasks.workunit.client.1.smithi135.stdout:9/741: sync 2024-03-30T04:28:32.287 INFO:tasks.workunit.client.1.smithi135.stdout:7/909: rename d4/db/d1a/d5c/d2c/d51/d56/dbd/ce2 to d4/db/d1a/d90/dea/dfb/c138 0 2024-03-30T04:28:32.288 INFO:tasks.workunit.client.1.smithi135.stdout:1/776: dwrite d16/d5a/d5f/d3a/fb1 [0,4194304] 0 2024-03-30T04:28:32.294 INFO:tasks.workunit.client.1.smithi135.stdout:9/742: mkdir d7/d55/d8d/ddb/df0 0 2024-03-30T04:28:32.294 INFO:tasks.workunit.client.1.smithi135.stdout:7/910: unlink d4/db/lf7 0 2024-03-30T04:28:32.301 INFO:tasks.workunit.client.1.smithi135.stdout:9/743: rename d7/d41/f43 to d7/d29/ff1 0 2024-03-30T04:28:32.302 INFO:tasks.workunit.client.1.smithi135.stdout:7/911: creat d4/db/d1a/d4b/d40/d6e/d12e/f139 x:0 0 0 2024-03-30T04:28:32.305 INFO:tasks.workunit.client.1.smithi135.stdout:9/744: creat d7/d4f/d5c/d63/dd2/db1/dbc/ff2 x:0 0 0 2024-03-30T04:28:32.310 INFO:tasks.workunit.client.1.smithi135.stdout:9/745: creat d7/d29/d78/da4/d37/dba/ff3 x:0 0 0 2024-03-30T04:28:32.313 INFO:tasks.workunit.client.1.smithi135.stdout:1/777: dwrite d16/d5a/d5f/d3d/fca [0,4194304] 0 2024-03-30T04:28:32.317 INFO:tasks.workunit.client.1.smithi135.stdout:9/746: creat d7/d4f/d5c/d63/dd2/db1/dbc/de6/ff4 x:0 0 0 2024-03-30T04:28:32.318 INFO:tasks.workunit.client.1.smithi135.stdout:0/850: dwrite d0/d13/d1e/d3a/f5a [0,4194304] 0 2024-03-30T04:28:32.324 INFO:tasks.workunit.client.1.smithi135.stdout:7/912: dwrite d4/db/d1a/d5c/d2c/d51/da8/fb7 [0,4194304] 0 2024-03-30T04:28:32.324 INFO:tasks.workunit.client.1.smithi135.stdout:0/851: creat d0/d19/d2d/f118 x:0 0 0 2024-03-30T04:28:32.329 INFO:tasks.workunit.client.1.smithi135.stdout:9/747: rmdir d7/d4f/d8f 0 2024-03-30T04:28:32.330 INFO:tasks.workunit.client.1.smithi135.stdout:0/852: creat d0/d13/d53/d7f/f119 x:0 0 0 2024-03-30T04:28:32.334 INFO:tasks.workunit.client.1.smithi135.stdout:7/913: rename d4/l27 to d4/db/d1a/d5c/d59/d76/de4/dfc/l13a 0 2024-03-30T04:28:32.337 INFO:tasks.workunit.client.1.smithi135.stdout:9/748: rmdir d7/d4f/d5c/d63/dd2/db1/dbc 39 2024-03-30T04:28:32.342 INFO:tasks.workunit.client.1.smithi135.stdout:0/853: mknod d0/d13/d91/dd2/c11a 0 2024-03-30T04:28:32.346 INFO:tasks.workunit.client.1.smithi135.stdout:9/749: unlink d7/d4f/d5c/fc1 0 2024-03-30T04:28:32.346 INFO:tasks.workunit.client.1.smithi135.stdout:1/778: dwrite d16/d5a/d5f/d3a/fa7 [0,4194304] 0 2024-03-30T04:28:32.352 INFO:tasks.workunit.client.1.smithi135.stdout:0/854: dread d0/d5/d9/fb [0,4194304] 0 2024-03-30T04:28:32.352 INFO:tasks.workunit.client.1.smithi135.stdout:1/779: mkdir d16/d5a/d5f/d3d/d8d/dc9/d100 0 2024-03-30T04:28:32.355 INFO:tasks.workunit.client.1.smithi135.stdout:7/914: dwrite d4/ff9 [0,4194304] 0 2024-03-30T04:28:32.358 INFO:tasks.workunit.client.1.smithi135.stdout:7/915: dread - d4/db/d79/d92/dcf/fd7 zero size 2024-03-30T04:28:32.364 INFO:tasks.workunit.client.1.smithi135.stdout:7/916: creat d4/db/d1a/d5c/d2c/d51/d11d/f13b x:0 0 0 2024-03-30T04:28:32.389 INFO:tasks.workunit.client.1.smithi135.stdout:0/855: dread d0/d5/d9/d23/d2a/d60/f109 [0,4194304] 0 2024-03-30T04:28:32.393 INFO:tasks.workunit.client.1.smithi135.stdout:0/856: unlink d0/d13/d53/d54/d99/la6 0 2024-03-30T04:28:32.406 INFO:tasks.workunit.client.1.smithi135.stdout:8/685: sync 2024-03-30T04:28:32.407 INFO:tasks.workunit.client.1.smithi135.stdout:7/917: dread d4/db/d9f/fba [0,4194304] 0 2024-03-30T04:28:32.417 INFO:tasks.workunit.client.1.smithi135.stdout:0/857: creat d0/d13/d1e/d3a/dc2/dd1/f11b x:0 0 0 2024-03-30T04:28:32.418 INFO:tasks.workunit.client.1.smithi135.stdout:8/686: mknod d6/d8/d15/d54/d8e/dbd/cde 0 2024-03-30T04:28:32.420 INFO:tasks.workunit.client.1.smithi135.stdout:7/918: getdents d4/db/d1a/d4b/d40/d6e/d12e 0 2024-03-30T04:28:32.422 INFO:tasks.workunit.client.1.smithi135.stdout:0/858: mkdir d0/d5/d9/d23/d2a/d60/de8/d11c 0 2024-03-30T04:28:32.423 INFO:tasks.workunit.client.1.smithi135.stdout:7/919: write d4/db/d1a/d90/f11b [246429,20424] 0 2024-03-30T04:28:32.432 INFO:tasks.workunit.client.1.smithi135.stdout:8/687: rename d6/d8/d15/d54/d8e/d72/f95 to d6/d8/fdf 0 2024-03-30T04:28:32.433 INFO:tasks.workunit.client.1.smithi135.stdout:7/920: mkdir d4/db/d1a/d5c/d59/d76/de4/d13c 0 2024-03-30T04:28:32.433 INFO:tasks.workunit.client.1.smithi135.stdout:8/688: chown d6/d8/c66 1588 1 2024-03-30T04:28:32.436 INFO:tasks.workunit.client.1.smithi135.stdout:7/921: symlink d4/db/d1a/d4b/l13d 0 2024-03-30T04:28:32.437 INFO:tasks.workunit.client.1.smithi135.stdout:8/689: creat d6/d8/d15/d54/d8e/fe0 x:0 0 0 2024-03-30T04:28:32.439 INFO:tasks.workunit.client.1.smithi135.stdout:7/922: fsync d4/db/d1a/d5c/d2c/f63 0 2024-03-30T04:28:32.443 INFO:tasks.workunit.client.1.smithi135.stdout:7/923: mkdir d4/db/d1a/d5c/d2c/d51/da8/d13e 0 2024-03-30T04:28:32.445 INFO:tasks.workunit.client.1.smithi135.stdout:8/690: link d6/d8/d14/da0/fb6 d6/d8/d15/d54/d7e/daf/dc1/fe1 0 2024-03-30T04:28:32.445 INFO:tasks.workunit.client.1.smithi135.stdout:7/924: dread - d4/db/d1a/d3d/f12a zero size 2024-03-30T04:28:32.447 INFO:tasks.workunit.client.1.smithi135.stdout:8/691: symlink d6/d23/d9d/le2 0 2024-03-30T04:28:32.448 INFO:tasks.workunit.client.1.smithi135.stdout:8/692: read - d6/d23/f82 zero size 2024-03-30T04:28:32.450 INFO:tasks.workunit.client.1.smithi135.stdout:0/859: rmdir d0 39 2024-03-30T04:28:32.452 INFO:tasks.workunit.client.1.smithi135.stdout:8/693: mknod d6/d8/d15/d2a/ce3 0 2024-03-30T04:28:32.453 INFO:tasks.workunit.client.1.smithi135.stdout:0/860: symlink d0/db5/l11d 0 2024-03-30T04:28:32.459 INFO:tasks.workunit.client.1.smithi135.stdout:8/694: link d6/d8/d15/d2a/c6f d6/d8/d15/d1f/ddb/ce4 0 2024-03-30T04:28:32.463 INFO:tasks.workunit.client.1.smithi135.stdout:7/925: dwrite d4/db/d1a/d90/f11b [0,4194304] 0 2024-03-30T04:28:32.466 INFO:tasks.workunit.client.1.smithi135.stdout:8/695: truncate d6/d8/d15/d2a/f33 906626 0 2024-03-30T04:28:32.467 INFO:tasks.workunit.client.1.smithi135.stdout:7/926: write d4/db/d79/d92/dcf/fd7 [917169,12502] 0 2024-03-30T04:28:32.470 INFO:tasks.workunit.client.1.smithi135.stdout:8/696: chown d6/d8/d15/d1f/ddb 7444 1 2024-03-30T04:28:32.475 INFO:tasks.workunit.client.1.smithi135.stdout:8/697: read d6/d8/d15/d2a/f3c [1273059,4570] 0 2024-03-30T04:28:32.478 INFO:tasks.workunit.client.1.smithi135.stdout:0/861: dwrite d0/d5/d9/d23/fa4 [0,4194304] 0 2024-03-30T04:28:32.481 INFO:tasks.workunit.client.1.smithi135.stdout:0/862: dread - d0/d34/dfc/f4a zero size 2024-03-30T04:28:32.484 INFO:tasks.workunit.client.1.smithi135.stdout:8/698: dread d6/d8/d15/d54/d8e/faa [0,4194304] 0 2024-03-30T04:28:32.487 INFO:tasks.workunit.client.1.smithi135.stdout:8/699: fsync d6/d8/d15/d1f/f4c 0 2024-03-30T04:28:32.490 INFO:tasks.workunit.client.1.smithi135.stdout:0/863: rename d0/d34/dfc/d63 to d0/d13/d91/d106/d11e 0 2024-03-30T04:28:32.490 INFO:tasks.workunit.client.1.smithi135.stdout:8/700: write d6/d8/f3b [349302,50003] 0 2024-03-30T04:28:32.494 INFO:tasks.workunit.client.1.smithi135.stdout:0/864: readlink d0/d5/d42/d87/lcd 0 2024-03-30T04:28:32.496 INFO:tasks.workunit.client.1.smithi135.stdout:8/701: chown d6/d8/d15/d2a/d35/c3a 642900 1 2024-03-30T04:28:32.496 INFO:tasks.workunit.client.1.smithi135.stdout:7/927: dwrite d4/db/d1a/d5c/d59/d76/f10d [0,4194304] 0 2024-03-30T04:28:32.497 INFO:tasks.workunit.client.1.smithi135.stdout:0/865: write d0/d13/d1e/f78 [5480173,79698] 0 2024-03-30T04:28:32.502 INFO:tasks.workunit.client.1.smithi135.stdout:0/866: rmdir d0/d5/d9/d23/d2a/d60/de8 39 2024-03-30T04:28:32.503 INFO:tasks.workunit.client.1.smithi135.stdout:0/867: readlink d0/d5/d42/d87/la2 0 2024-03-30T04:28:32.504 INFO:tasks.workunit.client.1.smithi135.stdout:0/868: dread - d0/d13/d1e/d3a/f72 zero size 2024-03-30T04:28:32.506 INFO:tasks.workunit.client.1.smithi135.stdout:0/869: symlink d0/d5/d42/l11f 0 2024-03-30T04:28:32.514 INFO:tasks.workunit.client.1.smithi135.stdout:8/702: dwrite d6/f11 [4194304,4194304] 0 2024-03-30T04:28:32.516 INFO:tasks.workunit.client.1.smithi135.stdout:8/703: stat d6/d8/d15/d54/f5d 0 2024-03-30T04:28:32.518 INFO:tasks.workunit.client.1.smithi135.stdout:0/870: dread d0/d34/dfc/d3b/ffa [0,4194304] 0 2024-03-30T04:28:32.519 INFO:tasks.workunit.client.1.smithi135.stdout:8/704: mkdir d6/de5 0 2024-03-30T04:28:32.520 INFO:tasks.workunit.client.1.smithi135.stdout:0/871: write d0/d19/d2d/fe0 [949242,14681] 0 2024-03-30T04:28:32.532 INFO:tasks.workunit.client.1.smithi135.stdout:0/872: dwrite d0/d1/fab [0,4194304] 0 2024-03-30T04:28:32.561 INFO:tasks.workunit.client.1.smithi135.stdout:0/873: dwrite d0/d5/d10/fca [0,4194304] 0 2024-03-30T04:28:32.563 INFO:tasks.workunit.client.1.smithi135.stdout:0/874: readlink d0/d13/d91/d106/d11e/le2 0 2024-03-30T04:28:32.567 INFO:tasks.workunit.client.1.smithi135.stdout:0/875: unlink d0/d1/f5e 0 2024-03-30T04:28:32.575 INFO:tasks.workunit.client.1.smithi135.stdout:7/928: dread d4/db/d1a/f24 [0,4194304] 0 2024-03-30T04:28:32.577 INFO:tasks.workunit.client.1.smithi135.stdout:4/706: sync 2024-03-30T04:28:32.578 INFO:tasks.workunit.client.1.smithi135.stdout:0/876: truncate d0/d13/d91/d106/d7e/fe5 135121 0 2024-03-30T04:28:32.579 INFO:tasks.workunit.client.1.smithi135.stdout:8/705: dread d6/d8/d15/d54/f58 [0,4194304] 0 2024-03-30T04:28:32.580 INFO:tasks.workunit.client.1.smithi135.stdout:4/707: mknod d10/d72/d7c/cf1 0 2024-03-30T04:28:32.584 INFO:tasks.workunit.client.1.smithi135.stdout:8/706: creat d6/d8/d15/d1f/ddb/fe6 x:0 0 0 2024-03-30T04:28:32.586 INFO:tasks.workunit.client.1.smithi135.stdout:7/929: dread d4/f11 [0,4194304] 0 2024-03-30T04:28:32.587 INFO:tasks.workunit.client.1.smithi135.stdout:0/877: rename d0/d13/d91/d106/dd5/df1 to d0/d5/d9/d23/d2a/d60/d120 0 2024-03-30T04:28:32.587 INFO:tasks.workunit.client.1.smithi135.stdout:8/707: rmdir d6/d8/d15/d54/d8e/dbd 39 2024-03-30T04:28:32.590 INFO:tasks.workunit.client.1.smithi135.stdout:0/878: read - d0/d19/fd3 zero size 2024-03-30T04:28:32.592 INFO:tasks.workunit.client.1.smithi135.stdout:0/879: stat d0/d5/d42/d87 0 2024-03-30T04:28:32.595 INFO:tasks.workunit.client.1.smithi135.stdout:4/708: rename d10/d1a/d32/c3c to d10/d1a/d32/d40/ddf/cf2 0 2024-03-30T04:28:32.599 INFO:tasks.workunit.client.1.smithi135.stdout:4/709: write d10/d1a/d42/f62 [1554146,76411] 0 2024-03-30T04:28:32.599 INFO:tasks.workunit.client.1.smithi135.stdout:7/930: dread d4/db/d1a/d90/f11b [0,4194304] 0 2024-03-30T04:28:32.604 INFO:tasks.workunit.client.1.smithi135.stdout:4/710: creat d10/ff3 x:0 0 0 2024-03-30T04:28:32.606 INFO:tasks.workunit.client.1.smithi135.stdout:8/708: dwrite d6/d8/d14/f63 [0,4194304] 0 2024-03-30T04:28:32.608 INFO:tasks.workunit.client.1.smithi135.stdout:7/931: link d4/db/d1a/l93 d4/db/d1a/l13f 0 2024-03-30T04:28:32.614 INFO:tasks.workunit.client.1.smithi135.stdout:4/711: creat d10/d1a/d32/d40/d92/db9/dcf/de8/ff4 x:0 0 0 2024-03-30T04:28:32.623 INFO:tasks.workunit.client.1.smithi135.stdout:3/755: sync 2024-03-30T04:28:32.623 INFO:tasks.workunit.client.1.smithi135.stdout:5/705: sync 2024-03-30T04:28:32.624 INFO:tasks.workunit.client.1.smithi135.stdout:2/690: sync 2024-03-30T04:28:32.624 INFO:tasks.workunit.client.1.smithi135.stdout:9/750: sync 2024-03-30T04:28:32.624 INFO:tasks.workunit.client.1.smithi135.stdout:6/820: sync 2024-03-30T04:28:32.624 INFO:tasks.workunit.client.1.smithi135.stdout:1/780: sync 2024-03-30T04:28:32.625 INFO:tasks.workunit.client.1.smithi135.stdout:5/706: stat d1/d3/d76/f98 0 2024-03-30T04:28:32.627 INFO:tasks.workunit.client.1.smithi135.stdout:0/880: dread d0/d19/f5c [0,4194304] 0 2024-03-30T04:28:32.630 INFO:tasks.workunit.client.1.smithi135.stdout:3/756: symlink d3/d54/dab/l102 0 2024-03-30T04:28:32.637 INFO:tasks.workunit.client.1.smithi135.stdout:8/709: dwrite d6/d23/d9d/dad/fb0 [0,4194304] 0 2024-03-30T04:28:32.637 INFO:tasks.workunit.client.1.smithi135.stdout:0/881: read d0/d13/d91/d106/f32 [795788,12277] 0 2024-03-30T04:28:32.638 INFO:tasks.workunit.client.1.smithi135.stdout:2/691: mkdir d4/d32/d12/d45/d6d/dea 0 2024-03-30T04:28:32.638 INFO:tasks.workunit.client.1.smithi135.stdout:6/821: creat d2/dd/d14/d2b/f108 x:0 0 0 2024-03-30T04:28:32.642 INFO:tasks.workunit.client.1.smithi135.stdout:7/932: dwrite d4/db/d1a/d3d/ddc/f2b [0,4194304] 0 2024-03-30T04:28:32.642 INFO:tasks.workunit.client.1.smithi135.stdout:8/710: chown d6/l87 174919 1 2024-03-30T04:28:32.646 INFO:tasks.workunit.client.1.smithi135.stdout:9/751: write d7/d29/d78/f9b [1638279,18286] 0 2024-03-30T04:28:32.646 INFO:tasks.workunit.client.1.smithi135.stdout:1/781: creat d16/d5a/d5f/d3d/df7/f101 x:0 0 0 2024-03-30T04:28:32.646 INFO:tasks.workunit.client.1.smithi135.stdout:0/882: chown d0/d5/d42/l11f 991126032 1 2024-03-30T04:28:32.647 INFO:tasks.workunit.client.1.smithi135.stdout:4/712: dwrite d10/d1a/d42/d46/f82 [0,4194304] 0 2024-03-30T04:28:32.656 INFO:tasks.workunit.client.1.smithi135.stdout:2/692: write d4/da/d8d/fab [284242,65136] 0 2024-03-30T04:28:32.656 INFO:tasks.workunit.client.1.smithi135.stdout:7/933: stat d4/db/d1a/d5c/d2c/d51/da8/fc2 0 2024-03-30T04:28:32.656 INFO:tasks.workunit.client.1.smithi135.stdout:6/822: symlink d2/d106/l109 0 2024-03-30T04:28:32.657 INFO:tasks.workunit.client.1.smithi135.stdout:0/883: symlink d0/d13/d91/dd2/df4/l121 0 2024-03-30T04:28:32.661 INFO:tasks.workunit.client.1.smithi135.stdout:4/713: write d10/d1a/d3d/f91 [4235136,59133] 0 2024-03-30T04:28:32.661 INFO:tasks.workunit.client.1.smithi135.stdout:2/693: write d4/d32/d2e/d2f/f64 [2507014,127787] 0 2024-03-30T04:28:32.661 INFO:tasks.workunit.client.1.smithi135.stdout:8/711: creat d6/dc7/fe7 x:0 0 0 2024-03-30T04:28:32.667 INFO:tasks.workunit.client.1.smithi135.stdout:1/782: unlink d16/d5a/d5f/d3a/dc6/fe6 0 2024-03-30T04:28:32.670 INFO:tasks.workunit.client.1.smithi135.stdout:6/823: symlink d2/dd/d40/db4/l10a 0 2024-03-30T04:28:32.671 INFO:tasks.workunit.client.1.smithi135.stdout:8/712: write d6/f25 [3506604,127798] 0 2024-03-30T04:28:32.671 INFO:tasks.workunit.client.1.smithi135.stdout:3/757: getdents d3/d50/d22/d74 0 2024-03-30T04:28:32.676 INFO:tasks.workunit.client.1.smithi135.stdout:5/707: dread d1/d7/d11/f8a [0,4194304] 0 2024-03-30T04:28:32.678 INFO:tasks.workunit.client.1.smithi135.stdout:7/934: write d4/db/d1a/d3d/fd4 [3312143,48622] 0 2024-03-30T04:28:32.679 INFO:tasks.workunit.client.1.smithi135.stdout:0/884: write d0/d13/d1e/d3a/dc2/ff3 [2718083,100155] 0 2024-03-30T04:28:32.679 INFO:tasks.workunit.client.1.smithi135.stdout:6/824: chown d2/d2e/l2f 17 1 2024-03-30T04:28:32.682 INFO:tasks.workunit.client.1.smithi135.stdout:8/713: mknod d6/d23/d9d/ddd/ce8 0 2024-03-30T04:28:32.688 INFO:tasks.workunit.client.1.smithi135.stdout:3/758: fsync d3/d10/d16/d23/f42 0 2024-03-30T04:28:32.692 INFO:tasks.workunit.client.1.smithi135.stdout:7/935: creat d4/db/d1a/d5c/d2c/d51/d11d/f140 x:0 0 0 2024-03-30T04:28:32.693 INFO:tasks.workunit.client.1.smithi135.stdout:0/885: chown d0/d5/d10/f31 14 1 2024-03-30T04:28:32.694 INFO:tasks.workunit.client.1.smithi135.stdout:6/825: creat d2/dd/d14/d30/d43/dcd/dbe/f10b x:0 0 0 2024-03-30T04:28:32.697 INFO:tasks.workunit.client.1.smithi135.stdout:3/759: chown d3/d54/d5c/cf1 88765877 1 2024-03-30T04:28:32.700 INFO:tasks.workunit.client.1.smithi135.stdout:0/886: fdatasync d0/d1/f43 0 2024-03-30T04:28:32.704 INFO:tasks.workunit.client.1.smithi135.stdout:3/760: chown d3/d50/l48 9385 1 2024-03-30T04:28:32.704 INFO:tasks.workunit.client.1.smithi135.stdout:9/752: dread d7/fe3 [4194304,4194304] 0 2024-03-30T04:28:32.704 INFO:tasks.workunit.client.1.smithi135.stdout:6/826: fsync d2/dd/d14/d2b/f108 0 2024-03-30T04:28:32.705 INFO:tasks.workunit.client.1.smithi135.stdout:2/694: write d4/da/f1e [3630873,1000] 0 2024-03-30T04:28:32.706 INFO:tasks.workunit.client.1.smithi135.stdout:0/887: chown d0/d13/d91/d106/f4b 1420 1 2024-03-30T04:28:32.708 INFO:tasks.workunit.client.1.smithi135.stdout:4/714: dwrite d10/d1a/d3d/dcb/fd6 [0,4194304] 0 2024-03-30T04:28:32.716 INFO:tasks.workunit.client.1.smithi135.stdout:3/761: creat d3/d10/d16/d41/d83/f103 x:0 0 0 2024-03-30T04:28:32.719 INFO:tasks.workunit.client.1.smithi135.stdout:7/936: getdents d4/db/d1a/d5c/d5d 0 2024-03-30T04:28:32.719 INFO:tasks.workunit.client.1.smithi135.stdout:2/695: dread - d4/d32/f75 zero size 2024-03-30T04:28:32.720 INFO:tasks.workunit.client.1.smithi135.stdout:9/753: symlink d7/d4f/d5c/d63/lf5 0 2024-03-30T04:28:32.722 INFO:tasks.workunit.client.1.smithi135.stdout:6/827: write d2/d68/fbf [553229,68466] 0 2024-03-30T04:28:32.724 INFO:tasks.workunit.client.1.smithi135.stdout:5/708: dwrite d1/d3/fa4 [0,4194304] 0 2024-03-30T04:28:32.727 INFO:tasks.workunit.client.1.smithi135.stdout:0/888: mkdir d0/d5/d9/d23/d2a/d60/d122 0 2024-03-30T04:28:32.727 INFO:tasks.workunit.client.1.smithi135.stdout:7/937: chown d4/db/d1a/d5c/d5d/c74 1949853571 1 2024-03-30T04:28:32.728 INFO:tasks.workunit.client.1.smithi135.stdout:4/715: rename d10/d1a/d3d/dcb to d10/d1a/d32/d40/d92/db9/dcf/df5 0 2024-03-30T04:28:32.729 INFO:tasks.workunit.client.1.smithi135.stdout:9/754: chown d7/d4f/d5c/d63/f64 38643 1 2024-03-30T04:28:32.730 INFO:tasks.workunit.client.1.smithi135.stdout:6/828: mknod d2/d68/c10c 0 2024-03-30T04:28:32.731 INFO:tasks.workunit.client.1.smithi135.stdout:2/696: chown d4/d32/d46/lb1 0 1 2024-03-30T04:28:32.733 INFO:tasks.workunit.client.1.smithi135.stdout:3/762: mkdir d3/d10/d16/d23/d6a/d94/dfa/d104 0 2024-03-30T04:28:32.734 INFO:tasks.workunit.client.1.smithi135.stdout:0/889: write d0/d13/d53/d7f/dc5/fe4 [868870,55404] 0 2024-03-30T04:28:32.734 INFO:tasks.workunit.client.1.smithi135.stdout:1/783: dwrite d16/d5a/ffe [0,4194304] 0 2024-03-30T04:28:32.740 INFO:tasks.workunit.client.1.smithi135.stdout:4/716: fdatasync d10/d72/fd0 0 2024-03-30T04:28:32.742 INFO:tasks.workunit.client.1.smithi135.stdout:7/938: dread - d4/db/d1a/d5c/d59/d113/f131 zero size 2024-03-30T04:28:32.743 INFO:tasks.workunit.client.1.smithi135.stdout:3/763: read d3/d50/d46/fb8 [2127202,74044] 0 2024-03-30T04:28:32.744 INFO:tasks.workunit.client.1.smithi135.stdout:8/714: dread d6/d8/d15/d1f/f44 [4194304,4194304] 0 2024-03-30T04:28:32.744 INFO:tasks.workunit.client.1.smithi135.stdout:7/939: dread - d4/db/d1a/d5c/d59/d76/f91 zero size 2024-03-30T04:28:32.749 INFO:tasks.workunit.client.1.smithi135.stdout:3/764: chown d3/fa6 93080 1 2024-03-30T04:28:32.749 INFO:tasks.workunit.client.1.smithi135.stdout:2/697: unlink d4/da/f1e 0 2024-03-30T04:28:32.751 INFO:tasks.workunit.client.1.smithi135.stdout:4/717: creat d10/d72/ff6 x:0 0 0 2024-03-30T04:28:32.754 INFO:tasks.workunit.client.1.smithi135.stdout:8/715: fsync d6/d8/d15/d1f/fbc 0 2024-03-30T04:28:32.756 INFO:tasks.workunit.client.1.smithi135.stdout:5/709: write d1/d45/f6b [5054094,103266] 0 2024-03-30T04:28:32.757 INFO:tasks.workunit.client.1.smithi135.stdout:9/755: write d7/d90/f94 [1511697,59119] 0 2024-03-30T04:28:32.758 INFO:tasks.workunit.client.1.smithi135.stdout:1/784: mknod d16/d5a/d5f/d3d/d8d/dc9/c102 0 2024-03-30T04:28:32.762 INFO:tasks.workunit.client.1.smithi135.stdout:9/756: dread - d7/d29/d78/da4/d37/f39 zero size 2024-03-30T04:28:32.763 INFO:tasks.workunit.client.1.smithi135.stdout:6/829: truncate d2/dd/d14/f15 349832 0 2024-03-30T04:28:32.773 INFO:tasks.workunit.client.1.smithi135.stdout:2/698: creat d4/da/d7a/feb x:0 0 0 2024-03-30T04:28:32.774 INFO:tasks.workunit.client.1.smithi135.stdout:3/765: rmdir d3/d10/d9c/dbe/dd6/d91 39 2024-03-30T04:28:32.776 INFO:tasks.workunit.client.1.smithi135.stdout:1/785: mkdir d16/d5a/dad/d103 0 2024-03-30T04:28:32.780 INFO:tasks.workunit.client.1.smithi135.stdout:0/890: write d0/d13/d1e/fb2 [1335278,79810] 0 2024-03-30T04:28:32.784 INFO:tasks.workunit.client.1.smithi135.stdout:2/699: symlink d4/d55/d77/lec 0 2024-03-30T04:28:32.785 INFO:tasks.workunit.client.1.smithi135.stdout:4/718: write d10/d1a/fba [3150408,3667] 0 2024-03-30T04:28:32.789 INFO:tasks.workunit.client.1.smithi135.stdout:5/710: write d1/da/d17/f3d [4245166,15604] 0 2024-03-30T04:28:32.790 INFO:tasks.workunit.client.1.smithi135.stdout:0/891: creat d0/d13/d91/d106/f123 x:0 0 0 2024-03-30T04:28:32.794 INFO:tasks.workunit.client.1.smithi135.stdout:6/830: getdents d2/dd/d14 0 2024-03-30T04:28:32.798 INFO:tasks.workunit.client.1.smithi135.stdout:7/940: dwrite d4/db/d79/d92/dcf/f114 [0,4194304] 0 2024-03-30T04:28:32.805 INFO:tasks.workunit.client.1.smithi135.stdout:0/892: truncate d0/d1/dd7/fdc 4658165 0 2024-03-30T04:28:32.807 INFO:tasks.workunit.client.1.smithi135.stdout:8/716: dwrite d6/d8/d15/fb4 [0,4194304] 0 2024-03-30T04:28:32.813 INFO:tasks.workunit.client.1.smithi135.stdout:6/831: creat d2/f10d x:0 0 0 2024-03-30T04:28:32.815 INFO:tasks.workunit.client.1.smithi135.stdout:7/941: write d4/db/d1a/d5c/d5d/f8d [2412411,40787] 0 2024-03-30T04:28:32.816 INFO:tasks.workunit.client.1.smithi135.stdout:0/893: dread d0/d13/d91/d106/dd5/f100 [0,4194304] 0 2024-03-30T04:28:32.818 INFO:tasks.workunit.client.1.smithi135.stdout:5/711: symlink d1/d3/d2b/d54/d5e/ld6 0 2024-03-30T04:28:32.823 INFO:tasks.workunit.client.1.smithi135.stdout:0/894: chown d0/d34/dfc/d3b/fb7 0 1 2024-03-30T04:28:32.825 INFO:tasks.workunit.client.1.smithi135.stdout:8/717: getdents d6/d8/d15 0 2024-03-30T04:28:32.828 INFO:tasks.workunit.client.1.smithi135.stdout:7/942: mknod d4/db/d1a/d5c/d2c/d51/da8/d13e/c141 0 2024-03-30T04:28:32.829 INFO:tasks.workunit.client.1.smithi135.stdout:1/786: dwrite d16/d29/d61/fdc [0,4194304] 0 2024-03-30T04:28:32.837 INFO:tasks.workunit.client.1.smithi135.stdout:4/719: truncate d10/d72/fd2 263705 0 2024-03-30T04:28:32.843 INFO:tasks.workunit.client.1.smithi135.stdout:6/832: dwrite d2/f9 [8388608,4194304] 0 2024-03-30T04:28:32.844 INFO:tasks.workunit.client.1.smithi135.stdout:3/766: dread d3/f11 [0,4194304] 0 2024-03-30T04:28:32.848 INFO:tasks.workunit.client.1.smithi135.stdout:1/787: symlink d16/d5a/d5f/d3d/d8d/l104 0 2024-03-30T04:28:32.850 INFO:tasks.workunit.client.1.smithi135.stdout:1/788: chown d16/d5a/d5f/d3d/db4 16 1 2024-03-30T04:28:32.853 INFO:tasks.workunit.client.1.smithi135.stdout:7/943: link d4/db/d1a/l93 d4/db/d1a/d5c/d59/d76/de4/l142 0 2024-03-30T04:28:32.854 INFO:tasks.workunit.client.1.smithi135.stdout:4/720: rename d10/d1a/d32/d40/c64 to d10/d1a/d32/d40/d92/db9/dcf/de8/cf7 0 2024-03-30T04:28:32.854 INFO:tasks.workunit.client.1.smithi135.stdout:3/767: symlink d3/d50/l105 0 2024-03-30T04:28:32.857 INFO:tasks.workunit.client.1.smithi135.stdout:8/718: dwrite d6/d8/f1c [0,4194304] 0 2024-03-30T04:28:32.857 INFO:tasks.workunit.client.1.smithi135.stdout:1/789: chown d16/d29/c59 28503 1 2024-03-30T04:28:32.857 INFO:tasks.workunit.client.1.smithi135.stdout:4/721: truncate d10/f59 938642 0 2024-03-30T04:28:32.859 INFO:tasks.workunit.client.1.smithi135.stdout:6/833: read d2/dd/d18/f22 [3030122,58051] 0 2024-03-30T04:28:32.865 INFO:tasks.workunit.client.1.smithi135.stdout:7/944: mknod d4/db/d79/d126/c143 0 2024-03-30T04:28:32.868 INFO:tasks.workunit.client.1.smithi135.stdout:4/722: read fd [1081423,31685] 0 2024-03-30T04:28:32.868 INFO:tasks.workunit.client.1.smithi135.stdout:3/768: truncate d3/d10/f31 1815186 0 2024-03-30T04:28:32.875 INFO:tasks.workunit.client.1.smithi135.stdout:8/719: rmdir d6/d8/d15/d54/d8e/d72/d8b/db2/dc6 0 2024-03-30T04:28:32.875 INFO:tasks.workunit.client.1.smithi135.stdout:6/834: dread d2/d68/d81/fb7 [0,4194304] 0 2024-03-30T04:28:32.876 INFO:tasks.workunit.client.1.smithi135.stdout:4/723: read d10/d1a/d42/f62 [2559867,6293] 0 2024-03-30T04:28:32.876 INFO:tasks.workunit.client.1.smithi135.stdout:7/945: mkdir d4/db/d1a/d3d/ddc/d144 0 2024-03-30T04:28:32.879 INFO:tasks.workunit.client.1.smithi135.stdout:6/835: unlink d2/l3b 0 2024-03-30T04:28:32.886 INFO:tasks.workunit.client.1.smithi135.stdout:8/720: fdatasync d6/d8/d15/d2a/da9/fba 0 2024-03-30T04:28:32.888 INFO:tasks.workunit.client.1.smithi135.stdout:1/790: dwrite d16/d9f/ff0 [0,4194304] 0 2024-03-30T04:28:32.897 INFO:tasks.workunit.client.1.smithi135.stdout:3/769: dwrite d3/d50/d22/f3d [4194304,4194304] 0 2024-03-30T04:28:32.912 INFO:tasks.workunit.client.1.smithi135.stdout:4/724: dwrite d10/d4d/d9c/fc5 [0,4194304] 0 2024-03-30T04:28:32.913 INFO:tasks.workunit.client.1.smithi135.stdout:4/725: stat d10/d4d/f87 0 2024-03-30T04:28:32.916 INFO:tasks.workunit.client.1.smithi135.stdout:4/726: fsync d10/d1a/d32/d40/d92/db9/dcf/df5/fd6 0 2024-03-30T04:28:32.917 INFO:tasks.workunit.client.1.smithi135.stdout:4/727: chown d10/d1a/f21 2277690 1 2024-03-30T04:28:32.918 INFO:tasks.workunit.client.1.smithi135.stdout:0/895: dread d0/d13/d91/d106/f1b [0,4194304] 0 2024-03-30T04:28:32.921 INFO:tasks.workunit.client.1.smithi135.stdout:0/896: read - d0/d19/fd3 zero size 2024-03-30T04:28:32.923 INFO:tasks.workunit.client.1.smithi135.stdout:0/897: chown d0/d13/d1e/d3a/f5a 6 1 2024-03-30T04:28:32.959 INFO:tasks.workunit.client.1.smithi135.stdout:8/721: chown d6/d8/l10 1 1 2024-03-30T04:28:32.960 INFO:tasks.workunit.client.1.smithi135.stdout:1/791: mkdir d16/d5a/d5f/d3d/d8d/dc9/da4/d99/d9a/d105 0 2024-03-30T04:28:32.973 INFO:tasks.workunit.client.1.smithi135.stdout:3/770: rename d3/d10/d16/d23/db6/dc1 to d3/d10/d106 0 2024-03-30T04:28:32.979 INFO:tasks.workunit.client.1.smithi135.stdout:6/836: symlink d2/l10e 0 2024-03-30T04:28:32.980 INFO:tasks.workunit.client.1.smithi135.stdout:8/722: rename d6/d8/d15/d54 to d6/d8/d15/d54/d8e/dbd/de9 22 2024-03-30T04:28:32.991 INFO:tasks.workunit.client.1.smithi135.stdout:1/792: dwrite d16/d5a/d5f/d3a/fb1 [0,4194304] 0 2024-03-30T04:28:33.001 INFO:tasks.workunit.client.1.smithi135.stdout:0/898: dread d0/d13/d1e/d3a/f61 [4194304,4194304] 0 2024-03-30T04:28:33.003 INFO:tasks.workunit.client.1.smithi135.stdout:6/837: creat d2/dd/d14/d2b/d86/d8e/f10f x:0 0 0 2024-03-30T04:28:33.007 INFO:tasks.workunit.client.1.smithi135.stdout:1/793: creat d16/d29/d32/dcc/f106 x:0 0 0 2024-03-30T04:28:33.008 INFO:tasks.workunit.client.1.smithi135.stdout:0/899: creat d0/d19/d2d/f124 x:0 0 0 2024-03-30T04:28:33.011 INFO:tasks.workunit.client.1.smithi135.stdout:6/838: creat d2/d68/f110 x:0 0 0 2024-03-30T04:28:33.015 INFO:tasks.workunit.client.1.smithi135.stdout:6/839: fsync d2/d36/d76/fd5 0 2024-03-30T04:28:33.015 INFO:tasks.workunit.client.1.smithi135.stdout:1/794: mknod d16/d29/d32/dcc/c107 0 2024-03-30T04:28:33.017 INFO:tasks.workunit.client.1.smithi135.stdout:6/840: chown d2/dd/d14/d2b/f108 1 1 2024-03-30T04:28:33.018 INFO:tasks.workunit.client.1.smithi135.stdout:1/795: stat d16/d5a/d5f/d3d/d8d/dc9/da4 0 2024-03-30T04:28:33.021 INFO:tasks.workunit.client.1.smithi135.stdout:6/841: write d2/dd/d40/db4/fee [996081,88504] 0 2024-03-30T04:28:33.022 INFO:tasks.workunit.client.1.smithi135.stdout:8/723: creat d6/d8/d15/d2a/fea x:0 0 0 2024-03-30T04:28:33.024 INFO:tasks.workunit.client.1.smithi135.stdout:0/900: read d0/d13/d91/d106/d7e/fe5 [41784,123703] 0 2024-03-30T04:28:33.025 INFO:tasks.workunit.client.1.smithi135.stdout:7/946: symlink d4/db/d1a/d3d/ddc/db0/l145 0 2024-03-30T04:28:33.028 INFO:tasks.workunit.client.1.smithi135.stdout:8/724: write d6/d8/d15/d1f/fbc [1488188,127165] 0 2024-03-30T04:28:33.032 INFO:tasks.workunit.client.1.smithi135.stdout:1/796: mknod d16/d5a/d5f/d3a/c108 0 2024-03-30T04:28:33.033 INFO:tasks.workunit.client.1.smithi135.stdout:6/842: truncate d2/dd/d14/f72 809441 0 2024-03-30T04:28:33.054 INFO:tasks.workunit.client.1.smithi135.stdout:1/797: link d16/d29/d2c/d3c/l70 d16/d5a/d5f/d3d/d8d/dc9/da4/d99/d9a/l109 0 2024-03-30T04:28:33.054 INFO:tasks.workunit.client.1.smithi135.stdout:8/725: getdents d6/dc7 0 2024-03-30T04:28:33.059 INFO:tasks.workunit.client.1.smithi135.stdout:1/798: mkdir d16/d29/d32/dcc/d10a 0 2024-03-30T04:28:33.060 INFO:tasks.workunit.client.1.smithi135.stdout:8/726: creat d6/d23/dca/feb x:0 0 0 2024-03-30T04:28:33.063 INFO:tasks.workunit.client.1.smithi135.stdout:1/799: rmdir d16/d5a/dad 39 2024-03-30T04:28:33.063 INFO:tasks.workunit.client.1.smithi135.stdout:8/727: truncate d6/d23/d9d/ddd/fd5 960193 0 2024-03-30T04:28:33.065 INFO:tasks.workunit.client.1.smithi135.stdout:1/800: write d16/d5a/f82 [492601,66774] 0 2024-03-30T04:28:33.069 INFO:tasks.workunit.client.1.smithi135.stdout:1/801: symlink d16/l10b 0 2024-03-30T04:28:33.072 INFO:tasks.workunit.client.1.smithi135.stdout:1/802: chown d16/d5a/d5f/d3d/d8d/dc9/da4/faf 31656962 1 2024-03-30T04:28:33.082 INFO:tasks.workunit.client.1.smithi135.stdout:8/728: dwrite d6/d8/d15/d2a/fea [0,4194304] 0 2024-03-30T04:28:33.082 INFO:tasks.workunit.client.1.smithi135.stdout:5/712: sync 2024-03-30T04:28:33.082 INFO:tasks.workunit.client.1.smithi135.stdout:2/700: sync 2024-03-30T04:28:33.082 INFO:tasks.workunit.client.1.smithi135.stdout:9/757: sync 2024-03-30T04:28:33.085 INFO:tasks.workunit.client.1.smithi135.stdout:3/771: write d3/d50/d46/f56 [1297759,79090] 0 2024-03-30T04:28:33.098 INFO:tasks.workunit.client.1.smithi135.stdout:8/729: symlink d6/de5/lec 0 2024-03-30T04:28:33.100 INFO:tasks.workunit.client.1.smithi135.stdout:2/701: dread d4/d6/f1a [0,4194304] 0 2024-03-30T04:28:33.101 INFO:tasks.workunit.client.1.smithi135.stdout:3/772: symlink d3/d6e/l107 0 2024-03-30T04:28:33.110 INFO:tasks.workunit.client.1.smithi135.stdout:7/947: write d4/db/d1a/d3d/ddc/db0/f134 [18177,22130] 0 2024-03-30T04:28:33.114 INFO:tasks.workunit.client.1.smithi135.stdout:0/901: dwrite d0/d13/d91/d106/f1b [0,4194304] 0 2024-03-30T04:28:33.118 INFO:tasks.workunit.client.1.smithi135.stdout:5/713: getdents d1/da 0 2024-03-30T04:28:33.118 INFO:tasks.workunit.client.1.smithi135.stdout:9/758: getdents d7/d29/d78/da4/d37/dba 0 2024-03-30T04:28:33.119 INFO:tasks.workunit.client.1.smithi135.stdout:6/843: link d2/c7 d2/dd/d14/d30/d43/dcd/dbe/c111 0 2024-03-30T04:28:33.121 INFO:tasks.workunit.client.1.smithi135.stdout:4/728: mknod d10/d4d/d89/cf8 0 2024-03-30T04:28:33.133 INFO:tasks.workunit.client.1.smithi135.stdout:7/948: getdents d4/db/d1a/d5c/d5d/db5 0 2024-03-30T04:28:33.134 INFO:tasks.workunit.client.1.smithi135.stdout:0/902: getdents d0/d13/d53/d7f/dc5 0 2024-03-30T04:28:33.137 INFO:tasks.workunit.client.1.smithi135.stdout:5/714: getdents d1/d45/d52 0 2024-03-30T04:28:33.138 INFO:tasks.workunit.client.1.smithi135.stdout:3/773: rmdir d3/d6e 39 2024-03-30T04:28:33.140 INFO:tasks.workunit.client.1.smithi135.stdout:6/844: dread d2/dd/d14/d30/f67 [0,4194304] 0 2024-03-30T04:28:33.140 INFO:tasks.workunit.client.1.smithi135.stdout:2/702: write d4/d6/f16 [4835940,6716] 0 2024-03-30T04:28:33.143 INFO:tasks.workunit.client.1.smithi135.stdout:4/729: write d10/d1a/d32/fb5 [193865,45] 0 2024-03-30T04:28:33.148 INFO:tasks.workunit.client.1.smithi135.stdout:5/715: chown d1/d45/d52/f9e 90 1 2024-03-30T04:28:33.148 INFO:tasks.workunit.client.1.smithi135.stdout:9/759: dwrite d7/d55/f5b [0,4194304] 0 2024-03-30T04:28:33.155 INFO:tasks.workunit.client.1.smithi135.stdout:0/903: creat d0/d34/dfc/f125 x:0 0 0 2024-03-30T04:28:33.156 INFO:tasks.workunit.client.1.smithi135.stdout:2/703: readlink d4/da/d8d/l90 0 2024-03-30T04:28:33.156 INFO:tasks.workunit.client.1.smithi135.stdout:1/803: dwrite d16/d5a/d5f/d3d/faa [0,4194304] 0 2024-03-30T04:28:33.159 INFO:tasks.workunit.client.1.smithi135.stdout:3/774: creat d3/d10/d16/d23/f108 x:0 0 0 2024-03-30T04:28:33.162 INFO:tasks.workunit.client.1.smithi135.stdout:7/949: dread d4/db/d1a/d3d/ddc/db0/db4/dcc/f112 [0,4194304] 0 2024-03-30T04:28:33.164 INFO:tasks.workunit.client.1.smithi135.stdout:9/760: read d7/d90/f9c [4023420,38306] 0 2024-03-30T04:28:33.167 INFO:tasks.workunit.client.1.smithi135.stdout:0/904: write d0/d13/d91/d106/d11e/d77/d86/fc9 [7886627,20292] 0 2024-03-30T04:28:33.173 INFO:tasks.workunit.client.1.smithi135.stdout:8/730: dwrite d6/d23/f82 [0,4194304] 0 2024-03-30T04:28:33.178 INFO:tasks.workunit.client.1.smithi135.stdout:5/716: rename d1/d7/d11/c81 to d1/da/d17/d70/db2/dad/cd7 0 2024-03-30T04:28:33.179 INFO:tasks.workunit.client.1.smithi135.stdout:3/775: rmdir d3/d50/dec 0 2024-03-30T04:28:33.181 INFO:tasks.workunit.client.1.smithi135.stdout:1/804: write d16/d5a/d5f/d3a/fa7 [3634468,110374] 0 2024-03-30T04:28:33.181 INFO:tasks.workunit.client.1.smithi135.stdout:0/905: write d0/d5/d10/fca [271151,92801] 0 2024-03-30T04:28:33.183 INFO:tasks.workunit.client.1.smithi135.stdout:7/950: creat d4/db/d1a/d5c/d59/f146 x:0 0 0 2024-03-30T04:28:33.184 INFO:tasks.workunit.client.1.smithi135.stdout:7/951: chown d4 161143569 1 2024-03-30T04:28:33.186 INFO:tasks.workunit.client.1.smithi135.stdout:5/717: stat d1/d3/d2b/d54/d5e/ld6 0 2024-03-30T04:28:33.187 INFO:tasks.workunit.client.1.smithi135.stdout:9/761: symlink d7/d4f/dee/lf6 0 2024-03-30T04:28:33.189 INFO:tasks.workunit.client.1.smithi135.stdout:4/730: dwrite d10/d1a/f21 [4194304,4194304] 0 2024-03-30T04:28:33.190 INFO:tasks.workunit.client.1.smithi135.stdout:6/845: dread d2/d68/d81/f88 [0,4194304] 0 2024-03-30T04:28:33.205 INFO:tasks.workunit.client.1.smithi135.stdout:9/762: mkdir d7/d4f/d5c/df7 0 2024-03-30T04:28:33.205 INFO:tasks.workunit.client.1.smithi135.stdout:0/906: mkdir d0/d19/d2d/d45/da8/d103/d126 0 2024-03-30T04:28:33.208 INFO:tasks.workunit.client.1.smithi135.stdout:3/776: write d3/f45 [252798,129094] 0 2024-03-30T04:28:33.208 INFO:tasks.workunit.client.1.smithi135.stdout:2/704: dwrite d4/d32/d12/f31 [0,4194304] 0 2024-03-30T04:28:33.208 INFO:tasks.workunit.client.1.smithi135.stdout:5/718: creat d1/da/d17/d70/db2/d79/d8e/fd8 x:0 0 0 2024-03-30T04:28:33.215 INFO:tasks.workunit.client.1.smithi135.stdout:2/705: write d4/d32/d2e/d9f/fcc [18094,30332] 0 2024-03-30T04:28:33.218 INFO:tasks.workunit.client.1.smithi135.stdout:6/846: rename d2/dd/d14/d30/d43/dcd/dad to d2/dd/d14/d30/d43/d112 0 2024-03-30T04:28:33.218 INFO:tasks.workunit.client.1.smithi135.stdout:4/731: mkdir d10/d1a/d2b/daf/dd4/dda/df9 0 2024-03-30T04:28:33.225 INFO:tasks.workunit.client.1.smithi135.stdout:7/952: symlink d4/db/d1a/d5c/d2c/d51/d56/l147 0 2024-03-30T04:28:33.227 INFO:tasks.workunit.client.1.smithi135.stdout:9/763: mkdir d7/df8 0 2024-03-30T04:28:33.227 INFO:tasks.workunit.client.1.smithi135.stdout:5/719: readlink d1/d3/d2b/d54/d6d/l8d 0 2024-03-30T04:28:33.230 INFO:tasks.workunit.client.1.smithi135.stdout:5/720: read d1/d45/f6b [5148562,127289] 0 2024-03-30T04:28:33.231 INFO:tasks.workunit.client.1.smithi135.stdout:8/731: dread d6/d8/d15/d1f/fbc [0,4194304] 0 2024-03-30T04:28:33.232 INFO:tasks.workunit.client.1.smithi135.stdout:4/732: creat d10/d4d/d89/d9f/daa/ded/ffa x:0 0 0 2024-03-30T04:28:33.237 INFO:tasks.workunit.client.1.smithi135.stdout:8/732: stat d6/d8/d14/fda 0 2024-03-30T04:28:33.242 INFO:tasks.workunit.client.1.smithi135.stdout:1/805: truncate d16/d5a/d5f/d3a/fb1 471056 0 2024-03-30T04:28:33.245 INFO:tasks.workunit.client.1.smithi135.stdout:2/706: truncate d4/d32/d2e/d2f/f64 1932023 0 2024-03-30T04:28:33.247 INFO:tasks.workunit.client.1.smithi135.stdout:3/777: dwrite d3/d54/d5c/da1/fa8 [4194304,4194304] 0 2024-03-30T04:28:33.250 INFO:tasks.workunit.client.1.smithi135.stdout:4/733: rename d10/d1a/d2b/daf to d10/d72/d7c/dfb 0 2024-03-30T04:28:33.253 INFO:tasks.workunit.client.1.smithi135.stdout:8/733: mkdir d6/d8/d15/d54/d8e/d72/d8b/db2/ded 0 2024-03-30T04:28:33.255 INFO:tasks.workunit.client.1.smithi135.stdout:3/778: write d3/db/f19 [24114,32616] 0 2024-03-30T04:28:33.258 INFO:tasks.workunit.client.1.smithi135.stdout:4/734: chown d10/d1a/d42/l5d 1621141 1 2024-03-30T04:28:33.259 INFO:tasks.workunit.client.1.smithi135.stdout:8/734: readlink d6/d8/d15/d54/d4f/l7c 0 2024-03-30T04:28:33.264 INFO:tasks.workunit.client.1.smithi135.stdout:3/779: dread - d3/d10/d16/d23/db6/fe2 zero size 2024-03-30T04:28:33.267 INFO:tasks.workunit.client.1.smithi135.stdout:8/735: chown d6/dc7/fe7 9 1 2024-03-30T04:28:33.269 INFO:tasks.workunit.client.1.smithi135.stdout:8/736: fdatasync d6/d23/d7b/fcd 0 2024-03-30T04:28:33.269 INFO:tasks.workunit.client.1.smithi135.stdout:5/721: rename d1/d45/d52/da5/dac/lb5 to d1/d7/d11/d3c/d4b/ld9 0 2024-03-30T04:28:33.270 INFO:tasks.workunit.client.1.smithi135.stdout:1/806: creat d16/d5a/d5f/d3d/d8d/dc9/d100/f10c x:0 0 0 2024-03-30T04:28:33.272 INFO:tasks.workunit.client.1.smithi135.stdout:8/737: stat d6/d8/d15/l1a 0 2024-03-30T04:28:33.273 INFO:tasks.workunit.client.1.smithi135.stdout:4/735: creat d10/d1a/d32/d40/d92/db9/dcf/ffc x:0 0 0 2024-03-30T04:28:33.273 INFO:tasks.workunit.client.1.smithi135.stdout:3/780: rmdir d3/d10/d9c/dbe/dd6 39 2024-03-30T04:28:33.274 INFO:tasks.workunit.client.1.smithi135.stdout:2/707: dwrite d4/d32/d46/f83 [4194304,4194304] 0 2024-03-30T04:28:33.277 INFO:tasks.workunit.client.1.smithi135.stdout:2/708: dread - d4/d6/d99/fc9 zero size 2024-03-30T04:28:33.277 INFO:tasks.workunit.client.1.smithi135.stdout:0/907: dread d0/d13/d1e/f4f [0,4194304] 0 2024-03-30T04:28:33.280 INFO:tasks.workunit.client.1.smithi135.stdout:5/722: truncate d1/d7/f59 93480 0 2024-03-30T04:28:33.281 INFO:tasks.workunit.client.1.smithi135.stdout:0/908: readlink d0/d5/d42/d87/la2 0 2024-03-30T04:28:33.295 INFO:tasks.workunit.client.1.smithi135.stdout:4/736: write d10/d1a/d42/d46/fc4 [183020,70142] 0 2024-03-30T04:28:33.298 INFO:tasks.workunit.client.1.smithi135.stdout:3/781: truncate d3/d54/d5c/da1/fa8 6434059 0 2024-03-30T04:28:33.299 INFO:tasks.workunit.client.1.smithi135.stdout:9/764: dread d7/f11 [0,4194304] 0 2024-03-30T04:28:33.300 INFO:tasks.workunit.client.1.smithi135.stdout:0/909: write d0/d5/d9/ff [350955,111029] 0 2024-03-30T04:28:33.301 INFO:tasks.workunit.client.1.smithi135.stdout:7/953: dread d4/db/d1a/d5c/d82/f106 [0,4194304] 0 2024-03-30T04:28:33.303 INFO:tasks.workunit.client.1.smithi135.stdout:9/765: chown d7/d41/d72 948402 1 2024-03-30T04:28:33.303 INFO:tasks.workunit.client.1.smithi135.stdout:2/709: dwrite d4/d32/d2e/d9f/fcc [0,4194304] 0 2024-03-30T04:28:33.307 INFO:tasks.workunit.client.1.smithi135.stdout:2/710: chown d4/d32/d2e/d67/db4 76 1 2024-03-30T04:28:33.309 INFO:tasks.workunit.client.1.smithi135.stdout:3/782: mkdir d3/d109 0 2024-03-30T04:28:33.310 INFO:tasks.workunit.client.1.smithi135.stdout:1/807: dread fb [0,4194304] 0 2024-03-30T04:28:33.313 INFO:tasks.workunit.client.1.smithi135.stdout:2/711: chown d4/d55/la5 34715189 1 2024-03-30T04:28:33.314 INFO:tasks.workunit.client.1.smithi135.stdout:9/766: dread d7/d29/d78/f8c [0,4194304] 0 2024-03-30T04:28:33.318 INFO:tasks.workunit.client.1.smithi135.stdout:7/954: creat d4/db/d1a/d5c/d59/d116/f148 x:0 0 0 2024-03-30T04:28:33.321 INFO:tasks.workunit.client.1.smithi135.stdout:1/808: mkdir d16/d5a/d5f/d3a/dc6/d10d 0 2024-03-30T04:28:33.322 INFO:tasks.workunit.client.1.smithi135.stdout:0/910: truncate d0/d13/d91/dd2/df4/fa9 491877 0 2024-03-30T04:28:33.325 INFO:tasks.workunit.client.1.smithi135.stdout:7/955: rmdir d4 39 2024-03-30T04:28:33.326 INFO:tasks.workunit.client.1.smithi135.stdout:2/712: mknod d4/d6/ced 0 2024-03-30T04:28:33.327 INFO:tasks.workunit.client.1.smithi135.stdout:9/767: chown d7/d4f/d5c/d63/dd2/db1/dbc/de6/fe9 1590077 1 2024-03-30T04:28:33.327 INFO:tasks.workunit.client.1.smithi135.stdout:5/723: dread d1/da/f18 [0,4194304] 0 2024-03-30T04:28:33.331 INFO:tasks.workunit.client.1.smithi135.stdout:0/911: dread - d0/d13/d91/d106/dd5/f10b zero size 2024-03-30T04:28:33.332 INFO:tasks.workunit.client.1.smithi135.stdout:4/737: dwrite d10/d1a/f27 [0,4194304] 0 2024-03-30T04:28:33.333 INFO:tasks.workunit.client.1.smithi135.stdout:5/724: chown d1/d45/c91 27672137 1 2024-03-30T04:28:33.336 INFO:tasks.workunit.client.1.smithi135.stdout:2/713: unlink d4/d32/d43/c6c 0 2024-03-30T04:28:33.339 INFO:tasks.workunit.client.1.smithi135.stdout:2/714: chown d4/d55/f5f 1046173 1 2024-03-30T04:28:33.340 INFO:tasks.workunit.client.1.smithi135.stdout:7/956: write d4/db/d1a/d5c/d2c/d2e/fe5 [1029983,4988] 0 2024-03-30T04:28:33.341 INFO:tasks.workunit.client.1.smithi135.stdout:2/715: chown d4/c7b 793666 1 2024-03-30T04:28:33.344 INFO:tasks.workunit.client.1.smithi135.stdout:1/809: dread d16/d5a/d5f/d3d/d8d/dc9/f84 [0,4194304] 0 2024-03-30T04:28:33.346 INFO:tasks.workunit.client.1.smithi135.stdout:4/738: mkdir d10/d1a/d32/d40/dfd 0 2024-03-30T04:28:33.347 INFO:tasks.workunit.client.1.smithi135.stdout:5/725: symlink d1/d45/d52/da5/lda 0 2024-03-30T04:28:33.351 INFO:tasks.workunit.client.1.smithi135.stdout:2/716: mkdir d4/d32/d12/d45/dac/dee 0 2024-03-30T04:28:33.353 INFO:tasks.workunit.client.1.smithi135.stdout:5/726: chown d1/da/d17/d70/f99 2 1 2024-03-30T04:28:33.357 INFO:tasks.workunit.client.1.smithi135.stdout:0/912: dwrite d0/d13/f74 [0,4194304] 0 2024-03-30T04:28:33.359 INFO:tasks.workunit.client.1.smithi135.stdout:5/727: stat d1/d7/d1a/d1f/d9b/fb8 0 2024-03-30T04:28:33.361 INFO:tasks.workunit.client.1.smithi135.stdout:1/810: creat d16/d5a/d5f/d3a/f10e x:0 0 0 2024-03-30T04:28:33.363 INFO:tasks.workunit.client.1.smithi135.stdout:5/728: mknod d1/da/cdb 0 2024-03-30T04:28:33.364 INFO:tasks.workunit.client.1.smithi135.stdout:3/783: dwrite d3/d54/d5c/da1/fa8 [4194304,4194304] 0 2024-03-30T04:28:33.367 INFO:tasks.workunit.client.1.smithi135.stdout:0/913: truncate d0/d34/dfc/d3b/ffa 2609194 0 2024-03-30T04:28:33.371 INFO:tasks.workunit.client.1.smithi135.stdout:9/768: truncate d7/d41/d72/f7d 4134326 0 2024-03-30T04:28:33.374 INFO:tasks.workunit.client.1.smithi135.stdout:7/957: dwrite d4/db/d1a/d5c/d2c/d51/d56/fa4 [0,4194304] 0 2024-03-30T04:28:33.375 INFO:tasks.workunit.client.1.smithi135.stdout:4/739: truncate d10/d1a/d42/f62 3501678 0 2024-03-30T04:28:33.376 INFO:tasks.workunit.client.1.smithi135.stdout:5/729: mknod d1/d3/d2b/cdc 0 2024-03-30T04:28:33.378 INFO:tasks.workunit.client.1.smithi135.stdout:1/811: creat d16/d5a/dad/d103/f10f x:0 0 0 2024-03-30T04:28:33.379 INFO:tasks.workunit.client.1.smithi135.stdout:5/730: read d1/d7/f2c [435150,9615] 0 2024-03-30T04:28:33.381 INFO:tasks.workunit.client.1.smithi135.stdout:3/784: creat d3/d10/d16/d41/f10a x:0 0 0 2024-03-30T04:28:33.384 INFO:tasks.workunit.client.1.smithi135.stdout:4/740: write d10/d1a/f27 [1624824,60948] 0 2024-03-30T04:28:33.384 INFO:tasks.workunit.client.1.smithi135.stdout:7/958: readlink d4/db/d1a/d5c/d2c/d51/da8/l8f 0 2024-03-30T04:28:33.384 INFO:tasks.workunit.client.1.smithi135.stdout:9/769: rmdir d7/d4f/d5c/df7 0 2024-03-30T04:28:33.387 INFO:tasks.workunit.client.1.smithi135.stdout:3/785: symlink d3/d10/d16/d23/db6/l10b 0 2024-03-30T04:28:33.389 INFO:tasks.workunit.client.1.smithi135.stdout:5/731: symlink d1/da/d4d/ldd 0 2024-03-30T04:28:33.390 INFO:tasks.workunit.client.1.smithi135.stdout:2/717: dwrite d4/d55/d77/f58 [0,4194304] 0 2024-03-30T04:28:33.395 INFO:tasks.workunit.client.1.smithi135.stdout:1/812: creat d16/d5a/d5f/d3a/dc6/d10d/f110 x:0 0 0 2024-03-30T04:28:33.396 INFO:tasks.workunit.client.1.smithi135.stdout:7/959: chown d4/db/d1a/d5c/d2c/d51/da8/c8b 3112978 1 2024-03-30T04:28:33.400 INFO:tasks.workunit.client.1.smithi135.stdout:2/718: write d4/d32/d2e/fd3 [435891,115939] 0 2024-03-30T04:28:33.405 INFO:tasks.workunit.client.1.smithi135.stdout:9/770: read d7/d29/d78/da4/d37/f3a [759570,124077] 0 2024-03-30T04:28:33.417 INFO:tasks.workunit.client.1.smithi135.stdout:2/719: write d4/da/f44 [2023513,111030] 0 2024-03-30T04:28:33.431 INFO:tasks.workunit.client.1.smithi135.stdout:3/786: write d3/d6e/f90 [994925,115398] 0 2024-03-30T04:28:33.431 INFO:tasks.workunit.client.1.smithi135.stdout:4/741: dwrite d10/d1a/d32/d40/d92/fb4 [0,4194304] 0 2024-03-30T04:28:33.435 INFO:tasks.workunit.client.1.smithi135.stdout:6/847: sync 2024-03-30T04:28:33.435 INFO:tasks.workunit.client.1.smithi135.stdout:8/738: sync 2024-03-30T04:28:33.439 INFO:tasks.workunit.client.1.smithi135.stdout:0/914: dread d0/d1/f29 [4194304,4194304] 0 2024-03-30T04:28:33.454 INFO:tasks.workunit.client.1.smithi135.stdout:5/732: link d1/d45/f83 d1/d7/d11/fde 0 2024-03-30T04:28:33.456 INFO:tasks.workunit.client.1.smithi135.stdout:7/960: mkdir d4/db/d1a/d5c/d2c/d51/d56/dbd/d149 0 2024-03-30T04:28:33.460 INFO:tasks.workunit.client.1.smithi135.stdout:4/742: mkdir d10/d1a/d42/d46/dfe 0 2024-03-30T04:28:33.470 INFO:tasks.workunit.client.1.smithi135.stdout:5/733: dwrite d1/d45/d52/f9e [0,4194304] 0 2024-03-30T04:28:33.473 INFO:tasks.workunit.client.1.smithi135.stdout:7/961: creat d4/db/d1a/d5c/d2c/d51/da8/d13e/f14a x:0 0 0 2024-03-30T04:28:33.476 INFO:tasks.workunit.client.1.smithi135.stdout:6/848: unlink d2/dd/d14/d30/d43/ce0 0 2024-03-30T04:28:33.479 INFO:tasks.workunit.client.1.smithi135.stdout:9/771: rmdir d7/d4f/dac 0 2024-03-30T04:28:33.481 INFO:tasks.workunit.client.1.smithi135.stdout:4/743: dread d10/d1a/d32/d40/f4b [0,4194304] 0 2024-03-30T04:28:33.481 INFO:tasks.workunit.client.1.smithi135.stdout:1/813: link d16/d5a/d5f/d3d/d8d/dc9/da4/d99/d9a/cdf d16/d5a/d5f/d3d/db4/c111 0 2024-03-30T04:28:33.489 INFO:tasks.workunit.client.1.smithi135.stdout:8/739: rename d6/d8/d14 to d6/d8/d15/d2a/dee 0 2024-03-30T04:28:33.490 INFO:tasks.workunit.client.1.smithi135.stdout:6/849: mknod d2/dd/d14/d30/d43/dcd/dc5/c113 0 2024-03-30T04:28:33.492 INFO:tasks.workunit.client.1.smithi135.stdout:9/772: mkdir d7/d41/db4/df9 0 2024-03-30T04:28:33.496 INFO:tasks.workunit.client.1.smithi135.stdout:4/744: creat d10/d4d/dc3/fff x:0 0 0 2024-03-30T04:28:33.497 INFO:tasks.workunit.client.1.smithi135.stdout:1/814: creat d16/d29/d2c/d3c/f112 x:0 0 0 2024-03-30T04:28:33.498 INFO:tasks.workunit.client.1.smithi135.stdout:4/745: chown d10/d72/d7c/l7f 300080990 1 2024-03-30T04:28:33.499 INFO:tasks.workunit.client.1.smithi135.stdout:7/962: creat d4/db/d1a/d3d/ddc/db0/de1/d130/f14b x:0 0 0 2024-03-30T04:28:33.500 INFO:tasks.workunit.client.1.smithi135.stdout:1/815: stat d16/d29/d2c/d43/fa2 0 2024-03-30T04:28:33.500 INFO:tasks.workunit.client.1.smithi135.stdout:6/850: creat d2/d106/f114 x:0 0 0 2024-03-30T04:28:33.502 INFO:tasks.workunit.client.1.smithi135.stdout:8/740: creat d6/d8/d15/d54/d7e/da6/fef x:0 0 0 2024-03-30T04:28:33.504 INFO:tasks.workunit.client.1.smithi135.stdout:4/746: creat d10/d72/d7c/dfb/f100 x:0 0 0 2024-03-30T04:28:33.506 INFO:tasks.workunit.client.1.smithi135.stdout:3/787: write d3/d10/d16/d23/fcb [14399,82995] 0 2024-03-30T04:28:33.508 INFO:tasks.workunit.client.1.smithi135.stdout:6/851: chown d2/dd/d14/d30/d43/dcd/dc5/c113 41775 1 2024-03-30T04:28:33.508 INFO:tasks.workunit.client.1.smithi135.stdout:4/747: chown d10/l8c 603 1 2024-03-30T04:28:33.511 INFO:tasks.workunit.client.1.smithi135.stdout:3/788: stat d3/d10/d16/d23/db6/fe2 0 2024-03-30T04:28:33.512 INFO:tasks.workunit.client.1.smithi135.stdout:1/816: dread d16/d5a/d5f/d3d/fcb [0,4194304] 0 2024-03-30T04:28:33.513 INFO:tasks.workunit.client.1.smithi135.stdout:2/720: truncate d4/d32/d46/dbc/dbe/fd5 1254276 0 2024-03-30T04:28:33.516 INFO:tasks.workunit.client.1.smithi135.stdout:6/852: chown d2/d68/c9c 31 1 2024-03-30T04:28:33.517 INFO:tasks.workunit.client.1.smithi135.stdout:4/748: readlink d10/d1a/d2b/l7b 0 2024-03-30T04:28:33.517 INFO:tasks.workunit.client.1.smithi135.stdout:8/741: rename d6/d8/d15/d54/d8e/d72/d8b/db2 to d6/d23/dca/df0 0 2024-03-30T04:28:33.518 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:33 smithi123 ceph-mon[35245]: pgmap v50: 65 pgs: 65 active+clean; 3.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 73 MiB/s rd, 177 MiB/s wr, 412 op/s 2024-03-30T04:28:33.520 INFO:tasks.workunit.client.1.smithi135.stdout:3/789: unlink d3/d50/d22/f72 0 2024-03-30T04:28:33.523 INFO:tasks.workunit.client.1.smithi135.stdout:5/734: truncate d1/d7/fb1 5858505 0 2024-03-30T04:28:33.524 INFO:tasks.workunit.client.1.smithi135.stdout:4/749: mkdir d10/d4d/d9c/d101 0 2024-03-30T04:28:33.525 INFO:tasks.workunit.client.1.smithi135.stdout:1/817: symlink d16/d5a/d5f/d3a/dc6/l113 0 2024-03-30T04:28:33.528 INFO:tasks.workunit.client.1.smithi135.stdout:0/915: dwrite d0/d5/d9/ff [4194304,4194304] 0 2024-03-30T04:28:33.528 INFO:tasks.workunit.client.1.smithi135.stdout:2/721: truncate d4/d32/d2e/d9f/fcc 4430276 0 2024-03-30T04:28:33.529 INFO:tasks.workunit.client.1.smithi135.stdout:8/742: mknod d6/d23/dca/df0/cf1 0 2024-03-30T04:28:33.534 INFO:tasks.workunit.client.1.smithi135.stdout:9/773: dread d7/d4f/d5c/d63/fc4 [0,4194304] 0 2024-03-30T04:28:33.536 INFO:tasks.workunit.client.1.smithi135.stdout:4/750: fsync d10/d1a/f86 0 2024-03-30T04:28:33.537 INFO:tasks.workunit.client.1.smithi135.stdout:6/853: truncate d2/dd/d18/f4b 841873 0 2024-03-30T04:28:33.538 INFO:tasks.workunit.client.1.smithi135.stdout:7/963: dread d4/db/f1c [0,4194304] 0 2024-03-30T04:28:33.540 INFO:tasks.workunit.client.1.smithi135.stdout:1/818: creat d16/d5a/d5f/d3d/df7/f114 x:0 0 0 2024-03-30T04:28:33.540 INFO:tasks.workunit.client.1.smithi135.stdout:0/916: rmdir d0/d13/d91/dd2 39 2024-03-30T04:28:33.547 INFO:tasks.workunit.client.1.smithi135.stdout:5/735: creat d1/d3/d2b/fdf x:0 0 0 2024-03-30T04:28:33.549 INFO:tasks.workunit.client.1.smithi135.stdout:3/790: dread d3/db/f8d [0,4194304] 0 2024-03-30T04:28:33.552 INFO:tasks.workunit.client.1.smithi135.stdout:2/722: truncate d4/d6/f1f 1622632 0 2024-03-30T04:28:33.557 INFO:tasks.workunit.client.1.smithi135.stdout:6/854: fsync d2/dd/d14/d2b/d86/d8e/f97 0 2024-03-30T04:28:33.557 INFO:tasks.workunit.client.1.smithi135.stdout:9/774: dread d7/fe3 [4194304,4194304] 0 2024-03-30T04:28:33.559 INFO:tasks.workunit.client.1.smithi135.stdout:5/736: symlink d1/d7/d11/le0 0 2024-03-30T04:28:33.564 INFO:tasks.workunit.client.1.smithi135.stdout:9/775: write d7/d4f/d5c/d63/dd2/db1/dbc/ff2 [199072,22072] 0 2024-03-30T04:28:33.567 INFO:tasks.workunit.client.1.smithi135.stdout:1/819: mknod d16/d5a/d5f/d3d/d8d/dc9/da4/c115 0 2024-03-30T04:28:33.571 INFO:tasks.workunit.client.1.smithi135.stdout:1/820: readlink d16/d29/d32/dcc/lc3 0 2024-03-30T04:28:33.571 INFO:tasks.workunit.client.1.smithi135.stdout:8/743: rename d6/d8/d15/d1f/f53 to d6/d8/d15/d54/d8e/ff2 0 2024-03-30T04:28:33.573 INFO:tasks.workunit.client.1.smithi135.stdout:5/737: dread d1/d7/f2c [0,4194304] 0 2024-03-30T04:28:33.574 INFO:tasks.workunit.client.1.smithi135.stdout:6/855: dread d2/dd/d14/d30/f60 [0,4194304] 0 2024-03-30T04:28:33.575 INFO:tasks.workunit.client.1.smithi135.stdout:3/791: symlink d3/d10/d16/d23/d6a/d94/dfa/d104/l10c 0 2024-03-30T04:28:33.582 INFO:tasks.workunit.client.1.smithi135.stdout:3/792: stat d3/c63 0 2024-03-30T04:28:33.583 INFO:tasks.workunit.client.1.smithi135.stdout:2/723: creat d4/fef x:0 0 0 2024-03-30T04:28:33.583 INFO:tasks.workunit.client.1.smithi135.stdout:1/821: rmdir d16/d5a/dad 39 2024-03-30T04:28:33.586 INFO:tasks.workunit.client.1.smithi135.stdout:8/744: symlink d6/d8/d15/d54/lf3 0 2024-03-30T04:28:33.589 INFO:tasks.workunit.client.1.smithi135.stdout:5/738: rename d1/d55 to d1/da/d17/d70/db2/d79/de1 0 2024-03-30T04:28:33.593 INFO:tasks.workunit.client.1.smithi135.stdout:0/917: dwrite d0/d13/d1e/fdd [0,4194304] 0 2024-03-30T04:28:33.595 INFO:tasks.workunit.client.1.smithi135.stdout:3/793: symlink d3/d10/d9c/l10d 0 2024-03-30T04:28:33.597 INFO:tasks.workunit.client.1.smithi135.stdout:8/745: symlink d6/dc7/lf4 0 2024-03-30T04:28:33.598 INFO:tasks.workunit.client.1.smithi135.stdout:7/964: dwrite d4/db/d1a/d5c/d2c/d51/d56/d81/f7f [0,4194304] 0 2024-03-30T04:28:33.600 INFO:tasks.workunit.client.1.smithi135.stdout:2/724: creat d4/d32/d46/dbc/dd0/ff0 x:0 0 0 2024-03-30T04:28:33.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:33 smithi135 ceph-mon[44364]: pgmap v50: 65 pgs: 65 active+clean; 3.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 73 MiB/s rd, 177 MiB/s wr, 412 op/s 2024-03-30T04:28:33.603 INFO:tasks.workunit.client.1.smithi135.stdout:6/856: rename d2/dd/c2a to d2/dd/d14/d38/dde/c115 0 2024-03-30T04:28:33.603 INFO:tasks.workunit.client.1.smithi135.stdout:4/751: dwrite d10/d72/fde [0,4194304] 0 2024-03-30T04:28:33.608 INFO:tasks.workunit.client.1.smithi135.stdout:7/965: chown d4/db 205 1 2024-03-30T04:28:33.608 INFO:tasks.workunit.client.1.smithi135.stdout:9/776: dread d7/d41/f5f [4194304,4194304] 0 2024-03-30T04:28:33.612 INFO:tasks.workunit.client.1.smithi135.stdout:5/739: read d1/d7/d11/d3c/fd4 [1694554,53990] 0 2024-03-30T04:28:33.621 INFO:tasks.workunit.client.1.smithi135.stdout:9/777: write d7/d55/f5b [382828,40390] 0 2024-03-30T04:28:33.621 INFO:tasks.workunit.client.1.smithi135.stdout:6/857: symlink d2/l116 0 2024-03-30T04:28:33.623 INFO:tasks.workunit.client.1.smithi135.stdout:2/725: mkdir d4/d32/d2e/d67/dad/de5/df1 0 2024-03-30T04:28:33.624 INFO:tasks.workunit.client.1.smithi135.stdout:4/752: mknod d10/d4d/dc1/c102 0 2024-03-30T04:28:33.627 INFO:tasks.workunit.client.1.smithi135.stdout:5/740: unlink d1/d7/d1a/c1e 0 2024-03-30T04:28:33.628 INFO:tasks.workunit.client.1.smithi135.stdout:8/746: creat d6/d8/d15/d2a/dee/ff5 x:0 0 0 2024-03-30T04:28:33.633 INFO:tasks.workunit.client.1.smithi135.stdout:9/778: creat d7/d29/ffa x:0 0 0 2024-03-30T04:28:33.635 INFO:tasks.workunit.client.1.smithi135.stdout:2/726: unlink d4/c7b 0 2024-03-30T04:28:33.642 INFO:tasks.workunit.client.1.smithi135.stdout:1/822: dwrite d16/d29/d2c/d43/d50/fe0 [0,4194304] 0 2024-03-30T04:28:33.643 INFO:tasks.workunit.client.1.smithi135.stdout:6/858: rename d2/ffa to d2/dd/d18/d87/f117 0 2024-03-30T04:28:33.643 INFO:tasks.workunit.client.1.smithi135.stdout:2/727: chown d4/d32/d2e/d67/dad 0 1 2024-03-30T04:28:33.646 INFO:tasks.workunit.client.1.smithi135.stdout:5/741: fsync d1/d7/d1a/fcd 0 2024-03-30T04:28:33.647 INFO:tasks.workunit.client.1.smithi135.stdout:3/794: dwrite d3/d50/f36 [4194304,4194304] 0 2024-03-30T04:28:33.655 INFO:tasks.workunit.client.1.smithi135.stdout:0/918: dread d0/d5/d9/d23/d2a/f51 [0,4194304] 0 2024-03-30T04:28:33.656 INFO:tasks.workunit.client.1.smithi135.stdout:9/779: truncate d7/d29/d78/da4/d37/dba/ff3 375397 0 2024-03-30T04:28:33.656 INFO:tasks.workunit.client.1.smithi135.stdout:3/795: unlink d3/d10/f5e 0 2024-03-30T04:28:33.658 INFO:tasks.workunit.client.1.smithi135.stdout:7/966: dread d4/db/d1a/d5c/d5d/f7b [4194304,4194304] 0 2024-03-30T04:28:33.660 INFO:tasks.workunit.client.1.smithi135.stdout:6/859: creat d2/dd/d14/d30/d43/d112/dae/f118 x:0 0 0 2024-03-30T04:28:33.662 INFO:tasks.workunit.client.1.smithi135.stdout:1/823: write d16/d29/d32/d9d/fd2 [451568,125193] 0 2024-03-30T04:28:33.671 INFO:tasks.workunit.client.1.smithi135.stdout:4/753: dread d10/d1a/f3a [0,4194304] 0 2024-03-30T04:28:33.673 INFO:tasks.workunit.client.1.smithi135.stdout:0/919: rename d0/d13/d91/d106/f4b to d0/d1/df9/f127 0 2024-03-30T04:28:33.676 INFO:tasks.workunit.client.1.smithi135.stdout:9/780: fsync d7/d29/d78/da4/fb6 0 2024-03-30T04:28:33.679 INFO:tasks.workunit.client.1.smithi135.stdout:6/860: fdatasync d2/dd/ff3 0 2024-03-30T04:28:33.680 INFO:tasks.workunit.client.1.smithi135.stdout:5/742: write d1/d45/f6b [4190016,103697] 0 2024-03-30T04:28:33.687 INFO:tasks.workunit.client.1.smithi135.stdout:7/967: truncate d4/db/d1a/d5c/d59/d113/f131 723006 0 2024-03-30T04:28:33.696 INFO:tasks.workunit.client.1.smithi135.stdout:9/781: unlink d7/d41/d72/f9f 0 2024-03-30T04:28:33.698 INFO:tasks.workunit.client.1.smithi135.stdout:1/824: dwrite d16/d5a/d5f/d3d/db4/fd0 [0,4194304] 0 2024-03-30T04:28:33.699 INFO:tasks.workunit.client.1.smithi135.stdout:9/782: stat d7/d29/d78/da4/f25 0 2024-03-30T04:28:33.700 INFO:tasks.workunit.client.1.smithi135.stdout:4/754: mkdir d10/d1a/d32/d40/d92/db9/dcf/de8/d103 0 2024-03-30T04:28:33.703 INFO:tasks.workunit.client.1.smithi135.stdout:9/783: readlink d7/d4f/d5c/led 0 2024-03-30T04:28:33.705 INFO:tasks.workunit.client.1.smithi135.stdout:6/861: mkdir d2/dd/d14/d30/d43/d119 0 2024-03-30T04:28:33.710 INFO:tasks.workunit.client.1.smithi135.stdout:6/862: fsync d2/dd/d14/f3a 0 2024-03-30T04:28:33.711 INFO:tasks.workunit.client.1.smithi135.stdout:2/728: dread d4/d55/f5f [0,4194304] 0 2024-03-30T04:28:33.712 INFO:tasks.workunit.client.1.smithi135.stdout:1/825: creat d16/d5a/d5f/d3d/db4/f116 x:0 0 0 2024-03-30T04:28:33.714 INFO:tasks.workunit.client.1.smithi135.stdout:8/747: dread d6/d8/d15/d2a/dee/d32/f62 [4194304,4194304] 0 2024-03-30T04:28:33.718 INFO:tasks.workunit.client.1.smithi135.stdout:9/784: chown d7/d29/d78/da4/l14 6172912 1 2024-03-30T04:28:33.718 INFO:tasks.workunit.client.1.smithi135.stdout:7/968: dread d4/db/f1c [0,4194304] 0 2024-03-30T04:28:33.719 INFO:tasks.workunit.client.1.smithi135.stdout:3/796: dread d3/d50/f36 [0,4194304] 0 2024-03-30T04:28:33.721 INFO:tasks.workunit.client.1.smithi135.stdout:4/755: dread d10/d1a/d42/f54 [0,4194304] 0 2024-03-30T04:28:33.727 INFO:tasks.workunit.client.1.smithi135.stdout:5/743: truncate d1/d7/f48 3622544 0 2024-03-30T04:28:33.727 INFO:tasks.workunit.client.1.smithi135.stdout:3/797: write d3/d10/d16/d23/d6a/fe3 [400942,6917] 0 2024-03-30T04:28:33.728 INFO:tasks.workunit.client.1.smithi135.stdout:9/785: rmdir d7/d41 39 2024-03-30T04:28:33.732 INFO:tasks.workunit.client.1.smithi135.stdout:0/920: dread d0/d13/fde [0,4194304] 0 2024-03-30T04:28:33.734 INFO:tasks.workunit.client.1.smithi135.stdout:7/969: mkdir d4/db/d1a/d5c/d59/d76/d14c 0 2024-03-30T04:28:33.737 INFO:tasks.workunit.client.1.smithi135.stdout:8/748: fsync d6/d8/d15/d2a/dee/d32/f62 0 2024-03-30T04:28:33.738 INFO:tasks.workunit.client.1.smithi135.stdout:2/729: mkdir d4/d32/dc2/dde/df2 0 2024-03-30T04:28:33.741 INFO:tasks.workunit.client.1.smithi135.stdout:3/798: unlink d3/c79 0 2024-03-30T04:28:33.741 INFO:tasks.workunit.client.1.smithi135.stdout:4/756: chown d10/d1a/d32/d40/ddf/cf2 31 1 2024-03-30T04:28:33.749 INFO:tasks.workunit.client.1.smithi135.stdout:6/863: dwrite d2/dd/d14/d2b/d86/fc3 [0,4194304] 0 2024-03-30T04:28:33.751 INFO:tasks.workunit.client.1.smithi135.stdout:5/744: write d1/da/f18 [3857380,89059] 0 2024-03-30T04:28:33.754 INFO:tasks.workunit.client.1.smithi135.stdout:0/921: rename d0/d5/d42/d87/lcd to d0/d5/d9/d23/d2a/dc4/l128 0 2024-03-30T04:28:33.762 INFO:tasks.workunit.client.1.smithi135.stdout:9/786: dread d7/d4f/d5c/d63/f64 [0,4194304] 0 2024-03-30T04:28:33.763 INFO:tasks.workunit.client.1.smithi135.stdout:0/922: rmdir d0/d5 39 2024-03-30T04:28:33.768 INFO:tasks.workunit.client.1.smithi135.stdout:1/826: dwrite d16/d5a/dad/d77/fe9 [0,4194304] 0 2024-03-30T04:28:33.778 INFO:tasks.workunit.client.1.smithi135.stdout:4/757: getdents d10/d4d/d89 0 2024-03-30T04:28:33.779 INFO:tasks.workunit.client.1.smithi135.stdout:7/970: dwrite d4/db/d1a/d5c/d2c/d51/f7e [0,4194304] 0 2024-03-30T04:28:33.779 INFO:tasks.workunit.client.1.smithi135.stdout:1/827: dread - d16/d5a/dad/d77/fb6 zero size 2024-03-30T04:28:33.781 INFO:tasks.workunit.client.1.smithi135.stdout:3/799: getdents d3/d10/d16/d23/db6 0 2024-03-30T04:28:33.782 INFO:tasks.workunit.client.1.smithi135.stdout:9/787: mkdir d7/df8/dfb 0 2024-03-30T04:28:33.787 INFO:tasks.workunit.client.1.smithi135.stdout:7/971: fdatasync d4/db/fa0 0 2024-03-30T04:28:33.790 INFO:tasks.workunit.client.1.smithi135.stdout:8/749: dwrite d6/d8/f17 [0,4194304] 0 2024-03-30T04:28:33.792 INFO:tasks.workunit.client.1.smithi135.stdout:5/745: rename d1/f4f to d1/da/d17/d70/db2/d79/de1/fe2 0 2024-03-30T04:28:33.799 INFO:tasks.workunit.client.1.smithi135.stdout:2/730: dread d4/d32/d43/fa9 [0,4194304] 0 2024-03-30T04:28:33.803 INFO:tasks.workunit.client.1.smithi135.stdout:7/972: unlink d4/db/d1a/d5c/d82/c9b 0 2024-03-30T04:28:33.809 INFO:tasks.workunit.client.1.smithi135.stdout:3/800: rename d3/d6e/fd2 to d3/d50/d22/f10e 0 2024-03-30T04:28:33.817 INFO:tasks.workunit.client.1.smithi135.stdout:5/746: unlink d1/cb 0 2024-03-30T04:28:33.828 INFO:tasks.workunit.client.1.smithi135.stdout:1/828: rename d16/d29/d32/dcc/f106 to d16/d29/d2c/d3c/f117 0 2024-03-30T04:28:33.828 INFO:tasks.workunit.client.1.smithi135.stdout:9/788: write d7/d4f/d89/f95 [870217,10777] 0 2024-03-30T04:28:33.829 INFO:tasks.workunit.client.1.smithi135.stdout:0/923: dwrite d0/d13/fa3 [0,4194304] 0 2024-03-30T04:28:33.833 INFO:tasks.workunit.client.1.smithi135.stdout:3/801: rmdir d3/d10/d16 39 2024-03-30T04:28:33.836 INFO:tasks.workunit.client.1.smithi135.stdout:3/802: readlink d3/d10/la2 0 2024-03-30T04:28:33.836 INFO:tasks.workunit.client.1.smithi135.stdout:1/829: chown d16/d29/d32/dcc/f95 643500 1 2024-03-30T04:28:33.840 INFO:tasks.workunit.client.1.smithi135.stdout:2/731: write d4/d32/d2e/f40 [5160332,89601] 0 2024-03-30T04:28:33.844 INFO:tasks.workunit.client.1.smithi135.stdout:5/747: creat d1/da/d17/fe3 x:0 0 0 2024-03-30T04:28:33.847 INFO:tasks.workunit.client.1.smithi135.stdout:1/830: readlink d16/d5a/d5f/d3d/la0 0 2024-03-30T04:28:33.853 INFO:tasks.workunit.client.1.smithi135.stdout:6/864: dread d2/d2e/f4c [0,4194304] 0 2024-03-30T04:28:33.854 INFO:tasks.workunit.client.1.smithi135.stdout:8/750: dwrite d6/d8/d15/d54/d7e/daf/dc1/fe1 [0,4194304] 0 2024-03-30T04:28:33.862 INFO:tasks.workunit.client.1.smithi135.stdout:9/789: dwrite d7/d29/d78/fd6 [0,4194304] 0 2024-03-30T04:28:33.865 INFO:tasks.workunit.client.1.smithi135.stdout:1/831: creat d16/d5a/dad/d77/f118 x:0 0 0 2024-03-30T04:28:33.871 INFO:tasks.workunit.client.1.smithi135.stdout:4/758: dread d10/d72/d7c/f83 [0,4194304] 0 2024-03-30T04:28:33.872 INFO:tasks.workunit.client.1.smithi135.stdout:9/790: write d7/d4f/d89/fc2 [4268920,49646] 0 2024-03-30T04:28:33.872 INFO:tasks.workunit.client.1.smithi135.stdout:5/748: creat d1/d7/d11/d3c/d4b/fe4 x:0 0 0 2024-03-30T04:28:33.874 INFO:tasks.workunit.client.1.smithi135.stdout:8/751: mknod d6/d8/d15/d2a/da9/cf6 0 2024-03-30T04:28:33.875 INFO:tasks.workunit.client.1.smithi135.stdout:6/865: truncate d2/d36/fd4 1091983 0 2024-03-30T04:28:33.879 INFO:tasks.workunit.client.1.smithi135.stdout:7/973: getdents d4/db/d1a/d90/dc7 0 2024-03-30T04:28:33.890 INFO:tasks.workunit.client.1.smithi135.stdout:1/832: unlink d16/d29/c59 0 2024-03-30T04:28:33.890 INFO:tasks.workunit.client.1.smithi135.stdout:7/974: read d4/db/d1a/d3d/ddc/db0/dfd/fa1 [669465,35541] 0 2024-03-30T04:28:33.898 INFO:tasks.workunit.client.1.smithi135.stdout:4/759: dwrite d10/d1a/d3d/f91 [0,4194304] 0 2024-03-30T04:28:33.900 INFO:tasks.workunit.client.1.smithi135.stdout:4/760: stat d10/d1a/ceb 0 2024-03-30T04:28:33.905 INFO:tasks.workunit.client.1.smithi135.stdout:6/866: creat d2/d2e/f11a x:0 0 0 2024-03-30T04:28:33.909 INFO:tasks.workunit.client.1.smithi135.stdout:7/975: dread d4/db/d1a/d3d/ddc/f2b [4194304,4194304] 0 2024-03-30T04:28:33.909 INFO:tasks.workunit.client.1.smithi135.stdout:1/833: creat d16/d29/d32/f119 x:0 0 0 2024-03-30T04:28:33.910 INFO:tasks.workunit.client.1.smithi135.stdout:5/749: chown d1/d3/d80 15 1 2024-03-30T04:28:33.913 INFO:tasks.workunit.client.1.smithi135.stdout:5/750: fsync d1/da/d17/f3d 0 2024-03-30T04:28:33.918 INFO:tasks.workunit.client.1.smithi135.stdout:0/924: link d0/d13/d1e/d3a/dc2/ff3 d0/d5/d9/d23/f129 0 2024-03-30T04:28:33.918 INFO:tasks.workunit.client.1.smithi135.stdout:8/752: dread d6/d23/f5c [0,4194304] 0 2024-03-30T04:28:33.921 INFO:tasks.workunit.client.1.smithi135.stdout:3/803: getdents d3/d50/d22 0 2024-03-30T04:28:33.922 INFO:tasks.workunit.client.1.smithi135.stdout:2/732: creat d4/d55/d77/ff3 x:0 0 0 2024-03-30T04:28:33.926 INFO:tasks.workunit.client.1.smithi135.stdout:6/867: mknod d2/dd/d14/d53/c11b 0 2024-03-30T04:28:33.929 INFO:tasks.workunit.client.1.smithi135.stdout:5/751: symlink d1/d7/d1a/d1f/d9b/le5 0 2024-03-30T04:28:33.932 INFO:tasks.workunit.client.1.smithi135.stdout:4/761: creat d10/d1a/d42/d46/dfe/f104 x:0 0 0 2024-03-30T04:28:33.932 INFO:tasks.workunit.client.1.smithi135.stdout:3/804: symlink d3/d54/d5c/da1/l10f 0 2024-03-30T04:28:33.933 INFO:tasks.workunit.client.1.smithi135.stdout:8/753: creat d6/d8/d15/d54/d8e/dbd/ff7 x:0 0 0 2024-03-30T04:28:33.934 INFO:tasks.workunit.client.1.smithi135.stdout:5/752: truncate d1/da/f18 4769897 0 2024-03-30T04:28:33.936 INFO:tasks.workunit.client.1.smithi135.stdout:6/868: rmdir d2/dd/d14/d2b/d86 39 2024-03-30T04:28:33.937 INFO:tasks.workunit.client.1.smithi135.stdout:4/762: write d10/d4d/d89/f8f [657844,116973] 0 2024-03-30T04:28:33.942 INFO:tasks.workunit.client.1.smithi135.stdout:3/805: chown d3/d10/d16/ca4 7 1 2024-03-30T04:28:33.946 INFO:tasks.workunit.client.1.smithi135.stdout:2/733: dwrite d4/d32/d2e/d2f/fbd [0,4194304] 0 2024-03-30T04:28:33.951 INFO:tasks.workunit.client.1.smithi135.stdout:5/753: rename d1/d7/f13 to d1/d45/d52/da5/fe6 0 2024-03-30T04:28:33.952 INFO:tasks.workunit.client.1.smithi135.stdout:7/976: link d4/db/d1a/l13f d4/db/d1a/d5c/d2c/d51/da8/d13e/l14d 0 2024-03-30T04:28:33.954 INFO:tasks.workunit.client.1.smithi135.stdout:7/977: stat d4/db/c132 0 2024-03-30T04:28:33.957 INFO:tasks.workunit.client.1.smithi135.stdout:0/925: link d0/d5/d9/d23/d2a/d60/le7 d0/d1/dd7/l12a 0 2024-03-30T04:28:33.959 INFO:tasks.workunit.client.1.smithi135.stdout:6/869: dread d2/d36/fdf [0,4194304] 0 2024-03-30T04:28:33.964 INFO:tasks.workunit.client.1.smithi135.stdout:2/734: symlink d4/d32/d12/d45/d6d/lf4 0 2024-03-30T04:28:33.965 INFO:tasks.workunit.client.1.smithi135.stdout:4/763: creat d10/d72/d7c/dfb/dd4/dda/df9/f105 x:0 0 0 2024-03-30T04:28:33.968 INFO:tasks.workunit.client.1.smithi135.stdout:7/978: mkdir d4/db/d1a/d5c/d2c/d51/d56/dbd/d149/d14e 0 2024-03-30T04:28:33.969 INFO:tasks.workunit.client.1.smithi135.stdout:8/754: creat d6/d8/d15/d54/d8e/d72/ff8 x:0 0 0 2024-03-30T04:28:33.970 INFO:tasks.workunit.client.1.smithi135.stdout:6/870: write d2/dd/d14/d2b/fc8 [3190324,78094] 0 2024-03-30T04:28:33.972 INFO:tasks.workunit.client.1.smithi135.stdout:2/735: symlink d4/d32/daf/lf5 0 2024-03-30T04:28:33.972 INFO:tasks.workunit.client.1.smithi135.stdout:4/764: write d10/f59 [1661973,111607] 0 2024-03-30T04:28:33.983 INFO:tasks.workunit.client.1.smithi135.stdout:5/754: dwrite d1/d7/ff [8388608,4194304] 0 2024-03-30T04:28:33.983 INFO:tasks.workunit.client.1.smithi135.stdout:7/979: write d4/db/d1a/d5c/d59/fb6 [402736,14174] 0 2024-03-30T04:28:33.998 INFO:tasks.workunit.client.1.smithi135.stdout:2/736: dwrite d4/da/d8d/fe2 [0,4194304] 0 2024-03-30T04:28:34.362 INFO:tasks.workunit.client.1.smithi135.stdout:4/765: truncate d10/d1a/d42/d66/f7e 424872 0 2024-03-30T04:28:34.374 INFO:tasks.workunit.client.1.smithi135.stdout:6/871: mknod d2/d68/da8/d105/c11c 0 2024-03-30T04:28:34.375 INFO:tasks.workunit.client.1.smithi135.stdout:1/834: truncate d16/d5a/dad/ff4 2914991 0 2024-03-30T04:28:34.375 INFO:tasks.workunit.client.1.smithi135.stdout:7/980: dwrite d4/db/d1a/d5c/d2c/d51/fd0 [8388608,4194304] 0 2024-03-30T04:28:34.375 INFO:tasks.workunit.client.1.smithi135.stdout:5/755: dwrite d1/da/d17/d70/db2/dad/fce [0,4194304] 0 2024-03-30T04:28:34.375 INFO:tasks.workunit.client.1.smithi135.stdout:0/926: rename d0/d1 to d0/d13/d53/d54/d12b 0 2024-03-30T04:28:34.375 INFO:tasks.workunit.client.1.smithi135.stdout:6/872: unlink d2/dd/c16 0 2024-03-30T04:28:34.375 INFO:tasks.workunit.client.1.smithi135.stdout:6/873: stat d2/dd/d14/d38/le1 0 2024-03-30T04:28:34.375 INFO:tasks.workunit.client.1.smithi135.stdout:1/835: fdatasync d16/d5a/f92 0 2024-03-30T04:28:34.375 INFO:tasks.workunit.client.1.smithi135.stdout:8/755: getdents d6/d8/d15/d54 0 2024-03-30T04:28:34.375 INFO:tasks.workunit.client.1.smithi135.stdout:2/737: truncate d4/d6/f1f 220372 0 2024-03-30T04:28:34.375 INFO:tasks.workunit.client.1.smithi135.stdout:4/766: dread d10/d1a/d32/f70 [0,4194304] 0 2024-03-30T04:28:34.375 INFO:tasks.workunit.client.1.smithi135.stdout:1/836: read f7 [412604,27502] 0 2024-03-30T04:28:34.375 INFO:tasks.workunit.client.1.smithi135.stdout:7/981: symlink d4/db/d1a/d5c/d2c/d51/l14f 0 2024-03-30T04:28:34.375 INFO:tasks.workunit.client.1.smithi135.stdout:3/806: dwrite d3/d50/f25 [0,4194304] 0 2024-03-30T04:28:34.375 INFO:tasks.workunit.client.1.smithi135.stdout:0/927: link d0/d5/d9/d23/d5d/la7 d0/d5/l12c 0 2024-03-30T04:28:34.376 INFO:tasks.workunit.client.1.smithi135.stdout:2/738: read d4/d32/d12/d45/f8e [2447132,120097] 0 2024-03-30T04:28:34.376 INFO:tasks.workunit.client.1.smithi135.stdout:5/756: creat d1/d3/d76/d92/da0/fe7 x:0 0 0 2024-03-30T04:28:34.376 INFO:tasks.workunit.client.1.smithi135.stdout:1/837: creat d16/d5a/d5f/d3a/dbc/f11a x:0 0 0 2024-03-30T04:28:34.376 INFO:tasks.workunit.client.1.smithi135.stdout:3/807: fsync d3/d10/d16/fed 0 2024-03-30T04:28:34.376 INFO:tasks.workunit.client.1.smithi135.stdout:1/838: chown f7 2 1 2024-03-30T04:28:34.376 INFO:tasks.workunit.client.1.smithi135.stdout:5/757: symlink d1/d45/d52/da5/le8 0 2024-03-30T04:28:34.376 INFO:tasks.workunit.client.1.smithi135.stdout:7/982: getdents d4/db/d1a/d5c 0 2024-03-30T04:28:34.376 INFO:tasks.workunit.client.1.smithi135.stdout:4/767: dwrite d10/d1a/d32/d40/d92/db9/dcf/ffc [0,4194304] 0 2024-03-30T04:28:34.376 INFO:tasks.workunit.client.1.smithi135.stdout:1/839: write d16/d9f/ff5 [1546397,57600] 0 2024-03-30T04:28:34.376 INFO:tasks.workunit.client.1.smithi135.stdout:3/808: symlink d3/d10/d16/d23/d6a/d86/d98/l110 0 2024-03-30T04:28:34.376 INFO:tasks.workunit.client.1.smithi135.stdout:6/874: dread d2/f12 [0,4194304] 0 2024-03-30T04:28:34.376 INFO:tasks.workunit.client.1.smithi135.stdout:8/756: dwrite d6/d8/d15/d54/d8e/d72/d73/f9b [0,4194304] 0 2024-03-30T04:28:34.376 INFO:tasks.workunit.client.1.smithi135.stdout:0/928: dwrite d0/d13/d53/d7f/d83/f10a [0,4194304] 0 2024-03-30T04:28:34.376 INFO:tasks.workunit.client.1.smithi135.stdout:7/983: symlink d4/db/d1a/d3d/ddc/db0/l150 0 2024-03-30T04:28:34.376 INFO:tasks.workunit.client.1.smithi135.stdout:1/840: chown d16/d29/d2c/d43/d50/fe0 1306322 1 2024-03-30T04:28:34.377 INFO:tasks.workunit.client.1.smithi135.stdout:3/809: symlink d3/d10/d9c/dbe/l111 0 2024-03-30T04:28:34.377 INFO:tasks.workunit.client.1.smithi135.stdout:8/757: write d6/d8/d15/d54/d8e/fe0 [534389,67102] 0 2024-03-30T04:28:34.377 INFO:tasks.workunit.client.1.smithi135.stdout:2/739: dread d4/d32/d2e/f8c [0,4194304] 0 2024-03-30T04:28:34.377 INFO:tasks.workunit.client.1.smithi135.stdout:0/929: mkdir d0/d19/d2d/d45/da8/d103/d12d 0 2024-03-30T04:28:34.377 INFO:tasks.workunit.client.1.smithi135.stdout:3/810: dread d3/d54/ff6 [0,4194304] 0 2024-03-30T04:28:34.377 INFO:tasks.workunit.client.1.smithi135.stdout:1/841: creat d16/d5a/f11b x:0 0 0 2024-03-30T04:28:34.377 INFO:tasks.workunit.client.1.smithi135.stdout:8/758: rmdir d6/d23/d9d 39 2024-03-30T04:28:34.377 INFO:tasks.workunit.client.1.smithi135.stdout:4/768: dwrite d10/d1a/fba [0,4194304] 0 2024-03-30T04:28:34.377 INFO:tasks.workunit.client.1.smithi135.stdout:3/811: mkdir d3/d10/d9c/dbe/d112 0 2024-03-30T04:28:34.377 INFO:tasks.workunit.client.1.smithi135.stdout:4/769: mkdir d10/d1a/d42/d66/dc7/d106 0 2024-03-30T04:28:34.377 INFO:tasks.workunit.client.1.smithi135.stdout:3/812: creat d3/d10/d16/d41/d83/f113 x:0 0 0 2024-03-30T04:28:34.377 INFO:tasks.workunit.client.1.smithi135.stdout:6/875: dwrite d2/dd/d14/d2b/d86/fe5 [0,4194304] 0 2024-03-30T04:28:34.377 INFO:tasks.workunit.client.1.smithi135.stdout:8/759: dread d6/d8/d15/d54/f5d [0,4194304] 0 2024-03-30T04:28:34.377 INFO:tasks.workunit.client.1.smithi135.stdout:8/760: readlink d6/d8/d15/d1f/l60 0 2024-03-30T04:28:34.377 INFO:tasks.workunit.client.1.smithi135.stdout:3/813: mknod d3/d10/d16/d23/d6a/df9/c114 0 2024-03-30T04:28:34.378 INFO:tasks.workunit.client.1.smithi135.stdout:6/876: mkdir d2/dd/d14/d38/d11d 0 2024-03-30T04:28:34.378 INFO:tasks.workunit.client.1.smithi135.stdout:0/930: dwrite d0/d5/d9/ff [0,4194304] 0 2024-03-30T04:28:34.378 INFO:tasks.workunit.client.1.smithi135.stdout:3/814: mknod d3/db/c115 0 2024-03-30T04:28:34.378 INFO:tasks.workunit.client.1.smithi135.stdout:6/877: chown d2/dd/d14/d2b/d86/d55/f7b 471 1 2024-03-30T04:28:34.378 INFO:tasks.workunit.client.1.smithi135.stdout:1/842: dwrite d16/d5a/d5f/d3d/df7/f114 [0,4194304] 0 2024-03-30T04:28:34.378 INFO:tasks.workunit.client.1.smithi135.stdout:8/761: read - d6/dc7/fe7 zero size 2024-03-30T04:28:34.378 INFO:tasks.workunit.client.1.smithi135.stdout:1/843: rmdir d16/d5a/dad 39 2024-03-30T04:28:34.378 INFO:tasks.workunit.client.1.smithi135.stdout:3/815: truncate d3/d10/d16/d23/d6a/d86/fd1 504118 0 2024-03-30T04:28:34.378 INFO:tasks.workunit.client.1.smithi135.stdout:8/762: rmdir d6/d8/d15/d54/d4f 39 2024-03-30T04:28:34.378 INFO:tasks.workunit.client.1.smithi135.stdout:6/878: fdatasync d2/dd/d14/d53/fb3 0 2024-03-30T04:28:34.378 INFO:tasks.workunit.client.1.smithi135.stdout:6/879: chown d2/dd 40109 1 2024-03-30T04:28:34.378 INFO:tasks.workunit.client.1.smithi135.stdout:0/931: read d0/d5/d10/fec [2814891,5776] 0 2024-03-30T04:28:34.379 INFO:tasks.workunit.client.1.smithi135.stdout:4/770: dwrite d10/d1a/d32/d40/d92/db9/dcf/de8/ff4 [0,4194304] 0 2024-03-30T04:28:34.379 INFO:tasks.workunit.client.1.smithi135.stdout:2/740: dread d4/d32/d2e/d67/db4/fc4 [0,4194304] 0 2024-03-30T04:28:34.379 INFO:tasks.workunit.client.1.smithi135.stdout:3/816: write d3/d54/ff6 [109648,22432] 0 2024-03-30T04:28:34.379 INFO:tasks.workunit.client.1.smithi135.stdout:8/763: mkdir d6/d23/df9 0 2024-03-30T04:28:34.379 INFO:tasks.workunit.client.1.smithi135.stdout:0/932: write d0/d13/d1e/f112 [725629,50595] 0 2024-03-30T04:28:34.379 INFO:tasks.workunit.client.1.smithi135.stdout:3/817: readlink d3/d10/d16/d41/d83/l84 0 2024-03-30T04:28:34.379 INFO:tasks.workunit.client.1.smithi135.stdout:8/764: chown d6/d8/f16 735322262 1 2024-03-30T04:28:34.379 INFO:tasks.workunit.client.1.smithi135.stdout:6/880: mkdir d2/dd/d14/d2b/d11e 0 2024-03-30T04:28:34.379 INFO:tasks.workunit.client.1.smithi135.stdout:4/771: truncate d10/d72/d7c/f9b 239380 0 2024-03-30T04:28:34.379 INFO:tasks.workunit.client.1.smithi135.stdout:8/765: unlink d6/d8/d15/d2a/fa4 0 2024-03-30T04:28:34.379 INFO:tasks.workunit.client.1.smithi135.stdout:3/818: dread d3/d50/d46/f58 [0,4194304] 0 2024-03-30T04:28:34.379 INFO:tasks.workunit.client.1.smithi135.stdout:6/881: rename d2/dd/d14/d53/l5c to d2/dd/d40/d52/l11f 0 2024-03-30T04:28:34.379 INFO:tasks.workunit.client.1.smithi135.stdout:2/741: link d4/d32/d2e/f4b d4/d32/d2e/d67/dad/de5/df1/ff6 0 2024-03-30T04:28:34.379 INFO:tasks.workunit.client.1.smithi135.stdout:2/742: write d4/da/fb [1471932,73716] 0 2024-03-30T04:28:34.379 INFO:tasks.workunit.client.1.smithi135.stdout:4/772: write d10/d4d/dc1/fd3 [712594,19631] 0 2024-03-30T04:28:34.380 INFO:tasks.workunit.client.1.smithi135.stdout:3/819: fdatasync d3/d50/d46/f59 0 2024-03-30T04:28:34.380 INFO:tasks.workunit.client.1.smithi135.stdout:1/844: dread d16/d5a/d5f/f4e [0,4194304] 0 2024-03-30T04:28:34.380 INFO:tasks.workunit.client.1.smithi135.stdout:0/933: rename d0/d5/d9/d23/d2a/d60/de8/d11c to d0/d13/d91/d106/d7e/db3/d12e 0 2024-03-30T04:28:34.380 INFO:tasks.workunit.client.1.smithi135.stdout:6/882: mknod d2/c120 0 2024-03-30T04:28:34.380 INFO:tasks.workunit.client.1.smithi135.stdout:0/934: chown d0/d13/d53/d7f/d83/fea 3999 1 2024-03-30T04:28:34.380 INFO:tasks.workunit.client.1.smithi135.stdout:3/820: creat d3/db/da9/dcc/f116 x:0 0 0 2024-03-30T04:28:34.380 INFO:tasks.workunit.client.1.smithi135.stdout:3/821: readlink d3/d10/d9c/dbe/l111 0 2024-03-30T04:28:34.380 INFO:tasks.workunit.client.1.smithi135.stdout:1/845: creat d16/d5a/d5f/d3d/db4/f11c x:0 0 0 2024-03-30T04:28:34.380 INFO:tasks.workunit.client.1.smithi135.stdout:8/766: write d6/d8/d15/d54/f5d [680956,37206] 0 2024-03-30T04:28:34.380 INFO:tasks.workunit.client.1.smithi135.stdout:0/935: mknod d0/d13/d91/d106/d7e/db3/d12e/c12f 0 2024-03-30T04:28:34.380 INFO:tasks.workunit.client.1.smithi135.stdout:4/773: truncate d10/d1a/d3d/f95 1783559 0 2024-03-30T04:28:34.380 INFO:tasks.workunit.client.1.smithi135.stdout:1/846: dread d16/d29/d2c/d3c/fc0 [0,4194304] 0 2024-03-30T04:28:34.380 INFO:tasks.workunit.client.1.smithi135.stdout:2/743: dread d4/d6/f61 [0,4194304] 0 2024-03-30T04:28:34.380 INFO:tasks.workunit.client.1.smithi135.stdout:4/774: read d10/d72/d7c/f83 [2875743,97386] 0 2024-03-30T04:28:34.381 INFO:tasks.workunit.client.1.smithi135.stdout:3/822: truncate d3/d10/d16/d23/f4b 556965 0 2024-03-30T04:28:34.381 INFO:tasks.workunit.client.1.smithi135.stdout:6/883: creat d2/f121 x:0 0 0 2024-03-30T04:28:34.381 INFO:tasks.workunit.client.1.smithi135.stdout:8/767: truncate d6/d8/d15/d2a/dee/f4e 1687505 0 2024-03-30T04:28:34.381 INFO:tasks.workunit.client.1.smithi135.stdout:0/936: mknod d0/d19/d2d/d45/c130 0 2024-03-30T04:28:34.381 INFO:tasks.workunit.client.1.smithi135.stdout:1/847: creat d16/d29/d2c/d43/d50/f11d x:0 0 0 2024-03-30T04:28:34.381 INFO:tasks.workunit.client.1.smithi135.stdout:3/823: unlink d3/fe8 0 2024-03-30T04:28:34.382 INFO:tasks.workunit.client.1.smithi135.stdout:2/744: creat d4/d32/d46/dbc/dbe/ff7 x:0 0 0 2024-03-30T04:28:34.382 INFO:tasks.workunit.client.1.smithi135.stdout:1/848: write d16/d5a/d5f/d3d/faa [1034198,116693] 0 2024-03-30T04:28:34.382 INFO:tasks.workunit.client.1.smithi135.stdout:4/775: fdatasync d10/d1a/d32/d40/f4b 0 2024-03-30T04:28:34.382 INFO:tasks.workunit.client.1.smithi135.stdout:6/884: chown d2/c4f 15970 1 2024-03-30T04:28:34.382 INFO:tasks.workunit.client.1.smithi135.stdout:1/849: fsync d16/d5a/d5f/d3d/df7/f114 0 2024-03-30T04:28:34.383 INFO:tasks.workunit.client.1.smithi135.stdout:3/824: unlink d3/c4a 0 2024-03-30T04:28:34.383 INFO:tasks.workunit.client.1.smithi135.stdout:8/768: creat d6/d8/d15/d54/d7e/da6/ffa x:0 0 0 2024-03-30T04:28:34.383 INFO:tasks.workunit.client.1.smithi135.stdout:6/885: mknod d2/dd/d40/c122 0 2024-03-30T04:28:34.383 INFO:tasks.workunit.client.1.smithi135.stdout:4/776: truncate d10/d1a/d42/f54 695729 0 2024-03-30T04:28:34.383 INFO:tasks.workunit.client.1.smithi135.stdout:6/886: dread d2/dd/d18/d87/fda [0,4194304] 0 2024-03-30T04:28:34.383 INFO:tasks.workunit.client.1.smithi135.stdout:8/769: unlink d6/d8/d15/l43 0 2024-03-30T04:28:34.383 INFO:tasks.workunit.client.1.smithi135.stdout:6/887: chown d2/dd/d14/l104 205 1 2024-03-30T04:28:34.383 INFO:tasks.workunit.client.1.smithi135.stdout:6/888: stat d2/dd/d14/d53/cb6 0 2024-03-30T04:28:34.384 INFO:tasks.workunit.client.1.smithi135.stdout:4/777: mkdir d10/d4d/d9c/d107 0 2024-03-30T04:28:34.384 INFO:tasks.workunit.client.1.smithi135.stdout:2/745: mknod d4/d55/d77/cf8 0 2024-03-30T04:28:34.384 INFO:tasks.workunit.client.1.smithi135.stdout:4/778: write d10/fe9 [804898,33686] 0 2024-03-30T04:28:34.384 INFO:tasks.workunit.client.1.smithi135.stdout:0/937: link d0/d13/d91/d106/d11e/d67/c104 d0/d19/d2d/d45/da8/d103/d12d/c131 0 2024-03-30T04:28:34.384 INFO:tasks.workunit.client.1.smithi135.stdout:0/938: chown d0/d13/d91/d106/d11e/f95 2013616 1 2024-03-30T04:28:34.384 INFO:tasks.workunit.client.1.smithi135.stdout:1/850: creat d16/d5a/d5f/d3d/d8d/dc9/d100/f11e x:0 0 0 2024-03-30T04:28:34.384 INFO:tasks.workunit.client.1.smithi135.stdout:8/770: mkdir d6/d8/d15/d54/d7e/daf/dfb 0 2024-03-30T04:28:34.384 INFO:tasks.workunit.client.1.smithi135.stdout:0/939: write d0/d13/d1e/fdd [1016129,99568] 0 2024-03-30T04:28:34.385 INFO:tasks.workunit.client.1.smithi135.stdout:2/746: creat d4/d6/d99/ff9 x:0 0 0 2024-03-30T04:28:34.385 INFO:tasks.workunit.client.1.smithi135.stdout:3/825: link d3/d10/d16/d41/d83/l101 d3/d109/l117 0 2024-03-30T04:28:34.385 INFO:tasks.workunit.client.1.smithi135.stdout:1/851: readlink d16/d29/d2c/lc8 0 2024-03-30T04:28:34.385 INFO:tasks.workunit.client.1.smithi135.stdout:8/771: symlink d6/d23/dca/lfc 0 2024-03-30T04:28:34.385 INFO:tasks.workunit.client.1.smithi135.stdout:0/940: stat d0/d13/d91/d106/d11e/d67 0 2024-03-30T04:28:34.385 INFO:tasks.workunit.client.1.smithi135.stdout:4/779: dread d10/f59 [0,4194304] 0 2024-03-30T04:28:34.386 INFO:tasks.workunit.client.1.smithi135.stdout:6/889: dwrite d2/d2e/f37 [0,4194304] 0 2024-03-30T04:28:34.386 INFO:tasks.workunit.client.1.smithi135.stdout:2/747: creat d4/d32/daf/ffa x:0 0 0 2024-03-30T04:28:34.386 INFO:tasks.workunit.client.1.smithi135.stdout:4/780: fsync d10/d4d/f87 0 2024-03-30T04:28:34.386 INFO:tasks.workunit.client.1.smithi135.stdout:0/941: truncate d0/d5/d9/fb 251176 0 2024-03-30T04:28:34.386 INFO:tasks.workunit.client.1.smithi135.stdout:4/781: write d10/d1a/d32/d40/d92/db9/dcf/de8/ff4 [3309602,4801] 0 2024-03-30T04:28:34.386 INFO:tasks.workunit.client.1.smithi135.stdout:6/890: write d2/dd/d14/d30/d43/dcd/dbe/f10b [955160,47688] 0 2024-03-30T04:28:34.386 INFO:tasks.workunit.client.1.smithi135.stdout:8/772: mkdir d6/d8/d15/d54/d7e/daf/dfb/dfd 0 2024-03-30T04:28:34.387 INFO:tasks.workunit.client.1.smithi135.stdout:2/748: rename d4/d32/d12/lc6 to d4/d32/d12/d45/d7d/lfb 0 2024-03-30T04:28:34.387 INFO:tasks.workunit.client.1.smithi135.stdout:0/942: symlink d0/d5/d42/d9f/df5/l132 0 2024-03-30T04:28:34.387 INFO:tasks.workunit.client.1.smithi135.stdout:3/826: write d3/d50/f12 [634560,20280] 0 2024-03-30T04:28:34.387 INFO:tasks.workunit.client.1.smithi135.stdout:0/943: chown d0/cb6 1 1 2024-03-30T04:28:34.387 INFO:tasks.workunit.client.1.smithi135.stdout:0/944: chown d0/d5/d9/d23/d2a 14 1 2024-03-30T04:28:34.387 INFO:tasks.workunit.client.1.smithi135.stdout:3/827: rename d3/d50/fc8 to d3/d50/d22/d33/d62/f118 0 2024-03-30T04:28:34.387 INFO:tasks.workunit.client.1.smithi135.stdout:4/782: write d10/d4d/d89/d9f/fa7 [1118950,3119] 0 2024-03-30T04:28:34.387 INFO:tasks.workunit.client.1.smithi135.stdout:8/773: dread d6/d8/d15/d54/d8e/fe0 [0,4194304] 0 2024-03-30T04:28:34.388 INFO:tasks.workunit.client.1.smithi135.stdout:8/774: symlink d6/d8/d15/d2a/dee/d32/lfe 0 2024-03-30T04:28:34.388 INFO:tasks.workunit.client.1.smithi135.stdout:4/783: write d10/d1a/d42/f54 [553622,52628] 0 2024-03-30T04:28:34.389 INFO:tasks.workunit.client.1.smithi135.stdout:6/891: dwrite d2/dd/d14/d2b/f85 [0,4194304] 0 2024-03-30T04:28:34.389 INFO:tasks.workunit.client.1.smithi135.stdout:4/784: stat d10/c24 0 2024-03-30T04:28:34.389 INFO:tasks.workunit.client.1.smithi135.stdout:0/945: dwrite d0/d19/d2d/f118 [0,4194304] 0 2024-03-30T04:28:34.393 INFO:tasks.workunit.client.1.smithi135.stdout:6/892: dread - d2/dd/ff3 zero size 2024-03-30T04:28:34.394 INFO:tasks.workunit.client.1.smithi135.stdout:3/828: dwrite d3/f9 [0,4194304] 0 2024-03-30T04:28:34.401 INFO:tasks.workunit.client.1.smithi135.stdout:4/785: symlink d10/d1a/d2b/l108 0 2024-03-30T04:28:34.402 INFO:tasks.workunit.client.1.smithi135.stdout:0/946: mknod d0/d5/d9/d23/d2a/d60/d122/c133 0 2024-03-30T04:28:34.407 INFO:tasks.workunit.client.1.smithi135.stdout:0/947: truncate d0/d19/d2d/f118 4946377 0 2024-03-30T04:28:34.407 INFO:tasks.workunit.client.1.smithi135.stdout:4/786: symlink d10/d72/d7c/l109 0 2024-03-30T04:28:34.411 INFO:tasks.workunit.client.1.smithi135.stdout:4/787: stat d10/d1a/d3d/f91 0 2024-03-30T04:28:34.412 INFO:tasks.workunit.client.1.smithi135.stdout:3/829: rename d3/d50/d22/d33/d62/f66 to d3/f119 0 2024-03-30T04:28:34.425 INFO:tasks.workunit.client.1.smithi135.stdout:4/788: rmdir d10/d72/d7c/dfb 39 2024-03-30T04:28:34.426 INFO:tasks.workunit.client.1.smithi135.stdout:6/893: truncate d2/dd/d14/d2b/d86/fc3 1690677 0 2024-03-30T04:28:34.435 INFO:tasks.workunit.client.1.smithi135.stdout:8/775: dwrite d6/d8/d15/d2a/dee/f5e [0,4194304] 0 2024-03-30T04:28:34.439 INFO:tasks.workunit.client.1.smithi135.stdout:0/948: dwrite d0/d13/d1e/fdd [0,4194304] 0 2024-03-30T04:28:34.444 INFO:tasks.workunit.client.1.smithi135.stdout:0/949: write d0/d5/d10/f11 [11646681,95666] 0 2024-03-30T04:28:34.446 INFO:tasks.workunit.client.1.smithi135.stdout:0/950: stat d0/db5/db8/fe3 0 2024-03-30T04:28:34.449 INFO:tasks.workunit.client.1.smithi135.stdout:0/951: chown d0/d13/l27 198 1 2024-03-30T04:28:34.453 INFO:tasks.workunit.client.1.smithi135.stdout:0/952: chown d0/d13/cd8 3102998 1 2024-03-30T04:28:34.466 INFO:tasks.workunit.client.1.smithi135.stdout:3/830: mkdir d3/db/d11a 0 2024-03-30T04:28:34.468 INFO:tasks.workunit.client.1.smithi135.stdout:6/894: chown d2/dd/d14/d2b/d86/d8e/caa 789473 1 2024-03-30T04:28:34.473 INFO:tasks.workunit.client.1.smithi135.stdout:8/776: rename d6/d8/c66 to d6/d8/d15/d54/d4f/cff 0 2024-03-30T04:28:34.476 INFO:tasks.workunit.client.1.smithi135.stdout:3/831: chown d3/d10/d9c/dbe/dd6/d91/cc3 268 1 2024-03-30T04:28:34.481 INFO:tasks.workunit.client.1.smithi135.stdout:6/895: getdents d2 0 2024-03-30T04:28:34.484 INFO:tasks.workunit.client.1.smithi135.stdout:4/789: dread d10/d1a/d32/d40/d92/fb4 [0,4194304] 0 2024-03-30T04:28:34.485 INFO:tasks.workunit.client.1.smithi135.stdout:6/896: chown d2/dd/d14/d2b/d86/d8e/f97 20387 1 2024-03-30T04:28:34.488 INFO:tasks.workunit.client.1.smithi135.stdout:6/897: truncate d2/d68/d81/ffb 756075 0 2024-03-30T04:28:34.488 INFO:tasks.workunit.client.1.smithi135.stdout:4/790: stat d10/d1a/d32/d40/d92/lac 0 2024-03-30T04:28:34.488 INFO:tasks.workunit.client.1.smithi135.stdout:0/953: write d0/d13/d91/dd2/df4/fc8 [131515,96034] 0 2024-03-30T04:28:34.497 INFO:tasks.workunit.client.1.smithi135.stdout:6/898: read d2/dd/d14/d2b/d86/d55/f74 [386226,37929] 0 2024-03-30T04:28:34.499 INFO:tasks.workunit.client.1.smithi135.stdout:0/954: getdents d0/d5/d9/d23/d2a/d60 0 2024-03-30T04:28:34.500 INFO:tasks.workunit.client.1.smithi135.stdout:4/791: write d10/d1a/d32/d40/d92/db9/dcf/df5/fcd [875609,79812] 0 2024-03-30T04:28:34.503 INFO:tasks.workunit.client.1.smithi135.stdout:4/792: fdatasync d10/d1a/f21 0 2024-03-30T04:28:34.503 INFO:tasks.workunit.client.1.smithi135.stdout:6/899: mkdir d2/d68/da8/d123 0 2024-03-30T04:28:34.503 INFO:tasks.workunit.client.1.smithi135.stdout:0/955: readlink d0/d5/d9/l3d 0 2024-03-30T04:28:34.505 INFO:tasks.workunit.client.1.smithi135.stdout:0/956: readlink d0/d13/d53/d7f/d83/d9c/l6c 0 2024-03-30T04:28:34.506 INFO:tasks.workunit.client.1.smithi135.stdout:8/777: dread d6/d8/d15/d2a/dee/f63 [4194304,4194304] 0 2024-03-30T04:28:34.506 INFO:tasks.workunit.client.1.smithi135.stdout:4/793: fsync d10/f74 0 2024-03-30T04:28:34.508 INFO:tasks.workunit.client.1.smithi135.stdout:6/900: mknod d2/c124 0 2024-03-30T04:28:34.512 INFO:tasks.workunit.client.1.smithi135.stdout:4/794: readlink d10/d1a/d32/l3b 0 2024-03-30T04:28:34.513 INFO:tasks.workunit.client.1.smithi135.stdout:0/957: rename d0/d13/d91/d106/dd5/ffe to d0/d13/d53/d7f/d83/f134 0 2024-03-30T04:28:34.524 INFO:tasks.workunit.client.1.smithi135.stdout:8/778: link d6/d8/d15/d54/d8e/d72/cb8 d6/d8/c100 0 2024-03-30T04:28:34.525 INFO:tasks.workunit.client.1.smithi135.stdout:0/958: getdents d0/d13/d91/d106/d11e/d77/d8b 0 2024-03-30T04:28:34.526 INFO:tasks.workunit.client.1.smithi135.stdout:6/901: write d2/dd/d14/d2b/f85 [3325525,15954] 0 2024-03-30T04:28:34.532 INFO:tasks.workunit.client.1.smithi135.stdout:6/902: stat d2/d36/d76/l8c 0 2024-03-30T04:28:34.535 INFO:tasks.workunit.client.1.smithi135.stdout:0/959: truncate d0/d19/d2d/d45/da8/f10f 157143 0 2024-03-30T04:28:34.536 INFO:tasks.workunit.client.1.smithi135.stdout:8/779: rename d6/d8/d15/d2a/c7f to d6/d8/d15/d54/d4f/c101 0 2024-03-30T04:28:34.539 INFO:tasks.workunit.client.1.smithi135.stdout:4/795: dwrite d10/f14 [0,4194304] 0 2024-03-30T04:28:34.539 INFO:tasks.workunit.client.1.smithi135.stdout:6/903: fsync d2/dd/d14/d30/d43/f78 0 2024-03-30T04:28:34.557 INFO:tasks.workunit.client.1.smithi135.stdout:8/780: getdents d6/d23 0 2024-03-30T04:28:34.560 INFO:tasks.workunit.client.1.smithi135.stdout:6/904: write d2/dd/d14/d30/f67 [1228263,127061] 0 2024-03-30T04:28:34.564 INFO:tasks.workunit.client.1.smithi135.stdout:8/781: stat d6/d8/d15/d54/d8e/dbd/cde 0 2024-03-30T04:28:34.567 INFO:tasks.workunit.client.1.smithi135.stdout:0/960: dwrite d0/d19/fd3 [0,4194304] 0 2024-03-30T04:28:34.568 INFO:tasks.workunit.client.1.smithi135.stdout:6/905: stat d2/dd/c19 0 2024-03-30T04:28:34.574 INFO:tasks.workunit.client.1.smithi135.stdout:0/961: chown d0/d19/d2d/d45/da8/d103/d12d 6292120 1 2024-03-30T04:28:34.580 INFO:tasks.workunit.client.1.smithi135.stdout:4/796: dwrite d10/d1a/d32/fa5 [0,4194304] 0 2024-03-30T04:28:34.581 INFO:tasks.workunit.client.1.smithi135.stdout:0/962: stat d0/d5/d9/cbe 0 2024-03-30T04:28:34.581 INFO:tasks.workunit.client.1.smithi135.stdout:8/782: creat d6/d8/d15/d54/d7e/daf/dfb/f102 x:0 0 0 2024-03-30T04:28:34.581 INFO:tasks.workunit.client.1.smithi135.stdout:6/906: truncate d2/f99 3443215 0 2024-03-30T04:28:34.584 INFO:tasks.workunit.client.1.smithi135.stdout:0/963: rmdir d0/d19/d2d/d79 39 2024-03-30T04:28:34.586 INFO:tasks.workunit.client.1.smithi135.stdout:6/907: chown d2/dd/d14/d30/d43/dcd/lb8 984361 1 2024-03-30T04:28:34.586 INFO:tasks.workunit.client.1.smithi135.stdout:0/964: stat d0/d5/d10/f31 0 2024-03-30T04:28:34.587 INFO:tasks.workunit.client.1.smithi135.stdout:4/797: creat d10/d1a/d32/d40/d92/db9/dc6/f10a x:0 0 0 2024-03-30T04:28:34.594 INFO:tasks.workunit.client.1.smithi135.stdout:4/798: dread - d10/d1a/d42/d66/dc7/fce zero size 2024-03-30T04:28:34.597 INFO:tasks.workunit.client.1.smithi135.stdout:0/965: creat d0/d13/d91/d106/d11e/d67/f135 x:0 0 0 2024-03-30T04:28:34.598 INFO:tasks.workunit.client.1.smithi135.stdout:6/908: rmdir d2/dd/d14/d30/d43/d119 0 2024-03-30T04:28:34.604 INFO:tasks.workunit.client.1.smithi135.stdout:6/909: mkdir d2/dd/d14/d30/d43/d125 0 2024-03-30T04:28:34.607 INFO:tasks.workunit.client.1.smithi135.stdout:6/910: rmdir d2/dd/d40/db4 39 2024-03-30T04:28:34.609 INFO:tasks.workunit.client.1.smithi135.stdout:8/783: dread d6/d8/d15/d54/d7e/daf/dc1/fe1 [0,4194304] 0 2024-03-30T04:28:34.616 INFO:tasks.workunit.client.1.smithi135.stdout:6/911: creat d2/dd/d14/d30/d43/d112/f126 x:0 0 0 2024-03-30T04:28:34.619 INFO:tasks.workunit.client.1.smithi135.stdout:8/784: unlink d6/d8/d15/d54/d8e/d72/cb8 0 2024-03-30T04:28:34.622 INFO:tasks.workunit.client.1.smithi135.stdout:4/799: dwrite d10/d72/fde [0,4194304] 0 2024-03-30T04:28:34.623 INFO:tasks.workunit.client.1.smithi135.stdout:6/912: mknod d2/dd/d40/db4/c127 0 2024-03-30T04:28:34.626 INFO:tasks.workunit.client.1.smithi135.stdout:4/800: readlink d10/d1a/d42/l77 0 2024-03-30T04:28:34.629 INFO:tasks.workunit.client.1.smithi135.stdout:0/966: dwrite d0/f12 [0,4194304] 0 2024-03-30T04:28:34.629 INFO:tasks.workunit.client.1.smithi135.stdout:8/785: mknod d6/d23/dca/df0/c103 0 2024-03-30T04:28:34.631 INFO:tasks.workunit.client.1.smithi135.stdout:6/913: mkdir d2/dd/d18/db0/d128 0 2024-03-30T04:28:34.632 INFO:tasks.workunit.client.1.smithi135.stdout:8/786: chown d6/dc7/fe7 1832877773 1 2024-03-30T04:28:34.634 INFO:tasks.workunit.client.1.smithi135.stdout:9/791: sync 2024-03-30T04:28:34.642 INFO:tasks.workunit.client.1.smithi135.stdout:4/801: getdents d10/d1a/d32/d40/d92 0 2024-03-30T04:28:34.643 INFO:tasks.workunit.client.1.smithi135.stdout:4/802: stat d10 0 2024-03-30T04:28:34.647 INFO:tasks.workunit.client.1.smithi135.stdout:6/914: symlink d2/dd/d14/d2b/d86/l129 0 2024-03-30T04:28:34.651 INFO:tasks.workunit.client.1.smithi135.stdout:0/967: getdents d0/d13/d91/d106/d11e/d77 0 2024-03-30T04:28:34.652 INFO:tasks.workunit.client.1.smithi135.stdout:0/968: chown d0/db5/db8 7918510 1 2024-03-30T04:28:34.663 INFO:tasks.workunit.client.1.smithi135.stdout:4/803: dread d10/f14 [0,4194304] 0 2024-03-30T04:28:34.665 INFO:tasks.workunit.client.1.smithi135.stdout:6/915: link d2/dd/d14/d2b/d86/fc3 d2/dd/d18/f12a 0 2024-03-30T04:28:34.666 INFO:tasks.workunit.client.1.smithi135.stdout:8/787: truncate d6/d8/d15/d2a/fea 915975 0 2024-03-30T04:28:34.671 INFO:tasks.workunit.client.1.smithi135.stdout:6/916: truncate d2/d68/ffe 321004 0 2024-03-30T04:28:34.675 INFO:tasks.workunit.client.1.smithi135.stdout:0/969: dwrite d0/d13/d53/d7f/f119 [0,4194304] 0 2024-03-30T04:28:34.676 INFO:tasks.workunit.client.1.smithi135.stdout:4/804: write d10/d1a/d42/f5e [1264423,103784] 0 2024-03-30T04:28:34.679 INFO:tasks.workunit.client.1.smithi135.stdout:6/917: mknod d2/dd/d40/db4/c12b 0 2024-03-30T04:28:34.684 INFO:tasks.workunit.client.1.smithi135.stdout:6/918: readlink d2/dd/d14/l104 0 2024-03-30T04:28:34.688 INFO:tasks.workunit.client.1.smithi135.stdout:0/970: link d0/d13/f74 d0/d13/d53/d54/d12b/df9/f136 0 2024-03-30T04:28:34.688 INFO:tasks.workunit.client.1.smithi135.stdout:6/919: read d2/dd/d18/f22 [560948,42007] 0 2024-03-30T04:28:34.692 INFO:tasks.workunit.client.1.smithi135.stdout:8/788: dwrite d6/f11 [0,4194304] 0 2024-03-30T04:28:34.693 INFO:tasks.workunit.client.1.smithi135.stdout:6/920: mkdir d2/dd/d18/db0/d12c 0 2024-03-30T04:28:34.700 INFO:tasks.workunit.client.1.smithi135.stdout:4/805: dread d10/d1a/d32/f70 [0,4194304] 0 2024-03-30T04:28:34.701 INFO:tasks.workunit.client.1.smithi135.stdout:6/921: creat d2/dd/d18/d87/f12d x:0 0 0 2024-03-30T04:28:34.705 INFO:tasks.workunit.client.1.smithi135.stdout:0/971: link d0/d5/d9/d23/d2a/d60/d122/c133 d0/d13/d91/d106/d7e/c137 0 2024-03-30T04:28:34.705 INFO:tasks.workunit.client.1.smithi135.stdout:8/789: dread - d6/d8/d15/d79/f9c zero size 2024-03-30T04:28:34.705 INFO:tasks.workunit.client.1.smithi135.stdout:6/922: write d2/d68/da8/fb5 [362178,85021] 0 2024-03-30T04:28:34.712 INFO:tasks.workunit.client.1.smithi135.stdout:8/790: chown d6/d8/d15/f8a 1491 1 2024-03-30T04:28:34.721 INFO:tasks.workunit.client.1.smithi135.stdout:0/972: creat d0/d5/d9/f138 x:0 0 0 2024-03-30T04:28:34.723 INFO:tasks.workunit.client.1.smithi135.stdout:8/791: read - d6/d8/d15/d54/d8e/d72/ff8 zero size 2024-03-30T04:28:34.732 INFO:tasks.workunit.client.1.smithi135.stdout:8/792: creat d6/d8/d15/d2a/da9/f104 x:0 0 0 2024-03-30T04:28:34.736 INFO:tasks.workunit.client.1.smithi135.stdout:8/793: chown d6/d23/f82 1421853731 1 2024-03-30T04:28:34.743 INFO:tasks.workunit.client.1.smithi135.stdout:6/923: dwrite d2/dd/fd8 [0,4194304] 0 2024-03-30T04:28:34.747 INFO:tasks.workunit.client.1.smithi135.stdout:8/794: creat d6/d8/d15/d54/d7e/daf/f105 x:0 0 0 2024-03-30T04:28:34.749 INFO:tasks.workunit.client.1.smithi135.stdout:0/973: dread d0/d13/d53/d7f/f119 [0,4194304] 0 2024-03-30T04:28:34.753 INFO:tasks.workunit.client.1.smithi135.stdout:4/806: dread d10/d1a/d2b/f31 [0,4194304] 0 2024-03-30T04:28:34.754 INFO:tasks.workunit.client.1.smithi135.stdout:4/807: readlink d10/l5b 0 2024-03-30T04:28:34.754 INFO:tasks.workunit.client.1.smithi135.stdout:8/795: truncate d6/d8/d15/d54/d8e/faa 836858 0 2024-03-30T04:28:34.755 INFO:tasks.workunit.client.1.smithi135.stdout:0/974: write d0/d13/d53/d7f/d83/fea [3095692,128562] 0 2024-03-30T04:28:34.756 INFO:tasks.workunit.client.1.smithi135.stdout:6/924: rename d2/dd/lf to d2/dd/l12e 0 2024-03-30T04:28:34.758 INFO:tasks.workunit.client.1.smithi135.stdout:4/808: fsync d10/d1a/d42/f5e 0 2024-03-30T04:28:34.760 INFO:tasks.workunit.client.1.smithi135.stdout:8/796: dread - d6/d8/d15/d54/d8e/dbd/ff7 zero size 2024-03-30T04:28:34.762 INFO:tasks.workunit.client.1.smithi135.stdout:6/925: write d2/d68/f110 [645342,53] 0 2024-03-30T04:28:34.763 INFO:tasks.workunit.client.1.smithi135.stdout:0/975: rmdir d0/d13/d91/d106/d7e/db3/d12e 39 2024-03-30T04:28:34.767 INFO:tasks.workunit.client.1.smithi135.stdout:0/976: truncate d0/d5/d9/fc1 943154 0 2024-03-30T04:28:34.771 INFO:tasks.workunit.client.1.smithi135.stdout:6/926: dread d2/dd/d14/f15 [0,4194304] 0 2024-03-30T04:28:34.771 INFO:tasks.workunit.client.1.smithi135.stdout:0/977: mknod d0/d5/d42/d9f/c139 0 2024-03-30T04:28:34.775 INFO:tasks.workunit.client.1.smithi135.stdout:6/927: truncate d2/dd/f17 3743553 0 2024-03-30T04:28:34.780 INFO:tasks.workunit.client.1.smithi135.stdout:0/978: link d0/d13/d1e/lf6 d0/d5/d9/l13a 0 2024-03-30T04:28:34.783 INFO:tasks.workunit.client.1.smithi135.stdout:4/809: dwrite d10/d4d/d89/f8b [0,4194304] 0 2024-03-30T04:28:34.797 INFO:tasks.workunit.client.1.smithi135.stdout:0/979: truncate d0/d5/d10/f2c 2884646 0 2024-03-30T04:28:34.810 INFO:tasks.workunit.client.1.smithi135.stdout:6/928: dwrite d2/dd/d14/d2b/f85 [0,4194304] 0 2024-03-30T04:28:34.815 INFO:tasks.workunit.client.1.smithi135.stdout:0/980: write d0/d13/d53/d7f/f119 [4191402,99479] 0 2024-03-30T04:28:34.816 INFO:tasks.workunit.client.1.smithi135.stdout:6/929: fdatasync d2/dd/d14/d38/f4a 0 2024-03-30T04:28:34.826 INFO:tasks.workunit.client.1.smithi135.stdout:0/981: creat d0/d5/d9/d23/d2a/d60/d122/f13b x:0 0 0 2024-03-30T04:28:34.827 INFO:tasks.workunit.client.1.smithi135.stdout:4/810: dwrite fd [0,4194304] 0 2024-03-30T04:28:34.838 INFO:tasks.workunit.client.1.smithi135.stdout:4/811: getdents d10/d1a/d42/d66/dd7 0 2024-03-30T04:28:34.841 INFO:tasks.workunit.client.1.smithi135.stdout:4/812: chown d10/d1a/d32/l44 99482 1 2024-03-30T04:28:34.844 INFO:tasks.workunit.client.1.smithi135.stdout:4/813: chown d10/d4d/d89/d9f/fa7 4 1 2024-03-30T04:28:34.853 INFO:tasks.workunit.client.1.smithi135.stdout:4/814: rename d10/d1a/d32/d40/d92/db9/dcf/ld9 to d10/d1a/d32/d40/d92/db9/dcf/de8/d103/l10b 0 2024-03-30T04:28:34.858 INFO:tasks.workunit.client.1.smithi135.stdout:4/815: chown d10/d1a/d42/l77 8251 1 2024-03-30T04:28:34.861 INFO:tasks.workunit.client.1.smithi135.stdout:0/982: write d0/d13/d91/d106/d11e/d77/d8b/dcc/fd6 [902471,112184] 0 2024-03-30T04:28:34.869 INFO:tasks.workunit.client.1.smithi135.stdout:0/983: chown d0/d19/d2d/d45/cd9 12995 1 2024-03-30T04:28:34.870 INFO:tasks.workunit.client.1.smithi135.stdout:7/984: sync 2024-03-30T04:28:34.870 INFO:tasks.workunit.client.1.smithi135.stdout:5/758: sync 2024-03-30T04:28:34.873 INFO:tasks.workunit.client.1.smithi135.stdout:4/816: rename d10/d1a/d3d/l80 to d10/d4d/l10c 0 2024-03-30T04:28:34.877 INFO:tasks.workunit.client.1.smithi135.stdout:7/985: dread - d4/db/d1a/d3d/ddc/f111 zero size 2024-03-30T04:28:34.879 INFO:tasks.workunit.client.1.smithi135.stdout:6/930: dwrite d2/dd/d14/d2b/d86/fd1 [0,4194304] 0 2024-03-30T04:28:34.879 INFO:tasks.workunit.client.1.smithi135.stdout:0/984: write d0/d5/d9/d23/d2a/d60/d122/f13b [986316,124631] 0 2024-03-30T04:28:34.883 INFO:tasks.workunit.client.1.smithi135.stdout:4/817: chown d10/d72/d7c/f9b 2 1 2024-03-30T04:28:34.884 INFO:tasks.workunit.client.1.smithi135.stdout:5/759: write d1/da/f9f [1029771,72094] 0 2024-03-30T04:28:34.885 INFO:tasks.workunit.client.1.smithi135.stdout:7/986: read d4/db/fa0 [1439642,65800] 0 2024-03-30T04:28:34.886 INFO:tasks.workunit.client.1.smithi135.stdout:0/985: chown d0/d13/d91/dd2 680483 1 2024-03-30T04:28:34.887 INFO:tasks.workunit.client.1.smithi135.stdout:6/931: chown d2/dd/d14/d38/c49 230 1 2024-03-30T04:28:34.898 INFO:tasks.workunit.client.1.smithi135.stdout:5/760: chown d1/d7/fb1 53616678 1 2024-03-30T04:28:34.904 INFO:tasks.workunit.client.1.smithi135.stdout:6/932: symlink d2/d2e/l12f 0 2024-03-30T04:28:34.906 INFO:tasks.workunit.client.1.smithi135.stdout:7/987: mknod d4/db/d1a/d3d/c151 0 2024-03-30T04:28:34.908 INFO:tasks.workunit.client.1.smithi135.stdout:6/933: stat d2/d106/f114 0 2024-03-30T04:28:34.913 INFO:tasks.workunit.client.1.smithi135.stdout:5/761: rename d1/d3/d76/d92/da0 to d1/d7/d1a/d1f/d9b/de9 0 2024-03-30T04:28:34.917 INFO:tasks.workunit.client.1.smithi135.stdout:6/934: creat d2/dd/d40/db4/f130 x:0 0 0 2024-03-30T04:28:34.917 INFO:tasks.workunit.client.1.smithi135.stdout:5/762: dread - d1/d45/d52/d8f/fd2 zero size 2024-03-30T04:28:34.918 INFO:tasks.workunit.client.1.smithi135.stdout:7/988: rename d4/db/d79/d126 to d4/db/d1a/d4b/d152 0 2024-03-30T04:28:34.925 INFO:tasks.workunit.client.1.smithi135.stdout:7/989: fdatasync d4/db/d1a/d5c/d2c/d51/da8/f108 0 2024-03-30T04:28:34.927 INFO:tasks.workunit.client.1.smithi135.stdout:4/818: dwrite d10/d72/d7c/fa2 [0,4194304] 0 2024-03-30T04:28:34.932 INFO:tasks.workunit.client.1.smithi135.stdout:5/763: rename d1/fa1 to d1/d7/d11/fea 0 2024-03-30T04:28:34.933 INFO:tasks.workunit.client.1.smithi135.stdout:0/986: dwrite d0/d5/d9/d23/d2a/d60/f109 [0,4194304] 0 2024-03-30T04:28:34.935 INFO:tasks.workunit.client.1.smithi135.stdout:6/935: creat d2/dd/d14/d38/d11d/f131 x:0 0 0 2024-03-30T04:28:34.936 INFO:tasks.workunit.client.1.smithi135.stdout:6/936: readlink d2/d2e/l12f 0 2024-03-30T04:28:34.938 INFO:tasks.workunit.client.1.smithi135.stdout:7/990: unlink d4/db/d1a/d5c/d2c/d2e/db9/df4/l136 0 2024-03-30T04:28:34.938 INFO:tasks.workunit.client.1.smithi135.stdout:6/937: stat d2/dd/d14/d2b/d11e 0 2024-03-30T04:28:34.942 INFO:tasks.workunit.client.1.smithi135.stdout:7/991: chown d4/db/d1a/d5c/d2c/d51/d56/d81/fef 6727217 1 2024-03-30T04:28:34.946 INFO:tasks.workunit.client.1.smithi135.stdout:7/992: readlink d4/db/d1a/d4b/lda 0 2024-03-30T04:28:34.946 INFO:tasks.workunit.client.1.smithi135.stdout:4/819: link d10/d1a/d3d/f91 d10/d72/d7c/dfb/dd4/dda/f10d 0 2024-03-30T04:28:34.946 INFO:tasks.workunit.client.1.smithi135.stdout:0/987: getdents d0/d5 0 2024-03-30T04:28:34.950 INFO:tasks.workunit.client.1.smithi135.stdout:4/820: fsync d10/d1a/d32/d40/d92/db9/dcf/de8/ff4 0 2024-03-30T04:28:34.951 INFO:tasks.workunit.client.1.smithi135.stdout:0/988: mknod d0/db5/c13c 0 2024-03-30T04:28:34.953 INFO:tasks.workunit.client.1.smithi135.stdout:4/821: truncate d10/d4d/d89/f8f 1265982 0 2024-03-30T04:28:34.959 INFO:tasks.workunit.client.1.smithi135.stdout:7/993: mkdir d4/db/d1a/d5c/d59/d76/d14c/d153 0 2024-03-30T04:28:34.961 INFO:tasks.workunit.client.1.smithi135.stdout:6/938: dread d2/dd/d18/f4b [0,4194304] 0 2024-03-30T04:28:34.962 INFO:tasks.workunit.client.1.smithi135.stdout:0/989: creat d0/d13/d53/d54/d99/f13d x:0 0 0 2024-03-30T04:28:34.962 INFO:tasks.workunit.client.1.smithi135.stdout:4/822: fdatasync d10/d1a/d42/f63 0 2024-03-30T04:28:34.967 INFO:tasks.workunit.client.1.smithi135.stdout:4/823: chown d10/d1a/d3d/c71 3740 1 2024-03-30T04:28:34.972 INFO:tasks.workunit.client.1.smithi135.stdout:6/939: creat d2/f132 x:0 0 0 2024-03-30T04:28:34.975 INFO:tasks.workunit.client.1.smithi135.stdout:5/764: dread d1/f6e [0,4194304] 0 2024-03-30T04:28:34.976 INFO:tasks.workunit.client.1.smithi135.stdout:7/994: creat d4/db/d1a/d5c/d2c/d51/d56/d81/f154 x:0 0 0 2024-03-30T04:28:34.979 INFO:tasks.workunit.client.1.smithi135.stdout:5/765: creat d1/da/d17/feb x:0 0 0 2024-03-30T04:28:34.984 INFO:tasks.workunit.client.1.smithi135.stdout:7/995: mknod d4/db/d1a/d4b/c155 0 2024-03-30T04:28:34.986 INFO:tasks.workunit.client.1.smithi135.stdout:0/990: write d0/d13/d1e/ff2 [122235,69583] 0 2024-03-30T04:28:34.994 INFO:tasks.workunit.client.1.smithi135.stdout:0/991: creat d0/d5/d42/f13e x:0 0 0 2024-03-30T04:28:34.999 INFO:tasks.workunit.client.1.smithi135.stdout:5/766: dwrite d1/d3/d76/f98 [0,4194304] 0 2024-03-30T04:28:35.008 INFO:tasks.workunit.client.1.smithi135.stdout:5/767: mkdir d1/d7/d1a/dec 0 2024-03-30T04:28:35.009 INFO:tasks.workunit.client.1.smithi135.stdout:7/996: dwrite d4/f8 [0,4194304] 0 2024-03-30T04:28:35.052 INFO:tasks.workunit.client.1.smithi135.stdout:1/852: sync 2024-03-30T04:28:35.055 INFO:tasks.workunit.client.1.smithi135.stdout:5/768: dread d1/da/f9f [0,4194304] 0 2024-03-30T04:28:35.062 INFO:tasks.workunit.client.1.smithi135.stdout:0/992: link d0/d19/d2d/f118 d0/d5/d9/d23/d2a/f13f 0 2024-03-30T04:28:35.063 INFO:tasks.workunit.client.1.smithi135.stdout:7/997: truncate d4/db/d1a/d90/dc7/fe9 28392 0 2024-03-30T04:28:35.066 INFO:tasks.workunit.client.1.smithi135.stdout:0/993: mknod d0/d13/d91/dd2/c140 0 2024-03-30T04:28:35.074 INFO:tasks.workunit.client.1.smithi135.stdout:0/994: rename d0/d5/d9/d23/d2a/d60/d122/f13b to d0/d13/d53/d54/d12b/df9/f141 0 2024-03-30T04:28:35.080 INFO:tasks.workunit.client.1.smithi135.stdout:7/998: write d4/db/d1a/d3d/ddc/db0/f134 [693492,80030] 0 2024-03-30T04:28:35.081 INFO:tasks.workunit.client.1.smithi135.stdout:0/995: mkdir d0/d13/d91/d106/d7e/db3/d12e/d142 0 2024-03-30T04:28:35.085 INFO:tasks.workunit.client.1.smithi135.stdout:1/853: dwrite d16/d5a/dad/d103/f10f [0,4194304] 0 2024-03-30T04:28:35.085 INFO:tasks.workunit.client.1.smithi135.stdout:7/999: symlink d4/db/d1a/d90/dea/l156 0 2024-03-30T04:28:35.096 INFO:tasks.workunit.client.1.smithi135.stdout:1/854: write d16/d29/d61/fd7 [193549,15278] 0 2024-03-30T04:28:35.102 INFO:tasks.workunit.client.1.smithi135.stdout:0/996: dread d0/d5/d9/d23/d2a/f51 [0,4194304] 0 2024-03-30T04:28:35.104 INFO:tasks.workunit.client.1.smithi135.stdout:0/997: chown d0/d5/d42/d87/la2 49304576 1 2024-03-30T04:28:35.122 INFO:tasks.workunit.client.1.smithi135.stdout:0/998: dwrite d0/d13/d91/d106/dd5/f100 [0,4194304] 0 2024-03-30T04:28:35.128 INFO:tasks.workunit.client.1.smithi135.stdout:0/999: rename d0/d34/d3f/c115 to d0/d13/d91/dd2/c143 0 2024-03-30T04:28:35.198 INFO:tasks.workunit.client.1.smithi135.stdout:2/749: sync 2024-03-30T04:28:35.199 INFO:tasks.workunit.client.1.smithi135.stdout:3/832: sync 2024-03-30T04:28:35.201 INFO:tasks.workunit.client.1.smithi135.stdout:2/750: mknod d4/d32/d12/d45/dac/cfc 0 2024-03-30T04:28:35.204 INFO:tasks.workunit.client.1.smithi135.stdout:3/833: getdents d3/d10/d16/d23/d6a/d86/d98 0 2024-03-30T04:28:35.207 INFO:tasks.workunit.client.1.smithi135.stdout:3/834: mknod d3/d10/d16/d23/d6a/d94/c11b 0 2024-03-30T04:28:35.208 INFO:tasks.workunit.client.1.smithi135.stdout:3/835: readlink d3/db/lda 0 2024-03-30T04:28:35.209 INFO:tasks.workunit.client.1.smithi135.stdout:3/836: chown d3/db/da9/dcc/lf0 23 1 2024-03-30T04:28:35.213 INFO:tasks.workunit.client.1.smithi135.stdout:3/837: symlink d3/d10/d16/d41/l11c 0 2024-03-30T04:28:35.215 INFO:tasks.workunit.client.1.smithi135.stdout:3/838: write d3/d10/d16/d23/d6a/d94/fb9 [1813997,122058] 0 2024-03-30T04:28:35.220 INFO:tasks.workunit.client.1.smithi135.stdout:2/751: dread d4/d32/f13 [0,4194304] 0 2024-03-30T04:28:35.227 INFO:tasks.workunit.client.1.smithi135.stdout:3/839: truncate d3/d10/d16/d23/d6a/fe3 121863 0 2024-03-30T04:28:35.228 INFO:tasks.workunit.client.1.smithi135.stdout:2/752: write d4/d32/f13 [19107,87052] 0 2024-03-30T04:28:35.234 INFO:tasks.workunit.client.1.smithi135.stdout:2/753: link d4/d32/d12/d45/d7d/lfb d4/d32/d12/d45/d7d/de7/lfd 0 2024-03-30T04:28:35.252 INFO:tasks.workunit.client.1.smithi135.stdout:3/840: dread d3/d50/d22/d74/f7c [0,4194304] 0 2024-03-30T04:28:35.253 INFO:tasks.workunit.client.1.smithi135.stdout:2/754: dread d4/da/f21 [0,4194304] 0 2024-03-30T04:28:35.254 INFO:tasks.workunit.client.1.smithi135.stdout:3/841: mkdir d3/d50/d46/d11d 0 2024-03-30T04:28:35.256 INFO:tasks.workunit.client.1.smithi135.stdout:2/755: creat d4/d6/d47/ffe x:0 0 0 2024-03-30T04:28:35.258 INFO:tasks.workunit.client.1.smithi135.stdout:3/842: mknod d3/d10/d16/d23/d6a/d86/d98/c11e 0 2024-03-30T04:28:35.259 INFO:tasks.workunit.client.1.smithi135.stdout:2/756: fsync d4/d32/d2e/d67/dad/fe1 0 2024-03-30T04:28:35.260 INFO:tasks.workunit.client.1.smithi135.stdout:3/843: readlink d3/d10/d9c/dbe/dd6/dd7/le5 0 2024-03-30T04:28:35.262 INFO:tasks.workunit.client.1.smithi135.stdout:2/757: getdents d4/d32/d12/d45/d6d/dea 0 2024-03-30T04:28:35.265 INFO:tasks.workunit.client.1.smithi135.stdout:3/844: dread d3/d10/d9c/dbe/dd6/dd7/f100 [0,4194304] 0 2024-03-30T04:28:35.268 INFO:tasks.workunit.client.1.smithi135.stdout:9/792: sync 2024-03-30T04:28:35.268 INFO:tasks.workunit.client.1.smithi135.stdout:4/824: sync 2024-03-30T04:28:35.268 INFO:tasks.workunit.client.1.smithi135.stdout:8/797: sync 2024-03-30T04:28:35.269 INFO:tasks.workunit.client.1.smithi135.stdout:3/845: creat d3/d54/f11f x:0 0 0 2024-03-30T04:28:35.270 INFO:tasks.workunit.client.1.smithi135.stdout:8/798: rmdir d6 39 2024-03-30T04:28:35.274 INFO:tasks.workunit.client.1.smithi135.stdout:9/793: rename d7/d90/cef to d7/d4f/d5c/d63/cfc 0 2024-03-30T04:28:35.276 INFO:tasks.workunit.client.1.smithi135.stdout:3/846: creat d3/d10/d9c/dbe/dd6/d91/f120 x:0 0 0 2024-03-30T04:28:35.278 INFO:tasks.workunit.client.1.smithi135.stdout:4/825: link d10/d4d/dc1/fbf d10/d4d/dc3/f10e 0 2024-03-30T04:28:35.278 INFO:tasks.workunit.client.1.smithi135.stdout:8/799: creat d6/d8/d15/f106 x:0 0 0 2024-03-30T04:28:35.280 INFO:tasks.workunit.client.1.smithi135.stdout:9/794: creat d7/d4f/d6e/dad/ffd x:0 0 0 2024-03-30T04:28:35.280 INFO:tasks.workunit.client.1.smithi135.stdout:4/826: write d10/fe9 [1705387,127256] 0 2024-03-30T04:28:35.281 INFO:tasks.workunit.client.1.smithi135.stdout:8/800: fdatasync d6/f50 0 2024-03-30T04:28:35.284 INFO:tasks.workunit.client.1.smithi135.stdout:9/795: write d7/d90/fa0 [1865000,112556] 0 2024-03-30T04:28:35.286 INFO:tasks.workunit.client.1.smithi135.stdout:2/758: dread d4/da/fb [0,4194304] 0 2024-03-30T04:28:35.288 INFO:tasks.workunit.client.1.smithi135.stdout:8/801: mkdir d6/d8/d15/d1f/ddb/d107 0 2024-03-30T04:28:35.291 INFO:tasks.workunit.client.1.smithi135.stdout:4/827: truncate d10/d1a/d3d/f5a 3947352 0 2024-03-30T04:28:35.291 INFO:tasks.workunit.client.1.smithi135.stdout:9/796: mkdir d7/d4f/d5c/d63/dd2/dfe 0 2024-03-30T04:28:35.295 INFO:tasks.workunit.client.1.smithi135.stdout:3/847: dwrite d3/d54/dab/fd3 [0,4194304] 0 2024-03-30T04:28:35.295 INFO:tasks.workunit.client.1.smithi135.stdout:2/759: write d4/d32/d12/d45/f8e [3991394,37805] 0 2024-03-30T04:28:35.304 INFO:tasks.workunit.client.1.smithi135.stdout:2/760: symlink d4/d32/d43/db5/lff 0 2024-03-30T04:28:35.312 INFO:tasks.workunit.client.1.smithi135.stdout:2/761: truncate d4/da/d8d/fab 1867467 0 2024-03-30T04:28:35.314 INFO:tasks.workunit.client.1.smithi135.stdout:9/797: dwrite d7/fe3 [0,4194304] 0 2024-03-30T04:28:35.316 INFO:tasks.workunit.client.1.smithi135.stdout:8/802: dread d6/f50 [4194304,4194304] 0 2024-03-30T04:28:35.318 INFO:tasks.workunit.client.1.smithi135.stdout:2/762: fdatasync d4/d32/d2e/f8c 0 2024-03-30T04:28:35.321 INFO:tasks.workunit.client.1.smithi135.stdout:8/803: stat d6/d8/d15/la2 0 2024-03-30T04:28:35.321 INFO:tasks.workunit.client.1.smithi135.stdout:9/798: symlink d7/d4f/d89/lff 0 2024-03-30T04:28:35.323 INFO:tasks.workunit.client.1.smithi135.stdout:2/763: fdatasync d4/d32/d2e/f40 0 2024-03-30T04:28:35.324 INFO:tasks.workunit.client.1.smithi135.stdout:9/799: symlink d7/d55/l100 0 2024-03-30T04:28:35.325 INFO:tasks.workunit.client.1.smithi135.stdout:2/764: read - d4/d6/d47/ffe zero size 2024-03-30T04:28:35.330 INFO:tasks.workunit.client.1.smithi135.stdout:8/804: truncate d6/d8/d15/d2a/fea 893323 0 2024-03-30T04:28:35.333 INFO:tasks.workunit.client.1.smithi135.stdout:2/765: getdents d4/d55/d77 0 2024-03-30T04:28:35.336 INFO:tasks.workunit.client.1.smithi135.stdout:2/766: rmdir d4/da/d8d 39 2024-03-30T04:28:35.345 INFO:tasks.workunit.client.1.smithi135.stdout:9/800: truncate d7/d55/f5b 433664 0 2024-03-30T04:28:35.346 INFO:tasks.workunit.client.1.smithi135.stdout:9/801: readlink d7/l4b 0 2024-03-30T04:28:35.346 INFO:tasks.workunit.client.1.smithi135.stdout:8/805: write d6/d8/d15/d2a/dee/f29 [4578377,117778] 0 2024-03-30T04:28:35.349 INFO:tasks.workunit.client.1.smithi135.stdout:8/806: dread - d6/d8/d15/d2a/da9/fba zero size 2024-03-30T04:28:35.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:34 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:28:35.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:34 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:28:35.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:34 smithi135 ceph-mon[44364]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:28:35.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:34 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:28:35.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:34 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:28:35.351 INFO:tasks.workunit.client.1.smithi135.stdout:9/802: creat d7/d4f/dee/f101 x:0 0 0 2024-03-30T04:28:35.352 INFO:tasks.workunit.client.1.smithi135.stdout:8/807: chown d6/ld 49254 1 2024-03-30T04:28:35.354 INFO:tasks.workunit.client.1.smithi135.stdout:3/848: dread d3/d54/dab/fde [0,4194304] 0 2024-03-30T04:28:35.354 INFO:tasks.workunit.client.1.smithi135.stdout:8/808: dread - d6/f86 zero size 2024-03-30T04:28:35.355 INFO:tasks.workunit.client.1.smithi135.stdout:2/767: dwrite d4/da/d8d/fe2 [0,4194304] 0 2024-03-30T04:28:35.356 INFO:tasks.workunit.client.1.smithi135.stdout:8/809: stat d6/d23/d7b/fcd 0 2024-03-30T04:28:35.358 INFO:tasks.workunit.client.1.smithi135.stdout:3/849: mkdir d3/d50/d22/d33/d121 0 2024-03-30T04:28:35.359 INFO:tasks.workunit.client.1.smithi135.stdout:2/768: mkdir d4/d32/d2e/d67/dad/de5/d100 0 2024-03-30T04:28:35.361 INFO:tasks.workunit.client.1.smithi135.stdout:3/850: write d3/d6e/f80 [4281275,87047] 0 2024-03-30T04:28:35.367 INFO:tasks.workunit.client.1.smithi135.stdout:2/769: link d4/fef d4/d55/dda/f101 0 2024-03-30T04:28:35.369 INFO:tasks.workunit.client.1.smithi135.stdout:3/851: symlink d3/d10/d16/d41/l122 0 2024-03-30T04:28:35.371 INFO:tasks.workunit.client.1.smithi135.stdout:3/852: fdatasync d3/f45 0 2024-03-30T04:28:35.374 INFO:tasks.workunit.client.1.smithi135.stdout:2/770: truncate d4/d32/d43/fa9 4540730 0 2024-03-30T04:28:35.376 INFO:tasks.workunit.client.1.smithi135.stdout:5/769: sync 2024-03-30T04:28:35.376 INFO:tasks.workunit.client.1.smithi135.stdout:3/853: fsync d3/d10/f31 0 2024-03-30T04:28:35.376 INFO:tasks.workunit.client.1.smithi135.stdout:6/940: sync 2024-03-30T04:28:35.378 INFO:tasks.workunit.client.1.smithi135.stdout:1/855: sync 2024-03-30T04:28:35.378 INFO:tasks.workunit.client.1.smithi135.stdout:8/810: dwrite d6/d8/d15/d54/d7e/daf/dc1/fe1 [0,4194304] 0 2024-03-30T04:28:35.380 INFO:tasks.workunit.client.1.smithi135.stdout:2/771: unlink d4/da/d8d/f95 0 2024-03-30T04:28:35.390 INFO:tasks.workunit.client.1.smithi135.stdout:9/803: dread d7/d29/f62 [0,4194304] 0 2024-03-30T04:28:35.392 INFO:tasks.workunit.client.1.smithi135.stdout:8/811: stat d6/d8/d15/d2a/dee/d32/c89 0 2024-03-30T04:28:35.401 INFO:tasks.workunit.client.1.smithi135.stdout:9/804: mknod d7/d4f/c102 0 2024-03-30T04:28:35.403 INFO:tasks.workunit.client.1.smithi135.stdout:3/854: write d3/d50/d46/f58 [35766,27217] 0 2024-03-30T04:28:35.407 INFO:tasks.workunit.client.1.smithi135.stdout:2/772: truncate d4/da/d7a/dd6/fdf 550852 0 2024-03-30T04:28:35.408 INFO:tasks.workunit.client.1.smithi135.stdout:9/805: mknod d7/d4f/d5c/d63/dd2/db1/dbc/dd8/c103 0 2024-03-30T04:28:35.414 INFO:tasks.workunit.client.1.smithi135.stdout:4/828: sync 2024-03-30T04:28:35.416 INFO:tasks.workunit.client.1.smithi135.stdout:2/773: mknod d4/d32/d12/d45/dac/c102 0 2024-03-30T04:28:35.419 INFO:tasks.workunit.client.1.smithi135.stdout:8/812: dwrite d6/d8/d15/d2a/dee/d32/f62 [4194304,4194304] 0 2024-03-30T04:28:35.423 INFO:tasks.workunit.client.1.smithi135.stdout:4/829: mkdir d10/d1a/d32/d40/d10f 0 2024-03-30T04:28:35.423 INFO:tasks.workunit.client.1.smithi135.stdout:5/770: dwrite d1/da/d17/d70/db2/d79/de1/f57 [0,4194304] 0 2024-03-30T04:28:35.435 INFO:tasks.workunit.client.1.smithi135.stdout:1/856: dwrite d16/d29/d61/f98 [0,4194304] 0 2024-03-30T04:28:35.436 INFO:tasks.workunit.client.1.smithi135.stdout:3/855: truncate d3/d50/d46/d4c/fb1 1080147 0 2024-03-30T04:28:35.437 INFO:tasks.workunit.client.1.smithi135.stdout:6/941: dwrite d2/dd/d14/d38/f7e [4194304,4194304] 0 2024-03-30T04:28:35.455 INFO:tasks.workunit.client.1.smithi135.stdout:9/806: dwrite d7/d29/d78/da4/f53 [0,4194304] 0 2024-03-30T04:28:35.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:34 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:28:35.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:34 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:28:35.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:34 smithi123 ceph-mon[35245]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:28:35.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:34 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:28:35.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:34 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:28:35.474 INFO:tasks.workunit.client.1.smithi135.stdout:8/813: fsync d6/d8/d15/d2a/f33 0 2024-03-30T04:28:35.477 INFO:tasks.workunit.client.1.smithi135.stdout:4/830: creat d10/d1a/d32/d40/d92/db9/dcf/f110 x:0 0 0 2024-03-30T04:28:35.478 INFO:tasks.workunit.client.1.smithi135.stdout:4/831: fdatasync d10/d1a/d42/f54 0 2024-03-30T04:28:35.480 INFO:tasks.workunit.client.1.smithi135.stdout:6/942: read d2/d2e/f31 [1895453,51368] 0 2024-03-30T04:28:35.483 INFO:tasks.workunit.client.1.smithi135.stdout:1/857: symlink d16/d5a/d5f/d3d/d8d/dc9/l11f 0 2024-03-30T04:28:35.484 INFO:tasks.workunit.client.1.smithi135.stdout:8/814: chown d6/d8/fa5 12 1 2024-03-30T04:28:35.485 INFO:tasks.workunit.client.1.smithi135.stdout:1/858: readlink d16/d5a/d5f/d3a/lff 0 2024-03-30T04:28:35.489 INFO:tasks.workunit.client.1.smithi135.stdout:1/859: write d16/d29/d61/f98 [427243,36285] 0 2024-03-30T04:28:35.489 INFO:tasks.workunit.client.1.smithi135.stdout:4/832: mknod d10/d1a/d32/d40/d92/db9/dcf/de8/d103/c111 0 2024-03-30T04:28:35.489 INFO:tasks.workunit.client.1.smithi135.stdout:6/943: symlink d2/dd/d40/d52/l133 0 2024-03-30T04:28:35.491 INFO:tasks.workunit.client.1.smithi135.stdout:5/771: getdents d1/da/d17/d70 0 2024-03-30T04:28:35.492 INFO:tasks.workunit.client.1.smithi135.stdout:8/815: stat d6/d8/d15/d2a/dee/d32/fab 0 2024-03-30T04:28:35.494 INFO:tasks.workunit.client.1.smithi135.stdout:1/860: rmdir d16/d29/d2c 39 2024-03-30T04:28:35.498 INFO:tasks.workunit.client.1.smithi135.stdout:4/833: mkdir d10/d1a/d32/d40/d92/db9/dcf/d112 0 2024-03-30T04:28:35.499 INFO:tasks.workunit.client.1.smithi135.stdout:4/834: stat d10/d1a/d2b/l7b 0 2024-03-30T04:28:35.500 INFO:tasks.workunit.client.1.smithi135.stdout:8/816: unlink d6/d8/d15/d79/f9c 0 2024-03-30T04:28:35.500 INFO:tasks.workunit.client.1.smithi135.stdout:6/944: dread d2/dd/d40/db4/fee [0,4194304] 0 2024-03-30T04:28:35.504 INFO:tasks.workunit.client.1.smithi135.stdout:5/772: creat d1/da/d17/d70/db2/d79/d8e/fed x:0 0 0 2024-03-30T04:28:35.504 INFO:tasks.workunit.client.1.smithi135.stdout:1/861: symlink d16/d29/l120 0 2024-03-30T04:28:35.507 INFO:tasks.workunit.client.1.smithi135.stdout:1/862: fsync d16/f2f 0 2024-03-30T04:28:35.507 INFO:tasks.workunit.client.1.smithi135.stdout:5/773: chown d1/d7/d11/d3c/l74 144416 1 2024-03-30T04:28:35.510 INFO:tasks.workunit.client.1.smithi135.stdout:9/807: write d7/d55/d8d/faf [1739418,42727] 0 2024-03-30T04:28:35.510 INFO:tasks.workunit.client.1.smithi135.stdout:4/835: symlink d10/d1a/d42/d46/def/l113 0 2024-03-30T04:28:35.511 INFO:tasks.workunit.client.1.smithi135.stdout:8/817: mkdir d6/d8/d15/d54/d7e/daf/dfb/dfd/d108 0 2024-03-30T04:28:35.514 INFO:tasks.workunit.client.1.smithi135.stdout:5/774: creat d1/d3/d76/fee x:0 0 0 2024-03-30T04:28:35.515 INFO:tasks.workunit.client.1.smithi135.stdout:1/863: truncate d16/d29/d2c/d3c/f117 624072 0 2024-03-30T04:28:35.516 INFO:tasks.workunit.client.1.smithi135.stdout:5/775: dread - d1/d7/d11/d3c/d4b/fe4 zero size 2024-03-30T04:28:35.516 INFO:tasks.workunit.client.1.smithi135.stdout:4/836: mknod d10/d1a/d42/de0/c114 0 2024-03-30T04:28:35.518 INFO:tasks.workunit.client.1.smithi135.stdout:1/864: chown d16/d5a/d5f/d3d/d8d/dc9/f72 1782169974 1 2024-03-30T04:28:35.519 INFO:tasks.workunit.client.1.smithi135.stdout:9/808: mknod d7/df8/dfb/c104 0 2024-03-30T04:28:35.520 INFO:tasks.workunit.client.1.smithi135.stdout:8/818: creat d6/d23/df9/f109 x:0 0 0 2024-03-30T04:28:35.521 INFO:tasks.workunit.client.1.smithi135.stdout:4/837: creat d10/d1a/d32/d40/d92/db9/dcf/de8/f115 x:0 0 0 2024-03-30T04:28:35.523 INFO:tasks.workunit.client.1.smithi135.stdout:9/809: write d7/d55/f6c [4881070,126461] 0 2024-03-30T04:28:35.527 INFO:tasks.workunit.client.1.smithi135.stdout:6/945: write d2/dd/d14/d2b/d86/d55/ff9 [737645,81099] 0 2024-03-30T04:28:35.528 INFO:tasks.workunit.client.1.smithi135.stdout:5/776: rename d1/d3/d2b/d54/l63 to d1/da/d17/d70/db2/dd1/lef 0 2024-03-30T04:28:35.529 INFO:tasks.workunit.client.1.smithi135.stdout:1/865: creat d16/d5a/d5f/d3d/d8d/dc9/da4/d26/f121 x:0 0 0 2024-03-30T04:28:35.530 INFO:tasks.workunit.client.1.smithi135.stdout:5/777: readlink d1/d45/lbe 0 2024-03-30T04:28:35.530 INFO:tasks.workunit.client.1.smithi135.stdout:4/838: mkdir d10/d1a/d42/d116 0 2024-03-30T04:28:35.531 INFO:tasks.workunit.client.1.smithi135.stdout:9/810: creat d7/d4f/d5c/d63/dd2/dfe/f105 x:0 0 0 2024-03-30T04:28:35.535 INFO:tasks.workunit.client.1.smithi135.stdout:1/866: truncate d16/d5a/d5f/d3a/fa7 4695223 0 2024-03-30T04:28:35.535 INFO:tasks.workunit.client.1.smithi135.stdout:6/946: rename d2/dd/d14/d38/dde/ce6 to d2/dd/d14/d38/d11d/c134 0 2024-03-30T04:28:35.543 INFO:tasks.workunit.client.1.smithi135.stdout:1/867: dread d16/d5a/d5f/d3d/fcb [0,4194304] 0 2024-03-30T04:28:35.547 INFO:tasks.workunit.client.1.smithi135.stdout:5/778: dread d1/d7/d1a/d1f/d9b/de9/f9c [0,4194304] 0 2024-03-30T04:28:35.550 INFO:tasks.workunit.client.1.smithi135.stdout:1/868: creat d16/d29/d32/dcc/d10a/f122 x:0 0 0 2024-03-30T04:28:35.551 INFO:tasks.workunit.client.1.smithi135.stdout:5/779: mkdir d1/da/d17/d70/df0 0 2024-03-30T04:28:35.551 INFO:tasks.workunit.client.1.smithi135.stdout:4/839: write d10/d72/fd2 [694027,119969] 0 2024-03-30T04:28:35.557 INFO:tasks.workunit.client.1.smithi135.stdout:5/780: dread d1/da/f84 [0,4194304] 0 2024-03-30T04:28:35.565 INFO:tasks.workunit.client.1.smithi135.stdout:6/947: dwrite d2/dd/d14/d38/f9f [0,4194304] 0 2024-03-30T04:28:35.565 INFO:tasks.workunit.client.1.smithi135.stdout:9/811: dwrite d7/d29/d78/da4/d37/f39 [0,4194304] 0 2024-03-30T04:28:35.573 INFO:tasks.workunit.client.1.smithi135.stdout:5/781: rename d1/d3/d76/d92 to d1/d45/d52/da5/df1 0 2024-03-30T04:28:35.574 INFO:tasks.workunit.client.1.smithi135.stdout:9/812: mknod d7/d29/d78/c106 0 2024-03-30T04:28:35.579 INFO:tasks.workunit.client.1.smithi135.stdout:5/782: chown d1/da/d4d/dbd 3 1 2024-03-30T04:28:35.585 INFO:tasks.workunit.client.1.smithi135.stdout:5/783: write d1/d45/d52/f90 [3144650,118656] 0 2024-03-30T04:28:35.588 INFO:tasks.workunit.client.1.smithi135.stdout:6/948: write d2/dd/d40/fd6 [685505,116683] 0 2024-03-30T04:28:35.590 INFO:tasks.workunit.client.1.smithi135.stdout:9/813: mkdir d7/d41/d72/d107 0 2024-03-30T04:28:35.593 INFO:tasks.workunit.client.1.smithi135.stdout:4/840: dwrite d10/d1a/d42/d66/dc7/fce [0,4194304] 0 2024-03-30T04:28:35.599 INFO:tasks.workunit.client.1.smithi135.stdout:6/949: rmdir d2/dd/d14/d30 39 2024-03-30T04:28:35.603 INFO:tasks.workunit.client.1.smithi135.stdout:1/869: dread d16/d5a/d5f/d3d/d8d/dc9/f72 [4194304,4194304] 0 2024-03-30T04:28:35.606 INFO:tasks.workunit.client.1.smithi135.stdout:9/814: getdents d7/d4f/d89 0 2024-03-30T04:28:35.609 INFO:tasks.workunit.client.1.smithi135.stdout:6/950: creat d2/dd/d18/db0/d12c/f135 x:0 0 0 2024-03-30T04:28:35.610 INFO:tasks.workunit.client.1.smithi135.stdout:1/870: chown d16/d5a/d5f/d3d/d8d/dc9/c102 28 1 2024-03-30T04:28:35.610 INFO:tasks.workunit.client.1.smithi135.stdout:4/841: write d10/d4d/f87 [441098,21339] 0 2024-03-30T04:28:35.612 INFO:tasks.workunit.client.1.smithi135.stdout:9/815: dread - d7/d4f/d5c/f5e zero size 2024-03-30T04:28:35.612 INFO:tasks.workunit.client.1.smithi135.stdout:4/842: chown d10 1006045540 1 2024-03-30T04:28:35.619 INFO:tasks.workunit.client.1.smithi135.stdout:4/843: mknod d10/d72/d7c/dfb/c117 0 2024-03-30T04:28:35.620 INFO:tasks.workunit.client.1.smithi135.stdout:1/871: dread d16/d5a/f92 [0,4194304] 0 2024-03-30T04:28:35.624 INFO:tasks.workunit.client.1.smithi135.stdout:6/951: link d2/dd/d14/f7f d2/f136 0 2024-03-30T04:28:35.626 INFO:tasks.workunit.client.1.smithi135.stdout:4/844: link d10/d72/ff6 d10/d1a/f118 0 2024-03-30T04:28:35.633 INFO:tasks.workunit.client.1.smithi135.stdout:4/845: fdatasync d10/d1a/d42/f62 0 2024-03-30T04:28:35.635 INFO:tasks.workunit.client.1.smithi135.stdout:9/816: dwrite d7/d29/d78/da4/d37/dba/fda [0,4194304] 0 2024-03-30T04:28:35.638 INFO:tasks.workunit.client.1.smithi135.stdout:4/846: fdatasync d10/d72/d7c/dfb/f100 0 2024-03-30T04:28:35.645 INFO:tasks.workunit.client.1.smithi135.stdout:6/952: dread d2/dd/d14/d2b/f85 [0,4194304] 0 2024-03-30T04:28:35.654 INFO:tasks.workunit.client.1.smithi135.stdout:9/817: dread d7/d29/d78/da4/d37/f39 [0,4194304] 0 2024-03-30T04:28:35.654 INFO:tasks.workunit.client.1.smithi135.stdout:1/872: dread d16/d5a/d5f/d3d/d8d/dc9/da4/d26/feb [0,4194304] 0 2024-03-30T04:28:35.655 INFO:tasks.workunit.client.1.smithi135.stdout:6/953: dread d2/dd/d14/d2b/d86/d55/fb1 [0,4194304] 0 2024-03-30T04:28:35.656 INFO:tasks.workunit.client.1.smithi135.stdout:1/873: write d16/d29/d2c/d3c/f3b [180249,50492] 0 2024-03-30T04:28:35.660 INFO:tasks.workunit.client.1.smithi135.stdout:9/818: write d7/d4f/d5c/f6d [134196,117306] 0 2024-03-30T04:28:35.662 INFO:tasks.workunit.client.1.smithi135.stdout:9/819: readlink d7/d41/le4 0 2024-03-30T04:28:35.662 INFO:tasks.workunit.client.1.smithi135.stdout:1/874: dread d16/d29/d2c/d3c/f117 [0,4194304] 0 2024-03-30T04:28:35.665 INFO:tasks.workunit.client.1.smithi135.stdout:1/875: readlink d16/d5a/d5f/d3d/d8d/dc9/da4/l63 0 2024-03-30T04:28:35.667 INFO:tasks.workunit.client.1.smithi135.stdout:1/876: creat d16/d9f/f123 x:0 0 0 2024-03-30T04:28:35.669 INFO:tasks.workunit.client.1.smithi135.stdout:9/820: dread d7/d4f/d5c/f9e [0,4194304] 0 2024-03-30T04:28:35.670 INFO:tasks.workunit.client.1.smithi135.stdout:6/954: dread d2/dd/d14/d2b/d86/fd1 [0,4194304] 0 2024-03-30T04:28:35.672 INFO:tasks.workunit.client.1.smithi135.stdout:9/821: fsync d7/d4f/d5c/d63/f64 0 2024-03-30T04:28:35.673 INFO:tasks.workunit.client.1.smithi135.stdout:9/822: readlink d7/d41/le4 0 2024-03-30T04:28:35.673 INFO:tasks.workunit.client.1.smithi135.stdout:1/877: truncate d16/d5a/d5f/d3d/d8d/dc9/f1f 8839081 0 2024-03-30T04:28:35.674 INFO:tasks.workunit.client.1.smithi135.stdout:6/955: creat d2/d68/f137 x:0 0 0 2024-03-30T04:28:35.680 INFO:tasks.workunit.client.1.smithi135.stdout:1/878: getdents d16/d5a/d5f/d3d/d8d/dc9 0 2024-03-30T04:28:35.681 INFO:tasks.workunit.client.1.smithi135.stdout:6/956: rename d2/dd/d14/d30/fbc to d2/dd/d14/d30/d43/f138 0 2024-03-30T04:28:35.686 INFO:tasks.workunit.client.1.smithi135.stdout:6/957: dread d2/dd/d14/d30/d43/f78 [0,4194304] 0 2024-03-30T04:28:35.688 INFO:tasks.workunit.client.1.smithi135.stdout:9/823: dwrite d7/d29/d78/f9b [0,4194304] 0 2024-03-30T04:28:35.691 INFO:tasks.workunit.client.1.smithi135.stdout:6/958: mknod d2/dd/d14/d53/c139 0 2024-03-30T04:28:35.694 INFO:tasks.workunit.client.1.smithi135.stdout:6/959: truncate d2/dd/d14/d30/d43/f69 4999752 0 2024-03-30T04:28:35.698 INFO:tasks.workunit.client.1.smithi135.stdout:6/960: rmdir d2/d2e 39 2024-03-30T04:28:35.699 INFO:tasks.workunit.client.1.smithi135.stdout:9/824: link d7/l4b d7/d4f/d5c/d63/dd2/dfe/l108 0 2024-03-30T04:28:35.700 INFO:tasks.workunit.client.1.smithi135.stdout:6/961: chown d2/dd/d14/d38/dde 1172440289 1 2024-03-30T04:28:35.701 INFO:tasks.workunit.client.1.smithi135.stdout:9/825: dread - d7/d4f/d5c/d63/fd9 zero size 2024-03-30T04:28:35.704 INFO:tasks.workunit.client.1.smithi135.stdout:9/826: fdatasync d7/d4f/d5c/d63/dd2/db1/dbc/ff2 0 2024-03-30T04:28:35.707 INFO:tasks.workunit.client.1.smithi135.stdout:6/962: dread d2/dd/d14/d2b/d86/d8e/ff6 [0,4194304] 0 2024-03-30T04:28:35.708 INFO:tasks.workunit.client.1.smithi135.stdout:9/827: creat d7/d29/d78/f109 x:0 0 0 2024-03-30T04:28:35.709 INFO:tasks.workunit.client.1.smithi135.stdout:6/963: rmdir d2/d68/d81 39 2024-03-30T04:28:35.710 INFO:tasks.workunit.client.1.smithi135.stdout:9/828: creat d7/d4f/dee/f10a x:0 0 0 2024-03-30T04:28:35.714 INFO:tasks.workunit.client.1.smithi135.stdout:6/964: mknod d2/dd/d14/d30/c13a 0 2024-03-30T04:28:35.716 INFO:tasks.workunit.client.1.smithi135.stdout:6/965: write d2/dd/d14/f3a [1149664,55407] 0 2024-03-30T04:28:35.720 INFO:tasks.workunit.client.1.smithi135.stdout:6/966: truncate d2/d2e/f37 5798961 0 2024-03-30T04:28:35.723 INFO:tasks.workunit.client.1.smithi135.stdout:6/967: mknod d2/dd/d14/d30/d43/d125/c13b 0 2024-03-30T04:28:35.725 INFO:tasks.workunit.client.1.smithi135.stdout:9/829: dwrite d7/f4e [0,4194304] 0 2024-03-30T04:28:35.727 INFO:tasks.workunit.client.1.smithi135.stdout:6/968: creat d2/d68/f13c x:0 0 0 2024-03-30T04:28:35.729 INFO:tasks.workunit.client.1.smithi135.stdout:6/969: rmdir d2/dd/d14/d30/d43/dcd/dc5/dd2 39 2024-03-30T04:28:35.733 INFO:tasks.workunit.client.1.smithi135.stdout:9/830: link d7/d4f/d5c/ld5 d7/d4f/d5c/l10b 0 2024-03-30T04:28:35.736 INFO:tasks.workunit.client.1.smithi135.stdout:9/831: rename d7/d41/db4/df9 to d7/d4f/dee/d10c 0 2024-03-30T04:28:35.739 INFO:tasks.workunit.client.1.smithi135.stdout:6/970: write d2/dd/d14/d2b/d86/f80 [337414,94693] 0 2024-03-30T04:28:35.742 INFO:tasks.workunit.client.1.smithi135.stdout:6/971: creat d2/dd/d14/d38/f13d x:0 0 0 2024-03-30T04:28:35.757 INFO:tasks.workunit.client.1.smithi135.stdout:6/972: dwrite d2/dd/d14/d38/f3e [4194304,4194304] 0 2024-03-30T04:28:35.763 INFO:tasks.workunit.client.1.smithi135.stdout:6/973: rename d2/d2e to d2/dd/d40/db4/d13e 0 2024-03-30T04:28:35.765 INFO:tasks.workunit.client.1.smithi135.stdout:6/974: read d2/d68/d81/fb7 [20886,52142] 0 2024-03-30T04:28:35.768 INFO:tasks.workunit.client.1.smithi135.stdout:6/975: write d2/dd/d14/d38/d11d/f131 [794089,130458] 0 2024-03-30T04:28:35.770 INFO:tasks.workunit.client.1.smithi135.stdout:6/976: mknod d2/dd/d18/db0/d128/c13f 0 2024-03-30T04:28:35.773 INFO:tasks.workunit.client.1.smithi135.stdout:6/977: mknod d2/dd/d14/d30/d43/dcd/dbe/c140 0 2024-03-30T04:28:35.788 INFO:tasks.workunit.client.1.smithi135.stdout:6/978: dwrite d2/d68/f83 [0,4194304] 0 2024-03-30T04:28:35.818 INFO:tasks.workunit.client.1.smithi135.stdout:6/979: dwrite d2/dd/d14/d38/f3e [0,4194304] 0 2024-03-30T04:28:35.823 INFO:tasks.workunit.client.1.smithi135.stdout:6/980: chown d2/dd/d14/d2b/d86/d55/fb2 880 1 2024-03-30T04:28:35.826 INFO:tasks.workunit.client.1.smithi135.stdout:6/981: write d2/dd/d14/d2b/fc8 [2593625,57598] 0 2024-03-30T04:28:35.844 INFO:tasks.workunit.client.1.smithi135.stdout:6/982: truncate d2/dd/d14/fab 306963 0 2024-03-30T04:28:35.846 INFO:tasks.workunit.client.1.smithi135.stdout:6/983: chown d2/c71 125 1 2024-03-30T04:28:35.848 INFO:tasks.workunit.client.1.smithi135.stdout:6/984: chown d2/dd/d14/d2b/d86/d8e/f97 0 1 2024-03-30T04:28:35.852 INFO:tasks.workunit.client.1.smithi135.stdout:6/985: write d2/dd/d14/d2b/d86/fe5 [2541107,53758] 0 2024-03-30T04:28:35.858 INFO:tasks.workunit.client.1.smithi135.stdout:6/986: rename d2/c124 to d2/dd/d14/d38/dde/c141 0 2024-03-30T04:28:35.884 INFO:tasks.workunit.client.1.smithi135.stdout:6/987: write d2/dd/d14/d38/f7e [714552,8136] 0 2024-03-30T04:28:35.886 INFO:tasks.workunit.client.1.smithi135.stdout:6/988: rmdir d2/dd/d18 39 2024-03-30T04:28:35.894 INFO:tasks.workunit.client.1.smithi135.stdout:6/989: dread d2/dd/d14/d2b/d86/f80 [0,4194304] 0 2024-03-30T04:28:35.896 INFO:tasks.workunit.client.1.smithi135.stdout:6/990: rmdir d2/dd/d14/d38/dde 39 2024-03-30T04:28:35.900 INFO:tasks.workunit.client.1.smithi135.stdout:6/991: creat d2/dd/d14/d38/dde/f142 x:0 0 0 2024-03-30T04:28:35.903 INFO:tasks.workunit.client.1.smithi135.stdout:6/992: dread - d2/dd/d18/db0/d12c/f135 zero size 2024-03-30T04:28:35.926 INFO:tasks.workunit.client.1.smithi135.stdout:6/993: dread d2/dd/d14/d2b/f50 [0,4194304] 0 2024-03-30T04:28:35.929 INFO:tasks.workunit.client.1.smithi135.stdout:6/994: unlink d2/c4f 0 2024-03-30T04:28:35.934 INFO:tasks.workunit.client.1.smithi135.stdout:6/995: creat d2/f143 x:0 0 0 2024-03-30T04:28:35.937 INFO:tasks.workunit.client.1.smithi135.stdout:6/996: creat d2/dd/d14/d2b/d86/d8e/f144 x:0 0 0 2024-03-30T04:28:35.939 INFO:tasks.workunit.client.1.smithi135.stdout:6/997: fdatasync d2/dd/d40/db4/d13e/f31 0 2024-03-30T04:28:35.944 INFO:tasks.workunit.client.1.smithi135.stdout:6/998: creat d2/dd/d14/d30/d43/dcd/dc5/dd2/f145 x:0 0 0 2024-03-30T04:28:35.946 INFO:tasks.workunit.client.1.smithi135.stdout:6/999: creat d2/dd/d14/d30/d43/f146 x:0 0 0 2024-03-30T04:28:35.953 INFO:tasks.workunit.client.1.smithi135.stdout:2/774: sync 2024-03-30T04:28:35.953 INFO:tasks.workunit.client.1.smithi135.stdout:3/856: sync 2024-03-30T04:28:35.958 INFO:tasks.workunit.client.1.smithi135.stdout:3/857: rename d3/d10/d16/f43 to d3/d10/d16/d23/d6a/d86/d98/d99/f123 0 2024-03-30T04:28:35.960 INFO:tasks.workunit.client.1.smithi135.stdout:3/858: mkdir d3/d6e/dfc/d124 0 2024-03-30T04:28:35.964 INFO:tasks.workunit.client.1.smithi135.stdout:3/859: mknod d3/d10/d9c/dbe/dd6/c125 0 2024-03-30T04:28:35.967 INFO:tasks.workunit.client.1.smithi135.stdout:2/775: dwrite d4/da/d7a/f88 [0,4194304] 0 2024-03-30T04:28:35.970 INFO:tasks.workunit.client.1.smithi135.stdout:2/776: write d4/d32/d2e/d67/dad/fe1 [1071487,69353] 0 2024-03-30T04:28:35.988 INFO:tasks.workunit.client.1.smithi135.stdout:3/860: dwrite d3/d10/d16/d23/fcb [0,4194304] 0 2024-03-30T04:28:35.990 INFO:tasks.workunit.client.1.smithi135.stdout:3/861: chown d3/d10/d16/d23/f108 1 1 2024-03-30T04:28:35.991 INFO:tasks.workunit.client.1.smithi135.stdout:2/777: dwrite d4/d32/d12/d45/f39 [0,4194304] 0 2024-03-30T04:28:35.995 INFO:tasks.workunit.client.1.smithi135.stdout:2/778: mknod d4/d32/d12/d45/d6d/c103 0 2024-03-30T04:28:35.997 INFO:tasks.workunit.client.1.smithi135.stdout:2/779: stat d4/d6/d47/l9b 0 2024-03-30T04:28:35.997 INFO:tasks.workunit.client.1.smithi135.stdout:3/862: link d3/d10/d16/d23/db6/l10b d3/d6e/dfc/d124/l126 0 2024-03-30T04:28:35.998 INFO:tasks.workunit.client.1.smithi135.stdout:2/780: chown d4/da/c93 155473025 1 2024-03-30T04:28:36.004 INFO:tasks.workunit.client.1.smithi135.stdout:3/863: mkdir d3/d10/d16/d23/d6a/d94/dfa/d104/d127 0 2024-03-30T04:28:36.017 INFO:tasks.workunit.client.1.smithi135.stdout:3/864: link d3/d10/d16/d41/d83/cad d3/d10/d9c/dbe/d112/c128 0 2024-03-30T04:28:36.018 INFO:tasks.workunit.client.1.smithi135.stdout:2/781: dwrite d4/da/d7a/f88 [4194304,4194304] 0 2024-03-30T04:28:36.020 INFO:tasks.workunit.client.1.smithi135.stdout:2/782: fsync d4/d6/f1a 0 2024-03-30T04:28:36.040 INFO:tasks.workunit.client.1.smithi135.stdout:2/783: dwrite d4/d32/d12/f3f [0,4194304] 0 2024-03-30T04:28:36.042 INFO:tasks.workunit.client.1.smithi135.stdout:2/784: fdatasync d4/f1c 0 2024-03-30T04:28:36.046 INFO:tasks.workunit.client.1.smithi135.stdout:2/785: mknod d4/d32/d46/dbc/c104 0 2024-03-30T04:28:36.051 INFO:tasks.workunit.client.1.smithi135.stdout:2/786: link d4/d32/d2e/f53 d4/d32/d2e/d67/dad/f105 0 2024-03-30T04:28:36.053 INFO:tasks.workunit.client.1.smithi135.stdout:2/787: creat d4/da/d7a/f106 x:0 0 0 2024-03-30T04:28:36.079 INFO:tasks.workunit.client.1.smithi135.stdout:2/788: dread d4/d32/d46/f83 [0,4194304] 0 2024-03-30T04:28:36.081 INFO:tasks.workunit.client.1.smithi135.stdout:2/789: write d4/d32/f33 [90410,49428] 0 2024-03-30T04:28:36.082 INFO:tasks.workunit.client.1.smithi135.stdout:2/790: chown d4/d32/d43/db5 160544 1 2024-03-30T04:28:36.086 INFO:tasks.workunit.client.1.smithi135.stdout:2/791: creat d4/d32/d12/d45/dac/f107 x:0 0 0 2024-03-30T04:28:36.089 INFO:tasks.workunit.client.1.smithi135.stdout:2/792: mknod d4/d32/d2e/d67/dad/de5/d100/c108 0 2024-03-30T04:28:36.093 INFO:tasks.workunit.client.1.smithi135.stdout:2/793: rename d4/da/d8d/lce to d4/d32/d2e/d67/dad/de5/df1/l109 0 2024-03-30T04:28:36.097 INFO:tasks.workunit.client.1.smithi135.stdout:2/794: rename d4/d32/d12/d45/dac/dee to d4/d32/d12/d45/d6d/dea/d10a 0 2024-03-30T04:28:36.098 INFO:tasks.workunit.client.1.smithi135.stdout:2/795: dread - d4/da/d7a/feb zero size 2024-03-30T04:28:36.100 INFO:tasks.workunit.client.1.smithi135.stdout:2/796: creat d4/d32/d12/d45/d3c/f10b x:0 0 0 2024-03-30T04:28:36.109 INFO:tasks.workunit.client.1.smithi135.stdout:2/797: dread d4/da/f59 [0,4194304] 0 2024-03-30T04:28:36.111 INFO:tasks.workunit.client.1.smithi135.stdout:2/798: read d4/d6/f16 [7659367,110444] 0 2024-03-30T04:28:36.115 INFO:tasks.workunit.client.1.smithi135.stdout:2/799: truncate d4/da/d7a/dd6/fdf 822418 0 2024-03-30T04:28:36.116 INFO:tasks.workunit.client.1.smithi135.stdout:2/800: fdatasync d4/d6/f16 0 2024-03-30T04:28:36.118 INFO:tasks.workunit.client.1.smithi135.stdout:2/801: creat d4/d55/f10c x:0 0 0 2024-03-30T04:28:36.121 INFO:tasks.workunit.client.1.smithi135.stdout:2/802: unlink d4/d32/d12/d45/d7d/de7/f8a 0 2024-03-30T04:28:36.122 INFO:tasks.workunit.client.1.smithi135.stdout:8/819: sync 2024-03-30T04:28:36.123 INFO:tasks.workunit.client.1.smithi135.stdout:2/803: mknod d4/d32/d2e/d67/dad/de5/c10d 0 2024-03-30T04:28:36.126 INFO:tasks.workunit.client.1.smithi135.stdout:8/820: creat d6/d8/d15/d79/f10a x:0 0 0 2024-03-30T04:28:36.127 INFO:tasks.workunit.client.1.smithi135.stdout:2/804: creat d4/d6/f10e x:0 0 0 2024-03-30T04:28:36.128 INFO:tasks.workunit.client.1.smithi135.stdout:8/821: chown d6/d8/d15/d2a/dee/f29 930170 1 2024-03-30T04:28:36.138 INFO:tasks.workunit.client.1.smithi135.stdout:2/805: write d4/d32/d43/f63 [2962942,10692] 0 2024-03-30T04:28:36.140 INFO:tasks.workunit.client.1.smithi135.stdout:2/806: creat d4/d55/f10f x:0 0 0 2024-03-30T04:28:36.143 INFO:tasks.workunit.client.1.smithi135.stdout:2/807: read - d4/d6/f6f zero size 2024-03-30T04:28:36.145 INFO:tasks.workunit.client.1.smithi135.stdout:8/822: dwrite d6/d8/d15/d54/d8e/faa [0,4194304] 0 2024-03-30T04:28:36.154 INFO:tasks.workunit.client.1.smithi135.stdout:2/808: dread d4/da/fb [0,4194304] 0 2024-03-30T04:28:36.161 INFO:tasks.workunit.client.1.smithi135.stdout:2/809: link d4/d32/d12/d45/d3c/c66 d4/d32/d12/d45/d6d/c110 0 2024-03-30T04:28:36.168 INFO:tasks.workunit.client.1.smithi135.stdout:8/823: dread d6/d8/f17 [0,4194304] 0 2024-03-30T04:28:36.172 INFO:tasks.workunit.client.1.smithi135.stdout:8/824: symlink d6/d8/d15/d54/d7e/l10b 0 2024-03-30T04:28:36.176 INFO:tasks.workunit.client.1.smithi135.stdout:2/810: dwrite d4/d55/f10f [0,4194304] 0 2024-03-30T04:28:36.177 INFO:tasks.workunit.client.1.smithi135.stdout:8/825: mkdir d6/d8/d15/d54/d7e/daf/dfb/dfd/d10c 0 2024-03-30T04:28:36.183 INFO:tasks.workunit.client.1.smithi135.stdout:8/826: truncate d6/d8/d15/d54/d7e/da6/fb9 764175 0 2024-03-30T04:28:36.184 INFO:tasks.workunit.client.1.smithi135.stdout:8/827: readlink d6/d8/d15/d54/d7e/l10b 0 2024-03-30T04:28:36.185 INFO:tasks.workunit.client.1.smithi135.stdout:8/828: readlink d6/d8/d15/d2a/lb7 0 2024-03-30T04:28:36.191 INFO:tasks.workunit.client.1.smithi135.stdout:5/784: sync 2024-03-30T04:28:36.191 INFO:tasks.workunit.client.1.smithi135.stdout:4/847: sync 2024-03-30T04:28:36.191 INFO:tasks.workunit.client.1.smithi135.stdout:8/829: rename d6/d8/d15/l1a to d6/d8/d15/d54/d7e/daf/dc1/l10d 0 2024-03-30T04:28:36.195 INFO:tasks.workunit.client.1.smithi135.stdout:5/785: creat d1/d45/d52/da5/dac/ff2 x:0 0 0 2024-03-30T04:28:36.196 INFO:tasks.workunit.client.1.smithi135.stdout:8/830: write d6/d8/f31 [8455511,21842] 0 2024-03-30T04:28:36.197 INFO:tasks.workunit.client.1.smithi135.stdout:4/848: creat d10/d1a/d32/d40/d92/db9/dcf/df5/f119 x:0 0 0 2024-03-30T04:28:36.198 INFO:tasks.workunit.client.1.smithi135.stdout:8/831: dread - d6/d8/d15/d1f/ddb/fe6 zero size 2024-03-30T04:28:36.200 INFO:tasks.workunit.client.1.smithi135.stdout:4/849: mkdir d10/d1a/d42/d46/d11a 0 2024-03-30T04:28:36.211 INFO:tasks.workunit.client.1.smithi135.stdout:5/786: truncate d1/da/d17/d70/db2/d79/de1/f57 485042 0 2024-03-30T04:28:36.214 INFO:tasks.workunit.client.1.smithi135.stdout:5/787: unlink d1/f39 0 2024-03-30T04:28:36.218 INFO:tasks.workunit.client.1.smithi135.stdout:8/832: dwrite d6/d8/d15/d79/f10a [0,4194304] 0 2024-03-30T04:28:36.227 INFO:tasks.workunit.client.1.smithi135.stdout:8/833: fdatasync d6/d8/d15/d2a/fea 0 2024-03-30T04:28:36.229 INFO:tasks.workunit.client.1.smithi135.stdout:4/850: dread d10/d1a/d32/d40/f8e [0,4194304] 0 2024-03-30T04:28:36.230 INFO:tasks.workunit.client.1.smithi135.stdout:4/851: readlink d10/d1a/l2d 0 2024-03-30T04:28:36.240 INFO:tasks.workunit.client.1.smithi135.stdout:5/788: dwrite d1/d7/d1a/fcd [0,4194304] 0 2024-03-30T04:28:36.240 INFO:tasks.workunit.client.1.smithi135.stdout:4/852: write fa [1841638,110842] 0 2024-03-30T04:28:36.241 INFO:tasks.workunit.client.1.smithi135.stdout:4/853: chown d10/d72/d7c/dfb/dd4 67025502 1 2024-03-30T04:28:36.244 INFO:tasks.workunit.client.1.smithi135.stdout:5/789: truncate d1/d7/d11/fde 966083 0 2024-03-30T04:28:36.248 INFO:tasks.workunit.client.1.smithi135.stdout:5/790: creat d1/da/d17/d70/db2/ff3 x:0 0 0 2024-03-30T04:28:36.250 INFO:tasks.workunit.client.1.smithi135.stdout:4/854: write d10/d72/f7d [3128586,38758] 0 2024-03-30T04:28:36.254 INFO:tasks.workunit.client.1.smithi135.stdout:4/855: mknod d10/d4d/dc1/db0/c11b 0 2024-03-30T04:28:36.261 INFO:tasks.workunit.client.1.smithi135.stdout:4/856: getdents d10/d1a/d42/d46/def 0 2024-03-30T04:28:36.263 INFO:tasks.workunit.client.1.smithi135.stdout:5/791: dread d1/f6e [0,4194304] 0 2024-03-30T04:28:36.264 INFO:tasks.workunit.client.1.smithi135.stdout:4/857: creat d10/d4d/d9c/d107/f11c x:0 0 0 2024-03-30T04:28:36.268 INFO:tasks.workunit.client.1.smithi135.stdout:4/858: unlink d10/d1a/d32/d40/d92/db9/dcf/df5/fcd 0 2024-03-30T04:28:36.273 INFO:tasks.workunit.client.1.smithi135.stdout:8/834: dread d6/d8/f31 [4194304,4194304] 0 2024-03-30T04:28:36.279 INFO:tasks.workunit.client.1.smithi135.stdout:5/792: dwrite d1/d45/d52/f9e [0,4194304] 0 2024-03-30T04:28:36.279 INFO:tasks.workunit.client.1.smithi135.stdout:8/835: getdents d6/d8/d15/d54/d7e/daf/dfb/dfd 0 2024-03-30T04:28:36.284 INFO:tasks.workunit.client.1.smithi135.stdout:4/859: dwrite d10/d1a/d42/f5e [0,4194304] 0 2024-03-30T04:28:36.290 INFO:tasks.workunit.client.1.smithi135.stdout:4/860: write d10/f14 [262394,76600] 0 2024-03-30T04:28:36.295 INFO:tasks.workunit.client.1.smithi135.stdout:4/861: fsync d10/d72/d7c/dfb/f100 0 2024-03-30T04:28:36.299 INFO:tasks.workunit.client.1.smithi135.stdout:8/836: link d6/d8/d15/d2a/cac d6/d8/c10e 0 2024-03-30T04:28:36.304 INFO:tasks.workunit.client.1.smithi135.stdout:4/862: creat d10/d1a/d42/d66/dc7/f11d x:0 0 0 2024-03-30T04:28:36.307 INFO:tasks.workunit.client.1.smithi135.stdout:5/793: dwrite d1/d3/d2b/f5b [0,4194304] 0 2024-03-30T04:28:36.309 INFO:tasks.workunit.client.1.smithi135.stdout:8/837: creat d6/d8/d15/d54/d8e/d72/f10f x:0 0 0 2024-03-30T04:28:36.310 INFO:tasks.workunit.client.1.smithi135.stdout:8/838: chown d6/fcc 58522 1 2024-03-30T04:28:36.312 INFO:tasks.workunit.client.1.smithi135.stdout:3/865: sync 2024-03-30T04:28:36.312 INFO:tasks.workunit.client.1.smithi135.stdout:9/832: sync 2024-03-30T04:28:36.312 INFO:tasks.workunit.client.1.smithi135.stdout:1/879: sync 2024-03-30T04:28:36.316 INFO:tasks.workunit.client.1.smithi135.stdout:8/839: chown d6/d8/d15/d54/d7e/da6 5133585 1 2024-03-30T04:28:36.326 INFO:tasks.workunit.client.1.smithi135.stdout:3/866: read - d3/d10/d16/d23/d6a/f7e zero size 2024-03-30T04:28:36.329 INFO:tasks.workunit.client.1.smithi135.stdout:5/794: write d1/d7/d11/f8a [3485680,110981] 0 2024-03-30T04:28:36.334 INFO:tasks.workunit.client.1.smithi135.stdout:5/795: mknod d1/d3/d2b/d54/d5e/cf4 0 2024-03-30T04:28:36.338 INFO:tasks.workunit.client.1.smithi135.stdout:3/867: link d3/db/da9/dcc/lf8 d3/d10/d16/d23/d6a/d94/l129 0 2024-03-30T04:28:36.343 INFO:tasks.workunit.client.1.smithi135.stdout:4/863: dwrite d10/d1a/f118 [0,4194304] 0 2024-03-30T04:28:36.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:36 smithi135 ceph-mon[44364]: pgmap v51: 65 pgs: 65 active+clean; 3.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 84 MiB/s rd, 201 MiB/s wr, 492 op/s 2024-03-30T04:28:36.351 INFO:tasks.workunit.client.1.smithi135.stdout:3/868: creat d3/d10/d9c/dbe/dd6/f12a x:0 0 0 2024-03-30T04:28:36.353 INFO:tasks.workunit.client.1.smithi135.stdout:4/864: mkdir d10/d72/d7c/d11e 0 2024-03-30T04:28:36.354 INFO:tasks.workunit.client.1.smithi135.stdout:9/833: dwrite d7/f26 [4194304,4194304] 0 2024-03-30T04:28:36.356 INFO:tasks.workunit.client.1.smithi135.stdout:8/840: dread d6/d8/d15/fb4 [0,4194304] 0 2024-03-30T04:28:36.358 INFO:tasks.workunit.client.1.smithi135.stdout:3/869: symlink d3/d10/d16/d41/d83/l12b 0 2024-03-30T04:28:36.359 INFO:tasks.workunit.client.1.smithi135.stdout:3/870: chown d3/d6e/dfc 44170927 1 2024-03-30T04:28:36.361 INFO:tasks.workunit.client.1.smithi135.stdout:1/880: dwrite d16/d29/d61/fe8 [0,4194304] 0 2024-03-30T04:28:36.363 INFO:tasks.workunit.client.1.smithi135.stdout:8/841: truncate d6/f86 830589 0 2024-03-30T04:28:36.371 INFO:tasks.workunit.client.1.smithi135.stdout:5/796: dwrite d1/da/d17/d70/db2/d79/de1/fe2 [0,4194304] 0 2024-03-30T04:28:36.372 INFO:tasks.workunit.client.1.smithi135.stdout:9/834: rename d7/d41/d72/la5 to d7/d4f/d5c/d63/dd2/db1/dbc/dd8/l10d 0 2024-03-30T04:28:36.372 INFO:tasks.workunit.client.1.smithi135.stdout:2/811: sync 2024-03-30T04:28:36.376 INFO:tasks.workunit.client.1.smithi135.stdout:9/835: chown d7/d55/d8d/lcf 77507 1 2024-03-30T04:28:36.380 INFO:tasks.workunit.client.1.smithi135.stdout:9/836: write d7/d41/d72/f74 [1245931,71452] 0 2024-03-30T04:28:36.384 INFO:tasks.workunit.client.1.smithi135.stdout:3/871: dread d3/d10/d16/d23/d6a/d86/fd1 [0,4194304] 0 2024-03-30T04:28:36.390 INFO:tasks.workunit.client.1.smithi135.stdout:1/881: symlink d16/d5a/d5f/d3d/d8d/dc9/da4/l124 0 2024-03-30T04:28:36.396 INFO:tasks.workunit.client.1.smithi135.stdout:1/882: chown d16/d5a/d5f/d3d/d8d/dc9/da4/d99/d9a/d105 384079 1 2024-03-30T04:28:36.398 INFO:tasks.workunit.client.1.smithi135.stdout:9/837: rename d7/d4f/d5c/d63/fc4 to d7/d4f/d5c/d63/dd2/f10e 0 2024-03-30T04:28:36.400 INFO:tasks.workunit.client.1.smithi135.stdout:5/797: dread d1/d45/d52/f9e [0,4194304] 0 2024-03-30T04:28:36.406 INFO:tasks.workunit.client.1.smithi135.stdout:3/872: creat d3/d10/d16/d23/d6a/d94/dfa/f12c x:0 0 0 2024-03-30T04:28:36.409 INFO:tasks.workunit.client.1.smithi135.stdout:9/838: unlink d7/d29/d78/da4/f1c 0 2024-03-30T04:28:36.411 INFO:tasks.workunit.client.1.smithi135.stdout:5/798: symlink d1/d45/d52/da5/dac/lf5 0 2024-03-30T04:28:36.413 INFO:tasks.workunit.client.1.smithi135.stdout:4/865: dread d10/d4d/d89/f8b [0,4194304] 0 2024-03-30T04:28:36.414 INFO:tasks.workunit.client.1.smithi135.stdout:1/883: unlink d16/d5a/d5f/d3d/db4/ff1 0 2024-03-30T04:28:36.415 INFO:tasks.workunit.client.1.smithi135.stdout:2/812: write d4/d32/d2e/d2f/f64 [1079323,8910] 0 2024-03-30T04:28:36.420 INFO:tasks.workunit.client.1.smithi135.stdout:8/842: dwrite d6/d8/d15/d54/d7e/da6/fb9 [0,4194304] 0 2024-03-30T04:28:36.424 INFO:tasks.workunit.client.1.smithi135.stdout:8/843: truncate d6/d8/d15/d2a/dee/f4e 2557255 0 2024-03-30T04:28:36.428 INFO:tasks.workunit.client.1.smithi135.stdout:2/813: creat d4/d32/d46/dbc/dbe/f111 x:0 0 0 2024-03-30T04:28:36.430 INFO:tasks.workunit.client.1.smithi135.stdout:9/839: link d7/d90/ld7 d7/d55/d8d/l10f 0 2024-03-30T04:28:36.432 INFO:tasks.workunit.client.1.smithi135.stdout:3/873: getdents d3/db 0 2024-03-30T04:28:36.440 INFO:tasks.workunit.client.1.smithi135.stdout:4/866: write d10/d1a/f86 [652208,1666] 0 2024-03-30T04:28:36.441 INFO:tasks.workunit.client.1.smithi135.stdout:5/799: getdents d1/d3 0 2024-03-30T04:28:36.443 INFO:tasks.workunit.client.1.smithi135.stdout:8/844: creat d6/d8/d15/d54/d7e/daf/dfb/dfd/f110 x:0 0 0 2024-03-30T04:28:36.443 INFO:tasks.workunit.client.1.smithi135.stdout:1/884: dread d16/d29/d32/f38 [0,4194304] 0 2024-03-30T04:28:36.445 INFO:tasks.workunit.client.1.smithi135.stdout:4/867: chown d10/d1a/d2b/l108 2026 1 2024-03-30T04:28:36.456 INFO:tasks.workunit.client.1.smithi135.stdout:2/814: dwrite d4/d32/d46/f91 [0,4194304] 0 2024-03-30T04:28:36.458 INFO:tasks.workunit.client.1.smithi135.stdout:2/815: dread - d4/da/d7a/f7c zero size 2024-03-30T04:28:36.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:36 smithi123 ceph-mon[35245]: pgmap v51: 65 pgs: 65 active+clean; 3.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 84 MiB/s rd, 201 MiB/s wr, 492 op/s 2024-03-30T04:28:36.480 INFO:tasks.workunit.client.1.smithi135.stdout:9/840: write d7/d29/d78/fe0 [991414,63791] 0 2024-03-30T04:28:36.480 INFO:tasks.workunit.client.1.smithi135.stdout:3/874: creat d3/d10/d9c/dbe/dd6/daf/f12d x:0 0 0 2024-03-30T04:28:36.482 INFO:tasks.workunit.client.1.smithi135.stdout:1/885: mkdir d16/d5a/d5f/d3d/db4/d125 0 2024-03-30T04:28:36.483 INFO:tasks.workunit.client.1.smithi135.stdout:4/868: chown d10/d1a/d32/fae 428826 1 2024-03-30T04:28:36.485 INFO:tasks.workunit.client.1.smithi135.stdout:5/800: mknod d1/d3/d2b/d54/cf6 0 2024-03-30T04:28:36.485 INFO:tasks.workunit.client.1.smithi135.stdout:3/875: read d3/f11 [3696381,86498] 0 2024-03-30T04:28:36.492 INFO:tasks.workunit.client.1.smithi135.stdout:8/845: truncate d6/d8/d15/d54/f58 4416627 0 2024-03-30T04:28:36.492 INFO:tasks.workunit.client.1.smithi135.stdout:3/876: fsync d3/d50/d46/f58 0 2024-03-30T04:28:36.493 INFO:tasks.workunit.client.1.smithi135.stdout:4/869: creat d10/d4d/dc3/f11f x:0 0 0 2024-03-30T04:28:36.496 INFO:tasks.workunit.client.1.smithi135.stdout:5/801: dread d1/d7/f43 [0,4194304] 0 2024-03-30T04:28:36.498 INFO:tasks.workunit.client.1.smithi135.stdout:1/886: rename d16/d29/d32/dcc/f7c to d16/d29/d2c/d3c/f126 0 2024-03-30T04:28:36.509 INFO:tasks.workunit.client.1.smithi135.stdout:9/841: dwrite d7/d4f/d89/fc2 [0,4194304] 0 2024-03-30T04:28:36.517 INFO:tasks.workunit.client.1.smithi135.stdout:4/870: rename d10/d4d/d89/d9f/daa to d10/d4d/dc3/d120 0 2024-03-30T04:28:36.518 INFO:tasks.workunit.client.1.smithi135.stdout:9/842: mkdir d7/d4f/dee/d110 0 2024-03-30T04:28:36.528 INFO:tasks.workunit.client.1.smithi135.stdout:1/887: getdents d16/d9f 0 2024-03-30T04:28:36.531 INFO:tasks.workunit.client.1.smithi135.stdout:9/843: symlink d7/l111 0 2024-03-30T04:28:36.534 INFO:tasks.workunit.client.1.smithi135.stdout:8/846: dread d6/f50 [0,4194304] 0 2024-03-30T04:28:36.534 INFO:tasks.workunit.client.1.smithi135.stdout:3/877: write d3/fa6 [4034174,25980] 0 2024-03-30T04:28:36.542 INFO:tasks.workunit.client.1.smithi135.stdout:1/888: rename d16/d29/fa9 to d16/d29/d32/d9d/f127 0 2024-03-30T04:28:36.549 INFO:tasks.workunit.client.1.smithi135.stdout:4/871: dwrite d10/d4d/d89/f8f [0,4194304] 0 2024-03-30T04:28:36.550 INFO:tasks.workunit.client.1.smithi135.stdout:3/878: mkdir d3/d10/d9c/dbe/dd6/d91/d12e 0 2024-03-30T04:28:36.555 INFO:tasks.workunit.client.1.smithi135.stdout:8/847: mknod d6/d8/d15/d54/d7e/daf/c111 0 2024-03-30T04:28:36.557 INFO:tasks.workunit.client.1.smithi135.stdout:2/816: dwrite d4/d32/d2e/d67/dad/f105 [0,4194304] 0 2024-03-30T04:28:36.566 INFO:tasks.workunit.client.1.smithi135.stdout:3/879: creat d3/d10/d16/d23/f12f x:0 0 0 2024-03-30T04:28:36.566 INFO:tasks.workunit.client.1.smithi135.stdout:5/802: dwrite d1/d7/d1a/d1f/f66 [0,4194304] 0 2024-03-30T04:28:36.570 INFO:tasks.workunit.client.1.smithi135.stdout:4/872: dread - d10/d72/d7c/dfb/dd4/fe5 zero size 2024-03-30T04:28:36.579 INFO:tasks.workunit.client.1.smithi135.stdout:9/844: dwrite d7/d4f/d6e/dad/feb [0,4194304] 0 2024-03-30T04:28:36.581 INFO:tasks.workunit.client.1.smithi135.stdout:8/848: rename d6/d23/d9d/le2 to d6/d8/d15/d79/l112 0 2024-03-30T04:28:36.585 INFO:tasks.workunit.client.1.smithi135.stdout:5/803: mkdir d1/d3/df7 0 2024-03-30T04:28:36.588 INFO:tasks.workunit.client.1.smithi135.stdout:1/889: link d16/d5a/d5f/d3d/d8d/dc9/da4/d99/c41 d16/d9f/c128 0 2024-03-30T04:28:36.591 INFO:tasks.workunit.client.1.smithi135.stdout:9/845: rename d7/d41/f59 to d7/d4f/d5c/f112 0 2024-03-30T04:28:36.598 INFO:tasks.workunit.client.1.smithi135.stdout:4/873: creat d10/d1a/d42/d66/dc7/d106/f121 x:0 0 0 2024-03-30T04:28:36.598 INFO:tasks.workunit.client.1.smithi135.stdout:5/804: dread - d1/da/d17/d70/db2/d79/de1/fa9 zero size 2024-03-30T04:28:36.599 INFO:tasks.workunit.client.1.smithi135.stdout:1/890: dread - d16/d29/d2c/dce/fcf zero size 2024-03-30T04:28:36.600 INFO:tasks.workunit.client.1.smithi135.stdout:4/874: readlink d10/d1a/d32/l3b 0 2024-03-30T04:28:36.605 INFO:tasks.workunit.client.1.smithi135.stdout:8/849: dread d6/d8/d15/d54/d8e/faa [0,4194304] 0 2024-03-30T04:28:36.614 INFO:tasks.workunit.client.1.smithi135.stdout:8/850: creat d6/d8/d15/d54/d7e/daf/dfb/dfd/f113 x:0 0 0 2024-03-30T04:28:36.619 INFO:tasks.workunit.client.1.smithi135.stdout:8/851: write d6/d23/f82 [4385184,75165] 0 2024-03-30T04:28:36.624 INFO:tasks.workunit.client.1.smithi135.stdout:4/875: getdents d10/d1a/d42/d66 0 2024-03-30T04:28:36.631 INFO:tasks.workunit.client.1.smithi135.stdout:3/880: dwrite d3/d10/d16/fa5 [4194304,4194304] 0 2024-03-30T04:28:36.634 INFO:tasks.workunit.client.1.smithi135.stdout:1/891: dwrite d16/d29/d2c/d43/d50/f11d [0,4194304] 0 2024-03-30T04:28:36.638 INFO:tasks.workunit.client.1.smithi135.stdout:3/881: truncate d3/d54/f11f 542681 0 2024-03-30T04:28:36.640 INFO:tasks.workunit.client.1.smithi135.stdout:4/876: unlink d10/d1a/d32/d40/c8a 0 2024-03-30T04:28:36.642 INFO:tasks.workunit.client.1.smithi135.stdout:5/805: rename d1/d3/d76/c86 to d1/da/d17/d70/db2/dad/cf8 0 2024-03-30T04:28:36.642 INFO:tasks.workunit.client.1.smithi135.stdout:8/852: write d6/d8/fa5 [59912,20340] 0 2024-03-30T04:28:36.644 INFO:tasks.workunit.client.1.smithi135.stdout:9/846: dwrite d7/d29/d78/da4/f35 [0,4194304] 0 2024-03-30T04:28:36.647 INFO:tasks.workunit.client.1.smithi135.stdout:3/882: chown d3/d10/f14 57 1 2024-03-30T04:28:36.648 INFO:tasks.workunit.client.1.smithi135.stdout:5/806: read - d1/da/d17/d70/db2/d79/d8e/fd8 zero size 2024-03-30T04:28:36.649 INFO:tasks.workunit.client.1.smithi135.stdout:3/883: write d3/f1d [4031653,40744] 0 2024-03-30T04:28:36.654 INFO:tasks.workunit.client.1.smithi135.stdout:1/892: mkdir d16/d129 0 2024-03-30T04:28:36.664 INFO:tasks.workunit.client.1.smithi135.stdout:5/807: creat d1/d7/d11/ff9 x:0 0 0 2024-03-30T04:28:36.664 INFO:tasks.workunit.client.1.smithi135.stdout:4/877: creat d10/d1a/d42/d66/dd7/f122 x:0 0 0 2024-03-30T04:28:36.665 INFO:tasks.workunit.client.1.smithi135.stdout:4/878: chown d10/d4d/c50 159691 1 2024-03-30T04:28:36.667 INFO:tasks.workunit.client.1.smithi135.stdout:1/893: chown d16/d5a/d5f/d3d/db4/c111 790441 1 2024-03-30T04:28:36.670 INFO:tasks.workunit.client.1.smithi135.stdout:1/894: fdatasync d16/d5a/dad/d103/f10f 0 2024-03-30T04:28:36.670 INFO:tasks.workunit.client.1.smithi135.stdout:4/879: mknod d10/d4d/d89/c123 0 2024-03-30T04:28:36.677 INFO:tasks.workunit.client.1.smithi135.stdout:1/895: mknod d16/d29/d61/c12a 0 2024-03-30T04:28:36.682 INFO:tasks.workunit.client.1.smithi135.stdout:1/896: write d16/f2f [3200991,81944] 0 2024-03-30T04:28:36.686 INFO:tasks.workunit.client.1.smithi135.stdout:9/847: dwrite d7/d41/f85 [0,4194304] 0 2024-03-30T04:28:36.695 INFO:tasks.workunit.client.1.smithi135.stdout:3/884: dread d3/d10/d16/d23/d6a/d94/ff2 [0,4194304] 0 2024-03-30T04:28:36.698 INFO:tasks.workunit.client.1.smithi135.stdout:9/848: write d7/d29/d78/da4/d37/f39 [4095603,11570] 0 2024-03-30T04:28:36.704 INFO:tasks.workunit.client.1.smithi135.stdout:1/897: dread d16/d29/d2c/d3c/fc0 [0,4194304] 0 2024-03-30T04:28:36.704 INFO:tasks.workunit.client.1.smithi135.stdout:4/880: dwrite d10/d4d/d89/d9f/fbd [0,4194304] 0 2024-03-30T04:28:36.711 INFO:tasks.workunit.client.1.smithi135.stdout:1/898: rename d16/d5a/d5f/d3d/d8d/dc9/f2b to d16/d5a/dad/d103/f12b 0 2024-03-30T04:28:36.712 INFO:tasks.workunit.client.1.smithi135.stdout:4/881: creat d10/d1a/d42/d66/f124 x:0 0 0 2024-03-30T04:28:36.713 INFO:tasks.workunit.client.1.smithi135.stdout:4/882: stat d10/d1a 0 2024-03-30T04:28:36.715 INFO:tasks.workunit.client.1.smithi135.stdout:4/883: write d10/fe9 [1385682,14311] 0 2024-03-30T04:28:36.716 INFO:tasks.workunit.client.1.smithi135.stdout:9/849: truncate d7/d4f/d5c/d63/dd2/db1/dbc/f97 994912 0 2024-03-30T04:28:36.718 INFO:tasks.workunit.client.1.smithi135.stdout:3/885: truncate d3/f1d 3021319 0 2024-03-30T04:28:36.722 INFO:tasks.workunit.client.1.smithi135.stdout:5/808: dread d1/d7/d1a/d1f/d9b/de9/f9c [0,4194304] 0 2024-03-30T04:28:36.723 INFO:tasks.workunit.client.1.smithi135.stdout:1/899: unlink d16/d29/d2c/d3c/f78 0 2024-03-30T04:28:36.725 INFO:tasks.workunit.client.1.smithi135.stdout:5/809: read d1/d7/fb1 [2833682,101067] 0 2024-03-30T04:28:36.733 INFO:tasks.workunit.client.1.smithi135.stdout:1/900: mknod d16/d5a/dad/d103/c12c 0 2024-03-30T04:28:36.733 INFO:tasks.workunit.client.1.smithi135.stdout:4/884: truncate d10/d72/fd0 3804149 0 2024-03-30T04:28:36.734 INFO:tasks.workunit.client.1.smithi135.stdout:3/886: creat d3/d10/d16/d23/d6a/d86/d98/d99/df7/f130 x:0 0 0 2024-03-30T04:28:36.734 INFO:tasks.workunit.client.1.smithi135.stdout:4/885: stat d10/d1a/ceb 0 2024-03-30T04:28:36.740 INFO:tasks.workunit.client.1.smithi135.stdout:3/887: fdatasync d3/db/fe 0 2024-03-30T04:28:36.744 INFO:tasks.workunit.client.1.smithi135.stdout:9/850: link d7/d4f/d5c/d63/dd2/db1/dbc/fdf d7/d4f/dee/d110/f113 0 2024-03-30T04:28:36.745 INFO:tasks.workunit.client.1.smithi135.stdout:4/886: mkdir d10/d1a/d32/d40/d92/db9/dcf/d125 0 2024-03-30T04:28:36.746 INFO:tasks.workunit.client.1.smithi135.stdout:9/851: readlink d7/d4f/d5c/d63/l65 0 2024-03-30T04:28:36.751 INFO:tasks.workunit.client.1.smithi135.stdout:3/888: creat d3/d54/d5c/da1/f131 x:0 0 0 2024-03-30T04:28:36.755 INFO:tasks.workunit.client.1.smithi135.stdout:1/901: unlink d16/d5a/d5f/d3d/d8d/dc9/f56 0 2024-03-30T04:28:36.755 INFO:tasks.workunit.client.1.smithi135.stdout:3/889: truncate d3/d10/d16/d41/d83/f103 249316 0 2024-03-30T04:28:36.757 INFO:tasks.workunit.client.1.smithi135.stdout:4/887: unlink d10/d1a/d42/de0/c114 0 2024-03-30T04:28:36.759 INFO:tasks.workunit.client.1.smithi135.stdout:9/852: symlink d7/d4f/dee/l114 0 2024-03-30T04:28:36.761 INFO:tasks.workunit.client.1.smithi135.stdout:1/902: mkdir d16/d29/d32/d12d 0 2024-03-30T04:28:36.768 INFO:tasks.workunit.client.1.smithi135.stdout:3/890: mknod d3/d10/d16/c132 0 2024-03-30T04:28:36.771 INFO:tasks.workunit.client.1.smithi135.stdout:5/810: write d1/d3/d2b/d54/f71 [4611005,48981] 0 2024-03-30T04:28:36.773 INFO:tasks.workunit.client.1.smithi135.stdout:4/888: dread d10/d1a/d42/d46/f9d [0,4194304] 0 2024-03-30T04:28:36.776 INFO:tasks.workunit.client.1.smithi135.stdout:1/903: mkdir d16/d5a/d5f/d3d/d8d/dc9/da4/d26/d12e 0 2024-03-30T04:28:36.776 INFO:tasks.workunit.client.1.smithi135.stdout:3/891: mkdir d3/d10/d16/d41/d83/d133 0 2024-03-30T04:28:36.779 INFO:tasks.workunit.client.1.smithi135.stdout:5/811: rename d1/da/d4d/ldd to d1/d7/d1a/d1f/lfa 0 2024-03-30T04:28:36.781 INFO:tasks.workunit.client.1.smithi135.stdout:9/853: dread d7/d4f/d89/f8a [0,4194304] 0 2024-03-30T04:28:36.782 INFO:tasks.workunit.client.1.smithi135.stdout:4/889: rename d10/d1a/l25 to d10/d1a/d2b/l126 0 2024-03-30T04:28:36.783 INFO:tasks.workunit.client.1.smithi135.stdout:3/892: rename d3/d10 to d3/d10/d9c/dbe/dd6/daf/de4/d134 22 2024-03-30T04:28:36.785 INFO:tasks.workunit.client.1.smithi135.stdout:9/854: fsync d7/d29/d78/da4/d37/dba/fda 0 2024-03-30T04:28:36.803 INFO:tasks.workunit.client.1.smithi135.stdout:9/855: creat d7/d4f/d6e/dad/dd4/f115 x:0 0 0 2024-03-30T04:28:36.803 INFO:tasks.workunit.client.1.smithi135.stdout:3/893: read d3/d10/d16/d23/d6a/f9e [430348,130384] 0 2024-03-30T04:28:36.806 INFO:tasks.workunit.client.1.smithi135.stdout:1/904: rename d16/d5a/d5f/d3a/dc6/l113 to d16/d5a/d5f/d3d/d8d/dc9/da4/d26/l12f 0 2024-03-30T04:28:36.810 INFO:tasks.workunit.client.1.smithi135.stdout:4/890: dwrite d10/d72/fde [4194304,4194304] 0 2024-03-30T04:28:36.812 INFO:tasks.workunit.client.1.smithi135.stdout:1/905: chown d16/d5a/d5f/d3a/f10e 530717663 1 2024-03-30T04:28:36.817 INFO:tasks.workunit.client.1.smithi135.stdout:9/856: unlink d7/d4f/d5c/d63/dd2/db1/dbc/fdf 0 2024-03-30T04:28:36.825 INFO:tasks.workunit.client.1.smithi135.stdout:4/891: symlink d10/d1a/d32/d40/d92/db9/dcf/d112/l127 0 2024-03-30T04:28:36.826 INFO:tasks.workunit.client.1.smithi135.stdout:9/857: mknod d7/d4f/d6e/c116 0 2024-03-30T04:28:36.828 INFO:tasks.workunit.client.1.smithi135.stdout:1/906: truncate d16/d5a/d5f/d3d/d8d/dc9/d100/f11e 778370 0 2024-03-30T04:28:36.833 INFO:tasks.workunit.client.1.smithi135.stdout:9/858: read d7/d29/d78/da4/d37/f3a [875258,62135] 0 2024-03-30T04:28:36.835 INFO:tasks.workunit.client.1.smithi135.stdout:5/812: dwrite d1/d7/fb1 [0,4194304] 0 2024-03-30T04:28:36.835 INFO:tasks.workunit.client.1.smithi135.stdout:4/892: rmdir d10/d1a/d42/d66 39 2024-03-30T04:28:36.840 INFO:tasks.workunit.client.1.smithi135.stdout:4/893: write d10/f74 [2147329,26155] 0 2024-03-30T04:28:36.841 INFO:tasks.workunit.client.1.smithi135.stdout:9/859: dread - d7/d29/fae zero size 2024-03-30T04:28:36.842 INFO:tasks.workunit.client.1.smithi135.stdout:5/813: chown d1/da/d17/d70/db2/d79/de1/c60 53510 1 2024-03-30T04:28:36.864 INFO:tasks.workunit.client.1.smithi135.stdout:4/894: dwrite d10/d72/d7c/fa2 [0,4194304] 0 2024-03-30T04:28:36.869 INFO:tasks.workunit.client.1.smithi135.stdout:4/895: chown d10/d4d/d89/cf8 2 1 2024-03-30T04:28:36.875 INFO:tasks.workunit.client.1.smithi135.stdout:3/894: dread d3/d50/d22/fc0 [0,4194304] 0 2024-03-30T04:28:36.881 INFO:tasks.workunit.client.1.smithi135.stdout:9/860: symlink d7/d55/d8d/ddb/de7/l117 0 2024-03-30T04:28:36.886 INFO:tasks.workunit.client.1.smithi135.stdout:5/814: dwrite d1/d3/d80/f93 [0,4194304] 0 2024-03-30T04:28:36.887 INFO:tasks.workunit.client.1.smithi135.stdout:3/895: read d3/d10/ffe [2770763,89677] 0 2024-03-30T04:28:36.889 INFO:tasks.workunit.client.1.smithi135.stdout:4/896: creat d10/d1a/d32/d40/f128 x:0 0 0 2024-03-30T04:28:36.893 INFO:tasks.workunit.client.1.smithi135.stdout:3/896: fsync d3/d10/d16/d23/fcb 0 2024-03-30T04:28:36.897 INFO:tasks.workunit.client.1.smithi135.stdout:3/897: chown d3/d50/d46/f21 1640231967 1 2024-03-30T04:28:36.907 INFO:tasks.workunit.client.1.smithi135.stdout:3/898: unlink d3/d10/d9c/dbe/d112/c128 0 2024-03-30T04:28:36.911 INFO:tasks.workunit.client.1.smithi135.stdout:1/907: dread d16/d5a/d5f/d3d/d8d/dc9/da4/f8b [0,4194304] 0 2024-03-30T04:28:36.914 INFO:tasks.workunit.client.1.smithi135.stdout:4/897: link d10/d1a/d32/d40/d92/lac d10/d1a/d32/d40/d92/l129 0 2024-03-30T04:28:36.925 INFO:tasks.workunit.client.1.smithi135.stdout:1/908: mknod d16/d5a/d5f/d3a/dc6/c130 0 2024-03-30T04:28:36.929 INFO:tasks.workunit.client.1.smithi135.stdout:3/899: dwrite d3/d50/f25 [0,4194304] 0 2024-03-30T04:28:36.938 INFO:tasks.workunit.client.1.smithi135.stdout:4/898: link d10/d72/d7c/dfb/ce2 d10/d1a/d32/c12a 0 2024-03-30T04:28:36.942 INFO:tasks.workunit.client.1.smithi135.stdout:5/815: dwrite d1/d3/d2b/f2f [0,4194304] 0 2024-03-30T04:28:36.946 INFO:tasks.workunit.client.1.smithi135.stdout:1/909: getdents d16/d5a/d5f 0 2024-03-30T04:28:36.955 INFO:tasks.workunit.client.1.smithi135.stdout:3/900: dwrite d3/d10/d16/d41/f10a [0,4194304] 0 2024-03-30T04:28:36.958 INFO:tasks.workunit.client.1.smithi135.stdout:3/901: chown d3/d50/d22/d33/d7b/c82 5934 1 2024-03-30T04:28:36.960 INFO:tasks.workunit.client.1.smithi135.stdout:4/899: rename d10/f57 to d10/d1a/d32/d40/d92/db9/dcf/df5/f12b 0 2024-03-30T04:28:36.962 INFO:tasks.workunit.client.1.smithi135.stdout:4/900: write d10/d1a/d42/d46/f82 [1258123,44970] 0 2024-03-30T04:28:36.969 INFO:tasks.workunit.client.1.smithi135.stdout:5/816: rename d1/d3/d2b/d54/d5e/cf4 to d1/d7/d1a/d1f/cfb 0 2024-03-30T04:28:36.975 INFO:tasks.workunit.client.1.smithi135.stdout:9/861: dread d7/d29/d78/da4/f53 [0,4194304] 0 2024-03-30T04:28:36.980 INFO:tasks.workunit.client.1.smithi135.stdout:1/910: rename d16/d5a/d5f/d3d/d8d/dc9/da4 to d16/d29/d2c/dce/d131 0 2024-03-30T04:28:36.984 INFO:tasks.workunit.client.1.smithi135.stdout:9/862: mkdir d7/df8/d118 0 2024-03-30T04:28:36.986 INFO:tasks.workunit.client.1.smithi135.stdout:9/863: chown d7/d4f/dee 4405 1 2024-03-30T04:28:36.999 INFO:tasks.workunit.client.1.smithi135.stdout:4/901: dwrite d10/d72/d7c/f9b [0,4194304] 0 2024-03-30T04:28:36.999 INFO:tasks.workunit.client.1.smithi135.stdout:3/902: dwrite d3/d10/d16/fed [0,4194304] 0 2024-03-30T04:28:37.002 INFO:tasks.workunit.client.1.smithi135.stdout:9/864: dread d7/d29/d78/da4/f53 [0,4194304] 0 2024-03-30T04:28:37.003 INFO:tasks.workunit.client.1.smithi135.stdout:4/902: fdatasync d10/d72/fde 0 2024-03-30T04:28:37.012 INFO:tasks.workunit.client.1.smithi135.stdout:3/903: creat d3/d10/d9b/f135 x:0 0 0 2024-03-30T04:28:37.015 INFO:tasks.workunit.client.1.smithi135.stdout:4/903: write d10/d1a/d42/d66/dc7/fce [1122820,10962] 0 2024-03-30T04:28:37.020 INFO:tasks.workunit.client.1.smithi135.stdout:4/904: mkdir d10/d4d/dc3/d120/d12c 0 2024-03-30T04:28:37.021 INFO:tasks.workunit.client.1.smithi135.stdout:3/904: symlink d3/d10/d9c/dbe/dd6/daf/de4/l136 0 2024-03-30T04:28:37.023 INFO:tasks.workunit.client.1.smithi135.stdout:4/905: read d10/f74 [2576607,4441] 0 2024-03-30T04:28:37.023 INFO:tasks.workunit.client.1.smithi135.stdout:3/905: chown d3/d10/d9c/dbe/dd6/daf/de4/l136 221115 1 2024-03-30T04:28:37.027 INFO:tasks.workunit.client.1.smithi135.stdout:4/906: fsync d10/d1a/d32/fa5 0 2024-03-30T04:28:37.029 INFO:tasks.workunit.client.1.smithi135.stdout:5/817: dread d1/d45/d52/da5/dac/fc2 [0,4194304] 0 2024-03-30T04:28:37.033 INFO:tasks.workunit.client.1.smithi135.stdout:5/818: dread - d1/d3/d76/fee zero size 2024-03-30T04:28:37.033 INFO:tasks.workunit.client.1.smithi135.stdout:1/911: dread d16/d5a/d5f/d3d/db4/fd0 [0,4194304] 0 2024-03-30T04:28:37.034 INFO:tasks.workunit.client.1.smithi135.stdout:2/817: sync 2024-03-30T04:28:37.038 INFO:tasks.workunit.client.1.smithi135.stdout:1/912: mknod d16/d5a/d5f/d3d/d8d/c132 0 2024-03-30T04:28:37.040 INFO:tasks.workunit.client.1.smithi135.stdout:2/818: symlink d4/d32/dc2/l112 0 2024-03-30T04:28:37.040 INFO:tasks.workunit.client.1.smithi135.stdout:4/907: truncate d10/d72/d7c/f9b 1079190 0 2024-03-30T04:28:37.046 INFO:tasks.workunit.client.1.smithi135.stdout:5/819: link d1/da/d17/d70/db2/f65 d1/da/d17/ffc 0 2024-03-30T04:28:37.048 INFO:tasks.workunit.client.1.smithi135.stdout:2/819: chown d4/da/fb 0 1 2024-03-30T04:28:37.049 INFO:tasks.workunit.client.1.smithi135.stdout:1/913: mknod d16/d5a/d5f/d3a/c133 0 2024-03-30T04:28:37.050 INFO:tasks.workunit.client.1.smithi135.stdout:3/906: dwrite d3/d10/d16/d23/f12f [0,4194304] 0 2024-03-30T04:28:37.056 INFO:tasks.workunit.client.1.smithi135.stdout:2/820: creat d4/d32/d2e/d9f/f113 x:0 0 0 2024-03-30T04:28:37.058 INFO:tasks.workunit.client.1.smithi135.stdout:9/865: dread d7/d4f/d6e/f87 [0,4194304] 0 2024-03-30T04:28:37.058 INFO:tasks.workunit.client.1.smithi135.stdout:1/914: write d16/d5a/dad/d77/fb6 [891176,2346] 0 2024-03-30T04:28:37.061 INFO:tasks.workunit.client.1.smithi135.stdout:2/821: chown d4/d32/d46/dbc/dbe/f111 8126 1 2024-03-30T04:28:37.064 INFO:tasks.workunit.client.1.smithi135.stdout:2/822: write d4/d32/f13 [2717517,116738] 0 2024-03-30T04:28:37.068 INFO:tasks.workunit.client.1.smithi135.stdout:4/908: dread d10/d4d/d89/f8b [0,4194304] 0 2024-03-30T04:28:37.069 INFO:tasks.workunit.client.1.smithi135.stdout:9/866: truncate d7/d29/d78/f9b 4815651 0 2024-03-30T04:28:37.070 INFO:tasks.workunit.client.1.smithi135.stdout:3/907: write d3/d50/d46/d4c/fb1 [63884,55464] 0 2024-03-30T04:28:37.074 INFO:tasks.workunit.client.1.smithi135.stdout:2/823: rename d4/d32/d46/dbc/c104 to d4/d55/d77/c114 0 2024-03-30T04:28:37.082 INFO:tasks.workunit.client.1.smithi135.stdout:9/867: creat d7/d4f/d5c/d63/dd2/db1/f119 x:0 0 0 2024-03-30T04:28:37.085 INFO:tasks.workunit.client.1.smithi135.stdout:5/820: dwrite d1/d7/d1a/d1f/d9b/fb8 [0,4194304] 0 2024-03-30T04:28:37.089 INFO:tasks.workunit.client.1.smithi135.stdout:9/868: chown d7/d29/d78/da4/l33 101634 1 2024-03-30T04:28:37.093 INFO:tasks.workunit.client.1.smithi135.stdout:5/821: mknod d1/cfd 0 2024-03-30T04:28:37.095 INFO:tasks.workunit.client.1.smithi135.stdout:1/915: dwrite d16/d29/d2c/d43/d50/fe4 [0,4194304] 0 2024-03-30T04:28:37.100 INFO:tasks.workunit.client.1.smithi135.stdout:9/869: creat d7/d4f/d5c/f11a x:0 0 0 2024-03-30T04:28:37.101 INFO:tasks.workunit.client.1.smithi135.stdout:5/822: fdatasync d1/d45/d52/d8f/fd2 0 2024-03-30T04:28:37.108 INFO:tasks.workunit.client.1.smithi135.stdout:4/909: dwrite d10/f74 [8388608,4194304] 0 2024-03-30T04:28:37.108 INFO:tasks.workunit.client.1.smithi135.stdout:3/908: dwrite d3/d50/d22/fc0 [0,4194304] 0 2024-03-30T04:28:37.111 INFO:tasks.workunit.client.1.smithi135.stdout:9/870: stat d7/d4f/d5c/d63/cfc 0 2024-03-30T04:28:37.113 INFO:tasks.workunit.client.1.smithi135.stdout:4/910: fsync d10/d1a/d32/d40/d92/db9/dcf/f110 0 2024-03-30T04:28:37.118 INFO:tasks.workunit.client.1.smithi135.stdout:9/871: fdatasync d7/f12 0 2024-03-30T04:28:37.118 INFO:tasks.workunit.client.1.smithi135.stdout:2/824: dwrite d4/d32/d2e/d2f/f38 [4194304,4194304] 0 2024-03-30T04:28:37.123 INFO:tasks.workunit.client.1.smithi135.stdout:8/853: sync 2024-03-30T04:28:37.133 INFO:tasks.workunit.client.1.smithi135.stdout:2/825: mkdir d4/d32/d2e/d67/dad/de5/d100/d115 0 2024-03-30T04:28:37.133 INFO:tasks.workunit.client.1.smithi135.stdout:3/909: link d3/d50/d22/d33/d7b/lbc d3/d10/d9c/l137 0 2024-03-30T04:28:37.134 INFO:tasks.workunit.client.1.smithi135.stdout:4/911: creat d10/d4d/d9c/d101/f12d x:0 0 0 2024-03-30T04:28:37.142 INFO:tasks.workunit.client.1.smithi135.stdout:1/916: dwrite d16/d29/d2c/d3c/f112 [0,4194304] 0 2024-03-30T04:28:37.143 INFO:tasks.workunit.client.1.smithi135.stdout:3/910: symlink d3/d109/l138 0 2024-03-30T04:28:37.144 INFO:tasks.workunit.client.1.smithi135.stdout:3/911: readlink d3/l8 0 2024-03-30T04:28:37.147 INFO:tasks.workunit.client.1.smithi135.stdout:1/917: write d16/d29/f35 [2440997,112943] 0 2024-03-30T04:28:37.149 INFO:tasks.workunit.client.1.smithi135.stdout:2/826: link d4/d32/d46/fba d4/d32/f116 0 2024-03-30T04:28:37.160 INFO:tasks.workunit.client.1.smithi135.stdout:3/912: symlink d3/d50/d22/d74/l139 0 2024-03-30T04:28:37.160 INFO:tasks.workunit.client.1.smithi135.stdout:8/854: dwrite d6/f5b [0,4194304] 0 2024-03-30T04:28:37.166 INFO:tasks.workunit.client.1.smithi135.stdout:1/918: rmdir d16/d29/d2c/d43/d50 39 2024-03-30T04:28:37.168 INFO:tasks.workunit.client.1.smithi135.stdout:8/855: write d6/d8/d15/d79/f10a [818394,95011] 0 2024-03-30T04:28:37.169 INFO:tasks.workunit.client.1.smithi135.stdout:5/823: dread d1/d45/d52/f90 [0,4194304] 0 2024-03-30T04:28:37.172 INFO:tasks.workunit.client.1.smithi135.stdout:3/913: mknod d3/d10/d16/d41/c13a 0 2024-03-30T04:28:37.175 INFO:tasks.workunit.client.1.smithi135.stdout:9/872: dread d7/f26 [0,4194304] 0 2024-03-30T04:28:37.180 INFO:tasks.workunit.client.1.smithi135.stdout:4/912: dwrite d10/d1a/d42/d66/dc7/d106/f121 [0,4194304] 0 2024-03-30T04:28:37.182 INFO:tasks.workunit.client.1.smithi135.stdout:9/873: write d7/d55/d8d/faf [4381091,18608] 0 2024-03-30T04:28:37.187 INFO:tasks.workunit.client.1.smithi135.stdout:8/856: creat d6/d8/d15/d54/d4f/f114 x:0 0 0 2024-03-30T04:28:37.188 INFO:tasks.workunit.client.1.smithi135.stdout:5/824: symlink d1/da/lfe 0 2024-03-30T04:28:37.193 INFO:tasks.workunit.client.1.smithi135.stdout:1/919: rmdir d16/d29/d2c/dce/d131/d26/d12e 0 2024-03-30T04:28:37.198 INFO:tasks.workunit.client.1.smithi135.stdout:8/857: fdatasync d6/d8/d15/d54/d7e/da6/fd3 0 2024-03-30T04:28:37.203 INFO:tasks.workunit.client.1.smithi135.stdout:8/858: stat d6/d8/fa5 0 2024-03-30T04:28:37.203 INFO:tasks.workunit.client.1.smithi135.stdout:2/827: dread d4/da/f44 [0,4194304] 0 2024-03-30T04:28:37.216 INFO:tasks.workunit.client.1.smithi135.stdout:1/920: write fb [2289365,102434] 0 2024-03-30T04:28:37.216 INFO:tasks.workunit.client.1.smithi135.stdout:3/914: dwrite d3/d10/d16/d23/db6/fe2 [0,4194304] 0 2024-03-30T04:28:37.226 INFO:tasks.workunit.client.1.smithi135.stdout:9/874: dwrite d7/d4f/d5c/d63/dd2/f10e [0,4194304] 0 2024-03-30T04:28:37.227 INFO:tasks.workunit.client.1.smithi135.stdout:8/859: getdents d6/d8/d15/d54/d8e/dbd 0 2024-03-30T04:28:37.234 INFO:tasks.workunit.client.1.smithi135.stdout:1/921: write d16/d5a/dad/d77/fe9 [4457000,36811] 0 2024-03-30T04:28:37.235 INFO:tasks.workunit.client.1.smithi135.stdout:5/825: dwrite d1/d45/d52/d8f/fd2 [0,4194304] 0 2024-03-30T04:28:37.235 INFO:tasks.workunit.client.1.smithi135.stdout:4/913: dwrite d10/d1a/d42/d66/dd7/f122 [0,4194304] 0 2024-03-30T04:28:37.243 INFO:tasks.workunit.client.1.smithi135.stdout:1/922: fdatasync d16/d5a/dad/d77/f118 0 2024-03-30T04:28:37.245 INFO:tasks.workunit.client.1.smithi135.stdout:8/860: fsync d6/fcc 0 2024-03-30T04:28:37.247 INFO:tasks.workunit.client.1.smithi135.stdout:2/828: dwrite d4/da/f59 [0,4194304] 0 2024-03-30T04:28:37.253 INFO:tasks.workunit.client.1.smithi135.stdout:1/923: mkdir d16/d5a/d134 0 2024-03-30T04:28:37.254 INFO:tasks.workunit.client.1.smithi135.stdout:8/861: mknod d6/d8/d15/d54/d7e/c115 0 2024-03-30T04:28:37.261 INFO:tasks.workunit.client.1.smithi135.stdout:9/875: write d7/d4f/d5c/d63/f81 [578518,107512] 0 2024-03-30T04:28:37.263 INFO:tasks.workunit.client.1.smithi135.stdout:8/862: creat d6/d8/d15/d54/d7e/daf/f116 x:0 0 0 2024-03-30T04:28:37.272 INFO:tasks.workunit.client.1.smithi135.stdout:5/826: dwrite d1/d45/f6b [0,4194304] 0 2024-03-30T04:28:37.276 INFO:tasks.workunit.client.1.smithi135.stdout:5/827: fdatasync d1/d3/d2b/fa8 0 2024-03-30T04:28:37.280 INFO:tasks.workunit.client.1.smithi135.stdout:5/828: rmdir d1/d45/d52 39 2024-03-30T04:28:37.283 INFO:tasks.workunit.client.1.smithi135.stdout:2/829: truncate d4/da/f44 4083752 0 2024-03-30T04:28:37.285 INFO:tasks.workunit.client.1.smithi135.stdout:1/924: truncate d16/d29/d2c/d3c/f112 341316 0 2024-03-30T04:28:37.287 INFO:tasks.workunit.client.1.smithi135.stdout:3/915: dread d3/d54/dab/fd3 [0,4194304] 0 2024-03-30T04:28:37.290 INFO:tasks.workunit.client.1.smithi135.stdout:2/830: write d4/d32/d12/d45/dac/f107 [718371,38997] 0 2024-03-30T04:28:37.292 INFO:tasks.workunit.client.1.smithi135.stdout:1/925: read - d16/d29/d2c/dce/fcf zero size 2024-03-30T04:28:37.293 INFO:tasks.workunit.client.1.smithi135.stdout:5/829: mknod d1/d7/d1a/d1f/d9b/de9/cff 0 2024-03-30T04:28:37.294 INFO:tasks.workunit.client.1.smithi135.stdout:9/876: dwrite d7/d4f/d6e/dad/feb [4194304,4194304] 0 2024-03-30T04:28:37.296 INFO:tasks.workunit.client.1.smithi135.stdout:3/916: fsync d3/db/da9/dcc/f116 0 2024-03-30T04:28:37.301 INFO:tasks.workunit.client.1.smithi135.stdout:8/863: dwrite d6/d8/d15/d54/f58 [4194304,4194304] 0 2024-03-30T04:28:37.304 INFO:tasks.workunit.client.1.smithi135.stdout:2/831: truncate d4/d32/d2e/d67/dad/fe1 2054251 0 2024-03-30T04:28:37.310 INFO:tasks.workunit.client.1.smithi135.stdout:3/917: creat d3/d109/f13b x:0 0 0 2024-03-30T04:28:37.314 INFO:tasks.workunit.client.1.smithi135.stdout:2/832: creat d4/d32/d12/d45/d7d/de7/f117 x:0 0 0 2024-03-30T04:28:37.314 INFO:tasks.workunit.client.1.smithi135.stdout:9/877: link d7/d4f/dee/f101 d7/d4f/d5c/d63/dd2/f11b 0 2024-03-30T04:28:37.320 INFO:tasks.workunit.client.1.smithi135.stdout:5/830: dread d1/d45/d52/f90 [0,4194304] 0 2024-03-30T04:28:37.320 INFO:tasks.workunit.client.1.smithi135.stdout:2/833: dread - d4/d32/d46/dbc/dbe/f111 zero size 2024-03-30T04:28:37.323 INFO:tasks.workunit.client.1.smithi135.stdout:1/926: write d16/d5a/dad/d103/f12b [278540,87928] 0 2024-03-30T04:28:37.334 INFO:tasks.workunit.client.1.smithi135.stdout:3/918: rename d3/d10/d9c/dbe/dd6/c125 to d3/d10/d9c/dbe/c13c 0 2024-03-30T04:28:37.335 INFO:tasks.workunit.client.1.smithi135.stdout:2/834: truncate d4/d32/d2e/f8c 3838227 0 2024-03-30T04:28:37.337 INFO:tasks.workunit.client.1.smithi135.stdout:8/864: dwrite d6/d8/d15/d54/d7e/daf/dc1/fe1 [4194304,4194304] 0 2024-03-30T04:28:37.338 INFO:tasks.workunit.client.1.smithi135.stdout:1/927: rename d16/d5a/d5f/d3d/d8d/dc9/d100/f10c to d16/d5a/d5f/d3d/df7/f135 0 2024-03-30T04:28:37.339 INFO:tasks.workunit.client.1.smithi135.stdout:8/865: stat d6/d8/d15/d2a/d35/c3a 0 2024-03-30T04:28:37.341 INFO:tasks.workunit.client.1.smithi135.stdout:9/878: write d7/f11 [4445464,117295] 0 2024-03-30T04:28:37.343 INFO:tasks.workunit.client.1.smithi135.stdout:8/866: dread - d6/d8/d15/d54/d7e/daf/dfb/dfd/f113 zero size 2024-03-30T04:28:37.352 INFO:tasks.workunit.client.1.smithi135.stdout:1/928: rmdir d16/d129 0 2024-03-30T04:28:37.354 INFO:tasks.workunit.client.1.smithi135.stdout:3/919: link d3/d50/l48 d3/d10/d9c/dbe/dd6/l13d 0 2024-03-30T04:28:37.354 INFO:tasks.workunit.client.1.smithi135.stdout:5/831: dwrite d1/f37 [0,4194304] 0 2024-03-30T04:28:37.360 INFO:tasks.workunit.client.1.smithi135.stdout:9/879: getdents d7/d55 0 2024-03-30T04:28:37.360 INFO:tasks.workunit.client.1.smithi135.stdout:2/835: dwrite d4/d32/d12/f31 [0,4194304] 0 2024-03-30T04:28:37.360 INFO:tasks.workunit.client.1.smithi135.stdout:9/880: chown d7/d55/d8d/ddb/df0 5203 1 2024-03-30T04:28:37.362 INFO:tasks.workunit.client.1.smithi135.stdout:5/832: chown d1/d3/f49 649071101 1 2024-03-30T04:28:37.364 INFO:tasks.workunit.client.1.smithi135.stdout:2/836: stat d4/d32/d12/d45/dac/ce4 0 2024-03-30T04:28:37.364 INFO:tasks.workunit.client.1.smithi135.stdout:1/929: unlink d16/d29/d2c/dce/d131/d99/d9a/cdf 0 2024-03-30T04:28:37.368 INFO:tasks.workunit.client.1.smithi135.stdout:3/920: mknod d3/d10/d16/d23/d6a/d86/d98/d99/df7/c13e 0 2024-03-30T04:28:37.376 INFO:tasks.workunit.client.1.smithi135.stdout:2/837: creat d4/d32/d12/d45/dac/f118 x:0 0 0 2024-03-30T04:28:37.376 INFO:tasks.workunit.client.1.smithi135.stdout:8/867: truncate d6/d8/f1c 3366155 0 2024-03-30T04:28:37.378 INFO:tasks.workunit.client.1.smithi135.stdout:9/881: symlink d7/d29/d78/da4/l11c 0 2024-03-30T04:28:37.380 INFO:tasks.workunit.client.1.smithi135.stdout:3/921: symlink d3/l13f 0 2024-03-30T04:28:37.382 INFO:tasks.workunit.client.1.smithi135.stdout:3/922: chown d3/d54/dab 2475 1 2024-03-30T04:28:37.382 INFO:tasks.workunit.client.1.smithi135.stdout:8/868: write d6/d8/d15/d2a/dee/f29 [1652118,20077] 0 2024-03-30T04:28:37.386 INFO:tasks.workunit.client.1.smithi135.stdout:9/882: chown d7/d4f/d6e/dad/lbe 928084375 1 2024-03-30T04:28:37.393 INFO:tasks.workunit.client.1.smithi135.stdout:9/883: creat d7/d55/d8d/f11d x:0 0 0 2024-03-30T04:28:37.399 INFO:tasks.workunit.client.1.smithi135.stdout:1/930: dwrite f7 [0,4194304] 0 2024-03-30T04:28:37.400 INFO:tasks.workunit.client.1.smithi135.stdout:9/884: creat d7/d29/f11e x:0 0 0 2024-03-30T04:28:37.407 INFO:tasks.workunit.client.1.smithi135.stdout:5/833: dread d1/d7/f48 [0,4194304] 0 2024-03-30T04:28:37.408 INFO:tasks.workunit.client.1.smithi135.stdout:1/931: mkdir d16/d5a/d5f/d3a/d136 0 2024-03-30T04:28:37.409 INFO:tasks.workunit.client.1.smithi135.stdout:3/923: dwrite d3/d50/d46/f1c [0,4194304] 0 2024-03-30T04:28:37.412 INFO:tasks.workunit.client.1.smithi135.stdout:9/885: symlink d7/d4f/dee/d110/l11f 0 2024-03-30T04:28:37.413 INFO:tasks.workunit.client.1.smithi135.stdout:1/932: dread - d16/d5a/dad/d77/f118 zero size 2024-03-30T04:28:37.417 INFO:tasks.workunit.client.1.smithi135.stdout:9/886: creat d7/d4f/d5c/d63/f120 x:0 0 0 2024-03-30T04:28:37.422 INFO:tasks.workunit.client.1.smithi135.stdout:9/887: rename d7/d4f/d5c/f112 to d7/d4f/d5c/d63/dd2/db1/dbc/de6/f121 0 2024-03-30T04:28:37.426 INFO:tasks.workunit.client.1.smithi135.stdout:1/933: dread d16/d5a/d5f/d3d/fcb [0,4194304] 0 2024-03-30T04:28:37.427 INFO:tasks.workunit.client.1.smithi135.stdout:9/888: fdatasync d7/d4f/d5c/d63/dd2/db1/dbc/de6/f121 0 2024-03-30T04:28:37.432 INFO:tasks.workunit.client.1.smithi135.stdout:9/889: unlink d7/d29/c36 0 2024-03-30T04:28:37.433 INFO:tasks.workunit.client.1.smithi135.stdout:1/934: truncate d16/d29/d32/fdd 905915 0 2024-03-30T04:28:37.434 INFO:tasks.workunit.client.1.smithi135.stdout:5/834: dwrite d1/da/d17/d70/db2/d79/de1/fa9 [0,4194304] 0 2024-03-30T04:28:37.447 INFO:tasks.workunit.client.1.smithi135.stdout:1/935: chown d16/d29/d2c/d43/c54 2396675 1 2024-03-30T04:28:37.448 INFO:tasks.workunit.client.1.smithi135.stdout:1/936: fdatasync d16/d29/d61/fdc 0 2024-03-30T04:28:37.471 INFO:tasks.workunit.client.1.smithi135.stdout:9/890: link d7/ff d7/d55/d8d/ddb/de7/f122 0 2024-03-30T04:28:37.473 INFO:tasks.workunit.client.1.smithi135.stdout:5/835: rmdir d1/da/d4d/dbd 0 2024-03-30T04:28:37.475 INFO:tasks.workunit.client.1.smithi135.stdout:1/937: creat d16/d29/d2c/d43/d50/f137 x:0 0 0 2024-03-30T04:28:37.475 INFO:tasks.workunit.client.1.smithi135.stdout:5/836: dread - d1/da/d17/fe3 zero size 2024-03-30T04:28:37.475 INFO:tasks.workunit.client.1.smithi135.stdout:9/891: symlink d7/d55/d8d/ddb/de7/l123 0 2024-03-30T04:28:37.478 INFO:tasks.workunit.client.1.smithi135.stdout:5/837: dread - d1/d45/d52/da5/fc1 zero size 2024-03-30T04:28:37.480 INFO:tasks.workunit.client.1.smithi135.stdout:9/892: mkdir d7/d4f/d5c/d63/dd2/db1/dbc/dd8/d124 0 2024-03-30T04:28:37.480 INFO:tasks.workunit.client.1.smithi135.stdout:1/938: getdents d16/d5a/d5f/d3d/d8d 0 2024-03-30T04:28:37.481 INFO:tasks.workunit.client.1.smithi135.stdout:5/838: mknod d1/d3/d2b/d54/d6d/c100 0 2024-03-30T04:28:37.485 INFO:tasks.workunit.client.1.smithi135.stdout:9/893: rename d7/d29/d78/da4/d37/f54 to d7/d4f/d89/f125 0 2024-03-30T04:28:37.488 INFO:tasks.workunit.client.1.smithi135.stdout:5/839: symlink d1/da/d17/d70/db2/d79/de1/l101 0 2024-03-30T04:28:37.494 INFO:tasks.workunit.client.1.smithi135.stdout:5/840: mknod d1/c102 0 2024-03-30T04:28:37.495 INFO:tasks.workunit.client.1.smithi135.stdout:9/894: getdents d7/d41/d72 0 2024-03-30T04:28:37.500 INFO:tasks.workunit.client.1.smithi135.stdout:9/895: getdents d7/df8/dfb 0 2024-03-30T04:28:37.502 INFO:tasks.workunit.client.1.smithi135.stdout:9/896: read - d7/d4f/d5c/d63/fd9 zero size 2024-03-30T04:28:37.505 INFO:tasks.workunit.client.1.smithi135.stdout:9/897: truncate d7/f4e 4905585 0 2024-03-30T04:28:37.510 INFO:tasks.workunit.client.1.smithi135.stdout:5/841: dwrite d1/d7/d11/ff9 [0,4194304] 0 2024-03-30T04:28:37.513 INFO:tasks.workunit.client.1.smithi135.stdout:9/898: creat d7/d4f/d6e/dad/f126 x:0 0 0 2024-03-30T04:28:37.516 INFO:tasks.workunit.client.1.smithi135.stdout:9/899: symlink d7/d55/d8d/ddb/l127 0 2024-03-30T04:28:37.521 INFO:tasks.workunit.client.1.smithi135.stdout:9/900: mkdir d7/d55/d8d/d128 0 2024-03-30T04:28:37.525 INFO:tasks.workunit.client.1.smithi135.stdout:1/939: dwrite f3 [4194304,4194304] 0 2024-03-30T04:28:37.527 INFO:tasks.workunit.client.1.smithi135.stdout:9/901: read d7/d4f/d5c/d63/dd2/db1/dbc/de6/fe9 [371623,119925] 0 2024-03-30T04:28:37.531 INFO:tasks.workunit.client.1.smithi135.stdout:1/940: write d16/d5a/d5f/d3d/df7/f135 [384174,83376] 0 2024-03-30T04:28:37.532 INFO:tasks.workunit.client.1.smithi135.stdout:9/902: truncate d7/d4f/dee/f101 95781 0 2024-03-30T04:28:37.533 INFO:tasks.workunit.client.1.smithi135.stdout:5/842: dread d1/da/d17/d70/db2/d79/de1/fe2 [4194304,4194304] 0 2024-03-30T04:28:37.536 INFO:tasks.workunit.client.1.smithi135.stdout:5/843: chown d1/d45/d52/da5/fe6 3 1 2024-03-30T04:28:37.537 INFO:tasks.workunit.client.1.smithi135.stdout:9/903: symlink d7/d4f/d5c/d63/l129 0 2024-03-30T04:28:37.538 INFO:tasks.workunit.client.1.smithi135.stdout:1/941: rename d16/d29/d32/dcc/fa8 to d16/d5a/dad/f138 0 2024-03-30T04:28:37.540 INFO:tasks.workunit.client.1.smithi135.stdout:5/844: chown d1/d3/la7 11 1 2024-03-30T04:28:37.549 INFO:tasks.workunit.client.1.smithi135.stdout:5/845: truncate d1/d3/fa2 2084052 0 2024-03-30T04:28:37.553 INFO:tasks.workunit.client.1.smithi135.stdout:5/846: write d1/d7/d1a/d1f/d9b/de9/f9c [983658,124167] 0 2024-03-30T04:28:37.558 INFO:tasks.workunit.client.1.smithi135.stdout:5/847: write d1/d7/d11/fea [23913,98449] 0 2024-03-30T04:28:37.558 INFO:tasks.workunit.client.1.smithi135.stdout:9/904: dwrite d7/f5a [0,4194304] 0 2024-03-30T04:28:37.560 INFO:tasks.workunit.client.1.smithi135.stdout:9/905: chown d7/d4f/d5c/f84 17663 1 2024-03-30T04:28:37.563 INFO:tasks.workunit.client.1.smithi135.stdout:9/906: fdatasync d7/d41/f5f 0 2024-03-30T04:28:37.564 INFO:tasks.workunit.client.1.smithi135.stdout:5/848: creat d1/d7/d11/d3c/f103 x:0 0 0 2024-03-30T04:28:37.566 INFO:tasks.workunit.client.1.smithi135.stdout:9/907: write d7/d29/d78/da4/f80 [5711026,125767] 0 2024-03-30T04:28:37.569 INFO:tasks.workunit.client.1.smithi135.stdout:5/849: mkdir d1/d3/d2b/d54/d5e/d104 0 2024-03-30T04:28:37.569 INFO:tasks.workunit.client.1.smithi135.stdout:9/908: stat d7/d29/d78/da4/d37/dba 0 2024-03-30T04:28:37.571 INFO:tasks.workunit.client.1.smithi135.stdout:9/909: stat d7/d29/d78/da4/f3d 0 2024-03-30T04:28:37.571 INFO:tasks.workunit.client.1.smithi135.stdout:5/850: symlink d1/d45/d52/d8f/l105 0 2024-03-30T04:28:37.572 INFO:tasks.workunit.client.1.smithi135.stdout:1/942: dread d16/d5a/dad/fc2 [0,4194304] 0 2024-03-30T04:28:37.573 INFO:tasks.workunit.client.1.smithi135.stdout:5/851: readlink d1/d7/d1a/d1f/l64 0 2024-03-30T04:28:37.575 INFO:tasks.workunit.client.1.smithi135.stdout:1/943: write d16/d29/d2c/d43/d50/f11d [4019654,56424] 0 2024-03-30T04:28:37.579 INFO:tasks.workunit.client.1.smithi135.stdout:1/944: fdatasync d16/d29/d2c/d43/fa2 0 2024-03-30T04:28:37.585 INFO:tasks.workunit.client.1.smithi135.stdout:1/945: getdents d16/d29/d61 0 2024-03-30T04:28:37.587 INFO:tasks.workunit.client.1.smithi135.stdout:1/946: write f3 [607373,52716] 0 2024-03-30T04:28:37.590 INFO:tasks.workunit.client.1.smithi135.stdout:9/910: dwrite d7/d55/d8d/faf [0,4194304] 0 2024-03-30T04:28:37.592 INFO:tasks.workunit.client.1.smithi135.stdout:1/947: creat d16/d9f/f139 x:0 0 0 2024-03-30T04:28:37.595 INFO:tasks.workunit.client.1.smithi135.stdout:5/852: dwrite d1/d3/fa4 [4194304,4194304] 0 2024-03-30T04:28:37.604 INFO:tasks.workunit.client.1.smithi135.stdout:1/948: fdatasync d16/d5a/d5f/d3d/d8d/dc9/fd5 0 2024-03-30T04:28:37.606 INFO:tasks.workunit.client.1.smithi135.stdout:5/853: mknod d1/d7/d1a/d1f/d9b/de9/c106 0 2024-03-30T04:28:37.609 INFO:tasks.workunit.client.1.smithi135.stdout:5/854: write d1/d7/d1a/d1f/f66 [5098060,84732] 0 2024-03-30T04:28:37.609 INFO:tasks.workunit.client.1.smithi135.stdout:1/949: write d16/d29/d2c/d43/d50/f11d [1548431,80502] 0 2024-03-30T04:28:37.611 INFO:tasks.workunit.client.1.smithi135.stdout:1/950: dread - d16/d5a/dad/d77/f118 zero size 2024-03-30T04:28:37.612 INFO:tasks.workunit.client.1.smithi135.stdout:5/855: fdatasync d1/d7/d11/d3c/d4b/fe4 0 2024-03-30T04:28:37.614 INFO:tasks.workunit.client.1.smithi135.stdout:5/856: readlink d1/d45/d52/da5/le8 0 2024-03-30T04:28:37.617 INFO:tasks.workunit.client.1.smithi135.stdout:1/951: getdents d16/d5a 0 2024-03-30T04:28:37.620 INFO:tasks.workunit.client.1.smithi135.stdout:1/952: creat d16/d5a/d5f/d3a/dc6/f13a x:0 0 0 2024-03-30T04:28:37.623 INFO:tasks.workunit.client.1.smithi135.stdout:1/953: mknod d16/d9f/c13b 0 2024-03-30T04:28:37.626 INFO:tasks.workunit.client.1.smithi135.stdout:9/911: dwrite d7/d4f/d89/fa6 [0,4194304] 0 2024-03-30T04:28:37.631 INFO:tasks.workunit.client.1.smithi135.stdout:1/954: rename d16/d29/d2c/d43 to d16/d29/d32/d13c 0 2024-03-30T04:28:37.632 INFO:tasks.workunit.client.1.smithi135.stdout:1/955: chown d16/d5a 2094 1 2024-03-30T04:28:37.648 INFO:tasks.workunit.client.1.smithi135.stdout:9/912: dwrite d7/d4f/dee/d110/f113 [0,4194304] 0 2024-03-30T04:28:37.664 INFO:tasks.workunit.client.1.smithi135.stdout:9/913: dwrite d7/d29/d78/da4/fb6 [0,4194304] 0 2024-03-30T04:28:37.679 INFO:tasks.workunit.client.1.smithi135.stdout:9/914: dwrite d7/f26 [4194304,4194304] 0 2024-03-30T04:28:37.696 INFO:tasks.workunit.client.1.smithi135.stdout:9/915: dwrite d7/d90/f94 [0,4194304] 0 2024-03-30T04:28:37.699 INFO:tasks.workunit.client.1.smithi135.stdout:9/916: mkdir d7/d55/d8d/ddb/d12a 0 2024-03-30T04:28:37.701 INFO:tasks.workunit.client.1.smithi135.stdout:9/917: chown d7/d4f/d5c/f84 165487 1 2024-03-30T04:28:37.706 INFO:tasks.workunit.client.1.smithi135.stdout:9/918: fdatasync d7/d4f/d5c/d63/dd2/db1/dbc/f9a 0 2024-03-30T04:28:37.708 INFO:tasks.workunit.client.1.smithi135.stdout:9/919: fdatasync d7/fe3 0 2024-03-30T04:28:37.711 INFO:tasks.workunit.client.1.smithi135.stdout:9/920: mknod d7/d55/d8d/ddb/df0/c12b 0 2024-03-30T04:28:37.725 INFO:tasks.workunit.client.1.smithi135.stdout:9/921: dwrite d7/d4f/d5c/d63/dd2/dfe/f105 [0,4194304] 0 2024-03-30T04:28:37.727 INFO:tasks.workunit.client.1.smithi135.stdout:9/922: chown d7/d4f/d5c/f9e 96250 1 2024-03-30T04:28:37.728 INFO:tasks.workunit.client.1.smithi135.stdout:9/923: dread - d7/d4f/d6e/dad/ffd zero size 2024-03-30T04:28:37.755 INFO:tasks.workunit.client.1.smithi135.stdout:9/924: dwrite d7/d41/d72/f74 [0,4194304] 0 2024-03-30T04:28:37.798 INFO:tasks.workunit.client.1.smithi135.stdout:9/925: dread d7/d29/d78/da4/f3d [0,4194304] 0 2024-03-30T04:28:37.800 INFO:tasks.workunit.client.1.smithi135.stdout:9/926: chown d7/d4f/d5c/d63/f120 14 1 2024-03-30T04:28:37.804 INFO:tasks.workunit.client.1.smithi135.stdout:9/927: creat d7/df8/dfb/f12c x:0 0 0 2024-03-30T04:28:37.808 INFO:tasks.workunit.client.1.smithi135.stdout:9/928: creat d7/df8/f12d x:0 0 0 2024-03-30T04:28:37.810 INFO:tasks.workunit.client.1.smithi135.stdout:9/929: chown d7/d4f/d5c/d63/l129 9898800 1 2024-03-30T04:28:37.825 INFO:tasks.workunit.client.1.smithi135.stdout:9/930: dread d7/d29/f46 [0,4194304] 0 2024-03-30T04:28:37.830 INFO:tasks.workunit.client.1.smithi135.stdout:9/931: symlink d7/d41/d72/d107/l12e 0 2024-03-30T04:28:37.833 INFO:tasks.workunit.client.1.smithi135.stdout:9/932: write d7/d55/f6c [6085803,56699] 0 2024-03-30T04:28:37.835 INFO:tasks.workunit.client.1.smithi135.stdout:9/933: stat d7/d29/f2c 0 2024-03-30T04:28:37.852 INFO:tasks.workunit.client.1.smithi135.stdout:9/934: dwrite d7/f4e [4194304,4194304] 0 2024-03-30T04:28:37.854 INFO:tasks.workunit.client.1.smithi135.stdout:9/935: fsync d7/d29/d78/f109 0 2024-03-30T04:28:37.866 INFO:tasks.workunit.client.1.smithi135.stdout:9/936: fsync d7/df8/f12d 0 2024-03-30T04:28:37.871 INFO:tasks.workunit.client.1.smithi135.stdout:9/937: mknod d7/d4f/d6e/c12f 0 2024-03-30T04:28:37.872 INFO:tasks.workunit.client.1.smithi135.stdout:9/938: fdatasync d7/d90/f94 0 2024-03-30T04:28:37.916 INFO:tasks.workunit.client.1.smithi135.stdout:4/914: sync 2024-03-30T04:28:37.917 INFO:tasks.workunit.client.1.smithi135.stdout:2/838: sync 2024-03-30T04:28:37.919 INFO:tasks.workunit.client.1.smithi135.stdout:4/915: chown d10/c2a 1 1 2024-03-30T04:28:37.962 INFO:tasks.workunit.client.1.smithi135.stdout:5/857: sync 2024-03-30T04:28:37.963 INFO:tasks.workunit.client.1.smithi135.stdout:1/956: sync 2024-03-30T04:28:37.963 INFO:tasks.workunit.client.1.smithi135.stdout:8/869: sync 2024-03-30T04:28:37.963 INFO:tasks.workunit.client.1.smithi135.stdout:3/924: sync 2024-03-30T04:28:37.968 INFO:tasks.workunit.client.1.smithi135.stdout:3/925: read - d3/d10/d16/d41/f6d zero size 2024-03-30T04:28:37.970 INFO:tasks.workunit.client.1.smithi135.stdout:2/839: unlink d4/d32/d12/f3f 0 2024-03-30T04:28:37.972 INFO:tasks.workunit.client.1.smithi135.stdout:9/939: write d7/d41/fde [970049,67349] 0 2024-03-30T04:28:37.976 INFO:tasks.workunit.client.1.smithi135.stdout:1/957: mknod d16/d29/d2c/dce/d131/d99/d9a/c13d 0 2024-03-30T04:28:37.982 INFO:tasks.workunit.client.1.smithi135.stdout:8/870: mkdir d6/d8/d15/d54/d4f/d117 0 2024-03-30T04:28:38.000 INFO:tasks.workunit.client.1.smithi135.stdout:3/926: rename d3/d10/d16/d41/d83/cc9 to d3/d50/d22/d33/d121/c140 0 2024-03-30T04:28:38.002 INFO:tasks.workunit.client.1.smithi135.stdout:9/940: symlink d7/d41/db4/l130 0 2024-03-30T04:28:38.004 INFO:tasks.workunit.client.1.smithi135.stdout:1/958: truncate d16/d29/d2c/dce/d131/faf 2920299 0 2024-03-30T04:28:38.004 INFO:tasks.workunit.client.1.smithi135.stdout:9/941: dread - d7/d29/f11e zero size 2024-03-30T04:28:38.009 INFO:tasks.workunit.client.1.smithi135.stdout:2/840: symlink d4/d32/d2e/l119 0 2024-03-30T04:28:38.009 INFO:tasks.workunit.client.1.smithi135.stdout:1/959: truncate d16/d5a/f82 4280641 0 2024-03-30T04:28:38.011 INFO:tasks.workunit.client.1.smithi135.stdout:3/927: creat d3/d50/d46/f141 x:0 0 0 2024-03-30T04:28:38.012 INFO:tasks.workunit.client.1.smithi135.stdout:3/928: stat d3/d109 0 2024-03-30T04:28:38.014 INFO:tasks.workunit.client.1.smithi135.stdout:8/871: truncate d6/d8/fdf 124080 0 2024-03-30T04:28:38.017 INFO:tasks.workunit.client.1.smithi135.stdout:8/872: chown d6/d8/d15/d54/d8e/d72/d8b 32537 1 2024-03-30T04:28:38.019 INFO:tasks.workunit.client.1.smithi135.stdout:1/960: creat d16/d29/d32/dcc/f13e x:0 0 0 2024-03-30T04:28:38.022 INFO:tasks.workunit.client.1.smithi135.stdout:1/961: readlink d16/d5a/d5f/d3a/l47 0 2024-03-30T04:28:38.030 INFO:tasks.workunit.client.1.smithi135.stdout:9/942: getdents d7/d4f/d89 0 2024-03-30T04:28:38.035 INFO:tasks.workunit.client.1.smithi135.stdout:9/943: read d7/d55/d8d/f9d [454502,77022] 0 2024-03-30T04:28:38.040 INFO:tasks.workunit.client.1.smithi135.stdout:9/944: mknod d7/d55/d8d/d128/c131 0 2024-03-30T04:28:38.050 INFO:tasks.workunit.client.1.smithi135.stdout:8/873: dread d6/d8/d15/d54/d8e/d72/d73/fcb [0,4194304] 0 2024-03-30T04:28:38.091 INFO:tasks.workunit.client.1.smithi135.stdout:5/858: mkdir d1/d3/d2b/d54/d107 0 2024-03-30T04:28:38.095 INFO:tasks.workunit.client.1.smithi135.stdout:5/859: creat d1/da/d17/d70/df0/f108 x:0 0 0 2024-03-30T04:28:38.099 INFO:tasks.workunit.client.1.smithi135.stdout:5/860: truncate d1/d7/d11/d3c/f103 295262 0 2024-03-30T04:28:38.101 INFO:tasks.workunit.client.1.smithi135.stdout:1/962: write d16/d29/d32/dcc/fc7 [101140,114428] 0 2024-03-30T04:28:38.102 INFO:tasks.workunit.client.1.smithi135.stdout:5/861: chown d1/d3/d2b/f5b 498906 1 2024-03-30T04:28:38.104 INFO:tasks.workunit.client.1.smithi135.stdout:2/841: dwrite d4/f1c [4194304,4194304] 0 2024-03-30T04:28:38.106 INFO:tasks.workunit.client.1.smithi135.stdout:5/862: dread - d1/da/d17/d70/f99 zero size 2024-03-30T04:28:38.106 INFO:tasks.workunit.client.1.smithi135.stdout:9/945: write d7/d4f/d89/f125 [105574,17621] 0 2024-03-30T04:28:38.107 INFO:tasks.workunit.client.1.smithi135.stdout:4/916: rmdir d10/d1a 39 2024-03-30T04:28:38.108 INFO:tasks.workunit.client.1.smithi135.stdout:2/842: creat d4/d32/d12/d45/d7d/de7/f11a x:0 0 0 2024-03-30T04:28:38.112 INFO:tasks.workunit.client.1.smithi135.stdout:4/917: symlink d10/d1a/d42/d116/l12e 0 2024-03-30T04:28:38.113 INFO:tasks.workunit.client.1.smithi135.stdout:4/918: chown d10/d1a/d42/l5d 46454387 1 2024-03-30T04:28:38.114 INFO:tasks.workunit.client.1.smithi135.stdout:4/919: stat d10/d1a/d2b/cbb 0 2024-03-30T04:28:38.117 INFO:tasks.workunit.client.1.smithi135.stdout:3/929: rename d3/d10/d16/d23/d6a/d86 to d3/d50/d22/d33/d7b/d142 0 2024-03-30T04:28:38.124 INFO:tasks.workunit.client.1.smithi135.stdout:1/963: dread d16/d5a/f92 [0,4194304] 0 2024-03-30T04:28:38.126 INFO:tasks.workunit.client.1.smithi135.stdout:3/930: creat d3/db/d11a/f143 x:0 0 0 2024-03-30T04:28:38.126 INFO:tasks.workunit.client.1.smithi135.stdout:4/920: dread d10/d4d/f87 [0,4194304] 0 2024-03-30T04:28:38.129 INFO:tasks.workunit.client.1.smithi135.stdout:8/874: rename d6/d8/d15/d2a/da9 to d6/d8/d15/d2a/dee/d118 0 2024-03-30T04:28:38.129 INFO:tasks.workunit.client.1.smithi135.stdout:1/964: chown f7 0 1 2024-03-30T04:28:38.130 INFO:tasks.workunit.client.1.smithi135.stdout:4/921: truncate d10/d72/ff6 4929505 0 2024-03-30T04:28:38.132 INFO:tasks.workunit.client.1.smithi135.stdout:3/931: symlink d3/d54/d5c/da1/l144 0 2024-03-30T04:28:38.132 INFO:tasks.workunit.client.1.smithi135.stdout:8/875: readlink d6/d8/d15/d1f/l60 0 2024-03-30T04:28:38.135 INFO:tasks.workunit.client.1.smithi135.stdout:2/843: rename d4/d32/d2e/d2f/f64 to d4/da/d7a/de6/f11b 0 2024-03-30T04:28:38.136 INFO:tasks.workunit.client.1.smithi135.stdout:2/844: chown d4/d55 382 1 2024-03-30T04:28:38.138 INFO:tasks.workunit.client.1.smithi135.stdout:1/965: mkdir d16/d29/d2c/d13f 0 2024-03-30T04:28:38.143 INFO:tasks.workunit.client.1.smithi135.stdout:3/932: mkdir d3/d54/d5c/da1/d145 0 2024-03-30T04:28:38.145 INFO:tasks.workunit.client.1.smithi135.stdout:9/946: rename d7/d4f/d89/fc2 to d7/d4f/d5c/d63/dd2/f132 0 2024-03-30T04:28:38.152 INFO:tasks.workunit.client.1.smithi135.stdout:8/876: link d6/d8/d15/d2a/f4b d6/dc7/f119 0 2024-03-30T04:28:38.155 INFO:tasks.workunit.client.1.smithi135.stdout:9/947: dread d7/d29/f46 [0,4194304] 0 2024-03-30T04:28:38.157 INFO:tasks.workunit.client.1.smithi135.stdout:8/877: dread - d6/d8/f9e zero size 2024-03-30T04:28:38.158 INFO:tasks.workunit.client.1.smithi135.stdout:4/922: write d10/d4d/dc1/f93 [538157,41759] 0 2024-03-30T04:28:38.160 INFO:tasks.workunit.client.1.smithi135.stdout:1/966: dwrite d16/d5a/f82 [0,4194304] 0 2024-03-30T04:28:38.164 INFO:tasks.workunit.client.1.smithi135.stdout:8/878: creat d6/d8/d15/d79/f11a x:0 0 0 2024-03-30T04:28:38.170 INFO:tasks.workunit.client.1.smithi135.stdout:3/933: dwrite d3/d10/d9c/dbe/dd6/daf/f12d [0,4194304] 0 2024-03-30T04:28:38.175 INFO:tasks.workunit.client.1.smithi135.stdout:3/934: read - d3/d10/d16/d23/f27 zero size 2024-03-30T04:28:38.177 INFO:tasks.workunit.client.1.smithi135.stdout:2/845: dwrite d4/d32/d46/dbc/fc3 [0,4194304] 0 2024-03-30T04:28:38.181 INFO:tasks.workunit.client.1.smithi135.stdout:3/935: stat d3/d50/d22/f87 0 2024-03-30T04:28:38.181 INFO:tasks.workunit.client.1.smithi135.stdout:8/879: rmdir d6/d8/d15/d54/d8e/d72/d8b 0 2024-03-30T04:28:38.188 INFO:tasks.workunit.client.1.smithi135.stdout:4/923: dwrite d10/d1a/d32/d40/d92/db9/dcf/de8/f115 [0,4194304] 0 2024-03-30T04:28:38.188 INFO:tasks.workunit.client.1.smithi135.stdout:8/880: chown d6/dc7/lf4 58254 1 2024-03-30T04:28:38.195 INFO:tasks.workunit.client.1.smithi135.stdout:8/881: mkdir d6/d8/d15/d54/d7e/daf/dfb/dfd/d108/d11b 0 2024-03-30T04:28:38.201 INFO:tasks.workunit.client.1.smithi135.stdout:8/882: mkdir d6/d23/d9d/d11c 0 2024-03-30T04:28:38.209 INFO:tasks.workunit.client.1.smithi135.stdout:8/883: fdatasync d6/d23/d7b/f98 0 2024-03-30T04:28:38.216 INFO:tasks.workunit.client.1.smithi135.stdout:9/948: dread d7/d29/d78/f9b [0,4194304] 0 2024-03-30T04:28:38.216 INFO:tasks.workunit.client.1.smithi135.stdout:8/884: getdents d6/d8/d15/d54/d7e/da6/da7 0 2024-03-30T04:28:38.223 INFO:tasks.workunit.client.1.smithi135.stdout:3/936: write d3/d10/f44 [1779690,96547] 0 2024-03-30T04:28:38.223 INFO:tasks.workunit.client.1.smithi135.stdout:2/846: write d4/da/d7a/dd6/fdf [861732,25977] 0 2024-03-30T04:28:38.225 INFO:tasks.workunit.client.1.smithi135.stdout:4/924: truncate d10/f59 1330195 0 2024-03-30T04:28:38.226 INFO:tasks.workunit.client.1.smithi135.stdout:2/847: write d4/d32/d2e/d67/dad/f105 [1209909,106213] 0 2024-03-30T04:28:38.228 INFO:tasks.workunit.client.1.smithi135.stdout:3/937: write d3/d10/d16/d23/f12f [2656974,1983] 0 2024-03-30T04:28:38.234 INFO:tasks.workunit.client.1.smithi135.stdout:3/938: symlink d3/d54/d5c/da1/d145/l146 0 2024-03-30T04:28:38.235 INFO:tasks.workunit.client.1.smithi135.stdout:4/925: truncate d10/f17 2378191 0 2024-03-30T04:28:38.244 INFO:tasks.workunit.client.1.smithi135.stdout:4/926: rename d10/d1a/c55 to d10/d1a/d3d/c12f 0 2024-03-30T04:28:38.244 INFO:tasks.workunit.client.1.smithi135.stdout:9/949: dwrite d7/d4f/d5c/d63/dd2/dfe/f105 [0,4194304] 0 2024-03-30T04:28:38.248 INFO:tasks.workunit.client.1.smithi135.stdout:4/927: chown d10/d1a/d32/d40/d92/db9/dcf/d125 494789056 1 2024-03-30T04:28:38.253 INFO:tasks.workunit.client.1.smithi135.stdout:3/939: write d3/d50/d46/f59 [230834,13055] 0 2024-03-30T04:28:38.254 INFO:tasks.workunit.client.1.smithi135.stdout:2/848: dwrite d4/d32/f13 [4194304,4194304] 0 2024-03-30T04:28:38.257 INFO:tasks.workunit.client.1.smithi135.stdout:4/928: mknod d10/d1a/d32/d40/d92/db9/dc6/c130 0 2024-03-30T04:28:38.259 INFO:tasks.workunit.client.1.smithi135.stdout:9/950: rename d7/d29/c8b to d7/d55/d8d/ddb/d12a/c133 0 2024-03-30T04:28:38.260 INFO:tasks.workunit.client.1.smithi135.stdout:8/885: dread d6/d8/f1c [0,4194304] 0 2024-03-30T04:28:38.262 INFO:tasks.workunit.client.1.smithi135.stdout:8/886: readlink d6/d8/d15/d54/d4f/l7c 0 2024-03-30T04:28:38.264 INFO:tasks.workunit.client.1.smithi135.stdout:3/940: mkdir d3/d10/d9c/d147 0 2024-03-30T04:28:38.267 INFO:tasks.workunit.client.1.smithi135.stdout:8/887: chown d6/d8/d15/d54/d7e/daf/c111 79610 1 2024-03-30T04:28:38.270 INFO:tasks.workunit.client.1.smithi135.stdout:9/951: chown d7/c49 383 1 2024-03-30T04:28:38.273 INFO:tasks.workunit.client.1.smithi135.stdout:3/941: mkdir d3/db/da9/dcc/d148 0 2024-03-30T04:28:38.276 INFO:tasks.workunit.client.1.smithi135.stdout:3/942: fsync d3/d54/dab/fde 0 2024-03-30T04:28:38.280 INFO:tasks.workunit.client.1.smithi135.stdout:9/952: rename d7/d4f/cab to d7/d55/d8d/ddb/c134 0 2024-03-30T04:28:38.286 INFO:tasks.workunit.client.1.smithi135.stdout:3/943: getdents d3/d50/d22/d33/d7b 0 2024-03-30T04:28:38.288 INFO:tasks.workunit.client.1.smithi135.stdout:2/849: truncate d4/d6/d99/fdd 2646334 0 2024-03-30T04:28:38.290 INFO:tasks.workunit.client.1.smithi135.stdout:8/888: dwrite d6/d8/d15/d54/d7e/daf/dc1/fe1 [4194304,4194304] 0 2024-03-30T04:28:38.290 INFO:tasks.workunit.client.1.smithi135.stdout:9/953: dread d7/d4f/d5c/d63/dd2/db1/dbc/f9a [0,4194304] 0 2024-03-30T04:28:38.293 INFO:tasks.workunit.client.1.smithi135.stdout:2/850: chown d4/da/d7a/f106 16896 1 2024-03-30T04:28:38.296 INFO:tasks.workunit.client.1.smithi135.stdout:8/889: readlink d6/d23/l7a 0 2024-03-30T04:28:38.297 INFO:tasks.workunit.client.1.smithi135.stdout:3/944: getdents d3/d50/d22 0 2024-03-30T04:28:38.299 INFO:tasks.workunit.client.1.smithi135.stdout:4/929: dwrite d10/d4d/f97 [0,4194304] 0 2024-03-30T04:28:38.300 INFO:tasks.workunit.client.1.smithi135.stdout:2/851: chown d4/d32/d2e/d67/dad/de5/df1/l109 1115 1 2024-03-30T04:28:38.304 INFO:tasks.workunit.client.1.smithi135.stdout:3/945: fsync d3/f11 0 2024-03-30T04:28:38.305 INFO:tasks.workunit.client.1.smithi135.stdout:2/852: stat d4/d32/d2e/f8c 0 2024-03-30T04:28:38.307 INFO:tasks.workunit.client.1.smithi135.stdout:4/930: read fa [2323255,49846] 0 2024-03-30T04:28:38.310 INFO:tasks.workunit.client.1.smithi135.stdout:4/931: chown d10/d1a/d42/d66/dd7 910969 1 2024-03-30T04:28:38.321 INFO:tasks.workunit.client.1.smithi135.stdout:2/853: getdents d4/d6/d99 0 2024-03-30T04:28:38.322 INFO:tasks.workunit.client.1.smithi135.stdout:9/954: dwrite d7/d29/f11e [0,4194304] 0 2024-03-30T04:28:38.327 INFO:tasks.workunit.client.1.smithi135.stdout:4/932: rename d10/d4d/d89/d9f/fa7 to d10/d4d/dc3/d120/ded/f131 0 2024-03-30T04:28:38.328 INFO:tasks.workunit.client.1.smithi135.stdout:2/854: write d4/d32/d12/d45/d7d/de7/f11a [627357,55659] 0 2024-03-30T04:28:38.331 INFO:tasks.workunit.client.1.smithi135.stdout:9/955: creat d7/d4f/d5c/d63/dd2/f135 x:0 0 0 2024-03-30T04:28:38.332 INFO:tasks.workunit.client.1.smithi135.stdout:4/933: chown d10/d72/d7c/l7f 880 1 2024-03-30T04:28:38.332 INFO:tasks.workunit.client.1.smithi135.stdout:2/855: readlink d4/d32/dc2/l112 0 2024-03-30T04:28:38.334 INFO:tasks.workunit.client.1.smithi135.stdout:3/946: dwrite d3/d10/d16/d23/db6/fe2 [0,4194304] 0 2024-03-30T04:28:38.336 INFO:tasks.workunit.client.1.smithi135.stdout:2/856: dread - d4/d6/d47/ffe zero size 2024-03-30T04:28:38.340 INFO:tasks.workunit.client.1.smithi135.stdout:4/934: read d10/d1a/d42/de0/ff0 [780084,51775] 0 2024-03-30T04:28:38.343 INFO:tasks.workunit.client.1.smithi135.stdout:2/857: creat d4/d32/d46/dbc/dd0/f11c x:0 0 0 2024-03-30T04:28:38.344 INFO:tasks.workunit.client.1.smithi135.stdout:3/947: truncate d3/d54/d5c/da1/f131 628536 0 2024-03-30T04:28:38.348 INFO:tasks.workunit.client.1.smithi135.stdout:8/890: dwrite d6/d8/f38 [4194304,4194304] 0 2024-03-30T04:28:38.351 INFO:tasks.workunit.client.1.smithi135.stdout:8/891: dread - d6/d23/d7b/fcd zero size 2024-03-30T04:28:38.358 INFO:tasks.workunit.client.1.smithi135.stdout:2/858: rename d4/c28 to d4/c11d 0 2024-03-30T04:28:38.360 INFO:tasks.workunit.client.1.smithi135.stdout:9/956: truncate d7/d4f/d6e/dad/feb 8364412 0 2024-03-30T04:28:38.361 INFO:tasks.workunit.client.1.smithi135.stdout:8/892: mknod d6/d8/d15/d54/d7e/daf/dfb/c11d 0 2024-03-30T04:28:38.363 INFO:tasks.workunit.client.1.smithi135.stdout:9/957: dread - d7/d4f/d5c/d63/f120 zero size 2024-03-30T04:28:38.365 INFO:tasks.workunit.client.1.smithi135.stdout:2/859: truncate d4/d32/d46/dbc/dd0/f11c 1012012 0 2024-03-30T04:28:38.367 INFO:tasks.workunit.client.1.smithi135.stdout:8/893: symlink d6/d8/d15/d54/d8e/d72/l11e 0 2024-03-30T04:28:38.371 INFO:tasks.workunit.client.1.smithi135.stdout:8/894: stat d6/d8/d15/d54/d4f/c101 0 2024-03-30T04:28:38.371 INFO:tasks.workunit.client.1.smithi135.stdout:9/958: link d7/d55/d8d/d128/c131 d7/d41/db4/c136 0 2024-03-30T04:28:38.373 INFO:tasks.workunit.client.1.smithi135.stdout:2/860: creat d4/f11e x:0 0 0 2024-03-30T04:28:38.375 INFO:tasks.workunit.client.1.smithi135.stdout:9/959: truncate d7/d29/ffa 141789 0 2024-03-30T04:28:38.377 INFO:tasks.workunit.client.1.smithi135.stdout:2/861: mkdir d4/d32/d46/dbc/dbe/d11f 0 2024-03-30T04:28:38.379 INFO:tasks.workunit.client.1.smithi135.stdout:9/960: unlink d7/d29/d78/da4/d37/dba/cd0 0 2024-03-30T04:28:38.382 INFO:tasks.workunit.client.1.smithi135.stdout:8/895: rename d6/d8/d15/d54/d7e/daf/dfb/dfd/f113 to d6/d8/d15/d54/d8e/f11f 0 2024-03-30T04:28:38.383 INFO:tasks.workunit.client.1.smithi135.stdout:8/896: chown d6/d23/df9 1963 1 2024-03-30T04:28:38.384 INFO:tasks.workunit.client.1.smithi135.stdout:9/961: creat d7/d4f/d5c/d63/dd2/db1/dbc/dd8/d124/f137 x:0 0 0 2024-03-30T04:28:38.386 INFO:tasks.workunit.client.1.smithi135.stdout:2/862: rename d4/d32/d12/d45/f8e to d4/d32/d43/db5/dcb/f120 0 2024-03-30T04:28:38.387 INFO:tasks.workunit.client.1.smithi135.stdout:4/935: dread d10/d1a/d42/de0/ff0 [0,4194304] 0 2024-03-30T04:28:38.395 INFO:tasks.workunit.client.1.smithi135.stdout:2/863: chown d4/d55/d77/l51 5773 1 2024-03-30T04:28:38.399 INFO:tasks.workunit.client.1.smithi135.stdout:2/864: write d4/d32/d46/dbc/fc3 [4276976,48296] 0 2024-03-30T04:28:38.404 INFO:tasks.workunit.client.1.smithi135.stdout:9/962: dwrite d7/d4f/d6e/dad/f126 [0,4194304] 0 2024-03-30T04:28:38.408 INFO:tasks.workunit.client.1.smithi135.stdout:2/865: rename d4/d32/d43/f63 to d4/d32/d46/dbc/dbe/d11f/f121 0 2024-03-30T04:28:38.416 INFO:tasks.workunit.client.1.smithi135.stdout:2/866: mkdir d4/d32/d2e/d67/dad/de5/df1/d122 0 2024-03-30T04:28:38.425 INFO:tasks.workunit.client.1.smithi135.stdout:8/897: dwrite d6/d8/d15/d2a/f33 [0,4194304] 0 2024-03-30T04:28:38.425 INFO:tasks.workunit.client.1.smithi135.stdout:2/867: write d4/d32/d2e/d67/dad/fe1 [2348448,38208] 0 2024-03-30T04:28:38.430 INFO:tasks.workunit.client.1.smithi135.stdout:4/936: dwrite d10/d1a/f5c [0,4194304] 0 2024-03-30T04:28:38.433 INFO:tasks.workunit.client.1.smithi135.stdout:2/868: read - d4/fef zero size 2024-03-30T04:28:38.437 INFO:tasks.workunit.client.1.smithi135.stdout:9/963: dwrite d7/d29/d78/da4/d37/dba/fda [0,4194304] 0 2024-03-30T04:28:38.439 INFO:tasks.workunit.client.1.smithi135.stdout:8/898: rename d6/d23/dca to d6/d8/d15/d2a/d35/d120 0 2024-03-30T04:28:38.443 INFO:tasks.workunit.client.1.smithi135.stdout:2/869: fsync d4/d32/d12/d45/fa0 0 2024-03-30T04:28:38.445 INFO:tasks.workunit.client.1.smithi135.stdout:4/937: rename d10/d1a/d42/d66/f84 to d10/d1a/d32/d40/d92/db9/dcf/de8/d103/f132 0 2024-03-30T04:28:38.456 INFO:tasks.workunit.client.1.smithi135.stdout:2/870: mkdir d4/d32/d2e/d9f/d123 0 2024-03-30T04:28:38.456 INFO:tasks.workunit.client.1.smithi135.stdout:9/964: rename d7/d4f/d5c/d63/dd2/db1/cea to d7/d55/d8d/c138 0 2024-03-30T04:28:38.460 INFO:tasks.workunit.client.1.smithi135.stdout:8/899: write d6/d8/d15/d2a/dee/d32/f9a [289989,112584] 0 2024-03-30T04:28:38.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:38 smithi123 ceph-mon[35245]: pgmap v52: 65 pgs: 65 active+clean; 3.1 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 70 MiB/s rd, 176 MiB/s wr, 464 op/s 2024-03-30T04:28:38.473 INFO:tasks.workunit.client.1.smithi135.stdout:4/938: dwrite d10/d4d/d9c/d101/f12d [0,4194304] 0 2024-03-30T04:28:38.490 INFO:tasks.workunit.client.1.smithi135.stdout:2/871: creat d4/da/d7a/de6/f124 x:0 0 0 2024-03-30T04:28:38.490 INFO:tasks.workunit.client.1.smithi135.stdout:9/965: rmdir d7/d4f/d5c/d63/dd2/db1/dbc 39 2024-03-30T04:28:38.492 INFO:tasks.workunit.client.1.smithi135.stdout:9/966: write d7/f4e [3236983,86917] 0 2024-03-30T04:28:38.495 INFO:tasks.workunit.client.1.smithi135.stdout:2/872: symlink d4/d32/d12/d45/d6d/l125 0 2024-03-30T04:28:38.496 INFO:tasks.workunit.client.1.smithi135.stdout:8/900: truncate d6/f86 1123444 0 2024-03-30T04:28:38.497 INFO:tasks.workunit.client.1.smithi135.stdout:9/967: symlink d7/d41/d72/l139 0 2024-03-30T04:28:38.498 INFO:tasks.workunit.client.1.smithi135.stdout:8/901: chown d6/d23/c65 201 1 2024-03-30T04:28:38.499 INFO:tasks.workunit.client.1.smithi135.stdout:2/873: chown d4/d32/d2e/d67/dad/de5/df1/ff6 140770 1 2024-03-30T04:28:38.501 INFO:tasks.workunit.client.1.smithi135.stdout:8/902: dread - d6/d8/d15/d2a/dee/ff5 zero size 2024-03-30T04:28:38.502 INFO:tasks.workunit.client.1.smithi135.stdout:9/968: readlink d7/d4f/d5c/d63/dd2/db1/dbc/dd8/l10d 0 2024-03-30T04:28:38.504 INFO:tasks.workunit.client.1.smithi135.stdout:2/874: mknod d4/d32/d12/d45/d6d/dea/c126 0 2024-03-30T04:28:38.505 INFO:tasks.workunit.client.1.smithi135.stdout:8/903: write d6/d8/d15/d1f/ddb/fe6 [408236,89611] 0 2024-03-30T04:28:38.505 INFO:tasks.workunit.client.1.smithi135.stdout:9/969: readlink d7/d4f/d5c/d63/dd2/db1/dbc/dd8/l10d 0 2024-03-30T04:28:38.508 INFO:tasks.workunit.client.1.smithi135.stdout:2/875: write d4/d6/d47/f86 [2024882,68719] 0 2024-03-30T04:28:38.508 INFO:tasks.workunit.client.1.smithi135.stdout:8/904: fsync d6/d8/d15/d54/d7e/daf/dfb/f102 0 2024-03-30T04:28:38.509 INFO:tasks.workunit.client.1.smithi135.stdout:2/876: stat d4/d32/d43/db5/lff 0 2024-03-30T04:28:38.511 INFO:tasks.workunit.client.1.smithi135.stdout:9/970: rename d7/f22 to d7/d4f/d5c/f13a 0 2024-03-30T04:28:38.512 INFO:tasks.workunit.client.1.smithi135.stdout:8/905: creat d6/d8/d15/d1f/ddb/f121 x:0 0 0 2024-03-30T04:28:38.514 INFO:tasks.workunit.client.1.smithi135.stdout:2/877: chown d4/d32/d2e/d9f/ld4 1437858022 1 2024-03-30T04:28:38.519 INFO:tasks.workunit.client.1.smithi135.stdout:2/878: unlink d4/da/f44 0 2024-03-30T04:28:38.522 INFO:tasks.workunit.client.1.smithi135.stdout:2/879: mkdir d4/d32/d2e/d67/dad/d127 0 2024-03-30T04:28:38.533 INFO:tasks.workunit.client.1.smithi135.stdout:4/939: write d10/d1a/d3d/f95 [1246570,66869] 0 2024-03-30T04:28:38.537 INFO:tasks.workunit.client.1.smithi135.stdout:4/940: rmdir d10/d4d/d9c/d107 39 2024-03-30T04:28:38.539 INFO:tasks.workunit.client.1.smithi135.stdout:9/971: write d7/d55/d8d/f9d [4145068,123800] 0 2024-03-30T04:28:38.540 INFO:tasks.workunit.client.1.smithi135.stdout:4/941: creat d10/d1a/d2b/f133 x:0 0 0 2024-03-30T04:28:38.541 INFO:tasks.workunit.client.1.smithi135.stdout:8/906: write d6/d8/d15/d2a/dee/fda [853239,71830] 0 2024-03-30T04:28:38.544 INFO:tasks.workunit.client.1.smithi135.stdout:8/907: chown d6/d8/d15/d54/d4f/cc8 163 1 2024-03-30T04:28:38.545 INFO:tasks.workunit.client.1.smithi135.stdout:2/880: truncate d4/d55/f10f 2277725 0 2024-03-30T04:28:38.547 INFO:tasks.workunit.client.1.smithi135.stdout:8/908: chown d6/d8/f39 2609 1 2024-03-30T04:28:38.549 INFO:tasks.workunit.client.1.smithi135.stdout:9/972: dread d7/d41/fde [0,4194304] 0 2024-03-30T04:28:38.549 INFO:tasks.workunit.client.1.smithi135.stdout:2/881: unlink d4/d32/d2e/d67/dad/f105 0 2024-03-30T04:28:38.551 INFO:tasks.workunit.client.1.smithi135.stdout:8/909: mkdir d6/dc7/d122 0 2024-03-30T04:28:38.553 INFO:tasks.workunit.client.1.smithi135.stdout:2/882: truncate d4/d6/d47/ffe 773894 0 2024-03-30T04:28:38.557 INFO:tasks.workunit.client.1.smithi135.stdout:2/883: creat d4/da/d7a/de6/f128 x:0 0 0 2024-03-30T04:28:38.558 INFO:tasks.workunit.client.1.smithi135.stdout:8/910: creat d6/f123 x:0 0 0 2024-03-30T04:28:38.559 INFO:tasks.workunit.client.1.smithi135.stdout:2/884: dread - d4/d32/f9e zero size 2024-03-30T04:28:38.562 INFO:tasks.workunit.client.1.smithi135.stdout:2/885: write d4/d55/f5f [4790237,74140] 0 2024-03-30T04:28:38.564 INFO:tasks.workunit.client.1.smithi135.stdout:8/911: symlink d6/d23/d9d/d11c/l124 0 2024-03-30T04:28:38.567 INFO:tasks.workunit.client.1.smithi135.stdout:2/886: unlink d4/d32/d46/dbc/dbe/d11f/f121 0 2024-03-30T04:28:38.570 INFO:tasks.workunit.client.1.smithi135.stdout:2/887: dread - d4/da/d7a/de6/f128 zero size 2024-03-30T04:28:38.572 INFO:tasks.workunit.client.1.smithi135.stdout:9/973: dwrite d7/d4f/d6e/dad/feb [8388608,4194304] 0 2024-03-30T04:28:38.575 INFO:tasks.workunit.client.1.smithi135.stdout:9/974: truncate d7/d4f/d89/f125 777842 0 2024-03-30T04:28:38.575 INFO:tasks.workunit.client.1.smithi135.stdout:8/912: rename d6/d8/d15/d54/d8e/d72/d73/f9b to d6/d8/d15/d54/d8e/f125 0 2024-03-30T04:28:38.576 INFO:tasks.workunit.client.1.smithi135.stdout:2/888: symlink d4/d32/d46/dbc/dbe/d11f/l129 0 2024-03-30T04:28:38.579 INFO:tasks.workunit.client.1.smithi135.stdout:8/913: dread - d6/d8/d15/d54/d7e/da6/ffa zero size 2024-03-30T04:28:38.580 INFO:tasks.workunit.client.1.smithi135.stdout:2/889: creat d4/d32/d43/db5/f12a x:0 0 0 2024-03-30T04:28:38.589 INFO:tasks.workunit.client.1.smithi135.stdout:9/975: dread d7/d4f/d6e/dad/feb [8388608,4194304] 0 2024-03-30T04:28:38.591 INFO:tasks.workunit.client.1.smithi135.stdout:8/914: rename d6/d8/d15/d1f/ddb/ce4 to d6/d8/d15/d54/d7e/da6/da7/c126 0 2024-03-30T04:28:38.593 INFO:tasks.workunit.client.1.smithi135.stdout:8/915: rmdir d6/de5 39 2024-03-30T04:28:38.595 INFO:tasks.workunit.client.1.smithi135.stdout:2/890: dread d4/d32/d12/fb6 [4194304,4194304] 0 2024-03-30T04:28:38.598 INFO:tasks.workunit.client.1.smithi135.stdout:2/891: write d4/da/fb [4107850,67] 0 2024-03-30T04:28:38.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:38 smithi135 ceph-mon[44364]: pgmap v52: 65 pgs: 65 active+clean; 3.1 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 70 MiB/s rd, 176 MiB/s wr, 464 op/s 2024-03-30T04:28:38.600 INFO:tasks.workunit.client.1.smithi135.stdout:9/976: dread d7/d4f/d5c/d63/dd2/db1/dbc/f97 [0,4194304] 0 2024-03-30T04:28:38.602 INFO:tasks.workunit.client.1.smithi135.stdout:2/892: write d4/d32/d46/dbc/dbe/ff7 [846653,80026] 0 2024-03-30T04:28:38.608 INFO:tasks.workunit.client.1.smithi135.stdout:2/893: mkdir d4/d32/d2e/d67/dad/de5/d100/d115/d12b 0 2024-03-30T04:28:38.614 INFO:tasks.workunit.client.1.smithi135.stdout:8/916: dwrite d6/d23/d9d/ddd/fd5 [0,4194304] 0 2024-03-30T04:28:38.620 INFO:tasks.workunit.client.1.smithi135.stdout:9/977: dwrite d7/d4f/d5c/d63/f64 [0,4194304] 0 2024-03-30T04:28:38.621 INFO:tasks.workunit.client.1.smithi135.stdout:2/894: truncate d4/d32/d43/db5/dcb/f120 74200 0 2024-03-30T04:28:38.624 INFO:tasks.workunit.client.1.smithi135.stdout:2/895: chown d4/d32/d12/d45/d6d/lf4 1026860406 1 2024-03-30T04:28:38.624 INFO:tasks.workunit.client.1.smithi135.stdout:8/917: creat d6/d8/d15/d2a/f127 x:0 0 0 2024-03-30T04:28:38.631 INFO:tasks.workunit.client.1.smithi135.stdout:8/918: symlink d6/d8/d15/d54/d7e/daf/dfb/l128 0 2024-03-30T04:28:38.635 INFO:tasks.workunit.client.1.smithi135.stdout:8/919: truncate d6/d8/fd8 105063 0 2024-03-30T04:28:38.638 INFO:tasks.workunit.client.1.smithi135.stdout:8/920: symlink d6/dc7/l129 0 2024-03-30T04:28:38.639 INFO:tasks.workunit.client.1.smithi135.stdout:2/896: dread d4/d32/d12/d45/d7d/de7/f11a [0,4194304] 0 2024-03-30T04:28:38.643 INFO:tasks.workunit.client.1.smithi135.stdout:8/921: write d6/d23/df9/f109 [432157,128297] 0 2024-03-30T04:28:38.644 INFO:tasks.workunit.client.1.smithi135.stdout:9/978: dwrite d7/d4f/d5c/f9e [0,4194304] 0 2024-03-30T04:28:38.649 INFO:tasks.workunit.client.1.smithi135.stdout:9/979: chown d7/d4f/d5c/d63/dd2/f132 340010911 1 2024-03-30T04:28:38.649 INFO:tasks.workunit.client.1.smithi135.stdout:9/980: write d7/d29/d78/da4/f25 [4645430,88950] 0 2024-03-30T04:28:38.654 INFO:tasks.workunit.client.1.smithi135.stdout:9/981: creat d7/d41/db4/f13b x:0 0 0 2024-03-30T04:28:38.664 INFO:tasks.workunit.client.1.smithi135.stdout:2/897: dwrite d4/d32/d46/dbc/dbe/f111 [0,4194304] 0 2024-03-30T04:28:38.666 INFO:tasks.workunit.client.1.smithi135.stdout:9/982: write d7/d29/d78/da4/fbd [15987,58674] 0 2024-03-30T04:28:38.670 INFO:tasks.workunit.client.1.smithi135.stdout:8/922: dwrite d6/d8/d15/d1f/ddb/fe6 [0,4194304] 0 2024-03-30T04:28:38.672 INFO:tasks.workunit.client.1.smithi135.stdout:2/898: creat d4/d32/d43/f12c x:0 0 0 2024-03-30T04:28:38.672 INFO:tasks.workunit.client.1.smithi135.stdout:9/983: mkdir d7/d4f/dee/d13c 0 2024-03-30T04:28:38.673 INFO:tasks.workunit.client.1.smithi135.stdout:8/923: stat d6/d8/d15/d2a/dee/d32 0 2024-03-30T04:28:38.674 INFO:tasks.workunit.client.1.smithi135.stdout:2/899: chown d4/d55/f5f 33108 1 2024-03-30T04:28:38.677 INFO:tasks.workunit.client.1.smithi135.stdout:2/900: fsync d4/d32/d12/f31 0 2024-03-30T04:28:38.677 INFO:tasks.workunit.client.1.smithi135.stdout:9/984: creat d7/d55/d8d/ddb/de7/f13d x:0 0 0 2024-03-30T04:28:38.680 INFO:tasks.workunit.client.1.smithi135.stdout:8/924: truncate d6/d23/d9d/dad/fb0 4239381 0 2024-03-30T04:28:38.682 INFO:tasks.workunit.client.1.smithi135.stdout:8/925: stat d6/d8/d15/lae 0 2024-03-30T04:28:38.682 INFO:tasks.workunit.client.1.smithi135.stdout:9/985: rename d7/d29/f2c to d7/d4f/d5c/d63/f13e 0 2024-03-30T04:28:38.687 INFO:tasks.workunit.client.1.smithi135.stdout:2/901: rename d4/c97 to d4/d32/d2e/d67/dad/de5/d100/d115/c12d 0 2024-03-30T04:28:38.688 INFO:tasks.workunit.client.1.smithi135.stdout:8/926: symlink d6/d8/d15/d54/d8e/d72/l12a 0 2024-03-30T04:28:38.691 INFO:tasks.workunit.client.1.smithi135.stdout:9/986: rename d7/d4f/cb5 to d7/d4f/c13f 0 2024-03-30T04:28:38.693 INFO:tasks.workunit.client.1.smithi135.stdout:9/987: write d7/f11 [542861,10914] 0 2024-03-30T04:28:38.695 INFO:tasks.workunit.client.1.smithi135.stdout:2/902: dread d4/d6/f61 [4194304,4194304] 0 2024-03-30T04:28:38.696 INFO:tasks.workunit.client.1.smithi135.stdout:9/988: creat d7/d55/d8d/f140 x:0 0 0 2024-03-30T04:28:38.698 INFO:tasks.workunit.client.1.smithi135.stdout:2/903: write d4/d32/d2e/d67/db4/fc4 [501316,10078] 0 2024-03-30T04:28:38.704 INFO:tasks.workunit.client.1.smithi135.stdout:2/904: mkdir d4/d55/d77/d12e 0 2024-03-30T04:28:38.714 INFO:tasks.workunit.client.1.smithi135.stdout:8/927: dread d6/d8/d15/d79/f10a [0,4194304] 0 2024-03-30T04:28:38.723 INFO:tasks.workunit.client.1.smithi135.stdout:2/905: dwrite d4/d6/d99/ff9 [0,4194304] 0 2024-03-30T04:28:38.723 INFO:tasks.workunit.client.1.smithi135.stdout:9/989: dwrite d7/d4f/d5c/d63/dd2/db1/dbc/de6/fe9 [0,4194304] 0 2024-03-30T04:28:38.735 INFO:tasks.workunit.client.1.smithi135.stdout:2/906: mkdir d4/d32/d2e/d67/dad/de5/df1/d12f 0 2024-03-30T04:28:38.740 INFO:tasks.workunit.client.1.smithi135.stdout:2/907: symlink d4/d32/d12/d45/d6d/dea/l130 0 2024-03-30T04:28:38.742 INFO:tasks.workunit.client.1.smithi135.stdout:9/990: truncate d7/d29/f11e 1574872 0 2024-03-30T04:28:38.744 INFO:tasks.workunit.client.1.smithi135.stdout:9/991: stat d7/d29/d78/da4/f16 0 2024-03-30T04:28:38.745 INFO:tasks.workunit.client.1.smithi135.stdout:2/908: symlink d4/d55/l131 0 2024-03-30T04:28:38.746 INFO:tasks.workunit.client.1.smithi135.stdout:9/992: fsync d7/d29/d78/da4/f25 0 2024-03-30T04:28:38.752 INFO:tasks.workunit.client.1.smithi135.stdout:2/909: truncate d4/d32/d12/d45/fa0 79594 0 2024-03-30T04:28:38.755 INFO:tasks.workunit.client.1.smithi135.stdout:8/928: dread d6/d8/d15/d2a/dee/f29 [0,4194304] 0 2024-03-30T04:28:38.756 INFO:tasks.workunit.client.1.smithi135.stdout:2/910: creat d4/d32/d2e/d67/db4/ddb/f132 x:0 0 0 2024-03-30T04:28:38.763 INFO:tasks.workunit.client.1.smithi135.stdout:9/993: dread d7/f5a [0,4194304] 0 2024-03-30T04:28:38.764 INFO:tasks.workunit.client.1.smithi135.stdout:2/911: creat d4/d32/d12/d45/d6d/f133 x:0 0 0 2024-03-30T04:28:38.765 INFO:tasks.workunit.client.1.smithi135.stdout:2/912: readlink d4/d32/l3a 0 2024-03-30T04:28:38.768 INFO:tasks.workunit.client.1.smithi135.stdout:2/913: dread - d4/d32/daf/ffa zero size 2024-03-30T04:28:38.773 INFO:tasks.workunit.client.1.smithi135.stdout:2/914: rename d4/d32/d2e/le0 to d4/d32/d12/l134 0 2024-03-30T04:28:38.785 INFO:tasks.workunit.client.1.smithi135.stdout:9/994: dwrite d7/d29/d78/da4/f35 [0,4194304] 0 2024-03-30T04:28:38.787 INFO:tasks.workunit.client.1.smithi135.stdout:8/929: dread d6/d8/f31 [4194304,4194304] 0 2024-03-30T04:28:38.791 INFO:tasks.workunit.client.1.smithi135.stdout:2/915: write d4/d32/d2e/d67/dad/de5/df1/ff6 [1948257,82674] 0 2024-03-30T04:28:38.792 INFO:tasks.workunit.client.1.smithi135.stdout:9/995: rmdir d7/d4f/d5c/d63/dd2/db1/dbc/dd8 39 2024-03-30T04:28:38.797 INFO:tasks.workunit.client.1.smithi135.stdout:9/996: dread d7/d4f/d5c/f6d [0,4194304] 0 2024-03-30T04:28:38.799 INFO:tasks.workunit.client.1.smithi135.stdout:2/916: rename d4/d32/d2e/d67/dad/de5/d100/d115 to d4/d6/d47/d135 0 2024-03-30T04:28:38.800 INFO:tasks.workunit.client.1.smithi135.stdout:8/930: creat d6/f12b x:0 0 0 2024-03-30T04:28:38.801 INFO:tasks.workunit.client.1.smithi135.stdout:8/931: chown d6 33260 1 2024-03-30T04:28:38.803 INFO:tasks.workunit.client.1.smithi135.stdout:2/917: write d4/d32/d2e/d67/dad/fe1 [1374519,74714] 0 2024-03-30T04:28:38.806 INFO:tasks.workunit.client.1.smithi135.stdout:9/997: rename d7/d55/d8d/ddb/df0 to d7/d55/d8d/ddb/d12a/d141 0 2024-03-30T04:28:38.811 INFO:tasks.workunit.client.1.smithi135.stdout:9/998: rename d7 to d7/d55/d8d/ddb/d12a/d142 22 2024-03-30T04:28:38.819 INFO:tasks.workunit.client.1.smithi135.stdout:8/932: link d6/d8/d15/d54/d8e/dbd/ff7 d6/d8/d15/d54/d7e/da6/f12c 0 2024-03-30T04:28:38.822 INFO:tasks.workunit.client.1.smithi135.stdout:4/942: sync 2024-03-30T04:28:38.822 INFO:tasks.workunit.client.1.smithi135.stdout:1/967: sync 2024-03-30T04:28:38.822 INFO:tasks.workunit.client.1.smithi135.stdout:3/948: sync 2024-03-30T04:28:38.823 INFO:tasks.workunit.client.1.smithi135.stdout:5/863: sync 2024-03-30T04:28:38.826 INFO:tasks.workunit.client.1.smithi135.stdout:4/943: write d10/d1a/d2b/f133 [743755,79223] 0 2024-03-30T04:28:38.835 INFO:tasks.workunit.client.1.smithi135.stdout:1/968: creat d16/d5a/d5f/f140 x:0 0 0 2024-03-30T04:28:38.840 INFO:tasks.workunit.client.1.smithi135.stdout:1/969: fsync d16/d29/d61/fdc 0 2024-03-30T04:28:38.840 INFO:tasks.workunit.client.1.smithi135.stdout:3/949: unlink d3/d10/d16/l4f 0 2024-03-30T04:28:38.847 INFO:tasks.workunit.client.1.smithi135.stdout:3/950: dread - d3/d10/d9b/f135 zero size 2024-03-30T04:28:38.854 INFO:tasks.workunit.client.1.smithi135.stdout:1/970: creat d16/d5a/d5f/d3a/d136/f141 x:0 0 0 2024-03-30T04:28:38.856 INFO:tasks.workunit.client.1.smithi135.stdout:9/999: dread d7/d4f/d5c/d63/fe2 [0,4194304] 0 2024-03-30T04:28:38.856 INFO:tasks.workunit.client.1.smithi135.stdout:2/918: dwrite d4/d6/f1f [0,4194304] 0 2024-03-30T04:28:38.861 INFO:tasks.workunit.client.1.smithi135.stdout:3/951: creat d3/d54/d5c/da1/d145/f149 x:0 0 0 2024-03-30T04:28:38.868 INFO:tasks.workunit.client.1.smithi135.stdout:1/971: write d16/f2f [1026804,48930] 0 2024-03-30T04:28:38.871 INFO:tasks.workunit.client.1.smithi135.stdout:2/919: rmdir d4/d32/d2e/d9f 39 2024-03-30T04:28:38.877 INFO:tasks.workunit.client.1.smithi135.stdout:8/933: dwrite d6/d8/d15/d2a/f4b [0,4194304] 0 2024-03-30T04:28:38.882 INFO:tasks.workunit.client.1.smithi135.stdout:2/920: creat d4/d32/d12/d45/d3c/f136 x:0 0 0 2024-03-30T04:28:38.883 INFO:tasks.workunit.client.1.smithi135.stdout:5/864: dread d1/d7/d1a/d1f/d9b/de9/f9c [0,4194304] 0 2024-03-30T04:28:38.884 INFO:tasks.workunit.client.1.smithi135.stdout:4/944: dwrite d10/d72/fd0 [0,4194304] 0 2024-03-30T04:28:38.887 INFO:tasks.workunit.client.1.smithi135.stdout:8/934: creat d6/d8/f12d x:0 0 0 2024-03-30T04:28:38.887 INFO:tasks.workunit.client.1.smithi135.stdout:2/921: dread - d4/d32/d43/db5/f12a zero size 2024-03-30T04:28:38.892 INFO:tasks.workunit.client.1.smithi135.stdout:5/865: creat d1/d3/d2b/d54/d6d/f109 x:0 0 0 2024-03-30T04:28:38.896 INFO:tasks.workunit.client.1.smithi135.stdout:1/972: truncate d16/d5a/d5f/d3d/db4/f11c 534482 0 2024-03-30T04:28:38.896 INFO:tasks.workunit.client.1.smithi135.stdout:2/922: rmdir d4/d32/d43/db5 39 2024-03-30T04:28:38.900 INFO:tasks.workunit.client.1.smithi135.stdout:5/866: mkdir d1/d45/d10a 0 2024-03-30T04:28:38.905 INFO:tasks.workunit.client.1.smithi135.stdout:1/973: creat d16/d29/d32/f142 x:0 0 0 2024-03-30T04:28:38.905 INFO:tasks.workunit.client.1.smithi135.stdout:5/867: chown d1/da/d17/d70/db2/ff3 49 1 2024-03-30T04:28:38.905 INFO:tasks.workunit.client.1.smithi135.stdout:2/923: symlink d4/d32/d12/d45/d6d/l137 0 2024-03-30T04:28:38.905 INFO:tasks.workunit.client.1.smithi135.stdout:4/945: truncate d10/d4d/dc1/fbf 340994 0 2024-03-30T04:28:38.919 INFO:tasks.workunit.client.1.smithi135.stdout:5/868: creat d1/d7/d11/d3c/f10b x:0 0 0 2024-03-30T04:28:38.919 INFO:tasks.workunit.client.1.smithi135.stdout:4/946: mknod d10/d1a/d32/d40/d92/db9/dcf/df5/c134 0 2024-03-30T04:28:38.924 INFO:tasks.workunit.client.1.smithi135.stdout:5/869: symlink d1/d7/d1a/d1f/l10c 0 2024-03-30T04:28:38.928 INFO:tasks.workunit.client.1.smithi135.stdout:4/947: symlink d10/d4d/d9c/l135 0 2024-03-30T04:28:38.935 INFO:tasks.workunit.client.1.smithi135.stdout:5/870: dread - d1/d45/f88 zero size 2024-03-30T04:28:38.939 INFO:tasks.workunit.client.1.smithi135.stdout:2/924: dwrite d4/d32/d2e/f40 [4194304,4194304] 0 2024-03-30T04:28:38.960 INFO:tasks.workunit.client.1.smithi135.stdout:8/935: dread d6/f8c [0,4194304] 0 2024-03-30T04:28:38.963 INFO:tasks.workunit.client.1.smithi135.stdout:4/948: write d10/d72/d7c/dfb/dd4/fe5 [171056,33552] 0 2024-03-30T04:28:38.968 INFO:tasks.workunit.client.1.smithi135.stdout:8/936: dread - d6/d8/d15/d54/d7e/da6/fef zero size 2024-03-30T04:28:38.972 INFO:tasks.workunit.client.1.smithi135.stdout:4/949: unlink d10/d1a/d2b/l108 0 2024-03-30T04:28:38.974 INFO:tasks.workunit.client.1.smithi135.stdout:1/974: dread d16/d29/d32/d13c/d50/fe0 [0,4194304] 0 2024-03-30T04:28:38.977 INFO:tasks.workunit.client.1.smithi135.stdout:8/937: unlink d6/d8/d15/f106 0 2024-03-30T04:28:38.978 INFO:tasks.workunit.client.1.smithi135.stdout:2/925: dwrite d4/d32/d2e/f4b [0,4194304] 0 2024-03-30T04:28:38.983 INFO:tasks.workunit.client.1.smithi135.stdout:1/975: dread - d16/d5a/d5f/d3a/d136/f141 zero size 2024-03-30T04:28:38.985 INFO:tasks.workunit.client.1.smithi135.stdout:8/938: symlink d6/d8/d15/d2a/d35/d120/df0/l12e 0 2024-03-30T04:28:38.989 INFO:tasks.workunit.client.1.smithi135.stdout:2/926: mkdir d4/d32/dc2/dde/d138 0 2024-03-30T04:28:38.990 INFO:tasks.workunit.client.1.smithi135.stdout:8/939: read d6/d8/d15/d54/f5d [367170,14798] 0 2024-03-30T04:28:38.998 INFO:tasks.workunit.client.1.smithi135.stdout:5/871: dread d1/d7/d11/ff9 [0,4194304] 0 2024-03-30T04:28:38.999 INFO:tasks.workunit.client.1.smithi135.stdout:2/927: creat d4/d32/d2e/d67/dad/de5/df1/d122/f139 x:0 0 0 2024-03-30T04:28:39.001 INFO:tasks.workunit.client.1.smithi135.stdout:2/928: fdatasync d4/f11e 0 2024-03-30T04:28:39.002 INFO:tasks.workunit.client.1.smithi135.stdout:4/950: dwrite d10/d1a/d42/f54 [0,4194304] 0 2024-03-30T04:28:39.005 INFO:tasks.workunit.client.1.smithi135.stdout:5/872: dread d1/d7/f59 [0,4194304] 0 2024-03-30T04:28:39.007 INFO:tasks.workunit.client.1.smithi135.stdout:4/951: unlink d10/d1a/d32/fae 0 2024-03-30T04:28:39.012 INFO:tasks.workunit.client.1.smithi135.stdout:4/952: mkdir d10/d1a/d42/d46/def/d136 0 2024-03-30T04:28:39.016 INFO:tasks.workunit.client.1.smithi135.stdout:4/953: chown d10/d1a/d32/d40/d92/fb4 89105 1 2024-03-30T04:28:39.016 INFO:tasks.workunit.client.1.smithi135.stdout:2/929: mknod d4/d32/d43/db5/dcb/c13a 0 2024-03-30T04:28:39.020 INFO:tasks.workunit.client.1.smithi135.stdout:5/873: symlink d1/d45/d10a/l10d 0 2024-03-30T04:28:39.021 INFO:tasks.workunit.client.1.smithi135.stdout:1/976: write d16/d29/d32/dcc/fd4 [1572768,26704] 0 2024-03-30T04:28:39.022 INFO:tasks.workunit.client.1.smithi135.stdout:8/940: truncate d6/dc7/f119 2837494 0 2024-03-30T04:28:39.023 INFO:tasks.workunit.client.1.smithi135.stdout:4/954: fsync d10/d1a/f22 0 2024-03-30T04:28:39.027 INFO:tasks.workunit.client.1.smithi135.stdout:5/874: rmdir d1/da/d17/d70/df0 39 2024-03-30T04:28:39.030 INFO:tasks.workunit.client.1.smithi135.stdout:1/977: symlink d16/d29/d32/dcc/l143 0 2024-03-30T04:28:39.035 INFO:tasks.workunit.client.1.smithi135.stdout:8/941: symlink d6/d8/d15/d2a/dee/d118/l12f 0 2024-03-30T04:28:39.037 INFO:tasks.workunit.client.1.smithi135.stdout:5/875: creat d1/d7/d11/f10e x:0 0 0 2024-03-30T04:28:39.039 INFO:tasks.workunit.client.1.smithi135.stdout:1/978: symlink d16/d29/l144 0 2024-03-30T04:28:39.045 INFO:tasks.workunit.client.1.smithi135.stdout:4/955: dwrite d10/f74 [4194304,4194304] 0 2024-03-30T04:28:39.050 INFO:tasks.workunit.client.1.smithi135.stdout:2/930: dread d4/d32/d2e/f5c [0,4194304] 0 2024-03-30T04:28:39.052 INFO:tasks.workunit.client.1.smithi135.stdout:1/979: rename d16/d29/d32/d13c/d50 to d16/d29/d32/d12d/d145 0 2024-03-30T04:28:39.053 INFO:tasks.workunit.client.1.smithi135.stdout:2/931: readlink d4/d32/d12/d45/d6d/l137 0 2024-03-30T04:28:39.057 INFO:tasks.workunit.client.1.smithi135.stdout:4/956: mkdir d10/d4d/d9c/d107/d137 0 2024-03-30T04:28:39.058 INFO:tasks.workunit.client.1.smithi135.stdout:1/980: mkdir d16/d29/d32/d146 0 2024-03-30T04:28:39.061 INFO:tasks.workunit.client.1.smithi135.stdout:2/932: symlink d4/da/d8d/l13b 0 2024-03-30T04:28:39.064 INFO:tasks.workunit.client.1.smithi135.stdout:5/876: dwrite d1/d3/d76/f98 [0,4194304] 0 2024-03-30T04:28:39.066 INFO:tasks.workunit.client.1.smithi135.stdout:5/877: dread - d1/d3/d2b/fdf zero size 2024-03-30T04:28:39.070 INFO:tasks.workunit.client.1.smithi135.stdout:2/933: mkdir d4/d32/d2e/d67/d13c 0 2024-03-30T04:28:39.070 INFO:tasks.workunit.client.1.smithi135.stdout:4/957: unlink d10/d1a/d32/d40/d92/lac 0 2024-03-30T04:28:39.074 INFO:tasks.workunit.client.1.smithi135.stdout:4/958: chown d10/d1a/cc0 1194372 1 2024-03-30T04:28:39.078 INFO:tasks.workunit.client.1.smithi135.stdout:1/981: write d16/f45 [1027129,83165] 0 2024-03-30T04:28:39.082 INFO:tasks.workunit.client.1.smithi135.stdout:4/959: creat d10/d1a/d32/d40/f138 x:0 0 0 2024-03-30T04:28:39.082 INFO:tasks.workunit.client.1.smithi135.stdout:5/878: getdents d1/d7/d11/d3c/d4b 0 2024-03-30T04:28:39.099 INFO:tasks.workunit.client.1.smithi135.stdout:5/879: creat d1/d45/f10f x:0 0 0 2024-03-30T04:28:39.100 INFO:tasks.workunit.client.1.smithi135.stdout:2/934: truncate d4/d32/d2e/d9f/fcc 332926 0 2024-03-30T04:28:39.101 INFO:tasks.workunit.client.1.smithi135.stdout:1/982: dread d16/d29/d32/dcc/fd4 [0,4194304] 0 2024-03-30T04:28:39.104 INFO:tasks.workunit.client.1.smithi135.stdout:1/983: chown d16/d29/d61 22 1 2024-03-30T04:28:39.104 INFO:tasks.workunit.client.1.smithi135.stdout:5/880: write d1/d7/f10 [1062455,68632] 0 2024-03-30T04:28:39.108 INFO:tasks.workunit.client.1.smithi135.stdout:1/984: symlink d16/l147 0 2024-03-30T04:28:39.119 INFO:tasks.workunit.client.1.smithi135.stdout:1/985: rmdir d16/d5a/dad 39 2024-03-30T04:28:39.121 INFO:tasks.workunit.client.1.smithi135.stdout:4/960: dwrite d10/d1a/d32/d40/d92/db9/dcf/de8/ff4 [4194304,4194304] 0 2024-03-30T04:28:39.131 INFO:tasks.workunit.client.1.smithi135.stdout:2/935: dwrite d4/d32/d2e/d2f/fb7 [0,4194304] 0 2024-03-30T04:28:39.135 INFO:tasks.workunit.client.1.smithi135.stdout:5/881: dwrite d1/d7/d11/d3c/d4b/fe4 [0,4194304] 0 2024-03-30T04:28:39.140 INFO:tasks.workunit.client.1.smithi135.stdout:1/986: mknod d16/d5a/d5f/c148 0 2024-03-30T04:28:39.141 INFO:tasks.workunit.client.1.smithi135.stdout:2/936: write d4/d32/d12/fb6 [6016609,18669] 0 2024-03-30T04:28:39.145 INFO:tasks.workunit.client.1.smithi135.stdout:1/987: truncate d16/d29/d32/f119 488413 0 2024-03-30T04:28:39.145 INFO:tasks.workunit.client.1.smithi135.stdout:2/937: dread - d4/da/d7a/f106 zero size 2024-03-30T04:28:39.150 INFO:tasks.workunit.client.1.smithi135.stdout:2/938: unlink d4/d6/ced 0 2024-03-30T04:28:39.161 INFO:tasks.workunit.client.1.smithi135.stdout:5/882: write d1/da/d17/d70/db2/f65 [5042417,56850] 0 2024-03-30T04:28:39.163 INFO:tasks.workunit.client.1.smithi135.stdout:1/988: write d16/d29/d32/dcc/f95 [7415,84276] 0 2024-03-30T04:28:39.166 INFO:tasks.workunit.client.1.smithi135.stdout:1/989: stat d16/d5a/d5f/d3a/d136/f141 0 2024-03-30T04:28:39.167 INFO:tasks.workunit.client.1.smithi135.stdout:2/939: dwrite d4/d32/d46/f91 [0,4194304] 0 2024-03-30T04:28:39.172 INFO:tasks.workunit.client.1.smithi135.stdout:1/990: readlink d16/d29/d2c/dce/d131/d26/lf9 0 2024-03-30T04:28:39.172 INFO:tasks.workunit.client.1.smithi135.stdout:2/940: symlink d4/d32/d2e/d67/dad/de5/df1/d122/l13d 0 2024-03-30T04:28:39.176 INFO:tasks.workunit.client.1.smithi135.stdout:2/941: mknod d4/d32/d46/dbc/c13e 0 2024-03-30T04:28:39.181 INFO:tasks.workunit.client.1.smithi135.stdout:1/991: creat d16/d5a/d5f/d3d/db4/d125/f149 x:0 0 0 2024-03-30T04:28:39.181 INFO:tasks.workunit.client.1.smithi135.stdout:2/942: mkdir d4/d55/d13f 0 2024-03-30T04:28:39.183 INFO:tasks.workunit.client.1.smithi135.stdout:3/952: sync 2024-03-30T04:28:39.187 INFO:tasks.workunit.client.1.smithi135.stdout:2/943: read d4/d32/d2e/d67/dad/fe1 [1988913,125691] 0 2024-03-30T04:28:39.187 INFO:tasks.workunit.client.1.smithi135.stdout:3/953: mknod d3/d109/c14a 0 2024-03-30T04:28:39.190 INFO:tasks.workunit.client.1.smithi135.stdout:2/944: fsync d4/d32/f9e 0 2024-03-30T04:28:39.191 INFO:tasks.workunit.client.1.smithi135.stdout:3/954: mknod d3/d10/d9c/dbe/dd6/dd7/c14b 0 2024-03-30T04:28:39.199 INFO:tasks.workunit.client.1.smithi135.stdout:3/955: rename d3/db/d11a/f143 to d3/d10/d16/d23/f14c 0 2024-03-30T04:28:39.202 INFO:tasks.workunit.client.1.smithi135.stdout:2/945: write d4/d32/d12/d45/d6d/fcf [324817,115599] 0 2024-03-30T04:28:39.203 INFO:tasks.workunit.client.1.smithi135.stdout:1/992: dwrite d16/d5a/dad/d103/f10f [0,4194304] 0 2024-03-30T04:28:39.207 INFO:tasks.workunit.client.1.smithi135.stdout:1/993: rmdir d16/d29/d2c/dce/d131/d99/d9a 39 2024-03-30T04:28:39.209 INFO:tasks.workunit.client.1.smithi135.stdout:2/946: dread d4/d32/d12/d45/d7d/de7/f11a [0,4194304] 0 2024-03-30T04:28:39.211 INFO:tasks.workunit.client.1.smithi135.stdout:2/947: stat d4/da/c93 0 2024-03-30T04:28:39.219 INFO:tasks.workunit.client.1.smithi135.stdout:2/948: symlink d4/d32/d12/d45/d6d/l140 0 2024-03-30T04:28:39.225 INFO:tasks.workunit.client.1.smithi135.stdout:2/949: rename d4/d32/d46/dbc/dbe to d4/d32/d12/d45/d6d/dea/d10a/d141 0 2024-03-30T04:28:39.233 INFO:tasks.workunit.client.1.smithi135.stdout:1/994: dwrite d16/d5a/d5f/d3d/fcb [0,4194304] 0 2024-03-30T04:28:39.233 INFO:tasks.workunit.client.1.smithi135.stdout:2/950: unlink d4/d32/d2e/d67/dad/de5/c10d 0 2024-03-30T04:28:39.241 INFO:tasks.workunit.client.1.smithi135.stdout:2/951: creat d4/d32/d2e/d67/dad/de5/df1/d122/f142 x:0 0 0 2024-03-30T04:28:39.243 INFO:tasks.workunit.client.1.smithi135.stdout:3/956: dread d3/d10/ffe [0,4194304] 0 2024-03-30T04:28:39.248 INFO:tasks.workunit.client.1.smithi135.stdout:2/952: chown d4/d32/d12/d45/d7d/de7/lfd 412882 1 2024-03-30T04:28:39.251 INFO:tasks.workunit.client.1.smithi135.stdout:3/957: fdatasync d3/d50/d22/d33/d7b/d142/f8c 0 2024-03-30T04:28:39.251 INFO:tasks.workunit.client.1.smithi135.stdout:2/953: write d4/d32/d12/d45/dac/f118 [990554,97936] 0 2024-03-30T04:28:39.254 INFO:tasks.workunit.client.1.smithi135.stdout:2/954: chown d4/d32/f116 143 1 2024-03-30T04:28:39.257 INFO:tasks.workunit.client.1.smithi135.stdout:3/958: mkdir d3/d10/d16/d23/d6a/d94/d14d 0 2024-03-30T04:28:39.258 INFO:tasks.workunit.client.1.smithi135.stdout:2/955: unlink d4/d32/d43/fca 0 2024-03-30T04:28:39.259 INFO:tasks.workunit.client.1.smithi135.stdout:1/995: truncate d16/d29/d32/d12d/d145/fe0 84500 0 2024-03-30T04:28:39.262 INFO:tasks.workunit.client.1.smithi135.stdout:2/956: mknod d4/d32/d2e/c143 0 2024-03-30T04:28:39.263 INFO:tasks.workunit.client.1.smithi135.stdout:1/996: readlink d16/d29/d2c/dce/d131/d26/lfa 0 2024-03-30T04:28:39.264 INFO:tasks.workunit.client.1.smithi135.stdout:2/957: readlink d4/d32/d12/l74 0 2024-03-30T04:28:39.265 INFO:tasks.workunit.client.1.smithi135.stdout:3/959: rename d3/d54/d5c/da1/d145 to d3/d10/d9c/dbe/dd6/d91/d14e 0 2024-03-30T04:28:39.268 INFO:tasks.workunit.client.1.smithi135.stdout:3/960: write d3/d54/f11f [1381577,25833] 0 2024-03-30T04:28:39.271 INFO:tasks.workunit.client.1.smithi135.stdout:1/997: rename d16/d29/d32/dcc/f13e to d16/d29/d32/dcc/d10a/f14a 0 2024-03-30T04:28:39.273 INFO:tasks.workunit.client.1.smithi135.stdout:1/998: write d16/d29/d61/fdc [1499410,106311] 0 2024-03-30T04:28:39.273 INFO:tasks.workunit.client.1.smithi135.stdout:3/961: truncate d3/d50/d22/d33/f61 8663645 0 2024-03-30T04:28:39.275 INFO:tasks.workunit.client.1.smithi135.stdout:1/999: write d16/d29/d32/dcc/f95 [187194,116501] 0 2024-03-30T04:28:39.278 INFO:tasks.workunit.client.1.smithi135.stdout:2/958: dread d4/d32/d2e/f5c [0,4194304] 0 2024-03-30T04:28:39.280 INFO:tasks.workunit.client.1.smithi135.stdout:2/959: chown d4/d32/d2e/d9f/ld4 182270 1 2024-03-30T04:28:39.281 INFO:tasks.workunit.client.1.smithi135.stdout:2/960: chown d4/d32/d2e/d2f/f30 56 1 2024-03-30T04:28:39.282 INFO:tasks.workunit.client.1.smithi135.stdout:2/961: stat d4/d6/d99/fc9 0 2024-03-30T04:28:39.285 INFO:tasks.workunit.client.1.smithi135.stdout:2/962: write d4/d32/d12/d45/dac/f118 [1715764,17660] 0 2024-03-30T04:28:39.298 INFO:tasks.workunit.client.1.smithi135.stdout:3/962: dread d3/d50/d46/fb8 [0,4194304] 0 2024-03-30T04:28:39.299 INFO:tasks.workunit.client.1.smithi135.stdout:2/963: dwrite d4/d32/d46/f91 [0,4194304] 0 2024-03-30T04:28:39.299 INFO:tasks.workunit.client.1.smithi135.stdout:3/963: write d3/fa6 [2230901,88288] 0 2024-03-30T04:28:39.303 INFO:tasks.workunit.client.1.smithi135.stdout:2/964: symlink d4/d32/d46/l144 0 2024-03-30T04:28:39.309 INFO:tasks.workunit.client.1.smithi135.stdout:3/964: link d3/d10/d16/d23/d6a/cf3 d3/d10/d9c/dbe/dd6/daf/c14f 0 2024-03-30T04:28:39.312 INFO:tasks.workunit.client.1.smithi135.stdout:3/965: creat d3/d50/d46/d11d/f150 x:0 0 0 2024-03-30T04:28:39.315 INFO:tasks.workunit.client.1.smithi135.stdout:3/966: creat d3/d10/d16/d23/d6a/df9/f151 x:0 0 0 2024-03-30T04:28:39.317 INFO:tasks.workunit.client.1.smithi135.stdout:3/967: dread - d3/db/da9/dcc/f116 zero size 2024-03-30T04:28:39.320 INFO:tasks.workunit.client.1.smithi135.stdout:3/968: read d3/d10/f44 [1539025,121846] 0 2024-03-30T04:28:39.326 INFO:tasks.workunit.client.1.smithi135.stdout:3/969: dread d3/d10/d9c/dbe/dd6/dd7/f100 [0,4194304] 0 2024-03-30T04:28:39.331 INFO:tasks.workunit.client.1.smithi135.stdout:3/970: rename d3/d54/c8e to d3/db/da9/dcc/c152 0 2024-03-30T04:28:39.333 INFO:tasks.workunit.client.1.smithi135.stdout:2/965: dread d4/d32/d2e/fd3 [0,4194304] 0 2024-03-30T04:28:39.336 INFO:tasks.workunit.client.1.smithi135.stdout:2/966: creat d4/d32/d43/db5/f145 x:0 0 0 2024-03-30T04:28:39.338 INFO:tasks.workunit.client.1.smithi135.stdout:2/967: write d4/da/d7a/dd6/fdf [1248995,49348] 0 2024-03-30T04:28:39.346 INFO:tasks.workunit.client.1.smithi135.stdout:3/971: dwrite d3/d10/d16/d23/f12f [0,4194304] 0 2024-03-30T04:28:39.351 INFO:tasks.workunit.client.1.smithi135.stdout:2/968: dread d4/d6/d99/ff9 [0,4194304] 0 2024-03-30T04:28:39.362 INFO:tasks.workunit.client.1.smithi135.stdout:3/972: dwrite d3/d50/f25 [0,4194304] 0 2024-03-30T04:28:39.362 INFO:tasks.workunit.client.1.smithi135.stdout:2/969: write d4/d55/d77/f58 [4391058,28927] 0 2024-03-30T04:28:39.363 INFO:tasks.workunit.client.1.smithi135.stdout:2/970: chown d4/d32/d2e/l94 1061629 1 2024-03-30T04:28:39.365 INFO:tasks.workunit.client.1.smithi135.stdout:3/973: symlink d3/d10/d16/d23/l153 0 2024-03-30T04:28:39.366 INFO:tasks.workunit.client.1.smithi135.stdout:2/971: chown d4/d32/d12/d45/d7d/lfb 0 1 2024-03-30T04:28:39.369 INFO:tasks.workunit.client.1.smithi135.stdout:3/974: symlink d3/d10/d9c/dbe/dd6/d91/d14e/l154 0 2024-03-30T04:28:39.371 INFO:tasks.workunit.client.1.smithi135.stdout:8/942: sync 2024-03-30T04:28:39.373 INFO:tasks.workunit.client.1.smithi135.stdout:3/975: rmdir d3/d10/d9c/dbe/dd6/dd7 39 2024-03-30T04:28:39.375 INFO:tasks.workunit.client.1.smithi135.stdout:8/943: fsync d6/fa 0 2024-03-30T04:28:39.378 INFO:tasks.workunit.client.1.smithi135.stdout:8/944: write d6/f5b [1813334,49544] 0 2024-03-30T04:28:39.381 INFO:tasks.workunit.client.1.smithi135.stdout:2/972: truncate d4/d6/f61 3483853 0 2024-03-30T04:28:39.381 INFO:tasks.workunit.client.1.smithi135.stdout:8/945: symlink d6/d8/d15/d54/d7e/l130 0 2024-03-30T04:28:39.383 INFO:tasks.workunit.client.1.smithi135.stdout:2/973: fsync d4/d6/f1f 0 2024-03-30T04:28:39.387 INFO:tasks.workunit.client.1.smithi135.stdout:8/946: rmdir d6/d8/d15/d54/d7e/daf/dfb/dfd/d108 39 2024-03-30T04:28:39.387 INFO:tasks.workunit.client.1.smithi135.stdout:2/974: fdatasync d4/da/d7a/de6/f124 0 2024-03-30T04:28:39.393 INFO:tasks.workunit.client.1.smithi135.stdout:8/947: symlink d6/d8/d15/d1f/dcf/l131 0 2024-03-30T04:28:39.395 INFO:tasks.workunit.client.1.smithi135.stdout:3/976: dwrite d3/d10/d9c/dbe/dd6/dd7/f100 [0,4194304] 0 2024-03-30T04:28:39.398 INFO:tasks.workunit.client.1.smithi135.stdout:3/977: chown d3/d50/d22/fdf 2 1 2024-03-30T04:28:39.398 INFO:tasks.workunit.client.1.smithi135.stdout:8/948: read d6/d8/d15/d54/d8e/faa [1536516,37519] 0 2024-03-30T04:28:39.405 INFO:tasks.workunit.client.1.smithi135.stdout:3/978: rename d3/d10/d16/d23/d6a/d94 to d3/d10/d16/d23/d6a/d155 0 2024-03-30T04:28:39.408 INFO:tasks.workunit.client.1.smithi135.stdout:3/979: fdatasync d3/d50/f25 0 2024-03-30T04:28:39.409 INFO:tasks.workunit.client.1.smithi135.stdout:2/975: dwrite d4/d32/d2e/d2f/fb7 [4194304,4194304] 0 2024-03-30T04:28:39.420 INFO:tasks.workunit.client.1.smithi135.stdout:2/976: mkdir d4/da/d146 0 2024-03-30T04:28:39.421 INFO:tasks.workunit.client.1.smithi135.stdout:8/949: truncate d6/d8/d15/d1f/ddb/fe6 368161 0 2024-03-30T04:28:39.423 INFO:tasks.workunit.client.1.smithi135.stdout:3/980: truncate d3/d50/d22/d74/f7c 5627724 0 2024-03-30T04:28:39.424 INFO:tasks.workunit.client.1.smithi135.stdout:8/950: fdatasync d6/d8/d15/d54/d8e/f11f 0 2024-03-30T04:28:39.425 INFO:tasks.workunit.client.1.smithi135.stdout:2/977: symlink d4/d32/dc2/dde/df2/l147 0 2024-03-30T04:28:39.430 INFO:tasks.workunit.client.1.smithi135.stdout:3/981: rename d3/d50/d46/f58 to d3/d50/d46/d4c/f156 0 2024-03-30T04:28:39.431 INFO:tasks.workunit.client.1.smithi135.stdout:2/978: symlink d4/d32/d43/db5/dcb/l148 0 2024-03-30T04:28:39.440 INFO:tasks.workunit.client.1.smithi135.stdout:2/979: rename d4/d32/d12/l134 to d4/d32/dc2/dde/d138/l149 0 2024-03-30T04:28:39.441 INFO:tasks.workunit.client.1.smithi135.stdout:4/961: sync 2024-03-30T04:28:39.442 INFO:tasks.workunit.client.1.smithi135.stdout:5/883: sync 2024-03-30T04:28:39.442 INFO:tasks.workunit.client.1.smithi135.stdout:8/951: dwrite d6/d8/d15/d2a/dee/f4e [0,4194304] 0 2024-03-30T04:28:39.444 INFO:tasks.workunit.client.1.smithi135.stdout:2/980: unlink d4/d32/d2e/c143 0 2024-03-30T04:28:39.448 INFO:tasks.workunit.client.1.smithi135.stdout:4/962: mknod d10/d1a/d42/de0/c139 0 2024-03-30T04:28:39.453 INFO:tasks.workunit.client.1.smithi135.stdout:5/884: mknod d1/d45/c110 0 2024-03-30T04:28:39.463 INFO:tasks.workunit.client.1.smithi135.stdout:4/963: unlink d10/d1a/d32/d40/f8e 0 2024-03-30T04:28:39.469 INFO:tasks.workunit.client.1.smithi135.stdout:3/982: dwrite d3/d50/d22/d74/f7c [4194304,4194304] 0 2024-03-30T04:28:39.474 INFO:tasks.workunit.client.1.smithi135.stdout:8/952: dwrite d6/d8/d15/d54/d8e/d72/f10f [0,4194304] 0 2024-03-30T04:28:39.476 INFO:tasks.workunit.client.1.smithi135.stdout:2/981: dwrite d4/d32/f78 [0,4194304] 0 2024-03-30T04:28:39.498 INFO:tasks.workunit.client.1.smithi135.stdout:5/885: fsync d1/d7/d11/fde 0 2024-03-30T04:28:39.502 INFO:tasks.workunit.client.1.smithi135.stdout:5/886: write d1/d7/d1a/d1f/d9b/de9/f9c [3883650,83687] 0 2024-03-30T04:28:39.504 INFO:tasks.workunit.client.1.smithi135.stdout:2/982: getdents d4/da/d146 0 2024-03-30T04:28:39.505 INFO:tasks.workunit.client.1.smithi135.stdout:3/983: link d3/d10/d16/d23/d6a/f9e d3/d10/d16/d41/d83/f157 0 2024-03-30T04:28:39.505 INFO:tasks.workunit.client.1.smithi135.stdout:5/887: stat d1/d7/d1a/d1f/d9b/de9 0 2024-03-30T04:28:39.506 INFO:tasks.workunit.client.1.smithi135.stdout:4/964: mknod d10/d1a/d32/d40/d92/c13a 0 2024-03-30T04:28:39.508 INFO:tasks.workunit.client.1.smithi135.stdout:4/965: chown d10/d1a/f22 13 1 2024-03-30T04:28:39.511 INFO:tasks.workunit.client.1.smithi135.stdout:3/984: unlink d3/d10/d16/d41/f6d 0 2024-03-30T04:28:39.514 INFO:tasks.workunit.client.1.smithi135.stdout:5/888: creat d1/da/d17/d70/df0/f111 x:0 0 0 2024-03-30T04:28:39.514 INFO:tasks.workunit.client.1.smithi135.stdout:3/985: chown d3/d50/d22/fc0 71 1 2024-03-30T04:28:39.523 INFO:tasks.workunit.client.1.smithi135.stdout:8/953: dwrite d6/d8/d15/d54/d8e/ff2 [0,4194304] 0 2024-03-30T04:28:39.533 INFO:tasks.workunit.client.1.smithi135.stdout:8/954: mknod d6/d8/d15/d54/d7e/daf/dfb/dfd/d10c/c132 0 2024-03-30T04:28:39.535 INFO:tasks.workunit.client.1.smithi135.stdout:2/983: dread d4/d6/f16 [4194304,4194304] 0 2024-03-30T04:28:39.538 INFO:tasks.workunit.client.1.smithi135.stdout:2/984: write d4/d6/d99/ff9 [1132476,4059] 0 2024-03-30T04:28:39.543 INFO:tasks.workunit.client.1.smithi135.stdout:8/955: link d6/d8/d15/d2a/f3c d6/d8/d15/d54/d7e/daf/dc1/f133 0 2024-03-30T04:28:39.544 INFO:tasks.workunit.client.1.smithi135.stdout:5/889: rmdir d1/da/d17/d70/df0 39 2024-03-30T04:28:39.545 INFO:tasks.workunit.client.1.smithi135.stdout:4/966: write d10/d72/d7c/f9b [1965187,85121] 0 2024-03-30T04:28:39.549 INFO:tasks.workunit.client.1.smithi135.stdout:8/956: symlink d6/d8/d15/d54/d4f/l134 0 2024-03-30T04:28:39.553 INFO:tasks.workunit.client.1.smithi135.stdout:4/967: mknod d10/d4d/d89/d9f/c13b 0 2024-03-30T04:28:39.555 INFO:tasks.workunit.client.1.smithi135.stdout:5/890: rename d1/d45/d52/da5/cbb to d1/d7/d1a/d1f/d9b/de9/c112 0 2024-03-30T04:28:39.555 INFO:tasks.workunit.client.1.smithi135.stdout:2/985: dread d4/d6/d99/ff9 [0,4194304] 0 2024-03-30T04:28:39.556 INFO:tasks.workunit.client.1.smithi135.stdout:3/986: dwrite d3/d10/d16/d23/d6a/d155/fba [0,4194304] 0 2024-03-30T04:28:39.556 INFO:tasks.workunit.client.1.smithi135.stdout:8/957: fsync d6/d8/f3b 0 2024-03-30T04:28:39.564 INFO:tasks.workunit.client.1.smithi135.stdout:3/987: mkdir d3/d158 0 2024-03-30T04:28:39.574 INFO:tasks.workunit.client.1.smithi135.stdout:4/968: link d10/d1a/d32/c12a d10/d4d/dc1/db0/c13c 0 2024-03-30T04:28:39.574 INFO:tasks.workunit.client.1.smithi135.stdout:3/988: dread d3/db/f8d [0,4194304] 0 2024-03-30T04:28:39.576 INFO:tasks.workunit.client.1.smithi135.stdout:4/969: readlink d10/d4d/dc1/l99 0 2024-03-30T04:28:39.576 INFO:tasks.workunit.client.1.smithi135.stdout:3/989: chown d3/d54/dab/l102 1248256 1 2024-03-30T04:28:39.577 INFO:tasks.workunit.client.1.smithi135.stdout:8/958: write d6/d8/d15/d2a/dee/f5e [2674193,30224] 0 2024-03-30T04:28:39.577 INFO:tasks.workunit.client.1.smithi135.stdout:5/891: write d1/da/d17/d70/f99 [322067,41926] 0 2024-03-30T04:28:39.580 INFO:tasks.workunit.client.1.smithi135.stdout:3/990: chown d3/d50/d22/f10e 0 1 2024-03-30T04:28:39.582 INFO:tasks.workunit.client.1.smithi135.stdout:5/892: dread - d1/da/d17/fe3 zero size 2024-03-30T04:28:39.583 INFO:tasks.workunit.client.1.smithi135.stdout:4/970: chown d10/d1a/d32/d40/d92/fb4 2082 1 2024-03-30T04:28:39.583 INFO:tasks.workunit.client.1.smithi135.stdout:8/959: rmdir d6/d8/d15/d54/d7e/daf/dc1 39 2024-03-30T04:28:39.585 INFO:tasks.workunit.client.1.smithi135.stdout:3/991: fsync d3/d54/d5c/f89 0 2024-03-30T04:28:39.590 INFO:tasks.workunit.client.1.smithi135.stdout:4/971: write d10/fcc [510158,85204] 0 2024-03-30T04:28:39.593 INFO:tasks.workunit.client.1.smithi135.stdout:3/992: rename d3/d50/d22/d33/d7b/d142/d98/d99/df7 to d3/d50/d22/d33/d121/d159 0 2024-03-30T04:28:39.594 INFO:tasks.workunit.client.1.smithi135.stdout:5/893: getdents d1/d45/d52/da5/df1 0 2024-03-30T04:28:39.595 INFO:tasks.workunit.client.1.smithi135.stdout:5/894: stat d1/f6e 0 2024-03-30T04:28:39.596 INFO:tasks.workunit.client.1.smithi135.stdout:4/972: dread - d10/d1a/d32/d40/d92/db9/dcf/f110 zero size 2024-03-30T04:28:39.597 INFO:tasks.workunit.client.1.smithi135.stdout:3/993: mkdir d3/d50/d22/d15a 0 2024-03-30T04:28:39.601 INFO:tasks.workunit.client.1.smithi135.stdout:2/986: dread d4/d32/d2e/d67/db4/fc4 [0,4194304] 0 2024-03-30T04:28:39.601 INFO:tasks.workunit.client.1.smithi135.stdout:4/973: unlink d10/d1a/d3d/f95 0 2024-03-30T04:28:39.602 INFO:tasks.workunit.client.1.smithi135.stdout:5/895: symlink d1/d7/d1a/d1f/d9b/de9/l113 0 2024-03-30T04:28:39.603 INFO:tasks.workunit.client.1.smithi135.stdout:2/987: chown d4/d32/d46/dbc 961 1 2024-03-30T04:28:39.605 INFO:tasks.workunit.client.1.smithi135.stdout:4/974: write d10/d72/d7c/fa2 [4660736,119540] 0 2024-03-30T04:28:39.605 INFO:tasks.workunit.client.1.smithi135.stdout:8/960: dwrite d6/d8/d15/d79/f10a [0,4194304] 0 2024-03-30T04:28:39.607 INFO:tasks.workunit.client.1.smithi135.stdout:5/896: fdatasync d1/da/f9f 0 2024-03-30T04:28:39.608 INFO:tasks.workunit.client.1.smithi135.stdout:3/994: unlink d3/d50/d22/d33/d7b/lbc 0 2024-03-30T04:28:39.610 INFO:tasks.workunit.client.1.smithi135.stdout:8/961: chown d6/d8/d15/d2a/dee/d32/fab 1097627591 1 2024-03-30T04:28:39.614 INFO:tasks.workunit.client.1.smithi135.stdout:2/988: rename d4/d32/d46/dbc/dd0/ld2 to d4/d32/dc2/dde/l14a 0 2024-03-30T04:28:39.615 INFO:tasks.workunit.client.1.smithi135.stdout:3/995: mknod d3/d50/d22/c15b 0 2024-03-30T04:28:39.618 INFO:tasks.workunit.client.1.smithi135.stdout:3/996: fdatasync d3/f119 0 2024-03-30T04:28:39.618 INFO:tasks.workunit.client.1.smithi135.stdout:2/989: fsync d4/d32/d12/d45/d6d/fcf 0 2024-03-30T04:28:39.626 INFO:tasks.workunit.client.1.smithi135.stdout:3/997: fsync d3/d50/d46/d4c/f156 0 2024-03-30T04:28:39.628 INFO:tasks.workunit.client.1.smithi135.stdout:5/897: link d1/d3/d2b/cdc d1/da/d17/d70/db2/d79/de1/c114 0 2024-03-30T04:28:39.628 INFO:tasks.workunit.client.1.smithi135.stdout:4/975: dwrite d10/d1a/d42/f54 [4194304,4194304] 0 2024-03-30T04:28:39.636 INFO:tasks.workunit.client.1.smithi135.stdout:8/962: getdents d6/d8/d15/d1f 0 2024-03-30T04:28:39.643 INFO:tasks.workunit.client.1.smithi135.stdout:2/990: dwrite d4/d32/d43/db5/f145 [0,4194304] 0 2024-03-30T04:28:39.650 INFO:tasks.workunit.client.1.smithi135.stdout:8/963: chown d6/d8/d15/d79/l112 343174849 1 2024-03-30T04:28:39.651 INFO:tasks.workunit.client.1.smithi135.stdout:2/991: dread - d4/d32/d12/d45/d3c/f10b zero size 2024-03-30T04:28:39.651 INFO:tasks.workunit.client.1.smithi135.stdout:5/898: truncate d1/d3/d2b/d54/fd5 635091 0 2024-03-30T04:28:39.652 INFO:tasks.workunit.client.1.smithi135.stdout:4/976: creat d10/d4d/d89/d9f/f13d x:0 0 0 2024-03-30T04:28:39.658 INFO:tasks.workunit.client.1.smithi135.stdout:5/899: dread - d1/d3/d2b/d54/d6d/f109 zero size 2024-03-30T04:28:39.660 INFO:tasks.workunit.client.1.smithi135.stdout:4/977: mkdir d10/d1a/d42/d66/dc7/d106/d13e 0 2024-03-30T04:28:39.664 INFO:tasks.workunit.client.1.smithi135.stdout:5/900: creat d1/d7/d1a/dec/f115 x:0 0 0 2024-03-30T04:28:39.668 INFO:tasks.workunit.client.1.smithi135.stdout:8/964: rename d6/d23/d9d/ddd/ce8 to d6/d8/d15/d2a/dee/c135 0 2024-03-30T04:28:39.671 INFO:tasks.workunit.client.1.smithi135.stdout:3/998: dread d3/d6e/f80 [4194304,4194304] 0 2024-03-30T04:28:39.672 INFO:tasks.workunit.client.1.smithi135.stdout:5/901: mkdir d1/d3/d2b/d54/d5e/d116 0 2024-03-30T04:28:39.675 INFO:tasks.workunit.client.1.smithi135.stdout:2/992: dwrite d4/d32/d2e/d2f/f38 [4194304,4194304] 0 2024-03-30T04:28:39.676 INFO:tasks.workunit.client.1.smithi135.stdout:8/965: mknod d6/d8/d15/d54/d8e/d72/d73/c136 0 2024-03-30T04:28:39.677 INFO:tasks.workunit.client.1.smithi135.stdout:3/999: read d3/d50/d22/f10e [768961,8769] 0 2024-03-30T04:28:39.682 INFO:tasks.workunit.client.1.smithi135.stdout:4/978: write d10/d72/d7c/dfb/dd4/dda/f10d [1863635,9075] 0 2024-03-30T04:28:39.684 INFO:tasks.workunit.client.1.smithi135.stdout:5/902: symlink d1/d3/d2b/d54/d5e/l117 0 2024-03-30T04:28:39.686 INFO:tasks.workunit.client.1.smithi135.stdout:8/966: creat d6/d23/df9/f137 x:0 0 0 2024-03-30T04:28:39.688 INFO:tasks.workunit.client.1.smithi135.stdout:4/979: chown d10/d4d/dc3/fff 100169119 1 2024-03-30T04:28:39.692 INFO:tasks.workunit.client.1.smithi135.stdout:4/980: rmdir d10 39 2024-03-30T04:28:39.695 INFO:tasks.workunit.client.1.smithi135.stdout:8/967: truncate d6/d8/d15/d1f/f44 8014619 0 2024-03-30T04:28:39.697 INFO:tasks.workunit.client.1.smithi135.stdout:5/903: mkdir d1/d3/d2b/d54/d5e/d116/d118 0 2024-03-30T04:28:39.699 INFO:tasks.workunit.client.1.smithi135.stdout:4/981: fsync d10/d1a/d32/d40/f138 0 2024-03-30T04:28:39.715 INFO:tasks.workunit.client.1.smithi135.stdout:5/904: write d1/d45/d52/da5/dac/fc2 [1591824,7018] 0 2024-03-30T04:28:39.717 INFO:tasks.workunit.client.1.smithi135.stdout:8/968: truncate d6/d8/d15/d2a/dee/f4e 2391720 0 2024-03-30T04:28:39.720 INFO:tasks.workunit.client.1.smithi135.stdout:2/993: dread d4/d55/f5f [4194304,4194304] 0 2024-03-30T04:28:39.721 INFO:tasks.workunit.client.1.smithi135.stdout:8/969: write d6/d23/f27 [1922716,3083] 0 2024-03-30T04:28:39.722 INFO:tasks.workunit.client.1.smithi135.stdout:2/994: chown d4/d32/d2e/d67/db4 3 1 2024-03-30T04:28:39.724 INFO:tasks.workunit.client.1.smithi135.stdout:4/982: dwrite d10/d1a/f118 [0,4194304] 0 2024-03-30T04:28:39.732 INFO:tasks.workunit.client.1.smithi135.stdout:4/983: unlink d10/l36 0 2024-03-30T04:28:39.733 INFO:tasks.workunit.client.1.smithi135.stdout:5/905: dread d1/da/d17/d70/db2/d79/de1/f57 [0,4194304] 0 2024-03-30T04:28:39.733 INFO:tasks.workunit.client.1.smithi135.stdout:8/970: mknod d6/dc7/d122/c138 0 2024-03-30T04:28:39.736 INFO:tasks.workunit.client.1.smithi135.stdout:5/906: write d1/d45/d52/d8f/fd2 [3458785,24755] 0 2024-03-30T04:28:39.736 INFO:tasks.workunit.client.1.smithi135.stdout:8/971: chown d6/d8/d15/d79/fb1 44578638 1 2024-03-30T04:28:39.741 INFO:tasks.workunit.client.1.smithi135.stdout:4/984: symlink d10/d72/d7c/dfb/dd4/dda/df9/l13f 0 2024-03-30T04:28:39.742 INFO:tasks.workunit.client.1.smithi135.stdout:8/972: mknod d6/dc7/d122/c139 0 2024-03-30T04:28:39.744 INFO:tasks.workunit.client.1.smithi135.stdout:5/907: dread - d1/d7/d1a/d1f/fa6 zero size 2024-03-30T04:28:39.751 INFO:tasks.workunit.client.1.smithi135.stdout:5/908: rename d1/d7/d1a/d1f/d9b/le5 to d1/d3/d76/l119 0 2024-03-30T04:28:39.758 INFO:tasks.workunit.client.1.smithi135.stdout:2/995: dwrite d4/d32/d2e/d2f/f30 [0,4194304] 0 2024-03-30T04:28:39.765 INFO:tasks.workunit.client.1.smithi135.stdout:2/996: write d4/da/fb [3577577,62261] 0 2024-03-30T04:28:39.767 INFO:tasks.workunit.client.1.smithi135.stdout:4/985: dwrite d10/d4d/dc3/d120/ded/f131 [4194304,4194304] 0 2024-03-30T04:28:39.769 INFO:tasks.workunit.client.1.smithi135.stdout:5/909: rename d1/d7/d1a/d1f/d9b/de9/fe7 to d1/da/d17/d70/db2/d79/de1/f11a 0 2024-03-30T04:28:39.773 INFO:tasks.workunit.client.1.smithi135.stdout:2/997: mkdir d4/d32/d2e/d67/db4/d14b 0 2024-03-30T04:28:39.785 INFO:tasks.workunit.client.1.smithi135.stdout:8/973: dwrite d6/d8/d15/d2a/dee/f67 [0,4194304] 0 2024-03-30T04:28:39.789 INFO:tasks.workunit.client.1.smithi135.stdout:5/910: dwrite d1/da/d17/feb [0,4194304] 0 2024-03-30T04:28:39.799 INFO:tasks.workunit.client.1.smithi135.stdout:2/998: write d4/d32/d12/d45/d7d/fc1 [3179255,114013] 0 2024-03-30T04:28:39.804 INFO:tasks.workunit.client.1.smithi135.stdout:2/999: stat d4/d32/d12/d45/d7d/lfb 0 2024-03-30T04:28:39.817 INFO:tasks.workunit.client.1.smithi135.stdout:5/911: dwrite d1/da/d17/fe3 [0,4194304] 0 2024-03-30T04:28:39.817 INFO:tasks.workunit.client.1.smithi135.stdout:4/986: dwrite d10/d4d/d89/d9f/fb6 [0,4194304] 0 2024-03-30T04:28:39.823 INFO:tasks.workunit.client.1.smithi135.stdout:4/987: mknod d10/d1a/d32/d40/d92/db9/dcf/d112/c140 0 2024-03-30T04:28:39.841 INFO:tasks.workunit.client.1.smithi135.stdout:5/912: dwrite d1/d7/d11/ff9 [0,4194304] 0 2024-03-30T04:28:39.851 INFO:tasks.workunit.client.1.smithi135.stdout:4/988: write d10/d4d/dc3/d120/fdd [37803,120014] 0 2024-03-30T04:28:39.858 INFO:tasks.workunit.client.1.smithi135.stdout:4/989: read - d10/d4d/dc3/f11f zero size 2024-03-30T04:28:39.861 INFO:tasks.workunit.client.1.smithi135.stdout:5/913: write d1/d7/f2c [424532,93875] 0 2024-03-30T04:28:39.870 INFO:tasks.workunit.client.1.smithi135.stdout:5/914: mkdir d1/d3/d11b 0 2024-03-30T04:28:39.873 INFO:tasks.workunit.client.1.smithi135.stdout:4/990: creat d10/d1a/d42/d46/f141 x:0 0 0 2024-03-30T04:28:39.881 INFO:tasks.workunit.client.1.smithi135.stdout:5/915: unlink d1/d3/l29 0 2024-03-30T04:28:39.889 INFO:tasks.workunit.client.1.smithi135.stdout:5/916: getdents d1/d7/d1a/dec 0 2024-03-30T04:28:39.889 INFO:tasks.workunit.client.1.smithi135.stdout:4/991: dread d10/d72/fd0 [0,4194304] 0 2024-03-30T04:28:39.892 INFO:tasks.workunit.client.1.smithi135.stdout:4/992: write d10/f14 [903268,15057] 0 2024-03-30T04:28:39.898 INFO:tasks.workunit.client.1.smithi135.stdout:4/993: fdatasync d10/d1a/d2b/f31 0 2024-03-30T04:28:39.902 INFO:tasks.workunit.client.1.smithi135.stdout:5/917: link d1/d3/d2b/d54/d6d/c100 d1/d45/d10a/c11c 0 2024-03-30T04:28:39.904 INFO:tasks.workunit.client.1.smithi135.stdout:4/994: dread - d10/d1a/d32/d40/d92/db9/dcf/df5/f119 zero size 2024-03-30T04:28:39.904 INFO:tasks.workunit.client.1.smithi135.stdout:5/918: dread - d1/d3/d2b/fa8 zero size 2024-03-30T04:28:39.905 INFO:tasks.workunit.client.1.smithi135.stdout:4/995: chown d10/f14 246 1 2024-03-30T04:28:39.909 INFO:tasks.workunit.client.1.smithi135.stdout:5/919: mkdir d1/d7/d11d 0 2024-03-30T04:28:39.910 INFO:tasks.workunit.client.1.smithi135.stdout:4/996: chown d10/d1a/d32/d40/d92/db9/dcf/ffc 1 1 2024-03-30T04:28:39.914 INFO:tasks.workunit.client.1.smithi135.stdout:4/997: symlink d10/d1a/d42/de0/l142 0 2024-03-30T04:28:39.917 INFO:tasks.workunit.client.1.smithi135.stdout:5/920: rename d1/d7/d11/cb4 to d1/da/d4d/c11e 0 2024-03-30T04:28:39.918 INFO:tasks.workunit.client.1.smithi135.stdout:5/921: write d1/d7/f2c [1730366,41662] 0 2024-03-30T04:28:39.921 INFO:tasks.workunit.client.1.smithi135.stdout:4/998: mknod d10/c143 0 2024-03-30T04:28:39.923 INFO:tasks.workunit.client.1.smithi135.stdout:5/922: unlink d1/c34 0 2024-03-30T04:28:39.927 INFO:tasks.workunit.client.1.smithi135.stdout:5/923: fsync d1/d3/d2b/f53 0 2024-03-30T04:28:39.929 INFO:tasks.workunit.client.1.smithi135.stdout:5/924: fsync d1/da/d17/ffc 0 2024-03-30T04:28:39.940 INFO:tasks.workunit.client.1.smithi135.stdout:4/999: write d10/d1a/d32/d40/d92/db9/dcf/ffc [131394,99806] 0 2024-03-30T04:28:39.942 INFO:tasks.workunit.client.1.smithi135.stdout:5/925: truncate d1/d3/d2b/f53 2375664 0 2024-03-30T04:28:40.352 INFO:tasks.workunit.client.1.smithi135.stdout:8/974: sync 2024-03-30T04:28:40.352 INFO:tasks.workunit.client.1.smithi135.stdout:5/926: sync 2024-03-30T04:28:40.371 INFO:tasks.workunit.client.1.smithi135.stdout:8/975: dwrite d6/d8/d15/d54/d8e/fe0 [0,4194304] 0 2024-03-30T04:28:40.374 INFO:tasks.workunit.client.1.smithi135.stdout:5/927: dread d1/d3/fa4 [0,4194304] 0 2024-03-30T04:28:40.394 INFO:tasks.workunit.client.1.smithi135.stdout:5/928: dwrite d1/da/d17/d70/db2/dad/fc9 [0,4194304] 0 2024-03-30T04:28:40.395 INFO:tasks.workunit.client.1.smithi135.stdout:8/976: dread d6/d8/d15/d54/d8e/faa [0,4194304] 0 2024-03-30T04:28:40.396 INFO:tasks.workunit.client.1.smithi135.stdout:5/929: dread d1/d7/f59 [0,4194304] 0 2024-03-30T04:28:40.397 INFO:tasks.workunit.client.1.smithi135.stdout:5/930: fdatasync d1/d3/d2b/d54/d6d/f109 0 2024-03-30T04:28:40.398 INFO:tasks.workunit.client.1.smithi135.stdout:8/977: write d6/d23/df9/f109 [1563428,42035] 0 2024-03-30T04:28:40.401 INFO:tasks.workunit.client.1.smithi135.stdout:5/931: symlink d1/d7/d1a/d1f/l11f 0 2024-03-30T04:28:40.402 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:40 smithi123 ceph-mon[35245]: pgmap v53: 65 pgs: 65 active+clean; 3.1 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 57 MiB/s rd, 143 MiB/s wr, 371 op/s 2024-03-30T04:28:40.416 INFO:tasks.workunit.client.1.smithi135.stdout:5/932: write d1/d7/fcb [618907,107760] 0 2024-03-30T04:28:40.416 INFO:tasks.workunit.client.1.smithi135.stdout:8/978: dwrite d6/d8/d15/d54/d8e/fe0 [0,4194304] 0 2024-03-30T04:28:40.420 INFO:tasks.workunit.client.1.smithi135.stdout:5/933: getdents d1/d3/df7 0 2024-03-30T04:28:40.421 INFO:tasks.workunit.client.1.smithi135.stdout:8/979: truncate d6/f50 8868883 0 2024-03-30T04:28:40.424 INFO:tasks.workunit.client.1.smithi135.stdout:5/934: mknod d1/da/d17/d70/df0/c120 0 2024-03-30T04:28:40.427 INFO:tasks.workunit.client.1.smithi135.stdout:5/935: write d1/da/d17/d70/db2/d79/de1/fa9 [4648630,6036] 0 2024-03-30T04:28:40.430 INFO:tasks.workunit.client.1.smithi135.stdout:5/936: mkdir d1/da/d4d/d121 0 2024-03-30T04:28:40.434 INFO:tasks.workunit.client.1.smithi135.stdout:5/937: mknod d1/d7/d1a/d1f/d9b/de9/c122 0 2024-03-30T04:28:40.439 INFO:tasks.workunit.client.1.smithi135.stdout:8/980: dwrite d6/d8/d15/d2a/dee/d32/f62 [0,4194304] 0 2024-03-30T04:28:40.439 INFO:tasks.workunit.client.1.smithi135.stdout:5/938: link d1/d7/f2c d1/d7/d11/d3c/d4b/f123 0 2024-03-30T04:28:40.444 INFO:tasks.workunit.client.1.smithi135.stdout:8/981: creat d6/d8/d15/d2a/d35/d120/df0/ded/f13a x:0 0 0 2024-03-30T04:28:40.447 INFO:tasks.workunit.client.1.smithi135.stdout:8/982: truncate d6/d8/d15/d54/d7e/daf/f105 531810 0 2024-03-30T04:28:40.453 INFO:tasks.workunit.client.1.smithi135.stdout:5/939: rename d1/d7/d11/d3c/d4b to d1/d7/d11/d124 0 2024-03-30T04:28:40.456 INFO:tasks.workunit.client.1.smithi135.stdout:8/983: rename d6/d8/d15/d54/d4f/c55 to d6/d8/d15/d54/d7e/daf/dc1/c13b 0 2024-03-30T04:28:40.458 INFO:tasks.workunit.client.1.smithi135.stdout:5/940: write d1/d45/d52/da5/fe6 [4468010,44035] 0 2024-03-30T04:28:40.468 INFO:tasks.workunit.client.1.smithi135.stdout:5/941: fdatasync d1/da/d17/d70/db2/d79/de1/fa9 0 2024-03-30T04:28:40.474 INFO:tasks.workunit.client.1.smithi135.stdout:8/984: dwrite d6/f11 [4194304,4194304] 0 2024-03-30T04:28:40.475 INFO:tasks.workunit.client.1.smithi135.stdout:8/985: fdatasync d6/d8/fa5 0 2024-03-30T04:28:40.496 INFO:tasks.workunit.client.1.smithi135.stdout:8/986: creat d6/d8/d15/d1f/dcf/f13c x:0 0 0 2024-03-30T04:28:40.499 INFO:tasks.workunit.client.1.smithi135.stdout:8/987: rmdir d6/d8/d15/d54/d7e/daf/dfb/dfd 39 2024-03-30T04:28:40.504 INFO:tasks.workunit.client.1.smithi135.stdout:8/988: fsync d6/d8/d15/d2a/f4b 0 2024-03-30T04:28:40.520 INFO:tasks.workunit.client.1.smithi135.stdout:5/942: truncate d1/d7/d11/f87 6501560 0 2024-03-30T04:28:40.523 INFO:tasks.workunit.client.1.smithi135.stdout:5/943: mkdir d1/d3/d125 0 2024-03-30T04:28:40.527 INFO:tasks.workunit.client.1.smithi135.stdout:5/944: creat d1/d7/d1a/f126 x:0 0 0 2024-03-30T04:28:40.529 INFO:tasks.workunit.client.1.smithi135.stdout:8/989: dwrite d6/d23/d9d/ddd/fbb [0,4194304] 0 2024-03-30T04:28:40.531 INFO:tasks.workunit.client.1.smithi135.stdout:5/945: unlink d1/le 0 2024-03-30T04:28:40.534 INFO:tasks.workunit.client.1.smithi135.stdout:8/990: symlink d6/d23/d9d/dad/l13d 0 2024-03-30T04:28:40.549 INFO:tasks.workunit.client.1.smithi135.stdout:5/946: truncate d1/d7/fcb 493633 0 2024-03-30T04:28:40.552 INFO:tasks.workunit.client.1.smithi135.stdout:5/947: creat d1/da/f127 x:0 0 0 2024-03-30T04:28:40.555 INFO:tasks.workunit.client.1.smithi135.stdout:5/948: read - d1/d3/d76/fee zero size 2024-03-30T04:28:40.558 INFO:tasks.workunit.client.1.smithi135.stdout:8/991: dwrite d6/f86 [0,4194304] 0 2024-03-30T04:28:40.558 INFO:tasks.workunit.client.1.smithi135.stdout:5/949: write d1/da/f1b [5193378,80876] 0 2024-03-30T04:28:40.563 INFO:tasks.workunit.client.1.smithi135.stdout:5/950: write d1/d3/d2b/f5b [878945,121063] 0 2024-03-30T04:28:40.565 INFO:tasks.workunit.client.1.smithi135.stdout:5/951: chown d1/d7/d11/d3c/fd4 9424 1 2024-03-30T04:28:40.576 INFO:tasks.workunit.client.1.smithi135.stdout:5/952: dread d1/da/d17/fe3 [0,4194304] 0 2024-03-30T04:28:40.589 INFO:tasks.workunit.client.1.smithi135.stdout:5/953: dwrite d1/d7/d11/f8a [0,4194304] 0 2024-03-30T04:28:40.592 INFO:tasks.workunit.client.1.smithi135.stdout:5/954: write d1/d3/d76/f98 [2009391,73089] 0 2024-03-30T04:28:40.595 INFO:tasks.workunit.client.1.smithi135.stdout:5/955: fdatasync d1/d45/f83 0 2024-03-30T04:28:40.597 INFO:tasks.workunit.client.1.smithi135.stdout:5/956: chown d1/d3/d80/f93 5 1 2024-03-30T04:28:40.598 INFO:tasks.workunit.client.1.smithi135.stdout:8/992: dread d6/d23/f82 [0,4194304] 0 2024-03-30T04:28:40.600 INFO:tasks.workunit.client.1.smithi135.stdout:8/993: readlink d6/d8/d15/d1f/l60 0 2024-03-30T04:28:40.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:40 smithi135 ceph-mon[44364]: pgmap v53: 65 pgs: 65 active+clean; 3.1 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 57 MiB/s rd, 143 MiB/s wr, 371 op/s 2024-03-30T04:28:40.602 INFO:tasks.workunit.client.1.smithi135.stdout:5/957: mknod d1/da/d17/d70/db2/c128 0 2024-03-30T04:28:40.605 INFO:tasks.workunit.client.1.smithi135.stdout:5/958: dread - d1/d45/d52/da5/fc1 zero size 2024-03-30T04:28:40.606 INFO:tasks.workunit.client.1.smithi135.stdout:5/959: mknod d1/d3/d2b/d54/d5e/d116/c129 0 2024-03-30T04:28:40.609 INFO:tasks.workunit.client.1.smithi135.stdout:5/960: rmdir d1/da/d17/d70/db2/dd1 39 2024-03-30T04:28:40.612 INFO:tasks.workunit.client.1.smithi135.stdout:8/994: dread d6/d8/d15/d79/f10a [0,4194304] 0 2024-03-30T04:28:40.612 INFO:tasks.workunit.client.1.smithi135.stdout:5/961: mkdir d1/da/d17/d70/d12a 0 2024-03-30T04:28:40.614 INFO:tasks.workunit.client.1.smithi135.stdout:8/995: dread - d6/dc7/fe7 zero size 2024-03-30T04:28:40.615 INFO:tasks.workunit.client.1.smithi135.stdout:5/962: mknod d1/da/d17/d70/df0/c12b 0 2024-03-30T04:28:40.616 INFO:tasks.workunit.client.1.smithi135.stdout:8/996: readlink d6/d8/d15/d2a/d35/ld0 0 2024-03-30T04:28:40.619 INFO:tasks.workunit.client.1.smithi135.stdout:8/997: fdatasync d6/d8/f71 0 2024-03-30T04:28:40.620 INFO:tasks.workunit.client.1.smithi135.stdout:5/963: mknod d1/d7/d1a/d1f/d9b/de9/c12c 0 2024-03-30T04:28:40.621 INFO:tasks.workunit.client.1.smithi135.stdout:8/998: chown d6/d8/d15/d54/d8e/cc9 3979 1 2024-03-30T04:28:40.621 INFO:tasks.workunit.client.1.smithi135.stdout:5/964: readlink d1/d3/d2b/d54/d5e/ld0 0 2024-03-30T04:28:40.625 INFO:tasks.workunit.client.1.smithi135.stdout:5/965: creat d1/d45/d52/da5/df1/f12d x:0 0 0 2024-03-30T04:28:40.632 INFO:tasks.workunit.client.1.smithi135.stdout:8/999: truncate d6/d8/d15/d54/d8e/fe0 1687325 0 2024-03-30T04:28:40.643 INFO:tasks.workunit.client.1.smithi135.stdout:5/966: dread d1/d45/d52/fb3 [0,4194304] 0 2024-03-30T04:28:40.645 INFO:tasks.workunit.client.1.smithi135.stdout:5/967: chown d1/d3/fa2 5479989 1 2024-03-30T04:28:40.649 INFO:tasks.workunit.client.1.smithi135.stdout:5/968: link d1/da/d17/d70/f99 d1/d45/d52/da5/dac/f12e 0 2024-03-30T04:28:40.651 INFO:tasks.workunit.client.1.smithi135.stdout:5/969: mkdir d1/d7/d1a/dec/d12f 0 2024-03-30T04:28:40.653 INFO:tasks.workunit.client.1.smithi135.stdout:5/970: symlink d1/d7/d1a/d1f/d9b/l130 0 2024-03-30T04:28:40.654 INFO:tasks.workunit.client.1.smithi135.stdout:5/971: chown d1/d45/c91 32761194 1 2024-03-30T04:28:40.660 INFO:tasks.workunit.client.1.smithi135.stdout:5/972: link d1/da/d17/d70/db2/d79/de1/l101 d1/d7/d11/l131 0 2024-03-30T04:28:40.674 INFO:tasks.workunit.client.1.smithi135.stdout:5/973: dwrite d1/d3/d2b/f2f [4194304,4194304] 0 2024-03-30T04:28:40.685 INFO:tasks.workunit.client.1.smithi135.stdout:5/974: dread d1/d7/fcb [0,4194304] 0 2024-03-30T04:28:40.699 INFO:tasks.workunit.client.1.smithi135.stdout:5/975: dwrite d1/d7/d11/fea [0,4194304] 0 2024-03-30T04:28:40.702 INFO:tasks.workunit.client.1.smithi135.stdout:5/976: chown d1/d7/d1a/d1f/d9b/ca3 133 1 2024-03-30T04:28:40.708 INFO:tasks.workunit.client.1.smithi135.stdout:5/977: link d1/d7/c3f d1/da/d4d/c132 0 2024-03-30T04:28:40.721 INFO:tasks.workunit.client.1.smithi135.stdout:5/978: dread d1/da/d17/d70/db2/dad/fc9 [0,4194304] 0 2024-03-30T04:28:40.733 INFO:tasks.workunit.client.1.smithi135.stdout:5/979: dread d1/d3/d2b/d54/f71 [0,4194304] 0 2024-03-30T04:28:40.735 INFO:tasks.workunit.client.1.smithi135.stdout:5/980: mknod d1/d45/d10a/c133 0 2024-03-30T04:28:40.749 INFO:tasks.workunit.client.1.smithi135.stdout:5/981: dwrite d1/d3/d2b/fa8 [0,4194304] 0 2024-03-30T04:28:40.759 INFO:tasks.workunit.client.1.smithi135.stdout:5/982: getdents d1/da/d17/d70 0 2024-03-30T04:28:40.766 INFO:tasks.workunit.client.1.smithi135.stdout:5/983: write d1/d3/d2b/f53 [108774,82602] 0 2024-03-30T04:28:40.772 INFO:tasks.workunit.client.1.smithi135.stdout:5/984: write d1/da/d17/f3d [2513605,23697] 0 2024-03-30T04:28:40.775 INFO:tasks.workunit.client.1.smithi135.stdout:5/985: mkdir d1/d7/d1a/d1f/d9b/de9/d134 0 2024-03-30T04:28:40.791 INFO:tasks.workunit.client.1.smithi135.stdout:5/986: dwrite d1/d7/f59 [0,4194304] 0 2024-03-30T04:28:40.794 INFO:tasks.workunit.client.1.smithi135.stdout:5/987: stat d1/d7/f10 0 2024-03-30T04:28:40.798 INFO:tasks.workunit.client.1.smithi135.stdout:5/988: mkdir d1/d3/d2b/d54/d107/d135 0 2024-03-30T04:28:40.800 INFO:tasks.workunit.client.1.smithi135.stdout:5/989: creat d1/d45/d52/d8f/f136 x:0 0 0 2024-03-30T04:28:40.804 INFO:tasks.workunit.client.1.smithi135.stdout:5/990: symlink d1/d7/d11d/l137 0 2024-03-30T04:28:40.808 INFO:tasks.workunit.client.1.smithi135.stdout:5/991: stat d1/da/d17/d70/db2/d79/de1/c114 0 2024-03-30T04:28:40.812 INFO:tasks.workunit.client.1.smithi135.stdout:5/992: mkdir d1/da/d4d/d121/d138 0 2024-03-30T04:28:40.819 INFO:tasks.workunit.client.1.smithi135.stdout:5/993: getdents d1/d7/d1a/d1f/d9b/de9/d134 0 2024-03-30T04:28:40.829 INFO:tasks.workunit.client.1.smithi135.stdout:5/994: link d1/d7/d11d/l137 d1/da/d17/d70/db2/d79/l139 0 2024-03-30T04:28:40.854 INFO:tasks.workunit.client.1.smithi135.stdout:5/995: dwrite d1/d45/d52/da5/df1/f12d [0,4194304] 0 2024-03-30T04:28:40.876 INFO:tasks.workunit.client.1.smithi135.stdout:5/996: dwrite d1/da/f9f [0,4194304] 0 2024-03-30T04:28:40.905 INFO:tasks.workunit.client.1.smithi135.stdout:5/997: dwrite d1/d7/f43 [0,4194304] 0 2024-03-30T04:28:40.912 INFO:tasks.workunit.client.1.smithi135.stdout:5/998: symlink d1/d45/d52/da5/l13a 0 2024-03-30T04:28:40.916 INFO:tasks.workunit.client.1.smithi135.stdout:5/999: mknod d1/d7/d1a/d1f/c13b 0 2024-03-30T04:28:40.934 INFO:tasks.workunit.client.1.smithi135.stderr:+ rm -rf -- ./tmp.e0cy5c8jtQ 2024-03-30T04:28:42.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:42 smithi123 ceph-mon[35245]: pgmap v54: 65 pgs: 65 active+clean; 3.0 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 80 MiB/s rd, 196 MiB/s wr, 531 op/s 2024-03-30T04:28:42.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:42 smithi135 ceph-mon[44364]: pgmap v54: 65 pgs: 65 active+clean; 3.0 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 80 MiB/s rd, 196 MiB/s wr, 531 op/s 2024-03-30T04:28:43.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:43 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:28:43.852 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:43 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:28:43.852 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:43 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:28:43.852 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:43 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:28:43.852 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:43 smithi135 ceph-mon[44364]: pgmap v55: 65 pgs: 65 active+clean; 3.0 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 51 MiB/s rd, 117 MiB/s wr, 340 op/s 2024-03-30T04:28:43.935 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:43 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:28:43.935 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:43 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:28:43.935 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:43 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:28:43.935 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:43 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:28:43.936 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:43 smithi123 ceph-mon[35245]: pgmap v55: 65 pgs: 65 active+clean; 3.0 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 51 MiB/s rd, 117 MiB/s wr, 340 op/s 2024-03-30T04:28:44.730 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-30T04:28:44.730 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-30T04:28:45.436 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:45 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:28:45.436 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:45 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:28:45.436 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:45 smithi123 ceph-mon[35245]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:28:45.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:45 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:28:45.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:45 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:28:45.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:45 smithi135 ceph-mon[44364]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:28:46.425 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:46 smithi123 ceph-mon[35245]: pgmap v56: 65 pgs: 65 active+clean; 2.6 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 52 MiB/s rd, 122 MiB/s wr, 365 op/s 2024-03-30T04:28:46.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:46 smithi135 ceph-mon[44364]: pgmap v56: 65 pgs: 65 active+clean; 2.6 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 52 MiB/s rd, 122 MiB/s wr, 365 op/s 2024-03-30T04:28:47.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:47 smithi123 ceph-mon[35245]: Standby manager daemon smithi123.xgksqq restarted 2024-03-30T04:28:47.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:47 smithi123 ceph-mon[35245]: Standby manager daemon smithi123.xgksqq started 2024-03-30T04:28:47.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:47 smithi123 ceph-mon[35245]: from='mgr.? 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi123.xgksqq/crt"}]: dispatch 2024-03-30T04:28:47.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:47 smithi123 ceph-mon[35245]: from='mgr.? 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T04:28:47.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:47 smithi123 ceph-mon[35245]: from='mgr.? 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi123.xgksqq/key"}]: dispatch 2024-03-30T04:28:47.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:47 smithi123 ceph-mon[35245]: from='mgr.? 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T04:28:47.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:47 smithi135 ceph-mon[44364]: Standby manager daemon smithi123.xgksqq restarted 2024-03-30T04:28:47.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:47 smithi135 ceph-mon[44364]: Standby manager daemon smithi123.xgksqq started 2024-03-30T04:28:47.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:47 smithi135 ceph-mon[44364]: from='mgr.? 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi123.xgksqq/crt"}]: dispatch 2024-03-30T04:28:47.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:47 smithi135 ceph-mon[44364]: from='mgr.? 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T04:28:47.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:47 smithi135 ceph-mon[44364]: from='mgr.? 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi123.xgksqq/key"}]: dispatch 2024-03-30T04:28:47.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:47 smithi135 ceph-mon[44364]: from='mgr.? 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T04:28:48.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:48 smithi123 ceph-mon[35245]: pgmap v57: 65 pgs: 65 active+clean; 2.1 GiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 42 MiB/s rd, 99 MiB/s wr, 352 op/s 2024-03-30T04:28:48.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:48 smithi123 ceph-mon[35245]: mgrmap e26: smithi135.kyblli(active, since 108s), standbys: smithi123.xgksqq 2024-03-30T04:28:48.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:48 smithi135 ceph-mon[44364]: pgmap v57: 65 pgs: 65 active+clean; 2.1 GiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 42 MiB/s rd, 99 MiB/s wr, 352 op/s 2024-03-30T04:28:48.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:48 smithi135 ceph-mon[44364]: mgrmap e26: smithi135.kyblli(active, since 108s), standbys: smithi123.xgksqq 2024-03-30T04:28:50.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:50 smithi123 ceph-mon[35245]: pgmap v58: 65 pgs: 65 active+clean; 2.1 GiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 25 MiB/s rd, 59 MiB/s wr, 251 op/s 2024-03-30T04:28:50.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:50 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:28:50.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:50 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:28:50.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:50 smithi123 ceph-mon[35245]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:28:50.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:50 smithi123 ceph-mon[35245]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:28:50.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:50 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:28:50.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:50 smithi123 ceph-mon[35245]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:28:50.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:50 smithi123 ceph-mon[35245]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "mgr fail", "who": "smithi135.kyblli"}]: dispatch 2024-03-30T04:28:50.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:50 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "mgr fail", "who": "smithi135.kyblli"}]: dispatch 2024-03-30T04:28:50.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:50 smithi123 ceph-mon[35245]: osdmap e48: 6 total, 6 up, 6 in 2024-03-30T04:28:50.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:50 smithi123 ceph-mon[35245]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' cmd='[{"prefix": "mgr fail", "who": "smithi135.kyblli"}]': finished 2024-03-30T04:28:50.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:50 smithi123 ceph-mon[35245]: mgrmap e27: smithi123.xgksqq(active, starting, since 0.0097043s) 2024-03-30T04:28:50.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:50 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mon metadata", "id": "smithi123"}]: dispatch 2024-03-30T04:28:50.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:50 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-03-30T04:28:50.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:50 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi123.dvtind"}]: dispatch 2024-03-30T04:28:50.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:50 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi123.tqxvzr"}]: dispatch 2024-03-30T04:28:50.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:50 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi135.zvnlap"}]: dispatch 2024-03-30T04:28:50.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:50 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi135.xwgjvb"}]: dispatch 2024-03-30T04:28:50.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:50 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mgr metadata", "who": "smithi123.xgksqq", "id": "smithi123.xgksqq"}]: dispatch 2024-03-30T04:28:50.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:50 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T04:28:50.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:50 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T04:28:50.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:50 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T04:28:50.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:50 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T04:28:50.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:50 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T04:28:50.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:50 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T04:28:50.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:50 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-30T04:28:50.469 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:50 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-30T04:28:50.469 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:50 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-30T04:28:50.469 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:50 smithi123 ceph-mon[35245]: Manager daemon smithi123.xgksqq is now available 2024-03-30T04:28:50.469 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:50 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:28:50.469 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:50 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:28:50.469 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:50 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi123.xgksqq/mirror_snapshot_schedule"}]: dispatch 2024-03-30T04:28:50.469 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:50 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi123.xgksqq/trash_purge_schedule"}]: dispatch 2024-03-30T04:28:50.577 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:50 smithi135 ceph-mon[44364]: pgmap v58: 65 pgs: 65 active+clean; 2.1 GiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 25 MiB/s rd, 59 MiB/s wr, 251 op/s 2024-03-30T04:28:50.577 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:50 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:28:50.577 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:50 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:28:50.577 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:50 smithi135 ceph-mon[44364]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:28:50.577 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:50 smithi135 ceph-mon[44364]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:28:50.577 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:50 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' 2024-03-30T04:28:50.578 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:50 smithi135 ceph-mon[44364]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:28:50.578 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:50 smithi135 ceph-mon[44364]: from='mgr.14608 172.21.15.135:0/592770984' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "mgr fail", "who": "smithi135.kyblli"}]: dispatch 2024-03-30T04:28:50.578 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:50 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "mgr fail", "who": "smithi135.kyblli"}]: dispatch 2024-03-30T04:28:50.578 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:50 smithi135 ceph-mon[44364]: osdmap e48: 6 total, 6 up, 6 in 2024-03-30T04:28:50.578 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:50 smithi135 ceph-mon[44364]: from='mgr.14608 ' entity='mgr.smithi135.kyblli' cmd='[{"prefix": "mgr fail", "who": "smithi135.kyblli"}]': finished 2024-03-30T04:28:50.578 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:50 smithi135 ceph-mon[44364]: mgrmap e27: smithi123.xgksqq(active, starting, since 0.0097043s) 2024-03-30T04:28:50.578 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:50 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mon metadata", "id": "smithi123"}]: dispatch 2024-03-30T04:28:50.578 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:50 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-03-30T04:28:50.578 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:50 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi123.dvtind"}]: dispatch 2024-03-30T04:28:50.578 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:50 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi123.tqxvzr"}]: dispatch 2024-03-30T04:28:50.578 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:50 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi135.zvnlap"}]: dispatch 2024-03-30T04:28:50.578 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:50 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi135.xwgjvb"}]: dispatch 2024-03-30T04:28:50.578 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:50 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mgr metadata", "who": "smithi123.xgksqq", "id": "smithi123.xgksqq"}]: dispatch 2024-03-30T04:28:50.579 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:50 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T04:28:50.579 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:50 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T04:28:50.579 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:50 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T04:28:50.579 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:50 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T04:28:50.579 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:50 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T04:28:50.579 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:50 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T04:28:50.579 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:50 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-30T04:28:50.579 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:50 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-30T04:28:50.579 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:50 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-30T04:28:50.579 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:50 smithi135 ceph-mon[44364]: Manager daemon smithi123.xgksqq is now available 2024-03-30T04:28:50.579 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:50 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:28:50.579 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:50 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:28:50.579 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:50 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi123.xgksqq/mirror_snapshot_schedule"}]: dispatch 2024-03-30T04:28:50.579 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:50 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi123.xgksqq/trash_purge_schedule"}]: dispatch 2024-03-30T04:28:51.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:51 smithi123 ceph-mon[35245]: [30/Mar/2024:04:28:50] ENGINE Bus STARTING 2024-03-30T04:28:51.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:51 smithi123 ceph-mon[35245]: [30/Mar/2024:04:28:50] ENGINE Serving on https://172.21.15.123:7150 2024-03-30T04:28:51.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:51 smithi123 ceph-mon[35245]: [30/Mar/2024:04:28:50] ENGINE Serving on http://172.21.15.123:8765 2024-03-30T04:28:51.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:51 smithi123 ceph-mon[35245]: [30/Mar/2024:04:28:50] ENGINE Bus STARTED 2024-03-30T04:28:51.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:51 smithi123 ceph-mon[35245]: mgrmap e28: smithi123.xgksqq(active, since 1.01128s) 2024-03-30T04:28:51.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:51 smithi135 ceph-mon[44364]: [30/Mar/2024:04:28:50] ENGINE Bus STARTING 2024-03-30T04:28:51.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:51 smithi135 ceph-mon[44364]: [30/Mar/2024:04:28:50] ENGINE Serving on https://172.21.15.123:7150 2024-03-30T04:28:51.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:51 smithi135 ceph-mon[44364]: [30/Mar/2024:04:28:50] ENGINE Serving on http://172.21.15.123:8765 2024-03-30T04:28:51.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:51 smithi135 ceph-mon[44364]: [30/Mar/2024:04:28:50] ENGINE Bus STARTED 2024-03-30T04:28:51.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:51 smithi135 ceph-mon[44364]: mgrmap e28: smithi123.xgksqq(active, since 1.01128s) 2024-03-30T04:28:52.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:52 smithi123 ceph-mon[35245]: pgmap v3: 65 pgs: 65 active+clean; 1.2 GiB data, 5.4 GiB used, 531 GiB / 536 GiB avail 2024-03-30T04:28:52.586 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:52 smithi135 ceph-mon[44364]: pgmap v3: 65 pgs: 65 active+clean; 1.2 GiB data, 5.4 GiB used, 531 GiB / 536 GiB avail 2024-03-30T04:28:53.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.181+0000 7f9be4a00700 1 -- 172.21.15.123:0/1273021255 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9be0070890 msgr2=0x7f9be0070d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:28:53.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.181+0000 7f9be4a00700 1 --2- 172.21.15.123:0/1273021255 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9be0070890 0x7f9be0070d00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc004970 tx=0x7f9bcc009e30 comp rx=0 tx=0).stop 2024-03-30T04:28:53.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.181+0000 7f9be4a00700 1 -- 172.21.15.123:0/1273021255 shutdown_connections 2024-03-30T04:28:53.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.181+0000 7f9be4a00700 1 --2- 172.21.15.123:0/1273021255 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9be0070890 0x7f9be0070d00 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:53.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.181+0000 7f9be4a00700 1 --2- 172.21.15.123:0/1273021255 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9be0071e40 0x7f9be0070350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:53.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.181+0000 7f9be4a00700 1 -- 172.21.15.123:0/1273021255 >> 172.21.15.123:0/1273021255 conn(0x7f9be00f8550 msgr2=0x7f9be00fa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:28:53.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.182+0000 7f9be4a00700 1 -- 172.21.15.123:0/1273021255 shutdown_connections 2024-03-30T04:28:53.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.182+0000 7f9be4a00700 1 -- 172.21.15.123:0/1273021255 wait complete. 2024-03-30T04:28:53.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.183+0000 7f9be4a00700 1 Processor -- start 2024-03-30T04:28:53.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.184+0000 7f9be4a00700 1 -- start start 2024-03-30T04:28:53.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.184+0000 7f9be4a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9be0070890 0x7f9be006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:28:53.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.184+0000 7f9be4a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9be0071e40 0x7f9be006c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:28:53.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.184+0000 7f9be4a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9be006cbf0 con 0x7f9be0071e40 2024-03-30T04:28:53.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.184+0000 7f9be4a00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9be006cd60 con 0x7f9be0070890 2024-03-30T04:28:53.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.184+0000 7f9bdde00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9be0070890 0x7f9be006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:28:53.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.184+0000 7f9bdde00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9be0070890 0x7f9be006fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.123:52384/0 (socket says 172.21.15.123:52384) 2024-03-30T04:28:53.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.184+0000 7f9bdde00700 1 -- 172.21.15.123:0/550106508 learned_addr learned my addr 172.21.15.123:0/550106508 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:28:53.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.185+0000 7f9bdde00700 1 -- 172.21.15.123:0/550106508 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9be0071e40 msgr2=0x7f9be006c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:28:53.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.185+0000 7f9bdde00700 1 --2- 172.21.15.123:0/550106508 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9be0071e40 0x7f9be006c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:53.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.185+0000 7f9bdde00700 1 -- 172.21.15.123:0/550106508 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bcc005040 con 0x7f9be0070890 2024-03-30T04:28:53.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.185+0000 7f9bdde00700 1 --2- 172.21.15.123:0/550106508 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9be0070890 0x7f9be006fb80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9bd000b790 tx=0x7f9bd0009220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:28:53.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.186+0000 7f9bd6000700 1 -- 172.21.15.123:0/550106508 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9bd000f6f0 con 0x7f9be0070890 2024-03-30T04:28:53.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.186+0000 7f9be4a00700 1 -- 172.21.15.123:0/550106508 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9be006d040 con 0x7f9be0070890 2024-03-30T04:28:53.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.186+0000 7f9bd6000700 1 -- 172.21.15.123:0/550106508 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9bd000fd00 con 0x7f9be0070890 2024-03-30T04:28:53.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.186+0000 7f9bd6000700 1 -- 172.21.15.123:0/550106508 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9bd00103a0 con 0x7f9be0070890 2024-03-30T04:28:53.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.186+0000 7f9be4a00700 1 -- 172.21.15.123:0/550106508 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9be0122440 con 0x7f9be0070890 2024-03-30T04:28:53.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.187+0000 7f9be4a00700 1 -- 172.21.15.123:0/550106508 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bc00052e0 con 0x7f9be0070890 2024-03-30T04:28:53.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.188+0000 7f9bd6000700 1 -- 172.21.15.123:0/550106508 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 28) v1 ==== 45626+0+0 (secure 0 0 0) 0x7f9bd000f850 con 0x7f9be0070890 2024-03-30T04:28:53.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.189+0000 7f9bd6000700 1 --2- 172.21.15.123:0/550106508 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f9bc40385c0 0x7f9bc403aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:28:53.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.189+0000 7f9bdd400700 1 --2- 172.21.15.123:0/550106508 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f9bc40385c0 0x7f9bc403aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:28:53.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.190+0000 7f9bdd400700 1 --2- 172.21.15.123:0/550106508 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f9bc40385c0 0x7f9bc403aa70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc000c00 tx=0x7f9bcc009e30 comp rx=0 tx=0).ready entity=mgr.14696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:28:53.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.192+0000 7f9bd6000700 1 -- 172.21.15.123:0/550106508 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f9bd0018300 con 0x7f9be0070890 2024-03-30T04:28:53.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.199+0000 7f9bd6000700 1 -- 172.21.15.123:0/550106508 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f9bd0011400 con 0x7f9be0070890 2024-03-30T04:28:53.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.203+0000 7f9bd6000700 1 -- 172.21.15.123:0/550106508 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mgrmap(e 29) v1 ==== 45682+0+0 (secure 0 0 0) 0x7f9bd00115e0 con 0x7f9be0070890 2024-03-30T04:28:53.210 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:53 smithi123 ceph-mon[35245]: pgmap v4: 65 pgs: 65 active+clean; 1.2 GiB data, 5.4 GiB used, 531 GiB / 536 GiB avail 2024-03-30T04:28:53.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.494+0000 7f9be4a00700 1 -- 172.21.15.123:0/550106508 --> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9bc0000bf0 con 0x7f9bc40385c0 2024-03-30T04:28:53.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.495+0000 7f9bd6000700 1 -- 172.21.15.123:0/550106508 <== mgr.14696 v2:172.21.15.123:6800/2125774379 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+329 (secure 0 0 0) 0x7f9bc0000bf0 con 0x7f9bc40385c0 2024-03-30T04:28:53.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.498+0000 7f9be4a00700 1 -- 172.21.15.123:0/550106508 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f9bc40385c0 msgr2=0x7f9bc403aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:28:53.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.498+0000 7f9be4a00700 1 --2- 172.21.15.123:0/550106508 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f9bc40385c0 0x7f9bc403aa70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc000c00 tx=0x7f9bcc009e30 comp rx=0 tx=0).stop 2024-03-30T04:28:53.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.498+0000 7f9be4a00700 1 -- 172.21.15.123:0/550106508 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9be0070890 msgr2=0x7f9be006fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:28:53.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.498+0000 7f9be4a00700 1 --2- 172.21.15.123:0/550106508 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9be0070890 0x7f9be006fb80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9bd000b790 tx=0x7f9bd0009220 comp rx=0 tx=0).stop 2024-03-30T04:28:53.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.498+0000 7f9be4a00700 1 -- 172.21.15.123:0/550106508 shutdown_connections 2024-03-30T04:28:53.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.498+0000 7f9be4a00700 1 --2- 172.21.15.123:0/550106508 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f9bc40385c0 0x7f9bc403aa70 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:53.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.498+0000 7f9be4a00700 1 --2- 172.21.15.123:0/550106508 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9be0070890 0x7f9be006fb80 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:53.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.498+0000 7f9be4a00700 1 --2- 172.21.15.123:0/550106508 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9be0071e40 0x7f9be006c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:53.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.498+0000 7f9be4a00700 1 -- 172.21.15.123:0/550106508 >> 172.21.15.123:0/550106508 conn(0x7f9be00f8550 msgr2=0x7f9be00f9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:28:53.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.498+0000 7f9be4a00700 1 -- 172.21.15.123:0/550106508 shutdown_connections 2024-03-30T04:28:53.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.499+0000 7f9be4a00700 1 -- 172.21.15.123:0/550106508 wait complete. 2024-03-30T04:28:53.514 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:28:53.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:53 smithi135 ceph-mon[44364]: pgmap v4: 65 pgs: 65 active+clean; 1.2 GiB data, 5.4 GiB used, 531 GiB / 536 GiB avail 2024-03-30T04:28:53.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.663+0000 7fbbce600700 1 -- 172.21.15.123:0/2916399945 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbbc80fdee0 msgr2=0x7fbbc80fe350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:28:53.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.663+0000 7fbbce600700 1 --2- 172.21.15.123:0/2916399945 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbbc80fdee0 0x7fbbc80fe350 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4004940 tx=0x7fbbb4009e30 comp rx=0 tx=0).stop 2024-03-30T04:28:53.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.664+0000 7fbbce600700 1 -- 172.21.15.123:0/2916399945 shutdown_connections 2024-03-30T04:28:53.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.664+0000 7fbbce600700 1 --2- 172.21.15.123:0/2916399945 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbbc80fdee0 0x7fbbc80fe350 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:53.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.664+0000 7fbbce600700 1 --2- 172.21.15.123:0/2916399945 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbbc80fcc30 0x7fbbc80fd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:53.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.664+0000 7fbbce600700 1 -- 172.21.15.123:0/2916399945 >> 172.21.15.123:0/2916399945 conn(0x7fbbc80f83d0 msgr2=0x7fbbc80fa7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:28:53.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.664+0000 7fbbce600700 1 -- 172.21.15.123:0/2916399945 shutdown_connections 2024-03-30T04:28:53.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.664+0000 7fbbce600700 1 -- 172.21.15.123:0/2916399945 wait complete. 2024-03-30T04:28:53.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.665+0000 7fbbce600700 1 Processor -- start 2024-03-30T04:28:53.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.665+0000 7fbbce600700 1 -- start start 2024-03-30T04:28:53.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.665+0000 7fbbce600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbbc80fcc30 0x7fbbc806e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:28:53.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.665+0000 7fbbce600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbbc80fdee0 0x7fbbc806e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:28:53.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.665+0000 7fbbce600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc8070050 con 0x7fbbc80fdee0 2024-03-30T04:28:53.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.665+0000 7fbbce600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc806ee90 con 0x7fbbc80fcc30 2024-03-30T04:28:53.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.666+0000 7fbbc7e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbbc80fdee0 0x7fbbc806e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:28:53.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.666+0000 7fbbc7e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbbc80fdee0 0x7fbbc806e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54550/0 (socket says 172.21.15.123:54550) 2024-03-30T04:28:53.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.666+0000 7fbbc7e00700 1 -- 172.21.15.123:0/1741955277 learned_addr learned my addr 172.21.15.123:0/1741955277 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:28:53.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.666+0000 7fbbcd200700 1 --2- 172.21.15.123:0/1741955277 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbbc80fcc30 0x7fbbc806e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:28:53.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.666+0000 7fbbc7e00700 1 -- 172.21.15.123:0/1741955277 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbbc80fcc30 msgr2=0x7fbbc806e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:28:53.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.666+0000 7fbbc7e00700 1 --2- 172.21.15.123:0/1741955277 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbbc80fcc30 0x7fbbc806e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:53.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.666+0000 7fbbc7e00700 1 -- 172.21.15.123:0/1741955277 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbb4005040 con 0x7fbbc80fdee0 2024-03-30T04:28:53.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.667+0000 7fbbc7e00700 1 --2- 172.21.15.123:0/1741955277 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbbc80fdee0 0x7fbbc806e950 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fbbb40137b0 tx=0x7fbbb4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:28:53.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.667+0000 7fbbc4c00700 1 -- 172.21.15.123:0/1741955277 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbbb400e070 con 0x7fbbc80fdee0 2024-03-30T04:28:53.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.667+0000 7fbbce600700 1 -- 172.21.15.123:0/1741955277 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbc806f0b0 con 0x7fbbc80fdee0 2024-03-30T04:28:53.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.667+0000 7fbbc4c00700 1 -- 172.21.15.123:0/1741955277 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbbb401c400 con 0x7fbbc80fdee0 2024-03-30T04:28:53.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.667+0000 7fbbc4c00700 1 -- 172.21.15.123:0/1741955277 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbbb400b650 con 0x7fbbc80fdee0 2024-03-30T04:28:53.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.667+0000 7fbbce600700 1 -- 172.21.15.123:0/1741955277 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbc811bc20 con 0x7fbbc80fdee0 2024-03-30T04:28:53.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.668+0000 7fbbce600700 1 -- 172.21.15.123:0/1741955277 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbc805f5f0 con 0x7fbbc80fdee0 2024-03-30T04:28:53.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.669+0000 7fbbc4c00700 1 -- 172.21.15.123:0/1741955277 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 29) v1 ==== 45682+0+0 (secure 0 0 0) 0x7fbbb401c9e0 con 0x7fbbc80fdee0 2024-03-30T04:28:53.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.670+0000 7fbbc4c00700 1 --2- 172.21.15.123:0/1741955277 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fbbb0038640 0x7fbbb003aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:28:53.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.670+0000 7fbbc4c00700 1 -- 172.21.15.123:0/1741955277 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fbbb404d7d0 con 0x7fbbc80fdee0 2024-03-30T04:28:53.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.670+0000 7fbbcd200700 1 --2- 172.21.15.123:0/1741955277 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fbbb0038640 0x7fbbb003aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:28:53.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.671+0000 7fbbcd200700 1 --2- 172.21.15.123:0/1741955277 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fbbb0038640 0x7fbbb003aaf0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fbbc806fcd0 tx=0x7fbbbc00b040 comp rx=0 tx=0).ready entity=mgr.14696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:28:53.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.677+0000 7fbbc4c00700 1 -- 172.21.15.123:0/1741955277 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fbbb401c570 con 0x7fbbc80fdee0 2024-03-30T04:28:53.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.983+0000 7fbbce600700 1 -- 172.21.15.123:0/1741955277 --> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbbc8102210 con 0x7fbbb0038640 2024-03-30T04:28:53.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.985+0000 7fbbc4c00700 1 -- 172.21.15.123:0/1741955277 <== mgr.14696 v2:172.21.15.123:6800/2125774379 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+329 (secure 0 0 0) 0x7fbbc8102210 con 0x7fbbb0038640 2024-03-30T04:28:53.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.990+0000 7fbbce600700 1 -- 172.21.15.123:0/1741955277 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fbbb0038640 msgr2=0x7fbbb003aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:28:53.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.990+0000 7fbbce600700 1 --2- 172.21.15.123:0/1741955277 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fbbb0038640 0x7fbbb003aaf0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fbbc806fcd0 tx=0x7fbbbc00b040 comp rx=0 tx=0).stop 2024-03-30T04:28:53.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.990+0000 7fbbce600700 1 -- 172.21.15.123:0/1741955277 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbbc80fdee0 msgr2=0x7fbbc806e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:28:53.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.990+0000 7fbbce600700 1 --2- 172.21.15.123:0/1741955277 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbbc80fdee0 0x7fbbc806e950 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fbbb40137b0 tx=0x7fbbb4009e30 comp rx=0 tx=0).stop 2024-03-30T04:28:53.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.990+0000 7fbbce600700 1 -- 172.21.15.123:0/1741955277 shutdown_connections 2024-03-30T04:28:53.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.990+0000 7fbbce600700 1 --2- 172.21.15.123:0/1741955277 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fbbb0038640 0x7fbbb003aaf0 secure :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fbbc806fcd0 tx=0x7fbbbc00b040 comp rx=0 tx=0).stop 2024-03-30T04:28:53.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.990+0000 7fbbce600700 1 --2- 172.21.15.123:0/1741955277 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbbc80fcc30 0x7fbbc806e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:53.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.990+0000 7fbbce600700 1 --2- 172.21.15.123:0/1741955277 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbbc80fdee0 0x7fbbc806e950 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:53.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.990+0000 7fbbce600700 1 -- 172.21.15.123:0/1741955277 >> 172.21.15.123:0/1741955277 conn(0x7fbbc80f83d0 msgr2=0x7fbbc80f9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:28:53.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.991+0000 7fbbce600700 1 -- 172.21.15.123:0/1741955277 shutdown_connections 2024-03-30T04:28:53.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:53.991+0000 7fbbce600700 1 -- 172.21.15.123:0/1741955277 wait complete. 2024-03-30T04:28:54.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.192+0000 7f651a400700 1 -- 172.21.15.123:0/2286085565 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f65140fe0d0 msgr2=0x7f65140fe540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:28:54.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.192+0000 7f651a400700 1 --2- 172.21.15.123:0/2286085565 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f65140fe0d0 0x7f65140fe540 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f65000048b0 tx=0x7f6500009e30 comp rx=0 tx=0).stop 2024-03-30T04:28:54.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.192+0000 7f651a400700 1 -- 172.21.15.123:0/2286085565 shutdown_connections 2024-03-30T04:28:54.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.192+0000 7f651a400700 1 --2- 172.21.15.123:0/2286085565 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f65140fe0d0 0x7f65140fe540 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:54.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.192+0000 7f651a400700 1 --2- 172.21.15.123:0/2286085565 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65140fcee0 0x7f65140fd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:54.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.192+0000 7f651a400700 1 -- 172.21.15.123:0/2286085565 >> 172.21.15.123:0/2286085565 conn(0x7f65140f8560 msgr2=0x7f65140fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:28:54.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.192+0000 7f651a400700 1 -- 172.21.15.123:0/2286085565 shutdown_connections 2024-03-30T04:28:54.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.192+0000 7f651a400700 1 -- 172.21.15.123:0/2286085565 wait complete. 2024-03-30T04:28:54.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.193+0000 7f651a400700 1 Processor -- start 2024-03-30T04:28:54.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.193+0000 7f651a400700 1 -- start start 2024-03-30T04:28:54.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.194+0000 7f651a400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65140fcee0 0x7f651410fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:28:54.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.194+0000 7f651a400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f65140fe0d0 0x7f6514110060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:28:54.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.194+0000 7f651a400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6514110660 con 0x7f65140fcee0 2024-03-30T04:28:54.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.194+0000 7f651a400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6514110eb0 con 0x7f65140fe0d0 2024-03-30T04:28:54.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.194+0000 7f6512a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f65140fe0d0 0x7f6514110060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:28:54.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.194+0000 7f6512a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f65140fe0d0 0x7f6514110060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.123:59824/0 (socket says 172.21.15.123:59824) 2024-03-30T04:28:54.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.194+0000 7f6512a00700 1 -- 172.21.15.123:0/2853501249 learned_addr learned my addr 172.21.15.123:0/2853501249 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:28:54.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.195+0000 7f6512a00700 1 -- 172.21.15.123:0/2853501249 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65140fcee0 msgr2=0x7f651410fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:28:54.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.195+0000 7f6512a00700 1 --2- 172.21.15.123:0/2853501249 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65140fcee0 0x7f651410fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:54.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.195+0000 7f6512a00700 1 -- 172.21.15.123:0/2853501249 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6500005040 con 0x7f65140fe0d0 2024-03-30T04:28:54.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.195+0000 7f6512a00700 1 --2- 172.21.15.123:0/2853501249 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f65140fe0d0 0x7f6514110060 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6500009fd0 tx=0x7f6500009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:28:54.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.196+0000 7f650b400700 1 -- 172.21.15.123:0/2853501249 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f650000e040 con 0x7f65140fe0d0 2024-03-30T04:28:54.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.196+0000 7f650b400700 1 -- 172.21.15.123:0/2853501249 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6500014c30 con 0x7f65140fe0d0 2024-03-30T04:28:54.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.196+0000 7f651a400700 1 -- 172.21.15.123:0/2853501249 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6514111130 con 0x7f65140fe0d0 2024-03-30T04:28:54.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.196+0000 7f650b400700 1 -- 172.21.15.123:0/2853501249 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6500012600 con 0x7f65140fe0d0 2024-03-30T04:28:54.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.196+0000 7f651a400700 1 -- 172.21.15.123:0/2853501249 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6514111610 con 0x7f65140fe0d0 2024-03-30T04:28:54.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.197+0000 7f651a400700 1 -- 172.21.15.123:0/2853501249 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6514101a60 con 0x7f65140fe0d0 2024-03-30T04:28:54.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.198+0000 7f650b400700 1 -- 172.21.15.123:0/2853501249 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 29) v1 ==== 45682+0+0 (secure 0 0 0) 0x7f650001c970 con 0x7f65140fe0d0 2024-03-30T04:28:54.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.198+0000 7f650b400700 1 --2- 172.21.15.123:0/2853501249 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f64f8038650 0x7f64f803ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:28:54.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.198+0000 7f650b400700 1 -- 172.21.15.123:0/2853501249 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f650004d9e0 con 0x7f65140fe0d0 2024-03-30T04:28:54.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.199+0000 7f6513400700 1 --2- 172.21.15.123:0/2853501249 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f64f8038650 0x7f64f803ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:28:54.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.200+0000 7f6513400700 1 --2- 172.21.15.123:0/2853501249 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f64f8038650 0x7f64f803ab00 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f65140fcc20 tx=0x7f650400b040 comp rx=0 tx=0).ready entity=mgr.14696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:28:54.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.207+0000 7f650b400700 1 -- 172.21.15.123:0/2853501249 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f650001b390 con 0x7f65140fe0d0 2024-03-30T04:28:54.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:54 smithi123 ceph-mon[35245]: mgrmap e29: smithi123.xgksqq(active, since 3s) 2024-03-30T04:28:54.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:54 smithi123 ceph-mon[35245]: from='client.24525 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:28:54.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.474+0000 7f651a400700 1 -- 172.21.15.123:0/2853501249 --> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f65141118f0 con 0x7f64f8038650 2024-03-30T04:28:54.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.488+0000 7f650b400700 1 -- 172.21.15.123:0/2853501249 <== mgr.14696 v2:172.21.15.123:6800/2125774379 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f65141118f0 con 0x7f64f8038650 2024-03-30T04:28:54.488 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:28:54.488 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (79s) 11s ago 10m 13.7M - 0.25.0 c8568f914cd2 58af5a4ba60a 2024-03-30T04:28:54.488 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (10m) 11s ago 10m 6651k - 17.2.7-797-g5c024335 f77085f6aab7 bceb8b8c4837 2024-03-30T04:28:54.488 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (9m) 11s ago 9m 6651k - 17.2.7-797-g5c024335 f77085f6aab7 cd0fe67b2d34 2024-03-30T04:28:54.488 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (65s) 11s ago 10m 73.3M - 9.4.7 954c08fa6188 df1613fb18c2 2024-03-30T04:28:54.488 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (4m) 11s ago 4m 130M - 17.2.7-797-g5c024335 f77085f6aab7 2df066fba7a6 2024-03-30T04:28:54.489 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (4m) 11s ago 4m 76.5M - 17.2.7-797-g5c024335 f77085f6aab7 1293b8520db1 2024-03-30T04:28:54.489 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (4m) 11s ago 4m 90.2M - 17.2.7-797-g5c024335 f77085f6aab7 0dc41326818c 2024-03-30T04:28:54.489 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (4m) 11s ago 4m 148M - 17.2.7-797-g5c024335 f77085f6aab7 aef8a0cfc553 2024-03-30T04:28:54.489 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (21s) 11s ago 12m 77.2M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:28:54.489 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283 running (2m) 11s ago 9m 497M - 18.2.2-1018-g470100a4 e7c444a08cf7 20e5e0a05742 2024-03-30T04:28:54.489 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (12m) 11s ago 12m 48.9M 2048M 17.2.7-797-g5c024335 f77085f6aab7 51d18587a7df 2024-03-30T04:28:54.489 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (9m) 11s ago 9m 38.9M 2048M 17.2.7-797-g5c024335 f77085f6aab7 f81052b09dd8 2024-03-30T04:28:54.489 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (85s) 11s ago 10m 7805k - 1.5.0 0da6a335fe13 f4da2b8a3030 2024-03-30T04:28:54.489 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (33s) 11s ago 9m 6870k - 1.5.0 0da6a335fe13 abf949a26182 2024-03-30T04:28:54.489 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (8m) 11s ago 8m 210M 1618M 17.2.7-797-g5c024335 f77085f6aab7 57a7ba3fd75e 2024-03-30T04:28:54.489 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (8m) 11s ago 8m 221M 1618M 17.2.7-797-g5c024335 f77085f6aab7 e1a6c1818a0c 2024-03-30T04:28:54.490 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (7m) 11s ago 7m 175M 1618M 17.2.7-797-g5c024335 f77085f6aab7 4c6a727567bd 2024-03-30T04:28:54.490 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (6m) 11s ago 6m 251M 2642M 17.2.7-797-g5c024335 f77085f6aab7 6fbdd1d46a9d 2024-03-30T04:28:54.490 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (6m) 11s ago 6m 201M 2642M 17.2.7-797-g5c024335 f77085f6aab7 b74988e947e5 2024-03-30T04:28:54.490 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (5m) 11s ago 5m 244M 2642M 17.2.7-797-g5c024335 f77085f6aab7 1fd3185f5484 2024-03-30T04:28:54.490 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (44s) 11s ago 10m 42.3M - 2.43.0 a07b618ecd1d 2bb11d4d817e 2024-03-30T04:28:54.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.491+0000 7f651a400700 1 -- 172.21.15.123:0/2853501249 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f64f8038650 msgr2=0x7f64f803ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:28:54.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.491+0000 7f651a400700 1 --2- 172.21.15.123:0/2853501249 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f64f8038650 0x7f64f803ab00 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f65140fcc20 tx=0x7f650400b040 comp rx=0 tx=0).stop 2024-03-30T04:28:54.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.491+0000 7f651a400700 1 -- 172.21.15.123:0/2853501249 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f65140fe0d0 msgr2=0x7f6514110060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:28:54.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.491+0000 7f651a400700 1 --2- 172.21.15.123:0/2853501249 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f65140fe0d0 0x7f6514110060 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6500009fd0 tx=0x7f6500009e30 comp rx=0 tx=0).stop 2024-03-30T04:28:54.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.491+0000 7f651a400700 1 -- 172.21.15.123:0/2853501249 shutdown_connections 2024-03-30T04:28:54.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.491+0000 7f651a400700 1 --2- 172.21.15.123:0/2853501249 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65140fcee0 0x7f651410fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:54.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.491+0000 7f651a400700 1 --2- 172.21.15.123:0/2853501249 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f64f8038650 0x7f64f803ab00 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:54.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.491+0000 7f651a400700 1 --2- 172.21.15.123:0/2853501249 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f65140fe0d0 0x7f6514110060 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:54.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.491+0000 7f651a400700 1 -- 172.21.15.123:0/2853501249 >> 172.21.15.123:0/2853501249 conn(0x7f65140f8560 msgr2=0x7f65140f9c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:28:54.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.491+0000 7f651a400700 1 -- 172.21.15.123:0/2853501249 shutdown_connections 2024-03-30T04:28:54.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.491+0000 7f651a400700 1 -- 172.21.15.123:0/2853501249 wait complete. 2024-03-30T04:28:54.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:54 smithi135 ceph-mon[44364]: mgrmap e29: smithi123.xgksqq(active, since 3s) 2024-03-30T04:28:54.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:54 smithi135 ceph-mon[44364]: from='client.24525 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:28:54.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.654+0000 7fadb9200700 1 -- 172.21.15.123:0/44670279 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadb40709a0 msgr2=0x7fadb4070e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:28:54.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.654+0000 7fadb9200700 1 --2- 172.21.15.123:0/44670279 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadb40709a0 0x7fadb4070e10 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fada0004940 tx=0x7fada0009e30 comp rx=0 tx=0).stop 2024-03-30T04:28:54.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.655+0000 7fadb9200700 1 -- 172.21.15.123:0/44670279 shutdown_connections 2024-03-30T04:28:54.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.655+0000 7fadb9200700 1 --2- 172.21.15.123:0/44670279 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadb40709a0 0x7fadb4070e10 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:54.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.655+0000 7fadb9200700 1 --2- 172.21.15.123:0/44670279 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fadb4071f50 0x7fadb4070460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:54.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.655+0000 7fadb9200700 1 -- 172.21.15.123:0/44670279 >> 172.21.15.123:0/44670279 conn(0x7fadb40f8560 msgr2=0x7fadb40fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:28:54.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.655+0000 7fadb9200700 1 -- 172.21.15.123:0/44670279 shutdown_connections 2024-03-30T04:28:54.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.655+0000 7fadb9200700 1 -- 172.21.15.123:0/44670279 wait complete. 2024-03-30T04:28:54.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.656+0000 7fadb9200700 1 Processor -- start 2024-03-30T04:28:54.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.656+0000 7fadb9200700 1 -- start start 2024-03-30T04:28:54.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.656+0000 7fadb9200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fadb40709a0 0x7fadb410faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:28:54.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.656+0000 7fadb9200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadb4071f50 0x7fadb410ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:28:54.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.656+0000 7fadb9200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadb41105e0 con 0x7fadb4071f50 2024-03-30T04:28:54.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.656+0000 7fadb9200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadb4110e30 con 0x7fadb40709a0 2024-03-30T04:28:54.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.656+0000 7fadb1400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadb4071f50 0x7fadb410ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:28:54.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.657+0000 7fadb1400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadb4071f50 0x7fadb410ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53560/0 (socket says 172.21.15.123:53560) 2024-03-30T04:28:54.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.657+0000 7fadb1400700 1 -- 172.21.15.123:0/48758827 learned_addr learned my addr 172.21.15.123:0/48758827 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:28:54.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.657+0000 7fadb1e00700 1 --2- 172.21.15.123:0/48758827 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fadb40709a0 0x7fadb410faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:28:54.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.657+0000 7fadb1400700 1 -- 172.21.15.123:0/48758827 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fadb40709a0 msgr2=0x7fadb410faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:28:54.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.657+0000 7fadb1400700 1 --2- 172.21.15.123:0/48758827 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fadb40709a0 0x7fadb410faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:54.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.657+0000 7fadb1400700 1 -- 172.21.15.123:0/48758827 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada0005040 con 0x7fadb4071f50 2024-03-30T04:28:54.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.657+0000 7fadb1400700 1 --2- 172.21.15.123:0/48758827 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadb4071f50 0x7fadb410ffe0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fada000b0d0 tx=0x7fada0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:28:54.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.658+0000 7fadaa000700 1 -- 172.21.15.123:0/48758827 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fada000e070 con 0x7fadb4071f50 2024-03-30T04:28:54.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.658+0000 7fadaa000700 1 -- 172.21.15.123:0/48758827 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fada000b410 con 0x7fadb4071f50 2024-03-30T04:28:54.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.658+0000 7fadaa000700 1 -- 172.21.15.123:0/48758827 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fada0012600 con 0x7fadb4071f50 2024-03-30T04:28:54.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.658+0000 7fadb9200700 1 -- 172.21.15.123:0/48758827 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadb41110b0 con 0x7fadb4071f50 2024-03-30T04:28:54.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.658+0000 7fadb9200700 1 -- 172.21.15.123:0/48758827 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadb41115f0 con 0x7fadb4071f50 2024-03-30T04:28:54.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.659+0000 7fadb9200700 1 -- 172.21.15.123:0/48758827 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadb405f5f0 con 0x7fadb4071f50 2024-03-30T04:28:54.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.659+0000 7fadaa000700 1 -- 172.21.15.123:0/48758827 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 29) v1 ==== 45682+0+0 (secure 0 0 0) 0x7fada0013a50 con 0x7fadb4071f50 2024-03-30T04:28:54.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.659+0000 7fadaa000700 1 --2- 172.21.15.123:0/48758827 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fad98038650 0x7fad9803ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:28:54.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.660+0000 7fadaa000700 1 -- 172.21.15.123:0/48758827 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fada004cdc0 con 0x7fadb4071f50 2024-03-30T04:28:54.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.660+0000 7fadb1e00700 1 --2- 172.21.15.123:0/48758827 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fad98038650 0x7fad9803ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:28:54.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.661+0000 7fadb1e00700 1 --2- 172.21.15.123:0/48758827 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fad98038650 0x7fad9803ab00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fada4009b80 tx=0x7fada400b040 comp rx=0 tx=0).ready entity=mgr.14696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:28:54.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:54.666+0000 7fadaa000700 1 -- 172.21.15.123:0/48758827 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fada000b580 con 0x7fadb4071f50 2024-03-30T04:28:55.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.088+0000 7fadb9200700 1 -- 172.21.15.123:0/48758827 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fadb4107a40 con 0x7fadb4071f50 2024-03-30T04:28:55.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.089+0000 7fadaa000700 1 -- 172.21.15.123:0/48758827 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+743 (secure 0 0 0) 0x7fada0022020 con 0x7fadb4071f50 2024-03-30T04:28:55.088 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:28:55.088 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:28:55.088 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 2 2024-03-30T04:28:55.088 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:28:55.088 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:28:55.088 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 1 2024-03-30T04:28:55.088 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:28:55.088 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:28:55.089 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 6 2024-03-30T04:28:55.089 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:28:55.089 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:28:55.089 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 4 2024-03-30T04:28:55.089 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:28:55.089 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:28:55.089 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 12, 2024-03-30T04:28:55.089 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 1 2024-03-30T04:28:55.089 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:28:55.089 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:28:55.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.092+0000 7fadb9200700 1 -- 172.21.15.123:0/48758827 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fad98038650 msgr2=0x7fad9803ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:28:55.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.092+0000 7fadb9200700 1 --2- 172.21.15.123:0/48758827 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fad98038650 0x7fad9803ab00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fada4009b80 tx=0x7fada400b040 comp rx=0 tx=0).stop 2024-03-30T04:28:55.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.092+0000 7fadb9200700 1 -- 172.21.15.123:0/48758827 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadb4071f50 msgr2=0x7fadb410ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:28:55.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.092+0000 7fadb9200700 1 --2- 172.21.15.123:0/48758827 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadb4071f50 0x7fadb410ffe0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fada000b0d0 tx=0x7fada0009e30 comp rx=0 tx=0).stop 2024-03-30T04:28:55.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.093+0000 7fadb9200700 1 -- 172.21.15.123:0/48758827 shutdown_connections 2024-03-30T04:28:55.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.093+0000 7fadb9200700 1 --2- 172.21.15.123:0/48758827 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fad98038650 0x7fad9803ab00 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:55.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.093+0000 7fadb9200700 1 --2- 172.21.15.123:0/48758827 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fadb40709a0 0x7fadb410faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:55.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.093+0000 7fadb9200700 1 --2- 172.21.15.123:0/48758827 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadb4071f50 0x7fadb410ffe0 secure :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fada000b0d0 tx=0x7fada0009e30 comp rx=0 tx=0).stop 2024-03-30T04:28:55.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.093+0000 7fadb9200700 1 -- 172.21.15.123:0/48758827 >> 172.21.15.123:0/48758827 conn(0x7fadb40f8560 msgr2=0x7fadb40f9c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:28:55.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.093+0000 7fadb9200700 1 -- 172.21.15.123:0/48758827 shutdown_connections 2024-03-30T04:28:55.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.093+0000 7fadb9200700 1 -- 172.21.15.123:0/48758827 wait complete. 2024-03-30T04:28:55.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.281+0000 7fe385200700 1 -- 172.21.15.123:0/491565361 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe380102140 msgr2=0x7fe380104520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:28:55.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.281+0000 7fe385200700 1 --2- 172.21.15.123:0/491565361 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe380102140 0x7fe380104520 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fe36c004940 tx=0x7fe36c009e30 comp rx=0 tx=0).stop 2024-03-30T04:28:55.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.281+0000 7fe385200700 1 -- 172.21.15.123:0/491565361 shutdown_connections 2024-03-30T04:28:55.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.281+0000 7fe385200700 1 --2- 172.21.15.123:0/491565361 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe380102140 0x7fe380104520 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:55.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.281+0000 7fe385200700 1 --2- 172.21.15.123:0/491565361 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe3800662e0 0x7fe380101c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:55.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.281+0000 7fe385200700 1 -- 172.21.15.123:0/491565361 >> 172.21.15.123:0/491565361 conn(0x7fe3800f7270 msgr2=0x7fe3800f96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:28:55.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.281+0000 7fe385200700 1 -- 172.21.15.123:0/491565361 shutdown_connections 2024-03-30T04:28:55.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.281+0000 7fe385200700 1 -- 172.21.15.123:0/491565361 wait complete. 2024-03-30T04:28:55.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.282+0000 7fe385200700 1 Processor -- start 2024-03-30T04:28:55.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.282+0000 7fe385200700 1 -- start start 2024-03-30T04:28:55.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.283+0000 7fe385200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3800662e0 0x7fe38010b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:28:55.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.283+0000 7fe385200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe380102140 0x7fe38010b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:28:55.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.283+0000 7fe385200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe38010c030 con 0x7fe3800662e0 2024-03-30T04:28:55.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.283+0000 7fe385200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe38010c880 con 0x7fe380102140 2024-03-30T04:28:55.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.283+0000 7fe37f400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3800662e0 0x7fe38010b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:28:55.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.283+0000 7fe37f400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3800662e0 0x7fe38010b4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53572/0 (socket says 172.21.15.123:53572) 2024-03-30T04:28:55.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.283+0000 7fe37f400700 1 -- 172.21.15.123:0/2481317794 learned_addr learned my addr 172.21.15.123:0/2481317794 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:28:55.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.283+0000 7fe37ea00700 1 --2- 172.21.15.123:0/2481317794 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe380102140 0x7fe38010b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:28:55.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.284+0000 7fe37f400700 1 -- 172.21.15.123:0/2481317794 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe380102140 msgr2=0x7fe38010b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:28:55.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.284+0000 7fe37f400700 1 --2- 172.21.15.123:0/2481317794 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe380102140 0x7fe38010b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:55.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.284+0000 7fe37f400700 1 -- 172.21.15.123:0/2481317794 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe36c005040 con 0x7fe3800662e0 2024-03-30T04:28:55.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.285+0000 7fe37f400700 1 --2- 172.21.15.123:0/2481317794 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3800662e0 0x7fe38010b4a0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fe37000eb40 tx=0x7fe37000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:28:55.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.285+0000 7fe377400700 1 -- 172.21.15.123:0/2481317794 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe370007cc0 con 0x7fe3800662e0 2024-03-30T04:28:55.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.285+0000 7fe385200700 1 -- 172.21.15.123:0/2481317794 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe38010cb00 con 0x7fe3800662e0 2024-03-30T04:28:55.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.285+0000 7fe385200700 1 -- 172.21.15.123:0/2481317794 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe38010d0c0 con 0x7fe3800662e0 2024-03-30T04:28:55.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.285+0000 7fe377400700 1 -- 172.21.15.123:0/2481317794 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe37000f040 con 0x7fe3800662e0 2024-03-30T04:28:55.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.285+0000 7fe377400700 1 -- 172.21.15.123:0/2481317794 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe37000c8f0 con 0x7fe3800662e0 2024-03-30T04:28:55.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.286+0000 7fe385200700 1 -- 172.21.15.123:0/2481317794 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3600052e0 con 0x7fe3800662e0 2024-03-30T04:28:55.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.287+0000 7fe377400700 1 -- 172.21.15.123:0/2481317794 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 29) v1 ==== 45682+0+0 (secure 0 0 0) 0x7fe37000ca50 con 0x7fe3800662e0 2024-03-30T04:28:55.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.289+0000 7fe377400700 1 --2- 172.21.15.123:0/2481317794 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fe364038640 0x7fe36403aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:28:55.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.289+0000 7fe377400700 1 -- 172.21.15.123:0/2481317794 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fe370009110 con 0x7fe3800662e0 2024-03-30T04:28:55.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.289+0000 7fe37ea00700 1 --2- 172.21.15.123:0/2481317794 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fe364038640 0x7fe36403aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:28:55.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.290+0000 7fe37ea00700 1 --2- 172.21.15.123:0/2481317794 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fe364038640 0x7fe36403aaf0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe36c00b0d0 tx=0x7fe36c009e30 comp rx=0 tx=0).ready entity=mgr.14696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:28:55.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.295+0000 7fe377400700 1 -- 172.21.15.123:0/2481317794 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fe37001b2c0 con 0x7fe3800662e0 2024-03-30T04:28:55.575 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-30T04:28:55.575 DEBUG:teuthology.orchestra.run.smithi135:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1/tmp 2024-03-30T04:28:55.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.589+0000 7fe385200700 1 -- 172.21.15.123:0/2481317794 --> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe360000bf0 con 0x7fe364038640 2024-03-30T04:28:55.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.590+0000 7fe377400700 1 -- 172.21.15.123:0/2481317794 <== mgr.14696 v2:172.21.15.123:6800/2125774379 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+329 (secure 0 0 0) 0x7fe360000bf0 con 0x7fe364038640 2024-03-30T04:28:55.589 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:28:55.589 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:28:55.589 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:28:55.589 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-30T04:28:55.589 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:28:55.589 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr" 2024-03-30T04:28:55.589 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:28:55.589 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "2/2 daemons upgraded", 2024-03-30T04:28:55.590 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "", 2024-03-30T04:28:55.590 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:28:55.590 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:28:55.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.593+0000 7fe385200700 1 -- 172.21.15.123:0/2481317794 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fe364038640 msgr2=0x7fe36403aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:28:55.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.593+0000 7fe385200700 1 --2- 172.21.15.123:0/2481317794 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fe364038640 0x7fe36403aaf0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe36c00b0d0 tx=0x7fe36c009e30 comp rx=0 tx=0).stop 2024-03-30T04:28:55.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.593+0000 7fe385200700 1 -- 172.21.15.123:0/2481317794 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3800662e0 msgr2=0x7fe38010b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:28:55.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.593+0000 7fe385200700 1 --2- 172.21.15.123:0/2481317794 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3800662e0 0x7fe38010b4a0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fe37000eb40 tx=0x7fe37000b040 comp rx=0 tx=0).stop 2024-03-30T04:28:55.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.593+0000 7fe385200700 1 -- 172.21.15.123:0/2481317794 shutdown_connections 2024-03-30T04:28:55.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.593+0000 7fe385200700 1 --2- 172.21.15.123:0/2481317794 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3800662e0 0x7fe38010b4a0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:55.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.593+0000 7fe385200700 1 --2- 172.21.15.123:0/2481317794 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fe364038640 0x7fe36403aaf0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:55.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.593+0000 7fe385200700 1 --2- 172.21.15.123:0/2481317794 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe380102140 0x7fe38010b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:28:55.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.593+0000 7fe385200700 1 -- 172.21.15.123:0/2481317794 >> 172.21.15.123:0/2481317794 conn(0x7fe3800f7270 msgr2=0x7fe3800f7690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:28:55.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.593+0000 7fe385200700 1 -- 172.21.15.123:0/2481317794 shutdown_connections 2024-03-30T04:28:55.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:28:55.593+0000 7fe385200700 1 -- 172.21.15.123:0/2481317794 wait complete. 2024-03-30T04:28:55.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:55 smithi135 ceph-mon[44364]: pgmap v5: 65 pgs: 65 active+clean; 1.2 GiB data, 5.4 GiB used, 531 GiB / 536 GiB avail 2024-03-30T04:28:55.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:55 smithi135 ceph-mon[44364]: from='client.14722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:28:55.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:55 smithi135 ceph-mon[44364]: from='client.24533 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:28:55.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:55 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:28:55.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:55 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:28:55.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:55 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/48758827' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:28:55.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:55 smithi123 ceph-mon[35245]: pgmap v5: 65 pgs: 65 active+clean; 1.2 GiB data, 5.4 GiB used, 531 GiB / 536 GiB avail 2024-03-30T04:28:55.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:55 smithi123 ceph-mon[35245]: from='client.14722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:28:55.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:55 smithi123 ceph-mon[35245]: from='client.24533 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:28:55.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:55 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:28:55.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:55 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:28:55.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:55 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/48758827' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:28:56.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:56 smithi135 ceph-mon[44364]: from='client.14732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:28:56.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:56 smithi123 ceph-mon[35245]: from='client.14732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:28:57.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:57 smithi135 ceph-mon[44364]: pgmap v6: 65 pgs: 65 active+clean; 1.2 GiB data, 5.4 GiB used, 531 GiB / 536 GiB avail 2024-03-30T04:28:57.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:57 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:28:57.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:57 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:28:57.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:57 smithi123 ceph-mon[35245]: pgmap v6: 65 pgs: 65 active+clean; 1.2 GiB data, 5.4 GiB used, 531 GiB / 536 GiB avail 2024-03-30T04:28:57.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:57 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:28:57.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:57 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:28:59.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:28:59 smithi135 ceph-mon[44364]: pgmap v7: 65 pgs: 65 active+clean; 569 MiB data, 3.6 GiB used, 533 GiB / 536 GiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 266 op/s 2024-03-30T04:28:59.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:28:59 smithi123 ceph-mon[35245]: pgmap v7: 65 pgs: 65 active+clean; 569 MiB data, 3.6 GiB used, 533 GiB / 536 GiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 266 op/s 2024-03-30T04:29:01.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:01 smithi135 ceph-mon[44364]: pgmap v8: 65 pgs: 65 active+clean; 569 MiB data, 3.6 GiB used, 533 GiB / 536 GiB avail; 891 KiB/s rd, 848 KiB/s wr, 207 op/s 2024-03-30T04:29:01.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:01 smithi123 ceph-mon[35245]: pgmap v8: 65 pgs: 65 active+clean; 569 MiB data, 3.6 GiB used, 533 GiB / 536 GiB avail; 891 KiB/s rd, 848 KiB/s wr, 207 op/s 2024-03-30T04:29:02.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:02 smithi123 ceph-mon[35245]: Standby manager daemon smithi135.kyblli started 2024-03-30T04:29:02.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:02 smithi123 ceph-mon[35245]: from='mgr.? 172.21.15.135:0/130318549' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi135.kyblli/crt"}]: dispatch 2024-03-30T04:29:02.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:02 smithi123 ceph-mon[35245]: from='mgr.? 172.21.15.135:0/130318549' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T04:29:02.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:02 smithi123 ceph-mon[35245]: from='mgr.? 172.21.15.135:0/130318549' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi135.kyblli/key"}]: dispatch 2024-03-30T04:29:02.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:02 smithi123 ceph-mon[35245]: from='mgr.? 172.21.15.135:0/130318549' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T04:29:02.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:02 smithi135 ceph-mon[44364]: Standby manager daemon smithi135.kyblli started 2024-03-30T04:29:02.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:02 smithi135 ceph-mon[44364]: from='mgr.? 172.21.15.135:0/130318549' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi135.kyblli/crt"}]: dispatch 2024-03-30T04:29:02.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:02 smithi135 ceph-mon[44364]: from='mgr.? 172.21.15.135:0/130318549' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T04:29:02.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:02 smithi135 ceph-mon[44364]: from='mgr.? 172.21.15.135:0/130318549' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi135.kyblli/key"}]: dispatch 2024-03-30T04:29:02.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:02 smithi135 ceph-mon[44364]: from='mgr.? 172.21.15.135:0/130318549' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T04:29:03.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:03 smithi135 ceph-mon[44364]: pgmap v9: 65 pgs: 65 active+clean; 288 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 373 op/s 2024-03-30T04:29:03.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:03 smithi135 ceph-mon[44364]: mgrmap e30: smithi123.xgksqq(active, since 12s), standbys: smithi135.kyblli 2024-03-30T04:29:03.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:03 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mgr metadata", "who": "smithi135.kyblli", "id": "smithi135.kyblli"}]: dispatch 2024-03-30T04:29:03.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:03 smithi123 ceph-mon[35245]: pgmap v9: 65 pgs: 65 active+clean; 288 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 373 op/s 2024-03-30T04:29:03.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:03 smithi123 ceph-mon[35245]: mgrmap e30: smithi123.xgksqq(active, since 12s), standbys: smithi135.kyblli 2024-03-30T04:29:03.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:03 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mgr metadata", "who": "smithi135.kyblli", "id": "smithi135.kyblli"}]: dispatch 2024-03-30T04:29:05.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:05 smithi135 ceph-mon[44364]: pgmap v10: 65 pgs: 65 active+clean; 288 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 342 op/s 2024-03-30T04:29:05.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:05 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:29:05.869 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:05 smithi123 ceph-mon[35245]: pgmap v10: 65 pgs: 65 active+clean; 288 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 342 op/s 2024-03-30T04:29:05.869 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:05 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:29:07.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:07 smithi135 ceph-mon[44364]: pgmap v11: 65 pgs: 65 active+clean; 288 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 342 op/s 2024-03-30T04:29:07.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:07 smithi123 ceph-mon[35245]: pgmap v11: 65 pgs: 65 active+clean; 288 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 342 op/s 2024-03-30T04:29:09.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:08 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:29:09.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:08 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:29:09.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:08 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:29:09.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:08 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:29:10.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:09 smithi135 ceph-mon[44364]: pgmap v12: 65 pgs: 65 active+clean; 289 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 540 op/s 2024-03-30T04:29:10.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:09 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:29:10.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:09 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:29:10.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:09 smithi123 ceph-mon[35245]: pgmap v12: 65 pgs: 65 active+clean; 289 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 540 op/s 2024-03-30T04:29:10.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:09 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:29:10.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:09 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:29:12.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:11 smithi135 ceph-mon[44364]: pgmap v13: 65 pgs: 65 active+clean; 289 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 385 op/s 2024-03-30T04:29:12.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:11 smithi123 ceph-mon[35245]: pgmap v13: 65 pgs: 65 active+clean; 289 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 385 op/s 2024-03-30T04:29:14.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:13 smithi135 ceph-mon[44364]: pgmap v14: 65 pgs: 65 active+clean; 285 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 2.2 MiB/s rd, 2.3 MiB/s wr, 594 op/s 2024-03-30T04:29:14.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:13 smithi123 ceph-mon[35245]: pgmap v14: 65 pgs: 65 active+clean; 285 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 2.2 MiB/s rd, 2.3 MiB/s wr, 594 op/s 2024-03-30T04:29:16.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:15 smithi135 ceph-mon[44364]: pgmap v15: 65 pgs: 65 active+clean; 285 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 407 op/s 2024-03-30T04:29:16.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:15 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:29:16.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:15 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:29:16.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:15 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:29:16.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:15 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:29:16.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:15 smithi123 ceph-mon[35245]: pgmap v15: 65 pgs: 65 active+clean; 285 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 407 op/s 2024-03-30T04:29:16.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:15 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:29:16.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:15 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:29:16.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:15 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:29:16.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:15 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:29:16.776 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:16 smithi135 ceph-mon[44364]: Updating smithi123:/etc/ceph/ceph.conf 2024-03-30T04:29:16.776 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:16 smithi135 ceph-mon[44364]: Updating smithi135:/etc/ceph/ceph.conf 2024-03-30T04:29:16.776 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:16 smithi135 ceph-mon[44364]: Updating smithi123:/var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T04:29:16.776 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:16 smithi135 ceph-mon[44364]: Updating smithi135:/var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T04:29:17.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:16 smithi123 ceph-mon[35245]: Updating smithi123:/etc/ceph/ceph.conf 2024-03-30T04:29:17.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:16 smithi123 ceph-mon[35245]: Updating smithi135:/etc/ceph/ceph.conf 2024-03-30T04:29:17.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:16 smithi123 ceph-mon[35245]: Updating smithi123:/var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T04:29:17.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:16 smithi123 ceph-mon[35245]: Updating smithi135:/var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T04:29:18.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:17 smithi135 ceph-mon[44364]: pgmap v16: 65 pgs: 65 active+clean; 283 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 502 op/s 2024-03-30T04:29:18.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:17 smithi135 ceph-mon[44364]: Updating smithi135:/etc/ceph/ceph.client.admin.keyring 2024-03-30T04:29:18.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:17 smithi135 ceph-mon[44364]: Updating smithi123:/etc/ceph/ceph.client.admin.keyring 2024-03-30T04:29:18.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:17 smithi135 ceph-mon[44364]: Updating smithi123:/var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T04:29:18.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:17 smithi135 ceph-mon[44364]: Updating smithi135:/var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T04:29:18.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:17 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:29:18.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:17 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:29:18.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:17 smithi123 ceph-mon[35245]: pgmap v16: 65 pgs: 65 active+clean; 283 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 502 op/s 2024-03-30T04:29:18.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:17 smithi123 ceph-mon[35245]: Updating smithi135:/etc/ceph/ceph.client.admin.keyring 2024-03-30T04:29:18.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:17 smithi123 ceph-mon[35245]: Updating smithi123:/etc/ceph/ceph.client.admin.keyring 2024-03-30T04:29:18.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:17 smithi123 ceph-mon[35245]: Updating smithi123:/var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T04:29:18.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:17 smithi123 ceph-mon[35245]: Updating smithi135:/var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T04:29:18.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:17 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:29:18.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:17 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:29:19.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:18 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:29:19.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:18 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:29:19.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:18 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:29:19.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:18 smithi135 ceph-mon[44364]: pgmap v17: 65 pgs: 65 active+clean; 285 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 642 op/s 2024-03-30T04:29:19.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:18 smithi135 ceph-mon[44364]: Reconfiguring prometheus.smithi123 (dependencies changed)... 2024-03-30T04:29:19.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:18 smithi135 ceph-mon[44364]: Reconfiguring daemon prometheus.smithi123 on smithi123 2024-03-30T04:29:19.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:18 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:29:19.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:18 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:29:19.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:18 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:29:19.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:18 smithi123 ceph-mon[35245]: pgmap v17: 65 pgs: 65 active+clean; 285 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 642 op/s 2024-03-30T04:29:19.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:18 smithi123 ceph-mon[35245]: Reconfiguring prometheus.smithi123 (dependencies changed)... 2024-03-30T04:29:19.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:18 smithi123 ceph-mon[35245]: Reconfiguring daemon prometheus.smithi123 on smithi123 2024-03-30T04:29:21.154 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:20 smithi123 ceph-mon[35245]: pgmap v18: 65 pgs: 65 active+clean; 285 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 443 op/s 2024-03-30T04:29:21.154 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:20 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:29:21.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:20 smithi135 ceph-mon[44364]: pgmap v18: 65 pgs: 65 active+clean; 285 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 443 op/s 2024-03-30T04:29:21.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:20 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:29:23.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:22 smithi123 ceph-mon[35245]: pgmap v19: 65 pgs: 65 active+clean; 276 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 651 op/s 2024-03-30T04:29:23.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:22 smithi135 ceph-mon[44364]: pgmap v19: 65 pgs: 65 active+clean; 276 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 651 op/s 2024-03-30T04:29:25.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:24 smithi123 ceph-mon[35245]: pgmap v20: 65 pgs: 65 active+clean; 276 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 442 op/s 2024-03-30T04:29:25.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:24 smithi135 ceph-mon[44364]: pgmap v20: 65 pgs: 65 active+clean; 276 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 442 op/s 2024-03-30T04:29:25.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:25.750+0000 7f2fc4c00700 1 -- 172.21.15.123:0/693055203 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fc00fe0d0 msgr2=0x7f2fc00fe540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:29:25.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:25.750+0000 7f2fc4c00700 1 --2- 172.21.15.123:0/693055203 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fc00fe0d0 0x7f2fc00fe540 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f2fac004940 tx=0x7f2fac009e30 comp rx=0 tx=0).stop 2024-03-30T04:29:25.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:25.750+0000 7f2fc4c00700 1 -- 172.21.15.123:0/693055203 shutdown_connections 2024-03-30T04:29:25.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:25.750+0000 7f2fc4c00700 1 --2- 172.21.15.123:0/693055203 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fc00fe0d0 0x7f2fc00fe540 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:25.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:25.750+0000 7f2fc4c00700 1 --2- 172.21.15.123:0/693055203 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2fc00fcee0 0x7f2fc00fd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:25.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:25.750+0000 7f2fc4c00700 1 -- 172.21.15.123:0/693055203 >> 172.21.15.123:0/693055203 conn(0x7f2fc00f8560 msgr2=0x7f2fc00fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:29:25.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:25.751+0000 7f2fc4c00700 1 -- 172.21.15.123:0/693055203 shutdown_connections 2024-03-30T04:29:25.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:25.751+0000 7f2fc4c00700 1 -- 172.21.15.123:0/693055203 wait complete. 2024-03-30T04:29:25.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:25.751+0000 7f2fc4c00700 1 Processor -- start 2024-03-30T04:29:25.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:25.751+0000 7f2fc4c00700 1 -- start start 2024-03-30T04:29:25.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:25.752+0000 7f2fc4c00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2fc00fcee0 0x7f2fc0071e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:29:25.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:25.752+0000 7f2fc4c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fc00fe0d0 0x7f2fc0070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:29:25.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:25.752+0000 7f2fc4c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc00708c0 con 0x7f2fc00fe0d0 2024-03-30T04:29:25.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:25.752+0000 7f2fc4c00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc0070a30 con 0x7f2fc00fcee0 2024-03-30T04:29:25.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:25.752+0000 7f2fbd400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fc00fe0d0 0x7f2fc0070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:29:25.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:25.752+0000 7f2fbd400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fc00fe0d0 0x7f2fc0070310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44130/0 (socket says 172.21.15.123:44130) 2024-03-30T04:29:25.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:25.752+0000 7f2fbd400700 1 -- 172.21.15.123:0/2322578722 learned_addr learned my addr 172.21.15.123:0/2322578722 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:29:25.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:25.752+0000 7f2fbde00700 1 --2- 172.21.15.123:0/2322578722 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2fc00fcee0 0x7f2fc0071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:29:25.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:25.752+0000 7f2fbd400700 1 -- 172.21.15.123:0/2322578722 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2fc00fcee0 msgr2=0x7f2fc0071e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:29:25.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:25.752+0000 7f2fbd400700 1 --2- 172.21.15.123:0/2322578722 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2fc00fcee0 0x7f2fc0071e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:25.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:25.752+0000 7f2fbd400700 1 -- 172.21.15.123:0/2322578722 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fac005040 con 0x7f2fc00fe0d0 2024-03-30T04:29:25.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:25.753+0000 7f2fbd400700 1 --2- 172.21.15.123:0/2322578722 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fc00fe0d0 0x7f2fc0070310 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f2fac013870 tx=0x7f2fac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:29:25.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:25.753+0000 7f2fb6000700 1 -- 172.21.15.123:0/2322578722 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2fac021070 con 0x7f2fc00fe0d0 2024-03-30T04:29:25.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:25.753+0000 7f2fc4c00700 1 -- 172.21.15.123:0/2322578722 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fc0070cb0 con 0x7f2fc00fe0d0 2024-03-30T04:29:25.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:25.753+0000 7f2fb6000700 1 -- 172.21.15.123:0/2322578722 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2fac013dc0 con 0x7f2fc00fe0d0 2024-03-30T04:29:25.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:25.753+0000 7f2fb6000700 1 -- 172.21.15.123:0/2322578722 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2fac00e070 con 0x7f2fc00fe0d0 2024-03-30T04:29:25.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:25.753+0000 7f2fc4c00700 1 -- 172.21.15.123:0/2322578722 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fc01159a0 con 0x7f2fc00fe0d0 2024-03-30T04:29:25.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:25.754+0000 7f2fc4c00700 1 -- 172.21.15.123:0/2322578722 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fc005f5f0 con 0x7f2fc00fe0d0 2024-03-30T04:29:25.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:25.755+0000 7f2fb6000700 1 -- 172.21.15.123:0/2322578722 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 30) v1 ==== 90807+0+0 (secure 0 0 0) 0x7f2fac01c600 con 0x7f2fc00fe0d0 2024-03-30T04:29:25.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:25.756+0000 7f2fb6000700 1 --2- 172.21.15.123:0/2322578722 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f2fa406c9c0 0x7f2fa406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:29:25.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:25.756+0000 7f2fb6000700 1 -- 172.21.15.123:0/2322578722 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f2fac08cf70 con 0x7f2fc00fe0d0 2024-03-30T04:29:25.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:25.756+0000 7f2fbde00700 1 --2- 172.21.15.123:0/2322578722 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f2fa406c9c0 0x7f2fa406ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:29:25.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:25.757+0000 7f2fbde00700 1 --2- 172.21.15.123:0/2322578722 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f2fa406c9c0 0x7f2fa406ee70 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2fc0071bd0 tx=0x7f2fb000b040 comp rx=0 tx=0).ready entity=mgr.14696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:29:25.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:25.761+0000 7f2fb6000700 1 -- 172.21.15.123:0/2322578722 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f2fac057210 con 0x7f2fc00fe0d0 2024-03-30T04:29:26.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.005+0000 7f2fc4c00700 1 -- 172.21.15.123:0/2322578722 --> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2fc0102570 con 0x7f2fa406c9c0 2024-03-30T04:29:26.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.006+0000 7f2fb6000700 1 -- 172.21.15.123:0/2322578722 <== mgr.14696 v2:172.21.15.123:6800/2125774379 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f2fc0102570 con 0x7f2fa406c9c0 2024-03-30T04:29:26.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.009+0000 7f2fc4c00700 1 -- 172.21.15.123:0/2322578722 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f2fa406c9c0 msgr2=0x7f2fa406ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:29:26.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.009+0000 7f2fc4c00700 1 --2- 172.21.15.123:0/2322578722 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f2fa406c9c0 0x7f2fa406ee70 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2fc0071bd0 tx=0x7f2fb000b040 comp rx=0 tx=0).stop 2024-03-30T04:29:26.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.009+0000 7f2fc4c00700 1 -- 172.21.15.123:0/2322578722 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fc00fe0d0 msgr2=0x7f2fc0070310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:29:26.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.009+0000 7f2fc4c00700 1 --2- 172.21.15.123:0/2322578722 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fc00fe0d0 0x7f2fc0070310 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f2fac013870 tx=0x7f2fac009e30 comp rx=0 tx=0).stop 2024-03-30T04:29:26.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.009+0000 7f2fc4c00700 1 -- 172.21.15.123:0/2322578722 shutdown_connections 2024-03-30T04:29:26.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.009+0000 7f2fc4c00700 1 --2- 172.21.15.123:0/2322578722 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f2fa406c9c0 0x7f2fa406ee70 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:26.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.009+0000 7f2fc4c00700 1 --2- 172.21.15.123:0/2322578722 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2fc00fcee0 0x7f2fc0071e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:26.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.009+0000 7f2fc4c00700 1 --2- 172.21.15.123:0/2322578722 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fc00fe0d0 0x7f2fc0070310 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:26.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.009+0000 7f2fc4c00700 1 -- 172.21.15.123:0/2322578722 >> 172.21.15.123:0/2322578722 conn(0x7f2fc00f8560 msgr2=0x7f2fc00f9d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:29:26.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.009+0000 7f2fc4c00700 1 -- 172.21.15.123:0/2322578722 shutdown_connections 2024-03-30T04:29:26.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.009+0000 7f2fc4c00700 1 -- 172.21.15.123:0/2322578722 wait complete. 2024-03-30T04:29:26.023 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:29:26.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.153+0000 7f4034400700 1 -- 172.21.15.123:0/127253457 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f402c0fcee0 msgr2=0x7f402c0fd310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:29:26.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.153+0000 7f4034400700 1 --2- 172.21.15.123:0/127253457 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f402c0fcee0 0x7f402c0fd310 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f4020004940 tx=0x7f4020009e30 comp rx=0 tx=0).stop 2024-03-30T04:29:26.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.153+0000 7f4034400700 1 -- 172.21.15.123:0/127253457 shutdown_connections 2024-03-30T04:29:26.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.153+0000 7f4034400700 1 --2- 172.21.15.123:0/127253457 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f402c0fe0d0 0x7f402c0fe540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:26.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.153+0000 7f4034400700 1 --2- 172.21.15.123:0/127253457 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f402c0fcee0 0x7f402c0fd310 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:26.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.153+0000 7f4034400700 1 -- 172.21.15.123:0/127253457 >> 172.21.15.123:0/127253457 conn(0x7f402c0f8560 msgr2=0x7f402c0fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:29:26.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.153+0000 7f4034400700 1 -- 172.21.15.123:0/127253457 shutdown_connections 2024-03-30T04:29:26.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.153+0000 7f4034400700 1 -- 172.21.15.123:0/127253457 wait complete. 2024-03-30T04:29:26.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.154+0000 7f4034400700 1 Processor -- start 2024-03-30T04:29:26.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.154+0000 7f4034400700 1 -- start start 2024-03-30T04:29:26.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.155+0000 7f4034400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f402c0fcee0 0x7f402c071e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:29:26.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.155+0000 7f4034400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f402c0fe0d0 0x7f402c070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:29:26.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.155+0000 7f4034400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f402c0708c0 con 0x7f402c0fe0d0 2024-03-30T04:29:26.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.155+0000 7f4034400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f402c070a30 con 0x7f402c0fcee0 2024-03-30T04:29:26.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.155+0000 7f4031000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f402c0fe0d0 0x7f402c070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:29:26.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.155+0000 7f4031000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f402c0fe0d0 0x7f402c070310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44150/0 (socket says 172.21.15.123:44150) 2024-03-30T04:29:26.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.155+0000 7f4031000700 1 -- 172.21.15.123:0/1762791127 learned_addr learned my addr 172.21.15.123:0/1762791127 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:29:26.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.155+0000 7f4031a00700 1 --2- 172.21.15.123:0/1762791127 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f402c0fcee0 0x7f402c071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:29:26.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.155+0000 7f4031000700 1 -- 172.21.15.123:0/1762791127 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f402c0fcee0 msgr2=0x7f402c071e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:29:26.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.155+0000 7f4031000700 1 --2- 172.21.15.123:0/1762791127 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f402c0fcee0 0x7f402c071e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:26.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.155+0000 7f4031000700 1 -- 172.21.15.123:0/1762791127 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4020005040 con 0x7f402c0fe0d0 2024-03-30T04:29:26.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.156+0000 7f4031000700 1 --2- 172.21.15.123:0/1762791127 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f402c0fe0d0 0x7f402c070310 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f401800eb40 tx=0x7f401800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:29:26.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.156+0000 7f4029600700 1 -- 172.21.15.123:0/1762791127 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4018007cc0 con 0x7f402c0fe0d0 2024-03-30T04:29:26.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.156+0000 7f4029600700 1 -- 172.21.15.123:0/1762791127 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f401800f040 con 0x7f402c0fe0d0 2024-03-30T04:29:26.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.156+0000 7f4029600700 1 -- 172.21.15.123:0/1762791127 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f401800c990 con 0x7f402c0fe0d0 2024-03-30T04:29:26.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.156+0000 7f4034400700 1 -- 172.21.15.123:0/1762791127 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f402c070cb0 con 0x7f402c0fe0d0 2024-03-30T04:29:26.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.156+0000 7f4034400700 1 -- 172.21.15.123:0/1762791127 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f402c1158f0 con 0x7f402c0fe0d0 2024-03-30T04:29:26.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.157+0000 7f4034400700 1 -- 172.21.15.123:0/1762791127 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f402c05f5f0 con 0x7f402c0fe0d0 2024-03-30T04:29:26.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.158+0000 7f4029600700 1 -- 172.21.15.123:0/1762791127 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 30) v1 ==== 90807+0+0 (secure 0 0 0) 0x7f401800caf0 con 0x7f402c0fe0d0 2024-03-30T04:29:26.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.159+0000 7f4029600700 1 --2- 172.21.15.123:0/1762791127 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f401406c9c0 0x7f401406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:29:26.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.159+0000 7f4029600700 1 -- 172.21.15.123:0/1762791127 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f401808b860 con 0x7f402c0fe0d0 2024-03-30T04:29:26.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.160+0000 7f4031a00700 1 --2- 172.21.15.123:0/1762791127 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f401406c9c0 0x7f401406ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:29:26.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.160+0000 7f4031a00700 1 --2- 172.21.15.123:0/1762791127 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f401406c9c0 0x7f401406ee70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f402000b0d0 tx=0x7f4020009e30 comp rx=0 tx=0).ready entity=mgr.14696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:29:26.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.164+0000 7f4029600700 1 -- 172.21.15.123:0/1762791127 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f4018055c50 con 0x7f402c0fe0d0 2024-03-30T04:29:26.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.408+0000 7f4034400700 1 -- 172.21.15.123:0/1762791127 --> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f402c1024e0 con 0x7f401406c9c0 2024-03-30T04:29:26.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.409+0000 7f4029600700 1 -- 172.21.15.123:0/1762791127 <== mgr.14696 v2:172.21.15.123:6800/2125774379 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f402c1024e0 con 0x7f401406c9c0 2024-03-30T04:29:26.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.411+0000 7f4034400700 1 -- 172.21.15.123:0/1762791127 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f401406c9c0 msgr2=0x7f401406ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:29:26.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.411+0000 7f4034400700 1 --2- 172.21.15.123:0/1762791127 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f401406c9c0 0x7f401406ee70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f402000b0d0 tx=0x7f4020009e30 comp rx=0 tx=0).stop 2024-03-30T04:29:26.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.411+0000 7f4034400700 1 -- 172.21.15.123:0/1762791127 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f402c0fe0d0 msgr2=0x7f402c070310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:29:26.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.411+0000 7f4034400700 1 --2- 172.21.15.123:0/1762791127 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f402c0fe0d0 0x7f402c070310 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f401800eb40 tx=0x7f401800b040 comp rx=0 tx=0).stop 2024-03-30T04:29:26.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.412+0000 7f4034400700 1 -- 172.21.15.123:0/1762791127 shutdown_connections 2024-03-30T04:29:26.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.412+0000 7f4034400700 1 --2- 172.21.15.123:0/1762791127 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f401406c9c0 0x7f401406ee70 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:26.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.412+0000 7f4034400700 1 --2- 172.21.15.123:0/1762791127 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f402c0fcee0 0x7f402c071e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:26.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.412+0000 7f4034400700 1 --2- 172.21.15.123:0/1762791127 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f402c0fe0d0 0x7f402c070310 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:26.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.412+0000 7f4034400700 1 -- 172.21.15.123:0/1762791127 >> 172.21.15.123:0/1762791127 conn(0x7f402c0f8560 msgr2=0x7f402c0f9cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:29:26.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.412+0000 7f4034400700 1 -- 172.21.15.123:0/1762791127 shutdown_connections 2024-03-30T04:29:26.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.412+0000 7f4034400700 1 -- 172.21.15.123:0/1762791127 wait complete. 2024-03-30T04:29:26.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.582+0000 7f65f7400700 1 -- 172.21.15.123:0/65694431 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65f00662e0 msgr2=0x7f65f00ff9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:29:26.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.582+0000 7f65f7400700 1 --2- 172.21.15.123:0/65694431 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65f00662e0 0x7f65f00ff9c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f65e4004940 tx=0x7f65e4009e30 comp rx=0 tx=0).stop 2024-03-30T04:29:26.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.582+0000 7f65f7400700 1 -- 172.21.15.123:0/65694431 shutdown_connections 2024-03-30T04:29:26.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.582+0000 7f65f7400700 1 --2- 172.21.15.123:0/65694431 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f65f00fff00 0x7f65f0102320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:26.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.582+0000 7f65f7400700 1 --2- 172.21.15.123:0/65694431 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65f00662e0 0x7f65f00ff9c0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:26.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.582+0000 7f65f7400700 1 -- 172.21.15.123:0/65694431 >> 172.21.15.123:0/65694431 conn(0x7f65f00f7430 msgr2=0x7f65f00f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:29:26.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.583+0000 7f65f7400700 1 -- 172.21.15.123:0/65694431 shutdown_connections 2024-03-30T04:29:26.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.583+0000 7f65f7400700 1 -- 172.21.15.123:0/65694431 wait complete. 2024-03-30T04:29:26.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.584+0000 7f65f7400700 1 Processor -- start 2024-03-30T04:29:26.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.584+0000 7f65f7400700 1 -- start start 2024-03-30T04:29:26.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.585+0000 7f65f7400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f65f00662e0 0x7f65f010fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:29:26.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.585+0000 7f65f7400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65f00fff00 0x7f65f01100e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:29:26.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.585+0000 7f65f7400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f01106e0 con 0x7f65f00fff00 2024-03-30T04:29:26.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.585+0000 7f65f7400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f0110f30 con 0x7f65f00662e0 2024-03-30T04:29:26.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.585+0000 7f65f4a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f65f00662e0 0x7f65f010fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:29:26.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.586+0000 7f65f4a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f65f00662e0 0x7f65f010fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.123:51330/0 (socket says 172.21.15.123:51330) 2024-03-30T04:29:26.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.586+0000 7f65f4a00700 1 -- 172.21.15.123:0/265643458 learned_addr learned my addr 172.21.15.123:0/265643458 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:29:26.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.586+0000 7f65f4a00700 1 -- 172.21.15.123:0/265643458 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65f00fff00 msgr2=0x7f65f01100e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:29:26.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.586+0000 7f65f4a00700 1 --2- 172.21.15.123:0/265643458 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65f00fff00 0x7f65f01100e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:26.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.586+0000 7f65f4a00700 1 -- 172.21.15.123:0/265643458 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e4005040 con 0x7f65f00662e0 2024-03-30T04:29:26.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.587+0000 7f65f4a00700 1 --2- 172.21.15.123:0/265643458 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f65f00662e0 0x7f65f010fba0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f65e400b3b0 tx=0x7f65e4009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:29:26.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.587+0000 7f65ecc00700 1 -- 172.21.15.123:0/265643458 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f65e400e040 con 0x7f65f00662e0 2024-03-30T04:29:26.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.587+0000 7f65ecc00700 1 -- 172.21.15.123:0/265643458 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f65e4004290 con 0x7f65f00662e0 2024-03-30T04:29:26.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.587+0000 7f65ecc00700 1 -- 172.21.15.123:0/265643458 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f65e4013cf0 con 0x7f65f00662e0 2024-03-30T04:29:26.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.588+0000 7f65f7400700 1 -- 172.21.15.123:0/265643458 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65f01111b0 con 0x7f65f00662e0 2024-03-30T04:29:26.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.588+0000 7f65f7400700 1 -- 172.21.15.123:0/265643458 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65f01116f0 con 0x7f65f00662e0 2024-03-30T04:29:26.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.592+0000 7f65f7400700 1 -- 172.21.15.123:0/265643458 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65f00f8fb0 con 0x7f65f00662e0 2024-03-30T04:29:26.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.595+0000 7f65ecc00700 1 -- 172.21.15.123:0/265643458 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 30) v1 ==== 90807+0+0 (secure 0 0 0) 0x7f65e4026020 con 0x7f65f00662e0 2024-03-30T04:29:26.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.596+0000 7f65ecc00700 1 --2- 172.21.15.123:0/265643458 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f65d806cb50 0x7f65d806f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:29:26.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.596+0000 7f65ecc00700 1 -- 172.21.15.123:0/265643458 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f65e40955c0 con 0x7f65f00662e0 2024-03-30T04:29:26.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.596+0000 7f65efe00700 1 --2- 172.21.15.123:0/265643458 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f65d806cb50 0x7f65d806f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:29:26.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.597+0000 7f65efe00700 1 --2- 172.21.15.123:0/265643458 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f65d806cb50 0x7f65d806f000 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f65dc004d00 tx=0x7f65dc009ce0 comp rx=0 tx=0).ready entity=mgr.14696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:29:26.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.601+0000 7f65ecc00700 1 -- 172.21.15.123:0/265643458 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f65e4063480 con 0x7f65f00662e0 2024-03-30T04:29:26.819 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:26 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:29:26.819 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:26 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:29:26.819 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:26 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-30T04:29:26.819 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:26 smithi135 ceph-mon[44364]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-30T04:29:26.819 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:26 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:29:26.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.878+0000 7f65f7400700 1 -- 172.21.15.123:0/265643458 --> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f65f002b310 con 0x7f65d806cb50 2024-03-30T04:29:26.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.888+0000 7f65ecc00700 1 -- 172.21.15.123:0/265643458 <== mgr.14696 v2:172.21.15.123:6800/2125774379 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f65f002b310 con 0x7f65d806cb50 2024-03-30T04:29:26.887 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:29:26.887 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (112s) 19s ago 11m 13.8M - 0.25.0 c8568f914cd2 58af5a4ba60a 2024-03-30T04:29:26.888 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (11m) 19s ago 11m 6651k - 17.2.7-797-g5c024335 f77085f6aab7 bceb8b8c4837 2024-03-30T04:29:26.888 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (10m) 32s ago 10m 6651k - 17.2.7-797-g5c024335 f77085f6aab7 cd0fe67b2d34 2024-03-30T04:29:26.888 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (97s) 19s ago 10m 74.4M - 9.4.7 954c08fa6188 df1613fb18c2 2024-03-30T04:29:26.888 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (4m) 19s ago 4m 132M - 17.2.7-797-g5c024335 f77085f6aab7 2df066fba7a6 2024-03-30T04:29:26.888 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (4m) 19s ago 4m 77.3M - 17.2.7-797-g5c024335 f77085f6aab7 1293b8520db1 2024-03-30T04:29:26.888 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (4m) 32s ago 4m 97.2M - 17.2.7-797-g5c024335 f77085f6aab7 0dc41326818c 2024-03-30T04:29:26.888 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (4m) 32s ago 4m 160M - 17.2.7-797-g5c024335 f77085f6aab7 aef8a0cfc553 2024-03-30T04:29:26.888 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (53s) 19s ago 12m 475M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:29:26.888 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283 running (2m) 32s ago 10m 62.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 20e5e0a05742 2024-03-30T04:29:26.888 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (12m) 19s ago 12m 52.0M 2048M 17.2.7-797-g5c024335 f77085f6aab7 51d18587a7df 2024-03-30T04:29:26.888 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (9m) 32s ago 9m 41.7M 2048M 17.2.7-797-g5c024335 f77085f6aab7 f81052b09dd8 2024-03-30T04:29:26.888 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (118s) 19s ago 11m 7889k - 1.5.0 0da6a335fe13 f4da2b8a3030 2024-03-30T04:29:26.889 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (66s) 32s ago 10m 7788k - 1.5.0 0da6a335fe13 abf949a26182 2024-03-30T04:29:26.889 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (9m) 19s ago 9m 196M 1618M 17.2.7-797-g5c024335 f77085f6aab7 57a7ba3fd75e 2024-03-30T04:29:26.889 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (8m) 19s ago 8m 218M 1618M 17.2.7-797-g5c024335 f77085f6aab7 e1a6c1818a0c 2024-03-30T04:29:26.889 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (7m) 19s ago 7m 168M 1618M 17.2.7-797-g5c024335 f77085f6aab7 4c6a727567bd 2024-03-30T04:29:26.889 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (7m) 32s ago 7m 223M 2642M 17.2.7-797-g5c024335 f77085f6aab7 6fbdd1d46a9d 2024-03-30T04:29:26.889 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (6m) 32s ago 6m 203M 2642M 17.2.7-797-g5c024335 f77085f6aab7 b74988e947e5 2024-03-30T04:29:26.889 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (6m) 32s ago 6m 208M 2642M 17.2.7-797-g5c024335 f77085f6aab7 1fd3185f5484 2024-03-30T04:29:26.889 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 starting - - - - 2024-03-30T04:29:26.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.891+0000 7f65f7400700 1 -- 172.21.15.123:0/265643458 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f65d806cb50 msgr2=0x7f65d806f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:29:26.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.891+0000 7f65f7400700 1 --2- 172.21.15.123:0/265643458 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f65d806cb50 0x7f65d806f000 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f65dc004d00 tx=0x7f65dc009ce0 comp rx=0 tx=0).stop 2024-03-30T04:29:26.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.891+0000 7f65f7400700 1 -- 172.21.15.123:0/265643458 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f65f00662e0 msgr2=0x7f65f010fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:29:26.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.891+0000 7f65f7400700 1 --2- 172.21.15.123:0/265643458 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f65f00662e0 0x7f65f010fba0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f65e400b3b0 tx=0x7f65e4009e30 comp rx=0 tx=0).stop 2024-03-30T04:29:26.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.891+0000 7f65f7400700 1 -- 172.21.15.123:0/265643458 shutdown_connections 2024-03-30T04:29:26.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.891+0000 7f65f7400700 1 --2- 172.21.15.123:0/265643458 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f65d806cb50 0x7f65d806f000 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:26.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.891+0000 7f65f7400700 1 --2- 172.21.15.123:0/265643458 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f65f00662e0 0x7f65f010fba0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:26.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.891+0000 7f65f7400700 1 --2- 172.21.15.123:0/265643458 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65f00fff00 0x7f65f01100e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:26.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.891+0000 7f65f7400700 1 -- 172.21.15.123:0/265643458 >> 172.21.15.123:0/265643458 conn(0x7f65f00f7430 msgr2=0x7f65f00fbb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:29:26.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.891+0000 7f65f7400700 1 -- 172.21.15.123:0/265643458 shutdown_connections 2024-03-30T04:29:26.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:26.892+0000 7f65f7400700 1 -- 172.21.15.123:0/265643458 wait complete. 2024-03-30T04:29:26.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:26 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:29:26.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:26 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:29:26.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:26 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-30T04:29:26.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:26 smithi123 ceph-mon[35245]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-30T04:29:26.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:26 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:29:27.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.033+0000 7f26b6400700 1 -- 172.21.15.123:0/3794326620 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26b00662e0 msgr2=0x7f26b00ff9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:29:27.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.033+0000 7f26b6400700 1 --2- 172.21.15.123:0/3794326620 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26b00662e0 0x7f26b00ff9c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f26a0004940 tx=0x7f26a0009e30 comp rx=0 tx=0).stop 2024-03-30T04:29:27.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.033+0000 7f26b6400700 1 -- 172.21.15.123:0/3794326620 shutdown_connections 2024-03-30T04:29:27.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.033+0000 7f26b6400700 1 --2- 172.21.15.123:0/3794326620 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f26b00fff00 0x7f26b0102320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:27.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.033+0000 7f26b6400700 1 --2- 172.21.15.123:0/3794326620 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26b00662e0 0x7f26b00ff9c0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:27.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.033+0000 7f26b6400700 1 -- 172.21.15.123:0/3794326620 >> 172.21.15.123:0/3794326620 conn(0x7f26b00f7430 msgr2=0x7f26b00f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:29:27.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.033+0000 7f26b6400700 1 -- 172.21.15.123:0/3794326620 shutdown_connections 2024-03-30T04:29:27.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.033+0000 7f26b6400700 1 -- 172.21.15.123:0/3794326620 wait complete. 2024-03-30T04:29:27.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.034+0000 7f26b6400700 1 Processor -- start 2024-03-30T04:29:27.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.034+0000 7f26b6400700 1 -- start start 2024-03-30T04:29:27.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.035+0000 7f26b6400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26b00662e0 0x7f26b010fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:29:27.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.035+0000 7f26b6400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f26b00fff00 0x7f26b0110060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:29:27.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.035+0000 7f26b6400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b0110660 con 0x7f26b00662e0 2024-03-30T04:29:27.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.035+0000 7f26b6400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b0110eb0 con 0x7f26b00fff00 2024-03-30T04:29:27.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.035+0000 7f26af400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26b00662e0 0x7f26b010fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:29:27.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.035+0000 7f26af400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26b00662e0 0x7f26b010fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44190/0 (socket says 172.21.15.123:44190) 2024-03-30T04:29:27.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.035+0000 7f26af400700 1 -- 172.21.15.123:0/324329181 learned_addr learned my addr 172.21.15.123:0/324329181 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:29:27.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.035+0000 7f26aea00700 1 --2- 172.21.15.123:0/324329181 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f26b00fff00 0x7f26b0110060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:29:27.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.035+0000 7f26af400700 1 -- 172.21.15.123:0/324329181 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f26b00fff00 msgr2=0x7f26b0110060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:29:27.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.035+0000 7f26af400700 1 --2- 172.21.15.123:0/324329181 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f26b00fff00 0x7f26b0110060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:27.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.035+0000 7f26af400700 1 -- 172.21.15.123:0/324329181 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26a0005040 con 0x7f26b00662e0 2024-03-30T04:29:27.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.036+0000 7f26af400700 1 --2- 172.21.15.123:0/324329181 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26b00662e0 0x7f26b010fb20 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f26a0000c00 tx=0x7f26a0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:29:27.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.036+0000 7f26a7400700 1 -- 172.21.15.123:0/324329181 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26a000e070 con 0x7f26b00662e0 2024-03-30T04:29:27.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.036+0000 7f26a7400700 1 -- 172.21.15.123:0/324329181 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f26a00093b0 con 0x7f26b00662e0 2024-03-30T04:29:27.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.036+0000 7f26a7400700 1 -- 172.21.15.123:0/324329181 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26a001cd70 con 0x7f26b00662e0 2024-03-30T04:29:27.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.036+0000 7f26b6400700 1 -- 172.21.15.123:0/324329181 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26b0111130 con 0x7f26b00662e0 2024-03-30T04:29:27.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.036+0000 7f26b6400700 1 -- 172.21.15.123:0/324329181 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26b0111610 con 0x7f26b00662e0 2024-03-30T04:29:27.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.037+0000 7f26a4c00700 1 -- 172.21.15.123:0/324329181 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26880052b0 con 0x7f26b00662e0 2024-03-30T04:29:27.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.038+0000 7f26a7400700 1 -- 172.21.15.123:0/324329181 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 30) v1 ==== 90807+0+0 (secure 0 0 0) 0x7f26a001c600 con 0x7f26b00662e0 2024-03-30T04:29:27.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.039+0000 7f26a7400700 1 --2- 172.21.15.123:0/324329181 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f269406c9c0 0x7f269406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:29:27.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.039+0000 7f26a7400700 1 -- 172.21.15.123:0/324329181 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f26a008d2b0 con 0x7f26b00662e0 2024-03-30T04:29:27.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.039+0000 7f26aea00700 1 --2- 172.21.15.123:0/324329181 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f269406c9c0 0x7f269406ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:29:27.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.040+0000 7f26aea00700 1 --2- 172.21.15.123:0/324329181 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f269406c9c0 0x7f269406ee70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f269c009b80 tx=0x7f269c00b040 comp rx=0 tx=0).ready entity=mgr.14696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:29:27.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.044+0000 7f26a7400700 1 -- 172.21.15.123:0/324329181 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f26a005b060 con 0x7f26b00662e0 2024-03-30T04:29:27.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.421+0000 7f26a4c00700 1 -- 172.21.15.123:0/324329181 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f2688005830 con 0x7f26b00662e0 2024-03-30T04:29:27.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.422+0000 7f26a7400700 1 -- 172.21.15.123:0/324329181 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+743 (secure 0 0 0) 0x7f26a005abf0 con 0x7f26b00662e0 2024-03-30T04:29:27.424 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:29:27.424 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:29:27.424 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 2 2024-03-30T04:29:27.424 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:29:27.424 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:29:27.424 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:29:27.424 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:29:27.424 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:29:27.425 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 6 2024-03-30T04:29:27.425 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:29:27.425 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:29:27.425 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 4 2024-03-30T04:29:27.425 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:29:27.425 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:29:27.425 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 12, 2024-03-30T04:29:27.425 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:29:27.425 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:29:27.425 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:29:27.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.426+0000 7f26a4c00700 1 -- 172.21.15.123:0/324329181 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f269406c9c0 msgr2=0x7f269406ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:29:27.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.426+0000 7f26a4c00700 1 --2- 172.21.15.123:0/324329181 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f269406c9c0 0x7f269406ee70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f269c009b80 tx=0x7f269c00b040 comp rx=0 tx=0).stop 2024-03-30T04:29:27.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.426+0000 7f26a4c00700 1 -- 172.21.15.123:0/324329181 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26b00662e0 msgr2=0x7f26b010fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:29:27.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.426+0000 7f26a4c00700 1 --2- 172.21.15.123:0/324329181 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26b00662e0 0x7f26b010fb20 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f26a0000c00 tx=0x7f26a0009e30 comp rx=0 tx=0).stop 2024-03-30T04:29:27.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.426+0000 7f26a4c00700 1 -- 172.21.15.123:0/324329181 shutdown_connections 2024-03-30T04:29:27.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.426+0000 7f26a4c00700 1 --2- 172.21.15.123:0/324329181 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26b00662e0 0x7f26b010fb20 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:27.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.426+0000 7f26a4c00700 1 --2- 172.21.15.123:0/324329181 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f269406c9c0 0x7f269406ee70 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:27.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.426+0000 7f26a4c00700 1 --2- 172.21.15.123:0/324329181 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f26b00fff00 0x7f26b0110060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:27.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.426+0000 7f26a4c00700 1 -- 172.21.15.123:0/324329181 >> 172.21.15.123:0/324329181 conn(0x7f26b00f7430 msgr2=0x7f26b00fbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:29:27.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.427+0000 7f26a4c00700 1 -- 172.21.15.123:0/324329181 shutdown_connections 2024-03-30T04:29:27.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.427+0000 7f26a4c00700 1 -- 172.21.15.123:0/324329181 wait complete. 2024-03-30T04:29:27.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.585+0000 7ff772200700 1 -- 172.21.15.123:0/3465927060 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff76c071f50 msgr2=0x7ff76c070460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:29:27.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.585+0000 7ff772200700 1 --2- 172.21.15.123:0/3465927060 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff76c071f50 0x7ff76c070460 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7ff75c004940 tx=0x7ff75c009e30 comp rx=0 tx=0).stop 2024-03-30T04:29:27.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.585+0000 7ff772200700 1 -- 172.21.15.123:0/3465927060 shutdown_connections 2024-03-30T04:29:27.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.585+0000 7ff772200700 1 --2- 172.21.15.123:0/3465927060 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff76c0709a0 0x7ff76c070e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:27.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.585+0000 7ff772200700 1 --2- 172.21.15.123:0/3465927060 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff76c071f50 0x7ff76c070460 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:27.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.585+0000 7ff772200700 1 -- 172.21.15.123:0/3465927060 >> 172.21.15.123:0/3465927060 conn(0x7ff76c0f8560 msgr2=0x7ff76c0fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:29:27.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.585+0000 7ff772200700 1 -- 172.21.15.123:0/3465927060 shutdown_connections 2024-03-30T04:29:27.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.585+0000 7ff772200700 1 -- 172.21.15.123:0/3465927060 wait complete. 2024-03-30T04:29:27.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.586+0000 7ff772200700 1 Processor -- start 2024-03-30T04:29:27.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.586+0000 7ff772200700 1 -- start start 2024-03-30T04:29:27.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.586+0000 7ff772200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff76c0709a0 0x7ff76c113ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:29:27.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.586+0000 7ff772200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff76c071f50 0x7ff76c114410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:29:27.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.586+0000 7ff772200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff76c114a10 con 0x7ff76c071f50 2024-03-30T04:29:27.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.586+0000 7ff772200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff76c115260 con 0x7ff76c0709a0 2024-03-30T04:29:27.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.587+0000 7ff76b400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff76c071f50 0x7ff76c114410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:29:27.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.587+0000 7ff76b400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff76c071f50 0x7ff76c114410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44198/0 (socket says 172.21.15.123:44198) 2024-03-30T04:29:27.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.587+0000 7ff76b400700 1 -- 172.21.15.123:0/3407083763 learned_addr learned my addr 172.21.15.123:0/3407083763 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:29:27.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.587+0000 7ff763400700 1 --2- 172.21.15.123:0/3407083763 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff76c0709a0 0x7ff76c113ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:29:27.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.587+0000 7ff76b400700 1 -- 172.21.15.123:0/3407083763 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff76c0709a0 msgr2=0x7ff76c113ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:29:27.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.587+0000 7ff76b400700 1 --2- 172.21.15.123:0/3407083763 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff76c0709a0 0x7ff76c113ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:27.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.587+0000 7ff76b400700 1 -- 172.21.15.123:0/3407083763 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff75c005040 con 0x7ff76c071f50 2024-03-30T04:29:27.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.587+0000 7ff76b400700 1 --2- 172.21.15.123:0/3407083763 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff76c071f50 0x7ff76c114410 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7ff75400eb40 tx=0x7ff75400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:29:27.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.588+0000 7ff763e00700 1 -- 172.21.15.123:0/3407083763 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff754007cc0 con 0x7ff76c071f50 2024-03-30T04:29:27.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.588+0000 7ff763e00700 1 -- 172.21.15.123:0/3407083763 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff75400f040 con 0x7ff76c071f50 2024-03-30T04:29:27.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.588+0000 7ff763e00700 1 -- 172.21.15.123:0/3407083763 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff75400c990 con 0x7ff76c071f50 2024-03-30T04:29:27.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.588+0000 7ff772200700 1 -- 172.21.15.123:0/3407083763 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff76c1154e0 con 0x7ff76c071f50 2024-03-30T04:29:27.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.588+0000 7ff772200700 1 -- 172.21.15.123:0/3407083763 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff76c1159d0 con 0x7ff76c071f50 2024-03-30T04:29:27.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.589+0000 7ff772200700 1 -- 172.21.15.123:0/3407083763 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff76c1011e0 con 0x7ff76c071f50 2024-03-30T04:29:27.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.590+0000 7ff763e00700 1 -- 172.21.15.123:0/3407083763 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 30) v1 ==== 90807+0+0 (secure 0 0 0) 0x7ff75400cc00 con 0x7ff76c071f50 2024-03-30T04:29:27.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.591+0000 7ff763e00700 1 --2- 172.21.15.123:0/3407083763 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7ff75006c9c0 0x7ff75006ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:29:27.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.592+0000 7ff763e00700 1 -- 172.21.15.123:0/3407083763 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7ff75408b930 con 0x7ff76c071f50 2024-03-30T04:29:27.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.592+0000 7ff763400700 1 --2- 172.21.15.123:0/3407083763 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7ff75006c9c0 0x7ff75006ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:29:27.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.592+0000 7ff763400700 1 --2- 172.21.15.123:0/3407083763 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7ff75006c9c0 0x7ff75006ee70 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff75c0137d0 tx=0x7ff75c009e30 comp rx=0 tx=0).ready entity=mgr.14696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:29:27.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.596+0000 7ff763e00700 1 -- 172.21.15.123:0/3407083763 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7ff754059730 con 0x7ff76c071f50 2024-03-30T04:29:27.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.855+0000 7ff772200700 1 -- 172.21.15.123:0/3407083763 --> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff76c115c40 con 0x7ff75006c9c0 2024-03-30T04:29:27.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.856+0000 7ff763e00700 1 -- 172.21.15.123:0/3407083763 <== mgr.14696 v2:172.21.15.123:6800/2125774379 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7ff76c115c40 con 0x7ff75006c9c0 2024-03-30T04:29:27.855 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:29:27.855 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:29:27.855 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:29:27.855 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-30T04:29:27.855 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:29:27.855 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr" 2024-03-30T04:29:27.855 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:29:27.855 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "2/2 daemons upgraded", 2024-03-30T04:29:27.855 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mgr daemons", 2024-03-30T04:29:27.856 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:29:27.856 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:29:27.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.858+0000 7ff772200700 1 -- 172.21.15.123:0/3407083763 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7ff75006c9c0 msgr2=0x7ff75006ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:29:27.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.858+0000 7ff772200700 1 --2- 172.21.15.123:0/3407083763 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7ff75006c9c0 0x7ff75006ee70 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff75c0137d0 tx=0x7ff75c009e30 comp rx=0 tx=0).stop 2024-03-30T04:29:27.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.859+0000 7ff772200700 1 -- 172.21.15.123:0/3407083763 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff76c071f50 msgr2=0x7ff76c114410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:29:27.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.859+0000 7ff772200700 1 --2- 172.21.15.123:0/3407083763 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff76c071f50 0x7ff76c114410 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7ff75400eb40 tx=0x7ff75400b040 comp rx=0 tx=0).stop 2024-03-30T04:29:27.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.859+0000 7ff772200700 1 -- 172.21.15.123:0/3407083763 shutdown_connections 2024-03-30T04:29:27.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.859+0000 7ff772200700 1 --2- 172.21.15.123:0/3407083763 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7ff75006c9c0 0x7ff75006ee70 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:27.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.859+0000 7ff772200700 1 --2- 172.21.15.123:0/3407083763 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff76c0709a0 0x7ff76c113ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:27.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.859+0000 7ff772200700 1 --2- 172.21.15.123:0/3407083763 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff76c071f50 0x7ff76c114410 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:27.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.859+0000 7ff772200700 1 -- 172.21.15.123:0/3407083763 >> 172.21.15.123:0/3407083763 conn(0x7ff76c0f8560 msgr2=0x7ff76c0f9cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:29:27.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.859+0000 7ff772200700 1 -- 172.21.15.123:0/3407083763 shutdown_connections 2024-03-30T04:29:27.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:27.859+0000 7ff772200700 1 -- 172.21.15.123:0/3407083763 wait complete. 2024-03-30T04:29:27.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:27 smithi123 ceph-mon[35245]: pgmap v21: 65 pgs: 65 active+clean; 274 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 630 op/s 2024-03-30T04:29:27.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:27 smithi123 ceph-mon[35245]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:29:27.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:27 smithi123 ceph-mon[35245]: from='client.14740 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:29:27.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:27 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/324329181' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:29:28.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:27 smithi135 ceph-mon[44364]: pgmap v21: 65 pgs: 65 active+clean; 274 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 630 op/s 2024-03-30T04:29:28.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:27 smithi135 ceph-mon[44364]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:29:28.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:27 smithi135 ceph-mon[44364]: from='client.14740 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:29:28.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:27 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/324329181' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:29:28.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:28 smithi135 ceph-mon[44364]: from='client.24553 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:29:28.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:28 smithi135 ceph-mon[44364]: Upgrade: Updating mgr.smithi135.kyblli 2024-03-30T04:29:28.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:28 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi135.kyblli", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T04:29:28.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:28 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T04:29:28.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:28 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:29:28.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:28 smithi135 ceph-mon[44364]: Deploying daemon mgr.smithi135.kyblli on smithi135 2024-03-30T04:29:28.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:28 smithi123 ceph-mon[35245]: from='client.24553 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:29:28.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:28 smithi123 ceph-mon[35245]: Upgrade: Updating mgr.smithi135.kyblli 2024-03-30T04:29:28.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:28 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi135.kyblli", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T04:29:28.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:28 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T04:29:28.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:28 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:29:28.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:28 smithi123 ceph-mon[35245]: Deploying daemon mgr.smithi135.kyblli on smithi135 2024-03-30T04:29:29.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:29 smithi123 ceph-mon[35245]: pgmap v22: 65 pgs: 65 active+clean; 274 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 1.8 MiB/s wr, 535 op/s 2024-03-30T04:29:29.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:29 smithi123 ceph-mon[35245]: from='client.14750 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:29:30.025 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:29 smithi135 ceph-mon[44364]: pgmap v22: 65 pgs: 65 active+clean; 274 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 1.8 MiB/s wr, 535 op/s 2024-03-30T04:29:30.025 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:29 smithi135 ceph-mon[44364]: from='client.14750 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:29:31.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:31 smithi135 ceph-mon[44364]: pgmap v23: 65 pgs: 65 active+clean; 274 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 395 op/s 2024-03-30T04:29:31.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:31 smithi123 ceph-mon[35245]: pgmap v23: 65 pgs: 65 active+clean; 274 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 395 op/s 2024-03-30T04:29:33.758 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:33 smithi135 ceph-mon[44364]: pgmap v24: 65 pgs: 65 active+clean; 275 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 519 op/s 2024-03-30T04:29:33.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:33 smithi123 ceph-mon[35245]: pgmap v24: 65 pgs: 65 active+clean; 275 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 519 op/s 2024-03-30T04:29:34.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:34 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:29:34.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:34 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:29:34.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:34 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:29:34.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:34 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:29:34.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:34 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:29:35.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:34 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:29:35.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:34 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:29:35.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:34 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:29:35.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:34 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:29:35.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:34 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:29:35.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:35 smithi123 ceph-mon[35245]: pgmap v25: 65 pgs: 65 active+clean; 275 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 964 KiB/s rd, 1010 KiB/s wr, 312 op/s 2024-03-30T04:29:35.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:35 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:29:36.061 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:35 smithi135 ceph-mon[44364]: pgmap v25: 65 pgs: 65 active+clean; 275 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 964 KiB/s rd, 1010 KiB/s wr, 312 op/s 2024-03-30T04:29:36.061 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:35 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:29:37.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:37 smithi123 ceph-mon[35245]: pgmap v26: 65 pgs: 65 active+clean; 272 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 432 op/s 2024-03-30T04:29:38.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:37 smithi135 ceph-mon[44364]: pgmap v26: 65 pgs: 65 active+clean; 272 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 432 op/s 2024-03-30T04:29:39.853 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:39 smithi135 ceph-mon[44364]: pgmap v27: 65 pgs: 65 active+clean; 272 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 847 KiB/s rd, 867 KiB/s wr, 245 op/s 2024-03-30T04:29:39.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:39 smithi123 ceph-mon[35245]: pgmap v27: 65 pgs: 65 active+clean; 272 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 847 KiB/s rd, 867 KiB/s wr, 245 op/s 2024-03-30T04:29:41.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:41 smithi135 ceph-mon[44364]: pgmap v28: 65 pgs: 65 active+clean; 272 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 847 KiB/s rd, 867 KiB/s wr, 245 op/s 2024-03-30T04:29:41.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:41 smithi123 ceph-mon[35245]: pgmap v28: 65 pgs: 65 active+clean; 272 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 847 KiB/s rd, 867 KiB/s wr, 245 op/s 2024-03-30T04:29:43.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:43 smithi135 ceph-mon[44364]: pgmap v29: 65 pgs: 65 active+clean; 271 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 353 op/s 2024-03-30T04:29:43.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:43 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:29:43.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:43 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:29:43.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:43 smithi123 ceph-mon[35245]: pgmap v29: 65 pgs: 65 active+clean; 271 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 353 op/s 2024-03-30T04:29:43.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:43 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:29:43.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:43 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:29:45.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:45 smithi135 ceph-mon[44364]: pgmap v30: 65 pgs: 65 active+clean; 271 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 865 KiB/s rd, 804 KiB/s wr, 228 op/s 2024-03-30T04:29:45.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:45 smithi123 ceph-mon[35245]: pgmap v30: 65 pgs: 65 active+clean; 271 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 865 KiB/s rd, 804 KiB/s wr, 228 op/s 2024-03-30T04:29:46.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:46 smithi123 ceph-mon[35245]: Standby manager daemon smithi135.kyblli restarted 2024-03-30T04:29:46.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:46 smithi123 ceph-mon[35245]: Standby manager daemon smithi135.kyblli started 2024-03-30T04:29:46.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:46 smithi123 ceph-mon[35245]: from='mgr.? 172.21.15.135:0/3871542189' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi135.kyblli/crt"}]: dispatch 2024-03-30T04:29:46.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:46 smithi123 ceph-mon[35245]: from='mgr.? 172.21.15.135:0/3871542189' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T04:29:46.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:46 smithi123 ceph-mon[35245]: from='mgr.? 172.21.15.135:0/3871542189' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi135.kyblli/key"}]: dispatch 2024-03-30T04:29:46.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:46 smithi123 ceph-mon[35245]: from='mgr.? 172.21.15.135:0/3871542189' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T04:29:46.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:46 smithi135 ceph-mon[44364]: Standby manager daemon smithi135.kyblli restarted 2024-03-30T04:29:46.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:46 smithi135 ceph-mon[44364]: Standby manager daemon smithi135.kyblli started 2024-03-30T04:29:46.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:46 smithi135 ceph-mon[44364]: from='mgr.? 172.21.15.135:0/3871542189' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi135.kyblli/crt"}]: dispatch 2024-03-30T04:29:46.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:46 smithi135 ceph-mon[44364]: from='mgr.? 172.21.15.135:0/3871542189' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T04:29:46.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:46 smithi135 ceph-mon[44364]: from='mgr.? 172.21.15.135:0/3871542189' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi135.kyblli/key"}]: dispatch 2024-03-30T04:29:46.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:46 smithi135 ceph-mon[44364]: from='mgr.? 172.21.15.135:0/3871542189' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T04:29:47.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:47 smithi135 ceph-mon[44364]: pgmap v31: 65 pgs: 65 active+clean; 272 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 340 op/s 2024-03-30T04:29:47.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:47 smithi135 ceph-mon[44364]: mgrmap e31: smithi123.xgksqq(active, since 56s), standbys: smithi135.kyblli 2024-03-30T04:29:47.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:47 smithi123 ceph-mon[35245]: pgmap v31: 65 pgs: 65 active+clean; 272 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 340 op/s 2024-03-30T04:29:47.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:47 smithi123 ceph-mon[35245]: mgrmap e31: smithi123.xgksqq(active, since 56s), standbys: smithi135.kyblli 2024-03-30T04:29:49.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:49 smithi135 ceph-mon[44364]: pgmap v32: 65 pgs: 65 active+clean; 272 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 788 KiB/s rd, 833 KiB/s wr, 220 op/s 2024-03-30T04:29:49.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:49 smithi123 ceph-mon[35245]: pgmap v32: 65 pgs: 65 active+clean; 272 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 788 KiB/s rd, 833 KiB/s wr, 220 op/s 2024-03-30T04:29:50.748 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.0... 2024-03-30T04:29:50.748 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-30T04:29:50.819 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:50 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:29:50.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:50 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:29:51.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:51 smithi135 ceph-mon[44364]: pgmap v33: 65 pgs: 65 active+clean; 272 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 788 KiB/s rd, 833 KiB/s wr, 220 op/s 2024-03-30T04:29:51.874 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:51 smithi123 ceph-mon[35245]: pgmap v33: 65 pgs: 65 active+clean; 272 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 788 KiB/s rd, 833 KiB/s wr, 220 op/s 2024-03-30T04:29:51.876 DEBUG:teuthology.parallel:result is None 2024-03-30T04:29:53.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:53 smithi135 ceph-mon[44364]: pgmap v34: 65 pgs: 65 active+clean; 268 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 347 op/s 2024-03-30T04:29:53.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:53 smithi123 ceph-mon[35245]: pgmap v34: 65 pgs: 65 active+clean; 268 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 347 op/s 2024-03-30T04:29:54.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:54 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:29:54.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:54 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:29:55.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:54 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:29:55.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:54 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:29:55.873 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:55 smithi135 ceph-mon[44364]: pgmap v35: 65 pgs: 65 active+clean; 268 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 922 KiB/s rd, 982 KiB/s wr, 238 op/s 2024-03-30T04:29:55.873 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:55 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:29:55.873 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:55 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:29:55.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:55 smithi123 ceph-mon[35245]: pgmap v35: 65 pgs: 65 active+clean; 268 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 922 KiB/s rd, 982 KiB/s wr, 238 op/s 2024-03-30T04:29:55.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:55 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:29:55.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:55 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:29:57.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:57 smithi123 ceph-mon[35245]: pgmap v36: 65 pgs: 65 active+clean; 265 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 342 op/s 2024-03-30T04:29:58.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.022+0000 7f67e8400700 1 -- 172.21.15.123:0/3253685558 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67e0102300 msgr2=0x7f67e01046e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:29:58.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.022+0000 7f67e8400700 1 --2- 172.21.15.123:0/3253685558 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67e0102300 0x7f67e01046e0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f67cc004940 tx=0x7f67cc009e30 comp rx=0 tx=0).stop 2024-03-30T04:29:58.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.022+0000 7f67e8400700 1 -- 172.21.15.123:0/3253685558 shutdown_connections 2024-03-30T04:29:58.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.022+0000 7f67e8400700 1 --2- 172.21.15.123:0/3253685558 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67e0102300 0x7f67e01046e0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:58.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.023+0000 7f67e8400700 1 --2- 172.21.15.123:0/3253685558 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f67e0066f50 0x7f67e0101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:58.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.023+0000 7f67e8400700 1 -- 172.21.15.123:0/3253685558 >> 172.21.15.123:0/3253685558 conn(0x7f67e00f7450 msgr2=0x7f67e00f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:29:58.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.023+0000 7f67e8400700 1 -- 172.21.15.123:0/3253685558 shutdown_connections 2024-03-30T04:29:58.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.023+0000 7f67e8400700 1 -- 172.21.15.123:0/3253685558 wait complete. 2024-03-30T04:29:58.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.023+0000 7f67e8400700 1 Processor -- start 2024-03-30T04:29:58.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.024+0000 7f67e8400700 1 -- start start 2024-03-30T04:29:58.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.024+0000 7f67e8400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f67e0066f50 0x7f67e010fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:29:58.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.024+0000 7f67e8400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67e0102300 0x7f67e010ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:29:58.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.024+0000 7f67e8400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67e01105c0 con 0x7f67e0102300 2024-03-30T04:29:58.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.024+0000 7f67e8400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67e0110730 con 0x7f67e0066f50 2024-03-30T04:29:58.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.024+0000 7f67e5000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67e0102300 0x7f67e010ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:29:58.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.024+0000 7f67e5a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f67e0066f50 0x7f67e010fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:29:58.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.024+0000 7f67e5000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67e0102300 0x7f67e010ffc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34632/0 (socket says 172.21.15.123:34632) 2024-03-30T04:29:58.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.024+0000 7f67e5000700 1 -- 172.21.15.123:0/3951980954 learned_addr learned my addr 172.21.15.123:0/3951980954 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:29:58.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.025+0000 7f67e5000700 1 -- 172.21.15.123:0/3951980954 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f67e0066f50 msgr2=0x7f67e010fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:29:58.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.025+0000 7f67e5000700 1 --2- 172.21.15.123:0/3951980954 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f67e0066f50 0x7f67e010fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:58.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.025+0000 7f67e5000700 1 -- 172.21.15.123:0/3951980954 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67cc005040 con 0x7f67e0102300 2024-03-30T04:29:58.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.025+0000 7f67e5000700 1 --2- 172.21.15.123:0/3951980954 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67e0102300 0x7f67e010ffc0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f67cc009380 tx=0x7f67cc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:29:58.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.025+0000 7f67dd600700 1 -- 172.21.15.123:0/3951980954 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f67cc00e040 con 0x7f67e0102300 2024-03-30T04:29:58.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.025+0000 7f67dd600700 1 -- 172.21.15.123:0/3951980954 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f67cc00bd20 con 0x7f67e0102300 2024-03-30T04:29:58.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.025+0000 7f67dd600700 1 -- 172.21.15.123:0/3951980954 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f67cc01c630 con 0x7f67e0102300 2024-03-30T04:29:58.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.025+0000 7f67e8400700 1 -- 172.21.15.123:0/3951980954 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67e0115130 con 0x7f67e0102300 2024-03-30T04:29:58.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.025+0000 7f67e8400700 1 -- 172.21.15.123:0/3951980954 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67e01156f0 con 0x7f67e0102300 2024-03-30T04:29:58.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.027+0000 7f67e8400700 1 -- 172.21.15.123:0/3951980954 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67e004c710 con 0x7f67e0102300 2024-03-30T04:29:58.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.027+0000 7f67dd600700 1 -- 172.21.15.123:0/3951980954 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 31) v1 ==== 90807+0+0 (secure 0 0 0) 0x7f67cc013420 con 0x7f67e0102300 2024-03-30T04:29:58.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.029+0000 7f67dd600700 1 --2- 172.21.15.123:0/3951980954 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f67c806c9d0 0x7f67c806ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:29:58.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.029+0000 7f67dd600700 1 -- 172.21.15.123:0/3951980954 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f67cc08c8e0 con 0x7f67e0102300 2024-03-30T04:29:58.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.029+0000 7f67e5a00700 1 --2- 172.21.15.123:0/3951980954 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f67c806c9d0 0x7f67c806ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:29:58.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.030+0000 7f67e5a00700 1 --2- 172.21.15.123:0/3951980954 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f67c806c9d0 0x7f67c806ee80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f67e0066c90 tx=0x7f67d400b040 comp rx=0 tx=0).ready entity=mgr.14696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:29:58.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.034+0000 7f67dd600700 1 -- 172.21.15.123:0/3951980954 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f67cc05a690 con 0x7f67e0102300 2024-03-30T04:29:58.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:57 smithi135 ceph-mon[44364]: pgmap v36: 65 pgs: 65 active+clean; 265 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 342 op/s 2024-03-30T04:29:58.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.325+0000 7f67e8400700 1 -- 172.21.15.123:0/3951980954 --> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f67e00f8f40 con 0x7f67c806c9d0 2024-03-30T04:29:58.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.326+0000 7f67dd600700 1 -- 172.21.15.123:0/3951980954 <== mgr.14696 v2:172.21.15.123:6800/2125774379 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f67e00f8f40 con 0x7f67c806c9d0 2024-03-30T04:29:58.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.328+0000 7f67e8400700 1 -- 172.21.15.123:0/3951980954 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f67c806c9d0 msgr2=0x7f67c806ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:29:58.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.328+0000 7f67e8400700 1 --2- 172.21.15.123:0/3951980954 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f67c806c9d0 0x7f67c806ee80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f67e0066c90 tx=0x7f67d400b040 comp rx=0 tx=0).stop 2024-03-30T04:29:58.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.328+0000 7f67e8400700 1 -- 172.21.15.123:0/3951980954 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67e0102300 msgr2=0x7f67e010ffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:29:58.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.328+0000 7f67e8400700 1 --2- 172.21.15.123:0/3951980954 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67e0102300 0x7f67e010ffc0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f67cc009380 tx=0x7f67cc009e30 comp rx=0 tx=0).stop 2024-03-30T04:29:58.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.329+0000 7f67e8400700 1 -- 172.21.15.123:0/3951980954 shutdown_connections 2024-03-30T04:29:58.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.329+0000 7f67e8400700 1 --2- 172.21.15.123:0/3951980954 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f67c806c9d0 0x7f67c806ee80 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:58.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.329+0000 7f67e8400700 1 --2- 172.21.15.123:0/3951980954 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f67e0066f50 0x7f67e010fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:58.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.329+0000 7f67e8400700 1 --2- 172.21.15.123:0/3951980954 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67e0102300 0x7f67e010ffc0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:58.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.329+0000 7f67e8400700 1 -- 172.21.15.123:0/3951980954 >> 172.21.15.123:0/3951980954 conn(0x7f67e00f7450 msgr2=0x7f67e00f7820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:29:58.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.329+0000 7f67e8400700 1 -- 172.21.15.123:0/3951980954 shutdown_connections 2024-03-30T04:29:58.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.329+0000 7f67e8400700 1 -- 172.21.15.123:0/3951980954 wait complete. 2024-03-30T04:29:58.340 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:29:58.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.473+0000 7f450d400700 1 -- 172.21.15.123:0/2545382535 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4508066240 msgr2=0x7f45080ff8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:29:58.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.473+0000 7f450d400700 1 --2- 172.21.15.123:0/2545382535 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4508066240 0x7f45080ff8a0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f44f8004940 tx=0x7f44f8009e30 comp rx=0 tx=0).stop 2024-03-30T04:29:58.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.473+0000 7f450d400700 1 -- 172.21.15.123:0/2545382535 shutdown_connections 2024-03-30T04:29:58.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.473+0000 7f450d400700 1 --2- 172.21.15.123:0/2545382535 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f45080ffde0 0x7f45081021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:58.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.473+0000 7f450d400700 1 --2- 172.21.15.123:0/2545382535 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4508066240 0x7f45080ff8a0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:58.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.473+0000 7f450d400700 1 -- 172.21.15.123:0/2545382535 >> 172.21.15.123:0/2545382535 conn(0x7f45080f7270 msgr2=0x7f45080f96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:29:58.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.473+0000 7f450d400700 1 -- 172.21.15.123:0/2545382535 shutdown_connections 2024-03-30T04:29:58.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.473+0000 7f450d400700 1 -- 172.21.15.123:0/2545382535 wait complete. 2024-03-30T04:29:58.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.475+0000 7f450d400700 1 Processor -- start 2024-03-30T04:29:58.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.475+0000 7f450d400700 1 -- start start 2024-03-30T04:29:58.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.475+0000 7f450d400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4508066240 0x7f450810fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:29:58.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.476+0000 7f450d400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f45080ffde0 0x7f450810ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:29:58.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.476+0000 7f450d400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4508110570 con 0x7f45080ffde0 2024-03-30T04:29:58.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.476+0000 7f450d400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4508110d20 con 0x7f4508066240 2024-03-30T04:29:58.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.476+0000 7f4507400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f45080ffde0 0x7f450810ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:29:58.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.476+0000 7f4507e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4508066240 0x7f450810fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:29:58.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.476+0000 7f4507400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f45080ffde0 0x7f450810ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34638/0 (socket says 172.21.15.123:34638) 2024-03-30T04:29:58.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.476+0000 7f4507e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4508066240 0x7f450810fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.123:55440/0 (socket says 172.21.15.123:55440) 2024-03-30T04:29:58.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.476+0000 7f4507400700 1 -- 172.21.15.123:0/384503052 learned_addr learned my addr 172.21.15.123:0/384503052 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:29:58.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.476+0000 7f4507e00700 1 -- 172.21.15.123:0/384503052 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f45080ffde0 msgr2=0x7f450810ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:29:58.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.476+0000 7f4507e00700 1 --2- 172.21.15.123:0/384503052 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f45080ffde0 0x7f450810ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:58.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.476+0000 7f4507e00700 1 -- 172.21.15.123:0/384503052 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44f8005040 con 0x7f4508066240 2024-03-30T04:29:58.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.477+0000 7f4507e00700 1 --2- 172.21.15.123:0/384503052 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4508066240 0x7f450810fa30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f44f8000c00 tx=0x7f44f8009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:29:58.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.477+0000 7f44ffe00700 1 -- 172.21.15.123:0/384503052 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44f8021070 con 0x7f4508066240 2024-03-30T04:29:58.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.477+0000 7f44ffe00700 1 -- 172.21.15.123:0/384503052 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f44f801c400 con 0x7f4508066240 2024-03-30T04:29:58.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.477+0000 7f44ffe00700 1 -- 172.21.15.123:0/384503052 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44f800e070 con 0x7f4508066240 2024-03-30T04:29:58.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.477+0000 7f450d400700 1 -- 172.21.15.123:0/384503052 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4508110fa0 con 0x7f4508066240 2024-03-30T04:29:58.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.477+0000 7f450d400700 1 -- 172.21.15.123:0/384503052 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45081114b0 con 0x7f4508066240 2024-03-30T04:29:58.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.479+0000 7f450d400700 1 -- 172.21.15.123:0/384503052 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45080f8df0 con 0x7f4508066240 2024-03-30T04:29:58.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.480+0000 7f44ffe00700 1 -- 172.21.15.123:0/384503052 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 31) v1 ==== 90807+0+0 (secure 0 0 0) 0x7f44f801c9c0 con 0x7f4508066240 2024-03-30T04:29:58.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.481+0000 7f44ffe00700 1 --2- 172.21.15.123:0/384503052 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f44ec06c9c0 0x7f44ec06ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:29:58.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.481+0000 7f44ffe00700 1 -- 172.21.15.123:0/384503052 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f44f808cdb0 con 0x7f4508066240 2024-03-30T04:29:58.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.481+0000 7f4507400700 1 --2- 172.21.15.123:0/384503052 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f44ec06c9c0 0x7f44ec06ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:29:58.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.482+0000 7f4507400700 1 --2- 172.21.15.123:0/384503052 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f44ec06c9c0 0x7f44ec06ee70 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f44f4008fd0 tx=0x7f44f400b040 comp rx=0 tx=0).ready entity=mgr.14696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:29:58.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.486+0000 7f44ffe00700 1 -- 172.21.15.123:0/384503052 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f44f805ab60 con 0x7f4508066240 2024-03-30T04:29:58.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.783+0000 7f450d400700 1 -- 172.21.15.123:0/384503052 --> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4508105740 con 0x7f44ec06c9c0 2024-03-30T04:29:58.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.785+0000 7f44ffe00700 1 -- 172.21.15.123:0/384503052 <== mgr.14696 v2:172.21.15.123:6800/2125774379 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f4508105740 con 0x7f44ec06c9c0 2024-03-30T04:29:58.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.787+0000 7f450d400700 1 -- 172.21.15.123:0/384503052 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f44ec06c9c0 msgr2=0x7f44ec06ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:29:58.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.787+0000 7f450d400700 1 --2- 172.21.15.123:0/384503052 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f44ec06c9c0 0x7f44ec06ee70 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f44f4008fd0 tx=0x7f44f400b040 comp rx=0 tx=0).stop 2024-03-30T04:29:58.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.787+0000 7f450d400700 1 -- 172.21.15.123:0/384503052 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4508066240 msgr2=0x7f450810fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:29:58.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.787+0000 7f450d400700 1 --2- 172.21.15.123:0/384503052 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4508066240 0x7f450810fa30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f44f8000c00 tx=0x7f44f8009e30 comp rx=0 tx=0).stop 2024-03-30T04:29:58.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.787+0000 7f450d400700 1 -- 172.21.15.123:0/384503052 shutdown_connections 2024-03-30T04:29:58.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.787+0000 7f450d400700 1 --2- 172.21.15.123:0/384503052 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f44ec06c9c0 0x7f44ec06ee70 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:58.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.787+0000 7f450d400700 1 --2- 172.21.15.123:0/384503052 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4508066240 0x7f450810fa30 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:58.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.787+0000 7f450d400700 1 --2- 172.21.15.123:0/384503052 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f45080ffde0 0x7f450810ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:58.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.788+0000 7f450d400700 1 -- 172.21.15.123:0/384503052 >> 172.21.15.123:0/384503052 conn(0x7f45080f7270 msgr2=0x7f45080fba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:29:58.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.788+0000 7f450d400700 1 -- 172.21.15.123:0/384503052 shutdown_connections 2024-03-30T04:29:58.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.788+0000 7f450d400700 1 -- 172.21.15.123:0/384503052 wait complete. 2024-03-30T04:29:58.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.934+0000 7faeee200700 1 -- 172.21.15.123:0/3415312868 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faee80fcc30 msgr2=0x7faee80fd040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:29:58.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.934+0000 7faeee200700 1 --2- 172.21.15.123:0/3415312868 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faee80fcc30 0x7faee80fd040 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7faedc004940 tx=0x7faedc009e30 comp rx=0 tx=0).stop 2024-03-30T04:29:58.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.934+0000 7faeee200700 1 -- 172.21.15.123:0/3415312868 shutdown_connections 2024-03-30T04:29:58.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.934+0000 7faeee200700 1 --2- 172.21.15.123:0/3415312868 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faee80fdee0 0x7faee80fe350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:58.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.934+0000 7faeee200700 1 --2- 172.21.15.123:0/3415312868 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faee80fcc30 0x7faee80fd040 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:58.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.934+0000 7faeee200700 1 -- 172.21.15.123:0/3415312868 >> 172.21.15.123:0/3415312868 conn(0x7faee80f83d0 msgr2=0x7faee80fa7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:29:58.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.934+0000 7faeee200700 1 -- 172.21.15.123:0/3415312868 shutdown_connections 2024-03-30T04:29:58.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.935+0000 7faeee200700 1 -- 172.21.15.123:0/3415312868 wait complete. 2024-03-30T04:29:58.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.935+0000 7faeee200700 1 Processor -- start 2024-03-30T04:29:58.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.935+0000 7faeee200700 1 -- start start 2024-03-30T04:29:58.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.936+0000 7faeee200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faee80fcc30 0x7faee810f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:29:58.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.936+0000 7faeee200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faee80fdee0 0x7faee810fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:29:58.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.936+0000 7faeee200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee8110420 con 0x7faee80fdee0 2024-03-30T04:29:58.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.936+0000 7faeee200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee8110c70 con 0x7faee80fcc30 2024-03-30T04:29:58.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.936+0000 7faee7e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faee80fdee0 0x7faee810fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:29:58.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.936+0000 7faeece00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faee80fcc30 0x7faee810f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:29:58.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.936+0000 7faee7e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faee80fdee0 0x7faee810fe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34660/0 (socket says 172.21.15.123:34660) 2024-03-30T04:29:58.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.936+0000 7faee7e00700 1 -- 172.21.15.123:0/4197727352 learned_addr learned my addr 172.21.15.123:0/4197727352 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:29:58.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.936+0000 7faee7e00700 1 -- 172.21.15.123:0/4197727352 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faee80fcc30 msgr2=0x7faee810f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:29:58.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.936+0000 7faee7e00700 1 --2- 172.21.15.123:0/4197727352 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faee80fcc30 0x7faee810f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:58.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.936+0000 7faee7e00700 1 -- 172.21.15.123:0/4197727352 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faed4005040 con 0x7faee80fdee0 2024-03-30T04:29:58.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.937+0000 7faee7e00700 1 --2- 172.21.15.123:0/4197727352 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faee80fdee0 0x7faee810fe20 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7faed40096d0 tx=0x7faed4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:29:58.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.937+0000 7faee4c00700 1 -- 172.21.15.123:0/4197727352 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faed400e840 con 0x7faee80fdee0 2024-03-30T04:29:58.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.937+0000 7faee4c00700 1 -- 172.21.15.123:0/4197727352 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faed400f040 con 0x7faee80fdee0 2024-03-30T04:29:58.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.937+0000 7faeee200700 1 -- 172.21.15.123:0/4197727352 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faedc005040 con 0x7faee80fdee0 2024-03-30T04:29:58.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.937+0000 7faeee200700 1 -- 172.21.15.123:0/4197727352 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faee81112a0 con 0x7faee80fdee0 2024-03-30T04:29:58.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.937+0000 7faee4c00700 1 -- 172.21.15.123:0/4197727352 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faed4013d30 con 0x7faee80fdee0 2024-03-30T04:29:58.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.938+0000 7faeee200700 1 -- 172.21.15.123:0/4197727352 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faee8101870 con 0x7faee80fdee0 2024-03-30T04:29:58.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.939+0000 7faee4c00700 1 -- 172.21.15.123:0/4197727352 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 31) v1 ==== 90807+0+0 (secure 0 0 0) 0x7faed400e9a0 con 0x7faee80fdee0 2024-03-30T04:29:58.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.940+0000 7faee4c00700 1 --2- 172.21.15.123:0/4197727352 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7faed006c9c0 0x7faed006ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:29:58.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.941+0000 7faee4c00700 1 -- 172.21.15.123:0/4197727352 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7faed408bb20 con 0x7faee80fdee0 2024-03-30T04:29:58.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.942+0000 7faeece00700 1 --2- 172.21.15.123:0/4197727352 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7faed006c9c0 0x7faed006ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:29:58.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.942+0000 7faeece00700 1 --2- 172.21.15.123:0/4197727352 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7faed006c9c0 0x7faed006ee70 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7faedc009fd0 tx=0x7faedc009e30 comp rx=0 tx=0).ready entity=mgr.14696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:29:58.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:58.946+0000 7faee4c00700 1 -- 172.21.15.123:0/4197727352 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7faed4059870 con 0x7faee80fdee0 2024-03-30T04:29:59.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.175+0000 7faeee200700 1 -- 172.21.15.123:0/4197727352 --> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7faee81057a0 con 0x7faed006c9c0 2024-03-30T04:29:59.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.186+0000 7faee4c00700 1 -- 172.21.15.123:0/4197727352 <== mgr.14696 v2:172.21.15.123:6800/2125774379 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7faee81057a0 con 0x7faed006c9c0 2024-03-30T04:29:59.183 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:29:59.184 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (2m) 16s ago 11m 15.1M - 0.25.0 c8568f914cd2 58af5a4ba60a 2024-03-30T04:29:59.184 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (11m) 16s ago 11m 6651k - 17.2.7-797-g5c024335 f77085f6aab7 bceb8b8c4837 2024-03-30T04:29:59.184 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (10m) 5s ago 10m 6651k - 17.2.7-797-g5c024335 f77085f6aab7 cd0fe67b2d34 2024-03-30T04:29:59.184 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (2m) 16s ago 11m 73.2M - 9.4.7 954c08fa6188 df1613fb18c2 2024-03-30T04:29:59.184 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (5m) 16s ago 5m 114M - 17.2.7-797-g5c024335 f77085f6aab7 2df066fba7a6 2024-03-30T04:29:59.184 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (5m) 16s ago 5m 60.0M - 17.2.7-797-g5c024335 f77085f6aab7 1293b8520db1 2024-03-30T04:29:59.184 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (5m) 5s ago 5m 84.2M - 17.2.7-797-g5c024335 f77085f6aab7 0dc41326818c 2024-03-30T04:29:59.184 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (5m) 5s ago 5m 148M - 17.2.7-797-g5c024335 f77085f6aab7 aef8a0cfc553 2024-03-30T04:29:59.184 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (85s) 16s ago 13m 493M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:29:59.184 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (26s) 5s ago 10m 77.3M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:29:59.184 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (13m) 16s ago 13m 48.8M 2048M 17.2.7-797-g5c024335 f77085f6aab7 51d18587a7df 2024-03-30T04:29:59.184 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (10m) 5s ago 10m 37.4M 2048M 17.2.7-797-g5c024335 f77085f6aab7 f81052b09dd8 2024-03-30T04:29:59.184 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (2m) 16s ago 11m 7805k - 1.5.0 0da6a335fe13 f4da2b8a3030 2024-03-30T04:29:59.185 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (98s) 5s ago 10m 7742k - 1.5.0 0da6a335fe13 abf949a26182 2024-03-30T04:29:59.185 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (9m) 16s ago 9m 212M 1618M 17.2.7-797-g5c024335 f77085f6aab7 57a7ba3fd75e 2024-03-30T04:29:59.185 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (9m) 16s ago 9m 226M 1618M 17.2.7-797-g5c024335 f77085f6aab7 e1a6c1818a0c 2024-03-30T04:29:59.185 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (8m) 16s ago 8m 181M 1618M 17.2.7-797-g5c024335 f77085f6aab7 4c6a727567bd 2024-03-30T04:29:59.185 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (8m) 5s ago 8m 245M 2642M 17.2.7-797-g5c024335 f77085f6aab7 6fbdd1d46a9d 2024-03-30T04:29:59.185 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (7m) 5s ago 7m 213M 2642M 17.2.7-797-g5c024335 f77085f6aab7 b74988e947e5 2024-03-30T04:29:59.185 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (7m) 5s ago 6m 210M 2642M 17.2.7-797-g5c024335 f77085f6aab7 1fd3185f5484 2024-03-30T04:29:59.185 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (34s) 16s ago 11m 41.2M - 2.43.0 a07b618ecd1d ebb0e6589f67 2024-03-30T04:29:59.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.188+0000 7faeee200700 1 -- 172.21.15.123:0/4197727352 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7faed006c9c0 msgr2=0x7faed006ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:29:59.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.188+0000 7faeee200700 1 --2- 172.21.15.123:0/4197727352 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7faed006c9c0 0x7faed006ee70 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7faedc009fd0 tx=0x7faedc009e30 comp rx=0 tx=0).stop 2024-03-30T04:29:59.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.188+0000 7faeee200700 1 -- 172.21.15.123:0/4197727352 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faee80fdee0 msgr2=0x7faee810fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:29:59.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.188+0000 7faeee200700 1 --2- 172.21.15.123:0/4197727352 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faee80fdee0 0x7faee810fe20 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7faed40096d0 tx=0x7faed4009e30 comp rx=0 tx=0).stop 2024-03-30T04:29:59.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.188+0000 7faeee200700 1 -- 172.21.15.123:0/4197727352 shutdown_connections 2024-03-30T04:29:59.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.188+0000 7faeee200700 1 --2- 172.21.15.123:0/4197727352 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7faed006c9c0 0x7faed006ee70 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:59.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.188+0000 7faeee200700 1 --2- 172.21.15.123:0/4197727352 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faee80fcc30 0x7faee810f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:59.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.188+0000 7faeee200700 1 --2- 172.21.15.123:0/4197727352 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faee80fdee0 0x7faee810fe20 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:59.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.188+0000 7faeee200700 1 -- 172.21.15.123:0/4197727352 >> 172.21.15.123:0/4197727352 conn(0x7faee80f83d0 msgr2=0x7faee80f9a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:29:59.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.188+0000 7faeee200700 1 -- 172.21.15.123:0/4197727352 shutdown_connections 2024-03-30T04:29:59.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.189+0000 7faeee200700 1 -- 172.21.15.123:0/4197727352 wait complete. 2024-03-30T04:29:59.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.332+0000 7f7267200700 1 -- 172.21.15.123:0/2501842361 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7268102140 msgr2=0x7f7268104520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:29:59.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.332+0000 7f7267200700 1 --2- 172.21.15.123:0/2501842361 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7268102140 0x7f7268104520 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f7254004940 tx=0x7f7254009e30 comp rx=0 tx=0).stop 2024-03-30T04:29:59.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.333+0000 7f7267200700 1 -- 172.21.15.123:0/2501842361 shutdown_connections 2024-03-30T04:29:59.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.333+0000 7f7267200700 1 --2- 172.21.15.123:0/2501842361 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7268102140 0x7f7268104520 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:59.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.333+0000 7f7267200700 1 --2- 172.21.15.123:0/2501842361 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f72680662e0 0x7f7268101c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:59.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.333+0000 7f7267200700 1 -- 172.21.15.123:0/2501842361 >> 172.21.15.123:0/2501842361 conn(0x7f72680f7270 msgr2=0x7f72680f96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:29:59.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.333+0000 7f7267200700 1 -- 172.21.15.123:0/2501842361 shutdown_connections 2024-03-30T04:29:59.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.333+0000 7f7267200700 1 -- 172.21.15.123:0/2501842361 wait complete. 2024-03-30T04:29:59.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.334+0000 7f7267200700 1 Processor -- start 2024-03-30T04:29:59.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.334+0000 7f7267200700 1 -- start start 2024-03-30T04:29:59.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.335+0000 7f7267200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f72680662e0 0x7f726810f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:29:59.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.335+0000 7f7267200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7268102140 0x7f726810fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:29:59.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.335+0000 7f7267200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7268110440 con 0x7f7268102140 2024-03-30T04:29:59.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.335+0000 7f7267200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7268110c90 con 0x7f72680662e0 2024-03-30T04:29:59.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.335+0000 7f7265400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7268102140 0x7f726810fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:29:59.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.335+0000 7f7265e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f72680662e0 0x7f726810f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:29:59.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.335+0000 7f7265400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7268102140 0x7f726810fe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34684/0 (socket says 172.21.15.123:34684) 2024-03-30T04:29:59.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.335+0000 7f7265400700 1 -- 172.21.15.123:0/1100386219 learned_addr learned my addr 172.21.15.123:0/1100386219 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:29:59.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.335+0000 7f7265400700 1 -- 172.21.15.123:0/1100386219 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f72680662e0 msgr2=0x7f726810f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:29:59.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.335+0000 7f7265400700 1 --2- 172.21.15.123:0/1100386219 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f72680662e0 0x7f726810f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:59.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.335+0000 7f7265400700 1 -- 172.21.15.123:0/1100386219 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7254005040 con 0x7f7268102140 2024-03-30T04:29:59.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.336+0000 7f7265400700 1 --2- 172.21.15.123:0/1100386219 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7268102140 0x7f726810fe40 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f725400b0d0 tx=0x7f7254009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:29:59.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.336+0000 7f725e000700 1 -- 172.21.15.123:0/1100386219 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f725400e070 con 0x7f7268102140 2024-03-30T04:29:59.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.336+0000 7f7267200700 1 -- 172.21.15.123:0/1100386219 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7268110f10 con 0x7f7268102140 2024-03-30T04:29:59.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.336+0000 7f725e000700 1 -- 172.21.15.123:0/1100386219 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f725400b410 con 0x7f7268102140 2024-03-30T04:29:59.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.336+0000 7f725e000700 1 -- 172.21.15.123:0/1100386219 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7254012600 con 0x7f7268102140 2024-03-30T04:29:59.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.336+0000 7f7267200700 1 -- 172.21.15.123:0/1100386219 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7268111450 con 0x7f7268102140 2024-03-30T04:29:59.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.338+0000 7f7267200700 1 -- 172.21.15.123:0/1100386219 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f726805f5f0 con 0x7f7268102140 2024-03-30T04:29:59.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.339+0000 7f725e000700 1 -- 172.21.15.123:0/1100386219 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 31) v1 ==== 90807+0+0 (secure 0 0 0) 0x7f725400b9f0 con 0x7f7268102140 2024-03-30T04:29:59.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.339+0000 7f725e000700 1 --2- 172.21.15.123:0/1100386219 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f724c06c9c0 0x7f724c06ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:29:59.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.340+0000 7f725e000700 1 -- 172.21.15.123:0/1100386219 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f725408c910 con 0x7f7268102140 2024-03-30T04:29:59.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.340+0000 7f7265e00700 1 --2- 172.21.15.123:0/1100386219 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f724c06c9c0 0x7f724c06ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:29:59.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.341+0000 7f7265e00700 1 --2- 172.21.15.123:0/1100386219 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f724c06c9c0 0x7f724c06ee70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7258009b80 tx=0x7f725800b040 comp rx=0 tx=0).ready entity=mgr.14696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:29:59.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.344+0000 7f725e000700 1 -- 172.21.15.123:0/1100386219 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f725405a6f0 con 0x7f7268102140 2024-03-30T04:29:59.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.705+0000 7f7267200700 1 -- 172.21.15.123:0/1100386219 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f726804bed0 con 0x7f7268102140 2024-03-30T04:29:59.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.705+0000 7f725e000700 1 -- 172.21.15.123:0/1100386219 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+743 (secure 0 0 0) 0x7f7254022300 con 0x7f7268102140 2024-03-30T04:29:59.704 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:29:59.704 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:29:59.704 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 2 2024-03-30T04:29:59.704 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:29:59.704 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:29:59.704 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:29:59.705 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:29:59.705 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:29:59.705 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 6 2024-03-30T04:29:59.705 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:29:59.705 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:29:59.705 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 4 2024-03-30T04:29:59.705 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:29:59.705 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:29:59.705 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 12, 2024-03-30T04:29:59.705 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:29:59.705 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:29:59.705 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:29:59.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.708+0000 7f7267200700 1 -- 172.21.15.123:0/1100386219 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f724c06c9c0 msgr2=0x7f724c06ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:29:59.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.708+0000 7f7267200700 1 --2- 172.21.15.123:0/1100386219 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f724c06c9c0 0x7f724c06ee70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7258009b80 tx=0x7f725800b040 comp rx=0 tx=0).stop 2024-03-30T04:29:59.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.708+0000 7f7267200700 1 -- 172.21.15.123:0/1100386219 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7268102140 msgr2=0x7f726810fe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:29:59.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.708+0000 7f7267200700 1 --2- 172.21.15.123:0/1100386219 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7268102140 0x7f726810fe40 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f725400b0d0 tx=0x7f7254009e30 comp rx=0 tx=0).stop 2024-03-30T04:29:59.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.708+0000 7f7267200700 1 -- 172.21.15.123:0/1100386219 shutdown_connections 2024-03-30T04:29:59.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.708+0000 7f7267200700 1 --2- 172.21.15.123:0/1100386219 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f724c06c9c0 0x7f724c06ee70 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:59.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.708+0000 7f7267200700 1 --2- 172.21.15.123:0/1100386219 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f72680662e0 0x7f726810f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:59.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.708+0000 7f7267200700 1 --2- 172.21.15.123:0/1100386219 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7268102140 0x7f726810fe40 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:59.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.708+0000 7f7267200700 1 -- 172.21.15.123:0/1100386219 >> 172.21.15.123:0/1100386219 conn(0x7f72680f7270 msgr2=0x7f72680f7690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:29:59.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.708+0000 7f7267200700 1 -- 172.21.15.123:0/1100386219 shutdown_connections 2024-03-30T04:29:59.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.708+0000 7f7267200700 1 -- 172.21.15.123:0/1100386219 wait complete. 2024-03-30T04:29:59.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.847+0000 7f6b02e00700 1 -- 172.21.15.123:0/1648751206 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6afc0fcee0 msgr2=0x7f6afc0fd310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:29:59.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.847+0000 7f6b02e00700 1 --2- 172.21.15.123:0/1648751206 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6afc0fcee0 0x7f6afc0fd310 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f6aec004940 tx=0x7f6aec009e30 comp rx=0 tx=0).stop 2024-03-30T04:29:59.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.847+0000 7f6b02e00700 1 -- 172.21.15.123:0/1648751206 shutdown_connections 2024-03-30T04:29:59.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.847+0000 7f6b02e00700 1 --2- 172.21.15.123:0/1648751206 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6afc0fe0d0 0x7f6afc0fe540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:59.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.847+0000 7f6b02e00700 1 --2- 172.21.15.123:0/1648751206 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6afc0fcee0 0x7f6afc0fd310 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:59.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.847+0000 7f6b02e00700 1 -- 172.21.15.123:0/1648751206 >> 172.21.15.123:0/1648751206 conn(0x7f6afc0f8560 msgr2=0x7f6afc0fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:29:59.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.847+0000 7f6b02e00700 1 -- 172.21.15.123:0/1648751206 shutdown_connections 2024-03-30T04:29:59.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.848+0000 7f6b02e00700 1 -- 172.21.15.123:0/1648751206 wait complete. 2024-03-30T04:29:59.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.849+0000 7f6b02e00700 1 Processor -- start 2024-03-30T04:29:59.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.849+0000 7f6b02e00700 1 -- start start 2024-03-30T04:29:59.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.849+0000 7f6b02e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6afc0fcee0 0x7f6afc071e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:29:59.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.849+0000 7f6b02e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6afc0fe0d0 0x7f6afc070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:29:59.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.849+0000 7f6b02e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6afc0708c0 con 0x7f6afc0fe0d0 2024-03-30T04:29:59.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.849+0000 7f6b02e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6afc070a30 con 0x7f6afc0fcee0 2024-03-30T04:29:59.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.850+0000 7f6afb400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6afc0fe0d0 0x7f6afc070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:29:59.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.850+0000 7f6afb400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6afc0fe0d0 0x7f6afc070310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34694/0 (socket says 172.21.15.123:34694) 2024-03-30T04:29:59.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.850+0000 7f6afb400700 1 -- 172.21.15.123:0/2985278961 learned_addr learned my addr 172.21.15.123:0/2985278961 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:29:59.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.850+0000 7f6afbe00700 1 --2- 172.21.15.123:0/2985278961 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6afc0fcee0 0x7f6afc071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:29:59.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.850+0000 7f6afb400700 1 -- 172.21.15.123:0/2985278961 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6afc0fcee0 msgr2=0x7f6afc071e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:29:59.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.850+0000 7f6afb400700 1 --2- 172.21.15.123:0/2985278961 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6afc0fcee0 0x7f6afc071e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:29:59.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.850+0000 7f6afb400700 1 -- 172.21.15.123:0/2985278961 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aec005040 con 0x7f6afc0fe0d0 2024-03-30T04:29:59.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.850+0000 7f6afb400700 1 --2- 172.21.15.123:0/2985278961 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6afc0fe0d0 0x7f6afc070310 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f6ae800eb70 tx=0x7f6ae800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:29:59.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.851+0000 7f6af3e00700 1 -- 172.21.15.123:0/2985278961 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ae8007cc0 con 0x7f6afc0fe0d0 2024-03-30T04:29:59.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.851+0000 7f6af3e00700 1 -- 172.21.15.123:0/2985278961 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6ae800f040 con 0x7f6afc0fe0d0 2024-03-30T04:29:59.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.851+0000 7f6b02e00700 1 -- 172.21.15.123:0/2985278961 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6afc070cb0 con 0x7f6afc0fe0d0 2024-03-30T04:29:59.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.851+0000 7f6af3e00700 1 -- 172.21.15.123:0/2985278961 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ae80095f0 con 0x7f6afc0fe0d0 2024-03-30T04:29:59.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.851+0000 7f6b02e00700 1 -- 172.21.15.123:0/2985278961 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6afc1159b0 con 0x7f6afc0fe0d0 2024-03-30T04:29:59.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.852+0000 7f6b02e00700 1 -- 172.21.15.123:0/2985278961 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6afc05f5f0 con 0x7f6afc0fe0d0 2024-03-30T04:29:59.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.853+0000 7f6af3e00700 1 -- 172.21.15.123:0/2985278961 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 31) v1 ==== 90807+0+0 (secure 0 0 0) 0x7f6ae8009750 con 0x7f6afc0fe0d0 2024-03-30T04:29:59.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.855+0000 7f6af3e00700 1 --2- 172.21.15.123:0/2985278961 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f6ae006c9c0 0x7f6ae006ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:29:59.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.855+0000 7f6af3e00700 1 -- 172.21.15.123:0/2985278961 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f6ae808ba40 con 0x7f6afc0fe0d0 2024-03-30T04:29:59.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.855+0000 7f6afbe00700 1 --2- 172.21.15.123:0/2985278961 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f6ae006c9c0 0x7f6ae006ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:29:59.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.856+0000 7f6afbe00700 1 --2- 172.21.15.123:0/2985278961 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f6ae006c9c0 0x7f6ae006ee70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6aec013870 tx=0x7f6aec009e30 comp rx=0 tx=0).ready entity=mgr.14696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:29:59.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:29:59.859+0000 7f6af3e00700 1 -- 172.21.15.123:0/2985278961 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f6ae8055c60 con 0x7f6afc0fe0d0 2024-03-30T04:29:59.955 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:59 smithi135 ceph-mon[44364]: pgmap v37: 65 pgs: 65 active+clean; 265 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 911 KiB/s rd, 878 KiB/s wr, 230 op/s 2024-03-30T04:29:59.955 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:29:59 smithi135 ceph-mon[44364]: from='client.14758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:29:59.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:59 smithi123 ceph-mon[35245]: pgmap v37: 65 pgs: 65 active+clean; 265 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 911 KiB/s rd, 878 KiB/s wr, 230 op/s 2024-03-30T04:29:59.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:29:59 smithi123 ceph-mon[35245]: from='client.14758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:30:00.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:00.140+0000 7f6b02e00700 1 -- 172.21.15.123:0/2985278961 --> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6afc1025c0 con 0x7f6ae006c9c0 2024-03-30T04:30:00.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:00.141+0000 7f6af3e00700 1 -- 172.21.15.123:0/2985278961 <== mgr.14696 v2:172.21.15.123:6800/2125774379 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f6afc1025c0 con 0x7f6ae006c9c0 2024-03-30T04:30:00.140 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:30:00.140 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:30:00.140 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:30:00.140 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-30T04:30:00.140 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:30:00.140 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr" 2024-03-30T04:30:00.140 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:30:00.140 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "2/2 daemons upgraded", 2024-03-30T04:30:00.140 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mgr daemons", 2024-03-30T04:30:00.141 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:30:00.141 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:30:00.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:00.144+0000 7f6b02e00700 1 -- 172.21.15.123:0/2985278961 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f6ae006c9c0 msgr2=0x7f6ae006ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:30:00.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:00.144+0000 7f6b02e00700 1 --2- 172.21.15.123:0/2985278961 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f6ae006c9c0 0x7f6ae006ee70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6aec013870 tx=0x7f6aec009e30 comp rx=0 tx=0).stop 2024-03-30T04:30:00.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:00.144+0000 7f6b02e00700 1 -- 172.21.15.123:0/2985278961 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6afc0fe0d0 msgr2=0x7f6afc070310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:30:00.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:00.144+0000 7f6b02e00700 1 --2- 172.21.15.123:0/2985278961 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6afc0fe0d0 0x7f6afc070310 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f6ae800eb70 tx=0x7f6ae800b040 comp rx=0 tx=0).stop 2024-03-30T04:30:00.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:00.144+0000 7f6b02e00700 1 -- 172.21.15.123:0/2985278961 shutdown_connections 2024-03-30T04:30:00.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:00.144+0000 7f6b02e00700 1 --2- 172.21.15.123:0/2985278961 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f6ae006c9c0 0x7f6ae006ee70 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:30:00.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:00.144+0000 7f6b02e00700 1 --2- 172.21.15.123:0/2985278961 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6afc0fcee0 0x7f6afc071e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:30:00.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:00.144+0000 7f6b02e00700 1 --2- 172.21.15.123:0/2985278961 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6afc0fe0d0 0x7f6afc070310 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:30:00.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:00.144+0000 7f6b02e00700 1 -- 172.21.15.123:0/2985278961 >> 172.21.15.123:0/2985278961 conn(0x7f6afc0f8560 msgr2=0x7f6afc0f9d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:30:00.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:00.144+0000 7f6b02e00700 1 -- 172.21.15.123:0/2985278961 shutdown_connections 2024-03-30T04:30:00.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:00.144+0000 7f6b02e00700 1 -- 172.21.15.123:0/2985278961 wait complete. 2024-03-30T04:30:00.362 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.1... 2024-03-30T04:30:00.363 DEBUG:teuthology.orchestra.run.smithi135:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.1 /home/ubuntu/cephtest/clone.client.1 2024-03-30T04:30:00.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:00 smithi123 ceph-mon[35245]: from='client.24573 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:30:00.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:00 smithi123 ceph-mon[35245]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:30:00.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:00 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/1100386219' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:00.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:00 smithi123 ceph-mon[35245]: overall HEALTH_OK 2024-03-30T04:30:01.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:00 smithi135 ceph-mon[44364]: from='client.24573 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:30:01.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:00 smithi135 ceph-mon[44364]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:30:01.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:00 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/1100386219' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:01.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:00 smithi135 ceph-mon[44364]: overall HEALTH_OK 2024-03-30T04:30:01.382 DEBUG:teuthology.parallel:result is None 2024-03-30T04:30:01.382 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-30T04:30:01.465 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-30T04:30:01.466 DEBUG:teuthology.orchestra.run.smithi135:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1 2024-03-30T04:30:01.566 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.1/client.1 2024-03-30T04:30:01.566 DEBUG:teuthology.parallel:result is None 2024-03-30T04:30:01.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:01 smithi123 ceph-mon[35245]: pgmap v38: 65 pgs: 65 active+clean; 265 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 911 KiB/s rd, 878 KiB/s wr, 230 op/s 2024-03-30T04:30:01.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:01 smithi123 ceph-mon[35245]: from='client.14774 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:30:01.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:01 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:01.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:01 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:01.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:01 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:30:01.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:01 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:30:01.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:01 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:01.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:01 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:30:01.718 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:01 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:01.718 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:01 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:01.718 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:01 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi123.xgksqq"}]: dispatch 2024-03-30T04:30:01.718 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:01 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi123.xgksqq"}]': finished 2024-03-30T04:30:01.718 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:01 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi135.kyblli"}]: dispatch 2024-03-30T04:30:01.718 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:01 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi135.kyblli"}]': finished 2024-03-30T04:30:01.718 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:01 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:01.718 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:01 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:01.718 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:01 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:01.718 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:01 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:01.718 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:01 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:01.718 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:01 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:01.718 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:01 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:01.718 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:01 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:01.719 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:01 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:01.719 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:01 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:01.719 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:01 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:01.719 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:01 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:01.719 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:01 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:01.719 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:01 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:02.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:01 smithi135 ceph-mon[44364]: pgmap v38: 65 pgs: 65 active+clean; 265 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 911 KiB/s rd, 878 KiB/s wr, 230 op/s 2024-03-30T04:30:02.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:01 smithi135 ceph-mon[44364]: from='client.14774 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:30:02.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:01 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:02.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:01 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:02.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:01 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:30:02.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:01 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:30:02.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:01 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:02.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:01 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:30:02.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:01 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:02.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:01 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:02.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:01 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi123.xgksqq"}]: dispatch 2024-03-30T04:30:02.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:01 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi123.xgksqq"}]': finished 2024-03-30T04:30:02.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:01 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi135.kyblli"}]: dispatch 2024-03-30T04:30:02.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:01 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi135.kyblli"}]': finished 2024-03-30T04:30:02.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:01 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:02.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:01 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:02.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:01 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:02.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:01 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:02.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:01 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:02.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:01 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:02.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:01 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:02.103 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:01 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:02.103 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:01 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:02.103 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:01 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:02.103 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:01 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:02.103 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:01 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:02.103 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:01 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:02.103 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:01 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:02.930 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:02 smithi123 ceph-mon[35245]: Upgrade: Setting container_image for all mgr 2024-03-30T04:30:02.931 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:02 smithi123 ceph-mon[35245]: Upgrade: Setting container_image for all rgw 2024-03-30T04:30:02.931 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:02 smithi123 ceph-mon[35245]: Upgrade: Setting container_image for all rbd-mirror 2024-03-30T04:30:02.931 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:02 smithi123 ceph-mon[35245]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-30T04:30:02.931 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:02 smithi123 ceph-mon[35245]: Upgrade: Setting container_image for all ceph-exporter 2024-03-30T04:30:02.931 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:02 smithi123 ceph-mon[35245]: Upgrade: Setting container_image for all iscsi 2024-03-30T04:30:02.931 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:02 smithi123 ceph-mon[35245]: Upgrade: Setting container_image for all nfs 2024-03-30T04:30:02.931 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:02 smithi123 ceph-mon[35245]: Upgrade: Setting container_image for all nvmeof 2024-03-30T04:30:03.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:02 smithi135 ceph-mon[44364]: Upgrade: Setting container_image for all mgr 2024-03-30T04:30:03.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:02 smithi135 ceph-mon[44364]: Upgrade: Setting container_image for all rgw 2024-03-30T04:30:03.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:02 smithi135 ceph-mon[44364]: Upgrade: Setting container_image for all rbd-mirror 2024-03-30T04:30:03.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:02 smithi135 ceph-mon[44364]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-30T04:30:03.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:02 smithi135 ceph-mon[44364]: Upgrade: Setting container_image for all ceph-exporter 2024-03-30T04:30:03.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:02 smithi135 ceph-mon[44364]: Upgrade: Setting container_image for all iscsi 2024-03-30T04:30:03.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:02 smithi135 ceph-mon[44364]: Upgrade: Setting container_image for all nfs 2024-03-30T04:30:03.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:02 smithi135 ceph-mon[44364]: Upgrade: Setting container_image for all nvmeof 2024-03-30T04:30:03.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:03 smithi123 ceph-mon[35245]: pgmap v39: 65 pgs: 65 active+clean; 262 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 316 op/s 2024-03-30T04:30:04.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:03 smithi135 ceph-mon[44364]: pgmap v39: 65 pgs: 65 active+clean; 262 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 316 op/s 2024-03-30T04:30:04.925 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:04 smithi123 ceph-mon[35245]: Upgrade: Updating node-exporter.smithi123 (1/2) 2024-03-30T04:30:04.925 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:04 smithi123 ceph-mon[35245]: Deploying daemon node-exporter.smithi123 on smithi123 2024-03-30T04:30:05.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:04 smithi135 ceph-mon[44364]: Upgrade: Updating node-exporter.smithi123 (1/2) 2024-03-30T04:30:05.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:04 smithi135 ceph-mon[44364]: Deploying daemon node-exporter.smithi123 on smithi123 2024-03-30T04:30:05.925 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:05 smithi123 ceph-mon[35245]: pgmap v40: 65 pgs: 65 active+clean; 262 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 638 KiB/s rd, 639 KiB/s wr, 188 op/s 2024-03-30T04:30:05.925 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:05 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:30:06.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:05 smithi135 ceph-mon[44364]: pgmap v40: 65 pgs: 65 active+clean; 262 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 638 KiB/s rd, 639 KiB/s wr, 188 op/s 2024-03-30T04:30:06.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:05 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:30:07.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:07 smithi135 ceph-mon[44364]: pgmap v41: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 694 KiB/s rd, 671 KiB/s wr, 218 op/s 2024-03-30T04:30:07.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:07 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:07.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:07 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:07.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:07 smithi123 ceph-mon[35245]: pgmap v41: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 694 KiB/s rd, 671 KiB/s wr, 218 op/s 2024-03-30T04:30:07.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:07 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:07.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:07 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:09.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:09 smithi135 ceph-mon[44364]: pgmap v42: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 333 KiB/s rd, 337 KiB/s wr, 114 op/s 2024-03-30T04:30:09.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:09 smithi135 ceph-mon[44364]: Upgrade: Updating node-exporter.smithi135 (2/2) 2024-03-30T04:30:09.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:09 smithi135 ceph-mon[44364]: Deploying daemon node-exporter.smithi135 on smithi135 2024-03-30T04:30:09.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:09 smithi123 ceph-mon[35245]: pgmap v42: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 333 KiB/s rd, 337 KiB/s wr, 114 op/s 2024-03-30T04:30:09.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:09 smithi123 ceph-mon[35245]: Upgrade: Updating node-exporter.smithi135 (2/2) 2024-03-30T04:30:09.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:09 smithi123 ceph-mon[35245]: Deploying daemon node-exporter.smithi135 on smithi135 2024-03-30T04:30:11.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:11 smithi135 ceph-mon[44364]: pgmap v43: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 333 KiB/s rd, 337 KiB/s wr, 114 op/s 2024-03-30T04:30:11.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:11 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:11.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:11 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:11.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:11 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:30:11.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:11 smithi123 ceph-mon[35245]: pgmap v43: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 333 KiB/s rd, 337 KiB/s wr, 114 op/s 2024-03-30T04:30:11.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:11 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:11.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:11 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:11.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:11 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:30:13.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:13 smithi123 ceph-mon[35245]: pgmap v44: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 334 KiB/s rd, 338 KiB/s wr, 117 op/s 2024-03-30T04:30:13.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:13 smithi135 ceph-mon[44364]: pgmap v44: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 334 KiB/s rd, 338 KiB/s wr, 117 op/s 2024-03-30T04:30:15.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:15 smithi135 ceph-mon[44364]: pgmap v45: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 57 KiB/s rd, 32 KiB/s wr, 32 op/s 2024-03-30T04:30:15.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:15 smithi123 ceph-mon[35245]: pgmap v45: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 57 KiB/s rd, 32 KiB/s wr, 32 op/s 2024-03-30T04:30:17.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:17 smithi135 ceph-mon[44364]: pgmap v46: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 58 KiB/s rd, 32 KiB/s wr, 35 op/s 2024-03-30T04:30:17.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:17 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:17.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:17 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:17.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:17 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:17.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:17 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:17.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:17 smithi123 ceph-mon[35245]: pgmap v46: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 58 KiB/s rd, 32 KiB/s wr, 35 op/s 2024-03-30T04:30:17.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:17 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:17.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:17 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:17.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:17 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:17.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:17 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:18.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:18 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:18.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:18 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:18.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:18 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:18.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:18 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:19.065 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:18 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:19.065 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:18 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:19.065 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:18 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:19.065 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:18 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:20.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:19 smithi135 ceph-mon[44364]: pgmap v47: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.2 KiB/s rd, 682 B/s wr, 5 op/s 2024-03-30T04:30:20.187 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:19 smithi123 ceph-mon[35245]: pgmap v47: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.2 KiB/s rd, 682 B/s wr, 5 op/s 2024-03-30T04:30:20.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:20 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:30:21.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:20 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:30:22.060 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:21 smithi123 ceph-mon[35245]: pgmap v48: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.2 KiB/s rd, 682 B/s wr, 5 op/s 2024-03-30T04:30:22.060 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:21 smithi123 ceph-mon[35245]: mgrmap e32: smithi123.xgksqq(active, since 90s), standbys: smithi135.kyblli 2024-03-30T04:30:22.074 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:21 smithi135 ceph-mon[44364]: pgmap v48: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.2 KiB/s rd, 682 B/s wr, 5 op/s 2024-03-30T04:30:22.074 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:21 smithi135 ceph-mon[44364]: mgrmap e32: smithi123.xgksqq(active, since 90s), standbys: smithi135.kyblli 2024-03-30T04:30:23.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:23 smithi123 ceph-mon[35245]: pgmap v49: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 3.3 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-03-30T04:30:23.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:23.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:23.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:23.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:23.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:30:23.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:30:23.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:23.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:30:23.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:23.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:23.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:23.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:23.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:23.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:23.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:23.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:23.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:23.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:23 smithi135 ceph-mon[44364]: pgmap v49: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 3.3 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-03-30T04:30:23.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:23.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:23.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:23.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:23.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:30:23.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:30:23.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:23.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:30:23.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:23.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:23.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:23.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:23.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:23.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:23.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:23.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:23.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:25.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:25 smithi123 ceph-mon[35245]: pgmap v50: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.0 KiB/s rd, 682 B/s wr, 5 op/s 2024-03-30T04:30:25.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:25 smithi135 ceph-mon[44364]: pgmap v50: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.0 KiB/s rd, 682 B/s wr, 5 op/s 2024-03-30T04:30:26.446 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:26 smithi123 ceph-mon[35245]: Upgrade: Updating prometheus.smithi123 2024-03-30T04:30:26.446 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:26 smithi123 ceph-mon[35245]: Deploying daemon prometheus.smithi123 on smithi123 2024-03-30T04:30:26.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:26 smithi135 ceph-mon[44364]: Upgrade: Updating prometheus.smithi123 2024-03-30T04:30:26.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:26 smithi135 ceph-mon[44364]: Deploying daemon prometheus.smithi123 on smithi123 2024-03-30T04:30:27.458 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:27 smithi123 ceph-mon[35245]: pgmap v51: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.8 KiB/s rd, 682 B/s wr, 7 op/s 2024-03-30T04:30:27.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:27 smithi135 ceph-mon[44364]: pgmap v51: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.8 KiB/s rd, 682 B/s wr, 7 op/s 2024-03-30T04:30:29.350 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:29 smithi123 ceph-mon[35245]: pgmap v52: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.0 KiB/s rd, 682 B/s wr, 4 op/s 2024-03-30T04:30:29.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:29 smithi135 ceph-mon[44364]: pgmap v52: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.0 KiB/s rd, 682 B/s wr, 4 op/s 2024-03-30T04:30:30.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.304+0000 7fa3b6000700 1 -- 172.21.15.123:0/3210209206 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3b00ff000 msgr2=0x7fa3b00ff410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:30:30.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.304+0000 7fa3b6000700 1 --2- 172.21.15.123:0/3210209206 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3b00ff000 0x7fa3b00ff410 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fa3a0004940 tx=0x7fa3a0009e30 comp rx=0 tx=0).stop 2024-03-30T04:30:30.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.305+0000 7fa3b6000700 1 -- 172.21.15.123:0/3210209206 shutdown_connections 2024-03-30T04:30:30.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.305+0000 7fa3b6000700 1 --2- 172.21.15.123:0/3210209206 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa3b01002b0 0x7fa3b0100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:30:30.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.305+0000 7fa3b6000700 1 --2- 172.21.15.123:0/3210209206 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3b00ff000 0x7fa3b00ff410 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:30:30.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.305+0000 7fa3b6000700 1 -- 172.21.15.123:0/3210209206 >> 172.21.15.123:0/3210209206 conn(0x7fa3b00fa950 msgr2=0x7fa3b00fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:30:30.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.306+0000 7fa3b6000700 1 -- 172.21.15.123:0/3210209206 shutdown_connections 2024-03-30T04:30:30.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.307+0000 7fa3b6000700 1 -- 172.21.15.123:0/3210209206 wait complete. 2024-03-30T04:30:30.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.308+0000 7fa3b6000700 1 Processor -- start 2024-03-30T04:30:30.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.309+0000 7fa3b6000700 1 -- start start 2024-03-30T04:30:30.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.309+0000 7fa3b6000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa3b00ff000 0x7fa3b0113ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:30:30.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.309+0000 7fa3b6000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3b01002b0 0x7fa3b0114420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:30:30.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.309+0000 7fa3b6000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b0114ab0 con 0x7fa3b01002b0 2024-03-30T04:30:30.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.309+0000 7fa3b6000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b0119450 con 0x7fa3b00ff000 2024-03-30T04:30:30.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.311+0000 7fa3aea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3b01002b0 0x7fa3b0114420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:30:30.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.311+0000 7fa3aea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3b01002b0 0x7fa3b0114420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55088/0 (socket says 172.21.15.123:55088) 2024-03-30T04:30:30.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.311+0000 7fa3aea00700 1 -- 172.21.15.123:0/265975604 learned_addr learned my addr 172.21.15.123:0/265975604 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:30:30.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.311+0000 7fa3aea00700 1 -- 172.21.15.123:0/265975604 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa3b00ff000 msgr2=0x7fa3b0113ee0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-30T04:30:30.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.311+0000 7fa3aea00700 1 --2- 172.21.15.123:0/265975604 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa3b00ff000 0x7fa3b0113ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:30:30.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.311+0000 7fa3aea00700 1 -- 172.21.15.123:0/265975604 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3a0005040 con 0x7fa3b01002b0 2024-03-30T04:30:30.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.312+0000 7fa3aea00700 1 --2- 172.21.15.123:0/265975604 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3b01002b0 0x7fa3b0114420 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fa39c00b580 tx=0x7fa39c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:30:30.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.312+0000 7fa3a7400700 1 -- 172.21.15.123:0/265975604 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa39c00f6f0 con 0x7fa3b01002b0 2024-03-30T04:30:30.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.313+0000 7fa3a7400700 1 -- 172.21.15.123:0/265975604 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa39c00fd00 con 0x7fa3b01002b0 2024-03-30T04:30:30.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.313+0000 7fa3a7400700 1 -- 172.21.15.123:0/265975604 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa39c011e30 con 0x7fa3b01002b0 2024-03-30T04:30:30.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.314+0000 7fa3b6000700 1 -- 172.21.15.123:0/265975604 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3b01196b0 con 0x7fa3b01002b0 2024-03-30T04:30:30.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.314+0000 7fa3b6000700 1 -- 172.21.15.123:0/265975604 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3b0119ba0 con 0x7fa3b01002b0 2024-03-30T04:30:30.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.315+0000 7fa3b6000700 1 -- 172.21.15.123:0/265975604 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3b0103c40 con 0x7fa3b01002b0 2024-03-30T04:30:30.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.318+0000 7fa3a7400700 1 -- 172.21.15.123:0/265975604 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 32) v1 ==== 90705+0+0 (secure 0 0 0) 0x7fa39c00f850 con 0x7fa3b01002b0 2024-03-30T04:30:30.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.319+0000 7fa3a7400700 1 --2- 172.21.15.123:0/265975604 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fa39406c970 0x7fa39406ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:30:30.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.319+0000 7fa3a7400700 1 -- 172.21.15.123:0/265975604 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fa39c08b130 con 0x7fa3b01002b0 2024-03-30T04:30:30.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.321+0000 7fa3af400700 1 --2- 172.21.15.123:0/265975604 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fa39406c970 0x7fa39406ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:30:30.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.322+0000 7fa3af400700 1 --2- 172.21.15.123:0/265975604 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fa39406c970 0x7fa39406ee20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa3a0000c00 tx=0x7fa3a0009e30 comp rx=0 tx=0).ready entity=mgr.14696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:30:30.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.327+0000 7fa3a7400700 1 -- 172.21.15.123:0/265975604 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fa39c059100 con 0x7fa3b01002b0 2024-03-30T04:30:30.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.557+0000 7fa3b6000700 1 -- 172.21.15.123:0/265975604 --> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa3b002b470 con 0x7fa39406c970 2024-03-30T04:30:30.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.559+0000 7fa3a7400700 1 -- 172.21.15.123:0/265975604 <== mgr.14696 v2:172.21.15.123:6800/2125774379 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+367 (secure 0 0 0) 0x7fa3b002b470 con 0x7fa39406c970 2024-03-30T04:30:30.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.561+0000 7fa3b6000700 1 -- 172.21.15.123:0/265975604 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fa39406c970 msgr2=0x7fa39406ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:30:30.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.561+0000 7fa3b6000700 1 --2- 172.21.15.123:0/265975604 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fa39406c970 0x7fa39406ee20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa3a0000c00 tx=0x7fa3a0009e30 comp rx=0 tx=0).stop 2024-03-30T04:30:30.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.561+0000 7fa3b6000700 1 -- 172.21.15.123:0/265975604 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3b01002b0 msgr2=0x7fa3b0114420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:30:30.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.561+0000 7fa3b6000700 1 --2- 172.21.15.123:0/265975604 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3b01002b0 0x7fa3b0114420 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fa39c00b580 tx=0x7fa39c0091a0 comp rx=0 tx=0).stop 2024-03-30T04:30:30.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.561+0000 7fa3b6000700 1 -- 172.21.15.123:0/265975604 shutdown_connections 2024-03-30T04:30:30.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.561+0000 7fa3b6000700 1 --2- 172.21.15.123:0/265975604 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fa39406c970 0x7fa39406ee20 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:30:30.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.561+0000 7fa3b6000700 1 --2- 172.21.15.123:0/265975604 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa3b00ff000 0x7fa3b0113ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:30:30.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.561+0000 7fa3b6000700 1 --2- 172.21.15.123:0/265975604 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3b01002b0 0x7fa3b0114420 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:30:30.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.561+0000 7fa3b6000700 1 -- 172.21.15.123:0/265975604 >> 172.21.15.123:0/265975604 conn(0x7fa3b00fa950 msgr2=0x7fa3b00fb280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:30:30.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.561+0000 7fa3b6000700 1 -- 172.21.15.123:0/265975604 shutdown_connections 2024-03-30T04:30:30.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.561+0000 7fa3b6000700 1 -- 172.21.15.123:0/265975604 wait complete. 2024-03-30T04:30:30.581 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:30:30.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.724+0000 7ff506800700 1 -- 172.21.15.123:0/3785385988 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff5000fe0d0 msgr2=0x7ff5000fe540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:30:30.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.724+0000 7ff506800700 1 --2- 172.21.15.123:0/3785385988 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff5000fe0d0 0x7ff5000fe540 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff4ec004940 tx=0x7ff4ec009e30 comp rx=0 tx=0).stop 2024-03-30T04:30:30.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.725+0000 7ff506800700 1 -- 172.21.15.123:0/3785385988 shutdown_connections 2024-03-30T04:30:30.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.725+0000 7ff506800700 1 --2- 172.21.15.123:0/3785385988 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff5000fe0d0 0x7ff5000fe540 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:30:30.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.725+0000 7ff506800700 1 --2- 172.21.15.123:0/3785385988 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff5000fcee0 0x7ff5000fd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:30:30.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.725+0000 7ff506800700 1 -- 172.21.15.123:0/3785385988 >> 172.21.15.123:0/3785385988 conn(0x7ff5000f8560 msgr2=0x7ff5000fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:30:30.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.725+0000 7ff506800700 1 -- 172.21.15.123:0/3785385988 shutdown_connections 2024-03-30T04:30:30.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.725+0000 7ff506800700 1 -- 172.21.15.123:0/3785385988 wait complete. 2024-03-30T04:30:30.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.726+0000 7ff506800700 1 Processor -- start 2024-03-30T04:30:30.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.726+0000 7ff506800700 1 -- start start 2024-03-30T04:30:30.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.726+0000 7ff506800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff5000fcee0 0x7ff500071e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:30:30.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.726+0000 7ff506800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff5000fe0d0 0x7ff500070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:30:30.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.726+0000 7ff506800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5000708c0 con 0x7ff5000fe0d0 2024-03-30T04:30:30.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.726+0000 7ff506800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff500070a30 con 0x7ff5000fcee0 2024-03-30T04:30:30.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.726+0000 7ff4fea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff5000fe0d0 0x7ff500070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:30:30.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.726+0000 7ff4fea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff5000fe0d0 0x7ff500070310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55110/0 (socket says 172.21.15.123:55110) 2024-03-30T04:30:30.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.726+0000 7ff4fea00700 1 -- 172.21.15.123:0/2650909230 learned_addr learned my addr 172.21.15.123:0/2650909230 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:30:30.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.727+0000 7ff4ff400700 1 --2- 172.21.15.123:0/2650909230 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff5000fcee0 0x7ff500071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:30:30.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.727+0000 7ff4fea00700 1 -- 172.21.15.123:0/2650909230 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff5000fcee0 msgr2=0x7ff500071e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:30:30.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.727+0000 7ff4fea00700 1 --2- 172.21.15.123:0/2650909230 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff5000fcee0 0x7ff500071e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:30:30.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.727+0000 7ff4fea00700 1 -- 172.21.15.123:0/2650909230 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4ec005040 con 0x7ff5000fe0d0 2024-03-30T04:30:30.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.727+0000 7ff4fea00700 1 --2- 172.21.15.123:0/2650909230 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff5000fe0d0 0x7ff500070310 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ff4ec009380 tx=0x7ff4ec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:30:30.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.727+0000 7ff4f7400700 1 -- 172.21.15.123:0/2650909230 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4ec00e040 con 0x7ff5000fe0d0 2024-03-30T04:30:30.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.727+0000 7ff4f7400700 1 -- 172.21.15.123:0/2650909230 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff4ec004290 con 0x7ff5000fe0d0 2024-03-30T04:30:30.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.727+0000 7ff4f7400700 1 -- 172.21.15.123:0/2650909230 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4ec01c3f0 con 0x7ff5000fe0d0 2024-03-30T04:30:30.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.728+0000 7ff506800700 1 -- 172.21.15.123:0/2650909230 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff500070cb0 con 0x7ff5000fe0d0 2024-03-30T04:30:30.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.728+0000 7ff506800700 1 -- 172.21.15.123:0/2650909230 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5001159a0 con 0x7ff5000fe0d0 2024-03-30T04:30:30.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.729+0000 7ff506800700 1 -- 172.21.15.123:0/2650909230 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff50005f5f0 con 0x7ff5000fe0d0 2024-03-30T04:30:30.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.730+0000 7ff4f7400700 1 -- 172.21.15.123:0/2650909230 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 32) v1 ==== 90705+0+0 (secure 0 0 0) 0x7ff4ec013420 con 0x7ff5000fe0d0 2024-03-30T04:30:30.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.731+0000 7ff4f7400700 1 --2- 172.21.15.123:0/2650909230 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7ff4e406c8b0 0x7ff4e406ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:30:30.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.731+0000 7ff4f7400700 1 -- 172.21.15.123:0/2650909230 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7ff4ec08ce30 con 0x7ff5000fe0d0 2024-03-30T04:30:30.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.731+0000 7ff4ff400700 1 --2- 172.21.15.123:0/2650909230 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7ff4e406c8b0 0x7ff4e406ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:30:30.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.732+0000 7ff4ff400700 1 --2- 172.21.15.123:0/2650909230 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7ff4e406c8b0 0x7ff4e406ed60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff500071bd0 tx=0x7ff4f000b040 comp rx=0 tx=0).ready entity=mgr.14696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:30:30.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.736+0000 7ff4f7400700 1 -- 172.21.15.123:0/2650909230 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7ff4ec057430 con 0x7ff5000fe0d0 2024-03-30T04:30:30.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.979+0000 7ff506800700 1 -- 172.21.15.123:0/2650909230 --> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff500102570 con 0x7ff4e406c8b0 2024-03-30T04:30:30.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.980+0000 7ff4f7400700 1 -- 172.21.15.123:0/2650909230 <== mgr.14696 v2:172.21.15.123:6800/2125774379 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+367 (secure 0 0 0) 0x7ff500102570 con 0x7ff4e406c8b0 2024-03-30T04:30:30.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.982+0000 7ff506800700 1 -- 172.21.15.123:0/2650909230 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7ff4e406c8b0 msgr2=0x7ff4e406ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:30:30.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.982+0000 7ff506800700 1 --2- 172.21.15.123:0/2650909230 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7ff4e406c8b0 0x7ff4e406ed60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff500071bd0 tx=0x7ff4f000b040 comp rx=0 tx=0).stop 2024-03-30T04:30:30.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.982+0000 7ff506800700 1 -- 172.21.15.123:0/2650909230 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff5000fe0d0 msgr2=0x7ff500070310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:30:30.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.982+0000 7ff506800700 1 --2- 172.21.15.123:0/2650909230 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff5000fe0d0 0x7ff500070310 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ff4ec009380 tx=0x7ff4ec009e30 comp rx=0 tx=0).stop 2024-03-30T04:30:30.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.982+0000 7ff506800700 1 -- 172.21.15.123:0/2650909230 shutdown_connections 2024-03-30T04:30:30.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.982+0000 7ff506800700 1 --2- 172.21.15.123:0/2650909230 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7ff4e406c8b0 0x7ff4e406ed60 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:30:30.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.983+0000 7ff506800700 1 --2- 172.21.15.123:0/2650909230 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff5000fcee0 0x7ff500071e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:30:30.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.983+0000 7ff506800700 1 --2- 172.21.15.123:0/2650909230 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff5000fe0d0 0x7ff500070310 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:30:30.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.983+0000 7ff506800700 1 -- 172.21.15.123:0/2650909230 >> 172.21.15.123:0/2650909230 conn(0x7ff5000f8560 msgr2=0x7ff5000f9d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:30:30.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.983+0000 7ff506800700 1 -- 172.21.15.123:0/2650909230 shutdown_connections 2024-03-30T04:30:30.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:30.983+0000 7ff506800700 1 -- 172.21.15.123:0/2650909230 wait complete. 2024-03-30T04:30:31.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.134+0000 7ff240a00700 1 -- 172.21.15.123:0/1972114435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff23c0709a0 msgr2=0x7ff23c070e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:30:31.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.134+0000 7ff240a00700 1 --2- 172.21.15.123:0/1972114435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff23c0709a0 0x7ff23c070e10 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7ff228004940 tx=0x7ff228009e30 comp rx=0 tx=0).stop 2024-03-30T04:30:31.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.134+0000 7ff240a00700 1 -- 172.21.15.123:0/1972114435 shutdown_connections 2024-03-30T04:30:31.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.134+0000 7ff240a00700 1 --2- 172.21.15.123:0/1972114435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff23c0709a0 0x7ff23c070e10 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:30:31.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.134+0000 7ff240a00700 1 --2- 172.21.15.123:0/1972114435 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff23c071f50 0x7ff23c070460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:30:31.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.134+0000 7ff240a00700 1 -- 172.21.15.123:0/1972114435 >> 172.21.15.123:0/1972114435 conn(0x7ff23c0f8560 msgr2=0x7ff23c0fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:30:31.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.134+0000 7ff240a00700 1 -- 172.21.15.123:0/1972114435 shutdown_connections 2024-03-30T04:30:31.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.134+0000 7ff240a00700 1 -- 172.21.15.123:0/1972114435 wait complete. 2024-03-30T04:30:31.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.135+0000 7ff240a00700 1 Processor -- start 2024-03-30T04:30:31.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.135+0000 7ff240a00700 1 -- start start 2024-03-30T04:30:31.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.135+0000 7ff240a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff23c0709a0 0x7ff23c10faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:30:31.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.135+0000 7ff240a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff23c071f50 0x7ff23c10ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:30:31.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.135+0000 7ff240a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff23c1105e0 con 0x7ff23c0709a0 2024-03-30T04:30:31.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.135+0000 7ff240a00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff23c110e30 con 0x7ff23c071f50 2024-03-30T04:30:31.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.136+0000 7ff239e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff23c0709a0 0x7ff23c10faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:30:31.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.136+0000 7ff239e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff23c0709a0 0x7ff23c10faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55126/0 (socket says 172.21.15.123:55126) 2024-03-30T04:30:31.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.136+0000 7ff239e00700 1 -- 172.21.15.123:0/3285690741 learned_addr learned my addr 172.21.15.123:0/3285690741 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:30:31.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.136+0000 7ff239400700 1 --2- 172.21.15.123:0/3285690741 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff23c071f50 0x7ff23c10ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:30:31.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.136+0000 7ff239e00700 1 -- 172.21.15.123:0/3285690741 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff23c071f50 msgr2=0x7ff23c10ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:30:31.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.136+0000 7ff239e00700 1 --2- 172.21.15.123:0/3285690741 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff23c071f50 0x7ff23c10ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:30:31.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.136+0000 7ff239e00700 1 -- 172.21.15.123:0/3285690741 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff228005040 con 0x7ff23c0709a0 2024-03-30T04:30:31.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.136+0000 7ff239e00700 1 --2- 172.21.15.123:0/3285690741 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff23c0709a0 0x7ff23c10faa0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ff22c00eb70 tx=0x7ff22c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:30:31.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.136+0000 7ff232000700 1 -- 172.21.15.123:0/3285690741 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff22c007cc0 con 0x7ff23c0709a0 2024-03-30T04:30:31.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.137+0000 7ff232000700 1 -- 172.21.15.123:0/3285690741 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff22c00f040 con 0x7ff23c0709a0 2024-03-30T04:30:31.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.137+0000 7ff232000700 1 -- 172.21.15.123:0/3285690741 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff22c00cad0 con 0x7ff23c0709a0 2024-03-30T04:30:31.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.137+0000 7ff240a00700 1 -- 172.21.15.123:0/3285690741 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff23c1110b0 con 0x7ff23c0709a0 2024-03-30T04:30:31.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.137+0000 7ff240a00700 1 -- 172.21.15.123:0/3285690741 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff23c1115f0 con 0x7ff23c0709a0 2024-03-30T04:30:31.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.138+0000 7ff240a00700 1 -- 172.21.15.123:0/3285690741 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff23c05f5f0 con 0x7ff23c0709a0 2024-03-30T04:30:31.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.139+0000 7ff232000700 1 -- 172.21.15.123:0/3285690741 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 32) v1 ==== 90705+0+0 (secure 0 0 0) 0x7ff22c00cc30 con 0x7ff23c0709a0 2024-03-30T04:30:31.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.140+0000 7ff232000700 1 --2- 172.21.15.123:0/3285690741 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7ff22006c8b0 0x7ff22006ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:30:31.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.140+0000 7ff232000700 1 -- 172.21.15.123:0/3285690741 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7ff22c08b650 con 0x7ff23c0709a0 2024-03-30T04:30:31.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.141+0000 7ff239400700 1 --2- 172.21.15.123:0/3285690741 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7ff22006c8b0 0x7ff22006ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:30:31.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.141+0000 7ff239400700 1 --2- 172.21.15.123:0/3285690741 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7ff22006c8b0 0x7ff22006ed60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff22800b0d0 tx=0x7ff228009e30 comp rx=0 tx=0).ready entity=mgr.14696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:30:31.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.145+0000 7ff232000700 1 -- 172.21.15.123:0/3285690741 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7ff22c0595d0 con 0x7ff23c0709a0 2024-03-30T04:30:31.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.394+0000 7ff240a00700 1 -- 172.21.15.123:0/3285690741 --> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff23c111980 con 0x7ff22006c8b0 2024-03-30T04:30:31.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.405+0000 7ff232000700 1 -- 172.21.15.123:0/3285690741 <== mgr.14696 v2:172.21.15.123:6800/2125774379 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7ff23c111980 con 0x7ff22006c8b0 2024-03-30T04:30:31.403 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:30:31.404 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (2m) 14s ago 12m 15.4M - 0.25.0 c8568f914cd2 58af5a4ba60a 2024-03-30T04:30:31.404 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (12m) 14s ago 12m 6651k - 17.2.7-797-g5c024335 f77085f6aab7 bceb8b8c4837 2024-03-30T04:30:31.404 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (11m) 14s ago 11m 6651k - 17.2.7-797-g5c024335 f77085f6aab7 cd0fe67b2d34 2024-03-30T04:30:31.404 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (2m) 14s ago 11m 73.7M - 9.4.7 954c08fa6188 df1613fb18c2 2024-03-30T04:30:31.404 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (5m) 14s ago 5m 109M - 17.2.7-797-g5c024335 f77085f6aab7 2df066fba7a6 2024-03-30T04:30:31.404 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (5m) 14s ago 5m 57.5M - 17.2.7-797-g5c024335 f77085f6aab7 1293b8520db1 2024-03-30T04:30:31.404 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (5m) 14s ago 5m 67.5M - 17.2.7-797-g5c024335 f77085f6aab7 0dc41326818c 2024-03-30T04:30:31.404 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (5m) 14s ago 5m 119M - 17.2.7-797-g5c024335 f77085f6aab7 aef8a0cfc553 2024-03-30T04:30:31.404 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (118s) 14s ago 13m 497M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:30:31.404 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (58s) 14s ago 11m 446M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:30:31.404 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (13m) 14s ago 14m 50.9M 2048M 17.2.7-797-g5c024335 f77085f6aab7 51d18587a7df 2024-03-30T04:30:31.405 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (11m) 14s ago 11m 39.5M 2048M 17.2.7-797-g5c024335 f77085f6aab7 f81052b09dd8 2024-03-30T04:30:31.405 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (25s) 14s ago 12m 2839k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:30:31.405 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (21s) 14s ago 11m 6958k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:30:31.405 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (10m) 14s ago 10m 218M 1618M 17.2.7-797-g5c024335 f77085f6aab7 57a7ba3fd75e 2024-03-30T04:30:31.405 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (9m) 14s ago 9m 234M 1618M 17.2.7-797-g5c024335 f77085f6aab7 e1a6c1818a0c 2024-03-30T04:30:31.405 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (9m) 14s ago 9m 187M 1618M 17.2.7-797-g5c024335 f77085f6aab7 4c6a727567bd 2024-03-30T04:30:31.405 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (8m) 14s ago 8m 256M 2642M 17.2.7-797-g5c024335 f77085f6aab7 6fbdd1d46a9d 2024-03-30T04:30:31.405 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (8m) 14s ago 8m 224M 2642M 17.2.7-797-g5c024335 f77085f6aab7 b74988e947e5 2024-03-30T04:30:31.405 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (7m) 14s ago 7m 218M 2642M 17.2.7-797-g5c024335 f77085f6aab7 1fd3185f5484 2024-03-30T04:30:31.405 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (66s) 14s ago 11m 43.0M - 2.43.0 a07b618ecd1d ebb0e6589f67 2024-03-30T04:30:31.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.407+0000 7ff240a00700 1 -- 172.21.15.123:0/3285690741 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7ff22006c8b0 msgr2=0x7ff22006ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:30:31.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.407+0000 7ff240a00700 1 --2- 172.21.15.123:0/3285690741 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7ff22006c8b0 0x7ff22006ed60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff22800b0d0 tx=0x7ff228009e30 comp rx=0 tx=0).stop 2024-03-30T04:30:31.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.407+0000 7ff240a00700 1 -- 172.21.15.123:0/3285690741 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff23c0709a0 msgr2=0x7ff23c10faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:30:31.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.407+0000 7ff240a00700 1 --2- 172.21.15.123:0/3285690741 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff23c0709a0 0x7ff23c10faa0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ff22c00eb70 tx=0x7ff22c00b040 comp rx=0 tx=0).stop 2024-03-30T04:30:31.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.407+0000 7ff240a00700 1 -- 172.21.15.123:0/3285690741 shutdown_connections 2024-03-30T04:30:31.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.407+0000 7ff240a00700 1 --2- 172.21.15.123:0/3285690741 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff23c0709a0 0x7ff23c10faa0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:30:31.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.407+0000 7ff240a00700 1 --2- 172.21.15.123:0/3285690741 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7ff22006c8b0 0x7ff22006ed60 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:30:31.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.407+0000 7ff240a00700 1 --2- 172.21.15.123:0/3285690741 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff23c071f50 0x7ff23c10ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:30:31.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.407+0000 7ff240a00700 1 -- 172.21.15.123:0/3285690741 >> 172.21.15.123:0/3285690741 conn(0x7ff23c0f8560 msgr2=0x7ff23c0f9c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:30:31.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.407+0000 7ff240a00700 1 -- 172.21.15.123:0/3285690741 shutdown_connections 2024-03-30T04:30:31.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.408+0000 7ff240a00700 1 -- 172.21.15.123:0/3285690741 wait complete. 2024-03-30T04:30:31.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:31 smithi123 ceph-mon[35245]: pgmap v53: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.0 KiB/s rd, 682 B/s wr, 4 op/s 2024-03-30T04:30:31.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:31 smithi123 ceph-mon[35245]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:30:31.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.550+0000 7f3b84600700 1 -- 172.21.15.123:0/806975490 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b7c071f50 msgr2=0x7f3b7c070460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:30:31.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.550+0000 7f3b84600700 1 --2- 172.21.15.123:0/806975490 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b7c071f50 0x7f3b7c070460 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f3b70004940 tx=0x7f3b70009e30 comp rx=0 tx=0).stop 2024-03-30T04:30:31.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.550+0000 7f3b84600700 1 -- 172.21.15.123:0/806975490 shutdown_connections 2024-03-30T04:30:31.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.550+0000 7f3b84600700 1 --2- 172.21.15.123:0/806975490 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b7c0709a0 0x7f3b7c070e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:30:31.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.550+0000 7f3b84600700 1 --2- 172.21.15.123:0/806975490 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b7c071f50 0x7f3b7c070460 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:30:31.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.550+0000 7f3b84600700 1 -- 172.21.15.123:0/806975490 >> 172.21.15.123:0/806975490 conn(0x7f3b7c0f8560 msgr2=0x7f3b7c0fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:30:31.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.550+0000 7f3b84600700 1 -- 172.21.15.123:0/806975490 shutdown_connections 2024-03-30T04:30:31.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.550+0000 7f3b84600700 1 -- 172.21.15.123:0/806975490 wait complete. 2024-03-30T04:30:31.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.551+0000 7f3b84600700 1 Processor -- start 2024-03-30T04:30:31.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.551+0000 7f3b84600700 1 -- start start 2024-03-30T04:30:31.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.551+0000 7f3b84600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b7c0709a0 0x7f3b7c113f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:30:31.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.551+0000 7f3b84600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b7c071f50 0x7f3b7c1144a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:30:31.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.551+0000 7f3b84600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b7c114b30 con 0x7f3b7c0709a0 2024-03-30T04:30:31.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.551+0000 7f3b84600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b7c115320 con 0x7f3b7c071f50 2024-03-30T04:30:31.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.551+0000 7f3b81c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b7c0709a0 0x7f3b7c113f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:30:31.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.552+0000 7f3b81c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b7c0709a0 0x7f3b7c113f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55146/0 (socket says 172.21.15.123:55146) 2024-03-30T04:30:31.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.552+0000 7f3b81c00700 1 -- 172.21.15.123:0/1269834748 learned_addr learned my addr 172.21.15.123:0/1269834748 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:30:31.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.552+0000 7f3b81200700 1 --2- 172.21.15.123:0/1269834748 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b7c071f50 0x7f3b7c1144a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:30:31.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.552+0000 7f3b81c00700 1 -- 172.21.15.123:0/1269834748 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b7c071f50 msgr2=0x7f3b7c1144a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:30:31.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.552+0000 7f3b81c00700 1 --2- 172.21.15.123:0/1269834748 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b7c071f50 0x7f3b7c1144a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:30:31.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.552+0000 7f3b81c00700 1 -- 172.21.15.123:0/1269834748 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b70005040 con 0x7f3b7c0709a0 2024-03-30T04:30:31.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.552+0000 7f3b81c00700 1 --2- 172.21.15.123:0/1269834748 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b7c0709a0 0x7f3b7c113f60 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f3b7000b3b0 tx=0x7f3b70009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:30:31.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.552+0000 7f3b79600700 1 -- 172.21.15.123:0/1269834748 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b7000e040 con 0x7f3b7c0709a0 2024-03-30T04:30:31.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.553+0000 7f3b79600700 1 -- 172.21.15.123:0/1269834748 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3b70004290 con 0x7f3b7c0709a0 2024-03-30T04:30:31.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.553+0000 7f3b79600700 1 -- 172.21.15.123:0/1269834748 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b7001c3f0 con 0x7f3b7c0709a0 2024-03-30T04:30:31.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.553+0000 7f3b84600700 1 -- 172.21.15.123:0/1269834748 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b7c1155a0 con 0x7f3b7c0709a0 2024-03-30T04:30:31.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.553+0000 7f3b84600700 1 -- 172.21.15.123:0/1269834748 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b7c115b60 con 0x7f3b7c0709a0 2024-03-30T04:30:31.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.554+0000 7f3b84600700 1 -- 172.21.15.123:0/1269834748 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b7c05f5f0 con 0x7f3b7c0709a0 2024-03-30T04:30:31.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.555+0000 7f3b79600700 1 -- 172.21.15.123:0/1269834748 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 32) v1 ==== 90705+0+0 (secure 0 0 0) 0x7f3b7000bc50 con 0x7f3b7c0709a0 2024-03-30T04:30:31.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.556+0000 7f3b79600700 1 --2- 172.21.15.123:0/1269834748 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f3b6406c8b0 0x7f3b6406ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:30:31.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.556+0000 7f3b79600700 1 -- 172.21.15.123:0/1269834748 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f3b7008cd90 con 0x7f3b7c0709a0 2024-03-30T04:30:31.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.556+0000 7f3b81200700 1 --2- 172.21.15.123:0/1269834748 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f3b6406c8b0 0x7f3b6406ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:30:31.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.557+0000 7f3b81200700 1 --2- 172.21.15.123:0/1269834748 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f3b6406c8b0 0x7f3b6406ed60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3b68006fd0 tx=0x7f3b68008040 comp rx=0 tx=0).ready entity=mgr.14696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:30:31.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.561+0000 7f3b79600700 1 -- 172.21.15.123:0/1269834748 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f3b7005ae20 con 0x7f3b7c0709a0 2024-03-30T04:30:31.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:31 smithi135 ceph-mon[44364]: pgmap v53: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.0 KiB/s rd, 682 B/s wr, 4 op/s 2024-03-30T04:30:31.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:31 smithi135 ceph-mon[44364]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:30:31.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.869+0000 7f3b84600700 1 -- 172.21.15.123:0/1269834748 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f3b7c04bed0 con 0x7f3b7c0709a0 2024-03-30T04:30:31.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.870+0000 7f3b79600700 1 -- 172.21.15.123:0/1269834748 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+743 (secure 0 0 0) 0x7f3b7005a9b0 con 0x7f3b7c0709a0 2024-03-30T04:30:31.869 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:30:31.869 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:30:31.869 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 2 2024-03-30T04:30:31.869 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:30:31.869 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:30:31.869 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:30:31.869 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:30:31.869 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:30:31.869 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 6 2024-03-30T04:30:31.870 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:30:31.870 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:30:31.870 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 4 2024-03-30T04:30:31.870 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:30:31.870 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:30:31.870 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 12, 2024-03-30T04:30:31.870 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:30:31.870 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:30:31.870 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:30:31.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.873+0000 7f3b84600700 1 -- 172.21.15.123:0/1269834748 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f3b6406c8b0 msgr2=0x7f3b6406ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:30:31.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.873+0000 7f3b84600700 1 --2- 172.21.15.123:0/1269834748 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f3b6406c8b0 0x7f3b6406ed60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3b68006fd0 tx=0x7f3b68008040 comp rx=0 tx=0).stop 2024-03-30T04:30:31.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.873+0000 7f3b84600700 1 -- 172.21.15.123:0/1269834748 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b7c0709a0 msgr2=0x7f3b7c113f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:30:31.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.873+0000 7f3b84600700 1 --2- 172.21.15.123:0/1269834748 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b7c0709a0 0x7f3b7c113f60 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f3b7000b3b0 tx=0x7f3b70009e30 comp rx=0 tx=0).stop 2024-03-30T04:30:31.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.873+0000 7f3b84600700 1 -- 172.21.15.123:0/1269834748 shutdown_connections 2024-03-30T04:30:31.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.873+0000 7f3b84600700 1 --2- 172.21.15.123:0/1269834748 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b7c0709a0 0x7f3b7c113f60 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:30:31.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.873+0000 7f3b84600700 1 --2- 172.21.15.123:0/1269834748 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f3b6406c8b0 0x7f3b6406ed60 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:30:31.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.873+0000 7f3b84600700 1 --2- 172.21.15.123:0/1269834748 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b7c071f50 0x7f3b7c1144a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:30:31.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.873+0000 7f3b84600700 1 -- 172.21.15.123:0/1269834748 >> 172.21.15.123:0/1269834748 conn(0x7f3b7c0f8560 msgr2=0x7f3b7c0f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:30:31.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.873+0000 7f3b84600700 1 -- 172.21.15.123:0/1269834748 shutdown_connections 2024-03-30T04:30:31.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:31.873+0000 7f3b84600700 1 -- 172.21.15.123:0/1269834748 wait complete. 2024-03-30T04:30:32.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:32.018+0000 7f4152200700 1 -- 172.21.15.123:0/782494008 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f414c0fcc30 msgr2=0x7f414c0fd040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:30:32.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:32.018+0000 7f4152200700 1 --2- 172.21.15.123:0/782494008 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f414c0fcc30 0x7f414c0fd040 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f4140004940 tx=0x7f4140009e30 comp rx=0 tx=0).stop 2024-03-30T04:30:32.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:32.018+0000 7f4152200700 1 -- 172.21.15.123:0/782494008 shutdown_connections 2024-03-30T04:30:32.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:32.018+0000 7f4152200700 1 --2- 172.21.15.123:0/782494008 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f414c0fdee0 0x7f414c0fe350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:30:32.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:32.018+0000 7f4152200700 1 --2- 172.21.15.123:0/782494008 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f414c0fcc30 0x7f414c0fd040 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:30:32.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:32.018+0000 7f4152200700 1 -- 172.21.15.123:0/782494008 >> 172.21.15.123:0/782494008 conn(0x7f414c0f83d0 msgr2=0x7f414c0fa7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:30:32.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:32.018+0000 7f4152200700 1 -- 172.21.15.123:0/782494008 shutdown_connections 2024-03-30T04:30:32.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:32.018+0000 7f4152200700 1 -- 172.21.15.123:0/782494008 wait complete. 2024-03-30T04:30:32.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:32.019+0000 7f4152200700 1 Processor -- start 2024-03-30T04:30:32.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:32.019+0000 7f4152200700 1 -- start start 2024-03-30T04:30:32.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:32.019+0000 7f4152200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f414c0fcc30 0x7f414c10f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:30:32.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:32.019+0000 7f4152200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f414c0fdee0 0x7f414c10ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:30:32.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:32.019+0000 7f4152200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f414c110510 con 0x7f414c0fdee0 2024-03-30T04:30:32.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:32.019+0000 7f4152200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f414c110d60 con 0x7f414c0fcc30 2024-03-30T04:30:32.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:32.020+0000 7f414be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f414c0fdee0 0x7f414c10ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:30:32.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:32.020+0000 7f414be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f414c0fdee0 0x7f414c10ff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55160/0 (socket says 172.21.15.123:55160) 2024-03-30T04:30:32.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:32.020+0000 7f414be00700 1 -- 172.21.15.123:0/3739382466 learned_addr learned my addr 172.21.15.123:0/3739382466 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:30:32.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:32.020+0000 7f4150e00700 1 --2- 172.21.15.123:0/3739382466 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f414c0fcc30 0x7f414c10f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:30:32.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:32.020+0000 7f414be00700 1 -- 172.21.15.123:0/3739382466 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f414c0fcc30 msgr2=0x7f414c10f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:30:32.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:32.020+0000 7f414be00700 1 --2- 172.21.15.123:0/3739382466 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f414c0fcc30 0x7f414c10f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:30:32.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:32.020+0000 7f414be00700 1 -- 172.21.15.123:0/3739382466 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4140005040 con 0x7f414c0fdee0 2024-03-30T04:30:32.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:32.020+0000 7f4150e00700 1 --2- 172.21.15.123:0/3739382466 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f414c0fcc30 0x7f414c10f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:30:32.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:32.020+0000 7f414be00700 1 --2- 172.21.15.123:0/3739382466 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f414c0fdee0 0x7f414c10ff10 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f4138002a50 tx=0x7f41380046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:30:32.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:32.020+0000 7f4148c00700 1 -- 172.21.15.123:0/3739382466 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4138007cc0 con 0x7f414c0fdee0 2024-03-30T04:30:32.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:32.021+0000 7f4152200700 1 -- 172.21.15.123:0/3739382466 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f414c111040 con 0x7f414c0fdee0 2024-03-30T04:30:32.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:32.021+0000 7f4148c00700 1 -- 172.21.15.123:0/3739382466 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f413800f040 con 0x7f414c0fdee0 2024-03-30T04:30:32.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:32.021+0000 7f4152200700 1 -- 172.21.15.123:0/3739382466 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f414c111580 con 0x7f414c0fdee0 2024-03-30T04:30:32.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:32.021+0000 7f4148c00700 1 -- 172.21.15.123:0/3739382466 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f413800da30 con 0x7f414c0fdee0 2024-03-30T04:30:32.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:32.022+0000 7f4152200700 1 -- 172.21.15.123:0/3739382466 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f414c101870 con 0x7f414c0fdee0 2024-03-30T04:30:32.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:32.023+0000 7f4148c00700 1 -- 172.21.15.123:0/3739382466 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 32) v1 ==== 90705+0+0 (secure 0 0 0) 0x7f413800dbb0 con 0x7f414c0fdee0 2024-03-30T04:30:32.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:32.025+0000 7f4148c00700 1 --2- 172.21.15.123:0/3739382466 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f413406c8b0 0x7f413406ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:30:32.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:32.026+0000 7f4148c00700 1 -- 172.21.15.123:0/3739382466 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f413808b230 con 0x7f414c0fdee0 2024-03-30T04:30:32.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:32.026+0000 7f4150e00700 1 --2- 172.21.15.123:0/3739382466 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f413406c8b0 0x7f413406ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:30:32.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:32.026+0000 7f4150e00700 1 --2- 172.21.15.123:0/3739382466 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f413406c8b0 0x7f413406ed60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4140004f90 tx=0x7f4140009e30 comp rx=0 tx=0).ready entity=mgr.14696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:30:32.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:32.028+0000 7f4148c00700 1 -- 172.21.15.123:0/3739382466 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f4138059100 con 0x7f414c0fdee0 2024-03-30T04:30:32.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:32.267+0000 7f4152200700 1 -- 172.21.15.123:0/3739382466 --> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f414c1117f0 con 0x7f413406c8b0 2024-03-30T04:30:32.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:32.269+0000 7f4148c00700 1 -- 172.21.15.123:0/3739382466 <== mgr.14696 v2:172.21.15.123:6800/2125774379 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+367 (secure 0 0 0) 0x7f414c1117f0 con 0x7f413406c8b0 2024-03-30T04:30:32.267 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:30:32.267 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:30:32.267 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:30:32.267 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-30T04:30:32.267 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:30:32.268 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr" 2024-03-30T04:30:32.268 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:30:32.268 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "2/2 daemons upgraded", 2024-03-30T04:30:32.268 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading prometheus daemons", 2024-03-30T04:30:32.268 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:30:32.268 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:30:32.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:32.271+0000 7f4152200700 1 -- 172.21.15.123:0/3739382466 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f413406c8b0 msgr2=0x7f413406ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:30:32.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:32.271+0000 7f4152200700 1 --2- 172.21.15.123:0/3739382466 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f413406c8b0 0x7f413406ed60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4140004f90 tx=0x7f4140009e30 comp rx=0 tx=0).stop 2024-03-30T04:30:32.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:32.271+0000 7f4152200700 1 -- 172.21.15.123:0/3739382466 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f414c0fdee0 msgr2=0x7f414c10ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:30:32.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:32.271+0000 7f4152200700 1 --2- 172.21.15.123:0/3739382466 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f414c0fdee0 0x7f414c10ff10 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f4138002a50 tx=0x7f41380046b0 comp rx=0 tx=0).stop 2024-03-30T04:30:32.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:32.271+0000 7f4152200700 1 -- 172.21.15.123:0/3739382466 shutdown_connections 2024-03-30T04:30:32.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:32.271+0000 7f4152200700 1 --2- 172.21.15.123:0/3739382466 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f413406c8b0 0x7f413406ed60 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:30:32.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:32.271+0000 7f4152200700 1 --2- 172.21.15.123:0/3739382466 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f414c0fcc30 0x7f414c10f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:30:32.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:32.271+0000 7f4152200700 1 --2- 172.21.15.123:0/3739382466 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f414c0fdee0 0x7f414c10ff10 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:30:32.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:32.271+0000 7f4152200700 1 -- 172.21.15.123:0/3739382466 >> 172.21.15.123:0/3739382466 conn(0x7f414c0f83d0 msgr2=0x7f414c0f9b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:30:32.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:32.271+0000 7f4152200700 1 -- 172.21.15.123:0/3739382466 shutdown_connections 2024-03-30T04:30:32.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:30:32.271+0000 7f4152200700 1 -- 172.21.15.123:0/3739382466 wait complete. 2024-03-30T04:30:32.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:32 smithi123 ceph-mon[35245]: from='client.14782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:30:32.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:32 smithi123 ceph-mon[35245]: from='client.14786 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:30:32.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:32 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:32.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:32 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:32.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:32 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:30:32.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:32 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/1269834748' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:32.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:32 smithi135 ceph-mon[44364]: from='client.14782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:30:32.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:32 smithi135 ceph-mon[44364]: from='client.14786 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:30:32.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:32 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:32.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:32 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:32.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:32 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:30:32.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:32 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/1269834748' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:33.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:33 smithi123 ceph-mon[35245]: pgmap v54: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.8 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-30T04:30:33.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:33 smithi123 ceph-mon[35245]: from='client.14794 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:30:33.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:33 smithi135 ceph-mon[44364]: pgmap v54: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.8 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-30T04:30:33.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:33 smithi135 ceph-mon[44364]: from='client.14794 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:30:35.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:35 smithi135 ceph-mon[44364]: pgmap v55: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:30:35.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:35 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:30:35.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:35 smithi123 ceph-mon[35245]: pgmap v55: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:30:35.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:35 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:30:37.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:37 smithi135 ceph-mon[44364]: pgmap v56: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:30:37.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:37 smithi123 ceph-mon[35245]: pgmap v56: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:30:38.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:38 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:38.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:38 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:39.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:38 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:39.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:38 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:39.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:39 smithi123 ceph-mon[35245]: pgmap v57: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:30:39.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:39 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:39.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:39 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:40.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:39 smithi135 ceph-mon[44364]: pgmap v57: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:30:40.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:39 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:40.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:39 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:41.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:41 smithi123 ceph-mon[35245]: pgmap v58: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:30:42.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:41 smithi135 ceph-mon[44364]: pgmap v58: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:30:43.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:43 smithi123 ceph-mon[35245]: pgmap v59: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:30:43.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:43 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:43.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:43 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:43.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:43 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:30:43.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:43 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:30:43.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:43 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:44.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:43 smithi135 ceph-mon[44364]: pgmap v59: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:30:44.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:43 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:44.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:43 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:44.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:43 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:30:44.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:43 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:30:44.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:43 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:44.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:44 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-30T04:30:44.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:44 smithi123 ceph-mon[35245]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-30T04:30:44.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:44 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:30:44.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:44 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:44.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:44 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:44.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:44 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:44.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:44 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:44.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:44 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:44.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:44 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:44.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:44 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:44.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:44 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:44.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:44 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:44.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:44 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:45.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:44 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-30T04:30:45.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:44 smithi135 ceph-mon[44364]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-30T04:30:45.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:44 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:30:45.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:44 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:45.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:44 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:45.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:44 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:45.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:44 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:45.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:44 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:45.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:44 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:45.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:44 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:45.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:44 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:45.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:44 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:45.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:44 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:30:45.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:45 smithi123 ceph-mon[35245]: pgmap v60: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:30:46.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:45 smithi135 ceph-mon[44364]: pgmap v60: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:30:46.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:46 smithi123 ceph-mon[35245]: Upgrade: Updating alertmanager.smithi123 2024-03-30T04:30:46.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:46 smithi123 ceph-mon[35245]: Deploying daemon alertmanager.smithi123 on smithi123 2024-03-30T04:30:47.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:46 smithi135 ceph-mon[44364]: Upgrade: Updating alertmanager.smithi123 2024-03-30T04:30:47.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:46 smithi135 ceph-mon[44364]: Deploying daemon alertmanager.smithi123 on smithi123 2024-03-30T04:30:47.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:47 smithi123 ceph-mon[35245]: pgmap v61: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:30:48.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:47 smithi135 ceph-mon[44364]: pgmap v61: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:30:49.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:49 smithi123 ceph-mon[35245]: pgmap v62: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:30:50.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:49 smithi135 ceph-mon[44364]: pgmap v62: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:30:50.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:50 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:30:51.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:50 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:30:51.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:51 smithi123 ceph-mon[35245]: pgmap v63: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:30:52.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:51 smithi135 ceph-mon[44364]: pgmap v63: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:30:53.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:52 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:53.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:52 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:53.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:52 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:30:53.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:52 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:53.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:52 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:53.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:52 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:30:53.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:53 smithi123 ceph-mon[35245]: pgmap v64: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:30:54.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:53 smithi135 ceph-mon[44364]: pgmap v64: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:30:56.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:55 smithi135 ceph-mon[44364]: pgmap v65: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:30:56.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:55 smithi123 ceph-mon[35245]: pgmap v65: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:30:58.058 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:57 smithi123 ceph-mon[35245]: pgmap v66: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:30:58.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:57 smithi135 ceph-mon[44364]: pgmap v66: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:30:58.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:58 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:58.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:58 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:58.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:58 smithi123 ceph-mon[35245]: pgmap v67: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:30:59.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:58 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:59.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:58 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:30:59.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:58 smithi135 ceph-mon[44364]: pgmap v67: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:31:00.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:59 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:31:00.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:30:59 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:31:00.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:59 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:31:00.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:30:59 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:31:01.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:01 smithi135 ceph-mon[44364]: pgmap v68: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:31:01.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:00 smithi123 ceph-mon[35245]: pgmap v68: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:31:02.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.434+0000 7f2d54a00700 1 -- 172.21.15.123:0/1292361601 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d500fdff0 msgr2=0x7f2d500fe460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:31:02.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.434+0000 7f2d54a00700 1 --2- 172.21.15.123:0/1292361601 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d500fdff0 0x7f2d500fe460 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c004940 tx=0x7f2d3c009e30 comp rx=0 tx=0).stop 2024-03-30T04:31:02.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.434+0000 7f2d54a00700 1 -- 172.21.15.123:0/1292361601 shutdown_connections 2024-03-30T04:31:02.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.434+0000 7f2d54a00700 1 --2- 172.21.15.123:0/1292361601 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d500fdff0 0x7f2d500fe460 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:02.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.434+0000 7f2d54a00700 1 --2- 172.21.15.123:0/1292361601 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2d500fce80 0x7f2d500fd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:02.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.434+0000 7f2d54a00700 1 -- 172.21.15.123:0/1292361601 >> 172.21.15.123:0/1292361601 conn(0x7f2d500f8560 msgr2=0x7f2d500fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:31:02.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.434+0000 7f2d54a00700 1 -- 172.21.15.123:0/1292361601 shutdown_connections 2024-03-30T04:31:02.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.435+0000 7f2d54a00700 1 -- 172.21.15.123:0/1292361601 wait complete. 2024-03-30T04:31:02.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.435+0000 7f2d54a00700 1 Processor -- start 2024-03-30T04:31:02.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.436+0000 7f2d54a00700 1 -- start start 2024-03-30T04:31:02.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.436+0000 7f2d54a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2d500fce80 0x7f2d5010d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:31:02.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.436+0000 7f2d54a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d500fdff0 0x7f2d5010dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:31:02.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.436+0000 7f2d54a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5010e4a0 con 0x7f2d500fdff0 2024-03-30T04:31:02.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.436+0000 7f2d54a00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5010ecf0 con 0x7f2d500fce80 2024-03-30T04:31:02.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.436+0000 7f2d4d400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d500fdff0 0x7f2d5010dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:31:02.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.436+0000 7f2d4de00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2d500fce80 0x7f2d5010d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:31:02.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.436+0000 7f2d4de00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2d500fce80 0x7f2d5010d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.123:36064/0 (socket says 172.21.15.123:36064) 2024-03-30T04:31:02.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.436+0000 7f2d4de00700 1 -- 172.21.15.123:0/3541797562 learned_addr learned my addr 172.21.15.123:0/3541797562 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:31:02.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.437+0000 7f2d4de00700 1 -- 172.21.15.123:0/3541797562 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d500fdff0 msgr2=0x7f2d5010dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:31:02.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.437+0000 7f2d4de00700 1 --2- 172.21.15.123:0/3541797562 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d500fdff0 0x7f2d5010dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:02.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.437+0000 7f2d4de00700 1 -- 172.21.15.123:0/3541797562 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d3c005040 con 0x7f2d500fce80 2024-03-30T04:31:02.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.437+0000 7f2d4d400700 1 --2- 172.21.15.123:0/3541797562 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d500fdff0 0x7f2d5010dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:31:02.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.437+0000 7f2d4de00700 1 --2- 172.21.15.123:0/3541797562 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2d500fce80 0x7f2d5010d960 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2d40002a50 tx=0x7f2d400046b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:31:02.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.437+0000 7f2d46000700 1 -- 172.21.15.123:0/3541797562 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d40007cc0 con 0x7f2d500fce80 2024-03-30T04:31:02.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.437+0000 7f2d54a00700 1 -- 172.21.15.123:0/3541797562 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d5010efd0 con 0x7f2d500fce80 2024-03-30T04:31:02.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.437+0000 7f2d46000700 1 -- 172.21.15.123:0/3541797562 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2d4000f040 con 0x7f2d500fce80 2024-03-30T04:31:02.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.437+0000 7f2d46000700 1 -- 172.21.15.123:0/3541797562 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d4000d9c0 con 0x7f2d500fce80 2024-03-30T04:31:02.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.438+0000 7f2d54a00700 1 -- 172.21.15.123:0/3541797562 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d5010f510 con 0x7f2d500fce80 2024-03-30T04:31:02.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.439+0000 7f2d54a00700 1 -- 172.21.15.123:0/3541797562 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d5005f5f0 con 0x7f2d500fce80 2024-03-30T04:31:02.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.440+0000 7f2d46000700 1 -- 172.21.15.123:0/3541797562 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 32) v1 ==== 90705+0+0 (secure 0 0 0) 0x7f2d4000db60 con 0x7f2d500fce80 2024-03-30T04:31:02.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.441+0000 7f2d46000700 1 --2- 172.21.15.123:0/3541797562 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f2d3406c8b0 0x7f2d3406ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:31:02.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.441+0000 7f2d46000700 1 -- 172.21.15.123:0/3541797562 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f2d4008b100 con 0x7f2d500fce80 2024-03-30T04:31:02.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.441+0000 7f2d4d400700 1 --2- 172.21.15.123:0/3541797562 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f2d3406c8b0 0x7f2d3406ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:31:02.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.442+0000 7f2d4d400700 1 --2- 172.21.15.123:0/3541797562 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f2d3406c8b0 0x7f2d3406ed60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c004f90 tx=0x7f2d3c009e30 comp rx=0 tx=0).ready entity=mgr.14696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:31:02.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.446+0000 7f2d46000700 1 -- 172.21.15.123:0/3541797562 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f2d40059050 con 0x7f2d500fce80 2024-03-30T04:31:02.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.683+0000 7f2d54a00700 1 -- 172.21.15.123:0/3541797562 --> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2d5010f890 con 0x7f2d3406c8b0 2024-03-30T04:31:02.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.685+0000 7f2d46000700 1 -- 172.21.15.123:0/3541797562 <== mgr.14696 v2:172.21.15.123:6800/2125774379 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+369 (secure 0 0 0) 0x7f2d5010f890 con 0x7f2d3406c8b0 2024-03-30T04:31:02.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.687+0000 7f2d54a00700 1 -- 172.21.15.123:0/3541797562 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f2d3406c8b0 msgr2=0x7f2d3406ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:31:02.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.687+0000 7f2d54a00700 1 --2- 172.21.15.123:0/3541797562 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f2d3406c8b0 0x7f2d3406ed60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c004f90 tx=0x7f2d3c009e30 comp rx=0 tx=0).stop 2024-03-30T04:31:02.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.687+0000 7f2d54a00700 1 -- 172.21.15.123:0/3541797562 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2d500fce80 msgr2=0x7f2d5010d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:31:02.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.687+0000 7f2d54a00700 1 --2- 172.21.15.123:0/3541797562 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2d500fce80 0x7f2d5010d960 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2d40002a50 tx=0x7f2d400046b0 comp rx=0 tx=0).stop 2024-03-30T04:31:02.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.687+0000 7f2d54a00700 1 -- 172.21.15.123:0/3541797562 shutdown_connections 2024-03-30T04:31:02.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.687+0000 7f2d54a00700 1 --2- 172.21.15.123:0/3541797562 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f2d3406c8b0 0x7f2d3406ed60 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:02.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.687+0000 7f2d54a00700 1 --2- 172.21.15.123:0/3541797562 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2d500fce80 0x7f2d5010d960 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:02.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.687+0000 7f2d54a00700 1 --2- 172.21.15.123:0/3541797562 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d500fdff0 0x7f2d5010dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:02.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.687+0000 7f2d54a00700 1 -- 172.21.15.123:0/3541797562 >> 172.21.15.123:0/3541797562 conn(0x7f2d500f8560 msgr2=0x7f2d500f9d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:31:02.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.688+0000 7f2d54a00700 1 -- 172.21.15.123:0/3541797562 shutdown_connections 2024-03-30T04:31:02.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.688+0000 7f2d54a00700 1 -- 172.21.15.123:0/3541797562 wait complete. 2024-03-30T04:31:02.700 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:31:02.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.822+0000 7fdabd600700 1 -- 172.21.15.123:0/1200120584 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdab80fe0d0 msgr2=0x7fdab80fe540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:31:02.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.822+0000 7fdabd600700 1 --2- 172.21.15.123:0/1200120584 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdab80fe0d0 0x7fdab80fe540 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fdaa4004940 tx=0x7fdaa4009e30 comp rx=0 tx=0).stop 2024-03-30T04:31:02.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.822+0000 7fdabd600700 1 -- 172.21.15.123:0/1200120584 shutdown_connections 2024-03-30T04:31:02.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.822+0000 7fdabd600700 1 --2- 172.21.15.123:0/1200120584 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdab80fe0d0 0x7fdab80fe540 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:02.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.822+0000 7fdabd600700 1 --2- 172.21.15.123:0/1200120584 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdab80fcee0 0x7fdab80fd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:02.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.822+0000 7fdabd600700 1 -- 172.21.15.123:0/1200120584 >> 172.21.15.123:0/1200120584 conn(0x7fdab80f8560 msgr2=0x7fdab80fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:31:02.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.823+0000 7fdabd600700 1 -- 172.21.15.123:0/1200120584 shutdown_connections 2024-03-30T04:31:02.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.823+0000 7fdabd600700 1 -- 172.21.15.123:0/1200120584 wait complete. 2024-03-30T04:31:02.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.823+0000 7fdabd600700 1 Processor -- start 2024-03-30T04:31:02.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.824+0000 7fdabd600700 1 -- start start 2024-03-30T04:31:02.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.824+0000 7fdabd600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdab80fcee0 0x7fdab810fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:31:02.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.824+0000 7fdabd600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdab80fe0d0 0x7fdab81100c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:31:02.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.824+0000 7fdabd600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdab81106c0 con 0x7fdab80fcee0 2024-03-30T04:31:02.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.824+0000 7fdabd600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdab8110f10 con 0x7fdab80fe0d0 2024-03-30T04:31:02.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.824+0000 7fdab6a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdab80fcee0 0x7fdab810fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:31:02.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.824+0000 7fdab6a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdab80fcee0 0x7fdab810fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43536/0 (socket says 172.21.15.123:43536) 2024-03-30T04:31:02.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.824+0000 7fdab6a00700 1 -- 172.21.15.123:0/4273832240 learned_addr learned my addr 172.21.15.123:0/4273832240 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:31:02.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.824+0000 7fdab6000700 1 --2- 172.21.15.123:0/4273832240 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdab80fe0d0 0x7fdab81100c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:31:02.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.824+0000 7fdab6a00700 1 -- 172.21.15.123:0/4273832240 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdab80fe0d0 msgr2=0x7fdab81100c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:31:02.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.824+0000 7fdab6a00700 1 --2- 172.21.15.123:0/4273832240 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdab80fe0d0 0x7fdab81100c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:02.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.824+0000 7fdab6a00700 1 -- 172.21.15.123:0/4273832240 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaa4005040 con 0x7fdab80fcee0 2024-03-30T04:31:02.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.825+0000 7fdab6a00700 1 --2- 172.21.15.123:0/4273832240 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdab80fcee0 0x7fdab810fb80 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fdaa800eb70 tx=0x7fdaa800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:31:02.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.825+0000 7fdaaea00700 1 -- 172.21.15.123:0/4273832240 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdaa8007cc0 con 0x7fdab80fcee0 2024-03-30T04:31:02.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.825+0000 7fdaaea00700 1 -- 172.21.15.123:0/4273832240 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdaa800f040 con 0x7fdab80fcee0 2024-03-30T04:31:02.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.825+0000 7fdabd600700 1 -- 172.21.15.123:0/4273832240 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdab8111190 con 0x7fdab80fcee0 2024-03-30T04:31:02.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.825+0000 7fdaaea00700 1 -- 172.21.15.123:0/4273832240 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdaa800cad0 con 0x7fdab80fcee0 2024-03-30T04:31:02.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.825+0000 7fdabd600700 1 -- 172.21.15.123:0/4273832240 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdab81116d0 con 0x7fdab80fcee0 2024-03-30T04:31:02.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.826+0000 7fdabd600700 1 -- 172.21.15.123:0/4273832240 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdab805f5f0 con 0x7fdab80fcee0 2024-03-30T04:31:02.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.827+0000 7fdaaea00700 1 -- 172.21.15.123:0/4273832240 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 32) v1 ==== 90705+0+0 (secure 0 0 0) 0x7fdaa800cc30 con 0x7fdab80fcee0 2024-03-30T04:31:02.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.828+0000 7fdaaea00700 1 --2- 172.21.15.123:0/4273832240 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fda9c06c8b0 0x7fda9c06ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:31:02.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.828+0000 7fdaaea00700 1 -- 172.21.15.123:0/4273832240 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fdaa808b5b0 con 0x7fdab80fcee0 2024-03-30T04:31:02.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.828+0000 7fdab6000700 1 --2- 172.21.15.123:0/4273832240 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fda9c06c8b0 0x7fda9c06ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:31:02.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.829+0000 7fdab6000700 1 --2- 172.21.15.123:0/4273832240 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fda9c06c8b0 0x7fda9c06ed60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdaa4009fd0 tx=0x7fdaa4009e30 comp rx=0 tx=0).ready entity=mgr.14696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:31:02.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:02.833+0000 7fdaaea00700 1 -- 172.21.15.123:0/4273832240 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fdaa80595b0 con 0x7fdab80fcee0 2024-03-30T04:31:03.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.064+0000 7fdabd600700 1 -- 172.21.15.123:0/4273832240 --> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdab8111ab0 con 0x7fda9c06c8b0 2024-03-30T04:31:03.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.065+0000 7fdaaea00700 1 -- 172.21.15.123:0/4273832240 <== mgr.14696 v2:172.21.15.123:6800/2125774379 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+369 (secure 0 0 0) 0x7fdab8111ab0 con 0x7fda9c06c8b0 2024-03-30T04:31:03.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.067+0000 7fdabd600700 1 -- 172.21.15.123:0/4273832240 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fda9c06c8b0 msgr2=0x7fda9c06ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:31:03.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.067+0000 7fdabd600700 1 --2- 172.21.15.123:0/4273832240 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fda9c06c8b0 0x7fda9c06ed60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdaa4009fd0 tx=0x7fdaa4009e30 comp rx=0 tx=0).stop 2024-03-30T04:31:03.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.068+0000 7fdabd600700 1 -- 172.21.15.123:0/4273832240 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdab80fcee0 msgr2=0x7fdab810fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:31:03.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.068+0000 7fdabd600700 1 --2- 172.21.15.123:0/4273832240 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdab80fcee0 0x7fdab810fb80 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fdaa800eb70 tx=0x7fdaa800b040 comp rx=0 tx=0).stop 2024-03-30T04:31:03.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.068+0000 7fdabd600700 1 -- 172.21.15.123:0/4273832240 shutdown_connections 2024-03-30T04:31:03.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.068+0000 7fdabd600700 1 --2- 172.21.15.123:0/4273832240 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdab80fcee0 0x7fdab810fb80 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:03.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.068+0000 7fdabd600700 1 --2- 172.21.15.123:0/4273832240 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fda9c06c8b0 0x7fda9c06ed60 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:03.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.068+0000 7fdabd600700 1 --2- 172.21.15.123:0/4273832240 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdab80fe0d0 0x7fdab81100c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:03.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.068+0000 7fdabd600700 1 -- 172.21.15.123:0/4273832240 >> 172.21.15.123:0/4273832240 conn(0x7fdab80f8560 msgr2=0x7fdab80f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:31:03.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.068+0000 7fdabd600700 1 -- 172.21.15.123:0/4273832240 shutdown_connections 2024-03-30T04:31:03.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.068+0000 7fdabd600700 1 -- 172.21.15.123:0/4273832240 wait complete. 2024-03-30T04:31:03.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.204+0000 7fd363600700 1 -- 172.21.15.123:0/385948414 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd35c0fdff0 msgr2=0x7fd35c0fe460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:31:03.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.204+0000 7fd363600700 1 --2- 172.21.15.123:0/385948414 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd35c0fdff0 0x7fd35c0fe460 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fd348004940 tx=0x7fd348009e30 comp rx=0 tx=0).stop 2024-03-30T04:31:03.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.205+0000 7fd363600700 1 -- 172.21.15.123:0/385948414 shutdown_connections 2024-03-30T04:31:03.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.205+0000 7fd363600700 1 --2- 172.21.15.123:0/385948414 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd35c0fdff0 0x7fd35c0fe460 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:03.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.205+0000 7fd363600700 1 --2- 172.21.15.123:0/385948414 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd35c0fce80 0x7fd35c0fd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:03.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.205+0000 7fd363600700 1 -- 172.21.15.123:0/385948414 >> 172.21.15.123:0/385948414 conn(0x7fd35c0f8540 msgr2=0x7fd35c0fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:31:03.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.205+0000 7fd363600700 1 -- 172.21.15.123:0/385948414 shutdown_connections 2024-03-30T04:31:03.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.205+0000 7fd363600700 1 -- 172.21.15.123:0/385948414 wait complete. 2024-03-30T04:31:03.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.206+0000 7fd363600700 1 Processor -- start 2024-03-30T04:31:03.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.206+0000 7fd363600700 1 -- start start 2024-03-30T04:31:03.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.206+0000 7fd363600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd35c0fce80 0x7fd35c10faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:31:03.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.206+0000 7fd363600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd35c0fdff0 0x7fd35c10ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:31:03.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.206+0000 7fd363600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd35c1105e0 con 0x7fd35c0fce80 2024-03-30T04:31:03.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.206+0000 7fd363600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd35c110e30 con 0x7fd35c0fdff0 2024-03-30T04:31:03.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.206+0000 7fd360c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd35c0fce80 0x7fd35c10faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:31:03.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.206+0000 7fd360c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd35c0fce80 0x7fd35c10faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43558/0 (socket says 172.21.15.123:43558) 2024-03-30T04:31:03.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.206+0000 7fd360c00700 1 -- 172.21.15.123:0/2051978077 learned_addr learned my addr 172.21.15.123:0/2051978077 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:31:03.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.206+0000 7fd35be00700 1 --2- 172.21.15.123:0/2051978077 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd35c0fdff0 0x7fd35c10ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:31:03.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.207+0000 7fd360c00700 1 -- 172.21.15.123:0/2051978077 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd35c0fdff0 msgr2=0x7fd35c10ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:31:03.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.207+0000 7fd360c00700 1 --2- 172.21.15.123:0/2051978077 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd35c0fdff0 0x7fd35c10ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:03.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.207+0000 7fd360c00700 1 -- 172.21.15.123:0/2051978077 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd348005040 con 0x7fd35c0fce80 2024-03-30T04:31:03.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.207+0000 7fd360c00700 1 --2- 172.21.15.123:0/2051978077 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd35c0fce80 0x7fd35c10faa0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fd35000eb40 tx=0x7fd35000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:31:03.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.207+0000 7fd358c00700 1 -- 172.21.15.123:0/2051978077 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd350007cc0 con 0x7fd35c0fce80 2024-03-30T04:31:03.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.207+0000 7fd358c00700 1 -- 172.21.15.123:0/2051978077 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd35000f040 con 0x7fd35c0fce80 2024-03-30T04:31:03.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.207+0000 7fd363600700 1 -- 172.21.15.123:0/2051978077 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd35c1110b0 con 0x7fd35c0fce80 2024-03-30T04:31:03.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.207+0000 7fd358c00700 1 -- 172.21.15.123:0/2051978077 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd350009570 con 0x7fd35c0fce80 2024-03-30T04:31:03.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.207+0000 7fd363600700 1 -- 172.21.15.123:0/2051978077 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd35c1115f0 con 0x7fd35c0fce80 2024-03-30T04:31:03.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.208+0000 7fd363600700 1 -- 172.21.15.123:0/2051978077 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd35c101980 con 0x7fd35c0fce80 2024-03-30T04:31:03.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.209+0000 7fd358c00700 1 -- 172.21.15.123:0/2051978077 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 32) v1 ==== 90705+0+0 (secure 0 0 0) 0x7fd3500096d0 con 0x7fd35c0fce80 2024-03-30T04:31:03.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.210+0000 7fd358c00700 1 --2- 172.21.15.123:0/2051978077 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fd34406c8b0 0x7fd34406ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:31:03.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.210+0000 7fd358c00700 1 -- 172.21.15.123:0/2051978077 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fd35008b780 con 0x7fd35c0fce80 2024-03-30T04:31:03.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.210+0000 7fd35be00700 1 --2- 172.21.15.123:0/2051978077 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fd34406c8b0 0x7fd34406ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:31:03.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.211+0000 7fd35be00700 1 --2- 172.21.15.123:0/2051978077 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fd34406c8b0 0x7fd34406ed60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd34800b0d0 tx=0x7fd348009e30 comp rx=0 tx=0).ready entity=mgr.14696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:31:03.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.214+0000 7fd358c00700 1 -- 172.21.15.123:0/2051978077 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fd3500597a0 con 0x7fd35c0fce80 2024-03-30T04:31:03.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:03 smithi123 ceph-mon[35245]: pgmap v69: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:31:03.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:03 smithi123 ceph-mon[35245]: from='client.24603 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:31:03.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:03 smithi135 ceph-mon[44364]: pgmap v69: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:31:03.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:03 smithi135 ceph-mon[44364]: from='client.24603 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:31:03.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.455+0000 7fd363600700 1 -- 172.21.15.123:0/2051978077 --> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd35c111810 con 0x7fd34406c8b0 2024-03-30T04:31:03.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.465+0000 7fd358c00700 1 -- 172.21.15.123:0/2051978077 <== mgr.14696 v2:172.21.15.123:6800/2125774379 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fd35c111810 con 0x7fd34406c8b0 2024-03-30T04:31:03.464 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:31:03.464 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (12s) 5s ago 12m 14.2M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:31:03.465 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (13m) 5s ago 13m 6651k - 17.2.7-797-g5c024335 f77085f6aab7 bceb8b8c4837 2024-03-30T04:31:03.465 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (11m) 46s ago 11m 6651k - 17.2.7-797-g5c024335 f77085f6aab7 cd0fe67b2d34 2024-03-30T04:31:03.465 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (3m) 5s ago 12m 73.8M - 9.4.7 954c08fa6188 df1613fb18c2 2024-03-30T04:31:03.465 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (6m) 5s ago 6m 109M - 17.2.7-797-g5c024335 f77085f6aab7 2df066fba7a6 2024-03-30T04:31:03.465 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (6m) 5s ago 6m 57.6M - 17.2.7-797-g5c024335 f77085f6aab7 1293b8520db1 2024-03-30T04:31:03.465 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (6m) 46s ago 6m 67.5M - 17.2.7-797-g5c024335 f77085f6aab7 0dc41326818c 2024-03-30T04:31:03.465 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (6m) 46s ago 6m 119M - 17.2.7-797-g5c024335 f77085f6aab7 aef8a0cfc553 2024-03-30T04:31:03.465 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (2m) 5s ago 14m 500M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:31:03.465 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (90s) 46s ago 11m 446M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:31:03.465 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (14m) 5s ago 14m 52.8M 2048M 17.2.7-797-g5c024335 f77085f6aab7 51d18587a7df 2024-03-30T04:31:03.466 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (11m) 46s ago 11m 39.5M 2048M 17.2.7-797-g5c024335 f77085f6aab7 f81052b09dd8 2024-03-30T04:31:03.466 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (58s) 5s ago 12m 7596k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:31:03.466 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (53s) 46s ago 11m 6958k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:31:03.466 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (10m) 5s ago 10m 219M 1618M 17.2.7-797-g5c024335 f77085f6aab7 57a7ba3fd75e 2024-03-30T04:31:03.466 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (10m) 5s ago 10m 234M 1618M 17.2.7-797-g5c024335 f77085f6aab7 e1a6c1818a0c 2024-03-30T04:31:03.466 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (9m) 5s ago 9m 187M 1618M 17.2.7-797-g5c024335 f77085f6aab7 4c6a727567bd 2024-03-30T04:31:03.466 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (9m) 46s ago 9m 256M 2642M 17.2.7-797-g5c024335 f77085f6aab7 6fbdd1d46a9d 2024-03-30T04:31:03.466 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (8m) 46s ago 8m 224M 2642M 17.2.7-797-g5c024335 f77085f6aab7 b74988e947e5 2024-03-30T04:31:03.466 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (8m) 46s ago 8m 218M 2642M 17.2.7-797-g5c024335 f77085f6aab7 1fd3185f5484 2024-03-30T04:31:03.466 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (32s) 5s ago 12m 43.9M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:31:03.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.468+0000 7fd363600700 1 -- 172.21.15.123:0/2051978077 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fd34406c8b0 msgr2=0x7fd34406ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:31:03.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.468+0000 7fd363600700 1 --2- 172.21.15.123:0/2051978077 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fd34406c8b0 0x7fd34406ed60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd34800b0d0 tx=0x7fd348009e30 comp rx=0 tx=0).stop 2024-03-30T04:31:03.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.468+0000 7fd363600700 1 -- 172.21.15.123:0/2051978077 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd35c0fce80 msgr2=0x7fd35c10faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:31:03.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.468+0000 7fd363600700 1 --2- 172.21.15.123:0/2051978077 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd35c0fce80 0x7fd35c10faa0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fd35000eb40 tx=0x7fd35000b040 comp rx=0 tx=0).stop 2024-03-30T04:31:03.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.468+0000 7fd363600700 1 -- 172.21.15.123:0/2051978077 shutdown_connections 2024-03-30T04:31:03.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.468+0000 7fd363600700 1 --2- 172.21.15.123:0/2051978077 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd35c0fce80 0x7fd35c10faa0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:03.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.468+0000 7fd363600700 1 --2- 172.21.15.123:0/2051978077 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fd34406c8b0 0x7fd34406ed60 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:03.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.468+0000 7fd363600700 1 --2- 172.21.15.123:0/2051978077 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd35c0fdff0 0x7fd35c10ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:03.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.468+0000 7fd363600700 1 -- 172.21.15.123:0/2051978077 >> 172.21.15.123:0/2051978077 conn(0x7fd35c0f8540 msgr2=0x7fd35c0f9cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:31:03.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.468+0000 7fd363600700 1 -- 172.21.15.123:0/2051978077 shutdown_connections 2024-03-30T04:31:03.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.468+0000 7fd363600700 1 -- 172.21.15.123:0/2051978077 wait complete. 2024-03-30T04:31:03.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.608+0000 7f8107400700 1 -- 172.21.15.123:0/1600019480 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8100071e40 msgr2=0x7f8100070350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:31:03.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.608+0000 7f8107400700 1 --2- 172.21.15.123:0/1600019480 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8100071e40 0x7f8100070350 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f80f4004940 tx=0x7f80f4009e30 comp rx=0 tx=0).stop 2024-03-30T04:31:03.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.608+0000 7f8107400700 1 -- 172.21.15.123:0/1600019480 shutdown_connections 2024-03-30T04:31:03.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.608+0000 7f8107400700 1 --2- 172.21.15.123:0/1600019480 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8100070890 0x7f8100070d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:03.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.608+0000 7f8107400700 1 --2- 172.21.15.123:0/1600019480 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8100071e40 0x7f8100070350 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:03.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.608+0000 7f8107400700 1 -- 172.21.15.123:0/1600019480 >> 172.21.15.123:0/1600019480 conn(0x7f81000f8550 msgr2=0x7f81000fa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:31:03.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.609+0000 7f8107400700 1 -- 172.21.15.123:0/1600019480 shutdown_connections 2024-03-30T04:31:03.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.609+0000 7f8107400700 1 -- 172.21.15.123:0/1600019480 wait complete. 2024-03-30T04:31:03.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.610+0000 7f8107400700 1 Processor -- start 2024-03-30T04:31:03.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.610+0000 7f8107400700 1 -- start start 2024-03-30T04:31:03.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.610+0000 7f8107400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8100070890 0x7f810010fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:31:03.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.610+0000 7f8107400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8100071e40 0x7f810010ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:31:03.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.610+0000 7f8107400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8100110540 con 0x7f8100070890 2024-03-30T04:31:03.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.610+0000 7f8107400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81001106b0 con 0x7f8100071e40 2024-03-30T04:31:03.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.611+0000 7f8104a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8100070890 0x7f810010fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:31:03.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.611+0000 7f8104a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8100070890 0x7f810010fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43576/0 (socket says 172.21.15.123:43576) 2024-03-30T04:31:03.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.611+0000 7f8104a00700 1 -- 172.21.15.123:0/3083539314 learned_addr learned my addr 172.21.15.123:0/3083539314 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:31:03.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.611+0000 7f80ffe00700 1 --2- 172.21.15.123:0/3083539314 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8100071e40 0x7f810010ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:31:03.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.611+0000 7f8104a00700 1 -- 172.21.15.123:0/3083539314 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8100071e40 msgr2=0x7f810010ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:31:03.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.611+0000 7f8104a00700 1 --2- 172.21.15.123:0/3083539314 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8100071e40 0x7f810010ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:03.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.611+0000 7f8104a00700 1 -- 172.21.15.123:0/3083539314 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80f4005040 con 0x7f8100070890 2024-03-30T04:31:03.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.611+0000 7f8104a00700 1 --2- 172.21.15.123:0/3083539314 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8100070890 0x7f810010fa00 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f80f4009380 tx=0x7f80f4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:31:03.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.611+0000 7f80fcc00700 1 -- 172.21.15.123:0/3083539314 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f80f400e040 con 0x7f8100070890 2024-03-30T04:31:03.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.612+0000 7f80fcc00700 1 -- 172.21.15.123:0/3083539314 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f80f4004290 con 0x7f8100070890 2024-03-30T04:31:03.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.612+0000 7f80fcc00700 1 -- 172.21.15.123:0/3083539314 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f80f401c3f0 con 0x7f8100070890 2024-03-30T04:31:03.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.612+0000 7f8107400700 1 -- 172.21.15.123:0/3083539314 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81001150b0 con 0x7f8100070890 2024-03-30T04:31:03.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.612+0000 7f8107400700 1 -- 172.21.15.123:0/3083539314 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8100115670 con 0x7f8100070890 2024-03-30T04:31:03.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.613+0000 7f8107400700 1 -- 172.21.15.123:0/3083539314 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f810004c710 con 0x7f8100070890 2024-03-30T04:31:03.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.614+0000 7f80fcc00700 1 -- 172.21.15.123:0/3083539314 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 32) v1 ==== 90705+0+0 (secure 0 0 0) 0x7f80f4013420 con 0x7f8100070890 2024-03-30T04:31:03.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.615+0000 7f80fcc00700 1 --2- 172.21.15.123:0/3083539314 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f80e806c8a0 0x7f80e806ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:31:03.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.615+0000 7f80fcc00700 1 -- 172.21.15.123:0/3083539314 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f80f408ce10 con 0x7f8100070890 2024-03-30T04:31:03.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.615+0000 7f80ffe00700 1 --2- 172.21.15.123:0/3083539314 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f80e806c8a0 0x7f80e806ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:31:03.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.616+0000 7f80ffe00700 1 --2- 172.21.15.123:0/3083539314 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f80e806c8a0 0x7f80e806ed50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f80ec009fd0 tx=0x7f80ec009f60 comp rx=0 tx=0).ready entity=mgr.14696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:31:03.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.619+0000 7f80fcc00700 1 -- 172.21.15.123:0/3083539314 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f80f405ade0 con 0x7f8100070890 2024-03-30T04:31:03.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.918+0000 7f8107400700 1 -- 172.21.15.123:0/3083539314 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f8100115b20 con 0x7f8100070890 2024-03-30T04:31:03.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.919+0000 7f80fcc00700 1 -- 172.21.15.123:0/3083539314 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+743 (secure 0 0 0) 0x7f80f405a970 con 0x7f8100070890 2024-03-30T04:31:03.918 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:31:03.918 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:31:03.919 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 2 2024-03-30T04:31:03.919 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:31:03.919 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:31:03.919 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:31:03.919 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:31:03.919 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:31:03.919 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 6 2024-03-30T04:31:03.919 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:31:03.919 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:31:03.919 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 4 2024-03-30T04:31:03.919 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:31:03.919 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:31:03.919 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 12, 2024-03-30T04:31:03.919 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:31:03.920 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:31:03.920 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:31:03.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.921+0000 7f8107400700 1 -- 172.21.15.123:0/3083539314 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f80e806c8a0 msgr2=0x7f80e806ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:31:03.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.921+0000 7f8107400700 1 --2- 172.21.15.123:0/3083539314 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f80e806c8a0 0x7f80e806ed50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f80ec009fd0 tx=0x7f80ec009f60 comp rx=0 tx=0).stop 2024-03-30T04:31:03.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.921+0000 7f8107400700 1 -- 172.21.15.123:0/3083539314 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8100070890 msgr2=0x7f810010fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:31:03.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.921+0000 7f8107400700 1 --2- 172.21.15.123:0/3083539314 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8100070890 0x7f810010fa00 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f80f4009380 tx=0x7f80f4009e30 comp rx=0 tx=0).stop 2024-03-30T04:31:03.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.922+0000 7f8107400700 1 -- 172.21.15.123:0/3083539314 shutdown_connections 2024-03-30T04:31:03.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.922+0000 7f8107400700 1 --2- 172.21.15.123:0/3083539314 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8100070890 0x7f810010fa00 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:03.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.922+0000 7f8107400700 1 --2- 172.21.15.123:0/3083539314 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f80e806c8a0 0x7f80e806ed50 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:03.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.922+0000 7f8107400700 1 --2- 172.21.15.123:0/3083539314 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8100071e40 0x7f810010ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:03.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.922+0000 7f8107400700 1 -- 172.21.15.123:0/3083539314 >> 172.21.15.123:0/3083539314 conn(0x7f81000f8550 msgr2=0x7f81000f9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:31:03.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.922+0000 7f8107400700 1 -- 172.21.15.123:0/3083539314 shutdown_connections 2024-03-30T04:31:03.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:03.922+0000 7f8107400700 1 -- 172.21.15.123:0/3083539314 wait complete. 2024-03-30T04:31:04.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:04.068+0000 7ff435c00700 1 -- 172.21.15.123:0/1959089545 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4300fce80 msgr2=0x7ff4300fd290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:31:04.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:04.068+0000 7ff435c00700 1 --2- 172.21.15.123:0/1959089545 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4300fce80 0x7ff4300fd290 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7ff420004940 tx=0x7ff420009e30 comp rx=0 tx=0).stop 2024-03-30T04:31:04.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:04.068+0000 7ff435c00700 1 -- 172.21.15.123:0/1959089545 shutdown_connections 2024-03-30T04:31:04.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:04.068+0000 7ff435c00700 1 --2- 172.21.15.123:0/1959089545 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff4300fdff0 0x7ff4300fe460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:04.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:04.068+0000 7ff435c00700 1 --2- 172.21.15.123:0/1959089545 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4300fce80 0x7ff4300fd290 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:04.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:04.068+0000 7ff435c00700 1 -- 172.21.15.123:0/1959089545 >> 172.21.15.123:0/1959089545 conn(0x7ff4300f8560 msgr2=0x7ff4300fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:31:04.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:04.068+0000 7ff435c00700 1 -- 172.21.15.123:0/1959089545 shutdown_connections 2024-03-30T04:31:04.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:04.068+0000 7ff435c00700 1 -- 172.21.15.123:0/1959089545 wait complete. 2024-03-30T04:31:04.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:04.069+0000 7ff435c00700 1 Processor -- start 2024-03-30T04:31:04.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:04.069+0000 7ff435c00700 1 -- start start 2024-03-30T04:31:04.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:04.069+0000 7ff435c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4300fce80 0x7ff43010fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:31:04.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:04.069+0000 7ff435c00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff4300fdff0 0x7ff430110070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:31:04.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:04.069+0000 7ff435c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff430110670 con 0x7ff4300fce80 2024-03-30T04:31:04.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:04.069+0000 7ff435c00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff430110ec0 con 0x7ff4300fdff0 2024-03-30T04:31:04.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:04.069+0000 7ff42ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4300fce80 0x7ff43010fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:31:04.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:04.069+0000 7ff42ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4300fce80 0x7ff43010fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44400/0 (socket says 172.21.15.123:44400) 2024-03-30T04:31:04.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:04.069+0000 7ff42ea00700 1 -- 172.21.15.123:0/2893595506 learned_addr learned my addr 172.21.15.123:0/2893595506 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:31:04.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:04.070+0000 7ff427e00700 1 --2- 172.21.15.123:0/2893595506 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff4300fdff0 0x7ff430110070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:31:04.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:04.070+0000 7ff42ea00700 1 -- 172.21.15.123:0/2893595506 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff4300fdff0 msgr2=0x7ff430110070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:31:04.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:04.070+0000 7ff42ea00700 1 --2- 172.21.15.123:0/2893595506 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff4300fdff0 0x7ff430110070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:04.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:04.070+0000 7ff42ea00700 1 -- 172.21.15.123:0/2893595506 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff420005040 con 0x7ff4300fce80 2024-03-30T04:31:04.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:04.070+0000 7ff42ea00700 1 --2- 172.21.15.123:0/2893595506 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4300fce80 0x7ff43010fb30 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff42000b3b0 tx=0x7ff420009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:31:04.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:04.070+0000 7ff426a00700 1 -- 172.21.15.123:0/2893595506 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff42000e040 con 0x7ff4300fce80 2024-03-30T04:31:04.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:04.070+0000 7ff426a00700 1 -- 172.21.15.123:0/2893595506 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff420004290 con 0x7ff4300fce80 2024-03-30T04:31:04.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:04.070+0000 7ff435c00700 1 -- 172.21.15.123:0/2893595506 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff430111140 con 0x7ff4300fce80 2024-03-30T04:31:04.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:04.070+0000 7ff426a00700 1 -- 172.21.15.123:0/2893595506 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff42001c3f0 con 0x7ff4300fce80 2024-03-30T04:31:04.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:04.070+0000 7ff435c00700 1 -- 172.21.15.123:0/2893595506 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff430111680 con 0x7ff4300fce80 2024-03-30T04:31:04.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:04.072+0000 7ff435c00700 1 -- 172.21.15.123:0/2893595506 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff43005f5f0 con 0x7ff4300fce80 2024-03-30T04:31:04.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:04.072+0000 7ff426a00700 1 -- 172.21.15.123:0/2893595506 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 32) v1 ==== 90705+0+0 (secure 0 0 0) 0x7ff420013420 con 0x7ff4300fce80 2024-03-30T04:31:04.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:04.073+0000 7ff426a00700 1 --2- 172.21.15.123:0/2893595506 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7ff41406c8b0 0x7ff41406ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:31:04.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:04.073+0000 7ff426a00700 1 -- 172.21.15.123:0/2893595506 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7ff42008ce70 con 0x7ff4300fce80 2024-03-30T04:31:04.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:04.073+0000 7ff427e00700 1 --2- 172.21.15.123:0/2893595506 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7ff41406c8b0 0x7ff41406ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:31:04.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:04.074+0000 7ff427e00700 1 --2- 172.21.15.123:0/2893595506 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7ff41406c8b0 0x7ff41406ed60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff418004f40 tx=0x7ff41800b040 comp rx=0 tx=0).ready entity=mgr.14696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:31:04.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:04.078+0000 7ff426a00700 1 -- 172.21.15.123:0/2893595506 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7ff42005adc0 con 0x7ff4300fce80 2024-03-30T04:31:04.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:04.319+0000 7ff435c00700 1 -- 172.21.15.123:0/2893595506 --> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff430111a00 con 0x7ff41406c8b0 2024-03-30T04:31:04.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:04.321+0000 7ff426a00700 1 -- 172.21.15.123:0/2893595506 <== mgr.14696 v2:172.21.15.123:6800/2125774379 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+364 (secure 0 0 0) 0x7ff430111a00 con 0x7ff41406c8b0 2024-03-30T04:31:04.319 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:31:04.319 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:31:04.319 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:31:04.320 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-30T04:31:04.320 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:31:04.320 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr" 2024-03-30T04:31:04.320 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:31:04.320 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "2/2 daemons upgraded", 2024-03-30T04:31:04.320 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading grafana daemons", 2024-03-30T04:31:04.320 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:31:04.320 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:31:04.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:04.323+0000 7ff435c00700 1 -- 172.21.15.123:0/2893595506 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7ff41406c8b0 msgr2=0x7ff41406ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:31:04.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:04.323+0000 7ff435c00700 1 --2- 172.21.15.123:0/2893595506 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7ff41406c8b0 0x7ff41406ed60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff418004f40 tx=0x7ff41800b040 comp rx=0 tx=0).stop 2024-03-30T04:31:04.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:04.323+0000 7ff435c00700 1 -- 172.21.15.123:0/2893595506 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4300fce80 msgr2=0x7ff43010fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:31:04.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:04.323+0000 7ff435c00700 1 --2- 172.21.15.123:0/2893595506 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4300fce80 0x7ff43010fb30 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff42000b3b0 tx=0x7ff420009e30 comp rx=0 tx=0).stop 2024-03-30T04:31:04.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:04.324+0000 7ff435c00700 1 -- 172.21.15.123:0/2893595506 shutdown_connections 2024-03-30T04:31:04.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:04.324+0000 7ff435c00700 1 --2- 172.21.15.123:0/2893595506 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4300fce80 0x7ff43010fb30 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:04.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:04.324+0000 7ff435c00700 1 --2- 172.21.15.123:0/2893595506 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7ff41406c8b0 0x7ff41406ed60 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:04.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:04.324+0000 7ff435c00700 1 --2- 172.21.15.123:0/2893595506 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff4300fdff0 0x7ff430110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:04.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:04.324+0000 7ff435c00700 1 -- 172.21.15.123:0/2893595506 >> 172.21.15.123:0/2893595506 conn(0x7ff4300f8560 msgr2=0x7ff4300f9c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:31:04.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:04.324+0000 7ff435c00700 1 -- 172.21.15.123:0/2893595506 shutdown_connections 2024-03-30T04:31:04.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:04.324+0000 7ff435c00700 1 -- 172.21.15.123:0/2893595506 wait complete. 2024-03-30T04:31:04.562 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:04 smithi123 ceph-mon[35245]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:31:04.562 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:04 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:31:04.562 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:04 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:31:04.563 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:04 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:31:04.563 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:04 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:31:04.563 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:04 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:31:04.563 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:04 smithi123 ceph-mon[35245]: from='client.14806 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:31:04.563 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:04 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-30T04:31:04.563 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:04 smithi123 ceph-mon[35245]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-30T04:31:04.563 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:04 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:31:04.563 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:04 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:04.563 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:04 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:04.563 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:04 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:04.563 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:04 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:04.563 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:04 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:04.563 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:04 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:04.563 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:04 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:04.564 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:04 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:04.564 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:04 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:04.564 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:04 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:04.564 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:04 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:04.564 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:04 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/3083539314' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:04.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:04 smithi135 ceph-mon[44364]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:31:04.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:04 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:31:04.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:04 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:31:04.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:04 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:31:04.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:04 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:31:04.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:04 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:31:04.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:04 smithi135 ceph-mon[44364]: from='client.14806 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:31:04.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:04 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-30T04:31:04.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:04 smithi135 ceph-mon[44364]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-30T04:31:04.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:04 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:31:04.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:04 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:04.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:04 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:04.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:04 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:04.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:04 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:04.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:04 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:04.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:04 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:04.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:04 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:04.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:04 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:04.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:04 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:04.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:04 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:04.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:04 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:04.603 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:04 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/3083539314' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:05.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:05 smithi135 ceph-mon[44364]: pgmap v70: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:31:05.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:05 smithi135 ceph-mon[44364]: from='client.14814 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:31:05.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:05 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:31:05.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:05 smithi123 ceph-mon[35245]: pgmap v70: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:31:05.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:05 smithi123 ceph-mon[35245]: from='client.14814 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:31:05.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:05 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:31:06.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:06 smithi135 ceph-mon[44364]: Upgrade: Updating grafana.smithi123 2024-03-30T04:31:06.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:06 smithi135 ceph-mon[44364]: Deploying daemon grafana.smithi123 on smithi123 2024-03-30T04:31:06.666 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:06 smithi123 ceph-mon[35245]: Upgrade: Updating grafana.smithi123 2024-03-30T04:31:06.666 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:06 smithi123 ceph-mon[35245]: Deploying daemon grafana.smithi123 on smithi123 2024-03-30T04:31:07.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:07 smithi123 ceph-mon[35245]: pgmap v71: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:31:07.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:07 smithi135 ceph-mon[44364]: pgmap v71: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:31:09.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:09 smithi123 ceph-mon[35245]: pgmap v72: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:31:09.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:09 smithi135 ceph-mon[44364]: pgmap v72: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:31:12.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:11 smithi135 ceph-mon[44364]: pgmap v73: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:31:12.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:11 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:31:12.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:11 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:31:12.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:11 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:31:12.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:11 smithi123 ceph-mon[35245]: pgmap v73: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:31:12.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:11 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:31:12.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:11 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:31:12.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:11 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:31:13.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:13 smithi123 ceph-mon[35245]: pgmap v74: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:31:14.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:13 smithi135 ceph-mon[44364]: pgmap v74: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:31:15.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:15 smithi123 ceph-mon[35245]: pgmap v75: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:31:16.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:15 smithi135 ceph-mon[44364]: pgmap v75: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:31:18.058 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:17 smithi123 ceph-mon[35245]: pgmap v76: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:31:18.058 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:17 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:31:18.058 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:17 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:31:18.103 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:17 smithi135 ceph-mon[44364]: pgmap v76: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:31:18.103 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:17 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:31:18.103 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:17 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:31:19.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:19 smithi135 ceph-mon[44364]: pgmap v77: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:31:19.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:19 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:31:19.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:19 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:31:19.626 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:19 smithi123 ceph-mon[35245]: pgmap v77: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:31:19.626 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:19 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:31:19.626 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:19 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:31:20.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:20 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:31:20.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:20 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:31:21.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:21 smithi135 ceph-mon[44364]: pgmap v78: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:31:21.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:21 smithi123 ceph-mon[35245]: pgmap v78: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:31:23.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: pgmap v79: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:31:23.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:31:23.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:31:23.852 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:31:23.852 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:31:23.852 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:31:23.852 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-30T04:31:23.852 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-30T04:31:23.852 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:31:23.852 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:23.852 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:23.852 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:23.852 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:23.852 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:23.853 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:23.853 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:23.853 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:23.853 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:23.853 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:23.853 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:23.853 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:23.853 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:23.853 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:23.853 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: Upgrade: Finalizing container_image settings 2024-03-30T04:31:23.853 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:31:23.853 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-30T04:31:23.853 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-30T04:31:23.853 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-30T04:31:23.854 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-30T04:31:23.854 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-30T04:31:23.854 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-30T04:31:23.854 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-30T04:31:23.854 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-30T04:31:23.854 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-03-30T04:31:23.854 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-30T04:31:23.854 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-30T04:31:23.854 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-30T04:31:23.854 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-30T04:31:23.854 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-30T04:31:23.854 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-30T04:31:23.854 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-03-30T04:31:23.855 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-30T04:31:23.855 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-30T04:31:23.855 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-30T04:31:23.855 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-30T04:31:23.855 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-30T04:31:23.855 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-30T04:31:23.855 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-30T04:31:23.855 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-30T04:31:23.855 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-30T04:31:23.855 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-30T04:31:23.855 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-03-30T04:31:23.855 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2024-03-30T04:31:23.856 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-30T04:31:23.856 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-30T04:31:23.856 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:23 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:31:23.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: pgmap v79: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:31:23.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:31:23.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:31:23.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:31:23.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:31:23.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:31:23.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-30T04:31:23.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-30T04:31:23.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:31:23.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:23.968 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:23.968 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:23.968 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:23.968 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:23.968 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:23.968 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:23.968 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:23.968 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:23.968 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:23.968 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:23.968 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:23.968 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:23.968 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:23.968 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: Upgrade: Finalizing container_image settings 2024-03-30T04:31:23.969 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:31:23.969 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-30T04:31:23.969 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-30T04:31:23.969 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-30T04:31:23.969 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-30T04:31:23.969 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-30T04:31:23.969 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-30T04:31:23.969 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-30T04:31:23.969 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-30T04:31:23.969 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-03-30T04:31:23.969 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-30T04:31:23.969 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-30T04:31:23.969 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-30T04:31:23.969 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-30T04:31:23.969 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-30T04:31:23.970 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-30T04:31:23.970 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-03-30T04:31:23.970 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-30T04:31:23.970 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-30T04:31:23.970 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-30T04:31:23.970 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-30T04:31:23.970 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-30T04:31:23.970 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-30T04:31:23.970 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-30T04:31:23.970 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-30T04:31:23.970 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-30T04:31:23.970 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-30T04:31:23.970 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-03-30T04:31:23.971 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2024-03-30T04:31:23.971 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-30T04:31:23.971 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-30T04:31:23.971 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:23 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:31:24.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:24 smithi135 ceph-mon[44364]: Upgrade: Complete! 2024-03-30T04:31:24.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:24 smithi135 ceph-mon[44364]: fsmap cephfs:2 {0=cephfs.smithi123.dvtind=up:active,1=cephfs.smithi135.zvnlap=up:active} 2 up:standby-replay 2024-03-30T04:31:24.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:24 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:31:24.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:24 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:31:24.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:24 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:31:24.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:24 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:31:24.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:24 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:31:24.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:24 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:31:24.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:24 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:31:24.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:24 smithi123 ceph-mon[35245]: Upgrade: Complete! 2024-03-30T04:31:24.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:24 smithi123 ceph-mon[35245]: fsmap cephfs:2 {0=cephfs.smithi123.dvtind=up:active,1=cephfs.smithi135.zvnlap=up:active} 2 up:standby-replay 2024-03-30T04:31:24.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:24 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:31:24.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:24 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:31:24.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:24 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:31:24.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:24 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:31:24.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:24 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:31:24.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:24 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:31:24.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:24 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:31:25.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:25 smithi135 ceph-mon[44364]: pgmap v80: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:31:25.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:25 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:31:25.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:25 smithi123 ceph-mon[35245]: pgmap v80: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:31:25.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:25 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:31:27.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:27 smithi135 ceph-mon[44364]: pgmap v81: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:31:27.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:27 smithi123 ceph-mon[35245]: pgmap v81: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:31:29.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:29 smithi135 ceph-mon[44364]: pgmap v82: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:31:29.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:29 smithi123 ceph-mon[35245]: pgmap v82: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:31:31.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:31 smithi135 ceph-mon[44364]: pgmap v83: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:31:31.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:31 smithi123 ceph-mon[35245]: pgmap v83: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:31:33.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:33 smithi135 ceph-mon[44364]: pgmap v84: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:31:33.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:33 smithi123 ceph-mon[35245]: pgmap v84: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:31:34.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:34.477+0000 7efd9e000700 1 -- 172.21.15.123:0/39937301 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd98071e40 msgr2=0x7efd98070350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:31:34.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:34.477+0000 7efd9e000700 1 --2- 172.21.15.123:0/39937301 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd98071e40 0x7efd98070350 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7efd88004940 tx=0x7efd88009e30 comp rx=0 tx=0).stop 2024-03-30T04:31:34.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:34.477+0000 7efd9e000700 1 -- 172.21.15.123:0/39937301 shutdown_connections 2024-03-30T04:31:34.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:34.477+0000 7efd9e000700 1 --2- 172.21.15.123:0/39937301 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efd98070890 0x7efd98070d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:34.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:34.477+0000 7efd9e000700 1 --2- 172.21.15.123:0/39937301 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd98071e40 0x7efd98070350 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:34.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:34.477+0000 7efd9e000700 1 -- 172.21.15.123:0/39937301 >> 172.21.15.123:0/39937301 conn(0x7efd980f8560 msgr2=0x7efd980fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:31:34.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:34.477+0000 7efd9e000700 1 -- 172.21.15.123:0/39937301 shutdown_connections 2024-03-30T04:31:34.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:34.477+0000 7efd9e000700 1 -- 172.21.15.123:0/39937301 wait complete. 2024-03-30T04:31:34.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:34.478+0000 7efd9e000700 1 Processor -- start 2024-03-30T04:31:34.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:34.478+0000 7efd9e000700 1 -- start start 2024-03-30T04:31:34.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:34.478+0000 7efd9e000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efd98070890 0x7efd9810fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:31:34.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:34.478+0000 7efd9e000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd98071e40 0x7efd981100d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:31:34.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:34.479+0000 7efd9e000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd981106d0 con 0x7efd98071e40 2024-03-30T04:31:34.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:34.479+0000 7efd9e000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd98110f20 con 0x7efd98070890 2024-03-30T04:31:34.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:34.479+0000 7efd96a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd98071e40 0x7efd981100d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:31:34.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:34.479+0000 7efd96a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd98071e40 0x7efd981100d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60322/0 (socket says 172.21.15.123:60322) 2024-03-30T04:31:34.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:34.479+0000 7efd96a00700 1 -- 172.21.15.123:0/3031556320 learned_addr learned my addr 172.21.15.123:0/3031556320 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:31:34.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:34.479+0000 7efd97400700 1 --2- 172.21.15.123:0/3031556320 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efd98070890 0x7efd9810fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:31:34.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:34.479+0000 7efd96a00700 1 -- 172.21.15.123:0/3031556320 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efd98070890 msgr2=0x7efd9810fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:31:34.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:34.479+0000 7efd96a00700 1 --2- 172.21.15.123:0/3031556320 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efd98070890 0x7efd9810fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:34.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:34.479+0000 7efd96a00700 1 -- 172.21.15.123:0/3031556320 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd88005040 con 0x7efd98071e40 2024-03-30T04:31:34.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:34.480+0000 7efd96a00700 1 --2- 172.21.15.123:0/3031556320 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd98071e40 0x7efd981100d0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7efd84002a50 tx=0x7efd840046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:31:34.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:34.480+0000 7efd8f400700 1 -- 172.21.15.123:0/3031556320 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd84007cc0 con 0x7efd98071e40 2024-03-30T04:31:34.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:34.480+0000 7efd8f400700 1 -- 172.21.15.123:0/3031556320 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efd8400f040 con 0x7efd98071e40 2024-03-30T04:31:34.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:34.480+0000 7efd8f400700 1 -- 172.21.15.123:0/3031556320 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd8400d9c0 con 0x7efd98071e40 2024-03-30T04:31:34.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:34.480+0000 7efd9e000700 1 -- 172.21.15.123:0/3031556320 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd98111200 con 0x7efd98071e40 2024-03-30T04:31:34.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:34.480+0000 7efd9e000700 1 -- 172.21.15.123:0/3031556320 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd98111740 con 0x7efd98071e40 2024-03-30T04:31:34.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:34.481+0000 7efd9e000700 1 -- 172.21.15.123:0/3031556320 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd9805f5f0 con 0x7efd98071e40 2024-03-30T04:31:34.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:34.482+0000 7efd8f400700 1 -- 172.21.15.123:0/3031556320 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 32) v1 ==== 90705+0+0 (secure 0 0 0) 0x7efd8400db20 con 0x7efd98071e40 2024-03-30T04:31:34.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:34.483+0000 7efd8f400700 1 --2- 172.21.15.123:0/3031556320 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7efd7c06c8b0 0x7efd7c06ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:31:34.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:34.483+0000 7efd8f400700 1 -- 172.21.15.123:0/3031556320 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7efd8408b7a0 con 0x7efd98071e40 2024-03-30T04:31:34.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:34.483+0000 7efd97400700 1 --2- 172.21.15.123:0/3031556320 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7efd7c06c8b0 0x7efd7c06ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:31:34.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:34.484+0000 7efd97400700 1 --2- 172.21.15.123:0/3031556320 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7efd7c06c8b0 0x7efd7c06ed60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7efd88000c00 tx=0x7efd88009e30 comp rx=0 tx=0).ready entity=mgr.14696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:31:34.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:34.488+0000 7efd8f400700 1 -- 172.21.15.123:0/3031556320 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7efd840596f0 con 0x7efd98071e40 2024-03-30T04:31:34.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:34.709+0000 7efd9e000700 1 -- 172.21.15.123:0/3031556320 --> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7efd98111b30 con 0x7efd7c06c8b0 2024-03-30T04:31:34.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:34.710+0000 7efd8f400700 1 -- 172.21.15.123:0/3031556320 <== mgr.14696 v2:172.21.15.123:6800/2125774379 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7efd98111b30 con 0x7efd7c06c8b0 2024-03-30T04:31:34.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:34.712+0000 7efd9e000700 1 -- 172.21.15.123:0/3031556320 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7efd7c06c8b0 msgr2=0x7efd7c06ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:31:34.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:34.712+0000 7efd9e000700 1 --2- 172.21.15.123:0/3031556320 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7efd7c06c8b0 0x7efd7c06ed60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7efd88000c00 tx=0x7efd88009e30 comp rx=0 tx=0).stop 2024-03-30T04:31:34.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:34.712+0000 7efd9e000700 1 -- 172.21.15.123:0/3031556320 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd98071e40 msgr2=0x7efd981100d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:31:34.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:34.712+0000 7efd9e000700 1 --2- 172.21.15.123:0/3031556320 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd98071e40 0x7efd981100d0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7efd84002a50 tx=0x7efd840046b0 comp rx=0 tx=0).stop 2024-03-30T04:31:34.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:34.713+0000 7efd9e000700 1 -- 172.21.15.123:0/3031556320 shutdown_connections 2024-03-30T04:31:34.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:34.713+0000 7efd9e000700 1 --2- 172.21.15.123:0/3031556320 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7efd7c06c8b0 0x7efd7c06ed60 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:34.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:34.713+0000 7efd9e000700 1 --2- 172.21.15.123:0/3031556320 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efd98070890 0x7efd9810fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:34.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:34.713+0000 7efd9e000700 1 --2- 172.21.15.123:0/3031556320 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd98071e40 0x7efd981100d0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:34.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:34.713+0000 7efd9e000700 1 -- 172.21.15.123:0/3031556320 >> 172.21.15.123:0/3031556320 conn(0x7efd980f8560 msgr2=0x7efd980f9d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:31:34.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:34.713+0000 7efd9e000700 1 -- 172.21.15.123:0/3031556320 shutdown_connections 2024-03-30T04:31:34.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:34.713+0000 7efd9e000700 1 -- 172.21.15.123:0/3031556320 wait complete. 2024-03-30T04:31:35.472 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -e sha1=470100a4ccbcf7769052bf09a5413881ccf1b97f -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2024-03-30T04:31:35.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:35 smithi123 ceph-mon[35245]: pgmap v85: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:31:35.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:35 smithi123 ceph-mon[35245]: from='client.14818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:31:35.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:35 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:31:35.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:35 smithi135 ceph-mon[44364]: pgmap v85: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:31:35.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:35 smithi135 ceph-mon[44364]: from='client.14818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:31:35.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:35 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:31:35.899 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:31:37.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:37.820+0000 7f2840a00700 1 -- 172.21.15.123:0/2991357829 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f283c070c10 msgr2=0x7f283c109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:31:37.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:37.820+0000 7f2840a00700 1 --2- 172.21.15.123:0/2991357829 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f283c070c10 0x7f283c109190 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f2828004970 tx=0x7f2828009e30 comp rx=0 tx=0).stop 2024-03-30T04:31:37.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:37.820+0000 7f2840a00700 1 -- 172.21.15.123:0/2991357829 shutdown_connections 2024-03-30T04:31:37.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:37.820+0000 7f2840a00700 1 --2- 172.21.15.123:0/2991357829 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f283c070c10 0x7f283c109190 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:37.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:37.820+0000 7f2840a00700 1 --2- 172.21.15.123:0/2991357829 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f283c070300 0x7f283c0706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:37.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:37.820+0000 7f2840a00700 1 -- 172.21.15.123:0/2991357829 >> 172.21.15.123:0/2991357829 conn(0x7f283c0f89d0 msgr2=0x7f283c0fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:31:37.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:37.820+0000 7f2840a00700 1 -- 172.21.15.123:0/2991357829 shutdown_connections 2024-03-30T04:31:37.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:37.820+0000 7f2840a00700 1 -- 172.21.15.123:0/2991357829 wait complete. 2024-03-30T04:31:37.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:37.821+0000 7f2840a00700 1 Processor -- start 2024-03-30T04:31:37.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:37.821+0000 7f2840a00700 1 -- start start 2024-03-30T04:31:37.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:37.821+0000 7f2840a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f283c070300 0x7f283c115ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:31:37.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:37.821+0000 7f2840a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f283c070c10 0x7f283c1161e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:31:37.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:37.821+0000 7f2840a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f283c116830 con 0x7f283c070300 2024-03-30T04:31:37.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:37.821+0000 7f2840a00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f283c10fc00 con 0x7f283c070c10 2024-03-30T04:31:37.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:37.821+0000 7f2839e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f283c070300 0x7f283c115ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:31:37.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:37.821+0000 7f2839e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f283c070300 0x7f283c115ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60336/0 (socket says 172.21.15.123:60336) 2024-03-30T04:31:37.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:37.821+0000 7f2839e00700 1 -- 172.21.15.123:0/3540230256 learned_addr learned my addr 172.21.15.123:0/3540230256 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:31:37.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:37.822+0000 7f2839400700 1 --2- 172.21.15.123:0/3540230256 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f283c070c10 0x7f283c1161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:31:37.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:37.822+0000 7f2839e00700 1 -- 172.21.15.123:0/3540230256 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f283c070c10 msgr2=0x7f283c1161e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:31:37.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:37.822+0000 7f2839e00700 1 --2- 172.21.15.123:0/3540230256 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f283c070c10 0x7f283c1161e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:37.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:37.822+0000 7f2839e00700 1 -- 172.21.15.123:0/3540230256 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2828005040 con 0x7f283c070300 2024-03-30T04:31:37.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:37.822+0000 7f2839400700 1 --2- 172.21.15.123:0/3540230256 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f283c070c10 0x7f283c1161e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:31:37.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:37.822+0000 7f2839e00700 1 --2- 172.21.15.123:0/3540230256 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f283c070300 0x7f283c115ca0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f282c002a50 tx=0x7f282c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:31:37.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:37.822+0000 7f2832000700 1 -- 172.21.15.123:0/3540230256 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f282c007cc0 con 0x7f283c070300 2024-03-30T04:31:37.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:37.822+0000 7f2832000700 1 -- 172.21.15.123:0/3540230256 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f282c00f040 con 0x7f283c070300 2024-03-30T04:31:37.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:37.822+0000 7f2840a00700 1 -- 172.21.15.123:0/3540230256 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f283c10fee0 con 0x7f283c070300 2024-03-30T04:31:37.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:37.822+0000 7f2832000700 1 -- 172.21.15.123:0/3540230256 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f282c009540 con 0x7f283c070300 2024-03-30T04:31:37.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:37.823+0000 7f2840a00700 1 -- 172.21.15.123:0/3540230256 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f283c110420 con 0x7f283c070300 2024-03-30T04:31:37.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:37.823+0000 7f2840a00700 1 -- 172.21.15.123:0/3540230256 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f283c071a40 con 0x7f283c070300 2024-03-30T04:31:37.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:37.825+0000 7f2832000700 1 -- 172.21.15.123:0/3540230256 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 32) v1 ==== 90705+0+0 (secure 0 0 0) 0x7f282c0096a0 con 0x7f283c070300 2024-03-30T04:31:37.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:37.826+0000 7f2832000700 1 --2- 172.21.15.123:0/3540230256 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f282006c8b0 0x7f282006ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:31:37.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:37.826+0000 7f2832000700 1 -- 172.21.15.123:0/3540230256 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f282c08b7c0 con 0x7f283c070300 2024-03-30T04:31:37.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:37.826+0000 7f2839400700 1 --2- 172.21.15.123:0/3540230256 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f282006c8b0 0x7f282006ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:31:37.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:37.827+0000 7f2839400700 1 --2- 172.21.15.123:0/3540230256 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f282006c8b0 0x7f282006ed60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f282800b800 tx=0x7f2828009e30 comp rx=0 tx=0).ready entity=mgr.14696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:31:37.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:37.831+0000 7f2832000700 1 -- 172.21.15.123:0/3540230256 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f282c0597b0 con 0x7f283c070300 2024-03-30T04:31:37.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:37 smithi135 ceph-mon[44364]: pgmap v86: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:31:37.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:37 smithi123 ceph-mon[35245]: pgmap v86: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:31:38.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:38.114+0000 7f2840a00700 1 -- 172.21.15.123:0/3540230256 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f283c04bf00 con 0x7f283c070300 2024-03-30T04:31:38.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:38.114+0000 7f2832000700 1 -- 172.21.15.123:0/3540230256 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+743 (secure 0 0 0) 0x7f282c059340 con 0x7f283c070300 2024-03-30T04:31:38.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:38.117+0000 7f2840a00700 1 -- 172.21.15.123:0/3540230256 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f282006c8b0 msgr2=0x7f282006ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:31:38.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:38.117+0000 7f2840a00700 1 --2- 172.21.15.123:0/3540230256 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f282006c8b0 0x7f282006ed60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f282800b800 tx=0x7f2828009e30 comp rx=0 tx=0).stop 2024-03-30T04:31:38.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:38.117+0000 7f2840a00700 1 -- 172.21.15.123:0/3540230256 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f283c070300 msgr2=0x7f283c115ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:31:38.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:38.117+0000 7f2840a00700 1 --2- 172.21.15.123:0/3540230256 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f283c070300 0x7f283c115ca0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f282c002a50 tx=0x7f282c0046b0 comp rx=0 tx=0).stop 2024-03-30T04:31:38.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:38.117+0000 7f2840a00700 1 -- 172.21.15.123:0/3540230256 shutdown_connections 2024-03-30T04:31:38.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:38.117+0000 7f2840a00700 1 --2- 172.21.15.123:0/3540230256 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f283c070300 0x7f283c115ca0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:38.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:38.117+0000 7f2840a00700 1 --2- 172.21.15.123:0/3540230256 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f282006c8b0 0x7f282006ed60 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:38.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:38.117+0000 7f2840a00700 1 --2- 172.21.15.123:0/3540230256 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f283c070c10 0x7f283c1161e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:38.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:38.117+0000 7f2840a00700 1 -- 172.21.15.123:0/3540230256 >> 172.21.15.123:0/3540230256 conn(0x7f283c0f89d0 msgr2=0x7f283c0f9f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:31:38.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:38.117+0000 7f2840a00700 1 -- 172.21.15.123:0/3540230256 shutdown_connections 2024-03-30T04:31:38.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:38.117+0000 7f2840a00700 1 -- 172.21.15.123:0/3540230256 wait complete. 2024-03-30T04:31:38.130 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:31:38.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:38 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/3540230256' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:38.812 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -e sha1=470100a4ccbcf7769052bf09a5413881ccf1b97f -- bash -c 'ceph versions | jq -e '"'"'.mgr | keys'"'"' | grep $sha1' 2024-03-30T04:31:38.853 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:38 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/3540230256' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:39.239 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:31:39.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:39 smithi135 ceph-mon[44364]: pgmap v87: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:31:39.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:39 smithi123 ceph-mon[35245]: pgmap v87: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:31:41.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:41.265+0000 7ff78b200700 1 -- 172.21.15.123:0/624149277 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff78c102c40 msgr2=0x7ff78c103010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:31:41.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:41.265+0000 7ff78b200700 1 --2- 172.21.15.123:0/624149277 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff78c102c40 0x7ff78c103010 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7ff77c004940 tx=0x7ff77c009e30 comp rx=0 tx=0).stop 2024-03-30T04:31:41.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:41.265+0000 7ff78b200700 1 -- 172.21.15.123:0/624149277 shutdown_connections 2024-03-30T04:31:41.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:41.265+0000 7ff78b200700 1 --2- 172.21.15.123:0/624149277 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff78c1035e0 0x7ff78c0fcc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:41.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:41.265+0000 7ff78b200700 1 --2- 172.21.15.123:0/624149277 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff78c102c40 0x7ff78c103010 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:41.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:41.265+0000 7ff78b200700 1 -- 172.21.15.123:0/624149277 >> 172.21.15.123:0/624149277 conn(0x7ff78c0f8830 msgr2=0x7ff78c0fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:31:41.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:41.265+0000 7ff78b200700 1 -- 172.21.15.123:0/624149277 shutdown_connections 2024-03-30T04:31:41.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:41.265+0000 7ff78b200700 1 -- 172.21.15.123:0/624149277 wait complete. 2024-03-30T04:31:41.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:41.266+0000 7ff78b200700 1 Processor -- start 2024-03-30T04:31:41.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:41.267+0000 7ff78b200700 1 -- start start 2024-03-30T04:31:41.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:41.267+0000 7ff78b200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff78c102c40 0x7ff78c113870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:31:41.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:41.267+0000 7ff78b200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff78c1035e0 0x7ff78c113db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:31:41.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:41.267+0000 7ff78b200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff78c114400 con 0x7ff78c1035e0 2024-03-30T04:31:41.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:41.267+0000 7ff78b200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff78c10d7f0 con 0x7ff78c102c40 2024-03-30T04:31:41.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:41.268+0000 7ff789400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff78c1035e0 0x7ff78c113db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:31:41.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:41.268+0000 7ff789400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff78c1035e0 0x7ff78c113db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60342/0 (socket says 172.21.15.123:60342) 2024-03-30T04:31:41.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:41.268+0000 7ff789400700 1 -- 172.21.15.123:0/2251817451 learned_addr learned my addr 172.21.15.123:0/2251817451 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:31:41.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:41.268+0000 7ff789e00700 1 --2- 172.21.15.123:0/2251817451 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff78c102c40 0x7ff78c113870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:31:41.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:41.268+0000 7ff789400700 1 -- 172.21.15.123:0/2251817451 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff78c102c40 msgr2=0x7ff78c113870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:31:41.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:41.268+0000 7ff789400700 1 --2- 172.21.15.123:0/2251817451 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff78c102c40 0x7ff78c113870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:41.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:41.268+0000 7ff789400700 1 -- 172.21.15.123:0/2251817451 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff77c005040 con 0x7ff78c1035e0 2024-03-30T04:31:41.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:41.268+0000 7ff789e00700 1 --2- 172.21.15.123:0/2251817451 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff78c102c40 0x7ff78c113870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:31:41.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:41.268+0000 7ff789400700 1 --2- 172.21.15.123:0/2251817451 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff78c1035e0 0x7ff78c113db0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7ff77800eb40 tx=0x7ff77800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:31:41.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:41.268+0000 7ff782000700 1 -- 172.21.15.123:0/2251817451 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff778007cc0 con 0x7ff78c1035e0 2024-03-30T04:31:41.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:41.269+0000 7ff78b200700 1 -- 172.21.15.123:0/2251817451 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff78c10da70 con 0x7ff78c1035e0 2024-03-30T04:31:41.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:41.269+0000 7ff782000700 1 -- 172.21.15.123:0/2251817451 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff77800f040 con 0x7ff78c1035e0 2024-03-30T04:31:41.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:41.269+0000 7ff782000700 1 -- 172.21.15.123:0/2251817451 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff778009570 con 0x7ff78c1035e0 2024-03-30T04:31:41.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:41.269+0000 7ff78b200700 1 -- 172.21.15.123:0/2251817451 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff78c10e030 con 0x7ff78c1035e0 2024-03-30T04:31:41.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:41.270+0000 7ff78b200700 1 -- 172.21.15.123:0/2251817451 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff78c0fe390 con 0x7ff78c1035e0 2024-03-30T04:31:41.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:41.271+0000 7ff782000700 1 -- 172.21.15.123:0/2251817451 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 32) v1 ==== 90705+0+0 (secure 0 0 0) 0x7ff7780096d0 con 0x7ff78c1035e0 2024-03-30T04:31:41.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:41.271+0000 7ff782000700 1 --2- 172.21.15.123:0/2251817451 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7ff77006c8b0 0x7ff77006ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:31:41.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:41.272+0000 7ff782000700 1 -- 172.21.15.123:0/2251817451 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7ff77808b4e0 con 0x7ff78c1035e0 2024-03-30T04:31:41.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:41.272+0000 7ff789e00700 1 --2- 172.21.15.123:0/2251817451 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7ff77006c8b0 0x7ff77006ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:31:41.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:41.272+0000 7ff789e00700 1 --2- 172.21.15.123:0/2251817451 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7ff77006c8b0 0x7ff77006ed60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff77c00b0d0 tx=0x7ff77c009e30 comp rx=0 tx=0).ready entity=mgr.14696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:31:41.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:41.276+0000 7ff782000700 1 -- 172.21.15.123:0/2251817451 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7ff778059580 con 0x7ff78c1035e0 2024-03-30T04:31:41.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:41.561+0000 7ff78b200700 1 -- 172.21.15.123:0/2251817451 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ff78c05f5f0 con 0x7ff78c1035e0 2024-03-30T04:31:41.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:41.562+0000 7ff782000700 1 -- 172.21.15.123:0/2251817451 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+743 (secure 0 0 0) 0x7ff778059110 con 0x7ff78c1035e0 2024-03-30T04:31:41.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:41.564+0000 7ff78b200700 1 -- 172.21.15.123:0/2251817451 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7ff77006c8b0 msgr2=0x7ff77006ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:31:41.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:41.564+0000 7ff78b200700 1 --2- 172.21.15.123:0/2251817451 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7ff77006c8b0 0x7ff77006ed60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff77c00b0d0 tx=0x7ff77c009e30 comp rx=0 tx=0).stop 2024-03-30T04:31:41.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:41.564+0000 7ff78b200700 1 -- 172.21.15.123:0/2251817451 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff78c1035e0 msgr2=0x7ff78c113db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:31:41.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:41.564+0000 7ff78b200700 1 --2- 172.21.15.123:0/2251817451 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff78c1035e0 0x7ff78c113db0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7ff77800eb40 tx=0x7ff77800b040 comp rx=0 tx=0).stop 2024-03-30T04:31:41.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:41.564+0000 7ff78b200700 1 -- 172.21.15.123:0/2251817451 shutdown_connections 2024-03-30T04:31:41.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:41.564+0000 7ff78b200700 1 --2- 172.21.15.123:0/2251817451 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7ff77006c8b0 0x7ff77006ed60 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:41.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:41.564+0000 7ff78b200700 1 --2- 172.21.15.123:0/2251817451 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff78c102c40 0x7ff78c113870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:41.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:41.564+0000 7ff78b200700 1 --2- 172.21.15.123:0/2251817451 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff78c1035e0 0x7ff78c113db0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:41.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:41.564+0000 7ff78b200700 1 -- 172.21.15.123:0/2251817451 >> 172.21.15.123:0/2251817451 conn(0x7ff78c0f8830 msgr2=0x7ff78c0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:31:41.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:41.565+0000 7ff78b200700 1 -- 172.21.15.123:0/2251817451 shutdown_connections 2024-03-30T04:31:41.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:41.565+0000 7ff78b200700 1 -- 172.21.15.123:0/2251817451 wait complete. 2024-03-30T04:31:41.577 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)" 2024-03-30T04:31:41.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:41 smithi135 ceph-mon[44364]: pgmap v88: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:31:41.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:41 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/2251817451' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:41.928 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:41 smithi123 ceph-mon[35245]: pgmap v88: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:31:41.928 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:41 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/2251817451' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:42.340 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -e sha1=470100a4ccbcf7769052bf09a5413881ccf1b97f -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 2'"'"'' 2024-03-30T04:31:42.766 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:31:43.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:43 smithi135 ceph-mon[44364]: pgmap v89: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:31:43.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:43 smithi123 ceph-mon[35245]: pgmap v89: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:31:44.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:44.752+0000 7f1e5bc00700 1 -- 172.21.15.123:0/2465136862 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1e54102e20 msgr2=0x7f1e541031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:31:44.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:44.752+0000 7f1e5bc00700 1 --2- 172.21.15.123:0/2465136862 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1e54102e20 0x7f1e541031f0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f1e48004940 tx=0x7f1e48009e30 comp rx=0 tx=0).stop 2024-03-30T04:31:44.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:44.752+0000 7f1e5bc00700 1 -- 172.21.15.123:0/2465136862 shutdown_connections 2024-03-30T04:31:44.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:44.752+0000 7f1e5bc00700 1 --2- 172.21.15.123:0/2465136862 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1e541037c0 0x7f1e540fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:44.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:44.752+0000 7f1e5bc00700 1 --2- 172.21.15.123:0/2465136862 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1e54102e20 0x7f1e541031f0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:44.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:44.752+0000 7f1e5bc00700 1 -- 172.21.15.123:0/2465136862 >> 172.21.15.123:0/2465136862 conn(0x7f1e540f8970 msgr2=0x7f1e540fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:31:44.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:44.752+0000 7f1e5bc00700 1 -- 172.21.15.123:0/2465136862 shutdown_connections 2024-03-30T04:31:44.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:44.752+0000 7f1e5bc00700 1 -- 172.21.15.123:0/2465136862 wait complete. 2024-03-30T04:31:44.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:44.753+0000 7f1e5bc00700 1 Processor -- start 2024-03-30T04:31:44.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:44.754+0000 7f1e5bc00700 1 -- start start 2024-03-30T04:31:44.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:44.754+0000 7f1e5bc00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1e54102e20 0x7f1e54115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:31:44.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:44.754+0000 7f1e5bc00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1e541037c0 0x7f1e54116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:31:44.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:44.754+0000 7f1e5bc00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e54116880 con 0x7f1e54102e20 2024-03-30T04:31:44.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:44.754+0000 7f1e5bc00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e5410fc50 con 0x7f1e541037c0 2024-03-30T04:31:44.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:44.754+0000 7f1e59200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1e54102e20 0x7f1e54115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:31:44.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:44.754+0000 7f1e59200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1e54102e20 0x7f1e54115cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56394/0 (socket says 172.21.15.123:56394) 2024-03-30T04:31:44.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:44.754+0000 7f1e59200700 1 -- 172.21.15.123:0/714534885 learned_addr learned my addr 172.21.15.123:0/714534885 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:31:44.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:44.755+0000 7f1e53e00700 1 --2- 172.21.15.123:0/714534885 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1e541037c0 0x7f1e54116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:31:44.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:44.755+0000 7f1e59200700 1 -- 172.21.15.123:0/714534885 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1e541037c0 msgr2=0x7f1e54116230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:31:44.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:44.755+0000 7f1e59200700 1 --2- 172.21.15.123:0/714534885 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1e541037c0 0x7f1e54116230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:44.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:44.755+0000 7f1e59200700 1 -- 172.21.15.123:0/714534885 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e48005040 con 0x7f1e54102e20 2024-03-30T04:31:44.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:44.755+0000 7f1e53e00700 1 --2- 172.21.15.123:0/714534885 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1e541037c0 0x7f1e54116230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:31:44.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:44.755+0000 7f1e59200700 1 --2- 172.21.15.123:0/714534885 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1e54102e20 0x7f1e54115cf0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f1e48009360 tx=0x7f1e48009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:31:44.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:44.755+0000 7f1e50c00700 1 -- 172.21.15.123:0/714534885 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e4800e040 con 0x7f1e54102e20 2024-03-30T04:31:44.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:44.755+0000 7f1e5bc00700 1 -- 172.21.15.123:0/714534885 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e5410fed0 con 0x7f1e54102e20 2024-03-30T04:31:44.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:44.755+0000 7f1e50c00700 1 -- 172.21.15.123:0/714534885 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1e48013420 con 0x7f1e54102e20 2024-03-30T04:31:44.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:44.755+0000 7f1e50c00700 1 -- 172.21.15.123:0/714534885 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e4801c5f0 con 0x7f1e54102e20 2024-03-30T04:31:44.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:44.755+0000 7f1e5bc00700 1 -- 172.21.15.123:0/714534885 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e54110490 con 0x7f1e54102e20 2024-03-30T04:31:44.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:44.756+0000 7f1e5bc00700 1 -- 172.21.15.123:0/714534885 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e540fe570 con 0x7f1e54102e20 2024-03-30T04:31:44.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:44.757+0000 7f1e50c00700 1 -- 172.21.15.123:0/714534885 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 32) v1 ==== 90705+0+0 (secure 0 0 0) 0x7f1e48013590 con 0x7f1e54102e20 2024-03-30T04:31:44.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:44.758+0000 7f1e50c00700 1 --2- 172.21.15.123:0/714534885 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f1e3c06c8b0 0x7f1e3c06ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:31:44.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:44.758+0000 7f1e50c00700 1 -- 172.21.15.123:0/714534885 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f1e4808cda0 con 0x7f1e54102e20 2024-03-30T04:31:44.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:44.758+0000 7f1e53e00700 1 --2- 172.21.15.123:0/714534885 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f1e3c06c8b0 0x7f1e3c06ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:31:44.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:44.759+0000 7f1e53e00700 1 --2- 172.21.15.123:0/714534885 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f1e3c06c8b0 0x7f1e3c06ed60 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1e40007fd0 tx=0x7f1e40009040 comp rx=0 tx=0).ready entity=mgr.14696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:31:44.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:44.763+0000 7f1e50c00700 1 -- 172.21.15.123:0/714534885 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f1e4805ada0 con 0x7f1e54102e20 2024-03-30T04:31:45.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:45.049+0000 7f1e5bc00700 1 -- 172.21.15.123:0/714534885 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f1e5405f5f0 con 0x7f1e54102e20 2024-03-30T04:31:45.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:45.049+0000 7f1e50c00700 1 -- 172.21.15.123:0/714534885 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+743 (secure 0 0 0) 0x7f1e4805a930 con 0x7f1e54102e20 2024-03-30T04:31:45.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:45.051+0000 7f1e5bc00700 1 -- 172.21.15.123:0/714534885 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f1e3c06c8b0 msgr2=0x7f1e3c06ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:31:45.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:45.051+0000 7f1e5bc00700 1 --2- 172.21.15.123:0/714534885 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f1e3c06c8b0 0x7f1e3c06ed60 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1e40007fd0 tx=0x7f1e40009040 comp rx=0 tx=0).stop 2024-03-30T04:31:45.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:45.051+0000 7f1e5bc00700 1 -- 172.21.15.123:0/714534885 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1e54102e20 msgr2=0x7f1e54115cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:31:45.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:45.051+0000 7f1e5bc00700 1 --2- 172.21.15.123:0/714534885 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1e54102e20 0x7f1e54115cf0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f1e48009360 tx=0x7f1e48009e30 comp rx=0 tx=0).stop 2024-03-30T04:31:45.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:45.052+0000 7f1e5bc00700 1 -- 172.21.15.123:0/714534885 shutdown_connections 2024-03-30T04:31:45.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:45.052+0000 7f1e5bc00700 1 --2- 172.21.15.123:0/714534885 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1e54102e20 0x7f1e54115cf0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:45.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:45.052+0000 7f1e5bc00700 1 --2- 172.21.15.123:0/714534885 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f1e3c06c8b0 0x7f1e3c06ed60 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:45.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:45.052+0000 7f1e5bc00700 1 --2- 172.21.15.123:0/714534885 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1e541037c0 0x7f1e54116230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:45.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:45.052+0000 7f1e5bc00700 1 -- 172.21.15.123:0/714534885 >> 172.21.15.123:0/714534885 conn(0x7f1e540f8970 msgr2=0x7f1e540f99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:31:45.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:45.052+0000 7f1e5bc00700 1 -- 172.21.15.123:0/714534885 shutdown_connections 2024-03-30T04:31:45.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:45.052+0000 7f1e5bc00700 1 -- 172.21.15.123:0/714534885 wait complete. 2024-03-30T04:31:45.064 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:31:45.693 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:45 smithi123 ceph-mon[35245]: pgmap v90: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:31:45.693 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:45 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/714534885' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:45.718 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -e sha1=470100a4ccbcf7769052bf09a5413881ccf1b97f -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 2'"'"'' 2024-03-30T04:31:45.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:45 smithi135 ceph-mon[44364]: pgmap v90: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:31:45.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:45 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/714534885' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:31:46.145 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:31:47.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:47 smithi123 ceph-mon[35245]: pgmap v91: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:31:47.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:47 smithi135 ceph-mon[44364]: pgmap v91: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:31:48.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:48.011+0000 7fa84b200700 1 -- 172.21.15.123:0/1868160232 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8441037a0 msgr2=0x7fa8440fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:31:48.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:48.011+0000 7fa84b200700 1 --2- 172.21.15.123:0/1868160232 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8441037a0 0x7fa8440fce10 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fa830004940 tx=0x7fa830009e30 comp rx=0 tx=0).stop 2024-03-30T04:31:48.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:48.011+0000 7fa84b200700 1 -- 172.21.15.123:0/1868160232 shutdown_connections 2024-03-30T04:31:48.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:48.011+0000 7fa84b200700 1 --2- 172.21.15.123:0/1868160232 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8441037a0 0x7fa8440fce10 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:48.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:48.011+0000 7fa84b200700 1 --2- 172.21.15.123:0/1868160232 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa844102e00 0x7fa8441031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:48.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:48.011+0000 7fa84b200700 1 -- 172.21.15.123:0/1868160232 >> 172.21.15.123:0/1868160232 conn(0x7fa8440f89b0 msgr2=0x7fa8440fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:31:48.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:48.012+0000 7fa84b200700 1 -- 172.21.15.123:0/1868160232 shutdown_connections 2024-03-30T04:31:48.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:48.012+0000 7fa84b200700 1 -- 172.21.15.123:0/1868160232 wait complete. 2024-03-30T04:31:48.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:48.013+0000 7fa84b200700 1 Processor -- start 2024-03-30T04:31:48.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:48.013+0000 7fa84b200700 1 -- start start 2024-03-30T04:31:48.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:48.013+0000 7fa84b200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa844102e00 0x7fa844113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:31:48.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:48.013+0000 7fa84b200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa8441037a0 0x7fa844114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:31:48.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:48.013+0000 7fa84b200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa844114690 con 0x7fa844102e00 2024-03-30T04:31:48.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:48.013+0000 7fa84b200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa84410da80 con 0x7fa8441037a0 2024-03-30T04:31:48.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:48.014+0000 7fa843e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa844102e00 0x7fa844113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:31:48.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:48.014+0000 7fa843e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa844102e00 0x7fa844113b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56402/0 (socket says 172.21.15.123:56402) 2024-03-30T04:31:48.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:48.014+0000 7fa843e00700 1 -- 172.21.15.123:0/1456483960 learned_addr learned my addr 172.21.15.123:0/1456483960 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:31:48.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:48.014+0000 7fa843400700 1 --2- 172.21.15.123:0/1456483960 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa8441037a0 0x7fa844114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:31:48.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:48.014+0000 7fa843e00700 1 -- 172.21.15.123:0/1456483960 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa8441037a0 msgr2=0x7fa844114040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:31:48.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:48.014+0000 7fa843e00700 1 --2- 172.21.15.123:0/1456483960 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa8441037a0 0x7fa844114040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:48.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:48.014+0000 7fa843e00700 1 -- 172.21.15.123:0/1456483960 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa830005040 con 0x7fa844102e00 2024-03-30T04:31:48.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:48.014+0000 7fa843400700 1 --2- 172.21.15.123:0/1456483960 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa8441037a0 0x7fa844114040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:31:48.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:48.014+0000 7fa843e00700 1 --2- 172.21.15.123:0/1456483960 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa844102e00 0x7fa844113b00 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fa834002a50 tx=0x7fa8340046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:31:48.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:48.015+0000 7fa83be00700 1 -- 172.21.15.123:0/1456483960 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa834007cc0 con 0x7fa844102e00 2024-03-30T04:31:48.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:48.015+0000 7fa83be00700 1 -- 172.21.15.123:0/1456483960 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa83400f040 con 0x7fa844102e00 2024-03-30T04:31:48.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:48.015+0000 7fa84b200700 1 -- 172.21.15.123:0/1456483960 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa84410dd60 con 0x7fa844102e00 2024-03-30T04:31:48.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:48.015+0000 7fa83be00700 1 -- 172.21.15.123:0/1456483960 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa834009540 con 0x7fa844102e00 2024-03-30T04:31:48.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:48.015+0000 7fa84b200700 1 -- 172.21.15.123:0/1456483960 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa84410e320 con 0x7fa844102e00 2024-03-30T04:31:48.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:48.016+0000 7fa84b200700 1 -- 172.21.15.123:0/1456483960 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8440fe550 con 0x7fa844102e00 2024-03-30T04:31:48.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:48.017+0000 7fa83be00700 1 -- 172.21.15.123:0/1456483960 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 32) v1 ==== 90705+0+0 (secure 0 0 0) 0x7fa8340096a0 con 0x7fa844102e00 2024-03-30T04:31:48.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:48.018+0000 7fa83be00700 1 --2- 172.21.15.123:0/1456483960 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fa82806c8b0 0x7fa82806ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:31:48.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:48.018+0000 7fa83be00700 1 -- 172.21.15.123:0/1456483960 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fa83408b7c0 con 0x7fa844102e00 2024-03-30T04:31:48.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:48.018+0000 7fa843400700 1 --2- 172.21.15.123:0/1456483960 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fa82806c8b0 0x7fa82806ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:31:48.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:48.019+0000 7fa843400700 1 --2- 172.21.15.123:0/1456483960 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fa82806c8b0 0x7fa82806ed60 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa84410ec40 tx=0x7fa830009e30 comp rx=0 tx=0).ready entity=mgr.14696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:31:48.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:48.023+0000 7fa83be00700 1 -- 172.21.15.123:0/1456483960 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fa8340597b0 con 0x7fa844102e00 2024-03-30T04:31:48.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:48.242+0000 7fa84b200700 1 -- 172.21.15.123:0/1456483960 --> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] -- mgr_command(tid 0: {"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f", "target": ["mon-mgr", ""]}) v1 -- 0x7fa844109e40 con 0x7fa82806c8b0 2024-03-30T04:31:49.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:49.812+0000 7fa83be00700 1 -- 172.21.15.123:0/1456483960 <== mgr.14696 v2:172.21.15.123:6800/2125774379 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+5094 (secure 0 0 0) 0x7fa83000b6e0 con 0x7fa82806c8b0 2024-03-30T04:31:49.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:49.815+0000 7fa84b200700 1 -- 172.21.15.123:0/1456483960 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fa82806c8b0 msgr2=0x7fa82806ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:31:49.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:49.815+0000 7fa84b200700 1 --2- 172.21.15.123:0/1456483960 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fa82806c8b0 0x7fa82806ed60 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa84410ec40 tx=0x7fa830009e30 comp rx=0 tx=0).stop 2024-03-30T04:31:49.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:49.815+0000 7fa84b200700 1 -- 172.21.15.123:0/1456483960 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa844102e00 msgr2=0x7fa844113b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:31:49.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:49.815+0000 7fa84b200700 1 --2- 172.21.15.123:0/1456483960 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa844102e00 0x7fa844113b00 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fa834002a50 tx=0x7fa8340046b0 comp rx=0 tx=0).stop 2024-03-30T04:31:49.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:49.815+0000 7fa84b200700 1 -- 172.21.15.123:0/1456483960 shutdown_connections 2024-03-30T04:31:49.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:49.815+0000 7fa84b200700 1 --2- 172.21.15.123:0/1456483960 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa844102e00 0x7fa844113b00 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:49.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:49.815+0000 7fa84b200700 1 --2- 172.21.15.123:0/1456483960 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fa82806c8b0 0x7fa82806ed60 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:49.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:49.815+0000 7fa84b200700 1 --2- 172.21.15.123:0/1456483960 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa8441037a0 0x7fa844114040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:49.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:49.815+0000 7fa84b200700 1 -- 172.21.15.123:0/1456483960 >> 172.21.15.123:0/1456483960 conn(0x7fa8440f89b0 msgr2=0x7fa8440f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:31:49.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:49.815+0000 7fa84b200700 1 -- 172.21.15.123:0/1456483960 shutdown_connections 2024-03-30T04:31:49.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:49.815+0000 7fa84b200700 1 -- 172.21.15.123:0/1456483960 wait complete. 2024-03-30T04:31:49.828 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:31:49.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:49 smithi135 ceph-mon[44364]: pgmap v92: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:31:49.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:49 smithi135 ceph-mon[44364]: from='client.14834 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:31:49.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:49 smithi123 ceph-mon[35245]: pgmap v92: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:31:49.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:49 smithi123 ceph-mon[35245]: from='client.14834 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:31:50.546 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -e sha1=470100a4ccbcf7769052bf09a5413881ccf1b97f -- bash -c 'ceph orch ps' 2024-03-30T04:31:50.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:50 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:31:50.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:50 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:31:50.971 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:31:51.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:51 smithi135 ceph-mon[44364]: pgmap v93: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:31:51.892 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:51 smithi123 ceph-mon[35245]: pgmap v93: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:31:52.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:52.921+0000 7f2c9ec00700 1 -- 172.21.15.123:0/3428665015 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c981037a0 msgr2=0x7f2c980fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:31:52.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:52.921+0000 7f2c9ec00700 1 --2- 172.21.15.123:0/3428665015 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c981037a0 0x7f2c980fce10 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f2c84004940 tx=0x7f2c84009e30 comp rx=0 tx=0).stop 2024-03-30T04:31:52.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:52.921+0000 7f2c9ec00700 1 -- 172.21.15.123:0/3428665015 shutdown_connections 2024-03-30T04:31:52.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:52.921+0000 7f2c9ec00700 1 --2- 172.21.15.123:0/3428665015 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c981037a0 0x7f2c980fce10 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:52.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:52.921+0000 7f2c9ec00700 1 --2- 172.21.15.123:0/3428665015 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2c98102e00 0x7f2c981031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:52.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:52.921+0000 7f2c9ec00700 1 -- 172.21.15.123:0/3428665015 >> 172.21.15.123:0/3428665015 conn(0x7f2c980f89b0 msgr2=0x7f2c980fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:31:52.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:52.921+0000 7f2c9ec00700 1 -- 172.21.15.123:0/3428665015 shutdown_connections 2024-03-30T04:31:52.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:52.921+0000 7f2c9ec00700 1 -- 172.21.15.123:0/3428665015 wait complete. 2024-03-30T04:31:52.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:52.922+0000 7f2c9ec00700 1 Processor -- start 2024-03-30T04:31:52.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:52.922+0000 7f2c9ec00700 1 -- start start 2024-03-30T04:31:52.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:52.922+0000 7f2c9ec00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2c98102e00 0x7f2c98115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:31:52.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:52.923+0000 7f2c9ec00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c981037a0 0x7f2c98116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:31:52.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:52.923+0000 7f2c9ec00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c981168f0 con 0x7f2c981037a0 2024-03-30T04:31:52.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:52.923+0000 7f2c9ec00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c9810fc30 con 0x7f2c98102e00 2024-03-30T04:31:52.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:52.923+0000 7f2c97400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c981037a0 0x7f2c98116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:31:52.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:52.923+0000 7f2c97400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c981037a0 0x7f2c98116210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56418/0 (socket says 172.21.15.123:56418) 2024-03-30T04:31:52.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:52.923+0000 7f2c97400700 1 -- 172.21.15.123:0/2294964618 learned_addr learned my addr 172.21.15.123:0/2294964618 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:31:52.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:52.923+0000 7f2c97400700 1 -- 172.21.15.123:0/2294964618 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2c98102e00 msgr2=0x7f2c98115cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:31:52.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:52.923+0000 7f2c97e00700 1 --2- 172.21.15.123:0/2294964618 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2c98102e00 0x7f2c98115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:31:52.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:52.923+0000 7f2c97400700 1 --2- 172.21.15.123:0/2294964618 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2c98102e00 0x7f2c98115cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:52.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:52.923+0000 7f2c97400700 1 -- 172.21.15.123:0/2294964618 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c84005040 con 0x7f2c981037a0 2024-03-30T04:31:52.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:52.923+0000 7f2c97e00700 1 --2- 172.21.15.123:0/2294964618 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2c98102e00 0x7f2c98115cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:31:52.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:52.923+0000 7f2c97400700 1 --2- 172.21.15.123:0/2294964618 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c981037a0 0x7f2c98116210 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f2c84009390 tx=0x7f2c84009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:31:52.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:52.924+0000 7f2c8fe00700 1 -- 172.21.15.123:0/2294964618 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c8400e040 con 0x7f2c981037a0 2024-03-30T04:31:52.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:52.924+0000 7f2c9ec00700 1 -- 172.21.15.123:0/2294964618 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c9810fe50 con 0x7f2c981037a0 2024-03-30T04:31:52.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:52.924+0000 7f2c8fe00700 1 -- 172.21.15.123:0/2294964618 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2c84013420 con 0x7f2c981037a0 2024-03-30T04:31:52.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:52.924+0000 7f2c8fe00700 1 -- 172.21.15.123:0/2294964618 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c8401c630 con 0x7f2c981037a0 2024-03-30T04:31:52.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:52.924+0000 7f2c9ec00700 1 -- 172.21.15.123:0/2294964618 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c98110410 con 0x7f2c981037a0 2024-03-30T04:31:52.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:52.925+0000 7f2c9ec00700 1 -- 172.21.15.123:0/2294964618 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c980fe550 con 0x7f2c981037a0 2024-03-30T04:31:52.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:52.926+0000 7f2c8fe00700 1 -- 172.21.15.123:0/2294964618 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 32) v1 ==== 90705+0+0 (secure 0 0 0) 0x7f2c840139e0 con 0x7f2c981037a0 2024-03-30T04:31:52.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:52.927+0000 7f2c8fe00700 1 --2- 172.21.15.123:0/2294964618 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f2c7c06c8b0 0x7f2c7c06ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:31:52.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:52.927+0000 7f2c8fe00700 1 -- 172.21.15.123:0/2294964618 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f2c8408cd40 con 0x7f2c981037a0 2024-03-30T04:31:52.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:52.927+0000 7f2c97e00700 1 --2- 172.21.15.123:0/2294964618 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f2c7c06c8b0 0x7f2c7c06ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:31:52.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:52.928+0000 7f2c97e00700 1 --2- 172.21.15.123:0/2294964618 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f2c7c06c8b0 0x7f2c7c06ed60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2c88006fd0 tx=0x7f2c88008040 comp rx=0 tx=0).ready entity=mgr.14696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:31:52.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:52.932+0000 7f2c8fe00700 1 -- 172.21.15.123:0/2294964618 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f2c8405ad30 con 0x7f2c981037a0 2024-03-30T04:31:53.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:53.145+0000 7f2c9ec00700 1 -- 172.21.15.123:0/2294964618 --> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2c9804bed0 con 0x7f2c7c06c8b0 2024-03-30T04:31:53.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:53.155+0000 7f2c8fe00700 1 -- 172.21.15.123:0/2294964618 <== mgr.14696 v2:172.21.15.123:6800/2125774379 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f2c9804bed0 con 0x7f2c7c06c8b0 2024-03-30T04:31:53.154 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:31:53.155 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (62s) 36s ago 13m 14.2M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:31:53.155 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (13m) 36s ago 13m 6651k - 17.2.7-797-g5c024335 f77085f6aab7 bceb8b8c4837 2024-03-30T04:31:53.155 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (12m) 96s ago 12m 6651k - 17.2.7-797-g5c024335 f77085f6aab7 cd0fe67b2d34 2024-03-30T04:31:53.155 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (42s) 36s ago 13m 43.3M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T04:31:53.155 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (7m) 36s ago 7m 109M - 17.2.7-797-g5c024335 f77085f6aab7 2df066fba7a6 2024-03-30T04:31:53.155 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (7m) 36s ago 7m 57.6M - 17.2.7-797-g5c024335 f77085f6aab7 1293b8520db1 2024-03-30T04:31:53.155 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (7m) 96s ago 7m 67.5M - 17.2.7-797-g5c024335 f77085f6aab7 0dc41326818c 2024-03-30T04:31:53.155 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (7m) 96s ago 7m 119M - 17.2.7-797-g5c024335 f77085f6aab7 aef8a0cfc553 2024-03-30T04:31:53.155 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (3m) 36s ago 15m 501M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:31:53.156 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (2m) 96s ago 12m 446M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:31:53.156 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (15m) 36s ago 15m 53.4M 2048M 17.2.7-797-g5c024335 f77085f6aab7 51d18587a7df 2024-03-30T04:31:53.156 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (12m) 96s ago 12m 39.5M 2048M 17.2.7-797-g5c024335 f77085f6aab7 f81052b09dd8 2024-03-30T04:31:53.156 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (107s) 36s ago 13m 7750k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:31:53.156 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (102s) 96s ago 12m 6958k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:31:53.156 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (11m) 36s ago 11m 219M 1618M 17.2.7-797-g5c024335 f77085f6aab7 57a7ba3fd75e 2024-03-30T04:31:53.156 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (11m) 36s ago 10m 234M 1618M 17.2.7-797-g5c024335 f77085f6aab7 e1a6c1818a0c 2024-03-30T04:31:53.156 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (10m) 36s ago 10m 187M 1618M 17.2.7-797-g5c024335 f77085f6aab7 4c6a727567bd 2024-03-30T04:31:53.156 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (9m) 96s ago 9m 256M 2642M 17.2.7-797-g5c024335 f77085f6aab7 6fbdd1d46a9d 2024-03-30T04:31:53.156 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (9m) 96s ago 9m 224M 2642M 17.2.7-797-g5c024335 f77085f6aab7 b74988e947e5 2024-03-30T04:31:53.157 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (8m) 96s ago 8m 218M 2642M 17.2.7-797-g5c024335 f77085f6aab7 1fd3185f5484 2024-03-30T04:31:53.157 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (82s) 36s ago 13m 44.6M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:31:53.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:53.158+0000 7f2c9ec00700 1 -- 172.21.15.123:0/2294964618 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f2c7c06c8b0 msgr2=0x7f2c7c06ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:31:53.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:53.158+0000 7f2c9ec00700 1 --2- 172.21.15.123:0/2294964618 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f2c7c06c8b0 0x7f2c7c06ed60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2c88006fd0 tx=0x7f2c88008040 comp rx=0 tx=0).stop 2024-03-30T04:31:53.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:53.158+0000 7f2c9ec00700 1 -- 172.21.15.123:0/2294964618 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c981037a0 msgr2=0x7f2c98116210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:31:53.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:53.158+0000 7f2c9ec00700 1 --2- 172.21.15.123:0/2294964618 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c981037a0 0x7f2c98116210 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f2c84009390 tx=0x7f2c84009e30 comp rx=0 tx=0).stop 2024-03-30T04:31:53.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:53.158+0000 7f2c9ec00700 1 -- 172.21.15.123:0/2294964618 shutdown_connections 2024-03-30T04:31:53.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:53.158+0000 7f2c9ec00700 1 --2- 172.21.15.123:0/2294964618 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f2c7c06c8b0 0x7f2c7c06ed60 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:53.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:53.158+0000 7f2c9ec00700 1 --2- 172.21.15.123:0/2294964618 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2c98102e00 0x7f2c98115cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:53.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:53.158+0000 7f2c9ec00700 1 --2- 172.21.15.123:0/2294964618 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c981037a0 0x7f2c98116210 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:53.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:53.158+0000 7f2c9ec00700 1 -- 172.21.15.123:0/2294964618 >> 172.21.15.123:0/2294964618 conn(0x7f2c980f89b0 msgr2=0x7f2c980f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:31:53.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:53.158+0000 7f2c9ec00700 1 -- 172.21.15.123:0/2294964618 shutdown_connections 2024-03-30T04:31:53.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:53.158+0000 7f2c9ec00700 1 -- 172.21.15.123:0/2294964618 wait complete. 2024-03-30T04:31:53.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:53 smithi123 ceph-mon[35245]: pgmap v94: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:31:53.867 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-03-30T04:31:53.869 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi123.front.sepia.ceph.com 2024-03-30T04:31:53.869 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -e sha1=470100a4ccbcf7769052bf09a5413881ccf1b97f -- bash -c 'ceph config set mgr mgr/orchestrator/fail_fs true' 2024-03-30T04:31:54.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:53 smithi135 ceph-mon[44364]: pgmap v94: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:31:54.299 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:31:54.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:54 smithi123 ceph-mon[35245]: from='client.14838 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:31:55.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:54 smithi135 ceph-mon[44364]: from='client.14838 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:31:55.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:55 smithi123 ceph-mon[35245]: pgmap v95: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:31:56.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:55 smithi135 ceph-mon[44364]: pgmap v95: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:31:56.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:56.302+0000 7f0acc600700 1 -- 172.21.15.123:0/3991741813 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ac41037a0 msgr2=0x7f0ac40fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:31:56.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:56.302+0000 7f0acc600700 1 --2- 172.21.15.123:0/3991741813 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ac41037a0 0x7f0ac40fce10 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0004940 tx=0x7f0ab0009e30 comp rx=0 tx=0).stop 2024-03-30T04:31:56.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:56.303+0000 7f0acc600700 1 -- 172.21.15.123:0/3991741813 shutdown_connections 2024-03-30T04:31:56.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:56.303+0000 7f0acc600700 1 --2- 172.21.15.123:0/3991741813 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ac41037a0 0x7f0ac40fce10 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:56.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:56.303+0000 7f0acc600700 1 --2- 172.21.15.123:0/3991741813 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0ac4102e00 0x7f0ac41031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:56.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:56.303+0000 7f0acc600700 1 -- 172.21.15.123:0/3991741813 >> 172.21.15.123:0/3991741813 conn(0x7f0ac40f89b0 msgr2=0x7f0ac40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:31:56.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:56.303+0000 7f0acc600700 1 -- 172.21.15.123:0/3991741813 shutdown_connections 2024-03-30T04:31:56.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:56.303+0000 7f0acc600700 1 -- 172.21.15.123:0/3991741813 wait complete. 2024-03-30T04:31:56.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:56.304+0000 7f0acc600700 1 Processor -- start 2024-03-30T04:31:56.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:56.304+0000 7f0acc600700 1 -- start start 2024-03-30T04:31:56.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:56.304+0000 7f0acc600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0ac4102e00 0x7f0ac40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:31:56.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:56.304+0000 7f0acc600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ac41037a0 0x7f0ac4070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:31:56.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:56.304+0000 7f0acc600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac4070d90 con 0x7f0ac41037a0 2024-03-30T04:31:56.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:56.304+0000 7f0acc600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac4070ed0 con 0x7f0ac4102e00 2024-03-30T04:31:56.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:56.304+0000 7f0ac9200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ac41037a0 0x7f0ac4070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:31:56.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:56.304+0000 7f0ac9200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ac41037a0 0x7f0ac4070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46998/0 (socket says 172.21.15.123:46998) 2024-03-30T04:31:56.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:56.304+0000 7f0ac9200700 1 -- 172.21.15.123:0/1377752325 learned_addr learned my addr 172.21.15.123:0/1377752325 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:31:56.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:56.305+0000 7f0ac9200700 1 -- 172.21.15.123:0/1377752325 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0ac4102e00 msgr2=0x7f0ac40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:31:56.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:56.305+0000 7f0ac9200700 1 --2- 172.21.15.123:0/1377752325 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0ac4102e00 0x7f0ac40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:56.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:56.305+0000 7f0ac9200700 1 -- 172.21.15.123:0/1377752325 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab0005040 con 0x7f0ac41037a0 2024-03-30T04:31:56.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:56.305+0000 7f0ac9200700 1 --2- 172.21.15.123:0/1377752325 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ac41037a0 0x7f0ac4070800 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f0ab00093b0 tx=0x7f0ab0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:31:56.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:56.305+0000 7f0ac1600700 1 -- 172.21.15.123:0/1377752325 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ab000e040 con 0x7f0ac41037a0 2024-03-30T04:31:56.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:56.305+0000 7f0ac1600700 1 -- 172.21.15.123:0/1377752325 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0ab0013420 con 0x7f0ac41037a0 2024-03-30T04:31:56.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:56.305+0000 7f0acc600700 1 -- 172.21.15.123:0/1377752325 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ac4071120 con 0x7f0ac41037a0 2024-03-30T04:31:56.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:56.305+0000 7f0ac1600700 1 -- 172.21.15.123:0/1377752325 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ab001c630 con 0x7f0ac41037a0 2024-03-30T04:31:56.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:56.305+0000 7f0acc600700 1 -- 172.21.15.123:0/1377752325 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ac411a230 con 0x7f0ac41037a0 2024-03-30T04:31:56.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:56.306+0000 7f0acc600700 1 -- 172.21.15.123:0/1377752325 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ac40fe550 con 0x7f0ac41037a0 2024-03-30T04:31:56.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:56.308+0000 7f0ac1600700 1 -- 172.21.15.123:0/1377752325 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 32) v1 ==== 90705+0+0 (secure 0 0 0) 0x7f0ab00139e0 con 0x7f0ac41037a0 2024-03-30T04:31:56.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:56.309+0000 7f0ac1600700 1 --2- 172.21.15.123:0/1377752325 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f0aac06c8a0 0x7f0aac06ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:31:56.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:56.309+0000 7f0ac1600700 1 -- 172.21.15.123:0/1377752325 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f0ab008d7b0 con 0x7f0ac41037a0 2024-03-30T04:31:56.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:56.309+0000 7f0ac9c00700 1 --2- 172.21.15.123:0/1377752325 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f0aac06c8a0 0x7f0aac06ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:31:56.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:56.310+0000 7f0ac9c00700 1 --2- 172.21.15.123:0/1377752325 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f0aac06c8a0 0x7f0aac06ed50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0ab8004d00 tx=0x7f0ab8009ce0 comp rx=0 tx=0).ready entity=mgr.14696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:31:56.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:56.312+0000 7f0ac1600700 1 -- 172.21.15.123:0/1377752325 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f0ab0057da0 con 0x7f0ac41037a0 2024-03-30T04:31:56.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:56.515+0000 7f0acc600700 1 -- 172.21.15.123:0/1377752325 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=mgr/orchestrator/fail_fs}] v 0) v1 -- 0x7f0ac405f5f0 con 0x7f0ac41037a0 2024-03-30T04:31:56.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:56.520+0000 7f0ac1600700 1 -- 172.21.15.123:0/1377752325 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/orchestrator/fail_fs}]=0 v47)=0 v47) v1 ==== 125+0+0 (secure 0 0 0) 0x7f0ab005b3c0 con 0x7f0ac41037a0 2024-03-30T04:31:56.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:56.522+0000 7f0acc600700 1 -- 172.21.15.123:0/1377752325 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f0aac06c8a0 msgr2=0x7f0aac06ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:31:56.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:56.522+0000 7f0acc600700 1 --2- 172.21.15.123:0/1377752325 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f0aac06c8a0 0x7f0aac06ed50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0ab8004d00 tx=0x7f0ab8009ce0 comp rx=0 tx=0).stop 2024-03-30T04:31:56.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:56.522+0000 7f0acc600700 1 -- 172.21.15.123:0/1377752325 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ac41037a0 msgr2=0x7f0ac4070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:31:56.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:56.522+0000 7f0acc600700 1 --2- 172.21.15.123:0/1377752325 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ac41037a0 0x7f0ac4070800 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f0ab00093b0 tx=0x7f0ab0009e30 comp rx=0 tx=0).stop 2024-03-30T04:31:56.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:56.523+0000 7f0acc600700 1 -- 172.21.15.123:0/1377752325 shutdown_connections 2024-03-30T04:31:56.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:56.523+0000 7f0acc600700 1 --2- 172.21.15.123:0/1377752325 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f0aac06c8a0 0x7f0aac06ed50 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:56.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:56.523+0000 7f0acc600700 1 --2- 172.21.15.123:0/1377752325 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0ac4102e00 0x7f0ac40702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:56.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:56.523+0000 7f0acc600700 1 --2- 172.21.15.123:0/1377752325 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ac41037a0 0x7f0ac4070800 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:56.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:56.523+0000 7f0acc600700 1 -- 172.21.15.123:0/1377752325 >> 172.21.15.123:0/1377752325 conn(0x7f0ac40f89b0 msgr2=0x7f0ac40f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:31:56.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:56.523+0000 7f0acc600700 1 -- 172.21.15.123:0/1377752325 shutdown_connections 2024-03-30T04:31:56.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:56.523+0000 7f0acc600700 1 -- 172.21.15.123:0/1377752325 wait complete. 2024-03-30T04:31:57.237 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-03-30T04:31:57.238 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi123.front.sepia.ceph.com 2024-03-30T04:31:57.238 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -e sha1=470100a4ccbcf7769052bf09a5413881ccf1b97f -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-03-30T04:31:57.667 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:31:57.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:57 smithi135 ceph-mon[44364]: pgmap v96: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-30T04:31:57.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:57 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/1377752325' entity='client.admin' 2024-03-30T04:31:57.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:57 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:31:57.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:57 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:31:57.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:57 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:31:57.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:57 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:31:57.946 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:57 smithi123 ceph-mon[35245]: pgmap v96: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-30T04:31:57.946 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:57 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/1377752325' entity='client.admin' 2024-03-30T04:31:57.946 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:57 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:31:57.946 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:57 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:31:57.946 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:57 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:31:57.946 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:57 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:31:59.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:59.716+0000 7ff165600700 1 -- 172.21.15.123:0/3395940238 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1601037a0 msgr2=0x7ff1600fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:31:59.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:59.716+0000 7ff165600700 1 --2- 172.21.15.123:0/3395940238 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1601037a0 0x7ff1600fce10 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7ff14c004940 tx=0x7ff14c009e30 comp rx=0 tx=0).stop 2024-03-30T04:31:59.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:59.716+0000 7ff165600700 1 -- 172.21.15.123:0/3395940238 shutdown_connections 2024-03-30T04:31:59.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:59.716+0000 7ff165600700 1 --2- 172.21.15.123:0/3395940238 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1601037a0 0x7ff1600fce10 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:59.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:59.716+0000 7ff165600700 1 --2- 172.21.15.123:0/3395940238 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff160102e00 0x7ff1601031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:59.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:59.716+0000 7ff165600700 1 -- 172.21.15.123:0/3395940238 >> 172.21.15.123:0/3395940238 conn(0x7ff1600f89b0 msgr2=0x7ff1600fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:31:59.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:59.716+0000 7ff165600700 1 -- 172.21.15.123:0/3395940238 shutdown_connections 2024-03-30T04:31:59.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:59.716+0000 7ff165600700 1 -- 172.21.15.123:0/3395940238 wait complete. 2024-03-30T04:31:59.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:59.717+0000 7ff165600700 1 Processor -- start 2024-03-30T04:31:59.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:59.717+0000 7ff165600700 1 -- start start 2024-03-30T04:31:59.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:31:59 smithi123 ceph-mon[35245]: pgmap v97: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-30T04:31:59.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:59.718+0000 7ff165600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff160102e00 0x7ff160115d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:31:59.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:59.718+0000 7ff165600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff1601037a0 0x7ff160116260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:31:59.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:59.718+0000 7ff165600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1601168b0 con 0x7ff160102e00 2024-03-30T04:31:59.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:59.718+0000 7ff165600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff16010fc80 con 0x7ff1601037a0 2024-03-30T04:31:59.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:59.718+0000 7ff15ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff160102e00 0x7ff160115d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:31:59.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:59.718+0000 7ff15ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff160102e00 0x7ff160115d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47014/0 (socket says 172.21.15.123:47014) 2024-03-30T04:31:59.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:59.718+0000 7ff15ea00700 1 -- 172.21.15.123:0/2200259963 learned_addr learned my addr 172.21.15.123:0/2200259963 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:31:59.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:59.718+0000 7ff15e000700 1 --2- 172.21.15.123:0/2200259963 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff1601037a0 0x7ff160116260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:31:59.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:59.718+0000 7ff15ea00700 1 -- 172.21.15.123:0/2200259963 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff1601037a0 msgr2=0x7ff160116260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:31:59.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:59.718+0000 7ff15ea00700 1 --2- 172.21.15.123:0/2200259963 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff1601037a0 0x7ff160116260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:59.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:59.718+0000 7ff15ea00700 1 -- 172.21.15.123:0/2200259963 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff14c005040 con 0x7ff160102e00 2024-03-30T04:31:59.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:59.719+0000 7ff15ea00700 1 --2- 172.21.15.123:0/2200259963 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff160102e00 0x7ff160115d20 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7ff150002a50 tx=0x7ff1500046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:31:59.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:59.719+0000 7ff156a00700 1 -- 172.21.15.123:0/2200259963 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff150007cc0 con 0x7ff160102e00 2024-03-30T04:31:59.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:59.719+0000 7ff165600700 1 -- 172.21.15.123:0/2200259963 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff16010ff60 con 0x7ff160102e00 2024-03-30T04:31:59.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:59.719+0000 7ff156a00700 1 -- 172.21.15.123:0/2200259963 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff15000f040 con 0x7ff160102e00 2024-03-30T04:31:59.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:59.719+0000 7ff156a00700 1 -- 172.21.15.123:0/2200259963 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff15000d9c0 con 0x7ff160102e00 2024-03-30T04:31:59.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:59.719+0000 7ff165600700 1 -- 172.21.15.123:0/2200259963 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff160110520 con 0x7ff160102e00 2024-03-30T04:31:59.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:59.720+0000 7ff165600700 1 -- 172.21.15.123:0/2200259963 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1600fe550 con 0x7ff160102e00 2024-03-30T04:31:59.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:59.721+0000 7ff156a00700 1 -- 172.21.15.123:0/2200259963 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 32) v1 ==== 90705+0+0 (secure 0 0 0) 0x7ff15000db20 con 0x7ff160102e00 2024-03-30T04:31:59.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:59.722+0000 7ff156a00700 1 --2- 172.21.15.123:0/2200259963 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7ff14406c8b0 0x7ff14406ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:31:59.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:59.722+0000 7ff156a00700 1 -- 172.21.15.123:0/2200259963 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7ff15008b860 con 0x7ff160102e00 2024-03-30T04:31:59.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:59.722+0000 7ff15e000700 1 --2- 172.21.15.123:0/2200259963 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7ff14406c8b0 0x7ff14406ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:31:59.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:59.723+0000 7ff15e000700 1 --2- 172.21.15.123:0/2200259963 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7ff14406c8b0 0x7ff14406ed60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff1601115c0 tx=0x7ff14c009e30 comp rx=0 tx=0).ready entity=mgr.14696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:31:59.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:59.727+0000 7ff156a00700 1 -- 172.21.15.123:0/2200259963 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7ff150059730 con 0x7ff160102e00 2024-03-30T04:31:59.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:31:59 smithi135 ceph-mon[44364]: pgmap v97: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-30T04:31:59.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:59.931+0000 7ff165600700 1 -- 172.21.15.123:0/2200259963 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7ff16004bed0 con 0x7ff160102e00 2024-03-30T04:31:59.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:59.931+0000 7ff156a00700 1 -- 172.21.15.123:0/2200259963 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v47)=0 v47) v1 ==== 155+0+0 (secure 0 0 0) 0x7ff1500592c0 con 0x7ff160102e00 2024-03-30T04:31:59.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:59.934+0000 7ff165600700 1 -- 172.21.15.123:0/2200259963 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7ff14406c8b0 msgr2=0x7ff14406ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:31:59.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:59.934+0000 7ff165600700 1 --2- 172.21.15.123:0/2200259963 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7ff14406c8b0 0x7ff14406ed60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff1601115c0 tx=0x7ff14c009e30 comp rx=0 tx=0).stop 2024-03-30T04:31:59.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:59.934+0000 7ff165600700 1 -- 172.21.15.123:0/2200259963 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff160102e00 msgr2=0x7ff160115d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:31:59.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:59.934+0000 7ff165600700 1 --2- 172.21.15.123:0/2200259963 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff160102e00 0x7ff160115d20 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7ff150002a50 tx=0x7ff1500046b0 comp rx=0 tx=0).stop 2024-03-30T04:31:59.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:59.934+0000 7ff165600700 1 -- 172.21.15.123:0/2200259963 shutdown_connections 2024-03-30T04:31:59.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:59.934+0000 7ff165600700 1 --2- 172.21.15.123:0/2200259963 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff160102e00 0x7ff160115d20 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:59.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:59.934+0000 7ff165600700 1 --2- 172.21.15.123:0/2200259963 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7ff14406c8b0 0x7ff14406ed60 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:59.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:59.934+0000 7ff165600700 1 --2- 172.21.15.123:0/2200259963 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff1601037a0 0x7ff160116260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:31:59.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:59.934+0000 7ff165600700 1 -- 172.21.15.123:0/2200259963 >> 172.21.15.123:0/2200259963 conn(0x7ff1600f89b0 msgr2=0x7ff160070710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:31:59.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:59.934+0000 7ff165600700 1 -- 172.21.15.123:0/2200259963 shutdown_connections 2024-03-30T04:31:59.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:31:59.934+0000 7ff165600700 1 -- 172.21.15.123:0/2200259963 wait complete. 2024-03-30T04:32:00.681 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -e sha1=470100a4ccbcf7769052bf09a5413881ccf1b97f -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-03-30T04:32:01.110 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:32:01.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:01 smithi123 ceph-mon[35245]: pgmap v98: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-30T04:32:01.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:01 smithi135 ceph-mon[44364]: pgmap v98: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-30T04:32:03.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:03.038+0000 7fd823600700 1 -- 172.21.15.123:0/1974241167 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd81c102e20 msgr2=0x7fd81c1031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:03.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:03.038+0000 7fd823600700 1 --2- 172.21.15.123:0/1974241167 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd81c102e20 0x7fd81c1031f0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fd810004940 tx=0x7fd810009e30 comp rx=0 tx=0).stop 2024-03-30T04:32:03.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:03.038+0000 7fd823600700 1 -- 172.21.15.123:0/1974241167 shutdown_connections 2024-03-30T04:32:03.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:03.038+0000 7fd823600700 1 --2- 172.21.15.123:0/1974241167 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd81c1037c0 0x7fd81c0fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:03.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:03.038+0000 7fd823600700 1 --2- 172.21.15.123:0/1974241167 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd81c102e20 0x7fd81c1031f0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:03.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:03.038+0000 7fd823600700 1 -- 172.21.15.123:0/1974241167 >> 172.21.15.123:0/1974241167 conn(0x7fd81c0f89d0 msgr2=0x7fd81c0fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:32:03.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:03.039+0000 7fd823600700 1 -- 172.21.15.123:0/1974241167 shutdown_connections 2024-03-30T04:32:03.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:03.039+0000 7fd823600700 1 -- 172.21.15.123:0/1974241167 wait complete. 2024-03-30T04:32:03.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:03.039+0000 7fd823600700 1 Processor -- start 2024-03-30T04:32:03.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:03.040+0000 7fd823600700 1 -- start start 2024-03-30T04:32:03.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:03.040+0000 7fd823600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd81c102e20 0x7fd81c115d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:32:03.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:03.040+0000 7fd823600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd81c1037c0 0x7fd81c116250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:32:03.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:03.040+0000 7fd823600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd81c1168a0 con 0x7fd81c1037c0 2024-03-30T04:32:03.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:03.040+0000 7fd823600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd81c10fc70 con 0x7fd81c102e20 2024-03-30T04:32:03.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:03.040+0000 7fd81be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd81c1037c0 0x7fd81c116250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:32:03.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:03.040+0000 7fd81be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd81c1037c0 0x7fd81c116250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47036/0 (socket says 172.21.15.123:47036) 2024-03-30T04:32:03.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:03.040+0000 7fd81be00700 1 -- 172.21.15.123:0/3324522588 learned_addr learned my addr 172.21.15.123:0/3324522588 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:32:03.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:03.041+0000 7fd820c00700 1 --2- 172.21.15.123:0/3324522588 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd81c102e20 0x7fd81c115d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:32:03.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:03.041+0000 7fd81be00700 1 -- 172.21.15.123:0/3324522588 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd81c102e20 msgr2=0x7fd81c115d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:03.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:03.041+0000 7fd81be00700 1 --2- 172.21.15.123:0/3324522588 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd81c102e20 0x7fd81c115d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:03.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:03.041+0000 7fd81be00700 1 -- 172.21.15.123:0/3324522588 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd810005040 con 0x7fd81c1037c0 2024-03-30T04:32:03.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:03.041+0000 7fd81be00700 1 --2- 172.21.15.123:0/3324522588 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd81c1037c0 0x7fd81c116250 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fd80800c890 tx=0x7fd8080044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:32:03.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:03.041+0000 7fd818c00700 1 -- 172.21.15.123:0/3324522588 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd80800a7d0 con 0x7fd81c1037c0 2024-03-30T04:32:03.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:03.041+0000 7fd823600700 1 -- 172.21.15.123:0/3324522588 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd81c10ff50 con 0x7fd81c1037c0 2024-03-30T04:32:03.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:03.041+0000 7fd818c00700 1 -- 172.21.15.123:0/3324522588 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd80800ade0 con 0x7fd81c1037c0 2024-03-30T04:32:03.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:03.041+0000 7fd818c00700 1 -- 172.21.15.123:0/3324522588 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd8080113a0 con 0x7fd81c1037c0 2024-03-30T04:32:03.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:03.041+0000 7fd823600700 1 -- 172.21.15.123:0/3324522588 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd81c110510 con 0x7fd81c1037c0 2024-03-30T04:32:03.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:03.043+0000 7fd823600700 1 -- 172.21.15.123:0/3324522588 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd81c0fe570 con 0x7fd81c1037c0 2024-03-30T04:32:03.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:03.044+0000 7fd818c00700 1 -- 172.21.15.123:0/3324522588 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 32) v1 ==== 90705+0+0 (secure 0 0 0) 0x7fd808012400 con 0x7fd81c1037c0 2024-03-30T04:32:03.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:03.045+0000 7fd818c00700 1 --2- 172.21.15.123:0/3324522588 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fd80406c8b0 0x7fd80406ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:32:03.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:03.045+0000 7fd818c00700 1 -- 172.21.15.123:0/3324522588 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fd80808b6e0 con 0x7fd81c1037c0 2024-03-30T04:32:03.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:03.045+0000 7fd820c00700 1 --2- 172.21.15.123:0/3324522588 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fd80406c8b0 0x7fd80406ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:32:03.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:03.046+0000 7fd820c00700 1 --2- 172.21.15.123:0/3324522588 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fd80406c8b0 0x7fd80406ed60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd81000b3b0 tx=0x7fd810009e30 comp rx=0 tx=0).ready entity=mgr.14696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:32:03.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:03.049+0000 7fd818c00700 1 -- 172.21.15.123:0/3324522588 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fd808059770 con 0x7fd81c1037c0 2024-03-30T04:32:03.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:03.256+0000 7fd823600700 1 -- 172.21.15.123:0/3324522588 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7fd81c05f5f0 con 0x7fd81c1037c0 2024-03-30T04:32:03.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:03.256+0000 7fd818c00700 1 -- 172.21.15.123:0/3324522588 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v47)=0 v47) v1 ==== 163+0+0 (secure 0 0 0) 0x7fd808059300 con 0x7fd81c1037c0 2024-03-30T04:32:03.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:03.259+0000 7fd823600700 1 -- 172.21.15.123:0/3324522588 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fd80406c8b0 msgr2=0x7fd80406ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:03.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:03.259+0000 7fd823600700 1 --2- 172.21.15.123:0/3324522588 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fd80406c8b0 0x7fd80406ed60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd81000b3b0 tx=0x7fd810009e30 comp rx=0 tx=0).stop 2024-03-30T04:32:03.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:03.259+0000 7fd823600700 1 -- 172.21.15.123:0/3324522588 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd81c1037c0 msgr2=0x7fd81c116250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:03.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:03.259+0000 7fd823600700 1 --2- 172.21.15.123:0/3324522588 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd81c1037c0 0x7fd81c116250 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fd80800c890 tx=0x7fd8080044c0 comp rx=0 tx=0).stop 2024-03-30T04:32:03.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:03.259+0000 7fd823600700 1 -- 172.21.15.123:0/3324522588 shutdown_connections 2024-03-30T04:32:03.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:03.259+0000 7fd823600700 1 --2- 172.21.15.123:0/3324522588 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fd80406c8b0 0x7fd80406ed60 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:03.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:03.259+0000 7fd823600700 1 --2- 172.21.15.123:0/3324522588 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd81c102e20 0x7fd81c115d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:03.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:03.259+0000 7fd823600700 1 --2- 172.21.15.123:0/3324522588 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd81c1037c0 0x7fd81c116250 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:03.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:03.259+0000 7fd823600700 1 -- 172.21.15.123:0/3324522588 >> 172.21.15.123:0/3324522588 conn(0x7fd81c0f89d0 msgr2=0x7fd81c0f9a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:32:03.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:03.259+0000 7fd823600700 1 -- 172.21.15.123:0/3324522588 shutdown_connections 2024-03-30T04:32:03.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:03.259+0000 7fd823600700 1 -- 172.21.15.123:0/3324522588 wait complete. 2024-03-30T04:32:03.629 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:03 smithi123 ceph-mon[35245]: pgmap v99: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-30T04:32:03.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:03 smithi135 ceph-mon[44364]: pgmap v99: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-30T04:32:04.000 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -e sha1=470100a4ccbcf7769052bf09a5413881ccf1b97f -- bash -c 'ceph config set global log_to_journald false --force' 2024-03-30T04:32:04.427 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:32:05.614 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:05 smithi123 ceph-mon[35245]: pgmap v100: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-30T04:32:05.614 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:05 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:32:05.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:05 smithi135 ceph-mon[44364]: pgmap v100: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-30T04:32:05.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:05 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:32:06.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:06.304+0000 7fbc7da00700 1 -- 172.21.15.123:0/4130706809 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc78102e00 msgr2=0x7fbc781031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:06.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:06.304+0000 7fbc7da00700 1 --2- 172.21.15.123:0/4130706809 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc78102e00 0x7fbc781031d0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fbc68004940 tx=0x7fbc68009e30 comp rx=0 tx=0).stop 2024-03-30T04:32:06.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:06.304+0000 7fbc7da00700 1 -- 172.21.15.123:0/4130706809 shutdown_connections 2024-03-30T04:32:06.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:06.304+0000 7fbc7da00700 1 --2- 172.21.15.123:0/4130706809 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbc781037a0 0x7fbc780fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:06.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:06.304+0000 7fbc7da00700 1 --2- 172.21.15.123:0/4130706809 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc78102e00 0x7fbc781031d0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:06.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:06.304+0000 7fbc7da00700 1 -- 172.21.15.123:0/4130706809 >> 172.21.15.123:0/4130706809 conn(0x7fbc780f89b0 msgr2=0x7fbc780fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:32:06.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:06.304+0000 7fbc7da00700 1 -- 172.21.15.123:0/4130706809 shutdown_connections 2024-03-30T04:32:06.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:06.304+0000 7fbc7da00700 1 -- 172.21.15.123:0/4130706809 wait complete. 2024-03-30T04:32:06.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:06.305+0000 7fbc7da00700 1 Processor -- start 2024-03-30T04:32:06.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:06.305+0000 7fbc7da00700 1 -- start start 2024-03-30T04:32:06.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:06.305+0000 7fbc7da00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbc78102e00 0x7fbc78113a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:32:06.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:06.305+0000 7fbc7da00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc781037a0 0x7fbc78113f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:32:06.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:06.305+0000 7fbc7da00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc781145e0 con 0x7fbc781037a0 2024-03-30T04:32:06.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:06.305+0000 7fbc7da00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc7810d9d0 con 0x7fbc78102e00 2024-03-30T04:32:06.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:06.305+0000 7fbc76000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc781037a0 0x7fbc78113f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:32:06.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:06.306+0000 7fbc76000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc781037a0 0x7fbc78113f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38660/0 (socket says 172.21.15.123:38660) 2024-03-30T04:32:06.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:06.306+0000 7fbc76000700 1 -- 172.21.15.123:0/4286386033 learned_addr learned my addr 172.21.15.123:0/4286386033 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:32:06.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:06.306+0000 7fbc76a00700 1 --2- 172.21.15.123:0/4286386033 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbc78102e00 0x7fbc78113a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:32:06.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:06.306+0000 7fbc76000700 1 -- 172.21.15.123:0/4286386033 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbc78102e00 msgr2=0x7fbc78113a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:06.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:06.306+0000 7fbc76000700 1 --2- 172.21.15.123:0/4286386033 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbc78102e00 0x7fbc78113a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:06.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:06.306+0000 7fbc76000700 1 -- 172.21.15.123:0/4286386033 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc68005040 con 0x7fbc781037a0 2024-03-30T04:32:06.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:06.306+0000 7fbc76a00700 1 --2- 172.21.15.123:0/4286386033 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbc78102e00 0x7fbc78113a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:32:06.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:06.306+0000 7fbc76000700 1 --2- 172.21.15.123:0/4286386033 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc781037a0 0x7fbc78113f90 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fbc6400eb40 tx=0x7fbc6400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:32:06.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:06.306+0000 7fbc6ea00700 1 -- 172.21.15.123:0/4286386033 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc64007cc0 con 0x7fbc781037a0 2024-03-30T04:32:06.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:06.306+0000 7fbc6ea00700 1 -- 172.21.15.123:0/4286386033 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbc6400f040 con 0x7fbc781037a0 2024-03-30T04:32:06.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:06.306+0000 7fbc7da00700 1 -- 172.21.15.123:0/4286386033 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc7810dc50 con 0x7fbc781037a0 2024-03-30T04:32:06.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:06.306+0000 7fbc6ea00700 1 -- 172.21.15.123:0/4286386033 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc64009570 con 0x7fbc781037a0 2024-03-30T04:32:06.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:06.307+0000 7fbc7da00700 1 -- 172.21.15.123:0/4286386033 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc7810e210 con 0x7fbc781037a0 2024-03-30T04:32:06.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:06.308+0000 7fbc7da00700 1 -- 172.21.15.123:0/4286386033 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc780fe550 con 0x7fbc781037a0 2024-03-30T04:32:06.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:06.309+0000 7fbc6ea00700 1 -- 172.21.15.123:0/4286386033 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 32) v1 ==== 90705+0+0 (secure 0 0 0) 0x7fbc640096d0 con 0x7fbc781037a0 2024-03-30T04:32:06.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:06.309+0000 7fbc6ea00700 1 --2- 172.21.15.123:0/4286386033 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fbc5c06c8b0 0x7fbc5c06ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:32:06.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:06.310+0000 7fbc6ea00700 1 -- 172.21.15.123:0/4286386033 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fbc6408b690 con 0x7fbc781037a0 2024-03-30T04:32:06.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:06.310+0000 7fbc76a00700 1 --2- 172.21.15.123:0/4286386033 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fbc5c06c8b0 0x7fbc5c06ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:32:06.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:06.310+0000 7fbc76a00700 1 --2- 172.21.15.123:0/4286386033 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fbc5c06c8b0 0x7fbc5c06ed60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbc680136f0 tx=0x7fbc68009e30 comp rx=0 tx=0).ready entity=mgr.14696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:32:06.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:06.314+0000 7fbc6ea00700 1 -- 172.21.15.123:0/4286386033 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fbc64059610 con 0x7fbc781037a0 2024-03-30T04:32:06.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:06.518+0000 7fbc7da00700 1 -- 172.21.15.123:0/4286386033 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7fbc78109dc0 con 0x7fbc781037a0 2024-03-30T04:32:06.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:06.519+0000 7fbc6ea00700 1 -- 172.21.15.123:0/4286386033 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v47)=0 v47) v1 ==== 135+0+0 (secure 0 0 0) 0x7fbc640591a0 con 0x7fbc781037a0 2024-03-30T04:32:06.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:06.521+0000 7fbc7da00700 1 -- 172.21.15.123:0/4286386033 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fbc5c06c8b0 msgr2=0x7fbc5c06ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:06.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:06.521+0000 7fbc7da00700 1 --2- 172.21.15.123:0/4286386033 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fbc5c06c8b0 0x7fbc5c06ed60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbc680136f0 tx=0x7fbc68009e30 comp rx=0 tx=0).stop 2024-03-30T04:32:06.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:06.521+0000 7fbc7da00700 1 -- 172.21.15.123:0/4286386033 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc781037a0 msgr2=0x7fbc78113f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:06.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:06.521+0000 7fbc7da00700 1 --2- 172.21.15.123:0/4286386033 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc781037a0 0x7fbc78113f90 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fbc6400eb40 tx=0x7fbc6400b040 comp rx=0 tx=0).stop 2024-03-30T04:32:06.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:06.521+0000 7fbc7da00700 1 -- 172.21.15.123:0/4286386033 shutdown_connections 2024-03-30T04:32:06.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:06.521+0000 7fbc7da00700 1 --2- 172.21.15.123:0/4286386033 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fbc5c06c8b0 0x7fbc5c06ed60 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:06.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:06.521+0000 7fbc7da00700 1 --2- 172.21.15.123:0/4286386033 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbc78102e00 0x7fbc78113a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:06.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:06.521+0000 7fbc7da00700 1 --2- 172.21.15.123:0/4286386033 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc781037a0 0x7fbc78113f90 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:06.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:06.521+0000 7fbc7da00700 1 -- 172.21.15.123:0/4286386033 >> 172.21.15.123:0/4286386033 conn(0x7fbc780f89b0 msgr2=0x7fbc780f99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:32:06.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:06.521+0000 7fbc7da00700 1 -- 172.21.15.123:0/4286386033 shutdown_connections 2024-03-30T04:32:06.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:06.521+0000 7fbc7da00700 1 -- 172.21.15.123:0/4286386033 wait complete. 2024-03-30T04:32:07.195 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -e sha1=470100a4ccbcf7769052bf09a5413881ccf1b97f -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-03-30T04:32:07.620 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:32:07.854 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:07 smithi135 ceph-mon[44364]: pgmap v101: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-30T04:32:07.878 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:07 smithi123 ceph-mon[35245]: pgmap v101: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-30T04:32:09.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:09.589+0000 7f7023800700 1 -- 172.21.15.123:0/3156730159 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f701c070300 msgr2=0x7f701c0706d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:09.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:09.589+0000 7f7023800700 1 --2- 172.21.15.123:0/3156730159 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f701c070300 0x7f701c0706d0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f7010004970 tx=0x7f7010009e30 comp rx=0 tx=0).stop 2024-03-30T04:32:09.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:09.590+0000 7f7023800700 1 -- 172.21.15.123:0/3156730159 shutdown_connections 2024-03-30T04:32:09.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:09.590+0000 7f7023800700 1 --2- 172.21.15.123:0/3156730159 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f701c070c10 0x7f701c109180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:09.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:09.590+0000 7f7023800700 1 --2- 172.21.15.123:0/3156730159 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f701c070300 0x7f701c0706d0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:09.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:09.590+0000 7f7023800700 1 -- 172.21.15.123:0/3156730159 >> 172.21.15.123:0/3156730159 conn(0x7f701c0f89a0 msgr2=0x7f701c0fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:32:09.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:09.590+0000 7f7023800700 1 -- 172.21.15.123:0/3156730159 shutdown_connections 2024-03-30T04:32:09.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:09.590+0000 7f7023800700 1 -- 172.21.15.123:0/3156730159 wait complete. 2024-03-30T04:32:09.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:09.590+0000 7f7023800700 1 Processor -- start 2024-03-30T04:32:09.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:09.591+0000 7f7023800700 1 -- start start 2024-03-30T04:32:09.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:09.591+0000 7f7023800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f701c070300 0x7f701c115d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:32:09.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:09.591+0000 7f7023800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f701c070c10 0x7f701c116260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:32:09.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:09.591+0000 7f7023800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f701c116940 con 0x7f701c070300 2024-03-30T04:32:09.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:09.591+0000 7f7023800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f701c10fc80 con 0x7f701c070c10 2024-03-30T04:32:09.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:09.591+0000 7f7020e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f701c070300 0x7f701c115d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:32:09.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:09.591+0000 7f7020e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f701c070300 0x7f701c115d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38676/0 (socket says 172.21.15.123:38676) 2024-03-30T04:32:09.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:09.591+0000 7f7020e00700 1 -- 172.21.15.123:0/3712482836 learned_addr learned my addr 172.21.15.123:0/3712482836 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:32:09.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:09.592+0000 7f701be00700 1 --2- 172.21.15.123:0/3712482836 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f701c070c10 0x7f701c116260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:32:09.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:09.592+0000 7f7020e00700 1 -- 172.21.15.123:0/3712482836 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f701c070c10 msgr2=0x7f701c116260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:09.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:09.592+0000 7f7020e00700 1 --2- 172.21.15.123:0/3712482836 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f701c070c10 0x7f701c116260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:09.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:09.592+0000 7f7020e00700 1 -- 172.21.15.123:0/3712482836 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7010005040 con 0x7f701c070300 2024-03-30T04:32:09.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:09.592+0000 7f7020e00700 1 --2- 172.21.15.123:0/3712482836 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f701c070300 0x7f701c115d20 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f7010004f50 tx=0x7f7010009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:32:09.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:09.592+0000 7f7018c00700 1 -- 172.21.15.123:0/3712482836 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f701000e040 con 0x7f701c070300 2024-03-30T04:32:09.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:09.592+0000 7f7023800700 1 -- 172.21.15.123:0/3712482836 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f701c10fea0 con 0x7f701c070300 2024-03-30T04:32:09.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:09.592+0000 7f7018c00700 1 -- 172.21.15.123:0/3712482836 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7010013420 con 0x7f701c070300 2024-03-30T04:32:09.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:09.592+0000 7f7018c00700 1 -- 172.21.15.123:0/3712482836 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f701001c610 con 0x7f701c070300 2024-03-30T04:32:09.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:09.592+0000 7f7023800700 1 -- 172.21.15.123:0/3712482836 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f701c1103e0 con 0x7f701c070300 2024-03-30T04:32:09.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:09.593+0000 7f7023800700 1 -- 172.21.15.123:0/3712482836 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f701c071a40 con 0x7f701c070300 2024-03-30T04:32:09.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:09.594+0000 7f7018c00700 1 -- 172.21.15.123:0/3712482836 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 32) v1 ==== 90705+0+0 (secure 0 0 0) 0x7f7010013590 con 0x7f701c070300 2024-03-30T04:32:09.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:09.595+0000 7f7018c00700 1 --2- 172.21.15.123:0/3712482836 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f700406c8b0 0x7f700406ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:32:09.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:09.595+0000 7f7018c00700 1 -- 172.21.15.123:0/3712482836 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f701008cd30 con 0x7f701c070300 2024-03-30T04:32:09.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:09.595+0000 7f701be00700 1 --2- 172.21.15.123:0/3712482836 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f700406c8b0 0x7f700406ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:32:09.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:09.596+0000 7f701be00700 1 --2- 172.21.15.123:0/3712482836 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f700406c8b0 0x7f700406ed60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7008006fd0 tx=0x7f7008008040 comp rx=0 tx=0).ready entity=mgr.14696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:32:09.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:09.600+0000 7f7018c00700 1 -- 172.21.15.123:0/3712482836 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f701005ac00 con 0x7f701c070300 2024-03-30T04:32:09.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:09.820+0000 7f7023800700 1 -- 172.21.15.123:0/3712482836 --> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f", "target": ["mon-mgr", ""]}) v1 -- 0x7f701c111370 con 0x7f700406c8b0 2024-03-30T04:32:09.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:09.827+0000 7f7018c00700 1 -- 172.21.15.123:0/3712482836 <== mgr.14696 v2:172.21.15.123:6800/2125774379 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7f701c111370 con 0x7f700406c8b0 2024-03-30T04:32:09.826 INFO:teuthology.orchestra.run.smithi123.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f 2024-03-30T04:32:09.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:09.829+0000 7f7023800700 1 -- 172.21.15.123:0/3712482836 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f700406c8b0 msgr2=0x7f700406ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:09.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:09.829+0000 7f7023800700 1 --2- 172.21.15.123:0/3712482836 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f700406c8b0 0x7f700406ed60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7008006fd0 tx=0x7f7008008040 comp rx=0 tx=0).stop 2024-03-30T04:32:09.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:09.829+0000 7f7023800700 1 -- 172.21.15.123:0/3712482836 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f701c070300 msgr2=0x7f701c115d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:09.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:09.829+0000 7f7023800700 1 --2- 172.21.15.123:0/3712482836 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f701c070300 0x7f701c115d20 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f7010004f50 tx=0x7f7010009e30 comp rx=0 tx=0).stop 2024-03-30T04:32:09.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:09.830+0000 7f7023800700 1 -- 172.21.15.123:0/3712482836 shutdown_connections 2024-03-30T04:32:09.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:09.830+0000 7f7023800700 1 --2- 172.21.15.123:0/3712482836 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f701c070300 0x7f701c115d20 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:09.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:09.830+0000 7f7023800700 1 --2- 172.21.15.123:0/3712482836 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f700406c8b0 0x7f700406ed60 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:09.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:09.830+0000 7f7023800700 1 --2- 172.21.15.123:0/3712482836 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f701c070c10 0x7f701c116260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:09.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:09.830+0000 7f7023800700 1 -- 172.21.15.123:0/3712482836 >> 172.21.15.123:0/3712482836 conn(0x7f701c0f89a0 msgr2=0x7f701c0f9f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:32:09.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:09.830+0000 7f7023800700 1 -- 172.21.15.123:0/3712482836 shutdown_connections 2024-03-30T04:32:09.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:09.830+0000 7f7023800700 1 -- 172.21.15.123:0/3712482836 wait complete. 2024-03-30T04:32:09.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:09 smithi135 ceph-mon[44364]: pgmap v102: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:32:09.851 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:09 smithi123 ceph-mon[35245]: pgmap v102: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:32:10.555 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-03-30T04:32:10.555 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi123.front.sepia.ceph.com 2024-03-30T04:32:10.556 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -e sha1=470100a4ccbcf7769052bf09a5413881ccf1b97f -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-03-30T04:32:11.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:10 smithi135 ceph-mon[44364]: from='client.14858 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:32:11.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:10 smithi135 ceph-mon[44364]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f 2024-03-30T04:32:11.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:10 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:11.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:10 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:32:11.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:10 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:32:11.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:10 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:32:11.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:10 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:11.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:10 smithi135 ceph-mon[44364]: pgmap v103: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:32:11.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:10 smithi135 ceph-mon[44364]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f 2024-03-30T04:32:11.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:10 smithi123 ceph-mon[35245]: from='client.14858 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:32:11.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:10 smithi123 ceph-mon[35245]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f 2024-03-30T04:32:11.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:10 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:11.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:10 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:32:11.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:10 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:32:11.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:10 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:32:11.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:10 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:11.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:10 smithi123 ceph-mon[35245]: pgmap v103: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:32:11.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:10 smithi123 ceph-mon[35245]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f 2024-03-30T04:32:11.906 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T04:32:13.188 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:12 smithi123 ceph-mon[35245]: pgmap v104: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:32:13.188 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:12 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:13.188 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:12 smithi123 ceph-mon[35245]: Upgrade: Target is version 18.2.2-1018-g470100a4 (reef) 2024-03-30T04:32:13.188 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:12 smithi123 ceph-mon[35245]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35, digests ['quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35'] 2024-03-30T04:32:13.188 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:12 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:32:13.189 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:12 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:13.189 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:12 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:32:13.189 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:12 smithi123 ceph-mon[35245]: Upgrade: Setting container_image for all mgr 2024-03-30T04:32:13.189 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:12 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:13.189 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:12 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-30T04:32:13.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:12 smithi135 ceph-mon[44364]: pgmap v104: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:32:13.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:12 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:13.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:12 smithi135 ceph-mon[44364]: Upgrade: Target is version 18.2.2-1018-g470100a4 (reef) 2024-03-30T04:32:13.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:12 smithi135 ceph-mon[44364]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35, digests ['quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35'] 2024-03-30T04:32:13.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:12 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:32:13.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:12 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:13.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:12 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:32:13.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:12 smithi135 ceph-mon[44364]: Upgrade: Setting container_image for all mgr 2024-03-30T04:32:13.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:12 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:13.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:12 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-30T04:32:14.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.097+0000 7f001c600700 1 -- 172.21.15.123:0/2294880435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0014102e20 msgr2=0x7f00141031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:14.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.097+0000 7f001c600700 1 --2- 172.21.15.123:0/2294880435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0014102e20 0x7f00141031f0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f0008004940 tx=0x7f0008009e30 comp rx=0 tx=0).stop 2024-03-30T04:32:14.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.097+0000 7f001c600700 1 -- 172.21.15.123:0/2294880435 shutdown_connections 2024-03-30T04:32:14.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.097+0000 7f001c600700 1 --2- 172.21.15.123:0/2294880435 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f00141037c0 0x7f00140fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:14.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.097+0000 7f001c600700 1 --2- 172.21.15.123:0/2294880435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0014102e20 0x7f00141031f0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:14.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.097+0000 7f001c600700 1 -- 172.21.15.123:0/2294880435 >> 172.21.15.123:0/2294880435 conn(0x7f00140f89d0 msgr2=0x7f00140fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:32:14.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.097+0000 7f001c600700 1 -- 172.21.15.123:0/2294880435 shutdown_connections 2024-03-30T04:32:14.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.097+0000 7f001c600700 1 -- 172.21.15.123:0/2294880435 wait complete. 2024-03-30T04:32:14.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.098+0000 7f001c600700 1 Processor -- start 2024-03-30T04:32:14.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.098+0000 7f001c600700 1 -- start start 2024-03-30T04:32:14.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.098+0000 7f001c600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0014102e20 0x7f0014113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:32:14.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.098+0000 7f001c600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f00141037c0 0x7f0014114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:32:14.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.098+0000 7f001c600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0014114690 con 0x7f0014102e20 2024-03-30T04:32:14.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.098+0000 7f001c600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f001410da80 con 0x7f00141037c0 2024-03-30T04:32:14.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.099+0000 7f0019c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0014102e20 0x7f0014113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:32:14.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.099+0000 7f0019c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0014102e20 0x7f0014113b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41098/0 (socket says 172.21.15.123:41098) 2024-03-30T04:32:14.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.099+0000 7f0019c00700 1 -- 172.21.15.123:0/2047885676 learned_addr learned my addr 172.21.15.123:0/2047885676 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:32:14.529 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:32:14.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.099+0000 7f0019c00700 1 -- 172.21.15.123:0/2047885676 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f00141037c0 msgr2=0x7f0014114040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:14.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.099+0000 7f0019c00700 1 --2- 172.21.15.123:0/2047885676 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f00141037c0 0x7f0014114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:14.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.099+0000 7f0019c00700 1 -- 172.21.15.123:0/2047885676 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0008005040 con 0x7f0014102e20 2024-03-30T04:32:14.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.099+0000 7f0019c00700 1 --2- 172.21.15.123:0/2047885676 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0014102e20 0x7f0014113b00 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f0008009390 tx=0x7f0008009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:32:14.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.099+0000 7f0011600700 1 -- 172.21.15.123:0/2047885676 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f000800e040 con 0x7f0014102e20 2024-03-30T04:32:14.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.099+0000 7f0011600700 1 -- 172.21.15.123:0/2047885676 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0008013420 con 0x7f0014102e20 2024-03-30T04:32:14.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.099+0000 7f001c600700 1 -- 172.21.15.123:0/2047885676 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f001410dd00 con 0x7f0014102e20 2024-03-30T04:32:14.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.100+0000 7f0011600700 1 -- 172.21.15.123:0/2047885676 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f000801c630 con 0x7f0014102e20 2024-03-30T04:32:14.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.100+0000 7f001c600700 1 -- 172.21.15.123:0/2047885676 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f001410e2c0 con 0x7f0014102e20 2024-03-30T04:32:14.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.101+0000 7f001c600700 1 -- 172.21.15.123:0/2047885676 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00140fe570 con 0x7f0014102e20 2024-03-30T04:32:14.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.102+0000 7f0011600700 1 -- 172.21.15.123:0/2047885676 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 32) v1 ==== 90705+0+0 (secure 0 0 0) 0x7f00080139e0 con 0x7f0014102e20 2024-03-30T04:32:14.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.102+0000 7f0011600700 1 --2- 172.21.15.123:0/2047885676 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7efffc06c8b0 0x7efffc06ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:32:14.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.103+0000 7f0011600700 1 -- 172.21.15.123:0/2047885676 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f000808cd00 con 0x7f0014102e20 2024-03-30T04:32:14.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.103+0000 7f0019200700 1 --2- 172.21.15.123:0/2047885676 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7efffc06c8b0 0x7efffc06ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:32:14.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.103+0000 7f0019200700 1 --2- 172.21.15.123:0/2047885676 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7efffc06c8b0 0x7efffc06ed60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f001410ec40 tx=0x7f0000009ce0 comp rx=0 tx=0).ready entity=mgr.14696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:32:14.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.107+0000 7f0011600700 1 -- 172.21.15.123:0/2047885676 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f0008090030 con 0x7f0014102e20 2024-03-30T04:32:14.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.334+0000 7f001c600700 1 -- 172.21.15.123:0/2047885676 --> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0014109e80 con 0x7efffc06c8b0 2024-03-30T04:32:14.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.335+0000 7f0011600700 1 -- 172.21.15.123:0/2047885676 <== mgr.14696 v2:172.21.15.123:6800/2125774379 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7f000000f040 con 0x7efffc06c8b0 2024-03-30T04:32:14.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.338+0000 7f001c600700 1 -- 172.21.15.123:0/2047885676 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7efffc06c8b0 msgr2=0x7efffc06ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:14.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.338+0000 7f001c600700 1 --2- 172.21.15.123:0/2047885676 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7efffc06c8b0 0x7efffc06ed60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f001410ec40 tx=0x7f0000009ce0 comp rx=0 tx=0).stop 2024-03-30T04:32:14.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.338+0000 7f001c600700 1 -- 172.21.15.123:0/2047885676 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0014102e20 msgr2=0x7f0014113b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:14.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.338+0000 7f001c600700 1 --2- 172.21.15.123:0/2047885676 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0014102e20 0x7f0014113b00 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f0008009390 tx=0x7f0008009e30 comp rx=0 tx=0).stop 2024-03-30T04:32:14.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.338+0000 7f001c600700 1 -- 172.21.15.123:0/2047885676 shutdown_connections 2024-03-30T04:32:14.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.338+0000 7f001c600700 1 --2- 172.21.15.123:0/2047885676 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0014102e20 0x7f0014113b00 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:14.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.338+0000 7f001c600700 1 --2- 172.21.15.123:0/2047885676 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7efffc06c8b0 0x7efffc06ed60 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:14.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.338+0000 7f001c600700 1 --2- 172.21.15.123:0/2047885676 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f00141037c0 0x7f0014114040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:14.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.338+0000 7f001c600700 1 -- 172.21.15.123:0/2047885676 >> 172.21.15.123:0/2047885676 conn(0x7f00140f89d0 msgr2=0x7f00140f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:32:14.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.338+0000 7f001c600700 1 -- 172.21.15.123:0/2047885676 shutdown_connections 2024-03-30T04:32:14.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.338+0000 7f001c600700 1 -- 172.21.15.123:0/2047885676 wait complete. 2024-03-30T04:32:14.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.473+0000 7f3f6d000700 1 -- 172.21.15.123:0/1110881299 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f68102e00 msgr2=0x7f3f681031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:14.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.473+0000 7f3f6d000700 1 --2- 172.21.15.123:0/1110881299 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f68102e00 0x7f3f681031d0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f3f58004940 tx=0x7f3f58009e30 comp rx=0 tx=0).stop 2024-03-30T04:32:14.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.473+0000 7f3f6d000700 1 -- 172.21.15.123:0/1110881299 shutdown_connections 2024-03-30T04:32:14.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.473+0000 7f3f6d000700 1 --2- 172.21.15.123:0/1110881299 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3f681037a0 0x7f3f680fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:14.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.473+0000 7f3f6d000700 1 --2- 172.21.15.123:0/1110881299 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f68102e00 0x7f3f681031d0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:14.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.473+0000 7f3f6d000700 1 -- 172.21.15.123:0/1110881299 >> 172.21.15.123:0/1110881299 conn(0x7f3f680f89b0 msgr2=0x7f3f680fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:32:14.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.474+0000 7f3f6d000700 1 -- 172.21.15.123:0/1110881299 shutdown_connections 2024-03-30T04:32:14.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.474+0000 7f3f6d000700 1 -- 172.21.15.123:0/1110881299 wait complete. 2024-03-30T04:32:14.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.474+0000 7f3f6d000700 1 Processor -- start 2024-03-30T04:32:14.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.474+0000 7f3f6d000700 1 -- start start 2024-03-30T04:32:14.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.475+0000 7f3f6d000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3f68102e00 0x7f3f68113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:32:14.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.475+0000 7f3f6d000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f681037a0 0x7f3f68114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:32:14.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.475+0000 7f3f6d000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f681146b0 con 0x7f3f681037a0 2024-03-30T04:32:14.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.475+0000 7f3f6d000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6810daa0 con 0x7f3f68102e00 2024-03-30T04:32:14.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.475+0000 7f3f65400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f681037a0 0x7f3f68114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:32:14.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.475+0000 7f3f65400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f681037a0 0x7f3f68114060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41112/0 (socket says 172.21.15.123:41112) 2024-03-30T04:32:14.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.475+0000 7f3f65400700 1 -- 172.21.15.123:0/1207089370 learned_addr learned my addr 172.21.15.123:0/1207089370 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:32:14.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.475+0000 7f3f65e00700 1 --2- 172.21.15.123:0/1207089370 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3f68102e00 0x7f3f68113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:32:14.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.475+0000 7f3f65400700 1 -- 172.21.15.123:0/1207089370 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3f68102e00 msgr2=0x7f3f68113b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:14.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.475+0000 7f3f65400700 1 --2- 172.21.15.123:0/1207089370 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3f68102e00 0x7f3f68113b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:14.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.475+0000 7f3f65400700 1 -- 172.21.15.123:0/1207089370 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f58005040 con 0x7f3f681037a0 2024-03-30T04:32:14.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.476+0000 7f3f65400700 1 --2- 172.21.15.123:0/1207089370 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f681037a0 0x7f3f68114060 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f3f5400c890 tx=0x7f3f540044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:32:14.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.476+0000 7f3f5e000700 1 -- 172.21.15.123:0/1207089370 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f5400a700 con 0x7f3f681037a0 2024-03-30T04:32:14.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.476+0000 7f3f5e000700 1 -- 172.21.15.123:0/1207089370 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3f5400ad10 con 0x7f3f681037a0 2024-03-30T04:32:14.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.476+0000 7f3f6d000700 1 -- 172.21.15.123:0/1207089370 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f6810dd80 con 0x7f3f681037a0 2024-03-30T04:32:14.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.476+0000 7f3f5e000700 1 -- 172.21.15.123:0/1207089370 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f54012e70 con 0x7f3f681037a0 2024-03-30T04:32:14.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.476+0000 7f3f6d000700 1 -- 172.21.15.123:0/1207089370 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f6810e340 con 0x7f3f681037a0 2024-03-30T04:32:14.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.477+0000 7f3f6d000700 1 -- 172.21.15.123:0/1207089370 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f680fe550 con 0x7f3f681037a0 2024-03-30T04:32:14.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.478+0000 7f3f5e000700 1 -- 172.21.15.123:0/1207089370 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 32) v1 ==== 90705+0+0 (secure 0 0 0) 0x7f3f5400a860 con 0x7f3f681037a0 2024-03-30T04:32:14.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.479+0000 7f3f5e000700 1 --2- 172.21.15.123:0/1207089370 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f3f4c06c8b0 0x7f3f4c06ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:32:14.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.479+0000 7f3f5e000700 1 -- 172.21.15.123:0/1207089370 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f3f5408bba0 con 0x7f3f681037a0 2024-03-30T04:32:14.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.479+0000 7f3f65e00700 1 --2- 172.21.15.123:0/1207089370 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f3f4c06c8b0 0x7f3f4c06ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:32:14.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.480+0000 7f3f65e00700 1 --2- 172.21.15.123:0/1207089370 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f3f4c06c8b0 0x7f3f4c06ed60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3f58009fd0 tx=0x7f3f58009e30 comp rx=0 tx=0).ready entity=mgr.14696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:32:14.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.483+0000 7f3f5e000700 1 -- 172.21.15.123:0/1207089370 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f3f54059af0 con 0x7f3f681037a0 2024-03-30T04:32:14.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.709+0000 7f3f6d000700 1 -- 172.21.15.123:0/1207089370 --> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3f68109dc0 con 0x7f3f4c06c8b0 2024-03-30T04:32:14.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.711+0000 7f3f5e000700 1 -- 172.21.15.123:0/1207089370 <== mgr.14696 v2:172.21.15.123:6800/2125774379 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7f3f68109dc0 con 0x7f3f4c06c8b0 2024-03-30T04:32:14.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.713+0000 7f3f6d000700 1 -- 172.21.15.123:0/1207089370 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f3f4c06c8b0 msgr2=0x7f3f4c06ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:14.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.713+0000 7f3f6d000700 1 --2- 172.21.15.123:0/1207089370 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f3f4c06c8b0 0x7f3f4c06ed60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3f58009fd0 tx=0x7f3f58009e30 comp rx=0 tx=0).stop 2024-03-30T04:32:14.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.713+0000 7f3f6d000700 1 -- 172.21.15.123:0/1207089370 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f681037a0 msgr2=0x7f3f68114060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:14.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.713+0000 7f3f6d000700 1 --2- 172.21.15.123:0/1207089370 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f681037a0 0x7f3f68114060 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f3f5400c890 tx=0x7f3f540044c0 comp rx=0 tx=0).stop 2024-03-30T04:32:14.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.714+0000 7f3f6d000700 1 -- 172.21.15.123:0/1207089370 shutdown_connections 2024-03-30T04:32:14.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.714+0000 7f3f6d000700 1 --2- 172.21.15.123:0/1207089370 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f3f4c06c8b0 0x7f3f4c06ed60 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:14.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.714+0000 7f3f6d000700 1 --2- 172.21.15.123:0/1207089370 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3f68102e00 0x7f3f68113b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:14.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.714+0000 7f3f6d000700 1 --2- 172.21.15.123:0/1207089370 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f681037a0 0x7f3f68114060 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:14.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.714+0000 7f3f6d000700 1 -- 172.21.15.123:0/1207089370 >> 172.21.15.123:0/1207089370 conn(0x7f3f680f89b0 msgr2=0x7f3f680f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:32:14.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.714+0000 7f3f6d000700 1 -- 172.21.15.123:0/1207089370 shutdown_connections 2024-03-30T04:32:14.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.714+0000 7f3f6d000700 1 -- 172.21.15.123:0/1207089370 wait complete. 2024-03-30T04:32:14.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.846+0000 7faae1400700 1 -- 172.21.15.123:0/3066591160 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faadc102e00 msgr2=0x7faadc1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:14.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.846+0000 7faae1400700 1 --2- 172.21.15.123:0/3066591160 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faadc102e00 0x7faadc1031d0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7faacc004940 tx=0x7faacc009e30 comp rx=0 tx=0).stop 2024-03-30T04:32:14.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.846+0000 7faae1400700 1 -- 172.21.15.123:0/3066591160 shutdown_connections 2024-03-30T04:32:14.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.846+0000 7faae1400700 1 --2- 172.21.15.123:0/3066591160 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faadc1037a0 0x7faadc0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:14.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.846+0000 7faae1400700 1 --2- 172.21.15.123:0/3066591160 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faadc102e00 0x7faadc1031d0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:14.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.846+0000 7faae1400700 1 -- 172.21.15.123:0/3066591160 >> 172.21.15.123:0/3066591160 conn(0x7faadc0f89b0 msgr2=0x7faadc0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:32:14.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.847+0000 7faae1400700 1 -- 172.21.15.123:0/3066591160 shutdown_connections 2024-03-30T04:32:14.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.847+0000 7faae1400700 1 -- 172.21.15.123:0/3066591160 wait complete. 2024-03-30T04:32:14.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.847+0000 7faae1400700 1 Processor -- start 2024-03-30T04:32:14.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.847+0000 7faae1400700 1 -- start start 2024-03-30T04:32:14.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.848+0000 7faae1400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faadc102e00 0x7faadc115c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:32:14.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.848+0000 7faae1400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faadc1037a0 0x7faadc116190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:32:14.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.848+0000 7faae1400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faadc1167e0 con 0x7faadc102e00 2024-03-30T04:32:14.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.848+0000 7faae1400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faadc10fbb0 con 0x7faadc1037a0 2024-03-30T04:32:14.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.848+0000 7faada800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faadc102e00 0x7faadc115c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:32:14.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.848+0000 7faada800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faadc102e00 0x7faadc115c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41124/0 (socket says 172.21.15.123:41124) 2024-03-30T04:32:14.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.848+0000 7faada800700 1 -- 172.21.15.123:0/2010447280 learned_addr learned my addr 172.21.15.123:0/2010447280 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:32:14.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.848+0000 7faad9e00700 1 --2- 172.21.15.123:0/2010447280 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faadc1037a0 0x7faadc116190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:32:14.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.849+0000 7faada800700 1 -- 172.21.15.123:0/2010447280 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faadc1037a0 msgr2=0x7faadc116190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:14.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.849+0000 7faada800700 1 --2- 172.21.15.123:0/2010447280 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faadc1037a0 0x7faadc116190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:14.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.849+0000 7faada800700 1 -- 172.21.15.123:0/2010447280 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faacc005040 con 0x7faadc102e00 2024-03-30T04:32:14.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.849+0000 7faad9e00700 1 --2- 172.21.15.123:0/2010447280 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faadc1037a0 0x7faadc116190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:32:14.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.849+0000 7faada800700 1 --2- 172.21.15.123:0/2010447280 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faadc102e00 0x7faadc115c50 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7faacc009390 tx=0x7faacc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:32:14.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.849+0000 7faad2a00700 1 -- 172.21.15.123:0/2010447280 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faacc00e040 con 0x7faadc102e00 2024-03-30T04:32:14.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.849+0000 7faad2a00700 1 -- 172.21.15.123:0/2010447280 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faacc013420 con 0x7faadc102e00 2024-03-30T04:32:14.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.849+0000 7faae1400700 1 -- 172.21.15.123:0/2010447280 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faadc10fe30 con 0x7faadc102e00 2024-03-30T04:32:14.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.849+0000 7faad2a00700 1 -- 172.21.15.123:0/2010447280 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faacc01c630 con 0x7faadc102e00 2024-03-30T04:32:14.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.849+0000 7faae1400700 1 -- 172.21.15.123:0/2010447280 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faadc1103f0 con 0x7faadc102e00 2024-03-30T04:32:14.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.850+0000 7faae1400700 1 -- 172.21.15.123:0/2010447280 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faadc0fe550 con 0x7faadc102e00 2024-03-30T04:32:14.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.851+0000 7faad2a00700 1 -- 172.21.15.123:0/2010447280 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 32) v1 ==== 90705+0+0 (secure 0 0 0) 0x7faacc0139e0 con 0x7faadc102e00 2024-03-30T04:32:14.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.852+0000 7faad2a00700 1 --2- 172.21.15.123:0/2010447280 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7faac006c8b0 0x7faac006ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:32:14.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.853+0000 7faad2a00700 1 -- 172.21.15.123:0/2010447280 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7faacc08ccc0 con 0x7faadc102e00 2024-03-30T04:32:14.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.853+0000 7faad9e00700 1 --2- 172.21.15.123:0/2010447280 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7faac006c8b0 0x7faac006ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:32:14.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.853+0000 7faad9e00700 1 --2- 172.21.15.123:0/2010447280 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7faac006c8b0 0x7faac006ed60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7faac8007fd0 tx=0x7faac8009f60 comp rx=0 tx=0).ready entity=mgr.14696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:32:14.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:14.857+0000 7faad2a00700 1 -- 172.21.15.123:0/2010447280 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7faacc092020 con 0x7faadc102e00 2024-03-30T04:32:15.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.080+0000 7faae1400700 1 -- 172.21.15.123:0/2010447280 --> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7faadc04bed0 con 0x7faac006c8b0 2024-03-30T04:32:15.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.091+0000 7faad2a00700 1 -- 172.21.15.123:0/2010447280 <== mgr.14696 v2:172.21.15.123:6800/2125774379 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7faadc04bed0 con 0x7faac006c8b0 2024-03-30T04:32:15.089 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:32:15.090 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (84s) 57s ago 13m 14.2M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:32:15.090 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (14m) 57s ago 14m 6651k - 17.2.7-797-g5c024335 f77085f6aab7 bceb8b8c4837 2024-03-30T04:32:15.090 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (13m) 118s ago 13m 6651k - 17.2.7-797-g5c024335 f77085f6aab7 cd0fe67b2d34 2024-03-30T04:32:15.090 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (64s) 57s ago 13m 43.3M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T04:32:15.090 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (7m) 57s ago 7m 109M - 17.2.7-797-g5c024335 f77085f6aab7 2df066fba7a6 2024-03-30T04:32:15.090 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (7m) 57s ago 7m 57.6M - 17.2.7-797-g5c024335 f77085f6aab7 1293b8520db1 2024-03-30T04:32:15.090 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (7m) 118s ago 7m 67.5M - 17.2.7-797-g5c024335 f77085f6aab7 0dc41326818c 2024-03-30T04:32:15.090 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (7m) 118s ago 7m 119M - 17.2.7-797-g5c024335 f77085f6aab7 aef8a0cfc553 2024-03-30T04:32:15.090 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (3m) 57s ago 15m 501M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:32:15.090 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (2m) 118s ago 12m 446M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:32:15.090 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (15m) 57s ago 15m 53.4M 2048M 17.2.7-797-g5c024335 f77085f6aab7 51d18587a7df 2024-03-30T04:32:15.090 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (12m) 118s ago 12m 39.5M 2048M 17.2.7-797-g5c024335 f77085f6aab7 f81052b09dd8 2024-03-30T04:32:15.090 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (2m) 57s ago 14m 7750k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:32:15.090 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (2m) 118s ago 12m 6958k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:32:15.091 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (11m) 57s ago 11m 219M 1618M 17.2.7-797-g5c024335 f77085f6aab7 57a7ba3fd75e 2024-03-30T04:32:15.091 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (11m) 57s ago 11m 234M 1618M 17.2.7-797-g5c024335 f77085f6aab7 e1a6c1818a0c 2024-03-30T04:32:15.091 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (10m) 57s ago 10m 187M 1618M 17.2.7-797-g5c024335 f77085f6aab7 4c6a727567bd 2024-03-30T04:32:15.091 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (10m) 118s ago 10m 256M 2642M 17.2.7-797-g5c024335 f77085f6aab7 6fbdd1d46a9d 2024-03-30T04:32:15.091 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (9m) 118s ago 9m 224M 2642M 17.2.7-797-g5c024335 f77085f6aab7 b74988e947e5 2024-03-30T04:32:15.091 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (9m) 118s ago 9m 218M 2642M 17.2.7-797-g5c024335 f77085f6aab7 1fd3185f5484 2024-03-30T04:32:15.091 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (104s) 57s ago 13m 44.6M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:32:15.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.093+0000 7faae1400700 1 -- 172.21.15.123:0/2010447280 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7faac006c8b0 msgr2=0x7faac006ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:15.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.093+0000 7faae1400700 1 --2- 172.21.15.123:0/2010447280 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7faac006c8b0 0x7faac006ed60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7faac8007fd0 tx=0x7faac8009f60 comp rx=0 tx=0).stop 2024-03-30T04:32:15.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.093+0000 7faae1400700 1 -- 172.21.15.123:0/2010447280 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faadc102e00 msgr2=0x7faadc115c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:15.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.093+0000 7faae1400700 1 --2- 172.21.15.123:0/2010447280 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faadc102e00 0x7faadc115c50 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7faacc009390 tx=0x7faacc009e30 comp rx=0 tx=0).stop 2024-03-30T04:32:15.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.093+0000 7faae1400700 1 -- 172.21.15.123:0/2010447280 shutdown_connections 2024-03-30T04:32:15.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.093+0000 7faae1400700 1 --2- 172.21.15.123:0/2010447280 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faadc102e00 0x7faadc115c50 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:15.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.093+0000 7faae1400700 1 --2- 172.21.15.123:0/2010447280 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7faac006c8b0 0x7faac006ed60 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:15.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.093+0000 7faae1400700 1 --2- 172.21.15.123:0/2010447280 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faadc1037a0 0x7faadc116190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:15.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.093+0000 7faae1400700 1 -- 172.21.15.123:0/2010447280 >> 172.21.15.123:0/2010447280 conn(0x7faadc0f89b0 msgr2=0x7faadc0f9900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:32:15.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.093+0000 7faae1400700 1 -- 172.21.15.123:0/2010447280 shutdown_connections 2024-03-30T04:32:15.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.093+0000 7faae1400700 1 -- 172.21.15.123:0/2010447280 wait complete. 2024-03-30T04:32:15.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:14 smithi123 ceph-mon[35245]: pgmap v105: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:32:15.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:14 smithi123 ceph-mon[35245]: from='client.14862 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:32:15.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:14 smithi123 ceph-mon[35245]: from='client.14866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:32:15.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.240+0000 7fd87da00700 1 -- 172.21.15.123:0/325188255 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd878070c70 msgr2=0x7fd878109110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:15.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.240+0000 7fd87da00700 1 --2- 172.21.15.123:0/325188255 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd878070c70 0x7fd878109110 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fd864004970 tx=0x7fd864009e30 comp rx=0 tx=0).stop 2024-03-30T04:32:15.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.240+0000 7fd87da00700 1 -- 172.21.15.123:0/325188255 shutdown_connections 2024-03-30T04:32:15.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.240+0000 7fd87da00700 1 --2- 172.21.15.123:0/325188255 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd878070c70 0x7fd878109110 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:15.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.240+0000 7fd87da00700 1 --2- 172.21.15.123:0/325188255 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd878070360 0x7fd878070730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:15.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.240+0000 7fd87da00700 1 -- 172.21.15.123:0/325188255 >> 172.21.15.123:0/325188255 conn(0x7fd8780f89b0 msgr2=0x7fd8780fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:32:15.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.240+0000 7fd87da00700 1 -- 172.21.15.123:0/325188255 shutdown_connections 2024-03-30T04:32:15.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.241+0000 7fd87da00700 1 -- 172.21.15.123:0/325188255 wait complete. 2024-03-30T04:32:15.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.241+0000 7fd87da00700 1 Processor -- start 2024-03-30T04:32:15.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.241+0000 7fd87da00700 1 -- start start 2024-03-30T04:32:15.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.242+0000 7fd87da00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd878070360 0x7fd878115ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:32:15.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.242+0000 7fd87da00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd878070c70 0x7fd8781161e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:32:15.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.242+0000 7fd87da00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd878116830 con 0x7fd878070c70 2024-03-30T04:32:15.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.242+0000 7fd87da00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd87810fc00 con 0x7fd878070360 2024-03-30T04:32:15.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.242+0000 7fd876000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd878070c70 0x7fd8781161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:32:15.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.242+0000 7fd876000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd878070c70 0x7fd8781161e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41142/0 (socket says 172.21.15.123:41142) 2024-03-30T04:32:15.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.242+0000 7fd876000700 1 -- 172.21.15.123:0/891826380 learned_addr learned my addr 172.21.15.123:0/891826380 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:32:15.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.242+0000 7fd876a00700 1 --2- 172.21.15.123:0/891826380 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd878070360 0x7fd878115ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:32:15.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.242+0000 7fd876000700 1 -- 172.21.15.123:0/891826380 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd878070360 msgr2=0x7fd878115ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:15.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.242+0000 7fd876000700 1 --2- 172.21.15.123:0/891826380 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd878070360 0x7fd878115ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:15.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.242+0000 7fd876000700 1 -- 172.21.15.123:0/891826380 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd864005040 con 0x7fd878070c70 2024-03-30T04:32:15.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.243+0000 7fd876a00700 1 --2- 172.21.15.123:0/891826380 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd878070360 0x7fd878115ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:32:15.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.243+0000 7fd876000700 1 --2- 172.21.15.123:0/891826380 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd878070c70 0x7fd8781161e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fd864005010 tx=0x7fd864009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:32:15.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.243+0000 7fd86ea00700 1 -- 172.21.15.123:0/891826380 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd86400e040 con 0x7fd878070c70 2024-03-30T04:32:15.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.243+0000 7fd86ea00700 1 -- 172.21.15.123:0/891826380 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd864004290 con 0x7fd878070c70 2024-03-30T04:32:15.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.243+0000 7fd87da00700 1 -- 172.21.15.123:0/891826380 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd87810fe80 con 0x7fd878070c70 2024-03-30T04:32:15.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.243+0000 7fd86ea00700 1 -- 172.21.15.123:0/891826380 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd864013e40 con 0x7fd878070c70 2024-03-30T04:32:15.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.243+0000 7fd87da00700 1 -- 172.21.15.123:0/891826380 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8781103c0 con 0x7fd878070c70 2024-03-30T04:32:15.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.244+0000 7fd87da00700 1 -- 172.21.15.123:0/891826380 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd878103bd0 con 0x7fd878070c70 2024-03-30T04:32:15.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.245+0000 7fd86ea00700 1 -- 172.21.15.123:0/891826380 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 32) v1 ==== 90705+0+0 (secure 0 0 0) 0x7fd864013620 con 0x7fd878070c70 2024-03-30T04:32:15.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.246+0000 7fd86ea00700 1 --2- 172.21.15.123:0/891826380 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fd85c06c8b0 0x7fd85c06ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:32:15.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.247+0000 7fd86ea00700 1 -- 172.21.15.123:0/891826380 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fd86408cc50 con 0x7fd878070c70 2024-03-30T04:32:15.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.247+0000 7fd876a00700 1 --2- 172.21.15.123:0/891826380 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fd85c06c8b0 0x7fd85c06ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:32:15.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.248+0000 7fd876a00700 1 --2- 172.21.15.123:0/891826380 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fd85c06c8b0 0x7fd85c06ed60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd868004600 tx=0x7fd86800b040 comp rx=0 tx=0).ready entity=mgr.14696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:32:15.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.251+0000 7fd86ea00700 1 -- 172.21.15.123:0/891826380 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7fd86405ace0 con 0x7fd878070c70 2024-03-30T04:32:15.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:14 smithi135 ceph-mon[44364]: pgmap v105: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:32:15.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:14 smithi135 ceph-mon[44364]: from='client.14862 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:32:15.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:14 smithi135 ceph-mon[44364]: from='client.14866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:32:15.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.554+0000 7fd87da00700 1 -- 172.21.15.123:0/891826380 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fd878111340 con 0x7fd878070c70 2024-03-30T04:32:15.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.555+0000 7fd86ea00700 1 -- 172.21.15.123:0/891826380 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+743 (secure 0 0 0) 0x7fd86405a870 con 0x7fd878070c70 2024-03-30T04:32:15.554 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:32:15.554 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:32:15.554 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 2 2024-03-30T04:32:15.554 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:32:15.554 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:32:15.554 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:32:15.554 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:32:15.554 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:32:15.554 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 6 2024-03-30T04:32:15.554 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:32:15.554 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:32:15.555 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 4 2024-03-30T04:32:15.555 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:32:15.555 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:32:15.555 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 12, 2024-03-30T04:32:15.555 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:32:15.555 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:32:15.555 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:32:15.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.557+0000 7fd87da00700 1 -- 172.21.15.123:0/891826380 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fd85c06c8b0 msgr2=0x7fd85c06ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:15.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.557+0000 7fd87da00700 1 --2- 172.21.15.123:0/891826380 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fd85c06c8b0 0x7fd85c06ed60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd868004600 tx=0x7fd86800b040 comp rx=0 tx=0).stop 2024-03-30T04:32:15.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.557+0000 7fd87da00700 1 -- 172.21.15.123:0/891826380 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd878070c70 msgr2=0x7fd8781161e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:15.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.557+0000 7fd87da00700 1 --2- 172.21.15.123:0/891826380 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd878070c70 0x7fd8781161e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fd864005010 tx=0x7fd864009e30 comp rx=0 tx=0).stop 2024-03-30T04:32:15.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.557+0000 7fd87da00700 1 -- 172.21.15.123:0/891826380 shutdown_connections 2024-03-30T04:32:15.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.557+0000 7fd87da00700 1 --2- 172.21.15.123:0/891826380 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7fd85c06c8b0 0x7fd85c06ed60 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:15.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.557+0000 7fd87da00700 1 --2- 172.21.15.123:0/891826380 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd878070360 0x7fd878115ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:15.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.557+0000 7fd87da00700 1 --2- 172.21.15.123:0/891826380 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd878070c70 0x7fd8781161e0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:15.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.557+0000 7fd87da00700 1 -- 172.21.15.123:0/891826380 >> 172.21.15.123:0/891826380 conn(0x7fd8780f89b0 msgr2=0x7fd8781015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:32:15.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.558+0000 7fd87da00700 1 -- 172.21.15.123:0/891826380 shutdown_connections 2024-03-30T04:32:15.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.558+0000 7fd87da00700 1 -- 172.21.15.123:0/891826380 wait complete. 2024-03-30T04:32:15.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.694+0000 7f1705c00700 1 -- 172.21.15.123:0/4176563438 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1700071f90 msgr2=0x7f1700070360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:15.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.694+0000 7f1705c00700 1 --2- 172.21.15.123:0/4176563438 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1700071f90 0x7f1700070360 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f16f0004970 tx=0x7f16f0009e30 comp rx=0 tx=0).stop 2024-03-30T04:32:15.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.694+0000 7f1705c00700 1 -- 172.21.15.123:0/4176563438 shutdown_connections 2024-03-30T04:32:15.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.694+0000 7f1705c00700 1 --2- 172.21.15.123:0/4176563438 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f17000708a0 0x7f1700070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:15.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.694+0000 7f1705c00700 1 --2- 172.21.15.123:0/4176563438 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1700071f90 0x7f1700070360 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:15.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.694+0000 7f1705c00700 1 -- 172.21.15.123:0/4176563438 >> 172.21.15.123:0/4176563438 conn(0x7f17000f89b0 msgr2=0x7f17000fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:32:15.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.694+0000 7f1705c00700 1 -- 172.21.15.123:0/4176563438 shutdown_connections 2024-03-30T04:32:15.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.694+0000 7f1705c00700 1 -- 172.21.15.123:0/4176563438 wait complete. 2024-03-30T04:32:15.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.695+0000 7f1705c00700 1 Processor -- start 2024-03-30T04:32:15.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.695+0000 7f1705c00700 1 -- start start 2024-03-30T04:32:15.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.695+0000 7f1705c00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f17000708a0 0x7f170011a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:32:15.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.695+0000 7f1705c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1700071f90 0x7f170011a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:32:15.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.695+0000 7f1705c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f170011ac50 con 0x7f1700071f90 2024-03-30T04:32:15.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.695+0000 7f1705c00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1700114020 con 0x7f17000708a0 2024-03-30T04:32:15.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.696+0000 7f16fe000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1700071f90 0x7f170011a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:32:15.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.696+0000 7f16fe000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1700071f90 0x7f170011a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41164/0 (socket says 172.21.15.123:41164) 2024-03-30T04:32:15.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.696+0000 7f16fe000700 1 -- 172.21.15.123:0/1941123039 learned_addr learned my addr 172.21.15.123:0/1941123039 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:32:15.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.696+0000 7f16fea00700 1 --2- 172.21.15.123:0/1941123039 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f17000708a0 0x7f170011a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:32:15.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.696+0000 7f16fe000700 1 -- 172.21.15.123:0/1941123039 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f17000708a0 msgr2=0x7f170011a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:15.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.696+0000 7f16fe000700 1 --2- 172.21.15.123:0/1941123039 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f17000708a0 0x7f170011a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:15.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.696+0000 7f16fe000700 1 -- 172.21.15.123:0/1941123039 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f0005040 con 0x7f1700071f90 2024-03-30T04:32:15.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.696+0000 7f16fe000700 1 --2- 172.21.15.123:0/1941123039 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1700071f90 0x7f170011a600 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f16ec00c8c0 tx=0x7f16ec0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:32:15.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.697+0000 7f16f6a00700 1 -- 172.21.15.123:0/1941123039 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16ec00a980 con 0x7f1700071f90 2024-03-30T04:32:15.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.697+0000 7f16f6a00700 1 -- 172.21.15.123:0/1941123039 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f16ec00ce60 con 0x7f1700071f90 2024-03-30T04:32:15.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.697+0000 7f1705c00700 1 -- 172.21.15.123:0/1941123039 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1700114300 con 0x7f1700071f90 2024-03-30T04:32:15.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.697+0000 7f16f6a00700 1 -- 172.21.15.123:0/1941123039 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16ec0113a0 con 0x7f1700071f90 2024-03-30T04:32:15.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.697+0000 7f1705c00700 1 -- 172.21.15.123:0/1941123039 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1700114840 con 0x7f1700071f90 2024-03-30T04:32:15.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.698+0000 7f1705c00700 1 -- 172.21.15.123:0/1941123039 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1700071aa0 con 0x7f1700071f90 2024-03-30T04:32:15.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.699+0000 7f16f6a00700 1 -- 172.21.15.123:0/1941123039 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 32) v1 ==== 90705+0+0 (secure 0 0 0) 0x7f16ec012400 con 0x7f1700071f90 2024-03-30T04:32:15.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.699+0000 7f16f6a00700 1 --2- 172.21.15.123:0/1941123039 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f16e406c8b0 0x7f16e406ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:32:15.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.700+0000 7f16f6a00700 1 -- 172.21.15.123:0/1941123039 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f16ec08bc50 con 0x7f1700071f90 2024-03-30T04:32:15.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.700+0000 7f16fea00700 1 --2- 172.21.15.123:0/1941123039 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f16e406c8b0 0x7f16e406ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:32:15.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.700+0000 7f16fea00700 1 --2- 172.21.15.123:0/1941123039 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f16e406c8b0 0x7f16e406ed60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f16f0009fd0 tx=0x7f16f0009e30 comp rx=0 tx=0).ready entity=mgr.14696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:32:15.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.704+0000 7f16f6a00700 1 -- 172.21.15.123:0/1941123039 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7f16ec059ba0 con 0x7f1700071f90 2024-03-30T04:32:15.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.965+0000 7f1705c00700 1 -- 172.21.15.123:0/1941123039 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f170005f5f0 con 0x7f1700071f90 2024-03-30T04:32:15.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.966+0000 7f16f6a00700 1 -- 172.21.15.123:0/1941123039 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+1855 (secure 0 0 0) 0x7f16ec059730 con 0x7f1700071f90 2024-03-30T04:32:15.965 INFO:teuthology.orchestra.run.smithi123.stdout:e22 2024-03-30T04:32:15.965 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:32:15.965 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:32:15.965 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:32:15.965 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:32:15.965 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:32:15.965 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:32:15.966 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 22 2024-03-30T04:32:15.966 INFO:teuthology.orchestra.run.smithi123.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-30T04:32:15.966 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:32:15.966 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:31:23.532125+0000 2024-03-30T04:32:15.966 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:32:15.966 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:32:15.966 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:32:15.966 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:32:15.966 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:32:15.966 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:32:15.966 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:32:15.966 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 46 2024-03-30T04:32:15.966 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:32:15.966 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T04:32:15.967 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T04:32:15.967 INFO:teuthology.orchestra.run.smithi123.stdout:up {0=14488,1=24309} 2024-03-30T04:32:15.967 INFO:teuthology.orchestra.run.smithi123.stdout:failed 2024-03-30T04:32:15.967 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:32:15.967 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:32:15.967 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:32:15.967 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:32:15.967 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:32:15.967 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:32:15.967 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T04:32:15.967 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{0:14488} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.123:6828/3037849622,v1:172.21.15.123:6829/3037849622] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:32:15.967 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{0:14500} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/2433675995,v1:172.21.15.123:6827/2433675995] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:32:15.967 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{1:24309} state up:active seq 4 export targets 0 join_fscid=1 addr [v2:172.21.15.135:6824/4050541062,v1:172.21.15.135:6825/4050541062] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:32:15.967 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{1:24319} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.135:6826/1096986122,v1:172.21.15.135:6827/1096986122] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:32:15.967 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:32:15.968 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:32:15.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.968+0000 7f1705c00700 1 -- 172.21.15.123:0/1941123039 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f16e406c8b0 msgr2=0x7f16e406ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:15.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.968+0000 7f1705c00700 1 --2- 172.21.15.123:0/1941123039 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f16e406c8b0 0x7f16e406ed60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f16f0009fd0 tx=0x7f16f0009e30 comp rx=0 tx=0).stop 2024-03-30T04:32:15.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.968+0000 7f1705c00700 1 -- 172.21.15.123:0/1941123039 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1700071f90 msgr2=0x7f170011a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:15.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.968+0000 7f1705c00700 1 --2- 172.21.15.123:0/1941123039 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1700071f90 0x7f170011a600 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f16ec00c8c0 tx=0x7f16ec0044c0 comp rx=0 tx=0).stop 2024-03-30T04:32:15.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.968+0000 7f1705c00700 1 -- 172.21.15.123:0/1941123039 shutdown_connections 2024-03-30T04:32:15.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.968+0000 7f1705c00700 1 --2- 172.21.15.123:0/1941123039 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7f16e406c8b0 0x7f16e406ed60 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:15.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.968+0000 7f1705c00700 1 --2- 172.21.15.123:0/1941123039 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f17000708a0 0x7f170011a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:15.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.968+0000 7f1705c00700 1 --2- 172.21.15.123:0/1941123039 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1700071f90 0x7f170011a600 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:15.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.968+0000 7f1705c00700 1 -- 172.21.15.123:0/1941123039 >> 172.21.15.123:0/1941123039 conn(0x7f17000f89b0 msgr2=0x7f17000f9b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:32:15.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.969+0000 7f1705c00700 1 -- 172.21.15.123:0/1941123039 shutdown_connections 2024-03-30T04:32:15.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:15.969+0000 7f1705c00700 1 -- 172.21.15.123:0/1941123039 wait complete. 2024-03-30T04:32:15.969 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 22 2024-03-30T04:32:16.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.109+0000 7faaa1000700 1 -- 172.21.15.123:0/1480448138 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa9c1037a0 msgr2=0x7faa9c0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:16.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.109+0000 7faaa1000700 1 --2- 172.21.15.123:0/1480448138 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa9c1037a0 0x7faa9c0fce10 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7faa88004940 tx=0x7faa88009e30 comp rx=0 tx=0).stop 2024-03-30T04:32:16.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.109+0000 7faaa1000700 1 -- 172.21.15.123:0/1480448138 shutdown_connections 2024-03-30T04:32:16.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.109+0000 7faaa1000700 1 --2- 172.21.15.123:0/1480448138 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa9c1037a0 0x7faa9c0fce10 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:16.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.109+0000 7faaa1000700 1 --2- 172.21.15.123:0/1480448138 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa9c102e00 0x7faa9c1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:16.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.109+0000 7faaa1000700 1 -- 172.21.15.123:0/1480448138 >> 172.21.15.123:0/1480448138 conn(0x7faa9c0f89b0 msgr2=0x7faa9c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:32:16.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.109+0000 7faaa1000700 1 -- 172.21.15.123:0/1480448138 shutdown_connections 2024-03-30T04:32:16.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.109+0000 7faaa1000700 1 -- 172.21.15.123:0/1480448138 wait complete. 2024-03-30T04:32:16.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.110+0000 7faaa1000700 1 Processor -- start 2024-03-30T04:32:16.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.110+0000 7faaa1000700 1 -- start start 2024-03-30T04:32:16.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.110+0000 7faaa1000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa9c102e00 0x7faa9c115da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:32:16.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.110+0000 7faaa1000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa9c1037a0 0x7faa9c1162e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:32:16.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.111+0000 7faaa1000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa9c116930 con 0x7faa9c1037a0 2024-03-30T04:32:16.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.111+0000 7faaa1000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa9c10fd00 con 0x7faa9c102e00 2024-03-30T04:32:16.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.111+0000 7faa99400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa9c1037a0 0x7faa9c1162e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:32:16.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.111+0000 7faa99400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa9c1037a0 0x7faa9c1162e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41172/0 (socket says 172.21.15.123:41172) 2024-03-30T04:32:16.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.111+0000 7faa99400700 1 -- 172.21.15.123:0/85278831 learned_addr learned my addr 172.21.15.123:0/85278831 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:32:16.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.111+0000 7faa99e00700 1 --2- 172.21.15.123:0/85278831 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa9c102e00 0x7faa9c115da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:32:16.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.111+0000 7faa99400700 1 -- 172.21.15.123:0/85278831 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa9c102e00 msgr2=0x7faa9c115da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:16.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.111+0000 7faa99400700 1 --2- 172.21.15.123:0/85278831 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa9c102e00 0x7faa9c115da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:16.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.111+0000 7faa99400700 1 -- 172.21.15.123:0/85278831 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa88005040 con 0x7faa9c1037a0 2024-03-30T04:32:16.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.111+0000 7faa99e00700 1 --2- 172.21.15.123:0/85278831 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa9c102e00 0x7faa9c115da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:32:16.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.111+0000 7faa99400700 1 --2- 172.21.15.123:0/85278831 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa9c1037a0 0x7faa9c1162e0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7faa88009390 tx=0x7faa88009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:32:16.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.112+0000 7faa92000700 1 -- 172.21.15.123:0/85278831 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa8800e040 con 0x7faa9c1037a0 2024-03-30T04:32:16.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.112+0000 7faa92000700 1 -- 172.21.15.123:0/85278831 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faa88013420 con 0x7faa9c1037a0 2024-03-30T04:32:16.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.112+0000 7faaa1000700 1 -- 172.21.15.123:0/85278831 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa9c10ff80 con 0x7faa9c1037a0 2024-03-30T04:32:16.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.112+0000 7faa92000700 1 -- 172.21.15.123:0/85278831 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa8801c630 con 0x7faa9c1037a0 2024-03-30T04:32:16.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.112+0000 7faaa1000700 1 -- 172.21.15.123:0/85278831 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa9c1104c0 con 0x7faa9c1037a0 2024-03-30T04:32:16.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.113+0000 7faaa1000700 1 -- 172.21.15.123:0/85278831 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa9c0fe550 con 0x7faa9c1037a0 2024-03-30T04:32:16.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.114+0000 7faa92000700 1 -- 172.21.15.123:0/85278831 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 32) v1 ==== 90705+0+0 (secure 0 0 0) 0x7faa880139e0 con 0x7faa9c1037a0 2024-03-30T04:32:16.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.115+0000 7faa92000700 1 --2- 172.21.15.123:0/85278831 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7faa8006c8b0 0x7faa8006ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:32:16.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.115+0000 7faa92000700 1 -- 172.21.15.123:0/85278831 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7faa8808cde0 con 0x7faa9c1037a0 2024-03-30T04:32:16.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.115+0000 7faa99e00700 1 --2- 172.21.15.123:0/85278831 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7faa8006c8b0 0x7faa8006ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:32:16.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.116+0000 7faa99e00700 1 --2- 172.21.15.123:0/85278831 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7faa8006c8b0 0x7faa8006ed60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7faa8c007fd0 tx=0x7faa8c009040 comp rx=0 tx=0).ready entity=mgr.14696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:32:16.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.119+0000 7faa92000700 1 -- 172.21.15.123:0/85278831 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7faa8805ad30 con 0x7faa9c1037a0 2024-03-30T04:32:16.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.351+0000 7faaa1000700 1 -- 172.21.15.123:0/85278831 --> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7faa9c109da0 con 0x7faa8006c8b0 2024-03-30T04:32:16.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:16 smithi135 ceph-mon[44364]: Upgrade: Updating mon.smithi123 2024-03-30T04:32:16.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:16 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:16.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:16 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T04:32:16.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:16 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T04:32:16.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:16 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:32:16.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:16 smithi135 ceph-mon[44364]: Deploying daemon mon.smithi123 on smithi123 2024-03-30T04:32:16.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:16 smithi135 ceph-mon[44364]: from='client.14870 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:32:16.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:16 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/891826380' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:32:16.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:16 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/1941123039' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:32:16.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.353+0000 7faa92000700 1 -- 172.21.15.123:0/85278831 <== mgr.14696 v2:172.21.15.123:6800/2125774379 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7faa9c109da0 con 0x7faa8006c8b0 2024-03-30T04:32:16.352 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:32:16.352 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:32:16.352 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:32:16.352 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T04:32:16.352 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:32:16.352 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr" 2024-03-30T04:32:16.352 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:32:16.352 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "2/21 daemons upgraded", 2024-03-30T04:32:16.352 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mon daemons", 2024-03-30T04:32:16.352 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:32:16.352 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:32:16.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.355+0000 7faaa1000700 1 -- 172.21.15.123:0/85278831 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7faa8006c8b0 msgr2=0x7faa8006ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:16.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.355+0000 7faaa1000700 1 --2- 172.21.15.123:0/85278831 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7faa8006c8b0 0x7faa8006ed60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7faa8c007fd0 tx=0x7faa8c009040 comp rx=0 tx=0).stop 2024-03-30T04:32:16.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.355+0000 7faaa1000700 1 -- 172.21.15.123:0/85278831 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa9c1037a0 msgr2=0x7faa9c1162e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:16.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.355+0000 7faaa1000700 1 --2- 172.21.15.123:0/85278831 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa9c1037a0 0x7faa9c1162e0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7faa88009390 tx=0x7faa88009e30 comp rx=0 tx=0).stop 2024-03-30T04:32:16.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.355+0000 7faaa1000700 1 -- 172.21.15.123:0/85278831 shutdown_connections 2024-03-30T04:32:16.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.355+0000 7faaa1000700 1 --2- 172.21.15.123:0/85278831 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7faa8006c8b0 0x7faa8006ed60 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:16.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.355+0000 7faaa1000700 1 --2- 172.21.15.123:0/85278831 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa9c102e00 0x7faa9c115da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:16.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.355+0000 7faaa1000700 1 --2- 172.21.15.123:0/85278831 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa9c1037a0 0x7faa9c1162e0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:16.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.355+0000 7faaa1000700 1 -- 172.21.15.123:0/85278831 >> 172.21.15.123:0/85278831 conn(0x7faa9c0f89b0 msgr2=0x7faa9c0f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:32:16.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.355+0000 7faaa1000700 1 -- 172.21.15.123:0/85278831 shutdown_connections 2024-03-30T04:32:16.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.355+0000 7faaa1000700 1 -- 172.21.15.123:0/85278831 wait complete. 2024-03-30T04:32:16.404 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:16 smithi123 ceph-mon[35245]: Upgrade: Updating mon.smithi123 2024-03-30T04:32:16.405 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:16 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:16.405 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:16 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T04:32:16.405 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:16 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T04:32:16.405 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:16 smithi123 ceph-mon[35245]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:32:16.405 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:16 smithi123 ceph-mon[35245]: Deploying daemon mon.smithi123 on smithi123 2024-03-30T04:32:16.405 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:16 smithi123 ceph-mon[35245]: from='client.14870 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:32:16.405 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:16 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/891826380' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:32:16.405 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:16 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/1941123039' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:32:16.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.502+0000 7ff2ee800700 1 -- 172.21.15.123:0/1757158220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2e8102e00 msgr2=0x7ff2e81031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:16.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.502+0000 7ff2ee800700 1 --2- 172.21.15.123:0/1757158220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2e8102e00 0x7ff2e81031d0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7ff2d8004940 tx=0x7ff2d8009e30 comp rx=0 tx=0).stop 2024-03-30T04:32:16.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.502+0000 7ff2ee800700 1 -- 172.21.15.123:0/1757158220 shutdown_connections 2024-03-30T04:32:16.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.502+0000 7ff2ee800700 1 --2- 172.21.15.123:0/1757158220 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff2e81037a0 0x7ff2e80fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:16.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.502+0000 7ff2ee800700 1 --2- 172.21.15.123:0/1757158220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2e8102e00 0x7ff2e81031d0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:16.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.502+0000 7ff2ee800700 1 -- 172.21.15.123:0/1757158220 >> 172.21.15.123:0/1757158220 conn(0x7ff2e80f89b0 msgr2=0x7ff2e80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:32:16.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.502+0000 7ff2ee800700 1 -- 172.21.15.123:0/1757158220 shutdown_connections 2024-03-30T04:32:16.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.502+0000 7ff2ee800700 1 -- 172.21.15.123:0/1757158220 wait complete. 2024-03-30T04:32:16.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.503+0000 7ff2ee800700 1 Processor -- start 2024-03-30T04:32:16.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.503+0000 7ff2ee800700 1 -- start start 2024-03-30T04:32:16.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.503+0000 7ff2ee800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2e8102e00 0x7ff2e80702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:32:16.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.503+0000 7ff2ee800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff2e81037a0 0x7ff2e8070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:32:16.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.503+0000 7ff2ee800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e8070d90 con 0x7ff2e8102e00 2024-03-30T04:32:16.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.503+0000 7ff2ee800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e8070ed0 con 0x7ff2e81037a0 2024-03-30T04:32:16.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.503+0000 7ff2e7400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2e8102e00 0x7ff2e80702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:32:16.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.503+0000 7ff2e7400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2e8102e00 0x7ff2e80702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41188/0 (socket says 172.21.15.123:41188) 2024-03-30T04:32:16.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.503+0000 7ff2e7400700 1 -- 172.21.15.123:0/2939650112 learned_addr learned my addr 172.21.15.123:0/2939650112 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:32:16.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.504+0000 7ff2e7400700 1 -- 172.21.15.123:0/2939650112 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff2e81037a0 msgr2=0x7ff2e8070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:16.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.504+0000 7ff2e7400700 1 --2- 172.21.15.123:0/2939650112 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff2e81037a0 0x7ff2e8070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:16.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.504+0000 7ff2e7400700 1 -- 172.21.15.123:0/2939650112 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2d8005040 con 0x7ff2e8102e00 2024-03-30T04:32:16.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.504+0000 7ff2e7400700 1 --2- 172.21.15.123:0/2939650112 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2e8102e00 0x7ff2e80702c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7ff2d8009390 tx=0x7ff2d8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:32:16.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.504+0000 7ff2df400700 1 -- 172.21.15.123:0/2939650112 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff2d800e040 con 0x7ff2e8102e00 2024-03-30T04:32:16.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.504+0000 7ff2df400700 1 -- 172.21.15.123:0/2939650112 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff2d8013420 con 0x7ff2e8102e00 2024-03-30T04:32:16.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.504+0000 7ff2df400700 1 -- 172.21.15.123:0/2939650112 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff2d801c630 con 0x7ff2e8102e00 2024-03-30T04:32:16.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.504+0000 7ff2ee800700 1 -- 172.21.15.123:0/2939650112 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2e8071120 con 0x7ff2e8102e00 2024-03-30T04:32:16.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.504+0000 7ff2ee800700 1 -- 172.21.15.123:0/2939650112 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2e811a250 con 0x7ff2e8102e00 2024-03-30T04:32:16.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.505+0000 7ff2ee800700 1 -- 172.21.15.123:0/2939650112 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2e80fe550 con 0x7ff2e8102e00 2024-03-30T04:32:16.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.506+0000 7ff2df400700 1 -- 172.21.15.123:0/2939650112 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 32) v1 ==== 90705+0+0 (secure 0 0 0) 0x7ff2d80139e0 con 0x7ff2e8102e00 2024-03-30T04:32:16.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.507+0000 7ff2df400700 1 --2- 172.21.15.123:0/2939650112 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7ff2cc06c8b0 0x7ff2cc06ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:32:16.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.507+0000 7ff2df400700 1 -- 172.21.15.123:0/2939650112 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7ff2d808cc20 con 0x7ff2e8102e00 2024-03-30T04:32:16.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.507+0000 7ff2e6a00700 1 --2- 172.21.15.123:0/2939650112 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7ff2cc06c8b0 0x7ff2cc06ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:32:16.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.508+0000 7ff2e6a00700 1 --2- 172.21.15.123:0/2939650112 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7ff2cc06c8b0 0x7ff2cc06ed60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff2e8071c00 tx=0x7ff2d4009ce0 comp rx=0 tx=0).ready entity=mgr.14696 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:32:16.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.512+0000 7ff2df400700 1 -- 172.21.15.123:0/2939650112 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181579 (secure 0 0 0) 0x7ff2d8057220 con 0x7ff2e8102e00 2024-03-30T04:32:16.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.798+0000 7ff2ee800700 1 -- 172.21.15.123:0/2939650112 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7ff2e805f5f0 con 0x7ff2e8102e00 2024-03-30T04:32:16.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.799+0000 7ff2df400700 1 -- 172.21.15.123:0/2939650112 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7ff2d805a840 con 0x7ff2e8102e00 2024-03-30T04:32:16.798 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_OK 2024-03-30T04:32:16.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.801+0000 7ff2ee800700 1 -- 172.21.15.123:0/2939650112 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7ff2cc06c8b0 msgr2=0x7ff2cc06ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:16.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.801+0000 7ff2ee800700 1 --2- 172.21.15.123:0/2939650112 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7ff2cc06c8b0 0x7ff2cc06ed60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff2e8071c00 tx=0x7ff2d4009ce0 comp rx=0 tx=0).stop 2024-03-30T04:32:16.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.801+0000 7ff2ee800700 1 -- 172.21.15.123:0/2939650112 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2e8102e00 msgr2=0x7ff2e80702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:16.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.801+0000 7ff2ee800700 1 --2- 172.21.15.123:0/2939650112 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2e8102e00 0x7ff2e80702c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7ff2d8009390 tx=0x7ff2d8009e30 comp rx=0 tx=0).stop 2024-03-30T04:32:16.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.801+0000 7ff2ee800700 1 -- 172.21.15.123:0/2939650112 shutdown_connections 2024-03-30T04:32:16.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.801+0000 7ff2ee800700 1 --2- 172.21.15.123:0/2939650112 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2e8102e00 0x7ff2e80702c0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:16.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.801+0000 7ff2ee800700 1 --2- 172.21.15.123:0/2939650112 >> [v2:172.21.15.123:6800/2125774379,v1:172.21.15.123:6801/2125774379] conn(0x7ff2cc06c8b0 0x7ff2cc06ed60 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:16.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.801+0000 7ff2ee800700 1 --2- 172.21.15.123:0/2939650112 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff2e81037a0 0x7ff2e8070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:16.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.801+0000 7ff2ee800700 1 -- 172.21.15.123:0/2939650112 >> 172.21.15.123:0/2939650112 conn(0x7ff2e80f89b0 msgr2=0x7ff2e80f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:32:16.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.802+0000 7ff2ee800700 1 -- 172.21.15.123:0/2939650112 shutdown_connections 2024-03-30T04:32:16.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:16.802+0000 7ff2ee800700 1 -- 172.21.15.123:0/2939650112 wait complete. 2024-03-30T04:32:17.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:17 smithi135 ceph-mon[44364]: pgmap v106: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:32:17.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:17 smithi135 ceph-mon[44364]: from='client.14882 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:32:17.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:17 smithi135 ceph-mon[44364]: from='client.? 172.21.15.123:0/2939650112' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:32:17.374 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:17 smithi123 ceph-mon[35245]: pgmap v106: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:32:17.374 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:17 smithi123 ceph-mon[35245]: from='client.14882 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:32:17.375 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:17 smithi123 ceph-mon[35245]: from='client.? 172.21.15.123:0/2939650112' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:32:17.375 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:17 smithi123 systemd[1]: Stopping Ceph mon.smithi123 for 2e77a2be-ee4c-11ee-b647-cb9ed24678a4... 2024-03-30T04:32:17.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:17 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-mon-smithi123[35221]: 2024-03-30T04:32:17.374+0000 7f4eec000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi123 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T04:32:17.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:17 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-mon-smithi123[35221]: 2024-03-30T04:32:17.374+0000 7f4eec000700 -1 mon.smithi123@0(leader) e2 *** Got Signal Terminated *** 2024-03-30T04:32:18.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:17 smithi123 bash[139171]: ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-mon-smithi123 2024-03-30T04:32:18.570 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:18 smithi123 systemd[1]: ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4@mon.smithi123.service: Succeeded. 2024-03-30T04:32:18.570 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:18 smithi123 systemd[1]: Stopped Ceph mon.smithi123 for 2e77a2be-ee4c-11ee-b647-cb9ed24678a4. 2024-03-30T04:32:18.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:18 smithi123 systemd[1]: Starting Ceph mon.smithi123 for 2e77a2be-ee4c-11ee-b647-cb9ed24678a4... 2024-03-30T04:32:19.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: set uid:gid to 167:167 (ceph:ceph) 2024-03-30T04:32:19.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable), process ceph-mon, pid 7 2024-03-30T04:32:19.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: pidfile_write: ignore empty --pid-file 2024-03-30T04:32:19.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: load: jerasure load: lrc 2024-03-30T04:32:19.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: RocksDB version: 7.9.2 2024-03-30T04:32:19.718 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Git sha 0 2024-03-30T04:32:19.718 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Compile date 2024-03-29 11:49:16 2024-03-30T04:32:19.718 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: DB SUMMARY 2024-03-30T04:32:19.718 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: DB Session ID: BGP7ZFGL1IZ52K4WQ7J5 2024-03-30T04:32:19.718 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: CURRENT file: CURRENT 2024-03-30T04:32:19.718 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: IDENTITY file: IDENTITY 2024-03-30T04:32:19.718 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: MANIFEST file: MANIFEST-000015 size: 1147 Bytes 2024-03-30T04:32:19.718 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi123/store.db dir, Total Num: 1, files: 000033.sst 2024-03-30T04:32:19.718 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi123/store.db: 000031.log size: 3153373 ; 2024-03-30T04:32:19.718 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.error_if_exists: 0 2024-03-30T04:32:19.718 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.create_if_missing: 0 2024-03-30T04:32:19.718 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.paranoid_checks: 1 2024-03-30T04:32:19.719 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-30T04:32:19.719 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-30T04:32:19.719 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-30T04:32:19.719 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.env: 0x558c5ca9e720 2024-03-30T04:32:19.719 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.fs: PosixFileSystem 2024-03-30T04:32:19.719 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.info_log: 0x558c5f7a73c0 2024-03-30T04:32:19.719 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.max_file_opening_threads: 16 2024-03-30T04:32:19.719 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.statistics: (nil) 2024-03-30T04:32:19.719 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.use_fsync: 0 2024-03-30T04:32:19.719 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.max_log_file_size: 0 2024-03-30T04:32:19.719 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-30T04:32:19.719 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-30T04:32:19.719 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.keep_log_file_num: 1000 2024-03-30T04:32:19.719 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.recycle_log_file_num: 0 2024-03-30T04:32:19.720 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.allow_fallocate: 1 2024-03-30T04:32:19.720 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.allow_mmap_reads: 0 2024-03-30T04:32:19.720 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.allow_mmap_writes: 0 2024-03-30T04:32:19.720 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.use_direct_reads: 0 2024-03-30T04:32:19.720 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-30T04:32:19.720 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.create_missing_column_families: 0 2024-03-30T04:32:19.720 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.db_log_dir: 2024-03-30T04:32:19.720 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.wal_dir: 2024-03-30T04:32:19.720 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-30T04:32:19.720 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-30T04:32:19.720 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-30T04:32:19.720 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-30T04:32:19.720 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-30T04:32:19.720 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-30T04:32:19.721 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.advise_random_on_open: 1 2024-03-30T04:32:19.721 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.db_write_buffer_size: 0 2024-03-30T04:32:19.721 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.write_buffer_manager: 0x558c5ea1e140 2024-03-30T04:32:19.721 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-30T04:32:19.721 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-30T04:32:19.721 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-30T04:32:19.721 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.rate_limiter: (nil) 2024-03-30T04:32:19.721 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-30T04:32:19.721 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.wal_recovery_mode: 2 2024-03-30T04:32:19.721 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.enable_thread_tracking: 0 2024-03-30T04:32:19.721 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.enable_pipelined_write: 0 2024-03-30T04:32:19.721 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.unordered_write: 0 2024-03-30T04:32:19.721 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-30T04:32:19.721 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-30T04:32:19.721 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-30T04:32:19.722 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-30T04:32:19.722 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.row_cache: None 2024-03-30T04:32:19.722 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.wal_filter: None 2024-03-30T04:32:19.722 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-30T04:32:19.722 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.allow_ingest_behind: 0 2024-03-30T04:32:19.722 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.two_write_queues: 0 2024-03-30T04:32:19.722 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.manual_wal_flush: 0 2024-03-30T04:32:19.722 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.wal_compression: 0 2024-03-30T04:32:19.722 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.atomic_flush: 0 2024-03-30T04:32:19.722 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-30T04:32:19.722 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-30T04:32:19.722 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-30T04:32:19.722 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.log_readahead_size: 0 2024-03-30T04:32:19.722 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-30T04:32:19.723 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.best_efforts_recovery: 0 2024-03-30T04:32:19.723 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-30T04:32:19.723 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-30T04:32:19.723 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.allow_data_in_errors: 0 2024-03-30T04:32:19.723 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.db_host_id: __hostname__ 2024-03-30T04:32:19.723 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-30T04:32:19.723 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.max_background_jobs: 2 2024-03-30T04:32:19.723 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.max_background_compactions: -1 2024-03-30T04:32:19.723 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.max_subcompactions: 1 2024-03-30T04:32:19.723 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-30T04:32:19.723 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-30T04:32:19.723 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-30T04:32:19.723 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.max_total_wal_size: 0 2024-03-30T04:32:19.723 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-30T04:32:19.724 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-30T04:32:19.724 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-30T04:32:19.724 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-30T04:32:19.724 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.max_open_files: -1 2024-03-30T04:32:19.724 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.bytes_per_sync: 0 2024-03-30T04:32:19.724 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-30T04:32:19.724 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-30T04:32:19.724 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.compaction_readahead_size: 0 2024-03-30T04:32:19.724 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.max_background_flushes: -1 2024-03-30T04:32:19.724 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Compression algorithms supported: 2024-03-30T04:32:19.724 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-30T04:32:19.724 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: kZSTD supported: 0 2024-03-30T04:32:19.724 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: kXpressCompression supported: 0 2024-03-30T04:32:19.724 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: kLZ4HCCompression supported: 1 2024-03-30T04:32:19.724 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: kZlibCompression supported: 1 2024-03-30T04:32:19.725 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: kSnappyCompression supported: 1 2024-03-30T04:32:19.725 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: kLZ4Compression supported: 1 2024-03-30T04:32:19.725 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: kBZip2Compression supported: 0 2024-03-30T04:32:19.725 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-30T04:32:19.725 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-30T04:32:19.725 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi123/store.db/MANIFEST-000015 2024-03-30T04:32:19.725 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-30T04:32:19.725 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-30T04:32:19.725 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.merge_operator: 2024-03-30T04:32:19.725 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.compaction_filter: None 2024-03-30T04:32:19.725 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.compaction_filter_factory: None 2024-03-30T04:32:19.725 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.sst_partitioner_factory: None 2024-03-30T04:32:19.725 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-30T04:32:19.725 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-30T04:32:19.725 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x558c5f7a74e0) 2024-03-30T04:32:19.726 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: cache_index_and_filter_blocks: 1 2024-03-30T04:32:19.726 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-30T04:32:19.726 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-30T04:32:19.726 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: pin_top_level_index_and_filter: 1 2024-03-30T04:32:19.726 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: index_type: 0 2024-03-30T04:32:19.726 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: data_block_index_type: 0 2024-03-30T04:32:19.726 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: index_shortening: 1 2024-03-30T04:32:19.726 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-30T04:32:19.726 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: checksum: 4 2024-03-30T04:32:19.726 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: no_block_cache: 0 2024-03-30T04:32:19.726 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: block_cache: 0x558c5eabf1f0 2024-03-30T04:32:19.726 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: block_cache_name: BinnedLRUCache 2024-03-30T04:32:19.726 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: block_cache_options: 2024-03-30T04:32:19.726 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: capacity : 536870912 2024-03-30T04:32:19.727 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: num_shard_bits : 4 2024-03-30T04:32:19.727 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: strict_capacity_limit : 0 2024-03-30T04:32:19.727 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: high_pri_pool_ratio: 0.000 2024-03-30T04:32:19.727 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: block_cache_compressed: (nil) 2024-03-30T04:32:19.727 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: persistent_cache: (nil) 2024-03-30T04:32:19.727 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: block_size: 4096 2024-03-30T04:32:19.727 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: block_size_deviation: 10 2024-03-30T04:32:19.727 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: block_restart_interval: 16 2024-03-30T04:32:19.727 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: index_block_restart_interval: 1 2024-03-30T04:32:19.727 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: metadata_block_size: 4096 2024-03-30T04:32:19.727 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: partition_filters: 0 2024-03-30T04:32:19.727 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: use_delta_encoding: 1 2024-03-30T04:32:19.727 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: filter_policy: bloomfilter 2024-03-30T04:32:19.727 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: whole_key_filtering: 1 2024-03-30T04:32:19.727 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: verify_compression: 0 2024-03-30T04:32:19.728 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: read_amp_bytes_per_bit: 0 2024-03-30T04:32:19.728 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: format_version: 5 2024-03-30T04:32:19.728 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: enable_index_compression: 1 2024-03-30T04:32:19.728 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: block_align: 0 2024-03-30T04:32:19.728 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: max_auto_readahead_size: 262144 2024-03-30T04:32:19.728 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: prepopulate_block_cache: 0 2024-03-30T04:32:19.728 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: initial_auto_readahead_size: 8192 2024-03-30T04:32:19.728 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: num_file_reads_for_auto_readahead: 2 2024-03-30T04:32:19.728 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.write_buffer_size: 33554432 2024-03-30T04:32:19.728 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.max_write_buffer_number: 2 2024-03-30T04:32:19.728 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.compression: NoCompression 2024-03-30T04:32:19.728 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.bottommost_compression: Disabled 2024-03-30T04:32:19.728 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.prefix_extractor: nullptr 2024-03-30T04:32:19.728 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-30T04:32:19.728 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.num_levels: 7 2024-03-30T04:32:19.729 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-30T04:32:19.729 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-30T04:32:19.729 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-30T04:32:19.729 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-30T04:32:19.729 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-30T04:32:19.729 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-30T04:32:19.729 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-30T04:32:19.729 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-30T04:32:19.729 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-30T04:32:19.729 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-30T04:32:19.729 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-30T04:32:19.729 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-30T04:32:19.729 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-30T04:32:19.729 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.compression_opts.level: 32767 2024-03-30T04:32:19.729 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.compression_opts.strategy: 0 2024-03-30T04:32:19.730 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-30T04:32:19.730 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-30T04:32:19.730 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-30T04:32:19.730 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-30T04:32:19.730 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.compression_opts.enabled: false 2024-03-30T04:32:19.730 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-30T04:32:19.730 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-30T04:32:19.730 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-30T04:32:19.730 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-30T04:32:19.730 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.target_file_size_base: 67108864 2024-03-30T04:32:19.730 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-30T04:32:19.730 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-30T04:32:19.730 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-30T04:32:19.730 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-30T04:32:19.731 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-30T04:32:19.731 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-30T04:32:19.731 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-30T04:32:19.731 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-30T04:32:19.731 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-30T04:32:19.731 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-30T04:32:19.731 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-30T04:32:19.731 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-30T04:32:19.731 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-30T04:32:19.731 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-30T04:32:19.731 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.arena_block_size: 1048576 2024-03-30T04:32:19.731 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-30T04:32:19.731 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-30T04:32:19.731 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.disable_auto_compactions: 0 2024-03-30T04:32:19.731 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-30T04:32:19.732 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-30T04:32:19.732 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-30T04:32:19.732 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-30T04:32:19.733 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-30T04:32:19.733 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-30T04:32:19.733 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-30T04:32:19.733 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-30T04:32:19.733 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-30T04:32:19.733 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-30T04:32:19.733 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.table_properties_collectors: 2024-03-30T04:32:19.733 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.inplace_update_support: 0 2024-03-30T04:32:19.733 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-30T04:32:19.733 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-30T04:32:19.733 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-30T04:32:19.733 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-30T04:32:19.733 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.bloom_locality: 0 2024-03-30T04:32:19.733 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.max_successive_merges: 0 2024-03-30T04:32:19.733 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-30T04:32:19.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.paranoid_file_checks: 0 2024-03-30T04:32:19.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.force_consistency_checks: 1 2024-03-30T04:32:19.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.report_bg_io_stats: 0 2024-03-30T04:32:19.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.ttl: 2592000 2024-03-30T04:32:19.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-30T04:32:19.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-30T04:32:19.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-30T04:32:19.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.enable_blob_files: false 2024-03-30T04:32:19.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.min_blob_size: 0 2024-03-30T04:32:19.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.blob_file_size: 268435456 2024-03-30T04:32:19.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-30T04:32:19.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-30T04:32:19.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-30T04:32:19.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-30T04:32:19.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-30T04:32:19.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.blob_file_starting_level: 0 2024-03-30T04:32:19.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-30T04:32:19.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 33.sst 2024-03-30T04:32:19.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-30T04:32:19.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi123/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 35, last_sequence is 11906, log_number is 31,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-30T04:32:19.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 31 2024-03-30T04:32:19.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 1f886c23-95f4-477d-819d-a086b1c53672 2024-03-30T04:32:19.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711773139271319, "job": 1, "event": "recovery_started", "wal_files": [31]} 2024-03-30T04:32:19.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #31 mode 2 2024-03-30T04:32:19.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711773139285461, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 36, "file_size": 2783393, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 11900, "largest_seqno": 14566, "table_properties": {"data_size": 2772200, "index_size": 6844, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 3333, "raw_key_size": 29385, "raw_average_key_size": 22, "raw_value_size": 2746428, "raw_average_value_size": 2091, "num_data_blocks": 311, "num_entries": 1313, "num_filter_entries": 1313, "num_deletions": 260, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711773139, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "1f886c23-95f4-477d-819d-a086b1c53672", "db_session_id": "BGP7ZFGL1IZ52K4WQ7J5", "orig_file_number": 36, "seqno_to_time_mapping": "N/A"}} 2024-03-30T04:32:19.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711773139285613, "job": 1, "event": "recovery_finished"} 2024-03-30T04:32:19.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: [db/version_set.cc:5047] Creating manifest 38 2024-03-30T04:32:19.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-30T04:32:19.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi123/store.db/000031.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-30T04:32:19.736 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x558c5eb5c000 2024-03-30T04:32:19.736 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: DB pointer 0x558c5eb4a000 2024-03-30T04:32:19.736 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-30T04:32:19.736 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-30T04:32:19.736 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: ** DB Stats ** 2024-03-30T04:32:19.736 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-30T04:32:19.736 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-30T04:32:19.736 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-30T04:32:19.736 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-30T04:32:19.736 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-30T04:32:19.736 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-30T04:32:19.736 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-30T04:32:19.736 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: 2024-03-30T04:32:19.736 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: ** Compaction Stats [default] ** 2024-03-30T04:32:19.737 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-30T04:32:19.737 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-30T04:32:19.737 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: L0 1/0 2.65 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 464.4 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-03-30T04:32:19.737 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: L6 1/0 6.42 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-30T04:32:19.737 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: Sum 2/0 9.07 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 464.4 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-03-30T04:32:19.737 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 464.4 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-03-30T04:32:19.737 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: 2024-03-30T04:32:19.737 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: ** Compaction Stats [default] ** 2024-03-30T04:32:19.737 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-30T04:32:19.737 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-30T04:32:19.737 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 464.4 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-03-30T04:32:19.737 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: 2024-03-30T04:32:19.737 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-30T04:32:19.737 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: 2024-03-30T04:32:19.737 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-30T04:32:19.738 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: Flush(GB): cumulative 0.003, interval 0.003 2024-03-30T04:32:19.738 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-30T04:32:19.738 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-30T04:32:19.738 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-30T04:32:19.738 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-30T04:32:19.738 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: Cumulative compaction: 0.00 GB write, 126.14 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-30T04:32:19.738 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: Interval compaction: 0.00 GB write, 126.14 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-30T04:32:19.738 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-30T04:32:19.738 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: Block cache BinnedLRUCache@0x558c5eabf1f0#7 capacity: 512.00 MB usage: 12.11 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-03-30T04:32:19.738 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,4.03 KB,0.0007689%) IndexBlock(1,8.08 KB,0.00154078%) Misc(1,0.00 KB,0%) 2024-03-30T04:32:19.738 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: 2024-03-30T04:32:19.738 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-30T04:32:19.738 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: starting mon.smithi123 rank 0 at public addrs [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] at bind addrs [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi123 fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 2024-03-30T04:32:19.738 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: mon.smithi123@-1(???) e2 preinit fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 2024-03-30T04:32:19.739 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: mon.smithi123@-1(???).mds e22 new map 2024-03-30T04:32:19.739 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: mon.smithi123@-1(???).mds e22 print_map 2024-03-30T04:32:19.739 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: e22 2024-03-30T04:32:19.739 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:32:19.739 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:32:19.739 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: legacy client fscid: 1 2024-03-30T04:32:19.739 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: 2024-03-30T04:32:19.739 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: Filesystem 'cephfs' (1) 2024-03-30T04:32:19.739 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: fs_name cephfs 2024-03-30T04:32:19.739 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: epoch 22 2024-03-30T04:32:19.739 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-30T04:32:19.739 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:32:19.739 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: modified 2024-03-30T04:31:23.532125+0000 2024-03-30T04:32:19.739 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: tableserver 0 2024-03-30T04:32:19.740 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: root 0 2024-03-30T04:32:19.740 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: session_timeout 60 2024-03-30T04:32:19.740 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: session_autoclose 300 2024-03-30T04:32:19.740 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: max_file_size 1099511627776 2024-03-30T04:32:19.740 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: max_xattr_size 65536 2024-03-30T04:32:19.740 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: required_client_features {} 2024-03-30T04:32:19.740 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: last_failure 0 2024-03-30T04:32:19.740 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: last_failure_osd_epoch 46 2024-03-30T04:32:19.740 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:32:19.740 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: max_mds 2 2024-03-30T04:32:19.740 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: in 0,1 2024-03-30T04:32:19.740 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: up {0=14488,1=24309} 2024-03-30T04:32:19.740 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: failed 2024-03-30T04:32:19.740 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: damaged 2024-03-30T04:32:19.741 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: stopped 2024-03-30T04:32:19.741 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: data_pools [3] 2024-03-30T04:32:19.741 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: metadata_pool 2 2024-03-30T04:32:19.741 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: inline_data disabled 2024-03-30T04:32:19.741 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: balancer 2024-03-30T04:32:19.741 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: bal_rank_mask -1 2024-03-30T04:32:19.741 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: standby_count_wanted 1 2024-03-30T04:32:19.741 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: [mds.cephfs.smithi123.dvtind{0:14488} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.123:6828/3037849622,v1:172.21.15.123:6829/3037849622] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:32:19.741 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: [mds.cephfs.smithi123.tqxvzr{0:14500} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/2433675995,v1:172.21.15.123:6827/2433675995] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:32:19.741 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: [mds.cephfs.smithi135.zvnlap{1:24309} state up:active seq 4 export targets 0 join_fscid=1 addr [v2:172.21.15.135:6824/4050541062,v1:172.21.15.135:6825/4050541062] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:32:19.741 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: [mds.cephfs.smithi135.xwgjvb{1:24319} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.135:6826/1096986122,v1:172.21.15.135:6827/1096986122] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:32:19.741 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: 2024-03-30T04:32:19.741 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout: 2024-03-30T04:32:19.741 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: mon.smithi123@-1(???).osd e48 crush map has features 3314933000852226048, adjusting msgr requires 2024-03-30T04:32:19.741 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: mon.smithi123@-1(???).osd e48 crush map has features 288514051259236352, adjusting msgr requires 2024-03-30T04:32:19.741 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: mon.smithi123@-1(???).osd e48 crush map has features 288514051259236352, adjusting msgr requires 2024-03-30T04:32:19.742 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: mon.smithi123@-1(???).osd e48 crush map has features 288514051259236352, adjusting msgr requires 2024-03-30T04:32:19.742 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 ceph-mon[139369]: mon.smithi123@-1(???).paxosservice(auth 1..21) refresh upgraded, format 0 -> 3 2024-03-30T04:32:19.742 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 bash[139331]: 2b3ed6979ed2e5ea675f6f7108a37b2099b8af34a1030b7c8e001a9c7e80b8a5 2024-03-30T04:32:19.742 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:19 smithi123 systemd[1]: Started Ceph mon.smithi123 for 2e77a2be-ee4c-11ee-b647-cb9ed24678a4. 2024-03-30T04:32:21.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:20 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mon metadata", "id": "smithi123"}]: dispatch 2024-03-30T04:32:21.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:20 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-03-30T04:32:21.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:20 smithi135 ceph-mon[44364]: pgmap v107: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:32:21.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:20 smithi135 ceph-mon[44364]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:32:21.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:20 smithi135 ceph-mon[44364]: mon.smithi123 calling monitor election 2024-03-30T04:32:21.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:20 smithi135 ceph-mon[44364]: mon.smithi123 is new leader, mons smithi123,smithi135 in quorum (ranks 0,1) 2024-03-30T04:32:21.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:20 smithi135 ceph-mon[44364]: monmap e2: 2 mons at {smithi123=[v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0],smithi135=[v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-30T04:32:21.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:20 smithi135 ceph-mon[44364]: fsmap cephfs:2 {0=cephfs.smithi123.dvtind=up:active,1=cephfs.smithi135.zvnlap=up:active} 2 up:standby-replay 2024-03-30T04:32:21.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:20 smithi135 ceph-mon[44364]: osdmap e48: 6 total, 6 up, 6 in 2024-03-30T04:32:21.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:20 smithi135 ceph-mon[44364]: mgrmap e32: smithi123.xgksqq(active, since 3m), standbys: smithi135.kyblli 2024-03-30T04:32:21.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:20 smithi135 ceph-mon[44364]: mgrmap e33: smithi123.xgksqq(active, since 3m), standbys: smithi135.kyblli 2024-03-30T04:32:21.103 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:20 smithi135 ceph-mon[44364]: overall HEALTH_OK 2024-03-30T04:32:21.103 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:20 smithi135 ceph-mon[44364]: from='mgr.14696 ' entity='' 2024-03-30T04:32:21.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:20 smithi123 ceph-mon[139369]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mon metadata", "id": "smithi123"}]: dispatch 2024-03-30T04:32:21.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:20 smithi123 ceph-mon[139369]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-03-30T04:32:21.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:20 smithi123 ceph-mon[139369]: pgmap v107: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:32:21.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:20 smithi123 ceph-mon[139369]: from='mgr.14696 172.21.15.123:0/1848587721' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:32:21.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:20 smithi123 ceph-mon[139369]: mon.smithi123 calling monitor election 2024-03-30T04:32:21.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:20 smithi123 ceph-mon[139369]: mon.smithi123 is new leader, mons smithi123,smithi135 in quorum (ranks 0,1) 2024-03-30T04:32:21.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:20 smithi123 ceph-mon[139369]: monmap e2: 2 mons at {smithi123=[v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0],smithi135=[v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-30T04:32:21.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:20 smithi123 ceph-mon[139369]: fsmap cephfs:2 {0=cephfs.smithi123.dvtind=up:active,1=cephfs.smithi135.zvnlap=up:active} 2 up:standby-replay 2024-03-30T04:32:21.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:20 smithi123 ceph-mon[139369]: osdmap e48: 6 total, 6 up, 6 in 2024-03-30T04:32:21.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:20 smithi123 ceph-mon[139369]: mgrmap e32: smithi123.xgksqq(active, since 3m), standbys: smithi135.kyblli 2024-03-30T04:32:21.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:20 smithi123 ceph-mon[139369]: mgrmap e33: smithi123.xgksqq(active, since 3m), standbys: smithi135.kyblli 2024-03-30T04:32:21.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:20 smithi123 ceph-mon[139369]: overall HEALTH_OK 2024-03-30T04:32:21.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:20 smithi123 ceph-mon[139369]: from='mgr.14696 ' entity='' 2024-03-30T04:32:30.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:30 smithi135 ceph-mon[44364]: Active manager daemon smithi123.xgksqq restarted 2024-03-30T04:32:30.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:30 smithi135 ceph-mon[44364]: Activating manager daemon smithi123.xgksqq 2024-03-30T04:32:30.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:30 smithi135 ceph-mon[44364]: osdmap e49: 6 total, 6 up, 6 in 2024-03-30T04:32:30.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:30 smithi135 ceph-mon[44364]: mgrmap e34: smithi123.xgksqq(active, starting, since 0.00717385s), standbys: smithi135.kyblli 2024-03-30T04:32:30.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:30 smithi135 ceph-mon[44364]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mon metadata", "id": "smithi123"}]: dispatch 2024-03-30T04:32:30.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:30 smithi135 ceph-mon[44364]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-03-30T04:32:30.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:30 smithi135 ceph-mon[44364]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi123.dvtind"}]: dispatch 2024-03-30T04:32:30.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:30 smithi135 ceph-mon[44364]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi123.tqxvzr"}]: dispatch 2024-03-30T04:32:30.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:30 smithi135 ceph-mon[44364]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi135.zvnlap"}]: dispatch 2024-03-30T04:32:30.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:30 smithi135 ceph-mon[44364]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi135.xwgjvb"}]: dispatch 2024-03-30T04:32:30.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:30 smithi135 ceph-mon[44364]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mgr metadata", "who": "smithi123.xgksqq", "id": "smithi123.xgksqq"}]: dispatch 2024-03-30T04:32:30.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:30 smithi135 ceph-mon[44364]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mgr metadata", "who": "smithi135.kyblli", "id": "smithi135.kyblli"}]: dispatch 2024-03-30T04:32:30.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:30 smithi135 ceph-mon[44364]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T04:32:30.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:30 smithi135 ceph-mon[44364]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T04:32:30.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:30 smithi135 ceph-mon[44364]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T04:32:30.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:30 smithi135 ceph-mon[44364]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T04:32:30.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:30 smithi135 ceph-mon[44364]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T04:32:30.603 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:30 smithi135 ceph-mon[44364]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T04:32:30.603 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:30 smithi135 ceph-mon[44364]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-30T04:32:30.603 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:30 smithi135 ceph-mon[44364]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-30T04:32:30.603 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:30 smithi135 ceph-mon[44364]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-30T04:32:30.603 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:30 smithi135 ceph-mon[44364]: Standby manager daemon smithi135.kyblli restarted 2024-03-30T04:32:30.603 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:30 smithi135 ceph-mon[44364]: Standby manager daemon smithi135.kyblli started 2024-03-30T04:32:30.603 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:30 smithi135 ceph-mon[44364]: from='mgr.? 172.21.15.135:0/1991958793' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi135.kyblli/crt"}]: dispatch 2024-03-30T04:32:30.603 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:30 smithi135 ceph-mon[44364]: from='mgr.? 172.21.15.135:0/1991958793' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T04:32:30.603 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:30 smithi135 ceph-mon[44364]: from='mgr.? 172.21.15.135:0/1991958793' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi135.kyblli/key"}]: dispatch 2024-03-30T04:32:30.603 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:30 smithi135 ceph-mon[44364]: from='mgr.? 172.21.15.135:0/1991958793' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T04:32:30.603 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:30 smithi135 ceph-mon[44364]: Manager daemon smithi123.xgksqq is now available 2024-03-30T04:32:30.603 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:30 smithi135 ceph-mon[44364]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:32:30.603 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:30 smithi135 ceph-mon[44364]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:32:30.604 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:30 smithi135 ceph-mon[44364]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi123.xgksqq/mirror_snapshot_schedule"}]: dispatch 2024-03-30T04:32:30.604 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:30 smithi135 ceph-mon[44364]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi123.xgksqq/trash_purge_schedule"}]: dispatch 2024-03-30T04:32:30.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:30 smithi123 ceph-mon[139369]: Active manager daemon smithi123.xgksqq restarted 2024-03-30T04:32:30.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:30 smithi123 ceph-mon[139369]: Activating manager daemon smithi123.xgksqq 2024-03-30T04:32:30.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:30 smithi123 ceph-mon[139369]: osdmap e49: 6 total, 6 up, 6 in 2024-03-30T04:32:30.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:30 smithi123 ceph-mon[139369]: mgrmap e34: smithi123.xgksqq(active, starting, since 0.00717385s), standbys: smithi135.kyblli 2024-03-30T04:32:30.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mon metadata", "id": "smithi123"}]: dispatch 2024-03-30T04:32:30.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-03-30T04:32:30.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi123.dvtind"}]: dispatch 2024-03-30T04:32:30.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi123.tqxvzr"}]: dispatch 2024-03-30T04:32:30.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi135.zvnlap"}]: dispatch 2024-03-30T04:32:30.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi135.xwgjvb"}]: dispatch 2024-03-30T04:32:30.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mgr metadata", "who": "smithi123.xgksqq", "id": "smithi123.xgksqq"}]: dispatch 2024-03-30T04:32:30.718 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mgr metadata", "who": "smithi135.kyblli", "id": "smithi135.kyblli"}]: dispatch 2024-03-30T04:32:30.718 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T04:32:30.718 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T04:32:30.718 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T04:32:30.718 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T04:32:30.718 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T04:32:30.718 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T04:32:30.718 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-30T04:32:30.718 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-30T04:32:30.718 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-30T04:32:30.718 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:30 smithi123 ceph-mon[139369]: Standby manager daemon smithi135.kyblli restarted 2024-03-30T04:32:30.718 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:30 smithi123 ceph-mon[139369]: Standby manager daemon smithi135.kyblli started 2024-03-30T04:32:30.718 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:30 smithi123 ceph-mon[139369]: from='mgr.? 172.21.15.135:0/1991958793' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi135.kyblli/crt"}]: dispatch 2024-03-30T04:32:30.719 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:30 smithi123 ceph-mon[139369]: from='mgr.? 172.21.15.135:0/1991958793' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T04:32:30.719 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:30 smithi123 ceph-mon[139369]: from='mgr.? 172.21.15.135:0/1991958793' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi135.kyblli/key"}]: dispatch 2024-03-30T04:32:30.719 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:30 smithi123 ceph-mon[139369]: from='mgr.? 172.21.15.135:0/1991958793' entity='mgr.smithi135.kyblli' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T04:32:30.719 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:30 smithi123 ceph-mon[139369]: Manager daemon smithi123.xgksqq is now available 2024-03-30T04:32:30.719 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:32:30.719 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:32:30.719 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi123.xgksqq/mirror_snapshot_schedule"}]: dispatch 2024-03-30T04:32:30.719 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi123.xgksqq/trash_purge_schedule"}]: dispatch 2024-03-30T04:32:31.621 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:31 smithi135 ceph-mon[44364]: [30/Mar/2024:04:32:30] ENGINE Bus STARTING 2024-03-30T04:32:31.621 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:31 smithi135 ceph-mon[44364]: [30/Mar/2024:04:32:30] ENGINE Serving on http://172.21.15.123:8765 2024-03-30T04:32:31.621 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:31 smithi135 ceph-mon[44364]: [30/Mar/2024:04:32:30] ENGINE Serving on https://172.21.15.123:7150 2024-03-30T04:32:31.621 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:31 smithi135 ceph-mon[44364]: [30/Mar/2024:04:32:30] ENGINE Bus STARTED 2024-03-30T04:32:31.621 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:31 smithi135 ceph-mon[44364]: mgrmap e35: smithi123.xgksqq(active, since 1.01359s), standbys: smithi135.kyblli 2024-03-30T04:32:31.621 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:31 smithi135 ceph-mon[44364]: pgmap v3: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-30T04:32:31.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:31 smithi123 ceph-mon[139369]: [30/Mar/2024:04:32:30] ENGINE Bus STARTING 2024-03-30T04:32:31.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:31 smithi123 ceph-mon[139369]: [30/Mar/2024:04:32:30] ENGINE Serving on http://172.21.15.123:8765 2024-03-30T04:32:31.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:31 smithi123 ceph-mon[139369]: [30/Mar/2024:04:32:30] ENGINE Serving on https://172.21.15.123:7150 2024-03-30T04:32:31.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:31 smithi123 ceph-mon[139369]: [30/Mar/2024:04:32:30] ENGINE Bus STARTED 2024-03-30T04:32:31.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:31 smithi123 ceph-mon[139369]: mgrmap e35: smithi123.xgksqq(active, since 1.01359s), standbys: smithi135.kyblli 2024-03-30T04:32:31.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:31 smithi123 ceph-mon[139369]: pgmap v3: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-30T04:32:32.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:32 smithi123 ceph-mon[139369]: pgmap v4: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-30T04:32:32.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:32 smithi123 ceph-mon[139369]: mgrmap e36: smithi123.xgksqq(active, since 2s), standbys: smithi135.kyblli 2024-03-30T04:32:32.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:32 smithi135 ceph-mon[44364]: pgmap v4: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-30T04:32:32.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:32 smithi135 ceph-mon[44364]: mgrmap e36: smithi123.xgksqq(active, since 2s), standbys: smithi135.kyblli 2024-03-30T04:32:34.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:34 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:34.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:34 smithi123 ceph-mon[139369]: pgmap v5: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-30T04:32:34.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:34 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:34.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:34 smithi123 ceph-mon[139369]: mgrmap e37: smithi123.xgksqq(active, since 4s), standbys: smithi135.kyblli 2024-03-30T04:32:34.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:34 smithi135 ceph-mon[44364]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:34.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:34 smithi135 ceph-mon[44364]: pgmap v5: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-30T04:32:34.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:34 smithi135 ceph-mon[44364]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:34.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:34 smithi135 ceph-mon[44364]: mgrmap e37: smithi123.xgksqq(active, since 4s), standbys: smithi135.kyblli 2024-03-30T04:32:35.520 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:35 smithi135 ceph-mon[44364]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:35.520 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:35 smithi135 ceph-mon[44364]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:35.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:35 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:35.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:35 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:37.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:36 smithi135 ceph-mon[44364]: pgmap v6: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-30T04:32:37.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:36 smithi135 ceph-mon[44364]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:37.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:36 smithi135 ceph-mon[44364]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:37.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:36 smithi123 ceph-mon[139369]: pgmap v6: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-30T04:32:37.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:36 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:37.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:36 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:38.162 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:37 smithi135 ceph-mon[44364]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:38.163 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:37 smithi135 ceph-mon[44364]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:38.163 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:37 smithi135 ceph-mon[44364]: pgmap v7: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 31 KiB/s rd, 0 B/s wr, 19 op/s 2024-03-30T04:32:38.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:37 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:38.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:37 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:38.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:37 smithi123 ceph-mon[139369]: pgmap v7: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 31 KiB/s rd, 0 B/s wr, 19 op/s 2024-03-30T04:32:40.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:39 smithi123 ceph-mon[139369]: Detected new or changed devices on smithi135 2024-03-30T04:32:40.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:39 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:40.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:39 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:40.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:39 smithi123 ceph-mon[139369]: pgmap v8: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 24 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-30T04:32:40.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:39 smithi135 ceph-mon[44364]: Detected new or changed devices on smithi135 2024-03-30T04:32:40.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:39 smithi135 ceph-mon[44364]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:40.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:39 smithi135 ceph-mon[44364]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:40.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:39 smithi135 ceph-mon[44364]: pgmap v8: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 24 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-30T04:32:42.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:42 smithi135 ceph-mon[44364]: Detected new or changed devices on smithi123 2024-03-30T04:32:42.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:42 smithi135 ceph-mon[44364]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:42.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:42 smithi135 ceph-mon[44364]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:42.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:42 smithi135 ceph-mon[44364]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:32:42.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:42 smithi135 ceph-mon[44364]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:32:42.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:42 smithi135 ceph-mon[44364]: Updating smithi123:/etc/ceph/ceph.conf 2024-03-30T04:32:42.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:42 smithi135 ceph-mon[44364]: Updating smithi135:/etc/ceph/ceph.conf 2024-03-30T04:32:42.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:42 smithi135 ceph-mon[44364]: pgmap v9: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 21 KiB/s rd, 0 B/s wr, 14 op/s 2024-03-30T04:32:42.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:42 smithi123 ceph-mon[139369]: Detected new or changed devices on smithi123 2024-03-30T04:32:42.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:42 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:42.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:42 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:42.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:42 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:32:42.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:42 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:32:42.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:42 smithi123 ceph-mon[139369]: Updating smithi123:/etc/ceph/ceph.conf 2024-03-30T04:32:42.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:42 smithi123 ceph-mon[139369]: Updating smithi135:/etc/ceph/ceph.conf 2024-03-30T04:32:42.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:42 smithi123 ceph-mon[139369]: pgmap v9: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 21 KiB/s rd, 0 B/s wr, 14 op/s 2024-03-30T04:32:43.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:43 smithi135 ceph-mon[44364]: Updating smithi135:/var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T04:32:43.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:43 smithi135 ceph-mon[44364]: Updating smithi123:/var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T04:32:43.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:43 smithi135 ceph-mon[44364]: Updating smithi135:/etc/ceph/ceph.client.admin.keyring 2024-03-30T04:32:43.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:43 smithi135 ceph-mon[44364]: Updating smithi123:/etc/ceph/ceph.client.admin.keyring 2024-03-30T04:32:43.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:43 smithi123 ceph-mon[139369]: Updating smithi135:/var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T04:32:43.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:43 smithi123 ceph-mon[139369]: Updating smithi123:/var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T04:32:43.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:43 smithi123 ceph-mon[139369]: Updating smithi135:/etc/ceph/ceph.client.admin.keyring 2024-03-30T04:32:43.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:43 smithi123 ceph-mon[139369]: Updating smithi123:/etc/ceph/ceph.client.admin.keyring 2024-03-30T04:32:44.754 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:44 smithi135 ceph-mon[44364]: Updating smithi135:/var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T04:32:44.755 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:44 smithi135 ceph-mon[44364]: Updating smithi123:/var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T04:32:44.755 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:44 smithi135 ceph-mon[44364]: pgmap v10: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-30T04:32:44.755 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:44 smithi135 ceph-mon[44364]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:44.755 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:44 smithi135 ceph-mon[44364]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:44.755 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:44 smithi135 ceph-mon[44364]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:44.755 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:44 smithi135 ceph-mon[44364]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:44.755 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:44 smithi135 ceph-mon[44364]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:44.755 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:44 smithi135 ceph-mon[44364]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:32:44.755 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:44 smithi135 ceph-mon[44364]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:32:44.755 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:44 smithi135 ceph-mon[44364]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-30T04:32:44.755 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:44 smithi135 ceph-mon[44364]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:32:44.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:44 smithi123 ceph-mon[139369]: Updating smithi135:/var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T04:32:44.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:44 smithi123 ceph-mon[139369]: Updating smithi123:/var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T04:32:44.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:44 smithi123 ceph-mon[139369]: pgmap v10: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-30T04:32:44.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:44.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:44.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:44.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:44.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:44.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:32:44.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:32:44.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-30T04:32:44.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:32:46.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:46 smithi123 ceph-mon[139369]: pgmap v11: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-30T04:32:46.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:46 smithi123 ceph-mon[139369]: Upgrade: Updating mon.smithi135 2024-03-30T04:32:46.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:46 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:46.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:46 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T04:32:46.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:46 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T04:32:46.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:46 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:32:46.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:46 smithi123 ceph-mon[139369]: Deploying daemon mon.smithi135 on smithi135 2024-03-30T04:32:46.749 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:46 smithi135 ceph-mon[44364]: pgmap v11: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-30T04:32:46.749 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:46 smithi135 ceph-mon[44364]: Upgrade: Updating mon.smithi135 2024-03-30T04:32:46.749 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:46 smithi135 ceph-mon[44364]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:46.749 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:46 smithi135 ceph-mon[44364]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T04:32:46.749 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:46 smithi135 ceph-mon[44364]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T04:32:46.749 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:46 smithi135 ceph-mon[44364]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:32:46.749 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:46 smithi135 ceph-mon[44364]: Deploying daemon mon.smithi135 on smithi135 2024-03-30T04:32:46.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:46.952+0000 7fa8c0400700 1 -- 172.21.15.123:0/1002932943 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8b8070300 msgr2=0x7fa8b80706d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:46.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:46.952+0000 7fa8c0400700 1 --2- 172.21.15.123:0/1002932943 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8b8070300 0x7fa8b80706d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac004970 tx=0x7fa8ac009e30 comp rx=0 tx=0).stop 2024-03-30T04:32:46.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:46.952+0000 7fa8c0400700 1 -- 172.21.15.123:0/1002932943 shutdown_connections 2024-03-30T04:32:46.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:46.952+0000 7fa8c0400700 1 --2- 172.21.15.123:0/1002932943 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa8b8070c10 0x7fa8b8109170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:46.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:46.952+0000 7fa8c0400700 1 --2- 172.21.15.123:0/1002932943 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8b8070300 0x7fa8b80706d0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:46.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:46.952+0000 7fa8c0400700 1 -- 172.21.15.123:0/1002932943 >> 172.21.15.123:0/1002932943 conn(0x7fa8b80f89b0 msgr2=0x7fa8b80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:32:46.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:46.952+0000 7fa8c0400700 1 -- 172.21.15.123:0/1002932943 shutdown_connections 2024-03-30T04:32:46.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:46.952+0000 7fa8c0400700 1 -- 172.21.15.123:0/1002932943 wait complete. 2024-03-30T04:32:46.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:46.954+0000 7fa8c0400700 1 Processor -- start 2024-03-30T04:32:46.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:46.954+0000 7fa8c0400700 1 -- start start 2024-03-30T04:32:46.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:46.954+0000 7fa8c0400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa8b8070300 0x7fa8b8115a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:32:46.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:46.954+0000 7fa8c0400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8b8070c10 0x7fa8b8115f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:32:46.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:46.954+0000 7fa8c0400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b81165e0 con 0x7fa8b8070c10 2024-03-30T04:32:46.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:46.954+0000 7fa8c0400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b810fbc0 con 0x7fa8b8070300 2024-03-30T04:32:46.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:46.955+0000 7fa8bd000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8b8070c10 0x7fa8b8115f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:32:46.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:46.955+0000 7fa8bd000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8b8070c10 0x7fa8b8115f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56328/0 (socket says 172.21.15.123:56328) 2024-03-30T04:32:46.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:46.955+0000 7fa8bda00700 1 --2- 172.21.15.123:0/2103405856 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa8b8070300 0x7fa8b8115a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:32:46.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:46.955+0000 7fa8bd000700 1 -- 172.21.15.123:0/2103405856 learned_addr learned my addr 172.21.15.123:0/2103405856 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:32:46.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:46.955+0000 7fa8bd000700 1 -- 172.21.15.123:0/2103405856 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa8b8070300 msgr2=0x7fa8b8115a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:46.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:46.955+0000 7fa8bd000700 1 --2- 172.21.15.123:0/2103405856 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa8b8070300 0x7fa8b8115a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:46.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:46.955+0000 7fa8bd000700 1 -- 172.21.15.123:0/2103405856 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8ac005040 con 0x7fa8b8070c10 2024-03-30T04:32:46.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:46.955+0000 7fa8bda00700 1 --2- 172.21.15.123:0/2103405856 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa8b8070300 0x7fa8b8115a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:32:46.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:46.955+0000 7fa8bd000700 1 --2- 172.21.15.123:0/2103405856 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8b8070c10 0x7fa8b8115f90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa8a4002a80 tx=0x7fa8a40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:32:46.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:46.956+0000 7fa8b5600700 1 -- 172.21.15.123:0/2103405856 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa8a4007cc0 con 0x7fa8b8070c10 2024-03-30T04:32:46.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:46.956+0000 7fa8b5600700 1 -- 172.21.15.123:0/2103405856 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa8a400f040 con 0x7fa8b8070c10 2024-03-30T04:32:46.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:46.956+0000 7fa8c0400700 1 -- 172.21.15.123:0/2103405856 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8b810fea0 con 0x7fa8b8070c10 2024-03-30T04:32:46.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:46.956+0000 7fa8b5600700 1 -- 172.21.15.123:0/2103405856 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa8a400da60 con 0x7fa8b8070c10 2024-03-30T04:32:46.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:46.956+0000 7fa8c0400700 1 -- 172.21.15.123:0/2103405856 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8b81103e0 con 0x7fa8b8070c10 2024-03-30T04:32:46.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:46.957+0000 7fa8c0400700 1 -- 172.21.15.123:0/2103405856 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8b8071a40 con 0x7fa8b8070c10 2024-03-30T04:32:46.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:46.958+0000 7fa8b5600700 1 -- 172.21.15.123:0/2103405856 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 37) v1 ==== 90946+0+0 (secure 0 0 0) 0x7fa8a400dbc0 con 0x7fa8b8070c10 2024-03-30T04:32:46.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:46.960+0000 7fa8b5600700 1 --2- 172.21.15.123:0/2103405856 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa8a006cd50 0x7fa8a006f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:32:46.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:46.960+0000 7fa8b5600700 1 -- 172.21.15.123:0/2103405856 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fa8a408bf30 con 0x7fa8b8070c10 2024-03-30T04:32:46.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:46.960+0000 7fa8bda00700 1 --2- 172.21.15.123:0/2103405856 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa8a006cd50 0x7fa8a006f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:32:46.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:46.961+0000 7fa8bda00700 1 --2- 172.21.15.123:0/2103405856 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa8a006cd50 0x7fa8a006f200 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac00b370 tx=0x7fa8ac009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:32:46.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:46.966+0000 7fa8b5600700 1 -- 172.21.15.123:0/2103405856 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa8a4059760 con 0x7fa8b8070c10 2024-03-30T04:32:47.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.187+0000 7fa8c0400700 1 -- 172.21.15.123:0/2103405856 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa8b810c020 con 0x7fa8a006cd50 2024-03-30T04:32:47.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.189+0000 7fa8b5600700 1 -- 172.21.15.123:0/2103405856 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7fa8b810c020 con 0x7fa8a006cd50 2024-03-30T04:32:47.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.191+0000 7fa8c0400700 1 -- 172.21.15.123:0/2103405856 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa8a006cd50 msgr2=0x7fa8a006f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:47.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.191+0000 7fa8c0400700 1 --2- 172.21.15.123:0/2103405856 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa8a006cd50 0x7fa8a006f200 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac00b370 tx=0x7fa8ac009e30 comp rx=0 tx=0).stop 2024-03-30T04:32:47.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.191+0000 7fa8c0400700 1 -- 172.21.15.123:0/2103405856 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8b8070c10 msgr2=0x7fa8b8115f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:47.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.191+0000 7fa8c0400700 1 --2- 172.21.15.123:0/2103405856 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8b8070c10 0x7fa8b8115f90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa8a4002a80 tx=0x7fa8a40046b0 comp rx=0 tx=0).stop 2024-03-30T04:32:47.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.192+0000 7fa8c0400700 1 -- 172.21.15.123:0/2103405856 shutdown_connections 2024-03-30T04:32:47.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.192+0000 7fa8c0400700 1 --2- 172.21.15.123:0/2103405856 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa8a006cd50 0x7fa8a006f200 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:47.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.192+0000 7fa8c0400700 1 --2- 172.21.15.123:0/2103405856 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa8b8070300 0x7fa8b8115a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:47.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.192+0000 7fa8c0400700 1 --2- 172.21.15.123:0/2103405856 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa8b8070c10 0x7fa8b8115f90 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:47.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.192+0000 7fa8c0400700 1 -- 172.21.15.123:0/2103405856 >> 172.21.15.123:0/2103405856 conn(0x7fa8b80f89b0 msgr2=0x7fa8b8105a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:32:47.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.192+0000 7fa8c0400700 1 -- 172.21.15.123:0/2103405856 shutdown_connections 2024-03-30T04:32:47.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.192+0000 7fa8c0400700 1 -- 172.21.15.123:0/2103405856 wait complete. 2024-03-30T04:32:47.204 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:32:47.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.329+0000 7f8b24c00700 1 -- 172.21.15.123:0/3766629120 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b201037a0 msgr2=0x7f8b200fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:47.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.329+0000 7f8b24c00700 1 --2- 172.21.15.123:0/3766629120 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b201037a0 0x7f8b200fce10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c004940 tx=0x7f8b0c009e30 comp rx=0 tx=0).stop 2024-03-30T04:32:47.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.329+0000 7f8b24c00700 1 -- 172.21.15.123:0/3766629120 shutdown_connections 2024-03-30T04:32:47.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.329+0000 7f8b24c00700 1 --2- 172.21.15.123:0/3766629120 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b201037a0 0x7f8b200fce10 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:47.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.329+0000 7f8b24c00700 1 --2- 172.21.15.123:0/3766629120 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8b20102e00 0x7f8b201031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:47.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.329+0000 7f8b24c00700 1 -- 172.21.15.123:0/3766629120 >> 172.21.15.123:0/3766629120 conn(0x7f8b200f89b0 msgr2=0x7f8b200fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:32:47.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.329+0000 7f8b24c00700 1 -- 172.21.15.123:0/3766629120 shutdown_connections 2024-03-30T04:32:47.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.330+0000 7f8b24c00700 1 -- 172.21.15.123:0/3766629120 wait complete. 2024-03-30T04:32:47.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.330+0000 7f8b24c00700 1 Processor -- start 2024-03-30T04:32:47.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.330+0000 7f8b24c00700 1 -- start start 2024-03-30T04:32:47.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.331+0000 7f8b24c00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8b20102e00 0x7f8b20115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:32:47.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.331+0000 7f8b24c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b201037a0 0x7f8b201161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:32:47.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.331+0000 7f8b24c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b201167f0 con 0x7f8b201037a0 2024-03-30T04:32:47.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.331+0000 7f8b24c00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b2010fbc0 con 0x7f8b20102e00 2024-03-30T04:32:47.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.331+0000 7f8b1d400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b201037a0 0x7f8b201161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:32:47.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.331+0000 7f8b1d400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b201037a0 0x7f8b201161a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56334/0 (socket says 172.21.15.123:56334) 2024-03-30T04:32:47.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.331+0000 7f8b1d400700 1 -- 172.21.15.123:0/3831912240 learned_addr learned my addr 172.21.15.123:0/3831912240 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:32:47.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.331+0000 7f8b1de00700 1 --2- 172.21.15.123:0/3831912240 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8b20102e00 0x7f8b20115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:32:47.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.331+0000 7f8b1d400700 1 -- 172.21.15.123:0/3831912240 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8b20102e00 msgr2=0x7f8b20115c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:47.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.331+0000 7f8b1d400700 1 --2- 172.21.15.123:0/3831912240 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8b20102e00 0x7f8b20115c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:47.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.331+0000 7f8b1d400700 1 -- 172.21.15.123:0/3831912240 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b0c005040 con 0x7f8b201037a0 2024-03-30T04:32:47.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.331+0000 7f8b1de00700 1 --2- 172.21.15.123:0/3831912240 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8b20102e00 0x7f8b20115c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:32:47.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.332+0000 7f8b1d400700 1 --2- 172.21.15.123:0/3831912240 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b201037a0 0x7f8b201161a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c008d50 tx=0x7f8b0c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:32:47.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.332+0000 7f8b16000700 1 -- 172.21.15.123:0/3831912240 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b0c00e030 con 0x7f8b201037a0 2024-03-30T04:32:47.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.332+0000 7f8b16000700 1 -- 172.21.15.123:0/3831912240 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8b0c013de0 con 0x7f8b201037a0 2024-03-30T04:32:47.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.332+0000 7f8b24c00700 1 -- 172.21.15.123:0/3831912240 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b2010fe40 con 0x7f8b201037a0 2024-03-30T04:32:47.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.332+0000 7f8b16000700 1 -- 172.21.15.123:0/3831912240 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b0c01cd90 con 0x7f8b201037a0 2024-03-30T04:32:47.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.332+0000 7f8b24c00700 1 -- 172.21.15.123:0/3831912240 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b20110400 con 0x7f8b201037a0 2024-03-30T04:32:47.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.333+0000 7f8b24c00700 1 -- 172.21.15.123:0/3831912240 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b200fe550 con 0x7f8b201037a0 2024-03-30T04:32:47.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.334+0000 7f8b16000700 1 -- 172.21.15.123:0/3831912240 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 37) v1 ==== 90946+0+0 (secure 0 0 0) 0x7f8b0c013930 con 0x7f8b201037a0 2024-03-30T04:32:47.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.335+0000 7f8b16000700 1 --2- 172.21.15.123:0/3831912240 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8b0406cd40 0x7f8b0406f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:32:47.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.335+0000 7f8b16000700 1 -- 172.21.15.123:0/3831912240 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f8b0c08db90 con 0x7f8b201037a0 2024-03-30T04:32:47.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.335+0000 7f8b1de00700 1 --2- 172.21.15.123:0/3831912240 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8b0406cd40 0x7f8b0406f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:32:47.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.336+0000 7f8b1de00700 1 --2- 172.21.15.123:0/3831912240 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8b0406cd40 0x7f8b0406f1f0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8b10004600 tx=0x7f8b1000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:32:47.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.341+0000 7f8b16000700 1 -- 172.21.15.123:0/3831912240 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8b0c05b440 con 0x7f8b201037a0 2024-03-30T04:32:47.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.563+0000 7f8b24c00700 1 -- 172.21.15.123:0/3831912240 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8b20109e10 con 0x7f8b0406cd40 2024-03-30T04:32:47.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.564+0000 7f8b16000700 1 -- 172.21.15.123:0/3831912240 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7f8b20109e10 con 0x7f8b0406cd40 2024-03-30T04:32:47.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.566+0000 7f8b24c00700 1 -- 172.21.15.123:0/3831912240 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8b0406cd40 msgr2=0x7f8b0406f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:47.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.566+0000 7f8b24c00700 1 --2- 172.21.15.123:0/3831912240 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8b0406cd40 0x7f8b0406f1f0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8b10004600 tx=0x7f8b1000b040 comp rx=0 tx=0).stop 2024-03-30T04:32:47.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.566+0000 7f8b24c00700 1 -- 172.21.15.123:0/3831912240 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b201037a0 msgr2=0x7f8b201161a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:47.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.566+0000 7f8b24c00700 1 --2- 172.21.15.123:0/3831912240 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b201037a0 0x7f8b201161a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c008d50 tx=0x7f8b0c009e30 comp rx=0 tx=0).stop 2024-03-30T04:32:47.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.566+0000 7f8b24c00700 1 -- 172.21.15.123:0/3831912240 shutdown_connections 2024-03-30T04:32:47.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.566+0000 7f8b24c00700 1 --2- 172.21.15.123:0/3831912240 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8b0406cd40 0x7f8b0406f1f0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:47.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.566+0000 7f8b24c00700 1 --2- 172.21.15.123:0/3831912240 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8b20102e00 0x7f8b20115c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:47.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.566+0000 7f8b24c00700 1 --2- 172.21.15.123:0/3831912240 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b201037a0 0x7f8b201161a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:47.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.566+0000 7f8b24c00700 1 -- 172.21.15.123:0/3831912240 >> 172.21.15.123:0/3831912240 conn(0x7f8b200f89b0 msgr2=0x7f8b200f99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:32:47.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.567+0000 7f8b24c00700 1 -- 172.21.15.123:0/3831912240 shutdown_connections 2024-03-30T04:32:47.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.567+0000 7f8b24c00700 1 -- 172.21.15.123:0/3831912240 wait complete. 2024-03-30T04:32:47.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.699+0000 7f665d600700 1 -- 172.21.15.123:0/3277710006 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6658102c40 msgr2=0x7f6658103010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:47.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.699+0000 7f665d600700 1 --2- 172.21.15.123:0/3277710006 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6658102c40 0x7f6658103010 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6648004940 tx=0x7f6648009e30 comp rx=0 tx=0).stop 2024-03-30T04:32:47.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.699+0000 7f665d600700 1 -- 172.21.15.123:0/3277710006 shutdown_connections 2024-03-30T04:32:47.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.699+0000 7f665d600700 1 --2- 172.21.15.123:0/3277710006 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f66581035e0 0x7f66580fcc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:47.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.699+0000 7f665d600700 1 --2- 172.21.15.123:0/3277710006 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6658102c40 0x7f6658103010 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:47.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.699+0000 7f665d600700 1 -- 172.21.15.123:0/3277710006 >> 172.21.15.123:0/3277710006 conn(0x7f66580f8830 msgr2=0x7f66580fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:32:47.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.699+0000 7f665d600700 1 -- 172.21.15.123:0/3277710006 shutdown_connections 2024-03-30T04:32:47.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.699+0000 7f665d600700 1 -- 172.21.15.123:0/3277710006 wait complete. 2024-03-30T04:32:47.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.700+0000 7f665d600700 1 Processor -- start 2024-03-30T04:32:47.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.700+0000 7f665d600700 1 -- start start 2024-03-30T04:32:47.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.701+0000 7f665d600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6658102c40 0x7f66580702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:32:47.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.701+0000 7f665d600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f66581035e0 0x7f6658070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:32:47.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.701+0000 7f665d600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6658070d90 con 0x7f6658102c40 2024-03-30T04:32:47.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.701+0000 7f665d600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6658070ed0 con 0x7f66581035e0 2024-03-30T04:32:47.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.701+0000 7f6657e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6658102c40 0x7f66580702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:32:47.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.701+0000 7f6657e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6658102c40 0x7f66580702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56344/0 (socket says 172.21.15.123:56344) 2024-03-30T04:32:47.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.701+0000 7f6657e00700 1 -- 172.21.15.123:0/3513339933 learned_addr learned my addr 172.21.15.123:0/3513339933 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:32:47.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.701+0000 7f6657400700 1 --2- 172.21.15.123:0/3513339933 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f66581035e0 0x7f6658070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:32:47.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.701+0000 7f6657e00700 1 -- 172.21.15.123:0/3513339933 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f66581035e0 msgr2=0x7f6658070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:47.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.701+0000 7f6657e00700 1 --2- 172.21.15.123:0/3513339933 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f66581035e0 0x7f6658070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:47.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.701+0000 7f6657e00700 1 -- 172.21.15.123:0/3513339933 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6648005040 con 0x7f6658102c40 2024-03-30T04:32:47.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.701+0000 7f6657400700 1 --2- 172.21.15.123:0/3513339933 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f66581035e0 0x7f6658070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:32:47.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.701+0000 7f6657e00700 1 --2- 172.21.15.123:0/3513339933 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6658102c40 0x7f66580702c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6648009360 tx=0x7f6648009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:32:47.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.702+0000 7f664fe00700 1 -- 172.21.15.123:0/3513339933 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f664800e040 con 0x7f6658102c40 2024-03-30T04:32:47.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.702+0000 7f664fe00700 1 -- 172.21.15.123:0/3513339933 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6648013420 con 0x7f6658102c40 2024-03-30T04:32:47.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.702+0000 7f665d600700 1 -- 172.21.15.123:0/3513339933 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6658071120 con 0x7f6658102c40 2024-03-30T04:32:47.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.702+0000 7f664fe00700 1 -- 172.21.15.123:0/3513339933 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f664801c5f0 con 0x7f6658102c40 2024-03-30T04:32:47.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.702+0000 7f665d600700 1 -- 172.21.15.123:0/3513339933 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f665811a080 con 0x7f6658102c40 2024-03-30T04:32:47.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.703+0000 7f665d600700 1 -- 172.21.15.123:0/3513339933 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66580fe390 con 0x7f6658102c40 2024-03-30T04:32:47.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.704+0000 7f664fe00700 1 -- 172.21.15.123:0/3513339933 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 37) v1 ==== 90946+0+0 (secure 0 0 0) 0x7f6648013590 con 0x7f6658102c40 2024-03-30T04:32:47.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.704+0000 7f664fe00700 1 --2- 172.21.15.123:0/3513339933 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f663c06cd40 0x7f663c06f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:32:47.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.705+0000 7f664fe00700 1 -- 172.21.15.123:0/3513339933 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f664808d550 con 0x7f6658102c40 2024-03-30T04:32:47.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.705+0000 7f6657400700 1 --2- 172.21.15.123:0/3513339933 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f663c06cd40 0x7f663c06f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:32:47.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.705+0000 7f6657400700 1 --2- 172.21.15.123:0/3513339933 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f663c06cd40 0x7f663c06f1f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6644009fd0 tx=0x7f664400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:32:47.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.709+0000 7f664fe00700 1 -- 172.21.15.123:0/3513339933 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6648057370 con 0x7f6658102c40 2024-03-30T04:32:47.745 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:47 smithi135 systemd[1]: Stopping Ceph mon.smithi135 for 2e77a2be-ee4c-11ee-b647-cb9ed24678a4... 2024-03-30T04:32:47.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.930+0000 7f665d600700 1 -- 172.21.15.123:0/3513339933 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f66580719b0 con 0x7f663c06cd40 2024-03-30T04:32:47.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.940+0000 7f664fe00700 1 -- 172.21.15.123:0/3513339933 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f66580719b0 con 0x7f663c06cd40 2024-03-30T04:32:47.938 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:32:47.938 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (117s) 12s ago 14m 14.8M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:32:47.938 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (14m) 12s ago 14m 6651k - 17.2.7-797-g5c024335 f77085f6aab7 bceb8b8c4837 2024-03-30T04:32:47.938 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (13m) 14s ago 13m 6651k - 17.2.7-797-g5c024335 f77085f6aab7 cd0fe67b2d34 2024-03-30T04:32:47.938 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (97s) 12s ago 14m 75.0M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T04:32:47.938 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (8m) 12s ago 8m 109M - 17.2.7-797-g5c024335 f77085f6aab7 2df066fba7a6 2024-03-30T04:32:47.938 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (8m) 12s ago 8m 57.8M - 17.2.7-797-g5c024335 f77085f6aab7 1293b8520db1 2024-03-30T04:32:47.938 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (8m) 14s ago 8m 67.4M - 17.2.7-797-g5c024335 f77085f6aab7 0dc41326818c 2024-03-30T04:32:47.938 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (8m) 14s ago 8m 119M - 17.2.7-797-g5c024335 f77085f6aab7 aef8a0cfc553 2024-03-30T04:32:47.938 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (4m) 12s ago 16m 469M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:32:47.939 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (3m) 14s ago 13m 446M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:32:47.939 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (29s) 12s ago 16m 27.5M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T04:32:47.939 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (13m) 14s ago 13m 46.0M 2048M 17.2.7-797-g5c024335 f77085f6aab7 f81052b09dd8 2024-03-30T04:32:47.939 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (2m) 12s ago 14m 7688k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:32:47.939 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (2m) 14s ago 13m 7616k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:32:47.939 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (12m) 12s ago 12m 221M 1618M 17.2.7-797-g5c024335 f77085f6aab7 57a7ba3fd75e 2024-03-30T04:32:47.939 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (11m) 12s ago 11m 235M 1618M 17.2.7-797-g5c024335 f77085f6aab7 e1a6c1818a0c 2024-03-30T04:32:47.939 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (11m) 12s ago 11m 188M 1618M 17.2.7-797-g5c024335 f77085f6aab7 4c6a727567bd 2024-03-30T04:32:47.939 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (10m) 14s ago 10m 257M 2642M 17.2.7-797-g5c024335 f77085f6aab7 6fbdd1d46a9d 2024-03-30T04:32:47.939 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (10m) 14s ago 10m 226M 2642M 17.2.7-797-g5c024335 f77085f6aab7 b74988e947e5 2024-03-30T04:32:47.939 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (9m) 14s ago 9m 219M 2642M 17.2.7-797-g5c024335 f77085f6aab7 1fd3185f5484 2024-03-30T04:32:47.939 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (2m) 12s ago 14m 47.8M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:32:47.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.942+0000 7f665d600700 1 -- 172.21.15.123:0/3513339933 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f663c06cd40 msgr2=0x7f663c06f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:47.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.942+0000 7f665d600700 1 --2- 172.21.15.123:0/3513339933 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f663c06cd40 0x7f663c06f1f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6644009fd0 tx=0x7f664400b040 comp rx=0 tx=0).stop 2024-03-30T04:32:47.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.942+0000 7f665d600700 1 -- 172.21.15.123:0/3513339933 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6658102c40 msgr2=0x7f66580702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:47.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.942+0000 7f665d600700 1 --2- 172.21.15.123:0/3513339933 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6658102c40 0x7f66580702c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6648009360 tx=0x7f6648009e30 comp rx=0 tx=0).stop 2024-03-30T04:32:47.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.943+0000 7f665d600700 1 -- 172.21.15.123:0/3513339933 shutdown_connections 2024-03-30T04:32:47.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.943+0000 7f665d600700 1 --2- 172.21.15.123:0/3513339933 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f663c06cd40 0x7f663c06f1f0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:47.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.943+0000 7f665d600700 1 --2- 172.21.15.123:0/3513339933 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6658102c40 0x7f66580702c0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:47.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.943+0000 7f665d600700 1 --2- 172.21.15.123:0/3513339933 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f66581035e0 0x7f6658070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:47.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.943+0000 7f665d600700 1 -- 172.21.15.123:0/3513339933 >> 172.21.15.123:0/3513339933 conn(0x7f66580f8830 msgr2=0x7f66580f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:32:47.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.943+0000 7f665d600700 1 -- 172.21.15.123:0/3513339933 shutdown_connections 2024-03-30T04:32:47.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:47.943+0000 7f665d600700 1 -- 172.21.15.123:0/3513339933 wait complete. 2024-03-30T04:32:48.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.079+0000 7f6eaa200700 1 -- 172.21.15.123:0/1156516538 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6ea4102c40 msgr2=0x7f6ea4103010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:48.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.079+0000 7f6eaa200700 1 --2- 172.21.15.123:0/1156516538 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6ea4102c40 0x7f6ea4103010 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6e98004940 tx=0x7f6e98009e30 comp rx=0 tx=0).stop 2024-03-30T04:32:48.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.079+0000 7f6eaa200700 1 -- 172.21.15.123:0/1156516538 shutdown_connections 2024-03-30T04:32:48.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.079+0000 7f6eaa200700 1 --2- 172.21.15.123:0/1156516538 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6ea41035e0 0x7f6ea40fcc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:48.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.079+0000 7f6eaa200700 1 --2- 172.21.15.123:0/1156516538 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6ea4102c40 0x7f6ea4103010 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:48.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.079+0000 7f6eaa200700 1 -- 172.21.15.123:0/1156516538 >> 172.21.15.123:0/1156516538 conn(0x7f6ea40f8830 msgr2=0x7f6ea40fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:32:48.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.079+0000 7f6eaa200700 1 -- 172.21.15.123:0/1156516538 shutdown_connections 2024-03-30T04:32:48.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.079+0000 7f6eaa200700 1 -- 172.21.15.123:0/1156516538 wait complete. 2024-03-30T04:32:48.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.080+0000 7f6eaa200700 1 Processor -- start 2024-03-30T04:32:48.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.080+0000 7f6eaa200700 1 -- start start 2024-03-30T04:32:48.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.080+0000 7f6eaa200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6ea4102c40 0x7f6ea4113920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:32:48.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.080+0000 7f6eaa200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6ea41035e0 0x7f6ea4113e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:32:48.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.080+0000 7f6eaa200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea4114540 con 0x7f6ea4102c40 2024-03-30T04:32:48.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.080+0000 7f6eaa200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea410d8a0 con 0x7f6ea41035e0 2024-03-30T04:32:48.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.080+0000 7f6ea8e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6ea4102c40 0x7f6ea4113920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:32:48.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.080+0000 7f6ea3e00700 1 -- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6ea41035e0 msgr2=0x7f6ea4113e60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.135:3300/0 2024-03-30T04:32:48.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.080+0000 7f6ea8e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6ea4102c40 0x7f6ea4113920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56430/0 (socket says 172.21.15.123:56430) 2024-03-30T04:32:48.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.080+0000 7f6ea8e00700 1 -- 172.21.15.123:0/2212202651 learned_addr learned my addr 172.21.15.123:0/2212202651 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:32:48.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.080+0000 7f6ea3e00700 1 --2- 172.21.15.123:0/2212202651 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6ea41035e0 0x7f6ea4113e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-30T04:32:48.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.081+0000 7f6ea8e00700 1 -- 172.21.15.123:0/2212202651 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6ea41035e0 msgr2=0x7f6ea4113e60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-30T04:32:48.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.081+0000 7f6ea8e00700 1 --2- 172.21.15.123:0/2212202651 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6ea41035e0 0x7f6ea4113e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:48.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.081+0000 7f6ea8e00700 1 -- 172.21.15.123:0/2212202651 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e98005040 con 0x7f6ea4102c40 2024-03-30T04:32:48.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.081+0000 7f6ea8e00700 1 --2- 172.21.15.123:0/2212202651 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6ea4102c40 0x7f6ea4113920 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6e98004090 tx=0x7f6e98009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:32:48.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.081+0000 7f6ea0c00700 1 -- 172.21.15.123:0/2212202651 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e9800e070 con 0x7f6ea4102c40 2024-03-30T04:32:48.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.081+0000 7f6ea0c00700 1 -- 172.21.15.123:0/2212202651 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6e98004290 con 0x7f6ea4102c40 2024-03-30T04:32:48.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.081+0000 7f6eaa200700 1 -- 172.21.15.123:0/2212202651 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ea410db20 con 0x7f6ea4102c40 2024-03-30T04:32:48.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.081+0000 7f6ea0c00700 1 -- 172.21.15.123:0/2212202651 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e98013cb0 con 0x7f6ea4102c40 2024-03-30T04:32:48.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.082+0000 7f6eaa200700 1 -- 172.21.15.123:0/2212202651 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ea410e080 con 0x7f6ea4102c40 2024-03-30T04:32:48.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.083+0000 7f6eaa200700 1 -- 172.21.15.123:0/2212202651 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ea40fe390 con 0x7f6ea4102c40 2024-03-30T04:32:48.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.083+0000 7f6ea0c00700 1 -- 172.21.15.123:0/2212202651 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 37) v1 ==== 90946+0+0 (secure 0 0 0) 0x7f6e9800ba80 con 0x7f6ea4102c40 2024-03-30T04:32:48.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.085+0000 7f6ea0c00700 1 --2- 172.21.15.123:0/2212202651 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6e8c06cce0 0x7f6e8c06f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:32:48.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.085+0000 7f6ea0c00700 1 -- 172.21.15.123:0/2212202651 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f6e9808da30 con 0x7f6ea4102c40 2024-03-30T04:32:48.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.085+0000 7f6ea3e00700 1 --2- 172.21.15.123:0/2212202651 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6e8c06cce0 0x7f6e8c06f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:32:48.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.086+0000 7f6ea3e00700 1 --2- 172.21.15.123:0/2212202651 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6e8c06cce0 0x7f6e8c06f190 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6e90006fd0 tx=0x7f6e90006df0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:32:48.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.090+0000 7f6ea0c00700 1 -- 172.21.15.123:0/2212202651 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6e9805b2e0 con 0x7f6ea4102c40 2024-03-30T04:32:48.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:47 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-mon-smithi135[44340]: 2024-03-30T04:32:47.746+0000 7f2a42e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi135 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T04:32:48.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:47 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-mon-smithi135[44340]: 2024-03-30T04:32:47.746+0000 7f2a42e00700 -1 mon.smithi135@1(peon) e2 *** Got Signal Terminated *** 2024-03-30T04:32:48.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.375+0000 7f6eaa200700 1 -- 172.21.15.123:0/2212202651 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f6ea4109d00 con 0x7f6ea4102c40 2024-03-30T04:32:48.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.376+0000 7f6ea0c00700 1 -- 172.21.15.123:0/2212202651 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+849 (secure 0 0 0) 0x7f6e9805ae70 con 0x7f6ea4102c40 2024-03-30T04:32:48.374 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:32:48.374 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:32:48.374 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:32:48.374 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 1 2024-03-30T04:32:48.374 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:32:48.374 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:32:48.374 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:32:48.374 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:32:48.374 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:32:48.374 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 6 2024-03-30T04:32:48.374 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:32:48.375 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:32:48.375 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 4 2024-03-30T04:32:48.375 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:32:48.375 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:32:48.375 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 11, 2024-03-30T04:32:48.375 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 3 2024-03-30T04:32:48.375 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:32:48.375 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:32:48.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.378+0000 7f6eaa200700 1 -- 172.21.15.123:0/2212202651 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6e8c06cce0 msgr2=0x7f6e8c06f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:48.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.378+0000 7f6eaa200700 1 --2- 172.21.15.123:0/2212202651 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6e8c06cce0 0x7f6e8c06f190 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6e90006fd0 tx=0x7f6e90006df0 comp rx=0 tx=0).stop 2024-03-30T04:32:48.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.378+0000 7f6eaa200700 1 -- 172.21.15.123:0/2212202651 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6ea4102c40 msgr2=0x7f6ea4113920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:48.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.378+0000 7f6eaa200700 1 --2- 172.21.15.123:0/2212202651 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6ea4102c40 0x7f6ea4113920 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6e98004090 tx=0x7f6e98009e30 comp rx=0 tx=0).stop 2024-03-30T04:32:48.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.378+0000 7f6eaa200700 1 -- 172.21.15.123:0/2212202651 shutdown_connections 2024-03-30T04:32:48.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.378+0000 7f6eaa200700 1 --2- 172.21.15.123:0/2212202651 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6e8c06cce0 0x7f6e8c06f190 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:48.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.378+0000 7f6eaa200700 1 --2- 172.21.15.123:0/2212202651 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6ea4102c40 0x7f6ea4113920 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:48.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.378+0000 7f6eaa200700 1 --2- 172.21.15.123:0/2212202651 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6ea41035e0 0x7f6ea4113e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:48.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.378+0000 7f6eaa200700 1 -- 172.21.15.123:0/2212202651 >> 172.21.15.123:0/2212202651 conn(0x7f6ea40f8830 msgr2=0x7f6ea40f9850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:32:48.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.378+0000 7f6eaa200700 1 -- 172.21.15.123:0/2212202651 shutdown_connections 2024-03-30T04:32:48.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.379+0000 7f6eaa200700 1 -- 172.21.15.123:0/2212202651 wait complete. 2024-03-30T04:32:48.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.512+0000 7f8500800700 1 -- 172.21.15.123:0/1654967541 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84f81037a0 msgr2=0x7f84f80fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:48.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.512+0000 7f8500800700 1 --2- 172.21.15.123:0/1654967541 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84f81037a0 0x7f84f80fce10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f84e8004940 tx=0x7f84e8009e30 comp rx=0 tx=0).stop 2024-03-30T04:32:48.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.512+0000 7f8500800700 1 -- 172.21.15.123:0/1654967541 shutdown_connections 2024-03-30T04:32:48.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.512+0000 7f8500800700 1 --2- 172.21.15.123:0/1654967541 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84f81037a0 0x7f84f80fce10 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:48.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.512+0000 7f8500800700 1 --2- 172.21.15.123:0/1654967541 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f84f8102e00 0x7f84f81031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:48.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.512+0000 7f8500800700 1 -- 172.21.15.123:0/1654967541 >> 172.21.15.123:0/1654967541 conn(0x7f84f80f89b0 msgr2=0x7f84f80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:32:48.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.512+0000 7f8500800700 1 -- 172.21.15.123:0/1654967541 shutdown_connections 2024-03-30T04:32:48.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.512+0000 7f8500800700 1 -- 172.21.15.123:0/1654967541 wait complete. 2024-03-30T04:32:48.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.513+0000 7f8500800700 1 Processor -- start 2024-03-30T04:32:48.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.513+0000 7f8500800700 1 -- start start 2024-03-30T04:32:48.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.514+0000 7f8500800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f84f8102e00 0x7f84f8115d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:32:48.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.514+0000 7f8500800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84f81037a0 0x7f84f8116280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:32:48.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.514+0000 7f8500800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84f8116960 con 0x7f84f81037a0 2024-03-30T04:32:48.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.514+0000 7f8500800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84f810fca0 con 0x7f84f8102e00 2024-03-30T04:32:48.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.514+0000 7f84fde00700 1 -- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f84f8102e00 msgr2=0x7f84f8115d40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.135:3300/0 2024-03-30T04:32:48.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.514+0000 7f84fd400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84f81037a0 0x7f84f8116280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:32:48.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.514+0000 7f84fde00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f84f8102e00 0x7f84f8115d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-30T04:32:48.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.514+0000 7f84fd400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84f81037a0 0x7f84f8116280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56446/0 (socket says 172.21.15.123:56446) 2024-03-30T04:32:48.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.514+0000 7f84fd400700 1 -- 172.21.15.123:0/1134765192 learned_addr learned my addr 172.21.15.123:0/1134765192 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:32:48.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.515+0000 7f84fd400700 1 -- 172.21.15.123:0/1134765192 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f84f8102e00 msgr2=0x7f84f8115d40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-30T04:32:48.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.515+0000 7f84fd400700 1 --2- 172.21.15.123:0/1134765192 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f84f8102e00 0x7f84f8115d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:48.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.515+0000 7f84fd400700 1 -- 172.21.15.123:0/1134765192 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e8005040 con 0x7f84f81037a0 2024-03-30T04:32:48.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.515+0000 7f84fd400700 1 --2- 172.21.15.123:0/1134765192 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84f81037a0 0x7f84f8116280 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f84e8000c00 tx=0x7f84e8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:32:48.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.515+0000 7f84f2000700 1 -- 172.21.15.123:0/1134765192 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f84e800e030 con 0x7f84f81037a0 2024-03-30T04:32:48.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.515+0000 7f84f2000700 1 -- 172.21.15.123:0/1134765192 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f84e80093b0 con 0x7f84f81037a0 2024-03-30T04:32:48.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.515+0000 7f8500800700 1 -- 172.21.15.123:0/1134765192 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84f810fec0 con 0x7f84f81037a0 2024-03-30T04:32:48.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.515+0000 7f84f2000700 1 -- 172.21.15.123:0/1134765192 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f84e8013c80 con 0x7f84f81037a0 2024-03-30T04:32:48.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.515+0000 7f8500800700 1 -- 172.21.15.123:0/1134765192 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84f8110480 con 0x7f84f81037a0 2024-03-30T04:32:48.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.516+0000 7f8500800700 1 -- 172.21.15.123:0/1134765192 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84f80fe550 con 0x7f84f81037a0 2024-03-30T04:32:48.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.517+0000 7f84f2000700 1 -- 172.21.15.123:0/1134765192 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 37) v1 ==== 90946+0+0 (secure 0 0 0) 0x7f84e8013420 con 0x7f84f81037a0 2024-03-30T04:32:48.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.518+0000 7f84f2000700 1 --2- 172.21.15.123:0/1134765192 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f84e006cc90 0x7f84e006f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:32:48.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.518+0000 7f84f2000700 1 -- 172.21.15.123:0/1134765192 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f84e808dc10 con 0x7f84f81037a0 2024-03-30T04:32:48.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.518+0000 7f84fde00700 1 --2- 172.21.15.123:0/1134765192 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f84e006cc90 0x7f84e006f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:32:48.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.519+0000 7f84fde00700 1 --2- 172.21.15.123:0/1134765192 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f84e006cc90 0x7f84e006f140 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f84ec006fd0 tx=0x7f84ec006df0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:32:48.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.523+0000 7f84f2000700 1 -- 172.21.15.123:0/1134765192 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f84e805b540 con 0x7f84f81037a0 2024-03-30T04:32:48.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:48 smithi135 bash[105823]: ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-mon-smithi135 2024-03-30T04:32:48.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.764+0000 7f8500800700 1 -- 172.21.15.123:0/1134765192 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f84f8109e60 con 0x7f84f81037a0 2024-03-30T04:32:48.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.765+0000 7f84f2000700 1 -- 172.21.15.123:0/1134765192 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+1893 (secure 0 0 0) 0x7f84e805b0d0 con 0x7f84f81037a0 2024-03-30T04:32:48.762 INFO:teuthology.orchestra.run.smithi123.stdout:e22 2024-03-30T04:32:48.763 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:32:48.763 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:32:48.763 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:32:48.763 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:32:48.763 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:32:48.763 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:32:48.763 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 22 2024-03-30T04:32:48.763 INFO:teuthology.orchestra.run.smithi123.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-30T04:32:48.763 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:32:48.763 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:31:23.532125+0000 2024-03-30T04:32:48.763 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:32:48.763 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:32:48.763 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:32:48.764 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:32:48.764 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:32:48.764 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T04:32:48.764 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:32:48.764 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:32:48.764 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 46 2024-03-30T04:32:48.764 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:32:48.764 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T04:32:48.764 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T04:32:48.764 INFO:teuthology.orchestra.run.smithi123.stdout:up {0=14488,1=24309} 2024-03-30T04:32:48.764 INFO:teuthology.orchestra.run.smithi123.stdout:failed 2024-03-30T04:32:48.764 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:32:48.764 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:32:48.764 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:32:48.764 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:32:48.765 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:32:48.765 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:32:48.765 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T04:32:48.765 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T04:32:48.765 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{0:14488} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.123:6828/3037849622,v1:172.21.15.123:6829/3037849622] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:32:48.765 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{0:14500} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/2433675995,v1:172.21.15.123:6827/2433675995] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:32:48.765 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{1:24309} state up:active seq 4 export targets 0 join_fscid=1 addr [v2:172.21.15.135:6824/4050541062,v1:172.21.15.135:6825/4050541062] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:32:48.765 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{1:24319} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.135:6826/1096986122,v1:172.21.15.135:6827/1096986122] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:32:48.765 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:32:48.765 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:32:48.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.767+0000 7f8500800700 1 -- 172.21.15.123:0/1134765192 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f84e006cc90 msgr2=0x7f84e006f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:48.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.767+0000 7f8500800700 1 --2- 172.21.15.123:0/1134765192 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f84e006cc90 0x7f84e006f140 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f84ec006fd0 tx=0x7f84ec006df0 comp rx=0 tx=0).stop 2024-03-30T04:32:48.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.767+0000 7f8500800700 1 -- 172.21.15.123:0/1134765192 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84f81037a0 msgr2=0x7f84f8116280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:48.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.767+0000 7f8500800700 1 --2- 172.21.15.123:0/1134765192 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84f81037a0 0x7f84f8116280 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f84e8000c00 tx=0x7f84e8009e30 comp rx=0 tx=0).stop 2024-03-30T04:32:48.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.767+0000 7f8500800700 1 -- 172.21.15.123:0/1134765192 shutdown_connections 2024-03-30T04:32:48.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.767+0000 7f8500800700 1 --2- 172.21.15.123:0/1134765192 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f84e006cc90 0x7f84e006f140 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:48.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.767+0000 7f8500800700 1 --2- 172.21.15.123:0/1134765192 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f84f8102e00 0x7f84f8115d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:48.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.767+0000 7f8500800700 1 --2- 172.21.15.123:0/1134765192 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84f81037a0 0x7f84f8116280 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:48.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.767+0000 7f8500800700 1 -- 172.21.15.123:0/1134765192 >> 172.21.15.123:0/1134765192 conn(0x7f84f80f89b0 msgr2=0x7f84f80f9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:32:48.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.767+0000 7f8500800700 1 -- 172.21.15.123:0/1134765192 shutdown_connections 2024-03-30T04:32:48.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.767+0000 7f8500800700 1 -- 172.21.15.123:0/1134765192 wait complete. 2024-03-30T04:32:48.767 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 22 2024-03-30T04:32:48.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.899+0000 7f650a200700 1 -- 172.21.15.123:0/3397820822 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6504102e00 msgr2=0x7f65041031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:48.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.899+0000 7f650a200700 1 --2- 172.21.15.123:0/3397820822 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6504102e00 0x7f65041031d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f64f4004940 tx=0x7f64f4009e30 comp rx=0 tx=0).stop 2024-03-30T04:32:48.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.899+0000 7f650a200700 1 -- 172.21.15.123:0/3397820822 shutdown_connections 2024-03-30T04:32:48.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.899+0000 7f650a200700 1 --2- 172.21.15.123:0/3397820822 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f65041037a0 0x7f65040fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:48.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.899+0000 7f650a200700 1 --2- 172.21.15.123:0/3397820822 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6504102e00 0x7f65041031d0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:48.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.899+0000 7f650a200700 1 -- 172.21.15.123:0/3397820822 >> 172.21.15.123:0/3397820822 conn(0x7f65040f89b0 msgr2=0x7f65040fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:32:48.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.899+0000 7f650a200700 1 -- 172.21.15.123:0/3397820822 shutdown_connections 2024-03-30T04:32:48.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.899+0000 7f650a200700 1 -- 172.21.15.123:0/3397820822 wait complete. 2024-03-30T04:32:48.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.900+0000 7f650a200700 1 Processor -- start 2024-03-30T04:32:48.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.900+0000 7f650a200700 1 -- start start 2024-03-30T04:32:48.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.901+0000 7f650a200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6504102e00 0x7f650410e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:32:48.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.901+0000 7f650a200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f65041037a0 0x7f650410b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:32:48.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.901+0000 7f650a200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f650410ef70 con 0x7f6504102e00 2024-03-30T04:32:48.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.901+0000 7f650a200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f650410f0b0 con 0x7f65041037a0 2024-03-30T04:32:48.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.901+0000 7f6502a00700 1 -- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f65041037a0 msgr2=0x7f650410b870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.135:3300/0 2024-03-30T04:32:48.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.901+0000 7f6502a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f65041037a0 0x7f650410b870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-30T04:32:48.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.901+0000 7f6503400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6504102e00 0x7f650410e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:32:48.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.901+0000 7f6503400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6504102e00 0x7f650410e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56472/0 (socket says 172.21.15.123:56472) 2024-03-30T04:32:48.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.901+0000 7f6503400700 1 -- 172.21.15.123:0/1934966044 learned_addr learned my addr 172.21.15.123:0/1934966044 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:32:48.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.901+0000 7f6503400700 1 -- 172.21.15.123:0/1934966044 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f65041037a0 msgr2=0x7f650410b870 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-30T04:32:48.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.901+0000 7f6503400700 1 --2- 172.21.15.123:0/1934966044 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f65041037a0 0x7f650410b870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:48.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.901+0000 7f6503400700 1 -- 172.21.15.123:0/1934966044 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64f4005040 con 0x7f6504102e00 2024-03-30T04:32:48.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.902+0000 7f6503400700 1 --2- 172.21.15.123:0/1934966044 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6504102e00 0x7f650410e890 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f64f4004090 tx=0x7f64f4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:32:48.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.902+0000 7f64fb400700 1 -- 172.21.15.123:0/1934966044 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f64f400e070 con 0x7f6504102e00 2024-03-30T04:32:48.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.902+0000 7f64fb400700 1 -- 172.21.15.123:0/1934966044 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f64f4004290 con 0x7f6504102e00 2024-03-30T04:32:48.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.902+0000 7f64fb400700 1 -- 172.21.15.123:0/1934966044 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f64f4013cb0 con 0x7f6504102e00 2024-03-30T04:32:48.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.902+0000 7f650a200700 1 -- 172.21.15.123:0/1934966044 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f650410be10 con 0x7f6504102e00 2024-03-30T04:32:48.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.902+0000 7f650a200700 1 -- 172.21.15.123:0/1934966044 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f650410c370 con 0x7f6504102e00 2024-03-30T04:32:48.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.903+0000 7f650a200700 1 -- 172.21.15.123:0/1934966044 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65040fe550 con 0x7f6504102e00 2024-03-30T04:32:48.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.904+0000 7f64fb400700 1 -- 172.21.15.123:0/1934966044 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 37) v1 ==== 90946+0+0 (secure 0 0 0) 0x7f64f400ba80 con 0x7f6504102e00 2024-03-30T04:32:48.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.905+0000 7f64fb400700 1 --2- 172.21.15.123:0/1934966044 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f64e806cce0 0x7f64e806f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:32:48.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.905+0000 7f64fb400700 1 -- 172.21.15.123:0/1934966044 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f64f408d8b0 con 0x7f6504102e00 2024-03-30T04:32:48.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.905+0000 7f6502a00700 1 --2- 172.21.15.123:0/1934966044 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f64e806cce0 0x7f64e806f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:32:48.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.906+0000 7f6502a00700 1 --2- 172.21.15.123:0/1934966044 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f64e806cce0 0x7f64e806f190 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f64f0006fd0 tx=0x7f64f0006df0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:32:48.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:48.909+0000 7f64fb400700 1 -- 172.21.15.123:0/1934966044 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f64f4057810 con 0x7f6504102e00 2024-03-30T04:32:49.053 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:48 smithi135 systemd[1]: ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4@mon.smithi135.service: Succeeded. 2024-03-30T04:32:49.053 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:48 smithi135 systemd[1]: Stopped Ceph mon.smithi135 for 2e77a2be-ee4c-11ee-b647-cb9ed24678a4. 2024-03-30T04:32:49.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.128+0000 7f650a200700 1 -- 172.21.15.123:0/1934966044 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f650404bed0 con 0x7f64e806cce0 2024-03-30T04:32:49.127 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:32:49.127 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:32:49.127 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:32:49.127 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T04:32:49.128 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:32:49.128 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr" 2024-03-30T04:32:49.128 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:32:49.128 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "3/21 daemons upgraded", 2024-03-30T04:32:49.128 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mon daemons", 2024-03-30T04:32:49.128 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:32:49.128 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:32:49.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.130+0000 7f64fb400700 1 -- 172.21.15.123:0/1934966044 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7f650404bed0 con 0x7f64e806cce0 2024-03-30T04:32:49.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.132+0000 7f650a200700 1 -- 172.21.15.123:0/1934966044 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f64e806cce0 msgr2=0x7f64e806f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:49.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.132+0000 7f650a200700 1 --2- 172.21.15.123:0/1934966044 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f64e806cce0 0x7f64e806f190 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f64f0006fd0 tx=0x7f64f0006df0 comp rx=0 tx=0).stop 2024-03-30T04:32:49.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.132+0000 7f650a200700 1 -- 172.21.15.123:0/1934966044 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6504102e00 msgr2=0x7f650410e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:49.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.132+0000 7f650a200700 1 --2- 172.21.15.123:0/1934966044 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6504102e00 0x7f650410e890 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f64f4004090 tx=0x7f64f4009e30 comp rx=0 tx=0).stop 2024-03-30T04:32:49.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.132+0000 7f650a200700 1 -- 172.21.15.123:0/1934966044 shutdown_connections 2024-03-30T04:32:49.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.132+0000 7f650a200700 1 --2- 172.21.15.123:0/1934966044 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f64e806cce0 0x7f64e806f190 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:49.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.132+0000 7f650a200700 1 --2- 172.21.15.123:0/1934966044 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6504102e00 0x7f650410e890 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:49.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.132+0000 7f650a200700 1 --2- 172.21.15.123:0/1934966044 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f65041037a0 0x7f650410b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:49.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.132+0000 7f650a200700 1 -- 172.21.15.123:0/1934966044 >> 172.21.15.123:0/1934966044 conn(0x7f65040f89b0 msgr2=0x7f6504070660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:32:49.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.132+0000 7f650a200700 1 -- 172.21.15.123:0/1934966044 shutdown_connections 2024-03-30T04:32:49.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.132+0000 7f650a200700 1 -- 172.21.15.123:0/1934966044 wait complete. 2024-03-30T04:32:49.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.264+0000 7f4474c00700 1 -- 172.21.15.123:0/3979086622 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4470102710 msgr2=0x7f44700fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:49.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.264+0000 7f4474c00700 1 --2- 172.21.15.123:0/3979086622 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4470102710 0x7f44700fbd80 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f445c004940 tx=0x7f445c009e30 comp rx=0 tx=0).stop 2024-03-30T04:32:49.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.264+0000 7f4474c00700 1 -- 172.21.15.123:0/3979086622 shutdown_connections 2024-03-30T04:32:49.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.264+0000 7f4474c00700 1 --2- 172.21.15.123:0/3979086622 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4470102710 0x7f44700fbd80 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:49.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.264+0000 7f4474c00700 1 --2- 172.21.15.123:0/3979086622 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4470101d70 0x7f4470102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:49.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.264+0000 7f4474c00700 1 -- 172.21.15.123:0/3979086622 >> 172.21.15.123:0/3979086622 conn(0x7f4470072bc0 msgr2=0x7f4470072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:32:49.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.265+0000 7f4474c00700 1 -- 172.21.15.123:0/3979086622 shutdown_connections 2024-03-30T04:32:49.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.265+0000 7f4474c00700 1 -- 172.21.15.123:0/3979086622 wait complete. 2024-03-30T04:32:49.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.265+0000 7f4474c00700 1 Processor -- start 2024-03-30T04:32:49.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.265+0000 7f4474c00700 1 -- start start 2024-03-30T04:32:49.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.266+0000 7f4474c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4470101d70 0x7f4470113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:32:49.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.266+0000 7f4474c00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4470102710 0x7f4470114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:32:49.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.266+0000 7f4474c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4470114720 con 0x7f4470101d70 2024-03-30T04:32:49.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.266+0000 7f4474c00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f447010da80 con 0x7f4470102710 2024-03-30T04:32:49.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.266+0000 7f446d400700 1 -- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4470102710 msgr2=0x7f4470114040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.135:3300/0 2024-03-30T04:32:49.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.266+0000 7f446d400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4470102710 0x7f4470114040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-30T04:32:49.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.266+0000 7f446de00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4470101d70 0x7f4470113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:32:49.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.266+0000 7f446de00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4470101d70 0x7f4470113b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56494/0 (socket says 172.21.15.123:56494) 2024-03-30T04:32:49.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.266+0000 7f446de00700 1 -- 172.21.15.123:0/1641799681 learned_addr learned my addr 172.21.15.123:0/1641799681 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:32:49.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.266+0000 7f446de00700 1 -- 172.21.15.123:0/1641799681 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4470102710 msgr2=0x7f4470114040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-30T04:32:49.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.266+0000 7f446de00700 1 --2- 172.21.15.123:0/1641799681 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4470102710 0x7f4470114040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:49.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.266+0000 7f446de00700 1 -- 172.21.15.123:0/1641799681 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f445c005040 con 0x7f4470101d70 2024-03-30T04:32:49.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.267+0000 7f446de00700 1 --2- 172.21.15.123:0/1641799681 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4470101d70 0x7f4470113b00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f446000b760 tx=0x7f4460009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:32:49.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.267+0000 7f4466000700 1 -- 172.21.15.123:0/1641799681 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4460010790 con 0x7f4470101d70 2024-03-30T04:32:49.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.267+0000 7f4466000700 1 -- 172.21.15.123:0/1641799681 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4460010da0 con 0x7f4470101d70 2024-03-30T04:32:49.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.267+0000 7f4474c00700 1 -- 172.21.15.123:0/1641799681 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f447010dd60 con 0x7f4470101d70 2024-03-30T04:32:49.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.267+0000 7f4466000700 1 -- 172.21.15.123:0/1641799681 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f446000f300 con 0x7f4470101d70 2024-03-30T04:32:49.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.267+0000 7f4474c00700 1 -- 172.21.15.123:0/1641799681 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f447010e320 con 0x7f4470101d70 2024-03-30T04:32:49.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.268+0000 7f4474c00700 1 -- 172.21.15.123:0/1641799681 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44700fd4c0 con 0x7f4470101d70 2024-03-30T04:32:49.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.269+0000 7f4466000700 1 -- 172.21.15.123:0/1641799681 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 37) v1 ==== 90946+0+0 (secure 0 0 0) 0x7f44600116d0 con 0x7f4470101d70 2024-03-30T04:32:49.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.270+0000 7f4466000700 1 --2- 172.21.15.123:0/1641799681 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f445406cce0 0x7f445406f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:32:49.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.271+0000 7f4466000700 1 -- 172.21.15.123:0/1641799681 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f446008c290 con 0x7f4470101d70 2024-03-30T04:32:49.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.271+0000 7f446d400700 1 --2- 172.21.15.123:0/1641799681 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f445406cce0 0x7f445406f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:32:49.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.271+0000 7f446d400700 1 --2- 172.21.15.123:0/1641799681 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f445406cce0 0x7f445406f190 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f445c000c00 tx=0x7f445c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:32:49.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.275+0000 7f4466000700 1 -- 172.21.15.123:0/1641799681 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f44600599c0 con 0x7f4470101d70 2024-03-30T04:32:49.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 systemd[1]: Starting Ceph mon.smithi135 for 2e77a2be-ee4c-11ee-b647-cb9ed24678a4... 2024-03-30T04:32:49.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.561+0000 7f4474c00700 1 -- 172.21.15.123:0/1641799681 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f447005f5f0 con 0x7f4470101d70 2024-03-30T04:32:49.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.562+0000 7f4466000700 1 -- 172.21.15.123:0/1641799681 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f44600597e0 con 0x7f4470101d70 2024-03-30T04:32:49.560 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_OK 2024-03-30T04:32:49.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.564+0000 7f4474c00700 1 -- 172.21.15.123:0/1641799681 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f445406cce0 msgr2=0x7f445406f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:49.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.564+0000 7f4474c00700 1 --2- 172.21.15.123:0/1641799681 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f445406cce0 0x7f445406f190 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f445c000c00 tx=0x7f445c009e30 comp rx=0 tx=0).stop 2024-03-30T04:32:49.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.564+0000 7f4474c00700 1 -- 172.21.15.123:0/1641799681 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4470101d70 msgr2=0x7f4470113b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:32:49.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.564+0000 7f4474c00700 1 --2- 172.21.15.123:0/1641799681 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4470101d70 0x7f4470113b00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f446000b760 tx=0x7f4460009220 comp rx=0 tx=0).stop 2024-03-30T04:32:49.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.564+0000 7f4474c00700 1 -- 172.21.15.123:0/1641799681 shutdown_connections 2024-03-30T04:32:49.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.564+0000 7f4474c00700 1 --2- 172.21.15.123:0/1641799681 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f445406cce0 0x7f445406f190 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:49.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.564+0000 7f4474c00700 1 --2- 172.21.15.123:0/1641799681 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4470101d70 0x7f4470113b00 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:49.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.564+0000 7f4474c00700 1 --2- 172.21.15.123:0/1641799681 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4470102710 0x7f4470114040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:32:49.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.564+0000 7f4474c00700 1 -- 172.21.15.123:0/1641799681 >> 172.21.15.123:0/1641799681 conn(0x7f4470072bc0 msgr2=0x7f4470105740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:32:49.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.564+0000 7f4474c00700 1 -- 172.21.15.123:0/1641799681 shutdown_connections 2024-03-30T04:32:49.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:32:49.565+0000 7f4474c00700 1 -- 172.21.15.123:0/1641799681 wait complete. 2024-03-30T04:32:50.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: set uid:gid to 167:167 (ceph:ceph) 2024-03-30T04:32:50.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable), process ceph-mon, pid 7 2024-03-30T04:32:50.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: pidfile_write: ignore empty --pid-file 2024-03-30T04:32:50.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: load: jerasure load: lrc 2024-03-30T04:32:50.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: RocksDB version: 7.9.2 2024-03-30T04:32:50.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Git sha 0 2024-03-30T04:32:50.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Compile date 2024-03-29 11:49:16 2024-03-30T04:32:50.103 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: DB SUMMARY 2024-03-30T04:32:50.103 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: DB Session ID: X98G7SCDYTX98NZQEY2F 2024-03-30T04:32:50.103 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: CURRENT file: CURRENT 2024-03-30T04:32:50.103 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: IDENTITY file: IDENTITY 2024-03-30T04:32:50.103 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1072 Bytes 2024-03-30T04:32:50.103 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi135/store.db dir, Total Num: 1, files: 000027.sst 2024-03-30T04:32:50.104 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi135/store.db: 000025.log size: 7394386 ; 2024-03-30T04:32:50.104 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.error_if_exists: 0 2024-03-30T04:32:50.104 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.create_if_missing: 0 2024-03-30T04:32:50.104 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.paranoid_checks: 1 2024-03-30T04:32:50.104 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-30T04:32:50.104 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-30T04:32:50.105 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-30T04:32:50.105 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.env: 0x5569ff67b720 2024-03-30T04:32:50.105 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.fs: PosixFileSystem 2024-03-30T04:32:50.105 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.info_log: 0x556a00e1b3c0 2024-03-30T04:32:50.105 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.max_file_opening_threads: 16 2024-03-30T04:32:50.105 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.statistics: (nil) 2024-03-30T04:32:50.105 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.use_fsync: 0 2024-03-30T04:32:50.105 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.max_log_file_size: 0 2024-03-30T04:32:50.105 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-30T04:32:50.106 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-30T04:32:50.106 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.keep_log_file_num: 1000 2024-03-30T04:32:50.106 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.recycle_log_file_num: 0 2024-03-30T04:32:50.106 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.allow_fallocate: 1 2024-03-30T04:32:50.106 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.allow_mmap_reads: 0 2024-03-30T04:32:50.106 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.allow_mmap_writes: 0 2024-03-30T04:32:50.106 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.use_direct_reads: 0 2024-03-30T04:32:50.106 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-30T04:32:50.107 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.create_missing_column_families: 0 2024-03-30T04:32:50.107 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.db_log_dir: 2024-03-30T04:32:50.107 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.wal_dir: 2024-03-30T04:32:50.109 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-30T04:32:50.109 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-30T04:32:50.109 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-30T04:32:50.109 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-30T04:32:50.110 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-30T04:32:50.110 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-30T04:32:50.110 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.advise_random_on_open: 1 2024-03-30T04:32:50.110 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.db_write_buffer_size: 0 2024-03-30T04:32:50.110 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.write_buffer_manager: 0x556a00092140 2024-03-30T04:32:50.110 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-30T04:32:50.110 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-30T04:32:50.110 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-30T04:32:50.110 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.rate_limiter: (nil) 2024-03-30T04:32:50.110 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-30T04:32:50.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.wal_recovery_mode: 2 2024-03-30T04:32:50.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.enable_thread_tracking: 0 2024-03-30T04:32:50.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.enable_pipelined_write: 0 2024-03-30T04:32:50.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.unordered_write: 0 2024-03-30T04:32:50.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-30T04:32:50.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-30T04:32:50.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-30T04:32:50.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-30T04:32:50.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.row_cache: None 2024-03-30T04:32:50.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.wal_filter: None 2024-03-30T04:32:50.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-30T04:32:50.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.allow_ingest_behind: 0 2024-03-30T04:32:50.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.two_write_queues: 0 2024-03-30T04:32:50.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.manual_wal_flush: 0 2024-03-30T04:32:50.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.wal_compression: 0 2024-03-30T04:32:50.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.atomic_flush: 0 2024-03-30T04:32:50.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-30T04:32:50.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-30T04:32:50.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-30T04:32:50.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.log_readahead_size: 0 2024-03-30T04:32:50.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-30T04:32:50.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.best_efforts_recovery: 0 2024-03-30T04:32:50.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-30T04:32:50.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-30T04:32:50.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.allow_data_in_errors: 0 2024-03-30T04:32:50.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.db_host_id: __hostname__ 2024-03-30T04:32:50.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-30T04:32:50.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.max_background_jobs: 2 2024-03-30T04:32:50.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.max_background_compactions: -1 2024-03-30T04:32:50.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.max_subcompactions: 1 2024-03-30T04:32:50.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-30T04:32:50.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-30T04:32:50.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-30T04:32:50.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.max_total_wal_size: 0 2024-03-30T04:32:50.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-30T04:32:50.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-30T04:32:50.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-30T04:32:50.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-30T04:32:50.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.max_open_files: -1 2024-03-30T04:32:50.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.bytes_per_sync: 0 2024-03-30T04:32:50.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-30T04:32:50.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-30T04:32:50.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.compaction_readahead_size: 0 2024-03-30T04:32:50.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.max_background_flushes: -1 2024-03-30T04:32:50.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Compression algorithms supported: 2024-03-30T04:32:50.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-30T04:32:50.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: kZSTD supported: 0 2024-03-30T04:32:50.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: kXpressCompression supported: 0 2024-03-30T04:32:50.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: kLZ4HCCompression supported: 1 2024-03-30T04:32:50.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: kZlibCompression supported: 1 2024-03-30T04:32:50.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: kSnappyCompression supported: 1 2024-03-30T04:32:50.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: kLZ4Compression supported: 1 2024-03-30T04:32:50.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: kBZip2Compression supported: 0 2024-03-30T04:32:50.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-30T04:32:50.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-30T04:32:50.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi135/store.db/MANIFEST-000009 2024-03-30T04:32:50.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-30T04:32:50.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-30T04:32:50.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.merge_operator: 2024-03-30T04:32:50.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.compaction_filter: None 2024-03-30T04:32:50.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.compaction_filter_factory: None 2024-03-30T04:32:50.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.sst_partitioner_factory: None 2024-03-30T04:32:50.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-30T04:32:50.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-30T04:32:50.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x556a00e1b4e0) 2024-03-30T04:32:50.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: cache_index_and_filter_blocks: 1 2024-03-30T04:32:50.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-30T04:32:50.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-30T04:32:50.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: pin_top_level_index_and_filter: 1 2024-03-30T04:32:50.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: index_type: 0 2024-03-30T04:32:50.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: data_block_index_type: 0 2024-03-30T04:32:50.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: index_shortening: 1 2024-03-30T04:32:50.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-30T04:32:50.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: checksum: 4 2024-03-30T04:32:50.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: no_block_cache: 0 2024-03-30T04:32:50.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: block_cache: 0x556a001331f0 2024-03-30T04:32:50.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: block_cache_name: BinnedLRUCache 2024-03-30T04:32:50.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: block_cache_options: 2024-03-30T04:32:50.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: capacity : 536870912 2024-03-30T04:32:50.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: num_shard_bits : 4 2024-03-30T04:32:50.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: strict_capacity_limit : 0 2024-03-30T04:32:50.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: high_pri_pool_ratio: 0.000 2024-03-30T04:32:50.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: block_cache_compressed: (nil) 2024-03-30T04:32:50.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: persistent_cache: (nil) 2024-03-30T04:32:50.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: block_size: 4096 2024-03-30T04:32:50.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: block_size_deviation: 10 2024-03-30T04:32:50.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: block_restart_interval: 16 2024-03-30T04:32:50.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: index_block_restart_interval: 1 2024-03-30T04:32:50.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: metadata_block_size: 4096 2024-03-30T04:32:50.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: partition_filters: 0 2024-03-30T04:32:50.118 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: use_delta_encoding: 1 2024-03-30T04:32:50.118 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: filter_policy: bloomfilter 2024-03-30T04:32:50.118 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: whole_key_filtering: 1 2024-03-30T04:32:50.118 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: verify_compression: 0 2024-03-30T04:32:50.118 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: read_amp_bytes_per_bit: 0 2024-03-30T04:32:50.118 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: format_version: 5 2024-03-30T04:32:50.118 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: enable_index_compression: 1 2024-03-30T04:32:50.118 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: block_align: 0 2024-03-30T04:32:50.118 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: max_auto_readahead_size: 262144 2024-03-30T04:32:50.118 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: prepopulate_block_cache: 0 2024-03-30T04:32:50.118 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: initial_auto_readahead_size: 8192 2024-03-30T04:32:50.118 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: num_file_reads_for_auto_readahead: 2 2024-03-30T04:32:50.118 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.write_buffer_size: 33554432 2024-03-30T04:32:50.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.max_write_buffer_number: 2 2024-03-30T04:32:50.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.compression: NoCompression 2024-03-30T04:32:50.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.bottommost_compression: Disabled 2024-03-30T04:32:50.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.prefix_extractor: nullptr 2024-03-30T04:32:50.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-30T04:32:50.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.num_levels: 7 2024-03-30T04:32:50.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-30T04:32:50.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-30T04:32:50.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-30T04:32:50.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-30T04:32:50.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-30T04:32:50.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-30T04:32:50.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-30T04:32:50.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-30T04:32:50.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-30T04:32:50.120 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-30T04:32:50.120 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-30T04:32:50.120 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-30T04:32:50.120 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-30T04:32:50.120 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.compression_opts.level: 32767 2024-03-30T04:32:50.120 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.compression_opts.strategy: 0 2024-03-30T04:32:50.120 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-30T04:32:50.120 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-30T04:32:50.120 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-30T04:32:50.120 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-30T04:32:50.120 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.compression_opts.enabled: false 2024-03-30T04:32:50.120 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-30T04:32:50.121 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-30T04:32:50.121 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-30T04:32:50.121 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-30T04:32:50.121 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.target_file_size_base: 67108864 2024-03-30T04:32:50.121 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-30T04:32:50.121 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-30T04:32:50.122 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-30T04:32:50.122 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-30T04:32:50.122 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-30T04:32:50.122 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-30T04:32:50.122 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-30T04:32:50.122 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-30T04:32:50.122 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-30T04:32:50.123 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-30T04:32:50.123 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-30T04:32:50.123 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-30T04:32:50.123 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-30T04:32:50.124 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-30T04:32:50.124 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.arena_block_size: 1048576 2024-03-30T04:32:50.124 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-30T04:32:50.124 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-30T04:32:50.124 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.disable_auto_compactions: 0 2024-03-30T04:32:50.124 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-30T04:32:50.124 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-30T04:32:50.124 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-30T04:32:50.125 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-30T04:32:50.125 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-30T04:32:50.125 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-30T04:32:50.125 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-30T04:32:50.125 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-30T04:32:50.125 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-30T04:32:50.126 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-30T04:32:50.126 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.table_properties_collectors: 2024-03-30T04:32:50.126 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.inplace_update_support: 0 2024-03-30T04:32:50.126 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-30T04:32:50.126 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-30T04:32:50.127 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-30T04:32:50.127 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-30T04:32:50.127 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.bloom_locality: 0 2024-03-30T04:32:50.127 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.max_successive_merges: 0 2024-03-30T04:32:50.127 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-30T04:32:50.127 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.paranoid_file_checks: 0 2024-03-30T04:32:50.127 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.force_consistency_checks: 1 2024-03-30T04:32:50.128 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.report_bg_io_stats: 0 2024-03-30T04:32:50.128 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.ttl: 2592000 2024-03-30T04:32:50.128 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-30T04:32:50.128 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-30T04:32:50.128 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-30T04:32:50.128 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.enable_blob_files: false 2024-03-30T04:32:50.128 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.min_blob_size: 0 2024-03-30T04:32:50.128 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.blob_file_size: 268435456 2024-03-30T04:32:50.129 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-30T04:32:50.129 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-30T04:32:50.129 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-30T04:32:50.129 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-30T04:32:50.129 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-30T04:32:50.129 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.blob_file_starting_level: 0 2024-03-30T04:32:50.130 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-30T04:32:50.130 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-03-30T04:32:50.130 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-30T04:32:50.130 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi135/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 12093, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-30T04:32:50.130 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-03-30T04:32:50.130 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 5341e256-8028-4727-9241-35cb1bb90899 2024-03-30T04:32:50.130 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711773169735969, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-03-30T04:32:50.131 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-03-30T04:32:50.131 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 bash[105972]: a66acfbe9e0ae714fca32d218f4decaf1000804a458e153e5963f9387de1f7f4 2024-03-30T04:32:50.131 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 systemd[1]: Started Ceph mon.smithi135 for 2e77a2be-ee4c-11ee-b647-cb9ed24678a4. 2024-03-30T04:32:50.131 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711773169763749, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 4445211, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12090, "largest_seqno": 15604, "table_properties": {"data_size": 4431844, "index_size": 8442, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 3909, "raw_key_size": 34610, "raw_average_key_size": 22, "raw_value_size": 4401227, "raw_average_value_size": 2857, "num_data_blocks": 382, "num_entries": 1540, "num_filter_entries": 1540, "num_deletions": 260, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711773169, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "5341e256-8028-4727-9241-35cb1bb90899", "db_session_id": "X98G7SCDYTX98NZQEY2F", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-03-30T04:32:50.131 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711773169763942, "job": 1, "event": "recovery_finished"} 2024-03-30T04:32:50.131 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-03-30T04:32:50.132 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-30T04:32:50.132 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi135/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-30T04:32:50.132 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x556a001d0000 2024-03-30T04:32:50.132 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: DB pointer 0x556a001be000 2024-03-30T04:32:50.132 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-30T04:32:50.132 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-30T04:32:50.132 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: ** DB Stats ** 2024-03-30T04:32:50.133 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-30T04:32:50.133 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-30T04:32:50.133 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-30T04:32:50.133 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-30T04:32:50.133 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-30T04:32:50.133 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-30T04:32:50.134 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-30T04:32:50.134 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: 2024-03-30T04:32:50.134 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: ** Compaction Stats [default] ** 2024-03-30T04:32:50.134 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-30T04:32:50.134 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-30T04:32:50.134 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: L0 1/0 4.24 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 458.2 0.01 0.00 1 0.009 0 0 0.0 0.0 2024-03-30T04:32:50.134 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: L6 1/0 6.42 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-30T04:32:50.134 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Sum 2/0 10.66 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 458.2 0.01 0.00 1 0.009 0 0 0.0 0.0 2024-03-30T04:32:50.135 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 458.2 0.01 0.00 1 0.009 0 0 0.0 0.0 2024-03-30T04:32:50.135 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: 2024-03-30T04:32:50.135 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: ** Compaction Stats [default] ** 2024-03-30T04:32:50.135 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-30T04:32:50.135 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-30T04:32:50.135 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 458.2 0.01 0.00 1 0.009 0 0 0.0 0.0 2024-03-30T04:32:50.135 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: 2024-03-30T04:32:50.135 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-30T04:32:50.135 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: 2024-03-30T04:32:50.135 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-30T04:32:50.135 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Flush(GB): cumulative 0.004, interval 0.004 2024-03-30T04:32:50.135 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-30T04:32:50.136 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-30T04:32:50.136 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-30T04:32:50.136 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-30T04:32:50.136 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Cumulative compaction: 0.00 GB write, 123.03 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-30T04:32:50.136 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Interval compaction: 0.00 GB write, 123.03 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-30T04:32:50.136 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-30T04:32:50.136 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Block cache BinnedLRUCache@0x556a001331f0#7 capacity: 512.00 MB usage: 13.11 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.6e-05 secs_since: 0 2024-03-30T04:32:50.136 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,4.03 KB,0.0007689%) IndexBlock(1,9.08 KB,0.00173151%) Misc(1,0.00 KB,0%) 2024-03-30T04:32:50.136 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: 2024-03-30T04:32:50.136 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-30T04:32:50.136 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: starting mon.smithi135 rank 1 at public addrs [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] at bind addrs [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi135 fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 2024-03-30T04:32:50.136 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: mon.smithi135@-1(???) e2 preinit fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 2024-03-30T04:32:50.136 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: mon.smithi135@-1(???).mds e22 new map 2024-03-30T04:32:50.137 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: mon.smithi135@-1(???).mds e22 print_map 2024-03-30T04:32:50.137 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: e22 2024-03-30T04:32:50.137 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:32:50.137 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:32:50.137 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: legacy client fscid: 1 2024-03-30T04:32:50.137 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: 2024-03-30T04:32:50.137 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Filesystem 'cephfs' (1) 2024-03-30T04:32:50.137 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: fs_name cephfs 2024-03-30T04:32:50.137 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: epoch 22 2024-03-30T04:32:50.137 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-30T04:32:50.137 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:32:50.137 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: modified 2024-03-30T04:31:23.532125+0000 2024-03-30T04:32:50.137 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: tableserver 0 2024-03-30T04:32:50.138 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: root 0 2024-03-30T04:32:50.138 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: session_timeout 60 2024-03-30T04:32:50.138 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: session_autoclose 300 2024-03-30T04:32:50.138 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: max_file_size 1099511627776 2024-03-30T04:32:50.138 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: max_xattr_size 65536 2024-03-30T04:32:50.138 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: required_client_features {} 2024-03-30T04:32:50.138 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: last_failure 0 2024-03-30T04:32:50.138 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: last_failure_osd_epoch 46 2024-03-30T04:32:50.138 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:32:50.138 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: max_mds 2 2024-03-30T04:32:50.138 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: in 0,1 2024-03-30T04:32:50.138 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: up {0=14488,1=24309} 2024-03-30T04:32:50.139 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: failed 2024-03-30T04:32:50.139 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: damaged 2024-03-30T04:32:50.139 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: stopped 2024-03-30T04:32:50.139 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: data_pools [3] 2024-03-30T04:32:50.139 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: metadata_pool 2 2024-03-30T04:32:50.139 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: inline_data disabled 2024-03-30T04:32:50.139 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: balancer 2024-03-30T04:32:50.139 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: bal_rank_mask -1 2024-03-30T04:32:50.139 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: standby_count_wanted 1 2024-03-30T04:32:50.139 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: [mds.cephfs.smithi123.dvtind{0:14488} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.123:6828/3037849622,v1:172.21.15.123:6829/3037849622] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:32:50.139 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: [mds.cephfs.smithi123.tqxvzr{0:14500} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/2433675995,v1:172.21.15.123:6827/2433675995] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:32:50.139 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: [mds.cephfs.smithi135.zvnlap{1:24309} state up:active seq 4 export targets 0 join_fscid=1 addr [v2:172.21.15.135:6824/4050541062,v1:172.21.15.135:6825/4050541062] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:32:50.139 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: [mds.cephfs.smithi135.xwgjvb{1:24319} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.135:6826/1096986122,v1:172.21.15.135:6827/1096986122] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:32:50.139 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: 2024-03-30T04:32:50.140 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: 2024-03-30T04:32:50.140 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: mon.smithi135@-1(???).osd e49 crush map has features 3314933000852226048, adjusting msgr requires 2024-03-30T04:32:50.140 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: mon.smithi135@-1(???).osd e49 crush map has features 288514051259236352, adjusting msgr requires 2024-03-30T04:32:50.140 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: mon.smithi135@-1(???).osd e49 crush map has features 288514051259236352, adjusting msgr requires 2024-03-30T04:32:50.140 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: mon.smithi135@-1(???).osd e49 crush map has features 288514051259236352, adjusting msgr requires 2024-03-30T04:32:50.140 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:49 smithi135 ceph-mon[106011]: mon.smithi135@-1(???).paxosservice(auth 1..23) refresh upgraded, format 0 -> 3 2024-03-30T04:32:51.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:50 smithi123 ceph-mon[139369]: overall HEALTH_OK 2024-03-30T04:32:51.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:50 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mon metadata", "id": "smithi123"}]: dispatch 2024-03-30T04:32:51.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:50 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-03-30T04:32:51.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:50 smithi123 ceph-mon[139369]: mon.smithi123 calling monitor election 2024-03-30T04:32:51.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:50 smithi123 ceph-mon[139369]: mon.smithi135 calling monitor election 2024-03-30T04:32:51.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:50 smithi123 ceph-mon[139369]: mon.smithi123 is new leader, mons smithi123,smithi135 in quorum (ranks 0,1) 2024-03-30T04:32:51.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:50 smithi123 ceph-mon[139369]: monmap e3: 2 mons at {smithi123=[v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0],smithi135=[v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-30T04:32:51.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:50 smithi123 ceph-mon[139369]: fsmap cephfs:2 {0=cephfs.smithi123.dvtind=up:active,1=cephfs.smithi135.zvnlap=up:active} 2 up:standby-replay 2024-03-30T04:32:51.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:50 smithi123 ceph-mon[139369]: osdmap e49: 6 total, 6 up, 6 in 2024-03-30T04:32:51.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:50 smithi123 ceph-mon[139369]: mgrmap e37: smithi123.xgksqq(active, since 20s), standbys: smithi135.kyblli 2024-03-30T04:32:51.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:50 smithi123 ceph-mon[139369]: overall HEALTH_OK 2024-03-30T04:32:51.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:50 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:51.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:50 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:51.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:50 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:32:51.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:50 smithi135 ceph-mon[106011]: overall HEALTH_OK 2024-03-30T04:32:51.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:50 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mon metadata", "id": "smithi123"}]: dispatch 2024-03-30T04:32:51.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:50 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-03-30T04:32:51.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:50 smithi135 ceph-mon[106011]: mon.smithi123 calling monitor election 2024-03-30T04:32:51.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:50 smithi135 ceph-mon[106011]: mon.smithi135 calling monitor election 2024-03-30T04:32:51.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:50 smithi135 ceph-mon[106011]: mon.smithi123 is new leader, mons smithi123,smithi135 in quorum (ranks 0,1) 2024-03-30T04:32:51.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:50 smithi135 ceph-mon[106011]: monmap e3: 2 mons at {smithi123=[v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0],smithi135=[v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-30T04:32:51.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:50 smithi135 ceph-mon[106011]: fsmap cephfs:2 {0=cephfs.smithi123.dvtind=up:active,1=cephfs.smithi135.zvnlap=up:active} 2 up:standby-replay 2024-03-30T04:32:51.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:50 smithi135 ceph-mon[106011]: osdmap e49: 6 total, 6 up, 6 in 2024-03-30T04:32:51.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:50 smithi135 ceph-mon[106011]: mgrmap e37: smithi123.xgksqq(active, since 20s), standbys: smithi135.kyblli 2024-03-30T04:32:51.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:50 smithi135 ceph-mon[106011]: overall HEALTH_OK 2024-03-30T04:32:51.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:50 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:51.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:50 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:51.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:50 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:32:52.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:51 smithi135 ceph-mon[106011]: pgmap v14: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:32:52.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:51 smithi123 ceph-mon[139369]: pgmap v14: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:32:55.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:54 smithi135 ceph-mon[106011]: pgmap v15: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:32:55.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:54 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:55.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:54 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:55.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:54 smithi123 ceph-mon[139369]: pgmap v15: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:32:55.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:54 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:55.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:54 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:56.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:55 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:56.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:55 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:56.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:55 smithi123 ceph-mon[139369]: pgmap v16: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:32:56.251 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:55 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:56.251 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:55 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:32:56.251 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:55 smithi135 ceph-mon[106011]: pgmap v16: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:32:58.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:32:58 smithi123 ceph-mon[139369]: pgmap v17: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:32:58.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:32:58 smithi135 ceph-mon[106011]: pgmap v17: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:33:00.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:00 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:00.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:00 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:00.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:00 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:33:00.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:00 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:33:00.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:00 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:00.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:00 smithi135 ceph-mon[106011]: pgmap v18: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:33:00.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:00 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:33:00.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:00 smithi135 ceph-mon[106011]: Reconfiguring mon.smithi123 (monmap changed)... 2024-03-30T04:33:00.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:00 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T04:33:00.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:00 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T04:33:00.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:00 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:33:00.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:00 smithi135 ceph-mon[106011]: Reconfiguring daemon mon.smithi123 on smithi123 2024-03-30T04:33:00.687 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:00 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:00.687 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:00 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:00.687 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:00 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:33:00.687 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:00 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:33:00.687 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:00 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:00.688 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:00 smithi123 ceph-mon[139369]: pgmap v18: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:33:00.688 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:00 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:33:00.688 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:00 smithi123 ceph-mon[139369]: Reconfiguring mon.smithi123 (monmap changed)... 2024-03-30T04:33:00.688 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:00 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T04:33:00.688 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:00 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T04:33:00.688 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:00 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:33:00.688 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:00 smithi123 ceph-mon[139369]: Reconfiguring daemon mon.smithi123 on smithi123 2024-03-30T04:33:02.762 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:02 smithi123 ceph-mon[139369]: pgmap v19: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:33:02.762 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:02 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:02.763 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:02 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:02.763 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:02 smithi123 ceph-mon[139369]: Reconfiguring mgr.smithi123.xgksqq (monmap changed)... 2024-03-30T04:33:02.763 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:02 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi123.xgksqq", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T04:33:02.763 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:02 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T04:33:02.763 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:02 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:33:02.763 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:02 smithi123 ceph-mon[139369]: Reconfiguring daemon mgr.smithi123.xgksqq on smithi123 2024-03-30T04:33:02.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:02 smithi135 ceph-mon[106011]: pgmap v19: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:33:02.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:02 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:02.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:02 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:02.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:02 smithi135 ceph-mon[106011]: Reconfiguring mgr.smithi123.xgksqq (monmap changed)... 2024-03-30T04:33:02.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:02 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi123.xgksqq", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T04:33:02.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:02 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T04:33:02.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:02 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:33:02.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:02 smithi135 ceph-mon[106011]: Reconfiguring daemon mgr.smithi123.xgksqq on smithi123 2024-03-30T04:33:04.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:04 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:04.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:04 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:04.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:04 smithi123 ceph-mon[139369]: Reconfiguring crash.smithi123 (monmap changed)... 2024-03-30T04:33:04.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:04 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi123", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-30T04:33:04.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:04 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:33:04.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:04 smithi123 ceph-mon[139369]: Reconfiguring daemon crash.smithi123 on smithi123 2024-03-30T04:33:04.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:04 smithi123 ceph-mon[139369]: pgmap v20: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:33:04.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:04 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:04.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:04 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:04.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:04 smithi135 ceph-mon[106011]: Reconfiguring crash.smithi123 (monmap changed)... 2024-03-30T04:33:04.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:04 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi123", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-30T04:33:04.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:04 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:33:04.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:04 smithi135 ceph-mon[106011]: Reconfiguring daemon crash.smithi123 on smithi123 2024-03-30T04:33:04.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:04 smithi135 ceph-mon[106011]: pgmap v20: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:33:06.180 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:05 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:06.180 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:05 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:06.180 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:05 smithi123 ceph-mon[139369]: Reconfiguring osd.0 (monmap changed)... 2024-03-30T04:33:06.180 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:05 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-30T04:33:06.180 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:05 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:33:06.180 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:05 smithi123 ceph-mon[139369]: Reconfiguring daemon osd.0 on smithi123 2024-03-30T04:33:06.180 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:05 smithi123 ceph-mon[139369]: pgmap v21: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:33:06.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:05 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:06.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:05 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:06.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:05 smithi135 ceph-mon[106011]: Reconfiguring osd.0 (monmap changed)... 2024-03-30T04:33:06.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:05 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-30T04:33:06.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:05 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:33:06.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:05 smithi135 ceph-mon[106011]: Reconfiguring daemon osd.0 on smithi123 2024-03-30T04:33:06.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:05 smithi135 ceph-mon[106011]: pgmap v21: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:33:07.927 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:07 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:07.928 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:07 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:07.928 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:07 smithi123 ceph-mon[139369]: Reconfiguring osd.1 (monmap changed)... 2024-03-30T04:33:07.928 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:07 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-30T04:33:07.928 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:07 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:33:07.928 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:07 smithi123 ceph-mon[139369]: Reconfiguring daemon osd.1 on smithi123 2024-03-30T04:33:08.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:07 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:08.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:07 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:08.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:07 smithi135 ceph-mon[106011]: Reconfiguring osd.1 (monmap changed)... 2024-03-30T04:33:08.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:07 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-30T04:33:08.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:07 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:33:08.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:07 smithi135 ceph-mon[106011]: Reconfiguring daemon osd.1 on smithi123 2024-03-30T04:33:08.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:08 smithi123 ceph-mon[139369]: pgmap v22: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:33:08.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:08 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:08.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:08 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:08.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:08 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-30T04:33:08.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:08 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:33:09.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:08 smithi135 ceph-mon[106011]: pgmap v22: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:33:09.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:08 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:09.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:08 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:09.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:08 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-30T04:33:09.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:08 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:33:09.945 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:09 smithi123 ceph-mon[139369]: Reconfiguring osd.2 (monmap changed)... 2024-03-30T04:33:09.945 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:09 smithi123 ceph-mon[139369]: Reconfiguring daemon osd.2 on smithi123 2024-03-30T04:33:10.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:09 smithi135 ceph-mon[106011]: Reconfiguring osd.2 (monmap changed)... 2024-03-30T04:33:10.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:09 smithi135 ceph-mon[106011]: Reconfiguring daemon osd.2 on smithi123 2024-03-30T04:33:10.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:10 smithi123 ceph-mon[139369]: pgmap v23: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:33:10.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:10 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:10.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:10 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:10.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:10 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi123.tqxvzr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T04:33:10.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:10 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:33:11.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:10 smithi135 ceph-mon[106011]: pgmap v23: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:33:11.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:10 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:11.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:10 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:11.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:10 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi123.tqxvzr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T04:33:11.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:10 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:33:11.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:11 smithi123 ceph-mon[139369]: Reconfiguring mds.cephfs.smithi123.tqxvzr (monmap changed)... 2024-03-30T04:33:11.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:11 smithi123 ceph-mon[139369]: Reconfiguring daemon mds.cephfs.smithi123.tqxvzr on smithi123 2024-03-30T04:33:12.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:11 smithi135 ceph-mon[106011]: Reconfiguring mds.cephfs.smithi123.tqxvzr (monmap changed)... 2024-03-30T04:33:12.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:11 smithi135 ceph-mon[106011]: Reconfiguring daemon mds.cephfs.smithi123.tqxvzr on smithi123 2024-03-30T04:33:13.092 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:12 smithi123 ceph-mon[139369]: pgmap v24: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:33:13.092 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:12 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:13.092 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:12 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:13.092 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:12 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi123.dvtind", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T04:33:13.092 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:12 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:33:13.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:12 smithi135 ceph-mon[106011]: pgmap v24: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:33:13.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:12 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:13.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:12 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:13.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:12 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi123.dvtind", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T04:33:13.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:12 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:33:14.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:13 smithi135 ceph-mon[106011]: Reconfiguring mds.cephfs.smithi123.dvtind (monmap changed)... 2024-03-30T04:33:14.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:13 smithi135 ceph-mon[106011]: Reconfiguring daemon mds.cephfs.smithi123.dvtind on smithi123 2024-03-30T04:33:14.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:13 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:14.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:13 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:14.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:13 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi135", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-30T04:33:14.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:13 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:33:14.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:13 smithi123 ceph-mon[139369]: Reconfiguring mds.cephfs.smithi123.dvtind (monmap changed)... 2024-03-30T04:33:14.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:13 smithi123 ceph-mon[139369]: Reconfiguring daemon mds.cephfs.smithi123.dvtind on smithi123 2024-03-30T04:33:14.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:13 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:14.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:13 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:14.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:13 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi135", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-30T04:33:14.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:13 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:33:15.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:14 smithi135 ceph-mon[106011]: pgmap v25: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:33:15.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:14 smithi135 ceph-mon[106011]: Reconfiguring crash.smithi135 (monmap changed)... 2024-03-30T04:33:15.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:14 smithi135 ceph-mon[106011]: Reconfiguring daemon crash.smithi135 on smithi135 2024-03-30T04:33:15.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:14 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:33:15.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:14 smithi123 ceph-mon[139369]: pgmap v25: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:33:15.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:14 smithi123 ceph-mon[139369]: Reconfiguring crash.smithi135 (monmap changed)... 2024-03-30T04:33:15.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:14 smithi123 ceph-mon[139369]: Reconfiguring daemon crash.smithi135 on smithi135 2024-03-30T04:33:15.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:14 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:33:16.688 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:16 smithi135 ceph-mon[106011]: pgmap v26: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:33:16.689 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:16 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:16.689 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:16 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:16.689 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:16 smithi135 ceph-mon[106011]: Reconfiguring mgr.smithi135.kyblli (monmap changed)... 2024-03-30T04:33:16.689 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:16 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi135.kyblli", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T04:33:16.689 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:16 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T04:33:16.689 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:16 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:33:16.689 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:16 smithi135 ceph-mon[106011]: Reconfiguring daemon mgr.smithi135.kyblli on smithi135 2024-03-30T04:33:16.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:16 smithi123 ceph-mon[139369]: pgmap v26: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:33:16.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:16 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:16.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:16 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:16.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:16 smithi123 ceph-mon[139369]: Reconfiguring mgr.smithi135.kyblli (monmap changed)... 2024-03-30T04:33:16.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:16 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi135.kyblli", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T04:33:16.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:16 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T04:33:16.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:16 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:33:16.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:16 smithi123 ceph-mon[139369]: Reconfiguring daemon mgr.smithi135.kyblli on smithi135 2024-03-30T04:33:18.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:18 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:18.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:18 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:18.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:18 smithi123 ceph-mon[139369]: Reconfiguring mon.smithi135 (monmap changed)... 2024-03-30T04:33:18.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:18 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T04:33:18.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:18 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T04:33:18.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:18 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:33:18.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:18 smithi123 ceph-mon[139369]: Reconfiguring daemon mon.smithi135 on smithi135 2024-03-30T04:33:18.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:18 smithi123 ceph-mon[139369]: pgmap v27: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:33:18.487 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:18 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:18.487 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:18 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:18.487 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:18 smithi135 ceph-mon[106011]: Reconfiguring mon.smithi135 (monmap changed)... 2024-03-30T04:33:18.487 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:18 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T04:33:18.487 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:18 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T04:33:18.487 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:18 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:33:18.487 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:18 smithi135 ceph-mon[106011]: Reconfiguring daemon mon.smithi135 on smithi135 2024-03-30T04:33:18.487 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:18 smithi135 ceph-mon[106011]: pgmap v27: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:33:19.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:19.714+0000 7f5197400700 1 -- 172.21.15.123:0/587923258 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51901037a0 msgr2=0x7f51900fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:19.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:19.714+0000 7f5197400700 1 --2- 172.21.15.123:0/587923258 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51901037a0 0x7f51900fce10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f517c004940 tx=0x7f517c009e30 comp rx=0 tx=0).stop 2024-03-30T04:33:19.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:19.714+0000 7f5197400700 1 -- 172.21.15.123:0/587923258 shutdown_connections 2024-03-30T04:33:19.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:19.714+0000 7f5197400700 1 --2- 172.21.15.123:0/587923258 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51901037a0 0x7f51900fce10 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:19.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:19.714+0000 7f5197400700 1 --2- 172.21.15.123:0/587923258 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5190102e00 0x7f51901031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:19.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:19.714+0000 7f5197400700 1 -- 172.21.15.123:0/587923258 >> 172.21.15.123:0/587923258 conn(0x7f51900f89b0 msgr2=0x7f51900fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:33:19.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:19.714+0000 7f5197400700 1 -- 172.21.15.123:0/587923258 shutdown_connections 2024-03-30T04:33:19.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:19.714+0000 7f5197400700 1 -- 172.21.15.123:0/587923258 wait complete. 2024-03-30T04:33:19.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:19.715+0000 7f5197400700 1 Processor -- start 2024-03-30T04:33:19.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:19.716+0000 7f5197400700 1 -- start start 2024-03-30T04:33:19.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:19.716+0000 7f5197400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5190102e00 0x7f51900702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:33:19.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:19.716+0000 7f5197400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f51901037a0 0x7f5190070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:33:19.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:19.716+0000 7f5197400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5190070d90 con 0x7f5190102e00 2024-03-30T04:33:19.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:19.716+0000 7f5197400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5190070ed0 con 0x7f51901037a0 2024-03-30T04:33:19.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:19.717+0000 7f5194a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5190102e00 0x7f51900702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:33:19.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:19.717+0000 7f5194a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5190102e00 0x7f51900702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50018/0 (socket says 172.21.15.123:50018) 2024-03-30T04:33:19.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:19.717+0000 7f5194a00700 1 -- 172.21.15.123:0/3216514738 learned_addr learned my addr 172.21.15.123:0/3216514738 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:33:19.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:19.717+0000 7f518fe00700 1 --2- 172.21.15.123:0/3216514738 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f51901037a0 0x7f5190070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:33:19.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:19.717+0000 7f5194a00700 1 -- 172.21.15.123:0/3216514738 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f51901037a0 msgr2=0x7f5190070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:19.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:19.717+0000 7f5194a00700 1 --2- 172.21.15.123:0/3216514738 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f51901037a0 0x7f5190070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:19.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:19.717+0000 7f5194a00700 1 -- 172.21.15.123:0/3216514738 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f517c005040 con 0x7f5190102e00 2024-03-30T04:33:19.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:19.717+0000 7f5194a00700 1 --2- 172.21.15.123:0/3216514738 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5190102e00 0x7f51900702c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f518400eb40 tx=0x7f518400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:33:19.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:19.717+0000 7f518cc00700 1 -- 172.21.15.123:0/3216514738 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5184007cc0 con 0x7f5190102e00 2024-03-30T04:33:19.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:19.718+0000 7f5197400700 1 -- 172.21.15.123:0/3216514738 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5190071120 con 0x7f5190102e00 2024-03-30T04:33:19.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:19.718+0000 7f518cc00700 1 -- 172.21.15.123:0/3216514738 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f518400f040 con 0x7f5190102e00 2024-03-30T04:33:19.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:19.718+0000 7f518cc00700 1 -- 172.21.15.123:0/3216514738 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f518400c930 con 0x7f5190102e00 2024-03-30T04:33:19.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:19.718+0000 7f5197400700 1 -- 172.21.15.123:0/3216514738 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f519011a1a0 con 0x7f5190102e00 2024-03-30T04:33:19.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:19.719+0000 7f5197400700 1 -- 172.21.15.123:0/3216514738 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51900fe550 con 0x7f5190102e00 2024-03-30T04:33:19.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:19.720+0000 7f518cc00700 1 -- 172.21.15.123:0/3216514738 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 37) v1 ==== 90946+0+0 (secure 0 0 0) 0x7f51840090d0 con 0x7f5190102e00 2024-03-30T04:33:19.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:19.720+0000 7f518cc00700 1 --2- 172.21.15.123:0/3216514738 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f517806cd40 0x7f517806f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:33:19.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:19.720+0000 7f518cc00700 1 -- 172.21.15.123:0/3216514738 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f518408bdd0 con 0x7f5190102e00 2024-03-30T04:33:19.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:19.721+0000 7f518fe00700 1 --2- 172.21.15.123:0/3216514738 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f517806cd40 0x7f517806f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:33:19.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:19.721+0000 7f518fe00700 1 --2- 172.21.15.123:0/3216514738 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f517806cd40 0x7f517806f1f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5190071c00 tx=0x7f517c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:33:19.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:19.726+0000 7f518cc00700 1 -- 172.21.15.123:0/3216514738 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5184055cc0 con 0x7f5190102e00 2024-03-30T04:33:19.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:19.949+0000 7f5197400700 1 -- 172.21.15.123:0/3216514738 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f51900719b0 con 0x7f517806cd40 2024-03-30T04:33:19.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:19.951+0000 7f518cc00700 1 -- 172.21.15.123:0/3216514738 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7f51900719b0 con 0x7f517806cd40 2024-03-30T04:33:19.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:19.953+0000 7f5197400700 1 -- 172.21.15.123:0/3216514738 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f517806cd40 msgr2=0x7f517806f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:19.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:19.953+0000 7f5197400700 1 --2- 172.21.15.123:0/3216514738 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f517806cd40 0x7f517806f1f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5190071c00 tx=0x7f517c009e30 comp rx=0 tx=0).stop 2024-03-30T04:33:19.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:19.953+0000 7f5197400700 1 -- 172.21.15.123:0/3216514738 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5190102e00 msgr2=0x7f51900702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:19.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:19.953+0000 7f5197400700 1 --2- 172.21.15.123:0/3216514738 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5190102e00 0x7f51900702c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f518400eb40 tx=0x7f518400b040 comp rx=0 tx=0).stop 2024-03-30T04:33:19.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:19.953+0000 7f5197400700 1 -- 172.21.15.123:0/3216514738 shutdown_connections 2024-03-30T04:33:19.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:19.953+0000 7f5197400700 1 --2- 172.21.15.123:0/3216514738 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f517806cd40 0x7f517806f1f0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:19.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:19.953+0000 7f5197400700 1 --2- 172.21.15.123:0/3216514738 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5190102e00 0x7f51900702c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:19.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:19.953+0000 7f5197400700 1 --2- 172.21.15.123:0/3216514738 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f51901037a0 0x7f5190070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:19.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:19.953+0000 7f5197400700 1 -- 172.21.15.123:0/3216514738 >> 172.21.15.123:0/3216514738 conn(0x7f51900f89b0 msgr2=0x7f51900f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:33:19.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:19.953+0000 7f5197400700 1 -- 172.21.15.123:0/3216514738 shutdown_connections 2024-03-30T04:33:19.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:19.953+0000 7f5197400700 1 -- 172.21.15.123:0/3216514738 wait complete. 2024-03-30T04:33:19.967 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:33:20.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.088+0000 7f4b9d000700 1 -- 172.21.15.123:0/2905196905 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b98102e00 msgr2=0x7f4b981031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:20.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.088+0000 7f4b9d000700 1 --2- 172.21.15.123:0/2905196905 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b98102e00 0x7f4b981031d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4b88004940 tx=0x7f4b88009e30 comp rx=0 tx=0).stop 2024-03-30T04:33:20.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.088+0000 7f4b9d000700 1 -- 172.21.15.123:0/2905196905 shutdown_connections 2024-03-30T04:33:20.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.088+0000 7f4b9d000700 1 --2- 172.21.15.123:0/2905196905 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4b981037a0 0x7f4b980fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:20.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.088+0000 7f4b9d000700 1 --2- 172.21.15.123:0/2905196905 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b98102e00 0x7f4b981031d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:20.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.088+0000 7f4b9d000700 1 -- 172.21.15.123:0/2905196905 >> 172.21.15.123:0/2905196905 conn(0x7f4b980f89b0 msgr2=0x7f4b980fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:33:20.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.088+0000 7f4b9d000700 1 -- 172.21.15.123:0/2905196905 shutdown_connections 2024-03-30T04:33:20.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.088+0000 7f4b9d000700 1 -- 172.21.15.123:0/2905196905 wait complete. 2024-03-30T04:33:20.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.089+0000 7f4b9d000700 1 Processor -- start 2024-03-30T04:33:20.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.089+0000 7f4b9d000700 1 -- start start 2024-03-30T04:33:20.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.090+0000 7f4b9d000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4b98102e00 0x7f4b98113a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:33:20.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.090+0000 7f4b9d000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b981037a0 0x7f4b98113f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:33:20.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.090+0000 7f4b9d000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b981145d0 con 0x7f4b981037a0 2024-03-30T04:33:20.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.090+0000 7f4b9d000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9810d9c0 con 0x7f4b98102e00 2024-03-30T04:33:20.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.090+0000 7f4b95400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b981037a0 0x7f4b98113f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:33:20.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.090+0000 7f4b95400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b981037a0 0x7f4b98113f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50038/0 (socket says 172.21.15.123:50038) 2024-03-30T04:33:20.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.090+0000 7f4b95400700 1 -- 172.21.15.123:0/2524844580 learned_addr learned my addr 172.21.15.123:0/2524844580 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:33:20.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.090+0000 7f4b95e00700 1 --2- 172.21.15.123:0/2524844580 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4b98102e00 0x7f4b98113a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:33:20.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.090+0000 7f4b95400700 1 -- 172.21.15.123:0/2524844580 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4b98102e00 msgr2=0x7f4b98113a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:20.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.090+0000 7f4b95400700 1 --2- 172.21.15.123:0/2524844580 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4b98102e00 0x7f4b98113a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:20.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.090+0000 7f4b95400700 1 -- 172.21.15.123:0/2524844580 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b88005040 con 0x7f4b981037a0 2024-03-30T04:33:20.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.091+0000 7f4b95400700 1 --2- 172.21.15.123:0/2524844580 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b981037a0 0x7f4b98113f80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4b8400eb40 tx=0x7f4b8400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:33:20.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.091+0000 7f4b8e000700 1 -- 172.21.15.123:0/2524844580 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b84007cc0 con 0x7f4b981037a0 2024-03-30T04:33:20.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.091+0000 7f4b8e000700 1 -- 172.21.15.123:0/2524844580 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4b8400f040 con 0x7f4b981037a0 2024-03-30T04:33:20.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.091+0000 7f4b9d000700 1 -- 172.21.15.123:0/2524844580 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b9810dc40 con 0x7f4b981037a0 2024-03-30T04:33:20.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.091+0000 7f4b8e000700 1 -- 172.21.15.123:0/2524844580 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b84009570 con 0x7f4b981037a0 2024-03-30T04:33:20.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.091+0000 7f4b9d000700 1 -- 172.21.15.123:0/2524844580 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b9810e200 con 0x7f4b981037a0 2024-03-30T04:33:20.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.092+0000 7f4b9d000700 1 -- 172.21.15.123:0/2524844580 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b980fe550 con 0x7f4b981037a0 2024-03-30T04:33:20.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.093+0000 7f4b8e000700 1 -- 172.21.15.123:0/2524844580 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 37) v1 ==== 90946+0+0 (secure 0 0 0) 0x7f4b840096d0 con 0x7f4b981037a0 2024-03-30T04:33:20.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.094+0000 7f4b8e000700 1 --2- 172.21.15.123:0/2524844580 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4b7c06cd40 0x7f4b7c06f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:33:20.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.094+0000 7f4b8e000700 1 -- 172.21.15.123:0/2524844580 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f4b8408be20 con 0x7f4b981037a0 2024-03-30T04:33:20.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.094+0000 7f4b95e00700 1 --2- 172.21.15.123:0/2524844580 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4b7c06cd40 0x7f4b7c06f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:33:20.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.095+0000 7f4b95e00700 1 --2- 172.21.15.123:0/2524844580 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4b7c06cd40 0x7f4b7c06f1f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4b88004560 tx=0x7f4b88009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:33:20.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.098+0000 7f4b8e000700 1 -- 172.21.15.123:0/2524844580 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4b84059700 con 0x7f4b981037a0 2024-03-30T04:33:20.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:19 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:20.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:19 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:20.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:19 smithi123 ceph-mon[139369]: Reconfiguring osd.3 (monmap changed)... 2024-03-30T04:33:20.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:19 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-30T04:33:20.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:19 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:33:20.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:19 smithi123 ceph-mon[139369]: Reconfiguring daemon osd.3 on smithi135 2024-03-30T04:33:20.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:19 smithi123 ceph-mon[139369]: pgmap v28: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:33:20.311 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:19 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:20.311 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:19 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:20.311 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:19 smithi135 ceph-mon[106011]: Reconfiguring osd.3 (monmap changed)... 2024-03-30T04:33:20.311 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:19 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-30T04:33:20.311 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:19 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:33:20.311 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:19 smithi135 ceph-mon[106011]: Reconfiguring daemon osd.3 on smithi135 2024-03-30T04:33:20.312 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:19 smithi135 ceph-mon[106011]: pgmap v28: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:33:20.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.322+0000 7f4b9d000700 1 -- 172.21.15.123:0/2524844580 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4b98109dc0 con 0x7f4b7c06cd40 2024-03-30T04:33:20.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.324+0000 7f4b8e000700 1 -- 172.21.15.123:0/2524844580 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7f4b98109dc0 con 0x7f4b7c06cd40 2024-03-30T04:33:20.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.326+0000 7f4b9d000700 1 -- 172.21.15.123:0/2524844580 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4b7c06cd40 msgr2=0x7f4b7c06f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:20.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.326+0000 7f4b9d000700 1 --2- 172.21.15.123:0/2524844580 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4b7c06cd40 0x7f4b7c06f1f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4b88004560 tx=0x7f4b88009e30 comp rx=0 tx=0).stop 2024-03-30T04:33:20.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.326+0000 7f4b9d000700 1 -- 172.21.15.123:0/2524844580 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b981037a0 msgr2=0x7f4b98113f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:20.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.326+0000 7f4b9d000700 1 --2- 172.21.15.123:0/2524844580 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b981037a0 0x7f4b98113f80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4b8400eb40 tx=0x7f4b8400b040 comp rx=0 tx=0).stop 2024-03-30T04:33:20.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.326+0000 7f4b9d000700 1 -- 172.21.15.123:0/2524844580 shutdown_connections 2024-03-30T04:33:20.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.326+0000 7f4b9d000700 1 --2- 172.21.15.123:0/2524844580 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4b7c06cd40 0x7f4b7c06f1f0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:20.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.326+0000 7f4b9d000700 1 --2- 172.21.15.123:0/2524844580 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4b98102e00 0x7f4b98113a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:20.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.326+0000 7f4b9d000700 1 --2- 172.21.15.123:0/2524844580 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b981037a0 0x7f4b98113f80 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:20.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.326+0000 7f4b9d000700 1 -- 172.21.15.123:0/2524844580 >> 172.21.15.123:0/2524844580 conn(0x7f4b980f89b0 msgr2=0x7f4b980f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:33:20.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.326+0000 7f4b9d000700 1 -- 172.21.15.123:0/2524844580 shutdown_connections 2024-03-30T04:33:20.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.326+0000 7f4b9d000700 1 -- 172.21.15.123:0/2524844580 wait complete. 2024-03-30T04:33:20.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.460+0000 7fdef0e00700 1 -- 172.21.15.123:0/2086519335 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdeec102e00 msgr2=0x7fdeec1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:20.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.460+0000 7fdef0e00700 1 --2- 172.21.15.123:0/2086519335 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdeec102e00 0x7fdeec1031d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fdedc004940 tx=0x7fdedc009e30 comp rx=0 tx=0).stop 2024-03-30T04:33:20.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.461+0000 7fdef0e00700 1 -- 172.21.15.123:0/2086519335 shutdown_connections 2024-03-30T04:33:20.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.461+0000 7fdef0e00700 1 --2- 172.21.15.123:0/2086519335 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdeec1037a0 0x7fdeec0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:20.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.461+0000 7fdef0e00700 1 --2- 172.21.15.123:0/2086519335 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdeec102e00 0x7fdeec1031d0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:20.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.461+0000 7fdef0e00700 1 -- 172.21.15.123:0/2086519335 >> 172.21.15.123:0/2086519335 conn(0x7fdeec0f89b0 msgr2=0x7fdeec0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:33:20.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.461+0000 7fdef0e00700 1 -- 172.21.15.123:0/2086519335 shutdown_connections 2024-03-30T04:33:20.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.461+0000 7fdef0e00700 1 -- 172.21.15.123:0/2086519335 wait complete. 2024-03-30T04:33:20.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.461+0000 7fdef0e00700 1 Processor -- start 2024-03-30T04:33:20.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.462+0000 7fdef0e00700 1 -- start start 2024-03-30T04:33:20.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.462+0000 7fdef0e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdeec102e00 0x7fdeec115aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:33:20.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.462+0000 7fdef0e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdeec1037a0 0x7fdeec115fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:33:20.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.462+0000 7fdef0e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeec116630 con 0x7fdeec102e00 2024-03-30T04:33:20.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.462+0000 7fdef0e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeec10fc10 con 0x7fdeec1037a0 2024-03-30T04:33:20.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.462+0000 7fdee9e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdeec102e00 0x7fdeec115aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:33:20.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.462+0000 7fdee9e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdeec102e00 0x7fdeec115aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50052/0 (socket says 172.21.15.123:50052) 2024-03-30T04:33:20.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.462+0000 7fdee9e00700 1 -- 172.21.15.123:0/2361382071 learned_addr learned my addr 172.21.15.123:0/2361382071 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:33:20.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.463+0000 7fdee9400700 1 --2- 172.21.15.123:0/2361382071 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdeec1037a0 0x7fdeec115fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:33:20.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.463+0000 7fdee9e00700 1 -- 172.21.15.123:0/2361382071 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdeec1037a0 msgr2=0x7fdeec115fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:20.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.463+0000 7fdee9e00700 1 --2- 172.21.15.123:0/2361382071 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdeec1037a0 0x7fdeec115fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:20.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.463+0000 7fdee9e00700 1 -- 172.21.15.123:0/2361382071 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdedc005040 con 0x7fdeec102e00 2024-03-30T04:33:20.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.463+0000 7fdee9e00700 1 --2- 172.21.15.123:0/2361382071 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdeec102e00 0x7fdeec115aa0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fdedc009360 tx=0x7fdedc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:33:20.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.463+0000 7fdee2000700 1 -- 172.21.15.123:0/2361382071 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdedc00e040 con 0x7fdeec102e00 2024-03-30T04:33:20.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.463+0000 7fdee2000700 1 -- 172.21.15.123:0/2361382071 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdedc013420 con 0x7fdeec102e00 2024-03-30T04:33:20.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.463+0000 7fdef0e00700 1 -- 172.21.15.123:0/2361382071 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdeec10fe90 con 0x7fdeec102e00 2024-03-30T04:33:20.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.463+0000 7fdee2000700 1 -- 172.21.15.123:0/2361382071 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdedc01c5f0 con 0x7fdeec102e00 2024-03-30T04:33:20.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.463+0000 7fdef0e00700 1 -- 172.21.15.123:0/2361382071 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdeec110450 con 0x7fdeec102e00 2024-03-30T04:33:20.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.465+0000 7fdef0e00700 1 -- 172.21.15.123:0/2361382071 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdeec0fe550 con 0x7fdeec102e00 2024-03-30T04:33:20.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.466+0000 7fdee2000700 1 -- 172.21.15.123:0/2361382071 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 37) v1 ==== 90946+0+0 (secure 0 0 0) 0x7fdedc013590 con 0x7fdeec102e00 2024-03-30T04:33:20.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.467+0000 7fdee2000700 1 --2- 172.21.15.123:0/2361382071 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fded006cd50 0x7fded006f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:33:20.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.467+0000 7fdee2000700 1 -- 172.21.15.123:0/2361382071 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fdedc08d4b0 con 0x7fdeec102e00 2024-03-30T04:33:20.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.467+0000 7fdee9400700 1 --2- 172.21.15.123:0/2361382071 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fded006cd50 0x7fded006f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:33:20.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.468+0000 7fdee9400700 1 --2- 172.21.15.123:0/2361382071 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fded006cd50 0x7fded006f200 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fdeec111340 tx=0x7fded8009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:33:20.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.471+0000 7fdee2000700 1 -- 172.21.15.123:0/2361382071 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fdedc05ae00 con 0x7fdeec102e00 2024-03-30T04:33:20.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.682+0000 7fdef0e00700 1 -- 172.21.15.123:0/2361382071 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fdeec1110f0 con 0x7fded006cd50 2024-03-30T04:33:20.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.692+0000 7fdee2000700 1 -- 172.21.15.123:0/2361382071 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fdeec1110f0 con 0x7fded006cd50 2024-03-30T04:33:20.691 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:33:20.691 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (2m) 44s ago 15m 14.8M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:33:20.691 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (15m) 44s ago 15m 6651k - 17.2.7-797-g5c024335 f77085f6aab7 bceb8b8c4837 2024-03-30T04:33:20.691 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (14m) 26s ago 14m 6651k - 17.2.7-797-g5c024335 f77085f6aab7 cd0fe67b2d34 2024-03-30T04:33:20.691 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (2m) 44s ago 14m 75.0M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T04:33:20.691 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (8m) 44s ago 8m 109M - 17.2.7-797-g5c024335 f77085f6aab7 2df066fba7a6 2024-03-30T04:33:20.691 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (8m) 44s ago 8m 57.8M - 17.2.7-797-g5c024335 f77085f6aab7 1293b8520db1 2024-03-30T04:33:20.691 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (8m) 26s ago 8m 67.4M - 17.2.7-797-g5c024335 f77085f6aab7 0dc41326818c 2024-03-30T04:33:20.692 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (8m) 26s ago 8m 119M - 17.2.7-797-g5c024335 f77085f6aab7 aef8a0cfc553 2024-03-30T04:33:20.692 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (4m) 44s ago 16m 469M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:33:20.692 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (3m) 26s ago 13m 447M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:33:20.692 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (61s) 44s ago 16m 27.5M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T04:33:20.692 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (31s) 26s ago 13m 23.8M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T04:33:20.692 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (3m) 44s ago 15m 7688k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:33:20.692 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (3m) 26s ago 14m 7772k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:33:20.692 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (13m) 44s ago 13m 221M 1618M 17.2.7-797-g5c024335 f77085f6aab7 57a7ba3fd75e 2024-03-30T04:33:20.692 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (12m) 44s ago 12m 235M 1618M 17.2.7-797-g5c024335 f77085f6aab7 e1a6c1818a0c 2024-03-30T04:33:20.692 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (11m) 44s ago 11m 188M 1618M 17.2.7-797-g5c024335 f77085f6aab7 4c6a727567bd 2024-03-30T04:33:20.692 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (11m) 26s ago 11m 258M 2642M 17.2.7-797-g5c024335 f77085f6aab7 6fbdd1d46a9d 2024-03-30T04:33:20.692 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (10m) 26s ago 10m 227M 2642M 17.2.7-797-g5c024335 f77085f6aab7 b74988e947e5 2024-03-30T04:33:20.693 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (10m) 26s ago 10m 219M 2642M 17.2.7-797-g5c024335 f77085f6aab7 1fd3185f5484 2024-03-30T04:33:20.693 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (2m) 44s ago 14m 47.8M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:33:20.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.694+0000 7fdef0e00700 1 -- 172.21.15.123:0/2361382071 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fded006cd50 msgr2=0x7fded006f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:20.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.694+0000 7fdef0e00700 1 --2- 172.21.15.123:0/2361382071 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fded006cd50 0x7fded006f200 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fdeec111340 tx=0x7fded8009ee0 comp rx=0 tx=0).stop 2024-03-30T04:33:20.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.694+0000 7fdef0e00700 1 -- 172.21.15.123:0/2361382071 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdeec102e00 msgr2=0x7fdeec115aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:20.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.694+0000 7fdef0e00700 1 --2- 172.21.15.123:0/2361382071 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdeec102e00 0x7fdeec115aa0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fdedc009360 tx=0x7fdedc009e30 comp rx=0 tx=0).stop 2024-03-30T04:33:20.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.694+0000 7fdef0e00700 1 -- 172.21.15.123:0/2361382071 shutdown_connections 2024-03-30T04:33:20.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.694+0000 7fdef0e00700 1 --2- 172.21.15.123:0/2361382071 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fded006cd50 0x7fded006f200 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:20.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.694+0000 7fdef0e00700 1 --2- 172.21.15.123:0/2361382071 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdeec102e00 0x7fdeec115aa0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:20.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.694+0000 7fdef0e00700 1 --2- 172.21.15.123:0/2361382071 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdeec1037a0 0x7fdeec115fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:20.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.694+0000 7fdef0e00700 1 -- 172.21.15.123:0/2361382071 >> 172.21.15.123:0/2361382071 conn(0x7fdeec0f89b0 msgr2=0x7fdeec0f99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:33:20.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.694+0000 7fdef0e00700 1 -- 172.21.15.123:0/2361382071 shutdown_connections 2024-03-30T04:33:20.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.694+0000 7fdef0e00700 1 -- 172.21.15.123:0/2361382071 wait complete. 2024-03-30T04:33:20.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.827+0000 7f00f3800700 1 -- 172.21.15.123:0/322371108 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00ec1037c0 msgr2=0x7f00ec0fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:20.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.827+0000 7f00f3800700 1 --2- 172.21.15.123:0/322371108 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00ec1037c0 0x7f00ec0fce30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f00d8004940 tx=0x7f00d8009e30 comp rx=0 tx=0).stop 2024-03-30T04:33:20.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.827+0000 7f00f3800700 1 -- 172.21.15.123:0/322371108 shutdown_connections 2024-03-30T04:33:20.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.827+0000 7f00f3800700 1 --2- 172.21.15.123:0/322371108 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00ec1037c0 0x7f00ec0fce30 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:20.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.827+0000 7f00f3800700 1 --2- 172.21.15.123:0/322371108 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f00ec102e20 0x7f00ec1031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:20.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.827+0000 7f00f3800700 1 -- 172.21.15.123:0/322371108 >> 172.21.15.123:0/322371108 conn(0x7f00ec0f89d0 msgr2=0x7f00ec0fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:33:20.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.828+0000 7f00f3800700 1 -- 172.21.15.123:0/322371108 shutdown_connections 2024-03-30T04:33:20.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.828+0000 7f00f3800700 1 -- 172.21.15.123:0/322371108 wait complete. 2024-03-30T04:33:20.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.828+0000 7f00f3800700 1 Processor -- start 2024-03-30T04:33:20.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.829+0000 7f00f3800700 1 -- start start 2024-03-30T04:33:20.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.829+0000 7f00f3800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00ec102e20 0x7f00ec115c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:33:20.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.829+0000 7f00f3800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f00ec1037c0 0x7f00ec116180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:33:20.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.829+0000 7f00f3800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ec1167d0 con 0x7f00ec102e20 2024-03-30T04:33:20.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.829+0000 7f00f3800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ec10fba0 con 0x7f00ec1037c0 2024-03-30T04:33:20.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.829+0000 7f00f0e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00ec102e20 0x7f00ec115c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:33:20.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.829+0000 7f00f0e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00ec102e20 0x7f00ec115c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50072/0 (socket says 172.21.15.123:50072) 2024-03-30T04:33:20.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.829+0000 7f00ebe00700 1 --2- 172.21.15.123:0/2991612059 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f00ec1037c0 0x7f00ec116180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:33:20.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.829+0000 7f00f0e00700 1 -- 172.21.15.123:0/2991612059 learned_addr learned my addr 172.21.15.123:0/2991612059 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:33:20.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.830+0000 7f00ebe00700 1 -- 172.21.15.123:0/2991612059 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00ec102e20 msgr2=0x7f00ec115c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:20.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.830+0000 7f00ebe00700 1 --2- 172.21.15.123:0/2991612059 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00ec102e20 0x7f00ec115c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:20.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.830+0000 7f00ebe00700 1 -- 172.21.15.123:0/2991612059 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00e0005040 con 0x7f00ec1037c0 2024-03-30T04:33:20.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.830+0000 7f00f0e00700 1 --2- 172.21.15.123:0/2991612059 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00ec102e20 0x7f00ec115c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:33:20.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.830+0000 7f00ebe00700 1 --2- 172.21.15.123:0/2991612059 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f00ec1037c0 0x7f00ec116180 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f00d8009fd0 tx=0x7f00d8009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:33:20.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.831+0000 7f00e8c00700 1 -- 172.21.15.123:0/2991612059 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f00d800e070 con 0x7f00ec1037c0 2024-03-30T04:33:20.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.831+0000 7f00e8c00700 1 -- 172.21.15.123:0/2991612059 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f00d80093b0 con 0x7f00ec1037c0 2024-03-30T04:33:20.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.831+0000 7f00e8c00700 1 -- 172.21.15.123:0/2991612059 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f00d801cc00 con 0x7f00ec1037c0 2024-03-30T04:33:20.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.831+0000 7f00f3800700 1 -- 172.21.15.123:0/2991612059 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00d8005040 con 0x7f00ec1037c0 2024-03-30T04:33:20.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.831+0000 7f00f3800700 1 -- 172.21.15.123:0/2991612059 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00ec110250 con 0x7f00ec1037c0 2024-03-30T04:33:20.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.833+0000 7f00f3800700 1 -- 172.21.15.123:0/2991612059 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00ec0fe570 con 0x7f00ec1037c0 2024-03-30T04:33:20.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.833+0000 7f00e8c00700 1 -- 172.21.15.123:0/2991612059 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 37) v1 ==== 90946+0+0 (secure 0 0 0) 0x7f00d8025400 con 0x7f00ec1037c0 2024-03-30T04:33:20.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.834+0000 7f00e8c00700 1 --2- 172.21.15.123:0/2991612059 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f00d406cd40 0x7f00d406f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:33:20.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.835+0000 7f00e8c00700 1 -- 172.21.15.123:0/2991612059 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f00d808d420 con 0x7f00ec1037c0 2024-03-30T04:33:20.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.835+0000 7f00f0e00700 1 --2- 172.21.15.123:0/2991612059 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f00d406cd40 0x7f00d406f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:33:20.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.835+0000 7f00f0e00700 1 --2- 172.21.15.123:0/2991612059 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f00d406cd40 0x7f00d406f1f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f00e00044e0 tx=0x7f00e000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:33:20.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:20.839+0000 7f00e8c00700 1 -- 172.21.15.123:0/2991612059 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f00d805ae00 con 0x7f00ec1037c0 2024-03-30T04:33:21.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.129+0000 7f00f3800700 1 -- 172.21.15.123:0/2991612059 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f00ec111290 con 0x7f00ec1037c0 2024-03-30T04:33:21.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.130+0000 7f00e8c00700 1 -- 172.21.15.123:0/2991612059 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+742 (secure 0 0 0) 0x7f00d805a990 con 0x7f00ec1037c0 2024-03-30T04:33:21.129 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:33:21.129 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:33:21.129 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:33:21.129 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:33:21.129 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:33:21.129 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:33:21.129 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:33:21.129 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:33:21.129 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 6 2024-03-30T04:33:21.129 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:33:21.130 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:33:21.130 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 4 2024-03-30T04:33:21.130 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:33:21.130 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:33:21.130 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 10, 2024-03-30T04:33:21.130 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 4 2024-03-30T04:33:21.130 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:33:21.130 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:33:21.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.133+0000 7f00f3800700 1 -- 172.21.15.123:0/2991612059 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f00d406cd40 msgr2=0x7f00d406f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:21.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.133+0000 7f00f3800700 1 --2- 172.21.15.123:0/2991612059 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f00d406cd40 0x7f00d406f1f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f00e00044e0 tx=0x7f00e000b040 comp rx=0 tx=0).stop 2024-03-30T04:33:21.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.133+0000 7f00f3800700 1 -- 172.21.15.123:0/2991612059 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f00ec1037c0 msgr2=0x7f00ec116180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:21.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.133+0000 7f00f3800700 1 --2- 172.21.15.123:0/2991612059 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f00ec1037c0 0x7f00ec116180 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f00d8009fd0 tx=0x7f00d8009e30 comp rx=0 tx=0).stop 2024-03-30T04:33:21.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.133+0000 7f00f3800700 1 -- 172.21.15.123:0/2991612059 shutdown_connections 2024-03-30T04:33:21.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.133+0000 7f00f3800700 1 --2- 172.21.15.123:0/2991612059 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f00d406cd40 0x7f00d406f1f0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:21.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.133+0000 7f00f3800700 1 --2- 172.21.15.123:0/2991612059 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00ec102e20 0x7f00ec115c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:21.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.133+0000 7f00f3800700 1 --2- 172.21.15.123:0/2991612059 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f00ec1037c0 0x7f00ec116180 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:21.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.133+0000 7f00f3800700 1 -- 172.21.15.123:0/2991612059 >> 172.21.15.123:0/2991612059 conn(0x7f00ec0f89d0 msgr2=0x7f00ec0f9970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:33:21.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.133+0000 7f00f3800700 1 -- 172.21.15.123:0/2991612059 shutdown_connections 2024-03-30T04:33:21.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.133+0000 7f00f3800700 1 -- 172.21.15.123:0/2991612059 wait complete. 2024-03-30T04:33:21.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.266+0000 7fd70f400700 1 -- 172.21.15.123:0/3180007597 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7081037a0 msgr2=0x7fd7080fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:21.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.266+0000 7fd70f400700 1 --2- 172.21.15.123:0/3180007597 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7081037a0 0x7fd7080fce10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd6f4004940 tx=0x7fd6f4009e30 comp rx=0 tx=0).stop 2024-03-30T04:33:21.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.266+0000 7fd70f400700 1 -- 172.21.15.123:0/3180007597 shutdown_connections 2024-03-30T04:33:21.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.266+0000 7fd70f400700 1 --2- 172.21.15.123:0/3180007597 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7081037a0 0x7fd7080fce10 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:21.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.266+0000 7fd70f400700 1 --2- 172.21.15.123:0/3180007597 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd708102e00 0x7fd7081031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:21.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.266+0000 7fd70f400700 1 -- 172.21.15.123:0/3180007597 >> 172.21.15.123:0/3180007597 conn(0x7fd7080f89b0 msgr2=0x7fd7080fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:33:21.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.266+0000 7fd70f400700 1 -- 172.21.15.123:0/3180007597 shutdown_connections 2024-03-30T04:33:21.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.266+0000 7fd70f400700 1 -- 172.21.15.123:0/3180007597 wait complete. 2024-03-30T04:33:21.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.267+0000 7fd70f400700 1 Processor -- start 2024-03-30T04:33:21.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.267+0000 7fd70f400700 1 -- start start 2024-03-30T04:33:21.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.267+0000 7fd70f400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd708102e00 0x7fd708115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:33:21.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.267+0000 7fd70f400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7081037a0 0x7fd7081161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:33:21.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.267+0000 7fd70f400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7081167f0 con 0x7fd7081037a0 2024-03-30T04:33:21.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.267+0000 7fd70f400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd70810fbc0 con 0x7fd708102e00 2024-03-30T04:33:21.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.268+0000 7fd707e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7081037a0 0x7fd7081161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:33:21.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.268+0000 7fd707e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7081037a0 0x7fd7081161a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50078/0 (socket says 172.21.15.123:50078) 2024-03-30T04:33:21.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.268+0000 7fd707e00700 1 -- 172.21.15.123:0/2028640662 learned_addr learned my addr 172.21.15.123:0/2028640662 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:33:21.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.268+0000 7fd70ca00700 1 --2- 172.21.15.123:0/2028640662 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd708102e00 0x7fd708115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:33:21.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.268+0000 7fd707e00700 1 -- 172.21.15.123:0/2028640662 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd708102e00 msgr2=0x7fd708115c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:21.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.268+0000 7fd707e00700 1 --2- 172.21.15.123:0/2028640662 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd708102e00 0x7fd708115c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:21.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.268+0000 7fd707e00700 1 -- 172.21.15.123:0/2028640662 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6f4005040 con 0x7fd7081037a0 2024-03-30T04:33:21.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.268+0000 7fd707e00700 1 --2- 172.21.15.123:0/2028640662 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7081037a0 0x7fd7081161a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd6f4000c00 tx=0x7fd6f4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:33:21.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.268+0000 7fd704c00700 1 -- 172.21.15.123:0/2028640662 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd6f400e070 con 0x7fd7081037a0 2024-03-30T04:33:21.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.268+0000 7fd704c00700 1 -- 172.21.15.123:0/2028640662 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd6f40093b0 con 0x7fd7081037a0 2024-03-30T04:33:21.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.268+0000 7fd704c00700 1 -- 172.21.15.123:0/2028640662 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd6f401ccb0 con 0x7fd7081037a0 2024-03-30T04:33:21.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.269+0000 7fd70f400700 1 -- 172.21.15.123:0/2028640662 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd70810fe40 con 0x7fd7081037a0 2024-03-30T04:33:21.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.269+0000 7fd70f400700 1 -- 172.21.15.123:0/2028640662 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd708110350 con 0x7fd7081037a0 2024-03-30T04:33:21.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.270+0000 7fd70f400700 1 -- 172.21.15.123:0/2028640662 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7080fe550 con 0x7fd7081037a0 2024-03-30T04:33:21.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.271+0000 7fd704c00700 1 -- 172.21.15.123:0/2028640662 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 37) v1 ==== 90946+0+0 (secure 0 0 0) 0x7fd6f401c490 con 0x7fd7081037a0 2024-03-30T04:33:21.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.271+0000 7fd704c00700 1 --2- 172.21.15.123:0/2028640662 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd6f006cd40 0x7fd6f006f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:33:21.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.272+0000 7fd704c00700 1 -- 172.21.15.123:0/2028640662 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fd6f408d770 con 0x7fd7081037a0 2024-03-30T04:33:21.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.272+0000 7fd70ca00700 1 --2- 172.21.15.123:0/2028640662 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd6f006cd40 0x7fd6f006f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:33:21.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.272+0000 7fd70ca00700 1 --2- 172.21.15.123:0/2028640662 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd6f006cd40 0x7fd6f006f1f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc009b80 tx=0x7fd6fc00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:33:21.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.276+0000 7fd704c00700 1 -- 172.21.15.123:0/2028640662 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd6f405b020 con 0x7fd7081037a0 2024-03-30T04:33:21.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.523+0000 7fd70f400700 1 -- 172.21.15.123:0/2028640662 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fd70805f5f0 con 0x7fd7081037a0 2024-03-30T04:33:21.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.523+0000 7fd704c00700 1 -- 172.21.15.123:0/2028640662 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 23 v23) v1 ==== 76+0+1876 (secure 0 0 0) 0x7fd6f4055e60 con 0x7fd7081037a0 2024-03-30T04:33:21.522 INFO:teuthology.orchestra.run.smithi123.stdout:e23 2024-03-30T04:33:21.522 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:33:21.522 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:33:21.523 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:33:21.523 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:33:21.523 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:33:21.523 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:33:21.523 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 23 2024-03-30T04:33:21.523 INFO:teuthology.orchestra.run.smithi123.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-30T04:33:21.523 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:33:21.523 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:32:48.448604+0000 2024-03-30T04:33:21.523 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:33:21.523 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:33:21.523 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:33:21.523 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:33:21.523 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:33:21.523 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T04:33:21.524 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:33:21.524 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:33:21.524 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 46 2024-03-30T04:33:21.524 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:33:21.524 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T04:33:21.524 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T04:33:21.524 INFO:teuthology.orchestra.run.smithi123.stdout:up {0=14488,1=24309} 2024-03-30T04:33:21.524 INFO:teuthology.orchestra.run.smithi123.stdout:failed 2024-03-30T04:33:21.524 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:33:21.524 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:33:21.524 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:33:21.524 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:33:21.524 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:33:21.524 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:33:21.525 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T04:33:21.525 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T04:33:21.525 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{0:14488} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.123:6828/3037849622,v1:172.21.15.123:6829/3037849622] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:33:21.525 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{0:14500} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/2433675995,v1:172.21.15.123:6827/2433675995] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:33:21.525 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{1:24309} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.135:6824/4050541062,v1:172.21.15.135:6825/4050541062] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:33:21.525 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{1:24319} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.135:6826/1096986122,v1:172.21.15.135:6827/1096986122] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:33:21.525 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:33:21.525 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:33:21.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.526+0000 7fd70f400700 1 -- 172.21.15.123:0/2028640662 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd6f006cd40 msgr2=0x7fd6f006f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:21.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.526+0000 7fd70f400700 1 --2- 172.21.15.123:0/2028640662 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd6f006cd40 0x7fd6f006f1f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc009b80 tx=0x7fd6fc00b040 comp rx=0 tx=0).stop 2024-03-30T04:33:21.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.526+0000 7fd70f400700 1 -- 172.21.15.123:0/2028640662 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7081037a0 msgr2=0x7fd7081161a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:21.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.526+0000 7fd70f400700 1 --2- 172.21.15.123:0/2028640662 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7081037a0 0x7fd7081161a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd6f4000c00 tx=0x7fd6f4009e30 comp rx=0 tx=0).stop 2024-03-30T04:33:21.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.526+0000 7fd70f400700 1 -- 172.21.15.123:0/2028640662 shutdown_connections 2024-03-30T04:33:21.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.526+0000 7fd70f400700 1 --2- 172.21.15.123:0/2028640662 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd6f006cd40 0x7fd6f006f1f0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:21.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.526+0000 7fd70f400700 1 --2- 172.21.15.123:0/2028640662 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd708102e00 0x7fd708115c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:21.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.526+0000 7fd70f400700 1 --2- 172.21.15.123:0/2028640662 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7081037a0 0x7fd7081161a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:21.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.526+0000 7fd70f400700 1 -- 172.21.15.123:0/2028640662 >> 172.21.15.123:0/2028640662 conn(0x7fd7080f89b0 msgr2=0x7fd7080f99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:33:21.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.526+0000 7fd70f400700 1 -- 172.21.15.123:0/2028640662 shutdown_connections 2024-03-30T04:33:21.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.526+0000 7fd70f400700 1 -- 172.21.15.123:0/2028640662 wait complete. 2024-03-30T04:33:21.527 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 23 2024-03-30T04:33:21.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.661+0000 7faba0600700 1 -- 172.21.15.123:0/385899228 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab981037a0 msgr2=0x7fab980fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:21.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.661+0000 7faba0600700 1 --2- 172.21.15.123:0/385899228 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab981037a0 0x7fab980fce10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fab84004940 tx=0x7fab84009e30 comp rx=0 tx=0).stop 2024-03-30T04:33:21.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.661+0000 7faba0600700 1 -- 172.21.15.123:0/385899228 shutdown_connections 2024-03-30T04:33:21.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.661+0000 7faba0600700 1 --2- 172.21.15.123:0/385899228 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab981037a0 0x7fab980fce10 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:21.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.661+0000 7faba0600700 1 --2- 172.21.15.123:0/385899228 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fab98102e00 0x7fab981031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:21.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.661+0000 7faba0600700 1 -- 172.21.15.123:0/385899228 >> 172.21.15.123:0/385899228 conn(0x7fab980f89b0 msgr2=0x7fab980fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:33:21.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.661+0000 7faba0600700 1 -- 172.21.15.123:0/385899228 shutdown_connections 2024-03-30T04:33:21.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.661+0000 7faba0600700 1 -- 172.21.15.123:0/385899228 wait complete. 2024-03-30T04:33:21.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.662+0000 7faba0600700 1 Processor -- start 2024-03-30T04:33:21.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.663+0000 7faba0600700 1 -- start start 2024-03-30T04:33:21.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.663+0000 7faba0600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fab98102e00 0x7fab980702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:33:21.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.663+0000 7faba0600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab981037a0 0x7fab98070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:33:21.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.663+0000 7faba0600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab980720b0 con 0x7fab981037a0 2024-03-30T04:33:21.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.663+0000 7faba0600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab98072220 con 0x7fab98102e00 2024-03-30T04:33:21.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.663+0000 7fab9d200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab981037a0 0x7fab98070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:33:21.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.663+0000 7fab9d200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab981037a0 0x7fab98070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50100/0 (socket says 172.21.15.123:50100) 2024-03-30T04:33:21.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.663+0000 7fab9d200700 1 -- 172.21.15.123:0/3666413207 learned_addr learned my addr 172.21.15.123:0/3666413207 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:33:21.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.663+0000 7fab9dc00700 1 --2- 172.21.15.123:0/3666413207 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fab98102e00 0x7fab980702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:33:21.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.664+0000 7fab9d200700 1 -- 172.21.15.123:0/3666413207 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fab98102e00 msgr2=0x7fab980702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:21.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.664+0000 7fab9d200700 1 --2- 172.21.15.123:0/3666413207 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fab98102e00 0x7fab980702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:21.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.664+0000 7fab9d200700 1 -- 172.21.15.123:0/3666413207 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab84005040 con 0x7fab981037a0 2024-03-30T04:33:21.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.664+0000 7fab9d200700 1 --2- 172.21.15.123:0/3666413207 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab981037a0 0x7fab98070800 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fab84009390 tx=0x7fab84009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:33:21.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.664+0000 7fab95600700 1 -- 172.21.15.123:0/3666413207 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab8400e040 con 0x7fab981037a0 2024-03-30T04:33:21.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.664+0000 7fab95600700 1 -- 172.21.15.123:0/3666413207 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fab84013420 con 0x7fab981037a0 2024-03-30T04:33:21.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.664+0000 7faba0600700 1 -- 172.21.15.123:0/3666413207 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab98070e00 con 0x7fab981037a0 2024-03-30T04:33:21.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.664+0000 7fab95600700 1 -- 172.21.15.123:0/3666413207 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab8401c630 con 0x7fab981037a0 2024-03-30T04:33:21.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.664+0000 7faba0600700 1 -- 172.21.15.123:0/3666413207 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab9811e700 con 0x7fab981037a0 2024-03-30T04:33:21.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.665+0000 7faba0600700 1 -- 172.21.15.123:0/3666413207 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab980fe550 con 0x7fab981037a0 2024-03-30T04:33:21.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.666+0000 7fab95600700 1 -- 172.21.15.123:0/3666413207 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 37) v1 ==== 90946+0+0 (secure 0 0 0) 0x7fab84013a00 con 0x7fab981037a0 2024-03-30T04:33:21.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.667+0000 7fab95600700 1 --2- 172.21.15.123:0/3666413207 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fab8006cd40 0x7fab8006f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:33:21.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.667+0000 7fab95600700 1 -- 172.21.15.123:0/3666413207 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fab8408d4d0 con 0x7fab981037a0 2024-03-30T04:33:21.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.667+0000 7fab9dc00700 1 --2- 172.21.15.123:0/3666413207 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fab8006cd40 0x7fab8006f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:33:21.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.668+0000 7fab9dc00700 1 --2- 172.21.15.123:0/3666413207 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fab8006cd40 0x7fab8006f1f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fab8c009fd0 tx=0x7fab8c009f60 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:33:21.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.671+0000 7fab95600700 1 -- 172.21.15.123:0/3666413207 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fab8405ae00 con 0x7fab981037a0 2024-03-30T04:33:21.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.892+0000 7faba0600700 1 -- 172.21.15.123:0/3666413207 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fab98109da0 con 0x7fab8006cd40 2024-03-30T04:33:21.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.893+0000 7fab95600700 1 -- 172.21.15.123:0/3666413207 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7fab98109da0 con 0x7fab8006cd40 2024-03-30T04:33:21.892 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:33:21.892 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:33:21.892 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:33:21.892 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T04:33:21.892 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:33:21.892 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T04:33:21.892 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T04:33:21.892 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:33:21.892 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "4/21 daemons upgraded", 2024-03-30T04:33:21.893 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mon daemons", 2024-03-30T04:33:21.893 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:33:21.893 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:33:21.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.896+0000 7faba0600700 1 -- 172.21.15.123:0/3666413207 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fab8006cd40 msgr2=0x7fab8006f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:21.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.896+0000 7faba0600700 1 --2- 172.21.15.123:0/3666413207 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fab8006cd40 0x7fab8006f1f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fab8c009fd0 tx=0x7fab8c009f60 comp rx=0 tx=0).stop 2024-03-30T04:33:21.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.896+0000 7faba0600700 1 -- 172.21.15.123:0/3666413207 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab981037a0 msgr2=0x7fab98070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:21.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.896+0000 7faba0600700 1 --2- 172.21.15.123:0/3666413207 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab981037a0 0x7fab98070800 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fab84009390 tx=0x7fab84009e30 comp rx=0 tx=0).stop 2024-03-30T04:33:21.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.896+0000 7faba0600700 1 -- 172.21.15.123:0/3666413207 shutdown_connections 2024-03-30T04:33:21.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.896+0000 7faba0600700 1 --2- 172.21.15.123:0/3666413207 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fab8006cd40 0x7fab8006f1f0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:21.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.896+0000 7faba0600700 1 --2- 172.21.15.123:0/3666413207 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fab98102e00 0x7fab980702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:21.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.896+0000 7faba0600700 1 --2- 172.21.15.123:0/3666413207 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab981037a0 0x7fab98070800 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:21.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.896+0000 7faba0600700 1 -- 172.21.15.123:0/3666413207 >> 172.21.15.123:0/3666413207 conn(0x7fab980f89b0 msgr2=0x7fab980f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:33:21.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.896+0000 7faba0600700 1 -- 172.21.15.123:0/3666413207 shutdown_connections 2024-03-30T04:33:21.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:21.896+0000 7faba0600700 1 -- 172.21.15.123:0/3666413207 wait complete. 2024-03-30T04:33:22.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:22.029+0000 7f6a50600700 1 -- 172.21.15.123:0/2900223642 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a480fdeb0 msgr2=0x7f6a480fe280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:22.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:22.029+0000 7f6a50600700 1 --2- 172.21.15.123:0/2900223642 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a480fdeb0 0x7f6a480fe280 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c004970 tx=0x7f6a3c009e30 comp rx=0 tx=0).stop 2024-03-30T04:33:22.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:22.029+0000 7f6a50600700 1 -- 172.21.15.123:0/2900223642 shutdown_connections 2024-03-30T04:33:22.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:22.029+0000 7f6a50600700 1 --2- 172.21.15.123:0/2900223642 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6a480fe850 0x7f6a48106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:22.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:22.029+0000 7f6a50600700 1 --2- 172.21.15.123:0/2900223642 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a480fdeb0 0x7f6a480fe280 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:22.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:22.029+0000 7f6a50600700 1 -- 172.21.15.123:0/2900223642 >> 172.21.15.123:0/2900223642 conn(0x7f6a48072bc0 msgr2=0x7f6a48072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:33:22.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:22.029+0000 7f6a50600700 1 -- 172.21.15.123:0/2900223642 shutdown_connections 2024-03-30T04:33:22.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:22.029+0000 7f6a50600700 1 -- 172.21.15.123:0/2900223642 wait complete. 2024-03-30T04:33:22.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:22.030+0000 7f6a50600700 1 Processor -- start 2024-03-30T04:33:22.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:22.030+0000 7f6a50600700 1 -- start start 2024-03-30T04:33:22.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:22.030+0000 7f6a50600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a480fdeb0 0x7f6a480fbd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:33:22.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:22.030+0000 7f6a50600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6a480fe850 0x7f6a480fc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:33:22.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:22.030+0000 7f6a50600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a480fc840 con 0x7f6a480fdeb0 2024-03-30T04:33:22.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:22.030+0000 7f6a50600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a480fc980 con 0x7f6a480fe850 2024-03-30T04:33:22.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:22.031+0000 7f6a4dc00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a480fdeb0 0x7f6a480fbd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:33:22.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:22.031+0000 7f6a4dc00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a480fdeb0 0x7f6a480fbd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50126/0 (socket says 172.21.15.123:50126) 2024-03-30T04:33:22.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:22.031+0000 7f6a4dc00700 1 -- 172.21.15.123:0/1925067513 learned_addr learned my addr 172.21.15.123:0/1925067513 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:33:22.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:22.031+0000 7f6a4d200700 1 --2- 172.21.15.123:0/1925067513 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6a480fe850 0x7f6a480fc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:33:22.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:22.031+0000 7f6a4dc00700 1 -- 172.21.15.123:0/1925067513 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6a480fe850 msgr2=0x7f6a480fc2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:22.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:22.031+0000 7f6a4dc00700 1 --2- 172.21.15.123:0/1925067513 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6a480fe850 0x7f6a480fc2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:22.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:22.031+0000 7f6a4dc00700 1 -- 172.21.15.123:0/1925067513 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a3c005040 con 0x7f6a480fdeb0 2024-03-30T04:33:22.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:22.031+0000 7f6a4dc00700 1 --2- 172.21.15.123:0/1925067513 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a480fdeb0 0x7f6a480fbd70 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c005010 tx=0x7f6a3c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:33:22.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:22.031+0000 7f6a45600700 1 -- 172.21.15.123:0/1925067513 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a3c00e040 con 0x7f6a480fdeb0 2024-03-30T04:33:22.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:22.032+0000 7f6a45600700 1 -- 172.21.15.123:0/1925067513 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6a3c004290 con 0x7f6a480fdeb0 2024-03-30T04:33:22.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:22.032+0000 7f6a50600700 1 -- 172.21.15.123:0/1925067513 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a48119d70 con 0x7f6a480fdeb0 2024-03-30T04:33:22.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:22.032+0000 7f6a45600700 1 -- 172.21.15.123:0/1925067513 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a3c013d00 con 0x7f6a480fdeb0 2024-03-30T04:33:22.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:22.032+0000 7f6a50600700 1 -- 172.21.15.123:0/1925067513 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a4811a170 con 0x7f6a480fdeb0 2024-03-30T04:33:22.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:22.033+0000 7f6a50600700 1 -- 172.21.15.123:0/1925067513 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a48065e10 con 0x7f6a480fdeb0 2024-03-30T04:33:22.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:22.034+0000 7f6a45600700 1 -- 172.21.15.123:0/1925067513 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 37) v1 ==== 90946+0+0 (secure 0 0 0) 0x7f6a3c0134b0 con 0x7f6a480fdeb0 2024-03-30T04:33:22.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:22.034+0000 7f6a45600700 1 --2- 172.21.15.123:0/1925067513 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6a3006cd40 0x7f6a3006f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:33:22.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:22.035+0000 7f6a45600700 1 -- 172.21.15.123:0/1925067513 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f6a3c08d3a0 con 0x7f6a480fdeb0 2024-03-30T04:33:22.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:22.035+0000 7f6a4d200700 1 --2- 172.21.15.123:0/1925067513 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6a3006cd40 0x7f6a3006f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:33:22.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:22.035+0000 7f6a4d200700 1 --2- 172.21.15.123:0/1925067513 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6a3006cd40 0x7f6a3006f1f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6a480fd660 tx=0x7f6a3400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:33:22.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:22.039+0000 7f6a45600700 1 -- 172.21.15.123:0/1925067513 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6a3c0571c0 con 0x7f6a480fdeb0 2024-03-30T04:33:22.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:21 smithi135 ceph-mon[106011]: from='client.34160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:33:22.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:21 smithi135 ceph-mon[106011]: from='client.34164 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:33:22.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:21 smithi135 ceph-mon[106011]: from='client.34168 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:33:22.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:21 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:22.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:21 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:22.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:21 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-30T04:33:22.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:21 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:33:22.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:21 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2991612059' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:33:22.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:21 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2028640662' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:33:22.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:21 smithi123 ceph-mon[139369]: from='client.34160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:33:22.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:21 smithi123 ceph-mon[139369]: from='client.34164 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:33:22.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:21 smithi123 ceph-mon[139369]: from='client.34168 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:33:22.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:21 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:22.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:21 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:22.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:21 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-30T04:33:22.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:21 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:33:22.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:21 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2991612059' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:33:22.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:21 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2028640662' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:33:22.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:22.321+0000 7f6a50600700 1 -- 172.21.15.123:0/1925067513 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f6a4805f5f0 con 0x7f6a480fdeb0 2024-03-30T04:33:22.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:22.321+0000 7f6a45600700 1 -- 172.21.15.123:0/1925067513 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f6a3c05a7e0 con 0x7f6a480fdeb0 2024-03-30T04:33:22.320 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_OK 2024-03-30T04:33:22.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:22.324+0000 7f6a50600700 1 -- 172.21.15.123:0/1925067513 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6a3006cd40 msgr2=0x7f6a3006f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:22.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:22.324+0000 7f6a50600700 1 --2- 172.21.15.123:0/1925067513 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6a3006cd40 0x7f6a3006f1f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6a480fd660 tx=0x7f6a3400b040 comp rx=0 tx=0).stop 2024-03-30T04:33:22.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:22.324+0000 7f6a50600700 1 -- 172.21.15.123:0/1925067513 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a480fdeb0 msgr2=0x7f6a480fbd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:22.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:22.324+0000 7f6a50600700 1 --2- 172.21.15.123:0/1925067513 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a480fdeb0 0x7f6a480fbd70 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c005010 tx=0x7f6a3c009e30 comp rx=0 tx=0).stop 2024-03-30T04:33:22.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:22.324+0000 7f6a50600700 1 -- 172.21.15.123:0/1925067513 shutdown_connections 2024-03-30T04:33:22.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:22.324+0000 7f6a50600700 1 --2- 172.21.15.123:0/1925067513 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6a3006cd40 0x7f6a3006f1f0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:22.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:22.324+0000 7f6a50600700 1 --2- 172.21.15.123:0/1925067513 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a480fdeb0 0x7f6a480fbd70 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:22.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:22.324+0000 7f6a50600700 1 --2- 172.21.15.123:0/1925067513 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6a480fe850 0x7f6a480fc2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:22.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:22.324+0000 7f6a50600700 1 -- 172.21.15.123:0/1925067513 >> 172.21.15.123:0/1925067513 conn(0x7f6a48072bc0 msgr2=0x7f6a480ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:33:22.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:22.324+0000 7f6a50600700 1 -- 172.21.15.123:0/1925067513 shutdown_connections 2024-03-30T04:33:22.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:22.324+0000 7f6a50600700 1 -- 172.21.15.123:0/1925067513 wait complete. 2024-03-30T04:33:23.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:22 smithi135 ceph-mon[106011]: Reconfiguring osd.4 (monmap changed)... 2024-03-30T04:33:23.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:22 smithi135 ceph-mon[106011]: Reconfiguring daemon osd.4 on smithi135 2024-03-30T04:33:23.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:22 smithi135 ceph-mon[106011]: pgmap v29: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:33:23.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:22 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/1925067513' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:33:23.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:22 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:23.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:22 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:23.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:22 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-30T04:33:23.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:22 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:33:23.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:22 smithi123 ceph-mon[139369]: Reconfiguring osd.4 (monmap changed)... 2024-03-30T04:33:23.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:22 smithi123 ceph-mon[139369]: Reconfiguring daemon osd.4 on smithi135 2024-03-30T04:33:23.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:22 smithi123 ceph-mon[139369]: pgmap v29: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:33:23.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:22 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/1925067513' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:33:23.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:22 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:23.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:22 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:23.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:22 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-30T04:33:23.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:22 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:33:24.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:23 smithi135 ceph-mon[106011]: from='client.34180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:33:24.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:23 smithi135 ceph-mon[106011]: Reconfiguring osd.5 (monmap changed)... 2024-03-30T04:33:24.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:23 smithi135 ceph-mon[106011]: Reconfiguring daemon osd.5 on smithi135 2024-03-30T04:33:24.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:23 smithi123 ceph-mon[139369]: from='client.34180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:33:24.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:23 smithi123 ceph-mon[139369]: Reconfiguring osd.5 (monmap changed)... 2024-03-30T04:33:24.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:23 smithi123 ceph-mon[139369]: Reconfiguring daemon osd.5 on smithi135 2024-03-30T04:33:25.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:24 smithi135 ceph-mon[106011]: pgmap v30: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:33:25.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:24 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:25.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:24 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:25.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:24 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi135.zvnlap", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T04:33:25.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:24 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:33:25.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:24 smithi123 ceph-mon[139369]: pgmap v30: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:33:25.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:24 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:25.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:24 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:25.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:24 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi135.zvnlap", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T04:33:25.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:24 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:33:26.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:25 smithi135 ceph-mon[106011]: Reconfiguring mds.cephfs.smithi135.zvnlap (monmap changed)... 2024-03-30T04:33:26.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:25 smithi135 ceph-mon[106011]: Reconfiguring daemon mds.cephfs.smithi135.zvnlap on smithi135 2024-03-30T04:33:26.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:25 smithi123 ceph-mon[139369]: Reconfiguring mds.cephfs.smithi135.zvnlap (monmap changed)... 2024-03-30T04:33:26.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:25 smithi123 ceph-mon[139369]: Reconfiguring daemon mds.cephfs.smithi135.zvnlap on smithi135 2024-03-30T04:33:27.189 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:26 smithi135 ceph-mon[106011]: pgmap v31: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:33:27.189 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:26 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:27.189 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:26 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:27.189 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:26 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi135.xwgjvb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T04:33:27.189 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:26 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:33:27.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:26 smithi123 ceph-mon[139369]: pgmap v31: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:33:27.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:26 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:27.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:26 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:27.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:26 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi135.xwgjvb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T04:33:27.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:26 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:33:28.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:27 smithi123 ceph-mon[139369]: Reconfiguring mds.cephfs.smithi135.xwgjvb (monmap changed)... 2024-03-30T04:33:28.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:27 smithi123 ceph-mon[139369]: Reconfiguring daemon mds.cephfs.smithi135.xwgjvb on smithi135 2024-03-30T04:33:28.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:27 smithi135 ceph-mon[106011]: Reconfiguring mds.cephfs.smithi135.xwgjvb (monmap changed)... 2024-03-30T04:33:28.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:27 smithi135 ceph-mon[106011]: Reconfiguring daemon mds.cephfs.smithi135.xwgjvb on smithi135 2024-03-30T04:33:29.103 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:28 smithi123 ceph-mon[139369]: pgmap v32: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:33:29.103 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:28 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:29.103 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:28 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:29.103 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:28 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:33:29.104 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:28 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:33:29.104 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:28 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:33:29.104 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:28 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:29.104 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:28 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi123"}]: dispatch 2024-03-30T04:33:29.104 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:28 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi123"}]': finished 2024-03-30T04:33:29.104 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:28 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi135"}]: dispatch 2024-03-30T04:33:29.104 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:28 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi135"}]': finished 2024-03-30T04:33:29.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:28 smithi135 ceph-mon[106011]: pgmap v32: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:33:29.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:28 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:29.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:28 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:29.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:28 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:33:29.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:28 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:33:29.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:28 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:33:29.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:28 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:29.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:28 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi123"}]: dispatch 2024-03-30T04:33:29.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:28 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi123"}]': finished 2024-03-30T04:33:29.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:28 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi135"}]: dispatch 2024-03-30T04:33:29.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:28 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi135"}]': finished 2024-03-30T04:33:30.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:29 smithi123 ceph-mon[139369]: Upgrade: Setting container_image for all mon 2024-03-30T04:33:30.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:29 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:33:30.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:29 smithi135 ceph-mon[106011]: Upgrade: Setting container_image for all mon 2024-03-30T04:33:30.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:29 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:33:30.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:30 smithi123 ceph-mon[139369]: pgmap v33: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:33:30.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:30 smithi123 ceph-mon[139369]: Upgrade: Updating crash.smithi123 (1/2) 2024-03-30T04:33:30.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:30.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi123", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-30T04:33:30.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:33:30.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:30 smithi123 ceph-mon[139369]: Deploying daemon crash.smithi123 on smithi123 2024-03-30T04:33:31.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:30 smithi135 ceph-mon[106011]: pgmap v33: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:33:31.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:30 smithi135 ceph-mon[106011]: Upgrade: Updating crash.smithi123 (1/2) 2024-03-30T04:33:31.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:30 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:31.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:30 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi123", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-30T04:33:31.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:30 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:33:31.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:30 smithi135 ceph-mon[106011]: Deploying daemon crash.smithi123 on smithi123 2024-03-30T04:33:32.168 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:31 smithi123 ceph-mon[139369]: pgmap v34: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:33:32.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:31 smithi135 ceph-mon[106011]: pgmap v34: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:33:34.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:34 smithi123 ceph-mon[139369]: pgmap v35: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:33:34.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:34 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:34.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:34 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:34.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:34 smithi135 ceph-mon[106011]: pgmap v35: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:33:34.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:34 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:34.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:34 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:36.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:36 smithi135 ceph-mon[106011]: pgmap v36: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:33:36.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:36 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:36.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:36 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi135", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-30T04:33:36.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:36 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:33:36.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:36 smithi123 ceph-mon[139369]: pgmap v36: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:33:36.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:36 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:36.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:36 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi135", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-30T04:33:36.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:36 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:33:37.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:37 smithi123 ceph-mon[139369]: Upgrade: Updating crash.smithi135 (2/2) 2024-03-30T04:33:37.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:37 smithi123 ceph-mon[139369]: Deploying daemon crash.smithi135 on smithi135 2024-03-30T04:33:37.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:37 smithi135 ceph-mon[106011]: Upgrade: Updating crash.smithi135 (2/2) 2024-03-30T04:33:37.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:37 smithi135 ceph-mon[106011]: Deploying daemon crash.smithi135 on smithi135 2024-03-30T04:33:38.556 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:38 smithi135 ceph-mon[106011]: pgmap v37: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:33:38.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:38 smithi123 ceph-mon[139369]: pgmap v37: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:33:40.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:40 smithi135 ceph-mon[106011]: pgmap v38: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:33:40.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:40 smithi123 ceph-mon[139369]: pgmap v38: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:33:41.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:41 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:41.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:41 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:41.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:41 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:33:41.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:41 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:41.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:41 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:41.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:41 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:33:42.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:42 smithi135 ceph-mon[106011]: pgmap v39: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:33:42.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:42 smithi123 ceph-mon[139369]: pgmap v39: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:33:44.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:44 smithi123 ceph-mon[139369]: pgmap v40: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:33:44.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:44.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:44.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:33:44.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:44 smithi135 ceph-mon[106011]: pgmap v40: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:33:44.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:44 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:44.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:44 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:44.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:44 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:33:46.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:46 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:46.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:46 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:46.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:46 smithi123 ceph-mon[139369]: pgmap v41: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:33:46.538 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:46 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:46.538 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:46 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:46.538 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:46 smithi135 ceph-mon[106011]: pgmap v41: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:33:47.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:47 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:47.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:47 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:47.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:47 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:47.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:47 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:49.001 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:48 smithi123 ceph-mon[139369]: pgmap v42: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:33:49.001 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:48 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:49.001 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:48 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:49.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:48 smithi135 ceph-mon[106011]: pgmap v42: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:33:49.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:48 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:49.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:48 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:50.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:50 smithi123 ceph-mon[139369]: pgmap v43: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:33:50.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:50 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:50.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:50 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:51.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:50 smithi135 ceph-mon[106011]: pgmap v43: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:33:51.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:50 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:51.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:50 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:52.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.469+0000 7f2b80e00700 1 -- 172.21.15.123:0/1066909671 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b7c071f90 msgr2=0x7f2b7c070360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:52.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.469+0000 7f2b80e00700 1 --2- 172.21.15.123:0/1066909671 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b7c071f90 0x7f2b7c070360 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c004970 tx=0x7f2b6c009e30 comp rx=0 tx=0).stop 2024-03-30T04:33:52.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.469+0000 7f2b80e00700 1 -- 172.21.15.123:0/1066909671 shutdown_connections 2024-03-30T04:33:52.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.469+0000 7f2b80e00700 1 --2- 172.21.15.123:0/1066909671 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2b7c0708a0 0x7f2b7c070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:52.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.469+0000 7f2b80e00700 1 --2- 172.21.15.123:0/1066909671 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b7c071f90 0x7f2b7c070360 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:52.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.469+0000 7f2b80e00700 1 -- 172.21.15.123:0/1066909671 >> 172.21.15.123:0/1066909671 conn(0x7f2b7c0f89b0 msgr2=0x7f2b7c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:33:52.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.469+0000 7f2b80e00700 1 -- 172.21.15.123:0/1066909671 shutdown_connections 2024-03-30T04:33:52.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.469+0000 7f2b80e00700 1 -- 172.21.15.123:0/1066909671 wait complete. 2024-03-30T04:33:52.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.470+0000 7f2b80e00700 1 Processor -- start 2024-03-30T04:33:52.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.470+0000 7f2b80e00700 1 -- start start 2024-03-30T04:33:52.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.470+0000 7f2b80e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b7c0708a0 0x7f2b7c11a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:33:52.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.470+0000 7f2b80e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2b7c071f90 0x7f2b7c11a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:33:52.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.470+0000 7f2b80e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7c11ac50 con 0x7f2b7c0708a0 2024-03-30T04:33:52.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.470+0000 7f2b80e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7c114020 con 0x7f2b7c071f90 2024-03-30T04:33:52.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.470+0000 7f2b79e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b7c0708a0 0x7f2b7c11a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:33:52.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.470+0000 7f2b79e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b7c0708a0 0x7f2b7c11a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39104/0 (socket says 172.21.15.123:39104) 2024-03-30T04:33:52.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.470+0000 7f2b79e00700 1 -- 172.21.15.123:0/424646110 learned_addr learned my addr 172.21.15.123:0/424646110 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:33:52.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.470+0000 7f2b79400700 1 --2- 172.21.15.123:0/424646110 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2b7c071f90 0x7f2b7c11a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:33:52.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.471+0000 7f2b79e00700 1 -- 172.21.15.123:0/424646110 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2b7c071f90 msgr2=0x7f2b7c11a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:52.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.471+0000 7f2b79e00700 1 --2- 172.21.15.123:0/424646110 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2b7c071f90 0x7f2b7c11a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:52.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.471+0000 7f2b79e00700 1 -- 172.21.15.123:0/424646110 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b6c005040 con 0x7f2b7c0708a0 2024-03-30T04:33:52.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.471+0000 7f2b79e00700 1 --2- 172.21.15.123:0/424646110 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b7c0708a0 0x7f2b7c11a0c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c005010 tx=0x7f2b6c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:33:52.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.471+0000 7f2b72000700 1 -- 172.21.15.123:0/424646110 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b6c00e040 con 0x7f2b7c0708a0 2024-03-30T04:33:52.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.471+0000 7f2b72000700 1 -- 172.21.15.123:0/424646110 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2b6c0098b0 con 0x7f2b7c0708a0 2024-03-30T04:33:52.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.471+0000 7f2b72000700 1 -- 172.21.15.123:0/424646110 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b6c013e80 con 0x7f2b7c0708a0 2024-03-30T04:33:52.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.471+0000 7f2b80e00700 1 -- 172.21.15.123:0/424646110 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b7c1142a0 con 0x7f2b7c0708a0 2024-03-30T04:33:52.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.472+0000 7f2b80e00700 1 -- 172.21.15.123:0/424646110 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b7c1147e0 con 0x7f2b7c0708a0 2024-03-30T04:33:52.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.473+0000 7f2b80e00700 1 -- 172.21.15.123:0/424646110 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b7c071aa0 con 0x7f2b7c0708a0 2024-03-30T04:33:52.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.473+0000 7f2b72000700 1 -- 172.21.15.123:0/424646110 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 37) v1 ==== 90946+0+0 (secure 0 0 0) 0x7f2b6c0136b0 con 0x7f2b7c0708a0 2024-03-30T04:33:52.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.474+0000 7f2b72000700 1 --2- 172.21.15.123:0/424646110 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2b6006cd40 0x7f2b6006f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:33:52.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.474+0000 7f2b72000700 1 -- 172.21.15.123:0/424646110 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f2b6c08d420 con 0x7f2b7c0708a0 2024-03-30T04:33:52.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.475+0000 7f2b79400700 1 --2- 172.21.15.123:0/424646110 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2b6006cd40 0x7f2b6006f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:33:52.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.475+0000 7f2b79400700 1 --2- 172.21.15.123:0/424646110 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2b6006cd40 0x7f2b6006f1f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c115960 tx=0x7f2b6800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:33:52.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.479+0000 7f2b72000700 1 -- 172.21.15.123:0/424646110 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2b6c05ae10 con 0x7f2b7c0708a0 2024-03-30T04:33:52.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.747+0000 7f2b80e00700 1 -- 172.21.15.123:0/424646110 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2b7c04bf40 con 0x7f2b6006cd40 2024-03-30T04:33:52.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.748+0000 7f2b72000700 1 -- 172.21.15.123:0/424646110 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f2b7c04bf40 con 0x7f2b6006cd40 2024-03-30T04:33:52.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.750+0000 7f2b80e00700 1 -- 172.21.15.123:0/424646110 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2b6006cd40 msgr2=0x7f2b6006f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:52.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.751+0000 7f2b80e00700 1 --2- 172.21.15.123:0/424646110 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2b6006cd40 0x7f2b6006f1f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c115960 tx=0x7f2b6800b040 comp rx=0 tx=0).stop 2024-03-30T04:33:52.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.751+0000 7f2b80e00700 1 -- 172.21.15.123:0/424646110 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b7c0708a0 msgr2=0x7f2b7c11a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:52.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.751+0000 7f2b80e00700 1 --2- 172.21.15.123:0/424646110 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b7c0708a0 0x7f2b7c11a0c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c005010 tx=0x7f2b6c009e30 comp rx=0 tx=0).stop 2024-03-30T04:33:52.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.751+0000 7f2b80e00700 1 -- 172.21.15.123:0/424646110 shutdown_connections 2024-03-30T04:33:52.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.751+0000 7f2b80e00700 1 --2- 172.21.15.123:0/424646110 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2b6006cd40 0x7f2b6006f1f0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:52.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.751+0000 7f2b80e00700 1 --2- 172.21.15.123:0/424646110 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b7c0708a0 0x7f2b7c11a0c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:52.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.751+0000 7f2b80e00700 1 --2- 172.21.15.123:0/424646110 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2b7c071f90 0x7f2b7c11a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:52.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.751+0000 7f2b80e00700 1 -- 172.21.15.123:0/424646110 >> 172.21.15.123:0/424646110 conn(0x7f2b7c0f89b0 msgr2=0x7f2b7c0f9b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:33:52.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.751+0000 7f2b80e00700 1 -- 172.21.15.123:0/424646110 shutdown_connections 2024-03-30T04:33:52.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.751+0000 7f2b80e00700 1 -- 172.21.15.123:0/424646110 wait complete. 2024-03-30T04:33:52.763 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:33:52.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.888+0000 7f2bca400700 1 -- 172.21.15.123:0/3422922504 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2bc40fe830 msgr2=0x7f2bc4106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:52.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.888+0000 7f2bca400700 1 --2- 172.21.15.123:0/3422922504 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2bc40fe830 0x7f2bc4106f00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2bb0004940 tx=0x7f2bb0009e30 comp rx=0 tx=0).stop 2024-03-30T04:33:52.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.889+0000 7f2bca400700 1 -- 172.21.15.123:0/3422922504 shutdown_connections 2024-03-30T04:33:52.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.889+0000 7f2bca400700 1 --2- 172.21.15.123:0/3422922504 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2bc40fe830 0x7f2bc4106f00 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:52.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.889+0000 7f2bca400700 1 --2- 172.21.15.123:0/3422922504 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2bc40fde90 0x7f2bc40fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:52.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.889+0000 7f2bca400700 1 -- 172.21.15.123:0/3422922504 >> 172.21.15.123:0/3422922504 conn(0x7f2bc4072bc0 msgr2=0x7f2bc4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:33:52.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.889+0000 7f2bca400700 1 -- 172.21.15.123:0/3422922504 shutdown_connections 2024-03-30T04:33:52.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.889+0000 7f2bca400700 1 -- 172.21.15.123:0/3422922504 wait complete. 2024-03-30T04:33:52.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.889+0000 7f2bca400700 1 Processor -- start 2024-03-30T04:33:52.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.890+0000 7f2bca400700 1 -- start start 2024-03-30T04:33:52.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.890+0000 7f2bca400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2bc40fde90 0x7f2bc40fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:33:52.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.890+0000 7f2bca400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2bc40fe830 0x7f2bc40fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:33:52.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.890+0000 7f2bca400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc40fc820 con 0x7f2bc40fe830 2024-03-30T04:33:52.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.890+0000 7f2bca400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc40fc960 con 0x7f2bc40fde90 2024-03-30T04:33:52.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.890+0000 7f2bc2a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2bc40fe830 0x7f2bc40fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:33:52.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.890+0000 7f2bc2a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2bc40fe830 0x7f2bc40fc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39122/0 (socket says 172.21.15.123:39122) 2024-03-30T04:33:52.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.890+0000 7f2bc2a00700 1 -- 172.21.15.123:0/942116825 learned_addr learned my addr 172.21.15.123:0/942116825 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:33:52.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.890+0000 7f2bc3400700 1 --2- 172.21.15.123:0/942116825 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2bc40fde90 0x7f2bc40fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:33:52.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.891+0000 7f2bc2a00700 1 -- 172.21.15.123:0/942116825 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2bc40fde90 msgr2=0x7f2bc40fbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:52.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.891+0000 7f2bc2a00700 1 --2- 172.21.15.123:0/942116825 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2bc40fde90 0x7f2bc40fbd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:52.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.891+0000 7f2bc2a00700 1 -- 172.21.15.123:0/942116825 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bb0005040 con 0x7f2bc40fe830 2024-03-30T04:33:52.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.891+0000 7f2bc3400700 1 --2- 172.21.15.123:0/942116825 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2bc40fde90 0x7f2bc40fbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:33:52.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.891+0000 7f2bc2a00700 1 --2- 172.21.15.123:0/942116825 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2bc40fe830 0x7f2bc40fc290 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2bb00093b0 tx=0x7f2bb0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:33:52.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.891+0000 7f2bbb400700 1 -- 172.21.15.123:0/942116825 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2bb000e040 con 0x7f2bc40fe830 2024-03-30T04:33:52.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.891+0000 7f2bbb400700 1 -- 172.21.15.123:0/942116825 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2bb0004290 con 0x7f2bc40fe830 2024-03-30T04:33:52.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.891+0000 7f2bca400700 1 -- 172.21.15.123:0/942116825 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bc40fcbb0 con 0x7f2bc40fe830 2024-03-30T04:33:52.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.891+0000 7f2bbb400700 1 -- 172.21.15.123:0/942116825 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2bb001c3f0 con 0x7f2bc40fe830 2024-03-30T04:33:52.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.891+0000 7f2bca400700 1 -- 172.21.15.123:0/942116825 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bc411a190 con 0x7f2bc40fe830 2024-03-30T04:33:52.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.892+0000 7f2bca400700 1 -- 172.21.15.123:0/942116825 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bc4065e30 con 0x7f2bc40fe830 2024-03-30T04:33:52.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.893+0000 7f2bbb400700 1 -- 172.21.15.123:0/942116825 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 37) v1 ==== 90946+0+0 (secure 0 0 0) 0x7f2bb0013790 con 0x7f2bc40fe830 2024-03-30T04:33:52.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.894+0000 7f2bbb400700 1 --2- 172.21.15.123:0/942116825 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2ba806cd90 0x7f2ba806f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:33:52.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.894+0000 7f2bbb400700 1 -- 172.21.15.123:0/942116825 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f2bb008d570 con 0x7f2bc40fe830 2024-03-30T04:33:52.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.895+0000 7f2bc3400700 1 --2- 172.21.15.123:0/942116825 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2ba806cd90 0x7f2ba806f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:33:52.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.895+0000 7f2bc3400700 1 --2- 172.21.15.123:0/942116825 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2ba806cd90 0x7f2ba806f240 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2bb4007fd0 tx=0x7f2bb4009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:33:52.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:52.899+0000 7f2bbb400700 1 -- 172.21.15.123:0/942116825 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2bb0057390 con 0x7f2bc40fe830 2024-03-30T04:33:52.956 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:52 smithi123 ceph-mon[139369]: pgmap v44: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:33:52.956 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:52 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:52.956 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:52 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:52.956 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:52 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:33:52.956 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:52 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:33:52.956 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:52 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:52.956 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:52 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:33:52.957 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:52 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:33:52.957 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:52 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:33:52.957 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:52 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:33:52.957 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:52 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:52.957 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:52 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi123"}]: dispatch 2024-03-30T04:33:52.957 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:52 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi123"}]': finished 2024-03-30T04:33:52.957 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:52 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi135"}]: dispatch 2024-03-30T04:33:52.957 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:52 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi135"}]': finished 2024-03-30T04:33:52.957 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:52 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-30T04:33:53.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:52 smithi135 ceph-mon[106011]: pgmap v44: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:33:53.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:52 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:53.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:52 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:53.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:52 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:33:53.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:52 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:33:53.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:52 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:53.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:52 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:33:53.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:52 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:33:53.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:52 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:33:53.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:52 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:33:53.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:52 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:53.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:52 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi123"}]: dispatch 2024-03-30T04:33:53.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:52 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi123"}]': finished 2024-03-30T04:33:53.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:52 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi135"}]: dispatch 2024-03-30T04:33:53.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:52 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi135"}]': finished 2024-03-30T04:33:53.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:52 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-30T04:33:53.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.136+0000 7f2bca400700 1 -- 172.21.15.123:0/942116825 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2bc404bed0 con 0x7f2ba806cd90 2024-03-30T04:33:53.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.138+0000 7f2bbb400700 1 -- 172.21.15.123:0/942116825 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f2bc404bed0 con 0x7f2ba806cd90 2024-03-30T04:33:53.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.140+0000 7f2bca400700 1 -- 172.21.15.123:0/942116825 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2ba806cd90 msgr2=0x7f2ba806f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:53.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.140+0000 7f2bca400700 1 --2- 172.21.15.123:0/942116825 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2ba806cd90 0x7f2ba806f240 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2bb4007fd0 tx=0x7f2bb4009040 comp rx=0 tx=0).stop 2024-03-30T04:33:53.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.140+0000 7f2bca400700 1 -- 172.21.15.123:0/942116825 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2bc40fe830 msgr2=0x7f2bc40fc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:53.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.140+0000 7f2bca400700 1 --2- 172.21.15.123:0/942116825 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2bc40fe830 0x7f2bc40fc290 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2bb00093b0 tx=0x7f2bb0009e30 comp rx=0 tx=0).stop 2024-03-30T04:33:53.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.140+0000 7f2bca400700 1 -- 172.21.15.123:0/942116825 shutdown_connections 2024-03-30T04:33:53.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.140+0000 7f2bca400700 1 --2- 172.21.15.123:0/942116825 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2ba806cd90 0x7f2ba806f240 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:53.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.140+0000 7f2bca400700 1 --2- 172.21.15.123:0/942116825 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2bc40fde90 0x7f2bc40fbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:53.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.140+0000 7f2bca400700 1 --2- 172.21.15.123:0/942116825 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2bc40fe830 0x7f2bc40fc290 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:53.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.140+0000 7f2bca400700 1 -- 172.21.15.123:0/942116825 >> 172.21.15.123:0/942116825 conn(0x7f2bc4072bc0 msgr2=0x7f2bc4103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:33:53.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.140+0000 7f2bca400700 1 -- 172.21.15.123:0/942116825 shutdown_connections 2024-03-30T04:33:53.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.140+0000 7f2bca400700 1 -- 172.21.15.123:0/942116825 wait complete. 2024-03-30T04:33:53.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.279+0000 7fb634c00700 1 -- 172.21.15.123:0/3782084347 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb630071f90 msgr2=0x7fb630070360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:53.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.279+0000 7fb634c00700 1 --2- 172.21.15.123:0/3782084347 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb630071f90 0x7fb630070360 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb620004970 tx=0x7fb620009e30 comp rx=0 tx=0).stop 2024-03-30T04:33:53.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.279+0000 7fb634c00700 1 -- 172.21.15.123:0/3782084347 shutdown_connections 2024-03-30T04:33:53.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.280+0000 7fb634c00700 1 --2- 172.21.15.123:0/3782084347 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb6300708a0 0x7fb630070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:53.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.280+0000 7fb634c00700 1 --2- 172.21.15.123:0/3782084347 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb630071f90 0x7fb630070360 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:53.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.280+0000 7fb634c00700 1 -- 172.21.15.123:0/3782084347 >> 172.21.15.123:0/3782084347 conn(0x7fb6300f89b0 msgr2=0x7fb6300fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:33:53.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.280+0000 7fb634c00700 1 -- 172.21.15.123:0/3782084347 shutdown_connections 2024-03-30T04:33:53.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.280+0000 7fb634c00700 1 -- 172.21.15.123:0/3782084347 wait complete. 2024-03-30T04:33:53.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.281+0000 7fb634c00700 1 Processor -- start 2024-03-30T04:33:53.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.281+0000 7fb634c00700 1 -- start start 2024-03-30T04:33:53.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.282+0000 7fb634c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6300708a0 0x7fb63011a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:33:53.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.282+0000 7fb634c00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb630071f90 0x7fb63011a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:33:53.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.282+0000 7fb634c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb63011ac50 con 0x7fb6300708a0 2024-03-30T04:33:53.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.282+0000 7fb634c00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb630114020 con 0x7fb630071f90 2024-03-30T04:33:53.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.282+0000 7fb62de00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6300708a0 0x7fb63011a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:33:53.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.282+0000 7fb62de00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6300708a0 0x7fb63011a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39138/0 (socket says 172.21.15.123:39138) 2024-03-30T04:33:53.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.282+0000 7fb62de00700 1 -- 172.21.15.123:0/2828044593 learned_addr learned my addr 172.21.15.123:0/2828044593 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:33:53.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.282+0000 7fb62d400700 1 --2- 172.21.15.123:0/2828044593 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb630071f90 0x7fb63011a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:33:53.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.282+0000 7fb62de00700 1 -- 172.21.15.123:0/2828044593 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb630071f90 msgr2=0x7fb63011a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:53.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.282+0000 7fb62de00700 1 --2- 172.21.15.123:0/2828044593 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb630071f90 0x7fb63011a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:53.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.282+0000 7fb62de00700 1 -- 172.21.15.123:0/2828044593 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb620005040 con 0x7fb6300708a0 2024-03-30T04:33:53.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.283+0000 7fb62de00700 1 --2- 172.21.15.123:0/2828044593 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6300708a0 0x7fb63011a0c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb620004f50 tx=0x7fb620009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:33:53.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.283+0000 7fb626000700 1 -- 172.21.15.123:0/2828044593 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb62000e040 con 0x7fb6300708a0 2024-03-30T04:33:53.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.283+0000 7fb634c00700 1 -- 172.21.15.123:0/2828044593 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6301142a0 con 0x7fb6300708a0 2024-03-30T04:33:53.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.283+0000 7fb626000700 1 -- 172.21.15.123:0/2828044593 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb620013420 con 0x7fb6300708a0 2024-03-30T04:33:53.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.283+0000 7fb626000700 1 -- 172.21.15.123:0/2828044593 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb62001c610 con 0x7fb6300708a0 2024-03-30T04:33:53.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.283+0000 7fb634c00700 1 -- 172.21.15.123:0/2828044593 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6301147e0 con 0x7fb6300708a0 2024-03-30T04:33:53.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.284+0000 7fb634c00700 1 -- 172.21.15.123:0/2828044593 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb630071aa0 con 0x7fb6300708a0 2024-03-30T04:33:53.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.285+0000 7fb626000700 1 -- 172.21.15.123:0/2828044593 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 37) v1 ==== 90946+0+0 (secure 0 0 0) 0x7fb620013590 con 0x7fb6300708a0 2024-03-30T04:33:53.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.286+0000 7fb626000700 1 --2- 172.21.15.123:0/2828044593 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb61406cd40 0x7fb61406f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:33:53.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.286+0000 7fb626000700 1 -- 172.21.15.123:0/2828044593 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fb62008d480 con 0x7fb6300708a0 2024-03-30T04:33:53.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.286+0000 7fb62d400700 1 --2- 172.21.15.123:0/2828044593 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb61406cd40 0x7fb61406f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:33:53.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.287+0000 7fb62d400700 1 --2- 172.21.15.123:0/2828044593 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb61406cd40 0x7fb61406f1f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb630115960 tx=0x7fb61c009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:33:53.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.294+0000 7fb626000700 1 -- 172.21.15.123:0/2828044593 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb62005ae80 con 0x7fb6300708a0 2024-03-30T04:33:53.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.517+0000 7fb634c00700 1 -- 172.21.15.123:0/2828044593 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb63004bed0 con 0x7fb61406cd40 2024-03-30T04:33:53.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.527+0000 7fb626000700 1 -- 172.21.15.123:0/2828044593 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fb63004bed0 con 0x7fb61406cd40 2024-03-30T04:33:53.525 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:33:53.526 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (3m) 6s ago 15m 14.9M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:33:53.526 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (19s) 6s ago 15m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T04:33:53.526 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (13s) 9s ago 14m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T04:33:53.526 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (2m) 6s ago 15m 74.3M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T04:33:53.526 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (9m) 6s ago 9m 109M - 17.2.7-797-g5c024335 f77085f6aab7 2df066fba7a6 2024-03-30T04:33:53.526 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (9m) 6s ago 9m 57.9M - 17.2.7-797-g5c024335 f77085f6aab7 1293b8520db1 2024-03-30T04:33:53.526 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (9m) 9s ago 9m 67.4M - 17.2.7-797-g5c024335 f77085f6aab7 0dc41326818c 2024-03-30T04:33:53.526 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (9m) 9s ago 9m 119M - 17.2.7-797-g5c024335 f77085f6aab7 aef8a0cfc553 2024-03-30T04:33:53.526 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (5m) 6s ago 17m 493M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:33:53.526 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (4m) 9s ago 14m 448M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:33:53.526 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (94s) 6s ago 17m 40.7M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T04:33:53.526 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (64s) 9s ago 14m 33.2M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T04:33:53.526 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (3m) 6s ago 15m 7331k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:33:53.526 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (3m) 9s ago 14m 7512k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:33:53.527 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (13m) 6s ago 13m 221M 1618M 17.2.7-797-g5c024335 f77085f6aab7 57a7ba3fd75e 2024-03-30T04:33:53.527 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (13m) 6s ago 12m 235M 1618M 17.2.7-797-g5c024335 f77085f6aab7 e1a6c1818a0c 2024-03-30T04:33:53.527 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (12m) 6s ago 12m 189M 1618M 17.2.7-797-g5c024335 f77085f6aab7 4c6a727567bd 2024-03-30T04:33:53.527 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (11m) 9s ago 11m 259M 2642M 17.2.7-797-g5c024335 f77085f6aab7 6fbdd1d46a9d 2024-03-30T04:33:53.527 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (11m) 9s ago 11m 227M 2642M 17.2.7-797-g5c024335 f77085f6aab7 b74988e947e5 2024-03-30T04:33:53.527 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (10m) 9s ago 10m 220M 2642M 17.2.7-797-g5c024335 f77085f6aab7 1fd3185f5484 2024-03-30T04:33:53.527 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (3m) 6s ago 15m 50.5M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:33:53.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.529+0000 7fb634c00700 1 -- 172.21.15.123:0/2828044593 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb61406cd40 msgr2=0x7fb61406f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:53.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.529+0000 7fb634c00700 1 --2- 172.21.15.123:0/2828044593 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb61406cd40 0x7fb61406f1f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb630115960 tx=0x7fb61c009ee0 comp rx=0 tx=0).stop 2024-03-30T04:33:53.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.529+0000 7fb634c00700 1 -- 172.21.15.123:0/2828044593 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6300708a0 msgr2=0x7fb63011a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:53.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.529+0000 7fb634c00700 1 --2- 172.21.15.123:0/2828044593 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6300708a0 0x7fb63011a0c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb620004f50 tx=0x7fb620009e30 comp rx=0 tx=0).stop 2024-03-30T04:33:53.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.529+0000 7fb634c00700 1 -- 172.21.15.123:0/2828044593 shutdown_connections 2024-03-30T04:33:53.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.529+0000 7fb634c00700 1 --2- 172.21.15.123:0/2828044593 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb61406cd40 0x7fb61406f1f0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:53.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.529+0000 7fb634c00700 1 --2- 172.21.15.123:0/2828044593 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6300708a0 0x7fb63011a0c0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:53.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.529+0000 7fb634c00700 1 --2- 172.21.15.123:0/2828044593 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb630071f90 0x7fb63011a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:53.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.529+0000 7fb634c00700 1 -- 172.21.15.123:0/2828044593 >> 172.21.15.123:0/2828044593 conn(0x7fb6300f89b0 msgr2=0x7fb6300f9b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:33:53.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.529+0000 7fb634c00700 1 -- 172.21.15.123:0/2828044593 shutdown_connections 2024-03-30T04:33:53.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.530+0000 7fb634c00700 1 -- 172.21.15.123:0/2828044593 wait complete. 2024-03-30T04:33:53.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.663+0000 7f2f48e00700 1 -- 172.21.15.123:0/2565147372 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f44070300 msgr2=0x7f2f440706d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:53.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.663+0000 7f2f48e00700 1 --2- 172.21.15.123:0/2565147372 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f44070300 0x7f2f440706d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2f34004970 tx=0x7f2f34009e30 comp rx=0 tx=0).stop 2024-03-30T04:33:53.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.663+0000 7f2f48e00700 1 -- 172.21.15.123:0/2565147372 shutdown_connections 2024-03-30T04:33:53.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.663+0000 7f2f48e00700 1 --2- 172.21.15.123:0/2565147372 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2f44070c10 0x7f2f44108fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:53.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.663+0000 7f2f48e00700 1 --2- 172.21.15.123:0/2565147372 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f44070300 0x7f2f440706d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:53.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.663+0000 7f2f48e00700 1 -- 172.21.15.123:0/2565147372 >> 172.21.15.123:0/2565147372 conn(0x7f2f440f87e0 msgr2=0x7f2f440fabf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:33:53.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.663+0000 7f2f48e00700 1 -- 172.21.15.123:0/2565147372 shutdown_connections 2024-03-30T04:33:53.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.663+0000 7f2f48e00700 1 -- 172.21.15.123:0/2565147372 wait complete. 2024-03-30T04:33:53.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.664+0000 7f2f48e00700 1 Processor -- start 2024-03-30T04:33:53.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.664+0000 7f2f48e00700 1 -- start start 2024-03-30T04:33:53.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.664+0000 7f2f48e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2f44070300 0x7f2f44115b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:33:53.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.664+0000 7f2f48e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f44070c10 0x7f2f44116090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:33:53.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.664+0000 7f2f48e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f441166e0 con 0x7f2f44070c10 2024-03-30T04:33:53.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.664+0000 7f2f48e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4410fad0 con 0x7f2f44070300 2024-03-30T04:33:53.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.665+0000 7f2f42a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f44070c10 0x7f2f44116090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:33:53.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.665+0000 7f2f42a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f44070c10 0x7f2f44116090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39150/0 (socket says 172.21.15.123:39150) 2024-03-30T04:33:53.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.665+0000 7f2f42a00700 1 -- 172.21.15.123:0/2643524011 learned_addr learned my addr 172.21.15.123:0/2643524011 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:33:53.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.665+0000 7f2f43400700 1 --2- 172.21.15.123:0/2643524011 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2f44070300 0x7f2f44115b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:33:53.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.665+0000 7f2f42a00700 1 -- 172.21.15.123:0/2643524011 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2f44070300 msgr2=0x7f2f44115b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:53.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.665+0000 7f2f42a00700 1 --2- 172.21.15.123:0/2643524011 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2f44070300 0x7f2f44115b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:53.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.665+0000 7f2f42a00700 1 -- 172.21.15.123:0/2643524011 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f34005040 con 0x7f2f44070c10 2024-03-30T04:33:53.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.666+0000 7f2f42a00700 1 --2- 172.21.15.123:0/2643524011 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f44070c10 0x7f2f44116090 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2f30002a50 tx=0x7f2f300046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:33:53.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.666+0000 7f2f3b400700 1 -- 172.21.15.123:0/2643524011 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f30007cc0 con 0x7f2f44070c10 2024-03-30T04:33:53.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.666+0000 7f2f3b400700 1 -- 172.21.15.123:0/2643524011 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2f3000f040 con 0x7f2f44070c10 2024-03-30T04:33:53.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.666+0000 7f2f48e00700 1 -- 172.21.15.123:0/2643524011 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f4410fdb0 con 0x7f2f44070c10 2024-03-30T04:33:53.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.666+0000 7f2f3b400700 1 -- 172.21.15.123:0/2643524011 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f3000d9c0 con 0x7f2f44070c10 2024-03-30T04:33:53.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.666+0000 7f2f48e00700 1 -- 172.21.15.123:0/2643524011 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f441102f0 con 0x7f2f44070c10 2024-03-30T04:33:53.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.667+0000 7f2f48e00700 1 -- 172.21.15.123:0/2643524011 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f44071a40 con 0x7f2f44070c10 2024-03-30T04:33:53.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.668+0000 7f2f3b400700 1 -- 172.21.15.123:0/2643524011 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 37) v1 ==== 90946+0+0 (secure 0 0 0) 0x7f2f3000db20 con 0x7f2f44070c10 2024-03-30T04:33:53.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.669+0000 7f2f3b400700 1 --2- 172.21.15.123:0/2643524011 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2f2806cd40 0x7f2f2806f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:33:53.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.669+0000 7f2f3b400700 1 -- 172.21.15.123:0/2643524011 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f2f3008bd50 con 0x7f2f44070c10 2024-03-30T04:33:53.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.669+0000 7f2f43400700 1 --2- 172.21.15.123:0/2643524011 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2f2806cd40 0x7f2f2806f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:33:53.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.670+0000 7f2f43400700 1 --2- 172.21.15.123:0/2643524011 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2f2806cd40 0x7f2f2806f1f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2f3400b370 tx=0x7f2f34009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:33:53.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.673+0000 7f2f3b400700 1 -- 172.21.15.123:0/2643524011 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2f30059580 con 0x7f2f44070c10 2024-03-30T04:33:53.915 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:53 smithi123 ceph-mon[139369]: Upgrade: Setting container_image for all crash 2024-03-30T04:33:53.915 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:53 smithi123 ceph-mon[139369]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-30T04:33:53.916 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:53 smithi123 ceph-mon[139369]: Upgrade: osd.0 is safe to restart 2024-03-30T04:33:53.916 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:53 smithi123 ceph-mon[139369]: from='client.34188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:33:53.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.968+0000 7f2f48e00700 1 -- 172.21.15.123:0/2643524011 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f2f441111d0 con 0x7f2f44070c10 2024-03-30T04:33:53.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.968+0000 7f2f3b400700 1 -- 172.21.15.123:0/2643524011 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+742 (secure 0 0 0) 0x7f2f30059110 con 0x7f2f44070c10 2024-03-30T04:33:53.967 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:33:53.967 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:33:53.967 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:33:53.967 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:33:53.967 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:33:53.968 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:33:53.968 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:33:53.968 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:33:53.968 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 6 2024-03-30T04:33:53.968 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:33:53.968 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:33:53.968 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 4 2024-03-30T04:33:53.968 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:33:53.968 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:33:53.968 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 10, 2024-03-30T04:33:53.968 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 4 2024-03-30T04:33:53.968 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:33:53.968 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:33:53.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.971+0000 7f2f48e00700 1 -- 172.21.15.123:0/2643524011 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2f2806cd40 msgr2=0x7f2f2806f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:53.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.971+0000 7f2f48e00700 1 --2- 172.21.15.123:0/2643524011 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2f2806cd40 0x7f2f2806f1f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2f3400b370 tx=0x7f2f34009e30 comp rx=0 tx=0).stop 2024-03-30T04:33:53.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.971+0000 7f2f48e00700 1 -- 172.21.15.123:0/2643524011 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f44070c10 msgr2=0x7f2f44116090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:53.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.971+0000 7f2f48e00700 1 --2- 172.21.15.123:0/2643524011 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f44070c10 0x7f2f44116090 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2f30002a50 tx=0x7f2f300046b0 comp rx=0 tx=0).stop 2024-03-30T04:33:53.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.971+0000 7f2f48e00700 1 -- 172.21.15.123:0/2643524011 shutdown_connections 2024-03-30T04:33:53.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.971+0000 7f2f48e00700 1 --2- 172.21.15.123:0/2643524011 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2f2806cd40 0x7f2f2806f1f0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:53.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.971+0000 7f2f48e00700 1 --2- 172.21.15.123:0/2643524011 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2f44070300 0x7f2f44115b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:53.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.971+0000 7f2f48e00700 1 --2- 172.21.15.123:0/2643524011 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f44070c10 0x7f2f44116090 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:53.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.971+0000 7f2f48e00700 1 -- 172.21.15.123:0/2643524011 >> 172.21.15.123:0/2643524011 conn(0x7f2f440f87e0 msgr2=0x7f2f440f9d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:33:53.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.971+0000 7f2f48e00700 1 -- 172.21.15.123:0/2643524011 shutdown_connections 2024-03-30T04:33:53.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:53.971+0000 7f2f48e00700 1 -- 172.21.15.123:0/2643524011 wait complete. 2024-03-30T04:33:54.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:53 smithi135 ceph-mon[106011]: Upgrade: Setting container_image for all crash 2024-03-30T04:33:54.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:53 smithi135 ceph-mon[106011]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-30T04:33:54.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:53 smithi135 ceph-mon[106011]: Upgrade: osd.0 is safe to restart 2024-03-30T04:33:54.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:53 smithi135 ceph-mon[106011]: from='client.34188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:33:54.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.118+0000 7fac96600700 1 -- 172.21.15.123:0/1491679822 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac900708a0 msgr2=0x7fac90070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:54.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.118+0000 7fac96600700 1 --2- 172.21.15.123:0/1491679822 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac900708a0 0x7fac90070d10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fac7c004970 tx=0x7fac7c009e30 comp rx=0 tx=0).stop 2024-03-30T04:33:54.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.118+0000 7fac96600700 1 -- 172.21.15.123:0/1491679822 shutdown_connections 2024-03-30T04:33:54.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.118+0000 7fac96600700 1 --2- 172.21.15.123:0/1491679822 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac900708a0 0x7fac90070d10 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:54.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.118+0000 7fac96600700 1 --2- 172.21.15.123:0/1491679822 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac90071f90 0x7fac90070360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:54.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.118+0000 7fac96600700 1 -- 172.21.15.123:0/1491679822 >> 172.21.15.123:0/1491679822 conn(0x7fac900f89b0 msgr2=0x7fac900fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:33:54.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.118+0000 7fac96600700 1 -- 172.21.15.123:0/1491679822 shutdown_connections 2024-03-30T04:33:54.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.118+0000 7fac96600700 1 -- 172.21.15.123:0/1491679822 wait complete. 2024-03-30T04:33:54.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.119+0000 7fac96600700 1 Processor -- start 2024-03-30T04:33:54.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.119+0000 7fac96600700 1 -- start start 2024-03-30T04:33:54.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.120+0000 7fac96600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac900708a0 0x7fac9011a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:33:54.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.120+0000 7fac96600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac90071f90 0x7fac9011a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:33:54.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.120+0000 7fac96600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac9011ac40 con 0x7fac900708a0 2024-03-30T04:33:54.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.120+0000 7fac96600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac90114010 con 0x7fac90071f90 2024-03-30T04:33:54.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.120+0000 7fac8f400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac900708a0 0x7fac9011a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:33:54.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.120+0000 7fac8f400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac900708a0 0x7fac9011a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45038/0 (socket says 172.21.15.123:45038) 2024-03-30T04:33:54.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.120+0000 7fac8f400700 1 -- 172.21.15.123:0/580758649 learned_addr learned my addr 172.21.15.123:0/580758649 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:33:54.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.120+0000 7fac8ea00700 1 --2- 172.21.15.123:0/580758649 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac90071f90 0x7fac9011a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:33:54.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.120+0000 7fac8f400700 1 -- 172.21.15.123:0/580758649 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac90071f90 msgr2=0x7fac9011a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:54.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.120+0000 7fac8f400700 1 --2- 172.21.15.123:0/580758649 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac90071f90 0x7fac9011a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:54.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.120+0000 7fac8f400700 1 -- 172.21.15.123:0/580758649 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac7c005040 con 0x7fac900708a0 2024-03-30T04:33:54.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.121+0000 7fac8f400700 1 --2- 172.21.15.123:0/580758649 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac900708a0 0x7fac9011a0b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fac8000eb40 tx=0x7fac8000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:33:54.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.121+0000 7fac87400700 1 -- 172.21.15.123:0/580758649 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac80007cc0 con 0x7fac900708a0 2024-03-30T04:33:54.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.121+0000 7fac87400700 1 -- 172.21.15.123:0/580758649 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fac8000f040 con 0x7fac900708a0 2024-03-30T04:33:54.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.121+0000 7fac96600700 1 -- 172.21.15.123:0/580758649 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac90114290 con 0x7fac900708a0 2024-03-30T04:33:54.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.121+0000 7fac87400700 1 -- 172.21.15.123:0/580758649 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac8000c990 con 0x7fac900708a0 2024-03-30T04:33:54.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.121+0000 7fac96600700 1 -- 172.21.15.123:0/580758649 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac901147a0 con 0x7fac900708a0 2024-03-30T04:33:54.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.122+0000 7fac96600700 1 -- 172.21.15.123:0/580758649 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac90071aa0 con 0x7fac900708a0 2024-03-30T04:33:54.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.123+0000 7fac87400700 1 -- 172.21.15.123:0/580758649 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 37) v1 ==== 90946+0+0 (secure 0 0 0) 0x7fac800090d0 con 0x7fac900708a0 2024-03-30T04:33:54.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.124+0000 7fac87400700 1 --2- 172.21.15.123:0/580758649 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fac7406cd40 0x7fac7406f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:33:54.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.124+0000 7fac87400700 1 -- 172.21.15.123:0/580758649 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fac8008be00 con 0x7fac900708a0 2024-03-30T04:33:54.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.125+0000 7fac8ea00700 1 --2- 172.21.15.123:0/580758649 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fac7406cd40 0x7fac7406f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:33:54.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.125+0000 7fac8ea00700 1 --2- 172.21.15.123:0/580758649 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fac7406cd40 0x7fac7406f1f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fac7c009fd0 tx=0x7fac7c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:33:54.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.129+0000 7fac87400700 1 -- 172.21.15.123:0/580758649 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fac80059760 con 0x7fac900708a0 2024-03-30T04:33:54.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.391+0000 7fac96600700 1 -- 172.21.15.123:0/580758649 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fac9004bed0 con 0x7fac900708a0 2024-03-30T04:33:54.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.391+0000 7fac87400700 1 -- 172.21.15.123:0/580758649 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 23 v23) v1 ==== 76+0+1876 (secure 0 0 0) 0x7fac80019770 con 0x7fac900708a0 2024-03-30T04:33:54.390 INFO:teuthology.orchestra.run.smithi123.stdout:e23 2024-03-30T04:33:54.390 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:33:54.390 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:33:54.390 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:33:54.390 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:33:54.391 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:33:54.391 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:33:54.391 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 23 2024-03-30T04:33:54.391 INFO:teuthology.orchestra.run.smithi123.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-30T04:33:54.391 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:33:54.391 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:32:48.448604+0000 2024-03-30T04:33:54.391 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:33:54.391 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:33:54.391 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:33:54.391 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:33:54.391 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:33:54.391 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T04:33:54.391 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:33:54.391 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:33:54.392 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 46 2024-03-30T04:33:54.392 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:33:54.392 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T04:33:54.392 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T04:33:54.392 INFO:teuthology.orchestra.run.smithi123.stdout:up {0=14488,1=24309} 2024-03-30T04:33:54.392 INFO:teuthology.orchestra.run.smithi123.stdout:failed 2024-03-30T04:33:54.392 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:33:54.392 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:33:54.392 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:33:54.392 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:33:54.392 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:33:54.392 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:33:54.392 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T04:33:54.392 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T04:33:54.393 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{0:14488} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.123:6828/3037849622,v1:172.21.15.123:6829/3037849622] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:33:54.393 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{0:14500} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/2433675995,v1:172.21.15.123:6827/2433675995] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:33:54.393 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{1:24309} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.135:6824/4050541062,v1:172.21.15.135:6825/4050541062] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:33:54.393 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{1:24319} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.135:6826/1096986122,v1:172.21.15.135:6827/1096986122] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:33:54.393 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:33:54.393 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:33:54.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.393+0000 7fac96600700 1 -- 172.21.15.123:0/580758649 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fac7406cd40 msgr2=0x7fac7406f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:54.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.393+0000 7fac96600700 1 --2- 172.21.15.123:0/580758649 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fac7406cd40 0x7fac7406f1f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fac7c009fd0 tx=0x7fac7c009e30 comp rx=0 tx=0).stop 2024-03-30T04:33:54.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.393+0000 7fac96600700 1 -- 172.21.15.123:0/580758649 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac900708a0 msgr2=0x7fac9011a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:54.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.393+0000 7fac96600700 1 --2- 172.21.15.123:0/580758649 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac900708a0 0x7fac9011a0b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fac8000eb40 tx=0x7fac8000b040 comp rx=0 tx=0).stop 2024-03-30T04:33:54.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.394+0000 7fac96600700 1 -- 172.21.15.123:0/580758649 shutdown_connections 2024-03-30T04:33:54.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.394+0000 7fac96600700 1 --2- 172.21.15.123:0/580758649 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fac7406cd40 0x7fac7406f1f0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:54.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.394+0000 7fac96600700 1 --2- 172.21.15.123:0/580758649 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac900708a0 0x7fac9011a0b0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:54.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.394+0000 7fac96600700 1 --2- 172.21.15.123:0/580758649 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac90071f90 0x7fac9011a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:54.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.394+0000 7fac96600700 1 -- 172.21.15.123:0/580758649 >> 172.21.15.123:0/580758649 conn(0x7fac900f89b0 msgr2=0x7fac900f9b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:33:54.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.394+0000 7fac96600700 1 -- 172.21.15.123:0/580758649 shutdown_connections 2024-03-30T04:33:54.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.394+0000 7fac96600700 1 -- 172.21.15.123:0/580758649 wait complete. 2024-03-30T04:33:54.394 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 23 2024-03-30T04:33:54.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.532+0000 7fd2e4e00700 1 -- 172.21.15.123:0/1279340642 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2e01059b0 msgr2=0x7fd2e00ff020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:54.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.532+0000 7fd2e4e00700 1 --2- 172.21.15.123:0/1279340642 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2e01059b0 0x7fd2e00ff020 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc004940 tx=0x7fd2cc009e30 comp rx=0 tx=0).stop 2024-03-30T04:33:54.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.532+0000 7fd2e4e00700 1 -- 172.21.15.123:0/1279340642 shutdown_connections 2024-03-30T04:33:54.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.532+0000 7fd2e4e00700 1 --2- 172.21.15.123:0/1279340642 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2e01059b0 0x7fd2e00ff020 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:54.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.532+0000 7fd2e4e00700 1 --2- 172.21.15.123:0/1279340642 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2e0105010 0x7fd2e01053e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:54.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.532+0000 7fd2e4e00700 1 -- 172.21.15.123:0/1279340642 >> 172.21.15.123:0/1279340642 conn(0x7fd2e00faba0 msgr2=0x7fd2e00fcfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:33:54.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.532+0000 7fd2e4e00700 1 -- 172.21.15.123:0/1279340642 shutdown_connections 2024-03-30T04:33:54.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.532+0000 7fd2e4e00700 1 -- 172.21.15.123:0/1279340642 wait complete. 2024-03-30T04:33:54.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.533+0000 7fd2e4e00700 1 Processor -- start 2024-03-30T04:33:54.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.533+0000 7fd2e4e00700 1 -- start start 2024-03-30T04:33:54.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.533+0000 7fd2e4e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2e0105010 0x7fd2e0115cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:33:54.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.533+0000 7fd2e4e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2e01059b0 0x7fd2e01161f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:33:54.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.533+0000 7fd2e4e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e0116840 con 0x7fd2e01059b0 2024-03-30T04:33:54.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.533+0000 7fd2e4e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e010fc60 con 0x7fd2e0105010 2024-03-30T04:33:54.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.534+0000 7fd2dd400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2e01059b0 0x7fd2e01161f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:33:54.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.534+0000 7fd2dd400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2e01059b0 0x7fd2e01161f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45044/0 (socket says 172.21.15.123:45044) 2024-03-30T04:33:54.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.534+0000 7fd2dd400700 1 -- 172.21.15.123:0/1420626643 learned_addr learned my addr 172.21.15.123:0/1420626643 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:33:54.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.534+0000 7fd2dde00700 1 --2- 172.21.15.123:0/1420626643 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2e0105010 0x7fd2e0115cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:33:54.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.534+0000 7fd2dd400700 1 -- 172.21.15.123:0/1420626643 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2e0105010 msgr2=0x7fd2e0115cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:54.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.534+0000 7fd2dd400700 1 --2- 172.21.15.123:0/1420626643 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2e0105010 0x7fd2e0115cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:54.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.534+0000 7fd2dd400700 1 -- 172.21.15.123:0/1420626643 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2cc005040 con 0x7fd2e01059b0 2024-03-30T04:33:54.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.534+0000 7fd2dde00700 1 --2- 172.21.15.123:0/1420626643 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2e0105010 0x7fd2e0115cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:33:54.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.534+0000 7fd2dd400700 1 --2- 172.21.15.123:0/1420626643 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2e01059b0 0x7fd2e01161f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc009390 tx=0x7fd2cc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:33:54.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.534+0000 7fd2d6000700 1 -- 172.21.15.123:0/1420626643 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd2cc00e040 con 0x7fd2e01059b0 2024-03-30T04:33:54.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.535+0000 7fd2e4e00700 1 -- 172.21.15.123:0/1420626643 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2e010fee0 con 0x7fd2e01059b0 2024-03-30T04:33:54.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.535+0000 7fd2d6000700 1 -- 172.21.15.123:0/1420626643 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd2cc013420 con 0x7fd2e01059b0 2024-03-30T04:33:54.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.535+0000 7fd2d6000700 1 -- 172.21.15.123:0/1420626643 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd2cc01c630 con 0x7fd2e01059b0 2024-03-30T04:33:54.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.535+0000 7fd2e4e00700 1 -- 172.21.15.123:0/1420626643 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2e01104a0 con 0x7fd2e01059b0 2024-03-30T04:33:54.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.536+0000 7fd2e4e00700 1 -- 172.21.15.123:0/1420626643 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2e0102320 con 0x7fd2e01059b0 2024-03-30T04:33:54.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.537+0000 7fd2d6000700 1 -- 172.21.15.123:0/1420626643 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 37) v1 ==== 90946+0+0 (secure 0 0 0) 0x7fd2cc013a00 con 0x7fd2e01059b0 2024-03-30T04:33:54.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.537+0000 7fd2d6000700 1 --2- 172.21.15.123:0/1420626643 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd2c406cd40 0x7fd2c406f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:33:54.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.538+0000 7fd2d6000700 1 -- 172.21.15.123:0/1420626643 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fd2cc08d5a0 con 0x7fd2e01059b0 2024-03-30T04:33:54.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.538+0000 7fd2dde00700 1 --2- 172.21.15.123:0/1420626643 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd2c406cd40 0x7fd2c406f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:33:54.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.538+0000 7fd2dde00700 1 --2- 172.21.15.123:0/1420626643 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd2c406cd40 0x7fd2c406f1f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd2d0007fd0 tx=0x7fd2d0009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:33:54.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.542+0000 7fd2d6000700 1 -- 172.21.15.123:0/1420626643 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd2cc05ae50 con 0x7fd2e01059b0 2024-03-30T04:33:54.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.779+0000 7fd2e4e00700 1 -- 172.21.15.123:0/1420626643 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd2e0111370 con 0x7fd2c406cd40 2024-03-30T04:33:54.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.780+0000 7fd2d6000700 1 -- 172.21.15.123:0/1420626643 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fd2e0111370 con 0x7fd2c406cd40 2024-03-30T04:33:54.779 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:33:54.779 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:33:54.779 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:33:54.779 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T04:33:54.779 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:33:54.779 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T04:33:54.779 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T04:33:54.779 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T04:33:54.779 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:33:54.779 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "6/21 daemons upgraded", 2024-03-30T04:33:54.779 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading osd daemons", 2024-03-30T04:33:54.779 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:33:54.779 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:33:54.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.782+0000 7fd2e4e00700 1 -- 172.21.15.123:0/1420626643 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd2c406cd40 msgr2=0x7fd2c406f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:54.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.782+0000 7fd2e4e00700 1 --2- 172.21.15.123:0/1420626643 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd2c406cd40 0x7fd2c406f1f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd2d0007fd0 tx=0x7fd2d0009040 comp rx=0 tx=0).stop 2024-03-30T04:33:54.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.782+0000 7fd2e4e00700 1 -- 172.21.15.123:0/1420626643 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2e01059b0 msgr2=0x7fd2e01161f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:54.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.782+0000 7fd2e4e00700 1 --2- 172.21.15.123:0/1420626643 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2e01059b0 0x7fd2e01161f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc009390 tx=0x7fd2cc009e30 comp rx=0 tx=0).stop 2024-03-30T04:33:54.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.783+0000 7fd2e4e00700 1 -- 172.21.15.123:0/1420626643 shutdown_connections 2024-03-30T04:33:54.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.783+0000 7fd2e4e00700 1 --2- 172.21.15.123:0/1420626643 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd2c406cd40 0x7fd2c406f1f0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:54.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.783+0000 7fd2e4e00700 1 --2- 172.21.15.123:0/1420626643 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2e0105010 0x7fd2e0115cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:54.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.783+0000 7fd2e4e00700 1 --2- 172.21.15.123:0/1420626643 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2e01059b0 0x7fd2e01161f0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:54.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.783+0000 7fd2e4e00700 1 -- 172.21.15.123:0/1420626643 >> 172.21.15.123:0/1420626643 conn(0x7fd2e00faba0 msgr2=0x7fd2e0108640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:33:54.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.783+0000 7fd2e4e00700 1 -- 172.21.15.123:0/1420626643 shutdown_connections 2024-03-30T04:33:54.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.783+0000 7fd2e4e00700 1 -- 172.21.15.123:0/1420626643 wait complete. 2024-03-30T04:33:54.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.920+0000 7f7f2a200700 1 -- 172.21.15.123:0/1835264830 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f24102e00 msgr2=0x7f7f241031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:54.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.920+0000 7f7f2a200700 1 --2- 172.21.15.123:0/1835264830 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f24102e00 0x7f7f241031d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7f14004940 tx=0x7f7f14009e30 comp rx=0 tx=0).stop 2024-03-30T04:33:54.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.920+0000 7f7f2a200700 1 -- 172.21.15.123:0/1835264830 shutdown_connections 2024-03-30T04:33:54.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.920+0000 7f7f2a200700 1 --2- 172.21.15.123:0/1835264830 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f241037a0 0x7f7f240fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:54.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.920+0000 7f7f2a200700 1 --2- 172.21.15.123:0/1835264830 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f24102e00 0x7f7f241031d0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:54.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.920+0000 7f7f2a200700 1 -- 172.21.15.123:0/1835264830 >> 172.21.15.123:0/1835264830 conn(0x7f7f240f89b0 msgr2=0x7f7f240fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:33:54.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.920+0000 7f7f2a200700 1 -- 172.21.15.123:0/1835264830 shutdown_connections 2024-03-30T04:33:54.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.920+0000 7f7f2a200700 1 -- 172.21.15.123:0/1835264830 wait complete. 2024-03-30T04:33:54.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.921+0000 7f7f2a200700 1 Processor -- start 2024-03-30T04:33:54.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.922+0000 7f7f2a200700 1 -- start start 2024-03-30T04:33:54.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.922+0000 7f7f2a200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f24102e00 0x7f7f24115aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:33:54.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.922+0000 7f7f2a200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f241037a0 0x7f7f24115fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:33:54.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.922+0000 7f7f2a200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f24116630 con 0x7f7f241037a0 2024-03-30T04:33:54.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.922+0000 7f7f2a200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2410fc10 con 0x7f7f24102e00 2024-03-30T04:33:54.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.923+0000 7f7f22a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f241037a0 0x7f7f24115fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:33:54.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.923+0000 7f7f22a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f241037a0 0x7f7f24115fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45056/0 (socket says 172.21.15.123:45056) 2024-03-30T04:33:54.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.923+0000 7f7f22a00700 1 -- 172.21.15.123:0/1876265781 learned_addr learned my addr 172.21.15.123:0/1876265781 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:33:54.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.923+0000 7f7f23400700 1 --2- 172.21.15.123:0/1876265781 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f24102e00 0x7f7f24115aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:33:54.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.923+0000 7f7f22a00700 1 -- 172.21.15.123:0/1876265781 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f24102e00 msgr2=0x7f7f24115aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:54.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.923+0000 7f7f22a00700 1 --2- 172.21.15.123:0/1876265781 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f24102e00 0x7f7f24115aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:54.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.923+0000 7f7f22a00700 1 -- 172.21.15.123:0/1876265781 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f14005040 con 0x7f7f241037a0 2024-03-30T04:33:54.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.923+0000 7f7f22a00700 1 --2- 172.21.15.123:0/1876265781 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f241037a0 0x7f7f24115fe0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7f10002a50 tx=0x7f7f10004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:33:54.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.923+0000 7f7f1b400700 1 -- 172.21.15.123:0/1876265781 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f10007cc0 con 0x7f7f241037a0 2024-03-30T04:33:54.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.923+0000 7f7f1b400700 1 -- 172.21.15.123:0/1876265781 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7f1000f040 con 0x7f7f241037a0 2024-03-30T04:33:54.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.923+0000 7f7f1b400700 1 -- 172.21.15.123:0/1876265781 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f1000d9a0 con 0x7f7f241037a0 2024-03-30T04:33:54.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.923+0000 7f7f2a200700 1 -- 172.21.15.123:0/1876265781 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f2410fef0 con 0x7f7f241037a0 2024-03-30T04:33:54.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.924+0000 7f7f2a200700 1 -- 172.21.15.123:0/1876265781 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f241104b0 con 0x7f7f241037a0 2024-03-30T04:33:54.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.924+0000 7f7f2a200700 1 -- 172.21.15.123:0/1876265781 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f240fe550 con 0x7f7f241037a0 2024-03-30T04:33:54.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.925+0000 7f7f1b400700 1 -- 172.21.15.123:0/1876265781 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 37) v1 ==== 90946+0+0 (secure 0 0 0) 0x7f7f1000dc60 con 0x7f7f241037a0 2024-03-30T04:33:54.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.926+0000 7f7f1b400700 1 --2- 172.21.15.123:0/1876265781 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7f0806cd50 0x7f7f0806f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:33:54.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.926+0000 7f7f1b400700 1 -- 172.21.15.123:0/1876265781 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f7f1008c100 con 0x7f7f241037a0 2024-03-30T04:33:54.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.927+0000 7f7f23400700 1 --2- 172.21.15.123:0/1876265781 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7f0806cd50 0x7f7f0806f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:33:54.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.927+0000 7f7f23400700 1 --2- 172.21.15.123:0/1876265781 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7f0806cd50 0x7f7f0806f200 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7f1400b3b0 tx=0x7f7f14009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:33:54.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:54.932+0000 7f7f1b400700 1 -- 172.21.15.123:0/1876265781 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7f10059930 con 0x7f7f241037a0 2024-03-30T04:33:54.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:54 smithi123 ceph-mon[139369]: from='client.34192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:33:54.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:54 smithi123 ceph-mon[139369]: pgmap v45: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:33:54.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:54 smithi123 ceph-mon[139369]: from='client.34196 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:33:54.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:54 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:54.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:54 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-30T04:33:54.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:54 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:33:54.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:54 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2643524011' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:33:54.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:54 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/580758649' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:33:55.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:54 smithi135 ceph-mon[106011]: from='client.34192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:33:55.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:54 smithi135 ceph-mon[106011]: pgmap v45: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:33:55.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:54 smithi135 ceph-mon[106011]: from='client.34196 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:33:55.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:54 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:33:55.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:54 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-30T04:33:55.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:54 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:33:55.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:54 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2643524011' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:33:55.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:54 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/580758649' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:33:55.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:55.226+0000 7f7f2a200700 1 -- 172.21.15.123:0/1876265781 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f7f24109dc0 con 0x7f7f241037a0 2024-03-30T04:33:55.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:55.226+0000 7f7f1b400700 1 -- 172.21.15.123:0/1876265781 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f7f100594c0 con 0x7f7f241037a0 2024-03-30T04:33:55.225 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_OK 2024-03-30T04:33:55.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:55.229+0000 7f7f2a200700 1 -- 172.21.15.123:0/1876265781 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7f0806cd50 msgr2=0x7f7f0806f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:55.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:55.229+0000 7f7f2a200700 1 --2- 172.21.15.123:0/1876265781 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7f0806cd50 0x7f7f0806f200 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7f1400b3b0 tx=0x7f7f14009e30 comp rx=0 tx=0).stop 2024-03-30T04:33:55.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:55.229+0000 7f7f2a200700 1 -- 172.21.15.123:0/1876265781 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f241037a0 msgr2=0x7f7f24115fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:33:55.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:55.229+0000 7f7f2a200700 1 --2- 172.21.15.123:0/1876265781 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f241037a0 0x7f7f24115fe0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7f10002a50 tx=0x7f7f10004600 comp rx=0 tx=0).stop 2024-03-30T04:33:55.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:55.229+0000 7f7f2a200700 1 -- 172.21.15.123:0/1876265781 shutdown_connections 2024-03-30T04:33:55.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:55.229+0000 7f7f2a200700 1 --2- 172.21.15.123:0/1876265781 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7f0806cd50 0x7f7f0806f200 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:55.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:55.229+0000 7f7f2a200700 1 --2- 172.21.15.123:0/1876265781 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f24102e00 0x7f7f24115aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:55.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:55.229+0000 7f7f2a200700 1 --2- 172.21.15.123:0/1876265781 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f241037a0 0x7f7f24115fe0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:33:55.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:55.229+0000 7f7f2a200700 1 -- 172.21.15.123:0/1876265781 >> 172.21.15.123:0/1876265781 conn(0x7f7f240f89b0 msgr2=0x7f7f240f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:33:55.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:55.229+0000 7f7f2a200700 1 -- 172.21.15.123:0/1876265781 shutdown_connections 2024-03-30T04:33:55.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:33:55.229+0000 7f7f2a200700 1 -- 172.21.15.123:0/1876265781 wait complete. 2024-03-30T04:33:55.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:55 smithi123 ceph-mon[139369]: Upgrade: Updating osd.0 2024-03-30T04:33:55.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:55 smithi123 ceph-mon[139369]: Deploying daemon osd.0 on smithi123 2024-03-30T04:33:55.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:55 smithi123 ceph-mon[139369]: from='client.34208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:33:55.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:55 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/1876265781' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:33:56.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:55 smithi135 ceph-mon[106011]: Upgrade: Updating osd.0 2024-03-30T04:33:56.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:55 smithi135 ceph-mon[106011]: Deploying daemon osd.0 on smithi123 2024-03-30T04:33:56.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:55 smithi135 ceph-mon[106011]: from='client.34208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:33:56.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:55 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/1876265781' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:33:56.945 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:56 smithi123 ceph-mon[139369]: pgmap v46: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:33:57.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:56 smithi135 ceph-mon[106011]: pgmap v46: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:33:57.967 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 30 04:33:57 smithi123 systemd[1]: Stopping Ceph osd.0 for 2e77a2be-ee4c-11ee-b647-cb9ed24678a4... 2024-03-30T04:33:58.466 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 30 04:33:58 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-0[58803]: 2024-03-30T04:33:58.224+0000 7fde6a000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T04:33:58.466 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 30 04:33:58 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-0[58803]: 2024-03-30T04:33:58.224+0000 7fde6a000700 -1 osd.0 49 *** Got signal Terminated *** 2024-03-30T04:33:58.466 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 30 04:33:58 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-0[58803]: 2024-03-30T04:33:58.224+0000 7fde6a000700 -1 osd.0 49 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T04:33:59.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:58 smithi135 ceph-mon[106011]: pgmap v47: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:33:59.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:58 smithi135 ceph-mon[106011]: osd.0 marked itself down and dead 2024-03-30T04:33:59.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:58 smithi123 ceph-mon[139369]: pgmap v47: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:33:59.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:58 smithi123 ceph-mon[139369]: osd.0 marked itself down and dead 2024-03-30T04:33:59.716 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 30 04:33:59 smithi123 bash[153262]: ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-0 2024-03-30T04:34:00.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:59 smithi135 ceph-mon[106011]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-30T04:34:00.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:59 smithi135 ceph-mon[106011]: osdmap e50: 6 total, 5 up, 6 in 2024-03-30T04:34:00.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:59 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:34:00.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:59 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:34:00.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:33:59 smithi135 ceph-mon[106011]: osdmap e51: 6 total, 5 up, 6 in 2024-03-30T04:34:00.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:59 smithi123 ceph-mon[139369]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-30T04:34:00.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:59 smithi123 ceph-mon[139369]: osdmap e50: 6 total, 5 up, 6 in 2024-03-30T04:34:00.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:59 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:34:00.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:59 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:34:00.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:33:59 smithi123 ceph-mon[139369]: osdmap e51: 6 total, 5 up, 6 in 2024-03-30T04:34:00.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:00 smithi123 ceph-mon[139369]: pgmap v49: 65 pgs: 9 stale+active+clean, 56 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-30T04:34:01.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:00 smithi135 ceph-mon[106011]: pgmap v49: 65 pgs: 9 stale+active+clean, 56 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-30T04:34:02.043 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:01 smithi123 ceph-mon[139369]: Health check failed: Degraded data redundancy: 61/324 objects degraded (18.827%), 19 pgs degraded (PG_DEGRADED) 2024-03-30T04:34:02.043 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:01 smithi123 ceph-mon[139369]: mgrmap e38: smithi123.xgksqq(active, since 92s), standbys: smithi135.kyblli 2024-03-30T04:34:02.043 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 30 04:34:01 smithi123 systemd[1]: ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4@osd.0.service: Succeeded. 2024-03-30T04:34:02.043 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 30 04:34:01 smithi123 systemd[1]: Stopped Ceph osd.0 for 2e77a2be-ee4c-11ee-b647-cb9ed24678a4. 2024-03-30T04:34:02.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:01 smithi135 ceph-mon[106011]: Health check failed: Degraded data redundancy: 61/324 objects degraded (18.827%), 19 pgs degraded (PG_DEGRADED) 2024-03-30T04:34:02.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:01 smithi135 ceph-mon[106011]: mgrmap e38: smithi123.xgksqq(active, since 92s), standbys: smithi135.kyblli 2024-03-30T04:34:02.466 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 30 04:34:02 smithi123 systemd[1]: Starting Ceph osd.0 for 2e77a2be-ee4c-11ee-b647-cb9ed24678a4... 2024-03-30T04:34:02.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:02 smithi123 ceph-mon[139369]: pgmap v51: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 767 B/s wr, 2 op/s; 61/324 objects degraded (18.827%) 2024-03-30T04:34:03.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:02 smithi135 ceph-mon[106011]: pgmap v51: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 767 B/s wr, 2 op/s; 61/324 objects degraded (18.827%) 2024-03-30T04:34:03.966 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 30 04:34:03 smithi123 bash[153534]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-30T04:34:03.967 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 30 04:34:03 smithi123 bash[153534]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-30T04:34:03.967 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 30 04:34:03 smithi123 bash[153534]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-30T04:34:03.967 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 30 04:34:03 smithi123 bash[153534]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-30T04:34:03.967 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 30 04:34:03 smithi123 bash[153534]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-03-30T04:34:03.967 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 30 04:34:03 smithi123 bash[153534]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-30T04:34:03.967 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 30 04:34:03 smithi123 bash[153534]: --> ceph-volume raw activate successful for osd ID: 0 2024-03-30T04:34:05.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:04 smithi135 ceph-mon[106011]: pgmap v52: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 767 B/s wr, 0 op/s; 61/324 objects degraded (18.827%) 2024-03-30T04:34:05.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:04 smithi123 ceph-mon[139369]: pgmap v52: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 767 B/s wr, 0 op/s; 61/324 objects degraded (18.827%) 2024-03-30T04:34:05.216 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 30 04:34:04 smithi123 ceph-osd[153850]: -- 172.21.15.123:0/2623300138 <== mon.0 v2:172.21.15.123:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55638a8eb1e0 con 0x55638a8bb000 2024-03-30T04:34:05.217 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 30 04:34:04 smithi123 bash[153812]: f8d62db1437b119b7524dd640accdb653eb470bb19e1c540d4dc7af84fbf278f 2024-03-30T04:34:05.217 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 30 04:34:04 smithi123 systemd[1]: Started Ceph osd.0 for 2e77a2be-ee4c-11ee-b647-cb9ed24678a4. 2024-03-30T04:34:05.716 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 30 04:34:05 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-0[153827]: 2024-03-30T04:34:05.408+0000 7f2717903640 -1 Falling back to public interface 2024-03-30T04:34:06.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:06 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:34:06.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:06 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:34:06.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:06 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:34:06.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:06 smithi135 ceph-mon[106011]: pgmap v53: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 767 B/s wr, 2 op/s; 61/324 objects degraded (18.827%) 2024-03-30T04:34:06.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:06 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:34:06.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:06 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:34:06.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:06 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:34:06.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:06 smithi123 ceph-mon[139369]: pgmap v53: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 767 B/s wr, 2 op/s; 61/324 objects degraded (18.827%) 2024-03-30T04:34:08.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:08 smithi123 ceph-mon[139369]: pgmap v54: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 710 B/s wr, 2 op/s; 61/324 objects degraded (18.827%) 2024-03-30T04:34:08.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:08 smithi135 ceph-mon[106011]: pgmap v54: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 710 B/s wr, 2 op/s; 61/324 objects degraded (18.827%) 2024-03-30T04:34:10.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:10 smithi135 ceph-mon[106011]: pgmap v55: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 614 B/s wr, 1 op/s; 61/324 objects degraded (18.827%) 2024-03-30T04:34:10.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:10 smithi123 ceph-mon[139369]: pgmap v55: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 614 B/s wr, 1 op/s; 61/324 objects degraded (18.827%) 2024-03-30T04:34:10.716 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 30 04:34:10 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-0[153827]: 2024-03-30T04:34:10.460+0000 7f2717903640 -1 osd.0 49 log_to_monitors true 2024-03-30T04:34:11.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:11 smithi123 ceph-mon[139369]: from='osd.0 [v2:172.21.15.123:6802/1500077196,v1:172.21.15.123:6803/1500077196]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-30T04:34:11.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:11 smithi135 ceph-mon[106011]: from='osd.0 [v2:172.21.15.123:6802/1500077196,v1:172.21.15.123:6803/1500077196]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-30T04:34:12.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:12 smithi135 ceph-mon[106011]: from='osd.0 [v2:172.21.15.123:6802/1500077196,v1:172.21.15.123:6803/1500077196]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-30T04:34:12.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:12 smithi135 ceph-mon[106011]: osdmap e52: 6 total, 5 up, 6 in 2024-03-30T04:34:12.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:12 smithi135 ceph-mon[106011]: from='osd.0 [v2:172.21.15.123:6802/1500077196,v1:172.21.15.123:6803/1500077196]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-03-30T04:34:12.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:12 smithi135 ceph-mon[106011]: pgmap v57: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 259 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s; 61/324 objects degraded (18.827%) 2024-03-30T04:34:12.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:12 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:34:12.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:12 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:34:12.718 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 30 04:34:12 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-0[153827]: 2024-03-30T04:34:12.361+0000 7f2701c00700 -1 osd.0 49 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T04:34:12.719 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:12 smithi123 ceph-mon[139369]: from='osd.0 [v2:172.21.15.123:6802/1500077196,v1:172.21.15.123:6803/1500077196]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-30T04:34:12.719 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:12 smithi123 ceph-mon[139369]: osdmap e52: 6 total, 5 up, 6 in 2024-03-30T04:34:12.719 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:12 smithi123 ceph-mon[139369]: from='osd.0 [v2:172.21.15.123:6802/1500077196,v1:172.21.15.123:6803/1500077196]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-03-30T04:34:12.719 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:12 smithi123 ceph-mon[139369]: pgmap v57: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 259 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s; 61/324 objects degraded (18.827%) 2024-03-30T04:34:12.719 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:12 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:34:12.720 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:12 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:34:13.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:13 smithi123 ceph-mon[139369]: from='osd.0 [v2:172.21.15.123:6802/1500077196,v1:172.21.15.123:6803/1500077196]' entity='osd.0' 2024-03-30T04:34:13.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:13 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:34:13.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:13 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:34:13.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:13 smithi135 ceph-mon[106011]: from='osd.0 [v2:172.21.15.123:6802/1500077196,v1:172.21.15.123:6803/1500077196]' entity='osd.0' 2024-03-30T04:34:13.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:13 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:34:13.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:13 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:34:14.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:14 smithi123 ceph-mon[139369]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-30T04:34:14.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:14 smithi123 ceph-mon[139369]: osd.0 [v2:172.21.15.123:6802/1500077196,v1:172.21.15.123:6803/1500077196] boot 2024-03-30T04:34:14.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:14 smithi123 ceph-mon[139369]: osdmap e53: 6 total, 6 up, 6 in 2024-03-30T04:34:14.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:14 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T04:34:14.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:14 smithi123 ceph-mon[139369]: pgmap v59: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 259 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s; 61/324 objects degraded (18.827%) 2024-03-30T04:34:14.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:14 smithi123 ceph-mon[139369]: osdmap e54: 6 total, 6 up, 6 in 2024-03-30T04:34:14.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:14 smithi135 ceph-mon[106011]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-30T04:34:14.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:14 smithi135 ceph-mon[106011]: osd.0 [v2:172.21.15.123:6802/1500077196,v1:172.21.15.123:6803/1500077196] boot 2024-03-30T04:34:14.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:14 smithi135 ceph-mon[106011]: osdmap e53: 6 total, 6 up, 6 in 2024-03-30T04:34:14.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:14 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T04:34:14.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:14 smithi135 ceph-mon[106011]: pgmap v59: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 259 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s; 61/324 objects degraded (18.827%) 2024-03-30T04:34:14.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:14 smithi135 ceph-mon[106011]: osdmap e54: 6 total, 6 up, 6 in 2024-03-30T04:34:15.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:15 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:34:15.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:15 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:34:15.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:15 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:34:15.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:15 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:34:16.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:16 smithi123 ceph-mon[139369]: pgmap v61: 65 pgs: 3 active+undersized+degraded, 6 active+undersized, 56 active+clean; 259 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 3.3 KiB/s rd, 6 op/s; 11/324 objects degraded (3.395%) 2024-03-30T04:34:16.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:16 smithi123 ceph-mon[139369]: Health check update: Degraded data redundancy: 11/324 objects degraded (3.395%), 3 pgs degraded (PG_DEGRADED) 2024-03-30T04:34:16.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:16 smithi135 ceph-mon[106011]: pgmap v61: 65 pgs: 3 active+undersized+degraded, 6 active+undersized, 56 active+clean; 259 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 3.3 KiB/s rd, 6 op/s; 11/324 objects degraded (3.395%) 2024-03-30T04:34:16.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:16 smithi135 ceph-mon[106011]: Health check update: Degraded data redundancy: 11/324 objects degraded (3.395%), 3 pgs degraded (PG_DEGRADED) 2024-03-30T04:34:17.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:17 smithi123 ceph-mon[139369]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 11/324 objects degraded (3.395%), 3 pgs degraded) 2024-03-30T04:34:17.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:17 smithi123 ceph-mon[139369]: Cluster is now healthy 2024-03-30T04:34:17.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:17 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:34:17.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:17 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:34:17.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:17 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:34:17.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:17 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:34:17.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:17 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:34:17.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:17 smithi135 ceph-mon[106011]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 11/324 objects degraded (3.395%), 3 pgs degraded) 2024-03-30T04:34:17.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:17 smithi135 ceph-mon[106011]: Cluster is now healthy 2024-03-30T04:34:17.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:17 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:34:17.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:17 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:34:17.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:17 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:34:17.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:17 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:34:17.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:17 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:34:18.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:18 smithi123 ceph-mon[139369]: pgmap v62: 65 pgs: 65 active+clean; 259 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:34:18.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:18 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:34:18.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:18 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:34:18.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:18 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:34:18.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:18 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:34:18.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:18 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-30T04:34:18.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:18 smithi123 ceph-mon[139369]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-30T04:34:18.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:18 smithi123 ceph-mon[139369]: Upgrade: osd.1 is safe to restart 2024-03-30T04:34:18.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:18 smithi135 ceph-mon[106011]: pgmap v62: 65 pgs: 65 active+clean; 259 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:34:18.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:18 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:34:18.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:18 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:34:18.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:18 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:34:18.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:18 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:34:18.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:18 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-30T04:34:18.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:18 smithi135 ceph-mon[106011]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-30T04:34:18.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:18 smithi135 ceph-mon[106011]: Upgrade: osd.1 is safe to restart 2024-03-30T04:34:20.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:20 smithi135 ceph-mon[106011]: pgmap v63: 65 pgs: 65 active+clean; 259 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-30T04:34:20.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:20 smithi135 ceph-mon[106011]: Upgrade: Updating osd.1 2024-03-30T04:34:20.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:20 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:34:20.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:20 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-30T04:34:20.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:20 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:34:20.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:20 smithi135 ceph-mon[106011]: Deploying daemon osd.1 on smithi123 2024-03-30T04:34:20.944 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:20 smithi123 ceph-mon[139369]: pgmap v63: 65 pgs: 65 active+clean; 259 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-30T04:34:20.944 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:20 smithi123 ceph-mon[139369]: Upgrade: Updating osd.1 2024-03-30T04:34:20.944 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:20 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:34:20.944 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:20 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-30T04:34:20.944 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:20 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:34:20.944 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:20 smithi123 ceph-mon[139369]: Deploying daemon osd.1 on smithi123 2024-03-30T04:34:22.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:22 smithi123 ceph-mon[139369]: pgmap v64: 65 pgs: 65 active+clean; 259 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-30T04:34:22.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:22 smithi135 ceph-mon[106011]: pgmap v64: 65 pgs: 65 active+clean; 259 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-30T04:34:23.937 INFO:journalctl@ceph.osd.1.smithi123.stdout:Mar 30 04:34:23 smithi123 systemd[1]: Stopping Ceph osd.1 for 2e77a2be-ee4c-11ee-b647-cb9ed24678a4... 2024-03-30T04:34:24.216 INFO:journalctl@ceph.osd.1.smithi123.stdout:Mar 30 04:34:23 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-1[67435]: 2024-03-30T04:34:23.937+0000 7f01b0400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T04:34:24.216 INFO:journalctl@ceph.osd.1.smithi123.stdout:Mar 30 04:34:23 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-1[67435]: 2024-03-30T04:34:23.937+0000 7f01b0400700 -1 osd.1 54 *** Got signal Terminated *** 2024-03-30T04:34:24.216 INFO:journalctl@ceph.osd.1.smithi123.stdout:Mar 30 04:34:23 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-1[67435]: 2024-03-30T04:34:23.937+0000 7f01b0400700 -1 osd.1 54 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T04:34:24.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:24 smithi135 ceph-mon[106011]: pgmap v65: 65 pgs: 65 active+clean; 259 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-30T04:34:24.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:24 smithi135 ceph-mon[106011]: osd.1 marked itself down and dead 2024-03-30T04:34:24.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:24 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:34:24.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:24 smithi123 ceph-mon[139369]: pgmap v65: 65 pgs: 65 active+clean; 259 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-30T04:34:24.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:24 smithi123 ceph-mon[139369]: osd.1 marked itself down and dead 2024-03-30T04:34:24.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:24 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:34:25.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.376+0000 7f6ae0a00700 1 -- 172.21.15.123:0/1432011176 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6adc071f90 msgr2=0x7f6adc070360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:34:25.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.376+0000 7f6ae0a00700 1 --2- 172.21.15.123:0/1432011176 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6adc071f90 0x7f6adc070360 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6acc004970 tx=0x7f6acc009e30 comp rx=0 tx=0).stop 2024-03-30T04:34:25.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.376+0000 7f6ae0a00700 1 -- 172.21.15.123:0/1432011176 shutdown_connections 2024-03-30T04:34:25.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.376+0000 7f6ae0a00700 1 --2- 172.21.15.123:0/1432011176 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6adc0708a0 0x7f6adc070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:25.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.376+0000 7f6ae0a00700 1 --2- 172.21.15.123:0/1432011176 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6adc071f90 0x7f6adc070360 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:25.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.376+0000 7f6ae0a00700 1 -- 172.21.15.123:0/1432011176 >> 172.21.15.123:0/1432011176 conn(0x7f6adc0f89d0 msgr2=0x7f6adc0fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:34:25.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.376+0000 7f6ae0a00700 1 -- 172.21.15.123:0/1432011176 shutdown_connections 2024-03-30T04:34:25.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.376+0000 7f6ae0a00700 1 -- 172.21.15.123:0/1432011176 wait complete. 2024-03-30T04:34:25.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.377+0000 7f6ae0a00700 1 Processor -- start 2024-03-30T04:34:25.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.377+0000 7f6ae0a00700 1 -- start start 2024-03-30T04:34:25.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.377+0000 7f6ae0a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6adc0708a0 0x7f6adc11a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:34:25.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.378+0000 7f6ae0a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6adc071f90 0x7f6adc11a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:34:25.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.378+0000 7f6ae0a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6adc11ac40 con 0x7f6adc0708a0 2024-03-30T04:34:25.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.378+0000 7f6ae0a00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6adc114010 con 0x7f6adc071f90 2024-03-30T04:34:25.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.378+0000 7f6ad9e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6adc0708a0 0x7f6adc11a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:34:25.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.378+0000 7f6ad9e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6adc0708a0 0x7f6adc11a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53472/0 (socket says 172.21.15.123:53472) 2024-03-30T04:34:25.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.378+0000 7f6ad9e00700 1 -- 172.21.15.123:0/1635421478 learned_addr learned my addr 172.21.15.123:0/1635421478 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:34:25.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.378+0000 7f6ad9400700 1 --2- 172.21.15.123:0/1635421478 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6adc071f90 0x7f6adc11a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:34:25.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.378+0000 7f6ad9e00700 1 -- 172.21.15.123:0/1635421478 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6adc071f90 msgr2=0x7f6adc11a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:34:25.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.378+0000 7f6ad9e00700 1 --2- 172.21.15.123:0/1635421478 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6adc071f90 0x7f6adc11a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:25.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.378+0000 7f6ad9e00700 1 -- 172.21.15.123:0/1635421478 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6acc005040 con 0x7f6adc0708a0 2024-03-30T04:34:25.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.379+0000 7f6ad9e00700 1 --2- 172.21.15.123:0/1635421478 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6adc0708a0 0x7f6adc11a0b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6acc005010 tx=0x7f6acc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:34:25.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.379+0000 7f6ad2000700 1 -- 172.21.15.123:0/1635421478 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6acc00e040 con 0x7f6adc0708a0 2024-03-30T04:34:25.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.379+0000 7f6ad2000700 1 -- 172.21.15.123:0/1635421478 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6acc0098b0 con 0x7f6adc0708a0 2024-03-30T04:34:25.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.379+0000 7f6ae0a00700 1 -- 172.21.15.123:0/1635421478 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6adc114290 con 0x7f6adc0708a0 2024-03-30T04:34:25.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.379+0000 7f6ad2000700 1 -- 172.21.15.123:0/1635421478 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6acc013e80 con 0x7f6adc0708a0 2024-03-30T04:34:25.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.379+0000 7f6ae0a00700 1 -- 172.21.15.123:0/1635421478 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6adc1147d0 con 0x7f6adc0708a0 2024-03-30T04:34:25.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.380+0000 7f6ae0a00700 1 -- 172.21.15.123:0/1635421478 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6adc071aa0 con 0x7f6adc0708a0 2024-03-30T04:34:25.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.381+0000 7f6ad2000700 1 -- 172.21.15.123:0/1635421478 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f6acc0136b0 con 0x7f6adc0708a0 2024-03-30T04:34:25.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.382+0000 7f6ad2000700 1 --2- 172.21.15.123:0/1635421478 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6ac006cc30 0x7f6ac006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:34:25.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.382+0000 7f6ad2000700 1 -- 172.21.15.123:0/1635421478 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f6acc08d220 con 0x7f6adc0708a0 2024-03-30T04:34:25.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.382+0000 7f6ad9400700 1 --2- 172.21.15.123:0/1635421478 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6ac006cc30 0x7f6ac006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:34:25.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.382+0000 7f6ad9400700 1 --2- 172.21.15.123:0/1635421478 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6ac006cc30 0x7f6ac006f0e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6adc115950 tx=0x7f6ac800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:34:25.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.386+0000 7f6ad2000700 1 -- 172.21.15.123:0/1635421478 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6acc05ad70 con 0x7f6adc0708a0 2024-03-30T04:34:25.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.632+0000 7f6ae0a00700 1 -- 172.21.15.123:0/1635421478 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6adc1104a0 con 0x7f6ac006cc30 2024-03-30T04:34:25.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.633+0000 7f6ad2000700 1 -- 172.21.15.123:0/1635421478 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f6ac800f040 con 0x7f6ac006cc30 2024-03-30T04:34:25.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.636+0000 7f6ae0a00700 1 -- 172.21.15.123:0/1635421478 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6ac006cc30 msgr2=0x7f6ac006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:34:25.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.636+0000 7f6ae0a00700 1 --2- 172.21.15.123:0/1635421478 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6ac006cc30 0x7f6ac006f0e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6adc115950 tx=0x7f6ac800b040 comp rx=0 tx=0).stop 2024-03-30T04:34:25.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.636+0000 7f6ae0a00700 1 -- 172.21.15.123:0/1635421478 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6adc0708a0 msgr2=0x7f6adc11a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:34:25.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.636+0000 7f6ae0a00700 1 --2- 172.21.15.123:0/1635421478 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6adc0708a0 0x7f6adc11a0b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6acc005010 tx=0x7f6acc009e30 comp rx=0 tx=0).stop 2024-03-30T04:34:25.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.636+0000 7f6ae0a00700 1 -- 172.21.15.123:0/1635421478 shutdown_connections 2024-03-30T04:34:25.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.636+0000 7f6ae0a00700 1 --2- 172.21.15.123:0/1635421478 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6ac006cc30 0x7f6ac006f0e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:25.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.636+0000 7f6ae0a00700 1 --2- 172.21.15.123:0/1635421478 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6adc0708a0 0x7f6adc11a0b0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:25.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.636+0000 7f6ae0a00700 1 --2- 172.21.15.123:0/1635421478 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6adc071f90 0x7f6adc11a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:25.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.636+0000 7f6ae0a00700 1 -- 172.21.15.123:0/1635421478 >> 172.21.15.123:0/1635421478 conn(0x7f6adc0f89d0 msgr2=0x7f6adc0f9bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:34:25.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.636+0000 7f6ae0a00700 1 -- 172.21.15.123:0/1635421478 shutdown_connections 2024-03-30T04:34:25.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.636+0000 7f6ae0a00700 1 -- 172.21.15.123:0/1635421478 wait complete. 2024-03-30T04:34:25.648 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:34:25.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.775+0000 7f664ea00700 1 -- 172.21.15.123:0/435380336 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66481037a0 msgr2=0x7f66480fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:34:25.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.775+0000 7f664ea00700 1 --2- 172.21.15.123:0/435380336 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66481037a0 0x7f66480fce10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6634004940 tx=0x7f6634009e30 comp rx=0 tx=0).stop 2024-03-30T04:34:25.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.776+0000 7f664ea00700 1 -- 172.21.15.123:0/435380336 shutdown_connections 2024-03-30T04:34:25.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.776+0000 7f664ea00700 1 --2- 172.21.15.123:0/435380336 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66481037a0 0x7f66480fce10 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:25.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.776+0000 7f664ea00700 1 --2- 172.21.15.123:0/435380336 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6648102e00 0x7f66481031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:25.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.776+0000 7f664ea00700 1 -- 172.21.15.123:0/435380336 >> 172.21.15.123:0/435380336 conn(0x7f66480f89b0 msgr2=0x7f66480fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:34:25.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.776+0000 7f664ea00700 1 -- 172.21.15.123:0/435380336 shutdown_connections 2024-03-30T04:34:25.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.776+0000 7f664ea00700 1 -- 172.21.15.123:0/435380336 wait complete. 2024-03-30T04:34:25.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.776+0000 7f664ea00700 1 Processor -- start 2024-03-30T04:34:25.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.777+0000 7f664ea00700 1 -- start start 2024-03-30T04:34:25.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.777+0000 7f664ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6648102e00 0x7f66480702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:34:25.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.777+0000 7f664ea00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f66481037a0 0x7f6648070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:34:25.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.777+0000 7f664ea00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6648070d90 con 0x7f6648102e00 2024-03-30T04:34:25.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.777+0000 7f664ea00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6648070ed0 con 0x7f66481037a0 2024-03-30T04:34:25.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.777+0000 7f6647e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6648102e00 0x7f66480702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:34:25.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.777+0000 7f6647e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6648102e00 0x7f66480702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53492/0 (socket says 172.21.15.123:53492) 2024-03-30T04:34:25.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.777+0000 7f6647e00700 1 -- 172.21.15.123:0/3182973028 learned_addr learned my addr 172.21.15.123:0/3182973028 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:34:25.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.777+0000 7f6647400700 1 --2- 172.21.15.123:0/3182973028 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f66481037a0 0x7f6648070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:34:25.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.778+0000 7f6647e00700 1 -- 172.21.15.123:0/3182973028 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f66481037a0 msgr2=0x7f6648070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:34:25.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.778+0000 7f6647e00700 1 --2- 172.21.15.123:0/3182973028 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f66481037a0 0x7f6648070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:25.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.778+0000 7f6647e00700 1 -- 172.21.15.123:0/3182973028 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6634005040 con 0x7f6648102e00 2024-03-30T04:34:25.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.778+0000 7f6647e00700 1 --2- 172.21.15.123:0/3182973028 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6648102e00 0x7f66480702c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6638002a50 tx=0x7f66380046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:34:25.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.778+0000 7f663fe00700 1 -- 172.21.15.123:0/3182973028 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6638007cc0 con 0x7f6648102e00 2024-03-30T04:34:25.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.778+0000 7f663fe00700 1 -- 172.21.15.123:0/3182973028 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f663800f040 con 0x7f6648102e00 2024-03-30T04:34:25.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.778+0000 7f664ea00700 1 -- 172.21.15.123:0/3182973028 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6648119e00 con 0x7f6648102e00 2024-03-30T04:34:25.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.778+0000 7f663fe00700 1 -- 172.21.15.123:0/3182973028 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f663800d9c0 con 0x7f6648102e00 2024-03-30T04:34:25.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.778+0000 7f664ea00700 1 -- 172.21.15.123:0/3182973028 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f664811a280 con 0x7f6648102e00 2024-03-30T04:34:25.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.779+0000 7f664ea00700 1 -- 172.21.15.123:0/3182973028 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66480fe550 con 0x7f6648102e00 2024-03-30T04:34:25.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.780+0000 7f663fe00700 1 -- 172.21.15.123:0/3182973028 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f663800db20 con 0x7f6648102e00 2024-03-30T04:34:25.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.781+0000 7f663fe00700 1 --2- 172.21.15.123:0/3182973028 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f662c06cc30 0x7f662c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:34:25.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.782+0000 7f663fe00700 1 -- 172.21.15.123:0/3182973028 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f663808bc80 con 0x7f6648102e00 2024-03-30T04:34:25.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.782+0000 7f6647400700 1 --2- 172.21.15.123:0/3182973028 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f662c06cc30 0x7f662c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:34:25.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.782+0000 7f6647400700 1 --2- 172.21.15.123:0/3182973028 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f662c06cc30 0x7f662c06f0e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6648071c00 tx=0x7f6634009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:34:25.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:25.786+0000 7f663fe00700 1 -- 172.21.15.123:0/3182973028 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6638055cc0 con 0x7f6648102e00 2024-03-30T04:34:25.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:25 smithi135 ceph-mon[106011]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-30T04:34:25.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:25 smithi135 ceph-mon[106011]: osdmap e55: 6 total, 5 up, 6 in 2024-03-30T04:34:25.966 INFO:journalctl@ceph.osd.1.smithi123.stdout:Mar 30 04:34:25 smithi123 bash[159676]: ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-1 2024-03-30T04:34:25.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:25 smithi123 ceph-mon[139369]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-30T04:34:25.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:25 smithi123 ceph-mon[139369]: osdmap e55: 6 total, 5 up, 6 in 2024-03-30T04:34:26.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.015+0000 7f664ea00700 1 -- 172.21.15.123:0/3182973028 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f66480719b0 con 0x7f662c06cc30 2024-03-30T04:34:26.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.016+0000 7f663fe00700 1 -- 172.21.15.123:0/3182973028 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f66480719b0 con 0x7f662c06cc30 2024-03-30T04:34:26.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.019+0000 7f664ea00700 1 -- 172.21.15.123:0/3182973028 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f662c06cc30 msgr2=0x7f662c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:34:26.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.019+0000 7f664ea00700 1 --2- 172.21.15.123:0/3182973028 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f662c06cc30 0x7f662c06f0e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6648071c00 tx=0x7f6634009e30 comp rx=0 tx=0).stop 2024-03-30T04:34:26.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.019+0000 7f664ea00700 1 -- 172.21.15.123:0/3182973028 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6648102e00 msgr2=0x7f66480702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:34:26.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.019+0000 7f664ea00700 1 --2- 172.21.15.123:0/3182973028 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6648102e00 0x7f66480702c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6638002a50 tx=0x7f66380046b0 comp rx=0 tx=0).stop 2024-03-30T04:34:26.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.019+0000 7f664ea00700 1 -- 172.21.15.123:0/3182973028 shutdown_connections 2024-03-30T04:34:26.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.019+0000 7f664ea00700 1 --2- 172.21.15.123:0/3182973028 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f662c06cc30 0x7f662c06f0e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:26.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.019+0000 7f664ea00700 1 --2- 172.21.15.123:0/3182973028 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6648102e00 0x7f66480702c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:26.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.019+0000 7f664ea00700 1 --2- 172.21.15.123:0/3182973028 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f66481037a0 0x7f6648070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:26.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.019+0000 7f664ea00700 1 -- 172.21.15.123:0/3182973028 >> 172.21.15.123:0/3182973028 conn(0x7f66480f89b0 msgr2=0x7f66480f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:34:26.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.019+0000 7f664ea00700 1 -- 172.21.15.123:0/3182973028 shutdown_connections 2024-03-30T04:34:26.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.019+0000 7f664ea00700 1 -- 172.21.15.123:0/3182973028 wait complete. 2024-03-30T04:34:26.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.165+0000 7fadd7200700 1 -- 172.21.15.123:0/3725923207 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadd8070360 msgr2=0x7fadd8070730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:34:26.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.165+0000 7fadd7200700 1 --2- 172.21.15.123:0/3725923207 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadd8070360 0x7fadd8070730 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fadc8004970 tx=0x7fadc8009e30 comp rx=0 tx=0).stop 2024-03-30T04:34:26.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.165+0000 7fadd7200700 1 -- 172.21.15.123:0/3725923207 shutdown_connections 2024-03-30T04:34:26.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.165+0000 7fadd7200700 1 --2- 172.21.15.123:0/3725923207 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fadd8070c70 0x7fadd8108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:26.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.165+0000 7fadd7200700 1 --2- 172.21.15.123:0/3725923207 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadd8070360 0x7fadd8070730 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:26.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.165+0000 7fadd7200700 1 -- 172.21.15.123:0/3725923207 >> 172.21.15.123:0/3725923207 conn(0x7fadd80f8830 msgr2=0x7fadd80fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:34:26.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.165+0000 7fadd7200700 1 -- 172.21.15.123:0/3725923207 shutdown_connections 2024-03-30T04:34:26.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.165+0000 7fadd7200700 1 -- 172.21.15.123:0/3725923207 wait complete. 2024-03-30T04:34:26.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.166+0000 7fadd7200700 1 Processor -- start 2024-03-30T04:34:26.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.166+0000 7fadd7200700 1 -- start start 2024-03-30T04:34:26.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.167+0000 7fadd7200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadd8070360 0x7fadd8115a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:34:26.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.167+0000 7fadd7200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fadd8070c70 0x7fadd8115fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:34:26.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.167+0000 7fadd7200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd8116620 con 0x7fadd8070360 2024-03-30T04:34:26.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.167+0000 7fadd7200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd810fa50 con 0x7fadd8070c70 2024-03-30T04:34:26.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.167+0000 7fadd5e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadd8070360 0x7fadd8115a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:34:26.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.167+0000 7fadd5e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadd8070360 0x7fadd8115a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53512/0 (socket says 172.21.15.123:53512) 2024-03-30T04:34:26.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.167+0000 7fadd5e00700 1 -- 172.21.15.123:0/21365802 learned_addr learned my addr 172.21.15.123:0/21365802 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:34:26.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.167+0000 7fadd5400700 1 --2- 172.21.15.123:0/21365802 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fadd8070c70 0x7fadd8115fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:34:26.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.167+0000 7fadd5e00700 1 -- 172.21.15.123:0/21365802 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fadd8070c70 msgr2=0x7fadd8115fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:34:26.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.167+0000 7fadd5e00700 1 --2- 172.21.15.123:0/21365802 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fadd8070c70 0x7fadd8115fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:26.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.167+0000 7fadd5e00700 1 -- 172.21.15.123:0/21365802 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadc8005040 con 0x7fadd8070360 2024-03-30T04:34:26.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.168+0000 7fadd5e00700 1 --2- 172.21.15.123:0/21365802 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadd8070360 0x7fadd8115a90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fadc8009380 tx=0x7fadc8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:34:26.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.168+0000 7fadce000700 1 -- 172.21.15.123:0/21365802 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fadc800e040 con 0x7fadd8070360 2024-03-30T04:34:26.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.168+0000 7fadce000700 1 -- 172.21.15.123:0/21365802 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fadc8013420 con 0x7fadd8070360 2024-03-30T04:34:26.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.168+0000 7fadd7200700 1 -- 172.21.15.123:0/21365802 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadd810fcd0 con 0x7fadd8070360 2024-03-30T04:34:26.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.168+0000 7fadce000700 1 -- 172.21.15.123:0/21365802 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fadc801c630 con 0x7fadd8070360 2024-03-30T04:34:26.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.168+0000 7fadd7200700 1 -- 172.21.15.123:0/21365802 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadd8110210 con 0x7fadd8070360 2024-03-30T04:34:26.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.169+0000 7fadd7200700 1 -- 172.21.15.123:0/21365802 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadd8071aa0 con 0x7fadd8070360 2024-03-30T04:34:26.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.170+0000 7fadce000700 1 -- 172.21.15.123:0/21365802 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fadc8013990 con 0x7fadd8070360 2024-03-30T04:34:26.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.171+0000 7fadce000700 1 --2- 172.21.15.123:0/21365802 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fadbc06cc20 0x7fadbc06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:34:26.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.171+0000 7fadce000700 1 -- 172.21.15.123:0/21365802 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fadc808d040 con 0x7fadd8070360 2024-03-30T04:34:26.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.171+0000 7fadd5400700 1 --2- 172.21.15.123:0/21365802 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fadbc06cc20 0x7fadbc06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:34:26.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.172+0000 7fadd5400700 1 --2- 172.21.15.123:0/21365802 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fadbc06cc20 0x7fadbc06f0d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fadd8104940 tx=0x7fadc400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:34:26.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.175+0000 7fadce000700 1 -- 172.21.15.123:0/21365802 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fadc805ac50 con 0x7fadd8070360 2024-03-30T04:34:26.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.405+0000 7fadd7200700 1 -- 172.21.15.123:0/21365802 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fadd804bed0 con 0x7fadbc06cc20 2024-03-30T04:34:26.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.416+0000 7fadce000700 1 -- 172.21.15.123:0/21365802 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fadd804bed0 con 0x7fadbc06cc20 2024-03-30T04:34:26.415 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:34:26.415 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (3m) 14s ago 16m 15.0M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:34:26.415 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (52s) 14s ago 16m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T04:34:26.415 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (46s) 42s ago 15m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T04:34:26.415 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (3m) 14s ago 15m 74.7M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T04:34:26.415 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (9m) 14s ago 9m 109M - 17.2.7-797-g5c024335 f77085f6aab7 2df066fba7a6 2024-03-30T04:34:26.415 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (9m) 14s ago 9m 58.0M - 17.2.7-797-g5c024335 f77085f6aab7 1293b8520db1 2024-03-30T04:34:26.415 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (9m) 42s ago 9m 67.4M - 17.2.7-797-g5c024335 f77085f6aab7 0dc41326818c 2024-03-30T04:34:26.415 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (9m) 42s ago 9m 119M - 17.2.7-797-g5c024335 f77085f6aab7 aef8a0cfc553 2024-03-30T04:34:26.415 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (5m) 14s ago 17m 496M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:34:26.415 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (4m) 42s ago 15m 448M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:34:26.416 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (2m) 14s ago 17m 43.9M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T04:34:26.416 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (97s) 42s ago 14m 33.2M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T04:34:26.416 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (4m) 14s ago 16m 7490k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:34:26.416 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (4m) 42s ago 15m 7512k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:34:26.416 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (21s) 14s ago 14m 11.0M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T04:34:26.416 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (13m) 14s ago 13m 235M 1618M 17.2.7-797-g5c024335 f77085f6aab7 e1a6c1818a0c 2024-03-30T04:34:26.416 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (12m) 14s ago 12m 189M 1618M 17.2.7-797-g5c024335 f77085f6aab7 4c6a727567bd 2024-03-30T04:34:26.416 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (12m) 42s ago 12m 259M 2642M 17.2.7-797-g5c024335 f77085f6aab7 6fbdd1d46a9d 2024-03-30T04:34:26.416 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (11m) 42s ago 11m 227M 2642M 17.2.7-797-g5c024335 f77085f6aab7 b74988e947e5 2024-03-30T04:34:26.416 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (11m) 42s ago 11m 220M 2642M 17.2.7-797-g5c024335 f77085f6aab7 1fd3185f5484 2024-03-30T04:34:26.416 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (3m) 14s ago 15m 51.2M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:34:26.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.419+0000 7fadd7200700 1 -- 172.21.15.123:0/21365802 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fadbc06cc20 msgr2=0x7fadbc06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:34:26.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.419+0000 7fadd7200700 1 --2- 172.21.15.123:0/21365802 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fadbc06cc20 0x7fadbc06f0d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fadd8104940 tx=0x7fadc400b040 comp rx=0 tx=0).stop 2024-03-30T04:34:26.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.419+0000 7fadd7200700 1 -- 172.21.15.123:0/21365802 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadd8070360 msgr2=0x7fadd8115a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:34:26.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.419+0000 7fadd7200700 1 --2- 172.21.15.123:0/21365802 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadd8070360 0x7fadd8115a90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fadc8009380 tx=0x7fadc8009e30 comp rx=0 tx=0).stop 2024-03-30T04:34:26.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.419+0000 7fadd7200700 1 -- 172.21.15.123:0/21365802 shutdown_connections 2024-03-30T04:34:26.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.419+0000 7fadd7200700 1 --2- 172.21.15.123:0/21365802 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fadbc06cc20 0x7fadbc06f0d0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:26.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.419+0000 7fadd7200700 1 --2- 172.21.15.123:0/21365802 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadd8070360 0x7fadd8115a90 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:26.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.419+0000 7fadd7200700 1 --2- 172.21.15.123:0/21365802 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fadd8070c70 0x7fadd8115fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:26.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.419+0000 7fadd7200700 1 -- 172.21.15.123:0/21365802 >> 172.21.15.123:0/21365802 conn(0x7fadd80f8830 msgr2=0x7fadd80f9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:34:26.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.419+0000 7fadd7200700 1 -- 172.21.15.123:0/21365802 shutdown_connections 2024-03-30T04:34:26.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.419+0000 7fadd7200700 1 -- 172.21.15.123:0/21365802 wait complete. 2024-03-30T04:34:26.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.553+0000 7f5c3bc00700 1 -- 172.21.15.123:0/1424417826 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c34070c70 msgr2=0x7f5c34109110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:34:26.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.553+0000 7f5c3bc00700 1 --2- 172.21.15.123:0/1424417826 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c34070c70 0x7f5c34109110 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5c20004970 tx=0x7f5c20009e30 comp rx=0 tx=0).stop 2024-03-30T04:34:26.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.553+0000 7f5c3bc00700 1 -- 172.21.15.123:0/1424417826 shutdown_connections 2024-03-30T04:34:26.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.553+0000 7f5c3bc00700 1 --2- 172.21.15.123:0/1424417826 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c34070c70 0x7f5c34109110 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:26.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.553+0000 7f5c3bc00700 1 --2- 172.21.15.123:0/1424417826 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5c34070360 0x7f5c34070730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:26.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.553+0000 7f5c3bc00700 1 -- 172.21.15.123:0/1424417826 >> 172.21.15.123:0/1424417826 conn(0x7f5c340f89b0 msgr2=0x7f5c340fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:34:26.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.553+0000 7f5c3bc00700 1 -- 172.21.15.123:0/1424417826 shutdown_connections 2024-03-30T04:34:26.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.553+0000 7f5c3bc00700 1 -- 172.21.15.123:0/1424417826 wait complete. 2024-03-30T04:34:26.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.554+0000 7f5c3bc00700 1 Processor -- start 2024-03-30T04:34:26.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.554+0000 7f5c3bc00700 1 -- start start 2024-03-30T04:34:26.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.555+0000 7f5c3bc00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5c34070360 0x7f5c34115ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:34:26.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.555+0000 7f5c3bc00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c34070c70 0x7f5c341161e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:34:26.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.555+0000 7f5c3bc00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c34116830 con 0x7f5c34070c70 2024-03-30T04:34:26.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.555+0000 7f5c3bc00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3410fc00 con 0x7f5c34070360 2024-03-30T04:34:26.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.555+0000 7f5c33e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c34070c70 0x7f5c341161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:34:26.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.555+0000 7f5c33e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c34070c70 0x7f5c341161e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53518/0 (socket says 172.21.15.123:53518) 2024-03-30T04:34:26.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.555+0000 7f5c33e00700 1 -- 172.21.15.123:0/141393598 learned_addr learned my addr 172.21.15.123:0/141393598 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:34:26.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.555+0000 7f5c39200700 1 --2- 172.21.15.123:0/141393598 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5c34070360 0x7f5c34115ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:34:26.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.555+0000 7f5c33e00700 1 -- 172.21.15.123:0/141393598 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5c34070360 msgr2=0x7f5c34115ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:34:26.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.555+0000 7f5c33e00700 1 --2- 172.21.15.123:0/141393598 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5c34070360 0x7f5c34115ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:26.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.555+0000 7f5c33e00700 1 -- 172.21.15.123:0/141393598 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c20005040 con 0x7f5c34070c70 2024-03-30T04:34:26.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.556+0000 7f5c33e00700 1 --2- 172.21.15.123:0/141393598 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c34070c70 0x7f5c341161e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5c20004f50 tx=0x7f5c20009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:34:26.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.556+0000 7f5c30c00700 1 -- 172.21.15.123:0/141393598 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c2000e040 con 0x7f5c34070c70 2024-03-30T04:34:26.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.556+0000 7f5c30c00700 1 -- 172.21.15.123:0/141393598 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5c20013420 con 0x7f5c34070c70 2024-03-30T04:34:26.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.556+0000 7f5c3bc00700 1 -- 172.21.15.123:0/141393598 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c3410fe80 con 0x7f5c34070c70 2024-03-30T04:34:26.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.556+0000 7f5c30c00700 1 -- 172.21.15.123:0/141393598 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c2001c630 con 0x7f5c34070c70 2024-03-30T04:34:26.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.556+0000 7f5c3bc00700 1 -- 172.21.15.123:0/141393598 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c341103c0 con 0x7f5c34070c70 2024-03-30T04:34:26.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.557+0000 7f5c3bc00700 1 -- 172.21.15.123:0/141393598 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c34103bd0 con 0x7f5c34070c70 2024-03-30T04:34:26.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.558+0000 7f5c30c00700 1 -- 172.21.15.123:0/141393598 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f5c20013590 con 0x7f5c34070c70 2024-03-30T04:34:26.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.559+0000 7f5c30c00700 1 --2- 172.21.15.123:0/141393598 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5c1c06cc30 0x7f5c1c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:34:26.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.559+0000 7f5c30c00700 1 -- 172.21.15.123:0/141393598 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f5c2008d080 con 0x7f5c34070c70 2024-03-30T04:34:26.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.559+0000 7f5c39200700 1 --2- 172.21.15.123:0/141393598 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5c1c06cc30 0x7f5c1c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:34:26.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.560+0000 7f5c39200700 1 --2- 172.21.15.123:0/141393598 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5c1c06cc30 0x7f5c1c06f0e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5c28004f40 tx=0x7f5c2800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:34:26.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.563+0000 7f5c30c00700 1 -- 172.21.15.123:0/141393598 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5c2005ac90 con 0x7f5c34070c70 2024-03-30T04:34:26.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:26 smithi135 ceph-mon[106011]: pgmap v67: 65 pgs: 29 peering, 3 stale+active+clean, 33 active+clean; 259 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 1 op/s 2024-03-30T04:34:26.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:26 smithi135 ceph-mon[106011]: Health check failed: Reduced data availability: 9 pgs peering (PG_AVAILABILITY) 2024-03-30T04:34:26.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:26 smithi135 ceph-mon[106011]: osdmap e56: 6 total, 5 up, 6 in 2024-03-30T04:34:26.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:26 smithi135 ceph-mon[106011]: from='client.34220 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:34:26.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.865+0000 7f5c3bc00700 1 -- 172.21.15.123:0/141393598 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f5c341112f0 con 0x7f5c34070c70 2024-03-30T04:34:26.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.866+0000 7f5c30c00700 1 -- 172.21.15.123:0/141393598 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f5c2005a820 con 0x7f5c34070c70 2024-03-30T04:34:26.865 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:34:26.865 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:34:26.865 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:34:26.865 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:34:26.865 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:34:26.865 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:34:26.865 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:34:26.865 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:34:26.865 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 4, 2024-03-30T04:34:26.865 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 1 2024-03-30T04:34:26.865 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:34:26.865 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:34:26.865 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 4 2024-03-30T04:34:26.865 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:34:26.866 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:34:26.866 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 8, 2024-03-30T04:34:26.866 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 5 2024-03-30T04:34:26.866 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:34:26.866 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:34:26.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.869+0000 7f5c3bc00700 1 -- 172.21.15.123:0/141393598 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5c1c06cc30 msgr2=0x7f5c1c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:34:26.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.869+0000 7f5c3bc00700 1 --2- 172.21.15.123:0/141393598 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5c1c06cc30 0x7f5c1c06f0e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5c28004f40 tx=0x7f5c2800b040 comp rx=0 tx=0).stop 2024-03-30T04:34:26.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.869+0000 7f5c3bc00700 1 -- 172.21.15.123:0/141393598 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c34070c70 msgr2=0x7f5c341161e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:34:26.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.869+0000 7f5c3bc00700 1 --2- 172.21.15.123:0/141393598 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c34070c70 0x7f5c341161e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5c20004f50 tx=0x7f5c20009e30 comp rx=0 tx=0).stop 2024-03-30T04:34:26.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.869+0000 7f5c3bc00700 1 -- 172.21.15.123:0/141393598 shutdown_connections 2024-03-30T04:34:26.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.869+0000 7f5c3bc00700 1 --2- 172.21.15.123:0/141393598 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5c1c06cc30 0x7f5c1c06f0e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:26.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.869+0000 7f5c3bc00700 1 --2- 172.21.15.123:0/141393598 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5c34070360 0x7f5c34115ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:26.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.869+0000 7f5c3bc00700 1 --2- 172.21.15.123:0/141393598 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c34070c70 0x7f5c341161e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:26.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.869+0000 7f5c3bc00700 1 -- 172.21.15.123:0/141393598 >> 172.21.15.123:0/141393598 conn(0x7f5c340f89b0 msgr2=0x7f5c341015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:34:26.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.869+0000 7f5c3bc00700 1 -- 172.21.15.123:0/141393598 shutdown_connections 2024-03-30T04:34:26.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:26.869+0000 7f5c3bc00700 1 -- 172.21.15.123:0/141393598 wait complete. 2024-03-30T04:34:26.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:26 smithi123 ceph-mon[139369]: pgmap v67: 65 pgs: 29 peering, 3 stale+active+clean, 33 active+clean; 259 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 1 op/s 2024-03-30T04:34:26.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:26 smithi123 ceph-mon[139369]: Health check failed: Reduced data availability: 9 pgs peering (PG_AVAILABILITY) 2024-03-30T04:34:26.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:26 smithi123 ceph-mon[139369]: osdmap e56: 6 total, 5 up, 6 in 2024-03-30T04:34:26.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:26 smithi123 ceph-mon[139369]: from='client.34220 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:34:27.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.009+0000 7fd7d7a00700 1 -- 172.21.15.123:0/3438148508 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7d0102e00 msgr2=0x7fd7d01031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:34:27.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.009+0000 7fd7d7a00700 1 --2- 172.21.15.123:0/3438148508 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7d0102e00 0x7fd7d01031d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd7c4004940 tx=0x7fd7c4009e30 comp rx=0 tx=0).stop 2024-03-30T04:34:27.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.009+0000 7fd7d7a00700 1 -- 172.21.15.123:0/3438148508 shutdown_connections 2024-03-30T04:34:27.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.009+0000 7fd7d7a00700 1 --2- 172.21.15.123:0/3438148508 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd7d01037a0 0x7fd7d00fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:27.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.009+0000 7fd7d7a00700 1 --2- 172.21.15.123:0/3438148508 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7d0102e00 0x7fd7d01031d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:27.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.009+0000 7fd7d7a00700 1 -- 172.21.15.123:0/3438148508 >> 172.21.15.123:0/3438148508 conn(0x7fd7d00f89b0 msgr2=0x7fd7d00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:34:27.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.009+0000 7fd7d7a00700 1 -- 172.21.15.123:0/3438148508 shutdown_connections 2024-03-30T04:34:27.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.009+0000 7fd7d7a00700 1 -- 172.21.15.123:0/3438148508 wait complete. 2024-03-30T04:34:27.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.010+0000 7fd7d7a00700 1 Processor -- start 2024-03-30T04:34:27.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.010+0000 7fd7d7a00700 1 -- start start 2024-03-30T04:34:27.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.010+0000 7fd7d7a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd7d0102e00 0x7fd7d0115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:34:27.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.010+0000 7fd7d7a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7d01037a0 0x7fd7d0116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:34:27.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.010+0000 7fd7d7a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d0116880 con 0x7fd7d01037a0 2024-03-30T04:34:27.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.010+0000 7fd7d7a00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d010fc50 con 0x7fd7d0102e00 2024-03-30T04:34:27.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.011+0000 7fd7cfe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7d01037a0 0x7fd7d0116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:34:27.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.011+0000 7fd7d5000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd7d0102e00 0x7fd7d0115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:34:27.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.011+0000 7fd7cfe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7d01037a0 0x7fd7d0116230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53548/0 (socket says 172.21.15.123:53548) 2024-03-30T04:34:27.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.011+0000 7fd7cfe00700 1 -- 172.21.15.123:0/3348983100 learned_addr learned my addr 172.21.15.123:0/3348983100 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:34:27.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.011+0000 7fd7d5000700 1 -- 172.21.15.123:0/3348983100 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7d01037a0 msgr2=0x7fd7d0116230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:34:27.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.011+0000 7fd7d5000700 1 --2- 172.21.15.123:0/3348983100 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7d01037a0 0x7fd7d0116230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:27.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.011+0000 7fd7d5000700 1 -- 172.21.15.123:0/3348983100 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c4005040 con 0x7fd7d0102e00 2024-03-30T04:34:27.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.011+0000 7fd7cfe00700 1 --2- 172.21.15.123:0/3348983100 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7d01037a0 0x7fd7d0116230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:34:27.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.011+0000 7fd7d5000700 1 --2- 172.21.15.123:0/3348983100 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd7d0102e00 0x7fd7d0115cf0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd7c4004940 tx=0x7fd7c4009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:34:27.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.012+0000 7fd7ccc00700 1 -- 172.21.15.123:0/3348983100 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd7c400e040 con 0x7fd7d0102e00 2024-03-30T04:34:27.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.012+0000 7fd7d7a00700 1 -- 172.21.15.123:0/3348983100 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7d010fed0 con 0x7fd7d0102e00 2024-03-30T04:34:27.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.012+0000 7fd7ccc00700 1 -- 172.21.15.123:0/3348983100 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd7c4004290 con 0x7fd7d0102e00 2024-03-30T04:34:27.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.012+0000 7fd7ccc00700 1 -- 172.21.15.123:0/3348983100 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd7c4013d60 con 0x7fd7d0102e00 2024-03-30T04:34:27.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.012+0000 7fd7d7a00700 1 -- 172.21.15.123:0/3348983100 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7d0110490 con 0x7fd7d0102e00 2024-03-30T04:34:27.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.013+0000 7fd7d7a00700 1 -- 172.21.15.123:0/3348983100 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7d00fe550 con 0x7fd7d0102e00 2024-03-30T04:34:27.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.014+0000 7fd7ccc00700 1 -- 172.21.15.123:0/3348983100 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fd7c40135b0 con 0x7fd7d0102e00 2024-03-30T04:34:27.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.015+0000 7fd7ccc00700 1 --2- 172.21.15.123:0/3348983100 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd7b806cc30 0x7fd7b806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:34:27.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.015+0000 7fd7ccc00700 1 -- 172.21.15.123:0/3348983100 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fd7c408d160 con 0x7fd7d0102e00 2024-03-30T04:34:27.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.016+0000 7fd7cfe00700 1 --2- 172.21.15.123:0/3348983100 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd7b806cc30 0x7fd7b806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:34:27.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.016+0000 7fd7cfe00700 1 --2- 172.21.15.123:0/3348983100 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd7b806cc30 0x7fd7b806f0e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd7d0111590 tx=0x7fd7bc00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:34:27.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.021+0000 7fd7ccc00700 1 -- 172.21.15.123:0/3348983100 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd7c405ac30 con 0x7fd7d0102e00 2024-03-30T04:34:27.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.292+0000 7fd7d7a00700 1 -- 172.21.15.123:0/3348983100 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fd7d004bf00 con 0x7fd7d0102e00 2024-03-30T04:34:27.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.293+0000 7fd7ccc00700 1 -- 172.21.15.123:0/3348983100 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 23 v23) v1 ==== 76+0+1876 (secure 0 0 0) 0x7fd7c405a7c0 con 0x7fd7d0102e00 2024-03-30T04:34:27.293 INFO:teuthology.orchestra.run.smithi123.stdout:e23 2024-03-30T04:34:27.293 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:34:27.293 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:34:27.293 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:34:27.293 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:34:27.293 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:34:27.293 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:34:27.293 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 23 2024-03-30T04:34:27.293 INFO:teuthology.orchestra.run.smithi123.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-30T04:34:27.293 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:34:27.294 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:32:48.448604+0000 2024-03-30T04:34:27.294 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:34:27.294 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:34:27.294 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:34:27.294 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:34:27.294 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:34:27.294 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T04:34:27.294 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:34:27.294 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:34:27.294 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 46 2024-03-30T04:34:27.295 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:34:27.295 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T04:34:27.295 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T04:34:27.295 INFO:teuthology.orchestra.run.smithi123.stdout:up {0=14488,1=24309} 2024-03-30T04:34:27.295 INFO:teuthology.orchestra.run.smithi123.stdout:failed 2024-03-30T04:34:27.295 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:34:27.295 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:34:27.295 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:34:27.295 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:34:27.295 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:34:27.295 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:34:27.296 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T04:34:27.296 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T04:34:27.296 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{0:14488} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.123:6828/3037849622,v1:172.21.15.123:6829/3037849622] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:34:27.296 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{0:14500} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/2433675995,v1:172.21.15.123:6827/2433675995] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:34:27.296 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{1:24309} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.135:6824/4050541062,v1:172.21.15.135:6825/4050541062] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:34:27.296 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{1:24319} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.135:6826/1096986122,v1:172.21.15.135:6827/1096986122] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:34:27.296 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:34:27.296 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:34:27.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.296+0000 7fd7d7a00700 1 -- 172.21.15.123:0/3348983100 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd7b806cc30 msgr2=0x7fd7b806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:34:27.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.296+0000 7fd7d7a00700 1 --2- 172.21.15.123:0/3348983100 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd7b806cc30 0x7fd7b806f0e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd7d0111590 tx=0x7fd7bc00b040 comp rx=0 tx=0).stop 2024-03-30T04:34:27.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.296+0000 7fd7d7a00700 1 -- 172.21.15.123:0/3348983100 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd7d0102e00 msgr2=0x7fd7d0115cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:34:27.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.296+0000 7fd7d7a00700 1 --2- 172.21.15.123:0/3348983100 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd7d0102e00 0x7fd7d0115cf0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd7c4004940 tx=0x7fd7c4009e30 comp rx=0 tx=0).stop 2024-03-30T04:34:27.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.296+0000 7fd7d7a00700 1 -- 172.21.15.123:0/3348983100 shutdown_connections 2024-03-30T04:34:27.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.296+0000 7fd7d7a00700 1 --2- 172.21.15.123:0/3348983100 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd7b806cc30 0x7fd7b806f0e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:27.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.296+0000 7fd7d7a00700 1 --2- 172.21.15.123:0/3348983100 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd7d0102e00 0x7fd7d0115cf0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:27.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.296+0000 7fd7d7a00700 1 --2- 172.21.15.123:0/3348983100 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7d01037a0 0x7fd7d0116230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:27.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.296+0000 7fd7d7a00700 1 -- 172.21.15.123:0/3348983100 >> 172.21.15.123:0/3348983100 conn(0x7fd7d00f89b0 msgr2=0x7fd7d00706d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:34:27.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.296+0000 7fd7d7a00700 1 -- 172.21.15.123:0/3348983100 shutdown_connections 2024-03-30T04:34:27.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.296+0000 7fd7d7a00700 1 -- 172.21.15.123:0/3348983100 wait complete. 2024-03-30T04:34:27.298 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 23 2024-03-30T04:34:27.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.432+0000 7fac0b200700 1 -- 172.21.15.123:0/1576898125 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac04102e00 msgr2=0x7fac041031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:34:27.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.432+0000 7fac0b200700 1 --2- 172.21.15.123:0/1576898125 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac04102e00 0x7fac041031d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fabf4004940 tx=0x7fabf4009e30 comp rx=0 tx=0).stop 2024-03-30T04:34:27.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.432+0000 7fac0b200700 1 -- 172.21.15.123:0/1576898125 shutdown_connections 2024-03-30T04:34:27.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.432+0000 7fac0b200700 1 --2- 172.21.15.123:0/1576898125 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac041037a0 0x7fac040fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:27.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.432+0000 7fac0b200700 1 --2- 172.21.15.123:0/1576898125 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac04102e00 0x7fac041031d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:27.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.432+0000 7fac0b200700 1 -- 172.21.15.123:0/1576898125 >> 172.21.15.123:0/1576898125 conn(0x7fac040f89b0 msgr2=0x7fac040fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:34:27.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.432+0000 7fac0b200700 1 -- 172.21.15.123:0/1576898125 shutdown_connections 2024-03-30T04:34:27.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.432+0000 7fac0b200700 1 -- 172.21.15.123:0/1576898125 wait complete. 2024-03-30T04:34:27.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.433+0000 7fac0b200700 1 Processor -- start 2024-03-30T04:34:27.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.433+0000 7fac0b200700 1 -- start start 2024-03-30T04:34:27.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.433+0000 7fac0b200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac04102e00 0x7fac04115d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:34:27.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.433+0000 7fac0b200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac041037a0 0x7fac04116270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:34:27.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.433+0000 7fac0b200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac041168c0 con 0x7fac041037a0 2024-03-30T04:34:27.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.433+0000 7fac0b200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac0410fc90 con 0x7fac04102e00 2024-03-30T04:34:27.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.434+0000 7fac03400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac041037a0 0x7fac04116270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:34:27.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.434+0000 7fac03400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac041037a0 0x7fac04116270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53564/0 (socket says 172.21.15.123:53564) 2024-03-30T04:34:27.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.434+0000 7fac03400700 1 -- 172.21.15.123:0/3398343476 learned_addr learned my addr 172.21.15.123:0/3398343476 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:34:27.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.434+0000 7fac03e00700 1 --2- 172.21.15.123:0/3398343476 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac04102e00 0x7fac04115d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:34:27.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.434+0000 7fac03400700 1 -- 172.21.15.123:0/3398343476 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac04102e00 msgr2=0x7fac04115d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:34:27.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.434+0000 7fac03400700 1 --2- 172.21.15.123:0/3398343476 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac04102e00 0x7fac04115d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:27.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.434+0000 7fac03400700 1 -- 172.21.15.123:0/3398343476 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf4005040 con 0x7fac041037a0 2024-03-30T04:34:27.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.434+0000 7fac03400700 1 --2- 172.21.15.123:0/3398343476 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac041037a0 0x7fac04116270 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fabf0002a80 tx=0x7fabf0004630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:34:27.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.434+0000 7fabfbe00700 1 -- 172.21.15.123:0/3398343476 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fabf0004e80 con 0x7fac041037a0 2024-03-30T04:34:27.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.435+0000 7fac0b200700 1 -- 172.21.15.123:0/3398343476 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac0410ff70 con 0x7fac041037a0 2024-03-30T04:34:27.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.435+0000 7fabfbe00700 1 -- 172.21.15.123:0/3398343476 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fabf000f040 con 0x7fac041037a0 2024-03-30T04:34:27.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.435+0000 7fabfbe00700 1 -- 172.21.15.123:0/3398343476 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fabf000da00 con 0x7fac041037a0 2024-03-30T04:34:27.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.435+0000 7fac0b200700 1 -- 172.21.15.123:0/3398343476 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac04110530 con 0x7fac041037a0 2024-03-30T04:34:27.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.436+0000 7fac0b200700 1 -- 172.21.15.123:0/3398343476 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac040fe550 con 0x7fac041037a0 2024-03-30T04:34:27.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.437+0000 7fabfbe00700 1 -- 172.21.15.123:0/3398343476 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fabf0007cc0 con 0x7fac041037a0 2024-03-30T04:34:27.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.438+0000 7fabfbe00700 1 --2- 172.21.15.123:0/3398343476 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fabe806cc30 0x7fabe806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:34:27.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.438+0000 7fabfbe00700 1 -- 172.21.15.123:0/3398343476 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fabf008b540 con 0x7fac041037a0 2024-03-30T04:34:27.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.438+0000 7fac03e00700 1 --2- 172.21.15.123:0/3398343476 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fabe806cc30 0x7fabe806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:34:27.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.439+0000 7fac03e00700 1 --2- 172.21.15.123:0/3398343476 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fabe806cc30 0x7fabe806f0e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fabf4009fd0 tx=0x7fabf4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:34:27.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.443+0000 7fabfbe00700 1 -- 172.21.15.123:0/3398343476 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fabf00590b0 con 0x7fac041037a0 2024-03-30T04:34:27.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.673+0000 7fac0b200700 1 -- 172.21.15.123:0/3398343476 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fac04109de0 con 0x7fabe806cc30 2024-03-30T04:34:27.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.674+0000 7fabfbe00700 1 -- 172.21.15.123:0/3398343476 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fac04109de0 con 0x7fabe806cc30 2024-03-30T04:34:27.673 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:34:27.673 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:34:27.673 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:34:27.673 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T04:34:27.673 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:34:27.673 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T04:34:27.673 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T04:34:27.674 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T04:34:27.674 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:34:27.674 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "7/21 daemons upgraded", 2024-03-30T04:34:27.674 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading osd daemons", 2024-03-30T04:34:27.674 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:34:27.674 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:34:27.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.677+0000 7fac0b200700 1 -- 172.21.15.123:0/3398343476 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fabe806cc30 msgr2=0x7fabe806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:34:27.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.677+0000 7fac0b200700 1 --2- 172.21.15.123:0/3398343476 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fabe806cc30 0x7fabe806f0e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fabf4009fd0 tx=0x7fabf4009e30 comp rx=0 tx=0).stop 2024-03-30T04:34:27.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.677+0000 7fac0b200700 1 -- 172.21.15.123:0/3398343476 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac041037a0 msgr2=0x7fac04116270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:34:27.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.677+0000 7fac0b200700 1 --2- 172.21.15.123:0/3398343476 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac041037a0 0x7fac04116270 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fabf0002a80 tx=0x7fabf0004630 comp rx=0 tx=0).stop 2024-03-30T04:34:27.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.677+0000 7fac0b200700 1 -- 172.21.15.123:0/3398343476 shutdown_connections 2024-03-30T04:34:27.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.677+0000 7fac0b200700 1 --2- 172.21.15.123:0/3398343476 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fabe806cc30 0x7fabe806f0e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:27.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.677+0000 7fac0b200700 1 --2- 172.21.15.123:0/3398343476 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac04102e00 0x7fac04115d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:27.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.677+0000 7fac0b200700 1 --2- 172.21.15.123:0/3398343476 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac041037a0 0x7fac04116270 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:27.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.677+0000 7fac0b200700 1 -- 172.21.15.123:0/3398343476 >> 172.21.15.123:0/3398343476 conn(0x7fac040f89b0 msgr2=0x7fac040704f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:34:27.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.677+0000 7fac0b200700 1 -- 172.21.15.123:0/3398343476 shutdown_connections 2024-03-30T04:34:27.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.677+0000 7fac0b200700 1 -- 172.21.15.123:0/3398343476 wait complete. 2024-03-30T04:34:27.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.810+0000 7f6165600700 1 -- 172.21.15.123:0/252878274 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61601037a0 msgr2=0x7f61600fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:34:27.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.810+0000 7f6165600700 1 --2- 172.21.15.123:0/252878274 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61601037a0 0x7f61600fce10 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f614c004940 tx=0x7f614c009e30 comp rx=0 tx=0).stop 2024-03-30T04:34:27.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.810+0000 7f6165600700 1 -- 172.21.15.123:0/252878274 shutdown_connections 2024-03-30T04:34:27.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.810+0000 7f6165600700 1 --2- 172.21.15.123:0/252878274 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61601037a0 0x7f61600fce10 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:27.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.810+0000 7f6165600700 1 --2- 172.21.15.123:0/252878274 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6160102e00 0x7f61601031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:27.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.810+0000 7f6165600700 1 -- 172.21.15.123:0/252878274 >> 172.21.15.123:0/252878274 conn(0x7f61600f89b0 msgr2=0x7f61600fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:34:27.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.811+0000 7f6165600700 1 -- 172.21.15.123:0/252878274 shutdown_connections 2024-03-30T04:34:27.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.811+0000 7f6165600700 1 -- 172.21.15.123:0/252878274 wait complete. 2024-03-30T04:34:27.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.811+0000 7f6165600700 1 Processor -- start 2024-03-30T04:34:27.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.811+0000 7f6165600700 1 -- start start 2024-03-30T04:34:27.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.812+0000 7f6165600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6160102e00 0x7f6160115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:34:27.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.812+0000 7f6165600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f61601037a0 0x7f61601161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:34:27.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.812+0000 7f6165600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6160116810 con 0x7f6160102e00 2024-03-30T04:34:27.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.812+0000 7f6165600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616010fbe0 con 0x7f61601037a0 2024-03-30T04:34:27.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.812+0000 7f615ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6160102e00 0x7f6160115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:34:27.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.812+0000 7f615ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6160102e00 0x7f6160115c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53580/0 (socket says 172.21.15.123:53580) 2024-03-30T04:34:27.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.812+0000 7f615ea00700 1 -- 172.21.15.123:0/2032139239 learned_addr learned my addr 172.21.15.123:0/2032139239 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:34:27.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.812+0000 7f615e000700 1 --2- 172.21.15.123:0/2032139239 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f61601037a0 0x7f61601161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:34:27.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.812+0000 7f615ea00700 1 -- 172.21.15.123:0/2032139239 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f61601037a0 msgr2=0x7f61601161c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:34:27.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.812+0000 7f615ea00700 1 --2- 172.21.15.123:0/2032139239 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f61601037a0 0x7f61601161c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:27.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.812+0000 7f615ea00700 1 -- 172.21.15.123:0/2032139239 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f614c005040 con 0x7f6160102e00 2024-03-30T04:34:27.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.812+0000 7f615e000700 1 --2- 172.21.15.123:0/2032139239 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f61601037a0 0x7f61601161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:34:27.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.813+0000 7f615ea00700 1 --2- 172.21.15.123:0/2032139239 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6160102e00 0x7f6160115c80 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f615000eb40 tx=0x7f615000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:34:27.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.813+0000 7f6156a00700 1 -- 172.21.15.123:0/2032139239 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6150007cc0 con 0x7f6160102e00 2024-03-30T04:34:27.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.813+0000 7f6156a00700 1 -- 172.21.15.123:0/2032139239 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f615000f040 con 0x7f6160102e00 2024-03-30T04:34:27.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.813+0000 7f6165600700 1 -- 172.21.15.123:0/2032139239 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f616010fe60 con 0x7f6160102e00 2024-03-30T04:34:27.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.813+0000 7f6156a00700 1 -- 172.21.15.123:0/2032139239 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6150009570 con 0x7f6160102e00 2024-03-30T04:34:27.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.813+0000 7f6165600700 1 -- 172.21.15.123:0/2032139239 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6160110420 con 0x7f6160102e00 2024-03-30T04:34:27.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.814+0000 7f6165600700 1 -- 172.21.15.123:0/2032139239 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61600fe550 con 0x7f6160102e00 2024-03-30T04:34:27.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.815+0000 7f6156a00700 1 -- 172.21.15.123:0/2032139239 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f61500096d0 con 0x7f6160102e00 2024-03-30T04:34:27.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.816+0000 7f6156a00700 1 --2- 172.21.15.123:0/2032139239 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f614406cc30 0x7f614406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:34:27.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.816+0000 7f6156a00700 1 -- 172.21.15.123:0/2032139239 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f615008bc80 con 0x7f6160102e00 2024-03-30T04:34:27.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.816+0000 7f615e000700 1 --2- 172.21.15.123:0/2032139239 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f614406cc30 0x7f614406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:34:27.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.817+0000 7f615e000700 1 --2- 172.21.15.123:0/2032139239 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f614406cc30 0x7f614406f0e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f614c000c00 tx=0x7f614c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:34:27.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:27.820+0000 7f6156a00700 1 -- 172.21.15.123:0/2032139239 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6150059820 con 0x7f6160102e00 2024-03-30T04:34:27.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:27 smithi135 ceph-mon[106011]: from='client.34224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:34:27.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:27 smithi135 ceph-mon[106011]: from='client.34228 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:34:27.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:27 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/141393598' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:34:27.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:27 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3348983100' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:34:27.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:27 smithi123 ceph-mon[139369]: from='client.34224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:34:27.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:27 smithi123 ceph-mon[139369]: from='client.34228 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:34:27.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:27 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/141393598' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:34:27.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:27 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3348983100' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:34:28.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:28.106+0000 7f6165600700 1 -- 172.21.15.123:0/2032139239 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f616004bed0 con 0x7f6160102e00 2024-03-30T04:34:28.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:28.107+0000 7f6156a00700 1 -- 172.21.15.123:0/2032139239 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+1180 (secure 0 0 0) 0x7f61500593b0 con 0x7f6160102e00 2024-03-30T04:34:28.106 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_WARN 1 osds down; Reduced data availability: 9 pgs peering; Degraded data redundancy: 6/324 objects degraded (1.852%), 2 pgs degraded 2024-03-30T04:34:28.106 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] OSD_DOWN: 1 osds down 2024-03-30T04:34:28.106 INFO:teuthology.orchestra.run.smithi123.stdout: osd.1 (root=default,host=smithi123) is down 2024-03-30T04:34:28.106 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] PG_AVAILABILITY: Reduced data availability: 9 pgs peering 2024-03-30T04:34:28.106 INFO:teuthology.orchestra.run.smithi123.stdout: pg 2.6 is stuck peering for 4m, current state peering, last acting [3,4] 2024-03-30T04:34:28.106 INFO:teuthology.orchestra.run.smithi123.stdout: pg 2.9 is stuck peering for 4m, current state peering, last acting [4,0] 2024-03-30T04:34:28.106 INFO:teuthology.orchestra.run.smithi123.stdout: pg 2.a is stuck peering for 4m, current state peering, last acting [4,3] 2024-03-30T04:34:28.107 INFO:teuthology.orchestra.run.smithi123.stdout: pg 2.d is stuck peering for 5m, current state peering, last acting [3,2] 2024-03-30T04:34:28.107 INFO:teuthology.orchestra.run.smithi123.stdout: pg 2.15 is stuck peering for 4m, current state peering, last acting [3,0] 2024-03-30T04:34:28.107 INFO:teuthology.orchestra.run.smithi123.stdout: pg 3.0 is stuck peering for 4m, current state peering, last acting [2,4] 2024-03-30T04:34:28.107 INFO:teuthology.orchestra.run.smithi123.stdout: pg 3.2 is stuck peering for 4m, current state peering, last acting [5,3] 2024-03-30T04:34:28.107 INFO:teuthology.orchestra.run.smithi123.stdout: pg 3.4 is stuck peering for 4m, current state peering, last acting [2,3] 2024-03-30T04:34:28.107 INFO:teuthology.orchestra.run.smithi123.stdout: pg 3.19 is stuck peering for 4m, current state peering, last acting [4,3] 2024-03-30T04:34:28.107 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 6/324 objects degraded (1.852%), 2 pgs degraded 2024-03-30T04:34:28.107 INFO:teuthology.orchestra.run.smithi123.stdout: pg 2.4 is active+undersized+degraded, acting [0,4] 2024-03-30T04:34:28.107 INFO:teuthology.orchestra.run.smithi123.stdout: pg 2.1b is active+undersized+degraded, acting [0,5] 2024-03-30T04:34:28.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:28.109+0000 7f6165600700 1 -- 172.21.15.123:0/2032139239 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f614406cc30 msgr2=0x7f614406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:34:28.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:28.109+0000 7f6165600700 1 --2- 172.21.15.123:0/2032139239 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f614406cc30 0x7f614406f0e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f614c000c00 tx=0x7f614c009e30 comp rx=0 tx=0).stop 2024-03-30T04:34:28.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:28.109+0000 7f6165600700 1 -- 172.21.15.123:0/2032139239 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6160102e00 msgr2=0x7f6160115c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:34:28.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:28.109+0000 7f6165600700 1 --2- 172.21.15.123:0/2032139239 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6160102e00 0x7f6160115c80 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f615000eb40 tx=0x7f615000b040 comp rx=0 tx=0).stop 2024-03-30T04:34:28.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:28.109+0000 7f6165600700 1 -- 172.21.15.123:0/2032139239 shutdown_connections 2024-03-30T04:34:28.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:28.109+0000 7f6165600700 1 --2- 172.21.15.123:0/2032139239 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f614406cc30 0x7f614406f0e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:28.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:28.109+0000 7f6165600700 1 --2- 172.21.15.123:0/2032139239 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6160102e00 0x7f6160115c80 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:28.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:28.109+0000 7f6165600700 1 --2- 172.21.15.123:0/2032139239 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f61601037a0 0x7f61601161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:28.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:28.109+0000 7f6165600700 1 -- 172.21.15.123:0/2032139239 >> 172.21.15.123:0/2032139239 conn(0x7f61600f89b0 msgr2=0x7f61600f99a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:34:28.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:28.110+0000 7f6165600700 1 -- 172.21.15.123:0/2032139239 shutdown_connections 2024-03-30T04:34:28.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:28.110+0000 7f6165600700 1 -- 172.21.15.123:0/2032139239 wait complete. 2024-03-30T04:34:28.466 INFO:journalctl@ceph.osd.1.smithi123.stdout:Mar 30 04:34:28 smithi123 systemd[1]: ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4@osd.1.service: Succeeded. 2024-03-30T04:34:28.467 INFO:journalctl@ceph.osd.1.smithi123.stdout:Mar 30 04:34:28 smithi123 systemd[1]: Stopped Ceph osd.1 for 2e77a2be-ee4c-11ee-b647-cb9ed24678a4. 2024-03-30T04:34:28.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:28 smithi135 ceph-mon[106011]: pgmap v69: 65 pgs: 2 active+undersized+degraded, 29 peering, 3 active+undersized, 31 active+clean; 259 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 6/324 objects degraded (1.852%) 2024-03-30T04:34:28.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:28 smithi135 ceph-mon[106011]: Health check failed: Degraded data redundancy: 6/324 objects degraded (1.852%), 2 pgs degraded (PG_DEGRADED) 2024-03-30T04:34:28.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:28 smithi135 ceph-mon[106011]: from='client.34240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:34:28.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:28 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2032139239' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:34:28.898 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:28 smithi123 ceph-mon[139369]: pgmap v69: 65 pgs: 2 active+undersized+degraded, 29 peering, 3 active+undersized, 31 active+clean; 259 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 6/324 objects degraded (1.852%) 2024-03-30T04:34:28.898 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:28 smithi123 ceph-mon[139369]: Health check failed: Degraded data redundancy: 6/324 objects degraded (1.852%), 2 pgs degraded (PG_DEGRADED) 2024-03-30T04:34:28.898 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:28 smithi123 ceph-mon[139369]: from='client.34240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:34:28.898 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:28 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2032139239' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:34:28.898 INFO:journalctl@ceph.osd.1.smithi123.stdout:Mar 30 04:34:28 smithi123 systemd[1]: Starting Ceph osd.1 for 2e77a2be-ee4c-11ee-b647-cb9ed24678a4... 2024-03-30T04:34:30.216 INFO:journalctl@ceph.osd.1.smithi123.stdout:Mar 30 04:34:29 smithi123 bash[160113]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-30T04:34:30.217 INFO:journalctl@ceph.osd.1.smithi123.stdout:Mar 30 04:34:29 smithi123 bash[160113]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-30T04:34:30.217 INFO:journalctl@ceph.osd.1.smithi123.stdout:Mar 30 04:34:29 smithi123 bash[160113]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-30T04:34:30.217 INFO:journalctl@ceph.osd.1.smithi123.stdout:Mar 30 04:34:29 smithi123 bash[160113]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-30T04:34:30.217 INFO:journalctl@ceph.osd.1.smithi123.stdout:Mar 30 04:34:29 smithi123 bash[160113]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-03-30T04:34:30.217 INFO:journalctl@ceph.osd.1.smithi123.stdout:Mar 30 04:34:29 smithi123 bash[160113]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-30T04:34:30.217 INFO:journalctl@ceph.osd.1.smithi123.stdout:Mar 30 04:34:29 smithi123 bash[160113]: --> ceph-volume raw activate successful for osd ID: 1 2024-03-30T04:34:30.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:30 smithi123 ceph-mon[139369]: pgmap v70: 65 pgs: 2 active+undersized+degraded, 29 peering, 3 active+undersized, 31 active+clean; 259 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 6/324 objects degraded (1.852%) 2024-03-30T04:34:30.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:34:30.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:34:30.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:30 smithi135 ceph-mon[106011]: pgmap v70: 65 pgs: 2 active+undersized+degraded, 29 peering, 3 active+undersized, 31 active+clean; 259 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 6/324 objects degraded (1.852%) 2024-03-30T04:34:30.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:30 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:34:30.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:30 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:34:31.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:31 smithi123 ceph-mon[139369]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs peering) 2024-03-30T04:34:31.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:31 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:34:31.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:31 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:34:31.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:31 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:34:31.717 INFO:journalctl@ceph.osd.1.smithi123.stdout:Mar 30 04:34:31 smithi123 ceph-osd[160425]: -- 172.21.15.123:0/2936417822 <== mon.0 v2:172.21.15.123:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560dc19211e0 con 0x560dc18f1000 2024-03-30T04:34:31.717 INFO:journalctl@ceph.osd.1.smithi123.stdout:Mar 30 04:34:31 smithi123 bash[160386]: 5d21109ef26683b9a9f30a18770b3cd6ce520e64632242d433181b5fe6a5710f 2024-03-30T04:34:31.717 INFO:journalctl@ceph.osd.1.smithi123.stdout:Mar 30 04:34:31 smithi123 systemd[1]: Started Ceph osd.1 for 2e77a2be-ee4c-11ee-b647-cb9ed24678a4. 2024-03-30T04:34:31.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:31 smithi135 ceph-mon[106011]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs peering) 2024-03-30T04:34:31.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:31 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:34:31.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:31 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:34:31.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:31 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:34:32.216 INFO:journalctl@ceph.osd.1.smithi123.stdout:Mar 30 04:34:31 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-1[160401]: 2024-03-30T04:34:31.878+0000 7f5a657ae640 -1 Falling back to public interface 2024-03-30T04:34:32.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:32 smithi135 ceph-mon[106011]: pgmap v71: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 259 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 49/324 objects degraded (15.123%) 2024-03-30T04:34:32.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:32 smithi123 ceph-mon[139369]: pgmap v71: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 259 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 49/324 objects degraded (15.123%) 2024-03-30T04:34:34.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:34 smithi135 ceph-mon[106011]: pgmap v72: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 259 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 49/324 objects degraded (15.123%) 2024-03-30T04:34:34.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:34 smithi135 ceph-mon[106011]: Health check update: Degraded data redundancy: 49/324 objects degraded (15.123%), 16 pgs degraded (PG_DEGRADED) 2024-03-30T04:34:34.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:34 smithi123 ceph-mon[139369]: pgmap v72: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 259 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 49/324 objects degraded (15.123%) 2024-03-30T04:34:34.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:34 smithi123 ceph-mon[139369]: Health check update: Degraded data redundancy: 49/324 objects degraded (15.123%), 16 pgs degraded (PG_DEGRADED) 2024-03-30T04:34:36.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:36 smithi135 ceph-mon[106011]: pgmap v73: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 259 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 49/324 objects degraded (15.123%) 2024-03-30T04:34:36.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:36 smithi123 ceph-mon[139369]: pgmap v73: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 259 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 49/324 objects degraded (15.123%) 2024-03-30T04:34:37.966 INFO:journalctl@ceph.osd.1.smithi123.stdout:Mar 30 04:34:37 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-1[160401]: 2024-03-30T04:34:37.590+0000 7f5a657ae640 -1 osd.1 54 log_to_monitors true 2024-03-30T04:34:38.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:38 smithi135 ceph-mon[106011]: pgmap v74: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 259 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 49/324 objects degraded (15.123%) 2024-03-30T04:34:38.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:38 smithi135 ceph-mon[106011]: from='osd.1 [v2:172.21.15.123:6810/4217392586,v1:172.21.15.123:6811/4217392586]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-30T04:34:38.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:38 smithi123 ceph-mon[139369]: pgmap v74: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 259 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 49/324 objects degraded (15.123%) 2024-03-30T04:34:38.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:38 smithi123 ceph-mon[139369]: from='osd.1 [v2:172.21.15.123:6810/4217392586,v1:172.21.15.123:6811/4217392586]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-30T04:34:38.967 INFO:journalctl@ceph.osd.1.smithi123.stdout:Mar 30 04:34:38 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-1[160401]: 2024-03-30T04:34:38.518+0000 7f5a54c00700 -1 osd.1 54 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T04:34:39.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:39 smithi135 ceph-mon[106011]: from='osd.1 [v2:172.21.15.123:6810/4217392586,v1:172.21.15.123:6811/4217392586]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-30T04:34:39.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:39 smithi135 ceph-mon[106011]: osdmap e57: 6 total, 5 up, 6 in 2024-03-30T04:34:39.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:39 smithi135 ceph-mon[106011]: from='osd.1 [v2:172.21.15.123:6810/4217392586,v1:172.21.15.123:6811/4217392586]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-03-30T04:34:39.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:39 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:34:39.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:39 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:34:39.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:39 smithi123 ceph-mon[139369]: from='osd.1 [v2:172.21.15.123:6810/4217392586,v1:172.21.15.123:6811/4217392586]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-30T04:34:39.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:39 smithi123 ceph-mon[139369]: osdmap e57: 6 total, 5 up, 6 in 2024-03-30T04:34:39.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:39 smithi123 ceph-mon[139369]: from='osd.1 [v2:172.21.15.123:6810/4217392586,v1:172.21.15.123:6811/4217392586]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-03-30T04:34:39.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:39 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:34:39.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:39 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:34:40.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:40 smithi123 ceph-mon[139369]: pgmap v76: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s; 49/324 objects degraded (15.123%) 2024-03-30T04:34:40.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:40 smithi123 ceph-mon[139369]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-30T04:34:40.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:40 smithi123 ceph-mon[139369]: osd.1 [v2:172.21.15.123:6810/4217392586,v1:172.21.15.123:6811/4217392586] boot 2024-03-30T04:34:40.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:40 smithi123 ceph-mon[139369]: osdmap e58: 6 total, 6 up, 6 in 2024-03-30T04:34:40.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:40 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T04:34:40.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:40 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:34:40.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:40 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:34:40.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:40 smithi135 ceph-mon[106011]: pgmap v76: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s; 49/324 objects degraded (15.123%) 2024-03-30T04:34:40.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:40 smithi135 ceph-mon[106011]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-30T04:34:40.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:40 smithi135 ceph-mon[106011]: osd.1 [v2:172.21.15.123:6810/4217392586,v1:172.21.15.123:6811/4217392586] boot 2024-03-30T04:34:40.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:40 smithi135 ceph-mon[106011]: osdmap e58: 6 total, 6 up, 6 in 2024-03-30T04:34:40.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:40 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T04:34:40.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:40 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:34:40.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:40 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:34:41.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:41 smithi123 ceph-mon[139369]: osdmap e59: 6 total, 6 up, 6 in 2024-03-30T04:34:41.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:41 smithi135 ceph-mon[106011]: osdmap e59: 6 total, 6 up, 6 in 2024-03-30T04:34:42.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:42 smithi135 ceph-mon[106011]: pgmap v79: 65 pgs: 20 peering, 7 active+undersized+degraded, 5 active+undersized, 33 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.3 KiB/s rd, 2 op/s; 20/324 objects degraded (6.173%) 2024-03-30T04:34:42.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:42 smithi135 ceph-mon[106011]: Health check update: Degraded data redundancy: 20/324 objects degraded (6.173%), 7 pgs degraded (PG_DEGRADED) 2024-03-30T04:34:42.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:42 smithi123 ceph-mon[139369]: pgmap v79: 65 pgs: 20 peering, 7 active+undersized+degraded, 5 active+undersized, 33 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.3 KiB/s rd, 2 op/s; 20/324 objects degraded (6.173%) 2024-03-30T04:34:42.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:42 smithi123 ceph-mon[139369]: Health check update: Degraded data redundancy: 20/324 objects degraded (6.173%), 7 pgs degraded (PG_DEGRADED) 2024-03-30T04:34:44.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:44 smithi123 ceph-mon[139369]: pgmap v80: 65 pgs: 20 peering, 7 active+undersized+degraded, 5 active+undersized, 33 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.3 KiB/s rd, 2 op/s; 20/324 objects degraded (6.173%) 2024-03-30T04:34:44.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:34:44.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:34:44.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:34:44.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:34:44.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:34:44.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:34:44.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:34:44.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:34:44.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:34:44.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-30T04:34:44.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:34:44.718 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:34:44.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:44 smithi135 ceph-mon[106011]: pgmap v80: 65 pgs: 20 peering, 7 active+undersized+degraded, 5 active+undersized, 33 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.3 KiB/s rd, 2 op/s; 20/324 objects degraded (6.173%) 2024-03-30T04:34:44.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:44 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:34:44.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:44 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:34:44.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:44 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:34:44.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:44 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:34:44.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:44 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:34:44.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:44 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:34:44.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:44 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:34:44.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:44 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:34:44.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:44 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:34:44.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:44 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-30T04:34:44.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:44 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:34:44.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:44 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:34:45.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:45 smithi135 ceph-mon[106011]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-30T04:34:45.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:45 smithi135 ceph-mon[106011]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-03-30T04:34:45.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:45 smithi123 ceph-mon[139369]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-30T04:34:45.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:45 smithi123 ceph-mon[139369]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-03-30T04:34:46.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:46 smithi135 ceph-mon[106011]: pgmap v81: 65 pgs: 65 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:34:46.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:46 smithi135 ceph-mon[106011]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 20/324 objects degraded (6.173%), 7 pgs degraded) 2024-03-30T04:34:46.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:46 smithi135 ceph-mon[106011]: Cluster is now healthy 2024-03-30T04:34:46.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:46 smithi123 ceph-mon[139369]: pgmap v81: 65 pgs: 65 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:34:46.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:46 smithi123 ceph-mon[139369]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 20/324 objects degraded (6.173%), 7 pgs degraded) 2024-03-30T04:34:46.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:46 smithi123 ceph-mon[139369]: Cluster is now healthy 2024-03-30T04:34:48.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:48 smithi135 ceph-mon[106011]: pgmap v82: 65 pgs: 65 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-30T04:34:48.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:48 smithi123 ceph-mon[139369]: pgmap v82: 65 pgs: 65 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-30T04:34:50.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:50 smithi135 ceph-mon[106011]: pgmap v83: 65 pgs: 65 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:34:50.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:50 smithi123 ceph-mon[139369]: pgmap v83: 65 pgs: 65 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:34:52.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:52 smithi123 ceph-mon[139369]: pgmap v84: 65 pgs: 65 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:34:52.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:52 smithi135 ceph-mon[106011]: pgmap v84: 65 pgs: 65 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:34:54.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:54 smithi123 ceph-mon[139369]: pgmap v85: 65 pgs: 65 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-30T04:34:54.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:54 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:34:54.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:54 smithi135 ceph-mon[106011]: pgmap v85: 65 pgs: 65 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-30T04:34:54.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:54 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:34:56.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:56 smithi123 ceph-mon[139369]: pgmap v86: 65 pgs: 65 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-30T04:34:56.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:56 smithi135 ceph-mon[106011]: pgmap v86: 65 pgs: 65 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-30T04:34:58.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.259+0000 7f50d2600700 1 -- 172.21.15.123:0/411370249 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50cc1037a0 msgr2=0x7f50cc0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:34:58.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.259+0000 7f50d2600700 1 --2- 172.21.15.123:0/411370249 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50cc1037a0 0x7f50cc0fce10 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f50b8004940 tx=0x7f50b8009e30 comp rx=0 tx=0).stop 2024-03-30T04:34:58.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.259+0000 7f50d2600700 1 -- 172.21.15.123:0/411370249 shutdown_connections 2024-03-30T04:34:58.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.259+0000 7f50d2600700 1 --2- 172.21.15.123:0/411370249 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50cc1037a0 0x7f50cc0fce10 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:58.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.259+0000 7f50d2600700 1 --2- 172.21.15.123:0/411370249 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50cc102e00 0x7f50cc1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:58.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.259+0000 7f50d2600700 1 -- 172.21.15.123:0/411370249 >> 172.21.15.123:0/411370249 conn(0x7f50cc0f89b0 msgr2=0x7f50cc0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:34:58.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.259+0000 7f50d2600700 1 -- 172.21.15.123:0/411370249 shutdown_connections 2024-03-30T04:34:58.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.259+0000 7f50d2600700 1 -- 172.21.15.123:0/411370249 wait complete. 2024-03-30T04:34:58.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.260+0000 7f50d2600700 1 Processor -- start 2024-03-30T04:34:58.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.260+0000 7f50d2600700 1 -- start start 2024-03-30T04:34:58.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.261+0000 7f50d2600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50cc102e00 0x7f50cc0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:34:58.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.261+0000 7f50d2600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50cc1037a0 0x7f50cc070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:34:58.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.261+0000 7f50d2600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50cc070d90 con 0x7f50cc1037a0 2024-03-30T04:34:58.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.261+0000 7f50d2600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50cc070ed0 con 0x7f50cc102e00 2024-03-30T04:34:58.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.261+0000 7f50caa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50cc1037a0 0x7f50cc070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:34:58.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.261+0000 7f50caa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50cc1037a0 0x7f50cc070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37242/0 (socket says 172.21.15.123:37242) 2024-03-30T04:34:58.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.261+0000 7f50caa00700 1 -- 172.21.15.123:0/3825722199 learned_addr learned my addr 172.21.15.123:0/3825722199 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:34:58.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.261+0000 7f50cb400700 1 --2- 172.21.15.123:0/3825722199 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50cc102e00 0x7f50cc0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:34:58.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.261+0000 7f50caa00700 1 -- 172.21.15.123:0/3825722199 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50cc102e00 msgr2=0x7f50cc0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:34:58.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.261+0000 7f50caa00700 1 --2- 172.21.15.123:0/3825722199 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50cc102e00 0x7f50cc0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:58.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.261+0000 7f50caa00700 1 -- 172.21.15.123:0/3825722199 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b8005040 con 0x7f50cc1037a0 2024-03-30T04:34:58.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.262+0000 7f50caa00700 1 --2- 172.21.15.123:0/3825722199 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50cc1037a0 0x7f50cc070800 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f50b8009390 tx=0x7f50b8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:34:58.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.262+0000 7f50c3400700 1 -- 172.21.15.123:0/3825722199 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f50b800e040 con 0x7f50cc1037a0 2024-03-30T04:34:58.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.262+0000 7f50c3400700 1 -- 172.21.15.123:0/3825722199 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f50b8013420 con 0x7f50cc1037a0 2024-03-30T04:34:58.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.262+0000 7f50d2600700 1 -- 172.21.15.123:0/3825722199 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50cc071120 con 0x7f50cc1037a0 2024-03-30T04:34:58.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.262+0000 7f50c3400700 1 -- 172.21.15.123:0/3825722199 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f50b801c630 con 0x7f50cc1037a0 2024-03-30T04:34:58.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.262+0000 7f50d2600700 1 -- 172.21.15.123:0/3825722199 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50cc11a230 con 0x7f50cc1037a0 2024-03-30T04:34:58.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.263+0000 7f50d2600700 1 -- 172.21.15.123:0/3825722199 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50cc0fe550 con 0x7f50cc1037a0 2024-03-30T04:34:58.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.264+0000 7f50c3400700 1 -- 172.21.15.123:0/3825722199 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f50b8013a00 con 0x7f50cc1037a0 2024-03-30T04:34:58.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.265+0000 7f50c3400700 1 --2- 172.21.15.123:0/3825722199 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f50b006cc30 0x7f50b006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:34:58.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.265+0000 7f50c3400700 1 -- 172.21.15.123:0/3825722199 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f50b808d090 con 0x7f50cc1037a0 2024-03-30T04:34:58.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.266+0000 7f50cb400700 1 --2- 172.21.15.123:0/3825722199 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f50b006cc30 0x7f50b006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:34:58.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.266+0000 7f50cb400700 1 --2- 172.21.15.123:0/3825722199 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f50b006cc30 0x7f50b006f0e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f50bc004f40 tx=0x7f50bc00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:34:58.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.270+0000 7f50c3400700 1 -- 172.21.15.123:0/3825722199 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f50b8057170 con 0x7f50cc1037a0 2024-03-30T04:34:58.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.495+0000 7f50d2600700 1 -- 172.21.15.123:0/3825722199 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f50cc109e40 con 0x7f50b006cc30 2024-03-30T04:34:58.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.496+0000 7f50c3400700 1 -- 172.21.15.123:0/3825722199 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f50bc00f040 con 0x7f50b006cc30 2024-03-30T04:34:58.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.498+0000 7f50d2600700 1 -- 172.21.15.123:0/3825722199 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f50b006cc30 msgr2=0x7f50b006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:34:58.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.498+0000 7f50d2600700 1 --2- 172.21.15.123:0/3825722199 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f50b006cc30 0x7f50b006f0e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f50bc004f40 tx=0x7f50bc00b040 comp rx=0 tx=0).stop 2024-03-30T04:34:58.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.498+0000 7f50d2600700 1 -- 172.21.15.123:0/3825722199 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50cc1037a0 msgr2=0x7f50cc070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:34:58.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.498+0000 7f50d2600700 1 --2- 172.21.15.123:0/3825722199 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50cc1037a0 0x7f50cc070800 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f50b8009390 tx=0x7f50b8009e30 comp rx=0 tx=0).stop 2024-03-30T04:34:58.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.498+0000 7f50d2600700 1 -- 172.21.15.123:0/3825722199 shutdown_connections 2024-03-30T04:34:58.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.498+0000 7f50d2600700 1 --2- 172.21.15.123:0/3825722199 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f50b006cc30 0x7f50b006f0e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:58.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.498+0000 7f50d2600700 1 --2- 172.21.15.123:0/3825722199 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50cc102e00 0x7f50cc0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:58.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.498+0000 7f50d2600700 1 --2- 172.21.15.123:0/3825722199 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50cc1037a0 0x7f50cc070800 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:58.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.498+0000 7f50d2600700 1 -- 172.21.15.123:0/3825722199 >> 172.21.15.123:0/3825722199 conn(0x7f50cc0f89b0 msgr2=0x7f50cc0f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:34:58.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.498+0000 7f50d2600700 1 -- 172.21.15.123:0/3825722199 shutdown_connections 2024-03-30T04:34:58.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.498+0000 7f50d2600700 1 -- 172.21.15.123:0/3825722199 wait complete. 2024-03-30T04:34:58.511 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:34:58.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.633+0000 7f3f46400700 1 -- 172.21.15.123:0/4290330535 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f40070c70 msgr2=0x7f3f40109110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:34:58.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.633+0000 7f3f46400700 1 --2- 172.21.15.123:0/4290330535 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f40070c70 0x7f3f40109110 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c004970 tx=0x7f3f2c009e30 comp rx=0 tx=0).stop 2024-03-30T04:34:58.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.633+0000 7f3f46400700 1 -- 172.21.15.123:0/4290330535 shutdown_connections 2024-03-30T04:34:58.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.633+0000 7f3f46400700 1 --2- 172.21.15.123:0/4290330535 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f40070c70 0x7f3f40109110 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:58.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.633+0000 7f3f46400700 1 --2- 172.21.15.123:0/4290330535 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3f40070360 0x7f3f40070730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:58.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.633+0000 7f3f46400700 1 -- 172.21.15.123:0/4290330535 >> 172.21.15.123:0/4290330535 conn(0x7f3f400f89b0 msgr2=0x7f3f400fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:34:58.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.633+0000 7f3f46400700 1 -- 172.21.15.123:0/4290330535 shutdown_connections 2024-03-30T04:34:58.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.633+0000 7f3f46400700 1 -- 172.21.15.123:0/4290330535 wait complete. 2024-03-30T04:34:58.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.634+0000 7f3f46400700 1 Processor -- start 2024-03-30T04:34:58.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.634+0000 7f3f46400700 1 -- start start 2024-03-30T04:34:58.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.634+0000 7f3f46400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f40070360 0x7f3f40115ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:34:58.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.634+0000 7f3f46400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3f40070c70 0x7f3f401161e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:34:58.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.634+0000 7f3f46400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f40116830 con 0x7f3f40070360 2024-03-30T04:34:58.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.634+0000 7f3f46400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4010fc00 con 0x7f3f40070c70 2024-03-30T04:34:58.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.634+0000 7f3f3f400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f40070360 0x7f3f40115ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:34:58.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.634+0000 7f3f3f400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f40070360 0x7f3f40115ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37264/0 (socket says 172.21.15.123:37264) 2024-03-30T04:34:58.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.634+0000 7f3f3f400700 1 -- 172.21.15.123:0/2050950646 learned_addr learned my addr 172.21.15.123:0/2050950646 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:34:58.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.635+0000 7f3f3ea00700 1 --2- 172.21.15.123:0/2050950646 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3f40070c70 0x7f3f401161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:34:58.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.635+0000 7f3f3f400700 1 -- 172.21.15.123:0/2050950646 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3f40070c70 msgr2=0x7f3f401161e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:34:58.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.635+0000 7f3f3f400700 1 --2- 172.21.15.123:0/2050950646 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3f40070c70 0x7f3f401161e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:58.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.635+0000 7f3f3f400700 1 -- 172.21.15.123:0/2050950646 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f2c005040 con 0x7f3f40070360 2024-03-30T04:34:58.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.635+0000 7f3f3f400700 1 --2- 172.21.15.123:0/2050950646 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f40070360 0x7f3f40115ca0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3f3000c8c0 tx=0x7f3f300044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:34:58.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.635+0000 7f3f37400700 1 -- 172.21.15.123:0/2050950646 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f3000a760 con 0x7f3f40070360 2024-03-30T04:34:58.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.635+0000 7f3f37400700 1 -- 172.21.15.123:0/2050950646 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3f3000ad70 con 0x7f3f40070360 2024-03-30T04:34:58.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.635+0000 7f3f46400700 1 -- 172.21.15.123:0/2050950646 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f4010fee0 con 0x7f3f40070360 2024-03-30T04:34:58.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.635+0000 7f3f37400700 1 -- 172.21.15.123:0/2050950646 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f300113a0 con 0x7f3f40070360 2024-03-30T04:34:58.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.636+0000 7f3f46400700 1 -- 172.21.15.123:0/2050950646 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f401103f0 con 0x7f3f40070360 2024-03-30T04:34:58.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.637+0000 7f3f46400700 1 -- 172.21.15.123:0/2050950646 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f40103bd0 con 0x7f3f40070360 2024-03-30T04:34:58.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.637+0000 7f3f37400700 1 -- 172.21.15.123:0/2050950646 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f3f30012400 con 0x7f3f40070360 2024-03-30T04:34:58.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.638+0000 7f3f37400700 1 --2- 172.21.15.123:0/2050950646 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3f2406cc20 0x7f3f2406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:34:58.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.638+0000 7f3f37400700 1 -- 172.21.15.123:0/2050950646 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f3f3008bfe0 con 0x7f3f40070360 2024-03-30T04:34:58.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.638+0000 7f3f3ea00700 1 --2- 172.21.15.123:0/2050950646 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3f2406cc20 0x7f3f2406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:34:58.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.639+0000 7f3f3ea00700 1 --2- 172.21.15.123:0/2050950646 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3f2406cc20 0x7f3f2406f0d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c004f50 tx=0x7f3f2c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:34:58.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.642+0000 7f3f37400700 1 -- 172.21.15.123:0/2050950646 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3f30059b50 con 0x7f3f40070360 2024-03-30T04:34:58.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:58 smithi123 ceph-mon[139369]: pgmap v87: 65 pgs: 65 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:34:58.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:58 smithi135 ceph-mon[106011]: pgmap v87: 65 pgs: 65 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:34:58.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.862+0000 7f3f46400700 1 -- 172.21.15.123:0/2050950646 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3f4010bfb0 con 0x7f3f2406cc20 2024-03-30T04:34:58.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.863+0000 7f3f37400700 1 -- 172.21.15.123:0/2050950646 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f3f4010bfb0 con 0x7f3f2406cc20 2024-03-30T04:34:58.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.865+0000 7f3f46400700 1 -- 172.21.15.123:0/2050950646 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3f2406cc20 msgr2=0x7f3f2406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:34:58.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.865+0000 7f3f46400700 1 --2- 172.21.15.123:0/2050950646 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3f2406cc20 0x7f3f2406f0d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c004f50 tx=0x7f3f2c009e30 comp rx=0 tx=0).stop 2024-03-30T04:34:58.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.865+0000 7f3f46400700 1 -- 172.21.15.123:0/2050950646 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f40070360 msgr2=0x7f3f40115ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:34:58.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.865+0000 7f3f46400700 1 --2- 172.21.15.123:0/2050950646 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f40070360 0x7f3f40115ca0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3f3000c8c0 tx=0x7f3f300044c0 comp rx=0 tx=0).stop 2024-03-30T04:34:58.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.865+0000 7f3f46400700 1 -- 172.21.15.123:0/2050950646 shutdown_connections 2024-03-30T04:34:58.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.865+0000 7f3f46400700 1 --2- 172.21.15.123:0/2050950646 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3f2406cc20 0x7f3f2406f0d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:58.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.865+0000 7f3f46400700 1 --2- 172.21.15.123:0/2050950646 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f40070360 0x7f3f40115ca0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:58.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.865+0000 7f3f46400700 1 --2- 172.21.15.123:0/2050950646 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3f40070c70 0x7f3f401161e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:58.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.865+0000 7f3f46400700 1 -- 172.21.15.123:0/2050950646 >> 172.21.15.123:0/2050950646 conn(0x7f3f400f89b0 msgr2=0x7f3f401015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:34:58.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.866+0000 7f3f46400700 1 -- 172.21.15.123:0/2050950646 shutdown_connections 2024-03-30T04:34:58.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:58.866+0000 7f3f46400700 1 -- 172.21.15.123:0/2050950646 wait complete. 2024-03-30T04:34:58.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.000+0000 7faa18800700 1 -- 172.21.15.123:0/3475491050 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa101037a0 msgr2=0x7faa100fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:34:58.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.000+0000 7faa18800700 1 --2- 172.21.15.123:0/3475491050 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa101037a0 0x7faa100fce10 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7faa00004940 tx=0x7faa00009e30 comp rx=0 tx=0).stop 2024-03-30T04:34:58.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.000+0000 7faa18800700 1 -- 172.21.15.123:0/3475491050 shutdown_connections 2024-03-30T04:34:58.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.000+0000 7faa18800700 1 --2- 172.21.15.123:0/3475491050 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa101037a0 0x7faa100fce10 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:59.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.000+0000 7faa18800700 1 --2- 172.21.15.123:0/3475491050 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa10102e00 0x7faa101031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:59.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.000+0000 7faa18800700 1 -- 172.21.15.123:0/3475491050 >> 172.21.15.123:0/3475491050 conn(0x7faa100f89b0 msgr2=0x7faa100fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:34:59.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.001+0000 7faa18800700 1 -- 172.21.15.123:0/3475491050 shutdown_connections 2024-03-30T04:34:59.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.001+0000 7faa18800700 1 -- 172.21.15.123:0/3475491050 wait complete. 2024-03-30T04:34:59.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.001+0000 7faa18800700 1 Processor -- start 2024-03-30T04:34:59.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.002+0000 7faa18800700 1 -- start start 2024-03-30T04:34:59.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.002+0000 7faa18800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa10102e00 0x7faa10115d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:34:59.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.002+0000 7faa18800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa101037a0 0x7faa10116260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:34:59.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.002+0000 7faa18800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa101168b0 con 0x7faa10102e00 2024-03-30T04:34:59.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.002+0000 7faa18800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa1010fc80 con 0x7faa101037a0 2024-03-30T04:34:59.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.002+0000 7faa15e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa10102e00 0x7faa10115d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:34:59.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.002+0000 7faa15e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa10102e00 0x7faa10115d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37276/0 (socket says 172.21.15.123:37276) 2024-03-30T04:34:59.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.002+0000 7faa15e00700 1 -- 172.21.15.123:0/2383693303 learned_addr learned my addr 172.21.15.123:0/2383693303 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:34:59.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.003+0000 7faa15400700 1 --2- 172.21.15.123:0/2383693303 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa101037a0 0x7faa10116260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:34:59.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.003+0000 7faa15e00700 1 -- 172.21.15.123:0/2383693303 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa101037a0 msgr2=0x7faa10116260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:34:59.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.003+0000 7faa15e00700 1 --2- 172.21.15.123:0/2383693303 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa101037a0 0x7faa10116260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:59.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.003+0000 7faa15e00700 1 -- 172.21.15.123:0/2383693303 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa00005040 con 0x7faa10102e00 2024-03-30T04:34:59.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.003+0000 7faa15400700 1 --2- 172.21.15.123:0/2383693303 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa101037a0 0x7faa10116260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:34:59.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.003+0000 7faa15e00700 1 --2- 172.21.15.123:0/2383693303 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa10102e00 0x7faa10115d20 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7faa04002a50 tx=0x7faa040046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:34:59.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.003+0000 7faa0a000700 1 -- 172.21.15.123:0/2383693303 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa04007cc0 con 0x7faa10102e00 2024-03-30T04:34:59.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.003+0000 7faa0a000700 1 -- 172.21.15.123:0/2383693303 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faa0400f040 con 0x7faa10102e00 2024-03-30T04:34:59.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.003+0000 7faa18800700 1 -- 172.21.15.123:0/2383693303 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa1010ff60 con 0x7faa10102e00 2024-03-30T04:34:59.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.003+0000 7faa0a000700 1 -- 172.21.15.123:0/2383693303 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa0400da70 con 0x7faa10102e00 2024-03-30T04:34:59.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.004+0000 7faa18800700 1 -- 172.21.15.123:0/2383693303 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa10110520 con 0x7faa10102e00 2024-03-30T04:34:59.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.004+0000 7faa18800700 1 -- 172.21.15.123:0/2383693303 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa100fe550 con 0x7faa10102e00 2024-03-30T04:34:59.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.005+0000 7faa0a000700 1 -- 172.21.15.123:0/2383693303 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7faa0400dc10 con 0x7faa10102e00 2024-03-30T04:34:59.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.006+0000 7faa0a000700 1 --2- 172.21.15.123:0/2383693303 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa9f806cc30 0x7fa9f806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:34:59.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.006+0000 7faa0a000700 1 -- 172.21.15.123:0/2383693303 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6324+0+0 (secure 0 0 0) 0x7faa0408b6c0 con 0x7faa10102e00 2024-03-30T04:34:59.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.006+0000 7faa15400700 1 --2- 172.21.15.123:0/2383693303 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa9f806cc30 0x7fa9f806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:34:59.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.007+0000 7faa15400700 1 --2- 172.21.15.123:0/2383693303 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa9f806cc30 0x7fa9f806f0e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7faa0000b3b0 tx=0x7faa00009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:34:59.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.011+0000 7faa0a000700 1 -- 172.21.15.123:0/2383693303 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7faa04059190 con 0x7faa10102e00 2024-03-30T04:34:59.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.222+0000 7faa18800700 1 -- 172.21.15.123:0/2383693303 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7faa1004bed0 con 0x7fa9f806cc30 2024-03-30T04:34:59.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.231+0000 7faa0a000700 1 -- 172.21.15.123:0/2383693303 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7faa1004bed0 con 0x7fa9f806cc30 2024-03-30T04:34:59.230 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:34:59.230 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (4m) 20s ago 16m 15.0M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:34:59.230 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (85s) 20s ago 16m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T04:34:59.230 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (79s) 75s ago 15m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T04:34:59.230 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (3m) 20s ago 16m 74.7M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T04:34:59.230 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (10m) 20s ago 10m 109M - 17.2.7-797-g5c024335 f77085f6aab7 2df066fba7a6 2024-03-30T04:34:59.231 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (10m) 20s ago 10m 58.0M - 17.2.7-797-g5c024335 f77085f6aab7 1293b8520db1 2024-03-30T04:34:59.231 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (10m) 75s ago 10m 67.4M - 17.2.7-797-g5c024335 f77085f6aab7 0dc41326818c 2024-03-30T04:34:59.231 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (10m) 75s ago 10m 119M - 17.2.7-797-g5c024335 f77085f6aab7 aef8a0cfc553 2024-03-30T04:34:59.231 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (6m) 20s ago 18m 497M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:34:59.231 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (5m) 75s ago 15m 448M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:34:59.231 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (2m) 20s ago 18m 44.3M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T04:34:59.231 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (2m) 75s ago 15m 33.2M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T04:34:59.231 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (4m) 20s ago 16m 7490k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:34:59.231 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (4m) 75s ago 15m 7512k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:34:59.231 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (54s) 20s ago 14m 97.9M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T04:34:59.231 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (28s) 20s ago 14m 9.86M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T04:34:59.231 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (13m) 20s ago 13m 188M 1618M 17.2.7-797-g5c024335 f77085f6aab7 4c6a727567bd 2024-03-30T04:34:59.231 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (13m) 75s ago 13m 259M 2642M 17.2.7-797-g5c024335 f77085f6aab7 6fbdd1d46a9d 2024-03-30T04:34:59.231 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (12m) 75s ago 12m 227M 2642M 17.2.7-797-g5c024335 f77085f6aab7 b74988e947e5 2024-03-30T04:34:59.231 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (12m) 75s ago 11m 220M 2642M 17.2.7-797-g5c024335 f77085f6aab7 1fd3185f5484 2024-03-30T04:34:59.232 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (4m) 20s ago 16m 51.3M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:34:59.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.234+0000 7faa18800700 1 -- 172.21.15.123:0/2383693303 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa9f806cc30 msgr2=0x7fa9f806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:34:59.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.234+0000 7faa18800700 1 --2- 172.21.15.123:0/2383693303 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa9f806cc30 0x7fa9f806f0e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7faa0000b3b0 tx=0x7faa00009e30 comp rx=0 tx=0).stop 2024-03-30T04:34:59.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.234+0000 7faa18800700 1 -- 172.21.15.123:0/2383693303 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa10102e00 msgr2=0x7faa10115d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:34:59.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.234+0000 7faa18800700 1 --2- 172.21.15.123:0/2383693303 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa10102e00 0x7faa10115d20 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7faa04002a50 tx=0x7faa040046b0 comp rx=0 tx=0).stop 2024-03-30T04:34:59.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.234+0000 7faa18800700 1 -- 172.21.15.123:0/2383693303 shutdown_connections 2024-03-30T04:34:59.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.234+0000 7faa18800700 1 --2- 172.21.15.123:0/2383693303 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa9f806cc30 0x7fa9f806f0e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:59.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.234+0000 7faa18800700 1 --2- 172.21.15.123:0/2383693303 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa10102e00 0x7faa10115d20 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:59.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.234+0000 7faa18800700 1 --2- 172.21.15.123:0/2383693303 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa101037a0 0x7faa10116260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:59.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.234+0000 7faa18800700 1 -- 172.21.15.123:0/2383693303 >> 172.21.15.123:0/2383693303 conn(0x7faa100f89b0 msgr2=0x7faa100f9a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:34:59.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.234+0000 7faa18800700 1 -- 172.21.15.123:0/2383693303 shutdown_connections 2024-03-30T04:34:59.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.234+0000 7faa18800700 1 -- 172.21.15.123:0/2383693303 wait complete. 2024-03-30T04:34:59.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.368+0000 7eff7ba00700 1 -- 172.21.15.123:0/3002920471 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff74070360 msgr2=0x7eff74070730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:34:59.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.368+0000 7eff7ba00700 1 --2- 172.21.15.123:0/3002920471 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff74070360 0x7eff74070730 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7eff68004970 tx=0x7eff68009e30 comp rx=0 tx=0).stop 2024-03-30T04:34:59.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.368+0000 7eff7ba00700 1 -- 172.21.15.123:0/3002920471 shutdown_connections 2024-03-30T04:34:59.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.368+0000 7eff7ba00700 1 --2- 172.21.15.123:0/3002920471 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7eff74070c70 0x7eff74109110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:59.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.368+0000 7eff7ba00700 1 --2- 172.21.15.123:0/3002920471 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff74070360 0x7eff74070730 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:59.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.368+0000 7eff7ba00700 1 -- 172.21.15.123:0/3002920471 >> 172.21.15.123:0/3002920471 conn(0x7eff740f89b0 msgr2=0x7eff740fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:34:59.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.368+0000 7eff7ba00700 1 -- 172.21.15.123:0/3002920471 shutdown_connections 2024-03-30T04:34:59.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.368+0000 7eff7ba00700 1 -- 172.21.15.123:0/3002920471 wait complete. 2024-03-30T04:34:59.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.369+0000 7eff7ba00700 1 Processor -- start 2024-03-30T04:34:59.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.369+0000 7eff7ba00700 1 -- start start 2024-03-30T04:34:59.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.369+0000 7eff7ba00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7eff74070360 0x7eff74115bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:34:59.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.369+0000 7eff7ba00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff74070c70 0x7eff74116110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:34:59.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.369+0000 7eff7ba00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff74116760 con 0x7eff74070c70 2024-03-30T04:34:59.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.369+0000 7eff7ba00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff7410fbe0 con 0x7eff74070360 2024-03-30T04:34:59.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.370+0000 7eff73e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff74070c70 0x7eff74116110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:34:59.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.370+0000 7eff73e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff74070c70 0x7eff74116110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37288/0 (socket says 172.21.15.123:37288) 2024-03-30T04:34:59.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.370+0000 7eff73e00700 1 -- 172.21.15.123:0/3660079623 learned_addr learned my addr 172.21.15.123:0/3660079623 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:34:59.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.370+0000 7eff79000700 1 --2- 172.21.15.123:0/3660079623 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7eff74070360 0x7eff74115bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:34:59.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.370+0000 7eff73e00700 1 -- 172.21.15.123:0/3660079623 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7eff74070360 msgr2=0x7eff74115bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:34:59.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.370+0000 7eff73e00700 1 --2- 172.21.15.123:0/3660079623 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7eff74070360 0x7eff74115bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:59.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.370+0000 7eff73e00700 1 -- 172.21.15.123:0/3660079623 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff68005040 con 0x7eff74070c70 2024-03-30T04:34:59.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.370+0000 7eff73e00700 1 --2- 172.21.15.123:0/3660079623 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff74070c70 0x7eff74116110 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7eff6000eb40 tx=0x7eff6000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:34:59.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.371+0000 7eff70c00700 1 -- 172.21.15.123:0/3660079623 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff60007cc0 con 0x7eff74070c70 2024-03-30T04:34:59.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.371+0000 7eff70c00700 1 -- 172.21.15.123:0/3660079623 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7eff6000f040 con 0x7eff74070c70 2024-03-30T04:34:59.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.371+0000 7eff7ba00700 1 -- 172.21.15.123:0/3660079623 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff7410fe60 con 0x7eff74070c70 2024-03-30T04:34:59.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.371+0000 7eff70c00700 1 -- 172.21.15.123:0/3660079623 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff6000c990 con 0x7eff74070c70 2024-03-30T04:34:59.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.371+0000 7eff7ba00700 1 -- 172.21.15.123:0/3660079623 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff741103a0 con 0x7eff74070c70 2024-03-30T04:34:59.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.372+0000 7eff7ba00700 1 -- 172.21.15.123:0/3660079623 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff74071aa0 con 0x7eff74070c70 2024-03-30T04:34:59.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.373+0000 7eff70c00700 1 -- 172.21.15.123:0/3660079623 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7eff6000caf0 con 0x7eff74070c70 2024-03-30T04:34:59.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.374+0000 7eff70c00700 1 --2- 172.21.15.123:0/3660079623 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7eff5c06cc30 0x7eff5c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:34:59.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.374+0000 7eff70c00700 1 -- 172.21.15.123:0/3660079623 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6324+0+0 (secure 0 0 0) 0x7eff6008bb80 con 0x7eff74070c70 2024-03-30T04:34:59.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.375+0000 7eff79000700 1 --2- 172.21.15.123:0/3660079623 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7eff5c06cc30 0x7eff5c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:34:59.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.375+0000 7eff79000700 1 --2- 172.21.15.123:0/3660079623 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7eff5c06cc30 0x7eff5c06f0e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7eff68009fd0 tx=0x7eff68009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:34:59.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.380+0000 7eff70c00700 1 -- 172.21.15.123:0/3660079623 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7eff60055c70 con 0x7eff74070c70 2024-03-30T04:34:59.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.691+0000 7eff7ba00700 1 -- 172.21.15.123:0/3660079623 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7eff74103b70 con 0x7eff74070c70 2024-03-30T04:34:59.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.692+0000 7eff70c00700 1 -- 172.21.15.123:0/3660079623 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7eff60059290 con 0x7eff74070c70 2024-03-30T04:34:59.691 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:34:59.691 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:34:59.691 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:34:59.691 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:34:59.691 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:34:59.691 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:34:59.691 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:34:59.691 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:34:59.691 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 4, 2024-03-30T04:34:59.691 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:34:59.692 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:34:59.692 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:34:59.692 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 4 2024-03-30T04:34:59.692 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:34:59.692 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:34:59.692 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 8, 2024-03-30T04:34:59.692 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T04:34:59.692 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:34:59.692 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:34:59.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.694+0000 7eff7ba00700 1 -- 172.21.15.123:0/3660079623 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7eff5c06cc30 msgr2=0x7eff5c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:34:59.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.694+0000 7eff7ba00700 1 --2- 172.21.15.123:0/3660079623 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7eff5c06cc30 0x7eff5c06f0e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7eff68009fd0 tx=0x7eff68009e30 comp rx=0 tx=0).stop 2024-03-30T04:34:59.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.694+0000 7eff7ba00700 1 -- 172.21.15.123:0/3660079623 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff74070c70 msgr2=0x7eff74116110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:34:59.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.694+0000 7eff7ba00700 1 --2- 172.21.15.123:0/3660079623 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff74070c70 0x7eff74116110 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7eff6000eb40 tx=0x7eff6000b040 comp rx=0 tx=0).stop 2024-03-30T04:34:59.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.695+0000 7eff7ba00700 1 -- 172.21.15.123:0/3660079623 shutdown_connections 2024-03-30T04:34:59.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.695+0000 7eff7ba00700 1 --2- 172.21.15.123:0/3660079623 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7eff5c06cc30 0x7eff5c06f0e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:59.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.695+0000 7eff7ba00700 1 --2- 172.21.15.123:0/3660079623 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7eff74070360 0x7eff74115bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:59.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.695+0000 7eff7ba00700 1 --2- 172.21.15.123:0/3660079623 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff74070c70 0x7eff74116110 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:59.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.695+0000 7eff7ba00700 1 -- 172.21.15.123:0/3660079623 >> 172.21.15.123:0/3660079623 conn(0x7eff740f89b0 msgr2=0x7eff741015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:34:59.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.695+0000 7eff7ba00700 1 -- 172.21.15.123:0/3660079623 shutdown_connections 2024-03-30T04:34:59.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.695+0000 7eff7ba00700 1 -- 172.21.15.123:0/3660079623 wait complete. 2024-03-30T04:34:59.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:59 smithi123 ceph-mon[139369]: from='client.34252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:34:59.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:59 smithi123 ceph-mon[139369]: from='client.34256 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:34:59.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:59 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-30T04:34:59.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:59 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:34:59.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:34:59 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:34:59.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:59 smithi135 ceph-mon[106011]: from='client.34252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:34:59.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:59 smithi135 ceph-mon[106011]: from='client.34256 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:34:59.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:59 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-30T04:34:59.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:59 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:34:59.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:34:59 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:34:59.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.853+0000 7f1ee0400700 1 -- 172.21.15.123:0/514927868 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ed8102e20 msgr2=0x7f1ed81031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:34:59.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.853+0000 7f1ee0400700 1 --2- 172.21.15.123:0/514927868 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ed8102e20 0x7f1ed81031f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc004940 tx=0x7f1ecc009e30 comp rx=0 tx=0).stop 2024-03-30T04:34:59.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.853+0000 7f1ee0400700 1 -- 172.21.15.123:0/514927868 shutdown_connections 2024-03-30T04:34:59.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.853+0000 7f1ee0400700 1 --2- 172.21.15.123:0/514927868 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1ed81037c0 0x7f1ed80fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:59.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.853+0000 7f1ee0400700 1 --2- 172.21.15.123:0/514927868 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ed8102e20 0x7f1ed81031f0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:59.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.853+0000 7f1ee0400700 1 -- 172.21.15.123:0/514927868 >> 172.21.15.123:0/514927868 conn(0x7f1ed80f89d0 msgr2=0x7f1ed80fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:34:59.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.853+0000 7f1ee0400700 1 -- 172.21.15.123:0/514927868 shutdown_connections 2024-03-30T04:34:59.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.853+0000 7f1ee0400700 1 -- 172.21.15.123:0/514927868 wait complete. 2024-03-30T04:34:59.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.854+0000 7f1ee0400700 1 Processor -- start 2024-03-30T04:34:59.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.854+0000 7f1ee0400700 1 -- start start 2024-03-30T04:34:59.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.854+0000 7f1ee0400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ed8102e20 0x7f1ed8115c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:34:59.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.854+0000 7f1ee0400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1ed81037c0 0x7f1ed81161d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:34:59.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.854+0000 7f1ee0400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ed8116820 con 0x7f1ed8102e20 2024-03-30T04:34:59.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.854+0000 7f1ee0400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ed810fbf0 con 0x7f1ed81037c0 2024-03-30T04:34:59.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.855+0000 7f1edda00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ed8102e20 0x7f1ed8115c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:34:59.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.855+0000 7f1edda00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ed8102e20 0x7f1ed8115c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37318/0 (socket says 172.21.15.123:37318) 2024-03-30T04:34:59.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.855+0000 7f1edda00700 1 -- 172.21.15.123:0/3928423898 learned_addr learned my addr 172.21.15.123:0/3928423898 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:34:59.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.855+0000 7f1edda00700 1 -- 172.21.15.123:0/3928423898 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1ed81037c0 msgr2=0x7f1ed81161d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:34:59.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.855+0000 7f1edda00700 1 --2- 172.21.15.123:0/3928423898 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1ed81037c0 0x7f1ed81161d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:34:59.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.855+0000 7f1edda00700 1 -- 172.21.15.123:0/3928423898 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ecc005040 con 0x7f1ed8102e20 2024-03-30T04:34:59.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.855+0000 7f1edda00700 1 --2- 172.21.15.123:0/3928423898 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ed8102e20 0x7f1ed8115c90 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc007f50 tx=0x7f1ecc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:34:59.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.855+0000 7f1ed5600700 1 -- 172.21.15.123:0/3928423898 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ecc021070 con 0x7f1ed8102e20 2024-03-30T04:34:59.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.855+0000 7f1ed5600700 1 -- 172.21.15.123:0/3928423898 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1ecc013e00 con 0x7f1ed8102e20 2024-03-30T04:34:59.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.855+0000 7f1ee0400700 1 -- 172.21.15.123:0/3928423898 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ed810fe70 con 0x7f1ed8102e20 2024-03-30T04:34:59.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.856+0000 7f1ed5600700 1 -- 172.21.15.123:0/3928423898 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ecc00e070 con 0x7f1ed8102e20 2024-03-30T04:34:59.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.856+0000 7f1ee0400700 1 -- 172.21.15.123:0/3928423898 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ed8110430 con 0x7f1ed8102e20 2024-03-30T04:34:59.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.857+0000 7f1ee0400700 1 -- 172.21.15.123:0/3928423898 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ed80fe570 con 0x7f1ed8102e20 2024-03-30T04:34:59.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.858+0000 7f1ed5600700 1 -- 172.21.15.123:0/3928423898 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f1ecc013950 con 0x7f1ed8102e20 2024-03-30T04:34:59.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.858+0000 7f1ed5600700 1 --2- 172.21.15.123:0/3928423898 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1ec006cc30 0x7f1ec006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:34:59.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.859+0000 7f1ed5600700 1 -- 172.21.15.123:0/3928423898 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f1ecc08d1e0 con 0x7f1ed8102e20 2024-03-30T04:34:59.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.859+0000 7f1edd000700 1 --2- 172.21.15.123:0/3928423898 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1ec006cc30 0x7f1ec006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:34:59.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.859+0000 7f1edd000700 1 --2- 172.21.15.123:0/3928423898 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1ec006cc30 0x7f1ec006f0e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1ed8111530 tx=0x7f1ec400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:34:59.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:34:59.863+0000 7f1ed5600700 1 -- 172.21.15.123:0/3928423898 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f1ecc05ad30 con 0x7f1ed8102e20 2024-03-30T04:35:00.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.121+0000 7f1ee0400700 1 -- 172.21.15.123:0/3928423898 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f1ed804bed0 con 0x7f1ed8102e20 2024-03-30T04:35:00.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.121+0000 7f1ed5600700 1 -- 172.21.15.123:0/3928423898 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 23 v23) v1 ==== 76+0+1876 (secure 0 0 0) 0x7f1ecc05a8c0 con 0x7f1ed8102e20 2024-03-30T04:35:00.120 INFO:teuthology.orchestra.run.smithi123.stdout:e23 2024-03-30T04:35:00.120 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:35:00.120 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:35:00.120 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:35:00.121 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:35:00.121 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:35:00.121 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:35:00.121 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 23 2024-03-30T04:35:00.121 INFO:teuthology.orchestra.run.smithi123.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-30T04:35:00.121 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:35:00.121 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:32:48.448604+0000 2024-03-30T04:35:00.121 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:35:00.121 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:35:00.121 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:35:00.121 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:35:00.121 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:35:00.121 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T04:35:00.121 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:35:00.121 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:35:00.122 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 46 2024-03-30T04:35:00.122 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:35:00.122 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T04:35:00.122 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T04:35:00.122 INFO:teuthology.orchestra.run.smithi123.stdout:up {0=14488,1=24309} 2024-03-30T04:35:00.122 INFO:teuthology.orchestra.run.smithi123.stdout:failed 2024-03-30T04:35:00.122 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:35:00.122 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:35:00.122 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:35:00.122 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:35:00.122 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:35:00.122 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:35:00.122 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T04:35:00.123 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T04:35:00.123 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{0:14488} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.123:6828/3037849622,v1:172.21.15.123:6829/3037849622] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:35:00.123 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{0:14500} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/2433675995,v1:172.21.15.123:6827/2433675995] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:35:00.123 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{1:24309} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.135:6824/4050541062,v1:172.21.15.135:6825/4050541062] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:35:00.123 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{1:24319} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.135:6826/1096986122,v1:172.21.15.135:6827/1096986122] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:35:00.123 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:35:00.123 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:35:00.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.124+0000 7f1ee0400700 1 -- 172.21.15.123:0/3928423898 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1ec006cc30 msgr2=0x7f1ec006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:35:00.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.124+0000 7f1ee0400700 1 --2- 172.21.15.123:0/3928423898 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1ec006cc30 0x7f1ec006f0e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1ed8111530 tx=0x7f1ec400b040 comp rx=0 tx=0).stop 2024-03-30T04:35:00.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.124+0000 7f1ee0400700 1 -- 172.21.15.123:0/3928423898 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ed8102e20 msgr2=0x7f1ed8115c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:35:00.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.124+0000 7f1ee0400700 1 --2- 172.21.15.123:0/3928423898 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ed8102e20 0x7f1ed8115c90 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc007f50 tx=0x7f1ecc009e30 comp rx=0 tx=0).stop 2024-03-30T04:35:00.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.124+0000 7f1ee0400700 1 -- 172.21.15.123:0/3928423898 shutdown_connections 2024-03-30T04:35:00.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.124+0000 7f1ee0400700 1 --2- 172.21.15.123:0/3928423898 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1ec006cc30 0x7f1ec006f0e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:00.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.124+0000 7f1ee0400700 1 --2- 172.21.15.123:0/3928423898 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ed8102e20 0x7f1ed8115c90 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:00.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.124+0000 7f1ee0400700 1 --2- 172.21.15.123:0/3928423898 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1ed81037c0 0x7f1ed81161d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:00.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.124+0000 7f1ee0400700 1 -- 172.21.15.123:0/3928423898 >> 172.21.15.123:0/3928423898 conn(0x7f1ed80f89d0 msgr2=0x7f1ed80f99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:35:00.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.124+0000 7f1ee0400700 1 -- 172.21.15.123:0/3928423898 shutdown_connections 2024-03-30T04:35:00.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.124+0000 7f1ee0400700 1 -- 172.21.15.123:0/3928423898 wait complete. 2024-03-30T04:35:00.124 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 23 2024-03-30T04:35:00.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.265+0000 7f7e33200700 1 -- 172.21.15.123:0/1342160320 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7e341035f0 msgr2=0x7f7e340fcc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:35:00.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.265+0000 7f7e33200700 1 --2- 172.21.15.123:0/1342160320 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7e341035f0 0x7f7e340fcc60 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7e20004940 tx=0x7f7e20009e30 comp rx=0 tx=0).stop 2024-03-30T04:35:00.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.265+0000 7f7e33200700 1 -- 172.21.15.123:0/1342160320 shutdown_connections 2024-03-30T04:35:00.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.265+0000 7f7e33200700 1 --2- 172.21.15.123:0/1342160320 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7e341035f0 0x7f7e340fcc60 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:00.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.265+0000 7f7e33200700 1 --2- 172.21.15.123:0/1342160320 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7e34102c50 0x7f7e34103020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:00.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.265+0000 7f7e33200700 1 -- 172.21.15.123:0/1342160320 >> 172.21.15.123:0/1342160320 conn(0x7f7e340f87e0 msgr2=0x7f7e340fabf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:35:00.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.265+0000 7f7e33200700 1 -- 172.21.15.123:0/1342160320 shutdown_connections 2024-03-30T04:35:00.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.265+0000 7f7e33200700 1 -- 172.21.15.123:0/1342160320 wait complete. 2024-03-30T04:35:00.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.266+0000 7f7e33200700 1 Processor -- start 2024-03-30T04:35:00.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.266+0000 7f7e33200700 1 -- start start 2024-03-30T04:35:00.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.266+0000 7f7e33200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7e34102c50 0x7f7e34113940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:35:00.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.266+0000 7f7e33200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7e341035f0 0x7f7e34113e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:35:00.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.266+0000 7f7e33200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e341144d0 con 0x7f7e34102c50 2024-03-30T04:35:00.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.266+0000 7f7e33200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3410d8c0 con 0x7f7e341035f0 2024-03-30T04:35:00.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.266+0000 7f7e31e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7e34102c50 0x7f7e34113940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:35:00.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.266+0000 7f7e31e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7e34102c50 0x7f7e34113940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37334/0 (socket says 172.21.15.123:37334) 2024-03-30T04:35:00.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.266+0000 7f7e31e00700 1 -- 172.21.15.123:0/3119250910 learned_addr learned my addr 172.21.15.123:0/3119250910 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:35:00.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.267+0000 7f7e31400700 1 --2- 172.21.15.123:0/3119250910 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7e341035f0 0x7f7e34113e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:35:00.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.267+0000 7f7e31e00700 1 -- 172.21.15.123:0/3119250910 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7e341035f0 msgr2=0x7f7e34113e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:35:00.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.267+0000 7f7e31e00700 1 --2- 172.21.15.123:0/3119250910 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7e341035f0 0x7f7e34113e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:00.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.267+0000 7f7e31e00700 1 -- 172.21.15.123:0/3119250910 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e20005040 con 0x7f7e34102c50 2024-03-30T04:35:00.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.267+0000 7f7e31400700 1 --2- 172.21.15.123:0/3119250910 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7e341035f0 0x7f7e34113e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:35:00.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.267+0000 7f7e31e00700 1 --2- 172.21.15.123:0/3119250910 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7e34102c50 0x7f7e34113940 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7e2400c8c0 tx=0x7f7e240044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:35:00.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.267+0000 7f7e2a000700 1 -- 172.21.15.123:0/3119250910 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e2400a980 con 0x7f7e34102c50 2024-03-30T04:35:00.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.267+0000 7f7e2a000700 1 -- 172.21.15.123:0/3119250910 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7e2400ce60 con 0x7f7e34102c50 2024-03-30T04:35:00.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.267+0000 7f7e2a000700 1 -- 172.21.15.123:0/3119250910 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e240113a0 con 0x7f7e34102c50 2024-03-30T04:35:00.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.267+0000 7f7e33200700 1 -- 172.21.15.123:0/3119250910 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e3410dba0 con 0x7f7e34102c50 2024-03-30T04:35:00.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.267+0000 7f7e33200700 1 -- 172.21.15.123:0/3119250910 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e3410e130 con 0x7f7e34102c50 2024-03-30T04:35:00.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.268+0000 7f7e33200700 1 -- 172.21.15.123:0/3119250910 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e340fe3a0 con 0x7f7e34102c50 2024-03-30T04:35:00.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.269+0000 7f7e2a000700 1 -- 172.21.15.123:0/3119250910 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f7e24012400 con 0x7f7e34102c50 2024-03-30T04:35:00.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.270+0000 7f7e2a000700 1 --2- 172.21.15.123:0/3119250910 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7e1806cc30 0x7f7e1806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:35:00.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.270+0000 7f7e2a000700 1 -- 172.21.15.123:0/3119250910 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f7e2408c150 con 0x7f7e34102c50 2024-03-30T04:35:00.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.270+0000 7f7e31400700 1 --2- 172.21.15.123:0/3119250910 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7e1806cc30 0x7f7e1806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:35:00.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.271+0000 7f7e31400700 1 --2- 172.21.15.123:0/3119250910 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7e1806cc30 0x7f7e1806f0e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7e3410eba0 tx=0x7f7e20009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:35:00.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.274+0000 7f7e2a000700 1 -- 172.21.15.123:0/3119250910 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7e24059c20 con 0x7f7e34102c50 2024-03-30T04:35:00.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.509+0000 7f7e33200700 1 -- 172.21.15.123:0/3119250910 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7e34109be0 con 0x7f7e1806cc30 2024-03-30T04:35:00.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.510+0000 7f7e2a000700 1 -- 172.21.15.123:0/3119250910 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f7e34109be0 con 0x7f7e1806cc30 2024-03-30T04:35:00.509 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:35:00.509 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:35:00.509 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:35:00.509 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T04:35:00.509 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:35:00.509 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T04:35:00.510 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T04:35:00.510 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T04:35:00.510 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:35:00.510 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "8/21 daemons upgraded", 2024-03-30T04:35:00.510 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading osd daemons", 2024-03-30T04:35:00.510 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:35:00.510 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:35:00.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.513+0000 7f7e33200700 1 -- 172.21.15.123:0/3119250910 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7e1806cc30 msgr2=0x7f7e1806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:35:00.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.513+0000 7f7e33200700 1 --2- 172.21.15.123:0/3119250910 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7e1806cc30 0x7f7e1806f0e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7e3410eba0 tx=0x7f7e20009e30 comp rx=0 tx=0).stop 2024-03-30T04:35:00.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.513+0000 7f7e33200700 1 -- 172.21.15.123:0/3119250910 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7e34102c50 msgr2=0x7f7e34113940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:35:00.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.513+0000 7f7e33200700 1 --2- 172.21.15.123:0/3119250910 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7e34102c50 0x7f7e34113940 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7e2400c8c0 tx=0x7f7e240044c0 comp rx=0 tx=0).stop 2024-03-30T04:35:00.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.513+0000 7f7e33200700 1 -- 172.21.15.123:0/3119250910 shutdown_connections 2024-03-30T04:35:00.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.513+0000 7f7e33200700 1 --2- 172.21.15.123:0/3119250910 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7e1806cc30 0x7f7e1806f0e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:00.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.513+0000 7f7e33200700 1 --2- 172.21.15.123:0/3119250910 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7e34102c50 0x7f7e34113940 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:00.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.513+0000 7f7e33200700 1 --2- 172.21.15.123:0/3119250910 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7e341035f0 0x7f7e34113e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:00.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.513+0000 7f7e33200700 1 -- 172.21.15.123:0/3119250910 >> 172.21.15.123:0/3119250910 conn(0x7f7e340f87e0 msgr2=0x7f7e340f9900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:35:00.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.513+0000 7f7e33200700 1 -- 172.21.15.123:0/3119250910 shutdown_connections 2024-03-30T04:35:00.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.514+0000 7f7e33200700 1 -- 172.21.15.123:0/3119250910 wait complete. 2024-03-30T04:35:00.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.649+0000 7ff826800700 1 -- 172.21.15.123:0/640312760 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff820102e00 msgr2=0x7ff8201031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:35:00.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.649+0000 7ff826800700 1 --2- 172.21.15.123:0/640312760 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff820102e00 0x7ff8201031d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff810004940 tx=0x7ff810009e30 comp rx=0 tx=0).stop 2024-03-30T04:35:00.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.649+0000 7ff826800700 1 -- 172.21.15.123:0/640312760 shutdown_connections 2024-03-30T04:35:00.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.649+0000 7ff826800700 1 --2- 172.21.15.123:0/640312760 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8201037a0 0x7ff8200fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:00.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.649+0000 7ff826800700 1 --2- 172.21.15.123:0/640312760 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff820102e00 0x7ff8201031d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:00.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.649+0000 7ff826800700 1 -- 172.21.15.123:0/640312760 >> 172.21.15.123:0/640312760 conn(0x7ff8200f89b0 msgr2=0x7ff8200fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:35:00.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.649+0000 7ff826800700 1 -- 172.21.15.123:0/640312760 shutdown_connections 2024-03-30T04:35:00.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.649+0000 7ff826800700 1 -- 172.21.15.123:0/640312760 wait complete. 2024-03-30T04:35:00.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.650+0000 7ff826800700 1 Processor -- start 2024-03-30T04:35:00.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.650+0000 7ff826800700 1 -- start start 2024-03-30T04:35:00.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.651+0000 7ff826800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff820102e00 0x7ff8200702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:35:00.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.651+0000 7ff826800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8201037a0 0x7ff820070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:35:00.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.651+0000 7ff826800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff820070d90 con 0x7ff8201037a0 2024-03-30T04:35:00.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.651+0000 7ff826800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff820070ed0 con 0x7ff820102e00 2024-03-30T04:35:00.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.651+0000 7ff81ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8201037a0 0x7ff820070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:35:00.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.651+0000 7ff81ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8201037a0 0x7ff820070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37348/0 (socket says 172.21.15.123:37348) 2024-03-30T04:35:00.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.651+0000 7ff81ea00700 1 -- 172.21.15.123:0/843561078 learned_addr learned my addr 172.21.15.123:0/843561078 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:35:00.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.651+0000 7ff81f400700 1 --2- 172.21.15.123:0/843561078 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff820102e00 0x7ff8200702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:35:00.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.651+0000 7ff81ea00700 1 -- 172.21.15.123:0/843561078 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff820102e00 msgr2=0x7ff8200702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:35:00.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.651+0000 7ff81ea00700 1 --2- 172.21.15.123:0/843561078 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff820102e00 0x7ff8200702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:00.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.651+0000 7ff81ea00700 1 -- 172.21.15.123:0/843561078 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff810005040 con 0x7ff8201037a0 2024-03-30T04:35:00.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.652+0000 7ff81ea00700 1 --2- 172.21.15.123:0/843561078 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8201037a0 0x7ff820070800 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff80c002a50 tx=0x7ff80c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:35:00.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.652+0000 7ff817400700 1 -- 172.21.15.123:0/843561078 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff80c007cc0 con 0x7ff8201037a0 2024-03-30T04:35:00.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.652+0000 7ff826800700 1 -- 172.21.15.123:0/843561078 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff820119df0 con 0x7ff8201037a0 2024-03-30T04:35:00.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.652+0000 7ff817400700 1 -- 172.21.15.123:0/843561078 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff80c00f040 con 0x7ff8201037a0 2024-03-30T04:35:00.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.652+0000 7ff817400700 1 -- 172.21.15.123:0/843561078 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff80c00d920 con 0x7ff8201037a0 2024-03-30T04:35:00.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.652+0000 7ff826800700 1 -- 172.21.15.123:0/843561078 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff82011a270 con 0x7ff8201037a0 2024-03-30T04:35:00.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.653+0000 7ff826800700 1 -- 172.21.15.123:0/843561078 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8200fe550 con 0x7ff8201037a0 2024-03-30T04:35:00.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.654+0000 7ff817400700 1 -- 172.21.15.123:0/843561078 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7ff80c009680 con 0x7ff8201037a0 2024-03-30T04:35:00.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.655+0000 7ff817400700 1 --2- 172.21.15.123:0/843561078 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ff80406cc30 0x7ff80406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:35:00.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.655+0000 7ff817400700 1 -- 172.21.15.123:0/843561078 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6324+0+0 (secure 0 0 0) 0x7ff80c08bd00 con 0x7ff8201037a0 2024-03-30T04:35:00.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.655+0000 7ff81f400700 1 --2- 172.21.15.123:0/843561078 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ff80406cc30 0x7ff80406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:35:00.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.656+0000 7ff81f400700 1 --2- 172.21.15.123:0/843561078 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ff80406cc30 0x7ff80406f0e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff81000b840 tx=0x7ff810009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:35:00.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.660+0000 7ff817400700 1 -- 172.21.15.123:0/843561078 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff80c055d40 con 0x7ff8201037a0 2024-03-30T04:35:00.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:00 smithi123 ceph-mon[139369]: from='client.34260 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:35:00.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:00 smithi123 ceph-mon[139369]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-30T04:35:00.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:00 smithi123 ceph-mon[139369]: Upgrade: osd.2 is safe to restart 2024-03-30T04:35:00.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:00 smithi123 ceph-mon[139369]: pgmap v88: 65 pgs: 65 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:35:00.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:00 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3660079623' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:35:00.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:00 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3928423898' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:35:00.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:00 smithi135 ceph-mon[106011]: from='client.34260 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:35:00.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:00 smithi135 ceph-mon[106011]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-30T04:35:00.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:00 smithi135 ceph-mon[106011]: Upgrade: osd.2 is safe to restart 2024-03-30T04:35:00.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:00 smithi135 ceph-mon[106011]: pgmap v88: 65 pgs: 65 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:35:00.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:00 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3660079623' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:35:00.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:00 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3928423898' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:35:00.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.957+0000 7ff826800700 1 -- 172.21.15.123:0/843561078 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7ff82005f5f0 con 0x7ff8201037a0 2024-03-30T04:35:00.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.958+0000 7ff817400700 1 -- 172.21.15.123:0/843561078 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7ff80c059360 con 0x7ff8201037a0 2024-03-30T04:35:00.956 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_OK 2024-03-30T04:35:00.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.960+0000 7ff826800700 1 -- 172.21.15.123:0/843561078 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ff80406cc30 msgr2=0x7ff80406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:35:00.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.960+0000 7ff826800700 1 --2- 172.21.15.123:0/843561078 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ff80406cc30 0x7ff80406f0e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff81000b840 tx=0x7ff810009e30 comp rx=0 tx=0).stop 2024-03-30T04:35:00.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.960+0000 7ff826800700 1 -- 172.21.15.123:0/843561078 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8201037a0 msgr2=0x7ff820070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:35:00.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.960+0000 7ff826800700 1 --2- 172.21.15.123:0/843561078 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8201037a0 0x7ff820070800 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff80c002a50 tx=0x7ff80c004600 comp rx=0 tx=0).stop 2024-03-30T04:35:00.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.960+0000 7ff826800700 1 -- 172.21.15.123:0/843561078 shutdown_connections 2024-03-30T04:35:00.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.960+0000 7ff826800700 1 --2- 172.21.15.123:0/843561078 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ff80406cc30 0x7ff80406f0e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:00.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.960+0000 7ff826800700 1 --2- 172.21.15.123:0/843561078 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff820102e00 0x7ff8200702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:00.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.960+0000 7ff826800700 1 --2- 172.21.15.123:0/843561078 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8201037a0 0x7ff820070800 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:00.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.960+0000 7ff826800700 1 -- 172.21.15.123:0/843561078 >> 172.21.15.123:0/843561078 conn(0x7ff8200f89b0 msgr2=0x7ff8200f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:35:00.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.960+0000 7ff826800700 1 -- 172.21.15.123:0/843561078 shutdown_connections 2024-03-30T04:35:00.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:00.960+0000 7ff826800700 1 -- 172.21.15.123:0/843561078 wait complete. 2024-03-30T04:35:01.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:01 smithi123 ceph-mon[139369]: from='client.34272 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:35:01.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:01 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/843561078' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:35:01.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:01 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:35:01.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:01 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-30T04:35:01.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:01 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:35:01.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:01 smithi135 ceph-mon[106011]: from='client.34272 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:35:01.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:01 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/843561078' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:35:01.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:01 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:35:01.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:01 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-30T04:35:01.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:01 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:35:02.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:02 smithi123 ceph-mon[139369]: Upgrade: Updating osd.2 2024-03-30T04:35:02.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:02 smithi123 ceph-mon[139369]: Deploying daemon osd.2 on smithi123 2024-03-30T04:35:02.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:02 smithi123 ceph-mon[139369]: pgmap v89: 65 pgs: 65 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:35:02.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:02 smithi135 ceph-mon[106011]: Upgrade: Updating osd.2 2024-03-30T04:35:02.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:02 smithi135 ceph-mon[106011]: Deploying daemon osd.2 on smithi123 2024-03-30T04:35:02.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:02 smithi135 ceph-mon[106011]: pgmap v89: 65 pgs: 65 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:35:04.721 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:04 smithi123 ceph-mon[139369]: pgmap v90: 65 pgs: 65 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:35:04.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:04 smithi135 ceph-mon[106011]: pgmap v90: 65 pgs: 65 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:35:05.716 INFO:journalctl@ceph.osd.2.smithi123.stdout:Mar 30 04:35:05 smithi123 systemd[1]: Stopping Ceph osd.2 for 2e77a2be-ee4c-11ee-b647-cb9ed24678a4... 2024-03-30T04:35:06.466 INFO:journalctl@ceph.osd.2.smithi123.stdout:Mar 30 04:35:06 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-2[77012]: 2024-03-30T04:35:06.049+0000 7fb24d400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T04:35:06.466 INFO:journalctl@ceph.osd.2.smithi123.stdout:Mar 30 04:35:06 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-2[77012]: 2024-03-30T04:35:06.049+0000 7fb24d400700 -1 osd.2 59 *** Got signal Terminated *** 2024-03-30T04:35:06.467 INFO:journalctl@ceph.osd.2.smithi123.stdout:Mar 30 04:35:06 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-2[77012]: 2024-03-30T04:35:06.049+0000 7fb24d400700 -1 osd.2 59 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T04:35:06.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:06 smithi135 ceph-mon[106011]: pgmap v91: 65 pgs: 65 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:35:06.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:06 smithi135 ceph-mon[106011]: osd.2 marked itself down and dead 2024-03-30T04:35:06.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:06 smithi123 ceph-mon[139369]: pgmap v91: 65 pgs: 65 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:35:06.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:06 smithi123 ceph-mon[139369]: osd.2 marked itself down and dead 2024-03-30T04:35:07.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:07 smithi135 ceph-mon[106011]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-30T04:35:07.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:07 smithi135 ceph-mon[106011]: osdmap e60: 6 total, 5 up, 6 in 2024-03-30T04:35:07.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:07 smithi123 ceph-mon[139369]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-30T04:35:07.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:07 smithi123 ceph-mon[139369]: osdmap e60: 6 total, 5 up, 6 in 2024-03-30T04:35:07.967 INFO:journalctl@ceph.osd.2.smithi123.stdout:Mar 30 04:35:07 smithi123 bash[166425]: ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-2 2024-03-30T04:35:08.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:08 smithi135 ceph-mon[106011]: pgmap v93: 65 pgs: 9 stale+active+clean, 56 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-30T04:35:08.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:08 smithi135 ceph-mon[106011]: osdmap e61: 6 total, 5 up, 6 in 2024-03-30T04:35:08.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:08 smithi123 ceph-mon[139369]: pgmap v93: 65 pgs: 9 stale+active+clean, 56 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-30T04:35:08.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:08 smithi123 ceph-mon[139369]: osdmap e61: 6 total, 5 up, 6 in 2024-03-30T04:35:10.203 INFO:journalctl@ceph.osd.2.smithi123.stdout:Mar 30 04:35:09 smithi123 systemd[1]: ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4@osd.2.service: Succeeded. 2024-03-30T04:35:10.204 INFO:journalctl@ceph.osd.2.smithi123.stdout:Mar 30 04:35:09 smithi123 systemd[1]: Stopped Ceph osd.2 for 2e77a2be-ee4c-11ee-b647-cb9ed24678a4. 2024-03-30T04:35:10.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:10 smithi123 ceph-mon[139369]: pgmap v95: 65 pgs: 3 active+undersized+degraded, 5 stale+active+clean, 4 active+undersized, 53 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 11/324 objects degraded (3.395%) 2024-03-30T04:35:10.467 INFO:journalctl@ceph.osd.2.smithi123.stdout:Mar 30 04:35:10 smithi123 systemd[1]: Starting Ceph osd.2 for 2e77a2be-ee4c-11ee-b647-cb9ed24678a4... 2024-03-30T04:35:10.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:10 smithi135 ceph-mon[106011]: pgmap v95: 65 pgs: 3 active+undersized+degraded, 5 stale+active+clean, 4 active+undersized, 53 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 11/324 objects degraded (3.395%) 2024-03-30T04:35:11.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:11 smithi135 ceph-mon[106011]: Health check failed: Degraded data redundancy: 11/324 objects degraded (3.395%), 3 pgs degraded (PG_DEGRADED) 2024-03-30T04:35:11.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:11 smithi123 ceph-mon[139369]: Health check failed: Degraded data redundancy: 11/324 objects degraded (3.395%), 3 pgs degraded (PG_DEGRADED) 2024-03-30T04:35:11.716 INFO:journalctl@ceph.osd.2.smithi123.stdout:Mar 30 04:35:11 smithi123 bash[166699]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-30T04:35:11.716 INFO:journalctl@ceph.osd.2.smithi123.stdout:Mar 30 04:35:11 smithi123 bash[166699]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-30T04:35:11.717 INFO:journalctl@ceph.osd.2.smithi123.stdout:Mar 30 04:35:11 smithi123 bash[166699]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-30T04:35:11.717 INFO:journalctl@ceph.osd.2.smithi123.stdout:Mar 30 04:35:11 smithi123 bash[166699]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-30T04:35:11.717 INFO:journalctl@ceph.osd.2.smithi123.stdout:Mar 30 04:35:11 smithi123 bash[166699]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-03-30T04:35:11.717 INFO:journalctl@ceph.osd.2.smithi123.stdout:Mar 30 04:35:11 smithi123 bash[166699]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-30T04:35:11.717 INFO:journalctl@ceph.osd.2.smithi123.stdout:Mar 30 04:35:11 smithi123 bash[166699]: --> ceph-volume raw activate successful for osd ID: 2 2024-03-30T04:35:12.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:12 smithi135 ceph-mon[106011]: pgmap v96: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 39/324 objects degraded (12.037%) 2024-03-30T04:35:12.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:12 smithi123 ceph-mon[139369]: pgmap v96: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 39/324 objects degraded (12.037%) 2024-03-30T04:35:13.216 INFO:journalctl@ceph.osd.2.smithi123.stdout:Mar 30 04:35:12 smithi123 ceph-osd[167011]: -- 172.21.15.123:0/2493549641 <== mon.0 v2:172.21.15.123:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55cc831871e0 con 0x55cc83156c00 2024-03-30T04:35:13.216 INFO:journalctl@ceph.osd.2.smithi123.stdout:Mar 30 04:35:12 smithi123 bash[166974]: 06673576e943f344f9c0f77cc5d7cc255170743d9b09d1a5d34e512d367c9c9e 2024-03-30T04:35:13.216 INFO:journalctl@ceph.osd.2.smithi123.stdout:Mar 30 04:35:12 smithi123 systemd[1]: Started Ceph osd.2 for 2e77a2be-ee4c-11ee-b647-cb9ed24678a4. 2024-03-30T04:35:13.716 INFO:journalctl@ceph.osd.2.smithi123.stdout:Mar 30 04:35:13 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-2[166988]: 2024-03-30T04:35:13.415+0000 7fd313d4d640 -1 Falling back to public interface 2024-03-30T04:35:14.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:14 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:35:14.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:14 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:35:14.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:14 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:35:14.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:14 smithi135 ceph-mon[106011]: pgmap v97: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 39/324 objects degraded (12.037%) 2024-03-30T04:35:14.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:14 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:35:14.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:14 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:35:14.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:14 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:35:14.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:14 smithi123 ceph-mon[139369]: pgmap v97: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 39/324 objects degraded (12.037%) 2024-03-30T04:35:15.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:15 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:35:15.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:15 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:35:15.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:15 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:35:15.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:15 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:35:16.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:16 smithi123 ceph-mon[139369]: pgmap v98: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 39/324 objects degraded (12.037%) 2024-03-30T04:35:16.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:16 smithi135 ceph-mon[106011]: pgmap v98: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 39/324 objects degraded (12.037%) 2024-03-30T04:35:18.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:18 smithi123 ceph-mon[139369]: pgmap v99: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 39/324 objects degraded (12.037%) 2024-03-30T04:35:18.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:18 smithi123 ceph-mon[139369]: from='osd.2 [v2:172.21.15.123:6818/3356591260,v1:172.21.15.123:6819/3356591260]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-30T04:35:18.717 INFO:journalctl@ceph.osd.2.smithi123.stdout:Mar 30 04:35:18 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-2[166988]: 2024-03-30T04:35:18.400+0000 7fd313d4d640 -1 osd.2 59 log_to_monitors true 2024-03-30T04:35:18.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:18 smithi135 ceph-mon[106011]: pgmap v99: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 39/324 objects degraded (12.037%) 2024-03-30T04:35:18.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:18 smithi135 ceph-mon[106011]: from='osd.2 [v2:172.21.15.123:6818/3356591260,v1:172.21.15.123:6819/3356591260]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-30T04:35:19.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:19 smithi123 ceph-mon[139369]: from='osd.2 [v2:172.21.15.123:6818/3356591260,v1:172.21.15.123:6819/3356591260]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-30T04:35:19.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:19 smithi123 ceph-mon[139369]: osdmap e62: 6 total, 5 up, 6 in 2024-03-30T04:35:19.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:19 smithi123 ceph-mon[139369]: from='osd.2 [v2:172.21.15.123:6818/3356591260,v1:172.21.15.123:6819/3356591260]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-03-30T04:35:19.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:19 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:35:19.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:19 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:35:19.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:19 smithi123 ceph-mon[139369]: Health check update: Degraded data redundancy: 39/324 objects degraded (12.037%), 12 pgs degraded (PG_DEGRADED) 2024-03-30T04:35:19.717 INFO:journalctl@ceph.osd.2.smithi123.stdout:Mar 30 04:35:19 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-2[166988]: 2024-03-30T04:35:19.481+0000 7fd2fe000700 -1 osd.2 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T04:35:19.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:19 smithi135 ceph-mon[106011]: from='osd.2 [v2:172.21.15.123:6818/3356591260,v1:172.21.15.123:6819/3356591260]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-30T04:35:19.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:19 smithi135 ceph-mon[106011]: osdmap e62: 6 total, 5 up, 6 in 2024-03-30T04:35:19.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:19 smithi135 ceph-mon[106011]: from='osd.2 [v2:172.21.15.123:6818/3356591260,v1:172.21.15.123:6819/3356591260]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-03-30T04:35:19.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:19 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:35:19.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:19 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:35:19.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:19 smithi135 ceph-mon[106011]: Health check update: Degraded data redundancy: 39/324 objects degraded (12.037%), 12 pgs degraded (PG_DEGRADED) 2024-03-30T04:35:20.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:20 smithi123 ceph-mon[139369]: pgmap v101: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 39/324 objects degraded (12.037%) 2024-03-30T04:35:20.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:20 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:35:20.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:20 smithi123 ceph-mon[139369]: from='osd.2 [v2:172.21.15.123:6818/3356591260,v1:172.21.15.123:6819/3356591260]' entity='osd.2' 2024-03-30T04:35:20.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:20 smithi135 ceph-mon[106011]: pgmap v101: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 39/324 objects degraded (12.037%) 2024-03-30T04:35:20.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:20 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:35:20.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:20 smithi135 ceph-mon[106011]: from='osd.2 [v2:172.21.15.123:6818/3356591260,v1:172.21.15.123:6819/3356591260]' entity='osd.2' 2024-03-30T04:35:21.543 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:21 smithi123 ceph-mon[139369]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-30T04:35:21.544 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:21 smithi123 ceph-mon[139369]: osd.2 [v2:172.21.15.123:6818/3356591260,v1:172.21.15.123:6819/3356591260] boot 2024-03-30T04:35:21.544 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:21 smithi123 ceph-mon[139369]: osdmap e63: 6 total, 6 up, 6 in 2024-03-30T04:35:21.544 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:21 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T04:35:21.544 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:21 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:35:21.544 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:21 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:35:21.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:21 smithi135 ceph-mon[106011]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-30T04:35:21.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:21 smithi135 ceph-mon[106011]: osd.2 [v2:172.21.15.123:6818/3356591260,v1:172.21.15.123:6819/3356591260] boot 2024-03-30T04:35:21.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:21 smithi135 ceph-mon[106011]: osdmap e63: 6 total, 6 up, 6 in 2024-03-30T04:35:21.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:21 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T04:35:21.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:21 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:35:21.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:21 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:35:22.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:22 smithi123 ceph-mon[139369]: pgmap v103: 65 pgs: 5 peering, 10 active+undersized+degraded, 12 active+undersized, 38 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s; 31/324 objects degraded (9.568%) 2024-03-30T04:35:22.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:22 smithi123 ceph-mon[139369]: osdmap e64: 6 total, 6 up, 6 in 2024-03-30T04:35:22.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:22 smithi135 ceph-mon[106011]: pgmap v103: 65 pgs: 5 peering, 10 active+undersized+degraded, 12 active+undersized, 38 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s; 31/324 objects degraded (9.568%) 2024-03-30T04:35:22.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:22 smithi135 ceph-mon[106011]: osdmap e64: 6 total, 6 up, 6 in 2024-03-30T04:35:24.626 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:24 smithi123 ceph-mon[139369]: pgmap v105: 65 pgs: 5 peering, 10 active+undersized+degraded, 12 active+undersized, 38 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 31/324 objects degraded (9.568%) 2024-03-30T04:35:24.626 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:24 smithi123 ceph-mon[139369]: Health check update: Degraded data redundancy: 31/324 objects degraded (9.568%), 10 pgs degraded (PG_DEGRADED) 2024-03-30T04:35:24.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:24 smithi135 ceph-mon[106011]: pgmap v105: 65 pgs: 5 peering, 10 active+undersized+degraded, 12 active+undersized, 38 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 31/324 objects degraded (9.568%) 2024-03-30T04:35:24.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:24 smithi135 ceph-mon[106011]: Health check update: Degraded data redundancy: 31/324 objects degraded (9.568%), 10 pgs degraded (PG_DEGRADED) 2024-03-30T04:35:26.059 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:26 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:35:26.060 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:26 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:35:26.060 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:26 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:35:26.060 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:26 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:35:26.060 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:26 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:35:26.060 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:26 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:35:26.060 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:26 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:35:26.060 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:26 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:35:26.060 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:26 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:35:26.060 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:26 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-30T04:35:26.060 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:26 smithi135 ceph-mon[106011]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-30T04:35:26.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:26 smithi135 ceph-mon[106011]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-03-30T04:35:26.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:26 smithi135 ceph-mon[106011]: pgmap v106: 65 pgs: 5 peering, 60 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-30T04:35:26.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:26 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:35:26.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:26 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:35:26.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:26 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:35:26.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:26 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:35:26.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:26 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:35:26.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:26 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:35:26.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:26 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:35:26.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:26 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:35:26.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:26 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:35:26.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:26 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-30T04:35:26.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:26 smithi123 ceph-mon[139369]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-30T04:35:26.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:26 smithi123 ceph-mon[139369]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-03-30T04:35:26.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:26 smithi123 ceph-mon[139369]: pgmap v106: 65 pgs: 5 peering, 60 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s rd, 4 op/s 2024-03-30T04:35:27.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:27 smithi135 ceph-mon[106011]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 31/324 objects degraded (9.568%), 10 pgs degraded) 2024-03-30T04:35:27.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:27 smithi135 ceph-mon[106011]: Cluster is now healthy 2024-03-30T04:35:27.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:27 smithi123 ceph-mon[139369]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 31/324 objects degraded (9.568%), 10 pgs degraded) 2024-03-30T04:35:27.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:27 smithi123 ceph-mon[139369]: Cluster is now healthy 2024-03-30T04:35:28.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:28 smithi135 ceph-mon[106011]: pgmap v107: 65 pgs: 65 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-30T04:35:28.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:28 smithi123 ceph-mon[139369]: pgmap v107: 65 pgs: 65 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-30T04:35:30.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:30 smithi123 ceph-mon[139369]: pgmap v108: 65 pgs: 65 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 688 B/s rd, 1 op/s 2024-03-30T04:35:30.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:35:30.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:35:30.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:30 smithi135 ceph-mon[106011]: pgmap v108: 65 pgs: 65 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 688 B/s rd, 1 op/s 2024-03-30T04:35:30.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:30 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:35:30.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:30 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:35:31.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.108+0000 7f84efa00700 1 -- 172.21.15.123:0/513589575 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84e81037a0 msgr2=0x7f84e80fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:35:31.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.109+0000 7f84efa00700 1 --2- 172.21.15.123:0/513589575 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84e81037a0 0x7f84e80fce10 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f84d4004940 tx=0x7f84d4009e30 comp rx=0 tx=0).stop 2024-03-30T04:35:31.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.109+0000 7f84efa00700 1 -- 172.21.15.123:0/513589575 shutdown_connections 2024-03-30T04:35:31.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.109+0000 7f84efa00700 1 --2- 172.21.15.123:0/513589575 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84e81037a0 0x7f84e80fce10 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:31.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.109+0000 7f84efa00700 1 --2- 172.21.15.123:0/513589575 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f84e8102e00 0x7f84e81031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:31.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.109+0000 7f84efa00700 1 -- 172.21.15.123:0/513589575 >> 172.21.15.123:0/513589575 conn(0x7f84e80f89b0 msgr2=0x7f84e80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:35:31.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.109+0000 7f84efa00700 1 -- 172.21.15.123:0/513589575 shutdown_connections 2024-03-30T04:35:31.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.109+0000 7f84efa00700 1 -- 172.21.15.123:0/513589575 wait complete. 2024-03-30T04:35:31.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.110+0000 7f84efa00700 1 Processor -- start 2024-03-30T04:35:31.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.110+0000 7f84efa00700 1 -- start start 2024-03-30T04:35:31.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.110+0000 7f84efa00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f84e8102e00 0x7f84e8115d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:35:31.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.110+0000 7f84efa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84e81037a0 0x7f84e8116240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:35:31.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.110+0000 7f84efa00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e8116890 con 0x7f84e81037a0 2024-03-30T04:35:31.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.110+0000 7f84efa00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e810fc60 con 0x7f84e8102e00 2024-03-30T04:35:31.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.110+0000 7f84e7e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84e81037a0 0x7f84e8116240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:35:31.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.110+0000 7f84e7e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84e81037a0 0x7f84e8116240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37062/0 (socket says 172.21.15.123:37062) 2024-03-30T04:35:31.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.110+0000 7f84e7e00700 1 -- 172.21.15.123:0/1680780781 learned_addr learned my addr 172.21.15.123:0/1680780781 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:35:31.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.111+0000 7f84ed000700 1 --2- 172.21.15.123:0/1680780781 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f84e8102e00 0x7f84e8115d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:35:31.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.111+0000 7f84e7e00700 1 -- 172.21.15.123:0/1680780781 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f84e8102e00 msgr2=0x7f84e8115d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:35:31.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.111+0000 7f84e7e00700 1 --2- 172.21.15.123:0/1680780781 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f84e8102e00 0x7f84e8115d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:31.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.111+0000 7f84e7e00700 1 -- 172.21.15.123:0/1680780781 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d4005040 con 0x7f84e81037a0 2024-03-30T04:35:31.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.111+0000 7f84e7e00700 1 --2- 172.21.15.123:0/1680780781 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84e81037a0 0x7f84e8116240 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f84d4009390 tx=0x7f84d4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:35:31.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.111+0000 7f84e4c00700 1 -- 172.21.15.123:0/1680780781 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f84d400e040 con 0x7f84e81037a0 2024-03-30T04:35:31.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.111+0000 7f84e4c00700 1 -- 172.21.15.123:0/1680780781 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f84d4013420 con 0x7f84e81037a0 2024-03-30T04:35:31.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.111+0000 7f84e4c00700 1 -- 172.21.15.123:0/1680780781 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f84d401c630 con 0x7f84e81037a0 2024-03-30T04:35:31.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.111+0000 7f84efa00700 1 -- 172.21.15.123:0/1680780781 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84e810fee0 con 0x7f84e81037a0 2024-03-30T04:35:31.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.111+0000 7f84efa00700 1 -- 172.21.15.123:0/1680780781 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84e81104a0 con 0x7f84e81037a0 2024-03-30T04:35:31.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.112+0000 7f84efa00700 1 -- 172.21.15.123:0/1680780781 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84e80fe550 con 0x7f84e81037a0 2024-03-30T04:35:31.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.113+0000 7f84e4c00700 1 -- 172.21.15.123:0/1680780781 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f84d40139e0 con 0x7f84e81037a0 2024-03-30T04:35:31.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.114+0000 7f84e4c00700 1 --2- 172.21.15.123:0/1680780781 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f84d006cc30 0x7f84d006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:35:31.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.114+0000 7f84e4c00700 1 -- 172.21.15.123:0/1680780781 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f84d408d2e0 con 0x7f84e81037a0 2024-03-30T04:35:31.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.115+0000 7f84ed000700 1 --2- 172.21.15.123:0/1680780781 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f84d006cc30 0x7f84d006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:35:31.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.115+0000 7f84ed000700 1 --2- 172.21.15.123:0/1680780781 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f84d006cc30 0x7f84d006f0e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f84dc004990 tx=0x7f84dc009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:35:31.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.120+0000 7f84e4c00700 1 -- 172.21.15.123:0/1680780781 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f84d405adb0 con 0x7f84e81037a0 2024-03-30T04:35:31.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.340+0000 7f84efa00700 1 -- 172.21.15.123:0/1680780781 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f84e8109e10 con 0x7f84d006cc30 2024-03-30T04:35:31.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.342+0000 7f84e4c00700 1 -- 172.21.15.123:0/1680780781 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f84e8109e10 con 0x7f84d006cc30 2024-03-30T04:35:31.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.344+0000 7f84efa00700 1 -- 172.21.15.123:0/1680780781 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f84d006cc30 msgr2=0x7f84d006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:35:31.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.344+0000 7f84efa00700 1 --2- 172.21.15.123:0/1680780781 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f84d006cc30 0x7f84d006f0e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f84dc004990 tx=0x7f84dc009ee0 comp rx=0 tx=0).stop 2024-03-30T04:35:31.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.344+0000 7f84efa00700 1 -- 172.21.15.123:0/1680780781 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84e81037a0 msgr2=0x7f84e8116240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:35:31.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.344+0000 7f84efa00700 1 --2- 172.21.15.123:0/1680780781 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84e81037a0 0x7f84e8116240 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f84d4009390 tx=0x7f84d4009e30 comp rx=0 tx=0).stop 2024-03-30T04:35:31.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.344+0000 7f84efa00700 1 -- 172.21.15.123:0/1680780781 shutdown_connections 2024-03-30T04:35:31.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.344+0000 7f84efa00700 1 --2- 172.21.15.123:0/1680780781 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f84d006cc30 0x7f84d006f0e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:31.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.344+0000 7f84efa00700 1 --2- 172.21.15.123:0/1680780781 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f84e8102e00 0x7f84e8115d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:31.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.344+0000 7f84efa00700 1 --2- 172.21.15.123:0/1680780781 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84e81037a0 0x7f84e8116240 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:31.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.344+0000 7f84efa00700 1 -- 172.21.15.123:0/1680780781 >> 172.21.15.123:0/1680780781 conn(0x7f84e80f89b0 msgr2=0x7f84e80f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:35:31.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.344+0000 7f84efa00700 1 -- 172.21.15.123:0/1680780781 shutdown_connections 2024-03-30T04:35:31.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.344+0000 7f84efa00700 1 -- 172.21.15.123:0/1680780781 wait complete. 2024-03-30T04:35:31.357 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:35:31.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.481+0000 7f338ba00700 1 -- 172.21.15.123:0/2500017036 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33840fe830 msgr2=0x7f3384106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:35:31.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.481+0000 7f338ba00700 1 --2- 172.21.15.123:0/2500017036 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33840fe830 0x7f3384106f00 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3370004940 tx=0x7f3370009e30 comp rx=0 tx=0).stop 2024-03-30T04:35:31.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.481+0000 7f338ba00700 1 -- 172.21.15.123:0/2500017036 shutdown_connections 2024-03-30T04:35:31.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.481+0000 7f338ba00700 1 --2- 172.21.15.123:0/2500017036 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33840fe830 0x7f3384106f00 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:31.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.481+0000 7f338ba00700 1 --2- 172.21.15.123:0/2500017036 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f33840fde90 0x7f33840fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:31.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.481+0000 7f338ba00700 1 -- 172.21.15.123:0/2500017036 >> 172.21.15.123:0/2500017036 conn(0x7f3384072bc0 msgr2=0x7f3384072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:35:31.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.481+0000 7f338ba00700 1 -- 172.21.15.123:0/2500017036 shutdown_connections 2024-03-30T04:35:31.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.481+0000 7f338ba00700 1 -- 172.21.15.123:0/2500017036 wait complete. 2024-03-30T04:35:31.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.482+0000 7f338ba00700 1 Processor -- start 2024-03-30T04:35:31.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.482+0000 7f338ba00700 1 -- start start 2024-03-30T04:35:31.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.482+0000 7f338ba00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33840fde90 0x7f3384115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:35:31.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.482+0000 7f338ba00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f33840fe830 0x7f33841161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:35:31.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.482+0000 7f338ba00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3384116810 con 0x7f33840fde90 2024-03-30T04:35:31.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.482+0000 7f338ba00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f338410fc30 con 0x7f33840fe830 2024-03-30T04:35:31.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.483+0000 7f3389000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33840fde90 0x7f3384115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:35:31.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.483+0000 7f3389000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33840fde90 0x7f3384115c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37078/0 (socket says 172.21.15.123:37078) 2024-03-30T04:35:31.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.483+0000 7f3389000700 1 -- 172.21.15.123:0/1434934483 learned_addr learned my addr 172.21.15.123:0/1434934483 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:35:31.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.483+0000 7f3383e00700 1 --2- 172.21.15.123:0/1434934483 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f33840fe830 0x7f33841161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:35:31.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.483+0000 7f3389000700 1 -- 172.21.15.123:0/1434934483 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f33840fe830 msgr2=0x7f33841161c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:35:31.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.483+0000 7f3389000700 1 --2- 172.21.15.123:0/1434934483 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f33840fe830 0x7f33841161c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:31.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.483+0000 7f3389000700 1 -- 172.21.15.123:0/1434934483 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3370005040 con 0x7f33840fde90 2024-03-30T04:35:31.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.483+0000 7f3389000700 1 --2- 172.21.15.123:0/1434934483 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33840fde90 0x7f3384115c80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f337800eb40 tx=0x7f337800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:35:31.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.483+0000 7f3380c00700 1 -- 172.21.15.123:0/1434934483 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3378007cc0 con 0x7f33840fde90 2024-03-30T04:35:31.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.483+0000 7f3380c00700 1 -- 172.21.15.123:0/1434934483 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f337800f040 con 0x7f33840fde90 2024-03-30T04:35:31.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.484+0000 7f3380c00700 1 -- 172.21.15.123:0/1434934483 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f337800c990 con 0x7f33840fde90 2024-03-30T04:35:31.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.484+0000 7f338ba00700 1 -- 172.21.15.123:0/1434934483 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f338410feb0 con 0x7f33840fde90 2024-03-30T04:35:31.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.484+0000 7f338ba00700 1 -- 172.21.15.123:0/1434934483 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33841103f0 con 0x7f33840fde90 2024-03-30T04:35:31.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.485+0000 7f338ba00700 1 -- 172.21.15.123:0/1434934483 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3384106a20 con 0x7f33840fde90 2024-03-30T04:35:31.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.486+0000 7f3380c00700 1 -- 172.21.15.123:0/1434934483 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f337800caf0 con 0x7f33840fde90 2024-03-30T04:35:31.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.487+0000 7f3380c00700 1 --2- 172.21.15.123:0/1434934483 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f336c06cc30 0x7f336c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:35:31.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.487+0000 7f3380c00700 1 -- 172.21.15.123:0/1434934483 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f337808bb80 con 0x7f33840fde90 2024-03-30T04:35:31.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.487+0000 7f3383e00700 1 --2- 172.21.15.123:0/1434934483 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f336c06cc30 0x7f336c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:35:31.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.488+0000 7f3383e00700 1 --2- 172.21.15.123:0/1434934483 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f336c06cc30 0x7f336c06f0e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3384111520 tx=0x7f3370009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:35:31.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.492+0000 7f3380c00700 1 -- 172.21.15.123:0/1434934483 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3378055c70 con 0x7f33840fde90 2024-03-30T04:35:31.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.712+0000 7f338ba00700 1 -- 172.21.15.123:0/1434934483 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f33841112d0 con 0x7f336c06cc30 2024-03-30T04:35:31.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.714+0000 7f3380c00700 1 -- 172.21.15.123:0/1434934483 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f33841112d0 con 0x7f336c06cc30 2024-03-30T04:35:31.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.716+0000 7f338ba00700 1 -- 172.21.15.123:0/1434934483 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f336c06cc30 msgr2=0x7f336c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:35:31.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.716+0000 7f338ba00700 1 --2- 172.21.15.123:0/1434934483 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f336c06cc30 0x7f336c06f0e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3384111520 tx=0x7f3370009e30 comp rx=0 tx=0).stop 2024-03-30T04:35:31.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.716+0000 7f338ba00700 1 -- 172.21.15.123:0/1434934483 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33840fde90 msgr2=0x7f3384115c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:35:31.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.716+0000 7f338ba00700 1 --2- 172.21.15.123:0/1434934483 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33840fde90 0x7f3384115c80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f337800eb40 tx=0x7f337800b040 comp rx=0 tx=0).stop 2024-03-30T04:35:31.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.716+0000 7f338ba00700 1 -- 172.21.15.123:0/1434934483 shutdown_connections 2024-03-30T04:35:31.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.716+0000 7f338ba00700 1 --2- 172.21.15.123:0/1434934483 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f336c06cc30 0x7f336c06f0e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:31.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.716+0000 7f338ba00700 1 --2- 172.21.15.123:0/1434934483 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33840fde90 0x7f3384115c80 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:31.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.716+0000 7f338ba00700 1 --2- 172.21.15.123:0/1434934483 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f33840fe830 0x7f33841161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:31.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.716+0000 7f338ba00700 1 -- 172.21.15.123:0/1434934483 >> 172.21.15.123:0/1434934483 conn(0x7f3384072bc0 msgr2=0x7f33840fd510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:35:31.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.716+0000 7f338ba00700 1 -- 172.21.15.123:0/1434934483 shutdown_connections 2024-03-30T04:35:31.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.716+0000 7f338ba00700 1 -- 172.21.15.123:0/1434934483 wait complete. 2024-03-30T04:35:31.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.850+0000 7f822f400700 1 -- 172.21.15.123:0/3138909525 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8228102e00 msgr2=0x7f82281031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:35:31.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.850+0000 7f822f400700 1 --2- 172.21.15.123:0/3138909525 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8228102e00 0x7f82281031d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f821c004940 tx=0x7f821c009e30 comp rx=0 tx=0).stop 2024-03-30T04:35:31.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.850+0000 7f822f400700 1 -- 172.21.15.123:0/3138909525 shutdown_connections 2024-03-30T04:35:31.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.850+0000 7f822f400700 1 --2- 172.21.15.123:0/3138909525 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f82281037a0 0x7f82280fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:31.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.850+0000 7f822f400700 1 --2- 172.21.15.123:0/3138909525 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8228102e00 0x7f82281031d0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:31.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.850+0000 7f822f400700 1 -- 172.21.15.123:0/3138909525 >> 172.21.15.123:0/3138909525 conn(0x7f82280f89b0 msgr2=0x7f82280fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:35:31.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.850+0000 7f822f400700 1 -- 172.21.15.123:0/3138909525 shutdown_connections 2024-03-30T04:35:31.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.850+0000 7f822f400700 1 -- 172.21.15.123:0/3138909525 wait complete. 2024-03-30T04:35:31.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.851+0000 7f822f400700 1 Processor -- start 2024-03-30T04:35:31.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.851+0000 7f822f400700 1 -- start start 2024-03-30T04:35:31.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.851+0000 7f822f400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8228102e00 0x7f8228115c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:35:31.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.851+0000 7f822f400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f82281037a0 0x7f8228116180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:35:31.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.851+0000 7f822f400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82281167d0 con 0x7f8228102e00 2024-03-30T04:35:31.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.851+0000 7f822f400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f822810fba0 con 0x7f82281037a0 2024-03-30T04:35:31.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.852+0000 7f822ca00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8228102e00 0x7f8228115c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:35:31.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.852+0000 7f822ca00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8228102e00 0x7f8228115c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37088/0 (socket says 172.21.15.123:37088) 2024-03-30T04:35:31.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.852+0000 7f822ca00700 1 -- 172.21.15.123:0/4201041554 learned_addr learned my addr 172.21.15.123:0/4201041554 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:35:31.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.852+0000 7f8227e00700 1 --2- 172.21.15.123:0/4201041554 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f82281037a0 0x7f8228116180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:35:31.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.852+0000 7f822ca00700 1 -- 172.21.15.123:0/4201041554 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f82281037a0 msgr2=0x7f8228116180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:35:31.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.852+0000 7f822ca00700 1 --2- 172.21.15.123:0/4201041554 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f82281037a0 0x7f8228116180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:31.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.852+0000 7f822ca00700 1 -- 172.21.15.123:0/4201041554 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f821c005040 con 0x7f8228102e00 2024-03-30T04:35:31.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.852+0000 7f8227e00700 1 --2- 172.21.15.123:0/4201041554 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f82281037a0 0x7f8228116180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:35:31.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.852+0000 7f822ca00700 1 --2- 172.21.15.123:0/4201041554 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8228102e00 0x7f8228115c40 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f821c007f50 tx=0x7f821c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:35:31.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.853+0000 7f8224c00700 1 -- 172.21.15.123:0/4201041554 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f821c021070 con 0x7f8228102e00 2024-03-30T04:35:31.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.853+0000 7f8224c00700 1 -- 172.21.15.123:0/4201041554 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f821c013e00 con 0x7f8228102e00 2024-03-30T04:35:31.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.853+0000 7f822f400700 1 -- 172.21.15.123:0/4201041554 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f822810fe20 con 0x7f8228102e00 2024-03-30T04:35:31.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.853+0000 7f8224c00700 1 -- 172.21.15.123:0/4201041554 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f821c00e070 con 0x7f8228102e00 2024-03-30T04:35:31.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.853+0000 7f822f400700 1 -- 172.21.15.123:0/4201041554 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82281103e0 con 0x7f8228102e00 2024-03-30T04:35:31.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.854+0000 7f822f400700 1 -- 172.21.15.123:0/4201041554 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82280fe550 con 0x7f8228102e00 2024-03-30T04:35:31.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.855+0000 7f8224c00700 1 -- 172.21.15.123:0/4201041554 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f821c013950 con 0x7f8228102e00 2024-03-30T04:35:31.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.856+0000 7f8224c00700 1 --2- 172.21.15.123:0/4201041554 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f821006cc30 0x7f821006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:35:31.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.856+0000 7f8224c00700 1 -- 172.21.15.123:0/4201041554 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f821c08d2c0 con 0x7f8228102e00 2024-03-30T04:35:31.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.856+0000 7f8227e00700 1 --2- 172.21.15.123:0/4201041554 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f821006cc30 0x7f821006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:35:31.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.856+0000 7f8227e00700 1 --2- 172.21.15.123:0/4201041554 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f821006cc30 0x7f821006f0e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8214004600 tx=0x7f821400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:35:31.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:31.860+0000 7f8224c00700 1 -- 172.21.15.123:0/4201041554 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f821c05ad90 con 0x7f8228102e00 2024-03-30T04:35:32.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.072+0000 7f822f400700 1 -- 172.21.15.123:0/4201041554 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f822804bf40 con 0x7f821006cc30 2024-03-30T04:35:32.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.082+0000 7f8224c00700 1 -- 172.21.15.123:0/4201041554 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f822804bf40 con 0x7f821006cc30 2024-03-30T04:35:32.081 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:35:32.081 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (4m) 12s ago 17m 14.7M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:35:32.081 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (118s) 12s ago 17m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T04:35:32.081 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (112s) 108s ago 16m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T04:35:32.082 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (4m) 12s ago 16m 74.1M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T04:35:32.082 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (10m) 12s ago 10m 109M - 17.2.7-797-g5c024335 f77085f6aab7 2df066fba7a6 2024-03-30T04:35:32.082 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (10m) 12s ago 10m 58.0M - 17.2.7-797-g5c024335 f77085f6aab7 1293b8520db1 2024-03-30T04:35:32.082 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (10m) 108s ago 10m 67.4M - 17.2.7-797-g5c024335 f77085f6aab7 0dc41326818c 2024-03-30T04:35:32.082 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (10m) 108s ago 10m 119M - 17.2.7-797-g5c024335 f77085f6aab7 aef8a0cfc553 2024-03-30T04:35:32.082 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (6m) 12s ago 18m 502M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:35:32.082 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (5m) 108s ago 16m 448M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:35:32.082 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (3m) 12s ago 19m 45.6M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T04:35:32.082 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (2m) 108s ago 16m 33.2M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T04:35:32.082 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (5m) 12s ago 17m 7583k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:35:32.082 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (5m) 108s ago 16m 7512k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:35:32.082 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (87s) 12s ago 15m 99.5M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T04:35:32.082 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (61s) 12s ago 14m 90.4M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T04:35:32.083 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (19s) 12s ago 14m 10.0M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T04:35:32.083 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (13m) 108s ago 13m 259M 2642M 17.2.7-797-g5c024335 f77085f6aab7 6fbdd1d46a9d 2024-03-30T04:35:32.083 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (13m) 108s ago 13m 227M 2642M 17.2.7-797-g5c024335 f77085f6aab7 b74988e947e5 2024-03-30T04:35:32.083 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (12m) 108s ago 12m 220M 2642M 17.2.7-797-g5c024335 f77085f6aab7 1fd3185f5484 2024-03-30T04:35:32.083 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (5m) 12s ago 16m 52.5M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:35:32.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.084+0000 7f822f400700 1 -- 172.21.15.123:0/4201041554 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f821006cc30 msgr2=0x7f821006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:35:32.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.084+0000 7f822f400700 1 --2- 172.21.15.123:0/4201041554 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f821006cc30 0x7f821006f0e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8214004600 tx=0x7f821400b040 comp rx=0 tx=0).stop 2024-03-30T04:35:32.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.084+0000 7f822f400700 1 -- 172.21.15.123:0/4201041554 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8228102e00 msgr2=0x7f8228115c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:35:32.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.084+0000 7f822f400700 1 --2- 172.21.15.123:0/4201041554 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8228102e00 0x7f8228115c40 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f821c007f50 tx=0x7f821c009e30 comp rx=0 tx=0).stop 2024-03-30T04:35:32.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.085+0000 7f822f400700 1 -- 172.21.15.123:0/4201041554 shutdown_connections 2024-03-30T04:35:32.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.085+0000 7f822f400700 1 --2- 172.21.15.123:0/4201041554 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f821006cc30 0x7f821006f0e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:32.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.085+0000 7f822f400700 1 --2- 172.21.15.123:0/4201041554 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8228102e00 0x7f8228115c40 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:32.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.085+0000 7f822f400700 1 --2- 172.21.15.123:0/4201041554 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f82281037a0 0x7f8228116180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:32.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.085+0000 7f822f400700 1 -- 172.21.15.123:0/4201041554 >> 172.21.15.123:0/4201041554 conn(0x7f82280f89b0 msgr2=0x7f82280f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:35:32.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.085+0000 7f822f400700 1 -- 172.21.15.123:0/4201041554 shutdown_connections 2024-03-30T04:35:32.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.085+0000 7f822f400700 1 -- 172.21.15.123:0/4201041554 wait complete. 2024-03-30T04:35:32.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.217+0000 7faa9dc00700 1 -- 172.21.15.123:0/545676209 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa98102c40 msgr2=0x7faa98103010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:35:32.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.217+0000 7faa9dc00700 1 --2- 172.21.15.123:0/545676209 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa98102c40 0x7faa98103010 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7faa88004940 tx=0x7faa88009e30 comp rx=0 tx=0).stop 2024-03-30T04:35:32.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.217+0000 7faa9dc00700 1 -- 172.21.15.123:0/545676209 shutdown_connections 2024-03-30T04:35:32.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.217+0000 7faa9dc00700 1 --2- 172.21.15.123:0/545676209 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa981035e0 0x7faa980fcc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:32.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.217+0000 7faa9dc00700 1 --2- 172.21.15.123:0/545676209 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa98102c40 0x7faa98103010 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:32.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.217+0000 7faa9dc00700 1 -- 172.21.15.123:0/545676209 >> 172.21.15.123:0/545676209 conn(0x7faa980f8830 msgr2=0x7faa980fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:35:32.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.217+0000 7faa9dc00700 1 -- 172.21.15.123:0/545676209 shutdown_connections 2024-03-30T04:35:32.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.217+0000 7faa9dc00700 1 -- 172.21.15.123:0/545676209 wait complete. 2024-03-30T04:35:32.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.218+0000 7faa9dc00700 1 Processor -- start 2024-03-30T04:35:32.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.218+0000 7faa9dc00700 1 -- start start 2024-03-30T04:35:32.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.219+0000 7faa9dc00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa98102c40 0x7faa980702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:35:32.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.219+0000 7faa9dc00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa981035e0 0x7faa98070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:35:32.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.219+0000 7faa9dc00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa98070d90 con 0x7faa981035e0 2024-03-30T04:35:32.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.219+0000 7faa9dc00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa98070ed0 con 0x7faa98102c40 2024-03-30T04:35:32.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.219+0000 7faa97400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa981035e0 0x7faa98070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:35:32.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.219+0000 7faa97400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa981035e0 0x7faa98070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37106/0 (socket says 172.21.15.123:37106) 2024-03-30T04:35:32.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.219+0000 7faa97400700 1 -- 172.21.15.123:0/342644061 learned_addr learned my addr 172.21.15.123:0/342644061 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:35:32.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.219+0000 7faa97e00700 1 --2- 172.21.15.123:0/342644061 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa98102c40 0x7faa980702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:35:32.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.219+0000 7faa97400700 1 -- 172.21.15.123:0/342644061 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa98102c40 msgr2=0x7faa980702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:35:32.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.219+0000 7faa97400700 1 --2- 172.21.15.123:0/342644061 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa98102c40 0x7faa980702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:32.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.219+0000 7faa97400700 1 -- 172.21.15.123:0/342644061 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa88005040 con 0x7faa981035e0 2024-03-30T04:35:32.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.220+0000 7faa97400700 1 --2- 172.21.15.123:0/342644061 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa981035e0 0x7faa98070800 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7faa84002a50 tx=0x7faa840046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:35:32.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.220+0000 7faa8fe00700 1 -- 172.21.15.123:0/342644061 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa84007cc0 con 0x7faa981035e0 2024-03-30T04:35:32.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.220+0000 7faa9dc00700 1 -- 172.21.15.123:0/342644061 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa98119b70 con 0x7faa981035e0 2024-03-30T04:35:32.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.220+0000 7faa8fe00700 1 -- 172.21.15.123:0/342644061 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faa8400f040 con 0x7faa981035e0 2024-03-30T04:35:32.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.220+0000 7faa8fe00700 1 -- 172.21.15.123:0/342644061 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa840095d0 con 0x7faa981035e0 2024-03-30T04:35:32.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.220+0000 7faa9dc00700 1 -- 172.21.15.123:0/342644061 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa98119ff0 con 0x7faa981035e0 2024-03-30T04:35:32.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.221+0000 7faa9dc00700 1 -- 172.21.15.123:0/342644061 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa980fe390 con 0x7faa981035e0 2024-03-30T04:35:32.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.222+0000 7faa8fe00700 1 -- 172.21.15.123:0/342644061 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7faa84009730 con 0x7faa981035e0 2024-03-30T04:35:32.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.223+0000 7faa8fe00700 1 --2- 172.21.15.123:0/342644061 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7faa7c06cc30 0x7faa7c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:35:32.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.223+0000 7faa8fe00700 1 -- 172.21.15.123:0/342644061 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6324+0+0 (secure 0 0 0) 0x7faa8408bc90 con 0x7faa981035e0 2024-03-30T04:35:32.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.223+0000 7faa97e00700 1 --2- 172.21.15.123:0/342644061 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7faa7c06cc30 0x7faa7c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:35:32.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.224+0000 7faa97e00700 1 --2- 172.21.15.123:0/342644061 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7faa7c06cc30 0x7faa7c06f0e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7faa8800b020 tx=0x7faa88009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:35:32.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.228+0000 7faa8fe00700 1 -- 172.21.15.123:0/342644061 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7faa84055cf0 con 0x7faa981035e0 2024-03-30T04:35:32.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.512+0000 7faa9dc00700 1 -- 172.21.15.123:0/342644061 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7faa98109c20 con 0x7faa981035e0 2024-03-30T04:35:32.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.513+0000 7faa8fe00700 1 -- 172.21.15.123:0/342644061 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7faa84059310 con 0x7faa981035e0 2024-03-30T04:35:32.511 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:35:32.511 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:35:32.512 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:35:32.512 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:35:32.512 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:35:32.512 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:35:32.512 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:35:32.512 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:35:32.512 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 3, 2024-03-30T04:35:32.512 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 3 2024-03-30T04:35:32.512 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:35:32.512 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:35:32.512 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 4 2024-03-30T04:35:32.512 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:35:32.512 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:35:32.512 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 7, 2024-03-30T04:35:32.513 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 7 2024-03-30T04:35:32.513 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:35:32.513 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:35:32.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.515+0000 7faa9dc00700 1 -- 172.21.15.123:0/342644061 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7faa7c06cc30 msgr2=0x7faa7c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:35:32.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.515+0000 7faa9dc00700 1 --2- 172.21.15.123:0/342644061 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7faa7c06cc30 0x7faa7c06f0e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7faa8800b020 tx=0x7faa88009e30 comp rx=0 tx=0).stop 2024-03-30T04:35:32.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.515+0000 7faa9dc00700 1 -- 172.21.15.123:0/342644061 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa981035e0 msgr2=0x7faa98070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:35:32.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.515+0000 7faa9dc00700 1 --2- 172.21.15.123:0/342644061 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa981035e0 0x7faa98070800 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7faa84002a50 tx=0x7faa840046b0 comp rx=0 tx=0).stop 2024-03-30T04:35:32.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.515+0000 7faa9dc00700 1 -- 172.21.15.123:0/342644061 shutdown_connections 2024-03-30T04:35:32.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.515+0000 7faa9dc00700 1 --2- 172.21.15.123:0/342644061 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7faa7c06cc30 0x7faa7c06f0e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:32.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.515+0000 7faa9dc00700 1 --2- 172.21.15.123:0/342644061 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa98102c40 0x7faa980702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:32.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.515+0000 7faa9dc00700 1 --2- 172.21.15.123:0/342644061 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa981035e0 0x7faa98070800 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:32.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.515+0000 7faa9dc00700 1 -- 172.21.15.123:0/342644061 >> 172.21.15.123:0/342644061 conn(0x7faa980f8830 msgr2=0x7faa980f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:35:32.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.515+0000 7faa9dc00700 1 -- 172.21.15.123:0/342644061 shutdown_connections 2024-03-30T04:35:32.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.515+0000 7faa9dc00700 1 -- 172.21.15.123:0/342644061 wait complete. 2024-03-30T04:35:32.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.648+0000 7f4ed3a00700 1 -- 172.21.15.123:0/1621832131 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ecc1037a0 msgr2=0x7f4ecc0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:35:32.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.648+0000 7f4ed3a00700 1 --2- 172.21.15.123:0/1621832131 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ecc1037a0 0x7f4ecc0fce10 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4eb8004940 tx=0x7f4eb8009e30 comp rx=0 tx=0).stop 2024-03-30T04:35:32.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.648+0000 7f4ed3a00700 1 -- 172.21.15.123:0/1621832131 shutdown_connections 2024-03-30T04:35:32.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.648+0000 7f4ed3a00700 1 --2- 172.21.15.123:0/1621832131 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ecc1037a0 0x7f4ecc0fce10 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:32.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.648+0000 7f4ed3a00700 1 --2- 172.21.15.123:0/1621832131 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4ecc102e00 0x7f4ecc1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:32.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.648+0000 7f4ed3a00700 1 -- 172.21.15.123:0/1621832131 >> 172.21.15.123:0/1621832131 conn(0x7f4ecc0f89b0 msgr2=0x7f4ecc0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:35:32.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.648+0000 7f4ed3a00700 1 -- 172.21.15.123:0/1621832131 shutdown_connections 2024-03-30T04:35:32.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.648+0000 7f4ed3a00700 1 -- 172.21.15.123:0/1621832131 wait complete. 2024-03-30T04:35:32.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.649+0000 7f4ed3a00700 1 Processor -- start 2024-03-30T04:35:32.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.649+0000 7f4ed3a00700 1 -- start start 2024-03-30T04:35:32.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.650+0000 7f4ed3a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4ecc102e00 0x7f4ecc113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:35:32.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.650+0000 7f4ed3a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ecc1037a0 0x7f4ecc113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:35:32.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.650+0000 7f4ed3a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ecc114600 con 0x7f4ecc1037a0 2024-03-30T04:35:32.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.650+0000 7f4ed3a00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ecc10d9f0 con 0x7f4ecc102e00 2024-03-30T04:35:32.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.650+0000 7f4ecbe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ecc1037a0 0x7f4ecc113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:35:32.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.650+0000 7f4ecbe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ecc1037a0 0x7f4ecc113fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37128/0 (socket says 172.21.15.123:37128) 2024-03-30T04:35:32.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.650+0000 7f4ecbe00700 1 -- 172.21.15.123:0/766394840 learned_addr learned my addr 172.21.15.123:0/766394840 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:35:32.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.650+0000 7f4ed1000700 1 --2- 172.21.15.123:0/766394840 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4ecc102e00 0x7f4ecc113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:35:32.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.650+0000 7f4ecbe00700 1 -- 172.21.15.123:0/766394840 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4ecc102e00 msgr2=0x7f4ecc113a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:35:32.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.650+0000 7f4ecbe00700 1 --2- 172.21.15.123:0/766394840 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4ecc102e00 0x7f4ecc113a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:32.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.650+0000 7f4ecbe00700 1 -- 172.21.15.123:0/766394840 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eb8005040 con 0x7f4ecc1037a0 2024-03-30T04:35:32.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.651+0000 7f4ecbe00700 1 --2- 172.21.15.123:0/766394840 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ecc1037a0 0x7f4ecc113fb0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4eb8009fd0 tx=0x7f4eb8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:35:32.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.651+0000 7f4ec8c00700 1 -- 172.21.15.123:0/766394840 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4eb800e030 con 0x7f4ecc1037a0 2024-03-30T04:35:32.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.651+0000 7f4ec8c00700 1 -- 172.21.15.123:0/766394840 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4eb80093b0 con 0x7f4ecc1037a0 2024-03-30T04:35:32.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.651+0000 7f4ed3a00700 1 -- 172.21.15.123:0/766394840 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ecc10dc70 con 0x7f4ecc1037a0 2024-03-30T04:35:32.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.651+0000 7f4ec8c00700 1 -- 172.21.15.123:0/766394840 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4eb801cc80 con 0x7f4ecc1037a0 2024-03-30T04:35:32.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.651+0000 7f4ed3a00700 1 -- 172.21.15.123:0/766394840 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ecc10e230 con 0x7f4ecc1037a0 2024-03-30T04:35:32.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.652+0000 7f4ed3a00700 1 -- 172.21.15.123:0/766394840 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ecc0fe550 con 0x7f4ecc1037a0 2024-03-30T04:35:32.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.653+0000 7f4ec8c00700 1 -- 172.21.15.123:0/766394840 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f4eb801c400 con 0x7f4ecc1037a0 2024-03-30T04:35:32.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.654+0000 7f4ec8c00700 1 --2- 172.21.15.123:0/766394840 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4eb406cc30 0x7f4eb406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:35:32.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.654+0000 7f4ec8c00700 1 -- 172.21.15.123:0/766394840 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f4eb808d710 con 0x7f4ecc1037a0 2024-03-30T04:35:32.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.654+0000 7f4ed1000700 1 --2- 172.21.15.123:0/766394840 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4eb406cc30 0x7f4eb406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:35:32.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.655+0000 7f4ed1000700 1 --2- 172.21.15.123:0/766394840 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4eb406cc30 0x7f4eb406f0e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4ec0009b80 tx=0x7f4ec000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:35:32.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.659+0000 7f4ec8c00700 1 -- 172.21.15.123:0/766394840 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4eb805b320 con 0x7f4ecc1037a0 2024-03-30T04:35:32.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:32 smithi123 ceph-mon[139369]: from='client.34284 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:35:32.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:32 smithi123 ceph-mon[139369]: pgmap v109: 65 pgs: 65 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-30T04:35:32.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:32 smithi123 ceph-mon[139369]: from='client.34288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:35:32.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:32 smithi135 ceph-mon[106011]: from='client.34284 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:35:32.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:32 smithi135 ceph-mon[106011]: pgmap v109: 65 pgs: 65 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-30T04:35:32.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:32 smithi135 ceph-mon[106011]: from='client.34288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:35:32.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.904+0000 7f4ed3a00700 1 -- 172.21.15.123:0/766394840 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f4ecc10f0c0 con 0x7f4ecc1037a0 2024-03-30T04:35:32.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.905+0000 7f4ec8c00700 1 -- 172.21.15.123:0/766394840 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 23 v23) v1 ==== 76+0+1876 (secure 0 0 0) 0x7f4eb805aeb0 con 0x7f4ecc1037a0 2024-03-30T04:35:32.903 INFO:teuthology.orchestra.run.smithi123.stdout:e23 2024-03-30T04:35:32.903 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:35:32.903 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:35:32.903 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:35:32.904 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:35:32.904 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:35:32.904 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:35:32.904 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 23 2024-03-30T04:35:32.904 INFO:teuthology.orchestra.run.smithi123.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-30T04:35:32.904 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:35:32.904 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:32:48.448604+0000 2024-03-30T04:35:32.904 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:35:32.904 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:35:32.904 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:35:32.904 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:35:32.904 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:35:32.904 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T04:35:32.904 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:35:32.904 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:35:32.905 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 46 2024-03-30T04:35:32.905 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:35:32.905 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T04:35:32.905 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T04:35:32.905 INFO:teuthology.orchestra.run.smithi123.stdout:up {0=14488,1=24309} 2024-03-30T04:35:32.905 INFO:teuthology.orchestra.run.smithi123.stdout:failed 2024-03-30T04:35:32.905 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:35:32.905 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:35:32.905 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:35:32.905 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:35:32.905 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:35:32.905 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:35:32.905 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T04:35:32.905 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T04:35:32.905 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{0:14488} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.123:6828/3037849622,v1:172.21.15.123:6829/3037849622] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:35:32.906 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{0:14500} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/2433675995,v1:172.21.15.123:6827/2433675995] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:35:32.906 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{1:24309} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.135:6824/4050541062,v1:172.21.15.135:6825/4050541062] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:35:32.906 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{1:24319} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.135:6826/1096986122,v1:172.21.15.135:6827/1096986122] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:35:32.906 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:35:32.906 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:35:32.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.907+0000 7f4ed3a00700 1 -- 172.21.15.123:0/766394840 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4eb406cc30 msgr2=0x7f4eb406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:35:32.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.907+0000 7f4ed3a00700 1 --2- 172.21.15.123:0/766394840 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4eb406cc30 0x7f4eb406f0e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4ec0009b80 tx=0x7f4ec000b040 comp rx=0 tx=0).stop 2024-03-30T04:35:32.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.907+0000 7f4ed3a00700 1 -- 172.21.15.123:0/766394840 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ecc1037a0 msgr2=0x7f4ecc113fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:35:32.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.907+0000 7f4ed3a00700 1 --2- 172.21.15.123:0/766394840 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ecc1037a0 0x7f4ecc113fb0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4eb8009fd0 tx=0x7f4eb8009e30 comp rx=0 tx=0).stop 2024-03-30T04:35:32.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.907+0000 7f4ed3a00700 1 -- 172.21.15.123:0/766394840 shutdown_connections 2024-03-30T04:35:32.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.907+0000 7f4ed3a00700 1 --2- 172.21.15.123:0/766394840 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4eb406cc30 0x7f4eb406f0e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:32.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.907+0000 7f4ed3a00700 1 --2- 172.21.15.123:0/766394840 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4ecc102e00 0x7f4ecc113a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:32.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.907+0000 7f4ed3a00700 1 --2- 172.21.15.123:0/766394840 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ecc1037a0 0x7f4ecc113fb0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:32.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.907+0000 7f4ed3a00700 1 -- 172.21.15.123:0/766394840 >> 172.21.15.123:0/766394840 conn(0x7f4ecc0f89b0 msgr2=0x7f4ecc0f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:35:32.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.907+0000 7f4ed3a00700 1 -- 172.21.15.123:0/766394840 shutdown_connections 2024-03-30T04:35:32.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:32.907+0000 7f4ed3a00700 1 -- 172.21.15.123:0/766394840 wait complete. 2024-03-30T04:35:32.909 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 23 2024-03-30T04:35:33.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.043+0000 7f9632e00700 1 -- 172.21.15.123:0/2731514491 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f962c1037a0 msgr2=0x7f962c0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:35:33.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.043+0000 7f9632e00700 1 --2- 172.21.15.123:0/2731514491 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f962c1037a0 0x7f962c0fce10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9618004940 tx=0x7f9618009e30 comp rx=0 tx=0).stop 2024-03-30T04:35:33.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.043+0000 7f9632e00700 1 -- 172.21.15.123:0/2731514491 shutdown_connections 2024-03-30T04:35:33.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.043+0000 7f9632e00700 1 --2- 172.21.15.123:0/2731514491 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f962c1037a0 0x7f962c0fce10 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:33.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.043+0000 7f9632e00700 1 --2- 172.21.15.123:0/2731514491 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f962c102e00 0x7f962c1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:33.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.043+0000 7f9632e00700 1 -- 172.21.15.123:0/2731514491 >> 172.21.15.123:0/2731514491 conn(0x7f962c0f89b0 msgr2=0x7f962c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:35:33.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.043+0000 7f9632e00700 1 -- 172.21.15.123:0/2731514491 shutdown_connections 2024-03-30T04:35:33.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.043+0000 7f9632e00700 1 -- 172.21.15.123:0/2731514491 wait complete. 2024-03-30T04:35:33.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.044+0000 7f9632e00700 1 Processor -- start 2024-03-30T04:35:33.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.044+0000 7f9632e00700 1 -- start start 2024-03-30T04:35:33.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.045+0000 7f9632e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f962c102e00 0x7f962c115d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:35:33.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.045+0000 7f9632e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f962c1037a0 0x7f962c116250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:35:33.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.045+0000 7f9632e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f962c1168a0 con 0x7f962c1037a0 2024-03-30T04:35:33.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.045+0000 7f9632e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f962c10fc70 con 0x7f962c102e00 2024-03-30T04:35:33.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.045+0000 7f962b400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f962c1037a0 0x7f962c116250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:35:33.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.045+0000 7f962b400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f962c1037a0 0x7f962c116250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37146/0 (socket says 172.21.15.123:37146) 2024-03-30T04:35:33.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.045+0000 7f962b400700 1 -- 172.21.15.123:0/1639002331 learned_addr learned my addr 172.21.15.123:0/1639002331 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:35:33.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.045+0000 7f962be00700 1 --2- 172.21.15.123:0/1639002331 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f962c102e00 0x7f962c115d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:35:33.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.046+0000 7f962b400700 1 -- 172.21.15.123:0/1639002331 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f962c102e00 msgr2=0x7f962c115d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:35:33.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.046+0000 7f962b400700 1 --2- 172.21.15.123:0/1639002331 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f962c102e00 0x7f962c115d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:33.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.046+0000 7f962b400700 1 -- 172.21.15.123:0/1639002331 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9618005040 con 0x7f962c1037a0 2024-03-30T04:35:33.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.046+0000 7f962b400700 1 --2- 172.21.15.123:0/1639002331 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f962c1037a0 0x7f962c116250 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9618009fd0 tx=0x7f9618009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:35:33.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.046+0000 7f9623e00700 1 -- 172.21.15.123:0/1639002331 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f961800e030 con 0x7f962c1037a0 2024-03-30T04:35:33.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.046+0000 7f9632e00700 1 -- 172.21.15.123:0/1639002331 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f962c10fef0 con 0x7f962c1037a0 2024-03-30T04:35:33.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.046+0000 7f9623e00700 1 -- 172.21.15.123:0/1639002331 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f96180093b0 con 0x7f962c1037a0 2024-03-30T04:35:33.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.046+0000 7f9623e00700 1 -- 172.21.15.123:0/1639002331 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f961801cc80 con 0x7f962c1037a0 2024-03-30T04:35:33.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.046+0000 7f9632e00700 1 -- 172.21.15.123:0/1639002331 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f962c110430 con 0x7f962c1037a0 2024-03-30T04:35:33.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.048+0000 7f9632e00700 1 -- 172.21.15.123:0/1639002331 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f962c0fe550 con 0x7f962c1037a0 2024-03-30T04:35:33.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.049+0000 7f9623e00700 1 -- 172.21.15.123:0/1639002331 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f961801c400 con 0x7f962c1037a0 2024-03-30T04:35:33.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.050+0000 7f9623e00700 1 --2- 172.21.15.123:0/1639002331 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f961006cc30 0x7f961006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:35:33.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.050+0000 7f9623e00700 1 -- 172.21.15.123:0/1639002331 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f961808d7c0 con 0x7f962c1037a0 2024-03-30T04:35:33.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.050+0000 7f962be00700 1 --2- 172.21.15.123:0/1639002331 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f961006cc30 0x7f961006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:35:33.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.051+0000 7f962be00700 1 --2- 172.21.15.123:0/1639002331 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f961006cc30 0x7f961006f0e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f961c009b80 tx=0x7f961c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:35:33.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.054+0000 7f9623e00700 1 -- 172.21.15.123:0/1639002331 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f961805b330 con 0x7f962c1037a0 2024-03-30T04:35:33.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.278+0000 7f9632e00700 1 -- 172.21.15.123:0/1639002331 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f962c109dc0 con 0x7f961006cc30 2024-03-30T04:35:33.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.280+0000 7f9623e00700 1 -- 172.21.15.123:0/1639002331 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f962c109dc0 con 0x7f961006cc30 2024-03-30T04:35:33.278 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:35:33.278 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:35:33.278 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:35:33.278 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T04:35:33.279 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:35:33.279 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T04:35:33.279 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T04:35:33.279 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T04:35:33.279 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:35:33.279 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "9/21 daemons upgraded", 2024-03-30T04:35:33.279 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading osd daemons", 2024-03-30T04:35:33.279 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:35:33.279 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:35:33.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.282+0000 7f9632e00700 1 -- 172.21.15.123:0/1639002331 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f961006cc30 msgr2=0x7f961006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:35:33.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.282+0000 7f9632e00700 1 --2- 172.21.15.123:0/1639002331 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f961006cc30 0x7f961006f0e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f961c009b80 tx=0x7f961c00b040 comp rx=0 tx=0).stop 2024-03-30T04:35:33.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.282+0000 7f9632e00700 1 -- 172.21.15.123:0/1639002331 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f962c1037a0 msgr2=0x7f962c116250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:35:33.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.282+0000 7f9632e00700 1 --2- 172.21.15.123:0/1639002331 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f962c1037a0 0x7f962c116250 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9618009fd0 tx=0x7f9618009e30 comp rx=0 tx=0).stop 2024-03-30T04:35:33.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.282+0000 7f9632e00700 1 -- 172.21.15.123:0/1639002331 shutdown_connections 2024-03-30T04:35:33.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.282+0000 7f9632e00700 1 --2- 172.21.15.123:0/1639002331 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f961006cc30 0x7f961006f0e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:33.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.282+0000 7f9632e00700 1 --2- 172.21.15.123:0/1639002331 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f962c102e00 0x7f962c115d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:33.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.282+0000 7f9632e00700 1 --2- 172.21.15.123:0/1639002331 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f962c1037a0 0x7f962c116250 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:33.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.282+0000 7f9632e00700 1 -- 172.21.15.123:0/1639002331 >> 172.21.15.123:0/1639002331 conn(0x7f962c0f89b0 msgr2=0x7f962c0f9980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:35:33.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.282+0000 7f9632e00700 1 -- 172.21.15.123:0/1639002331 shutdown_connections 2024-03-30T04:35:33.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.282+0000 7f9632e00700 1 -- 172.21.15.123:0/1639002331 wait complete. 2024-03-30T04:35:33.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.414+0000 7f0d00600700 1 -- 172.21.15.123:0/2002393918 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0cf8102e00 msgr2=0x7f0cf81031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:35:33.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.414+0000 7f0d00600700 1 --2- 172.21.15.123:0/2002393918 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0cf8102e00 0x7f0cf81031d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0cec004940 tx=0x7f0cec009e30 comp rx=0 tx=0).stop 2024-03-30T04:35:33.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.414+0000 7f0d00600700 1 -- 172.21.15.123:0/2002393918 shutdown_connections 2024-03-30T04:35:33.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.414+0000 7f0d00600700 1 --2- 172.21.15.123:0/2002393918 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0cf81037a0 0x7f0cf80fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:33.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.414+0000 7f0d00600700 1 --2- 172.21.15.123:0/2002393918 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0cf8102e00 0x7f0cf81031d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:33.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.414+0000 7f0d00600700 1 -- 172.21.15.123:0/2002393918 >> 172.21.15.123:0/2002393918 conn(0x7f0cf80f89b0 msgr2=0x7f0cf80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:35:33.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.414+0000 7f0d00600700 1 -- 172.21.15.123:0/2002393918 shutdown_connections 2024-03-30T04:35:33.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.414+0000 7f0d00600700 1 -- 172.21.15.123:0/2002393918 wait complete. 2024-03-30T04:35:33.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.415+0000 7f0d00600700 1 Processor -- start 2024-03-30T04:35:33.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.415+0000 7f0d00600700 1 -- start start 2024-03-30T04:35:33.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.415+0000 7f0d00600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0cf8102e00 0x7f0cf8115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:35:33.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.415+0000 7f0d00600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0cf81037a0 0x7f0cf8116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:35:33.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.415+0000 7f0d00600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf8116860 con 0x7f0cf81037a0 2024-03-30T04:35:33.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.415+0000 7f0d00600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf810fc30 con 0x7f0cf8102e00 2024-03-30T04:35:33.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.416+0000 7f0cfd200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0cf81037a0 0x7f0cf8116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:35:33.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.416+0000 7f0cfd200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0cf81037a0 0x7f0cf8116210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37176/0 (socket says 172.21.15.123:37176) 2024-03-30T04:35:33.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.416+0000 7f0cfd200700 1 -- 172.21.15.123:0/81890030 learned_addr learned my addr 172.21.15.123:0/81890030 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:35:33.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.416+0000 7f0cfdc00700 1 --2- 172.21.15.123:0/81890030 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0cf8102e00 0x7f0cf8115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:35:33.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.416+0000 7f0cfd200700 1 -- 172.21.15.123:0/81890030 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0cf8102e00 msgr2=0x7f0cf8115cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:35:33.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.416+0000 7f0cfd200700 1 --2- 172.21.15.123:0/81890030 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0cf8102e00 0x7f0cf8115cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:33.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.416+0000 7f0cfd200700 1 -- 172.21.15.123:0/81890030 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cec005040 con 0x7f0cf81037a0 2024-03-30T04:35:33.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.416+0000 7f0cfd200700 1 --2- 172.21.15.123:0/81890030 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0cf81037a0 0x7f0cf8116210 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4002a50 tx=0x7f0ce4004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:35:33.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.417+0000 7f0cf5600700 1 -- 172.21.15.123:0/81890030 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ce4007cc0 con 0x7f0cf81037a0 2024-03-30T04:35:33.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.417+0000 7f0cf5600700 1 -- 172.21.15.123:0/81890030 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0ce400f040 con 0x7f0cf81037a0 2024-03-30T04:35:33.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.417+0000 7f0d00600700 1 -- 172.21.15.123:0/81890030 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cf810ff10 con 0x7f0cf81037a0 2024-03-30T04:35:33.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.417+0000 7f0cf5600700 1 -- 172.21.15.123:0/81890030 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ce400d9a0 con 0x7f0cf81037a0 2024-03-30T04:35:33.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.417+0000 7f0d00600700 1 -- 172.21.15.123:0/81890030 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cf81104d0 con 0x7f0cf81037a0 2024-03-30T04:35:33.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.418+0000 7f0d00600700 1 -- 172.21.15.123:0/81890030 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cf80fe550 con 0x7f0cf81037a0 2024-03-30T04:35:33.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.419+0000 7f0cf5600700 1 -- 172.21.15.123:0/81890030 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f0ce400dc60 con 0x7f0cf81037a0 2024-03-30T04:35:33.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.420+0000 7f0cf5600700 1 --2- 172.21.15.123:0/81890030 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0ce006cc30 0x7f0ce006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:35:33.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.420+0000 7f0cf5600700 1 -- 172.21.15.123:0/81890030 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f0ce408be70 con 0x7f0cf81037a0 2024-03-30T04:35:33.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.420+0000 7f0cfdc00700 1 --2- 172.21.15.123:0/81890030 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0ce006cc30 0x7f0ce006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:35:33.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.421+0000 7f0cfdc00700 1 --2- 172.21.15.123:0/81890030 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0ce006cc30 0x7f0ce006f0e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0cec00b020 tx=0x7f0cec009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:35:33.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.424+0000 7f0cf5600700 1 -- 172.21.15.123:0/81890030 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0ce40598c0 con 0x7f0cf81037a0 2024-03-30T04:35:33.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.708+0000 7f0d00600700 1 -- 172.21.15.123:0/81890030 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f0cf804bed0 con 0x7f0cf81037a0 2024-03-30T04:35:33.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.709+0000 7f0cf5600700 1 -- 172.21.15.123:0/81890030 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f0ce4059450 con 0x7f0cf81037a0 2024-03-30T04:35:33.707 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_OK 2024-03-30T04:35:33.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.711+0000 7f0d00600700 1 -- 172.21.15.123:0/81890030 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0ce006cc30 msgr2=0x7f0ce006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:35:33.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.711+0000 7f0d00600700 1 --2- 172.21.15.123:0/81890030 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0ce006cc30 0x7f0ce006f0e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0cec00b020 tx=0x7f0cec009e30 comp rx=0 tx=0).stop 2024-03-30T04:35:33.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.711+0000 7f0d00600700 1 -- 172.21.15.123:0/81890030 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0cf81037a0 msgr2=0x7f0cf8116210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:35:33.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.711+0000 7f0d00600700 1 --2- 172.21.15.123:0/81890030 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0cf81037a0 0x7f0cf8116210 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4002a50 tx=0x7f0ce4004600 comp rx=0 tx=0).stop 2024-03-30T04:35:33.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.711+0000 7f0d00600700 1 -- 172.21.15.123:0/81890030 shutdown_connections 2024-03-30T04:35:33.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.711+0000 7f0d00600700 1 --2- 172.21.15.123:0/81890030 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0ce006cc30 0x7f0ce006f0e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:33.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.711+0000 7f0d00600700 1 --2- 172.21.15.123:0/81890030 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0cf8102e00 0x7f0cf8115cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:33.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.711+0000 7f0d00600700 1 --2- 172.21.15.123:0/81890030 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0cf81037a0 0x7f0cf8116210 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:35:33.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.711+0000 7f0d00600700 1 -- 172.21.15.123:0/81890030 >> 172.21.15.123:0/81890030 conn(0x7f0cf80f89b0 msgr2=0x7f0cf80f9980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:35:33.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.711+0000 7f0d00600700 1 -- 172.21.15.123:0/81890030 shutdown_connections 2024-03-30T04:35:33.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:35:33.711+0000 7f0d00600700 1 -- 172.21.15.123:0/81890030 wait complete. 2024-03-30T04:35:33.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:33 smithi123 ceph-mon[139369]: from='client.34292 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:35:33.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:33 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/342644061' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:35:33.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:33 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/766394840' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:35:33.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:33 smithi135 ceph-mon[106011]: from='client.34292 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:35:33.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:33 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/342644061' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:35:33.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:33 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/766394840' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:35:34.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:34 smithi123 ceph-mon[139369]: from='client.34304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:35:34.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:34 smithi123 ceph-mon[139369]: pgmap v110: 65 pgs: 65 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 2 op/s 2024-03-30T04:35:34.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:34 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/81890030' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:35:34.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:34 smithi135 ceph-mon[106011]: from='client.34304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:35:34.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:34 smithi135 ceph-mon[106011]: pgmap v110: 65 pgs: 65 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 2 op/s 2024-03-30T04:35:34.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:34 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/81890030' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:35:35.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:35 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:35:35.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:35 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:35:36.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:36 smithi135 ceph-mon[106011]: pgmap v111: 65 pgs: 65 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-30T04:35:36.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:36 smithi123 ceph-mon[139369]: pgmap v111: 65 pgs: 65 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-30T04:35:38.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:38 smithi135 ceph-mon[106011]: pgmap v112: 65 pgs: 65 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:35:38.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:38 smithi123 ceph-mon[139369]: pgmap v112: 65 pgs: 65 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:35:40.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:40 smithi123 ceph-mon[139369]: pgmap v113: 65 pgs: 65 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:35:40.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:40 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-30T04:35:40.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:40 smithi135 ceph-mon[106011]: pgmap v113: 65 pgs: 65 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:35:40.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:40 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-30T04:35:41.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:41 smithi135 ceph-mon[106011]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-30T04:35:41.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:41 smithi135 ceph-mon[106011]: Upgrade: osd.3 is safe to restart 2024-03-30T04:35:41.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:41 smithi123 ceph-mon[139369]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-30T04:35:41.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:41 smithi123 ceph-mon[139369]: Upgrade: osd.3 is safe to restart 2024-03-30T04:35:42.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:42 smithi135 ceph-mon[106011]: pgmap v114: 65 pgs: 65 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:35:42.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:42 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:35:42.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:42 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-30T04:35:42.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:42 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:35:42.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:42 smithi123 ceph-mon[139369]: pgmap v114: 65 pgs: 65 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:35:42.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:42 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:35:42.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:42 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-30T04:35:42.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:42 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:35:43.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:43 smithi123 ceph-mon[139369]: Upgrade: Updating osd.3 2024-03-30T04:35:43.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:43 smithi123 ceph-mon[139369]: Deploying daemon osd.3 on smithi135 2024-03-30T04:35:43.812 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:43 smithi135 ceph-mon[106011]: Upgrade: Updating osd.3 2024-03-30T04:35:43.812 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:43 smithi135 ceph-mon[106011]: Deploying daemon osd.3 on smithi135 2024-03-30T04:35:44.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:44 smithi135 ceph-mon[106011]: pgmap v115: 65 pgs: 65 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:35:44.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:44 smithi123 ceph-mon[139369]: pgmap v115: 65 pgs: 65 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:35:45.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:45 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:35:45.765 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:45 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:35:46.101 INFO:journalctl@ceph.osd.3.smithi135.stdout:Mar 30 04:35:46 smithi135 systemd[1]: Stopping Ceph osd.3 for 2e77a2be-ee4c-11ee-b647-cb9ed24678a4... 2024-03-30T04:35:46.601 INFO:journalctl@ceph.osd.3.smithi135.stdout:Mar 30 04:35:46 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-3[50690]: 2024-03-30T04:35:46.333+0000 7f260d400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T04:35:46.601 INFO:journalctl@ceph.osd.3.smithi135.stdout:Mar 30 04:35:46 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-3[50690]: 2024-03-30T04:35:46.333+0000 7f260d400700 -1 osd.3 64 *** Got signal Terminated *** 2024-03-30T04:35:46.601 INFO:journalctl@ceph.osd.3.smithi135.stdout:Mar 30 04:35:46 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-3[50690]: 2024-03-30T04:35:46.333+0000 7f260d400700 -1 osd.3 64 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T04:35:46.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:46 smithi135 ceph-mon[106011]: pgmap v116: 65 pgs: 65 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:35:46.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:46 smithi135 ceph-mon[106011]: osd.3 marked itself down and dead 2024-03-30T04:35:46.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:46 smithi123 ceph-mon[139369]: pgmap v116: 65 pgs: 65 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:35:46.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:46 smithi123 ceph-mon[139369]: osd.3 marked itself down and dead 2024-03-30T04:35:47.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:47 smithi135 ceph-mon[106011]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-30T04:35:47.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:47 smithi135 ceph-mon[106011]: osdmap e65: 6 total, 5 up, 6 in 2024-03-30T04:35:47.601 INFO:journalctl@ceph.osd.3.smithi135.stdout:Mar 30 04:35:47 smithi135 bash[118202]: ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-3 2024-03-30T04:35:47.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:47 smithi123 ceph-mon[139369]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-30T04:35:47.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:47 smithi123 ceph-mon[139369]: osdmap e65: 6 total, 5 up, 6 in 2024-03-30T04:35:48.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:48 smithi135 ceph-mon[106011]: osdmap e66: 6 total, 5 up, 6 in 2024-03-30T04:35:48.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:48 smithi135 ceph-mon[106011]: pgmap v119: 65 pgs: 16 stale+active+clean, 49 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:35:48.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:48 smithi123 ceph-mon[139369]: osdmap e66: 6 total, 5 up, 6 in 2024-03-30T04:35:48.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:48 smithi123 ceph-mon[139369]: pgmap v119: 65 pgs: 16 stale+active+clean, 49 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:35:49.599 INFO:journalctl@ceph.osd.3.smithi135.stdout:Mar 30 04:35:49 smithi135 systemd[1]: ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4@osd.3.service: Succeeded. 2024-03-30T04:35:49.599 INFO:journalctl@ceph.osd.3.smithi135.stdout:Mar 30 04:35:49 smithi135 systemd[1]: Stopped Ceph osd.3 for 2e77a2be-ee4c-11ee-b647-cb9ed24678a4. 2024-03-30T04:35:50.101 INFO:journalctl@ceph.osd.3.smithi135.stdout:Mar 30 04:35:49 smithi135 systemd[1]: Starting Ceph osd.3 for 2e77a2be-ee4c-11ee-b647-cb9ed24678a4... 2024-03-30T04:35:50.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:50 smithi123 ceph-mon[139369]: pgmap v120: 65 pgs: 7 active+undersized+degraded, 13 stale+active+clean, 6 active+undersized, 39 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 21/324 objects degraded (6.481%) 2024-03-30T04:35:50.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:50 smithi135 ceph-mon[106011]: pgmap v120: 65 pgs: 7 active+undersized+degraded, 13 stale+active+clean, 6 active+undersized, 39 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 21/324 objects degraded (6.481%) 2024-03-30T04:35:51.263 INFO:journalctl@ceph.osd.3.smithi135.stdout:Mar 30 04:35:50 smithi135 bash[118474]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-30T04:35:51.264 INFO:journalctl@ceph.osd.3.smithi135.stdout:Mar 30 04:35:50 smithi135 bash[118474]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-30T04:35:51.264 INFO:journalctl@ceph.osd.3.smithi135.stdout:Mar 30 04:35:50 smithi135 bash[118474]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-30T04:35:51.264 INFO:journalctl@ceph.osd.3.smithi135.stdout:Mar 30 04:35:50 smithi135 bash[118474]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-30T04:35:51.264 INFO:journalctl@ceph.osd.3.smithi135.stdout:Mar 30 04:35:50 smithi135 bash[118474]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3/block 2024-03-30T04:35:51.264 INFO:journalctl@ceph.osd.3.smithi135.stdout:Mar 30 04:35:50 smithi135 bash[118474]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-30T04:35:51.264 INFO:journalctl@ceph.osd.3.smithi135.stdout:Mar 30 04:35:50 smithi135 bash[118474]: --> ceph-volume raw activate successful for osd ID: 3 2024-03-30T04:35:51.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:51 smithi135 ceph-mon[106011]: Health check failed: Degraded data redundancy: 21/324 objects degraded (6.481%), 7 pgs degraded (PG_DEGRADED) 2024-03-30T04:35:51.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:51 smithi123 ceph-mon[139369]: Health check failed: Degraded data redundancy: 21/324 objects degraded (6.481%), 7 pgs degraded (PG_DEGRADED) 2024-03-30T04:35:52.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:52 smithi135 ceph-mon[106011]: pgmap v121: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 68/324 objects degraded (20.988%) 2024-03-30T04:35:52.601 INFO:journalctl@ceph.osd.3.smithi135.stdout:Mar 30 04:35:52 smithi135 ceph-osd[118762]: -- 172.21.15.135:0/1494950824 <== mon.1 v2:172.21.15.135:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555f87ddf1e0 con 0x555f87daec00 2024-03-30T04:35:52.601 INFO:journalctl@ceph.osd.3.smithi135.stdout:Mar 30 04:35:52 smithi135 bash[118724]: acbbc38441e433c947bdb03f9a1af46e2bb8a807f07a66a97ec692a47388b57a 2024-03-30T04:35:52.601 INFO:journalctl@ceph.osd.3.smithi135.stdout:Mar 30 04:35:52 smithi135 systemd[1]: Started Ceph osd.3 for 2e77a2be-ee4c-11ee-b647-cb9ed24678a4. 2024-03-30T04:35:52.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:52 smithi123 ceph-mon[139369]: pgmap v121: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 68/324 objects degraded (20.988%) 2024-03-30T04:35:53.100 INFO:journalctl@ceph.osd.3.smithi135.stdout:Mar 30 04:35:52 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-3[118739]: 2024-03-30T04:35:52.775+0000 7f1e5b49a640 -1 Falling back to public interface 2024-03-30T04:35:53.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:53 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:35:53.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:53 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:35:53.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:53 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:35:53.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:53 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:35:53.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:53 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:35:53.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:53 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:35:54.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:54 smithi135 ceph-mon[106011]: pgmap v122: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 68/324 objects degraded (20.988%) 2024-03-30T04:35:54.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:54 smithi123 ceph-mon[139369]: pgmap v122: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 68/324 objects degraded (20.988%) 2024-03-30T04:35:56.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:56 smithi123 ceph-mon[139369]: pgmap v123: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 68/324 objects degraded (20.988%) 2024-03-30T04:35:56.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:56 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:35:56.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:56 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:35:56.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:56 smithi135 ceph-mon[106011]: pgmap v123: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 68/324 objects degraded (20.988%) 2024-03-30T04:35:56.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:56 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:35:56.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:56 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:35:58.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:58 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:35:58.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:58 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:35:58.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:58 smithi123 ceph-mon[139369]: pgmap v124: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 68/324 objects degraded (20.988%) 2024-03-30T04:35:58.542 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:58 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:35:58.542 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:58 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:35:58.542 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:58 smithi135 ceph-mon[106011]: pgmap v124: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 68/324 objects degraded (20.988%) 2024-03-30T04:35:58.850 INFO:journalctl@ceph.osd.3.smithi135.stdout:Mar 30 04:35:58 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-3[118739]: 2024-03-30T04:35:58.542+0000 7f1e5b49a640 -1 osd.3 64 log_to_monitors true 2024-03-30T04:35:59.200 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:59 smithi135 ceph-mon[106011]: from='osd.3 [v2:172.21.15.135:6800/978771295,v1:172.21.15.135:6801/978771295]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-30T04:35:59.200 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:35:59 smithi135 ceph-mon[106011]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-30T04:35:59.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:59 smithi123 ceph-mon[139369]: from='osd.3 [v2:172.21.15.135:6800/978771295,v1:172.21.15.135:6801/978771295]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-30T04:35:59.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:35:59 smithi123 ceph-mon[139369]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-30T04:35:59.601 INFO:journalctl@ceph.osd.3.smithi135.stdout:Mar 30 04:35:59 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-3[118739]: 2024-03-30T04:35:59.200+0000 7f1e4a800700 -1 osd.3 64 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T04:36:00.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:00 smithi123 ceph-mon[139369]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-30T04:36:00.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:00 smithi123 ceph-mon[139369]: osdmap e67: 6 total, 5 up, 6 in 2024-03-30T04:36:00.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:00 smithi123 ceph-mon[139369]: from='osd.3 [v2:172.21.15.135:6800/978771295,v1:172.21.15.135:6801/978771295]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-03-30T04:36:00.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:00 smithi123 ceph-mon[139369]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-03-30T04:36:00.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:00 smithi123 ceph-mon[139369]: Health check update: Degraded data redundancy: 68/324 objects degraded (20.988%), 19 pgs degraded (PG_DEGRADED) 2024-03-30T04:36:00.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:00 smithi123 ceph-mon[139369]: pgmap v126: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 68/324 objects degraded (20.988%) 2024-03-30T04:36:00.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:00 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:36:00.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:00 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:36:00.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:00 smithi135 ceph-mon[106011]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-30T04:36:00.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:00 smithi135 ceph-mon[106011]: osdmap e67: 6 total, 5 up, 6 in 2024-03-30T04:36:00.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:00 smithi135 ceph-mon[106011]: from='osd.3 [v2:172.21.15.135:6800/978771295,v1:172.21.15.135:6801/978771295]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-03-30T04:36:00.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:00 smithi135 ceph-mon[106011]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-03-30T04:36:00.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:00 smithi135 ceph-mon[106011]: Health check update: Degraded data redundancy: 68/324 objects degraded (20.988%), 19 pgs degraded (PG_DEGRADED) 2024-03-30T04:36:00.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:00 smithi135 ceph-mon[106011]: pgmap v126: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 68/324 objects degraded (20.988%) 2024-03-30T04:36:00.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:00 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:36:00.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:00 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:36:01.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:01 smithi135 ceph-mon[106011]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-30T04:36:01.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:01 smithi135 ceph-mon[106011]: osd.3 [v2:172.21.15.135:6800/978771295,v1:172.21.15.135:6801/978771295] boot 2024-03-30T04:36:01.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:01 smithi135 ceph-mon[106011]: osdmap e68: 6 total, 6 up, 6 in 2024-03-30T04:36:01.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:01 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T04:36:01.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:01 smithi123 ceph-mon[139369]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-30T04:36:01.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:01 smithi123 ceph-mon[139369]: osd.3 [v2:172.21.15.135:6800/978771295,v1:172.21.15.135:6801/978771295] boot 2024-03-30T04:36:01.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:01 smithi123 ceph-mon[139369]: osdmap e68: 6 total, 6 up, 6 in 2024-03-30T04:36:01.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:01 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T04:36:02.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:02 smithi123 ceph-mon[139369]: osdmap e69: 6 total, 6 up, 6 in 2024-03-30T04:36:02.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:02 smithi123 ceph-mon[139369]: pgmap v129: 65 pgs: 8 peering, 15 active+undersized+degraded, 11 active+undersized, 31 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s; 55/324 objects degraded (16.975%) 2024-03-30T04:36:02.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:02 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:36:02.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:02 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:36:02.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:02 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:36:02.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:02 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:36:02.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:02 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:36:02.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:02 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:36:02.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:02 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:36:02.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:02 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:36:02.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:02 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:36:02.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:02 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-30T04:36:02.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:02 smithi135 ceph-mon[106011]: osdmap e69: 6 total, 6 up, 6 in 2024-03-30T04:36:02.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:02 smithi135 ceph-mon[106011]: pgmap v129: 65 pgs: 8 peering, 15 active+undersized+degraded, 11 active+undersized, 31 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s; 55/324 objects degraded (16.975%) 2024-03-30T04:36:02.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:02 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:36:02.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:02 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:36:02.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:02 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:36:02.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:02 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:36:02.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:02 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:36:02.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:02 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:36:02.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:02 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:36:02.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:02 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:36:02.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:02 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:36:02.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:02 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-30T04:36:03.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:03 smithi123 ceph-mon[139369]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-30T04:36:03.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:03 smithi123 ceph-mon[139369]: Upgrade: unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-03-30T04:36:03.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:03 smithi135 ceph-mon[106011]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-30T04:36:03.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:03 smithi135 ceph-mon[106011]: Upgrade: unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-03-30T04:36:03.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:03.864+0000 7f91a2000700 1 -- 172.21.15.123:0/3440418254 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f919c1037c0 msgr2=0x7f919c0fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:03.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:03.864+0000 7f91a2000700 1 --2- 172.21.15.123:0/3440418254 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f919c1037c0 0x7f919c0fce30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9188004940 tx=0x7f9188009e30 comp rx=0 tx=0).stop 2024-03-30T04:36:03.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:03.864+0000 7f91a2000700 1 -- 172.21.15.123:0/3440418254 shutdown_connections 2024-03-30T04:36:03.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:03.864+0000 7f91a2000700 1 --2- 172.21.15.123:0/3440418254 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f919c1037c0 0x7f919c0fce30 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:03.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:03.864+0000 7f91a2000700 1 --2- 172.21.15.123:0/3440418254 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f919c102e20 0x7f919c1031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:03.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:03.864+0000 7f91a2000700 1 -- 172.21.15.123:0/3440418254 >> 172.21.15.123:0/3440418254 conn(0x7f919c0f89d0 msgr2=0x7f919c0fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:36:03.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:03.864+0000 7f91a2000700 1 -- 172.21.15.123:0/3440418254 shutdown_connections 2024-03-30T04:36:03.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:03.864+0000 7f91a2000700 1 -- 172.21.15.123:0/3440418254 wait complete. 2024-03-30T04:36:03.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:03.865+0000 7f91a2000700 1 Processor -- start 2024-03-30T04:36:03.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:03.865+0000 7f91a2000700 1 -- start start 2024-03-30T04:36:03.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:03.866+0000 7f91a2000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f919c102e20 0x7f919c115c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:36:03.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:03.866+0000 7f91a2000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f919c1037c0 0x7f919c116190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:36:03.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:03.866+0000 7f91a2000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f919c1167e0 con 0x7f919c1037c0 2024-03-30T04:36:03.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:03.866+0000 7f91a2000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f919c10fbb0 con 0x7f919c102e20 2024-03-30T04:36:03.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:03.866+0000 7f919aa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f919c1037c0 0x7f919c116190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:36:03.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:03.866+0000 7f919aa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f919c1037c0 0x7f919c116190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36606/0 (socket says 172.21.15.123:36606) 2024-03-30T04:36:03.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:03.866+0000 7f919aa00700 1 -- 172.21.15.123:0/2140199493 learned_addr learned my addr 172.21.15.123:0/2140199493 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:36:03.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:03.866+0000 7f919b400700 1 --2- 172.21.15.123:0/2140199493 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f919c102e20 0x7f919c115c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:36:03.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:03.866+0000 7f919aa00700 1 -- 172.21.15.123:0/2140199493 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f919c102e20 msgr2=0x7f919c115c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:03.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:03.866+0000 7f919aa00700 1 --2- 172.21.15.123:0/2140199493 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f919c102e20 0x7f919c115c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:03.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:03.866+0000 7f919aa00700 1 -- 172.21.15.123:0/2140199493 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9188005040 con 0x7f919c1037c0 2024-03-30T04:36:03.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:03.867+0000 7f919aa00700 1 --2- 172.21.15.123:0/2140199493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f919c1037c0 0x7f919c116190 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f91880093b0 tx=0x7f9188009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:36:03.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:03.867+0000 7f9193400700 1 -- 172.21.15.123:0/2140199493 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f918800e040 con 0x7f919c1037c0 2024-03-30T04:36:03.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:03.867+0000 7f9193400700 1 -- 172.21.15.123:0/2140199493 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9188013420 con 0x7f919c1037c0 2024-03-30T04:36:03.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:03.867+0000 7f91a2000700 1 -- 172.21.15.123:0/2140199493 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f919c10fe30 con 0x7f919c1037c0 2024-03-30T04:36:03.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:03.867+0000 7f9193400700 1 -- 172.21.15.123:0/2140199493 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f918801c630 con 0x7f919c1037c0 2024-03-30T04:36:03.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:03.867+0000 7f91a2000700 1 -- 172.21.15.123:0/2140199493 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f919c1103f0 con 0x7f919c1037c0 2024-03-30T04:36:03.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:03.868+0000 7f91a2000700 1 -- 172.21.15.123:0/2140199493 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f919c0fe570 con 0x7f919c1037c0 2024-03-30T04:36:03.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:03.869+0000 7f9193400700 1 -- 172.21.15.123:0/2140199493 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f91880139e0 con 0x7f919c1037c0 2024-03-30T04:36:03.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:03.870+0000 7f9193400700 1 --2- 172.21.15.123:0/2140199493 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f918006cc20 0x7f918006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:36:03.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:03.870+0000 7f9193400700 1 -- 172.21.15.123:0/2140199493 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f918808d180 con 0x7f919c1037c0 2024-03-30T04:36:03.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:03.870+0000 7f919b400700 1 --2- 172.21.15.123:0/2140199493 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f918006cc20 0x7f918006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:36:03.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:03.871+0000 7f919b400700 1 --2- 172.21.15.123:0/2140199493 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f918006cc20 0x7f918006f0d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f918c009fd0 tx=0x7f918c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:36:03.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:03.875+0000 7f9193400700 1 -- 172.21.15.123:0/2140199493 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f918805ad00 con 0x7f919c1037c0 2024-03-30T04:36:04.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.095+0000 7f91a2000700 1 -- 172.21.15.123:0/2140199493 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f919c109e30 con 0x7f918006cc20 2024-03-30T04:36:04.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.097+0000 7f9193400700 1 -- 172.21.15.123:0/2140199493 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f919c109e30 con 0x7f918006cc20 2024-03-30T04:36:04.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.099+0000 7f91a2000700 1 -- 172.21.15.123:0/2140199493 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f918006cc20 msgr2=0x7f918006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:04.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.099+0000 7f91a2000700 1 --2- 172.21.15.123:0/2140199493 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f918006cc20 0x7f918006f0d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f918c009fd0 tx=0x7f918c00b040 comp rx=0 tx=0).stop 2024-03-30T04:36:04.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.099+0000 7f91a2000700 1 -- 172.21.15.123:0/2140199493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f919c1037c0 msgr2=0x7f919c116190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:04.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.099+0000 7f91a2000700 1 --2- 172.21.15.123:0/2140199493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f919c1037c0 0x7f919c116190 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f91880093b0 tx=0x7f9188009e30 comp rx=0 tx=0).stop 2024-03-30T04:36:04.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.099+0000 7f91a2000700 1 -- 172.21.15.123:0/2140199493 shutdown_connections 2024-03-30T04:36:04.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.099+0000 7f91a2000700 1 --2- 172.21.15.123:0/2140199493 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f918006cc20 0x7f918006f0d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:04.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.099+0000 7f91a2000700 1 --2- 172.21.15.123:0/2140199493 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f919c102e20 0x7f919c115c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:04.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.099+0000 7f91a2000700 1 --2- 172.21.15.123:0/2140199493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f919c1037c0 0x7f919c116190 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:04.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.099+0000 7f91a2000700 1 -- 172.21.15.123:0/2140199493 >> 172.21.15.123:0/2140199493 conn(0x7f919c0f89d0 msgr2=0x7f919c0f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:36:04.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.099+0000 7f91a2000700 1 -- 172.21.15.123:0/2140199493 shutdown_connections 2024-03-30T04:36:04.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.099+0000 7f91a2000700 1 -- 172.21.15.123:0/2140199493 wait complete. 2024-03-30T04:36:04.112 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:36:04.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.234+0000 7fa08ee00700 1 -- 172.21.15.123:0/1275646205 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa088102e00 msgr2=0x7fa0881031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:04.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.234+0000 7fa08ee00700 1 --2- 172.21.15.123:0/1275646205 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa088102e00 0x7fa0881031d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa078004940 tx=0x7fa078009e30 comp rx=0 tx=0).stop 2024-03-30T04:36:04.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.234+0000 7fa08ee00700 1 -- 172.21.15.123:0/1275646205 shutdown_connections 2024-03-30T04:36:04.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.234+0000 7fa08ee00700 1 --2- 172.21.15.123:0/1275646205 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa0881037a0 0x7fa0880fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:04.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.234+0000 7fa08ee00700 1 --2- 172.21.15.123:0/1275646205 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa088102e00 0x7fa0881031d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:04.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.234+0000 7fa08ee00700 1 -- 172.21.15.123:0/1275646205 >> 172.21.15.123:0/1275646205 conn(0x7fa0880f89b0 msgr2=0x7fa0880fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:36:04.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.234+0000 7fa08ee00700 1 -- 172.21.15.123:0/1275646205 shutdown_connections 2024-03-30T04:36:04.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.234+0000 7fa08ee00700 1 -- 172.21.15.123:0/1275646205 wait complete. 2024-03-30T04:36:04.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.235+0000 7fa08ee00700 1 Processor -- start 2024-03-30T04:36:04.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.235+0000 7fa08ee00700 1 -- start start 2024-03-30T04:36:04.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.235+0000 7fa08ee00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa088102e00 0x7fa088115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:36:04.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.235+0000 7fa08ee00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0881037a0 0x7fa088116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:36:04.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.235+0000 7fa08ee00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa088116860 con 0x7fa0881037a0 2024-03-30T04:36:04.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.235+0000 7fa08ee00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa08810fc30 con 0x7fa088102e00 2024-03-30T04:36:04.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.236+0000 7fa087400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0881037a0 0x7fa088116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:36:04.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.236+0000 7fa087400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0881037a0 0x7fa088116210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36628/0 (socket says 172.21.15.123:36628) 2024-03-30T04:36:04.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.236+0000 7fa087400700 1 -- 172.21.15.123:0/386210303 learned_addr learned my addr 172.21.15.123:0/386210303 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:36:04.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.236+0000 7fa087e00700 1 --2- 172.21.15.123:0/386210303 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa088102e00 0x7fa088115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:36:04.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.236+0000 7fa087400700 1 -- 172.21.15.123:0/386210303 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa088102e00 msgr2=0x7fa088115cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:04.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.236+0000 7fa087400700 1 --2- 172.21.15.123:0/386210303 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa088102e00 0x7fa088115cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:04.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.236+0000 7fa087400700 1 -- 172.21.15.123:0/386210303 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa078005040 con 0x7fa0881037a0 2024-03-30T04:36:04.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.236+0000 7fa087400700 1 --2- 172.21.15.123:0/386210303 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0881037a0 0x7fa088116210 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa07400c800 tx=0x7fa074004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:36:04.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.236+0000 7fa07fe00700 1 -- 172.21.15.123:0/386210303 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0740106f0 con 0x7fa0881037a0 2024-03-30T04:36:04.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.237+0000 7fa07fe00700 1 -- 172.21.15.123:0/386210303 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa074010d00 con 0x7fa0881037a0 2024-03-30T04:36:04.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.237+0000 7fa08ee00700 1 -- 172.21.15.123:0/386210303 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa08810ff10 con 0x7fa0881037a0 2024-03-30T04:36:04.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.237+0000 7fa07fe00700 1 -- 172.21.15.123:0/386210303 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0740113a0 con 0x7fa0881037a0 2024-03-30T04:36:04.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.237+0000 7fa08ee00700 1 -- 172.21.15.123:0/386210303 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0881104d0 con 0x7fa0881037a0 2024-03-30T04:36:04.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.238+0000 7fa08ee00700 1 -- 172.21.15.123:0/386210303 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0880fe550 con 0x7fa0881037a0 2024-03-30T04:36:04.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.239+0000 7fa07fe00700 1 -- 172.21.15.123:0/386210303 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fa074012400 con 0x7fa0881037a0 2024-03-30T04:36:04.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.240+0000 7fa07fe00700 1 --2- 172.21.15.123:0/386210303 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa06c06cc30 0x7fa06c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:36:04.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.240+0000 7fa07fe00700 1 -- 172.21.15.123:0/386210303 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fa07408bf00 con 0x7fa0881037a0 2024-03-30T04:36:04.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.240+0000 7fa087e00700 1 --2- 172.21.15.123:0/386210303 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa06c06cc30 0x7fa06c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:36:04.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.241+0000 7fa087e00700 1 --2- 172.21.15.123:0/386210303 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa06c06cc30 0x7fa06c06f0e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa078009fd0 tx=0x7fa078009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:36:04.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.244+0000 7fa07fe00700 1 -- 172.21.15.123:0/386210303 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa074059a50 con 0x7fa0881037a0 2024-03-30T04:36:04.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:04 smithi123 ceph-mon[139369]: pgmap v130: 65 pgs: 8 peering, 15 active+undersized+degraded, 11 active+undersized, 31 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s; 55/324 objects degraded (16.975%) 2024-03-30T04:36:04.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.467+0000 7fa08ee00700 1 -- 172.21.15.123:0/386210303 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa088109e30 con 0x7fa06c06cc30 2024-03-30T04:36:04.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.469+0000 7fa07fe00700 1 -- 172.21.15.123:0/386210303 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7fa088109e30 con 0x7fa06c06cc30 2024-03-30T04:36:04.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.471+0000 7fa08ee00700 1 -- 172.21.15.123:0/386210303 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa06c06cc30 msgr2=0x7fa06c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:04.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.471+0000 7fa08ee00700 1 --2- 172.21.15.123:0/386210303 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa06c06cc30 0x7fa06c06f0e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa078009fd0 tx=0x7fa078009e30 comp rx=0 tx=0).stop 2024-03-30T04:36:04.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.471+0000 7fa08ee00700 1 -- 172.21.15.123:0/386210303 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0881037a0 msgr2=0x7fa088116210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:04.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.471+0000 7fa08ee00700 1 --2- 172.21.15.123:0/386210303 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0881037a0 0x7fa088116210 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa07400c800 tx=0x7fa074004470 comp rx=0 tx=0).stop 2024-03-30T04:36:04.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.471+0000 7fa08ee00700 1 -- 172.21.15.123:0/386210303 shutdown_connections 2024-03-30T04:36:04.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.471+0000 7fa08ee00700 1 --2- 172.21.15.123:0/386210303 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa06c06cc30 0x7fa06c06f0e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:04.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.471+0000 7fa08ee00700 1 --2- 172.21.15.123:0/386210303 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa088102e00 0x7fa088115cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:04.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.471+0000 7fa08ee00700 1 --2- 172.21.15.123:0/386210303 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0881037a0 0x7fa088116210 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:04.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.471+0000 7fa08ee00700 1 -- 172.21.15.123:0/386210303 >> 172.21.15.123:0/386210303 conn(0x7fa0880f89b0 msgr2=0x7fa0880f99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:36:04.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.471+0000 7fa08ee00700 1 -- 172.21.15.123:0/386210303 shutdown_connections 2024-03-30T04:36:04.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.471+0000 7fa08ee00700 1 -- 172.21.15.123:0/386210303 wait complete. 2024-03-30T04:36:04.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:04 smithi135 ceph-mon[106011]: pgmap v130: 65 pgs: 8 peering, 15 active+undersized+degraded, 11 active+undersized, 31 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s; 55/324 objects degraded (16.975%) 2024-03-30T04:36:04.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.604+0000 7fa3c4000700 1 -- 172.21.15.123:0/2895385521 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3bc1037a0 msgr2=0x7fa3bc0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:04.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.604+0000 7fa3c4000700 1 --2- 172.21.15.123:0/2895385521 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3bc1037a0 0x7fa3bc0fce10 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa3a8004940 tx=0x7fa3a8009e30 comp rx=0 tx=0).stop 2024-03-30T04:36:04.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.605+0000 7fa3c4000700 1 -- 172.21.15.123:0/2895385521 shutdown_connections 2024-03-30T04:36:04.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.605+0000 7fa3c4000700 1 --2- 172.21.15.123:0/2895385521 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3bc1037a0 0x7fa3bc0fce10 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:04.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.605+0000 7fa3c4000700 1 --2- 172.21.15.123:0/2895385521 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa3bc102e00 0x7fa3bc1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:04.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.605+0000 7fa3c4000700 1 -- 172.21.15.123:0/2895385521 >> 172.21.15.123:0/2895385521 conn(0x7fa3bc0f89b0 msgr2=0x7fa3bc0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:36:04.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.605+0000 7fa3c4000700 1 -- 172.21.15.123:0/2895385521 shutdown_connections 2024-03-30T04:36:04.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.605+0000 7fa3c4000700 1 -- 172.21.15.123:0/2895385521 wait complete. 2024-03-30T04:36:04.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.606+0000 7fa3c4000700 1 Processor -- start 2024-03-30T04:36:04.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.606+0000 7fa3c4000700 1 -- start start 2024-03-30T04:36:04.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.606+0000 7fa3c4000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa3bc102e00 0x7fa3bc113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:36:04.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.606+0000 7fa3c4000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3bc1037a0 0x7fa3bc114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:36:04.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.606+0000 7fa3c4000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3bc114700 con 0x7fa3bc1037a0 2024-03-30T04:36:04.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.606+0000 7fa3c4000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3bc10da60 con 0x7fa3bc102e00 2024-03-30T04:36:04.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.606+0000 7fa3c0c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3bc1037a0 0x7fa3bc114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:36:04.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.606+0000 7fa3c0c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3bc1037a0 0x7fa3bc114020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36634/0 (socket says 172.21.15.123:36634) 2024-03-30T04:36:04.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.606+0000 7fa3c0c00700 1 -- 172.21.15.123:0/2448043986 learned_addr learned my addr 172.21.15.123:0/2448043986 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:36:04.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.607+0000 7fa3c0c00700 1 -- 172.21.15.123:0/2448043986 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa3bc102e00 msgr2=0x7fa3bc113ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:04.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.607+0000 7fa3c0c00700 1 --2- 172.21.15.123:0/2448043986 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa3bc102e00 0x7fa3bc113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:04.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.607+0000 7fa3c0c00700 1 -- 172.21.15.123:0/2448043986 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3a8005040 con 0x7fa3bc1037a0 2024-03-30T04:36:04.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.607+0000 7fa3c0c00700 1 --2- 172.21.15.123:0/2448043986 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3bc1037a0 0x7fa3bc114020 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa3a8009390 tx=0x7fa3a8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:36:04.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.607+0000 7fa3b9600700 1 -- 172.21.15.123:0/2448043986 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3a800e040 con 0x7fa3bc1037a0 2024-03-30T04:36:04.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.607+0000 7fa3b9600700 1 -- 172.21.15.123:0/2448043986 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa3a8013420 con 0x7fa3bc1037a0 2024-03-30T04:36:04.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.607+0000 7fa3c4000700 1 -- 172.21.15.123:0/2448043986 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3bc10dce0 con 0x7fa3bc1037a0 2024-03-30T04:36:04.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.607+0000 7fa3b9600700 1 -- 172.21.15.123:0/2448043986 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3a801c630 con 0x7fa3bc1037a0 2024-03-30T04:36:04.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.607+0000 7fa3c4000700 1 -- 172.21.15.123:0/2448043986 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3bc10e2a0 con 0x7fa3bc1037a0 2024-03-30T04:36:04.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.608+0000 7fa3c4000700 1 -- 172.21.15.123:0/2448043986 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3bc0fe550 con 0x7fa3bc1037a0 2024-03-30T04:36:04.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.609+0000 7fa3b9600700 1 -- 172.21.15.123:0/2448043986 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fa3a80139e0 con 0x7fa3bc1037a0 2024-03-30T04:36:04.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.610+0000 7fa3b9600700 1 --2- 172.21.15.123:0/2448043986 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa3a406cc30 0x7fa3a406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:36:04.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.610+0000 7fa3b9600700 1 -- 172.21.15.123:0/2448043986 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fa3a808d2e0 con 0x7fa3bc1037a0 2024-03-30T04:36:04.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.610+0000 7fa3c1600700 1 --2- 172.21.15.123:0/2448043986 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa3a406cc30 0x7fa3a406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:36:04.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.611+0000 7fa3c1600700 1 --2- 172.21.15.123:0/2448043986 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa3a406cc30 0x7fa3a406f0e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa3b0006fd0 tx=0x7fa3b0006db0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:36:04.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.615+0000 7fa3b9600700 1 -- 172.21.15.123:0/2448043986 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa3a805adb0 con 0x7fa3bc1037a0 2024-03-30T04:36:04.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.824+0000 7fa3c4000700 1 -- 172.21.15.123:0/2448043986 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fa3bc109e80 con 0x7fa3a406cc30 2024-03-30T04:36:04.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.833+0000 7fa3b9600700 1 -- 172.21.15.123:0/2448043986 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fa3bc109e80 con 0x7fa3a406cc30 2024-03-30T04:36:04.832 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:36:04.832 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (5m) 45s ago 17m 14.7M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:36:04.832 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (2m) 45s ago 18m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T04:36:04.832 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (2m) 8s ago 16m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T04:36:04.832 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (4m) 45s ago 17m 74.1M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T04:36:04.832 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (11m) 45s ago 11m 109M - 17.2.7-797-g5c024335 f77085f6aab7 2df066fba7a6 2024-03-30T04:36:04.832 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (11m) 45s ago 11m 58.0M - 17.2.7-797-g5c024335 f77085f6aab7 1293b8520db1 2024-03-30T04:36:04.832 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (11m) 8s ago 11m 67.5M - 17.2.7-797-g5c024335 f77085f6aab7 0dc41326818c 2024-03-30T04:36:04.832 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (11m) 8s ago 11m 119M - 17.2.7-797-g5c024335 f77085f6aab7 aef8a0cfc553 2024-03-30T04:36:04.832 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (7m) 45s ago 19m 502M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:36:04.832 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (6m) 8s ago 16m 449M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:36:04.832 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (3m) 45s ago 19m 45.6M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T04:36:04.833 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (3m) 8s ago 16m 37.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T04:36:04.833 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (5m) 45s ago 17m 7583k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:36:04.833 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (5m) 8s ago 16m 7922k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:36:04.833 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (2m) 45s ago 15m 99.5M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T04:36:04.833 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (93s) 45s ago 15m 90.4M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T04:36:04.833 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (52s) 45s ago 14m 10.0M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T04:36:04.833 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (13s) 8s ago 14m 10.9M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T04:36:04.833 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (13m) 8s ago 13m 228M 2642M 17.2.7-797-g5c024335 f77085f6aab7 b74988e947e5 2024-03-30T04:36:04.833 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (13m) 8s ago 13m 219M 2642M 17.2.7-797-g5c024335 f77085f6aab7 1fd3185f5484 2024-03-30T04:36:04.833 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (5m) 45s ago 17m 52.5M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:36:04.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.836+0000 7fa3c4000700 1 -- 172.21.15.123:0/2448043986 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa3a406cc30 msgr2=0x7fa3a406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:04.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.836+0000 7fa3c4000700 1 --2- 172.21.15.123:0/2448043986 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa3a406cc30 0x7fa3a406f0e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa3b0006fd0 tx=0x7fa3b0006db0 comp rx=0 tx=0).stop 2024-03-30T04:36:04.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.836+0000 7fa3c4000700 1 -- 172.21.15.123:0/2448043986 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3bc1037a0 msgr2=0x7fa3bc114020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:04.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.836+0000 7fa3c4000700 1 --2- 172.21.15.123:0/2448043986 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3bc1037a0 0x7fa3bc114020 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa3a8009390 tx=0x7fa3a8009e30 comp rx=0 tx=0).stop 2024-03-30T04:36:04.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.836+0000 7fa3c4000700 1 -- 172.21.15.123:0/2448043986 shutdown_connections 2024-03-30T04:36:04.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.836+0000 7fa3c4000700 1 --2- 172.21.15.123:0/2448043986 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa3a406cc30 0x7fa3a406f0e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:04.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.836+0000 7fa3c4000700 1 --2- 172.21.15.123:0/2448043986 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa3bc102e00 0x7fa3bc113ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:04.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.836+0000 7fa3c4000700 1 --2- 172.21.15.123:0/2448043986 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3bc1037a0 0x7fa3bc114020 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:04.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.836+0000 7fa3c4000700 1 -- 172.21.15.123:0/2448043986 >> 172.21.15.123:0/2448043986 conn(0x7fa3bc0f89b0 msgr2=0x7fa3bc0f99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:36:04.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.836+0000 7fa3c4000700 1 -- 172.21.15.123:0/2448043986 shutdown_connections 2024-03-30T04:36:04.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.836+0000 7fa3c4000700 1 -- 172.21.15.123:0/2448043986 wait complete. 2024-03-30T04:36:04.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.968+0000 7f615da00700 1 -- 172.21.15.123:0/221424642 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6158102e00 msgr2=0x7f61581031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:04.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.968+0000 7f615da00700 1 --2- 172.21.15.123:0/221424642 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6158102e00 0x7f61581031d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6148004940 tx=0x7f6148009e30 comp rx=0 tx=0).stop 2024-03-30T04:36:04.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.968+0000 7f615da00700 1 -- 172.21.15.123:0/221424642 shutdown_connections 2024-03-30T04:36:04.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.968+0000 7f615da00700 1 --2- 172.21.15.123:0/221424642 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f61581037a0 0x7f61580fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:04.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.968+0000 7f615da00700 1 --2- 172.21.15.123:0/221424642 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6158102e00 0x7f61581031d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:04.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.968+0000 7f615da00700 1 -- 172.21.15.123:0/221424642 >> 172.21.15.123:0/221424642 conn(0x7f61580f89b0 msgr2=0x7f61580fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:36:04.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.968+0000 7f615da00700 1 -- 172.21.15.123:0/221424642 shutdown_connections 2024-03-30T04:36:04.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.968+0000 7f615da00700 1 -- 172.21.15.123:0/221424642 wait complete. 2024-03-30T04:36:04.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.969+0000 7f615da00700 1 Processor -- start 2024-03-30T04:36:04.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.969+0000 7f615da00700 1 -- start start 2024-03-30T04:36:04.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.969+0000 7f615da00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6158102e00 0x7f61580702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:36:04.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.969+0000 7f615da00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f61581037a0 0x7f6158070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:36:04.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.969+0000 7f615da00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6158070d90 con 0x7f6158102e00 2024-03-30T04:36:04.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.969+0000 7f615da00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6158070ed0 con 0x7f61581037a0 2024-03-30T04:36:04.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.969+0000 7f6156a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6158102e00 0x7f61580702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:36:04.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.969+0000 7f6156a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6158102e00 0x7f61580702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36654/0 (socket says 172.21.15.123:36654) 2024-03-30T04:36:04.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.969+0000 7f6156a00700 1 -- 172.21.15.123:0/2442678259 learned_addr learned my addr 172.21.15.123:0/2442678259 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:36:04.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.970+0000 7f6156a00700 1 -- 172.21.15.123:0/2442678259 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f61581037a0 msgr2=0x7f6158070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:04.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.970+0000 7f6156a00700 1 --2- 172.21.15.123:0/2442678259 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f61581037a0 0x7f6158070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:04.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.970+0000 7f6156a00700 1 -- 172.21.15.123:0/2442678259 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6148005040 con 0x7f6158102e00 2024-03-30T04:36:04.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.970+0000 7f6156a00700 1 --2- 172.21.15.123:0/2442678259 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6158102e00 0x7f61580702c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6148009390 tx=0x7f6148009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:36:04.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.970+0000 7f614ea00700 1 -- 172.21.15.123:0/2442678259 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f614800e040 con 0x7f6158102e00 2024-03-30T04:36:04.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.970+0000 7f614ea00700 1 -- 172.21.15.123:0/2442678259 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6148013420 con 0x7f6158102e00 2024-03-30T04:36:04.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.970+0000 7f615da00700 1 -- 172.21.15.123:0/2442678259 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6158071120 con 0x7f6158102e00 2024-03-30T04:36:04.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.970+0000 7f614ea00700 1 -- 172.21.15.123:0/2442678259 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f614801c630 con 0x7f6158102e00 2024-03-30T04:36:04.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.970+0000 7f615da00700 1 -- 172.21.15.123:0/2442678259 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f615811a170 con 0x7f6158102e00 2024-03-30T04:36:04.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.972+0000 7f615da00700 1 -- 172.21.15.123:0/2442678259 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61580fe550 con 0x7f6158102e00 2024-03-30T04:36:04.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.973+0000 7f614ea00700 1 -- 172.21.15.123:0/2442678259 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f61480139e0 con 0x7f6158102e00 2024-03-30T04:36:04.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.973+0000 7f614ea00700 1 --2- 172.21.15.123:0/2442678259 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f613c06cc30 0x7f613c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:36:04.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.973+0000 7f614ea00700 1 -- 172.21.15.123:0/2442678259 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f614808d1f0 con 0x7f6158102e00 2024-03-30T04:36:04.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.974+0000 7f6156000700 1 --2- 172.21.15.123:0/2442678259 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f613c06cc30 0x7f613c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:36:04.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.974+0000 7f6156000700 1 --2- 172.21.15.123:0/2442678259 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f613c06cc30 0x7f613c06f0e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6144009b80 tx=0x7f6144009d60 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:36:04.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:04.978+0000 7f614ea00700 1 -- 172.21.15.123:0/2442678259 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f61480572d0 con 0x7f6158102e00 2024-03-30T04:36:05.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.263+0000 7f615da00700 1 -- 172.21.15.123:0/2442678259 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f615804bed0 con 0x7f6158102e00 2024-03-30T04:36:05.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.264+0000 7f614ea00700 1 -- 172.21.15.123:0/2442678259 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f614805a8f0 con 0x7f6158102e00 2024-03-30T04:36:05.262 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:36:05.262 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:36:05.263 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:36:05.263 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:36:05.263 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:36:05.263 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:36:05.263 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:36:05.263 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:36:05.263 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 2, 2024-03-30T04:36:05.263 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 4 2024-03-30T04:36:05.263 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:36:05.263 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:36:05.263 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 4 2024-03-30T04:36:05.263 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:36:05.264 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:36:05.264 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 6, 2024-03-30T04:36:05.264 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 8 2024-03-30T04:36:05.264 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:36:05.264 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:36:05.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.266+0000 7f615da00700 1 -- 172.21.15.123:0/2442678259 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f613c06cc30 msgr2=0x7f613c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:05.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.266+0000 7f615da00700 1 --2- 172.21.15.123:0/2442678259 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f613c06cc30 0x7f613c06f0e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6144009b80 tx=0x7f6144009d60 comp rx=0 tx=0).stop 2024-03-30T04:36:05.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.266+0000 7f615da00700 1 -- 172.21.15.123:0/2442678259 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6158102e00 msgr2=0x7f61580702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:05.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.266+0000 7f615da00700 1 --2- 172.21.15.123:0/2442678259 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6158102e00 0x7f61580702c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6148009390 tx=0x7f6148009e30 comp rx=0 tx=0).stop 2024-03-30T04:36:05.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.266+0000 7f615da00700 1 -- 172.21.15.123:0/2442678259 shutdown_connections 2024-03-30T04:36:05.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.266+0000 7f615da00700 1 --2- 172.21.15.123:0/2442678259 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f613c06cc30 0x7f613c06f0e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:05.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.266+0000 7f615da00700 1 --2- 172.21.15.123:0/2442678259 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6158102e00 0x7f61580702c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:05.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.266+0000 7f615da00700 1 --2- 172.21.15.123:0/2442678259 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f61581037a0 0x7f6158070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:05.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.266+0000 7f615da00700 1 -- 172.21.15.123:0/2442678259 >> 172.21.15.123:0/2442678259 conn(0x7f61580f89b0 msgr2=0x7f61580f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:36:05.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.266+0000 7f615da00700 1 -- 172.21.15.123:0/2442678259 shutdown_connections 2024-03-30T04:36:05.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.266+0000 7f615da00700 1 -- 172.21.15.123:0/2442678259 wait complete. 2024-03-30T04:36:05.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.400+0000 7fde8fc00700 1 -- 172.21.15.123:0/2657041622 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde881037a0 msgr2=0x7fde880fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:05.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.400+0000 7fde8fc00700 1 --2- 172.21.15.123:0/2657041622 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde881037a0 0x7fde880fce10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fde74004940 tx=0x7fde74009e30 comp rx=0 tx=0).stop 2024-03-30T04:36:05.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.400+0000 7fde8fc00700 1 -- 172.21.15.123:0/2657041622 shutdown_connections 2024-03-30T04:36:05.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.400+0000 7fde8fc00700 1 --2- 172.21.15.123:0/2657041622 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde881037a0 0x7fde880fce10 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:05.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.400+0000 7fde8fc00700 1 --2- 172.21.15.123:0/2657041622 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fde88102e00 0x7fde881031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:05.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.400+0000 7fde8fc00700 1 -- 172.21.15.123:0/2657041622 >> 172.21.15.123:0/2657041622 conn(0x7fde880f89b0 msgr2=0x7fde880fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:36:05.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.400+0000 7fde8fc00700 1 -- 172.21.15.123:0/2657041622 shutdown_connections 2024-03-30T04:36:05.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.400+0000 7fde8fc00700 1 -- 172.21.15.123:0/2657041622 wait complete. 2024-03-30T04:36:05.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.401+0000 7fde8fc00700 1 Processor -- start 2024-03-30T04:36:05.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.401+0000 7fde8fc00700 1 -- start start 2024-03-30T04:36:05.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.402+0000 7fde8fc00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fde88102e00 0x7fde88113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:36:05.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.402+0000 7fde8fc00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde881037a0 0x7fde88114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:36:05.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.402+0000 7fde8fc00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde88114690 con 0x7fde881037a0 2024-03-30T04:36:05.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.402+0000 7fde8fc00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8810da80 con 0x7fde88102e00 2024-03-30T04:36:05.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.402+0000 7fde87e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde881037a0 0x7fde88114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:36:05.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.402+0000 7fde87e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde881037a0 0x7fde88114040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36672/0 (socket says 172.21.15.123:36672) 2024-03-30T04:36:05.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.402+0000 7fde87e00700 1 -- 172.21.15.123:0/785362780 learned_addr learned my addr 172.21.15.123:0/785362780 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:36:05.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.402+0000 7fde8d200700 1 --2- 172.21.15.123:0/785362780 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fde88102e00 0x7fde88113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:36:05.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.403+0000 7fde87e00700 1 -- 172.21.15.123:0/785362780 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fde88102e00 msgr2=0x7fde88113b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:05.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.403+0000 7fde87e00700 1 --2- 172.21.15.123:0/785362780 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fde88102e00 0x7fde88113b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:05.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.403+0000 7fde87e00700 1 -- 172.21.15.123:0/785362780 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde74005040 con 0x7fde881037a0 2024-03-30T04:36:05.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.403+0000 7fde87e00700 1 --2- 172.21.15.123:0/785362780 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde881037a0 0x7fde88114040 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fde74009390 tx=0x7fde74009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:36:05.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.403+0000 7fde84c00700 1 -- 172.21.15.123:0/785362780 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde7400e040 con 0x7fde881037a0 2024-03-30T04:36:05.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.403+0000 7fde84c00700 1 -- 172.21.15.123:0/785362780 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fde74013420 con 0x7fde881037a0 2024-03-30T04:36:05.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.403+0000 7fde8fc00700 1 -- 172.21.15.123:0/785362780 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde8810dd00 con 0x7fde881037a0 2024-03-30T04:36:05.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.403+0000 7fde84c00700 1 -- 172.21.15.123:0/785362780 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde7401c630 con 0x7fde881037a0 2024-03-30T04:36:05.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.403+0000 7fde8fc00700 1 -- 172.21.15.123:0/785362780 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde8810e2c0 con 0x7fde881037a0 2024-03-30T04:36:05.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.404+0000 7fde8fc00700 1 -- 172.21.15.123:0/785362780 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde880fe550 con 0x7fde881037a0 2024-03-30T04:36:05.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.405+0000 7fde84c00700 1 -- 172.21.15.123:0/785362780 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fde740139e0 con 0x7fde881037a0 2024-03-30T04:36:05.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.406+0000 7fde84c00700 1 --2- 172.21.15.123:0/785362780 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fde7006cc30 0x7fde7006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:36:05.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.407+0000 7fde84c00700 1 -- 172.21.15.123:0/785362780 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fde7408d2e0 con 0x7fde881037a0 2024-03-30T04:36:05.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.407+0000 7fde8d200700 1 --2- 172.21.15.123:0/785362780 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fde7006cc30 0x7fde7006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:36:05.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.407+0000 7fde8d200700 1 --2- 172.21.15.123:0/785362780 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fde7006cc30 0x7fde7006f0e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fde7c007fd0 tx=0x7fde7c009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:36:05.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.411+0000 7fde84c00700 1 -- 172.21.15.123:0/785362780 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fde7405adb0 con 0x7fde881037a0 2024-03-30T04:36:05.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:05 smithi123 ceph-mon[139369]: from='client.34316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:36:05.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:05 smithi123 ceph-mon[139369]: Health check update: Degraded data redundancy: 55/324 objects degraded (16.975%), 15 pgs degraded (PG_DEGRADED) 2024-03-30T04:36:05.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:05 smithi123 ceph-mon[139369]: from='client.34320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:36:05.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:05 smithi123 ceph-mon[139369]: from='client.34324 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:36:05.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:05 smithi135 ceph-mon[106011]: from='client.34316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:36:05.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:05 smithi135 ceph-mon[106011]: Health check update: Degraded data redundancy: 55/324 objects degraded (16.975%), 15 pgs degraded (PG_DEGRADED) 2024-03-30T04:36:05.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:05 smithi135 ceph-mon[106011]: from='client.34320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:36:05.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:05 smithi135 ceph-mon[106011]: from='client.34324 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:36:05.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.657+0000 7fde8fc00700 1 -- 172.21.15.123:0/785362780 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fde88109da0 con 0x7fde881037a0 2024-03-30T04:36:05.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.657+0000 7fde84c00700 1 -- 172.21.15.123:0/785362780 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 23 v23) v1 ==== 76+0+1876 (secure 0 0 0) 0x7fde7405a940 con 0x7fde881037a0 2024-03-30T04:36:05.656 INFO:teuthology.orchestra.run.smithi123.stdout:e23 2024-03-30T04:36:05.656 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:36:05.656 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:36:05.656 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:36:05.656 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:36:05.656 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:36:05.656 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:36:05.656 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 23 2024-03-30T04:36:05.656 INFO:teuthology.orchestra.run.smithi123.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-30T04:36:05.657 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:36:05.657 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:32:48.448604+0000 2024-03-30T04:36:05.657 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:36:05.657 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:36:05.657 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:36:05.657 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:36:05.657 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:36:05.658 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T04:36:05.658 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:36:05.658 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:36:05.658 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 46 2024-03-30T04:36:05.658 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:36:05.658 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T04:36:05.659 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T04:36:05.659 INFO:teuthology.orchestra.run.smithi123.stdout:up {0=14488,1=24309} 2024-03-30T04:36:05.659 INFO:teuthology.orchestra.run.smithi123.stdout:failed 2024-03-30T04:36:05.659 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:36:05.659 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:36:05.659 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:36:05.659 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:36:05.659 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:36:05.659 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:36:05.659 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T04:36:05.659 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T04:36:05.659 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{0:14488} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.123:6828/3037849622,v1:172.21.15.123:6829/3037849622] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:36:05.660 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{0:14500} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/2433675995,v1:172.21.15.123:6827/2433675995] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:36:05.660 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{1:24309} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.135:6824/4050541062,v1:172.21.15.135:6825/4050541062] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:36:05.660 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{1:24319} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.135:6826/1096986122,v1:172.21.15.135:6827/1096986122] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:36:05.660 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:36:05.660 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:36:05.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.659+0000 7fde8fc00700 1 -- 172.21.15.123:0/785362780 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fde7006cc30 msgr2=0x7fde7006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:05.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.659+0000 7fde8fc00700 1 --2- 172.21.15.123:0/785362780 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fde7006cc30 0x7fde7006f0e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fde7c007fd0 tx=0x7fde7c009040 comp rx=0 tx=0).stop 2024-03-30T04:36:05.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.659+0000 7fde8fc00700 1 -- 172.21.15.123:0/785362780 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde881037a0 msgr2=0x7fde88114040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:05.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.659+0000 7fde8fc00700 1 --2- 172.21.15.123:0/785362780 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde881037a0 0x7fde88114040 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fde74009390 tx=0x7fde74009e30 comp rx=0 tx=0).stop 2024-03-30T04:36:05.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.660+0000 7fde8fc00700 1 -- 172.21.15.123:0/785362780 shutdown_connections 2024-03-30T04:36:05.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.660+0000 7fde8fc00700 1 --2- 172.21.15.123:0/785362780 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fde7006cc30 0x7fde7006f0e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:05.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.660+0000 7fde8fc00700 1 --2- 172.21.15.123:0/785362780 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fde88102e00 0x7fde88113b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:05.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.660+0000 7fde8fc00700 1 --2- 172.21.15.123:0/785362780 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde881037a0 0x7fde88114040 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:05.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.660+0000 7fde8fc00700 1 -- 172.21.15.123:0/785362780 >> 172.21.15.123:0/785362780 conn(0x7fde880f89b0 msgr2=0x7fde880f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:36:05.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.660+0000 7fde8fc00700 1 -- 172.21.15.123:0/785362780 shutdown_connections 2024-03-30T04:36:05.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.660+0000 7fde8fc00700 1 -- 172.21.15.123:0/785362780 wait complete. 2024-03-30T04:36:05.663 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 23 2024-03-30T04:36:05.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.792+0000 7fe366800700 1 -- 172.21.15.123:0/2885269174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3601037a0 msgr2=0x7fe3600fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:05.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.792+0000 7fe366800700 1 --2- 172.21.15.123:0/2885269174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3601037a0 0x7fe3600fce10 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fe34c004940 tx=0x7fe34c009e30 comp rx=0 tx=0).stop 2024-03-30T04:36:05.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.792+0000 7fe366800700 1 -- 172.21.15.123:0/2885269174 shutdown_connections 2024-03-30T04:36:05.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.792+0000 7fe366800700 1 --2- 172.21.15.123:0/2885269174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3601037a0 0x7fe3600fce10 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:05.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.792+0000 7fe366800700 1 --2- 172.21.15.123:0/2885269174 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe360102e00 0x7fe3601031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:05.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.792+0000 7fe366800700 1 -- 172.21.15.123:0/2885269174 >> 172.21.15.123:0/2885269174 conn(0x7fe3600f89b0 msgr2=0x7fe3600fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:36:05.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.792+0000 7fe366800700 1 -- 172.21.15.123:0/2885269174 shutdown_connections 2024-03-30T04:36:05.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.792+0000 7fe366800700 1 -- 172.21.15.123:0/2885269174 wait complete. 2024-03-30T04:36:05.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.793+0000 7fe366800700 1 Processor -- start 2024-03-30T04:36:05.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.793+0000 7fe366800700 1 -- start start 2024-03-30T04:36:05.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.793+0000 7fe366800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe360102e00 0x7fe360115d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:36:05.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.793+0000 7fe366800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe3601037a0 0x7fe360116240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:36:05.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.793+0000 7fe366800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe360116890 con 0x7fe360102e00 2024-03-30T04:36:05.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.793+0000 7fe366800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe36010fc60 con 0x7fe3601037a0 2024-03-30T04:36:05.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.794+0000 7fe35f400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe360102e00 0x7fe360115d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:36:05.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.794+0000 7fe35f400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe360102e00 0x7fe360115d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36684/0 (socket says 172.21.15.123:36684) 2024-03-30T04:36:05.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.794+0000 7fe35f400700 1 -- 172.21.15.123:0/3377672705 learned_addr learned my addr 172.21.15.123:0/3377672705 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:36:05.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.794+0000 7fe35ea00700 1 --2- 172.21.15.123:0/3377672705 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe3601037a0 0x7fe360116240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:36:05.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.794+0000 7fe35f400700 1 -- 172.21.15.123:0/3377672705 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe3601037a0 msgr2=0x7fe360116240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:05.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.794+0000 7fe35f400700 1 --2- 172.21.15.123:0/3377672705 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe3601037a0 0x7fe360116240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:05.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.794+0000 7fe35f400700 1 -- 172.21.15.123:0/3377672705 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe34c005040 con 0x7fe360102e00 2024-03-30T04:36:05.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.794+0000 7fe35f400700 1 --2- 172.21.15.123:0/3377672705 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe360102e00 0x7fe360115d00 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe350002a50 tx=0x7fe3500046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:36:05.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.794+0000 7fe357400700 1 -- 172.21.15.123:0/3377672705 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe350007cc0 con 0x7fe360102e00 2024-03-30T04:36:05.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.795+0000 7fe357400700 1 -- 172.21.15.123:0/3377672705 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe35000f040 con 0x7fe360102e00 2024-03-30T04:36:05.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.795+0000 7fe366800700 1 -- 172.21.15.123:0/3377672705 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe36010ff40 con 0x7fe360102e00 2024-03-30T04:36:05.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.795+0000 7fe357400700 1 -- 172.21.15.123:0/3377672705 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe35000d9c0 con 0x7fe360102e00 2024-03-30T04:36:05.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.795+0000 7fe366800700 1 -- 172.21.15.123:0/3377672705 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe360110500 con 0x7fe360102e00 2024-03-30T04:36:05.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.796+0000 7fe366800700 1 -- 172.21.15.123:0/3377672705 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3600fe550 con 0x7fe360102e00 2024-03-30T04:36:05.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.797+0000 7fe357400700 1 -- 172.21.15.123:0/3377672705 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fe35000db20 con 0x7fe360102e00 2024-03-30T04:36:05.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.798+0000 7fe357400700 1 --2- 172.21.15.123:0/3377672705 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe34406cc30 0x7fe34406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:36:05.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.798+0000 7fe357400700 1 -- 172.21.15.123:0/3377672705 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fe35008bca0 con 0x7fe360102e00 2024-03-30T04:36:05.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.798+0000 7fe35ea00700 1 --2- 172.21.15.123:0/3377672705 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe34406cc30 0x7fe34406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:36:05.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.799+0000 7fe35ea00700 1 --2- 172.21.15.123:0/3377672705 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe34406cc30 0x7fe34406f0e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe3601115a0 tx=0x7fe34c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:36:05.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:05.803+0000 7fe357400700 1 -- 172.21.15.123:0/3377672705 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe350059770 con 0x7fe360102e00 2024-03-30T04:36:06.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.024+0000 7fe366800700 1 -- 172.21.15.123:0/3377672705 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe36004bf40 con 0x7fe34406cc30 2024-03-30T04:36:06.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.026+0000 7fe357400700 1 -- 172.21.15.123:0/3377672705 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7fe36004bf40 con 0x7fe34406cc30 2024-03-30T04:36:06.024 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:36:06.024 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:36:06.024 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:36:06.024 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T04:36:06.024 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:36:06.025 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T04:36:06.025 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T04:36:06.025 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T04:36:06.025 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:36:06.025 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "10/21 daemons upgraded", 2024-03-30T04:36:06.025 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading osd daemons", 2024-03-30T04:36:06.025 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:36:06.025 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:36:06.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.028+0000 7fe366800700 1 -- 172.21.15.123:0/3377672705 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe34406cc30 msgr2=0x7fe34406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:06.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.028+0000 7fe366800700 1 --2- 172.21.15.123:0/3377672705 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe34406cc30 0x7fe34406f0e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe3601115a0 tx=0x7fe34c009e30 comp rx=0 tx=0).stop 2024-03-30T04:36:06.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.028+0000 7fe366800700 1 -- 172.21.15.123:0/3377672705 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe360102e00 msgr2=0x7fe360115d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:06.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.028+0000 7fe366800700 1 --2- 172.21.15.123:0/3377672705 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe360102e00 0x7fe360115d00 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe350002a50 tx=0x7fe3500046b0 comp rx=0 tx=0).stop 2024-03-30T04:36:06.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.028+0000 7fe366800700 1 -- 172.21.15.123:0/3377672705 shutdown_connections 2024-03-30T04:36:06.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.028+0000 7fe366800700 1 --2- 172.21.15.123:0/3377672705 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe34406cc30 0x7fe34406f0e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:06.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.028+0000 7fe366800700 1 --2- 172.21.15.123:0/3377672705 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe360102e00 0x7fe360115d00 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:06.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.028+0000 7fe366800700 1 --2- 172.21.15.123:0/3377672705 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe3601037a0 0x7fe360116240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:06.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.028+0000 7fe366800700 1 -- 172.21.15.123:0/3377672705 >> 172.21.15.123:0/3377672705 conn(0x7fe3600f89b0 msgr2=0x7fe3600f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:36:06.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.028+0000 7fe366800700 1 -- 172.21.15.123:0/3377672705 shutdown_connections 2024-03-30T04:36:06.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.028+0000 7fe366800700 1 -- 172.21.15.123:0/3377672705 wait complete. 2024-03-30T04:36:06.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.161+0000 7f6251e00700 1 -- 172.21.15.123:0/1136254476 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f624c1037a0 msgr2=0x7f624c0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:06.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.161+0000 7f6251e00700 1 --2- 172.21.15.123:0/1136254476 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f624c1037a0 0x7f624c0fce10 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6238004940 tx=0x7f6238009e30 comp rx=0 tx=0).stop 2024-03-30T04:36:06.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.161+0000 7f6251e00700 1 -- 172.21.15.123:0/1136254476 shutdown_connections 2024-03-30T04:36:06.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.161+0000 7f6251e00700 1 --2- 172.21.15.123:0/1136254476 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f624c1037a0 0x7f624c0fce10 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:06.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.161+0000 7f6251e00700 1 --2- 172.21.15.123:0/1136254476 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f624c102e00 0x7f624c1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:06.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.161+0000 7f6251e00700 1 -- 172.21.15.123:0/1136254476 >> 172.21.15.123:0/1136254476 conn(0x7f624c0f89b0 msgr2=0x7f624c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:36:06.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.161+0000 7f6251e00700 1 -- 172.21.15.123:0/1136254476 shutdown_connections 2024-03-30T04:36:06.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.161+0000 7f6251e00700 1 -- 172.21.15.123:0/1136254476 wait complete. 2024-03-30T04:36:06.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.162+0000 7f6251e00700 1 Processor -- start 2024-03-30T04:36:06.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.162+0000 7f6251e00700 1 -- start start 2024-03-30T04:36:06.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.163+0000 7f6251e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f624c102e00 0x7f624c115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:36:06.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.163+0000 7f6251e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f624c1037a0 0x7f624c1161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:36:06.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.163+0000 7f6251e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f624c116810 con 0x7f624c102e00 2024-03-30T04:36:06.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.163+0000 7f6251e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f624c10fbe0 con 0x7f624c1037a0 2024-03-30T04:36:06.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.163+0000 7f624aa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f624c102e00 0x7f624c115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:36:06.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.163+0000 7f624aa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f624c102e00 0x7f624c115c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36696/0 (socket says 172.21.15.123:36696) 2024-03-30T04:36:06.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.163+0000 7f624aa00700 1 -- 172.21.15.123:0/3458864868 learned_addr learned my addr 172.21.15.123:0/3458864868 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:36:06.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.163+0000 7f624a000700 1 --2- 172.21.15.123:0/3458864868 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f624c1037a0 0x7f624c1161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:36:06.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.163+0000 7f624aa00700 1 -- 172.21.15.123:0/3458864868 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f624c1037a0 msgr2=0x7f624c1161c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:06.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.163+0000 7f624aa00700 1 --2- 172.21.15.123:0/3458864868 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f624c1037a0 0x7f624c1161c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:06.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.163+0000 7f624aa00700 1 -- 172.21.15.123:0/3458864868 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6238005040 con 0x7f624c102e00 2024-03-30T04:36:06.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.163+0000 7f624a000700 1 --2- 172.21.15.123:0/3458864868 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f624c1037a0 0x7f624c1161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:36:06.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.163+0000 7f624aa00700 1 --2- 172.21.15.123:0/3458864868 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f624c102e00 0x7f624c115c80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f623c00eb40 tx=0x7f623c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:36:06.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.164+0000 7f6242a00700 1 -- 172.21.15.123:0/3458864868 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f623c007cc0 con 0x7f624c102e00 2024-03-30T04:36:06.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.164+0000 7f6242a00700 1 -- 172.21.15.123:0/3458864868 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f623c00f040 con 0x7f624c102e00 2024-03-30T04:36:06.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.164+0000 7f6251e00700 1 -- 172.21.15.123:0/3458864868 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f624c10fe60 con 0x7f624c102e00 2024-03-30T04:36:06.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.164+0000 7f6242a00700 1 -- 172.21.15.123:0/3458864868 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f623c009570 con 0x7f624c102e00 2024-03-30T04:36:06.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.164+0000 7f6251e00700 1 -- 172.21.15.123:0/3458864868 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f624c110420 con 0x7f624c102e00 2024-03-30T04:36:06.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.165+0000 7f6251e00700 1 -- 172.21.15.123:0/3458864868 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f624c0fe550 con 0x7f624c102e00 2024-03-30T04:36:06.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.166+0000 7f6242a00700 1 -- 172.21.15.123:0/3458864868 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f623c0096d0 con 0x7f624c102e00 2024-03-30T04:36:06.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.167+0000 7f6242a00700 1 --2- 172.21.15.123:0/3458864868 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f623006cc30 0x7f623006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:36:06.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.167+0000 7f6242a00700 1 -- 172.21.15.123:0/3458864868 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f623c08bc80 con 0x7f624c102e00 2024-03-30T04:36:06.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.167+0000 7f624a000700 1 --2- 172.21.15.123:0/3458864868 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f623006cc30 0x7f623006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:36:06.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.168+0000 7f624a000700 1 --2- 172.21.15.123:0/3458864868 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f623006cc30 0x7f623006f0e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6238000c00 tx=0x7f6238009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:36:06.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.171+0000 7f6242a00700 1 -- 172.21.15.123:0/3458864868 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f623c059820 con 0x7f624c102e00 2024-03-30T04:36:06.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.457+0000 7f6251e00700 1 -- 172.21.15.123:0/3458864868 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f624c04bed0 con 0x7f624c102e00 2024-03-30T04:36:06.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.457+0000 7f6242a00700 1 -- 172.21.15.123:0/3458864868 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f623c0593b0 con 0x7f624c102e00 2024-03-30T04:36:06.456 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_OK 2024-03-30T04:36:06.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.459+0000 7f6251e00700 1 -- 172.21.15.123:0/3458864868 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f623006cc30 msgr2=0x7f623006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:06.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.459+0000 7f6251e00700 1 --2- 172.21.15.123:0/3458864868 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f623006cc30 0x7f623006f0e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6238000c00 tx=0x7f6238009e30 comp rx=0 tx=0).stop 2024-03-30T04:36:06.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.459+0000 7f6251e00700 1 -- 172.21.15.123:0/3458864868 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f624c102e00 msgr2=0x7f624c115c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:06.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.459+0000 7f6251e00700 1 --2- 172.21.15.123:0/3458864868 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f624c102e00 0x7f624c115c80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f623c00eb40 tx=0x7f623c00b040 comp rx=0 tx=0).stop 2024-03-30T04:36:06.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.459+0000 7f6251e00700 1 -- 172.21.15.123:0/3458864868 shutdown_connections 2024-03-30T04:36:06.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.459+0000 7f6251e00700 1 --2- 172.21.15.123:0/3458864868 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f623006cc30 0x7f623006f0e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:06.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.459+0000 7f6251e00700 1 --2- 172.21.15.123:0/3458864868 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f624c102e00 0x7f624c115c80 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:06.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.459+0000 7f6251e00700 1 --2- 172.21.15.123:0/3458864868 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f624c1037a0 0x7f624c1161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:06.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.459+0000 7f6251e00700 1 -- 172.21.15.123:0/3458864868 >> 172.21.15.123:0/3458864868 conn(0x7f624c0f89b0 msgr2=0x7f624c0f99a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:36:06.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.460+0000 7f6251e00700 1 -- 172.21.15.123:0/3458864868 shutdown_connections 2024-03-30T04:36:06.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:06.460+0000 7f6251e00700 1 -- 172.21.15.123:0/3458864868 wait complete. 2024-03-30T04:36:06.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:06 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2442678259' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:36:06.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:06 smithi135 ceph-mon[106011]: pgmap v131: 65 pgs: 65 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-30T04:36:06.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:06 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/785362780' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:36:06.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:06 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2442678259' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:36:06.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:06 smithi123 ceph-mon[139369]: pgmap v131: 65 pgs: 65 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-30T04:36:06.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:06 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/785362780' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:36:07.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:07 smithi135 ceph-mon[106011]: from='client.34336 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:36:07.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:07 smithi135 ceph-mon[106011]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 55/324 objects degraded (16.975%), 15 pgs degraded) 2024-03-30T04:36:07.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:07 smithi135 ceph-mon[106011]: Cluster is now healthy 2024-03-30T04:36:07.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:07 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3458864868' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:36:07.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:07 smithi123 ceph-mon[139369]: from='client.34336 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:36:07.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:07 smithi123 ceph-mon[139369]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 55/324 objects degraded (16.975%), 15 pgs degraded) 2024-03-30T04:36:07.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:07 smithi123 ceph-mon[139369]: Cluster is now healthy 2024-03-30T04:36:07.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:07 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3458864868' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:36:08.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:08 smithi123 ceph-mon[139369]: pgmap v132: 65 pgs: 65 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-30T04:36:08.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:08 smithi135 ceph-mon[106011]: pgmap v132: 65 pgs: 65 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-30T04:36:10.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:10 smithi123 ceph-mon[139369]: pgmap v133: 65 pgs: 65 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-30T04:36:10.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:10 smithi135 ceph-mon[106011]: pgmap v133: 65 pgs: 65 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-30T04:36:12.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:12 smithi135 ceph-mon[106011]: pgmap v134: 65 pgs: 65 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-30T04:36:12.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:12 smithi123 ceph-mon[139369]: pgmap v134: 65 pgs: 65 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-30T04:36:14.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:14 smithi135 ceph-mon[106011]: pgmap v135: 65 pgs: 65 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-30T04:36:14.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:14 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:36:14.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:14 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:36:14.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:14 smithi123 ceph-mon[139369]: pgmap v135: 65 pgs: 65 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-30T04:36:14.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:14 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:36:14.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:14 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:36:15.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:15 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:36:15.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:15 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:36:16.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:16 smithi135 ceph-mon[106011]: pgmap v136: 65 pgs: 65 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-30T04:36:16.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:16 smithi123 ceph-mon[139369]: pgmap v136: 65 pgs: 65 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-30T04:36:17.763 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:17 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-30T04:36:17.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:17 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-30T04:36:18.749 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:18 smithi135 ceph-mon[106011]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-30T04:36:18.749 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:18 smithi135 ceph-mon[106011]: Upgrade: osd.4 is safe to restart 2024-03-30T04:36:18.749 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:18 smithi135 ceph-mon[106011]: pgmap v137: 65 pgs: 65 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:36:18.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:18 smithi123 ceph-mon[139369]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-30T04:36:18.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:18 smithi123 ceph-mon[139369]: Upgrade: osd.4 is safe to restart 2024-03-30T04:36:18.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:18 smithi123 ceph-mon[139369]: pgmap v137: 65 pgs: 65 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:36:20.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:19 smithi135 ceph-mon[106011]: Upgrade: Updating osd.4 2024-03-30T04:36:20.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:19 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:36:20.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:19 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-30T04:36:20.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:19 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:36:20.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:19 smithi135 ceph-mon[106011]: Deploying daemon osd.4 on smithi135 2024-03-30T04:36:20.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:19 smithi123 ceph-mon[139369]: Upgrade: Updating osd.4 2024-03-30T04:36:20.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:19 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:36:20.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:19 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-30T04:36:20.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:19 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:36:20.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:19 smithi123 ceph-mon[139369]: Deploying daemon osd.4 on smithi135 2024-03-30T04:36:21.067 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:20 smithi135 ceph-mon[106011]: pgmap v138: 65 pgs: 65 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-30T04:36:21.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:20 smithi123 ceph-mon[139369]: pgmap v138: 65 pgs: 65 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-30T04:36:22.813 INFO:journalctl@ceph.osd.4.smithi135.stdout:Mar 30 04:36:22 smithi135 systemd[1]: Stopping Ceph osd.4 for 2e77a2be-ee4c-11ee-b647-cb9ed24678a4... 2024-03-30T04:36:23.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:22 smithi135 ceph-mon[106011]: pgmap v139: 65 pgs: 65 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-30T04:36:23.101 INFO:journalctl@ceph.osd.4.smithi135.stdout:Mar 30 04:36:23 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-4[58761]: 2024-03-30T04:36:23.014+0000 7f5415400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T04:36:23.101 INFO:journalctl@ceph.osd.4.smithi135.stdout:Mar 30 04:36:23 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-4[58761]: 2024-03-30T04:36:23.014+0000 7f5415400700 -1 osd.4 69 *** Got signal Terminated *** 2024-03-30T04:36:23.101 INFO:journalctl@ceph.osd.4.smithi135.stdout:Mar 30 04:36:23 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-4[58761]: 2024-03-30T04:36:23.014+0000 7f5415400700 -1 osd.4 69 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T04:36:23.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:22 smithi123 ceph-mon[139369]: pgmap v139: 65 pgs: 65 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-30T04:36:24.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:23 smithi135 ceph-mon[106011]: osd.4 marked itself down and dead 2024-03-30T04:36:24.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:23 smithi123 ceph-mon[139369]: osd.4 marked itself down and dead 2024-03-30T04:36:24.822 INFO:journalctl@ceph.osd.4.smithi135.stdout:Mar 30 04:36:24 smithi135 bash[124973]: ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-4 2024-03-30T04:36:25.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:24 smithi135 ceph-mon[106011]: pgmap v140: 65 pgs: 65 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:36:25.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:24 smithi135 ceph-mon[106011]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-30T04:36:25.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:24 smithi135 ceph-mon[106011]: osdmap e70: 6 total, 5 up, 6 in 2024-03-30T04:36:25.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:24 smithi123 ceph-mon[139369]: pgmap v140: 65 pgs: 65 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:36:25.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:24 smithi123 ceph-mon[139369]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-30T04:36:25.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:24 smithi123 ceph-mon[139369]: osdmap e70: 6 total, 5 up, 6 in 2024-03-30T04:36:25.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:25 smithi135 ceph-mon[106011]: osdmap e71: 6 total, 5 up, 6 in 2024-03-30T04:36:26.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:25 smithi123 ceph-mon[139369]: osdmap e71: 6 total, 5 up, 6 in 2024-03-30T04:36:27.012 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:26 smithi135 ceph-mon[106011]: pgmap v143: 65 pgs: 2 active+undersized+degraded, 11 peering, 4 stale+active+clean, 3 active+undersized, 45 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 5/324 objects degraded (1.543%) 2024-03-30T04:36:27.012 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:26 smithi135 ceph-mon[106011]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-30T04:36:27.012 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:26 smithi135 ceph-mon[106011]: Health check failed: Degraded data redundancy: 5/324 objects degraded (1.543%), 2 pgs degraded (PG_DEGRADED) 2024-03-30T04:36:27.012 INFO:journalctl@ceph.osd.4.smithi135.stdout:Mar 30 04:36:26 smithi135 systemd[1]: ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4@osd.4.service: Succeeded. 2024-03-30T04:36:27.012 INFO:journalctl@ceph.osd.4.smithi135.stdout:Mar 30 04:36:26 smithi135 systemd[1]: Stopped Ceph osd.4 for 2e77a2be-ee4c-11ee-b647-cb9ed24678a4. 2024-03-30T04:36:27.013 INFO:journalctl@ceph.osd.4.smithi135.stdout:Mar 30 04:36:27 smithi135 systemd[1]: Starting Ceph osd.4 for 2e77a2be-ee4c-11ee-b647-cb9ed24678a4... 2024-03-30T04:36:27.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:26 smithi123 ceph-mon[139369]: pgmap v143: 65 pgs: 2 active+undersized+degraded, 11 peering, 4 stale+active+clean, 3 active+undersized, 45 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 5/324 objects degraded (1.543%) 2024-03-30T04:36:27.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:26 smithi123 ceph-mon[139369]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-30T04:36:27.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:26 smithi123 ceph-mon[139369]: Health check failed: Degraded data redundancy: 5/324 objects degraded (1.543%), 2 pgs degraded (PG_DEGRADED) 2024-03-30T04:36:28.601 INFO:journalctl@ceph.osd.4.smithi135.stdout:Mar 30 04:36:28 smithi135 bash[125240]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-30T04:36:28.601 INFO:journalctl@ceph.osd.4.smithi135.stdout:Mar 30 04:36:28 smithi135 bash[125240]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-30T04:36:28.601 INFO:journalctl@ceph.osd.4.smithi135.stdout:Mar 30 04:36:28 smithi135 bash[125240]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-30T04:36:28.601 INFO:journalctl@ceph.osd.4.smithi135.stdout:Mar 30 04:36:28 smithi135 bash[125240]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-30T04:36:28.601 INFO:journalctl@ceph.osd.4.smithi135.stdout:Mar 30 04:36:28 smithi135 bash[125240]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-4/block 2024-03-30T04:36:28.601 INFO:journalctl@ceph.osd.4.smithi135.stdout:Mar 30 04:36:28 smithi135 bash[125240]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-30T04:36:28.601 INFO:journalctl@ceph.osd.4.smithi135.stdout:Mar 30 04:36:28 smithi135 bash[125240]: --> ceph-volume raw activate successful for osd ID: 4 2024-03-30T04:36:29.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:28 smithi135 ceph-mon[106011]: pgmap v144: 65 pgs: 6 active+undersized+degraded, 11 peering, 2 stale+active+clean, 7 active+undersized, 39 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 18/324 objects degraded (5.556%) 2024-03-30T04:36:29.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:28 smithi123 ceph-mon[139369]: pgmap v144: 65 pgs: 6 active+undersized+degraded, 11 peering, 2 stale+active+clean, 7 active+undersized, 39 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 18/324 objects degraded (5.556%) 2024-03-30T04:36:30.100 INFO:journalctl@ceph.osd.4.smithi135.stdout:Mar 30 04:36:29 smithi135 ceph-osd[125533]: -- 172.21.15.135:0/456554803 <== mon.1 v2:172.21.15.135:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55848b8d91e0 con 0x55848b8a8c00 2024-03-30T04:36:30.100 INFO:journalctl@ceph.osd.4.smithi135.stdout:Mar 30 04:36:29 smithi135 bash[125495]: c7b2e84c0f81f26af593b35bd8c8e639d0f6a36e24ffa5f9385bf3ec4eae3c2a 2024-03-30T04:36:30.101 INFO:journalctl@ceph.osd.4.smithi135.stdout:Mar 30 04:36:29 smithi135 systemd[1]: Started Ceph osd.4 for 2e77a2be-ee4c-11ee-b647-cb9ed24678a4. 2024-03-30T04:36:30.600 INFO:journalctl@ceph.osd.4.smithi135.stdout:Mar 30 04:36:30 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-4[125510]: 2024-03-30T04:36:30.231+0000 7fa646938640 -1 Falling back to public interface 2024-03-30T04:36:30.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:30 smithi135 ceph-mon[106011]: pgmap v145: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 54/324 objects degraded (16.667%) 2024-03-30T04:36:30.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:30 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:36:30.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:30 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:36:30.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:30 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:36:30.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:30 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:36:30.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:30 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:36:30.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:30 smithi135 ceph-mon[106011]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-30T04:36:30.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:30 smithi123 ceph-mon[139369]: pgmap v145: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 54/324 objects degraded (16.667%) 2024-03-30T04:36:30.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:36:30.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:36:30.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:36:30.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:36:30.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:36:30.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:30 smithi123 ceph-mon[139369]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-30T04:36:32.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:32 smithi123 ceph-mon[139369]: pgmap v146: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 54/324 objects degraded (16.667%) 2024-03-30T04:36:32.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:32 smithi135 ceph-mon[106011]: pgmap v146: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 54/324 objects degraded (16.667%) 2024-03-30T04:36:34.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:34 smithi123 ceph-mon[139369]: pgmap v147: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 54/324 objects degraded (16.667%) 2024-03-30T04:36:34.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:34 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:36:34.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:34 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:36:34.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:34 smithi123 ceph-mon[139369]: Health check update: Degraded data redundancy: 54/324 objects degraded (16.667%), 16 pgs degraded (PG_DEGRADED) 2024-03-30T04:36:35.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:34 smithi135 ceph-mon[106011]: pgmap v147: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 54/324 objects degraded (16.667%) 2024-03-30T04:36:35.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:34 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:36:35.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:34 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:36:35.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:34 smithi135 ceph-mon[106011]: Health check update: Degraded data redundancy: 54/324 objects degraded (16.667%), 16 pgs degraded (PG_DEGRADED) 2024-03-30T04:36:35.994 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:35 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:36:35.994 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:35 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:36:36.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:35 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:36:36.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:35 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:36:36.324 INFO:journalctl@ceph.osd.4.smithi135.stdout:Mar 30 04:36:35 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-4[125510]: 2024-03-30T04:36:35.994+0000 7fa646938640 -1 osd.4 69 log_to_monitors true 2024-03-30T04:36:36.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.608+0000 7f8dabe00700 1 -- 172.21.15.123:0/2976069286 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8da4102e00 msgr2=0x7f8da41031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:36.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.608+0000 7f8dabe00700 1 --2- 172.21.15.123:0/2976069286 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8da4102e00 0x7f8da41031d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8d98004940 tx=0x7f8d98009e30 comp rx=0 tx=0).stop 2024-03-30T04:36:36.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.608+0000 7f8dabe00700 1 -- 172.21.15.123:0/2976069286 shutdown_connections 2024-03-30T04:36:36.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.608+0000 7f8dabe00700 1 --2- 172.21.15.123:0/2976069286 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8da41037a0 0x7f8da40fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:36.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.608+0000 7f8dabe00700 1 --2- 172.21.15.123:0/2976069286 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8da4102e00 0x7f8da41031d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:36.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.608+0000 7f8dabe00700 1 -- 172.21.15.123:0/2976069286 >> 172.21.15.123:0/2976069286 conn(0x7f8da40f89b0 msgr2=0x7f8da40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:36:36.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.608+0000 7f8dabe00700 1 -- 172.21.15.123:0/2976069286 shutdown_connections 2024-03-30T04:36:36.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.608+0000 7f8dabe00700 1 -- 172.21.15.123:0/2976069286 wait complete. 2024-03-30T04:36:36.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.609+0000 7f8dabe00700 1 Processor -- start 2024-03-30T04:36:36.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.609+0000 7f8dabe00700 1 -- start start 2024-03-30T04:36:36.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.610+0000 7f8dabe00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8da4102e00 0x7f8da4115d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:36:36.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.610+0000 7f8dabe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8da41037a0 0x7f8da4116250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:36:36.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.610+0000 7f8dabe00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da41168a0 con 0x7f8da41037a0 2024-03-30T04:36:36.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.610+0000 7f8dabe00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da410fc70 con 0x7f8da4102e00 2024-03-30T04:36:36.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.610+0000 7f8da8a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8da41037a0 0x7f8da4116250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:36:36.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.610+0000 7f8da8a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8da41037a0 0x7f8da4116250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46444/0 (socket says 172.21.15.123:46444) 2024-03-30T04:36:36.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.610+0000 7f8da9400700 1 --2- 172.21.15.123:0/3126642835 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8da4102e00 0x7f8da4115d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:36:36.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.610+0000 7f8da8a00700 1 -- 172.21.15.123:0/3126642835 learned_addr learned my addr 172.21.15.123:0/3126642835 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:36:36.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.611+0000 7f8da8a00700 1 -- 172.21.15.123:0/3126642835 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8da4102e00 msgr2=0x7f8da4115d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:36.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.611+0000 7f8da8a00700 1 --2- 172.21.15.123:0/3126642835 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8da4102e00 0x7f8da4115d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:36.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.611+0000 7f8da8a00700 1 -- 172.21.15.123:0/3126642835 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d98005040 con 0x7f8da41037a0 2024-03-30T04:36:36.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.611+0000 7f8da9400700 1 --2- 172.21.15.123:0/3126642835 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8da4102e00 0x7f8da4115d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:36:36.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.611+0000 7f8da8a00700 1 --2- 172.21.15.123:0/3126642835 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8da41037a0 0x7f8da4116250 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8d9000c890 tx=0x7f8d900044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:36:36.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.611+0000 7f8da1600700 1 -- 172.21.15.123:0/3126642835 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d9000a700 con 0x7f8da41037a0 2024-03-30T04:36:36.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.611+0000 7f8da1600700 1 -- 172.21.15.123:0/3126642835 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8d9000ad10 con 0x7f8da41037a0 2024-03-30T04:36:36.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.611+0000 7f8dabe00700 1 -- 172.21.15.123:0/3126642835 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8da410ff50 con 0x7f8da41037a0 2024-03-30T04:36:36.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.611+0000 7f8da1600700 1 -- 172.21.15.123:0/3126642835 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d90012e70 con 0x7f8da41037a0 2024-03-30T04:36:36.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.611+0000 7f8dabe00700 1 -- 172.21.15.123:0/3126642835 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8da4110510 con 0x7f8da41037a0 2024-03-30T04:36:36.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.613+0000 7f8dabe00700 1 -- 172.21.15.123:0/3126642835 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8da40fe550 con 0x7f8da41037a0 2024-03-30T04:36:36.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.613+0000 7f8da1600700 1 -- 172.21.15.123:0/3126642835 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f8d9000a860 con 0x7f8da41037a0 2024-03-30T04:36:36.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.615+0000 7f8da1600700 1 --2- 172.21.15.123:0/3126642835 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8d8c06cc30 0x7f8d8c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:36:36.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.615+0000 7f8da1600700 1 -- 172.21.15.123:0/3126642835 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f8d9008c000 con 0x7f8da41037a0 2024-03-30T04:36:36.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.615+0000 7f8da9400700 1 --2- 172.21.15.123:0/3126642835 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8d8c06cc30 0x7f8d8c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:36:36.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.615+0000 7f8da9400700 1 --2- 172.21.15.123:0/3126642835 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8d8c06cc30 0x7f8d8c06f0e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8d9800b3b0 tx=0x7f8d98009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:36:36.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.619+0000 7f8da1600700 1 -- 172.21.15.123:0/3126642835 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8d90059b70 con 0x7f8da41037a0 2024-03-30T04:36:36.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.838+0000 7f8dabe00700 1 -- 172.21.15.123:0/3126642835 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8da4109e30 con 0x7f8d8c06cc30 2024-03-30T04:36:36.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.840+0000 7f8da1600700 1 -- 172.21.15.123:0/3126642835 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f8da4109e30 con 0x7f8d8c06cc30 2024-03-30T04:36:36.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.842+0000 7f8dabe00700 1 -- 172.21.15.123:0/3126642835 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8d8c06cc30 msgr2=0x7f8d8c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:36.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.842+0000 7f8dabe00700 1 --2- 172.21.15.123:0/3126642835 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8d8c06cc30 0x7f8d8c06f0e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8d9800b3b0 tx=0x7f8d98009e30 comp rx=0 tx=0).stop 2024-03-30T04:36:36.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.842+0000 7f8dabe00700 1 -- 172.21.15.123:0/3126642835 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8da41037a0 msgr2=0x7f8da4116250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:36.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.842+0000 7f8dabe00700 1 --2- 172.21.15.123:0/3126642835 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8da41037a0 0x7f8da4116250 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8d9000c890 tx=0x7f8d900044c0 comp rx=0 tx=0).stop 2024-03-30T04:36:36.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.842+0000 7f8dabe00700 1 -- 172.21.15.123:0/3126642835 shutdown_connections 2024-03-30T04:36:36.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.842+0000 7f8dabe00700 1 --2- 172.21.15.123:0/3126642835 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8d8c06cc30 0x7f8d8c06f0e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:36.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.842+0000 7f8dabe00700 1 --2- 172.21.15.123:0/3126642835 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8da4102e00 0x7f8da4115d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:36.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.842+0000 7f8dabe00700 1 --2- 172.21.15.123:0/3126642835 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8da41037a0 0x7f8da4116250 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:36.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.842+0000 7f8dabe00700 1 -- 172.21.15.123:0/3126642835 >> 172.21.15.123:0/3126642835 conn(0x7f8da40f89b0 msgr2=0x7f8da40704f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:36:36.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.843+0000 7f8dabe00700 1 -- 172.21.15.123:0/3126642835 shutdown_connections 2024-03-30T04:36:36.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.843+0000 7f8dabe00700 1 -- 172.21.15.123:0/3126642835 wait complete. 2024-03-30T04:36:36.857 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:36:36.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.981+0000 7f913d400700 1 -- 172.21.15.123:0/3037733708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91381037c0 msgr2=0x7f91380fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:36.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.981+0000 7f913d400700 1 --2- 172.21.15.123:0/3037733708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91381037c0 0x7f91380fce30 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f9120004940 tx=0x7f9120009e30 comp rx=0 tx=0).stop 2024-03-30T04:36:36.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.981+0000 7f913d400700 1 -- 172.21.15.123:0/3037733708 shutdown_connections 2024-03-30T04:36:36.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.981+0000 7f913d400700 1 --2- 172.21.15.123:0/3037733708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91381037c0 0x7f91380fce30 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:36.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.981+0000 7f913d400700 1 --2- 172.21.15.123:0/3037733708 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9138102e20 0x7f91381031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:36.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.981+0000 7f913d400700 1 -- 172.21.15.123:0/3037733708 >> 172.21.15.123:0/3037733708 conn(0x7f91380f89d0 msgr2=0x7f91380fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:36:36.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.981+0000 7f913d400700 1 -- 172.21.15.123:0/3037733708 shutdown_connections 2024-03-30T04:36:36.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.981+0000 7f913d400700 1 -- 172.21.15.123:0/3037733708 wait complete. 2024-03-30T04:36:36.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.982+0000 7f913d400700 1 Processor -- start 2024-03-30T04:36:36.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.982+0000 7f913d400700 1 -- start start 2024-03-30T04:36:36.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.982+0000 7f913d400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9138102e20 0x7f9138115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:36:36.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.982+0000 7f913d400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91381037c0 0x7f9138116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:36:36.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.982+0000 7f913d400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9138116860 con 0x7f91381037c0 2024-03-30T04:36:36.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.982+0000 7f913d400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f913810fc30 con 0x7f9138102e20 2024-03-30T04:36:36.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.983+0000 7f912fe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91381037c0 0x7f9138116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:36:36.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.983+0000 7f912fe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91381037c0 0x7f9138116210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46466/0 (socket says 172.21.15.123:46466) 2024-03-30T04:36:36.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.983+0000 7f912fe00700 1 -- 172.21.15.123:0/1808123580 learned_addr learned my addr 172.21.15.123:0/1808123580 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:36:36.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.983+0000 7f9136800700 1 --2- 172.21.15.123:0/1808123580 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9138102e20 0x7f9138115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:36:36.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.983+0000 7f912fe00700 1 -- 172.21.15.123:0/1808123580 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9138102e20 msgr2=0x7f9138115cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:36.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.983+0000 7f912fe00700 1 --2- 172.21.15.123:0/1808123580 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9138102e20 0x7f9138115cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:36.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.983+0000 7f912fe00700 1 -- 172.21.15.123:0/1808123580 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9120005040 con 0x7f91381037c0 2024-03-30T04:36:36.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.983+0000 7f912fe00700 1 --2- 172.21.15.123:0/1808123580 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91381037c0 0x7f9138116210 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f9120009390 tx=0x7f9120009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:36:36.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.983+0000 7f912ea00700 1 -- 172.21.15.123:0/1808123580 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f912000e040 con 0x7f91381037c0 2024-03-30T04:36:36.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.983+0000 7f913d400700 1 -- 172.21.15.123:0/1808123580 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f913810feb0 con 0x7f91381037c0 2024-03-30T04:36:36.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.983+0000 7f912ea00700 1 -- 172.21.15.123:0/1808123580 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9120013420 con 0x7f91381037c0 2024-03-30T04:36:36.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.984+0000 7f912ea00700 1 -- 172.21.15.123:0/1808123580 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f912001c630 con 0x7f91381037c0 2024-03-30T04:36:36.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.984+0000 7f913d400700 1 -- 172.21.15.123:0/1808123580 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9138110470 con 0x7f91381037c0 2024-03-30T04:36:36.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.985+0000 7f913d400700 1 -- 172.21.15.123:0/1808123580 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91380fe570 con 0x7f91381037c0 2024-03-30T04:36:36.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.986+0000 7f912ea00700 1 -- 172.21.15.123:0/1808123580 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f9120013a00 con 0x7f91381037c0 2024-03-30T04:36:36.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.986+0000 7f912ea00700 1 --2- 172.21.15.123:0/1808123580 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f911c06cc30 0x7f911c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:36:36.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.987+0000 7f912ea00700 1 -- 172.21.15.123:0/1808123580 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f912008d160 con 0x7f91381037c0 2024-03-30T04:36:36.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.987+0000 7f9136800700 1 --2- 172.21.15.123:0/1808123580 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f911c06cc30 0x7f911c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:36:36.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.987+0000 7f9136800700 1 --2- 172.21.15.123:0/1808123580 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f911c06cc30 0x7f911c06f0e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9128004f40 tx=0x7f912800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:36:36.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:36.991+0000 7f912ea00700 1 -- 172.21.15.123:0/1808123580 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f912005ac30 con 0x7f91381037c0 2024-03-30T04:36:37.017 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:36 smithi135 ceph-mon[106011]: pgmap v148: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 2 op/s; 54/324 objects degraded (16.667%) 2024-03-30T04:36:37.017 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:36 smithi135 ceph-mon[106011]: from='osd.4 [v2:172.21.15.135:6808/4133828133,v1:172.21.15.135:6809/4133828133]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-30T04:36:37.017 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:36 smithi135 ceph-mon[106011]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-30T04:36:37.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.209+0000 7f913d400700 1 -- 172.21.15.123:0/1808123580 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9138109dc0 con 0x7f911c06cc30 2024-03-30T04:36:37.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.210+0000 7f912ea00700 1 -- 172.21.15.123:0/1808123580 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f9138109dc0 con 0x7f911c06cc30 2024-03-30T04:36:37.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.212+0000 7f913d400700 1 -- 172.21.15.123:0/1808123580 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f911c06cc30 msgr2=0x7f911c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:37.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.212+0000 7f913d400700 1 --2- 172.21.15.123:0/1808123580 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f911c06cc30 0x7f911c06f0e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9128004f40 tx=0x7f912800b040 comp rx=0 tx=0).stop 2024-03-30T04:36:37.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.213+0000 7f913d400700 1 -- 172.21.15.123:0/1808123580 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91381037c0 msgr2=0x7f9138116210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:37.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.213+0000 7f913d400700 1 --2- 172.21.15.123:0/1808123580 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91381037c0 0x7f9138116210 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f9120009390 tx=0x7f9120009e30 comp rx=0 tx=0).stop 2024-03-30T04:36:37.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.213+0000 7f913d400700 1 -- 172.21.15.123:0/1808123580 shutdown_connections 2024-03-30T04:36:37.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.213+0000 7f913d400700 1 --2- 172.21.15.123:0/1808123580 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f911c06cc30 0x7f911c06f0e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:37.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.213+0000 7f913d400700 1 --2- 172.21.15.123:0/1808123580 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9138102e20 0x7f9138115cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:37.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.213+0000 7f913d400700 1 --2- 172.21.15.123:0/1808123580 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91381037c0 0x7f9138116210 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:37.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.213+0000 7f913d400700 1 -- 172.21.15.123:0/1808123580 >> 172.21.15.123:0/1808123580 conn(0x7f91380f89d0 msgr2=0x7f91380706a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:36:37.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.213+0000 7f913d400700 1 -- 172.21.15.123:0/1808123580 shutdown_connections 2024-03-30T04:36:37.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.213+0000 7f913d400700 1 -- 172.21.15.123:0/1808123580 wait complete. 2024-03-30T04:36:37.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:36 smithi123 ceph-mon[139369]: pgmap v148: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 2 op/s; 54/324 objects degraded (16.667%) 2024-03-30T04:36:37.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:36 smithi123 ceph-mon[139369]: from='osd.4 [v2:172.21.15.135:6808/4133828133,v1:172.21.15.135:6809/4133828133]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-30T04:36:37.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:36 smithi123 ceph-mon[139369]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-30T04:36:37.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.346+0000 7f5f0d600700 1 -- 172.21.15.123:0/3169162622 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f080708a0 msgr2=0x7f5f08070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:37.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.346+0000 7f5f0d600700 1 --2- 172.21.15.123:0/3169162622 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f080708a0 0x7f5f08070d10 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5ef4004970 tx=0x7f5ef4009e30 comp rx=0 tx=0).stop 2024-03-30T04:36:37.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.347+0000 7f5f0d600700 1 -- 172.21.15.123:0/3169162622 shutdown_connections 2024-03-30T04:36:37.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.347+0000 7f5f0d600700 1 --2- 172.21.15.123:0/3169162622 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f080708a0 0x7f5f08070d10 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:37.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.347+0000 7f5f0d600700 1 --2- 172.21.15.123:0/3169162622 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5f08071f90 0x7f5f08070360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:37.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.347+0000 7f5f0d600700 1 -- 172.21.15.123:0/3169162622 >> 172.21.15.123:0/3169162622 conn(0x7f5f080f89b0 msgr2=0x7f5f080fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:36:37.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.347+0000 7f5f0d600700 1 -- 172.21.15.123:0/3169162622 shutdown_connections 2024-03-30T04:36:37.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.347+0000 7f5f0d600700 1 -- 172.21.15.123:0/3169162622 wait complete. 2024-03-30T04:36:37.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.347+0000 7f5f0d600700 1 Processor -- start 2024-03-30T04:36:37.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.348+0000 7f5f0d600700 1 -- start start 2024-03-30T04:36:37.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.348+0000 7f5f0d600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5f080708a0 0x7f5f0811a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:36:37.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.348+0000 7f5f0d600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f08071f90 0x7f5f0811a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:36:37.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.348+0000 7f5f0d600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0811ac40 con 0x7f5f08071f90 2024-03-30T04:36:37.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.348+0000 7f5f0d600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f08114010 con 0x7f5f080708a0 2024-03-30T04:36:37.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.348+0000 7f5f06000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f08071f90 0x7f5f0811a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:36:37.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.348+0000 7f5f06a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5f080708a0 0x7f5f0811a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:36:37.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.348+0000 7f5f06000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f08071f90 0x7f5f0811a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46496/0 (socket says 172.21.15.123:46496) 2024-03-30T04:36:37.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.348+0000 7f5f06000700 1 -- 172.21.15.123:0/3855869268 learned_addr learned my addr 172.21.15.123:0/3855869268 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:36:37.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.349+0000 7f5f06000700 1 -- 172.21.15.123:0/3855869268 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5f080708a0 msgr2=0x7f5f0811a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:37.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.349+0000 7f5f06000700 1 --2- 172.21.15.123:0/3855869268 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5f080708a0 0x7f5f0811a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:37.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.349+0000 7f5f06000700 1 -- 172.21.15.123:0/3855869268 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ef4005040 con 0x7f5f08071f90 2024-03-30T04:36:37.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.349+0000 7f5f06a00700 1 --2- 172.21.15.123:0/3855869268 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5f080708a0 0x7f5f0811a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:36:37.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.349+0000 7f5f06000700 1 --2- 172.21.15.123:0/3855869268 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f08071f90 0x7f5f0811a5f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5ef4005010 tx=0x7f5ef4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:36:37.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.349+0000 7f5efea00700 1 -- 172.21.15.123:0/3855869268 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ef400e040 con 0x7f5f08071f90 2024-03-30T04:36:37.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.349+0000 7f5efea00700 1 -- 172.21.15.123:0/3855869268 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5ef4004290 con 0x7f5f08071f90 2024-03-30T04:36:37.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.349+0000 7f5f0d600700 1 -- 172.21.15.123:0/3855869268 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f08114290 con 0x7f5f08071f90 2024-03-30T04:36:37.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.349+0000 7f5efea00700 1 -- 172.21.15.123:0/3855869268 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ef4013d40 con 0x7f5f08071f90 2024-03-30T04:36:37.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.349+0000 7f5f0d600700 1 -- 172.21.15.123:0/3855869268 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f081147d0 con 0x7f5f08071f90 2024-03-30T04:36:37.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.350+0000 7f5f0d600700 1 -- 172.21.15.123:0/3855869268 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f08071aa0 con 0x7f5f08071f90 2024-03-30T04:36:37.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.351+0000 7f5efea00700 1 -- 172.21.15.123:0/3855869268 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f5ef4013520 con 0x7f5f08071f90 2024-03-30T04:36:37.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.352+0000 7f5efea00700 1 --2- 172.21.15.123:0/3855869268 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5eec06cc30 0x7f5eec06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:36:37.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.352+0000 7f5efea00700 1 -- 172.21.15.123:0/3855869268 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f5ef408d200 con 0x7f5f08071f90 2024-03-30T04:36:37.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.352+0000 7f5f06a00700 1 --2- 172.21.15.123:0/3855869268 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5eec06cc30 0x7f5eec06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:36:37.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.353+0000 7f5f06a00700 1 --2- 172.21.15.123:0/3855869268 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5eec06cc30 0x7f5eec06f0e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5ef8009b80 tx=0x7f5ef800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:36:37.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.357+0000 7f5efea00700 1 -- 172.21.15.123:0/3855869268 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5ef405ad80 con 0x7f5f08071f90 2024-03-30T04:36:37.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.571+0000 7f5f0d600700 1 -- 172.21.15.123:0/3855869268 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f5f08110450 con 0x7f5eec06cc30 2024-03-30T04:36:37.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.581+0000 7f5efea00700 1 -- 172.21.15.123:0/3855869268 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f5f08110450 con 0x7f5eec06cc30 2024-03-30T04:36:37.579 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:36:37.579 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (5m) 78s ago 18m 14.7M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:36:37.579 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (3m) 78s ago 18m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T04:36:37.580 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (2m) 3s ago 17m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T04:36:37.580 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (5m) 78s ago 18m 74.1M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T04:36:37.580 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (11m) 78s ago 11m 109M - 17.2.7-797-g5c024335 f77085f6aab7 2df066fba7a6 2024-03-30T04:36:37.580 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (12m) 78s ago 12m 58.0M - 17.2.7-797-g5c024335 f77085f6aab7 1293b8520db1 2024-03-30T04:36:37.580 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (11m) 3s ago 11m 67.6M - 17.2.7-797-g5c024335 f77085f6aab7 0dc41326818c 2024-03-30T04:36:37.580 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (12m) 3s ago 11m 119M - 17.2.7-797-g5c024335 f77085f6aab7 aef8a0cfc553 2024-03-30T04:36:37.580 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (8m) 78s ago 19m 502M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:36:37.580 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (7m) 3s ago 17m 449M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:36:37.580 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (4m) 78s ago 20m 45.6M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T04:36:37.580 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (3m) 3s ago 17m 38.8M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T04:36:37.580 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (6m) 78s ago 18m 7583k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:36:37.580 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (6m) 3s ago 17m 7708k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:36:37.580 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (2m) 78s ago 16m 99.5M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T04:36:37.581 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (2m) 78s ago 15m 90.4M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T04:36:37.581 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (85s) 78s ago 15m 10.0M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T04:36:37.581 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (45s) 3s ago 14m 94.5M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T04:36:37.581 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (8s) 3s ago 14m 11.0M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T04:36:37.581 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (13m) 3s ago 13m 219M 2642M 17.2.7-797-g5c024335 f77085f6aab7 1fd3185f5484 2024-03-30T04:36:37.581 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (6m) 78s ago 17m 52.5M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:36:37.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.583+0000 7f5f0d600700 1 -- 172.21.15.123:0/3855869268 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5eec06cc30 msgr2=0x7f5eec06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:37.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.583+0000 7f5f0d600700 1 --2- 172.21.15.123:0/3855869268 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5eec06cc30 0x7f5eec06f0e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5ef8009b80 tx=0x7f5ef800b040 comp rx=0 tx=0).stop 2024-03-30T04:36:37.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.583+0000 7f5f0d600700 1 -- 172.21.15.123:0/3855869268 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f08071f90 msgr2=0x7f5f0811a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:37.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.583+0000 7f5f0d600700 1 --2- 172.21.15.123:0/3855869268 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f08071f90 0x7f5f0811a5f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5ef4005010 tx=0x7f5ef4009e30 comp rx=0 tx=0).stop 2024-03-30T04:36:37.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.583+0000 7f5f0d600700 1 -- 172.21.15.123:0/3855869268 shutdown_connections 2024-03-30T04:36:37.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.583+0000 7f5f0d600700 1 --2- 172.21.15.123:0/3855869268 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5eec06cc30 0x7f5eec06f0e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:37.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.583+0000 7f5f0d600700 1 --2- 172.21.15.123:0/3855869268 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5f080708a0 0x7f5f0811a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:37.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.583+0000 7f5f0d600700 1 --2- 172.21.15.123:0/3855869268 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f08071f90 0x7f5f0811a5f0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:37.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.583+0000 7f5f0d600700 1 -- 172.21.15.123:0/3855869268 >> 172.21.15.123:0/3855869268 conn(0x7f5f080f89b0 msgr2=0x7f5f080f9b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:36:37.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.583+0000 7f5f0d600700 1 -- 172.21.15.123:0/3855869268 shutdown_connections 2024-03-30T04:36:37.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.583+0000 7f5f0d600700 1 -- 172.21.15.123:0/3855869268 wait complete. 2024-03-30T04:36:37.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.715+0000 7f4108200700 1 -- 172.21.15.123:0/1767271098 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4100102e00 msgr2=0x7f41001031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:37.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.715+0000 7f4108200700 1 --2- 172.21.15.123:0/1767271098 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4100102e00 0x7f41001031d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f40f4004940 tx=0x7f40f4009e30 comp rx=0 tx=0).stop 2024-03-30T04:36:37.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.715+0000 7f4108200700 1 -- 172.21.15.123:0/1767271098 shutdown_connections 2024-03-30T04:36:37.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.715+0000 7f4108200700 1 --2- 172.21.15.123:0/1767271098 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f41001037a0 0x7f41000fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:37.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.715+0000 7f4108200700 1 --2- 172.21.15.123:0/1767271098 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4100102e00 0x7f41001031d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:37.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.715+0000 7f4108200700 1 -- 172.21.15.123:0/1767271098 >> 172.21.15.123:0/1767271098 conn(0x7f41000f89b0 msgr2=0x7f41000fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:36:37.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.715+0000 7f4108200700 1 -- 172.21.15.123:0/1767271098 shutdown_connections 2024-03-30T04:36:37.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.715+0000 7f4108200700 1 -- 172.21.15.123:0/1767271098 wait complete. 2024-03-30T04:36:37.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.716+0000 7f4108200700 1 Processor -- start 2024-03-30T04:36:37.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.716+0000 7f4108200700 1 -- start start 2024-03-30T04:36:37.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.716+0000 7f4108200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4100102e00 0x7f4100115d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:36:37.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.716+0000 7f4108200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f41001037a0 0x7f4100116250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:36:37.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.716+0000 7f4108200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41001168a0 con 0x7f4100102e00 2024-03-30T04:36:37.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.716+0000 7f4108200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f410010fc70 con 0x7f41001037a0 2024-03-30T04:36:37.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.716+0000 7f4105800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4100102e00 0x7f4100115d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:36:37.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.717+0000 7f4105800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4100102e00 0x7f4100115d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46520/0 (socket says 172.21.15.123:46520) 2024-03-30T04:36:37.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.717+0000 7f4105800700 1 -- 172.21.15.123:0/2311079629 learned_addr learned my addr 172.21.15.123:0/2311079629 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:36:37.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.717+0000 7f4104e00700 1 --2- 172.21.15.123:0/2311079629 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f41001037a0 0x7f4100116250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:36:37.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.717+0000 7f4105800700 1 -- 172.21.15.123:0/2311079629 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f41001037a0 msgr2=0x7f4100116250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:37.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.717+0000 7f4105800700 1 --2- 172.21.15.123:0/2311079629 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f41001037a0 0x7f4100116250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:37.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.717+0000 7f4105800700 1 -- 172.21.15.123:0/2311079629 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f4005040 con 0x7f4100102e00 2024-03-30T04:36:37.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.717+0000 7f4105800700 1 --2- 172.21.15.123:0/2311079629 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4100102e00 0x7f4100115d10 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f40f4009390 tx=0x7f40f4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:36:37.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.717+0000 7f40fd600700 1 -- 172.21.15.123:0/2311079629 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40f400e040 con 0x7f4100102e00 2024-03-30T04:36:37.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.717+0000 7f40fd600700 1 -- 172.21.15.123:0/2311079629 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f40f4013420 con 0x7f4100102e00 2024-03-30T04:36:37.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.717+0000 7f40fd600700 1 -- 172.21.15.123:0/2311079629 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40f401c630 con 0x7f4100102e00 2024-03-30T04:36:37.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.717+0000 7f4108200700 1 -- 172.21.15.123:0/2311079629 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f410010fef0 con 0x7f4100102e00 2024-03-30T04:36:37.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.718+0000 7f4108200700 1 -- 172.21.15.123:0/2311079629 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41001104b0 con 0x7f4100102e00 2024-03-30T04:36:37.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.719+0000 7f4108200700 1 -- 172.21.15.123:0/2311079629 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41000fe550 con 0x7f4100102e00 2024-03-30T04:36:37.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.719+0000 7f40fd600700 1 -- 172.21.15.123:0/2311079629 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f40f40139e0 con 0x7f4100102e00 2024-03-30T04:36:37.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.720+0000 7f40fd600700 1 --2- 172.21.15.123:0/2311079629 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f40e806cc30 0x7f40e806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:36:37.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.720+0000 7f40fd600700 1 -- 172.21.15.123:0/2311079629 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f40f408d350 con 0x7f4100102e00 2024-03-30T04:36:37.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.721+0000 7f4104e00700 1 --2- 172.21.15.123:0/2311079629 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f40e806cc30 0x7f40e806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:36:37.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.721+0000 7f4104e00700 1 --2- 172.21.15.123:0/2311079629 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f40e806cc30 0x7f40e806f0e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f41001115b0 tx=0x7f40ec00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:36:37.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:37.725+0000 7f40fd600700 1 -- 172.21.15.123:0/2311079629 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f40f405ae20 con 0x7f4100102e00 2024-03-30T04:36:38.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.016+0000 7f4108200700 1 -- 172.21.15.123:0/2311079629 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f410004bed0 con 0x7f4100102e00 2024-03-30T04:36:38.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.016+0000 7f40fd600700 1 -- 172.21.15.123:0/2311079629 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f40f405a9b0 con 0x7f4100102e00 2024-03-30T04:36:38.015 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:36:38.015 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:36:38.015 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:36:38.015 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:36:38.015 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:36:38.015 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:36:38.015 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:36:38.015 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:36:38.016 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:36:38.016 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 4 2024-03-30T04:36:38.016 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:36:38.016 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:36:38.016 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 4 2024-03-30T04:36:38.016 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:36:38.016 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:36:38.016 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 5, 2024-03-30T04:36:38.016 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 8 2024-03-30T04:36:38.016 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:36:38.016 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:36:38.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.018+0000 7f4108200700 1 -- 172.21.15.123:0/2311079629 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f40e806cc30 msgr2=0x7f40e806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:38.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.018+0000 7f4108200700 1 --2- 172.21.15.123:0/2311079629 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f40e806cc30 0x7f40e806f0e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f41001115b0 tx=0x7f40ec00b040 comp rx=0 tx=0).stop 2024-03-30T04:36:38.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.018+0000 7f4108200700 1 -- 172.21.15.123:0/2311079629 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4100102e00 msgr2=0x7f4100115d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:38.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.018+0000 7f4108200700 1 --2- 172.21.15.123:0/2311079629 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4100102e00 0x7f4100115d10 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f40f4009390 tx=0x7f40f4009e30 comp rx=0 tx=0).stop 2024-03-30T04:36:38.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.019+0000 7f4108200700 1 -- 172.21.15.123:0/2311079629 shutdown_connections 2024-03-30T04:36:38.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.019+0000 7f4108200700 1 --2- 172.21.15.123:0/2311079629 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f40e806cc30 0x7f40e806f0e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:38.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.019+0000 7f4108200700 1 --2- 172.21.15.123:0/2311079629 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4100102e00 0x7f4100115d10 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:38.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.019+0000 7f4108200700 1 --2- 172.21.15.123:0/2311079629 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f41001037a0 0x7f4100116250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:38.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.019+0000 7f4108200700 1 -- 172.21.15.123:0/2311079629 >> 172.21.15.123:0/2311079629 conn(0x7f41000f89b0 msgr2=0x7f41000704f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:36:38.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.019+0000 7f4108200700 1 -- 172.21.15.123:0/2311079629 shutdown_connections 2024-03-30T04:36:38.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.019+0000 7f4108200700 1 -- 172.21.15.123:0/2311079629 wait complete. 2024-03-30T04:36:38.101 INFO:journalctl@ceph.osd.4.smithi135.stdout:Mar 30 04:36:37 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-4[125510]: 2024-03-30T04:36:37.827+0000 7fa630c00700 -1 osd.4 69 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T04:36:38.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:37 smithi135 ceph-mon[106011]: from='client.34346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:36:38.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:37 smithi135 ceph-mon[106011]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-30T04:36:38.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:37 smithi135 ceph-mon[106011]: osdmap e72: 6 total, 5 up, 6 in 2024-03-30T04:36:38.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:37 smithi135 ceph-mon[106011]: from='osd.4 [v2:172.21.15.135:6808/4133828133,v1:172.21.15.135:6809/4133828133]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-03-30T04:36:38.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:37 smithi135 ceph-mon[106011]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-03-30T04:36:38.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:37 smithi135 ceph-mon[106011]: from='osd.4 ' entity='osd.4' 2024-03-30T04:36:38.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.152+0000 7f0d55000700 1 -- 172.21.15.123:0/4001787713 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d501037c0 msgr2=0x7f0d500fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:38.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.152+0000 7f0d55000700 1 --2- 172.21.15.123:0/4001787713 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d501037c0 0x7f0d500fce30 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c004940 tx=0x7f0d3c009e30 comp rx=0 tx=0).stop 2024-03-30T04:36:38.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.152+0000 7f0d55000700 1 -- 172.21.15.123:0/4001787713 shutdown_connections 2024-03-30T04:36:38.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.152+0000 7f0d55000700 1 --2- 172.21.15.123:0/4001787713 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d501037c0 0x7f0d500fce30 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:38.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.152+0000 7f0d55000700 1 --2- 172.21.15.123:0/4001787713 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0d50102e20 0x7f0d501031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:38.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.152+0000 7f0d55000700 1 -- 172.21.15.123:0/4001787713 >> 172.21.15.123:0/4001787713 conn(0x7f0d500f89d0 msgr2=0x7f0d500fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:36:38.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.153+0000 7f0d55000700 1 -- 172.21.15.123:0/4001787713 shutdown_connections 2024-03-30T04:36:38.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.153+0000 7f0d55000700 1 -- 172.21.15.123:0/4001787713 wait complete. 2024-03-30T04:36:38.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.153+0000 7f0d55000700 1 Processor -- start 2024-03-30T04:36:38.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.153+0000 7f0d55000700 1 -- start start 2024-03-30T04:36:38.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.154+0000 7f0d55000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d50102e20 0x7f0d50113ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:36:38.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.154+0000 7f0d55000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0d501037c0 0x7f0d50114010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:36:38.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.154+0000 7f0d55000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d50114660 con 0x7f0d50102e20 2024-03-30T04:36:38.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.154+0000 7f0d55000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5010da50 con 0x7f0d501037c0 2024-03-30T04:36:38.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.154+0000 7f0d4de00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d50102e20 0x7f0d50113ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:36:38.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.154+0000 7f0d4de00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d50102e20 0x7f0d50113ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46526/0 (socket says 172.21.15.123:46526) 2024-03-30T04:36:38.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.154+0000 7f0d4de00700 1 -- 172.21.15.123:0/317408841 learned_addr learned my addr 172.21.15.123:0/317408841 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:36:38.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.154+0000 7f0d4d400700 1 --2- 172.21.15.123:0/317408841 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0d501037c0 0x7f0d50114010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:36:38.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.155+0000 7f0d4de00700 1 -- 172.21.15.123:0/317408841 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0d501037c0 msgr2=0x7f0d50114010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:38.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.155+0000 7f0d4de00700 1 --2- 172.21.15.123:0/317408841 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0d501037c0 0x7f0d50114010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:38.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.155+0000 7f0d4de00700 1 -- 172.21.15.123:0/317408841 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d3c005040 con 0x7f0d50102e20 2024-03-30T04:36:38.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.155+0000 7f0d4de00700 1 --2- 172.21.15.123:0/317408841 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d50102e20 0x7f0d50113ad0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0d40002a50 tx=0x7f0d400046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:36:38.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.155+0000 7f0d46000700 1 -- 172.21.15.123:0/317408841 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d40007cc0 con 0x7f0d50102e20 2024-03-30T04:36:38.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.155+0000 7f0d46000700 1 -- 172.21.15.123:0/317408841 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0d4000f040 con 0x7f0d50102e20 2024-03-30T04:36:38.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.155+0000 7f0d46000700 1 -- 172.21.15.123:0/317408841 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d4000d9c0 con 0x7f0d50102e20 2024-03-30T04:36:38.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.155+0000 7f0d55000700 1 -- 172.21.15.123:0/317408841 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d5010dd30 con 0x7f0d50102e20 2024-03-30T04:36:38.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.155+0000 7f0d55000700 1 -- 172.21.15.123:0/317408841 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d5010e2c0 con 0x7f0d50102e20 2024-03-30T04:36:38.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.156+0000 7f0d55000700 1 -- 172.21.15.123:0/317408841 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d500fe570 con 0x7f0d50102e20 2024-03-30T04:36:38.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.157+0000 7f0d46000700 1 -- 172.21.15.123:0/317408841 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f0d4000db20 con 0x7f0d50102e20 2024-03-30T04:36:38.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.158+0000 7f0d46000700 1 --2- 172.21.15.123:0/317408841 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0d3406cc30 0x7f0d3406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:36:38.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.158+0000 7f0d46000700 1 -- 172.21.15.123:0/317408841 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f0d4008bc00 con 0x7f0d50102e20 2024-03-30T04:36:38.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.158+0000 7f0d4d400700 1 --2- 172.21.15.123:0/317408841 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0d3406cc30 0x7f0d3406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:36:38.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.159+0000 7f0d4d400700 1 --2- 172.21.15.123:0/317408841 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0d3406cc30 0x7f0d3406f0e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0d5010ed30 tx=0x7f0d3c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:36:38.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.162+0000 7f0d46000700 1 -- 172.21.15.123:0/317408841 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0d40091020 con 0x7f0d50102e20 2024-03-30T04:36:38.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:37 smithi123 ceph-mon[139369]: from='client.34346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:36:38.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:37 smithi123 ceph-mon[139369]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-30T04:36:38.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:37 smithi123 ceph-mon[139369]: osdmap e72: 6 total, 5 up, 6 in 2024-03-30T04:36:38.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:37 smithi123 ceph-mon[139369]: from='osd.4 [v2:172.21.15.135:6808/4133828133,v1:172.21.15.135:6809/4133828133]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-03-30T04:36:38.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:37 smithi123 ceph-mon[139369]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-03-30T04:36:38.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:37 smithi123 ceph-mon[139369]: from='osd.4 ' entity='osd.4' 2024-03-30T04:36:38.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.409+0000 7f0d55000700 1 -- 172.21.15.123:0/317408841 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0d50109e10 con 0x7f0d50102e20 2024-03-30T04:36:38.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.409+0000 7f0d46000700 1 -- 172.21.15.123:0/317408841 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 23 v23) v1 ==== 76+0+1876 (secure 0 0 0) 0x7f0d40059770 con 0x7f0d50102e20 2024-03-30T04:36:38.408 INFO:teuthology.orchestra.run.smithi123.stdout:e23 2024-03-30T04:36:38.408 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:36:38.408 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:36:38.408 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:36:38.408 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:36:38.408 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:36:38.408 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:36:38.408 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 23 2024-03-30T04:36:38.408 INFO:teuthology.orchestra.run.smithi123.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-30T04:36:38.409 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:36:38.409 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:32:48.448604+0000 2024-03-30T04:36:38.409 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:36:38.409 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:36:38.409 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:36:38.409 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:36:38.409 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:36:38.409 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T04:36:38.409 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:36:38.409 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:36:38.409 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 46 2024-03-30T04:36:38.409 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:36:38.409 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T04:36:38.409 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T04:36:38.409 INFO:teuthology.orchestra.run.smithi123.stdout:up {0=14488,1=24309} 2024-03-30T04:36:38.410 INFO:teuthology.orchestra.run.smithi123.stdout:failed 2024-03-30T04:36:38.410 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:36:38.410 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:36:38.410 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:36:38.410 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:36:38.410 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:36:38.410 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:36:38.410 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T04:36:38.410 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T04:36:38.410 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{0:14488} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.123:6828/3037849622,v1:172.21.15.123:6829/3037849622] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:36:38.410 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{0:14500} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/2433675995,v1:172.21.15.123:6827/2433675995] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:36:38.410 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{1:24309} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.135:6824/4050541062,v1:172.21.15.135:6825/4050541062] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:36:38.410 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{1:24319} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.135:6826/1096986122,v1:172.21.15.135:6827/1096986122] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:36:38.411 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:36:38.411 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:36:38.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.411+0000 7f0d55000700 1 -- 172.21.15.123:0/317408841 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0d3406cc30 msgr2=0x7f0d3406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:38.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.411+0000 7f0d55000700 1 --2- 172.21.15.123:0/317408841 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0d3406cc30 0x7f0d3406f0e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0d5010ed30 tx=0x7f0d3c009e30 comp rx=0 tx=0).stop 2024-03-30T04:36:38.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.411+0000 7f0d55000700 1 -- 172.21.15.123:0/317408841 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d50102e20 msgr2=0x7f0d50113ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:38.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.411+0000 7f0d55000700 1 --2- 172.21.15.123:0/317408841 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d50102e20 0x7f0d50113ad0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0d40002a50 tx=0x7f0d400046b0 comp rx=0 tx=0).stop 2024-03-30T04:36:38.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.412+0000 7f0d55000700 1 -- 172.21.15.123:0/317408841 shutdown_connections 2024-03-30T04:36:38.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.412+0000 7f0d55000700 1 --2- 172.21.15.123:0/317408841 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0d3406cc30 0x7f0d3406f0e0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:38.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.412+0000 7f0d55000700 1 --2- 172.21.15.123:0/317408841 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d50102e20 0x7f0d50113ad0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:38.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.412+0000 7f0d55000700 1 --2- 172.21.15.123:0/317408841 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0d501037c0 0x7f0d50114010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:38.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.412+0000 7f0d55000700 1 -- 172.21.15.123:0/317408841 >> 172.21.15.123:0/317408841 conn(0x7f0d500f89d0 msgr2=0x7f0d500f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:36:38.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.412+0000 7f0d55000700 1 -- 172.21.15.123:0/317408841 shutdown_connections 2024-03-30T04:36:38.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.412+0000 7f0d55000700 1 -- 172.21.15.123:0/317408841 wait complete. 2024-03-30T04:36:38.412 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 23 2024-03-30T04:36:38.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.546+0000 7ff3f9c00700 1 -- 172.21.15.123:0/1806251186 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3f41037c0 msgr2=0x7ff3f40fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:38.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.546+0000 7ff3f9c00700 1 --2- 172.21.15.123:0/1806251186 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3f41037c0 0x7ff3f40fce30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff3e0004940 tx=0x7ff3e0009e30 comp rx=0 tx=0).stop 2024-03-30T04:36:38.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.546+0000 7ff3f9c00700 1 -- 172.21.15.123:0/1806251186 shutdown_connections 2024-03-30T04:36:38.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.546+0000 7ff3f9c00700 1 --2- 172.21.15.123:0/1806251186 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3f41037c0 0x7ff3f40fce30 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:38.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.546+0000 7ff3f9c00700 1 --2- 172.21.15.123:0/1806251186 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff3f4102e20 0x7ff3f41031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:38.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.546+0000 7ff3f9c00700 1 -- 172.21.15.123:0/1806251186 >> 172.21.15.123:0/1806251186 conn(0x7ff3f40f89d0 msgr2=0x7ff3f40fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:36:38.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.546+0000 7ff3f9c00700 1 -- 172.21.15.123:0/1806251186 shutdown_connections 2024-03-30T04:36:38.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.546+0000 7ff3f9c00700 1 -- 172.21.15.123:0/1806251186 wait complete. 2024-03-30T04:36:38.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.547+0000 7ff3f9c00700 1 Processor -- start 2024-03-30T04:36:38.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.547+0000 7ff3f9c00700 1 -- start start 2024-03-30T04:36:38.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.548+0000 7ff3f9c00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff3f4102e20 0x7ff3f40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:36:38.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.548+0000 7ff3f9c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3f41037c0 0x7ff3f4070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:36:38.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.548+0000 7ff3f9c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f4070d90 con 0x7ff3f41037c0 2024-03-30T04:36:38.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.548+0000 7ff3f9c00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f4070ed0 con 0x7ff3f4102e20 2024-03-30T04:36:38.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.548+0000 7ff3f2000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3f41037c0 0x7ff3f4070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:36:38.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.548+0000 7ff3f2000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3f41037c0 0x7ff3f4070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46554/0 (socket says 172.21.15.123:46554) 2024-03-30T04:36:38.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.548+0000 7ff3f2000700 1 -- 172.21.15.123:0/61441779 learned_addr learned my addr 172.21.15.123:0/61441779 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:36:38.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.548+0000 7ff3f2a00700 1 --2- 172.21.15.123:0/61441779 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff3f4102e20 0x7ff3f40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:36:38.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.548+0000 7ff3f2000700 1 -- 172.21.15.123:0/61441779 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff3f4102e20 msgr2=0x7ff3f40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:38.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.548+0000 7ff3f2000700 1 --2- 172.21.15.123:0/61441779 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff3f4102e20 0x7ff3f40702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:38.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.548+0000 7ff3f2000700 1 -- 172.21.15.123:0/61441779 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e0005040 con 0x7ff3f41037c0 2024-03-30T04:36:38.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.549+0000 7ff3f2000700 1 --2- 172.21.15.123:0/61441779 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3f41037c0 0x7ff3f4070800 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff3e0009390 tx=0x7ff3e0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:36:38.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.549+0000 7ff3eaa00700 1 -- 172.21.15.123:0/61441779 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3e000e040 con 0x7ff3f41037c0 2024-03-30T04:36:38.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.549+0000 7ff3eaa00700 1 -- 172.21.15.123:0/61441779 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff3e0004290 con 0x7ff3f41037c0 2024-03-30T04:36:38.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.549+0000 7ff3f9c00700 1 -- 172.21.15.123:0/61441779 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3f4071120 con 0x7ff3f41037c0 2024-03-30T04:36:38.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.549+0000 7ff3f9c00700 1 -- 172.21.15.123:0/61441779 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3f411a200 con 0x7ff3f41037c0 2024-03-30T04:36:38.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.550+0000 7ff3eaa00700 1 -- 172.21.15.123:0/61441779 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3e0013e60 con 0x7ff3f41037c0 2024-03-30T04:36:38.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.550+0000 7ff3f9c00700 1 -- 172.21.15.123:0/61441779 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3f40fe570 con 0x7ff3f41037c0 2024-03-30T04:36:38.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.551+0000 7ff3eaa00700 1 -- 172.21.15.123:0/61441779 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7ff3e0013690 con 0x7ff3f41037c0 2024-03-30T04:36:38.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.552+0000 7ff3eaa00700 1 --2- 172.21.15.123:0/61441779 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ff3d806cc30 0x7ff3d806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:36:38.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.552+0000 7ff3eaa00700 1 -- 172.21.15.123:0/61441779 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6324+0+0 (secure 0 0 0) 0x7ff3e008d200 con 0x7ff3f41037c0 2024-03-30T04:36:38.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.552+0000 7ff3f2a00700 1 --2- 172.21.15.123:0/61441779 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ff3d806cc30 0x7ff3d806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:36:38.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.553+0000 7ff3f2a00700 1 --2- 172.21.15.123:0/61441779 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ff3d806cc30 0x7ff3d806f0e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff3e4004f40 tx=0x7ff3e400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:36:38.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.556+0000 7ff3eaa00700 1 -- 172.21.15.123:0/61441779 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff3e00572f0 con 0x7ff3f41037c0 2024-03-30T04:36:38.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.778+0000 7ff3f9c00700 1 -- 172.21.15.123:0/61441779 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff3f4109e30 con 0x7ff3d806cc30 2024-03-30T04:36:38.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.780+0000 7ff3eaa00700 1 -- 172.21.15.123:0/61441779 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7ff3f4109e30 con 0x7ff3d806cc30 2024-03-30T04:36:38.778 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:36:38.778 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:36:38.778 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:36:38.779 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T04:36:38.779 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:36:38.779 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T04:36:38.779 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T04:36:38.779 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T04:36:38.779 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:36:38.779 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "11/21 daemons upgraded", 2024-03-30T04:36:38.779 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading osd daemons", 2024-03-30T04:36:38.779 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:36:38.779 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:36:38.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.782+0000 7ff3f9c00700 1 -- 172.21.15.123:0/61441779 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ff3d806cc30 msgr2=0x7ff3d806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:38.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.782+0000 7ff3f9c00700 1 --2- 172.21.15.123:0/61441779 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ff3d806cc30 0x7ff3d806f0e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff3e4004f40 tx=0x7ff3e400b040 comp rx=0 tx=0).stop 2024-03-30T04:36:38.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.782+0000 7ff3f9c00700 1 -- 172.21.15.123:0/61441779 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3f41037c0 msgr2=0x7ff3f4070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:38.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.782+0000 7ff3f9c00700 1 --2- 172.21.15.123:0/61441779 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3f41037c0 0x7ff3f4070800 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff3e0009390 tx=0x7ff3e0009e30 comp rx=0 tx=0).stop 2024-03-30T04:36:38.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.782+0000 7ff3f9c00700 1 -- 172.21.15.123:0/61441779 shutdown_connections 2024-03-30T04:36:38.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.782+0000 7ff3f9c00700 1 --2- 172.21.15.123:0/61441779 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ff3d806cc30 0x7ff3d806f0e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:38.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.782+0000 7ff3f9c00700 1 --2- 172.21.15.123:0/61441779 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff3f4102e20 0x7ff3f40702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:38.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.782+0000 7ff3f9c00700 1 --2- 172.21.15.123:0/61441779 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3f41037c0 0x7ff3f4070800 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:38.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.782+0000 7ff3f9c00700 1 -- 172.21.15.123:0/61441779 >> 172.21.15.123:0/61441779 conn(0x7ff3f40f89d0 msgr2=0x7ff3f40f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:36:38.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.782+0000 7ff3f9c00700 1 -- 172.21.15.123:0/61441779 shutdown_connections 2024-03-30T04:36:38.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.783+0000 7ff3f9c00700 1 -- 172.21.15.123:0/61441779 wait complete. 2024-03-30T04:36:38.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.929+0000 7f91b4000700 1 -- 172.21.15.123:0/688862566 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91ac103720 msgr2=0x7f91ac0fcd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:38.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.929+0000 7f91b4000700 1 --2- 172.21.15.123:0/688862566 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91ac103720 0x7f91ac0fcd90 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9198004940 tx=0x7f9198009e30 comp rx=0 tx=0).stop 2024-03-30T04:36:38.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.929+0000 7f91b4000700 1 -- 172.21.15.123:0/688862566 shutdown_connections 2024-03-30T04:36:38.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.929+0000 7f91b4000700 1 --2- 172.21.15.123:0/688862566 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91ac103720 0x7f91ac0fcd90 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:38.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.929+0000 7f91b4000700 1 --2- 172.21.15.123:0/688862566 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91ac102d80 0x7f91ac103150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:38.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.929+0000 7f91b4000700 1 -- 172.21.15.123:0/688862566 >> 172.21.15.123:0/688862566 conn(0x7f91ac0f8930 msgr2=0x7f91ac0fad40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:36:38.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.929+0000 7f91b4000700 1 -- 172.21.15.123:0/688862566 shutdown_connections 2024-03-30T04:36:38.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.929+0000 7f91b4000700 1 -- 172.21.15.123:0/688862566 wait complete. 2024-03-30T04:36:38.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.930+0000 7f91b4000700 1 Processor -- start 2024-03-30T04:36:38.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.930+0000 7f91b4000700 1 -- start start 2024-03-30T04:36:38.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.930+0000 7f91b4000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91ac102d80 0x7f91ac115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:36:38.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.930+0000 7f91b4000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91ac103720 0x7f91ac1161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:36:38.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.930+0000 7f91b4000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91ac1168a0 con 0x7f91ac102d80 2024-03-30T04:36:38.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.930+0000 7f91b4000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91ac10fbe0 con 0x7f91ac103720 2024-03-30T04:36:38.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.930+0000 7f91b1600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91ac102d80 0x7f91ac115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:36:38.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.930+0000 7f91b1600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91ac102d80 0x7f91ac115c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46570/0 (socket says 172.21.15.123:46570) 2024-03-30T04:36:38.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.930+0000 7f91b1600700 1 -- 172.21.15.123:0/2952622559 learned_addr learned my addr 172.21.15.123:0/2952622559 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:36:38.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.931+0000 7f91b0c00700 1 --2- 172.21.15.123:0/2952622559 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91ac103720 0x7f91ac1161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:36:38.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.931+0000 7f91b1600700 1 -- 172.21.15.123:0/2952622559 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91ac103720 msgr2=0x7f91ac1161c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:38.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.931+0000 7f91b1600700 1 --2- 172.21.15.123:0/2952622559 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91ac103720 0x7f91ac1161c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:38.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.931+0000 7f91b1600700 1 -- 172.21.15.123:0/2952622559 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9198005040 con 0x7f91ac102d80 2024-03-30T04:36:38.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.931+0000 7f91b1600700 1 --2- 172.21.15.123:0/2952622559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91ac102d80 0x7f91ac115c80 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f91a000b790 tx=0x7f91a0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:36:38.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.931+0000 7f91a9600700 1 -- 172.21.15.123:0/2952622559 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f91a000f770 con 0x7f91ac102d80 2024-03-30T04:36:38.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.931+0000 7f91a9600700 1 -- 172.21.15.123:0/2952622559 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f91a000fd80 con 0x7f91ac102d80 2024-03-30T04:36:38.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.931+0000 7f91a9600700 1 -- 172.21.15.123:0/2952622559 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f91a000d310 con 0x7f91ac102d80 2024-03-30T04:36:38.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.931+0000 7f91b4000700 1 -- 172.21.15.123:0/2952622559 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91ac10fe60 con 0x7f91ac102d80 2024-03-30T04:36:38.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.932+0000 7f91b4000700 1 -- 172.21.15.123:0/2952622559 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91ac110420 con 0x7f91ac102d80 2024-03-30T04:36:38.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.933+0000 7f91b4000700 1 -- 172.21.15.123:0/2952622559 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91ac0fe4d0 con 0x7f91ac102d80 2024-03-30T04:36:38.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.934+0000 7f91a9600700 1 -- 172.21.15.123:0/2952622559 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f91a0010420 con 0x7f91ac102d80 2024-03-30T04:36:38.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.935+0000 7f91a9600700 1 --2- 172.21.15.123:0/2952622559 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f919406cc30 0x7f919406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:36:38.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.935+0000 7f91a9600700 1 -- 172.21.15.123:0/2952622559 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f91a008bf60 con 0x7f91ac102d80 2024-03-30T04:36:38.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.935+0000 7f91b0c00700 1 --2- 172.21.15.123:0/2952622559 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f919406cc30 0x7f919406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:36:38.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.936+0000 7f91b0c00700 1 --2- 172.21.15.123:0/2952622559 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f919406cc30 0x7f919406f0e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f91ac111520 tx=0x7f9198009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:36:38.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:38.940+0000 7f91a9600700 1 -- 172.21.15.123:0/2952622559 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f91a0059ab0 con 0x7f91ac102d80 2024-03-30T04:36:38.964 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:38 smithi135 ceph-mon[106011]: from='client.34350 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:36:38.964 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:38 smithi135 ceph-mon[106011]: pgmap v150: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s; 54/324 objects degraded (16.667%) 2024-03-30T04:36:38.965 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:38 smithi135 ceph-mon[106011]: from='client.34354 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:36:38.965 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:38 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2311079629' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:36:38.965 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:38 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/317408841' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:36:38.965 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:38 smithi135 ceph-mon[106011]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-30T04:36:38.965 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:38 smithi135 ceph-mon[106011]: osd.4 [v2:172.21.15.135:6808/4133828133,v1:172.21.15.135:6809/4133828133] boot 2024-03-30T04:36:38.965 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:38 smithi135 ceph-mon[106011]: osdmap e73: 6 total, 6 up, 6 in 2024-03-30T04:36:38.965 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:38 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T04:36:39.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:38 smithi123 ceph-mon[139369]: from='client.34350 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:36:39.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:38 smithi123 ceph-mon[139369]: pgmap v150: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s; 54/324 objects degraded (16.667%) 2024-03-30T04:36:39.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:38 smithi123 ceph-mon[139369]: from='client.34354 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:36:39.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:38 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2311079629' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:36:39.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:38 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/317408841' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:36:39.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:38 smithi123 ceph-mon[139369]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-30T04:36:39.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:38 smithi123 ceph-mon[139369]: osd.4 [v2:172.21.15.135:6808/4133828133,v1:172.21.15.135:6809/4133828133] boot 2024-03-30T04:36:39.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:38 smithi123 ceph-mon[139369]: osdmap e73: 6 total, 6 up, 6 in 2024-03-30T04:36:39.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:38 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T04:36:39.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:39.224+0000 7f91b4000700 1 -- 172.21.15.123:0/2952622559 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f91ac04bf00 con 0x7f91ac102d80 2024-03-30T04:36:39.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:39.224+0000 7f91a9600700 1 -- 172.21.15.123:0/2952622559 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+1073 (secure 0 0 0) 0x7f91a0059640 con 0x7f91ac102d80 2024-03-30T04:36:39.223 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_WARN Degraded data redundancy: 54/324 objects degraded (16.667%), 16 pgs degraded 2024-03-30T04:36:39.223 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 54/324 objects degraded (16.667%), 16 pgs degraded 2024-03-30T04:36:39.223 INFO:teuthology.orchestra.run.smithi123.stdout: pg 2.4 is active+undersized+degraded, acting [1,0] 2024-03-30T04:36:39.223 INFO:teuthology.orchestra.run.smithi123.stdout: pg 2.5 is active+undersized+degraded, acting [3,0] 2024-03-30T04:36:39.223 INFO:teuthology.orchestra.run.smithi123.stdout: pg 2.6 is active+undersized+degraded, acting [1,3] 2024-03-30T04:36:39.223 INFO:teuthology.orchestra.run.smithi123.stdout: pg 2.7 is active+undersized+degraded, acting [3,2] 2024-03-30T04:36:39.223 INFO:teuthology.orchestra.run.smithi123.stdout: pg 2.9 is active+undersized+degraded, acting [1,0] 2024-03-30T04:36:39.223 INFO:teuthology.orchestra.run.smithi123.stdout: pg 2.a is active+undersized+degraded, acting [1,3] 2024-03-30T04:36:39.223 INFO:teuthology.orchestra.run.smithi123.stdout: pg 2.b is active+undersized+degraded, acting [3,5] 2024-03-30T04:36:39.224 INFO:teuthology.orchestra.run.smithi123.stdout: pg 2.f is active+undersized+degraded, acting [0,5] 2024-03-30T04:36:39.224 INFO:teuthology.orchestra.run.smithi123.stdout: pg 2.11 is active+undersized+degraded, acting [3,1] 2024-03-30T04:36:39.224 INFO:teuthology.orchestra.run.smithi123.stdout: pg 2.13 is active+undersized+degraded, acting [0,2] 2024-03-30T04:36:39.224 INFO:teuthology.orchestra.run.smithi123.stdout: pg 2.14 is active+undersized+degraded, acting [3,5] 2024-03-30T04:36:39.224 INFO:teuthology.orchestra.run.smithi123.stdout: pg 2.18 is active+undersized+degraded, acting [5,3] 2024-03-30T04:36:39.224 INFO:teuthology.orchestra.run.smithi123.stdout: pg 2.19 is active+undersized+degraded, acting [0,2] 2024-03-30T04:36:39.224 INFO:teuthology.orchestra.run.smithi123.stdout: pg 2.1a is active+undersized+degraded, acting [3,5] 2024-03-30T04:36:39.224 INFO:teuthology.orchestra.run.smithi123.stdout: pg 2.1c is active+undersized+degraded, acting [5,2] 2024-03-30T04:36:39.224 INFO:teuthology.orchestra.run.smithi123.stdout: pg 2.1f is active+undersized+degraded, acting [0,3] 2024-03-30T04:36:39.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:39.227+0000 7f91b4000700 1 -- 172.21.15.123:0/2952622559 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f919406cc30 msgr2=0x7f919406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:39.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:39.227+0000 7f91b4000700 1 --2- 172.21.15.123:0/2952622559 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f919406cc30 0x7f919406f0e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f91ac111520 tx=0x7f9198009e30 comp rx=0 tx=0).stop 2024-03-30T04:36:39.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:39.227+0000 7f91b4000700 1 -- 172.21.15.123:0/2952622559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91ac102d80 msgr2=0x7f91ac115c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:36:39.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:39.227+0000 7f91b4000700 1 --2- 172.21.15.123:0/2952622559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91ac102d80 0x7f91ac115c80 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f91a000b790 tx=0x7f91a0009250 comp rx=0 tx=0).stop 2024-03-30T04:36:39.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:39.227+0000 7f91b4000700 1 -- 172.21.15.123:0/2952622559 shutdown_connections 2024-03-30T04:36:39.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:39.227+0000 7f91b4000700 1 --2- 172.21.15.123:0/2952622559 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f919406cc30 0x7f919406f0e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:39.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:39.227+0000 7f91b4000700 1 --2- 172.21.15.123:0/2952622559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91ac102d80 0x7f91ac115c80 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:39.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:39.227+0000 7f91b4000700 1 --2- 172.21.15.123:0/2952622559 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91ac103720 0x7f91ac1161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:36:39.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:39.227+0000 7f91b4000700 1 -- 172.21.15.123:0/2952622559 >> 172.21.15.123:0/2952622559 conn(0x7f91ac0f8930 msgr2=0x7f91ac0f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:36:39.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:39.227+0000 7f91b4000700 1 -- 172.21.15.123:0/2952622559 shutdown_connections 2024-03-30T04:36:39.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:36:39.227+0000 7f91b4000700 1 -- 172.21.15.123:0/2952622559 wait complete. 2024-03-30T04:36:40.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:39 smithi123 ceph-mon[139369]: from='client.34366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:36:40.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:39 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2952622559' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:36:40.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:39 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:36:40.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:39 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:36:40.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:39 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:36:40.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:39 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:36:40.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:39 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:36:40.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:39 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:36:40.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:39 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:36:40.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:39 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:36:40.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:39 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:36:40.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:39 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-30T04:36:40.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:39 smithi123 ceph-mon[139369]: osdmap e74: 6 total, 6 up, 6 in 2024-03-30T04:36:40.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:39 smithi135 ceph-mon[106011]: from='client.34366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:36:40.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:39 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2952622559' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:36:40.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:39 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:36:40.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:39 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:36:40.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:39 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:36:40.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:39 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:36:40.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:39 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:36:40.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:39 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:36:40.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:39 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:36:40.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:39 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:36:40.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:39 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:36:40.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:39 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-30T04:36:40.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:39 smithi135 ceph-mon[106011]: osdmap e74: 6 total, 6 up, 6 in 2024-03-30T04:36:41.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:40 smithi123 ceph-mon[139369]: pgmap v152: 65 pgs: 10 peering, 11 active+undersized+degraded, 13 active+undersized, 31 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 2 op/s; 31/324 objects degraded (9.568%) 2024-03-30T04:36:41.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:40 smithi123 ceph-mon[139369]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-30T04:36:41.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:40 smithi123 ceph-mon[139369]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-03-30T04:36:41.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:40 smithi123 ceph-mon[139369]: Health check update: Degraded data redundancy: 31/324 objects degraded (9.568%), 11 pgs degraded (PG_DEGRADED) 2024-03-30T04:36:41.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:40 smithi135 ceph-mon[106011]: pgmap v152: 65 pgs: 10 peering, 11 active+undersized+degraded, 13 active+undersized, 31 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 2 op/s; 31/324 objects degraded (9.568%) 2024-03-30T04:36:41.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:40 smithi135 ceph-mon[106011]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-30T04:36:41.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:40 smithi135 ceph-mon[106011]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-03-30T04:36:41.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:40 smithi135 ceph-mon[106011]: Health check update: Degraded data redundancy: 31/324 objects degraded (9.568%), 11 pgs degraded (PG_DEGRADED) 2024-03-30T04:36:43.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:42 smithi123 ceph-mon[139369]: pgmap v154: 65 pgs: 10 peering, 7 active+undersized+degraded, 7 active+undersized, 41 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1023 B/s rd, 1 op/s; 17/324 objects degraded (5.247%) 2024-03-30T04:36:43.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:42 smithi135 ceph-mon[106011]: pgmap v154: 65 pgs: 10 peering, 7 active+undersized+degraded, 7 active+undersized, 41 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1023 B/s rd, 1 op/s; 17/324 objects degraded (5.247%) 2024-03-30T04:36:45.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:44 smithi123 ceph-mon[139369]: pgmap v155: 65 pgs: 10 peering, 4 active+undersized+degraded, 4 active+undersized, 47 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 935 B/s rd, 1 op/s; 10/324 objects degraded (3.086%) 2024-03-30T04:36:45.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:36:45.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:36:45.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:44 smithi135 ceph-mon[106011]: pgmap v155: 65 pgs: 10 peering, 4 active+undersized+degraded, 4 active+undersized, 47 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 935 B/s rd, 1 op/s; 10/324 objects degraded (3.086%) 2024-03-30T04:36:45.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:44 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:36:45.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:44 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:36:46.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:45 smithi123 ceph-mon[139369]: pgmap v156: 65 pgs: 65 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-30T04:36:46.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:45 smithi123 ceph-mon[139369]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 10/324 objects degraded (3.086%), 4 pgs degraded) 2024-03-30T04:36:46.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:45 smithi123 ceph-mon[139369]: Cluster is now healthy 2024-03-30T04:36:46.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:45 smithi135 ceph-mon[106011]: pgmap v156: 65 pgs: 65 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-30T04:36:46.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:45 smithi135 ceph-mon[106011]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 10/324 objects degraded (3.086%), 4 pgs degraded) 2024-03-30T04:36:46.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:45 smithi135 ceph-mon[106011]: Cluster is now healthy 2024-03-30T04:36:48.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:48 smithi135 ceph-mon[106011]: pgmap v157: 65 pgs: 65 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-30T04:36:48.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:48 smithi123 ceph-mon[139369]: pgmap v157: 65 pgs: 65 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-30T04:36:50.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:50 smithi123 ceph-mon[139369]: pgmap v158: 65 pgs: 65 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-30T04:36:50.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:50 smithi135 ceph-mon[106011]: pgmap v158: 65 pgs: 65 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-30T04:36:52.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:52 smithi135 ceph-mon[106011]: pgmap v159: 65 pgs: 65 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-30T04:36:52.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:52 smithi123 ceph-mon[139369]: pgmap v159: 65 pgs: 65 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-30T04:36:54.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:54 smithi135 ceph-mon[106011]: pgmap v160: 65 pgs: 65 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:36:54.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:54 smithi123 ceph-mon[139369]: pgmap v160: 65 pgs: 65 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:36:55.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:55 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:36:55.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:55 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-30T04:36:55.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:55 smithi135 ceph-mon[106011]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-30T04:36:55.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:55 smithi135 ceph-mon[106011]: Upgrade: osd.5 is safe to restart 2024-03-30T04:36:55.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:55 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:36:55.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:55 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-30T04:36:55.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:55 smithi123 ceph-mon[139369]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-30T04:36:55.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:55 smithi123 ceph-mon[139369]: Upgrade: osd.5 is safe to restart 2024-03-30T04:36:56.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:56 smithi135 ceph-mon[106011]: pgmap v161: 65 pgs: 65 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:36:56.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:56 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:36:56.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:56 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-30T04:36:56.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:56 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:36:56.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:56 smithi123 ceph-mon[139369]: pgmap v161: 65 pgs: 65 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T04:36:56.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:56 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:36:56.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:56 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-30T04:36:56.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:56 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:36:57.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:57 smithi135 ceph-mon[106011]: Upgrade: Updating osd.5 2024-03-30T04:36:57.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:57 smithi135 ceph-mon[106011]: Deploying daemon osd.5 on smithi135 2024-03-30T04:36:57.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:57 smithi123 ceph-mon[139369]: Upgrade: Updating osd.5 2024-03-30T04:36:57.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:57 smithi123 ceph-mon[139369]: Deploying daemon osd.5 on smithi135 2024-03-30T04:36:58.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:36:58 smithi135 ceph-mon[106011]: pgmap v162: 65 pgs: 65 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:36:58.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:36:58 smithi123 ceph-mon[139369]: pgmap v162: 65 pgs: 65 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T04:37:00.434 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 04:37:00 smithi135 systemd[1]: Stopping Ceph osd.5 for 2e77a2be-ee4c-11ee-b647-cb9ed24678a4... 2024-03-30T04:37:00.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:00 smithi123 ceph-mon[139369]: pgmap v163: 65 pgs: 65 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-30T04:37:00.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:00 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:00.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:00 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:37:00.736 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:00 smithi135 ceph-mon[106011]: pgmap v163: 65 pgs: 65 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-30T04:37:00.737 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:00 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:00.737 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:00 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:37:01.100 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 04:37:00 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-5[67404]: 2024-03-30T04:37:00.737+0000 7f4f12000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T04:37:01.101 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 04:37:00 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-5[67404]: 2024-03-30T04:37:00.737+0000 7f4f12000700 -1 osd.5 74 *** Got signal Terminated *** 2024-03-30T04:37:01.101 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 04:37:00 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-5[67404]: 2024-03-30T04:37:00.737+0000 7f4f12000700 -1 osd.5 74 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T04:37:01.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:01 smithi123 ceph-mon[139369]: osd.5 marked itself down and dead 2024-03-30T04:37:01.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:01 smithi135 ceph-mon[106011]: osd.5 marked itself down and dead 2024-03-30T04:37:02.601 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 04:37:02 smithi135 bash[131028]: ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-5 2024-03-30T04:37:02.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:02 smithi135 ceph-mon[106011]: pgmap v164: 65 pgs: 65 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-30T04:37:02.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:02 smithi135 ceph-mon[106011]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-30T04:37:02.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:02 smithi135 ceph-mon[106011]: osdmap e75: 6 total, 5 up, 6 in 2024-03-30T04:37:02.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:02 smithi123 ceph-mon[139369]: pgmap v164: 65 pgs: 65 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-30T04:37:02.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:02 smithi123 ceph-mon[139369]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-30T04:37:02.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:02 smithi123 ceph-mon[139369]: osdmap e75: 6 total, 5 up, 6 in 2024-03-30T04:37:03.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:03 smithi135 ceph-mon[106011]: osdmap e76: 6 total, 5 up, 6 in 2024-03-30T04:37:03.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:03 smithi123 ceph-mon[139369]: osdmap e76: 6 total, 5 up, 6 in 2024-03-30T04:37:04.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:04 smithi123 ceph-mon[139369]: pgmap v167: 65 pgs: 7 peering, 9 stale+active+clean, 49 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s 2024-03-30T04:37:04.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:04 smithi123 ceph-mon[139369]: Health check failed: Reduced data availability: 4 pgs peering (PG_AVAILABILITY) 2024-03-30T04:37:04.827 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:04 smithi135 ceph-mon[106011]: pgmap v167: 65 pgs: 7 peering, 9 stale+active+clean, 49 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s 2024-03-30T04:37:04.827 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:04 smithi135 ceph-mon[106011]: Health check failed: Reduced data availability: 4 pgs peering (PG_AVAILABILITY) 2024-03-30T04:37:04.827 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 04:37:04 smithi135 systemd[1]: ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4@osd.5.service: Succeeded. 2024-03-30T04:37:04.827 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 04:37:04 smithi135 systemd[1]: Stopped Ceph osd.5 for 2e77a2be-ee4c-11ee-b647-cb9ed24678a4. 2024-03-30T04:37:05.100 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 04:37:04 smithi135 systemd[1]: Starting Ceph osd.5 for 2e77a2be-ee4c-11ee-b647-cb9ed24678a4... 2024-03-30T04:37:05.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:05 smithi135 ceph-mon[106011]: Health check failed: Degraded data redundancy: 46/324 objects degraded (14.198%), 11 pgs degraded (PG_DEGRADED) 2024-03-30T04:37:05.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:05 smithi123 ceph-mon[139369]: Health check failed: Degraded data redundancy: 46/324 objects degraded (14.198%), 11 pgs degraded (PG_DEGRADED) 2024-03-30T04:37:06.511 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 04:37:06 smithi135 bash[131297]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-30T04:37:06.512 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 04:37:06 smithi135 bash[131297]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-30T04:37:06.512 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 04:37:06 smithi135 bash[131297]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-30T04:37:06.512 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 04:37:06 smithi135 bash[131297]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-30T04:37:06.512 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 04:37:06 smithi135 bash[131297]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5/block 2024-03-30T04:37:06.512 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 04:37:06 smithi135 bash[131297]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-30T04:37:06.512 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 04:37:06 smithi135 bash[131297]: --> ceph-volume raw activate successful for osd ID: 5 2024-03-30T04:37:06.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:06 smithi135 ceph-mon[106011]: pgmap v168: 65 pgs: 11 active+undersized+degraded, 7 peering, 2 stale+active+clean, 7 active+undersized, 38 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 46/324 objects degraded (14.198%) 2024-03-30T04:37:06.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:06 smithi123 ceph-mon[139369]: pgmap v168: 65 pgs: 11 active+undersized+degraded, 7 peering, 2 stale+active+clean, 7 active+undersized, 38 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 46/324 objects degraded (14.198%) 2024-03-30T04:37:07.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:07 smithi135 ceph-mon[106011]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs peering) 2024-03-30T04:37:07.851 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 04:37:07 smithi135 ceph-osd[131590]: -- 172.21.15.135:0/749523276 <== mon.1 v2:172.21.15.135:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b41af4b1e0 con 0x55b41af1ac00 2024-03-30T04:37:07.851 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 04:37:07 smithi135 bash[131552]: 27615b61023134c06c0b741a14489cafe1da1e575cf397ac914bd8da5945d700 2024-03-30T04:37:07.851 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 04:37:07 smithi135 systemd[1]: Started Ceph osd.5 for 2e77a2be-ee4c-11ee-b647-cb9ed24678a4. 2024-03-30T04:37:07.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:07 smithi123 ceph-mon[139369]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs peering) 2024-03-30T04:37:08.350 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 04:37:08 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-5[131567]: 2024-03-30T04:37:08.091+0000 7fadde5b7640 -1 Falling back to public interface 2024-03-30T04:37:08.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:08 smithi123 ceph-mon[139369]: pgmap v169: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 53/324 objects degraded (16.358%) 2024-03-30T04:37:08.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:08 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:08.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:08 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:08.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:08 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:37:09.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:08 smithi135 ceph-mon[106011]: pgmap v169: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 53/324 objects degraded (16.358%) 2024-03-30T04:37:09.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:08 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:09.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:08 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:09.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:08 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:37:09.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.374+0000 7f19c9e00700 1 -- 172.21.15.123:0/1495689124 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f19c4102e20 msgr2=0x7f19c41031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:09.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.374+0000 7f19c9e00700 1 --2- 172.21.15.123:0/1495689124 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f19c4102e20 0x7f19c41031f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f19b4004940 tx=0x7f19b4009e30 comp rx=0 tx=0).stop 2024-03-30T04:37:09.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.374+0000 7f19c9e00700 1 -- 172.21.15.123:0/1495689124 shutdown_connections 2024-03-30T04:37:09.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.374+0000 7f19c9e00700 1 --2- 172.21.15.123:0/1495689124 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f19c41037c0 0x7f19c40fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:09.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.374+0000 7f19c9e00700 1 --2- 172.21.15.123:0/1495689124 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f19c4102e20 0x7f19c41031f0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:09.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.374+0000 7f19c9e00700 1 -- 172.21.15.123:0/1495689124 >> 172.21.15.123:0/1495689124 conn(0x7f19c40f89d0 msgr2=0x7f19c40fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:37:09.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.374+0000 7f19c9e00700 1 -- 172.21.15.123:0/1495689124 shutdown_connections 2024-03-30T04:37:09.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.374+0000 7f19c9e00700 1 -- 172.21.15.123:0/1495689124 wait complete. 2024-03-30T04:37:09.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.375+0000 7f19c9e00700 1 Processor -- start 2024-03-30T04:37:09.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.376+0000 7f19c9e00700 1 -- start start 2024-03-30T04:37:09.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.376+0000 7f19c9e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f19c4102e20 0x7f19c4113a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:37:09.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.376+0000 7f19c9e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f19c41037c0 0x7f19c4113fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:37:09.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.376+0000 7f19c9e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c4114610 con 0x7f19c41037c0 2024-03-30T04:37:09.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.376+0000 7f19c9e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c410da00 con 0x7f19c4102e20 2024-03-30T04:37:09.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.376+0000 7f19c2000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f19c41037c0 0x7f19c4113fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:37:09.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.377+0000 7f19c2000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f19c41037c0 0x7f19c4113fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34186/0 (socket says 172.21.15.123:34186) 2024-03-30T04:37:09.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.377+0000 7f19c2000700 1 -- 172.21.15.123:0/2430883769 learned_addr learned my addr 172.21.15.123:0/2430883769 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:37:09.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.377+0000 7f19c2a00700 1 --2- 172.21.15.123:0/2430883769 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f19c4102e20 0x7f19c4113a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:37:09.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.377+0000 7f19c2000700 1 -- 172.21.15.123:0/2430883769 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f19c4102e20 msgr2=0x7f19c4113a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:09.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.377+0000 7f19c2000700 1 --2- 172.21.15.123:0/2430883769 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f19c4102e20 0x7f19c4113a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:09.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.377+0000 7f19c2000700 1 -- 172.21.15.123:0/2430883769 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19b4005040 con 0x7f19c41037c0 2024-03-30T04:37:09.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.377+0000 7f19c2a00700 1 --2- 172.21.15.123:0/2430883769 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f19c4102e20 0x7f19c4113a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:37:09.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.377+0000 7f19c2000700 1 --2- 172.21.15.123:0/2430883769 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f19c41037c0 0x7f19c4113fc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f19b000eb40 tx=0x7f19b000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:37:09.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.377+0000 7f19baa00700 1 -- 172.21.15.123:0/2430883769 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19b0007cc0 con 0x7f19c41037c0 2024-03-30T04:37:09.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.378+0000 7f19baa00700 1 -- 172.21.15.123:0/2430883769 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f19b000f040 con 0x7f19c41037c0 2024-03-30T04:37:09.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.378+0000 7f19c9e00700 1 -- 172.21.15.123:0/2430883769 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19c410dc80 con 0x7f19c41037c0 2024-03-30T04:37:09.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.378+0000 7f19baa00700 1 -- 172.21.15.123:0/2430883769 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19b000c880 con 0x7f19c41037c0 2024-03-30T04:37:09.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.378+0000 7f19c9e00700 1 -- 172.21.15.123:0/2430883769 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19c410e240 con 0x7f19c41037c0 2024-03-30T04:37:09.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.379+0000 7f19c9e00700 1 -- 172.21.15.123:0/2430883769 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19c40fe570 con 0x7f19c41037c0 2024-03-30T04:37:09.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.380+0000 7f19baa00700 1 -- 172.21.15.123:0/2430883769 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f19b000cab0 con 0x7f19c41037c0 2024-03-30T04:37:09.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.381+0000 7f19baa00700 1 --2- 172.21.15.123:0/2430883769 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f19a806cc30 0x7f19a806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:37:09.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.381+0000 7f19baa00700 1 -- 172.21.15.123:0/2430883769 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f19b008baa0 con 0x7f19c41037c0 2024-03-30T04:37:09.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.381+0000 7f19c2a00700 1 --2- 172.21.15.123:0/2430883769 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f19a806cc30 0x7f19a806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:37:09.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.381+0000 7f19c2a00700 1 --2- 172.21.15.123:0/2430883769 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f19a806cc30 0x7f19a806f0e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f19b400b0d0 tx=0x7f19b401c040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:37:09.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.385+0000 7f19baa00700 1 -- 172.21.15.123:0/2430883769 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f19b00595a0 con 0x7f19c41037c0 2024-03-30T04:37:09.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.610+0000 7f19c9e00700 1 -- 172.21.15.123:0/2430883769 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f19c4113750 con 0x7f19a806cc30 2024-03-30T04:37:09.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.611+0000 7f19baa00700 1 -- 172.21.15.123:0/2430883769 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f19c4113750 con 0x7f19a806cc30 2024-03-30T04:37:09.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.614+0000 7f19c9e00700 1 -- 172.21.15.123:0/2430883769 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f19a806cc30 msgr2=0x7f19a806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:09.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.614+0000 7f19c9e00700 1 --2- 172.21.15.123:0/2430883769 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f19a806cc30 0x7f19a806f0e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f19b400b0d0 tx=0x7f19b401c040 comp rx=0 tx=0).stop 2024-03-30T04:37:09.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.614+0000 7f19c9e00700 1 -- 172.21.15.123:0/2430883769 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f19c41037c0 msgr2=0x7f19c4113fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:09.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.614+0000 7f19c9e00700 1 --2- 172.21.15.123:0/2430883769 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f19c41037c0 0x7f19c4113fc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f19b000eb40 tx=0x7f19b000b040 comp rx=0 tx=0).stop 2024-03-30T04:37:09.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.614+0000 7f19c9e00700 1 -- 172.21.15.123:0/2430883769 shutdown_connections 2024-03-30T04:37:09.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.614+0000 7f19c9e00700 1 --2- 172.21.15.123:0/2430883769 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f19a806cc30 0x7f19a806f0e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:09.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.614+0000 7f19c9e00700 1 --2- 172.21.15.123:0/2430883769 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f19c4102e20 0x7f19c4113a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:09.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.614+0000 7f19c9e00700 1 --2- 172.21.15.123:0/2430883769 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f19c41037c0 0x7f19c4113fc0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:09.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.614+0000 7f19c9e00700 1 -- 172.21.15.123:0/2430883769 >> 172.21.15.123:0/2430883769 conn(0x7f19c40f89d0 msgr2=0x7f19c4105ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:37:09.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.614+0000 7f19c9e00700 1 -- 172.21.15.123:0/2430883769 shutdown_connections 2024-03-30T04:37:09.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.614+0000 7f19c9e00700 1 -- 172.21.15.123:0/2430883769 wait complete. 2024-03-30T04:37:09.626 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:37:09.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.747+0000 7f203be00700 1 -- 172.21.15.123:0/2342516654 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f203c105800 msgr2=0x7f203c0fee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:09.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.747+0000 7f203be00700 1 --2- 172.21.15.123:0/2342516654 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f203c105800 0x7f203c0fee70 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2028004940 tx=0x7f2028009e30 comp rx=0 tx=0).stop 2024-03-30T04:37:09.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.748+0000 7f203be00700 1 -- 172.21.15.123:0/2342516654 shutdown_connections 2024-03-30T04:37:09.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.748+0000 7f203be00700 1 --2- 172.21.15.123:0/2342516654 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f203c105800 0x7f203c0fee70 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:09.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.748+0000 7f203be00700 1 --2- 172.21.15.123:0/2342516654 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f203c104e60 0x7f203c105230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:09.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.748+0000 7f203be00700 1 -- 172.21.15.123:0/2342516654 >> 172.21.15.123:0/2342516654 conn(0x7f203c0fab10 msgr2=0x7f203c0fcef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:37:09.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.748+0000 7f203be00700 1 -- 172.21.15.123:0/2342516654 shutdown_connections 2024-03-30T04:37:09.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.748+0000 7f203be00700 1 -- 172.21.15.123:0/2342516654 wait complete. 2024-03-30T04:37:09.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.748+0000 7f203be00700 1 Processor -- start 2024-03-30T04:37:09.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.749+0000 7f203be00700 1 -- start start 2024-03-30T04:37:09.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.749+0000 7f203be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f203c104e60 0x7f203c115b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:37:09.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.749+0000 7f203be00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f203c105800 0x7f203c116090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:37:09.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.749+0000 7f203be00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f203c1166e0 con 0x7f203c104e60 2024-03-30T04:37:09.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.749+0000 7f203be00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f203c10fad0 con 0x7f203c105800 2024-03-30T04:37:09.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.749+0000 7f203aa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f203c104e60 0x7f203c115b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:37:09.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.749+0000 7f203aa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f203c104e60 0x7f203c115b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34206/0 (socket says 172.21.15.123:34206) 2024-03-30T04:37:09.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.749+0000 7f203aa00700 1 -- 172.21.15.123:0/1319000783 learned_addr learned my addr 172.21.15.123:0/1319000783 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:37:09.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.749+0000 7f203a000700 1 --2- 172.21.15.123:0/1319000783 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f203c105800 0x7f203c116090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:37:09.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.750+0000 7f203aa00700 1 -- 172.21.15.123:0/1319000783 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f203c105800 msgr2=0x7f203c116090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:09.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.750+0000 7f203aa00700 1 --2- 172.21.15.123:0/1319000783 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f203c105800 0x7f203c116090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:09.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.750+0000 7f203aa00700 1 -- 172.21.15.123:0/1319000783 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2028005040 con 0x7f203c104e60 2024-03-30T04:37:09.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.750+0000 7f203aa00700 1 --2- 172.21.15.123:0/1319000783 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f203c104e60 0x7f203c115b50 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f202c002a50 tx=0x7f202c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:37:09.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.750+0000 7f2032a00700 1 -- 172.21.15.123:0/1319000783 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f202c007cc0 con 0x7f203c104e60 2024-03-30T04:37:09.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.750+0000 7f2032a00700 1 -- 172.21.15.123:0/1319000783 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f202c00f040 con 0x7f203c104e60 2024-03-30T04:37:09.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.750+0000 7f203be00700 1 -- 172.21.15.123:0/1319000783 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f203c10fdb0 con 0x7f203c104e60 2024-03-30T04:37:09.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.750+0000 7f2032a00700 1 -- 172.21.15.123:0/1319000783 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f202c00d9c0 con 0x7f203c104e60 2024-03-30T04:37:09.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.750+0000 7f203be00700 1 -- 172.21.15.123:0/1319000783 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f203c110340 con 0x7f203c104e60 2024-03-30T04:37:09.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.751+0000 7f203be00700 1 -- 172.21.15.123:0/1319000783 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f203c1005b0 con 0x7f203c104e60 2024-03-30T04:37:09.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.752+0000 7f2032a00700 1 -- 172.21.15.123:0/1319000783 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f202c00db20 con 0x7f203c104e60 2024-03-30T04:37:09.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.754+0000 7f2032a00700 1 --2- 172.21.15.123:0/1319000783 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f202006cc30 0x7f202006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:37:09.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.754+0000 7f2032a00700 1 -- 172.21.15.123:0/1319000783 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f202c08bca0 con 0x7f203c104e60 2024-03-30T04:37:09.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.754+0000 7f203a000700 1 --2- 172.21.15.123:0/1319000783 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f202006cc30 0x7f202006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:37:09.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.754+0000 7f203a000700 1 --2- 172.21.15.123:0/1319000783 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f202006cc30 0x7f202006f0e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f203c110db0 tx=0x7f2028009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:37:09.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.758+0000 7f2032a00700 1 -- 172.21.15.123:0/1319000783 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f202c059770 con 0x7f203c104e60 2024-03-30T04:37:09.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.978+0000 7f203be00700 1 -- 172.21.15.123:0/1319000783 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f203c10be50 con 0x7f202006cc30 2024-03-30T04:37:09.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.979+0000 7f2032a00700 1 -- 172.21.15.123:0/1319000783 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f203c10be50 con 0x7f202006cc30 2024-03-30T04:37:09.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.981+0000 7f203be00700 1 -- 172.21.15.123:0/1319000783 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f202006cc30 msgr2=0x7f202006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:09.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.981+0000 7f203be00700 1 --2- 172.21.15.123:0/1319000783 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f202006cc30 0x7f202006f0e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f203c110db0 tx=0x7f2028009e30 comp rx=0 tx=0).stop 2024-03-30T04:37:09.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.981+0000 7f203be00700 1 -- 172.21.15.123:0/1319000783 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f203c104e60 msgr2=0x7f203c115b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:09.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.981+0000 7f203be00700 1 --2- 172.21.15.123:0/1319000783 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f203c104e60 0x7f203c115b50 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f202c002a50 tx=0x7f202c0046b0 comp rx=0 tx=0).stop 2024-03-30T04:37:09.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.981+0000 7f203be00700 1 -- 172.21.15.123:0/1319000783 shutdown_connections 2024-03-30T04:37:09.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.981+0000 7f203be00700 1 --2- 172.21.15.123:0/1319000783 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f202006cc30 0x7f202006f0e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:09.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.981+0000 7f203be00700 1 --2- 172.21.15.123:0/1319000783 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f203c104e60 0x7f203c115b50 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:09.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.981+0000 7f203be00700 1 --2- 172.21.15.123:0/1319000783 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f203c105800 0x7f203c116090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:09.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.981+0000 7f203be00700 1 -- 172.21.15.123:0/1319000783 >> 172.21.15.123:0/1319000783 conn(0x7f203c0fab10 msgr2=0x7f203c0fcbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:37:09.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.982+0000 7f203be00700 1 -- 172.21.15.123:0/1319000783 shutdown_connections 2024-03-30T04:37:09.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:09.982+0000 7f203be00700 1 -- 172.21.15.123:0/1319000783 wait complete. 2024-03-30T04:37:10.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.114+0000 7f40ebc00700 1 -- 172.21.15.123:0/835885410 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40e4102e20 msgr2=0x7f40e41031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:10.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.114+0000 7f40ebc00700 1 --2- 172.21.15.123:0/835885410 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40e4102e20 0x7f40e41031f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f40d8004940 tx=0x7f40d8009e30 comp rx=0 tx=0).stop 2024-03-30T04:37:10.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.114+0000 7f40ebc00700 1 -- 172.21.15.123:0/835885410 shutdown_connections 2024-03-30T04:37:10.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.114+0000 7f40ebc00700 1 --2- 172.21.15.123:0/835885410 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f40e41037c0 0x7f40e40fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:10.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.114+0000 7f40ebc00700 1 --2- 172.21.15.123:0/835885410 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40e4102e20 0x7f40e41031f0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:10.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.114+0000 7f40ebc00700 1 -- 172.21.15.123:0/835885410 >> 172.21.15.123:0/835885410 conn(0x7f40e40f8970 msgr2=0x7f40e40fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:37:10.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.114+0000 7f40ebc00700 1 -- 172.21.15.123:0/835885410 shutdown_connections 2024-03-30T04:37:10.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.114+0000 7f40ebc00700 1 -- 172.21.15.123:0/835885410 wait complete. 2024-03-30T04:37:10.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.115+0000 7f40ebc00700 1 Processor -- start 2024-03-30T04:37:10.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.115+0000 7f40ebc00700 1 -- start start 2024-03-30T04:37:10.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.115+0000 7f40ebc00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40e4102e20 0x7f40e40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:37:10.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.115+0000 7f40ebc00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f40e41037c0 0x7f40e4070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:37:10.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.115+0000 7f40ebc00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e4070d90 con 0x7f40e4102e20 2024-03-30T04:37:10.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.115+0000 7f40ebc00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e4070ed0 con 0x7f40e41037c0 2024-03-30T04:37:10.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.116+0000 7f40e9200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40e4102e20 0x7f40e40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:37:10.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.116+0000 7f40e9200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40e4102e20 0x7f40e40702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34224/0 (socket says 172.21.15.123:34224) 2024-03-30T04:37:10.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.116+0000 7f40e9200700 1 -- 172.21.15.123:0/492888109 learned_addr learned my addr 172.21.15.123:0/492888109 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:37:10.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.116+0000 7f40e3e00700 1 --2- 172.21.15.123:0/492888109 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f40e41037c0 0x7f40e4070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:37:10.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.116+0000 7f40e9200700 1 -- 172.21.15.123:0/492888109 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f40e41037c0 msgr2=0x7f40e4070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:10.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.116+0000 7f40e9200700 1 --2- 172.21.15.123:0/492888109 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f40e41037c0 0x7f40e4070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:10.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.116+0000 7f40e9200700 1 -- 172.21.15.123:0/492888109 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40d8005040 con 0x7f40e4102e20 2024-03-30T04:37:10.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.116+0000 7f40e9200700 1 --2- 172.21.15.123:0/492888109 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40e4102e20 0x7f40e40702c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f40d8009390 tx=0x7f40d8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:37:10.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.117+0000 7f40e0c00700 1 -- 172.21.15.123:0/492888109 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40d800e040 con 0x7f40e4102e20 2024-03-30T04:37:10.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.117+0000 7f40e0c00700 1 -- 172.21.15.123:0/492888109 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f40d8013420 con 0x7f40e4102e20 2024-03-30T04:37:10.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.117+0000 7f40ebc00700 1 -- 172.21.15.123:0/492888109 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40e4071120 con 0x7f40e4102e20 2024-03-30T04:37:10.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.117+0000 7f40e0c00700 1 -- 172.21.15.123:0/492888109 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40d801c630 con 0x7f40e4102e20 2024-03-30T04:37:10.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.117+0000 7f40ebc00700 1 -- 172.21.15.123:0/492888109 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40e411a210 con 0x7f40e4102e20 2024-03-30T04:37:10.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.118+0000 7f40ebc00700 1 -- 172.21.15.123:0/492888109 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40e40fe570 con 0x7f40e4102e20 2024-03-30T04:37:10.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.119+0000 7f40e0c00700 1 -- 172.21.15.123:0/492888109 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f40d80139e0 con 0x7f40e4102e20 2024-03-30T04:37:10.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.120+0000 7f40e0c00700 1 --2- 172.21.15.123:0/492888109 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f40cc06cc30 0x7f40cc06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:37:10.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.120+0000 7f40e0c00700 1 -- 172.21.15.123:0/492888109 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f40d808d330 con 0x7f40e4102e20 2024-03-30T04:37:10.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.120+0000 7f40e3e00700 1 --2- 172.21.15.123:0/492888109 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f40cc06cc30 0x7f40cc06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:37:10.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.121+0000 7f40e3e00700 1 --2- 172.21.15.123:0/492888109 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f40cc06cc30 0x7f40cc06f0e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f40e4071c00 tx=0x7f40d000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:37:10.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.124+0000 7f40e0c00700 1 -- 172.21.15.123:0/492888109 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f40d8057370 con 0x7f40e4102e20 2024-03-30T04:37:10.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.336+0000 7f40ebc00700 1 -- 172.21.15.123:0/492888109 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f40e40719b0 con 0x7f40cc06cc30 2024-03-30T04:37:10.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.346+0000 7f40e0c00700 1 -- 172.21.15.123:0/492888109 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f40e40719b0 con 0x7f40cc06cc30 2024-03-30T04:37:10.344 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:37:10.344 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (6m) 111s ago 18m 14.7M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:37:10.344 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (3m) 111s ago 19m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T04:37:10.345 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (3m) 36s ago 17m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T04:37:10.345 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (6m) 111s ago 18m 74.1M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T04:37:10.345 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (12m) 111s ago 12m 109M - 17.2.7-797-g5c024335 f77085f6aab7 2df066fba7a6 2024-03-30T04:37:10.345 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (12m) 111s ago 12m 58.0M - 17.2.7-797-g5c024335 f77085f6aab7 1293b8520db1 2024-03-30T04:37:10.345 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (12m) 36s ago 12m 67.6M - 17.2.7-797-g5c024335 f77085f6aab7 0dc41326818c 2024-03-30T04:37:10.345 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (12m) 36s ago 12m 119M - 17.2.7-797-g5c024335 f77085f6aab7 aef8a0cfc553 2024-03-30T04:37:10.345 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (8m) 111s ago 20m 502M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:37:10.345 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (7m) 36s ago 17m 449M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:37:10.345 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (4m) 111s ago 20m 45.6M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T04:37:10.345 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (4m) 36s ago 17m 38.8M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T04:37:10.345 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (7m) 111s ago 19m 7583k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:37:10.345 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (7m) 36s ago 17m 7708k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:37:10.345 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (3m) 111s ago 16m 99.5M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T04:37:10.345 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (2m) 111s ago 16m 90.4M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T04:37:10.346 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (117s) 111s ago 15m 10.0M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T04:37:10.346 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (78s) 36s ago 15m 94.5M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T04:37:10.346 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (41s) 36s ago 14m 11.0M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T04:37:10.346 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 starting - - - 2642M 2024-03-30T04:37:10.346 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (6m) 111s ago 18m 52.5M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:37:10.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.348+0000 7f40ebc00700 1 -- 172.21.15.123:0/492888109 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f40cc06cc30 msgr2=0x7f40cc06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:10.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.348+0000 7f40ebc00700 1 --2- 172.21.15.123:0/492888109 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f40cc06cc30 0x7f40cc06f0e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f40e4071c00 tx=0x7f40d000b040 comp rx=0 tx=0).stop 2024-03-30T04:37:10.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.348+0000 7f40ebc00700 1 -- 172.21.15.123:0/492888109 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40e4102e20 msgr2=0x7f40e40702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:10.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.348+0000 7f40ebc00700 1 --2- 172.21.15.123:0/492888109 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40e4102e20 0x7f40e40702c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f40d8009390 tx=0x7f40d8009e30 comp rx=0 tx=0).stop 2024-03-30T04:37:10.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.348+0000 7f40ebc00700 1 -- 172.21.15.123:0/492888109 shutdown_connections 2024-03-30T04:37:10.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.348+0000 7f40ebc00700 1 --2- 172.21.15.123:0/492888109 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f40cc06cc30 0x7f40cc06f0e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:10.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.348+0000 7f40ebc00700 1 --2- 172.21.15.123:0/492888109 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40e4102e20 0x7f40e40702c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:10.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.348+0000 7f40ebc00700 1 --2- 172.21.15.123:0/492888109 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f40e41037c0 0x7f40e4070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:10.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.348+0000 7f40ebc00700 1 -- 172.21.15.123:0/492888109 >> 172.21.15.123:0/492888109 conn(0x7f40e40f8970 msgr2=0x7f40e40f9a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:37:10.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.348+0000 7f40ebc00700 1 -- 172.21.15.123:0/492888109 shutdown_connections 2024-03-30T04:37:10.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.348+0000 7f40ebc00700 1 -- 172.21.15.123:0/492888109 wait complete. 2024-03-30T04:37:10.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.482+0000 7f5263e00700 1 -- 172.21.15.123:0/3909223223 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5264102c40 msgr2=0x7f5264103010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:10.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.482+0000 7f5263e00700 1 --2- 172.21.15.123:0/3909223223 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5264102c40 0x7f5264103010 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5254004940 tx=0x7f5254009e30 comp rx=0 tx=0).stop 2024-03-30T04:37:10.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.482+0000 7f5263e00700 1 -- 172.21.15.123:0/3909223223 shutdown_connections 2024-03-30T04:37:10.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.482+0000 7f5263e00700 1 --2- 172.21.15.123:0/3909223223 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f52641035e0 0x7f52640fcc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:10.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.482+0000 7f5263e00700 1 --2- 172.21.15.123:0/3909223223 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5264102c40 0x7f5264103010 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:10.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.482+0000 7f5263e00700 1 -- 172.21.15.123:0/3909223223 >> 172.21.15.123:0/3909223223 conn(0x7f52640f8830 msgr2=0x7f52640fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:37:10.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.482+0000 7f5263e00700 1 -- 172.21.15.123:0/3909223223 shutdown_connections 2024-03-30T04:37:10.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.482+0000 7f5263e00700 1 -- 172.21.15.123:0/3909223223 wait complete. 2024-03-30T04:37:10.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.483+0000 7f5263e00700 1 Processor -- start 2024-03-30T04:37:10.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.483+0000 7f5263e00700 1 -- start start 2024-03-30T04:37:10.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.483+0000 7f5263e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5264102c40 0x7f5264115a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:37:10.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.483+0000 7f5263e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52641035e0 0x7f5264115fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:37:10.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.483+0000 7f5263e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5264116610 con 0x7f52641035e0 2024-03-30T04:37:10.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.483+0000 7f5263e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f526410fa00 con 0x7f5264102c40 2024-03-30T04:37:10.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.484+0000 7f5262000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52641035e0 0x7f5264115fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:37:10.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.484+0000 7f5262000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52641035e0 0x7f5264115fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34236/0 (socket says 172.21.15.123:34236) 2024-03-30T04:37:10.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.484+0000 7f5262000700 1 -- 172.21.15.123:0/1889694048 learned_addr learned my addr 172.21.15.123:0/1889694048 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:37:10.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.484+0000 7f5262a00700 1 --2- 172.21.15.123:0/1889694048 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5264102c40 0x7f5264115a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:37:10.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.484+0000 7f5262000700 1 -- 172.21.15.123:0/1889694048 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5264102c40 msgr2=0x7f5264115a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:10.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.484+0000 7f5262000700 1 --2- 172.21.15.123:0/1889694048 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5264102c40 0x7f5264115a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:10.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.484+0000 7f5262000700 1 -- 172.21.15.123:0/1889694048 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5254005040 con 0x7f52641035e0 2024-03-30T04:37:10.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.484+0000 7f5262000700 1 --2- 172.21.15.123:0/1889694048 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52641035e0 0x7f5264115fc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f525000eb40 tx=0x7f525000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:37:10.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.485+0000 7f525aa00700 1 -- 172.21.15.123:0/1889694048 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5250007cc0 con 0x7f52641035e0 2024-03-30T04:37:10.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.485+0000 7f525aa00700 1 -- 172.21.15.123:0/1889694048 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f525000f040 con 0x7f52641035e0 2024-03-30T04:37:10.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.485+0000 7f5263e00700 1 -- 172.21.15.123:0/1889694048 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f526410fc80 con 0x7f52641035e0 2024-03-30T04:37:10.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.485+0000 7f525aa00700 1 -- 172.21.15.123:0/1889694048 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f525000c8f0 con 0x7f52641035e0 2024-03-30T04:37:10.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.485+0000 7f5263e00700 1 -- 172.21.15.123:0/1889694048 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5264110240 con 0x7f52641035e0 2024-03-30T04:37:10.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.486+0000 7f5263e00700 1 -- 172.21.15.123:0/1889694048 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52640fe390 con 0x7f52641035e0 2024-03-30T04:37:10.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.487+0000 7f525aa00700 1 -- 172.21.15.123:0/1889694048 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f525000cb60 con 0x7f52641035e0 2024-03-30T04:37:10.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.488+0000 7f525aa00700 1 --2- 172.21.15.123:0/1889694048 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f524806cc20 0x7f524806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:37:10.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.488+0000 7f525aa00700 1 -- 172.21.15.123:0/1889694048 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f525008bad0 con 0x7f52641035e0 2024-03-30T04:37:10.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.488+0000 7f5262a00700 1 --2- 172.21.15.123:0/1889694048 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f524806cc20 0x7f524806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:37:10.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.489+0000 7f5262a00700 1 --2- 172.21.15.123:0/1889694048 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f524806cc20 0x7f524806f0d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5254009fd0 tx=0x7f5254009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:37:10.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.492+0000 7f525aa00700 1 -- 172.21.15.123:0/1889694048 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f52500595d0 con 0x7f52641035e0 2024-03-30T04:37:10.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:10 smithi123 ceph-mon[139369]: pgmap v170: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 53/324 objects degraded (16.358%) 2024-03-30T04:37:10.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:10 smithi123 ceph-mon[139369]: from='client.34376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:37:10.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.775+0000 7f5263e00700 1 -- 172.21.15.123:0/1889694048 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f5264109bf0 con 0x7f52641035e0 2024-03-30T04:37:10.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.775+0000 7f525aa00700 1 -- 172.21.15.123:0/1889694048 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+740 (secure 0 0 0) 0x7f5250059160 con 0x7f52641035e0 2024-03-30T04:37:10.774 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:37:10.774 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:37:10.774 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:37:10.774 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:37:10.774 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:37:10.774 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:37:10.775 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:37:10.775 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:37:10.775 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 5 2024-03-30T04:37:10.775 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:37:10.775 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:37:10.775 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 4 2024-03-30T04:37:10.775 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:37:10.775 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:37:10.775 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 4, 2024-03-30T04:37:10.775 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 9 2024-03-30T04:37:10.775 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:37:10.775 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:37:10.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.778+0000 7f5263e00700 1 -- 172.21.15.123:0/1889694048 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f524806cc20 msgr2=0x7f524806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:10.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.778+0000 7f5263e00700 1 --2- 172.21.15.123:0/1889694048 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f524806cc20 0x7f524806f0d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5254009fd0 tx=0x7f5254009e30 comp rx=0 tx=0).stop 2024-03-30T04:37:10.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.778+0000 7f5263e00700 1 -- 172.21.15.123:0/1889694048 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52641035e0 msgr2=0x7f5264115fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:10.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.778+0000 7f5263e00700 1 --2- 172.21.15.123:0/1889694048 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52641035e0 0x7f5264115fc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f525000eb40 tx=0x7f525000b040 comp rx=0 tx=0).stop 2024-03-30T04:37:10.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.778+0000 7f5263e00700 1 -- 172.21.15.123:0/1889694048 shutdown_connections 2024-03-30T04:37:10.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.778+0000 7f5263e00700 1 --2- 172.21.15.123:0/1889694048 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f524806cc20 0x7f524806f0d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:10.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.778+0000 7f5263e00700 1 --2- 172.21.15.123:0/1889694048 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5264102c40 0x7f5264115a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:10.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.778+0000 7f5263e00700 1 --2- 172.21.15.123:0/1889694048 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52641035e0 0x7f5264115fc0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:10.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.778+0000 7f5263e00700 1 -- 172.21.15.123:0/1889694048 >> 172.21.15.123:0/1889694048 conn(0x7f52640f8830 msgr2=0x7f52640f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:37:10.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.778+0000 7f5263e00700 1 -- 172.21.15.123:0/1889694048 shutdown_connections 2024-03-30T04:37:10.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.778+0000 7f5263e00700 1 -- 172.21.15.123:0/1889694048 wait complete. 2024-03-30T04:37:10.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:10 smithi135 ceph-mon[106011]: pgmap v170: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 53/324 objects degraded (16.358%) 2024-03-30T04:37:10.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:10 smithi135 ceph-mon[106011]: from='client.34376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:37:10.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.910+0000 7fa982000700 1 -- 172.21.15.123:0/956151417 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa97c1037a0 msgr2=0x7fa97c0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:10.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.910+0000 7fa982000700 1 --2- 172.21.15.123:0/956151417 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa97c1037a0 0x7fa97c0fce10 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa968004940 tx=0x7fa968009e30 comp rx=0 tx=0).stop 2024-03-30T04:37:10.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.911+0000 7fa982000700 1 -- 172.21.15.123:0/956151417 shutdown_connections 2024-03-30T04:37:10.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.911+0000 7fa982000700 1 --2- 172.21.15.123:0/956151417 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa97c1037a0 0x7fa97c0fce10 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:10.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.911+0000 7fa982000700 1 --2- 172.21.15.123:0/956151417 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa97c102e00 0x7fa97c1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:10.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.911+0000 7fa982000700 1 -- 172.21.15.123:0/956151417 >> 172.21.15.123:0/956151417 conn(0x7fa97c0f89b0 msgr2=0x7fa97c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:37:10.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.911+0000 7fa982000700 1 -- 172.21.15.123:0/956151417 shutdown_connections 2024-03-30T04:37:10.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.911+0000 7fa982000700 1 -- 172.21.15.123:0/956151417 wait complete. 2024-03-30T04:37:10.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.911+0000 7fa982000700 1 Processor -- start 2024-03-30T04:37:10.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.912+0000 7fa982000700 1 -- start start 2024-03-30T04:37:10.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.912+0000 7fa982000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa97c102e00 0x7fa97c115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:37:10.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.912+0000 7fa982000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa97c1037a0 0x7fa97c1161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:37:10.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.912+0000 7fa982000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa97c116810 con 0x7fa97c102e00 2024-03-30T04:37:10.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.912+0000 7fa982000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa97c10fbe0 con 0x7fa97c1037a0 2024-03-30T04:37:10.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.912+0000 7fa97b400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa97c102e00 0x7fa97c115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:37:10.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.912+0000 7fa97b400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa97c102e00 0x7fa97c115c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34262/0 (socket says 172.21.15.123:34262) 2024-03-30T04:37:10.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.912+0000 7fa97aa00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa97c1037a0 0x7fa97c1161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:37:10.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.912+0000 7fa97b400700 1 -- 172.21.15.123:0/1999410486 learned_addr learned my addr 172.21.15.123:0/1999410486 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:37:10.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.913+0000 7fa97b400700 1 -- 172.21.15.123:0/1999410486 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa97c1037a0 msgr2=0x7fa97c1161c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:10.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.913+0000 7fa97b400700 1 --2- 172.21.15.123:0/1999410486 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa97c1037a0 0x7fa97c1161c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:10.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.913+0000 7fa97b400700 1 -- 172.21.15.123:0/1999410486 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa968005040 con 0x7fa97c102e00 2024-03-30T04:37:10.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.913+0000 7fa97b400700 1 --2- 172.21.15.123:0/1999410486 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa97c102e00 0x7fa97c115c80 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa96c002a80 tx=0x7fa96c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:37:10.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.913+0000 7fa973400700 1 -- 172.21.15.123:0/1999410486 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa96c007cc0 con 0x7fa97c102e00 2024-03-30T04:37:10.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.913+0000 7fa973400700 1 -- 172.21.15.123:0/1999410486 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa96c00f040 con 0x7fa97c102e00 2024-03-30T04:37:10.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.913+0000 7fa982000700 1 -- 172.21.15.123:0/1999410486 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa97c10fec0 con 0x7fa97c102e00 2024-03-30T04:37:10.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.913+0000 7fa973400700 1 -- 172.21.15.123:0/1999410486 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa96c00da60 con 0x7fa97c102e00 2024-03-30T04:37:10.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.913+0000 7fa982000700 1 -- 172.21.15.123:0/1999410486 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa97c110480 con 0x7fa97c102e00 2024-03-30T04:37:10.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.914+0000 7fa982000700 1 -- 172.21.15.123:0/1999410486 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa97c0fe550 con 0x7fa97c102e00 2024-03-30T04:37:10.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.915+0000 7fa973400700 1 -- 172.21.15.123:0/1999410486 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fa96c00dbc0 con 0x7fa97c102e00 2024-03-30T04:37:10.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.916+0000 7fa973400700 1 --2- 172.21.15.123:0/1999410486 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa96006cc30 0x7fa96006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:37:10.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.916+0000 7fa973400700 1 -- 172.21.15.123:0/1999410486 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fa96c08bb50 con 0x7fa97c102e00 2024-03-30T04:37:10.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.916+0000 7fa97aa00700 1 --2- 172.21.15.123:0/1999410486 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa96006cc30 0x7fa96006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:37:10.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.917+0000 7fa97aa00700 1 --2- 172.21.15.123:0/1999410486 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa96006cc30 0x7fa96006f0e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa97c111520 tx=0x7fa968009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:37:10.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:10.921+0000 7fa973400700 1 -- 172.21.15.123:0/1999410486 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa96c0596c0 con 0x7fa97c102e00 2024-03-30T04:37:11.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.172+0000 7fa982000700 1 -- 172.21.15.123:0/1999410486 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fa97c04bed0 con 0x7fa97c102e00 2024-03-30T04:37:11.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.172+0000 7fa973400700 1 -- 172.21.15.123:0/1999410486 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 23 v23) v1 ==== 76+0+1876 (secure 0 0 0) 0x7fa96c059250 con 0x7fa97c102e00 2024-03-30T04:37:11.171 INFO:teuthology.orchestra.run.smithi123.stdout:e23 2024-03-30T04:37:11.171 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:37:11.171 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:37:11.171 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:37:11.171 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:37:11.171 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:37:11.171 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:37:11.171 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 23 2024-03-30T04:37:11.171 INFO:teuthology.orchestra.run.smithi123.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-30T04:37:11.171 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:37:11.171 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:32:48.448604+0000 2024-03-30T04:37:11.171 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:37:11.171 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:37:11.172 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:37:11.172 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:37:11.172 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:37:11.172 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T04:37:11.172 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:37:11.172 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:37:11.172 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 46 2024-03-30T04:37:11.172 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:37:11.172 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T04:37:11.172 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T04:37:11.172 INFO:teuthology.orchestra.run.smithi123.stdout:up {0=14488,1=24309} 2024-03-30T04:37:11.172 INFO:teuthology.orchestra.run.smithi123.stdout:failed 2024-03-30T04:37:11.172 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:37:11.172 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:37:11.172 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:37:11.173 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:37:11.173 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:37:11.173 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:37:11.173 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T04:37:11.173 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T04:37:11.173 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{0:14488} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.123:6828/3037849622,v1:172.21.15.123:6829/3037849622] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:37:11.173 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{0:14500} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/2433675995,v1:172.21.15.123:6827/2433675995] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:37:11.173 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{1:24309} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.135:6824/4050541062,v1:172.21.15.135:6825/4050541062] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:37:11.173 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{1:24319} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.135:6826/1096986122,v1:172.21.15.135:6827/1096986122] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:37:11.173 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:37:11.173 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:37:11.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.174+0000 7fa982000700 1 -- 172.21.15.123:0/1999410486 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa96006cc30 msgr2=0x7fa96006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:11.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.174+0000 7fa982000700 1 --2- 172.21.15.123:0/1999410486 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa96006cc30 0x7fa96006f0e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa97c111520 tx=0x7fa968009e30 comp rx=0 tx=0).stop 2024-03-30T04:37:11.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.174+0000 7fa982000700 1 -- 172.21.15.123:0/1999410486 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa97c102e00 msgr2=0x7fa97c115c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:11.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.174+0000 7fa982000700 1 --2- 172.21.15.123:0/1999410486 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa97c102e00 0x7fa97c115c80 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa96c002a80 tx=0x7fa96c0046b0 comp rx=0 tx=0).stop 2024-03-30T04:37:11.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.174+0000 7fa982000700 1 -- 172.21.15.123:0/1999410486 shutdown_connections 2024-03-30T04:37:11.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.174+0000 7fa982000700 1 --2- 172.21.15.123:0/1999410486 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa96006cc30 0x7fa96006f0e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:11.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.174+0000 7fa982000700 1 --2- 172.21.15.123:0/1999410486 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa97c102e00 0x7fa97c115c80 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:11.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.174+0000 7fa982000700 1 --2- 172.21.15.123:0/1999410486 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa97c1037a0 0x7fa97c1161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:11.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.174+0000 7fa982000700 1 -- 172.21.15.123:0/1999410486 >> 172.21.15.123:0/1999410486 conn(0x7fa97c0f89b0 msgr2=0x7fa97c0f99a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:37:11.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.175+0000 7fa982000700 1 -- 172.21.15.123:0/1999410486 shutdown_connections 2024-03-30T04:37:11.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.175+0000 7fa982000700 1 -- 172.21.15.123:0/1999410486 wait complete. 2024-03-30T04:37:11.175 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 23 2024-03-30T04:37:11.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.316+0000 7f3689200700 1 -- 172.21.15.123:0/122674301 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f36840708a0 msgr2=0x7f3684070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:11.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.316+0000 7f3689200700 1 --2- 172.21.15.123:0/122674301 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f36840708a0 0x7f3684070d10 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3670004970 tx=0x7f3670009e30 comp rx=0 tx=0).stop 2024-03-30T04:37:11.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.316+0000 7f3689200700 1 -- 172.21.15.123:0/122674301 shutdown_connections 2024-03-30T04:37:11.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.316+0000 7f3689200700 1 --2- 172.21.15.123:0/122674301 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f36840708a0 0x7f3684070d10 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:11.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.316+0000 7f3689200700 1 --2- 172.21.15.123:0/122674301 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3684071f90 0x7f3684070360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:11.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.316+0000 7f3689200700 1 -- 172.21.15.123:0/122674301 >> 172.21.15.123:0/122674301 conn(0x7f36840f89b0 msgr2=0x7f36840fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:37:11.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.316+0000 7f3689200700 1 -- 172.21.15.123:0/122674301 shutdown_connections 2024-03-30T04:37:11.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.316+0000 7f3689200700 1 -- 172.21.15.123:0/122674301 wait complete. 2024-03-30T04:37:11.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.317+0000 7f3689200700 1 Processor -- start 2024-03-30T04:37:11.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.317+0000 7f3689200700 1 -- start start 2024-03-30T04:37:11.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.317+0000 7f3689200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f36840708a0 0x7f3684119df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:37:11.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.317+0000 7f3689200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3684071f90 0x7f368411a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:37:11.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.317+0000 7f3689200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f368411a980 con 0x7f36840708a0 2024-03-30T04:37:11.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.317+0000 7f3689200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3684114010 con 0x7f3684071f90 2024-03-30T04:37:11.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.318+0000 7f3681e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f36840708a0 0x7f3684119df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:37:11.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.318+0000 7f3681e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f36840708a0 0x7f3684119df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34284/0 (socket says 172.21.15.123:34284) 2024-03-30T04:37:11.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.318+0000 7f3681e00700 1 -- 172.21.15.123:0/330027393 learned_addr learned my addr 172.21.15.123:0/330027393 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:37:11.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.318+0000 7f3681400700 1 --2- 172.21.15.123:0/330027393 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3684071f90 0x7f368411a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:37:11.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.318+0000 7f3681e00700 1 -- 172.21.15.123:0/330027393 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3684071f90 msgr2=0x7f368411a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:11.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.318+0000 7f3681e00700 1 --2- 172.21.15.123:0/330027393 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3684071f90 0x7f368411a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:11.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.318+0000 7f3681e00700 1 -- 172.21.15.123:0/330027393 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3674005040 con 0x7f36840708a0 2024-03-30T04:37:11.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.318+0000 7f3681e00700 1 --2- 172.21.15.123:0/330027393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f36840708a0 0x7f3684119df0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f36740096d0 tx=0x7f3674009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:37:11.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.319+0000 7f367a000700 1 -- 172.21.15.123:0/330027393 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f367400e840 con 0x7f36840708a0 2024-03-30T04:37:11.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.319+0000 7f367a000700 1 -- 172.21.15.123:0/330027393 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f367400f040 con 0x7f36840708a0 2024-03-30T04:37:11.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.319+0000 7f367a000700 1 -- 172.21.15.123:0/330027393 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3674013d30 con 0x7f36840708a0 2024-03-30T04:37:11.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.319+0000 7f3689200700 1 -- 172.21.15.123:0/330027393 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3670005040 con 0x7f36840708a0 2024-03-30T04:37:11.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.319+0000 7f3689200700 1 -- 172.21.15.123:0/330027393 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3684114640 con 0x7f36840708a0 2024-03-30T04:37:11.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.320+0000 7f3689200700 1 -- 172.21.15.123:0/330027393 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3684071aa0 con 0x7f36840708a0 2024-03-30T04:37:11.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.321+0000 7f367a000700 1 -- 172.21.15.123:0/330027393 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f367400e9a0 con 0x7f36840708a0 2024-03-30T04:37:11.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.322+0000 7f367a000700 1 --2- 172.21.15.123:0/330027393 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f366806cc20 0x7f366806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:37:11.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.322+0000 7f367a000700 1 -- 172.21.15.123:0/330027393 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f367408bd50 con 0x7f36840708a0 2024-03-30T04:37:11.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.322+0000 7f3681400700 1 --2- 172.21.15.123:0/330027393 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f366806cc20 0x7f366806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:37:11.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.322+0000 7f3681400700 1 --2- 172.21.15.123:0/330027393 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f366806cc20 0x7f366806f0d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f368410cfa0 tx=0x7f3670009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:37:11.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.326+0000 7f367a000700 1 -- 172.21.15.123:0/330027393 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3674055d90 con 0x7f36840708a0 2024-03-30T04:37:11.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.552+0000 7f3689200700 1 -- 172.21.15.123:0/330027393 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f36841103e0 con 0x7f366806cc20 2024-03-30T04:37:11.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.553+0000 7f367a000700 1 -- 172.21.15.123:0/330027393 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f36841103e0 con 0x7f366806cc20 2024-03-30T04:37:11.552 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:37:11.552 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:37:11.552 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:37:11.552 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T04:37:11.552 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:37:11.552 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T04:37:11.552 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T04:37:11.552 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T04:37:11.552 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T04:37:11.552 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:37:11.552 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "12/21 daemons upgraded", 2024-03-30T04:37:11.552 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading osd daemons", 2024-03-30T04:37:11.552 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:37:11.553 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:37:11.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.556+0000 7f3689200700 1 -- 172.21.15.123:0/330027393 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f366806cc20 msgr2=0x7f366806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:11.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.556+0000 7f3689200700 1 --2- 172.21.15.123:0/330027393 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f366806cc20 0x7f366806f0d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f368410cfa0 tx=0x7f3670009e30 comp rx=0 tx=0).stop 2024-03-30T04:37:11.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.556+0000 7f3689200700 1 -- 172.21.15.123:0/330027393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f36840708a0 msgr2=0x7f3684119df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:11.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.556+0000 7f3689200700 1 --2- 172.21.15.123:0/330027393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f36840708a0 0x7f3684119df0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f36740096d0 tx=0x7f3674009e30 comp rx=0 tx=0).stop 2024-03-30T04:37:11.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.556+0000 7f3689200700 1 -- 172.21.15.123:0/330027393 shutdown_connections 2024-03-30T04:37:11.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.556+0000 7f3689200700 1 --2- 172.21.15.123:0/330027393 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f366806cc20 0x7f366806f0d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:11.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.556+0000 7f3689200700 1 --2- 172.21.15.123:0/330027393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f36840708a0 0x7f3684119df0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:11.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.556+0000 7f3689200700 1 --2- 172.21.15.123:0/330027393 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3684071f90 0x7f368411a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:11.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.556+0000 7f3689200700 1 -- 172.21.15.123:0/330027393 >> 172.21.15.123:0/330027393 conn(0x7f36840f89b0 msgr2=0x7f36840f9b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:37:11.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.556+0000 7f3689200700 1 -- 172.21.15.123:0/330027393 shutdown_connections 2024-03-30T04:37:11.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.556+0000 7f3689200700 1 -- 172.21.15.123:0/330027393 wait complete. 2024-03-30T04:37:11.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.689+0000 7f49b5400700 1 -- 172.21.15.123:0/1487550078 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49b0102e00 msgr2=0x7f49b01031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:11.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.689+0000 7f49b5400700 1 --2- 172.21.15.123:0/1487550078 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49b0102e00 0x7f49b01031d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f49a0004940 tx=0x7f49a0009e30 comp rx=0 tx=0).stop 2024-03-30T04:37:11.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.689+0000 7f49b5400700 1 -- 172.21.15.123:0/1487550078 shutdown_connections 2024-03-30T04:37:11.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.689+0000 7f49b5400700 1 --2- 172.21.15.123:0/1487550078 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49b01037a0 0x7f49b00fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:11.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.689+0000 7f49b5400700 1 --2- 172.21.15.123:0/1487550078 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49b0102e00 0x7f49b01031d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:11.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.689+0000 7f49b5400700 1 -- 172.21.15.123:0/1487550078 >> 172.21.15.123:0/1487550078 conn(0x7f49b00f89b0 msgr2=0x7f49b00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:37:11.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.689+0000 7f49b5400700 1 -- 172.21.15.123:0/1487550078 shutdown_connections 2024-03-30T04:37:11.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.689+0000 7f49b5400700 1 -- 172.21.15.123:0/1487550078 wait complete. 2024-03-30T04:37:11.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.690+0000 7f49b5400700 1 Processor -- start 2024-03-30T04:37:11.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.690+0000 7f49b5400700 1 -- start start 2024-03-30T04:37:11.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.690+0000 7f49b5400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49b0102e00 0x7f49b00702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:37:11.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.690+0000 7f49b5400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49b01037a0 0x7f49b0070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:37:11.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.690+0000 7f49b5400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49b0070d90 con 0x7f49b0102e00 2024-03-30T04:37:11.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.690+0000 7f49b5400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49b0070ed0 con 0x7f49b01037a0 2024-03-30T04:37:11.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.690+0000 7f49ae800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49b0102e00 0x7f49b00702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:37:11.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.690+0000 7f49ae800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49b0102e00 0x7f49b00702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34296/0 (socket says 172.21.15.123:34296) 2024-03-30T04:37:11.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.690+0000 7f49ae800700 1 -- 172.21.15.123:0/235311635 learned_addr learned my addr 172.21.15.123:0/235311635 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:37:11.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.690+0000 7f49ade00700 1 --2- 172.21.15.123:0/235311635 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49b01037a0 0x7f49b0070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:37:11.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.691+0000 7f49ae800700 1 -- 172.21.15.123:0/235311635 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49b01037a0 msgr2=0x7f49b0070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:11.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.691+0000 7f49ae800700 1 --2- 172.21.15.123:0/235311635 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49b01037a0 0x7f49b0070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:11.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.691+0000 7f49ae800700 1 -- 172.21.15.123:0/235311635 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a0005040 con 0x7f49b0102e00 2024-03-30T04:37:11.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.691+0000 7f49ae800700 1 --2- 172.21.15.123:0/235311635 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49b0102e00 0x7f49b00702c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f49a0009360 tx=0x7f49a0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:37:11.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.691+0000 7f49a6a00700 1 -- 172.21.15.123:0/235311635 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f49a000e040 con 0x7f49b0102e00 2024-03-30T04:37:11.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.691+0000 7f49a6a00700 1 -- 172.21.15.123:0/235311635 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f49a0013420 con 0x7f49b0102e00 2024-03-30T04:37:11.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.691+0000 7f49a6a00700 1 -- 172.21.15.123:0/235311635 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f49a001c5d0 con 0x7f49b0102e00 2024-03-30T04:37:11.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.691+0000 7f49b5400700 1 -- 172.21.15.123:0/235311635 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49b0071120 con 0x7f49b0102e00 2024-03-30T04:37:11.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.691+0000 7f49b5400700 1 -- 172.21.15.123:0/235311635 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49b011a250 con 0x7f49b0102e00 2024-03-30T04:37:11.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.692+0000 7f49b5400700 1 -- 172.21.15.123:0/235311635 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49b00fe550 con 0x7f49b0102e00 2024-03-30T04:37:11.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.693+0000 7f49a6a00700 1 -- 172.21.15.123:0/235311635 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f49a0013590 con 0x7f49b0102e00 2024-03-30T04:37:11.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.694+0000 7f49a6a00700 1 --2- 172.21.15.123:0/235311635 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f499406cc30 0x7f499406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:37:11.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.694+0000 7f49a6a00700 1 -- 172.21.15.123:0/235311635 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f49a008d1d0 con 0x7f49b0102e00 2024-03-30T04:37:11.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.694+0000 7f49ade00700 1 --2- 172.21.15.123:0/235311635 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f499406cc30 0x7f499406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:37:11.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.695+0000 7f49ade00700 1 --2- 172.21.15.123:0/235311635 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f499406cc30 0x7f499406f0e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f49b0071c00 tx=0x7f499c009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:37:11.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.698+0000 7f49a6a00700 1 -- 172.21.15.123:0/235311635 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f49a0057190 con 0x7f49b0102e00 2024-03-30T04:37:11.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:11 smithi123 ceph-mon[139369]: from='client.34380 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:37:11.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:11 smithi123 ceph-mon[139369]: from='client.34384 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:37:11.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:11 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/1889694048' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:37:11.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:11 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/1999410486' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:37:11.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:11 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:11.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:11 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:11.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:11 smithi135 ceph-mon[106011]: from='client.34380 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:37:11.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:11 smithi135 ceph-mon[106011]: from='client.34384 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:37:11.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:11 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/1889694048' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:37:11.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:11 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/1999410486' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:37:11.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:11 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:11.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:11 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:11.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.986+0000 7f49b5400700 1 -- 172.21.15.123:0/235311635 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f49b005f5f0 con 0x7f49b0102e00 2024-03-30T04:37:11.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.986+0000 7f49a6a00700 1 -- 172.21.15.123:0/235311635 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+1053 (secure 0 0 0) 0x7f49a005a7b0 con 0x7f49b0102e00 2024-03-30T04:37:11.985 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 53/324 objects degraded (16.358%), 14 pgs degraded 2024-03-30T04:37:11.985 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] OSD_DOWN: 1 osds down 2024-03-30T04:37:11.985 INFO:teuthology.orchestra.run.smithi123.stdout: osd.5 (root=default,host=smithi135) is down 2024-03-30T04:37:11.985 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 53/324 objects degraded (16.358%), 14 pgs degraded 2024-03-30T04:37:11.985 INFO:teuthology.orchestra.run.smithi123.stdout: pg 2.2 is active+undersized+degraded, acting [1,0] 2024-03-30T04:37:11.985 INFO:teuthology.orchestra.run.smithi123.stdout: pg 2.3 is active+undersized+degraded, acting [2,1] 2024-03-30T04:37:11.986 INFO:teuthology.orchestra.run.smithi123.stdout: pg 2.8 is active+undersized+degraded, acting [3,0] 2024-03-30T04:37:11.986 INFO:teuthology.orchestra.run.smithi123.stdout: pg 2.b is active+undersized+degraded, acting [3,4] 2024-03-30T04:37:11.986 INFO:teuthology.orchestra.run.smithi123.stdout: pg 2.f is active+undersized+degraded, acting [4,0] 2024-03-30T04:37:11.986 INFO:teuthology.orchestra.run.smithi123.stdout: pg 2.14 is active+undersized+degraded, acting [3,4] 2024-03-30T04:37:11.986 INFO:teuthology.orchestra.run.smithi123.stdout: pg 2.16 is active+undersized+degraded, acting [3,2] 2024-03-30T04:37:11.986 INFO:teuthology.orchestra.run.smithi123.stdout: pg 2.17 is active+undersized+degraded, acting [2,1] 2024-03-30T04:37:11.986 INFO:teuthology.orchestra.run.smithi123.stdout: pg 2.18 is active+undersized+degraded, acting [4,3] 2024-03-30T04:37:11.986 INFO:teuthology.orchestra.run.smithi123.stdout: pg 2.1a is active+undersized+degraded, acting [3,4] 2024-03-30T04:37:11.986 INFO:teuthology.orchestra.run.smithi123.stdout: pg 2.1b is active+undersized+degraded, acting [1,0] 2024-03-30T04:37:11.986 INFO:teuthology.orchestra.run.smithi123.stdout: pg 2.1c is active+undersized+degraded, acting [4,2] 2024-03-30T04:37:11.986 INFO:teuthology.orchestra.run.smithi123.stdout: pg 2.1d is active+undersized+degraded, acting [3,0] 2024-03-30T04:37:11.986 INFO:teuthology.orchestra.run.smithi123.stdout: pg 2.1e is active+undersized+degraded, acting [2,0] 2024-03-30T04:37:11.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.989+0000 7f49b5400700 1 -- 172.21.15.123:0/235311635 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f499406cc30 msgr2=0x7f499406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:11.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.989+0000 7f49b5400700 1 --2- 172.21.15.123:0/235311635 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f499406cc30 0x7f499406f0e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f49b0071c00 tx=0x7f499c009040 comp rx=0 tx=0).stop 2024-03-30T04:37:11.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.989+0000 7f49b5400700 1 -- 172.21.15.123:0/235311635 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49b0102e00 msgr2=0x7f49b00702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:11.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.989+0000 7f49b5400700 1 --2- 172.21.15.123:0/235311635 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49b0102e00 0x7f49b00702c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f49a0009360 tx=0x7f49a0009e30 comp rx=0 tx=0).stop 2024-03-30T04:37:11.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.989+0000 7f49b5400700 1 -- 172.21.15.123:0/235311635 shutdown_connections 2024-03-30T04:37:11.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.989+0000 7f49b5400700 1 --2- 172.21.15.123:0/235311635 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f499406cc30 0x7f499406f0e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:11.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.989+0000 7f49b5400700 1 --2- 172.21.15.123:0/235311635 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49b0102e00 0x7f49b00702c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:11.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.989+0000 7f49b5400700 1 --2- 172.21.15.123:0/235311635 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49b01037a0 0x7f49b0070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:11.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.989+0000 7f49b5400700 1 -- 172.21.15.123:0/235311635 >> 172.21.15.123:0/235311635 conn(0x7f49b00f89b0 msgr2=0x7f49b00f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:37:11.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.989+0000 7f49b5400700 1 -- 172.21.15.123:0/235311635 shutdown_connections 2024-03-30T04:37:11.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:11.989+0000 7f49b5400700 1 -- 172.21.15.123:0/235311635 wait complete. 2024-03-30T04:37:12.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:12 smithi123 ceph-mon[139369]: pgmap v171: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 53/324 objects degraded (16.358%) 2024-03-30T04:37:12.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:12 smithi123 ceph-mon[139369]: from='client.34396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:37:12.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:12 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/235311635' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:37:12.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:12 smithi135 ceph-mon[106011]: pgmap v171: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 53/324 objects degraded (16.358%) 2024-03-30T04:37:12.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:12 smithi135 ceph-mon[106011]: from='client.34396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:37:12.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:12 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/235311635' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:37:13.492 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 04:37:13 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-5[131567]: 2024-03-30T04:37:13.173+0000 7fadde5b7640 -1 osd.5 74 log_to_monitors true 2024-03-30T04:37:13.850 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 04:37:13 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-5[131567]: 2024-03-30T04:37:13.529+0000 7fadcda00700 -1 osd.5 74 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T04:37:13.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:13 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:13.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:13 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:13.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:13 smithi135 ceph-mon[106011]: from='osd.5 [v2:172.21.15.135:6816/1977885887,v1:172.21.15.135:6817/1977885887]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-30T04:37:13.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:13 smithi135 ceph-mon[106011]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-30T04:37:13.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:13 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:13.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:13 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:13.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:13 smithi123 ceph-mon[139369]: from='osd.5 [v2:172.21.15.135:6816/1977885887,v1:172.21.15.135:6817/1977885887]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-30T04:37:13.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:13 smithi123 ceph-mon[139369]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-30T04:37:14.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:14 smithi135 ceph-mon[106011]: pgmap v172: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 53/324 objects degraded (16.358%) 2024-03-30T04:37:14.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:14 smithi135 ceph-mon[106011]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-30T04:37:14.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:14 smithi135 ceph-mon[106011]: osdmap e77: 6 total, 5 up, 6 in 2024-03-30T04:37:14.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:14 smithi135 ceph-mon[106011]: from='osd.5 [v2:172.21.15.135:6816/1977885887,v1:172.21.15.135:6817/1977885887]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-03-30T04:37:14.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:14 smithi135 ceph-mon[106011]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-03-30T04:37:14.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:14 smithi135 ceph-mon[106011]: Health check update: Degraded data redundancy: 53/324 objects degraded (16.358%), 14 pgs degraded (PG_DEGRADED) 2024-03-30T04:37:14.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:14 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:14.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:14 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:37:14.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:14 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:14.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:14 smithi123 ceph-mon[139369]: pgmap v172: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 53/324 objects degraded (16.358%) 2024-03-30T04:37:14.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:14 smithi123 ceph-mon[139369]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-30T04:37:14.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:14 smithi123 ceph-mon[139369]: osdmap e77: 6 total, 5 up, 6 in 2024-03-30T04:37:14.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:14 smithi123 ceph-mon[139369]: from='osd.5 [v2:172.21.15.135:6816/1977885887,v1:172.21.15.135:6817/1977885887]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-03-30T04:37:14.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:14 smithi123 ceph-mon[139369]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-03-30T04:37:14.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:14 smithi123 ceph-mon[139369]: Health check update: Degraded data redundancy: 53/324 objects degraded (16.358%), 14 pgs degraded (PG_DEGRADED) 2024-03-30T04:37:14.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:14 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:14.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:14 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:37:14.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:14 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:15.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:15 smithi135 ceph-mon[106011]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-30T04:37:15.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:15 smithi135 ceph-mon[106011]: osd.5 [v2:172.21.15.135:6816/1977885887,v1:172.21.15.135:6817/1977885887] boot 2024-03-30T04:37:15.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:15 smithi135 ceph-mon[106011]: osdmap e78: 6 total, 6 up, 6 in 2024-03-30T04:37:15.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:15 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T04:37:15.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:15 smithi123 ceph-mon[139369]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-30T04:37:15.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:15 smithi123 ceph-mon[139369]: osd.5 [v2:172.21.15.135:6816/1977885887,v1:172.21.15.135:6817/1977885887] boot 2024-03-30T04:37:15.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:15 smithi123 ceph-mon[139369]: osdmap e78: 6 total, 6 up, 6 in 2024-03-30T04:37:15.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:15 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T04:37:16.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:16 smithi135 ceph-mon[106011]: pgmap v175: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 259 MiB data, 992 MiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s; 53/324 objects degraded (16.358%) 2024-03-30T04:37:16.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:16 smithi135 ceph-mon[106011]: osdmap e79: 6 total, 6 up, 6 in 2024-03-30T04:37:16.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:16 smithi123 ceph-mon[139369]: pgmap v175: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 259 MiB data, 992 MiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s; 53/324 objects degraded (16.358%) 2024-03-30T04:37:16.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:16 smithi123 ceph-mon[139369]: osdmap e79: 6 total, 6 up, 6 in 2024-03-30T04:37:18.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:17 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:18.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:17 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:18.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:17 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:37:18.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:17 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:37:18.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:17 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:18.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:17 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:37:18.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:17 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:37:18.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:17 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:37:18.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:17 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:37:18.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:17 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:37:18.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:17 smithi123 ceph-mon[139369]: Upgrade: Setting container_image for all osd 2024-03-30T04:37:18.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:17 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:18.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:17 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-03-30T04:37:18.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:17 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-03-30T04:37:18.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:17 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-03-30T04:37:18.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:17 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-03-30T04:37:18.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:17 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-03-30T04:37:18.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:17 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-03-30T04:37:18.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:17 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-03-30T04:37:18.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:17 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-03-30T04:37:18.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:17 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-03-30T04:37:18.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:17 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-03-30T04:37:18.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:17 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-03-30T04:37:18.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:17 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-03-30T04:37:18.219 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:17 smithi123 ceph-mon[139369]: Upgrade: Setting require_osd_release to 18 reef 2024-03-30T04:37:18.219 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:17 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd require-osd-release", "release": "reef"}]: dispatch 2024-03-30T04:37:18.219 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:17 smithi123 ceph-mon[139369]: pgmap v177: 65 pgs: 12 active+undersized+degraded, 12 active+undersized, 41 active+clean; 259 MiB data, 993 MiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 2 op/s; 49/324 objects degraded (15.123%) 2024-03-30T04:37:18.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:17 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:18.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:17 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:18.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:17 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:37:18.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:17 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:37:18.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:17 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:18.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:17 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:37:18.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:17 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:37:18.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:17 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:37:18.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:17 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:37:18.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:17 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:37:18.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:17 smithi135 ceph-mon[106011]: Upgrade: Setting container_image for all osd 2024-03-30T04:37:18.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:17 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:18.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:17 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-03-30T04:37:18.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:17 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-03-30T04:37:18.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:17 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-03-30T04:37:18.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:17 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-03-30T04:37:18.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:17 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-03-30T04:37:18.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:17 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-03-30T04:37:18.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:17 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-03-30T04:37:18.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:17 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-03-30T04:37:18.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:17 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-03-30T04:37:18.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:17 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-03-30T04:37:18.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:17 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-03-30T04:37:18.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:17 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-03-30T04:37:18.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:17 smithi135 ceph-mon[106011]: Upgrade: Setting require_osd_release to 18 reef 2024-03-30T04:37:18.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:17 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd require-osd-release", "release": "reef"}]: dispatch 2024-03-30T04:37:18.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:17 smithi135 ceph-mon[106011]: pgmap v177: 65 pgs: 12 active+undersized+degraded, 12 active+undersized, 41 active+clean; 259 MiB data, 993 MiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 2 op/s; 49/324 objects degraded (15.123%) 2024-03-30T04:37:19.514 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:19 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "osd require-osd-release", "release": "reef"}]': finished 2024-03-30T04:37:19.514 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:19 smithi123 ceph-mon[139369]: osdmap e80: 6 total, 6 up, 6 in 2024-03-30T04:37:19.514 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:19 smithi123 ceph-mon[139369]: Upgrade: Disabling standby-replay for filesystem cephfs 2024-03-30T04:37:19.515 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:19 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:19.515 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:19 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "0"}]: dispatch 2024-03-30T04:37:19.515 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:19 smithi123 ceph-mon[139369]: osdmap e81: 6 total, 6 up, 6 in 2024-03-30T04:37:19.593 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:19 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "osd require-osd-release", "release": "reef"}]': finished 2024-03-30T04:37:19.593 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:19 smithi135 ceph-mon[106011]: osdmap e80: 6 total, 6 up, 6 in 2024-03-30T04:37:19.593 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:19 smithi135 ceph-mon[106011]: Upgrade: Disabling standby-replay for filesystem cephfs 2024-03-30T04:37:19.593 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:19 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:19.593 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:19 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "0"}]: dispatch 2024-03-30T04:37:19.593 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:19 smithi135 ceph-mon[106011]: osdmap e81: 6 total, 6 up, 6 in 2024-03-30T04:37:19.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:19 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-mon-smithi123[139346]: 2024-03-30T04:37:19.515+0000 7f50e4c00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-30T04:37:20.548 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:20 smithi123 ceph-mon[139369]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-03-30T04:37:20.548 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:20 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "0"}]': finished 2024-03-30T04:37:20.548 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:20 smithi123 ceph-mon[139369]: fsmap cephfs:2 {0=cephfs.smithi123.dvtind=up:active,1=cephfs.smithi135.zvnlap=up:active} 2024-03-30T04:37:20.548 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:20 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:37:20.548 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:20 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:37:20.548 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:20 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:37:20.548 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:20 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:20.549 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:20 smithi123 ceph-mon[139369]: Health check update: Degraded data redundancy: 49/324 objects degraded (15.123%), 12 pgs degraded (PG_DEGRADED) 2024-03-30T04:37:20.549 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:20 smithi123 ceph-mon[139369]: pgmap v180: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:37:20.549 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:20 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:37:20.549 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:20 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:37:20.549 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:20 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:37:20.549 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:20 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:37:20.549 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:20 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:37:20.549 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:20 smithi123 ceph-mon[139369]: Upgrade: failing fs cephfs for rapid multi-rank mds upgrade 2024-03-30T04:37:20.549 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:20 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-30T04:37:20.549 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:20 smithi123 ceph-mon[139369]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-30T04:37:20.549 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:20 smithi123 ceph-mon[139369]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-30T04:37:20.549 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:20 smithi123 ceph-mon[139369]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-03-30T04:37:20.549 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:20 smithi123 ceph-mon[139369]: osdmap e82: 6 total, 6 up, 6 in 2024-03-30T04:37:20.549 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:20 smithi123 ceph-mon[139369]: mds.? [v2:172.21.15.135:6826/1741352336,v1:172.21.15.135:6827/1741352336] up:boot 2024-03-30T04:37:20.549 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:20 smithi123 ceph-mon[139369]: mds.? [v2:172.21.15.123:6826/2891461179,v1:172.21.15.123:6827/2891461179] up:boot 2024-03-30T04:37:20.550 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:20 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-30T04:37:20.550 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:20 smithi123 ceph-mon[139369]: fsmap cephfs:0/2 2 up:standby, 2 failed 2024-03-30T04:37:20.550 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:20 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi123.tqxvzr"}]: dispatch 2024-03-30T04:37:20.550 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:20 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi135.xwgjvb"}]: dispatch 2024-03-30T04:37:20.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:20 smithi135 ceph-mon[106011]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-03-30T04:37:20.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:20 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "0"}]': finished 2024-03-30T04:37:20.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:20 smithi135 ceph-mon[106011]: fsmap cephfs:2 {0=cephfs.smithi123.dvtind=up:active,1=cephfs.smithi135.zvnlap=up:active} 2024-03-30T04:37:20.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:20 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:37:20.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:20 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:37:20.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:20 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:37:20.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:20 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:20.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:20 smithi135 ceph-mon[106011]: Health check update: Degraded data redundancy: 49/324 objects degraded (15.123%), 12 pgs degraded (PG_DEGRADED) 2024-03-30T04:37:20.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:20 smithi135 ceph-mon[106011]: pgmap v180: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:37:20.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:20 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:37:20.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:20 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:37:20.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:20 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:37:20.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:20 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:37:20.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:20 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:37:20.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:20 smithi135 ceph-mon[106011]: Upgrade: failing fs cephfs for rapid multi-rank mds upgrade 2024-03-30T04:37:20.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:20 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-30T04:37:20.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:20 smithi135 ceph-mon[106011]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-30T04:37:20.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:20 smithi135 ceph-mon[106011]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-30T04:37:20.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:20 smithi135 ceph-mon[106011]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-03-30T04:37:20.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:20 smithi135 ceph-mon[106011]: osdmap e82: 6 total, 6 up, 6 in 2024-03-30T04:37:20.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:20 smithi135 ceph-mon[106011]: mds.? [v2:172.21.15.135:6826/1741352336,v1:172.21.15.135:6827/1741352336] up:boot 2024-03-30T04:37:20.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:20 smithi135 ceph-mon[106011]: mds.? [v2:172.21.15.123:6826/2891461179,v1:172.21.15.123:6827/2891461179] up:boot 2024-03-30T04:37:20.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:20 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-30T04:37:20.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:20 smithi135 ceph-mon[106011]: fsmap cephfs:0/2 2 up:standby, 2 failed 2024-03-30T04:37:20.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:20 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi123.tqxvzr"}]: dispatch 2024-03-30T04:37:20.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:20 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi135.xwgjvb"}]: dispatch 2024-03-30T04:37:21.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:21 smithi135 ceph-mon[106011]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 49/324 objects degraded (15.123%), 12 pgs degraded) 2024-03-30T04:37:21.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:21 smithi135 ceph-mon[106011]: mds.? [v2:172.21.15.135:6824/2944944016,v1:172.21.15.135:6825/2944944016] up:boot 2024-03-30T04:37:21.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:21 smithi135 ceph-mon[106011]: mds.? [v2:172.21.15.123:6828/3902778632,v1:172.21.15.123:6829/3902778632] up:boot 2024-03-30T04:37:21.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:21 smithi135 ceph-mon[106011]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-03-30T04:37:21.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:21 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi135.zvnlap"}]: dispatch 2024-03-30T04:37:21.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:21 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi123.dvtind"}]: dispatch 2024-03-30T04:37:21.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:21 smithi123 ceph-mon[139369]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 49/324 objects degraded (15.123%), 12 pgs degraded) 2024-03-30T04:37:21.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:21 smithi123 ceph-mon[139369]: mds.? [v2:172.21.15.135:6824/2944944016,v1:172.21.15.135:6825/2944944016] up:boot 2024-03-30T04:37:21.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:21 smithi123 ceph-mon[139369]: mds.? [v2:172.21.15.123:6828/3902778632,v1:172.21.15.123:6829/3902778632] up:boot 2024-03-30T04:37:21.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:21 smithi123 ceph-mon[139369]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-03-30T04:37:21.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:21 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi135.zvnlap"}]: dispatch 2024-03-30T04:37:21.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:21 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi123.dvtind"}]: dispatch 2024-03-30T04:37:22.797 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:22 smithi123 ceph-mon[139369]: pgmap v182: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:37:22.797 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:22 smithi123 ceph-mon[139369]: Upgrade: Updating mds.cephfs.smithi123.tqxvzr 2024-03-30T04:37:22.797 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:22 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:22.797 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:22 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi123.tqxvzr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T04:37:22.797 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:22 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:37:22.797 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:22 smithi123 ceph-mon[139369]: Deploying daemon mds.cephfs.smithi123.tqxvzr on smithi123 2024-03-30T04:37:22.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:22 smithi135 ceph-mon[106011]: pgmap v182: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:37:22.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:22 smithi135 ceph-mon[106011]: Upgrade: Updating mds.cephfs.smithi123.tqxvzr 2024-03-30T04:37:22.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:22 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:22.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:22 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi123.tqxvzr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T04:37:22.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:22 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:37:22.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:22 smithi135 ceph-mon[106011]: Deploying daemon mds.cephfs.smithi123.tqxvzr on smithi123 2024-03-30T04:37:25.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:24 smithi135 ceph-mon[106011]: pgmap v183: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:37:25.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:24 smithi135 ceph-mon[106011]: osdmap e83: 6 total, 6 up, 6 in 2024-03-30T04:37:25.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:24 smithi135 ceph-mon[106011]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-03-30T04:37:25.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:24 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:25.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:24 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi123.tqxvzr"}]: dispatch 2024-03-30T04:37:25.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:24 smithi123 ceph-mon[139369]: pgmap v183: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:37:25.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:24 smithi123 ceph-mon[139369]: osdmap e83: 6 total, 6 up, 6 in 2024-03-30T04:37:25.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:24 smithi123 ceph-mon[139369]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-03-30T04:37:25.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:24 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:25.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:24 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi123.tqxvzr"}]: dispatch 2024-03-30T04:37:27.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:26 smithi135 ceph-mon[106011]: pgmap v185: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:37:27.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:26 smithi123 ceph-mon[139369]: pgmap v185: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:37:29.046 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:28 smithi123 ceph-mon[139369]: pgmap v186: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 127 B/s rd, 0 op/s 2024-03-30T04:37:29.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:28 smithi135 ceph-mon[106011]: pgmap v186: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 127 B/s rd, 0 op/s 2024-03-30T04:37:30.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:30 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:30.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:30 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:30.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:30 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:37:30.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:30 smithi135 ceph-mon[106011]: pgmap v187: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 103 B/s rd, 0 op/s 2024-03-30T04:37:30.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:30 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:30.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:30 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:37:30.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:30 smithi135 ceph-mon[106011]: mds.? [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] up:boot 2024-03-30T04:37:30.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:30 smithi135 ceph-mon[106011]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-03-30T04:37:30.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:30 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi123.tqxvzr"}]: dispatch 2024-03-30T04:37:30.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:30.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:30.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:37:30.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:30 smithi123 ceph-mon[139369]: pgmap v187: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 103 B/s rd, 0 op/s 2024-03-30T04:37:30.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:30.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:37:30.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:30 smithi123 ceph-mon[139369]: mds.? [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] up:boot 2024-03-30T04:37:30.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:30 smithi123 ceph-mon[139369]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-03-30T04:37:30.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi123.tqxvzr"}]: dispatch 2024-03-30T04:37:32.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:32 smithi123 ceph-mon[139369]: pgmap v188: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 102 B/s rd, 0 op/s 2024-03-30T04:37:32.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:32 smithi135 ceph-mon[106011]: pgmap v188: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 102 B/s rd, 0 op/s 2024-03-30T04:37:34.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:34 smithi135 ceph-mon[106011]: pgmap v189: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 102 B/s rd, 0 op/s 2024-03-30T04:37:34.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:34 smithi123 ceph-mon[139369]: pgmap v189: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 102 B/s rd, 0 op/s 2024-03-30T04:37:36.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:36 smithi123 ceph-mon[139369]: pgmap v190: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:37:36.969 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:36 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:36.970 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:36 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:37.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:36 smithi135 ceph-mon[106011]: pgmap v190: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:37:37.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:36 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:37.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:36 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:38.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:37 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:38.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:37 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:38.171 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:37 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:38.172 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:37 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:39.086 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:38 smithi123 ceph-mon[139369]: pgmap v191: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:37:39.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:38 smithi135 ceph-mon[106011]: pgmap v191: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:37:40.700 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:40 smithi123 ceph-mon[139369]: pgmap v192: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:37:40.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:40 smithi135 ceph-mon[106011]: pgmap v192: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:37:42.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.145+0000 7f7c04c00700 1 -- 172.21.15.123:0/860921700 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c00071f90 msgr2=0x7f7c00070360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:42.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.145+0000 7f7c04c00700 1 --2- 172.21.15.123:0/860921700 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c00071f90 0x7f7c00070360 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7bf0004970 tx=0x7f7bf0009e30 comp rx=0 tx=0).stop 2024-03-30T04:37:42.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.145+0000 7f7c04c00700 1 -- 172.21.15.123:0/860921700 shutdown_connections 2024-03-30T04:37:42.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.145+0000 7f7c04c00700 1 --2- 172.21.15.123:0/860921700 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7c000708a0 0x7f7c00070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:42.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.145+0000 7f7c04c00700 1 --2- 172.21.15.123:0/860921700 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c00071f90 0x7f7c00070360 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:42.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.145+0000 7f7c04c00700 1 -- 172.21.15.123:0/860921700 >> 172.21.15.123:0/860921700 conn(0x7f7c000f89d0 msgr2=0x7f7c000fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:37:42.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.145+0000 7f7c04c00700 1 -- 172.21.15.123:0/860921700 shutdown_connections 2024-03-30T04:37:42.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.146+0000 7f7c04c00700 1 -- 172.21.15.123:0/860921700 wait complete. 2024-03-30T04:37:42.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.147+0000 7f7c04c00700 1 Processor -- start 2024-03-30T04:37:42.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.147+0000 7f7c04c00700 1 -- start start 2024-03-30T04:37:42.146 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.147+0000 7f7c04c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c000708a0 0x7f7c0011a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:37:42.146 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.147+0000 7f7c04c00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7c00071f90 0x7f7c0011a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:37:42.146 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.148+0000 7f7c04c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0011ac40 con 0x7f7c000708a0 2024-03-30T04:37:42.146 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.148+0000 7f7c04c00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c00114010 con 0x7f7c00071f90 2024-03-30T04:37:42.146 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.148+0000 7f7bfde00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c000708a0 0x7f7c0011a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:37:42.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.148+0000 7f7bfde00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c000708a0 0x7f7c0011a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48534/0 (socket says 172.21.15.123:48534) 2024-03-30T04:37:42.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.148+0000 7f7bfde00700 1 -- 172.21.15.123:0/2151746604 learned_addr learned my addr 172.21.15.123:0/2151746604 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:37:42.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.148+0000 7f7bfd400700 1 --2- 172.21.15.123:0/2151746604 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7c00071f90 0x7f7c0011a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:37:42.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.148+0000 7f7bfde00700 1 -- 172.21.15.123:0/2151746604 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7c00071f90 msgr2=0x7f7c0011a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:42.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.148+0000 7f7bfde00700 1 --2- 172.21.15.123:0/2151746604 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7c00071f90 0x7f7c0011a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:42.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.148+0000 7f7bfde00700 1 -- 172.21.15.123:0/2151746604 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bf0005040 con 0x7f7c000708a0 2024-03-30T04:37:42.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.148+0000 7f7bfde00700 1 --2- 172.21.15.123:0/2151746604 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c000708a0 0x7f7c0011a0b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7bf0004f50 tx=0x7f7bf0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:37:42.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.149+0000 7f7bf6000700 1 -- 172.21.15.123:0/2151746604 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7bf000e040 con 0x7f7c000708a0 2024-03-30T04:37:42.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.149+0000 7f7bf6000700 1 -- 172.21.15.123:0/2151746604 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7bf0013420 con 0x7f7c000708a0 2024-03-30T04:37:42.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.149+0000 7f7c04c00700 1 -- 172.21.15.123:0/2151746604 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c00114290 con 0x7f7c000708a0 2024-03-30T04:37:42.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.149+0000 7f7bf6000700 1 -- 172.21.15.123:0/2151746604 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7bf001c630 con 0x7f7c000708a0 2024-03-30T04:37:42.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.149+0000 7f7c04c00700 1 -- 172.21.15.123:0/2151746604 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c001147d0 con 0x7f7c000708a0 2024-03-30T04:37:42.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.150+0000 7f7c04c00700 1 -- 172.21.15.123:0/2151746604 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c00071aa0 con 0x7f7c000708a0 2024-03-30T04:37:42.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.151+0000 7f7bf6000700 1 -- 172.21.15.123:0/2151746604 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f7bf0013590 con 0x7f7c000708a0 2024-03-30T04:37:42.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.153+0000 7f7bf6000700 1 --2- 172.21.15.123:0/2151746604 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7be406cc30 0x7f7be406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:37:42.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.153+0000 7f7bf6000700 1 -- 172.21.15.123:0/2151746604 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7bf008d340 con 0x7f7c000708a0 2024-03-30T04:37:42.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.153+0000 7f7bfd400700 1 --2- 172.21.15.123:0/2151746604 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7be406cc30 0x7f7be406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:37:42.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.154+0000 7f7bfd400700 1 --2- 172.21.15.123:0/2151746604 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7be406cc30 0x7f7be406f0e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7c00115950 tx=0x7f7bec00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:37:42.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.157+0000 7f7bf6000700 1 -- 172.21.15.123:0/2151746604 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7bf005ad90 con 0x7f7c000708a0 2024-03-30T04:37:42.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.391+0000 7f7c04c00700 1 -- 172.21.15.123:0/2151746604 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7c0004bf70 con 0x7f7be406cc30 2024-03-30T04:37:42.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.393+0000 7f7bf6000700 1 -- 172.21.15.123:0/2151746604 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f7bec00f040 con 0x7f7be406cc30 2024-03-30T04:37:42.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.395+0000 7f7c04c00700 1 -- 172.21.15.123:0/2151746604 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7be406cc30 msgr2=0x7f7be406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:42.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.395+0000 7f7c04c00700 1 --2- 172.21.15.123:0/2151746604 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7be406cc30 0x7f7be406f0e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7c00115950 tx=0x7f7bec00b040 comp rx=0 tx=0).stop 2024-03-30T04:37:42.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.395+0000 7f7c04c00700 1 -- 172.21.15.123:0/2151746604 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c000708a0 msgr2=0x7f7c0011a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:42.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.395+0000 7f7c04c00700 1 --2- 172.21.15.123:0/2151746604 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c000708a0 0x7f7c0011a0b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7bf0004f50 tx=0x7f7bf0009e30 comp rx=0 tx=0).stop 2024-03-30T04:37:42.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.395+0000 7f7c04c00700 1 -- 172.21.15.123:0/2151746604 shutdown_connections 2024-03-30T04:37:42.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.395+0000 7f7c04c00700 1 --2- 172.21.15.123:0/2151746604 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7be406cc30 0x7f7be406f0e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:42.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.395+0000 7f7c04c00700 1 --2- 172.21.15.123:0/2151746604 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c000708a0 0x7f7c0011a0b0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:42.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.395+0000 7f7c04c00700 1 --2- 172.21.15.123:0/2151746604 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7c00071f90 0x7f7c0011a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:42.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.395+0000 7f7c04c00700 1 -- 172.21.15.123:0/2151746604 >> 172.21.15.123:0/2151746604 conn(0x7f7c000f89d0 msgr2=0x7f7c000f9bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:37:42.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.395+0000 7f7c04c00700 1 -- 172.21.15.123:0/2151746604 shutdown_connections 2024-03-30T04:37:42.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.395+0000 7f7c04c00700 1 -- 172.21.15.123:0/2151746604 wait complete. 2024-03-30T04:37:42.409 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:37:42.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.542+0000 7f02e1200700 1 -- 172.21.15.123:0/526078014 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02dc1037a0 msgr2=0x7f02dc0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:42.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.542+0000 7f02e1200700 1 --2- 172.21.15.123:0/526078014 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02dc1037a0 0x7f02dc0fce10 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f02c8004940 tx=0x7f02c8009e30 comp rx=0 tx=0).stop 2024-03-30T04:37:42.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.542+0000 7f02e1200700 1 -- 172.21.15.123:0/526078014 shutdown_connections 2024-03-30T04:37:42.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.542+0000 7f02e1200700 1 --2- 172.21.15.123:0/526078014 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02dc1037a0 0x7f02dc0fce10 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:42.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.542+0000 7f02e1200700 1 --2- 172.21.15.123:0/526078014 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f02dc102e00 0x7f02dc1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:42.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.542+0000 7f02e1200700 1 -- 172.21.15.123:0/526078014 >> 172.21.15.123:0/526078014 conn(0x7f02dc0f89b0 msgr2=0x7f02dc0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:37:42.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.542+0000 7f02e1200700 1 -- 172.21.15.123:0/526078014 shutdown_connections 2024-03-30T04:37:42.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.542+0000 7f02e1200700 1 -- 172.21.15.123:0/526078014 wait complete. 2024-03-30T04:37:42.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.543+0000 7f02e1200700 1 Processor -- start 2024-03-30T04:37:42.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.543+0000 7f02e1200700 1 -- start start 2024-03-30T04:37:42.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.544+0000 7f02e1200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f02dc102e00 0x7f02dc0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:37:42.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.544+0000 7f02e1200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02dc1037a0 0x7f02dc070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:37:42.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.544+0000 7f02e1200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02dc070d90 con 0x7f02dc1037a0 2024-03-30T04:37:42.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.544+0000 7f02e1200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02dc070ed0 con 0x7f02dc102e00 2024-03-30T04:37:42.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.544+0000 7f02d9400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02dc1037a0 0x7f02dc070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:37:42.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.544+0000 7f02d9400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02dc1037a0 0x7f02dc070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48554/0 (socket says 172.21.15.123:48554) 2024-03-30T04:37:42.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.544+0000 7f02d9400700 1 -- 172.21.15.123:0/3103809687 learned_addr learned my addr 172.21.15.123:0/3103809687 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:37:42.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.544+0000 7f02d9e00700 1 --2- 172.21.15.123:0/3103809687 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f02dc102e00 0x7f02dc0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:37:42.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.544+0000 7f02d9400700 1 -- 172.21.15.123:0/3103809687 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f02dc102e00 msgr2=0x7f02dc0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:42.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.544+0000 7f02d9400700 1 --2- 172.21.15.123:0/3103809687 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f02dc102e00 0x7f02dc0702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:42.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.544+0000 7f02d9400700 1 -- 172.21.15.123:0/3103809687 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02c8005040 con 0x7f02dc1037a0 2024-03-30T04:37:42.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.545+0000 7f02d9400700 1 --2- 172.21.15.123:0/3103809687 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02dc1037a0 0x7f02dc070800 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f02c80093b0 tx=0x7f02c8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:37:42.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.545+0000 7f02d2000700 1 -- 172.21.15.123:0/3103809687 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f02c800e040 con 0x7f02dc1037a0 2024-03-30T04:37:42.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.545+0000 7f02e1200700 1 -- 172.21.15.123:0/3103809687 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02dc071120 con 0x7f02dc1037a0 2024-03-30T04:37:42.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.545+0000 7f02d2000700 1 -- 172.21.15.123:0/3103809687 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f02c8013420 con 0x7f02dc1037a0 2024-03-30T04:37:42.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.545+0000 7f02d2000700 1 -- 172.21.15.123:0/3103809687 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f02c801c630 con 0x7f02dc1037a0 2024-03-30T04:37:42.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.545+0000 7f02e1200700 1 -- 172.21.15.123:0/3103809687 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02dc11a230 con 0x7f02dc1037a0 2024-03-30T04:37:42.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.546+0000 7f02e1200700 1 -- 172.21.15.123:0/3103809687 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02dc0fe550 con 0x7f02dc1037a0 2024-03-30T04:37:42.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.548+0000 7f02d2000700 1 -- 172.21.15.123:0/3103809687 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f02c80139e0 con 0x7f02dc1037a0 2024-03-30T04:37:42.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.549+0000 7f02d2000700 1 --2- 172.21.15.123:0/3103809687 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f02c006cc20 0x7f02c006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:37:42.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.549+0000 7f02d2000700 1 -- 172.21.15.123:0/3103809687 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f02c808d2e0 con 0x7f02dc1037a0 2024-03-30T04:37:42.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.549+0000 7f02d9e00700 1 --2- 172.21.15.123:0/3103809687 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f02c006cc20 0x7f02c006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:37:42.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.550+0000 7f02d9e00700 1 --2- 172.21.15.123:0/3103809687 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f02c006cc20 0x7f02c006f0d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f02cc004d60 tx=0x7f02cc00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:37:42.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.554+0000 7f02d2000700 1 -- 172.21.15.123:0/3103809687 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f02c80572a0 con 0x7f02dc1037a0 2024-03-30T04:37:42.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:42 smithi123 ceph-mon[139369]: pgmap v193: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:37:42.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:42 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:42.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:42 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:42.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:42 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:37:42.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:42 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:37:42.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:42 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:42.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:42 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:37:42.718 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:42 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:37:42.718 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:42 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:37:42.718 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:42 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:37:42.718 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:42 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:37:42.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.783+0000 7f02e1200700 1 -- 172.21.15.123:0/3103809687 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f02dc109e40 con 0x7f02c006cc20 2024-03-30T04:37:42.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.785+0000 7f02d2000700 1 -- 172.21.15.123:0/3103809687 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f02cc00f040 con 0x7f02c006cc20 2024-03-30T04:37:42.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.787+0000 7f02e1200700 1 -- 172.21.15.123:0/3103809687 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f02c006cc20 msgr2=0x7f02c006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:42.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.787+0000 7f02e1200700 1 --2- 172.21.15.123:0/3103809687 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f02c006cc20 0x7f02c006f0d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f02cc004d60 tx=0x7f02cc00b040 comp rx=0 tx=0).stop 2024-03-30T04:37:42.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.787+0000 7f02e1200700 1 -- 172.21.15.123:0/3103809687 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02dc1037a0 msgr2=0x7f02dc070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:42.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.787+0000 7f02e1200700 1 --2- 172.21.15.123:0/3103809687 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02dc1037a0 0x7f02dc070800 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f02c80093b0 tx=0x7f02c8009e30 comp rx=0 tx=0).stop 2024-03-30T04:37:42.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.787+0000 7f02e1200700 1 -- 172.21.15.123:0/3103809687 shutdown_connections 2024-03-30T04:37:42.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.787+0000 7f02e1200700 1 --2- 172.21.15.123:0/3103809687 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f02c006cc20 0x7f02c006f0d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:42.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.787+0000 7f02e1200700 1 --2- 172.21.15.123:0/3103809687 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f02dc102e00 0x7f02dc0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:42.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.787+0000 7f02e1200700 1 --2- 172.21.15.123:0/3103809687 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02dc1037a0 0x7f02dc070800 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:42.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.787+0000 7f02e1200700 1 -- 172.21.15.123:0/3103809687 >> 172.21.15.123:0/3103809687 conn(0x7f02dc0f89b0 msgr2=0x7f02dc0f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:37:42.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.787+0000 7f02e1200700 1 -- 172.21.15.123:0/3103809687 shutdown_connections 2024-03-30T04:37:42.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.787+0000 7f02e1200700 1 -- 172.21.15.123:0/3103809687 wait complete. 2024-03-30T04:37:42.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:42 smithi135 ceph-mon[106011]: pgmap v193: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:37:42.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:42 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:42.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:42 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:42.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:42 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:37:42.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:42 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:37:42.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:42 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:42.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:42 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:37:42.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:42 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:37:42.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:42 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:37:42.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:42 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:37:42.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:42 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:37:42.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.932+0000 7f0bb5e00700 1 -- 172.21.15.123:0/1076939175 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bb00fde20 msgr2=0x7f0bb00fe1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:42.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.932+0000 7f0bb5e00700 1 --2- 172.21.15.123:0/1076939175 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bb00fde20 0x7f0bb00fe1f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0ba0004940 tx=0x7f0ba0009e30 comp rx=0 tx=0).stop 2024-03-30T04:37:42.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.932+0000 7f0bb5e00700 1 -- 172.21.15.123:0/1076939175 shutdown_connections 2024-03-30T04:37:42.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.932+0000 7f0bb5e00700 1 --2- 172.21.15.123:0/1076939175 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0bb00fe7c0 0x7f0bb0106e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:42.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.932+0000 7f0bb5e00700 1 --2- 172.21.15.123:0/1076939175 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bb00fde20 0x7f0bb00fe1f0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:42.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.932+0000 7f0bb5e00700 1 -- 172.21.15.123:0/1076939175 >> 172.21.15.123:0/1076939175 conn(0x7f0bb00f7850 msgr2=0x7f0bb00f9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:37:42.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.932+0000 7f0bb5e00700 1 -- 172.21.15.123:0/1076939175 shutdown_connections 2024-03-30T04:37:42.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.932+0000 7f0bb5e00700 1 -- 172.21.15.123:0/1076939175 wait complete. 2024-03-30T04:37:42.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.933+0000 7f0bb5e00700 1 Processor -- start 2024-03-30T04:37:42.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.933+0000 7f0bb5e00700 1 -- start start 2024-03-30T04:37:42.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.933+0000 7f0bb5e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bb00fde20 0x7f0bb0115ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:37:42.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.933+0000 7f0bb5e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0bb00fe7c0 0x7f0bb0116000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:37:42.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.933+0000 7f0bb5e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb0116650 con 0x7f0bb00fde20 2024-03-30T04:37:42.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.933+0000 7f0bb5e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb010fc30 con 0x7f0bb00fe7c0 2024-03-30T04:37:42.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.934+0000 7f0baea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bb00fde20 0x7f0bb0115ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:37:42.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.934+0000 7f0baea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bb00fde20 0x7f0bb0115ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48568/0 (socket says 172.21.15.123:48568) 2024-03-30T04:37:42.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.934+0000 7f0baea00700 1 -- 172.21.15.123:0/1094959776 learned_addr learned my addr 172.21.15.123:0/1094959776 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:37:42.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.934+0000 7f0bae000700 1 --2- 172.21.15.123:0/1094959776 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0bb00fe7c0 0x7f0bb0116000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:37:42.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.934+0000 7f0baea00700 1 -- 172.21.15.123:0/1094959776 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0bb00fe7c0 msgr2=0x7f0bb0116000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:42.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.934+0000 7f0baea00700 1 --2- 172.21.15.123:0/1094959776 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0bb00fe7c0 0x7f0bb0116000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:42.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.934+0000 7f0baea00700 1 -- 172.21.15.123:0/1094959776 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba0005040 con 0x7f0bb00fde20 2024-03-30T04:37:42.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.934+0000 7f0baea00700 1 --2- 172.21.15.123:0/1094959776 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bb00fde20 0x7f0bb0115ac0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0ba0009390 tx=0x7f0ba0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:37:42.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.935+0000 7f0ba6a00700 1 -- 172.21.15.123:0/1094959776 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ba000e040 con 0x7f0bb00fde20 2024-03-30T04:37:42.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.935+0000 7f0ba6a00700 1 -- 172.21.15.123:0/1094959776 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0ba0013420 con 0x7f0bb00fde20 2024-03-30T04:37:42.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.935+0000 7f0bb5e00700 1 -- 172.21.15.123:0/1094959776 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bb010feb0 con 0x7f0bb00fde20 2024-03-30T04:37:42.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.935+0000 7f0ba6a00700 1 -- 172.21.15.123:0/1094959776 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ba001c630 con 0x7f0bb00fde20 2024-03-30T04:37:42.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.935+0000 7f0bb5e00700 1 -- 172.21.15.123:0/1094959776 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bb01103f0 con 0x7f0bb00fde20 2024-03-30T04:37:42.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.936+0000 7f0bb5e00700 1 -- 172.21.15.123:0/1094959776 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bb0066b50 con 0x7f0bb00fde20 2024-03-30T04:37:42.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.937+0000 7f0ba6a00700 1 -- 172.21.15.123:0/1094959776 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f0ba0013a20 con 0x7f0bb00fde20 2024-03-30T04:37:42.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.937+0000 7f0ba6a00700 1 --2- 172.21.15.123:0/1094959776 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0b9406cc20 0x7f0b9406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:37:42.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.938+0000 7f0ba6a00700 1 -- 172.21.15.123:0/1094959776 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f0ba008d300 con 0x7f0bb00fde20 2024-03-30T04:37:42.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.938+0000 7f0bae000700 1 --2- 172.21.15.123:0/1094959776 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0b9406cc20 0x7f0b9406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:37:42.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.938+0000 7f0bae000700 1 --2- 172.21.15.123:0/1094959776 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0b9406cc20 0x7f0b9406f0d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0bb0111360 tx=0x7f0b9c009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:37:42.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:42.942+0000 7f0ba6a00700 1 -- 172.21.15.123:0/1094959776 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0ba005ad50 con 0x7f0bb00fde20 2024-03-30T04:37:43.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.160+0000 7f0bb5e00700 1 -- 172.21.15.123:0/1094959776 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f0bb0109e10 con 0x7f0b9406cc20 2024-03-30T04:37:43.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.170+0000 7f0ba6a00700 1 -- 172.21.15.123:0/1094959776 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f0b9c004e30 con 0x7f0b9406cc20 2024-03-30T04:37:43.169 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:37:43.169 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (6m) 7s ago 19m 15.0M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:37:43.169 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (4m) 7s ago 19m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T04:37:43.170 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (4m) 31s ago 18m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T04:37:43.170 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (6m) 7s ago 19m 73.7M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T04:37:43.170 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (13m) 7s ago 13m 10.5M - 17.2.7-797-g5c024335 f77085f6aab7 2df066fba7a6 2024-03-30T04:37:43.170 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (14s) 7s ago 13m 9785k - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T04:37:43.170 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (12m) 31s ago 12m 67.6M - 17.2.7-797-g5c024335 f77085f6aab7 0dc41326818c 2024-03-30T04:37:43.170 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (13m) 31s ago 13m 119M - 17.2.7-797-g5c024335 f77085f6aab7 aef8a0cfc553 2024-03-30T04:37:43.170 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (9m) 7s ago 21m 506M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:37:43.170 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (8m) 31s ago 18m 450M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:37:43.170 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (5m) 7s ago 21m 52.4M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T04:37:43.171 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (4m) 31s ago 18m 39.8M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T04:37:43.171 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (7m) 7s ago 19m 8166k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:37:43.171 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (7m) 31s ago 18m 8044k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:37:43.171 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (3m) 7s ago 17m 103M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T04:37:43.171 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (3m) 7s ago 16m 93.3M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T04:37:43.171 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (2m) 7s ago 16m 86.2M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T04:37:43.171 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (111s) 31s ago 15m 96.1M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T04:37:43.171 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (73s) 31s ago 15m 96.7M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T04:37:43.172 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (36s) 31s ago 14m 11.0M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T04:37:43.172 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (7m) 7s ago 18m 53.7M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:37:43.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.172+0000 7f0bb5e00700 1 -- 172.21.15.123:0/1094959776 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0b9406cc20 msgr2=0x7f0b9406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:43.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.172+0000 7f0bb5e00700 1 --2- 172.21.15.123:0/1094959776 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0b9406cc20 0x7f0b9406f0d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0bb0111360 tx=0x7f0b9c009ee0 comp rx=0 tx=0).stop 2024-03-30T04:37:43.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.172+0000 7f0bb5e00700 1 -- 172.21.15.123:0/1094959776 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bb00fde20 msgr2=0x7f0bb0115ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:43.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.172+0000 7f0bb5e00700 1 --2- 172.21.15.123:0/1094959776 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bb00fde20 0x7f0bb0115ac0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0ba0009390 tx=0x7f0ba0009e30 comp rx=0 tx=0).stop 2024-03-30T04:37:43.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.173+0000 7f0bb5e00700 1 -- 172.21.15.123:0/1094959776 shutdown_connections 2024-03-30T04:37:43.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.173+0000 7f0bb5e00700 1 --2- 172.21.15.123:0/1094959776 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0b9406cc20 0x7f0b9406f0d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:43.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.173+0000 7f0bb5e00700 1 --2- 172.21.15.123:0/1094959776 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bb00fde20 0x7f0bb0115ac0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:43.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.173+0000 7f0bb5e00700 1 --2- 172.21.15.123:0/1094959776 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0bb00fe7c0 0x7f0bb0116000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:43.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.173+0000 7f0bb5e00700 1 -- 172.21.15.123:0/1094959776 >> 172.21.15.123:0/1094959776 conn(0x7f0bb00f7850 msgr2=0x7f0bb00f8df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:37:43.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.173+0000 7f0bb5e00700 1 -- 172.21.15.123:0/1094959776 shutdown_connections 2024-03-30T04:37:43.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.173+0000 7f0bb5e00700 1 -- 172.21.15.123:0/1094959776 wait complete. 2024-03-30T04:37:43.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.308+0000 7f9111e00700 1 -- 172.21.15.123:0/239180693 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f910c1035e0 msgr2=0x7f910c0fcc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:43.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.308+0000 7f9111e00700 1 --2- 172.21.15.123:0/239180693 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f910c1035e0 0x7f910c0fcc50 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f90f8004940 tx=0x7f90f8009e30 comp rx=0 tx=0).stop 2024-03-30T04:37:43.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.308+0000 7f9111e00700 1 -- 172.21.15.123:0/239180693 shutdown_connections 2024-03-30T04:37:43.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.308+0000 7f9111e00700 1 --2- 172.21.15.123:0/239180693 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f910c1035e0 0x7f910c0fcc50 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:43.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.308+0000 7f9111e00700 1 --2- 172.21.15.123:0/239180693 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f910c102c40 0x7f910c103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:43.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.308+0000 7f9111e00700 1 -- 172.21.15.123:0/239180693 >> 172.21.15.123:0/239180693 conn(0x7f910c0f8830 msgr2=0x7f910c0fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:37:43.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.308+0000 7f9111e00700 1 -- 172.21.15.123:0/239180693 shutdown_connections 2024-03-30T04:37:43.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.308+0000 7f9111e00700 1 -- 172.21.15.123:0/239180693 wait complete. 2024-03-30T04:37:43.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.309+0000 7f9111e00700 1 Processor -- start 2024-03-30T04:37:43.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.309+0000 7f9111e00700 1 -- start start 2024-03-30T04:37:43.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.309+0000 7f9111e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f910c102c40 0x7f910c1138e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:37:43.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.309+0000 7f9111e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f910c1035e0 0x7f910c113e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:37:43.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.309+0000 7f9111e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f910c114470 con 0x7f910c102c40 2024-03-30T04:37:43.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.310+0000 7f9111e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f910c10d860 con 0x7f910c1035e0 2024-03-30T04:37:43.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.310+0000 7f9110a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f910c102c40 0x7f910c1138e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:37:43.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.310+0000 7f9110a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f910c102c40 0x7f910c1138e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48592/0 (socket says 172.21.15.123:48592) 2024-03-30T04:37:43.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.310+0000 7f9110a00700 1 -- 172.21.15.123:0/833442651 learned_addr learned my addr 172.21.15.123:0/833442651 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:37:43.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.310+0000 7f910be00700 1 --2- 172.21.15.123:0/833442651 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f910c1035e0 0x7f910c113e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:37:43.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.310+0000 7f9110a00700 1 -- 172.21.15.123:0/833442651 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f910c1035e0 msgr2=0x7f910c113e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:43.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.310+0000 7f9110a00700 1 --2- 172.21.15.123:0/833442651 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f910c1035e0 0x7f910c113e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:43.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.310+0000 7f9110a00700 1 -- 172.21.15.123:0/833442651 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90f8005040 con 0x7f910c102c40 2024-03-30T04:37:43.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.310+0000 7f910be00700 1 --2- 172.21.15.123:0/833442651 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f910c1035e0 0x7f910c113e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:37:43.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.310+0000 7f9110a00700 1 --2- 172.21.15.123:0/833442651 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f910c102c40 0x7f910c1138e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9100002a50 tx=0x7f91000046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:37:43.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.311+0000 7f9108c00700 1 -- 172.21.15.123:0/833442651 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9100007cc0 con 0x7f910c102c40 2024-03-30T04:37:43.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.311+0000 7f9108c00700 1 -- 172.21.15.123:0/833442651 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f910000f040 con 0x7f910c102c40 2024-03-30T04:37:43.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.311+0000 7f9111e00700 1 -- 172.21.15.123:0/833442651 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f910c10db40 con 0x7f910c102c40 2024-03-30T04:37:43.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.311+0000 7f9108c00700 1 -- 172.21.15.123:0/833442651 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9100009540 con 0x7f910c102c40 2024-03-30T04:37:43.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.311+0000 7f9111e00700 1 -- 172.21.15.123:0/833442651 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f910c10e0d0 con 0x7f910c102c40 2024-03-30T04:37:43.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.312+0000 7f9111e00700 1 -- 172.21.15.123:0/833442651 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f910c0fe390 con 0x7f910c102c40 2024-03-30T04:37:43.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.313+0000 7f9108c00700 1 -- 172.21.15.123:0/833442651 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f91000096a0 con 0x7f910c102c40 2024-03-30T04:37:43.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.314+0000 7f9108c00700 1 --2- 172.21.15.123:0/833442651 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f90f406cc30 0x7f90f406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:37:43.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.314+0000 7f9108c00700 1 -- 172.21.15.123:0/833442651 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f910008bee0 con 0x7f910c102c40 2024-03-30T04:37:43.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.314+0000 7f910be00700 1 --2- 172.21.15.123:0/833442651 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f90f406cc30 0x7f90f406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:37:43.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.314+0000 7f910be00700 1 --2- 172.21.15.123:0/833442651 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f90f406cc30 0x7f90f406f0e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f910c10eb40 tx=0x7f90f8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:37:43.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.318+0000 7f9108c00700 1 -- 172.21.15.123:0/833442651 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f91000598b0 con 0x7f910c102c40 2024-03-30T04:37:43.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.618+0000 7f9111e00700 1 -- 172.21.15.123:0/833442651 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f910c109bd0 con 0x7f910c102c40 2024-03-30T04:37:43.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.619+0000 7f9108c00700 1 -- 172.21.15.123:0/833442651 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f9100059440 con 0x7f910c102c40 2024-03-30T04:37:43.618 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:37:43.618 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:37:43.618 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:37:43.618 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:37:43.618 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:37:43.618 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:37:43.618 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:37:43.618 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:37:43.618 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T04:37:43.618 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:37:43.618 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:37:43.619 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 3, 2024-03-30T04:37:43.619 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 1 2024-03-30T04:37:43.619 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:37:43.619 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:37:43.619 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 3, 2024-03-30T04:37:43.619 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 11 2024-03-30T04:37:43.619 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:37:43.619 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:37:43.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.621+0000 7f9111e00700 1 -- 172.21.15.123:0/833442651 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f90f406cc30 msgr2=0x7f90f406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:43.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.621+0000 7f9111e00700 1 --2- 172.21.15.123:0/833442651 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f90f406cc30 0x7f90f406f0e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f910c10eb40 tx=0x7f90f8009e30 comp rx=0 tx=0).stop 2024-03-30T04:37:43.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.621+0000 7f9111e00700 1 -- 172.21.15.123:0/833442651 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f910c102c40 msgr2=0x7f910c1138e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:43.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.621+0000 7f9111e00700 1 --2- 172.21.15.123:0/833442651 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f910c102c40 0x7f910c1138e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9100002a50 tx=0x7f91000046b0 comp rx=0 tx=0).stop 2024-03-30T04:37:43.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.621+0000 7f9111e00700 1 -- 172.21.15.123:0/833442651 shutdown_connections 2024-03-30T04:37:43.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.621+0000 7f9111e00700 1 --2- 172.21.15.123:0/833442651 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f90f406cc30 0x7f90f406f0e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:43.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.621+0000 7f9111e00700 1 --2- 172.21.15.123:0/833442651 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f910c102c40 0x7f910c1138e0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:43.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.621+0000 7f9111e00700 1 --2- 172.21.15.123:0/833442651 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f910c1035e0 0x7f910c113e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:43.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.621+0000 7f9111e00700 1 -- 172.21.15.123:0/833442651 >> 172.21.15.123:0/833442651 conn(0x7f910c0f8830 msgr2=0x7f910c0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:37:43.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.622+0000 7f9111e00700 1 -- 172.21.15.123:0/833442651 shutdown_connections 2024-03-30T04:37:43.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.622+0000 7f9111e00700 1 -- 172.21.15.123:0/833442651 wait complete. 2024-03-30T04:37:43.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.767+0000 7fd299e00700 1 -- 172.21.15.123:0/177025585 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd294102e00 msgr2=0x7fd2941031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:43.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.767+0000 7fd299e00700 1 --2- 172.21.15.123:0/177025585 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd294102e00 0x7fd2941031d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd284004940 tx=0x7fd284009e30 comp rx=0 tx=0).stop 2024-03-30T04:37:43.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.767+0000 7fd299e00700 1 -- 172.21.15.123:0/177025585 shutdown_connections 2024-03-30T04:37:43.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.767+0000 7fd299e00700 1 --2- 172.21.15.123:0/177025585 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2941037a0 0x7fd2940fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:43.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.767+0000 7fd299e00700 1 --2- 172.21.15.123:0/177025585 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd294102e00 0x7fd2941031d0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:43.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.767+0000 7fd299e00700 1 -- 172.21.15.123:0/177025585 >> 172.21.15.123:0/177025585 conn(0x7fd2940f89b0 msgr2=0x7fd2940fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:37:43.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.767+0000 7fd299e00700 1 -- 172.21.15.123:0/177025585 shutdown_connections 2024-03-30T04:37:43.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.767+0000 7fd299e00700 1 -- 172.21.15.123:0/177025585 wait complete. 2024-03-30T04:37:43.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.768+0000 7fd299e00700 1 Processor -- start 2024-03-30T04:37:43.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.768+0000 7fd299e00700 1 -- start start 2024-03-30T04:37:43.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.768+0000 7fd299e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd294102e00 0x7fd294115ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:37:43.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.768+0000 7fd299e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2941037a0 0x7fd2941161e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:37:43.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.768+0000 7fd299e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd294116830 con 0x7fd2941037a0 2024-03-30T04:37:43.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.768+0000 7fd299e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd29410fc00 con 0x7fd294102e00 2024-03-30T04:37:43.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.768+0000 7fd292000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2941037a0 0x7fd2941161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:37:43.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.769+0000 7fd292000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2941037a0 0x7fd2941161e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48606/0 (socket says 172.21.15.123:48606) 2024-03-30T04:37:43.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.769+0000 7fd292000700 1 -- 172.21.15.123:0/3558503479 learned_addr learned my addr 172.21.15.123:0/3558503479 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:37:43.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.769+0000 7fd292a00700 1 --2- 172.21.15.123:0/3558503479 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd294102e00 0x7fd294115ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:37:43.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.769+0000 7fd292000700 1 -- 172.21.15.123:0/3558503479 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd294102e00 msgr2=0x7fd294115ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:43.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.769+0000 7fd292000700 1 --2- 172.21.15.123:0/3558503479 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd294102e00 0x7fd294115ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:43.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.769+0000 7fd292000700 1 -- 172.21.15.123:0/3558503479 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd284005040 con 0x7fd2941037a0 2024-03-30T04:37:43.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.769+0000 7fd292000700 1 --2- 172.21.15.123:0/3558503479 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2941037a0 0x7fd2941161e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd28000eb40 tx=0x7fd28000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:37:43.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.769+0000 7fd28aa00700 1 -- 172.21.15.123:0/3558503479 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd280007cc0 con 0x7fd2941037a0 2024-03-30T04:37:43.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.769+0000 7fd299e00700 1 -- 172.21.15.123:0/3558503479 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd29410fe80 con 0x7fd2941037a0 2024-03-30T04:37:43.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.770+0000 7fd28aa00700 1 -- 172.21.15.123:0/3558503479 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd28000f040 con 0x7fd2941037a0 2024-03-30T04:37:43.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.770+0000 7fd28aa00700 1 -- 172.21.15.123:0/3558503479 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd28000ca30 con 0x7fd2941037a0 2024-03-30T04:37:43.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.770+0000 7fd299e00700 1 -- 172.21.15.123:0/3558503479 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd294110440 con 0x7fd2941037a0 2024-03-30T04:37:43.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.771+0000 7fd299e00700 1 -- 172.21.15.123:0/3558503479 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2940fe550 con 0x7fd2941037a0 2024-03-30T04:37:43.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.772+0000 7fd28aa00700 1 -- 172.21.15.123:0/3558503479 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fd28000cb90 con 0x7fd2941037a0 2024-03-30T04:37:43.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.773+0000 7fd28aa00700 1 --2- 172.21.15.123:0/3558503479 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd27806cc30 0x7fd27806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:37:43.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.773+0000 7fd28aa00700 1 -- 172.21.15.123:0/3558503479 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd28008bbe0 con 0x7fd2941037a0 2024-03-30T04:37:43.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.773+0000 7fd292a00700 1 --2- 172.21.15.123:0/3558503479 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd27806cc30 0x7fd27806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:37:43.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.774+0000 7fd292a00700 1 --2- 172.21.15.123:0/3558503479 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd27806cc30 0x7fd27806f0e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd2840045a0 tx=0x7fd284009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:37:43.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:43.778+0000 7fd28aa00700 1 -- 172.21.15.123:0/3558503479 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd280059630 con 0x7fd2941037a0 2024-03-30T04:37:43.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:43 smithi135 ceph-mon[106011]: from='client.34424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:37:43.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:43 smithi135 ceph-mon[106011]: from='client.34428 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:37:43.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:43 smithi123 ceph-mon[139369]: from='client.34424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:37:43.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:43 smithi123 ceph-mon[139369]: from='client.34428 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:37:44.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.061+0000 7fd299e00700 1 -- 172.21.15.123:0/3558503479 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fd294109de0 con 0x7fd2941037a0 2024-03-30T04:37:44.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.062+0000 7fd28aa00700 1 -- 172.21.15.123:0/3558503479 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1845 (secure 0 0 0) 0x7fd2800591c0 con 0x7fd2941037a0 2024-03-30T04:37:44.060 INFO:teuthology.orchestra.run.smithi123.stdout:e28 2024-03-30T04:37:44.060 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:37:44.060 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:37:44.060 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:37:44.061 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:37:44.061 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:37:44.061 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:37:44.061 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T04:37:44.061 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T04:37:44.061 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:37:44.061 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T04:37:44.061 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:37:44.061 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:37:44.061 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:37:44.061 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:37:44.061 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:37:44.061 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T04:37:44.061 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:37:44.062 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:37:44.062 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T04:37:44.062 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:37:44.062 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T04:37:44.062 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T04:37:44.062 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T04:37:44.062 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T04:37:44.063 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:37:44.063 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:37:44.063 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:37:44.063 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:37:44.063 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:37:44.063 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:37:44.064 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T04:37:44.064 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T04:37:44.064 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:37:44.064 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:37:44.064 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T04:37:44.065 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:37:44.065 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:37:44.066 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44315} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/1741352336,v1:172.21.15.135:6827/1741352336] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:37:44.066 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44323} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/2944944016,v1:172.21.15.135:6825/2944944016] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:37:44.066 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:44325} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/3902778632,v1:172.21.15.123:6829/3902778632] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:37:44.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.064+0000 7fd299e00700 1 -- 172.21.15.123:0/3558503479 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd27806cc30 msgr2=0x7fd27806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:44.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.064+0000 7fd299e00700 1 --2- 172.21.15.123:0/3558503479 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd27806cc30 0x7fd27806f0e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd2840045a0 tx=0x7fd284009e30 comp rx=0 tx=0).stop 2024-03-30T04:37:44.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.064+0000 7fd299e00700 1 -- 172.21.15.123:0/3558503479 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2941037a0 msgr2=0x7fd2941161e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:44.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.064+0000 7fd299e00700 1 --2- 172.21.15.123:0/3558503479 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2941037a0 0x7fd2941161e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd28000eb40 tx=0x7fd28000b040 comp rx=0 tx=0).stop 2024-03-30T04:37:44.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.064+0000 7fd299e00700 1 -- 172.21.15.123:0/3558503479 shutdown_connections 2024-03-30T04:37:44.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.064+0000 7fd299e00700 1 --2- 172.21.15.123:0/3558503479 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd27806cc30 0x7fd27806f0e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:44.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.064+0000 7fd299e00700 1 --2- 172.21.15.123:0/3558503479 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd294102e00 0x7fd294115ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:44.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.064+0000 7fd299e00700 1 --2- 172.21.15.123:0/3558503479 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2941037a0 0x7fd2941161e0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:44.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.064+0000 7fd299e00700 1 -- 172.21.15.123:0/3558503479 >> 172.21.15.123:0/3558503479 conn(0x7fd2940f89b0 msgr2=0x7fd2940f9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:37:44.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.064+0000 7fd299e00700 1 -- 172.21.15.123:0/3558503479 shutdown_connections 2024-03-30T04:37:44.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.064+0000 7fd299e00700 1 -- 172.21.15.123:0/3558503479 wait complete. 2024-03-30T04:37:44.069 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 28 2024-03-30T04:37:44.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.204+0000 7ffba2e00700 1 -- 172.21.15.123:0/3621680417 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb9c1037a0 msgr2=0x7ffb9c0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:44.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.204+0000 7ffba2e00700 1 --2- 172.21.15.123:0/3621680417 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb9c1037a0 0x7ffb9c0fce10 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ffb88004940 tx=0x7ffb88009e30 comp rx=0 tx=0).stop 2024-03-30T04:37:44.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.205+0000 7ffba2e00700 1 -- 172.21.15.123:0/3621680417 shutdown_connections 2024-03-30T04:37:44.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.205+0000 7ffba2e00700 1 --2- 172.21.15.123:0/3621680417 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb9c1037a0 0x7ffb9c0fce10 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:44.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.205+0000 7ffba2e00700 1 --2- 172.21.15.123:0/3621680417 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffb9c102e00 0x7ffb9c1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:44.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.205+0000 7ffba2e00700 1 -- 172.21.15.123:0/3621680417 >> 172.21.15.123:0/3621680417 conn(0x7ffb9c0f89b0 msgr2=0x7ffb9c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:37:44.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.205+0000 7ffba2e00700 1 -- 172.21.15.123:0/3621680417 shutdown_connections 2024-03-30T04:37:44.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.205+0000 7ffba2e00700 1 -- 172.21.15.123:0/3621680417 wait complete. 2024-03-30T04:37:44.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.205+0000 7ffba2e00700 1 Processor -- start 2024-03-30T04:37:44.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.206+0000 7ffba2e00700 1 -- start start 2024-03-30T04:37:44.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.206+0000 7ffba2e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb9c102e00 0x7ffb9c115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:37:44.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.206+0000 7ffba2e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffb9c1037a0 0x7ffb9c1161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:37:44.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.206+0000 7ffba2e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb9c116810 con 0x7ffb9c102e00 2024-03-30T04:37:44.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.206+0000 7ffba2e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb9c10fbe0 con 0x7ffb9c1037a0 2024-03-30T04:37:44.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.206+0000 7ffb9be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb9c102e00 0x7ffb9c115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:37:44.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.206+0000 7ffb9be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb9c102e00 0x7ffb9c115c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34454/0 (socket says 172.21.15.123:34454) 2024-03-30T04:37:44.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.206+0000 7ffb9be00700 1 -- 172.21.15.123:0/2904055043 learned_addr learned my addr 172.21.15.123:0/2904055043 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:37:44.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.206+0000 7ffb9b400700 1 --2- 172.21.15.123:0/2904055043 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffb9c1037a0 0x7ffb9c1161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:37:44.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.207+0000 7ffb9be00700 1 -- 172.21.15.123:0/2904055043 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffb9c1037a0 msgr2=0x7ffb9c1161c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:44.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.207+0000 7ffb9be00700 1 --2- 172.21.15.123:0/2904055043 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffb9c1037a0 0x7ffb9c1161c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:44.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.207+0000 7ffb9be00700 1 -- 172.21.15.123:0/2904055043 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb88005040 con 0x7ffb9c102e00 2024-03-30T04:37:44.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.207+0000 7ffb9be00700 1 --2- 172.21.15.123:0/2904055043 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb9c102e00 0x7ffb9c115c80 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c00eb40 tx=0x7ffb8c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:37:44.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.207+0000 7ffb93e00700 1 -- 172.21.15.123:0/2904055043 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb8c007cc0 con 0x7ffb9c102e00 2024-03-30T04:37:44.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.207+0000 7ffb93e00700 1 -- 172.21.15.123:0/2904055043 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffb8c00f040 con 0x7ffb9c102e00 2024-03-30T04:37:44.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.207+0000 7ffba2e00700 1 -- 172.21.15.123:0/2904055043 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb9c10fe60 con 0x7ffb9c102e00 2024-03-30T04:37:44.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.207+0000 7ffb93e00700 1 -- 172.21.15.123:0/2904055043 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb8c00c990 con 0x7ffb9c102e00 2024-03-30T04:37:44.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.207+0000 7ffba2e00700 1 -- 172.21.15.123:0/2904055043 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb9c110420 con 0x7ffb9c102e00 2024-03-30T04:37:44.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.208+0000 7ffba2e00700 1 -- 172.21.15.123:0/2904055043 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb9c0fe550 con 0x7ffb9c102e00 2024-03-30T04:37:44.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.209+0000 7ffb93e00700 1 -- 172.21.15.123:0/2904055043 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7ffb8c0090d0 con 0x7ffb9c102e00 2024-03-30T04:37:44.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.210+0000 7ffb93e00700 1 --2- 172.21.15.123:0/2904055043 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ffb8006cc30 0x7ffb8006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:37:44.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.210+0000 7ffb93e00700 1 -- 172.21.15.123:0/2904055043 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ffb8c08bbf0 con 0x7ffb9c102e00 2024-03-30T04:37:44.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.210+0000 7ffb9b400700 1 --2- 172.21.15.123:0/2904055043 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ffb8006cc30 0x7ffb8006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:37:44.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.211+0000 7ffb9b400700 1 --2- 172.21.15.123:0/2904055043 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ffb8006cc30 0x7ffb8006f0e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ffb9c111520 tx=0x7ffb88009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:37:44.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.214+0000 7ffb93e00700 1 -- 172.21.15.123:0/2904055043 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ffb8c0596f0 con 0x7ffb9c102e00 2024-03-30T04:37:44.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.448+0000 7ffba2e00700 1 -- 172.21.15.123:0/2904055043 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ffb9c04bf40 con 0x7ffb8006cc30 2024-03-30T04:37:44.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.449+0000 7ffb93e00700 1 -- 172.21.15.123:0/2904055043 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ffb9c04bf40 con 0x7ffb8006cc30 2024-03-30T04:37:44.447 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:37:44.447 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:37:44.447 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:37:44.448 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T04:37:44.448 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:37:44.448 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T04:37:44.448 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T04:37:44.448 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T04:37:44.448 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T04:37:44.448 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:37:44.448 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "13/21 daemons upgraded", 2024-03-30T04:37:44.448 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T04:37:44.448 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:37:44.448 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:37:44.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.452+0000 7ffba2e00700 1 -- 172.21.15.123:0/2904055043 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ffb8006cc30 msgr2=0x7ffb8006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:44.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.452+0000 7ffba2e00700 1 --2- 172.21.15.123:0/2904055043 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ffb8006cc30 0x7ffb8006f0e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ffb9c111520 tx=0x7ffb88009e30 comp rx=0 tx=0).stop 2024-03-30T04:37:44.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.452+0000 7ffba2e00700 1 -- 172.21.15.123:0/2904055043 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb9c102e00 msgr2=0x7ffb9c115c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:44.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.452+0000 7ffba2e00700 1 --2- 172.21.15.123:0/2904055043 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb9c102e00 0x7ffb9c115c80 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c00eb40 tx=0x7ffb8c00b040 comp rx=0 tx=0).stop 2024-03-30T04:37:44.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.452+0000 7ffba2e00700 1 -- 172.21.15.123:0/2904055043 shutdown_connections 2024-03-30T04:37:44.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.452+0000 7ffba2e00700 1 --2- 172.21.15.123:0/2904055043 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ffb8006cc30 0x7ffb8006f0e0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:44.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.452+0000 7ffba2e00700 1 --2- 172.21.15.123:0/2904055043 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb9c102e00 0x7ffb9c115c80 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:44.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.452+0000 7ffba2e00700 1 --2- 172.21.15.123:0/2904055043 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffb9c1037a0 0x7ffb9c1161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:44.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.452+0000 7ffba2e00700 1 -- 172.21.15.123:0/2904055043 >> 172.21.15.123:0/2904055043 conn(0x7ffb9c0f89b0 msgr2=0x7ffb9c0f99a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:37:44.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.452+0000 7ffba2e00700 1 -- 172.21.15.123:0/2904055043 shutdown_connections 2024-03-30T04:37:44.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.452+0000 7ffba2e00700 1 -- 172.21.15.123:0/2904055043 wait complete. 2024-03-30T04:37:44.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.592+0000 7f3b8e400700 1 -- 172.21.15.123:0/832606216 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b88102e00 msgr2=0x7f3b881031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:44.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.592+0000 7f3b8e400700 1 --2- 172.21.15.123:0/832606216 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b88102e00 0x7f3b881031d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f3b78004940 tx=0x7f3b78009e30 comp rx=0 tx=0).stop 2024-03-30T04:37:44.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.592+0000 7f3b8e400700 1 -- 172.21.15.123:0/832606216 shutdown_connections 2024-03-30T04:37:44.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.592+0000 7f3b8e400700 1 --2- 172.21.15.123:0/832606216 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b881037a0 0x7f3b880fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:44.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.592+0000 7f3b8e400700 1 --2- 172.21.15.123:0/832606216 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b88102e00 0x7f3b881031d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:44.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.592+0000 7f3b8e400700 1 -- 172.21.15.123:0/832606216 >> 172.21.15.123:0/832606216 conn(0x7f3b880f89b0 msgr2=0x7f3b880fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:37:44.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.592+0000 7f3b8e400700 1 -- 172.21.15.123:0/832606216 shutdown_connections 2024-03-30T04:37:44.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.592+0000 7f3b8e400700 1 -- 172.21.15.123:0/832606216 wait complete. 2024-03-30T04:37:44.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.593+0000 7f3b8e400700 1 Processor -- start 2024-03-30T04:37:44.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.594+0000 7f3b8e400700 1 -- start start 2024-03-30T04:37:44.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.594+0000 7f3b8e400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b88102e00 0x7f3b880702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:37:44.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.594+0000 7f3b8e400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b881037a0 0x7f3b88070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:37:44.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.594+0000 7f3b8e400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b88070e20 con 0x7f3b88102e00 2024-03-30T04:37:44.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.594+0000 7f3b8e400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b88070f60 con 0x7f3b881037a0 2024-03-30T04:37:44.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.594+0000 7f3b87400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b88102e00 0x7f3b880702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:37:44.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.594+0000 7f3b87400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b88102e00 0x7f3b880702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34480/0 (socket says 172.21.15.123:34480) 2024-03-30T04:37:44.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.594+0000 7f3b87400700 1 -- 172.21.15.123:0/3320469729 learned_addr learned my addr 172.21.15.123:0/3320469729 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:37:44.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.595+0000 7f3b86a00700 1 --2- 172.21.15.123:0/3320469729 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b881037a0 0x7f3b88070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:37:44.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.595+0000 7f3b87400700 1 -- 172.21.15.123:0/3320469729 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b881037a0 msgr2=0x7f3b88070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:44.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.595+0000 7f3b87400700 1 --2- 172.21.15.123:0/3320469729 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b881037a0 0x7f3b88070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:44.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.595+0000 7f3b87400700 1 -- 172.21.15.123:0/3320469729 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b78005040 con 0x7f3b88102e00 2024-03-30T04:37:44.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.595+0000 7f3b86a00700 1 --2- 172.21.15.123:0/3320469729 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b881037a0 0x7f3b88070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:37:44.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.595+0000 7f3b87400700 1 --2- 172.21.15.123:0/3320469729 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b88102e00 0x7f3b880702c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f3b78009390 tx=0x7f3b78009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:37:44.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.595+0000 7f3b7f400700 1 -- 172.21.15.123:0/3320469729 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b7800e040 con 0x7f3b88102e00 2024-03-30T04:37:44.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.595+0000 7f3b7f400700 1 -- 172.21.15.123:0/3320469729 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3b78013420 con 0x7f3b88102e00 2024-03-30T04:37:44.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.595+0000 7f3b8e400700 1 -- 172.21.15.123:0/3320469729 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b88119da0 con 0x7f3b88102e00 2024-03-30T04:37:44.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.595+0000 7f3b7f400700 1 -- 172.21.15.123:0/3320469729 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b7801c630 con 0x7f3b88102e00 2024-03-30T04:37:44.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.596+0000 7f3b8e400700 1 -- 172.21.15.123:0/3320469729 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b8811a280 con 0x7f3b88102e00 2024-03-30T04:37:44.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.597+0000 7f3b8e400700 1 -- 172.21.15.123:0/3320469729 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b880fe550 con 0x7f3b88102e00 2024-03-30T04:37:44.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.597+0000 7f3b7f400700 1 -- 172.21.15.123:0/3320469729 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f3b780139e0 con 0x7f3b88102e00 2024-03-30T04:37:44.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.598+0000 7f3b7f400700 1 --2- 172.21.15.123:0/3320469729 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3b6c06cc30 0x7f3b6c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:37:44.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.598+0000 7f3b7f400700 1 -- 172.21.15.123:0/3320469729 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f3b7808d260 con 0x7f3b88102e00 2024-03-30T04:37:44.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.599+0000 7f3b86a00700 1 --2- 172.21.15.123:0/3320469729 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3b6c06cc30 0x7f3b6c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:37:44.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.599+0000 7f3b86a00700 1 --2- 172.21.15.123:0/3320469729 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3b6c06cc30 0x7f3b6c06f0e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3b74006fd0 tx=0x7f3b74008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:37:44.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.603+0000 7f3b7f400700 1 -- 172.21.15.123:0/3320469729 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3b780572a0 con 0x7f3b88102e00 2024-03-30T04:37:44.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:44 smithi135 ceph-mon[106011]: from='client.34432 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:37:44.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:44 smithi135 ceph-mon[106011]: pgmap v194: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:37:44.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:44 smithi135 ceph-mon[106011]: Upgrade: Updating mds.cephfs.smithi123.dvtind 2024-03-30T04:37:44.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:44 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:44.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:44 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi123.dvtind", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T04:37:44.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:44 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:37:44.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:44 smithi135 ceph-mon[106011]: Deploying daemon mds.cephfs.smithi123.dvtind on smithi123 2024-03-30T04:37:44.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:44 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/833442651' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:37:44.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:44 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3558503479' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:37:44.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:44 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:37:44.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.890+0000 7f3b8e400700 1 -- 172.21.15.123:0/3320469729 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f3b8805f5f0 con 0x7f3b88102e00 2024-03-30T04:37:44.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.891+0000 7f3b7f400700 1 -- 172.21.15.123:0/3320469729 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f3b8805f5f0 con 0x7f3b88102e00 2024-03-30T04:37:44.889 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T04:37:44.889 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T04:37:44.889 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T04:37:44.889 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T04:37:44.889 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T04:37:44.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.893+0000 7f3b8e400700 1 -- 172.21.15.123:0/3320469729 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3b6c06cc30 msgr2=0x7f3b6c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:44.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.893+0000 7f3b8e400700 1 --2- 172.21.15.123:0/3320469729 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3b6c06cc30 0x7f3b6c06f0e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3b74006fd0 tx=0x7f3b74008040 comp rx=0 tx=0).stop 2024-03-30T04:37:44.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.893+0000 7f3b8e400700 1 -- 172.21.15.123:0/3320469729 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b88102e00 msgr2=0x7f3b880702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:37:44.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.893+0000 7f3b8e400700 1 --2- 172.21.15.123:0/3320469729 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b88102e00 0x7f3b880702c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f3b78009390 tx=0x7f3b78009e30 comp rx=0 tx=0).stop 2024-03-30T04:37:44.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.893+0000 7f3b8e400700 1 -- 172.21.15.123:0/3320469729 shutdown_connections 2024-03-30T04:37:44.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.893+0000 7f3b8e400700 1 --2- 172.21.15.123:0/3320469729 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3b6c06cc30 0x7f3b6c06f0e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:44.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.893+0000 7f3b8e400700 1 --2- 172.21.15.123:0/3320469729 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b88102e00 0x7f3b880702c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:44.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.893+0000 7f3b8e400700 1 --2- 172.21.15.123:0/3320469729 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b881037a0 0x7f3b88070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:37:44.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.893+0000 7f3b8e400700 1 -- 172.21.15.123:0/3320469729 >> 172.21.15.123:0/3320469729 conn(0x7f3b880f89b0 msgr2=0x7f3b880f9a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:37:44.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.893+0000 7f3b8e400700 1 -- 172.21.15.123:0/3320469729 shutdown_connections 2024-03-30T04:37:44.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:37:44.893+0000 7f3b8e400700 1 -- 172.21.15.123:0/3320469729 wait complete. 2024-03-30T04:37:44.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:44 smithi123 ceph-mon[139369]: from='client.34432 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:37:44.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:44 smithi123 ceph-mon[139369]: pgmap v194: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:37:44.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:44 smithi123 ceph-mon[139369]: Upgrade: Updating mds.cephfs.smithi123.dvtind 2024-03-30T04:37:44.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:44.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi123.dvtind", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T04:37:44.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:37:44.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:44 smithi123 ceph-mon[139369]: Deploying daemon mds.cephfs.smithi123.dvtind on smithi123 2024-03-30T04:37:44.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:44 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/833442651' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:37:44.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:44 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3558503479' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:37:44.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:37:45.684 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:45 smithi123 ceph-mon[139369]: from='client.34444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:37:45.684 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:45 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3320469729' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:37:45.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:45 smithi135 ceph-mon[106011]: from='client.34444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:37:45.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:45 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3320469729' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:37:46.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:46 smithi135 ceph-mon[106011]: pgmap v195: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:37:46.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:46 smithi135 ceph-mon[106011]: osdmap e84: 6 total, 6 up, 6 in 2024-03-30T04:37:46.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:46 smithi123 ceph-mon[139369]: pgmap v195: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:37:46.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:46 smithi123 ceph-mon[139369]: osdmap e84: 6 total, 6 up, 6 in 2024-03-30T04:37:47.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:47 smithi135 ceph-mon[106011]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-03-30T04:37:47.851 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:47 smithi123 ceph-mon[139369]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-03-30T04:37:48.631 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:48 smithi123 ceph-mon[139369]: pgmap v197: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:37:48.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:48 smithi135 ceph-mon[106011]: pgmap v197: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:37:50.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:50 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:50.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:50 smithi123 ceph-mon[139369]: pgmap v198: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:37:50.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:50 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:50.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:50 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:37:50.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:50 smithi123 ceph-mon[139369]: mds.? [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] up:boot 2024-03-30T04:37:50.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:50 smithi123 ceph-mon[139369]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-03-30T04:37:50.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:50 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi123.dvtind"}]: dispatch 2024-03-30T04:37:50.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:50 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:50.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:50 smithi135 ceph-mon[106011]: pgmap v198: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:37:50.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:50 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:50.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:50 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:37:50.852 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:50 smithi135 ceph-mon[106011]: mds.? [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] up:boot 2024-03-30T04:37:50.852 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:50 smithi135 ceph-mon[106011]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-03-30T04:37:50.852 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:50 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi123.dvtind"}]: dispatch 2024-03-30T04:37:52.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:52 smithi123 ceph-mon[139369]: pgmap v199: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:37:52.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:52 smithi135 ceph-mon[106011]: pgmap v199: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:37:54.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:54 smithi123 ceph-mon[139369]: pgmap v200: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:37:54.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:54 smithi135 ceph-mon[106011]: pgmap v200: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:37:56.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:55 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:56.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:55 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:56.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:55 smithi123 ceph-mon[139369]: pgmap v201: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:37:56.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:55 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:56.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:55 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:56.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:55 smithi135 ceph-mon[106011]: pgmap v201: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:37:57.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:57 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:57.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:57 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:57.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:57 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:57.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:57 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:37:58.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:37:58 smithi135 ceph-mon[106011]: pgmap v202: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:37:58.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:37:58 smithi123 ceph-mon[139369]: pgmap v202: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:00.708 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:00 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:38:00.709 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:00 smithi123 ceph-mon[139369]: pgmap v203: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:00.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:00 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:38:00.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:00 smithi135 ceph-mon[106011]: pgmap v203: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:02.038 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:01 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:38:02.038 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:01 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:38:02.038 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:01 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:38:02.038 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:01 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:38:02.038 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:01 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:38:02.038 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:01 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:38:02.038 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:01 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:38:02.038 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:01 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:38:02.038 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:01 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:38:02.039 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:01 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:38:02.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:01 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:38:02.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:01 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:38:02.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:01 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:38:02.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:01 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:38:02.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:01 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:38:02.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:01 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:38:02.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:01 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:38:02.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:01 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:38:02.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:01 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:38:02.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:01 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:38:03.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:02 smithi135 ceph-mon[106011]: pgmap v204: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:03.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:02 smithi123 ceph-mon[139369]: pgmap v204: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:04.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:03 smithi135 ceph-mon[106011]: Upgrade: Updating mds.cephfs.smithi135.zvnlap 2024-03-30T04:38:04.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:03 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:38:04.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:03 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi135.zvnlap", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T04:38:04.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:03 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:38:04.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:03 smithi135 ceph-mon[106011]: Deploying daemon mds.cephfs.smithi135.zvnlap on smithi135 2024-03-30T04:38:04.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:03 smithi123 ceph-mon[139369]: Upgrade: Updating mds.cephfs.smithi135.zvnlap 2024-03-30T04:38:04.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:03 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:38:04.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:03 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi135.zvnlap", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T04:38:04.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:03 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:38:04.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:03 smithi123 ceph-mon[139369]: Deploying daemon mds.cephfs.smithi135.zvnlap on smithi135 2024-03-30T04:38:04.925 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:04 smithi135 ceph-mon[106011]: pgmap v205: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:05.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:04 smithi123 ceph-mon[139369]: pgmap v205: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:06.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:06 smithi123 ceph-mon[139369]: osdmap e85: 6 total, 6 up, 6 in 2024-03-30T04:38:06.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:06 smithi123 ceph-mon[139369]: pgmap v207: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:06.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:06 smithi123 ceph-mon[139369]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-03-30T04:38:06.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:06 smithi135 ceph-mon[106011]: osdmap e85: 6 total, 6 up, 6 in 2024-03-30T04:38:06.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:06 smithi135 ceph-mon[106011]: pgmap v207: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:06.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:06 smithi135 ceph-mon[106011]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-03-30T04:38:08.704 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:08 smithi135 ceph-mon[106011]: pgmap v208: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:08.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:08 smithi123 ceph-mon[139369]: pgmap v208: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:10.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:10 smithi135 ceph-mon[106011]: pgmap v209: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:10.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:10 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:38:10.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:10 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:38:10.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:10 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:38:10.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:10 smithi135 ceph-mon[106011]: mds.? [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] up:boot 2024-03-30T04:38:10.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:10 smithi135 ceph-mon[106011]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-03-30T04:38:10.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:10 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi135.zvnlap"}]: dispatch 2024-03-30T04:38:10.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:10 smithi123 ceph-mon[139369]: pgmap v209: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:10.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:10 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:38:10.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:10 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:38:10.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:10 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:38:10.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:10 smithi123 ceph-mon[139369]: mds.? [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] up:boot 2024-03-30T04:38:10.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:10 smithi123 ceph-mon[139369]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-03-30T04:38:10.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:10 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi135.zvnlap"}]: dispatch 2024-03-30T04:38:12.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:12 smithi135 ceph-mon[106011]: pgmap v210: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:12.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:12 smithi123 ceph-mon[139369]: pgmap v210: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:14.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:14 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:38:14.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:14 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:38:14.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:14 smithi123 ceph-mon[139369]: pgmap v211: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:14.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:14 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:38:14.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:14 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:38:14.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:14 smithi135 ceph-mon[106011]: pgmap v211: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:15.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.048+0000 7f7798800700 1 -- 172.21.15.123:0/2519371996 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7790102e00 msgr2=0x7f77901031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:15.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.048+0000 7f7798800700 1 --2- 172.21.15.123:0/2519371996 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7790102e00 0x7f77901031d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7784004940 tx=0x7f7784009e30 comp rx=0 tx=0).stop 2024-03-30T04:38:15.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.048+0000 7f7798800700 1 -- 172.21.15.123:0/2519371996 shutdown_connections 2024-03-30T04:38:15.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.048+0000 7f7798800700 1 --2- 172.21.15.123:0/2519371996 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f77901037a0 0x7f77900fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:15.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.048+0000 7f7798800700 1 --2- 172.21.15.123:0/2519371996 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7790102e00 0x7f77901031d0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:15.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.048+0000 7f7798800700 1 -- 172.21.15.123:0/2519371996 >> 172.21.15.123:0/2519371996 conn(0x7f77900f89b0 msgr2=0x7f77900fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:38:15.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.048+0000 7f7798800700 1 -- 172.21.15.123:0/2519371996 shutdown_connections 2024-03-30T04:38:15.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.048+0000 7f7798800700 1 -- 172.21.15.123:0/2519371996 wait complete. 2024-03-30T04:38:15.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.049+0000 7f7798800700 1 Processor -- start 2024-03-30T04:38:15.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.050+0000 7f7798800700 1 -- start start 2024-03-30T04:38:15.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.050+0000 7f7798800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7790102e00 0x7f7790115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:38:15.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.050+0000 7f7798800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77901037a0 0x7f7790116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:38:15.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.050+0000 7f7798800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7790116860 con 0x7f77901037a0 2024-03-30T04:38:15.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.050+0000 7f7798800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f779010fc30 con 0x7f7790102e00 2024-03-30T04:38:15.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.050+0000 7f7795400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77901037a0 0x7f7790116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:38:15.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.050+0000 7f7795400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77901037a0 0x7f7790116210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47654/0 (socket says 172.21.15.123:47654) 2024-03-30T04:38:15.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.051+0000 7f7795400700 1 -- 172.21.15.123:0/3806271382 learned_addr learned my addr 172.21.15.123:0/3806271382 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:38:15.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.051+0000 7f7795e00700 1 --2- 172.21.15.123:0/3806271382 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7790102e00 0x7f7790115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:38:15.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.051+0000 7f7795400700 1 -- 172.21.15.123:0/3806271382 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7790102e00 msgr2=0x7f7790115cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:15.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.051+0000 7f7795400700 1 --2- 172.21.15.123:0/3806271382 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7790102e00 0x7f7790115cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:15.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.051+0000 7f7795400700 1 -- 172.21.15.123:0/3806271382 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7784005040 con 0x7f77901037a0 2024-03-30T04:38:15.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.051+0000 7f7795400700 1 --2- 172.21.15.123:0/3806271382 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77901037a0 0x7f7790116210 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f7780002a50 tx=0x7f7780004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:38:15.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.051+0000 7f778a000700 1 -- 172.21.15.123:0/3806271382 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7780007cc0 con 0x7f77901037a0 2024-03-30T04:38:15.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.051+0000 7f778a000700 1 -- 172.21.15.123:0/3806271382 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f778000f040 con 0x7f77901037a0 2024-03-30T04:38:15.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.051+0000 7f778a000700 1 -- 172.21.15.123:0/3806271382 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f778000d920 con 0x7f77901037a0 2024-03-30T04:38:15.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.051+0000 7f7798800700 1 -- 172.21.15.123:0/3806271382 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f779010ff10 con 0x7f77901037a0 2024-03-30T04:38:15.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.052+0000 7f7798800700 1 -- 172.21.15.123:0/3806271382 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7790110480 con 0x7f77901037a0 2024-03-30T04:38:15.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.053+0000 7f7798800700 1 -- 172.21.15.123:0/3806271382 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77900fe550 con 0x7f77901037a0 2024-03-30T04:38:15.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.054+0000 7f778a000700 1 -- 172.21.15.123:0/3806271382 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f7780009680 con 0x7f77901037a0 2024-03-30T04:38:15.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.054+0000 7f778a000700 1 --2- 172.21.15.123:0/3806271382 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f777806cc30 0x7f777806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:38:15.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.055+0000 7f778a000700 1 -- 172.21.15.123:0/3806271382 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f778008bdb0 con 0x7f77901037a0 2024-03-30T04:38:15.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.055+0000 7f7795e00700 1 --2- 172.21.15.123:0/3806271382 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f777806cc30 0x7f777806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:38:15.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.055+0000 7f7795e00700 1 --2- 172.21.15.123:0/3806271382 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f777806cc30 0x7f777806f0e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7784009fd0 tx=0x7f7784009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:38:15.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.060+0000 7f778a000700 1 -- 172.21.15.123:0/3806271382 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7780059780 con 0x7f77901037a0 2024-03-30T04:38:15.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.287+0000 7f7798800700 1 -- 172.21.15.123:0/3806271382 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7790109e30 con 0x7f777806cc30 2024-03-30T04:38:15.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.288+0000 7f778a000700 1 -- 172.21.15.123:0/3806271382 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f7790109e30 con 0x7f777806cc30 2024-03-30T04:38:15.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.290+0000 7f7798800700 1 -- 172.21.15.123:0/3806271382 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f777806cc30 msgr2=0x7f777806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:15.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.290+0000 7f7798800700 1 --2- 172.21.15.123:0/3806271382 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f777806cc30 0x7f777806f0e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7784009fd0 tx=0x7f7784009e30 comp rx=0 tx=0).stop 2024-03-30T04:38:15.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.290+0000 7f7798800700 1 -- 172.21.15.123:0/3806271382 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77901037a0 msgr2=0x7f7790116210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:15.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.290+0000 7f7798800700 1 --2- 172.21.15.123:0/3806271382 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77901037a0 0x7f7790116210 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f7780002a50 tx=0x7f7780004600 comp rx=0 tx=0).stop 2024-03-30T04:38:15.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.290+0000 7f7798800700 1 -- 172.21.15.123:0/3806271382 shutdown_connections 2024-03-30T04:38:15.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.290+0000 7f7798800700 1 --2- 172.21.15.123:0/3806271382 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f777806cc30 0x7f777806f0e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:15.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.290+0000 7f7798800700 1 --2- 172.21.15.123:0/3806271382 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7790102e00 0x7f7790115cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:15.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.290+0000 7f7798800700 1 --2- 172.21.15.123:0/3806271382 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77901037a0 0x7f7790116210 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:15.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.290+0000 7f7798800700 1 -- 172.21.15.123:0/3806271382 >> 172.21.15.123:0/3806271382 conn(0x7f77900f89b0 msgr2=0x7f77900f99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:38:15.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.291+0000 7f7798800700 1 -- 172.21.15.123:0/3806271382 shutdown_connections 2024-03-30T04:38:15.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.291+0000 7f7798800700 1 -- 172.21.15.123:0/3806271382 wait complete. 2024-03-30T04:38:15.304 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:38:15.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.427+0000 7f2446e00700 1 -- 172.21.15.123:0/574794067 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2440102e20 msgr2=0x7f24401031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:15.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.427+0000 7f2446e00700 1 --2- 172.21.15.123:0/574794067 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2440102e20 0x7f24401031f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2430004940 tx=0x7f2430009e30 comp rx=0 tx=0).stop 2024-03-30T04:38:15.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.427+0000 7f2446e00700 1 -- 172.21.15.123:0/574794067 shutdown_connections 2024-03-30T04:38:15.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.427+0000 7f2446e00700 1 --2- 172.21.15.123:0/574794067 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f24401037c0 0x7f24400fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:15.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.427+0000 7f2446e00700 1 --2- 172.21.15.123:0/574794067 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2440102e20 0x7f24401031f0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:15.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.427+0000 7f2446e00700 1 -- 172.21.15.123:0/574794067 >> 172.21.15.123:0/574794067 conn(0x7f24400f89d0 msgr2=0x7f24400fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:38:15.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.427+0000 7f2446e00700 1 -- 172.21.15.123:0/574794067 shutdown_connections 2024-03-30T04:38:15.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.427+0000 7f2446e00700 1 -- 172.21.15.123:0/574794067 wait complete. 2024-03-30T04:38:15.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.428+0000 7f2446e00700 1 Processor -- start 2024-03-30T04:38:15.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.428+0000 7f2446e00700 1 -- start start 2024-03-30T04:38:15.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.428+0000 7f2446e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2440102e20 0x7f2440115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:38:15.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.428+0000 7f2446e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f24401037c0 0x7f24401161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:38:15.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.428+0000 7f2446e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2440116800 con 0x7f2440102e20 2024-03-30T04:38:15.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.428+0000 7f2446e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f244010fbd0 con 0x7f24401037c0 2024-03-30T04:38:15.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.429+0000 7f243fe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2440102e20 0x7f2440115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:38:15.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.429+0000 7f243fe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2440102e20 0x7f2440115c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47674/0 (socket says 172.21.15.123:47674) 2024-03-30T04:38:15.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.429+0000 7f243fe00700 1 -- 172.21.15.123:0/1779883685 learned_addr learned my addr 172.21.15.123:0/1779883685 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:38:15.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.429+0000 7f243f400700 1 --2- 172.21.15.123:0/1779883685 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f24401037c0 0x7f24401161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:38:15.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.429+0000 7f243fe00700 1 -- 172.21.15.123:0/1779883685 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f24401037c0 msgr2=0x7f24401161b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:15.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.429+0000 7f243fe00700 1 --2- 172.21.15.123:0/1779883685 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f24401037c0 0x7f24401161b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:15.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.429+0000 7f243fe00700 1 -- 172.21.15.123:0/1779883685 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2430005040 con 0x7f2440102e20 2024-03-30T04:38:15.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.429+0000 7f243fe00700 1 --2- 172.21.15.123:0/1779883685 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2440102e20 0x7f2440115c70 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2430007f50 tx=0x7f2430009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:38:15.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.429+0000 7f2437e00700 1 -- 172.21.15.123:0/1779883685 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2430021070 con 0x7f2440102e20 2024-03-30T04:38:15.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.430+0000 7f2437e00700 1 -- 172.21.15.123:0/1779883685 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f24300093b0 con 0x7f2440102e20 2024-03-30T04:38:15.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.430+0000 7f2446e00700 1 -- 172.21.15.123:0/1779883685 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f244010fe50 con 0x7f2440102e20 2024-03-30T04:38:15.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.430+0000 7f2437e00700 1 -- 172.21.15.123:0/1779883685 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f243000e070 con 0x7f2440102e20 2024-03-30T04:38:15.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.430+0000 7f2446e00700 1 -- 172.21.15.123:0/1779883685 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24401103b0 con 0x7f2440102e20 2024-03-30T04:38:15.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.431+0000 7f2446e00700 1 -- 172.21.15.123:0/1779883685 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24400fe570 con 0x7f2440102e20 2024-03-30T04:38:15.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.432+0000 7f2437e00700 1 -- 172.21.15.123:0/1779883685 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f243001c400 con 0x7f2440102e20 2024-03-30T04:38:15.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.433+0000 7f2437e00700 1 --2- 172.21.15.123:0/1779883685 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f242406cc30 0x7f242406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:38:15.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.433+0000 7f2437e00700 1 -- 172.21.15.123:0/1779883685 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f243008d060 con 0x7f2440102e20 2024-03-30T04:38:15.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.433+0000 7f243f400700 1 --2- 172.21.15.123:0/1779883685 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f242406cc30 0x7f242406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:38:15.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.434+0000 7f243f400700 1 --2- 172.21.15.123:0/1779883685 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f242406cc30 0x7f242406f0e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2440111510 tx=0x7f242c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:38:15.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.437+0000 7f2437e00700 1 -- 172.21.15.123:0/1779883685 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f243005abf0 con 0x7f2440102e20 2024-03-30T04:38:15.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.658+0000 7f2446e00700 1 -- 172.21.15.123:0/1779883685 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f244004bf70 con 0x7f242406cc30 2024-03-30T04:38:15.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.660+0000 7f2437e00700 1 -- 172.21.15.123:0/1779883685 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f242c00f040 con 0x7f242406cc30 2024-03-30T04:38:15.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.662+0000 7f2446e00700 1 -- 172.21.15.123:0/1779883685 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f242406cc30 msgr2=0x7f242406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:15.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.662+0000 7f2446e00700 1 --2- 172.21.15.123:0/1779883685 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f242406cc30 0x7f242406f0e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2440111510 tx=0x7f242c00b040 comp rx=0 tx=0).stop 2024-03-30T04:38:15.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.662+0000 7f2446e00700 1 -- 172.21.15.123:0/1779883685 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2440102e20 msgr2=0x7f2440115c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:15.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.662+0000 7f2446e00700 1 --2- 172.21.15.123:0/1779883685 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2440102e20 0x7f2440115c70 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2430007f50 tx=0x7f2430009e30 comp rx=0 tx=0).stop 2024-03-30T04:38:15.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.662+0000 7f2446e00700 1 -- 172.21.15.123:0/1779883685 shutdown_connections 2024-03-30T04:38:15.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.662+0000 7f2446e00700 1 --2- 172.21.15.123:0/1779883685 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f242406cc30 0x7f242406f0e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:15.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.662+0000 7f2446e00700 1 --2- 172.21.15.123:0/1779883685 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2440102e20 0x7f2440115c70 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:15.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.662+0000 7f2446e00700 1 --2- 172.21.15.123:0/1779883685 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f24401037c0 0x7f24401161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:15.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.662+0000 7f2446e00700 1 -- 172.21.15.123:0/1779883685 >> 172.21.15.123:0/1779883685 conn(0x7f24400f89d0 msgr2=0x7f2440070620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:38:15.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.662+0000 7f2446e00700 1 -- 172.21.15.123:0/1779883685 shutdown_connections 2024-03-30T04:38:15.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.662+0000 7f2446e00700 1 -- 172.21.15.123:0/1779883685 wait complete. 2024-03-30T04:38:15.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:15 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:38:15.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:15 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:38:15.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:15 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:38:15.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.796+0000 7ff65ca00700 1 -- 172.21.15.123:0/3155143620 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff6580fe800 msgr2=0x7ff658106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:15.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.796+0000 7ff65ca00700 1 --2- 172.21.15.123:0/3155143620 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff6580fe800 0x7ff658106e40 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff644004970 tx=0x7ff644009e30 comp rx=0 tx=0).stop 2024-03-30T04:38:15.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.796+0000 7ff65ca00700 1 -- 172.21.15.123:0/3155143620 shutdown_connections 2024-03-30T04:38:15.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.796+0000 7ff65ca00700 1 --2- 172.21.15.123:0/3155143620 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff6580fe800 0x7ff658106e40 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:15.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.796+0000 7ff65ca00700 1 --2- 172.21.15.123:0/3155143620 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff6580fde60 0x7ff6580fe230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:15.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.796+0000 7ff65ca00700 1 -- 172.21.15.123:0/3155143620 >> 172.21.15.123:0/3155143620 conn(0x7ff6580f7870 msgr2=0x7ff6580f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:38:15.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.796+0000 7ff65ca00700 1 -- 172.21.15.123:0/3155143620 shutdown_connections 2024-03-30T04:38:15.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.796+0000 7ff65ca00700 1 -- 172.21.15.123:0/3155143620 wait complete. 2024-03-30T04:38:15.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.797+0000 7ff65ca00700 1 Processor -- start 2024-03-30T04:38:15.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.797+0000 7ff65ca00700 1 -- start start 2024-03-30T04:38:15.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.797+0000 7ff65ca00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff6580fde60 0x7ff6580fbc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:38:15.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.797+0000 7ff65ca00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff6580fe800 0x7ff6580fc1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:38:15.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.797+0000 7ff65ca00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6580fdbe0 con 0x7ff6580fde60 2024-03-30T04:38:15.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.797+0000 7ff65ca00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6580fc710 con 0x7ff6580fe800 2024-03-30T04:38:15.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.798+0000 7ff655e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff6580fde60 0x7ff6580fbc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:38:15.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.798+0000 7ff655e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff6580fde60 0x7ff6580fbc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47694/0 (socket says 172.21.15.123:47694) 2024-03-30T04:38:15.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.798+0000 7ff655e00700 1 -- 172.21.15.123:0/2684451067 learned_addr learned my addr 172.21.15.123:0/2684451067 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:38:15.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.798+0000 7ff655400700 1 --2- 172.21.15.123:0/2684451067 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff6580fe800 0x7ff6580fc1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:38:15.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.798+0000 7ff655e00700 1 -- 172.21.15.123:0/2684451067 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff6580fe800 msgr2=0x7ff6580fc1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:15.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.798+0000 7ff655e00700 1 --2- 172.21.15.123:0/2684451067 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff6580fe800 0x7ff6580fc1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:15.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.798+0000 7ff655e00700 1 -- 172.21.15.123:0/2684451067 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff644005040 con 0x7ff6580fde60 2024-03-30T04:38:15.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.798+0000 7ff655400700 1 --2- 172.21.15.123:0/2684451067 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff6580fe800 0x7ff6580fc1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:38:15.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.798+0000 7ff655e00700 1 --2- 172.21.15.123:0/2684451067 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff6580fde60 0x7ff6580fbc90 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff648002a80 tx=0x7ff6480046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:38:15.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.799+0000 7ff64e000700 1 -- 172.21.15.123:0/2684451067 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff648007cc0 con 0x7ff6580fde60 2024-03-30T04:38:15.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.799+0000 7ff64e000700 1 -- 172.21.15.123:0/2684451067 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff64800f040 con 0x7ff6580fde60 2024-03-30T04:38:15.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.799+0000 7ff65ca00700 1 -- 172.21.15.123:0/2684451067 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6580fc9f0 con 0x7ff6580fde60 2024-03-30T04:38:15.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.799+0000 7ff64e000700 1 -- 172.21.15.123:0/2684451067 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff64800da60 con 0x7ff6580fde60 2024-03-30T04:38:15.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.799+0000 7ff65ca00700 1 -- 172.21.15.123:0/2684451067 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff658119f40 con 0x7ff6580fde60 2024-03-30T04:38:15.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.800+0000 7ff65ca00700 1 -- 172.21.15.123:0/2684451067 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff658066b10 con 0x7ff6580fde60 2024-03-30T04:38:15.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.801+0000 7ff64e000700 1 -- 172.21.15.123:0/2684451067 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7ff64800dbc0 con 0x7ff6580fde60 2024-03-30T04:38:15.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.802+0000 7ff64e000700 1 --2- 172.21.15.123:0/2684451067 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ff63c06cc70 0x7ff63c06f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:38:15.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.802+0000 7ff64e000700 1 -- 172.21.15.123:0/2684451067 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff64808bd20 con 0x7ff6580fde60 2024-03-30T04:38:15.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.802+0000 7ff655400700 1 --2- 172.21.15.123:0/2684451067 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ff63c06cc70 0x7ff63c06f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:38:15.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.803+0000 7ff655400700 1 --2- 172.21.15.123:0/2684451067 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ff63c06cc70 0x7ff63c06f120 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff64400b370 tx=0x7ff644009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:38:15.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:15.806+0000 7ff64e000700 1 -- 172.21.15.123:0/2684451067 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff648055c60 con 0x7ff6580fde60 2024-03-30T04:38:15.806 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:15 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:38:15.806 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:15 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:38:15.806 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:15 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:38:16.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.023+0000 7ff65ca00700 1 -- 172.21.15.123:0/2684451067 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff6580fd240 con 0x7ff63c06cc70 2024-03-30T04:38:16.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.033+0000 7ff64e000700 1 -- 172.21.15.123:0/2684451067 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7ff6580fd240 con 0x7ff63c06cc70 2024-03-30T04:38:16.031 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:38:16.031 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (7m) 21s ago 19m 15.0M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:38:16.031 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (4m) 21s ago 20m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T04:38:16.032 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (4m) 2s ago 19m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T04:38:16.032 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (7m) 21s ago 19m 74.0M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T04:38:16.032 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (27s) 21s ago 13m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T04:38:16.032 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (47s) 21s ago 13m 10.9M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T04:38:16.032 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (13m) 2s ago 13m 10.7M - 17.2.7-797-g5c024335 f77085f6aab7 0dc41326818c 2024-03-30T04:38:16.032 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (7s) 2s ago 13m 9747k - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T04:38:16.032 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (9m) 21s ago 21m 506M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:38:16.032 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (8m) 2s ago 18m 450M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:38:16.032 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (5m) 21s ago 21m 53.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T04:38:16.032 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (5m) 2s ago 18m 42.6M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T04:38:16.032 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (8m) 21s ago 20m 8052k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:38:16.032 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (8m) 2s ago 18m 7880k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:38:16.032 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (4m) 21s ago 17m 104M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T04:38:16.033 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (3m) 21s ago 17m 93.4M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T04:38:16.033 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (3m) 21s ago 16m 86.3M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T04:38:16.033 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (2m) 2s ago 16m 97.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T04:38:16.033 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (106s) 2s ago 15m 97.9M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T04:38:16.033 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (68s) 2s ago 15m 85.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T04:38:16.033 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (7m) 21s ago 19m 53.7M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:38:16.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.035+0000 7ff65ca00700 1 -- 172.21.15.123:0/2684451067 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ff63c06cc70 msgr2=0x7ff63c06f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:16.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.035+0000 7ff65ca00700 1 --2- 172.21.15.123:0/2684451067 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ff63c06cc70 0x7ff63c06f120 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff64400b370 tx=0x7ff644009e30 comp rx=0 tx=0).stop 2024-03-30T04:38:16.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.035+0000 7ff65ca00700 1 -- 172.21.15.123:0/2684451067 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff6580fde60 msgr2=0x7ff6580fbc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:16.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.035+0000 7ff65ca00700 1 --2- 172.21.15.123:0/2684451067 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff6580fde60 0x7ff6580fbc90 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff648002a80 tx=0x7ff6480046b0 comp rx=0 tx=0).stop 2024-03-30T04:38:16.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.035+0000 7ff65ca00700 1 -- 172.21.15.123:0/2684451067 shutdown_connections 2024-03-30T04:38:16.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.035+0000 7ff65ca00700 1 --2- 172.21.15.123:0/2684451067 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ff63c06cc70 0x7ff63c06f120 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:16.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.035+0000 7ff65ca00700 1 --2- 172.21.15.123:0/2684451067 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff6580fde60 0x7ff6580fbc90 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:16.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.035+0000 7ff65ca00700 1 --2- 172.21.15.123:0/2684451067 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff6580fe800 0x7ff6580fc1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:16.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.035+0000 7ff65ca00700 1 -- 172.21.15.123:0/2684451067 >> 172.21.15.123:0/2684451067 conn(0x7ff6580f7870 msgr2=0x7ff6580f8d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:38:16.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.035+0000 7ff65ca00700 1 -- 172.21.15.123:0/2684451067 shutdown_connections 2024-03-30T04:38:16.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.035+0000 7ff65ca00700 1 -- 172.21.15.123:0/2684451067 wait complete. 2024-03-30T04:38:16.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.166+0000 7f49dd000700 1 -- 172.21.15.123:0/3030010927 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49d8102e00 msgr2=0x7f49d81031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:16.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.166+0000 7f49dd000700 1 --2- 172.21.15.123:0/3030010927 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49d8102e00 0x7f49d81031d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f49c8004940 tx=0x7f49c8009e30 comp rx=0 tx=0).stop 2024-03-30T04:38:16.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.166+0000 7f49dd000700 1 -- 172.21.15.123:0/3030010927 shutdown_connections 2024-03-30T04:38:16.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.166+0000 7f49dd000700 1 --2- 172.21.15.123:0/3030010927 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49d81037a0 0x7f49d80fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:16.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.166+0000 7f49dd000700 1 --2- 172.21.15.123:0/3030010927 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49d8102e00 0x7f49d81031d0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:16.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.166+0000 7f49dd000700 1 -- 172.21.15.123:0/3030010927 >> 172.21.15.123:0/3030010927 conn(0x7f49d80f89b0 msgr2=0x7f49d80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:38:16.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.167+0000 7f49dd000700 1 -- 172.21.15.123:0/3030010927 shutdown_connections 2024-03-30T04:38:16.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.167+0000 7f49dd000700 1 -- 172.21.15.123:0/3030010927 wait complete. 2024-03-30T04:38:16.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.168+0000 7f49dd000700 1 Processor -- start 2024-03-30T04:38:16.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.168+0000 7f49dd000700 1 -- start start 2024-03-30T04:38:16.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.168+0000 7f49dd000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49d8102e00 0x7f49d80702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:38:16.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.168+0000 7f49dd000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49d81037a0 0x7f49d8070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:38:16.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.168+0000 7f49dd000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d8070d90 con 0x7f49d81037a0 2024-03-30T04:38:16.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.168+0000 7f49dd000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d8070ed0 con 0x7f49d8102e00 2024-03-30T04:38:16.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.169+0000 7f49d5400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49d81037a0 0x7f49d8070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:38:16.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.169+0000 7f49d5400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49d81037a0 0x7f49d8070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47712/0 (socket says 172.21.15.123:47712) 2024-03-30T04:38:16.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.169+0000 7f49d5400700 1 -- 172.21.15.123:0/3041323192 learned_addr learned my addr 172.21.15.123:0/3041323192 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:38:16.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.169+0000 7f49d5e00700 1 --2- 172.21.15.123:0/3041323192 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49d8102e00 0x7f49d80702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:38:16.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.169+0000 7f49d5400700 1 -- 172.21.15.123:0/3041323192 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49d8102e00 msgr2=0x7f49d80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:16.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.169+0000 7f49d5400700 1 --2- 172.21.15.123:0/3041323192 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49d8102e00 0x7f49d80702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:16.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.169+0000 7f49d5400700 1 -- 172.21.15.123:0/3041323192 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c8005040 con 0x7f49d81037a0 2024-03-30T04:38:16.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.169+0000 7f49d5400700 1 --2- 172.21.15.123:0/3041323192 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49d81037a0 0x7f49d8070800 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f49c4002a50 tx=0x7f49c4004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:38:16.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.170+0000 7f49ce000700 1 -- 172.21.15.123:0/3041323192 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f49c4007cc0 con 0x7f49d81037a0 2024-03-30T04:38:16.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.170+0000 7f49ce000700 1 -- 172.21.15.123:0/3041323192 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f49c400f040 con 0x7f49d81037a0 2024-03-30T04:38:16.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.170+0000 7f49dd000700 1 -- 172.21.15.123:0/3041323192 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49d8119e20 con 0x7f49d81037a0 2024-03-30T04:38:16.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.170+0000 7f49ce000700 1 -- 172.21.15.123:0/3041323192 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f49c400d920 con 0x7f49d81037a0 2024-03-30T04:38:16.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.170+0000 7f49dd000700 1 -- 172.21.15.123:0/3041323192 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49d811a2a0 con 0x7f49d81037a0 2024-03-30T04:38:16.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.171+0000 7f49dd000700 1 -- 172.21.15.123:0/3041323192 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49d80fe550 con 0x7f49d81037a0 2024-03-30T04:38:16.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.172+0000 7f49ce000700 1 -- 172.21.15.123:0/3041323192 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f49c4009680 con 0x7f49d81037a0 2024-03-30T04:38:16.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.172+0000 7f49ce000700 1 --2- 172.21.15.123:0/3041323192 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f49bc06cc20 0x7f49bc06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:38:16.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.173+0000 7f49ce000700 1 -- 172.21.15.123:0/3041323192 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f49c408bdb0 con 0x7f49d81037a0 2024-03-30T04:38:16.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.173+0000 7f49d5e00700 1 --2- 172.21.15.123:0/3041323192 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f49bc06cc20 0x7f49bc06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:38:16.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.173+0000 7f49d5e00700 1 --2- 172.21.15.123:0/3041323192 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f49bc06cc20 0x7f49bc06f0d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f49c8004f90 tx=0x7f49c8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:38:16.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.177+0000 7f49ce000700 1 -- 172.21.15.123:0/3041323192 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f49c4055cf0 con 0x7f49d81037a0 2024-03-30T04:38:16.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.463+0000 7f49dd000700 1 -- 172.21.15.123:0/3041323192 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f49d804bed0 con 0x7f49d81037a0 2024-03-30T04:38:16.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.464+0000 7f49ce000700 1 -- 172.21.15.123:0/3041323192 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f49c4059310 con 0x7f49d81037a0 2024-03-30T04:38:16.462 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:38:16.463 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:38:16.463 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:38:16.463 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:38:16.463 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:38:16.463 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:38:16.463 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:38:16.463 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:38:16.463 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T04:38:16.463 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:38:16.463 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:38:16.463 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:38:16.463 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 3 2024-03-30T04:38:16.463 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:38:16.463 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:38:16.464 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:38:16.464 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 13 2024-03-30T04:38:16.464 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:38:16.464 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:38:16.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.466+0000 7f49dd000700 1 -- 172.21.15.123:0/3041323192 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f49bc06cc20 msgr2=0x7f49bc06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:16.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.466+0000 7f49dd000700 1 --2- 172.21.15.123:0/3041323192 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f49bc06cc20 0x7f49bc06f0d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f49c8004f90 tx=0x7f49c8009e30 comp rx=0 tx=0).stop 2024-03-30T04:38:16.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.466+0000 7f49dd000700 1 -- 172.21.15.123:0/3041323192 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49d81037a0 msgr2=0x7f49d8070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:16.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.466+0000 7f49dd000700 1 --2- 172.21.15.123:0/3041323192 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49d81037a0 0x7f49d8070800 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f49c4002a50 tx=0x7f49c4004600 comp rx=0 tx=0).stop 2024-03-30T04:38:16.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.466+0000 7f49dd000700 1 -- 172.21.15.123:0/3041323192 shutdown_connections 2024-03-30T04:38:16.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.466+0000 7f49dd000700 1 --2- 172.21.15.123:0/3041323192 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f49bc06cc20 0x7f49bc06f0d0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:16.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.466+0000 7f49dd000700 1 --2- 172.21.15.123:0/3041323192 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49d8102e00 0x7f49d80702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:16.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.466+0000 7f49dd000700 1 --2- 172.21.15.123:0/3041323192 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49d81037a0 0x7f49d8070800 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:16.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.466+0000 7f49dd000700 1 -- 172.21.15.123:0/3041323192 >> 172.21.15.123:0/3041323192 conn(0x7f49d80f89b0 msgr2=0x7f49d80f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:38:16.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.466+0000 7f49dd000700 1 -- 172.21.15.123:0/3041323192 shutdown_connections 2024-03-30T04:38:16.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.467+0000 7f49dd000700 1 -- 172.21.15.123:0/3041323192 wait complete. 2024-03-30T04:38:16.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.599+0000 7fa4e0400700 1 -- 172.21.15.123:0/4291439563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4d8070360 msgr2=0x7fa4d8070730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:16.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.599+0000 7fa4e0400700 1 --2- 172.21.15.123:0/4291439563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4d8070360 0x7fa4d8070730 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc004970 tx=0x7fa4cc009e30 comp rx=0 tx=0).stop 2024-03-30T04:38:16.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.599+0000 7fa4e0400700 1 -- 172.21.15.123:0/4291439563 shutdown_connections 2024-03-30T04:38:16.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.599+0000 7fa4e0400700 1 --2- 172.21.15.123:0/4291439563 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa4d8070c70 0x7fa4d8109110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:16.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.599+0000 7fa4e0400700 1 --2- 172.21.15.123:0/4291439563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4d8070360 0x7fa4d8070730 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:16.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.599+0000 7fa4e0400700 1 -- 172.21.15.123:0/4291439563 >> 172.21.15.123:0/4291439563 conn(0x7fa4d80f89b0 msgr2=0x7fa4d80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:38:16.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.600+0000 7fa4e0400700 1 -- 172.21.15.123:0/4291439563 shutdown_connections 2024-03-30T04:38:16.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.600+0000 7fa4e0400700 1 -- 172.21.15.123:0/4291439563 wait complete. 2024-03-30T04:38:16.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.600+0000 7fa4e0400700 1 Processor -- start 2024-03-30T04:38:16.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.601+0000 7fa4e0400700 1 -- start start 2024-03-30T04:38:16.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.601+0000 7fa4e0400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa4d8070360 0x7fa4d8115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:38:16.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.601+0000 7fa4e0400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4d8070c70 0x7fa4d81161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:38:16.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.601+0000 7fa4e0400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d8116810 con 0x7fa4d8070c70 2024-03-30T04:38:16.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.601+0000 7fa4e0400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d810fbe0 con 0x7fa4d8070360 2024-03-30T04:38:16.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.601+0000 7fa4dd000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4d8070c70 0x7fa4d81161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:38:16.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.601+0000 7fa4dd000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4d8070c70 0x7fa4d81161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47728/0 (socket says 172.21.15.123:47728) 2024-03-30T04:38:16.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.601+0000 7fa4dd000700 1 -- 172.21.15.123:0/2892763045 learned_addr learned my addr 172.21.15.123:0/2892763045 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:38:16.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.601+0000 7fa4dda00700 1 --2- 172.21.15.123:0/2892763045 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa4d8070360 0x7fa4d8115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:38:16.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.602+0000 7fa4dd000700 1 -- 172.21.15.123:0/2892763045 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa4d8070360 msgr2=0x7fa4d8115c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:16.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.602+0000 7fa4dd000700 1 --2- 172.21.15.123:0/2892763045 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa4d8070360 0x7fa4d8115c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:16.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.602+0000 7fa4dd000700 1 -- 172.21.15.123:0/2892763045 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4cc005040 con 0x7fa4d8070c70 2024-03-30T04:38:16.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.602+0000 7fa4dd000700 1 --2- 172.21.15.123:0/2892763045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4d8070c70 0x7fa4d81161c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa4c400eb40 tx=0x7fa4c400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:38:16.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.602+0000 7fa4d5600700 1 -- 172.21.15.123:0/2892763045 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa4c4007cc0 con 0x7fa4d8070c70 2024-03-30T04:38:16.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.602+0000 7fa4d5600700 1 -- 172.21.15.123:0/2892763045 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa4c400f040 con 0x7fa4d8070c70 2024-03-30T04:38:16.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.602+0000 7fa4d5600700 1 -- 172.21.15.123:0/2892763045 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa4c400c990 con 0x7fa4d8070c70 2024-03-30T04:38:16.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.602+0000 7fa4e0400700 1 -- 172.21.15.123:0/2892763045 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4d810fe60 con 0x7fa4d8070c70 2024-03-30T04:38:16.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.602+0000 7fa4e0400700 1 -- 172.21.15.123:0/2892763045 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4d81103a0 con 0x7fa4d8070c70 2024-03-30T04:38:16.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:16 smithi135 ceph-mon[106011]: from='client.34458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:38:16.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:16 smithi135 ceph-mon[106011]: pgmap v212: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:16.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:16 smithi135 ceph-mon[106011]: from='client.34462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:38:16.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.603+0000 7fa4e0400700 1 -- 172.21.15.123:0/2892763045 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4d8103b90 con 0x7fa4d8070c70 2024-03-30T04:38:16.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.604+0000 7fa4d5600700 1 -- 172.21.15.123:0/2892763045 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fa4c400cc00 con 0x7fa4d8070c70 2024-03-30T04:38:16.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.605+0000 7fa4d5600700 1 --2- 172.21.15.123:0/2892763045 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa4c006cc30 0x7fa4c006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:38:16.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.605+0000 7fa4d5600700 1 -- 172.21.15.123:0/2892763045 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa4c408bcd0 con 0x7fa4d8070c70 2024-03-30T04:38:16.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.606+0000 7fa4dda00700 1 --2- 172.21.15.123:0/2892763045 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa4c006cc30 0x7fa4c006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:38:16.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.606+0000 7fa4dda00700 1 --2- 172.21.15.123:0/2892763045 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa4c006cc30 0x7fa4c006f0e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc009fd0 tx=0x7fa4cc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:38:16.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.610+0000 7fa4d5600700 1 -- 172.21.15.123:0/2892763045 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa4c4059750 con 0x7fa4d8070c70 2024-03-30T04:38:16.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:16 smithi123 ceph-mon[139369]: from='client.34458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:38:16.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:16 smithi123 ceph-mon[139369]: pgmap v212: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:16.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:16 smithi123 ceph-mon[139369]: from='client.34462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:38:16.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.857+0000 7fa4e0400700 1 -- 172.21.15.123:0/2892763045 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fa4d81112d0 con 0x7fa4d8070c70 2024-03-30T04:38:16.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.857+0000 7fa4d5600700 1 -- 172.21.15.123:0/2892763045 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1843 (secure 0 0 0) 0x7fa4c40592e0 con 0x7fa4d8070c70 2024-03-30T04:38:16.856 INFO:teuthology.orchestra.run.smithi123.stdout:e32 2024-03-30T04:38:16.856 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:38:16.856 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:38:16.856 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:38:16.856 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:38:16.856 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:38:16.856 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:38:16.857 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T04:38:16.857 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T04:38:16.857 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:38:16.857 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T04:38:16.857 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:38:16.857 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:38:16.857 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:38:16.857 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:38:16.857 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:38:16.857 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T04:38:16.857 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:38:16.857 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:38:16.857 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T04:38:16.857 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:38:16.858 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T04:38:16.858 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T04:38:16.858 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T04:38:16.858 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T04:38:16.858 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:38:16.858 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:38:16.858 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:38:16.858 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:38:16.858 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:38:16.858 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:38:16.858 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T04:38:16.858 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T04:38:16.858 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:38:16.859 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:38:16.859 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T04:38:16.859 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:38:16.859 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:38:16.859 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:38:16.859 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44315} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/1741352336,v1:172.21.15.135:6827/1741352336] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:38:16.859 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:38:16.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.860+0000 7fa4e0400700 1 -- 172.21.15.123:0/2892763045 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa4c006cc30 msgr2=0x7fa4c006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:16.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.860+0000 7fa4e0400700 1 --2- 172.21.15.123:0/2892763045 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa4c006cc30 0x7fa4c006f0e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc009fd0 tx=0x7fa4cc009e30 comp rx=0 tx=0).stop 2024-03-30T04:38:16.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.860+0000 7fa4e0400700 1 -- 172.21.15.123:0/2892763045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4d8070c70 msgr2=0x7fa4d81161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:16.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.860+0000 7fa4e0400700 1 --2- 172.21.15.123:0/2892763045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4d8070c70 0x7fa4d81161c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa4c400eb40 tx=0x7fa4c400b040 comp rx=0 tx=0).stop 2024-03-30T04:38:16.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.860+0000 7fa4e0400700 1 -- 172.21.15.123:0/2892763045 shutdown_connections 2024-03-30T04:38:16.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.860+0000 7fa4e0400700 1 --2- 172.21.15.123:0/2892763045 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa4c006cc30 0x7fa4c006f0e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:16.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.860+0000 7fa4e0400700 1 --2- 172.21.15.123:0/2892763045 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa4d8070360 0x7fa4d8115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:16.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.860+0000 7fa4e0400700 1 --2- 172.21.15.123:0/2892763045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4d8070c70 0x7fa4d81161c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:16.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.860+0000 7fa4e0400700 1 -- 172.21.15.123:0/2892763045 >> 172.21.15.123:0/2892763045 conn(0x7fa4d80f89b0 msgr2=0x7fa4d81015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:38:16.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.860+0000 7fa4e0400700 1 -- 172.21.15.123:0/2892763045 shutdown_connections 2024-03-30T04:38:16.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.860+0000 7fa4e0400700 1 -- 172.21.15.123:0/2892763045 wait complete. 2024-03-30T04:38:16.860 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 32 2024-03-30T04:38:16.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.994+0000 7efd99a00700 1 -- 172.21.15.123:0/3381236424 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd941037a0 msgr2=0x7efd940fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:16.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.994+0000 7efd99a00700 1 --2- 172.21.15.123:0/3381236424 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd941037a0 0x7efd940fce10 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7efd80004940 tx=0x7efd80009e30 comp rx=0 tx=0).stop 2024-03-30T04:38:16.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.994+0000 7efd99a00700 1 -- 172.21.15.123:0/3381236424 shutdown_connections 2024-03-30T04:38:16.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.994+0000 7efd99a00700 1 --2- 172.21.15.123:0/3381236424 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd941037a0 0x7efd940fce10 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:16.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.994+0000 7efd99a00700 1 --2- 172.21.15.123:0/3381236424 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efd94102e00 0x7efd941031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:16.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.994+0000 7efd99a00700 1 -- 172.21.15.123:0/3381236424 >> 172.21.15.123:0/3381236424 conn(0x7efd940f89b0 msgr2=0x7efd940fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:38:16.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.994+0000 7efd99a00700 1 -- 172.21.15.123:0/3381236424 shutdown_connections 2024-03-30T04:38:16.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.994+0000 7efd99a00700 1 -- 172.21.15.123:0/3381236424 wait complete. 2024-03-30T04:38:16.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.995+0000 7efd99a00700 1 Processor -- start 2024-03-30T04:38:16.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.995+0000 7efd99a00700 1 -- start start 2024-03-30T04:38:16.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.995+0000 7efd99a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd94102e00 0x7efd94115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:38:16.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.995+0000 7efd99a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efd941037a0 0x7efd941161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:38:16.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.995+0000 7efd99a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd94116810 con 0x7efd94102e00 2024-03-30T04:38:16.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.995+0000 7efd99a00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd9410fbe0 con 0x7efd941037a0 2024-03-30T04:38:16.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.996+0000 7efd92a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd94102e00 0x7efd94115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:38:16.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.996+0000 7efd92a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd94102e00 0x7efd94115c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47748/0 (socket says 172.21.15.123:47748) 2024-03-30T04:38:16.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.996+0000 7efd92a00700 1 -- 172.21.15.123:0/2807375345 learned_addr learned my addr 172.21.15.123:0/2807375345 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:38:16.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.996+0000 7efd92000700 1 --2- 172.21.15.123:0/2807375345 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efd941037a0 0x7efd941161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:38:16.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.996+0000 7efd92a00700 1 -- 172.21.15.123:0/2807375345 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efd941037a0 msgr2=0x7efd941161c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:16.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.996+0000 7efd92a00700 1 --2- 172.21.15.123:0/2807375345 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efd941037a0 0x7efd941161c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:16.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.996+0000 7efd92a00700 1 -- 172.21.15.123:0/2807375345 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd80005040 con 0x7efd94102e00 2024-03-30T04:38:16.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.996+0000 7efd92a00700 1 --2- 172.21.15.123:0/2807375345 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd94102e00 0x7efd94115c80 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7efd8400eb40 tx=0x7efd8400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:38:16.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.996+0000 7efd8aa00700 1 -- 172.21.15.123:0/2807375345 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd84007cc0 con 0x7efd94102e00 2024-03-30T04:38:16.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.996+0000 7efd99a00700 1 -- 172.21.15.123:0/2807375345 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd9410fe60 con 0x7efd94102e00 2024-03-30T04:38:16.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.996+0000 7efd8aa00700 1 -- 172.21.15.123:0/2807375345 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efd8400f040 con 0x7efd94102e00 2024-03-30T04:38:16.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.997+0000 7efd8aa00700 1 -- 172.21.15.123:0/2807375345 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd8400c990 con 0x7efd94102e00 2024-03-30T04:38:16.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.997+0000 7efd99a00700 1 -- 172.21.15.123:0/2807375345 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd94110420 con 0x7efd94102e00 2024-03-30T04:38:16.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.998+0000 7efd99a00700 1 -- 172.21.15.123:0/2807375345 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd940fe550 con 0x7efd94102e00 2024-03-30T04:38:16.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.999+0000 7efd8aa00700 1 -- 172.21.15.123:0/2807375345 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7efd8400cc00 con 0x7efd94102e00 2024-03-30T04:38:16.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:16.999+0000 7efd8aa00700 1 --2- 172.21.15.123:0/2807375345 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7efd7806cc30 0x7efd7806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:38:16.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.000+0000 7efd8aa00700 1 -- 172.21.15.123:0/2807375345 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7efd8408bc50 con 0x7efd94102e00 2024-03-30T04:38:16.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.000+0000 7efd92000700 1 --2- 172.21.15.123:0/2807375345 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7efd7806cc30 0x7efd7806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:38:16.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.000+0000 7efd92000700 1 --2- 172.21.15.123:0/2807375345 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7efd7806cc30 0x7efd7806f0e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7efd94111520 tx=0x7efd80009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:38:17.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.004+0000 7efd8aa00700 1 -- 172.21.15.123:0/2807375345 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7efd84059750 con 0x7efd94102e00 2024-03-30T04:38:17.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.225+0000 7efd99a00700 1 -- 172.21.15.123:0/2807375345 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7efd9404bf40 con 0x7efd7806cc30 2024-03-30T04:38:17.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.226+0000 7efd8aa00700 1 -- 172.21.15.123:0/2807375345 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7efd9404bf40 con 0x7efd7806cc30 2024-03-30T04:38:17.225 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:38:17.225 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:38:17.225 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:38:17.225 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T04:38:17.225 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:38:17.225 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T04:38:17.225 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T04:38:17.225 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T04:38:17.225 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T04:38:17.225 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:38:17.226 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "15/21 daemons upgraded", 2024-03-30T04:38:17.226 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T04:38:17.226 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:38:17.226 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:38:17.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.228+0000 7efd99a00700 1 -- 172.21.15.123:0/2807375345 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7efd7806cc30 msgr2=0x7efd7806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:17.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.228+0000 7efd99a00700 1 --2- 172.21.15.123:0/2807375345 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7efd7806cc30 0x7efd7806f0e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7efd94111520 tx=0x7efd80009e30 comp rx=0 tx=0).stop 2024-03-30T04:38:17.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.228+0000 7efd99a00700 1 -- 172.21.15.123:0/2807375345 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd94102e00 msgr2=0x7efd94115c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:17.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.228+0000 7efd99a00700 1 --2- 172.21.15.123:0/2807375345 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd94102e00 0x7efd94115c80 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7efd8400eb40 tx=0x7efd8400b040 comp rx=0 tx=0).stop 2024-03-30T04:38:17.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.228+0000 7efd99a00700 1 -- 172.21.15.123:0/2807375345 shutdown_connections 2024-03-30T04:38:17.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.228+0000 7efd99a00700 1 --2- 172.21.15.123:0/2807375345 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7efd7806cc30 0x7efd7806f0e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:17.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.228+0000 7efd99a00700 1 --2- 172.21.15.123:0/2807375345 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd94102e00 0x7efd94115c80 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:17.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.228+0000 7efd99a00700 1 --2- 172.21.15.123:0/2807375345 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efd941037a0 0x7efd941161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:17.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.229+0000 7efd99a00700 1 -- 172.21.15.123:0/2807375345 >> 172.21.15.123:0/2807375345 conn(0x7efd940f89b0 msgr2=0x7efd940f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:38:17.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.229+0000 7efd99a00700 1 -- 172.21.15.123:0/2807375345 shutdown_connections 2024-03-30T04:38:17.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.229+0000 7efd99a00700 1 -- 172.21.15.123:0/2807375345 wait complete. 2024-03-30T04:38:17.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.361+0000 7fa641600700 1 -- 172.21.15.123:0/1324485870 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa63c1037c0 msgr2=0x7fa63c0fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:17.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.361+0000 7fa641600700 1 --2- 172.21.15.123:0/1324485870 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa63c1037c0 0x7fa63c0fce30 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa628004940 tx=0x7fa628009e30 comp rx=0 tx=0).stop 2024-03-30T04:38:17.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.361+0000 7fa641600700 1 -- 172.21.15.123:0/1324485870 shutdown_connections 2024-03-30T04:38:17.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.361+0000 7fa641600700 1 --2- 172.21.15.123:0/1324485870 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa63c1037c0 0x7fa63c0fce30 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:17.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.361+0000 7fa641600700 1 --2- 172.21.15.123:0/1324485870 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa63c102e20 0x7fa63c1031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:17.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.361+0000 7fa641600700 1 -- 172.21.15.123:0/1324485870 >> 172.21.15.123:0/1324485870 conn(0x7fa63c0f8970 msgr2=0x7fa63c0fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:38:17.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.361+0000 7fa641600700 1 -- 172.21.15.123:0/1324485870 shutdown_connections 2024-03-30T04:38:17.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.361+0000 7fa641600700 1 -- 172.21.15.123:0/1324485870 wait complete. 2024-03-30T04:38:17.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.362+0000 7fa641600700 1 Processor -- start 2024-03-30T04:38:17.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.362+0000 7fa641600700 1 -- start start 2024-03-30T04:38:17.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.362+0000 7fa641600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa63c102e20 0x7fa63c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:38:17.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.363+0000 7fa641600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa63c1037c0 0x7fa63c070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:38:17.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.363+0000 7fa641600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa63c070d90 con 0x7fa63c1037c0 2024-03-30T04:38:17.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.363+0000 7fa641600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa63c070ed0 con 0x7fa63c102e20 2024-03-30T04:38:17.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.363+0000 7fa63a000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa63c1037c0 0x7fa63c070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:38:17.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.363+0000 7fa63a000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa63c1037c0 0x7fa63c070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47760/0 (socket says 172.21.15.123:47760) 2024-03-30T04:38:17.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.363+0000 7fa63a000700 1 -- 172.21.15.123:0/2170004237 learned_addr learned my addr 172.21.15.123:0/2170004237 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:38:17.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.363+0000 7fa63aa00700 1 --2- 172.21.15.123:0/2170004237 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa63c102e20 0x7fa63c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:38:17.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.363+0000 7fa63a000700 1 -- 172.21.15.123:0/2170004237 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa63c102e20 msgr2=0x7fa63c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:17.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.363+0000 7fa63a000700 1 --2- 172.21.15.123:0/2170004237 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa63c102e20 0x7fa63c0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:17.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.363+0000 7fa63a000700 1 -- 172.21.15.123:0/2170004237 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa628005040 con 0x7fa63c1037c0 2024-03-30T04:38:17.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.363+0000 7fa63a000700 1 --2- 172.21.15.123:0/2170004237 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa63c1037c0 0x7fa63c070800 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa628009390 tx=0x7fa628009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:38:17.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.364+0000 7fa632a00700 1 -- 172.21.15.123:0/2170004237 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa62800e040 con 0x7fa63c1037c0 2024-03-30T04:38:17.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.364+0000 7fa641600700 1 -- 172.21.15.123:0/2170004237 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa63c071120 con 0x7fa63c1037c0 2024-03-30T04:38:17.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.364+0000 7fa632a00700 1 -- 172.21.15.123:0/2170004237 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa628013420 con 0x7fa63c1037c0 2024-03-30T04:38:17.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.364+0000 7fa632a00700 1 -- 172.21.15.123:0/2170004237 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa62801c630 con 0x7fa63c1037c0 2024-03-30T04:38:17.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.364+0000 7fa641600700 1 -- 172.21.15.123:0/2170004237 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa63c11a210 con 0x7fa63c1037c0 2024-03-30T04:38:17.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.365+0000 7fa641600700 1 -- 172.21.15.123:0/2170004237 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa63c0fe570 con 0x7fa63c1037c0 2024-03-30T04:38:17.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.366+0000 7fa632a00700 1 -- 172.21.15.123:0/2170004237 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fa628013a00 con 0x7fa63c1037c0 2024-03-30T04:38:17.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.367+0000 7fa632a00700 1 --2- 172.21.15.123:0/2170004237 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa62006cc30 0x7fa62006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:38:17.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.367+0000 7fa632a00700 1 -- 172.21.15.123:0/2170004237 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa62808d130 con 0x7fa63c1037c0 2024-03-30T04:38:17.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.367+0000 7fa63aa00700 1 --2- 172.21.15.123:0/2170004237 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa62006cc30 0x7fa62006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:38:17.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.368+0000 7fa63aa00700 1 --2- 172.21.15.123:0/2170004237 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa62006cc30 0x7fa62006f0e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa62c004f40 tx=0x7fa62c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:38:17.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.372+0000 7fa632a00700 1 -- 172.21.15.123:0/2170004237 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa628057170 con 0x7fa63c1037c0 2024-03-30T04:38:17.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.656+0000 7fa641600700 1 -- 172.21.15.123:0/2170004237 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fa63c05f5f0 con 0x7fa63c1037c0 2024-03-30T04:38:17.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.657+0000 7fa632a00700 1 -- 172.21.15.123:0/2170004237 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fa62805a790 con 0x7fa63c1037c0 2024-03-30T04:38:17.656 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T04:38:17.656 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T04:38:17.656 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T04:38:17.656 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T04:38:17.656 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T04:38:17.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.659+0000 7fa641600700 1 -- 172.21.15.123:0/2170004237 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa62006cc30 msgr2=0x7fa62006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:17.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.659+0000 7fa641600700 1 --2- 172.21.15.123:0/2170004237 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa62006cc30 0x7fa62006f0e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa62c004f40 tx=0x7fa62c00b040 comp rx=0 tx=0).stop 2024-03-30T04:38:17.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.659+0000 7fa641600700 1 -- 172.21.15.123:0/2170004237 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa63c1037c0 msgr2=0x7fa63c070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:17.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.659+0000 7fa641600700 1 --2- 172.21.15.123:0/2170004237 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa63c1037c0 0x7fa63c070800 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa628009390 tx=0x7fa628009e30 comp rx=0 tx=0).stop 2024-03-30T04:38:17.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.659+0000 7fa641600700 1 -- 172.21.15.123:0/2170004237 shutdown_connections 2024-03-30T04:38:17.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.659+0000 7fa641600700 1 --2- 172.21.15.123:0/2170004237 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa62006cc30 0x7fa62006f0e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:17.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.659+0000 7fa641600700 1 --2- 172.21.15.123:0/2170004237 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa63c102e20 0x7fa63c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:17.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.659+0000 7fa641600700 1 --2- 172.21.15.123:0/2170004237 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa63c1037c0 0x7fa63c070800 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:17.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.659+0000 7fa641600700 1 -- 172.21.15.123:0/2170004237 >> 172.21.15.123:0/2170004237 conn(0x7fa63c0f8970 msgr2=0x7fa63c0f9a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:38:17.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.659+0000 7fa641600700 1 -- 172.21.15.123:0/2170004237 shutdown_connections 2024-03-30T04:38:17.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:17.660+0000 7fa641600700 1 -- 172.21.15.123:0/2170004237 wait complete. 2024-03-30T04:38:17.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:17 smithi123 ceph-mon[139369]: from='client.34466 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:38:17.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:17 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3041323192' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:38:17.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:17 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2892763045' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:38:17.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:17 smithi135 ceph-mon[106011]: from='client.34466 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:38:17.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:17 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3041323192' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:38:17.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:17 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2892763045' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:38:18.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:18 smithi123 ceph-mon[139369]: from='client.34478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:38:18.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:18 smithi123 ceph-mon[139369]: pgmap v213: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:18.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:18 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2170004237' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:38:18.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:18 smithi135 ceph-mon[106011]: from='client.34478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:38:18.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:18 smithi135 ceph-mon[106011]: pgmap v213: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:18.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:18 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2170004237' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:38:20.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:20 smithi123 ceph-mon[139369]: pgmap v214: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:20.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:20 smithi135 ceph-mon[106011]: pgmap v214: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:22.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:22 smithi135 ceph-mon[106011]: pgmap v215: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:22.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:22 smithi123 ceph-mon[139369]: pgmap v215: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:24.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:24 smithi135 ceph-mon[106011]: pgmap v216: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:24.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:24 smithi123 ceph-mon[139369]: pgmap v216: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:26.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:26 smithi135 ceph-mon[106011]: pgmap v217: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:26.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:26 smithi123 ceph-mon[139369]: pgmap v217: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:28.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:28 smithi135 ceph-mon[106011]: pgmap v218: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:28.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:28 smithi123 ceph-mon[139369]: pgmap v218: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:30.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:38:30.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:30 smithi123 ceph-mon[139369]: pgmap v219: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:30.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:30 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:38:30.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:30 smithi135 ceph-mon[106011]: pgmap v219: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:32.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:32 smithi135 ceph-mon[106011]: pgmap v220: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:32.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:32 smithi123 ceph-mon[139369]: pgmap v220: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:34.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:34 smithi135 ceph-mon[106011]: pgmap v221: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:34.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:34 smithi123 ceph-mon[139369]: pgmap v221: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:36.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:36 smithi135 ceph-mon[106011]: pgmap v222: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:36.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:36 smithi123 ceph-mon[139369]: pgmap v222: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:38.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:38 smithi135 ceph-mon[106011]: pgmap v223: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:38.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:38 smithi123 ceph-mon[139369]: pgmap v223: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:40.718 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:40 smithi123 ceph-mon[139369]: pgmap v224: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:40.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:40 smithi135 ceph-mon[106011]: pgmap v224: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:42.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:42 smithi135 ceph-mon[106011]: pgmap v225: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:42.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:42 smithi123 ceph-mon[139369]: pgmap v225: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:44.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:44 smithi135 ceph-mon[106011]: pgmap v226: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:44.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:44 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:38:44.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:44 smithi123 ceph-mon[139369]: pgmap v226: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:44.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:38:46.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:46 smithi135 ceph-mon[106011]: pgmap v227: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:46.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:46 smithi123 ceph-mon[139369]: pgmap v227: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:47.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:47.807+0000 7fa7a1800700 1 -- 172.21.15.123:0/1901747382 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa79c102e00 msgr2=0x7fa79c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:47.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:47.807+0000 7fa7a1800700 1 --2- 172.21.15.123:0/1901747382 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa79c102e00 0x7fa79c1031d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa78c004940 tx=0x7fa78c009e30 comp rx=0 tx=0).stop 2024-03-30T04:38:47.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:47.807+0000 7fa7a1800700 1 -- 172.21.15.123:0/1901747382 shutdown_connections 2024-03-30T04:38:47.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:47.807+0000 7fa7a1800700 1 --2- 172.21.15.123:0/1901747382 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa79c1037a0 0x7fa79c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:47.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:47.807+0000 7fa7a1800700 1 --2- 172.21.15.123:0/1901747382 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa79c102e00 0x7fa79c1031d0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:47.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:47.807+0000 7fa7a1800700 1 -- 172.21.15.123:0/1901747382 >> 172.21.15.123:0/1901747382 conn(0x7fa79c0f89b0 msgr2=0x7fa79c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:38:47.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:47.807+0000 7fa7a1800700 1 -- 172.21.15.123:0/1901747382 shutdown_connections 2024-03-30T04:38:47.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:47.808+0000 7fa7a1800700 1 -- 172.21.15.123:0/1901747382 wait complete. 2024-03-30T04:38:47.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:47.808+0000 7fa7a1800700 1 Processor -- start 2024-03-30T04:38:47.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:47.808+0000 7fa7a1800700 1 -- start start 2024-03-30T04:38:47.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:47.809+0000 7fa7a1800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa79c102e00 0x7fa79c115d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:38:47.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:47.809+0000 7fa7a1800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa79c1037a0 0x7fa79c116250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:38:47.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:47.809+0000 7fa7a1800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa79c1168a0 con 0x7fa79c102e00 2024-03-30T04:38:47.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:47.809+0000 7fa7a1800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa79c10fc70 con 0x7fa79c1037a0 2024-03-30T04:38:47.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:47.809+0000 7fa79aa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa79c102e00 0x7fa79c115d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:38:47.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:47.809+0000 7fa79aa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa79c102e00 0x7fa79c115d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33560/0 (socket says 172.21.15.123:33560) 2024-03-30T04:38:47.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:47.809+0000 7fa79aa00700 1 -- 172.21.15.123:0/2095065944 learned_addr learned my addr 172.21.15.123:0/2095065944 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:38:47.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:47.809+0000 7fa79a000700 1 --2- 172.21.15.123:0/2095065944 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa79c1037a0 0x7fa79c116250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:38:47.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:47.809+0000 7fa79aa00700 1 -- 172.21.15.123:0/2095065944 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa79c1037a0 msgr2=0x7fa79c116250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:47.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:47.809+0000 7fa79aa00700 1 --2- 172.21.15.123:0/2095065944 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa79c1037a0 0x7fa79c116250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:47.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:47.809+0000 7fa79aa00700 1 -- 172.21.15.123:0/2095065944 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa78c005040 con 0x7fa79c102e00 2024-03-30T04:38:47.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:47.810+0000 7fa79aa00700 1 --2- 172.21.15.123:0/2095065944 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa79c102e00 0x7fa79c115d10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa78c009390 tx=0x7fa78c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:38:47.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:47.810+0000 7fa792a00700 1 -- 172.21.15.123:0/2095065944 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa78c00e040 con 0x7fa79c102e00 2024-03-30T04:38:47.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:47.810+0000 7fa792a00700 1 -- 172.21.15.123:0/2095065944 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa78c013420 con 0x7fa79c102e00 2024-03-30T04:38:47.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:47.810+0000 7fa7a1800700 1 -- 172.21.15.123:0/2095065944 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa79c10fef0 con 0x7fa79c102e00 2024-03-30T04:38:47.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:47.810+0000 7fa792a00700 1 -- 172.21.15.123:0/2095065944 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa78c01c630 con 0x7fa79c102e00 2024-03-30T04:38:47.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:47.810+0000 7fa7a1800700 1 -- 172.21.15.123:0/2095065944 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa79c1104b0 con 0x7fa79c102e00 2024-03-30T04:38:47.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:47.811+0000 7fa7a1800700 1 -- 172.21.15.123:0/2095065944 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa79c0fe550 con 0x7fa79c102e00 2024-03-30T04:38:47.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:47.812+0000 7fa792a00700 1 -- 172.21.15.123:0/2095065944 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fa78c0139e0 con 0x7fa79c102e00 2024-03-30T04:38:47.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:47.813+0000 7fa792a00700 1 --2- 172.21.15.123:0/2095065944 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa78006cc30 0x7fa78006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:38:47.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:47.813+0000 7fa792a00700 1 -- 172.21.15.123:0/2095065944 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa78c08d250 con 0x7fa79c102e00 2024-03-30T04:38:47.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:47.813+0000 7fa79a000700 1 --2- 172.21.15.123:0/2095065944 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa78006cc30 0x7fa78006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:38:47.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:47.814+0000 7fa79a000700 1 --2- 172.21.15.123:0/2095065944 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa78006cc30 0x7fa78006f0e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa79c1115b0 tx=0x7fa788009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:38:47.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:47.818+0000 7fa792a00700 1 -- 172.21.15.123:0/2095065944 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa78c05add0 con 0x7fa79c102e00 2024-03-30T04:38:48.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.050+0000 7fa7a1800700 1 -- 172.21.15.123:0/2095065944 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa79c04bf70 con 0x7fa78006cc30 2024-03-30T04:38:48.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.052+0000 7fa792a00700 1 -- 172.21.15.123:0/2095065944 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fa788004e30 con 0x7fa78006cc30 2024-03-30T04:38:48.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.054+0000 7fa7a1800700 1 -- 172.21.15.123:0/2095065944 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa78006cc30 msgr2=0x7fa78006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:48.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.054+0000 7fa7a1800700 1 --2- 172.21.15.123:0/2095065944 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa78006cc30 0x7fa78006f0e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa79c1115b0 tx=0x7fa788009ee0 comp rx=0 tx=0).stop 2024-03-30T04:38:48.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.054+0000 7fa7a1800700 1 -- 172.21.15.123:0/2095065944 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa79c102e00 msgr2=0x7fa79c115d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:48.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.054+0000 7fa7a1800700 1 --2- 172.21.15.123:0/2095065944 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa79c102e00 0x7fa79c115d10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa78c009390 tx=0x7fa78c009e30 comp rx=0 tx=0).stop 2024-03-30T04:38:48.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.054+0000 7fa7a1800700 1 -- 172.21.15.123:0/2095065944 shutdown_connections 2024-03-30T04:38:48.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.054+0000 7fa7a1800700 1 --2- 172.21.15.123:0/2095065944 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa78006cc30 0x7fa78006f0e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:48.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.054+0000 7fa7a1800700 1 --2- 172.21.15.123:0/2095065944 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa79c102e00 0x7fa79c115d10 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:48.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.054+0000 7fa7a1800700 1 --2- 172.21.15.123:0/2095065944 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa79c1037a0 0x7fa79c116250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:48.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.054+0000 7fa7a1800700 1 -- 172.21.15.123:0/2095065944 >> 172.21.15.123:0/2095065944 conn(0x7fa79c0f89b0 msgr2=0x7fa79c0704f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:38:48.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.054+0000 7fa7a1800700 1 -- 172.21.15.123:0/2095065944 shutdown_connections 2024-03-30T04:38:48.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.054+0000 7fa7a1800700 1 -- 172.21.15.123:0/2095065944 wait complete. 2024-03-30T04:38:48.068 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:38:48.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.190+0000 7ff4c7000700 1 -- 172.21.15.123:0/193323534 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4c00fe830 msgr2=0x7ff4c0106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:48.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.190+0000 7ff4c7000700 1 --2- 172.21.15.123:0/193323534 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4c00fe830 0x7ff4c0106f00 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac004940 tx=0x7ff4ac009e30 comp rx=0 tx=0).stop 2024-03-30T04:38:48.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.190+0000 7ff4c7000700 1 -- 172.21.15.123:0/193323534 shutdown_connections 2024-03-30T04:38:48.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.190+0000 7ff4c7000700 1 --2- 172.21.15.123:0/193323534 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4c00fe830 0x7ff4c0106f00 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:48.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.190+0000 7ff4c7000700 1 --2- 172.21.15.123:0/193323534 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff4c00fde90 0x7ff4c00fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:48.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.190+0000 7ff4c7000700 1 -- 172.21.15.123:0/193323534 >> 172.21.15.123:0/193323534 conn(0x7ff4c0072bc0 msgr2=0x7ff4c0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:38:48.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.190+0000 7ff4c7000700 1 -- 172.21.15.123:0/193323534 shutdown_connections 2024-03-30T04:38:48.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.190+0000 7ff4c7000700 1 -- 172.21.15.123:0/193323534 wait complete. 2024-03-30T04:38:48.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.191+0000 7ff4c7000700 1 Processor -- start 2024-03-30T04:38:48.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.191+0000 7ff4c7000700 1 -- start start 2024-03-30T04:38:48.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.192+0000 7ff4c7000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff4c00fde90 0x7ff4c0115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:38:48.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.192+0000 7ff4c7000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4c00fe830 0x7ff4c01161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:38:48.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.192+0000 7ff4c7000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c0116810 con 0x7ff4c00fe830 2024-03-30T04:38:48.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.192+0000 7ff4c7000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c010fc30 con 0x7ff4c00fde90 2024-03-30T04:38:48.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.192+0000 7ff4bf400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4c00fe830 0x7ff4c01161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:38:48.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.192+0000 7ff4bf400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4c00fe830 0x7ff4c01161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33584/0 (socket says 172.21.15.123:33584) 2024-03-30T04:38:48.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.192+0000 7ff4bf400700 1 -- 172.21.15.123:0/3473507458 learned_addr learned my addr 172.21.15.123:0/3473507458 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:38:48.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.192+0000 7ff4bfe00700 1 --2- 172.21.15.123:0/3473507458 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff4c00fde90 0x7ff4c0115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:38:48.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.192+0000 7ff4bf400700 1 -- 172.21.15.123:0/3473507458 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff4c00fde90 msgr2=0x7ff4c0115c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:48.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.192+0000 7ff4bf400700 1 --2- 172.21.15.123:0/3473507458 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff4c00fde90 0x7ff4c0115c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:48.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.192+0000 7ff4bf400700 1 -- 172.21.15.123:0/3473507458 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4ac005040 con 0x7ff4c00fe830 2024-03-30T04:38:48.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.192+0000 7ff4bfe00700 1 --2- 172.21.15.123:0/3473507458 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff4c00fde90 0x7ff4c0115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:38:48.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.192+0000 7ff4bf400700 1 --2- 172.21.15.123:0/3473507458 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4c00fe830 0x7ff4c01161c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac000c00 tx=0x7ff4ac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:38:48.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.193+0000 7ff4b7e00700 1 -- 172.21.15.123:0/3473507458 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4ac021070 con 0x7ff4c00fe830 2024-03-30T04:38:48.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.193+0000 7ff4b7e00700 1 -- 172.21.15.123:0/3473507458 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff4ac013e00 con 0x7ff4c00fe830 2024-03-30T04:38:48.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.193+0000 7ff4c7000700 1 -- 172.21.15.123:0/3473507458 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4c010feb0 con 0x7ff4c00fe830 2024-03-30T04:38:48.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.193+0000 7ff4b7e00700 1 -- 172.21.15.123:0/3473507458 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4ac00e070 con 0x7ff4c00fe830 2024-03-30T04:38:48.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.193+0000 7ff4c7000700 1 -- 172.21.15.123:0/3473507458 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4c01103f0 con 0x7ff4c00fe830 2024-03-30T04:38:48.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.194+0000 7ff4c7000700 1 -- 172.21.15.123:0/3473507458 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4c0106a20 con 0x7ff4c00fe830 2024-03-30T04:38:48.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.195+0000 7ff4b7e00700 1 -- 172.21.15.123:0/3473507458 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7ff4ac013950 con 0x7ff4c00fe830 2024-03-30T04:38:48.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.196+0000 7ff4b7e00700 1 --2- 172.21.15.123:0/3473507458 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ff4a406cc30 0x7ff4a406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:38:48.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.196+0000 7ff4b7e00700 1 -- 172.21.15.123:0/3473507458 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff4ac08d120 con 0x7ff4c00fe830 2024-03-30T04:38:48.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.196+0000 7ff4bfe00700 1 --2- 172.21.15.123:0/3473507458 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ff4a406cc30 0x7ff4a406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:38:48.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.196+0000 7ff4bfe00700 1 --2- 172.21.15.123:0/3473507458 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ff4a406cc30 0x7ff4a406f0e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff4b0004600 tx=0x7ff4b000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:38:48.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.200+0000 7ff4b7e00700 1 -- 172.21.15.123:0/3473507458 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff4ac0570e0 con 0x7ff4c00fe830 2024-03-30T04:38:48.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.419+0000 7ff4c7000700 1 -- 172.21.15.123:0/3473507458 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff4c01112d0 con 0x7ff4a406cc30 2024-03-30T04:38:48.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.420+0000 7ff4b7e00700 1 -- 172.21.15.123:0/3473507458 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ff4c01112d0 con 0x7ff4a406cc30 2024-03-30T04:38:48.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.422+0000 7ff4c7000700 1 -- 172.21.15.123:0/3473507458 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ff4a406cc30 msgr2=0x7ff4a406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:48.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.422+0000 7ff4c7000700 1 --2- 172.21.15.123:0/3473507458 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ff4a406cc30 0x7ff4a406f0e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff4b0004600 tx=0x7ff4b000b040 comp rx=0 tx=0).stop 2024-03-30T04:38:48.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.422+0000 7ff4c7000700 1 -- 172.21.15.123:0/3473507458 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4c00fe830 msgr2=0x7ff4c01161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:48.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.422+0000 7ff4c7000700 1 --2- 172.21.15.123:0/3473507458 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4c00fe830 0x7ff4c01161c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac000c00 tx=0x7ff4ac009e30 comp rx=0 tx=0).stop 2024-03-30T04:38:48.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.423+0000 7ff4c7000700 1 -- 172.21.15.123:0/3473507458 shutdown_connections 2024-03-30T04:38:48.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.423+0000 7ff4c7000700 1 --2- 172.21.15.123:0/3473507458 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ff4a406cc30 0x7ff4a406f0e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:48.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.423+0000 7ff4c7000700 1 --2- 172.21.15.123:0/3473507458 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff4c00fde90 0x7ff4c0115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:48.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.423+0000 7ff4c7000700 1 --2- 172.21.15.123:0/3473507458 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4c00fe830 0x7ff4c01161c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:48.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.423+0000 7ff4c7000700 1 -- 172.21.15.123:0/3473507458 >> 172.21.15.123:0/3473507458 conn(0x7ff4c0072bc0 msgr2=0x7ff4c00fd510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:38:48.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.423+0000 7ff4c7000700 1 -- 172.21.15.123:0/3473507458 shutdown_connections 2024-03-30T04:38:48.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.423+0000 7ff4c7000700 1 -- 172.21.15.123:0/3473507458 wait complete. 2024-03-30T04:38:48.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.557+0000 7f0b86c00700 1 -- 172.21.15.123:0/434041766 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b80070c70 msgr2=0x7f0b80109110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:48.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.557+0000 7f0b86c00700 1 --2- 172.21.15.123:0/434041766 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b80070c70 0x7f0b80109110 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c004970 tx=0x7f0b6c009e30 comp rx=0 tx=0).stop 2024-03-30T04:38:48.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.557+0000 7f0b86c00700 1 -- 172.21.15.123:0/434041766 shutdown_connections 2024-03-30T04:38:48.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.557+0000 7f0b86c00700 1 --2- 172.21.15.123:0/434041766 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b80070c70 0x7f0b80109110 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:48.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.557+0000 7f0b86c00700 1 --2- 172.21.15.123:0/434041766 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0b80070360 0x7f0b80070730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:48.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.557+0000 7f0b86c00700 1 -- 172.21.15.123:0/434041766 >> 172.21.15.123:0/434041766 conn(0x7f0b800f89b0 msgr2=0x7f0b800fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:38:48.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.557+0000 7f0b86c00700 1 -- 172.21.15.123:0/434041766 shutdown_connections 2024-03-30T04:38:48.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.557+0000 7f0b86c00700 1 -- 172.21.15.123:0/434041766 wait complete. 2024-03-30T04:38:48.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.558+0000 7f0b86c00700 1 Processor -- start 2024-03-30T04:38:48.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.558+0000 7f0b86c00700 1 -- start start 2024-03-30T04:38:48.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.558+0000 7f0b86c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b80070360 0x7f0b80115ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:38:48.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.558+0000 7f0b86c00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0b80070c70 0x7f0b801161e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:38:48.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.558+0000 7f0b86c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b80116830 con 0x7f0b80070360 2024-03-30T04:38:48.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.558+0000 7f0b86c00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b8010fc00 con 0x7f0b80070c70 2024-03-30T04:38:48.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.558+0000 7f0b7fe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b80070360 0x7f0b80115ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:38:48.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.558+0000 7f0b7fe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b80070360 0x7f0b80115ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33602/0 (socket says 172.21.15.123:33602) 2024-03-30T04:38:48.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.558+0000 7f0b7fe00700 1 -- 172.21.15.123:0/1859842632 learned_addr learned my addr 172.21.15.123:0/1859842632 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:38:48.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.559+0000 7f0b7fe00700 1 -- 172.21.15.123:0/1859842632 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0b80070c70 msgr2=0x7f0b801161e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:48.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.559+0000 7f0b7fe00700 1 --2- 172.21.15.123:0/1859842632 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0b80070c70 0x7f0b801161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:48.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.559+0000 7f0b7fe00700 1 -- 172.21.15.123:0/1859842632 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b6c005040 con 0x7f0b80070360 2024-03-30T04:38:48.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.559+0000 7f0b7fe00700 1 --2- 172.21.15.123:0/1859842632 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b80070360 0x7f0b80115ca0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0b70002a50 tx=0x7f0b700046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:38:48.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.559+0000 7f0b77e00700 1 -- 172.21.15.123:0/1859842632 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b70007cc0 con 0x7f0b80070360 2024-03-30T04:38:48.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.559+0000 7f0b77e00700 1 -- 172.21.15.123:0/1859842632 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0b7000f040 con 0x7f0b80070360 2024-03-30T04:38:48.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.559+0000 7f0b86c00700 1 -- 172.21.15.123:0/1859842632 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b8010fee0 con 0x7f0b80070360 2024-03-30T04:38:48.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.559+0000 7f0b77e00700 1 -- 172.21.15.123:0/1859842632 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b70009540 con 0x7f0b80070360 2024-03-30T04:38:48.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.559+0000 7f0b86c00700 1 -- 172.21.15.123:0/1859842632 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b80110420 con 0x7f0b80070360 2024-03-30T04:38:48.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.560+0000 7f0b86c00700 1 -- 172.21.15.123:0/1859842632 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b80103bd0 con 0x7f0b80070360 2024-03-30T04:38:48.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.561+0000 7f0b77e00700 1 -- 172.21.15.123:0/1859842632 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f0b700096a0 con 0x7f0b80070360 2024-03-30T04:38:48.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.562+0000 7f0b77e00700 1 --2- 172.21.15.123:0/1859842632 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0b6406cc30 0x7f0b6406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:38:48.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.562+0000 7f0b77e00700 1 -- 172.21.15.123:0/1859842632 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f0b7008be60 con 0x7f0b80070360 2024-03-30T04:38:48.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.563+0000 7f0b7f400700 1 --2- 172.21.15.123:0/1859842632 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0b6406cc30 0x7f0b6406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:38:48.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.563+0000 7f0b7f400700 1 --2- 172.21.15.123:0/1859842632 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0b6406cc30 0x7f0b6406f0e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c000c00 tx=0x7f0b6c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:38:48.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.567+0000 7f0b77e00700 1 -- 172.21.15.123:0/1859842632 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0b700598b0 con 0x7f0b80070360 2024-03-30T04:38:48.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.779+0000 7f0b86c00700 1 -- 172.21.15.123:0/1859842632 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f0b80111360 con 0x7f0b6406cc30 2024-03-30T04:38:48.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.789+0000 7f0b77e00700 1 -- 172.21.15.123:0/1859842632 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f0b80111360 con 0x7f0b6406cc30 2024-03-30T04:38:48.788 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:38:48.788 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (7m) 53s ago 20m 15.0M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:38:48.788 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (5m) 53s ago 20m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T04:38:48.788 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (5m) 35s ago 19m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T04:38:48.788 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (7m) 53s ago 20m 74.0M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T04:38:48.788 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (59s) 53s ago 14m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T04:38:48.788 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (80s) 53s ago 14m 10.9M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T04:38:48.788 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (14m) 35s ago 14m 10.7M - 17.2.7-797-g5c024335 f77085f6aab7 0dc41326818c 2024-03-30T04:38:48.788 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (39s) 35s ago 14m 9747k - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T04:38:48.788 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (10m) 53s ago 22m 506M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:38:48.788 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (9m) 35s ago 19m 450M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:38:48.788 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (6m) 53s ago 22m 53.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T04:38:48.788 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (5m) 35s ago 19m 42.6M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T04:38:48.789 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (8m) 53s ago 20m 8052k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:38:48.789 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (8m) 35s ago 19m 7880k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:38:48.789 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (4m) 53s ago 18m 104M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T04:38:48.789 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (4m) 53s ago 17m 93.4M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T04:38:48.789 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (3m) 53s ago 17m 86.3M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T04:38:48.789 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (2m) 35s ago 16m 97.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T04:38:48.789 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (2m) 35s ago 16m 97.9M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T04:38:48.789 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (101s) 35s ago 15m 85.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T04:38:48.789 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (8m) 53s ago 20m 53.7M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:38:48.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.791+0000 7f0b86c00700 1 -- 172.21.15.123:0/1859842632 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0b6406cc30 msgr2=0x7f0b6406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:48.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.791+0000 7f0b86c00700 1 --2- 172.21.15.123:0/1859842632 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0b6406cc30 0x7f0b6406f0e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c000c00 tx=0x7f0b6c009e30 comp rx=0 tx=0).stop 2024-03-30T04:38:48.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.791+0000 7f0b86c00700 1 -- 172.21.15.123:0/1859842632 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b80070360 msgr2=0x7f0b80115ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:48.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.791+0000 7f0b86c00700 1 --2- 172.21.15.123:0/1859842632 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b80070360 0x7f0b80115ca0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0b70002a50 tx=0x7f0b700046b0 comp rx=0 tx=0).stop 2024-03-30T04:38:48.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.791+0000 7f0b86c00700 1 -- 172.21.15.123:0/1859842632 shutdown_connections 2024-03-30T04:38:48.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.791+0000 7f0b86c00700 1 --2- 172.21.15.123:0/1859842632 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0b6406cc30 0x7f0b6406f0e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:48.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.792+0000 7f0b86c00700 1 --2- 172.21.15.123:0/1859842632 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b80070360 0x7f0b80115ca0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:48.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.792+0000 7f0b86c00700 1 --2- 172.21.15.123:0/1859842632 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0b80070c70 0x7f0b801161e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:48.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.792+0000 7f0b86c00700 1 -- 172.21.15.123:0/1859842632 >> 172.21.15.123:0/1859842632 conn(0x7f0b800f89b0 msgr2=0x7f0b801015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:38:48.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.792+0000 7f0b86c00700 1 -- 172.21.15.123:0/1859842632 shutdown_connections 2024-03-30T04:38:48.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.792+0000 7f0b86c00700 1 -- 172.21.15.123:0/1859842632 wait complete. 2024-03-30T04:38:48.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:48 smithi135 ceph-mon[106011]: pgmap v228: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:48.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.924+0000 7f2005c00700 1 -- 172.21.15.123:0/4000528476 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2000070c10 msgr2=0x7f2000109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:48.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.924+0000 7f2005c00700 1 --2- 172.21.15.123:0/4000528476 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2000070c10 0x7f2000109190 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1fec004970 tx=0x7f1fec009e30 comp rx=0 tx=0).stop 2024-03-30T04:38:48.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.924+0000 7f2005c00700 1 -- 172.21.15.123:0/4000528476 shutdown_connections 2024-03-30T04:38:48.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.924+0000 7f2005c00700 1 --2- 172.21.15.123:0/4000528476 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2000070c10 0x7f2000109190 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:48.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.924+0000 7f2005c00700 1 --2- 172.21.15.123:0/4000528476 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2000070300 0x7f20000706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:48.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.924+0000 7f2005c00700 1 -- 172.21.15.123:0/4000528476 >> 172.21.15.123:0/4000528476 conn(0x7f20000f8970 msgr2=0x7f20000fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:38:48.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.924+0000 7f2005c00700 1 -- 172.21.15.123:0/4000528476 shutdown_connections 2024-03-30T04:38:48.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.924+0000 7f2005c00700 1 -- 172.21.15.123:0/4000528476 wait complete. 2024-03-30T04:38:48.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.925+0000 7f2005c00700 1 Processor -- start 2024-03-30T04:38:48.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.925+0000 7f2005c00700 1 -- start start 2024-03-30T04:38:48.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.925+0000 7f2005c00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2000070300 0x7f2000115d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:38:48.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.925+0000 7f2005c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2000070c10 0x7f2000116270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:38:48.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.925+0000 7f2005c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20001168c0 con 0x7f2000070c10 2024-03-30T04:38:48.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.925+0000 7f2005c00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f200010fc90 con 0x7f2000070300 2024-03-30T04:38:48.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.926+0000 7f1ffe000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2000070c10 0x7f2000116270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:38:48.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.926+0000 7f1ffe000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2000070c10 0x7f2000116270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33608/0 (socket says 172.21.15.123:33608) 2024-03-30T04:38:48.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.926+0000 7f1ffe000700 1 -- 172.21.15.123:0/1651732757 learned_addr learned my addr 172.21.15.123:0/1651732757 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:38:48.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.926+0000 7f1ffea00700 1 --2- 172.21.15.123:0/1651732757 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2000070300 0x7f2000115d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:38:48.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.926+0000 7f1ffe000700 1 -- 172.21.15.123:0/1651732757 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2000070300 msgr2=0x7f2000115d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:48.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.926+0000 7f1ffe000700 1 --2- 172.21.15.123:0/1651732757 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2000070300 0x7f2000115d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:48.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.926+0000 7f1ffe000700 1 -- 172.21.15.123:0/1651732757 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fec005040 con 0x7f2000070c10 2024-03-30T04:38:48.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.926+0000 7f1ffe000700 1 --2- 172.21.15.123:0/1651732757 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2000070c10 0x7f2000116270 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1fec005010 tx=0x7f1fec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:38:48.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.926+0000 7f1ff6a00700 1 -- 172.21.15.123:0/1651732757 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1fec00e040 con 0x7f2000070c10 2024-03-30T04:38:48.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.927+0000 7f1ff6a00700 1 -- 172.21.15.123:0/1651732757 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1fec004290 con 0x7f2000070c10 2024-03-30T04:38:48.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.927+0000 7f1ff6a00700 1 -- 172.21.15.123:0/1651732757 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1fec013d00 con 0x7f2000070c10 2024-03-30T04:38:48.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.927+0000 7f2005c00700 1 -- 172.21.15.123:0/1651732757 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f200010ff10 con 0x7f2000070c10 2024-03-30T04:38:48.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.927+0000 7f2005c00700 1 -- 172.21.15.123:0/1651732757 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2000110450 con 0x7f2000070c10 2024-03-30T04:38:48.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.928+0000 7f2005c00700 1 -- 172.21.15.123:0/1651732757 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2000071a40 con 0x7f2000070c10 2024-03-30T04:38:48.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.929+0000 7f1ff6a00700 1 -- 172.21.15.123:0/1651732757 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f1fec0134b0 con 0x7f2000070c10 2024-03-30T04:38:48.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.930+0000 7f1ff6a00700 1 --2- 172.21.15.123:0/1651732757 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1fe406cc30 0x7f1fe406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:38:48.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.930+0000 7f1ff6a00700 1 -- 172.21.15.123:0/1651732757 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f1fec08d1d0 con 0x7f2000070c10 2024-03-30T04:38:48.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.930+0000 7f1ffea00700 1 --2- 172.21.15.123:0/1651732757 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1fe406cc30 0x7f1fe406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:38:48.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.930+0000 7f1ffea00700 1 --2- 172.21.15.123:0/1651732757 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1fe406cc30 0x7f1fe406f0e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1ff0009b80 tx=0x7f1ff000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:38:48.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:48.934+0000 7f1ff6a00700 1 -- 172.21.15.123:0/1651732757 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f1fec05ac20 con 0x7f2000070c10 2024-03-30T04:38:48.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:48 smithi123 ceph-mon[139369]: pgmap v228: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:49.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.220+0000 7f2005c00700 1 -- 172.21.15.123:0/1651732757 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f200010c000 con 0x7f2000070c10 2024-03-30T04:38:49.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.221+0000 7f1ff6a00700 1 -- 172.21.15.123:0/1651732757 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f1fec05a7b0 con 0x7f2000070c10 2024-03-30T04:38:49.219 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:38:49.219 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:38:49.220 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:38:49.220 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:38:49.220 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:38:49.220 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:38:49.220 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:38:49.220 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:38:49.220 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T04:38:49.220 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:38:49.220 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:38:49.220 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:38:49.220 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 3 2024-03-30T04:38:49.220 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:38:49.220 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:38:49.220 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:38:49.221 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 13 2024-03-30T04:38:49.221 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:38:49.221 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:38:49.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.223+0000 7f2005c00700 1 -- 172.21.15.123:0/1651732757 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1fe406cc30 msgr2=0x7f1fe406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:49.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.223+0000 7f2005c00700 1 --2- 172.21.15.123:0/1651732757 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1fe406cc30 0x7f1fe406f0e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1ff0009b80 tx=0x7f1ff000b040 comp rx=0 tx=0).stop 2024-03-30T04:38:49.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.223+0000 7f2005c00700 1 -- 172.21.15.123:0/1651732757 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2000070c10 msgr2=0x7f2000116270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:49.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.223+0000 7f2005c00700 1 --2- 172.21.15.123:0/1651732757 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2000070c10 0x7f2000116270 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1fec005010 tx=0x7f1fec009e30 comp rx=0 tx=0).stop 2024-03-30T04:38:49.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.223+0000 7f2005c00700 1 -- 172.21.15.123:0/1651732757 shutdown_connections 2024-03-30T04:38:49.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.223+0000 7f2005c00700 1 --2- 172.21.15.123:0/1651732757 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1fe406cc30 0x7f1fe406f0e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:49.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.223+0000 7f2005c00700 1 --2- 172.21.15.123:0/1651732757 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2000070300 0x7f2000115d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:49.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.223+0000 7f2005c00700 1 --2- 172.21.15.123:0/1651732757 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2000070c10 0x7f2000116270 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:49.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.223+0000 7f2005c00700 1 -- 172.21.15.123:0/1651732757 >> 172.21.15.123:0/1651732757 conn(0x7f20000f8970 msgr2=0x7f20001053c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:38:49.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.223+0000 7f2005c00700 1 -- 172.21.15.123:0/1651732757 shutdown_connections 2024-03-30T04:38:49.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.223+0000 7f2005c00700 1 -- 172.21.15.123:0/1651732757 wait complete. 2024-03-30T04:38:49.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.356+0000 7f0f6ea00700 1 -- 172.21.15.123:0/1182144 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f68102e00 msgr2=0x7f0f681031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:49.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.356+0000 7f0f6ea00700 1 --2- 172.21.15.123:0/1182144 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f68102e00 0x7f0f681031d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0f58004940 tx=0x7f0f58009e30 comp rx=0 tx=0).stop 2024-03-30T04:38:49.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.356+0000 7f0f6ea00700 1 -- 172.21.15.123:0/1182144 shutdown_connections 2024-03-30T04:38:49.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.356+0000 7f0f6ea00700 1 --2- 172.21.15.123:0/1182144 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0f681037a0 0x7f0f680fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:49.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.356+0000 7f0f6ea00700 1 --2- 172.21.15.123:0/1182144 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f68102e00 0x7f0f681031d0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:49.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.356+0000 7f0f6ea00700 1 -- 172.21.15.123:0/1182144 >> 172.21.15.123:0/1182144 conn(0x7f0f680f89b0 msgr2=0x7f0f680fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:38:49.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.356+0000 7f0f6ea00700 1 -- 172.21.15.123:0/1182144 shutdown_connections 2024-03-30T04:38:49.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.356+0000 7f0f6ea00700 1 -- 172.21.15.123:0/1182144 wait complete. 2024-03-30T04:38:49.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.357+0000 7f0f6ea00700 1 Processor -- start 2024-03-30T04:38:49.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.357+0000 7f0f6ea00700 1 -- start start 2024-03-30T04:38:49.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.357+0000 7f0f6ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f68102e00 0x7f0f68115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:38:49.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.358+0000 7f0f6ea00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0f681037a0 0x7f0f68116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:38:49.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.358+0000 7f0f6ea00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f68116860 con 0x7f0f68102e00 2024-03-30T04:38:49.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.358+0000 7f0f6ea00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f6810fc30 con 0x7f0f681037a0 2024-03-30T04:38:49.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.358+0000 7f0f67e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f68102e00 0x7f0f68115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:38:49.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.358+0000 7f0f67e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f68102e00 0x7f0f68115cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33620/0 (socket says 172.21.15.123:33620) 2024-03-30T04:38:49.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.358+0000 7f0f67e00700 1 -- 172.21.15.123:0/1995586084 learned_addr learned my addr 172.21.15.123:0/1995586084 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:38:49.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.358+0000 7f0f67400700 1 --2- 172.21.15.123:0/1995586084 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0f681037a0 0x7f0f68116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:38:49.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.358+0000 7f0f67e00700 1 -- 172.21.15.123:0/1995586084 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0f681037a0 msgr2=0x7f0f68116210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:49.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.358+0000 7f0f67e00700 1 --2- 172.21.15.123:0/1995586084 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0f681037a0 0x7f0f68116210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:49.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.358+0000 7f0f67e00700 1 -- 172.21.15.123:0/1995586084 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f58005040 con 0x7f0f68102e00 2024-03-30T04:38:49.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.358+0000 7f0f67e00700 1 --2- 172.21.15.123:0/1995586084 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f68102e00 0x7f0f68115cd0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0f58009360 tx=0x7f0f58009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:38:49.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.359+0000 7f0f5fe00700 1 -- 172.21.15.123:0/1995586084 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f5800e040 con 0x7f0f68102e00 2024-03-30T04:38:49.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.359+0000 7f0f5fe00700 1 -- 172.21.15.123:0/1995586084 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0f58013420 con 0x7f0f68102e00 2024-03-30T04:38:49.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.359+0000 7f0f6ea00700 1 -- 172.21.15.123:0/1995586084 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f6810feb0 con 0x7f0f68102e00 2024-03-30T04:38:49.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.359+0000 7f0f5fe00700 1 -- 172.21.15.123:0/1995586084 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f5801c5d0 con 0x7f0f68102e00 2024-03-30T04:38:49.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.359+0000 7f0f6ea00700 1 -- 172.21.15.123:0/1995586084 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f68110470 con 0x7f0f68102e00 2024-03-30T04:38:49.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.360+0000 7f0f6ea00700 1 -- 172.21.15.123:0/1995586084 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f680fe550 con 0x7f0f68102e00 2024-03-30T04:38:49.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.361+0000 7f0f5fe00700 1 -- 172.21.15.123:0/1995586084 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f0f58013590 con 0x7f0f68102e00 2024-03-30T04:38:49.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.362+0000 7f0f5fe00700 1 --2- 172.21.15.123:0/1995586084 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0f4c06cc30 0x7f0f4c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:38:49.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.362+0000 7f0f5fe00700 1 -- 172.21.15.123:0/1995586084 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f0f5808d1f0 con 0x7f0f68102e00 2024-03-30T04:38:49.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.362+0000 7f0f67400700 1 --2- 172.21.15.123:0/1995586084 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0f4c06cc30 0x7f0f4c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:38:49.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.363+0000 7f0f67400700 1 --2- 172.21.15.123:0/1995586084 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0f4c06cc30 0x7f0f4c06f0e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0f68111570 tx=0x7f0f54009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:38:49.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.366+0000 7f0f5fe00700 1 -- 172.21.15.123:0/1995586084 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0f5805ac40 con 0x7f0f68102e00 2024-03-30T04:38:49.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.614+0000 7f0f6ea00700 1 -- 172.21.15.123:0/1995586084 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0f6804bed0 con 0x7f0f68102e00 2024-03-30T04:38:49.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.614+0000 7f0f5fe00700 1 -- 172.21.15.123:0/1995586084 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f0f5805a7d0 con 0x7f0f68102e00 2024-03-30T04:38:49.613 INFO:teuthology.orchestra.run.smithi123.stdout:e32 2024-03-30T04:38:49.613 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:38:49.613 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:38:49.613 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:38:49.613 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:38:49.613 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:38:49.613 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:38:49.613 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T04:38:49.614 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T04:38:49.614 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:38:49.614 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T04:38:49.614 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:38:49.614 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:38:49.614 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:38:49.614 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:38:49.614 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:38:49.614 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T04:38:49.614 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:38:49.614 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:38:49.614 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T04:38:49.614 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:38:49.615 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T04:38:49.615 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T04:38:49.615 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T04:38:49.615 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T04:38:49.615 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:38:49.615 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:38:49.615 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:38:49.615 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:38:49.615 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:38:49.615 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:38:49.615 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T04:38:49.615 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T04:38:49.615 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:38:49.615 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:38:49.615 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T04:38:49.616 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:38:49.616 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:38:49.616 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:38:49.616 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44315} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/1741352336,v1:172.21.15.135:6827/1741352336] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:38:49.616 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:38:49.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.617+0000 7f0f6ea00700 1 -- 172.21.15.123:0/1995586084 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0f4c06cc30 msgr2=0x7f0f4c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:49.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.617+0000 7f0f6ea00700 1 --2- 172.21.15.123:0/1995586084 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0f4c06cc30 0x7f0f4c06f0e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0f68111570 tx=0x7f0f54009040 comp rx=0 tx=0).stop 2024-03-30T04:38:49.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.617+0000 7f0f6ea00700 1 -- 172.21.15.123:0/1995586084 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f68102e00 msgr2=0x7f0f68115cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:49.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.617+0000 7f0f6ea00700 1 --2- 172.21.15.123:0/1995586084 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f68102e00 0x7f0f68115cd0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0f58009360 tx=0x7f0f58009e30 comp rx=0 tx=0).stop 2024-03-30T04:38:49.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.617+0000 7f0f6ea00700 1 -- 172.21.15.123:0/1995586084 shutdown_connections 2024-03-30T04:38:49.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.617+0000 7f0f6ea00700 1 --2- 172.21.15.123:0/1995586084 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0f4c06cc30 0x7f0f4c06f0e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:49.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.617+0000 7f0f6ea00700 1 --2- 172.21.15.123:0/1995586084 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f68102e00 0x7f0f68115cd0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:49.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.617+0000 7f0f6ea00700 1 --2- 172.21.15.123:0/1995586084 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0f681037a0 0x7f0f68116210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:49.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.617+0000 7f0f6ea00700 1 -- 172.21.15.123:0/1995586084 >> 172.21.15.123:0/1995586084 conn(0x7f0f680f89b0 msgr2=0x7f0f680f99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:38:49.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.617+0000 7f0f6ea00700 1 -- 172.21.15.123:0/1995586084 shutdown_connections 2024-03-30T04:38:49.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.617+0000 7f0f6ea00700 1 -- 172.21.15.123:0/1995586084 wait complete. 2024-03-30T04:38:49.617 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 32 2024-03-30T04:38:49.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.750+0000 7f246ce00700 1 -- 172.21.15.123:0/1644005024 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f24681035e0 msgr2=0x7f24680fcc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:49.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.750+0000 7f246ce00700 1 --2- 172.21.15.123:0/1644005024 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f24681035e0 0x7f24680fcc50 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2454004940 tx=0x7f2454009e30 comp rx=0 tx=0).stop 2024-03-30T04:38:49.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.750+0000 7f246ce00700 1 -- 172.21.15.123:0/1644005024 shutdown_connections 2024-03-30T04:38:49.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.750+0000 7f246ce00700 1 --2- 172.21.15.123:0/1644005024 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f24681035e0 0x7f24680fcc50 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:49.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.750+0000 7f246ce00700 1 --2- 172.21.15.123:0/1644005024 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2468102c40 0x7f2468103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:49.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.750+0000 7f246ce00700 1 -- 172.21.15.123:0/1644005024 >> 172.21.15.123:0/1644005024 conn(0x7f24680f8830 msgr2=0x7f24680fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:38:49.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.750+0000 7f246ce00700 1 -- 172.21.15.123:0/1644005024 shutdown_connections 2024-03-30T04:38:49.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.751+0000 7f246ce00700 1 -- 172.21.15.123:0/1644005024 wait complete. 2024-03-30T04:38:49.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.751+0000 7f246ce00700 1 Processor -- start 2024-03-30T04:38:49.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.751+0000 7f246ce00700 1 -- start start 2024-03-30T04:38:49.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.752+0000 7f246ce00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2468102c40 0x7f2468115ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:38:49.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.752+0000 7f246ce00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f24681035e0 0x7f2468116020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:38:49.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.752+0000 7f246ce00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2468116670 con 0x7f24681035e0 2024-03-30T04:38:49.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.752+0000 7f246ce00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f246810fa60 con 0x7f2468102c40 2024-03-30T04:38:49.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.752+0000 7f2466a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f24681035e0 0x7f2468116020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:38:49.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.752+0000 7f2467400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2468102c40 0x7f2468115ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:38:49.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.752+0000 7f2466a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f24681035e0 0x7f2468116020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33626/0 (socket says 172.21.15.123:33626) 2024-03-30T04:38:49.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.752+0000 7f2467400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2468102c40 0x7f2468115ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.123:34088/0 (socket says 172.21.15.123:34088) 2024-03-30T04:38:49.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.752+0000 7f2466a00700 1 -- 172.21.15.123:0/582609787 learned_addr learned my addr 172.21.15.123:0/582609787 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:38:49.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.753+0000 7f2466a00700 1 -- 172.21.15.123:0/582609787 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2468102c40 msgr2=0x7f2468115ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:49.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.753+0000 7f2466a00700 1 --2- 172.21.15.123:0/582609787 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2468102c40 0x7f2468115ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:49.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.753+0000 7f2466a00700 1 -- 172.21.15.123:0/582609787 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2454005040 con 0x7f24681035e0 2024-03-30T04:38:49.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.753+0000 7f2466a00700 1 --2- 172.21.15.123:0/582609787 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f24681035e0 0x7f2468116020 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f2454009380 tx=0x7f2454009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:38:49.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.753+0000 7f245f400700 1 -- 172.21.15.123:0/582609787 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f245400e040 con 0x7f24681035e0 2024-03-30T04:38:49.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.753+0000 7f245f400700 1 -- 172.21.15.123:0/582609787 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2454013420 con 0x7f24681035e0 2024-03-30T04:38:49.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.753+0000 7f245f400700 1 -- 172.21.15.123:0/582609787 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f245401c630 con 0x7f24681035e0 2024-03-30T04:38:49.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.753+0000 7f246ce00700 1 -- 172.21.15.123:0/582609787 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f246810fce0 con 0x7f24681035e0 2024-03-30T04:38:49.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.753+0000 7f246ce00700 1 -- 172.21.15.123:0/582609787 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24681102a0 con 0x7f24681035e0 2024-03-30T04:38:49.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.754+0000 7f246ce00700 1 -- 172.21.15.123:0/582609787 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24680fe390 con 0x7f24681035e0 2024-03-30T04:38:49.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.755+0000 7f245f400700 1 -- 172.21.15.123:0/582609787 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f24540139b0 con 0x7f24681035e0 2024-03-30T04:38:49.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.756+0000 7f245f400700 1 --2- 172.21.15.123:0/582609787 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f244c06cc20 0x7f244c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:38:49.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.756+0000 7f245f400700 1 -- 172.21.15.123:0/582609787 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f245408d170 con 0x7f24681035e0 2024-03-30T04:38:49.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.757+0000 7f2467400700 1 --2- 172.21.15.123:0/582609787 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f244c06cc20 0x7f244c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:38:49.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.757+0000 7f2467400700 1 --2- 172.21.15.123:0/582609787 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f244c06cc20 0x7f244c06f0d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2458004f40 tx=0x7f245800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:38:49.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.761+0000 7f245f400700 1 -- 172.21.15.123:0/582609787 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f245405ac40 con 0x7f24681035e0 2024-03-30T04:38:49.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:49 smithi135 ceph-mon[106011]: from='client.34486 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:38:49.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:49 smithi135 ceph-mon[106011]: from='client.34490 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:38:49.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:49 smithi135 ceph-mon[106011]: from='client.34494 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:38:49.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:49 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/1651732757' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:38:49.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:49 smithi123 ceph-mon[139369]: from='client.34486 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:38:49.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:49 smithi123 ceph-mon[139369]: from='client.34490 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:38:49.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:49 smithi123 ceph-mon[139369]: from='client.34494 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:38:49.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:49 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/1651732757' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:38:49.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.986+0000 7f246ce00700 1 -- 172.21.15.123:0/582609787 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2468109c40 con 0x7f244c06cc20 2024-03-30T04:38:49.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.988+0000 7f245f400700 1 -- 172.21.15.123:0/582609787 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f2468109c40 con 0x7f244c06cc20 2024-03-30T04:38:49.987 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:38:49.987 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:38:49.987 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:38:49.987 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T04:38:49.987 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:38:49.987 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T04:38:49.987 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T04:38:49.987 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T04:38:49.987 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T04:38:49.987 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:38:49.987 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "15/21 daemons upgraded", 2024-03-30T04:38:49.988 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T04:38:49.988 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:38:49.988 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:38:49.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.990+0000 7f246ce00700 1 -- 172.21.15.123:0/582609787 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f244c06cc20 msgr2=0x7f244c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:49.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.990+0000 7f246ce00700 1 --2- 172.21.15.123:0/582609787 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f244c06cc20 0x7f244c06f0d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2458004f40 tx=0x7f245800b040 comp rx=0 tx=0).stop 2024-03-30T04:38:49.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.990+0000 7f246ce00700 1 -- 172.21.15.123:0/582609787 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f24681035e0 msgr2=0x7f2468116020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:49.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.990+0000 7f246ce00700 1 --2- 172.21.15.123:0/582609787 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f24681035e0 0x7f2468116020 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f2454009380 tx=0x7f2454009e30 comp rx=0 tx=0).stop 2024-03-30T04:38:49.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.990+0000 7f246ce00700 1 -- 172.21.15.123:0/582609787 shutdown_connections 2024-03-30T04:38:49.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.990+0000 7f246ce00700 1 --2- 172.21.15.123:0/582609787 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f244c06cc20 0x7f244c06f0d0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:49.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.990+0000 7f246ce00700 1 --2- 172.21.15.123:0/582609787 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2468102c40 0x7f2468115ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:49.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.990+0000 7f246ce00700 1 --2- 172.21.15.123:0/582609787 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f24681035e0 0x7f2468116020 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:49.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.990+0000 7f246ce00700 1 -- 172.21.15.123:0/582609787 >> 172.21.15.123:0/582609787 conn(0x7f24680f8830 msgr2=0x7f24680f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:38:49.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.990+0000 7f246ce00700 1 -- 172.21.15.123:0/582609787 shutdown_connections 2024-03-30T04:38:49.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:49.990+0000 7f246ce00700 1 -- 172.21.15.123:0/582609787 wait complete. 2024-03-30T04:38:50.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:50.124+0000 7fa217e00700 1 -- 172.21.15.123:0/1712068168 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2181035e0 msgr2=0x7fa2180fcc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:50.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:50.124+0000 7fa217e00700 1 --2- 172.21.15.123:0/1712068168 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2181035e0 0x7fa2180fcc50 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa204004940 tx=0x7fa204009e30 comp rx=0 tx=0).stop 2024-03-30T04:38:50.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:50.124+0000 7fa217e00700 1 -- 172.21.15.123:0/1712068168 shutdown_connections 2024-03-30T04:38:50.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:50.124+0000 7fa217e00700 1 --2- 172.21.15.123:0/1712068168 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2181035e0 0x7fa2180fcc50 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:50.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:50.124+0000 7fa217e00700 1 --2- 172.21.15.123:0/1712068168 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa218102c40 0x7fa218103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:50.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:50.124+0000 7fa217e00700 1 -- 172.21.15.123:0/1712068168 >> 172.21.15.123:0/1712068168 conn(0x7fa2180f8830 msgr2=0x7fa2180fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:38:50.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:50.124+0000 7fa217e00700 1 -- 172.21.15.123:0/1712068168 shutdown_connections 2024-03-30T04:38:50.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:50.124+0000 7fa217e00700 1 -- 172.21.15.123:0/1712068168 wait complete. 2024-03-30T04:38:50.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:50.125+0000 7fa217e00700 1 Processor -- start 2024-03-30T04:38:50.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:50.125+0000 7fa217e00700 1 -- start start 2024-03-30T04:38:50.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:50.126+0000 7fa217e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa218102c40 0x7fa218115a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:38:50.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:50.126+0000 7fa217e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2181035e0 0x7fa218115fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:38:50.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:50.126+0000 7fa217e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2181165f0 con 0x7fa2181035e0 2024-03-30T04:38:50.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:50.126+0000 7fa217e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa21810f9e0 con 0x7fa218102c40 2024-03-30T04:38:50.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:50.126+0000 7fa216000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2181035e0 0x7fa218115fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:38:50.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:50.126+0000 7fa216000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2181035e0 0x7fa218115fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33636/0 (socket says 172.21.15.123:33636) 2024-03-30T04:38:50.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:50.126+0000 7fa216000700 1 -- 172.21.15.123:0/2507978178 learned_addr learned my addr 172.21.15.123:0/2507978178 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:38:50.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:50.126+0000 7fa216a00700 1 --2- 172.21.15.123:0/2507978178 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa218102c40 0x7fa218115a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:38:50.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:50.126+0000 7fa216000700 1 -- 172.21.15.123:0/2507978178 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa218102c40 msgr2=0x7fa218115a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:50.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:50.126+0000 7fa216000700 1 --2- 172.21.15.123:0/2507978178 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa218102c40 0x7fa218115a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:50.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:50.126+0000 7fa216000700 1 -- 172.21.15.123:0/2507978178 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa204005040 con 0x7fa2181035e0 2024-03-30T04:38:50.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:50.127+0000 7fa216000700 1 --2- 172.21.15.123:0/2507978178 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2181035e0 0x7fa218115fa0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa204009390 tx=0x7fa204009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:38:50.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:50.127+0000 7fa20ea00700 1 -- 172.21.15.123:0/2507978178 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa20400e040 con 0x7fa2181035e0 2024-03-30T04:38:50.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:50.127+0000 7fa20ea00700 1 -- 172.21.15.123:0/2507978178 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa204013420 con 0x7fa2181035e0 2024-03-30T04:38:50.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:50.127+0000 7fa217e00700 1 -- 172.21.15.123:0/2507978178 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa21810fc60 con 0x7fa2181035e0 2024-03-30T04:38:50.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:50.127+0000 7fa20ea00700 1 -- 172.21.15.123:0/2507978178 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa20401c630 con 0x7fa2181035e0 2024-03-30T04:38:50.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:50.127+0000 7fa217e00700 1 -- 172.21.15.123:0/2507978178 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa218110220 con 0x7fa2181035e0 2024-03-30T04:38:50.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:50.128+0000 7fa217e00700 1 -- 172.21.15.123:0/2507978178 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2180fe390 con 0x7fa2181035e0 2024-03-30T04:38:50.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:50.130+0000 7fa20ea00700 1 -- 172.21.15.123:0/2507978178 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fa204013a00 con 0x7fa2181035e0 2024-03-30T04:38:50.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:50.131+0000 7fa20ea00700 1 --2- 172.21.15.123:0/2507978178 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa1fc06cc30 0x7fa1fc06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:38:50.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:50.131+0000 7fa20ea00700 1 -- 172.21.15.123:0/2507978178 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa20408dd10 con 0x7fa2181035e0 2024-03-30T04:38:50.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:50.131+0000 7fa216a00700 1 --2- 172.21.15.123:0/2507978178 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa1fc06cc30 0x7fa1fc06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:38:50.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:50.132+0000 7fa216a00700 1 --2- 172.21.15.123:0/2507978178 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa1fc06cc30 0x7fa1fc06f0e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa208009fd0 tx=0x7fa20800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:38:50.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:50.134+0000 7fa20ea00700 1 -- 172.21.15.123:0/2507978178 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa20405b9c0 con 0x7fa2181035e0 2024-03-30T04:38:50.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:50.418+0000 7fa217e00700 1 -- 172.21.15.123:0/2507978178 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fa21810fdf0 con 0x7fa2181035e0 2024-03-30T04:38:50.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:50.419+0000 7fa20ea00700 1 -- 172.21.15.123:0/2507978178 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fa21810fdf0 con 0x7fa2181035e0 2024-03-30T04:38:50.417 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T04:38:50.418 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T04:38:50.418 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T04:38:50.418 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T04:38:50.418 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T04:38:50.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:50.421+0000 7fa217e00700 1 -- 172.21.15.123:0/2507978178 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa1fc06cc30 msgr2=0x7fa1fc06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:50.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:50.421+0000 7fa217e00700 1 --2- 172.21.15.123:0/2507978178 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa1fc06cc30 0x7fa1fc06f0e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa208009fd0 tx=0x7fa20800b040 comp rx=0 tx=0).stop 2024-03-30T04:38:50.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:50.421+0000 7fa217e00700 1 -- 172.21.15.123:0/2507978178 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2181035e0 msgr2=0x7fa218115fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:38:50.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:50.421+0000 7fa217e00700 1 --2- 172.21.15.123:0/2507978178 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2181035e0 0x7fa218115fa0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa204009390 tx=0x7fa204009e30 comp rx=0 tx=0).stop 2024-03-30T04:38:50.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:50.421+0000 7fa217e00700 1 -- 172.21.15.123:0/2507978178 shutdown_connections 2024-03-30T04:38:50.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:50.421+0000 7fa217e00700 1 --2- 172.21.15.123:0/2507978178 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa1fc06cc30 0x7fa1fc06f0e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:50.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:50.421+0000 7fa217e00700 1 --2- 172.21.15.123:0/2507978178 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa218102c40 0x7fa218115a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:50.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:50.421+0000 7fa217e00700 1 --2- 172.21.15.123:0/2507978178 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2181035e0 0x7fa218115fa0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:38:50.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:50.421+0000 7fa217e00700 1 -- 172.21.15.123:0/2507978178 >> 172.21.15.123:0/2507978178 conn(0x7fa2180f8830 msgr2=0x7fa2180f9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:38:50.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:50.421+0000 7fa217e00700 1 -- 172.21.15.123:0/2507978178 shutdown_connections 2024-03-30T04:38:50.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:38:50.422+0000 7fa217e00700 1 -- 172.21.15.123:0/2507978178 wait complete. 2024-03-30T04:38:50.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:50 smithi135 ceph-mon[106011]: pgmap v229: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:50.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:50 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/1995586084' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:38:50.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:50 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2507978178' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:38:50.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:50 smithi123 ceph-mon[139369]: pgmap v229: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:50.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:50 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/1995586084' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:38:50.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:50 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2507978178' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:38:51.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:51 smithi135 ceph-mon[106011]: from='client.34506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:38:51.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:51 smithi123 ceph-mon[139369]: from='client.34506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:38:52.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:52 smithi135 ceph-mon[106011]: pgmap v230: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:52.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:52 smithi123 ceph-mon[139369]: pgmap v230: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:54.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:54 smithi135 ceph-mon[106011]: pgmap v231: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:54.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:54 smithi123 ceph-mon[139369]: pgmap v231: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:56.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:56 smithi135 ceph-mon[106011]: pgmap v232: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:56.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:56 smithi123 ceph-mon[139369]: pgmap v232: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:58.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:38:58 smithi135 ceph-mon[106011]: pgmap v233: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:38:58.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:38:58 smithi123 ceph-mon[139369]: pgmap v233: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:00.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:39:00 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:39:00.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:39:00 smithi123 ceph-mon[139369]: pgmap v234: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:00.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:39:00 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:39:00.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:39:00 smithi135 ceph-mon[106011]: pgmap v234: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:02.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:39:02 smithi135 ceph-mon[106011]: pgmap v235: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:02.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:39:02 smithi123 ceph-mon[139369]: pgmap v235: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:04.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:39:04 smithi135 ceph-mon[106011]: pgmap v236: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:04.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:39:04 smithi123 ceph-mon[139369]: pgmap v236: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:06.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:39:06 smithi135 ceph-mon[106011]: pgmap v237: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:06.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:39:06 smithi123 ceph-mon[139369]: pgmap v237: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:08.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:39:08 smithi135 ceph-mon[106011]: pgmap v238: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:08.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:39:08 smithi123 ceph-mon[139369]: pgmap v238: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:10.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:39:10 smithi123 ceph-mon[139369]: pgmap v239: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:10.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:39:10 smithi135 ceph-mon[106011]: pgmap v239: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:12.853 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:39:12 smithi135 ceph-mon[106011]: pgmap v240: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:12.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:39:12 smithi123 ceph-mon[139369]: pgmap v240: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:14.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:39:14 smithi135 ceph-mon[106011]: pgmap v241: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:14.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:39:14 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:39:14.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:39:14 smithi123 ceph-mon[139369]: pgmap v241: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:14.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:39:14 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:39:16.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:39:16 smithi135 ceph-mon[106011]: pgmap v242: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:16.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:39:16 smithi123 ceph-mon[139369]: pgmap v242: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:18.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:39:18 smithi135 ceph-mon[106011]: pgmap v243: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:18.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:39:18 smithi123 ceph-mon[139369]: pgmap v243: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:20.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.569+0000 7f7f3f200700 1 -- 172.21.15.123:0/3609674199 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f381037a0 msgr2=0x7f7f380fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:20.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.569+0000 7f7f3f200700 1 --2- 172.21.15.123:0/3609674199 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f381037a0 0x7f7f380fce10 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7f24004940 tx=0x7f7f24009e30 comp rx=0 tx=0).stop 2024-03-30T04:39:20.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.569+0000 7f7f3f200700 1 -- 172.21.15.123:0/3609674199 shutdown_connections 2024-03-30T04:39:20.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.569+0000 7f7f3f200700 1 --2- 172.21.15.123:0/3609674199 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f381037a0 0x7f7f380fce10 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:20.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.569+0000 7f7f3f200700 1 --2- 172.21.15.123:0/3609674199 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f38102e00 0x7f7f381031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:20.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.569+0000 7f7f3f200700 1 -- 172.21.15.123:0/3609674199 >> 172.21.15.123:0/3609674199 conn(0x7f7f380f89b0 msgr2=0x7f7f380fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:39:20.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.569+0000 7f7f3f200700 1 -- 172.21.15.123:0/3609674199 shutdown_connections 2024-03-30T04:39:20.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.569+0000 7f7f3f200700 1 -- 172.21.15.123:0/3609674199 wait complete. 2024-03-30T04:39:20.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.570+0000 7f7f3f200700 1 Processor -- start 2024-03-30T04:39:20.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.570+0000 7f7f3f200700 1 -- start start 2024-03-30T04:39:20.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.571+0000 7f7f3f200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f38102e00 0x7f7f380702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:39:20.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.571+0000 7f7f3f200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f381037a0 0x7f7f38070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:39:20.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.571+0000 7f7f3f200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f38070d90 con 0x7f7f381037a0 2024-03-30T04:39:20.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.571+0000 7f7f3f200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f38070ed0 con 0x7f7f38102e00 2024-03-30T04:39:20.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.571+0000 7f7f37400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f381037a0 0x7f7f38070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:39:20.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.571+0000 7f7f37400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f381037a0 0x7f7f38070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48676/0 (socket says 172.21.15.123:48676) 2024-03-30T04:39:20.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.571+0000 7f7f37400700 1 -- 172.21.15.123:0/4053756616 learned_addr learned my addr 172.21.15.123:0/4053756616 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:39:20.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.572+0000 7f7f37e00700 1 --2- 172.21.15.123:0/4053756616 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f38102e00 0x7f7f380702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:39:20.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.572+0000 7f7f37400700 1 -- 172.21.15.123:0/4053756616 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f38102e00 msgr2=0x7f7f380702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:20.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.572+0000 7f7f37400700 1 --2- 172.21.15.123:0/4053756616 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f38102e00 0x7f7f380702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:20.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.572+0000 7f7f37400700 1 -- 172.21.15.123:0/4053756616 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f24005040 con 0x7f7f381037a0 2024-03-30T04:39:20.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.572+0000 7f7f37400700 1 --2- 172.21.15.123:0/4053756616 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f381037a0 0x7f7f38070800 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7f24009390 tx=0x7f7f24009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:39:20.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.572+0000 7f7f2fe00700 1 -- 172.21.15.123:0/4053756616 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f2400e040 con 0x7f7f381037a0 2024-03-30T04:39:20.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.572+0000 7f7f2fe00700 1 -- 172.21.15.123:0/4053756616 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7f24013420 con 0x7f7f381037a0 2024-03-30T04:39:20.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.572+0000 7f7f3f200700 1 -- 172.21.15.123:0/4053756616 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f38071120 con 0x7f7f381037a0 2024-03-30T04:39:20.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.572+0000 7f7f2fe00700 1 -- 172.21.15.123:0/4053756616 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f2401c630 con 0x7f7f381037a0 2024-03-30T04:39:20.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.573+0000 7f7f3f200700 1 -- 172.21.15.123:0/4053756616 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f3811a230 con 0x7f7f381037a0 2024-03-30T04:39:20.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.573+0000 7f7f3f200700 1 -- 172.21.15.123:0/4053756616 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f380fe550 con 0x7f7f381037a0 2024-03-30T04:39:20.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.574+0000 7f7f2fe00700 1 -- 172.21.15.123:0/4053756616 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f7f24013a00 con 0x7f7f381037a0 2024-03-30T04:39:20.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.575+0000 7f7f2fe00700 1 --2- 172.21.15.123:0/4053756616 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7f1c06cc30 0x7f7f1c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:39:20.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.575+0000 7f7f2fe00700 1 -- 172.21.15.123:0/4053756616 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7f2408d1b0 con 0x7f7f381037a0 2024-03-30T04:39:20.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.576+0000 7f7f37e00700 1 --2- 172.21.15.123:0/4053756616 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7f1c06cc30 0x7f7f1c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:39:20.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.576+0000 7f7f37e00700 1 --2- 172.21.15.123:0/4053756616 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7f1c06cc30 0x7f7f1c06f0e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7f28004f40 tx=0x7f7f2800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:39:20.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.580+0000 7f7f2fe00700 1 -- 172.21.15.123:0/4053756616 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7f24057170 con 0x7f7f381037a0 2024-03-30T04:39:20.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:39:20 smithi123 ceph-mon[139369]: pgmap v244: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:20.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.799+0000 7f7f3f200700 1 -- 172.21.15.123:0/4053756616 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7f38109e40 con 0x7f7f1c06cc30 2024-03-30T04:39:20.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.801+0000 7f7f2fe00700 1 -- 172.21.15.123:0/4053756616 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f7f2800f040 con 0x7f7f1c06cc30 2024-03-30T04:39:20.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.803+0000 7f7f3f200700 1 -- 172.21.15.123:0/4053756616 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7f1c06cc30 msgr2=0x7f7f1c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:20.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.803+0000 7f7f3f200700 1 --2- 172.21.15.123:0/4053756616 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7f1c06cc30 0x7f7f1c06f0e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7f28004f40 tx=0x7f7f2800b040 comp rx=0 tx=0).stop 2024-03-30T04:39:20.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.803+0000 7f7f3f200700 1 -- 172.21.15.123:0/4053756616 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f381037a0 msgr2=0x7f7f38070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:20.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.803+0000 7f7f3f200700 1 --2- 172.21.15.123:0/4053756616 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f381037a0 0x7f7f38070800 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7f24009390 tx=0x7f7f24009e30 comp rx=0 tx=0).stop 2024-03-30T04:39:20.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.803+0000 7f7f3f200700 1 -- 172.21.15.123:0/4053756616 shutdown_connections 2024-03-30T04:39:20.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.803+0000 7f7f3f200700 1 --2- 172.21.15.123:0/4053756616 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7f1c06cc30 0x7f7f1c06f0e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:20.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.803+0000 7f7f3f200700 1 --2- 172.21.15.123:0/4053756616 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f38102e00 0x7f7f380702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:20.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.803+0000 7f7f3f200700 1 --2- 172.21.15.123:0/4053756616 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f381037a0 0x7f7f38070800 secure :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7f24009390 tx=0x7f7f24009e30 comp rx=0 tx=0).stop 2024-03-30T04:39:20.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.803+0000 7f7f3f200700 1 -- 172.21.15.123:0/4053756616 >> 172.21.15.123:0/4053756616 conn(0x7f7f380f89b0 msgr2=0x7f7f380f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:39:20.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.803+0000 7f7f3f200700 1 -- 172.21.15.123:0/4053756616 shutdown_connections 2024-03-30T04:39:20.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.803+0000 7f7f3f200700 1 -- 172.21.15.123:0/4053756616 wait complete. 2024-03-30T04:39:20.816 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:39:20.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:39:20 smithi135 ceph-mon[106011]: pgmap v244: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:20.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.938+0000 7f5c07000700 1 -- 172.21.15.123:0/1840075427 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c00102e00 msgr2=0x7f5c001031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:20.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.938+0000 7f5c07000700 1 --2- 172.21.15.123:0/1840075427 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c00102e00 0x7f5c001031d0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5bf0004940 tx=0x7f5bf0009e30 comp rx=0 tx=0).stop 2024-03-30T04:39:20.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.938+0000 7f5c07000700 1 -- 172.21.15.123:0/1840075427 shutdown_connections 2024-03-30T04:39:20.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.938+0000 7f5c07000700 1 --2- 172.21.15.123:0/1840075427 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5c001037a0 0x7f5c000fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:20.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.938+0000 7f5c07000700 1 --2- 172.21.15.123:0/1840075427 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c00102e00 0x7f5c001031d0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:20.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.938+0000 7f5c07000700 1 -- 172.21.15.123:0/1840075427 >> 172.21.15.123:0/1840075427 conn(0x7f5c000f89b0 msgr2=0x7f5c000fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:39:20.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.938+0000 7f5c07000700 1 -- 172.21.15.123:0/1840075427 shutdown_connections 2024-03-30T04:39:20.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.939+0000 7f5c07000700 1 -- 172.21.15.123:0/1840075427 wait complete. 2024-03-30T04:39:20.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.939+0000 7f5c07000700 1 Processor -- start 2024-03-30T04:39:20.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.939+0000 7f5c07000700 1 -- start start 2024-03-30T04:39:20.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.940+0000 7f5c07000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c00102e00 0x7f5c000702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:39:20.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.940+0000 7f5c07000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5c001037a0 0x7f5c00070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:39:20.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.940+0000 7f5c07000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c00070d90 con 0x7f5c00102e00 2024-03-30T04:39:20.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.940+0000 7f5c07000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c00070ed0 con 0x7f5c001037a0 2024-03-30T04:39:20.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.940+0000 7f5bffe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c00102e00 0x7f5c000702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:39:20.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.940+0000 7f5bffe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c00102e00 0x7f5c000702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48702/0 (socket says 172.21.15.123:48702) 2024-03-30T04:39:20.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.940+0000 7f5bffe00700 1 -- 172.21.15.123:0/1440481856 learned_addr learned my addr 172.21.15.123:0/1440481856 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:39:20.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.940+0000 7f5bff400700 1 --2- 172.21.15.123:0/1440481856 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5c001037a0 0x7f5c00070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:39:20.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.940+0000 7f5bffe00700 1 -- 172.21.15.123:0/1440481856 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5c001037a0 msgr2=0x7f5c00070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:20.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.940+0000 7f5bffe00700 1 --2- 172.21.15.123:0/1440481856 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5c001037a0 0x7f5c00070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:20.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.940+0000 7f5bffe00700 1 -- 172.21.15.123:0/1440481856 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bf0005040 con 0x7f5c00102e00 2024-03-30T04:39:20.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.940+0000 7f5bff400700 1 --2- 172.21.15.123:0/1440481856 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5c001037a0 0x7f5c00070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:39:20.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.941+0000 7f5bffe00700 1 --2- 172.21.15.123:0/1440481856 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c00102e00 0x7f5c000702c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5bf0009360 tx=0x7f5bf0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:39:20.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.941+0000 7f5bf7e00700 1 -- 172.21.15.123:0/1440481856 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5bf000e040 con 0x7f5c00102e00 2024-03-30T04:39:20.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.941+0000 7f5bf7e00700 1 -- 172.21.15.123:0/1440481856 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5bf0013420 con 0x7f5c00102e00 2024-03-30T04:39:20.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.941+0000 7f5bf7e00700 1 -- 172.21.15.123:0/1440481856 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5bf001c5f0 con 0x7f5c00102e00 2024-03-30T04:39:20.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.941+0000 7f5c07000700 1 -- 172.21.15.123:0/1440481856 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c00071120 con 0x7f5c00102e00 2024-03-30T04:39:20.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.941+0000 7f5c07000700 1 -- 172.21.15.123:0/1440481856 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c0011a250 con 0x7f5c00102e00 2024-03-30T04:39:20.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.942+0000 7f5c07000700 1 -- 172.21.15.123:0/1440481856 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c000fe550 con 0x7f5c00102e00 2024-03-30T04:39:20.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.943+0000 7f5bf7e00700 1 -- 172.21.15.123:0/1440481856 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f5bf0013590 con 0x7f5c00102e00 2024-03-30T04:39:20.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.944+0000 7f5bf7e00700 1 --2- 172.21.15.123:0/1440481856 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5be406cc30 0x7f5be406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:39:20.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.944+0000 7f5bf7e00700 1 -- 172.21.15.123:0/1440481856 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5bf008d270 con 0x7f5c00102e00 2024-03-30T04:39:20.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.944+0000 7f5bff400700 1 --2- 172.21.15.123:0/1440481856 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5be406cc30 0x7f5be406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:39:20.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.945+0000 7f5bff400700 1 --2- 172.21.15.123:0/1440481856 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5be406cc30 0x7f5be406f0e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5bec007fd0 tx=0x7f5bec009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:39:20.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:20.949+0000 7f5bf7e00700 1 -- 172.21.15.123:0/1440481856 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5bf00572b0 con 0x7f5c00102e00 2024-03-30T04:39:21.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.178+0000 7f5c07000700 1 -- 172.21.15.123:0/1440481856 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5c000719b0 con 0x7f5be406cc30 2024-03-30T04:39:21.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.179+0000 7f5bf7e00700 1 -- 172.21.15.123:0/1440481856 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f5c000719b0 con 0x7f5be406cc30 2024-03-30T04:39:21.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.181+0000 7f5c07000700 1 -- 172.21.15.123:0/1440481856 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5be406cc30 msgr2=0x7f5be406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:21.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.181+0000 7f5c07000700 1 --2- 172.21.15.123:0/1440481856 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5be406cc30 0x7f5be406f0e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5bec007fd0 tx=0x7f5bec009040 comp rx=0 tx=0).stop 2024-03-30T04:39:21.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.181+0000 7f5c07000700 1 -- 172.21.15.123:0/1440481856 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c00102e00 msgr2=0x7f5c000702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:21.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.181+0000 7f5c07000700 1 --2- 172.21.15.123:0/1440481856 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c00102e00 0x7f5c000702c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5bf0009360 tx=0x7f5bf0009e30 comp rx=0 tx=0).stop 2024-03-30T04:39:21.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.181+0000 7f5c07000700 1 -- 172.21.15.123:0/1440481856 shutdown_connections 2024-03-30T04:39:21.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.181+0000 7f5c07000700 1 --2- 172.21.15.123:0/1440481856 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5be406cc30 0x7f5be406f0e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:21.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.181+0000 7f5c07000700 1 --2- 172.21.15.123:0/1440481856 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c00102e00 0x7f5c000702c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:21.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.181+0000 7f5c07000700 1 --2- 172.21.15.123:0/1440481856 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5c001037a0 0x7f5c00070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:21.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.181+0000 7f5c07000700 1 -- 172.21.15.123:0/1440481856 >> 172.21.15.123:0/1440481856 conn(0x7f5c000f89b0 msgr2=0x7f5c000f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:39:21.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.181+0000 7f5c07000700 1 -- 172.21.15.123:0/1440481856 shutdown_connections 2024-03-30T04:39:21.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.182+0000 7f5c07000700 1 -- 172.21.15.123:0/1440481856 wait complete. 2024-03-30T04:39:21.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.315+0000 7fa7aba00700 1 -- 172.21.15.123:0/83910195 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7a41037a0 msgr2=0x7fa7a40fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:21.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.315+0000 7fa7aba00700 1 --2- 172.21.15.123:0/83910195 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7a41037a0 0x7fa7a40fce10 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa790004940 tx=0x7fa790009e30 comp rx=0 tx=0).stop 2024-03-30T04:39:21.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.316+0000 7fa7aba00700 1 -- 172.21.15.123:0/83910195 shutdown_connections 2024-03-30T04:39:21.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.316+0000 7fa7aba00700 1 --2- 172.21.15.123:0/83910195 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7a41037a0 0x7fa7a40fce10 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:21.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.316+0000 7fa7aba00700 1 --2- 172.21.15.123:0/83910195 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa7a4102e00 0x7fa7a41031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:21.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.316+0000 7fa7aba00700 1 -- 172.21.15.123:0/83910195 >> 172.21.15.123:0/83910195 conn(0x7fa7a40f89b0 msgr2=0x7fa7a40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:39:21.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.316+0000 7fa7aba00700 1 -- 172.21.15.123:0/83910195 shutdown_connections 2024-03-30T04:39:21.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.316+0000 7fa7aba00700 1 -- 172.21.15.123:0/83910195 wait complete. 2024-03-30T04:39:21.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.316+0000 7fa7aba00700 1 Processor -- start 2024-03-30T04:39:21.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.317+0000 7fa7aba00700 1 -- start start 2024-03-30T04:39:21.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.317+0000 7fa7aba00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa7a4102e00 0x7fa7a40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:39:21.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.317+0000 7fa7aba00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7a41037a0 0x7fa7a4070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:39:21.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.317+0000 7fa7aba00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a4070d90 con 0x7fa7a41037a0 2024-03-30T04:39:21.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.317+0000 7fa7aba00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a4070ed0 con 0x7fa7a4102e00 2024-03-30T04:39:21.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.317+0000 7fa7a3e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7a41037a0 0x7fa7a4070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:39:21.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.317+0000 7fa7a3e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7a41037a0 0x7fa7a4070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48720/0 (socket says 172.21.15.123:48720) 2024-03-30T04:39:21.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.317+0000 7fa7a3e00700 1 -- 172.21.15.123:0/226675239 learned_addr learned my addr 172.21.15.123:0/226675239 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:39:21.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.318+0000 7fa7a9000700 1 --2- 172.21.15.123:0/226675239 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa7a4102e00 0x7fa7a40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:39:21.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.318+0000 7fa7a3e00700 1 -- 172.21.15.123:0/226675239 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa7a4102e00 msgr2=0x7fa7a40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:21.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.318+0000 7fa7a3e00700 1 --2- 172.21.15.123:0/226675239 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa7a4102e00 0x7fa7a40702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:21.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.318+0000 7fa7a3e00700 1 -- 172.21.15.123:0/226675239 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa790005040 con 0x7fa7a41037a0 2024-03-30T04:39:21.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.318+0000 7fa7a3e00700 1 --2- 172.21.15.123:0/226675239 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7a41037a0 0x7fa7a4070800 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa790009390 tx=0x7fa790009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:39:21.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.318+0000 7fa7a0c00700 1 -- 172.21.15.123:0/226675239 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa79000e040 con 0x7fa7a41037a0 2024-03-30T04:39:21.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.318+0000 7fa7a0c00700 1 -- 172.21.15.123:0/226675239 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa790013420 con 0x7fa7a41037a0 2024-03-30T04:39:21.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.318+0000 7fa7a0c00700 1 -- 172.21.15.123:0/226675239 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa79001c630 con 0x7fa7a41037a0 2024-03-30T04:39:21.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.318+0000 7fa7aba00700 1 -- 172.21.15.123:0/226675239 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7a4071120 con 0x7fa7a41037a0 2024-03-30T04:39:21.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.318+0000 7fa7aba00700 1 -- 172.21.15.123:0/226675239 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7a411a230 con 0x7fa7a41037a0 2024-03-30T04:39:21.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.319+0000 7fa7aba00700 1 -- 172.21.15.123:0/226675239 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7a40fe550 con 0x7fa7a41037a0 2024-03-30T04:39:21.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.320+0000 7fa7a0c00700 1 -- 172.21.15.123:0/226675239 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fa7900139e0 con 0x7fa7a41037a0 2024-03-30T04:39:21.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.321+0000 7fa7a0c00700 1 --2- 172.21.15.123:0/226675239 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa78c06cc30 0x7fa78c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:39:21.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.321+0000 7fa7a0c00700 1 -- 172.21.15.123:0/226675239 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa79008d2b0 con 0x7fa7a41037a0 2024-03-30T04:39:21.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.321+0000 7fa7a9000700 1 --2- 172.21.15.123:0/226675239 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa78c06cc30 0x7fa78c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:39:21.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.322+0000 7fa7a9000700 1 --2- 172.21.15.123:0/226675239 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa78c06cc30 0x7fa78c06f0e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa798004990 tx=0x7fa798009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:39:21.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.326+0000 7fa7a0c00700 1 -- 172.21.15.123:0/226675239 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa7900572f0 con 0x7fa7a41037a0 2024-03-30T04:39:21.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.542+0000 7fa7aba00700 1 -- 172.21.15.123:0/226675239 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fa7a4109e40 con 0x7fa78c06cc30 2024-03-30T04:39:21.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.551+0000 7fa7a0c00700 1 -- 172.21.15.123:0/226675239 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7fa798004e60 con 0x7fa78c06cc30 2024-03-30T04:39:21.550 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:39:21.550 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (8m) 86s ago 21m 15.0M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:39:21.550 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (5m) 86s ago 21m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T04:39:21.550 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (5m) 68s ago 20m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T04:39:21.550 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (8m) 86s ago 20m 74.0M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T04:39:21.550 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (92s) 86s ago 14m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T04:39:21.550 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (112s) 86s ago 14m 10.9M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T04:39:21.550 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (14m) 68s ago 14m 10.7M - 17.2.7-797-g5c024335 f77085f6aab7 0dc41326818c 2024-03-30T04:39:21.551 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (72s) 68s ago 14m 9747k - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T04:39:21.551 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (10m) 86s ago 22m 506M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:39:21.551 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (9m) 68s ago 19m 450M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:39:21.551 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (7m) 86s ago 22m 53.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T04:39:21.551 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (6m) 68s ago 19m 42.6M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T04:39:21.551 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (9m) 86s ago 21m 8052k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:39:21.551 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (9m) 68s ago 20m 7880k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:39:21.551 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (5m) 86s ago 19m 104M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T04:39:21.551 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (4m) 86s ago 18m 93.4M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T04:39:21.551 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (4m) 86s ago 17m 86.3M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T04:39:21.551 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (3m) 68s ago 17m 97.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T04:39:21.551 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (2m) 68s ago 16m 97.9M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T04:39:21.551 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (2m) 68s ago 16m 85.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T04:39:21.551 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (8m) 86s ago 20m 53.7M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:39:21.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.554+0000 7fa7aba00700 1 -- 172.21.15.123:0/226675239 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa78c06cc30 msgr2=0x7fa78c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:21.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.554+0000 7fa7aba00700 1 --2- 172.21.15.123:0/226675239 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa78c06cc30 0x7fa78c06f0e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa798004990 tx=0x7fa798009ee0 comp rx=0 tx=0).stop 2024-03-30T04:39:21.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.554+0000 7fa7aba00700 1 -- 172.21.15.123:0/226675239 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7a41037a0 msgr2=0x7fa7a4070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:21.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.554+0000 7fa7aba00700 1 --2- 172.21.15.123:0/226675239 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7a41037a0 0x7fa7a4070800 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa790009390 tx=0x7fa790009e30 comp rx=0 tx=0).stop 2024-03-30T04:39:21.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.554+0000 7fa7aba00700 1 -- 172.21.15.123:0/226675239 shutdown_connections 2024-03-30T04:39:21.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.554+0000 7fa7aba00700 1 --2- 172.21.15.123:0/226675239 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa78c06cc30 0x7fa78c06f0e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:21.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.554+0000 7fa7aba00700 1 --2- 172.21.15.123:0/226675239 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa7a4102e00 0x7fa7a40702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:21.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.554+0000 7fa7aba00700 1 --2- 172.21.15.123:0/226675239 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7a41037a0 0x7fa7a4070800 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:21.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.554+0000 7fa7aba00700 1 -- 172.21.15.123:0/226675239 >> 172.21.15.123:0/226675239 conn(0x7fa7a40f89b0 msgr2=0x7fa7a40f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:39:21.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.554+0000 7fa7aba00700 1 -- 172.21.15.123:0/226675239 shutdown_connections 2024-03-30T04:39:21.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.555+0000 7fa7aba00700 1 -- 172.21.15.123:0/226675239 wait complete. 2024-03-30T04:39:21.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.687+0000 7faccac00700 1 -- 172.21.15.123:0/2032076405 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7facc41037a0 msgr2=0x7facc40fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:21.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.687+0000 7faccac00700 1 --2- 172.21.15.123:0/2032076405 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7facc41037a0 0x7facc40fce10 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7facb0004940 tx=0x7facb0009e30 comp rx=0 tx=0).stop 2024-03-30T04:39:21.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.687+0000 7faccac00700 1 -- 172.21.15.123:0/2032076405 shutdown_connections 2024-03-30T04:39:21.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.687+0000 7faccac00700 1 --2- 172.21.15.123:0/2032076405 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7facc41037a0 0x7facc40fce10 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:21.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.687+0000 7faccac00700 1 --2- 172.21.15.123:0/2032076405 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7facc4102e00 0x7facc41031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:21.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.687+0000 7faccac00700 1 -- 172.21.15.123:0/2032076405 >> 172.21.15.123:0/2032076405 conn(0x7facc40f89b0 msgr2=0x7facc40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:39:21.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.687+0000 7faccac00700 1 -- 172.21.15.123:0/2032076405 shutdown_connections 2024-03-30T04:39:21.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.688+0000 7faccac00700 1 -- 172.21.15.123:0/2032076405 wait complete. 2024-03-30T04:39:21.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.688+0000 7faccac00700 1 Processor -- start 2024-03-30T04:39:21.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.688+0000 7faccac00700 1 -- start start 2024-03-30T04:39:21.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.689+0000 7faccac00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7facc4102e00 0x7facc4113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:39:21.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.689+0000 7faccac00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7facc41037a0 0x7facc4114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:39:21.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.689+0000 7faccac00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc4114690 con 0x7facc4102e00 2024-03-30T04:39:21.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.689+0000 7faccac00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc410da80 con 0x7facc41037a0 2024-03-30T04:39:21.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.689+0000 7facc3e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7facc4102e00 0x7facc4113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:39:21.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.689+0000 7facc3e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7facc4102e00 0x7facc4113b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48728/0 (socket says 172.21.15.123:48728) 2024-03-30T04:39:21.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.689+0000 7facc3e00700 1 -- 172.21.15.123:0/839386928 learned_addr learned my addr 172.21.15.123:0/839386928 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:39:21.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.689+0000 7facc3e00700 1 -- 172.21.15.123:0/839386928 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7facc41037a0 msgr2=0x7facc4114040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:21.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.689+0000 7facc3e00700 1 --2- 172.21.15.123:0/839386928 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7facc41037a0 0x7facc4114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:21.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.689+0000 7facc3e00700 1 -- 172.21.15.123:0/839386928 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facb0005040 con 0x7facc4102e00 2024-03-30T04:39:21.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.690+0000 7facc3e00700 1 --2- 172.21.15.123:0/839386928 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7facc4102e00 0x7facc4113b00 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7facb4002a50 tx=0x7facb40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:39:21.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.690+0000 7facbbe00700 1 -- 172.21.15.123:0/839386928 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7facb4007cc0 con 0x7facc4102e00 2024-03-30T04:39:21.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.690+0000 7faccac00700 1 -- 172.21.15.123:0/839386928 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facc410dd60 con 0x7facc4102e00 2024-03-30T04:39:21.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.690+0000 7facbbe00700 1 -- 172.21.15.123:0/839386928 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7facb400f040 con 0x7facc4102e00 2024-03-30T04:39:21.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.690+0000 7facbbe00700 1 -- 172.21.15.123:0/839386928 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7facb4009500 con 0x7facc4102e00 2024-03-30T04:39:21.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.690+0000 7faccac00700 1 -- 172.21.15.123:0/839386928 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facc410e320 con 0x7facc4102e00 2024-03-30T04:39:21.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.691+0000 7faccac00700 1 -- 172.21.15.123:0/839386928 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facc40fe550 con 0x7facc4102e00 2024-03-30T04:39:21.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.692+0000 7facbbe00700 1 -- 172.21.15.123:0/839386928 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7facb40096a0 con 0x7facc4102e00 2024-03-30T04:39:21.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.693+0000 7facbbe00700 1 --2- 172.21.15.123:0/839386928 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7faca806cc30 0x7faca806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:39:21.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.693+0000 7facbbe00700 1 -- 172.21.15.123:0/839386928 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7facb408bf00 con 0x7facc4102e00 2024-03-30T04:39:21.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.693+0000 7facc3400700 1 --2- 172.21.15.123:0/839386928 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7faca806cc30 0x7faca806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:39:21.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.694+0000 7facc3400700 1 --2- 172.21.15.123:0/839386928 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7faca806cc30 0x7faca806f0e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7facb0004f90 tx=0x7facb0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:39:21.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.697+0000 7facbbe00700 1 -- 172.21.15.123:0/839386928 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7facb40598d0 con 0x7facc4102e00 2024-03-30T04:39:21.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:39:21 smithi123 ceph-mon[139369]: from='client.34514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:39:21.854 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:39:21 smithi135 ceph-mon[106011]: from='client.34514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:39:21.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.982+0000 7faccac00700 1 -- 172.21.15.123:0/839386928 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7facc404bed0 con 0x7facc4102e00 2024-03-30T04:39:21.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.982+0000 7facbbe00700 1 -- 172.21.15.123:0/839386928 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7facb4059460 con 0x7facc4102e00 2024-03-30T04:39:21.981 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:39:21.981 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:39:21.981 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:39:21.981 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:39:21.981 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:39:21.981 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:39:21.981 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:39:21.981 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:39:21.981 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T04:39:21.982 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:39:21.982 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:39:21.982 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:39:21.982 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 3 2024-03-30T04:39:21.982 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:39:21.982 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:39:21.982 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:39:21.982 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 13 2024-03-30T04:39:21.982 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:39:21.982 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:39:21.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.985+0000 7faccac00700 1 -- 172.21.15.123:0/839386928 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7faca806cc30 msgr2=0x7faca806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:21.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.985+0000 7faccac00700 1 --2- 172.21.15.123:0/839386928 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7faca806cc30 0x7faca806f0e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7facb0004f90 tx=0x7facb0009e30 comp rx=0 tx=0).stop 2024-03-30T04:39:21.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.985+0000 7faccac00700 1 -- 172.21.15.123:0/839386928 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7facc4102e00 msgr2=0x7facc4113b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:21.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.985+0000 7faccac00700 1 --2- 172.21.15.123:0/839386928 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7facc4102e00 0x7facc4113b00 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7facb4002a50 tx=0x7facb40046b0 comp rx=0 tx=0).stop 2024-03-30T04:39:21.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.985+0000 7faccac00700 1 -- 172.21.15.123:0/839386928 shutdown_connections 2024-03-30T04:39:21.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.985+0000 7faccac00700 1 --2- 172.21.15.123:0/839386928 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7faca806cc30 0x7faca806f0e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:21.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.985+0000 7faccac00700 1 --2- 172.21.15.123:0/839386928 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7facc4102e00 0x7facc4113b00 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:21.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.985+0000 7faccac00700 1 --2- 172.21.15.123:0/839386928 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7facc41037a0 0x7facc4114040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:21.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.985+0000 7faccac00700 1 -- 172.21.15.123:0/839386928 >> 172.21.15.123:0/839386928 conn(0x7facc40f89b0 msgr2=0x7facc40f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:39:21.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.985+0000 7faccac00700 1 -- 172.21.15.123:0/839386928 shutdown_connections 2024-03-30T04:39:21.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:21.985+0000 7faccac00700 1 -- 172.21.15.123:0/839386928 wait complete. 2024-03-30T04:39:22.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.118+0000 7fe43be00700 1 -- 172.21.15.123:0/3139136796 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4341037a0 msgr2=0x7fe4340fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:22.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.118+0000 7fe43be00700 1 --2- 172.21.15.123:0/3139136796 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4341037a0 0x7fe4340fce10 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe420004940 tx=0x7fe420009e30 comp rx=0 tx=0).stop 2024-03-30T04:39:22.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.118+0000 7fe43be00700 1 -- 172.21.15.123:0/3139136796 shutdown_connections 2024-03-30T04:39:22.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.118+0000 7fe43be00700 1 --2- 172.21.15.123:0/3139136796 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4341037a0 0x7fe4340fce10 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:22.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.118+0000 7fe43be00700 1 --2- 172.21.15.123:0/3139136796 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe434102e00 0x7fe4341031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:22.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.118+0000 7fe43be00700 1 -- 172.21.15.123:0/3139136796 >> 172.21.15.123:0/3139136796 conn(0x7fe4340f89b0 msgr2=0x7fe4340fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:39:22.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.118+0000 7fe43be00700 1 -- 172.21.15.123:0/3139136796 shutdown_connections 2024-03-30T04:39:22.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.118+0000 7fe43be00700 1 -- 172.21.15.123:0/3139136796 wait complete. 2024-03-30T04:39:22.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.119+0000 7fe43be00700 1 Processor -- start 2024-03-30T04:39:22.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.119+0000 7fe43be00700 1 -- start start 2024-03-30T04:39:22.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.119+0000 7fe43be00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe434102e00 0x7fe434115d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:39:22.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.119+0000 7fe43be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4341037a0 0x7fe434116260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:39:22.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.119+0000 7fe43be00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4341168b0 con 0x7fe4341037a0 2024-03-30T04:39:22.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.119+0000 7fe43be00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe43410fc80 con 0x7fe434102e00 2024-03-30T04:39:22.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.119+0000 7fe438a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4341037a0 0x7fe434116260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:39:22.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.119+0000 7fe438a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4341037a0 0x7fe434116260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48746/0 (socket says 172.21.15.123:48746) 2024-03-30T04:39:22.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.120+0000 7fe438a00700 1 -- 172.21.15.123:0/3387418763 learned_addr learned my addr 172.21.15.123:0/3387418763 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:39:22.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.120+0000 7fe439400700 1 --2- 172.21.15.123:0/3387418763 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe434102e00 0x7fe434115d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:39:22.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.120+0000 7fe438a00700 1 -- 172.21.15.123:0/3387418763 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe434102e00 msgr2=0x7fe434115d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:22.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.120+0000 7fe438a00700 1 --2- 172.21.15.123:0/3387418763 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe434102e00 0x7fe434115d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:22.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.120+0000 7fe438a00700 1 -- 172.21.15.123:0/3387418763 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe420005040 con 0x7fe4341037a0 2024-03-30T04:39:22.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.120+0000 7fe438a00700 1 --2- 172.21.15.123:0/3387418763 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4341037a0 0x7fe434116260 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe420009390 tx=0x7fe420009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:39:22.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.120+0000 7fe431600700 1 -- 172.21.15.123:0/3387418763 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe42000e040 con 0x7fe4341037a0 2024-03-30T04:39:22.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.120+0000 7fe431600700 1 -- 172.21.15.123:0/3387418763 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe420013420 con 0x7fe4341037a0 2024-03-30T04:39:22.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.120+0000 7fe431600700 1 -- 172.21.15.123:0/3387418763 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe42001c630 con 0x7fe4341037a0 2024-03-30T04:39:22.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.120+0000 7fe43be00700 1 -- 172.21.15.123:0/3387418763 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe43410ff00 con 0x7fe4341037a0 2024-03-30T04:39:22.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.121+0000 7fe43be00700 1 -- 172.21.15.123:0/3387418763 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4341104c0 con 0x7fe4341037a0 2024-03-30T04:39:22.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.122+0000 7fe43be00700 1 -- 172.21.15.123:0/3387418763 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4340fe550 con 0x7fe4341037a0 2024-03-30T04:39:22.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.123+0000 7fe431600700 1 -- 172.21.15.123:0/3387418763 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fe420013a00 con 0x7fe4341037a0 2024-03-30T04:39:22.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.124+0000 7fe431600700 1 --2- 172.21.15.123:0/3387418763 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe41c06cc30 0x7fe41c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:39:22.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.124+0000 7fe431600700 1 -- 172.21.15.123:0/3387418763 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe42008d0e0 con 0x7fe4341037a0 2024-03-30T04:39:22.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.124+0000 7fe439400700 1 --2- 172.21.15.123:0/3387418763 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe41c06cc30 0x7fe41c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:39:22.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.125+0000 7fe439400700 1 --2- 172.21.15.123:0/3387418763 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe41c06cc30 0x7fe41c06f0e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe428004f40 tx=0x7fe42800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:39:22.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.128+0000 7fe431600700 1 -- 172.21.15.123:0/3387418763 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe42005abb0 con 0x7fe4341037a0 2024-03-30T04:39:22.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.372+0000 7fe43be00700 1 -- 172.21.15.123:0/3387418763 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe43404bed0 con 0x7fe4341037a0 2024-03-30T04:39:22.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.373+0000 7fe431600700 1 -- 172.21.15.123:0/3387418763 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1843 (secure 0 0 0) 0x7fe42005a740 con 0x7fe4341037a0 2024-03-30T04:39:22.371 INFO:teuthology.orchestra.run.smithi123.stdout:e32 2024-03-30T04:39:22.371 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:39:22.371 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:39:22.372 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:39:22.372 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:39:22.372 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:39:22.372 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:39:22.372 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T04:39:22.372 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T04:39:22.372 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:39:22.372 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T04:39:22.372 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:39:22.372 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:39:22.372 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:39:22.372 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:39:22.372 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:39:22.372 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T04:39:22.373 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:39:22.373 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:39:22.373 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T04:39:22.373 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:39:22.373 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T04:39:22.373 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T04:39:22.373 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T04:39:22.373 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T04:39:22.373 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:39:22.373 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:39:22.373 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:39:22.373 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:39:22.374 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:39:22.374 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:39:22.374 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T04:39:22.374 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T04:39:22.374 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:39:22.374 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:39:22.374 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T04:39:22.374 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:39:22.374 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:39:22.374 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:39:22.374 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44315} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/1741352336,v1:172.21.15.135:6827/1741352336] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:39:22.374 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:39:22.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.375+0000 7fe43be00700 1 -- 172.21.15.123:0/3387418763 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe41c06cc30 msgr2=0x7fe41c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:22.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.375+0000 7fe43be00700 1 --2- 172.21.15.123:0/3387418763 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe41c06cc30 0x7fe41c06f0e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe428004f40 tx=0x7fe42800b040 comp rx=0 tx=0).stop 2024-03-30T04:39:22.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.375+0000 7fe43be00700 1 -- 172.21.15.123:0/3387418763 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4341037a0 msgr2=0x7fe434116260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:22.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.375+0000 7fe43be00700 1 --2- 172.21.15.123:0/3387418763 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4341037a0 0x7fe434116260 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe420009390 tx=0x7fe420009e30 comp rx=0 tx=0).stop 2024-03-30T04:39:22.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.375+0000 7fe43be00700 1 -- 172.21.15.123:0/3387418763 shutdown_connections 2024-03-30T04:39:22.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.375+0000 7fe43be00700 1 --2- 172.21.15.123:0/3387418763 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe41c06cc30 0x7fe41c06f0e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:22.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.375+0000 7fe43be00700 1 --2- 172.21.15.123:0/3387418763 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe434102e00 0x7fe434115d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:22.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.375+0000 7fe43be00700 1 --2- 172.21.15.123:0/3387418763 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4341037a0 0x7fe434116260 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:22.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.375+0000 7fe43be00700 1 -- 172.21.15.123:0/3387418763 >> 172.21.15.123:0/3387418763 conn(0x7fe4340f89b0 msgr2=0x7fe4340f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:39:22.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.375+0000 7fe43be00700 1 -- 172.21.15.123:0/3387418763 shutdown_connections 2024-03-30T04:39:22.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.375+0000 7fe43be00700 1 -- 172.21.15.123:0/3387418763 wait complete. 2024-03-30T04:39:22.376 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 32 2024-03-30T04:39:22.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.511+0000 7f27a8400700 1 -- 172.21.15.123:0/856248973 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27a0102e00 msgr2=0x7f27a01031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:22.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.511+0000 7f27a8400700 1 --2- 172.21.15.123:0/856248973 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27a0102e00 0x7f27a01031d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f2794004940 tx=0x7f2794009e30 comp rx=0 tx=0).stop 2024-03-30T04:39:22.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.511+0000 7f27a8400700 1 -- 172.21.15.123:0/856248973 shutdown_connections 2024-03-30T04:39:22.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.511+0000 7f27a8400700 1 --2- 172.21.15.123:0/856248973 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f27a01037a0 0x7f27a00fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:22.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.511+0000 7f27a8400700 1 --2- 172.21.15.123:0/856248973 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27a0102e00 0x7f27a01031d0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:22.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.511+0000 7f27a8400700 1 -- 172.21.15.123:0/856248973 >> 172.21.15.123:0/856248973 conn(0x7f27a00f89b0 msgr2=0x7f27a00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:39:22.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.511+0000 7f27a8400700 1 -- 172.21.15.123:0/856248973 shutdown_connections 2024-03-30T04:39:22.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.511+0000 7f27a8400700 1 -- 172.21.15.123:0/856248973 wait complete. 2024-03-30T04:39:22.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.512+0000 7f27a8400700 1 Processor -- start 2024-03-30T04:39:22.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.512+0000 7f27a8400700 1 -- start start 2024-03-30T04:39:22.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.512+0000 7f27a8400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27a0102e00 0x7f27a00702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:39:22.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.512+0000 7f27a8400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f27a01037a0 0x7f27a0070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:39:22.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.512+0000 7f27a8400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a0070d90 con 0x7f27a0102e00 2024-03-30T04:39:22.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.512+0000 7f27a8400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a0070ed0 con 0x7f27a01037a0 2024-03-30T04:39:22.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.513+0000 7f27a5a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27a0102e00 0x7f27a00702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:39:22.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.513+0000 7f27a5a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27a0102e00 0x7f27a00702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48764/0 (socket says 172.21.15.123:48764) 2024-03-30T04:39:22.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.513+0000 7f27a5a00700 1 -- 172.21.15.123:0/1265296021 learned_addr learned my addr 172.21.15.123:0/1265296021 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:39:22.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.513+0000 7f27a5000700 1 --2- 172.21.15.123:0/1265296021 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f27a01037a0 0x7f27a0070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:39:22.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.513+0000 7f27a5a00700 1 -- 172.21.15.123:0/1265296021 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f27a01037a0 msgr2=0x7f27a0070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:22.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.513+0000 7f27a5a00700 1 --2- 172.21.15.123:0/1265296021 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f27a01037a0 0x7f27a0070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:22.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.513+0000 7f27a5a00700 1 -- 172.21.15.123:0/1265296021 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2794005040 con 0x7f27a0102e00 2024-03-30T04:39:22.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.513+0000 7f27a5a00700 1 --2- 172.21.15.123:0/1265296021 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27a0102e00 0x7f27a00702c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f2794009360 tx=0x7f2794009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:39:22.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.513+0000 7f279d600700 1 -- 172.21.15.123:0/1265296021 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f279400e040 con 0x7f27a0102e00 2024-03-30T04:39:22.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.514+0000 7f27a8400700 1 -- 172.21.15.123:0/1265296021 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27a0071120 con 0x7f27a0102e00 2024-03-30T04:39:22.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.514+0000 7f279d600700 1 -- 172.21.15.123:0/1265296021 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2794013420 con 0x7f27a0102e00 2024-03-30T04:39:22.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.514+0000 7f279d600700 1 -- 172.21.15.123:0/1265296021 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f279401c5f0 con 0x7f27a0102e00 2024-03-30T04:39:22.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.514+0000 7f27a8400700 1 -- 172.21.15.123:0/1265296021 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27a011a270 con 0x7f27a0102e00 2024-03-30T04:39:22.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.515+0000 7f27a8400700 1 -- 172.21.15.123:0/1265296021 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27a00fe550 con 0x7f27a0102e00 2024-03-30T04:39:22.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.516+0000 7f279d600700 1 -- 172.21.15.123:0/1265296021 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f2794013590 con 0x7f27a0102e00 2024-03-30T04:39:22.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.516+0000 7f279d600700 1 --2- 172.21.15.123:0/1265296021 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f278806cc30 0x7f278806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:39:22.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.516+0000 7f279d600700 1 -- 172.21.15.123:0/1265296021 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f279408d340 con 0x7f27a0102e00 2024-03-30T04:39:22.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.517+0000 7f27a5000700 1 --2- 172.21.15.123:0/1265296021 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f278806cc30 0x7f278806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:39:22.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.517+0000 7f27a5000700 1 --2- 172.21.15.123:0/1265296021 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f278806cc30 0x7f278806f0e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f278c009fd0 tx=0x7f278c009f20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:39:22.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.521+0000 7f279d600700 1 -- 172.21.15.123:0/1265296021 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2794057300 con 0x7f27a0102e00 2024-03-30T04:39:22.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:39:22 smithi123 ceph-mon[139369]: from='client.34518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:39:22.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:39:22 smithi123 ceph-mon[139369]: pgmap v245: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:22.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:39:22 smithi123 ceph-mon[139369]: from='client.34522 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:39:22.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:39:22 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/839386928' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:39:22.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:39:22 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3387418763' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:39:22.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.743+0000 7f27a8400700 1 -- 172.21.15.123:0/1265296021 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f27a00719b0 con 0x7f278806cc30 2024-03-30T04:39:22.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.744+0000 7f279d600700 1 -- 172.21.15.123:0/1265296021 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f27a00719b0 con 0x7f278806cc30 2024-03-30T04:39:22.743 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:39:22.743 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:39:22.743 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:39:22.743 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T04:39:22.743 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:39:22.743 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T04:39:22.743 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T04:39:22.743 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T04:39:22.743 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T04:39:22.743 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:39:22.744 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "15/21 daemons upgraded", 2024-03-30T04:39:22.744 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T04:39:22.744 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:39:22.744 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:39:22.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.746+0000 7f27a8400700 1 -- 172.21.15.123:0/1265296021 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f278806cc30 msgr2=0x7f278806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:22.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.746+0000 7f27a8400700 1 --2- 172.21.15.123:0/1265296021 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f278806cc30 0x7f278806f0e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f278c009fd0 tx=0x7f278c009f20 comp rx=0 tx=0).stop 2024-03-30T04:39:22.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.747+0000 7f27a8400700 1 -- 172.21.15.123:0/1265296021 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27a0102e00 msgr2=0x7f27a00702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:22.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.747+0000 7f27a8400700 1 --2- 172.21.15.123:0/1265296021 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27a0102e00 0x7f27a00702c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f2794009360 tx=0x7f2794009e30 comp rx=0 tx=0).stop 2024-03-30T04:39:22.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.747+0000 7f27a8400700 1 -- 172.21.15.123:0/1265296021 shutdown_connections 2024-03-30T04:39:22.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.747+0000 7f27a8400700 1 --2- 172.21.15.123:0/1265296021 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f278806cc30 0x7f278806f0e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:22.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.747+0000 7f27a8400700 1 --2- 172.21.15.123:0/1265296021 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27a0102e00 0x7f27a00702c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:22.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.747+0000 7f27a8400700 1 --2- 172.21.15.123:0/1265296021 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f27a01037a0 0x7f27a0070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:22.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.747+0000 7f27a8400700 1 -- 172.21.15.123:0/1265296021 >> 172.21.15.123:0/1265296021 conn(0x7f27a00f89b0 msgr2=0x7f27a00f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:39:22.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.747+0000 7f27a8400700 1 -- 172.21.15.123:0/1265296021 shutdown_connections 2024-03-30T04:39:22.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.747+0000 7f27a8400700 1 -- 172.21.15.123:0/1265296021 wait complete. 2024-03-30T04:39:22.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:39:22 smithi135 ceph-mon[106011]: from='client.34518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:39:22.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:39:22 smithi135 ceph-mon[106011]: pgmap v245: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:22.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:39:22 smithi135 ceph-mon[106011]: from='client.34522 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:39:22.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:39:22 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/839386928' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:39:22.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:39:22 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3387418763' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:39:22.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.880+0000 7f4e7be00700 1 -- 172.21.15.123:0/2894699943 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e7c0fe5e0 msgr2=0x7f4e7c106c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:22.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.880+0000 7f4e7be00700 1 --2- 172.21.15.123:0/2894699943 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e7c0fe5e0 0x7f4e7c106c20 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f4e68004940 tx=0x7f4e68009e30 comp rx=0 tx=0).stop 2024-03-30T04:39:22.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.880+0000 7f4e7be00700 1 -- 172.21.15.123:0/2894699943 shutdown_connections 2024-03-30T04:39:22.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.880+0000 7f4e7be00700 1 --2- 172.21.15.123:0/2894699943 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e7c0fe5e0 0x7f4e7c106c20 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:22.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.880+0000 7f4e7be00700 1 --2- 172.21.15.123:0/2894699943 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4e7c0fdc40 0x7f4e7c0fe010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:22.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.880+0000 7f4e7be00700 1 -- 172.21.15.123:0/2894699943 >> 172.21.15.123:0/2894699943 conn(0x7f4e7c0f7670 msgr2=0x7f4e7c0f9a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:39:22.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.880+0000 7f4e7be00700 1 -- 172.21.15.123:0/2894699943 shutdown_connections 2024-03-30T04:39:22.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.880+0000 7f4e7be00700 1 -- 172.21.15.123:0/2894699943 wait complete. 2024-03-30T04:39:22.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.881+0000 7f4e7be00700 1 Processor -- start 2024-03-30T04:39:22.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.881+0000 7f4e7be00700 1 -- start start 2024-03-30T04:39:22.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.881+0000 7f4e7be00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4e7c0fdc40 0x7f4e7c1139a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:39:22.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.881+0000 7f4e7be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e7c0fe5e0 0x7f4e7c113ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:39:22.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.881+0000 7f4e7be00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7c114530 con 0x7f4e7c0fe5e0 2024-03-30T04:39:22.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.881+0000 7f4e7be00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7c10d920 con 0x7f4e7c0fdc40 2024-03-30T04:39:22.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.881+0000 7f4e7a000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e7c0fe5e0 0x7f4e7c113ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:39:22.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.881+0000 7f4e7a000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e7c0fe5e0 0x7f4e7c113ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48790/0 (socket says 172.21.15.123:48790) 2024-03-30T04:39:22.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.881+0000 7f4e7a000700 1 -- 172.21.15.123:0/1716316320 learned_addr learned my addr 172.21.15.123:0/1716316320 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:39:22.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.881+0000 7f4e7aa00700 1 --2- 172.21.15.123:0/1716316320 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4e7c0fdc40 0x7f4e7c1139a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:39:22.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.882+0000 7f4e7a000700 1 -- 172.21.15.123:0/1716316320 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4e7c0fdc40 msgr2=0x7f4e7c1139a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:22.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.882+0000 7f4e7a000700 1 --2- 172.21.15.123:0/1716316320 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4e7c0fdc40 0x7f4e7c1139a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:22.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.882+0000 7f4e7a000700 1 -- 172.21.15.123:0/1716316320 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e68005040 con 0x7f4e7c0fe5e0 2024-03-30T04:39:22.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.882+0000 7f4e7a000700 1 --2- 172.21.15.123:0/1716316320 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e7c0fe5e0 0x7f4e7c113ee0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4e6800b3b0 tx=0x7f4e68009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:39:22.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.882+0000 7f4e72a00700 1 -- 172.21.15.123:0/1716316320 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e6800e040 con 0x7f4e7c0fe5e0 2024-03-30T04:39:22.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.882+0000 7f4e72a00700 1 -- 172.21.15.123:0/1716316320 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4e68013420 con 0x7f4e7c0fe5e0 2024-03-30T04:39:22.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.882+0000 7f4e72a00700 1 -- 172.21.15.123:0/1716316320 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e6801c630 con 0x7f4e7c0fe5e0 2024-03-30T04:39:22.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.882+0000 7f4e7be00700 1 -- 172.21.15.123:0/1716316320 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e7c10dba0 con 0x7f4e7c0fe5e0 2024-03-30T04:39:22.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.882+0000 7f4e7be00700 1 -- 172.21.15.123:0/1716316320 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e7c10e0e0 con 0x7f4e7c0fe5e0 2024-03-30T04:39:22.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.883+0000 7f4e7be00700 1 -- 172.21.15.123:0/1716316320 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e7c066930 con 0x7f4e7c0fe5e0 2024-03-30T04:39:22.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.884+0000 7f4e72a00700 1 -- 172.21.15.123:0/1716316320 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f4e68013590 con 0x7f4e7c0fe5e0 2024-03-30T04:39:22.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.886+0000 7f4e72a00700 1 --2- 172.21.15.123:0/1716316320 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4e6006cc20 0x7f4e6006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:39:22.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.886+0000 7f4e72a00700 1 -- 172.21.15.123:0/1716316320 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f4e6808d1d0 con 0x7f4e7c0fe5e0 2024-03-30T04:39:22.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.886+0000 7f4e7aa00700 1 --2- 172.21.15.123:0/1716316320 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4e6006cc20 0x7f4e6006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:39:22.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.886+0000 7f4e7aa00700 1 --2- 172.21.15.123:0/1716316320 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4e6006cc20 0x7f4e6006f0d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c004f40 tx=0x7f4e6c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:39:22.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:22.890+0000 7f4e72a00700 1 -- 172.21.15.123:0/1716316320 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4e6805aca0 con 0x7f4e7c0fe5e0 2024-03-30T04:39:23.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:23.174+0000 7f4e7be00700 1 -- 172.21.15.123:0/1716316320 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f4e7c109ba0 con 0x7f4e7c0fe5e0 2024-03-30T04:39:23.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:23.174+0000 7f4e72a00700 1 -- 172.21.15.123:0/1716316320 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f4e6805a830 con 0x7f4e7c0fe5e0 2024-03-30T04:39:23.173 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T04:39:23.173 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T04:39:23.173 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T04:39:23.173 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T04:39:23.173 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T04:39:23.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:23.176+0000 7f4e7be00700 1 -- 172.21.15.123:0/1716316320 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4e6006cc20 msgr2=0x7f4e6006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:23.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:23.176+0000 7f4e7be00700 1 --2- 172.21.15.123:0/1716316320 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4e6006cc20 0x7f4e6006f0d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c004f40 tx=0x7f4e6c00b040 comp rx=0 tx=0).stop 2024-03-30T04:39:23.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:23.176+0000 7f4e7be00700 1 -- 172.21.15.123:0/1716316320 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e7c0fe5e0 msgr2=0x7f4e7c113ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:23.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:23.176+0000 7f4e7be00700 1 --2- 172.21.15.123:0/1716316320 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e7c0fe5e0 0x7f4e7c113ee0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4e6800b3b0 tx=0x7f4e68009e30 comp rx=0 tx=0).stop 2024-03-30T04:39:23.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:23.176+0000 7f4e7be00700 1 -- 172.21.15.123:0/1716316320 shutdown_connections 2024-03-30T04:39:23.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:23.176+0000 7f4e7be00700 1 --2- 172.21.15.123:0/1716316320 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4e6006cc20 0x7f4e6006f0d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:23.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:23.176+0000 7f4e7be00700 1 --2- 172.21.15.123:0/1716316320 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4e7c0fdc40 0x7f4e7c1139a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:23.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:23.176+0000 7f4e7be00700 1 --2- 172.21.15.123:0/1716316320 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e7c0fe5e0 0x7f4e7c113ee0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:23.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:23.176+0000 7f4e7be00700 1 -- 172.21.15.123:0/1716316320 >> 172.21.15.123:0/1716316320 conn(0x7f4e7c0f7670 msgr2=0x7f4e7c0f8c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:39:23.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:23.177+0000 7f4e7be00700 1 -- 172.21.15.123:0/1716316320 shutdown_connections 2024-03-30T04:39:23.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:23.177+0000 7f4e7be00700 1 -- 172.21.15.123:0/1716316320 wait complete. 2024-03-30T04:39:23.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:39:23 smithi123 ceph-mon[139369]: from='client.34534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:39:23.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:39:23 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/1716316320' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:39:23.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:39:23 smithi135 ceph-mon[106011]: from='client.34534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:39:23.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:39:23 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/1716316320' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:39:24.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:39:24 smithi123 ceph-mon[139369]: pgmap v246: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:24.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:39:24 smithi135 ceph-mon[106011]: pgmap v246: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:26.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:39:26 smithi135 ceph-mon[106011]: pgmap v247: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:26.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:39:26 smithi123 ceph-mon[139369]: pgmap v247: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:28.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:39:28 smithi135 ceph-mon[106011]: pgmap v248: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:28.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:39:28 smithi123 ceph-mon[139369]: pgmap v248: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:30.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:39:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:39:30.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:39:30 smithi123 ceph-mon[139369]: pgmap v249: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:30.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:39:30 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:39:30.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:39:30 smithi135 ceph-mon[106011]: pgmap v249: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:32.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:39:32 smithi135 ceph-mon[106011]: pgmap v250: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:32.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:39:32 smithi123 ceph-mon[139369]: pgmap v250: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:34.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:39:34 smithi135 ceph-mon[106011]: pgmap v251: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:34.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:39:34 smithi123 ceph-mon[139369]: pgmap v251: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:36.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:39:36 smithi135 ceph-mon[106011]: pgmap v252: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:36.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:39:36 smithi123 ceph-mon[139369]: pgmap v252: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:38.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:39:38 smithi135 ceph-mon[106011]: pgmap v253: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:38.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:39:38 smithi123 ceph-mon[139369]: pgmap v253: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:40.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:39:40 smithi123 ceph-mon[139369]: pgmap v254: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:40.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:39:40 smithi135 ceph-mon[106011]: pgmap v254: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:42.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:39:42 smithi135 ceph-mon[106011]: pgmap v255: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:42.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:39:42 smithi123 ceph-mon[139369]: pgmap v255: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:44.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:39:44 smithi135 ceph-mon[106011]: pgmap v256: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:44.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:39:44 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:39:44.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:39:44 smithi123 ceph-mon[139369]: pgmap v256: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:44.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:39:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:39:46.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:39:46 smithi135 ceph-mon[106011]: pgmap v257: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:46.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:39:46 smithi123 ceph-mon[139369]: pgmap v257: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:48.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:39:48 smithi135 ceph-mon[106011]: pgmap v258: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:48.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:39:48 smithi123 ceph-mon[139369]: pgmap v258: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:50.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:39:50 smithi123 ceph-mon[139369]: pgmap v259: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:50.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:39:50 smithi135 ceph-mon[106011]: pgmap v259: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:52.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:39:52 smithi135 ceph-mon[106011]: pgmap v260: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:52.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:39:52 smithi123 ceph-mon[139369]: pgmap v260: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:53.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.328+0000 7fa03c000700 1 -- 172.21.15.123:0/4094083668 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa034102e20 msgr2=0x7fa0341031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:53.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.328+0000 7fa03c000700 1 --2- 172.21.15.123:0/4094083668 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa034102e20 0x7fa0341031f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa028004940 tx=0x7fa028009e30 comp rx=0 tx=0).stop 2024-03-30T04:39:53.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.328+0000 7fa03c000700 1 -- 172.21.15.123:0/4094083668 shutdown_connections 2024-03-30T04:39:53.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.328+0000 7fa03c000700 1 --2- 172.21.15.123:0/4094083668 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa0341037c0 0x7fa0340fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:53.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.328+0000 7fa03c000700 1 --2- 172.21.15.123:0/4094083668 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa034102e20 0x7fa0341031f0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:53.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.328+0000 7fa03c000700 1 -- 172.21.15.123:0/4094083668 >> 172.21.15.123:0/4094083668 conn(0x7fa0340f89d0 msgr2=0x7fa0340fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:39:53.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.328+0000 7fa03c000700 1 -- 172.21.15.123:0/4094083668 shutdown_connections 2024-03-30T04:39:53.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.328+0000 7fa03c000700 1 -- 172.21.15.123:0/4094083668 wait complete. 2024-03-30T04:39:53.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.329+0000 7fa03c000700 1 Processor -- start 2024-03-30T04:39:53.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.329+0000 7fa03c000700 1 -- start start 2024-03-30T04:39:53.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.330+0000 7fa03c000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa034102e20 0x7fa034115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:39:53.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.330+0000 7fa03c000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0341037c0 0x7fa034116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:39:53.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.330+0000 7fa03c000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa034116880 con 0x7fa0341037c0 2024-03-30T04:39:53.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.330+0000 7fa03c000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa03410fc50 con 0x7fa034102e20 2024-03-30T04:39:53.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.330+0000 7fa038c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0341037c0 0x7fa034116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:39:53.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.330+0000 7fa038c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0341037c0 0x7fa034116230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35242/0 (socket says 172.21.15.123:35242) 2024-03-30T04:39:53.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.330+0000 7fa038c00700 1 -- 172.21.15.123:0/3165581898 learned_addr learned my addr 172.21.15.123:0/3165581898 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:39:53.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.330+0000 7fa039600700 1 --2- 172.21.15.123:0/3165581898 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa034102e20 0x7fa034115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:39:53.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.330+0000 7fa038c00700 1 -- 172.21.15.123:0/3165581898 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa034102e20 msgr2=0x7fa034115cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:53.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.330+0000 7fa038c00700 1 --2- 172.21.15.123:0/3165581898 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa034102e20 0x7fa034115cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:53.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.330+0000 7fa038c00700 1 -- 172.21.15.123:0/3165581898 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa028005040 con 0x7fa0341037c0 2024-03-30T04:39:53.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.331+0000 7fa038c00700 1 --2- 172.21.15.123:0/3165581898 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0341037c0 0x7fa034116230 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa020002a50 tx=0x7fa020004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:39:53.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.331+0000 7fa031600700 1 -- 172.21.15.123:0/3165581898 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa020007cc0 con 0x7fa0341037c0 2024-03-30T04:39:53.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.331+0000 7fa031600700 1 -- 172.21.15.123:0/3165581898 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa02000f040 con 0x7fa0341037c0 2024-03-30T04:39:53.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.331+0000 7fa031600700 1 -- 172.21.15.123:0/3165581898 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa02000d920 con 0x7fa0341037c0 2024-03-30T04:39:53.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.331+0000 7fa03c000700 1 -- 172.21.15.123:0/3165581898 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa03410ff30 con 0x7fa0341037c0 2024-03-30T04:39:53.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.331+0000 7fa03c000700 1 -- 172.21.15.123:0/3165581898 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0341104f0 con 0x7fa0341037c0 2024-03-30T04:39:53.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.332+0000 7fa03c000700 1 -- 172.21.15.123:0/3165581898 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0340fe570 con 0x7fa0341037c0 2024-03-30T04:39:53.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.333+0000 7fa031600700 1 -- 172.21.15.123:0/3165581898 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fa020009680 con 0x7fa0341037c0 2024-03-30T04:39:53.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.335+0000 7fa031600700 1 --2- 172.21.15.123:0/3165581898 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa01c06cc30 0x7fa01c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:39:53.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.335+0000 7fa031600700 1 -- 172.21.15.123:0/3165581898 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa02008bd80 con 0x7fa0341037c0 2024-03-30T04:39:53.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.336+0000 7fa039600700 1 --2- 172.21.15.123:0/3165581898 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa01c06cc30 0x7fa01c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:39:53.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.336+0000 7fa039600700 1 --2- 172.21.15.123:0/3165581898 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa01c06cc30 0x7fa01c06f0e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa028009fd0 tx=0x7fa028009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:39:53.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.340+0000 7fa031600700 1 -- 172.21.15.123:0/3165581898 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa0200597d0 con 0x7fa0341037c0 2024-03-30T04:39:53.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.565+0000 7fa03c000700 1 -- 172.21.15.123:0/3165581898 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa034109e50 con 0x7fa01c06cc30 2024-03-30T04:39:53.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.566+0000 7fa031600700 1 -- 172.21.15.123:0/3165581898 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fa034109e50 con 0x7fa01c06cc30 2024-03-30T04:39:53.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.569+0000 7fa03c000700 1 -- 172.21.15.123:0/3165581898 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa01c06cc30 msgr2=0x7fa01c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:53.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.569+0000 7fa03c000700 1 --2- 172.21.15.123:0/3165581898 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa01c06cc30 0x7fa01c06f0e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa028009fd0 tx=0x7fa028009e30 comp rx=0 tx=0).stop 2024-03-30T04:39:53.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.569+0000 7fa03c000700 1 -- 172.21.15.123:0/3165581898 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0341037c0 msgr2=0x7fa034116230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:53.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.569+0000 7fa03c000700 1 --2- 172.21.15.123:0/3165581898 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0341037c0 0x7fa034116230 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa020002a50 tx=0x7fa020004600 comp rx=0 tx=0).stop 2024-03-30T04:39:53.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.569+0000 7fa03c000700 1 -- 172.21.15.123:0/3165581898 shutdown_connections 2024-03-30T04:39:53.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.569+0000 7fa03c000700 1 --2- 172.21.15.123:0/3165581898 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa01c06cc30 0x7fa01c06f0e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:53.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.569+0000 7fa03c000700 1 --2- 172.21.15.123:0/3165581898 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa034102e20 0x7fa034115cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:53.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.569+0000 7fa03c000700 1 --2- 172.21.15.123:0/3165581898 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0341037c0 0x7fa034116230 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:53.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.569+0000 7fa03c000700 1 -- 172.21.15.123:0/3165581898 >> 172.21.15.123:0/3165581898 conn(0x7fa0340f89d0 msgr2=0x7fa0340f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:39:53.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.569+0000 7fa03c000700 1 -- 172.21.15.123:0/3165581898 shutdown_connections 2024-03-30T04:39:53.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.569+0000 7fa03c000700 1 -- 172.21.15.123:0/3165581898 wait complete. 2024-03-30T04:39:53.582 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:39:53.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.705+0000 7fb2baa00700 1 -- 172.21.15.123:0/2343320763 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2b41037a0 msgr2=0x7fb2b40fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:53.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.705+0000 7fb2baa00700 1 --2- 172.21.15.123:0/2343320763 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2b41037a0 0x7fb2b40fce10 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb2a0004940 tx=0x7fb2a0009e30 comp rx=0 tx=0).stop 2024-03-30T04:39:53.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.705+0000 7fb2baa00700 1 -- 172.21.15.123:0/2343320763 shutdown_connections 2024-03-30T04:39:53.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.705+0000 7fb2baa00700 1 --2- 172.21.15.123:0/2343320763 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2b41037a0 0x7fb2b40fce10 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:53.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.705+0000 7fb2baa00700 1 --2- 172.21.15.123:0/2343320763 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb2b4102e00 0x7fb2b41031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:53.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.705+0000 7fb2baa00700 1 -- 172.21.15.123:0/2343320763 >> 172.21.15.123:0/2343320763 conn(0x7fb2b40f89b0 msgr2=0x7fb2b40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:39:53.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.705+0000 7fb2baa00700 1 -- 172.21.15.123:0/2343320763 shutdown_connections 2024-03-30T04:39:53.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.705+0000 7fb2baa00700 1 -- 172.21.15.123:0/2343320763 wait complete. 2024-03-30T04:39:53.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.706+0000 7fb2baa00700 1 Processor -- start 2024-03-30T04:39:53.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.706+0000 7fb2baa00700 1 -- start start 2024-03-30T04:39:53.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.706+0000 7fb2baa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2b4102e00 0x7fb2b4115d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:39:53.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.706+0000 7fb2baa00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb2b41037a0 0x7fb2b4116260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:39:53.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.706+0000 7fb2baa00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b41168b0 con 0x7fb2b4102e00 2024-03-30T04:39:53.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.706+0000 7fb2baa00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b410fc80 con 0x7fb2b41037a0 2024-03-30T04:39:53.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.706+0000 7fb2b3e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2b4102e00 0x7fb2b4115d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:39:53.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.707+0000 7fb2b3e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2b4102e00 0x7fb2b4115d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35260/0 (socket says 172.21.15.123:35260) 2024-03-30T04:39:53.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.707+0000 7fb2b3e00700 1 -- 172.21.15.123:0/1238825071 learned_addr learned my addr 172.21.15.123:0/1238825071 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:39:53.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.707+0000 7fb2b3400700 1 --2- 172.21.15.123:0/1238825071 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb2b41037a0 0x7fb2b4116260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:39:53.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.707+0000 7fb2b3e00700 1 -- 172.21.15.123:0/1238825071 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb2b41037a0 msgr2=0x7fb2b4116260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:53.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.707+0000 7fb2b3e00700 1 --2- 172.21.15.123:0/1238825071 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb2b41037a0 0x7fb2b4116260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:53.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.707+0000 7fb2b3e00700 1 -- 172.21.15.123:0/1238825071 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2a0005040 con 0x7fb2b4102e00 2024-03-30T04:39:53.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.707+0000 7fb2b3e00700 1 --2- 172.21.15.123:0/1238825071 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2b4102e00 0x7fb2b4115d20 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb2a4002a50 tx=0x7fb2a40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:39:53.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.708+0000 7fb2abe00700 1 -- 172.21.15.123:0/1238825071 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb2a4007cc0 con 0x7fb2b4102e00 2024-03-30T04:39:53.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.708+0000 7fb2abe00700 1 -- 172.21.15.123:0/1238825071 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb2a400f040 con 0x7fb2b4102e00 2024-03-30T04:39:53.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.708+0000 7fb2baa00700 1 -- 172.21.15.123:0/1238825071 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2b410ff60 con 0x7fb2b4102e00 2024-03-30T04:39:53.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.708+0000 7fb2abe00700 1 -- 172.21.15.123:0/1238825071 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb2a400da70 con 0x7fb2b4102e00 2024-03-30T04:39:53.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.708+0000 7fb2baa00700 1 -- 172.21.15.123:0/1238825071 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2b4110520 con 0x7fb2b4102e00 2024-03-30T04:39:53.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.709+0000 7fb2baa00700 1 -- 172.21.15.123:0/1238825071 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2b40fe550 con 0x7fb2b4102e00 2024-03-30T04:39:53.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.710+0000 7fb2abe00700 1 -- 172.21.15.123:0/1238825071 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fb2a400dc10 con 0x7fb2b4102e00 2024-03-30T04:39:53.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.711+0000 7fb2abe00700 1 --2- 172.21.15.123:0/1238825071 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb29806cc20 0x7fb29806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:39:53.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.711+0000 7fb2abe00700 1 -- 172.21.15.123:0/1238825071 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb2a408b810 con 0x7fb2b4102e00 2024-03-30T04:39:53.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.711+0000 7fb2b3400700 1 --2- 172.21.15.123:0/1238825071 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb29806cc20 0x7fb29806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:39:53.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.711+0000 7fb2b3400700 1 --2- 172.21.15.123:0/1238825071 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb29806cc20 0x7fb29806f0d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb2a000b3b0 tx=0x7fb2a0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:39:53.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.715+0000 7fb2abe00700 1 -- 172.21.15.123:0/1238825071 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb2a40591e0 con 0x7fb2b4102e00 2024-03-30T04:39:53.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.938+0000 7fb2baa00700 1 -- 172.21.15.123:0/1238825071 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb2b404bf40 con 0x7fb29806cc20 2024-03-30T04:39:53.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.939+0000 7fb2abe00700 1 -- 172.21.15.123:0/1238825071 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fb2b404bf40 con 0x7fb29806cc20 2024-03-30T04:39:53.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.941+0000 7fb2baa00700 1 -- 172.21.15.123:0/1238825071 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb29806cc20 msgr2=0x7fb29806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:53.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.941+0000 7fb2baa00700 1 --2- 172.21.15.123:0/1238825071 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb29806cc20 0x7fb29806f0d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb2a000b3b0 tx=0x7fb2a0009e30 comp rx=0 tx=0).stop 2024-03-30T04:39:53.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.941+0000 7fb2baa00700 1 -- 172.21.15.123:0/1238825071 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2b4102e00 msgr2=0x7fb2b4115d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:53.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.941+0000 7fb2baa00700 1 --2- 172.21.15.123:0/1238825071 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2b4102e00 0x7fb2b4115d20 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb2a4002a50 tx=0x7fb2a40046b0 comp rx=0 tx=0).stop 2024-03-30T04:39:53.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.942+0000 7fb2baa00700 1 -- 172.21.15.123:0/1238825071 shutdown_connections 2024-03-30T04:39:53.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.942+0000 7fb2baa00700 1 --2- 172.21.15.123:0/1238825071 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb29806cc20 0x7fb29806f0d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:53.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.942+0000 7fb2baa00700 1 --2- 172.21.15.123:0/1238825071 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2b4102e00 0x7fb2b4115d20 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:53.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.942+0000 7fb2baa00700 1 --2- 172.21.15.123:0/1238825071 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb2b41037a0 0x7fb2b4116260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:53.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.942+0000 7fb2baa00700 1 -- 172.21.15.123:0/1238825071 >> 172.21.15.123:0/1238825071 conn(0x7fb2b40f89b0 msgr2=0x7fb2b4070750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:39:53.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.942+0000 7fb2baa00700 1 -- 172.21.15.123:0/1238825071 shutdown_connections 2024-03-30T04:39:53.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:53.942+0000 7fb2baa00700 1 -- 172.21.15.123:0/1238825071 wait complete. 2024-03-30T04:39:54.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.077+0000 7f4514200700 1 -- 172.21.15.123:0/234468195 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f450c1037a0 msgr2=0x7f450c0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:54.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.077+0000 7f4514200700 1 --2- 172.21.15.123:0/234468195 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f450c1037a0 0x7f450c0fce10 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f44f8004940 tx=0x7f44f8009e30 comp rx=0 tx=0).stop 2024-03-30T04:39:54.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.077+0000 7f4514200700 1 -- 172.21.15.123:0/234468195 shutdown_connections 2024-03-30T04:39:54.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.077+0000 7f4514200700 1 --2- 172.21.15.123:0/234468195 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f450c1037a0 0x7f450c0fce10 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:54.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.077+0000 7f4514200700 1 --2- 172.21.15.123:0/234468195 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f450c102e00 0x7f450c1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:54.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.077+0000 7f4514200700 1 -- 172.21.15.123:0/234468195 >> 172.21.15.123:0/234468195 conn(0x7f450c0f89b0 msgr2=0x7f450c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:39:54.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.077+0000 7f4514200700 1 -- 172.21.15.123:0/234468195 shutdown_connections 2024-03-30T04:39:54.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.077+0000 7f4514200700 1 -- 172.21.15.123:0/234468195 wait complete. 2024-03-30T04:39:54.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.078+0000 7f4514200700 1 Processor -- start 2024-03-30T04:39:54.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.078+0000 7f4514200700 1 -- start start 2024-03-30T04:39:54.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.078+0000 7f4514200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f450c102e00 0x7f450c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:39:54.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.078+0000 7f4514200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f450c1037a0 0x7f450c070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:39:54.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.078+0000 7f4514200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f450c070d90 con 0x7f450c1037a0 2024-03-30T04:39:54.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.078+0000 7f4514200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f450c070ed0 con 0x7f450c102e00 2024-03-30T04:39:54.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.079+0000 7f4510e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f450c1037a0 0x7f450c070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:39:54.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.079+0000 7f4510e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f450c1037a0 0x7f450c070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57190/0 (socket says 172.21.15.123:57190) 2024-03-30T04:39:54.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.079+0000 7f4510e00700 1 -- 172.21.15.123:0/2541537063 learned_addr learned my addr 172.21.15.123:0/2541537063 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:39:54.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.079+0000 7f4511800700 1 --2- 172.21.15.123:0/2541537063 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f450c102e00 0x7f450c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:39:54.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.079+0000 7f4510e00700 1 -- 172.21.15.123:0/2541537063 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f450c102e00 msgr2=0x7f450c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:54.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.079+0000 7f4510e00700 1 --2- 172.21.15.123:0/2541537063 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f450c102e00 0x7f450c0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:54.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.079+0000 7f4510e00700 1 -- 172.21.15.123:0/2541537063 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44f8005040 con 0x7f450c1037a0 2024-03-30T04:39:54.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.079+0000 7f4510e00700 1 --2- 172.21.15.123:0/2541537063 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f450c1037a0 0x7f450c070800 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f44f8008d50 tx=0x7f44f8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:39:54.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.079+0000 7f4509600700 1 -- 172.21.15.123:0/2541537063 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44f800e030 con 0x7f450c1037a0 2024-03-30T04:39:54.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.079+0000 7f4509600700 1 -- 172.21.15.123:0/2541537063 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f44f80093b0 con 0x7f450c1037a0 2024-03-30T04:39:54.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.080+0000 7f4514200700 1 -- 172.21.15.123:0/2541537063 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f450c071120 con 0x7f450c1037a0 2024-03-30T04:39:54.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.080+0000 7f4509600700 1 -- 172.21.15.123:0/2541537063 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44f801cbb0 con 0x7f450c1037a0 2024-03-30T04:39:54.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.080+0000 7f4514200700 1 -- 172.21.15.123:0/2541537063 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f450c11a1a0 con 0x7f450c1037a0 2024-03-30T04:39:54.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.081+0000 7f4514200700 1 -- 172.21.15.123:0/2541537063 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f450c0fe550 con 0x7f450c1037a0 2024-03-30T04:39:54.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.082+0000 7f4509600700 1 -- 172.21.15.123:0/2541537063 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f44f8013930 con 0x7f450c1037a0 2024-03-30T04:39:54.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.082+0000 7f4509600700 1 --2- 172.21.15.123:0/2541537063 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f44f406cc30 0x7f44f406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:39:54.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.083+0000 7f4509600700 1 -- 172.21.15.123:0/2541537063 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f44f808d880 con 0x7f450c1037a0 2024-03-30T04:39:54.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.083+0000 7f4511800700 1 --2- 172.21.15.123:0/2541537063 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f44f406cc30 0x7f44f406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:39:54.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.083+0000 7f4511800700 1 --2- 172.21.15.123:0/2541537063 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f44f406cc30 0x7f44f406f0e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4500009b80 tx=0x7f450000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:39:54.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.087+0000 7f4509600700 1 -- 172.21.15.123:0/2541537063 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f44f80577c0 con 0x7f450c1037a0 2024-03-30T04:39:54.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.300+0000 7f4514200700 1 -- 172.21.15.123:0/2541537063 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f450c109e40 con 0x7f44f406cc30 2024-03-30T04:39:54.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.309+0000 7f4509600700 1 -- 172.21.15.123:0/2541537063 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f450000f040 con 0x7f44f406cc30 2024-03-30T04:39:54.308 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:39:54.308 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (9m) 119s ago 21m 15.0M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:39:54.308 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (6m) 119s ago 21m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T04:39:54.308 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (6m) 101s ago 20m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T04:39:54.308 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (8m) 119s ago 21m 74.0M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T04:39:54.308 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (2m) 119s ago 15m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T04:39:54.308 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (2m) 119s ago 15m 10.9M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T04:39:54.308 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (15m) 101s ago 15m 10.7M - 17.2.7-797-g5c024335 f77085f6aab7 0dc41326818c 2024-03-30T04:39:54.308 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (105s) 101s ago 15m 9747k - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T04:39:54.308 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (11m) 119s ago 23m 506M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:39:54.308 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (10m) 101s ago 20m 450M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:39:54.308 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (7m) 119s ago 23m 53.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T04:39:54.308 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (7m) 101s ago 20m 42.6M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T04:39:54.309 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (9m) 119s ago 21m 8052k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:39:54.309 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (9m) 101s ago 20m 7880k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:39:54.309 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (5m) 119s ago 19m 104M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T04:39:54.309 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (5m) 119s ago 19m 93.4M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T04:39:54.309 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (4m) 119s ago 18m 86.3M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T04:39:54.309 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (4m) 101s ago 17m 97.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T04:39:54.309 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (3m) 101s ago 17m 97.9M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T04:39:54.309 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (2m) 101s ago 16m 85.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T04:39:54.309 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (9m) 119s ago 21m 53.7M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:39:54.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.311+0000 7f4514200700 1 -- 172.21.15.123:0/2541537063 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f44f406cc30 msgr2=0x7f44f406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:54.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.311+0000 7f4514200700 1 --2- 172.21.15.123:0/2541537063 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f44f406cc30 0x7f44f406f0e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4500009b80 tx=0x7f450000b040 comp rx=0 tx=0).stop 2024-03-30T04:39:54.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.311+0000 7f4514200700 1 -- 172.21.15.123:0/2541537063 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f450c1037a0 msgr2=0x7f450c070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:54.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.312+0000 7f4514200700 1 --2- 172.21.15.123:0/2541537063 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f450c1037a0 0x7f450c070800 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f44f8008d50 tx=0x7f44f8009e30 comp rx=0 tx=0).stop 2024-03-30T04:39:54.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.312+0000 7f4514200700 1 -- 172.21.15.123:0/2541537063 shutdown_connections 2024-03-30T04:39:54.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.312+0000 7f4514200700 1 --2- 172.21.15.123:0/2541537063 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f44f406cc30 0x7f44f406f0e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:54.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.312+0000 7f4514200700 1 --2- 172.21.15.123:0/2541537063 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f450c102e00 0x7f450c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:54.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.312+0000 7f4514200700 1 --2- 172.21.15.123:0/2541537063 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f450c1037a0 0x7f450c070800 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:54.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.312+0000 7f4514200700 1 -- 172.21.15.123:0/2541537063 >> 172.21.15.123:0/2541537063 conn(0x7f450c0f89b0 msgr2=0x7f450c0f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:39:54.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.312+0000 7f4514200700 1 -- 172.21.15.123:0/2541537063 shutdown_connections 2024-03-30T04:39:54.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.312+0000 7f4514200700 1 -- 172.21.15.123:0/2541537063 wait complete. 2024-03-30T04:39:54.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.444+0000 7fee89e00700 1 -- 172.21.15.123:0/1429265132 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee84102e00 msgr2=0x7fee841031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:54.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.444+0000 7fee89e00700 1 --2- 172.21.15.123:0/1429265132 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee84102e00 0x7fee841031d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fee74004940 tx=0x7fee74009e30 comp rx=0 tx=0).stop 2024-03-30T04:39:54.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.444+0000 7fee89e00700 1 -- 172.21.15.123:0/1429265132 shutdown_connections 2024-03-30T04:39:54.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.444+0000 7fee89e00700 1 --2- 172.21.15.123:0/1429265132 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fee841037a0 0x7fee840fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:54.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.444+0000 7fee89e00700 1 --2- 172.21.15.123:0/1429265132 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee84102e00 0x7fee841031d0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:54.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.444+0000 7fee89e00700 1 -- 172.21.15.123:0/1429265132 >> 172.21.15.123:0/1429265132 conn(0x7fee840f89b0 msgr2=0x7fee840fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:39:54.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.444+0000 7fee89e00700 1 -- 172.21.15.123:0/1429265132 shutdown_connections 2024-03-30T04:39:54.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.444+0000 7fee89e00700 1 -- 172.21.15.123:0/1429265132 wait complete. 2024-03-30T04:39:54.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.445+0000 7fee89e00700 1 Processor -- start 2024-03-30T04:39:54.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.445+0000 7fee89e00700 1 -- start start 2024-03-30T04:39:54.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.446+0000 7fee89e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fee84102e00 0x7fee84115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:39:54.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.446+0000 7fee89e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee841037a0 0x7fee84116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:39:54.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.446+0000 7fee89e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee84116880 con 0x7fee841037a0 2024-03-30T04:39:54.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.446+0000 7fee89e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee8410fc50 con 0x7fee84102e00 2024-03-30T04:39:54.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.446+0000 7fee82000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee841037a0 0x7fee84116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:39:54.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.446+0000 7fee82000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee841037a0 0x7fee84116230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57206/0 (socket says 172.21.15.123:57206) 2024-03-30T04:39:54.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.446+0000 7fee82000700 1 -- 172.21.15.123:0/589214838 learned_addr learned my addr 172.21.15.123:0/589214838 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:39:54.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.446+0000 7fee82a00700 1 --2- 172.21.15.123:0/589214838 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fee84102e00 0x7fee84115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:39:54.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.446+0000 7fee82000700 1 -- 172.21.15.123:0/589214838 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fee84102e00 msgr2=0x7fee84115cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:54.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.446+0000 7fee82000700 1 --2- 172.21.15.123:0/589214838 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fee84102e00 0x7fee84115cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:54.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.446+0000 7fee82000700 1 -- 172.21.15.123:0/589214838 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee74005040 con 0x7fee841037a0 2024-03-30T04:39:54.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.446+0000 7fee82000700 1 --2- 172.21.15.123:0/589214838 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee841037a0 0x7fee84116230 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fee7000c890 tx=0x7fee700044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:39:54.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.447+0000 7fee7aa00700 1 -- 172.21.15.123:0/589214838 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee7000a700 con 0x7fee841037a0 2024-03-30T04:39:54.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.447+0000 7fee7aa00700 1 -- 172.21.15.123:0/589214838 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fee7000ad10 con 0x7fee841037a0 2024-03-30T04:39:54.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.447+0000 7fee89e00700 1 -- 172.21.15.123:0/589214838 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee8410ff30 con 0x7fee841037a0 2024-03-30T04:39:54.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.447+0000 7fee7aa00700 1 -- 172.21.15.123:0/589214838 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee70012e70 con 0x7fee841037a0 2024-03-30T04:39:54.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.447+0000 7fee89e00700 1 -- 172.21.15.123:0/589214838 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee841104f0 con 0x7fee841037a0 2024-03-30T04:39:54.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.448+0000 7fee89e00700 1 -- 172.21.15.123:0/589214838 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee840fe550 con 0x7fee841037a0 2024-03-30T04:39:54.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.449+0000 7fee7aa00700 1 -- 172.21.15.123:0/589214838 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fee7000a860 con 0x7fee841037a0 2024-03-30T04:39:54.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.450+0000 7fee7aa00700 1 --2- 172.21.15.123:0/589214838 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fee6806cc20 0x7fee6806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:39:54.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.450+0000 7fee7aa00700 1 -- 172.21.15.123:0/589214838 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fee7008c120 con 0x7fee841037a0 2024-03-30T04:39:54.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.450+0000 7fee82a00700 1 --2- 172.21.15.123:0/589214838 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fee6806cc20 0x7fee6806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:39:54.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.451+0000 7fee82a00700 1 --2- 172.21.15.123:0/589214838 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fee6806cc20 0x7fee6806f0d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fee74004f90 tx=0x7fee74009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:39:54.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.455+0000 7fee7aa00700 1 -- 172.21.15.123:0/589214838 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fee70059b70 con 0x7fee841037a0 2024-03-30T04:39:54.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.737+0000 7fee89e00700 1 -- 172.21.15.123:0/589214838 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fee8404bed0 con 0x7fee841037a0 2024-03-30T04:39:54.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.738+0000 7fee7aa00700 1 -- 172.21.15.123:0/589214838 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7fee70059700 con 0x7fee841037a0 2024-03-30T04:39:54.736 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:39:54.736 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:39:54.736 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:39:54.736 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:39:54.737 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:39:54.737 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:39:54.737 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:39:54.737 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:39:54.737 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T04:39:54.737 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:39:54.737 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:39:54.737 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:39:54.737 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 3 2024-03-30T04:39:54.737 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:39:54.737 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:39:54.737 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:39:54.737 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 13 2024-03-30T04:39:54.737 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:39:54.738 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:39:54.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.740+0000 7fee89e00700 1 -- 172.21.15.123:0/589214838 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fee6806cc20 msgr2=0x7fee6806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:54.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.740+0000 7fee89e00700 1 --2- 172.21.15.123:0/589214838 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fee6806cc20 0x7fee6806f0d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fee74004f90 tx=0x7fee74009e30 comp rx=0 tx=0).stop 2024-03-30T04:39:54.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.740+0000 7fee89e00700 1 -- 172.21.15.123:0/589214838 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee841037a0 msgr2=0x7fee84116230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:54.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.740+0000 7fee89e00700 1 --2- 172.21.15.123:0/589214838 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee841037a0 0x7fee84116230 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fee7000c890 tx=0x7fee700044c0 comp rx=0 tx=0).stop 2024-03-30T04:39:54.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.740+0000 7fee89e00700 1 -- 172.21.15.123:0/589214838 shutdown_connections 2024-03-30T04:39:54.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.740+0000 7fee89e00700 1 --2- 172.21.15.123:0/589214838 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fee6806cc20 0x7fee6806f0d0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:54.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.740+0000 7fee89e00700 1 --2- 172.21.15.123:0/589214838 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fee84102e00 0x7fee84115cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:54.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.740+0000 7fee89e00700 1 --2- 172.21.15.123:0/589214838 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee841037a0 0x7fee84116230 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:54.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.740+0000 7fee89e00700 1 -- 172.21.15.123:0/589214838 >> 172.21.15.123:0/589214838 conn(0x7fee840f89b0 msgr2=0x7fee840706d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:39:54.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.741+0000 7fee89e00700 1 -- 172.21.15.123:0/589214838 shutdown_connections 2024-03-30T04:39:54.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.741+0000 7fee89e00700 1 -- 172.21.15.123:0/589214838 wait complete. 2024-03-30T04:39:54.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:39:54 smithi135 ceph-mon[106011]: pgmap v261: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:54.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:39:54 smithi135 ceph-mon[106011]: from='client.34542 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:39:54.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.874+0000 7f7458400700 1 -- 172.21.15.123:0/1060109938 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7450102e00 msgr2=0x7f74501031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:54.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.874+0000 7f7458400700 1 --2- 172.21.15.123:0/1060109938 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7450102e00 0x7f74501031d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f7444004940 tx=0x7f7444009e30 comp rx=0 tx=0).stop 2024-03-30T04:39:54.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.874+0000 7f7458400700 1 -- 172.21.15.123:0/1060109938 shutdown_connections 2024-03-30T04:39:54.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.874+0000 7f7458400700 1 --2- 172.21.15.123:0/1060109938 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f74501037a0 0x7f74500fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:54.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.874+0000 7f7458400700 1 --2- 172.21.15.123:0/1060109938 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7450102e00 0x7f74501031d0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:54.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.874+0000 7f7458400700 1 -- 172.21.15.123:0/1060109938 >> 172.21.15.123:0/1060109938 conn(0x7f74500f89b0 msgr2=0x7f74500fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:39:54.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.874+0000 7f7458400700 1 -- 172.21.15.123:0/1060109938 shutdown_connections 2024-03-30T04:39:54.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.874+0000 7f7458400700 1 -- 172.21.15.123:0/1060109938 wait complete. 2024-03-30T04:39:54.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.875+0000 7f7458400700 1 Processor -- start 2024-03-30T04:39:54.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.875+0000 7f7458400700 1 -- start start 2024-03-30T04:39:54.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.875+0000 7f7458400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7450102e00 0x7f7450115cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:39:54.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.875+0000 7f7458400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74501037a0 0x7f7450116200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:39:54.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.875+0000 7f7458400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74501168e0 con 0x7f74501037a0 2024-03-30T04:39:54.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.875+0000 7f7458400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f745010fc20 con 0x7f7450102e00 2024-03-30T04:39:54.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.876+0000 7f7455000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74501037a0 0x7f7450116200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:39:54.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.876+0000 7f7455000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74501037a0 0x7f7450116200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57234/0 (socket says 172.21.15.123:57234) 2024-03-30T04:39:54.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.876+0000 7f7455000700 1 -- 172.21.15.123:0/1045008691 learned_addr learned my addr 172.21.15.123:0/1045008691 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:39:54.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.876+0000 7f7455a00700 1 --2- 172.21.15.123:0/1045008691 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7450102e00 0x7f7450115cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:39:54.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.876+0000 7f7455000700 1 -- 172.21.15.123:0/1045008691 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7450102e00 msgr2=0x7f7450115cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:54.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.876+0000 7f7455000700 1 --2- 172.21.15.123:0/1045008691 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7450102e00 0x7f7450115cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:54.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.876+0000 7f7455000700 1 -- 172.21.15.123:0/1045008691 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7444005040 con 0x7f74501037a0 2024-03-30T04:39:54.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.876+0000 7f7455a00700 1 --2- 172.21.15.123:0/1045008691 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7450102e00 0x7f7450115cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:39:54.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.876+0000 7f7455000700 1 --2- 172.21.15.123:0/1045008691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74501037a0 0x7f7450116200 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f743c00b790 tx=0x7f743c009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:39:54.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.876+0000 7f744d600700 1 -- 172.21.15.123:0/1045008691 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f743c00f6f0 con 0x7f74501037a0 2024-03-30T04:39:54.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.877+0000 7f7458400700 1 -- 172.21.15.123:0/1045008691 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f745010fea0 con 0x7f74501037a0 2024-03-30T04:39:54.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.877+0000 7f744d600700 1 -- 172.21.15.123:0/1045008691 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f743c00fd00 con 0x7f74501037a0 2024-03-30T04:39:54.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.877+0000 7f744d600700 1 -- 172.21.15.123:0/1045008691 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f743c01a400 con 0x7f74501037a0 2024-03-30T04:39:54.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.877+0000 7f7458400700 1 -- 172.21.15.123:0/1045008691 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7450110460 con 0x7f74501037a0 2024-03-30T04:39:54.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.878+0000 7f7458400700 1 -- 172.21.15.123:0/1045008691 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74500fe550 con 0x7f74501037a0 2024-03-30T04:39:54.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.879+0000 7f744d600700 1 -- 172.21.15.123:0/1045008691 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f743c00f850 con 0x7f74501037a0 2024-03-30T04:39:54.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.880+0000 7f744d600700 1 --2- 172.21.15.123:0/1045008691 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f743806cc30 0x7f743806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:39:54.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.880+0000 7f744d600700 1 -- 172.21.15.123:0/1045008691 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f743c08bee0 con 0x7f74501037a0 2024-03-30T04:39:54.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.880+0000 7f7455a00700 1 --2- 172.21.15.123:0/1045008691 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f743806cc30 0x7f743806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:39:54.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.881+0000 7f7455a00700 1 --2- 172.21.15.123:0/1045008691 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f743806cc30 0x7f743806f0e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f7444000c00 tx=0x7f7444009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:39:54.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:54.884+0000 7f744d600700 1 -- 172.21.15.123:0/1045008691 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f743c059930 con 0x7f74501037a0 2024-03-30T04:39:54.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:39:54 smithi123 ceph-mon[139369]: pgmap v261: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:54.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:39:54 smithi123 ceph-mon[139369]: from='client.34542 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:39:55.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.128+0000 7f7458400700 1 -- 172.21.15.123:0/1045008691 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f745004bed0 con 0x7f74501037a0 2024-03-30T04:39:55.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.129+0000 7f744d600700 1 -- 172.21.15.123:0/1045008691 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f743c0594c0 con 0x7f74501037a0 2024-03-30T04:39:55.127 INFO:teuthology.orchestra.run.smithi123.stdout:e32 2024-03-30T04:39:55.127 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:39:55.128 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:39:55.128 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:39:55.128 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:39:55.128 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:39:55.128 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:39:55.128 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T04:39:55.128 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T04:39:55.128 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:39:55.128 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T04:39:55.128 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:39:55.128 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:39:55.128 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:39:55.128 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:39:55.128 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:39:55.129 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T04:39:55.129 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:39:55.129 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:39:55.129 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T04:39:55.129 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:39:55.129 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T04:39:55.129 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T04:39:55.129 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T04:39:55.129 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T04:39:55.129 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:39:55.129 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:39:55.129 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:39:55.129 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:39:55.129 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:39:55.129 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:39:55.130 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T04:39:55.130 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T04:39:55.130 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:39:55.130 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:39:55.130 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T04:39:55.130 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:39:55.130 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:39:55.130 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:39:55.130 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44315} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/1741352336,v1:172.21.15.135:6827/1741352336] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:39:55.130 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:39:55.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.131+0000 7f7458400700 1 -- 172.21.15.123:0/1045008691 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f743806cc30 msgr2=0x7f743806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:55.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.131+0000 7f7458400700 1 --2- 172.21.15.123:0/1045008691 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f743806cc30 0x7f743806f0e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f7444000c00 tx=0x7f7444009e30 comp rx=0 tx=0).stop 2024-03-30T04:39:55.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.131+0000 7f7458400700 1 -- 172.21.15.123:0/1045008691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74501037a0 msgr2=0x7f7450116200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:55.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.131+0000 7f7458400700 1 --2- 172.21.15.123:0/1045008691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74501037a0 0x7f7450116200 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f743c00b790 tx=0x7f743c009220 comp rx=0 tx=0).stop 2024-03-30T04:39:55.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.131+0000 7f7458400700 1 -- 172.21.15.123:0/1045008691 shutdown_connections 2024-03-30T04:39:55.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.131+0000 7f7458400700 1 --2- 172.21.15.123:0/1045008691 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f743806cc30 0x7f743806f0e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:55.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.131+0000 7f7458400700 1 --2- 172.21.15.123:0/1045008691 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7450102e00 0x7f7450115cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:55.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.131+0000 7f7458400700 1 --2- 172.21.15.123:0/1045008691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74501037a0 0x7f7450116200 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:55.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.131+0000 7f7458400700 1 -- 172.21.15.123:0/1045008691 >> 172.21.15.123:0/1045008691 conn(0x7f74500f89b0 msgr2=0x7f7450071ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:39:55.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.131+0000 7f7458400700 1 -- 172.21.15.123:0/1045008691 shutdown_connections 2024-03-30T04:39:55.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.131+0000 7f7458400700 1 -- 172.21.15.123:0/1045008691 wait complete. 2024-03-30T04:39:55.132 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 32 2024-03-30T04:39:55.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.265+0000 7f2e70e00700 1 -- 172.21.15.123:0/3276524100 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e6c1037a0 msgr2=0x7f2e6c0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:55.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.265+0000 7f2e70e00700 1 --2- 172.21.15.123:0/3276524100 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e6c1037a0 0x7f2e6c0fce10 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f2e58004940 tx=0x7f2e58009e30 comp rx=0 tx=0).stop 2024-03-30T04:39:55.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.265+0000 7f2e70e00700 1 -- 172.21.15.123:0/3276524100 shutdown_connections 2024-03-30T04:39:55.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.265+0000 7f2e70e00700 1 --2- 172.21.15.123:0/3276524100 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e6c1037a0 0x7f2e6c0fce10 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:55.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.265+0000 7f2e70e00700 1 --2- 172.21.15.123:0/3276524100 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2e6c102e00 0x7f2e6c1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:55.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.265+0000 7f2e70e00700 1 -- 172.21.15.123:0/3276524100 >> 172.21.15.123:0/3276524100 conn(0x7f2e6c0f89b0 msgr2=0x7f2e6c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:39:55.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.265+0000 7f2e70e00700 1 -- 172.21.15.123:0/3276524100 shutdown_connections 2024-03-30T04:39:55.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.265+0000 7f2e70e00700 1 -- 172.21.15.123:0/3276524100 wait complete. 2024-03-30T04:39:55.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.266+0000 7f2e70e00700 1 Processor -- start 2024-03-30T04:39:55.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.266+0000 7f2e70e00700 1 -- start start 2024-03-30T04:39:55.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.266+0000 7f2e70e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e6c102e00 0x7f2e6c115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:39:55.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.267+0000 7f2e70e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2e6c1037a0 0x7f2e6c116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:39:55.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.267+0000 7f2e70e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6c1168f0 con 0x7f2e6c102e00 2024-03-30T04:39:55.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.267+0000 7f2e70e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6c10fc30 con 0x7f2e6c1037a0 2024-03-30T04:39:55.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.267+0000 7f2e69e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e6c102e00 0x7f2e6c115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:39:55.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.267+0000 7f2e69e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e6c102e00 0x7f2e6c115cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57256/0 (socket says 172.21.15.123:57256) 2024-03-30T04:39:55.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.267+0000 7f2e69e00700 1 -- 172.21.15.123:0/462672555 learned_addr learned my addr 172.21.15.123:0/462672555 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:39:55.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.267+0000 7f2e69e00700 1 -- 172.21.15.123:0/462672555 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2e6c1037a0 msgr2=0x7f2e6c116210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:55.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.267+0000 7f2e69e00700 1 --2- 172.21.15.123:0/462672555 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2e6c1037a0 0x7f2e6c116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:55.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.267+0000 7f2e69e00700 1 -- 172.21.15.123:0/462672555 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e58005040 con 0x7f2e6c102e00 2024-03-30T04:39:55.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.267+0000 7f2e69e00700 1 --2- 172.21.15.123:0/462672555 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e6c102e00 0x7f2e6c115cd0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c00b790 tx=0x7f2e5c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:39:55.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.268+0000 7f2e62000700 1 -- 172.21.15.123:0/462672555 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e5c00f770 con 0x7f2e6c102e00 2024-03-30T04:39:55.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.268+0000 7f2e70e00700 1 -- 172.21.15.123:0/462672555 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e6c10feb0 con 0x7f2e6c102e00 2024-03-30T04:39:55.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.268+0000 7f2e62000700 1 -- 172.21.15.123:0/462672555 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2e5c00fd80 con 0x7f2e6c102e00 2024-03-30T04:39:55.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.268+0000 7f2e62000700 1 -- 172.21.15.123:0/462672555 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e5c00d310 con 0x7f2e6c102e00 2024-03-30T04:39:55.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.268+0000 7f2e70e00700 1 -- 172.21.15.123:0/462672555 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e6c110470 con 0x7f2e6c102e00 2024-03-30T04:39:55.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.269+0000 7f2e70e00700 1 -- 172.21.15.123:0/462672555 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e6c0fe550 con 0x7f2e6c102e00 2024-03-30T04:39:55.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.270+0000 7f2e62000700 1 -- 172.21.15.123:0/462672555 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f2e5c010420 con 0x7f2e6c102e00 2024-03-30T04:39:55.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.271+0000 7f2e62000700 1 --2- 172.21.15.123:0/462672555 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2e5006cc30 0x7f2e5006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:39:55.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.271+0000 7f2e62000700 1 -- 172.21.15.123:0/462672555 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f2e5c08bf20 con 0x7f2e6c102e00 2024-03-30T04:39:55.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.271+0000 7f2e69400700 1 --2- 172.21.15.123:0/462672555 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2e5006cc30 0x7f2e5006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:39:55.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.272+0000 7f2e69400700 1 --2- 172.21.15.123:0/462672555 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2e5006cc30 0x7f2e5006f0e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2e58004f90 tx=0x7f2e58009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:39:55.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.275+0000 7f2e62000700 1 -- 172.21.15.123:0/462672555 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2e5c0bf9f0 con 0x7f2e6c102e00 2024-03-30T04:39:55.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.493+0000 7f2e70e00700 1 -- 172.21.15.123:0/462672555 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2e6c109eb0 con 0x7f2e5006cc30 2024-03-30T04:39:55.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.494+0000 7f2e62000700 1 -- 172.21.15.123:0/462672555 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f2e580096f0 con 0x7f2e5006cc30 2024-03-30T04:39:55.493 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:39:55.493 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:39:55.493 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:39:55.493 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T04:39:55.493 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:39:55.493 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T04:39:55.493 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T04:39:55.493 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T04:39:55.493 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T04:39:55.493 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:39:55.493 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "15/21 daemons upgraded", 2024-03-30T04:39:55.493 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T04:39:55.493 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:39:55.494 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:39:55.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.496+0000 7f2e70e00700 1 -- 172.21.15.123:0/462672555 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2e5006cc30 msgr2=0x7f2e5006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:55.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.496+0000 7f2e70e00700 1 --2- 172.21.15.123:0/462672555 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2e5006cc30 0x7f2e5006f0e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2e58004f90 tx=0x7f2e58009e30 comp rx=0 tx=0).stop 2024-03-30T04:39:55.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.496+0000 7f2e70e00700 1 -- 172.21.15.123:0/462672555 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e6c102e00 msgr2=0x7f2e6c115cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:55.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.496+0000 7f2e70e00700 1 --2- 172.21.15.123:0/462672555 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e6c102e00 0x7f2e6c115cd0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c00b790 tx=0x7f2e5c009250 comp rx=0 tx=0).stop 2024-03-30T04:39:55.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.497+0000 7f2e70e00700 1 -- 172.21.15.123:0/462672555 shutdown_connections 2024-03-30T04:39:55.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.497+0000 7f2e70e00700 1 --2- 172.21.15.123:0/462672555 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2e5006cc30 0x7f2e5006f0e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:55.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.497+0000 7f2e70e00700 1 --2- 172.21.15.123:0/462672555 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e6c102e00 0x7f2e6c115cd0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:55.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.497+0000 7f2e70e00700 1 --2- 172.21.15.123:0/462672555 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2e6c1037a0 0x7f2e6c116210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:55.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.497+0000 7f2e70e00700 1 -- 172.21.15.123:0/462672555 >> 172.21.15.123:0/462672555 conn(0x7f2e6c0f89b0 msgr2=0x7f2e6c0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:39:55.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.497+0000 7f2e70e00700 1 -- 172.21.15.123:0/462672555 shutdown_connections 2024-03-30T04:39:55.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.497+0000 7f2e70e00700 1 -- 172.21.15.123:0/462672555 wait complete. 2024-03-30T04:39:55.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.634+0000 7fc3f3200700 1 -- 172.21.15.123:0/2811152272 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3ec1037b0 msgr2=0x7fc3ec0fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:55.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.634+0000 7fc3f3200700 1 --2- 172.21.15.123:0/2811152272 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3ec1037b0 0x7fc3ec0fce20 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8004940 tx=0x7fc3d8009e30 comp rx=0 tx=0).stop 2024-03-30T04:39:55.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.634+0000 7fc3f3200700 1 -- 172.21.15.123:0/2811152272 shutdown_connections 2024-03-30T04:39:55.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.634+0000 7fc3f3200700 1 --2- 172.21.15.123:0/2811152272 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3ec1037b0 0x7fc3ec0fce20 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:55.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.634+0000 7fc3f3200700 1 --2- 172.21.15.123:0/2811152272 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc3ec102e10 0x7fc3ec1031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:55.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.634+0000 7fc3f3200700 1 -- 172.21.15.123:0/2811152272 >> 172.21.15.123:0/2811152272 conn(0x7fc3ec0f89a0 msgr2=0x7fc3ec0fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:39:55.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.634+0000 7fc3f3200700 1 -- 172.21.15.123:0/2811152272 shutdown_connections 2024-03-30T04:39:55.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.634+0000 7fc3f3200700 1 -- 172.21.15.123:0/2811152272 wait complete. 2024-03-30T04:39:55.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.635+0000 7fc3f3200700 1 Processor -- start 2024-03-30T04:39:55.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.635+0000 7fc3f3200700 1 -- start start 2024-03-30T04:39:55.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.635+0000 7fc3f3200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc3ec102e10 0x7fc3ec115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:39:55.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.635+0000 7fc3f3200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3ec1037b0 0x7fc3ec116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:39:55.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.635+0000 7fc3f3200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ec116860 con 0x7fc3ec1037b0 2024-03-30T04:39:55.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.635+0000 7fc3f3200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ec10fc30 con 0x7fc3ec102e10 2024-03-30T04:39:55.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.636+0000 7fc3eb400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3ec1037b0 0x7fc3ec116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:39:55.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.636+0000 7fc3eb400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3ec1037b0 0x7fc3ec116210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57272/0 (socket says 172.21.15.123:57272) 2024-03-30T04:39:55.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.636+0000 7fc3eb400700 1 -- 172.21.15.123:0/3581587110 learned_addr learned my addr 172.21.15.123:0/3581587110 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:39:55.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.636+0000 7fc3ebe00700 1 --2- 172.21.15.123:0/3581587110 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc3ec102e10 0x7fc3ec115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:39:55.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.636+0000 7fc3eb400700 1 -- 172.21.15.123:0/3581587110 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc3ec102e10 msgr2=0x7fc3ec115cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:55.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.636+0000 7fc3eb400700 1 --2- 172.21.15.123:0/3581587110 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc3ec102e10 0x7fc3ec115cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:55.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.636+0000 7fc3eb400700 1 -- 172.21.15.123:0/3581587110 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d8005040 con 0x7fc3ec1037b0 2024-03-30T04:39:55.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.636+0000 7fc3eb400700 1 --2- 172.21.15.123:0/3581587110 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3ec1037b0 0x7fc3ec116210 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8009390 tx=0x7fc3d8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:39:55.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.637+0000 7fc3e3e00700 1 -- 172.21.15.123:0/3581587110 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc3d800e040 con 0x7fc3ec1037b0 2024-03-30T04:39:55.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.637+0000 7fc3f3200700 1 -- 172.21.15.123:0/3581587110 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3ec10feb0 con 0x7fc3ec1037b0 2024-03-30T04:39:55.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.637+0000 7fc3e3e00700 1 -- 172.21.15.123:0/3581587110 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc3d8013420 con 0x7fc3ec1037b0 2024-03-30T04:39:55.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.637+0000 7fc3e3e00700 1 -- 172.21.15.123:0/3581587110 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc3d801c630 con 0x7fc3ec1037b0 2024-03-30T04:39:55.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.637+0000 7fc3f3200700 1 -- 172.21.15.123:0/3581587110 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3ec110470 con 0x7fc3ec1037b0 2024-03-30T04:39:55.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.638+0000 7fc3f3200700 1 -- 172.21.15.123:0/3581587110 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3ec0fe560 con 0x7fc3ec1037b0 2024-03-30T04:39:55.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.639+0000 7fc3e3e00700 1 -- 172.21.15.123:0/3581587110 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fc3d8013a00 con 0x7fc3ec1037b0 2024-03-30T04:39:55.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.640+0000 7fc3e3e00700 1 --2- 172.21.15.123:0/3581587110 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc3d006cc30 0x7fc3d006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:39:55.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.640+0000 7fc3e3e00700 1 -- 172.21.15.123:0/3581587110 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc3d808d1e0 con 0x7fc3ec1037b0 2024-03-30T04:39:55.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.640+0000 7fc3ebe00700 1 --2- 172.21.15.123:0/3581587110 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc3d006cc30 0x7fc3d006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:39:55.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.641+0000 7fc3ebe00700 1 --2- 172.21.15.123:0/3581587110 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc3d006cc30 0x7fc3d006f0e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc004f60 tx=0x7fc3dc00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:39:55.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.644+0000 7fc3e3e00700 1 -- 172.21.15.123:0/3581587110 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc3d805ac30 con 0x7fc3ec1037b0 2024-03-30T04:39:55.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:39:55 smithi135 ceph-mon[106011]: from='client.34546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:39:55.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:39:55 smithi135 ceph-mon[106011]: from='client.34550 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:39:55.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:39:55 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/589214838' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:39:55.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:39:55 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/1045008691' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:39:55.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.923+0000 7fc3f3200700 1 -- 172.21.15.123:0/3581587110 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc3ec109dd0 con 0x7fc3ec1037b0 2024-03-30T04:39:55.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.924+0000 7fc3e3e00700 1 -- 172.21.15.123:0/3581587110 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fc3d805a7c0 con 0x7fc3ec1037b0 2024-03-30T04:39:55.923 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T04:39:55.923 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T04:39:55.923 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T04:39:55.923 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T04:39:55.923 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T04:39:55.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.926+0000 7fc3f3200700 1 -- 172.21.15.123:0/3581587110 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc3d006cc30 msgr2=0x7fc3d006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:55.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.926+0000 7fc3f3200700 1 --2- 172.21.15.123:0/3581587110 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc3d006cc30 0x7fc3d006f0e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc004f60 tx=0x7fc3dc00b040 comp rx=0 tx=0).stop 2024-03-30T04:39:55.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.926+0000 7fc3f3200700 1 -- 172.21.15.123:0/3581587110 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3ec1037b0 msgr2=0x7fc3ec116210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:39:55.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.926+0000 7fc3f3200700 1 --2- 172.21.15.123:0/3581587110 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3ec1037b0 0x7fc3ec116210 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8009390 tx=0x7fc3d8009e30 comp rx=0 tx=0).stop 2024-03-30T04:39:55.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.926+0000 7fc3f3200700 1 -- 172.21.15.123:0/3581587110 shutdown_connections 2024-03-30T04:39:55.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.926+0000 7fc3f3200700 1 --2- 172.21.15.123:0/3581587110 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc3d006cc30 0x7fc3d006f0e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:55.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.926+0000 7fc3f3200700 1 --2- 172.21.15.123:0/3581587110 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc3ec102e10 0x7fc3ec115cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:55.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.926+0000 7fc3f3200700 1 --2- 172.21.15.123:0/3581587110 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3ec1037b0 0x7fc3ec116210 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:39:55.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.926+0000 7fc3f3200700 1 -- 172.21.15.123:0/3581587110 >> 172.21.15.123:0/3581587110 conn(0x7fc3ec0f89a0 msgr2=0x7fc3ec0f9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:39:55.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.926+0000 7fc3f3200700 1 -- 172.21.15.123:0/3581587110 shutdown_connections 2024-03-30T04:39:55.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:39:55.926+0000 7fc3f3200700 1 -- 172.21.15.123:0/3581587110 wait complete. 2024-03-30T04:39:55.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:39:55 smithi123 ceph-mon[139369]: from='client.34546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:39:55.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:39:55 smithi123 ceph-mon[139369]: from='client.34550 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:39:55.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:39:55 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/589214838' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:39:55.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:39:55 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/1045008691' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:39:56.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:39:56 smithi135 ceph-mon[106011]: pgmap v262: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:56.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:39:56 smithi135 ceph-mon[106011]: from='client.34562 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:39:56.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:39:56 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3581587110' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:39:56.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:39:56 smithi123 ceph-mon[139369]: pgmap v262: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:56.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:39:56 smithi123 ceph-mon[139369]: from='client.34562 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:39:56.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:39:56 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3581587110' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:39:58.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:39:58 smithi135 ceph-mon[106011]: pgmap v263: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:39:58.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:39:58 smithi123 ceph-mon[139369]: pgmap v263: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:00.422 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:40:00 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-mon-smithi123[139346]: 2024-03-30T04:39:59.999+0000 7f50e7e00700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T04:40:00.422 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:40:00 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-mon-smithi123[139346]: 2024-03-30T04:39:59.999+0000 7f50e7e00700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T04:40:00.422 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:40:00 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-mon-smithi123[139346]: 2024-03-30T04:39:59.999+0000 7f50e7e00700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2024-03-30T04:40:00.423 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:40:00 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-mon-smithi123[139346]: 2024-03-30T04:39:59.999+0000 7f50e7e00700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T04:40:00.423 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:40:00 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-mon-smithi123[139346]: 2024-03-30T04:39:59.999+0000 7f50e7e00700 -1 log_channel(cluster) log [ERR] : fs cephfs is offline because no MDS is active for it. 2024-03-30T04:40:00.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:40:00 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:40:00.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:40:00 smithi123 ceph-mon[139369]: pgmap v264: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:00.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:40:00 smithi123 ceph-mon[139369]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T04:40:00.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:40:00 smithi123 ceph-mon[139369]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T04:40:00.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:40:00 smithi123 ceph-mon[139369]: fs cephfs is degraded 2024-03-30T04:40:00.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:40:00 smithi123 ceph-mon[139369]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T04:40:00.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:40:00 smithi123 ceph-mon[139369]: fs cephfs is offline because no MDS is active for it. 2024-03-30T04:40:00.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:40:00 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:40:00.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:40:00 smithi135 ceph-mon[106011]: pgmap v264: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:00.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:40:00 smithi135 ceph-mon[106011]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T04:40:00.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:40:00 smithi135 ceph-mon[106011]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T04:40:00.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:40:00 smithi135 ceph-mon[106011]: fs cephfs is degraded 2024-03-30T04:40:00.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:40:00 smithi135 ceph-mon[106011]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T04:40:00.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:40:00 smithi135 ceph-mon[106011]: fs cephfs is offline because no MDS is active for it. 2024-03-30T04:40:02.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:40:02 smithi135 ceph-mon[106011]: pgmap v265: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:02.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:40:02 smithi123 ceph-mon[139369]: pgmap v265: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:04.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:40:04 smithi135 ceph-mon[106011]: pgmap v266: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:04.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:40:04 smithi123 ceph-mon[139369]: pgmap v266: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:06.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:40:06 smithi135 ceph-mon[106011]: pgmap v267: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:06.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:40:06 smithi123 ceph-mon[139369]: pgmap v267: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:08.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:40:08 smithi135 ceph-mon[106011]: pgmap v268: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:08.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:40:08 smithi123 ceph-mon[139369]: pgmap v268: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:10.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:40:10 smithi123 ceph-mon[139369]: pgmap v269: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:10.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:40:10 smithi135 ceph-mon[106011]: pgmap v269: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:12.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:40:12 smithi135 ceph-mon[106011]: pgmap v270: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:12.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:40:12 smithi123 ceph-mon[139369]: pgmap v270: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:14.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:40:14 smithi135 ceph-mon[106011]: pgmap v271: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:14.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:40:14 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:40:14.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:40:14 smithi123 ceph-mon[139369]: pgmap v271: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:14.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:40:14 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:40:16.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:40:16 smithi135 ceph-mon[106011]: pgmap v272: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:16.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:40:16 smithi123 ceph-mon[139369]: pgmap v272: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:18.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:40:18 smithi135 ceph-mon[106011]: pgmap v273: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:18.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:40:18 smithi123 ceph-mon[139369]: pgmap v273: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:20.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:40:20 smithi123 ceph-mon[139369]: pgmap v274: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:20.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:40:20 smithi135 ceph-mon[106011]: pgmap v274: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:22.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:40:22 smithi135 ceph-mon[106011]: pgmap v275: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:22.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:40:22 smithi123 ceph-mon[139369]: pgmap v275: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:24.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:40:24 smithi135 ceph-mon[106011]: pgmap v276: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:24.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:40:24 smithi123 ceph-mon[139369]: pgmap v276: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:26.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.076+0000 7fb18c400700 1 -- 172.21.15.123:0/2107221455 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb184102e00 msgr2=0x7fb1841031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:40:26.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.076+0000 7fb18c400700 1 --2- 172.21.15.123:0/2107221455 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb184102e00 0x7fb1841031d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb178004940 tx=0x7fb178009e30 comp rx=0 tx=0).stop 2024-03-30T04:40:26.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.076+0000 7fb18c400700 1 -- 172.21.15.123:0/2107221455 shutdown_connections 2024-03-30T04:40:26.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.076+0000 7fb18c400700 1 --2- 172.21.15.123:0/2107221455 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb1841037a0 0x7fb1840fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:26.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.076+0000 7fb18c400700 1 --2- 172.21.15.123:0/2107221455 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb184102e00 0x7fb1841031d0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:26.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.076+0000 7fb18c400700 1 -- 172.21.15.123:0/2107221455 >> 172.21.15.123:0/2107221455 conn(0x7fb1840f89b0 msgr2=0x7fb1840fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:40:26.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.076+0000 7fb18c400700 1 -- 172.21.15.123:0/2107221455 shutdown_connections 2024-03-30T04:40:26.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.076+0000 7fb18c400700 1 -- 172.21.15.123:0/2107221455 wait complete. 2024-03-30T04:40:26.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.077+0000 7fb18c400700 1 Processor -- start 2024-03-30T04:40:26.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.077+0000 7fb18c400700 1 -- start start 2024-03-30T04:40:26.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.078+0000 7fb18c400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb184102e00 0x7fb184113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:40:26.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.078+0000 7fb18c400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb1841037a0 0x7fb184114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:40:26.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.078+0000 7fb18c400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1841146b0 con 0x7fb184102e00 2024-03-30T04:40:26.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.078+0000 7fb18c400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb18410daa0 con 0x7fb1841037a0 2024-03-30T04:40:26.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.078+0000 7fb189a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb184102e00 0x7fb184113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:40:26.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.078+0000 7fb189a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb184102e00 0x7fb184113b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46104/0 (socket says 172.21.15.123:46104) 2024-03-30T04:40:26.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.078+0000 7fb189a00700 1 -- 172.21.15.123:0/1949764360 learned_addr learned my addr 172.21.15.123:0/1949764360 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:40:26.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.078+0000 7fb189000700 1 --2- 172.21.15.123:0/1949764360 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb1841037a0 0x7fb184114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:40:26.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.079+0000 7fb189a00700 1 -- 172.21.15.123:0/1949764360 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb1841037a0 msgr2=0x7fb184114060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:40:26.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.079+0000 7fb189a00700 1 --2- 172.21.15.123:0/1949764360 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb1841037a0 0x7fb184114060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:26.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.079+0000 7fb189a00700 1 -- 172.21.15.123:0/1949764360 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb178005040 con 0x7fb184102e00 2024-03-30T04:40:26.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.079+0000 7fb189a00700 1 --2- 172.21.15.123:0/1949764360 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb184102e00 0x7fb184113b20 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb178009390 tx=0x7fb178009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:40:26.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.079+0000 7fb181600700 1 -- 172.21.15.123:0/1949764360 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb17800e040 con 0x7fb184102e00 2024-03-30T04:40:26.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.079+0000 7fb181600700 1 -- 172.21.15.123:0/1949764360 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb178013420 con 0x7fb184102e00 2024-03-30T04:40:26.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.079+0000 7fb181600700 1 -- 172.21.15.123:0/1949764360 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb17801c630 con 0x7fb184102e00 2024-03-30T04:40:26.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.079+0000 7fb18c400700 1 -- 172.21.15.123:0/1949764360 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb18410dd20 con 0x7fb184102e00 2024-03-30T04:40:26.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.079+0000 7fb18c400700 1 -- 172.21.15.123:0/1949764360 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb18410e2e0 con 0x7fb184102e00 2024-03-30T04:40:26.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.080+0000 7fb18c400700 1 -- 172.21.15.123:0/1949764360 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1840fe550 con 0x7fb184102e00 2024-03-30T04:40:26.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.081+0000 7fb181600700 1 -- 172.21.15.123:0/1949764360 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fb1780139e0 con 0x7fb184102e00 2024-03-30T04:40:26.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.083+0000 7fb181600700 1 --2- 172.21.15.123:0/1949764360 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb16c06cd00 0x7fb16c06f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:40:26.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.083+0000 7fb181600700 1 -- 172.21.15.123:0/1949764360 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb17808d2f0 con 0x7fb184102e00 2024-03-30T04:40:26.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.083+0000 7fb189000700 1 --2- 172.21.15.123:0/1949764360 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb16c06cd00 0x7fb16c06f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:40:26.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.084+0000 7fb189000700 1 --2- 172.21.15.123:0/1949764360 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb16c06cd00 0x7fb16c06f1b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb18410ec60 tx=0x7fb17000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:40:26.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.088+0000 7fb181600700 1 -- 172.21.15.123:0/1949764360 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb17805adc0 con 0x7fb184102e00 2024-03-30T04:40:26.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.305+0000 7fb18c400700 1 -- 172.21.15.123:0/1949764360 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb184109e30 con 0x7fb16c06cd00 2024-03-30T04:40:26.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.306+0000 7fb181600700 1 -- 172.21.15.123:0/1949764360 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fb184109e30 con 0x7fb16c06cd00 2024-03-30T04:40:26.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.309+0000 7fb18c400700 1 -- 172.21.15.123:0/1949764360 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb16c06cd00 msgr2=0x7fb16c06f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:40:26.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.309+0000 7fb18c400700 1 --2- 172.21.15.123:0/1949764360 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb16c06cd00 0x7fb16c06f1b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb18410ec60 tx=0x7fb17000b040 comp rx=0 tx=0).stop 2024-03-30T04:40:26.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.309+0000 7fb18c400700 1 -- 172.21.15.123:0/1949764360 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb184102e00 msgr2=0x7fb184113b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:40:26.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.309+0000 7fb18c400700 1 --2- 172.21.15.123:0/1949764360 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb184102e00 0x7fb184113b20 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb178009390 tx=0x7fb178009e30 comp rx=0 tx=0).stop 2024-03-30T04:40:26.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.309+0000 7fb18c400700 1 -- 172.21.15.123:0/1949764360 shutdown_connections 2024-03-30T04:40:26.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.309+0000 7fb18c400700 1 --2- 172.21.15.123:0/1949764360 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb16c06cd00 0x7fb16c06f1b0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:26.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.309+0000 7fb18c400700 1 --2- 172.21.15.123:0/1949764360 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb184102e00 0x7fb184113b20 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:26.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.309+0000 7fb18c400700 1 --2- 172.21.15.123:0/1949764360 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb1841037a0 0x7fb184114060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:26.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.309+0000 7fb18c400700 1 -- 172.21.15.123:0/1949764360 >> 172.21.15.123:0/1949764360 conn(0x7fb1840f89b0 msgr2=0x7fb1840f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:40:26.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.309+0000 7fb18c400700 1 -- 172.21.15.123:0/1949764360 shutdown_connections 2024-03-30T04:40:26.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.309+0000 7fb18c400700 1 -- 172.21.15.123:0/1949764360 wait complete. 2024-03-30T04:40:26.322 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:40:26.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.443+0000 7f5b0ca00700 1 -- 172.21.15.123:0/893631141 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b08102c40 msgr2=0x7f5b08103010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:40:26.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.443+0000 7f5b0ca00700 1 --2- 172.21.15.123:0/893631141 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b08102c40 0x7f5b08103010 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f5af8004940 tx=0x7f5af8009e30 comp rx=0 tx=0).stop 2024-03-30T04:40:26.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.443+0000 7f5b0ca00700 1 -- 172.21.15.123:0/893631141 shutdown_connections 2024-03-30T04:40:26.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.443+0000 7f5b0ca00700 1 --2- 172.21.15.123:0/893631141 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5b081035e0 0x7f5b080fcc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:26.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.443+0000 7f5b0ca00700 1 --2- 172.21.15.123:0/893631141 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b08102c40 0x7f5b08103010 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:26.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.443+0000 7f5b0ca00700 1 -- 172.21.15.123:0/893631141 >> 172.21.15.123:0/893631141 conn(0x7f5b080f8830 msgr2=0x7f5b080fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:40:26.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.443+0000 7f5b0ca00700 1 -- 172.21.15.123:0/893631141 shutdown_connections 2024-03-30T04:40:26.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.443+0000 7f5b0ca00700 1 -- 172.21.15.123:0/893631141 wait complete. 2024-03-30T04:40:26.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.444+0000 7f5b0ca00700 1 Processor -- start 2024-03-30T04:40:26.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.444+0000 7f5b0ca00700 1 -- start start 2024-03-30T04:40:26.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.445+0000 7f5b0ca00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5b08102c40 0x7f5b08115b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:40:26.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.445+0000 7f5b0ca00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b081035e0 0x7f5b08116060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:40:26.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.445+0000 7f5b0ca00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b08116740 con 0x7f5b081035e0 2024-03-30T04:40:26.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.445+0000 7f5b0ca00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0810faa0 con 0x7f5b08102c40 2024-03-30T04:40:26.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.445+0000 7f5b05e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b081035e0 0x7f5b08116060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:40:26.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.445+0000 7f5b05e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b081035e0 0x7f5b08116060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46116/0 (socket says 172.21.15.123:46116) 2024-03-30T04:40:26.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.445+0000 7f5b05e00700 1 -- 172.21.15.123:0/1899965406 learned_addr learned my addr 172.21.15.123:0/1899965406 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:40:26.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.445+0000 7f5b06800700 1 --2- 172.21.15.123:0/1899965406 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5b08102c40 0x7f5b08115b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:40:26.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.445+0000 7f5b05e00700 1 -- 172.21.15.123:0/1899965406 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5b08102c40 msgr2=0x7f5b08115b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:40:26.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.445+0000 7f5b05e00700 1 --2- 172.21.15.123:0/1899965406 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5b08102c40 0x7f5b08115b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:26.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.445+0000 7f5b05e00700 1 -- 172.21.15.123:0/1899965406 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5af8005040 con 0x7f5b081035e0 2024-03-30T04:40:26.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.446+0000 7f5b05e00700 1 --2- 172.21.15.123:0/1899965406 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b081035e0 0x7f5b08116060 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5af400b790 tx=0x7f5af4009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:40:26.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.446+0000 7f5afea00700 1 -- 172.21.15.123:0/1899965406 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5af400f6f0 con 0x7f5b081035e0 2024-03-30T04:40:26.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.446+0000 7f5afea00700 1 -- 172.21.15.123:0/1899965406 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5af400fd00 con 0x7f5b081035e0 2024-03-30T04:40:26.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.446+0000 7f5b0ca00700 1 -- 172.21.15.123:0/1899965406 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b0810fd80 con 0x7f5b081035e0 2024-03-30T04:40:26.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.446+0000 7f5afea00700 1 -- 172.21.15.123:0/1899965406 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5af40103a0 con 0x7f5b081035e0 2024-03-30T04:40:26.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.446+0000 7f5b0ca00700 1 -- 172.21.15.123:0/1899965406 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b08110340 con 0x7f5b081035e0 2024-03-30T04:40:26.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.447+0000 7f5b0ca00700 1 -- 172.21.15.123:0/1899965406 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b080fe390 con 0x7f5b081035e0 2024-03-30T04:40:26.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.448+0000 7f5afea00700 1 -- 172.21.15.123:0/1899965406 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f5af400f850 con 0x7f5b081035e0 2024-03-30T04:40:26.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.449+0000 7f5afea00700 1 --2- 172.21.15.123:0/1899965406 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5aec06cc30 0x7f5aec06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:40:26.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.450+0000 7f5afea00700 1 -- 172.21.15.123:0/1899965406 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5af408be10 con 0x7f5b081035e0 2024-03-30T04:40:26.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.450+0000 7f5b06800700 1 --2- 172.21.15.123:0/1899965406 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5aec06cc30 0x7f5aec06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:40:26.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.450+0000 7f5b06800700 1 --2- 172.21.15.123:0/1899965406 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5aec06cc30 0x7f5aec06f0e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5af8009fd0 tx=0x7f5af8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:40:26.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.454+0000 7f5afea00700 1 -- 172.21.15.123:0/1899965406 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5af40598e0 con 0x7f5b081035e0 2024-03-30T04:40:26.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.677+0000 7f5b0ca00700 1 -- 172.21.15.123:0/1899965406 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5b08109c70 con 0x7f5aec06cc30 2024-03-30T04:40:26.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.678+0000 7f5afea00700 1 -- 172.21.15.123:0/1899965406 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f5b08109c70 con 0x7f5aec06cc30 2024-03-30T04:40:26.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.681+0000 7f5b0ca00700 1 -- 172.21.15.123:0/1899965406 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5aec06cc30 msgr2=0x7f5aec06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:40:26.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.681+0000 7f5b0ca00700 1 --2- 172.21.15.123:0/1899965406 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5aec06cc30 0x7f5aec06f0e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5af8009fd0 tx=0x7f5af8009e30 comp rx=0 tx=0).stop 2024-03-30T04:40:26.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.681+0000 7f5b0ca00700 1 -- 172.21.15.123:0/1899965406 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b081035e0 msgr2=0x7f5b08116060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:40:26.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.681+0000 7f5b0ca00700 1 --2- 172.21.15.123:0/1899965406 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b081035e0 0x7f5b08116060 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5af400b790 tx=0x7f5af4009220 comp rx=0 tx=0).stop 2024-03-30T04:40:26.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.681+0000 7f5b0ca00700 1 -- 172.21.15.123:0/1899965406 shutdown_connections 2024-03-30T04:40:26.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.681+0000 7f5b0ca00700 1 --2- 172.21.15.123:0/1899965406 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5aec06cc30 0x7f5aec06f0e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:26.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.681+0000 7f5b0ca00700 1 --2- 172.21.15.123:0/1899965406 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5b08102c40 0x7f5b08115b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:26.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.681+0000 7f5b0ca00700 1 --2- 172.21.15.123:0/1899965406 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b081035e0 0x7f5b08116060 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:26.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.681+0000 7f5b0ca00700 1 -- 172.21.15.123:0/1899965406 >> 172.21.15.123:0/1899965406 conn(0x7f5b080f8830 msgr2=0x7f5b080f97d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:40:26.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.681+0000 7f5b0ca00700 1 -- 172.21.15.123:0/1899965406 shutdown_connections 2024-03-30T04:40:26.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.681+0000 7f5b0ca00700 1 -- 172.21.15.123:0/1899965406 wait complete. 2024-03-30T04:40:26.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.815+0000 7f6924a00700 1 -- 172.21.15.123:0/1743372526 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6920105010 msgr2=0x7f69201053e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:40:26.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.815+0000 7f6924a00700 1 --2- 172.21.15.123:0/1743372526 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6920105010 0x7f69201053e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f6910004940 tx=0x7f6910009e30 comp rx=0 tx=0).stop 2024-03-30T04:40:26.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.815+0000 7f6924a00700 1 -- 172.21.15.123:0/1743372526 shutdown_connections 2024-03-30T04:40:26.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.815+0000 7f6924a00700 1 --2- 172.21.15.123:0/1743372526 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f69201059b0 0x7f69200ff020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:26.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.815+0000 7f6924a00700 1 --2- 172.21.15.123:0/1743372526 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6920105010 0x7f69201053e0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:26.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.815+0000 7f6924a00700 1 -- 172.21.15.123:0/1743372526 >> 172.21.15.123:0/1743372526 conn(0x7f69200faba0 msgr2=0x7f69200fcfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:40:26.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.815+0000 7f6924a00700 1 -- 172.21.15.123:0/1743372526 shutdown_connections 2024-03-30T04:40:26.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.815+0000 7f6924a00700 1 -- 172.21.15.123:0/1743372526 wait complete. 2024-03-30T04:40:26.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.816+0000 7f6924a00700 1 Processor -- start 2024-03-30T04:40:26.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.816+0000 7f6924a00700 1 -- start start 2024-03-30T04:40:26.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.816+0000 7f6924a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6920105010 0x7f6920115cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:40:26.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.816+0000 7f6924a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f69201059b0 0x7f6920116200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:40:26.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.816+0000 7f6924a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6920116850 con 0x7f6920105010 2024-03-30T04:40:26.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.816+0000 7f6924a00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f692010fc20 con 0x7f69201059b0 2024-03-30T04:40:26.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.817+0000 7f691de00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6920105010 0x7f6920115cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:40:26.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.817+0000 7f691de00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6920105010 0x7f6920115cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46130/0 (socket says 172.21.15.123:46130) 2024-03-30T04:40:26.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.817+0000 7f691de00700 1 -- 172.21.15.123:0/4133823754 learned_addr learned my addr 172.21.15.123:0/4133823754 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:40:26.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.817+0000 7f691d400700 1 --2- 172.21.15.123:0/4133823754 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f69201059b0 0x7f6920116200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:40:26.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.817+0000 7f691de00700 1 -- 172.21.15.123:0/4133823754 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f69201059b0 msgr2=0x7f6920116200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:40:26.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.817+0000 7f691de00700 1 --2- 172.21.15.123:0/4133823754 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f69201059b0 0x7f6920116200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:26.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.817+0000 7f691de00700 1 -- 172.21.15.123:0/4133823754 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6910005040 con 0x7f6920105010 2024-03-30T04:40:26.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.817+0000 7f691d400700 1 --2- 172.21.15.123:0/4133823754 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f69201059b0 0x7f6920116200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:40:26.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.817+0000 7f691de00700 1 --2- 172.21.15.123:0/4133823754 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6920105010 0x7f6920115cc0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f6910009360 tx=0x7f6910009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:40:26.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.817+0000 7f6916000700 1 -- 172.21.15.123:0/4133823754 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f691000e040 con 0x7f6920105010 2024-03-30T04:40:26.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.817+0000 7f6916000700 1 -- 172.21.15.123:0/4133823754 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6910013420 con 0x7f6920105010 2024-03-30T04:40:26.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.817+0000 7f6924a00700 1 -- 172.21.15.123:0/4133823754 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f692010fea0 con 0x7f6920105010 2024-03-30T04:40:26.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.817+0000 7f6916000700 1 -- 172.21.15.123:0/4133823754 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f691001c5f0 con 0x7f6920105010 2024-03-30T04:40:26.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.818+0000 7f6924a00700 1 -- 172.21.15.123:0/4133823754 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6920110460 con 0x7f6920105010 2024-03-30T04:40:26.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.819+0000 7f6924a00700 1 -- 172.21.15.123:0/4133823754 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6920100760 con 0x7f6920105010 2024-03-30T04:40:26.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.819+0000 7f6916000700 1 -- 172.21.15.123:0/4133823754 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f6910013590 con 0x7f6920105010 2024-03-30T04:40:26.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.820+0000 7f6916000700 1 --2- 172.21.15.123:0/4133823754 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f690406cc30 0x7f690406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:40:26.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.820+0000 7f6916000700 1 -- 172.21.15.123:0/4133823754 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f691008d340 con 0x7f6920105010 2024-03-30T04:40:26.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.821+0000 7f691d400700 1 --2- 172.21.15.123:0/4133823754 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f690406cc30 0x7f690406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:40:26.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.821+0000 7f691d400700 1 --2- 172.21.15.123:0/4133823754 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f690406cc30 0x7f690406f0e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f690c009fd0 tx=0x7f690c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:40:26.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:26.825+0000 7f6916000700 1 -- 172.21.15.123:0/4133823754 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f691005ad90 con 0x7f6920105010 2024-03-30T04:40:26.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:40:26 smithi135 ceph-mon[106011]: pgmap v277: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:26.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:40:26 smithi123 ceph-mon[139369]: pgmap v277: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:27.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.037+0000 7f6924a00700 1 -- 172.21.15.123:0/4133823754 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f692004bed0 con 0x7f690406cc30 2024-03-30T04:40:27.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.047+0000 7f6916000700 1 -- 172.21.15.123:0/4133823754 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f692004bed0 con 0x7f690406cc30 2024-03-30T04:40:27.045 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:40:27.045 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (9m) 2m ago 22m 15.0M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:40:27.045 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (6m) 2m ago 22m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T04:40:27.046 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (6m) 2m ago 21m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T04:40:27.046 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (9m) 2m ago 21m 74.0M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T04:40:27.046 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (2m) 2m ago 15m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T04:40:27.046 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (2m) 2m ago 15m 10.9M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T04:40:27.046 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (15m) 2m ago 15m 10.7M - 17.2.7-797-g5c024335 f77085f6aab7 0dc41326818c 2024-03-30T04:40:27.046 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (2m) 2m ago 15m 9747k - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T04:40:27.046 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (11m) 2m ago 23m 506M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:40:27.046 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (10m) 2m ago 21m 450M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:40:27.046 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (8m) 2m ago 23m 53.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T04:40:27.046 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (7m) 2m ago 20m 42.6M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T04:40:27.046 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (10m) 2m ago 22m 8052k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:40:27.046 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (10m) 2m ago 21m 7880k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:40:27.046 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (6m) 2m ago 20m 104M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T04:40:27.047 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (5m) 2m ago 19m 93.4M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T04:40:27.047 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (5m) 2m ago 18m 86.3M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T04:40:27.047 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (4m) 2m ago 18m 97.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T04:40:27.047 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (3m) 2m ago 17m 97.9M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T04:40:27.047 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (3m) 2m ago 17m 85.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T04:40:27.047 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (9m) 2m ago 21m 53.7M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:40:27.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.049+0000 7f6924a00700 1 -- 172.21.15.123:0/4133823754 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f690406cc30 msgr2=0x7f690406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:40:27.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.049+0000 7f6924a00700 1 --2- 172.21.15.123:0/4133823754 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f690406cc30 0x7f690406f0e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f690c009fd0 tx=0x7f690c00b040 comp rx=0 tx=0).stop 2024-03-30T04:40:27.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.049+0000 7f6924a00700 1 -- 172.21.15.123:0/4133823754 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6920105010 msgr2=0x7f6920115cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:40:27.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.049+0000 7f6924a00700 1 --2- 172.21.15.123:0/4133823754 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6920105010 0x7f6920115cc0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f6910009360 tx=0x7f6910009e30 comp rx=0 tx=0).stop 2024-03-30T04:40:27.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.049+0000 7f6924a00700 1 -- 172.21.15.123:0/4133823754 shutdown_connections 2024-03-30T04:40:27.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.049+0000 7f6924a00700 1 --2- 172.21.15.123:0/4133823754 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f690406cc30 0x7f690406f0e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:27.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.049+0000 7f6924a00700 1 --2- 172.21.15.123:0/4133823754 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6920105010 0x7f6920115cc0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:27.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.049+0000 7f6924a00700 1 --2- 172.21.15.123:0/4133823754 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f69201059b0 0x7f6920116200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:27.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.049+0000 7f6924a00700 1 -- 172.21.15.123:0/4133823754 >> 172.21.15.123:0/4133823754 conn(0x7f69200faba0 msgr2=0x7f69200fbd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:40:27.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.049+0000 7f6924a00700 1 -- 172.21.15.123:0/4133823754 shutdown_connections 2024-03-30T04:40:27.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.050+0000 7f6924a00700 1 -- 172.21.15.123:0/4133823754 wait complete. 2024-03-30T04:40:27.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.182+0000 7f281a600700 1 -- 172.21.15.123:0/2520827708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2814102e00 msgr2=0x7f28141031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:40:27.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.182+0000 7f281a600700 1 --2- 172.21.15.123:0/2520827708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2814102e00 0x7f28141031d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2804004940 tx=0x7f2804009e30 comp rx=0 tx=0).stop 2024-03-30T04:40:27.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.182+0000 7f281a600700 1 -- 172.21.15.123:0/2520827708 shutdown_connections 2024-03-30T04:40:27.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.182+0000 7f281a600700 1 --2- 172.21.15.123:0/2520827708 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f28141037a0 0x7f28140fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:27.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.182+0000 7f281a600700 1 --2- 172.21.15.123:0/2520827708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2814102e00 0x7f28141031d0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:27.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.182+0000 7f281a600700 1 -- 172.21.15.123:0/2520827708 >> 172.21.15.123:0/2520827708 conn(0x7f28140f89b0 msgr2=0x7f28140fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:40:27.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.182+0000 7f281a600700 1 -- 172.21.15.123:0/2520827708 shutdown_connections 2024-03-30T04:40:27.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.182+0000 7f281a600700 1 -- 172.21.15.123:0/2520827708 wait complete. 2024-03-30T04:40:27.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.183+0000 7f281a600700 1 Processor -- start 2024-03-30T04:40:27.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.183+0000 7f281a600700 1 -- start start 2024-03-30T04:40:27.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.184+0000 7f281a600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2814102e00 0x7f28140702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:40:27.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.184+0000 7f281a600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f28141037a0 0x7f2814070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:40:27.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.184+0000 7f281a600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2814070d90 con 0x7f2814102e00 2024-03-30T04:40:27.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.184+0000 7f281a600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2814070ed0 con 0x7f28141037a0 2024-03-30T04:40:27.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.184+0000 7f2813400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2814102e00 0x7f28140702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:40:27.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.184+0000 7f2813400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2814102e00 0x7f28140702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46160/0 (socket says 172.21.15.123:46160) 2024-03-30T04:40:27.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.184+0000 7f2813400700 1 -- 172.21.15.123:0/4225979832 learned_addr learned my addr 172.21.15.123:0/4225979832 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:40:27.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.184+0000 7f2812a00700 1 --2- 172.21.15.123:0/4225979832 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f28141037a0 0x7f2814070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:40:27.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.184+0000 7f2813400700 1 -- 172.21.15.123:0/4225979832 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f28141037a0 msgr2=0x7f2814070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:40:27.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.184+0000 7f2813400700 1 --2- 172.21.15.123:0/4225979832 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f28141037a0 0x7f2814070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:27.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.184+0000 7f2813400700 1 -- 172.21.15.123:0/4225979832 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2804005040 con 0x7f2814102e00 2024-03-30T04:40:27.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.185+0000 7f2813400700 1 --2- 172.21.15.123:0/4225979832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2814102e00 0x7f28140702c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2804009390 tx=0x7f2804009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:40:27.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.185+0000 7f280b400700 1 -- 172.21.15.123:0/4225979832 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f280400e040 con 0x7f2814102e00 2024-03-30T04:40:27.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.185+0000 7f280b400700 1 -- 172.21.15.123:0/4225979832 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2804013420 con 0x7f2814102e00 2024-03-30T04:40:27.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.185+0000 7f281a600700 1 -- 172.21.15.123:0/4225979832 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2814071120 con 0x7f2814102e00 2024-03-30T04:40:27.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.185+0000 7f280b400700 1 -- 172.21.15.123:0/4225979832 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f280401c630 con 0x7f2814102e00 2024-03-30T04:40:27.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.185+0000 7f281a600700 1 -- 172.21.15.123:0/4225979832 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f281411a250 con 0x7f2814102e00 2024-03-30T04:40:27.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.186+0000 7f281a600700 1 -- 172.21.15.123:0/4225979832 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28140fe550 con 0x7f2814102e00 2024-03-30T04:40:27.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.187+0000 7f280b400700 1 -- 172.21.15.123:0/4225979832 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f28040139e0 con 0x7f2814102e00 2024-03-30T04:40:27.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.188+0000 7f280b400700 1 --2- 172.21.15.123:0/4225979832 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f27f806cc30 0x7f27f806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:40:27.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.188+0000 7f280b400700 1 -- 172.21.15.123:0/4225979832 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f280408d260 con 0x7f2814102e00 2024-03-30T04:40:27.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.188+0000 7f2812a00700 1 --2- 172.21.15.123:0/4225979832 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f27f806cc30 0x7f27f806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:40:27.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.189+0000 7f2812a00700 1 --2- 172.21.15.123:0/4225979832 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f27f806cc30 0x7f27f806f0e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f2814071c00 tx=0x7f2800009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:40:27.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.192+0000 7f280b400700 1 -- 172.21.15.123:0/4225979832 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f28040572a0 con 0x7f2814102e00 2024-03-30T04:40:27.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.473+0000 7f281a600700 1 -- 172.21.15.123:0/4225979832 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f28140719b0 con 0x7f2814102e00 2024-03-30T04:40:27.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.473+0000 7f280b400700 1 -- 172.21.15.123:0/4225979832 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f280405a8c0 con 0x7f2814102e00 2024-03-30T04:40:27.556 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:40:27.556 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:40:27.556 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:40:27.556 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:40:27.556 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:40:27.557 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:40:27.557 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:40:27.557 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:40:27.557 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T04:40:27.557 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:40:27.557 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:40:27.557 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:40:27.557 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 3 2024-03-30T04:40:27.557 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:40:27.557 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:40:27.557 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:40:27.557 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 13 2024-03-30T04:40:27.558 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:40:27.558 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:40:27.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.476+0000 7f281a600700 1 -- 172.21.15.123:0/4225979832 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f27f806cc30 msgr2=0x7f27f806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:40:27.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.476+0000 7f281a600700 1 --2- 172.21.15.123:0/4225979832 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f27f806cc30 0x7f27f806f0e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f2814071c00 tx=0x7f2800009ee0 comp rx=0 tx=0).stop 2024-03-30T04:40:27.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.476+0000 7f281a600700 1 -- 172.21.15.123:0/4225979832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2814102e00 msgr2=0x7f28140702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:40:27.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.476+0000 7f281a600700 1 --2- 172.21.15.123:0/4225979832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2814102e00 0x7f28140702c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2804009390 tx=0x7f2804009e30 comp rx=0 tx=0).stop 2024-03-30T04:40:27.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.476+0000 7f281a600700 1 -- 172.21.15.123:0/4225979832 shutdown_connections 2024-03-30T04:40:27.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.476+0000 7f281a600700 1 --2- 172.21.15.123:0/4225979832 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f27f806cc30 0x7f27f806f0e0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:27.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.476+0000 7f281a600700 1 --2- 172.21.15.123:0/4225979832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2814102e00 0x7f28140702c0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:27.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.476+0000 7f281a600700 1 --2- 172.21.15.123:0/4225979832 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f28141037a0 0x7f2814070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:27.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.476+0000 7f281a600700 1 -- 172.21.15.123:0/4225979832 >> 172.21.15.123:0/4225979832 conn(0x7f28140f89b0 msgr2=0x7f28140f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:40:27.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.476+0000 7f281a600700 1 -- 172.21.15.123:0/4225979832 shutdown_connections 2024-03-30T04:40:27.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.476+0000 7f281a600700 1 -- 172.21.15.123:0/4225979832 wait complete. 2024-03-30T04:40:27.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.613+0000 7fadd4e00700 1 -- 172.21.15.123:0/295035305 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadd0102e00 msgr2=0x7fadd01031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:40:27.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.613+0000 7fadd4e00700 1 --2- 172.21.15.123:0/295035305 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadd0102e00 0x7fadd01031d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fadc0004940 tx=0x7fadc0009e30 comp rx=0 tx=0).stop 2024-03-30T04:40:27.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.613+0000 7fadd4e00700 1 -- 172.21.15.123:0/295035305 shutdown_connections 2024-03-30T04:40:27.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.613+0000 7fadd4e00700 1 --2- 172.21.15.123:0/295035305 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fadd01037a0 0x7fadd00fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:27.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.613+0000 7fadd4e00700 1 --2- 172.21.15.123:0/295035305 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadd0102e00 0x7fadd01031d0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:27.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.613+0000 7fadd4e00700 1 -- 172.21.15.123:0/295035305 >> 172.21.15.123:0/295035305 conn(0x7fadd00f89b0 msgr2=0x7fadd00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:40:27.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.613+0000 7fadd4e00700 1 -- 172.21.15.123:0/295035305 shutdown_connections 2024-03-30T04:40:27.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.613+0000 7fadd4e00700 1 -- 172.21.15.123:0/295035305 wait complete. 2024-03-30T04:40:27.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.614+0000 7fadd4e00700 1 Processor -- start 2024-03-30T04:40:27.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.614+0000 7fadd4e00700 1 -- start start 2024-03-30T04:40:27.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.614+0000 7fadd4e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadd0102e00 0x7fadd0113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:40:27.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.614+0000 7fadd4e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fadd01037a0 0x7fadd0114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:40:27.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.614+0000 7fadd4e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd01146b0 con 0x7fadd0102e00 2024-03-30T04:40:27.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.614+0000 7fadd4e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd010daa0 con 0x7fadd01037a0 2024-03-30T04:40:27.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.615+0000 7fadcde00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadd0102e00 0x7fadd0113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:40:27.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.615+0000 7fadcde00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadd0102e00 0x7fadd0113b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46186/0 (socket says 172.21.15.123:46186) 2024-03-30T04:40:27.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.615+0000 7fadcde00700 1 -- 172.21.15.123:0/4134261301 learned_addr learned my addr 172.21.15.123:0/4134261301 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:40:27.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.615+0000 7fadcd400700 1 --2- 172.21.15.123:0/4134261301 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fadd01037a0 0x7fadd0114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:40:27.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.615+0000 7fadcde00700 1 -- 172.21.15.123:0/4134261301 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fadd01037a0 msgr2=0x7fadd0114060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:40:27.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.615+0000 7fadcde00700 1 --2- 172.21.15.123:0/4134261301 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fadd01037a0 0x7fadd0114060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:27.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.615+0000 7fadcde00700 1 -- 172.21.15.123:0/4134261301 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadc0005040 con 0x7fadd0102e00 2024-03-30T04:40:27.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.615+0000 7fadcd400700 1 --2- 172.21.15.123:0/4134261301 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fadd01037a0 0x7fadd0114060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:40:27.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.615+0000 7fadcde00700 1 --2- 172.21.15.123:0/4134261301 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadd0102e00 0x7fadd0113b20 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fadc0009390 tx=0x7fadc0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:40:27.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.615+0000 7fadc6000700 1 -- 172.21.15.123:0/4134261301 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fadc000e040 con 0x7fadd0102e00 2024-03-30T04:40:27.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.615+0000 7fadc6000700 1 -- 172.21.15.123:0/4134261301 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fadc0013420 con 0x7fadd0102e00 2024-03-30T04:40:27.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.616+0000 7fadd4e00700 1 -- 172.21.15.123:0/4134261301 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadd010dd20 con 0x7fadd0102e00 2024-03-30T04:40:27.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.616+0000 7fadc6000700 1 -- 172.21.15.123:0/4134261301 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fadc001c630 con 0x7fadd0102e00 2024-03-30T04:40:27.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.616+0000 7fadd4e00700 1 -- 172.21.15.123:0/4134261301 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadd010e2e0 con 0x7fadd0102e00 2024-03-30T04:40:27.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.617+0000 7fadd4e00700 1 -- 172.21.15.123:0/4134261301 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadd00fe550 con 0x7fadd0102e00 2024-03-30T04:40:27.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.617+0000 7fadc6000700 1 -- 172.21.15.123:0/4134261301 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fadc00139e0 con 0x7fadd0102e00 2024-03-30T04:40:27.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.618+0000 7fadc6000700 1 --2- 172.21.15.123:0/4134261301 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fadb406cc30 0x7fadb406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:40:27.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.618+0000 7fadc6000700 1 -- 172.21.15.123:0/4134261301 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fadc008d220 con 0x7fadd0102e00 2024-03-30T04:40:27.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.619+0000 7fadcd400700 1 --2- 172.21.15.123:0/4134261301 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fadb406cc30 0x7fadb406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:40:27.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.619+0000 7fadcd400700 1 --2- 172.21.15.123:0/4134261301 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fadb406cc30 0x7fadb406f0e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fadbc009fd0 tx=0x7fadbc00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:40:27.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.623+0000 7fadc6000700 1 -- 172.21.15.123:0/4134261301 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fadc005ae60 con 0x7fadd0102e00 2024-03-30T04:40:27.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:40:27 smithi135 ceph-mon[106011]: from='client.34570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:40:27.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:40:27 smithi135 ceph-mon[106011]: from='client.34574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:40:27.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:40:27 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/4225979832' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:40:27.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.865+0000 7fadd4e00700 1 -- 172.21.15.123:0/4134261301 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fadd004bed0 con 0x7fadd0102e00 2024-03-30T04:40:27.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.866+0000 7fadc6000700 1 -- 172.21.15.123:0/4134261301 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1843 (secure 0 0 0) 0x7fadc005a9f0 con 0x7fadd0102e00 2024-03-30T04:40:27.864 INFO:teuthology.orchestra.run.smithi123.stdout:e32 2024-03-30T04:40:27.864 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:40:27.864 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:40:27.865 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:40:27.865 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:40:27.865 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:40:27.865 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:40:27.865 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T04:40:27.865 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T04:40:27.865 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:40:27.865 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T04:40:27.865 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:40:27.865 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:40:27.865 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:40:27.865 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:40:27.866 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:40:27.866 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T04:40:27.866 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:40:27.866 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:40:27.866 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T04:40:27.866 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:40:27.866 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T04:40:27.866 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T04:40:27.866 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T04:40:27.866 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T04:40:27.866 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:40:27.866 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:40:27.866 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:40:27.867 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:40:27.867 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:40:27.867 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:40:27.867 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T04:40:27.867 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T04:40:27.867 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:40:27.867 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:40:27.867 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T04:40:27.867 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:40:27.867 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:40:27.867 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:40:27.867 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44315} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/1741352336,v1:172.21.15.135:6827/1741352336] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:40:27.867 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:40:27.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.868+0000 7fadd4e00700 1 -- 172.21.15.123:0/4134261301 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fadb406cc30 msgr2=0x7fadb406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:40:27.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.868+0000 7fadd4e00700 1 --2- 172.21.15.123:0/4134261301 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fadb406cc30 0x7fadb406f0e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fadbc009fd0 tx=0x7fadbc00b040 comp rx=0 tx=0).stop 2024-03-30T04:40:27.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.868+0000 7fadd4e00700 1 -- 172.21.15.123:0/4134261301 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadd0102e00 msgr2=0x7fadd0113b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:40:27.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.868+0000 7fadd4e00700 1 --2- 172.21.15.123:0/4134261301 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadd0102e00 0x7fadd0113b20 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fadc0009390 tx=0x7fadc0009e30 comp rx=0 tx=0).stop 2024-03-30T04:40:27.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.868+0000 7fadd4e00700 1 -- 172.21.15.123:0/4134261301 shutdown_connections 2024-03-30T04:40:27.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.868+0000 7fadd4e00700 1 --2- 172.21.15.123:0/4134261301 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fadb406cc30 0x7fadb406f0e0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:27.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.868+0000 7fadd4e00700 1 --2- 172.21.15.123:0/4134261301 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadd0102e00 0x7fadd0113b20 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:27.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.868+0000 7fadd4e00700 1 --2- 172.21.15.123:0/4134261301 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fadd01037a0 0x7fadd0114060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:27.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.868+0000 7fadd4e00700 1 -- 172.21.15.123:0/4134261301 >> 172.21.15.123:0/4134261301 conn(0x7fadd00f89b0 msgr2=0x7fadd00f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:40:27.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.868+0000 7fadd4e00700 1 -- 172.21.15.123:0/4134261301 shutdown_connections 2024-03-30T04:40:27.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:27.868+0000 7fadd4e00700 1 -- 172.21.15.123:0/4134261301 wait complete. 2024-03-30T04:40:27.869 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 32 2024-03-30T04:40:27.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:40:27 smithi123 ceph-mon[139369]: from='client.34570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:40:27.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:40:27 smithi123 ceph-mon[139369]: from='client.34574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:40:27.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:40:27 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/4225979832' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:40:28.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.003+0000 7fdb02200700 1 -- 172.21.15.123:0/1694152224 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdafc0fe830 msgr2=0x7fdafc106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:40:28.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.003+0000 7fdb02200700 1 --2- 172.21.15.123:0/1694152224 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdafc0fe830 0x7fdafc106f00 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fdae8004940 tx=0x7fdae8009e30 comp rx=0 tx=0).stop 2024-03-30T04:40:28.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.003+0000 7fdb02200700 1 -- 172.21.15.123:0/1694152224 shutdown_connections 2024-03-30T04:40:28.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.003+0000 7fdb02200700 1 --2- 172.21.15.123:0/1694152224 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdafc0fe830 0x7fdafc106f00 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:28.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.003+0000 7fdb02200700 1 --2- 172.21.15.123:0/1694152224 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdafc0fde90 0x7fdafc0fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:28.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.003+0000 7fdb02200700 1 -- 172.21.15.123:0/1694152224 >> 172.21.15.123:0/1694152224 conn(0x7fdafc072bc0 msgr2=0x7fdafc072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:40:28.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.004+0000 7fdb02200700 1 -- 172.21.15.123:0/1694152224 shutdown_connections 2024-03-30T04:40:28.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.004+0000 7fdb02200700 1 -- 172.21.15.123:0/1694152224 wait complete. 2024-03-30T04:40:28.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.004+0000 7fdb02200700 1 Processor -- start 2024-03-30T04:40:28.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.004+0000 7fdb02200700 1 -- start start 2024-03-30T04:40:28.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.005+0000 7fdb02200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdafc0fde90 0x7fdafc115d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:40:28.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.005+0000 7fdb02200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdafc0fe830 0x7fdafc116270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:40:28.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.005+0000 7fdb02200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdafc1168c0 con 0x7fdafc0fde90 2024-03-30T04:40:28.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.005+0000 7fdb02200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdafc10fc90 con 0x7fdafc0fe830 2024-03-30T04:40:28.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.005+0000 7fdafb400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdafc0fde90 0x7fdafc115d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:40:28.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.005+0000 7fdafb400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdafc0fde90 0x7fdafc115d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46208/0 (socket says 172.21.15.123:46208) 2024-03-30T04:40:28.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.005+0000 7fdafb400700 1 -- 172.21.15.123:0/3712724945 learned_addr learned my addr 172.21.15.123:0/3712724945 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:40:28.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.005+0000 7fdafb400700 1 -- 172.21.15.123:0/3712724945 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdafc0fe830 msgr2=0x7fdafc116270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:40:28.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.005+0000 7fdafb400700 1 --2- 172.21.15.123:0/3712724945 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdafc0fe830 0x7fdafc116270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:28.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.005+0000 7fdafb400700 1 -- 172.21.15.123:0/3712724945 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdae8005040 con 0x7fdafc0fde90 2024-03-30T04:40:28.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.006+0000 7fdafb400700 1 --2- 172.21.15.123:0/3712724945 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdafc0fde90 0x7fdafc115d30 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fdaec002a50 tx=0x7fdaec0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:40:28.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.006+0000 7fdaf3400700 1 -- 172.21.15.123:0/3712724945 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdaec007cc0 con 0x7fdafc0fde90 2024-03-30T04:40:28.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.006+0000 7fdaf3400700 1 -- 172.21.15.123:0/3712724945 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdaec00f040 con 0x7fdafc0fde90 2024-03-30T04:40:28.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.006+0000 7fdb02200700 1 -- 172.21.15.123:0/3712724945 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdafc10ff70 con 0x7fdafc0fde90 2024-03-30T04:40:28.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.006+0000 7fdaf3400700 1 -- 172.21.15.123:0/3712724945 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdaec009540 con 0x7fdafc0fde90 2024-03-30T04:40:28.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.006+0000 7fdb02200700 1 -- 172.21.15.123:0/3712724945 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdafc1104b0 con 0x7fdafc0fde90 2024-03-30T04:40:28.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.007+0000 7fdb02200700 1 -- 172.21.15.123:0/3712724945 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdafc106a20 con 0x7fdafc0fde90 2024-03-30T04:40:28.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.008+0000 7fdaf3400700 1 -- 172.21.15.123:0/3712724945 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fdaec0096a0 con 0x7fdafc0fde90 2024-03-30T04:40:28.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.009+0000 7fdaf3400700 1 --2- 172.21.15.123:0/3712724945 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fdae006cc30 0x7fdae006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:40:28.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.009+0000 7fdaf3400700 1 -- 172.21.15.123:0/3712724945 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fdaec08bee0 con 0x7fdafc0fde90 2024-03-30T04:40:28.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.009+0000 7fdafaa00700 1 --2- 172.21.15.123:0/3712724945 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fdae006cc30 0x7fdae006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:40:28.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.010+0000 7fdafaa00700 1 --2- 172.21.15.123:0/3712724945 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fdae006cc30 0x7fdae006f0e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fdae8004f90 tx=0x7fdae8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:40:28.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.013+0000 7fdaf3400700 1 -- 172.21.15.123:0/3712724945 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fdaec0598b0 con 0x7fdafc0fde90 2024-03-30T04:40:28.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.234+0000 7fdb02200700 1 -- 172.21.15.123:0/3712724945 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdafc1113f0 con 0x7fdae006cc30 2024-03-30T04:40:28.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.236+0000 7fdaf3400700 1 -- 172.21.15.123:0/3712724945 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fdafc1113f0 con 0x7fdae006cc30 2024-03-30T04:40:28.234 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:40:28.234 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:40:28.234 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:40:28.235 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T04:40:28.235 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:40:28.235 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T04:40:28.235 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T04:40:28.235 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T04:40:28.235 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T04:40:28.235 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:40:28.235 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "15/21 daemons upgraded", 2024-03-30T04:40:28.235 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T04:40:28.236 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:40:28.236 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:40:28.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.238+0000 7fdb02200700 1 -- 172.21.15.123:0/3712724945 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fdae006cc30 msgr2=0x7fdae006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:40:28.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.238+0000 7fdb02200700 1 --2- 172.21.15.123:0/3712724945 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fdae006cc30 0x7fdae006f0e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fdae8004f90 tx=0x7fdae8009e30 comp rx=0 tx=0).stop 2024-03-30T04:40:28.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.238+0000 7fdb02200700 1 -- 172.21.15.123:0/3712724945 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdafc0fde90 msgr2=0x7fdafc115d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:40:28.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.238+0000 7fdb02200700 1 --2- 172.21.15.123:0/3712724945 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdafc0fde90 0x7fdafc115d30 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fdaec002a50 tx=0x7fdaec0046b0 comp rx=0 tx=0).stop 2024-03-30T04:40:28.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.238+0000 7fdb02200700 1 -- 172.21.15.123:0/3712724945 shutdown_connections 2024-03-30T04:40:28.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.238+0000 7fdb02200700 1 --2- 172.21.15.123:0/3712724945 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fdae006cc30 0x7fdae006f0e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:28.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.238+0000 7fdb02200700 1 --2- 172.21.15.123:0/3712724945 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdafc0fde90 0x7fdafc115d30 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:28.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.238+0000 7fdb02200700 1 --2- 172.21.15.123:0/3712724945 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdafc0fe830 0x7fdafc116270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:28.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.238+0000 7fdb02200700 1 -- 172.21.15.123:0/3712724945 >> 172.21.15.123:0/3712724945 conn(0x7fdafc072bc0 msgr2=0x7fdafc0fb840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:40:28.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.238+0000 7fdb02200700 1 -- 172.21.15.123:0/3712724945 shutdown_connections 2024-03-30T04:40:28.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.238+0000 7fdb02200700 1 -- 172.21.15.123:0/3712724945 wait complete. 2024-03-30T04:40:28.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.371+0000 7f5ce3200700 1 -- 172.21.15.123:0/2915983196 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5cdc102c40 msgr2=0x7f5cdc103010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:40:28.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.371+0000 7f5ce3200700 1 --2- 172.21.15.123:0/2915983196 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5cdc102c40 0x7f5cdc103010 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5cd0004940 tx=0x7f5cd0009e30 comp rx=0 tx=0).stop 2024-03-30T04:40:28.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.371+0000 7f5ce3200700 1 -- 172.21.15.123:0/2915983196 shutdown_connections 2024-03-30T04:40:28.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.371+0000 7f5ce3200700 1 --2- 172.21.15.123:0/2915983196 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5cdc1035e0 0x7f5cdc0fcc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:28.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.371+0000 7f5ce3200700 1 --2- 172.21.15.123:0/2915983196 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5cdc102c40 0x7f5cdc103010 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:28.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.371+0000 7f5ce3200700 1 -- 172.21.15.123:0/2915983196 >> 172.21.15.123:0/2915983196 conn(0x7f5cdc0f8830 msgr2=0x7f5cdc0fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:40:28.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.371+0000 7f5ce3200700 1 -- 172.21.15.123:0/2915983196 shutdown_connections 2024-03-30T04:40:28.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.371+0000 7f5ce3200700 1 -- 172.21.15.123:0/2915983196 wait complete. 2024-03-30T04:40:28.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.372+0000 7f5ce3200700 1 Processor -- start 2024-03-30T04:40:28.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.372+0000 7f5ce3200700 1 -- start start 2024-03-30T04:40:28.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.372+0000 7f5ce3200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5cdc102c40 0x7f5cdc113950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:40:28.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.372+0000 7f5ce3200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5cdc1035e0 0x7f5cdc113e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:40:28.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.373+0000 7f5ce3200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cdc1144e0 con 0x7f5cdc102c40 2024-03-30T04:40:28.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.373+0000 7f5ce3200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cdc10d8d0 con 0x7f5cdc1035e0 2024-03-30T04:40:28.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.373+0000 7f5ce1e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5cdc102c40 0x7f5cdc113950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:40:28.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.373+0000 7f5ce1e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5cdc102c40 0x7f5cdc113950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46222/0 (socket says 172.21.15.123:46222) 2024-03-30T04:40:28.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.373+0000 7f5ce1e00700 1 -- 172.21.15.123:0/1611489375 learned_addr learned my addr 172.21.15.123:0/1611489375 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:40:28.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.373+0000 7f5ce1400700 1 --2- 172.21.15.123:0/1611489375 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5cdc1035e0 0x7f5cdc113e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:40:28.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.373+0000 7f5ce1e00700 1 -- 172.21.15.123:0/1611489375 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5cdc1035e0 msgr2=0x7f5cdc113e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:40:28.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.373+0000 7f5ce1e00700 1 --2- 172.21.15.123:0/1611489375 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5cdc1035e0 0x7f5cdc113e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:28.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.373+0000 7f5ce1e00700 1 -- 172.21.15.123:0/1611489375 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cd0005040 con 0x7f5cdc102c40 2024-03-30T04:40:28.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.373+0000 7f5ce1e00700 1 --2- 172.21.15.123:0/1611489375 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5cdc102c40 0x7f5cdc113950 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5cd0009360 tx=0x7f5cd0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:40:28.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.374+0000 7f5cd6000700 1 -- 172.21.15.123:0/1611489375 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5cd000e040 con 0x7f5cdc102c40 2024-03-30T04:40:28.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.374+0000 7f5ce3200700 1 -- 172.21.15.123:0/1611489375 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cdc10db50 con 0x7f5cdc102c40 2024-03-30T04:40:28.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.374+0000 7f5cd6000700 1 -- 172.21.15.123:0/1611489375 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5cd0013420 con 0x7f5cdc102c40 2024-03-30T04:40:28.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.374+0000 7f5cd6000700 1 -- 172.21.15.123:0/1611489375 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5cd001c5f0 con 0x7f5cdc102c40 2024-03-30T04:40:28.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.374+0000 7f5ce3200700 1 -- 172.21.15.123:0/1611489375 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cdc10e110 con 0x7f5cdc102c40 2024-03-30T04:40:28.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.375+0000 7f5ce3200700 1 -- 172.21.15.123:0/1611489375 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cdc0fe390 con 0x7f5cdc102c40 2024-03-30T04:40:28.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.376+0000 7f5cd6000700 1 -- 172.21.15.123:0/1611489375 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f5cd0013590 con 0x7f5cdc102c40 2024-03-30T04:40:28.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.377+0000 7f5cd6000700 1 --2- 172.21.15.123:0/1611489375 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5cc406cc30 0x7f5cc406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:40:28.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.377+0000 7f5cd6000700 1 -- 172.21.15.123:0/1611489375 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5cd008d340 con 0x7f5cdc102c40 2024-03-30T04:40:28.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.377+0000 7f5ce1400700 1 --2- 172.21.15.123:0/1611489375 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5cc406cc30 0x7f5cc406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:40:28.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.377+0000 7f5ce1400700 1 --2- 172.21.15.123:0/1611489375 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5cc406cc30 0x7f5cc406f0e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc007fd0 tx=0x7f5ccc009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:40:28.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.381+0000 7f5cd6000700 1 -- 172.21.15.123:0/1611489375 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5cd005ad90 con 0x7f5cdc102c40 2024-03-30T04:40:28.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.664+0000 7f5ce3200700 1 -- 172.21.15.123:0/1611489375 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f5cdc109bf0 con 0x7f5cdc102c40 2024-03-30T04:40:28.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.664+0000 7f5cd6000700 1 -- 172.21.15.123:0/1611489375 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f5cd005a920 con 0x7f5cdc102c40 2024-03-30T04:40:28.663 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T04:40:28.663 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T04:40:28.663 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T04:40:28.663 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T04:40:28.663 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T04:40:28.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.666+0000 7f5ce3200700 1 -- 172.21.15.123:0/1611489375 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5cc406cc30 msgr2=0x7f5cc406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:40:28.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.666+0000 7f5ce3200700 1 --2- 172.21.15.123:0/1611489375 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5cc406cc30 0x7f5cc406f0e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc007fd0 tx=0x7f5ccc009040 comp rx=0 tx=0).stop 2024-03-30T04:40:28.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.666+0000 7f5ce3200700 1 -- 172.21.15.123:0/1611489375 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5cdc102c40 msgr2=0x7f5cdc113950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:40:28.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.666+0000 7f5ce3200700 1 --2- 172.21.15.123:0/1611489375 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5cdc102c40 0x7f5cdc113950 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5cd0009360 tx=0x7f5cd0009e30 comp rx=0 tx=0).stop 2024-03-30T04:40:28.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.666+0000 7f5ce3200700 1 -- 172.21.15.123:0/1611489375 shutdown_connections 2024-03-30T04:40:28.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.666+0000 7f5ce3200700 1 --2- 172.21.15.123:0/1611489375 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5cc406cc30 0x7f5cc406f0e0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:28.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.666+0000 7f5ce3200700 1 --2- 172.21.15.123:0/1611489375 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5cdc102c40 0x7f5cdc113950 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:28.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.666+0000 7f5ce3200700 1 --2- 172.21.15.123:0/1611489375 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5cdc1035e0 0x7f5cdc113e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:28.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.666+0000 7f5ce3200700 1 -- 172.21.15.123:0/1611489375 >> 172.21.15.123:0/1611489375 conn(0x7f5cdc0f8830 msgr2=0x7f5cdc0f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:40:28.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.667+0000 7f5ce3200700 1 -- 172.21.15.123:0/1611489375 shutdown_connections 2024-03-30T04:40:28.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:28.667+0000 7f5ce3200700 1 -- 172.21.15.123:0/1611489375 wait complete. 2024-03-30T04:40:28.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:40:28 smithi135 ceph-mon[106011]: from='client.34578 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:40:28.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:40:28 smithi135 ceph-mon[106011]: pgmap v278: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:28.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:40:28 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/4134261301' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:40:28.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:40:28 smithi123 ceph-mon[139369]: from='client.34578 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:40:28.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:40:28 smithi123 ceph-mon[139369]: pgmap v278: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:28.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:40:28 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/4134261301' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:40:29.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:40:29 smithi135 ceph-mon[106011]: from='client.34590 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:40:29.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:40:29 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/1611489375' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:40:29.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:40:29 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:40:29.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:40:29 smithi123 ceph-mon[139369]: from='client.34590 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:40:29.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:40:29 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/1611489375' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:40:29.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:40:29 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:40:30.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:40:30 smithi135 ceph-mon[106011]: pgmap v279: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:30.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:40:30 smithi123 ceph-mon[139369]: pgmap v279: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:32.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:40:32 smithi135 ceph-mon[106011]: pgmap v280: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:32.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:40:32 smithi123 ceph-mon[139369]: pgmap v280: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:34.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:40:34 smithi135 ceph-mon[106011]: pgmap v281: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:34.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:40:34 smithi123 ceph-mon[139369]: pgmap v281: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:36.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:40:36 smithi135 ceph-mon[106011]: pgmap v282: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:36.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:40:36 smithi123 ceph-mon[139369]: pgmap v282: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:38.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:40:38 smithi135 ceph-mon[106011]: pgmap v283: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:38.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:40:38 smithi123 ceph-mon[139369]: pgmap v283: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:40.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:40:40 smithi123 ceph-mon[139369]: pgmap v284: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:40.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:40:40 smithi135 ceph-mon[106011]: pgmap v284: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:42.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:40:42 smithi135 ceph-mon[106011]: pgmap v285: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:42.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:40:42 smithi123 ceph-mon[139369]: pgmap v285: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:44.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:40:44 smithi135 ceph-mon[106011]: pgmap v286: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:44.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:40:44 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:40:44.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:40:44 smithi123 ceph-mon[139369]: pgmap v286: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:44.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:40:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:40:46.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:40:46 smithi135 ceph-mon[106011]: pgmap v287: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:46.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:40:46 smithi123 ceph-mon[139369]: pgmap v287: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:48.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:40:48 smithi135 ceph-mon[106011]: pgmap v288: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:48.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:40:48 smithi123 ceph-mon[139369]: pgmap v288: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:50.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:40:50 smithi123 ceph-mon[139369]: pgmap v289: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:50.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:40:50 smithi135 ceph-mon[106011]: pgmap v289: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:52.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:40:52 smithi135 ceph-mon[106011]: pgmap v290: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:52.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:40:52 smithi123 ceph-mon[139369]: pgmap v290: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:54.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:40:54 smithi135 ceph-mon[106011]: pgmap v291: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:54.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:40:54 smithi123 ceph-mon[139369]: pgmap v291: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:56.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:40:56 smithi135 ceph-mon[106011]: pgmap v292: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:56.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:40:56 smithi123 ceph-mon[139369]: pgmap v292: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:58.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:58.816+0000 7fb9fcc00700 1 -- 172.21.15.123:0/1147274262 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9f8102e20 msgr2=0x7fb9f81031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:40:58.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:58.816+0000 7fb9fcc00700 1 --2- 172.21.15.123:0/1147274262 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9f8102e20 0x7fb9f81031f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fb9e8004940 tx=0x7fb9e8009e30 comp rx=0 tx=0).stop 2024-03-30T04:40:58.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:58.816+0000 7fb9fcc00700 1 -- 172.21.15.123:0/1147274262 shutdown_connections 2024-03-30T04:40:58.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:58.816+0000 7fb9fcc00700 1 --2- 172.21.15.123:0/1147274262 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb9f81037c0 0x7fb9f80fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:58.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:58.816+0000 7fb9fcc00700 1 --2- 172.21.15.123:0/1147274262 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9f8102e20 0x7fb9f81031f0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:58.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:58.816+0000 7fb9fcc00700 1 -- 172.21.15.123:0/1147274262 >> 172.21.15.123:0/1147274262 conn(0x7fb9f80f89d0 msgr2=0x7fb9f80fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:40:58.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:58.816+0000 7fb9fcc00700 1 -- 172.21.15.123:0/1147274262 shutdown_connections 2024-03-30T04:40:58.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:58.816+0000 7fb9fcc00700 1 -- 172.21.15.123:0/1147274262 wait complete. 2024-03-30T04:40:58.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:58.817+0000 7fb9fcc00700 1 Processor -- start 2024-03-30T04:40:58.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:58.817+0000 7fb9fcc00700 1 -- start start 2024-03-30T04:40:58.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:58.817+0000 7fb9fcc00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb9f8102e20 0x7fb9f8115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:40:58.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:58.817+0000 7fb9fcc00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9f81037c0 0x7fb9f8116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:40:58.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:58.817+0000 7fb9fcc00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f8116880 con 0x7fb9f81037c0 2024-03-30T04:40:58.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:58.817+0000 7fb9fcc00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f810fc50 con 0x7fb9f8102e20 2024-03-30T04:40:58.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:58.818+0000 7fb9f5400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9f81037c0 0x7fb9f8116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:40:58.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:58.818+0000 7fb9f5400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9f81037c0 0x7fb9f8116230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44672/0 (socket says 172.21.15.123:44672) 2024-03-30T04:40:58.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:58.818+0000 7fb9f5400700 1 -- 172.21.15.123:0/2352228418 learned_addr learned my addr 172.21.15.123:0/2352228418 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:40:58.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:58.818+0000 7fb9f5400700 1 -- 172.21.15.123:0/2352228418 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb9f8102e20 msgr2=0x7fb9f8115cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:40:58.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:58.818+0000 7fb9f5400700 1 --2- 172.21.15.123:0/2352228418 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb9f8102e20 0x7fb9f8115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:58.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:58.818+0000 7fb9f5400700 1 -- 172.21.15.123:0/2352228418 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9e8005040 con 0x7fb9f81037c0 2024-03-30T04:40:58.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:58.818+0000 7fb9f5400700 1 --2- 172.21.15.123:0/2352228418 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9f81037c0 0x7fb9f8116230 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb9e400c890 tx=0x7fb9e40044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:40:58.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:58.818+0000 7fb9ee000700 1 -- 172.21.15.123:0/2352228418 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9e400a7d0 con 0x7fb9f81037c0 2024-03-30T04:40:58.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:58.818+0000 7fb9fcc00700 1 -- 172.21.15.123:0/2352228418 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9f810ff30 con 0x7fb9f81037c0 2024-03-30T04:40:58.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:58.819+0000 7fb9ee000700 1 -- 172.21.15.123:0/2352228418 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb9e400ade0 con 0x7fb9f81037c0 2024-03-30T04:40:58.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:58.819+0000 7fb9ee000700 1 -- 172.21.15.123:0/2352228418 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9e40113a0 con 0x7fb9f81037c0 2024-03-30T04:40:58.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:58.819+0000 7fb9fcc00700 1 -- 172.21.15.123:0/2352228418 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9f81104f0 con 0x7fb9f81037c0 2024-03-30T04:40:58.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:58.820+0000 7fb9fcc00700 1 -- 172.21.15.123:0/2352228418 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9f80fe570 con 0x7fb9f81037c0 2024-03-30T04:40:58.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:58.821+0000 7fb9ee000700 1 -- 172.21.15.123:0/2352228418 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fb9e4012400 con 0x7fb9f81037c0 2024-03-30T04:40:58.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:58.821+0000 7fb9ee000700 1 --2- 172.21.15.123:0/2352228418 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb9dc06cc30 0x7fb9dc06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:40:58.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:58.822+0000 7fb9ee000700 1 -- 172.21.15.123:0/2352228418 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb9e408c0b0 con 0x7fb9f81037c0 2024-03-30T04:40:58.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:58.822+0000 7fb9f5e00700 1 --2- 172.21.15.123:0/2352228418 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb9dc06cc30 0x7fb9dc06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:40:58.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:58.822+0000 7fb9f5e00700 1 --2- 172.21.15.123:0/2352228418 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb9dc06cc30 0x7fb9dc06f0e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb9e8004f90 tx=0x7fb9e8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:40:58.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:58.826+0000 7fb9ee000700 1 -- 172.21.15.123:0/2352228418 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb9e4059b80 con 0x7fb9f81037c0 2024-03-30T04:40:58.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:40:58 smithi135 ceph-mon[106011]: pgmap v293: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:58.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:40:58 smithi123 ceph-mon[139369]: pgmap v293: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:40:59.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.046+0000 7fb9fcc00700 1 -- 172.21.15.123:0/2352228418 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb9f8109de0 con 0x7fb9dc06cc30 2024-03-30T04:40:59.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.047+0000 7fb9ee000700 1 -- 172.21.15.123:0/2352228418 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fb9f8109de0 con 0x7fb9dc06cc30 2024-03-30T04:40:59.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.049+0000 7fb9fcc00700 1 -- 172.21.15.123:0/2352228418 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb9dc06cc30 msgr2=0x7fb9dc06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:40:59.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.049+0000 7fb9fcc00700 1 --2- 172.21.15.123:0/2352228418 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb9dc06cc30 0x7fb9dc06f0e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb9e8004f90 tx=0x7fb9e8009e30 comp rx=0 tx=0).stop 2024-03-30T04:40:59.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.049+0000 7fb9fcc00700 1 -- 172.21.15.123:0/2352228418 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9f81037c0 msgr2=0x7fb9f8116230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:40:59.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.049+0000 7fb9fcc00700 1 --2- 172.21.15.123:0/2352228418 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9f81037c0 0x7fb9f8116230 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb9e400c890 tx=0x7fb9e40044c0 comp rx=0 tx=0).stop 2024-03-30T04:40:59.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.050+0000 7fb9fcc00700 1 -- 172.21.15.123:0/2352228418 shutdown_connections 2024-03-30T04:40:59.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.050+0000 7fb9fcc00700 1 --2- 172.21.15.123:0/2352228418 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb9dc06cc30 0x7fb9dc06f0e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:59.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.050+0000 7fb9fcc00700 1 --2- 172.21.15.123:0/2352228418 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb9f8102e20 0x7fb9f8115cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:59.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.050+0000 7fb9fcc00700 1 --2- 172.21.15.123:0/2352228418 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9f81037c0 0x7fb9f8116230 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:59.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.050+0000 7fb9fcc00700 1 -- 172.21.15.123:0/2352228418 >> 172.21.15.123:0/2352228418 conn(0x7fb9f80f89d0 msgr2=0x7fb9f80f9a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:40:59.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.050+0000 7fb9fcc00700 1 -- 172.21.15.123:0/2352228418 shutdown_connections 2024-03-30T04:40:59.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.050+0000 7fb9fcc00700 1 -- 172.21.15.123:0/2352228418 wait complete. 2024-03-30T04:40:59.063 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:40:59.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.185+0000 7fbdc6800700 1 -- 172.21.15.123:0/2892695348 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdc0071f90 msgr2=0x7fbdc0070360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:40:59.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.185+0000 7fbdc6800700 1 --2- 172.21.15.123:0/2892695348 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdc0071f90 0x7fbdc0070360 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fbdb0004970 tx=0x7fbdb0009e30 comp rx=0 tx=0).stop 2024-03-30T04:40:59.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.185+0000 7fbdc6800700 1 -- 172.21.15.123:0/2892695348 shutdown_connections 2024-03-30T04:40:59.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.185+0000 7fbdc6800700 1 --2- 172.21.15.123:0/2892695348 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbdc00708a0 0x7fbdc0070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:59.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.185+0000 7fbdc6800700 1 --2- 172.21.15.123:0/2892695348 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdc0071f90 0x7fbdc0070360 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:59.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.185+0000 7fbdc6800700 1 -- 172.21.15.123:0/2892695348 >> 172.21.15.123:0/2892695348 conn(0x7fbdc00f89b0 msgr2=0x7fbdc00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:40:59.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.185+0000 7fbdc6800700 1 -- 172.21.15.123:0/2892695348 shutdown_connections 2024-03-30T04:40:59.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.185+0000 7fbdc6800700 1 -- 172.21.15.123:0/2892695348 wait complete. 2024-03-30T04:40:59.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.186+0000 7fbdc6800700 1 Processor -- start 2024-03-30T04:40:59.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.186+0000 7fbdc6800700 1 -- start start 2024-03-30T04:40:59.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.186+0000 7fbdc6800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdc00708a0 0x7fbdc011a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:40:59.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.186+0000 7fbdc6800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbdc0071f90 0x7fbdc011a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:40:59.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.186+0000 7fbdc6800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc011ac50 con 0x7fbdc00708a0 2024-03-30T04:40:59.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.186+0000 7fbdc6800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc0114020 con 0x7fbdc0071f90 2024-03-30T04:40:59.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.187+0000 7fbdbf400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdc00708a0 0x7fbdc011a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:40:59.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.187+0000 7fbdbf400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdc00708a0 0x7fbdc011a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44694/0 (socket says 172.21.15.123:44694) 2024-03-30T04:40:59.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.187+0000 7fbdbf400700 1 -- 172.21.15.123:0/2369821080 learned_addr learned my addr 172.21.15.123:0/2369821080 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:40:59.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.187+0000 7fbdbf400700 1 -- 172.21.15.123:0/2369821080 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbdc0071f90 msgr2=0x7fbdc011a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:40:59.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.187+0000 7fbdbf400700 1 --2- 172.21.15.123:0/2369821080 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbdc0071f90 0x7fbdc011a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:59.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.187+0000 7fbdbf400700 1 -- 172.21.15.123:0/2369821080 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdb0005040 con 0x7fbdc00708a0 2024-03-30T04:40:59.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.187+0000 7fbdbf400700 1 --2- 172.21.15.123:0/2369821080 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdc00708a0 0x7fbdc011a0c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fbdb0004f50 tx=0x7fbdb0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:40:59.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.187+0000 7fbdb7400700 1 -- 172.21.15.123:0/2369821080 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbdb000e040 con 0x7fbdc00708a0 2024-03-30T04:40:59.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.187+0000 7fbdb7400700 1 -- 172.21.15.123:0/2369821080 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbdb0013420 con 0x7fbdc00708a0 2024-03-30T04:40:59.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.187+0000 7fbdc6800700 1 -- 172.21.15.123:0/2369821080 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdc01142a0 con 0x7fbdc00708a0 2024-03-30T04:40:59.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.188+0000 7fbdb7400700 1 -- 172.21.15.123:0/2369821080 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbdb001c610 con 0x7fbdc00708a0 2024-03-30T04:40:59.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.188+0000 7fbdc6800700 1 -- 172.21.15.123:0/2369821080 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdc01147e0 con 0x7fbdc00708a0 2024-03-30T04:40:59.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.189+0000 7fbdc6800700 1 -- 172.21.15.123:0/2369821080 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdc0071aa0 con 0x7fbdc00708a0 2024-03-30T04:40:59.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.190+0000 7fbdb7400700 1 -- 172.21.15.123:0/2369821080 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fbdb0013590 con 0x7fbdc00708a0 2024-03-30T04:40:59.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.190+0000 7fbdb7400700 1 --2- 172.21.15.123:0/2369821080 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbda406cc30 0x7fbda406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:40:59.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.191+0000 7fbdb7400700 1 -- 172.21.15.123:0/2369821080 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fbdb008d310 con 0x7fbdc00708a0 2024-03-30T04:40:59.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.191+0000 7fbdb7e00700 1 --2- 172.21.15.123:0/2369821080 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbda406cc30 0x7fbda406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:40:59.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.191+0000 7fbdb7e00700 1 --2- 172.21.15.123:0/2369821080 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbda406cc30 0x7fbda406f0e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fbdc0115960 tx=0x7fbda8009ce0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:40:59.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.195+0000 7fbdb7400700 1 -- 172.21.15.123:0/2369821080 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fbdb005ae10 con 0x7fbdc00708a0 2024-03-30T04:40:59.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.417+0000 7fbdc6800700 1 -- 172.21.15.123:0/2369821080 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbdc01103e0 con 0x7fbda406cc30 2024-03-30T04:40:59.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.419+0000 7fbdb7400700 1 -- 172.21.15.123:0/2369821080 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fbdc01103e0 con 0x7fbda406cc30 2024-03-30T04:40:59.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.421+0000 7fbdc6800700 1 -- 172.21.15.123:0/2369821080 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbda406cc30 msgr2=0x7fbda406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:40:59.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.421+0000 7fbdc6800700 1 --2- 172.21.15.123:0/2369821080 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbda406cc30 0x7fbda406f0e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fbdc0115960 tx=0x7fbda8009ce0 comp rx=0 tx=0).stop 2024-03-30T04:40:59.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.421+0000 7fbdc6800700 1 -- 172.21.15.123:0/2369821080 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdc00708a0 msgr2=0x7fbdc011a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:40:59.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.421+0000 7fbdc6800700 1 --2- 172.21.15.123:0/2369821080 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdc00708a0 0x7fbdc011a0c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fbdb0004f50 tx=0x7fbdb0009e30 comp rx=0 tx=0).stop 2024-03-30T04:40:59.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.421+0000 7fbdc6800700 1 -- 172.21.15.123:0/2369821080 shutdown_connections 2024-03-30T04:40:59.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.421+0000 7fbdc6800700 1 --2- 172.21.15.123:0/2369821080 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbda406cc30 0x7fbda406f0e0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:59.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.421+0000 7fbdc6800700 1 --2- 172.21.15.123:0/2369821080 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdc00708a0 0x7fbdc011a0c0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:59.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.421+0000 7fbdc6800700 1 --2- 172.21.15.123:0/2369821080 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbdc0071f90 0x7fbdc011a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:59.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.421+0000 7fbdc6800700 1 -- 172.21.15.123:0/2369821080 >> 172.21.15.123:0/2369821080 conn(0x7fbdc00f89b0 msgr2=0x7fbdc00f9b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:40:59.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.421+0000 7fbdc6800700 1 -- 172.21.15.123:0/2369821080 shutdown_connections 2024-03-30T04:40:59.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.421+0000 7fbdc6800700 1 -- 172.21.15.123:0/2369821080 wait complete. 2024-03-30T04:40:59.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.558+0000 7f7f07e00700 1 -- 172.21.15.123:0/3260157839 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f08070c70 msgr2=0x7f7f08108f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:40:59.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.558+0000 7f7f07e00700 1 --2- 172.21.15.123:0/3260157839 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f08070c70 0x7f7f08108f50 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7ef40048b0 tx=0x7f7ef4009e30 comp rx=0 tx=0).stop 2024-03-30T04:40:59.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.558+0000 7f7f07e00700 1 -- 172.21.15.123:0/3260157839 shutdown_connections 2024-03-30T04:40:59.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.558+0000 7f7f07e00700 1 --2- 172.21.15.123:0/3260157839 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f08070c70 0x7f7f08108f50 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:59.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.558+0000 7f7f07e00700 1 --2- 172.21.15.123:0/3260157839 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f08070360 0x7f7f08070730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:59.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.558+0000 7f7f07e00700 1 -- 172.21.15.123:0/3260157839 >> 172.21.15.123:0/3260157839 conn(0x7f7f080f8830 msgr2=0x7f7f080fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:40:59.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.558+0000 7f7f07e00700 1 -- 172.21.15.123:0/3260157839 shutdown_connections 2024-03-30T04:40:59.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.558+0000 7f7f07e00700 1 -- 172.21.15.123:0/3260157839 wait complete. 2024-03-30T04:40:59.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.559+0000 7f7f07e00700 1 Processor -- start 2024-03-30T04:40:59.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.559+0000 7f7f07e00700 1 -- start start 2024-03-30T04:40:59.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.560+0000 7f7f07e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f08070360 0x7f7f08115ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:40:59.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.560+0000 7f7f07e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f08070c70 0x7f7f08116000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:40:59.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.560+0000 7f7f07e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f08116650 con 0x7f7f08070360 2024-03-30T04:40:59.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.560+0000 7f7f07e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0810fa40 con 0x7f7f08070c70 2024-03-30T04:40:59.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.560+0000 7f7f06a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f08070360 0x7f7f08115ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:40:59.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.560+0000 7f7f06a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f08070360 0x7f7f08115ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44702/0 (socket says 172.21.15.123:44702) 2024-03-30T04:40:59.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.560+0000 7f7f06a00700 1 -- 172.21.15.123:0/21858012 learned_addr learned my addr 172.21.15.123:0/21858012 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:40:59.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.560+0000 7f7f06a00700 1 -- 172.21.15.123:0/21858012 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f08070c70 msgr2=0x7f7f08116000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:40:59.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.560+0000 7f7f06a00700 1 --2- 172.21.15.123:0/21858012 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f08070c70 0x7f7f08116000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:59.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.560+0000 7f7f06a00700 1 -- 172.21.15.123:0/21858012 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ef4005040 con 0x7f7f08070360 2024-03-30T04:40:59.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.560+0000 7f7f06a00700 1 --2- 172.21.15.123:0/21858012 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f08070360 0x7f7f08115ac0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f7ef80041d0 tx=0x7f7ef800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:40:59.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.561+0000 7f7efea00700 1 -- 172.21.15.123:0/21858012 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ef8007cc0 con 0x7f7f08070360 2024-03-30T04:40:59.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.561+0000 7f7efea00700 1 -- 172.21.15.123:0/21858012 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7ef800f040 con 0x7f7f08070360 2024-03-30T04:40:59.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.561+0000 7f7efea00700 1 -- 172.21.15.123:0/21858012 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ef800cae0 con 0x7f7f08070360 2024-03-30T04:40:59.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.561+0000 7f7f07e00700 1 -- 172.21.15.123:0/21858012 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f0810fcc0 con 0x7f7f08070360 2024-03-30T04:40:59.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.561+0000 7f7f07e00700 1 -- 172.21.15.123:0/21858012 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f08110200 con 0x7f7f08070360 2024-03-30T04:40:59.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.562+0000 7f7f07e00700 1 -- 172.21.15.123:0/21858012 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f08071aa0 con 0x7f7f08070360 2024-03-30T04:40:59.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.563+0000 7f7efea00700 1 -- 172.21.15.123:0/21858012 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f7ef800cc40 con 0x7f7f08070360 2024-03-30T04:40:59.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.564+0000 7f7efea00700 1 --2- 172.21.15.123:0/21858012 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7eec06cc20 0x7f7eec06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:40:59.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.564+0000 7f7efea00700 1 -- 172.21.15.123:0/21858012 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7ef808bba0 con 0x7f7f08070360 2024-03-30T04:40:59.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.564+0000 7f7f06000700 1 --2- 172.21.15.123:0/21858012 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7eec06cc20 0x7f7eec06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:40:59.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.564+0000 7f7f06000700 1 --2- 172.21.15.123:0/21858012 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7eec06cc20 0x7f7eec06f0d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7ef4004fb0 tx=0x7f7ef4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:40:59.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.568+0000 7f7efea00700 1 -- 172.21.15.123:0/21858012 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7ef80596a0 con 0x7f7f08070360 2024-03-30T04:40:59.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.781+0000 7f7f07e00700 1 -- 172.21.15.123:0/21858012 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f7f0810be60 con 0x7f7eec06cc20 2024-03-30T04:40:59.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.791+0000 7f7efea00700 1 -- 172.21.15.123:0/21858012 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f7f0810be60 con 0x7f7eec06cc20 2024-03-30T04:40:59.790 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:40:59.790 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (10m) 3m ago 22m 15.0M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:40:59.790 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (7m) 3m ago 22m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T04:40:59.790 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (7m) 2m ago 21m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T04:40:59.790 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (9m) 3m ago 22m 74.0M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T04:40:59.790 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (3m) 3m ago 16m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T04:40:59.790 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (3m) 3m ago 16m 10.9M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T04:40:59.790 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (16m) 2m ago 16m 10.7M - 17.2.7-797-g5c024335 f77085f6aab7 0dc41326818c 2024-03-30T04:40:59.790 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (2m) 2m ago 16m 9747k - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T04:40:59.790 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (12m) 3m ago 24m 506M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:40:59.790 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (11m) 2m ago 21m 450M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:40:59.790 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (8m) 3m ago 24m 53.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T04:40:59.791 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (8m) 2m ago 21m 42.6M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T04:40:59.791 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (10m) 3m ago 22m 8052k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:40:59.791 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (10m) 2m ago 21m 7880k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:40:59.791 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (6m) 3m ago 20m 104M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T04:40:59.791 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (6m) 3m ago 20m 93.4M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T04:40:59.791 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (5m) 3m ago 19m 86.3M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T04:40:59.791 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (5m) 2m ago 19m 97.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T04:40:59.791 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (4m) 2m ago 18m 97.9M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T04:40:59.791 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (3m) 2m ago 18m 85.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T04:40:59.791 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (10m) 3m ago 22m 53.7M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:40:59.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.793+0000 7f7f07e00700 1 -- 172.21.15.123:0/21858012 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7eec06cc20 msgr2=0x7f7eec06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:40:59.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.793+0000 7f7f07e00700 1 --2- 172.21.15.123:0/21858012 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7eec06cc20 0x7f7eec06f0d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7ef4004fb0 tx=0x7f7ef4009e30 comp rx=0 tx=0).stop 2024-03-30T04:40:59.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.793+0000 7f7f07e00700 1 -- 172.21.15.123:0/21858012 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f08070360 msgr2=0x7f7f08115ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:40:59.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.793+0000 7f7f07e00700 1 --2- 172.21.15.123:0/21858012 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f08070360 0x7f7f08115ac0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f7ef80041d0 tx=0x7f7ef800b040 comp rx=0 tx=0).stop 2024-03-30T04:40:59.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.794+0000 7f7f07e00700 1 -- 172.21.15.123:0/21858012 shutdown_connections 2024-03-30T04:40:59.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.794+0000 7f7f07e00700 1 --2- 172.21.15.123:0/21858012 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7eec06cc20 0x7f7eec06f0d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:59.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.794+0000 7f7f07e00700 1 --2- 172.21.15.123:0/21858012 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f08070360 0x7f7f08115ac0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:59.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.794+0000 7f7f07e00700 1 --2- 172.21.15.123:0/21858012 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f08070c70 0x7f7f08116000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:59.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.794+0000 7f7f07e00700 1 -- 172.21.15.123:0/21858012 >> 172.21.15.123:0/21858012 conn(0x7f7f080f8830 msgr2=0x7f7f080f9d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:40:59.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.794+0000 7f7f07e00700 1 -- 172.21.15.123:0/21858012 shutdown_connections 2024-03-30T04:40:59.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.794+0000 7f7f07e00700 1 -- 172.21.15.123:0/21858012 wait complete. 2024-03-30T04:40:59.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.927+0000 7fcabdc00700 1 -- 172.21.15.123:0/3725980116 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcab8102e00 msgr2=0x7fcab81031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:40:59.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.927+0000 7fcabdc00700 1 --2- 172.21.15.123:0/3725980116 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcab8102e00 0x7fcab81031d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fcaa8004940 tx=0x7fcaa8009e30 comp rx=0 tx=0).stop 2024-03-30T04:40:59.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.927+0000 7fcabdc00700 1 -- 172.21.15.123:0/3725980116 shutdown_connections 2024-03-30T04:40:59.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.927+0000 7fcabdc00700 1 --2- 172.21.15.123:0/3725980116 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcab81037a0 0x7fcab80fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:59.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.927+0000 7fcabdc00700 1 --2- 172.21.15.123:0/3725980116 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcab8102e00 0x7fcab81031d0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:59.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.927+0000 7fcabdc00700 1 -- 172.21.15.123:0/3725980116 >> 172.21.15.123:0/3725980116 conn(0x7fcab80f89b0 msgr2=0x7fcab80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:40:59.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.927+0000 7fcabdc00700 1 -- 172.21.15.123:0/3725980116 shutdown_connections 2024-03-30T04:40:59.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.927+0000 7fcabdc00700 1 -- 172.21.15.123:0/3725980116 wait complete. 2024-03-30T04:40:59.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.928+0000 7fcabdc00700 1 Processor -- start 2024-03-30T04:40:59.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.928+0000 7fcabdc00700 1 -- start start 2024-03-30T04:40:59.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.928+0000 7fcabdc00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcab8102e00 0x7fcab8113a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:40:59.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.928+0000 7fcabdc00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcab81037a0 0x7fcab8113fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:40:59.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.928+0000 7fcabdc00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcab81145f0 con 0x7fcab81037a0 2024-03-30T04:40:59.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.928+0000 7fcabdc00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcab810d9e0 con 0x7fcab8102e00 2024-03-30T04:40:59.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.929+0000 7fcab6000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcab81037a0 0x7fcab8113fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:40:59.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.929+0000 7fcab6000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcab81037a0 0x7fcab8113fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44714/0 (socket says 172.21.15.123:44714) 2024-03-30T04:40:59.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.929+0000 7fcab6000700 1 -- 172.21.15.123:0/3765032807 learned_addr learned my addr 172.21.15.123:0/3765032807 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:40:59.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.929+0000 7fcab6a00700 1 --2- 172.21.15.123:0/3765032807 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcab8102e00 0x7fcab8113a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:40:59.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.929+0000 7fcab6000700 1 -- 172.21.15.123:0/3765032807 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcab8102e00 msgr2=0x7fcab8113a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:40:59.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.929+0000 7fcab6000700 1 --2- 172.21.15.123:0/3765032807 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcab8102e00 0x7fcab8113a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:40:59.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.929+0000 7fcab6000700 1 -- 172.21.15.123:0/3765032807 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaa8005040 con 0x7fcab81037a0 2024-03-30T04:40:59.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.929+0000 7fcab6000700 1 --2- 172.21.15.123:0/3765032807 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcab81037a0 0x7fcab8113fa0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fcaa400eb40 tx=0x7fcaa400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:40:59.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.929+0000 7fcaaea00700 1 -- 172.21.15.123:0/3765032807 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcaa4007cc0 con 0x7fcab81037a0 2024-03-30T04:40:59.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.929+0000 7fcabdc00700 1 -- 172.21.15.123:0/3765032807 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcab810dc60 con 0x7fcab81037a0 2024-03-30T04:40:59.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.929+0000 7fcaaea00700 1 -- 172.21.15.123:0/3765032807 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcaa400f040 con 0x7fcab81037a0 2024-03-30T04:40:59.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.929+0000 7fcaaea00700 1 -- 172.21.15.123:0/3765032807 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcaa4009510 con 0x7fcab81037a0 2024-03-30T04:40:59.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.930+0000 7fcabdc00700 1 -- 172.21.15.123:0/3765032807 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcab810e220 con 0x7fcab81037a0 2024-03-30T04:40:59.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.931+0000 7fcabdc00700 1 -- 172.21.15.123:0/3765032807 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcab80fe550 con 0x7fcab81037a0 2024-03-30T04:40:59.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.932+0000 7fcaaea00700 1 -- 172.21.15.123:0/3765032807 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fcaa4009670 con 0x7fcab81037a0 2024-03-30T04:40:59.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.932+0000 7fcaaea00700 1 --2- 172.21.15.123:0/3765032807 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fca9c06cc30 0x7fca9c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:40:59.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.933+0000 7fcaaea00700 1 -- 172.21.15.123:0/3765032807 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fcaa408bdb0 con 0x7fcab81037a0 2024-03-30T04:40:59.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.933+0000 7fcab6a00700 1 --2- 172.21.15.123:0/3765032807 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fca9c06cc30 0x7fca9c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:40:59.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.933+0000 7fcab6a00700 1 --2- 172.21.15.123:0/3765032807 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fca9c06cc30 0x7fca9c06f0e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fcaa800b0d0 tx=0x7fcaa8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:40:59.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:40:59.937+0000 7fcaaea00700 1 -- 172.21.15.123:0/3765032807 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fcaa4059780 con 0x7fcab81037a0 2024-03-30T04:41:00.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.229+0000 7fcabdc00700 1 -- 172.21.15.123:0/3765032807 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fcab8109de0 con 0x7fcab81037a0 2024-03-30T04:41:00.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.230+0000 7fcaaea00700 1 -- 172.21.15.123:0/3765032807 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7fcaa4059310 con 0x7fcab81037a0 2024-03-30T04:41:00.228 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:41:00.228 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:41:00.229 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:41:00.229 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:41:00.229 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:41:00.229 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:41:00.229 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:41:00.229 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:41:00.229 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T04:41:00.229 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:41:00.229 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:41:00.229 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:41:00.229 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 3 2024-03-30T04:41:00.229 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:41:00.229 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:41:00.229 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:41:00.229 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 13 2024-03-30T04:41:00.229 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:41:00.230 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:41:00.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.232+0000 7fcabdc00700 1 -- 172.21.15.123:0/3765032807 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fca9c06cc30 msgr2=0x7fca9c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:41:00.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.232+0000 7fcabdc00700 1 --2- 172.21.15.123:0/3765032807 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fca9c06cc30 0x7fca9c06f0e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fcaa800b0d0 tx=0x7fcaa8009e30 comp rx=0 tx=0).stop 2024-03-30T04:41:00.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.232+0000 7fcabdc00700 1 -- 172.21.15.123:0/3765032807 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcab81037a0 msgr2=0x7fcab8113fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:41:00.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.232+0000 7fcabdc00700 1 --2- 172.21.15.123:0/3765032807 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcab81037a0 0x7fcab8113fa0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fcaa400eb40 tx=0x7fcaa400b040 comp rx=0 tx=0).stop 2024-03-30T04:41:00.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.232+0000 7fcabdc00700 1 -- 172.21.15.123:0/3765032807 shutdown_connections 2024-03-30T04:41:00.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.232+0000 7fcabdc00700 1 --2- 172.21.15.123:0/3765032807 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fca9c06cc30 0x7fca9c06f0e0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:00.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.232+0000 7fcabdc00700 1 --2- 172.21.15.123:0/3765032807 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcab8102e00 0x7fcab8113a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:00.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.232+0000 7fcabdc00700 1 --2- 172.21.15.123:0/3765032807 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcab81037a0 0x7fcab8113fa0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:00.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.232+0000 7fcabdc00700 1 -- 172.21.15.123:0/3765032807 >> 172.21.15.123:0/3765032807 conn(0x7fcab80f89b0 msgr2=0x7fcab80f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:41:00.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.233+0000 7fcabdc00700 1 -- 172.21.15.123:0/3765032807 shutdown_connections 2024-03-30T04:41:00.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.233+0000 7fcabdc00700 1 -- 172.21.15.123:0/3765032807 wait complete. 2024-03-30T04:41:00.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.365+0000 7f2125a00700 1 -- 172.21.15.123:0/2965440336 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21201037a0 msgr2=0x7f21200fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:41:00.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.365+0000 7f2125a00700 1 --2- 172.21.15.123:0/2965440336 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21201037a0 0x7f21200fce10 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f210c004940 tx=0x7f210c009e30 comp rx=0 tx=0).stop 2024-03-30T04:41:00.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.365+0000 7f2125a00700 1 -- 172.21.15.123:0/2965440336 shutdown_connections 2024-03-30T04:41:00.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.365+0000 7f2125a00700 1 --2- 172.21.15.123:0/2965440336 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21201037a0 0x7f21200fce10 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:00.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.365+0000 7f2125a00700 1 --2- 172.21.15.123:0/2965440336 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2120102e00 0x7f21201031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:00.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.365+0000 7f2125a00700 1 -- 172.21.15.123:0/2965440336 >> 172.21.15.123:0/2965440336 conn(0x7f21200f89b0 msgr2=0x7f21200fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:41:00.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.365+0000 7f2125a00700 1 -- 172.21.15.123:0/2965440336 shutdown_connections 2024-03-30T04:41:00.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.365+0000 7f2125a00700 1 -- 172.21.15.123:0/2965440336 wait complete. 2024-03-30T04:41:00.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.365+0000 7f2125a00700 1 Processor -- start 2024-03-30T04:41:00.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.366+0000 7f2125a00700 1 -- start start 2024-03-30T04:41:00.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.366+0000 7f2125a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2120102e00 0x7f21200702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:41:00.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.366+0000 7f2125a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f21201037a0 0x7f2120070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:41:00.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.366+0000 7f2125a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2120070e20 con 0x7f2120102e00 2024-03-30T04:41:00.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.366+0000 7f2125a00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2120070f60 con 0x7f21201037a0 2024-03-30T04:41:00.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.366+0000 7f211ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2120102e00 0x7f21200702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:41:00.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.366+0000 7f211ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2120102e00 0x7f21200702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44726/0 (socket says 172.21.15.123:44726) 2024-03-30T04:41:00.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.366+0000 7f211ea00700 1 -- 172.21.15.123:0/2720027753 learned_addr learned my addr 172.21.15.123:0/2720027753 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:41:00.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.366+0000 7f211e000700 1 --2- 172.21.15.123:0/2720027753 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f21201037a0 0x7f2120070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:41:00.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.367+0000 7f211ea00700 1 -- 172.21.15.123:0/2720027753 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f21201037a0 msgr2=0x7f2120070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:41:00.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.367+0000 7f211ea00700 1 --2- 172.21.15.123:0/2720027753 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f21201037a0 0x7f2120070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:00.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.367+0000 7f211ea00700 1 -- 172.21.15.123:0/2720027753 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f210c005040 con 0x7f2120102e00 2024-03-30T04:41:00.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.367+0000 7f211ea00700 1 --2- 172.21.15.123:0/2720027753 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2120102e00 0x7f21200702c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f211000b790 tx=0x7f2110009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:41:00.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.367+0000 7f2116a00700 1 -- 172.21.15.123:0/2720027753 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f211000f6f0 con 0x7f2120102e00 2024-03-30T04:41:00.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.367+0000 7f2116a00700 1 -- 172.21.15.123:0/2720027753 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f211000fd00 con 0x7f2120102e00 2024-03-30T04:41:00.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.367+0000 7f2125a00700 1 -- 172.21.15.123:0/2720027753 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2120119de0 con 0x7f2120102e00 2024-03-30T04:41:00.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.367+0000 7f2116a00700 1 -- 172.21.15.123:0/2720027753 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f21100103a0 con 0x7f2120102e00 2024-03-30T04:41:00.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.367+0000 7f2125a00700 1 -- 172.21.15.123:0/2720027753 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f212011a2f0 con 0x7f2120102e00 2024-03-30T04:41:00.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.368+0000 7f2125a00700 1 -- 172.21.15.123:0/2720027753 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21200fe550 con 0x7f2120102e00 2024-03-30T04:41:00.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.369+0000 7f2116a00700 1 -- 172.21.15.123:0/2720027753 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f211000f850 con 0x7f2120102e00 2024-03-30T04:41:00.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.370+0000 7f2116a00700 1 --2- 172.21.15.123:0/2720027753 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f210406cc30 0x7f210406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:41:00.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.370+0000 7f2116a00700 1 -- 172.21.15.123:0/2720027753 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f211008be60 con 0x7f2120102e00 2024-03-30T04:41:00.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.371+0000 7f211e000700 1 --2- 172.21.15.123:0/2720027753 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f210406cc30 0x7f210406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:41:00.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.371+0000 7f211e000700 1 --2- 172.21.15.123:0/2720027753 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f210406cc30 0x7f210406f0e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f2120071c00 tx=0x7f210c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:41:00.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.375+0000 7f2116a00700 1 -- 172.21.15.123:0/2720027753 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2110055ea0 con 0x7f2120102e00 2024-03-30T04:41:00.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.622+0000 7f2125a00700 1 -- 172.21.15.123:0/2720027753 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f21200719b0 con 0x7f2120102e00 2024-03-30T04:41:00.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.623+0000 7f2116a00700 1 -- 172.21.15.123:0/2720027753 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f21100594c0 con 0x7f2120102e00 2024-03-30T04:41:00.621 INFO:teuthology.orchestra.run.smithi123.stdout:e32 2024-03-30T04:41:00.621 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:41:00.621 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:41:00.621 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:41:00.622 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:41:00.622 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:41:00.622 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:41:00.622 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T04:41:00.622 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T04:41:00.622 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:41:00.622 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T04:41:00.622 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:41:00.622 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:41:00.622 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:41:00.622 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:41:00.622 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:41:00.622 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T04:41:00.622 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:41:00.623 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:41:00.623 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T04:41:00.623 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:41:00.623 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T04:41:00.623 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T04:41:00.623 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T04:41:00.623 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T04:41:00.623 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:41:00.623 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:41:00.623 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:41:00.623 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:41:00.623 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:41:00.623 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:41:00.623 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T04:41:00.624 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T04:41:00.624 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:41:00.624 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:41:00.624 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T04:41:00.624 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:41:00.624 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:41:00.624 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:41:00.624 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44315} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/1741352336,v1:172.21.15.135:6827/1741352336] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:41:00.624 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:41:00.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.625+0000 7f2125a00700 1 -- 172.21.15.123:0/2720027753 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f210406cc30 msgr2=0x7f210406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:41:00.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.625+0000 7f2125a00700 1 --2- 172.21.15.123:0/2720027753 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f210406cc30 0x7f210406f0e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f2120071c00 tx=0x7f210c009e30 comp rx=0 tx=0).stop 2024-03-30T04:41:00.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.625+0000 7f2125a00700 1 -- 172.21.15.123:0/2720027753 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2120102e00 msgr2=0x7f21200702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:41:00.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.625+0000 7f2125a00700 1 --2- 172.21.15.123:0/2720027753 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2120102e00 0x7f21200702c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f211000b790 tx=0x7f2110009250 comp rx=0 tx=0).stop 2024-03-30T04:41:00.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.625+0000 7f2125a00700 1 -- 172.21.15.123:0/2720027753 shutdown_connections 2024-03-30T04:41:00.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.625+0000 7f2125a00700 1 --2- 172.21.15.123:0/2720027753 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f210406cc30 0x7f210406f0e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:00.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.625+0000 7f2125a00700 1 --2- 172.21.15.123:0/2720027753 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2120102e00 0x7f21200702c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:00.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.625+0000 7f2125a00700 1 --2- 172.21.15.123:0/2720027753 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f21201037a0 0x7f2120070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:00.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.625+0000 7f2125a00700 1 -- 172.21.15.123:0/2720027753 >> 172.21.15.123:0/2720027753 conn(0x7f21200f89b0 msgr2=0x7f21200f99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:41:00.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.625+0000 7f2125a00700 1 -- 172.21.15.123:0/2720027753 shutdown_connections 2024-03-30T04:41:00.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.625+0000 7f2125a00700 1 -- 172.21.15.123:0/2720027753 wait complete. 2024-03-30T04:41:00.626 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 32 2024-03-30T04:41:00.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:41:00 smithi123 ceph-mon[139369]: from='client.34598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:41:00.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:41:00 smithi123 ceph-mon[139369]: from='client.34602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:41:00.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:41:00 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:41:00.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:41:00 smithi123 ceph-mon[139369]: pgmap v294: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:00.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:41:00 smithi123 ceph-mon[139369]: from='client.34606 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:41:00.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:41:00 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3765032807' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:41:00.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.759+0000 7ff338000700 1 -- 172.21.15.123:0/954612757 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3301037a0 msgr2=0x7ff3300fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:41:00.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.759+0000 7ff338000700 1 --2- 172.21.15.123:0/954612757 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3301037a0 0x7ff3300fce10 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ff31c004940 tx=0x7ff31c009e30 comp rx=0 tx=0).stop 2024-03-30T04:41:00.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.759+0000 7ff338000700 1 -- 172.21.15.123:0/954612757 shutdown_connections 2024-03-30T04:41:00.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.759+0000 7ff338000700 1 --2- 172.21.15.123:0/954612757 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3301037a0 0x7ff3300fce10 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:00.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.759+0000 7ff338000700 1 --2- 172.21.15.123:0/954612757 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff330102e00 0x7ff3301031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:00.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.759+0000 7ff338000700 1 -- 172.21.15.123:0/954612757 >> 172.21.15.123:0/954612757 conn(0x7ff3300f89b0 msgr2=0x7ff3300fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:41:00.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.759+0000 7ff338000700 1 -- 172.21.15.123:0/954612757 shutdown_connections 2024-03-30T04:41:00.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.759+0000 7ff338000700 1 -- 172.21.15.123:0/954612757 wait complete. 2024-03-30T04:41:00.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.760+0000 7ff338000700 1 Processor -- start 2024-03-30T04:41:00.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.761+0000 7ff338000700 1 -- start start 2024-03-30T04:41:00.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.761+0000 7ff338000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff330102e00 0x7ff330113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:41:00.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.761+0000 7ff338000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3301037a0 0x7ff330114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:41:00.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.761+0000 7ff338000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff330114690 con 0x7ff3301037a0 2024-03-30T04:41:00.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.761+0000 7ff338000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff33010da80 con 0x7ff330102e00 2024-03-30T04:41:00.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.761+0000 7ff334c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3301037a0 0x7ff330114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:41:00.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.761+0000 7ff334c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3301037a0 0x7ff330114040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44748/0 (socket says 172.21.15.123:44748) 2024-03-30T04:41:00.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.761+0000 7ff334c00700 1 -- 172.21.15.123:0/1197561429 learned_addr learned my addr 172.21.15.123:0/1197561429 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:41:00.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.761+0000 7ff335600700 1 --2- 172.21.15.123:0/1197561429 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff330102e00 0x7ff330113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:41:00.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.762+0000 7ff334c00700 1 -- 172.21.15.123:0/1197561429 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff330102e00 msgr2=0x7ff330113b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:41:00.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.762+0000 7ff334c00700 1 --2- 172.21.15.123:0/1197561429 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff330102e00 0x7ff330113b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:00.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.762+0000 7ff334c00700 1 -- 172.21.15.123:0/1197561429 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff31c005040 con 0x7ff3301037a0 2024-03-30T04:41:00.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.762+0000 7ff335600700 1 --2- 172.21.15.123:0/1197561429 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff330102e00 0x7ff330113b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:41:00.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.762+0000 7ff334c00700 1 --2- 172.21.15.123:0/1197561429 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3301037a0 0x7ff330114040 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ff31c009390 tx=0x7ff31c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:41:00.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.762+0000 7ff32d600700 1 -- 172.21.15.123:0/1197561429 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff31c00e040 con 0x7ff3301037a0 2024-03-30T04:41:00.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.762+0000 7ff32d600700 1 -- 172.21.15.123:0/1197561429 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff31c013420 con 0x7ff3301037a0 2024-03-30T04:41:00.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.762+0000 7ff32d600700 1 -- 172.21.15.123:0/1197561429 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff31c01c630 con 0x7ff3301037a0 2024-03-30T04:41:00.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.762+0000 7ff338000700 1 -- 172.21.15.123:0/1197561429 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff33010dd00 con 0x7ff3301037a0 2024-03-30T04:41:00.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.762+0000 7ff338000700 1 -- 172.21.15.123:0/1197561429 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff33010e2c0 con 0x7ff3301037a0 2024-03-30T04:41:00.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.763+0000 7ff338000700 1 -- 172.21.15.123:0/1197561429 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3300fe550 con 0x7ff3301037a0 2024-03-30T04:41:00.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.764+0000 7ff32d600700 1 -- 172.21.15.123:0/1197561429 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7ff31c013a00 con 0x7ff3301037a0 2024-03-30T04:41:00.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.765+0000 7ff32d600700 1 --2- 172.21.15.123:0/1197561429 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ff31806cc30 0x7ff31806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:41:00.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.765+0000 7ff32d600700 1 -- 172.21.15.123:0/1197561429 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff31c08d200 con 0x7ff3301037a0 2024-03-30T04:41:00.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.765+0000 7ff335600700 1 --2- 172.21.15.123:0/1197561429 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ff31806cc30 0x7ff31806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:41:00.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.766+0000 7ff335600700 1 --2- 172.21.15.123:0/1197561429 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ff31806cc30 0x7ff31806f0e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff324009fd0 tx=0x7ff324009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:41:00.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.769+0000 7ff32d600700 1 -- 172.21.15.123:0/1197561429 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff31c05ad90 con 0x7ff3301037a0 2024-03-30T04:41:00.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:41:00 smithi135 ceph-mon[106011]: from='client.34598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:41:00.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:41:00 smithi135 ceph-mon[106011]: from='client.34602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:41:00.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:41:00 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:41:00.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:41:00 smithi135 ceph-mon[106011]: pgmap v294: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:00.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:41:00 smithi135 ceph-mon[106011]: from='client.34606 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:41:00.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:41:00 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3765032807' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:41:00.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.989+0000 7ff338000700 1 -- 172.21.15.123:0/1197561429 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff330109e10 con 0x7ff31806cc30 2024-03-30T04:41:00.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.990+0000 7ff32d600700 1 -- 172.21.15.123:0/1197561429 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ff330109e10 con 0x7ff31806cc30 2024-03-30T04:41:00.989 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:41:00.989 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:41:00.989 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:41:00.989 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T04:41:00.989 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:41:00.989 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T04:41:00.989 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T04:41:00.989 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T04:41:00.989 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T04:41:00.989 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:41:00.989 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "15/21 daemons upgraded", 2024-03-30T04:41:00.990 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T04:41:00.990 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:41:00.990 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:41:00.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.993+0000 7ff338000700 1 -- 172.21.15.123:0/1197561429 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ff31806cc30 msgr2=0x7ff31806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:41:00.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.993+0000 7ff338000700 1 --2- 172.21.15.123:0/1197561429 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ff31806cc30 0x7ff31806f0e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff324009fd0 tx=0x7ff324009ee0 comp rx=0 tx=0).stop 2024-03-30T04:41:00.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.993+0000 7ff338000700 1 -- 172.21.15.123:0/1197561429 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3301037a0 msgr2=0x7ff330114040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:41:00.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.993+0000 7ff338000700 1 --2- 172.21.15.123:0/1197561429 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3301037a0 0x7ff330114040 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ff31c009390 tx=0x7ff31c009e30 comp rx=0 tx=0).stop 2024-03-30T04:41:00.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.993+0000 7ff338000700 1 -- 172.21.15.123:0/1197561429 shutdown_connections 2024-03-30T04:41:00.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.993+0000 7ff338000700 1 --2- 172.21.15.123:0/1197561429 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ff31806cc30 0x7ff31806f0e0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:00.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.993+0000 7ff338000700 1 --2- 172.21.15.123:0/1197561429 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff330102e00 0x7ff330113b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:00.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.993+0000 7ff338000700 1 --2- 172.21.15.123:0/1197561429 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3301037a0 0x7ff330114040 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:00.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.993+0000 7ff338000700 1 -- 172.21.15.123:0/1197561429 >> 172.21.15.123:0/1197561429 conn(0x7ff3300f89b0 msgr2=0x7ff3300f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:41:00.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.993+0000 7ff338000700 1 -- 172.21.15.123:0/1197561429 shutdown_connections 2024-03-30T04:41:00.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:00.993+0000 7ff338000700 1 -- 172.21.15.123:0/1197561429 wait complete. 2024-03-30T04:41:01.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:01.124+0000 7fe9a9a00700 1 -- 172.21.15.123:0/1428675076 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9a41037a0 msgr2=0x7fe9a40fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:41:01.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:01.124+0000 7fe9a9a00700 1 --2- 172.21.15.123:0/1428675076 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9a41037a0 0x7fe9a40fce10 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe990004940 tx=0x7fe990009e30 comp rx=0 tx=0).stop 2024-03-30T04:41:01.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:01.125+0000 7fe9a9a00700 1 -- 172.21.15.123:0/1428675076 shutdown_connections 2024-03-30T04:41:01.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:01.125+0000 7fe9a9a00700 1 --2- 172.21.15.123:0/1428675076 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9a41037a0 0x7fe9a40fce10 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:01.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:01.125+0000 7fe9a9a00700 1 --2- 172.21.15.123:0/1428675076 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe9a4102e00 0x7fe9a41031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:01.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:01.125+0000 7fe9a9a00700 1 -- 172.21.15.123:0/1428675076 >> 172.21.15.123:0/1428675076 conn(0x7fe9a40f89b0 msgr2=0x7fe9a40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:41:01.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:01.125+0000 7fe9a9a00700 1 -- 172.21.15.123:0/1428675076 shutdown_connections 2024-03-30T04:41:01.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:01.125+0000 7fe9a9a00700 1 -- 172.21.15.123:0/1428675076 wait complete. 2024-03-30T04:41:01.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:01.125+0000 7fe9a9a00700 1 Processor -- start 2024-03-30T04:41:01.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:01.126+0000 7fe9a9a00700 1 -- start start 2024-03-30T04:41:01.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:01.126+0000 7fe9a9a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9a4102e00 0x7fe9a4113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:41:01.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:01.126+0000 7fe9a9a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe9a41037a0 0x7fe9a4114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:41:01.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:01.126+0000 7fe9a9a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a4114690 con 0x7fe9a4102e00 2024-03-30T04:41:01.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:01.126+0000 7fe9a9a00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a410da80 con 0x7fe9a41037a0 2024-03-30T04:41:01.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:01.126+0000 7fe9a2a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9a4102e00 0x7fe9a4113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:41:01.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:01.126+0000 7fe9a2a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9a4102e00 0x7fe9a4113b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44766/0 (socket says 172.21.15.123:44766) 2024-03-30T04:41:01.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:01.126+0000 7fe9a2a00700 1 -- 172.21.15.123:0/1769003252 learned_addr learned my addr 172.21.15.123:0/1769003252 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:41:01.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:01.126+0000 7fe9a2000700 1 --2- 172.21.15.123:0/1769003252 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe9a41037a0 0x7fe9a4114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:41:01.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:01.127+0000 7fe9a2a00700 1 -- 172.21.15.123:0/1769003252 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe9a41037a0 msgr2=0x7fe9a4114040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:41:01.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:01.127+0000 7fe9a2a00700 1 --2- 172.21.15.123:0/1769003252 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe9a41037a0 0x7fe9a4114040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:01.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:01.127+0000 7fe9a2a00700 1 -- 172.21.15.123:0/1769003252 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe990005040 con 0x7fe9a4102e00 2024-03-30T04:41:01.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:01.127+0000 7fe9a2a00700 1 --2- 172.21.15.123:0/1769003252 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9a4102e00 0x7fe9a4113b00 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe994002a50 tx=0x7fe9940046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:41:01.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:01.127+0000 7fe99aa00700 1 -- 172.21.15.123:0/1769003252 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe994007cc0 con 0x7fe9a4102e00 2024-03-30T04:41:01.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:01.127+0000 7fe9a9a00700 1 -- 172.21.15.123:0/1769003252 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9a410dd60 con 0x7fe9a4102e00 2024-03-30T04:41:01.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:01.127+0000 7fe99aa00700 1 -- 172.21.15.123:0/1769003252 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe99400f040 con 0x7fe9a4102e00 2024-03-30T04:41:01.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:01.127+0000 7fe99aa00700 1 -- 172.21.15.123:0/1769003252 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe99400d9c0 con 0x7fe9a4102e00 2024-03-30T04:41:01.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:01.127+0000 7fe9a9a00700 1 -- 172.21.15.123:0/1769003252 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9a410e2f0 con 0x7fe9a4102e00 2024-03-30T04:41:01.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:01.128+0000 7fe9a9a00700 1 -- 172.21.15.123:0/1769003252 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9a40fe550 con 0x7fe9a4102e00 2024-03-30T04:41:01.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:01.129+0000 7fe99aa00700 1 -- 172.21.15.123:0/1769003252 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fe99400db20 con 0x7fe9a4102e00 2024-03-30T04:41:01.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:01.130+0000 7fe99aa00700 1 --2- 172.21.15.123:0/1769003252 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe98806cc30 0x7fe98806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:41:01.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:01.130+0000 7fe99aa00700 1 -- 172.21.15.123:0/1769003252 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe99408bde0 con 0x7fe9a4102e00 2024-03-30T04:41:01.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:01.130+0000 7fe9a2000700 1 --2- 172.21.15.123:0/1769003252 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe98806cc30 0x7fe98806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:41:01.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:01.131+0000 7fe9a2000700 1 --2- 172.21.15.123:0/1769003252 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe98806cc30 0x7fe98806f0e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe9a410ed60 tx=0x7fe990009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:41:01.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:01.134+0000 7fe99aa00700 1 -- 172.21.15.123:0/1769003252 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe9940597b0 con 0x7fe9a4102e00 2024-03-30T04:41:01.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:01.416+0000 7fe9a9a00700 1 -- 172.21.15.123:0/1769003252 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fe9a4109da0 con 0x7fe9a4102e00 2024-03-30T04:41:01.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:01.416+0000 7fe99aa00700 1 -- 172.21.15.123:0/1769003252 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fe994059340 con 0x7fe9a4102e00 2024-03-30T04:41:01.414 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T04:41:01.415 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T04:41:01.415 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T04:41:01.415 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T04:41:01.415 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T04:41:01.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:01.418+0000 7fe9a9a00700 1 -- 172.21.15.123:0/1769003252 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe98806cc30 msgr2=0x7fe98806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:41:01.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:01.418+0000 7fe9a9a00700 1 --2- 172.21.15.123:0/1769003252 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe98806cc30 0x7fe98806f0e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe9a410ed60 tx=0x7fe990009e30 comp rx=0 tx=0).stop 2024-03-30T04:41:01.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:01.418+0000 7fe9a9a00700 1 -- 172.21.15.123:0/1769003252 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9a4102e00 msgr2=0x7fe9a4113b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:41:01.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:01.418+0000 7fe9a9a00700 1 --2- 172.21.15.123:0/1769003252 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9a4102e00 0x7fe9a4113b00 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe994002a50 tx=0x7fe9940046b0 comp rx=0 tx=0).stop 2024-03-30T04:41:01.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:01.418+0000 7fe9a9a00700 1 -- 172.21.15.123:0/1769003252 shutdown_connections 2024-03-30T04:41:01.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:01.418+0000 7fe9a9a00700 1 --2- 172.21.15.123:0/1769003252 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe98806cc30 0x7fe98806f0e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:01.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:01.418+0000 7fe9a9a00700 1 --2- 172.21.15.123:0/1769003252 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9a4102e00 0x7fe9a4113b00 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:01.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:01.418+0000 7fe9a9a00700 1 --2- 172.21.15.123:0/1769003252 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe9a41037a0 0x7fe9a4114040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:01.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:01.418+0000 7fe9a9a00700 1 -- 172.21.15.123:0/1769003252 >> 172.21.15.123:0/1769003252 conn(0x7fe9a40f89b0 msgr2=0x7fe9a40f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:41:01.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:01.419+0000 7fe9a9a00700 1 -- 172.21.15.123:0/1769003252 shutdown_connections 2024-03-30T04:41:01.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:01.419+0000 7fe9a9a00700 1 -- 172.21.15.123:0/1769003252 wait complete. 2024-03-30T04:41:01.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:41:01 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2720027753' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:41:01.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:41:01 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/1769003252' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:41:01.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:41:01 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2720027753' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:41:01.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:41:01 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/1769003252' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:41:02.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:41:02 smithi123 ceph-mon[139369]: from='client.34618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:41:02.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:41:02 smithi123 ceph-mon[139369]: pgmap v295: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:02.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:41:02 smithi135 ceph-mon[106011]: from='client.34618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:41:02.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:41:02 smithi135 ceph-mon[106011]: pgmap v295: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:04.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:41:04 smithi135 ceph-mon[106011]: pgmap v296: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:04.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:41:04 smithi123 ceph-mon[139369]: pgmap v296: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:06.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:41:06 smithi135 ceph-mon[106011]: pgmap v297: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:06.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:41:06 smithi123 ceph-mon[139369]: pgmap v297: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:08.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:41:08 smithi135 ceph-mon[106011]: pgmap v298: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:08.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:41:08 smithi123 ceph-mon[139369]: pgmap v298: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:10.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:41:10 smithi123 ceph-mon[139369]: pgmap v299: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:10.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:41:10 smithi135 ceph-mon[106011]: pgmap v299: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:12.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:41:12 smithi135 ceph-mon[106011]: pgmap v300: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:12.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:41:12 smithi123 ceph-mon[139369]: pgmap v300: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:14.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:41:14 smithi135 ceph-mon[106011]: pgmap v301: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:14.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:41:14 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:41:14.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:41:14 smithi123 ceph-mon[139369]: pgmap v301: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:14.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:41:14 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:41:16.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:41:16 smithi135 ceph-mon[106011]: pgmap v302: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:16.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:41:16 smithi123 ceph-mon[139369]: pgmap v302: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:18.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:41:18 smithi135 ceph-mon[106011]: pgmap v303: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:18.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:41:18 smithi123 ceph-mon[139369]: pgmap v303: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:20.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:41:20 smithi123 ceph-mon[139369]: pgmap v304: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:20.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:41:20 smithi135 ceph-mon[106011]: pgmap v304: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:22.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:41:22 smithi135 ceph-mon[106011]: pgmap v305: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:22.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:41:22 smithi123 ceph-mon[139369]: pgmap v305: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:24.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:41:24 smithi135 ceph-mon[106011]: pgmap v306: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:24.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:41:24 smithi123 ceph-mon[139369]: pgmap v306: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:26.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:41:26 smithi135 ceph-mon[106011]: pgmap v307: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:26.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:41:26 smithi123 ceph-mon[139369]: pgmap v307: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:28.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:41:28 smithi135 ceph-mon[106011]: pgmap v308: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:28.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:41:28 smithi123 ceph-mon[139369]: pgmap v308: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:29.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:41:29 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:41:29.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:41:29 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:41:30.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:41:30 smithi135 ceph-mon[106011]: pgmap v309: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:30.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:41:30 smithi123 ceph-mon[139369]: pgmap v309: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:31.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.569+0000 7f907e000700 1 -- 172.21.15.123:0/327197679 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9078102e00 msgr2=0x7f90781031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:41:31.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.569+0000 7f907e000700 1 --2- 172.21.15.123:0/327197679 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9078102e00 0x7f90781031d0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f9068004940 tx=0x7f9068009e30 comp rx=0 tx=0).stop 2024-03-30T04:41:31.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.569+0000 7f907e000700 1 -- 172.21.15.123:0/327197679 shutdown_connections 2024-03-30T04:41:31.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.569+0000 7f907e000700 1 --2- 172.21.15.123:0/327197679 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f90781037a0 0x7f90780fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:31.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.569+0000 7f907e000700 1 --2- 172.21.15.123:0/327197679 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9078102e00 0x7f90781031d0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:31.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.569+0000 7f907e000700 1 -- 172.21.15.123:0/327197679 >> 172.21.15.123:0/327197679 conn(0x7f90780f89b0 msgr2=0x7f90780fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:41:31.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.570+0000 7f907e000700 1 -- 172.21.15.123:0/327197679 shutdown_connections 2024-03-30T04:41:31.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.570+0000 7f907e000700 1 -- 172.21.15.123:0/327197679 wait complete. 2024-03-30T04:41:31.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.571+0000 7f907e000700 1 Processor -- start 2024-03-30T04:41:31.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.571+0000 7f907e000700 1 -- start start 2024-03-30T04:41:31.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.571+0000 7f907e000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9078102e00 0x7f9078115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:41:31.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.572+0000 7f907e000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f90781037a0 0x7f9078116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:41:31.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.572+0000 7f907e000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9078116880 con 0x7f9078102e00 2024-03-30T04:41:31.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.572+0000 7f907e000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f907810fc50 con 0x7f90781037a0 2024-03-30T04:41:31.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.572+0000 7f9077400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9078102e00 0x7f9078115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:41:31.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.572+0000 7f9077400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9078102e00 0x7f9078115cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50410/0 (socket says 172.21.15.123:50410) 2024-03-30T04:41:31.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.572+0000 7f9077400700 1 -- 172.21.15.123:0/704765961 learned_addr learned my addr 172.21.15.123:0/704765961 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:41:31.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.572+0000 7f9076a00700 1 --2- 172.21.15.123:0/704765961 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f90781037a0 0x7f9078116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:41:31.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.572+0000 7f9077400700 1 -- 172.21.15.123:0/704765961 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f90781037a0 msgr2=0x7f9078116230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:41:31.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.572+0000 7f9077400700 1 --2- 172.21.15.123:0/704765961 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f90781037a0 0x7f9078116230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:31.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.572+0000 7f9077400700 1 -- 172.21.15.123:0/704765961 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9068005040 con 0x7f9078102e00 2024-03-30T04:41:31.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.572+0000 7f9077400700 1 --2- 172.21.15.123:0/704765961 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9078102e00 0x7f9078115cf0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f9068009390 tx=0x7f9068009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:41:31.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.573+0000 7f906f400700 1 -- 172.21.15.123:0/704765961 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f906800e040 con 0x7f9078102e00 2024-03-30T04:41:31.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.573+0000 7f907e000700 1 -- 172.21.15.123:0/704765961 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f907810fed0 con 0x7f9078102e00 2024-03-30T04:41:31.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.573+0000 7f906f400700 1 -- 172.21.15.123:0/704765961 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9068013420 con 0x7f9078102e00 2024-03-30T04:41:31.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.573+0000 7f906f400700 1 -- 172.21.15.123:0/704765961 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f906801c630 con 0x7f9078102e00 2024-03-30T04:41:31.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.573+0000 7f907e000700 1 -- 172.21.15.123:0/704765961 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9078110490 con 0x7f9078102e00 2024-03-30T04:41:31.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.574+0000 7f907e000700 1 -- 172.21.15.123:0/704765961 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90780fe550 con 0x7f9078102e00 2024-03-30T04:41:31.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.575+0000 7f906f400700 1 -- 172.21.15.123:0/704765961 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f90680139e0 con 0x7f9078102e00 2024-03-30T04:41:31.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.576+0000 7f906f400700 1 --2- 172.21.15.123:0/704765961 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f905c06cc30 0x7f905c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:41:31.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.576+0000 7f906f400700 1 -- 172.21.15.123:0/704765961 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f906808d320 con 0x7f9078102e00 2024-03-30T04:41:31.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.576+0000 7f9076a00700 1 --2- 172.21.15.123:0/704765961 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f905c06cc30 0x7f905c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:41:31.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.577+0000 7f9076a00700 1 --2- 172.21.15.123:0/704765961 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f905c06cc30 0x7f905c06f0e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f9078111590 tx=0x7f9064009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:41:31.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.581+0000 7f906f400700 1 -- 172.21.15.123:0/704765961 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f906805ae20 con 0x7f9078102e00 2024-03-30T04:41:31.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.802+0000 7f907e000700 1 -- 172.21.15.123:0/704765961 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f907804bf40 con 0x7f905c06cc30 2024-03-30T04:41:31.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.803+0000 7f906f400700 1 -- 172.21.15.123:0/704765961 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f907804bf40 con 0x7f905c06cc30 2024-03-30T04:41:31.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.805+0000 7f907e000700 1 -- 172.21.15.123:0/704765961 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f905c06cc30 msgr2=0x7f905c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:41:31.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.805+0000 7f907e000700 1 --2- 172.21.15.123:0/704765961 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f905c06cc30 0x7f905c06f0e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f9078111590 tx=0x7f9064009ee0 comp rx=0 tx=0).stop 2024-03-30T04:41:31.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.805+0000 7f907e000700 1 -- 172.21.15.123:0/704765961 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9078102e00 msgr2=0x7f9078115cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:41:31.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.805+0000 7f907e000700 1 --2- 172.21.15.123:0/704765961 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9078102e00 0x7f9078115cf0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f9068009390 tx=0x7f9068009e30 comp rx=0 tx=0).stop 2024-03-30T04:41:31.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.805+0000 7f907e000700 1 -- 172.21.15.123:0/704765961 shutdown_connections 2024-03-30T04:41:31.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.805+0000 7f907e000700 1 --2- 172.21.15.123:0/704765961 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f905c06cc30 0x7f905c06f0e0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:31.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.805+0000 7f907e000700 1 --2- 172.21.15.123:0/704765961 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9078102e00 0x7f9078115cf0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:31.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.805+0000 7f907e000700 1 --2- 172.21.15.123:0/704765961 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f90781037a0 0x7f9078116230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:31.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.805+0000 7f907e000700 1 -- 172.21.15.123:0/704765961 >> 172.21.15.123:0/704765961 conn(0x7f90780f89b0 msgr2=0x7f90780f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:41:31.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.805+0000 7f907e000700 1 -- 172.21.15.123:0/704765961 shutdown_connections 2024-03-30T04:41:31.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.805+0000 7f907e000700 1 -- 172.21.15.123:0/704765961 wait complete. 2024-03-30T04:41:31.819 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:41:31.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.940+0000 7fbd6be00700 1 -- 172.21.15.123:0/3725053767 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbd6c102c40 msgr2=0x7fbd6c103010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:41:31.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.940+0000 7fbd6be00700 1 --2- 172.21.15.123:0/3725053767 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbd6c102c40 0x7fbd6c103010 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c004940 tx=0x7fbd5c009e30 comp rx=0 tx=0).stop 2024-03-30T04:41:31.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.940+0000 7fbd6be00700 1 -- 172.21.15.123:0/3725053767 shutdown_connections 2024-03-30T04:41:31.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.940+0000 7fbd6be00700 1 --2- 172.21.15.123:0/3725053767 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbd6c1035e0 0x7fbd6c0fcc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:31.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.940+0000 7fbd6be00700 1 --2- 172.21.15.123:0/3725053767 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbd6c102c40 0x7fbd6c103010 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:31.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.940+0000 7fbd6be00700 1 -- 172.21.15.123:0/3725053767 >> 172.21.15.123:0/3725053767 conn(0x7fbd6c0f8830 msgr2=0x7fbd6c0fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:41:31.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.940+0000 7fbd6be00700 1 -- 172.21.15.123:0/3725053767 shutdown_connections 2024-03-30T04:41:31.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.941+0000 7fbd6be00700 1 -- 172.21.15.123:0/3725053767 wait complete. 2024-03-30T04:41:31.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.941+0000 7fbd6be00700 1 Processor -- start 2024-03-30T04:41:31.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.941+0000 7fbd6be00700 1 -- start start 2024-03-30T04:41:31.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.942+0000 7fbd6be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbd6c102c40 0x7fbd6c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:41:31.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.942+0000 7fbd6be00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbd6c1035e0 0x7fbd6c070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:41:31.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.942+0000 7fbd6be00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6c070e20 con 0x7fbd6c102c40 2024-03-30T04:41:31.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.942+0000 7fbd6be00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6c070f60 con 0x7fbd6c1035e0 2024-03-30T04:41:31.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.942+0000 7fbd6aa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbd6c102c40 0x7fbd6c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:41:31.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.942+0000 7fbd6aa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbd6c102c40 0x7fbd6c0702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50420/0 (socket says 172.21.15.123:50420) 2024-03-30T04:41:31.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.942+0000 7fbd6aa00700 1 -- 172.21.15.123:0/1128106106 learned_addr learned my addr 172.21.15.123:0/1128106106 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:41:31.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.942+0000 7fbd6a000700 1 --2- 172.21.15.123:0/1128106106 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbd6c1035e0 0x7fbd6c070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:41:31.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.942+0000 7fbd6aa00700 1 -- 172.21.15.123:0/1128106106 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbd6c1035e0 msgr2=0x7fbd6c070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:41:31.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.942+0000 7fbd6aa00700 1 --2- 172.21.15.123:0/1128106106 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbd6c1035e0 0x7fbd6c070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:31.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.942+0000 7fbd6aa00700 1 -- 172.21.15.123:0/1128106106 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd5c005040 con 0x7fbd6c102c40 2024-03-30T04:41:31.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.942+0000 7fbd6a000700 1 --2- 172.21.15.123:0/1128106106 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbd6c1035e0 0x7fbd6c070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:41:31.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.943+0000 7fbd6aa00700 1 --2- 172.21.15.123:0/1128106106 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbd6c102c40 0x7fbd6c0702c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c009390 tx=0x7fbd5c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:41:31.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.943+0000 7fbd62a00700 1 -- 172.21.15.123:0/1128106106 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd5c00e040 con 0x7fbd6c102c40 2024-03-30T04:41:31.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.943+0000 7fbd62a00700 1 -- 172.21.15.123:0/1128106106 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbd5c013420 con 0x7fbd6c102c40 2024-03-30T04:41:31.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.943+0000 7fbd62a00700 1 -- 172.21.15.123:0/1128106106 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd5c01c630 con 0x7fbd6c102c40 2024-03-30T04:41:31.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.943+0000 7fbd6be00700 1 -- 172.21.15.123:0/1128106106 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd6c119bf0 con 0x7fbd6c102c40 2024-03-30T04:41:31.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.943+0000 7fbd6be00700 1 -- 172.21.15.123:0/1128106106 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd6c11a0d0 con 0x7fbd6c102c40 2024-03-30T04:41:31.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.944+0000 7fbd6be00700 1 -- 172.21.15.123:0/1128106106 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd6c0fe390 con 0x7fbd6c102c40 2024-03-30T04:41:31.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.945+0000 7fbd62a00700 1 -- 172.21.15.123:0/1128106106 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fbd5c0139e0 con 0x7fbd6c102c40 2024-03-30T04:41:31.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.946+0000 7fbd62a00700 1 --2- 172.21.15.123:0/1128106106 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbd5006cc30 0x7fbd5006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:41:31.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.946+0000 7fbd62a00700 1 -- 172.21.15.123:0/1128106106 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fbd5c08d220 con 0x7fbd6c102c40 2024-03-30T04:41:31.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.946+0000 7fbd6a000700 1 --2- 172.21.15.123:0/1128106106 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbd5006cc30 0x7fbd5006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:41:31.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.947+0000 7fbd6a000700 1 --2- 172.21.15.123:0/1128106106 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbd5006cc30 0x7fbd5006f0e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c071c00 tx=0x7fbd58008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:41:31.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:31.951+0000 7fbd62a00700 1 -- 172.21.15.123:0/1128106106 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fbd5c0572a0 con 0x7fbd6c102c40 2024-03-30T04:41:32.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.172+0000 7fbd6be00700 1 -- 172.21.15.123:0/1128106106 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbd6c0719b0 con 0x7fbd5006cc30 2024-03-30T04:41:32.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.174+0000 7fbd62a00700 1 -- 172.21.15.123:0/1128106106 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fbd6c0719b0 con 0x7fbd5006cc30 2024-03-30T04:41:32.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.176+0000 7fbd6be00700 1 -- 172.21.15.123:0/1128106106 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbd5006cc30 msgr2=0x7fbd5006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:41:32.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.176+0000 7fbd6be00700 1 --2- 172.21.15.123:0/1128106106 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbd5006cc30 0x7fbd5006f0e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c071c00 tx=0x7fbd58008040 comp rx=0 tx=0).stop 2024-03-30T04:41:32.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.176+0000 7fbd6be00700 1 -- 172.21.15.123:0/1128106106 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbd6c102c40 msgr2=0x7fbd6c0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:41:32.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.176+0000 7fbd6be00700 1 --2- 172.21.15.123:0/1128106106 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbd6c102c40 0x7fbd6c0702c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c009390 tx=0x7fbd5c009e30 comp rx=0 tx=0).stop 2024-03-30T04:41:32.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.176+0000 7fbd6be00700 1 -- 172.21.15.123:0/1128106106 shutdown_connections 2024-03-30T04:41:32.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.176+0000 7fbd6be00700 1 --2- 172.21.15.123:0/1128106106 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbd5006cc30 0x7fbd5006f0e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:32.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.176+0000 7fbd6be00700 1 --2- 172.21.15.123:0/1128106106 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbd6c102c40 0x7fbd6c0702c0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:32.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.176+0000 7fbd6be00700 1 --2- 172.21.15.123:0/1128106106 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbd6c1035e0 0x7fbd6c070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:32.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.176+0000 7fbd6be00700 1 -- 172.21.15.123:0/1128106106 >> 172.21.15.123:0/1128106106 conn(0x7fbd6c0f8830 msgr2=0x7fbd6c0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:41:32.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.176+0000 7fbd6be00700 1 -- 172.21.15.123:0/1128106106 shutdown_connections 2024-03-30T04:41:32.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.176+0000 7fbd6be00700 1 -- 172.21.15.123:0/1128106106 wait complete. 2024-03-30T04:41:32.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.310+0000 7fac70e00700 1 -- 172.21.15.123:0/3079526988 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac6c102e20 msgr2=0x7fac6c1031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:41:32.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.310+0000 7fac70e00700 1 --2- 172.21.15.123:0/3079526988 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac6c102e20 0x7fac6c1031f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fac5c004940 tx=0x7fac5c009e30 comp rx=0 tx=0).stop 2024-03-30T04:41:32.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.310+0000 7fac70e00700 1 -- 172.21.15.123:0/3079526988 shutdown_connections 2024-03-30T04:41:32.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.310+0000 7fac70e00700 1 --2- 172.21.15.123:0/3079526988 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac6c1037c0 0x7fac6c0fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:32.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.310+0000 7fac70e00700 1 --2- 172.21.15.123:0/3079526988 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac6c102e20 0x7fac6c1031f0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:32.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.310+0000 7fac70e00700 1 -- 172.21.15.123:0/3079526988 >> 172.21.15.123:0/3079526988 conn(0x7fac6c0f89d0 msgr2=0x7fac6c0fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:41:32.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.310+0000 7fac70e00700 1 -- 172.21.15.123:0/3079526988 shutdown_connections 2024-03-30T04:41:32.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.310+0000 7fac70e00700 1 -- 172.21.15.123:0/3079526988 wait complete. 2024-03-30T04:41:32.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.311+0000 7fac70e00700 1 Processor -- start 2024-03-30T04:41:32.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.311+0000 7fac70e00700 1 -- start start 2024-03-30T04:41:32.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.311+0000 7fac70e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac6c102e20 0x7fac6c115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:41:32.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.311+0000 7fac70e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac6c1037c0 0x7fac6c116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:41:32.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.311+0000 7fac70e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac6c116880 con 0x7fac6c102e20 2024-03-30T04:41:32.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.311+0000 7fac70e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac6c10fc50 con 0x7fac6c1037c0 2024-03-30T04:41:32.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.311+0000 7fac69e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac6c102e20 0x7fac6c115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:41:32.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.312+0000 7fac69e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac6c102e20 0x7fac6c115cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50436/0 (socket says 172.21.15.123:50436) 2024-03-30T04:41:32.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.312+0000 7fac69e00700 1 -- 172.21.15.123:0/3306634229 learned_addr learned my addr 172.21.15.123:0/3306634229 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:41:32.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.312+0000 7fac69e00700 1 -- 172.21.15.123:0/3306634229 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac6c1037c0 msgr2=0x7fac6c116230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:41:32.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.312+0000 7fac69e00700 1 --2- 172.21.15.123:0/3306634229 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac6c1037c0 0x7fac6c116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:32.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.312+0000 7fac69e00700 1 -- 172.21.15.123:0/3306634229 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac5c005040 con 0x7fac6c102e20 2024-03-30T04:41:32.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.312+0000 7fac69e00700 1 --2- 172.21.15.123:0/3306634229 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac6c102e20 0x7fac6c115cf0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fac5c009390 tx=0x7fac5c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:41:32.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.312+0000 7fac62000700 1 -- 172.21.15.123:0/3306634229 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac5c00e040 con 0x7fac6c102e20 2024-03-30T04:41:32.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.312+0000 7fac62000700 1 -- 172.21.15.123:0/3306634229 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fac5c013420 con 0x7fac6c102e20 2024-03-30T04:41:32.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.312+0000 7fac70e00700 1 -- 172.21.15.123:0/3306634229 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac6c10fed0 con 0x7fac6c102e20 2024-03-30T04:41:32.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.312+0000 7fac62000700 1 -- 172.21.15.123:0/3306634229 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac5c01c630 con 0x7fac6c102e20 2024-03-30T04:41:32.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.313+0000 7fac70e00700 1 -- 172.21.15.123:0/3306634229 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac6c110490 con 0x7fac6c102e20 2024-03-30T04:41:32.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.314+0000 7fac70e00700 1 -- 172.21.15.123:0/3306634229 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac6c0fe570 con 0x7fac6c102e20 2024-03-30T04:41:32.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.315+0000 7fac62000700 1 -- 172.21.15.123:0/3306634229 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fac5c0139e0 con 0x7fac6c102e20 2024-03-30T04:41:32.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.316+0000 7fac62000700 1 --2- 172.21.15.123:0/3306634229 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fac5006cc30 0x7fac5006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:41:32.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.316+0000 7fac62000700 1 -- 172.21.15.123:0/3306634229 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fac5c08d250 con 0x7fac6c102e20 2024-03-30T04:41:32.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.316+0000 7fac69400700 1 --2- 172.21.15.123:0/3306634229 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fac5006cc30 0x7fac5006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:41:32.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.317+0000 7fac69400700 1 --2- 172.21.15.123:0/3306634229 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fac5006cc30 0x7fac5006f0e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fac6c111590 tx=0x7fac58009ce0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:41:32.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.320+0000 7fac62000700 1 -- 172.21.15.123:0/3306634229 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fac5c05add0 con 0x7fac6c102e20 2024-03-30T04:41:32.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.533+0000 7fac70e00700 1 -- 172.21.15.123:0/3306634229 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fac6c04bed0 con 0x7fac5006cc30 2024-03-30T04:41:32.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.543+0000 7fac62000700 1 -- 172.21.15.123:0/3306634229 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fac6c04bed0 con 0x7fac5006cc30 2024-03-30T04:41:32.542 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:41:32.542 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (10m) 3m ago 23m 15.0M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:41:32.542 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (7m) 3m ago 23m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T04:41:32.542 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (7m) 3m ago 22m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T04:41:32.542 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (10m) 3m ago 22m 74.0M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T04:41:32.542 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (3m) 3m ago 16m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T04:41:32.542 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (4m) 3m ago 16m 10.9M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T04:41:32.543 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (16m) 3m ago 16m 10.7M - 17.2.7-797-g5c024335 f77085f6aab7 0dc41326818c 2024-03-30T04:41:32.543 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (3m) 3m ago 16m 9747k - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T04:41:32.543 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (12m) 3m ago 24m 506M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:41:32.543 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (11m) 3m ago 22m 450M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:41:32.543 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (9m) 3m ago 25m 53.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T04:41:32.543 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (8m) 3m ago 22m 42.6M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T04:41:32.543 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (11m) 3m ago 23m 8052k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:41:32.543 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (11m) 3m ago 22m 7880k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:41:32.543 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (7m) 3m ago 21m 104M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T04:41:32.543 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (7m) 3m ago 20m 93.4M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T04:41:32.543 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (6m) 3m ago 20m 86.3M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T04:41:32.543 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (5m) 3m ago 19m 97.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T04:41:32.543 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (5m) 3m ago 19m 97.9M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T04:41:32.543 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (4m) 3m ago 18m 85.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T04:41:32.544 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (11m) 3m ago 22m 53.7M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:41:32.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.546+0000 7fac70e00700 1 -- 172.21.15.123:0/3306634229 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fac5006cc30 msgr2=0x7fac5006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:41:32.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.546+0000 7fac70e00700 1 --2- 172.21.15.123:0/3306634229 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fac5006cc30 0x7fac5006f0e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fac6c111590 tx=0x7fac58009ce0 comp rx=0 tx=0).stop 2024-03-30T04:41:32.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.546+0000 7fac70e00700 1 -- 172.21.15.123:0/3306634229 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac6c102e20 msgr2=0x7fac6c115cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:41:32.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.546+0000 7fac70e00700 1 --2- 172.21.15.123:0/3306634229 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac6c102e20 0x7fac6c115cf0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fac5c009390 tx=0x7fac5c009e30 comp rx=0 tx=0).stop 2024-03-30T04:41:32.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.546+0000 7fac70e00700 1 -- 172.21.15.123:0/3306634229 shutdown_connections 2024-03-30T04:41:32.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.546+0000 7fac70e00700 1 --2- 172.21.15.123:0/3306634229 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fac5006cc30 0x7fac5006f0e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:32.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.546+0000 7fac70e00700 1 --2- 172.21.15.123:0/3306634229 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac6c102e20 0x7fac6c115cf0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:32.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.546+0000 7fac70e00700 1 --2- 172.21.15.123:0/3306634229 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac6c1037c0 0x7fac6c116230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:32.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.546+0000 7fac70e00700 1 -- 172.21.15.123:0/3306634229 >> 172.21.15.123:0/3306634229 conn(0x7fac6c0f89d0 msgr2=0x7fac6c0f9a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:41:32.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.546+0000 7fac70e00700 1 -- 172.21.15.123:0/3306634229 shutdown_connections 2024-03-30T04:41:32.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.546+0000 7fac70e00700 1 -- 172.21.15.123:0/3306634229 wait complete. 2024-03-30T04:41:32.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.682+0000 7fe0dca00700 1 -- 172.21.15.123:0/3331355045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe0d80708a0 msgr2=0x7fe0d8070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:41:32.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.682+0000 7fe0dca00700 1 --2- 172.21.15.123:0/3331355045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe0d80708a0 0x7fe0d8070d10 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fe0c4004970 tx=0x7fe0c4009e30 comp rx=0 tx=0).stop 2024-03-30T04:41:32.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.682+0000 7fe0dca00700 1 -- 172.21.15.123:0/3331355045 shutdown_connections 2024-03-30T04:41:32.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.682+0000 7fe0dca00700 1 --2- 172.21.15.123:0/3331355045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe0d80708a0 0x7fe0d8070d10 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:32.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.682+0000 7fe0dca00700 1 --2- 172.21.15.123:0/3331355045 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe0d8071f90 0x7fe0d8070360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:32.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.682+0000 7fe0dca00700 1 -- 172.21.15.123:0/3331355045 >> 172.21.15.123:0/3331355045 conn(0x7fe0d80f89b0 msgr2=0x7fe0d80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:41:32.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.682+0000 7fe0dca00700 1 -- 172.21.15.123:0/3331355045 shutdown_connections 2024-03-30T04:41:32.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.682+0000 7fe0dca00700 1 -- 172.21.15.123:0/3331355045 wait complete. 2024-03-30T04:41:32.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.683+0000 7fe0dca00700 1 Processor -- start 2024-03-30T04:41:32.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.683+0000 7fe0dca00700 1 -- start start 2024-03-30T04:41:32.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.684+0000 7fe0dca00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe0d80708a0 0x7fe0d811a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:41:32.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.684+0000 7fe0dca00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe0d8071f90 0x7fe0d811a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:41:32.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.684+0000 7fe0dca00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d811ab90 con 0x7fe0d80708a0 2024-03-30T04:41:32.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.684+0000 7fe0dca00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d8114010 con 0x7fe0d8071f90 2024-03-30T04:41:32.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.684+0000 7fe0d5e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe0d80708a0 0x7fe0d811a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:41:32.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.684+0000 7fe0d5e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe0d80708a0 0x7fe0d811a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50446/0 (socket says 172.21.15.123:50446) 2024-03-30T04:41:32.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.684+0000 7fe0d5e00700 1 -- 172.21.15.123:0/932806810 learned_addr learned my addr 172.21.15.123:0/932806810 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:41:32.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.684+0000 7fe0d5400700 1 --2- 172.21.15.123:0/932806810 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe0d8071f90 0x7fe0d811a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:41:32.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.685+0000 7fe0d5e00700 1 -- 172.21.15.123:0/932806810 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe0d8071f90 msgr2=0x7fe0d811a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:41:32.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.685+0000 7fe0d5e00700 1 --2- 172.21.15.123:0/932806810 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe0d8071f90 0x7fe0d811a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:32.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.685+0000 7fe0d5e00700 1 -- 172.21.15.123:0/932806810 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c4005040 con 0x7fe0d80708a0 2024-03-30T04:41:32.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.685+0000 7fe0d5e00700 1 --2- 172.21.15.123:0/932806810 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe0d80708a0 0x7fe0d811a000 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fe0c800eb40 tx=0x7fe0c800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:41:32.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.685+0000 7fe0ce000700 1 -- 172.21.15.123:0/932806810 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe0c8007cc0 con 0x7fe0d80708a0 2024-03-30T04:41:32.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.685+0000 7fe0ce000700 1 -- 172.21.15.123:0/932806810 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe0c800f040 con 0x7fe0d80708a0 2024-03-30T04:41:32.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.685+0000 7fe0ce000700 1 -- 172.21.15.123:0/932806810 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe0c800c990 con 0x7fe0d80708a0 2024-03-30T04:41:32.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.685+0000 7fe0dca00700 1 -- 172.21.15.123:0/932806810 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0d8114290 con 0x7fe0d80708a0 2024-03-30T04:41:32.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.685+0000 7fe0dca00700 1 -- 172.21.15.123:0/932806810 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0d81147d0 con 0x7fe0d80708a0 2024-03-30T04:41:32.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.687+0000 7fe0dca00700 1 -- 172.21.15.123:0/932806810 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0d8071aa0 con 0x7fe0d80708a0 2024-03-30T04:41:32.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.687+0000 7fe0ce000700 1 -- 172.21.15.123:0/932806810 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fe0c800caf0 con 0x7fe0d80708a0 2024-03-30T04:41:32.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.688+0000 7fe0ce000700 1 --2- 172.21.15.123:0/932806810 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe0bc06cc30 0x7fe0bc06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:41:32.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.688+0000 7fe0ce000700 1 -- 172.21.15.123:0/932806810 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe0c808bb80 con 0x7fe0d80708a0 2024-03-30T04:41:32.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.689+0000 7fe0d5400700 1 --2- 172.21.15.123:0/932806810 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe0bc06cc30 0x7fe0bc06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:41:32.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.689+0000 7fe0d5400700 1 --2- 172.21.15.123:0/932806810 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe0bc06cc30 0x7fe0bc06f0e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe0d810cf40 tx=0x7fe0c4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:41:32.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.692+0000 7fe0ce000700 1 -- 172.21.15.123:0/932806810 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe0c8055c70 con 0x7fe0d80708a0 2024-03-30T04:41:32.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:41:32 smithi135 ceph-mon[106011]: pgmap v310: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:32.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:41:32 smithi135 ceph-mon[106011]: from='client.34626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:41:32.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:41:32 smithi123 ceph-mon[139369]: pgmap v310: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:32.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:41:32 smithi123 ceph-mon[139369]: from='client.34626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:41:32.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.977+0000 7fe0dca00700 1 -- 172.21.15.123:0/932806810 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe0d81103e0 con 0x7fe0d80708a0 2024-03-30T04:41:32.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.978+0000 7fe0ce000700 1 -- 172.21.15.123:0/932806810 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7fe0c8059290 con 0x7fe0d80708a0 2024-03-30T04:41:32.976 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:41:32.976 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:41:32.976 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:41:32.976 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:41:32.976 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:41:32.976 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:41:32.976 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:41:32.977 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:41:32.977 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T04:41:32.977 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:41:32.977 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:41:32.977 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:41:32.977 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 3 2024-03-30T04:41:32.977 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:41:32.977 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:41:32.977 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:41:32.977 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 13 2024-03-30T04:41:32.977 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:41:32.977 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:41:32.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.980+0000 7fe0dca00700 1 -- 172.21.15.123:0/932806810 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe0bc06cc30 msgr2=0x7fe0bc06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:41:32.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.980+0000 7fe0dca00700 1 --2- 172.21.15.123:0/932806810 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe0bc06cc30 0x7fe0bc06f0e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe0d810cf40 tx=0x7fe0c4009e30 comp rx=0 tx=0).stop 2024-03-30T04:41:32.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.980+0000 7fe0dca00700 1 -- 172.21.15.123:0/932806810 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe0d80708a0 msgr2=0x7fe0d811a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:41:32.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.980+0000 7fe0dca00700 1 --2- 172.21.15.123:0/932806810 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe0d80708a0 0x7fe0d811a000 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fe0c800eb40 tx=0x7fe0c800b040 comp rx=0 tx=0).stop 2024-03-30T04:41:32.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.980+0000 7fe0dca00700 1 -- 172.21.15.123:0/932806810 shutdown_connections 2024-03-30T04:41:32.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.980+0000 7fe0dca00700 1 --2- 172.21.15.123:0/932806810 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe0bc06cc30 0x7fe0bc06f0e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:32.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.980+0000 7fe0dca00700 1 --2- 172.21.15.123:0/932806810 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe0d80708a0 0x7fe0d811a000 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:32.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.980+0000 7fe0dca00700 1 --2- 172.21.15.123:0/932806810 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe0d8071f90 0x7fe0d811a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:32.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.980+0000 7fe0dca00700 1 -- 172.21.15.123:0/932806810 >> 172.21.15.123:0/932806810 conn(0x7fe0d80f89b0 msgr2=0x7fe0d80f9b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:41:32.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.980+0000 7fe0dca00700 1 -- 172.21.15.123:0/932806810 shutdown_connections 2024-03-30T04:41:32.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:32.981+0000 7fe0dca00700 1 -- 172.21.15.123:0/932806810 wait complete. 2024-03-30T04:41:33.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.114+0000 7f7d07200700 1 -- 172.21.15.123:0/2912413914 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d081035e0 msgr2=0x7f7d080fcc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:41:33.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.114+0000 7f7d07200700 1 --2- 172.21.15.123:0/2912413914 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d081035e0 0x7f7d080fcc50 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f7cf4004940 tx=0x7f7cf4009e30 comp rx=0 tx=0).stop 2024-03-30T04:41:33.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.114+0000 7f7d07200700 1 -- 172.21.15.123:0/2912413914 shutdown_connections 2024-03-30T04:41:33.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.114+0000 7f7d07200700 1 --2- 172.21.15.123:0/2912413914 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d081035e0 0x7f7d080fcc50 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:33.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.114+0000 7f7d07200700 1 --2- 172.21.15.123:0/2912413914 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d08102c40 0x7f7d08103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:33.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.114+0000 7f7d07200700 1 -- 172.21.15.123:0/2912413914 >> 172.21.15.123:0/2912413914 conn(0x7f7d080f8830 msgr2=0x7f7d080fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:41:33.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.114+0000 7f7d07200700 1 -- 172.21.15.123:0/2912413914 shutdown_connections 2024-03-30T04:41:33.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.114+0000 7f7d07200700 1 -- 172.21.15.123:0/2912413914 wait complete. 2024-03-30T04:41:33.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.115+0000 7f7d07200700 1 Processor -- start 2024-03-30T04:41:33.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.115+0000 7f7d07200700 1 -- start start 2024-03-30T04:41:33.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.116+0000 7f7d07200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d08102c40 0x7f7d080702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:41:33.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.116+0000 7f7d07200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d081035e0 0x7f7d08070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:41:33.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.116+0000 7f7d07200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d08070d90 con 0x7f7d08102c40 2024-03-30T04:41:33.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.116+0000 7f7d07200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d08070ed0 con 0x7f7d081035e0 2024-03-30T04:41:33.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.116+0000 7f7d05e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d08102c40 0x7f7d080702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:41:33.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.116+0000 7f7d05e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d08102c40 0x7f7d080702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50464/0 (socket says 172.21.15.123:50464) 2024-03-30T04:41:33.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.116+0000 7f7d05e00700 1 -- 172.21.15.123:0/4081088865 learned_addr learned my addr 172.21.15.123:0/4081088865 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:41:33.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.116+0000 7f7d05400700 1 --2- 172.21.15.123:0/4081088865 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d081035e0 0x7f7d08070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:41:33.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.116+0000 7f7d05e00700 1 -- 172.21.15.123:0/4081088865 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d081035e0 msgr2=0x7f7d08070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:41:33.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.116+0000 7f7d05e00700 1 --2- 172.21.15.123:0/4081088865 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d081035e0 0x7f7d08070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:33.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.116+0000 7f7d05e00700 1 -- 172.21.15.123:0/4081088865 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cf4005040 con 0x7f7d08102c40 2024-03-30T04:41:33.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.116+0000 7f7d05400700 1 --2- 172.21.15.123:0/4081088865 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d081035e0 0x7f7d08070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:41:33.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.116+0000 7f7d05e00700 1 --2- 172.21.15.123:0/4081088865 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d08102c40 0x7f7d080702c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f7cf8002a50 tx=0x7f7cf80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:41:33.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.117+0000 7f7cfe000700 1 -- 172.21.15.123:0/4081088865 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7cf8007cc0 con 0x7f7d08102c40 2024-03-30T04:41:33.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.117+0000 7f7d07200700 1 -- 172.21.15.123:0/4081088865 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d08119be0 con 0x7f7d08102c40 2024-03-30T04:41:33.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.117+0000 7f7cfe000700 1 -- 172.21.15.123:0/4081088865 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7cf800f040 con 0x7f7d08102c40 2024-03-30T04:41:33.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.117+0000 7f7cfe000700 1 -- 172.21.15.123:0/4081088865 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7cf8009540 con 0x7f7d08102c40 2024-03-30T04:41:33.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.117+0000 7f7d07200700 1 -- 172.21.15.123:0/4081088865 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d0811a060 con 0x7f7d08102c40 2024-03-30T04:41:33.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.118+0000 7f7d07200700 1 -- 172.21.15.123:0/4081088865 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d080fe390 con 0x7f7d08102c40 2024-03-30T04:41:33.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.119+0000 7f7cfe000700 1 -- 172.21.15.123:0/4081088865 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f7cf80096a0 con 0x7f7d08102c40 2024-03-30T04:41:33.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.120+0000 7f7cfe000700 1 --2- 172.21.15.123:0/4081088865 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7cec06cc30 0x7f7cec06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:41:33.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.120+0000 7f7cfe000700 1 -- 172.21.15.123:0/4081088865 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7cf808bde0 con 0x7f7d08102c40 2024-03-30T04:41:33.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.120+0000 7f7d05400700 1 --2- 172.21.15.123:0/4081088865 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7cec06cc30 0x7f7cec06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:41:33.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.121+0000 7f7d05400700 1 --2- 172.21.15.123:0/4081088865 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7cec06cc30 0x7f7cec06f0e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7d08071c00 tx=0x7f7cf4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:41:33.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.124+0000 7f7cfe000700 1 -- 172.21.15.123:0/4081088865 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7cf8055da0 con 0x7f7d08102c40 2024-03-30T04:41:33.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.370+0000 7f7d07200700 1 -- 172.21.15.123:0/4081088865 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f7d080719b0 con 0x7f7d08102c40 2024-03-30T04:41:33.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.370+0000 7f7cfe000700 1 -- 172.21.15.123:0/4081088865 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f7cf80593c0 con 0x7f7d08102c40 2024-03-30T04:41:33.369 INFO:teuthology.orchestra.run.smithi123.stdout:e32 2024-03-30T04:41:33.369 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:41:33.369 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:41:33.369 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:41:33.369 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:41:33.369 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:41:33.369 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:41:33.369 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T04:41:33.369 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T04:41:33.370 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:41:33.370 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T04:41:33.370 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:41:33.370 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:41:33.370 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:41:33.370 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:41:33.370 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:41:33.370 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T04:41:33.370 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:41:33.370 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:41:33.370 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T04:41:33.370 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:41:33.370 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T04:41:33.370 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T04:41:33.371 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T04:41:33.371 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T04:41:33.371 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:41:33.371 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:41:33.371 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:41:33.371 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:41:33.371 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:41:33.371 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:41:33.371 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T04:41:33.371 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T04:41:33.371 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:41:33.371 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:41:33.371 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T04:41:33.371 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:41:33.372 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:41:33.372 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:41:33.372 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44315} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/1741352336,v1:172.21.15.135:6827/1741352336] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:41:33.372 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:41:33.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.373+0000 7f7d07200700 1 -- 172.21.15.123:0/4081088865 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7cec06cc30 msgr2=0x7f7cec06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:41:33.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.373+0000 7f7d07200700 1 --2- 172.21.15.123:0/4081088865 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7cec06cc30 0x7f7cec06f0e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7d08071c00 tx=0x7f7cf4009e30 comp rx=0 tx=0).stop 2024-03-30T04:41:33.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.373+0000 7f7d07200700 1 -- 172.21.15.123:0/4081088865 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d08102c40 msgr2=0x7f7d080702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:41:33.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.373+0000 7f7d07200700 1 --2- 172.21.15.123:0/4081088865 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d08102c40 0x7f7d080702c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f7cf8002a50 tx=0x7f7cf80046b0 comp rx=0 tx=0).stop 2024-03-30T04:41:33.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.373+0000 7f7d07200700 1 -- 172.21.15.123:0/4081088865 shutdown_connections 2024-03-30T04:41:33.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.373+0000 7f7d07200700 1 --2- 172.21.15.123:0/4081088865 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7cec06cc30 0x7f7cec06f0e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:33.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.373+0000 7f7d07200700 1 --2- 172.21.15.123:0/4081088865 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d08102c40 0x7f7d080702c0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:33.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.373+0000 7f7d07200700 1 --2- 172.21.15.123:0/4081088865 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d081035e0 0x7f7d08070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:33.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.373+0000 7f7d07200700 1 -- 172.21.15.123:0/4081088865 >> 172.21.15.123:0/4081088865 conn(0x7f7d080f8830 msgr2=0x7f7d080f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:41:33.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.373+0000 7f7d07200700 1 -- 172.21.15.123:0/4081088865 shutdown_connections 2024-03-30T04:41:33.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.373+0000 7f7d07200700 1 -- 172.21.15.123:0/4081088865 wait complete. 2024-03-30T04:41:33.373 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 32 2024-03-30T04:41:33.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.507+0000 7f359b800700 1 -- 172.21.15.123:0/3449006917 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3594102e00 msgr2=0x7f35941031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:41:33.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.507+0000 7f359b800700 1 --2- 172.21.15.123:0/3449006917 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3594102e00 0x7f35941031d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f3588004940 tx=0x7f3588009e30 comp rx=0 tx=0).stop 2024-03-30T04:41:33.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.507+0000 7f359b800700 1 -- 172.21.15.123:0/3449006917 shutdown_connections 2024-03-30T04:41:33.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.507+0000 7f359b800700 1 --2- 172.21.15.123:0/3449006917 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f35941037a0 0x7f35940fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:33.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.507+0000 7f359b800700 1 --2- 172.21.15.123:0/3449006917 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3594102e00 0x7f35941031d0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:33.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.507+0000 7f359b800700 1 -- 172.21.15.123:0/3449006917 >> 172.21.15.123:0/3449006917 conn(0x7f35940f89b0 msgr2=0x7f35940fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:41:33.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.507+0000 7f359b800700 1 -- 172.21.15.123:0/3449006917 shutdown_connections 2024-03-30T04:41:33.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.508+0000 7f359b800700 1 -- 172.21.15.123:0/3449006917 wait complete. 2024-03-30T04:41:33.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.508+0000 7f359b800700 1 Processor -- start 2024-03-30T04:41:33.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.508+0000 7f359b800700 1 -- start start 2024-03-30T04:41:33.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.509+0000 7f359b800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3594102e00 0x7f3594115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:41:33.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.509+0000 7f359b800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f35941037a0 0x7f3594116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:41:33.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.509+0000 7f359b800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3594116860 con 0x7f3594102e00 2024-03-30T04:41:33.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.509+0000 7f359b800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f359410fc30 con 0x7f35941037a0 2024-03-30T04:41:33.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.509+0000 7f3598e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3594102e00 0x7f3594115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:41:33.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.509+0000 7f3598e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3594102e00 0x7f3594115cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50472/0 (socket says 172.21.15.123:50472) 2024-03-30T04:41:33.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.509+0000 7f3598e00700 1 -- 172.21.15.123:0/3089377259 learned_addr learned my addr 172.21.15.123:0/3089377259 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:41:33.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.509+0000 7f3593e00700 1 --2- 172.21.15.123:0/3089377259 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f35941037a0 0x7f3594116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:41:33.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.510+0000 7f3598e00700 1 -- 172.21.15.123:0/3089377259 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f35941037a0 msgr2=0x7f3594116210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:41:33.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.510+0000 7f3598e00700 1 --2- 172.21.15.123:0/3089377259 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f35941037a0 0x7f3594116210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:33.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.510+0000 7f3598e00700 1 -- 172.21.15.123:0/3089377259 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3588005040 con 0x7f3594102e00 2024-03-30T04:41:33.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.510+0000 7f3598e00700 1 --2- 172.21.15.123:0/3089377259 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3594102e00 0x7f3594115cd0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3588009390 tx=0x7f3588009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:41:33.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.510+0000 7f3590c00700 1 -- 172.21.15.123:0/3089377259 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f358800e040 con 0x7f3594102e00 2024-03-30T04:41:33.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.510+0000 7f359b800700 1 -- 172.21.15.123:0/3089377259 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f359410feb0 con 0x7f3594102e00 2024-03-30T04:41:33.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.510+0000 7f3590c00700 1 -- 172.21.15.123:0/3089377259 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3588013420 con 0x7f3594102e00 2024-03-30T04:41:33.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.510+0000 7f3590c00700 1 -- 172.21.15.123:0/3089377259 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f358801c630 con 0x7f3594102e00 2024-03-30T04:41:33.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.510+0000 7f359b800700 1 -- 172.21.15.123:0/3089377259 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3594110470 con 0x7f3594102e00 2024-03-30T04:41:33.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.511+0000 7f359b800700 1 -- 172.21.15.123:0/3089377259 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35940fe550 con 0x7f3594102e00 2024-03-30T04:41:33.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.512+0000 7f3590c00700 1 -- 172.21.15.123:0/3089377259 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f35880139e0 con 0x7f3594102e00 2024-03-30T04:41:33.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.513+0000 7f3590c00700 1 --2- 172.21.15.123:0/3089377259 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f357c06cc30 0x7f357c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:41:33.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.513+0000 7f3590c00700 1 -- 172.21.15.123:0/3089377259 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f358808d3d0 con 0x7f3594102e00 2024-03-30T04:41:33.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.513+0000 7f3593e00700 1 --2- 172.21.15.123:0/3089377259 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f357c06cc30 0x7f357c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:41:33.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.514+0000 7f3593e00700 1 --2- 172.21.15.123:0/3089377259 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f357c06cc30 0x7f357c06f0e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3594111570 tx=0x7f358000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:41:33.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.518+0000 7f3590c00700 1 -- 172.21.15.123:0/3089377259 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f358805ae20 con 0x7f3594102e00 2024-03-30T04:41:33.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.741+0000 7f359b800700 1 -- 172.21.15.123:0/3089377259 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f359404bf70 con 0x7f357c06cc30 2024-03-30T04:41:33.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.742+0000 7f3590c00700 1 -- 172.21.15.123:0/3089377259 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f358000f040 con 0x7f357c06cc30 2024-03-30T04:41:33.740 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:41:33.741 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:41:33.741 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:41:33.741 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T04:41:33.741 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:41:33.741 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T04:41:33.741 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T04:41:33.741 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T04:41:33.741 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T04:41:33.741 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:41:33.741 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "15/21 daemons upgraded", 2024-03-30T04:41:33.741 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T04:41:33.741 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:41:33.741 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:41:33.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.745+0000 7f359b800700 1 -- 172.21.15.123:0/3089377259 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f357c06cc30 msgr2=0x7f357c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:41:33.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.745+0000 7f359b800700 1 --2- 172.21.15.123:0/3089377259 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f357c06cc30 0x7f357c06f0e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3594111570 tx=0x7f358000b040 comp rx=0 tx=0).stop 2024-03-30T04:41:33.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.745+0000 7f359b800700 1 -- 172.21.15.123:0/3089377259 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3594102e00 msgr2=0x7f3594115cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:41:33.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.745+0000 7f359b800700 1 --2- 172.21.15.123:0/3089377259 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3594102e00 0x7f3594115cd0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3588009390 tx=0x7f3588009e30 comp rx=0 tx=0).stop 2024-03-30T04:41:33.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.745+0000 7f359b800700 1 -- 172.21.15.123:0/3089377259 shutdown_connections 2024-03-30T04:41:33.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.745+0000 7f359b800700 1 --2- 172.21.15.123:0/3089377259 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f357c06cc30 0x7f357c06f0e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:33.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.745+0000 7f359b800700 1 --2- 172.21.15.123:0/3089377259 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3594102e00 0x7f3594115cd0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:33.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.745+0000 7f359b800700 1 --2- 172.21.15.123:0/3089377259 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f35941037a0 0x7f3594116210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:33.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.745+0000 7f359b800700 1 -- 172.21.15.123:0/3089377259 >> 172.21.15.123:0/3089377259 conn(0x7f35940f89b0 msgr2=0x7f35940f9980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:41:33.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.745+0000 7f359b800700 1 -- 172.21.15.123:0/3089377259 shutdown_connections 2024-03-30T04:41:33.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.745+0000 7f359b800700 1 -- 172.21.15.123:0/3089377259 wait complete. 2024-03-30T04:41:33.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:41:33 smithi135 ceph-mon[106011]: from='client.34630 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:41:33.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:41:33 smithi135 ceph-mon[106011]: from='client.34634 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:41:33.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:41:33 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/932806810' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:41:33.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:41:33 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/4081088865' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:41:33.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.878+0000 7fb010000700 1 -- 172.21.15.123:0/1761956421 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb008102e00 msgr2=0x7fb0081031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:41:33.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.878+0000 7fb010000700 1 --2- 172.21.15.123:0/1761956421 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb008102e00 0x7fb0081031d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7faffc004940 tx=0x7faffc009e30 comp rx=0 tx=0).stop 2024-03-30T04:41:33.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.878+0000 7fb010000700 1 -- 172.21.15.123:0/1761956421 shutdown_connections 2024-03-30T04:41:33.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.878+0000 7fb010000700 1 --2- 172.21.15.123:0/1761956421 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb0081037a0 0x7fb0080fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:33.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.878+0000 7fb010000700 1 --2- 172.21.15.123:0/1761956421 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb008102e00 0x7fb0081031d0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:33.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.878+0000 7fb010000700 1 -- 172.21.15.123:0/1761956421 >> 172.21.15.123:0/1761956421 conn(0x7fb0080f89b0 msgr2=0x7fb0080fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:41:33.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.878+0000 7fb010000700 1 -- 172.21.15.123:0/1761956421 shutdown_connections 2024-03-30T04:41:33.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.878+0000 7fb010000700 1 -- 172.21.15.123:0/1761956421 wait complete. 2024-03-30T04:41:33.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.879+0000 7fb010000700 1 Processor -- start 2024-03-30T04:41:33.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.879+0000 7fb010000700 1 -- start start 2024-03-30T04:41:33.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.879+0000 7fb010000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb008102e00 0x7fb008115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:41:33.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.879+0000 7fb010000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0081037a0 0x7fb0081161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:41:33.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.879+0000 7fb010000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb008116800 con 0x7fb0081037a0 2024-03-30T04:41:33.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.879+0000 7fb010000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb00810fbd0 con 0x7fb008102e00 2024-03-30T04:41:33.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.880+0000 7fb00cc00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0081037a0 0x7fb0081161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:41:33.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.880+0000 7fb00cc00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0081037a0 0x7fb0081161b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39842/0 (socket says 172.21.15.123:39842) 2024-03-30T04:41:33.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.880+0000 7fb00cc00700 1 -- 172.21.15.123:0/3590811635 learned_addr learned my addr 172.21.15.123:0/3590811635 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:41:33.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.880+0000 7fb00d600700 1 --2- 172.21.15.123:0/3590811635 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb008102e00 0x7fb008115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:41:33.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.880+0000 7fb00cc00700 1 -- 172.21.15.123:0/3590811635 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb008102e00 msgr2=0x7fb008115c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:41:33.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.880+0000 7fb00cc00700 1 --2- 172.21.15.123:0/3590811635 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb008102e00 0x7fb008115c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:33.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.880+0000 7fb00cc00700 1 -- 172.21.15.123:0/3590811635 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faffc005040 con 0x7fb0081037a0 2024-03-30T04:41:33.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.880+0000 7fb00d600700 1 --2- 172.21.15.123:0/3590811635 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb008102e00 0x7fb008115c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:41:33.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.880+0000 7fb00cc00700 1 --2- 172.21.15.123:0/3590811635 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0081037a0 0x7fb0081161b0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7faff4002a50 tx=0x7faff40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:41:33.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.880+0000 7fb005600700 1 -- 172.21.15.123:0/3590811635 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faff4007cc0 con 0x7fb0081037a0 2024-03-30T04:41:33.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.880+0000 7fb005600700 1 -- 172.21.15.123:0/3590811635 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faff400f040 con 0x7fb0081037a0 2024-03-30T04:41:33.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.880+0000 7fb010000700 1 -- 172.21.15.123:0/3590811635 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb00810feb0 con 0x7fb0081037a0 2024-03-30T04:41:33.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.881+0000 7fb005600700 1 -- 172.21.15.123:0/3590811635 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faff40095d0 con 0x7fb0081037a0 2024-03-30T04:41:33.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.881+0000 7fb010000700 1 -- 172.21.15.123:0/3590811635 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb008110470 con 0x7fb0081037a0 2024-03-30T04:41:33.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.882+0000 7fb010000700 1 -- 172.21.15.123:0/3590811635 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0080fe550 con 0x7fb0081037a0 2024-03-30T04:41:33.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.883+0000 7fb005600700 1 -- 172.21.15.123:0/3590811635 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7faff4009730 con 0x7fb0081037a0 2024-03-30T04:41:33.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.885+0000 7fb005600700 1 --2- 172.21.15.123:0/3590811635 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7faff006cc30 0x7faff006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:41:33.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.885+0000 7fb005600700 1 -- 172.21.15.123:0/3590811635 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7faff408bc60 con 0x7fb0081037a0 2024-03-30T04:41:33.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.885+0000 7fb00d600700 1 --2- 172.21.15.123:0/3590811635 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7faff006cc30 0x7faff006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:41:33.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.886+0000 7fb00d600700 1 --2- 172.21.15.123:0/3590811635 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7faff006cc30 0x7faff006f0e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7faffc00b840 tx=0x7faffc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:41:33.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:33.889+0000 7fb005600700 1 -- 172.21.15.123:0/3590811635 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7faff40597e0 con 0x7fb0081037a0 2024-03-30T04:41:33.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:41:33 smithi123 ceph-mon[139369]: from='client.34630 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:41:33.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:41:33 smithi123 ceph-mon[139369]: from='client.34634 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:41:33.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:41:33 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/932806810' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:41:33.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:41:33 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/4081088865' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:41:34.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:34.174+0000 7fb010000700 1 -- 172.21.15.123:0/3590811635 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fb00804bed0 con 0x7fb0081037a0 2024-03-30T04:41:34.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:34.174+0000 7fb005600700 1 -- 172.21.15.123:0/3590811635 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7faff4059370 con 0x7fb0081037a0 2024-03-30T04:41:34.173 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T04:41:34.173 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T04:41:34.173 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T04:41:34.173 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T04:41:34.173 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T04:41:34.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:34.177+0000 7fb010000700 1 -- 172.21.15.123:0/3590811635 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7faff006cc30 msgr2=0x7faff006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:41:34.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:34.177+0000 7fb010000700 1 --2- 172.21.15.123:0/3590811635 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7faff006cc30 0x7faff006f0e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7faffc00b840 tx=0x7faffc009e30 comp rx=0 tx=0).stop 2024-03-30T04:41:34.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:34.177+0000 7fb010000700 1 -- 172.21.15.123:0/3590811635 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0081037a0 msgr2=0x7fb0081161b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:41:34.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:34.177+0000 7fb010000700 1 --2- 172.21.15.123:0/3590811635 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0081037a0 0x7fb0081161b0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7faff4002a50 tx=0x7faff40046b0 comp rx=0 tx=0).stop 2024-03-30T04:41:34.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:34.177+0000 7fb010000700 1 -- 172.21.15.123:0/3590811635 shutdown_connections 2024-03-30T04:41:34.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:34.177+0000 7fb010000700 1 --2- 172.21.15.123:0/3590811635 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7faff006cc30 0x7faff006f0e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:34.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:34.177+0000 7fb010000700 1 --2- 172.21.15.123:0/3590811635 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb008102e00 0x7fb008115c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:34.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:34.177+0000 7fb010000700 1 --2- 172.21.15.123:0/3590811635 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0081037a0 0x7fb0081161b0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:41:34.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:34.177+0000 7fb010000700 1 -- 172.21.15.123:0/3590811635 >> 172.21.15.123:0/3590811635 conn(0x7fb0080f89b0 msgr2=0x7fb0080f9960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:41:34.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:34.177+0000 7fb010000700 1 -- 172.21.15.123:0/3590811635 shutdown_connections 2024-03-30T04:41:34.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:41:34.177+0000 7fb010000700 1 -- 172.21.15.123:0/3590811635 wait complete. 2024-03-30T04:41:34.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:41:34 smithi135 ceph-mon[106011]: pgmap v311: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:34.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:41:34 smithi135 ceph-mon[106011]: from='client.34646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:41:34.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:41:34 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3590811635' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:41:34.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:41:34 smithi123 ceph-mon[139369]: pgmap v311: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:34.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:41:34 smithi123 ceph-mon[139369]: from='client.34646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:41:34.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:41:34 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3590811635' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:41:36.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:41:36 smithi135 ceph-mon[106011]: pgmap v312: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:36.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:41:36 smithi123 ceph-mon[139369]: pgmap v312: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:38.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:41:38 smithi135 ceph-mon[106011]: pgmap v313: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:38.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:41:38 smithi123 ceph-mon[139369]: pgmap v313: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:40.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:41:40 smithi123 ceph-mon[139369]: pgmap v314: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:40.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:41:40 smithi135 ceph-mon[106011]: pgmap v314: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:42.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:41:42 smithi135 ceph-mon[106011]: pgmap v315: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:42.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:41:42 smithi123 ceph-mon[139369]: pgmap v315: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:44.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:41:44 smithi135 ceph-mon[106011]: pgmap v316: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:44.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:41:44 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:41:44.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:41:44 smithi123 ceph-mon[139369]: pgmap v316: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:44.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:41:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:41:46.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:41:46 smithi135 ceph-mon[106011]: pgmap v317: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:46.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:41:46 smithi123 ceph-mon[139369]: pgmap v317: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:48.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:41:48 smithi135 ceph-mon[106011]: pgmap v318: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:48.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:41:48 smithi123 ceph-mon[139369]: pgmap v318: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:50.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:41:50 smithi123 ceph-mon[139369]: pgmap v319: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:50.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:41:50 smithi135 ceph-mon[106011]: pgmap v319: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:52.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:41:52 smithi135 ceph-mon[106011]: pgmap v320: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:52.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:41:52 smithi123 ceph-mon[139369]: pgmap v320: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:54.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:41:54 smithi135 ceph-mon[106011]: pgmap v321: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:54.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:41:54 smithi123 ceph-mon[139369]: pgmap v321: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:56.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:41:56 smithi135 ceph-mon[106011]: pgmap v322: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:56.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:41:56 smithi123 ceph-mon[139369]: pgmap v322: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:58.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:41:58 smithi135 ceph-mon[106011]: pgmap v323: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:58.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:41:58 smithi123 ceph-mon[139369]: pgmap v323: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:41:59.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:41:59 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:41:59.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:41:59 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:42:00.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:42:00 smithi135 ceph-mon[106011]: pgmap v324: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:00.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:42:00 smithi123 ceph-mon[139369]: pgmap v324: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:02.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:42:02 smithi135 ceph-mon[106011]: pgmap v325: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:02.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:42:02 smithi123 ceph-mon[139369]: pgmap v325: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:04.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.326+0000 7fa351400700 1 -- 172.21.15.123:0/4009492107 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa34c1037a0 msgr2=0x7fa34c0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:04.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.326+0000 7fa351400700 1 --2- 172.21.15.123:0/4009492107 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa34c1037a0 0x7fa34c0fce10 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa338004940 tx=0x7fa338009e30 comp rx=0 tx=0).stop 2024-03-30T04:42:04.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.326+0000 7fa351400700 1 -- 172.21.15.123:0/4009492107 shutdown_connections 2024-03-30T04:42:04.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.326+0000 7fa351400700 1 --2- 172.21.15.123:0/4009492107 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa34c1037a0 0x7fa34c0fce10 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:04.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.326+0000 7fa351400700 1 --2- 172.21.15.123:0/4009492107 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa34c102e00 0x7fa34c1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:04.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.326+0000 7fa351400700 1 -- 172.21.15.123:0/4009492107 >> 172.21.15.123:0/4009492107 conn(0x7fa34c0f89b0 msgr2=0x7fa34c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:42:04.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.326+0000 7fa351400700 1 -- 172.21.15.123:0/4009492107 shutdown_connections 2024-03-30T04:42:04.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.327+0000 7fa351400700 1 -- 172.21.15.123:0/4009492107 wait complete. 2024-03-30T04:42:04.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.327+0000 7fa351400700 1 Processor -- start 2024-03-30T04:42:04.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.327+0000 7fa351400700 1 -- start start 2024-03-30T04:42:04.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.328+0000 7fa351400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa34c102e00 0x7fa34c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:42:04.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.328+0000 7fa351400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa34c1037a0 0x7fa34c070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:42:04.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.328+0000 7fa351400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa34c070d90 con 0x7fa34c102e00 2024-03-30T04:42:04.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.328+0000 7fa351400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa34c070ed0 con 0x7fa34c1037a0 2024-03-30T04:42:04.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.328+0000 7fa34a800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa34c102e00 0x7fa34c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:42:04.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.328+0000 7fa34a800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa34c102e00 0x7fa34c0702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49578/0 (socket says 172.21.15.123:49578) 2024-03-30T04:42:04.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.328+0000 7fa34a800700 1 -- 172.21.15.123:0/2750741394 learned_addr learned my addr 172.21.15.123:0/2750741394 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:42:04.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.328+0000 7fa349e00700 1 --2- 172.21.15.123:0/2750741394 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa34c1037a0 0x7fa34c070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:42:04.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.328+0000 7fa34a800700 1 -- 172.21.15.123:0/2750741394 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa34c1037a0 msgr2=0x7fa34c070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:04.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.328+0000 7fa34a800700 1 --2- 172.21.15.123:0/2750741394 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa34c1037a0 0x7fa34c070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:04.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.328+0000 7fa34a800700 1 -- 172.21.15.123:0/2750741394 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa338005040 con 0x7fa34c102e00 2024-03-30T04:42:04.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.328+0000 7fa349e00700 1 --2- 172.21.15.123:0/2750741394 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa34c1037a0 0x7fa34c070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:42:04.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.329+0000 7fa34a800700 1 --2- 172.21.15.123:0/2750741394 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa34c102e00 0x7fa34c0702c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa33c002a50 tx=0x7fa33c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:42:04.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.329+0000 7fa342a00700 1 -- 172.21.15.123:0/2750741394 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa33c007cc0 con 0x7fa34c102e00 2024-03-30T04:42:04.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.329+0000 7fa342a00700 1 -- 172.21.15.123:0/2750741394 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa33c00f040 con 0x7fa34c102e00 2024-03-30T04:42:04.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.329+0000 7fa342a00700 1 -- 172.21.15.123:0/2750741394 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa33c009540 con 0x7fa34c102e00 2024-03-30T04:42:04.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.329+0000 7fa351400700 1 -- 172.21.15.123:0/2750741394 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa34c119e00 con 0x7fa34c102e00 2024-03-30T04:42:04.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.329+0000 7fa351400700 1 -- 172.21.15.123:0/2750741394 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa34c11a280 con 0x7fa34c102e00 2024-03-30T04:42:04.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.330+0000 7fa351400700 1 -- 172.21.15.123:0/2750741394 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa34c0fe550 con 0x7fa34c102e00 2024-03-30T04:42:04.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.331+0000 7fa342a00700 1 -- 172.21.15.123:0/2750741394 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fa33c0096a0 con 0x7fa34c102e00 2024-03-30T04:42:04.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.332+0000 7fa342a00700 1 --2- 172.21.15.123:0/2750741394 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa33006cc30 0x7fa33006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:42:04.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.332+0000 7fa342a00700 1 -- 172.21.15.123:0/2750741394 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa33c08be00 con 0x7fa34c102e00 2024-03-30T04:42:04.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.332+0000 7fa349e00700 1 --2- 172.21.15.123:0/2750741394 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa33006cc30 0x7fa33006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:42:04.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.333+0000 7fa349e00700 1 --2- 172.21.15.123:0/2750741394 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa33006cc30 0x7fa33006f0e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa34c071c00 tx=0x7fa338009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:42:04.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.337+0000 7fa342a00700 1 -- 172.21.15.123:0/2750741394 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa33c055dc0 con 0x7fa34c102e00 2024-03-30T04:42:04.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.559+0000 7fa351400700 1 -- 172.21.15.123:0/2750741394 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa34c0719b0 con 0x7fa33006cc30 2024-03-30T04:42:04.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.561+0000 7fa342a00700 1 -- 172.21.15.123:0/2750741394 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fa34c0719b0 con 0x7fa33006cc30 2024-03-30T04:42:04.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.563+0000 7fa351400700 1 -- 172.21.15.123:0/2750741394 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa33006cc30 msgr2=0x7fa33006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:04.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.563+0000 7fa351400700 1 --2- 172.21.15.123:0/2750741394 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa33006cc30 0x7fa33006f0e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa34c071c00 tx=0x7fa338009e30 comp rx=0 tx=0).stop 2024-03-30T04:42:04.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.563+0000 7fa351400700 1 -- 172.21.15.123:0/2750741394 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa34c102e00 msgr2=0x7fa34c0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:04.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.563+0000 7fa351400700 1 --2- 172.21.15.123:0/2750741394 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa34c102e00 0x7fa34c0702c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa33c002a50 tx=0x7fa33c0046b0 comp rx=0 tx=0).stop 2024-03-30T04:42:04.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.563+0000 7fa351400700 1 -- 172.21.15.123:0/2750741394 shutdown_connections 2024-03-30T04:42:04.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.563+0000 7fa351400700 1 --2- 172.21.15.123:0/2750741394 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa33006cc30 0x7fa33006f0e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:04.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.563+0000 7fa351400700 1 --2- 172.21.15.123:0/2750741394 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa34c102e00 0x7fa34c0702c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:04.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.563+0000 7fa351400700 1 --2- 172.21.15.123:0/2750741394 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa34c1037a0 0x7fa34c070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:04.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.563+0000 7fa351400700 1 -- 172.21.15.123:0/2750741394 >> 172.21.15.123:0/2750741394 conn(0x7fa34c0f89b0 msgr2=0x7fa34c0f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:42:04.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.563+0000 7fa351400700 1 -- 172.21.15.123:0/2750741394 shutdown_connections 2024-03-30T04:42:04.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.563+0000 7fa351400700 1 -- 172.21.15.123:0/2750741394 wait complete. 2024-03-30T04:42:04.576 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:42:04.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.700+0000 7f55dd600700 1 -- 172.21.15.123:0/692361294 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55d81037a0 msgr2=0x7f55d80fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:04.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.700+0000 7f55dd600700 1 --2- 172.21.15.123:0/692361294 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55d81037a0 0x7f55d80fce10 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f55c4004940 tx=0x7f55c4009e30 comp rx=0 tx=0).stop 2024-03-30T04:42:04.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.700+0000 7f55dd600700 1 -- 172.21.15.123:0/692361294 shutdown_connections 2024-03-30T04:42:04.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.700+0000 7f55dd600700 1 --2- 172.21.15.123:0/692361294 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55d81037a0 0x7f55d80fce10 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:04.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.700+0000 7f55dd600700 1 --2- 172.21.15.123:0/692361294 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f55d8102e00 0x7f55d81031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:04.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.700+0000 7f55dd600700 1 -- 172.21.15.123:0/692361294 >> 172.21.15.123:0/692361294 conn(0x7f55d80f89b0 msgr2=0x7f55d80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:42:04.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.700+0000 7f55dd600700 1 -- 172.21.15.123:0/692361294 shutdown_connections 2024-03-30T04:42:04.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.700+0000 7f55dd600700 1 -- 172.21.15.123:0/692361294 wait complete. 2024-03-30T04:42:04.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.701+0000 7f55dd600700 1 Processor -- start 2024-03-30T04:42:04.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.701+0000 7f55dd600700 1 -- start start 2024-03-30T04:42:04.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.701+0000 7f55dd600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f55d8102e00 0x7f55d80702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:42:04.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.701+0000 7f55dd600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55d81037a0 0x7f55d8070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:42:04.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.701+0000 7f55dd600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55d8070d90 con 0x7f55d81037a0 2024-03-30T04:42:04.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.701+0000 7f55dd600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55d8070ed0 con 0x7f55d8102e00 2024-03-30T04:42:04.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.702+0000 7f55d6000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55d81037a0 0x7f55d8070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:42:04.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.702+0000 7f55d6000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55d81037a0 0x7f55d8070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49596/0 (socket says 172.21.15.123:49596) 2024-03-30T04:42:04.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.702+0000 7f55d6000700 1 -- 172.21.15.123:0/2827337420 learned_addr learned my addr 172.21.15.123:0/2827337420 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:42:04.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.702+0000 7f55d6a00700 1 --2- 172.21.15.123:0/2827337420 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f55d8102e00 0x7f55d80702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:42:04.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.702+0000 7f55d6000700 1 -- 172.21.15.123:0/2827337420 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f55d8102e00 msgr2=0x7f55d80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:04.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.702+0000 7f55d6000700 1 --2- 172.21.15.123:0/2827337420 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f55d8102e00 0x7f55d80702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:04.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.702+0000 7f55d6000700 1 -- 172.21.15.123:0/2827337420 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55c4005040 con 0x7f55d81037a0 2024-03-30T04:42:04.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.702+0000 7f55d6a00700 1 --2- 172.21.15.123:0/2827337420 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f55d8102e00 0x7f55d80702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:42:04.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.702+0000 7f55d6000700 1 --2- 172.21.15.123:0/2827337420 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55d81037a0 0x7f55d8070800 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f55c4009fd0 tx=0x7f55c4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:42:04.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.703+0000 7f55cea00700 1 -- 172.21.15.123:0/2827337420 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f55c400e030 con 0x7f55d81037a0 2024-03-30T04:42:04.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.703+0000 7f55dd600700 1 -- 172.21.15.123:0/2827337420 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55d8071120 con 0x7f55d81037a0 2024-03-30T04:42:04.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.703+0000 7f55cea00700 1 -- 172.21.15.123:0/2827337420 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f55c4013e00 con 0x7f55d81037a0 2024-03-30T04:42:04.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.703+0000 7f55cea00700 1 -- 172.21.15.123:0/2827337420 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f55c401ccd0 con 0x7f55d81037a0 2024-03-30T04:42:04.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.703+0000 7f55dd600700 1 -- 172.21.15.123:0/2827337420 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55d811a1a0 con 0x7f55d81037a0 2024-03-30T04:42:04.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.704+0000 7f55dd600700 1 -- 172.21.15.123:0/2827337420 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55d80fe550 con 0x7f55d81037a0 2024-03-30T04:42:04.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.705+0000 7f55cea00700 1 -- 172.21.15.123:0/2827337420 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f55c401c500 con 0x7f55d81037a0 2024-03-30T04:42:04.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.706+0000 7f55cea00700 1 --2- 172.21.15.123:0/2827337420 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f55bc06cc30 0x7f55bc06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:42:04.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.706+0000 7f55cea00700 1 -- 172.21.15.123:0/2827337420 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f55c408d930 con 0x7f55d81037a0 2024-03-30T04:42:04.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.706+0000 7f55d6a00700 1 --2- 172.21.15.123:0/2827337420 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f55bc06cc30 0x7f55bc06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:42:04.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.707+0000 7f55d6a00700 1 --2- 172.21.15.123:0/2827337420 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f55bc06cc30 0x7f55bc06f0e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f55c8009fd0 tx=0x7f55c800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:42:04.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.711+0000 7f55cea00700 1 -- 172.21.15.123:0/2827337420 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f55c40578f0 con 0x7f55d81037a0 2024-03-30T04:42:04.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:42:04 smithi135 ceph-mon[106011]: pgmap v326: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:04.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.931+0000 7f55dd600700 1 -- 172.21.15.123:0/2827337420 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f55d8109e60 con 0x7f55bc06cc30 2024-03-30T04:42:04.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.933+0000 7f55cea00700 1 -- 172.21.15.123:0/2827337420 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f55c8009420 con 0x7f55bc06cc30 2024-03-30T04:42:04.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.935+0000 7f55dd600700 1 -- 172.21.15.123:0/2827337420 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f55bc06cc30 msgr2=0x7f55bc06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:04.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.935+0000 7f55dd600700 1 --2- 172.21.15.123:0/2827337420 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f55bc06cc30 0x7f55bc06f0e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f55c8009fd0 tx=0x7f55c800b040 comp rx=0 tx=0).stop 2024-03-30T04:42:04.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.935+0000 7f55dd600700 1 -- 172.21.15.123:0/2827337420 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55d81037a0 msgr2=0x7f55d8070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:04.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.935+0000 7f55dd600700 1 --2- 172.21.15.123:0/2827337420 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55d81037a0 0x7f55d8070800 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f55c4009fd0 tx=0x7f55c4009e30 comp rx=0 tx=0).stop 2024-03-30T04:42:04.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.935+0000 7f55dd600700 1 -- 172.21.15.123:0/2827337420 shutdown_connections 2024-03-30T04:42:04.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.935+0000 7f55dd600700 1 --2- 172.21.15.123:0/2827337420 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f55bc06cc30 0x7f55bc06f0e0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:04.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.935+0000 7f55dd600700 1 --2- 172.21.15.123:0/2827337420 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f55d8102e00 0x7f55d80702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:04.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.935+0000 7f55dd600700 1 --2- 172.21.15.123:0/2827337420 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55d81037a0 0x7f55d8070800 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:04.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.935+0000 7f55dd600700 1 -- 172.21.15.123:0/2827337420 >> 172.21.15.123:0/2827337420 conn(0x7f55d80f89b0 msgr2=0x7f55d80f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:42:04.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.935+0000 7f55dd600700 1 -- 172.21.15.123:0/2827337420 shutdown_connections 2024-03-30T04:42:04.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:04.935+0000 7f55dd600700 1 -- 172.21.15.123:0/2827337420 wait complete. 2024-03-30T04:42:04.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:42:04 smithi123 ceph-mon[139369]: pgmap v326: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:05.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.070+0000 7f2cbbc00700 1 -- 172.21.15.123:0/2793194841 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2cb4070c10 msgr2=0x7f2cb4109150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:05.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.070+0000 7f2cbbc00700 1 --2- 172.21.15.123:0/2793194841 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2cb4070c10 0x7f2cb4109150 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2ca0004970 tx=0x7f2ca0009e30 comp rx=0 tx=0).stop 2024-03-30T04:42:05.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.070+0000 7f2cbbc00700 1 -- 172.21.15.123:0/2793194841 shutdown_connections 2024-03-30T04:42:05.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.070+0000 7f2cbbc00700 1 --2- 172.21.15.123:0/2793194841 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2cb4070c10 0x7f2cb4109150 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:05.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.070+0000 7f2cbbc00700 1 --2- 172.21.15.123:0/2793194841 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2cb4070300 0x7f2cb40706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:05.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.070+0000 7f2cbbc00700 1 -- 172.21.15.123:0/2793194841 >> 172.21.15.123:0/2793194841 conn(0x7f2cb40f8950 msgr2=0x7f2cb40fad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:42:05.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.070+0000 7f2cbbc00700 1 -- 172.21.15.123:0/2793194841 shutdown_connections 2024-03-30T04:42:05.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.070+0000 7f2cbbc00700 1 -- 172.21.15.123:0/2793194841 wait complete. 2024-03-30T04:42:05.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.071+0000 7f2cbbc00700 1 Processor -- start 2024-03-30T04:42:05.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.071+0000 7f2cbbc00700 1 -- start start 2024-03-30T04:42:05.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.071+0000 7f2cbbc00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2cb4070300 0x7f2cb4115a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:42:05.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.071+0000 7f2cbbc00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2cb4070c10 0x7f2cb4115f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:42:05.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.071+0000 7f2cbbc00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb41165d0 con 0x7f2cb4070300 2024-03-30T04:42:05.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.071+0000 7f2cbbc00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb410fbb0 con 0x7f2cb4070c10 2024-03-30T04:42:05.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.071+0000 7f2cb9200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2cb4070300 0x7f2cb4115a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:42:05.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.071+0000 7f2cb9200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2cb4070300 0x7f2cb4115a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49604/0 (socket says 172.21.15.123:49604) 2024-03-30T04:42:05.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.071+0000 7f2cb9200700 1 -- 172.21.15.123:0/592918014 learned_addr learned my addr 172.21.15.123:0/592918014 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:42:05.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.072+0000 7f2cb3e00700 1 --2- 172.21.15.123:0/592918014 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2cb4070c10 0x7f2cb4115f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:42:05.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.072+0000 7f2cb9200700 1 -- 172.21.15.123:0/592918014 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2cb4070c10 msgr2=0x7f2cb4115f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:05.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.072+0000 7f2cb9200700 1 --2- 172.21.15.123:0/592918014 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2cb4070c10 0x7f2cb4115f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:05.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.072+0000 7f2cb9200700 1 -- 172.21.15.123:0/592918014 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca0005040 con 0x7f2cb4070300 2024-03-30T04:42:05.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.072+0000 7f2cb9200700 1 --2- 172.21.15.123:0/592918014 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2cb4070300 0x7f2cb4115a40 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f2ca800eb40 tx=0x7f2ca800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:42:05.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.072+0000 7f2cb0c00700 1 -- 172.21.15.123:0/592918014 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ca8007cc0 con 0x7f2cb4070300 2024-03-30T04:42:05.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.072+0000 7f2cbbc00700 1 -- 172.21.15.123:0/592918014 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cb410fe30 con 0x7f2cb4070300 2024-03-30T04:42:05.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.072+0000 7f2cb0c00700 1 -- 172.21.15.123:0/592918014 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2ca800f040 con 0x7f2cb4070300 2024-03-30T04:42:05.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.072+0000 7f2cb0c00700 1 -- 172.21.15.123:0/592918014 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ca8009570 con 0x7f2cb4070300 2024-03-30T04:42:05.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.072+0000 7f2cbbc00700 1 -- 172.21.15.123:0/592918014 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cb4110370 con 0x7f2cb4070300 2024-03-30T04:42:05.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.073+0000 7f2cbbc00700 1 -- 172.21.15.123:0/592918014 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cb4071a40 con 0x7f2cb4070300 2024-03-30T04:42:05.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.074+0000 7f2cb0c00700 1 -- 172.21.15.123:0/592918014 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f2ca80096d0 con 0x7f2cb4070300 2024-03-30T04:42:05.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.075+0000 7f2cb0c00700 1 --2- 172.21.15.123:0/592918014 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2c9c06cc20 0x7f2c9c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:42:05.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.075+0000 7f2cb0c00700 1 -- 172.21.15.123:0/592918014 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f2ca808bda0 con 0x7f2cb4070300 2024-03-30T04:42:05.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.075+0000 7f2cb3e00700 1 --2- 172.21.15.123:0/592918014 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2c9c06cc20 0x7f2c9c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:42:05.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.076+0000 7f2cb3e00700 1 --2- 172.21.15.123:0/592918014 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2c9c06cc20 0x7f2c9c06f0d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2ca0004e30 tx=0x7f2ca0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:42:05.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.079+0000 7f2cb0c00700 1 -- 172.21.15.123:0/592918014 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2ca8059820 con 0x7f2cb4070300 2024-03-30T04:42:05.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.292+0000 7f2cbbc00700 1 -- 172.21.15.123:0/592918014 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2cb4111100 con 0x7f2c9c06cc20 2024-03-30T04:42:05.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.302+0000 7f2cb0c00700 1 -- 172.21.15.123:0/592918014 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f2cb4111100 con 0x7f2c9c06cc20 2024-03-30T04:42:05.299 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:42:05.299 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (11m) 4m ago 23m 15.0M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:42:05.300 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (8m) 4m ago 24m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T04:42:05.300 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (8m) 3m ago 22m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T04:42:05.300 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (10m) 4m ago 23m 74.0M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T04:42:05.300 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (4m) 4m ago 17m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T04:42:05.300 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (4m) 4m ago 17m 10.9M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T04:42:05.300 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (17m) 3m ago 17m 10.7M - 17.2.7-797-g5c024335 f77085f6aab7 0dc41326818c 2024-03-30T04:42:05.300 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (3m) 3m ago 17m 9747k - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T04:42:05.300 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (13m) 4m ago 25m 506M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:42:05.300 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (12m) 3m ago 22m 450M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:42:05.300 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (9m) 4m ago 25m 53.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T04:42:05.300 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (9m) 3m ago 22m 42.6M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T04:42:05.300 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (11m) 4m ago 23m 8052k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:42:05.300 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (11m) 3m ago 22m 7880k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:42:05.300 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (8m) 4m ago 21m 104M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T04:42:05.301 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (7m) 4m ago 21m 93.4M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T04:42:05.301 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (6m) 4m ago 20m 86.3M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T04:42:05.301 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (6m) 3m ago 20m 97.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T04:42:05.301 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (5m) 3m ago 19m 97.9M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T04:42:05.301 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (4m) 3m ago 19m 85.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T04:42:05.301 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (11m) 4m ago 23m 53.7M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:42:05.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.304+0000 7f2cbbc00700 1 -- 172.21.15.123:0/592918014 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2c9c06cc20 msgr2=0x7f2c9c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:05.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.304+0000 7f2cbbc00700 1 --2- 172.21.15.123:0/592918014 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2c9c06cc20 0x7f2c9c06f0d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2ca0004e30 tx=0x7f2ca0009e30 comp rx=0 tx=0).stop 2024-03-30T04:42:05.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.304+0000 7f2cbbc00700 1 -- 172.21.15.123:0/592918014 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2cb4070300 msgr2=0x7f2cb4115a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:05.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.304+0000 7f2cbbc00700 1 --2- 172.21.15.123:0/592918014 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2cb4070300 0x7f2cb4115a40 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f2ca800eb40 tx=0x7f2ca800b040 comp rx=0 tx=0).stop 2024-03-30T04:42:05.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.304+0000 7f2cbbc00700 1 -- 172.21.15.123:0/592918014 shutdown_connections 2024-03-30T04:42:05.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.304+0000 7f2cbbc00700 1 --2- 172.21.15.123:0/592918014 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2c9c06cc20 0x7f2c9c06f0d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:05.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.304+0000 7f2cbbc00700 1 --2- 172.21.15.123:0/592918014 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2cb4070300 0x7f2cb4115a40 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:05.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.304+0000 7f2cbbc00700 1 --2- 172.21.15.123:0/592918014 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2cb4070c10 0x7f2cb4115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:05.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.304+0000 7f2cbbc00700 1 -- 172.21.15.123:0/592918014 >> 172.21.15.123:0/592918014 conn(0x7f2cb40f8950 msgr2=0x7f2cb4105a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:42:05.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.305+0000 7f2cbbc00700 1 -- 172.21.15.123:0/592918014 shutdown_connections 2024-03-30T04:42:05.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.305+0000 7f2cbbc00700 1 -- 172.21.15.123:0/592918014 wait complete. 2024-03-30T04:42:05.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.437+0000 7fccd3200700 1 -- 172.21.15.123:0/477588673 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcccc102e00 msgr2=0x7fcccc1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:05.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.437+0000 7fccd3200700 1 --2- 172.21.15.123:0/477588673 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcccc102e00 0x7fcccc1031d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fccbc004940 tx=0x7fccbc009e30 comp rx=0 tx=0).stop 2024-03-30T04:42:05.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.437+0000 7fccd3200700 1 -- 172.21.15.123:0/477588673 shutdown_connections 2024-03-30T04:42:05.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.437+0000 7fccd3200700 1 --2- 172.21.15.123:0/477588673 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcccc1037a0 0x7fcccc0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:05.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.437+0000 7fccd3200700 1 --2- 172.21.15.123:0/477588673 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcccc102e00 0x7fcccc1031d0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:05.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.437+0000 7fccd3200700 1 -- 172.21.15.123:0/477588673 >> 172.21.15.123:0/477588673 conn(0x7fcccc0f89b0 msgr2=0x7fcccc0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:42:05.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.437+0000 7fccd3200700 1 -- 172.21.15.123:0/477588673 shutdown_connections 2024-03-30T04:42:05.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.437+0000 7fccd3200700 1 -- 172.21.15.123:0/477588673 wait complete. 2024-03-30T04:42:05.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.438+0000 7fccd3200700 1 Processor -- start 2024-03-30T04:42:05.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.439+0000 7fccd3200700 1 -- start start 2024-03-30T04:42:05.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.439+0000 7fccd3200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcccc102e00 0x7fcccc0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:42:05.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.439+0000 7fccd3200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcccc1037a0 0x7fcccc070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:42:05.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.439+0000 7fccd3200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc070d90 con 0x7fcccc102e00 2024-03-30T04:42:05.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.439+0000 7fccd3200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc070ed0 con 0x7fcccc1037a0 2024-03-30T04:42:05.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.439+0000 7fcccbe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcccc102e00 0x7fcccc0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:42:05.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.440+0000 7fcccbe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcccc102e00 0x7fcccc0702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49626/0 (socket says 172.21.15.123:49626) 2024-03-30T04:42:05.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.440+0000 7fcccbe00700 1 -- 172.21.15.123:0/4011541068 learned_addr learned my addr 172.21.15.123:0/4011541068 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:42:05.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.440+0000 7fcccb400700 1 --2- 172.21.15.123:0/4011541068 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcccc1037a0 0x7fcccc070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:42:05.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.440+0000 7fcccbe00700 1 -- 172.21.15.123:0/4011541068 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcccc1037a0 msgr2=0x7fcccc070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:05.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.440+0000 7fcccbe00700 1 --2- 172.21.15.123:0/4011541068 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcccc1037a0 0x7fcccc070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:05.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.440+0000 7fcccbe00700 1 -- 172.21.15.123:0/4011541068 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccbc005040 con 0x7fcccc102e00 2024-03-30T04:42:05.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.440+0000 7fcccbe00700 1 --2- 172.21.15.123:0/4011541068 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcccc102e00 0x7fcccc0702c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fccbc009360 tx=0x7fccbc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:42:05.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.440+0000 7fccc3e00700 1 -- 172.21.15.123:0/4011541068 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fccbc00e040 con 0x7fcccc102e00 2024-03-30T04:42:05.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.440+0000 7fccc3e00700 1 -- 172.21.15.123:0/4011541068 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fccbc013420 con 0x7fcccc102e00 2024-03-30T04:42:05.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.440+0000 7fccd3200700 1 -- 172.21.15.123:0/4011541068 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcccc071120 con 0x7fcccc102e00 2024-03-30T04:42:05.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.440+0000 7fccc3e00700 1 -- 172.21.15.123:0/4011541068 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fccbc01c5f0 con 0x7fcccc102e00 2024-03-30T04:42:05.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.441+0000 7fccd3200700 1 -- 172.21.15.123:0/4011541068 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcccc11a270 con 0x7fcccc102e00 2024-03-30T04:42:05.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.442+0000 7fccd3200700 1 -- 172.21.15.123:0/4011541068 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcccc0fe550 con 0x7fcccc102e00 2024-03-30T04:42:05.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.442+0000 7fccc3e00700 1 -- 172.21.15.123:0/4011541068 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fccbc013590 con 0x7fcccc102e00 2024-03-30T04:42:05.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.443+0000 7fccc3e00700 1 --2- 172.21.15.123:0/4011541068 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fccb006cc30 0x7fccb006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:42:05.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.443+0000 7fccc3e00700 1 -- 172.21.15.123:0/4011541068 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fccbc08d340 con 0x7fcccc102e00 2024-03-30T04:42:05.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.444+0000 7fcccb400700 1 --2- 172.21.15.123:0/4011541068 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fccb006cc30 0x7fccb006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:42:05.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.444+0000 7fcccb400700 1 --2- 172.21.15.123:0/4011541068 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fccb006cc30 0x7fccb006f0e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fccb8009fd0 tx=0x7fccb8009f20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:42:05.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.448+0000 7fccc3e00700 1 -- 172.21.15.123:0/4011541068 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fccbc057300 con 0x7fcccc102e00 2024-03-30T04:42:05.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.732+0000 7fccd3200700 1 -- 172.21.15.123:0/4011541068 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fcccc0719b0 con 0x7fcccc102e00 2024-03-30T04:42:05.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.732+0000 7fccc3e00700 1 -- 172.21.15.123:0/4011541068 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7fccbc05a920 con 0x7fcccc102e00 2024-03-30T04:42:05.730 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:42:05.730 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:42:05.730 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:42:05.730 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:42:05.730 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:42:05.730 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:42:05.730 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:42:05.730 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:42:05.730 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T04:42:05.731 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:42:05.731 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:42:05.731 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:42:05.731 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 3 2024-03-30T04:42:05.731 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:42:05.731 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:42:05.731 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:42:05.731 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 13 2024-03-30T04:42:05.731 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:42:05.731 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:42:05.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.735+0000 7fccd3200700 1 -- 172.21.15.123:0/4011541068 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fccb006cc30 msgr2=0x7fccb006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:05.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.735+0000 7fccd3200700 1 --2- 172.21.15.123:0/4011541068 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fccb006cc30 0x7fccb006f0e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fccb8009fd0 tx=0x7fccb8009f20 comp rx=0 tx=0).stop 2024-03-30T04:42:05.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.735+0000 7fccd3200700 1 -- 172.21.15.123:0/4011541068 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcccc102e00 msgr2=0x7fcccc0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:05.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.735+0000 7fccd3200700 1 --2- 172.21.15.123:0/4011541068 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcccc102e00 0x7fcccc0702c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fccbc009360 tx=0x7fccbc009e30 comp rx=0 tx=0).stop 2024-03-30T04:42:05.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.735+0000 7fccd3200700 1 -- 172.21.15.123:0/4011541068 shutdown_connections 2024-03-30T04:42:05.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.735+0000 7fccd3200700 1 --2- 172.21.15.123:0/4011541068 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fccb006cc30 0x7fccb006f0e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:05.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.735+0000 7fccd3200700 1 --2- 172.21.15.123:0/4011541068 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcccc102e00 0x7fcccc0702c0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:05.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.735+0000 7fccd3200700 1 --2- 172.21.15.123:0/4011541068 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcccc1037a0 0x7fcccc070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:05.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.735+0000 7fccd3200700 1 -- 172.21.15.123:0/4011541068 >> 172.21.15.123:0/4011541068 conn(0x7fcccc0f89b0 msgr2=0x7fcccc0f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:42:05.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.735+0000 7fccd3200700 1 -- 172.21.15.123:0/4011541068 shutdown_connections 2024-03-30T04:42:05.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.735+0000 7fccd3200700 1 -- 172.21.15.123:0/4011541068 wait complete. 2024-03-30T04:42:05.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:42:05 smithi135 ceph-mon[106011]: from='client.34654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:42:05.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:42:05 smithi135 ceph-mon[106011]: from='client.34658 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:42:05.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.867+0000 7f01cda00700 1 -- 172.21.15.123:0/1911694694 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01c8102e00 msgr2=0x7f01c81031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:05.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.867+0000 7f01cda00700 1 --2- 172.21.15.123:0/1911694694 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01c8102e00 0x7f01c81031d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f01b8004940 tx=0x7f01b8009e30 comp rx=0 tx=0).stop 2024-03-30T04:42:05.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.867+0000 7f01cda00700 1 -- 172.21.15.123:0/1911694694 shutdown_connections 2024-03-30T04:42:05.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.867+0000 7f01cda00700 1 --2- 172.21.15.123:0/1911694694 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f01c81037a0 0x7f01c80fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:05.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.867+0000 7f01cda00700 1 --2- 172.21.15.123:0/1911694694 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01c8102e00 0x7f01c81031d0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:05.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.867+0000 7f01cda00700 1 -- 172.21.15.123:0/1911694694 >> 172.21.15.123:0/1911694694 conn(0x7f01c80f89b0 msgr2=0x7f01c80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:42:05.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.867+0000 7f01cda00700 1 -- 172.21.15.123:0/1911694694 shutdown_connections 2024-03-30T04:42:05.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.867+0000 7f01cda00700 1 -- 172.21.15.123:0/1911694694 wait complete. 2024-03-30T04:42:05.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.868+0000 7f01cda00700 1 Processor -- start 2024-03-30T04:42:05.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.868+0000 7f01cda00700 1 -- start start 2024-03-30T04:42:05.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.868+0000 7f01cda00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01c8102e00 0x7f01c8115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:42:05.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.868+0000 7f01cda00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f01c81037a0 0x7f01c8116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:42:05.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.868+0000 7f01cda00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c8116880 con 0x7f01c8102e00 2024-03-30T04:42:05.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.868+0000 7f01cda00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c810fc50 con 0x7f01c81037a0 2024-03-30T04:42:05.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.869+0000 7f01c6a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01c8102e00 0x7f01c8115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:42:05.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.869+0000 7f01c6a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01c8102e00 0x7f01c8115cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49638/0 (socket says 172.21.15.123:49638) 2024-03-30T04:42:05.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.869+0000 7f01c6a00700 1 -- 172.21.15.123:0/80407327 learned_addr learned my addr 172.21.15.123:0/80407327 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:42:05.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.869+0000 7f01c6a00700 1 -- 172.21.15.123:0/80407327 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f01c81037a0 msgr2=0x7f01c8116230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:05.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.869+0000 7f01c6a00700 1 --2- 172.21.15.123:0/80407327 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f01c81037a0 0x7f01c8116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:05.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.869+0000 7f01c6a00700 1 -- 172.21.15.123:0/80407327 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b8005040 con 0x7f01c8102e00 2024-03-30T04:42:05.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.869+0000 7f01c6a00700 1 --2- 172.21.15.123:0/80407327 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01c8102e00 0x7f01c8115cf0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f01b8009390 tx=0x7f01b8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:42:05.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.870+0000 7f01bea00700 1 -- 172.21.15.123:0/80407327 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f01b800e040 con 0x7f01c8102e00 2024-03-30T04:42:05.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.870+0000 7f01bea00700 1 -- 172.21.15.123:0/80407327 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f01b8013420 con 0x7f01c8102e00 2024-03-30T04:42:05.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.870+0000 7f01cda00700 1 -- 172.21.15.123:0/80407327 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01c810fed0 con 0x7f01c8102e00 2024-03-30T04:42:05.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.870+0000 7f01bea00700 1 -- 172.21.15.123:0/80407327 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f01b801c630 con 0x7f01c8102e00 2024-03-30T04:42:05.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.870+0000 7f01cda00700 1 -- 172.21.15.123:0/80407327 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01c8110490 con 0x7f01c8102e00 2024-03-30T04:42:05.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.871+0000 7f01cda00700 1 -- 172.21.15.123:0/80407327 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01c80fe550 con 0x7f01c8102e00 2024-03-30T04:42:05.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.872+0000 7f01bea00700 1 -- 172.21.15.123:0/80407327 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f01b80139e0 con 0x7f01c8102e00 2024-03-30T04:42:05.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.873+0000 7f01bea00700 1 --2- 172.21.15.123:0/80407327 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f01ac06cc30 0x7f01ac06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:42:05.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.873+0000 7f01bea00700 1 -- 172.21.15.123:0/80407327 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f01b808d320 con 0x7f01c8102e00 2024-03-30T04:42:05.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.873+0000 7f01c6000700 1 --2- 172.21.15.123:0/80407327 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f01ac06cc30 0x7f01ac06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:42:05.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.874+0000 7f01c6000700 1 --2- 172.21.15.123:0/80407327 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f01ac06cc30 0x7f01ac06f0e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f01c8111590 tx=0x7f01b4009ce0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:42:05.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:05.878+0000 7f01bea00700 1 -- 172.21.15.123:0/80407327 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f01b805ae20 con 0x7f01c8102e00 2024-03-30T04:42:05.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:42:05 smithi123 ceph-mon[139369]: from='client.34654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:42:05.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:42:05 smithi123 ceph-mon[139369]: from='client.34658 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:42:06.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.124+0000 7f01cda00700 1 -- 172.21.15.123:0/80407327 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f01c804bed0 con 0x7f01c8102e00 2024-03-30T04:42:06.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.124+0000 7f01bea00700 1 -- 172.21.15.123:0/80407327 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f01b805a9b0 con 0x7f01c8102e00 2024-03-30T04:42:06.122 INFO:teuthology.orchestra.run.smithi123.stdout:e32 2024-03-30T04:42:06.122 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:42:06.122 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:42:06.122 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:42:06.122 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:42:06.122 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:42:06.122 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:42:06.123 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T04:42:06.123 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T04:42:06.123 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:42:06.123 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T04:42:06.123 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:42:06.123 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:42:06.123 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:42:06.123 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:42:06.123 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:42:06.123 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T04:42:06.123 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:42:06.123 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:42:06.123 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T04:42:06.123 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:42:06.123 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T04:42:06.124 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T04:42:06.124 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T04:42:06.124 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T04:42:06.124 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:42:06.124 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:42:06.124 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:42:06.124 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:42:06.124 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:42:06.124 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:42:06.124 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T04:42:06.124 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T04:42:06.124 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:42:06.124 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:42:06.124 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T04:42:06.124 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:42:06.124 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:42:06.125 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:42:06.125 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44315} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/1741352336,v1:172.21.15.135:6827/1741352336] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:42:06.125 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:42:06.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.127+0000 7f01cda00700 1 -- 172.21.15.123:0/80407327 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f01ac06cc30 msgr2=0x7f01ac06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:06.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.127+0000 7f01cda00700 1 --2- 172.21.15.123:0/80407327 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f01ac06cc30 0x7f01ac06f0e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f01c8111590 tx=0x7f01b4009ce0 comp rx=0 tx=0).stop 2024-03-30T04:42:06.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.127+0000 7f01cda00700 1 -- 172.21.15.123:0/80407327 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01c8102e00 msgr2=0x7f01c8115cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:06.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.127+0000 7f01cda00700 1 --2- 172.21.15.123:0/80407327 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01c8102e00 0x7f01c8115cf0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f01b8009390 tx=0x7f01b8009e30 comp rx=0 tx=0).stop 2024-03-30T04:42:06.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.127+0000 7f01cda00700 1 -- 172.21.15.123:0/80407327 shutdown_connections 2024-03-30T04:42:06.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.127+0000 7f01cda00700 1 --2- 172.21.15.123:0/80407327 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f01ac06cc30 0x7f01ac06f0e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:06.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.127+0000 7f01cda00700 1 --2- 172.21.15.123:0/80407327 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01c8102e00 0x7f01c8115cf0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:06.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.127+0000 7f01cda00700 1 --2- 172.21.15.123:0/80407327 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f01c81037a0 0x7f01c8116230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:06.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.127+0000 7f01cda00700 1 -- 172.21.15.123:0/80407327 >> 172.21.15.123:0/80407327 conn(0x7f01c80f89b0 msgr2=0x7f01c80706d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:42:06.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.127+0000 7f01cda00700 1 -- 172.21.15.123:0/80407327 shutdown_connections 2024-03-30T04:42:06.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.127+0000 7f01cda00700 1 -- 172.21.15.123:0/80407327 wait complete. 2024-03-30T04:42:06.126 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 32 2024-03-30T04:42:06.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.261+0000 7f352ac00700 1 -- 172.21.15.123:0/3188624951 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35241037a0 msgr2=0x7f35240fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:06.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.261+0000 7f352ac00700 1 --2- 172.21.15.123:0/3188624951 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35241037a0 0x7f35240fce10 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f3510004940 tx=0x7f3510009e30 comp rx=0 tx=0).stop 2024-03-30T04:42:06.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.261+0000 7f352ac00700 1 -- 172.21.15.123:0/3188624951 shutdown_connections 2024-03-30T04:42:06.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.261+0000 7f352ac00700 1 --2- 172.21.15.123:0/3188624951 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35241037a0 0x7f35240fce10 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:06.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.261+0000 7f352ac00700 1 --2- 172.21.15.123:0/3188624951 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3524102e00 0x7f35241031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:06.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.261+0000 7f352ac00700 1 -- 172.21.15.123:0/3188624951 >> 172.21.15.123:0/3188624951 conn(0x7f35240f89b0 msgr2=0x7f35240fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:42:06.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.261+0000 7f352ac00700 1 -- 172.21.15.123:0/3188624951 shutdown_connections 2024-03-30T04:42:06.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.261+0000 7f352ac00700 1 -- 172.21.15.123:0/3188624951 wait complete. 2024-03-30T04:42:06.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.262+0000 7f352ac00700 1 Processor -- start 2024-03-30T04:42:06.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.262+0000 7f352ac00700 1 -- start start 2024-03-30T04:42:06.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.262+0000 7f352ac00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3524102e00 0x7f3524115d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:42:06.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.262+0000 7f352ac00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f35241037a0 0x7f3524116240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:42:06.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.262+0000 7f352ac00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3524116890 con 0x7f3524102e00 2024-03-30T04:42:06.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.262+0000 7f352ac00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f352410fc60 con 0x7f35241037a0 2024-03-30T04:42:06.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.262+0000 7f3523e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3524102e00 0x7f3524115d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:42:06.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.262+0000 7f3523e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3524102e00 0x7f3524115d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49664/0 (socket says 172.21.15.123:49664) 2024-03-30T04:42:06.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.263+0000 7f3523e00700 1 -- 172.21.15.123:0/4183870232 learned_addr learned my addr 172.21.15.123:0/4183870232 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:42:06.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.263+0000 7f3523400700 1 --2- 172.21.15.123:0/4183870232 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f35241037a0 0x7f3524116240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:42:06.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.263+0000 7f3523e00700 1 -- 172.21.15.123:0/4183870232 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f35241037a0 msgr2=0x7f3524116240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:06.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.263+0000 7f3523e00700 1 --2- 172.21.15.123:0/4183870232 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f35241037a0 0x7f3524116240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:06.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.263+0000 7f3523e00700 1 -- 172.21.15.123:0/4183870232 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3510005040 con 0x7f3524102e00 2024-03-30T04:42:06.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.263+0000 7f3523e00700 1 --2- 172.21.15.123:0/4183870232 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3524102e00 0x7f3524115d00 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f3514002a50 tx=0x7f35140046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:42:06.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.263+0000 7f351be00700 1 -- 172.21.15.123:0/4183870232 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3514007cc0 con 0x7f3524102e00 2024-03-30T04:42:06.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.263+0000 7f351be00700 1 -- 172.21.15.123:0/4183870232 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f351400f040 con 0x7f3524102e00 2024-03-30T04:42:06.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.263+0000 7f352ac00700 1 -- 172.21.15.123:0/4183870232 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f352410ff40 con 0x7f3524102e00 2024-03-30T04:42:06.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.263+0000 7f351be00700 1 -- 172.21.15.123:0/4183870232 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f351400d9c0 con 0x7f3524102e00 2024-03-30T04:42:06.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.263+0000 7f352ac00700 1 -- 172.21.15.123:0/4183870232 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3524110500 con 0x7f3524102e00 2024-03-30T04:42:06.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.265+0000 7f352ac00700 1 -- 172.21.15.123:0/4183870232 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35240fe550 con 0x7f3524102e00 2024-03-30T04:42:06.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.266+0000 7f351be00700 1 -- 172.21.15.123:0/4183870232 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f351400db20 con 0x7f3524102e00 2024-03-30T04:42:06.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.267+0000 7f351be00700 1 --2- 172.21.15.123:0/4183870232 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f350806cc30 0x7f350806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:42:06.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.267+0000 7f351be00700 1 -- 172.21.15.123:0/4183870232 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f351408bd20 con 0x7f3524102e00 2024-03-30T04:42:06.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.267+0000 7f3523400700 1 --2- 172.21.15.123:0/4183870232 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f350806cc30 0x7f350806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:42:06.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.268+0000 7f3523400700 1 --2- 172.21.15.123:0/4183870232 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f350806cc30 0x7f350806f0e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f35241115a0 tx=0x7f3510009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:42:06.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.272+0000 7f351be00700 1 -- 172.21.15.123:0/4183870232 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3514059770 con 0x7f3524102e00 2024-03-30T04:42:06.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.493+0000 7f352ac00700 1 -- 172.21.15.123:0/4183870232 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f352404bf40 con 0x7f350806cc30 2024-03-30T04:42:06.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.494+0000 7f351be00700 1 -- 172.21.15.123:0/4183870232 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f352404bf40 con 0x7f350806cc30 2024-03-30T04:42:06.492 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:42:06.492 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:42:06.492 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:42:06.492 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T04:42:06.492 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:42:06.492 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T04:42:06.492 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T04:42:06.492 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T04:42:06.492 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T04:42:06.492 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:42:06.492 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "15/21 daemons upgraded", 2024-03-30T04:42:06.493 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T04:42:06.493 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:42:06.493 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:42:06.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.497+0000 7f352ac00700 1 -- 172.21.15.123:0/4183870232 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f350806cc30 msgr2=0x7f350806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:06.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.497+0000 7f352ac00700 1 --2- 172.21.15.123:0/4183870232 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f350806cc30 0x7f350806f0e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f35241115a0 tx=0x7f3510009e30 comp rx=0 tx=0).stop 2024-03-30T04:42:06.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.497+0000 7f352ac00700 1 -- 172.21.15.123:0/4183870232 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3524102e00 msgr2=0x7f3524115d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:06.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.497+0000 7f352ac00700 1 --2- 172.21.15.123:0/4183870232 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3524102e00 0x7f3524115d00 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f3514002a50 tx=0x7f35140046b0 comp rx=0 tx=0).stop 2024-03-30T04:42:06.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.497+0000 7f352ac00700 1 -- 172.21.15.123:0/4183870232 shutdown_connections 2024-03-30T04:42:06.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.497+0000 7f352ac00700 1 --2- 172.21.15.123:0/4183870232 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f350806cc30 0x7f350806f0e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:06.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.497+0000 7f352ac00700 1 --2- 172.21.15.123:0/4183870232 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3524102e00 0x7f3524115d00 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:06.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.497+0000 7f352ac00700 1 --2- 172.21.15.123:0/4183870232 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f35241037a0 0x7f3524116240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:06.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.497+0000 7f352ac00700 1 -- 172.21.15.123:0/4183870232 >> 172.21.15.123:0/4183870232 conn(0x7f35240f89b0 msgr2=0x7f35240f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:42:06.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.497+0000 7f352ac00700 1 -- 172.21.15.123:0/4183870232 shutdown_connections 2024-03-30T04:42:06.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.497+0000 7f352ac00700 1 -- 172.21.15.123:0/4183870232 wait complete. 2024-03-30T04:42:06.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.632+0000 7fcb5aa00700 1 -- 172.21.15.123:0/1568514140 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb540fe830 msgr2=0x7fcb54106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:06.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.632+0000 7fcb5aa00700 1 --2- 172.21.15.123:0/1568514140 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb540fe830 0x7fcb54106f00 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fcb40004940 tx=0x7fcb40009e30 comp rx=0 tx=0).stop 2024-03-30T04:42:06.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.632+0000 7fcb5aa00700 1 -- 172.21.15.123:0/1568514140 shutdown_connections 2024-03-30T04:42:06.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.632+0000 7fcb5aa00700 1 --2- 172.21.15.123:0/1568514140 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb540fe830 0x7fcb54106f00 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:06.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.632+0000 7fcb5aa00700 1 --2- 172.21.15.123:0/1568514140 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcb540fde90 0x7fcb540fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:06.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.632+0000 7fcb5aa00700 1 -- 172.21.15.123:0/1568514140 >> 172.21.15.123:0/1568514140 conn(0x7fcb54072bc0 msgr2=0x7fcb54072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:42:06.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.632+0000 7fcb5aa00700 1 -- 172.21.15.123:0/1568514140 shutdown_connections 2024-03-30T04:42:06.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.632+0000 7fcb5aa00700 1 -- 172.21.15.123:0/1568514140 wait complete. 2024-03-30T04:42:06.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.633+0000 7fcb5aa00700 1 Processor -- start 2024-03-30T04:42:06.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.633+0000 7fcb5aa00700 1 -- start start 2024-03-30T04:42:06.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.633+0000 7fcb5aa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb540fde90 0x7fcb54115d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:42:06.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.633+0000 7fcb5aa00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcb540fe830 0x7fcb54116250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:42:06.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.633+0000 7fcb5aa00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb541168a0 con 0x7fcb540fde90 2024-03-30T04:42:06.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.633+0000 7fcb5aa00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb5410fc70 con 0x7fcb540fe830 2024-03-30T04:42:06.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.633+0000 7fcb53e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb540fde90 0x7fcb54115d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:42:06.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.633+0000 7fcb53e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb540fde90 0x7fcb54115d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49680/0 (socket says 172.21.15.123:49680) 2024-03-30T04:42:06.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.633+0000 7fcb53e00700 1 -- 172.21.15.123:0/1215716772 learned_addr learned my addr 172.21.15.123:0/1215716772 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:42:06.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.634+0000 7fcb53400700 1 --2- 172.21.15.123:0/1215716772 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcb540fe830 0x7fcb54116250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:42:06.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.634+0000 7fcb53e00700 1 -- 172.21.15.123:0/1215716772 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcb540fe830 msgr2=0x7fcb54116250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:06.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.634+0000 7fcb53e00700 1 --2- 172.21.15.123:0/1215716772 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcb540fe830 0x7fcb54116250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:06.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.634+0000 7fcb53e00700 1 -- 172.21.15.123:0/1215716772 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb40005040 con 0x7fcb540fde90 2024-03-30T04:42:06.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.634+0000 7fcb53400700 1 --2- 172.21.15.123:0/1215716772 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcb540fe830 0x7fcb54116250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:42:06.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.634+0000 7fcb53e00700 1 --2- 172.21.15.123:0/1215716772 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb540fde90 0x7fcb54115d10 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fcb44002a50 tx=0x7fcb440046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:42:06.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.634+0000 7fcb4be00700 1 -- 172.21.15.123:0/1215716772 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb44007cc0 con 0x7fcb540fde90 2024-03-30T04:42:06.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.634+0000 7fcb4be00700 1 -- 172.21.15.123:0/1215716772 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcb4400f040 con 0x7fcb540fde90 2024-03-30T04:42:06.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.634+0000 7fcb4be00700 1 -- 172.21.15.123:0/1215716772 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb44009540 con 0x7fcb540fde90 2024-03-30T04:42:06.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.634+0000 7fcb5aa00700 1 -- 172.21.15.123:0/1215716772 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb5410ff50 con 0x7fcb540fde90 2024-03-30T04:42:06.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.634+0000 7fcb5aa00700 1 -- 172.21.15.123:0/1215716772 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb54110490 con 0x7fcb540fde90 2024-03-30T04:42:06.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.635+0000 7fcb5aa00700 1 -- 172.21.15.123:0/1215716772 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb54106a20 con 0x7fcb540fde90 2024-03-30T04:42:06.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.636+0000 7fcb4be00700 1 -- 172.21.15.123:0/1215716772 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fcb440096a0 con 0x7fcb540fde90 2024-03-30T04:42:06.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.637+0000 7fcb4be00700 1 --2- 172.21.15.123:0/1215716772 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fcb3806cc30 0x7fcb3806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:42:06.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.637+0000 7fcb4be00700 1 -- 172.21.15.123:0/1215716772 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fcb4408bde0 con 0x7fcb540fde90 2024-03-30T04:42:06.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.637+0000 7fcb53400700 1 --2- 172.21.15.123:0/1215716772 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fcb3806cc30 0x7fcb3806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:42:06.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.638+0000 7fcb53400700 1 --2- 172.21.15.123:0/1215716772 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fcb3806cc30 0x7fcb3806f0e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fcb4000b3b0 tx=0x7fcb40009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:42:06.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.641+0000 7fcb4be00700 1 -- 172.21.15.123:0/1215716772 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fcb44059830 con 0x7fcb540fde90 2024-03-30T04:42:06.854 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:42:06 smithi135 ceph-mon[106011]: from='client.34662 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:42:06.855 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:42:06 smithi135 ceph-mon[106011]: pgmap v327: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:06.855 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:42:06 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/4011541068' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:42:06.855 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:42:06 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/80407327' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:42:06.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.922+0000 7fcb5aa00700 1 -- 172.21.15.123:0/1215716772 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fcb5404bed0 con 0x7fcb540fde90 2024-03-30T04:42:06.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.923+0000 7fcb4be00700 1 -- 172.21.15.123:0/1215716772 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fcb440593c0 con 0x7fcb540fde90 2024-03-30T04:42:06.921 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T04:42:06.921 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T04:42:06.921 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T04:42:06.921 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T04:42:06.921 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T04:42:06.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.925+0000 7fcb5aa00700 1 -- 172.21.15.123:0/1215716772 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fcb3806cc30 msgr2=0x7fcb3806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:06.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.925+0000 7fcb5aa00700 1 --2- 172.21.15.123:0/1215716772 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fcb3806cc30 0x7fcb3806f0e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fcb4000b3b0 tx=0x7fcb40009e30 comp rx=0 tx=0).stop 2024-03-30T04:42:06.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.925+0000 7fcb5aa00700 1 -- 172.21.15.123:0/1215716772 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb540fde90 msgr2=0x7fcb54115d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:06.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.925+0000 7fcb5aa00700 1 --2- 172.21.15.123:0/1215716772 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb540fde90 0x7fcb54115d10 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fcb44002a50 tx=0x7fcb440046b0 comp rx=0 tx=0).stop 2024-03-30T04:42:06.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.925+0000 7fcb5aa00700 1 -- 172.21.15.123:0/1215716772 shutdown_connections 2024-03-30T04:42:06.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.925+0000 7fcb5aa00700 1 --2- 172.21.15.123:0/1215716772 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fcb3806cc30 0x7fcb3806f0e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:06.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.925+0000 7fcb5aa00700 1 --2- 172.21.15.123:0/1215716772 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb540fde90 0x7fcb54115d10 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:06.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.925+0000 7fcb5aa00700 1 --2- 172.21.15.123:0/1215716772 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcb540fe830 0x7fcb54116250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:06.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.925+0000 7fcb5aa00700 1 -- 172.21.15.123:0/1215716772 >> 172.21.15.123:0/1215716772 conn(0x7fcb54072bc0 msgr2=0x7fcb540fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:42:06.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.925+0000 7fcb5aa00700 1 -- 172.21.15.123:0/1215716772 shutdown_connections 2024-03-30T04:42:06.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:06.925+0000 7fcb5aa00700 1 -- 172.21.15.123:0/1215716772 wait complete. 2024-03-30T04:42:06.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:42:06 smithi123 ceph-mon[139369]: from='client.34662 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:42:06.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:42:06 smithi123 ceph-mon[139369]: pgmap v327: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:06.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:42:06 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/4011541068' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:42:06.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:42:06 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/80407327' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:42:07.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:42:07 smithi135 ceph-mon[106011]: from='client.34674 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:42:07.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:42:07 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/1215716772' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:42:07.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:42:07 smithi123 ceph-mon[139369]: from='client.34674 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:42:07.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:42:07 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/1215716772' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:42:08.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:42:08 smithi135 ceph-mon[106011]: pgmap v328: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:08.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:42:08 smithi123 ceph-mon[139369]: pgmap v328: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:10.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:42:10 smithi123 ceph-mon[139369]: pgmap v329: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:10.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:42:10 smithi135 ceph-mon[106011]: pgmap v329: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:12.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:42:12 smithi135 ceph-mon[106011]: pgmap v330: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:12.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:42:12 smithi123 ceph-mon[139369]: pgmap v330: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:14.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:42:14 smithi135 ceph-mon[106011]: pgmap v331: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:14.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:42:14 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:42:14.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:42:14 smithi123 ceph-mon[139369]: pgmap v331: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:14.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:42:14 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:42:16.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:42:16 smithi135 ceph-mon[106011]: pgmap v332: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:16.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:42:16 smithi123 ceph-mon[139369]: pgmap v332: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:18.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:42:18 smithi135 ceph-mon[106011]: pgmap v333: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:18.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:42:18 smithi123 ceph-mon[139369]: pgmap v333: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:20.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:42:20 smithi123 ceph-mon[139369]: pgmap v334: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:20.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:42:20 smithi135 ceph-mon[106011]: pgmap v334: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:22.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:42:22 smithi135 ceph-mon[106011]: pgmap v335: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:22.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:42:22 smithi123 ceph-mon[139369]: pgmap v335: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:24.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:42:24 smithi135 ceph-mon[106011]: pgmap v336: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:24.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:42:24 smithi123 ceph-mon[139369]: pgmap v336: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:26.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:42:26 smithi135 ceph-mon[106011]: pgmap v337: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:26.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:42:26 smithi123 ceph-mon[139369]: pgmap v337: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:28.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:42:28 smithi135 ceph-mon[106011]: pgmap v338: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:28.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:42:28 smithi123 ceph-mon[139369]: pgmap v338: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:29.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:42:29 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:42:29.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:42:29 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:42:30.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:42:30 smithi135 ceph-mon[106011]: pgmap v339: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:30.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:42:30 smithi123 ceph-mon[139369]: pgmap v339: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:32.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:42:32 smithi135 ceph-mon[106011]: pgmap v340: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:32.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:42:32 smithi123 ceph-mon[139369]: pgmap v340: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:34.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:42:34 smithi135 ceph-mon[106011]: pgmap v341: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:34.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:42:34 smithi123 ceph-mon[139369]: pgmap v341: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:36.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:42:36 smithi135 ceph-mon[106011]: pgmap v342: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:36.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:42:36 smithi123 ceph-mon[139369]: pgmap v342: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:37.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.076+0000 7f3505c00700 1 -- 172.21.15.123:0/1258996983 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35001035f0 msgr2=0x7f35000fcc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:37.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.076+0000 7f3505c00700 1 --2- 172.21.15.123:0/1258996983 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35001035f0 0x7f35000fcc60 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f34ec004940 tx=0x7f34ec009e30 comp rx=0 tx=0).stop 2024-03-30T04:42:37.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.077+0000 7f3505c00700 1 -- 172.21.15.123:0/1258996983 shutdown_connections 2024-03-30T04:42:37.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.077+0000 7f3505c00700 1 --2- 172.21.15.123:0/1258996983 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35001035f0 0x7f35000fcc60 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:37.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.077+0000 7f3505c00700 1 --2- 172.21.15.123:0/1258996983 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3500102c50 0x7f3500103020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:37.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.077+0000 7f3505c00700 1 -- 172.21.15.123:0/1258996983 >> 172.21.15.123:0/1258996983 conn(0x7f35000f87e0 msgr2=0x7f35000fabf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:42:37.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.077+0000 7f3505c00700 1 -- 172.21.15.123:0/1258996983 shutdown_connections 2024-03-30T04:42:37.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.077+0000 7f3505c00700 1 -- 172.21.15.123:0/1258996983 wait complete. 2024-03-30T04:42:37.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.077+0000 7f3505c00700 1 Processor -- start 2024-03-30T04:42:37.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.078+0000 7f3505c00700 1 -- start start 2024-03-30T04:42:37.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.078+0000 7f3505c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3500102c50 0x7f3500113940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:42:37.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.078+0000 7f3505c00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f35001035f0 0x7f3500113e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:42:37.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.078+0000 7f3505c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35001144d0 con 0x7f3500102c50 2024-03-30T04:42:37.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.078+0000 7f3505c00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f350010d8c0 con 0x7f35001035f0 2024-03-30T04:42:37.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.078+0000 7f34ffe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3500102c50 0x7f3500113940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:42:37.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.078+0000 7f34ffe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3500102c50 0x7f3500113940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48242/0 (socket says 172.21.15.123:48242) 2024-03-30T04:42:37.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.078+0000 7f34ffe00700 1 -- 172.21.15.123:0/867186677 learned_addr learned my addr 172.21.15.123:0/867186677 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:42:37.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.079+0000 7f34ff400700 1 --2- 172.21.15.123:0/867186677 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f35001035f0 0x7f3500113e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:42:37.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.079+0000 7f34ffe00700 1 -- 172.21.15.123:0/867186677 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f35001035f0 msgr2=0x7f3500113e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:37.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.079+0000 7f34ffe00700 1 --2- 172.21.15.123:0/867186677 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f35001035f0 0x7f3500113e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:37.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.079+0000 7f34ffe00700 1 -- 172.21.15.123:0/867186677 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34ec005040 con 0x7f3500102c50 2024-03-30T04:42:37.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.079+0000 7f34ffe00700 1 --2- 172.21.15.123:0/867186677 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3500102c50 0x7f3500113940 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f34f000c8c0 tx=0x7f34f00044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:42:37.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.079+0000 7f34f7e00700 1 -- 172.21.15.123:0/867186677 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f34f000a760 con 0x7f3500102c50 2024-03-30T04:42:37.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.079+0000 7f34f7e00700 1 -- 172.21.15.123:0/867186677 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f34f000ad70 con 0x7f3500102c50 2024-03-30T04:42:37.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.079+0000 7f3505c00700 1 -- 172.21.15.123:0/867186677 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f350010dba0 con 0x7f3500102c50 2024-03-30T04:42:37.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.079+0000 7f34f7e00700 1 -- 172.21.15.123:0/867186677 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f34f00113a0 con 0x7f3500102c50 2024-03-30T04:42:37.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.080+0000 7f3505c00700 1 -- 172.21.15.123:0/867186677 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f350010e130 con 0x7f3500102c50 2024-03-30T04:42:37.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.081+0000 7f3505c00700 1 -- 172.21.15.123:0/867186677 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35000fe3a0 con 0x7f3500102c50 2024-03-30T04:42:37.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.082+0000 7f34f7e00700 1 -- 172.21.15.123:0/867186677 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f34f0012400 con 0x7f3500102c50 2024-03-30T04:42:37.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.082+0000 7f34f7e00700 1 --2- 172.21.15.123:0/867186677 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f34e406cc30 0x7f34e406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:42:37.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.083+0000 7f34f7e00700 1 -- 172.21.15.123:0/867186677 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f34f008bfe0 con 0x7f3500102c50 2024-03-30T04:42:37.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.083+0000 7f34ff400700 1 --2- 172.21.15.123:0/867186677 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f34e406cc30 0x7f34e406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:42:37.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.083+0000 7f34ff400700 1 --2- 172.21.15.123:0/867186677 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f34e406cc30 0x7f34e406f0e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f350010eba0 tx=0x7f34ec009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:42:37.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.087+0000 7f34f7e00700 1 -- 172.21.15.123:0/867186677 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f34f0059af0 con 0x7f3500102c50 2024-03-30T04:42:37.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.308+0000 7f3505c00700 1 -- 172.21.15.123:0/867186677 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3500109be0 con 0x7f34e406cc30 2024-03-30T04:42:37.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.310+0000 7f34f7e00700 1 -- 172.21.15.123:0/867186677 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f3500109be0 con 0x7f34e406cc30 2024-03-30T04:42:37.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.312+0000 7f3505c00700 1 -- 172.21.15.123:0/867186677 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f34e406cc30 msgr2=0x7f34e406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:37.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.312+0000 7f3505c00700 1 --2- 172.21.15.123:0/867186677 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f34e406cc30 0x7f34e406f0e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f350010eba0 tx=0x7f34ec009e30 comp rx=0 tx=0).stop 2024-03-30T04:42:37.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.312+0000 7f3505c00700 1 -- 172.21.15.123:0/867186677 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3500102c50 msgr2=0x7f3500113940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:37.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.312+0000 7f3505c00700 1 --2- 172.21.15.123:0/867186677 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3500102c50 0x7f3500113940 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f34f000c8c0 tx=0x7f34f00044c0 comp rx=0 tx=0).stop 2024-03-30T04:42:37.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.312+0000 7f3505c00700 1 -- 172.21.15.123:0/867186677 shutdown_connections 2024-03-30T04:42:37.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.312+0000 7f3505c00700 1 --2- 172.21.15.123:0/867186677 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f34e406cc30 0x7f34e406f0e0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:37.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.312+0000 7f3505c00700 1 --2- 172.21.15.123:0/867186677 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3500102c50 0x7f3500113940 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:37.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.312+0000 7f3505c00700 1 --2- 172.21.15.123:0/867186677 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f35001035f0 0x7f3500113e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:37.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.312+0000 7f3505c00700 1 -- 172.21.15.123:0/867186677 >> 172.21.15.123:0/867186677 conn(0x7f35000f87e0 msgr2=0x7f35000f9900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:42:37.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.312+0000 7f3505c00700 1 -- 172.21.15.123:0/867186677 shutdown_connections 2024-03-30T04:42:37.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.312+0000 7f3505c00700 1 -- 172.21.15.123:0/867186677 wait complete. 2024-03-30T04:42:37.325 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:42:37.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.445+0000 7f9520600700 1 -- 172.21.15.123:0/1219234260 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95180fe830 msgr2=0x7f9518106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:37.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.445+0000 7f9520600700 1 --2- 172.21.15.123:0/1219234260 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95180fe830 0x7f9518106f00 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f9504004940 tx=0x7f9504009e30 comp rx=0 tx=0).stop 2024-03-30T04:42:37.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.445+0000 7f9520600700 1 -- 172.21.15.123:0/1219234260 shutdown_connections 2024-03-30T04:42:37.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.445+0000 7f9520600700 1 --2- 172.21.15.123:0/1219234260 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95180fe830 0x7f9518106f00 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:37.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.445+0000 7f9520600700 1 --2- 172.21.15.123:0/1219234260 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f95180fde90 0x7f95180fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:37.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.445+0000 7f9520600700 1 -- 172.21.15.123:0/1219234260 >> 172.21.15.123:0/1219234260 conn(0x7f9518072bc0 msgr2=0x7f9518072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:42:37.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.445+0000 7f9520600700 1 -- 172.21.15.123:0/1219234260 shutdown_connections 2024-03-30T04:42:37.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.445+0000 7f9520600700 1 -- 172.21.15.123:0/1219234260 wait complete. 2024-03-30T04:42:37.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.446+0000 7f9520600700 1 Processor -- start 2024-03-30T04:42:37.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.446+0000 7f9520600700 1 -- start start 2024-03-30T04:42:37.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.446+0000 7f9520600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95180fde90 0x7f9518113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:42:37.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.446+0000 7f9520600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f95180fe830 0x7f9518114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:42:37.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.446+0000 7f9520600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9518114670 con 0x7f95180fde90 2024-03-30T04:42:37.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.446+0000 7f9520600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f951810da60 con 0x7f95180fe830 2024-03-30T04:42:37.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.446+0000 7f951dc00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95180fde90 0x7f9518113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:42:37.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.446+0000 7f951dc00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95180fde90 0x7f9518113ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48260/0 (socket says 172.21.15.123:48260) 2024-03-30T04:42:37.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.446+0000 7f951dc00700 1 -- 172.21.15.123:0/1706464101 learned_addr learned my addr 172.21.15.123:0/1706464101 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:42:37.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.447+0000 7f951d200700 1 --2- 172.21.15.123:0/1706464101 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f95180fe830 0x7f9518114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:42:37.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.447+0000 7f951dc00700 1 -- 172.21.15.123:0/1706464101 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f95180fe830 msgr2=0x7f9518114020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:37.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.447+0000 7f951dc00700 1 --2- 172.21.15.123:0/1706464101 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f95180fe830 0x7f9518114020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:37.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.447+0000 7f951dc00700 1 -- 172.21.15.123:0/1706464101 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9504005040 con 0x7f95180fde90 2024-03-30T04:42:37.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.447+0000 7f951d200700 1 --2- 172.21.15.123:0/1706464101 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f95180fe830 0x7f9518114020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:42:37.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.447+0000 7f951dc00700 1 --2- 172.21.15.123:0/1706464101 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95180fde90 0x7f9518113ae0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f950c00c830 tx=0x7f950c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:42:37.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.447+0000 7f9515600700 1 -- 172.21.15.123:0/1706464101 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f950c00a760 con 0x7f95180fde90 2024-03-30T04:42:37.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.447+0000 7f9520600700 1 -- 172.21.15.123:0/1706464101 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f951810dd40 con 0x7f95180fde90 2024-03-30T04:42:37.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.447+0000 7f9515600700 1 -- 172.21.15.123:0/1706464101 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f950c00ad70 con 0x7f95180fde90 2024-03-30T04:42:37.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.447+0000 7f9515600700 1 -- 172.21.15.123:0/1706464101 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f950c0113a0 con 0x7f95180fde90 2024-03-30T04:42:37.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.448+0000 7f9520600700 1 -- 172.21.15.123:0/1706464101 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f951810e250 con 0x7f95180fde90 2024-03-30T04:42:37.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.448+0000 7f9520600700 1 -- 172.21.15.123:0/1706464101 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9518065e30 con 0x7f95180fde90 2024-03-30T04:42:37.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.449+0000 7f9515600700 1 -- 172.21.15.123:0/1706464101 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f950c012400 con 0x7f95180fde90 2024-03-30T04:42:37.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.450+0000 7f9515600700 1 --2- 172.21.15.123:0/1706464101 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f950006cc30 0x7f950006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:42:37.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.450+0000 7f9515600700 1 -- 172.21.15.123:0/1706464101 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f950c08c120 con 0x7f95180fde90 2024-03-30T04:42:37.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.450+0000 7f951d200700 1 --2- 172.21.15.123:0/1706464101 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f950006cc30 0x7f950006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:42:37.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.451+0000 7f951d200700 1 --2- 172.21.15.123:0/1706464101 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f950006cc30 0x7f950006f0e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f951810ed40 tx=0x7f9504009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:42:37.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.454+0000 7f9515600700 1 -- 172.21.15.123:0/1706464101 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f950c059b70 con 0x7f95180fde90 2024-03-30T04:42:37.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.681+0000 7f9520600700 1 -- 172.21.15.123:0/1706464101 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9518109dd0 con 0x7f950006cc30 2024-03-30T04:42:37.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.682+0000 7f9515600700 1 -- 172.21.15.123:0/1706464101 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f9518109dd0 con 0x7f950006cc30 2024-03-30T04:42:37.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.685+0000 7f9520600700 1 -- 172.21.15.123:0/1706464101 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f950006cc30 msgr2=0x7f950006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:37.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.685+0000 7f9520600700 1 --2- 172.21.15.123:0/1706464101 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f950006cc30 0x7f950006f0e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f951810ed40 tx=0x7f9504009e30 comp rx=0 tx=0).stop 2024-03-30T04:42:37.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.685+0000 7f9520600700 1 -- 172.21.15.123:0/1706464101 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95180fde90 msgr2=0x7f9518113ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:37.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.685+0000 7f9520600700 1 --2- 172.21.15.123:0/1706464101 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95180fde90 0x7f9518113ae0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f950c00c830 tx=0x7f950c004470 comp rx=0 tx=0).stop 2024-03-30T04:42:37.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.685+0000 7f9520600700 1 -- 172.21.15.123:0/1706464101 shutdown_connections 2024-03-30T04:42:37.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.685+0000 7f9520600700 1 --2- 172.21.15.123:0/1706464101 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f950006cc30 0x7f950006f0e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:37.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.685+0000 7f9520600700 1 --2- 172.21.15.123:0/1706464101 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95180fde90 0x7f9518113ae0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:37.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.685+0000 7f9520600700 1 --2- 172.21.15.123:0/1706464101 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f95180fe830 0x7f9518114020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:37.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.685+0000 7f9520600700 1 -- 172.21.15.123:0/1706464101 >> 172.21.15.123:0/1706464101 conn(0x7f9518072bc0 msgr2=0x7f9518103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:42:37.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.685+0000 7f9520600700 1 -- 172.21.15.123:0/1706464101 shutdown_connections 2024-03-30T04:42:37.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.685+0000 7f9520600700 1 -- 172.21.15.123:0/1706464101 wait complete. 2024-03-30T04:42:37.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.818+0000 7f8ece200700 1 -- 172.21.15.123:0/529215212 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ec81037a0 msgr2=0x7f8ec80fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:37.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.818+0000 7f8ece200700 1 --2- 172.21.15.123:0/529215212 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ec81037a0 0x7f8ec80fce10 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f8eb4004940 tx=0x7f8eb4009e30 comp rx=0 tx=0).stop 2024-03-30T04:42:37.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.818+0000 7f8ece200700 1 -- 172.21.15.123:0/529215212 shutdown_connections 2024-03-30T04:42:37.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.818+0000 7f8ece200700 1 --2- 172.21.15.123:0/529215212 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ec81037a0 0x7f8ec80fce10 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:37.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.818+0000 7f8ece200700 1 --2- 172.21.15.123:0/529215212 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8ec8102e00 0x7f8ec81031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:37.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.818+0000 7f8ece200700 1 -- 172.21.15.123:0/529215212 >> 172.21.15.123:0/529215212 conn(0x7f8ec80f89b0 msgr2=0x7f8ec80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:42:37.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.818+0000 7f8ece200700 1 -- 172.21.15.123:0/529215212 shutdown_connections 2024-03-30T04:42:37.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.818+0000 7f8ece200700 1 -- 172.21.15.123:0/529215212 wait complete. 2024-03-30T04:42:37.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.819+0000 7f8ece200700 1 Processor -- start 2024-03-30T04:42:37.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.819+0000 7f8ece200700 1 -- start start 2024-03-30T04:42:37.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.820+0000 7f8ece200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ec8102e00 0x7f8ec8113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:42:37.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.820+0000 7f8ece200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8ec81037a0 0x7f8ec8114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:42:37.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.820+0000 7f8ece200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec8114690 con 0x7f8ec8102e00 2024-03-30T04:42:37.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.820+0000 7f8ece200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec810da80 con 0x7f8ec81037a0 2024-03-30T04:42:37.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.820+0000 7f8ec7400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ec8102e00 0x7f8ec8113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:42:37.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.820+0000 7f8ec7400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ec8102e00 0x7f8ec8113b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48284/0 (socket says 172.21.15.123:48284) 2024-03-30T04:42:37.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.820+0000 7f8ec7400700 1 -- 172.21.15.123:0/2711331616 learned_addr learned my addr 172.21.15.123:0/2711331616 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:42:37.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.820+0000 7f8ec6a00700 1 --2- 172.21.15.123:0/2711331616 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8ec81037a0 0x7f8ec8114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:42:37.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.820+0000 7f8ec7400700 1 -- 172.21.15.123:0/2711331616 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8ec81037a0 msgr2=0x7f8ec8114040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:37.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.820+0000 7f8ec7400700 1 --2- 172.21.15.123:0/2711331616 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8ec81037a0 0x7f8ec8114040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:37.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.820+0000 7f8ec7400700 1 -- 172.21.15.123:0/2711331616 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb4005040 con 0x7f8ec8102e00 2024-03-30T04:42:37.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.821+0000 7f8ec7400700 1 --2- 172.21.15.123:0/2711331616 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ec8102e00 0x7f8ec8113b00 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8eb8002a50 tx=0x7f8eb80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:42:37.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.821+0000 7f8ebf400700 1 -- 172.21.15.123:0/2711331616 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8eb8007cc0 con 0x7f8ec8102e00 2024-03-30T04:42:37.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.821+0000 7f8ebf400700 1 -- 172.21.15.123:0/2711331616 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8eb800f040 con 0x7f8ec8102e00 2024-03-30T04:42:37.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.821+0000 7f8ece200700 1 -- 172.21.15.123:0/2711331616 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ec810dd60 con 0x7f8ec8102e00 2024-03-30T04:42:37.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.821+0000 7f8ebf400700 1 -- 172.21.15.123:0/2711331616 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8eb800d9c0 con 0x7f8ec8102e00 2024-03-30T04:42:37.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.821+0000 7f8ece200700 1 -- 172.21.15.123:0/2711331616 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ec810e2f0 con 0x7f8ec8102e00 2024-03-30T04:42:37.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.822+0000 7f8ece200700 1 -- 172.21.15.123:0/2711331616 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ec80fe550 con 0x7f8ec8102e00 2024-03-30T04:42:37.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.823+0000 7f8ebf400700 1 -- 172.21.15.123:0/2711331616 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f8eb800db20 con 0x7f8ec8102e00 2024-03-30T04:42:37.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.824+0000 7f8ebf400700 1 --2- 172.21.15.123:0/2711331616 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8eac06cc30 0x7f8eac06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:42:37.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.824+0000 7f8ebf400700 1 -- 172.21.15.123:0/2711331616 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f8eb808bde0 con 0x7f8ec8102e00 2024-03-30T04:42:37.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.824+0000 7f8ec6a00700 1 --2- 172.21.15.123:0/2711331616 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8eac06cc30 0x7f8eac06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:42:37.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.825+0000 7f8ec6a00700 1 --2- 172.21.15.123:0/2711331616 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8eac06cc30 0x7f8eac06f0e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8ec810ed60 tx=0x7f8eb4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:42:37.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:37.828+0000 7f8ebf400700 1 -- 172.21.15.123:0/2711331616 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8eb80597b0 con 0x7f8ec8102e00 2024-03-30T04:42:38.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.042+0000 7f8ece200700 1 -- 172.21.15.123:0/2711331616 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f8ec8109da0 con 0x7f8eac06cc30 2024-03-30T04:42:38.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.052+0000 7f8ebf400700 1 -- 172.21.15.123:0/2711331616 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f8ec8109da0 con 0x7f8eac06cc30 2024-03-30T04:42:38.050 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:42:38.050 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (11m) 4m ago 24m 15.0M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:42:38.050 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (9m) 4m ago 24m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T04:42:38.051 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (8m) 4m ago 23m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T04:42:38.051 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (11m) 4m ago 24m 74.0M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T04:42:38.051 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (4m) 4m ago 17m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T04:42:38.051 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (5m) 4m ago 18m 10.9M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T04:42:38.051 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (17m) 4m ago 17m 10.7M - 17.2.7-797-g5c024335 f77085f6aab7 0dc41326818c 2024-03-30T04:42:38.051 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (4m) 4m ago 18m 9747k - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T04:42:38.051 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (14m) 4m ago 25m 506M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:42:38.052 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (13m) 4m ago 23m 450M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:42:38.052 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (10m) 4m ago 26m 53.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T04:42:38.052 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (9m) 4m ago 23m 42.6M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T04:42:38.052 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (12m) 4m ago 24m 8052k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:42:38.052 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (12m) 4m ago 23m 7880k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:42:38.052 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (8m) 4m ago 22m 104M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T04:42:38.052 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (8m) 4m ago 21m 93.4M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T04:42:38.052 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (7m) 4m ago 21m 86.3M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T04:42:38.052 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (6m) 4m ago 20m 97.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T04:42:38.052 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (6m) 4m ago 20m 97.9M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T04:42:38.053 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (5m) 4m ago 19m 85.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T04:42:38.053 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (12m) 4m ago 23m 53.7M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:42:38.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.054+0000 7f8ece200700 1 -- 172.21.15.123:0/2711331616 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8eac06cc30 msgr2=0x7f8eac06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:38.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.054+0000 7f8ece200700 1 --2- 172.21.15.123:0/2711331616 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8eac06cc30 0x7f8eac06f0e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8ec810ed60 tx=0x7f8eb4009e30 comp rx=0 tx=0).stop 2024-03-30T04:42:38.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.054+0000 7f8ece200700 1 -- 172.21.15.123:0/2711331616 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ec8102e00 msgr2=0x7f8ec8113b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:38.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.054+0000 7f8ece200700 1 --2- 172.21.15.123:0/2711331616 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ec8102e00 0x7f8ec8113b00 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8eb8002a50 tx=0x7f8eb80046b0 comp rx=0 tx=0).stop 2024-03-30T04:42:38.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.054+0000 7f8ece200700 1 -- 172.21.15.123:0/2711331616 shutdown_connections 2024-03-30T04:42:38.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.054+0000 7f8ece200700 1 --2- 172.21.15.123:0/2711331616 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8eac06cc30 0x7f8eac06f0e0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:38.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.054+0000 7f8ece200700 1 --2- 172.21.15.123:0/2711331616 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ec8102e00 0x7f8ec8113b00 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:38.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.054+0000 7f8ece200700 1 --2- 172.21.15.123:0/2711331616 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8ec81037a0 0x7f8ec8114040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:38.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.054+0000 7f8ece200700 1 -- 172.21.15.123:0/2711331616 >> 172.21.15.123:0/2711331616 conn(0x7f8ec80f89b0 msgr2=0x7f8ec80f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:42:38.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.054+0000 7f8ece200700 1 -- 172.21.15.123:0/2711331616 shutdown_connections 2024-03-30T04:42:38.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.054+0000 7f8ece200700 1 -- 172.21.15.123:0/2711331616 wait complete. 2024-03-30T04:42:38.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.188+0000 7fd787e00700 1 -- 172.21.15.123:0/23849513 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7801037a0 msgr2=0x7fd7800fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:38.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.188+0000 7fd787e00700 1 --2- 172.21.15.123:0/23849513 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7801037a0 0x7fd7800fce10 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fd76c004940 tx=0x7fd76c009e30 comp rx=0 tx=0).stop 2024-03-30T04:42:38.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.188+0000 7fd787e00700 1 -- 172.21.15.123:0/23849513 shutdown_connections 2024-03-30T04:42:38.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.188+0000 7fd787e00700 1 --2- 172.21.15.123:0/23849513 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7801037a0 0x7fd7800fce10 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:38.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.188+0000 7fd787e00700 1 --2- 172.21.15.123:0/23849513 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd780102e00 0x7fd7801031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:38.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.188+0000 7fd787e00700 1 -- 172.21.15.123:0/23849513 >> 172.21.15.123:0/23849513 conn(0x7fd7800f89b0 msgr2=0x7fd7800fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:42:38.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.188+0000 7fd787e00700 1 -- 172.21.15.123:0/23849513 shutdown_connections 2024-03-30T04:42:38.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.188+0000 7fd787e00700 1 -- 172.21.15.123:0/23849513 wait complete. 2024-03-30T04:42:38.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.189+0000 7fd787e00700 1 Processor -- start 2024-03-30T04:42:38.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.189+0000 7fd787e00700 1 -- start start 2024-03-30T04:42:38.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.189+0000 7fd787e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd780102e00 0x7fd780115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:42:38.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.189+0000 7fd787e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd7801037a0 0x7fd7801161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:42:38.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.189+0000 7fd787e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd780116800 con 0x7fd780102e00 2024-03-30T04:42:38.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.189+0000 7fd787e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd78010fc80 con 0x7fd7801037a0 2024-03-30T04:42:38.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.189+0000 7fd785400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd780102e00 0x7fd780115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:42:38.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.189+0000 7fd785400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd780102e00 0x7fd780115c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48300/0 (socket says 172.21.15.123:48300) 2024-03-30T04:42:38.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.189+0000 7fd785400700 1 -- 172.21.15.123:0/1775890042 learned_addr learned my addr 172.21.15.123:0/1775890042 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:42:38.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.190+0000 7fd784a00700 1 --2- 172.21.15.123:0/1775890042 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd7801037a0 0x7fd7801161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:42:38.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.190+0000 7fd785400700 1 -- 172.21.15.123:0/1775890042 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd7801037a0 msgr2=0x7fd7801161b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:38.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.190+0000 7fd785400700 1 --2- 172.21.15.123:0/1775890042 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd7801037a0 0x7fd7801161b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:38.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.190+0000 7fd785400700 1 -- 172.21.15.123:0/1775890042 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd76c005040 con 0x7fd780102e00 2024-03-30T04:42:38.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.190+0000 7fd784a00700 1 --2- 172.21.15.123:0/1775890042 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd7801037a0 0x7fd7801161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:42:38.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.190+0000 7fd785400700 1 --2- 172.21.15.123:0/1775890042 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd780102e00 0x7fd780115c70 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fd77400c8c0 tx=0x7fd7740044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:42:38.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.190+0000 7fd77d600700 1 -- 172.21.15.123:0/1775890042 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd77400a980 con 0x7fd780102e00 2024-03-30T04:42:38.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.190+0000 7fd77d600700 1 -- 172.21.15.123:0/1775890042 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd77400ce60 con 0x7fd780102e00 2024-03-30T04:42:38.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.190+0000 7fd77d600700 1 -- 172.21.15.123:0/1775890042 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd7740113a0 con 0x7fd780102e00 2024-03-30T04:42:38.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.190+0000 7fd787e00700 1 -- 172.21.15.123:0/1775890042 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd78010ff60 con 0x7fd780102e00 2024-03-30T04:42:38.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.190+0000 7fd787e00700 1 -- 172.21.15.123:0/1775890042 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd780110520 con 0x7fd780102e00 2024-03-30T04:42:38.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.191+0000 7fd787e00700 1 -- 172.21.15.123:0/1775890042 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7800fe550 con 0x7fd780102e00 2024-03-30T04:42:38.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.192+0000 7fd77d600700 1 -- 172.21.15.123:0/1775890042 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fd774012400 con 0x7fd780102e00 2024-03-30T04:42:38.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.193+0000 7fd77d600700 1 --2- 172.21.15.123:0/1775890042 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd76806cc30 0x7fd76806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:42:38.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.193+0000 7fd77d600700 1 -- 172.21.15.123:0/1775890042 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd77408c0f0 con 0x7fd780102e00 2024-03-30T04:42:38.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.194+0000 7fd784a00700 1 --2- 172.21.15.123:0/1775890042 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd76806cc30 0x7fd76806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:42:38.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.194+0000 7fd784a00700 1 --2- 172.21.15.123:0/1775890042 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd76806cc30 0x7fd76806f0e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd76c00b3b0 tx=0x7fd76c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:42:38.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.198+0000 7fd77d600700 1 -- 172.21.15.123:0/1775890042 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd774056130 con 0x7fd780102e00 2024-03-30T04:42:38.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.481+0000 7fd787e00700 1 -- 172.21.15.123:0/1775890042 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fd78004bed0 con 0x7fd780102e00 2024-03-30T04:42:38.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.482+0000 7fd77d600700 1 -- 172.21.15.123:0/1775890042 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7fd774059750 con 0x7fd780102e00 2024-03-30T04:42:38.480 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:42:38.480 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:42:38.480 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:42:38.480 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:42:38.480 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:42:38.480 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:42:38.480 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:42:38.481 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:42:38.481 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T04:42:38.481 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:42:38.481 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:42:38.481 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:42:38.481 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 3 2024-03-30T04:42:38.481 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:42:38.481 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:42:38.481 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:42:38.481 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 13 2024-03-30T04:42:38.481 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:42:38.481 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:42:38.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.484+0000 7fd787e00700 1 -- 172.21.15.123:0/1775890042 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd76806cc30 msgr2=0x7fd76806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:38.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.484+0000 7fd787e00700 1 --2- 172.21.15.123:0/1775890042 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd76806cc30 0x7fd76806f0e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd76c00b3b0 tx=0x7fd76c009e30 comp rx=0 tx=0).stop 2024-03-30T04:42:38.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.484+0000 7fd787e00700 1 -- 172.21.15.123:0/1775890042 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd780102e00 msgr2=0x7fd780115c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:38.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.484+0000 7fd787e00700 1 --2- 172.21.15.123:0/1775890042 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd780102e00 0x7fd780115c70 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fd77400c8c0 tx=0x7fd7740044c0 comp rx=0 tx=0).stop 2024-03-30T04:42:38.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.484+0000 7fd787e00700 1 -- 172.21.15.123:0/1775890042 shutdown_connections 2024-03-30T04:42:38.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.484+0000 7fd787e00700 1 --2- 172.21.15.123:0/1775890042 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd76806cc30 0x7fd76806f0e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:38.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.484+0000 7fd787e00700 1 --2- 172.21.15.123:0/1775890042 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd780102e00 0x7fd780115c70 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:38.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.484+0000 7fd787e00700 1 --2- 172.21.15.123:0/1775890042 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd7801037a0 0x7fd7801161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:38.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.484+0000 7fd787e00700 1 -- 172.21.15.123:0/1775890042 >> 172.21.15.123:0/1775890042 conn(0x7fd7800f89b0 msgr2=0x7fd7800f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:42:38.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.484+0000 7fd787e00700 1 -- 172.21.15.123:0/1775890042 shutdown_connections 2024-03-30T04:42:38.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.484+0000 7fd787e00700 1 -- 172.21.15.123:0/1775890042 wait complete. 2024-03-30T04:42:38.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.619+0000 7fd6bda00700 1 -- 172.21.15.123:0/3038625425 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd6b81037a0 msgr2=0x7fd6b80fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:38.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.619+0000 7fd6bda00700 1 --2- 172.21.15.123:0/3038625425 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd6b81037a0 0x7fd6b80fce10 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fd6a4004940 tx=0x7fd6a4009e30 comp rx=0 tx=0).stop 2024-03-30T04:42:38.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.619+0000 7fd6bda00700 1 -- 172.21.15.123:0/3038625425 shutdown_connections 2024-03-30T04:42:38.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.619+0000 7fd6bda00700 1 --2- 172.21.15.123:0/3038625425 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd6b81037a0 0x7fd6b80fce10 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:38.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.619+0000 7fd6bda00700 1 --2- 172.21.15.123:0/3038625425 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd6b8102e00 0x7fd6b81031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:38.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.619+0000 7fd6bda00700 1 -- 172.21.15.123:0/3038625425 >> 172.21.15.123:0/3038625425 conn(0x7fd6b80f89b0 msgr2=0x7fd6b80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:42:38.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.619+0000 7fd6bda00700 1 -- 172.21.15.123:0/3038625425 shutdown_connections 2024-03-30T04:42:38.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.619+0000 7fd6bda00700 1 -- 172.21.15.123:0/3038625425 wait complete. 2024-03-30T04:42:38.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.620+0000 7fd6bda00700 1 Processor -- start 2024-03-30T04:42:38.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.620+0000 7fd6bda00700 1 -- start start 2024-03-30T04:42:38.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.621+0000 7fd6bda00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd6b8102e00 0x7fd6b8113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:42:38.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.621+0000 7fd6bda00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd6b81037a0 0x7fd6b8114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:42:38.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.621+0000 7fd6bda00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b8114690 con 0x7fd6b81037a0 2024-03-30T04:42:38.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.621+0000 7fd6bda00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b810da80 con 0x7fd6b8102e00 2024-03-30T04:42:38.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.621+0000 7fd6b6000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd6b81037a0 0x7fd6b8114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:42:38.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.621+0000 7fd6b6000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd6b81037a0 0x7fd6b8114040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48318/0 (socket says 172.21.15.123:48318) 2024-03-30T04:42:38.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.621+0000 7fd6b6000700 1 -- 172.21.15.123:0/2191139440 learned_addr learned my addr 172.21.15.123:0/2191139440 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:42:38.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.622+0000 7fd6b6a00700 1 --2- 172.21.15.123:0/2191139440 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd6b8102e00 0x7fd6b8113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:42:38.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.622+0000 7fd6b6000700 1 -- 172.21.15.123:0/2191139440 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd6b8102e00 msgr2=0x7fd6b8113b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:38.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.622+0000 7fd6b6000700 1 --2- 172.21.15.123:0/2191139440 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd6b8102e00 0x7fd6b8113b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:38.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.622+0000 7fd6b6000700 1 -- 172.21.15.123:0/2191139440 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a4005040 con 0x7fd6b81037a0 2024-03-30T04:42:38.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.622+0000 7fd6b6000700 1 --2- 172.21.15.123:0/2191139440 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd6b81037a0 0x7fd6b8114040 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fd6a4009390 tx=0x7fd6a4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:42:38.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.622+0000 7fd6aea00700 1 -- 172.21.15.123:0/2191139440 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd6a400e040 con 0x7fd6b81037a0 2024-03-30T04:42:38.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.622+0000 7fd6aea00700 1 -- 172.21.15.123:0/2191139440 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd6a4013420 con 0x7fd6b81037a0 2024-03-30T04:42:38.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.622+0000 7fd6bda00700 1 -- 172.21.15.123:0/2191139440 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6b810dd00 con 0x7fd6b81037a0 2024-03-30T04:42:38.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.622+0000 7fd6aea00700 1 -- 172.21.15.123:0/2191139440 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd6a401c630 con 0x7fd6b81037a0 2024-03-30T04:42:38.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.622+0000 7fd6bda00700 1 -- 172.21.15.123:0/2191139440 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6b810e2c0 con 0x7fd6b81037a0 2024-03-30T04:42:38.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.623+0000 7fd6bda00700 1 -- 172.21.15.123:0/2191139440 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6b80fe550 con 0x7fd6b81037a0 2024-03-30T04:42:38.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.624+0000 7fd6aea00700 1 -- 172.21.15.123:0/2191139440 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fd6a4013a00 con 0x7fd6b81037a0 2024-03-30T04:42:38.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.625+0000 7fd6aea00700 1 --2- 172.21.15.123:0/2191139440 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd69c06cc30 0x7fd69c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:42:38.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.625+0000 7fd6aea00700 1 -- 172.21.15.123:0/2191139440 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd6a408d2d0 con 0x7fd6b81037a0 2024-03-30T04:42:38.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.626+0000 7fd6b6a00700 1 --2- 172.21.15.123:0/2191139440 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd69c06cc30 0x7fd69c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:42:38.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.626+0000 7fd6b6a00700 1 --2- 172.21.15.123:0/2191139440 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd69c06cc30 0x7fd69c06f0e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd6a8007fd0 tx=0x7fd6a8009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:42:38.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.630+0000 7fd6aea00700 1 -- 172.21.15.123:0/2191139440 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd6a405ada0 con 0x7fd6b81037a0 2024-03-30T04:42:38.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:42:38 smithi135 ceph-mon[106011]: from='client.34682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:42:38.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:42:38 smithi135 ceph-mon[106011]: pgmap v343: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:38.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:42:38 smithi135 ceph-mon[106011]: from='client.34686 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:42:38.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:42:38 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/1775890042' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:42:38.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.873+0000 7fd6bda00700 1 -- 172.21.15.123:0/2191139440 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fd6b8109da0 con 0x7fd6b81037a0 2024-03-30T04:42:38.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.873+0000 7fd6aea00700 1 -- 172.21.15.123:0/2191139440 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1843 (secure 0 0 0) 0x7fd6a405a930 con 0x7fd6b81037a0 2024-03-30T04:42:38.872 INFO:teuthology.orchestra.run.smithi123.stdout:e32 2024-03-30T04:42:38.872 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:42:38.872 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:42:38.872 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:42:38.872 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:42:38.872 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:42:38.872 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:42:38.872 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T04:42:38.872 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T04:42:38.872 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:42:38.872 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T04:42:38.872 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:42:38.873 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:42:38.873 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:42:38.873 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:42:38.873 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:42:38.873 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T04:42:38.873 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:42:38.873 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:42:38.873 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T04:42:38.873 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:42:38.873 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T04:42:38.873 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T04:42:38.873 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T04:42:38.873 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T04:42:38.874 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:42:38.874 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:42:38.874 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:42:38.874 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:42:38.874 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:42:38.874 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:42:38.874 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T04:42:38.874 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T04:42:38.874 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:42:38.874 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:42:38.874 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T04:42:38.874 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:42:38.874 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:42:38.874 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:42:38.875 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44315} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/1741352336,v1:172.21.15.135:6827/1741352336] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:42:38.875 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:42:38.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.876+0000 7fd6bda00700 1 -- 172.21.15.123:0/2191139440 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd69c06cc30 msgr2=0x7fd69c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:38.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.876+0000 7fd6bda00700 1 --2- 172.21.15.123:0/2191139440 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd69c06cc30 0x7fd69c06f0e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd6a8007fd0 tx=0x7fd6a8009040 comp rx=0 tx=0).stop 2024-03-30T04:42:38.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.876+0000 7fd6bda00700 1 -- 172.21.15.123:0/2191139440 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd6b81037a0 msgr2=0x7fd6b8114040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:38.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.876+0000 7fd6bda00700 1 --2- 172.21.15.123:0/2191139440 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd6b81037a0 0x7fd6b8114040 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fd6a4009390 tx=0x7fd6a4009e30 comp rx=0 tx=0).stop 2024-03-30T04:42:38.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.876+0000 7fd6bda00700 1 -- 172.21.15.123:0/2191139440 shutdown_connections 2024-03-30T04:42:38.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.876+0000 7fd6bda00700 1 --2- 172.21.15.123:0/2191139440 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd69c06cc30 0x7fd69c06f0e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:38.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.876+0000 7fd6bda00700 1 --2- 172.21.15.123:0/2191139440 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd6b8102e00 0x7fd6b8113b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:38.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.876+0000 7fd6bda00700 1 --2- 172.21.15.123:0/2191139440 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd6b81037a0 0x7fd6b8114040 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:38.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.876+0000 7fd6bda00700 1 -- 172.21.15.123:0/2191139440 >> 172.21.15.123:0/2191139440 conn(0x7fd6b80f89b0 msgr2=0x7fd6b80f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:42:38.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.876+0000 7fd6bda00700 1 -- 172.21.15.123:0/2191139440 shutdown_connections 2024-03-30T04:42:38.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:38.876+0000 7fd6bda00700 1 -- 172.21.15.123:0/2191139440 wait complete. 2024-03-30T04:42:38.876 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 32 2024-03-30T04:42:38.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:42:38 smithi123 ceph-mon[139369]: from='client.34682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:42:38.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:42:38 smithi123 ceph-mon[139369]: pgmap v343: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:38.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:42:38 smithi123 ceph-mon[139369]: from='client.34686 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:42:38.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:42:38 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/1775890042' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:42:39.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.011+0000 7f833e600700 1 -- 172.21.15.123:0/2798598012 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8338102e20 msgr2=0x7f83381031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:39.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.011+0000 7f833e600700 1 --2- 172.21.15.123:0/2798598012 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8338102e20 0x7f83381031f0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f8328004940 tx=0x7f8328009e30 comp rx=0 tx=0).stop 2024-03-30T04:42:39.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.011+0000 7f833e600700 1 -- 172.21.15.123:0/2798598012 shutdown_connections 2024-03-30T04:42:39.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.011+0000 7f833e600700 1 --2- 172.21.15.123:0/2798598012 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f83381037c0 0x7f83380fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:39.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.011+0000 7f833e600700 1 --2- 172.21.15.123:0/2798598012 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8338102e20 0x7f83381031f0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:39.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.011+0000 7f833e600700 1 -- 172.21.15.123:0/2798598012 >> 172.21.15.123:0/2798598012 conn(0x7f83380f89d0 msgr2=0x7f83380fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:42:39.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.011+0000 7f833e600700 1 -- 172.21.15.123:0/2798598012 shutdown_connections 2024-03-30T04:42:39.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.011+0000 7f833e600700 1 -- 172.21.15.123:0/2798598012 wait complete. 2024-03-30T04:42:39.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.012+0000 7f833e600700 1 Processor -- start 2024-03-30T04:42:39.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.013+0000 7f833e600700 1 -- start start 2024-03-30T04:42:39.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.013+0000 7f833e600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8338102e20 0x7f8338113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:42:39.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.013+0000 7f833e600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f83381037c0 0x7f8338114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:42:39.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.013+0000 7f833e600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338114690 con 0x7f8338102e20 2024-03-30T04:42:39.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.013+0000 7f833e600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833810da80 con 0x7f83381037c0 2024-03-30T04:42:39.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.013+0000 7f8337400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8338102e20 0x7f8338113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:42:39.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.013+0000 7f8337400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8338102e20 0x7f8338113b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48340/0 (socket says 172.21.15.123:48340) 2024-03-30T04:42:39.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.014+0000 7f8337400700 1 -- 172.21.15.123:0/2614689407 learned_addr learned my addr 172.21.15.123:0/2614689407 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:42:39.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.014+0000 7f8336a00700 1 --2- 172.21.15.123:0/2614689407 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f83381037c0 0x7f8338114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:42:39.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.014+0000 7f8337400700 1 -- 172.21.15.123:0/2614689407 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f83381037c0 msgr2=0x7f8338114040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:39.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.014+0000 7f8337400700 1 --2- 172.21.15.123:0/2614689407 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f83381037c0 0x7f8338114040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:39.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.014+0000 7f8337400700 1 -- 172.21.15.123:0/2614689407 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8328005040 con 0x7f8338102e20 2024-03-30T04:42:39.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.014+0000 7f8337400700 1 --2- 172.21.15.123:0/2614689407 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8338102e20 0x7f8338113b00 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f8328009360 tx=0x7f8328009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:42:39.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.014+0000 7f832f400700 1 -- 172.21.15.123:0/2614689407 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f832800e040 con 0x7f8338102e20 2024-03-30T04:42:39.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.014+0000 7f832f400700 1 -- 172.21.15.123:0/2614689407 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8328013420 con 0x7f8338102e20 2024-03-30T04:42:39.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.014+0000 7f833e600700 1 -- 172.21.15.123:0/2614689407 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f833810dd00 con 0x7f8338102e20 2024-03-30T04:42:39.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.014+0000 7f832f400700 1 -- 172.21.15.123:0/2614689407 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f832801c5f0 con 0x7f8338102e20 2024-03-30T04:42:39.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.015+0000 7f833e600700 1 -- 172.21.15.123:0/2614689407 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f833810e2c0 con 0x7f8338102e20 2024-03-30T04:42:39.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.016+0000 7f833e600700 1 -- 172.21.15.123:0/2614689407 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83380fe570 con 0x7f8338102e20 2024-03-30T04:42:39.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.017+0000 7f832f400700 1 -- 172.21.15.123:0/2614689407 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f8328013590 con 0x7f8338102e20 2024-03-30T04:42:39.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.017+0000 7f832f400700 1 --2- 172.21.15.123:0/2614689407 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f831c06cc30 0x7f831c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:42:39.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.017+0000 7f832f400700 1 -- 172.21.15.123:0/2614689407 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f832808d180 con 0x7f8338102e20 2024-03-30T04:42:39.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.018+0000 7f8336a00700 1 --2- 172.21.15.123:0/2614689407 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f831c06cc30 0x7f831c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:42:39.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.018+0000 7f8336a00700 1 --2- 172.21.15.123:0/2614689407 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f831c06cc30 0x7f831c06f0e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8324007fd0 tx=0x7f8324009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:42:39.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.022+0000 7f832f400700 1 -- 172.21.15.123:0/2614689407 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f832805ad10 con 0x7f8338102e20 2024-03-30T04:42:39.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.241+0000 7f833e600700 1 -- 172.21.15.123:0/2614689407 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8338109de0 con 0x7f831c06cc30 2024-03-30T04:42:39.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.242+0000 7f832f400700 1 -- 172.21.15.123:0/2614689407 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f8338109de0 con 0x7f831c06cc30 2024-03-30T04:42:39.240 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:42:39.241 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:42:39.241 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:42:39.241 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T04:42:39.241 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:42:39.241 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T04:42:39.241 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T04:42:39.241 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T04:42:39.241 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T04:42:39.241 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:42:39.241 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "15/21 daemons upgraded", 2024-03-30T04:42:39.241 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T04:42:39.241 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:42:39.241 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:42:39.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.244+0000 7f833e600700 1 -- 172.21.15.123:0/2614689407 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f831c06cc30 msgr2=0x7f831c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:39.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.245+0000 7f833e600700 1 --2- 172.21.15.123:0/2614689407 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f831c06cc30 0x7f831c06f0e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8324007fd0 tx=0x7f8324009040 comp rx=0 tx=0).stop 2024-03-30T04:42:39.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.245+0000 7f833e600700 1 -- 172.21.15.123:0/2614689407 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8338102e20 msgr2=0x7f8338113b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:39.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.245+0000 7f833e600700 1 --2- 172.21.15.123:0/2614689407 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8338102e20 0x7f8338113b00 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f8328009360 tx=0x7f8328009e30 comp rx=0 tx=0).stop 2024-03-30T04:42:39.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.245+0000 7f833e600700 1 -- 172.21.15.123:0/2614689407 shutdown_connections 2024-03-30T04:42:39.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.245+0000 7f833e600700 1 --2- 172.21.15.123:0/2614689407 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f831c06cc30 0x7f831c06f0e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:39.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.245+0000 7f833e600700 1 --2- 172.21.15.123:0/2614689407 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8338102e20 0x7f8338113b00 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:39.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.245+0000 7f833e600700 1 --2- 172.21.15.123:0/2614689407 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f83381037c0 0x7f8338114040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:39.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.245+0000 7f833e600700 1 -- 172.21.15.123:0/2614689407 >> 172.21.15.123:0/2614689407 conn(0x7f83380f89d0 msgr2=0x7f83380f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:42:39.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.245+0000 7f833e600700 1 -- 172.21.15.123:0/2614689407 shutdown_connections 2024-03-30T04:42:39.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.245+0000 7f833e600700 1 -- 172.21.15.123:0/2614689407 wait complete. 2024-03-30T04:42:39.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.379+0000 7f64abc00700 1 -- 172.21.15.123:0/4259815397 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64a41037a0 msgr2=0x7f64a40fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:39.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.379+0000 7f64abc00700 1 --2- 172.21.15.123:0/4259815397 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64a41037a0 0x7f64a40fce10 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f6490004940 tx=0x7f6490009e30 comp rx=0 tx=0).stop 2024-03-30T04:42:39.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.379+0000 7f64abc00700 1 -- 172.21.15.123:0/4259815397 shutdown_connections 2024-03-30T04:42:39.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.379+0000 7f64abc00700 1 --2- 172.21.15.123:0/4259815397 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64a41037a0 0x7f64a40fce10 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:39.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.379+0000 7f64abc00700 1 --2- 172.21.15.123:0/4259815397 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f64a4102e00 0x7f64a41031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:39.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.379+0000 7f64abc00700 1 -- 172.21.15.123:0/4259815397 >> 172.21.15.123:0/4259815397 conn(0x7f64a40f89b0 msgr2=0x7f64a40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:42:39.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.379+0000 7f64abc00700 1 -- 172.21.15.123:0/4259815397 shutdown_connections 2024-03-30T04:42:39.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.379+0000 7f64abc00700 1 -- 172.21.15.123:0/4259815397 wait complete. 2024-03-30T04:42:39.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.380+0000 7f64abc00700 1 Processor -- start 2024-03-30T04:42:39.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.380+0000 7f64abc00700 1 -- start start 2024-03-30T04:42:39.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.380+0000 7f64abc00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f64a4102e00 0x7f64a4113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:42:39.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.380+0000 7f64abc00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64a41037a0 0x7f64a4114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:42:39.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.380+0000 7f64abc00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a4114690 con 0x7f64a41037a0 2024-03-30T04:42:39.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.380+0000 7f64abc00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a410da80 con 0x7f64a4102e00 2024-03-30T04:42:39.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.380+0000 7f64a3e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64a41037a0 0x7f64a4114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:42:39.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.380+0000 7f64a3e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64a41037a0 0x7f64a4114040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48368/0 (socket says 172.21.15.123:48368) 2024-03-30T04:42:39.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.381+0000 7f64a3e00700 1 -- 172.21.15.123:0/3166001884 learned_addr learned my addr 172.21.15.123:0/3166001884 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:42:39.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.381+0000 7f64a9200700 1 --2- 172.21.15.123:0/3166001884 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f64a4102e00 0x7f64a4113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:42:39.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.381+0000 7f64a3e00700 1 -- 172.21.15.123:0/3166001884 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f64a4102e00 msgr2=0x7f64a4113b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:39.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.381+0000 7f64a3e00700 1 --2- 172.21.15.123:0/3166001884 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f64a4102e00 0x7f64a4113b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:39.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.381+0000 7f64a3e00700 1 -- 172.21.15.123:0/3166001884 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6490005040 con 0x7f64a41037a0 2024-03-30T04:42:39.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.381+0000 7f64a3e00700 1 --2- 172.21.15.123:0/3166001884 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64a41037a0 0x7f64a4114040 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f6490009390 tx=0x7f6490009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:42:39.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.381+0000 7f64a0c00700 1 -- 172.21.15.123:0/3166001884 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f649000e040 con 0x7f64a41037a0 2024-03-30T04:42:39.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.381+0000 7f64abc00700 1 -- 172.21.15.123:0/3166001884 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64a410dd00 con 0x7f64a41037a0 2024-03-30T04:42:39.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.381+0000 7f64a0c00700 1 -- 172.21.15.123:0/3166001884 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6490004290 con 0x7f64a41037a0 2024-03-30T04:42:39.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.381+0000 7f64a0c00700 1 -- 172.21.15.123:0/3166001884 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6490013e70 con 0x7f64a41037a0 2024-03-30T04:42:39.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.382+0000 7f64abc00700 1 -- 172.21.15.123:0/3166001884 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64a410e2c0 con 0x7f64a41037a0 2024-03-30T04:42:39.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.383+0000 7f64abc00700 1 -- 172.21.15.123:0/3166001884 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64a40fe550 con 0x7f64a41037a0 2024-03-30T04:42:39.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.384+0000 7f64a0c00700 1 -- 172.21.15.123:0/3166001884 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f6490013690 con 0x7f64a41037a0 2024-03-30T04:42:39.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.385+0000 7f64a0c00700 1 --2- 172.21.15.123:0/3166001884 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f648c06cbc0 0x7f648c06f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:42:39.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.385+0000 7f64a0c00700 1 -- 172.21.15.123:0/3166001884 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f649008d230 con 0x7f64a41037a0 2024-03-30T04:42:39.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.385+0000 7f64a9200700 1 --2- 172.21.15.123:0/3166001884 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f648c06cbc0 0x7f648c06f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:42:39.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.386+0000 7f64a9200700 1 --2- 172.21.15.123:0/3166001884 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f648c06cbc0 0x7f648c06f070 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f6498004f40 tx=0x7f649800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:42:39.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.390+0000 7f64a0c00700 1 -- 172.21.15.123:0/3166001884 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f649005adb0 con 0x7f64a41037a0 2024-03-30T04:42:39.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.675+0000 7f64abc00700 1 -- 172.21.15.123:0/3166001884 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f64a405f5f0 con 0x7f64a41037a0 2024-03-30T04:42:39.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.675+0000 7f64a0c00700 1 -- 172.21.15.123:0/3166001884 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f6490013420 con 0x7f64a41037a0 2024-03-30T04:42:39.674 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T04:42:39.674 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T04:42:39.674 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T04:42:39.674 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T04:42:39.674 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T04:42:39.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.678+0000 7f64abc00700 1 -- 172.21.15.123:0/3166001884 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f648c06cbc0 msgr2=0x7f648c06f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:39.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.678+0000 7f64abc00700 1 --2- 172.21.15.123:0/3166001884 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f648c06cbc0 0x7f648c06f070 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f6498004f40 tx=0x7f649800b040 comp rx=0 tx=0).stop 2024-03-30T04:42:39.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.678+0000 7f64abc00700 1 -- 172.21.15.123:0/3166001884 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64a41037a0 msgr2=0x7f64a4114040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:42:39.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.678+0000 7f64abc00700 1 --2- 172.21.15.123:0/3166001884 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64a41037a0 0x7f64a4114040 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f6490009390 tx=0x7f6490009e30 comp rx=0 tx=0).stop 2024-03-30T04:42:39.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.678+0000 7f64abc00700 1 -- 172.21.15.123:0/3166001884 shutdown_connections 2024-03-30T04:42:39.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.678+0000 7f64abc00700 1 --2- 172.21.15.123:0/3166001884 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f648c06cbc0 0x7f648c06f070 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:39.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.678+0000 7f64abc00700 1 --2- 172.21.15.123:0/3166001884 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f64a4102e00 0x7f64a4113b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:39.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.678+0000 7f64abc00700 1 --2- 172.21.15.123:0/3166001884 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64a41037a0 0x7f64a4114040 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:42:39.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.678+0000 7f64abc00700 1 -- 172.21.15.123:0/3166001884 >> 172.21.15.123:0/3166001884 conn(0x7f64a40f89b0 msgr2=0x7f64a40f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:42:39.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.678+0000 7f64abc00700 1 -- 172.21.15.123:0/3166001884 shutdown_connections 2024-03-30T04:42:39.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:42:39.678+0000 7f64abc00700 1 -- 172.21.15.123:0/3166001884 wait complete. 2024-03-30T04:42:39.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:42:39 smithi135 ceph-mon[106011]: from='client.34690 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:42:39.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:42:39 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2191139440' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:42:39.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:42:39 smithi123 ceph-mon[139369]: from='client.34690 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:42:39.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:42:39 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2191139440' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:42:40.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:42:40 smithi123 ceph-mon[139369]: from='client.34702 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:42:40.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:42:40 smithi123 ceph-mon[139369]: pgmap v344: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:40.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:42:40 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3166001884' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:42:41.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:42:40 smithi135 ceph-mon[106011]: from='client.34702 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:42:41.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:42:40 smithi135 ceph-mon[106011]: pgmap v344: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:41.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:42:40 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3166001884' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:42:42.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:42:42 smithi123 ceph-mon[139369]: pgmap v345: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:43.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:42:42 smithi135 ceph-mon[106011]: pgmap v345: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:44.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:42:44 smithi123 ceph-mon[139369]: pgmap v346: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:44.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:42:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:42:45.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:42:44 smithi135 ceph-mon[106011]: pgmap v346: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:45.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:42:44 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:42:46.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:42:46 smithi123 ceph-mon[139369]: pgmap v347: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:47.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:42:46 smithi135 ceph-mon[106011]: pgmap v347: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:48.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:42:48 smithi123 ceph-mon[139369]: pgmap v348: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:49.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:42:48 smithi135 ceph-mon[106011]: pgmap v348: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:50.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:42:50 smithi135 ceph-mon[106011]: pgmap v349: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:50.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:42:50 smithi123 ceph-mon[139369]: pgmap v349: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:52.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:42:52 smithi135 ceph-mon[106011]: pgmap v350: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:52.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:42:52 smithi123 ceph-mon[139369]: pgmap v350: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:54.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:42:54 smithi135 ceph-mon[106011]: pgmap v351: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:54.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:42:54 smithi123 ceph-mon[139369]: pgmap v351: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:56.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:42:56 smithi135 ceph-mon[106011]: pgmap v352: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:56.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:42:56 smithi123 ceph-mon[139369]: pgmap v352: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:58.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:42:58 smithi135 ceph-mon[106011]: pgmap v353: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:58.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:42:58 smithi123 ceph-mon[139369]: pgmap v353: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:42:59.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:42:59 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:42:59.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:42:59 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:43:00.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:43:00 smithi135 ceph-mon[106011]: pgmap v354: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:00.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:43:00 smithi123 ceph-mon[139369]: pgmap v354: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:02.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:43:02 smithi135 ceph-mon[106011]: pgmap v355: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:02.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:43:02 smithi123 ceph-mon[139369]: pgmap v355: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:04.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:43:04 smithi135 ceph-mon[106011]: pgmap v356: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:04.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:43:04 smithi123 ceph-mon[139369]: pgmap v356: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:06.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:43:06 smithi123 ceph-mon[139369]: pgmap v357: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:07.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:43:06 smithi135 ceph-mon[106011]: pgmap v357: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:08.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:43:08 smithi123 ceph-mon[139369]: pgmap v358: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:09.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:43:08 smithi135 ceph-mon[106011]: pgmap v358: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:09.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:09.824+0000 7fdf70e00700 1 -- 172.21.15.123:0/4018108150 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf6c102e00 msgr2=0x7fdf6c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:09.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:09.824+0000 7fdf70e00700 1 --2- 172.21.15.123:0/4018108150 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf6c102e00 0x7fdf6c1031d0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c004940 tx=0x7fdf5c009e30 comp rx=0 tx=0).stop 2024-03-30T04:43:09.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:09.825+0000 7fdf70e00700 1 -- 172.21.15.123:0/4018108150 shutdown_connections 2024-03-30T04:43:09.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:09.825+0000 7fdf70e00700 1 --2- 172.21.15.123:0/4018108150 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf6c1037a0 0x7fdf6c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:09.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:09.825+0000 7fdf70e00700 1 --2- 172.21.15.123:0/4018108150 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf6c102e00 0x7fdf6c1031d0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:09.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:09.825+0000 7fdf70e00700 1 -- 172.21.15.123:0/4018108150 >> 172.21.15.123:0/4018108150 conn(0x7fdf6c0f89b0 msgr2=0x7fdf6c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:43:09.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:09.825+0000 7fdf70e00700 1 -- 172.21.15.123:0/4018108150 shutdown_connections 2024-03-30T04:43:09.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:09.825+0000 7fdf70e00700 1 -- 172.21.15.123:0/4018108150 wait complete. 2024-03-30T04:43:09.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:09.825+0000 7fdf70e00700 1 Processor -- start 2024-03-30T04:43:09.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:09.826+0000 7fdf70e00700 1 -- start start 2024-03-30T04:43:09.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:09.826+0000 7fdf70e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf6c102e00 0x7fdf6c113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:43:09.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:09.826+0000 7fdf70e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf6c1037a0 0x7fdf6c114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:43:09.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:09.826+0000 7fdf70e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6c1146b0 con 0x7fdf6c1037a0 2024-03-30T04:43:09.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:09.826+0000 7fdf70e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6c10daa0 con 0x7fdf6c102e00 2024-03-30T04:43:09.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:09.826+0000 7fdf69400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf6c1037a0 0x7fdf6c114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:43:09.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:09.826+0000 7fdf69400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf6c1037a0 0x7fdf6c114060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42144/0 (socket says 172.21.15.123:42144) 2024-03-30T04:43:09.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:09.826+0000 7fdf69400700 1 -- 172.21.15.123:0/2112222731 learned_addr learned my addr 172.21.15.123:0/2112222731 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:43:09.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:09.827+0000 7fdf69e00700 1 --2- 172.21.15.123:0/2112222731 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf6c102e00 0x7fdf6c113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:43:09.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:09.827+0000 7fdf69400700 1 -- 172.21.15.123:0/2112222731 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf6c102e00 msgr2=0x7fdf6c113b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:09.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:09.827+0000 7fdf69400700 1 --2- 172.21.15.123:0/2112222731 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf6c102e00 0x7fdf6c113b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:09.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:09.827+0000 7fdf69400700 1 -- 172.21.15.123:0/2112222731 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf5c005040 con 0x7fdf6c1037a0 2024-03-30T04:43:09.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:09.827+0000 7fdf69400700 1 --2- 172.21.15.123:0/2112222731 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf6c1037a0 0x7fdf6c114060 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fdf58002a50 tx=0x7fdf58004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:43:09.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:09.827+0000 7fdf62000700 1 -- 172.21.15.123:0/2112222731 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf58007cc0 con 0x7fdf6c1037a0 2024-03-30T04:43:09.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:09.827+0000 7fdf62000700 1 -- 172.21.15.123:0/2112222731 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdf5800f040 con 0x7fdf6c1037a0 2024-03-30T04:43:09.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:09.827+0000 7fdf62000700 1 -- 172.21.15.123:0/2112222731 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf5800d9a0 con 0x7fdf6c1037a0 2024-03-30T04:43:09.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:09.827+0000 7fdf70e00700 1 -- 172.21.15.123:0/2112222731 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf6c10dd80 con 0x7fdf6c1037a0 2024-03-30T04:43:09.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:09.827+0000 7fdf70e00700 1 -- 172.21.15.123:0/2112222731 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf6c10e340 con 0x7fdf6c1037a0 2024-03-30T04:43:09.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:09.828+0000 7fdf70e00700 1 -- 172.21.15.123:0/2112222731 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf6c0fe550 con 0x7fdf6c1037a0 2024-03-30T04:43:09.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:09.829+0000 7fdf62000700 1 -- 172.21.15.123:0/2112222731 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fdf5800dc60 con 0x7fdf6c1037a0 2024-03-30T04:43:09.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:09.830+0000 7fdf62000700 1 --2- 172.21.15.123:0/2112222731 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fdf5006cc30 0x7fdf5006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:43:09.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:09.830+0000 7fdf62000700 1 -- 172.21.15.123:0/2112222731 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fdf5808bd10 con 0x7fdf6c1037a0 2024-03-30T04:43:09.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:09.831+0000 7fdf69e00700 1 --2- 172.21.15.123:0/2112222731 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fdf5006cc30 0x7fdf5006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:43:09.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:09.831+0000 7fdf69e00700 1 --2- 172.21.15.123:0/2112222731 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fdf5006cc30 0x7fdf5006f0e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c00b020 tx=0x7fdf5c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:43:09.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:09.835+0000 7fdf62000700 1 -- 172.21.15.123:0/2112222731 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fdf580596e0 con 0x7fdf6c1037a0 2024-03-30T04:43:10.055 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.056+0000 7fdf70e00700 1 -- 172.21.15.123:0/2112222731 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdf6c109dc0 con 0x7fdf5006cc30 2024-03-30T04:43:10.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.058+0000 7fdf62000700 1 -- 172.21.15.123:0/2112222731 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fdf6c109dc0 con 0x7fdf5006cc30 2024-03-30T04:43:10.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.060+0000 7fdf70e00700 1 -- 172.21.15.123:0/2112222731 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fdf5006cc30 msgr2=0x7fdf5006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:10.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.060+0000 7fdf70e00700 1 --2- 172.21.15.123:0/2112222731 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fdf5006cc30 0x7fdf5006f0e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c00b020 tx=0x7fdf5c009e30 comp rx=0 tx=0).stop 2024-03-30T04:43:10.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.060+0000 7fdf70e00700 1 -- 172.21.15.123:0/2112222731 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf6c1037a0 msgr2=0x7fdf6c114060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:10.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.060+0000 7fdf70e00700 1 --2- 172.21.15.123:0/2112222731 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf6c1037a0 0x7fdf6c114060 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fdf58002a50 tx=0x7fdf58004600 comp rx=0 tx=0).stop 2024-03-30T04:43:10.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.060+0000 7fdf70e00700 1 -- 172.21.15.123:0/2112222731 shutdown_connections 2024-03-30T04:43:10.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.060+0000 7fdf70e00700 1 --2- 172.21.15.123:0/2112222731 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fdf5006cc30 0x7fdf5006f0e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:10.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.060+0000 7fdf70e00700 1 --2- 172.21.15.123:0/2112222731 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf6c102e00 0x7fdf6c113b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:10.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.060+0000 7fdf70e00700 1 --2- 172.21.15.123:0/2112222731 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf6c1037a0 0x7fdf6c114060 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:10.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.060+0000 7fdf70e00700 1 -- 172.21.15.123:0/2112222731 >> 172.21.15.123:0/2112222731 conn(0x7fdf6c0f89b0 msgr2=0x7fdf6c0f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:43:10.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.061+0000 7fdf70e00700 1 -- 172.21.15.123:0/2112222731 shutdown_connections 2024-03-30T04:43:10.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.061+0000 7fdf70e00700 1 -- 172.21.15.123:0/2112222731 wait complete. 2024-03-30T04:43:10.073 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:43:10.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.196+0000 7f9a86c00700 1 -- 172.21.15.123:0/310104143 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a80070360 msgr2=0x7f9a80070730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:10.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.196+0000 7f9a86c00700 1 --2- 172.21.15.123:0/310104143 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a80070360 0x7f9a80070730 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f9a70004970 tx=0x7f9a70009e30 comp rx=0 tx=0).stop 2024-03-30T04:43:10.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.196+0000 7f9a86c00700 1 -- 172.21.15.123:0/310104143 shutdown_connections 2024-03-30T04:43:10.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.196+0000 7f9a86c00700 1 --2- 172.21.15.123:0/310104143 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9a80070c70 0x7f9a80109110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:10.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.196+0000 7f9a86c00700 1 --2- 172.21.15.123:0/310104143 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a80070360 0x7f9a80070730 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:10.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.196+0000 7f9a86c00700 1 -- 172.21.15.123:0/310104143 >> 172.21.15.123:0/310104143 conn(0x7f9a800f89b0 msgr2=0x7f9a800fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:43:10.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.196+0000 7f9a86c00700 1 -- 172.21.15.123:0/310104143 shutdown_connections 2024-03-30T04:43:10.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.196+0000 7f9a86c00700 1 -- 172.21.15.123:0/310104143 wait complete. 2024-03-30T04:43:10.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.197+0000 7f9a86c00700 1 Processor -- start 2024-03-30T04:43:10.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.197+0000 7f9a86c00700 1 -- start start 2024-03-30T04:43:10.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.197+0000 7f9a86c00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9a80070360 0x7f9a80115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:43:10.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.197+0000 7f9a86c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a80070c70 0x7f9a801161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:43:10.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.197+0000 7f9a86c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a80116810 con 0x7f9a80070c70 2024-03-30T04:43:10.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.197+0000 7f9a86c00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8010fbe0 con 0x7f9a80070360 2024-03-30T04:43:10.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.198+0000 7f9a7f400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a80070c70 0x7f9a801161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:43:10.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.198+0000 7f9a7f400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a80070c70 0x7f9a801161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42156/0 (socket says 172.21.15.123:42156) 2024-03-30T04:43:10.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.198+0000 7f9a7f400700 1 -- 172.21.15.123:0/230861044 learned_addr learned my addr 172.21.15.123:0/230861044 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:43:10.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.198+0000 7f9a7f400700 1 -- 172.21.15.123:0/230861044 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9a80070360 msgr2=0x7f9a80115c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:10.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.198+0000 7f9a7f400700 1 --2- 172.21.15.123:0/230861044 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9a80070360 0x7f9a80115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:10.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.198+0000 7f9a7f400700 1 -- 172.21.15.123:0/230861044 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a70005040 con 0x7f9a80070c70 2024-03-30T04:43:10.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.198+0000 7f9a7f400700 1 --2- 172.21.15.123:0/230861044 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a80070c70 0x7f9a801161c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c00eb40 tx=0x7f9a6c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:43:10.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.198+0000 7f9a77e00700 1 -- 172.21.15.123:0/230861044 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a6c007cc0 con 0x7f9a80070c70 2024-03-30T04:43:10.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.199+0000 7f9a86c00700 1 -- 172.21.15.123:0/230861044 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a8010fe60 con 0x7f9a80070c70 2024-03-30T04:43:10.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.199+0000 7f9a77e00700 1 -- 172.21.15.123:0/230861044 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9a6c00f040 con 0x7f9a80070c70 2024-03-30T04:43:10.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.199+0000 7f9a77e00700 1 -- 172.21.15.123:0/230861044 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a6c009570 con 0x7f9a80070c70 2024-03-30T04:43:10.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.199+0000 7f9a86c00700 1 -- 172.21.15.123:0/230861044 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a801103a0 con 0x7f9a80070c70 2024-03-30T04:43:10.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.200+0000 7f9a86c00700 1 -- 172.21.15.123:0/230861044 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a80103b90 con 0x7f9a80070c70 2024-03-30T04:43:10.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.201+0000 7f9a77e00700 1 -- 172.21.15.123:0/230861044 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f9a6c0096d0 con 0x7f9a80070c70 2024-03-30T04:43:10.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.202+0000 7f9a77e00700 1 --2- 172.21.15.123:0/230861044 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9a6406cc30 0x7f9a6406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:43:10.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.202+0000 7f9a77e00700 1 -- 172.21.15.123:0/230861044 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f9a6c08bcb0 con 0x7f9a80070c70 2024-03-30T04:43:10.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.202+0000 7f9a7fe00700 1 --2- 172.21.15.123:0/230861044 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9a6406cc30 0x7f9a6406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:43:10.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.203+0000 7f9a7fe00700 1 --2- 172.21.15.123:0/230861044 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9a6406cc30 0x7f9a6406f0e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9a7000b760 tx=0x7f9a70009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:43:10.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.206+0000 7f9a77e00700 1 -- 172.21.15.123:0/230861044 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9a6c0597b0 con 0x7f9a80070c70 2024-03-30T04:43:10.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.427+0000 7f9a86c00700 1 -- 172.21.15.123:0/230861044 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9a80111340 con 0x7f9a6406cc30 2024-03-30T04:43:10.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.428+0000 7f9a77e00700 1 -- 172.21.15.123:0/230861044 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f9a80111340 con 0x7f9a6406cc30 2024-03-30T04:43:10.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.430+0000 7f9a86c00700 1 -- 172.21.15.123:0/230861044 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9a6406cc30 msgr2=0x7f9a6406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:10.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.430+0000 7f9a86c00700 1 --2- 172.21.15.123:0/230861044 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9a6406cc30 0x7f9a6406f0e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9a7000b760 tx=0x7f9a70009e30 comp rx=0 tx=0).stop 2024-03-30T04:43:10.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.430+0000 7f9a86c00700 1 -- 172.21.15.123:0/230861044 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a80070c70 msgr2=0x7f9a801161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:10.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.430+0000 7f9a86c00700 1 --2- 172.21.15.123:0/230861044 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a80070c70 0x7f9a801161c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c00eb40 tx=0x7f9a6c00b040 comp rx=0 tx=0).stop 2024-03-30T04:43:10.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.430+0000 7f9a86c00700 1 -- 172.21.15.123:0/230861044 shutdown_connections 2024-03-30T04:43:10.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.430+0000 7f9a86c00700 1 --2- 172.21.15.123:0/230861044 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9a6406cc30 0x7f9a6406f0e0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:10.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.430+0000 7f9a86c00700 1 --2- 172.21.15.123:0/230861044 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9a80070360 0x7f9a80115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:10.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.430+0000 7f9a86c00700 1 --2- 172.21.15.123:0/230861044 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a80070c70 0x7f9a801161c0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:10.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.430+0000 7f9a86c00700 1 -- 172.21.15.123:0/230861044 >> 172.21.15.123:0/230861044 conn(0x7f9a800f89b0 msgr2=0x7f9a801015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:43:10.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.431+0000 7f9a86c00700 1 -- 172.21.15.123:0/230861044 shutdown_connections 2024-03-30T04:43:10.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.431+0000 7f9a86c00700 1 -- 172.21.15.123:0/230861044 wait complete. 2024-03-30T04:43:10.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.567+0000 7f0084000700 1 -- 172.21.15.123:0/3073801251 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f007c1037a0 msgr2=0x7f007c0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:10.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.567+0000 7f0084000700 1 --2- 172.21.15.123:0/3073801251 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f007c1037a0 0x7f007c0fce10 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f0068004940 tx=0x7f0068009e30 comp rx=0 tx=0).stop 2024-03-30T04:43:10.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.567+0000 7f0084000700 1 -- 172.21.15.123:0/3073801251 shutdown_connections 2024-03-30T04:43:10.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.567+0000 7f0084000700 1 --2- 172.21.15.123:0/3073801251 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f007c1037a0 0x7f007c0fce10 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:10.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.567+0000 7f0084000700 1 --2- 172.21.15.123:0/3073801251 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f007c102e00 0x7f007c1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:10.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.567+0000 7f0084000700 1 -- 172.21.15.123:0/3073801251 >> 172.21.15.123:0/3073801251 conn(0x7f007c0f89b0 msgr2=0x7f007c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:43:10.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.567+0000 7f0084000700 1 -- 172.21.15.123:0/3073801251 shutdown_connections 2024-03-30T04:43:10.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.567+0000 7f0084000700 1 -- 172.21.15.123:0/3073801251 wait complete. 2024-03-30T04:43:10.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.568+0000 7f0084000700 1 Processor -- start 2024-03-30T04:43:10.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.568+0000 7f0084000700 1 -- start start 2024-03-30T04:43:10.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.568+0000 7f0084000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f007c102e00 0x7f007c115da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:43:10.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.568+0000 7f0084000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f007c1037a0 0x7f007c1162e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:43:10.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.568+0000 7f0084000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f007c116930 con 0x7f007c1037a0 2024-03-30T04:43:10.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.568+0000 7f0084000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f007c10fd00 con 0x7f007c102e00 2024-03-30T04:43:10.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.569+0000 7f0080c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f007c1037a0 0x7f007c1162e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:43:10.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.569+0000 7f0080c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f007c1037a0 0x7f007c1162e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42174/0 (socket says 172.21.15.123:42174) 2024-03-30T04:43:10.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.569+0000 7f0080c00700 1 -- 172.21.15.123:0/721799200 learned_addr learned my addr 172.21.15.123:0/721799200 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:43:10.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.569+0000 7f0081600700 1 --2- 172.21.15.123:0/721799200 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f007c102e00 0x7f007c115da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:43:10.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.569+0000 7f0080c00700 1 -- 172.21.15.123:0/721799200 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f007c102e00 msgr2=0x7f007c115da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:10.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.569+0000 7f0080c00700 1 --2- 172.21.15.123:0/721799200 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f007c102e00 0x7f007c115da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:10.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.569+0000 7f0080c00700 1 -- 172.21.15.123:0/721799200 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0068005040 con 0x7f007c1037a0 2024-03-30T04:43:10.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.569+0000 7f0080c00700 1 --2- 172.21.15.123:0/721799200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f007c1037a0 0x7f007c1162e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f0068009390 tx=0x7f0068009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:43:10.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.570+0000 7f0079600700 1 -- 172.21.15.123:0/721799200 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f006800e040 con 0x7f007c1037a0 2024-03-30T04:43:10.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.570+0000 7f0079600700 1 -- 172.21.15.123:0/721799200 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0068013420 con 0x7f007c1037a0 2024-03-30T04:43:10.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.570+0000 7f0084000700 1 -- 172.21.15.123:0/721799200 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f007c10ff80 con 0x7f007c1037a0 2024-03-30T04:43:10.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.570+0000 7f0079600700 1 -- 172.21.15.123:0/721799200 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f006801c630 con 0x7f007c1037a0 2024-03-30T04:43:10.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.570+0000 7f0084000700 1 -- 172.21.15.123:0/721799200 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f007c1104c0 con 0x7f007c1037a0 2024-03-30T04:43:10.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.571+0000 7f0084000700 1 -- 172.21.15.123:0/721799200 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f007c0fe550 con 0x7f007c1037a0 2024-03-30T04:43:10.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.572+0000 7f0079600700 1 -- 172.21.15.123:0/721799200 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f0068013a00 con 0x7f007c1037a0 2024-03-30T04:43:10.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.573+0000 7f0079600700 1 --2- 172.21.15.123:0/721799200 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f006406cc30 0x7f006406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:43:10.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.573+0000 7f0079600700 1 -- 172.21.15.123:0/721799200 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f006808d1b0 con 0x7f007c1037a0 2024-03-30T04:43:10.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.573+0000 7f0081600700 1 --2- 172.21.15.123:0/721799200 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f006406cc30 0x7f006406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:43:10.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.574+0000 7f0081600700 1 --2- 172.21.15.123:0/721799200 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f006406cc30 0x7f006406f0e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0070004f40 tx=0x7f007000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:43:10.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.577+0000 7f0079600700 1 -- 172.21.15.123:0/721799200 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f006805ac00 con 0x7f007c1037a0 2024-03-30T04:43:10.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.791+0000 7f0084000700 1 -- 172.21.15.123:0/721799200 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f007c109e10 con 0x7f006406cc30 2024-03-30T04:43:10.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.801+0000 7f0079600700 1 -- 172.21.15.123:0/721799200 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f007c109e10 con 0x7f006406cc30 2024-03-30T04:43:10.799 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:43:10.799 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (12m) 5m ago 24m 15.0M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:43:10.799 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (9m) 5m ago 25m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T04:43:10.799 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (9m) 4m ago 23m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T04:43:10.800 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (12m) 5m ago 24m 74.0M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T04:43:10.800 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (5m) 5m ago 18m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T04:43:10.800 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (5m) 5m ago 18m 10.9M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T04:43:10.800 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (18m) 4m ago 18m 10.7M - 17.2.7-797-g5c024335 f77085f6aab7 0dc41326818c 2024-03-30T04:43:10.800 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (5m) 4m ago 18m 9747k - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T04:43:10.800 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (14m) 5m ago 26m 506M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:43:10.800 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (13m) 4m ago 23m 450M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:43:10.800 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (10m) 5m ago 26m 53.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T04:43:10.800 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (10m) 4m ago 23m 42.6M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T04:43:10.800 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (13m) 5m ago 25m 8052k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:43:10.800 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (13m) 4m ago 23m 7880k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:43:10.800 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (9m) 5m ago 22m 104M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T04:43:10.800 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (8m) 5m ago 22m 93.4M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T04:43:10.800 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (7m) 5m ago 21m 86.3M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T04:43:10.801 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (7m) 4m ago 21m 97.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T04:43:10.801 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (6m) 4m ago 20m 97.9M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T04:43:10.801 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (6m) 4m ago 20m 85.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T04:43:10.801 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (12m) 5m ago 24m 53.7M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:43:10.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.803+0000 7f0084000700 1 -- 172.21.15.123:0/721799200 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f006406cc30 msgr2=0x7f006406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:10.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.803+0000 7f0084000700 1 --2- 172.21.15.123:0/721799200 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f006406cc30 0x7f006406f0e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0070004f40 tx=0x7f007000b040 comp rx=0 tx=0).stop 2024-03-30T04:43:10.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.803+0000 7f0084000700 1 -- 172.21.15.123:0/721799200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f007c1037a0 msgr2=0x7f007c1162e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:10.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.803+0000 7f0084000700 1 --2- 172.21.15.123:0/721799200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f007c1037a0 0x7f007c1162e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f0068009390 tx=0x7f0068009e30 comp rx=0 tx=0).stop 2024-03-30T04:43:10.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.803+0000 7f0084000700 1 -- 172.21.15.123:0/721799200 shutdown_connections 2024-03-30T04:43:10.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.803+0000 7f0084000700 1 --2- 172.21.15.123:0/721799200 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f006406cc30 0x7f006406f0e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:10.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.803+0000 7f0084000700 1 --2- 172.21.15.123:0/721799200 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f007c102e00 0x7f007c115da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:10.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.803+0000 7f0084000700 1 --2- 172.21.15.123:0/721799200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f007c1037a0 0x7f007c1162e0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:10.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.803+0000 7f0084000700 1 -- 172.21.15.123:0/721799200 >> 172.21.15.123:0/721799200 conn(0x7f007c0f89b0 msgr2=0x7f007c0f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:43:10.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.804+0000 7f0084000700 1 -- 172.21.15.123:0/721799200 shutdown_connections 2024-03-30T04:43:10.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.804+0000 7f0084000700 1 -- 172.21.15.123:0/721799200 wait complete. 2024-03-30T04:43:10.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:43:10 smithi135 ceph-mon[106011]: pgmap v359: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:10.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.935+0000 7f33cf600700 1 -- 172.21.15.123:0/1454884791 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33c81037a0 msgr2=0x7f33c80fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:10.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.935+0000 7f33cf600700 1 --2- 172.21.15.123:0/1454884791 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33c81037a0 0x7f33c80fce10 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f33b4004940 tx=0x7f33b4009e30 comp rx=0 tx=0).stop 2024-03-30T04:43:10.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.935+0000 7f33cf600700 1 -- 172.21.15.123:0/1454884791 shutdown_connections 2024-03-30T04:43:10.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.935+0000 7f33cf600700 1 --2- 172.21.15.123:0/1454884791 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33c81037a0 0x7f33c80fce10 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:10.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.935+0000 7f33cf600700 1 --2- 172.21.15.123:0/1454884791 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f33c8102e00 0x7f33c81031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:10.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.935+0000 7f33cf600700 1 -- 172.21.15.123:0/1454884791 >> 172.21.15.123:0/1454884791 conn(0x7f33c80f89b0 msgr2=0x7f33c80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:43:10.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.935+0000 7f33cf600700 1 -- 172.21.15.123:0/1454884791 shutdown_connections 2024-03-30T04:43:10.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.935+0000 7f33cf600700 1 -- 172.21.15.123:0/1454884791 wait complete. 2024-03-30T04:43:10.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.936+0000 7f33cf600700 1 Processor -- start 2024-03-30T04:43:10.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.936+0000 7f33cf600700 1 -- start start 2024-03-30T04:43:10.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.937+0000 7f33cf600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33c8102e00 0x7f33c8115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:43:10.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.937+0000 7f33cf600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f33c81037a0 0x7f33c8116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:43:10.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.937+0000 7f33cf600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33c81168f0 con 0x7f33c8102e00 2024-03-30T04:43:10.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.937+0000 7f33cf600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33c810fc30 con 0x7f33c81037a0 2024-03-30T04:43:10.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.937+0000 7f33ccc00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33c8102e00 0x7f33c8115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:43:10.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.937+0000 7f33ccc00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33c8102e00 0x7f33c8115cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42188/0 (socket says 172.21.15.123:42188) 2024-03-30T04:43:10.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.937+0000 7f33ccc00700 1 -- 172.21.15.123:0/3611496412 learned_addr learned my addr 172.21.15.123:0/3611496412 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:43:10.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.937+0000 7f33c7e00700 1 --2- 172.21.15.123:0/3611496412 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f33c81037a0 0x7f33c8116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:43:10.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.937+0000 7f33ccc00700 1 -- 172.21.15.123:0/3611496412 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f33c81037a0 msgr2=0x7f33c8116210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:10.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.937+0000 7f33ccc00700 1 --2- 172.21.15.123:0/3611496412 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f33c81037a0 0x7f33c8116210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:10.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.937+0000 7f33ccc00700 1 -- 172.21.15.123:0/3611496412 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33b4005040 con 0x7f33c8102e00 2024-03-30T04:43:10.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.938+0000 7f33ccc00700 1 --2- 172.21.15.123:0/3611496412 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33c8102e00 0x7f33c8115cd0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f33bc00b790 tx=0x7f33bc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:43:10.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.938+0000 7f33c4c00700 1 -- 172.21.15.123:0/3611496412 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f33bc00f6f0 con 0x7f33c8102e00 2024-03-30T04:43:10.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.938+0000 7f33c4c00700 1 -- 172.21.15.123:0/3611496412 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f33bc00fd00 con 0x7f33c8102e00 2024-03-30T04:43:10.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.938+0000 7f33c4c00700 1 -- 172.21.15.123:0/3611496412 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f33bc0103a0 con 0x7f33c8102e00 2024-03-30T04:43:10.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.938+0000 7f33cf600700 1 -- 172.21.15.123:0/3611496412 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33c810feb0 con 0x7f33c8102e00 2024-03-30T04:43:10.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.938+0000 7f33cf600700 1 -- 172.21.15.123:0/3611496412 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33c8110470 con 0x7f33c8102e00 2024-03-30T04:43:10.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.939+0000 7f33cf600700 1 -- 172.21.15.123:0/3611496412 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33c80fe550 con 0x7f33c8102e00 2024-03-30T04:43:10.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.940+0000 7f33c4c00700 1 -- 172.21.15.123:0/3611496412 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f33bc00f850 con 0x7f33c8102e00 2024-03-30T04:43:10.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.941+0000 7f33c4c00700 1 --2- 172.21.15.123:0/3611496412 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f33b006cc30 0x7f33b006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:43:10.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.941+0000 7f33c4c00700 1 -- 172.21.15.123:0/3611496412 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f33bc08be60 con 0x7f33c8102e00 2024-03-30T04:43:10.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.941+0000 7f33c7e00700 1 --2- 172.21.15.123:0/3611496412 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f33b006cc30 0x7f33b006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:43:10.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.942+0000 7f33c7e00700 1 --2- 172.21.15.123:0/3611496412 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f33b006cc30 0x7f33b006f0e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f33c8111570 tx=0x7f33b4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:43:10.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:10.945+0000 7f33c4c00700 1 -- 172.21.15.123:0/3611496412 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f33bc059930 con 0x7f33c8102e00 2024-03-30T04:43:10.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:43:10 smithi123 ceph-mon[139369]: pgmap v359: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:11.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.232+0000 7f33cf600700 1 -- 172.21.15.123:0/3611496412 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f33c804bf60 con 0x7f33c8102e00 2024-03-30T04:43:11.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.233+0000 7f33c4c00700 1 -- 172.21.15.123:0/3611496412 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f33bc0594c0 con 0x7f33c8102e00 2024-03-30T04:43:11.231 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:43:11.232 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:43:11.232 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:43:11.232 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:43:11.232 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:43:11.232 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:43:11.232 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:43:11.232 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:43:11.232 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T04:43:11.232 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:43:11.232 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:43:11.232 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:43:11.232 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 3 2024-03-30T04:43:11.232 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:43:11.232 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:43:11.233 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:43:11.233 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 13 2024-03-30T04:43:11.233 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:43:11.233 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:43:11.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.235+0000 7f33cf600700 1 -- 172.21.15.123:0/3611496412 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f33b006cc30 msgr2=0x7f33b006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:11.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.235+0000 7f33cf600700 1 --2- 172.21.15.123:0/3611496412 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f33b006cc30 0x7f33b006f0e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f33c8111570 tx=0x7f33b4009e30 comp rx=0 tx=0).stop 2024-03-30T04:43:11.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.235+0000 7f33cf600700 1 -- 172.21.15.123:0/3611496412 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33c8102e00 msgr2=0x7f33c8115cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:11.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.235+0000 7f33cf600700 1 --2- 172.21.15.123:0/3611496412 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33c8102e00 0x7f33c8115cd0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f33bc00b790 tx=0x7f33bc009250 comp rx=0 tx=0).stop 2024-03-30T04:43:11.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.235+0000 7f33cf600700 1 -- 172.21.15.123:0/3611496412 shutdown_connections 2024-03-30T04:43:11.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.235+0000 7f33cf600700 1 --2- 172.21.15.123:0/3611496412 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f33b006cc30 0x7f33b006f0e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:11.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.235+0000 7f33cf600700 1 --2- 172.21.15.123:0/3611496412 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33c8102e00 0x7f33c8115cd0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:11.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.235+0000 7f33cf600700 1 --2- 172.21.15.123:0/3611496412 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f33c81037a0 0x7f33c8116210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:11.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.235+0000 7f33cf600700 1 -- 172.21.15.123:0/3611496412 >> 172.21.15.123:0/3611496412 conn(0x7f33c80f89b0 msgr2=0x7f33c80f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:43:11.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.236+0000 7f33cf600700 1 -- 172.21.15.123:0/3611496412 shutdown_connections 2024-03-30T04:43:11.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.236+0000 7f33cf600700 1 -- 172.21.15.123:0/3611496412 wait complete. 2024-03-30T04:43:11.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.369+0000 7fe746800700 1 -- 172.21.15.123:0/343280008 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7401037a0 msgr2=0x7fe7400fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:11.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.369+0000 7fe746800700 1 --2- 172.21.15.123:0/343280008 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7401037a0 0x7fe7400fce10 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe72c004940 tx=0x7fe72c009e30 comp rx=0 tx=0).stop 2024-03-30T04:43:11.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.369+0000 7fe746800700 1 -- 172.21.15.123:0/343280008 shutdown_connections 2024-03-30T04:43:11.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.369+0000 7fe746800700 1 --2- 172.21.15.123:0/343280008 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7401037a0 0x7fe7400fce10 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:11.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.369+0000 7fe746800700 1 --2- 172.21.15.123:0/343280008 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe740102e00 0x7fe7401031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:11.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.369+0000 7fe746800700 1 -- 172.21.15.123:0/343280008 >> 172.21.15.123:0/343280008 conn(0x7fe7400f89b0 msgr2=0x7fe7400fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:43:11.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.369+0000 7fe746800700 1 -- 172.21.15.123:0/343280008 shutdown_connections 2024-03-30T04:43:11.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.370+0000 7fe746800700 1 -- 172.21.15.123:0/343280008 wait complete. 2024-03-30T04:43:11.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.370+0000 7fe746800700 1 Processor -- start 2024-03-30T04:43:11.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.370+0000 7fe746800700 1 -- start start 2024-03-30T04:43:11.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.371+0000 7fe746800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe740102e00 0x7fe740113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:43:11.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.371+0000 7fe746800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7401037a0 0x7fe740114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:43:11.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.371+0000 7fe746800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe740114690 con 0x7fe7401037a0 2024-03-30T04:43:11.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.371+0000 7fe746800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe74010da80 con 0x7fe740102e00 2024-03-30T04:43:11.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.371+0000 7fe73ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7401037a0 0x7fe740114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:43:11.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.371+0000 7fe73ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7401037a0 0x7fe740114040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42212/0 (socket says 172.21.15.123:42212) 2024-03-30T04:43:11.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.371+0000 7fe73ea00700 1 -- 172.21.15.123:0/3300130343 learned_addr learned my addr 172.21.15.123:0/3300130343 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:43:11.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.371+0000 7fe73f400700 1 --2- 172.21.15.123:0/3300130343 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe740102e00 0x7fe740113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:43:11.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.371+0000 7fe73ea00700 1 -- 172.21.15.123:0/3300130343 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe740102e00 msgr2=0x7fe740113b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:11.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.371+0000 7fe73ea00700 1 --2- 172.21.15.123:0/3300130343 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe740102e00 0x7fe740113b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:11.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.371+0000 7fe73ea00700 1 -- 172.21.15.123:0/3300130343 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe72c005040 con 0x7fe7401037a0 2024-03-30T04:43:11.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.372+0000 7fe73ea00700 1 --2- 172.21.15.123:0/3300130343 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7401037a0 0x7fe740114040 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fe72c009390 tx=0x7fe72c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:43:11.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.372+0000 7fe737400700 1 -- 172.21.15.123:0/3300130343 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe72c00e040 con 0x7fe7401037a0 2024-03-30T04:43:11.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.372+0000 7fe737400700 1 -- 172.21.15.123:0/3300130343 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe72c013420 con 0x7fe7401037a0 2024-03-30T04:43:11.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.372+0000 7fe746800700 1 -- 172.21.15.123:0/3300130343 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe74010dd00 con 0x7fe7401037a0 2024-03-30T04:43:11.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.372+0000 7fe737400700 1 -- 172.21.15.123:0/3300130343 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe72c01c630 con 0x7fe7401037a0 2024-03-30T04:43:11.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.372+0000 7fe746800700 1 -- 172.21.15.123:0/3300130343 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe74010e2c0 con 0x7fe7401037a0 2024-03-30T04:43:11.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.373+0000 7fe746800700 1 -- 172.21.15.123:0/3300130343 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7400fe550 con 0x7fe7401037a0 2024-03-30T04:43:11.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.374+0000 7fe737400700 1 -- 172.21.15.123:0/3300130343 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fe72c013a00 con 0x7fe7401037a0 2024-03-30T04:43:11.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.375+0000 7fe737400700 1 --2- 172.21.15.123:0/3300130343 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe72406cc30 0x7fe72406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:43:11.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.375+0000 7fe737400700 1 -- 172.21.15.123:0/3300130343 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe72c08d1e0 con 0x7fe7401037a0 2024-03-30T04:43:11.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.375+0000 7fe73f400700 1 --2- 172.21.15.123:0/3300130343 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe72406cc30 0x7fe72406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:43:11.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.376+0000 7fe73f400700 1 --2- 172.21.15.123:0/3300130343 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe72406cc30 0x7fe72406f0e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe730004f40 tx=0x7fe73000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:43:11.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.380+0000 7fe737400700 1 -- 172.21.15.123:0/3300130343 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe72c05ac30 con 0x7fe7401037a0 2024-03-30T04:43:11.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.627+0000 7fe746800700 1 -- 172.21.15.123:0/3300130343 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe74010f150 con 0x7fe7401037a0 2024-03-30T04:43:11.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.628+0000 7fe737400700 1 -- 172.21.15.123:0/3300130343 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1843 (secure 0 0 0) 0x7fe72c05a7c0 con 0x7fe7401037a0 2024-03-30T04:43:11.626 INFO:teuthology.orchestra.run.smithi123.stdout:e32 2024-03-30T04:43:11.626 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:43:11.626 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:43:11.627 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:43:11.627 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:43:11.627 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:43:11.627 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:43:11.627 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T04:43:11.627 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T04:43:11.627 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:43:11.627 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T04:43:11.627 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:43:11.627 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:43:11.627 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:43:11.627 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:43:11.627 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:43:11.627 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T04:43:11.628 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:43:11.628 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:43:11.628 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T04:43:11.628 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:43:11.628 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T04:43:11.628 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T04:43:11.628 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T04:43:11.628 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T04:43:11.628 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:43:11.628 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:43:11.628 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:43:11.628 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:43:11.628 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:43:11.628 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:43:11.628 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T04:43:11.629 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T04:43:11.629 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:43:11.629 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:43:11.629 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T04:43:11.629 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:43:11.629 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:43:11.629 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:43:11.629 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44315} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/1741352336,v1:172.21.15.135:6827/1741352336] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:43:11.629 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:43:11.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.630+0000 7fe746800700 1 -- 172.21.15.123:0/3300130343 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe72406cc30 msgr2=0x7fe72406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:11.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.630+0000 7fe746800700 1 --2- 172.21.15.123:0/3300130343 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe72406cc30 0x7fe72406f0e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe730004f40 tx=0x7fe73000b040 comp rx=0 tx=0).stop 2024-03-30T04:43:11.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.630+0000 7fe746800700 1 -- 172.21.15.123:0/3300130343 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7401037a0 msgr2=0x7fe740114040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:11.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.630+0000 7fe746800700 1 --2- 172.21.15.123:0/3300130343 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7401037a0 0x7fe740114040 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fe72c009390 tx=0x7fe72c009e30 comp rx=0 tx=0).stop 2024-03-30T04:43:11.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.630+0000 7fe746800700 1 -- 172.21.15.123:0/3300130343 shutdown_connections 2024-03-30T04:43:11.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.630+0000 7fe746800700 1 --2- 172.21.15.123:0/3300130343 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe72406cc30 0x7fe72406f0e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:11.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.630+0000 7fe746800700 1 --2- 172.21.15.123:0/3300130343 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe740102e00 0x7fe740113b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:11.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.630+0000 7fe746800700 1 --2- 172.21.15.123:0/3300130343 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7401037a0 0x7fe740114040 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:11.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.630+0000 7fe746800700 1 -- 172.21.15.123:0/3300130343 >> 172.21.15.123:0/3300130343 conn(0x7fe7400f89b0 msgr2=0x7fe7400f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:43:11.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.630+0000 7fe746800700 1 -- 172.21.15.123:0/3300130343 shutdown_connections 2024-03-30T04:43:11.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.630+0000 7fe746800700 1 -- 172.21.15.123:0/3300130343 wait complete. 2024-03-30T04:43:11.631 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 32 2024-03-30T04:43:11.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.765+0000 7f8f2a400700 1 -- 172.21.15.123:0/213200388 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f241037a0 msgr2=0x7f8f240fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:11.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.765+0000 7f8f2a400700 1 --2- 172.21.15.123:0/213200388 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f241037a0 0x7f8f240fce10 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f8f10004940 tx=0x7f8f10009e30 comp rx=0 tx=0).stop 2024-03-30T04:43:11.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.765+0000 7f8f2a400700 1 -- 172.21.15.123:0/213200388 shutdown_connections 2024-03-30T04:43:11.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.765+0000 7f8f2a400700 1 --2- 172.21.15.123:0/213200388 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f241037a0 0x7f8f240fce10 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:11.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.765+0000 7f8f2a400700 1 --2- 172.21.15.123:0/213200388 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8f24102e00 0x7f8f241031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:11.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.765+0000 7f8f2a400700 1 -- 172.21.15.123:0/213200388 >> 172.21.15.123:0/213200388 conn(0x7f8f240f89b0 msgr2=0x7f8f240fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:43:11.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.765+0000 7f8f2a400700 1 -- 172.21.15.123:0/213200388 shutdown_connections 2024-03-30T04:43:11.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.765+0000 7f8f2a400700 1 -- 172.21.15.123:0/213200388 wait complete. 2024-03-30T04:43:11.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.766+0000 7f8f2a400700 1 Processor -- start 2024-03-30T04:43:11.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.766+0000 7f8f2a400700 1 -- start start 2024-03-30T04:43:11.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.767+0000 7f8f2a400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8f24102e00 0x7f8f240702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:43:11.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.767+0000 7f8f2a400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f241037a0 0x7f8f24070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:43:11.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.767+0000 7f8f2a400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f24070d90 con 0x7f8f241037a0 2024-03-30T04:43:11.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.767+0000 7f8f2a400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f24070ed0 con 0x7f8f24102e00 2024-03-30T04:43:11.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.767+0000 7f8f22a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f241037a0 0x7f8f24070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:43:11.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.767+0000 7f8f22a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f241037a0 0x7f8f24070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42234/0 (socket says 172.21.15.123:42234) 2024-03-30T04:43:11.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.767+0000 7f8f22a00700 1 -- 172.21.15.123:0/667422616 learned_addr learned my addr 172.21.15.123:0/667422616 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:43:11.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.767+0000 7f8f23400700 1 --2- 172.21.15.123:0/667422616 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8f24102e00 0x7f8f240702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:43:11.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.767+0000 7f8f22a00700 1 -- 172.21.15.123:0/667422616 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8f24102e00 msgr2=0x7f8f240702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:11.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.767+0000 7f8f22a00700 1 --2- 172.21.15.123:0/667422616 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8f24102e00 0x7f8f240702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:11.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.767+0000 7f8f22a00700 1 -- 172.21.15.123:0/667422616 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f10005040 con 0x7f8f241037a0 2024-03-30T04:43:11.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.767+0000 7f8f23400700 1 --2- 172.21.15.123:0/667422616 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8f24102e00 0x7f8f240702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:43:11.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.768+0000 7f8f22a00700 1 --2- 172.21.15.123:0/667422616 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f241037a0 0x7f8f24070800 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f8f10009390 tx=0x7f8f10009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:43:11.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.768+0000 7f8f1b400700 1 -- 172.21.15.123:0/667422616 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f1000e040 con 0x7f8f241037a0 2024-03-30T04:43:11.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.768+0000 7f8f1b400700 1 -- 172.21.15.123:0/667422616 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8f10013420 con 0x7f8f241037a0 2024-03-30T04:43:11.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.768+0000 7f8f1b400700 1 -- 172.21.15.123:0/667422616 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f1001c630 con 0x7f8f241037a0 2024-03-30T04:43:11.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.768+0000 7f8f2a400700 1 -- 172.21.15.123:0/667422616 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f24071120 con 0x7f8f241037a0 2024-03-30T04:43:11.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.768+0000 7f8f2a400700 1 -- 172.21.15.123:0/667422616 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f2411a230 con 0x7f8f241037a0 2024-03-30T04:43:11.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.769+0000 7f8f2a400700 1 -- 172.21.15.123:0/667422616 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f240fe550 con 0x7f8f241037a0 2024-03-30T04:43:11.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.770+0000 7f8f1b400700 1 -- 172.21.15.123:0/667422616 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f8f10013a00 con 0x7f8f241037a0 2024-03-30T04:43:11.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.771+0000 7f8f1b400700 1 --2- 172.21.15.123:0/667422616 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8f0806cc30 0x7f8f0806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:43:11.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.771+0000 7f8f1b400700 1 -- 172.21.15.123:0/667422616 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f8f1008d2a0 con 0x7f8f241037a0 2024-03-30T04:43:11.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.771+0000 7f8f23400700 1 --2- 172.21.15.123:0/667422616 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8f0806cc30 0x7f8f0806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:43:11.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.772+0000 7f8f23400700 1 --2- 172.21.15.123:0/667422616 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8f0806cc30 0x7f8f0806f0e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8f14009fd0 tx=0x7f8f14009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:43:11.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:11.776+0000 7f8f1b400700 1 -- 172.21.15.123:0/667422616 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8f100572e0 con 0x7f8f241037a0 2024-03-30T04:43:11.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:43:11 smithi135 ceph-mon[106011]: from='client.34710 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:43:11.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:43:11 smithi135 ceph-mon[106011]: from='client.34714 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:43:11.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:43:11 smithi135 ceph-mon[106011]: from='client.34718 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:43:11.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:43:11 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3611496412' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:43:11.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:43:11 smithi123 ceph-mon[139369]: from='client.34710 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:43:11.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:43:11 smithi123 ceph-mon[139369]: from='client.34714 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:43:11.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:43:11 smithi123 ceph-mon[139369]: from='client.34718 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:43:11.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:43:11 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3611496412' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:43:12.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.004+0000 7f8f2a400700 1 -- 172.21.15.123:0/667422616 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8f24109e40 con 0x7f8f0806cc30 2024-03-30T04:43:12.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.006+0000 7f8f1b400700 1 -- 172.21.15.123:0/667422616 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f8f14009970 con 0x7f8f0806cc30 2024-03-30T04:43:12.004 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:43:12.004 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:43:12.004 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:43:12.004 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T04:43:12.004 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:43:12.004 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T04:43:12.004 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T04:43:12.005 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T04:43:12.005 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T04:43:12.005 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:43:12.005 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "15/21 daemons upgraded", 2024-03-30T04:43:12.005 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T04:43:12.005 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:43:12.005 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:43:12.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.009+0000 7f8f2a400700 1 -- 172.21.15.123:0/667422616 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8f0806cc30 msgr2=0x7f8f0806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:12.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.009+0000 7f8f2a400700 1 --2- 172.21.15.123:0/667422616 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8f0806cc30 0x7f8f0806f0e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8f14009fd0 tx=0x7f8f14009ee0 comp rx=0 tx=0).stop 2024-03-30T04:43:12.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.009+0000 7f8f2a400700 1 -- 172.21.15.123:0/667422616 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f241037a0 msgr2=0x7f8f24070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:12.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.009+0000 7f8f2a400700 1 --2- 172.21.15.123:0/667422616 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f241037a0 0x7f8f24070800 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f8f10009390 tx=0x7f8f10009e30 comp rx=0 tx=0).stop 2024-03-30T04:43:12.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.009+0000 7f8f2a400700 1 -- 172.21.15.123:0/667422616 shutdown_connections 2024-03-30T04:43:12.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.009+0000 7f8f2a400700 1 --2- 172.21.15.123:0/667422616 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8f0806cc30 0x7f8f0806f0e0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:12.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.009+0000 7f8f2a400700 1 --2- 172.21.15.123:0/667422616 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8f24102e00 0x7f8f240702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:12.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.009+0000 7f8f2a400700 1 --2- 172.21.15.123:0/667422616 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f241037a0 0x7f8f24070800 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:12.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.009+0000 7f8f2a400700 1 -- 172.21.15.123:0/667422616 >> 172.21.15.123:0/667422616 conn(0x7f8f240f89b0 msgr2=0x7f8f240f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:43:12.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.009+0000 7f8f2a400700 1 -- 172.21.15.123:0/667422616 shutdown_connections 2024-03-30T04:43:12.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.009+0000 7f8f2a400700 1 -- 172.21.15.123:0/667422616 wait complete. 2024-03-30T04:43:12.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.143+0000 7ffa40800700 1 -- 172.21.15.123:0/3654213265 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa38102e00 msgr2=0x7ffa381031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:12.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.143+0000 7ffa40800700 1 --2- 172.21.15.123:0/3654213265 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa38102e00 0x7ffa381031d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c004940 tx=0x7ffa2c009e30 comp rx=0 tx=0).stop 2024-03-30T04:43:12.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.143+0000 7ffa40800700 1 -- 172.21.15.123:0/3654213265 shutdown_connections 2024-03-30T04:43:12.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.143+0000 7ffa40800700 1 --2- 172.21.15.123:0/3654213265 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffa381037a0 0x7ffa380fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:12.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.143+0000 7ffa40800700 1 --2- 172.21.15.123:0/3654213265 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa38102e00 0x7ffa381031d0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:12.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.143+0000 7ffa40800700 1 -- 172.21.15.123:0/3654213265 >> 172.21.15.123:0/3654213265 conn(0x7ffa380f89b0 msgr2=0x7ffa380fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:43:12.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.143+0000 7ffa40800700 1 -- 172.21.15.123:0/3654213265 shutdown_connections 2024-03-30T04:43:12.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.143+0000 7ffa40800700 1 -- 172.21.15.123:0/3654213265 wait complete. 2024-03-30T04:43:12.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.144+0000 7ffa40800700 1 Processor -- start 2024-03-30T04:43:12.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.144+0000 7ffa40800700 1 -- start start 2024-03-30T04:43:12.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.145+0000 7ffa40800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa38102e00 0x7ffa38113a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:43:12.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.145+0000 7ffa40800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffa381037a0 0x7ffa38113f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:43:12.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.145+0000 7ffa40800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa381145d0 con 0x7ffa38102e00 2024-03-30T04:43:12.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.145+0000 7ffa40800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa3810d9c0 con 0x7ffa381037a0 2024-03-30T04:43:12.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.145+0000 7ffa3de00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa38102e00 0x7ffa38113a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:43:12.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.145+0000 7ffa3de00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa38102e00 0x7ffa38113a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42242/0 (socket says 172.21.15.123:42242) 2024-03-30T04:43:12.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.145+0000 7ffa3de00700 1 -- 172.21.15.123:0/2486852199 learned_addr learned my addr 172.21.15.123:0/2486852199 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:43:12.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.145+0000 7ffa3d400700 1 --2- 172.21.15.123:0/2486852199 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffa381037a0 0x7ffa38113f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:43:12.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.145+0000 7ffa3de00700 1 -- 172.21.15.123:0/2486852199 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffa381037a0 msgr2=0x7ffa38113f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:12.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.145+0000 7ffa3de00700 1 --2- 172.21.15.123:0/2486852199 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffa381037a0 0x7ffa38113f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:12.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.145+0000 7ffa3de00700 1 -- 172.21.15.123:0/2486852199 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa2c005040 con 0x7ffa38102e00 2024-03-30T04:43:12.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.145+0000 7ffa3de00700 1 --2- 172.21.15.123:0/2486852199 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa38102e00 0x7ffa38113a40 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c009390 tx=0x7ffa2c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:43:12.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.146+0000 7ffa32000700 1 -- 172.21.15.123:0/2486852199 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa2c00e040 con 0x7ffa38102e00 2024-03-30T04:43:12.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.146+0000 7ffa32000700 1 -- 172.21.15.123:0/2486852199 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffa2c013420 con 0x7ffa38102e00 2024-03-30T04:43:12.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.146+0000 7ffa32000700 1 -- 172.21.15.123:0/2486852199 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa2c01c630 con 0x7ffa38102e00 2024-03-30T04:43:12.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.146+0000 7ffa40800700 1 -- 172.21.15.123:0/2486852199 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa3810dc40 con 0x7ffa38102e00 2024-03-30T04:43:12.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.146+0000 7ffa40800700 1 -- 172.21.15.123:0/2486852199 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa3810e200 con 0x7ffa38102e00 2024-03-30T04:43:12.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.147+0000 7ffa40800700 1 -- 172.21.15.123:0/2486852199 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa380fe550 con 0x7ffa38102e00 2024-03-30T04:43:12.146 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.148+0000 7ffa32000700 1 -- 172.21.15.123:0/2486852199 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7ffa2c0139e0 con 0x7ffa38102e00 2024-03-30T04:43:12.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.149+0000 7ffa32000700 1 --2- 172.21.15.123:0/2486852199 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ffa2006cc30 0x7ffa2006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:43:12.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.149+0000 7ffa32000700 1 -- 172.21.15.123:0/2486852199 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ffa2c08d2a0 con 0x7ffa38102e00 2024-03-30T04:43:12.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.149+0000 7ffa3d400700 1 --2- 172.21.15.123:0/2486852199 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ffa2006cc30 0x7ffa2006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:43:12.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.150+0000 7ffa3d400700 1 --2- 172.21.15.123:0/2486852199 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ffa2006cc30 0x7ffa2006f0e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ffa28009fd0 tx=0x7ffa28009f60 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:43:12.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.154+0000 7ffa32000700 1 -- 172.21.15.123:0/2486852199 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ffa2c05ae20 con 0x7ffa38102e00 2024-03-30T04:43:12.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.435+0000 7ffa40800700 1 -- 172.21.15.123:0/2486852199 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7ffa3804bed0 con 0x7ffa38102e00 2024-03-30T04:43:12.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.435+0000 7ffa32000700 1 -- 172.21.15.123:0/2486852199 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7ffa2c05a9b0 con 0x7ffa38102e00 2024-03-30T04:43:12.434 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T04:43:12.434 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T04:43:12.434 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T04:43:12.434 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T04:43:12.434 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T04:43:12.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.438+0000 7ffa40800700 1 -- 172.21.15.123:0/2486852199 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ffa2006cc30 msgr2=0x7ffa2006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:12.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.438+0000 7ffa40800700 1 --2- 172.21.15.123:0/2486852199 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ffa2006cc30 0x7ffa2006f0e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ffa28009fd0 tx=0x7ffa28009f60 comp rx=0 tx=0).stop 2024-03-30T04:43:12.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.438+0000 7ffa40800700 1 -- 172.21.15.123:0/2486852199 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa38102e00 msgr2=0x7ffa38113a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:12.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.438+0000 7ffa40800700 1 --2- 172.21.15.123:0/2486852199 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa38102e00 0x7ffa38113a40 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c009390 tx=0x7ffa2c009e30 comp rx=0 tx=0).stop 2024-03-30T04:43:12.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.438+0000 7ffa40800700 1 -- 172.21.15.123:0/2486852199 shutdown_connections 2024-03-30T04:43:12.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.438+0000 7ffa40800700 1 --2- 172.21.15.123:0/2486852199 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ffa2006cc30 0x7ffa2006f0e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:12.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.438+0000 7ffa40800700 1 --2- 172.21.15.123:0/2486852199 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa38102e00 0x7ffa38113a40 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:12.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.438+0000 7ffa40800700 1 --2- 172.21.15.123:0/2486852199 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffa381037a0 0x7ffa38113f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:12.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.438+0000 7ffa40800700 1 -- 172.21.15.123:0/2486852199 >> 172.21.15.123:0/2486852199 conn(0x7ffa380f89b0 msgr2=0x7ffa380f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:43:12.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.438+0000 7ffa40800700 1 -- 172.21.15.123:0/2486852199 shutdown_connections 2024-03-30T04:43:12.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:12.438+0000 7ffa40800700 1 -- 172.21.15.123:0/2486852199 wait complete. 2024-03-30T04:43:12.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:43:12 smithi135 ceph-mon[106011]: pgmap v360: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:12.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:43:12 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3300130343' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:43:12.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:43:12 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2486852199' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:43:12.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:43:12 smithi123 ceph-mon[139369]: pgmap v360: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:12.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:43:12 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3300130343' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:43:12.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:43:12 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2486852199' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:43:13.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:43:13 smithi135 ceph-mon[106011]: from='client.34730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:43:13.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:43:13 smithi123 ceph-mon[139369]: from='client.34730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:43:14.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:43:14 smithi135 ceph-mon[106011]: pgmap v361: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:14.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:43:14 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:43:14.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:43:14 smithi123 ceph-mon[139369]: pgmap v361: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:14.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:43:14 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:43:16.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:43:16 smithi135 ceph-mon[106011]: pgmap v362: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:16.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:43:16 smithi123 ceph-mon[139369]: pgmap v362: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:18.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:43:18 smithi135 ceph-mon[106011]: pgmap v363: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:18.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:43:18 smithi123 ceph-mon[139369]: pgmap v363: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:20.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:43:20 smithi135 ceph-mon[106011]: pgmap v364: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:20.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:43:20 smithi123 ceph-mon[139369]: pgmap v364: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:22.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:43:22 smithi135 ceph-mon[106011]: pgmap v365: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:22.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:43:22 smithi123 ceph-mon[139369]: pgmap v365: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:24.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:43:24 smithi135 ceph-mon[106011]: pgmap v366: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:24.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:43:24 smithi123 ceph-mon[139369]: pgmap v366: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:26.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:43:26 smithi135 ceph-mon[106011]: pgmap v367: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:26.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:43:26 smithi123 ceph-mon[139369]: pgmap v367: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:28.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:43:28 smithi135 ceph-mon[106011]: pgmap v368: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:28.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:43:28 smithi123 ceph-mon[139369]: pgmap v368: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:29.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:43:29 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:43:29.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:43:29 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:43:30.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:43:30 smithi135 ceph-mon[106011]: pgmap v369: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:30.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:43:30 smithi123 ceph-mon[139369]: pgmap v369: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:32.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:43:32 smithi123 ceph-mon[139369]: pgmap v370: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:33.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:43:32 smithi135 ceph-mon[106011]: pgmap v370: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:34.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:43:34 smithi123 ceph-mon[139369]: pgmap v371: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:35.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:43:34 smithi135 ceph-mon[106011]: pgmap v371: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:36.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:43:36 smithi123 ceph-mon[139369]: pgmap v372: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:37.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:43:36 smithi135 ceph-mon[106011]: pgmap v372: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:38.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:43:38 smithi123 ceph-mon[139369]: pgmap v373: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:39.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:43:38 smithi135 ceph-mon[106011]: pgmap v373: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:40.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:43:40 smithi135 ceph-mon[106011]: pgmap v374: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:40.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:43:40 smithi123 ceph-mon[139369]: pgmap v374: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:42.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.587+0000 7f6a74a00700 1 -- 172.21.15.123:0/1277493275 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a70102c40 msgr2=0x7f6a70103010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:42.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.587+0000 7f6a74a00700 1 --2- 172.21.15.123:0/1277493275 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a70102c40 0x7f6a70103010 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f6a60004940 tx=0x7f6a60009e30 comp rx=0 tx=0).stop 2024-03-30T04:43:42.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.588+0000 7f6a74a00700 1 -- 172.21.15.123:0/1277493275 shutdown_connections 2024-03-30T04:43:42.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.588+0000 7f6a74a00700 1 --2- 172.21.15.123:0/1277493275 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6a701035e0 0x7f6a700fcc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:42.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.588+0000 7f6a74a00700 1 --2- 172.21.15.123:0/1277493275 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a70102c40 0x7f6a70103010 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:42.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.588+0000 7f6a74a00700 1 -- 172.21.15.123:0/1277493275 >> 172.21.15.123:0/1277493275 conn(0x7f6a700f8830 msgr2=0x7f6a700fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:43:42.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.588+0000 7f6a74a00700 1 -- 172.21.15.123:0/1277493275 shutdown_connections 2024-03-30T04:43:42.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.588+0000 7f6a74a00700 1 -- 172.21.15.123:0/1277493275 wait complete. 2024-03-30T04:43:42.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.589+0000 7f6a74a00700 1 Processor -- start 2024-03-30T04:43:42.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.589+0000 7f6a74a00700 1 -- start start 2024-03-30T04:43:42.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.589+0000 7f6a74a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6a70102c40 0x7f6a70115b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:43:42.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.589+0000 7f6a74a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a701035e0 0x7f6a70116090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:43:42.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.589+0000 7f6a74a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a701166e0 con 0x7f6a701035e0 2024-03-30T04:43:42.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.589+0000 7f6a74a00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a7010fad0 con 0x7f6a70102c40 2024-03-30T04:43:42.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.589+0000 7f6a6ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a701035e0 0x7f6a70116090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:43:42.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.589+0000 7f6a6ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a701035e0 0x7f6a70116090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56358/0 (socket says 172.21.15.123:56358) 2024-03-30T04:43:42.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.590+0000 7f6a6ea00700 1 -- 172.21.15.123:0/2892026211 learned_addr learned my addr 172.21.15.123:0/2892026211 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:43:42.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.590+0000 7f6a6f400700 1 --2- 172.21.15.123:0/2892026211 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6a70102c40 0x7f6a70115b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:43:42.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.590+0000 7f6a6ea00700 1 -- 172.21.15.123:0/2892026211 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6a70102c40 msgr2=0x7f6a70115b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:42.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.590+0000 7f6a6ea00700 1 --2- 172.21.15.123:0/2892026211 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6a70102c40 0x7f6a70115b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:42.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.590+0000 7f6a6ea00700 1 -- 172.21.15.123:0/2892026211 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a60005040 con 0x7f6a701035e0 2024-03-30T04:43:42.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.590+0000 7f6a6ea00700 1 --2- 172.21.15.123:0/2892026211 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a701035e0 0x7f6a70116090 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c002a80 tx=0x7f6a5c004630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:43:42.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.592+0000 7f6a67400700 1 -- 172.21.15.123:0/2892026211 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a5c004e80 con 0x7f6a701035e0 2024-03-30T04:43:42.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.592+0000 7f6a67400700 1 -- 172.21.15.123:0/2892026211 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6a5c00f040 con 0x7f6a701035e0 2024-03-30T04:43:42.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.592+0000 7f6a74a00700 1 -- 172.21.15.123:0/2892026211 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a7010fdb0 con 0x7f6a701035e0 2024-03-30T04:43:42.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.592+0000 7f6a67400700 1 -- 172.21.15.123:0/2892026211 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a5c00da00 con 0x7f6a701035e0 2024-03-30T04:43:42.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.593+0000 7f6a74a00700 1 -- 172.21.15.123:0/2892026211 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a70110370 con 0x7f6a701035e0 2024-03-30T04:43:42.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.594+0000 7f6a74a00700 1 -- 172.21.15.123:0/2892026211 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a700fe390 con 0x7f6a701035e0 2024-03-30T04:43:42.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.595+0000 7f6a67400700 1 -- 172.21.15.123:0/2892026211 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f6a5c007cc0 con 0x7f6a701035e0 2024-03-30T04:43:42.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.596+0000 7f6a67400700 1 --2- 172.21.15.123:0/2892026211 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6a5406cc30 0x7f6a5406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:43:42.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.596+0000 7f6a67400700 1 -- 172.21.15.123:0/2892026211 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f6a5c08b660 con 0x7f6a701035e0 2024-03-30T04:43:42.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.596+0000 7f6a6f400700 1 --2- 172.21.15.123:0/2892026211 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6a5406cc30 0x7f6a5406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:43:42.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.597+0000 7f6a6f400700 1 --2- 172.21.15.123:0/2892026211 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6a5406cc30 0x7f6a5406f0e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6a60009fd0 tx=0x7f6a60009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:43:42.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.601+0000 7f6a67400700 1 -- 172.21.15.123:0/2892026211 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6a5c0590b0 con 0x7f6a701035e0 2024-03-30T04:43:42.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.820+0000 7f6a74a00700 1 -- 172.21.15.123:0/2892026211 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6a70109c80 con 0x7f6a5406cc30 2024-03-30T04:43:42.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.821+0000 7f6a67400700 1 -- 172.21.15.123:0/2892026211 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f6a70109c80 con 0x7f6a5406cc30 2024-03-30T04:43:42.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.823+0000 7f6a74a00700 1 -- 172.21.15.123:0/2892026211 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6a5406cc30 msgr2=0x7f6a5406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:42.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.823+0000 7f6a74a00700 1 --2- 172.21.15.123:0/2892026211 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6a5406cc30 0x7f6a5406f0e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6a60009fd0 tx=0x7f6a60009e30 comp rx=0 tx=0).stop 2024-03-30T04:43:42.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.823+0000 7f6a74a00700 1 -- 172.21.15.123:0/2892026211 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a701035e0 msgr2=0x7f6a70116090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:42.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.823+0000 7f6a74a00700 1 --2- 172.21.15.123:0/2892026211 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a701035e0 0x7f6a70116090 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c002a80 tx=0x7f6a5c004630 comp rx=0 tx=0).stop 2024-03-30T04:43:42.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.823+0000 7f6a74a00700 1 -- 172.21.15.123:0/2892026211 shutdown_connections 2024-03-30T04:43:42.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.823+0000 7f6a74a00700 1 --2- 172.21.15.123:0/2892026211 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6a5406cc30 0x7f6a5406f0e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:42.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.823+0000 7f6a74a00700 1 --2- 172.21.15.123:0/2892026211 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6a70102c40 0x7f6a70115b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:42.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.823+0000 7f6a74a00700 1 --2- 172.21.15.123:0/2892026211 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a701035e0 0x7f6a70116090 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:42.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.823+0000 7f6a74a00700 1 -- 172.21.15.123:0/2892026211 >> 172.21.15.123:0/2892026211 conn(0x7f6a700f8830 msgr2=0x7f6a700f9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:43:42.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.824+0000 7f6a74a00700 1 -- 172.21.15.123:0/2892026211 shutdown_connections 2024-03-30T04:43:42.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.824+0000 7f6a74a00700 1 -- 172.21.15.123:0/2892026211 wait complete. 2024-03-30T04:43:42.836 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:43:42.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:43:42 smithi135 ceph-mon[106011]: pgmap v375: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:42.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.957+0000 7f1f1ac00700 1 -- 172.21.15.123:0/1115549731 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f141037c0 msgr2=0x7f1f140fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:42.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.957+0000 7f1f1ac00700 1 --2- 172.21.15.123:0/1115549731 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f141037c0 0x7f1f140fce30 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f1f00004940 tx=0x7f1f00009e30 comp rx=0 tx=0).stop 2024-03-30T04:43:42.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.957+0000 7f1f1ac00700 1 -- 172.21.15.123:0/1115549731 shutdown_connections 2024-03-30T04:43:42.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.957+0000 7f1f1ac00700 1 --2- 172.21.15.123:0/1115549731 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f141037c0 0x7f1f140fce30 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:42.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.957+0000 7f1f1ac00700 1 --2- 172.21.15.123:0/1115549731 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1f14102e20 0x7f1f141031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:42.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.957+0000 7f1f1ac00700 1 -- 172.21.15.123:0/1115549731 >> 172.21.15.123:0/1115549731 conn(0x7f1f140f89d0 msgr2=0x7f1f140fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:43:42.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.957+0000 7f1f1ac00700 1 -- 172.21.15.123:0/1115549731 shutdown_connections 2024-03-30T04:43:42.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.957+0000 7f1f1ac00700 1 -- 172.21.15.123:0/1115549731 wait complete. 2024-03-30T04:43:42.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.958+0000 7f1f1ac00700 1 Processor -- start 2024-03-30T04:43:42.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.958+0000 7f1f1ac00700 1 -- start start 2024-03-30T04:43:42.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.958+0000 7f1f1ac00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f14102e20 0x7f1f140702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:43:42.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.958+0000 7f1f1ac00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1f141037c0 0x7f1f14070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:43:42.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.958+0000 7f1f1ac00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f14070d90 con 0x7f1f14102e20 2024-03-30T04:43:42.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.958+0000 7f1f1ac00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f14070ed0 con 0x7f1f141037c0 2024-03-30T04:43:42.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.959+0000 7f1f13e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f14102e20 0x7f1f140702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:43:42.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.959+0000 7f1f13e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f14102e20 0x7f1f140702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56376/0 (socket says 172.21.15.123:56376) 2024-03-30T04:43:42.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.959+0000 7f1f13e00700 1 -- 172.21.15.123:0/822928004 learned_addr learned my addr 172.21.15.123:0/822928004 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:43:42.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.959+0000 7f1f13400700 1 --2- 172.21.15.123:0/822928004 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1f141037c0 0x7f1f14070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:43:42.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.959+0000 7f1f13e00700 1 -- 172.21.15.123:0/822928004 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1f141037c0 msgr2=0x7f1f14070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:42.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.959+0000 7f1f13e00700 1 --2- 172.21.15.123:0/822928004 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1f141037c0 0x7f1f14070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:42.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.959+0000 7f1f13e00700 1 -- 172.21.15.123:0/822928004 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f00005040 con 0x7f1f14102e20 2024-03-30T04:43:42.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.959+0000 7f1f13400700 1 --2- 172.21.15.123:0/822928004 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1f141037c0 0x7f1f14070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:43:42.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.959+0000 7f1f13e00700 1 --2- 172.21.15.123:0/822928004 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f14102e20 0x7f1f140702c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f1f0400eb40 tx=0x7f1f0400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:43:42.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.960+0000 7f1f0be00700 1 -- 172.21.15.123:0/822928004 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f04007cc0 con 0x7f1f14102e20 2024-03-30T04:43:42.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.960+0000 7f1f0be00700 1 -- 172.21.15.123:0/822928004 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1f0400f040 con 0x7f1f14102e20 2024-03-30T04:43:42.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.960+0000 7f1f0be00700 1 -- 172.21.15.123:0/822928004 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f04009570 con 0x7f1f14102e20 2024-03-30T04:43:42.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.960+0000 7f1f1ac00700 1 -- 172.21.15.123:0/822928004 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f14071120 con 0x7f1f14102e20 2024-03-30T04:43:42.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.960+0000 7f1f1ac00700 1 -- 172.21.15.123:0/822928004 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f1411a170 con 0x7f1f14102e20 2024-03-30T04:43:42.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.961+0000 7f1f1ac00700 1 -- 172.21.15.123:0/822928004 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f140fe570 con 0x7f1f14102e20 2024-03-30T04:43:42.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.962+0000 7f1f0be00700 1 -- 172.21.15.123:0/822928004 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f1f040096d0 con 0x7f1f14102e20 2024-03-30T04:43:42.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.963+0000 7f1f0be00700 1 --2- 172.21.15.123:0/822928004 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1ef806cc20 0x7f1ef806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:43:42.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.963+0000 7f1f0be00700 1 -- 172.21.15.123:0/822928004 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f1f0408bca0 con 0x7f1f14102e20 2024-03-30T04:43:42.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.963+0000 7f1f13400700 1 --2- 172.21.15.123:0/822928004 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1ef806cc20 0x7f1ef806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:43:42.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.964+0000 7f1f13400700 1 --2- 172.21.15.123:0/822928004 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1ef806cc20 0x7f1ef806f0d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1f0000b0d0 tx=0x7f1f00009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:43:42.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:43:42 smithi123 ceph-mon[139369]: pgmap v375: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:42.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:42.968+0000 7f1f0be00700 1 -- 172.21.15.123:0/822928004 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f1f04055d10 con 0x7f1f14102e20 2024-03-30T04:43:43.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.194+0000 7f1f1ac00700 1 -- 172.21.15.123:0/822928004 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1f140719b0 con 0x7f1ef806cc20 2024-03-30T04:43:43.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.196+0000 7f1f0be00700 1 -- 172.21.15.123:0/822928004 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f1f140719b0 con 0x7f1ef806cc20 2024-03-30T04:43:43.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.198+0000 7f1f1ac00700 1 -- 172.21.15.123:0/822928004 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1ef806cc20 msgr2=0x7f1ef806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:43.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.198+0000 7f1f1ac00700 1 --2- 172.21.15.123:0/822928004 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1ef806cc20 0x7f1ef806f0d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1f0000b0d0 tx=0x7f1f00009e30 comp rx=0 tx=0).stop 2024-03-30T04:43:43.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.198+0000 7f1f1ac00700 1 -- 172.21.15.123:0/822928004 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f14102e20 msgr2=0x7f1f140702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:43.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.198+0000 7f1f1ac00700 1 --2- 172.21.15.123:0/822928004 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f14102e20 0x7f1f140702c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f1f0400eb40 tx=0x7f1f0400b040 comp rx=0 tx=0).stop 2024-03-30T04:43:43.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.198+0000 7f1f1ac00700 1 -- 172.21.15.123:0/822928004 shutdown_connections 2024-03-30T04:43:43.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.198+0000 7f1f1ac00700 1 --2- 172.21.15.123:0/822928004 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1ef806cc20 0x7f1ef806f0d0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:43.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.198+0000 7f1f1ac00700 1 --2- 172.21.15.123:0/822928004 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f14102e20 0x7f1f140702c0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:43.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.198+0000 7f1f1ac00700 1 --2- 172.21.15.123:0/822928004 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1f141037c0 0x7f1f14070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:43.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.198+0000 7f1f1ac00700 1 -- 172.21.15.123:0/822928004 >> 172.21.15.123:0/822928004 conn(0x7f1f140f89d0 msgr2=0x7f1f140f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:43:43.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.199+0000 7f1f1ac00700 1 -- 172.21.15.123:0/822928004 shutdown_connections 2024-03-30T04:43:43.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.199+0000 7f1f1ac00700 1 -- 172.21.15.123:0/822928004 wait complete. 2024-03-30T04:43:43.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.332+0000 7fcdd2e00700 1 -- 172.21.15.123:0/3961427590 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcdcc102e20 msgr2=0x7fcdcc1031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:43.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.332+0000 7fcdd2e00700 1 --2- 172.21.15.123:0/3961427590 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcdcc102e20 0x7fcdcc1031f0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc004940 tx=0x7fcdbc009e30 comp rx=0 tx=0).stop 2024-03-30T04:43:43.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.332+0000 7fcdd2e00700 1 -- 172.21.15.123:0/3961427590 shutdown_connections 2024-03-30T04:43:43.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.332+0000 7fcdd2e00700 1 --2- 172.21.15.123:0/3961427590 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcdcc1037c0 0x7fcdcc0fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:43.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.332+0000 7fcdd2e00700 1 --2- 172.21.15.123:0/3961427590 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcdcc102e20 0x7fcdcc1031f0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:43.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.332+0000 7fcdd2e00700 1 -- 172.21.15.123:0/3961427590 >> 172.21.15.123:0/3961427590 conn(0x7fcdcc0f8970 msgr2=0x7fcdcc0fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:43:43.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.332+0000 7fcdd2e00700 1 -- 172.21.15.123:0/3961427590 shutdown_connections 2024-03-30T04:43:43.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.332+0000 7fcdd2e00700 1 -- 172.21.15.123:0/3961427590 wait complete. 2024-03-30T04:43:43.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.333+0000 7fcdd2e00700 1 Processor -- start 2024-03-30T04:43:43.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.333+0000 7fcdd2e00700 1 -- start start 2024-03-30T04:43:43.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.334+0000 7fcdd2e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcdcc102e20 0x7fcdcc113a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:43:43.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.334+0000 7fcdd2e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcdcc1037c0 0x7fcdcc113f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:43:43.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.334+0000 7fcdd2e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdcc1145e0 con 0x7fcdcc1037c0 2024-03-30T04:43:43.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.334+0000 7fcdd2e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdcc10d9d0 con 0x7fcdcc102e20 2024-03-30T04:43:43.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.334+0000 7fcdcb400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcdcc1037c0 0x7fcdcc113f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:43:43.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.334+0000 7fcdcb400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcdcc1037c0 0x7fcdcc113f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56408/0 (socket says 172.21.15.123:56408) 2024-03-30T04:43:43.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.334+0000 7fcdcb400700 1 -- 172.21.15.123:0/2491433551 learned_addr learned my addr 172.21.15.123:0/2491433551 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:43:43.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.334+0000 7fcdcbe00700 1 --2- 172.21.15.123:0/2491433551 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcdcc102e20 0x7fcdcc113a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:43:43.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.334+0000 7fcdcb400700 1 -- 172.21.15.123:0/2491433551 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcdcc102e20 msgr2=0x7fcdcc113a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:43.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.334+0000 7fcdcb400700 1 --2- 172.21.15.123:0/2491433551 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcdcc102e20 0x7fcdcc113a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:43.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.334+0000 7fcdcb400700 1 -- 172.21.15.123:0/2491433551 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdbc005040 con 0x7fcdcc1037c0 2024-03-30T04:43:43.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.335+0000 7fcdcb400700 1 --2- 172.21.15.123:0/2491433551 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcdcc1037c0 0x7fcdcc113f90 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fcdb8002a50 tx=0x7fcdb80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:43:43.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.335+0000 7fcdc3e00700 1 -- 172.21.15.123:0/2491433551 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcdb8007cc0 con 0x7fcdcc1037c0 2024-03-30T04:43:43.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.335+0000 7fcdc3e00700 1 -- 172.21.15.123:0/2491433551 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcdb800f040 con 0x7fcdcc1037c0 2024-03-30T04:43:43.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.335+0000 7fcdc3e00700 1 -- 172.21.15.123:0/2491433551 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcdb800da30 con 0x7fcdcc1037c0 2024-03-30T04:43:43.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.335+0000 7fcdd2e00700 1 -- 172.21.15.123:0/2491433551 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdcc10dcb0 con 0x7fcdcc1037c0 2024-03-30T04:43:43.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.335+0000 7fcdd2e00700 1 -- 172.21.15.123:0/2491433551 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdcc10e270 con 0x7fcdcc1037c0 2024-03-30T04:43:43.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.336+0000 7fcdd2e00700 1 -- 172.21.15.123:0/2491433551 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdcc0fe570 con 0x7fcdcc1037c0 2024-03-30T04:43:43.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.338+0000 7fcdc3e00700 1 -- 172.21.15.123:0/2491433551 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fcdb800db90 con 0x7fcdcc1037c0 2024-03-30T04:43:43.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.338+0000 7fcdc3e00700 1 --2- 172.21.15.123:0/2491433551 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fcdb006cc30 0x7fcdb006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:43:43.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.339+0000 7fcdc3e00700 1 -- 172.21.15.123:0/2491433551 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fcdb808bca0 con 0x7fcdcc1037c0 2024-03-30T04:43:43.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.339+0000 7fcdcbe00700 1 --2- 172.21.15.123:0/2491433551 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fcdb006cc30 0x7fcdb006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:43:43.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.339+0000 7fcdcbe00700 1 --2- 172.21.15.123:0/2491433551 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fcdb006cc30 0x7fcdb006f0e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc009fd0 tx=0x7fcdbc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:43:43.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.343+0000 7fcdc3e00700 1 -- 172.21.15.123:0/2491433551 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fcdb80596f0 con 0x7fcdcc1037c0 2024-03-30T04:43:43.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.559+0000 7fcdd2e00700 1 -- 172.21.15.123:0/2491433551 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fcdcc109e80 con 0x7fcdb006cc30 2024-03-30T04:43:43.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.569+0000 7fcdc3e00700 1 -- 172.21.15.123:0/2491433551 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fcdbc008ee0 con 0x7fcdb006cc30 2024-03-30T04:43:43.567 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:43:43.567 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (12m) 5m ago 25m 15.0M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:43:43.567 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (10m) 5m ago 25m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T04:43:43.567 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (10m) 5m ago 24m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T04:43:43.567 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (12m) 5m ago 25m 74.0M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T04:43:43.567 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (5m) 5m ago 19m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T04:43:43.567 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (6m) 5m ago 19m 10.9M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T04:43:43.567 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (18m) 5m ago 18m 10.7M - 17.2.7-797-g5c024335 f77085f6aab7 0dc41326818c 2024-03-30T04:43:43.568 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (5m) 5m ago 19m 9747k - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T04:43:43.568 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (15m) 5m ago 27m 506M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:43:43.568 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (14m) 5m ago 24m 450M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:43:43.568 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (11m) 5m ago 27m 53.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T04:43:43.568 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (10m) 5m ago 24m 42.6M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T04:43:43.568 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (13m) 5m ago 25m 8052k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:43:43.568 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (13m) 5m ago 24m 7880k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:43:43.568 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (9m) 5m ago 23m 104M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T04:43:43.568 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (9m) 5m ago 22m 93.4M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T04:43:43.568 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (8m) 5m ago 22m 86.3M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T04:43:43.568 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (7m) 5m ago 21m 97.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T04:43:43.568 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (7m) 5m ago 21m 97.9M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T04:43:43.568 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (6m) 5m ago 20m 85.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T04:43:43.568 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (13m) 5m ago 24m 53.7M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:43:43.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.571+0000 7fcdd2e00700 1 -- 172.21.15.123:0/2491433551 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fcdb006cc30 msgr2=0x7fcdb006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:43.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.571+0000 7fcdd2e00700 1 --2- 172.21.15.123:0/2491433551 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fcdb006cc30 0x7fcdb006f0e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc009fd0 tx=0x7fcdbc009e30 comp rx=0 tx=0).stop 2024-03-30T04:43:43.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.571+0000 7fcdd2e00700 1 -- 172.21.15.123:0/2491433551 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcdcc1037c0 msgr2=0x7fcdcc113f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:43.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.571+0000 7fcdd2e00700 1 --2- 172.21.15.123:0/2491433551 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcdcc1037c0 0x7fcdcc113f90 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fcdb8002a50 tx=0x7fcdb80046b0 comp rx=0 tx=0).stop 2024-03-30T04:43:43.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.571+0000 7fcdd2e00700 1 -- 172.21.15.123:0/2491433551 shutdown_connections 2024-03-30T04:43:43.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.571+0000 7fcdd2e00700 1 --2- 172.21.15.123:0/2491433551 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fcdb006cc30 0x7fcdb006f0e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:43.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.571+0000 7fcdd2e00700 1 --2- 172.21.15.123:0/2491433551 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcdcc102e20 0x7fcdcc113a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:43.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.571+0000 7fcdd2e00700 1 --2- 172.21.15.123:0/2491433551 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcdcc1037c0 0x7fcdcc113f90 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:43.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.571+0000 7fcdd2e00700 1 -- 172.21.15.123:0/2491433551 >> 172.21.15.123:0/2491433551 conn(0x7fcdcc0f8970 msgr2=0x7fcdcc0f99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:43:43.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.571+0000 7fcdd2e00700 1 -- 172.21.15.123:0/2491433551 shutdown_connections 2024-03-30T04:43:43.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.571+0000 7fcdd2e00700 1 -- 172.21.15.123:0/2491433551 wait complete. 2024-03-30T04:43:43.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.705+0000 7f8b85e00700 1 -- 172.21.15.123:0/1086792196 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b80102c40 msgr2=0x7f8b80103010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:43.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.705+0000 7f8b85e00700 1 --2- 172.21.15.123:0/1086792196 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b80102c40 0x7f8b80103010 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f8b74004940 tx=0x7f8b74009e30 comp rx=0 tx=0).stop 2024-03-30T04:43:43.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.705+0000 7f8b85e00700 1 -- 172.21.15.123:0/1086792196 shutdown_connections 2024-03-30T04:43:43.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.705+0000 7f8b85e00700 1 --2- 172.21.15.123:0/1086792196 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8b801035e0 0x7f8b800fcc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:43.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.705+0000 7f8b85e00700 1 --2- 172.21.15.123:0/1086792196 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b80102c40 0x7f8b80103010 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:43.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.705+0000 7f8b85e00700 1 -- 172.21.15.123:0/1086792196 >> 172.21.15.123:0/1086792196 conn(0x7f8b800f8830 msgr2=0x7f8b800fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:43:43.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.705+0000 7f8b85e00700 1 -- 172.21.15.123:0/1086792196 shutdown_connections 2024-03-30T04:43:43.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.705+0000 7f8b85e00700 1 -- 172.21.15.123:0/1086792196 wait complete. 2024-03-30T04:43:43.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.706+0000 7f8b85e00700 1 Processor -- start 2024-03-30T04:43:43.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.707+0000 7f8b85e00700 1 -- start start 2024-03-30T04:43:43.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.707+0000 7f8b85e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b80102c40 0x7f8b800702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:43:43.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.707+0000 7f8b85e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8b801035e0 0x7f8b80070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:43:43.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.707+0000 7f8b85e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b80070d90 con 0x7f8b80102c40 2024-03-30T04:43:43.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.707+0000 7f8b85e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b80070ed0 con 0x7f8b801035e0 2024-03-30T04:43:43.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.707+0000 7f8b84a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b80102c40 0x7f8b800702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:43:43.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.707+0000 7f8b84a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b80102c40 0x7f8b800702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56416/0 (socket says 172.21.15.123:56416) 2024-03-30T04:43:43.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.707+0000 7f8b84a00700 1 -- 172.21.15.123:0/1850217120 learned_addr learned my addr 172.21.15.123:0/1850217120 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:43:43.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.707+0000 7f8b7fe00700 1 --2- 172.21.15.123:0/1850217120 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8b801035e0 0x7f8b80070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:43:43.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.708+0000 7f8b84a00700 1 -- 172.21.15.123:0/1850217120 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8b801035e0 msgr2=0x7f8b80070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:43.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.708+0000 7f8b84a00700 1 --2- 172.21.15.123:0/1850217120 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8b801035e0 0x7f8b80070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:43.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.708+0000 7f8b84a00700 1 -- 172.21.15.123:0/1850217120 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b74005040 con 0x7f8b80102c40 2024-03-30T04:43:43.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.708+0000 7f8b84a00700 1 --2- 172.21.15.123:0/1850217120 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b80102c40 0x7f8b800702c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f8b74009390 tx=0x7f8b74009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:43:43.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.708+0000 7f8b7cc00700 1 -- 172.21.15.123:0/1850217120 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b7400e040 con 0x7f8b80102c40 2024-03-30T04:43:43.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.708+0000 7f8b7cc00700 1 -- 172.21.15.123:0/1850217120 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8b74013420 con 0x7f8b80102c40 2024-03-30T04:43:43.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.708+0000 7f8b7cc00700 1 -- 172.21.15.123:0/1850217120 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b7401c630 con 0x7f8b80102c40 2024-03-30T04:43:43.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.708+0000 7f8b85e00700 1 -- 172.21.15.123:0/1850217120 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b80071120 con 0x7f8b80102c40 2024-03-30T04:43:43.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.709+0000 7f8b85e00700 1 -- 172.21.15.123:0/1850217120 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b8011a080 con 0x7f8b80102c40 2024-03-30T04:43:43.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.710+0000 7f8b85e00700 1 -- 172.21.15.123:0/1850217120 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b800fe390 con 0x7f8b80102c40 2024-03-30T04:43:43.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.711+0000 7f8b7cc00700 1 -- 172.21.15.123:0/1850217120 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f8b740139e0 con 0x7f8b80102c40 2024-03-30T04:43:43.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.711+0000 7f8b7cc00700 1 --2- 172.21.15.123:0/1850217120 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8b6806cc30 0x7f8b6806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:43:43.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.711+0000 7f8b7cc00700 1 -- 172.21.15.123:0/1850217120 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f8b7408d310 con 0x7f8b80102c40 2024-03-30T04:43:43.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.712+0000 7f8b7fe00700 1 --2- 172.21.15.123:0/1850217120 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8b6806cc30 0x7f8b6806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:43:43.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.712+0000 7f8b7fe00700 1 --2- 172.21.15.123:0/1850217120 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8b6806cc30 0x7f8b6806f0e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8b80071c00 tx=0x7f8b6c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:43:43.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:43.716+0000 7f8b7cc00700 1 -- 172.21.15.123:0/1850217120 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8b74057350 con 0x7f8b80102c40 2024-03-30T04:43:43.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:43:43 smithi135 ceph-mon[106011]: from='client.34738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:43:43.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:43:43 smithi123 ceph-mon[139369]: from='client.34738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:43:44.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.008+0000 7f8b85e00700 1 -- 172.21.15.123:0/1850217120 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f8b800719b0 con 0x7f8b80102c40 2024-03-30T04:43:44.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.009+0000 7f8b7cc00700 1 -- 172.21.15.123:0/1850217120 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f8b7405a970 con 0x7f8b80102c40 2024-03-30T04:43:44.007 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:43:44.007 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:43:44.007 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:43:44.007 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:43:44.008 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:43:44.008 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:43:44.008 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:43:44.008 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:43:44.008 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T04:43:44.008 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:43:44.008 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:43:44.008 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:43:44.008 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 3 2024-03-30T04:43:44.008 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:43:44.008 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:43:44.008 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:43:44.008 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 13 2024-03-30T04:43:44.009 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:43:44.009 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:43:44.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.011+0000 7f8b85e00700 1 -- 172.21.15.123:0/1850217120 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8b6806cc30 msgr2=0x7f8b6806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:44.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.011+0000 7f8b85e00700 1 --2- 172.21.15.123:0/1850217120 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8b6806cc30 0x7f8b6806f0e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8b80071c00 tx=0x7f8b6c00b040 comp rx=0 tx=0).stop 2024-03-30T04:43:44.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.011+0000 7f8b85e00700 1 -- 172.21.15.123:0/1850217120 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b80102c40 msgr2=0x7f8b800702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:44.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.011+0000 7f8b85e00700 1 --2- 172.21.15.123:0/1850217120 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b80102c40 0x7f8b800702c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f8b74009390 tx=0x7f8b74009e30 comp rx=0 tx=0).stop 2024-03-30T04:43:44.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.011+0000 7f8b85e00700 1 -- 172.21.15.123:0/1850217120 shutdown_connections 2024-03-30T04:43:44.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.011+0000 7f8b85e00700 1 --2- 172.21.15.123:0/1850217120 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8b6806cc30 0x7f8b6806f0e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:44.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.011+0000 7f8b85e00700 1 --2- 172.21.15.123:0/1850217120 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b80102c40 0x7f8b800702c0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:44.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.011+0000 7f8b85e00700 1 --2- 172.21.15.123:0/1850217120 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8b801035e0 0x7f8b80070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:44.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.011+0000 7f8b85e00700 1 -- 172.21.15.123:0/1850217120 >> 172.21.15.123:0/1850217120 conn(0x7f8b800f8830 msgr2=0x7f8b800f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:43:44.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.011+0000 7f8b85e00700 1 -- 172.21.15.123:0/1850217120 shutdown_connections 2024-03-30T04:43:44.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.012+0000 7f8b85e00700 1 -- 172.21.15.123:0/1850217120 wait complete. 2024-03-30T04:43:44.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.145+0000 7fea9f200700 1 -- 172.21.15.123:0/1259992110 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea981035f0 msgr2=0x7fea980fcc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:44.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.145+0000 7fea9f200700 1 --2- 172.21.15.123:0/1259992110 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea981035f0 0x7fea980fcc60 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fea88004940 tx=0x7fea88009e30 comp rx=0 tx=0).stop 2024-03-30T04:43:44.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.145+0000 7fea9f200700 1 -- 172.21.15.123:0/1259992110 shutdown_connections 2024-03-30T04:43:44.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.145+0000 7fea9f200700 1 --2- 172.21.15.123:0/1259992110 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea981035f0 0x7fea980fcc60 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:44.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.145+0000 7fea9f200700 1 --2- 172.21.15.123:0/1259992110 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea98102c50 0x7fea98103020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:44.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.145+0000 7fea9f200700 1 -- 172.21.15.123:0/1259992110 >> 172.21.15.123:0/1259992110 conn(0x7fea980f87e0 msgr2=0x7fea980fabf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:43:44.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.145+0000 7fea9f200700 1 -- 172.21.15.123:0/1259992110 shutdown_connections 2024-03-30T04:43:44.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.145+0000 7fea9f200700 1 -- 172.21.15.123:0/1259992110 wait complete. 2024-03-30T04:43:44.146 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.146+0000 7fea9f200700 1 Processor -- start 2024-03-30T04:43:44.146 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.146+0000 7fea9f200700 1 -- start start 2024-03-30T04:43:44.146 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.147+0000 7fea9f200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea98102c50 0x7fea98115a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:43:44.146 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.147+0000 7fea9f200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea981035f0 0x7fea98115fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:43:44.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.147+0000 7fea9f200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea98116620 con 0x7fea98102c50 2024-03-30T04:43:44.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.147+0000 7fea9f200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea9810fa10 con 0x7fea981035f0 2024-03-30T04:43:44.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.147+0000 7fea9de00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea98102c50 0x7fea98115a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:43:44.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.147+0000 7fea9de00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea98102c50 0x7fea98115a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46986/0 (socket says 172.21.15.123:46986) 2024-03-30T04:43:44.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.147+0000 7fea9de00700 1 -- 172.21.15.123:0/315870605 learned_addr learned my addr 172.21.15.123:0/315870605 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:43:44.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.147+0000 7fea9d400700 1 --2- 172.21.15.123:0/315870605 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea981035f0 0x7fea98115fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:43:44.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.147+0000 7fea9de00700 1 -- 172.21.15.123:0/315870605 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea981035f0 msgr2=0x7fea98115fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:44.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.147+0000 7fea9de00700 1 --2- 172.21.15.123:0/315870605 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea981035f0 0x7fea98115fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:44.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.147+0000 7fea9de00700 1 -- 172.21.15.123:0/315870605 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea88005040 con 0x7fea98102c50 2024-03-30T04:43:44.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.147+0000 7fea9de00700 1 --2- 172.21.15.123:0/315870605 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea98102c50 0x7fea98115a90 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fea8c00eb40 tx=0x7fea8c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:43:44.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.148+0000 7fea92000700 1 -- 172.21.15.123:0/315870605 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea8c007cc0 con 0x7fea98102c50 2024-03-30T04:43:44.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.148+0000 7fea92000700 1 -- 172.21.15.123:0/315870605 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fea8c00f040 con 0x7fea98102c50 2024-03-30T04:43:44.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.148+0000 7fea9f200700 1 -- 172.21.15.123:0/315870605 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea9810fc90 con 0x7fea98102c50 2024-03-30T04:43:44.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.148+0000 7fea92000700 1 -- 172.21.15.123:0/315870605 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea8c009570 con 0x7fea98102c50 2024-03-30T04:43:44.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.148+0000 7fea9f200700 1 -- 172.21.15.123:0/315870605 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea98110220 con 0x7fea98102c50 2024-03-30T04:43:44.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.149+0000 7fea9f200700 1 -- 172.21.15.123:0/315870605 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea980fe3a0 con 0x7fea98102c50 2024-03-30T04:43:44.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.150+0000 7fea92000700 1 -- 172.21.15.123:0/315870605 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fea8c0096d0 con 0x7fea98102c50 2024-03-30T04:43:44.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.150+0000 7fea92000700 1 --2- 172.21.15.123:0/315870605 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fea8006cc30 0x7fea8006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:43:44.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.151+0000 7fea92000700 1 -- 172.21.15.123:0/315870605 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fea8c08bcb0 con 0x7fea98102c50 2024-03-30T04:43:44.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.151+0000 7fea9d400700 1 --2- 172.21.15.123:0/315870605 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fea8006cc30 0x7fea8006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:43:44.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.151+0000 7fea9d400700 1 --2- 172.21.15.123:0/315870605 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fea8006cc30 0x7fea8006f0e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fea981113c0 tx=0x7fea88009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:43:44.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.155+0000 7fea92000700 1 -- 172.21.15.123:0/315870605 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fea8c0597b0 con 0x7fea98102c50 2024-03-30T04:43:44.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.402+0000 7fea9f200700 1 -- 172.21.15.123:0/315870605 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fea98109be0 con 0x7fea98102c50 2024-03-30T04:43:44.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.402+0000 7fea92000700 1 -- 172.21.15.123:0/315870605 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1843 (secure 0 0 0) 0x7fea8c019750 con 0x7fea98102c50 2024-03-30T04:43:44.401 INFO:teuthology.orchestra.run.smithi123.stdout:e32 2024-03-30T04:43:44.401 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:43:44.401 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:43:44.401 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:43:44.401 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:43:44.401 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:43:44.401 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:43:44.401 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T04:43:44.401 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T04:43:44.401 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:43:44.402 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T04:43:44.402 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:43:44.402 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:43:44.402 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:43:44.402 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:43:44.402 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:43:44.402 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T04:43:44.402 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:43:44.402 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:43:44.402 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T04:43:44.402 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:43:44.402 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T04:43:44.403 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T04:43:44.403 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T04:43:44.403 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T04:43:44.403 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:43:44.403 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:43:44.403 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:43:44.403 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:43:44.403 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:43:44.403 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:43:44.403 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T04:43:44.403 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T04:43:44.403 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:43:44.404 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:43:44.404 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T04:43:44.404 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:43:44.404 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:43:44.404 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:43:44.404 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44315} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/1741352336,v1:172.21.15.135:6827/1741352336] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:43:44.404 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:43:44.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.404+0000 7fea9f200700 1 -- 172.21.15.123:0/315870605 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fea8006cc30 msgr2=0x7fea8006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:44.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.404+0000 7fea9f200700 1 --2- 172.21.15.123:0/315870605 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fea8006cc30 0x7fea8006f0e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fea981113c0 tx=0x7fea88009e30 comp rx=0 tx=0).stop 2024-03-30T04:43:44.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.404+0000 7fea9f200700 1 -- 172.21.15.123:0/315870605 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea98102c50 msgr2=0x7fea98115a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:44.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.404+0000 7fea9f200700 1 --2- 172.21.15.123:0/315870605 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea98102c50 0x7fea98115a90 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fea8c00eb40 tx=0x7fea8c00b040 comp rx=0 tx=0).stop 2024-03-30T04:43:44.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.405+0000 7fea9f200700 1 -- 172.21.15.123:0/315870605 shutdown_connections 2024-03-30T04:43:44.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.405+0000 7fea9f200700 1 --2- 172.21.15.123:0/315870605 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fea8006cc30 0x7fea8006f0e0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:44.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.405+0000 7fea9f200700 1 --2- 172.21.15.123:0/315870605 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea98102c50 0x7fea98115a90 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:44.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.405+0000 7fea9f200700 1 --2- 172.21.15.123:0/315870605 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea981035f0 0x7fea98115fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:44.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.405+0000 7fea9f200700 1 -- 172.21.15.123:0/315870605 >> 172.21.15.123:0/315870605 conn(0x7fea980f87e0 msgr2=0x7fea980f97d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:43:44.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.405+0000 7fea9f200700 1 -- 172.21.15.123:0/315870605 shutdown_connections 2024-03-30T04:43:44.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.405+0000 7fea9f200700 1 -- 172.21.15.123:0/315870605 wait complete. 2024-03-30T04:43:44.406 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 32 2024-03-30T04:43:44.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.543+0000 7fbcb9c00700 1 -- 172.21.15.123:0/4072524062 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbcb4102e00 msgr2=0x7fbcb41031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:44.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.543+0000 7fbcb9c00700 1 --2- 172.21.15.123:0/4072524062 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbcb4102e00 0x7fbcb41031d0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fbca4004940 tx=0x7fbca4009e30 comp rx=0 tx=0).stop 2024-03-30T04:43:44.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.543+0000 7fbcb9c00700 1 -- 172.21.15.123:0/4072524062 shutdown_connections 2024-03-30T04:43:44.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.543+0000 7fbcb9c00700 1 --2- 172.21.15.123:0/4072524062 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbcb41037a0 0x7fbcb40fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:44.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.543+0000 7fbcb9c00700 1 --2- 172.21.15.123:0/4072524062 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbcb4102e00 0x7fbcb41031d0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:44.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.543+0000 7fbcb9c00700 1 -- 172.21.15.123:0/4072524062 >> 172.21.15.123:0/4072524062 conn(0x7fbcb40f89b0 msgr2=0x7fbcb40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:43:44.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.543+0000 7fbcb9c00700 1 -- 172.21.15.123:0/4072524062 shutdown_connections 2024-03-30T04:43:44.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.543+0000 7fbcb9c00700 1 -- 172.21.15.123:0/4072524062 wait complete. 2024-03-30T04:43:44.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.544+0000 7fbcb9c00700 1 Processor -- start 2024-03-30T04:43:44.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.544+0000 7fbcb9c00700 1 -- start start 2024-03-30T04:43:44.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.545+0000 7fbcb9c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbcb4102e00 0x7fbcb40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:43:44.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.545+0000 7fbcb9c00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbcb41037a0 0x7fbcb4070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:43:44.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.545+0000 7fbcb9c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb4070d90 con 0x7fbcb4102e00 2024-03-30T04:43:44.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.545+0000 7fbcb9c00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb4070ed0 con 0x7fbcb41037a0 2024-03-30T04:43:44.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.545+0000 7fbcb2a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbcb4102e00 0x7fbcb40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:43:44.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.545+0000 7fbcb2a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbcb4102e00 0x7fbcb40702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47002/0 (socket says 172.21.15.123:47002) 2024-03-30T04:43:44.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.545+0000 7fbcb2a00700 1 -- 172.21.15.123:0/4278032640 learned_addr learned my addr 172.21.15.123:0/4278032640 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:43:44.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.545+0000 7fbcb2000700 1 --2- 172.21.15.123:0/4278032640 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbcb41037a0 0x7fbcb4070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:43:44.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.545+0000 7fbcb2a00700 1 -- 172.21.15.123:0/4278032640 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbcb41037a0 msgr2=0x7fbcb4070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:44.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.545+0000 7fbcb2a00700 1 --2- 172.21.15.123:0/4278032640 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbcb41037a0 0x7fbcb4070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:44.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.545+0000 7fbcb2a00700 1 -- 172.21.15.123:0/4278032640 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca4005040 con 0x7fbcb4102e00 2024-03-30T04:43:44.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.545+0000 7fbcb2000700 1 --2- 172.21.15.123:0/4278032640 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbcb41037a0 0x7fbcb4070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:43:44.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.546+0000 7fbcb2a00700 1 --2- 172.21.15.123:0/4278032640 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbcb4102e00 0x7fbcb40702c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fbca4009390 tx=0x7fbca4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:43:44.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.546+0000 7fbcaaa00700 1 -- 172.21.15.123:0/4278032640 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbca400e040 con 0x7fbcb4102e00 2024-03-30T04:43:44.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.546+0000 7fbcaaa00700 1 -- 172.21.15.123:0/4278032640 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbca4013420 con 0x7fbcb4102e00 2024-03-30T04:43:44.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.546+0000 7fbcaaa00700 1 -- 172.21.15.123:0/4278032640 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbca401c630 con 0x7fbcb4102e00 2024-03-30T04:43:44.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.546+0000 7fbcb9c00700 1 -- 172.21.15.123:0/4278032640 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcb4071120 con 0x7fbcb4102e00 2024-03-30T04:43:44.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.546+0000 7fbcb9c00700 1 -- 172.21.15.123:0/4278032640 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcb411a250 con 0x7fbcb4102e00 2024-03-30T04:43:44.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.547+0000 7fbcb9c00700 1 -- 172.21.15.123:0/4278032640 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcb40fe550 con 0x7fbcb4102e00 2024-03-30T04:43:44.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.548+0000 7fbcaaa00700 1 -- 172.21.15.123:0/4278032640 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fbca40139e0 con 0x7fbcb4102e00 2024-03-30T04:43:44.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.549+0000 7fbcaaa00700 1 --2- 172.21.15.123:0/4278032640 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbc9806cc30 0x7fbc9806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:43:44.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.549+0000 7fbcaaa00700 1 -- 172.21.15.123:0/4278032640 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fbca408d1a0 con 0x7fbcb4102e00 2024-03-30T04:43:44.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.549+0000 7fbcb2000700 1 --2- 172.21.15.123:0/4278032640 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbc9806cc30 0x7fbc9806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:43:44.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.550+0000 7fbcb2000700 1 --2- 172.21.15.123:0/4278032640 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbc9806cc30 0x7fbc9806f0e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fbca0009fd0 tx=0x7fbca0009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:43:44.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.553+0000 7fbcaaa00700 1 -- 172.21.15.123:0/4278032640 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fbca40572a0 con 0x7fbcb4102e00 2024-03-30T04:43:44.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.775+0000 7fbcb9c00700 1 -- 172.21.15.123:0/4278032640 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbcb40719b0 con 0x7fbc9806cc30 2024-03-30T04:43:44.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.777+0000 7fbcaaa00700 1 -- 172.21.15.123:0/4278032640 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fbcb40719b0 con 0x7fbc9806cc30 2024-03-30T04:43:44.775 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:43:44.775 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:43:44.775 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:43:44.775 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T04:43:44.775 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:43:44.775 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T04:43:44.775 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T04:43:44.776 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T04:43:44.776 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T04:43:44.776 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:43:44.776 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "15/21 daemons upgraded", 2024-03-30T04:43:44.776 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T04:43:44.776 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:43:44.776 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:43:44.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.779+0000 7fbcb9c00700 1 -- 172.21.15.123:0/4278032640 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbc9806cc30 msgr2=0x7fbc9806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:44.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.779+0000 7fbcb9c00700 1 --2- 172.21.15.123:0/4278032640 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbc9806cc30 0x7fbc9806f0e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fbca0009fd0 tx=0x7fbca0009ee0 comp rx=0 tx=0).stop 2024-03-30T04:43:44.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.779+0000 7fbcb9c00700 1 -- 172.21.15.123:0/4278032640 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbcb4102e00 msgr2=0x7fbcb40702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:44.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.779+0000 7fbcb9c00700 1 --2- 172.21.15.123:0/4278032640 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbcb4102e00 0x7fbcb40702c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fbca4009390 tx=0x7fbca4009e30 comp rx=0 tx=0).stop 2024-03-30T04:43:44.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.779+0000 7fbcb9c00700 1 -- 172.21.15.123:0/4278032640 shutdown_connections 2024-03-30T04:43:44.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.779+0000 7fbcb9c00700 1 --2- 172.21.15.123:0/4278032640 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbc9806cc30 0x7fbc9806f0e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:44.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.779+0000 7fbcb9c00700 1 --2- 172.21.15.123:0/4278032640 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbcb4102e00 0x7fbcb40702c0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:44.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.779+0000 7fbcb9c00700 1 --2- 172.21.15.123:0/4278032640 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbcb41037a0 0x7fbcb4070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:44.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.779+0000 7fbcb9c00700 1 -- 172.21.15.123:0/4278032640 >> 172.21.15.123:0/4278032640 conn(0x7fbcb40f89b0 msgr2=0x7fbcb40f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:43:44.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.779+0000 7fbcb9c00700 1 -- 172.21.15.123:0/4278032640 shutdown_connections 2024-03-30T04:43:44.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.779+0000 7fbcb9c00700 1 -- 172.21.15.123:0/4278032640 wait complete. 2024-03-30T04:43:44.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:43:44 smithi135 ceph-mon[106011]: from='client.34742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:43:44.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:43:44 smithi135 ceph-mon[106011]: pgmap v376: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:44.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:43:44 smithi135 ceph-mon[106011]: from='client.34746 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:43:44.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:43:44 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/1850217120' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:43:44.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:43:44 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/315870605' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:43:44.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:43:44 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:43:44.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.912+0000 7fde4f600700 1 -- 172.21.15.123:0/2313959189 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde481037c0 msgr2=0x7fde480fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:44.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.912+0000 7fde4f600700 1 --2- 172.21.15.123:0/2313959189 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde481037c0 0x7fde480fce30 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fde34004940 tx=0x7fde34009e30 comp rx=0 tx=0).stop 2024-03-30T04:43:44.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.912+0000 7fde4f600700 1 -- 172.21.15.123:0/2313959189 shutdown_connections 2024-03-30T04:43:44.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.912+0000 7fde4f600700 1 --2- 172.21.15.123:0/2313959189 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde481037c0 0x7fde480fce30 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:44.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.912+0000 7fde4f600700 1 --2- 172.21.15.123:0/2313959189 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fde48102e20 0x7fde481031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:44.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.912+0000 7fde4f600700 1 -- 172.21.15.123:0/2313959189 >> 172.21.15.123:0/2313959189 conn(0x7fde480f89d0 msgr2=0x7fde480fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:43:44.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.912+0000 7fde4f600700 1 -- 172.21.15.123:0/2313959189 shutdown_connections 2024-03-30T04:43:44.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.912+0000 7fde4f600700 1 -- 172.21.15.123:0/2313959189 wait complete. 2024-03-30T04:43:44.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.913+0000 7fde4f600700 1 Processor -- start 2024-03-30T04:43:44.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.914+0000 7fde4f600700 1 -- start start 2024-03-30T04:43:44.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.914+0000 7fde4f600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde48102e20 0x7fde480702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:43:44.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.914+0000 7fde4f600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fde481037c0 0x7fde48070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:43:44.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.914+0000 7fde4f600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde48070d90 con 0x7fde48102e20 2024-03-30T04:43:44.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.914+0000 7fde4f600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde48070ed0 con 0x7fde481037c0 2024-03-30T04:43:44.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.914+0000 7fde4cc00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde48102e20 0x7fde480702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:43:44.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.914+0000 7fde4cc00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde48102e20 0x7fde480702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47026/0 (socket says 172.21.15.123:47026) 2024-03-30T04:43:44.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.914+0000 7fde4cc00700 1 -- 172.21.15.123:0/561063460 learned_addr learned my addr 172.21.15.123:0/561063460 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:43:44.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.914+0000 7fde47e00700 1 --2- 172.21.15.123:0/561063460 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fde481037c0 0x7fde48070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:43:44.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.915+0000 7fde4cc00700 1 -- 172.21.15.123:0/561063460 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fde481037c0 msgr2=0x7fde48070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:44.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.915+0000 7fde4cc00700 1 --2- 172.21.15.123:0/561063460 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fde481037c0 0x7fde48070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:44.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.915+0000 7fde4cc00700 1 -- 172.21.15.123:0/561063460 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde34005040 con 0x7fde48102e20 2024-03-30T04:43:44.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.915+0000 7fde4cc00700 1 --2- 172.21.15.123:0/561063460 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde48102e20 0x7fde480702c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fde3c002a50 tx=0x7fde3c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:43:44.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.915+0000 7fde44c00700 1 -- 172.21.15.123:0/561063460 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde3c007cc0 con 0x7fde48102e20 2024-03-30T04:43:44.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.915+0000 7fde44c00700 1 -- 172.21.15.123:0/561063460 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fde3c00f040 con 0x7fde48102e20 2024-03-30T04:43:44.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.915+0000 7fde4f600700 1 -- 172.21.15.123:0/561063460 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde48119dd0 con 0x7fde48102e20 2024-03-30T04:43:44.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.915+0000 7fde44c00700 1 -- 172.21.15.123:0/561063460 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde3c00d9c0 con 0x7fde48102e20 2024-03-30T04:43:44.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.915+0000 7fde4f600700 1 -- 172.21.15.123:0/561063460 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde4811a250 con 0x7fde48102e20 2024-03-30T04:43:44.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.917+0000 7fde4f600700 1 -- 172.21.15.123:0/561063460 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde480fe570 con 0x7fde48102e20 2024-03-30T04:43:44.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.918+0000 7fde44c00700 1 -- 172.21.15.123:0/561063460 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fde3c00db20 con 0x7fde48102e20 2024-03-30T04:43:44.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.918+0000 7fde44c00700 1 --2- 172.21.15.123:0/561063460 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fde3006cc30 0x7fde3006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:43:44.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.919+0000 7fde44c00700 1 -- 172.21.15.123:0/561063460 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fde3c08bc80 con 0x7fde48102e20 2024-03-30T04:43:44.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.919+0000 7fde47e00700 1 --2- 172.21.15.123:0/561063460 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fde3006cc30 0x7fde3006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:43:44.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.919+0000 7fde47e00700 1 --2- 172.21.15.123:0/561063460 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fde3006cc30 0x7fde3006f0e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fde48071c00 tx=0x7fde34009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:43:44.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:44.923+0000 7fde44c00700 1 -- 172.21.15.123:0/561063460 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fde3c055cc0 con 0x7fde48102e20 2024-03-30T04:43:44.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:43:44 smithi123 ceph-mon[139369]: from='client.34742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:43:44.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:43:44 smithi123 ceph-mon[139369]: pgmap v376: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:44.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:43:44 smithi123 ceph-mon[139369]: from='client.34746 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:43:44.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:43:44 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/1850217120' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:43:44.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:43:44 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/315870605' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:43:44.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:43:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:43:45.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:45.204+0000 7fde4f600700 1 -- 172.21.15.123:0/561063460 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fde4805f5f0 con 0x7fde48102e20 2024-03-30T04:43:45.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:45.204+0000 7fde44c00700 1 -- 172.21.15.123:0/561063460 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fde4805f5f0 con 0x7fde48102e20 2024-03-30T04:43:45.203 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T04:43:45.203 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T04:43:45.203 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T04:43:45.203 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T04:43:45.203 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T04:43:45.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:45.206+0000 7fde4f600700 1 -- 172.21.15.123:0/561063460 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fde3006cc30 msgr2=0x7fde3006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:45.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:45.206+0000 7fde4f600700 1 --2- 172.21.15.123:0/561063460 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fde3006cc30 0x7fde3006f0e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fde48071c00 tx=0x7fde34009e30 comp rx=0 tx=0).stop 2024-03-30T04:43:45.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:45.206+0000 7fde4f600700 1 -- 172.21.15.123:0/561063460 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde48102e20 msgr2=0x7fde480702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:43:45.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:45.207+0000 7fde4f600700 1 --2- 172.21.15.123:0/561063460 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde48102e20 0x7fde480702c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fde3c002a50 tx=0x7fde3c0046b0 comp rx=0 tx=0).stop 2024-03-30T04:43:45.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:45.207+0000 7fde4f600700 1 -- 172.21.15.123:0/561063460 shutdown_connections 2024-03-30T04:43:45.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:45.207+0000 7fde4f600700 1 --2- 172.21.15.123:0/561063460 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fde3006cc30 0x7fde3006f0e0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:45.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:45.207+0000 7fde4f600700 1 --2- 172.21.15.123:0/561063460 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde48102e20 0x7fde480702c0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:45.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:45.207+0000 7fde4f600700 1 --2- 172.21.15.123:0/561063460 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fde481037c0 0x7fde48070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:43:45.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:45.207+0000 7fde4f600700 1 -- 172.21.15.123:0/561063460 >> 172.21.15.123:0/561063460 conn(0x7fde480f89d0 msgr2=0x7fde480f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:43:45.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:45.207+0000 7fde4f600700 1 -- 172.21.15.123:0/561063460 shutdown_connections 2024-03-30T04:43:45.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:43:45.207+0000 7fde4f600700 1 -- 172.21.15.123:0/561063460 wait complete. 2024-03-30T04:43:45.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:43:45 smithi123 ceph-mon[139369]: from='client.34758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:43:45.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:43:45 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/561063460' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:43:46.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:43:45 smithi135 ceph-mon[106011]: from='client.34758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:43:46.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:43:45 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/561063460' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:43:46.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:43:46 smithi123 ceph-mon[139369]: pgmap v377: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:47.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:43:46 smithi135 ceph-mon[106011]: pgmap v377: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:48.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:43:48 smithi123 ceph-mon[139369]: pgmap v378: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:49.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:43:48 smithi135 ceph-mon[106011]: pgmap v378: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:50.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:43:50 smithi135 ceph-mon[106011]: pgmap v379: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:50.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:43:50 smithi123 ceph-mon[139369]: pgmap v379: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:52.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:43:52 smithi135 ceph-mon[106011]: pgmap v380: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:52.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:43:52 smithi123 ceph-mon[139369]: pgmap v380: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:54.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:43:54 smithi135 ceph-mon[106011]: pgmap v381: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:54.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:43:54 smithi123 ceph-mon[139369]: pgmap v381: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:56.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:43:56 smithi135 ceph-mon[106011]: pgmap v382: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:56.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:43:56 smithi123 ceph-mon[139369]: pgmap v382: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:58.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:43:58 smithi123 ceph-mon[139369]: pgmap v383: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:59.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:43:58 smithi135 ceph-mon[106011]: pgmap v383: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:43:59.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:43:59 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:44:00.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:43:59 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:44:00.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:44:00 smithi123 ceph-mon[139369]: pgmap v384: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:01.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:44:00 smithi135 ceph-mon[106011]: pgmap v384: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:02.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:44:02 smithi123 ceph-mon[139369]: pgmap v385: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:03.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:44:02 smithi135 ceph-mon[106011]: pgmap v385: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:04.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:44:04 smithi123 ceph-mon[139369]: pgmap v386: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:05.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:44:04 smithi135 ceph-mon[106011]: pgmap v386: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:06.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:44:06 smithi123 ceph-mon[139369]: pgmap v387: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:07.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:44:06 smithi135 ceph-mon[106011]: pgmap v387: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:08.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:44:08 smithi123 ceph-mon[139369]: pgmap v388: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:09.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:44:08 smithi135 ceph-mon[106011]: pgmap v388: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:10.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:44:10 smithi135 ceph-mon[106011]: pgmap v389: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:10.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:44:10 smithi123 ceph-mon[139369]: pgmap v389: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:12.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:44:12 smithi123 ceph-mon[139369]: pgmap v390: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:13.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:44:12 smithi135 ceph-mon[106011]: pgmap v390: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:14.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:44:14 smithi123 ceph-mon[139369]: pgmap v391: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:14.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:44:14 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:44:15.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:44:14 smithi135 ceph-mon[106011]: pgmap v391: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:15.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:44:14 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:44:15.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.356+0000 7f4410000700 1 -- 172.21.15.123:0/944722779 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4408102e00 msgr2=0x7f44081031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:15.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.356+0000 7f4410000700 1 --2- 172.21.15.123:0/944722779 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4408102e00 0x7f44081031d0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f43fc004940 tx=0x7f43fc009e30 comp rx=0 tx=0).stop 2024-03-30T04:44:15.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.356+0000 7f4410000700 1 -- 172.21.15.123:0/944722779 shutdown_connections 2024-03-30T04:44:15.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.356+0000 7f4410000700 1 --2- 172.21.15.123:0/944722779 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f44081037a0 0x7f44080fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:15.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.356+0000 7f4410000700 1 --2- 172.21.15.123:0/944722779 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4408102e00 0x7f44081031d0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:15.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.356+0000 7f4410000700 1 -- 172.21.15.123:0/944722779 >> 172.21.15.123:0/944722779 conn(0x7f44080f89b0 msgr2=0x7f44080fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:44:15.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.356+0000 7f4410000700 1 -- 172.21.15.123:0/944722779 shutdown_connections 2024-03-30T04:44:15.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.356+0000 7f4410000700 1 -- 172.21.15.123:0/944722779 wait complete. 2024-03-30T04:44:15.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.357+0000 7f4410000700 1 Processor -- start 2024-03-30T04:44:15.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.357+0000 7f4410000700 1 -- start start 2024-03-30T04:44:15.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.358+0000 7f4410000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4408102e00 0x7f44080702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:44:15.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.358+0000 7f4410000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f44081037a0 0x7f4408070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:44:15.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.358+0000 7f4410000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4408070d90 con 0x7f4408102e00 2024-03-30T04:44:15.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.358+0000 7f4410000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4408070ed0 con 0x7f44081037a0 2024-03-30T04:44:15.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.358+0000 7f440d600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4408102e00 0x7f44080702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:44:15.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.358+0000 7f440d600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4408102e00 0x7f44080702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50674/0 (socket says 172.21.15.123:50674) 2024-03-30T04:44:15.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.358+0000 7f440d600700 1 -- 172.21.15.123:0/1121082928 learned_addr learned my addr 172.21.15.123:0/1121082928 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:44:15.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.358+0000 7f440cc00700 1 --2- 172.21.15.123:0/1121082928 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f44081037a0 0x7f4408070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:44:15.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.359+0000 7f440d600700 1 -- 172.21.15.123:0/1121082928 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f44081037a0 msgr2=0x7f4408070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:15.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.359+0000 7f440d600700 1 --2- 172.21.15.123:0/1121082928 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f44081037a0 0x7f4408070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:15.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.359+0000 7f440d600700 1 -- 172.21.15.123:0/1121082928 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43fc005040 con 0x7f4408102e00 2024-03-30T04:44:15.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.359+0000 7f440d600700 1 --2- 172.21.15.123:0/1121082928 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4408102e00 0x7f44080702c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f43fc009360 tx=0x7f43fc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:44:15.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.359+0000 7f4405600700 1 -- 172.21.15.123:0/1121082928 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f43fc00e040 con 0x7f4408102e00 2024-03-30T04:44:15.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.359+0000 7f4405600700 1 -- 172.21.15.123:0/1121082928 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f43fc013420 con 0x7f4408102e00 2024-03-30T04:44:15.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.359+0000 7f4410000700 1 -- 172.21.15.123:0/1121082928 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4408071120 con 0x7f4408102e00 2024-03-30T04:44:15.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.359+0000 7f4405600700 1 -- 172.21.15.123:0/1121082928 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f43fc01c5d0 con 0x7f4408102e00 2024-03-30T04:44:15.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.359+0000 7f4410000700 1 -- 172.21.15.123:0/1121082928 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f440811a250 con 0x7f4408102e00 2024-03-30T04:44:15.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.360+0000 7f4410000700 1 -- 172.21.15.123:0/1121082928 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44080fe550 con 0x7f4408102e00 2024-03-30T04:44:15.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.362+0000 7f4405600700 1 -- 172.21.15.123:0/1121082928 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f43fc013590 con 0x7f4408102e00 2024-03-30T04:44:15.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.362+0000 7f4405600700 1 --2- 172.21.15.123:0/1121082928 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f43f006cc30 0x7f43f006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:44:15.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.362+0000 7f4405600700 1 -- 172.21.15.123:0/1121082928 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f43fc08d230 con 0x7f4408102e00 2024-03-30T04:44:15.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.363+0000 7f440cc00700 1 --2- 172.21.15.123:0/1121082928 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f43f006cc30 0x7f43f006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:44:15.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.363+0000 7f440cc00700 1 --2- 172.21.15.123:0/1121082928 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f43f006cc30 0x7f43f006f0e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4408071c00 tx=0x7f43f400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:44:15.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.367+0000 7f4405600700 1 -- 172.21.15.123:0/1121082928 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f43fc057170 con 0x7f4408102e00 2024-03-30T04:44:15.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.590+0000 7f4410000700 1 -- 172.21.15.123:0/1121082928 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f44080719b0 con 0x7f43f006cc30 2024-03-30T04:44:15.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.591+0000 7f4405600700 1 -- 172.21.15.123:0/1121082928 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f44080719b0 con 0x7f43f006cc30 2024-03-30T04:44:15.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.593+0000 7f4410000700 1 -- 172.21.15.123:0/1121082928 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f43f006cc30 msgr2=0x7f43f006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:15.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.593+0000 7f4410000700 1 --2- 172.21.15.123:0/1121082928 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f43f006cc30 0x7f43f006f0e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4408071c00 tx=0x7f43f400b040 comp rx=0 tx=0).stop 2024-03-30T04:44:15.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.593+0000 7f4410000700 1 -- 172.21.15.123:0/1121082928 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4408102e00 msgr2=0x7f44080702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:15.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.593+0000 7f4410000700 1 --2- 172.21.15.123:0/1121082928 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4408102e00 0x7f44080702c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f43fc009360 tx=0x7f43fc009e30 comp rx=0 tx=0).stop 2024-03-30T04:44:15.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.594+0000 7f4410000700 1 -- 172.21.15.123:0/1121082928 shutdown_connections 2024-03-30T04:44:15.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.594+0000 7f4410000700 1 --2- 172.21.15.123:0/1121082928 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f43f006cc30 0x7f43f006f0e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:15.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.594+0000 7f4410000700 1 --2- 172.21.15.123:0/1121082928 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4408102e00 0x7f44080702c0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:15.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.594+0000 7f4410000700 1 --2- 172.21.15.123:0/1121082928 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f44081037a0 0x7f4408070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:15.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.594+0000 7f4410000700 1 -- 172.21.15.123:0/1121082928 >> 172.21.15.123:0/1121082928 conn(0x7f44080f89b0 msgr2=0x7f44080f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:44:15.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.594+0000 7f4410000700 1 -- 172.21.15.123:0/1121082928 shutdown_connections 2024-03-30T04:44:15.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.594+0000 7f4410000700 1 -- 172.21.15.123:0/1121082928 wait complete. 2024-03-30T04:44:15.607 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:44:15.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.727+0000 7f2d2aa00700 1 -- 172.21.15.123:0/3487805424 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d24071f90 msgr2=0x7f2d24070360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:15.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.727+0000 7f2d2aa00700 1 --2- 172.21.15.123:0/3487805424 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d24071f90 0x7f2d24070360 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f2d14004970 tx=0x7f2d14009e30 comp rx=0 tx=0).stop 2024-03-30T04:44:15.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.727+0000 7f2d2aa00700 1 -- 172.21.15.123:0/3487805424 shutdown_connections 2024-03-30T04:44:15.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.727+0000 7f2d2aa00700 1 --2- 172.21.15.123:0/3487805424 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2d240708a0 0x7f2d24070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:15.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.727+0000 7f2d2aa00700 1 --2- 172.21.15.123:0/3487805424 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d24071f90 0x7f2d24070360 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:15.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.727+0000 7f2d2aa00700 1 -- 172.21.15.123:0/3487805424 >> 172.21.15.123:0/3487805424 conn(0x7f2d240f89b0 msgr2=0x7f2d240fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:44:15.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.727+0000 7f2d2aa00700 1 -- 172.21.15.123:0/3487805424 shutdown_connections 2024-03-30T04:44:15.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.727+0000 7f2d2aa00700 1 -- 172.21.15.123:0/3487805424 wait complete. 2024-03-30T04:44:15.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.728+0000 7f2d2aa00700 1 Processor -- start 2024-03-30T04:44:15.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.728+0000 7f2d2aa00700 1 -- start start 2024-03-30T04:44:15.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.729+0000 7f2d2aa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d240708a0 0x7f2d2411a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:44:15.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.729+0000 7f2d2aa00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2d24071f90 0x7f2d2411a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:44:15.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.729+0000 7f2d2aa00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2411aba0 con 0x7f2d240708a0 2024-03-30T04:44:15.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.729+0000 7f2d2aa00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d24114020 con 0x7f2d24071f90 2024-03-30T04:44:15.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.729+0000 7f2d23e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d240708a0 0x7f2d2411a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:44:15.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.729+0000 7f2d23e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d240708a0 0x7f2d2411a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50694/0 (socket says 172.21.15.123:50694) 2024-03-30T04:44:15.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.729+0000 7f2d23e00700 1 -- 172.21.15.123:0/3910620649 learned_addr learned my addr 172.21.15.123:0/3910620649 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:44:15.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.729+0000 7f2d23400700 1 --2- 172.21.15.123:0/3910620649 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2d24071f90 0x7f2d2411a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:44:15.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.729+0000 7f2d23e00700 1 -- 172.21.15.123:0/3910620649 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2d24071f90 msgr2=0x7f2d2411a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:15.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.729+0000 7f2d23e00700 1 --2- 172.21.15.123:0/3910620649 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2d24071f90 0x7f2d2411a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:15.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.729+0000 7f2d23e00700 1 -- 172.21.15.123:0/3910620649 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d14005040 con 0x7f2d240708a0 2024-03-30T04:44:15.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.730+0000 7f2d23e00700 1 --2- 172.21.15.123:0/3910620649 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d240708a0 0x7f2d2411a010 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f2d14004f50 tx=0x7f2d14009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:44:15.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.730+0000 7f2d1be00700 1 -- 172.21.15.123:0/3910620649 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d1400e040 con 0x7f2d240708a0 2024-03-30T04:44:15.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.730+0000 7f2d1be00700 1 -- 172.21.15.123:0/3910620649 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2d14013420 con 0x7f2d240708a0 2024-03-30T04:44:15.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.730+0000 7f2d2aa00700 1 -- 172.21.15.123:0/3910620649 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d241142a0 con 0x7f2d240708a0 2024-03-30T04:44:15.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.730+0000 7f2d1be00700 1 -- 172.21.15.123:0/3910620649 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d1401c630 con 0x7f2d240708a0 2024-03-30T04:44:15.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.730+0000 7f2d2aa00700 1 -- 172.21.15.123:0/3910620649 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d241147e0 con 0x7f2d240708a0 2024-03-30T04:44:15.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.731+0000 7f2d2aa00700 1 -- 172.21.15.123:0/3910620649 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d24071aa0 con 0x7f2d240708a0 2024-03-30T04:44:15.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.732+0000 7f2d1be00700 1 -- 172.21.15.123:0/3910620649 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f2d14013590 con 0x7f2d240708a0 2024-03-30T04:44:15.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.733+0000 7f2d1be00700 1 --2- 172.21.15.123:0/3910620649 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2d0806cc30 0x7f2d0806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:44:15.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.733+0000 7f2d1be00700 1 -- 172.21.15.123:0/3910620649 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f2d1408d340 con 0x7f2d240708a0 2024-03-30T04:44:15.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.733+0000 7f2d23400700 1 --2- 172.21.15.123:0/3910620649 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2d0806cc30 0x7f2d0806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:44:15.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.734+0000 7f2d23400700 1 --2- 172.21.15.123:0/3910620649 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2d0806cc30 0x7f2d0806f0e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2d2410cfc0 tx=0x7f2d10009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:44:15.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.738+0000 7f2d1be00700 1 -- 172.21.15.123:0/3910620649 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2d14057300 con 0x7f2d240708a0 2024-03-30T04:44:15.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.963+0000 7f2d2aa00700 1 -- 172.21.15.123:0/3910620649 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2d241103e0 con 0x7f2d0806cc30 2024-03-30T04:44:15.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.964+0000 7f2d1be00700 1 -- 172.21.15.123:0/3910620649 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f2d241103e0 con 0x7f2d0806cc30 2024-03-30T04:44:15.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.966+0000 7f2d2aa00700 1 -- 172.21.15.123:0/3910620649 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2d0806cc30 msgr2=0x7f2d0806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:15.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.966+0000 7f2d2aa00700 1 --2- 172.21.15.123:0/3910620649 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2d0806cc30 0x7f2d0806f0e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2d2410cfc0 tx=0x7f2d10009040 comp rx=0 tx=0).stop 2024-03-30T04:44:15.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.966+0000 7f2d2aa00700 1 -- 172.21.15.123:0/3910620649 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d240708a0 msgr2=0x7f2d2411a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:15.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.966+0000 7f2d2aa00700 1 --2- 172.21.15.123:0/3910620649 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d240708a0 0x7f2d2411a010 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f2d14004f50 tx=0x7f2d14009e30 comp rx=0 tx=0).stop 2024-03-30T04:44:15.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.966+0000 7f2d2aa00700 1 -- 172.21.15.123:0/3910620649 shutdown_connections 2024-03-30T04:44:15.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.966+0000 7f2d2aa00700 1 --2- 172.21.15.123:0/3910620649 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2d0806cc30 0x7f2d0806f0e0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:15.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.966+0000 7f2d2aa00700 1 --2- 172.21.15.123:0/3910620649 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d240708a0 0x7f2d2411a010 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:15.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.966+0000 7f2d2aa00700 1 --2- 172.21.15.123:0/3910620649 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2d24071f90 0x7f2d2411a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:15.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.966+0000 7f2d2aa00700 1 -- 172.21.15.123:0/3910620649 >> 172.21.15.123:0/3910620649 conn(0x7f2d240f89b0 msgr2=0x7f2d240f9b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:44:15.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.966+0000 7f2d2aa00700 1 -- 172.21.15.123:0/3910620649 shutdown_connections 2024-03-30T04:44:15.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:15.967+0000 7f2d2aa00700 1 -- 172.21.15.123:0/3910620649 wait complete. 2024-03-30T04:44:16.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.099+0000 7ff30ce00700 1 -- 172.21.15.123:0/259396337 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3081037a0 msgr2=0x7ff3080fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:16.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.099+0000 7ff30ce00700 1 --2- 172.21.15.123:0/259396337 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3081037a0 0x7ff3080fce10 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7ff2f4004940 tx=0x7ff2f4009e30 comp rx=0 tx=0).stop 2024-03-30T04:44:16.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.099+0000 7ff30ce00700 1 -- 172.21.15.123:0/259396337 shutdown_connections 2024-03-30T04:44:16.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.099+0000 7ff30ce00700 1 --2- 172.21.15.123:0/259396337 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3081037a0 0x7ff3080fce10 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:16.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.099+0000 7ff30ce00700 1 --2- 172.21.15.123:0/259396337 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff308102e00 0x7ff3081031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:16.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.099+0000 7ff30ce00700 1 -- 172.21.15.123:0/259396337 >> 172.21.15.123:0/259396337 conn(0x7ff3080f89b0 msgr2=0x7ff3080fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:44:16.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.099+0000 7ff30ce00700 1 -- 172.21.15.123:0/259396337 shutdown_connections 2024-03-30T04:44:16.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.099+0000 7ff30ce00700 1 -- 172.21.15.123:0/259396337 wait complete. 2024-03-30T04:44:16.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.100+0000 7ff30ce00700 1 Processor -- start 2024-03-30T04:44:16.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.100+0000 7ff30ce00700 1 -- start start 2024-03-30T04:44:16.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.101+0000 7ff30ce00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff308102e00 0x7ff308115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:44:16.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.101+0000 7ff30ce00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff3081037a0 0x7ff308116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:44:16.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.101+0000 7ff30ce00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3081168f0 con 0x7ff308102e00 2024-03-30T04:44:16.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.101+0000 7ff30ce00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff30810fc30 con 0x7ff3081037a0 2024-03-30T04:44:16.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.101+0000 7ff305e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff308102e00 0x7ff308115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:44:16.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.101+0000 7ff305e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff308102e00 0x7ff308115cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50708/0 (socket says 172.21.15.123:50708) 2024-03-30T04:44:16.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.101+0000 7ff305e00700 1 -- 172.21.15.123:0/4283475079 learned_addr learned my addr 172.21.15.123:0/4283475079 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:44:16.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.101+0000 7ff305400700 1 --2- 172.21.15.123:0/4283475079 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff3081037a0 0x7ff308116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:44:16.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.101+0000 7ff305e00700 1 -- 172.21.15.123:0/4283475079 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff3081037a0 msgr2=0x7ff308116210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:16.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.101+0000 7ff305e00700 1 --2- 172.21.15.123:0/4283475079 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff3081037a0 0x7ff308116210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:16.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.101+0000 7ff305e00700 1 -- 172.21.15.123:0/4283475079 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f4005040 con 0x7ff308102e00 2024-03-30T04:44:16.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.102+0000 7ff305e00700 1 --2- 172.21.15.123:0/4283475079 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff308102e00 0x7ff308115cd0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ff2f800b790 tx=0x7ff2f8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:44:16.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.102+0000 7ff2fe000700 1 -- 172.21.15.123:0/4283475079 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff2f800f6f0 con 0x7ff308102e00 2024-03-30T04:44:16.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.102+0000 7ff2fe000700 1 -- 172.21.15.123:0/4283475079 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff2f800fd00 con 0x7ff308102e00 2024-03-30T04:44:16.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.102+0000 7ff2fe000700 1 -- 172.21.15.123:0/4283475079 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff2f80103a0 con 0x7ff308102e00 2024-03-30T04:44:16.102 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.102+0000 7ff30ce00700 1 -- 172.21.15.123:0/4283475079 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff30810feb0 con 0x7ff308102e00 2024-03-30T04:44:16.102 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.102+0000 7ff30ce00700 1 -- 172.21.15.123:0/4283475079 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff308110470 con 0x7ff308102e00 2024-03-30T04:44:16.102 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.103+0000 7ff30ce00700 1 -- 172.21.15.123:0/4283475079 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3080fe550 con 0x7ff308102e00 2024-03-30T04:44:16.102 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.104+0000 7ff2fe000700 1 -- 172.21.15.123:0/4283475079 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7ff2f800f850 con 0x7ff308102e00 2024-03-30T04:44:16.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.105+0000 7ff2fe000700 1 --2- 172.21.15.123:0/4283475079 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ff2ec06cc30 0x7ff2ec06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:44:16.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.105+0000 7ff2fe000700 1 -- 172.21.15.123:0/4283475079 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff2f808be20 con 0x7ff308102e00 2024-03-30T04:44:16.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.105+0000 7ff305400700 1 --2- 172.21.15.123:0/4283475079 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ff2ec06cc30 0x7ff2ec06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:44:16.104 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.106+0000 7ff305400700 1 --2- 172.21.15.123:0/4283475079 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ff2ec06cc30 0x7ff2ec06f0e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff308111570 tx=0x7ff2f4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:44:16.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.109+0000 7ff2fe000700 1 -- 172.21.15.123:0/4283475079 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff2f8059930 con 0x7ff308102e00 2024-03-30T04:44:16.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.318+0000 7ff30ce00700 1 -- 172.21.15.123:0/4283475079 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff308111320 con 0x7ff2ec06cc30 2024-03-30T04:44:16.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.327+0000 7ff2fe000700 1 -- 172.21.15.123:0/4283475079 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7ff308111320 con 0x7ff2ec06cc30 2024-03-30T04:44:16.326 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:44:16.326 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (13m) 6m ago 25m 15.0M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:44:16.326 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (10m) 6m ago 26m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T04:44:16.326 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (10m) 6m ago 25m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T04:44:16.326 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (13m) 6m ago 25m 74.0M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T04:44:16.326 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (6m) 6m ago 19m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T04:44:16.326 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (6m) 6m ago 19m 10.9M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T04:44:16.327 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (19m) 6m ago 19m 10.7M - 17.2.7-797-g5c024335 f77085f6aab7 0dc41326818c 2024-03-30T04:44:16.327 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (6m) 6m ago 19m 9747k - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T04:44:16.327 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (15m) 6m ago 27m 506M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:44:16.327 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (14m) 6m ago 24m 450M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:44:16.327 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (11m) 6m ago 27m 53.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T04:44:16.327 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (11m) 6m ago 24m 42.6M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T04:44:16.327 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (14m) 6m ago 26m 8052k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:44:16.327 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (14m) 6m ago 24m 7880k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:44:16.327 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (10m) 6m ago 23m 104M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T04:44:16.327 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (9m) 6m ago 23m 93.4M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T04:44:16.327 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (9m) 6m ago 22m 86.3M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T04:44:16.327 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (8m) 6m ago 22m 97.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T04:44:16.327 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (7m) 6m ago 21m 97.9M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T04:44:16.328 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (7m) 6m ago 21m 85.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T04:44:16.328 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (13m) 6m ago 25m 53.7M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:44:16.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.330+0000 7ff30ce00700 1 -- 172.21.15.123:0/4283475079 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ff2ec06cc30 msgr2=0x7ff2ec06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:16.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.330+0000 7ff30ce00700 1 --2- 172.21.15.123:0/4283475079 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ff2ec06cc30 0x7ff2ec06f0e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff308111570 tx=0x7ff2f4009e30 comp rx=0 tx=0).stop 2024-03-30T04:44:16.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.330+0000 7ff30ce00700 1 -- 172.21.15.123:0/4283475079 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff308102e00 msgr2=0x7ff308115cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:16.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.330+0000 7ff30ce00700 1 --2- 172.21.15.123:0/4283475079 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff308102e00 0x7ff308115cd0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ff2f800b790 tx=0x7ff2f8009250 comp rx=0 tx=0).stop 2024-03-30T04:44:16.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.330+0000 7ff30ce00700 1 -- 172.21.15.123:0/4283475079 shutdown_connections 2024-03-30T04:44:16.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.330+0000 7ff30ce00700 1 --2- 172.21.15.123:0/4283475079 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ff2ec06cc30 0x7ff2ec06f0e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:16.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.330+0000 7ff30ce00700 1 --2- 172.21.15.123:0/4283475079 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff308102e00 0x7ff308115cd0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:16.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.330+0000 7ff30ce00700 1 --2- 172.21.15.123:0/4283475079 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff3081037a0 0x7ff308116210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:16.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.330+0000 7ff30ce00700 1 -- 172.21.15.123:0/4283475079 >> 172.21.15.123:0/4283475079 conn(0x7ff3080f89b0 msgr2=0x7ff3080f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:44:16.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.330+0000 7ff30ce00700 1 -- 172.21.15.123:0/4283475079 shutdown_connections 2024-03-30T04:44:16.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.330+0000 7ff30ce00700 1 -- 172.21.15.123:0/4283475079 wait complete. 2024-03-30T04:44:16.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.462+0000 7fb720800700 1 -- 172.21.15.123:0/3810926926 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7181037a0 msgr2=0x7fb7180fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:16.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.462+0000 7fb720800700 1 --2- 172.21.15.123:0/3810926926 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7181037a0 0x7fb7180fce10 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb708004940 tx=0x7fb708009e30 comp rx=0 tx=0).stop 2024-03-30T04:44:16.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.462+0000 7fb720800700 1 -- 172.21.15.123:0/3810926926 shutdown_connections 2024-03-30T04:44:16.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.462+0000 7fb720800700 1 --2- 172.21.15.123:0/3810926926 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7181037a0 0x7fb7180fce10 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:16.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.462+0000 7fb720800700 1 --2- 172.21.15.123:0/3810926926 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb718102e00 0x7fb7181031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:16.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.462+0000 7fb720800700 1 -- 172.21.15.123:0/3810926926 >> 172.21.15.123:0/3810926926 conn(0x7fb7180f89b0 msgr2=0x7fb7180fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:44:16.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.462+0000 7fb720800700 1 -- 172.21.15.123:0/3810926926 shutdown_connections 2024-03-30T04:44:16.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.462+0000 7fb720800700 1 -- 172.21.15.123:0/3810926926 wait complete. 2024-03-30T04:44:16.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.463+0000 7fb720800700 1 Processor -- start 2024-03-30T04:44:16.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.463+0000 7fb720800700 1 -- start start 2024-03-30T04:44:16.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.463+0000 7fb720800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb718102e00 0x7fb718115a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:44:16.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.463+0000 7fb720800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7181037a0 0x7fb718115f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:44:16.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.463+0000 7fb720800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7181165a0 con 0x7fb7181037a0 2024-03-30T04:44:16.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.463+0000 7fb720800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb71810fba0 con 0x7fb718102e00 2024-03-30T04:44:16.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.463+0000 7fb71d400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7181037a0 0x7fb718115f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:44:16.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.463+0000 7fb71d400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7181037a0 0x7fb718115f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50728/0 (socket says 172.21.15.123:50728) 2024-03-30T04:44:16.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.463+0000 7fb71d400700 1 -- 172.21.15.123:0/2905892594 learned_addr learned my addr 172.21.15.123:0/2905892594 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:44:16.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.464+0000 7fb71de00700 1 --2- 172.21.15.123:0/2905892594 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb718102e00 0x7fb718115a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:44:16.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.464+0000 7fb71d400700 1 -- 172.21.15.123:0/2905892594 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb718102e00 msgr2=0x7fb718115a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:16.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.464+0000 7fb71d400700 1 --2- 172.21.15.123:0/2905892594 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb718102e00 0x7fb718115a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:16.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.464+0000 7fb71d400700 1 -- 172.21.15.123:0/2905892594 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb708005040 con 0x7fb7181037a0 2024-03-30T04:44:16.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.464+0000 7fb71d400700 1 --2- 172.21.15.123:0/2905892594 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7181037a0 0x7fb718115f50 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fb708008d50 tx=0x7fb708009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:44:16.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.464+0000 7fb712000700 1 -- 172.21.15.123:0/2905892594 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb70800e030 con 0x7fb7181037a0 2024-03-30T04:44:16.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.464+0000 7fb712000700 1 -- 172.21.15.123:0/2905892594 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb7080093b0 con 0x7fb7181037a0 2024-03-30T04:44:16.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.464+0000 7fb720800700 1 -- 172.21.15.123:0/2905892594 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb71810fe20 con 0x7fb7181037a0 2024-03-30T04:44:16.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.464+0000 7fb712000700 1 -- 172.21.15.123:0/2905892594 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb70801cbb0 con 0x7fb7181037a0 2024-03-30T04:44:16.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.464+0000 7fb720800700 1 -- 172.21.15.123:0/2905892594 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7181103e0 con 0x7fb7181037a0 2024-03-30T04:44:16.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.466+0000 7fb720800700 1 -- 172.21.15.123:0/2905892594 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7180fe550 con 0x7fb7181037a0 2024-03-30T04:44:16.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.467+0000 7fb712000700 1 -- 172.21.15.123:0/2905892594 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fb708013930 con 0x7fb7181037a0 2024-03-30T04:44:16.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.468+0000 7fb712000700 1 --2- 172.21.15.123:0/2905892594 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb70006cc20 0x7fb70006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:44:16.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.468+0000 7fb712000700 1 -- 172.21.15.123:0/2905892594 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb70808d850 con 0x7fb7181037a0 2024-03-30T04:44:16.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.468+0000 7fb71de00700 1 --2- 172.21.15.123:0/2905892594 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb70006cc20 0x7fb70006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:44:16.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.469+0000 7fb71de00700 1 --2- 172.21.15.123:0/2905892594 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb70006cc20 0x7fb70006f0d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb70c009b80 tx=0x7fb70c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:44:16.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.473+0000 7fb712000700 1 -- 172.21.15.123:0/2905892594 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb70805b2a0 con 0x7fb7181037a0 2024-03-30T04:44:16.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.756+0000 7fb720800700 1 -- 172.21.15.123:0/2905892594 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fb718109da0 con 0x7fb7181037a0 2024-03-30T04:44:16.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.757+0000 7fb712000700 1 -- 172.21.15.123:0/2905892594 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7fb70805ae30 con 0x7fb7181037a0 2024-03-30T04:44:16.755 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:44:16.755 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:44:16.755 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:44:16.756 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:44:16.756 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:44:16.756 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:44:16.756 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:44:16.756 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:44:16.756 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T04:44:16.756 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:44:16.756 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:44:16.756 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:44:16.756 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 3 2024-03-30T04:44:16.756 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:44:16.756 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:44:16.756 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:44:16.756 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 13 2024-03-30T04:44:16.757 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:44:16.757 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:44:16.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.759+0000 7fb720800700 1 -- 172.21.15.123:0/2905892594 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb70006cc20 msgr2=0x7fb70006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:16.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.759+0000 7fb720800700 1 --2- 172.21.15.123:0/2905892594 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb70006cc20 0x7fb70006f0d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb70c009b80 tx=0x7fb70c00b040 comp rx=0 tx=0).stop 2024-03-30T04:44:16.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.759+0000 7fb720800700 1 -- 172.21.15.123:0/2905892594 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7181037a0 msgr2=0x7fb718115f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:16.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.759+0000 7fb720800700 1 --2- 172.21.15.123:0/2905892594 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7181037a0 0x7fb718115f50 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fb708008d50 tx=0x7fb708009e30 comp rx=0 tx=0).stop 2024-03-30T04:44:16.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.759+0000 7fb720800700 1 -- 172.21.15.123:0/2905892594 shutdown_connections 2024-03-30T04:44:16.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.759+0000 7fb720800700 1 --2- 172.21.15.123:0/2905892594 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb70006cc20 0x7fb70006f0d0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:16.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.759+0000 7fb720800700 1 --2- 172.21.15.123:0/2905892594 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb718102e00 0x7fb718115a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:16.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.759+0000 7fb720800700 1 --2- 172.21.15.123:0/2905892594 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7181037a0 0x7fb718115f50 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:16.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.759+0000 7fb720800700 1 -- 172.21.15.123:0/2905892594 >> 172.21.15.123:0/2905892594 conn(0x7fb7180f89b0 msgr2=0x7fb7180704f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:44:16.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.760+0000 7fb720800700 1 -- 172.21.15.123:0/2905892594 shutdown_connections 2024-03-30T04:44:16.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.760+0000 7fb720800700 1 -- 172.21.15.123:0/2905892594 wait complete. 2024-03-30T04:44:16.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.893+0000 7f424e400700 1 -- 172.21.15.123:0/2530909827 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42480fbc80 msgr2=0x7f42480fc050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:16.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.893+0000 7f424e400700 1 --2- 172.21.15.123:0/2530909827 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42480fbc80 0x7f42480fc050 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f4238004940 tx=0x7f4238009e30 comp rx=0 tx=0).stop 2024-03-30T04:44:16.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.893+0000 7f424e400700 1 -- 172.21.15.123:0/2530909827 shutdown_connections 2024-03-30T04:44:16.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.893+0000 7f424e400700 1 --2- 172.21.15.123:0/2530909827 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f42480fc590 0x7f42481092f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:16.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.893+0000 7f424e400700 1 --2- 172.21.15.123:0/2530909827 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42480fbc80 0x7f42480fc050 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:16.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.893+0000 7f424e400700 1 -- 172.21.15.123:0/2530909827 >> 172.21.15.123:0/2530909827 conn(0x7f4248072bc0 msgr2=0x7f4248072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:44:16.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.893+0000 7f424e400700 1 -- 172.21.15.123:0/2530909827 shutdown_connections 2024-03-30T04:44:16.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.893+0000 7f424e400700 1 -- 172.21.15.123:0/2530909827 wait complete. 2024-03-30T04:44:16.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.894+0000 7f424e400700 1 Processor -- start 2024-03-30T04:44:16.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.894+0000 7f424e400700 1 -- start start 2024-03-30T04:44:16.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.894+0000 7f424e400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42480fbc80 0x7f4248115d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:44:16.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.894+0000 7f424e400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f42480fc590 0x7f4248116270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:44:16.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.894+0000 7f424e400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42481168c0 con 0x7f42480fbc80 2024-03-30T04:44:16.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.894+0000 7f424e400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f424810fc90 con 0x7f42480fc590 2024-03-30T04:44:16.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.894+0000 7f4247400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42480fbc80 0x7f4248115d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:44:16.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.895+0000 7f4247400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42480fbc80 0x7f4248115d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50746/0 (socket says 172.21.15.123:50746) 2024-03-30T04:44:16.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.895+0000 7f4247400700 1 -- 172.21.15.123:0/1644328584 learned_addr learned my addr 172.21.15.123:0/1644328584 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:44:16.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.895+0000 7f4246a00700 1 --2- 172.21.15.123:0/1644328584 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f42480fc590 0x7f4248116270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:44:16.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.895+0000 7f4247400700 1 -- 172.21.15.123:0/1644328584 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f42480fc590 msgr2=0x7f4248116270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:16.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.895+0000 7f4247400700 1 --2- 172.21.15.123:0/1644328584 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f42480fc590 0x7f4248116270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:16.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.895+0000 7f4247400700 1 -- 172.21.15.123:0/1644328584 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4238005040 con 0x7f42480fbc80 2024-03-30T04:44:16.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.895+0000 7f4247400700 1 --2- 172.21.15.123:0/1644328584 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42480fbc80 0x7f4248115d30 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f4238009390 tx=0x7f4238009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:44:16.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.895+0000 7f423f400700 1 -- 172.21.15.123:0/1644328584 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f423800e040 con 0x7f42480fbc80 2024-03-30T04:44:16.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.895+0000 7f423f400700 1 -- 172.21.15.123:0/1644328584 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4238013420 con 0x7f42480fbc80 2024-03-30T04:44:16.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.896+0000 7f423f400700 1 -- 172.21.15.123:0/1644328584 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f423801c630 con 0x7f42480fbc80 2024-03-30T04:44:16.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.896+0000 7f424e400700 1 -- 172.21.15.123:0/1644328584 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f424810ff10 con 0x7f42480fbc80 2024-03-30T04:44:16.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.896+0000 7f424e400700 1 -- 172.21.15.123:0/1644328584 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4248110400 con 0x7f42480fbc80 2024-03-30T04:44:16.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.897+0000 7f424e400700 1 -- 172.21.15.123:0/1644328584 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4248065e30 con 0x7f42480fbc80 2024-03-30T04:44:16.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.898+0000 7f423f400700 1 -- 172.21.15.123:0/1644328584 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f42380139e0 con 0x7f42480fbc80 2024-03-30T04:44:16.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.898+0000 7f423f400700 1 --2- 172.21.15.123:0/1644328584 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f422c06cc30 0x7f422c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:44:16.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.899+0000 7f423f400700 1 -- 172.21.15.123:0/1644328584 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f423808d3d0 con 0x7f42480fbc80 2024-03-30T04:44:16.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.899+0000 7f4246a00700 1 --2- 172.21.15.123:0/1644328584 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f422c06cc30 0x7f422c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:44:16.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.899+0000 7f4246a00700 1 --2- 172.21.15.123:0/1644328584 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f422c06cc30 0x7f422c06f0e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f42481115d0 tx=0x7f4234009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:44:16.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:16.903+0000 7f423f400700 1 -- 172.21.15.123:0/1644328584 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f423805ae20 con 0x7f42480fbc80 2024-03-30T04:44:16.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:44:16 smithi123 ceph-mon[139369]: pgmap v392: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:16.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:44:16 smithi123 ceph-mon[139369]: from='client.34766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:44:16.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:44:16 smithi123 ceph-mon[139369]: from='client.34770 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:44:17.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:44:16 smithi135 ceph-mon[106011]: pgmap v392: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:17.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:44:16 smithi135 ceph-mon[106011]: from='client.34766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:44:17.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:44:16 smithi135 ceph-mon[106011]: from='client.34770 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:44:17.146 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.147+0000 7f424e400700 1 -- 172.21.15.123:0/1644328584 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f424804bed0 con 0x7f42480fbc80 2024-03-30T04:44:17.146 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.148+0000 7f423f400700 1 -- 172.21.15.123:0/1644328584 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f423805a9b0 con 0x7f42480fbc80 2024-03-30T04:44:17.146 INFO:teuthology.orchestra.run.smithi123.stdout:e32 2024-03-30T04:44:17.146 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:44:17.146 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:44:17.147 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:44:17.147 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:44:17.147 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:44:17.147 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:44:17.147 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T04:44:17.147 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T04:44:17.147 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:44:17.147 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T04:44:17.147 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:44:17.147 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:44:17.147 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:44:17.147 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:44:17.147 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:44:17.147 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T04:44:17.148 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:44:17.148 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:44:17.148 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T04:44:17.148 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:44:17.148 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T04:44:17.148 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T04:44:17.148 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T04:44:17.148 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T04:44:17.148 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:44:17.148 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:44:17.148 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:44:17.148 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:44:17.148 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:44:17.148 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:44:17.148 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T04:44:17.149 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T04:44:17.149 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:44:17.149 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:44:17.149 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T04:44:17.149 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:44:17.149 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:44:17.149 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:44:17.149 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44315} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/1741352336,v1:172.21.15.135:6827/1741352336] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:44:17.149 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:44:17.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.150+0000 7f424e400700 1 -- 172.21.15.123:0/1644328584 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f422c06cc30 msgr2=0x7f422c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:17.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.150+0000 7f424e400700 1 --2- 172.21.15.123:0/1644328584 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f422c06cc30 0x7f422c06f0e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f42481115d0 tx=0x7f4234009040 comp rx=0 tx=0).stop 2024-03-30T04:44:17.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.150+0000 7f424e400700 1 -- 172.21.15.123:0/1644328584 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42480fbc80 msgr2=0x7f4248115d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:17.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.150+0000 7f424e400700 1 --2- 172.21.15.123:0/1644328584 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42480fbc80 0x7f4248115d30 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f4238009390 tx=0x7f4238009e30 comp rx=0 tx=0).stop 2024-03-30T04:44:17.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.150+0000 7f424e400700 1 -- 172.21.15.123:0/1644328584 shutdown_connections 2024-03-30T04:44:17.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.150+0000 7f424e400700 1 --2- 172.21.15.123:0/1644328584 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f422c06cc30 0x7f422c06f0e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:17.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.150+0000 7f424e400700 1 --2- 172.21.15.123:0/1644328584 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42480fbc80 0x7f4248115d30 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:17.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.150+0000 7f424e400700 1 --2- 172.21.15.123:0/1644328584 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f42480fc590 0x7f4248116270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:17.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.150+0000 7f424e400700 1 -- 172.21.15.123:0/1644328584 >> 172.21.15.123:0/1644328584 conn(0x7f4248072bc0 msgr2=0x7f4248105c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:44:17.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.150+0000 7f424e400700 1 -- 172.21.15.123:0/1644328584 shutdown_connections 2024-03-30T04:44:17.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.150+0000 7f424e400700 1 -- 172.21.15.123:0/1644328584 wait complete. 2024-03-30T04:44:17.151 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 32 2024-03-30T04:44:17.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.286+0000 7f567e400700 1 -- 172.21.15.123:0/1154404827 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5678070300 msgr2=0x7f56780706d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:17.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.286+0000 7f567e400700 1 --2- 172.21.15.123:0/1154404827 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5678070300 0x7f56780706d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f5668004970 tx=0x7f5668009e30 comp rx=0 tx=0).stop 2024-03-30T04:44:17.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.286+0000 7f567e400700 1 -- 172.21.15.123:0/1154404827 shutdown_connections 2024-03-30T04:44:17.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.286+0000 7f567e400700 1 --2- 172.21.15.123:0/1154404827 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5678070c10 0x7f5678109170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:17.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.286+0000 7f567e400700 1 --2- 172.21.15.123:0/1154404827 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5678070300 0x7f56780706d0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:17.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.286+0000 7f567e400700 1 -- 172.21.15.123:0/1154404827 >> 172.21.15.123:0/1154404827 conn(0x7f56780f89b0 msgr2=0x7f56780fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:44:17.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.286+0000 7f567e400700 1 -- 172.21.15.123:0/1154404827 shutdown_connections 2024-03-30T04:44:17.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.286+0000 7f567e400700 1 -- 172.21.15.123:0/1154404827 wait complete. 2024-03-30T04:44:17.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.287+0000 7f567e400700 1 Processor -- start 2024-03-30T04:44:17.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.287+0000 7f567e400700 1 -- start start 2024-03-30T04:44:17.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.287+0000 7f567e400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5678070300 0x7f5678115a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:44:17.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.287+0000 7f567e400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5678070c10 0x7f5678115fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:44:17.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.288+0000 7f567e400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56781166a0 con 0x7f5678070c10 2024-03-30T04:44:17.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.288+0000 7f567e400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f567810fc30 con 0x7f5678070300 2024-03-30T04:44:17.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.288+0000 7f5676a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5678070c10 0x7f5678115fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:44:17.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.288+0000 7f5676a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5678070c10 0x7f5678115fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50766/0 (socket says 172.21.15.123:50766) 2024-03-30T04:44:17.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.288+0000 7f5676a00700 1 -- 172.21.15.123:0/3158540398 learned_addr learned my addr 172.21.15.123:0/3158540398 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:44:17.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.288+0000 7f5677400700 1 --2- 172.21.15.123:0/3158540398 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5678070300 0x7f5678115a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:44:17.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.288+0000 7f5676a00700 1 -- 172.21.15.123:0/3158540398 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5678070300 msgr2=0x7f5678115a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:17.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.288+0000 7f5676a00700 1 --2- 172.21.15.123:0/3158540398 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5678070300 0x7f5678115a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:17.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.288+0000 7f5676a00700 1 -- 172.21.15.123:0/3158540398 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5668005040 con 0x7f5678070c10 2024-03-30T04:44:17.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.288+0000 7f5676a00700 1 --2- 172.21.15.123:0/3158540398 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5678070c10 0x7f5678115fc0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f566400b790 tx=0x7f5664009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:44:17.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.289+0000 7f566f400700 1 -- 172.21.15.123:0/3158540398 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f566400f770 con 0x7f5678070c10 2024-03-30T04:44:17.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.289+0000 7f566f400700 1 -- 172.21.15.123:0/3158540398 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f566400fd80 con 0x7f5678070c10 2024-03-30T04:44:17.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.289+0000 7f567e400700 1 -- 172.21.15.123:0/3158540398 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f567810ff10 con 0x7f5678070c10 2024-03-30T04:44:17.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.289+0000 7f566f400700 1 -- 172.21.15.123:0/3158540398 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f566400d310 con 0x7f5678070c10 2024-03-30T04:44:17.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.289+0000 7f567e400700 1 -- 172.21.15.123:0/3158540398 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5678110450 con 0x7f5678070c10 2024-03-30T04:44:17.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.290+0000 7f567e400700 1 -- 172.21.15.123:0/3158540398 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5678071a40 con 0x7f5678070c10 2024-03-30T04:44:17.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.291+0000 7f566f400700 1 -- 172.21.15.123:0/3158540398 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f5664010420 con 0x7f5678070c10 2024-03-30T04:44:17.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.292+0000 7f566f400700 1 --2- 172.21.15.123:0/3158540398 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f565c06cc20 0x7f565c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:44:17.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.292+0000 7f566f400700 1 -- 172.21.15.123:0/3158540398 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f566408bfa0 con 0x7f5678070c10 2024-03-30T04:44:17.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.292+0000 7f5677400700 1 --2- 172.21.15.123:0/3158540398 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f565c06cc20 0x7f565c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:44:17.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.293+0000 7f5677400700 1 --2- 172.21.15.123:0/3158540398 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f565c06cc20 0x7f565c06f0d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f5668004ee0 tx=0x7f5668009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:44:17.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.297+0000 7f566f400700 1 -- 172.21.15.123:0/3158540398 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f56640bf9f0 con 0x7f5678070c10 2024-03-30T04:44:17.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.516+0000 7f567e400700 1 -- 172.21.15.123:0/3158540398 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f567810c0a0 con 0x7f565c06cc20 2024-03-30T04:44:17.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.518+0000 7f566f400700 1 -- 172.21.15.123:0/3158540398 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f567810c0a0 con 0x7f565c06cc20 2024-03-30T04:44:17.516 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:44:17.516 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:44:17.516 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:44:17.516 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T04:44:17.516 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:44:17.516 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T04:44:17.516 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T04:44:17.517 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T04:44:17.517 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T04:44:17.517 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:44:17.517 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "15/21 daemons upgraded", 2024-03-30T04:44:17.517 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T04:44:17.517 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:44:17.517 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:44:17.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.520+0000 7f567e400700 1 -- 172.21.15.123:0/3158540398 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f565c06cc20 msgr2=0x7f565c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:17.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.520+0000 7f567e400700 1 --2- 172.21.15.123:0/3158540398 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f565c06cc20 0x7f565c06f0d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f5668004ee0 tx=0x7f5668009e30 comp rx=0 tx=0).stop 2024-03-30T04:44:17.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.520+0000 7f567e400700 1 -- 172.21.15.123:0/3158540398 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5678070c10 msgr2=0x7f5678115fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:17.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.520+0000 7f567e400700 1 --2- 172.21.15.123:0/3158540398 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5678070c10 0x7f5678115fc0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f566400b790 tx=0x7f5664009250 comp rx=0 tx=0).stop 2024-03-30T04:44:17.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.520+0000 7f567e400700 1 -- 172.21.15.123:0/3158540398 shutdown_connections 2024-03-30T04:44:17.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.520+0000 7f567e400700 1 --2- 172.21.15.123:0/3158540398 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f565c06cc20 0x7f565c06f0d0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:17.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.520+0000 7f567e400700 1 --2- 172.21.15.123:0/3158540398 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5678070300 0x7f5678115a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:17.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.520+0000 7f567e400700 1 --2- 172.21.15.123:0/3158540398 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5678070c10 0x7f5678115fc0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:17.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.520+0000 7f567e400700 1 -- 172.21.15.123:0/3158540398 >> 172.21.15.123:0/3158540398 conn(0x7f56780f89b0 msgr2=0x7f56780f9ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:44:17.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.520+0000 7f567e400700 1 -- 172.21.15.123:0/3158540398 shutdown_connections 2024-03-30T04:44:17.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.520+0000 7f567e400700 1 -- 172.21.15.123:0/3158540398 wait complete. 2024-03-30T04:44:17.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.654+0000 7fd665600700 1 -- 172.21.15.123:0/2959261884 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd660102d80 msgr2=0x7fd660103150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:17.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.654+0000 7fd665600700 1 --2- 172.21.15.123:0/2959261884 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd660102d80 0x7fd660103150 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd650004940 tx=0x7fd650009e30 comp rx=0 tx=0).stop 2024-03-30T04:44:17.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.655+0000 7fd665600700 1 -- 172.21.15.123:0/2959261884 shutdown_connections 2024-03-30T04:44:17.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.655+0000 7fd665600700 1 --2- 172.21.15.123:0/2959261884 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd660103720 0x7fd6600fcd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:17.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.655+0000 7fd665600700 1 --2- 172.21.15.123:0/2959261884 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd660102d80 0x7fd660103150 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:17.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.655+0000 7fd665600700 1 -- 172.21.15.123:0/2959261884 >> 172.21.15.123:0/2959261884 conn(0x7fd6600f8930 msgr2=0x7fd6600fad40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:44:17.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.655+0000 7fd665600700 1 -- 172.21.15.123:0/2959261884 shutdown_connections 2024-03-30T04:44:17.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.655+0000 7fd665600700 1 -- 172.21.15.123:0/2959261884 wait complete. 2024-03-30T04:44:17.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.655+0000 7fd665600700 1 Processor -- start 2024-03-30T04:44:17.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.656+0000 7fd665600700 1 -- start start 2024-03-30T04:44:17.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.656+0000 7fd665600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd660102d80 0x7fd660115c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:44:17.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.656+0000 7fd665600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd660103720 0x7fd660116190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:44:17.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.656+0000 7fd665600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6601167e0 con 0x7fd660102d80 2024-03-30T04:44:17.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.656+0000 7fd665600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd66010fbb0 con 0x7fd660103720 2024-03-30T04:44:17.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.656+0000 7fd65ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd660102d80 0x7fd660115c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:44:17.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.656+0000 7fd65ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd660102d80 0x7fd660115c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50774/0 (socket says 172.21.15.123:50774) 2024-03-30T04:44:17.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.656+0000 7fd65ea00700 1 -- 172.21.15.123:0/2115746444 learned_addr learned my addr 172.21.15.123:0/2115746444 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:44:17.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.656+0000 7fd65e000700 1 --2- 172.21.15.123:0/2115746444 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd660103720 0x7fd660116190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:44:17.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.657+0000 7fd65ea00700 1 -- 172.21.15.123:0/2115746444 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd660103720 msgr2=0x7fd660116190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:17.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.657+0000 7fd65ea00700 1 --2- 172.21.15.123:0/2115746444 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd660103720 0x7fd660116190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:17.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.657+0000 7fd65ea00700 1 -- 172.21.15.123:0/2115746444 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd650005040 con 0x7fd660102d80 2024-03-30T04:44:17.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.657+0000 7fd65ea00700 1 --2- 172.21.15.123:0/2115746444 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd660102d80 0x7fd660115c50 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd650009390 tx=0x7fd650009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:44:17.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.657+0000 7fd656a00700 1 -- 172.21.15.123:0/2115746444 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd65000e040 con 0x7fd660102d80 2024-03-30T04:44:17.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.657+0000 7fd656a00700 1 -- 172.21.15.123:0/2115746444 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd650013420 con 0x7fd660102d80 2024-03-30T04:44:17.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.657+0000 7fd665600700 1 -- 172.21.15.123:0/2115746444 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd66010fe30 con 0x7fd660102d80 2024-03-30T04:44:17.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.657+0000 7fd656a00700 1 -- 172.21.15.123:0/2115746444 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd65001c630 con 0x7fd660102d80 2024-03-30T04:44:17.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.657+0000 7fd665600700 1 -- 172.21.15.123:0/2115746444 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6601103f0 con 0x7fd660102d80 2024-03-30T04:44:17.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.658+0000 7fd665600700 1 -- 172.21.15.123:0/2115746444 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6600fe4d0 con 0x7fd660102d80 2024-03-30T04:44:17.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.659+0000 7fd656a00700 1 -- 172.21.15.123:0/2115746444 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fd6500139e0 con 0x7fd660102d80 2024-03-30T04:44:17.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.660+0000 7fd656a00700 1 --2- 172.21.15.123:0/2115746444 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd64406cc30 0x7fd64406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:44:17.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.660+0000 7fd656a00700 1 -- 172.21.15.123:0/2115746444 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd65008d250 con 0x7fd660102d80 2024-03-30T04:44:17.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.661+0000 7fd65e000700 1 --2- 172.21.15.123:0/2115746444 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd64406cc30 0x7fd64406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:44:17.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.661+0000 7fd65e000700 1 --2- 172.21.15.123:0/2115746444 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd64406cc30 0x7fd64406f0e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd6601114f0 tx=0x7fd64c009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:44:17.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.665+0000 7fd656a00700 1 -- 172.21.15.123:0/2115746444 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd65005add0 con 0x7fd660102d80 2024-03-30T04:44:17.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.947+0000 7fd665600700 1 -- 172.21.15.123:0/2115746444 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fd66005f5f0 con 0x7fd660102d80 2024-03-30T04:44:17.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.947+0000 7fd656a00700 1 -- 172.21.15.123:0/2115746444 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fd65005a960 con 0x7fd660102d80 2024-03-30T04:44:17.946 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T04:44:17.946 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T04:44:17.946 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T04:44:17.946 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T04:44:17.946 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T04:44:17.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.950+0000 7fd665600700 1 -- 172.21.15.123:0/2115746444 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd64406cc30 msgr2=0x7fd64406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:17.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.950+0000 7fd665600700 1 --2- 172.21.15.123:0/2115746444 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd64406cc30 0x7fd64406f0e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd6601114f0 tx=0x7fd64c009ee0 comp rx=0 tx=0).stop 2024-03-30T04:44:17.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.950+0000 7fd665600700 1 -- 172.21.15.123:0/2115746444 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd660102d80 msgr2=0x7fd660115c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:17.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.950+0000 7fd665600700 1 --2- 172.21.15.123:0/2115746444 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd660102d80 0x7fd660115c50 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd650009390 tx=0x7fd650009e30 comp rx=0 tx=0).stop 2024-03-30T04:44:17.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.950+0000 7fd665600700 1 -- 172.21.15.123:0/2115746444 shutdown_connections 2024-03-30T04:44:17.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.950+0000 7fd665600700 1 --2- 172.21.15.123:0/2115746444 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd64406cc30 0x7fd64406f0e0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:17.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.950+0000 7fd665600700 1 --2- 172.21.15.123:0/2115746444 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd660102d80 0x7fd660115c50 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:17.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.950+0000 7fd665600700 1 --2- 172.21.15.123:0/2115746444 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd660103720 0x7fd660116190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:17.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.950+0000 7fd665600700 1 -- 172.21.15.123:0/2115746444 >> 172.21.15.123:0/2115746444 conn(0x7fd6600f8930 msgr2=0x7fd6600f9910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:44:17.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.950+0000 7fd665600700 1 -- 172.21.15.123:0/2115746444 shutdown_connections 2024-03-30T04:44:17.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:17.950+0000 7fd665600700 1 -- 172.21.15.123:0/2115746444 wait complete. 2024-03-30T04:44:17.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:44:17 smithi123 ceph-mon[139369]: from='client.34774 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:44:17.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:44:17 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2905892594' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:44:17.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:44:17 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/1644328584' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:44:18.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:44:17 smithi135 ceph-mon[106011]: from='client.34774 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:44:18.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:44:17 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2905892594' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:44:18.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:44:17 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/1644328584' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:44:18.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:44:18 smithi123 ceph-mon[139369]: from='client.34786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:44:18.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:44:18 smithi123 ceph-mon[139369]: pgmap v393: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:18.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:44:18 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2115746444' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:44:19.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:44:18 smithi135 ceph-mon[106011]: from='client.34786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:44:19.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:44:18 smithi135 ceph-mon[106011]: pgmap v393: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:19.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:44:18 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2115746444' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:44:20.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:44:20 smithi135 ceph-mon[106011]: pgmap v394: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:20.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:44:20 smithi123 ceph-mon[139369]: pgmap v394: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:22.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:44:22 smithi135 ceph-mon[106011]: pgmap v395: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:22.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:44:22 smithi123 ceph-mon[139369]: pgmap v395: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:24.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:44:24 smithi123 ceph-mon[139369]: pgmap v396: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:25.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:44:24 smithi135 ceph-mon[106011]: pgmap v396: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:26.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:44:26 smithi123 ceph-mon[139369]: pgmap v397: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:27.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:44:26 smithi135 ceph-mon[106011]: pgmap v397: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:28.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:44:28 smithi123 ceph-mon[139369]: pgmap v398: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:29.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:44:28 smithi135 ceph-mon[106011]: pgmap v398: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:29.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:44:29 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:44:30.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:44:29 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:44:30.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:44:30 smithi123 ceph-mon[139369]: pgmap v399: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:31.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:44:30 smithi135 ceph-mon[106011]: pgmap v399: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:32.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:44:32 smithi123 ceph-mon[139369]: pgmap v400: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:33.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:44:32 smithi135 ceph-mon[106011]: pgmap v400: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:34.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:44:34 smithi123 ceph-mon[139369]: pgmap v401: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:35.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:44:34 smithi135 ceph-mon[106011]: pgmap v401: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:36.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:44:36 smithi123 ceph-mon[139369]: pgmap v402: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:37.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:44:36 smithi135 ceph-mon[106011]: pgmap v402: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:38.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:44:38 smithi123 ceph-mon[139369]: pgmap v403: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:39.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:44:38 smithi135 ceph-mon[106011]: pgmap v403: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:40.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:44:40 smithi135 ceph-mon[106011]: pgmap v404: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:40.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:44:40 smithi123 ceph-mon[139369]: pgmap v404: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:42.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:44:42 smithi123 ceph-mon[139369]: pgmap v405: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:43.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:44:42 smithi135 ceph-mon[106011]: pgmap v405: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:44.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:44:44 smithi123 ceph-mon[139369]: pgmap v406: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:44.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:44:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:44:45.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:44:44 smithi135 ceph-mon[106011]: pgmap v406: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:45.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:44:44 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:44:46.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:44:46 smithi123 ceph-mon[139369]: pgmap v407: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:47.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:44:46 smithi135 ceph-mon[106011]: pgmap v407: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:48.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.102+0000 7f1b66e00700 1 -- 172.21.15.123:0/1810004881 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b600708a0 msgr2=0x7f1b60070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:48.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.102+0000 7f1b66e00700 1 --2- 172.21.15.123:0/1810004881 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b600708a0 0x7f1b60070d10 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c004970 tx=0x7f1b4c009e30 comp rx=0 tx=0).stop 2024-03-30T04:44:48.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.102+0000 7f1b66e00700 1 -- 172.21.15.123:0/1810004881 shutdown_connections 2024-03-30T04:44:48.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.102+0000 7f1b66e00700 1 --2- 172.21.15.123:0/1810004881 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b600708a0 0x7f1b60070d10 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:48.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.102+0000 7f1b66e00700 1 --2- 172.21.15.123:0/1810004881 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1b60071f90 0x7f1b60070360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:48.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.102+0000 7f1b66e00700 1 -- 172.21.15.123:0/1810004881 >> 172.21.15.123:0/1810004881 conn(0x7f1b600f89b0 msgr2=0x7f1b600fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:44:48.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.102+0000 7f1b66e00700 1 -- 172.21.15.123:0/1810004881 shutdown_connections 2024-03-30T04:44:48.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.102+0000 7f1b66e00700 1 -- 172.21.15.123:0/1810004881 wait complete. 2024-03-30T04:44:48.102 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.103+0000 7f1b66e00700 1 Processor -- start 2024-03-30T04:44:48.102 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.103+0000 7f1b66e00700 1 -- start start 2024-03-30T04:44:48.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.103+0000 7f1b66e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1b600708a0 0x7f1b6011a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:44:48.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.103+0000 7f1b66e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b60071f90 0x7f1b6011a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:44:48.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.104+0000 7f1b66e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6011ac40 con 0x7f1b60071f90 2024-03-30T04:44:48.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.104+0000 7f1b66e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b60114010 con 0x7f1b600708a0 2024-03-30T04:44:48.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.104+0000 7f1b5f400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b60071f90 0x7f1b6011a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:44:48.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.104+0000 7f1b5f400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b60071f90 0x7f1b6011a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36052/0 (socket says 172.21.15.123:36052) 2024-03-30T04:44:48.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.104+0000 7f1b5f400700 1 -- 172.21.15.123:0/2085054657 learned_addr learned my addr 172.21.15.123:0/2085054657 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:44:48.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.104+0000 7f1b5fe00700 1 --2- 172.21.15.123:0/2085054657 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1b600708a0 0x7f1b6011a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:44:48.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.104+0000 7f1b5f400700 1 -- 172.21.15.123:0/2085054657 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1b600708a0 msgr2=0x7f1b6011a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:48.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.104+0000 7f1b5f400700 1 --2- 172.21.15.123:0/2085054657 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1b600708a0 0x7f1b6011a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:48.104 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.104+0000 7f1b5f400700 1 -- 172.21.15.123:0/2085054657 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b4c005040 con 0x7f1b60071f90 2024-03-30T04:44:48.104 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.104+0000 7f1b5f400700 1 --2- 172.21.15.123:0/2085054657 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b60071f90 0x7f1b6011a5f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c004f50 tx=0x7f1b4c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:44:48.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.105+0000 7f1b57e00700 1 -- 172.21.15.123:0/2085054657 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b4c00e040 con 0x7f1b60071f90 2024-03-30T04:44:48.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.105+0000 7f1b57e00700 1 -- 172.21.15.123:0/2085054657 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1b4c013420 con 0x7f1b60071f90 2024-03-30T04:44:48.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.105+0000 7f1b57e00700 1 -- 172.21.15.123:0/2085054657 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b4c01c630 con 0x7f1b60071f90 2024-03-30T04:44:48.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.105+0000 7f1b66e00700 1 -- 172.21.15.123:0/2085054657 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b60114290 con 0x7f1b60071f90 2024-03-30T04:44:48.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.105+0000 7f1b66e00700 1 -- 172.21.15.123:0/2085054657 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b601147d0 con 0x7f1b60071f90 2024-03-30T04:44:48.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.106+0000 7f1b66e00700 1 -- 172.21.15.123:0/2085054657 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b60071aa0 con 0x7f1b60071f90 2024-03-30T04:44:48.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.107+0000 7f1b57e00700 1 -- 172.21.15.123:0/2085054657 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f1b4c013590 con 0x7f1b60071f90 2024-03-30T04:44:48.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.108+0000 7f1b57e00700 1 --2- 172.21.15.123:0/2085054657 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1b4406cc30 0x7f1b4406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:44:48.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.108+0000 7f1b57e00700 1 -- 172.21.15.123:0/2085054657 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f1b4c08d1a0 con 0x7f1b60071f90 2024-03-30T04:44:48.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.108+0000 7f1b5fe00700 1 --2- 172.21.15.123:0/2085054657 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1b4406cc30 0x7f1b4406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:44:48.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.108+0000 7f1b5fe00700 1 --2- 172.21.15.123:0/2085054657 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1b4406cc30 0x7f1b4406f0e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1b50004f40 tx=0x7f1b5000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:44:48.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.112+0000 7f1b57e00700 1 -- 172.21.15.123:0/2085054657 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f1b4c05ade0 con 0x7f1b60071f90 2024-03-30T04:44:48.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.332+0000 7f1b66e00700 1 -- 172.21.15.123:0/2085054657 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1b601103e0 con 0x7f1b4406cc30 2024-03-30T04:44:48.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.334+0000 7f1b57e00700 1 -- 172.21.15.123:0/2085054657 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f1b601103e0 con 0x7f1b4406cc30 2024-03-30T04:44:48.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.336+0000 7f1b66e00700 1 -- 172.21.15.123:0/2085054657 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1b4406cc30 msgr2=0x7f1b4406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:48.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.336+0000 7f1b66e00700 1 --2- 172.21.15.123:0/2085054657 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1b4406cc30 0x7f1b4406f0e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1b50004f40 tx=0x7f1b5000b040 comp rx=0 tx=0).stop 2024-03-30T04:44:48.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.336+0000 7f1b66e00700 1 -- 172.21.15.123:0/2085054657 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b60071f90 msgr2=0x7f1b6011a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:48.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.336+0000 7f1b66e00700 1 --2- 172.21.15.123:0/2085054657 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b60071f90 0x7f1b6011a5f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c004f50 tx=0x7f1b4c009e30 comp rx=0 tx=0).stop 2024-03-30T04:44:48.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.336+0000 7f1b66e00700 1 -- 172.21.15.123:0/2085054657 shutdown_connections 2024-03-30T04:44:48.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.336+0000 7f1b66e00700 1 --2- 172.21.15.123:0/2085054657 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1b4406cc30 0x7f1b4406f0e0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:48.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.336+0000 7f1b66e00700 1 --2- 172.21.15.123:0/2085054657 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1b600708a0 0x7f1b6011a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:48.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.336+0000 7f1b66e00700 1 --2- 172.21.15.123:0/2085054657 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b60071f90 0x7f1b6011a5f0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:48.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.336+0000 7f1b66e00700 1 -- 172.21.15.123:0/2085054657 >> 172.21.15.123:0/2085054657 conn(0x7f1b600f89b0 msgr2=0x7f1b600f9b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:44:48.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.336+0000 7f1b66e00700 1 -- 172.21.15.123:0/2085054657 shutdown_connections 2024-03-30T04:44:48.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.336+0000 7f1b66e00700 1 -- 172.21.15.123:0/2085054657 wait complete. 2024-03-30T04:44:48.349 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:44:48.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.470+0000 7fac8d000700 1 -- 172.21.15.123:0/3217082964 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac88070c10 msgr2=0x7fac88109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:48.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.470+0000 7fac8d000700 1 --2- 172.21.15.123:0/3217082964 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac88070c10 0x7fac88109170 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fac74004970 tx=0x7fac74009e30 comp rx=0 tx=0).stop 2024-03-30T04:44:48.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.470+0000 7fac8d000700 1 -- 172.21.15.123:0/3217082964 shutdown_connections 2024-03-30T04:44:48.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.470+0000 7fac8d000700 1 --2- 172.21.15.123:0/3217082964 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac88070c10 0x7fac88109170 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:48.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.470+0000 7fac8d000700 1 --2- 172.21.15.123:0/3217082964 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac88070300 0x7fac880706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:48.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.470+0000 7fac8d000700 1 -- 172.21.15.123:0/3217082964 >> 172.21.15.123:0/3217082964 conn(0x7fac880f89b0 msgr2=0x7fac880fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:44:48.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.470+0000 7fac8d000700 1 -- 172.21.15.123:0/3217082964 shutdown_connections 2024-03-30T04:44:48.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.470+0000 7fac8d000700 1 -- 172.21.15.123:0/3217082964 wait complete. 2024-03-30T04:44:48.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.471+0000 7fac8d000700 1 Processor -- start 2024-03-30T04:44:48.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.472+0000 7fac8d000700 1 -- start start 2024-03-30T04:44:48.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.472+0000 7fac8d000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac88070300 0x7fac88115a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:44:48.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.472+0000 7fac8d000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac88070c10 0x7fac88115fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:44:48.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.472+0000 7fac8d000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac88116610 con 0x7fac88070c10 2024-03-30T04:44:48.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.472+0000 7fac8d000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac8810fbf0 con 0x7fac88070300 2024-03-30T04:44:48.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.473+0000 7fac85400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac88070c10 0x7fac88115fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:44:48.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.473+0000 7fac85400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac88070c10 0x7fac88115fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36074/0 (socket says 172.21.15.123:36074) 2024-03-30T04:44:48.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.473+0000 7fac85400700 1 -- 172.21.15.123:0/445157576 learned_addr learned my addr 172.21.15.123:0/445157576 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:44:48.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.473+0000 7fac85e00700 1 --2- 172.21.15.123:0/445157576 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac88070300 0x7fac88115a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:44:48.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.473+0000 7fac85400700 1 -- 172.21.15.123:0/445157576 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac88070300 msgr2=0x7fac88115a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:48.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.473+0000 7fac85400700 1 --2- 172.21.15.123:0/445157576 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac88070300 0x7fac88115a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:48.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.473+0000 7fac85400700 1 -- 172.21.15.123:0/445157576 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac74005040 con 0x7fac88070c10 2024-03-30T04:44:48.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.473+0000 7fac85e00700 1 --2- 172.21.15.123:0/445157576 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac88070300 0x7fac88115a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:44:48.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.473+0000 7fac85400700 1 --2- 172.21.15.123:0/445157576 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac88070c10 0x7fac88115fc0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fac74005010 tx=0x7fac74009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:44:48.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.473+0000 7fac7e000700 1 -- 172.21.15.123:0/445157576 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac7400e040 con 0x7fac88070c10 2024-03-30T04:44:48.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.473+0000 7fac7e000700 1 -- 172.21.15.123:0/445157576 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fac74004290 con 0x7fac88070c10 2024-03-30T04:44:48.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.473+0000 7fac8d000700 1 -- 172.21.15.123:0/445157576 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac8810fe70 con 0x7fac88070c10 2024-03-30T04:44:48.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.474+0000 7fac7e000700 1 -- 172.21.15.123:0/445157576 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac74013e40 con 0x7fac88070c10 2024-03-30T04:44:48.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.474+0000 7fac8d000700 1 -- 172.21.15.123:0/445157576 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac881103b0 con 0x7fac88070c10 2024-03-30T04:44:48.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.475+0000 7fac8d000700 1 -- 172.21.15.123:0/445157576 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac88071a40 con 0x7fac88070c10 2024-03-30T04:44:48.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.476+0000 7fac7e000700 1 -- 172.21.15.123:0/445157576 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fac74013620 con 0x7fac88070c10 2024-03-30T04:44:48.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.476+0000 7fac7e000700 1 --2- 172.21.15.123:0/445157576 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fac6c06cbc0 0x7fac6c06f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:44:48.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.477+0000 7fac7e000700 1 -- 172.21.15.123:0/445157576 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fac7408d360 con 0x7fac88070c10 2024-03-30T04:44:48.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.477+0000 7fac85e00700 1 --2- 172.21.15.123:0/445157576 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fac6c06cbc0 0x7fac6c06f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:44:48.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.477+0000 7fac85e00700 1 --2- 172.21.15.123:0/445157576 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fac6c06cbc0 0x7fac6c06f070 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fac78004600 tx=0x7fac7800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:44:48.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.481+0000 7fac7e000700 1 -- 172.21.15.123:0/445157576 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fac7405adb0 con 0x7fac88070c10 2024-03-30T04:44:48.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.705+0000 7fac8d000700 1 -- 172.21.15.123:0/445157576 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fac8810c030 con 0x7fac6c06cbc0 2024-03-30T04:44:48.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.706+0000 7fac7e000700 1 -- 172.21.15.123:0/445157576 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fac8810c030 con 0x7fac6c06cbc0 2024-03-30T04:44:48.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.709+0000 7fac8d000700 1 -- 172.21.15.123:0/445157576 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fac6c06cbc0 msgr2=0x7fac6c06f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:48.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.709+0000 7fac8d000700 1 --2- 172.21.15.123:0/445157576 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fac6c06cbc0 0x7fac6c06f070 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fac78004600 tx=0x7fac7800b040 comp rx=0 tx=0).stop 2024-03-30T04:44:48.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.709+0000 7fac8d000700 1 -- 172.21.15.123:0/445157576 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac88070c10 msgr2=0x7fac88115fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:48.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.709+0000 7fac8d000700 1 --2- 172.21.15.123:0/445157576 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac88070c10 0x7fac88115fc0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fac74005010 tx=0x7fac74009e30 comp rx=0 tx=0).stop 2024-03-30T04:44:48.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.709+0000 7fac8d000700 1 -- 172.21.15.123:0/445157576 shutdown_connections 2024-03-30T04:44:48.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.709+0000 7fac8d000700 1 --2- 172.21.15.123:0/445157576 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fac6c06cbc0 0x7fac6c06f070 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:48.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.709+0000 7fac8d000700 1 --2- 172.21.15.123:0/445157576 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac88070300 0x7fac88115a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:48.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.709+0000 7fac8d000700 1 --2- 172.21.15.123:0/445157576 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac88070c10 0x7fac88115fc0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:48.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.709+0000 7fac8d000700 1 -- 172.21.15.123:0/445157576 >> 172.21.15.123:0/445157576 conn(0x7fac880f89b0 msgr2=0x7fac88105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:44:48.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.709+0000 7fac8d000700 1 -- 172.21.15.123:0/445157576 shutdown_connections 2024-03-30T04:44:48.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.709+0000 7fac8d000700 1 -- 172.21.15.123:0/445157576 wait complete. 2024-03-30T04:44:48.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.843+0000 7f98d3e00700 1 -- 172.21.15.123:0/3940704167 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98d4070c70 msgr2=0x7f98d4108f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:48.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.843+0000 7f98d3e00700 1 --2- 172.21.15.123:0/3940704167 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98d4070c70 0x7f98d4108f50 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f98c0004970 tx=0x7f98c0009e30 comp rx=0 tx=0).stop 2024-03-30T04:44:48.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.843+0000 7f98d3e00700 1 -- 172.21.15.123:0/3940704167 shutdown_connections 2024-03-30T04:44:48.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.843+0000 7f98d3e00700 1 --2- 172.21.15.123:0/3940704167 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98d4070c70 0x7f98d4108f50 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:48.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.843+0000 7f98d3e00700 1 --2- 172.21.15.123:0/3940704167 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f98d4070360 0x7f98d4070730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:48.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.843+0000 7f98d3e00700 1 -- 172.21.15.123:0/3940704167 >> 172.21.15.123:0/3940704167 conn(0x7f98d40f8830 msgr2=0x7f98d40fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:44:48.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.843+0000 7f98d3e00700 1 -- 172.21.15.123:0/3940704167 shutdown_connections 2024-03-30T04:44:48.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.843+0000 7f98d3e00700 1 -- 172.21.15.123:0/3940704167 wait complete. 2024-03-30T04:44:48.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.844+0000 7f98d3e00700 1 Processor -- start 2024-03-30T04:44:48.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.844+0000 7f98d3e00700 1 -- start start 2024-03-30T04:44:48.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.845+0000 7f98d3e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98d4070360 0x7f98d4115ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:44:48.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.845+0000 7f98d3e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f98d4070c70 0x7f98d4116000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:44:48.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.845+0000 7f98d3e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d4116650 con 0x7f98d4070360 2024-03-30T04:44:48.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.845+0000 7f98d3e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d410fa40 con 0x7f98d4070c70 2024-03-30T04:44:48.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.845+0000 7f98d2a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98d4070360 0x7f98d4115ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:44:48.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.845+0000 7f98d2a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98d4070360 0x7f98d4115ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36090/0 (socket says 172.21.15.123:36090) 2024-03-30T04:44:48.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.845+0000 7f98d2a00700 1 -- 172.21.15.123:0/3545010032 learned_addr learned my addr 172.21.15.123:0/3545010032 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:44:48.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.845+0000 7f98d2000700 1 --2- 172.21.15.123:0/3545010032 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f98d4070c70 0x7f98d4116000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:44:48.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.845+0000 7f98d2a00700 1 -- 172.21.15.123:0/3545010032 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f98d4070c70 msgr2=0x7f98d4116000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:48.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.845+0000 7f98d2a00700 1 --2- 172.21.15.123:0/3545010032 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f98d4070c70 0x7f98d4116000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:48.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.845+0000 7f98d2a00700 1 -- 172.21.15.123:0/3545010032 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c0005040 con 0x7f98d4070360 2024-03-30T04:44:48.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.846+0000 7f98d2a00700 1 --2- 172.21.15.123:0/3545010032 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98d4070360 0x7f98d4115ac0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f98c400eb40 tx=0x7f98c400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:44:48.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.846+0000 7f98caa00700 1 -- 172.21.15.123:0/3545010032 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98c4007cc0 con 0x7f98d4070360 2024-03-30T04:44:48.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.846+0000 7f98caa00700 1 -- 172.21.15.123:0/3545010032 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f98c400f040 con 0x7f98d4070360 2024-03-30T04:44:48.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.846+0000 7f98d3e00700 1 -- 172.21.15.123:0/3545010032 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98d410fcc0 con 0x7f98d4070360 2024-03-30T04:44:48.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.846+0000 7f98caa00700 1 -- 172.21.15.123:0/3545010032 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98c4009570 con 0x7f98d4070360 2024-03-30T04:44:48.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.846+0000 7f98d3e00700 1 -- 172.21.15.123:0/3545010032 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98d4110200 con 0x7f98d4070360 2024-03-30T04:44:48.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.847+0000 7f98d3e00700 1 -- 172.21.15.123:0/3545010032 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98d4071aa0 con 0x7f98d4070360 2024-03-30T04:44:48.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.848+0000 7f98caa00700 1 -- 172.21.15.123:0/3545010032 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f98c40096d0 con 0x7f98d4070360 2024-03-30T04:44:48.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.849+0000 7f98caa00700 1 --2- 172.21.15.123:0/3545010032 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f98b806cc30 0x7f98b806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:44:48.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.849+0000 7f98caa00700 1 -- 172.21.15.123:0/3545010032 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f98c408bcb0 con 0x7f98d4070360 2024-03-30T04:44:48.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.849+0000 7f98d2000700 1 --2- 172.21.15.123:0/3545010032 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f98b806cc30 0x7f98b806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:44:48.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.849+0000 7f98d2000700 1 --2- 172.21.15.123:0/3545010032 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f98b806cc30 0x7f98b806f0e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f98c0009fd0 tx=0x7f98c0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:44:48.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:48.853+0000 7f98caa00700 1 -- 172.21.15.123:0/3545010032 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f98c40597b0 con 0x7f98d4070360 2024-03-30T04:44:48.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:44:48 smithi123 ceph-mon[139369]: pgmap v408: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:49.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.066+0000 7f98d3e00700 1 -- 172.21.15.123:0/3545010032 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f98d410bdf0 con 0x7f98b806cc30 2024-03-30T04:44:49.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.076+0000 7f98caa00700 1 -- 172.21.15.123:0/3545010032 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f98d410bdf0 con 0x7f98b806cc30 2024-03-30T04:44:49.075 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:44:49.075 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (13m) 6m ago 26m 15.0M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:44:49.075 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (11m) 6m ago 26m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T04:44:49.075 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (11m) 6m ago 25m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T04:44:49.075 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (13m) 6m ago 26m 74.0M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T04:44:49.075 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (7m) 6m ago 20m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T04:44:49.075 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (7m) 6m ago 20m 10.9M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T04:44:49.075 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (20m) 6m ago 20m 10.7M - 17.2.7-797-g5c024335 f77085f6aab7 0dc41326818c 2024-03-30T04:44:49.075 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (6m) 6m ago 20m 9747k - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T04:44:49.075 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (16m) 6m ago 28m 506M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:44:49.075 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (15m) 6m ago 25m 450M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:44:49.076 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (12m) 6m ago 28m 53.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T04:44:49.076 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (11m) 6m ago 25m 42.6M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T04:44:49.076 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (14m) 6m ago 26m 8052k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:44:49.076 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (14m) 6m ago 25m 7880k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:44:49.076 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (10m) 6m ago 24m 104M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T04:44:49.076 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (10m) 6m ago 23m 93.4M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T04:44:49.076 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (9m) 6m ago 23m 86.3M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T04:44:49.076 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (8m) 6m ago 22m 97.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T04:44:49.076 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (8m) 6m ago 22m 97.9M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T04:44:49.076 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (7m) 6m ago 21m 85.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T04:44:49.076 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (14m) 6m ago 26m 53.7M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:44:49.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.079+0000 7f98d3e00700 1 -- 172.21.15.123:0/3545010032 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f98b806cc30 msgr2=0x7f98b806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:49.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.079+0000 7f98d3e00700 1 --2- 172.21.15.123:0/3545010032 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f98b806cc30 0x7f98b806f0e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f98c0009fd0 tx=0x7f98c0009e30 comp rx=0 tx=0).stop 2024-03-30T04:44:49.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.079+0000 7f98d3e00700 1 -- 172.21.15.123:0/3545010032 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98d4070360 msgr2=0x7f98d4115ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:49.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.079+0000 7f98d3e00700 1 --2- 172.21.15.123:0/3545010032 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98d4070360 0x7f98d4115ac0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f98c400eb40 tx=0x7f98c400b040 comp rx=0 tx=0).stop 2024-03-30T04:44:49.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.079+0000 7f98d3e00700 1 -- 172.21.15.123:0/3545010032 shutdown_connections 2024-03-30T04:44:49.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.079+0000 7f98d3e00700 1 --2- 172.21.15.123:0/3545010032 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f98b806cc30 0x7f98b806f0e0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:49.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.079+0000 7f98d3e00700 1 --2- 172.21.15.123:0/3545010032 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98d4070360 0x7f98d4115ac0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:49.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.079+0000 7f98d3e00700 1 --2- 172.21.15.123:0/3545010032 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f98d4070c70 0x7f98d4116000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:49.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.079+0000 7f98d3e00700 1 -- 172.21.15.123:0/3545010032 >> 172.21.15.123:0/3545010032 conn(0x7f98d40f8830 msgr2=0x7f98d40f9d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:44:49.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.079+0000 7f98d3e00700 1 -- 172.21.15.123:0/3545010032 shutdown_connections 2024-03-30T04:44:49.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.079+0000 7f98d3e00700 1 -- 172.21.15.123:0/3545010032 wait complete. 2024-03-30T04:44:49.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:44:48 smithi135 ceph-mon[106011]: pgmap v408: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:49.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.210+0000 7f1595a00700 1 -- 172.21.15.123:0/965589384 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1590102e20 msgr2=0x7f15901031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:49.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.210+0000 7f1595a00700 1 --2- 172.21.15.123:0/965589384 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1590102e20 0x7f15901031f0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f1580004940 tx=0x7f1580009e30 comp rx=0 tx=0).stop 2024-03-30T04:44:49.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.210+0000 7f1595a00700 1 -- 172.21.15.123:0/965589384 shutdown_connections 2024-03-30T04:44:49.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.210+0000 7f1595a00700 1 --2- 172.21.15.123:0/965589384 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f15901037c0 0x7f15900fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:49.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.210+0000 7f1595a00700 1 --2- 172.21.15.123:0/965589384 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1590102e20 0x7f15901031f0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:49.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.210+0000 7f1595a00700 1 -- 172.21.15.123:0/965589384 >> 172.21.15.123:0/965589384 conn(0x7f15900f8970 msgr2=0x7f15900fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:44:49.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.210+0000 7f1595a00700 1 -- 172.21.15.123:0/965589384 shutdown_connections 2024-03-30T04:44:49.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.210+0000 7f1595a00700 1 -- 172.21.15.123:0/965589384 wait complete. 2024-03-30T04:44:49.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.211+0000 7f1595a00700 1 Processor -- start 2024-03-30T04:44:49.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.211+0000 7f1595a00700 1 -- start start 2024-03-30T04:44:49.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.211+0000 7f1595a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1590102e20 0x7f1590115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:44:49.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.211+0000 7f1595a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15901037c0 0x7f1590116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:44:49.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.211+0000 7f1595a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1590116880 con 0x7f15901037c0 2024-03-30T04:44:49.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.211+0000 7f1595a00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f159010fc50 con 0x7f1590102e20 2024-03-30T04:44:49.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.212+0000 7f158e000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15901037c0 0x7f1590116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:44:49.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.212+0000 7f158e000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15901037c0 0x7f1590116230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36106/0 (socket says 172.21.15.123:36106) 2024-03-30T04:44:49.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.212+0000 7f158e000700 1 -- 172.21.15.123:0/1703583951 learned_addr learned my addr 172.21.15.123:0/1703583951 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:44:49.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.212+0000 7f158ea00700 1 --2- 172.21.15.123:0/1703583951 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1590102e20 0x7f1590115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:44:49.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.212+0000 7f158e000700 1 -- 172.21.15.123:0/1703583951 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1590102e20 msgr2=0x7f1590115cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:49.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.212+0000 7f158e000700 1 --2- 172.21.15.123:0/1703583951 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1590102e20 0x7f1590115cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:49.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.212+0000 7f158e000700 1 -- 172.21.15.123:0/1703583951 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1580005040 con 0x7f15901037c0 2024-03-30T04:44:49.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.212+0000 7f158e000700 1 --2- 172.21.15.123:0/1703583951 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15901037c0 0x7f1590116230 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f157c002a50 tx=0x7f157c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:44:49.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.212+0000 7f1586a00700 1 -- 172.21.15.123:0/1703583951 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f157c007cc0 con 0x7f15901037c0 2024-03-30T04:44:49.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.212+0000 7f1595a00700 1 -- 172.21.15.123:0/1703583951 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f159010ff30 con 0x7f15901037c0 2024-03-30T04:44:49.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.212+0000 7f1586a00700 1 -- 172.21.15.123:0/1703583951 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f157c00f040 con 0x7f15901037c0 2024-03-30T04:44:49.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.212+0000 7f1586a00700 1 -- 172.21.15.123:0/1703583951 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f157c00d920 con 0x7f15901037c0 2024-03-30T04:44:49.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.213+0000 7f1595a00700 1 -- 172.21.15.123:0/1703583951 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15901104f0 con 0x7f15901037c0 2024-03-30T04:44:49.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.214+0000 7f1595a00700 1 -- 172.21.15.123:0/1703583951 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15900fe570 con 0x7f15901037c0 2024-03-30T04:44:49.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.214+0000 7f1586a00700 1 -- 172.21.15.123:0/1703583951 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f157c009680 con 0x7f15901037c0 2024-03-30T04:44:49.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.215+0000 7f1586a00700 1 --2- 172.21.15.123:0/1703583951 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f157406cc30 0x7f157406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:44:49.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.215+0000 7f1586a00700 1 -- 172.21.15.123:0/1703583951 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f157c08be30 con 0x7f15901037c0 2024-03-30T04:44:49.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.216+0000 7f158ea00700 1 --2- 172.21.15.123:0/1703583951 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f157406cc30 0x7f157406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:44:49.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.216+0000 7f158ea00700 1 --2- 172.21.15.123:0/1703583951 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f157406cc30 0x7f157406f0e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f158000b840 tx=0x7f1580009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:44:49.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.220+0000 7f1586a00700 1 -- 172.21.15.123:0/1703583951 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f157c059800 con 0x7f15901037c0 2024-03-30T04:44:49.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.507+0000 7f1595a00700 1 -- 172.21.15.123:0/1703583951 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f159004bed0 con 0x7f15901037c0 2024-03-30T04:44:49.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.508+0000 7f1586a00700 1 -- 172.21.15.123:0/1703583951 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f157c059390 con 0x7f15901037c0 2024-03-30T04:44:49.506 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:44:49.506 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:44:49.506 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:44:49.507 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:44:49.507 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:44:49.507 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:44:49.507 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:44:49.507 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:44:49.507 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T04:44:49.507 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:44:49.507 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:44:49.507 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:44:49.507 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 3 2024-03-30T04:44:49.507 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:44:49.507 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:44:49.507 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:44:49.507 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 13 2024-03-30T04:44:49.507 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:44:49.508 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:44:49.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.510+0000 7f1595a00700 1 -- 172.21.15.123:0/1703583951 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f157406cc30 msgr2=0x7f157406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:49.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.510+0000 7f1595a00700 1 --2- 172.21.15.123:0/1703583951 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f157406cc30 0x7f157406f0e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f158000b840 tx=0x7f1580009e30 comp rx=0 tx=0).stop 2024-03-30T04:44:49.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.510+0000 7f1595a00700 1 -- 172.21.15.123:0/1703583951 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15901037c0 msgr2=0x7f1590116230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:49.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.510+0000 7f1595a00700 1 --2- 172.21.15.123:0/1703583951 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15901037c0 0x7f1590116230 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f157c002a50 tx=0x7f157c004600 comp rx=0 tx=0).stop 2024-03-30T04:44:49.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.510+0000 7f1595a00700 1 -- 172.21.15.123:0/1703583951 shutdown_connections 2024-03-30T04:44:49.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.510+0000 7f1595a00700 1 --2- 172.21.15.123:0/1703583951 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f157406cc30 0x7f157406f0e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:49.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.510+0000 7f1595a00700 1 --2- 172.21.15.123:0/1703583951 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1590102e20 0x7f1590115cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:49.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.510+0000 7f1595a00700 1 --2- 172.21.15.123:0/1703583951 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15901037c0 0x7f1590116230 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:49.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.510+0000 7f1595a00700 1 -- 172.21.15.123:0/1703583951 >> 172.21.15.123:0/1703583951 conn(0x7f15900f8970 msgr2=0x7f15900f99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:44:49.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.511+0000 7f1595a00700 1 -- 172.21.15.123:0/1703583951 shutdown_connections 2024-03-30T04:44:49.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.511+0000 7f1595a00700 1 -- 172.21.15.123:0/1703583951 wait complete. 2024-03-30T04:44:49.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.644+0000 7f4462000700 1 -- 172.21.15.123:0/1792789067 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f445c070c10 msgr2=0x7f445c109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:49.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.645+0000 7f4462000700 1 --2- 172.21.15.123:0/1792789067 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f445c070c10 0x7f445c109170 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f4448004970 tx=0x7f4448009e30 comp rx=0 tx=0).stop 2024-03-30T04:44:49.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.645+0000 7f4462000700 1 -- 172.21.15.123:0/1792789067 shutdown_connections 2024-03-30T04:44:49.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.645+0000 7f4462000700 1 --2- 172.21.15.123:0/1792789067 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f445c070c10 0x7f445c109170 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:49.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.645+0000 7f4462000700 1 --2- 172.21.15.123:0/1792789067 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f445c070300 0x7f445c0706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:49.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.645+0000 7f4462000700 1 -- 172.21.15.123:0/1792789067 >> 172.21.15.123:0/1792789067 conn(0x7f445c0f89b0 msgr2=0x7f445c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:44:49.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.645+0000 7f4462000700 1 -- 172.21.15.123:0/1792789067 shutdown_connections 2024-03-30T04:44:49.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.645+0000 7f4462000700 1 -- 172.21.15.123:0/1792789067 wait complete. 2024-03-30T04:44:49.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.646+0000 7f4462000700 1 Processor -- start 2024-03-30T04:44:49.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.646+0000 7f4462000700 1 -- start start 2024-03-30T04:44:49.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.646+0000 7f4462000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f445c070300 0x7f445c115a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:44:49.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.646+0000 7f4462000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f445c070c10 0x7f445c115f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:44:49.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.646+0000 7f4462000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f445c1165c0 con 0x7f445c070300 2024-03-30T04:44:49.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.646+0000 7f4462000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f445c10fbe0 con 0x7f445c070c10 2024-03-30T04:44:49.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.646+0000 7f445b400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f445c070300 0x7f445c115a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:44:49.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.646+0000 7f445b400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f445c070300 0x7f445c115a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36132/0 (socket says 172.21.15.123:36132) 2024-03-30T04:44:49.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.646+0000 7f445b400700 1 -- 172.21.15.123:0/90655579 learned_addr learned my addr 172.21.15.123:0/90655579 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:44:49.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.647+0000 7f445b400700 1 -- 172.21.15.123:0/90655579 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f445c070c10 msgr2=0x7f445c115f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:49.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.647+0000 7f445b400700 1 --2- 172.21.15.123:0/90655579 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f445c070c10 0x7f445c115f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:49.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.647+0000 7f445b400700 1 -- 172.21.15.123:0/90655579 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4448005040 con 0x7f445c070300 2024-03-30T04:44:49.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.647+0000 7f445b400700 1 --2- 172.21.15.123:0/90655579 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f445c070300 0x7f445c115a30 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f444c002a50 tx=0x7f444c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:44:49.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.647+0000 7f4453400700 1 -- 172.21.15.123:0/90655579 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f444c007cc0 con 0x7f445c070300 2024-03-30T04:44:49.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.647+0000 7f4453400700 1 -- 172.21.15.123:0/90655579 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f444c00f040 con 0x7f445c070300 2024-03-30T04:44:49.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.647+0000 7f4462000700 1 -- 172.21.15.123:0/90655579 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f445c10fec0 con 0x7f445c070300 2024-03-30T04:44:49.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.647+0000 7f4453400700 1 -- 172.21.15.123:0/90655579 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f444c00da70 con 0x7f445c070300 2024-03-30T04:44:49.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.647+0000 7f4462000700 1 -- 172.21.15.123:0/90655579 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f445c110400 con 0x7f445c070300 2024-03-30T04:44:49.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.648+0000 7f4462000700 1 -- 172.21.15.123:0/90655579 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f445c071a40 con 0x7f445c070300 2024-03-30T04:44:49.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.649+0000 7f4453400700 1 -- 172.21.15.123:0/90655579 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f444c00dc10 con 0x7f445c070300 2024-03-30T04:44:49.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.650+0000 7f4453400700 1 --2- 172.21.15.123:0/90655579 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f444006cc20 0x7f444006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:44:49.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.650+0000 7f4453400700 1 -- 172.21.15.123:0/90655579 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f444c08b740 con 0x7f445c070300 2024-03-30T04:44:49.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.650+0000 7f445aa00700 1 --2- 172.21.15.123:0/90655579 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f444006cc20 0x7f444006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:44:49.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.651+0000 7f445aa00700 1 --2- 172.21.15.123:0/90655579 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f444006cc20 0x7f444006f0d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f4448000c00 tx=0x7f4448009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:44:49.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.655+0000 7f4453400700 1 -- 172.21.15.123:0/90655579 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f444c055700 con 0x7f445c070300 2024-03-30T04:44:49.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.899+0000 7f4462000700 1 -- 172.21.15.123:0/90655579 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f445c10bfe0 con 0x7f445c070300 2024-03-30T04:44:49.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.899+0000 7f4453400700 1 -- 172.21.15.123:0/90655579 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f444c058d20 con 0x7f445c070300 2024-03-30T04:44:49.898 INFO:teuthology.orchestra.run.smithi123.stdout:e32 2024-03-30T04:44:49.898 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:44:49.898 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:44:49.898 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:44:49.898 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:44:49.898 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:44:49.898 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:44:49.898 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T04:44:49.898 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T04:44:49.898 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:44:49.898 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T04:44:49.899 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:44:49.899 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:44:49.899 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:44:49.899 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:44:49.899 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:44:49.899 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T04:44:49.899 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:44:49.899 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:44:49.899 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T04:44:49.899 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:44:49.899 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T04:44:49.899 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T04:44:49.899 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T04:44:49.899 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T04:44:49.899 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:44:49.900 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:44:49.900 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:44:49.900 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:44:49.900 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:44:49.900 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:44:49.900 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T04:44:49.900 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T04:44:49.900 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:44:49.900 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:44:49.900 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T04:44:49.900 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:44:49.900 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:44:49.900 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:44:49.900 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44315} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/1741352336,v1:172.21.15.135:6827/1741352336] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:44:49.900 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:44:49.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.901+0000 7f4462000700 1 -- 172.21.15.123:0/90655579 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f444006cc20 msgr2=0x7f444006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:49.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.901+0000 7f4462000700 1 --2- 172.21.15.123:0/90655579 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f444006cc20 0x7f444006f0d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f4448000c00 tx=0x7f4448009e30 comp rx=0 tx=0).stop 2024-03-30T04:44:49.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.901+0000 7f4462000700 1 -- 172.21.15.123:0/90655579 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f445c070300 msgr2=0x7f445c115a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:49.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.901+0000 7f4462000700 1 --2- 172.21.15.123:0/90655579 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f445c070300 0x7f445c115a30 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f444c002a50 tx=0x7f444c0046b0 comp rx=0 tx=0).stop 2024-03-30T04:44:49.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.902+0000 7f4462000700 1 -- 172.21.15.123:0/90655579 shutdown_connections 2024-03-30T04:44:49.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.902+0000 7f4462000700 1 --2- 172.21.15.123:0/90655579 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f444006cc20 0x7f444006f0d0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:49.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.902+0000 7f4462000700 1 --2- 172.21.15.123:0/90655579 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f445c070300 0x7f445c115a30 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:49.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.902+0000 7f4462000700 1 --2- 172.21.15.123:0/90655579 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f445c070c10 0x7f445c115f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:49.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.902+0000 7f4462000700 1 -- 172.21.15.123:0/90655579 >> 172.21.15.123:0/90655579 conn(0x7f445c0f89b0 msgr2=0x7f445c0f9ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:44:49.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.902+0000 7f4462000700 1 -- 172.21.15.123:0/90655579 shutdown_connections 2024-03-30T04:44:49.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:49.902+0000 7f4462000700 1 -- 172.21.15.123:0/90655579 wait complete. 2024-03-30T04:44:49.902 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 32 2024-03-30T04:44:49.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:44:49 smithi123 ceph-mon[139369]: from='client.34794 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:44:49.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:44:49 smithi123 ceph-mon[139369]: from='client.34798 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:44:49.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:44:49 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/1703583951' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:44:50.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.036+0000 7f97c5600700 1 -- 172.21.15.123:0/860218376 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f97c0102e00 msgr2=0x7f97c01031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:50.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.036+0000 7f97c5600700 1 --2- 172.21.15.123:0/860218376 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f97c0102e00 0x7f97c01031d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f97b0004940 tx=0x7f97b0009e30 comp rx=0 tx=0).stop 2024-03-30T04:44:50.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.036+0000 7f97c5600700 1 -- 172.21.15.123:0/860218376 shutdown_connections 2024-03-30T04:44:50.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.036+0000 7f97c5600700 1 --2- 172.21.15.123:0/860218376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97c01037a0 0x7f97c00fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:50.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.036+0000 7f97c5600700 1 --2- 172.21.15.123:0/860218376 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f97c0102e00 0x7f97c01031d0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:50.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.036+0000 7f97c5600700 1 -- 172.21.15.123:0/860218376 >> 172.21.15.123:0/860218376 conn(0x7f97c00f89b0 msgr2=0x7f97c00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:44:50.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.036+0000 7f97c5600700 1 -- 172.21.15.123:0/860218376 shutdown_connections 2024-03-30T04:44:50.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.036+0000 7f97c5600700 1 -- 172.21.15.123:0/860218376 wait complete. 2024-03-30T04:44:50.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.037+0000 7f97c5600700 1 Processor -- start 2024-03-30T04:44:50.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.037+0000 7f97c5600700 1 -- start start 2024-03-30T04:44:50.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.038+0000 7f97c5600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f97c0102e00 0x7f97c0115c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:44:50.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.038+0000 7f97c5600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97c01037a0 0x7f97c01161d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:44:50.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.038+0000 7f97c5600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c0116820 con 0x7f97c01037a0 2024-03-30T04:44:50.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.038+0000 7f97c5600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c010fbf0 con 0x7f97c0102e00 2024-03-30T04:44:50.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.038+0000 7f97be000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97c01037a0 0x7f97c01161d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:44:50.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.038+0000 7f97be000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97c01037a0 0x7f97c01161d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36142/0 (socket says 172.21.15.123:36142) 2024-03-30T04:44:50.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.038+0000 7f97be000700 1 -- 172.21.15.123:0/3959765929 learned_addr learned my addr 172.21.15.123:0/3959765929 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:44:50.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.038+0000 7f97bea00700 1 --2- 172.21.15.123:0/3959765929 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f97c0102e00 0x7f97c0115c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:44:50.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.038+0000 7f97be000700 1 -- 172.21.15.123:0/3959765929 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f97c0102e00 msgr2=0x7f97c0115c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:50.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.038+0000 7f97be000700 1 --2- 172.21.15.123:0/3959765929 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f97c0102e00 0x7f97c0115c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:50.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.038+0000 7f97be000700 1 -- 172.21.15.123:0/3959765929 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97ac005040 con 0x7f97c01037a0 2024-03-30T04:44:50.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.039+0000 7f97be000700 1 --2- 172.21.15.123:0/3959765929 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97c01037a0 0x7f97c01161d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f97ac009700 tx=0x7f97ac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:44:50.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.039+0000 7f97b6a00700 1 -- 172.21.15.123:0/3959765929 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97ac00e7e0 con 0x7f97c01037a0 2024-03-30T04:44:50.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.039+0000 7f97b6a00700 1 -- 172.21.15.123:0/3959765929 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f97ac00f040 con 0x7f97c01037a0 2024-03-30T04:44:50.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.039+0000 7f97c5600700 1 -- 172.21.15.123:0/3959765929 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97b0005040 con 0x7f97c01037a0 2024-03-30T04:44:50.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.039+0000 7f97b6a00700 1 -- 172.21.15.123:0/3959765929 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97ac013d90 con 0x7f97c01037a0 2024-03-30T04:44:50.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.039+0000 7f97c5600700 1 -- 172.21.15.123:0/3959765929 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97c01102a0 con 0x7f97c01037a0 2024-03-30T04:44:50.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.040+0000 7f97c5600700 1 -- 172.21.15.123:0/3959765929 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97c00fe550 con 0x7f97c01037a0 2024-03-30T04:44:50.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.041+0000 7f97b6a00700 1 -- 172.21.15.123:0/3959765929 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f97ac00e940 con 0x7f97c01037a0 2024-03-30T04:44:50.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.042+0000 7f97b6a00700 1 --2- 172.21.15.123:0/3959765929 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f97a406cc30 0x7f97a406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:44:50.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.042+0000 7f97b6a00700 1 -- 172.21.15.123:0/3959765929 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f97ac08bf30 con 0x7f97c01037a0 2024-03-30T04:44:50.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.042+0000 7f97bea00700 1 --2- 172.21.15.123:0/3959765929 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f97a406cc30 0x7f97a406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:44:50.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.043+0000 7f97bea00700 1 --2- 172.21.15.123:0/3959765929 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f97a406cc30 0x7f97a406f0e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f97b0004560 tx=0x7f97b0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:44:50.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.046+0000 7f97b6a00700 1 -- 172.21.15.123:0/3959765929 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f97ac059900 con 0x7f97c01037a0 2024-03-30T04:44:50.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:44:49 smithi135 ceph-mon[106011]: from='client.34794 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:44:50.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:44:49 smithi135 ceph-mon[106011]: from='client.34798 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:44:50.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:44:49 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/1703583951' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:44:50.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.264+0000 7f97c5600700 1 -- 172.21.15.123:0/3959765929 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f97c0109e30 con 0x7f97a406cc30 2024-03-30T04:44:50.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.266+0000 7f97b6a00700 1 -- 172.21.15.123:0/3959765929 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f97c0109e30 con 0x7f97a406cc30 2024-03-30T04:44:50.264 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:44:50.264 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:44:50.264 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:44:50.264 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T04:44:50.264 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:44:50.264 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T04:44:50.265 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T04:44:50.265 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T04:44:50.265 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T04:44:50.265 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:44:50.265 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "15/21 daemons upgraded", 2024-03-30T04:44:50.265 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T04:44:50.265 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:44:50.265 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:44:50.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.268+0000 7f97c5600700 1 -- 172.21.15.123:0/3959765929 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f97a406cc30 msgr2=0x7f97a406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:50.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.268+0000 7f97c5600700 1 --2- 172.21.15.123:0/3959765929 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f97a406cc30 0x7f97a406f0e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f97b0004560 tx=0x7f97b0009e30 comp rx=0 tx=0).stop 2024-03-30T04:44:50.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.268+0000 7f97c5600700 1 -- 172.21.15.123:0/3959765929 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97c01037a0 msgr2=0x7f97c01161d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:50.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.268+0000 7f97c5600700 1 --2- 172.21.15.123:0/3959765929 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97c01037a0 0x7f97c01161d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f97ac009700 tx=0x7f97ac009e30 comp rx=0 tx=0).stop 2024-03-30T04:44:50.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.268+0000 7f97c5600700 1 -- 172.21.15.123:0/3959765929 shutdown_connections 2024-03-30T04:44:50.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.268+0000 7f97c5600700 1 --2- 172.21.15.123:0/3959765929 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f97a406cc30 0x7f97a406f0e0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:50.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.268+0000 7f97c5600700 1 --2- 172.21.15.123:0/3959765929 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f97c0102e00 0x7f97c0115c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:50.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.268+0000 7f97c5600700 1 --2- 172.21.15.123:0/3959765929 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97c01037a0 0x7f97c01161d0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:50.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.268+0000 7f97c5600700 1 -- 172.21.15.123:0/3959765929 >> 172.21.15.123:0/3959765929 conn(0x7f97c00f89b0 msgr2=0x7f97c00f9910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:44:50.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.268+0000 7f97c5600700 1 -- 172.21.15.123:0/3959765929 shutdown_connections 2024-03-30T04:44:50.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.268+0000 7f97c5600700 1 -- 172.21.15.123:0/3959765929 wait complete. 2024-03-30T04:44:50.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.399+0000 7f08cb200700 1 -- 172.21.15.123:0/2555382968 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08c40fe850 msgr2=0x7f08c4106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:50.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.399+0000 7f08cb200700 1 --2- 172.21.15.123:0/2555382968 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08c40fe850 0x7f08c4106f20 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f08b0004970 tx=0x7f08b0009e30 comp rx=0 tx=0).stop 2024-03-30T04:44:50.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.399+0000 7f08cb200700 1 -- 172.21.15.123:0/2555382968 shutdown_connections 2024-03-30T04:44:50.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.399+0000 7f08cb200700 1 --2- 172.21.15.123:0/2555382968 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08c40fe850 0x7f08c4106f20 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:50.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.399+0000 7f08cb200700 1 --2- 172.21.15.123:0/2555382968 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f08c40fdeb0 0x7f08c40fe280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:50.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.399+0000 7f08cb200700 1 -- 172.21.15.123:0/2555382968 >> 172.21.15.123:0/2555382968 conn(0x7f08c4072bc0 msgr2=0x7f08c4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:44:50.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.400+0000 7f08cb200700 1 -- 172.21.15.123:0/2555382968 shutdown_connections 2024-03-30T04:44:50.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.400+0000 7f08cb200700 1 -- 172.21.15.123:0/2555382968 wait complete. 2024-03-30T04:44:50.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.400+0000 7f08cb200700 1 Processor -- start 2024-03-30T04:44:50.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.400+0000 7f08cb200700 1 -- start start 2024-03-30T04:44:50.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.401+0000 7f08cb200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08c40fdeb0 0x7f08c4115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:44:50.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.401+0000 7f08cb200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f08c40fe850 0x7f08c4116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:44:50.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.401+0000 7f08cb200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08c4116790 con 0x7f08c40fdeb0 2024-03-30T04:44:50.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.401+0000 7f08cb200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08c410fb80 con 0x7f08c40fe850 2024-03-30T04:44:50.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.401+0000 7f08c3e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08c40fdeb0 0x7f08c4115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:44:50.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.401+0000 7f08c3e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08c40fdeb0 0x7f08c4115c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36164/0 (socket says 172.21.15.123:36164) 2024-03-30T04:44:50.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.401+0000 7f08c3e00700 1 -- 172.21.15.123:0/2767818175 learned_addr learned my addr 172.21.15.123:0/2767818175 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:44:50.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.401+0000 7f08c3400700 1 --2- 172.21.15.123:0/2767818175 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f08c40fe850 0x7f08c4116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:44:50.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.402+0000 7f08c3e00700 1 -- 172.21.15.123:0/2767818175 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f08c40fe850 msgr2=0x7f08c4116140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:50.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.402+0000 7f08c3e00700 1 --2- 172.21.15.123:0/2767818175 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f08c40fe850 0x7f08c4116140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:50.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.402+0000 7f08c3e00700 1 -- 172.21.15.123:0/2767818175 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08b0005040 con 0x7f08c40fdeb0 2024-03-30T04:44:50.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.402+0000 7f08c3400700 1 --2- 172.21.15.123:0/2767818175 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f08c40fe850 0x7f08c4116140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:44:50.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.402+0000 7f08c3e00700 1 --2- 172.21.15.123:0/2767818175 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08c40fdeb0 0x7f08c4115c00 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f08b400eb40 tx=0x7f08b400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:44:50.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.402+0000 7f08bbe00700 1 -- 172.21.15.123:0/2767818175 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f08b4007cc0 con 0x7f08c40fdeb0 2024-03-30T04:44:50.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.402+0000 7f08bbe00700 1 -- 172.21.15.123:0/2767818175 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f08b400f040 con 0x7f08c40fdeb0 2024-03-30T04:44:50.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.402+0000 7f08cb200700 1 -- 172.21.15.123:0/2767818175 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08c410fe00 con 0x7f08c40fdeb0 2024-03-30T04:44:50.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.402+0000 7f08bbe00700 1 -- 172.21.15.123:0/2767818175 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f08b4009570 con 0x7f08c40fdeb0 2024-03-30T04:44:50.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.402+0000 7f08cb200700 1 -- 172.21.15.123:0/2767818175 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08c4110310 con 0x7f08c40fdeb0 2024-03-30T04:44:50.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.403+0000 7f08cb200700 1 -- 172.21.15.123:0/2767818175 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08c41069f0 con 0x7f08c40fdeb0 2024-03-30T04:44:50.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.404+0000 7f08bbe00700 1 -- 172.21.15.123:0/2767818175 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f08b40096d0 con 0x7f08c40fdeb0 2024-03-30T04:44:50.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.405+0000 7f08bbe00700 1 --2- 172.21.15.123:0/2767818175 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f08a806cc30 0x7f08a806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:44:50.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.406+0000 7f08bbe00700 1 -- 172.21.15.123:0/2767818175 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f08b408bcb0 con 0x7f08c40fdeb0 2024-03-30T04:44:50.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.406+0000 7f08c3400700 1 --2- 172.21.15.123:0/2767818175 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f08a806cc30 0x7f08a806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:44:50.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.406+0000 7f08c3400700 1 --2- 172.21.15.123:0/2767818175 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f08a806cc30 0x7f08a806f0e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f08c4110e60 tx=0x7f08b0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:44:50.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.410+0000 7f08bbe00700 1 -- 172.21.15.123:0/2767818175 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f08b40597b0 con 0x7f08c40fdeb0 2024-03-30T04:44:50.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.693+0000 7f08cb200700 1 -- 172.21.15.123:0/2767818175 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f08c404bed0 con 0x7f08c40fdeb0 2024-03-30T04:44:50.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.693+0000 7f08bbe00700 1 -- 172.21.15.123:0/2767818175 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f08b4019100 con 0x7f08c40fdeb0 2024-03-30T04:44:50.692 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T04:44:50.692 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T04:44:50.692 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T04:44:50.692 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T04:44:50.692 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T04:44:50.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.696+0000 7f08cb200700 1 -- 172.21.15.123:0/2767818175 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f08a806cc30 msgr2=0x7f08a806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:50.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.696+0000 7f08cb200700 1 --2- 172.21.15.123:0/2767818175 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f08a806cc30 0x7f08a806f0e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f08c4110e60 tx=0x7f08b0009e30 comp rx=0 tx=0).stop 2024-03-30T04:44:50.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.696+0000 7f08cb200700 1 -- 172.21.15.123:0/2767818175 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08c40fdeb0 msgr2=0x7f08c4115c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:44:50.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.696+0000 7f08cb200700 1 --2- 172.21.15.123:0/2767818175 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08c40fdeb0 0x7f08c4115c00 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f08b400eb40 tx=0x7f08b400b040 comp rx=0 tx=0).stop 2024-03-30T04:44:50.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.696+0000 7f08cb200700 1 -- 172.21.15.123:0/2767818175 shutdown_connections 2024-03-30T04:44:50.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.696+0000 7f08cb200700 1 --2- 172.21.15.123:0/2767818175 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f08a806cc30 0x7f08a806f0e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:50.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.696+0000 7f08cb200700 1 --2- 172.21.15.123:0/2767818175 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08c40fdeb0 0x7f08c4115c00 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:50.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.696+0000 7f08cb200700 1 --2- 172.21.15.123:0/2767818175 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f08c40fe850 0x7f08c4116140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:44:50.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.696+0000 7f08cb200700 1 -- 172.21.15.123:0/2767818175 >> 172.21.15.123:0/2767818175 conn(0x7f08c4072bc0 msgr2=0x7f08c40fb640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:44:50.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.696+0000 7f08cb200700 1 -- 172.21.15.123:0/2767818175 shutdown_connections 2024-03-30T04:44:50.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:44:50.696+0000 7f08cb200700 1 -- 172.21.15.123:0/2767818175 wait complete. 2024-03-30T04:44:50.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:44:50 smithi123 ceph-mon[139369]: from='client.34802 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:44:50.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:44:50 smithi123 ceph-mon[139369]: pgmap v409: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:50.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:44:50 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/90655579' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:44:51.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:44:50 smithi135 ceph-mon[106011]: from='client.34802 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:44:51.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:44:50 smithi135 ceph-mon[106011]: pgmap v409: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:51.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:44:50 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/90655579' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:44:51.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:44:51 smithi123 ceph-mon[139369]: from='client.34814 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:44:51.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:44:51 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2767818175' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:44:52.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:44:51 smithi135 ceph-mon[106011]: from='client.34814 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:44:52.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:44:51 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2767818175' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:44:52.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:44:52 smithi123 ceph-mon[139369]: pgmap v410: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:53.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:44:52 smithi135 ceph-mon[106011]: pgmap v410: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:54.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:44:54 smithi123 ceph-mon[139369]: pgmap v411: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:55.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:44:54 smithi135 ceph-mon[106011]: pgmap v411: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:56.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:44:56 smithi123 ceph-mon[139369]: pgmap v412: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:57.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:44:56 smithi135 ceph-mon[106011]: pgmap v412: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:58.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:44:58 smithi123 ceph-mon[139369]: pgmap v413: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:59.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:44:58 smithi135 ceph-mon[106011]: pgmap v413: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:44:59.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:44:59 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:45:00.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:44:59 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:45:00.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:45:00 smithi123 ceph-mon[139369]: pgmap v414: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:01.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:45:00 smithi135 ceph-mon[106011]: pgmap v414: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:02.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:45:02 smithi123 ceph-mon[139369]: pgmap v415: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:03.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:45:02 smithi135 ceph-mon[106011]: pgmap v415: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:04.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:45:04 smithi123 ceph-mon[139369]: pgmap v416: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:05.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:45:04 smithi135 ceph-mon[106011]: pgmap v416: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:06.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:45:06 smithi123 ceph-mon[139369]: pgmap v417: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:07.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:45:06 smithi135 ceph-mon[106011]: pgmap v417: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:08.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:45:08 smithi123 ceph-mon[139369]: pgmap v418: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:09.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:45:08 smithi135 ceph-mon[106011]: pgmap v418: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:10.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:45:10 smithi135 ceph-mon[106011]: pgmap v419: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:10.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:45:10 smithi123 ceph-mon[139369]: pgmap v419: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:12.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:45:12 smithi123 ceph-mon[139369]: pgmap v420: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:13.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:45:12 smithi135 ceph-mon[106011]: pgmap v420: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:14.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:45:14 smithi123 ceph-mon[139369]: pgmap v421: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:14.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:45:14 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:45:15.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:45:14 smithi135 ceph-mon[106011]: pgmap v421: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:15.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:45:14 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:45:16.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:45:16 smithi123 ceph-mon[139369]: pgmap v422: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:17.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:45:16 smithi135 ceph-mon[106011]: pgmap v422: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:18.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:45:18 smithi123 ceph-mon[139369]: pgmap v423: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:19.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:45:18 smithi135 ceph-mon[106011]: pgmap v423: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:20.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:20.843+0000 7feb8d600700 1 -- 172.21.15.123:0/1128053425 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb88105090 msgr2=0x7feb88105460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:20.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:20.843+0000 7feb8d600700 1 --2- 172.21.15.123:0/1128053425 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb88105090 0x7feb88105460 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7feb78004940 tx=0x7feb78009e30 comp rx=0 tx=0).stop 2024-03-30T04:45:20.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:20.843+0000 7feb8d600700 1 -- 172.21.15.123:0/1128053425 shutdown_connections 2024-03-30T04:45:20.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:20.843+0000 7feb8d600700 1 --2- 172.21.15.123:0/1128053425 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feb88105a30 0x7feb880ff0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:20.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:20.843+0000 7feb8d600700 1 --2- 172.21.15.123:0/1128053425 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb88105090 0x7feb88105460 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:20.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:20.843+0000 7feb8d600700 1 -- 172.21.15.123:0/1128053425 >> 172.21.15.123:0/1128053425 conn(0x7feb880fac00 msgr2=0x7feb880fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:45:20.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:20.843+0000 7feb8d600700 1 -- 172.21.15.123:0/1128053425 shutdown_connections 2024-03-30T04:45:20.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:20.843+0000 7feb8d600700 1 -- 172.21.15.123:0/1128053425 wait complete. 2024-03-30T04:45:20.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:20.844+0000 7feb8d600700 1 Processor -- start 2024-03-30T04:45:20.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:20.844+0000 7feb8d600700 1 -- start start 2024-03-30T04:45:20.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:20.844+0000 7feb8d600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb88105090 0x7feb88115a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:45:20.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:20.844+0000 7feb8d600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feb88105a30 0x7feb88115fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:45:20.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:20.844+0000 7feb8d600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb88116620 con 0x7feb88105090 2024-03-30T04:45:20.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:20.844+0000 7feb8d600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb8810fc00 con 0x7feb88105a30 2024-03-30T04:45:20.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:20.845+0000 7feb86a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb88105090 0x7feb88115a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:45:20.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:20.845+0000 7feb86a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb88105090 0x7feb88115a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43376/0 (socket says 172.21.15.123:43376) 2024-03-30T04:45:20.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:20.845+0000 7feb86a00700 1 -- 172.21.15.123:0/3074862236 learned_addr learned my addr 172.21.15.123:0/3074862236 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:45:20.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:20.845+0000 7feb86a00700 1 -- 172.21.15.123:0/3074862236 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feb88105a30 msgr2=0x7feb88115fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:20.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:20.845+0000 7feb86a00700 1 --2- 172.21.15.123:0/3074862236 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feb88105a30 0x7feb88115fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:20.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:20.845+0000 7feb86a00700 1 -- 172.21.15.123:0/3074862236 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb78005040 con 0x7feb88105090 2024-03-30T04:45:20.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:20.845+0000 7feb86a00700 1 --2- 172.21.15.123:0/3074862236 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb88105090 0x7feb88115a90 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7feb78009180 tx=0x7feb78009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:45:20.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:20.845+0000 7feb7ea00700 1 -- 172.21.15.123:0/3074862236 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb7800e040 con 0x7feb88105090 2024-03-30T04:45:20.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:20.845+0000 7feb7ea00700 1 -- 172.21.15.123:0/3074862236 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feb78004290 con 0x7feb88105090 2024-03-30T04:45:20.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:20.845+0000 7feb8d600700 1 -- 172.21.15.123:0/3074862236 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb8810fe80 con 0x7feb88105090 2024-03-30T04:45:20.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:20.845+0000 7feb7ea00700 1 -- 172.21.15.123:0/3074862236 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb7801c3f0 con 0x7feb88105090 2024-03-30T04:45:20.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:20.846+0000 7feb8d600700 1 -- 172.21.15.123:0/3074862236 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb88110440 con 0x7feb88105090 2024-03-30T04:45:20.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:20.847+0000 7feb8d600700 1 -- 172.21.15.123:0/3074862236 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb881023a0 con 0x7feb88105090 2024-03-30T04:45:20.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:20.848+0000 7feb7ea00700 1 -- 172.21.15.123:0/3074862236 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7feb78013790 con 0x7feb88105090 2024-03-30T04:45:20.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:20.848+0000 7feb7ea00700 1 --2- 172.21.15.123:0/3074862236 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7feb6c06cc20 0x7feb6c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:45:20.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:20.849+0000 7feb7ea00700 1 -- 172.21.15.123:0/3074862236 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7feb7808d370 con 0x7feb88105090 2024-03-30T04:45:20.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:20.849+0000 7feb86000700 1 --2- 172.21.15.123:0/3074862236 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7feb6c06cc20 0x7feb6c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:45:20.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:20.849+0000 7feb86000700 1 --2- 172.21.15.123:0/3074862236 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7feb6c06cc20 0x7feb6c06f0d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7feb88111330 tx=0x7feb74009ce0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:45:20.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:45:20 smithi135 ceph-mon[106011]: pgmap v424: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:20.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:20.853+0000 7feb7ea00700 1 -- 172.21.15.123:0/3074862236 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7feb7805adc0 con 0x7feb88105090 2024-03-30T04:45:20.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:45:20 smithi123 ceph-mon[139369]: pgmap v424: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:21.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.076+0000 7feb8d600700 1 -- 172.21.15.123:0/3074862236 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7feb8810c050 con 0x7feb6c06cc20 2024-03-30T04:45:21.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.078+0000 7feb7ea00700 1 -- 172.21.15.123:0/3074862236 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7feb8810c050 con 0x7feb6c06cc20 2024-03-30T04:45:21.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.080+0000 7feb8d600700 1 -- 172.21.15.123:0/3074862236 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7feb6c06cc20 msgr2=0x7feb6c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:21.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.080+0000 7feb8d600700 1 --2- 172.21.15.123:0/3074862236 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7feb6c06cc20 0x7feb6c06f0d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7feb88111330 tx=0x7feb74009ce0 comp rx=0 tx=0).stop 2024-03-30T04:45:21.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.080+0000 7feb8d600700 1 -- 172.21.15.123:0/3074862236 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb88105090 msgr2=0x7feb88115a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:21.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.080+0000 7feb8d600700 1 --2- 172.21.15.123:0/3074862236 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb88105090 0x7feb88115a90 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7feb78009180 tx=0x7feb78009e30 comp rx=0 tx=0).stop 2024-03-30T04:45:21.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.080+0000 7feb8d600700 1 -- 172.21.15.123:0/3074862236 shutdown_connections 2024-03-30T04:45:21.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.080+0000 7feb8d600700 1 --2- 172.21.15.123:0/3074862236 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7feb6c06cc20 0x7feb6c06f0d0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:21.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.080+0000 7feb8d600700 1 --2- 172.21.15.123:0/3074862236 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb88105090 0x7feb88115a90 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:21.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.080+0000 7feb8d600700 1 --2- 172.21.15.123:0/3074862236 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feb88105a30 0x7feb88115fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:21.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.080+0000 7feb8d600700 1 -- 172.21.15.123:0/3074862236 >> 172.21.15.123:0/3074862236 conn(0x7feb880fac00 msgr2=0x7feb880fc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:45:21.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.080+0000 7feb8d600700 1 -- 172.21.15.123:0/3074862236 shutdown_connections 2024-03-30T04:45:21.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.080+0000 7feb8d600700 1 -- 172.21.15.123:0/3074862236 wait complete. 2024-03-30T04:45:21.093 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:45:21.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.216+0000 7f95d8c00700 1 -- 172.21.15.123:0/2522218788 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95d41037a0 msgr2=0x7f95d40fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:21.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.216+0000 7f95d8c00700 1 --2- 172.21.15.123:0/2522218788 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95d41037a0 0x7f95d40fce10 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f95c0004940 tx=0x7f95c0009e30 comp rx=0 tx=0).stop 2024-03-30T04:45:21.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.216+0000 7f95d8c00700 1 -- 172.21.15.123:0/2522218788 shutdown_connections 2024-03-30T04:45:21.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.216+0000 7f95d8c00700 1 --2- 172.21.15.123:0/2522218788 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95d41037a0 0x7f95d40fce10 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:21.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.216+0000 7f95d8c00700 1 --2- 172.21.15.123:0/2522218788 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f95d4102e00 0x7f95d41031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:21.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.216+0000 7f95d8c00700 1 -- 172.21.15.123:0/2522218788 >> 172.21.15.123:0/2522218788 conn(0x7f95d40f89b0 msgr2=0x7f95d40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:45:21.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.216+0000 7f95d8c00700 1 -- 172.21.15.123:0/2522218788 shutdown_connections 2024-03-30T04:45:21.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.216+0000 7f95d8c00700 1 -- 172.21.15.123:0/2522218788 wait complete. 2024-03-30T04:45:21.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.217+0000 7f95d8c00700 1 Processor -- start 2024-03-30T04:45:21.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.217+0000 7f95d8c00700 1 -- start start 2024-03-30T04:45:21.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.217+0000 7f95d8c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95d4102e00 0x7f95d4115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:45:21.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.217+0000 7f95d8c00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f95d41037a0 0x7f95d4116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:45:21.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.217+0000 7f95d8c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d4116880 con 0x7f95d4102e00 2024-03-30T04:45:21.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.217+0000 7f95d8c00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d410fc50 con 0x7f95d41037a0 2024-03-30T04:45:21.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.217+0000 7f95d1e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95d4102e00 0x7f95d4115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:45:21.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.217+0000 7f95d1e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95d4102e00 0x7f95d4115cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43404/0 (socket says 172.21.15.123:43404) 2024-03-30T04:45:21.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.217+0000 7f95d1e00700 1 -- 172.21.15.123:0/948963501 learned_addr learned my addr 172.21.15.123:0/948963501 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:45:21.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.218+0000 7f95d1400700 1 --2- 172.21.15.123:0/948963501 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f95d41037a0 0x7f95d4116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:45:21.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.218+0000 7f95d1e00700 1 -- 172.21.15.123:0/948963501 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f95d41037a0 msgr2=0x7f95d4116230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:21.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.218+0000 7f95d1e00700 1 --2- 172.21.15.123:0/948963501 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f95d41037a0 0x7f95d4116230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:21.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.218+0000 7f95d1e00700 1 -- 172.21.15.123:0/948963501 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95c0005040 con 0x7f95d4102e00 2024-03-30T04:45:21.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.218+0000 7f95d1e00700 1 --2- 172.21.15.123:0/948963501 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95d4102e00 0x7f95d4115cf0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f95c4002a50 tx=0x7f95c40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:45:21.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.218+0000 7f95ca000700 1 -- 172.21.15.123:0/948963501 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f95c4007cc0 con 0x7f95d4102e00 2024-03-30T04:45:21.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.218+0000 7f95ca000700 1 -- 172.21.15.123:0/948963501 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f95c400f040 con 0x7f95d4102e00 2024-03-30T04:45:21.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.218+0000 7f95d8c00700 1 -- 172.21.15.123:0/948963501 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95d410ff30 con 0x7f95d4102e00 2024-03-30T04:45:21.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.218+0000 7f95ca000700 1 -- 172.21.15.123:0/948963501 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f95c4009540 con 0x7f95d4102e00 2024-03-30T04:45:21.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.218+0000 7f95d8c00700 1 -- 172.21.15.123:0/948963501 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95d41104f0 con 0x7f95d4102e00 2024-03-30T04:45:21.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.219+0000 7f95d8c00700 1 -- 172.21.15.123:0/948963501 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95d40fe550 con 0x7f95d4102e00 2024-03-30T04:45:21.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.220+0000 7f95ca000700 1 -- 172.21.15.123:0/948963501 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f95c40096a0 con 0x7f95d4102e00 2024-03-30T04:45:21.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.221+0000 7f95ca000700 1 --2- 172.21.15.123:0/948963501 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f95b806cc30 0x7f95b806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:45:21.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.221+0000 7f95ca000700 1 -- 172.21.15.123:0/948963501 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f95c408bde0 con 0x7f95d4102e00 2024-03-30T04:45:21.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.221+0000 7f95d1400700 1 --2- 172.21.15.123:0/948963501 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f95b806cc30 0x7f95b806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:45:21.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.222+0000 7f95d1400700 1 --2- 172.21.15.123:0/948963501 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f95b806cc30 0x7f95b806f0e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f95d4111590 tx=0x7f95c0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:45:21.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.226+0000 7f95ca000700 1 -- 172.21.15.123:0/948963501 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f95c4059830 con 0x7f95d4102e00 2024-03-30T04:45:21.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.448+0000 7f95d8c00700 1 -- 172.21.15.123:0/948963501 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f95d404bf40 con 0x7f95b806cc30 2024-03-30T04:45:21.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.449+0000 7f95ca000700 1 -- 172.21.15.123:0/948963501 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f95d404bf40 con 0x7f95b806cc30 2024-03-30T04:45:21.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.452+0000 7f95d8c00700 1 -- 172.21.15.123:0/948963501 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f95b806cc30 msgr2=0x7f95b806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:21.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.452+0000 7f95d8c00700 1 --2- 172.21.15.123:0/948963501 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f95b806cc30 0x7f95b806f0e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f95d4111590 tx=0x7f95c0009e30 comp rx=0 tx=0).stop 2024-03-30T04:45:21.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.452+0000 7f95d8c00700 1 -- 172.21.15.123:0/948963501 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95d4102e00 msgr2=0x7f95d4115cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:21.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.452+0000 7f95d8c00700 1 --2- 172.21.15.123:0/948963501 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95d4102e00 0x7f95d4115cf0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f95c4002a50 tx=0x7f95c40046b0 comp rx=0 tx=0).stop 2024-03-30T04:45:21.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.452+0000 7f95d8c00700 1 -- 172.21.15.123:0/948963501 shutdown_connections 2024-03-30T04:45:21.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.452+0000 7f95d8c00700 1 --2- 172.21.15.123:0/948963501 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f95b806cc30 0x7f95b806f0e0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:21.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.452+0000 7f95d8c00700 1 --2- 172.21.15.123:0/948963501 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95d4102e00 0x7f95d4115cf0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:21.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.452+0000 7f95d8c00700 1 --2- 172.21.15.123:0/948963501 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f95d41037a0 0x7f95d4116230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:21.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.452+0000 7f95d8c00700 1 -- 172.21.15.123:0/948963501 >> 172.21.15.123:0/948963501 conn(0x7f95d40f89b0 msgr2=0x7f95d40f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:45:21.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.452+0000 7f95d8c00700 1 -- 172.21.15.123:0/948963501 shutdown_connections 2024-03-30T04:45:21.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.452+0000 7f95d8c00700 1 -- 172.21.15.123:0/948963501 wait complete. 2024-03-30T04:45:21.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.587+0000 7f7132e00700 1 -- 172.21.15.123:0/3386267022 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f712c1037a0 msgr2=0x7f712c0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:21.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.587+0000 7f7132e00700 1 --2- 172.21.15.123:0/3386267022 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f712c1037a0 0x7f712c0fce10 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f7118004940 tx=0x7f7118009e30 comp rx=0 tx=0).stop 2024-03-30T04:45:21.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.587+0000 7f7132e00700 1 -- 172.21.15.123:0/3386267022 shutdown_connections 2024-03-30T04:45:21.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.587+0000 7f7132e00700 1 --2- 172.21.15.123:0/3386267022 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f712c1037a0 0x7f712c0fce10 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:21.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.587+0000 7f7132e00700 1 --2- 172.21.15.123:0/3386267022 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f712c102e00 0x7f712c1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:21.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.587+0000 7f7132e00700 1 -- 172.21.15.123:0/3386267022 >> 172.21.15.123:0/3386267022 conn(0x7f712c0f89b0 msgr2=0x7f712c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:45:21.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.587+0000 7f7132e00700 1 -- 172.21.15.123:0/3386267022 shutdown_connections 2024-03-30T04:45:21.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.587+0000 7f7132e00700 1 -- 172.21.15.123:0/3386267022 wait complete. 2024-03-30T04:45:21.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.588+0000 7f7132e00700 1 Processor -- start 2024-03-30T04:45:21.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.588+0000 7f7132e00700 1 -- start start 2024-03-30T04:45:21.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.589+0000 7f7132e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f712c102e00 0x7f712c113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:45:21.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.589+0000 7f7132e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f712c1037a0 0x7f712c113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:45:21.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.589+0000 7f7132e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f712c114600 con 0x7f712c102e00 2024-03-30T04:45:21.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.589+0000 7f7132e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f712c10d9f0 con 0x7f712c1037a0 2024-03-30T04:45:21.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.589+0000 7f712be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f712c102e00 0x7f712c113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:45:21.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.589+0000 7f712be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f712c102e00 0x7f712c113a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43420/0 (socket says 172.21.15.123:43420) 2024-03-30T04:45:21.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.589+0000 7f712be00700 1 -- 172.21.15.123:0/229867760 learned_addr learned my addr 172.21.15.123:0/229867760 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:45:21.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.589+0000 7f712b400700 1 --2- 172.21.15.123:0/229867760 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f712c1037a0 0x7f712c113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:45:21.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.589+0000 7f712be00700 1 -- 172.21.15.123:0/229867760 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f712c1037a0 msgr2=0x7f712c113fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:21.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.589+0000 7f712be00700 1 --2- 172.21.15.123:0/229867760 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f712c1037a0 0x7f712c113fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:21.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.589+0000 7f712be00700 1 -- 172.21.15.123:0/229867760 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7118005040 con 0x7f712c102e00 2024-03-30T04:45:21.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.590+0000 7f712be00700 1 --2- 172.21.15.123:0/229867760 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f712c102e00 0x7f712c113a70 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f711c002a80 tx=0x7f711c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:45:21.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.590+0000 7f7123e00700 1 -- 172.21.15.123:0/229867760 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f711c007cc0 con 0x7f712c102e00 2024-03-30T04:45:21.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.590+0000 7f7123e00700 1 -- 172.21.15.123:0/229867760 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f711c00f040 con 0x7f712c102e00 2024-03-30T04:45:21.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.590+0000 7f7132e00700 1 -- 172.21.15.123:0/229867760 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f712c10dcd0 con 0x7f712c102e00 2024-03-30T04:45:21.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.590+0000 7f7123e00700 1 -- 172.21.15.123:0/229867760 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f711c00da60 con 0x7f712c102e00 2024-03-30T04:45:21.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.590+0000 7f7132e00700 1 -- 172.21.15.123:0/229867760 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f712c10e260 con 0x7f712c102e00 2024-03-30T04:45:21.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.591+0000 7f7132e00700 1 -- 172.21.15.123:0/229867760 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f712c0fe550 con 0x7f712c102e00 2024-03-30T04:45:21.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.592+0000 7f7123e00700 1 -- 172.21.15.123:0/229867760 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f711c00dbc0 con 0x7f712c102e00 2024-03-30T04:45:21.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.593+0000 7f7123e00700 1 --2- 172.21.15.123:0/229867760 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f711006cc30 0x7f711006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:45:21.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.593+0000 7f7123e00700 1 -- 172.21.15.123:0/229867760 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f711c08bd20 con 0x7f712c102e00 2024-03-30T04:45:21.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.593+0000 7f712b400700 1 --2- 172.21.15.123:0/229867760 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f711006cc30 0x7f711006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:45:21.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.594+0000 7f712b400700 1 --2- 172.21.15.123:0/229867760 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f711006cc30 0x7f711006f0e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f712c10ecd0 tx=0x7f7118009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:45:21.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.597+0000 7f7123e00700 1 -- 172.21.15.123:0/229867760 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f711c0596f0 con 0x7f712c102e00 2024-03-30T04:45:21.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.807+0000 7f7132e00700 1 -- 172.21.15.123:0/229867760 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f712c109da0 con 0x7f711006cc30 2024-03-30T04:45:21.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.817+0000 7f7123e00700 1 -- 172.21.15.123:0/229867760 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f712c109da0 con 0x7f711006cc30 2024-03-30T04:45:21.816 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:45:21.816 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (14m) 7m ago 27m 15.0M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:45:21.816 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (11m) 7m ago 27m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T04:45:21.816 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (11m) 7m ago 26m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T04:45:21.816 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (14m) 7m ago 26m 74.0M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T04:45:21.816 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (7m) 7m ago 20m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T04:45:21.816 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (7m) 7m ago 20m 10.9M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T04:45:21.817 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (20m) 7m ago 20m 10.7M - 17.2.7-797-g5c024335 f77085f6aab7 0dc41326818c 2024-03-30T04:45:21.817 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (7m) 7m ago 20m 9747k - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T04:45:21.817 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (16m) 7m ago 28m 506M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:45:21.817 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (15m) 7m ago 25m 450M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:45:21.817 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (13m) 7m ago 28m 53.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T04:45:21.817 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (12m) 7m ago 25m 42.6M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T04:45:21.817 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (15m) 7m ago 27m 8052k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:45:21.817 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (15m) 7m ago 26m 7880k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:45:21.817 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (11m) 7m ago 25m 104M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T04:45:21.817 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (10m) 7m ago 24m 93.4M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T04:45:21.817 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (10m) 7m ago 23m 86.3M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T04:45:21.817 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (9m) 7m ago 23m 97.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T04:45:21.817 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (8m) 7m ago 22m 97.9M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T04:45:21.817 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (8m) 7m ago 22m 85.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T04:45:21.818 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (14m) 7m ago 26m 53.7M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:45:21.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.820+0000 7f7132e00700 1 -- 172.21.15.123:0/229867760 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f711006cc30 msgr2=0x7f711006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:21.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.820+0000 7f7132e00700 1 --2- 172.21.15.123:0/229867760 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f711006cc30 0x7f711006f0e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f712c10ecd0 tx=0x7f7118009e30 comp rx=0 tx=0).stop 2024-03-30T04:45:21.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.820+0000 7f7132e00700 1 -- 172.21.15.123:0/229867760 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f712c102e00 msgr2=0x7f712c113a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:21.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.820+0000 7f7132e00700 1 --2- 172.21.15.123:0/229867760 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f712c102e00 0x7f712c113a70 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f711c002a80 tx=0x7f711c0046b0 comp rx=0 tx=0).stop 2024-03-30T04:45:21.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.820+0000 7f7132e00700 1 -- 172.21.15.123:0/229867760 shutdown_connections 2024-03-30T04:45:21.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.820+0000 7f7132e00700 1 --2- 172.21.15.123:0/229867760 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f711006cc30 0x7f711006f0e0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:21.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.820+0000 7f7132e00700 1 --2- 172.21.15.123:0/229867760 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f712c102e00 0x7f712c113a70 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:21.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.820+0000 7f7132e00700 1 --2- 172.21.15.123:0/229867760 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f712c1037a0 0x7f712c113fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:21.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.820+0000 7f7132e00700 1 -- 172.21.15.123:0/229867760 >> 172.21.15.123:0/229867760 conn(0x7f712c0f89b0 msgr2=0x7f712c0f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:45:21.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.820+0000 7f7132e00700 1 -- 172.21.15.123:0/229867760 shutdown_connections 2024-03-30T04:45:21.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.820+0000 7f7132e00700 1 -- 172.21.15.123:0/229867760 wait complete. 2024-03-30T04:45:21.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.953+0000 7f316ae00700 1 -- 172.21.15.123:0/2890492052 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3164102e00 msgr2=0x7f31641031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:21.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.953+0000 7f316ae00700 1 --2- 172.21.15.123:0/2890492052 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3164102e00 0x7f31641031d0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f3154004940 tx=0x7f3154009e30 comp rx=0 tx=0).stop 2024-03-30T04:45:21.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.953+0000 7f316ae00700 1 -- 172.21.15.123:0/2890492052 shutdown_connections 2024-03-30T04:45:21.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.953+0000 7f316ae00700 1 --2- 172.21.15.123:0/2890492052 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31641037a0 0x7f31640fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:21.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.953+0000 7f316ae00700 1 --2- 172.21.15.123:0/2890492052 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3164102e00 0x7f31641031d0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:21.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.953+0000 7f316ae00700 1 -- 172.21.15.123:0/2890492052 >> 172.21.15.123:0/2890492052 conn(0x7f31640f89b0 msgr2=0x7f31640fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:45:21.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.953+0000 7f316ae00700 1 -- 172.21.15.123:0/2890492052 shutdown_connections 2024-03-30T04:45:21.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.953+0000 7f316ae00700 1 -- 172.21.15.123:0/2890492052 wait complete. 2024-03-30T04:45:21.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.954+0000 7f316ae00700 1 Processor -- start 2024-03-30T04:45:21.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.954+0000 7f316ae00700 1 -- start start 2024-03-30T04:45:21.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.954+0000 7f316ae00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3164102e00 0x7f3164115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:45:21.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.954+0000 7f316ae00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31641037a0 0x7f3164116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:45:21.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.954+0000 7f316ae00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3164116860 con 0x7f3164102e00 2024-03-30T04:45:21.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.954+0000 7f316ae00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f316410fc30 con 0x7f31641037a0 2024-03-30T04:45:21.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.955+0000 7f3163e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3164102e00 0x7f3164115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:45:21.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.955+0000 7f3163e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3164102e00 0x7f3164115cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43438/0 (socket says 172.21.15.123:43438) 2024-03-30T04:45:21.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.955+0000 7f3163e00700 1 -- 172.21.15.123:0/3597396382 learned_addr learned my addr 172.21.15.123:0/3597396382 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:45:21.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.955+0000 7f3163e00700 1 -- 172.21.15.123:0/3597396382 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31641037a0 msgr2=0x7f3164116210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:21.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.955+0000 7f3163e00700 1 --2- 172.21.15.123:0/3597396382 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31641037a0 0x7f3164116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:21.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.955+0000 7f3163e00700 1 -- 172.21.15.123:0/3597396382 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3154005040 con 0x7f3164102e00 2024-03-30T04:45:21.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.955+0000 7f3163e00700 1 --2- 172.21.15.123:0/3597396382 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3164102e00 0x7f3164115cd0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f3154009390 tx=0x7f3154009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:45:21.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.956+0000 7f315be00700 1 -- 172.21.15.123:0/3597396382 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f315400e040 con 0x7f3164102e00 2024-03-30T04:45:21.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.956+0000 7f315be00700 1 -- 172.21.15.123:0/3597396382 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3154013420 con 0x7f3164102e00 2024-03-30T04:45:21.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.956+0000 7f316ae00700 1 -- 172.21.15.123:0/3597396382 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f316410feb0 con 0x7f3164102e00 2024-03-30T04:45:21.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.956+0000 7f315be00700 1 -- 172.21.15.123:0/3597396382 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f315401c630 con 0x7f3164102e00 2024-03-30T04:45:21.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.956+0000 7f316ae00700 1 -- 172.21.15.123:0/3597396382 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3164110470 con 0x7f3164102e00 2024-03-30T04:45:21.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.957+0000 7f316ae00700 1 -- 172.21.15.123:0/3597396382 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31640fe550 con 0x7f3164102e00 2024-03-30T04:45:21.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.958+0000 7f315be00700 1 -- 172.21.15.123:0/3597396382 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f31540139e0 con 0x7f3164102e00 2024-03-30T04:45:21.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.958+0000 7f315be00700 1 --2- 172.21.15.123:0/3597396382 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f314806cc30 0x7f314806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:45:21.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.959+0000 7f315be00700 1 -- 172.21.15.123:0/3597396382 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f315408d240 con 0x7f3164102e00 2024-03-30T04:45:21.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.959+0000 7f3163400700 1 --2- 172.21.15.123:0/3597396382 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f314806cc30 0x7f314806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:45:21.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.959+0000 7f3163400700 1 --2- 172.21.15.123:0/3597396382 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f314806cc30 0x7f314806f0e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3164111570 tx=0x7f3150009ce0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:45:21.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:21.963+0000 7f315be00700 1 -- 172.21.15.123:0/3597396382 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f315405adc0 con 0x7f3164102e00 2024-03-30T04:45:22.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.250+0000 7f316ae00700 1 -- 172.21.15.123:0/3597396382 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f3164109dc0 con 0x7f3164102e00 2024-03-30T04:45:22.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.251+0000 7f315be00700 1 -- 172.21.15.123:0/3597396382 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f315405a950 con 0x7f3164102e00 2024-03-30T04:45:22.249 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:45:22.249 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:45:22.249 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:45:22.250 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:45:22.250 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:45:22.250 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:45:22.250 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:45:22.250 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:45:22.250 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T04:45:22.250 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:45:22.250 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:45:22.250 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:45:22.250 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 3 2024-03-30T04:45:22.250 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:45:22.250 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:45:22.250 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:45:22.250 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 13 2024-03-30T04:45:22.251 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:45:22.251 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:45:22.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.253+0000 7f316ae00700 1 -- 172.21.15.123:0/3597396382 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f314806cc30 msgr2=0x7f314806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:22.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.253+0000 7f316ae00700 1 --2- 172.21.15.123:0/3597396382 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f314806cc30 0x7f314806f0e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3164111570 tx=0x7f3150009ce0 comp rx=0 tx=0).stop 2024-03-30T04:45:22.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.253+0000 7f316ae00700 1 -- 172.21.15.123:0/3597396382 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3164102e00 msgr2=0x7f3164115cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:22.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.253+0000 7f316ae00700 1 --2- 172.21.15.123:0/3597396382 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3164102e00 0x7f3164115cd0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f3154009390 tx=0x7f3154009e30 comp rx=0 tx=0).stop 2024-03-30T04:45:22.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.253+0000 7f316ae00700 1 -- 172.21.15.123:0/3597396382 shutdown_connections 2024-03-30T04:45:22.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.253+0000 7f316ae00700 1 --2- 172.21.15.123:0/3597396382 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f314806cc30 0x7f314806f0e0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:22.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.253+0000 7f316ae00700 1 --2- 172.21.15.123:0/3597396382 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3164102e00 0x7f3164115cd0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:22.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.253+0000 7f316ae00700 1 --2- 172.21.15.123:0/3597396382 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31641037a0 0x7f3164116210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:22.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.253+0000 7f316ae00700 1 -- 172.21.15.123:0/3597396382 >> 172.21.15.123:0/3597396382 conn(0x7f31640f89b0 msgr2=0x7f31640f99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:45:22.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.253+0000 7f316ae00700 1 -- 172.21.15.123:0/3597396382 shutdown_connections 2024-03-30T04:45:22.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.253+0000 7f316ae00700 1 -- 172.21.15.123:0/3597396382 wait complete. 2024-03-30T04:45:22.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.388+0000 7fcbeba00700 1 -- 172.21.15.123:0/4062653543 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbe4070300 msgr2=0x7fcbe40706d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:22.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.388+0000 7fcbeba00700 1 --2- 172.21.15.123:0/4062653543 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbe4070300 0x7fcbe40706d0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fcbd8004970 tx=0x7fcbd8009e30 comp rx=0 tx=0).stop 2024-03-30T04:45:22.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.388+0000 7fcbeba00700 1 -- 172.21.15.123:0/4062653543 shutdown_connections 2024-03-30T04:45:22.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.388+0000 7fcbeba00700 1 --2- 172.21.15.123:0/4062653543 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcbe4070c10 0x7fcbe4109190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:22.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.388+0000 7fcbeba00700 1 --2- 172.21.15.123:0/4062653543 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbe4070300 0x7fcbe40706d0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:22.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.388+0000 7fcbeba00700 1 -- 172.21.15.123:0/4062653543 >> 172.21.15.123:0/4062653543 conn(0x7fcbe40f8970 msgr2=0x7fcbe40fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:45:22.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.388+0000 7fcbeba00700 1 -- 172.21.15.123:0/4062653543 shutdown_connections 2024-03-30T04:45:22.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.388+0000 7fcbeba00700 1 -- 172.21.15.123:0/4062653543 wait complete. 2024-03-30T04:45:22.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.389+0000 7fcbeba00700 1 Processor -- start 2024-03-30T04:45:22.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.389+0000 7fcbeba00700 1 -- start start 2024-03-30T04:45:22.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.390+0000 7fcbeba00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbe4070300 0x7fcbe4115c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:45:22.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.390+0000 7fcbeba00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcbe4070c10 0x7fcbe41161d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:45:22.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.390+0000 7fcbeba00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe4116820 con 0x7fcbe4070300 2024-03-30T04:45:22.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.390+0000 7fcbeba00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe410fc30 con 0x7fcbe4070c10 2024-03-30T04:45:22.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.390+0000 7fcbe9000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbe4070300 0x7fcbe4115c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:45:22.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.390+0000 7fcbe9000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbe4070300 0x7fcbe4115c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43458/0 (socket says 172.21.15.123:43458) 2024-03-30T04:45:22.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.390+0000 7fcbe9000700 1 -- 172.21.15.123:0/602295518 learned_addr learned my addr 172.21.15.123:0/602295518 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:45:22.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.390+0000 7fcbe3e00700 1 --2- 172.21.15.123:0/602295518 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcbe4070c10 0x7fcbe41161d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:45:22.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.390+0000 7fcbe9000700 1 -- 172.21.15.123:0/602295518 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcbe4070c10 msgr2=0x7fcbe41161d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:22.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.390+0000 7fcbe9000700 1 --2- 172.21.15.123:0/602295518 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcbe4070c10 0x7fcbe41161d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:22.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.390+0000 7fcbe9000700 1 -- 172.21.15.123:0/602295518 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbd8005040 con 0x7fcbe4070300 2024-03-30T04:45:22.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.391+0000 7fcbe9000700 1 --2- 172.21.15.123:0/602295518 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbe4070300 0x7fcbe4115c90 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fcbd8005010 tx=0x7fcbd8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:45:22.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.391+0000 7fcbe0c00700 1 -- 172.21.15.123:0/602295518 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcbd800e040 con 0x7fcbe4070300 2024-03-30T04:45:22.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.391+0000 7fcbe0c00700 1 -- 172.21.15.123:0/602295518 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcbd8004290 con 0x7fcbe4070300 2024-03-30T04:45:22.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.391+0000 7fcbe0c00700 1 -- 172.21.15.123:0/602295518 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcbd8013e40 con 0x7fcbe4070300 2024-03-30T04:45:22.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.391+0000 7fcbeba00700 1 -- 172.21.15.123:0/602295518 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbe410feb0 con 0x7fcbe4070300 2024-03-30T04:45:22.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.391+0000 7fcbeba00700 1 -- 172.21.15.123:0/602295518 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbe41103f0 con 0x7fcbe4070300 2024-03-30T04:45:22.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.392+0000 7fcbeba00700 1 -- 172.21.15.123:0/602295518 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbe4071a40 con 0x7fcbe4070300 2024-03-30T04:45:22.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.393+0000 7fcbe0c00700 1 -- 172.21.15.123:0/602295518 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fcbd8013620 con 0x7fcbe4070300 2024-03-30T04:45:22.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.394+0000 7fcbe0c00700 1 --2- 172.21.15.123:0/602295518 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fcbcc06cc10 0x7fcbcc06f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:45:22.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.394+0000 7fcbe0c00700 1 -- 172.21.15.123:0/602295518 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fcbd808d360 con 0x7fcbe4070300 2024-03-30T04:45:22.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.394+0000 7fcbe3e00700 1 --2- 172.21.15.123:0/602295518 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fcbcc06cc10 0x7fcbcc06f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:45:22.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.395+0000 7fcbe3e00700 1 --2- 172.21.15.123:0/602295518 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fcbcc06cc10 0x7fcbcc06f0c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fcbd0004600 tx=0x7fcbd000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:45:22.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.398+0000 7fcbe0c00700 1 -- 172.21.15.123:0/602295518 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fcbd8057320 con 0x7fcbe4070300 2024-03-30T04:45:22.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.643+0000 7fcbeba00700 1 -- 172.21.15.123:0/602295518 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fcbe404bed0 con 0x7fcbe4070300 2024-03-30T04:45:22.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.643+0000 7fcbe0c00700 1 -- 172.21.15.123:0/602295518 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1843 (secure 0 0 0) 0x7fcbd805a940 con 0x7fcbe4070300 2024-03-30T04:45:22.642 INFO:teuthology.orchestra.run.smithi123.stdout:e32 2024-03-30T04:45:22.642 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:45:22.643 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:45:22.643 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:45:22.643 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:45:22.643 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:45:22.643 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:45:22.643 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T04:45:22.643 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T04:45:22.643 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:45:22.644 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T04:45:22.644 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:45:22.644 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:45:22.644 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:45:22.644 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:45:22.644 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:45:22.645 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T04:45:22.645 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:45:22.645 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:45:22.645 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T04:45:22.645 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:45:22.645 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T04:45:22.645 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T04:45:22.646 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T04:45:22.646 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T04:45:22.646 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:45:22.646 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:45:22.646 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:45:22.646 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:45:22.647 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:45:22.647 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:45:22.647 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T04:45:22.647 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T04:45:22.647 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:45:22.647 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:45:22.648 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T04:45:22.648 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:45:22.648 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:45:22.648 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:45:22.648 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44315} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/1741352336,v1:172.21.15.135:6827/1741352336] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:45:22.648 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:45:22.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.646+0000 7fcbeba00700 1 -- 172.21.15.123:0/602295518 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fcbcc06cc10 msgr2=0x7fcbcc06f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:22.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.646+0000 7fcbeba00700 1 --2- 172.21.15.123:0/602295518 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fcbcc06cc10 0x7fcbcc06f0c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fcbd0004600 tx=0x7fcbd000b040 comp rx=0 tx=0).stop 2024-03-30T04:45:22.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.646+0000 7fcbeba00700 1 -- 172.21.15.123:0/602295518 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbe4070300 msgr2=0x7fcbe4115c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:22.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.646+0000 7fcbeba00700 1 --2- 172.21.15.123:0/602295518 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbe4070300 0x7fcbe4115c90 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fcbd8005010 tx=0x7fcbd8009e30 comp rx=0 tx=0).stop 2024-03-30T04:45:22.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.646+0000 7fcbeba00700 1 -- 172.21.15.123:0/602295518 shutdown_connections 2024-03-30T04:45:22.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.646+0000 7fcbeba00700 1 --2- 172.21.15.123:0/602295518 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fcbcc06cc10 0x7fcbcc06f0c0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:22.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.646+0000 7fcbeba00700 1 --2- 172.21.15.123:0/602295518 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbe4070300 0x7fcbe4115c90 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:22.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.646+0000 7fcbeba00700 1 --2- 172.21.15.123:0/602295518 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcbe4070c10 0x7fcbe41161d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:22.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.646+0000 7fcbeba00700 1 -- 172.21.15.123:0/602295518 >> 172.21.15.123:0/602295518 conn(0x7fcbe40f8970 msgr2=0x7fcbe40f9ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:45:22.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.646+0000 7fcbeba00700 1 -- 172.21.15.123:0/602295518 shutdown_connections 2024-03-30T04:45:22.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.646+0000 7fcbeba00700 1 -- 172.21.15.123:0/602295518 wait complete. 2024-03-30T04:45:22.651 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 32 2024-03-30T04:45:22.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.779+0000 7f458ea00700 1 -- 172.21.15.123:0/3855826131 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4588102e00 msgr2=0x7f45881031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:22.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.779+0000 7f458ea00700 1 --2- 172.21.15.123:0/3855826131 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4588102e00 0x7f45881031d0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f4578004940 tx=0x7f4578009e30 comp rx=0 tx=0).stop 2024-03-30T04:45:22.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.779+0000 7f458ea00700 1 -- 172.21.15.123:0/3855826131 shutdown_connections 2024-03-30T04:45:22.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.779+0000 7f458ea00700 1 --2- 172.21.15.123:0/3855826131 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f45881037a0 0x7f45880fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:22.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.779+0000 7f458ea00700 1 --2- 172.21.15.123:0/3855826131 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4588102e00 0x7f45881031d0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:22.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.779+0000 7f458ea00700 1 -- 172.21.15.123:0/3855826131 >> 172.21.15.123:0/3855826131 conn(0x7f45880f89b0 msgr2=0x7f45880fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:45:22.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.779+0000 7f458ea00700 1 -- 172.21.15.123:0/3855826131 shutdown_connections 2024-03-30T04:45:22.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.779+0000 7f458ea00700 1 -- 172.21.15.123:0/3855826131 wait complete. 2024-03-30T04:45:22.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.780+0000 7f458ea00700 1 Processor -- start 2024-03-30T04:45:22.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.780+0000 7f458ea00700 1 -- start start 2024-03-30T04:45:22.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.780+0000 7f458ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4588102e00 0x7f4588113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:45:22.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.780+0000 7f458ea00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f45881037a0 0x7f4588114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:45:22.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.780+0000 7f458ea00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45881146b0 con 0x7f4588102e00 2024-03-30T04:45:22.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.780+0000 7f458ea00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f458810daa0 con 0x7f45881037a0 2024-03-30T04:45:22.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.781+0000 7f4587e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4588102e00 0x7f4588113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:45:22.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.781+0000 7f4587e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4588102e00 0x7f4588113b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43478/0 (socket says 172.21.15.123:43478) 2024-03-30T04:45:22.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.781+0000 7f4587e00700 1 -- 172.21.15.123:0/2190137204 learned_addr learned my addr 172.21.15.123:0/2190137204 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:45:22.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.781+0000 7f4587400700 1 --2- 172.21.15.123:0/2190137204 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f45881037a0 0x7f4588114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:45:22.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.781+0000 7f4587e00700 1 -- 172.21.15.123:0/2190137204 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f45881037a0 msgr2=0x7f4588114060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:22.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.781+0000 7f4587e00700 1 --2- 172.21.15.123:0/2190137204 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f45881037a0 0x7f4588114060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:22.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.781+0000 7f4587e00700 1 -- 172.21.15.123:0/2190137204 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4578005040 con 0x7f4588102e00 2024-03-30T04:45:22.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.781+0000 7f4587e00700 1 --2- 172.21.15.123:0/2190137204 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4588102e00 0x7f4588113b20 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f4578009390 tx=0x7f4578009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:45:22.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.782+0000 7f457fe00700 1 -- 172.21.15.123:0/2190137204 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f457800e040 con 0x7f4588102e00 2024-03-30T04:45:22.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.782+0000 7f457fe00700 1 -- 172.21.15.123:0/2190137204 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4578013420 con 0x7f4588102e00 2024-03-30T04:45:22.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.782+0000 7f458ea00700 1 -- 172.21.15.123:0/2190137204 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f458810dd20 con 0x7f4588102e00 2024-03-30T04:45:22.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.782+0000 7f457fe00700 1 -- 172.21.15.123:0/2190137204 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f457801c630 con 0x7f4588102e00 2024-03-30T04:45:22.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.782+0000 7f458ea00700 1 -- 172.21.15.123:0/2190137204 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f458810e2e0 con 0x7f4588102e00 2024-03-30T04:45:22.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.783+0000 7f458ea00700 1 -- 172.21.15.123:0/2190137204 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45880fe550 con 0x7f4588102e00 2024-03-30T04:45:22.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.784+0000 7f457fe00700 1 -- 172.21.15.123:0/2190137204 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f45780139e0 con 0x7f4588102e00 2024-03-30T04:45:22.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.784+0000 7f457fe00700 1 --2- 172.21.15.123:0/2190137204 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f456c06cc30 0x7f456c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:45:22.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.785+0000 7f457fe00700 1 -- 172.21.15.123:0/2190137204 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f457808d3d0 con 0x7f4588102e00 2024-03-30T04:45:22.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.785+0000 7f4587400700 1 --2- 172.21.15.123:0/2190137204 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f456c06cc30 0x7f456c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:45:22.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.785+0000 7f4587400700 1 --2- 172.21.15.123:0/2190137204 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f456c06cc30 0x7f456c06f0e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f458810ec60 tx=0x7f457400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:45:22.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:22.789+0000 7f457fe00700 1 -- 172.21.15.123:0/2190137204 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f457805ae20 con 0x7f4588102e00 2024-03-30T04:45:22.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:45:22 smithi123 ceph-mon[139369]: from='client.34822 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:45:22.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:45:22 smithi123 ceph-mon[139369]: from='client.34826 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:45:22.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:45:22 smithi123 ceph-mon[139369]: pgmap v425: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:22.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:45:22 smithi123 ceph-mon[139369]: from='client.34830 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:45:22.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:45:22 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3597396382' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:45:23.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.009+0000 7f458ea00700 1 -- 172.21.15.123:0/2190137204 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4588109e30 con 0x7f456c06cc30 2024-03-30T04:45:23.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.011+0000 7f457fe00700 1 -- 172.21.15.123:0/2190137204 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f4588109e30 con 0x7f456c06cc30 2024-03-30T04:45:23.009 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:45:23.009 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:45:23.009 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:45:23.009 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T04:45:23.009 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:45:23.009 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T04:45:23.009 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T04:45:23.010 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T04:45:23.010 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T04:45:23.010 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:45:23.010 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "15/21 daemons upgraded", 2024-03-30T04:45:23.010 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T04:45:23.010 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:45:23.010 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:45:23.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.013+0000 7f458ea00700 1 -- 172.21.15.123:0/2190137204 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f456c06cc30 msgr2=0x7f456c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:23.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.013+0000 7f458ea00700 1 --2- 172.21.15.123:0/2190137204 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f456c06cc30 0x7f456c06f0e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f458810ec60 tx=0x7f457400b040 comp rx=0 tx=0).stop 2024-03-30T04:45:23.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.013+0000 7f458ea00700 1 -- 172.21.15.123:0/2190137204 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4588102e00 msgr2=0x7f4588113b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:23.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.013+0000 7f458ea00700 1 --2- 172.21.15.123:0/2190137204 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4588102e00 0x7f4588113b20 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f4578009390 tx=0x7f4578009e30 comp rx=0 tx=0).stop 2024-03-30T04:45:23.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.013+0000 7f458ea00700 1 -- 172.21.15.123:0/2190137204 shutdown_connections 2024-03-30T04:45:23.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.013+0000 7f458ea00700 1 --2- 172.21.15.123:0/2190137204 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f456c06cc30 0x7f456c06f0e0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:23.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.013+0000 7f458ea00700 1 --2- 172.21.15.123:0/2190137204 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4588102e00 0x7f4588113b20 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:23.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.013+0000 7f458ea00700 1 --2- 172.21.15.123:0/2190137204 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f45881037a0 0x7f4588114060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:23.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.013+0000 7f458ea00700 1 -- 172.21.15.123:0/2190137204 >> 172.21.15.123:0/2190137204 conn(0x7f45880f89b0 msgr2=0x7f45880f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:45:23.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.013+0000 7f458ea00700 1 -- 172.21.15.123:0/2190137204 shutdown_connections 2024-03-30T04:45:23.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.013+0000 7f458ea00700 1 -- 172.21.15.123:0/2190137204 wait complete. 2024-03-30T04:45:23.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:45:22 smithi135 ceph-mon[106011]: from='client.34822 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:45:23.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:45:22 smithi135 ceph-mon[106011]: from='client.34826 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:45:23.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:45:22 smithi135 ceph-mon[106011]: pgmap v425: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:23.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:45:22 smithi135 ceph-mon[106011]: from='client.34830 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:45:23.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:45:22 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3597396382' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:45:23.146 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.148+0000 7f42c1200700 1 -- 172.21.15.123:0/2363721331 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42bc102e00 msgr2=0x7f42bc1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:23.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.148+0000 7f42c1200700 1 --2- 172.21.15.123:0/2363721331 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42bc102e00 0x7f42bc1031d0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f42ac004940 tx=0x7f42ac009e30 comp rx=0 tx=0).stop 2024-03-30T04:45:23.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.148+0000 7f42c1200700 1 -- 172.21.15.123:0/2363721331 shutdown_connections 2024-03-30T04:45:23.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.148+0000 7f42c1200700 1 --2- 172.21.15.123:0/2363721331 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f42bc1037a0 0x7f42bc0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:23.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.148+0000 7f42c1200700 1 --2- 172.21.15.123:0/2363721331 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42bc102e00 0x7f42bc1031d0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:23.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.148+0000 7f42c1200700 1 -- 172.21.15.123:0/2363721331 >> 172.21.15.123:0/2363721331 conn(0x7f42bc0f89b0 msgr2=0x7f42bc0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:45:23.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.148+0000 7f42c1200700 1 -- 172.21.15.123:0/2363721331 shutdown_connections 2024-03-30T04:45:23.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.148+0000 7f42c1200700 1 -- 172.21.15.123:0/2363721331 wait complete. 2024-03-30T04:45:23.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.149+0000 7f42c1200700 1 Processor -- start 2024-03-30T04:45:23.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.149+0000 7f42c1200700 1 -- start start 2024-03-30T04:45:23.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.149+0000 7f42c1200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42bc102e00 0x7f42bc115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:45:23.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.149+0000 7f42c1200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f42bc1037a0 0x7f42bc116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:45:23.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.149+0000 7f42c1200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42bc116880 con 0x7f42bc102e00 2024-03-30T04:45:23.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.149+0000 7f42c1200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42bc10fc50 con 0x7f42bc1037a0 2024-03-30T04:45:23.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.150+0000 7f42b9e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42bc102e00 0x7f42bc115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:45:23.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.150+0000 7f42b9e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42bc102e00 0x7f42bc115cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43508/0 (socket says 172.21.15.123:43508) 2024-03-30T04:45:23.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.150+0000 7f42b9e00700 1 -- 172.21.15.123:0/3340299045 learned_addr learned my addr 172.21.15.123:0/3340299045 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:45:23.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.150+0000 7f42b9e00700 1 -- 172.21.15.123:0/3340299045 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f42bc1037a0 msgr2=0x7f42bc116230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:23.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.150+0000 7f42b9e00700 1 --2- 172.21.15.123:0/3340299045 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f42bc1037a0 0x7f42bc116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:23.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.150+0000 7f42b9e00700 1 -- 172.21.15.123:0/3340299045 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42ac005040 con 0x7f42bc102e00 2024-03-30T04:45:23.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.150+0000 7f42b9e00700 1 --2- 172.21.15.123:0/3340299045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42bc102e00 0x7f42bc115cf0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f42ac009390 tx=0x7f42ac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:45:23.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.150+0000 7f42b2000700 1 -- 172.21.15.123:0/3340299045 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f42ac00e040 con 0x7f42bc102e00 2024-03-30T04:45:23.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.151+0000 7f42b2000700 1 -- 172.21.15.123:0/3340299045 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f42ac013420 con 0x7f42bc102e00 2024-03-30T04:45:23.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.151+0000 7f42c1200700 1 -- 172.21.15.123:0/3340299045 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42bc10fed0 con 0x7f42bc102e00 2024-03-30T04:45:23.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.151+0000 7f42b2000700 1 -- 172.21.15.123:0/3340299045 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f42ac01c630 con 0x7f42bc102e00 2024-03-30T04:45:23.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.151+0000 7f42c1200700 1 -- 172.21.15.123:0/3340299045 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42bc110490 con 0x7f42bc102e00 2024-03-30T04:45:23.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.152+0000 7f42c1200700 1 -- 172.21.15.123:0/3340299045 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42bc0fe550 con 0x7f42bc102e00 2024-03-30T04:45:23.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.153+0000 7f42b2000700 1 -- 172.21.15.123:0/3340299045 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f42ac0139e0 con 0x7f42bc102e00 2024-03-30T04:45:23.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.154+0000 7f42b2000700 1 --2- 172.21.15.123:0/3340299045 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f42a006cc30 0x7f42a006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:45:23.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.154+0000 7f42b2000700 1 -- 172.21.15.123:0/3340299045 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f42ac08d250 con 0x7f42bc102e00 2024-03-30T04:45:23.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.154+0000 7f42b9400700 1 --2- 172.21.15.123:0/3340299045 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f42a006cc30 0x7f42a006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:45:23.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.155+0000 7f42b9400700 1 --2- 172.21.15.123:0/3340299045 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f42a006cc30 0x7f42a006f0e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f42bc111590 tx=0x7f42a8009ce0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:45:23.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.158+0000 7f42b2000700 1 -- 172.21.15.123:0/3340299045 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f42ac05add0 con 0x7f42bc102e00 2024-03-30T04:45:23.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.443+0000 7f42c1200700 1 -- 172.21.15.123:0/3340299045 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f42bc05f5f0 con 0x7f42bc102e00 2024-03-30T04:45:23.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.443+0000 7f42b2000700 1 -- 172.21.15.123:0/3340299045 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f42ac05a960 con 0x7f42bc102e00 2024-03-30T04:45:23.442 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T04:45:23.442 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T04:45:23.442 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T04:45:23.442 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T04:45:23.442 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T04:45:23.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.445+0000 7f42c1200700 1 -- 172.21.15.123:0/3340299045 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f42a006cc30 msgr2=0x7f42a006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:23.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.445+0000 7f42c1200700 1 --2- 172.21.15.123:0/3340299045 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f42a006cc30 0x7f42a006f0e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f42bc111590 tx=0x7f42a8009ce0 comp rx=0 tx=0).stop 2024-03-30T04:45:23.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.445+0000 7f42c1200700 1 -- 172.21.15.123:0/3340299045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42bc102e00 msgr2=0x7f42bc115cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:23.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.445+0000 7f42c1200700 1 --2- 172.21.15.123:0/3340299045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42bc102e00 0x7f42bc115cf0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f42ac009390 tx=0x7f42ac009e30 comp rx=0 tx=0).stop 2024-03-30T04:45:23.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.446+0000 7f42c1200700 1 -- 172.21.15.123:0/3340299045 shutdown_connections 2024-03-30T04:45:23.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.446+0000 7f42c1200700 1 --2- 172.21.15.123:0/3340299045 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f42a006cc30 0x7f42a006f0e0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:23.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.446+0000 7f42c1200700 1 --2- 172.21.15.123:0/3340299045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42bc102e00 0x7f42bc115cf0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:23.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.446+0000 7f42c1200700 1 --2- 172.21.15.123:0/3340299045 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f42bc1037a0 0x7f42bc116230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:23.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.446+0000 7f42c1200700 1 -- 172.21.15.123:0/3340299045 >> 172.21.15.123:0/3340299045 conn(0x7f42bc0f89b0 msgr2=0x7f42bc0f9980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:45:23.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.446+0000 7f42c1200700 1 -- 172.21.15.123:0/3340299045 shutdown_connections 2024-03-30T04:45:23.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:23.446+0000 7f42c1200700 1 -- 172.21.15.123:0/3340299045 wait complete. 2024-03-30T04:45:23.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:45:23 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/602295518' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:45:23.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:45:23 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3340299045' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:45:24.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:45:23 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/602295518' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:45:24.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:45:23 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3340299045' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:45:24.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:45:24 smithi123 ceph-mon[139369]: from='client.34842 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:45:24.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:45:24 smithi123 ceph-mon[139369]: pgmap v426: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:25.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:45:24 smithi135 ceph-mon[106011]: from='client.34842 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:45:25.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:45:24 smithi135 ceph-mon[106011]: pgmap v426: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:26.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:45:26 smithi123 ceph-mon[139369]: pgmap v427: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:27.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:45:26 smithi135 ceph-mon[106011]: pgmap v427: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:28.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:45:28 smithi123 ceph-mon[139369]: pgmap v428: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:29.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:45:28 smithi135 ceph-mon[106011]: pgmap v428: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:29.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:45:29 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:45:30.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:45:29 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:45:30.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:45:30 smithi123 ceph-mon[139369]: pgmap v429: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:31.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:45:30 smithi135 ceph-mon[106011]: pgmap v429: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:32.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:45:32 smithi123 ceph-mon[139369]: pgmap v430: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:33.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:45:32 smithi135 ceph-mon[106011]: pgmap v430: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:34.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:45:34 smithi123 ceph-mon[139369]: pgmap v431: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:35.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:45:34 smithi135 ceph-mon[106011]: pgmap v431: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:36.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:45:36 smithi123 ceph-mon[139369]: pgmap v432: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:37.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:45:36 smithi135 ceph-mon[106011]: pgmap v432: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:38.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:45:38 smithi123 ceph-mon[139369]: pgmap v433: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:39.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:45:38 smithi135 ceph-mon[106011]: pgmap v433: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:40.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:45:40 smithi135 ceph-mon[106011]: pgmap v434: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:40.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:45:40 smithi123 ceph-mon[139369]: pgmap v434: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:42.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:45:42 smithi123 ceph-mon[139369]: pgmap v435: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:43.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:45:42 smithi135 ceph-mon[106011]: pgmap v435: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:44.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:45:44 smithi123 ceph-mon[139369]: pgmap v436: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:44.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:45:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:45:45.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:45:44 smithi135 ceph-mon[106011]: pgmap v436: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:45.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:45:44 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:45:46.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:45:46 smithi123 ceph-mon[139369]: pgmap v437: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:47.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:45:46 smithi135 ceph-mon[106011]: pgmap v437: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:48.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:45:48 smithi123 ceph-mon[139369]: pgmap v438: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:49.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:45:48 smithi135 ceph-mon[106011]: pgmap v438: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:50.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:45:50 smithi135 ceph-mon[106011]: pgmap v439: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:50.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:45:50 smithi123 ceph-mon[139369]: pgmap v439: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:52.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:45:52 smithi123 ceph-mon[139369]: pgmap v440: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:53.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:45:52 smithi135 ceph-mon[106011]: pgmap v440: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:53.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.598+0000 7f4da1e00700 1 -- 172.21.15.123:0/195937419 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d9c102e00 msgr2=0x7f4d9c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:53.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.598+0000 7f4da1e00700 1 --2- 172.21.15.123:0/195937419 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d9c102e00 0x7f4d9c1031d0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f4d8c004940 tx=0x7f4d8c009e30 comp rx=0 tx=0).stop 2024-03-30T04:45:53.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.598+0000 7f4da1e00700 1 -- 172.21.15.123:0/195937419 shutdown_connections 2024-03-30T04:45:53.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.598+0000 7f4da1e00700 1 --2- 172.21.15.123:0/195937419 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4d9c1037a0 0x7f4d9c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:53.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.598+0000 7f4da1e00700 1 --2- 172.21.15.123:0/195937419 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d9c102e00 0x7f4d9c1031d0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:53.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.598+0000 7f4da1e00700 1 -- 172.21.15.123:0/195937419 >> 172.21.15.123:0/195937419 conn(0x7f4d9c0f89b0 msgr2=0x7f4d9c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:45:53.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.598+0000 7f4da1e00700 1 -- 172.21.15.123:0/195937419 shutdown_connections 2024-03-30T04:45:53.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.598+0000 7f4da1e00700 1 -- 172.21.15.123:0/195937419 wait complete. 2024-03-30T04:45:53.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.599+0000 7f4da1e00700 1 Processor -- start 2024-03-30T04:45:53.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.599+0000 7f4da1e00700 1 -- start start 2024-03-30T04:45:53.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.600+0000 7f4da1e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4d9c102e00 0x7f4d9c115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:45:53.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.600+0000 7f4da1e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d9c1037a0 0x7f4d9c116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:45:53.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.600+0000 7f4da1e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d9c116860 con 0x7f4d9c1037a0 2024-03-30T04:45:53.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.600+0000 7f4da1e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d9c10fc30 con 0x7f4d9c102e00 2024-03-30T04:45:53.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.600+0000 7f4d9a000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d9c1037a0 0x7f4d9c116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:45:53.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.600+0000 7f4d9a000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d9c1037a0 0x7f4d9c116210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56456/0 (socket says 172.21.15.123:56456) 2024-03-30T04:45:53.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.600+0000 7f4d9a000700 1 -- 172.21.15.123:0/2319124063 learned_addr learned my addr 172.21.15.123:0/2319124063 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:45:53.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.600+0000 7f4d9aa00700 1 --2- 172.21.15.123:0/2319124063 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4d9c102e00 0x7f4d9c115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:45:53.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.600+0000 7f4d9a000700 1 -- 172.21.15.123:0/2319124063 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4d9c102e00 msgr2=0x7f4d9c115cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:53.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.600+0000 7f4d9a000700 1 --2- 172.21.15.123:0/2319124063 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4d9c102e00 0x7f4d9c115cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:53.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.600+0000 7f4d9a000700 1 -- 172.21.15.123:0/2319124063 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d8c005040 con 0x7f4d9c1037a0 2024-03-30T04:45:53.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.601+0000 7f4d9a000700 1 --2- 172.21.15.123:0/2319124063 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d9c1037a0 0x7f4d9c116210 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f4d88002a50 tx=0x7f4d88004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:45:53.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.601+0000 7f4d92a00700 1 -- 172.21.15.123:0/2319124063 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d88007cc0 con 0x7f4d9c1037a0 2024-03-30T04:45:53.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.601+0000 7f4d92a00700 1 -- 172.21.15.123:0/2319124063 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4d8800f040 con 0x7f4d9c1037a0 2024-03-30T04:45:53.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.601+0000 7f4d92a00700 1 -- 172.21.15.123:0/2319124063 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d8800d920 con 0x7f4d9c1037a0 2024-03-30T04:45:53.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.601+0000 7f4da1e00700 1 -- 172.21.15.123:0/2319124063 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d9c10ff10 con 0x7f4d9c1037a0 2024-03-30T04:45:53.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.601+0000 7f4da1e00700 1 -- 172.21.15.123:0/2319124063 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d9c110480 con 0x7f4d9c1037a0 2024-03-30T04:45:53.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.602+0000 7f4da1e00700 1 -- 172.21.15.123:0/2319124063 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d9c0fe550 con 0x7f4d9c1037a0 2024-03-30T04:45:53.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.603+0000 7f4d92a00700 1 -- 172.21.15.123:0/2319124063 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f4d88009680 con 0x7f4d9c1037a0 2024-03-30T04:45:53.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.604+0000 7f4d92a00700 1 --2- 172.21.15.123:0/2319124063 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4d8006cc30 0x7f4d8006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:45:53.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.604+0000 7f4d92a00700 1 -- 172.21.15.123:0/2319124063 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f4d8808bdb0 con 0x7f4d9c1037a0 2024-03-30T04:45:53.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.604+0000 7f4d9aa00700 1 --2- 172.21.15.123:0/2319124063 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4d8006cc30 0x7f4d8006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:45:53.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.605+0000 7f4d9aa00700 1 --2- 172.21.15.123:0/2319124063 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4d8006cc30 0x7f4d8006f0e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4d8c009fd0 tx=0x7f4d8c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:45:53.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.609+0000 7f4d92a00700 1 -- 172.21.15.123:0/2319124063 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4d88059780 con 0x7f4d9c1037a0 2024-03-30T04:45:53.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.829+0000 7f4da1e00700 1 -- 172.21.15.123:0/2319124063 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4d9c109e30 con 0x7f4d8006cc30 2024-03-30T04:45:53.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.831+0000 7f4d92a00700 1 -- 172.21.15.123:0/2319124063 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f4d9c109e30 con 0x7f4d8006cc30 2024-03-30T04:45:53.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.833+0000 7f4da1e00700 1 -- 172.21.15.123:0/2319124063 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4d8006cc30 msgr2=0x7f4d8006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:53.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.833+0000 7f4da1e00700 1 --2- 172.21.15.123:0/2319124063 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4d8006cc30 0x7f4d8006f0e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4d8c009fd0 tx=0x7f4d8c009e30 comp rx=0 tx=0).stop 2024-03-30T04:45:53.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.833+0000 7f4da1e00700 1 -- 172.21.15.123:0/2319124063 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d9c1037a0 msgr2=0x7f4d9c116210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:53.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.833+0000 7f4da1e00700 1 --2- 172.21.15.123:0/2319124063 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d9c1037a0 0x7f4d9c116210 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f4d88002a50 tx=0x7f4d88004600 comp rx=0 tx=0).stop 2024-03-30T04:45:53.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.833+0000 7f4da1e00700 1 -- 172.21.15.123:0/2319124063 shutdown_connections 2024-03-30T04:45:53.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.833+0000 7f4da1e00700 1 --2- 172.21.15.123:0/2319124063 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4d8006cc30 0x7f4d8006f0e0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:53.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.833+0000 7f4da1e00700 1 --2- 172.21.15.123:0/2319124063 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4d9c102e00 0x7f4d9c115cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:53.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.833+0000 7f4da1e00700 1 --2- 172.21.15.123:0/2319124063 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d9c1037a0 0x7f4d9c116210 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:53.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.833+0000 7f4da1e00700 1 -- 172.21.15.123:0/2319124063 >> 172.21.15.123:0/2319124063 conn(0x7f4d9c0f89b0 msgr2=0x7f4d9c0f99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:45:53.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.833+0000 7f4da1e00700 1 -- 172.21.15.123:0/2319124063 shutdown_connections 2024-03-30T04:45:53.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.833+0000 7f4da1e00700 1 -- 172.21.15.123:0/2319124063 wait complete. 2024-03-30T04:45:53.846 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:45:53.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.968+0000 7fc597000700 1 -- 172.21.15.123:0/1205523905 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc590102e20 msgr2=0x7fc5901031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:53.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.968+0000 7fc597000700 1 --2- 172.21.15.123:0/1205523905 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc590102e20 0x7fc5901031f0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fc580004940 tx=0x7fc580009e30 comp rx=0 tx=0).stop 2024-03-30T04:45:53.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.968+0000 7fc597000700 1 -- 172.21.15.123:0/1205523905 shutdown_connections 2024-03-30T04:45:53.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.968+0000 7fc597000700 1 --2- 172.21.15.123:0/1205523905 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc5901037c0 0x7fc5900fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:53.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.968+0000 7fc597000700 1 --2- 172.21.15.123:0/1205523905 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc590102e20 0x7fc5901031f0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:53.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.968+0000 7fc597000700 1 -- 172.21.15.123:0/1205523905 >> 172.21.15.123:0/1205523905 conn(0x7fc5900f89d0 msgr2=0x7fc5900fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:45:53.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.968+0000 7fc597000700 1 -- 172.21.15.123:0/1205523905 shutdown_connections 2024-03-30T04:45:53.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.968+0000 7fc597000700 1 -- 172.21.15.123:0/1205523905 wait complete. 2024-03-30T04:45:53.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.969+0000 7fc597000700 1 Processor -- start 2024-03-30T04:45:53.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.969+0000 7fc597000700 1 -- start start 2024-03-30T04:45:53.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.969+0000 7fc597000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc590102e20 0x7fc590115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:45:53.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.969+0000 7fc597000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5901037c0 0x7fc590116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:45:53.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.969+0000 7fc597000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc590116880 con 0x7fc5901037c0 2024-03-30T04:45:53.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.969+0000 7fc597000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc59010fc50 con 0x7fc590102e20 2024-03-30T04:45:53.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.970+0000 7fc58f400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5901037c0 0x7fc590116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:45:53.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.970+0000 7fc58f400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5901037c0 0x7fc590116230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58344/0 (socket says 172.21.15.123:58344) 2024-03-30T04:45:53.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.970+0000 7fc58f400700 1 -- 172.21.15.123:0/1094511803 learned_addr learned my addr 172.21.15.123:0/1094511803 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:45:53.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.970+0000 7fc58fe00700 1 --2- 172.21.15.123:0/1094511803 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc590102e20 0x7fc590115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:45:53.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.970+0000 7fc58f400700 1 -- 172.21.15.123:0/1094511803 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc590102e20 msgr2=0x7fc590115cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:53.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.970+0000 7fc58f400700 1 --2- 172.21.15.123:0/1094511803 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc590102e20 0x7fc590115cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:53.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.970+0000 7fc58f400700 1 -- 172.21.15.123:0/1094511803 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc580005040 con 0x7fc5901037c0 2024-03-30T04:45:53.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.970+0000 7fc58f400700 1 --2- 172.21.15.123:0/1094511803 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5901037c0 0x7fc590116230 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fc57c002a50 tx=0x7fc57c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:45:53.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.970+0000 7fc587e00700 1 -- 172.21.15.123:0/1094511803 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc57c007cc0 con 0x7fc5901037c0 2024-03-30T04:45:53.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.971+0000 7fc587e00700 1 -- 172.21.15.123:0/1094511803 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc57c00f040 con 0x7fc5901037c0 2024-03-30T04:45:53.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.971+0000 7fc597000700 1 -- 172.21.15.123:0/1094511803 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc59010ff30 con 0x7fc5901037c0 2024-03-30T04:45:53.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.971+0000 7fc587e00700 1 -- 172.21.15.123:0/1094511803 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc57c00d9a0 con 0x7fc5901037c0 2024-03-30T04:45:53.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.971+0000 7fc597000700 1 -- 172.21.15.123:0/1094511803 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5901104f0 con 0x7fc5901037c0 2024-03-30T04:45:53.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.972+0000 7fc597000700 1 -- 172.21.15.123:0/1094511803 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5900fe570 con 0x7fc5901037c0 2024-03-30T04:45:53.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.973+0000 7fc587e00700 1 -- 172.21.15.123:0/1094511803 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fc57c00dc60 con 0x7fc5901037c0 2024-03-30T04:45:53.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.974+0000 7fc587e00700 1 --2- 172.21.15.123:0/1094511803 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc57406cc30 0x7fc57406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:45:53.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.974+0000 7fc587e00700 1 -- 172.21.15.123:0/1094511803 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc57c08bef0 con 0x7fc5901037c0 2024-03-30T04:45:53.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.974+0000 7fc58fe00700 1 --2- 172.21.15.123:0/1094511803 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc57406cc30 0x7fc57406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:45:53.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.975+0000 7fc58fe00700 1 --2- 172.21.15.123:0/1094511803 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc57406cc30 0x7fc57406f0e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc58000b020 tx=0x7fc580009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:45:53.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:53.979+0000 7fc587e00700 1 -- 172.21.15.123:0/1094511803 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc57c0598c0 con 0x7fc5901037c0 2024-03-30T04:45:54.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.201+0000 7fc597000700 1 -- 172.21.15.123:0/1094511803 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc590109e50 con 0x7fc57406cc30 2024-03-30T04:45:54.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.203+0000 7fc587e00700 1 -- 172.21.15.123:0/1094511803 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc590109e50 con 0x7fc57406cc30 2024-03-30T04:45:54.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.205+0000 7fc597000700 1 -- 172.21.15.123:0/1094511803 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc57406cc30 msgr2=0x7fc57406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:54.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.205+0000 7fc597000700 1 --2- 172.21.15.123:0/1094511803 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc57406cc30 0x7fc57406f0e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc58000b020 tx=0x7fc580009e30 comp rx=0 tx=0).stop 2024-03-30T04:45:54.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.205+0000 7fc597000700 1 -- 172.21.15.123:0/1094511803 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5901037c0 msgr2=0x7fc590116230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:54.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.205+0000 7fc597000700 1 --2- 172.21.15.123:0/1094511803 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5901037c0 0x7fc590116230 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fc57c002a50 tx=0x7fc57c004600 comp rx=0 tx=0).stop 2024-03-30T04:45:54.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.205+0000 7fc597000700 1 -- 172.21.15.123:0/1094511803 shutdown_connections 2024-03-30T04:45:54.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.205+0000 7fc597000700 1 --2- 172.21.15.123:0/1094511803 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc57406cc30 0x7fc57406f0e0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:54.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.205+0000 7fc597000700 1 --2- 172.21.15.123:0/1094511803 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc590102e20 0x7fc590115cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:54.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.205+0000 7fc597000700 1 --2- 172.21.15.123:0/1094511803 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5901037c0 0x7fc590116230 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:54.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.205+0000 7fc597000700 1 -- 172.21.15.123:0/1094511803 >> 172.21.15.123:0/1094511803 conn(0x7fc5900f89d0 msgr2=0x7fc5900f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:45:54.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.205+0000 7fc597000700 1 -- 172.21.15.123:0/1094511803 shutdown_connections 2024-03-30T04:45:54.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.205+0000 7fc597000700 1 -- 172.21.15.123:0/1094511803 wait complete. 2024-03-30T04:45:54.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.339+0000 7fa883200700 1 -- 172.21.15.123:0/2312769979 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa884070c70 msgr2=0x7fa884108f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:54.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.339+0000 7fa883200700 1 --2- 172.21.15.123:0/2312769979 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa884070c70 0x7fa884108f50 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fa870004970 tx=0x7fa870009e30 comp rx=0 tx=0).stop 2024-03-30T04:45:54.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.339+0000 7fa883200700 1 -- 172.21.15.123:0/2312769979 shutdown_connections 2024-03-30T04:45:54.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.339+0000 7fa883200700 1 --2- 172.21.15.123:0/2312769979 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa884070c70 0x7fa884108f50 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:54.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.339+0000 7fa883200700 1 --2- 172.21.15.123:0/2312769979 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa884070360 0x7fa884070730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:54.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.339+0000 7fa883200700 1 -- 172.21.15.123:0/2312769979 >> 172.21.15.123:0/2312769979 conn(0x7fa8840f8830 msgr2=0x7fa8840fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:45:54.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.339+0000 7fa883200700 1 -- 172.21.15.123:0/2312769979 shutdown_connections 2024-03-30T04:45:54.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.339+0000 7fa883200700 1 -- 172.21.15.123:0/2312769979 wait complete. 2024-03-30T04:45:54.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.340+0000 7fa883200700 1 Processor -- start 2024-03-30T04:45:54.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.340+0000 7fa883200700 1 -- start start 2024-03-30T04:45:54.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.340+0000 7fa883200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa884070360 0x7fa884115ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:45:54.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.340+0000 7fa883200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa884070c70 0x7fa884116000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:45:54.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.340+0000 7fa883200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa884116650 con 0x7fa884070c70 2024-03-30T04:45:54.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.340+0000 7fa883200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa88410fa40 con 0x7fa884070360 2024-03-30T04:45:54.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.340+0000 7fa881400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa884070c70 0x7fa884116000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:45:54.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.340+0000 7fa881400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa884070c70 0x7fa884116000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58362/0 (socket says 172.21.15.123:58362) 2024-03-30T04:45:54.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.340+0000 7fa881400700 1 -- 172.21.15.123:0/1265387585 learned_addr learned my addr 172.21.15.123:0/1265387585 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:45:54.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.341+0000 7fa881e00700 1 --2- 172.21.15.123:0/1265387585 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa884070360 0x7fa884115ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:45:54.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.341+0000 7fa881400700 1 -- 172.21.15.123:0/1265387585 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa884070360 msgr2=0x7fa884115ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:54.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.341+0000 7fa881400700 1 --2- 172.21.15.123:0/1265387585 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa884070360 0x7fa884115ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:54.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.341+0000 7fa881400700 1 -- 172.21.15.123:0/1265387585 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa870005040 con 0x7fa884070c70 2024-03-30T04:45:54.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.341+0000 7fa881400700 1 --2- 172.21.15.123:0/1265387585 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa884070c70 0x7fa884116000 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fa870009360 tx=0x7fa870009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:45:54.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.341+0000 7fa87a000700 1 -- 172.21.15.123:0/1265387585 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa87000e040 con 0x7fa884070c70 2024-03-30T04:45:54.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.341+0000 7fa87a000700 1 -- 172.21.15.123:0/1265387585 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa870004290 con 0x7fa884070c70 2024-03-30T04:45:54.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.341+0000 7fa883200700 1 -- 172.21.15.123:0/1265387585 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa88410fcc0 con 0x7fa884070c70 2024-03-30T04:45:54.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.341+0000 7fa87a000700 1 -- 172.21.15.123:0/1265387585 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa87001c3f0 con 0x7fa884070c70 2024-03-30T04:45:54.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.341+0000 7fa883200700 1 -- 172.21.15.123:0/1265387585 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa884110200 con 0x7fa884070c70 2024-03-30T04:45:54.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.342+0000 7fa883200700 1 -- 172.21.15.123:0/1265387585 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa884071aa0 con 0x7fa884070c70 2024-03-30T04:45:54.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.343+0000 7fa87a000700 1 -- 172.21.15.123:0/1265387585 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fa870013420 con 0x7fa884070c70 2024-03-30T04:45:54.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.345+0000 7fa87a000700 1 --2- 172.21.15.123:0/1265387585 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa86806cc30 0x7fa86806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:45:54.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.345+0000 7fa87a000700 1 -- 172.21.15.123:0/1265387585 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa87008d4d0 con 0x7fa884070c70 2024-03-30T04:45:54.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.345+0000 7fa881e00700 1 --2- 172.21.15.123:0/1265387585 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa86806cc30 0x7fa86806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:45:54.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.346+0000 7fa881e00700 1 --2- 172.21.15.123:0/1265387585 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa86806cc30 0x7fa86806f0e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa874007fd0 tx=0x7fa874009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:45:54.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.349+0000 7fa87a000700 1 -- 172.21.15.123:0/1265387585 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa87005afd0 con 0x7fa884070c70 2024-03-30T04:45:54.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.562+0000 7fa883200700 1 -- 172.21.15.123:0/1265387585 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fa884111180 con 0x7fa86806cc30 2024-03-30T04:45:54.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.572+0000 7fa87a000700 1 -- 172.21.15.123:0/1265387585 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fa884111180 con 0x7fa86806cc30 2024-03-30T04:45:54.570 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:45:54.571 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (15m) 7m ago 27m 15.0M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:45:54.571 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (12m) 7m ago 27m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T04:45:54.571 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (12m) 7m ago 26m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T04:45:54.571 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (14m) 7m ago 27m 74.0M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T04:45:54.571 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (8m) 7m ago 21m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T04:45:54.571 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (8m) 7m ago 21m 10.9M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T04:45:54.571 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (21m) 7m ago 21m 10.7M - 17.2.7-797-g5c024335 f77085f6aab7 0dc41326818c 2024-03-30T04:45:54.571 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (7m) 7m ago 21m 9747k - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T04:45:54.571 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (17m) 7m ago 29m 506M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:45:54.571 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (16m) 7m ago 26m 450M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:45:54.572 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (13m) 7m ago 29m 53.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T04:45:54.572 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (13m) 7m ago 26m 42.6M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T04:45:54.572 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (15m) 7m ago 27m 8052k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:45:54.572 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (15m) 7m ago 26m 7880k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:45:54.572 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (11m) 7m ago 25m 104M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T04:45:54.572 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (11m) 7m ago 25m 93.4M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T04:45:54.572 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (10m) 7m ago 24m 86.3M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T04:45:54.572 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (10m) 7m ago 23m 97.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T04:45:54.572 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (9m) 7m ago 23m 97.9M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T04:45:54.572 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (8m) 7m ago 22m 85.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T04:45:54.572 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (15m) 7m ago 27m 53.7M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:45:54.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.574+0000 7fa883200700 1 -- 172.21.15.123:0/1265387585 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa86806cc30 msgr2=0x7fa86806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:54.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.574+0000 7fa883200700 1 --2- 172.21.15.123:0/1265387585 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa86806cc30 0x7fa86806f0e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa874007fd0 tx=0x7fa874009040 comp rx=0 tx=0).stop 2024-03-30T04:45:54.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.575+0000 7fa883200700 1 -- 172.21.15.123:0/1265387585 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa884070c70 msgr2=0x7fa884116000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:54.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.575+0000 7fa883200700 1 --2- 172.21.15.123:0/1265387585 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa884070c70 0x7fa884116000 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fa870009360 tx=0x7fa870009e30 comp rx=0 tx=0).stop 2024-03-30T04:45:54.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.575+0000 7fa883200700 1 -- 172.21.15.123:0/1265387585 shutdown_connections 2024-03-30T04:45:54.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.575+0000 7fa883200700 1 --2- 172.21.15.123:0/1265387585 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa86806cc30 0x7fa86806f0e0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:54.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.575+0000 7fa883200700 1 --2- 172.21.15.123:0/1265387585 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa884070360 0x7fa884115ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:54.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.575+0000 7fa883200700 1 --2- 172.21.15.123:0/1265387585 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa884070c70 0x7fa884116000 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:54.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.575+0000 7fa883200700 1 -- 172.21.15.123:0/1265387585 >> 172.21.15.123:0/1265387585 conn(0x7fa8840f8830 msgr2=0x7fa8840f9d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:45:54.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.575+0000 7fa883200700 1 -- 172.21.15.123:0/1265387585 shutdown_connections 2024-03-30T04:45:54.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.575+0000 7fa883200700 1 -- 172.21.15.123:0/1265387585 wait complete. 2024-03-30T04:45:54.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.708+0000 7f11a9a00700 1 -- 172.21.15.123:0/3958008139 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11a41037a0 msgr2=0x7f11a40fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:54.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.708+0000 7f11a9a00700 1 --2- 172.21.15.123:0/3958008139 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11a41037a0 0x7f11a40fce10 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f1190004940 tx=0x7f1190009e30 comp rx=0 tx=0).stop 2024-03-30T04:45:54.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.708+0000 7f11a9a00700 1 -- 172.21.15.123:0/3958008139 shutdown_connections 2024-03-30T04:45:54.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.708+0000 7f11a9a00700 1 --2- 172.21.15.123:0/3958008139 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11a41037a0 0x7f11a40fce10 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:54.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.708+0000 7f11a9a00700 1 --2- 172.21.15.123:0/3958008139 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f11a4102e00 0x7f11a41031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:54.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.709+0000 7f11a9a00700 1 -- 172.21.15.123:0/3958008139 >> 172.21.15.123:0/3958008139 conn(0x7f11a40f89b0 msgr2=0x7f11a40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:45:54.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.709+0000 7f11a9a00700 1 -- 172.21.15.123:0/3958008139 shutdown_connections 2024-03-30T04:45:54.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.709+0000 7f11a9a00700 1 -- 172.21.15.123:0/3958008139 wait complete. 2024-03-30T04:45:54.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.709+0000 7f11a9a00700 1 Processor -- start 2024-03-30T04:45:54.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.710+0000 7f11a9a00700 1 -- start start 2024-03-30T04:45:54.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.710+0000 7f11a9a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f11a4102e00 0x7f11a4113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:45:54.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.710+0000 7f11a9a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11a41037a0 0x7f11a4114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:45:54.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.710+0000 7f11a9a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11a4114690 con 0x7f11a41037a0 2024-03-30T04:45:54.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.710+0000 7f11a9a00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11a410da80 con 0x7f11a4102e00 2024-03-30T04:45:54.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.710+0000 7f11a2000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11a41037a0 0x7f11a4114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:45:54.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.710+0000 7f11a2000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11a41037a0 0x7f11a4114040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58384/0 (socket says 172.21.15.123:58384) 2024-03-30T04:45:54.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.710+0000 7f11a2000700 1 -- 172.21.15.123:0/208816100 learned_addr learned my addr 172.21.15.123:0/208816100 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:45:54.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.710+0000 7f11a2a00700 1 --2- 172.21.15.123:0/208816100 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f11a4102e00 0x7f11a4113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:45:54.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.711+0000 7f11a2000700 1 -- 172.21.15.123:0/208816100 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f11a4102e00 msgr2=0x7f11a4113b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:54.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.711+0000 7f11a2000700 1 --2- 172.21.15.123:0/208816100 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f11a4102e00 0x7f11a4113b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:54.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.711+0000 7f11a2000700 1 -- 172.21.15.123:0/208816100 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1190005040 con 0x7f11a41037a0 2024-03-30T04:45:54.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.711+0000 7f11a2a00700 1 --2- 172.21.15.123:0/208816100 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f11a4102e00 0x7f11a4113b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:45:54.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.711+0000 7f11a2000700 1 --2- 172.21.15.123:0/208816100 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11a41037a0 0x7f11a4114040 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f1190009390 tx=0x7f1190009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:45:54.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.711+0000 7f119aa00700 1 -- 172.21.15.123:0/208816100 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f119000e040 con 0x7f11a41037a0 2024-03-30T04:45:54.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.711+0000 7f119aa00700 1 -- 172.21.15.123:0/208816100 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1190013420 con 0x7f11a41037a0 2024-03-30T04:45:54.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.711+0000 7f119aa00700 1 -- 172.21.15.123:0/208816100 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f119001c630 con 0x7f11a41037a0 2024-03-30T04:45:54.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.711+0000 7f11a9a00700 1 -- 172.21.15.123:0/208816100 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11a410dd00 con 0x7f11a41037a0 2024-03-30T04:45:54.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.711+0000 7f11a9a00700 1 -- 172.21.15.123:0/208816100 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11a410e2c0 con 0x7f11a41037a0 2024-03-30T04:45:54.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.712+0000 7f11a9a00700 1 -- 172.21.15.123:0/208816100 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11a40fe550 con 0x7f11a41037a0 2024-03-30T04:45:54.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.713+0000 7f119aa00700 1 -- 172.21.15.123:0/208816100 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f11900139e0 con 0x7f11a41037a0 2024-03-30T04:45:54.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.714+0000 7f119aa00700 1 --2- 172.21.15.123:0/208816100 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f118806cc30 0x7f118806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:45:54.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.714+0000 7f119aa00700 1 -- 172.21.15.123:0/208816100 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f119008d2a0 con 0x7f11a41037a0 2024-03-30T04:45:54.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.714+0000 7f11a2a00700 1 --2- 172.21.15.123:0/208816100 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f118806cc30 0x7f118806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:45:54.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.715+0000 7f11a2a00700 1 --2- 172.21.15.123:0/208816100 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f118806cc30 0x7f118806f0e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1194009fd0 tx=0x7f1194009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:45:54.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:54.719+0000 7f119aa00700 1 -- 172.21.15.123:0/208816100 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f119005ad70 con 0x7f11a41037a0 2024-03-30T04:45:54.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:45:54 smithi123 ceph-mon[139369]: pgmap v441: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:54.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:45:54 smithi123 ceph-mon[139369]: from='client.34850 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:45:55.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.003+0000 7f11a9a00700 1 -- 172.21.15.123:0/208816100 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f11a405f5f0 con 0x7f11a41037a0 2024-03-30T04:45:55.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.004+0000 7f119aa00700 1 -- 172.21.15.123:0/208816100 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f119005a900 con 0x7f11a41037a0 2024-03-30T04:45:55.002 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:45:55.002 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:45:55.002 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:45:55.002 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:45:55.003 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:45:55.003 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:45:55.003 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:45:55.003 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:45:55.003 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T04:45:55.003 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:45:55.003 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:45:55.003 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:45:55.003 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 3 2024-03-30T04:45:55.003 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:45:55.003 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:45:55.003 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:45:55.003 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 13 2024-03-30T04:45:55.004 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:45:55.004 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:45:55.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.006+0000 7f11a9a00700 1 -- 172.21.15.123:0/208816100 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f118806cc30 msgr2=0x7f118806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:55.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.006+0000 7f11a9a00700 1 --2- 172.21.15.123:0/208816100 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f118806cc30 0x7f118806f0e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1194009fd0 tx=0x7f1194009ee0 comp rx=0 tx=0).stop 2024-03-30T04:45:55.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.006+0000 7f11a9a00700 1 -- 172.21.15.123:0/208816100 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11a41037a0 msgr2=0x7f11a4114040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:55.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.006+0000 7f11a9a00700 1 --2- 172.21.15.123:0/208816100 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11a41037a0 0x7f11a4114040 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f1190009390 tx=0x7f1190009e30 comp rx=0 tx=0).stop 2024-03-30T04:45:55.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.006+0000 7f11a9a00700 1 -- 172.21.15.123:0/208816100 shutdown_connections 2024-03-30T04:45:55.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.006+0000 7f11a9a00700 1 --2- 172.21.15.123:0/208816100 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f118806cc30 0x7f118806f0e0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:55.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.006+0000 7f11a9a00700 1 --2- 172.21.15.123:0/208816100 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f11a4102e00 0x7f11a4113b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:55.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.006+0000 7f11a9a00700 1 --2- 172.21.15.123:0/208816100 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11a41037a0 0x7f11a4114040 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:55.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.006+0000 7f11a9a00700 1 -- 172.21.15.123:0/208816100 >> 172.21.15.123:0/208816100 conn(0x7f11a40f89b0 msgr2=0x7f11a40f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:45:55.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.006+0000 7f11a9a00700 1 -- 172.21.15.123:0/208816100 shutdown_connections 2024-03-30T04:45:55.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.006+0000 7f11a9a00700 1 -- 172.21.15.123:0/208816100 wait complete. 2024-03-30T04:45:55.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:45:54 smithi135 ceph-mon[106011]: pgmap v441: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:55.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:45:54 smithi135 ceph-mon[106011]: from='client.34850 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:45:55.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.139+0000 7f0cc3600700 1 -- 172.21.15.123:0/2091882777 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0cbc1037a0 msgr2=0x7f0cbc0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:55.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.139+0000 7f0cc3600700 1 --2- 172.21.15.123:0/2091882777 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0cbc1037a0 0x7f0cbc0fce10 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f0ca8004940 tx=0x7f0ca8009e30 comp rx=0 tx=0).stop 2024-03-30T04:45:55.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.140+0000 7f0cc3600700 1 -- 172.21.15.123:0/2091882777 shutdown_connections 2024-03-30T04:45:55.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.140+0000 7f0cc3600700 1 --2- 172.21.15.123:0/2091882777 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0cbc1037a0 0x7f0cbc0fce10 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:55.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.140+0000 7f0cc3600700 1 --2- 172.21.15.123:0/2091882777 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0cbc102e00 0x7f0cbc1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:55.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.140+0000 7f0cc3600700 1 -- 172.21.15.123:0/2091882777 >> 172.21.15.123:0/2091882777 conn(0x7f0cbc0f89b0 msgr2=0x7f0cbc0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:45:55.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.140+0000 7f0cc3600700 1 -- 172.21.15.123:0/2091882777 shutdown_connections 2024-03-30T04:45:55.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.140+0000 7f0cc3600700 1 -- 172.21.15.123:0/2091882777 wait complete. 2024-03-30T04:45:55.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.140+0000 7f0cc3600700 1 Processor -- start 2024-03-30T04:45:55.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.141+0000 7f0cc3600700 1 -- start start 2024-03-30T04:45:55.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.141+0000 7f0cc3600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0cbc102e00 0x7f0cbc113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:45:55.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.141+0000 7f0cc3600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0cbc1037a0 0x7f0cbc114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:45:55.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.141+0000 7f0cc3600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cbc114690 con 0x7f0cbc1037a0 2024-03-30T04:45:55.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.141+0000 7f0cc3600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cbc10da80 con 0x7f0cbc102e00 2024-03-30T04:45:55.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.141+0000 7f0cbbe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0cbc1037a0 0x7f0cbc114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:45:55.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.141+0000 7f0cbbe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0cbc1037a0 0x7f0cbc114040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58402/0 (socket says 172.21.15.123:58402) 2024-03-30T04:45:55.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.141+0000 7f0cbbe00700 1 -- 172.21.15.123:0/950282733 learned_addr learned my addr 172.21.15.123:0/950282733 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:45:55.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.142+0000 7f0cc0c00700 1 --2- 172.21.15.123:0/950282733 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0cbc102e00 0x7f0cbc113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:45:55.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.142+0000 7f0cbbe00700 1 -- 172.21.15.123:0/950282733 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0cbc102e00 msgr2=0x7f0cbc113b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:55.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.142+0000 7f0cbbe00700 1 --2- 172.21.15.123:0/950282733 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0cbc102e00 0x7f0cbc113b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:55.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.142+0000 7f0cbbe00700 1 -- 172.21.15.123:0/950282733 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca8005040 con 0x7f0cbc1037a0 2024-03-30T04:45:55.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.142+0000 7f0cbbe00700 1 --2- 172.21.15.123:0/950282733 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0cbc1037a0 0x7f0cbc114040 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f0ca8009380 tx=0x7f0ca8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:45:55.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.142+0000 7f0cb8c00700 1 -- 172.21.15.123:0/950282733 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ca800e040 con 0x7f0cbc1037a0 2024-03-30T04:45:55.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.142+0000 7f0cb8c00700 1 -- 172.21.15.123:0/950282733 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0ca8013420 con 0x7f0cbc1037a0 2024-03-30T04:45:55.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.142+0000 7f0cc3600700 1 -- 172.21.15.123:0/950282733 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cbc10dd00 con 0x7f0cbc1037a0 2024-03-30T04:45:55.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.142+0000 7f0cb8c00700 1 -- 172.21.15.123:0/950282733 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ca801c630 con 0x7f0cbc1037a0 2024-03-30T04:45:55.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.142+0000 7f0cc3600700 1 -- 172.21.15.123:0/950282733 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cbc10e2c0 con 0x7f0cbc1037a0 2024-03-30T04:45:55.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.143+0000 7f0cc3600700 1 -- 172.21.15.123:0/950282733 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cbc0fe550 con 0x7f0cbc1037a0 2024-03-30T04:45:55.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.144+0000 7f0cb8c00700 1 -- 172.21.15.123:0/950282733 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f0ca80139e0 con 0x7f0cbc1037a0 2024-03-30T04:45:55.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.145+0000 7f0cb8c00700 1 --2- 172.21.15.123:0/950282733 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0ca406cc20 0x7f0ca406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:45:55.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.145+0000 7f0cb8c00700 1 -- 172.21.15.123:0/950282733 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f0ca808d290 con 0x7f0cbc1037a0 2024-03-30T04:45:55.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.145+0000 7f0cc0c00700 1 --2- 172.21.15.123:0/950282733 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0ca406cc20 0x7f0ca406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:45:55.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.146+0000 7f0cc0c00700 1 --2- 172.21.15.123:0/950282733 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0ca406cc20 0x7f0ca406f0d0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0cb00049b0 tx=0x7f0cb0009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:45:55.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.150+0000 7f0cb8c00700 1 -- 172.21.15.123:0/950282733 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0ca805ace0 con 0x7f0cbc1037a0 2024-03-30T04:45:55.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.396+0000 7f0cc3600700 1 -- 172.21.15.123:0/950282733 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0cbc109da0 con 0x7f0cbc1037a0 2024-03-30T04:45:55.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.396+0000 7f0cb8c00700 1 -- 172.21.15.123:0/950282733 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f0ca805a870 con 0x7f0cbc1037a0 2024-03-30T04:45:55.394 INFO:teuthology.orchestra.run.smithi123.stdout:e32 2024-03-30T04:45:55.395 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:45:55.395 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:45:55.395 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:45:55.395 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:45:55.395 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:45:55.395 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:45:55.395 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T04:45:55.395 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T04:45:55.395 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:45:55.395 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T04:45:55.395 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:45:55.395 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:45:55.395 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:45:55.395 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:45:55.396 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:45:55.396 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T04:45:55.396 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:45:55.396 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:45:55.396 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T04:45:55.396 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:45:55.396 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T04:45:55.396 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T04:45:55.396 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T04:45:55.396 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T04:45:55.396 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:45:55.396 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:45:55.396 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:45:55.396 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:45:55.397 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:45:55.397 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:45:55.397 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T04:45:55.397 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T04:45:55.397 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:45:55.397 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:45:55.397 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T04:45:55.397 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:45:55.397 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:45:55.397 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:45:55.397 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44315} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/1741352336,v1:172.21.15.135:6827/1741352336] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:45:55.397 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:45:55.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.398+0000 7f0cc3600700 1 -- 172.21.15.123:0/950282733 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0ca406cc20 msgr2=0x7f0ca406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:55.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.398+0000 7f0cc3600700 1 --2- 172.21.15.123:0/950282733 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0ca406cc20 0x7f0ca406f0d0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0cb00049b0 tx=0x7f0cb0009ee0 comp rx=0 tx=0).stop 2024-03-30T04:45:55.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.398+0000 7f0cc3600700 1 -- 172.21.15.123:0/950282733 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0cbc1037a0 msgr2=0x7f0cbc114040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:55.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.398+0000 7f0cc3600700 1 --2- 172.21.15.123:0/950282733 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0cbc1037a0 0x7f0cbc114040 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f0ca8009380 tx=0x7f0ca8009e30 comp rx=0 tx=0).stop 2024-03-30T04:45:55.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.398+0000 7f0cc3600700 1 -- 172.21.15.123:0/950282733 shutdown_connections 2024-03-30T04:45:55.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.398+0000 7f0cc3600700 1 --2- 172.21.15.123:0/950282733 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0ca406cc20 0x7f0ca406f0d0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:55.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.398+0000 7f0cc3600700 1 --2- 172.21.15.123:0/950282733 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0cbc102e00 0x7f0cbc113b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:55.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.398+0000 7f0cc3600700 1 --2- 172.21.15.123:0/950282733 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0cbc1037a0 0x7f0cbc114040 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:55.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.398+0000 7f0cc3600700 1 -- 172.21.15.123:0/950282733 >> 172.21.15.123:0/950282733 conn(0x7f0cbc0f89b0 msgr2=0x7f0cbc0f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:45:55.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.399+0000 7f0cc3600700 1 -- 172.21.15.123:0/950282733 shutdown_connections 2024-03-30T04:45:55.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.399+0000 7f0cc3600700 1 -- 172.21.15.123:0/950282733 wait complete. 2024-03-30T04:45:55.399 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 32 2024-03-30T04:45:55.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.531+0000 7f08eaa00700 1 -- 172.21.15.123:0/702013253 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08e41037c0 msgr2=0x7f08e40fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:55.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.531+0000 7f08eaa00700 1 --2- 172.21.15.123:0/702013253 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08e41037c0 0x7f08e40fce30 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f08d0004940 tx=0x7f08d0009e30 comp rx=0 tx=0).stop 2024-03-30T04:45:55.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.531+0000 7f08eaa00700 1 -- 172.21.15.123:0/702013253 shutdown_connections 2024-03-30T04:45:55.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.531+0000 7f08eaa00700 1 --2- 172.21.15.123:0/702013253 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08e41037c0 0x7f08e40fce30 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:55.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.531+0000 7f08eaa00700 1 --2- 172.21.15.123:0/702013253 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f08e4102e20 0x7f08e41031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:55.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.531+0000 7f08eaa00700 1 -- 172.21.15.123:0/702013253 >> 172.21.15.123:0/702013253 conn(0x7f08e40f89d0 msgr2=0x7f08e40fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:45:55.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.531+0000 7f08eaa00700 1 -- 172.21.15.123:0/702013253 shutdown_connections 2024-03-30T04:45:55.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.531+0000 7f08eaa00700 1 -- 172.21.15.123:0/702013253 wait complete. 2024-03-30T04:45:55.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.532+0000 7f08eaa00700 1 Processor -- start 2024-03-30T04:45:55.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.532+0000 7f08eaa00700 1 -- start start 2024-03-30T04:45:55.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.533+0000 7f08eaa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08e4102e20 0x7f08e4113ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:45:55.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.533+0000 7f08eaa00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f08e41037c0 0x7f08e4114010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:45:55.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.533+0000 7f08eaa00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e4114660 con 0x7f08e4102e20 2024-03-30T04:45:55.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.533+0000 7f08eaa00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e410da50 con 0x7f08e41037c0 2024-03-30T04:45:55.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.533+0000 7f08e3e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08e4102e20 0x7f08e4113ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:45:55.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.533+0000 7f08e3e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08e4102e20 0x7f08e4113ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58410/0 (socket says 172.21.15.123:58410) 2024-03-30T04:45:55.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.533+0000 7f08e3e00700 1 -- 172.21.15.123:0/4190669203 learned_addr learned my addr 172.21.15.123:0/4190669203 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:45:55.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.533+0000 7f08e3400700 1 --2- 172.21.15.123:0/4190669203 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f08e41037c0 0x7f08e4114010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:45:55.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.533+0000 7f08e3e00700 1 -- 172.21.15.123:0/4190669203 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f08e41037c0 msgr2=0x7f08e4114010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:55.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.533+0000 7f08e3e00700 1 --2- 172.21.15.123:0/4190669203 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f08e41037c0 0x7f08e4114010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:55.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.533+0000 7f08e3e00700 1 -- 172.21.15.123:0/4190669203 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08d0005040 con 0x7f08e4102e20 2024-03-30T04:45:55.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.533+0000 7f08e3400700 1 --2- 172.21.15.123:0/4190669203 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f08e41037c0 0x7f08e4114010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:45:55.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.534+0000 7f08e3e00700 1 --2- 172.21.15.123:0/4190669203 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08e4102e20 0x7f08e4113ad0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f08d4002a50 tx=0x7f08d40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:45:55.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.534+0000 7f08dbe00700 1 -- 172.21.15.123:0/4190669203 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f08d4007cc0 con 0x7f08e4102e20 2024-03-30T04:45:55.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.534+0000 7f08dbe00700 1 -- 172.21.15.123:0/4190669203 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f08d400f040 con 0x7f08e4102e20 2024-03-30T04:45:55.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.534+0000 7f08dbe00700 1 -- 172.21.15.123:0/4190669203 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f08d4009540 con 0x7f08e4102e20 2024-03-30T04:45:55.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.534+0000 7f08eaa00700 1 -- 172.21.15.123:0/4190669203 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08e410dd30 con 0x7f08e4102e20 2024-03-30T04:45:55.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.534+0000 7f08eaa00700 1 -- 172.21.15.123:0/4190669203 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08e410e2f0 con 0x7f08e4102e20 2024-03-30T04:45:55.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.535+0000 7f08eaa00700 1 -- 172.21.15.123:0/4190669203 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08e40fe570 con 0x7f08e4102e20 2024-03-30T04:45:55.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.536+0000 7f08dbe00700 1 -- 172.21.15.123:0/4190669203 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f08d40096a0 con 0x7f08e4102e20 2024-03-30T04:45:55.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.537+0000 7f08dbe00700 1 --2- 172.21.15.123:0/4190669203 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f08c806cc30 0x7f08c806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:45:55.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.537+0000 7f08dbe00700 1 -- 172.21.15.123:0/4190669203 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f08d408bde0 con 0x7f08e4102e20 2024-03-30T04:45:55.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.537+0000 7f08e3400700 1 --2- 172.21.15.123:0/4190669203 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f08c806cc30 0x7f08c806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:45:55.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.537+0000 7f08e3400700 1 --2- 172.21.15.123:0/4190669203 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f08c806cc30 0x7f08c806f0e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f08e410ec10 tx=0x7f08d0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:45:55.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.541+0000 7f08dbe00700 1 -- 172.21.15.123:0/4190669203 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f08d4059830 con 0x7f08e4102e20 2024-03-30T04:45:55.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.767+0000 7f08eaa00700 1 -- 172.21.15.123:0/4190669203 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f08e4109dc0 con 0x7f08c806cc30 2024-03-30T04:45:55.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.769+0000 7f08dbe00700 1 -- 172.21.15.123:0/4190669203 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f08e4109dc0 con 0x7f08c806cc30 2024-03-30T04:45:55.767 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:45:55.767 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:45:55.767 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:45:55.767 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T04:45:55.767 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:45:55.767 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T04:45:55.767 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T04:45:55.767 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T04:45:55.768 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T04:45:55.768 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:45:55.768 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "15/21 daemons upgraded", 2024-03-30T04:45:55.768 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T04:45:55.768 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:45:55.768 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:45:55.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.771+0000 7f08eaa00700 1 -- 172.21.15.123:0/4190669203 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f08c806cc30 msgr2=0x7f08c806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:55.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.771+0000 7f08eaa00700 1 --2- 172.21.15.123:0/4190669203 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f08c806cc30 0x7f08c806f0e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f08e410ec10 tx=0x7f08d0009e30 comp rx=0 tx=0).stop 2024-03-30T04:45:55.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.771+0000 7f08eaa00700 1 -- 172.21.15.123:0/4190669203 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08e4102e20 msgr2=0x7f08e4113ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:55.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.771+0000 7f08eaa00700 1 --2- 172.21.15.123:0/4190669203 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08e4102e20 0x7f08e4113ad0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f08d4002a50 tx=0x7f08d40046b0 comp rx=0 tx=0).stop 2024-03-30T04:45:55.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.771+0000 7f08eaa00700 1 -- 172.21.15.123:0/4190669203 shutdown_connections 2024-03-30T04:45:55.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.771+0000 7f08eaa00700 1 --2- 172.21.15.123:0/4190669203 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f08c806cc30 0x7f08c806f0e0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:55.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.771+0000 7f08eaa00700 1 --2- 172.21.15.123:0/4190669203 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08e4102e20 0x7f08e4113ad0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:55.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.771+0000 7f08eaa00700 1 --2- 172.21.15.123:0/4190669203 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f08e41037c0 0x7f08e4114010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:55.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.771+0000 7f08eaa00700 1 -- 172.21.15.123:0/4190669203 >> 172.21.15.123:0/4190669203 conn(0x7f08e40f89d0 msgr2=0x7f08e40f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:45:55.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.771+0000 7f08eaa00700 1 -- 172.21.15.123:0/4190669203 shutdown_connections 2024-03-30T04:45:55.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.771+0000 7f08eaa00700 1 -- 172.21.15.123:0/4190669203 wait complete. 2024-03-30T04:45:55.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.904+0000 7fb451a00700 1 -- 172.21.15.123:0/1382755420 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb44c1037a0 msgr2=0x7fb44c0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:55.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.904+0000 7fb451a00700 1 --2- 172.21.15.123:0/1382755420 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb44c1037a0 0x7fb44c0fce10 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb438004940 tx=0x7fb438009e30 comp rx=0 tx=0).stop 2024-03-30T04:45:55.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.904+0000 7fb451a00700 1 -- 172.21.15.123:0/1382755420 shutdown_connections 2024-03-30T04:45:55.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.904+0000 7fb451a00700 1 --2- 172.21.15.123:0/1382755420 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb44c1037a0 0x7fb44c0fce10 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:55.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.904+0000 7fb451a00700 1 --2- 172.21.15.123:0/1382755420 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb44c102e00 0x7fb44c1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:55.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.904+0000 7fb451a00700 1 -- 172.21.15.123:0/1382755420 >> 172.21.15.123:0/1382755420 conn(0x7fb44c0f89b0 msgr2=0x7fb44c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:45:55.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.904+0000 7fb451a00700 1 -- 172.21.15.123:0/1382755420 shutdown_connections 2024-03-30T04:45:55.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.904+0000 7fb451a00700 1 -- 172.21.15.123:0/1382755420 wait complete. 2024-03-30T04:45:55.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.905+0000 7fb451a00700 1 Processor -- start 2024-03-30T04:45:55.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.905+0000 7fb451a00700 1 -- start start 2024-03-30T04:45:55.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.906+0000 7fb451a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb44c102e00 0x7fb44c115d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:45:55.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.906+0000 7fb451a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb44c1037a0 0x7fb44c116240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:45:55.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.906+0000 7fb451a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb44c116890 con 0x7fb44c1037a0 2024-03-30T04:45:55.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.906+0000 7fb451a00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb44c10fc60 con 0x7fb44c102e00 2024-03-30T04:45:55.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.906+0000 7fb44a000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb44c1037a0 0x7fb44c116240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:45:55.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.906+0000 7fb44a000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb44c1037a0 0x7fb44c116240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58428/0 (socket says 172.21.15.123:58428) 2024-03-30T04:45:55.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.906+0000 7fb44a000700 1 -- 172.21.15.123:0/4152160363 learned_addr learned my addr 172.21.15.123:0/4152160363 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:45:55.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.906+0000 7fb44aa00700 1 --2- 172.21.15.123:0/4152160363 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb44c102e00 0x7fb44c115d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:45:55.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.906+0000 7fb44a000700 1 -- 172.21.15.123:0/4152160363 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb44c102e00 msgr2=0x7fb44c115d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:55.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.906+0000 7fb44a000700 1 --2- 172.21.15.123:0/4152160363 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb44c102e00 0x7fb44c115d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:55.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.906+0000 7fb44a000700 1 -- 172.21.15.123:0/4152160363 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb438005040 con 0x7fb44c1037a0 2024-03-30T04:45:55.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.906+0000 7fb44a000700 1 --2- 172.21.15.123:0/4152160363 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb44c1037a0 0x7fb44c116240 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fb438009390 tx=0x7fb438009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:45:55.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.907+0000 7fb442a00700 1 -- 172.21.15.123:0/4152160363 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb43800e040 con 0x7fb44c1037a0 2024-03-30T04:45:55.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.907+0000 7fb442a00700 1 -- 172.21.15.123:0/4152160363 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb438013420 con 0x7fb44c1037a0 2024-03-30T04:45:55.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.907+0000 7fb451a00700 1 -- 172.21.15.123:0/4152160363 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb44c10fee0 con 0x7fb44c1037a0 2024-03-30T04:45:55.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.907+0000 7fb442a00700 1 -- 172.21.15.123:0/4152160363 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb43801c630 con 0x7fb44c1037a0 2024-03-30T04:45:55.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.907+0000 7fb451a00700 1 -- 172.21.15.123:0/4152160363 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb44c1104a0 con 0x7fb44c1037a0 2024-03-30T04:45:55.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.908+0000 7fb451a00700 1 -- 172.21.15.123:0/4152160363 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb44c0fe550 con 0x7fb44c1037a0 2024-03-30T04:45:55.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.909+0000 7fb442a00700 1 -- 172.21.15.123:0/4152160363 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fb438013a00 con 0x7fb44c1037a0 2024-03-30T04:45:55.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.910+0000 7fb442a00700 1 --2- 172.21.15.123:0/4152160363 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb43006cc30 0x7fb43006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:45:55.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.910+0000 7fb442a00700 1 -- 172.21.15.123:0/4152160363 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb43808d0e0 con 0x7fb44c1037a0 2024-03-30T04:45:55.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.910+0000 7fb44aa00700 1 --2- 172.21.15.123:0/4152160363 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb43006cc30 0x7fb43006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:45:55.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.910+0000 7fb44aa00700 1 --2- 172.21.15.123:0/4152160363 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb43006cc30 0x7fb43006f0e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb43c004f40 tx=0x7fb43c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:45:55.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:55.914+0000 7fb442a00700 1 -- 172.21.15.123:0/4152160363 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb43805abb0 con 0x7fb44c1037a0 2024-03-30T04:45:55.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:45:55 smithi123 ceph-mon[139369]: from='client.34854 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:45:55.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:45:55 smithi123 ceph-mon[139369]: from='client.34858 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:45:55.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:45:55 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/208816100' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:45:55.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:45:55 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/950282733' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:45:56.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:45:55 smithi135 ceph-mon[106011]: from='client.34854 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:45:56.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:45:55 smithi135 ceph-mon[106011]: from='client.34858 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:45:56.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:45:55 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/208816100' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:45:56.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:45:55 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/950282733' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:45:56.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:56.198+0000 7fb451a00700 1 -- 172.21.15.123:0/4152160363 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fb44c04bed0 con 0x7fb44c1037a0 2024-03-30T04:45:56.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:56.198+0000 7fb442a00700 1 -- 172.21.15.123:0/4152160363 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fb43805a740 con 0x7fb44c1037a0 2024-03-30T04:45:56.196 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T04:45:56.197 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T04:45:56.197 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T04:45:56.197 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T04:45:56.197 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T04:45:56.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:56.200+0000 7fb451a00700 1 -- 172.21.15.123:0/4152160363 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb43006cc30 msgr2=0x7fb43006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:56.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:56.200+0000 7fb451a00700 1 --2- 172.21.15.123:0/4152160363 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb43006cc30 0x7fb43006f0e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb43c004f40 tx=0x7fb43c00b040 comp rx=0 tx=0).stop 2024-03-30T04:45:56.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:56.200+0000 7fb451a00700 1 -- 172.21.15.123:0/4152160363 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb44c1037a0 msgr2=0x7fb44c116240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:45:56.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:56.200+0000 7fb451a00700 1 --2- 172.21.15.123:0/4152160363 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb44c1037a0 0x7fb44c116240 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fb438009390 tx=0x7fb438009e30 comp rx=0 tx=0).stop 2024-03-30T04:45:56.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:56.200+0000 7fb451a00700 1 -- 172.21.15.123:0/4152160363 shutdown_connections 2024-03-30T04:45:56.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:56.200+0000 7fb451a00700 1 --2- 172.21.15.123:0/4152160363 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb43006cc30 0x7fb43006f0e0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:56.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:56.200+0000 7fb451a00700 1 --2- 172.21.15.123:0/4152160363 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb44c102e00 0x7fb44c115d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:56.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:56.200+0000 7fb451a00700 1 --2- 172.21.15.123:0/4152160363 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb44c1037a0 0x7fb44c116240 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:45:56.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:56.200+0000 7fb451a00700 1 -- 172.21.15.123:0/4152160363 >> 172.21.15.123:0/4152160363 conn(0x7fb44c0f89b0 msgr2=0x7fb44c0f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:45:56.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:56.201+0000 7fb451a00700 1 -- 172.21.15.123:0/4152160363 shutdown_connections 2024-03-30T04:45:56.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:45:56.201+0000 7fb451a00700 1 -- 172.21.15.123:0/4152160363 wait complete. 2024-03-30T04:45:56.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:45:56 smithi123 ceph-mon[139369]: pgmap v442: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:56.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:45:56 smithi123 ceph-mon[139369]: from='client.34870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:45:56.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:45:56 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/4152160363' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:45:57.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:45:56 smithi135 ceph-mon[106011]: pgmap v442: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:57.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:45:56 smithi135 ceph-mon[106011]: from='client.34870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:45:57.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:45:56 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/4152160363' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:45:58.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:45:58 smithi123 ceph-mon[139369]: pgmap v443: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:59.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:45:58 smithi135 ceph-mon[106011]: pgmap v443: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:45:59.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:45:59 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:46:00.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:45:59 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:46:00.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:46:00 smithi123 ceph-mon[139369]: pgmap v444: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:01.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:46:00 smithi135 ceph-mon[106011]: pgmap v444: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:02.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:46:02 smithi123 ceph-mon[139369]: pgmap v445: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:03.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:46:02 smithi135 ceph-mon[106011]: pgmap v445: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:04.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:46:04 smithi123 ceph-mon[139369]: pgmap v446: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:05.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:46:04 smithi135 ceph-mon[106011]: pgmap v446: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:06.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:46:06 smithi123 ceph-mon[139369]: pgmap v447: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:07.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:46:06 smithi135 ceph-mon[106011]: pgmap v447: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:08.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:46:08 smithi123 ceph-mon[139369]: pgmap v448: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:09.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:46:08 smithi135 ceph-mon[106011]: pgmap v448: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:10.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:46:10 smithi135 ceph-mon[106011]: pgmap v449: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:10.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:46:10 smithi123 ceph-mon[139369]: pgmap v449: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:12.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:46:12 smithi123 ceph-mon[139369]: pgmap v450: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:13.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:46:12 smithi135 ceph-mon[106011]: pgmap v450: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:14.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:46:14 smithi123 ceph-mon[139369]: pgmap v451: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:14.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:46:14 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:46:15.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:46:14 smithi135 ceph-mon[106011]: pgmap v451: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:15.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:46:14 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:46:16.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:46:16 smithi123 ceph-mon[139369]: pgmap v452: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:17.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:46:16 smithi135 ceph-mon[106011]: pgmap v452: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:18.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:46:18 smithi123 ceph-mon[139369]: pgmap v453: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:19.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:46:18 smithi135 ceph-mon[106011]: pgmap v453: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:20.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:46:20 smithi135 ceph-mon[106011]: pgmap v454: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:20.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:46:20 smithi123 ceph-mon[139369]: pgmap v454: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:22.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:46:22 smithi123 ceph-mon[139369]: pgmap v455: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:23.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:46:22 smithi135 ceph-mon[106011]: pgmap v455: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:24.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:46:24 smithi123 ceph-mon[139369]: pgmap v456: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:25.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:46:24 smithi135 ceph-mon[106011]: pgmap v456: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:26.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.349+0000 7fecaea00700 1 -- 172.21.15.123:0/735968791 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feca8102e00 msgr2=0x7feca81031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:46:26.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.349+0000 7fecaea00700 1 --2- 172.21.15.123:0/735968791 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feca8102e00 0x7feca81031d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fec98004940 tx=0x7fec98009e30 comp rx=0 tx=0).stop 2024-03-30T04:46:26.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.349+0000 7fecaea00700 1 -- 172.21.15.123:0/735968791 shutdown_connections 2024-03-30T04:46:26.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.349+0000 7fecaea00700 1 --2- 172.21.15.123:0/735968791 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feca81037a0 0x7feca80fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:26.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.349+0000 7fecaea00700 1 --2- 172.21.15.123:0/735968791 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feca8102e00 0x7feca81031d0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:26.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.349+0000 7fecaea00700 1 -- 172.21.15.123:0/735968791 >> 172.21.15.123:0/735968791 conn(0x7feca80f89b0 msgr2=0x7feca80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:46:26.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.349+0000 7fecaea00700 1 -- 172.21.15.123:0/735968791 shutdown_connections 2024-03-30T04:46:26.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.349+0000 7fecaea00700 1 -- 172.21.15.123:0/735968791 wait complete. 2024-03-30T04:46:26.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.350+0000 7fecaea00700 1 Processor -- start 2024-03-30T04:46:26.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.350+0000 7fecaea00700 1 -- start start 2024-03-30T04:46:26.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.350+0000 7fecaea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feca8102e00 0x7feca80702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:46:26.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.350+0000 7fecaea00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feca81037a0 0x7feca8070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:46:26.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.350+0000 7fecaea00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feca8070d90 con 0x7feca8102e00 2024-03-30T04:46:26.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.350+0000 7fecaea00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feca8070ed0 con 0x7feca81037a0 2024-03-30T04:46:26.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.351+0000 7feca7e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feca8102e00 0x7feca80702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:46:26.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.351+0000 7feca7e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feca8102e00 0x7feca80702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50388/0 (socket says 172.21.15.123:50388) 2024-03-30T04:46:26.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.351+0000 7feca7e00700 1 -- 172.21.15.123:0/1917683249 learned_addr learned my addr 172.21.15.123:0/1917683249 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:46:26.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.351+0000 7feca7400700 1 --2- 172.21.15.123:0/1917683249 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feca81037a0 0x7feca8070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:46:26.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.351+0000 7feca7e00700 1 -- 172.21.15.123:0/1917683249 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feca81037a0 msgr2=0x7feca8070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:46:26.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.351+0000 7feca7e00700 1 --2- 172.21.15.123:0/1917683249 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feca81037a0 0x7feca8070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:26.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.351+0000 7feca7e00700 1 -- 172.21.15.123:0/1917683249 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec98005040 con 0x7feca8102e00 2024-03-30T04:46:26.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.351+0000 7feca7e00700 1 --2- 172.21.15.123:0/1917683249 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feca8102e00 0x7feca80702c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fec98009390 tx=0x7fec98009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:46:26.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.352+0000 7fec9fe00700 1 -- 172.21.15.123:0/1917683249 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec9800e040 con 0x7feca8102e00 2024-03-30T04:46:26.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.352+0000 7fecaea00700 1 -- 172.21.15.123:0/1917683249 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feca8071120 con 0x7feca8102e00 2024-03-30T04:46:26.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.352+0000 7fec9fe00700 1 -- 172.21.15.123:0/1917683249 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fec98013420 con 0x7feca8102e00 2024-03-30T04:46:26.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.352+0000 7fecaea00700 1 -- 172.21.15.123:0/1917683249 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feca811a170 con 0x7feca8102e00 2024-03-30T04:46:26.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.352+0000 7fec9fe00700 1 -- 172.21.15.123:0/1917683249 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec980255e0 con 0x7feca8102e00 2024-03-30T04:46:26.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.353+0000 7fecaea00700 1 -- 172.21.15.123:0/1917683249 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feca80fe550 con 0x7feca8102e00 2024-03-30T04:46:26.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.354+0000 7fec9fe00700 1 -- 172.21.15.123:0/1917683249 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fec980139e0 con 0x7feca8102e00 2024-03-30T04:46:26.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.355+0000 7fec9fe00700 1 --2- 172.21.15.123:0/1917683249 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fec8c06cc30 0x7fec8c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:46:26.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.355+0000 7fec9fe00700 1 -- 172.21.15.123:0/1917683249 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fec9808d350 con 0x7feca8102e00 2024-03-30T04:46:26.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.355+0000 7feca7400700 1 --2- 172.21.15.123:0/1917683249 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fec8c06cc30 0x7fec8c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:46:26.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.355+0000 7feca7400700 1 --2- 172.21.15.123:0/1917683249 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fec8c06cc30 0x7fec8c06f0e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fec94009fd0 tx=0x7fec94009f60 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:46:26.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.359+0000 7fec9fe00700 1 -- 172.21.15.123:0/1917683249 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fec98057310 con 0x7feca8102e00 2024-03-30T04:46:26.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.581+0000 7fecaea00700 1 -- 172.21.15.123:0/1917683249 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7feca80719b0 con 0x7fec8c06cc30 2024-03-30T04:46:26.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.582+0000 7fec9fe00700 1 -- 172.21.15.123:0/1917683249 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7feca80719b0 con 0x7fec8c06cc30 2024-03-30T04:46:26.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.584+0000 7fecaea00700 1 -- 172.21.15.123:0/1917683249 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fec8c06cc30 msgr2=0x7fec8c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:46:26.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.584+0000 7fecaea00700 1 --2- 172.21.15.123:0/1917683249 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fec8c06cc30 0x7fec8c06f0e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fec94009fd0 tx=0x7fec94009f60 comp rx=0 tx=0).stop 2024-03-30T04:46:26.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.584+0000 7fecaea00700 1 -- 172.21.15.123:0/1917683249 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feca8102e00 msgr2=0x7feca80702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:46:26.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.584+0000 7fecaea00700 1 --2- 172.21.15.123:0/1917683249 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feca8102e00 0x7feca80702c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fec98009390 tx=0x7fec98009e30 comp rx=0 tx=0).stop 2024-03-30T04:46:26.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.585+0000 7fecaea00700 1 -- 172.21.15.123:0/1917683249 shutdown_connections 2024-03-30T04:46:26.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.585+0000 7fecaea00700 1 --2- 172.21.15.123:0/1917683249 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fec8c06cc30 0x7fec8c06f0e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:26.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.585+0000 7fecaea00700 1 --2- 172.21.15.123:0/1917683249 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feca8102e00 0x7feca80702c0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:26.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.585+0000 7fecaea00700 1 --2- 172.21.15.123:0/1917683249 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feca81037a0 0x7feca8070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:26.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.585+0000 7fecaea00700 1 -- 172.21.15.123:0/1917683249 >> 172.21.15.123:0/1917683249 conn(0x7feca80f89b0 msgr2=0x7feca80f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:46:26.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.585+0000 7fecaea00700 1 -- 172.21.15.123:0/1917683249 shutdown_connections 2024-03-30T04:46:26.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.585+0000 7fecaea00700 1 -- 172.21.15.123:0/1917683249 wait complete. 2024-03-30T04:46:26.597 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:46:26.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.720+0000 7f6e4a000700 1 -- 172.21.15.123:0/1554711076 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6e441037a0 msgr2=0x7f6e440fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:46:26.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.720+0000 7f6e4a000700 1 --2- 172.21.15.123:0/1554711076 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6e441037a0 0x7f6e440fce10 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f6e30004940 tx=0x7f6e30009e30 comp rx=0 tx=0).stop 2024-03-30T04:46:26.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.720+0000 7f6e4a000700 1 -- 172.21.15.123:0/1554711076 shutdown_connections 2024-03-30T04:46:26.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.720+0000 7f6e4a000700 1 --2- 172.21.15.123:0/1554711076 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6e441037a0 0x7f6e440fce10 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:26.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.720+0000 7f6e4a000700 1 --2- 172.21.15.123:0/1554711076 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6e44102e00 0x7f6e441031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:26.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.720+0000 7f6e4a000700 1 -- 172.21.15.123:0/1554711076 >> 172.21.15.123:0/1554711076 conn(0x7f6e440f89b0 msgr2=0x7f6e440fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:46:26.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.720+0000 7f6e4a000700 1 -- 172.21.15.123:0/1554711076 shutdown_connections 2024-03-30T04:46:26.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.721+0000 7f6e4a000700 1 -- 172.21.15.123:0/1554711076 wait complete. 2024-03-30T04:46:26.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.721+0000 7f6e4a000700 1 Processor -- start 2024-03-30T04:46:26.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.721+0000 7f6e4a000700 1 -- start start 2024-03-30T04:46:26.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.722+0000 7f6e4a000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6e44102e00 0x7f6e44115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:46:26.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.722+0000 7f6e4a000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6e441037a0 0x7f6e44116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:46:26.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.722+0000 7f6e4a000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e44116880 con 0x7f6e44102e00 2024-03-30T04:46:26.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.722+0000 7f6e4a000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4410fc50 con 0x7f6e441037a0 2024-03-30T04:46:26.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.722+0000 7f6e43400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6e44102e00 0x7f6e44115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:46:26.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.722+0000 7f6e43400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6e44102e00 0x7f6e44115cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50408/0 (socket says 172.21.15.123:50408) 2024-03-30T04:46:26.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.722+0000 7f6e43400700 1 -- 172.21.15.123:0/4128950240 learned_addr learned my addr 172.21.15.123:0/4128950240 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:46:26.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.722+0000 7f6e42a00700 1 --2- 172.21.15.123:0/4128950240 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6e441037a0 0x7f6e44116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:46:26.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.722+0000 7f6e43400700 1 -- 172.21.15.123:0/4128950240 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6e441037a0 msgr2=0x7f6e44116230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:46:26.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.722+0000 7f6e43400700 1 --2- 172.21.15.123:0/4128950240 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6e441037a0 0x7f6e44116230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:26.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.722+0000 7f6e43400700 1 -- 172.21.15.123:0/4128950240 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e30005040 con 0x7f6e44102e00 2024-03-30T04:46:26.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.723+0000 7f6e43400700 1 --2- 172.21.15.123:0/4128950240 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6e44102e00 0x7f6e44115cf0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f6e3400c8c0 tx=0x7f6e340044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:46:26.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.723+0000 7f6e3b400700 1 -- 172.21.15.123:0/4128950240 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e3400a980 con 0x7f6e44102e00 2024-03-30T04:46:26.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.723+0000 7f6e3b400700 1 -- 172.21.15.123:0/4128950240 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6e3400ce60 con 0x7f6e44102e00 2024-03-30T04:46:26.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.723+0000 7f6e4a000700 1 -- 172.21.15.123:0/4128950240 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e4410ff30 con 0x7f6e44102e00 2024-03-30T04:46:26.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.723+0000 7f6e3b400700 1 -- 172.21.15.123:0/4128950240 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e340113a0 con 0x7f6e44102e00 2024-03-30T04:46:26.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.723+0000 7f6e4a000700 1 -- 172.21.15.123:0/4128950240 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e441104f0 con 0x7f6e44102e00 2024-03-30T04:46:26.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.724+0000 7f6e4a000700 1 -- 172.21.15.123:0/4128950240 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e440fe550 con 0x7f6e44102e00 2024-03-30T04:46:26.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.725+0000 7f6e3b400700 1 -- 172.21.15.123:0/4128950240 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f6e34012400 con 0x7f6e44102e00 2024-03-30T04:46:26.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.726+0000 7f6e3b400700 1 --2- 172.21.15.123:0/4128950240 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6e2806cc30 0x7f6e2806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:46:26.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.727+0000 7f6e3b400700 1 -- 172.21.15.123:0/4128950240 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f6e3408c0f0 con 0x7f6e44102e00 2024-03-30T04:46:26.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.727+0000 7f6e42a00700 1 --2- 172.21.15.123:0/4128950240 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6e2806cc30 0x7f6e2806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:46:26.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.727+0000 7f6e42a00700 1 --2- 172.21.15.123:0/4128950240 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6e2806cc30 0x7f6e2806f0e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6e44111590 tx=0x7f6e30009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:46:26.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.731+0000 7f6e3b400700 1 -- 172.21.15.123:0/4128950240 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6e34059bc0 con 0x7f6e44102e00 2024-03-30T04:46:26.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.952+0000 7f6e4a000700 1 -- 172.21.15.123:0/4128950240 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6e4404bed0 con 0x7f6e2806cc30 2024-03-30T04:46:26.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.954+0000 7f6e3b400700 1 -- 172.21.15.123:0/4128950240 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f6e4404bed0 con 0x7f6e2806cc30 2024-03-30T04:46:26.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.956+0000 7f6e4a000700 1 -- 172.21.15.123:0/4128950240 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6e2806cc30 msgr2=0x7f6e2806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:46:26.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.956+0000 7f6e4a000700 1 --2- 172.21.15.123:0/4128950240 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6e2806cc30 0x7f6e2806f0e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6e44111590 tx=0x7f6e30009e30 comp rx=0 tx=0).stop 2024-03-30T04:46:26.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.956+0000 7f6e4a000700 1 -- 172.21.15.123:0/4128950240 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6e44102e00 msgr2=0x7f6e44115cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:46:26.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.956+0000 7f6e4a000700 1 --2- 172.21.15.123:0/4128950240 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6e44102e00 0x7f6e44115cf0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f6e3400c8c0 tx=0x7f6e340044c0 comp rx=0 tx=0).stop 2024-03-30T04:46:26.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.956+0000 7f6e4a000700 1 -- 172.21.15.123:0/4128950240 shutdown_connections 2024-03-30T04:46:26.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.956+0000 7f6e4a000700 1 --2- 172.21.15.123:0/4128950240 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6e2806cc30 0x7f6e2806f0e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:26.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.956+0000 7f6e4a000700 1 --2- 172.21.15.123:0/4128950240 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6e44102e00 0x7f6e44115cf0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:26.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.956+0000 7f6e4a000700 1 --2- 172.21.15.123:0/4128950240 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6e441037a0 0x7f6e44116230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:26.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.956+0000 7f6e4a000700 1 -- 172.21.15.123:0/4128950240 >> 172.21.15.123:0/4128950240 conn(0x7f6e440f89b0 msgr2=0x7f6e440f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:46:26.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.956+0000 7f6e4a000700 1 -- 172.21.15.123:0/4128950240 shutdown_connections 2024-03-30T04:46:26.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:26.956+0000 7f6e4a000700 1 -- 172.21.15.123:0/4128950240 wait complete. 2024-03-30T04:46:26.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:46:26 smithi123 ceph-mon[139369]: pgmap v457: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:27.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.090+0000 7fc91ac00700 1 -- 172.21.15.123:0/1005088614 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc9141037a0 msgr2=0x7fc9140fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:46:27.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.090+0000 7fc91ac00700 1 --2- 172.21.15.123:0/1005088614 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc9141037a0 0x7fc9140fce10 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fc900004940 tx=0x7fc900009e30 comp rx=0 tx=0).stop 2024-03-30T04:46:27.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.090+0000 7fc91ac00700 1 -- 172.21.15.123:0/1005088614 shutdown_connections 2024-03-30T04:46:27.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.090+0000 7fc91ac00700 1 --2- 172.21.15.123:0/1005088614 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc9141037a0 0x7fc9140fce10 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:27.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.090+0000 7fc91ac00700 1 --2- 172.21.15.123:0/1005088614 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc914102e00 0x7fc9141031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:27.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.090+0000 7fc91ac00700 1 -- 172.21.15.123:0/1005088614 >> 172.21.15.123:0/1005088614 conn(0x7fc9140f89b0 msgr2=0x7fc9140fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:46:27.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.090+0000 7fc91ac00700 1 -- 172.21.15.123:0/1005088614 shutdown_connections 2024-03-30T04:46:27.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.091+0000 7fc91ac00700 1 -- 172.21.15.123:0/1005088614 wait complete. 2024-03-30T04:46:27.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.091+0000 7fc91ac00700 1 Processor -- start 2024-03-30T04:46:27.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.091+0000 7fc91ac00700 1 -- start start 2024-03-30T04:46:27.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.092+0000 7fc91ac00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc914102e00 0x7fc914115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:46:27.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.092+0000 7fc91ac00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc9141037a0 0x7fc9141161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:46:27.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.092+0000 7fc91ac00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc914116810 con 0x7fc914102e00 2024-03-30T04:46:27.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.092+0000 7fc91ac00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc91410fbe0 con 0x7fc9141037a0 2024-03-30T04:46:27.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.092+0000 7fc913e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc914102e00 0x7fc914115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:46:27.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.092+0000 7fc913e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc914102e00 0x7fc914115c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50424/0 (socket says 172.21.15.123:50424) 2024-03-30T04:46:27.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.092+0000 7fc913e00700 1 -- 172.21.15.123:0/1512624874 learned_addr learned my addr 172.21.15.123:0/1512624874 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:46:27.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.092+0000 7fc913400700 1 --2- 172.21.15.123:0/1512624874 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc9141037a0 0x7fc9141161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:46:27.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.092+0000 7fc913e00700 1 -- 172.21.15.123:0/1512624874 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc9141037a0 msgr2=0x7fc9141161c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:46:27.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.092+0000 7fc913e00700 1 --2- 172.21.15.123:0/1512624874 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc9141037a0 0x7fc9141161c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:27.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.092+0000 7fc913e00700 1 -- 172.21.15.123:0/1512624874 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc900005040 con 0x7fc914102e00 2024-03-30T04:46:27.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.093+0000 7fc913e00700 1 --2- 172.21.15.123:0/1512624874 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc914102e00 0x7fc914115c80 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fc90400eb40 tx=0x7fc90400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:46:27.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.093+0000 7fc90be00700 1 -- 172.21.15.123:0/1512624874 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc904007cc0 con 0x7fc914102e00 2024-03-30T04:46:27.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.093+0000 7fc90be00700 1 -- 172.21.15.123:0/1512624874 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc90400f040 con 0x7fc914102e00 2024-03-30T04:46:27.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.093+0000 7fc91ac00700 1 -- 172.21.15.123:0/1512624874 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc91410fe60 con 0x7fc914102e00 2024-03-30T04:46:27.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.093+0000 7fc90be00700 1 -- 172.21.15.123:0/1512624874 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc90400c990 con 0x7fc914102e00 2024-03-30T04:46:27.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.093+0000 7fc91ac00700 1 -- 172.21.15.123:0/1512624874 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc914110420 con 0x7fc914102e00 2024-03-30T04:46:27.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.094+0000 7fc91ac00700 1 -- 172.21.15.123:0/1512624874 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9140fe550 con 0x7fc914102e00 2024-03-30T04:46:27.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.095+0000 7fc90be00700 1 -- 172.21.15.123:0/1512624874 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fc90400caf0 con 0x7fc914102e00 2024-03-30T04:46:27.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.096+0000 7fc90be00700 1 --2- 172.21.15.123:0/1512624874 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc8f806cc30 0x7fc8f806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:46:27.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.096+0000 7fc90be00700 1 -- 172.21.15.123:0/1512624874 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc90408bb80 con 0x7fc914102e00 2024-03-30T04:46:27.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.096+0000 7fc913400700 1 --2- 172.21.15.123:0/1512624874 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc8f806cc30 0x7fc8f806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:46:27.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.097+0000 7fc913400700 1 --2- 172.21.15.123:0/1512624874 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc8f806cc30 0x7fc8f806f0e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc914111520 tx=0x7fc900009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:46:27.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.100+0000 7fc90be00700 1 -- 172.21.15.123:0/1512624874 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc904059700 con 0x7fc914102e00 2024-03-30T04:46:27.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:46:26 smithi135 ceph-mon[106011]: pgmap v457: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:27.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.311+0000 7fc91ac00700 1 -- 172.21.15.123:0/1512624874 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc91404bf40 con 0x7fc8f806cc30 2024-03-30T04:46:27.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.321+0000 7fc90be00700 1 -- 172.21.15.123:0/1512624874 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fc91404bf40 con 0x7fc8f806cc30 2024-03-30T04:46:27.319 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:46:27.319 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (15m) 8m ago 28m 15.0M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:46:27.319 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (12m) 8m ago 28m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T04:46:27.319 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (12m) 8m ago 27m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T04:46:27.319 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (15m) 8m ago 27m 74.0M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T04:46:27.319 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (8m) 8m ago 21m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T04:46:27.320 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (8m) 8m ago 21m 10.9M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T04:46:27.320 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (21m) 8m ago 21m 10.7M - 17.2.7-797-g5c024335 f77085f6aab7 0dc41326818c 2024-03-30T04:46:27.320 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (8m) 8m ago 21m 9747k - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T04:46:27.320 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (17m) 8m ago 29m 506M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:46:27.320 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (16m) 8m ago 27m 450M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:46:27.320 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (14m) 8m ago 29m 53.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T04:46:27.320 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (13m) 8m ago 26m 42.6M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T04:46:27.320 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (16m) 8m ago 28m 8052k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:46:27.320 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (16m) 8m ago 27m 7880k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:46:27.320 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (12m) 8m ago 26m 104M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T04:46:27.320 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (11m) 8m ago 25m 93.4M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T04:46:27.320 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (11m) 8m ago 24m 86.3M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T04:46:27.320 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (10m) 8m ago 24m 97.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T04:46:27.320 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (9m) 8m ago 23m 97.9M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T04:46:27.321 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (9m) 8m ago 23m 85.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T04:46:27.321 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (15m) 8m ago 27m 53.7M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:46:27.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.323+0000 7fc91ac00700 1 -- 172.21.15.123:0/1512624874 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc8f806cc30 msgr2=0x7fc8f806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:46:27.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.323+0000 7fc91ac00700 1 --2- 172.21.15.123:0/1512624874 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc8f806cc30 0x7fc8f806f0e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc914111520 tx=0x7fc900009e30 comp rx=0 tx=0).stop 2024-03-30T04:46:27.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.323+0000 7fc91ac00700 1 -- 172.21.15.123:0/1512624874 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc914102e00 msgr2=0x7fc914115c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:46:27.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.323+0000 7fc91ac00700 1 --2- 172.21.15.123:0/1512624874 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc914102e00 0x7fc914115c80 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fc90400eb40 tx=0x7fc90400b040 comp rx=0 tx=0).stop 2024-03-30T04:46:27.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.323+0000 7fc91ac00700 1 -- 172.21.15.123:0/1512624874 shutdown_connections 2024-03-30T04:46:27.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.323+0000 7fc91ac00700 1 --2- 172.21.15.123:0/1512624874 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc8f806cc30 0x7fc8f806f0e0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:27.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.323+0000 7fc91ac00700 1 --2- 172.21.15.123:0/1512624874 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc914102e00 0x7fc914115c80 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:27.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.323+0000 7fc91ac00700 1 --2- 172.21.15.123:0/1512624874 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc9141037a0 0x7fc9141161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:27.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.323+0000 7fc91ac00700 1 -- 172.21.15.123:0/1512624874 >> 172.21.15.123:0/1512624874 conn(0x7fc9140f89b0 msgr2=0x7fc9140f99a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:46:27.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.323+0000 7fc91ac00700 1 -- 172.21.15.123:0/1512624874 shutdown_connections 2024-03-30T04:46:27.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.324+0000 7fc91ac00700 1 -- 172.21.15.123:0/1512624874 wait complete. 2024-03-30T04:46:27.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.455+0000 7fc084200700 1 -- 172.21.15.123:0/2121289499 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc07c1037a0 msgr2=0x7fc07c0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:46:27.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.455+0000 7fc084200700 1 --2- 172.21.15.123:0/2121289499 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc07c1037a0 0x7fc07c0fce10 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fc068004940 tx=0x7fc068009e30 comp rx=0 tx=0).stop 2024-03-30T04:46:27.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.456+0000 7fc084200700 1 -- 172.21.15.123:0/2121289499 shutdown_connections 2024-03-30T04:46:27.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.456+0000 7fc084200700 1 --2- 172.21.15.123:0/2121289499 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc07c1037a0 0x7fc07c0fce10 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:27.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.456+0000 7fc084200700 1 --2- 172.21.15.123:0/2121289499 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc07c102e00 0x7fc07c1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:27.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.456+0000 7fc084200700 1 -- 172.21.15.123:0/2121289499 >> 172.21.15.123:0/2121289499 conn(0x7fc07c0f89b0 msgr2=0x7fc07c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:46:27.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.456+0000 7fc084200700 1 -- 172.21.15.123:0/2121289499 shutdown_connections 2024-03-30T04:46:27.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.456+0000 7fc084200700 1 -- 172.21.15.123:0/2121289499 wait complete. 2024-03-30T04:46:27.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.456+0000 7fc084200700 1 Processor -- start 2024-03-30T04:46:27.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.457+0000 7fc084200700 1 -- start start 2024-03-30T04:46:27.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.457+0000 7fc084200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc07c102e00 0x7fc07c115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:46:27.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.457+0000 7fc084200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc07c1037a0 0x7fc07c1161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:46:27.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.457+0000 7fc084200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc07c1167f0 con 0x7fc07c1037a0 2024-03-30T04:46:27.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.457+0000 7fc084200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc07c10fbc0 con 0x7fc07c102e00 2024-03-30T04:46:27.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.457+0000 7fc080e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc07c1037a0 0x7fc07c1161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:46:27.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.457+0000 7fc080e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc07c1037a0 0x7fc07c1161a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50444/0 (socket says 172.21.15.123:50444) 2024-03-30T04:46:27.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.457+0000 7fc080e00700 1 -- 172.21.15.123:0/2253090609 learned_addr learned my addr 172.21.15.123:0/2253090609 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:46:27.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.457+0000 7fc081800700 1 --2- 172.21.15.123:0/2253090609 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc07c102e00 0x7fc07c115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:46:27.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.458+0000 7fc080e00700 1 -- 172.21.15.123:0/2253090609 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc07c102e00 msgr2=0x7fc07c115c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:46:27.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.458+0000 7fc080e00700 1 --2- 172.21.15.123:0/2253090609 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc07c102e00 0x7fc07c115c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:27.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.458+0000 7fc080e00700 1 -- 172.21.15.123:0/2253090609 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc068005040 con 0x7fc07c1037a0 2024-03-30T04:46:27.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.458+0000 7fc080e00700 1 --2- 172.21.15.123:0/2253090609 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc07c1037a0 0x7fc07c1161a0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fc068009390 tx=0x7fc068009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:46:27.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.458+0000 7fc079600700 1 -- 172.21.15.123:0/2253090609 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc06800e040 con 0x7fc07c1037a0 2024-03-30T04:46:27.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.458+0000 7fc079600700 1 -- 172.21.15.123:0/2253090609 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc068013420 con 0x7fc07c1037a0 2024-03-30T04:46:27.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.458+0000 7fc079600700 1 -- 172.21.15.123:0/2253090609 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc06801c630 con 0x7fc07c1037a0 2024-03-30T04:46:27.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.458+0000 7fc084200700 1 -- 172.21.15.123:0/2253090609 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc07c10fe40 con 0x7fc07c1037a0 2024-03-30T04:46:27.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.458+0000 7fc084200700 1 -- 172.21.15.123:0/2253090609 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc07c1103b0 con 0x7fc07c1037a0 2024-03-30T04:46:27.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.459+0000 7fc084200700 1 -- 172.21.15.123:0/2253090609 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc07c0fe550 con 0x7fc07c1037a0 2024-03-30T04:46:27.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.460+0000 7fc079600700 1 -- 172.21.15.123:0/2253090609 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fc068013a00 con 0x7fc07c1037a0 2024-03-30T04:46:27.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.461+0000 7fc079600700 1 --2- 172.21.15.123:0/2253090609 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc06406cc30 0x7fc06406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:46:27.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.461+0000 7fc079600700 1 -- 172.21.15.123:0/2253090609 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc06808d390 con 0x7fc07c1037a0 2024-03-30T04:46:27.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.461+0000 7fc081800700 1 --2- 172.21.15.123:0/2253090609 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc06406cc30 0x7fc06406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:46:27.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.462+0000 7fc081800700 1 --2- 172.21.15.123:0/2253090609 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc06406cc30 0x7fc06406f0e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc070009fd0 tx=0x7fc070009f60 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:46:27.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.465+0000 7fc079600700 1 -- 172.21.15.123:0/2253090609 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc06805ade0 con 0x7fc07c1037a0 2024-03-30T04:46:27.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.753+0000 7fc084200700 1 -- 172.21.15.123:0/2253090609 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc07c04bed0 con 0x7fc07c1037a0 2024-03-30T04:46:27.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.754+0000 7fc079600700 1 -- 172.21.15.123:0/2253090609 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7fc06805a970 con 0x7fc07c1037a0 2024-03-30T04:46:27.752 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:46:27.752 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:46:27.753 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:46:27.753 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:46:27.753 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:46:27.753 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:46:27.753 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:46:27.753 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:46:27.753 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T04:46:27.753 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:46:27.753 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:46:27.753 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:46:27.753 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 3 2024-03-30T04:46:27.753 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:46:27.753 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:46:27.753 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:46:27.753 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 13 2024-03-30T04:46:27.754 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:46:27.754 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:46:27.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.756+0000 7fc084200700 1 -- 172.21.15.123:0/2253090609 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc06406cc30 msgr2=0x7fc06406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:46:27.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.756+0000 7fc084200700 1 --2- 172.21.15.123:0/2253090609 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc06406cc30 0x7fc06406f0e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc070009fd0 tx=0x7fc070009f60 comp rx=0 tx=0).stop 2024-03-30T04:46:27.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.756+0000 7fc084200700 1 -- 172.21.15.123:0/2253090609 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc07c1037a0 msgr2=0x7fc07c1161a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:46:27.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.756+0000 7fc084200700 1 --2- 172.21.15.123:0/2253090609 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc07c1037a0 0x7fc07c1161a0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fc068009390 tx=0x7fc068009e30 comp rx=0 tx=0).stop 2024-03-30T04:46:27.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.757+0000 7fc084200700 1 -- 172.21.15.123:0/2253090609 shutdown_connections 2024-03-30T04:46:27.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.757+0000 7fc084200700 1 --2- 172.21.15.123:0/2253090609 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc06406cc30 0x7fc06406f0e0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:27.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.757+0000 7fc084200700 1 --2- 172.21.15.123:0/2253090609 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc07c102e00 0x7fc07c115c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:27.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.757+0000 7fc084200700 1 --2- 172.21.15.123:0/2253090609 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc07c1037a0 0x7fc07c1161a0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:27.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.757+0000 7fc084200700 1 -- 172.21.15.123:0/2253090609 >> 172.21.15.123:0/2253090609 conn(0x7fc07c0f89b0 msgr2=0x7fc07c0f99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:46:27.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.757+0000 7fc084200700 1 -- 172.21.15.123:0/2253090609 shutdown_connections 2024-03-30T04:46:27.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.757+0000 7fc084200700 1 -- 172.21.15.123:0/2253090609 wait complete. 2024-03-30T04:46:27.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.890+0000 7f2623e00700 1 -- 172.21.15.123:0/201114371 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f261c0708a0 msgr2=0x7f261c070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:46:27.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.890+0000 7f2623e00700 1 --2- 172.21.15.123:0/201114371 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f261c0708a0 0x7f261c070d10 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f2608004970 tx=0x7f2608009e30 comp rx=0 tx=0).stop 2024-03-30T04:46:27.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.890+0000 7f2623e00700 1 -- 172.21.15.123:0/201114371 shutdown_connections 2024-03-30T04:46:27.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.890+0000 7f2623e00700 1 --2- 172.21.15.123:0/201114371 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f261c0708a0 0x7f261c070d10 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:27.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.890+0000 7f2623e00700 1 --2- 172.21.15.123:0/201114371 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f261c071f90 0x7f261c070360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:27.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.890+0000 7f2623e00700 1 -- 172.21.15.123:0/201114371 >> 172.21.15.123:0/201114371 conn(0x7f261c0f89b0 msgr2=0x7f261c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:46:27.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.890+0000 7f2623e00700 1 -- 172.21.15.123:0/201114371 shutdown_connections 2024-03-30T04:46:27.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.890+0000 7f2623e00700 1 -- 172.21.15.123:0/201114371 wait complete. 2024-03-30T04:46:27.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.891+0000 7f2623e00700 1 Processor -- start 2024-03-30T04:46:27.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.891+0000 7f2623e00700 1 -- start start 2024-03-30T04:46:27.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.891+0000 7f2623e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f261c0708a0 0x7f261c11a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:46:27.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.891+0000 7f2623e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f261c071f90 0x7f261c11a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:46:27.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.891+0000 7f2623e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f261c11ab90 con 0x7f261c0708a0 2024-03-30T04:46:27.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.891+0000 7f2623e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f261c114010 con 0x7f261c071f90 2024-03-30T04:46:27.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.891+0000 7f2621400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f261c0708a0 0x7f261c11a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:46:27.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.891+0000 7f2621400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f261c0708a0 0x7f261c11a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50474/0 (socket says 172.21.15.123:50474) 2024-03-30T04:46:27.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.891+0000 7f2621400700 1 -- 172.21.15.123:0/2881710724 learned_addr learned my addr 172.21.15.123:0/2881710724 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:46:27.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.892+0000 7f2621400700 1 -- 172.21.15.123:0/2881710724 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f261c071f90 msgr2=0x7f261c11a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:46:27.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.892+0000 7f2621400700 1 --2- 172.21.15.123:0/2881710724 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f261c071f90 0x7f261c11a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:27.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.892+0000 7f2621400700 1 -- 172.21.15.123:0/2881710724 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2608005040 con 0x7f261c0708a0 2024-03-30T04:46:27.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.892+0000 7f2621400700 1 --2- 172.21.15.123:0/2881710724 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f261c0708a0 0x7f261c11a000 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f2610002a50 tx=0x7f26100046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:46:27.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.892+0000 7f2619600700 1 -- 172.21.15.123:0/2881710724 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2610007cc0 con 0x7f261c0708a0 2024-03-30T04:46:27.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.892+0000 7f2619600700 1 -- 172.21.15.123:0/2881710724 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f261000f040 con 0x7f261c0708a0 2024-03-30T04:46:27.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.892+0000 7f2623e00700 1 -- 172.21.15.123:0/2881710724 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f261c1142f0 con 0x7f261c0708a0 2024-03-30T04:46:27.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.892+0000 7f2619600700 1 -- 172.21.15.123:0/2881710724 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2610009540 con 0x7f261c0708a0 2024-03-30T04:46:27.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.892+0000 7f2623e00700 1 -- 172.21.15.123:0/2881710724 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f261c114830 con 0x7f261c0708a0 2024-03-30T04:46:27.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.893+0000 7f2623e00700 1 -- 172.21.15.123:0/2881710724 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f261c071aa0 con 0x7f261c0708a0 2024-03-30T04:46:27.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.894+0000 7f2619600700 1 -- 172.21.15.123:0/2881710724 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f26100096a0 con 0x7f261c0708a0 2024-03-30T04:46:27.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.896+0000 7f2619600700 1 --2- 172.21.15.123:0/2881710724 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f260406cc30 0x7f260406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:46:27.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.896+0000 7f2619600700 1 -- 172.21.15.123:0/2881710724 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f261008bda0 con 0x7f261c0708a0 2024-03-30T04:46:27.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.896+0000 7f2620a00700 1 --2- 172.21.15.123:0/2881710724 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f260406cc30 0x7f260406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:46:27.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.896+0000 7f2620a00700 1 --2- 172.21.15.123:0/2881710724 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f260406cc30 0x7f260406f0e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2608000c00 tx=0x7f2608009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:46:27.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:27.900+0000 7f2619600700 1 -- 172.21.15.123:0/2881710724 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2610055da0 con 0x7f261c0708a0 2024-03-30T04:46:27.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:46:27 smithi123 ceph-mon[139369]: from='client.34878 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:46:27.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:46:27 smithi123 ceph-mon[139369]: from='client.34882 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:46:28.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:46:27 smithi135 ceph-mon[106011]: from='client.34878 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:46:28.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:46:27 smithi135 ceph-mon[106011]: from='client.34882 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:46:28.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.147+0000 7f2623e00700 1 -- 172.21.15.123:0/2881710724 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f261c04bed0 con 0x7f261c0708a0 2024-03-30T04:46:28.146 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.147+0000 7f2619600700 1 -- 172.21.15.123:0/2881710724 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f26100593c0 con 0x7f261c0708a0 2024-03-30T04:46:28.146 INFO:teuthology.orchestra.run.smithi123.stdout:e32 2024-03-30T04:46:28.146 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:46:28.146 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:46:28.146 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:46:28.146 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:46:28.146 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:46:28.146 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:46:28.146 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T04:46:28.146 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T04:46:28.147 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:46:28.147 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T04:46:28.147 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:46:28.147 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:46:28.147 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:46:28.147 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:46:28.147 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:46:28.147 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T04:46:28.147 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:46:28.147 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:46:28.147 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T04:46:28.147 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:46:28.147 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T04:46:28.147 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T04:46:28.148 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T04:46:28.148 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T04:46:28.148 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:46:28.148 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:46:28.148 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:46:28.148 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:46:28.148 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:46:28.148 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:46:28.148 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T04:46:28.148 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T04:46:28.148 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:46:28.149 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:46:28.149 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T04:46:28.149 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:46:28.149 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:46:28.149 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:46:28.149 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44315} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/1741352336,v1:172.21.15.135:6827/1741352336] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:46:28.149 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:46:28.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.150+0000 7f2623e00700 1 -- 172.21.15.123:0/2881710724 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f260406cc30 msgr2=0x7f260406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:46:28.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.150+0000 7f2623e00700 1 --2- 172.21.15.123:0/2881710724 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f260406cc30 0x7f260406f0e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2608000c00 tx=0x7f2608009e30 comp rx=0 tx=0).stop 2024-03-30T04:46:28.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.150+0000 7f2623e00700 1 -- 172.21.15.123:0/2881710724 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f261c0708a0 msgr2=0x7f261c11a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:46:28.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.150+0000 7f2623e00700 1 --2- 172.21.15.123:0/2881710724 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f261c0708a0 0x7f261c11a000 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f2610002a50 tx=0x7f26100046b0 comp rx=0 tx=0).stop 2024-03-30T04:46:28.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.150+0000 7f2623e00700 1 -- 172.21.15.123:0/2881710724 shutdown_connections 2024-03-30T04:46:28.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.150+0000 7f2623e00700 1 --2- 172.21.15.123:0/2881710724 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f260406cc30 0x7f260406f0e0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:28.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.150+0000 7f2623e00700 1 --2- 172.21.15.123:0/2881710724 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f261c0708a0 0x7f261c11a000 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:28.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.150+0000 7f2623e00700 1 --2- 172.21.15.123:0/2881710724 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f261c071f90 0x7f261c11a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:28.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.150+0000 7f2623e00700 1 -- 172.21.15.123:0/2881710724 >> 172.21.15.123:0/2881710724 conn(0x7f261c0f89b0 msgr2=0x7f261c0f9b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:46:28.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.150+0000 7f2623e00700 1 -- 172.21.15.123:0/2881710724 shutdown_connections 2024-03-30T04:46:28.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.150+0000 7f2623e00700 1 -- 172.21.15.123:0/2881710724 wait complete. 2024-03-30T04:46:28.151 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 32 2024-03-30T04:46:28.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.284+0000 7fc5f8800700 1 -- 172.21.15.123:0/326461201 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5f0102e00 msgr2=0x7fc5f01031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:46:28.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.284+0000 7fc5f8800700 1 --2- 172.21.15.123:0/326461201 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5f0102e00 0x7fc5f01031d0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fc5e4004940 tx=0x7fc5e4009e30 comp rx=0 tx=0).stop 2024-03-30T04:46:28.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.284+0000 7fc5f8800700 1 -- 172.21.15.123:0/326461201 shutdown_connections 2024-03-30T04:46:28.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.284+0000 7fc5f8800700 1 --2- 172.21.15.123:0/326461201 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc5f01037a0 0x7fc5f00fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:28.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.284+0000 7fc5f8800700 1 --2- 172.21.15.123:0/326461201 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5f0102e00 0x7fc5f01031d0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:28.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.284+0000 7fc5f8800700 1 -- 172.21.15.123:0/326461201 >> 172.21.15.123:0/326461201 conn(0x7fc5f00f89b0 msgr2=0x7fc5f00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:46:28.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.284+0000 7fc5f8800700 1 -- 172.21.15.123:0/326461201 shutdown_connections 2024-03-30T04:46:28.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.284+0000 7fc5f8800700 1 -- 172.21.15.123:0/326461201 wait complete. 2024-03-30T04:46:28.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.285+0000 7fc5f8800700 1 Processor -- start 2024-03-30T04:46:28.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.285+0000 7fc5f8800700 1 -- start start 2024-03-30T04:46:28.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.285+0000 7fc5f8800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc5f0102e00 0x7fc5f0115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:46:28.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.285+0000 7fc5f8800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5f01037a0 0x7fc5f0116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:46:28.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.285+0000 7fc5f8800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f0116880 con 0x7fc5f01037a0 2024-03-30T04:46:28.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.285+0000 7fc5f8800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f010fc50 con 0x7fc5f0102e00 2024-03-30T04:46:28.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.286+0000 7fc5f5400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5f01037a0 0x7fc5f0116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:46:28.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.286+0000 7fc5f5400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5f01037a0 0x7fc5f0116230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50488/0 (socket says 172.21.15.123:50488) 2024-03-30T04:46:28.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.286+0000 7fc5f5400700 1 -- 172.21.15.123:0/3694289433 learned_addr learned my addr 172.21.15.123:0/3694289433 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:46:28.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.286+0000 7fc5f5e00700 1 --2- 172.21.15.123:0/3694289433 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc5f0102e00 0x7fc5f0115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:46:28.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.286+0000 7fc5f5400700 1 -- 172.21.15.123:0/3694289433 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc5f0102e00 msgr2=0x7fc5f0115cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:46:28.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.286+0000 7fc5f5400700 1 --2- 172.21.15.123:0/3694289433 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc5f0102e00 0x7fc5f0115cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:28.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.286+0000 7fc5f5400700 1 -- 172.21.15.123:0/3694289433 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e4005040 con 0x7fc5f01037a0 2024-03-30T04:46:28.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.286+0000 7fc5f5400700 1 --2- 172.21.15.123:0/3694289433 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5f01037a0 0x7fc5f0116230 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fc5e0002a50 tx=0x7fc5e0004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:46:28.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.287+0000 7fc5ea000700 1 -- 172.21.15.123:0/3694289433 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc5e0007cc0 con 0x7fc5f01037a0 2024-03-30T04:46:28.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.287+0000 7fc5ea000700 1 -- 172.21.15.123:0/3694289433 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc5e000f040 con 0x7fc5f01037a0 2024-03-30T04:46:28.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.287+0000 7fc5f8800700 1 -- 172.21.15.123:0/3694289433 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5f010ff30 con 0x7fc5f01037a0 2024-03-30T04:46:28.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.287+0000 7fc5ea000700 1 -- 172.21.15.123:0/3694289433 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc5e000d920 con 0x7fc5f01037a0 2024-03-30T04:46:28.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.287+0000 7fc5f8800700 1 -- 172.21.15.123:0/3694289433 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5f01104f0 con 0x7fc5f01037a0 2024-03-30T04:46:28.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.288+0000 7fc5f8800700 1 -- 172.21.15.123:0/3694289433 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5f00fe550 con 0x7fc5f01037a0 2024-03-30T04:46:28.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.289+0000 7fc5ea000700 1 -- 172.21.15.123:0/3694289433 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fc5e0009680 con 0x7fc5f01037a0 2024-03-30T04:46:28.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.290+0000 7fc5ea000700 1 --2- 172.21.15.123:0/3694289433 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc5d806cc20 0x7fc5d806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:46:28.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.290+0000 7fc5ea000700 1 -- 172.21.15.123:0/3694289433 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc5e008bd80 con 0x7fc5f01037a0 2024-03-30T04:46:28.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.290+0000 7fc5f5e00700 1 --2- 172.21.15.123:0/3694289433 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc5d806cc20 0x7fc5d806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:46:28.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.290+0000 7fc5f5e00700 1 --2- 172.21.15.123:0/3694289433 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc5d806cc20 0x7fc5d806f0d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fc5e4004f90 tx=0x7fc5e4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:46:28.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.294+0000 7fc5ea000700 1 -- 172.21.15.123:0/3694289433 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc5e00597d0 con 0x7fc5f01037a0 2024-03-30T04:46:28.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.515+0000 7fc5f8800700 1 -- 172.21.15.123:0/3694289433 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc5f0109e30 con 0x7fc5d806cc20 2024-03-30T04:46:28.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.516+0000 7fc5ea000700 1 -- 172.21.15.123:0/3694289433 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc5f0109e30 con 0x7fc5d806cc20 2024-03-30T04:46:28.515 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:46:28.515 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:46:28.515 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:46:28.515 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T04:46:28.515 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:46:28.515 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T04:46:28.515 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T04:46:28.515 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T04:46:28.515 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T04:46:28.515 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:46:28.515 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "15/21 daemons upgraded", 2024-03-30T04:46:28.515 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T04:46:28.516 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:46:28.516 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:46:28.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.519+0000 7fc5f8800700 1 -- 172.21.15.123:0/3694289433 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc5d806cc20 msgr2=0x7fc5d806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:46:28.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.519+0000 7fc5f8800700 1 --2- 172.21.15.123:0/3694289433 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc5d806cc20 0x7fc5d806f0d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fc5e4004f90 tx=0x7fc5e4009e30 comp rx=0 tx=0).stop 2024-03-30T04:46:28.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.519+0000 7fc5f8800700 1 -- 172.21.15.123:0/3694289433 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5f01037a0 msgr2=0x7fc5f0116230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:46:28.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.519+0000 7fc5f8800700 1 --2- 172.21.15.123:0/3694289433 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5f01037a0 0x7fc5f0116230 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fc5e0002a50 tx=0x7fc5e0004600 comp rx=0 tx=0).stop 2024-03-30T04:46:28.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.519+0000 7fc5f8800700 1 -- 172.21.15.123:0/3694289433 shutdown_connections 2024-03-30T04:46:28.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.519+0000 7fc5f8800700 1 --2- 172.21.15.123:0/3694289433 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc5d806cc20 0x7fc5d806f0d0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:28.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.519+0000 7fc5f8800700 1 --2- 172.21.15.123:0/3694289433 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc5f0102e00 0x7fc5f0115cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:28.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.519+0000 7fc5f8800700 1 --2- 172.21.15.123:0/3694289433 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5f01037a0 0x7fc5f0116230 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:28.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.519+0000 7fc5f8800700 1 -- 172.21.15.123:0/3694289433 >> 172.21.15.123:0/3694289433 conn(0x7fc5f00f89b0 msgr2=0x7fc5f00706d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:46:28.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.519+0000 7fc5f8800700 1 -- 172.21.15.123:0/3694289433 shutdown_connections 2024-03-30T04:46:28.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.519+0000 7fc5f8800700 1 -- 172.21.15.123:0/3694289433 wait complete. 2024-03-30T04:46:28.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.653+0000 7f4d54600700 1 -- 172.21.15.123:0/4127619288 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d4c102e00 msgr2=0x7f4d4c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:46:28.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.653+0000 7f4d54600700 1 --2- 172.21.15.123:0/4127619288 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d4c102e00 0x7f4d4c1031d0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f4d40004940 tx=0x7f4d40009e30 comp rx=0 tx=0).stop 2024-03-30T04:46:28.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.653+0000 7f4d54600700 1 -- 172.21.15.123:0/4127619288 shutdown_connections 2024-03-30T04:46:28.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.653+0000 7f4d54600700 1 --2- 172.21.15.123:0/4127619288 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4d4c1037a0 0x7f4d4c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:28.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.653+0000 7f4d54600700 1 --2- 172.21.15.123:0/4127619288 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d4c102e00 0x7f4d4c1031d0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:28.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.653+0000 7f4d54600700 1 -- 172.21.15.123:0/4127619288 >> 172.21.15.123:0/4127619288 conn(0x7f4d4c0f89b0 msgr2=0x7f4d4c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:46:28.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.653+0000 7f4d54600700 1 -- 172.21.15.123:0/4127619288 shutdown_connections 2024-03-30T04:46:28.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.653+0000 7f4d54600700 1 -- 172.21.15.123:0/4127619288 wait complete. 2024-03-30T04:46:28.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.654+0000 7f4d54600700 1 Processor -- start 2024-03-30T04:46:28.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.654+0000 7f4d54600700 1 -- start start 2024-03-30T04:46:28.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.654+0000 7f4d54600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4d4c102e00 0x7f4d4c115c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:46:28.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.654+0000 7f4d54600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d4c1037a0 0x7f4d4c1161d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:46:28.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.655+0000 7f4d54600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4c116820 con 0x7f4d4c1037a0 2024-03-30T04:46:28.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.655+0000 7f4d54600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4c10fbf0 con 0x7f4d4c102e00 2024-03-30T04:46:28.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.655+0000 7f4d51200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d4c1037a0 0x7f4d4c1161d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:46:28.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.655+0000 7f4d51200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d4c1037a0 0x7f4d4c1161d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50508/0 (socket says 172.21.15.123:50508) 2024-03-30T04:46:28.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.655+0000 7f4d51200700 1 -- 172.21.15.123:0/3065692292 learned_addr learned my addr 172.21.15.123:0/3065692292 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:46:28.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.655+0000 7f4d51c00700 1 --2- 172.21.15.123:0/3065692292 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4d4c102e00 0x7f4d4c115c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:46:28.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.655+0000 7f4d51200700 1 -- 172.21.15.123:0/3065692292 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4d4c102e00 msgr2=0x7f4d4c115c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:46:28.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.655+0000 7f4d51200700 1 --2- 172.21.15.123:0/3065692292 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4d4c102e00 0x7f4d4c115c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:28.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.655+0000 7f4d51200700 1 -- 172.21.15.123:0/3065692292 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d40005040 con 0x7f4d4c1037a0 2024-03-30T04:46:28.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.655+0000 7f4d51200700 1 --2- 172.21.15.123:0/3065692292 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d4c1037a0 0x7f4d4c1161d0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f4d3800eb40 tx=0x7f4d3800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:46:28.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.656+0000 7f4d49600700 1 -- 172.21.15.123:0/3065692292 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d38007cc0 con 0x7f4d4c1037a0 2024-03-30T04:46:28.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.656+0000 7f4d49600700 1 -- 172.21.15.123:0/3065692292 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4d3800f040 con 0x7f4d4c1037a0 2024-03-30T04:46:28.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.656+0000 7f4d54600700 1 -- 172.21.15.123:0/3065692292 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d4c10fe70 con 0x7f4d4c1037a0 2024-03-30T04:46:28.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.656+0000 7f4d49600700 1 -- 172.21.15.123:0/3065692292 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d3800ca30 con 0x7f4d4c1037a0 2024-03-30T04:46:28.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.656+0000 7f4d54600700 1 -- 172.21.15.123:0/3065692292 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d4c110430 con 0x7f4d4c1037a0 2024-03-30T04:46:28.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.657+0000 7f4d54600700 1 -- 172.21.15.123:0/3065692292 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d4c0fe550 con 0x7f4d4c1037a0 2024-03-30T04:46:28.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.659+0000 7f4d49600700 1 -- 172.21.15.123:0/3065692292 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f4d3800cb90 con 0x7f4d4c1037a0 2024-03-30T04:46:28.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.660+0000 7f4d49600700 1 --2- 172.21.15.123:0/3065692292 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4d3406cc30 0x7f4d3406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:46:28.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.660+0000 7f4d49600700 1 -- 172.21.15.123:0/3065692292 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f4d3808bf20 con 0x7f4d4c1037a0 2024-03-30T04:46:28.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.660+0000 7f4d51c00700 1 --2- 172.21.15.123:0/3065692292 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4d3406cc30 0x7f4d3406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:46:28.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.661+0000 7f4d51c00700 1 --2- 172.21.15.123:0/3065692292 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4d3406cc30 0x7f4d3406f0e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f4d400045a0 tx=0x7f4d40009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:46:28.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.663+0000 7f4d49600700 1 -- 172.21.15.123:0/3065692292 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4d38059970 con 0x7f4d4c1037a0 2024-03-30T04:46:28.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.943+0000 7f4d54600700 1 -- 172.21.15.123:0/3065692292 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f4d4c05f5f0 con 0x7f4d4c1037a0 2024-03-30T04:46:28.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.944+0000 7f4d49600700 1 -- 172.21.15.123:0/3065692292 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f4d38019020 con 0x7f4d4c1037a0 2024-03-30T04:46:28.942 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T04:46:28.942 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T04:46:28.942 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T04:46:28.943 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T04:46:28.943 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T04:46:28.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.946+0000 7f4d54600700 1 -- 172.21.15.123:0/3065692292 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4d3406cc30 msgr2=0x7f4d3406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:46:28.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.946+0000 7f4d54600700 1 --2- 172.21.15.123:0/3065692292 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4d3406cc30 0x7f4d3406f0e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f4d400045a0 tx=0x7f4d40009e30 comp rx=0 tx=0).stop 2024-03-30T04:46:28.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.946+0000 7f4d54600700 1 -- 172.21.15.123:0/3065692292 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d4c1037a0 msgr2=0x7f4d4c1161d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:46:28.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.946+0000 7f4d54600700 1 --2- 172.21.15.123:0/3065692292 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d4c1037a0 0x7f4d4c1161d0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f4d3800eb40 tx=0x7f4d3800b040 comp rx=0 tx=0).stop 2024-03-30T04:46:28.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.946+0000 7f4d54600700 1 -- 172.21.15.123:0/3065692292 shutdown_connections 2024-03-30T04:46:28.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.946+0000 7f4d54600700 1 --2- 172.21.15.123:0/3065692292 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4d3406cc30 0x7f4d3406f0e0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:28.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.946+0000 7f4d54600700 1 --2- 172.21.15.123:0/3065692292 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4d4c102e00 0x7f4d4c115c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:28.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.946+0000 7f4d54600700 1 --2- 172.21.15.123:0/3065692292 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d4c1037a0 0x7f4d4c1161d0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:28.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.946+0000 7f4d54600700 1 -- 172.21.15.123:0/3065692292 >> 172.21.15.123:0/3065692292 conn(0x7f4d4c0f89b0 msgr2=0x7f4d4c0f9960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:46:28.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.947+0000 7f4d54600700 1 -- 172.21.15.123:0/3065692292 shutdown_connections 2024-03-30T04:46:28.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:28.947+0000 7f4d54600700 1 -- 172.21.15.123:0/3065692292 wait complete. 2024-03-30T04:46:28.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:46:28 smithi123 ceph-mon[139369]: from='client.34886 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:46:28.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:46:28 smithi123 ceph-mon[139369]: pgmap v458: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:28.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:46:28 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2253090609' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:46:28.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:46:28 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2881710724' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:46:29.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:46:28 smithi135 ceph-mon[106011]: from='client.34886 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:46:29.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:46:28 smithi135 ceph-mon[106011]: pgmap v458: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:29.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:46:28 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2253090609' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:46:29.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:46:28 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2881710724' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:46:29.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:46:29 smithi123 ceph-mon[139369]: from='client.34898 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:46:29.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:46:29 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3065692292' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:46:29.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:46:29 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:46:30.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:46:29 smithi135 ceph-mon[106011]: from='client.34898 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:46:30.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:46:29 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3065692292' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:46:30.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:46:29 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:46:30.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:46:30 smithi123 ceph-mon[139369]: pgmap v459: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:31.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:46:30 smithi135 ceph-mon[106011]: pgmap v459: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:32.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:46:32 smithi123 ceph-mon[139369]: pgmap v460: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:33.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:46:32 smithi135 ceph-mon[106011]: pgmap v460: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:34.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:46:34 smithi123 ceph-mon[139369]: pgmap v461: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:35.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:46:34 smithi135 ceph-mon[106011]: pgmap v461: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:36.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:46:36 smithi123 ceph-mon[139369]: pgmap v462: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:37.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:46:36 smithi135 ceph-mon[106011]: pgmap v462: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:38.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:46:38 smithi123 ceph-mon[139369]: pgmap v463: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:39.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:46:38 smithi135 ceph-mon[106011]: pgmap v463: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:40.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:46:40 smithi135 ceph-mon[106011]: pgmap v464: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:40.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:46:40 smithi123 ceph-mon[139369]: pgmap v464: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:42.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:46:42 smithi123 ceph-mon[139369]: pgmap v465: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:43.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:46:42 smithi135 ceph-mon[106011]: pgmap v465: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:44.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:46:44 smithi123 ceph-mon[139369]: pgmap v466: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:44.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:46:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:46:45.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:46:44 smithi135 ceph-mon[106011]: pgmap v466: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:45.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:46:44 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:46:46.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:46:46 smithi123 ceph-mon[139369]: pgmap v467: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:47.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:46:46 smithi135 ceph-mon[106011]: pgmap v467: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:48.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:46:48 smithi123 ceph-mon[139369]: pgmap v468: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:49.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:46:48 smithi135 ceph-mon[106011]: pgmap v468: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:50.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:46:50 smithi135 ceph-mon[106011]: pgmap v469: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:50.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:46:50 smithi123 ceph-mon[139369]: pgmap v469: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:52.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:46:52 smithi123 ceph-mon[139369]: pgmap v470: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:53.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:46:52 smithi135 ceph-mon[106011]: pgmap v470: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:54.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:46:54 smithi123 ceph-mon[139369]: pgmap v471: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:55.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:46:54 smithi135 ceph-mon[106011]: pgmap v471: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:56.902 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:46:56 smithi135 ceph-mon[106011]: pgmap v472: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:56.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:46:56 smithi123 ceph-mon[139369]: pgmap v472: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:58.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:46:58 smithi123 ceph-mon[139369]: pgmap v473: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:59.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.094+0000 7ffbd9800700 1 -- 172.21.15.123:0/446215199 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffbd4105990 msgr2=0x7ffbd40ff000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:46:59.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.094+0000 7ffbd9800700 1 --2- 172.21.15.123:0/446215199 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffbd4105990 0x7ffbd40ff000 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7ffbc0004940 tx=0x7ffbc0009e30 comp rx=0 tx=0).stop 2024-03-30T04:46:59.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.094+0000 7ffbd9800700 1 -- 172.21.15.123:0/446215199 shutdown_connections 2024-03-30T04:46:59.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.094+0000 7ffbd9800700 1 --2- 172.21.15.123:0/446215199 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffbd4105990 0x7ffbd40ff000 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:59.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.094+0000 7ffbd9800700 1 --2- 172.21.15.123:0/446215199 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffbd4104ff0 0x7ffbd41053c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:59.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.094+0000 7ffbd9800700 1 -- 172.21.15.123:0/446215199 >> 172.21.15.123:0/446215199 conn(0x7ffbd40faba0 msgr2=0x7ffbd40fcfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:46:59.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.094+0000 7ffbd9800700 1 -- 172.21.15.123:0/446215199 shutdown_connections 2024-03-30T04:46:59.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.094+0000 7ffbd9800700 1 -- 172.21.15.123:0/446215199 wait complete. 2024-03-30T04:46:59.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.095+0000 7ffbd9800700 1 Processor -- start 2024-03-30T04:46:59.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.095+0000 7ffbd9800700 1 -- start start 2024-03-30T04:46:59.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.096+0000 7ffbd9800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffbd4104ff0 0x7ffbd4115b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:46:59.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.096+0000 7ffbd9800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffbd4105990 0x7ffbd41160a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:46:59.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.096+0000 7ffbd9800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd41166f0 con 0x7ffbd4105990 2024-03-30T04:46:59.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.096+0000 7ffbd9800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd410fcd0 con 0x7ffbd4104ff0 2024-03-30T04:46:59.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.096+0000 7ffbd2000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffbd4105990 0x7ffbd41160a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:46:59.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.096+0000 7ffbd2000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffbd4105990 0x7ffbd41160a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37974/0 (socket says 172.21.15.123:37974) 2024-03-30T04:46:59.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.096+0000 7ffbd2000700 1 -- 172.21.15.123:0/4025750241 learned_addr learned my addr 172.21.15.123:0/4025750241 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:46:59.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.096+0000 7ffbd2a00700 1 --2- 172.21.15.123:0/4025750241 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffbd4104ff0 0x7ffbd4115b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:46:59.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.096+0000 7ffbd2000700 1 -- 172.21.15.123:0/4025750241 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffbd4104ff0 msgr2=0x7ffbd4115b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:46:59.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.096+0000 7ffbd2000700 1 --2- 172.21.15.123:0/4025750241 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffbd4104ff0 0x7ffbd4115b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:59.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.096+0000 7ffbd2000700 1 -- 172.21.15.123:0/4025750241 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbc0005040 con 0x7ffbd4105990 2024-03-30T04:46:59.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.097+0000 7ffbd2000700 1 --2- 172.21.15.123:0/4025750241 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffbd4105990 0x7ffbd41160a0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7ffbc0009390 tx=0x7ffbc0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:46:59.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.097+0000 7ffbcaa00700 1 -- 172.21.15.123:0/4025750241 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffbc000e040 con 0x7ffbd4105990 2024-03-30T04:46:59.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.097+0000 7ffbcaa00700 1 -- 172.21.15.123:0/4025750241 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffbc0013420 con 0x7ffbd4105990 2024-03-30T04:46:59.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.097+0000 7ffbd9800700 1 -- 172.21.15.123:0/4025750241 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbd410ff50 con 0x7ffbd4105990 2024-03-30T04:46:59.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.097+0000 7ffbcaa00700 1 -- 172.21.15.123:0/4025750241 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffbc001c630 con 0x7ffbd4105990 2024-03-30T04:46:59.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.097+0000 7ffbd9800700 1 -- 172.21.15.123:0/4025750241 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbd4110490 con 0x7ffbd4105990 2024-03-30T04:46:59.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.098+0000 7ffbd9800700 1 -- 172.21.15.123:0/4025750241 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbd4100740 con 0x7ffbd4105990 2024-03-30T04:46:59.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.099+0000 7ffbcaa00700 1 -- 172.21.15.123:0/4025750241 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7ffbc0013a00 con 0x7ffbd4105990 2024-03-30T04:46:59.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.100+0000 7ffbcaa00700 1 --2- 172.21.15.123:0/4025750241 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ffbb806cc20 0x7ffbb806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:46:59.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.100+0000 7ffbcaa00700 1 -- 172.21.15.123:0/4025750241 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ffbc008d120 con 0x7ffbd4105990 2024-03-30T04:46:59.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.100+0000 7ffbd2a00700 1 --2- 172.21.15.123:0/4025750241 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ffbb806cc20 0x7ffbb806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:46:59.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.101+0000 7ffbd2a00700 1 --2- 172.21.15.123:0/4025750241 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ffbb806cc20 0x7ffbb806f0d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ffbc4004f60 tx=0x7ffbc400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:46:59.102 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.105+0000 7ffbcaa00700 1 -- 172.21.15.123:0/4025750241 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ffbc005abf0 con 0x7ffbd4105990 2024-03-30T04:46:59.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:46:58 smithi135 ceph-mon[106011]: pgmap v473: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:46:59.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.326+0000 7ffbd9800700 1 -- 172.21.15.123:0/4025750241 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ffbd410bfb0 con 0x7ffbb806cc20 2024-03-30T04:46:59.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.327+0000 7ffbcaa00700 1 -- 172.21.15.123:0/4025750241 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ffbd410bfb0 con 0x7ffbb806cc20 2024-03-30T04:46:59.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.329+0000 7ffbd9800700 1 -- 172.21.15.123:0/4025750241 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ffbb806cc20 msgr2=0x7ffbb806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:46:59.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.329+0000 7ffbd9800700 1 --2- 172.21.15.123:0/4025750241 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ffbb806cc20 0x7ffbb806f0d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ffbc4004f60 tx=0x7ffbc400b040 comp rx=0 tx=0).stop 2024-03-30T04:46:59.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.329+0000 7ffbd9800700 1 -- 172.21.15.123:0/4025750241 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffbd4105990 msgr2=0x7ffbd41160a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:46:59.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.329+0000 7ffbd9800700 1 --2- 172.21.15.123:0/4025750241 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffbd4105990 0x7ffbd41160a0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7ffbc0009390 tx=0x7ffbc0009e30 comp rx=0 tx=0).stop 2024-03-30T04:46:59.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.330+0000 7ffbd9800700 1 -- 172.21.15.123:0/4025750241 shutdown_connections 2024-03-30T04:46:59.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.330+0000 7ffbd9800700 1 --2- 172.21.15.123:0/4025750241 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ffbb806cc20 0x7ffbb806f0d0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:59.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.330+0000 7ffbd9800700 1 --2- 172.21.15.123:0/4025750241 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffbd4104ff0 0x7ffbd4115b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:59.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.330+0000 7ffbd9800700 1 --2- 172.21.15.123:0/4025750241 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffbd4105990 0x7ffbd41160a0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:59.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.330+0000 7ffbd9800700 1 -- 172.21.15.123:0/4025750241 >> 172.21.15.123:0/4025750241 conn(0x7ffbd40faba0 msgr2=0x7ffbd40fca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:46:59.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.330+0000 7ffbd9800700 1 -- 172.21.15.123:0/4025750241 shutdown_connections 2024-03-30T04:46:59.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.330+0000 7ffbd9800700 1 -- 172.21.15.123:0/4025750241 wait complete. 2024-03-30T04:46:59.341 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:46:59.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.465+0000 7f73f5a00700 1 -- 172.21.15.123:0/3933199440 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73f00fdca0 msgr2=0x7f73f00fe070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:46:59.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.465+0000 7f73f5a00700 1 --2- 172.21.15.123:0/3933199440 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73f00fdca0 0x7f73f00fe070 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f73e0004940 tx=0x7f73e0009e30 comp rx=0 tx=0).stop 2024-03-30T04:46:59.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.465+0000 7f73f5a00700 1 -- 172.21.15.123:0/3933199440 shutdown_connections 2024-03-30T04:46:59.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.465+0000 7f73f5a00700 1 --2- 172.21.15.123:0/3933199440 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f73f00fe5b0 0x7f73f0106c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:59.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.465+0000 7f73f5a00700 1 --2- 172.21.15.123:0/3933199440 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73f00fdca0 0x7f73f00fe070 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:59.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.465+0000 7f73f5a00700 1 -- 172.21.15.123:0/3933199440 >> 172.21.15.123:0/3933199440 conn(0x7f73f00f7670 msgr2=0x7f73f00f9a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:46:59.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.465+0000 7f73f5a00700 1 -- 172.21.15.123:0/3933199440 shutdown_connections 2024-03-30T04:46:59.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.465+0000 7f73f5a00700 1 -- 172.21.15.123:0/3933199440 wait complete. 2024-03-30T04:46:59.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.466+0000 7f73f5a00700 1 Processor -- start 2024-03-30T04:46:59.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.466+0000 7f73f5a00700 1 -- start start 2024-03-30T04:46:59.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.466+0000 7f73f5a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73f00fdca0 0x7f73f01138e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:46:59.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.466+0000 7f73f5a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f73f00fe5b0 0x7f73f0113e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:46:59.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.466+0000 7f73f5a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73f0114470 con 0x7f73f00fdca0 2024-03-30T04:46:59.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.466+0000 7f73f5a00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73f010d860 con 0x7f73f00fe5b0 2024-03-30T04:46:59.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.467+0000 7f73efe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73f00fdca0 0x7f73f01138e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:46:59.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.467+0000 7f73efe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73f00fdca0 0x7f73f01138e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37996/0 (socket says 172.21.15.123:37996) 2024-03-30T04:46:59.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.467+0000 7f73efe00700 1 -- 172.21.15.123:0/3832065238 learned_addr learned my addr 172.21.15.123:0/3832065238 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:46:59.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.467+0000 7f73ef400700 1 --2- 172.21.15.123:0/3832065238 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f73f00fe5b0 0x7f73f0113e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:46:59.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.467+0000 7f73efe00700 1 -- 172.21.15.123:0/3832065238 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f73f00fe5b0 msgr2=0x7f73f0113e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:46:59.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.467+0000 7f73efe00700 1 --2- 172.21.15.123:0/3832065238 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f73f00fe5b0 0x7f73f0113e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:59.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.467+0000 7f73efe00700 1 -- 172.21.15.123:0/3832065238 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e0005040 con 0x7f73f00fdca0 2024-03-30T04:46:59.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.467+0000 7f73ef400700 1 --2- 172.21.15.123:0/3832065238 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f73f00fe5b0 0x7f73f0113e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:46:59.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.467+0000 7f73efe00700 1 --2- 172.21.15.123:0/3832065238 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73f00fdca0 0x7f73f01138e0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f73e0009390 tx=0x7f73e0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:46:59.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.467+0000 7f73e7e00700 1 -- 172.21.15.123:0/3832065238 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f73e000e040 con 0x7f73f00fdca0 2024-03-30T04:46:59.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.468+0000 7f73e7e00700 1 -- 172.21.15.123:0/3832065238 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f73e0013420 con 0x7f73f00fdca0 2024-03-30T04:46:59.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.468+0000 7f73f5a00700 1 -- 172.21.15.123:0/3832065238 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73f010dae0 con 0x7f73f00fdca0 2024-03-30T04:46:59.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.468+0000 7f73e7e00700 1 -- 172.21.15.123:0/3832065238 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f73e001c630 con 0x7f73f00fdca0 2024-03-30T04:46:59.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.468+0000 7f73f5a00700 1 -- 172.21.15.123:0/3832065238 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73f010e020 con 0x7f73f00fdca0 2024-03-30T04:46:59.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.469+0000 7f73f5a00700 1 -- 172.21.15.123:0/3832065238 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73f00668d0 con 0x7f73f00fdca0 2024-03-30T04:46:59.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.470+0000 7f73e7e00700 1 -- 172.21.15.123:0/3832065238 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f73e00139e0 con 0x7f73f00fdca0 2024-03-30T04:46:59.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.470+0000 7f73e7e00700 1 --2- 172.21.15.123:0/3832065238 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f73d406cc20 0x7f73d406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:46:59.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.471+0000 7f73e7e00700 1 -- 172.21.15.123:0/3832065238 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f73e008d320 con 0x7f73f00fdca0 2024-03-30T04:46:59.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.471+0000 7f73ef400700 1 --2- 172.21.15.123:0/3832065238 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f73d406cc20 0x7f73d406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:46:59.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.471+0000 7f73ef400700 1 --2- 172.21.15.123:0/3832065238 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f73d406cc20 0x7f73d406f0d0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f73dc007fd0 tx=0x7f73dc009f60 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:46:59.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.475+0000 7f73e7e00700 1 -- 172.21.15.123:0/3832065238 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f73e005ae20 con 0x7f73f00fdca0 2024-03-30T04:46:59.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.702+0000 7f73f5a00700 1 -- 172.21.15.123:0/3832065238 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f73f0109bf0 con 0x7f73d406cc20 2024-03-30T04:46:59.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.704+0000 7f73e7e00700 1 -- 172.21.15.123:0/3832065238 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f73f0109bf0 con 0x7f73d406cc20 2024-03-30T04:46:59.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.706+0000 7f73f5a00700 1 -- 172.21.15.123:0/3832065238 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f73d406cc20 msgr2=0x7f73d406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:46:59.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.706+0000 7f73f5a00700 1 --2- 172.21.15.123:0/3832065238 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f73d406cc20 0x7f73d406f0d0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f73dc007fd0 tx=0x7f73dc009f60 comp rx=0 tx=0).stop 2024-03-30T04:46:59.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.706+0000 7f73f5a00700 1 -- 172.21.15.123:0/3832065238 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73f00fdca0 msgr2=0x7f73f01138e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:46:59.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.706+0000 7f73f5a00700 1 --2- 172.21.15.123:0/3832065238 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73f00fdca0 0x7f73f01138e0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f73e0009390 tx=0x7f73e0009e30 comp rx=0 tx=0).stop 2024-03-30T04:46:59.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.706+0000 7f73f5a00700 1 -- 172.21.15.123:0/3832065238 shutdown_connections 2024-03-30T04:46:59.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.706+0000 7f73f5a00700 1 --2- 172.21.15.123:0/3832065238 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f73d406cc20 0x7f73d406f0d0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:59.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.706+0000 7f73f5a00700 1 --2- 172.21.15.123:0/3832065238 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73f00fdca0 0x7f73f01138e0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:59.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.706+0000 7f73f5a00700 1 --2- 172.21.15.123:0/3832065238 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f73f00fe5b0 0x7f73f0113e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:59.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.706+0000 7f73f5a00700 1 -- 172.21.15.123:0/3832065238 >> 172.21.15.123:0/3832065238 conn(0x7f73f00f7670 msgr2=0x7f73f00f8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:46:59.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.707+0000 7f73f5a00700 1 -- 172.21.15.123:0/3832065238 shutdown_connections 2024-03-30T04:46:59.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.707+0000 7f73f5a00700 1 -- 172.21.15.123:0/3832065238 wait complete. 2024-03-30T04:46:59.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.840+0000 7f2cfce00700 1 -- 172.21.15.123:0/1058588756 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2cf8102e00 msgr2=0x7f2cf81031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:46:59.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.840+0000 7f2cfce00700 1 --2- 172.21.15.123:0/1058588756 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2cf8102e00 0x7f2cf81031d0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f2ce8004940 tx=0x7f2ce8009e30 comp rx=0 tx=0).stop 2024-03-30T04:46:59.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.840+0000 7f2cfce00700 1 -- 172.21.15.123:0/1058588756 shutdown_connections 2024-03-30T04:46:59.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.840+0000 7f2cfce00700 1 --2- 172.21.15.123:0/1058588756 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2cf81037a0 0x7f2cf80fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:59.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.840+0000 7f2cfce00700 1 --2- 172.21.15.123:0/1058588756 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2cf8102e00 0x7f2cf81031d0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:59.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.840+0000 7f2cfce00700 1 -- 172.21.15.123:0/1058588756 >> 172.21.15.123:0/1058588756 conn(0x7f2cf80f89b0 msgr2=0x7f2cf80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:46:59.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.840+0000 7f2cfce00700 1 -- 172.21.15.123:0/1058588756 shutdown_connections 2024-03-30T04:46:59.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.840+0000 7f2cfce00700 1 -- 172.21.15.123:0/1058588756 wait complete. 2024-03-30T04:46:59.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.841+0000 7f2cfce00700 1 Processor -- start 2024-03-30T04:46:59.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.841+0000 7f2cfce00700 1 -- start start 2024-03-30T04:46:59.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.841+0000 7f2cfce00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2cf8102e00 0x7f2cf8115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:46:59.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.841+0000 7f2cfce00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2cf81037a0 0x7f2cf8116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:46:59.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.841+0000 7f2cfce00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf8116860 con 0x7f2cf81037a0 2024-03-30T04:46:59.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.841+0000 7f2cfce00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf810fc30 con 0x7f2cf8102e00 2024-03-30T04:46:59.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.841+0000 7f2cf5400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2cf81037a0 0x7f2cf8116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:46:59.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.841+0000 7f2cf5400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2cf81037a0 0x7f2cf8116210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38018/0 (socket says 172.21.15.123:38018) 2024-03-30T04:46:59.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.841+0000 7f2cf5400700 1 -- 172.21.15.123:0/3202763909 learned_addr learned my addr 172.21.15.123:0/3202763909 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:46:59.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.842+0000 7f2cf5e00700 1 --2- 172.21.15.123:0/3202763909 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2cf8102e00 0x7f2cf8115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:46:59.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.842+0000 7f2cf5400700 1 -- 172.21.15.123:0/3202763909 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2cf8102e00 msgr2=0x7f2cf8115cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:46:59.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.842+0000 7f2cf5400700 1 --2- 172.21.15.123:0/3202763909 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2cf8102e00 0x7f2cf8115cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:46:59.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.842+0000 7f2cf5400700 1 -- 172.21.15.123:0/3202763909 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ce8005040 con 0x7f2cf81037a0 2024-03-30T04:46:59.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.842+0000 7f2cf5400700 1 --2- 172.21.15.123:0/3202763909 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2cf81037a0 0x7f2cf8116210 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f2ce4002a50 tx=0x7f2ce4004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:46:59.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.842+0000 7f2cee000700 1 -- 172.21.15.123:0/3202763909 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ce4007cc0 con 0x7f2cf81037a0 2024-03-30T04:46:59.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.842+0000 7f2cee000700 1 -- 172.21.15.123:0/3202763909 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2ce400f040 con 0x7f2cf81037a0 2024-03-30T04:46:59.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.842+0000 7f2cfce00700 1 -- 172.21.15.123:0/3202763909 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cf810ff10 con 0x7f2cf81037a0 2024-03-30T04:46:59.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.842+0000 7f2cee000700 1 -- 172.21.15.123:0/3202763909 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ce400d920 con 0x7f2cf81037a0 2024-03-30T04:46:59.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.842+0000 7f2cfce00700 1 -- 172.21.15.123:0/3202763909 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cf81104d0 con 0x7f2cf81037a0 2024-03-30T04:46:59.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.844+0000 7f2cfce00700 1 -- 172.21.15.123:0/3202763909 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cf80fe550 con 0x7f2cf81037a0 2024-03-30T04:46:59.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.845+0000 7f2cee000700 1 -- 172.21.15.123:0/3202763909 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f2ce4009680 con 0x7f2cf81037a0 2024-03-30T04:46:59.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.846+0000 7f2cee000700 1 --2- 172.21.15.123:0/3202763909 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2cdc06cc30 0x7f2cdc06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:46:59.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.846+0000 7f2cee000700 1 -- 172.21.15.123:0/3202763909 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f2ce408be50 con 0x7f2cf81037a0 2024-03-30T04:46:59.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.846+0000 7f2cf5e00700 1 --2- 172.21.15.123:0/3202763909 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2cdc06cc30 0x7f2cdc06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:46:59.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.847+0000 7f2cf5e00700 1 --2- 172.21.15.123:0/3202763909 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2cdc06cc30 0x7f2cdc06f0e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2ce8009fd0 tx=0x7f2ce8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:46:59.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:46:59.851+0000 7f2cee000700 1 -- 172.21.15.123:0/3202763909 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2ce4059820 con 0x7f2cf81037a0 2024-03-30T04:46:59.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:46:59 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:47:00.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.064+0000 7f2cfce00700 1 -- 172.21.15.123:0/3202763909 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2cf8109e30 con 0x7f2cdc06cc30 2024-03-30T04:47:00.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.074+0000 7f2cee000700 1 -- 172.21.15.123:0/3202763909 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f2cf8109e30 con 0x7f2cdc06cc30 2024-03-30T04:47:00.072 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:47:00.072 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (16m) 9m ago 28m 15.0M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:47:00.072 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (13m) 9m ago 28m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T04:47:00.072 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (13m) 8m ago 27m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T04:47:00.072 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (15m) 9m ago 28m 74.0M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T04:47:00.072 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (9m) 9m ago 22m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T04:47:00.072 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (9m) 9m ago 22m 10.9M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T04:47:00.072 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (22m) 8m ago 22m 10.7M - 17.2.7-797-g5c024335 f77085f6aab7 0dc41326818c 2024-03-30T04:47:00.072 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (8m) 8m ago 22m 9747k - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T04:47:00.072 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (18m) 9m ago 30m 506M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:47:00.072 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (17m) 8m ago 27m 450M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:47:00.072 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (14m) 9m ago 30m 53.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T04:47:00.072 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (14m) 8m ago 27m 42.6M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T04:47:00.073 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (16m) 9m ago 28m 8052k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:47:00.073 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (16m) 8m ago 27m 7880k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:47:00.073 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (12m) 9m ago 26m 104M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T04:47:00.073 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (12m) 9m ago 26m 93.4M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T04:47:00.073 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (11m) 9m ago 25m 86.3M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T04:47:00.073 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (11m) 8m ago 25m 97.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T04:47:00.073 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (10m) 8m ago 24m 97.9M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T04:47:00.073 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (9m) 8m ago 24m 85.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T04:47:00.073 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (16m) 9m ago 28m 53.7M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:47:00.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.076+0000 7f2cfce00700 1 -- 172.21.15.123:0/3202763909 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2cdc06cc30 msgr2=0x7f2cdc06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:47:00.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.076+0000 7f2cfce00700 1 --2- 172.21.15.123:0/3202763909 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2cdc06cc30 0x7f2cdc06f0e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2ce8009fd0 tx=0x7f2ce8009e30 comp rx=0 tx=0).stop 2024-03-30T04:47:00.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.076+0000 7f2cfce00700 1 -- 172.21.15.123:0/3202763909 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2cf81037a0 msgr2=0x7f2cf8116210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:47:00.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.076+0000 7f2cfce00700 1 --2- 172.21.15.123:0/3202763909 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2cf81037a0 0x7f2cf8116210 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f2ce4002a50 tx=0x7f2ce4004600 comp rx=0 tx=0).stop 2024-03-30T04:47:00.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.076+0000 7f2cfce00700 1 -- 172.21.15.123:0/3202763909 shutdown_connections 2024-03-30T04:47:00.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.076+0000 7f2cfce00700 1 --2- 172.21.15.123:0/3202763909 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2cdc06cc30 0x7f2cdc06f0e0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:00.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.076+0000 7f2cfce00700 1 --2- 172.21.15.123:0/3202763909 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2cf8102e00 0x7f2cf8115cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:00.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.076+0000 7f2cfce00700 1 --2- 172.21.15.123:0/3202763909 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2cf81037a0 0x7f2cf8116210 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:00.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.076+0000 7f2cfce00700 1 -- 172.21.15.123:0/3202763909 >> 172.21.15.123:0/3202763909 conn(0x7f2cf80f89b0 msgr2=0x7f2cf80f9980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:47:00.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.076+0000 7f2cfce00700 1 -- 172.21.15.123:0/3202763909 shutdown_connections 2024-03-30T04:47:00.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.077+0000 7f2cfce00700 1 -- 172.21.15.123:0/3202763909 wait complete. 2024-03-30T04:47:00.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:46:59 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:47:00.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.207+0000 7fe12e000700 1 -- 172.21.15.123:0/219400647 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe1281037a0 msgr2=0x7fe1280fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:47:00.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.207+0000 7fe12e000700 1 --2- 172.21.15.123:0/219400647 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe1281037a0 0x7fe1280fce10 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fe114004940 tx=0x7fe114009e30 comp rx=0 tx=0).stop 2024-03-30T04:47:00.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.208+0000 7fe12e000700 1 -- 172.21.15.123:0/219400647 shutdown_connections 2024-03-30T04:47:00.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.208+0000 7fe12e000700 1 --2- 172.21.15.123:0/219400647 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe1281037a0 0x7fe1280fce10 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:00.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.208+0000 7fe12e000700 1 --2- 172.21.15.123:0/219400647 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe128102e00 0x7fe1281031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:00.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.208+0000 7fe12e000700 1 -- 172.21.15.123:0/219400647 >> 172.21.15.123:0/219400647 conn(0x7fe1280f89b0 msgr2=0x7fe1280fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:47:00.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.208+0000 7fe12e000700 1 -- 172.21.15.123:0/219400647 shutdown_connections 2024-03-30T04:47:00.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.208+0000 7fe12e000700 1 -- 172.21.15.123:0/219400647 wait complete. 2024-03-30T04:47:00.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.208+0000 7fe12e000700 1 Processor -- start 2024-03-30T04:47:00.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.209+0000 7fe12e000700 1 -- start start 2024-03-30T04:47:00.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.209+0000 7fe12e000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe128102e00 0x7fe1280702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:47:00.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.209+0000 7fe12e000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe1281037a0 0x7fe128070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:47:00.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.209+0000 7fe12e000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe128070d90 con 0x7fe1281037a0 2024-03-30T04:47:00.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.209+0000 7fe12e000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe128070ed0 con 0x7fe128102e00 2024-03-30T04:47:00.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.209+0000 7fe126a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe1281037a0 0x7fe128070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:47:00.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.209+0000 7fe126a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe1281037a0 0x7fe128070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38040/0 (socket says 172.21.15.123:38040) 2024-03-30T04:47:00.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.209+0000 7fe126a00700 1 -- 172.21.15.123:0/2169346168 learned_addr learned my addr 172.21.15.123:0/2169346168 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:47:00.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.209+0000 7fe127400700 1 --2- 172.21.15.123:0/2169346168 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe128102e00 0x7fe1280702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:47:00.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.210+0000 7fe126a00700 1 -- 172.21.15.123:0/2169346168 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe128102e00 msgr2=0x7fe1280702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:47:00.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.210+0000 7fe126a00700 1 --2- 172.21.15.123:0/2169346168 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe128102e00 0x7fe1280702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:00.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.210+0000 7fe126a00700 1 -- 172.21.15.123:0/2169346168 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe114005040 con 0x7fe1281037a0 2024-03-30T04:47:00.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.210+0000 7fe126a00700 1 --2- 172.21.15.123:0/2169346168 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe1281037a0 0x7fe128070800 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fe114009390 tx=0x7fe114009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:47:00.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.210+0000 7fe11f400700 1 -- 172.21.15.123:0/2169346168 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe11400e040 con 0x7fe1281037a0 2024-03-30T04:47:00.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.210+0000 7fe11f400700 1 -- 172.21.15.123:0/2169346168 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe114013420 con 0x7fe1281037a0 2024-03-30T04:47:00.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.210+0000 7fe12e000700 1 -- 172.21.15.123:0/2169346168 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe128071120 con 0x7fe1281037a0 2024-03-30T04:47:00.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.210+0000 7fe11f400700 1 -- 172.21.15.123:0/2169346168 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe11401c630 con 0x7fe1281037a0 2024-03-30T04:47:00.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.210+0000 7fe12e000700 1 -- 172.21.15.123:0/2169346168 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe12811a1a0 con 0x7fe1281037a0 2024-03-30T04:47:00.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.211+0000 7fe12e000700 1 -- 172.21.15.123:0/2169346168 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1280fe550 con 0x7fe1281037a0 2024-03-30T04:47:00.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.212+0000 7fe11f400700 1 -- 172.21.15.123:0/2169346168 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fe114013a00 con 0x7fe1281037a0 2024-03-30T04:47:00.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.213+0000 7fe11f400700 1 --2- 172.21.15.123:0/2169346168 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe10c06cc30 0x7fe10c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:47:00.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.213+0000 7fe11f400700 1 -- 172.21.15.123:0/2169346168 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe11408d0e0 con 0x7fe1281037a0 2024-03-30T04:47:00.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.213+0000 7fe127400700 1 --2- 172.21.15.123:0/2169346168 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe10c06cc30 0x7fe10c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:47:00.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.214+0000 7fe127400700 1 --2- 172.21.15.123:0/2169346168 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe10c06cc30 0x7fe10c06f0e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fe118009fd0 tx=0x7fe11800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:47:00.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.217+0000 7fe11f400700 1 -- 172.21.15.123:0/2169346168 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe114057120 con 0x7fe1281037a0 2024-03-30T04:47:00.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.505+0000 7fe12e000700 1 -- 172.21.15.123:0/2169346168 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe128109dd0 con 0x7fe1281037a0 2024-03-30T04:47:00.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.505+0000 7fe11f400700 1 -- 172.21.15.123:0/2169346168 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7fe11405a740 con 0x7fe1281037a0 2024-03-30T04:47:00.503 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:47:00.503 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:47:00.503 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:47:00.503 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:47:00.503 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:47:00.503 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:47:00.503 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:47:00.503 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:47:00.503 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T04:47:00.503 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:47:00.503 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:47:00.503 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:47:00.503 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 3 2024-03-30T04:47:00.504 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:47:00.504 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:47:00.504 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:47:00.504 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 13 2024-03-30T04:47:00.504 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:47:00.504 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:47:00.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.508+0000 7fe12e000700 1 -- 172.21.15.123:0/2169346168 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe10c06cc30 msgr2=0x7fe10c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:47:00.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.508+0000 7fe12e000700 1 --2- 172.21.15.123:0/2169346168 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe10c06cc30 0x7fe10c06f0e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fe118009fd0 tx=0x7fe11800b040 comp rx=0 tx=0).stop 2024-03-30T04:47:00.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.508+0000 7fe12e000700 1 -- 172.21.15.123:0/2169346168 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe1281037a0 msgr2=0x7fe128070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:47:00.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.508+0000 7fe12e000700 1 --2- 172.21.15.123:0/2169346168 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe1281037a0 0x7fe128070800 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fe114009390 tx=0x7fe114009e30 comp rx=0 tx=0).stop 2024-03-30T04:47:00.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.508+0000 7fe12e000700 1 -- 172.21.15.123:0/2169346168 shutdown_connections 2024-03-30T04:47:00.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.508+0000 7fe12e000700 1 --2- 172.21.15.123:0/2169346168 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe10c06cc30 0x7fe10c06f0e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:00.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.508+0000 7fe12e000700 1 --2- 172.21.15.123:0/2169346168 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe128102e00 0x7fe1280702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:00.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.508+0000 7fe12e000700 1 --2- 172.21.15.123:0/2169346168 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe1281037a0 0x7fe128070800 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:00.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.508+0000 7fe12e000700 1 -- 172.21.15.123:0/2169346168 >> 172.21.15.123:0/2169346168 conn(0x7fe1280f89b0 msgr2=0x7fe1280f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:47:00.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.508+0000 7fe12e000700 1 -- 172.21.15.123:0/2169346168 shutdown_connections 2024-03-30T04:47:00.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.508+0000 7fe12e000700 1 -- 172.21.15.123:0/2169346168 wait complete. 2024-03-30T04:47:00.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.639+0000 7fba4b000700 1 -- 172.21.15.123:0/1707499511 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba440fe830 msgr2=0x7fba44106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:47:00.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.639+0000 7fba4b000700 1 --2- 172.21.15.123:0/1707499511 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba440fe830 0x7fba44106f00 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fba30004940 tx=0x7fba30009e30 comp rx=0 tx=0).stop 2024-03-30T04:47:00.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.639+0000 7fba4b000700 1 -- 172.21.15.123:0/1707499511 shutdown_connections 2024-03-30T04:47:00.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.639+0000 7fba4b000700 1 --2- 172.21.15.123:0/1707499511 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba440fe830 0x7fba44106f00 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:00.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.639+0000 7fba4b000700 1 --2- 172.21.15.123:0/1707499511 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fba440fde90 0x7fba440fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:00.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.639+0000 7fba4b000700 1 -- 172.21.15.123:0/1707499511 >> 172.21.15.123:0/1707499511 conn(0x7fba44072bc0 msgr2=0x7fba44072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:47:00.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.640+0000 7fba4b000700 1 -- 172.21.15.123:0/1707499511 shutdown_connections 2024-03-30T04:47:00.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.640+0000 7fba4b000700 1 -- 172.21.15.123:0/1707499511 wait complete. 2024-03-30T04:47:00.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.640+0000 7fba4b000700 1 Processor -- start 2024-03-30T04:47:00.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.641+0000 7fba4b000700 1 -- start start 2024-03-30T04:47:00.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.641+0000 7fba4b000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba440fde90 0x7fba44115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:47:00.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.641+0000 7fba4b000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fba440fe830 0x7fba441161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:47:00.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.641+0000 7fba4b000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba44116800 con 0x7fba440fde90 2024-03-30T04:47:00.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.641+0000 7fba4b000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba4410fbd0 con 0x7fba440fe830 2024-03-30T04:47:00.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.641+0000 7fba43e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba440fde90 0x7fba44115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:47:00.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.641+0000 7fba43e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba440fde90 0x7fba44115c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38066/0 (socket says 172.21.15.123:38066) 2024-03-30T04:47:00.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.641+0000 7fba43e00700 1 -- 172.21.15.123:0/3534341558 learned_addr learned my addr 172.21.15.123:0/3534341558 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:47:00.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.641+0000 7fba43400700 1 --2- 172.21.15.123:0/3534341558 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fba440fe830 0x7fba441161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:47:00.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.642+0000 7fba43e00700 1 -- 172.21.15.123:0/3534341558 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fba440fe830 msgr2=0x7fba441161b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:47:00.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.642+0000 7fba43e00700 1 --2- 172.21.15.123:0/3534341558 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fba440fe830 0x7fba441161b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:00.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.642+0000 7fba43e00700 1 -- 172.21.15.123:0/3534341558 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba30005040 con 0x7fba440fde90 2024-03-30T04:47:00.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.642+0000 7fba43e00700 1 --2- 172.21.15.123:0/3534341558 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba440fde90 0x7fba44115c70 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fba3400eb40 tx=0x7fba3400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:47:00.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.642+0000 7fba3be00700 1 -- 172.21.15.123:0/3534341558 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba34007cc0 con 0x7fba440fde90 2024-03-30T04:47:00.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.642+0000 7fba3be00700 1 -- 172.21.15.123:0/3534341558 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fba3400f040 con 0x7fba440fde90 2024-03-30T04:47:00.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.642+0000 7fba4b000700 1 -- 172.21.15.123:0/3534341558 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba4410fe50 con 0x7fba440fde90 2024-03-30T04:47:00.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.642+0000 7fba3be00700 1 -- 172.21.15.123:0/3534341558 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba3400c990 con 0x7fba440fde90 2024-03-30T04:47:00.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.642+0000 7fba4b000700 1 -- 172.21.15.123:0/3534341558 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba44110390 con 0x7fba440fde90 2024-03-30T04:47:00.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.643+0000 7fba4b000700 1 -- 172.21.15.123:0/3534341558 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba44106a20 con 0x7fba440fde90 2024-03-30T04:47:00.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.644+0000 7fba3be00700 1 -- 172.21.15.123:0/3534341558 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fba3400cc00 con 0x7fba440fde90 2024-03-30T04:47:00.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.645+0000 7fba3be00700 1 --2- 172.21.15.123:0/3534341558 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fba2806cc30 0x7fba2806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:47:00.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.645+0000 7fba3be00700 1 -- 172.21.15.123:0/3534341558 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fba3408bcd0 con 0x7fba440fde90 2024-03-30T04:47:00.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.645+0000 7fba43400700 1 --2- 172.21.15.123:0/3534341558 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fba2806cc30 0x7fba2806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:47:00.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.646+0000 7fba43400700 1 --2- 172.21.15.123:0/3534341558 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fba2806cc30 0x7fba2806f0e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fba44111510 tx=0x7fba30009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:47:00.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.650+0000 7fba3be00700 1 -- 172.21.15.123:0/3534341558 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fba34059750 con 0x7fba440fde90 2024-03-30T04:47:00.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.895+0000 7fba4b000700 1 -- 172.21.15.123:0/3534341558 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fba4404bed0 con 0x7fba440fde90 2024-03-30T04:47:00.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.895+0000 7fba3be00700 1 -- 172.21.15.123:0/3534341558 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1843 (secure 0 0 0) 0x7fba340592e0 con 0x7fba440fde90 2024-03-30T04:47:00.894 INFO:teuthology.orchestra.run.smithi123.stdout:e32 2024-03-30T04:47:00.894 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:47:00.894 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:47:00.894 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:47:00.894 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:47:00.894 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:47:00.894 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:47:00.894 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T04:47:00.894 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T04:47:00.894 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:47:00.894 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T04:47:00.895 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:47:00.895 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:47:00.895 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:47:00.895 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:47:00.895 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:47:00.895 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T04:47:00.895 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:47:00.895 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:47:00.895 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T04:47:00.895 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:47:00.895 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T04:47:00.895 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T04:47:00.895 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T04:47:00.895 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T04:47:00.896 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:47:00.896 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:47:00.896 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:47:00.896 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:47:00.896 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:47:00.896 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:47:00.896 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T04:47:00.896 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T04:47:00.896 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:47:00.896 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:47:00.896 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T04:47:00.896 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:47:00.896 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:47:00.896 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:47:00.897 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44315} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/1741352336,v1:172.21.15.135:6827/1741352336] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:47:00.897 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:47:00.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.898+0000 7fba4b000700 1 -- 172.21.15.123:0/3534341558 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fba2806cc30 msgr2=0x7fba2806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:47:00.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.898+0000 7fba4b000700 1 --2- 172.21.15.123:0/3534341558 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fba2806cc30 0x7fba2806f0e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fba44111510 tx=0x7fba30009e30 comp rx=0 tx=0).stop 2024-03-30T04:47:00.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.898+0000 7fba4b000700 1 -- 172.21.15.123:0/3534341558 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba440fde90 msgr2=0x7fba44115c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:47:00.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.898+0000 7fba4b000700 1 --2- 172.21.15.123:0/3534341558 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba440fde90 0x7fba44115c70 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fba3400eb40 tx=0x7fba3400b040 comp rx=0 tx=0).stop 2024-03-30T04:47:00.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.898+0000 7fba4b000700 1 -- 172.21.15.123:0/3534341558 shutdown_connections 2024-03-30T04:47:00.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.898+0000 7fba4b000700 1 --2- 172.21.15.123:0/3534341558 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fba2806cc30 0x7fba2806f0e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:00.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.898+0000 7fba4b000700 1 --2- 172.21.15.123:0/3534341558 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba440fde90 0x7fba44115c70 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:00.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.898+0000 7fba4b000700 1 --2- 172.21.15.123:0/3534341558 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fba440fe830 0x7fba441161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:00.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.898+0000 7fba4b000700 1 -- 172.21.15.123:0/3534341558 >> 172.21.15.123:0/3534341558 conn(0x7fba44072bc0 msgr2=0x7fba440fb6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:47:00.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.898+0000 7fba4b000700 1 -- 172.21.15.123:0/3534341558 shutdown_connections 2024-03-30T04:47:00.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:00.898+0000 7fba4b000700 1 -- 172.21.15.123:0/3534341558 wait complete. 2024-03-30T04:47:00.899 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 32 2024-03-30T04:47:00.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:47:00 smithi123 ceph-mon[139369]: from='client.34906 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:47:00.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:47:00 smithi123 ceph-mon[139369]: pgmap v474: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:00.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:47:00 smithi123 ceph-mon[139369]: from='client.34910 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:47:00.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:47:00 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2169346168' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:47:01.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.033+0000 7f7c83800700 1 -- 172.21.15.123:0/1621853563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c7c102e00 msgr2=0x7f7c7c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:47:01.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.033+0000 7f7c83800700 1 --2- 172.21.15.123:0/1621853563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c7c102e00 0x7f7c7c1031d0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f7c70004940 tx=0x7f7c70009e30 comp rx=0 tx=0).stop 2024-03-30T04:47:01.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.033+0000 7f7c83800700 1 -- 172.21.15.123:0/1621853563 shutdown_connections 2024-03-30T04:47:01.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.033+0000 7f7c83800700 1 --2- 172.21.15.123:0/1621853563 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7c7c1037a0 0x7f7c7c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:01.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.033+0000 7f7c83800700 1 --2- 172.21.15.123:0/1621853563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c7c102e00 0x7f7c7c1031d0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:01.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.033+0000 7f7c83800700 1 -- 172.21.15.123:0/1621853563 >> 172.21.15.123:0/1621853563 conn(0x7f7c7c0f89b0 msgr2=0x7f7c7c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:47:01.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.033+0000 7f7c83800700 1 -- 172.21.15.123:0/1621853563 shutdown_connections 2024-03-30T04:47:01.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.033+0000 7f7c83800700 1 -- 172.21.15.123:0/1621853563 wait complete. 2024-03-30T04:47:01.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.034+0000 7f7c83800700 1 Processor -- start 2024-03-30T04:47:01.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.034+0000 7f7c83800700 1 -- start start 2024-03-30T04:47:01.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.034+0000 7f7c83800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c7c102e00 0x7f7c7c115d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:47:01.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.034+0000 7f7c83800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7c7c1037a0 0x7f7c7c116250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:47:01.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.034+0000 7f7c83800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7c1168a0 con 0x7f7c7c102e00 2024-03-30T04:47:01.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.034+0000 7f7c83800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7c10fc70 con 0x7f7c7c1037a0 2024-03-30T04:47:01.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.035+0000 7f7c80e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c7c102e00 0x7f7c7c115d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:47:01.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.035+0000 7f7c80e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c7c102e00 0x7f7c7c115d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38094/0 (socket says 172.21.15.123:38094) 2024-03-30T04:47:01.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.035+0000 7f7c80e00700 1 -- 172.21.15.123:0/3103764796 learned_addr learned my addr 172.21.15.123:0/3103764796 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:47:01.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.035+0000 7f7c7be00700 1 --2- 172.21.15.123:0/3103764796 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7c7c1037a0 0x7f7c7c116250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:47:01.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.035+0000 7f7c80e00700 1 -- 172.21.15.123:0/3103764796 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7c7c1037a0 msgr2=0x7f7c7c116250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:47:01.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.035+0000 7f7c80e00700 1 --2- 172.21.15.123:0/3103764796 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7c7c1037a0 0x7f7c7c116250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:01.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.035+0000 7f7c80e00700 1 -- 172.21.15.123:0/3103764796 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c70005040 con 0x7f7c7c102e00 2024-03-30T04:47:01.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.035+0000 7f7c80e00700 1 --2- 172.21.15.123:0/3103764796 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c7c102e00 0x7f7c7c115d10 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f7c70009390 tx=0x7f7c70009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:47:01.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.035+0000 7f7c78c00700 1 -- 172.21.15.123:0/3103764796 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c7000e040 con 0x7f7c7c102e00 2024-03-30T04:47:01.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.035+0000 7f7c78c00700 1 -- 172.21.15.123:0/3103764796 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7c70013420 con 0x7f7c7c102e00 2024-03-30T04:47:01.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.036+0000 7f7c83800700 1 -- 172.21.15.123:0/3103764796 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c7c10fef0 con 0x7f7c7c102e00 2024-03-30T04:47:01.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.036+0000 7f7c78c00700 1 -- 172.21.15.123:0/3103764796 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c7001c630 con 0x7f7c7c102e00 2024-03-30T04:47:01.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.036+0000 7f7c83800700 1 -- 172.21.15.123:0/3103764796 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c7c1104b0 con 0x7f7c7c102e00 2024-03-30T04:47:01.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.037+0000 7f7c83800700 1 -- 172.21.15.123:0/3103764796 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c7c0fe550 con 0x7f7c7c102e00 2024-03-30T04:47:01.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.038+0000 7f7c78c00700 1 -- 172.21.15.123:0/3103764796 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f7c700139e0 con 0x7f7c7c102e00 2024-03-30T04:47:01.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.038+0000 7f7c78c00700 1 --2- 172.21.15.123:0/3103764796 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7c6406cc30 0x7f7c6406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:47:01.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.039+0000 7f7c78c00700 1 -- 172.21.15.123:0/3103764796 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7c7008d340 con 0x7f7c7c102e00 2024-03-30T04:47:01.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.039+0000 7f7c7be00700 1 --2- 172.21.15.123:0/3103764796 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7c6406cc30 0x7f7c6406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:47:01.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.039+0000 7f7c7be00700 1 --2- 172.21.15.123:0/3103764796 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7c6406cc30 0x7f7c6406f0e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c1115b0 tx=0x7f7c6800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:47:01.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.043+0000 7f7c78c00700 1 -- 172.21.15.123:0/3103764796 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7c7005ae10 con 0x7f7c7c102e00 2024-03-30T04:47:01.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:47:00 smithi135 ceph-mon[106011]: from='client.34906 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:47:01.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:47:00 smithi135 ceph-mon[106011]: pgmap v474: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:01.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:47:00 smithi135 ceph-mon[106011]: from='client.34910 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:47:01.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:47:00 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2169346168' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:47:01.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.262+0000 7f7c83800700 1 -- 172.21.15.123:0/3103764796 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7c7c04bf40 con 0x7f7c6406cc30 2024-03-30T04:47:01.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.263+0000 7f7c78c00700 1 -- 172.21.15.123:0/3103764796 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f7c7c04bf40 con 0x7f7c6406cc30 2024-03-30T04:47:01.261 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:47:01.261 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:47:01.262 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:47:01.262 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T04:47:01.262 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:47:01.262 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T04:47:01.262 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T04:47:01.262 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T04:47:01.262 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T04:47:01.262 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:47:01.262 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "15/21 daemons upgraded", 2024-03-30T04:47:01.262 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T04:47:01.262 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:47:01.262 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:47:01.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.265+0000 7f7c83800700 1 -- 172.21.15.123:0/3103764796 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7c6406cc30 msgr2=0x7f7c6406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:47:01.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.265+0000 7f7c83800700 1 --2- 172.21.15.123:0/3103764796 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7c6406cc30 0x7f7c6406f0e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c1115b0 tx=0x7f7c6800b040 comp rx=0 tx=0).stop 2024-03-30T04:47:01.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.265+0000 7f7c83800700 1 -- 172.21.15.123:0/3103764796 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c7c102e00 msgr2=0x7f7c7c115d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:47:01.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.265+0000 7f7c83800700 1 --2- 172.21.15.123:0/3103764796 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c7c102e00 0x7f7c7c115d10 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f7c70009390 tx=0x7f7c70009e30 comp rx=0 tx=0).stop 2024-03-30T04:47:01.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.266+0000 7f7c83800700 1 -- 172.21.15.123:0/3103764796 shutdown_connections 2024-03-30T04:47:01.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.266+0000 7f7c83800700 1 --2- 172.21.15.123:0/3103764796 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7c6406cc30 0x7f7c6406f0e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:01.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.266+0000 7f7c83800700 1 --2- 172.21.15.123:0/3103764796 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c7c102e00 0x7f7c7c115d10 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:01.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.266+0000 7f7c83800700 1 --2- 172.21.15.123:0/3103764796 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7c7c1037a0 0x7f7c7c116250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:01.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.266+0000 7f7c83800700 1 -- 172.21.15.123:0/3103764796 >> 172.21.15.123:0/3103764796 conn(0x7f7c7c0f89b0 msgr2=0x7f7c7c0704f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:47:01.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.266+0000 7f7c83800700 1 -- 172.21.15.123:0/3103764796 shutdown_connections 2024-03-30T04:47:01.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.266+0000 7f7c83800700 1 -- 172.21.15.123:0/3103764796 wait complete. 2024-03-30T04:47:01.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.399+0000 7f5db4200700 1 -- 172.21.15.123:0/2630208507 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5dac1037a0 msgr2=0x7f5dac0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:47:01.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.399+0000 7f5db4200700 1 --2- 172.21.15.123:0/2630208507 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5dac1037a0 0x7f5dac0fce10 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f5d98004940 tx=0x7f5d98009e30 comp rx=0 tx=0).stop 2024-03-30T04:47:01.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.399+0000 7f5db4200700 1 -- 172.21.15.123:0/2630208507 shutdown_connections 2024-03-30T04:47:01.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.399+0000 7f5db4200700 1 --2- 172.21.15.123:0/2630208507 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5dac1037a0 0x7f5dac0fce10 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:01.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.399+0000 7f5db4200700 1 --2- 172.21.15.123:0/2630208507 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5dac102e00 0x7f5dac1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:01.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.399+0000 7f5db4200700 1 -- 172.21.15.123:0/2630208507 >> 172.21.15.123:0/2630208507 conn(0x7f5dac0f89b0 msgr2=0x7f5dac0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:47:01.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.399+0000 7f5db4200700 1 -- 172.21.15.123:0/2630208507 shutdown_connections 2024-03-30T04:47:01.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.399+0000 7f5db4200700 1 -- 172.21.15.123:0/2630208507 wait complete. 2024-03-30T04:47:01.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.400+0000 7f5db4200700 1 Processor -- start 2024-03-30T04:47:01.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.400+0000 7f5db4200700 1 -- start start 2024-03-30T04:47:01.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.400+0000 7f5db4200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5dac102e00 0x7f5dac115d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:47:01.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.400+0000 7f5db4200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5dac1037a0 0x7f5dac116260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:47:01.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.400+0000 7f5db4200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dac1168b0 con 0x7f5dac1037a0 2024-03-30T04:47:01.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.400+0000 7f5db4200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dac10fc80 con 0x7f5dac102e00 2024-03-30T04:47:01.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.401+0000 7f5db0e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5dac1037a0 0x7f5dac116260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:47:01.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.401+0000 7f5db0e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5dac1037a0 0x7f5dac116260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38100/0 (socket says 172.21.15.123:38100) 2024-03-30T04:47:01.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.401+0000 7f5db0e00700 1 -- 172.21.15.123:0/779548358 learned_addr learned my addr 172.21.15.123:0/779548358 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:47:01.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.401+0000 7f5db1800700 1 --2- 172.21.15.123:0/779548358 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5dac102e00 0x7f5dac115d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:47:01.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.401+0000 7f5db0e00700 1 -- 172.21.15.123:0/779548358 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5dac102e00 msgr2=0x7f5dac115d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:47:01.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.401+0000 7f5db0e00700 1 --2- 172.21.15.123:0/779548358 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5dac102e00 0x7f5dac115d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:01.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.401+0000 7f5db0e00700 1 -- 172.21.15.123:0/779548358 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d98005040 con 0x7f5dac1037a0 2024-03-30T04:47:01.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.401+0000 7f5db0e00700 1 --2- 172.21.15.123:0/779548358 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5dac1037a0 0x7f5dac116260 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f5d98009390 tx=0x7f5d98009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:47:01.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.401+0000 7f5da9600700 1 -- 172.21.15.123:0/779548358 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d9800e040 con 0x7f5dac1037a0 2024-03-30T04:47:01.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.401+0000 7f5da9600700 1 -- 172.21.15.123:0/779548358 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5d98013420 con 0x7f5dac1037a0 2024-03-30T04:47:01.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.401+0000 7f5db4200700 1 -- 172.21.15.123:0/779548358 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dac10ff00 con 0x7f5dac1037a0 2024-03-30T04:47:01.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.401+0000 7f5da9600700 1 -- 172.21.15.123:0/779548358 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d9801c630 con 0x7f5dac1037a0 2024-03-30T04:47:01.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.402+0000 7f5db4200700 1 -- 172.21.15.123:0/779548358 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dac1104c0 con 0x7f5dac1037a0 2024-03-30T04:47:01.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.403+0000 7f5db4200700 1 -- 172.21.15.123:0/779548358 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dac0fe550 con 0x7f5dac1037a0 2024-03-30T04:47:01.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.404+0000 7f5da9600700 1 -- 172.21.15.123:0/779548358 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f5d98013a00 con 0x7f5dac1037a0 2024-03-30T04:47:01.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.404+0000 7f5da9600700 1 --2- 172.21.15.123:0/779548358 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5d9406cc30 0x7f5d9406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:47:01.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.405+0000 7f5da9600700 1 -- 172.21.15.123:0/779548358 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5d9808d2c0 con 0x7f5dac1037a0 2024-03-30T04:47:01.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.405+0000 7f5db1800700 1 --2- 172.21.15.123:0/779548358 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5d9406cc30 0x7f5d9406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:47:01.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.405+0000 7f5db1800700 1 --2- 172.21.15.123:0/779548358 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5d9406cc30 0x7f5d9406f0e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f5da0009fd0 tx=0x7f5da0009f20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:47:01.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.409+0000 7f5da9600700 1 -- 172.21.15.123:0/779548358 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5d9805ad90 con 0x7f5dac1037a0 2024-03-30T04:47:01.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.689+0000 7f5db4200700 1 -- 172.21.15.123:0/779548358 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f5dac05f5f0 con 0x7f5dac1037a0 2024-03-30T04:47:01.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.689+0000 7f5da9600700 1 -- 172.21.15.123:0/779548358 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f5d9805a920 con 0x7f5dac1037a0 2024-03-30T04:47:01.688 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T04:47:01.688 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T04:47:01.688 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T04:47:01.688 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T04:47:01.688 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T04:47:01.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.692+0000 7f5db4200700 1 -- 172.21.15.123:0/779548358 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5d9406cc30 msgr2=0x7f5d9406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:47:01.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.692+0000 7f5db4200700 1 --2- 172.21.15.123:0/779548358 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5d9406cc30 0x7f5d9406f0e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f5da0009fd0 tx=0x7f5da0009f20 comp rx=0 tx=0).stop 2024-03-30T04:47:01.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.692+0000 7f5db4200700 1 -- 172.21.15.123:0/779548358 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5dac1037a0 msgr2=0x7f5dac116260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:47:01.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.692+0000 7f5db4200700 1 --2- 172.21.15.123:0/779548358 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5dac1037a0 0x7f5dac116260 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f5d98009390 tx=0x7f5d98009e30 comp rx=0 tx=0).stop 2024-03-30T04:47:01.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.692+0000 7f5db4200700 1 -- 172.21.15.123:0/779548358 shutdown_connections 2024-03-30T04:47:01.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.692+0000 7f5db4200700 1 --2- 172.21.15.123:0/779548358 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5d9406cc30 0x7f5d9406f0e0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:01.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.692+0000 7f5db4200700 1 --2- 172.21.15.123:0/779548358 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5dac102e00 0x7f5dac115d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:01.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.692+0000 7f5db4200700 1 --2- 172.21.15.123:0/779548358 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5dac1037a0 0x7f5dac116260 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:01.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.692+0000 7f5db4200700 1 -- 172.21.15.123:0/779548358 >> 172.21.15.123:0/779548358 conn(0x7f5dac0f89b0 msgr2=0x7f5dac0f9a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:47:01.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.692+0000 7f5db4200700 1 -- 172.21.15.123:0/779548358 shutdown_connections 2024-03-30T04:47:01.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:01.692+0000 7f5db4200700 1 -- 172.21.15.123:0/779548358 wait complete. 2024-03-30T04:47:01.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:47:01 smithi123 ceph-mon[139369]: from='client.34914 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:47:01.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:47:01 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3534341558' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:47:02.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:47:01 smithi135 ceph-mon[106011]: from='client.34914 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:47:02.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:47:01 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3534341558' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:47:02.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:47:02 smithi123 ceph-mon[139369]: from='client.34926 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:47:02.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:47:02 smithi123 ceph-mon[139369]: pgmap v475: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:02.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:47:02 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/779548358' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:47:03.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:47:02 smithi135 ceph-mon[106011]: from='client.34926 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:47:03.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:47:02 smithi135 ceph-mon[106011]: pgmap v475: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:03.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:47:02 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/779548358' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:47:04.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:47:04 smithi123 ceph-mon[139369]: pgmap v476: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:05.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:47:04 smithi135 ceph-mon[106011]: pgmap v476: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:06.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:47:06 smithi123 ceph-mon[139369]: pgmap v477: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:07.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:47:06 smithi135 ceph-mon[106011]: pgmap v477: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:08.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:47:08 smithi123 ceph-mon[139369]: pgmap v478: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:09.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:47:08 smithi135 ceph-mon[106011]: pgmap v478: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:10.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:47:10 smithi135 ceph-mon[106011]: pgmap v479: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:10.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:47:10 smithi123 ceph-mon[139369]: pgmap v479: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:12.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:47:12 smithi123 ceph-mon[139369]: pgmap v480: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:13.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:47:12 smithi135 ceph-mon[106011]: pgmap v480: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:14.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:47:14 smithi123 ceph-mon[139369]: pgmap v481: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:14.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:47:14 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:47:15.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:47:14 smithi135 ceph-mon[106011]: pgmap v481: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:15.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:47:14 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:47:16.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:47:16 smithi123 ceph-mon[139369]: pgmap v482: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:17.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:47:16 smithi135 ceph-mon[106011]: pgmap v482: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:18.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:47:18 smithi123 ceph-mon[139369]: pgmap v483: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:19.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:47:18 smithi135 ceph-mon[106011]: pgmap v483: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:20.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:47:20 smithi135 ceph-mon[106011]: pgmap v484: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:20.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:47:20 smithi123 ceph-mon[139369]: pgmap v484: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:22.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:47:22 smithi123 ceph-mon[139369]: pgmap v485: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:23.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:47:22 smithi135 ceph-mon[106011]: pgmap v485: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:24.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:47:24 smithi123 ceph-mon[139369]: pgmap v486: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:25.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:47:24 smithi135 ceph-mon[106011]: pgmap v486: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:26.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:47:26 smithi123 ceph-mon[139369]: pgmap v487: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:27.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:47:26 smithi135 ceph-mon[106011]: pgmap v487: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:28.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:47:28 smithi123 ceph-mon[139369]: pgmap v488: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:29.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:47:28 smithi135 ceph-mon[106011]: pgmap v488: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:29.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:47:29 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:47:30.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:47:29 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:47:30.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:47:30 smithi123 ceph-mon[139369]: pgmap v489: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:31.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:47:30 smithi135 ceph-mon[106011]: pgmap v489: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:31.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:31.838+0000 7f31cc400700 1 -- 172.21.15.123:0/873047720 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31c41037a0 msgr2=0x7f31c40fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:47:31.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:31.838+0000 7f31cc400700 1 --2- 172.21.15.123:0/873047720 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31c41037a0 0x7f31c40fce10 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f31b0004940 tx=0x7f31b0009e30 comp rx=0 tx=0).stop 2024-03-30T04:47:31.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:31.838+0000 7f31cc400700 1 -- 172.21.15.123:0/873047720 shutdown_connections 2024-03-30T04:47:31.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:31.838+0000 7f31cc400700 1 --2- 172.21.15.123:0/873047720 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31c41037a0 0x7f31c40fce10 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:31.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:31.838+0000 7f31cc400700 1 --2- 172.21.15.123:0/873047720 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31c4102e00 0x7f31c41031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:31.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:31.838+0000 7f31cc400700 1 -- 172.21.15.123:0/873047720 >> 172.21.15.123:0/873047720 conn(0x7f31c40f89b0 msgr2=0x7f31c40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:47:31.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:31.839+0000 7f31cc400700 1 -- 172.21.15.123:0/873047720 shutdown_connections 2024-03-30T04:47:31.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:31.839+0000 7f31cc400700 1 -- 172.21.15.123:0/873047720 wait complete. 2024-03-30T04:47:31.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:31.839+0000 7f31cc400700 1 Processor -- start 2024-03-30T04:47:31.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:31.840+0000 7f31cc400700 1 -- start start 2024-03-30T04:47:31.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:31.840+0000 7f31cc400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31c4102e00 0x7f31c4115a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:47:31.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:31.840+0000 7f31cc400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31c41037a0 0x7f31c4115f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:47:31.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:31.840+0000 7f31cc400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c4116590 con 0x7f31c4102e00 2024-03-30T04:47:31.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:31.840+0000 7f31cc400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c410fbb0 con 0x7f31c41037a0 2024-03-30T04:47:31.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:31.840+0000 7f31c9a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31c4102e00 0x7f31c4115a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:47:31.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:31.840+0000 7f31c9a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31c4102e00 0x7f31c4115a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58768/0 (socket says 172.21.15.123:58768) 2024-03-30T04:47:31.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:31.840+0000 7f31c9a00700 1 -- 172.21.15.123:0/2065430994 learned_addr learned my addr 172.21.15.123:0/2065430994 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:47:31.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:31.840+0000 7f31c9000700 1 --2- 172.21.15.123:0/2065430994 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31c41037a0 0x7f31c4115f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:47:31.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:31.840+0000 7f31c9a00700 1 -- 172.21.15.123:0/2065430994 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31c41037a0 msgr2=0x7f31c4115f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:47:31.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:31.841+0000 7f31c9a00700 1 --2- 172.21.15.123:0/2065430994 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31c41037a0 0x7f31c4115f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:31.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:31.841+0000 7f31c9a00700 1 -- 172.21.15.123:0/2065430994 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31b0005040 con 0x7f31c4102e00 2024-03-30T04:47:31.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:31.841+0000 7f31c9a00700 1 --2- 172.21.15.123:0/2065430994 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31c4102e00 0x7f31c4115a00 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f31b800eb40 tx=0x7f31b800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:47:31.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:31.841+0000 7f31c1600700 1 -- 172.21.15.123:0/2065430994 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31b8007cc0 con 0x7f31c4102e00 2024-03-30T04:47:31.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:31.841+0000 7f31c1600700 1 -- 172.21.15.123:0/2065430994 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f31b800f040 con 0x7f31c4102e00 2024-03-30T04:47:31.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:31.841+0000 7f31c1600700 1 -- 172.21.15.123:0/2065430994 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31b800c990 con 0x7f31c4102e00 2024-03-30T04:47:31.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:31.841+0000 7f31cc400700 1 -- 172.21.15.123:0/2065430994 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31c410fe30 con 0x7f31c4102e00 2024-03-30T04:47:31.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:31.841+0000 7f31cc400700 1 -- 172.21.15.123:0/2065430994 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31c41103f0 con 0x7f31c4102e00 2024-03-30T04:47:31.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:31.842+0000 7f31cc400700 1 -- 172.21.15.123:0/2065430994 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31c40fe550 con 0x7f31c4102e00 2024-03-30T04:47:31.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:31.843+0000 7f31c1600700 1 -- 172.21.15.123:0/2065430994 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f31b800caf0 con 0x7f31c4102e00 2024-03-30T04:47:31.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:31.844+0000 7f31c1600700 1 --2- 172.21.15.123:0/2065430994 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f31ac06cc20 0x7f31ac06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:47:31.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:31.844+0000 7f31c1600700 1 -- 172.21.15.123:0/2065430994 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f31b808bc00 con 0x7f31c4102e00 2024-03-30T04:47:31.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:31.844+0000 7f31c9000700 1 --2- 172.21.15.123:0/2065430994 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f31ac06cc20 0x7f31ac06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:47:31.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:31.845+0000 7f31c9000700 1 --2- 172.21.15.123:0/2065430994 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f31ac06cc20 0x7f31ac06f0d0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f31c41112e0 tx=0x7f31b0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:47:31.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:31.849+0000 7f31c1600700 1 -- 172.21.15.123:0/2065430994 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f31b8055c70 con 0x7f31c4102e00 2024-03-30T04:47:32.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.070+0000 7f31cc400700 1 -- 172.21.15.123:0/2065430994 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f31c4111090 con 0x7f31ac06cc20 2024-03-30T04:47:32.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.072+0000 7f31c1600700 1 -- 172.21.15.123:0/2065430994 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f31c4111090 con 0x7f31ac06cc20 2024-03-30T04:47:32.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.074+0000 7f31cc400700 1 -- 172.21.15.123:0/2065430994 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f31ac06cc20 msgr2=0x7f31ac06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:47:32.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.074+0000 7f31cc400700 1 --2- 172.21.15.123:0/2065430994 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f31ac06cc20 0x7f31ac06f0d0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f31c41112e0 tx=0x7f31b0009e30 comp rx=0 tx=0).stop 2024-03-30T04:47:32.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.074+0000 7f31cc400700 1 -- 172.21.15.123:0/2065430994 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31c4102e00 msgr2=0x7f31c4115a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:47:32.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.074+0000 7f31cc400700 1 --2- 172.21.15.123:0/2065430994 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31c4102e00 0x7f31c4115a00 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f31b800eb40 tx=0x7f31b800b040 comp rx=0 tx=0).stop 2024-03-30T04:47:32.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.074+0000 7f31cc400700 1 -- 172.21.15.123:0/2065430994 shutdown_connections 2024-03-30T04:47:32.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.074+0000 7f31cc400700 1 --2- 172.21.15.123:0/2065430994 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f31ac06cc20 0x7f31ac06f0d0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:32.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.074+0000 7f31cc400700 1 --2- 172.21.15.123:0/2065430994 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31c4102e00 0x7f31c4115a00 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:32.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.074+0000 7f31cc400700 1 --2- 172.21.15.123:0/2065430994 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31c41037a0 0x7f31c4115f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:32.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.074+0000 7f31cc400700 1 -- 172.21.15.123:0/2065430994 >> 172.21.15.123:0/2065430994 conn(0x7f31c40f89b0 msgr2=0x7f31c40f9910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:47:32.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.074+0000 7f31cc400700 1 -- 172.21.15.123:0/2065430994 shutdown_connections 2024-03-30T04:47:32.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.074+0000 7f31cc400700 1 -- 172.21.15.123:0/2065430994 wait complete. 2024-03-30T04:47:32.089 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:47:32.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.209+0000 7f0471800700 1 -- 172.21.15.123:0/928867808 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f046c102e00 msgr2=0x7f046c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:47:32.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.209+0000 7f0471800700 1 --2- 172.21.15.123:0/928867808 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f046c102e00 0x7f046c1031d0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f045c004940 tx=0x7f045c009e30 comp rx=0 tx=0).stop 2024-03-30T04:47:32.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.209+0000 7f0471800700 1 -- 172.21.15.123:0/928867808 shutdown_connections 2024-03-30T04:47:32.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.209+0000 7f0471800700 1 --2- 172.21.15.123:0/928867808 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f046c1037a0 0x7f046c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:32.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.209+0000 7f0471800700 1 --2- 172.21.15.123:0/928867808 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f046c102e00 0x7f046c1031d0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:32.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.209+0000 7f0471800700 1 -- 172.21.15.123:0/928867808 >> 172.21.15.123:0/928867808 conn(0x7f046c0f89b0 msgr2=0x7f046c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:47:32.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.209+0000 7f0471800700 1 -- 172.21.15.123:0/928867808 shutdown_connections 2024-03-30T04:47:32.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.209+0000 7f0471800700 1 -- 172.21.15.123:0/928867808 wait complete. 2024-03-30T04:47:32.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.210+0000 7f0471800700 1 Processor -- start 2024-03-30T04:47:32.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.210+0000 7f0471800700 1 -- start start 2024-03-30T04:47:32.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.210+0000 7f0471800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f046c102e00 0x7f046c113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:47:32.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.210+0000 7f0471800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f046c1037a0 0x7f046c114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:47:32.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.210+0000 7f0471800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f046c1146b0 con 0x7f046c1037a0 2024-03-30T04:47:32.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.210+0000 7f0471800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f046c10daa0 con 0x7f046c102e00 2024-03-30T04:47:32.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.211+0000 7f046a000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f046c1037a0 0x7f046c114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:47:32.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.211+0000 7f046a000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f046c1037a0 0x7f046c114060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58772/0 (socket says 172.21.15.123:58772) 2024-03-30T04:47:32.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.211+0000 7f046a000700 1 -- 172.21.15.123:0/1420742639 learned_addr learned my addr 172.21.15.123:0/1420742639 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:47:32.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.211+0000 7f046aa00700 1 --2- 172.21.15.123:0/1420742639 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f046c102e00 0x7f046c113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:47:32.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.211+0000 7f046a000700 1 -- 172.21.15.123:0/1420742639 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f046c102e00 msgr2=0x7f046c113b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:47:32.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.211+0000 7f046a000700 1 --2- 172.21.15.123:0/1420742639 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f046c102e00 0x7f046c113b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:32.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.211+0000 7f046a000700 1 -- 172.21.15.123:0/1420742639 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f045c005040 con 0x7f046c1037a0 2024-03-30T04:47:32.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.211+0000 7f046aa00700 1 --2- 172.21.15.123:0/1420742639 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f046c102e00 0x7f046c113b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:47:32.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.211+0000 7f046a000700 1 --2- 172.21.15.123:0/1420742639 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f046c1037a0 0x7f046c114060 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f0458002a50 tx=0x7f0458004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:47:32.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.211+0000 7f0462a00700 1 -- 172.21.15.123:0/1420742639 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0458007cc0 con 0x7f046c1037a0 2024-03-30T04:47:32.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.211+0000 7f0462a00700 1 -- 172.21.15.123:0/1420742639 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f045800f040 con 0x7f046c1037a0 2024-03-30T04:47:32.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.211+0000 7f0471800700 1 -- 172.21.15.123:0/1420742639 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f046c10dd80 con 0x7f046c1037a0 2024-03-30T04:47:32.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.211+0000 7f0462a00700 1 -- 172.21.15.123:0/1420742639 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f045800d9a0 con 0x7f046c1037a0 2024-03-30T04:47:32.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.212+0000 7f0471800700 1 -- 172.21.15.123:0/1420742639 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f046c10e340 con 0x7f046c1037a0 2024-03-30T04:47:32.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.213+0000 7f0471800700 1 -- 172.21.15.123:0/1420742639 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f046c0fe550 con 0x7f046c1037a0 2024-03-30T04:47:32.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.213+0000 7f0462a00700 1 -- 172.21.15.123:0/1420742639 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f045800dc60 con 0x7f046c1037a0 2024-03-30T04:47:32.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.214+0000 7f0462a00700 1 --2- 172.21.15.123:0/1420742639 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f045006cc30 0x7f045006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:47:32.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.214+0000 7f0462a00700 1 -- 172.21.15.123:0/1420742639 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f045808bef0 con 0x7f046c1037a0 2024-03-30T04:47:32.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.215+0000 7f046aa00700 1 --2- 172.21.15.123:0/1420742639 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f045006cc30 0x7f045006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:47:32.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.215+0000 7f046aa00700 1 --2- 172.21.15.123:0/1420742639 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f045006cc30 0x7f045006f0e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f045c00b840 tx=0x7f045c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:47:32.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.219+0000 7f0462a00700 1 -- 172.21.15.123:0/1420742639 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f04580598c0 con 0x7f046c1037a0 2024-03-30T04:47:32.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.439+0000 7f0471800700 1 -- 172.21.15.123:0/1420742639 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f046c1137f0 con 0x7f045006cc30 2024-03-30T04:47:32.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.440+0000 7f0462a00700 1 -- 172.21.15.123:0/1420742639 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f046c1137f0 con 0x7f045006cc30 2024-03-30T04:47:32.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.442+0000 7f0471800700 1 -- 172.21.15.123:0/1420742639 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f045006cc30 msgr2=0x7f045006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:47:32.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.442+0000 7f0471800700 1 --2- 172.21.15.123:0/1420742639 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f045006cc30 0x7f045006f0e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f045c00b840 tx=0x7f045c009e30 comp rx=0 tx=0).stop 2024-03-30T04:47:32.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.442+0000 7f0471800700 1 -- 172.21.15.123:0/1420742639 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f046c1037a0 msgr2=0x7f046c114060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:47:32.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.442+0000 7f0471800700 1 --2- 172.21.15.123:0/1420742639 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f046c1037a0 0x7f046c114060 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f0458002a50 tx=0x7f0458004600 comp rx=0 tx=0).stop 2024-03-30T04:47:32.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.443+0000 7f0471800700 1 -- 172.21.15.123:0/1420742639 shutdown_connections 2024-03-30T04:47:32.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.443+0000 7f0471800700 1 --2- 172.21.15.123:0/1420742639 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f045006cc30 0x7f045006f0e0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:32.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.443+0000 7f0471800700 1 --2- 172.21.15.123:0/1420742639 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f046c102e00 0x7f046c113b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:32.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.443+0000 7f0471800700 1 --2- 172.21.15.123:0/1420742639 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f046c1037a0 0x7f046c114060 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:32.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.443+0000 7f0471800700 1 -- 172.21.15.123:0/1420742639 >> 172.21.15.123:0/1420742639 conn(0x7f046c0f89b0 msgr2=0x7f046c0f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:47:32.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.443+0000 7f0471800700 1 -- 172.21.15.123:0/1420742639 shutdown_connections 2024-03-30T04:47:32.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.443+0000 7f0471800700 1 -- 172.21.15.123:0/1420742639 wait complete. 2024-03-30T04:47:32.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.575+0000 7f0abca00700 1 -- 172.21.15.123:0/587290911 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ab81037a0 msgr2=0x7f0ab80fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:47:32.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.575+0000 7f0abca00700 1 --2- 172.21.15.123:0/587290911 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ab81037a0 0x7f0ab80fce10 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f0aa4004940 tx=0x7f0aa4009e30 comp rx=0 tx=0).stop 2024-03-30T04:47:32.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.575+0000 7f0abca00700 1 -- 172.21.15.123:0/587290911 shutdown_connections 2024-03-30T04:47:32.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.575+0000 7f0abca00700 1 --2- 172.21.15.123:0/587290911 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ab81037a0 0x7f0ab80fce10 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:32.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.575+0000 7f0abca00700 1 --2- 172.21.15.123:0/587290911 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0ab8102e00 0x7f0ab81031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:32.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.575+0000 7f0abca00700 1 -- 172.21.15.123:0/587290911 >> 172.21.15.123:0/587290911 conn(0x7f0ab80f89b0 msgr2=0x7f0ab80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:47:32.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.575+0000 7f0abca00700 1 -- 172.21.15.123:0/587290911 shutdown_connections 2024-03-30T04:47:32.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.575+0000 7f0abca00700 1 -- 172.21.15.123:0/587290911 wait complete. 2024-03-30T04:47:32.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.576+0000 7f0abca00700 1 Processor -- start 2024-03-30T04:47:32.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.576+0000 7f0abca00700 1 -- start start 2024-03-30T04:47:32.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.576+0000 7f0abca00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0ab8102e00 0x7f0ab80702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:47:32.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.576+0000 7f0abca00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ab81037a0 0x7f0ab8070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:47:32.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.577+0000 7f0abca00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab8070d90 con 0x7f0ab81037a0 2024-03-30T04:47:32.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.577+0000 7f0abca00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab8070ed0 con 0x7f0ab8102e00 2024-03-30T04:47:32.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.577+0000 7f0ab5400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ab81037a0 0x7f0ab8070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:47:32.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.577+0000 7f0ab5400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ab81037a0 0x7f0ab8070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58788/0 (socket says 172.21.15.123:58788) 2024-03-30T04:47:32.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.577+0000 7f0ab5400700 1 -- 172.21.15.123:0/3209924664 learned_addr learned my addr 172.21.15.123:0/3209924664 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:47:32.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.577+0000 7f0ab5e00700 1 --2- 172.21.15.123:0/3209924664 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0ab8102e00 0x7f0ab80702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:47:32.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.577+0000 7f0ab5400700 1 -- 172.21.15.123:0/3209924664 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0ab8102e00 msgr2=0x7f0ab80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:47:32.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.577+0000 7f0ab5400700 1 --2- 172.21.15.123:0/3209924664 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0ab8102e00 0x7f0ab80702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:32.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.577+0000 7f0ab5400700 1 -- 172.21.15.123:0/3209924664 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aa4005040 con 0x7f0ab81037a0 2024-03-30T04:47:32.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.577+0000 7f0ab5e00700 1 --2- 172.21.15.123:0/3209924664 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0ab8102e00 0x7f0ab80702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:47:32.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.577+0000 7f0ab5400700 1 --2- 172.21.15.123:0/3209924664 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ab81037a0 0x7f0ab8070800 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f0aa4009390 tx=0x7f0aa4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:47:32.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.578+0000 7f0aae000700 1 -- 172.21.15.123:0/3209924664 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0aa400e040 con 0x7f0ab81037a0 2024-03-30T04:47:32.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.578+0000 7f0aae000700 1 -- 172.21.15.123:0/3209924664 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0aa4013420 con 0x7f0ab81037a0 2024-03-30T04:47:32.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.578+0000 7f0abca00700 1 -- 172.21.15.123:0/3209924664 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ab8071120 con 0x7f0ab81037a0 2024-03-30T04:47:32.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.578+0000 7f0aae000700 1 -- 172.21.15.123:0/3209924664 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0aa401c630 con 0x7f0ab81037a0 2024-03-30T04:47:32.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.578+0000 7f0abca00700 1 -- 172.21.15.123:0/3209924664 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ab811a230 con 0x7f0ab81037a0 2024-03-30T04:47:32.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.579+0000 7f0abca00700 1 -- 172.21.15.123:0/3209924664 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ab80fe550 con 0x7f0ab81037a0 2024-03-30T04:47:32.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.580+0000 7f0aae000700 1 -- 172.21.15.123:0/3209924664 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f0aa40139e0 con 0x7f0ab81037a0 2024-03-30T04:47:32.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.581+0000 7f0aae000700 1 --2- 172.21.15.123:0/3209924664 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0a9c06cc30 0x7f0a9c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:47:32.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.581+0000 7f0aae000700 1 -- 172.21.15.123:0/3209924664 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f0aa408d380 con 0x7f0ab81037a0 2024-03-30T04:47:32.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.581+0000 7f0ab5e00700 1 --2- 172.21.15.123:0/3209924664 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0a9c06cc30 0x7f0a9c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:47:32.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.581+0000 7f0ab5e00700 1 --2- 172.21.15.123:0/3209924664 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0a9c06cc30 0x7f0a9c06f0e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0aa8007fd0 tx=0x7f0aa8009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:47:32.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.585+0000 7f0aae000700 1 -- 172.21.15.123:0/3209924664 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0aa4057340 con 0x7f0ab81037a0 2024-03-30T04:47:32.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.797+0000 7f0abca00700 1 -- 172.21.15.123:0/3209924664 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f0ab8109e10 con 0x7f0a9c06cc30 2024-03-30T04:47:32.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.807+0000 7f0aae000700 1 -- 172.21.15.123:0/3209924664 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f0ab8109e10 con 0x7f0a9c06cc30 2024-03-30T04:47:32.805 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:47:32.805 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (16m) 9m ago 29m 15.0M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:47:32.805 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (13m) 9m ago 29m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T04:47:32.805 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (13m) 9m ago 28m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T04:47:32.805 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (16m) 9m ago 28m 74.0M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T04:47:32.805 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (9m) 9m ago 22m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T04:47:32.805 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (10m) 9m ago 22m 10.9M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T04:47:32.805 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (22m) 9m ago 22m 10.7M - 17.2.7-797-g5c024335 f77085f6aab7 0dc41326818c 2024-03-30T04:47:32.806 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (9m) 9m ago 22m 9747k - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T04:47:32.806 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (18m) 9m ago 30m 506M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:47:32.806 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (17m) 9m ago 28m 450M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:47:32.806 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (15m) 9m ago 31m 53.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T04:47:32.806 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (14m) 9m ago 28m 42.6M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T04:47:32.806 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (17m) 9m ago 29m 8052k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:47:32.806 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (17m) 9m ago 28m 7880k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:47:32.806 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (13m) 9m ago 27m 104M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T04:47:32.806 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (13m) 9m ago 26m 93.4M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T04:47:32.806 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (12m) 9m ago 26m 86.3M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T04:47:32.806 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (11m) 9m ago 25m 97.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T04:47:32.806 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (11m) 9m ago 25m 97.9M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T04:47:32.806 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (10m) 9m ago 24m 85.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T04:47:32.806 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (17m) 9m ago 28m 53.7M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:47:32.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.809+0000 7f0abca00700 1 -- 172.21.15.123:0/3209924664 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0a9c06cc30 msgr2=0x7f0a9c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:47:32.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.809+0000 7f0abca00700 1 --2- 172.21.15.123:0/3209924664 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0a9c06cc30 0x7f0a9c06f0e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0aa8007fd0 tx=0x7f0aa8009040 comp rx=0 tx=0).stop 2024-03-30T04:47:32.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.809+0000 7f0abca00700 1 -- 172.21.15.123:0/3209924664 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ab81037a0 msgr2=0x7f0ab8070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:47:32.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.809+0000 7f0abca00700 1 --2- 172.21.15.123:0/3209924664 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ab81037a0 0x7f0ab8070800 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f0aa4009390 tx=0x7f0aa4009e30 comp rx=0 tx=0).stop 2024-03-30T04:47:32.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.809+0000 7f0abca00700 1 -- 172.21.15.123:0/3209924664 shutdown_connections 2024-03-30T04:47:32.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.809+0000 7f0abca00700 1 --2- 172.21.15.123:0/3209924664 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0a9c06cc30 0x7f0a9c06f0e0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:32.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.809+0000 7f0abca00700 1 --2- 172.21.15.123:0/3209924664 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0ab8102e00 0x7f0ab80702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:32.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.809+0000 7f0abca00700 1 --2- 172.21.15.123:0/3209924664 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ab81037a0 0x7f0ab8070800 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:32.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.809+0000 7f0abca00700 1 -- 172.21.15.123:0/3209924664 >> 172.21.15.123:0/3209924664 conn(0x7f0ab80f89b0 msgr2=0x7f0ab80f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:47:32.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.810+0000 7f0abca00700 1 -- 172.21.15.123:0/3209924664 shutdown_connections 2024-03-30T04:47:32.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.810+0000 7f0abca00700 1 -- 172.21.15.123:0/3209924664 wait complete. 2024-03-30T04:47:32.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.941+0000 7f50ac800700 1 -- 172.21.15.123:0/780035005 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50a41037a0 msgr2=0x7f50a40fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:47:32.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.941+0000 7f50ac800700 1 --2- 172.21.15.123:0/780035005 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50a41037a0 0x7f50a40fce10 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f5094004940 tx=0x7f5094009e30 comp rx=0 tx=0).stop 2024-03-30T04:47:32.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.941+0000 7f50ac800700 1 -- 172.21.15.123:0/780035005 shutdown_connections 2024-03-30T04:47:32.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.941+0000 7f50ac800700 1 --2- 172.21.15.123:0/780035005 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50a41037a0 0x7f50a40fce10 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:32.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.941+0000 7f50ac800700 1 --2- 172.21.15.123:0/780035005 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50a4102e00 0x7f50a41031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:32.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.941+0000 7f50ac800700 1 -- 172.21.15.123:0/780035005 >> 172.21.15.123:0/780035005 conn(0x7f50a40f89b0 msgr2=0x7f50a40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:47:32.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.941+0000 7f50ac800700 1 -- 172.21.15.123:0/780035005 shutdown_connections 2024-03-30T04:47:32.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.941+0000 7f50ac800700 1 -- 172.21.15.123:0/780035005 wait complete. 2024-03-30T04:47:32.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.942+0000 7f50ac800700 1 Processor -- start 2024-03-30T04:47:32.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.942+0000 7f50ac800700 1 -- start start 2024-03-30T04:47:32.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.942+0000 7f50ac800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50a4102e00 0x7f50a40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:47:32.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.942+0000 7f50ac800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50a41037a0 0x7f50a4070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:47:32.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.942+0000 7f50ac800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50a4070d90 con 0x7f50a4102e00 2024-03-30T04:47:32.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.942+0000 7f50ac800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50a4070ed0 con 0x7f50a41037a0 2024-03-30T04:47:32.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.942+0000 7f50a9e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50a4102e00 0x7f50a40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:47:32.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.943+0000 7f50a9e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50a4102e00 0x7f50a40702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58818/0 (socket says 172.21.15.123:58818) 2024-03-30T04:47:32.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.943+0000 7f50a9e00700 1 -- 172.21.15.123:0/20644598 learned_addr learned my addr 172.21.15.123:0/20644598 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:47:32.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.943+0000 7f50a9400700 1 --2- 172.21.15.123:0/20644598 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50a41037a0 0x7f50a4070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:47:32.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.943+0000 7f50a9e00700 1 -- 172.21.15.123:0/20644598 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50a41037a0 msgr2=0x7f50a4070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:47:32.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.943+0000 7f50a9e00700 1 --2- 172.21.15.123:0/20644598 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50a41037a0 0x7f50a4070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:32.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.943+0000 7f50a9e00700 1 -- 172.21.15.123:0/20644598 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5094005040 con 0x7f50a4102e00 2024-03-30T04:47:32.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.943+0000 7f50a9e00700 1 --2- 172.21.15.123:0/20644598 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50a4102e00 0x7f50a40702c0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f5098002a50 tx=0x7f50980046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:47:32.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.943+0000 7f509e000700 1 -- 172.21.15.123:0/20644598 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5098007cc0 con 0x7f50a4102e00 2024-03-30T04:47:32.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.943+0000 7f509e000700 1 -- 172.21.15.123:0/20644598 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f509800f040 con 0x7f50a4102e00 2024-03-30T04:47:32.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.943+0000 7f509e000700 1 -- 172.21.15.123:0/20644598 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f509800d9c0 con 0x7f50a4102e00 2024-03-30T04:47:32.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.943+0000 7f50ac800700 1 -- 172.21.15.123:0/20644598 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50a4119e00 con 0x7f50a4102e00 2024-03-30T04:47:32.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.944+0000 7f50ac800700 1 -- 172.21.15.123:0/20644598 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50a411a280 con 0x7f50a4102e00 2024-03-30T04:47:32.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.945+0000 7f50ac800700 1 -- 172.21.15.123:0/20644598 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50a40fe550 con 0x7f50a4102e00 2024-03-30T04:47:32.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.945+0000 7f509e000700 1 -- 172.21.15.123:0/20644598 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f509800db20 con 0x7f50a4102e00 2024-03-30T04:47:32.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.946+0000 7f509e000700 1 --2- 172.21.15.123:0/20644598 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f508c06cc30 0x7f508c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:47:32.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.946+0000 7f509e000700 1 -- 172.21.15.123:0/20644598 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f509808bd00 con 0x7f50a4102e00 2024-03-30T04:47:32.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.947+0000 7f50a9400700 1 --2- 172.21.15.123:0/20644598 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f508c06cc30 0x7f508c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:47:32.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.947+0000 7f50a9400700 1 --2- 172.21.15.123:0/20644598 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f508c06cc30 0x7f508c06f0e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f50a4071c00 tx=0x7f5094009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:47:32.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:32.951+0000 7f509e000700 1 -- 172.21.15.123:0/20644598 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5098055cc0 con 0x7f50a4102e00 2024-03-30T04:47:32.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:47:32 smithi123 ceph-mon[139369]: pgmap v490: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:33.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:47:32 smithi135 ceph-mon[106011]: pgmap v490: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:33.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.235+0000 7f50ac800700 1 -- 172.21.15.123:0/20644598 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f50a40719b0 con 0x7f50a4102e00 2024-03-30T04:47:33.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.236+0000 7f509e000700 1 -- 172.21.15.123:0/20644598 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f50980592e0 con 0x7f50a4102e00 2024-03-30T04:47:33.234 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:47:33.234 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:47:33.234 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:47:33.235 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:47:33.235 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:47:33.235 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:47:33.235 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:47:33.235 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:47:33.235 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T04:47:33.235 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:47:33.235 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:47:33.235 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:47:33.235 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 3 2024-03-30T04:47:33.235 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:47:33.235 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:47:33.235 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:47:33.236 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 13 2024-03-30T04:47:33.236 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:47:33.236 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:47:33.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.238+0000 7f50ac800700 1 -- 172.21.15.123:0/20644598 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f508c06cc30 msgr2=0x7f508c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:47:33.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.238+0000 7f50ac800700 1 --2- 172.21.15.123:0/20644598 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f508c06cc30 0x7f508c06f0e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f50a4071c00 tx=0x7f5094009e30 comp rx=0 tx=0).stop 2024-03-30T04:47:33.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.239+0000 7f50ac800700 1 -- 172.21.15.123:0/20644598 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50a4102e00 msgr2=0x7f50a40702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:47:33.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.239+0000 7f50ac800700 1 --2- 172.21.15.123:0/20644598 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50a4102e00 0x7f50a40702c0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f5098002a50 tx=0x7f50980046b0 comp rx=0 tx=0).stop 2024-03-30T04:47:33.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.239+0000 7f50ac800700 1 -- 172.21.15.123:0/20644598 shutdown_connections 2024-03-30T04:47:33.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.239+0000 7f50ac800700 1 --2- 172.21.15.123:0/20644598 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f508c06cc30 0x7f508c06f0e0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:33.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.239+0000 7f50ac800700 1 --2- 172.21.15.123:0/20644598 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50a4102e00 0x7f50a40702c0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:33.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.239+0000 7f50ac800700 1 --2- 172.21.15.123:0/20644598 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50a41037a0 0x7f50a4070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:33.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.239+0000 7f50ac800700 1 -- 172.21.15.123:0/20644598 >> 172.21.15.123:0/20644598 conn(0x7f50a40f89b0 msgr2=0x7f50a40f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:47:33.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.239+0000 7f50ac800700 1 -- 172.21.15.123:0/20644598 shutdown_connections 2024-03-30T04:47:33.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.239+0000 7f50ac800700 1 -- 172.21.15.123:0/20644598 wait complete. 2024-03-30T04:47:33.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.371+0000 7f7669800700 1 -- 172.21.15.123:0/48143318 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7664102e20 msgr2=0x7f76641031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:47:33.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.371+0000 7f7669800700 1 --2- 172.21.15.123:0/48143318 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7664102e20 0x7f76641031f0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f7654004940 tx=0x7f7654009e30 comp rx=0 tx=0).stop 2024-03-30T04:47:33.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.371+0000 7f7669800700 1 -- 172.21.15.123:0/48143318 shutdown_connections 2024-03-30T04:47:33.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.371+0000 7f7669800700 1 --2- 172.21.15.123:0/48143318 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f76641037c0 0x7f76640fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:33.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.371+0000 7f7669800700 1 --2- 172.21.15.123:0/48143318 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7664102e20 0x7f76641031f0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:33.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.371+0000 7f7669800700 1 -- 172.21.15.123:0/48143318 >> 172.21.15.123:0/48143318 conn(0x7f76640f89d0 msgr2=0x7f76640fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:47:33.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.371+0000 7f7669800700 1 -- 172.21.15.123:0/48143318 shutdown_connections 2024-03-30T04:47:33.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.371+0000 7f7669800700 1 -- 172.21.15.123:0/48143318 wait complete. 2024-03-30T04:47:33.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.372+0000 7f7669800700 1 Processor -- start 2024-03-30T04:47:33.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.372+0000 7f7669800700 1 -- start start 2024-03-30T04:47:33.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.373+0000 7f7669800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7664102e20 0x7f7664113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:47:33.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.373+0000 7f7669800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76641037c0 0x7f7664114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:47:33.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.373+0000 7f7669800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7664114690 con 0x7f76641037c0 2024-03-30T04:47:33.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.373+0000 7f7669800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f766410da80 con 0x7f7664102e20 2024-03-30T04:47:33.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.373+0000 7f7662000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76641037c0 0x7f7664114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:47:33.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.373+0000 7f7662000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76641037c0 0x7f7664114040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58830/0 (socket says 172.21.15.123:58830) 2024-03-30T04:47:33.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.373+0000 7f7662000700 1 -- 172.21.15.123:0/1140101934 learned_addr learned my addr 172.21.15.123:0/1140101934 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:47:33.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.373+0000 7f7662a00700 1 --2- 172.21.15.123:0/1140101934 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7664102e20 0x7f7664113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:47:33.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.374+0000 7f7662000700 1 -- 172.21.15.123:0/1140101934 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7664102e20 msgr2=0x7f7664113b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:47:33.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.374+0000 7f7662000700 1 --2- 172.21.15.123:0/1140101934 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7664102e20 0x7f7664113b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:33.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.374+0000 7f7662000700 1 -- 172.21.15.123:0/1140101934 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7654005040 con 0x7f76641037c0 2024-03-30T04:47:33.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.374+0000 7f7662a00700 1 --2- 172.21.15.123:0/1140101934 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7664102e20 0x7f7664113b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:47:33.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.374+0000 7f7662000700 1 --2- 172.21.15.123:0/1140101934 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76641037c0 0x7f7664114040 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f7650002a50 tx=0x7f7650004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:47:33.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.374+0000 7f765aa00700 1 -- 172.21.15.123:0/1140101934 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7650007cc0 con 0x7f76641037c0 2024-03-30T04:47:33.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.374+0000 7f765aa00700 1 -- 172.21.15.123:0/1140101934 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f765000f040 con 0x7f76641037c0 2024-03-30T04:47:33.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.374+0000 7f7669800700 1 -- 172.21.15.123:0/1140101934 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f766410dd60 con 0x7f76641037c0 2024-03-30T04:47:33.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.374+0000 7f765aa00700 1 -- 172.21.15.123:0/1140101934 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f765000d9a0 con 0x7f76641037c0 2024-03-30T04:47:33.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.374+0000 7f7669800700 1 -- 172.21.15.123:0/1140101934 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f766410e320 con 0x7f76641037c0 2024-03-30T04:47:33.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.376+0000 7f7669800700 1 -- 172.21.15.123:0/1140101934 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76640fe570 con 0x7f76641037c0 2024-03-30T04:47:33.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.376+0000 7f765aa00700 1 -- 172.21.15.123:0/1140101934 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f765000dc10 con 0x7f76641037c0 2024-03-30T04:47:33.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.377+0000 7f765aa00700 1 --2- 172.21.15.123:0/1140101934 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f764806cc30 0x7f764806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:47:33.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.377+0000 7f765aa00700 1 -- 172.21.15.123:0/1140101934 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f765008bdb0 con 0x7f76641037c0 2024-03-30T04:47:33.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.378+0000 7f7662a00700 1 --2- 172.21.15.123:0/1140101934 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f764806cc30 0x7f764806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:47:33.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.378+0000 7f7662a00700 1 --2- 172.21.15.123:0/1140101934 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f764806cc30 0x7f764806f0e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f7654000c00 tx=0x7f7654009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:47:33.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.382+0000 7f765aa00700 1 -- 172.21.15.123:0/1140101934 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7650059880 con 0x7f76641037c0 2024-03-30T04:47:33.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.626+0000 7f7669800700 1 -- 172.21.15.123:0/1140101934 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f7664109de0 con 0x7f76641037c0 2024-03-30T04:47:33.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.626+0000 7f765aa00700 1 -- 172.21.15.123:0/1140101934 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f7650059410 con 0x7f76641037c0 2024-03-30T04:47:33.625 INFO:teuthology.orchestra.run.smithi123.stdout:e32 2024-03-30T04:47:33.625 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:47:33.625 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:47:33.625 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:47:33.625 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:47:33.625 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:47:33.625 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:47:33.625 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T04:47:33.625 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T04:47:33.625 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:47:33.626 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T04:47:33.626 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:47:33.626 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:47:33.626 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:47:33.626 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:47:33.626 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:47:33.626 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T04:47:33.626 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:47:33.626 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:47:33.626 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T04:47:33.626 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:47:33.626 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T04:47:33.626 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T04:47:33.626 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T04:47:33.627 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T04:47:33.627 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:47:33.627 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:47:33.627 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:47:33.627 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:47:33.627 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:47:33.627 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:47:33.627 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T04:47:33.627 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T04:47:33.627 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:47:33.627 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:47:33.627 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T04:47:33.627 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:47:33.627 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:47:33.628 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:47:33.628 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44315} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/1741352336,v1:172.21.15.135:6827/1741352336] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:47:33.628 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:47:33.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.629+0000 7f7669800700 1 -- 172.21.15.123:0/1140101934 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f764806cc30 msgr2=0x7f764806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:47:33.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.629+0000 7f7669800700 1 --2- 172.21.15.123:0/1140101934 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f764806cc30 0x7f764806f0e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f7654000c00 tx=0x7f7654009e30 comp rx=0 tx=0).stop 2024-03-30T04:47:33.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.629+0000 7f7669800700 1 -- 172.21.15.123:0/1140101934 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76641037c0 msgr2=0x7f7664114040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:47:33.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.629+0000 7f7669800700 1 --2- 172.21.15.123:0/1140101934 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76641037c0 0x7f7664114040 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f7650002a50 tx=0x7f7650004600 comp rx=0 tx=0).stop 2024-03-30T04:47:33.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.629+0000 7f7669800700 1 -- 172.21.15.123:0/1140101934 shutdown_connections 2024-03-30T04:47:33.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.629+0000 7f7669800700 1 --2- 172.21.15.123:0/1140101934 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f764806cc30 0x7f764806f0e0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:33.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.629+0000 7f7669800700 1 --2- 172.21.15.123:0/1140101934 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7664102e20 0x7f7664113b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:33.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.629+0000 7f7669800700 1 --2- 172.21.15.123:0/1140101934 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76641037c0 0x7f7664114040 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:33.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.629+0000 7f7669800700 1 -- 172.21.15.123:0/1140101934 >> 172.21.15.123:0/1140101934 conn(0x7f76640f89d0 msgr2=0x7f76640f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:47:33.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.629+0000 7f7669800700 1 -- 172.21.15.123:0/1140101934 shutdown_connections 2024-03-30T04:47:33.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.629+0000 7f7669800700 1 -- 172.21.15.123:0/1140101934 wait complete. 2024-03-30T04:47:33.629 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 32 2024-03-30T04:47:33.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.763+0000 7f7f48400700 1 -- 172.21.15.123:0/402389990 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f401037a0 msgr2=0x7f7f400fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:47:33.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.763+0000 7f7f48400700 1 --2- 172.21.15.123:0/402389990 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f401037a0 0x7f7f400fce10 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c004940 tx=0x7f7f2c009e30 comp rx=0 tx=0).stop 2024-03-30T04:47:33.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.763+0000 7f7f48400700 1 -- 172.21.15.123:0/402389990 shutdown_connections 2024-03-30T04:47:33.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.763+0000 7f7f48400700 1 --2- 172.21.15.123:0/402389990 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f401037a0 0x7f7f400fce10 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:33.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.763+0000 7f7f48400700 1 --2- 172.21.15.123:0/402389990 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f40102e00 0x7f7f401031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:33.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.763+0000 7f7f48400700 1 -- 172.21.15.123:0/402389990 >> 172.21.15.123:0/402389990 conn(0x7f7f400f89b0 msgr2=0x7f7f400fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:47:33.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.764+0000 7f7f48400700 1 -- 172.21.15.123:0/402389990 shutdown_connections 2024-03-30T04:47:33.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.764+0000 7f7f48400700 1 -- 172.21.15.123:0/402389990 wait complete. 2024-03-30T04:47:33.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.765+0000 7f7f48400700 1 Processor -- start 2024-03-30T04:47:33.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.765+0000 7f7f48400700 1 -- start start 2024-03-30T04:47:33.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.765+0000 7f7f48400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f40102e00 0x7f7f40113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:47:33.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.765+0000 7f7f48400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f401037a0 0x7f7f40114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:47:33.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.765+0000 7f7f48400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f40114690 con 0x7f7f401037a0 2024-03-30T04:47:33.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.765+0000 7f7f48400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4010da80 con 0x7f7f40102e00 2024-03-30T04:47:33.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.766+0000 7f7f45000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f401037a0 0x7f7f40114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:47:33.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.766+0000 7f7f45000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f401037a0 0x7f7f40114040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58840/0 (socket says 172.21.15.123:58840) 2024-03-30T04:47:33.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.766+0000 7f7f45000700 1 -- 172.21.15.123:0/1091725436 learned_addr learned my addr 172.21.15.123:0/1091725436 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:47:33.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.766+0000 7f7f45a00700 1 --2- 172.21.15.123:0/1091725436 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f40102e00 0x7f7f40113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:47:33.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.766+0000 7f7f45000700 1 -- 172.21.15.123:0/1091725436 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f40102e00 msgr2=0x7f7f40113b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:47:33.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.766+0000 7f7f45000700 1 --2- 172.21.15.123:0/1091725436 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f40102e00 0x7f7f40113b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:33.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.766+0000 7f7f45000700 1 -- 172.21.15.123:0/1091725436 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f2c005040 con 0x7f7f401037a0 2024-03-30T04:47:33.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.766+0000 7f7f45000700 1 --2- 172.21.15.123:0/1091725436 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f401037a0 0x7f7f40114040 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c009390 tx=0x7f7f2c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:47:33.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.767+0000 7f7f3d600700 1 -- 172.21.15.123:0/1091725436 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f2c00e040 con 0x7f7f401037a0 2024-03-30T04:47:33.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.767+0000 7f7f3d600700 1 -- 172.21.15.123:0/1091725436 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7f2c013420 con 0x7f7f401037a0 2024-03-30T04:47:33.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.767+0000 7f7f3d600700 1 -- 172.21.15.123:0/1091725436 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f2c01c630 con 0x7f7f401037a0 2024-03-30T04:47:33.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.767+0000 7f7f48400700 1 -- 172.21.15.123:0/1091725436 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f4010dd00 con 0x7f7f401037a0 2024-03-30T04:47:33.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.767+0000 7f7f48400700 1 -- 172.21.15.123:0/1091725436 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f4010e2c0 con 0x7f7f401037a0 2024-03-30T04:47:33.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.768+0000 7f7f48400700 1 -- 172.21.15.123:0/1091725436 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f400fe550 con 0x7f7f401037a0 2024-03-30T04:47:33.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.769+0000 7f7f3d600700 1 -- 172.21.15.123:0/1091725436 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f7f2c013a00 con 0x7f7f401037a0 2024-03-30T04:47:33.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.770+0000 7f7f3d600700 1 --2- 172.21.15.123:0/1091725436 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7f2806cc30 0x7f7f2806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:47:33.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.770+0000 7f7f3d600700 1 -- 172.21.15.123:0/1091725436 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7f2c08d020 con 0x7f7f401037a0 2024-03-30T04:47:33.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.770+0000 7f7f45a00700 1 --2- 172.21.15.123:0/1091725436 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7f2806cc30 0x7f7f2806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:47:33.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.771+0000 7f7f45a00700 1 --2- 172.21.15.123:0/1091725436 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7f2806cc30 0x7f7f2806f0e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f7f34004f40 tx=0x7f7f3400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:47:33.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.774+0000 7f7f3d600700 1 -- 172.21.15.123:0/1091725436 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7f2c05abb0 con 0x7f7f401037a0 2024-03-30T04:47:33.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:47:33 smithi123 ceph-mon[139369]: from='client.34934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:47:33.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:47:33 smithi123 ceph-mon[139369]: from='client.34938 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:47:33.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:47:33 smithi123 ceph-mon[139369]: from='client.34942 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:47:33.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:47:33 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/20644598' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:47:33.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:47:33 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/1140101934' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:47:33.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.997+0000 7f7f48400700 1 -- 172.21.15.123:0/1091725436 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7f40109da0 con 0x7f7f2806cc30 2024-03-30T04:47:33.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:33.998+0000 7f7f3d600700 1 -- 172.21.15.123:0/1091725436 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f7f40109da0 con 0x7f7f2806cc30 2024-03-30T04:47:33.996 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:47:33.996 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:47:33.996 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:47:33.996 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T04:47:33.996 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:47:33.996 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T04:47:33.997 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T04:47:33.997 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T04:47:33.997 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T04:47:33.997 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:47:33.997 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "15/21 daemons upgraded", 2024-03-30T04:47:33.997 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T04:47:33.997 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:47:33.997 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:47:33.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.000+0000 7f7f48400700 1 -- 172.21.15.123:0/1091725436 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7f2806cc30 msgr2=0x7f7f2806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:47:33.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.000+0000 7f7f48400700 1 --2- 172.21.15.123:0/1091725436 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7f2806cc30 0x7f7f2806f0e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f7f34004f40 tx=0x7f7f3400b040 comp rx=0 tx=0).stop 2024-03-30T04:47:33.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.000+0000 7f7f48400700 1 -- 172.21.15.123:0/1091725436 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f401037a0 msgr2=0x7f7f40114040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:47:33.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.000+0000 7f7f48400700 1 --2- 172.21.15.123:0/1091725436 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f401037a0 0x7f7f40114040 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c009390 tx=0x7f7f2c009e30 comp rx=0 tx=0).stop 2024-03-30T04:47:33.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.000+0000 7f7f48400700 1 -- 172.21.15.123:0/1091725436 shutdown_connections 2024-03-30T04:47:33.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.000+0000 7f7f48400700 1 --2- 172.21.15.123:0/1091725436 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7f2806cc30 0x7f7f2806f0e0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:33.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.000+0000 7f7f48400700 1 --2- 172.21.15.123:0/1091725436 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f40102e00 0x7f7f40113b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:33.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.000+0000 7f7f48400700 1 --2- 172.21.15.123:0/1091725436 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f401037a0 0x7f7f40114040 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:33.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.000+0000 7f7f48400700 1 -- 172.21.15.123:0/1091725436 >> 172.21.15.123:0/1091725436 conn(0x7f7f400f89b0 msgr2=0x7f7f400f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:47:33.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.001+0000 7f7f48400700 1 -- 172.21.15.123:0/1091725436 shutdown_connections 2024-03-30T04:47:33.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.001+0000 7f7f48400700 1 -- 172.21.15.123:0/1091725436 wait complete. 2024-03-30T04:47:34.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:47:33 smithi135 ceph-mon[106011]: from='client.34934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:47:34.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:47:33 smithi135 ceph-mon[106011]: from='client.34938 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:47:34.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:47:33 smithi135 ceph-mon[106011]: from='client.34942 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:47:34.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:47:33 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/20644598' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:47:34.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:47:33 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/1140101934' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:47:34.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.134+0000 7f27ab600700 1 -- 172.21.15.123:0/3259069625 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27a4102e00 msgr2=0x7f27a41031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:47:34.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.134+0000 7f27ab600700 1 --2- 172.21.15.123:0/3259069625 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27a4102e00 0x7f27a41031d0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f2798004940 tx=0x7f2798009e30 comp rx=0 tx=0).stop 2024-03-30T04:47:34.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.134+0000 7f27ab600700 1 -- 172.21.15.123:0/3259069625 shutdown_connections 2024-03-30T04:47:34.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.134+0000 7f27ab600700 1 --2- 172.21.15.123:0/3259069625 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f27a41037a0 0x7f27a40fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:34.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.134+0000 7f27ab600700 1 --2- 172.21.15.123:0/3259069625 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27a4102e00 0x7f27a41031d0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:34.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.134+0000 7f27ab600700 1 -- 172.21.15.123:0/3259069625 >> 172.21.15.123:0/3259069625 conn(0x7f27a40f89b0 msgr2=0x7f27a40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:47:34.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.134+0000 7f27ab600700 1 -- 172.21.15.123:0/3259069625 shutdown_connections 2024-03-30T04:47:34.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.134+0000 7f27ab600700 1 -- 172.21.15.123:0/3259069625 wait complete. 2024-03-30T04:47:34.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.135+0000 7f27ab600700 1 Processor -- start 2024-03-30T04:47:34.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.135+0000 7f27ab600700 1 -- start start 2024-03-30T04:47:34.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.135+0000 7f27ab600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27a4102e00 0x7f27a4115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:47:34.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.135+0000 7f27ab600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f27a41037a0 0x7f27a41161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:47:34.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.135+0000 7f27ab600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a4116810 con 0x7f27a4102e00 2024-03-30T04:47:34.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.135+0000 7f27ab600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a410fbe0 con 0x7f27a41037a0 2024-03-30T04:47:34.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.136+0000 7f27a8c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27a4102e00 0x7f27a4115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:47:34.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.136+0000 7f27a8c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27a4102e00 0x7f27a4115c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41044/0 (socket says 172.21.15.123:41044) 2024-03-30T04:47:34.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.136+0000 7f27a8c00700 1 -- 172.21.15.123:0/4114135546 learned_addr learned my addr 172.21.15.123:0/4114135546 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:47:34.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.136+0000 7f27a8c00700 1 -- 172.21.15.123:0/4114135546 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f27a41037a0 msgr2=0x7f27a41161c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:47:34.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.136+0000 7f27a8c00700 1 --2- 172.21.15.123:0/4114135546 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f27a41037a0 0x7f27a41161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:34.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.136+0000 7f27a8c00700 1 -- 172.21.15.123:0/4114135546 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2798005040 con 0x7f27a4102e00 2024-03-30T04:47:34.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.136+0000 7f27a8c00700 1 --2- 172.21.15.123:0/4114135546 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27a4102e00 0x7f27a4115c80 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f2798009390 tx=0x7f2798009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:47:34.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.136+0000 7f27a0c00700 1 -- 172.21.15.123:0/4114135546 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f279800e040 con 0x7f27a4102e00 2024-03-30T04:47:34.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.136+0000 7f27a0c00700 1 -- 172.21.15.123:0/4114135546 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2798013420 con 0x7f27a4102e00 2024-03-30T04:47:34.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.136+0000 7f27a0c00700 1 -- 172.21.15.123:0/4114135546 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f279801c630 con 0x7f27a4102e00 2024-03-30T04:47:34.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.136+0000 7f27ab600700 1 -- 172.21.15.123:0/4114135546 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27a410fe60 con 0x7f27a4102e00 2024-03-30T04:47:34.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.137+0000 7f27ab600700 1 -- 172.21.15.123:0/4114135546 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27a41103d0 con 0x7f27a4102e00 2024-03-30T04:47:34.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.138+0000 7f27ab600700 1 -- 172.21.15.123:0/4114135546 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27a40fe550 con 0x7f27a4102e00 2024-03-30T04:47:34.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.138+0000 7f27a0c00700 1 -- 172.21.15.123:0/4114135546 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f27980139e0 con 0x7f27a4102e00 2024-03-30T04:47:34.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.140+0000 7f27a0c00700 1 --2- 172.21.15.123:0/4114135546 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f278c06cc30 0x7f278c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:47:34.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.140+0000 7f27a0c00700 1 -- 172.21.15.123:0/4114135546 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f279808d320 con 0x7f27a4102e00 2024-03-30T04:47:34.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.140+0000 7f27a3e00700 1 --2- 172.21.15.123:0/4114135546 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f278c06cc30 0x7f278c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:47:34.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.141+0000 7f27a3e00700 1 --2- 172.21.15.123:0/4114135546 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f278c06cc30 0x7f278c06f0e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2790009b80 tx=0x7f2790009d60 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:47:34.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.144+0000 7f27a0c00700 1 -- 172.21.15.123:0/4114135546 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f279805ae20 con 0x7f27a4102e00 2024-03-30T04:47:34.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.426+0000 7f27ab600700 1 -- 172.21.15.123:0/4114135546 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f27a404bed0 con 0x7f27a4102e00 2024-03-30T04:47:34.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.427+0000 7f27a0c00700 1 -- 172.21.15.123:0/4114135546 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f279805a9b0 con 0x7f27a4102e00 2024-03-30T04:47:34.425 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T04:47:34.425 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T04:47:34.425 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T04:47:34.425 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T04:47:34.426 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T04:47:34.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.429+0000 7f27ab600700 1 -- 172.21.15.123:0/4114135546 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f278c06cc30 msgr2=0x7f278c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:47:34.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.429+0000 7f27ab600700 1 --2- 172.21.15.123:0/4114135546 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f278c06cc30 0x7f278c06f0e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2790009b80 tx=0x7f2790009d60 comp rx=0 tx=0).stop 2024-03-30T04:47:34.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.429+0000 7f27ab600700 1 -- 172.21.15.123:0/4114135546 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27a4102e00 msgr2=0x7f27a4115c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:47:34.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.429+0000 7f27ab600700 1 --2- 172.21.15.123:0/4114135546 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27a4102e00 0x7f27a4115c80 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f2798009390 tx=0x7f2798009e30 comp rx=0 tx=0).stop 2024-03-30T04:47:34.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.429+0000 7f27ab600700 1 -- 172.21.15.123:0/4114135546 shutdown_connections 2024-03-30T04:47:34.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.429+0000 7f27ab600700 1 --2- 172.21.15.123:0/4114135546 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f278c06cc30 0x7f278c06f0e0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:34.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.429+0000 7f27ab600700 1 --2- 172.21.15.123:0/4114135546 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27a4102e00 0x7f27a4115c80 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:34.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.429+0000 7f27ab600700 1 --2- 172.21.15.123:0/4114135546 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f27a41037a0 0x7f27a41161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:47:34.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.429+0000 7f27ab600700 1 -- 172.21.15.123:0/4114135546 >> 172.21.15.123:0/4114135546 conn(0x7f27a40f89b0 msgr2=0x7f27a40f9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:47:34.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.429+0000 7f27ab600700 1 -- 172.21.15.123:0/4114135546 shutdown_connections 2024-03-30T04:47:34.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:47:34.429+0000 7f27ab600700 1 -- 172.21.15.123:0/4114135546 wait complete. 2024-03-30T04:47:34.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:47:34 smithi123 ceph-mon[139369]: pgmap v491: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:34.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:47:34 smithi123 ceph-mon[139369]: from='client.34954 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:47:34.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:47:34 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/4114135546' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:47:35.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:47:34 smithi135 ceph-mon[106011]: pgmap v491: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:35.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:47:34 smithi135 ceph-mon[106011]: from='client.34954 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:47:35.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:47:34 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/4114135546' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:47:36.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:47:36 smithi123 ceph-mon[139369]: pgmap v492: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:37.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:47:36 smithi135 ceph-mon[106011]: pgmap v492: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:38.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:47:38 smithi123 ceph-mon[139369]: pgmap v493: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:39.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:47:38 smithi135 ceph-mon[106011]: pgmap v493: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:40.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:47:40 smithi135 ceph-mon[106011]: pgmap v494: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:40.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:47:40 smithi123 ceph-mon[139369]: pgmap v494: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:42.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:47:42 smithi123 ceph-mon[139369]: pgmap v495: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:43.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:47:42 smithi135 ceph-mon[106011]: pgmap v495: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:44.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:47:44 smithi123 ceph-mon[139369]: pgmap v496: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:44.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:47:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:47:45.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:47:44 smithi135 ceph-mon[106011]: pgmap v496: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:45.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:47:44 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:47:46.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:47:46 smithi123 ceph-mon[139369]: pgmap v497: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:47.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:47:46 smithi135 ceph-mon[106011]: pgmap v497: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:48.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:47:48 smithi123 ceph-mon[139369]: pgmap v498: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:49.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:47:48 smithi135 ceph-mon[106011]: pgmap v498: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:50.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:47:50 smithi135 ceph-mon[106011]: pgmap v499: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:50.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:47:50 smithi123 ceph-mon[139369]: pgmap v499: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:52.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:47:52 smithi123 ceph-mon[139369]: pgmap v500: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:53.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:47:52 smithi135 ceph-mon[106011]: pgmap v500: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:54.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:47:54 smithi123 ceph-mon[139369]: pgmap v501: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:55.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:47:54 smithi135 ceph-mon[106011]: pgmap v501: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:56.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:47:56 smithi123 ceph-mon[139369]: pgmap v502: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:57.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:47:56 smithi135 ceph-mon[106011]: pgmap v502: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:58.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:47:58 smithi123 ceph-mon[139369]: pgmap v503: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:59.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:47:58 smithi135 ceph-mon[106011]: pgmap v503: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:47:59.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:47:59 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:48:00.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:47:59 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:48:00.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:48:00 smithi123 ceph-mon[139369]: pgmap v504: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:01.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:48:00 smithi135 ceph-mon[106011]: pgmap v504: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:02.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:48:02 smithi123 ceph-mon[139369]: pgmap v505: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:03.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:48:02 smithi135 ceph-mon[106011]: pgmap v505: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:04.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.579+0000 7faf53e00700 1 -- 172.21.15.123:0/2931778883 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf4c1037a0 msgr2=0x7faf4c0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:04.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.579+0000 7faf53e00700 1 --2- 172.21.15.123:0/2931778883 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf4c1037a0 0x7faf4c0fce10 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7faf38004940 tx=0x7faf38009e30 comp rx=0 tx=0).stop 2024-03-30T04:48:04.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.579+0000 7faf53e00700 1 -- 172.21.15.123:0/2931778883 shutdown_connections 2024-03-30T04:48:04.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.579+0000 7faf53e00700 1 --2- 172.21.15.123:0/2931778883 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf4c1037a0 0x7faf4c0fce10 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:04.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.579+0000 7faf53e00700 1 --2- 172.21.15.123:0/2931778883 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faf4c102e00 0x7faf4c1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:04.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.579+0000 7faf53e00700 1 -- 172.21.15.123:0/2931778883 >> 172.21.15.123:0/2931778883 conn(0x7faf4c0f89b0 msgr2=0x7faf4c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:48:04.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.579+0000 7faf53e00700 1 -- 172.21.15.123:0/2931778883 shutdown_connections 2024-03-30T04:48:04.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.579+0000 7faf53e00700 1 -- 172.21.15.123:0/2931778883 wait complete. 2024-03-30T04:48:04.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.580+0000 7faf53e00700 1 Processor -- start 2024-03-30T04:48:04.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.580+0000 7faf53e00700 1 -- start start 2024-03-30T04:48:04.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.580+0000 7faf53e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faf4c102e00 0x7faf4c115da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:48:04.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.580+0000 7faf53e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf4c1037a0 0x7faf4c1162e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:48:04.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.580+0000 7faf53e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf4c116930 con 0x7faf4c1037a0 2024-03-30T04:48:04.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.580+0000 7faf53e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf4c10fd00 con 0x7faf4c102e00 2024-03-30T04:48:04.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.580+0000 7faf50a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf4c1037a0 0x7faf4c1162e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:48:04.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.581+0000 7faf50a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf4c1037a0 0x7faf4c1162e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58896/0 (socket says 172.21.15.123:58896) 2024-03-30T04:48:04.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.581+0000 7faf50a00700 1 -- 172.21.15.123:0/867850691 learned_addr learned my addr 172.21.15.123:0/867850691 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:48:04.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.581+0000 7faf51400700 1 --2- 172.21.15.123:0/867850691 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faf4c102e00 0x7faf4c115da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:48:04.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.581+0000 7faf50a00700 1 -- 172.21.15.123:0/867850691 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faf4c102e00 msgr2=0x7faf4c115da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:04.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.581+0000 7faf50a00700 1 --2- 172.21.15.123:0/867850691 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faf4c102e00 0x7faf4c115da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:04.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.581+0000 7faf50a00700 1 -- 172.21.15.123:0/867850691 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf38005040 con 0x7faf4c1037a0 2024-03-30T04:48:04.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.581+0000 7faf50a00700 1 --2- 172.21.15.123:0/867850691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf4c1037a0 0x7faf4c1162e0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7faf38009390 tx=0x7faf38009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:48:04.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.581+0000 7faf49600700 1 -- 172.21.15.123:0/867850691 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf3800e040 con 0x7faf4c1037a0 2024-03-30T04:48:04.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.582+0000 7faf49600700 1 -- 172.21.15.123:0/867850691 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faf38013420 con 0x7faf4c1037a0 2024-03-30T04:48:04.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.582+0000 7faf53e00700 1 -- 172.21.15.123:0/867850691 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf4c10ff80 con 0x7faf4c1037a0 2024-03-30T04:48:04.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.582+0000 7faf49600700 1 -- 172.21.15.123:0/867850691 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf3801c630 con 0x7faf4c1037a0 2024-03-30T04:48:04.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.582+0000 7faf53e00700 1 -- 172.21.15.123:0/867850691 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf4c1104c0 con 0x7faf4c1037a0 2024-03-30T04:48:04.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.583+0000 7faf53e00700 1 -- 172.21.15.123:0/867850691 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf4c0fe550 con 0x7faf4c1037a0 2024-03-30T04:48:04.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.584+0000 7faf49600700 1 -- 172.21.15.123:0/867850691 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7faf38013a00 con 0x7faf4c1037a0 2024-03-30T04:48:04.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.584+0000 7faf49600700 1 --2- 172.21.15.123:0/867850691 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7faf3406cc30 0x7faf3406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:48:04.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.584+0000 7faf49600700 1 -- 172.21.15.123:0/867850691 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7faf3808d0e0 con 0x7faf4c1037a0 2024-03-30T04:48:04.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.585+0000 7faf51400700 1 --2- 172.21.15.123:0/867850691 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7faf3406cc30 0x7faf3406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:48:04.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.585+0000 7faf51400700 1 --2- 172.21.15.123:0/867850691 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7faf3406cc30 0x7faf3406f0e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7faf400041d0 tx=0x7faf400043c0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:48:04.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.589+0000 7faf49600700 1 -- 172.21.15.123:0/867850691 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7faf3805abb0 con 0x7faf4c1037a0 2024-03-30T04:48:04.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.808+0000 7faf53e00700 1 -- 172.21.15.123:0/867850691 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7faf4c109e10 con 0x7faf3406cc30 2024-03-30T04:48:04.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.810+0000 7faf49600700 1 -- 172.21.15.123:0/867850691 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7faf4c109e10 con 0x7faf3406cc30 2024-03-30T04:48:04.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.812+0000 7faf53e00700 1 -- 172.21.15.123:0/867850691 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7faf3406cc30 msgr2=0x7faf3406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:04.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.812+0000 7faf53e00700 1 --2- 172.21.15.123:0/867850691 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7faf3406cc30 0x7faf3406f0e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7faf400041d0 tx=0x7faf400043c0 comp rx=0 tx=0).stop 2024-03-30T04:48:04.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.812+0000 7faf53e00700 1 -- 172.21.15.123:0/867850691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf4c1037a0 msgr2=0x7faf4c1162e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:04.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.812+0000 7faf53e00700 1 --2- 172.21.15.123:0/867850691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf4c1037a0 0x7faf4c1162e0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7faf38009390 tx=0x7faf38009e30 comp rx=0 tx=0).stop 2024-03-30T04:48:04.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.812+0000 7faf53e00700 1 -- 172.21.15.123:0/867850691 shutdown_connections 2024-03-30T04:48:04.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.812+0000 7faf53e00700 1 --2- 172.21.15.123:0/867850691 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7faf3406cc30 0x7faf3406f0e0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:04.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.812+0000 7faf53e00700 1 --2- 172.21.15.123:0/867850691 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faf4c102e00 0x7faf4c115da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:04.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.812+0000 7faf53e00700 1 --2- 172.21.15.123:0/867850691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf4c1037a0 0x7faf4c1162e0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:04.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.812+0000 7faf53e00700 1 -- 172.21.15.123:0/867850691 >> 172.21.15.123:0/867850691 conn(0x7faf4c0f89b0 msgr2=0x7faf4c0f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:48:04.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.812+0000 7faf53e00700 1 -- 172.21.15.123:0/867850691 shutdown_connections 2024-03-30T04:48:04.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.812+0000 7faf53e00700 1 -- 172.21.15.123:0/867850691 wait complete. 2024-03-30T04:48:04.825 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:48:04.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.946+0000 7f8198600700 1 -- 172.21.15.123:0/3902876734 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81901037c0 msgr2=0x7f81900fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:04.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.946+0000 7f8198600700 1 --2- 172.21.15.123:0/3902876734 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81901037c0 0x7f81900fce30 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f817c004940 tx=0x7f817c009e30 comp rx=0 tx=0).stop 2024-03-30T04:48:04.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.946+0000 7f8198600700 1 -- 172.21.15.123:0/3902876734 shutdown_connections 2024-03-30T04:48:04.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.946+0000 7f8198600700 1 --2- 172.21.15.123:0/3902876734 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81901037c0 0x7f81900fce30 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:04.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.946+0000 7f8198600700 1 --2- 172.21.15.123:0/3902876734 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8190102e20 0x7f81901031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:04.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.946+0000 7f8198600700 1 -- 172.21.15.123:0/3902876734 >> 172.21.15.123:0/3902876734 conn(0x7f81900f89d0 msgr2=0x7f81900fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:48:04.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.946+0000 7f8198600700 1 -- 172.21.15.123:0/3902876734 shutdown_connections 2024-03-30T04:48:04.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.946+0000 7f8198600700 1 -- 172.21.15.123:0/3902876734 wait complete. 2024-03-30T04:48:04.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.947+0000 7f8198600700 1 Processor -- start 2024-03-30T04:48:04.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.947+0000 7f8198600700 1 -- start start 2024-03-30T04:48:04.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.947+0000 7f8198600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8190102e20 0x7f81900702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:48:04.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.948+0000 7f8198600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81901037c0 0x7f8190070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:48:04.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.948+0000 7f8198600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8190070d90 con 0x7f81901037c0 2024-03-30T04:48:04.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.948+0000 7f8198600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8190070ed0 con 0x7f8190102e20 2024-03-30T04:48:04.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.948+0000 7f8195200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81901037c0 0x7f8190070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:48:04.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.948+0000 7f8195200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81901037c0 0x7f8190070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58914/0 (socket says 172.21.15.123:58914) 2024-03-30T04:48:04.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.948+0000 7f8195200700 1 -- 172.21.15.123:0/3752704405 learned_addr learned my addr 172.21.15.123:0/3752704405 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:48:04.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.948+0000 7f8195c00700 1 --2- 172.21.15.123:0/3752704405 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8190102e20 0x7f81900702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:48:04.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.948+0000 7f8195200700 1 -- 172.21.15.123:0/3752704405 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8190102e20 msgr2=0x7f81900702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:04.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.948+0000 7f8195200700 1 --2- 172.21.15.123:0/3752704405 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8190102e20 0x7f81900702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:04.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.948+0000 7f8195200700 1 -- 172.21.15.123:0/3752704405 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f817c005040 con 0x7f81901037c0 2024-03-30T04:48:04.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.948+0000 7f8195c00700 1 --2- 172.21.15.123:0/3752704405 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8190102e20 0x7f81900702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:48:04.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.948+0000 7f8195200700 1 --2- 172.21.15.123:0/3752704405 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81901037c0 0x7f8190070800 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f817c009390 tx=0x7f817c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:48:04.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.949+0000 7f818d600700 1 -- 172.21.15.123:0/3752704405 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f817c00e040 con 0x7f81901037c0 2024-03-30T04:48:04.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.949+0000 7f818d600700 1 -- 172.21.15.123:0/3752704405 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f817c013420 con 0x7f81901037c0 2024-03-30T04:48:04.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.949+0000 7f8198600700 1 -- 172.21.15.123:0/3752704405 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8190071120 con 0x7f81901037c0 2024-03-30T04:48:04.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.949+0000 7f818d600700 1 -- 172.21.15.123:0/3752704405 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f817c01c630 con 0x7f81901037c0 2024-03-30T04:48:04.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.949+0000 7f8198600700 1 -- 172.21.15.123:0/3752704405 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f819011a200 con 0x7f81901037c0 2024-03-30T04:48:04.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.950+0000 7f8198600700 1 -- 172.21.15.123:0/3752704405 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81900fe570 con 0x7f81901037c0 2024-03-30T04:48:04.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.951+0000 7f818d600700 1 -- 172.21.15.123:0/3752704405 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f817c0139e0 con 0x7f81901037c0 2024-03-30T04:48:04.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.953+0000 7f818d600700 1 --2- 172.21.15.123:0/3752704405 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f817806cc30 0x7f817806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:48:04.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.953+0000 7f818d600700 1 -- 172.21.15.123:0/3752704405 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f817c08d380 con 0x7f81901037c0 2024-03-30T04:48:04.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.953+0000 7f8195c00700 1 --2- 172.21.15.123:0/3752704405 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f817806cc30 0x7f817806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:48:04.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.953+0000 7f8195c00700 1 --2- 172.21.15.123:0/3752704405 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f817806cc30 0x7f817806f0e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8184009fd0 tx=0x7f818400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:48:04.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:04.957+0000 7f818d600700 1 -- 172.21.15.123:0/3752704405 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f817c057340 con 0x7f81901037c0 2024-03-30T04:48:04.965 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:48:04 smithi123 ceph-mon[139369]: pgmap v506: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:05.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:48:04 smithi135 ceph-mon[106011]: pgmap v506: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:05.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.179+0000 7f8198600700 1 -- 172.21.15.123:0/3752704405 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8190109e60 con 0x7f817806cc30 2024-03-30T04:48:05.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.181+0000 7f818d600700 1 -- 172.21.15.123:0/3752704405 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f818400f040 con 0x7f817806cc30 2024-03-30T04:48:05.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.183+0000 7f8198600700 1 -- 172.21.15.123:0/3752704405 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f817806cc30 msgr2=0x7f817806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:05.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.183+0000 7f8198600700 1 --2- 172.21.15.123:0/3752704405 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f817806cc30 0x7f817806f0e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8184009fd0 tx=0x7f818400b040 comp rx=0 tx=0).stop 2024-03-30T04:48:05.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.183+0000 7f8198600700 1 -- 172.21.15.123:0/3752704405 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81901037c0 msgr2=0x7f8190070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:05.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.183+0000 7f8198600700 1 --2- 172.21.15.123:0/3752704405 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81901037c0 0x7f8190070800 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f817c009390 tx=0x7f817c009e30 comp rx=0 tx=0).stop 2024-03-30T04:48:05.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.183+0000 7f8198600700 1 -- 172.21.15.123:0/3752704405 shutdown_connections 2024-03-30T04:48:05.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.183+0000 7f8198600700 1 --2- 172.21.15.123:0/3752704405 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f817806cc30 0x7f817806f0e0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:05.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.183+0000 7f8198600700 1 --2- 172.21.15.123:0/3752704405 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8190102e20 0x7f81900702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:05.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.183+0000 7f8198600700 1 --2- 172.21.15.123:0/3752704405 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81901037c0 0x7f8190070800 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:05.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.183+0000 7f8198600700 1 -- 172.21.15.123:0/3752704405 >> 172.21.15.123:0/3752704405 conn(0x7f81900f89d0 msgr2=0x7f81900f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:48:05.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.183+0000 7f8198600700 1 -- 172.21.15.123:0/3752704405 shutdown_connections 2024-03-30T04:48:05.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.183+0000 7f8198600700 1 -- 172.21.15.123:0/3752704405 wait complete. 2024-03-30T04:48:05.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.316+0000 7fe056200700 1 -- 172.21.15.123:0/1451990680 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe0501037a0 msgr2=0x7fe0500fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:05.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.316+0000 7fe056200700 1 --2- 172.21.15.123:0/1451990680 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe0501037a0 0x7fe0500fce10 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fe03c004940 tx=0x7fe03c009e30 comp rx=0 tx=0).stop 2024-03-30T04:48:05.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.316+0000 7fe056200700 1 -- 172.21.15.123:0/1451990680 shutdown_connections 2024-03-30T04:48:05.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.316+0000 7fe056200700 1 --2- 172.21.15.123:0/1451990680 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe0501037a0 0x7fe0500fce10 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:05.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.316+0000 7fe056200700 1 --2- 172.21.15.123:0/1451990680 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe050102e00 0x7fe0501031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:05.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.316+0000 7fe056200700 1 -- 172.21.15.123:0/1451990680 >> 172.21.15.123:0/1451990680 conn(0x7fe0500f89b0 msgr2=0x7fe0500fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:48:05.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.317+0000 7fe056200700 1 -- 172.21.15.123:0/1451990680 shutdown_connections 2024-03-30T04:48:05.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.317+0000 7fe056200700 1 -- 172.21.15.123:0/1451990680 wait complete. 2024-03-30T04:48:05.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.317+0000 7fe056200700 1 Processor -- start 2024-03-30T04:48:05.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.317+0000 7fe056200700 1 -- start start 2024-03-30T04:48:05.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.318+0000 7fe056200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe050102e00 0x7fe0500702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:48:05.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.318+0000 7fe056200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe0501037a0 0x7fe050070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:48:05.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.318+0000 7fe056200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe050070d90 con 0x7fe050102e00 2024-03-30T04:48:05.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.318+0000 7fe056200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe050070ed0 con 0x7fe0501037a0 2024-03-30T04:48:05.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.318+0000 7fe04f400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe050102e00 0x7fe0500702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:48:05.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.318+0000 7fe04f400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe050102e00 0x7fe0500702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58940/0 (socket says 172.21.15.123:58940) 2024-03-30T04:48:05.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.318+0000 7fe04f400700 1 -- 172.21.15.123:0/2878290527 learned_addr learned my addr 172.21.15.123:0/2878290527 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:48:05.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.318+0000 7fe04ea00700 1 --2- 172.21.15.123:0/2878290527 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe0501037a0 0x7fe050070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:48:05.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.318+0000 7fe04f400700 1 -- 172.21.15.123:0/2878290527 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe0501037a0 msgr2=0x7fe050070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:05.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.318+0000 7fe04f400700 1 --2- 172.21.15.123:0/2878290527 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe0501037a0 0x7fe050070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:05.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.318+0000 7fe04f400700 1 -- 172.21.15.123:0/2878290527 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe03c005040 con 0x7fe050102e00 2024-03-30T04:48:05.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.319+0000 7fe04f400700 1 --2- 172.21.15.123:0/2878290527 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe050102e00 0x7fe0500702c0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fe04000c8c0 tx=0x7fe0400044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:48:05.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.319+0000 7fe047400700 1 -- 172.21.15.123:0/2878290527 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe04000a760 con 0x7fe050102e00 2024-03-30T04:48:05.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.319+0000 7fe047400700 1 -- 172.21.15.123:0/2878290527 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe04000ad70 con 0x7fe050102e00 2024-03-30T04:48:05.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.319+0000 7fe056200700 1 -- 172.21.15.123:0/2878290527 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe050119e00 con 0x7fe050102e00 2024-03-30T04:48:05.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.319+0000 7fe047400700 1 -- 172.21.15.123:0/2878290527 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe0400113a0 con 0x7fe050102e00 2024-03-30T04:48:05.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.319+0000 7fe056200700 1 -- 172.21.15.123:0/2878290527 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe05011a280 con 0x7fe050102e00 2024-03-30T04:48:05.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.320+0000 7fe056200700 1 -- 172.21.15.123:0/2878290527 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0500fe550 con 0x7fe050102e00 2024-03-30T04:48:05.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.321+0000 7fe047400700 1 -- 172.21.15.123:0/2878290527 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fe040012400 con 0x7fe050102e00 2024-03-30T04:48:05.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.322+0000 7fe047400700 1 --2- 172.21.15.123:0/2878290527 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe03406cc30 0x7fe03406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:48:05.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.322+0000 7fe047400700 1 -- 172.21.15.123:0/2878290527 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe04008c0f0 con 0x7fe050102e00 2024-03-30T04:48:05.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.322+0000 7fe04ea00700 1 --2- 172.21.15.123:0/2878290527 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe03406cc30 0x7fe03406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:48:05.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.323+0000 7fe04ea00700 1 --2- 172.21.15.123:0/2878290527 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe03406cc30 0x7fe03406f0e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe050071c00 tx=0x7fe03c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:48:05.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.326+0000 7fe047400700 1 -- 172.21.15.123:0/2878290527 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe0400560b0 con 0x7fe050102e00 2024-03-30T04:48:05.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.538+0000 7fe056200700 1 -- 172.21.15.123:0/2878290527 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fe0500719b0 con 0x7fe03406cc30 2024-03-30T04:48:05.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.548+0000 7fe047400700 1 -- 172.21.15.123:0/2878290527 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fe0500719b0 con 0x7fe03406cc30 2024-03-30T04:48:05.546 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:48:05.546 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (17m) 10m ago 29m 15.0M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:48:05.546 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (14m) 10m ago 30m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T04:48:05.546 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (14m) 9m ago 28m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T04:48:05.546 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (16m) 10m ago 29m 74.0M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T04:48:05.546 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (10m) 10m ago 23m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T04:48:05.547 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (10m) 10m ago 23m 10.9M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T04:48:05.547 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (23m) 9m ago 23m 10.7M - 17.2.7-797-g5c024335 f77085f6aab7 0dc41326818c 2024-03-30T04:48:05.547 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (9m) 9m ago 23m 9747k - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T04:48:05.547 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (19m) 10m ago 31m 506M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:48:05.547 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (18m) 9m ago 28m 450M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:48:05.547 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (15m) 10m ago 31m 53.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T04:48:05.547 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (15m) 9m ago 28m 42.6M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T04:48:05.547 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (18m) 10m ago 29m 8052k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:48:05.547 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (17m) 9m ago 28m 7880k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:48:05.547 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (14m) 10m ago 27m 104M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T04:48:05.547 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (13m) 10m ago 27m 93.4M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T04:48:05.547 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (12m) 10m ago 26m 86.3M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T04:48:05.547 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (12m) 9m ago 26m 97.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T04:48:05.547 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (11m) 9m ago 25m 97.9M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T04:48:05.548 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (10m) 9m ago 25m 85.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T04:48:05.548 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (17m) 10m ago 29m 53.7M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:48:05.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.550+0000 7fe056200700 1 -- 172.21.15.123:0/2878290527 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe03406cc30 msgr2=0x7fe03406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:05.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.550+0000 7fe056200700 1 --2- 172.21.15.123:0/2878290527 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe03406cc30 0x7fe03406f0e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe050071c00 tx=0x7fe03c009e30 comp rx=0 tx=0).stop 2024-03-30T04:48:05.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.550+0000 7fe056200700 1 -- 172.21.15.123:0/2878290527 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe050102e00 msgr2=0x7fe0500702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:05.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.550+0000 7fe056200700 1 --2- 172.21.15.123:0/2878290527 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe050102e00 0x7fe0500702c0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fe04000c8c0 tx=0x7fe0400044c0 comp rx=0 tx=0).stop 2024-03-30T04:48:05.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.550+0000 7fe056200700 1 -- 172.21.15.123:0/2878290527 shutdown_connections 2024-03-30T04:48:05.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.550+0000 7fe056200700 1 --2- 172.21.15.123:0/2878290527 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe03406cc30 0x7fe03406f0e0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:05.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.550+0000 7fe056200700 1 --2- 172.21.15.123:0/2878290527 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe050102e00 0x7fe0500702c0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:05.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.550+0000 7fe056200700 1 --2- 172.21.15.123:0/2878290527 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe0501037a0 0x7fe050070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:05.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.550+0000 7fe056200700 1 -- 172.21.15.123:0/2878290527 >> 172.21.15.123:0/2878290527 conn(0x7fe0500f89b0 msgr2=0x7fe0500f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:48:05.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.550+0000 7fe056200700 1 -- 172.21.15.123:0/2878290527 shutdown_connections 2024-03-30T04:48:05.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.550+0000 7fe056200700 1 -- 172.21.15.123:0/2878290527 wait complete. 2024-03-30T04:48:05.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.684+0000 7fd1c1a00700 1 -- 172.21.15.123:0/2696262376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1bc1037c0 msgr2=0x7fd1bc0fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:05.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.684+0000 7fd1c1a00700 1 --2- 172.21.15.123:0/2696262376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1bc1037c0 0x7fd1bc0fce30 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fd1a8004940 tx=0x7fd1a8009e30 comp rx=0 tx=0).stop 2024-03-30T04:48:05.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.684+0000 7fd1c1a00700 1 -- 172.21.15.123:0/2696262376 shutdown_connections 2024-03-30T04:48:05.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.684+0000 7fd1c1a00700 1 --2- 172.21.15.123:0/2696262376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1bc1037c0 0x7fd1bc0fce30 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:05.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.684+0000 7fd1c1a00700 1 --2- 172.21.15.123:0/2696262376 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd1bc102e20 0x7fd1bc1031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:05.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.684+0000 7fd1c1a00700 1 -- 172.21.15.123:0/2696262376 >> 172.21.15.123:0/2696262376 conn(0x7fd1bc0f89d0 msgr2=0x7fd1bc0fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:48:05.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.684+0000 7fd1c1a00700 1 -- 172.21.15.123:0/2696262376 shutdown_connections 2024-03-30T04:48:05.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.684+0000 7fd1c1a00700 1 -- 172.21.15.123:0/2696262376 wait complete. 2024-03-30T04:48:05.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.685+0000 7fd1c1a00700 1 Processor -- start 2024-03-30T04:48:05.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.685+0000 7fd1c1a00700 1 -- start start 2024-03-30T04:48:05.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.685+0000 7fd1c1a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd1bc102e20 0x7fd1bc0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:48:05.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.685+0000 7fd1c1a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1bc1037c0 0x7fd1bc070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:48:05.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.685+0000 7fd1c1a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc0720b0 con 0x7fd1bc1037c0 2024-03-30T04:48:05.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.685+0000 7fd1c1a00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc072220 con 0x7fd1bc102e20 2024-03-30T04:48:05.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.685+0000 7fd1ba000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1bc1037c0 0x7fd1bc070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:48:05.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.686+0000 7fd1baa00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd1bc102e20 0x7fd1bc0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:48:05.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.686+0000 7fd1ba000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1bc1037c0 0x7fd1bc070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58962/0 (socket says 172.21.15.123:58962) 2024-03-30T04:48:05.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.686+0000 7fd1ba000700 1 -- 172.21.15.123:0/876995630 learned_addr learned my addr 172.21.15.123:0/876995630 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:48:05.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.686+0000 7fd1ba000700 1 -- 172.21.15.123:0/876995630 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd1bc102e20 msgr2=0x7fd1bc0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:05.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.686+0000 7fd1ba000700 1 --2- 172.21.15.123:0/876995630 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd1bc102e20 0x7fd1bc0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:05.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.686+0000 7fd1ba000700 1 -- 172.21.15.123:0/876995630 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1a8005040 con 0x7fd1bc1037c0 2024-03-30T04:48:05.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.686+0000 7fd1ba000700 1 --2- 172.21.15.123:0/876995630 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1bc1037c0 0x7fd1bc070800 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fd1a8009fd0 tx=0x7fd1a8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:48:05.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.687+0000 7fd1b2a00700 1 -- 172.21.15.123:0/876995630 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1a800e030 con 0x7fd1bc1037c0 2024-03-30T04:48:05.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.687+0000 7fd1b2a00700 1 -- 172.21.15.123:0/876995630 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd1a80093b0 con 0x7fd1bc1037c0 2024-03-30T04:48:05.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.687+0000 7fd1c1a00700 1 -- 172.21.15.123:0/876995630 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1bc070e00 con 0x7fd1bc1037c0 2024-03-30T04:48:05.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.687+0000 7fd1b2a00700 1 -- 172.21.15.123:0/876995630 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1a801cc80 con 0x7fd1bc1037c0 2024-03-30T04:48:05.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.687+0000 7fd1c1a00700 1 -- 172.21.15.123:0/876995630 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1bc11e6d0 con 0x7fd1bc1037c0 2024-03-30T04:48:05.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.688+0000 7fd1c1a00700 1 -- 172.21.15.123:0/876995630 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1bc0fe570 con 0x7fd1bc1037c0 2024-03-30T04:48:05.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.689+0000 7fd1b2a00700 1 -- 172.21.15.123:0/876995630 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fd1a801c400 con 0x7fd1bc1037c0 2024-03-30T04:48:05.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.690+0000 7fd1b2a00700 1 --2- 172.21.15.123:0/876995630 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd1a006cc30 0x7fd1a006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:48:05.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.690+0000 7fd1b2a00700 1 -- 172.21.15.123:0/876995630 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd1a808d850 con 0x7fd1bc1037c0 2024-03-30T04:48:05.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.690+0000 7fd1baa00700 1 --2- 172.21.15.123:0/876995630 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd1a006cc30 0x7fd1a006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:48:05.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.690+0000 7fd1baa00700 1 --2- 172.21.15.123:0/876995630 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd1a006cc30 0x7fd1a006f0e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac009b80 tx=0x7fd1ac00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:48:05.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.694+0000 7fd1b2a00700 1 -- 172.21.15.123:0/876995630 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd1a805b320 con 0x7fd1bc1037c0 2024-03-30T04:48:05.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:48:05 smithi123 ceph-mon[139369]: from='client.34962 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:48:05.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.977+0000 7fd1c1a00700 1 -- 172.21.15.123:0/876995630 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fd1bc070f90 con 0x7fd1bc1037c0 2024-03-30T04:48:05.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.977+0000 7fd1b2a00700 1 -- 172.21.15.123:0/876995630 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7fd1bc070f90 con 0x7fd1bc1037c0 2024-03-30T04:48:05.976 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:48:05.976 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:48:05.976 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:48:05.976 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:48:05.976 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:48:05.976 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:48:05.976 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:48:05.976 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:48:05.976 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T04:48:05.976 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:48:05.977 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:48:05.977 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:48:05.977 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 3 2024-03-30T04:48:05.977 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:48:05.977 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:48:05.977 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:48:05.977 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 13 2024-03-30T04:48:05.977 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:48:05.977 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:48:05.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.980+0000 7fd1c1a00700 1 -- 172.21.15.123:0/876995630 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd1a006cc30 msgr2=0x7fd1a006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:05.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.980+0000 7fd1c1a00700 1 --2- 172.21.15.123:0/876995630 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd1a006cc30 0x7fd1a006f0e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac009b80 tx=0x7fd1ac00b040 comp rx=0 tx=0).stop 2024-03-30T04:48:05.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.980+0000 7fd1c1a00700 1 -- 172.21.15.123:0/876995630 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1bc1037c0 msgr2=0x7fd1bc070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:05.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.980+0000 7fd1c1a00700 1 --2- 172.21.15.123:0/876995630 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1bc1037c0 0x7fd1bc070800 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fd1a8009fd0 tx=0x7fd1a8009e30 comp rx=0 tx=0).stop 2024-03-30T04:48:05.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.980+0000 7fd1c1a00700 1 -- 172.21.15.123:0/876995630 shutdown_connections 2024-03-30T04:48:05.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.980+0000 7fd1c1a00700 1 --2- 172.21.15.123:0/876995630 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd1a006cc30 0x7fd1a006f0e0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:05.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.980+0000 7fd1c1a00700 1 --2- 172.21.15.123:0/876995630 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd1bc102e20 0x7fd1bc0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:05.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.980+0000 7fd1c1a00700 1 --2- 172.21.15.123:0/876995630 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1bc1037c0 0x7fd1bc070800 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:05.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.980+0000 7fd1c1a00700 1 -- 172.21.15.123:0/876995630 >> 172.21.15.123:0/876995630 conn(0x7fd1bc0f89d0 msgr2=0x7fd1bc0f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:48:05.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.980+0000 7fd1c1a00700 1 -- 172.21.15.123:0/876995630 shutdown_connections 2024-03-30T04:48:05.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:05.980+0000 7fd1c1a00700 1 -- 172.21.15.123:0/876995630 wait complete. 2024-03-30T04:48:06.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:48:05 smithi135 ceph-mon[106011]: from='client.34962 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:48:06.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.114+0000 7f4140200700 1 -- 172.21.15.123:0/2688673913 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4138102e20 msgr2=0x7f41381031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:06.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.114+0000 7f4140200700 1 --2- 172.21.15.123:0/2688673913 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4138102e20 0x7f41381031f0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f412c004940 tx=0x7f412c009e30 comp rx=0 tx=0).stop 2024-03-30T04:48:06.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.114+0000 7f4140200700 1 -- 172.21.15.123:0/2688673913 shutdown_connections 2024-03-30T04:48:06.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.114+0000 7f4140200700 1 --2- 172.21.15.123:0/2688673913 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f41381037c0 0x7f41380fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:06.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.114+0000 7f4140200700 1 --2- 172.21.15.123:0/2688673913 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4138102e20 0x7f41381031f0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:06.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.114+0000 7f4140200700 1 -- 172.21.15.123:0/2688673913 >> 172.21.15.123:0/2688673913 conn(0x7f41380f89d0 msgr2=0x7f41380fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:48:06.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.114+0000 7f4140200700 1 -- 172.21.15.123:0/2688673913 shutdown_connections 2024-03-30T04:48:06.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.114+0000 7f4140200700 1 -- 172.21.15.123:0/2688673913 wait complete. 2024-03-30T04:48:06.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.115+0000 7f4140200700 1 Processor -- start 2024-03-30T04:48:06.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.115+0000 7f4140200700 1 -- start start 2024-03-30T04:48:06.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.115+0000 7f4140200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4138102e20 0x7f4138115c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:48:06.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.115+0000 7f4140200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f41381037c0 0x7f41381161d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:48:06.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.116+0000 7f4140200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4138116820 con 0x7f4138102e20 2024-03-30T04:48:06.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.116+0000 7f4140200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f413810fbf0 con 0x7f41381037c0 2024-03-30T04:48:06.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.116+0000 7f413d800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4138102e20 0x7f4138115c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:48:06.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.116+0000 7f413d800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4138102e20 0x7f4138115c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58970/0 (socket says 172.21.15.123:58970) 2024-03-30T04:48:06.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.116+0000 7f413d800700 1 -- 172.21.15.123:0/1618542359 learned_addr learned my addr 172.21.15.123:0/1618542359 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:48:06.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.116+0000 7f413ce00700 1 --2- 172.21.15.123:0/1618542359 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f41381037c0 0x7f41381161d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:48:06.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.116+0000 7f413d800700 1 -- 172.21.15.123:0/1618542359 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f41381037c0 msgr2=0x7f41381161d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:06.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.116+0000 7f413d800700 1 --2- 172.21.15.123:0/1618542359 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f41381037c0 0x7f41381161d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:06.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.116+0000 7f413d800700 1 -- 172.21.15.123:0/1618542359 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f412c005040 con 0x7f4138102e20 2024-03-30T04:48:06.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.116+0000 7f413d800700 1 --2- 172.21.15.123:0/1618542359 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4138102e20 0x7f4138115c90 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f412c008d80 tx=0x7f412c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:48:06.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.117+0000 7f4135600700 1 -- 172.21.15.123:0/1618542359 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f412c00e030 con 0x7f4138102e20 2024-03-30T04:48:06.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.117+0000 7f4135600700 1 -- 172.21.15.123:0/1618542359 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f412c013de0 con 0x7f4138102e20 2024-03-30T04:48:06.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.117+0000 7f4140200700 1 -- 172.21.15.123:0/1618542359 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f413810fe70 con 0x7f4138102e20 2024-03-30T04:48:06.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.117+0000 7f4135600700 1 -- 172.21.15.123:0/1618542359 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f412c01cd60 con 0x7f4138102e20 2024-03-30T04:48:06.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.117+0000 7f4140200700 1 -- 172.21.15.123:0/1618542359 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4138110430 con 0x7f4138102e20 2024-03-30T04:48:06.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.118+0000 7f4140200700 1 -- 172.21.15.123:0/1618542359 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41380fe570 con 0x7f4138102e20 2024-03-30T04:48:06.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.119+0000 7f4135600700 1 -- 172.21.15.123:0/1618542359 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f412c01c590 con 0x7f4138102e20 2024-03-30T04:48:06.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.120+0000 7f4135600700 1 --2- 172.21.15.123:0/1618542359 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f412006cc30 0x7f412006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:48:06.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.120+0000 7f4135600700 1 -- 172.21.15.123:0/1618542359 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f412c08d8c0 con 0x7f4138102e20 2024-03-30T04:48:06.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.120+0000 7f413ce00700 1 --2- 172.21.15.123:0/1618542359 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f412006cc30 0x7f412006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:48:06.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.120+0000 7f413ce00700 1 --2- 172.21.15.123:0/1618542359 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f412006cc30 0x7f412006f0e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f4124004600 tx=0x7f412400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:48:06.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.124+0000 7f4135600700 1 -- 172.21.15.123:0/1618542359 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f412c05b390 con 0x7f4138102e20 2024-03-30T04:48:06.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.370+0000 7f4140200700 1 -- 172.21.15.123:0/1618542359 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f413804bed0 con 0x7f4138102e20 2024-03-30T04:48:06.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.370+0000 7f4135600700 1 -- 172.21.15.123:0/1618542359 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f412c05af20 con 0x7f4138102e20 2024-03-30T04:48:06.369 INFO:teuthology.orchestra.run.smithi123.stdout:e32 2024-03-30T04:48:06.369 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:48:06.369 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:48:06.369 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:48:06.369 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:48:06.369 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:48:06.369 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:48:06.369 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T04:48:06.369 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T04:48:06.369 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:48:06.369 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T04:48:06.370 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:48:06.370 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:48:06.370 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:48:06.370 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:48:06.370 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:48:06.370 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T04:48:06.370 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:48:06.370 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:48:06.370 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T04:48:06.370 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:48:06.370 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T04:48:06.370 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T04:48:06.371 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T04:48:06.371 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T04:48:06.371 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:48:06.371 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:48:06.371 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:48:06.371 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:48:06.371 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:48:06.371 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:48:06.371 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T04:48:06.371 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T04:48:06.371 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:48:06.371 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:48:06.371 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T04:48:06.371 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:48:06.371 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:48:06.372 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:48:06.372 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44315} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/1741352336,v1:172.21.15.135:6827/1741352336] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:48:06.372 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:48:06.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.373+0000 7f4140200700 1 -- 172.21.15.123:0/1618542359 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f412006cc30 msgr2=0x7f412006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:06.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.373+0000 7f4140200700 1 --2- 172.21.15.123:0/1618542359 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f412006cc30 0x7f412006f0e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f4124004600 tx=0x7f412400b040 comp rx=0 tx=0).stop 2024-03-30T04:48:06.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.373+0000 7f4140200700 1 -- 172.21.15.123:0/1618542359 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4138102e20 msgr2=0x7f4138115c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:06.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.373+0000 7f4140200700 1 --2- 172.21.15.123:0/1618542359 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4138102e20 0x7f4138115c90 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f412c008d80 tx=0x7f412c009e30 comp rx=0 tx=0).stop 2024-03-30T04:48:06.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.373+0000 7f4140200700 1 -- 172.21.15.123:0/1618542359 shutdown_connections 2024-03-30T04:48:06.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.373+0000 7f4140200700 1 --2- 172.21.15.123:0/1618542359 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f412006cc30 0x7f412006f0e0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:06.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.373+0000 7f4140200700 1 --2- 172.21.15.123:0/1618542359 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4138102e20 0x7f4138115c90 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:06.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.373+0000 7f4140200700 1 --2- 172.21.15.123:0/1618542359 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f41381037c0 0x7f41381161d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:06.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.373+0000 7f4140200700 1 -- 172.21.15.123:0/1618542359 >> 172.21.15.123:0/1618542359 conn(0x7f41380f89d0 msgr2=0x7f41380f99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:48:06.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.373+0000 7f4140200700 1 -- 172.21.15.123:0/1618542359 shutdown_connections 2024-03-30T04:48:06.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.373+0000 7f4140200700 1 -- 172.21.15.123:0/1618542359 wait complete. 2024-03-30T04:48:06.373 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 32 2024-03-30T04:48:06.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.507+0000 7fb18d400700 1 -- 172.21.15.123:0/31800607 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb188102e00 msgr2=0x7fb1881031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:06.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.507+0000 7fb18d400700 1 --2- 172.21.15.123:0/31800607 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb188102e00 0x7fb1881031d0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fb178004940 tx=0x7fb178009e30 comp rx=0 tx=0).stop 2024-03-30T04:48:06.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.507+0000 7fb18d400700 1 -- 172.21.15.123:0/31800607 shutdown_connections 2024-03-30T04:48:06.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.507+0000 7fb18d400700 1 --2- 172.21.15.123:0/31800607 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb1881037a0 0x7fb1880fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:06.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.507+0000 7fb18d400700 1 --2- 172.21.15.123:0/31800607 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb188102e00 0x7fb1881031d0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:06.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.507+0000 7fb18d400700 1 -- 172.21.15.123:0/31800607 >> 172.21.15.123:0/31800607 conn(0x7fb1880f89b0 msgr2=0x7fb1880fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:48:06.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.507+0000 7fb18d400700 1 -- 172.21.15.123:0/31800607 shutdown_connections 2024-03-30T04:48:06.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.507+0000 7fb18d400700 1 -- 172.21.15.123:0/31800607 wait complete. 2024-03-30T04:48:06.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.508+0000 7fb18d400700 1 Processor -- start 2024-03-30T04:48:06.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.508+0000 7fb18d400700 1 -- start start 2024-03-30T04:48:06.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.508+0000 7fb18d400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb188102e00 0x7fb188115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:48:06.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.508+0000 7fb18d400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb1881037a0 0x7fb1881161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:48:06.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.508+0000 7fb18d400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb188116810 con 0x7fb188102e00 2024-03-30T04:48:06.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.508+0000 7fb18d400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb18810fbe0 con 0x7fb1881037a0 2024-03-30T04:48:06.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.508+0000 7fb186800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb188102e00 0x7fb188115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:48:06.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.509+0000 7fb186800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb188102e00 0x7fb188115c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58984/0 (socket says 172.21.15.123:58984) 2024-03-30T04:48:06.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.509+0000 7fb186800700 1 -- 172.21.15.123:0/2453773462 learned_addr learned my addr 172.21.15.123:0/2453773462 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:48:06.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.509+0000 7fb17fe00700 1 --2- 172.21.15.123:0/2453773462 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb1881037a0 0x7fb1881161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:48:06.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.509+0000 7fb186800700 1 -- 172.21.15.123:0/2453773462 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb1881037a0 msgr2=0x7fb1881161c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:06.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.509+0000 7fb186800700 1 --2- 172.21.15.123:0/2453773462 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb1881037a0 0x7fb1881161c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:06.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.509+0000 7fb186800700 1 -- 172.21.15.123:0/2453773462 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb178005040 con 0x7fb188102e00 2024-03-30T04:48:06.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.509+0000 7fb186800700 1 --2- 172.21.15.123:0/2453773462 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb188102e00 0x7fb188115c80 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fb178007f50 tx=0x7fb178009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:48:06.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.509+0000 7fb17ea00700 1 -- 172.21.15.123:0/2453773462 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb178021070 con 0x7fb188102e00 2024-03-30T04:48:06.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.509+0000 7fb18d400700 1 -- 172.21.15.123:0/2453773462 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb18810fe60 con 0x7fb188102e00 2024-03-30T04:48:06.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.510+0000 7fb17ea00700 1 -- 172.21.15.123:0/2453773462 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb1780093b0 con 0x7fb188102e00 2024-03-30T04:48:06.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.510+0000 7fb17ea00700 1 -- 172.21.15.123:0/2453773462 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb17800e070 con 0x7fb188102e00 2024-03-30T04:48:06.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.510+0000 7fb18d400700 1 -- 172.21.15.123:0/2453773462 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1881103c0 con 0x7fb188102e00 2024-03-30T04:48:06.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.511+0000 7fb18d400700 1 -- 172.21.15.123:0/2453773462 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1880fe550 con 0x7fb188102e00 2024-03-30T04:48:06.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.511+0000 7fb17ea00700 1 -- 172.21.15.123:0/2453773462 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fb17801c400 con 0x7fb188102e00 2024-03-30T04:48:06.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.512+0000 7fb17ea00700 1 --2- 172.21.15.123:0/2453773462 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb16c06cc30 0x7fb16c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:48:06.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.512+0000 7fb17ea00700 1 -- 172.21.15.123:0/2453773462 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb17808d120 con 0x7fb188102e00 2024-03-30T04:48:06.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.513+0000 7fb17fe00700 1 --2- 172.21.15.123:0/2453773462 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb16c06cc30 0x7fb16c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:48:06.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.513+0000 7fb17fe00700 1 --2- 172.21.15.123:0/2453773462 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb16c06cc30 0x7fb16c06f0e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb188111520 tx=0x7fb17000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:48:06.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.517+0000 7fb17ea00700 1 -- 172.21.15.123:0/2453773462 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb17805abf0 con 0x7fb188102e00 2024-03-30T04:48:06.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.739+0000 7fb18d400700 1 -- 172.21.15.123:0/2453773462 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb18804bf40 con 0x7fb16c06cc30 2024-03-30T04:48:06.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.740+0000 7fb17ea00700 1 -- 172.21.15.123:0/2453773462 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fb18804bf40 con 0x7fb16c06cc30 2024-03-30T04:48:06.738 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:48:06.738 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:48:06.738 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:48:06.739 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T04:48:06.739 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:48:06.739 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T04:48:06.739 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T04:48:06.739 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T04:48:06.739 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T04:48:06.739 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:48:06.739 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "15/21 daemons upgraded", 2024-03-30T04:48:06.739 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T04:48:06.739 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:48:06.739 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:48:06.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.742+0000 7fb18d400700 1 -- 172.21.15.123:0/2453773462 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb16c06cc30 msgr2=0x7fb16c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:06.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.742+0000 7fb18d400700 1 --2- 172.21.15.123:0/2453773462 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb16c06cc30 0x7fb16c06f0e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb188111520 tx=0x7fb17000b040 comp rx=0 tx=0).stop 2024-03-30T04:48:06.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.742+0000 7fb18d400700 1 -- 172.21.15.123:0/2453773462 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb188102e00 msgr2=0x7fb188115c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:06.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.742+0000 7fb18d400700 1 --2- 172.21.15.123:0/2453773462 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb188102e00 0x7fb188115c80 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fb178007f50 tx=0x7fb178009e30 comp rx=0 tx=0).stop 2024-03-30T04:48:06.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.742+0000 7fb18d400700 1 -- 172.21.15.123:0/2453773462 shutdown_connections 2024-03-30T04:48:06.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.742+0000 7fb18d400700 1 --2- 172.21.15.123:0/2453773462 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb16c06cc30 0x7fb16c06f0e0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:06.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.742+0000 7fb18d400700 1 --2- 172.21.15.123:0/2453773462 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb188102e00 0x7fb188115c80 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:06.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.742+0000 7fb18d400700 1 --2- 172.21.15.123:0/2453773462 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb1881037a0 0x7fb1881161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:06.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.742+0000 7fb18d400700 1 -- 172.21.15.123:0/2453773462 >> 172.21.15.123:0/2453773462 conn(0x7fb1880f89b0 msgr2=0x7fb1880f9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:48:06.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.743+0000 7fb18d400700 1 -- 172.21.15.123:0/2453773462 shutdown_connections 2024-03-30T04:48:06.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.743+0000 7fb18d400700 1 -- 172.21.15.123:0/2453773462 wait complete. 2024-03-30T04:48:06.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.875+0000 7f4bba200700 1 -- 172.21.15.123:0/3497707842 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4bb4102e00 msgr2=0x7f4bb41031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:06.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.875+0000 7f4bba200700 1 --2- 172.21.15.123:0/3497707842 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4bb4102e00 0x7f4bb41031d0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f4ba4004940 tx=0x7f4ba4009e30 comp rx=0 tx=0).stop 2024-03-30T04:48:06.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.875+0000 7f4bba200700 1 -- 172.21.15.123:0/3497707842 shutdown_connections 2024-03-30T04:48:06.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.875+0000 7f4bba200700 1 --2- 172.21.15.123:0/3497707842 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4bb41037a0 0x7f4bb40fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:06.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.875+0000 7f4bba200700 1 --2- 172.21.15.123:0/3497707842 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4bb4102e00 0x7f4bb41031d0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:06.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.875+0000 7f4bba200700 1 -- 172.21.15.123:0/3497707842 >> 172.21.15.123:0/3497707842 conn(0x7f4bb40f89b0 msgr2=0x7f4bb40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:48:06.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.875+0000 7f4bba200700 1 -- 172.21.15.123:0/3497707842 shutdown_connections 2024-03-30T04:48:06.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.875+0000 7f4bba200700 1 -- 172.21.15.123:0/3497707842 wait complete. 2024-03-30T04:48:06.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.876+0000 7f4bba200700 1 Processor -- start 2024-03-30T04:48:06.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.876+0000 7f4bba200700 1 -- start start 2024-03-30T04:48:06.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.877+0000 7f4bba200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4bb4102e00 0x7f4bb4115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:48:06.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.877+0000 7f4bba200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4bb41037a0 0x7f4bb4116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:48:06.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.877+0000 7f4bba200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb4116860 con 0x7f4bb41037a0 2024-03-30T04:48:06.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.877+0000 7f4bba200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb410fc30 con 0x7f4bb4102e00 2024-03-30T04:48:06.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.877+0000 7f4bb2a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4bb41037a0 0x7f4bb4116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:48:06.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.877+0000 7f4bb2a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4bb41037a0 0x7f4bb4116210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58996/0 (socket says 172.21.15.123:58996) 2024-03-30T04:48:06.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.877+0000 7f4bb2a00700 1 -- 172.21.15.123:0/3932559052 learned_addr learned my addr 172.21.15.123:0/3932559052 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:48:06.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.877+0000 7f4bb3400700 1 --2- 172.21.15.123:0/3932559052 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4bb4102e00 0x7f4bb4115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:48:06.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.877+0000 7f4bb2a00700 1 -- 172.21.15.123:0/3932559052 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4bb4102e00 msgr2=0x7f4bb4115cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:06.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.877+0000 7f4bb2a00700 1 --2- 172.21.15.123:0/3932559052 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4bb4102e00 0x7f4bb4115cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:06.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.877+0000 7f4bb2a00700 1 -- 172.21.15.123:0/3932559052 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba4005040 con 0x7f4bb41037a0 2024-03-30T04:48:06.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.878+0000 7f4bb2a00700 1 --2- 172.21.15.123:0/3932559052 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4bb41037a0 0x7f4bb4116210 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f4ba0002a50 tx=0x7f4ba0004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:48:06.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.878+0000 7f4bab400700 1 -- 172.21.15.123:0/3932559052 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ba0007cc0 con 0x7f4bb41037a0 2024-03-30T04:48:06.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.878+0000 7f4bab400700 1 -- 172.21.15.123:0/3932559052 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4ba000f040 con 0x7f4bb41037a0 2024-03-30T04:48:06.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.878+0000 7f4bba200700 1 -- 172.21.15.123:0/3932559052 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bb410ff10 con 0x7f4bb41037a0 2024-03-30T04:48:06.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.878+0000 7f4bab400700 1 -- 172.21.15.123:0/3932559052 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ba000d920 con 0x7f4bb41037a0 2024-03-30T04:48:06.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.878+0000 7f4bba200700 1 -- 172.21.15.123:0/3932559052 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bb41104d0 con 0x7f4bb41037a0 2024-03-30T04:48:06.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.879+0000 7f4bba200700 1 -- 172.21.15.123:0/3932559052 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bb40fe550 con 0x7f4bb41037a0 2024-03-30T04:48:06.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.880+0000 7f4bab400700 1 -- 172.21.15.123:0/3932559052 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f4ba0009680 con 0x7f4bb41037a0 2024-03-30T04:48:06.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.881+0000 7f4bab400700 1 --2- 172.21.15.123:0/3932559052 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4b9806cc30 0x7f4b9806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:48:06.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.881+0000 7f4bab400700 1 -- 172.21.15.123:0/3932559052 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f4ba008bd10 con 0x7f4bb41037a0 2024-03-30T04:48:06.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.881+0000 7f4bb3400700 1 --2- 172.21.15.123:0/3932559052 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4b9806cc30 0x7f4b9806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:48:06.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.882+0000 7f4bb3400700 1 --2- 172.21.15.123:0/3932559052 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4b9806cc30 0x7f4b9806f0e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f4ba400b840 tx=0x7f4ba4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:48:06.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:06.885+0000 7f4bab400700 1 -- 172.21.15.123:0/3932559052 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4ba0059760 con 0x7f4bb41037a0 2024-03-30T04:48:06.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:48:06 smithi123 ceph-mon[139369]: from='client.34966 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:48:06.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:48:06 smithi123 ceph-mon[139369]: from='client.34970 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:48:06.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:48:06 smithi123 ceph-mon[139369]: pgmap v507: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:06.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:48:06 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/876995630' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:48:06.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:48:06 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/1618542359' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:48:07.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:48:06 smithi135 ceph-mon[106011]: from='client.34966 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:48:07.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:48:06 smithi135 ceph-mon[106011]: from='client.34970 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:48:07.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:48:06 smithi135 ceph-mon[106011]: pgmap v507: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:07.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:48:06 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/876995630' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:48:07.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:48:06 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/1618542359' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:48:07.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:07.168+0000 7f4bba200700 1 -- 172.21.15.123:0/3932559052 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f4bb405f5f0 con 0x7f4bb41037a0 2024-03-30T04:48:07.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:07.168+0000 7f4bab400700 1 -- 172.21.15.123:0/3932559052 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f4ba00592f0 con 0x7f4bb41037a0 2024-03-30T04:48:07.167 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T04:48:07.167 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T04:48:07.167 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T04:48:07.167 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T04:48:07.167 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T04:48:07.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:07.170+0000 7f4bba200700 1 -- 172.21.15.123:0/3932559052 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4b9806cc30 msgr2=0x7f4b9806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:07.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:07.170+0000 7f4bba200700 1 --2- 172.21.15.123:0/3932559052 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4b9806cc30 0x7f4b9806f0e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f4ba400b840 tx=0x7f4ba4009e30 comp rx=0 tx=0).stop 2024-03-30T04:48:07.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:07.170+0000 7f4bba200700 1 -- 172.21.15.123:0/3932559052 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4bb41037a0 msgr2=0x7f4bb4116210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:07.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:07.170+0000 7f4bba200700 1 --2- 172.21.15.123:0/3932559052 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4bb41037a0 0x7f4bb4116210 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f4ba0002a50 tx=0x7f4ba0004600 comp rx=0 tx=0).stop 2024-03-30T04:48:07.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:07.170+0000 7f4bba200700 1 -- 172.21.15.123:0/3932559052 shutdown_connections 2024-03-30T04:48:07.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:07.170+0000 7f4bba200700 1 --2- 172.21.15.123:0/3932559052 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4b9806cc30 0x7f4b9806f0e0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:07.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:07.170+0000 7f4bba200700 1 --2- 172.21.15.123:0/3932559052 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4bb4102e00 0x7f4bb4115cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:07.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:07.170+0000 7f4bba200700 1 --2- 172.21.15.123:0/3932559052 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4bb41037a0 0x7f4bb4116210 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:07.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:07.170+0000 7f4bba200700 1 -- 172.21.15.123:0/3932559052 >> 172.21.15.123:0/3932559052 conn(0x7f4bb40f89b0 msgr2=0x7f4bb40f99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:48:07.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:07.171+0000 7f4bba200700 1 -- 172.21.15.123:0/3932559052 shutdown_connections 2024-03-30T04:48:07.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:07.171+0000 7f4bba200700 1 -- 172.21.15.123:0/3932559052 wait complete. 2024-03-30T04:48:07.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:48:07 smithi123 ceph-mon[139369]: from='client.34982 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:48:07.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:48:07 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3932559052' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:48:08.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:48:07 smithi135 ceph-mon[106011]: from='client.34982 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:48:08.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:48:07 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3932559052' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:48:08.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:48:08 smithi123 ceph-mon[139369]: pgmap v508: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:09.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:48:08 smithi135 ceph-mon[106011]: pgmap v508: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:10.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:48:10 smithi135 ceph-mon[106011]: pgmap v509: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:10.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:48:10 smithi123 ceph-mon[139369]: pgmap v509: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:12.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:48:12 smithi123 ceph-mon[139369]: pgmap v510: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:13.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:48:12 smithi135 ceph-mon[106011]: pgmap v510: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:14.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:48:14 smithi123 ceph-mon[139369]: pgmap v511: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:14.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:48:14 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:48:15.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:48:14 smithi135 ceph-mon[106011]: pgmap v511: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:15.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:48:14 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:48:16.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:48:16 smithi123 ceph-mon[139369]: pgmap v512: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:17.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:48:16 smithi135 ceph-mon[106011]: pgmap v512: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:18.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:48:18 smithi123 ceph-mon[139369]: pgmap v513: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:19.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:48:18 smithi135 ceph-mon[106011]: pgmap v513: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:20.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:48:20 smithi135 ceph-mon[106011]: pgmap v514: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:20.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:48:20 smithi123 ceph-mon[139369]: pgmap v514: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:22.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:48:22 smithi123 ceph-mon[139369]: pgmap v515: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:23.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:48:22 smithi135 ceph-mon[106011]: pgmap v515: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:24.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:48:24 smithi123 ceph-mon[139369]: pgmap v516: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:25.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:48:24 smithi135 ceph-mon[106011]: pgmap v516: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:26.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:48:26 smithi123 ceph-mon[139369]: pgmap v517: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:27.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:48:26 smithi135 ceph-mon[106011]: pgmap v517: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:28.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:48:28 smithi123 ceph-mon[139369]: pgmap v518: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:29.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:48:28 smithi135 ceph-mon[106011]: pgmap v518: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:29.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:48:29 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:48:30.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:48:29 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:48:30.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:48:30 smithi123 ceph-mon[139369]: pgmap v519: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:31.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:48:30 smithi135 ceph-mon[106011]: pgmap v519: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:32.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:48:32 smithi123 ceph-mon[139369]: pgmap v520: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:33.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:48:32 smithi135 ceph-mon[106011]: pgmap v520: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:34.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:48:34 smithi123 ceph-mon[139369]: pgmap v521: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:35.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:48:34 smithi135 ceph-mon[106011]: pgmap v521: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:36.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:48:36 smithi123 ceph-mon[139369]: pgmap v522: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:37.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:48:36 smithi135 ceph-mon[106011]: pgmap v522: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:37.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.334+0000 7f8b9f600700 1 -- 172.21.15.123:0/1987363332 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b98102e00 msgr2=0x7f8b981031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:37.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.334+0000 7f8b9f600700 1 --2- 172.21.15.123:0/1987363332 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b98102e00 0x7f8b981031d0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c004940 tx=0x7f8b8c009e30 comp rx=0 tx=0).stop 2024-03-30T04:48:37.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.334+0000 7f8b9f600700 1 -- 172.21.15.123:0/1987363332 shutdown_connections 2024-03-30T04:48:37.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.334+0000 7f8b9f600700 1 --2- 172.21.15.123:0/1987363332 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8b981037a0 0x7f8b980fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:37.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.334+0000 7f8b9f600700 1 --2- 172.21.15.123:0/1987363332 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b98102e00 0x7f8b981031d0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:37.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.334+0000 7f8b9f600700 1 -- 172.21.15.123:0/1987363332 >> 172.21.15.123:0/1987363332 conn(0x7f8b980f89b0 msgr2=0x7f8b980fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:48:37.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.334+0000 7f8b9f600700 1 -- 172.21.15.123:0/1987363332 shutdown_connections 2024-03-30T04:48:37.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.334+0000 7f8b9f600700 1 -- 172.21.15.123:0/1987363332 wait complete. 2024-03-30T04:48:37.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.335+0000 7f8b9f600700 1 Processor -- start 2024-03-30T04:48:37.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.335+0000 7f8b9f600700 1 -- start start 2024-03-30T04:48:37.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.336+0000 7f8b9f600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b98102e00 0x7f8b98115ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:48:37.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.336+0000 7f8b9f600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8b981037a0 0x7f8b981161e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:48:37.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.336+0000 7f8b9f600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b98116830 con 0x7f8b98102e00 2024-03-30T04:48:37.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.336+0000 7f8b9f600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9810fc00 con 0x7f8b981037a0 2024-03-30T04:48:37.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.336+0000 7f8b9cc00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b98102e00 0x7f8b98115ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:48:37.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.336+0000 7f8b9cc00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b98102e00 0x7f8b98115ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54890/0 (socket says 172.21.15.123:54890) 2024-03-30T04:48:37.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.336+0000 7f8b9cc00700 1 -- 172.21.15.123:0/560112825 learned_addr learned my addr 172.21.15.123:0/560112825 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:48:37.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.336+0000 7f8b97e00700 1 --2- 172.21.15.123:0/560112825 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8b981037a0 0x7f8b981161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:48:37.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.337+0000 7f8b9cc00700 1 -- 172.21.15.123:0/560112825 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8b981037a0 msgr2=0x7f8b981161e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:37.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.337+0000 7f8b9cc00700 1 --2- 172.21.15.123:0/560112825 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8b981037a0 0x7f8b981161e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:37.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.337+0000 7f8b9cc00700 1 -- 172.21.15.123:0/560112825 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b8c005040 con 0x7f8b98102e00 2024-03-30T04:48:37.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.337+0000 7f8b9cc00700 1 --2- 172.21.15.123:0/560112825 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b98102e00 0x7f8b98115ca0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c008d80 tx=0x7f8b8c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:48:37.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.337+0000 7f8b94c00700 1 -- 172.21.15.123:0/560112825 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b8c00e030 con 0x7f8b98102e00 2024-03-30T04:48:37.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.337+0000 7f8b9f600700 1 -- 172.21.15.123:0/560112825 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b9810fe80 con 0x7f8b98102e00 2024-03-30T04:48:37.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.337+0000 7f8b94c00700 1 -- 172.21.15.123:0/560112825 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8b8c013de0 con 0x7f8b98102e00 2024-03-30T04:48:37.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.337+0000 7f8b94c00700 1 -- 172.21.15.123:0/560112825 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b8c01cd00 con 0x7f8b98102e00 2024-03-30T04:48:37.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.337+0000 7f8b9f600700 1 -- 172.21.15.123:0/560112825 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b98110440 con 0x7f8b98102e00 2024-03-30T04:48:37.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.339+0000 7f8b9f600700 1 -- 172.21.15.123:0/560112825 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b980fe550 con 0x7f8b98102e00 2024-03-30T04:48:37.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.340+0000 7f8b94c00700 1 -- 172.21.15.123:0/560112825 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f8b8c013930 con 0x7f8b98102e00 2024-03-30T04:48:37.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.340+0000 7f8b94c00700 1 --2- 172.21.15.123:0/560112825 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8b8006cc30 0x7f8b8006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:48:37.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.341+0000 7f8b94c00700 1 -- 172.21.15.123:0/560112825 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f8b8c08d940 con 0x7f8b98102e00 2024-03-30T04:48:37.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.341+0000 7f8b97e00700 1 --2- 172.21.15.123:0/560112825 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8b8006cc30 0x7f8b8006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:48:37.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.341+0000 7f8b97e00700 1 --2- 172.21.15.123:0/560112825 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8b8006cc30 0x7f8b8006f0e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8b98111540 tx=0x7f8b8400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:48:37.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.345+0000 7f8b94c00700 1 -- 172.21.15.123:0/560112825 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8b8c05b390 con 0x7f8b98102e00 2024-03-30T04:48:37.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.568+0000 7f8b9f600700 1 -- 172.21.15.123:0/560112825 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8b9804bed0 con 0x7f8b8006cc30 2024-03-30T04:48:37.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.569+0000 7f8b94c00700 1 -- 172.21.15.123:0/560112825 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f8b9804bed0 con 0x7f8b8006cc30 2024-03-30T04:48:37.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.571+0000 7f8b9f600700 1 -- 172.21.15.123:0/560112825 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8b8006cc30 msgr2=0x7f8b8006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:37.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.571+0000 7f8b9f600700 1 --2- 172.21.15.123:0/560112825 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8b8006cc30 0x7f8b8006f0e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8b98111540 tx=0x7f8b8400b040 comp rx=0 tx=0).stop 2024-03-30T04:48:37.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.571+0000 7f8b9f600700 1 -- 172.21.15.123:0/560112825 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b98102e00 msgr2=0x7f8b98115ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:37.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.571+0000 7f8b9f600700 1 --2- 172.21.15.123:0/560112825 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b98102e00 0x7f8b98115ca0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c008d80 tx=0x7f8b8c009e30 comp rx=0 tx=0).stop 2024-03-30T04:48:37.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.571+0000 7f8b9f600700 1 -- 172.21.15.123:0/560112825 shutdown_connections 2024-03-30T04:48:37.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.571+0000 7f8b9f600700 1 --2- 172.21.15.123:0/560112825 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8b8006cc30 0x7f8b8006f0e0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:37.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.571+0000 7f8b9f600700 1 --2- 172.21.15.123:0/560112825 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b98102e00 0x7f8b98115ca0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:37.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.571+0000 7f8b9f600700 1 --2- 172.21.15.123:0/560112825 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8b981037a0 0x7f8b981161e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:37.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.571+0000 7f8b9f600700 1 -- 172.21.15.123:0/560112825 >> 172.21.15.123:0/560112825 conn(0x7f8b980f89b0 msgr2=0x7f8b980f9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:48:37.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.572+0000 7f8b9f600700 1 -- 172.21.15.123:0/560112825 shutdown_connections 2024-03-30T04:48:37.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.572+0000 7f8b9f600700 1 -- 172.21.15.123:0/560112825 wait complete. 2024-03-30T04:48:37.584 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:48:37.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.707+0000 7fe5ace00700 1 -- 172.21.15.123:0/499556229 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5a81059a0 msgr2=0x7fe5a80ff010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:37.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.707+0000 7fe5ace00700 1 --2- 172.21.15.123:0/499556229 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5a81059a0 0x7fe5a80ff010 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fe594004940 tx=0x7fe594009e30 comp rx=0 tx=0).stop 2024-03-30T04:48:37.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.707+0000 7fe5ace00700 1 -- 172.21.15.123:0/499556229 shutdown_connections 2024-03-30T04:48:37.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.707+0000 7fe5ace00700 1 --2- 172.21.15.123:0/499556229 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5a81059a0 0x7fe5a80ff010 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:37.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.707+0000 7fe5ace00700 1 --2- 172.21.15.123:0/499556229 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe5a8105000 0x7fe5a81053d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:37.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.707+0000 7fe5ace00700 1 -- 172.21.15.123:0/499556229 >> 172.21.15.123:0/499556229 conn(0x7fe5a80fabd0 msgr2=0x7fe5a80fcfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:48:37.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.707+0000 7fe5ace00700 1 -- 172.21.15.123:0/499556229 shutdown_connections 2024-03-30T04:48:37.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.707+0000 7fe5ace00700 1 -- 172.21.15.123:0/499556229 wait complete. 2024-03-30T04:48:37.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.708+0000 7fe5ace00700 1 Processor -- start 2024-03-30T04:48:37.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.708+0000 7fe5ace00700 1 -- start start 2024-03-30T04:48:37.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.708+0000 7fe5ace00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5a8105000 0x7fe5a8115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:48:37.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.708+0000 7fe5ace00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe5a81059a0 0x7fe5a8116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:48:37.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.708+0000 7fe5ace00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a8116860 con 0x7fe5a8105000 2024-03-30T04:48:37.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.708+0000 7fe5ace00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a810fc30 con 0x7fe5a81059a0 2024-03-30T04:48:37.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.709+0000 7fe5a5e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5a8105000 0x7fe5a8115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:48:37.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.709+0000 7fe5a5e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5a8105000 0x7fe5a8115cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54918/0 (socket says 172.21.15.123:54918) 2024-03-30T04:48:37.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.709+0000 7fe5a5e00700 1 -- 172.21.15.123:0/2849711481 learned_addr learned my addr 172.21.15.123:0/2849711481 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:48:37.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.709+0000 7fe5a5e00700 1 -- 172.21.15.123:0/2849711481 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe5a81059a0 msgr2=0x7fe5a8116210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:37.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.709+0000 7fe5a5e00700 1 --2- 172.21.15.123:0/2849711481 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe5a81059a0 0x7fe5a8116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:37.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.709+0000 7fe5a5e00700 1 -- 172.21.15.123:0/2849711481 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe594005040 con 0x7fe5a8105000 2024-03-30T04:48:37.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.709+0000 7fe5a5e00700 1 --2- 172.21.15.123:0/2849711481 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5a8105000 0x7fe5a8115cd0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fe598002a50 tx=0x7fe5980046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:48:37.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.709+0000 7fe59e000700 1 -- 172.21.15.123:0/2849711481 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe598007cc0 con 0x7fe5a8105000 2024-03-30T04:48:37.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.710+0000 7fe59e000700 1 -- 172.21.15.123:0/2849711481 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe59800f040 con 0x7fe5a8105000 2024-03-30T04:48:37.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.710+0000 7fe5ace00700 1 -- 172.21.15.123:0/2849711481 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5a810ff10 con 0x7fe5a8105000 2024-03-30T04:48:37.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.710+0000 7fe59e000700 1 -- 172.21.15.123:0/2849711481 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe598009540 con 0x7fe5a8105000 2024-03-30T04:48:37.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.710+0000 7fe5ace00700 1 -- 172.21.15.123:0/2849711481 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5a81104d0 con 0x7fe5a8105000 2024-03-30T04:48:37.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.711+0000 7fe5ace00700 1 -- 172.21.15.123:0/2849711481 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5a8102310 con 0x7fe5a8105000 2024-03-30T04:48:37.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.712+0000 7fe59e000700 1 -- 172.21.15.123:0/2849711481 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fe5980096a0 con 0x7fe5a8105000 2024-03-30T04:48:37.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.712+0000 7fe59e000700 1 --2- 172.21.15.123:0/2849711481 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe58c06cc30 0x7fe58c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:48:37.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.713+0000 7fe59e000700 1 -- 172.21.15.123:0/2849711481 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe59808bde0 con 0x7fe5a8105000 2024-03-30T04:48:37.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.713+0000 7fe5a5400700 1 --2- 172.21.15.123:0/2849711481 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe58c06cc30 0x7fe58c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:48:37.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.713+0000 7fe5a5400700 1 --2- 172.21.15.123:0/2849711481 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe58c06cc30 0x7fe58c06f0e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe594004f90 tx=0x7fe594009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:48:37.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.717+0000 7fe59e000700 1 -- 172.21.15.123:0/2849711481 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe598059830 con 0x7fe5a8105000 2024-03-30T04:48:37.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.939+0000 7fe5ace00700 1 -- 172.21.15.123:0/2849711481 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe5a8111390 con 0x7fe58c06cc30 2024-03-30T04:48:37.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.940+0000 7fe59e000700 1 -- 172.21.15.123:0/2849711481 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fe5a8111390 con 0x7fe58c06cc30 2024-03-30T04:48:37.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.943+0000 7fe5ace00700 1 -- 172.21.15.123:0/2849711481 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe58c06cc30 msgr2=0x7fe58c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:37.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.943+0000 7fe5ace00700 1 --2- 172.21.15.123:0/2849711481 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe58c06cc30 0x7fe58c06f0e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe594004f90 tx=0x7fe594009e30 comp rx=0 tx=0).stop 2024-03-30T04:48:37.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.943+0000 7fe5ace00700 1 -- 172.21.15.123:0/2849711481 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5a8105000 msgr2=0x7fe5a8115cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:37.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.943+0000 7fe5ace00700 1 --2- 172.21.15.123:0/2849711481 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5a8105000 0x7fe5a8115cd0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fe598002a50 tx=0x7fe5980046b0 comp rx=0 tx=0).stop 2024-03-30T04:48:37.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.943+0000 7fe5ace00700 1 -- 172.21.15.123:0/2849711481 shutdown_connections 2024-03-30T04:48:37.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.943+0000 7fe5ace00700 1 --2- 172.21.15.123:0/2849711481 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe58c06cc30 0x7fe58c06f0e0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:37.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.943+0000 7fe5ace00700 1 --2- 172.21.15.123:0/2849711481 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5a8105000 0x7fe5a8115cd0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:37.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.943+0000 7fe5ace00700 1 --2- 172.21.15.123:0/2849711481 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe5a81059a0 0x7fe5a8116210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:37.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.943+0000 7fe5ace00700 1 -- 172.21.15.123:0/2849711481 >> 172.21.15.123:0/2849711481 conn(0x7fe5a80fabd0 msgr2=0x7fe5a80fcf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:48:37.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.943+0000 7fe5ace00700 1 -- 172.21.15.123:0/2849711481 shutdown_connections 2024-03-30T04:48:37.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:37.943+0000 7fe5ace00700 1 -- 172.21.15.123:0/2849711481 wait complete. 2024-03-30T04:48:38.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.077+0000 7fe98fe00700 1 -- 172.21.15.123:0/3978132601 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe990070360 msgr2=0x7fe990070730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:38.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.077+0000 7fe98fe00700 1 --2- 172.21.15.123:0/3978132601 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe990070360 0x7fe990070730 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fe980004970 tx=0x7fe980009e30 comp rx=0 tx=0).stop 2024-03-30T04:48:38.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.077+0000 7fe98fe00700 1 -- 172.21.15.123:0/3978132601 shutdown_connections 2024-03-30T04:48:38.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.077+0000 7fe98fe00700 1 --2- 172.21.15.123:0/3978132601 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe990070c70 0x7fe990108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:38.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.077+0000 7fe98fe00700 1 --2- 172.21.15.123:0/3978132601 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe990070360 0x7fe990070730 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:38.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.077+0000 7fe98fe00700 1 -- 172.21.15.123:0/3978132601 >> 172.21.15.123:0/3978132601 conn(0x7fe9900f8830 msgr2=0x7fe9900fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:48:38.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.077+0000 7fe98fe00700 1 -- 172.21.15.123:0/3978132601 shutdown_connections 2024-03-30T04:48:38.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.077+0000 7fe98fe00700 1 -- 172.21.15.123:0/3978132601 wait complete. 2024-03-30T04:48:38.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.078+0000 7fe98fe00700 1 Processor -- start 2024-03-30T04:48:38.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.078+0000 7fe98fe00700 1 -- start start 2024-03-30T04:48:38.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.079+0000 7fe98fe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe990070360 0x7fe990115af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:48:38.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.079+0000 7fe98fe00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe990070c70 0x7fe990116030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:48:38.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.079+0000 7fe98fe00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe990116680 con 0x7fe990070360 2024-03-30T04:48:38.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.079+0000 7fe98fe00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe99010fa70 con 0x7fe990070c70 2024-03-30T04:48:38.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.079+0000 7fe98ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe990070360 0x7fe990115af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:48:38.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.079+0000 7fe98ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe990070360 0x7fe990115af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54950/0 (socket says 172.21.15.123:54950) 2024-03-30T04:48:38.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.079+0000 7fe98ea00700 1 -- 172.21.15.123:0/3486408376 learned_addr learned my addr 172.21.15.123:0/3486408376 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:48:38.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.079+0000 7fe98ea00700 1 -- 172.21.15.123:0/3486408376 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe990070c70 msgr2=0x7fe990116030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:38.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.079+0000 7fe98e000700 1 --2- 172.21.15.123:0/3486408376 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe990070c70 0x7fe990116030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:48:38.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.079+0000 7fe98ea00700 1 --2- 172.21.15.123:0/3486408376 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe990070c70 0x7fe990116030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:38.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.079+0000 7fe98ea00700 1 -- 172.21.15.123:0/3486408376 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe980005040 con 0x7fe990070360 2024-03-30T04:48:38.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.079+0000 7fe98e000700 1 --2- 172.21.15.123:0/3486408376 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe990070c70 0x7fe990116030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:48:38.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.080+0000 7fe98ea00700 1 --2- 172.21.15.123:0/3486408376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe990070360 0x7fe990115af0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fe980009360 tx=0x7fe980009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:48:38.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.080+0000 7fe986a00700 1 -- 172.21.15.123:0/3486408376 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe98000e040 con 0x7fe990070360 2024-03-30T04:48:38.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.080+0000 7fe986a00700 1 -- 172.21.15.123:0/3486408376 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe980004290 con 0x7fe990070360 2024-03-30T04:48:38.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.080+0000 7fe986a00700 1 -- 172.21.15.123:0/3486408376 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe98001c3f0 con 0x7fe990070360 2024-03-30T04:48:38.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.080+0000 7fe98fe00700 1 -- 172.21.15.123:0/3486408376 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe99010fcf0 con 0x7fe990070360 2024-03-30T04:48:38.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.080+0000 7fe98fe00700 1 -- 172.21.15.123:0/3486408376 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe990110230 con 0x7fe990070360 2024-03-30T04:48:38.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.081+0000 7fe98fe00700 1 -- 172.21.15.123:0/3486408376 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe990071aa0 con 0x7fe990070360 2024-03-30T04:48:38.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.082+0000 7fe986a00700 1 -- 172.21.15.123:0/3486408376 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fe980013420 con 0x7fe990070360 2024-03-30T04:48:38.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.083+0000 7fe986a00700 1 --2- 172.21.15.123:0/3486408376 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe97406cc30 0x7fe97406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:48:38.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.083+0000 7fe986a00700 1 -- 172.21.15.123:0/3486408376 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe98008d450 con 0x7fe990070360 2024-03-30T04:48:38.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.083+0000 7fe98e000700 1 --2- 172.21.15.123:0/3486408376 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe97406cc30 0x7fe97406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:48:38.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.084+0000 7fe98e000700 1 --2- 172.21.15.123:0/3486408376 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe97406cc30 0x7fe97406f0e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe97c009fd0 tx=0x7fe97c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:48:38.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.088+0000 7fe986a00700 1 -- 172.21.15.123:0/3486408376 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe98005afd0 con 0x7fe990070360 2024-03-30T04:48:38.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.298+0000 7fe98fe00700 1 -- 172.21.15.123:0/3486408376 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fe99010bdf0 con 0x7fe97406cc30 2024-03-30T04:48:38.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.308+0000 7fe986a00700 1 -- 172.21.15.123:0/3486408376 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fe99010bdf0 con 0x7fe97406cc30 2024-03-30T04:48:38.306 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:48:38.306 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (17m) 10m ago 30m 15.0M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:48:38.306 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (15m) 10m ago 30m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T04:48:38.306 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (14m) 10m ago 29m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T04:48:38.306 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (17m) 10m ago 30m 74.0M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T04:48:38.306 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (10m) 10m ago 23m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T04:48:38.306 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (11m) 10m ago 24m 10.9M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T04:48:38.306 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (23m) 10m ago 23m 10.7M - 17.2.7-797-g5c024335 f77085f6aab7 0dc41326818c 2024-03-30T04:48:38.306 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (10m) 10m ago 24m 9747k - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T04:48:38.307 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (20m) 10m ago 31m 506M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:48:38.307 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (19m) 10m ago 29m 450M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:48:38.307 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (16m) 10m ago 32m 53.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T04:48:38.307 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (15m) 10m ago 29m 42.6M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T04:48:38.307 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (18m) 10m ago 30m 8052k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:48:38.307 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (18m) 10m ago 29m 7880k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:48:38.307 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (14m) 10m ago 28m 104M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T04:48:38.307 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (14m) 10m ago 27m 93.4M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T04:48:38.307 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (13m) 10m ago 27m 86.3M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T04:48:38.307 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (12m) 10m ago 26m 97.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T04:48:38.307 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (12m) 10m ago 26m 97.9M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T04:48:38.307 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (11m) 10m ago 25m 85.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T04:48:38.307 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (18m) 10m ago 29m 53.7M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:48:38.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.310+0000 7fe98fe00700 1 -- 172.21.15.123:0/3486408376 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe97406cc30 msgr2=0x7fe97406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:38.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.310+0000 7fe98fe00700 1 --2- 172.21.15.123:0/3486408376 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe97406cc30 0x7fe97406f0e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe97c009fd0 tx=0x7fe97c00b040 comp rx=0 tx=0).stop 2024-03-30T04:48:38.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.310+0000 7fe98fe00700 1 -- 172.21.15.123:0/3486408376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe990070360 msgr2=0x7fe990115af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:38.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.310+0000 7fe98fe00700 1 --2- 172.21.15.123:0/3486408376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe990070360 0x7fe990115af0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fe980009360 tx=0x7fe980009e30 comp rx=0 tx=0).stop 2024-03-30T04:48:38.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.310+0000 7fe98fe00700 1 -- 172.21.15.123:0/3486408376 shutdown_connections 2024-03-30T04:48:38.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.310+0000 7fe98fe00700 1 --2- 172.21.15.123:0/3486408376 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe97406cc30 0x7fe97406f0e0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:38.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.310+0000 7fe98fe00700 1 --2- 172.21.15.123:0/3486408376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe990070360 0x7fe990115af0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:38.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.310+0000 7fe98fe00700 1 --2- 172.21.15.123:0/3486408376 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe990070c70 0x7fe990116030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:38.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.310+0000 7fe98fe00700 1 -- 172.21.15.123:0/3486408376 >> 172.21.15.123:0/3486408376 conn(0x7fe9900f8830 msgr2=0x7fe9900f9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:48:38.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.310+0000 7fe98fe00700 1 -- 172.21.15.123:0/3486408376 shutdown_connections 2024-03-30T04:48:38.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.310+0000 7fe98fe00700 1 -- 172.21.15.123:0/3486408376 wait complete. 2024-03-30T04:48:38.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.443+0000 7f1b56e00700 1 -- 172.21.15.123:0/2620005289 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b50070c10 msgr2=0x7f1b50109150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:38.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.443+0000 7f1b56e00700 1 --2- 172.21.15.123:0/2620005289 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b50070c10 0x7f1b50109150 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c004970 tx=0x7f1b3c009e30 comp rx=0 tx=0).stop 2024-03-30T04:48:38.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.443+0000 7f1b56e00700 1 -- 172.21.15.123:0/2620005289 shutdown_connections 2024-03-30T04:48:38.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.443+0000 7f1b56e00700 1 --2- 172.21.15.123:0/2620005289 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b50070c10 0x7f1b50109150 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:38.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.443+0000 7f1b56e00700 1 --2- 172.21.15.123:0/2620005289 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1b50070300 0x7f1b500706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:38.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.443+0000 7f1b56e00700 1 -- 172.21.15.123:0/2620005289 >> 172.21.15.123:0/2620005289 conn(0x7f1b500f8950 msgr2=0x7f1b500fad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:48:38.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.443+0000 7f1b56e00700 1 -- 172.21.15.123:0/2620005289 shutdown_connections 2024-03-30T04:48:38.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.443+0000 7f1b56e00700 1 -- 172.21.15.123:0/2620005289 wait complete. 2024-03-30T04:48:38.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.444+0000 7f1b56e00700 1 Processor -- start 2024-03-30T04:48:38.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.444+0000 7f1b56e00700 1 -- start start 2024-03-30T04:48:38.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.444+0000 7f1b56e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1b50070300 0x7f1b50115a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:48:38.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.444+0000 7f1b56e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b50070c10 0x7f1b50115f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:48:38.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.444+0000 7f1b56e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b501165d0 con 0x7f1b50070c10 2024-03-30T04:48:38.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.444+0000 7f1b56e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5010fbb0 con 0x7f1b50070300 2024-03-30T04:48:38.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.445+0000 7f1b4f400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b50070c10 0x7f1b50115f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:48:38.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.445+0000 7f1b4f400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b50070c10 0x7f1b50115f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54966/0 (socket says 172.21.15.123:54966) 2024-03-30T04:48:38.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.445+0000 7f1b4f400700 1 -- 172.21.15.123:0/458393940 learned_addr learned my addr 172.21.15.123:0/458393940 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:48:38.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.445+0000 7f1b4fe00700 1 --2- 172.21.15.123:0/458393940 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1b50070300 0x7f1b50115a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:48:38.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.445+0000 7f1b4f400700 1 -- 172.21.15.123:0/458393940 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1b50070300 msgr2=0x7f1b50115a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:38.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.445+0000 7f1b4f400700 1 --2- 172.21.15.123:0/458393940 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1b50070300 0x7f1b50115a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:38.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.445+0000 7f1b4f400700 1 -- 172.21.15.123:0/458393940 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b3c005040 con 0x7f1b50070c10 2024-03-30T04:48:38.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.445+0000 7f1b4f400700 1 --2- 172.21.15.123:0/458393940 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b50070c10 0x7f1b50115f80 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c005010 tx=0x7f1b3c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:48:38.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.446+0000 7f1b47e00700 1 -- 172.21.15.123:0/458393940 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b3c00e040 con 0x7f1b50070c10 2024-03-30T04:48:38.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.446+0000 7f1b47e00700 1 -- 172.21.15.123:0/458393940 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1b3c004290 con 0x7f1b50070c10 2024-03-30T04:48:38.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.446+0000 7f1b56e00700 1 -- 172.21.15.123:0/458393940 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b5010fe30 con 0x7f1b50070c10 2024-03-30T04:48:38.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.446+0000 7f1b47e00700 1 -- 172.21.15.123:0/458393940 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b3c013e40 con 0x7f1b50070c10 2024-03-30T04:48:38.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.446+0000 7f1b56e00700 1 -- 172.21.15.123:0/458393940 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b50110370 con 0x7f1b50070c10 2024-03-30T04:48:38.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.447+0000 7f1b56e00700 1 -- 172.21.15.123:0/458393940 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b50071a40 con 0x7f1b50070c10 2024-03-30T04:48:38.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.448+0000 7f1b47e00700 1 -- 172.21.15.123:0/458393940 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f1b3c013620 con 0x7f1b50070c10 2024-03-30T04:48:38.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.449+0000 7f1b47e00700 1 --2- 172.21.15.123:0/458393940 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1b3406cc20 0x7f1b3406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:48:38.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.449+0000 7f1b47e00700 1 -- 172.21.15.123:0/458393940 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f1b3c08d200 con 0x7f1b50070c10 2024-03-30T04:48:38.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.449+0000 7f1b4fe00700 1 --2- 172.21.15.123:0/458393940 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1b3406cc20 0x7f1b3406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:48:38.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.450+0000 7f1b4fe00700 1 --2- 172.21.15.123:0/458393940 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1b3406cc20 0x7f1b3406f0d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1b40004600 tx=0x7f1b4000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:48:38.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.453+0000 7f1b47e00700 1 -- 172.21.15.123:0/458393940 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f1b3c05ad90 con 0x7f1b50070c10 2024-03-30T04:48:38.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.737+0000 7f1b56e00700 1 -- 172.21.15.123:0/458393940 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f1b5010bfa0 con 0x7f1b50070c10 2024-03-30T04:48:38.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.738+0000 7f1b47e00700 1 -- 172.21.15.123:0/458393940 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f1b3c05a920 con 0x7f1b50070c10 2024-03-30T04:48:38.736 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:48:38.736 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:48:38.736 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:48:38.736 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:48:38.737 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:48:38.737 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:48:38.737 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:48:38.737 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:48:38.737 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T04:48:38.737 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:48:38.737 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:48:38.737 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:48:38.737 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 3 2024-03-30T04:48:38.737 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:48:38.737 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:48:38.737 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:48:38.737 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 13 2024-03-30T04:48:38.737 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:48:38.738 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:48:38.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.740+0000 7f1b56e00700 1 -- 172.21.15.123:0/458393940 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1b3406cc20 msgr2=0x7f1b3406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:38.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.740+0000 7f1b56e00700 1 --2- 172.21.15.123:0/458393940 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1b3406cc20 0x7f1b3406f0d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1b40004600 tx=0x7f1b4000b040 comp rx=0 tx=0).stop 2024-03-30T04:48:38.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.740+0000 7f1b56e00700 1 -- 172.21.15.123:0/458393940 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b50070c10 msgr2=0x7f1b50115f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:38.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.740+0000 7f1b56e00700 1 --2- 172.21.15.123:0/458393940 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b50070c10 0x7f1b50115f80 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c005010 tx=0x7f1b3c009e30 comp rx=0 tx=0).stop 2024-03-30T04:48:38.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.740+0000 7f1b56e00700 1 -- 172.21.15.123:0/458393940 shutdown_connections 2024-03-30T04:48:38.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.740+0000 7f1b56e00700 1 --2- 172.21.15.123:0/458393940 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1b3406cc20 0x7f1b3406f0d0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:38.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.740+0000 7f1b56e00700 1 --2- 172.21.15.123:0/458393940 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1b50070300 0x7f1b50115a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:38.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.740+0000 7f1b56e00700 1 --2- 172.21.15.123:0/458393940 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b50070c10 0x7f1b50115f80 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:38.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.740+0000 7f1b56e00700 1 -- 172.21.15.123:0/458393940 >> 172.21.15.123:0/458393940 conn(0x7f1b500f8950 msgr2=0x7f1b50105a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:48:38.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.740+0000 7f1b56e00700 1 -- 172.21.15.123:0/458393940 shutdown_connections 2024-03-30T04:48:38.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.740+0000 7f1b56e00700 1 -- 172.21.15.123:0/458393940 wait complete. 2024-03-30T04:48:38.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.873+0000 7f6fb1200700 1 -- 172.21.15.123:0/3147988369 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fac102c40 msgr2=0x7f6fac103010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:38.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.873+0000 7f6fb1200700 1 --2- 172.21.15.123:0/3147988369 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fac102c40 0x7f6fac103010 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c004940 tx=0x7f6f9c009e30 comp rx=0 tx=0).stop 2024-03-30T04:48:38.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.873+0000 7f6fb1200700 1 -- 172.21.15.123:0/3147988369 shutdown_connections 2024-03-30T04:48:38.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.873+0000 7f6fb1200700 1 --2- 172.21.15.123:0/3147988369 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6fac1035e0 0x7f6fac0fcc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:38.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.873+0000 7f6fb1200700 1 --2- 172.21.15.123:0/3147988369 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fac102c40 0x7f6fac103010 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:38.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.873+0000 7f6fb1200700 1 -- 172.21.15.123:0/3147988369 >> 172.21.15.123:0/3147988369 conn(0x7f6fac0f8830 msgr2=0x7f6fac0fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:48:38.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.873+0000 7f6fb1200700 1 -- 172.21.15.123:0/3147988369 shutdown_connections 2024-03-30T04:48:38.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.873+0000 7f6fb1200700 1 -- 172.21.15.123:0/3147988369 wait complete. 2024-03-30T04:48:38.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.874+0000 7f6fb1200700 1 Processor -- start 2024-03-30T04:48:38.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.874+0000 7f6fb1200700 1 -- start start 2024-03-30T04:48:38.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.874+0000 7f6fb1200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fac102c40 0x7f6fac113950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:48:38.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.874+0000 7f6fb1200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6fac1035e0 0x7f6fac113e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:48:38.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.874+0000 7f6fb1200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fac1144e0 con 0x7f6fac102c40 2024-03-30T04:48:38.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.874+0000 7f6fb1200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fac10d8d0 con 0x7f6fac1035e0 2024-03-30T04:48:38.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.875+0000 7f6fab400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fac102c40 0x7f6fac113950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:48:38.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.875+0000 7f6fab400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fac102c40 0x7f6fac113950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54984/0 (socket says 172.21.15.123:54984) 2024-03-30T04:48:38.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.875+0000 7f6fab400700 1 -- 172.21.15.123:0/3296870361 learned_addr learned my addr 172.21.15.123:0/3296870361 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:48:38.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.875+0000 7f6faaa00700 1 --2- 172.21.15.123:0/3296870361 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6fac1035e0 0x7f6fac113e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:48:38.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.875+0000 7f6fab400700 1 -- 172.21.15.123:0/3296870361 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6fac1035e0 msgr2=0x7f6fac113e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:38.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.875+0000 7f6fab400700 1 --2- 172.21.15.123:0/3296870361 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6fac1035e0 0x7f6fac113e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:38.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.875+0000 7f6fab400700 1 -- 172.21.15.123:0/3296870361 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f9c005040 con 0x7f6fac102c40 2024-03-30T04:48:38.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.875+0000 7f6fab400700 1 --2- 172.21.15.123:0/3296870361 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fac102c40 0x7f6fac113950 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c00b3b0 tx=0x7f6f9c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:48:38.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.876+0000 7f6fa3400700 1 -- 172.21.15.123:0/3296870361 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f9c00e040 con 0x7f6fac102c40 2024-03-30T04:48:38.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.876+0000 7f6fa3400700 1 -- 172.21.15.123:0/3296870361 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6f9c013420 con 0x7f6fac102c40 2024-03-30T04:48:38.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.876+0000 7f6fb1200700 1 -- 172.21.15.123:0/3296870361 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fac10db50 con 0x7f6fac102c40 2024-03-30T04:48:38.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.876+0000 7f6fa3400700 1 -- 172.21.15.123:0/3296870361 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f9c01c630 con 0x7f6fac102c40 2024-03-30T04:48:38.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.876+0000 7f6fb1200700 1 -- 172.21.15.123:0/3296870361 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fac10e110 con 0x7f6fac102c40 2024-03-30T04:48:38.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.877+0000 7f6fb1200700 1 -- 172.21.15.123:0/3296870361 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fac0fe390 con 0x7f6fac102c40 2024-03-30T04:48:38.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.878+0000 7f6fa3400700 1 -- 172.21.15.123:0/3296870361 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f6f9c013970 con 0x7f6fac102c40 2024-03-30T04:48:38.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.878+0000 7f6fa3400700 1 --2- 172.21.15.123:0/3296870361 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6f9006cc20 0x7f6f9006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:48:38.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.879+0000 7f6fa3400700 1 -- 172.21.15.123:0/3296870361 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f6f9c08d230 con 0x7f6fac102c40 2024-03-30T04:48:38.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.879+0000 7f6faaa00700 1 --2- 172.21.15.123:0/3296870361 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6f9006cc20 0x7f6f9006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:48:38.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.879+0000 7f6faaa00700 1 --2- 172.21.15.123:0/3296870361 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6f9006cc20 0x7f6f9006f0d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6fac10ea90 tx=0x7f6f98009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:48:38.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:38.883+0000 7f6fa3400700 1 -- 172.21.15.123:0/3296870361 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6f9c05ad00 con 0x7f6fac102c40 2024-03-30T04:48:38.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:48:38 smithi123 ceph-mon[139369]: from='client.34990 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:48:38.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:48:38 smithi123 ceph-mon[139369]: pgmap v523: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:38.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:48:38 smithi123 ceph-mon[139369]: from='client.34994 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:48:39.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:48:38 smithi135 ceph-mon[106011]: from='client.34990 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:48:39.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:48:38 smithi135 ceph-mon[106011]: pgmap v523: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:39.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:48:38 smithi135 ceph-mon[106011]: from='client.34994 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:48:39.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.134+0000 7f6fb1200700 1 -- 172.21.15.123:0/3296870361 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f6fac109c40 con 0x7f6fac102c40 2024-03-30T04:48:39.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.134+0000 7f6fa3400700 1 -- 172.21.15.123:0/3296870361 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f6f9c05a890 con 0x7f6fac102c40 2024-03-30T04:48:39.133 INFO:teuthology.orchestra.run.smithi123.stdout:e32 2024-03-30T04:48:39.133 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:48:39.133 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:48:39.133 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:48:39.133 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:48:39.133 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:48:39.133 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:48:39.133 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T04:48:39.133 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T04:48:39.133 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:48:39.133 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T04:48:39.133 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:48:39.133 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:48:39.134 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:48:39.134 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:48:39.134 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:48:39.134 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T04:48:39.134 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:48:39.134 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:48:39.134 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T04:48:39.134 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:48:39.134 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T04:48:39.134 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T04:48:39.134 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T04:48:39.134 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T04:48:39.134 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:48:39.135 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:48:39.135 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:48:39.135 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:48:39.135 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:48:39.135 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:48:39.135 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T04:48:39.135 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T04:48:39.135 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:48:39.135 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:48:39.135 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T04:48:39.135 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:48:39.135 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:48:39.135 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:48:39.135 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44315} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/1741352336,v1:172.21.15.135:6827/1741352336] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:48:39.136 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:48:39.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.137+0000 7f6fb1200700 1 -- 172.21.15.123:0/3296870361 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6f9006cc20 msgr2=0x7f6f9006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:39.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.137+0000 7f6fb1200700 1 --2- 172.21.15.123:0/3296870361 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6f9006cc20 0x7f6f9006f0d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6fac10ea90 tx=0x7f6f98009ee0 comp rx=0 tx=0).stop 2024-03-30T04:48:39.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.137+0000 7f6fb1200700 1 -- 172.21.15.123:0/3296870361 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fac102c40 msgr2=0x7f6fac113950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:39.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.137+0000 7f6fb1200700 1 --2- 172.21.15.123:0/3296870361 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fac102c40 0x7f6fac113950 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c00b3b0 tx=0x7f6f9c009e30 comp rx=0 tx=0).stop 2024-03-30T04:48:39.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.137+0000 7f6fb1200700 1 -- 172.21.15.123:0/3296870361 shutdown_connections 2024-03-30T04:48:39.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.137+0000 7f6fb1200700 1 --2- 172.21.15.123:0/3296870361 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6f9006cc20 0x7f6f9006f0d0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:39.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.137+0000 7f6fb1200700 1 --2- 172.21.15.123:0/3296870361 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fac102c40 0x7f6fac113950 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:39.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.137+0000 7f6fb1200700 1 --2- 172.21.15.123:0/3296870361 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6fac1035e0 0x7f6fac113e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:39.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.137+0000 7f6fb1200700 1 -- 172.21.15.123:0/3296870361 >> 172.21.15.123:0/3296870361 conn(0x7f6fac0f8830 msgr2=0x7f6fac0f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:48:39.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.137+0000 7f6fb1200700 1 -- 172.21.15.123:0/3296870361 shutdown_connections 2024-03-30T04:48:39.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.137+0000 7f6fb1200700 1 -- 172.21.15.123:0/3296870361 wait complete. 2024-03-30T04:48:39.137 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 32 2024-03-30T04:48:39.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.272+0000 7f50ec000700 1 -- 172.21.15.123:0/210814922 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50e4070c70 msgr2=0x7f50e4109110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:39.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.273+0000 7f50ec000700 1 --2- 172.21.15.123:0/210814922 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50e4070c70 0x7f50e4109110 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f50d0004970 tx=0x7f50d0009e30 comp rx=0 tx=0).stop 2024-03-30T04:48:39.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.273+0000 7f50ec000700 1 -- 172.21.15.123:0/210814922 shutdown_connections 2024-03-30T04:48:39.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.273+0000 7f50ec000700 1 --2- 172.21.15.123:0/210814922 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50e4070c70 0x7f50e4109110 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:39.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.273+0000 7f50ec000700 1 --2- 172.21.15.123:0/210814922 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50e4070360 0x7f50e4070730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:39.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.273+0000 7f50ec000700 1 -- 172.21.15.123:0/210814922 >> 172.21.15.123:0/210814922 conn(0x7f50e40f89b0 msgr2=0x7f50e40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:48:39.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.273+0000 7f50ec000700 1 -- 172.21.15.123:0/210814922 shutdown_connections 2024-03-30T04:48:39.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.273+0000 7f50ec000700 1 -- 172.21.15.123:0/210814922 wait complete. 2024-03-30T04:48:39.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.274+0000 7f50ec000700 1 Processor -- start 2024-03-30T04:48:39.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.274+0000 7f50ec000700 1 -- start start 2024-03-30T04:48:39.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.274+0000 7f50ec000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50e4070360 0x7f50e4115ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:48:39.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.274+0000 7f50ec000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50e4070c70 0x7f50e41161e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:48:39.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.274+0000 7f50ec000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e4116830 con 0x7f50e4070c70 2024-03-30T04:48:39.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.274+0000 7f50ec000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e410fc00 con 0x7f50e4070360 2024-03-30T04:48:39.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.274+0000 7f50e8c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50e4070c70 0x7f50e41161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:48:39.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.274+0000 7f50e8c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50e4070c70 0x7f50e41161e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54992/0 (socket says 172.21.15.123:54992) 2024-03-30T04:48:39.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.274+0000 7f50e8c00700 1 -- 172.21.15.123:0/656540471 learned_addr learned my addr 172.21.15.123:0/656540471 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:48:39.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.275+0000 7f50e9600700 1 --2- 172.21.15.123:0/656540471 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50e4070360 0x7f50e4115ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:48:39.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.275+0000 7f50e8c00700 1 -- 172.21.15.123:0/656540471 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50e4070360 msgr2=0x7f50e4115ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:39.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.275+0000 7f50e8c00700 1 --2- 172.21.15.123:0/656540471 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50e4070360 0x7f50e4115ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:39.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.275+0000 7f50e8c00700 1 -- 172.21.15.123:0/656540471 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50d0005040 con 0x7f50e4070c70 2024-03-30T04:48:39.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.275+0000 7f50e8c00700 1 --2- 172.21.15.123:0/656540471 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50e4070c70 0x7f50e41161e0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f50d0009360 tx=0x7f50d0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:48:39.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.275+0000 7f50e1600700 1 -- 172.21.15.123:0/656540471 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f50d000e040 con 0x7f50e4070c70 2024-03-30T04:48:39.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.275+0000 7f50ec000700 1 -- 172.21.15.123:0/656540471 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50e410fe80 con 0x7f50e4070c70 2024-03-30T04:48:39.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.275+0000 7f50e1600700 1 -- 172.21.15.123:0/656540471 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f50d0004290 con 0x7f50e4070c70 2024-03-30T04:48:39.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.275+0000 7f50e1600700 1 -- 172.21.15.123:0/656540471 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f50d001c3f0 con 0x7f50e4070c70 2024-03-30T04:48:39.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.276+0000 7f50ec000700 1 -- 172.21.15.123:0/656540471 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50e41103c0 con 0x7f50e4070c70 2024-03-30T04:48:39.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.276+0000 7f50ec000700 1 -- 172.21.15.123:0/656540471 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50e4103bd0 con 0x7f50e4070c70 2024-03-30T04:48:39.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.278+0000 7f50e1600700 1 -- 172.21.15.123:0/656540471 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f50d0013420 con 0x7f50e4070c70 2024-03-30T04:48:39.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.278+0000 7f50e1600700 1 --2- 172.21.15.123:0/656540471 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f50cc06cc30 0x7f50cc06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:48:39.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.278+0000 7f50e1600700 1 -- 172.21.15.123:0/656540471 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f50d008d390 con 0x7f50e4070c70 2024-03-30T04:48:39.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.279+0000 7f50e9600700 1 --2- 172.21.15.123:0/656540471 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f50cc06cc30 0x7f50cc06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:48:39.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.279+0000 7f50e9600700 1 --2- 172.21.15.123:0/656540471 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f50cc06cc30 0x7f50cc06f0e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f50d8007fd0 tx=0x7f50d8009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:48:39.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.283+0000 7f50e1600700 1 -- 172.21.15.123:0/656540471 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f50d005afd0 con 0x7f50e4070c70 2024-03-30T04:48:39.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.507+0000 7f50ec000700 1 -- 172.21.15.123:0/656540471 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f50e41112f0 con 0x7f50cc06cc30 2024-03-30T04:48:39.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.509+0000 7f50e1600700 1 -- 172.21.15.123:0/656540471 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f50e41112f0 con 0x7f50cc06cc30 2024-03-30T04:48:39.507 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:48:39.507 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:48:39.507 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:48:39.507 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T04:48:39.507 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:48:39.507 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T04:48:39.507 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T04:48:39.507 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T04:48:39.508 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T04:48:39.508 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:48:39.508 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "15/21 daemons upgraded", 2024-03-30T04:48:39.508 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T04:48:39.508 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:48:39.508 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:48:39.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.511+0000 7f50ec000700 1 -- 172.21.15.123:0/656540471 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f50cc06cc30 msgr2=0x7f50cc06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:39.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.511+0000 7f50ec000700 1 --2- 172.21.15.123:0/656540471 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f50cc06cc30 0x7f50cc06f0e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f50d8007fd0 tx=0x7f50d8009040 comp rx=0 tx=0).stop 2024-03-30T04:48:39.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.511+0000 7f50ec000700 1 -- 172.21.15.123:0/656540471 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50e4070c70 msgr2=0x7f50e41161e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:39.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.511+0000 7f50ec000700 1 --2- 172.21.15.123:0/656540471 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50e4070c70 0x7f50e41161e0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f50d0009360 tx=0x7f50d0009e30 comp rx=0 tx=0).stop 2024-03-30T04:48:39.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.511+0000 7f50ec000700 1 -- 172.21.15.123:0/656540471 shutdown_connections 2024-03-30T04:48:39.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.511+0000 7f50ec000700 1 --2- 172.21.15.123:0/656540471 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f50cc06cc30 0x7f50cc06f0e0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:39.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.511+0000 7f50ec000700 1 --2- 172.21.15.123:0/656540471 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50e4070360 0x7f50e4115ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:39.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.511+0000 7f50ec000700 1 --2- 172.21.15.123:0/656540471 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50e4070c70 0x7f50e41161e0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:39.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.511+0000 7f50ec000700 1 -- 172.21.15.123:0/656540471 >> 172.21.15.123:0/656540471 conn(0x7f50e40f89b0 msgr2=0x7f50e41015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:48:39.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.511+0000 7f50ec000700 1 -- 172.21.15.123:0/656540471 shutdown_connections 2024-03-30T04:48:39.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.512+0000 7f50ec000700 1 -- 172.21.15.123:0/656540471 wait complete. 2024-03-30T04:48:39.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.646+0000 7f390a400700 1 -- 172.21.15.123:0/3018452281 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3904102e00 msgr2=0x7f39041031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:39.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.646+0000 7f390a400700 1 --2- 172.21.15.123:0/3018452281 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3904102e00 0x7f39041031d0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f38f4004940 tx=0x7f38f4009e30 comp rx=0 tx=0).stop 2024-03-30T04:48:39.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.646+0000 7f390a400700 1 -- 172.21.15.123:0/3018452281 shutdown_connections 2024-03-30T04:48:39.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.646+0000 7f390a400700 1 --2- 172.21.15.123:0/3018452281 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f39041037a0 0x7f39040fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:39.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.646+0000 7f390a400700 1 --2- 172.21.15.123:0/3018452281 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3904102e00 0x7f39041031d0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:39.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.646+0000 7f390a400700 1 -- 172.21.15.123:0/3018452281 >> 172.21.15.123:0/3018452281 conn(0x7f39040f89b0 msgr2=0x7f39040fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:48:39.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.646+0000 7f390a400700 1 -- 172.21.15.123:0/3018452281 shutdown_connections 2024-03-30T04:48:39.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.646+0000 7f390a400700 1 -- 172.21.15.123:0/3018452281 wait complete. 2024-03-30T04:48:39.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.647+0000 7f390a400700 1 Processor -- start 2024-03-30T04:48:39.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.647+0000 7f390a400700 1 -- start start 2024-03-30T04:48:39.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.647+0000 7f390a400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3904102e00 0x7f3904115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:48:39.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.647+0000 7f390a400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f39041037a0 0x7f3904116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:48:39.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.647+0000 7f390a400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3904116880 con 0x7f39041037a0 2024-03-30T04:48:39.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.647+0000 7f390a400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f390410fc50 con 0x7f3904102e00 2024-03-30T04:48:39.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.647+0000 7f3902a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f39041037a0 0x7f3904116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:48:39.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.647+0000 7f3902a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f39041037a0 0x7f3904116230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55016/0 (socket says 172.21.15.123:55016) 2024-03-30T04:48:39.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.647+0000 7f3902a00700 1 -- 172.21.15.123:0/3610260066 learned_addr learned my addr 172.21.15.123:0/3610260066 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:48:39.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.648+0000 7f3903400700 1 --2- 172.21.15.123:0/3610260066 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3904102e00 0x7f3904115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:48:39.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.648+0000 7f3902a00700 1 -- 172.21.15.123:0/3610260066 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3904102e00 msgr2=0x7f3904115cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:39.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.648+0000 7f3902a00700 1 --2- 172.21.15.123:0/3610260066 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3904102e00 0x7f3904115cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:39.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.648+0000 7f3902a00700 1 -- 172.21.15.123:0/3610260066 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38f4005040 con 0x7f39041037a0 2024-03-30T04:48:39.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.648+0000 7f3902a00700 1 --2- 172.21.15.123:0/3610260066 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f39041037a0 0x7f3904116230 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f38f0002a50 tx=0x7f38f0004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:48:39.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.648+0000 7f38fb400700 1 -- 172.21.15.123:0/3610260066 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f38f0007cc0 con 0x7f39041037a0 2024-03-30T04:48:39.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.648+0000 7f38fb400700 1 -- 172.21.15.123:0/3610260066 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f38f000f040 con 0x7f39041037a0 2024-03-30T04:48:39.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.648+0000 7f390a400700 1 -- 172.21.15.123:0/3610260066 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f390410ff30 con 0x7f39041037a0 2024-03-30T04:48:39.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.648+0000 7f38fb400700 1 -- 172.21.15.123:0/3610260066 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f38f000d920 con 0x7f39041037a0 2024-03-30T04:48:39.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.648+0000 7f390a400700 1 -- 172.21.15.123:0/3610260066 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39041104f0 con 0x7f39041037a0 2024-03-30T04:48:39.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.650+0000 7f390a400700 1 -- 172.21.15.123:0/3610260066 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39040fe550 con 0x7f39041037a0 2024-03-30T04:48:39.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.651+0000 7f38fb400700 1 -- 172.21.15.123:0/3610260066 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f38f0009680 con 0x7f39041037a0 2024-03-30T04:48:39.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.652+0000 7f38fb400700 1 --2- 172.21.15.123:0/3610260066 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f38e806cc30 0x7f38e806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:48:39.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.652+0000 7f38fb400700 1 -- 172.21.15.123:0/3610260066 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f38f008bc90 con 0x7f39041037a0 2024-03-30T04:48:39.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.652+0000 7f3903400700 1 --2- 172.21.15.123:0/3610260066 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f38e806cc30 0x7f38e806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:48:39.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.652+0000 7f3903400700 1 --2- 172.21.15.123:0/3610260066 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f38e806cc30 0x7f38e806f0e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f38f4009fd0 tx=0x7f38f4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:48:39.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.656+0000 7f38fb400700 1 -- 172.21.15.123:0/3610260066 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f38f0059760 con 0x7f39041037a0 2024-03-30T04:48:39.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.950+0000 7f390a400700 1 -- 172.21.15.123:0/3610260066 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f390404bed0 con 0x7f39041037a0 2024-03-30T04:48:39.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.950+0000 7f38fb400700 1 -- 172.21.15.123:0/3610260066 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f38f00592f0 con 0x7f39041037a0 2024-03-30T04:48:39.949 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T04:48:39.949 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T04:48:39.949 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T04:48:39.949 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T04:48:39.949 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T04:48:39.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.953+0000 7f390a400700 1 -- 172.21.15.123:0/3610260066 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f38e806cc30 msgr2=0x7f38e806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:39.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.953+0000 7f390a400700 1 --2- 172.21.15.123:0/3610260066 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f38e806cc30 0x7f38e806f0e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f38f4009fd0 tx=0x7f38f4009e30 comp rx=0 tx=0).stop 2024-03-30T04:48:39.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.953+0000 7f390a400700 1 -- 172.21.15.123:0/3610260066 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f39041037a0 msgr2=0x7f3904116230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:48:39.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.953+0000 7f390a400700 1 --2- 172.21.15.123:0/3610260066 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f39041037a0 0x7f3904116230 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f38f0002a50 tx=0x7f38f0004600 comp rx=0 tx=0).stop 2024-03-30T04:48:39.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.953+0000 7f390a400700 1 -- 172.21.15.123:0/3610260066 shutdown_connections 2024-03-30T04:48:39.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.953+0000 7f390a400700 1 --2- 172.21.15.123:0/3610260066 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f38e806cc30 0x7f38e806f0e0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:39.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.953+0000 7f390a400700 1 --2- 172.21.15.123:0/3610260066 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3904102e00 0x7f3904115cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:39.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.953+0000 7f390a400700 1 --2- 172.21.15.123:0/3610260066 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f39041037a0 0x7f3904116230 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:48:39.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.953+0000 7f390a400700 1 -- 172.21.15.123:0/3610260066 >> 172.21.15.123:0/3610260066 conn(0x7f39040f89b0 msgr2=0x7f39040706d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:48:39.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.953+0000 7f390a400700 1 -- 172.21.15.123:0/3610260066 shutdown_connections 2024-03-30T04:48:39.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:48:39.953+0000 7f390a400700 1 -- 172.21.15.123:0/3610260066 wait complete. 2024-03-30T04:48:39.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:48:39 smithi123 ceph-mon[139369]: from='client.34998 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:48:39.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:48:39 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/458393940' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:48:39.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:48:39 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3296870361' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:48:40.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:48:39 smithi135 ceph-mon[106011]: from='client.34998 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:48:40.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:48:39 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/458393940' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:48:40.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:48:39 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3296870361' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:48:40.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:48:40 smithi123 ceph-mon[139369]: from='client.35010 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:48:40.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:48:40 smithi123 ceph-mon[139369]: pgmap v524: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:40.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:48:40 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3610260066' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:48:41.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:48:40 smithi135 ceph-mon[106011]: from='client.35010 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:48:41.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:48:40 smithi135 ceph-mon[106011]: pgmap v524: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:41.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:48:40 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3610260066' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:48:42.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:48:42 smithi123 ceph-mon[139369]: pgmap v525: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:43.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:48:42 smithi135 ceph-mon[106011]: pgmap v525: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:44.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:48:44 smithi123 ceph-mon[139369]: pgmap v526: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:44.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:48:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:48:45.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:48:44 smithi135 ceph-mon[106011]: pgmap v526: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:45.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:48:44 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:48:46.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:48:46 smithi123 ceph-mon[139369]: pgmap v527: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:47.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:48:46 smithi135 ceph-mon[106011]: pgmap v527: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:48.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:48:48 smithi123 ceph-mon[139369]: pgmap v528: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:49.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:48:48 smithi135 ceph-mon[106011]: pgmap v528: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:50.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:48:50 smithi135 ceph-mon[106011]: pgmap v529: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:50.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:48:50 smithi123 ceph-mon[139369]: pgmap v529: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:52.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:48:52 smithi123 ceph-mon[139369]: pgmap v530: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:53.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:48:52 smithi135 ceph-mon[106011]: pgmap v530: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:54.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:48:54 smithi123 ceph-mon[139369]: pgmap v531: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:55.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:48:54 smithi135 ceph-mon[106011]: pgmap v531: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:56.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:48:56 smithi123 ceph-mon[139369]: pgmap v532: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:57.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:48:56 smithi135 ceph-mon[106011]: pgmap v532: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:58.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:48:58 smithi123 ceph-mon[139369]: pgmap v533: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:59.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:48:58 smithi135 ceph-mon[106011]: pgmap v533: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:48:59.795 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:48:59 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:48:59.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:48:59 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:49:00.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:49:00 smithi123 ceph-mon[139369]: pgmap v534: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:01.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:49:00 smithi135 ceph-mon[106011]: pgmap v534: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:02.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:49:02 smithi123 ceph-mon[139369]: pgmap v535: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:03.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:49:02 smithi135 ceph-mon[106011]: pgmap v535: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:04.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:49:04 smithi123 ceph-mon[139369]: pgmap v536: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:05.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:49:04 smithi135 ceph-mon[106011]: pgmap v536: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:06.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:49:06 smithi123 ceph-mon[139369]: pgmap v537: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:07.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:49:06 smithi135 ceph-mon[106011]: pgmap v537: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:08.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:49:08 smithi123 ceph-mon[139369]: pgmap v538: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:09.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:49:08 smithi135 ceph-mon[106011]: pgmap v538: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:10.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.100+0000 7fc7cf200700 1 -- 172.21.15.123:0/3601356557 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7c8071f90 msgr2=0x7fc7c8070360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:10.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.100+0000 7fc7cf200700 1 --2- 172.21.15.123:0/3601356557 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7c8071f90 0x7fc7c8070360 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fc7b8004970 tx=0x7fc7b8009e30 comp rx=0 tx=0).stop 2024-03-30T04:49:10.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.100+0000 7fc7cf200700 1 -- 172.21.15.123:0/3601356557 shutdown_connections 2024-03-30T04:49:10.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.100+0000 7fc7cf200700 1 --2- 172.21.15.123:0/3601356557 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7c80708a0 0x7fc7c8070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:10.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.100+0000 7fc7cf200700 1 --2- 172.21.15.123:0/3601356557 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7c8071f90 0x7fc7c8070360 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:10.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.100+0000 7fc7cf200700 1 -- 172.21.15.123:0/3601356557 >> 172.21.15.123:0/3601356557 conn(0x7fc7c80f89b0 msgr2=0x7fc7c80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:49:10.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.101+0000 7fc7cf200700 1 -- 172.21.15.123:0/3601356557 shutdown_connections 2024-03-30T04:49:10.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.101+0000 7fc7cf200700 1 -- 172.21.15.123:0/3601356557 wait complete. 2024-03-30T04:49:10.102 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.101+0000 7fc7cf200700 1 Processor -- start 2024-03-30T04:49:10.102 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.102+0000 7fc7cf200700 1 -- start start 2024-03-30T04:49:10.102 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.102+0000 7fc7cf200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7c80708a0 0x7fc7c811a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:49:10.102 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.102+0000 7fc7cf200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7c8071f90 0x7fc7c811a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:49:10.102 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.102+0000 7fc7cf200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c811ad50 con 0x7fc7c80708a0 2024-03-30T04:49:10.102 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.102+0000 7fc7cf200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c8114090 con 0x7fc7c8071f90 2024-03-30T04:49:10.102 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.102+0000 7fc7c7e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7c80708a0 0x7fc7c811a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:49:10.102 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.102+0000 7fc7c7e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7c80708a0 0x7fc7c811a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44498/0 (socket says 172.21.15.123:44498) 2024-03-30T04:49:10.102 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.102+0000 7fc7c7e00700 1 -- 172.21.15.123:0/2795901939 learned_addr learned my addr 172.21.15.123:0/2795901939 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:49:10.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.102+0000 7fc7c7400700 1 --2- 172.21.15.123:0/2795901939 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7c8071f90 0x7fc7c811a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:49:10.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.103+0000 7fc7c7e00700 1 -- 172.21.15.123:0/2795901939 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7c8071f90 msgr2=0x7fc7c811a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:10.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.103+0000 7fc7c7e00700 1 --2- 172.21.15.123:0/2795901939 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7c8071f90 0x7fc7c811a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:10.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.103+0000 7fc7c7e00700 1 -- 172.21.15.123:0/2795901939 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7b8005040 con 0x7fc7c80708a0 2024-03-30T04:49:10.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.103+0000 7fc7c7e00700 1 --2- 172.21.15.123:0/2795901939 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7c80708a0 0x7fc7c811a130 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fc7b8009360 tx=0x7fc7b8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:49:10.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.103+0000 7fc7bfe00700 1 -- 172.21.15.123:0/2795901939 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc7b800e040 con 0x7fc7c80708a0 2024-03-30T04:49:10.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.103+0000 7fc7bfe00700 1 -- 172.21.15.123:0/2795901939 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc7b8004290 con 0x7fc7c80708a0 2024-03-30T04:49:10.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.103+0000 7fc7cf200700 1 -- 172.21.15.123:0/2795901939 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7c81142b0 con 0x7fc7c80708a0 2024-03-30T04:49:10.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.103+0000 7fc7bfe00700 1 -- 172.21.15.123:0/2795901939 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc7b8013e70 con 0x7fc7c80708a0 2024-03-30T04:49:10.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.103+0000 7fc7cf200700 1 -- 172.21.15.123:0/2795901939 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7c81147f0 con 0x7fc7c80708a0 2024-03-30T04:49:10.104 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.104+0000 7fc7cf200700 1 -- 172.21.15.123:0/2795901939 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7c8071aa0 con 0x7fc7c80708a0 2024-03-30T04:49:10.104 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.105+0000 7fc7bfe00700 1 -- 172.21.15.123:0/2795901939 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fc7b8013620 con 0x7fc7c80708a0 2024-03-30T04:49:10.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.107+0000 7fc7bfe00700 1 --2- 172.21.15.123:0/2795901939 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc7ac06cc30 0x7fc7ac06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:49:10.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.107+0000 7fc7bfe00700 1 -- 172.21.15.123:0/2795901939 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc7b808d300 con 0x7fc7c80708a0 2024-03-30T04:49:10.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.107+0000 7fc7c7400700 1 --2- 172.21.15.123:0/2795901939 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc7ac06cc30 0x7fc7ac06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:49:10.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.107+0000 7fc7c7400700 1 --2- 172.21.15.123:0/2795901939 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc7ac06cc30 0x7fc7ac06f0e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fc7c81159d0 tx=0x7fc7b40091a0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:49:10.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.111+0000 7fc7bfe00700 1 -- 172.21.15.123:0/2795901939 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc7b805add0 con 0x7fc7c80708a0 2024-03-30T04:49:10.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.335+0000 7fc7cf200700 1 -- 172.21.15.123:0/2795901939 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc7c804bf70 con 0x7fc7ac06cc30 2024-03-30T04:49:10.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.336+0000 7fc7bfe00700 1 -- 172.21.15.123:0/2795901939 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc7b400e6e0 con 0x7fc7ac06cc30 2024-03-30T04:49:10.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.338+0000 7fc7cf200700 1 -- 172.21.15.123:0/2795901939 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc7ac06cc30 msgr2=0x7fc7ac06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:10.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.338+0000 7fc7cf200700 1 --2- 172.21.15.123:0/2795901939 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc7ac06cc30 0x7fc7ac06f0e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fc7c81159d0 tx=0x7fc7b40091a0 comp rx=0 tx=0).stop 2024-03-30T04:49:10.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.338+0000 7fc7cf200700 1 -- 172.21.15.123:0/2795901939 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7c80708a0 msgr2=0x7fc7c811a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:10.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.338+0000 7fc7cf200700 1 --2- 172.21.15.123:0/2795901939 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7c80708a0 0x7fc7c811a130 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fc7b8009360 tx=0x7fc7b8009e30 comp rx=0 tx=0).stop 2024-03-30T04:49:10.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.338+0000 7fc7cf200700 1 -- 172.21.15.123:0/2795901939 shutdown_connections 2024-03-30T04:49:10.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.338+0000 7fc7cf200700 1 --2- 172.21.15.123:0/2795901939 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc7ac06cc30 0x7fc7ac06f0e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:10.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.338+0000 7fc7cf200700 1 --2- 172.21.15.123:0/2795901939 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7c80708a0 0x7fc7c811a130 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:10.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.338+0000 7fc7cf200700 1 --2- 172.21.15.123:0/2795901939 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7c8071f90 0x7fc7c811a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:10.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.338+0000 7fc7cf200700 1 -- 172.21.15.123:0/2795901939 >> 172.21.15.123:0/2795901939 conn(0x7fc7c80f89b0 msgr2=0x7fc7c80f9b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:49:10.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.339+0000 7fc7cf200700 1 -- 172.21.15.123:0/2795901939 shutdown_connections 2024-03-30T04:49:10.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.339+0000 7fc7cf200700 1 -- 172.21.15.123:0/2795901939 wait complete. 2024-03-30T04:49:10.352 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:49:10.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.473+0000 7f16b3e00700 1 -- 172.21.15.123:0/3151956984 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16ac1037a0 msgr2=0x7f16ac0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:10.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.473+0000 7f16b3e00700 1 --2- 172.21.15.123:0/3151956984 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16ac1037a0 0x7f16ac0fce10 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f1698004940 tx=0x7f1698009e30 comp rx=0 tx=0).stop 2024-03-30T04:49:10.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.473+0000 7f16b3e00700 1 -- 172.21.15.123:0/3151956984 shutdown_connections 2024-03-30T04:49:10.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.473+0000 7f16b3e00700 1 --2- 172.21.15.123:0/3151956984 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16ac1037a0 0x7f16ac0fce10 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:10.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.473+0000 7f16b3e00700 1 --2- 172.21.15.123:0/3151956984 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f16ac102e00 0x7f16ac1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:10.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.473+0000 7f16b3e00700 1 -- 172.21.15.123:0/3151956984 >> 172.21.15.123:0/3151956984 conn(0x7f16ac0f89b0 msgr2=0x7f16ac0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:49:10.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.473+0000 7f16b3e00700 1 -- 172.21.15.123:0/3151956984 shutdown_connections 2024-03-30T04:49:10.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.473+0000 7f16b3e00700 1 -- 172.21.15.123:0/3151956984 wait complete. 2024-03-30T04:49:10.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.474+0000 7f16b3e00700 1 Processor -- start 2024-03-30T04:49:10.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.474+0000 7f16b3e00700 1 -- start start 2024-03-30T04:49:10.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.474+0000 7f16b3e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f16ac102e00 0x7f16ac115da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:49:10.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.474+0000 7f16b3e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16ac1037a0 0x7f16ac1162e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:49:10.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.474+0000 7f16b3e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16ac116930 con 0x7f16ac1037a0 2024-03-30T04:49:10.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.474+0000 7f16b3e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16ac10fd00 con 0x7f16ac102e00 2024-03-30T04:49:10.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.475+0000 7f16b0a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16ac1037a0 0x7f16ac1162e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:49:10.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.475+0000 7f16b0a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16ac1037a0 0x7f16ac1162e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44514/0 (socket says 172.21.15.123:44514) 2024-03-30T04:49:10.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.475+0000 7f16b0a00700 1 -- 172.21.15.123:0/256320239 learned_addr learned my addr 172.21.15.123:0/256320239 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:49:10.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.475+0000 7f16b1400700 1 --2- 172.21.15.123:0/256320239 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f16ac102e00 0x7f16ac115da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:49:10.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.475+0000 7f16b0a00700 1 -- 172.21.15.123:0/256320239 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f16ac102e00 msgr2=0x7f16ac115da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:10.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.475+0000 7f16b0a00700 1 --2- 172.21.15.123:0/256320239 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f16ac102e00 0x7f16ac115da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:10.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.475+0000 7f16b0a00700 1 -- 172.21.15.123:0/256320239 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1698005040 con 0x7f16ac1037a0 2024-03-30T04:49:10.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.475+0000 7f16b1400700 1 --2- 172.21.15.123:0/256320239 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f16ac102e00 0x7f16ac115da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:49:10.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.475+0000 7f16b0a00700 1 --2- 172.21.15.123:0/256320239 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16ac1037a0 0x7f16ac1162e0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f1698009390 tx=0x7f1698009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:49:10.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.476+0000 7f16a9600700 1 -- 172.21.15.123:0/256320239 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f169800e040 con 0x7f16ac1037a0 2024-03-30T04:49:10.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.476+0000 7f16a9600700 1 -- 172.21.15.123:0/256320239 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1698013420 con 0x7f16ac1037a0 2024-03-30T04:49:10.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.476+0000 7f16b3e00700 1 -- 172.21.15.123:0/256320239 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16ac10ff80 con 0x7f16ac1037a0 2024-03-30T04:49:10.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.476+0000 7f16a9600700 1 -- 172.21.15.123:0/256320239 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f169801c630 con 0x7f16ac1037a0 2024-03-30T04:49:10.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.476+0000 7f16b3e00700 1 -- 172.21.15.123:0/256320239 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16ac1104c0 con 0x7f16ac1037a0 2024-03-30T04:49:10.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.477+0000 7f16b3e00700 1 -- 172.21.15.123:0/256320239 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16ac0fe550 con 0x7f16ac1037a0 2024-03-30T04:49:10.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.478+0000 7f16a9600700 1 -- 172.21.15.123:0/256320239 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f1698013a00 con 0x7f16ac1037a0 2024-03-30T04:49:10.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.479+0000 7f16a9600700 1 --2- 172.21.15.123:0/256320239 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f169406cc30 0x7f169406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:49:10.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.479+0000 7f16a9600700 1 -- 172.21.15.123:0/256320239 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f169808d2d0 con 0x7f16ac1037a0 2024-03-30T04:49:10.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.479+0000 7f16b1400700 1 --2- 172.21.15.123:0/256320239 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f169406cc30 0x7f169406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:49:10.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.480+0000 7f16b1400700 1 --2- 172.21.15.123:0/256320239 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f169406cc30 0x7f169406f0e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f16a0009fd0 tx=0x7f16a0009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:49:10.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.483+0000 7f16a9600700 1 -- 172.21.15.123:0/256320239 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f169805ada0 con 0x7f16ac1037a0 2024-03-30T04:49:10.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.704+0000 7f16b3e00700 1 -- 172.21.15.123:0/256320239 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f16ac109e10 con 0x7f169406cc30 2024-03-30T04:49:10.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.705+0000 7f16a9600700 1 -- 172.21.15.123:0/256320239 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f16ac109e10 con 0x7f169406cc30 2024-03-30T04:49:10.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.708+0000 7f16b3e00700 1 -- 172.21.15.123:0/256320239 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f169406cc30 msgr2=0x7f169406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:10.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.708+0000 7f16b3e00700 1 --2- 172.21.15.123:0/256320239 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f169406cc30 0x7f169406f0e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f16a0009fd0 tx=0x7f16a0009ee0 comp rx=0 tx=0).stop 2024-03-30T04:49:10.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.708+0000 7f16b3e00700 1 -- 172.21.15.123:0/256320239 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16ac1037a0 msgr2=0x7f16ac1162e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:10.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.708+0000 7f16b3e00700 1 --2- 172.21.15.123:0/256320239 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16ac1037a0 0x7f16ac1162e0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f1698009390 tx=0x7f1698009e30 comp rx=0 tx=0).stop 2024-03-30T04:49:10.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.708+0000 7f16b3e00700 1 -- 172.21.15.123:0/256320239 shutdown_connections 2024-03-30T04:49:10.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.708+0000 7f16b3e00700 1 --2- 172.21.15.123:0/256320239 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f169406cc30 0x7f169406f0e0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:10.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.708+0000 7f16b3e00700 1 --2- 172.21.15.123:0/256320239 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f16ac102e00 0x7f16ac115da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:10.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.708+0000 7f16b3e00700 1 --2- 172.21.15.123:0/256320239 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16ac1037a0 0x7f16ac1162e0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:10.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.708+0000 7f16b3e00700 1 -- 172.21.15.123:0/256320239 >> 172.21.15.123:0/256320239 conn(0x7f16ac0f89b0 msgr2=0x7f16ac0f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:49:10.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.708+0000 7f16b3e00700 1 -- 172.21.15.123:0/256320239 shutdown_connections 2024-03-30T04:49:10.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.708+0000 7f16b3e00700 1 -- 172.21.15.123:0/256320239 wait complete. 2024-03-30T04:49:10.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.843+0000 7f5843800700 1 -- 172.21.15.123:0/602349318 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f583c1037a0 msgr2=0x7f583c0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:10.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.843+0000 7f5843800700 1 --2- 172.21.15.123:0/602349318 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f583c1037a0 0x7f583c0fce10 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f5828004940 tx=0x7f5828009e30 comp rx=0 tx=0).stop 2024-03-30T04:49:10.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.843+0000 7f5843800700 1 -- 172.21.15.123:0/602349318 shutdown_connections 2024-03-30T04:49:10.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.843+0000 7f5843800700 1 --2- 172.21.15.123:0/602349318 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f583c1037a0 0x7f583c0fce10 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:10.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.843+0000 7f5843800700 1 --2- 172.21.15.123:0/602349318 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f583c102e00 0x7f583c1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:10.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.843+0000 7f5843800700 1 -- 172.21.15.123:0/602349318 >> 172.21.15.123:0/602349318 conn(0x7f583c0f89b0 msgr2=0x7f583c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:49:10.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.843+0000 7f5843800700 1 -- 172.21.15.123:0/602349318 shutdown_connections 2024-03-30T04:49:10.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.843+0000 7f5843800700 1 -- 172.21.15.123:0/602349318 wait complete. 2024-03-30T04:49:10.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.844+0000 7f5843800700 1 Processor -- start 2024-03-30T04:49:10.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.844+0000 7f5843800700 1 -- start start 2024-03-30T04:49:10.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.844+0000 7f5843800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f583c102e00 0x7f583c115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:49:10.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.844+0000 7f5843800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f583c1037a0 0x7f583c116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:49:10.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.844+0000 7f5843800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f583c116880 con 0x7f583c102e00 2024-03-30T04:49:10.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.844+0000 7f5843800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f583c10fc50 con 0x7f583c1037a0 2024-03-30T04:49:10.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.844+0000 7f5840e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f583c102e00 0x7f583c115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:49:10.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.845+0000 7f5840e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f583c102e00 0x7f583c115cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44536/0 (socket says 172.21.15.123:44536) 2024-03-30T04:49:10.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.845+0000 7f5840e00700 1 -- 172.21.15.123:0/672289257 learned_addr learned my addr 172.21.15.123:0/672289257 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:49:10.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.845+0000 7f583be00700 1 --2- 172.21.15.123:0/672289257 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f583c1037a0 0x7f583c116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:49:10.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.845+0000 7f5840e00700 1 -- 172.21.15.123:0/672289257 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f583c1037a0 msgr2=0x7f583c116230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:10.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.845+0000 7f5840e00700 1 --2- 172.21.15.123:0/672289257 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f583c1037a0 0x7f583c116230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:10.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.845+0000 7f5840e00700 1 -- 172.21.15.123:0/672289257 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5828005040 con 0x7f583c102e00 2024-03-30T04:49:10.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.845+0000 7f5840e00700 1 --2- 172.21.15.123:0/672289257 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f583c102e00 0x7f583c115cf0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f5830002a50 tx=0x7f58300046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:49:10.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.845+0000 7f5838c00700 1 -- 172.21.15.123:0/672289257 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5830007cc0 con 0x7f583c102e00 2024-03-30T04:49:10.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.845+0000 7f5838c00700 1 -- 172.21.15.123:0/672289257 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f583000f040 con 0x7f583c102e00 2024-03-30T04:49:10.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.845+0000 7f5843800700 1 -- 172.21.15.123:0/672289257 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f583c10ff30 con 0x7f583c102e00 2024-03-30T04:49:10.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.845+0000 7f5838c00700 1 -- 172.21.15.123:0/672289257 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f583000d9c0 con 0x7f583c102e00 2024-03-30T04:49:10.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.846+0000 7f5843800700 1 -- 172.21.15.123:0/672289257 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f583c1104f0 con 0x7f583c102e00 2024-03-30T04:49:10.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.847+0000 7f5843800700 1 -- 172.21.15.123:0/672289257 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f583c0fe550 con 0x7f583c102e00 2024-03-30T04:49:10.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.847+0000 7f5838c00700 1 -- 172.21.15.123:0/672289257 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f583000db20 con 0x7f583c102e00 2024-03-30T04:49:10.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.848+0000 7f5838c00700 1 --2- 172.21.15.123:0/672289257 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f582406cc30 0x7f582406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:49:10.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.848+0000 7f5838c00700 1 -- 172.21.15.123:0/672289257 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f583008bd20 con 0x7f583c102e00 2024-03-30T04:49:10.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.849+0000 7f583be00700 1 --2- 172.21.15.123:0/672289257 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f582406cc30 0x7f582406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:49:10.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.849+0000 7f583be00700 1 --2- 172.21.15.123:0/672289257 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f582406cc30 0x7f582406f0e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f583c111590 tx=0x7f5828009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:49:10.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:49:10 smithi135 ceph-mon[106011]: pgmap v539: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:10.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:10.853+0000 7f5838c00700 1 -- 172.21.15.123:0/672289257 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5830059770 con 0x7f583c102e00 2024-03-30T04:49:10.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:49:10 smithi123 ceph-mon[139369]: pgmap v539: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:11.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.068+0000 7f5843800700 1 -- 172.21.15.123:0/672289257 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f583c04bf40 con 0x7f582406cc30 2024-03-30T04:49:11.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.077+0000 7f5838c00700 1 -- 172.21.15.123:0/672289257 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f583c04bf40 con 0x7f582406cc30 2024-03-30T04:49:11.076 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:49:11.076 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (18m) 11m ago 30m 15.0M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:49:11.076 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (15m) 11m ago 31m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T04:49:11.076 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (15m) 10m ago 29m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T04:49:11.076 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (18m) 11m ago 30m 74.0M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T04:49:11.076 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (11m) 11m ago 24m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T04:49:11.076 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (11m) 11m ago 24m 10.9M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T04:49:11.076 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (24m) 10m ago 24m 10.7M - 17.2.7-797-g5c024335 f77085f6aab7 0dc41326818c 2024-03-30T04:49:11.076 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (11m) 10m ago 24m 9747k - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T04:49:11.076 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (20m) 11m ago 32m 506M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:49:11.076 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (19m) 10m ago 29m 450M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:49:11.076 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (16m) 11m ago 32m 53.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T04:49:11.077 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (16m) 10m ago 29m 42.6M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T04:49:11.077 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (19m) 11m ago 31m 8052k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:49:11.077 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (19m) 10m ago 29m 7880k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:49:11.077 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (15m) 11m ago 28m 104M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T04:49:11.077 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (14m) 11m ago 28m 93.4M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T04:49:11.077 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (13m) 11m ago 27m 86.3M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T04:49:11.077 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (13m) 10m ago 27m 97.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T04:49:11.077 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (12m) 10m ago 26m 97.9M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T04:49:11.077 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (12m) 10m ago 26m 85.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T04:49:11.077 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (18m) 11m ago 30m 53.7M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:49:11.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.080+0000 7f5843800700 1 -- 172.21.15.123:0/672289257 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f582406cc30 msgr2=0x7f582406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:11.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.080+0000 7f5843800700 1 --2- 172.21.15.123:0/672289257 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f582406cc30 0x7f582406f0e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f583c111590 tx=0x7f5828009e30 comp rx=0 tx=0).stop 2024-03-30T04:49:11.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.080+0000 7f5843800700 1 -- 172.21.15.123:0/672289257 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f583c102e00 msgr2=0x7f583c115cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:11.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.080+0000 7f5843800700 1 --2- 172.21.15.123:0/672289257 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f583c102e00 0x7f583c115cf0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f5830002a50 tx=0x7f58300046b0 comp rx=0 tx=0).stop 2024-03-30T04:49:11.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.080+0000 7f5843800700 1 -- 172.21.15.123:0/672289257 shutdown_connections 2024-03-30T04:49:11.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.080+0000 7f5843800700 1 --2- 172.21.15.123:0/672289257 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f582406cc30 0x7f582406f0e0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:11.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.080+0000 7f5843800700 1 --2- 172.21.15.123:0/672289257 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f583c102e00 0x7f583c115cf0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:11.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.080+0000 7f5843800700 1 --2- 172.21.15.123:0/672289257 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f583c1037a0 0x7f583c116230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:11.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.080+0000 7f5843800700 1 -- 172.21.15.123:0/672289257 >> 172.21.15.123:0/672289257 conn(0x7f583c0f89b0 msgr2=0x7f583c0f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:49:11.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.080+0000 7f5843800700 1 -- 172.21.15.123:0/672289257 shutdown_connections 2024-03-30T04:49:11.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.080+0000 7f5843800700 1 -- 172.21.15.123:0/672289257 wait complete. 2024-03-30T04:49:11.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.213+0000 7efc78e00700 1 -- 172.21.15.123:0/1495029292 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc741037a0 msgr2=0x7efc740fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:11.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.213+0000 7efc78e00700 1 --2- 172.21.15.123:0/1495029292 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc741037a0 0x7efc740fce10 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7efc60004940 tx=0x7efc60009e30 comp rx=0 tx=0).stop 2024-03-30T04:49:11.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.213+0000 7efc78e00700 1 -- 172.21.15.123:0/1495029292 shutdown_connections 2024-03-30T04:49:11.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.213+0000 7efc78e00700 1 --2- 172.21.15.123:0/1495029292 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc741037a0 0x7efc740fce10 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:11.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.213+0000 7efc78e00700 1 --2- 172.21.15.123:0/1495029292 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc74102e00 0x7efc741031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:11.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.213+0000 7efc78e00700 1 -- 172.21.15.123:0/1495029292 >> 172.21.15.123:0/1495029292 conn(0x7efc740f89b0 msgr2=0x7efc740fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:49:11.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.213+0000 7efc78e00700 1 -- 172.21.15.123:0/1495029292 shutdown_connections 2024-03-30T04:49:11.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.213+0000 7efc78e00700 1 -- 172.21.15.123:0/1495029292 wait complete. 2024-03-30T04:49:11.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.214+0000 7efc78e00700 1 Processor -- start 2024-03-30T04:49:11.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.214+0000 7efc78e00700 1 -- start start 2024-03-30T04:49:11.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.214+0000 7efc78e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc74102e00 0x7efc740702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:49:11.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.214+0000 7efc78e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc741037a0 0x7efc74070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:49:11.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.214+0000 7efc78e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc74070d90 con 0x7efc741037a0 2024-03-30T04:49:11.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.214+0000 7efc78e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc74070ed0 con 0x7efc74102e00 2024-03-30T04:49:11.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.214+0000 7efc71400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc741037a0 0x7efc74070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:49:11.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.214+0000 7efc71400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc741037a0 0x7efc74070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44546/0 (socket says 172.21.15.123:44546) 2024-03-30T04:49:11.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.214+0000 7efc71400700 1 -- 172.21.15.123:0/1204768112 learned_addr learned my addr 172.21.15.123:0/1204768112 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:49:11.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.214+0000 7efc71e00700 1 --2- 172.21.15.123:0/1204768112 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc74102e00 0x7efc740702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:49:11.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.215+0000 7efc71400700 1 -- 172.21.15.123:0/1204768112 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc74102e00 msgr2=0x7efc740702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:11.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.215+0000 7efc71400700 1 --2- 172.21.15.123:0/1204768112 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc74102e00 0x7efc740702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:11.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.215+0000 7efc71400700 1 -- 172.21.15.123:0/1204768112 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc60005040 con 0x7efc741037a0 2024-03-30T04:49:11.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.215+0000 7efc71400700 1 --2- 172.21.15.123:0/1204768112 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc741037a0 0x7efc74070800 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7efc60009390 tx=0x7efc60009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:49:11.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.215+0000 7efc6a000700 1 -- 172.21.15.123:0/1204768112 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc6000e040 con 0x7efc741037a0 2024-03-30T04:49:11.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.215+0000 7efc78e00700 1 -- 172.21.15.123:0/1204768112 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc74071120 con 0x7efc741037a0 2024-03-30T04:49:11.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.215+0000 7efc6a000700 1 -- 172.21.15.123:0/1204768112 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efc60013420 con 0x7efc741037a0 2024-03-30T04:49:11.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.215+0000 7efc6a000700 1 -- 172.21.15.123:0/1204768112 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc6001c630 con 0x7efc741037a0 2024-03-30T04:49:11.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.216+0000 7efc78e00700 1 -- 172.21.15.123:0/1204768112 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc7411a230 con 0x7efc741037a0 2024-03-30T04:49:11.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.217+0000 7efc78e00700 1 -- 172.21.15.123:0/1204768112 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc740fe550 con 0x7efc741037a0 2024-03-30T04:49:11.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.218+0000 7efc6a000700 1 -- 172.21.15.123:0/1204768112 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7efc60013a00 con 0x7efc741037a0 2024-03-30T04:49:11.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.218+0000 7efc6a000700 1 --2- 172.21.15.123:0/1204768112 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7efc5806cc30 0x7efc5806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:49:11.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.218+0000 7efc6a000700 1 -- 172.21.15.123:0/1204768112 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7efc6008d1b0 con 0x7efc741037a0 2024-03-30T04:49:11.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.219+0000 7efc71e00700 1 --2- 172.21.15.123:0/1204768112 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7efc5806cc30 0x7efc5806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:49:11.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.219+0000 7efc71e00700 1 --2- 172.21.15.123:0/1204768112 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7efc5806cc30 0x7efc5806f0e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7efc64004f40 tx=0x7efc6400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:49:11.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.223+0000 7efc6a000700 1 -- 172.21.15.123:0/1204768112 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7efc60057170 con 0x7efc741037a0 2024-03-30T04:49:11.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.513+0000 7efc78e00700 1 -- 172.21.15.123:0/1204768112 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7efc7404bed0 con 0x7efc741037a0 2024-03-30T04:49:11.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.513+0000 7efc6a000700 1 -- 172.21.15.123:0/1204768112 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7efc6005a790 con 0x7efc741037a0 2024-03-30T04:49:11.512 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:49:11.512 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:49:11.512 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:49:11.512 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:49:11.512 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:49:11.512 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:49:11.512 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:49:11.512 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:49:11.512 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T04:49:11.513 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:49:11.513 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:49:11.513 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:49:11.513 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 3 2024-03-30T04:49:11.513 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:49:11.513 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:49:11.513 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:49:11.513 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 13 2024-03-30T04:49:11.513 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:49:11.513 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:49:11.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.516+0000 7efc78e00700 1 -- 172.21.15.123:0/1204768112 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7efc5806cc30 msgr2=0x7efc5806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:11.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.516+0000 7efc78e00700 1 --2- 172.21.15.123:0/1204768112 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7efc5806cc30 0x7efc5806f0e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7efc64004f40 tx=0x7efc6400b040 comp rx=0 tx=0).stop 2024-03-30T04:49:11.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.516+0000 7efc78e00700 1 -- 172.21.15.123:0/1204768112 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc741037a0 msgr2=0x7efc74070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:11.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.516+0000 7efc78e00700 1 --2- 172.21.15.123:0/1204768112 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc741037a0 0x7efc74070800 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7efc60009390 tx=0x7efc60009e30 comp rx=0 tx=0).stop 2024-03-30T04:49:11.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.516+0000 7efc78e00700 1 -- 172.21.15.123:0/1204768112 shutdown_connections 2024-03-30T04:49:11.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.516+0000 7efc78e00700 1 --2- 172.21.15.123:0/1204768112 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7efc5806cc30 0x7efc5806f0e0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:11.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.516+0000 7efc78e00700 1 --2- 172.21.15.123:0/1204768112 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc74102e00 0x7efc740702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:11.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.516+0000 7efc78e00700 1 --2- 172.21.15.123:0/1204768112 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc741037a0 0x7efc74070800 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:11.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.516+0000 7efc78e00700 1 -- 172.21.15.123:0/1204768112 >> 172.21.15.123:0/1204768112 conn(0x7efc740f89b0 msgr2=0x7efc740f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:49:11.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.516+0000 7efc78e00700 1 -- 172.21.15.123:0/1204768112 shutdown_connections 2024-03-30T04:49:11.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.516+0000 7efc78e00700 1 -- 172.21.15.123:0/1204768112 wait complete. 2024-03-30T04:49:11.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.649+0000 7f9b66c00700 1 -- 172.21.15.123:0/2913716034 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b601037a0 msgr2=0x7f9b600fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:11.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.649+0000 7f9b66c00700 1 --2- 172.21.15.123:0/2913716034 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b601037a0 0x7f9b600fce10 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c004940 tx=0x7f9b4c009e30 comp rx=0 tx=0).stop 2024-03-30T04:49:11.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.649+0000 7f9b66c00700 1 -- 172.21.15.123:0/2913716034 shutdown_connections 2024-03-30T04:49:11.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.649+0000 7f9b66c00700 1 --2- 172.21.15.123:0/2913716034 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b601037a0 0x7f9b600fce10 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:11.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.649+0000 7f9b66c00700 1 --2- 172.21.15.123:0/2913716034 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9b60102e00 0x7f9b601031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:11.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.649+0000 7f9b66c00700 1 -- 172.21.15.123:0/2913716034 >> 172.21.15.123:0/2913716034 conn(0x7f9b600f89b0 msgr2=0x7f9b600fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:49:11.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.650+0000 7f9b66c00700 1 -- 172.21.15.123:0/2913716034 shutdown_connections 2024-03-30T04:49:11.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.650+0000 7f9b66c00700 1 -- 172.21.15.123:0/2913716034 wait complete. 2024-03-30T04:49:11.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.650+0000 7f9b66c00700 1 Processor -- start 2024-03-30T04:49:11.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.650+0000 7f9b66c00700 1 -- start start 2024-03-30T04:49:11.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.651+0000 7f9b66c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b60102e00 0x7f9b60115d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:49:11.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.651+0000 7f9b66c00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9b601037a0 0x7f9b60116240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:49:11.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.651+0000 7f9b66c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b60116890 con 0x7f9b60102e00 2024-03-30T04:49:11.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.651+0000 7f9b66c00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b6010fc60 con 0x7f9b601037a0 2024-03-30T04:49:11.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.651+0000 7f9b5fe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b60102e00 0x7f9b60115d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:49:11.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.651+0000 7f9b5fe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b60102e00 0x7f9b60115d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44576/0 (socket says 172.21.15.123:44576) 2024-03-30T04:49:11.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.651+0000 7f9b5fe00700 1 -- 172.21.15.123:0/2885672230 learned_addr learned my addr 172.21.15.123:0/2885672230 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:49:11.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.651+0000 7f9b5f400700 1 --2- 172.21.15.123:0/2885672230 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9b601037a0 0x7f9b60116240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:49:11.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.651+0000 7f9b5fe00700 1 -- 172.21.15.123:0/2885672230 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9b601037a0 msgr2=0x7f9b60116240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:11.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.651+0000 7f9b5fe00700 1 --2- 172.21.15.123:0/2885672230 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9b601037a0 0x7f9b60116240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:11.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.652+0000 7f9b5fe00700 1 -- 172.21.15.123:0/2885672230 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b4c005040 con 0x7f9b60102e00 2024-03-30T04:49:11.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.652+0000 7f9b5fe00700 1 --2- 172.21.15.123:0/2885672230 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b60102e00 0x7f9b60115d00 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f9b5000c8c0 tx=0x7f9b500044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:49:11.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.652+0000 7f9b57e00700 1 -- 172.21.15.123:0/2885672230 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9b5000a760 con 0x7f9b60102e00 2024-03-30T04:49:11.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.652+0000 7f9b57e00700 1 -- 172.21.15.123:0/2885672230 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9b5000ad70 con 0x7f9b60102e00 2024-03-30T04:49:11.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.652+0000 7f9b66c00700 1 -- 172.21.15.123:0/2885672230 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b6010ff40 con 0x7f9b60102e00 2024-03-30T04:49:11.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.652+0000 7f9b57e00700 1 -- 172.21.15.123:0/2885672230 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9b500113a0 con 0x7f9b60102e00 2024-03-30T04:49:11.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.652+0000 7f9b66c00700 1 -- 172.21.15.123:0/2885672230 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b60110500 con 0x7f9b60102e00 2024-03-30T04:49:11.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.653+0000 7f9b66c00700 1 -- 172.21.15.123:0/2885672230 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b600fe550 con 0x7f9b60102e00 2024-03-30T04:49:11.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.654+0000 7f9b57e00700 1 -- 172.21.15.123:0/2885672230 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f9b50012400 con 0x7f9b60102e00 2024-03-30T04:49:11.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.656+0000 7f9b57e00700 1 --2- 172.21.15.123:0/2885672230 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9b4406cc30 0x7f9b4406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:49:11.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.656+0000 7f9b57e00700 1 -- 172.21.15.123:0/2885672230 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f9b5008bfe0 con 0x7f9b60102e00 2024-03-30T04:49:11.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.656+0000 7f9b5f400700 1 --2- 172.21.15.123:0/2885672230 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9b4406cc30 0x7f9b4406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:49:11.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.656+0000 7f9b5f400700 1 --2- 172.21.15.123:0/2885672230 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9b4406cc30 0x7f9b4406f0e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f9b601115a0 tx=0x7f9b4c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:49:11.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.660+0000 7f9b57e00700 1 -- 172.21.15.123:0/2885672230 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9b50059af0 con 0x7f9b60102e00 2024-03-30T04:49:11.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:49:11 smithi135 ceph-mon[106011]: from='client.35018 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:49:11.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:49:11 smithi135 ceph-mon[106011]: from='client.35022 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:49:11.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:49:11 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/1204768112' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:49:11.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.910+0000 7f9b66c00700 1 -- 172.21.15.123:0/2885672230 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f9b6004bed0 con 0x7f9b60102e00 2024-03-30T04:49:11.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.911+0000 7f9b57e00700 1 -- 172.21.15.123:0/2885672230 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f9b50059680 con 0x7f9b60102e00 2024-03-30T04:49:11.909 INFO:teuthology.orchestra.run.smithi123.stdout:e32 2024-03-30T04:49:11.909 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:49:11.909 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:49:11.909 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:49:11.909 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:49:11.909 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:49:11.909 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:49:11.910 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T04:49:11.910 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T04:49:11.910 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:49:11.910 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T04:49:11.910 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:49:11.910 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:49:11.910 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:49:11.910 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:49:11.910 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:49:11.910 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T04:49:11.910 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:49:11.910 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:49:11.911 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T04:49:11.911 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:49:11.911 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T04:49:11.911 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T04:49:11.911 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T04:49:11.911 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T04:49:11.911 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:49:11.911 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:49:11.911 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:49:11.911 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:49:11.911 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:49:11.912 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:49:11.912 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T04:49:11.912 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T04:49:11.912 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:49:11.912 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:49:11.912 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T04:49:11.912 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:49:11.912 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:49:11.912 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:49:11.912 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44315} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/1741352336,v1:172.21.15.135:6827/1741352336] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:49:11.912 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:49:11.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.913+0000 7f9b66c00700 1 -- 172.21.15.123:0/2885672230 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9b4406cc30 msgr2=0x7f9b4406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:11.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.913+0000 7f9b66c00700 1 --2- 172.21.15.123:0/2885672230 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9b4406cc30 0x7f9b4406f0e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f9b601115a0 tx=0x7f9b4c009e30 comp rx=0 tx=0).stop 2024-03-30T04:49:11.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.913+0000 7f9b66c00700 1 -- 172.21.15.123:0/2885672230 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b60102e00 msgr2=0x7f9b60115d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:11.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.913+0000 7f9b66c00700 1 --2- 172.21.15.123:0/2885672230 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b60102e00 0x7f9b60115d00 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f9b5000c8c0 tx=0x7f9b500044c0 comp rx=0 tx=0).stop 2024-03-30T04:49:11.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.913+0000 7f9b66c00700 1 -- 172.21.15.123:0/2885672230 shutdown_connections 2024-03-30T04:49:11.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.913+0000 7f9b66c00700 1 --2- 172.21.15.123:0/2885672230 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9b4406cc30 0x7f9b4406f0e0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:11.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.913+0000 7f9b66c00700 1 --2- 172.21.15.123:0/2885672230 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b60102e00 0x7f9b60115d00 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:11.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.913+0000 7f9b66c00700 1 --2- 172.21.15.123:0/2885672230 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9b601037a0 0x7f9b60116240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:11.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.913+0000 7f9b66c00700 1 -- 172.21.15.123:0/2885672230 >> 172.21.15.123:0/2885672230 conn(0x7f9b600f89b0 msgr2=0x7f9b600f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:49:11.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.913+0000 7f9b66c00700 1 -- 172.21.15.123:0/2885672230 shutdown_connections 2024-03-30T04:49:11.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:11.913+0000 7f9b66c00700 1 -- 172.21.15.123:0/2885672230 wait complete. 2024-03-30T04:49:11.914 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 32 2024-03-30T04:49:11.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:49:11 smithi123 ceph-mon[139369]: from='client.35018 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:49:11.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:49:11 smithi123 ceph-mon[139369]: from='client.35022 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:49:11.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:49:11 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/1204768112' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:49:12.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.049+0000 7f3488e00700 1 -- 172.21.15.123:0/2876291874 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3484102e00 msgr2=0x7f34841031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:12.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.049+0000 7f3488e00700 1 --2- 172.21.15.123:0/2876291874 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3484102e00 0x7f34841031d0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f3474004940 tx=0x7f3474009e30 comp rx=0 tx=0).stop 2024-03-30T04:49:12.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.049+0000 7f3488e00700 1 -- 172.21.15.123:0/2876291874 shutdown_connections 2024-03-30T04:49:12.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.049+0000 7f3488e00700 1 --2- 172.21.15.123:0/2876291874 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f34841037a0 0x7f34840fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:12.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.049+0000 7f3488e00700 1 --2- 172.21.15.123:0/2876291874 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3484102e00 0x7f34841031d0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:12.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.049+0000 7f3488e00700 1 -- 172.21.15.123:0/2876291874 >> 172.21.15.123:0/2876291874 conn(0x7f34840f89b0 msgr2=0x7f34840fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:49:12.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.049+0000 7f3488e00700 1 -- 172.21.15.123:0/2876291874 shutdown_connections 2024-03-30T04:49:12.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.050+0000 7f3488e00700 1 -- 172.21.15.123:0/2876291874 wait complete. 2024-03-30T04:49:12.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.050+0000 7f3488e00700 1 Processor -- start 2024-03-30T04:49:12.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.050+0000 7f3488e00700 1 -- start start 2024-03-30T04:49:12.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.051+0000 7f3488e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3484102e00 0x7f3484115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:49:12.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.051+0000 7f3488e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f34841037a0 0x7f3484116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:49:12.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.051+0000 7f3488e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3484116880 con 0x7f3484102e00 2024-03-30T04:49:12.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.051+0000 7f3488e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f348410fc50 con 0x7f34841037a0 2024-03-30T04:49:12.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.051+0000 7f3481e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3484102e00 0x7f3484115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:49:12.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.051+0000 7f3481e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3484102e00 0x7f3484115cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44600/0 (socket says 172.21.15.123:44600) 2024-03-30T04:49:12.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.051+0000 7f3481e00700 1 -- 172.21.15.123:0/1914457327 learned_addr learned my addr 172.21.15.123:0/1914457327 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:49:12.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.051+0000 7f3481400700 1 --2- 172.21.15.123:0/1914457327 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f34841037a0 0x7f3484116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:49:12.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.051+0000 7f3481e00700 1 -- 172.21.15.123:0/1914457327 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f34841037a0 msgr2=0x7f3484116230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:12.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.051+0000 7f3481e00700 1 --2- 172.21.15.123:0/1914457327 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f34841037a0 0x7f3484116230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:12.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.051+0000 7f3481e00700 1 -- 172.21.15.123:0/1914457327 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3474005040 con 0x7f3484102e00 2024-03-30T04:49:12.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.052+0000 7f3481e00700 1 --2- 172.21.15.123:0/1914457327 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3484102e00 0x7f3484115cf0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f3474009360 tx=0x7f3474009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:49:12.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.052+0000 7f347a000700 1 -- 172.21.15.123:0/1914457327 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f347400e040 con 0x7f3484102e00 2024-03-30T04:49:12.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.052+0000 7f347a000700 1 -- 172.21.15.123:0/1914457327 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3474013420 con 0x7f3484102e00 2024-03-30T04:49:12.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.052+0000 7f3488e00700 1 -- 172.21.15.123:0/1914457327 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f348410fed0 con 0x7f3484102e00 2024-03-30T04:49:12.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.052+0000 7f347a000700 1 -- 172.21.15.123:0/1914457327 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f347401c5d0 con 0x7f3484102e00 2024-03-30T04:49:12.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.052+0000 7f3488e00700 1 -- 172.21.15.123:0/1914457327 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3484110490 con 0x7f3484102e00 2024-03-30T04:49:12.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.053+0000 7f3488e00700 1 -- 172.21.15.123:0/1914457327 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34840fe550 con 0x7f3484102e00 2024-03-30T04:49:12.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.054+0000 7f347a000700 1 -- 172.21.15.123:0/1914457327 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f3474013590 con 0x7f3484102e00 2024-03-30T04:49:12.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.055+0000 7f347a000700 1 --2- 172.21.15.123:0/1914457327 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f346806cc30 0x7f346806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:49:12.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.055+0000 7f347a000700 1 -- 172.21.15.123:0/1914457327 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f347408d1b0 con 0x7f3484102e00 2024-03-30T04:49:12.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.055+0000 7f3481400700 1 --2- 172.21.15.123:0/1914457327 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f346806cc30 0x7f346806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:49:12.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.056+0000 7f3481400700 1 --2- 172.21.15.123:0/1914457327 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f346806cc30 0x7f346806f0e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f3484111590 tx=0x7f347000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:49:12.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.059+0000 7f347a000700 1 -- 172.21.15.123:0/1914457327 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f347405ac00 con 0x7f3484102e00 2024-03-30T04:49:12.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.281+0000 7f3488e00700 1 -- 172.21.15.123:0/1914457327 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f348404bed0 con 0x7f346806cc30 2024-03-30T04:49:12.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.283+0000 7f347a000700 1 -- 172.21.15.123:0/1914457327 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f348404bed0 con 0x7f346806cc30 2024-03-30T04:49:12.281 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:49:12.281 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:49:12.281 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:49:12.281 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T04:49:12.281 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:49:12.281 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T04:49:12.281 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T04:49:12.281 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T04:49:12.281 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T04:49:12.282 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:49:12.282 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "15/21 daemons upgraded", 2024-03-30T04:49:12.282 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T04:49:12.282 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:49:12.282 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:49:12.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.285+0000 7f3488e00700 1 -- 172.21.15.123:0/1914457327 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f346806cc30 msgr2=0x7f346806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:12.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.285+0000 7f3488e00700 1 --2- 172.21.15.123:0/1914457327 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f346806cc30 0x7f346806f0e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f3484111590 tx=0x7f347000b040 comp rx=0 tx=0).stop 2024-03-30T04:49:12.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.285+0000 7f3488e00700 1 -- 172.21.15.123:0/1914457327 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3484102e00 msgr2=0x7f3484115cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:12.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.285+0000 7f3488e00700 1 --2- 172.21.15.123:0/1914457327 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3484102e00 0x7f3484115cf0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f3474009360 tx=0x7f3474009e30 comp rx=0 tx=0).stop 2024-03-30T04:49:12.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.285+0000 7f3488e00700 1 -- 172.21.15.123:0/1914457327 shutdown_connections 2024-03-30T04:49:12.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.285+0000 7f3488e00700 1 --2- 172.21.15.123:0/1914457327 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f346806cc30 0x7f346806f0e0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:12.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.285+0000 7f3488e00700 1 --2- 172.21.15.123:0/1914457327 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3484102e00 0x7f3484115cf0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:12.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.285+0000 7f3488e00700 1 --2- 172.21.15.123:0/1914457327 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f34841037a0 0x7f3484116230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:12.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.285+0000 7f3488e00700 1 -- 172.21.15.123:0/1914457327 >> 172.21.15.123:0/1914457327 conn(0x7f34840f89b0 msgr2=0x7f34840706d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:49:12.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.285+0000 7f3488e00700 1 -- 172.21.15.123:0/1914457327 shutdown_connections 2024-03-30T04:49:12.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.285+0000 7f3488e00700 1 -- 172.21.15.123:0/1914457327 wait complete. 2024-03-30T04:49:12.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.419+0000 7febe9400700 1 -- 172.21.15.123:0/990099997 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7febe4102e00 msgr2=0x7febe41031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:12.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.419+0000 7febe9400700 1 --2- 172.21.15.123:0/990099997 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7febe4102e00 0x7febe41031d0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7febd4004940 tx=0x7febd4009e30 comp rx=0 tx=0).stop 2024-03-30T04:49:12.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.419+0000 7febe9400700 1 -- 172.21.15.123:0/990099997 shutdown_connections 2024-03-30T04:49:12.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.419+0000 7febe9400700 1 --2- 172.21.15.123:0/990099997 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7febe41037a0 0x7febe40fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:12.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.419+0000 7febe9400700 1 --2- 172.21.15.123:0/990099997 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7febe4102e00 0x7febe41031d0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:12.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.419+0000 7febe9400700 1 -- 172.21.15.123:0/990099997 >> 172.21.15.123:0/990099997 conn(0x7febe40f89b0 msgr2=0x7febe40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:49:12.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.419+0000 7febe9400700 1 -- 172.21.15.123:0/990099997 shutdown_connections 2024-03-30T04:49:12.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.419+0000 7febe9400700 1 -- 172.21.15.123:0/990099997 wait complete. 2024-03-30T04:49:12.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.420+0000 7febe9400700 1 Processor -- start 2024-03-30T04:49:12.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.420+0000 7febe9400700 1 -- start start 2024-03-30T04:49:12.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.421+0000 7febe9400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7febe4102e00 0x7febe4113a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:49:12.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.421+0000 7febe9400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7febe41037a0 0x7febe4113f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:49:12.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.421+0000 7febe9400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe41145d0 con 0x7febe4102e00 2024-03-30T04:49:12.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.421+0000 7febe9400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe410d9c0 con 0x7febe41037a0 2024-03-30T04:49:12.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.421+0000 7febe2800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7febe4102e00 0x7febe4113a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:49:12.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.421+0000 7febe2800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7febe4102e00 0x7febe4113a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44632/0 (socket says 172.21.15.123:44632) 2024-03-30T04:49:12.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.421+0000 7febe2800700 1 -- 172.21.15.123:0/3709296518 learned_addr learned my addr 172.21.15.123:0/3709296518 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:49:12.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.421+0000 7febdbe00700 1 --2- 172.21.15.123:0/3709296518 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7febe41037a0 0x7febe4113f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:49:12.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.421+0000 7febe2800700 1 -- 172.21.15.123:0/3709296518 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7febe41037a0 msgr2=0x7febe4113f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:12.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.421+0000 7febe2800700 1 --2- 172.21.15.123:0/3709296518 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7febe41037a0 0x7febe4113f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:12.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.421+0000 7febe2800700 1 -- 172.21.15.123:0/3709296518 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd4005040 con 0x7febe4102e00 2024-03-30T04:49:12.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.422+0000 7febe2800700 1 --2- 172.21.15.123:0/3709296518 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7febe4102e00 0x7febe4113a40 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7febd4007f50 tx=0x7febd4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:49:12.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.422+0000 7febdaa00700 1 -- 172.21.15.123:0/3709296518 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7febd4021070 con 0x7febe4102e00 2024-03-30T04:49:12.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.422+0000 7febdaa00700 1 -- 172.21.15.123:0/3709296518 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7febd40093b0 con 0x7febe4102e00 2024-03-30T04:49:12.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.422+0000 7febdaa00700 1 -- 172.21.15.123:0/3709296518 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7febd400e070 con 0x7febe4102e00 2024-03-30T04:49:12.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.422+0000 7febe9400700 1 -- 172.21.15.123:0/3709296518 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febe410dc40 con 0x7febe4102e00 2024-03-30T04:49:12.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.422+0000 7febe9400700 1 -- 172.21.15.123:0/3709296518 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febe410e1a0 con 0x7febe4102e00 2024-03-30T04:49:12.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.423+0000 7febe9400700 1 -- 172.21.15.123:0/3709296518 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febe40fe550 con 0x7febe4102e00 2024-03-30T04:49:12.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.424+0000 7febdaa00700 1 -- 172.21.15.123:0/3709296518 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7febd401c400 con 0x7febe4102e00 2024-03-30T04:49:12.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.425+0000 7febdaa00700 1 --2- 172.21.15.123:0/3709296518 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7febc806cc30 0x7febc806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:49:12.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.425+0000 7febdaa00700 1 -- 172.21.15.123:0/3709296518 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7febd408d120 con 0x7febe4102e00 2024-03-30T04:49:12.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.425+0000 7febdbe00700 1 --2- 172.21.15.123:0/3709296518 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7febc806cc30 0x7febc806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:49:12.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.426+0000 7febdbe00700 1 --2- 172.21.15.123:0/3709296518 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7febc806cc30 0x7febc806f0e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7febe410eb80 tx=0x7febcc00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:49:12.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.430+0000 7febdaa00700 1 -- 172.21.15.123:0/3709296518 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7febd405abf0 con 0x7febe4102e00 2024-03-30T04:49:12.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.717+0000 7febe9400700 1 -- 172.21.15.123:0/3709296518 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7febe4109e50 con 0x7febe4102e00 2024-03-30T04:49:12.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.717+0000 7febdaa00700 1 -- 172.21.15.123:0/3709296518 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7febd405a780 con 0x7febe4102e00 2024-03-30T04:49:12.715 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T04:49:12.716 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T04:49:12.716 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T04:49:12.716 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T04:49:12.716 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T04:49:12.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.719+0000 7febe9400700 1 -- 172.21.15.123:0/3709296518 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7febc806cc30 msgr2=0x7febc806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:12.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.719+0000 7febe9400700 1 --2- 172.21.15.123:0/3709296518 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7febc806cc30 0x7febc806f0e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7febe410eb80 tx=0x7febcc00b040 comp rx=0 tx=0).stop 2024-03-30T04:49:12.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.719+0000 7febe9400700 1 -- 172.21.15.123:0/3709296518 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7febe4102e00 msgr2=0x7febe4113a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:12.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.719+0000 7febe9400700 1 --2- 172.21.15.123:0/3709296518 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7febe4102e00 0x7febe4113a40 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7febd4007f50 tx=0x7febd4009e30 comp rx=0 tx=0).stop 2024-03-30T04:49:12.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.720+0000 7febe9400700 1 -- 172.21.15.123:0/3709296518 shutdown_connections 2024-03-30T04:49:12.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.720+0000 7febe9400700 1 --2- 172.21.15.123:0/3709296518 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7febc806cc30 0x7febc806f0e0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:12.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.720+0000 7febe9400700 1 --2- 172.21.15.123:0/3709296518 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7febe4102e00 0x7febe4113a40 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:12.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.720+0000 7febe9400700 1 --2- 172.21.15.123:0/3709296518 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7febe41037a0 0x7febe4113f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:12.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.720+0000 7febe9400700 1 -- 172.21.15.123:0/3709296518 >> 172.21.15.123:0/3709296518 conn(0x7febe40f89b0 msgr2=0x7febe40f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:49:12.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.720+0000 7febe9400700 1 -- 172.21.15.123:0/3709296518 shutdown_connections 2024-03-30T04:49:12.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:12.720+0000 7febe9400700 1 -- 172.21.15.123:0/3709296518 wait complete. 2024-03-30T04:49:12.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:49:12 smithi135 ceph-mon[106011]: from='client.35026 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:49:12.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:49:12 smithi135 ceph-mon[106011]: pgmap v540: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:12.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:49:12 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2885672230' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:49:12.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:49:12 smithi123 ceph-mon[139369]: from='client.35026 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:49:12.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:49:12 smithi123 ceph-mon[139369]: pgmap v540: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:12.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:49:12 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2885672230' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:49:13.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:49:13 smithi135 ceph-mon[106011]: from='client.35038 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:49:13.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:49:13 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3709296518' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:49:13.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:49:13 smithi123 ceph-mon[139369]: from='client.35038 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:49:13.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:49:13 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3709296518' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:49:14.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:49:14 smithi135 ceph-mon[106011]: pgmap v541: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:14.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:49:14 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:49:14.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:49:14 smithi123 ceph-mon[139369]: pgmap v541: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:14.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:49:14 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:49:16.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:49:16 smithi123 ceph-mon[139369]: pgmap v542: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:17.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:49:16 smithi135 ceph-mon[106011]: pgmap v542: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:18.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:49:18 smithi123 ceph-mon[139369]: pgmap v543: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:19.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:49:18 smithi135 ceph-mon[106011]: pgmap v543: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:20.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:49:20 smithi135 ceph-mon[106011]: pgmap v544: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:20.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:49:20 smithi123 ceph-mon[139369]: pgmap v544: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:22.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:49:22 smithi123 ceph-mon[139369]: pgmap v545: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:23.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:49:22 smithi135 ceph-mon[106011]: pgmap v545: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:24.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:49:24 smithi123 ceph-mon[139369]: pgmap v546: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:25.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:49:24 smithi135 ceph-mon[106011]: pgmap v546: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:26.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:49:26 smithi123 ceph-mon[139369]: pgmap v547: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:27.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:49:26 smithi135 ceph-mon[106011]: pgmap v547: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:28.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:49:28 smithi123 ceph-mon[139369]: pgmap v548: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:29.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:49:28 smithi135 ceph-mon[106011]: pgmap v548: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:29.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:49:29 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:49:30.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:49:29 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:49:30.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:49:30 smithi123 ceph-mon[139369]: pgmap v549: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:31.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:49:30 smithi135 ceph-mon[106011]: pgmap v549: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:32.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:49:32 smithi123 ceph-mon[139369]: pgmap v550: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:33.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:49:32 smithi135 ceph-mon[106011]: pgmap v550: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:34.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:49:34 smithi123 ceph-mon[139369]: pgmap v551: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:35.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:49:34 smithi135 ceph-mon[106011]: pgmap v551: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:36.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:49:36 smithi123 ceph-mon[139369]: pgmap v552: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:37.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:49:36 smithi135 ceph-mon[106011]: pgmap v552: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:38.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:49:38 smithi123 ceph-mon[139369]: pgmap v553: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:39.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:49:38 smithi135 ceph-mon[106011]: pgmap v553: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:40.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:49:40 smithi135 ceph-mon[106011]: pgmap v554: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:40.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:49:40 smithi123 ceph-mon[139369]: pgmap v554: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:42.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:42.869+0000 7f49d1000700 1 -- 172.21.15.123:0/1767859909 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49cc1059b0 msgr2=0x7f49cc0ff020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:42.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:42.869+0000 7f49d1000700 1 --2- 172.21.15.123:0/1767859909 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49cc1059b0 0x7f49cc0ff020 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f49b8004940 tx=0x7f49b8009e30 comp rx=0 tx=0).stop 2024-03-30T04:49:42.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:42.870+0000 7f49d1000700 1 -- 172.21.15.123:0/1767859909 shutdown_connections 2024-03-30T04:49:42.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:42.870+0000 7f49d1000700 1 --2- 172.21.15.123:0/1767859909 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49cc1059b0 0x7f49cc0ff020 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:42.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:42.870+0000 7f49d1000700 1 --2- 172.21.15.123:0/1767859909 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49cc105010 0x7f49cc1053e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:42.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:42.870+0000 7f49d1000700 1 -- 172.21.15.123:0/1767859909 >> 172.21.15.123:0/1767859909 conn(0x7f49cc0faba0 msgr2=0x7f49cc0fcfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:49:42.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:42.870+0000 7f49d1000700 1 -- 172.21.15.123:0/1767859909 shutdown_connections 2024-03-30T04:49:42.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:42.870+0000 7f49d1000700 1 -- 172.21.15.123:0/1767859909 wait complete. 2024-03-30T04:49:42.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:42.870+0000 7f49d1000700 1 Processor -- start 2024-03-30T04:49:42.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:42.871+0000 7f49d1000700 1 -- start start 2024-03-30T04:49:42.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:42.871+0000 7f49d1000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49cc105010 0x7f49cc115cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:49:42.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:42.871+0000 7f49d1000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49cc1059b0 0x7f49cc1161f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:49:42.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:42.871+0000 7f49d1000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49cc116840 con 0x7f49cc105010 2024-03-30T04:49:42.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:42.871+0000 7f49d1000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49cc10fc60 con 0x7f49cc1059b0 2024-03-30T04:49:42.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:42.871+0000 7f49c9e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49cc105010 0x7f49cc115cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:49:42.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:42.871+0000 7f49c9e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49cc105010 0x7f49cc115cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33504/0 (socket says 172.21.15.123:33504) 2024-03-30T04:49:42.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:42.871+0000 7f49c9e00700 1 -- 172.21.15.123:0/1769459056 learned_addr learned my addr 172.21.15.123:0/1769459056 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:49:42.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:42.872+0000 7f49c9400700 1 --2- 172.21.15.123:0/1769459056 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49cc1059b0 0x7f49cc1161f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:49:42.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:42.872+0000 7f49c9e00700 1 -- 172.21.15.123:0/1769459056 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49cc1059b0 msgr2=0x7f49cc1161f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:42.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:42.872+0000 7f49c9e00700 1 --2- 172.21.15.123:0/1769459056 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49cc1059b0 0x7f49cc1161f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:42.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:42.872+0000 7f49c9e00700 1 -- 172.21.15.123:0/1769459056 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49b8005040 con 0x7f49cc105010 2024-03-30T04:49:42.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:42.872+0000 7f49c9400700 1 --2- 172.21.15.123:0/1769459056 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49cc1059b0 0x7f49cc1161f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:49:42.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:42.872+0000 7f49c9e00700 1 --2- 172.21.15.123:0/1769459056 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49cc105010 0x7f49cc115cb0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f49bc00c8c0 tx=0x7f49bc0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:49:42.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:42.872+0000 7f49c2000700 1 -- 172.21.15.123:0/1769459056 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f49bc00a980 con 0x7f49cc105010 2024-03-30T04:49:42.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:42.872+0000 7f49c2000700 1 -- 172.21.15.123:0/1769459056 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f49bc00ce60 con 0x7f49cc105010 2024-03-30T04:49:42.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:42.872+0000 7f49d1000700 1 -- 172.21.15.123:0/1769459056 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49cc10ff40 con 0x7f49cc105010 2024-03-30T04:49:42.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:42.872+0000 7f49c2000700 1 -- 172.21.15.123:0/1769459056 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f49bc0113a0 con 0x7f49cc105010 2024-03-30T04:49:42.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:42.872+0000 7f49d1000700 1 -- 172.21.15.123:0/1769459056 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49cc110500 con 0x7f49cc105010 2024-03-30T04:49:42.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:42.873+0000 7f49d1000700 1 -- 172.21.15.123:0/1769459056 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49cc102320 con 0x7f49cc105010 2024-03-30T04:49:42.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:42.874+0000 7f49c2000700 1 -- 172.21.15.123:0/1769459056 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f49bc012400 con 0x7f49cc105010 2024-03-30T04:49:42.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:42.876+0000 7f49c2000700 1 --2- 172.21.15.123:0/1769459056 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f49b006cc30 0x7f49b006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:49:42.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:42.876+0000 7f49c2000700 1 -- 172.21.15.123:0/1769459056 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f49bc08c0f0 con 0x7f49cc105010 2024-03-30T04:49:42.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:42.876+0000 7f49c9400700 1 --2- 172.21.15.123:0/1769459056 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f49b006cc30 0x7f49b006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:49:42.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:42.876+0000 7f49c9400700 1 --2- 172.21.15.123:0/1769459056 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f49b006cc30 0x7f49b006f0e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f49b800b3b0 tx=0x7f49b8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:49:42.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:42.881+0000 7f49c2000700 1 -- 172.21.15.123:0/1769459056 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f49bc059bc0 con 0x7f49cc105010 2024-03-30T04:49:42.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:49:42 smithi123 ceph-mon[139369]: pgmap v555: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:43.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.101+0000 7f49d1000700 1 -- 172.21.15.123:0/1769459056 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f49cc10bfb0 con 0x7f49b006cc30 2024-03-30T04:49:43.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.102+0000 7f49c2000700 1 -- 172.21.15.123:0/1769459056 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f49cc10bfb0 con 0x7f49b006cc30 2024-03-30T04:49:43.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:49:42 smithi135 ceph-mon[106011]: pgmap v555: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:43.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.105+0000 7f49d1000700 1 -- 172.21.15.123:0/1769459056 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f49b006cc30 msgr2=0x7f49b006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:43.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.105+0000 7f49d1000700 1 --2- 172.21.15.123:0/1769459056 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f49b006cc30 0x7f49b006f0e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f49b800b3b0 tx=0x7f49b8009e30 comp rx=0 tx=0).stop 2024-03-30T04:49:43.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.105+0000 7f49d1000700 1 -- 172.21.15.123:0/1769459056 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49cc105010 msgr2=0x7f49cc115cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:43.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.105+0000 7f49d1000700 1 --2- 172.21.15.123:0/1769459056 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49cc105010 0x7f49cc115cb0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f49bc00c8c0 tx=0x7f49bc0044c0 comp rx=0 tx=0).stop 2024-03-30T04:49:43.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.105+0000 7f49d1000700 1 -- 172.21.15.123:0/1769459056 shutdown_connections 2024-03-30T04:49:43.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.105+0000 7f49d1000700 1 --2- 172.21.15.123:0/1769459056 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f49b006cc30 0x7f49b006f0e0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:43.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.105+0000 7f49d1000700 1 --2- 172.21.15.123:0/1769459056 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49cc105010 0x7f49cc115cb0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:43.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.105+0000 7f49d1000700 1 --2- 172.21.15.123:0/1769459056 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49cc1059b0 0x7f49cc1161f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:43.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.105+0000 7f49d1000700 1 -- 172.21.15.123:0/1769459056 >> 172.21.15.123:0/1769459056 conn(0x7f49cc0faba0 msgr2=0x7f49cc108640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:49:43.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.105+0000 7f49d1000700 1 -- 172.21.15.123:0/1769459056 shutdown_connections 2024-03-30T04:49:43.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.105+0000 7f49d1000700 1 -- 172.21.15.123:0/1769459056 wait complete. 2024-03-30T04:49:43.117 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:49:43.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.240+0000 7f6df1600700 1 -- 172.21.15.123:0/1731446085 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6dec102e00 msgr2=0x7f6dec1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:43.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.240+0000 7f6df1600700 1 --2- 172.21.15.123:0/1731446085 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6dec102e00 0x7f6dec1031d0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc004940 tx=0x7f6ddc009e30 comp rx=0 tx=0).stop 2024-03-30T04:49:43.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.240+0000 7f6df1600700 1 -- 172.21.15.123:0/1731446085 shutdown_connections 2024-03-30T04:49:43.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.240+0000 7f6df1600700 1 --2- 172.21.15.123:0/1731446085 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6dec1037a0 0x7f6dec0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:43.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.240+0000 7f6df1600700 1 --2- 172.21.15.123:0/1731446085 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6dec102e00 0x7f6dec1031d0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:43.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.240+0000 7f6df1600700 1 -- 172.21.15.123:0/1731446085 >> 172.21.15.123:0/1731446085 conn(0x7f6dec0f89b0 msgr2=0x7f6dec0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:49:43.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.240+0000 7f6df1600700 1 -- 172.21.15.123:0/1731446085 shutdown_connections 2024-03-30T04:49:43.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.240+0000 7f6df1600700 1 -- 172.21.15.123:0/1731446085 wait complete. 2024-03-30T04:49:43.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.241+0000 7f6df1600700 1 Processor -- start 2024-03-30T04:49:43.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.241+0000 7f6df1600700 1 -- start start 2024-03-30T04:49:43.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.242+0000 7f6df1600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6dec102e00 0x7f6dec0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:49:43.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.242+0000 7f6df1600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6dec1037a0 0x7f6dec070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:49:43.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.242+0000 7f6df1600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dec070d90 con 0x7f6dec102e00 2024-03-30T04:49:43.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.242+0000 7f6df1600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dec070ed0 con 0x7f6dec1037a0 2024-03-30T04:49:43.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.242+0000 7f6deaa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6dec102e00 0x7f6dec0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:49:43.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.242+0000 7f6deaa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6dec102e00 0x7f6dec0702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33522/0 (socket says 172.21.15.123:33522) 2024-03-30T04:49:43.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.242+0000 7f6deaa00700 1 -- 172.21.15.123:0/3501507807 learned_addr learned my addr 172.21.15.123:0/3501507807 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:49:43.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.242+0000 7f6deaa00700 1 -- 172.21.15.123:0/3501507807 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6dec1037a0 msgr2=0x7f6dec070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:43.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.242+0000 7f6deaa00700 1 --2- 172.21.15.123:0/3501507807 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6dec1037a0 0x7f6dec070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:43.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.242+0000 7f6deaa00700 1 -- 172.21.15.123:0/3501507807 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ddc005040 con 0x7f6dec102e00 2024-03-30T04:49:43.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.243+0000 7f6deaa00700 1 --2- 172.21.15.123:0/3501507807 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6dec102e00 0x7f6dec0702c0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc009360 tx=0x7f6ddc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:49:43.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.243+0000 7f6de2a00700 1 -- 172.21.15.123:0/3501507807 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ddc00e040 con 0x7f6dec102e00 2024-03-30T04:49:43.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.243+0000 7f6de2a00700 1 -- 172.21.15.123:0/3501507807 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6ddc013420 con 0x7f6dec102e00 2024-03-30T04:49:43.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.243+0000 7f6df1600700 1 -- 172.21.15.123:0/3501507807 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dec071120 con 0x7f6dec102e00 2024-03-30T04:49:43.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.243+0000 7f6de2a00700 1 -- 172.21.15.123:0/3501507807 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ddc01c5f0 con 0x7f6dec102e00 2024-03-30T04:49:43.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.243+0000 7f6df1600700 1 -- 172.21.15.123:0/3501507807 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dec11a250 con 0x7f6dec102e00 2024-03-30T04:49:43.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.244+0000 7f6df1600700 1 -- 172.21.15.123:0/3501507807 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dec0fe550 con 0x7f6dec102e00 2024-03-30T04:49:43.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.245+0000 7f6de2a00700 1 -- 172.21.15.123:0/3501507807 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f6ddc013590 con 0x7f6dec102e00 2024-03-30T04:49:43.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.246+0000 7f6de2a00700 1 --2- 172.21.15.123:0/3501507807 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6dd006cc30 0x7f6dd006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:49:43.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.246+0000 7f6de2a00700 1 -- 172.21.15.123:0/3501507807 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f6ddc08d1b0 con 0x7f6dec102e00 2024-03-30T04:49:43.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.246+0000 7f6dea000700 1 --2- 172.21.15.123:0/3501507807 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6dd006cc30 0x7f6dd006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:49:43.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.247+0000 7f6dea000700 1 --2- 172.21.15.123:0/3501507807 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6dd006cc30 0x7f6dd006f0e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6dec071c00 tx=0x7f6dd8009ce0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:49:43.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.250+0000 7f6de2a00700 1 -- 172.21.15.123:0/3501507807 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6ddc0572b0 con 0x7f6dec102e00 2024-03-30T04:49:43.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.472+0000 7f6df1600700 1 -- 172.21.15.123:0/3501507807 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6dec04bed0 con 0x7f6dd006cc30 2024-03-30T04:49:43.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.473+0000 7f6de2a00700 1 -- 172.21.15.123:0/3501507807 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f6dec04bed0 con 0x7f6dd006cc30 2024-03-30T04:49:43.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.475+0000 7f6df1600700 1 -- 172.21.15.123:0/3501507807 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6dd006cc30 msgr2=0x7f6dd006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:43.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.475+0000 7f6df1600700 1 --2- 172.21.15.123:0/3501507807 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6dd006cc30 0x7f6dd006f0e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6dec071c00 tx=0x7f6dd8009ce0 comp rx=0 tx=0).stop 2024-03-30T04:49:43.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.475+0000 7f6df1600700 1 -- 172.21.15.123:0/3501507807 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6dec102e00 msgr2=0x7f6dec0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:43.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.475+0000 7f6df1600700 1 --2- 172.21.15.123:0/3501507807 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6dec102e00 0x7f6dec0702c0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc009360 tx=0x7f6ddc009e30 comp rx=0 tx=0).stop 2024-03-30T04:49:43.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.475+0000 7f6df1600700 1 -- 172.21.15.123:0/3501507807 shutdown_connections 2024-03-30T04:49:43.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.475+0000 7f6df1600700 1 --2- 172.21.15.123:0/3501507807 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6dd006cc30 0x7f6dd006f0e0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:43.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.475+0000 7f6df1600700 1 --2- 172.21.15.123:0/3501507807 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6dec102e00 0x7f6dec0702c0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:43.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.475+0000 7f6df1600700 1 --2- 172.21.15.123:0/3501507807 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6dec1037a0 0x7f6dec070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:43.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.475+0000 7f6df1600700 1 -- 172.21.15.123:0/3501507807 >> 172.21.15.123:0/3501507807 conn(0x7f6dec0f89b0 msgr2=0x7f6dec0f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:49:43.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.476+0000 7f6df1600700 1 -- 172.21.15.123:0/3501507807 shutdown_connections 2024-03-30T04:49:43.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.476+0000 7f6df1600700 1 -- 172.21.15.123:0/3501507807 wait complete. 2024-03-30T04:49:43.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.611+0000 7fb76f600700 1 -- 172.21.15.123:0/3208230295 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb768102e00 msgr2=0x7fb7681031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:43.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.611+0000 7fb76f600700 1 --2- 172.21.15.123:0/3208230295 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb768102e00 0x7fb7681031d0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fb75c004940 tx=0x7fb75c009e30 comp rx=0 tx=0).stop 2024-03-30T04:49:43.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.611+0000 7fb76f600700 1 -- 172.21.15.123:0/3208230295 shutdown_connections 2024-03-30T04:49:43.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.611+0000 7fb76f600700 1 --2- 172.21.15.123:0/3208230295 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb7681037a0 0x7fb7680fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:43.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.611+0000 7fb76f600700 1 --2- 172.21.15.123:0/3208230295 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb768102e00 0x7fb7681031d0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:43.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.611+0000 7fb76f600700 1 -- 172.21.15.123:0/3208230295 >> 172.21.15.123:0/3208230295 conn(0x7fb7680f89b0 msgr2=0x7fb7680fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:49:43.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.612+0000 7fb76f600700 1 -- 172.21.15.123:0/3208230295 shutdown_connections 2024-03-30T04:49:43.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.612+0000 7fb76f600700 1 -- 172.21.15.123:0/3208230295 wait complete. 2024-03-30T04:49:43.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.613+0000 7fb76f600700 1 Processor -- start 2024-03-30T04:49:43.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.613+0000 7fb76f600700 1 -- start start 2024-03-30T04:49:43.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.613+0000 7fb76f600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb768102e00 0x7fb768113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:49:43.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.613+0000 7fb76f600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7681037a0 0x7fb768114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:49:43.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.613+0000 7fb76f600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7681146b0 con 0x7fb7681037a0 2024-03-30T04:49:43.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.613+0000 7fb76f600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb76810daa0 con 0x7fb768102e00 2024-03-30T04:49:43.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.614+0000 7fb767e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7681037a0 0x7fb768114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:49:43.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.614+0000 7fb767e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7681037a0 0x7fb768114060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33554/0 (socket says 172.21.15.123:33554) 2024-03-30T04:49:43.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.614+0000 7fb767e00700 1 -- 172.21.15.123:0/945116677 learned_addr learned my addr 172.21.15.123:0/945116677 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:49:43.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.614+0000 7fb76cc00700 1 --2- 172.21.15.123:0/945116677 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb768102e00 0x7fb768113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:49:43.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.614+0000 7fb767e00700 1 -- 172.21.15.123:0/945116677 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb768102e00 msgr2=0x7fb768113b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:43.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.614+0000 7fb767e00700 1 --2- 172.21.15.123:0/945116677 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb768102e00 0x7fb768113b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:43.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.614+0000 7fb767e00700 1 -- 172.21.15.123:0/945116677 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb75c005040 con 0x7fb7681037a0 2024-03-30T04:49:43.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.614+0000 7fb767e00700 1 --2- 172.21.15.123:0/945116677 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7681037a0 0x7fb768114060 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fb754002a50 tx=0x7fb754004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:49:43.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.615+0000 7fb764c00700 1 -- 172.21.15.123:0/945116677 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb754007cc0 con 0x7fb7681037a0 2024-03-30T04:49:43.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.615+0000 7fb76f600700 1 -- 172.21.15.123:0/945116677 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb76810dd80 con 0x7fb7681037a0 2024-03-30T04:49:43.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.615+0000 7fb764c00700 1 -- 172.21.15.123:0/945116677 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb75400f040 con 0x7fb7681037a0 2024-03-30T04:49:43.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.615+0000 7fb764c00700 1 -- 172.21.15.123:0/945116677 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb75400d920 con 0x7fb7681037a0 2024-03-30T04:49:43.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.615+0000 7fb76f600700 1 -- 172.21.15.123:0/945116677 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb76810e340 con 0x7fb7681037a0 2024-03-30T04:49:43.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.616+0000 7fb76f600700 1 -- 172.21.15.123:0/945116677 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7680fe550 con 0x7fb7681037a0 2024-03-30T04:49:43.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.617+0000 7fb764c00700 1 -- 172.21.15.123:0/945116677 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fb754009680 con 0x7fb7681037a0 2024-03-30T04:49:43.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.618+0000 7fb764c00700 1 --2- 172.21.15.123:0/945116677 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb75006cc30 0x7fb75006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:49:43.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.618+0000 7fb764c00700 1 -- 172.21.15.123:0/945116677 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb75408be50 con 0x7fb7681037a0 2024-03-30T04:49:43.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.618+0000 7fb76cc00700 1 --2- 172.21.15.123:0/945116677 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb75006cc30 0x7fb75006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:49:43.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.619+0000 7fb76cc00700 1 --2- 172.21.15.123:0/945116677 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb75006cc30 0x7fb75006f0e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb75c009fd0 tx=0x7fb75c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:49:43.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.622+0000 7fb764c00700 1 -- 172.21.15.123:0/945116677 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb754059820 con 0x7fb7681037a0 2024-03-30T04:49:43.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.835+0000 7fb76f600700 1 -- 172.21.15.123:0/945116677 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb768109dc0 con 0x7fb75006cc30 2024-03-30T04:49:43.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.845+0000 7fb764c00700 1 -- 172.21.15.123:0/945116677 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fb768109dc0 con 0x7fb75006cc30 2024-03-30T04:49:43.843 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:49:43.843 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (18m) 11m ago 31m 15.0M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:49:43.843 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (16m) 11m ago 31m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T04:49:43.844 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (16m) 11m ago 30m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T04:49:43.844 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (18m) 11m ago 31m 74.0M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T04:49:43.844 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (11m) 11m ago 25m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T04:49:43.844 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (12m) 11m ago 25m 10.9M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T04:49:43.844 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (24m) 11m ago 24m 10.7M - 17.2.7-797-g5c024335 f77085f6aab7 0dc41326818c 2024-03-30T04:49:43.844 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (11m) 11m ago 25m 9747k - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T04:49:43.844 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (21m) 11m ago 33m 506M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:49:43.844 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (20m) 11m ago 30m 450M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:49:43.844 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (17m) 11m ago 33m 53.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T04:49:43.844 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (16m) 11m ago 30m 42.6M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T04:49:43.844 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (19m) 11m ago 31m 8052k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:49:43.844 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (19m) 11m ago 30m 7880k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:49:43.844 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (15m) 11m ago 29m 104M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T04:49:43.844 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (15m) 11m ago 28m 93.4M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T04:49:43.845 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (14m) 11m ago 28m 86.3M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T04:49:43.845 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (13m) 11m ago 27m 97.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T04:49:43.845 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (13m) 11m ago 27m 97.9M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T04:49:43.845 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (12m) 11m ago 26m 85.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T04:49:43.845 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (19m) 11m ago 30m 53.7M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:49:43.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.847+0000 7fb76f600700 1 -- 172.21.15.123:0/945116677 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb75006cc30 msgr2=0x7fb75006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:43.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.847+0000 7fb76f600700 1 --2- 172.21.15.123:0/945116677 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb75006cc30 0x7fb75006f0e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb75c009fd0 tx=0x7fb75c009e30 comp rx=0 tx=0).stop 2024-03-30T04:49:43.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.847+0000 7fb76f600700 1 -- 172.21.15.123:0/945116677 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7681037a0 msgr2=0x7fb768114060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:43.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.847+0000 7fb76f600700 1 --2- 172.21.15.123:0/945116677 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7681037a0 0x7fb768114060 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fb754002a50 tx=0x7fb754004600 comp rx=0 tx=0).stop 2024-03-30T04:49:43.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.847+0000 7fb76f600700 1 -- 172.21.15.123:0/945116677 shutdown_connections 2024-03-30T04:49:43.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.847+0000 7fb76f600700 1 --2- 172.21.15.123:0/945116677 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb75006cc30 0x7fb75006f0e0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:43.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.847+0000 7fb76f600700 1 --2- 172.21.15.123:0/945116677 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb768102e00 0x7fb768113b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:43.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.847+0000 7fb76f600700 1 --2- 172.21.15.123:0/945116677 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7681037a0 0x7fb768114060 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:43.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.847+0000 7fb76f600700 1 -- 172.21.15.123:0/945116677 >> 172.21.15.123:0/945116677 conn(0x7fb7680f89b0 msgr2=0x7fb7680f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:49:43.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.848+0000 7fb76f600700 1 -- 172.21.15.123:0/945116677 shutdown_connections 2024-03-30T04:49:43.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.848+0000 7fb76f600700 1 -- 172.21.15.123:0/945116677 wait complete. 2024-03-30T04:49:43.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.980+0000 7ff304a00700 1 -- 172.21.15.123:0/4119032039 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff300102e00 msgr2=0x7ff3001031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:43.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.980+0000 7ff304a00700 1 --2- 172.21.15.123:0/4119032039 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff300102e00 0x7ff3001031d0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7ff2f0004940 tx=0x7ff2f0009e30 comp rx=0 tx=0).stop 2024-03-30T04:49:43.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.980+0000 7ff304a00700 1 -- 172.21.15.123:0/4119032039 shutdown_connections 2024-03-30T04:49:43.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.980+0000 7ff304a00700 1 --2- 172.21.15.123:0/4119032039 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff3001037a0 0x7ff3000fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:43.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.980+0000 7ff304a00700 1 --2- 172.21.15.123:0/4119032039 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff300102e00 0x7ff3001031d0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:43.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.980+0000 7ff304a00700 1 -- 172.21.15.123:0/4119032039 >> 172.21.15.123:0/4119032039 conn(0x7ff3000f89b0 msgr2=0x7ff3000fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:49:43.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.980+0000 7ff304a00700 1 -- 172.21.15.123:0/4119032039 shutdown_connections 2024-03-30T04:49:43.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.981+0000 7ff304a00700 1 -- 172.21.15.123:0/4119032039 wait complete. 2024-03-30T04:49:43.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.981+0000 7ff304a00700 1 Processor -- start 2024-03-30T04:49:43.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.981+0000 7ff304a00700 1 -- start start 2024-03-30T04:49:43.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.982+0000 7ff304a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff300102e00 0x7ff300113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:49:43.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.982+0000 7ff304a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff3001037a0 0x7ff300114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:49:43.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.982+0000 7ff304a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3001146b0 con 0x7ff300102e00 2024-03-30T04:49:43.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.982+0000 7ff304a00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff30010daa0 con 0x7ff3001037a0 2024-03-30T04:49:43.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.982+0000 7ff2fde00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff300102e00 0x7ff300113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:49:43.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.982+0000 7ff2fde00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff300102e00 0x7ff300113b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39120/0 (socket says 172.21.15.123:39120) 2024-03-30T04:49:43.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.982+0000 7ff2fde00700 1 -- 172.21.15.123:0/4017028187 learned_addr learned my addr 172.21.15.123:0/4017028187 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:49:43.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.982+0000 7ff2fd400700 1 --2- 172.21.15.123:0/4017028187 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff3001037a0 0x7ff300114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:49:43.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.982+0000 7ff2fde00700 1 -- 172.21.15.123:0/4017028187 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff3001037a0 msgr2=0x7ff300114060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:43.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.982+0000 7ff2fde00700 1 --2- 172.21.15.123:0/4017028187 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff3001037a0 0x7ff300114060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:43.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.982+0000 7ff2fde00700 1 -- 172.21.15.123:0/4017028187 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f0005040 con 0x7ff300102e00 2024-03-30T04:49:43.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.983+0000 7ff2fde00700 1 --2- 172.21.15.123:0/4017028187 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff300102e00 0x7ff300113b20 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7ff2f0009390 tx=0x7ff2f0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:49:43.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.983+0000 7ff2f6000700 1 -- 172.21.15.123:0/4017028187 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff2f000e040 con 0x7ff300102e00 2024-03-30T04:49:43.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.983+0000 7ff2f6000700 1 -- 172.21.15.123:0/4017028187 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff2f0013420 con 0x7ff300102e00 2024-03-30T04:49:43.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.983+0000 7ff304a00700 1 -- 172.21.15.123:0/4017028187 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff30010dd20 con 0x7ff300102e00 2024-03-30T04:49:43.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.983+0000 7ff2f6000700 1 -- 172.21.15.123:0/4017028187 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff2f001c630 con 0x7ff300102e00 2024-03-30T04:49:43.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.983+0000 7ff304a00700 1 -- 172.21.15.123:0/4017028187 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff30010e2e0 con 0x7ff300102e00 2024-03-30T04:49:43.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.984+0000 7ff304a00700 1 -- 172.21.15.123:0/4017028187 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3000fe550 con 0x7ff300102e00 2024-03-30T04:49:43.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.985+0000 7ff2f6000700 1 -- 172.21.15.123:0/4017028187 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7ff2f00139e0 con 0x7ff300102e00 2024-03-30T04:49:43.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.986+0000 7ff2f6000700 1 --2- 172.21.15.123:0/4017028187 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ff2e406cc30 0x7ff2e406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:49:43.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.986+0000 7ff2f6000700 1 -- 172.21.15.123:0/4017028187 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff2f008d240 con 0x7ff300102e00 2024-03-30T04:49:43.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.986+0000 7ff2fd400700 1 --2- 172.21.15.123:0/4017028187 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ff2e406cc30 0x7ff2e406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:49:43.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.987+0000 7ff2fd400700 1 --2- 172.21.15.123:0/4017028187 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ff2e406cc30 0x7ff2e406f0e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec007fd0 tx=0x7ff2ec009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:49:43.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:43.990+0000 7ff2f6000700 1 -- 172.21.15.123:0/4017028187 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff2f005adc0 con 0x7ff300102e00 2024-03-30T04:49:44.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.276+0000 7ff304a00700 1 -- 172.21.15.123:0/4017028187 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ff300109dc0 con 0x7ff300102e00 2024-03-30T04:49:44.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.277+0000 7ff2f6000700 1 -- 172.21.15.123:0/4017028187 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7ff2f005a950 con 0x7ff300102e00 2024-03-30T04:49:44.275 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:49:44.275 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:49:44.275 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:49:44.275 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:49:44.275 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:49:44.276 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:49:44.276 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:49:44.276 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:49:44.276 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T04:49:44.276 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:49:44.276 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:49:44.276 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:49:44.276 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 3 2024-03-30T04:49:44.276 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:49:44.276 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:49:44.276 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:49:44.276 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 13 2024-03-30T04:49:44.276 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:49:44.276 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:49:44.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.279+0000 7ff304a00700 1 -- 172.21.15.123:0/4017028187 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ff2e406cc30 msgr2=0x7ff2e406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:44.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.279+0000 7ff304a00700 1 --2- 172.21.15.123:0/4017028187 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ff2e406cc30 0x7ff2e406f0e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec007fd0 tx=0x7ff2ec009040 comp rx=0 tx=0).stop 2024-03-30T04:49:44.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.279+0000 7ff304a00700 1 -- 172.21.15.123:0/4017028187 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff300102e00 msgr2=0x7ff300113b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:44.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.279+0000 7ff304a00700 1 --2- 172.21.15.123:0/4017028187 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff300102e00 0x7ff300113b20 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7ff2f0009390 tx=0x7ff2f0009e30 comp rx=0 tx=0).stop 2024-03-30T04:49:44.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.279+0000 7ff304a00700 1 -- 172.21.15.123:0/4017028187 shutdown_connections 2024-03-30T04:49:44.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.279+0000 7ff304a00700 1 --2- 172.21.15.123:0/4017028187 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ff2e406cc30 0x7ff2e406f0e0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:44.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.279+0000 7ff304a00700 1 --2- 172.21.15.123:0/4017028187 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff300102e00 0x7ff300113b20 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:44.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.279+0000 7ff304a00700 1 --2- 172.21.15.123:0/4017028187 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff3001037a0 0x7ff300114060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:44.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.279+0000 7ff304a00700 1 -- 172.21.15.123:0/4017028187 >> 172.21.15.123:0/4017028187 conn(0x7ff3000f89b0 msgr2=0x7ff3000f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:49:44.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.279+0000 7ff304a00700 1 -- 172.21.15.123:0/4017028187 shutdown_connections 2024-03-30T04:49:44.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.279+0000 7ff304a00700 1 -- 172.21.15.123:0/4017028187 wait complete. 2024-03-30T04:49:44.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.412+0000 7f4450000700 1 -- 172.21.15.123:0/896820736 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44481037a0 msgr2=0x7f44480fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:44.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.412+0000 7f4450000700 1 --2- 172.21.15.123:0/896820736 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44481037a0 0x7f44480fce10 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f4434004940 tx=0x7f4434009e30 comp rx=0 tx=0).stop 2024-03-30T04:49:44.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.412+0000 7f4450000700 1 -- 172.21.15.123:0/896820736 shutdown_connections 2024-03-30T04:49:44.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.412+0000 7f4450000700 1 --2- 172.21.15.123:0/896820736 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44481037a0 0x7f44480fce10 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:44.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.412+0000 7f4450000700 1 --2- 172.21.15.123:0/896820736 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4448102e00 0x7f44481031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:44.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.412+0000 7f4450000700 1 -- 172.21.15.123:0/896820736 >> 172.21.15.123:0/896820736 conn(0x7f44480f89b0 msgr2=0x7f44480fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:49:44.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.412+0000 7f4450000700 1 -- 172.21.15.123:0/896820736 shutdown_connections 2024-03-30T04:49:44.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.412+0000 7f4450000700 1 -- 172.21.15.123:0/896820736 wait complete. 2024-03-30T04:49:44.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.413+0000 7f4450000700 1 Processor -- start 2024-03-30T04:49:44.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.413+0000 7f4450000700 1 -- start start 2024-03-30T04:49:44.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.413+0000 7f4450000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4448102e00 0x7f4448113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:49:44.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.413+0000 7f4450000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44481037a0 0x7f4448114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:49:44.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.413+0000 7f4450000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4448114690 con 0x7f44481037a0 2024-03-30T04:49:44.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.413+0000 7f4450000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f444810da80 con 0x7f4448102e00 2024-03-30T04:49:44.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.413+0000 7f444cc00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44481037a0 0x7f4448114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:49:44.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.413+0000 7f444cc00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44481037a0 0x7f4448114040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39128/0 (socket says 172.21.15.123:39128) 2024-03-30T04:49:44.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.413+0000 7f444cc00700 1 -- 172.21.15.123:0/3431662395 learned_addr learned my addr 172.21.15.123:0/3431662395 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:49:44.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.414+0000 7f444d600700 1 --2- 172.21.15.123:0/3431662395 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4448102e00 0x7f4448113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:49:44.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.414+0000 7f444cc00700 1 -- 172.21.15.123:0/3431662395 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4448102e00 msgr2=0x7f4448113b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:44.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.414+0000 7f444cc00700 1 --2- 172.21.15.123:0/3431662395 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4448102e00 0x7f4448113b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:44.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.414+0000 7f444cc00700 1 -- 172.21.15.123:0/3431662395 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4434005040 con 0x7f44481037a0 2024-03-30T04:49:44.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.414+0000 7f444d600700 1 --2- 172.21.15.123:0/3431662395 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4448102e00 0x7f4448113b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:49:44.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.414+0000 7f444cc00700 1 --2- 172.21.15.123:0/3431662395 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44481037a0 0x7f4448114040 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f4434009380 tx=0x7f4434009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:49:44.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.414+0000 7f4445600700 1 -- 172.21.15.123:0/3431662395 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f443400e040 con 0x7f44481037a0 2024-03-30T04:49:44.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.414+0000 7f4445600700 1 -- 172.21.15.123:0/3431662395 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4434013420 con 0x7f44481037a0 2024-03-30T04:49:44.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.414+0000 7f4450000700 1 -- 172.21.15.123:0/3431662395 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f444810dd00 con 0x7f44481037a0 2024-03-30T04:49:44.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.414+0000 7f4445600700 1 -- 172.21.15.123:0/3431662395 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f443401c630 con 0x7f44481037a0 2024-03-30T04:49:44.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.414+0000 7f4450000700 1 -- 172.21.15.123:0/3431662395 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f444810e2c0 con 0x7f44481037a0 2024-03-30T04:49:44.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.416+0000 7f4450000700 1 -- 172.21.15.123:0/3431662395 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44480fe550 con 0x7f44481037a0 2024-03-30T04:49:44.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.416+0000 7f4445600700 1 -- 172.21.15.123:0/3431662395 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f44340139e0 con 0x7f44481037a0 2024-03-30T04:49:44.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.417+0000 7f4445600700 1 --2- 172.21.15.123:0/3431662395 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f443006cc20 0x7f443006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:49:44.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.417+0000 7f4445600700 1 -- 172.21.15.123:0/3431662395 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f443408d2a0 con 0x7f44481037a0 2024-03-30T04:49:44.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.418+0000 7f444d600700 1 --2- 172.21.15.123:0/3431662395 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f443006cc20 0x7f443006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:49:44.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.418+0000 7f444d600700 1 --2- 172.21.15.123:0/3431662395 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f443006cc20 0x7f443006f0d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f443c009fd0 tx=0x7f443c009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:49:44.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.422+0000 7f4445600700 1 -- 172.21.15.123:0/3431662395 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f443405acf0 con 0x7f44481037a0 2024-03-30T04:49:44.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.672+0000 7f4450000700 1 -- 172.21.15.123:0/3431662395 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f4448109da0 con 0x7f44481037a0 2024-03-30T04:49:44.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.673+0000 7f4445600700 1 -- 172.21.15.123:0/3431662395 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f443405a880 con 0x7f44481037a0 2024-03-30T04:49:44.671 INFO:teuthology.orchestra.run.smithi123.stdout:e32 2024-03-30T04:49:44.671 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:49:44.671 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:49:44.671 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:49:44.671 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:49:44.672 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:49:44.672 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:49:44.672 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T04:49:44.672 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T04:49:44.672 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:49:44.672 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T04:49:44.672 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:49:44.672 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:49:44.672 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:49:44.672 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:49:44.672 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:49:44.672 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T04:49:44.672 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:49:44.672 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:49:44.672 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T04:49:44.673 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:49:44.673 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T04:49:44.673 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T04:49:44.673 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T04:49:44.673 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T04:49:44.673 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:49:44.673 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:49:44.673 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:49:44.673 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:49:44.673 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:49:44.673 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:49:44.673 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T04:49:44.673 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T04:49:44.673 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:49:44.674 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:49:44.674 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T04:49:44.674 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:49:44.674 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:49:44.674 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:49:44.674 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44315} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/1741352336,v1:172.21.15.135:6827/1741352336] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:49:44.674 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:49:44.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.675+0000 7f4450000700 1 -- 172.21.15.123:0/3431662395 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f443006cc20 msgr2=0x7f443006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:44.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.675+0000 7f4450000700 1 --2- 172.21.15.123:0/3431662395 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f443006cc20 0x7f443006f0d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f443c009fd0 tx=0x7f443c009ee0 comp rx=0 tx=0).stop 2024-03-30T04:49:44.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.675+0000 7f4450000700 1 -- 172.21.15.123:0/3431662395 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44481037a0 msgr2=0x7f4448114040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:44.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.675+0000 7f4450000700 1 --2- 172.21.15.123:0/3431662395 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44481037a0 0x7f4448114040 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f4434009380 tx=0x7f4434009e30 comp rx=0 tx=0).stop 2024-03-30T04:49:44.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.675+0000 7f4450000700 1 -- 172.21.15.123:0/3431662395 shutdown_connections 2024-03-30T04:49:44.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.675+0000 7f4450000700 1 --2- 172.21.15.123:0/3431662395 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f443006cc20 0x7f443006f0d0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:44.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.675+0000 7f4450000700 1 --2- 172.21.15.123:0/3431662395 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4448102e00 0x7f4448113b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:44.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.675+0000 7f4450000700 1 --2- 172.21.15.123:0/3431662395 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44481037a0 0x7f4448114040 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:44.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.675+0000 7f4450000700 1 -- 172.21.15.123:0/3431662395 >> 172.21.15.123:0/3431662395 conn(0x7f44480f89b0 msgr2=0x7f44480f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:49:44.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.675+0000 7f4450000700 1 -- 172.21.15.123:0/3431662395 shutdown_connections 2024-03-30T04:49:44.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.675+0000 7f4450000700 1 -- 172.21.15.123:0/3431662395 wait complete. 2024-03-30T04:49:44.676 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 32 2024-03-30T04:49:44.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.809+0000 7f154f000700 1 -- 172.21.15.123:0/3706293958 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1548105810 msgr2=0x7f15480feed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:44.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.809+0000 7f154f000700 1 --2- 172.21.15.123:0/3706293958 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1548105810 0x7f15480feed0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f1534004940 tx=0x7f1534009e30 comp rx=0 tx=0).stop 2024-03-30T04:49:44.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.809+0000 7f154f000700 1 -- 172.21.15.123:0/3706293958 shutdown_connections 2024-03-30T04:49:44.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.809+0000 7f154f000700 1 --2- 172.21.15.123:0/3706293958 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1548105810 0x7f15480feed0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:44.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.809+0000 7f154f000700 1 --2- 172.21.15.123:0/3706293958 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1548104e70 0x7f1548105240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:44.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.809+0000 7f154f000700 1 -- 172.21.15.123:0/3706293958 >> 172.21.15.123:0/3706293958 conn(0x7f15480fab10 msgr2=0x7f15480fcef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:49:44.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.809+0000 7f154f000700 1 -- 172.21.15.123:0/3706293958 shutdown_connections 2024-03-30T04:49:44.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.809+0000 7f154f000700 1 -- 172.21.15.123:0/3706293958 wait complete. 2024-03-30T04:49:44.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.810+0000 7f154f000700 1 Processor -- start 2024-03-30T04:49:44.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.810+0000 7f154f000700 1 -- start start 2024-03-30T04:49:44.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.811+0000 7f154f000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1548104e70 0x7f1548115ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:49:44.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.811+0000 7f154f000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1548105810 0x7f1548115ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:49:44.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.811+0000 7f154f000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1548116640 con 0x7f1548104e70 2024-03-30T04:49:44.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.811+0000 7f154f000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f154810fa30 con 0x7f1548105810 2024-03-30T04:49:44.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.811+0000 7f154dc00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1548104e70 0x7f1548115ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:49:44.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.811+0000 7f154dc00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1548104e70 0x7f1548115ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39154/0 (socket says 172.21.15.123:39154) 2024-03-30T04:49:44.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.811+0000 7f154dc00700 1 -- 172.21.15.123:0/1611607062 learned_addr learned my addr 172.21.15.123:0/1611607062 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:49:44.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.811+0000 7f154d200700 1 --2- 172.21.15.123:0/1611607062 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1548105810 0x7f1548115ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:49:44.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.811+0000 7f154dc00700 1 -- 172.21.15.123:0/1611607062 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1548105810 msgr2=0x7f1548115ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:44.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.811+0000 7f154dc00700 1 --2- 172.21.15.123:0/1611607062 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1548105810 0x7f1548115ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:44.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.811+0000 7f154dc00700 1 -- 172.21.15.123:0/1611607062 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1534005040 con 0x7f1548104e70 2024-03-30T04:49:44.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.811+0000 7f154dc00700 1 --2- 172.21.15.123:0/1611607062 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1548104e70 0x7f1548115ab0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f153c002a50 tx=0x7f153c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:49:44.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.812+0000 7f1545600700 1 -- 172.21.15.123:0/1611607062 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f153c007cc0 con 0x7f1548104e70 2024-03-30T04:49:44.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.812+0000 7f1545600700 1 -- 172.21.15.123:0/1611607062 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f153c00f040 con 0x7f1548104e70 2024-03-30T04:49:44.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.812+0000 7f154f000700 1 -- 172.21.15.123:0/1611607062 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f154810fd10 con 0x7f1548104e70 2024-03-30T04:49:44.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.812+0000 7f1545600700 1 -- 172.21.15.123:0/1611607062 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f153c009540 con 0x7f1548104e70 2024-03-30T04:49:44.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.812+0000 7f154f000700 1 -- 172.21.15.123:0/1611607062 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15481102d0 con 0x7f1548104e70 2024-03-30T04:49:44.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.813+0000 7f154f000700 1 -- 172.21.15.123:0/1611607062 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15481005c0 con 0x7f1548104e70 2024-03-30T04:49:44.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.814+0000 7f1545600700 1 -- 172.21.15.123:0/1611607062 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f153c0096a0 con 0x7f1548104e70 2024-03-30T04:49:44.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.815+0000 7f1545600700 1 --2- 172.21.15.123:0/1611607062 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f153006cc20 0x7f153006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:49:44.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.815+0000 7f1545600700 1 -- 172.21.15.123:0/1611607062 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f153c08bde0 con 0x7f1548104e70 2024-03-30T04:49:44.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.815+0000 7f154d200700 1 --2- 172.21.15.123:0/1611607062 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f153006cc20 0x7f153006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:49:44.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.815+0000 7f154d200700 1 --2- 172.21.15.123:0/1611607062 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f153006cc20 0x7f153006f0d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f1548110bf0 tx=0x7f1534009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:49:44.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:44.819+0000 7f1545600700 1 -- 172.21.15.123:0/1611607062 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f153c059830 con 0x7f1548104e70 2024-03-30T04:49:44.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:49:44 smithi123 ceph-mon[139369]: from='client.35046 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:49:44.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:49:44 smithi123 ceph-mon[139369]: from='client.35050 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:49:44.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:49:44 smithi123 ceph-mon[139369]: pgmap v556: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:44.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:49:44 smithi123 ceph-mon[139369]: from='client.35054 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:49:44.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:49:44 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/4017028187' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:49:44.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:49:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:49:44.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:49:44 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3431662395' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:49:45.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.049+0000 7f154f000700 1 -- 172.21.15.123:0/1611607062 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f154810be10 con 0x7f153006cc20 2024-03-30T04:49:45.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.051+0000 7f1545600700 1 -- 172.21.15.123:0/1611607062 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f154810be10 con 0x7f153006cc20 2024-03-30T04:49:45.049 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:49:45.049 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:49:45.049 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:49:45.049 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T04:49:45.049 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:49:45.049 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T04:49:45.049 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T04:49:45.050 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T04:49:45.050 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T04:49:45.050 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:49:45.050 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "15/21 daemons upgraded", 2024-03-30T04:49:45.050 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T04:49:45.050 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:49:45.050 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:49:45.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.053+0000 7f154f000700 1 -- 172.21.15.123:0/1611607062 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f153006cc20 msgr2=0x7f153006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:45.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.053+0000 7f154f000700 1 --2- 172.21.15.123:0/1611607062 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f153006cc20 0x7f153006f0d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f1548110bf0 tx=0x7f1534009e30 comp rx=0 tx=0).stop 2024-03-30T04:49:45.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.053+0000 7f154f000700 1 -- 172.21.15.123:0/1611607062 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1548104e70 msgr2=0x7f1548115ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:45.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.053+0000 7f154f000700 1 --2- 172.21.15.123:0/1611607062 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1548104e70 0x7f1548115ab0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f153c002a50 tx=0x7f153c0046b0 comp rx=0 tx=0).stop 2024-03-30T04:49:45.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.053+0000 7f154f000700 1 -- 172.21.15.123:0/1611607062 shutdown_connections 2024-03-30T04:49:45.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.053+0000 7f154f000700 1 --2- 172.21.15.123:0/1611607062 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f153006cc20 0x7f153006f0d0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:45.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.053+0000 7f154f000700 1 --2- 172.21.15.123:0/1611607062 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1548104e70 0x7f1548115ab0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:45.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.053+0000 7f154f000700 1 --2- 172.21.15.123:0/1611607062 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1548105810 0x7f1548115ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:45.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.053+0000 7f154f000700 1 -- 172.21.15.123:0/1611607062 >> 172.21.15.123:0/1611607062 conn(0x7f15480fab10 msgr2=0x7f15480fcaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:49:45.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.053+0000 7f154f000700 1 -- 172.21.15.123:0/1611607062 shutdown_connections 2024-03-30T04:49:45.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.053+0000 7f154f000700 1 -- 172.21.15.123:0/1611607062 wait complete. 2024-03-30T04:49:45.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:49:44 smithi135 ceph-mon[106011]: from='client.35046 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:49:45.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:49:44 smithi135 ceph-mon[106011]: from='client.35050 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:49:45.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:49:44 smithi135 ceph-mon[106011]: pgmap v556: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:45.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:49:44 smithi135 ceph-mon[106011]: from='client.35054 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:49:45.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:49:44 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/4017028187' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:49:45.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:49:44 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:49:45.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:49:44 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3431662395' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:49:45.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.187+0000 7fb87d800700 1 -- 172.21.15.123:0/164177407 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8780fe830 msgr2=0x7fb878106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:45.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.187+0000 7fb87d800700 1 --2- 172.21.15.123:0/164177407 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8780fe830 0x7fb878106f00 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fb864004940 tx=0x7fb864009e30 comp rx=0 tx=0).stop 2024-03-30T04:49:45.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.187+0000 7fb87d800700 1 -- 172.21.15.123:0/164177407 shutdown_connections 2024-03-30T04:49:45.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.187+0000 7fb87d800700 1 --2- 172.21.15.123:0/164177407 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8780fe830 0x7fb878106f00 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:45.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.187+0000 7fb87d800700 1 --2- 172.21.15.123:0/164177407 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb8780fde90 0x7fb8780fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:45.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.187+0000 7fb87d800700 1 -- 172.21.15.123:0/164177407 >> 172.21.15.123:0/164177407 conn(0x7fb878072bc0 msgr2=0x7fb878072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:49:45.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.188+0000 7fb87d800700 1 -- 172.21.15.123:0/164177407 shutdown_connections 2024-03-30T04:49:45.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.188+0000 7fb87d800700 1 -- 172.21.15.123:0/164177407 wait complete. 2024-03-30T04:49:45.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.188+0000 7fb87d800700 1 Processor -- start 2024-03-30T04:49:45.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.189+0000 7fb87d800700 1 -- start start 2024-03-30T04:49:45.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.189+0000 7fb87d800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb8780fde90 0x7fb8780fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:49:45.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.189+0000 7fb87d800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8780fe830 0x7fb8780fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:49:45.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.189+0000 7fb87d800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8780fc820 con 0x7fb8780fe830 2024-03-30T04:49:45.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.189+0000 7fb87d800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8780fc960 con 0x7fb8780fde90 2024-03-30T04:49:45.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.189+0000 7fb876000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8780fe830 0x7fb8780fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:49:45.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.189+0000 7fb876000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8780fe830 0x7fb8780fc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39172/0 (socket says 172.21.15.123:39172) 2024-03-30T04:49:45.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.189+0000 7fb876000700 1 -- 172.21.15.123:0/3536710691 learned_addr learned my addr 172.21.15.123:0/3536710691 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:49:45.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.189+0000 7fb876a00700 1 --2- 172.21.15.123:0/3536710691 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb8780fde90 0x7fb8780fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:49:45.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.190+0000 7fb876000700 1 -- 172.21.15.123:0/3536710691 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb8780fde90 msgr2=0x7fb8780fbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:45.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.190+0000 7fb876000700 1 --2- 172.21.15.123:0/3536710691 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb8780fde90 0x7fb8780fbd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:45.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.190+0000 7fb876000700 1 -- 172.21.15.123:0/3536710691 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb864005040 con 0x7fb8780fe830 2024-03-30T04:49:45.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.190+0000 7fb876a00700 1 --2- 172.21.15.123:0/3536710691 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb8780fde90 0x7fb8780fbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:49:45.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.190+0000 7fb876000700 1 --2- 172.21.15.123:0/3536710691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8780fe830 0x7fb8780fc290 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fb864000c00 tx=0x7fb864009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:49:45.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.190+0000 7fb86ea00700 1 -- 172.21.15.123:0/3536710691 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb86400e040 con 0x7fb8780fe830 2024-03-30T04:49:45.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.190+0000 7fb86ea00700 1 -- 172.21.15.123:0/3536710691 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb864013420 con 0x7fb8780fe830 2024-03-30T04:49:45.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.190+0000 7fb87d800700 1 -- 172.21.15.123:0/3536710691 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8780fcbb0 con 0x7fb8780fe830 2024-03-30T04:49:45.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.190+0000 7fb86ea00700 1 -- 172.21.15.123:0/3536710691 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb86401c630 con 0x7fb8780fe830 2024-03-30T04:49:45.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.190+0000 7fb87d800700 1 -- 172.21.15.123:0/3536710691 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb87811a190 con 0x7fb8780fe830 2024-03-30T04:49:45.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.191+0000 7fb87d800700 1 -- 172.21.15.123:0/3536710691 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb878065e30 con 0x7fb8780fe830 2024-03-30T04:49:45.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.192+0000 7fb86ea00700 1 -- 172.21.15.123:0/3536710691 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fb864013970 con 0x7fb8780fe830 2024-03-30T04:49:45.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.193+0000 7fb86ea00700 1 --2- 172.21.15.123:0/3536710691 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb85c06cc30 0x7fb85c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:49:45.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.193+0000 7fb86ea00700 1 -- 172.21.15.123:0/3536710691 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb86408d210 con 0x7fb8780fe830 2024-03-30T04:49:45.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.193+0000 7fb876a00700 1 --2- 172.21.15.123:0/3536710691 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb85c06cc30 0x7fb85c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:49:45.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.194+0000 7fb876a00700 1 --2- 172.21.15.123:0/3536710691 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb85c06cc30 0x7fb85c06f0e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb868009fd0 tx=0x7fb868009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:49:45.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.197+0000 7fb86ea00700 1 -- 172.21.15.123:0/3536710691 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb864057250 con 0x7fb8780fe830 2024-03-30T04:49:45.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.482+0000 7fb87d800700 1 -- 172.21.15.123:0/3536710691 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fb878109e00 con 0x7fb8780fe830 2024-03-30T04:49:45.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.483+0000 7fb86ea00700 1 -- 172.21.15.123:0/3536710691 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fb86405a870 con 0x7fb8780fe830 2024-03-30T04:49:45.481 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T04:49:45.481 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T04:49:45.481 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T04:49:45.481 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T04:49:45.481 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T04:49:45.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.485+0000 7fb87d800700 1 -- 172.21.15.123:0/3536710691 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb85c06cc30 msgr2=0x7fb85c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:45.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.485+0000 7fb87d800700 1 --2- 172.21.15.123:0/3536710691 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb85c06cc30 0x7fb85c06f0e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb868009fd0 tx=0x7fb868009ee0 comp rx=0 tx=0).stop 2024-03-30T04:49:45.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.485+0000 7fb87d800700 1 -- 172.21.15.123:0/3536710691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8780fe830 msgr2=0x7fb8780fc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:49:45.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.485+0000 7fb87d800700 1 --2- 172.21.15.123:0/3536710691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8780fe830 0x7fb8780fc290 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fb864000c00 tx=0x7fb864009e30 comp rx=0 tx=0).stop 2024-03-30T04:49:45.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.485+0000 7fb87d800700 1 -- 172.21.15.123:0/3536710691 shutdown_connections 2024-03-30T04:49:45.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.485+0000 7fb87d800700 1 --2- 172.21.15.123:0/3536710691 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb85c06cc30 0x7fb85c06f0e0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:45.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.485+0000 7fb87d800700 1 --2- 172.21.15.123:0/3536710691 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb8780fde90 0x7fb8780fbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:45.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.485+0000 7fb87d800700 1 --2- 172.21.15.123:0/3536710691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8780fe830 0x7fb8780fc290 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:49:45.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.485+0000 7fb87d800700 1 -- 172.21.15.123:0/3536710691 >> 172.21.15.123:0/3536710691 conn(0x7fb878072bc0 msgr2=0x7fb878103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:49:45.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.485+0000 7fb87d800700 1 -- 172.21.15.123:0/3536710691 shutdown_connections 2024-03-30T04:49:45.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:49:45.485+0000 7fb87d800700 1 -- 172.21.15.123:0/3536710691 wait complete. 2024-03-30T04:49:45.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:49:45 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3536710691' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:49:46.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:49:45 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3536710691' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:49:46.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:49:46 smithi123 ceph-mon[139369]: from='client.35066 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:49:46.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:49:46 smithi123 ceph-mon[139369]: pgmap v557: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:47.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:49:46 smithi135 ceph-mon[106011]: from='client.35066 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:49:47.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:49:46 smithi135 ceph-mon[106011]: pgmap v557: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:48.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:49:48 smithi123 ceph-mon[139369]: pgmap v558: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:49.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:49:48 smithi135 ceph-mon[106011]: pgmap v558: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:50.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:49:50 smithi135 ceph-mon[106011]: pgmap v559: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:50.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:49:50 smithi123 ceph-mon[139369]: pgmap v559: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:52.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:49:52 smithi123 ceph-mon[139369]: pgmap v560: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:53.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:49:52 smithi135 ceph-mon[106011]: pgmap v560: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:54.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:49:54 smithi123 ceph-mon[139369]: pgmap v561: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:55.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:49:54 smithi135 ceph-mon[106011]: pgmap v561: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:56.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:49:56 smithi123 ceph-mon[139369]: pgmap v562: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:57.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:49:56 smithi135 ceph-mon[106011]: pgmap v562: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:58.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:49:58 smithi123 ceph-mon[139369]: pgmap v563: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:59.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:49:58 smithi135 ceph-mon[106011]: pgmap v563: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:49:59.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:49:59 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:50:00.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:49:59 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:50:00.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:50:00 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-mon-smithi123[139346]: 2024-03-30T04:49:59.999+0000 7f50e7e00700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T04:50:00.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:50:00 smithi123 ceph-mon[139369]: pgmap v564: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:00.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:50:00 smithi123 ceph-mon[139369]: overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T04:50:01.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:50:00 smithi135 ceph-mon[106011]: pgmap v564: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:01.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:50:00 smithi135 ceph-mon[106011]: overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T04:50:02.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:50:02 smithi123 ceph-mon[139369]: pgmap v565: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:03.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:50:02 smithi135 ceph-mon[106011]: pgmap v565: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:04.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:50:04 smithi123 ceph-mon[139369]: pgmap v566: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:05.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:50:04 smithi135 ceph-mon[106011]: pgmap v566: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:06.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:50:06 smithi123 ceph-mon[139369]: pgmap v567: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:07.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:50:06 smithi135 ceph-mon[106011]: pgmap v567: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:09.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:50:08 smithi135 ceph-mon[106011]: pgmap v568: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:09.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:50:08 smithi123 ceph-mon[139369]: pgmap v568: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:10.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:50:10 smithi135 ceph-mon[106011]: pgmap v569: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:10.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:50:10 smithi123 ceph-mon[139369]: pgmap v569: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:12.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:50:12 smithi123 ceph-mon[139369]: pgmap v570: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:13.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:50:12 smithi135 ceph-mon[106011]: pgmap v570: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:14.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:50:14 smithi123 ceph-mon[139369]: pgmap v571: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:14.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:50:14 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:50:15.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:50:14 smithi135 ceph-mon[106011]: pgmap v571: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:15.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:50:14 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:50:15.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:15.632+0000 7fbcb1a00700 1 -- 172.21.15.123:0/4060329283 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbcac102e00 msgr2=0x7fbcac1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:15.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:15.632+0000 7fbcb1a00700 1 --2- 172.21.15.123:0/4060329283 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbcac102e00 0x7fbcac1031d0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c004940 tx=0x7fbc9c009e30 comp rx=0 tx=0).stop 2024-03-30T04:50:15.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:15.632+0000 7fbcb1a00700 1 -- 172.21.15.123:0/4060329283 shutdown_connections 2024-03-30T04:50:15.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:15.632+0000 7fbcb1a00700 1 --2- 172.21.15.123:0/4060329283 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbcac1037a0 0x7fbcac0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:15.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:15.632+0000 7fbcb1a00700 1 --2- 172.21.15.123:0/4060329283 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbcac102e00 0x7fbcac1031d0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:15.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:15.632+0000 7fbcb1a00700 1 -- 172.21.15.123:0/4060329283 >> 172.21.15.123:0/4060329283 conn(0x7fbcac0f89b0 msgr2=0x7fbcac0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:50:15.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:15.632+0000 7fbcb1a00700 1 -- 172.21.15.123:0/4060329283 shutdown_connections 2024-03-30T04:50:15.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:15.632+0000 7fbcb1a00700 1 -- 172.21.15.123:0/4060329283 wait complete. 2024-03-30T04:50:15.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:15.633+0000 7fbcb1a00700 1 Processor -- start 2024-03-30T04:50:15.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:15.633+0000 7fbcb1a00700 1 -- start start 2024-03-30T04:50:15.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:15.634+0000 7fbcb1a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbcac102e00 0x7fbcac115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:50:15.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:15.634+0000 7fbcb1a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbcac1037a0 0x7fbcac116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:50:15.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:15.634+0000 7fbcb1a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcac116880 con 0x7fbcac102e00 2024-03-30T04:50:15.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:15.634+0000 7fbcb1a00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcac10fc50 con 0x7fbcac1037a0 2024-03-30T04:50:15.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:15.634+0000 7fbcaaa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbcac102e00 0x7fbcac115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:50:15.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:15.634+0000 7fbcaaa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbcac102e00 0x7fbcac115cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33568/0 (socket says 172.21.15.123:33568) 2024-03-30T04:50:15.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:15.634+0000 7fbcaaa00700 1 -- 172.21.15.123:0/116609819 learned_addr learned my addr 172.21.15.123:0/116609819 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:50:15.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:15.634+0000 7fbcaa000700 1 --2- 172.21.15.123:0/116609819 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbcac1037a0 0x7fbcac116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:50:15.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:15.634+0000 7fbcaaa00700 1 -- 172.21.15.123:0/116609819 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbcac1037a0 msgr2=0x7fbcac116230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:15.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:15.634+0000 7fbcaaa00700 1 --2- 172.21.15.123:0/116609819 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbcac1037a0 0x7fbcac116230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:15.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:15.634+0000 7fbcaaa00700 1 -- 172.21.15.123:0/116609819 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc9c005040 con 0x7fbcac102e00 2024-03-30T04:50:15.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:15.635+0000 7fbcaaa00700 1 --2- 172.21.15.123:0/116609819 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbcac102e00 0x7fbcac115cf0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c009390 tx=0x7fbc9c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:50:15.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:15.635+0000 7fbca2a00700 1 -- 172.21.15.123:0/116609819 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc9c00e040 con 0x7fbcac102e00 2024-03-30T04:50:15.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:15.635+0000 7fbca2a00700 1 -- 172.21.15.123:0/116609819 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbc9c013420 con 0x7fbcac102e00 2024-03-30T04:50:15.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:15.635+0000 7fbcb1a00700 1 -- 172.21.15.123:0/116609819 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcac10fed0 con 0x7fbcac102e00 2024-03-30T04:50:15.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:15.635+0000 7fbca2a00700 1 -- 172.21.15.123:0/116609819 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc9c01c630 con 0x7fbcac102e00 2024-03-30T04:50:15.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:15.635+0000 7fbcb1a00700 1 -- 172.21.15.123:0/116609819 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcac110490 con 0x7fbcac102e00 2024-03-30T04:50:15.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:15.636+0000 7fbcb1a00700 1 -- 172.21.15.123:0/116609819 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcac0fe550 con 0x7fbcac102e00 2024-03-30T04:50:15.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:15.637+0000 7fbca2a00700 1 -- 172.21.15.123:0/116609819 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fbc9c0139e0 con 0x7fbcac102e00 2024-03-30T04:50:15.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:15.638+0000 7fbca2a00700 1 --2- 172.21.15.123:0/116609819 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbc9006cc30 0x7fbc9006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:50:15.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:15.638+0000 7fbca2a00700 1 -- 172.21.15.123:0/116609819 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fbc9c08de80 con 0x7fbcac102e00 2024-03-30T04:50:15.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:15.638+0000 7fbcaa000700 1 --2- 172.21.15.123:0/116609819 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbc9006cc30 0x7fbc9006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:50:15.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:15.639+0000 7fbcaa000700 1 --2- 172.21.15.123:0/116609819 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbc9006cc30 0x7fbc9006f0e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fbcac111590 tx=0x7fbc9800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:50:15.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:15.642+0000 7fbca2a00700 1 -- 172.21.15.123:0/116609819 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fbc9c05b950 con 0x7fbcac102e00 2024-03-30T04:50:15.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:15.863+0000 7fbcb1a00700 1 -- 172.21.15.123:0/116609819 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbcac04bf70 con 0x7fbc9006cc30 2024-03-30T04:50:15.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:15.864+0000 7fbca2a00700 1 -- 172.21.15.123:0/116609819 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fbc9800f040 con 0x7fbc9006cc30 2024-03-30T04:50:15.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:15.867+0000 7fbcb1a00700 1 -- 172.21.15.123:0/116609819 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbc9006cc30 msgr2=0x7fbc9006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:15.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:15.867+0000 7fbcb1a00700 1 --2- 172.21.15.123:0/116609819 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbc9006cc30 0x7fbc9006f0e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fbcac111590 tx=0x7fbc9800b040 comp rx=0 tx=0).stop 2024-03-30T04:50:15.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:15.867+0000 7fbcb1a00700 1 -- 172.21.15.123:0/116609819 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbcac102e00 msgr2=0x7fbcac115cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:15.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:15.867+0000 7fbcb1a00700 1 --2- 172.21.15.123:0/116609819 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbcac102e00 0x7fbcac115cf0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c009390 tx=0x7fbc9c009e30 comp rx=0 tx=0).stop 2024-03-30T04:50:15.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:15.867+0000 7fbcb1a00700 1 -- 172.21.15.123:0/116609819 shutdown_connections 2024-03-30T04:50:15.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:15.867+0000 7fbcb1a00700 1 --2- 172.21.15.123:0/116609819 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbc9006cc30 0x7fbc9006f0e0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:15.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:15.867+0000 7fbcb1a00700 1 --2- 172.21.15.123:0/116609819 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbcac102e00 0x7fbcac115cf0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:15.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:15.867+0000 7fbcb1a00700 1 --2- 172.21.15.123:0/116609819 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbcac1037a0 0x7fbcac116230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:15.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:15.867+0000 7fbcb1a00700 1 -- 172.21.15.123:0/116609819 >> 172.21.15.123:0/116609819 conn(0x7fbcac0f89b0 msgr2=0x7fbcac0706d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:50:15.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:15.867+0000 7fbcb1a00700 1 -- 172.21.15.123:0/116609819 shutdown_connections 2024-03-30T04:50:15.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:15.867+0000 7fbcb1a00700 1 -- 172.21.15.123:0/116609819 wait complete. 2024-03-30T04:50:15.879 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:50:16.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.001+0000 7f87f9e00700 1 -- 172.21.15.123:0/726385569 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87f41037a0 msgr2=0x7f87f40fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:16.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.001+0000 7f87f9e00700 1 --2- 172.21.15.123:0/726385569 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87f41037a0 0x7f87f40fce10 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f87e0004940 tx=0x7f87e0009e30 comp rx=0 tx=0).stop 2024-03-30T04:50:16.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.001+0000 7f87f9e00700 1 -- 172.21.15.123:0/726385569 shutdown_connections 2024-03-30T04:50:16.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.001+0000 7f87f9e00700 1 --2- 172.21.15.123:0/726385569 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87f41037a0 0x7f87f40fce10 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:16.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.001+0000 7f87f9e00700 1 --2- 172.21.15.123:0/726385569 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f87f4102e00 0x7f87f41031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:16.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.001+0000 7f87f9e00700 1 -- 172.21.15.123:0/726385569 >> 172.21.15.123:0/726385569 conn(0x7f87f40f89b0 msgr2=0x7f87f40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:50:16.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.001+0000 7f87f9e00700 1 -- 172.21.15.123:0/726385569 shutdown_connections 2024-03-30T04:50:16.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.001+0000 7f87f9e00700 1 -- 172.21.15.123:0/726385569 wait complete. 2024-03-30T04:50:16.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.002+0000 7f87f9e00700 1 Processor -- start 2024-03-30T04:50:16.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.002+0000 7f87f9e00700 1 -- start start 2024-03-30T04:50:16.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.003+0000 7f87f9e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87f4102e00 0x7f87f40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:50:16.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.003+0000 7f87f9e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f87f41037a0 0x7f87f4070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:50:16.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.003+0000 7f87f9e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f4070d90 con 0x7f87f4102e00 2024-03-30T04:50:16.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.003+0000 7f87f9e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f4070ed0 con 0x7f87f41037a0 2024-03-30T04:50:16.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.003+0000 7f87f2a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87f4102e00 0x7f87f40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:50:16.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.003+0000 7f87f2a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87f4102e00 0x7f87f40702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33590/0 (socket says 172.21.15.123:33590) 2024-03-30T04:50:16.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.003+0000 7f87f2a00700 1 -- 172.21.15.123:0/839358794 learned_addr learned my addr 172.21.15.123:0/839358794 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:50:16.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.003+0000 7f87f2000700 1 --2- 172.21.15.123:0/839358794 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f87f41037a0 0x7f87f4070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:50:16.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.003+0000 7f87f2a00700 1 -- 172.21.15.123:0/839358794 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f87f41037a0 msgr2=0x7f87f4070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:16.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.003+0000 7f87f2a00700 1 --2- 172.21.15.123:0/839358794 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f87f41037a0 0x7f87f4070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:16.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.003+0000 7f87f2a00700 1 -- 172.21.15.123:0/839358794 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e0005040 con 0x7f87f4102e00 2024-03-30T04:50:16.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.003+0000 7f87f2000700 1 --2- 172.21.15.123:0/839358794 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f87f41037a0 0x7f87f4070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:50:16.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.004+0000 7f87f2a00700 1 --2- 172.21.15.123:0/839358794 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87f4102e00 0x7f87f40702c0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f87e4002a50 tx=0x7f87e40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:50:16.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.004+0000 7f87eaa00700 1 -- 172.21.15.123:0/839358794 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f87e4007cc0 con 0x7f87f4102e00 2024-03-30T04:50:16.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.004+0000 7f87eaa00700 1 -- 172.21.15.123:0/839358794 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f87e400f040 con 0x7f87f4102e00 2024-03-30T04:50:16.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.004+0000 7f87f9e00700 1 -- 172.21.15.123:0/839358794 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87f4119e00 con 0x7f87f4102e00 2024-03-30T04:50:16.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.004+0000 7f87eaa00700 1 -- 172.21.15.123:0/839358794 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f87e4009540 con 0x7f87f4102e00 2024-03-30T04:50:16.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.004+0000 7f87f9e00700 1 -- 172.21.15.123:0/839358794 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87f411a280 con 0x7f87f4102e00 2024-03-30T04:50:16.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.005+0000 7f87f9e00700 1 -- 172.21.15.123:0/839358794 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87f40fe550 con 0x7f87f4102e00 2024-03-30T04:50:16.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.006+0000 7f87eaa00700 1 -- 172.21.15.123:0/839358794 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f87e40096a0 con 0x7f87f4102e00 2024-03-30T04:50:16.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.007+0000 7f87eaa00700 1 --2- 172.21.15.123:0/839358794 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f87d806cc30 0x7f87d806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:50:16.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.007+0000 7f87eaa00700 1 -- 172.21.15.123:0/839358794 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f87e408bed0 con 0x7f87f4102e00 2024-03-30T04:50:16.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.007+0000 7f87f2000700 1 --2- 172.21.15.123:0/839358794 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f87d806cc30 0x7f87d806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:50:16.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.007+0000 7f87f2000700 1 --2- 172.21.15.123:0/839358794 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f87d806cc30 0x7f87d806f0e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f87e000b3b0 tx=0x7f87e0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:50:16.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.011+0000 7f87eaa00700 1 -- 172.21.15.123:0/839358794 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f87e4055e10 con 0x7f87f4102e00 2024-03-30T04:50:16.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.230+0000 7f87f9e00700 1 -- 172.21.15.123:0/839358794 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f87f40719b0 con 0x7f87d806cc30 2024-03-30T04:50:16.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.231+0000 7f87eaa00700 1 -- 172.21.15.123:0/839358794 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f87f40719b0 con 0x7f87d806cc30 2024-03-30T04:50:16.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.233+0000 7f87f9e00700 1 -- 172.21.15.123:0/839358794 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f87d806cc30 msgr2=0x7f87d806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:16.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.233+0000 7f87f9e00700 1 --2- 172.21.15.123:0/839358794 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f87d806cc30 0x7f87d806f0e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f87e000b3b0 tx=0x7f87e0009e30 comp rx=0 tx=0).stop 2024-03-30T04:50:16.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.233+0000 7f87f9e00700 1 -- 172.21.15.123:0/839358794 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87f4102e00 msgr2=0x7f87f40702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:16.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.233+0000 7f87f9e00700 1 --2- 172.21.15.123:0/839358794 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87f4102e00 0x7f87f40702c0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f87e4002a50 tx=0x7f87e40046b0 comp rx=0 tx=0).stop 2024-03-30T04:50:16.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.233+0000 7f87f9e00700 1 -- 172.21.15.123:0/839358794 shutdown_connections 2024-03-30T04:50:16.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.233+0000 7f87f9e00700 1 --2- 172.21.15.123:0/839358794 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f87d806cc30 0x7f87d806f0e0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:16.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.233+0000 7f87f9e00700 1 --2- 172.21.15.123:0/839358794 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87f4102e00 0x7f87f40702c0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:16.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.233+0000 7f87f9e00700 1 --2- 172.21.15.123:0/839358794 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f87f41037a0 0x7f87f4070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:16.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.233+0000 7f87f9e00700 1 -- 172.21.15.123:0/839358794 >> 172.21.15.123:0/839358794 conn(0x7f87f40f89b0 msgr2=0x7f87f40f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:50:16.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.233+0000 7f87f9e00700 1 -- 172.21.15.123:0/839358794 shutdown_connections 2024-03-30T04:50:16.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.234+0000 7f87f9e00700 1 -- 172.21.15.123:0/839358794 wait complete. 2024-03-30T04:50:16.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.367+0000 7f2c4d000700 1 -- 172.21.15.123:0/789179877 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c481037a0 msgr2=0x7f2c480fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:16.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.367+0000 7f2c4d000700 1 --2- 172.21.15.123:0/789179877 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c481037a0 0x7f2c480fce10 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f2c34004940 tx=0x7f2c34009e30 comp rx=0 tx=0).stop 2024-03-30T04:50:16.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.367+0000 7f2c4d000700 1 -- 172.21.15.123:0/789179877 shutdown_connections 2024-03-30T04:50:16.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.367+0000 7f2c4d000700 1 --2- 172.21.15.123:0/789179877 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c481037a0 0x7f2c480fce10 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:16.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.367+0000 7f2c4d000700 1 --2- 172.21.15.123:0/789179877 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2c48102e00 0x7f2c481031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:16.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.367+0000 7f2c4d000700 1 -- 172.21.15.123:0/789179877 >> 172.21.15.123:0/789179877 conn(0x7f2c480f89b0 msgr2=0x7f2c480fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:50:16.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.367+0000 7f2c4d000700 1 -- 172.21.15.123:0/789179877 shutdown_connections 2024-03-30T04:50:16.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.367+0000 7f2c4d000700 1 -- 172.21.15.123:0/789179877 wait complete. 2024-03-30T04:50:16.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.368+0000 7f2c4d000700 1 Processor -- start 2024-03-30T04:50:16.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.368+0000 7f2c4d000700 1 -- start start 2024-03-30T04:50:16.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.369+0000 7f2c4d000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2c48102e00 0x7f2c480702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:50:16.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.369+0000 7f2c4d000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c481037a0 0x7f2c48070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:50:16.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.369+0000 7f2c4d000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c48070d90 con 0x7f2c481037a0 2024-03-30T04:50:16.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.369+0000 7f2c4d000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c48070ed0 con 0x7f2c48102e00 2024-03-30T04:50:16.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.369+0000 7f2c45400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c481037a0 0x7f2c48070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:50:16.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.369+0000 7f2c45400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c481037a0 0x7f2c48070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33596/0 (socket says 172.21.15.123:33596) 2024-03-30T04:50:16.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.369+0000 7f2c45400700 1 -- 172.21.15.123:0/1019660397 learned_addr learned my addr 172.21.15.123:0/1019660397 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:50:16.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.369+0000 7f2c45e00700 1 --2- 172.21.15.123:0/1019660397 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2c48102e00 0x7f2c480702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:50:16.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.370+0000 7f2c45400700 1 -- 172.21.15.123:0/1019660397 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2c48102e00 msgr2=0x7f2c480702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:16.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.370+0000 7f2c45400700 1 --2- 172.21.15.123:0/1019660397 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2c48102e00 0x7f2c480702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:16.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.370+0000 7f2c45400700 1 -- 172.21.15.123:0/1019660397 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c34005040 con 0x7f2c481037a0 2024-03-30T04:50:16.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.370+0000 7f2c45400700 1 --2- 172.21.15.123:0/1019660397 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c481037a0 0x7f2c48070800 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f2c34009390 tx=0x7f2c34009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:50:16.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.370+0000 7f2c3e000700 1 -- 172.21.15.123:0/1019660397 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c3400e040 con 0x7f2c481037a0 2024-03-30T04:50:16.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.370+0000 7f2c3e000700 1 -- 172.21.15.123:0/1019660397 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2c34013420 con 0x7f2c481037a0 2024-03-30T04:50:16.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.370+0000 7f2c4d000700 1 -- 172.21.15.123:0/1019660397 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c48071120 con 0x7f2c481037a0 2024-03-30T04:50:16.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.370+0000 7f2c3e000700 1 -- 172.21.15.123:0/1019660397 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c3401c630 con 0x7f2c481037a0 2024-03-30T04:50:16.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.370+0000 7f2c4d000700 1 -- 172.21.15.123:0/1019660397 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c4811a230 con 0x7f2c481037a0 2024-03-30T04:50:16.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.371+0000 7f2c4d000700 1 -- 172.21.15.123:0/1019660397 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c480fe550 con 0x7f2c481037a0 2024-03-30T04:50:16.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.372+0000 7f2c3e000700 1 -- 172.21.15.123:0/1019660397 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f2c34013a00 con 0x7f2c481037a0 2024-03-30T04:50:16.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.373+0000 7f2c3e000700 1 --2- 172.21.15.123:0/1019660397 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2c2c06cc30 0x7f2c2c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:50:16.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.373+0000 7f2c3e000700 1 -- 172.21.15.123:0/1019660397 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f2c3408d0e0 con 0x7f2c481037a0 2024-03-30T04:50:16.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.373+0000 7f2c45e00700 1 --2- 172.21.15.123:0/1019660397 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2c2c06cc30 0x7f2c2c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:50:16.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.374+0000 7f2c45e00700 1 --2- 172.21.15.123:0/1019660397 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2c2c06cc30 0x7f2c2c06f0e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f2c38004f40 tx=0x7f2c3800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:50:16.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.377+0000 7f2c3e000700 1 -- 172.21.15.123:0/1019660397 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2c34057120 con 0x7f2c481037a0 2024-03-30T04:50:16.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.590+0000 7f2c4d000700 1 -- 172.21.15.123:0/1019660397 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2c48109e40 con 0x7f2c2c06cc30 2024-03-30T04:50:16.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.600+0000 7f2c3e000700 1 -- 172.21.15.123:0/1019660397 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f2c3800f040 con 0x7f2c2c06cc30 2024-03-30T04:50:16.598 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:50:16.599 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (19m) 12m ago 31m 15.0M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:50:16.599 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (16m) 12m ago 32m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T04:50:16.599 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (16m) 12m ago 31m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T04:50:16.599 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (19m) 12m ago 31m 74.0M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T04:50:16.599 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (12m) 12m ago 25m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T04:50:16.599 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (12m) 12m ago 25m 10.9M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T04:50:16.599 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (25m) 12m ago 25m 10.7M - 17.2.7-797-g5c024335 f77085f6aab7 0dc41326818c 2024-03-30T04:50:16.599 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (12m) 12m ago 25m 9747k - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T04:50:16.599 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (21m) 12m ago 33m 506M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:50:16.599 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (20m) 12m ago 30m 450M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:50:16.599 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (17m) 12m ago 33m 53.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T04:50:16.599 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (17m) 12m ago 30m 42.6M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T04:50:16.599 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (20m) 12m ago 32m 8052k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:50:16.599 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (20m) 12m ago 30m 7880k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:50:16.600 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (16m) 12m ago 29m 104M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T04:50:16.600 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (15m) 12m ago 29m 93.4M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T04:50:16.600 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (15m) 12m ago 28m 86.3M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T04:50:16.600 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (14m) 12m ago 28m 97.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T04:50:16.600 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (13m) 12m ago 27m 97.9M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T04:50:16.600 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (13m) 12m ago 27m 85.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T04:50:16.600 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (19m) 12m ago 31m 53.7M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:50:16.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.602+0000 7f2c4d000700 1 -- 172.21.15.123:0/1019660397 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2c2c06cc30 msgr2=0x7f2c2c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:16.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.602+0000 7f2c4d000700 1 --2- 172.21.15.123:0/1019660397 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2c2c06cc30 0x7f2c2c06f0e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f2c38004f40 tx=0x7f2c3800b040 comp rx=0 tx=0).stop 2024-03-30T04:50:16.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.602+0000 7f2c4d000700 1 -- 172.21.15.123:0/1019660397 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c481037a0 msgr2=0x7f2c48070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:16.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.602+0000 7f2c4d000700 1 --2- 172.21.15.123:0/1019660397 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c481037a0 0x7f2c48070800 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f2c34009390 tx=0x7f2c34009e30 comp rx=0 tx=0).stop 2024-03-30T04:50:16.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.602+0000 7f2c4d000700 1 -- 172.21.15.123:0/1019660397 shutdown_connections 2024-03-30T04:50:16.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.602+0000 7f2c4d000700 1 --2- 172.21.15.123:0/1019660397 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2c2c06cc30 0x7f2c2c06f0e0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:16.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.602+0000 7f2c4d000700 1 --2- 172.21.15.123:0/1019660397 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2c48102e00 0x7f2c480702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:16.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.602+0000 7f2c4d000700 1 --2- 172.21.15.123:0/1019660397 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c481037a0 0x7f2c48070800 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:16.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.602+0000 7f2c4d000700 1 -- 172.21.15.123:0/1019660397 >> 172.21.15.123:0/1019660397 conn(0x7f2c480f89b0 msgr2=0x7f2c480f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:50:16.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.603+0000 7f2c4d000700 1 -- 172.21.15.123:0/1019660397 shutdown_connections 2024-03-30T04:50:16.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.603+0000 7f2c4d000700 1 -- 172.21.15.123:0/1019660397 wait complete. 2024-03-30T04:50:16.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.737+0000 7f3f4d600700 1 -- 172.21.15.123:0/1746388285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f48065630 msgr2=0x7f3f48065a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:16.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.737+0000 7f3f4d600700 1 --2- 172.21.15.123:0/1746388285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f48065630 0x7f3f48065a80 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f3f34004940 tx=0x7f3f34009e30 comp rx=0 tx=0).stop 2024-03-30T04:50:16.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.737+0000 7f3f4d600700 1 -- 172.21.15.123:0/1746388285 shutdown_connections 2024-03-30T04:50:16.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.737+0000 7f3f4d600700 1 --2- 172.21.15.123:0/1746388285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f48065630 0x7f3f48065a80 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:16.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.737+0000 7f3f4d600700 1 --2- 172.21.15.123:0/1746388285 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3f48102ca0 0x7f3f48103070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:16.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.737+0000 7f3f4d600700 1 -- 172.21.15.123:0/1746388285 >> 172.21.15.123:0/1746388285 conn(0x7f3f48071b70 msgr2=0x7f3f48071f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:50:16.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.737+0000 7f3f4d600700 1 -- 172.21.15.123:0/1746388285 shutdown_connections 2024-03-30T04:50:16.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.737+0000 7f3f4d600700 1 -- 172.21.15.123:0/1746388285 wait complete. 2024-03-30T04:50:16.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.738+0000 7f3f4d600700 1 Processor -- start 2024-03-30T04:50:16.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.738+0000 7f3f4d600700 1 -- start start 2024-03-30T04:50:16.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.738+0000 7f3f4d600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f48065630 0x7f3f48115b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:50:16.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.739+0000 7f3f4d600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3f48102ca0 0x7f3f48116070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:50:16.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.739+0000 7f3f4d600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f481166c0 con 0x7f3f48065630 2024-03-30T04:50:16.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.739+0000 7f3f4d600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4810fab0 con 0x7f3f48102ca0 2024-03-30T04:50:16.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.739+0000 7f3f47e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f48065630 0x7f3f48115b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:50:16.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.739+0000 7f3f47e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f48065630 0x7f3f48115b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33612/0 (socket says 172.21.15.123:33612) 2024-03-30T04:50:16.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.739+0000 7f3f47e00700 1 -- 172.21.15.123:0/973664725 learned_addr learned my addr 172.21.15.123:0/973664725 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:50:16.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.739+0000 7f3f47400700 1 --2- 172.21.15.123:0/973664725 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3f48102ca0 0x7f3f48116070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:50:16.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.739+0000 7f3f47e00700 1 -- 172.21.15.123:0/973664725 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3f48102ca0 msgr2=0x7f3f48116070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:16.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.739+0000 7f3f47e00700 1 --2- 172.21.15.123:0/973664725 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3f48102ca0 0x7f3f48116070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:16.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.739+0000 7f3f47e00700 1 -- 172.21.15.123:0/973664725 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f34005040 con 0x7f3f48065630 2024-03-30T04:50:16.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.739+0000 7f3f47e00700 1 --2- 172.21.15.123:0/973664725 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f48065630 0x7f3f48115b30 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f3f3800c830 tx=0x7f3f38004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:50:16.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.740+0000 7f3f3fe00700 1 -- 172.21.15.123:0/973664725 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f3800a760 con 0x7f3f48065630 2024-03-30T04:50:16.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.740+0000 7f3f3fe00700 1 -- 172.21.15.123:0/973664725 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3f3800ad70 con 0x7f3f48065630 2024-03-30T04:50:16.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.740+0000 7f3f4d600700 1 -- 172.21.15.123:0/973664725 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f4810fd90 con 0x7f3f48065630 2024-03-30T04:50:16.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.740+0000 7f3f3fe00700 1 -- 172.21.15.123:0/973664725 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f380113a0 con 0x7f3f48065630 2024-03-30T04:50:16.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.740+0000 7f3f4d600700 1 -- 172.21.15.123:0/973664725 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f481102d0 con 0x7f3f48065630 2024-03-30T04:50:16.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.741+0000 7f3f4d600700 1 -- 172.21.15.123:0/973664725 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f48106840 con 0x7f3f48065630 2024-03-30T04:50:16.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.742+0000 7f3f3fe00700 1 -- 172.21.15.123:0/973664725 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f3f38012400 con 0x7f3f48065630 2024-03-30T04:50:16.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.743+0000 7f3f3fe00700 1 --2- 172.21.15.123:0/973664725 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3f2c06cc30 0x7f3f2c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:50:16.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.743+0000 7f3f3fe00700 1 -- 172.21.15.123:0/973664725 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f3f3808c120 con 0x7f3f48065630 2024-03-30T04:50:16.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.743+0000 7f3f47400700 1 --2- 172.21.15.123:0/973664725 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3f2c06cc30 0x7f3f2c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:50:16.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.743+0000 7f3f47400700 1 --2- 172.21.15.123:0/973664725 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3f2c06cc30 0x7f3f2c06f0e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f3f48110c70 tx=0x7f3f34009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:50:16.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:16.747+0000 7f3f3fe00700 1 -- 172.21.15.123:0/973664725 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3f38059b70 con 0x7f3f48065630 2024-03-30T04:50:16.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:50:16 smithi123 ceph-mon[139369]: pgmap v572: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:16.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:50:16 smithi123 ceph-mon[139369]: from='client.35074 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:50:17.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.032+0000 7f3f4d600700 1 -- 172.21.15.123:0/973664725 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f3f48111180 con 0x7f3f48065630 2024-03-30T04:50:17.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.033+0000 7f3f3fe00700 1 -- 172.21.15.123:0/973664725 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f3f38059700 con 0x7f3f48065630 2024-03-30T04:50:17.031 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:50:17.031 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:50:17.031 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:50:17.031 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:50:17.031 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:50:17.032 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:50:17.032 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:50:17.032 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:50:17.032 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T04:50:17.032 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:50:17.032 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:50:17.032 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:50:17.032 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 3 2024-03-30T04:50:17.032 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:50:17.032 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:50:17.032 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:50:17.032 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 13 2024-03-30T04:50:17.032 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:50:17.032 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:50:17.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.035+0000 7f3f4d600700 1 -- 172.21.15.123:0/973664725 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3f2c06cc30 msgr2=0x7f3f2c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:17.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.035+0000 7f3f4d600700 1 --2- 172.21.15.123:0/973664725 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3f2c06cc30 0x7f3f2c06f0e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f3f48110c70 tx=0x7f3f34009e30 comp rx=0 tx=0).stop 2024-03-30T04:50:17.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.035+0000 7f3f4d600700 1 -- 172.21.15.123:0/973664725 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f48065630 msgr2=0x7f3f48115b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:17.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.035+0000 7f3f4d600700 1 --2- 172.21.15.123:0/973664725 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f48065630 0x7f3f48115b30 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f3f3800c830 tx=0x7f3f38004470 comp rx=0 tx=0).stop 2024-03-30T04:50:17.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.035+0000 7f3f4d600700 1 -- 172.21.15.123:0/973664725 shutdown_connections 2024-03-30T04:50:17.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.035+0000 7f3f4d600700 1 --2- 172.21.15.123:0/973664725 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3f2c06cc30 0x7f3f2c06f0e0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:17.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.035+0000 7f3f4d600700 1 --2- 172.21.15.123:0/973664725 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f48065630 0x7f3f48115b30 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:17.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.035+0000 7f3f4d600700 1 --2- 172.21.15.123:0/973664725 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3f48102ca0 0x7f3f48116070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:17.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.035+0000 7f3f4d600700 1 -- 172.21.15.123:0/973664725 >> 172.21.15.123:0/973664725 conn(0x7f3f48071b70 msgr2=0x7f3f481039c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:50:17.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.035+0000 7f3f4d600700 1 -- 172.21.15.123:0/973664725 shutdown_connections 2024-03-30T04:50:17.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.035+0000 7f3f4d600700 1 -- 172.21.15.123:0/973664725 wait complete. 2024-03-30T04:50:17.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:50:16 smithi135 ceph-mon[106011]: pgmap v572: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:17.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:50:16 smithi135 ceph-mon[106011]: from='client.35074 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:50:17.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.169+0000 7f076f400700 1 -- 172.21.15.123:0/2564208115 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07681037c0 msgr2=0x7f07680fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:17.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.169+0000 7f076f400700 1 --2- 172.21.15.123:0/2564208115 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07681037c0 0x7f07680fce30 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f0754004940 tx=0x7f0754009e30 comp rx=0 tx=0).stop 2024-03-30T04:50:17.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.169+0000 7f076f400700 1 -- 172.21.15.123:0/2564208115 shutdown_connections 2024-03-30T04:50:17.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.169+0000 7f076f400700 1 --2- 172.21.15.123:0/2564208115 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07681037c0 0x7f07680fce30 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:17.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.169+0000 7f076f400700 1 --2- 172.21.15.123:0/2564208115 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0768102e20 0x7f07681031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:17.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.169+0000 7f076f400700 1 -- 172.21.15.123:0/2564208115 >> 172.21.15.123:0/2564208115 conn(0x7f07680f89d0 msgr2=0x7f07680fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:50:17.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.169+0000 7f076f400700 1 -- 172.21.15.123:0/2564208115 shutdown_connections 2024-03-30T04:50:17.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.169+0000 7f076f400700 1 -- 172.21.15.123:0/2564208115 wait complete. 2024-03-30T04:50:17.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.170+0000 7f076f400700 1 Processor -- start 2024-03-30T04:50:17.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.170+0000 7f076f400700 1 -- start start 2024-03-30T04:50:17.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.171+0000 7f076f400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0768102e20 0x7f07680702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:50:17.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.171+0000 7f076f400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f07681037c0 0x7f0768070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:50:17.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.171+0000 7f076f400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07680720b0 con 0x7f0768102e20 2024-03-30T04:50:17.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.171+0000 7f076f400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0768072220 con 0x7f07681037c0 2024-03-30T04:50:17.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.171+0000 7f076ca00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0768102e20 0x7f07680702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:50:17.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.171+0000 7f076ca00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0768102e20 0x7f07680702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33632/0 (socket says 172.21.15.123:33632) 2024-03-30T04:50:17.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.171+0000 7f076ca00700 1 -- 172.21.15.123:0/1236121614 learned_addr learned my addr 172.21.15.123:0/1236121614 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:50:17.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.171+0000 7f0767e00700 1 --2- 172.21.15.123:0/1236121614 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f07681037c0 0x7f0768070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:50:17.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.171+0000 7f076ca00700 1 -- 172.21.15.123:0/1236121614 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f07681037c0 msgr2=0x7f0768070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:17.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.171+0000 7f076ca00700 1 --2- 172.21.15.123:0/1236121614 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f07681037c0 0x7f0768070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:17.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.171+0000 7f076ca00700 1 -- 172.21.15.123:0/1236121614 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0754005040 con 0x7f0768102e20 2024-03-30T04:50:17.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.171+0000 7f076ca00700 1 --2- 172.21.15.123:0/1236121614 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0768102e20 0x7f07680702c0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f075c00eb40 tx=0x7f075c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:50:17.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.172+0000 7f0764c00700 1 -- 172.21.15.123:0/1236121614 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f075c007cc0 con 0x7f0768102e20 2024-03-30T04:50:17.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.172+0000 7f0764c00700 1 -- 172.21.15.123:0/1236121614 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f075c00f040 con 0x7f0768102e20 2024-03-30T04:50:17.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.172+0000 7f076f400700 1 -- 172.21.15.123:0/1236121614 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0768070e00 con 0x7f0768102e20 2024-03-30T04:50:17.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.172+0000 7f0764c00700 1 -- 172.21.15.123:0/1236121614 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f075c00c990 con 0x7f0768102e20 2024-03-30T04:50:17.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.172+0000 7f076f400700 1 -- 172.21.15.123:0/1236121614 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f076811e6d0 con 0x7f0768102e20 2024-03-30T04:50:17.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.173+0000 7f076f400700 1 -- 172.21.15.123:0/1236121614 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07680fe570 con 0x7f0768102e20 2024-03-30T04:50:17.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.174+0000 7f0764c00700 1 -- 172.21.15.123:0/1236121614 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f075c00caf0 con 0x7f0768102e20 2024-03-30T04:50:17.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.175+0000 7f0764c00700 1 --2- 172.21.15.123:0/1236121614 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f075006cc20 0x7f075006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:50:17.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.175+0000 7f0764c00700 1 -- 172.21.15.123:0/1236121614 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f075c08bc00 con 0x7f0768102e20 2024-03-30T04:50:17.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.175+0000 7f0767e00700 1 --2- 172.21.15.123:0/1236121614 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f075006cc20 0x7f075006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:50:17.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.176+0000 7f0767e00700 1 --2- 172.21.15.123:0/1236121614 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f075006cc20 0x7f075006f0d0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f075400b0d0 tx=0x7f0754009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:50:17.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.179+0000 7f0764c00700 1 -- 172.21.15.123:0/1236121614 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f075c059700 con 0x7f0768102e20 2024-03-30T04:50:17.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.430+0000 7f076f400700 1 -- 172.21.15.123:0/1236121614 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f076804bf00 con 0x7f0768102e20 2024-03-30T04:50:17.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.430+0000 7f0764c00700 1 -- 172.21.15.123:0/1236121614 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f075c019750 con 0x7f0768102e20 2024-03-30T04:50:17.429 INFO:teuthology.orchestra.run.smithi123.stdout:e32 2024-03-30T04:50:17.429 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:50:17.429 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:50:17.429 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:50:17.429 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:50:17.429 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:50:17.429 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:50:17.429 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T04:50:17.429 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T04:50:17.429 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:50:17.430 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T04:50:17.430 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:50:17.430 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:50:17.430 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:50:17.430 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:50:17.430 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:50:17.430 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T04:50:17.430 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:50:17.430 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:50:17.430 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T04:50:17.430 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:50:17.430 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T04:50:17.430 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T04:50:17.430 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T04:50:17.431 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T04:50:17.431 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:50:17.431 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:50:17.431 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:50:17.431 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:50:17.431 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:50:17.431 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:50:17.431 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T04:50:17.431 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T04:50:17.431 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:50:17.431 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:50:17.431 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T04:50:17.431 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:50:17.431 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:50:17.432 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:50:17.432 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44315} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/1741352336,v1:172.21.15.135:6827/1741352336] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:50:17.432 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:50:17.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.433+0000 7f076f400700 1 -- 172.21.15.123:0/1236121614 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f075006cc20 msgr2=0x7f075006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:17.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.433+0000 7f076f400700 1 --2- 172.21.15.123:0/1236121614 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f075006cc20 0x7f075006f0d0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f075400b0d0 tx=0x7f0754009e30 comp rx=0 tx=0).stop 2024-03-30T04:50:17.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.433+0000 7f076f400700 1 -- 172.21.15.123:0/1236121614 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0768102e20 msgr2=0x7f07680702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:17.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.433+0000 7f076f400700 1 --2- 172.21.15.123:0/1236121614 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0768102e20 0x7f07680702c0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f075c00eb40 tx=0x7f075c00b040 comp rx=0 tx=0).stop 2024-03-30T04:50:17.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.433+0000 7f076f400700 1 -- 172.21.15.123:0/1236121614 shutdown_connections 2024-03-30T04:50:17.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.433+0000 7f076f400700 1 --2- 172.21.15.123:0/1236121614 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f075006cc20 0x7f075006f0d0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:17.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.433+0000 7f076f400700 1 --2- 172.21.15.123:0/1236121614 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0768102e20 0x7f07680702c0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:17.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.433+0000 7f076f400700 1 --2- 172.21.15.123:0/1236121614 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f07681037c0 0x7f0768070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:17.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.433+0000 7f076f400700 1 -- 172.21.15.123:0/1236121614 >> 172.21.15.123:0/1236121614 conn(0x7f07680f89d0 msgr2=0x7f07680f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:50:17.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.433+0000 7f076f400700 1 -- 172.21.15.123:0/1236121614 shutdown_connections 2024-03-30T04:50:17.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.433+0000 7f076f400700 1 -- 172.21.15.123:0/1236121614 wait complete. 2024-03-30T04:50:17.433 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 32 2024-03-30T04:50:17.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.568+0000 7f17bde00700 1 -- 172.21.15.123:0/1821199200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17b81059b0 msgr2=0x7f17b80ff020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:17.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.568+0000 7f17bde00700 1 --2- 172.21.15.123:0/1821199200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17b81059b0 0x7f17b80ff020 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f17a4004940 tx=0x7f17a4009e30 comp rx=0 tx=0).stop 2024-03-30T04:50:17.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.568+0000 7f17bde00700 1 -- 172.21.15.123:0/1821199200 shutdown_connections 2024-03-30T04:50:17.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.568+0000 7f17bde00700 1 --2- 172.21.15.123:0/1821199200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17b81059b0 0x7f17b80ff020 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:17.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.568+0000 7f17bde00700 1 --2- 172.21.15.123:0/1821199200 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f17b8105010 0x7f17b81053e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:17.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.568+0000 7f17bde00700 1 -- 172.21.15.123:0/1821199200 >> 172.21.15.123:0/1821199200 conn(0x7f17b80faba0 msgr2=0x7f17b80fcfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:50:17.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.568+0000 7f17bde00700 1 -- 172.21.15.123:0/1821199200 shutdown_connections 2024-03-30T04:50:17.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.568+0000 7f17bde00700 1 -- 172.21.15.123:0/1821199200 wait complete. 2024-03-30T04:50:17.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.569+0000 7f17bde00700 1 Processor -- start 2024-03-30T04:50:17.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.569+0000 7f17bde00700 1 -- start start 2024-03-30T04:50:17.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.570+0000 7f17bde00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f17b8105010 0x7f17b8115cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:50:17.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.570+0000 7f17bde00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17b81059b0 0x7f17b81161f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:50:17.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.570+0000 7f17bde00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17b8116840 con 0x7f17b81059b0 2024-03-30T04:50:17.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.570+0000 7f17bde00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17b810fc60 con 0x7f17b8105010 2024-03-30T04:50:17.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.570+0000 7f17b6000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17b81059b0 0x7f17b81161f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:50:17.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.570+0000 7f17b6000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17b81059b0 0x7f17b81161f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33652/0 (socket says 172.21.15.123:33652) 2024-03-30T04:50:17.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.570+0000 7f17b6000700 1 -- 172.21.15.123:0/3652468208 learned_addr learned my addr 172.21.15.123:0/3652468208 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:50:17.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.570+0000 7f17b6a00700 1 --2- 172.21.15.123:0/3652468208 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f17b8105010 0x7f17b8115cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:50:17.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.570+0000 7f17b6000700 1 -- 172.21.15.123:0/3652468208 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f17b8105010 msgr2=0x7f17b8115cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:17.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.570+0000 7f17b6000700 1 --2- 172.21.15.123:0/3652468208 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f17b8105010 0x7f17b8115cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:17.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.570+0000 7f17b6000700 1 -- 172.21.15.123:0/3652468208 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a4005040 con 0x7f17b81059b0 2024-03-30T04:50:17.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.571+0000 7f17b6000700 1 --2- 172.21.15.123:0/3652468208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17b81059b0 0x7f17b81161f0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f17a4009fd0 tx=0x7f17a4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:50:17.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.571+0000 7f17aea00700 1 -- 172.21.15.123:0/3652468208 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17a400e030 con 0x7f17b81059b0 2024-03-30T04:50:17.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.571+0000 7f17bde00700 1 -- 172.21.15.123:0/3652468208 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17b810fee0 con 0x7f17b81059b0 2024-03-30T04:50:17.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.571+0000 7f17aea00700 1 -- 172.21.15.123:0/3652468208 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f17a40093b0 con 0x7f17b81059b0 2024-03-30T04:50:17.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.571+0000 7f17aea00700 1 -- 172.21.15.123:0/3652468208 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17a401cc70 con 0x7f17b81059b0 2024-03-30T04:50:17.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.571+0000 7f17bde00700 1 -- 172.21.15.123:0/3652468208 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17b81104a0 con 0x7f17b81059b0 2024-03-30T04:50:17.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.572+0000 7f17bde00700 1 -- 172.21.15.123:0/3652468208 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17b8102320 con 0x7f17b81059b0 2024-03-30T04:50:17.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.573+0000 7f17aea00700 1 -- 172.21.15.123:0/3652468208 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f17a401c400 con 0x7f17b81059b0 2024-03-30T04:50:17.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.574+0000 7f17aea00700 1 --2- 172.21.15.123:0/3652468208 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f179c06cc30 0x7f179c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:50:17.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.574+0000 7f17aea00700 1 -- 172.21.15.123:0/3652468208 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f17a408d850 con 0x7f17b81059b0 2024-03-30T04:50:17.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.574+0000 7f17b6a00700 1 --2- 172.21.15.123:0/3652468208 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f179c06cc30 0x7f179c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:50:17.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.575+0000 7f17b6a00700 1 --2- 172.21.15.123:0/3652468208 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f179c06cc30 0x7f179c06f0e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f17a8009b80 tx=0x7f17a800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:50:17.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.578+0000 7f17aea00700 1 -- 172.21.15.123:0/3652468208 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f17a405b2a0 con 0x7f17b81059b0 2024-03-30T04:50:17.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.801+0000 7f17bde00700 1 -- 172.21.15.123:0/3652468208 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f17b8111370 con 0x7f179c06cc30 2024-03-30T04:50:17.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.802+0000 7f17aea00700 1 -- 172.21.15.123:0/3652468208 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f17b8111370 con 0x7f179c06cc30 2024-03-30T04:50:17.800 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:50:17.801 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:50:17.801 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:50:17.801 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T04:50:17.801 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:50:17.801 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T04:50:17.801 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T04:50:17.801 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T04:50:17.801 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T04:50:17.801 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:50:17.801 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "15/21 daemons upgraded", 2024-03-30T04:50:17.801 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T04:50:17.802 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:50:17.802 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:50:17.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.805+0000 7f17bde00700 1 -- 172.21.15.123:0/3652468208 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f179c06cc30 msgr2=0x7f179c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:17.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.805+0000 7f17bde00700 1 --2- 172.21.15.123:0/3652468208 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f179c06cc30 0x7f179c06f0e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f17a8009b80 tx=0x7f17a800b040 comp rx=0 tx=0).stop 2024-03-30T04:50:17.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.805+0000 7f17bde00700 1 -- 172.21.15.123:0/3652468208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17b81059b0 msgr2=0x7f17b81161f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:17.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.805+0000 7f17bde00700 1 --2- 172.21.15.123:0/3652468208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17b81059b0 0x7f17b81161f0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f17a4009fd0 tx=0x7f17a4009e30 comp rx=0 tx=0).stop 2024-03-30T04:50:17.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.805+0000 7f17bde00700 1 -- 172.21.15.123:0/3652468208 shutdown_connections 2024-03-30T04:50:17.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.805+0000 7f17bde00700 1 --2- 172.21.15.123:0/3652468208 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f179c06cc30 0x7f179c06f0e0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:17.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.805+0000 7f17bde00700 1 --2- 172.21.15.123:0/3652468208 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f17b8105010 0x7f17b8115cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:17.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.805+0000 7f17bde00700 1 --2- 172.21.15.123:0/3652468208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17b81059b0 0x7f17b81161f0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:17.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.805+0000 7f17bde00700 1 -- 172.21.15.123:0/3652468208 >> 172.21.15.123:0/3652468208 conn(0x7f17b80faba0 msgr2=0x7f17b8109600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:50:17.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.805+0000 7f17bde00700 1 -- 172.21.15.123:0/3652468208 shutdown_connections 2024-03-30T04:50:17.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.805+0000 7f17bde00700 1 -- 172.21.15.123:0/3652468208 wait complete. 2024-03-30T04:50:17.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.939+0000 7f295a800700 1 -- 172.21.15.123:0/157115855 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29541037a0 msgr2=0x7f29540fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:17.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.939+0000 7f295a800700 1 --2- 172.21.15.123:0/157115855 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29541037a0 0x7f29540fce10 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f2940004940 tx=0x7f2940009e30 comp rx=0 tx=0).stop 2024-03-30T04:50:17.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.939+0000 7f295a800700 1 -- 172.21.15.123:0/157115855 shutdown_connections 2024-03-30T04:50:17.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.939+0000 7f295a800700 1 --2- 172.21.15.123:0/157115855 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29541037a0 0x7f29540fce10 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:17.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.939+0000 7f295a800700 1 --2- 172.21.15.123:0/157115855 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2954102e00 0x7f29541031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:17.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.939+0000 7f295a800700 1 -- 172.21.15.123:0/157115855 >> 172.21.15.123:0/157115855 conn(0x7f29540f89b0 msgr2=0x7f29540fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:50:17.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.939+0000 7f295a800700 1 -- 172.21.15.123:0/157115855 shutdown_connections 2024-03-30T04:50:17.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.939+0000 7f295a800700 1 -- 172.21.15.123:0/157115855 wait complete. 2024-03-30T04:50:17.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.940+0000 7f295a800700 1 Processor -- start 2024-03-30T04:50:17.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.940+0000 7f295a800700 1 -- start start 2024-03-30T04:50:17.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.940+0000 7f295a800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2954102e00 0x7f2954115da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:50:17.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.940+0000 7f295a800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29541037a0 0x7f29541162e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:50:17.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.940+0000 7f295a800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2954116930 con 0x7f29541037a0 2024-03-30T04:50:17.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.940+0000 7f295a800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f295410fd00 con 0x7f2954102e00 2024-03-30T04:50:17.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.940+0000 7f2952a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29541037a0 0x7f29541162e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:50:17.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.941+0000 7f2952a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29541037a0 0x7f29541162e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33664/0 (socket says 172.21.15.123:33664) 2024-03-30T04:50:17.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.941+0000 7f2952a00700 1 -- 172.21.15.123:0/3662729198 learned_addr learned my addr 172.21.15.123:0/3662729198 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:50:17.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.941+0000 7f2953400700 1 --2- 172.21.15.123:0/3662729198 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2954102e00 0x7f2954115da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:50:17.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.941+0000 7f2952a00700 1 -- 172.21.15.123:0/3662729198 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2954102e00 msgr2=0x7f2954115da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:17.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.941+0000 7f2952a00700 1 --2- 172.21.15.123:0/3662729198 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2954102e00 0x7f2954115da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:17.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.941+0000 7f2952a00700 1 -- 172.21.15.123:0/3662729198 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2940005040 con 0x7f29541037a0 2024-03-30T04:50:17.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.941+0000 7f2952a00700 1 --2- 172.21.15.123:0/3662729198 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29541037a0 0x7f29541162e0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f2940009390 tx=0x7f2940009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:50:17.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.941+0000 7f294b400700 1 -- 172.21.15.123:0/3662729198 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f294000e040 con 0x7f29541037a0 2024-03-30T04:50:17.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.941+0000 7f294b400700 1 -- 172.21.15.123:0/3662729198 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2940013420 con 0x7f29541037a0 2024-03-30T04:50:17.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.941+0000 7f294b400700 1 -- 172.21.15.123:0/3662729198 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f294001c630 con 0x7f29541037a0 2024-03-30T04:50:17.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.941+0000 7f295a800700 1 -- 172.21.15.123:0/3662729198 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f295410ff80 con 0x7f29541037a0 2024-03-30T04:50:17.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.942+0000 7f295a800700 1 -- 172.21.15.123:0/3662729198 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29541104c0 con 0x7f29541037a0 2024-03-30T04:50:17.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.943+0000 7f295a800700 1 -- 172.21.15.123:0/3662729198 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29540fe550 con 0x7f29541037a0 2024-03-30T04:50:17.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.943+0000 7f294b400700 1 -- 172.21.15.123:0/3662729198 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f2940013a00 con 0x7f29541037a0 2024-03-30T04:50:17.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.944+0000 7f294b400700 1 --2- 172.21.15.123:0/3662729198 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f293806cc30 0x7f293806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:50:17.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.944+0000 7f294b400700 1 -- 172.21.15.123:0/3662729198 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f294008d2d0 con 0x7f29541037a0 2024-03-30T04:50:17.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.945+0000 7f2953400700 1 --2- 172.21.15.123:0/3662729198 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f293806cc30 0x7f293806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:50:17.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.945+0000 7f2953400700 1 --2- 172.21.15.123:0/3662729198 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f293806cc30 0x7f293806f0e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f2944004990 tx=0x7f2944009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:50:17.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:17.949+0000 7f294b400700 1 -- 172.21.15.123:0/3662729198 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f294005ada0 con 0x7f29541037a0 2024-03-30T04:50:17.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:50:17 smithi123 ceph-mon[139369]: from='client.35078 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:50:17.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:50:17 smithi123 ceph-mon[139369]: from='client.35082 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:50:17.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:50:17 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/973664725' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:50:17.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:50:17 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/1236121614' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:50:18.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:50:17 smithi135 ceph-mon[106011]: from='client.35078 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:50:18.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:50:17 smithi135 ceph-mon[106011]: from='client.35082 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:50:18.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:50:17 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/973664725' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:50:18.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:50:17 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/1236121614' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:50:18.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:18.239+0000 7f295a800700 1 -- 172.21.15.123:0/3662729198 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f295405f5f0 con 0x7f29541037a0 2024-03-30T04:50:18.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:18.239+0000 7f294b400700 1 -- 172.21.15.123:0/3662729198 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f294005a930 con 0x7f29541037a0 2024-03-30T04:50:18.238 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T04:50:18.238 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T04:50:18.238 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T04:50:18.238 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T04:50:18.238 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T04:50:18.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:18.242+0000 7f295a800700 1 -- 172.21.15.123:0/3662729198 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f293806cc30 msgr2=0x7f293806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:18.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:18.242+0000 7f295a800700 1 --2- 172.21.15.123:0/3662729198 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f293806cc30 0x7f293806f0e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f2944004990 tx=0x7f2944009ee0 comp rx=0 tx=0).stop 2024-03-30T04:50:18.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:18.242+0000 7f295a800700 1 -- 172.21.15.123:0/3662729198 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29541037a0 msgr2=0x7f29541162e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:18.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:18.242+0000 7f295a800700 1 --2- 172.21.15.123:0/3662729198 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29541037a0 0x7f29541162e0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f2940009390 tx=0x7f2940009e30 comp rx=0 tx=0).stop 2024-03-30T04:50:18.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:18.242+0000 7f295a800700 1 -- 172.21.15.123:0/3662729198 shutdown_connections 2024-03-30T04:50:18.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:18.242+0000 7f295a800700 1 --2- 172.21.15.123:0/3662729198 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f293806cc30 0x7f293806f0e0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:18.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:18.242+0000 7f295a800700 1 --2- 172.21.15.123:0/3662729198 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2954102e00 0x7f2954115da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:18.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:18.242+0000 7f295a800700 1 --2- 172.21.15.123:0/3662729198 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29541037a0 0x7f29541162e0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:18.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:18.242+0000 7f295a800700 1 -- 172.21.15.123:0/3662729198 >> 172.21.15.123:0/3662729198 conn(0x7f29540f89b0 msgr2=0x7f29540f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:50:18.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:18.242+0000 7f295a800700 1 -- 172.21.15.123:0/3662729198 shutdown_connections 2024-03-30T04:50:18.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:18.242+0000 7f295a800700 1 -- 172.21.15.123:0/3662729198 wait complete. 2024-03-30T04:50:18.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:50:18 smithi123 ceph-mon[139369]: pgmap v573: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:18.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:50:18 smithi123 ceph-mon[139369]: from='client.35094 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:50:18.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:50:18 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3662729198' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:50:19.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:50:18 smithi135 ceph-mon[106011]: pgmap v573: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:19.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:50:18 smithi135 ceph-mon[106011]: from='client.35094 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:50:19.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:50:18 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3662729198' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:50:20.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:50:20 smithi135 ceph-mon[106011]: pgmap v574: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:20.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:50:20 smithi123 ceph-mon[139369]: pgmap v574: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:22.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:50:22 smithi123 ceph-mon[139369]: pgmap v575: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:23.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:50:22 smithi135 ceph-mon[106011]: pgmap v575: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:24.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:50:24 smithi123 ceph-mon[139369]: pgmap v576: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:25.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:50:24 smithi135 ceph-mon[106011]: pgmap v576: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:26.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:50:26 smithi123 ceph-mon[139369]: pgmap v577: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:27.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:50:26 smithi135 ceph-mon[106011]: pgmap v577: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:28.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:50:28 smithi123 ceph-mon[139369]: pgmap v578: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:29.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:50:28 smithi135 ceph-mon[106011]: pgmap v578: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:29.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:50:29 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:50:30.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:50:29 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:50:30.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:50:30 smithi123 ceph-mon[139369]: pgmap v579: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:31.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:50:30 smithi135 ceph-mon[106011]: pgmap v579: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:32.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:50:32 smithi123 ceph-mon[139369]: pgmap v580: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:33.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:50:32 smithi135 ceph-mon[106011]: pgmap v580: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:35.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:50:34 smithi135 ceph-mon[106011]: pgmap v581: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:35.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:50:34 smithi123 ceph-mon[139369]: pgmap v581: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:37.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:50:36 smithi135 ceph-mon[106011]: pgmap v582: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:37.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:50:36 smithi123 ceph-mon[139369]: pgmap v582: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:39.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:50:38 smithi135 ceph-mon[106011]: pgmap v583: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:39.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:50:38 smithi123 ceph-mon[139369]: pgmap v583: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:40.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:50:40 smithi135 ceph-mon[106011]: pgmap v584: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:40.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:50:40 smithi123 ceph-mon[139369]: pgmap v584: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:42.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:50:42 smithi123 ceph-mon[139369]: pgmap v585: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:43.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:50:42 smithi135 ceph-mon[106011]: pgmap v585: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:44.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:50:44 smithi123 ceph-mon[139369]: pgmap v586: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:44.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:50:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:50:45.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:50:44 smithi135 ceph-mon[106011]: pgmap v586: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:45.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:50:44 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:50:46.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:50:46 smithi123 ceph-mon[139369]: pgmap v587: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:47.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:50:46 smithi135 ceph-mon[106011]: pgmap v587: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:48.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.390+0000 7f4dcee00700 1 -- 172.21.15.123:0/2345683339 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4dc8102e20 msgr2=0x7f4dc81031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:48.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.390+0000 7f4dcee00700 1 --2- 172.21.15.123:0/2345683339 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4dc8102e20 0x7f4dc81031f0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f4db8004940 tx=0x7f4db8009e30 comp rx=0 tx=0).stop 2024-03-30T04:50:48.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.390+0000 7f4dcee00700 1 -- 172.21.15.123:0/2345683339 shutdown_connections 2024-03-30T04:50:48.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.390+0000 7f4dcee00700 1 --2- 172.21.15.123:0/2345683339 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4dc81037c0 0x7f4dc80fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:48.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.390+0000 7f4dcee00700 1 --2- 172.21.15.123:0/2345683339 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4dc8102e20 0x7f4dc81031f0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:48.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.390+0000 7f4dcee00700 1 -- 172.21.15.123:0/2345683339 >> 172.21.15.123:0/2345683339 conn(0x7f4dc80f89d0 msgr2=0x7f4dc80fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:50:48.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.390+0000 7f4dcee00700 1 -- 172.21.15.123:0/2345683339 shutdown_connections 2024-03-30T04:50:48.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.390+0000 7f4dcee00700 1 -- 172.21.15.123:0/2345683339 wait complete. 2024-03-30T04:50:48.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.391+0000 7f4dcee00700 1 Processor -- start 2024-03-30T04:50:48.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.391+0000 7f4dcee00700 1 -- start start 2024-03-30T04:50:48.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.391+0000 7f4dcee00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4dc8102e20 0x7f4dc80702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:50:48.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.391+0000 7f4dcee00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4dc81037c0 0x7f4dc8070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:50:48.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.391+0000 7f4dcee00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc8070d90 con 0x7f4dc8102e20 2024-03-30T04:50:48.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.391+0000 7f4dcee00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc8070ed0 con 0x7f4dc81037c0 2024-03-30T04:50:48.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.391+0000 7f4dc7e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4dc8102e20 0x7f4dc80702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:50:48.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.392+0000 7f4dc7e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4dc8102e20 0x7f4dc80702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60028/0 (socket says 172.21.15.123:60028) 2024-03-30T04:50:48.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.392+0000 7f4dc7e00700 1 -- 172.21.15.123:0/1360242254 learned_addr learned my addr 172.21.15.123:0/1360242254 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:50:48.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.392+0000 7f4dc7400700 1 --2- 172.21.15.123:0/1360242254 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4dc81037c0 0x7f4dc8070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:50:48.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.392+0000 7f4dc7e00700 1 -- 172.21.15.123:0/1360242254 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4dc81037c0 msgr2=0x7f4dc8070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:48.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.392+0000 7f4dc7e00700 1 --2- 172.21.15.123:0/1360242254 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4dc81037c0 0x7f4dc8070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:48.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.392+0000 7f4dc7e00700 1 -- 172.21.15.123:0/1360242254 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db8005040 con 0x7f4dc8102e20 2024-03-30T04:50:48.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.392+0000 7f4dc7400700 1 --2- 172.21.15.123:0/1360242254 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4dc81037c0 0x7f4dc8070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:50:48.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.392+0000 7f4dc7e00700 1 --2- 172.21.15.123:0/1360242254 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4dc8102e20 0x7f4dc80702c0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f4db8009390 tx=0x7f4db8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:50:48.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.392+0000 7f4dbfe00700 1 -- 172.21.15.123:0/1360242254 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4db800e040 con 0x7f4dc8102e20 2024-03-30T04:50:48.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.392+0000 7f4dbfe00700 1 -- 172.21.15.123:0/1360242254 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4db8013420 con 0x7f4dc8102e20 2024-03-30T04:50:48.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.392+0000 7f4dbfe00700 1 -- 172.21.15.123:0/1360242254 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4db801c630 con 0x7f4dc8102e20 2024-03-30T04:50:48.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.392+0000 7f4dcee00700 1 -- 172.21.15.123:0/1360242254 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dc8071120 con 0x7f4dc8102e20 2024-03-30T04:50:48.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.393+0000 7f4dcee00700 1 -- 172.21.15.123:0/1360242254 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dc811a1b0 con 0x7f4dc8102e20 2024-03-30T04:50:48.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.394+0000 7f4dcee00700 1 -- 172.21.15.123:0/1360242254 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dc80fe570 con 0x7f4dc8102e20 2024-03-30T04:50:48.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.395+0000 7f4dbfe00700 1 -- 172.21.15.123:0/1360242254 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f4db80139e0 con 0x7f4dc8102e20 2024-03-30T04:50:48.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.396+0000 7f4dbfe00700 1 --2- 172.21.15.123:0/1360242254 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4dac06cc30 0x7f4dac06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:50:48.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.396+0000 7f4dbfe00700 1 -- 172.21.15.123:0/1360242254 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f4db808d260 con 0x7f4dc8102e20 2024-03-30T04:50:48.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.396+0000 7f4dc7400700 1 --2- 172.21.15.123:0/1360242254 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4dac06cc30 0x7f4dac06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:50:48.395 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.397+0000 7f4dc7400700 1 --2- 172.21.15.123:0/1360242254 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4dac06cc30 0x7f4dac06f0e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f4db4009b80 tx=0x7f4db400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:50:48.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.401+0000 7f4dbfe00700 1 -- 172.21.15.123:0/1360242254 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4db80572a0 con 0x7f4dc8102e20 2024-03-30T04:50:48.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.622+0000 7f4dcee00700 1 -- 172.21.15.123:0/1360242254 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4dc80719b0 con 0x7f4dac06cc30 2024-03-30T04:50:48.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.623+0000 7f4dbfe00700 1 -- 172.21.15.123:0/1360242254 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f4dc80719b0 con 0x7f4dac06cc30 2024-03-30T04:50:48.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.626+0000 7f4dcee00700 1 -- 172.21.15.123:0/1360242254 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4dac06cc30 msgr2=0x7f4dac06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:48.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.626+0000 7f4dcee00700 1 --2- 172.21.15.123:0/1360242254 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4dac06cc30 0x7f4dac06f0e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f4db4009b80 tx=0x7f4db400b040 comp rx=0 tx=0).stop 2024-03-30T04:50:48.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.626+0000 7f4dcee00700 1 -- 172.21.15.123:0/1360242254 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4dc8102e20 msgr2=0x7f4dc80702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:48.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.626+0000 7f4dcee00700 1 --2- 172.21.15.123:0/1360242254 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4dc8102e20 0x7f4dc80702c0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f4db8009390 tx=0x7f4db8009e30 comp rx=0 tx=0).stop 2024-03-30T04:50:48.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.626+0000 7f4dcee00700 1 -- 172.21.15.123:0/1360242254 shutdown_connections 2024-03-30T04:50:48.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.626+0000 7f4dcee00700 1 --2- 172.21.15.123:0/1360242254 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4dac06cc30 0x7f4dac06f0e0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:48.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.626+0000 7f4dcee00700 1 --2- 172.21.15.123:0/1360242254 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4dc8102e20 0x7f4dc80702c0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:48.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.626+0000 7f4dcee00700 1 --2- 172.21.15.123:0/1360242254 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4dc81037c0 0x7f4dc8070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:48.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.626+0000 7f4dcee00700 1 -- 172.21.15.123:0/1360242254 >> 172.21.15.123:0/1360242254 conn(0x7f4dc80f89d0 msgr2=0x7f4dc8105ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:50:48.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.626+0000 7f4dcee00700 1 -- 172.21.15.123:0/1360242254 shutdown_connections 2024-03-30T04:50:48.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.626+0000 7f4dcee00700 1 -- 172.21.15.123:0/1360242254 wait complete. 2024-03-30T04:50:48.639 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:50:48.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.762+0000 7fc372600700 1 -- 172.21.15.123:0/1709716043 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc36c102e00 msgr2=0x7fc36c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:48.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.762+0000 7fc372600700 1 --2- 172.21.15.123:0/1709716043 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc36c102e00 0x7fc36c1031d0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fc35c004940 tx=0x7fc35c009e30 comp rx=0 tx=0).stop 2024-03-30T04:50:48.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.762+0000 7fc372600700 1 -- 172.21.15.123:0/1709716043 shutdown_connections 2024-03-30T04:50:48.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.762+0000 7fc372600700 1 --2- 172.21.15.123:0/1709716043 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc36c1037a0 0x7fc36c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:48.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.762+0000 7fc372600700 1 --2- 172.21.15.123:0/1709716043 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc36c102e00 0x7fc36c1031d0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:48.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.762+0000 7fc372600700 1 -- 172.21.15.123:0/1709716043 >> 172.21.15.123:0/1709716043 conn(0x7fc36c0f89b0 msgr2=0x7fc36c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:50:48.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.762+0000 7fc372600700 1 -- 172.21.15.123:0/1709716043 shutdown_connections 2024-03-30T04:50:48.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.762+0000 7fc372600700 1 -- 172.21.15.123:0/1709716043 wait complete. 2024-03-30T04:50:48.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.763+0000 7fc372600700 1 Processor -- start 2024-03-30T04:50:48.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.763+0000 7fc372600700 1 -- start start 2024-03-30T04:50:48.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.763+0000 7fc372600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc36c102e00 0x7fc36c113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:50:48.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.763+0000 7fc372600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc36c1037a0 0x7fc36c114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:50:48.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.763+0000 7fc372600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc36c1146b0 con 0x7fc36c1037a0 2024-03-30T04:50:48.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.763+0000 7fc372600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc36c10daa0 con 0x7fc36c102e00 2024-03-30T04:50:48.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.764+0000 7fc36aa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc36c1037a0 0x7fc36c114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:50:48.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.764+0000 7fc36aa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc36c1037a0 0x7fc36c114060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60042/0 (socket says 172.21.15.123:60042) 2024-03-30T04:50:48.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.764+0000 7fc36aa00700 1 -- 172.21.15.123:0/2049897213 learned_addr learned my addr 172.21.15.123:0/2049897213 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:50:48.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.764+0000 7fc36b400700 1 --2- 172.21.15.123:0/2049897213 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc36c102e00 0x7fc36c113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:50:48.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.764+0000 7fc36aa00700 1 -- 172.21.15.123:0/2049897213 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc36c102e00 msgr2=0x7fc36c113b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:48.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.764+0000 7fc36aa00700 1 --2- 172.21.15.123:0/2049897213 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc36c102e00 0x7fc36c113b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:48.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.764+0000 7fc36aa00700 1 -- 172.21.15.123:0/2049897213 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc35c005040 con 0x7fc36c1037a0 2024-03-30T04:50:48.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.764+0000 7fc36aa00700 1 --2- 172.21.15.123:0/2049897213 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc36c1037a0 0x7fc36c114060 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fc358002a50 tx=0x7fc358004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:50:48.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.765+0000 7fc363400700 1 -- 172.21.15.123:0/2049897213 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc358007cc0 con 0x7fc36c1037a0 2024-03-30T04:50:48.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.765+0000 7fc363400700 1 -- 172.21.15.123:0/2049897213 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc35800f040 con 0x7fc36c1037a0 2024-03-30T04:50:48.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.765+0000 7fc372600700 1 -- 172.21.15.123:0/2049897213 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc36c10dd80 con 0x7fc36c1037a0 2024-03-30T04:50:48.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.765+0000 7fc363400700 1 -- 172.21.15.123:0/2049897213 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc35800d920 con 0x7fc36c1037a0 2024-03-30T04:50:48.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.765+0000 7fc372600700 1 -- 172.21.15.123:0/2049897213 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc36c10e340 con 0x7fc36c1037a0 2024-03-30T04:50:48.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.766+0000 7fc372600700 1 -- 172.21.15.123:0/2049897213 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc36c0fe550 con 0x7fc36c1037a0 2024-03-30T04:50:48.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.767+0000 7fc363400700 1 -- 172.21.15.123:0/2049897213 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fc358009680 con 0x7fc36c1037a0 2024-03-30T04:50:48.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.768+0000 7fc363400700 1 --2- 172.21.15.123:0/2049897213 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc35006cc30 0x7fc35006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:50:48.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.768+0000 7fc363400700 1 -- 172.21.15.123:0/2049897213 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc35808bc90 con 0x7fc36c1037a0 2024-03-30T04:50:48.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.768+0000 7fc36b400700 1 --2- 172.21.15.123:0/2049897213 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc35006cc30 0x7fc35006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:50:48.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.768+0000 7fc36b400700 1 --2- 172.21.15.123:0/2049897213 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc35006cc30 0x7fc35006f0e0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc35c009fd0 tx=0x7fc35c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:50:48.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.772+0000 7fc363400700 1 -- 172.21.15.123:0/2049897213 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc358059760 con 0x7fc36c1037a0 2024-03-30T04:50:48.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:50:48 smithi123 ceph-mon[139369]: pgmap v588: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:48.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.991+0000 7fc372600700 1 -- 172.21.15.123:0/2049897213 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc36c109dc0 con 0x7fc35006cc30 2024-03-30T04:50:48.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.993+0000 7fc363400700 1 -- 172.21.15.123:0/2049897213 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc36c109dc0 con 0x7fc35006cc30 2024-03-30T04:50:48.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.995+0000 7fc372600700 1 -- 172.21.15.123:0/2049897213 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc35006cc30 msgr2=0x7fc35006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:48.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.995+0000 7fc372600700 1 --2- 172.21.15.123:0/2049897213 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc35006cc30 0x7fc35006f0e0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc35c009fd0 tx=0x7fc35c009e30 comp rx=0 tx=0).stop 2024-03-30T04:50:48.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.995+0000 7fc372600700 1 -- 172.21.15.123:0/2049897213 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc36c1037a0 msgr2=0x7fc36c114060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:48.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.995+0000 7fc372600700 1 --2- 172.21.15.123:0/2049897213 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc36c1037a0 0x7fc36c114060 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fc358002a50 tx=0x7fc358004600 comp rx=0 tx=0).stop 2024-03-30T04:50:48.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.995+0000 7fc372600700 1 -- 172.21.15.123:0/2049897213 shutdown_connections 2024-03-30T04:50:48.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.995+0000 7fc372600700 1 --2- 172.21.15.123:0/2049897213 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc35006cc30 0x7fc35006f0e0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:48.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.995+0000 7fc372600700 1 --2- 172.21.15.123:0/2049897213 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc36c102e00 0x7fc36c113b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:48.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.995+0000 7fc372600700 1 --2- 172.21.15.123:0/2049897213 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc36c1037a0 0x7fc36c114060 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:48.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.995+0000 7fc372600700 1 -- 172.21.15.123:0/2049897213 >> 172.21.15.123:0/2049897213 conn(0x7fc36c0f89b0 msgr2=0x7fc36c0f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:50:48.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.995+0000 7fc372600700 1 -- 172.21.15.123:0/2049897213 shutdown_connections 2024-03-30T04:50:48.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:48.995+0000 7fc372600700 1 -- 172.21.15.123:0/2049897213 wait complete. 2024-03-30T04:50:49.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:50:48 smithi135 ceph-mon[106011]: pgmap v588: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:49.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.129+0000 7fb682600700 1 -- 172.21.15.123:0/4128927519 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb67c0fbc80 msgr2=0x7fb67c0fc050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:49.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.129+0000 7fb682600700 1 --2- 172.21.15.123:0/4128927519 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb67c0fbc80 0x7fb67c0fc050 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fb66c004940 tx=0x7fb66c009e30 comp rx=0 tx=0).stop 2024-03-30T04:50:49.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.129+0000 7fb682600700 1 -- 172.21.15.123:0/4128927519 shutdown_connections 2024-03-30T04:50:49.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.129+0000 7fb682600700 1 --2- 172.21.15.123:0/4128927519 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb67c0fc590 0x7fb67c1092f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:49.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.129+0000 7fb682600700 1 --2- 172.21.15.123:0/4128927519 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb67c0fbc80 0x7fb67c0fc050 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:49.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.129+0000 7fb682600700 1 -- 172.21.15.123:0/4128927519 >> 172.21.15.123:0/4128927519 conn(0x7fb67c072bc0 msgr2=0x7fb67c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:50:49.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.129+0000 7fb682600700 1 -- 172.21.15.123:0/4128927519 shutdown_connections 2024-03-30T04:50:49.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.129+0000 7fb682600700 1 -- 172.21.15.123:0/4128927519 wait complete. 2024-03-30T04:50:49.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.130+0000 7fb682600700 1 Processor -- start 2024-03-30T04:50:49.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.130+0000 7fb682600700 1 -- start start 2024-03-30T04:50:49.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.130+0000 7fb682600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb67c0fbc80 0x7fb67c115cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:50:49.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.130+0000 7fb682600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb67c0fc590 0x7fb67c1161f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:50:49.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.130+0000 7fb682600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb67c116840 con 0x7fb67c0fbc80 2024-03-30T04:50:49.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.130+0000 7fb682600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb67c10fc10 con 0x7fb67c0fc590 2024-03-30T04:50:49.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.131+0000 7fb67b400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb67c0fbc80 0x7fb67c115cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:50:49.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.131+0000 7fb67b400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb67c0fbc80 0x7fb67c115cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60066/0 (socket says 172.21.15.123:60066) 2024-03-30T04:50:49.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.131+0000 7fb67b400700 1 -- 172.21.15.123:0/3726871260 learned_addr learned my addr 172.21.15.123:0/3726871260 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:50:49.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.131+0000 7fb67b400700 1 -- 172.21.15.123:0/3726871260 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb67c0fc590 msgr2=0x7fb67c1161f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:49.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.131+0000 7fb67b400700 1 --2- 172.21.15.123:0/3726871260 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb67c0fc590 0x7fb67c1161f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:49.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.131+0000 7fb67b400700 1 -- 172.21.15.123:0/3726871260 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb66c005040 con 0x7fb67c0fbc80 2024-03-30T04:50:49.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.131+0000 7fb67b400700 1 --2- 172.21.15.123:0/3726871260 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb67c0fbc80 0x7fb67c115cb0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fb66c009390 tx=0x7fb66c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:50:49.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.131+0000 7fb673400700 1 -- 172.21.15.123:0/3726871260 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb66c00e040 con 0x7fb67c0fbc80 2024-03-30T04:50:49.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.132+0000 7fb673400700 1 -- 172.21.15.123:0/3726871260 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb66c013420 con 0x7fb67c0fbc80 2024-03-30T04:50:49.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.132+0000 7fb682600700 1 -- 172.21.15.123:0/3726871260 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb67c10fe90 con 0x7fb67c0fbc80 2024-03-30T04:50:49.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.132+0000 7fb673400700 1 -- 172.21.15.123:0/3726871260 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb66c01c630 con 0x7fb67c0fbc80 2024-03-30T04:50:49.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.132+0000 7fb682600700 1 -- 172.21.15.123:0/3726871260 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb67c110450 con 0x7fb67c0fbc80 2024-03-30T04:50:49.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.133+0000 7fb682600700 1 -- 172.21.15.123:0/3726871260 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb67c065e30 con 0x7fb67c0fbc80 2024-03-30T04:50:49.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.134+0000 7fb673400700 1 -- 172.21.15.123:0/3726871260 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fb66c0139e0 con 0x7fb67c0fbc80 2024-03-30T04:50:49.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.134+0000 7fb673400700 1 --2- 172.21.15.123:0/3726871260 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb66006cc30 0x7fb66006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:50:49.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.135+0000 7fb673400700 1 -- 172.21.15.123:0/3726871260 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb66c08d320 con 0x7fb67c0fbc80 2024-03-30T04:50:49.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.135+0000 7fb67aa00700 1 --2- 172.21.15.123:0/3726871260 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb66006cc30 0x7fb66006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:50:49.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.135+0000 7fb67aa00700 1 --2- 172.21.15.123:0/3726871260 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb66006cc30 0x7fb66006f0e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb67c111550 tx=0x7fb668009ce0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:50:49.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.139+0000 7fb673400700 1 -- 172.21.15.123:0/3726871260 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb66c05ae20 con 0x7fb67c0fbc80 2024-03-30T04:50:49.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.349+0000 7fb682600700 1 -- 172.21.15.123:0/3726871260 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb67c10c040 con 0x7fb66006cc30 2024-03-30T04:50:49.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.359+0000 7fb673400700 1 -- 172.21.15.123:0/3726871260 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fb67c10c040 con 0x7fb66006cc30 2024-03-30T04:50:49.358 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:50:49.358 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (19m) 12m ago 32m 15.0M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:50:49.358 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (17m) 12m ago 32m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T04:50:49.358 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (17m) 12m ago 31m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T04:50:49.358 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (19m) 12m ago 32m 74.0M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T04:50:49.358 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (13m) 12m ago 26m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T04:50:49.358 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (13m) 12m ago 26m 10.9M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T04:50:49.358 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (26m) 12m ago 26m 10.7M - 17.2.7-797-g5c024335 f77085f6aab7 0dc41326818c 2024-03-30T04:50:49.358 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (12m) 12m ago 26m 9747k - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T04:50:49.358 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (22m) 12m ago 34m 506M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:50:49.359 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (21m) 12m ago 31m 450M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:50:49.359 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (18m) 12m ago 34m 53.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T04:50:49.359 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (18m) 12m ago 31m 42.6M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T04:50:49.359 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (20m) 12m ago 32m 8052k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:50:49.359 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (20m) 12m ago 31m 7880k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:50:49.359 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (16m) 12m ago 30m 104M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T04:50:49.359 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (16m) 12m ago 29m 93.4M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T04:50:49.359 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (15m) 12m ago 29m 86.3M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T04:50:49.359 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (14m) 12m ago 28m 97.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T04:50:49.359 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (14m) 12m ago 28m 97.9M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T04:50:49.359 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (13m) 12m ago 27m 85.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T04:50:49.359 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (20m) 12m ago 32m 53.7M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:50:49.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.362+0000 7fb682600700 1 -- 172.21.15.123:0/3726871260 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb66006cc30 msgr2=0x7fb66006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:49.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.362+0000 7fb682600700 1 --2- 172.21.15.123:0/3726871260 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb66006cc30 0x7fb66006f0e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb67c111550 tx=0x7fb668009ce0 comp rx=0 tx=0).stop 2024-03-30T04:50:49.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.362+0000 7fb682600700 1 -- 172.21.15.123:0/3726871260 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb67c0fbc80 msgr2=0x7fb67c115cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:49.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.362+0000 7fb682600700 1 --2- 172.21.15.123:0/3726871260 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb67c0fbc80 0x7fb67c115cb0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fb66c009390 tx=0x7fb66c009e30 comp rx=0 tx=0).stop 2024-03-30T04:50:49.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.362+0000 7fb682600700 1 -- 172.21.15.123:0/3726871260 shutdown_connections 2024-03-30T04:50:49.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.362+0000 7fb682600700 1 --2- 172.21.15.123:0/3726871260 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb66006cc30 0x7fb66006f0e0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:49.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.362+0000 7fb682600700 1 --2- 172.21.15.123:0/3726871260 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb67c0fbc80 0x7fb67c115cb0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:49.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.362+0000 7fb682600700 1 --2- 172.21.15.123:0/3726871260 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb67c0fc590 0x7fb67c1161f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:49.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.362+0000 7fb682600700 1 -- 172.21.15.123:0/3726871260 >> 172.21.15.123:0/3726871260 conn(0x7fb67c072bc0 msgr2=0x7fb67c105ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:50:49.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.362+0000 7fb682600700 1 -- 172.21.15.123:0/3726871260 shutdown_connections 2024-03-30T04:50:49.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.362+0000 7fb682600700 1 -- 172.21.15.123:0/3726871260 wait complete. 2024-03-30T04:50:49.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.495+0000 7f1088e00700 1 -- 172.21.15.123:0/2837260720 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1084071f90 msgr2=0x7f1084070360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:49.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.495+0000 7f1088e00700 1 --2- 172.21.15.123:0/2837260720 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1084071f90 0x7f1084070360 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f1074004970 tx=0x7f1074009e30 comp rx=0 tx=0).stop 2024-03-30T04:50:49.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.495+0000 7f1088e00700 1 -- 172.21.15.123:0/2837260720 shutdown_connections 2024-03-30T04:50:49.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.495+0000 7f1088e00700 1 --2- 172.21.15.123:0/2837260720 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10840708a0 0x7f1084070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:49.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.495+0000 7f1088e00700 1 --2- 172.21.15.123:0/2837260720 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1084071f90 0x7f1084070360 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:49.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.495+0000 7f1088e00700 1 -- 172.21.15.123:0/2837260720 >> 172.21.15.123:0/2837260720 conn(0x7f10840f89b0 msgr2=0x7f10840fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:50:49.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.495+0000 7f1088e00700 1 -- 172.21.15.123:0/2837260720 shutdown_connections 2024-03-30T04:50:49.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.495+0000 7f1088e00700 1 -- 172.21.15.123:0/2837260720 wait complete. 2024-03-30T04:50:49.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.496+0000 7f1088e00700 1 Processor -- start 2024-03-30T04:50:49.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.496+0000 7f1088e00700 1 -- start start 2024-03-30T04:50:49.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.496+0000 7f1088e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10840708a0 0x7f108411a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:50:49.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.496+0000 7f1088e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1084071f90 0x7f108411a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:50:49.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.496+0000 7f1088e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f108411aba0 con 0x7f1084071f90 2024-03-30T04:50:49.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.496+0000 7f1088e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1084114020 con 0x7f10840708a0 2024-03-30T04:50:49.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.496+0000 7f1081400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1084071f90 0x7f108411a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:50:49.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.497+0000 7f1081400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1084071f90 0x7f108411a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60084/0 (socket says 172.21.15.123:60084) 2024-03-30T04:50:49.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.497+0000 7f1081400700 1 -- 172.21.15.123:0/2095302615 learned_addr learned my addr 172.21.15.123:0/2095302615 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:50:49.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.497+0000 7f1081e00700 1 --2- 172.21.15.123:0/2095302615 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10840708a0 0x7f108411a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:50:49.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.497+0000 7f1081400700 1 -- 172.21.15.123:0/2095302615 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10840708a0 msgr2=0x7f108411a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:49.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.497+0000 7f1081400700 1 --2- 172.21.15.123:0/2095302615 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10840708a0 0x7f108411a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:49.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.497+0000 7f1081400700 1 -- 172.21.15.123:0/2095302615 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1074005040 con 0x7f1084071f90 2024-03-30T04:50:49.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.497+0000 7f1081e00700 1 --2- 172.21.15.123:0/2095302615 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10840708a0 0x7f108411a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:50:49.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.497+0000 7f1081400700 1 --2- 172.21.15.123:0/2095302615 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1084071f90 0x7f108411a550 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f1070002a50 tx=0x7f10700046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:50:49.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.497+0000 7f107a000700 1 -- 172.21.15.123:0/2095302615 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1070007cc0 con 0x7f1084071f90 2024-03-30T04:50:49.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.497+0000 7f107a000700 1 -- 172.21.15.123:0/2095302615 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f107000f040 con 0x7f1084071f90 2024-03-30T04:50:49.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.497+0000 7f107a000700 1 -- 172.21.15.123:0/2095302615 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1070009540 con 0x7f1084071f90 2024-03-30T04:50:49.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.497+0000 7f1088e00700 1 -- 172.21.15.123:0/2095302615 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1084114300 con 0x7f1084071f90 2024-03-30T04:50:49.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.498+0000 7f1088e00700 1 -- 172.21.15.123:0/2095302615 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10841147f0 con 0x7f1084071f90 2024-03-30T04:50:49.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.499+0000 7f1088e00700 1 -- 172.21.15.123:0/2095302615 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1084071aa0 con 0x7f1084071f90 2024-03-30T04:50:49.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.500+0000 7f107a000700 1 -- 172.21.15.123:0/2095302615 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f10700096a0 con 0x7f1084071f90 2024-03-30T04:50:49.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.500+0000 7f107a000700 1 --2- 172.21.15.123:0/2095302615 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f106806cc30 0x7f106806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:50:49.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.500+0000 7f107a000700 1 -- 172.21.15.123:0/2095302615 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f107008bc50 con 0x7f1084071f90 2024-03-30T04:50:49.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.501+0000 7f1081e00700 1 --2- 172.21.15.123:0/2095302615 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f106806cc30 0x7f106806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:50:49.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.501+0000 7f1081e00700 1 --2- 172.21.15.123:0/2095302615 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f106806cc30 0x7f106806f0e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f107400b800 tx=0x7f1074009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:50:49.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.505+0000 7f107a000700 1 -- 172.21.15.123:0/2095302615 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f1070055c10 con 0x7f1084071f90 2024-03-30T04:50:49.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.793+0000 7f1088e00700 1 -- 172.21.15.123:0/2095302615 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f108404bed0 con 0x7f1084071f90 2024-03-30T04:50:49.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.794+0000 7f107a000700 1 -- 172.21.15.123:0/2095302615 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f1070059230 con 0x7f1084071f90 2024-03-30T04:50:49.792 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:50:49.792 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:50:49.792 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:50:49.792 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:50:49.792 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:50:49.792 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:50:49.793 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:50:49.793 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:50:49.793 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T04:50:49.793 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:50:49.793 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:50:49.793 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:50:49.793 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 3 2024-03-30T04:50:49.793 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:50:49.793 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:50:49.793 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:50:49.793 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 13 2024-03-30T04:50:49.793 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:50:49.793 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:50:49.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.796+0000 7f1088e00700 1 -- 172.21.15.123:0/2095302615 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f106806cc30 msgr2=0x7f106806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:49.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.796+0000 7f1088e00700 1 --2- 172.21.15.123:0/2095302615 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f106806cc30 0x7f106806f0e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f107400b800 tx=0x7f1074009e30 comp rx=0 tx=0).stop 2024-03-30T04:50:49.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.796+0000 7f1088e00700 1 -- 172.21.15.123:0/2095302615 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1084071f90 msgr2=0x7f108411a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:49.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.796+0000 7f1088e00700 1 --2- 172.21.15.123:0/2095302615 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1084071f90 0x7f108411a550 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f1070002a50 tx=0x7f10700046b0 comp rx=0 tx=0).stop 2024-03-30T04:50:49.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.797+0000 7f1088e00700 1 -- 172.21.15.123:0/2095302615 shutdown_connections 2024-03-30T04:50:49.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.797+0000 7f1088e00700 1 --2- 172.21.15.123:0/2095302615 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f106806cc30 0x7f106806f0e0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:49.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.797+0000 7f1088e00700 1 --2- 172.21.15.123:0/2095302615 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10840708a0 0x7f108411a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:49.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.797+0000 7f1088e00700 1 --2- 172.21.15.123:0/2095302615 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1084071f90 0x7f108411a550 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:49.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.797+0000 7f1088e00700 1 -- 172.21.15.123:0/2095302615 >> 172.21.15.123:0/2095302615 conn(0x7f10840f89b0 msgr2=0x7f10840f9b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:50:49.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.797+0000 7f1088e00700 1 -- 172.21.15.123:0/2095302615 shutdown_connections 2024-03-30T04:50:49.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.797+0000 7f1088e00700 1 -- 172.21.15.123:0/2095302615 wait complete. 2024-03-30T04:50:49.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.930+0000 7fcf12e00700 1 -- 172.21.15.123:0/1120656572 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf0c070360 msgr2=0x7fcf0c070730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:49.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.930+0000 7fcf12e00700 1 --2- 172.21.15.123:0/1120656572 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf0c070360 0x7fcf0c070730 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fcefc004970 tx=0x7fcefc009e30 comp rx=0 tx=0).stop 2024-03-30T04:50:49.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.931+0000 7fcf12e00700 1 -- 172.21.15.123:0/1120656572 shutdown_connections 2024-03-30T04:50:49.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.931+0000 7fcf12e00700 1 --2- 172.21.15.123:0/1120656572 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcf0c070c70 0x7fcf0c109110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:49.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.931+0000 7fcf12e00700 1 --2- 172.21.15.123:0/1120656572 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf0c070360 0x7fcf0c070730 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:49.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.931+0000 7fcf12e00700 1 -- 172.21.15.123:0/1120656572 >> 172.21.15.123:0/1120656572 conn(0x7fcf0c0f89b0 msgr2=0x7fcf0c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:50:49.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.931+0000 7fcf12e00700 1 -- 172.21.15.123:0/1120656572 shutdown_connections 2024-03-30T04:50:49.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.931+0000 7fcf12e00700 1 -- 172.21.15.123:0/1120656572 wait complete. 2024-03-30T04:50:49.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.931+0000 7fcf12e00700 1 Processor -- start 2024-03-30T04:50:49.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.932+0000 7fcf12e00700 1 -- start start 2024-03-30T04:50:49.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.932+0000 7fcf12e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf0c070360 0x7fcf0c115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:50:49.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.932+0000 7fcf12e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcf0c070c70 0x7fcf0c1161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:50:49.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.932+0000 7fcf12e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf0c116810 con 0x7fcf0c070360 2024-03-30T04:50:49.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.932+0000 7fcf12e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf0c10fbe0 con 0x7fcf0c070c70 2024-03-30T04:50:49.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.932+0000 7fcf0be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf0c070360 0x7fcf0c115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:50:49.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.932+0000 7fcf0be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf0c070360 0x7fcf0c115c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60100/0 (socket says 172.21.15.123:60100) 2024-03-30T04:50:49.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.932+0000 7fcf0be00700 1 -- 172.21.15.123:0/1622699392 learned_addr learned my addr 172.21.15.123:0/1622699392 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:50:49.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.933+0000 7fcf0b400700 1 --2- 172.21.15.123:0/1622699392 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcf0c070c70 0x7fcf0c1161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:50:49.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.933+0000 7fcf0be00700 1 -- 172.21.15.123:0/1622699392 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcf0c070c70 msgr2=0x7fcf0c1161c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:49.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.933+0000 7fcf0be00700 1 --2- 172.21.15.123:0/1622699392 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcf0c070c70 0x7fcf0c1161c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:49.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.933+0000 7fcf0be00700 1 -- 172.21.15.123:0/1622699392 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcefc005040 con 0x7fcf0c070360 2024-03-30T04:50:49.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.933+0000 7fcf0b400700 1 --2- 172.21.15.123:0/1622699392 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcf0c070c70 0x7fcf0c1161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:50:49.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.933+0000 7fcf0be00700 1 --2- 172.21.15.123:0/1622699392 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf0c070360 0x7fcf0c115c80 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fcefc005010 tx=0x7fcefc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:50:49.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.933+0000 7fcf03e00700 1 -- 172.21.15.123:0/1622699392 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcefc00e040 con 0x7fcf0c070360 2024-03-30T04:50:49.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.933+0000 7fcf12e00700 1 -- 172.21.15.123:0/1622699392 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf0c10fe60 con 0x7fcf0c070360 2024-03-30T04:50:49.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.933+0000 7fcf03e00700 1 -- 172.21.15.123:0/1622699392 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcefc004290 con 0x7fcf0c070360 2024-03-30T04:50:49.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.933+0000 7fcf03e00700 1 -- 172.21.15.123:0/1622699392 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcefc013e40 con 0x7fcf0c070360 2024-03-30T04:50:49.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.933+0000 7fcf12e00700 1 -- 172.21.15.123:0/1622699392 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf0c1103a0 con 0x7fcf0c070360 2024-03-30T04:50:49.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.934+0000 7fcf12e00700 1 -- 172.21.15.123:0/1622699392 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf0c103b90 con 0x7fcf0c070360 2024-03-30T04:50:49.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.935+0000 7fcf03e00700 1 -- 172.21.15.123:0/1622699392 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fcefc013620 con 0x7fcf0c070360 2024-03-30T04:50:49.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.936+0000 7fcf03e00700 1 --2- 172.21.15.123:0/1622699392 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fcef006cbc0 0x7fcef006f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:50:49.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.936+0000 7fcf03e00700 1 -- 172.21.15.123:0/1622699392 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fcefc08d360 con 0x7fcf0c070360 2024-03-30T04:50:49.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.936+0000 7fcf0b400700 1 --2- 172.21.15.123:0/1622699392 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fcef006cbc0 0x7fcef006f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:50:49.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.937+0000 7fcf0b400700 1 --2- 172.21.15.123:0/1622699392 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fcef006cbc0 0x7fcef006f070 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fcef8004600 tx=0x7fcef800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:50:49.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:49.941+0000 7fcf03e00700 1 -- 172.21.15.123:0/1622699392 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fcefc05adb0 con 0x7fcf0c070360 2024-03-30T04:50:50.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:50:49 smithi135 ceph-mon[106011]: from='client.35102 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:50:50.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:50:49 smithi135 ceph-mon[106011]: from='client.35106 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:50:50.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.187+0000 7fcf12e00700 1 -- 172.21.15.123:0/1622699392 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fcf0c04bed0 con 0x7fcf0c070360 2024-03-30T04:50:50.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.188+0000 7fcf03e00700 1 -- 172.21.15.123:0/1622699392 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1843 (secure 0 0 0) 0x7fcefc05a940 con 0x7fcf0c070360 2024-03-30T04:50:50.186 INFO:teuthology.orchestra.run.smithi123.stdout:e32 2024-03-30T04:50:50.186 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:50:50.186 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:50:50.186 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:50:50.187 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:50:50.187 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:50:50.187 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:50:50.187 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T04:50:50.187 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T04:50:50.187 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:50:50.187 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T04:50:50.187 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:50:50.187 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:50:50.187 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:50:50.187 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:50:50.187 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:50:50.187 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T04:50:50.187 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:50:50.188 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:50:50.188 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T04:50:50.188 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:50:50.188 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T04:50:50.188 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T04:50:50.188 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T04:50:50.188 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T04:50:50.188 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:50:50.188 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:50:50.188 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:50:50.188 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:50:50.188 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:50:50.188 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:50:50.188 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T04:50:50.189 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T04:50:50.189 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:50:50.189 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:50:50.189 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T04:50:50.189 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:50:50.189 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:50:50.189 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:50:50.189 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44315} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/1741352336,v1:172.21.15.135:6827/1741352336] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:50:50.189 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:50:50.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.190+0000 7fcf12e00700 1 -- 172.21.15.123:0/1622699392 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fcef006cbc0 msgr2=0x7fcef006f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:50.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.190+0000 7fcf12e00700 1 --2- 172.21.15.123:0/1622699392 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fcef006cbc0 0x7fcef006f070 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fcef8004600 tx=0x7fcef800b040 comp rx=0 tx=0).stop 2024-03-30T04:50:50.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.190+0000 7fcf12e00700 1 -- 172.21.15.123:0/1622699392 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf0c070360 msgr2=0x7fcf0c115c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:50.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.190+0000 7fcf12e00700 1 --2- 172.21.15.123:0/1622699392 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf0c070360 0x7fcf0c115c80 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fcefc005010 tx=0x7fcefc009e30 comp rx=0 tx=0).stop 2024-03-30T04:50:50.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.190+0000 7fcf12e00700 1 -- 172.21.15.123:0/1622699392 shutdown_connections 2024-03-30T04:50:50.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.190+0000 7fcf12e00700 1 --2- 172.21.15.123:0/1622699392 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fcef006cbc0 0x7fcef006f070 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:50.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.190+0000 7fcf12e00700 1 --2- 172.21.15.123:0/1622699392 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf0c070360 0x7fcf0c115c80 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:50.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.190+0000 7fcf12e00700 1 --2- 172.21.15.123:0/1622699392 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcf0c070c70 0x7fcf0c1161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:50.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.190+0000 7fcf12e00700 1 -- 172.21.15.123:0/1622699392 >> 172.21.15.123:0/1622699392 conn(0x7fcf0c0f89b0 msgr2=0x7fcf0c1015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:50:50.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.190+0000 7fcf12e00700 1 -- 172.21.15.123:0/1622699392 shutdown_connections 2024-03-30T04:50:50.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.190+0000 7fcf12e00700 1 -- 172.21.15.123:0/1622699392 wait complete. 2024-03-30T04:50:50.191 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 32 2024-03-30T04:50:50.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:50:49 smithi123 ceph-mon[139369]: from='client.35102 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:50:50.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:50:49 smithi123 ceph-mon[139369]: from='client.35106 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:50:50.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.326+0000 7f6533000700 1 -- 172.21.15.123:0/2484692909 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f652c102e00 msgr2=0x7f652c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:50.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.326+0000 7f6533000700 1 --2- 172.21.15.123:0/2484692909 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f652c102e00 0x7f652c1031d0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f651c004940 tx=0x7f651c009e30 comp rx=0 tx=0).stop 2024-03-30T04:50:50.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.326+0000 7f6533000700 1 -- 172.21.15.123:0/2484692909 shutdown_connections 2024-03-30T04:50:50.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.326+0000 7f6533000700 1 --2- 172.21.15.123:0/2484692909 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f652c1037a0 0x7f652c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:50.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.326+0000 7f6533000700 1 --2- 172.21.15.123:0/2484692909 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f652c102e00 0x7f652c1031d0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:50.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.326+0000 7f6533000700 1 -- 172.21.15.123:0/2484692909 >> 172.21.15.123:0/2484692909 conn(0x7f652c0f89b0 msgr2=0x7f652c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:50:50.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.326+0000 7f6533000700 1 -- 172.21.15.123:0/2484692909 shutdown_connections 2024-03-30T04:50:50.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.326+0000 7f6533000700 1 -- 172.21.15.123:0/2484692909 wait complete. 2024-03-30T04:50:50.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.327+0000 7f6533000700 1 Processor -- start 2024-03-30T04:50:50.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.327+0000 7f6533000700 1 -- start start 2024-03-30T04:50:50.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.327+0000 7f6533000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f652c102e00 0x7f652c115d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:50:50.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.327+0000 7f6533000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f652c1037a0 0x7f652c116270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:50:50.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.327+0000 7f6533000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f652c1168c0 con 0x7f652c1037a0 2024-03-30T04:50:50.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.327+0000 7f6533000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f652c10fc90 con 0x7f652c102e00 2024-03-30T04:50:50.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.328+0000 7f652b400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f652c1037a0 0x7f652c116270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:50:50.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.328+0000 7f652b400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f652c1037a0 0x7f652c116270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60130/0 (socket says 172.21.15.123:60130) 2024-03-30T04:50:50.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.328+0000 7f652b400700 1 -- 172.21.15.123:0/1035484115 learned_addr learned my addr 172.21.15.123:0/1035484115 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:50:50.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.328+0000 7f652b400700 1 -- 172.21.15.123:0/1035484115 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f652c102e00 msgr2=0x7f652c115d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:50.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.328+0000 7f652be00700 1 --2- 172.21.15.123:0/1035484115 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f652c102e00 0x7f652c115d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:50:50.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.328+0000 7f652b400700 1 --2- 172.21.15.123:0/1035484115 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f652c102e00 0x7f652c115d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:50.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.328+0000 7f652b400700 1 -- 172.21.15.123:0/1035484115 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f651c005040 con 0x7f652c1037a0 2024-03-30T04:50:50.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.328+0000 7f652b400700 1 --2- 172.21.15.123:0/1035484115 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f652c1037a0 0x7f652c116270 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f6518002a80 tx=0x7f6518004630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:50:50.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.329+0000 7f6523e00700 1 -- 172.21.15.123:0/1035484115 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6518007cc0 con 0x7f652c1037a0 2024-03-30T04:50:50.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.329+0000 7f6523e00700 1 -- 172.21.15.123:0/1035484115 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f651800f040 con 0x7f652c1037a0 2024-03-30T04:50:50.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.329+0000 7f6533000700 1 -- 172.21.15.123:0/1035484115 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f652c10ff70 con 0x7f652c1037a0 2024-03-30T04:50:50.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.329+0000 7f6523e00700 1 -- 172.21.15.123:0/1035484115 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f651800da40 con 0x7f652c1037a0 2024-03-30T04:50:50.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.329+0000 7f6533000700 1 -- 172.21.15.123:0/1035484115 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f652c110530 con 0x7f652c1037a0 2024-03-30T04:50:50.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.330+0000 7f6533000700 1 -- 172.21.15.123:0/1035484115 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f652c0fe550 con 0x7f652c1037a0 2024-03-30T04:50:50.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.331+0000 7f6523e00700 1 -- 172.21.15.123:0/1035484115 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f651800dbe0 con 0x7f652c1037a0 2024-03-30T04:50:50.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.332+0000 7f6523e00700 1 --2- 172.21.15.123:0/1035484115 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f651006cc30 0x7f651006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:50:50.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.332+0000 7f6523e00700 1 -- 172.21.15.123:0/1035484115 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f651808b830 con 0x7f652c1037a0 2024-03-30T04:50:50.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.332+0000 7f652be00700 1 --2- 172.21.15.123:0/1035484115 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f651006cc30 0x7f651006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:50:50.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.333+0000 7f652be00700 1 --2- 172.21.15.123:0/1035484115 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f651006cc30 0x7f651006f0e0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f651c00b3b0 tx=0x7f651c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:50:50.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.336+0000 7f6523e00700 1 -- 172.21.15.123:0/1035484115 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6518059200 con 0x7f652c1037a0 2024-03-30T04:50:50.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.557+0000 7f6533000700 1 -- 172.21.15.123:0/1035484115 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f652c109e50 con 0x7f651006cc30 2024-03-30T04:50:50.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.559+0000 7f6523e00700 1 -- 172.21.15.123:0/1035484115 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f652c109e50 con 0x7f651006cc30 2024-03-30T04:50:50.557 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:50:50.557 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:50:50.557 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:50:50.557 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T04:50:50.557 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:50:50.557 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T04:50:50.557 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T04:50:50.557 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T04:50:50.557 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T04:50:50.558 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:50:50.558 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "15/21 daemons upgraded", 2024-03-30T04:50:50.558 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T04:50:50.558 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:50:50.558 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:50:50.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.561+0000 7f6533000700 1 -- 172.21.15.123:0/1035484115 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f651006cc30 msgr2=0x7f651006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:50.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.561+0000 7f6533000700 1 --2- 172.21.15.123:0/1035484115 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f651006cc30 0x7f651006f0e0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f651c00b3b0 tx=0x7f651c009e30 comp rx=0 tx=0).stop 2024-03-30T04:50:50.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.561+0000 7f6533000700 1 -- 172.21.15.123:0/1035484115 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f652c1037a0 msgr2=0x7f652c116270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:50.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.561+0000 7f6533000700 1 --2- 172.21.15.123:0/1035484115 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f652c1037a0 0x7f652c116270 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f6518002a80 tx=0x7f6518004630 comp rx=0 tx=0).stop 2024-03-30T04:50:50.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.561+0000 7f6533000700 1 -- 172.21.15.123:0/1035484115 shutdown_connections 2024-03-30T04:50:50.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.561+0000 7f6533000700 1 --2- 172.21.15.123:0/1035484115 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f651006cc30 0x7f651006f0e0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:50.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.561+0000 7f6533000700 1 --2- 172.21.15.123:0/1035484115 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f652c102e00 0x7f652c115d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:50.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.561+0000 7f6533000700 1 --2- 172.21.15.123:0/1035484115 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f652c1037a0 0x7f652c116270 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:50.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.561+0000 7f6533000700 1 -- 172.21.15.123:0/1035484115 >> 172.21.15.123:0/1035484115 conn(0x7f652c0f89b0 msgr2=0x7f652c0704f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:50:50.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.561+0000 7f6533000700 1 -- 172.21.15.123:0/1035484115 shutdown_connections 2024-03-30T04:50:50.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.561+0000 7f6533000700 1 -- 172.21.15.123:0/1035484115 wait complete. 2024-03-30T04:50:50.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.696+0000 7fdcc5000700 1 -- 172.21.15.123:0/2393001712 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdcc0102e20 msgr2=0x7fdcc01031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:50.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.696+0000 7fdcc5000700 1 --2- 172.21.15.123:0/2393001712 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdcc0102e20 0x7fdcc01031f0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fdcb0004940 tx=0x7fdcb0009e30 comp rx=0 tx=0).stop 2024-03-30T04:50:50.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.696+0000 7fdcc5000700 1 -- 172.21.15.123:0/2393001712 shutdown_connections 2024-03-30T04:50:50.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.696+0000 7fdcc5000700 1 --2- 172.21.15.123:0/2393001712 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdcc01037c0 0x7fdcc00fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:50.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.696+0000 7fdcc5000700 1 --2- 172.21.15.123:0/2393001712 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdcc0102e20 0x7fdcc01031f0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:50.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.696+0000 7fdcc5000700 1 -- 172.21.15.123:0/2393001712 >> 172.21.15.123:0/2393001712 conn(0x7fdcc00f89d0 msgr2=0x7fdcc00fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:50:50.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.696+0000 7fdcc5000700 1 -- 172.21.15.123:0/2393001712 shutdown_connections 2024-03-30T04:50:50.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.696+0000 7fdcc5000700 1 -- 172.21.15.123:0/2393001712 wait complete. 2024-03-30T04:50:50.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.697+0000 7fdcc5000700 1 Processor -- start 2024-03-30T04:50:50.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.697+0000 7fdcc5000700 1 -- start start 2024-03-30T04:50:50.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.697+0000 7fdcc5000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdcc0102e20 0x7fdcc0115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:50:50.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.697+0000 7fdcc5000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdcc01037c0 0x7fdcc0116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:50:50.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.697+0000 7fdcc5000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc0116880 con 0x7fdcc0102e20 2024-03-30T04:50:50.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.697+0000 7fdcc5000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc010fc50 con 0x7fdcc01037c0 2024-03-30T04:50:50.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.697+0000 7fdcbde00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdcc0102e20 0x7fdcc0115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:50:50.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.697+0000 7fdcbde00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdcc0102e20 0x7fdcc0115cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60144/0 (socket says 172.21.15.123:60144) 2024-03-30T04:50:50.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.697+0000 7fdcbde00700 1 -- 172.21.15.123:0/210057120 learned_addr learned my addr 172.21.15.123:0/210057120 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:50:50.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.698+0000 7fdcbd400700 1 --2- 172.21.15.123:0/210057120 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdcc01037c0 0x7fdcc0116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:50:50.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.698+0000 7fdcbde00700 1 -- 172.21.15.123:0/210057120 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdcc01037c0 msgr2=0x7fdcc0116230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:50.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.698+0000 7fdcbde00700 1 --2- 172.21.15.123:0/210057120 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdcc01037c0 0x7fdcc0116230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:50.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.698+0000 7fdcbde00700 1 -- 172.21.15.123:0/210057120 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcb0005040 con 0x7fdcc0102e20 2024-03-30T04:50:50.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.698+0000 7fdcbde00700 1 --2- 172.21.15.123:0/210057120 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdcc0102e20 0x7fdcc0115cf0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fdcb0009360 tx=0x7fdcb0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:50:50.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.698+0000 7fdcb6000700 1 -- 172.21.15.123:0/210057120 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdcb000e040 con 0x7fdcc0102e20 2024-03-30T04:50:50.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.698+0000 7fdcb6000700 1 -- 172.21.15.123:0/210057120 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdcb0013420 con 0x7fdcc0102e20 2024-03-30T04:50:50.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.698+0000 7fdcc5000700 1 -- 172.21.15.123:0/210057120 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcc010fed0 con 0x7fdcc0102e20 2024-03-30T04:50:50.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.698+0000 7fdcb6000700 1 -- 172.21.15.123:0/210057120 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdcb001c5f0 con 0x7fdcc0102e20 2024-03-30T04:50:50.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.698+0000 7fdcc5000700 1 -- 172.21.15.123:0/210057120 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcc0110490 con 0x7fdcc0102e20 2024-03-30T04:50:50.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.700+0000 7fdcc5000700 1 -- 172.21.15.123:0/210057120 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcc00fe570 con 0x7fdcc0102e20 2024-03-30T04:50:50.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.701+0000 7fdcb6000700 1 -- 172.21.15.123:0/210057120 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fdcb0013590 con 0x7fdcc0102e20 2024-03-30T04:50:50.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.702+0000 7fdcb6000700 1 --2- 172.21.15.123:0/210057120 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fdca406cc30 0x7fdca406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:50:50.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.702+0000 7fdcb6000700 1 -- 172.21.15.123:0/210057120 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fdcb008d320 con 0x7fdcc0102e20 2024-03-30T04:50:50.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.702+0000 7fdcbd400700 1 --2- 172.21.15.123:0/210057120 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fdca406cc30 0x7fdca406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:50:50.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.702+0000 7fdcbd400700 1 --2- 172.21.15.123:0/210057120 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fdca406cc30 0x7fdca406f0e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fdcac007fd0 tx=0x7fdcac009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:50:50.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.706+0000 7fdcb6000700 1 -- 172.21.15.123:0/210057120 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fdcb005ae20 con 0x7fdcc0102e20 2024-03-30T04:50:50.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.990+0000 7fdcc5000700 1 -- 172.21.15.123:0/210057120 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fdcc004bed0 con 0x7fdcc0102e20 2024-03-30T04:50:50.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.990+0000 7fdcb6000700 1 -- 172.21.15.123:0/210057120 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fdcb005a9b0 con 0x7fdcc0102e20 2024-03-30T04:50:50.989 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T04:50:50.989 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T04:50:50.989 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T04:50:50.989 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T04:50:50.989 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T04:50:50.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.992+0000 7fdcc5000700 1 -- 172.21.15.123:0/210057120 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fdca406cc30 msgr2=0x7fdca406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:50.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.992+0000 7fdcc5000700 1 --2- 172.21.15.123:0/210057120 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fdca406cc30 0x7fdca406f0e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fdcac007fd0 tx=0x7fdcac009040 comp rx=0 tx=0).stop 2024-03-30T04:50:50.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.993+0000 7fdcc5000700 1 -- 172.21.15.123:0/210057120 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdcc0102e20 msgr2=0x7fdcc0115cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:50:50.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.993+0000 7fdcc5000700 1 --2- 172.21.15.123:0/210057120 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdcc0102e20 0x7fdcc0115cf0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fdcb0009360 tx=0x7fdcb0009e30 comp rx=0 tx=0).stop 2024-03-30T04:50:50.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.993+0000 7fdcc5000700 1 -- 172.21.15.123:0/210057120 shutdown_connections 2024-03-30T04:50:50.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.993+0000 7fdcc5000700 1 --2- 172.21.15.123:0/210057120 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fdca406cc30 0x7fdca406f0e0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:50.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.993+0000 7fdcc5000700 1 --2- 172.21.15.123:0/210057120 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdcc0102e20 0x7fdcc0115cf0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:50.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.993+0000 7fdcc5000700 1 --2- 172.21.15.123:0/210057120 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdcc01037c0 0x7fdcc0116230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:50:50.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.993+0000 7fdcc5000700 1 -- 172.21.15.123:0/210057120 >> 172.21.15.123:0/210057120 conn(0x7fdcc00f89d0 msgr2=0x7fdcc00f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:50:50.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.993+0000 7fdcc5000700 1 -- 172.21.15.123:0/210057120 shutdown_connections 2024-03-30T04:50:50.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:50:50.993+0000 7fdcc5000700 1 -- 172.21.15.123:0/210057120 wait complete. 2024-03-30T04:50:51.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:50:50 smithi135 ceph-mon[106011]: from='client.35110 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:50:51.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:50:50 smithi135 ceph-mon[106011]: pgmap v589: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:51.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:50:50 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2095302615' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:50:51.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:50:50 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/1622699392' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:50:51.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:50:50 smithi123 ceph-mon[139369]: from='client.35110 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:50:51.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:50:50 smithi123 ceph-mon[139369]: pgmap v589: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:51.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:50:50 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2095302615' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:50:51.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:50:50 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/1622699392' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:50:52.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:50:51 smithi135 ceph-mon[106011]: from='client.35122 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:50:52.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:50:51 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/210057120' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:50:52.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:50:51 smithi123 ceph-mon[139369]: from='client.35122 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:50:52.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:50:51 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/210057120' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:50:53.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:50:52 smithi135 ceph-mon[106011]: pgmap v590: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:53.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:50:52 smithi123 ceph-mon[139369]: pgmap v590: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:55.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:50:54 smithi135 ceph-mon[106011]: pgmap v591: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:55.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:50:54 smithi123 ceph-mon[139369]: pgmap v591: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:57.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:50:56 smithi135 ceph-mon[106011]: pgmap v592: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:57.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:50:56 smithi123 ceph-mon[139369]: pgmap v592: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:59.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:50:58 smithi135 ceph-mon[106011]: pgmap v593: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:50:59.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:50:58 smithi123 ceph-mon[139369]: pgmap v593: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:00.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:50:59 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:51:00.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:50:59 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:51:01.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:51:00 smithi135 ceph-mon[106011]: pgmap v594: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:01.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:51:00 smithi123 ceph-mon[139369]: pgmap v594: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:03.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:51:02 smithi135 ceph-mon[106011]: pgmap v595: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:03.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:51:02 smithi123 ceph-mon[139369]: pgmap v595: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:05.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:51:04 smithi135 ceph-mon[106011]: pgmap v596: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:05.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:51:04 smithi123 ceph-mon[139369]: pgmap v596: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:07.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:51:06 smithi135 ceph-mon[106011]: pgmap v597: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:07.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:51:06 smithi123 ceph-mon[139369]: pgmap v597: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:09.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:51:08 smithi135 ceph-mon[106011]: pgmap v598: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:09.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:51:08 smithi123 ceph-mon[139369]: pgmap v598: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:10.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:51:10 smithi123 ceph-mon[139369]: pgmap v599: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:11.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:51:10 smithi135 ceph-mon[106011]: pgmap v599: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:12.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:51:12 smithi123 ceph-mon[139369]: pgmap v600: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:13.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:51:12 smithi135 ceph-mon[106011]: pgmap v600: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:14.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:51:14 smithi123 ceph-mon[139369]: pgmap v601: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:14.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:51:14 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:51:15.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:51:14 smithi135 ceph-mon[106011]: pgmap v601: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:15.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:51:14 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:51:17.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:51:16 smithi135 ceph-mon[106011]: pgmap v602: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:17.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:51:16 smithi123 ceph-mon[139369]: pgmap v602: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:19.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:51:18 smithi135 ceph-mon[106011]: pgmap v603: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:19.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:51:18 smithi123 ceph-mon[139369]: pgmap v603: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:20.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:51:20 smithi123 ceph-mon[139369]: pgmap v604: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:21.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:51:20 smithi135 ceph-mon[106011]: pgmap v604: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:21.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.142+0000 7f1353c00700 1 -- 172.21.15.123:0/966812632 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f134c102e00 msgr2=0x7f134c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:21.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.142+0000 7f1353c00700 1 --2- 172.21.15.123:0/966812632 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f134c102e00 0x7f134c1031d0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f1340004940 tx=0x7f1340009e30 comp rx=0 tx=0).stop 2024-03-30T04:51:21.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.142+0000 7f1353c00700 1 -- 172.21.15.123:0/966812632 shutdown_connections 2024-03-30T04:51:21.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.142+0000 7f1353c00700 1 --2- 172.21.15.123:0/966812632 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f134c1037a0 0x7f134c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:21.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.142+0000 7f1353c00700 1 --2- 172.21.15.123:0/966812632 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f134c102e00 0x7f134c1031d0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:21.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.142+0000 7f1353c00700 1 -- 172.21.15.123:0/966812632 >> 172.21.15.123:0/966812632 conn(0x7f134c0f89b0 msgr2=0x7f134c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:51:21.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.143+0000 7f1353c00700 1 -- 172.21.15.123:0/966812632 shutdown_connections 2024-03-30T04:51:21.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.143+0000 7f1353c00700 1 -- 172.21.15.123:0/966812632 wait complete. 2024-03-30T04:51:21.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.143+0000 7f1353c00700 1 Processor -- start 2024-03-30T04:51:21.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.144+0000 7f1353c00700 1 -- start start 2024-03-30T04:51:21.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.144+0000 7f1353c00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f134c102e00 0x7f134c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:51:21.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.144+0000 7f1353c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f134c1037a0 0x7f134c070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:51:21.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.144+0000 7f1353c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f134c070d90 con 0x7f134c1037a0 2024-03-30T04:51:21.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.144+0000 7f1353c00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f134c070ed0 con 0x7f134c102e00 2024-03-30T04:51:21.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.144+0000 7f134be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f134c1037a0 0x7f134c070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:51:21.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.144+0000 7f134be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f134c1037a0 0x7f134c070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60238/0 (socket says 172.21.15.123:60238) 2024-03-30T04:51:21.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.144+0000 7f134be00700 1 -- 172.21.15.123:0/2686336810 learned_addr learned my addr 172.21.15.123:0/2686336810 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:51:21.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.145+0000 7f1351200700 1 --2- 172.21.15.123:0/2686336810 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f134c102e00 0x7f134c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:51:21.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.145+0000 7f134be00700 1 -- 172.21.15.123:0/2686336810 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f134c102e00 msgr2=0x7f134c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:21.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.145+0000 7f134be00700 1 --2- 172.21.15.123:0/2686336810 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f134c102e00 0x7f134c0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:21.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.145+0000 7f134be00700 1 -- 172.21.15.123:0/2686336810 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1340005040 con 0x7f134c1037a0 2024-03-30T04:51:21.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.145+0000 7f134be00700 1 --2- 172.21.15.123:0/2686336810 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f134c1037a0 0x7f134c070800 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f133800c800 tx=0x7f1338004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:51:21.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.145+0000 7f1348c00700 1 -- 172.21.15.123:0/2686336810 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f133800a760 con 0x7f134c1037a0 2024-03-30T04:51:21.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.145+0000 7f1353c00700 1 -- 172.21.15.123:0/2686336810 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f134c119e20 con 0x7f134c1037a0 2024-03-30T04:51:21.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.145+0000 7f1348c00700 1 -- 172.21.15.123:0/2686336810 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f133800ad70 con 0x7f134c1037a0 2024-03-30T04:51:21.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.145+0000 7f1348c00700 1 -- 172.21.15.123:0/2686336810 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13380113a0 con 0x7f134c1037a0 2024-03-30T04:51:21.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.145+0000 7f1353c00700 1 -- 172.21.15.123:0/2686336810 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f134c11a2a0 con 0x7f134c1037a0 2024-03-30T04:51:21.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.146+0000 7f1353c00700 1 -- 172.21.15.123:0/2686336810 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f134c0fe550 con 0x7f134c1037a0 2024-03-30T04:51:21.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.147+0000 7f1348c00700 1 -- 172.21.15.123:0/2686336810 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f1338012400 con 0x7f134c1037a0 2024-03-30T04:51:21.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.148+0000 7f1348c00700 1 --2- 172.21.15.123:0/2686336810 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f133406cc30 0x7f133406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:51:21.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.149+0000 7f1348c00700 1 -- 172.21.15.123:0/2686336810 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f133808c0d0 con 0x7f134c1037a0 2024-03-30T04:51:21.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.149+0000 7f1351200700 1 --2- 172.21.15.123:0/2686336810 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f133406cc30 0x7f133406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:51:21.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.149+0000 7f1351200700 1 --2- 172.21.15.123:0/2686336810 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f133406cc30 0x7f133406f0e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f134000b020 tx=0x7f1340009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:51:21.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.153+0000 7f1348c00700 1 -- 172.21.15.123:0/2686336810 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f1338056090 con 0x7f134c1037a0 2024-03-30T04:51:21.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.378+0000 7f1353c00700 1 -- 172.21.15.123:0/2686336810 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f134c04bed0 con 0x7f133406cc30 2024-03-30T04:51:21.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.380+0000 7f1348c00700 1 -- 172.21.15.123:0/2686336810 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f134c04bed0 con 0x7f133406cc30 2024-03-30T04:51:21.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.382+0000 7f1353c00700 1 -- 172.21.15.123:0/2686336810 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f133406cc30 msgr2=0x7f133406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:21.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.382+0000 7f1353c00700 1 --2- 172.21.15.123:0/2686336810 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f133406cc30 0x7f133406f0e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f134000b020 tx=0x7f1340009e30 comp rx=0 tx=0).stop 2024-03-30T04:51:21.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.382+0000 7f1353c00700 1 -- 172.21.15.123:0/2686336810 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f134c1037a0 msgr2=0x7f134c070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:21.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.382+0000 7f1353c00700 1 --2- 172.21.15.123:0/2686336810 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f134c1037a0 0x7f134c070800 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f133800c800 tx=0x7f1338004470 comp rx=0 tx=0).stop 2024-03-30T04:51:21.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.382+0000 7f1353c00700 1 -- 172.21.15.123:0/2686336810 shutdown_connections 2024-03-30T04:51:21.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.382+0000 7f1353c00700 1 --2- 172.21.15.123:0/2686336810 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f133406cc30 0x7f133406f0e0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:21.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.382+0000 7f1353c00700 1 --2- 172.21.15.123:0/2686336810 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f134c102e00 0x7f134c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:21.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.382+0000 7f1353c00700 1 --2- 172.21.15.123:0/2686336810 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f134c1037a0 0x7f134c070800 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:21.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.382+0000 7f1353c00700 1 -- 172.21.15.123:0/2686336810 >> 172.21.15.123:0/2686336810 conn(0x7f134c0f89b0 msgr2=0x7f134c0f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:51:21.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.382+0000 7f1353c00700 1 -- 172.21.15.123:0/2686336810 shutdown_connections 2024-03-30T04:51:21.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.382+0000 7f1353c00700 1 -- 172.21.15.123:0/2686336810 wait complete. 2024-03-30T04:51:21.395 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:51:21.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.517+0000 7f29e3600700 1 -- 172.21.15.123:0/497358563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29dc102e00 msgr2=0x7f29dc1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:21.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.517+0000 7f29e3600700 1 --2- 172.21.15.123:0/497358563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29dc102e00 0x7f29dc1031d0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f29d0004940 tx=0x7f29d0009e30 comp rx=0 tx=0).stop 2024-03-30T04:51:21.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.518+0000 7f29e3600700 1 -- 172.21.15.123:0/497358563 shutdown_connections 2024-03-30T04:51:21.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.518+0000 7f29e3600700 1 --2- 172.21.15.123:0/497358563 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f29dc1037a0 0x7f29dc0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:21.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.518+0000 7f29e3600700 1 --2- 172.21.15.123:0/497358563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29dc102e00 0x7f29dc1031d0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:21.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.518+0000 7f29e3600700 1 -- 172.21.15.123:0/497358563 >> 172.21.15.123:0/497358563 conn(0x7f29dc0f89b0 msgr2=0x7f29dc0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:51:21.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.518+0000 7f29e3600700 1 -- 172.21.15.123:0/497358563 shutdown_connections 2024-03-30T04:51:21.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.518+0000 7f29e3600700 1 -- 172.21.15.123:0/497358563 wait complete. 2024-03-30T04:51:21.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.519+0000 7f29e3600700 1 Processor -- start 2024-03-30T04:51:21.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.519+0000 7f29e3600700 1 -- start start 2024-03-30T04:51:21.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.519+0000 7f29e3600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f29dc102e00 0x7f29dc0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:51:21.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.519+0000 7f29e3600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29dc1037a0 0x7f29dc070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:51:21.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.519+0000 7f29e3600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29dc070d90 con 0x7f29dc1037a0 2024-03-30T04:51:21.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.519+0000 7f29e3600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29dc070ed0 con 0x7f29dc102e00 2024-03-30T04:51:21.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.519+0000 7f29dbe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29dc1037a0 0x7f29dc070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:51:21.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.519+0000 7f29dbe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29dc1037a0 0x7f29dc070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60264/0 (socket says 172.21.15.123:60264) 2024-03-30T04:51:21.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.519+0000 7f29dbe00700 1 -- 172.21.15.123:0/612299022 learned_addr learned my addr 172.21.15.123:0/612299022 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:51:21.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.520+0000 7f29e0c00700 1 --2- 172.21.15.123:0/612299022 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f29dc102e00 0x7f29dc0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:51:21.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.520+0000 7f29dbe00700 1 -- 172.21.15.123:0/612299022 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f29dc102e00 msgr2=0x7f29dc0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:21.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.520+0000 7f29dbe00700 1 --2- 172.21.15.123:0/612299022 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f29dc102e00 0x7f29dc0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:21.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.520+0000 7f29dbe00700 1 -- 172.21.15.123:0/612299022 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29d0005040 con 0x7f29dc1037a0 2024-03-30T04:51:21.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.520+0000 7f29e0c00700 1 --2- 172.21.15.123:0/612299022 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f29dc102e00 0x7f29dc0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:51:21.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.520+0000 7f29dbe00700 1 --2- 172.21.15.123:0/612299022 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29dc1037a0 0x7f29dc070800 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f29c8002a50 tx=0x7f29c8004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:51:21.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.520+0000 7f29d8c00700 1 -- 172.21.15.123:0/612299022 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f29c8007cc0 con 0x7f29dc1037a0 2024-03-30T04:51:21.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.520+0000 7f29d8c00700 1 -- 172.21.15.123:0/612299022 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f29c800f040 con 0x7f29dc1037a0 2024-03-30T04:51:21.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.520+0000 7f29e3600700 1 -- 172.21.15.123:0/612299022 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29dc119e20 con 0x7f29dc1037a0 2024-03-30T04:51:21.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.520+0000 7f29d8c00700 1 -- 172.21.15.123:0/612299022 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f29c800d9a0 con 0x7f29dc1037a0 2024-03-30T04:51:21.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.520+0000 7f29e3600700 1 -- 172.21.15.123:0/612299022 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29dc11a2a0 con 0x7f29dc1037a0 2024-03-30T04:51:21.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.521+0000 7f29e3600700 1 -- 172.21.15.123:0/612299022 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29dc0fe550 con 0x7f29dc1037a0 2024-03-30T04:51:21.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.522+0000 7f29d8c00700 1 -- 172.21.15.123:0/612299022 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f29c800dc60 con 0x7f29dc1037a0 2024-03-30T04:51:21.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.523+0000 7f29d8c00700 1 --2- 172.21.15.123:0/612299022 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f29c406cc30 0x7f29c406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:51:21.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.524+0000 7f29d8c00700 1 -- 172.21.15.123:0/612299022 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f29c808bef0 con 0x7f29dc1037a0 2024-03-30T04:51:21.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.524+0000 7f29e0c00700 1 --2- 172.21.15.123:0/612299022 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f29c406cc30 0x7f29c406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:51:21.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.524+0000 7f29e0c00700 1 --2- 172.21.15.123:0/612299022 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f29c406cc30 0x7f29c406f0e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f29d000b840 tx=0x7f29d0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:51:21.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.528+0000 7f29d8c00700 1 -- 172.21.15.123:0/612299022 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f29c8055e30 con 0x7f29dc1037a0 2024-03-30T04:51:21.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.750+0000 7f29e3600700 1 -- 172.21.15.123:0/612299022 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f29dc04bed0 con 0x7f29c406cc30 2024-03-30T04:51:21.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.752+0000 7f29d8c00700 1 -- 172.21.15.123:0/612299022 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f29dc04bed0 con 0x7f29c406cc30 2024-03-30T04:51:21.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.754+0000 7f29e3600700 1 -- 172.21.15.123:0/612299022 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f29c406cc30 msgr2=0x7f29c406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:21.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.754+0000 7f29e3600700 1 --2- 172.21.15.123:0/612299022 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f29c406cc30 0x7f29c406f0e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f29d000b840 tx=0x7f29d0009e30 comp rx=0 tx=0).stop 2024-03-30T04:51:21.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.754+0000 7f29e3600700 1 -- 172.21.15.123:0/612299022 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29dc1037a0 msgr2=0x7f29dc070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:21.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.754+0000 7f29e3600700 1 --2- 172.21.15.123:0/612299022 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29dc1037a0 0x7f29dc070800 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f29c8002a50 tx=0x7f29c8004600 comp rx=0 tx=0).stop 2024-03-30T04:51:21.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.754+0000 7f29e3600700 1 -- 172.21.15.123:0/612299022 shutdown_connections 2024-03-30T04:51:21.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.754+0000 7f29e3600700 1 --2- 172.21.15.123:0/612299022 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f29c406cc30 0x7f29c406f0e0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:21.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.754+0000 7f29e3600700 1 --2- 172.21.15.123:0/612299022 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f29dc102e00 0x7f29dc0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:21.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.754+0000 7f29e3600700 1 --2- 172.21.15.123:0/612299022 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29dc1037a0 0x7f29dc070800 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:21.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.754+0000 7f29e3600700 1 -- 172.21.15.123:0/612299022 >> 172.21.15.123:0/612299022 conn(0x7f29dc0f89b0 msgr2=0x7f29dc0f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:51:21.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.755+0000 7f29e3600700 1 -- 172.21.15.123:0/612299022 shutdown_connections 2024-03-30T04:51:21.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.755+0000 7f29e3600700 1 -- 172.21.15.123:0/612299022 wait complete. 2024-03-30T04:51:21.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.888+0000 7f3720400700 1 -- 172.21.15.123:0/665715214 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3718102e00 msgr2=0x7f37181031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:21.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.888+0000 7f3720400700 1 --2- 172.21.15.123:0/665715214 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3718102e00 0x7f37181031d0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f370c004940 tx=0x7f370c009e30 comp rx=0 tx=0).stop 2024-03-30T04:51:21.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.888+0000 7f3720400700 1 -- 172.21.15.123:0/665715214 shutdown_connections 2024-03-30T04:51:21.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.888+0000 7f3720400700 1 --2- 172.21.15.123:0/665715214 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f37181037a0 0x7f37180fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:21.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.888+0000 7f3720400700 1 --2- 172.21.15.123:0/665715214 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3718102e00 0x7f37181031d0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:21.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.888+0000 7f3720400700 1 -- 172.21.15.123:0/665715214 >> 172.21.15.123:0/665715214 conn(0x7f37180f89b0 msgr2=0x7f37180fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:51:21.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.888+0000 7f3720400700 1 -- 172.21.15.123:0/665715214 shutdown_connections 2024-03-30T04:51:21.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.888+0000 7f3720400700 1 -- 172.21.15.123:0/665715214 wait complete. 2024-03-30T04:51:21.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.889+0000 7f3720400700 1 Processor -- start 2024-03-30T04:51:21.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.889+0000 7f3720400700 1 -- start start 2024-03-30T04:51:21.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.889+0000 7f3720400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3718102e00 0x7f37180702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:51:21.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.889+0000 7f3720400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f37181037a0 0x7f3718070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:51:21.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.889+0000 7f3720400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3718070d90 con 0x7f37181037a0 2024-03-30T04:51:21.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.889+0000 7f3720400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3718070ed0 con 0x7f3718102e00 2024-03-30T04:51:21.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.890+0000 7f371d000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f37181037a0 0x7f3718070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:51:21.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.890+0000 7f371d000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f37181037a0 0x7f3718070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60288/0 (socket says 172.21.15.123:60288) 2024-03-30T04:51:21.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.890+0000 7f371d000700 1 -- 172.21.15.123:0/4169846979 learned_addr learned my addr 172.21.15.123:0/4169846979 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:51:21.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.890+0000 7f371da00700 1 --2- 172.21.15.123:0/4169846979 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3718102e00 0x7f37180702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:51:21.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.890+0000 7f371d000700 1 -- 172.21.15.123:0/4169846979 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3718102e00 msgr2=0x7f37180702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:21.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.890+0000 7f371d000700 1 --2- 172.21.15.123:0/4169846979 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3718102e00 0x7f37180702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:21.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.890+0000 7f371d000700 1 -- 172.21.15.123:0/4169846979 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f370c005040 con 0x7f37181037a0 2024-03-30T04:51:21.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.890+0000 7f371d000700 1 --2- 172.21.15.123:0/4169846979 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f37181037a0 0x7f3718070800 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f3704002a50 tx=0x7f37040046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:51:21.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.891+0000 7f3715600700 1 -- 172.21.15.123:0/4169846979 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3704007cc0 con 0x7f37181037a0 2024-03-30T04:51:21.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.891+0000 7f3715600700 1 -- 172.21.15.123:0/4169846979 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f370400f040 con 0x7f37181037a0 2024-03-30T04:51:21.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.891+0000 7f3715600700 1 -- 172.21.15.123:0/4169846979 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f370400da30 con 0x7f37181037a0 2024-03-30T04:51:21.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.891+0000 7f3720400700 1 -- 172.21.15.123:0/4169846979 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3718119d40 con 0x7f37181037a0 2024-03-30T04:51:21.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.891+0000 7f3720400700 1 -- 172.21.15.123:0/4169846979 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f371811a1c0 con 0x7f37181037a0 2024-03-30T04:51:21.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.892+0000 7f3720400700 1 -- 172.21.15.123:0/4169846979 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37180fe550 con 0x7f37181037a0 2024-03-30T04:51:21.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.893+0000 7f3715600700 1 -- 172.21.15.123:0/4169846979 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f370400db90 con 0x7f37181037a0 2024-03-30T04:51:21.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.894+0000 7f3715600700 1 --2- 172.21.15.123:0/4169846979 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f370006cc30 0x7f370006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:51:21.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.894+0000 7f3715600700 1 -- 172.21.15.123:0/4169846979 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f370408bcf0 con 0x7f37181037a0 2024-03-30T04:51:21.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.894+0000 7f371da00700 1 --2- 172.21.15.123:0/4169846979 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f370006cc30 0x7f370006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:51:21.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.895+0000 7f371da00700 1 --2- 172.21.15.123:0/4169846979 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f370006cc30 0x7f370006f0e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f370c009fd0 tx=0x7f370c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:51:21.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:21.898+0000 7f3715600700 1 -- 172.21.15.123:0/4169846979 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3704055c30 con 0x7f37181037a0 2024-03-30T04:51:22.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.108+0000 7f3720400700 1 -- 172.21.15.123:0/4169846979 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f371804bed0 con 0x7f370006cc30 2024-03-30T04:51:22.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.118+0000 7f3715600700 1 -- 172.21.15.123:0/4169846979 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f371804bed0 con 0x7f370006cc30 2024-03-30T04:51:22.116 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:51:22.116 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (20m) 13m ago 33m 15.0M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:51:22.116 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (17m) 13m ago 33m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T04:51:22.116 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (17m) 13m ago 32m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T04:51:22.116 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (20m) 13m ago 32m 74.0M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T04:51:22.116 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (13m) 13m ago 26m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T04:51:22.116 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (13m) 13m ago 26m 10.9M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T04:51:22.117 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (26m) 13m ago 26m 10.7M - 17.2.7-797-g5c024335 f77085f6aab7 0dc41326818c 2024-03-30T04:51:22.117 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (13m) 13m ago 26m 9747k - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T04:51:22.117 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (22m) 13m ago 34m 506M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:51:22.117 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (21m) 13m ago 31m 450M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:51:22.117 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (19m) 13m ago 34m 53.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T04:51:22.117 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (18m) 13m ago 31m 42.6M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T04:51:22.117 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (21m) 13m ago 33m 8052k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:51:22.117 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (21m) 13m ago 32m 7880k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:51:22.117 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (17m) 13m ago 31m 104M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T04:51:22.117 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (16m) 13m ago 30m 93.4M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T04:51:22.117 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (16m) 13m ago 29m 86.3M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T04:51:22.117 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (15m) 13m ago 29m 97.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T04:51:22.117 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (14m) 13m ago 28m 97.9M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T04:51:22.117 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (14m) 13m ago 28m 85.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T04:51:22.117 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (20m) 13m ago 32m 53.7M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:51:22.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.120+0000 7f3720400700 1 -- 172.21.15.123:0/4169846979 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f370006cc30 msgr2=0x7f370006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:22.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.120+0000 7f3720400700 1 --2- 172.21.15.123:0/4169846979 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f370006cc30 0x7f370006f0e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f370c009fd0 tx=0x7f370c009e30 comp rx=0 tx=0).stop 2024-03-30T04:51:22.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.120+0000 7f3720400700 1 -- 172.21.15.123:0/4169846979 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f37181037a0 msgr2=0x7f3718070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:22.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.120+0000 7f3720400700 1 --2- 172.21.15.123:0/4169846979 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f37181037a0 0x7f3718070800 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f3704002a50 tx=0x7f37040046b0 comp rx=0 tx=0).stop 2024-03-30T04:51:22.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.120+0000 7f3720400700 1 -- 172.21.15.123:0/4169846979 shutdown_connections 2024-03-30T04:51:22.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.120+0000 7f3720400700 1 --2- 172.21.15.123:0/4169846979 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f370006cc30 0x7f370006f0e0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:22.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.120+0000 7f3720400700 1 --2- 172.21.15.123:0/4169846979 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3718102e00 0x7f37180702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:22.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.120+0000 7f3720400700 1 --2- 172.21.15.123:0/4169846979 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f37181037a0 0x7f3718070800 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:22.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.120+0000 7f3720400700 1 -- 172.21.15.123:0/4169846979 >> 172.21.15.123:0/4169846979 conn(0x7f37180f89b0 msgr2=0x7f37180f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:51:22.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.120+0000 7f3720400700 1 -- 172.21.15.123:0/4169846979 shutdown_connections 2024-03-30T04:51:22.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.120+0000 7f3720400700 1 -- 172.21.15.123:0/4169846979 wait complete. 2024-03-30T04:51:22.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.251+0000 7fd296600700 1 -- 172.21.15.123:0/3683751135 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd290070360 msgr2=0x7fd290070730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:22.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.251+0000 7fd296600700 1 --2- 172.21.15.123:0/3683751135 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd290070360 0x7fd290070730 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fd280004970 tx=0x7fd280009e30 comp rx=0 tx=0).stop 2024-03-30T04:51:22.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.251+0000 7fd296600700 1 -- 172.21.15.123:0/3683751135 shutdown_connections 2024-03-30T04:51:22.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.251+0000 7fd296600700 1 --2- 172.21.15.123:0/3683751135 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd290070c70 0x7fd290109110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:22.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.251+0000 7fd296600700 1 --2- 172.21.15.123:0/3683751135 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd290070360 0x7fd290070730 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:22.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.251+0000 7fd296600700 1 -- 172.21.15.123:0/3683751135 >> 172.21.15.123:0/3683751135 conn(0x7fd2900f89b0 msgr2=0x7fd2900fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:51:22.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.251+0000 7fd296600700 1 -- 172.21.15.123:0/3683751135 shutdown_connections 2024-03-30T04:51:22.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.251+0000 7fd296600700 1 -- 172.21.15.123:0/3683751135 wait complete. 2024-03-30T04:51:22.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.252+0000 7fd296600700 1 Processor -- start 2024-03-30T04:51:22.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.252+0000 7fd296600700 1 -- start start 2024-03-30T04:51:22.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.252+0000 7fd296600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd290070360 0x7fd290115d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:51:22.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.252+0000 7fd296600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd290070c70 0x7fd290116240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:51:22.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.252+0000 7fd296600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd290116920 con 0x7fd290070360 2024-03-30T04:51:22.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.252+0000 7fd296600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd29010fc60 con 0x7fd290070c70 2024-03-30T04:51:22.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.252+0000 7fd28f400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd290070360 0x7fd290115d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:51:22.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.252+0000 7fd28f400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd290070360 0x7fd290115d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60300/0 (socket says 172.21.15.123:60300) 2024-03-30T04:51:22.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.252+0000 7fd28f400700 1 -- 172.21.15.123:0/4254548171 learned_addr learned my addr 172.21.15.123:0/4254548171 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:51:22.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.253+0000 7fd28ea00700 1 --2- 172.21.15.123:0/4254548171 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd290070c70 0x7fd290116240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:51:22.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.253+0000 7fd28f400700 1 -- 172.21.15.123:0/4254548171 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd290070c70 msgr2=0x7fd290116240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:22.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.253+0000 7fd28f400700 1 --2- 172.21.15.123:0/4254548171 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd290070c70 0x7fd290116240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:22.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.253+0000 7fd28f400700 1 -- 172.21.15.123:0/4254548171 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd280005040 con 0x7fd290070360 2024-03-30T04:51:22.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.253+0000 7fd28f400700 1 --2- 172.21.15.123:0/4254548171 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd290070360 0x7fd290115d00 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fd280004f50 tx=0x7fd280009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:51:22.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.253+0000 7fd287400700 1 -- 172.21.15.123:0/4254548171 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd28000e040 con 0x7fd290070360 2024-03-30T04:51:22.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.253+0000 7fd287400700 1 -- 172.21.15.123:0/4254548171 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd280013420 con 0x7fd290070360 2024-03-30T04:51:22.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.253+0000 7fd296600700 1 -- 172.21.15.123:0/4254548171 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd29010fe80 con 0x7fd290070360 2024-03-30T04:51:22.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.253+0000 7fd287400700 1 -- 172.21.15.123:0/4254548171 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd28001c630 con 0x7fd290070360 2024-03-30T04:51:22.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.254+0000 7fd296600700 1 -- 172.21.15.123:0/4254548171 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2901103c0 con 0x7fd290070360 2024-03-30T04:51:22.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.255+0000 7fd296600700 1 -- 172.21.15.123:0/4254548171 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd290071aa0 con 0x7fd290070360 2024-03-30T04:51:22.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.255+0000 7fd287400700 1 -- 172.21.15.123:0/4254548171 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fd280013590 con 0x7fd290070360 2024-03-30T04:51:22.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.256+0000 7fd287400700 1 --2- 172.21.15.123:0/4254548171 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd27406cc30 0x7fd27406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:51:22.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.256+0000 7fd287400700 1 -- 172.21.15.123:0/4254548171 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd28008d1f0 con 0x7fd290070360 2024-03-30T04:51:22.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.257+0000 7fd28ea00700 1 --2- 172.21.15.123:0/4254548171 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd27406cc30 0x7fd27406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:51:22.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.257+0000 7fd28ea00700 1 --2- 172.21.15.123:0/4254548171 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd27406cc30 0x7fd27406f0e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd2901115a0 tx=0x7fd27c0091a0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:51:22.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.261+0000 7fd287400700 1 -- 172.21.15.123:0/4254548171 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd28005ac40 con 0x7fd290070360 2024-03-30T04:51:22.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.543+0000 7fd296600700 1 -- 172.21.15.123:0/4254548171 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fd29004bed0 con 0x7fd290070360 2024-03-30T04:51:22.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.543+0000 7fd287400700 1 -- 172.21.15.123:0/4254548171 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7fd28005a7d0 con 0x7fd290070360 2024-03-30T04:51:22.542 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:51:22.542 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:51:22.542 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:51:22.542 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:51:22.542 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:51:22.542 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:51:22.542 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:51:22.542 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:51:22.542 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T04:51:22.542 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:51:22.542 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:51:22.543 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:51:22.543 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 3 2024-03-30T04:51:22.543 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:51:22.543 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:51:22.543 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:51:22.543 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 13 2024-03-30T04:51:22.543 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:51:22.543 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:51:22.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.545+0000 7fd296600700 1 -- 172.21.15.123:0/4254548171 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd27406cc30 msgr2=0x7fd27406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:22.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.545+0000 7fd296600700 1 --2- 172.21.15.123:0/4254548171 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd27406cc30 0x7fd27406f0e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd2901115a0 tx=0x7fd27c0091a0 comp rx=0 tx=0).stop 2024-03-30T04:51:22.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.545+0000 7fd296600700 1 -- 172.21.15.123:0/4254548171 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd290070360 msgr2=0x7fd290115d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:22.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.545+0000 7fd296600700 1 --2- 172.21.15.123:0/4254548171 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd290070360 0x7fd290115d00 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fd280004f50 tx=0x7fd280009e30 comp rx=0 tx=0).stop 2024-03-30T04:51:22.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.545+0000 7fd296600700 1 -- 172.21.15.123:0/4254548171 shutdown_connections 2024-03-30T04:51:22.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.545+0000 7fd296600700 1 --2- 172.21.15.123:0/4254548171 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd27406cc30 0x7fd27406f0e0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:22.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.545+0000 7fd296600700 1 --2- 172.21.15.123:0/4254548171 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd290070360 0x7fd290115d00 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:22.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.545+0000 7fd296600700 1 --2- 172.21.15.123:0/4254548171 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd290070c70 0x7fd290116240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:22.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.545+0000 7fd296600700 1 -- 172.21.15.123:0/4254548171 >> 172.21.15.123:0/4254548171 conn(0x7fd2900f89b0 msgr2=0x7fd2900f9f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:51:22.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.546+0000 7fd296600700 1 -- 172.21.15.123:0/4254548171 shutdown_connections 2024-03-30T04:51:22.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.546+0000 7fd296600700 1 -- 172.21.15.123:0/4254548171 wait complete. 2024-03-30T04:51:22.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.679+0000 7f60e6400700 1 -- 172.21.15.123:0/3371088554 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60e0102e00 msgr2=0x7f60e01031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:22.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.679+0000 7f60e6400700 1 --2- 172.21.15.123:0/3371088554 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60e0102e00 0x7f60e01031d0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f60d0004940 tx=0x7f60d0009e30 comp rx=0 tx=0).stop 2024-03-30T04:51:22.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.679+0000 7f60e6400700 1 -- 172.21.15.123:0/3371088554 shutdown_connections 2024-03-30T04:51:22.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.679+0000 7f60e6400700 1 --2- 172.21.15.123:0/3371088554 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f60e01037a0 0x7f60e00fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:22.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.679+0000 7f60e6400700 1 --2- 172.21.15.123:0/3371088554 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60e0102e00 0x7f60e01031d0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:22.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.679+0000 7f60e6400700 1 -- 172.21.15.123:0/3371088554 >> 172.21.15.123:0/3371088554 conn(0x7f60e00f89b0 msgr2=0x7f60e00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:51:22.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.679+0000 7f60e6400700 1 -- 172.21.15.123:0/3371088554 shutdown_connections 2024-03-30T04:51:22.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.679+0000 7f60e6400700 1 -- 172.21.15.123:0/3371088554 wait complete. 2024-03-30T04:51:22.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.680+0000 7f60e6400700 1 Processor -- start 2024-03-30T04:51:22.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.680+0000 7f60e6400700 1 -- start start 2024-03-30T04:51:22.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.680+0000 7f60e6400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60e0102e00 0x7f60e0115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:51:22.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.680+0000 7f60e6400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f60e01037a0 0x7f60e0116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:51:22.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.680+0000 7f60e6400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60e0116860 con 0x7f60e0102e00 2024-03-30T04:51:22.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.680+0000 7f60e6400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60e010fc30 con 0x7f60e01037a0 2024-03-30T04:51:22.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.681+0000 7f60df400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60e0102e00 0x7f60e0115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:51:22.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.681+0000 7f60df400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60e0102e00 0x7f60e0115cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60314/0 (socket says 172.21.15.123:60314) 2024-03-30T04:51:22.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.681+0000 7f60df400700 1 -- 172.21.15.123:0/1511659724 learned_addr learned my addr 172.21.15.123:0/1511659724 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:51:22.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.681+0000 7f60dea00700 1 --2- 172.21.15.123:0/1511659724 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f60e01037a0 0x7f60e0116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:51:22.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.681+0000 7f60df400700 1 -- 172.21.15.123:0/1511659724 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f60e01037a0 msgr2=0x7f60e0116210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:22.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.681+0000 7f60df400700 1 --2- 172.21.15.123:0/1511659724 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f60e01037a0 0x7f60e0116210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:22.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.681+0000 7f60df400700 1 -- 172.21.15.123:0/1511659724 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60d0005040 con 0x7f60e0102e00 2024-03-30T04:51:22.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.681+0000 7f60df400700 1 --2- 172.21.15.123:0/1511659724 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60e0102e00 0x7f60e0115cd0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f60d0009360 tx=0x7f60d0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:51:22.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.681+0000 7f60d7400700 1 -- 172.21.15.123:0/1511659724 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f60d000e040 con 0x7f60e0102e00 2024-03-30T04:51:22.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.682+0000 7f60d7400700 1 -- 172.21.15.123:0/1511659724 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f60d0013420 con 0x7f60e0102e00 2024-03-30T04:51:22.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.682+0000 7f60e6400700 1 -- 172.21.15.123:0/1511659724 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60e010feb0 con 0x7f60e0102e00 2024-03-30T04:51:22.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.682+0000 7f60d7400700 1 -- 172.21.15.123:0/1511659724 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f60d001c630 con 0x7f60e0102e00 2024-03-30T04:51:22.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.682+0000 7f60e6400700 1 -- 172.21.15.123:0/1511659724 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60e0110470 con 0x7f60e0102e00 2024-03-30T04:51:22.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.683+0000 7f60e6400700 1 -- 172.21.15.123:0/1511659724 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60e00fe550 con 0x7f60e0102e00 2024-03-30T04:51:22.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.684+0000 7f60d7400700 1 -- 172.21.15.123:0/1511659724 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f60d0013590 con 0x7f60e0102e00 2024-03-30T04:51:22.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.684+0000 7f60d7400700 1 --2- 172.21.15.123:0/1511659724 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f60c406cc30 0x7f60c406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:51:22.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.685+0000 7f60d7400700 1 -- 172.21.15.123:0/1511659724 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f60d008d3d0 con 0x7f60e0102e00 2024-03-30T04:51:22.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.685+0000 7f60dea00700 1 --2- 172.21.15.123:0/1511659724 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f60c406cc30 0x7f60c406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:51:22.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.685+0000 7f60dea00700 1 --2- 172.21.15.123:0/1511659724 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f60c406cc30 0x7f60c406f0e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f60e0111570 tx=0x7f60cc009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:51:22.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.689+0000 7f60d7400700 1 -- 172.21.15.123:0/1511659724 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f60d005aed0 con 0x7f60e0102e00 2024-03-30T04:51:22.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.934+0000 7f60e6400700 1 -- 172.21.15.123:0/1511659724 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f60e004bf00 con 0x7f60e0102e00 2024-03-30T04:51:22.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.934+0000 7f60d7400700 1 -- 172.21.15.123:0/1511659724 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f60d005aa60 con 0x7f60e0102e00 2024-03-30T04:51:22.933 INFO:teuthology.orchestra.run.smithi123.stdout:e32 2024-03-30T04:51:22.933 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:51:22.933 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:51:22.933 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:51:22.933 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:51:22.933 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:51:22.933 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:51:22.933 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T04:51:22.933 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T04:51:22.933 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:51:22.934 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T04:51:22.934 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:51:22.934 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:51:22.934 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:51:22.934 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:51:22.934 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:51:22.934 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T04:51:22.934 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:51:22.934 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:51:22.934 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T04:51:22.934 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:51:22.934 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T04:51:22.934 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T04:51:22.934 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T04:51:22.935 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T04:51:22.935 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:51:22.935 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:51:22.935 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:51:22.935 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:51:22.935 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:51:22.935 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:51:22.935 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T04:51:22.935 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T04:51:22.935 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:51:22.935 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:51:22.935 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T04:51:22.935 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:51:22.935 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:51:22.935 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:51:22.936 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44315} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/1741352336,v1:172.21.15.135:6827/1741352336] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:51:22.936 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:51:22.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.937+0000 7f60e6400700 1 -- 172.21.15.123:0/1511659724 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f60c406cc30 msgr2=0x7f60c406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:22.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.937+0000 7f60e6400700 1 --2- 172.21.15.123:0/1511659724 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f60c406cc30 0x7f60c406f0e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f60e0111570 tx=0x7f60cc009ee0 comp rx=0 tx=0).stop 2024-03-30T04:51:22.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.937+0000 7f60e6400700 1 -- 172.21.15.123:0/1511659724 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60e0102e00 msgr2=0x7f60e0115cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:22.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.937+0000 7f60e6400700 1 --2- 172.21.15.123:0/1511659724 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60e0102e00 0x7f60e0115cd0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f60d0009360 tx=0x7f60d0009e30 comp rx=0 tx=0).stop 2024-03-30T04:51:22.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.937+0000 7f60e6400700 1 -- 172.21.15.123:0/1511659724 shutdown_connections 2024-03-30T04:51:22.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.937+0000 7f60e6400700 1 --2- 172.21.15.123:0/1511659724 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f60c406cc30 0x7f60c406f0e0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:22.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.937+0000 7f60e6400700 1 --2- 172.21.15.123:0/1511659724 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60e0102e00 0x7f60e0115cd0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:22.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.937+0000 7f60e6400700 1 --2- 172.21.15.123:0/1511659724 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f60e01037a0 0x7f60e0116210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:22.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.937+0000 7f60e6400700 1 -- 172.21.15.123:0/1511659724 >> 172.21.15.123:0/1511659724 conn(0x7f60e00f89b0 msgr2=0x7f60e00f9980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:51:22.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.937+0000 7f60e6400700 1 -- 172.21.15.123:0/1511659724 shutdown_connections 2024-03-30T04:51:22.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:22.937+0000 7f60e6400700 1 -- 172.21.15.123:0/1511659724 wait complete. 2024-03-30T04:51:22.937 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 32 2024-03-30T04:51:22.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:51:22 smithi123 ceph-mon[139369]: from='client.35130 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:51:22.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:51:22 smithi123 ceph-mon[139369]: pgmap v605: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:22.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:51:22 smithi123 ceph-mon[139369]: from='client.35134 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:51:22.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:51:22 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/4254548171' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:51:23.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.073+0000 7f406be00700 1 -- 172.21.15.123:0/2505686498 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40640708a0 msgr2=0x7f4064070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:23.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.073+0000 7f406be00700 1 --2- 172.21.15.123:0/2505686498 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40640708a0 0x7f4064070d10 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f4050004970 tx=0x7f4050009e30 comp rx=0 tx=0).stop 2024-03-30T04:51:23.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.073+0000 7f406be00700 1 -- 172.21.15.123:0/2505686498 shutdown_connections 2024-03-30T04:51:23.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.073+0000 7f406be00700 1 --2- 172.21.15.123:0/2505686498 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40640708a0 0x7f4064070d10 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:23.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.073+0000 7f406be00700 1 --2- 172.21.15.123:0/2505686498 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4064071f90 0x7f4064070360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:23.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.073+0000 7f406be00700 1 -- 172.21.15.123:0/2505686498 >> 172.21.15.123:0/2505686498 conn(0x7f40640f8970 msgr2=0x7f40640fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:51:23.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.073+0000 7f406be00700 1 -- 172.21.15.123:0/2505686498 shutdown_connections 2024-03-30T04:51:23.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.073+0000 7f406be00700 1 -- 172.21.15.123:0/2505686498 wait complete. 2024-03-30T04:51:23.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.074+0000 7f406be00700 1 Processor -- start 2024-03-30T04:51:23.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.074+0000 7f406be00700 1 -- start start 2024-03-30T04:51:23.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.074+0000 7f406be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40640708a0 0x7f406411a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:51:23.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.074+0000 7f406be00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4064071f90 0x7f406411a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:51:23.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.074+0000 7f406be00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f406411ab90 con 0x7f40640708a0 2024-03-30T04:51:23.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.074+0000 7f406be00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4064114010 con 0x7f4064071f90 2024-03-30T04:51:23.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.075+0000 7f4069400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40640708a0 0x7f406411a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:51:23.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.075+0000 7f4069400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40640708a0 0x7f406411a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60338/0 (socket says 172.21.15.123:60338) 2024-03-30T04:51:23.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.075+0000 7f4069400700 1 -- 172.21.15.123:0/3379646668 learned_addr learned my addr 172.21.15.123:0/3379646668 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:51:23.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.075+0000 7f4068a00700 1 --2- 172.21.15.123:0/3379646668 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4064071f90 0x7f406411a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:51:23.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.075+0000 7f4069400700 1 -- 172.21.15.123:0/3379646668 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4064071f90 msgr2=0x7f406411a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:23.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.075+0000 7f4069400700 1 --2- 172.21.15.123:0/3379646668 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4064071f90 0x7f406411a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:23.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.075+0000 7f4069400700 1 -- 172.21.15.123:0/3379646668 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4050005040 con 0x7f40640708a0 2024-03-30T04:51:23.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.076+0000 7f4069400700 1 --2- 172.21.15.123:0/3379646668 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40640708a0 0x7f406411a000 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f405800eb40 tx=0x7f405800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:51:23.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.076+0000 7f4061600700 1 -- 172.21.15.123:0/3379646668 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4058007cc0 con 0x7f40640708a0 2024-03-30T04:51:23.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.076+0000 7f4061600700 1 -- 172.21.15.123:0/3379646668 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f405800f040 con 0x7f40640708a0 2024-03-30T04:51:23.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.076+0000 7f406be00700 1 -- 172.21.15.123:0/3379646668 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4064114290 con 0x7f40640708a0 2024-03-30T04:51:23.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.076+0000 7f4061600700 1 -- 172.21.15.123:0/3379646668 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f405800c990 con 0x7f40640708a0 2024-03-30T04:51:23.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.076+0000 7f406be00700 1 -- 172.21.15.123:0/3379646668 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40641147a0 con 0x7f40640708a0 2024-03-30T04:51:23.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.077+0000 7f406be00700 1 -- 172.21.15.123:0/3379646668 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4064071aa0 con 0x7f40640708a0 2024-03-30T04:51:23.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.078+0000 7f4061600700 1 -- 172.21.15.123:0/3379646668 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f405800cc00 con 0x7f40640708a0 2024-03-30T04:51:23.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.079+0000 7f4061600700 1 --2- 172.21.15.123:0/3379646668 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f404c06cc30 0x7f404c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:51:23.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.079+0000 7f4061600700 1 -- 172.21.15.123:0/3379646668 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f405808bcd0 con 0x7f40640708a0 2024-03-30T04:51:23.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.079+0000 7f4068a00700 1 --2- 172.21.15.123:0/3379646668 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f404c06cc30 0x7f404c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:51:23.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.080+0000 7f4068a00700 1 --2- 172.21.15.123:0/3379646668 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f404c06cc30 0x7f404c06f0e0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f4050009fd0 tx=0x7f4050009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:51:23.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.083+0000 7f4061600700 1 -- 172.21.15.123:0/3379646668 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4058055cc0 con 0x7f40640708a0 2024-03-30T04:51:23.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:51:22 smithi135 ceph-mon[106011]: from='client.35130 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:51:23.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:51:22 smithi135 ceph-mon[106011]: pgmap v605: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:23.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:51:22 smithi135 ceph-mon[106011]: from='client.35134 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:51:23.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:51:22 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/4254548171' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:51:23.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.303+0000 7f406be00700 1 -- 172.21.15.123:0/3379646668 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f406404bed0 con 0x7f404c06cc30 2024-03-30T04:51:23.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.305+0000 7f4061600700 1 -- 172.21.15.123:0/3379646668 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f406404bed0 con 0x7f404c06cc30 2024-03-30T04:51:23.303 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:51:23.303 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:51:23.303 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:51:23.303 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T04:51:23.303 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:51:23.303 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T04:51:23.303 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T04:51:23.303 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T04:51:23.303 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T04:51:23.304 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:51:23.304 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "15/21 daemons upgraded", 2024-03-30T04:51:23.304 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T04:51:23.304 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:51:23.304 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:51:23.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.307+0000 7f406be00700 1 -- 172.21.15.123:0/3379646668 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f404c06cc30 msgr2=0x7f404c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:23.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.307+0000 7f406be00700 1 --2- 172.21.15.123:0/3379646668 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f404c06cc30 0x7f404c06f0e0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f4050009fd0 tx=0x7f4050009e30 comp rx=0 tx=0).stop 2024-03-30T04:51:23.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.307+0000 7f406be00700 1 -- 172.21.15.123:0/3379646668 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40640708a0 msgr2=0x7f406411a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:23.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.307+0000 7f406be00700 1 --2- 172.21.15.123:0/3379646668 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40640708a0 0x7f406411a000 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f405800eb40 tx=0x7f405800b040 comp rx=0 tx=0).stop 2024-03-30T04:51:23.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.307+0000 7f406be00700 1 -- 172.21.15.123:0/3379646668 shutdown_connections 2024-03-30T04:51:23.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.307+0000 7f406be00700 1 --2- 172.21.15.123:0/3379646668 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f404c06cc30 0x7f404c06f0e0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:23.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.307+0000 7f406be00700 1 --2- 172.21.15.123:0/3379646668 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40640708a0 0x7f406411a000 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:23.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.307+0000 7f406be00700 1 --2- 172.21.15.123:0/3379646668 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4064071f90 0x7f406411a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:23.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.307+0000 7f406be00700 1 -- 172.21.15.123:0/3379646668 >> 172.21.15.123:0/3379646668 conn(0x7f40640f8970 msgr2=0x7f40640f9b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:51:23.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.307+0000 7f406be00700 1 -- 172.21.15.123:0/3379646668 shutdown_connections 2024-03-30T04:51:23.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.307+0000 7f406be00700 1 -- 172.21.15.123:0/3379646668 wait complete. 2024-03-30T04:51:23.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.440+0000 7f69f4200700 1 -- 172.21.15.123:0/4075379818 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69ec102e00 msgr2=0x7f69ec1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:23.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.440+0000 7f69f4200700 1 --2- 172.21.15.123:0/4075379818 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69ec102e00 0x7f69ec1031d0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f69e0004940 tx=0x7f69e0009e30 comp rx=0 tx=0).stop 2024-03-30T04:51:23.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.440+0000 7f69f4200700 1 -- 172.21.15.123:0/4075379818 shutdown_connections 2024-03-30T04:51:23.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.440+0000 7f69f4200700 1 --2- 172.21.15.123:0/4075379818 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f69ec1037a0 0x7f69ec0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:23.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.441+0000 7f69f4200700 1 --2- 172.21.15.123:0/4075379818 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69ec102e00 0x7f69ec1031d0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:23.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.441+0000 7f69f4200700 1 -- 172.21.15.123:0/4075379818 >> 172.21.15.123:0/4075379818 conn(0x7f69ec0f89b0 msgr2=0x7f69ec0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:51:23.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.441+0000 7f69f4200700 1 -- 172.21.15.123:0/4075379818 shutdown_connections 2024-03-30T04:51:23.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.441+0000 7f69f4200700 1 -- 172.21.15.123:0/4075379818 wait complete. 2024-03-30T04:51:23.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.441+0000 7f69f4200700 1 Processor -- start 2024-03-30T04:51:23.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.442+0000 7f69f4200700 1 -- start start 2024-03-30T04:51:23.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.442+0000 7f69f4200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f69ec102e00 0x7f69ec115d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:51:23.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.442+0000 7f69f4200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69ec1037a0 0x7f69ec116250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:51:23.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.442+0000 7f69f4200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69ec1168a0 con 0x7f69ec1037a0 2024-03-30T04:51:23.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.442+0000 7f69f4200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69ec10fc70 con 0x7f69ec102e00 2024-03-30T04:51:23.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.442+0000 7f69f0e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69ec1037a0 0x7f69ec116250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:51:23.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.442+0000 7f69f0e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69ec1037a0 0x7f69ec116250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60354/0 (socket says 172.21.15.123:60354) 2024-03-30T04:51:23.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.442+0000 7f69f0e00700 1 -- 172.21.15.123:0/4108982206 learned_addr learned my addr 172.21.15.123:0/4108982206 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:51:23.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.442+0000 7f69f1800700 1 --2- 172.21.15.123:0/4108982206 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f69ec102e00 0x7f69ec115d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:51:23.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.443+0000 7f69f0e00700 1 -- 172.21.15.123:0/4108982206 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f69ec102e00 msgr2=0x7f69ec115d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:23.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.443+0000 7f69f0e00700 1 --2- 172.21.15.123:0/4108982206 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f69ec102e00 0x7f69ec115d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:23.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.443+0000 7f69f0e00700 1 -- 172.21.15.123:0/4108982206 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69e0005040 con 0x7f69ec1037a0 2024-03-30T04:51:23.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.443+0000 7f69f1800700 1 --2- 172.21.15.123:0/4108982206 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f69ec102e00 0x7f69ec115d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:51:23.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.443+0000 7f69f0e00700 1 --2- 172.21.15.123:0/4108982206 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69ec1037a0 0x7f69ec116250 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f69d8002a50 tx=0x7f69d8004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:51:23.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.443+0000 7f69e9600700 1 -- 172.21.15.123:0/4108982206 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69d8007cc0 con 0x7f69ec1037a0 2024-03-30T04:51:23.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.443+0000 7f69e9600700 1 -- 172.21.15.123:0/4108982206 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f69d800f040 con 0x7f69ec1037a0 2024-03-30T04:51:23.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.443+0000 7f69e9600700 1 -- 172.21.15.123:0/4108982206 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69d800d990 con 0x7f69ec1037a0 2024-03-30T04:51:23.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.443+0000 7f69f4200700 1 -- 172.21.15.123:0/4108982206 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69ec10ff50 con 0x7f69ec1037a0 2024-03-30T04:51:23.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.443+0000 7f69f4200700 1 -- 172.21.15.123:0/4108982206 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69ec110510 con 0x7f69ec1037a0 2024-03-30T04:51:23.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.444+0000 7f69f4200700 1 -- 172.21.15.123:0/4108982206 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69ec0fe550 con 0x7f69ec1037a0 2024-03-30T04:51:23.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.445+0000 7f69e9600700 1 -- 172.21.15.123:0/4108982206 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f69d8009710 con 0x7f69ec1037a0 2024-03-30T04:51:23.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.447+0000 7f69e9600700 1 --2- 172.21.15.123:0/4108982206 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f69d406cc30 0x7f69d406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:51:23.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.447+0000 7f69e9600700 1 -- 172.21.15.123:0/4108982206 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f69d808bda0 con 0x7f69ec1037a0 2024-03-30T04:51:23.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.447+0000 7f69f1800700 1 --2- 172.21.15.123:0/4108982206 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f69d406cc30 0x7f69d406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:51:23.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.448+0000 7f69f1800700 1 --2- 172.21.15.123:0/4108982206 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f69d406cc30 0x7f69d406f0e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f69e0000c00 tx=0x7f69e0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:51:23.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.451+0000 7f69e9600700 1 -- 172.21.15.123:0/4108982206 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f69d80595e0 con 0x7f69ec1037a0 2024-03-30T04:51:23.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.741+0000 7f69f4200700 1 -- 172.21.15.123:0/4108982206 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f69ec04bed0 con 0x7f69ec1037a0 2024-03-30T04:51:23.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.742+0000 7f69e9600700 1 -- 172.21.15.123:0/4108982206 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f69d8059400 con 0x7f69ec1037a0 2024-03-30T04:51:23.740 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T04:51:23.740 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T04:51:23.740 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T04:51:23.740 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T04:51:23.740 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T04:51:23.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.744+0000 7f69f4200700 1 -- 172.21.15.123:0/4108982206 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f69d406cc30 msgr2=0x7f69d406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:23.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.744+0000 7f69f4200700 1 --2- 172.21.15.123:0/4108982206 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f69d406cc30 0x7f69d406f0e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f69e0000c00 tx=0x7f69e0009e30 comp rx=0 tx=0).stop 2024-03-30T04:51:23.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.744+0000 7f69f4200700 1 -- 172.21.15.123:0/4108982206 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69ec1037a0 msgr2=0x7f69ec116250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:23.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.744+0000 7f69f4200700 1 --2- 172.21.15.123:0/4108982206 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69ec1037a0 0x7f69ec116250 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f69d8002a50 tx=0x7f69d8004600 comp rx=0 tx=0).stop 2024-03-30T04:51:23.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.744+0000 7f69f4200700 1 -- 172.21.15.123:0/4108982206 shutdown_connections 2024-03-30T04:51:23.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.744+0000 7f69f4200700 1 --2- 172.21.15.123:0/4108982206 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f69d406cc30 0x7f69d406f0e0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:23.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.744+0000 7f69f4200700 1 --2- 172.21.15.123:0/4108982206 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f69ec102e00 0x7f69ec115d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:23.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.744+0000 7f69f4200700 1 --2- 172.21.15.123:0/4108982206 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69ec1037a0 0x7f69ec116250 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:23.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.744+0000 7f69f4200700 1 -- 172.21.15.123:0/4108982206 >> 172.21.15.123:0/4108982206 conn(0x7f69ec0f89b0 msgr2=0x7f69ec0704f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:51:23.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.744+0000 7f69f4200700 1 -- 172.21.15.123:0/4108982206 shutdown_connections 2024-03-30T04:51:23.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:23.744+0000 7f69f4200700 1 -- 172.21.15.123:0/4108982206 wait complete. 2024-03-30T04:51:23.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:51:23 smithi123 ceph-mon[139369]: from='client.35138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:51:23.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:51:23 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/1511659724' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:51:24.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:51:23 smithi135 ceph-mon[106011]: from='client.35138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:51:24.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:51:23 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/1511659724' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:51:25.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:51:24 smithi135 ceph-mon[106011]: from='client.35150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:51:25.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:51:24 smithi135 ceph-mon[106011]: pgmap v606: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:25.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:51:24 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/4108982206' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:51:25.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:51:24 smithi123 ceph-mon[139369]: from='client.35150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:51:25.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:51:24 smithi123 ceph-mon[139369]: pgmap v606: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:25.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:51:24 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/4108982206' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:51:27.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:51:26 smithi135 ceph-mon[106011]: pgmap v607: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:27.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:51:26 smithi123 ceph-mon[139369]: pgmap v607: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:29.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:51:28 smithi135 ceph-mon[106011]: pgmap v608: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:29.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:51:28 smithi123 ceph-mon[139369]: pgmap v608: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:30.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:51:29 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:51:30.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:51:29 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:51:31.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:51:30 smithi135 ceph-mon[106011]: pgmap v609: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:31.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:51:30 smithi123 ceph-mon[139369]: pgmap v609: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:33.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:51:32 smithi135 ceph-mon[106011]: pgmap v610: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:33.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:51:32 smithi123 ceph-mon[139369]: pgmap v610: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:35.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:51:34 smithi135 ceph-mon[106011]: pgmap v611: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:35.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:51:34 smithi123 ceph-mon[139369]: pgmap v611: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:37.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:51:36 smithi135 ceph-mon[106011]: pgmap v612: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:37.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:51:36 smithi123 ceph-mon[139369]: pgmap v612: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:39.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:51:38 smithi135 ceph-mon[106011]: pgmap v613: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:39.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:51:38 smithi123 ceph-mon[139369]: pgmap v613: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:40.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:51:40 smithi123 ceph-mon[139369]: pgmap v614: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:41.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:51:40 smithi135 ceph-mon[106011]: pgmap v614: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:43.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:51:42 smithi135 ceph-mon[106011]: pgmap v615: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:43.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:51:42 smithi123 ceph-mon[139369]: pgmap v615: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:45.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:51:44 smithi135 ceph-mon[106011]: pgmap v616: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:45.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:51:44 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:51:45.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:51:44 smithi123 ceph-mon[139369]: pgmap v616: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:45.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:51:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:51:47.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:51:46 smithi135 ceph-mon[106011]: pgmap v617: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:47.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:51:46 smithi123 ceph-mon[139369]: pgmap v617: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:49.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:51:48 smithi135 ceph-mon[106011]: pgmap v618: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:49.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:51:48 smithi123 ceph-mon[139369]: pgmap v618: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:50.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:51:50 smithi123 ceph-mon[139369]: pgmap v619: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:51.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:51:50 smithi135 ceph-mon[106011]: pgmap v619: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:53.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:51:52 smithi135 ceph-mon[106011]: pgmap v620: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:53.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:51:52 smithi123 ceph-mon[139369]: pgmap v620: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:53.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:53.891+0000 7f0da5c00700 1 -- 172.21.15.123:0/3665027988 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0da01037a0 msgr2=0x7f0da00fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:53.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:53.891+0000 7f0da5c00700 1 --2- 172.21.15.123:0/3665027988 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0da01037a0 0x7f0da00fce10 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c004940 tx=0x7f0d8c009e30 comp rx=0 tx=0).stop 2024-03-30T04:51:53.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:53.892+0000 7f0da5c00700 1 -- 172.21.15.123:0/3665027988 shutdown_connections 2024-03-30T04:51:53.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:53.892+0000 7f0da5c00700 1 --2- 172.21.15.123:0/3665027988 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0da01037a0 0x7f0da00fce10 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:53.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:53.892+0000 7f0da5c00700 1 --2- 172.21.15.123:0/3665027988 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0da0102e00 0x7f0da01031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:53.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:53.892+0000 7f0da5c00700 1 -- 172.21.15.123:0/3665027988 >> 172.21.15.123:0/3665027988 conn(0x7f0da00f89b0 msgr2=0x7f0da00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:51:53.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:53.892+0000 7f0da5c00700 1 -- 172.21.15.123:0/3665027988 shutdown_connections 2024-03-30T04:51:53.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:53.892+0000 7f0da5c00700 1 -- 172.21.15.123:0/3665027988 wait complete. 2024-03-30T04:51:53.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:53.892+0000 7f0da5c00700 1 Processor -- start 2024-03-30T04:51:53.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:53.893+0000 7f0da5c00700 1 -- start start 2024-03-30T04:51:53.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:53.893+0000 7f0da5c00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0da0102e00 0x7f0da0115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:51:53.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:53.893+0000 7f0da5c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0da01037a0 0x7f0da0116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:51:53.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:53.893+0000 7f0da5c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0da0116880 con 0x7f0da01037a0 2024-03-30T04:51:53.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:53.893+0000 7f0da5c00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0da010fc50 con 0x7f0da0102e00 2024-03-30T04:51:53.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:53.893+0000 7f0d9e000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0da01037a0 0x7f0da0116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:51:53.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:53.893+0000 7f0d9e000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0da01037a0 0x7f0da0116230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52184/0 (socket says 172.21.15.123:52184) 2024-03-30T04:51:53.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:53.893+0000 7f0d9e000700 1 -- 172.21.15.123:0/2465482738 learned_addr learned my addr 172.21.15.123:0/2465482738 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:51:53.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:53.894+0000 7f0d9e000700 1 -- 172.21.15.123:0/2465482738 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0da0102e00 msgr2=0x7f0da0115cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:53.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:53.894+0000 7f0d9e000700 1 --2- 172.21.15.123:0/2465482738 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0da0102e00 0x7f0da0115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:53.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:53.894+0000 7f0d9e000700 1 -- 172.21.15.123:0/2465482738 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d8c005040 con 0x7f0da01037a0 2024-03-30T04:51:53.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:53.894+0000 7f0d9e000700 1 --2- 172.21.15.123:0/2465482738 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0da01037a0 0x7f0da0116230 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c009390 tx=0x7f0d8c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:51:53.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:53.894+0000 7f0d96a00700 1 -- 172.21.15.123:0/2465482738 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d8c00e040 con 0x7f0da01037a0 2024-03-30T04:51:53.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:53.894+0000 7f0d96a00700 1 -- 172.21.15.123:0/2465482738 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0d8c013420 con 0x7f0da01037a0 2024-03-30T04:51:53.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:53.894+0000 7f0da5c00700 1 -- 172.21.15.123:0/2465482738 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0da010fed0 con 0x7f0da01037a0 2024-03-30T04:51:53.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:53.894+0000 7f0d96a00700 1 -- 172.21.15.123:0/2465482738 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d8c01c630 con 0x7f0da01037a0 2024-03-30T04:51:53.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:53.894+0000 7f0da5c00700 1 -- 172.21.15.123:0/2465482738 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0da0110490 con 0x7f0da01037a0 2024-03-30T04:51:53.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:53.895+0000 7f0da5c00700 1 -- 172.21.15.123:0/2465482738 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0da00fe550 con 0x7f0da01037a0 2024-03-30T04:51:53.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:53.896+0000 7f0d96a00700 1 -- 172.21.15.123:0/2465482738 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f0d8c0139e0 con 0x7f0da01037a0 2024-03-30T04:51:53.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:53.897+0000 7f0d96a00700 1 --2- 172.21.15.123:0/2465482738 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0d8406cc30 0x7f0d8406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:51:53.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:53.897+0000 7f0d96a00700 1 -- 172.21.15.123:0/2465482738 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f0d8c08d360 con 0x7f0da01037a0 2024-03-30T04:51:53.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:53.897+0000 7f0d9ea00700 1 --2- 172.21.15.123:0/2465482738 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0d8406cc30 0x7f0d8406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:51:53.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:53.898+0000 7f0d9ea00700 1 --2- 172.21.15.123:0/2465482738 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0d8406cc30 0x7f0d8406f0e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f0d90004b70 tx=0x7f0d90007cc0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:51:53.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:53.902+0000 7f0d96a00700 1 -- 172.21.15.123:0/2465482738 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0d8c05adb0 con 0x7f0da01037a0 2024-03-30T04:51:54.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.125+0000 7f0da5c00700 1 -- 172.21.15.123:0/2465482738 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0da0109da0 con 0x7f0d8406cc30 2024-03-30T04:51:54.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.126+0000 7f0d96a00700 1 -- 172.21.15.123:0/2465482738 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f0da0109da0 con 0x7f0d8406cc30 2024-03-30T04:51:54.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.129+0000 7f0da5c00700 1 -- 172.21.15.123:0/2465482738 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0d8406cc30 msgr2=0x7f0d8406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:54.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.129+0000 7f0da5c00700 1 --2- 172.21.15.123:0/2465482738 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0d8406cc30 0x7f0d8406f0e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f0d90004b70 tx=0x7f0d90007cc0 comp rx=0 tx=0).stop 2024-03-30T04:51:54.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.129+0000 7f0da5c00700 1 -- 172.21.15.123:0/2465482738 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0da01037a0 msgr2=0x7f0da0116230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:54.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.129+0000 7f0da5c00700 1 --2- 172.21.15.123:0/2465482738 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0da01037a0 0x7f0da0116230 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c009390 tx=0x7f0d8c009e30 comp rx=0 tx=0).stop 2024-03-30T04:51:54.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.129+0000 7f0da5c00700 1 -- 172.21.15.123:0/2465482738 shutdown_connections 2024-03-30T04:51:54.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.129+0000 7f0da5c00700 1 --2- 172.21.15.123:0/2465482738 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0d8406cc30 0x7f0d8406f0e0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:54.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.129+0000 7f0da5c00700 1 --2- 172.21.15.123:0/2465482738 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0da0102e00 0x7f0da0115cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:54.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.129+0000 7f0da5c00700 1 --2- 172.21.15.123:0/2465482738 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0da01037a0 0x7f0da0116230 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:54.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.129+0000 7f0da5c00700 1 -- 172.21.15.123:0/2465482738 >> 172.21.15.123:0/2465482738 conn(0x7f0da00f89b0 msgr2=0x7f0da00f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:51:54.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.129+0000 7f0da5c00700 1 -- 172.21.15.123:0/2465482738 shutdown_connections 2024-03-30T04:51:54.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.129+0000 7f0da5c00700 1 -- 172.21.15.123:0/2465482738 wait complete. 2024-03-30T04:51:54.142 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:51:54.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.264+0000 7f32b3400700 1 -- 172.21.15.123:0/3491264441 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32ac102e00 msgr2=0x7f32ac1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:54.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.264+0000 7f32b3400700 1 --2- 172.21.15.123:0/3491264441 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32ac102e00 0x7f32ac1031d0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f32a0004940 tx=0x7f32a0009e30 comp rx=0 tx=0).stop 2024-03-30T04:51:54.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.264+0000 7f32b3400700 1 -- 172.21.15.123:0/3491264441 shutdown_connections 2024-03-30T04:51:54.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.264+0000 7f32b3400700 1 --2- 172.21.15.123:0/3491264441 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f32ac1037a0 0x7f32ac0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:54.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.264+0000 7f32b3400700 1 --2- 172.21.15.123:0/3491264441 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32ac102e00 0x7f32ac1031d0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:54.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.264+0000 7f32b3400700 1 -- 172.21.15.123:0/3491264441 >> 172.21.15.123:0/3491264441 conn(0x7f32ac0f89b0 msgr2=0x7f32ac0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:51:54.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.264+0000 7f32b3400700 1 -- 172.21.15.123:0/3491264441 shutdown_connections 2024-03-30T04:51:54.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.265+0000 7f32b3400700 1 -- 172.21.15.123:0/3491264441 wait complete. 2024-03-30T04:51:54.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.265+0000 7f32b3400700 1 Processor -- start 2024-03-30T04:51:54.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.265+0000 7f32b3400700 1 -- start start 2024-03-30T04:51:54.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.266+0000 7f32b3400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32ac102e00 0x7f32ac0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:51:54.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.266+0000 7f32b3400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f32ac1037a0 0x7f32ac070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:51:54.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.266+0000 7f32b3400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32ac070d90 con 0x7f32ac102e00 2024-03-30T04:51:54.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.266+0000 7f32b3400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32ac070ed0 con 0x7f32ac1037a0 2024-03-30T04:51:54.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.266+0000 7f32b0a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32ac102e00 0x7f32ac0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:51:54.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.266+0000 7f32b0a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32ac102e00 0x7f32ac0702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52206/0 (socket says 172.21.15.123:52206) 2024-03-30T04:51:54.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.266+0000 7f32b0a00700 1 -- 172.21.15.123:0/1546579968 learned_addr learned my addr 172.21.15.123:0/1546579968 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:51:54.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.266+0000 7f32abe00700 1 --2- 172.21.15.123:0/1546579968 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f32ac1037a0 0x7f32ac070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:51:54.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.266+0000 7f32b0a00700 1 -- 172.21.15.123:0/1546579968 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f32ac1037a0 msgr2=0x7f32ac070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:54.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.266+0000 7f32b0a00700 1 --2- 172.21.15.123:0/1546579968 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f32ac1037a0 0x7f32ac070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:54.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.267+0000 7f32b0a00700 1 -- 172.21.15.123:0/1546579968 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a0005040 con 0x7f32ac102e00 2024-03-30T04:51:54.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.267+0000 7f32b0a00700 1 --2- 172.21.15.123:0/1546579968 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32ac102e00 0x7f32ac0702c0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f32a0009390 tx=0x7f32a0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:51:54.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.267+0000 7f32a8c00700 1 -- 172.21.15.123:0/1546579968 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32a000e040 con 0x7f32ac102e00 2024-03-30T04:51:54.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.267+0000 7f32a8c00700 1 -- 172.21.15.123:0/1546579968 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f32a0013420 con 0x7f32ac102e00 2024-03-30T04:51:54.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.267+0000 7f32b3400700 1 -- 172.21.15.123:0/1546579968 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32ac071120 con 0x7f32ac102e00 2024-03-30T04:51:54.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.267+0000 7f32a8c00700 1 -- 172.21.15.123:0/1546579968 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32a001c630 con 0x7f32ac102e00 2024-03-30T04:51:54.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.267+0000 7f32b3400700 1 -- 172.21.15.123:0/1546579968 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32ac11a250 con 0x7f32ac102e00 2024-03-30T04:51:54.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.268+0000 7f32b3400700 1 -- 172.21.15.123:0/1546579968 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32ac0fe550 con 0x7f32ac102e00 2024-03-30T04:51:54.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.269+0000 7f32a8c00700 1 -- 172.21.15.123:0/1546579968 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f32a00139e0 con 0x7f32ac102e00 2024-03-30T04:51:54.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.270+0000 7f32a8c00700 1 --2- 172.21.15.123:0/1546579968 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f329406cc30 0x7f329406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:51:54.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.270+0000 7f32a8c00700 1 -- 172.21.15.123:0/1546579968 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f32a008d310 con 0x7f32ac102e00 2024-03-30T04:51:54.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.270+0000 7f32abe00700 1 --2- 172.21.15.123:0/1546579968 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f329406cc30 0x7f329406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:51:54.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.271+0000 7f32abe00700 1 --2- 172.21.15.123:0/1546579968 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f329406cc30 0x7f329406f0e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f32ac071c00 tx=0x7f3298009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:51:54.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.275+0000 7f32a8c00700 1 -- 172.21.15.123:0/1546579968 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f32a00572d0 con 0x7f32ac102e00 2024-03-30T04:51:54.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.497+0000 7f32b3400700 1 -- 172.21.15.123:0/1546579968 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f32ac0719b0 con 0x7f329406cc30 2024-03-30T04:51:54.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.498+0000 7f32a8c00700 1 -- 172.21.15.123:0/1546579968 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f32ac0719b0 con 0x7f329406cc30 2024-03-30T04:51:54.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.500+0000 7f32b3400700 1 -- 172.21.15.123:0/1546579968 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f329406cc30 msgr2=0x7f329406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:54.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.501+0000 7f32b3400700 1 --2- 172.21.15.123:0/1546579968 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f329406cc30 0x7f329406f0e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f32ac071c00 tx=0x7f3298009ee0 comp rx=0 tx=0).stop 2024-03-30T04:51:54.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.501+0000 7f32b3400700 1 -- 172.21.15.123:0/1546579968 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32ac102e00 msgr2=0x7f32ac0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:54.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.501+0000 7f32b3400700 1 --2- 172.21.15.123:0/1546579968 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32ac102e00 0x7f32ac0702c0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f32a0009390 tx=0x7f32a0009e30 comp rx=0 tx=0).stop 2024-03-30T04:51:54.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.501+0000 7f32b3400700 1 -- 172.21.15.123:0/1546579968 shutdown_connections 2024-03-30T04:51:54.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.501+0000 7f32b3400700 1 --2- 172.21.15.123:0/1546579968 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f329406cc30 0x7f329406f0e0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:54.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.501+0000 7f32b3400700 1 --2- 172.21.15.123:0/1546579968 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32ac102e00 0x7f32ac0702c0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:54.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.501+0000 7f32b3400700 1 --2- 172.21.15.123:0/1546579968 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f32ac1037a0 0x7f32ac070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:54.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.501+0000 7f32b3400700 1 -- 172.21.15.123:0/1546579968 >> 172.21.15.123:0/1546579968 conn(0x7f32ac0f89b0 msgr2=0x7f32ac0f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:51:54.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.501+0000 7f32b3400700 1 -- 172.21.15.123:0/1546579968 shutdown_connections 2024-03-30T04:51:54.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.501+0000 7f32b3400700 1 -- 172.21.15.123:0/1546579968 wait complete. 2024-03-30T04:51:54.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.634+0000 7fec3b000700 1 -- 172.21.15.123:0/1127598301 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec34104ff0 msgr2=0x7fec341053c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:54.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.634+0000 7fec3b000700 1 --2- 172.21.15.123:0/1127598301 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec34104ff0 0x7fec341053c0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fec24004940 tx=0x7fec24009e30 comp rx=0 tx=0).stop 2024-03-30T04:51:54.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.635+0000 7fec3b000700 1 -- 172.21.15.123:0/1127598301 shutdown_connections 2024-03-30T04:51:54.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.635+0000 7fec3b000700 1 --2- 172.21.15.123:0/1127598301 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fec34105990 0x7fec340ff000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:54.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.635+0000 7fec3b000700 1 --2- 172.21.15.123:0/1127598301 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec34104ff0 0x7fec341053c0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:54.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.635+0000 7fec3b000700 1 -- 172.21.15.123:0/1127598301 >> 172.21.15.123:0/1127598301 conn(0x7fec340faba0 msgr2=0x7fec340fcfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:51:54.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.635+0000 7fec3b000700 1 -- 172.21.15.123:0/1127598301 shutdown_connections 2024-03-30T04:51:54.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.635+0000 7fec3b000700 1 -- 172.21.15.123:0/1127598301 wait complete. 2024-03-30T04:51:54.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.635+0000 7fec3b000700 1 Processor -- start 2024-03-30T04:51:54.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.636+0000 7fec3b000700 1 -- start start 2024-03-30T04:51:54.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.636+0000 7fec3b000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec34104ff0 0x7fec34115a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:51:54.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.636+0000 7fec3b000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fec34105990 0x7fec34115f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:51:54.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.636+0000 7fec3b000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec34116590 con 0x7fec34104ff0 2024-03-30T04:51:54.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.636+0000 7fec3b000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec3410fb90 con 0x7fec34105990 2024-03-30T04:51:54.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.636+0000 7fec33e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec34104ff0 0x7fec34115a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:51:54.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.636+0000 7fec33e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec34104ff0 0x7fec34115a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52228/0 (socket says 172.21.15.123:52228) 2024-03-30T04:51:54.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.636+0000 7fec33e00700 1 -- 172.21.15.123:0/3770230639 learned_addr learned my addr 172.21.15.123:0/3770230639 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:51:54.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.637+0000 7fec33400700 1 --2- 172.21.15.123:0/3770230639 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fec34105990 0x7fec34115f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:51:54.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.637+0000 7fec33e00700 1 -- 172.21.15.123:0/3770230639 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fec34105990 msgr2=0x7fec34115f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:54.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.637+0000 7fec33e00700 1 --2- 172.21.15.123:0/3770230639 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fec34105990 0x7fec34115f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:54.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.637+0000 7fec33e00700 1 -- 172.21.15.123:0/3770230639 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec24005040 con 0x7fec34104ff0 2024-03-30T04:51:54.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.637+0000 7fec33400700 1 --2- 172.21.15.123:0/3770230639 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fec34105990 0x7fec34115f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:51:54.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.637+0000 7fec33e00700 1 --2- 172.21.15.123:0/3770230639 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec34104ff0 0x7fec34115a00 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fec24008d80 tx=0x7fec24009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:51:54.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.637+0000 7fec2be00700 1 -- 172.21.15.123:0/3770230639 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec2400e030 con 0x7fec34104ff0 2024-03-30T04:51:54.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.637+0000 7fec2be00700 1 -- 172.21.15.123:0/3770230639 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fec24013de0 con 0x7fec34104ff0 2024-03-30T04:51:54.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.637+0000 7fec2be00700 1 -- 172.21.15.123:0/3770230639 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec2401cd60 con 0x7fec34104ff0 2024-03-30T04:51:54.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.637+0000 7fec3b000700 1 -- 172.21.15.123:0/3770230639 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec3410fe10 con 0x7fec34104ff0 2024-03-30T04:51:54.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.637+0000 7fec3b000700 1 -- 172.21.15.123:0/3770230639 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec341103d0 con 0x7fec34104ff0 2024-03-30T04:51:54.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.638+0000 7fec3b000700 1 -- 172.21.15.123:0/3770230639 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec34100740 con 0x7fec34104ff0 2024-03-30T04:51:54.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.639+0000 7fec2be00700 1 -- 172.21.15.123:0/3770230639 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fec2401c590 con 0x7fec34104ff0 2024-03-30T04:51:54.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.640+0000 7fec2be00700 1 --2- 172.21.15.123:0/3770230639 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fec1806cc20 0x7fec1806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:51:54.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.640+0000 7fec2be00700 1 -- 172.21.15.123:0/3770230639 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fec2408d8b0 con 0x7fec34104ff0 2024-03-30T04:51:54.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.640+0000 7fec33400700 1 --2- 172.21.15.123:0/3770230639 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fec1806cc20 0x7fec1806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:51:54.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.641+0000 7fec33400700 1 --2- 172.21.15.123:0/3770230639 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fec1806cc20 0x7fec1806f0d0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fec20004600 tx=0x7fec2000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:51:54.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.645+0000 7fec2be00700 1 -- 172.21.15.123:0/3770230639 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fec2405b380 con 0x7fec34104ff0 2024-03-30T04:51:54.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.858+0000 7fec3b000700 1 -- 172.21.15.123:0/3770230639 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fec34111050 con 0x7fec1806cc20 2024-03-30T04:51:54.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.868+0000 7fec2be00700 1 -- 172.21.15.123:0/3770230639 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fec34111050 con 0x7fec1806cc20 2024-03-30T04:51:54.866 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:51:54.866 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (21m) 13m ago 33m 15.0M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:51:54.866 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (18m) 13m ago 33m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T04:51:54.866 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (18m) 13m ago 32m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T04:51:54.866 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (20m) 13m ago 33m 74.0M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T04:51:54.866 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (14m) 13m ago 27m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T04:51:54.867 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (14m) 13m ago 27m 10.9M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T04:51:54.867 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (27m) 13m ago 27m 10.7M - 17.2.7-797-g5c024335 f77085f6aab7 0dc41326818c 2024-03-30T04:51:54.867 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (13m) 13m ago 27m 9747k - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T04:51:54.867 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (23m) 13m ago 35m 506M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:51:54.867 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (22m) 13m ago 32m 450M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:51:54.867 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (19m) 13m ago 35m 53.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T04:51:54.867 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (19m) 13m ago 32m 42.6M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T04:51:54.867 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (21m) 13m ago 33m 8052k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:51:54.867 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (21m) 13m ago 32m 7880k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:51:54.867 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (17m) 13m ago 31m 104M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T04:51:54.867 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (17m) 13m ago 31m 93.4M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T04:51:54.867 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (16m) 13m ago 30m 86.3M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T04:51:54.867 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (16m) 13m ago 29m 97.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T04:51:54.867 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (15m) 13m ago 29m 97.9M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T04:51:54.867 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (14m) 13m ago 28m 85.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T04:51:54.867 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (21m) 13m ago 33m 53.7M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:51:54.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.870+0000 7fec3b000700 1 -- 172.21.15.123:0/3770230639 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fec1806cc20 msgr2=0x7fec1806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:54.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.870+0000 7fec3b000700 1 --2- 172.21.15.123:0/3770230639 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fec1806cc20 0x7fec1806f0d0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fec20004600 tx=0x7fec2000b040 comp rx=0 tx=0).stop 2024-03-30T04:51:54.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.870+0000 7fec3b000700 1 -- 172.21.15.123:0/3770230639 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec34104ff0 msgr2=0x7fec34115a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:54.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.870+0000 7fec3b000700 1 --2- 172.21.15.123:0/3770230639 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec34104ff0 0x7fec34115a00 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fec24008d80 tx=0x7fec24009e30 comp rx=0 tx=0).stop 2024-03-30T04:51:54.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.870+0000 7fec3b000700 1 -- 172.21.15.123:0/3770230639 shutdown_connections 2024-03-30T04:51:54.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.870+0000 7fec3b000700 1 --2- 172.21.15.123:0/3770230639 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fec1806cc20 0x7fec1806f0d0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:54.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.870+0000 7fec3b000700 1 --2- 172.21.15.123:0/3770230639 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec34104ff0 0x7fec34115a00 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:54.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.870+0000 7fec3b000700 1 --2- 172.21.15.123:0/3770230639 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fec34105990 0x7fec34115f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:54.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.870+0000 7fec3b000700 1 -- 172.21.15.123:0/3770230639 >> 172.21.15.123:0/3770230639 conn(0x7fec340faba0 msgr2=0x7fec340fbb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:51:54.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.870+0000 7fec3b000700 1 -- 172.21.15.123:0/3770230639 shutdown_connections 2024-03-30T04:51:54.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:54.870+0000 7fec3b000700 1 -- 172.21.15.123:0/3770230639 wait complete. 2024-03-30T04:51:55.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.003+0000 7fd522c00700 1 -- 172.21.15.123:0/2771742891 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd51c102710 msgr2=0x7fd51c0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:55.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.003+0000 7fd522c00700 1 --2- 172.21.15.123:0/2771742891 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd51c102710 0x7fd51c0fbd80 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fd508004940 tx=0x7fd508009e30 comp rx=0 tx=0).stop 2024-03-30T04:51:55.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.003+0000 7fd522c00700 1 -- 172.21.15.123:0/2771742891 shutdown_connections 2024-03-30T04:51:55.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.003+0000 7fd522c00700 1 --2- 172.21.15.123:0/2771742891 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd51c102710 0x7fd51c0fbd80 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:55.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.003+0000 7fd522c00700 1 --2- 172.21.15.123:0/2771742891 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd51c101d70 0x7fd51c102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:55.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.003+0000 7fd522c00700 1 -- 172.21.15.123:0/2771742891 >> 172.21.15.123:0/2771742891 conn(0x7fd51c072bc0 msgr2=0x7fd51c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:51:55.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.003+0000 7fd522c00700 1 -- 172.21.15.123:0/2771742891 shutdown_connections 2024-03-30T04:51:55.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.003+0000 7fd522c00700 1 -- 172.21.15.123:0/2771742891 wait complete. 2024-03-30T04:51:55.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.004+0000 7fd522c00700 1 Processor -- start 2024-03-30T04:51:55.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.004+0000 7fd522c00700 1 -- start start 2024-03-30T04:51:55.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.004+0000 7fd522c00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd51c101d70 0x7fd51c115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:51:55.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.004+0000 7fd522c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd51c102710 0x7fd51c116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:51:55.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.004+0000 7fd522c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd51c116790 con 0x7fd51c102710 2024-03-30T04:51:55.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.004+0000 7fd522c00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd51c10fb80 con 0x7fd51c101d70 2024-03-30T04:51:55.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.004+0000 7fd51b400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd51c102710 0x7fd51c116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:51:55.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.004+0000 7fd51b400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd51c102710 0x7fd51c116140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52244/0 (socket says 172.21.15.123:52244) 2024-03-30T04:51:55.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.004+0000 7fd51b400700 1 -- 172.21.15.123:0/3521432180 learned_addr learned my addr 172.21.15.123:0/3521432180 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:51:55.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.005+0000 7fd51b400700 1 -- 172.21.15.123:0/3521432180 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd51c101d70 msgr2=0x7fd51c115c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:55.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.005+0000 7fd51b400700 1 --2- 172.21.15.123:0/3521432180 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd51c101d70 0x7fd51c115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:55.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.005+0000 7fd51b400700 1 -- 172.21.15.123:0/3521432180 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd508005040 con 0x7fd51c102710 2024-03-30T04:51:55.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.005+0000 7fd51b400700 1 --2- 172.21.15.123:0/3521432180 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd51c102710 0x7fd51c116140 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fd508009fd0 tx=0x7fd508009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:51:55.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.005+0000 7fd513e00700 1 -- 172.21.15.123:0/3521432180 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd50800e030 con 0x7fd51c102710 2024-03-30T04:51:55.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.005+0000 7fd513e00700 1 -- 172.21.15.123:0/3521432180 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd508013e00 con 0x7fd51c102710 2024-03-30T04:51:55.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.005+0000 7fd513e00700 1 -- 172.21.15.123:0/3521432180 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd50801ccd0 con 0x7fd51c102710 2024-03-30T04:51:55.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.005+0000 7fd522c00700 1 -- 172.21.15.123:0/3521432180 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd51c10fe00 con 0x7fd51c102710 2024-03-30T04:51:55.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.005+0000 7fd522c00700 1 -- 172.21.15.123:0/3521432180 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd51c1103c0 con 0x7fd51c102710 2024-03-30T04:51:55.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.006+0000 7fd522c00700 1 -- 172.21.15.123:0/3521432180 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd51c0fd4c0 con 0x7fd51c102710 2024-03-30T04:51:55.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.007+0000 7fd513e00700 1 -- 172.21.15.123:0/3521432180 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fd50801c500 con 0x7fd51c102710 2024-03-30T04:51:55.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.009+0000 7fd513e00700 1 --2- 172.21.15.123:0/3521432180 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd50006cc30 0x7fd50006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:51:55.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.009+0000 7fd513e00700 1 -- 172.21.15.123:0/3521432180 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd50808d8b0 con 0x7fd51c102710 2024-03-30T04:51:55.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.009+0000 7fd51be00700 1 --2- 172.21.15.123:0/3521432180 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd50006cc30 0x7fd50006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:51:55.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.009+0000 7fd51be00700 1 --2- 172.21.15.123:0/3521432180 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd50006cc30 0x7fd50006f0e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd50c009fd0 tx=0x7fd50c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:51:55.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.013+0000 7fd513e00700 1 -- 172.21.15.123:0/3521432180 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd50805b380 con 0x7fd51c102710 2024-03-30T04:51:55.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:51:54 smithi135 ceph-mon[106011]: pgmap v621: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:55.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:51:54 smithi123 ceph-mon[139369]: pgmap v621: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:55.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.295+0000 7fd522c00700 1 -- 172.21.15.123:0/3521432180 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fd51c108d10 con 0x7fd51c102710 2024-03-30T04:51:55.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.296+0000 7fd513e00700 1 -- 172.21.15.123:0/3521432180 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7fd50805af10 con 0x7fd51c102710 2024-03-30T04:51:55.294 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:51:55.294 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:51:55.294 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:51:55.294 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:51:55.294 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:51:55.294 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:51:55.294 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:51:55.295 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:51:55.295 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T04:51:55.295 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:51:55.295 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:51:55.295 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:51:55.295 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 3 2024-03-30T04:51:55.295 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:51:55.295 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:51:55.295 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:51:55.295 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 13 2024-03-30T04:51:55.295 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:51:55.295 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:51:55.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.298+0000 7fd522c00700 1 -- 172.21.15.123:0/3521432180 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd50006cc30 msgr2=0x7fd50006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:55.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.298+0000 7fd522c00700 1 --2- 172.21.15.123:0/3521432180 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd50006cc30 0x7fd50006f0e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd50c009fd0 tx=0x7fd50c00b040 comp rx=0 tx=0).stop 2024-03-30T04:51:55.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.298+0000 7fd522c00700 1 -- 172.21.15.123:0/3521432180 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd51c102710 msgr2=0x7fd51c116140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:55.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.298+0000 7fd522c00700 1 --2- 172.21.15.123:0/3521432180 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd51c102710 0x7fd51c116140 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fd508009fd0 tx=0x7fd508009e30 comp rx=0 tx=0).stop 2024-03-30T04:51:55.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.298+0000 7fd522c00700 1 -- 172.21.15.123:0/3521432180 shutdown_connections 2024-03-30T04:51:55.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.298+0000 7fd522c00700 1 --2- 172.21.15.123:0/3521432180 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd50006cc30 0x7fd50006f0e0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:55.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.298+0000 7fd522c00700 1 --2- 172.21.15.123:0/3521432180 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd51c101d70 0x7fd51c115c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:55.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.298+0000 7fd522c00700 1 --2- 172.21.15.123:0/3521432180 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd51c102710 0x7fd51c116140 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:55.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.298+0000 7fd522c00700 1 -- 172.21.15.123:0/3521432180 >> 172.21.15.123:0/3521432180 conn(0x7fd51c072bc0 msgr2=0x7fd51c0658e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:51:55.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.298+0000 7fd522c00700 1 -- 172.21.15.123:0/3521432180 shutdown_connections 2024-03-30T04:51:55.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.298+0000 7fd522c00700 1 -- 172.21.15.123:0/3521432180 wait complete. 2024-03-30T04:51:55.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.431+0000 7f5928800700 1 -- 172.21.15.123:0/3675022817 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f59201037a0 msgr2=0x7f59200fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:55.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.431+0000 7f5928800700 1 --2- 172.21.15.123:0/3675022817 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f59201037a0 0x7f59200fce10 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f5910004940 tx=0x7f5910009e30 comp rx=0 tx=0).stop 2024-03-30T04:51:55.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.432+0000 7f5928800700 1 -- 172.21.15.123:0/3675022817 shutdown_connections 2024-03-30T04:51:55.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.432+0000 7f5928800700 1 --2- 172.21.15.123:0/3675022817 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f59201037a0 0x7f59200fce10 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:55.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.432+0000 7f5928800700 1 --2- 172.21.15.123:0/3675022817 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5920102e00 0x7f59201031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:55.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.432+0000 7f5928800700 1 -- 172.21.15.123:0/3675022817 >> 172.21.15.123:0/3675022817 conn(0x7f59200f89b0 msgr2=0x7f59200fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:51:55.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.432+0000 7f5928800700 1 -- 172.21.15.123:0/3675022817 shutdown_connections 2024-03-30T04:51:55.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.432+0000 7f5928800700 1 -- 172.21.15.123:0/3675022817 wait complete. 2024-03-30T04:51:55.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.432+0000 7f5928800700 1 Processor -- start 2024-03-30T04:51:55.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.432+0000 7f5928800700 1 -- start start 2024-03-30T04:51:55.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.433+0000 7f5928800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5920102e00 0x7f59200702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:51:55.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.433+0000 7f5928800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f59201037a0 0x7f5920070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:51:55.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.433+0000 7f5928800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5920070d90 con 0x7f5920102e00 2024-03-30T04:51:55.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.433+0000 7f5928800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5920070ed0 con 0x7f59201037a0 2024-03-30T04:51:55.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.433+0000 7f5925e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5920102e00 0x7f59200702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:51:55.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.433+0000 7f5925e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5920102e00 0x7f59200702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52264/0 (socket says 172.21.15.123:52264) 2024-03-30T04:51:55.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.433+0000 7f5925e00700 1 -- 172.21.15.123:0/958996768 learned_addr learned my addr 172.21.15.123:0/958996768 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:51:55.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.433+0000 7f5925400700 1 --2- 172.21.15.123:0/958996768 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f59201037a0 0x7f5920070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:51:55.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.434+0000 7f5925e00700 1 -- 172.21.15.123:0/958996768 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f59201037a0 msgr2=0x7f5920070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:55.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.434+0000 7f5925e00700 1 --2- 172.21.15.123:0/958996768 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f59201037a0 0x7f5920070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:55.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.434+0000 7f5925e00700 1 -- 172.21.15.123:0/958996768 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5910005040 con 0x7f5920102e00 2024-03-30T04:51:55.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.434+0000 7f5925e00700 1 --2- 172.21.15.123:0/958996768 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5920102e00 0x7f59200702c0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f5914002a50 tx=0x7f59140046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:51:55.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.434+0000 7f591a000700 1 -- 172.21.15.123:0/958996768 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5914007cc0 con 0x7f5920102e00 2024-03-30T04:51:55.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.434+0000 7f591a000700 1 -- 172.21.15.123:0/958996768 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f591400f040 con 0x7f5920102e00 2024-03-30T04:51:55.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.434+0000 7f5928800700 1 -- 172.21.15.123:0/958996768 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5920119e00 con 0x7f5920102e00 2024-03-30T04:51:55.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.434+0000 7f591a000700 1 -- 172.21.15.123:0/958996768 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f591400d9c0 con 0x7f5920102e00 2024-03-30T04:51:55.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.434+0000 7f5928800700 1 -- 172.21.15.123:0/958996768 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f592011a280 con 0x7f5920102e00 2024-03-30T04:51:55.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.435+0000 7f5928800700 1 -- 172.21.15.123:0/958996768 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59200fe550 con 0x7f5920102e00 2024-03-30T04:51:55.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.436+0000 7f591a000700 1 -- 172.21.15.123:0/958996768 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f591400db20 con 0x7f5920102e00 2024-03-30T04:51:55.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.437+0000 7f591a000700 1 --2- 172.21.15.123:0/958996768 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f590806cc30 0x7f590806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:51:55.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.437+0000 7f591a000700 1 -- 172.21.15.123:0/958996768 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f591408bd00 con 0x7f5920102e00 2024-03-30T04:51:55.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.438+0000 7f5925400700 1 --2- 172.21.15.123:0/958996768 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f590806cc30 0x7f590806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:51:55.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.438+0000 7f5925400700 1 --2- 172.21.15.123:0/958996768 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f590806cc30 0x7f590806f0e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f5920071c00 tx=0x7f5910009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:51:55.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.442+0000 7f591a000700 1 -- 172.21.15.123:0/958996768 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5914055cc0 con 0x7f5920102e00 2024-03-30T04:51:55.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.688+0000 7f5928800700 1 -- 172.21.15.123:0/958996768 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f59200719b0 con 0x7f5920102e00 2024-03-30T04:51:55.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.688+0000 7f591a000700 1 -- 172.21.15.123:0/958996768 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f59140592e0 con 0x7f5920102e00 2024-03-30T04:51:55.687 INFO:teuthology.orchestra.run.smithi123.stdout:e32 2024-03-30T04:51:55.687 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:51:55.687 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:51:55.687 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:51:55.687 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:51:55.687 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:51:55.687 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:51:55.687 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T04:51:55.687 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T04:51:55.687 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:51:55.687 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T04:51:55.687 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:51:55.687 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:51:55.687 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:51:55.688 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:51:55.688 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:51:55.688 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T04:51:55.688 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:51:55.688 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:51:55.688 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T04:51:55.688 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:51:55.688 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T04:51:55.688 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T04:51:55.688 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T04:51:55.688 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T04:51:55.688 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:51:55.688 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:51:55.688 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:51:55.689 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:51:55.689 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:51:55.689 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:51:55.689 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T04:51:55.689 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T04:51:55.689 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:51:55.689 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:51:55.689 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T04:51:55.689 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:51:55.689 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:51:55.689 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:51:55.689 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44315} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/1741352336,v1:172.21.15.135:6827/1741352336] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:51:55.689 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:51:55.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.690+0000 7f5928800700 1 -- 172.21.15.123:0/958996768 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f590806cc30 msgr2=0x7f590806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:55.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.690+0000 7f5928800700 1 --2- 172.21.15.123:0/958996768 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f590806cc30 0x7f590806f0e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f5920071c00 tx=0x7f5910009e30 comp rx=0 tx=0).stop 2024-03-30T04:51:55.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.690+0000 7f5928800700 1 -- 172.21.15.123:0/958996768 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5920102e00 msgr2=0x7f59200702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:55.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.690+0000 7f5928800700 1 --2- 172.21.15.123:0/958996768 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5920102e00 0x7f59200702c0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f5914002a50 tx=0x7f59140046b0 comp rx=0 tx=0).stop 2024-03-30T04:51:55.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.690+0000 7f5928800700 1 -- 172.21.15.123:0/958996768 shutdown_connections 2024-03-30T04:51:55.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.690+0000 7f5928800700 1 --2- 172.21.15.123:0/958996768 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f590806cc30 0x7f590806f0e0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:55.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.690+0000 7f5928800700 1 --2- 172.21.15.123:0/958996768 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5920102e00 0x7f59200702c0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:55.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.690+0000 7f5928800700 1 --2- 172.21.15.123:0/958996768 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f59201037a0 0x7f5920070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:55.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.690+0000 7f5928800700 1 -- 172.21.15.123:0/958996768 >> 172.21.15.123:0/958996768 conn(0x7f59200f89b0 msgr2=0x7f59200f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:51:55.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.691+0000 7f5928800700 1 -- 172.21.15.123:0/958996768 shutdown_connections 2024-03-30T04:51:55.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.691+0000 7f5928800700 1 -- 172.21.15.123:0/958996768 wait complete. 2024-03-30T04:51:55.691 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 32 2024-03-30T04:51:55.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.827+0000 7fbac9000700 1 -- 172.21.15.123:0/851222605 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbac4102e00 msgr2=0x7fbac41031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:55.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.827+0000 7fbac9000700 1 --2- 172.21.15.123:0/851222605 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbac4102e00 0x7fbac41031d0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fbab4004940 tx=0x7fbab4009e30 comp rx=0 tx=0).stop 2024-03-30T04:51:55.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.827+0000 7fbac9000700 1 -- 172.21.15.123:0/851222605 shutdown_connections 2024-03-30T04:51:55.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.827+0000 7fbac9000700 1 --2- 172.21.15.123:0/851222605 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbac41037a0 0x7fbac40fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:55.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.827+0000 7fbac9000700 1 --2- 172.21.15.123:0/851222605 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbac4102e00 0x7fbac41031d0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:55.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.827+0000 7fbac9000700 1 -- 172.21.15.123:0/851222605 >> 172.21.15.123:0/851222605 conn(0x7fbac40f89b0 msgr2=0x7fbac40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:51:55.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.827+0000 7fbac9000700 1 -- 172.21.15.123:0/851222605 shutdown_connections 2024-03-30T04:51:55.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.827+0000 7fbac9000700 1 -- 172.21.15.123:0/851222605 wait complete. 2024-03-30T04:51:55.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.828+0000 7fbac9000700 1 Processor -- start 2024-03-30T04:51:55.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.828+0000 7fbac9000700 1 -- start start 2024-03-30T04:51:55.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.829+0000 7fbac9000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbac4102e00 0x7fbac4115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:51:55.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.829+0000 7fbac9000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbac41037a0 0x7fbac4116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:51:55.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.829+0000 7fbac9000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac4116860 con 0x7fbac4102e00 2024-03-30T04:51:55.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.829+0000 7fbac9000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac410fc30 con 0x7fbac41037a0 2024-03-30T04:51:55.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.829+0000 7fbac1e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbac4102e00 0x7fbac4115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:51:55.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.829+0000 7fbac1e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbac4102e00 0x7fbac4115cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52282/0 (socket says 172.21.15.123:52282) 2024-03-30T04:51:55.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.829+0000 7fbac1e00700 1 -- 172.21.15.123:0/4054129826 learned_addr learned my addr 172.21.15.123:0/4054129826 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:51:55.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.829+0000 7fbac1400700 1 --2- 172.21.15.123:0/4054129826 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbac41037a0 0x7fbac4116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:51:55.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.829+0000 7fbac1e00700 1 -- 172.21.15.123:0/4054129826 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbac41037a0 msgr2=0x7fbac4116210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:55.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.829+0000 7fbac1e00700 1 --2- 172.21.15.123:0/4054129826 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbac41037a0 0x7fbac4116210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:55.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.829+0000 7fbac1e00700 1 -- 172.21.15.123:0/4054129826 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbab4005040 con 0x7fbac4102e00 2024-03-30T04:51:55.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.829+0000 7fbac1400700 1 --2- 172.21.15.123:0/4054129826 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbac41037a0 0x7fbac4116210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:51:55.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.829+0000 7fbac1e00700 1 --2- 172.21.15.123:0/4054129826 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbac4102e00 0x7fbac4115cd0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fbab4009390 tx=0x7fbab4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:51:55.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.830+0000 7fbaba000700 1 -- 172.21.15.123:0/4054129826 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbab400e040 con 0x7fbac4102e00 2024-03-30T04:51:55.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.830+0000 7fbaba000700 1 -- 172.21.15.123:0/4054129826 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbab4013420 con 0x7fbac4102e00 2024-03-30T04:51:55.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.830+0000 7fbac9000700 1 -- 172.21.15.123:0/4054129826 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbac410feb0 con 0x7fbac4102e00 2024-03-30T04:51:55.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.830+0000 7fbaba000700 1 -- 172.21.15.123:0/4054129826 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbab401c630 con 0x7fbac4102e00 2024-03-30T04:51:55.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.830+0000 7fbac9000700 1 -- 172.21.15.123:0/4054129826 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbac4110470 con 0x7fbac4102e00 2024-03-30T04:51:55.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.831+0000 7fbac9000700 1 -- 172.21.15.123:0/4054129826 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbac40fe550 con 0x7fbac4102e00 2024-03-30T04:51:55.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.832+0000 7fbaba000700 1 -- 172.21.15.123:0/4054129826 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fbab40139e0 con 0x7fbac4102e00 2024-03-30T04:51:55.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.833+0000 7fbaba000700 1 --2- 172.21.15.123:0/4054129826 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbaa806cc30 0x7fbaa806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:51:55.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.833+0000 7fbaba000700 1 -- 172.21.15.123:0/4054129826 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fbab408d250 con 0x7fbac4102e00 2024-03-30T04:51:55.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.833+0000 7fbac1400700 1 --2- 172.21.15.123:0/4054129826 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbaa806cc30 0x7fbaa806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:51:55.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.833+0000 7fbac1400700 1 --2- 172.21.15.123:0/4054129826 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbaa806cc30 0x7fbaa806f0e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fbab0009fd0 tx=0x7fbab000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:51:55.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:55.837+0000 7fbaba000700 1 -- 172.21.15.123:0/4054129826 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fbab405add0 con 0x7fbac4102e00 2024-03-30T04:51:56.055 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.057+0000 7fbac9000700 1 -- 172.21.15.123:0/4054129826 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbac404bf40 con 0x7fbaa806cc30 2024-03-30T04:51:56.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.058+0000 7fbaba000700 1 -- 172.21.15.123:0/4054129826 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fbac404bf40 con 0x7fbaa806cc30 2024-03-30T04:51:56.056 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:51:56.056 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:51:56.056 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:51:56.056 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T04:51:56.057 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:51:56.057 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T04:51:56.057 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T04:51:56.057 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T04:51:56.057 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T04:51:56.058 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:51:56.058 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "15/21 daemons upgraded", 2024-03-30T04:51:56.058 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T04:51:56.058 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:51:56.058 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:51:56.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.060+0000 7fbac9000700 1 -- 172.21.15.123:0/4054129826 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbaa806cc30 msgr2=0x7fbaa806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:56.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.060+0000 7fbac9000700 1 --2- 172.21.15.123:0/4054129826 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbaa806cc30 0x7fbaa806f0e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fbab0009fd0 tx=0x7fbab000b040 comp rx=0 tx=0).stop 2024-03-30T04:51:56.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.060+0000 7fbac9000700 1 -- 172.21.15.123:0/4054129826 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbac4102e00 msgr2=0x7fbac4115cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:56.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.060+0000 7fbac9000700 1 --2- 172.21.15.123:0/4054129826 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbac4102e00 0x7fbac4115cd0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fbab4009390 tx=0x7fbab4009e30 comp rx=0 tx=0).stop 2024-03-30T04:51:56.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.061+0000 7fbac9000700 1 -- 172.21.15.123:0/4054129826 shutdown_connections 2024-03-30T04:51:56.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.061+0000 7fbac9000700 1 --2- 172.21.15.123:0/4054129826 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbaa806cc30 0x7fbaa806f0e0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:56.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.061+0000 7fbac9000700 1 --2- 172.21.15.123:0/4054129826 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbac4102e00 0x7fbac4115cd0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:56.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.061+0000 7fbac9000700 1 --2- 172.21.15.123:0/4054129826 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbac41037a0 0x7fbac4116210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:56.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.061+0000 7fbac9000700 1 -- 172.21.15.123:0/4054129826 >> 172.21.15.123:0/4054129826 conn(0x7fbac40f89b0 msgr2=0x7fbac40f9980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:51:56.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.061+0000 7fbac9000700 1 -- 172.21.15.123:0/4054129826 shutdown_connections 2024-03-30T04:51:56.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.061+0000 7fbac9000700 1 -- 172.21.15.123:0/4054129826 wait complete. 2024-03-30T04:51:56.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:51:55 smithi135 ceph-mon[106011]: from='client.35158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:51:56.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:51:55 smithi135 ceph-mon[106011]: from='client.35162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:51:56.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:51:55 smithi135 ceph-mon[106011]: from='client.35166 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:51:56.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:51:55 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3521432180' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:51:56.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:51:55 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/958996768' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:51:56.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.195+0000 7ffb1b200700 1 -- 172.21.15.123:0/1520981759 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb1c102c40 msgr2=0x7ffb1c103010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:56.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.195+0000 7ffb1b200700 1 --2- 172.21.15.123:0/1520981759 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb1c102c40 0x7ffb1c103010 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c004940 tx=0x7ffb0c009e30 comp rx=0 tx=0).stop 2024-03-30T04:51:56.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.195+0000 7ffb1b200700 1 -- 172.21.15.123:0/1520981759 shutdown_connections 2024-03-30T04:51:56.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.195+0000 7ffb1b200700 1 --2- 172.21.15.123:0/1520981759 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffb1c1035e0 0x7ffb1c0fcc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:56.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.195+0000 7ffb1b200700 1 --2- 172.21.15.123:0/1520981759 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb1c102c40 0x7ffb1c103010 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:56.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.195+0000 7ffb1b200700 1 -- 172.21.15.123:0/1520981759 >> 172.21.15.123:0/1520981759 conn(0x7ffb1c0f8830 msgr2=0x7ffb1c0fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:51:56.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.195+0000 7ffb1b200700 1 -- 172.21.15.123:0/1520981759 shutdown_connections 2024-03-30T04:51:56.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.195+0000 7ffb1b200700 1 -- 172.21.15.123:0/1520981759 wait complete. 2024-03-30T04:51:56.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.196+0000 7ffb1b200700 1 Processor -- start 2024-03-30T04:51:56.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.196+0000 7ffb1b200700 1 -- start start 2024-03-30T04:51:56.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.196+0000 7ffb1b200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb1c102c40 0x7ffb1c115b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:51:56.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.197+0000 7ffb1b200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffb1c1035e0 0x7ffb1c1160c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:51:56.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.197+0000 7ffb1b200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1c116710 con 0x7ffb1c102c40 2024-03-30T04:51:56.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.197+0000 7ffb1b200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1c10fb20 con 0x7ffb1c1035e0 2024-03-30T04:51:56.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.197+0000 7ffb19e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb1c102c40 0x7ffb1c115b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:51:56.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.197+0000 7ffb19e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb1c102c40 0x7ffb1c115b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52292/0 (socket says 172.21.15.123:52292) 2024-03-30T04:51:56.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.197+0000 7ffb19e00700 1 -- 172.21.15.123:0/3330216145 learned_addr learned my addr 172.21.15.123:0/3330216145 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:51:56.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.197+0000 7ffb19400700 1 --2- 172.21.15.123:0/3330216145 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffb1c1035e0 0x7ffb1c1160c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:51:56.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.197+0000 7ffb19e00700 1 -- 172.21.15.123:0/3330216145 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffb1c1035e0 msgr2=0x7ffb1c1160c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:56.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.197+0000 7ffb19e00700 1 --2- 172.21.15.123:0/3330216145 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffb1c1035e0 0x7ffb1c1160c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:56.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.197+0000 7ffb19e00700 1 -- 172.21.15.123:0/3330216145 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb0c005040 con 0x7ffb1c102c40 2024-03-30T04:51:56.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.197+0000 7ffb19400700 1 --2- 172.21.15.123:0/3330216145 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffb1c1035e0 0x7ffb1c1160c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:51:56.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.197+0000 7ffb19e00700 1 --2- 172.21.15.123:0/3330216145 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb1c102c40 0x7ffb1c115b80 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c009390 tx=0x7ffb0c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:51:56.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.198+0000 7ffb12000700 1 -- 172.21.15.123:0/3330216145 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb0c00e040 con 0x7ffb1c102c40 2024-03-30T04:51:56.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.198+0000 7ffb1b200700 1 -- 172.21.15.123:0/3330216145 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb1c10fda0 con 0x7ffb1c102c40 2024-03-30T04:51:56.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.198+0000 7ffb12000700 1 -- 172.21.15.123:0/3330216145 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffb0c013420 con 0x7ffb1c102c40 2024-03-30T04:51:56.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.198+0000 7ffb12000700 1 -- 172.21.15.123:0/3330216145 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb0c01c630 con 0x7ffb1c102c40 2024-03-30T04:51:56.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.198+0000 7ffb1b200700 1 -- 172.21.15.123:0/3330216145 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb1c1102e0 con 0x7ffb1c102c40 2024-03-30T04:51:56.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.199+0000 7ffb1b200700 1 -- 172.21.15.123:0/3330216145 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb1c0fe390 con 0x7ffb1c102c40 2024-03-30T04:51:56.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.200+0000 7ffb12000700 1 -- 172.21.15.123:0/3330216145 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7ffb0c0139e0 con 0x7ffb1c102c40 2024-03-30T04:51:56.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.201+0000 7ffb12000700 1 --2- 172.21.15.123:0/3330216145 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ffb0006cc30 0x7ffb0006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:51:56.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.201+0000 7ffb12000700 1 -- 172.21.15.123:0/3330216145 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ffb0c08d320 con 0x7ffb1c102c40 2024-03-30T04:51:56.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.201+0000 7ffb19400700 1 --2- 172.21.15.123:0/3330216145 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ffb0006cc30 0x7ffb0006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:51:56.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.202+0000 7ffb19400700 1 --2- 172.21.15.123:0/3330216145 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ffb0006cc30 0x7ffb0006f0e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ffb08009fd0 tx=0x7ffb08009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:51:56.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.206+0000 7ffb12000700 1 -- 172.21.15.123:0/3330216145 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ffb0c057390 con 0x7ffb1c102c40 2024-03-30T04:51:56.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:51:55 smithi123 ceph-mon[139369]: from='client.35158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:51:56.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:51:55 smithi123 ceph-mon[139369]: from='client.35162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:51:56.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:51:55 smithi123 ceph-mon[139369]: from='client.35166 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:51:56.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:51:55 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3521432180' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:51:56.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:51:55 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/958996768' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:51:56.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.487+0000 7ffb1b200700 1 -- 172.21.15.123:0/3330216145 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7ffb1c04bed0 con 0x7ffb1c102c40 2024-03-30T04:51:56.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.487+0000 7ffb12000700 1 -- 172.21.15.123:0/3330216145 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7ffb0c05a9b0 con 0x7ffb1c102c40 2024-03-30T04:51:56.486 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T04:51:56.486 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T04:51:56.486 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T04:51:56.486 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T04:51:56.486 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T04:51:56.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.490+0000 7ffb1b200700 1 -- 172.21.15.123:0/3330216145 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ffb0006cc30 msgr2=0x7ffb0006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:56.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.490+0000 7ffb1b200700 1 --2- 172.21.15.123:0/3330216145 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ffb0006cc30 0x7ffb0006f0e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ffb08009fd0 tx=0x7ffb08009ee0 comp rx=0 tx=0).stop 2024-03-30T04:51:56.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.490+0000 7ffb1b200700 1 -- 172.21.15.123:0/3330216145 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb1c102c40 msgr2=0x7ffb1c115b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:51:56.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.490+0000 7ffb1b200700 1 --2- 172.21.15.123:0/3330216145 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb1c102c40 0x7ffb1c115b80 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c009390 tx=0x7ffb0c009e30 comp rx=0 tx=0).stop 2024-03-30T04:51:56.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.490+0000 7ffb1b200700 1 -- 172.21.15.123:0/3330216145 shutdown_connections 2024-03-30T04:51:56.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.490+0000 7ffb1b200700 1 --2- 172.21.15.123:0/3330216145 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ffb0006cc30 0x7ffb0006f0e0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:56.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.490+0000 7ffb1b200700 1 --2- 172.21.15.123:0/3330216145 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb1c102c40 0x7ffb1c115b80 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:56.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.490+0000 7ffb1b200700 1 --2- 172.21.15.123:0/3330216145 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffb1c1035e0 0x7ffb1c1160c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:51:56.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.490+0000 7ffb1b200700 1 -- 172.21.15.123:0/3330216145 >> 172.21.15.123:0/3330216145 conn(0x7ffb1c0f8830 msgr2=0x7ffb1c0f9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:51:56.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.490+0000 7ffb1b200700 1 -- 172.21.15.123:0/3330216145 shutdown_connections 2024-03-30T04:51:56.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:51:56.490+0000 7ffb1b200700 1 -- 172.21.15.123:0/3330216145 wait complete. 2024-03-30T04:51:57.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:51:56 smithi135 ceph-mon[106011]: pgmap v622: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:57.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:51:56 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3330216145' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:51:57.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:51:56 smithi123 ceph-mon[139369]: pgmap v622: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:57.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:51:56 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3330216145' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:51:58.057 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:51:57 smithi123 ceph-mon[139369]: from='client.35178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:51:58.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:51:57 smithi135 ceph-mon[106011]: from='client.35178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:51:59.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:51:58 smithi135 ceph-mon[106011]: pgmap v623: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:51:59.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:51:58 smithi123 ceph-mon[139369]: pgmap v623: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:00.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:51:59 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:52:00.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:51:59 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:52:01.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:52:00 smithi135 ceph-mon[106011]: pgmap v624: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:01.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:52:00 smithi123 ceph-mon[139369]: pgmap v624: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:03.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:52:02 smithi135 ceph-mon[106011]: pgmap v625: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:03.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:52:02 smithi123 ceph-mon[139369]: pgmap v625: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:05.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:52:04 smithi135 ceph-mon[106011]: pgmap v626: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:05.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:52:04 smithi123 ceph-mon[139369]: pgmap v626: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:07.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:52:06 smithi135 ceph-mon[106011]: pgmap v627: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:07.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:52:06 smithi123 ceph-mon[139369]: pgmap v627: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:09.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:52:08 smithi135 ceph-mon[106011]: pgmap v628: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:09.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:52:08 smithi123 ceph-mon[139369]: pgmap v628: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:10.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:52:10 smithi123 ceph-mon[139369]: pgmap v629: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:11.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:52:10 smithi135 ceph-mon[106011]: pgmap v629: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:13.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:52:12 smithi135 ceph-mon[106011]: pgmap v630: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:13.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:52:12 smithi123 ceph-mon[139369]: pgmap v630: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:15.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:52:14 smithi135 ceph-mon[106011]: pgmap v631: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:15.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:52:14 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:52:15.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:52:14 smithi123 ceph-mon[139369]: pgmap v631: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:15.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:52:14 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:52:17.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:52:16 smithi135 ceph-mon[106011]: pgmap v632: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:17.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:52:16 smithi123 ceph-mon[139369]: pgmap v632: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:19.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:52:18 smithi135 ceph-mon[106011]: pgmap v633: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:19.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:52:18 smithi123 ceph-mon[139369]: pgmap v633: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:20.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:52:20 smithi123 ceph-mon[139369]: pgmap v634: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:21.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:52:20 smithi135 ceph-mon[106011]: pgmap v634: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:23.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:52:22 smithi135 ceph-mon[106011]: pgmap v635: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:23.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:52:22 smithi123 ceph-mon[139369]: pgmap v635: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:25.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:52:24 smithi135 ceph-mon[106011]: pgmap v636: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:25.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:52:24 smithi123 ceph-mon[139369]: pgmap v636: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:26.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:26.639+0000 7f1046c00700 1 -- 172.21.15.123:0/3731714587 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10401037a0 msgr2=0x7f10400fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:52:26.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:26.639+0000 7f1046c00700 1 --2- 172.21.15.123:0/3731714587 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10401037a0 0x7f10400fce10 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f102c004940 tx=0x7f102c009e30 comp rx=0 tx=0).stop 2024-03-30T04:52:26.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:26.639+0000 7f1046c00700 1 -- 172.21.15.123:0/3731714587 shutdown_connections 2024-03-30T04:52:26.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:26.639+0000 7f1046c00700 1 --2- 172.21.15.123:0/3731714587 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10401037a0 0x7f10400fce10 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:52:26.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:26.639+0000 7f1046c00700 1 --2- 172.21.15.123:0/3731714587 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1040102e00 0x7f10401031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:52:26.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:26.639+0000 7f1046c00700 1 -- 172.21.15.123:0/3731714587 >> 172.21.15.123:0/3731714587 conn(0x7f10400f89b0 msgr2=0x7f10400fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:52:26.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:26.640+0000 7f1046c00700 1 -- 172.21.15.123:0/3731714587 shutdown_connections 2024-03-30T04:52:26.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:26.640+0000 7f1046c00700 1 -- 172.21.15.123:0/3731714587 wait complete. 2024-03-30T04:52:26.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:26.641+0000 7f1046c00700 1 Processor -- start 2024-03-30T04:52:26.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:26.641+0000 7f1046c00700 1 -- start start 2024-03-30T04:52:26.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:26.641+0000 7f1046c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1040102e00 0x7f10400702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:52:26.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:26.642+0000 7f1046c00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10401037a0 0x7f1040070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:52:26.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:26.642+0000 7f1046c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1040070d90 con 0x7f1040102e00 2024-03-30T04:52:26.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:26.642+0000 7f1046c00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1040070ed0 con 0x7f10401037a0 2024-03-30T04:52:26.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:26.642+0000 7f103fe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1040102e00 0x7f10400702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:52:26.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:26.642+0000 7f103fe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1040102e00 0x7f10400702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42946/0 (socket says 172.21.15.123:42946) 2024-03-30T04:52:26.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:26.642+0000 7f103fe00700 1 -- 172.21.15.123:0/3351799130 learned_addr learned my addr 172.21.15.123:0/3351799130 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:52:26.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:26.642+0000 7f103f400700 1 --2- 172.21.15.123:0/3351799130 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10401037a0 0x7f1040070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:52:26.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:26.642+0000 7f103fe00700 1 -- 172.21.15.123:0/3351799130 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10401037a0 msgr2=0x7f1040070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:52:26.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:26.642+0000 7f103fe00700 1 --2- 172.21.15.123:0/3351799130 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10401037a0 0x7f1040070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:52:26.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:26.642+0000 7f103fe00700 1 -- 172.21.15.123:0/3351799130 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f102c005040 con 0x7f1040102e00 2024-03-30T04:52:26.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:26.642+0000 7f103fe00700 1 --2- 172.21.15.123:0/3351799130 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1040102e00 0x7f10400702c0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f1030002a50 tx=0x7f10300046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:52:26.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:26.643+0000 7f1037e00700 1 -- 172.21.15.123:0/3351799130 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1030007cc0 con 0x7f1040102e00 2024-03-30T04:52:26.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:26.643+0000 7f1046c00700 1 -- 172.21.15.123:0/3351799130 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1040119e00 con 0x7f1040102e00 2024-03-30T04:52:26.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:26.643+0000 7f1037e00700 1 -- 172.21.15.123:0/3351799130 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f103000f040 con 0x7f1040102e00 2024-03-30T04:52:26.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:26.643+0000 7f1037e00700 1 -- 172.21.15.123:0/3351799130 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f103000d9c0 con 0x7f1040102e00 2024-03-30T04:52:26.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:26.643+0000 7f1046c00700 1 -- 172.21.15.123:0/3351799130 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f104011a280 con 0x7f1040102e00 2024-03-30T04:52:26.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:26.644+0000 7f1046c00700 1 -- 172.21.15.123:0/3351799130 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10400fe550 con 0x7f1040102e00 2024-03-30T04:52:26.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:26.645+0000 7f1037e00700 1 -- 172.21.15.123:0/3351799130 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f103000db20 con 0x7f1040102e00 2024-03-30T04:52:26.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:26.646+0000 7f1037e00700 1 --2- 172.21.15.123:0/3351799130 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f102406cc20 0x7f102406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:52:26.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:26.646+0000 7f1037e00700 1 -- 172.21.15.123:0/3351799130 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f103008be20 con 0x7f1040102e00 2024-03-30T04:52:26.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:26.647+0000 7f103f400700 1 --2- 172.21.15.123:0/3351799130 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f102406cc20 0x7f102406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:52:26.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:26.647+0000 7f103f400700 1 --2- 172.21.15.123:0/3351799130 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f102406cc20 0x7f102406f0d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1040071c00 tx=0x7f102c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:52:26.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:26.651+0000 7f1037e00700 1 -- 172.21.15.123:0/3351799130 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f1030055d60 con 0x7f1040102e00 2024-03-30T04:52:26.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:26.872+0000 7f1046c00700 1 -- 172.21.15.123:0/3351799130 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f10400719b0 con 0x7f102406cc20 2024-03-30T04:52:26.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:26.873+0000 7f1037e00700 1 -- 172.21.15.123:0/3351799130 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f10400719b0 con 0x7f102406cc20 2024-03-30T04:52:26.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:26.876+0000 7f1046c00700 1 -- 172.21.15.123:0/3351799130 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f102406cc20 msgr2=0x7f102406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:52:26.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:26.876+0000 7f1046c00700 1 --2- 172.21.15.123:0/3351799130 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f102406cc20 0x7f102406f0d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1040071c00 tx=0x7f102c009e30 comp rx=0 tx=0).stop 2024-03-30T04:52:26.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:26.876+0000 7f1046c00700 1 -- 172.21.15.123:0/3351799130 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1040102e00 msgr2=0x7f10400702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:52:26.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:26.876+0000 7f1046c00700 1 --2- 172.21.15.123:0/3351799130 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1040102e00 0x7f10400702c0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f1030002a50 tx=0x7f10300046b0 comp rx=0 tx=0).stop 2024-03-30T04:52:26.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:26.876+0000 7f1046c00700 1 -- 172.21.15.123:0/3351799130 shutdown_connections 2024-03-30T04:52:26.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:26.876+0000 7f1046c00700 1 --2- 172.21.15.123:0/3351799130 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f102406cc20 0x7f102406f0d0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:52:26.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:26.876+0000 7f1046c00700 1 --2- 172.21.15.123:0/3351799130 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1040102e00 0x7f10400702c0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:52:26.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:26.876+0000 7f1046c00700 1 --2- 172.21.15.123:0/3351799130 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10401037a0 0x7f1040070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:52:26.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:26.876+0000 7f1046c00700 1 -- 172.21.15.123:0/3351799130 >> 172.21.15.123:0/3351799130 conn(0x7f10400f89b0 msgr2=0x7f10400f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:52:26.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:26.876+0000 7f1046c00700 1 -- 172.21.15.123:0/3351799130 shutdown_connections 2024-03-30T04:52:26.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:26.876+0000 7f1046c00700 1 -- 172.21.15.123:0/3351799130 wait complete. 2024-03-30T04:52:26.888 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:52:27.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.011+0000 7f2800e00700 1 -- 172.21.15.123:0/1046457886 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27fc102e20 msgr2=0x7f27fc1031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:52:27.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.011+0000 7f2800e00700 1 --2- 172.21.15.123:0/1046457886 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27fc102e20 0x7f27fc1031f0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f27ec004940 tx=0x7f27ec009e30 comp rx=0 tx=0).stop 2024-03-30T04:52:27.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.011+0000 7f2800e00700 1 -- 172.21.15.123:0/1046457886 shutdown_connections 2024-03-30T04:52:27.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.011+0000 7f2800e00700 1 --2- 172.21.15.123:0/1046457886 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f27fc1037c0 0x7f27fc0fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:52:27.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.011+0000 7f2800e00700 1 --2- 172.21.15.123:0/1046457886 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27fc102e20 0x7f27fc1031f0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:52:27.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.011+0000 7f2800e00700 1 -- 172.21.15.123:0/1046457886 >> 172.21.15.123:0/1046457886 conn(0x7f27fc0f89d0 msgr2=0x7f27fc0fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:52:27.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.011+0000 7f2800e00700 1 -- 172.21.15.123:0/1046457886 shutdown_connections 2024-03-30T04:52:27.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.011+0000 7f2800e00700 1 -- 172.21.15.123:0/1046457886 wait complete. 2024-03-30T04:52:27.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.012+0000 7f2800e00700 1 Processor -- start 2024-03-30T04:52:27.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.012+0000 7f2800e00700 1 -- start start 2024-03-30T04:52:27.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.012+0000 7f2800e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f27fc102e20 0x7f27fc115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:52:27.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.012+0000 7f2800e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27fc1037c0 0x7f27fc116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:52:27.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.013+0000 7f2800e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27fc116880 con 0x7f27fc1037c0 2024-03-30T04:52:27.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.013+0000 7f2800e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27fc10fc50 con 0x7f27fc102e20 2024-03-30T04:52:27.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.013+0000 7f27f9400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27fc1037c0 0x7f27fc116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:52:27.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.013+0000 7f27f9400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27fc1037c0 0x7f27fc116230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42962/0 (socket says 172.21.15.123:42962) 2024-03-30T04:52:27.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.013+0000 7f27f9400700 1 -- 172.21.15.123:0/1676075602 learned_addr learned my addr 172.21.15.123:0/1676075602 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:52:27.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.013+0000 7f27f9e00700 1 --2- 172.21.15.123:0/1676075602 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f27fc102e20 0x7f27fc115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:52:27.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.013+0000 7f27f9400700 1 -- 172.21.15.123:0/1676075602 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f27fc102e20 msgr2=0x7f27fc115cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:52:27.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.013+0000 7f27f9400700 1 --2- 172.21.15.123:0/1676075602 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f27fc102e20 0x7f27fc115cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:52:27.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.013+0000 7f27f9400700 1 -- 172.21.15.123:0/1676075602 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27ec005040 con 0x7f27fc1037c0 2024-03-30T04:52:27.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.013+0000 7f27f9400700 1 --2- 172.21.15.123:0/1676075602 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27fc1037c0 0x7f27fc116230 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f27e800c890 tx=0x7f27e80044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:52:27.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.014+0000 7f27f2000700 1 -- 172.21.15.123:0/1676075602 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f27e800a700 con 0x7f27fc1037c0 2024-03-30T04:52:27.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.014+0000 7f27f2000700 1 -- 172.21.15.123:0/1676075602 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f27e800ad10 con 0x7f27fc1037c0 2024-03-30T04:52:27.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.014+0000 7f2800e00700 1 -- 172.21.15.123:0/1676075602 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27fc10ff30 con 0x7f27fc1037c0 2024-03-30T04:52:27.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.014+0000 7f27f2000700 1 -- 172.21.15.123:0/1676075602 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f27e8012e70 con 0x7f27fc1037c0 2024-03-30T04:52:27.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.014+0000 7f2800e00700 1 -- 172.21.15.123:0/1676075602 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27fc1104f0 con 0x7f27fc1037c0 2024-03-30T04:52:27.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.015+0000 7f2800e00700 1 -- 172.21.15.123:0/1676075602 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27fc0fe570 con 0x7f27fc1037c0 2024-03-30T04:52:27.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.016+0000 7f27f2000700 1 -- 172.21.15.123:0/1676075602 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f27e800a860 con 0x7f27fc1037c0 2024-03-30T04:52:27.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.016+0000 7f27f2000700 1 --2- 172.21.15.123:0/1676075602 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f27e006cc30 0x7f27e006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:52:27.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.016+0000 7f27f2000700 1 -- 172.21.15.123:0/1676075602 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f27e808c050 con 0x7f27fc1037c0 2024-03-30T04:52:27.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.017+0000 7f27f9e00700 1 --2- 172.21.15.123:0/1676075602 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f27e006cc30 0x7f27e006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:52:27.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.017+0000 7f27f9e00700 1 --2- 172.21.15.123:0/1676075602 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f27e006cc30 0x7f27e006f0e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f27ec00b840 tx=0x7f27ec009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:52:27.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.021+0000 7f27f2000700 1 -- 172.21.15.123:0/1676075602 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f27e8059b20 con 0x7f27fc1037c0 2024-03-30T04:52:27.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:52:26 smithi135 ceph-mon[106011]: pgmap v637: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:27.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:52:26 smithi123 ceph-mon[139369]: pgmap v637: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:27.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.240+0000 7f2800e00700 1 -- 172.21.15.123:0/1676075602 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f27fc109e50 con 0x7f27e006cc30 2024-03-30T04:52:27.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.241+0000 7f27f2000700 1 -- 172.21.15.123:0/1676075602 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f27fc109e50 con 0x7f27e006cc30 2024-03-30T04:52:27.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.244+0000 7f2800e00700 1 -- 172.21.15.123:0/1676075602 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f27e006cc30 msgr2=0x7f27e006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:52:27.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.244+0000 7f2800e00700 1 --2- 172.21.15.123:0/1676075602 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f27e006cc30 0x7f27e006f0e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f27ec00b840 tx=0x7f27ec009e30 comp rx=0 tx=0).stop 2024-03-30T04:52:27.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.244+0000 7f2800e00700 1 -- 172.21.15.123:0/1676075602 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27fc1037c0 msgr2=0x7f27fc116230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:52:27.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.244+0000 7f2800e00700 1 --2- 172.21.15.123:0/1676075602 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27fc1037c0 0x7f27fc116230 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f27e800c890 tx=0x7f27e80044c0 comp rx=0 tx=0).stop 2024-03-30T04:52:27.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.244+0000 7f2800e00700 1 -- 172.21.15.123:0/1676075602 shutdown_connections 2024-03-30T04:52:27.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.244+0000 7f2800e00700 1 --2- 172.21.15.123:0/1676075602 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f27e006cc30 0x7f27e006f0e0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:52:27.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.244+0000 7f2800e00700 1 --2- 172.21.15.123:0/1676075602 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f27fc102e20 0x7f27fc115cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:52:27.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.244+0000 7f2800e00700 1 --2- 172.21.15.123:0/1676075602 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27fc1037c0 0x7f27fc116230 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:52:27.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.244+0000 7f2800e00700 1 -- 172.21.15.123:0/1676075602 >> 172.21.15.123:0/1676075602 conn(0x7f27fc0f89d0 msgr2=0x7f27fc0f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:52:27.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.244+0000 7f2800e00700 1 -- 172.21.15.123:0/1676075602 shutdown_connections 2024-03-30T04:52:27.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.244+0000 7f2800e00700 1 -- 172.21.15.123:0/1676075602 wait complete. 2024-03-30T04:52:27.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.376+0000 7ffb62400700 1 -- 172.21.15.123:0/419837395 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb5c1035f0 msgr2=0x7ffb5c0fcc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:52:27.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.376+0000 7ffb62400700 1 --2- 172.21.15.123:0/419837395 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb5c1035f0 0x7ffb5c0fcc60 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7ffb48004940 tx=0x7ffb48009e30 comp rx=0 tx=0).stop 2024-03-30T04:52:27.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.376+0000 7ffb62400700 1 -- 172.21.15.123:0/419837395 shutdown_connections 2024-03-30T04:52:27.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.376+0000 7ffb62400700 1 --2- 172.21.15.123:0/419837395 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb5c1035f0 0x7ffb5c0fcc60 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:52:27.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.376+0000 7ffb62400700 1 --2- 172.21.15.123:0/419837395 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffb5c102c50 0x7ffb5c103020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:52:27.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.376+0000 7ffb62400700 1 -- 172.21.15.123:0/419837395 >> 172.21.15.123:0/419837395 conn(0x7ffb5c0f87e0 msgr2=0x7ffb5c0fabf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:52:27.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.376+0000 7ffb62400700 1 -- 172.21.15.123:0/419837395 shutdown_connections 2024-03-30T04:52:27.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.376+0000 7ffb62400700 1 -- 172.21.15.123:0/419837395 wait complete. 2024-03-30T04:52:27.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.377+0000 7ffb62400700 1 Processor -- start 2024-03-30T04:52:27.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.377+0000 7ffb62400700 1 -- start start 2024-03-30T04:52:27.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.378+0000 7ffb62400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb5c102c50 0x7ffb5c115ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:52:27.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.378+0000 7ffb62400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffb5c1035f0 0x7ffb5c115ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:52:27.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.378+0000 7ffb62400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5c116640 con 0x7ffb5c102c50 2024-03-30T04:52:27.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.378+0000 7ffb62400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5c10fa30 con 0x7ffb5c1035f0 2024-03-30T04:52:27.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.378+0000 7ffb61000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb5c102c50 0x7ffb5c115ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:52:27.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.378+0000 7ffb61000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb5c102c50 0x7ffb5c115ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42970/0 (socket says 172.21.15.123:42970) 2024-03-30T04:52:27.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.378+0000 7ffb61000700 1 -- 172.21.15.123:0/2926790681 learned_addr learned my addr 172.21.15.123:0/2926790681 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:52:27.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.378+0000 7ffb61000700 1 -- 172.21.15.123:0/2926790681 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffb5c1035f0 msgr2=0x7ffb5c115ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:52:27.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.378+0000 7ffb5be00700 1 --2- 172.21.15.123:0/2926790681 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffb5c1035f0 0x7ffb5c115ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:52:27.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.378+0000 7ffb61000700 1 --2- 172.21.15.123:0/2926790681 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffb5c1035f0 0x7ffb5c115ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:52:27.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.378+0000 7ffb61000700 1 -- 172.21.15.123:0/2926790681 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb48005040 con 0x7ffb5c102c50 2024-03-30T04:52:27.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.378+0000 7ffb5be00700 1 --2- 172.21.15.123:0/2926790681 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffb5c1035f0 0x7ffb5c115ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:52:27.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.379+0000 7ffb61000700 1 --2- 172.21.15.123:0/2926790681 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb5c102c50 0x7ffb5c115ab0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7ffb5000eb40 tx=0x7ffb5000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:52:27.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.379+0000 7ffb58c00700 1 -- 172.21.15.123:0/2926790681 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb50007cc0 con 0x7ffb5c102c50 2024-03-30T04:52:27.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.379+0000 7ffb58c00700 1 -- 172.21.15.123:0/2926790681 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffb5000f040 con 0x7ffb5c102c50 2024-03-30T04:52:27.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.379+0000 7ffb62400700 1 -- 172.21.15.123:0/2926790681 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb5c10fcb0 con 0x7ffb5c102c50 2024-03-30T04:52:27.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.379+0000 7ffb58c00700 1 -- 172.21.15.123:0/2926790681 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb50009570 con 0x7ffb5c102c50 2024-03-30T04:52:27.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.379+0000 7ffb62400700 1 -- 172.21.15.123:0/2926790681 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb5c110270 con 0x7ffb5c102c50 2024-03-30T04:52:27.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.380+0000 7ffb62400700 1 -- 172.21.15.123:0/2926790681 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb5c0fe3a0 con 0x7ffb5c102c50 2024-03-30T04:52:27.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.381+0000 7ffb58c00700 1 -- 172.21.15.123:0/2926790681 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7ffb500096d0 con 0x7ffb5c102c50 2024-03-30T04:52:27.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.382+0000 7ffb58c00700 1 --2- 172.21.15.123:0/2926790681 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ffb4406cc30 0x7ffb4406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:52:27.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.382+0000 7ffb58c00700 1 -- 172.21.15.123:0/2926790681 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ffb5008bda0 con 0x7ffb5c102c50 2024-03-30T04:52:27.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.382+0000 7ffb5be00700 1 --2- 172.21.15.123:0/2926790681 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ffb4406cc30 0x7ffb4406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:52:27.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.383+0000 7ffb5be00700 1 --2- 172.21.15.123:0/2926790681 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ffb4406cc30 0x7ffb4406f0e0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ffb48009fd0 tx=0x7ffb48009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:52:27.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.386+0000 7ffb58c00700 1 -- 172.21.15.123:0/2926790681 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ffb50059820 con 0x7ffb5c102c50 2024-03-30T04:52:27.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.604+0000 7ffb62400700 1 -- 172.21.15.123:0/2926790681 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ffb5c109be0 con 0x7ffb4406cc30 2024-03-30T04:52:27.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.614+0000 7ffb58c00700 1 -- 172.21.15.123:0/2926790681 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7ffb5c109be0 con 0x7ffb4406cc30 2024-03-30T04:52:27.613 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:52:27.613 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (21m) 14m ago 34m 15.0M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:52:27.613 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (18m) 14m ago 34m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T04:52:27.613 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (18m) 14m ago 33m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T04:52:27.613 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (21m) 14m ago 33m 74.0M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T04:52:27.613 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (14m) 14m ago 27m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T04:52:27.613 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (14m) 14m ago 27m 10.9M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T04:52:27.613 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (27m) 14m ago 27m 10.7M - 17.2.7-797-g5c024335 f77085f6aab7 0dc41326818c 2024-03-30T04:52:27.613 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (14m) 14m ago 27m 9747k - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T04:52:27.613 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (23m) 14m ago 35m 506M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:52:27.613 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (22m) 14m ago 33m 450M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:52:27.613 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (20m) 14m ago 35m 53.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T04:52:27.614 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (19m) 14m ago 32m 42.6M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T04:52:27.614 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (22m) 14m ago 34m 8052k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:52:27.614 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (22m) 14m ago 33m 7880k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:52:27.614 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (18m) 14m ago 32m 104M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T04:52:27.614 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (17m) 14m ago 31m 93.4M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T04:52:27.614 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (17m) 14m ago 31m 86.3M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T04:52:27.614 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (16m) 14m ago 30m 97.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T04:52:27.614 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (15m) 14m ago 29m 97.9M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T04:52:27.614 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (15m) 14m ago 29m 85.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T04:52:27.614 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (21m) 14m ago 33m 53.7M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:52:27.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.617+0000 7ffb62400700 1 -- 172.21.15.123:0/2926790681 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ffb4406cc30 msgr2=0x7ffb4406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:52:27.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.617+0000 7ffb62400700 1 --2- 172.21.15.123:0/2926790681 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ffb4406cc30 0x7ffb4406f0e0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ffb48009fd0 tx=0x7ffb48009e30 comp rx=0 tx=0).stop 2024-03-30T04:52:27.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.617+0000 7ffb62400700 1 -- 172.21.15.123:0/2926790681 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb5c102c50 msgr2=0x7ffb5c115ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:52:27.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.617+0000 7ffb62400700 1 --2- 172.21.15.123:0/2926790681 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb5c102c50 0x7ffb5c115ab0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7ffb5000eb40 tx=0x7ffb5000b040 comp rx=0 tx=0).stop 2024-03-30T04:52:27.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.617+0000 7ffb62400700 1 -- 172.21.15.123:0/2926790681 shutdown_connections 2024-03-30T04:52:27.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.617+0000 7ffb62400700 1 --2- 172.21.15.123:0/2926790681 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ffb4406cc30 0x7ffb4406f0e0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:52:27.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.617+0000 7ffb62400700 1 --2- 172.21.15.123:0/2926790681 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb5c102c50 0x7ffb5c115ab0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:52:27.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.617+0000 7ffb62400700 1 --2- 172.21.15.123:0/2926790681 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffb5c1035f0 0x7ffb5c115ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:52:27.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.617+0000 7ffb62400700 1 -- 172.21.15.123:0/2926790681 >> 172.21.15.123:0/2926790681 conn(0x7ffb5c0f87e0 msgr2=0x7ffb5c0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:52:27.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.617+0000 7ffb62400700 1 -- 172.21.15.123:0/2926790681 shutdown_connections 2024-03-30T04:52:27.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.617+0000 7ffb62400700 1 -- 172.21.15.123:0/2926790681 wait complete. 2024-03-30T04:52:27.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.750+0000 7f0ac2400700 1 -- 172.21.15.123:0/1043149859 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0abc102e00 msgr2=0x7f0abc1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:52:27.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.750+0000 7f0ac2400700 1 --2- 172.21.15.123:0/1043149859 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0abc102e00 0x7f0abc1031d0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f0aac004940 tx=0x7f0aac009e30 comp rx=0 tx=0).stop 2024-03-30T04:52:27.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.750+0000 7f0ac2400700 1 -- 172.21.15.123:0/1043149859 shutdown_connections 2024-03-30T04:52:27.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.750+0000 7f0ac2400700 1 --2- 172.21.15.123:0/1043149859 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0abc1037a0 0x7f0abc0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:52:27.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.750+0000 7f0ac2400700 1 --2- 172.21.15.123:0/1043149859 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0abc102e00 0x7f0abc1031d0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:52:27.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.750+0000 7f0ac2400700 1 -- 172.21.15.123:0/1043149859 >> 172.21.15.123:0/1043149859 conn(0x7f0abc0f89b0 msgr2=0x7f0abc0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:52:27.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.750+0000 7f0ac2400700 1 -- 172.21.15.123:0/1043149859 shutdown_connections 2024-03-30T04:52:27.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.750+0000 7f0ac2400700 1 -- 172.21.15.123:0/1043149859 wait complete. 2024-03-30T04:52:27.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.751+0000 7f0ac2400700 1 Processor -- start 2024-03-30T04:52:27.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.751+0000 7f0ac2400700 1 -- start start 2024-03-30T04:52:27.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.751+0000 7f0ac2400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0abc102e00 0x7f0abc115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:52:27.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.751+0000 7f0ac2400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0abc1037a0 0x7f0abc116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:52:27.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.751+0000 7f0ac2400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0abc116880 con 0x7f0abc1037a0 2024-03-30T04:52:27.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.751+0000 7f0ac2400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0abc10fc50 con 0x7f0abc102e00 2024-03-30T04:52:27.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.752+0000 7f0abaa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0abc1037a0 0x7f0abc116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:52:27.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.752+0000 7f0abaa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0abc1037a0 0x7f0abc116230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42986/0 (socket says 172.21.15.123:42986) 2024-03-30T04:52:27.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.752+0000 7f0abaa00700 1 -- 172.21.15.123:0/2090185891 learned_addr learned my addr 172.21.15.123:0/2090185891 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:52:27.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.752+0000 7f0abb400700 1 --2- 172.21.15.123:0/2090185891 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0abc102e00 0x7f0abc115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:52:27.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.752+0000 7f0abaa00700 1 -- 172.21.15.123:0/2090185891 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0abc102e00 msgr2=0x7f0abc115cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:52:27.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.752+0000 7f0abaa00700 1 --2- 172.21.15.123:0/2090185891 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0abc102e00 0x7f0abc115cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:52:27.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.752+0000 7f0abaa00700 1 -- 172.21.15.123:0/2090185891 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aac005040 con 0x7f0abc1037a0 2024-03-30T04:52:27.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.752+0000 7f0abaa00700 1 --2- 172.21.15.123:0/2090185891 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0abc1037a0 0x7f0abc116230 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f0aa8002a50 tx=0x7f0aa8004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:52:27.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.752+0000 7f0ab3400700 1 -- 172.21.15.123:0/2090185891 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0aa8007cc0 con 0x7f0abc1037a0 2024-03-30T04:52:27.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.752+0000 7f0ab3400700 1 -- 172.21.15.123:0/2090185891 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0aa800f040 con 0x7f0abc1037a0 2024-03-30T04:52:27.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.752+0000 7f0ac2400700 1 -- 172.21.15.123:0/2090185891 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0abc10ff30 con 0x7f0abc1037a0 2024-03-30T04:52:27.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.752+0000 7f0ab3400700 1 -- 172.21.15.123:0/2090185891 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0aa800d9a0 con 0x7f0abc1037a0 2024-03-30T04:52:27.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.753+0000 7f0ac2400700 1 -- 172.21.15.123:0/2090185891 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0abc1104f0 con 0x7f0abc1037a0 2024-03-30T04:52:27.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.754+0000 7f0ac2400700 1 -- 172.21.15.123:0/2090185891 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0abc0fe550 con 0x7f0abc1037a0 2024-03-30T04:52:27.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.755+0000 7f0ab3400700 1 -- 172.21.15.123:0/2090185891 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f0aa800dc10 con 0x7f0abc1037a0 2024-03-30T04:52:27.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.756+0000 7f0ab3400700 1 --2- 172.21.15.123:0/2090185891 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0aa006cc30 0x7f0aa006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:52:27.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.756+0000 7f0ab3400700 1 -- 172.21.15.123:0/2090185891 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f0aa808bdb0 con 0x7f0abc1037a0 2024-03-30T04:52:27.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.756+0000 7f0abb400700 1 --2- 172.21.15.123:0/2090185891 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0aa006cc30 0x7f0aa006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:52:27.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.756+0000 7f0abb400700 1 --2- 172.21.15.123:0/2090185891 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0aa006cc30 0x7f0aa006f0e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f0aac00b3b0 tx=0x7f0aac009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:52:27.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:27.760+0000 7f0ab3400700 1 -- 172.21.15.123:0/2090185891 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0aa8059880 con 0x7f0abc1037a0 2024-03-30T04:52:28.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.049+0000 7f0ac2400700 1 -- 172.21.15.123:0/2090185891 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0abc04bed0 con 0x7f0abc1037a0 2024-03-30T04:52:28.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.050+0000 7f0ab3400700 1 -- 172.21.15.123:0/2090185891 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f0aa8059410 con 0x7f0abc1037a0 2024-03-30T04:52:28.048 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:52:28.048 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:52:28.048 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:52:28.048 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:52:28.049 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:52:28.049 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:52:28.049 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:52:28.049 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:52:28.049 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T04:52:28.049 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:52:28.049 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:52:28.049 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:52:28.049 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 3 2024-03-30T04:52:28.049 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:52:28.049 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:52:28.049 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:52:28.049 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 13 2024-03-30T04:52:28.049 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:52:28.050 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:52:28.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.052+0000 7f0ac2400700 1 -- 172.21.15.123:0/2090185891 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0aa006cc30 msgr2=0x7f0aa006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:52:28.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.052+0000 7f0ac2400700 1 --2- 172.21.15.123:0/2090185891 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0aa006cc30 0x7f0aa006f0e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f0aac00b3b0 tx=0x7f0aac009e30 comp rx=0 tx=0).stop 2024-03-30T04:52:28.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.052+0000 7f0ac2400700 1 -- 172.21.15.123:0/2090185891 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0abc1037a0 msgr2=0x7f0abc116230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:52:28.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.052+0000 7f0ac2400700 1 --2- 172.21.15.123:0/2090185891 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0abc1037a0 0x7f0abc116230 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f0aa8002a50 tx=0x7f0aa8004600 comp rx=0 tx=0).stop 2024-03-30T04:52:28.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.052+0000 7f0ac2400700 1 -- 172.21.15.123:0/2090185891 shutdown_connections 2024-03-30T04:52:28.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.052+0000 7f0ac2400700 1 --2- 172.21.15.123:0/2090185891 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0aa006cc30 0x7f0aa006f0e0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:52:28.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.052+0000 7f0ac2400700 1 --2- 172.21.15.123:0/2090185891 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0abc102e00 0x7f0abc115cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:52:28.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.052+0000 7f0ac2400700 1 --2- 172.21.15.123:0/2090185891 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0abc1037a0 0x7f0abc116230 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:52:28.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.052+0000 7f0ac2400700 1 -- 172.21.15.123:0/2090185891 >> 172.21.15.123:0/2090185891 conn(0x7f0abc0f89b0 msgr2=0x7f0abc0706d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:52:28.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.053+0000 7f0ac2400700 1 -- 172.21.15.123:0/2090185891 shutdown_connections 2024-03-30T04:52:28.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.053+0000 7f0ac2400700 1 -- 172.21.15.123:0/2090185891 wait complete. 2024-03-30T04:52:28.057 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:52:27 smithi123 ceph-mon[139369]: from='client.35186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:52:28.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:52:27 smithi135 ceph-mon[106011]: from='client.35186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:52:28.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.188+0000 7f87ce600700 1 -- 172.21.15.123:0/1693630586 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87c8102e00 msgr2=0x7f87c81031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:52:28.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.188+0000 7f87ce600700 1 --2- 172.21.15.123:0/1693630586 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87c8102e00 0x7f87c81031d0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f87b8004940 tx=0x7f87b8009e30 comp rx=0 tx=0).stop 2024-03-30T04:52:28.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.188+0000 7f87ce600700 1 -- 172.21.15.123:0/1693630586 shutdown_connections 2024-03-30T04:52:28.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.188+0000 7f87ce600700 1 --2- 172.21.15.123:0/1693630586 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f87c81037a0 0x7f87c80fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:52:28.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.188+0000 7f87ce600700 1 --2- 172.21.15.123:0/1693630586 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87c8102e00 0x7f87c81031d0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:52:28.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.188+0000 7f87ce600700 1 -- 172.21.15.123:0/1693630586 >> 172.21.15.123:0/1693630586 conn(0x7f87c80f89b0 msgr2=0x7f87c80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:52:28.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.188+0000 7f87ce600700 1 -- 172.21.15.123:0/1693630586 shutdown_connections 2024-03-30T04:52:28.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.188+0000 7f87ce600700 1 -- 172.21.15.123:0/1693630586 wait complete. 2024-03-30T04:52:28.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.189+0000 7f87ce600700 1 Processor -- start 2024-03-30T04:52:28.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.189+0000 7f87ce600700 1 -- start start 2024-03-30T04:52:28.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.189+0000 7f87ce600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f87c8102e00 0x7f87c8115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:52:28.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.189+0000 7f87ce600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87c81037a0 0x7f87c81161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:52:28.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.189+0000 7f87ce600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c8116800 con 0x7f87c81037a0 2024-03-30T04:52:28.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.189+0000 7f87ce600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c810fbd0 con 0x7f87c8102e00 2024-03-30T04:52:28.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.190+0000 7f87c6a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87c81037a0 0x7f87c81161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:52:28.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.190+0000 7f87c6a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87c81037a0 0x7f87c81161b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43006/0 (socket says 172.21.15.123:43006) 2024-03-30T04:52:28.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.190+0000 7f87c6a00700 1 -- 172.21.15.123:0/2748377701 learned_addr learned my addr 172.21.15.123:0/2748377701 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:52:28.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.190+0000 7f87c7400700 1 --2- 172.21.15.123:0/2748377701 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f87c8102e00 0x7f87c8115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:52:28.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.190+0000 7f87c6a00700 1 -- 172.21.15.123:0/2748377701 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f87c8102e00 msgr2=0x7f87c8115c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:52:28.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.190+0000 7f87c6a00700 1 --2- 172.21.15.123:0/2748377701 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f87c8102e00 0x7f87c8115c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:52:28.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.190+0000 7f87c6a00700 1 -- 172.21.15.123:0/2748377701 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87b8005040 con 0x7f87c81037a0 2024-03-30T04:52:28.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.190+0000 7f87c6a00700 1 --2- 172.21.15.123:0/2748377701 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87c81037a0 0x7f87c81161b0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f87b400eb40 tx=0x7f87b400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:52:28.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.191+0000 7f87bf400700 1 -- 172.21.15.123:0/2748377701 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f87b4007cc0 con 0x7f87c81037a0 2024-03-30T04:52:28.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.191+0000 7f87ce600700 1 -- 172.21.15.123:0/2748377701 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87c810fe50 con 0x7f87c81037a0 2024-03-30T04:52:28.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.191+0000 7f87bf400700 1 -- 172.21.15.123:0/2748377701 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f87b400f040 con 0x7f87c81037a0 2024-03-30T04:52:28.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.191+0000 7f87bf400700 1 -- 172.21.15.123:0/2748377701 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f87b400c8f0 con 0x7f87c81037a0 2024-03-30T04:52:28.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.191+0000 7f87ce600700 1 -- 172.21.15.123:0/2748377701 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87c8110410 con 0x7f87c81037a0 2024-03-30T04:52:28.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.192+0000 7f87ce600700 1 -- 172.21.15.123:0/2748377701 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87c80fe550 con 0x7f87c81037a0 2024-03-30T04:52:28.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.193+0000 7f87bf400700 1 -- 172.21.15.123:0/2748377701 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f87b400cb60 con 0x7f87c81037a0 2024-03-30T04:52:28.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.194+0000 7f87bf400700 1 --2- 172.21.15.123:0/2748377701 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f87ac06cc30 0x7f87ac06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:52:28.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.194+0000 7f87bf400700 1 -- 172.21.15.123:0/2748377701 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f87b408bb50 con 0x7f87c81037a0 2024-03-30T04:52:28.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.194+0000 7f87c7400700 1 --2- 172.21.15.123:0/2748377701 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f87ac06cc30 0x7f87ac06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:52:28.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.195+0000 7f87c7400700 1 --2- 172.21.15.123:0/2748377701 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f87ac06cc30 0x7f87ac06f0e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f87b80045a0 tx=0x7f87b8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:52:28.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.198+0000 7f87bf400700 1 -- 172.21.15.123:0/2748377701 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f87b40595d0 con 0x7f87c81037a0 2024-03-30T04:52:28.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.440+0000 7f87ce600700 1 -- 172.21.15.123:0/2748377701 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f87c804bed0 con 0x7f87c81037a0 2024-03-30T04:52:28.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.441+0000 7f87bf400700 1 -- 172.21.15.123:0/2748377701 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f87b4059160 con 0x7f87c81037a0 2024-03-30T04:52:28.439 INFO:teuthology.orchestra.run.smithi123.stdout:e32 2024-03-30T04:52:28.440 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:52:28.440 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:52:28.440 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:52:28.440 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:52:28.440 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:52:28.440 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:52:28.440 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T04:52:28.440 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T04:52:28.440 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:52:28.440 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T04:52:28.440 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:52:28.440 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:52:28.440 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:52:28.440 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:52:28.441 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:52:28.441 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T04:52:28.441 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:52:28.441 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:52:28.441 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T04:52:28.441 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:52:28.441 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T04:52:28.441 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T04:52:28.441 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T04:52:28.441 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T04:52:28.441 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:52:28.441 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:52:28.441 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:52:28.441 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:52:28.441 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:52:28.442 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:52:28.442 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T04:52:28.442 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T04:52:28.442 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:52:28.442 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:52:28.442 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T04:52:28.442 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:52:28.442 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:52:28.442 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:52:28.442 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44315} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/1741352336,v1:172.21.15.135:6827/1741352336] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:52:28.442 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:52:28.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.443+0000 7f87ce600700 1 -- 172.21.15.123:0/2748377701 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f87ac06cc30 msgr2=0x7f87ac06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:52:28.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.443+0000 7f87ce600700 1 --2- 172.21.15.123:0/2748377701 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f87ac06cc30 0x7f87ac06f0e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f87b80045a0 tx=0x7f87b8009e30 comp rx=0 tx=0).stop 2024-03-30T04:52:28.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.443+0000 7f87ce600700 1 -- 172.21.15.123:0/2748377701 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87c81037a0 msgr2=0x7f87c81161b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:52:28.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.443+0000 7f87ce600700 1 --2- 172.21.15.123:0/2748377701 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87c81037a0 0x7f87c81161b0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f87b400eb40 tx=0x7f87b400b040 comp rx=0 tx=0).stop 2024-03-30T04:52:28.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.443+0000 7f87ce600700 1 -- 172.21.15.123:0/2748377701 shutdown_connections 2024-03-30T04:52:28.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.443+0000 7f87ce600700 1 --2- 172.21.15.123:0/2748377701 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f87ac06cc30 0x7f87ac06f0e0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:52:28.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.443+0000 7f87ce600700 1 --2- 172.21.15.123:0/2748377701 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f87c8102e00 0x7f87c8115c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:52:28.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.443+0000 7f87ce600700 1 --2- 172.21.15.123:0/2748377701 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87c81037a0 0x7f87c81161b0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:52:28.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.443+0000 7f87ce600700 1 -- 172.21.15.123:0/2748377701 >> 172.21.15.123:0/2748377701 conn(0x7f87c80f89b0 msgr2=0x7f87c80f9960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:52:28.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.444+0000 7f87ce600700 1 -- 172.21.15.123:0/2748377701 shutdown_connections 2024-03-30T04:52:28.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.444+0000 7f87ce600700 1 -- 172.21.15.123:0/2748377701 wait complete. 2024-03-30T04:52:28.444 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 32 2024-03-30T04:52:28.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.578+0000 7f1c2ae00700 1 -- 172.21.15.123:0/4011266888 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c24102e00 msgr2=0x7f1c241031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:52:28.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.578+0000 7f1c2ae00700 1 --2- 172.21.15.123:0/4011266888 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c24102e00 0x7f1c241031d0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f1c14004940 tx=0x7f1c14009e30 comp rx=0 tx=0).stop 2024-03-30T04:52:28.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.578+0000 7f1c2ae00700 1 -- 172.21.15.123:0/4011266888 shutdown_connections 2024-03-30T04:52:28.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.578+0000 7f1c2ae00700 1 --2- 172.21.15.123:0/4011266888 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1c241037a0 0x7f1c240fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:52:28.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.578+0000 7f1c2ae00700 1 --2- 172.21.15.123:0/4011266888 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c24102e00 0x7f1c241031d0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:52:28.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.578+0000 7f1c2ae00700 1 -- 172.21.15.123:0/4011266888 >> 172.21.15.123:0/4011266888 conn(0x7f1c240f89b0 msgr2=0x7f1c240fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:52:28.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.578+0000 7f1c2ae00700 1 -- 172.21.15.123:0/4011266888 shutdown_connections 2024-03-30T04:52:28.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.578+0000 7f1c2ae00700 1 -- 172.21.15.123:0/4011266888 wait complete. 2024-03-30T04:52:28.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.579+0000 7f1c2ae00700 1 Processor -- start 2024-03-30T04:52:28.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.579+0000 7f1c2ae00700 1 -- start start 2024-03-30T04:52:28.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.579+0000 7f1c2ae00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1c24102e00 0x7f1c240702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:52:28.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.579+0000 7f1c2ae00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c241037a0 0x7f1c24070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:52:28.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.580+0000 7f1c2ae00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c24070d90 con 0x7f1c241037a0 2024-03-30T04:52:28.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.580+0000 7f1c2ae00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c24070ed0 con 0x7f1c24102e00 2024-03-30T04:52:28.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.580+0000 7f1c23400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c241037a0 0x7f1c24070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:52:28.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.580+0000 7f1c23400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c241037a0 0x7f1c24070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43036/0 (socket says 172.21.15.123:43036) 2024-03-30T04:52:28.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.580+0000 7f1c23400700 1 -- 172.21.15.123:0/968860115 learned_addr learned my addr 172.21.15.123:0/968860115 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:52:28.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.580+0000 7f1c23e00700 1 --2- 172.21.15.123:0/968860115 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1c24102e00 0x7f1c240702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:52:28.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.580+0000 7f1c23400700 1 -- 172.21.15.123:0/968860115 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1c24102e00 msgr2=0x7f1c240702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:52:28.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.580+0000 7f1c23400700 1 --2- 172.21.15.123:0/968860115 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1c24102e00 0x7f1c240702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:52:28.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.580+0000 7f1c23400700 1 -- 172.21.15.123:0/968860115 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c14005040 con 0x7f1c241037a0 2024-03-30T04:52:28.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.580+0000 7f1c23400700 1 --2- 172.21.15.123:0/968860115 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c241037a0 0x7f1c24070800 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f1c10002a50 tx=0x7f1c10004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:52:28.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.581+0000 7f1c1be00700 1 -- 172.21.15.123:0/968860115 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1c10007cc0 con 0x7f1c241037a0 2024-03-30T04:52:28.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.581+0000 7f1c1be00700 1 -- 172.21.15.123:0/968860115 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1c1000f040 con 0x7f1c241037a0 2024-03-30T04:52:28.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.581+0000 7f1c2ae00700 1 -- 172.21.15.123:0/968860115 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c24119e20 con 0x7f1c241037a0 2024-03-30T04:52:28.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.581+0000 7f1c1be00700 1 -- 172.21.15.123:0/968860115 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1c1000d920 con 0x7f1c241037a0 2024-03-30T04:52:28.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.581+0000 7f1c2ae00700 1 -- 172.21.15.123:0/968860115 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c2411a2a0 con 0x7f1c241037a0 2024-03-30T04:52:28.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.582+0000 7f1c2ae00700 1 -- 172.21.15.123:0/968860115 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c240fe550 con 0x7f1c241037a0 2024-03-30T04:52:28.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.583+0000 7f1c1be00700 1 -- 172.21.15.123:0/968860115 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f1c10009680 con 0x7f1c241037a0 2024-03-30T04:52:28.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.584+0000 7f1c1be00700 1 --2- 172.21.15.123:0/968860115 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1c0806cc30 0x7f1c0806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:52:28.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.584+0000 7f1c1be00700 1 -- 172.21.15.123:0/968860115 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f1c1008bcf0 con 0x7f1c241037a0 2024-03-30T04:52:28.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.584+0000 7f1c23e00700 1 --2- 172.21.15.123:0/968860115 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1c0806cc30 0x7f1c0806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:52:28.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.585+0000 7f1c23e00700 1 --2- 172.21.15.123:0/968860115 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1c0806cc30 0x7f1c0806f0e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f1c1400b840 tx=0x7f1c14009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:52:28.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.588+0000 7f1c1be00700 1 -- 172.21.15.123:0/968860115 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f1c10055cb0 con 0x7f1c241037a0 2024-03-30T04:52:28.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.808+0000 7f1c2ae00700 1 -- 172.21.15.123:0/968860115 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1c2404bed0 con 0x7f1c0806cc30 2024-03-30T04:52:28.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.809+0000 7f1c1be00700 1 -- 172.21.15.123:0/968860115 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f1c2404bed0 con 0x7f1c0806cc30 2024-03-30T04:52:28.807 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:52:28.807 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:52:28.807 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:52:28.808 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T04:52:28.808 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:52:28.808 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T04:52:28.808 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T04:52:28.808 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T04:52:28.808 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T04:52:28.808 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:52:28.808 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "15/21 daemons upgraded", 2024-03-30T04:52:28.808 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T04:52:28.808 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:52:28.808 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:52:28.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.811+0000 7f1c2ae00700 1 -- 172.21.15.123:0/968860115 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1c0806cc30 msgr2=0x7f1c0806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:52:28.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.811+0000 7f1c2ae00700 1 --2- 172.21.15.123:0/968860115 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1c0806cc30 0x7f1c0806f0e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f1c1400b840 tx=0x7f1c14009e30 comp rx=0 tx=0).stop 2024-03-30T04:52:28.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.811+0000 7f1c2ae00700 1 -- 172.21.15.123:0/968860115 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c241037a0 msgr2=0x7f1c24070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:52:28.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.811+0000 7f1c2ae00700 1 --2- 172.21.15.123:0/968860115 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c241037a0 0x7f1c24070800 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f1c10002a50 tx=0x7f1c10004600 comp rx=0 tx=0).stop 2024-03-30T04:52:28.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.811+0000 7f1c2ae00700 1 -- 172.21.15.123:0/968860115 shutdown_connections 2024-03-30T04:52:28.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.811+0000 7f1c2ae00700 1 --2- 172.21.15.123:0/968860115 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1c0806cc30 0x7f1c0806f0e0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:52:28.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.811+0000 7f1c2ae00700 1 --2- 172.21.15.123:0/968860115 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1c24102e00 0x7f1c240702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:52:28.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.811+0000 7f1c2ae00700 1 --2- 172.21.15.123:0/968860115 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c241037a0 0x7f1c24070800 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:52:28.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.811+0000 7f1c2ae00700 1 -- 172.21.15.123:0/968860115 >> 172.21.15.123:0/968860115 conn(0x7f1c240f89b0 msgr2=0x7f1c240f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:52:28.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.812+0000 7f1c2ae00700 1 -- 172.21.15.123:0/968860115 shutdown_connections 2024-03-30T04:52:28.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.812+0000 7f1c2ae00700 1 -- 172.21.15.123:0/968860115 wait complete. 2024-03-30T04:52:28.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.944+0000 7ffb51c00700 1 -- 172.21.15.123:0/2170123388 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb4c1037a0 msgr2=0x7ffb4c0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:52:28.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.944+0000 7ffb51c00700 1 --2- 172.21.15.123:0/2170123388 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb4c1037a0 0x7ffb4c0fce10 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7ffb38004940 tx=0x7ffb38009e30 comp rx=0 tx=0).stop 2024-03-30T04:52:28.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.944+0000 7ffb51c00700 1 -- 172.21.15.123:0/2170123388 shutdown_connections 2024-03-30T04:52:28.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.944+0000 7ffb51c00700 1 --2- 172.21.15.123:0/2170123388 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb4c1037a0 0x7ffb4c0fce10 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:52:28.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.944+0000 7ffb51c00700 1 --2- 172.21.15.123:0/2170123388 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffb4c102e00 0x7ffb4c1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:52:28.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.944+0000 7ffb51c00700 1 -- 172.21.15.123:0/2170123388 >> 172.21.15.123:0/2170123388 conn(0x7ffb4c0f89b0 msgr2=0x7ffb4c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:52:28.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.944+0000 7ffb51c00700 1 -- 172.21.15.123:0/2170123388 shutdown_connections 2024-03-30T04:52:28.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.944+0000 7ffb51c00700 1 -- 172.21.15.123:0/2170123388 wait complete. 2024-03-30T04:52:28.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.945+0000 7ffb51c00700 1 Processor -- start 2024-03-30T04:52:28.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.946+0000 7ffb51c00700 1 -- start start 2024-03-30T04:52:28.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.946+0000 7ffb51c00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffb4c102e00 0x7ffb4c113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:52:28.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.946+0000 7ffb51c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb4c1037a0 0x7ffb4c113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:52:28.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.946+0000 7ffb51c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4c114600 con 0x7ffb4c1037a0 2024-03-30T04:52:28.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.946+0000 7ffb51c00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4c10d9f0 con 0x7ffb4c102e00 2024-03-30T04:52:28.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.946+0000 7ffb4a000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb4c1037a0 0x7ffb4c113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:52:28.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.947+0000 7ffb4a000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb4c1037a0 0x7ffb4c113fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43054/0 (socket says 172.21.15.123:43054) 2024-03-30T04:52:28.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.947+0000 7ffb4a000700 1 -- 172.21.15.123:0/1129364093 learned_addr learned my addr 172.21.15.123:0/1129364093 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:52:28.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.947+0000 7ffb4aa00700 1 --2- 172.21.15.123:0/1129364093 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffb4c102e00 0x7ffb4c113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:52:28.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.947+0000 7ffb4a000700 1 -- 172.21.15.123:0/1129364093 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffb4c102e00 msgr2=0x7ffb4c113a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:52:28.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.947+0000 7ffb4a000700 1 --2- 172.21.15.123:0/1129364093 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffb4c102e00 0x7ffb4c113a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:52:28.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.947+0000 7ffb4a000700 1 -- 172.21.15.123:0/1129364093 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb38005040 con 0x7ffb4c1037a0 2024-03-30T04:52:28.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.947+0000 7ffb4aa00700 1 --2- 172.21.15.123:0/1129364093 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffb4c102e00 0x7ffb4c113a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:52:28.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.947+0000 7ffb4a000700 1 --2- 172.21.15.123:0/1129364093 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb4c1037a0 0x7ffb4c113fb0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7ffb38008d50 tx=0x7ffb38009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:52:28.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.947+0000 7ffb42a00700 1 -- 172.21.15.123:0/1129364093 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb3800e030 con 0x7ffb4c1037a0 2024-03-30T04:52:28.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.947+0000 7ffb51c00700 1 -- 172.21.15.123:0/1129364093 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb4c10dc70 con 0x7ffb4c1037a0 2024-03-30T04:52:28.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.947+0000 7ffb42a00700 1 -- 172.21.15.123:0/1129364093 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffb38013de0 con 0x7ffb4c1037a0 2024-03-30T04:52:28.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.947+0000 7ffb42a00700 1 -- 172.21.15.123:0/1129364093 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb3801cd90 con 0x7ffb4c1037a0 2024-03-30T04:52:28.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.948+0000 7ffb51c00700 1 -- 172.21.15.123:0/1129364093 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb4c10e230 con 0x7ffb4c1037a0 2024-03-30T04:52:28.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.949+0000 7ffb51c00700 1 -- 172.21.15.123:0/1129364093 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb4c0fe550 con 0x7ffb4c1037a0 2024-03-30T04:52:28.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.950+0000 7ffb42a00700 1 -- 172.21.15.123:0/1129364093 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7ffb38013930 con 0x7ffb4c1037a0 2024-03-30T04:52:28.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.951+0000 7ffb42a00700 1 --2- 172.21.15.123:0/1129364093 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ffb3006cc30 0x7ffb3006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:52:28.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.951+0000 7ffb42a00700 1 -- 172.21.15.123:0/1129364093 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ffb3808d980 con 0x7ffb4c1037a0 2024-03-30T04:52:28.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.951+0000 7ffb4aa00700 1 --2- 172.21.15.123:0/1129364093 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ffb3006cc30 0x7ffb3006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:52:28.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.951+0000 7ffb4aa00700 1 --2- 172.21.15.123:0/1129364093 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ffb3006cc30 0x7ffb3006f0e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c004600 tx=0x7ffb3c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:52:28.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:28.955+0000 7ffb42a00700 1 -- 172.21.15.123:0/1129364093 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ffb3805b3d0 con 0x7ffb4c1037a0 2024-03-30T04:52:29.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:52:28 smithi135 ceph-mon[106011]: from='client.35190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:52:29.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:52:28 smithi135 ceph-mon[106011]: from='client.35194 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:52:29.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:52:28 smithi135 ceph-mon[106011]: pgmap v638: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:29.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:52:28 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2090185891' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:52:29.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:52:28 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2748377701' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:52:29.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:52:28 smithi123 ceph-mon[139369]: from='client.35190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:52:29.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:52:28 smithi123 ceph-mon[139369]: from='client.35194 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:52:29.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:52:28 smithi123 ceph-mon[139369]: pgmap v638: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:29.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:52:28 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2090185891' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:52:29.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:52:28 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2748377701' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:52:29.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:29.238+0000 7ffb51c00700 1 -- 172.21.15.123:0/1129364093 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7ffb4c05f5f0 con 0x7ffb4c1037a0 2024-03-30T04:52:29.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:29.238+0000 7ffb42a00700 1 -- 172.21.15.123:0/1129364093 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7ffb3805af60 con 0x7ffb4c1037a0 2024-03-30T04:52:29.237 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T04:52:29.237 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T04:52:29.237 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T04:52:29.237 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T04:52:29.237 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T04:52:29.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:29.241+0000 7ffb51c00700 1 -- 172.21.15.123:0/1129364093 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ffb3006cc30 msgr2=0x7ffb3006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:52:29.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:29.241+0000 7ffb51c00700 1 --2- 172.21.15.123:0/1129364093 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ffb3006cc30 0x7ffb3006f0e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c004600 tx=0x7ffb3c00b040 comp rx=0 tx=0).stop 2024-03-30T04:52:29.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:29.241+0000 7ffb51c00700 1 -- 172.21.15.123:0/1129364093 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb4c1037a0 msgr2=0x7ffb4c113fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:52:29.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:29.241+0000 7ffb51c00700 1 --2- 172.21.15.123:0/1129364093 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb4c1037a0 0x7ffb4c113fb0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7ffb38008d50 tx=0x7ffb38009e30 comp rx=0 tx=0).stop 2024-03-30T04:52:29.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:29.241+0000 7ffb51c00700 1 -- 172.21.15.123:0/1129364093 shutdown_connections 2024-03-30T04:52:29.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:29.241+0000 7ffb51c00700 1 --2- 172.21.15.123:0/1129364093 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7ffb3006cc30 0x7ffb3006f0e0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:52:29.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:29.241+0000 7ffb51c00700 1 --2- 172.21.15.123:0/1129364093 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffb4c102e00 0x7ffb4c113a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:52:29.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:29.241+0000 7ffb51c00700 1 --2- 172.21.15.123:0/1129364093 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb4c1037a0 0x7ffb4c113fb0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:52:29.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:29.241+0000 7ffb51c00700 1 -- 172.21.15.123:0/1129364093 >> 172.21.15.123:0/1129364093 conn(0x7ffb4c0f89b0 msgr2=0x7ffb4c0f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:52:29.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:29.241+0000 7ffb51c00700 1 -- 172.21.15.123:0/1129364093 shutdown_connections 2024-03-30T04:52:29.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:29.241+0000 7ffb51c00700 1 -- 172.21.15.123:0/1129364093 wait complete. 2024-03-30T04:52:30.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:52:29 smithi135 ceph-mon[106011]: from='client.35206 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:52:30.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:52:29 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/1129364093' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:52:30.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:52:29 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:52:30.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:52:29 smithi123 ceph-mon[139369]: from='client.35206 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:52:30.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:52:29 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/1129364093' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:52:30.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:52:29 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:52:31.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:52:30 smithi135 ceph-mon[106011]: pgmap v639: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:31.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:52:30 smithi123 ceph-mon[139369]: pgmap v639: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:33.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:52:32 smithi135 ceph-mon[106011]: pgmap v640: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:33.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:52:32 smithi123 ceph-mon[139369]: pgmap v640: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:35.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:52:34 smithi135 ceph-mon[106011]: pgmap v641: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:35.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:52:34 smithi123 ceph-mon[139369]: pgmap v641: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:37.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:52:36 smithi135 ceph-mon[106011]: pgmap v642: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:37.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:52:36 smithi123 ceph-mon[139369]: pgmap v642: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:39.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:52:38 smithi135 ceph-mon[106011]: pgmap v643: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:39.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:52:38 smithi123 ceph-mon[139369]: pgmap v643: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:40.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:52:40 smithi123 ceph-mon[139369]: pgmap v644: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:41.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:52:40 smithi135 ceph-mon[106011]: pgmap v644: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:43.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:52:42 smithi135 ceph-mon[106011]: pgmap v645: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:43.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:52:42 smithi123 ceph-mon[139369]: pgmap v645: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:45.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:52:44 smithi135 ceph-mon[106011]: pgmap v646: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:45.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:52:44 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:52:45.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:52:44 smithi123 ceph-mon[139369]: pgmap v646: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:45.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:52:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:52:47.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:52:46 smithi135 ceph-mon[106011]: pgmap v647: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:47.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:52:46 smithi123 ceph-mon[139369]: pgmap v647: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:49.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:52:48 smithi135 ceph-mon[106011]: pgmap v648: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:49.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:52:48 smithi123 ceph-mon[139369]: pgmap v648: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:50.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:52:50 smithi123 ceph-mon[139369]: pgmap v649: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:51.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:52:50 smithi135 ceph-mon[106011]: pgmap v649: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:53.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:52:52 smithi135 ceph-mon[106011]: pgmap v650: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:53.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:52:52 smithi123 ceph-mon[139369]: pgmap v650: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:55.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:52:54 smithi135 ceph-mon[106011]: pgmap v651: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:55.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:52:54 smithi123 ceph-mon[139369]: pgmap v651: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:57.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:52:56 smithi135 ceph-mon[106011]: pgmap v652: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:57.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:52:56 smithi123 ceph-mon[139369]: pgmap v652: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:59.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:52:58 smithi135 ceph-mon[106011]: pgmap v653: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:59.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:52:58 smithi123 ceph-mon[139369]: pgmap v653: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:52:59.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.388+0000 7f60edc00700 1 -- 172.21.15.123:0/3653761232 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60e8071f90 msgr2=0x7f60e8070360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:52:59.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.388+0000 7f60edc00700 1 --2- 172.21.15.123:0/3653761232 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60e8071f90 0x7f60e8070360 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f60d8004970 tx=0x7f60d8009e30 comp rx=0 tx=0).stop 2024-03-30T04:52:59.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.388+0000 7f60edc00700 1 -- 172.21.15.123:0/3653761232 shutdown_connections 2024-03-30T04:52:59.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.388+0000 7f60edc00700 1 --2- 172.21.15.123:0/3653761232 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f60e80708a0 0x7f60e8070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:52:59.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.388+0000 7f60edc00700 1 --2- 172.21.15.123:0/3653761232 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60e8071f90 0x7f60e8070360 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:52:59.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.388+0000 7f60edc00700 1 -- 172.21.15.123:0/3653761232 >> 172.21.15.123:0/3653761232 conn(0x7f60e80f89b0 msgr2=0x7f60e80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:52:59.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.388+0000 7f60edc00700 1 -- 172.21.15.123:0/3653761232 shutdown_connections 2024-03-30T04:52:59.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.388+0000 7f60edc00700 1 -- 172.21.15.123:0/3653761232 wait complete. 2024-03-30T04:52:59.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.389+0000 7f60edc00700 1 Processor -- start 2024-03-30T04:52:59.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.390+0000 7f60edc00700 1 -- start start 2024-03-30T04:52:59.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.390+0000 7f60edc00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f60e80708a0 0x7f60e811a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:52:59.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.390+0000 7f60edc00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60e8071f90 0x7f60e811a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:52:59.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.390+0000 7f60edc00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60e811ac50 con 0x7f60e8071f90 2024-03-30T04:52:59.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.390+0000 7f60edc00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60e8114020 con 0x7f60e80708a0 2024-03-30T04:52:59.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.391+0000 7f60e6000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60e8071f90 0x7f60e811a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:52:59.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.391+0000 7f60e6000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60e8071f90 0x7f60e811a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49552/0 (socket says 172.21.15.123:49552) 2024-03-30T04:52:59.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.391+0000 7f60e6000700 1 -- 172.21.15.123:0/14931982 learned_addr learned my addr 172.21.15.123:0/14931982 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:52:59.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.391+0000 7f60e6a00700 1 --2- 172.21.15.123:0/14931982 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f60e80708a0 0x7f60e811a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:52:59.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.391+0000 7f60e6000700 1 -- 172.21.15.123:0/14931982 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f60e80708a0 msgr2=0x7f60e811a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:52:59.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.391+0000 7f60e6000700 1 --2- 172.21.15.123:0/14931982 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f60e80708a0 0x7f60e811a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:52:59.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.391+0000 7f60e6000700 1 -- 172.21.15.123:0/14931982 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60d8005040 con 0x7f60e8071f90 2024-03-30T04:52:59.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.391+0000 7f60e6a00700 1 --2- 172.21.15.123:0/14931982 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f60e80708a0 0x7f60e811a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:52:59.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.391+0000 7f60e6000700 1 --2- 172.21.15.123:0/14931982 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60e8071f90 0x7f60e811a600 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f60d400eb40 tx=0x7f60d400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:52:59.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.391+0000 7f60dea00700 1 -- 172.21.15.123:0/14931982 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f60d4007cc0 con 0x7f60e8071f90 2024-03-30T04:52:59.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.392+0000 7f60dea00700 1 -- 172.21.15.123:0/14931982 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f60d400f040 con 0x7f60e8071f90 2024-03-30T04:52:59.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.392+0000 7f60edc00700 1 -- 172.21.15.123:0/14931982 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60e81142a0 con 0x7f60e8071f90 2024-03-30T04:52:59.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.392+0000 7f60dea00700 1 -- 172.21.15.123:0/14931982 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f60d4009570 con 0x7f60e8071f90 2024-03-30T04:52:59.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.392+0000 7f60edc00700 1 -- 172.21.15.123:0/14931982 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60e81147e0 con 0x7f60e8071f90 2024-03-30T04:52:59.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.393+0000 7f60edc00700 1 -- 172.21.15.123:0/14931982 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60e8071aa0 con 0x7f60e8071f90 2024-03-30T04:52:59.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.394+0000 7f60dea00700 1 -- 172.21.15.123:0/14931982 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f60d40096d0 con 0x7f60e8071f90 2024-03-30T04:52:59.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.394+0000 7f60dea00700 1 --2- 172.21.15.123:0/14931982 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f60cc06cc30 0x7f60cc06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:52:59.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.394+0000 7f60dea00700 1 -- 172.21.15.123:0/14931982 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f60d408bcb0 con 0x7f60e8071f90 2024-03-30T04:52:59.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.395+0000 7f60e6a00700 1 --2- 172.21.15.123:0/14931982 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f60cc06cc30 0x7f60cc06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:52:59.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.395+0000 7f60e6a00700 1 --2- 172.21.15.123:0/14931982 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f60cc06cc30 0x7f60cc06f0e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f60d8009fd0 tx=0x7f60d8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:52:59.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.399+0000 7f60dea00700 1 -- 172.21.15.123:0/14931982 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f60d40597b0 con 0x7f60e8071f90 2024-03-30T04:52:59.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.625+0000 7f60edc00700 1 -- 172.21.15.123:0/14931982 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f60e804bed0 con 0x7f60cc06cc30 2024-03-30T04:52:59.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.627+0000 7f60dea00700 1 -- 172.21.15.123:0/14931982 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f60e804bed0 con 0x7f60cc06cc30 2024-03-30T04:52:59.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.629+0000 7f60edc00700 1 -- 172.21.15.123:0/14931982 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f60cc06cc30 msgr2=0x7f60cc06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:52:59.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.629+0000 7f60edc00700 1 --2- 172.21.15.123:0/14931982 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f60cc06cc30 0x7f60cc06f0e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f60d8009fd0 tx=0x7f60d8009e30 comp rx=0 tx=0).stop 2024-03-30T04:52:59.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.629+0000 7f60edc00700 1 -- 172.21.15.123:0/14931982 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60e8071f90 msgr2=0x7f60e811a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:52:59.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.629+0000 7f60edc00700 1 --2- 172.21.15.123:0/14931982 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60e8071f90 0x7f60e811a600 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f60d400eb40 tx=0x7f60d400b040 comp rx=0 tx=0).stop 2024-03-30T04:52:59.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.629+0000 7f60edc00700 1 -- 172.21.15.123:0/14931982 shutdown_connections 2024-03-30T04:52:59.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.629+0000 7f60edc00700 1 --2- 172.21.15.123:0/14931982 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f60cc06cc30 0x7f60cc06f0e0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:52:59.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.629+0000 7f60edc00700 1 --2- 172.21.15.123:0/14931982 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f60e80708a0 0x7f60e811a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:52:59.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.629+0000 7f60edc00700 1 --2- 172.21.15.123:0/14931982 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60e8071f90 0x7f60e811a600 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:52:59.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.629+0000 7f60edc00700 1 -- 172.21.15.123:0/14931982 >> 172.21.15.123:0/14931982 conn(0x7f60e80f89b0 msgr2=0x7f60e80f9b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:52:59.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.629+0000 7f60edc00700 1 -- 172.21.15.123:0/14931982 shutdown_connections 2024-03-30T04:52:59.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.629+0000 7f60edc00700 1 -- 172.21.15.123:0/14931982 wait complete. 2024-03-30T04:52:59.642 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:52:59.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.765+0000 7f968b000700 1 -- 172.21.15.123:0/3660925538 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9684102e00 msgr2=0x7f96841031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:52:59.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.765+0000 7f968b000700 1 --2- 172.21.15.123:0/3660925538 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9684102e00 0x7f96841031d0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f9674004940 tx=0x7f9674009e30 comp rx=0 tx=0).stop 2024-03-30T04:52:59.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.765+0000 7f968b000700 1 -- 172.21.15.123:0/3660925538 shutdown_connections 2024-03-30T04:52:59.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.765+0000 7f968b000700 1 --2- 172.21.15.123:0/3660925538 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f96841037a0 0x7f96840fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:52:59.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.765+0000 7f968b000700 1 --2- 172.21.15.123:0/3660925538 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9684102e00 0x7f96841031d0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:52:59.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.765+0000 7f968b000700 1 -- 172.21.15.123:0/3660925538 >> 172.21.15.123:0/3660925538 conn(0x7f96840f89b0 msgr2=0x7f96840fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:52:59.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.765+0000 7f968b000700 1 -- 172.21.15.123:0/3660925538 shutdown_connections 2024-03-30T04:52:59.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.765+0000 7f968b000700 1 -- 172.21.15.123:0/3660925538 wait complete. 2024-03-30T04:52:59.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.766+0000 7f968b000700 1 Processor -- start 2024-03-30T04:52:59.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.766+0000 7f968b000700 1 -- start start 2024-03-30T04:52:59.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.766+0000 7f968b000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9684102e00 0x7f9684115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:52:59.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.766+0000 7f968b000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f96841037a0 0x7f9684116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:52:59.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.766+0000 7f968b000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9684116860 con 0x7f9684102e00 2024-03-30T04:52:59.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.766+0000 7f968b000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f968410fc30 con 0x7f96841037a0 2024-03-30T04:52:59.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.767+0000 7f9683e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9684102e00 0x7f9684115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:52:59.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.767+0000 7f9683e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9684102e00 0x7f9684115cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49582/0 (socket says 172.21.15.123:49582) 2024-03-30T04:52:59.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.767+0000 7f9683e00700 1 -- 172.21.15.123:0/871232811 learned_addr learned my addr 172.21.15.123:0/871232811 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:52:59.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.767+0000 7f9683e00700 1 -- 172.21.15.123:0/871232811 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f96841037a0 msgr2=0x7f9684116210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:52:59.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.767+0000 7f9683e00700 1 --2- 172.21.15.123:0/871232811 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f96841037a0 0x7f9684116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:52:59.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.767+0000 7f9683e00700 1 -- 172.21.15.123:0/871232811 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9674005040 con 0x7f9684102e00 2024-03-30T04:52:59.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.767+0000 7f9683e00700 1 --2- 172.21.15.123:0/871232811 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9684102e00 0x7f9684115cd0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f9674009360 tx=0x7f9674009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:52:59.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.767+0000 7f967be00700 1 -- 172.21.15.123:0/871232811 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f967400e040 con 0x7f9684102e00 2024-03-30T04:52:59.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.767+0000 7f967be00700 1 -- 172.21.15.123:0/871232811 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9674013420 con 0x7f9684102e00 2024-03-30T04:52:59.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.767+0000 7f968b000700 1 -- 172.21.15.123:0/871232811 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f968410feb0 con 0x7f9684102e00 2024-03-30T04:52:59.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.767+0000 7f967be00700 1 -- 172.21.15.123:0/871232811 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f967401c5f0 con 0x7f9684102e00 2024-03-30T04:52:59.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.768+0000 7f968b000700 1 -- 172.21.15.123:0/871232811 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9684110470 con 0x7f9684102e00 2024-03-30T04:52:59.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.769+0000 7f968b000700 1 -- 172.21.15.123:0/871232811 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96840fe550 con 0x7f9684102e00 2024-03-30T04:52:59.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.769+0000 7f967be00700 1 -- 172.21.15.123:0/871232811 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f9674013590 con 0x7f9684102e00 2024-03-30T04:52:59.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.770+0000 7f967be00700 1 --2- 172.21.15.123:0/871232811 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f966806cc30 0x7f966806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:52:59.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.770+0000 7f967be00700 1 -- 172.21.15.123:0/871232811 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f967408d240 con 0x7f9684102e00 2024-03-30T04:52:59.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.771+0000 7f9683400700 1 --2- 172.21.15.123:0/871232811 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f966806cc30 0x7f966806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:52:59.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.771+0000 7f9683400700 1 --2- 172.21.15.123:0/871232811 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f966806cc30 0x7f966806f0e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f9684111570 tx=0x7f9670009ce0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:52:59.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.775+0000 7f967be00700 1 -- 172.21.15.123:0/871232811 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f967405ad10 con 0x7f9684102e00 2024-03-30T04:52:59.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.997+0000 7f968b000700 1 -- 172.21.15.123:0/871232811 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9684109e30 con 0x7f966806cc30 2024-03-30T04:52:59.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:52:59.999+0000 7f967be00700 1 -- 172.21.15.123:0/871232811 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f9684109e30 con 0x7f966806cc30 2024-03-30T04:53:00.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.004+0000 7f968b000700 1 -- 172.21.15.123:0/871232811 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f966806cc30 msgr2=0x7f966806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:53:00.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.004+0000 7f968b000700 1 --2- 172.21.15.123:0/871232811 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f966806cc30 0x7f966806f0e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f9684111570 tx=0x7f9670009ce0 comp rx=0 tx=0).stop 2024-03-30T04:53:00.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.004+0000 7f968b000700 1 -- 172.21.15.123:0/871232811 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9684102e00 msgr2=0x7f9684115cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:53:00.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.004+0000 7f968b000700 1 --2- 172.21.15.123:0/871232811 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9684102e00 0x7f9684115cd0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f9674009360 tx=0x7f9674009e30 comp rx=0 tx=0).stop 2024-03-30T04:53:00.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.004+0000 7f968b000700 1 -- 172.21.15.123:0/871232811 shutdown_connections 2024-03-30T04:53:00.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.004+0000 7f968b000700 1 --2- 172.21.15.123:0/871232811 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f966806cc30 0x7f966806f0e0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:00.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.004+0000 7f968b000700 1 --2- 172.21.15.123:0/871232811 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9684102e00 0x7f9684115cd0 secure :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f9674009360 tx=0x7f9674009e30 comp rx=0 tx=0).stop 2024-03-30T04:53:00.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.004+0000 7f968b000700 1 --2- 172.21.15.123:0/871232811 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f96841037a0 0x7f9684116210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:00.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.004+0000 7f968b000700 1 -- 172.21.15.123:0/871232811 >> 172.21.15.123:0/871232811 conn(0x7f96840f89b0 msgr2=0x7f96840f9980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:53:00.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.005+0000 7f968b000700 1 -- 172.21.15.123:0/871232811 shutdown_connections 2024-03-30T04:53:00.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.005+0000 7f968b000700 1 -- 172.21.15.123:0/871232811 wait complete. 2024-03-30T04:53:00.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:52:59 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:53:00.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.139+0000 7f0afae00700 1 -- 172.21.15.123:0/3851026407 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0af4102e00 msgr2=0x7f0af41031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:53:00.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.139+0000 7f0afae00700 1 --2- 172.21.15.123:0/3851026407 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0af4102e00 0x7f0af41031d0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f0ae4004940 tx=0x7f0ae4009e30 comp rx=0 tx=0).stop 2024-03-30T04:53:00.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.139+0000 7f0afae00700 1 -- 172.21.15.123:0/3851026407 shutdown_connections 2024-03-30T04:53:00.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.139+0000 7f0afae00700 1 --2- 172.21.15.123:0/3851026407 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0af41037a0 0x7f0af40fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:00.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.139+0000 7f0afae00700 1 --2- 172.21.15.123:0/3851026407 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0af4102e00 0x7f0af41031d0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:00.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.139+0000 7f0afae00700 1 -- 172.21.15.123:0/3851026407 >> 172.21.15.123:0/3851026407 conn(0x7f0af40f89b0 msgr2=0x7f0af40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:53:00.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.140+0000 7f0afae00700 1 -- 172.21.15.123:0/3851026407 shutdown_connections 2024-03-30T04:53:00.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.140+0000 7f0afae00700 1 -- 172.21.15.123:0/3851026407 wait complete. 2024-03-30T04:53:00.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.140+0000 7f0afae00700 1 Processor -- start 2024-03-30T04:53:00.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.141+0000 7f0afae00700 1 -- start start 2024-03-30T04:53:00.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.141+0000 7f0afae00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0af4102e00 0x7f0af4115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:53:00.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.141+0000 7f0afae00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0af41037a0 0x7f0af4116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:53:00.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.141+0000 7f0afae00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0af4116880 con 0x7f0af4102e00 2024-03-30T04:53:00.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.141+0000 7f0afae00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0af410fc50 con 0x7f0af41037a0 2024-03-30T04:53:00.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.141+0000 7f0af3e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0af4102e00 0x7f0af4115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:53:00.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.141+0000 7f0af3e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0af4102e00 0x7f0af4115cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49588/0 (socket says 172.21.15.123:49588) 2024-03-30T04:53:00.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.141+0000 7f0af3e00700 1 -- 172.21.15.123:0/1666447740 learned_addr learned my addr 172.21.15.123:0/1666447740 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:53:00.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.141+0000 7f0af3400700 1 --2- 172.21.15.123:0/1666447740 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0af41037a0 0x7f0af4116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:53:00.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.142+0000 7f0af3e00700 1 -- 172.21.15.123:0/1666447740 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0af41037a0 msgr2=0x7f0af4116230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:53:00.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.142+0000 7f0af3e00700 1 --2- 172.21.15.123:0/1666447740 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0af41037a0 0x7f0af4116230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:00.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.142+0000 7f0af3e00700 1 -- 172.21.15.123:0/1666447740 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ae4005040 con 0x7f0af4102e00 2024-03-30T04:53:00.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.142+0000 7f0af3e00700 1 --2- 172.21.15.123:0/1666447740 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0af4102e00 0x7f0af4115cf0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f0ae4009390 tx=0x7f0ae4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:53:00.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.142+0000 7f0aebe00700 1 -- 172.21.15.123:0/1666447740 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ae400e040 con 0x7f0af4102e00 2024-03-30T04:53:00.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.142+0000 7f0aebe00700 1 -- 172.21.15.123:0/1666447740 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0ae4013420 con 0x7f0af4102e00 2024-03-30T04:53:00.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.142+0000 7f0afae00700 1 -- 172.21.15.123:0/1666447740 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0af410fed0 con 0x7f0af4102e00 2024-03-30T04:53:00.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.142+0000 7f0aebe00700 1 -- 172.21.15.123:0/1666447740 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ae401c630 con 0x7f0af4102e00 2024-03-30T04:53:00.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.142+0000 7f0afae00700 1 -- 172.21.15.123:0/1666447740 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0af4110490 con 0x7f0af4102e00 2024-03-30T04:53:00.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.143+0000 7f0afae00700 1 -- 172.21.15.123:0/1666447740 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0af40fe550 con 0x7f0af4102e00 2024-03-30T04:53:00.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.144+0000 7f0aebe00700 1 -- 172.21.15.123:0/1666447740 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f0ae40139e0 con 0x7f0af4102e00 2024-03-30T04:53:00.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.145+0000 7f0aebe00700 1 --2- 172.21.15.123:0/1666447740 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0ad806cc30 0x7f0ad806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:53:00.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.145+0000 7f0aebe00700 1 -- 172.21.15.123:0/1666447740 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f0ae408d320 con 0x7f0af4102e00 2024-03-30T04:53:00.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.145+0000 7f0af3400700 1 --2- 172.21.15.123:0/1666447740 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0ad806cc30 0x7f0ad806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:53:00.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.146+0000 7f0af3400700 1 --2- 172.21.15.123:0/1666447740 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0ad806cc30 0x7f0ad806f0e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f0af4111590 tx=0x7f0ae0009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:53:00.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.149+0000 7f0aebe00700 1 -- 172.21.15.123:0/1666447740 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0ae405ae20 con 0x7f0af4102e00 2024-03-30T04:53:00.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:52:59 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:53:00.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.363+0000 7f0afae00700 1 -- 172.21.15.123:0/1666447740 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f0af404bed0 con 0x7f0ad806cc30 2024-03-30T04:53:00.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.373+0000 7f0aebe00700 1 -- 172.21.15.123:0/1666447740 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f0af404bed0 con 0x7f0ad806cc30 2024-03-30T04:53:00.371 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:53:00.371 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (22m) 15m ago 34m 15.0M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:53:00.371 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (19m) 15m ago 34m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T04:53:00.371 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (19m) 14m ago 33m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T04:53:00.372 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (21m) 15m ago 34m 74.0M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T04:53:00.372 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (15m) 15m ago 28m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T04:53:00.372 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (15m) 15m ago 28m 10.9M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T04:53:00.372 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (28m) 14m ago 28m 10.7M - 17.2.7-797-g5c024335 f77085f6aab7 0dc41326818c 2024-03-30T04:53:00.372 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (14m) 14m ago 28m 9747k - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T04:53:00.372 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (24m) 15m ago 36m 506M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:53:00.372 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (23m) 14m ago 33m 450M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:53:00.372 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (20m) 15m ago 36m 53.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T04:53:00.372 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (20m) 14m ago 33m 42.6M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T04:53:00.372 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (22m) 15m ago 34m 8052k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:53:00.372 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (22m) 14m ago 33m 7880k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:53:00.372 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (18m) 15m ago 32m 104M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T04:53:00.373 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (18m) 15m ago 32m 93.4M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T04:53:00.373 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (17m) 15m ago 31m 86.3M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T04:53:00.373 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (17m) 14m ago 31m 97.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T04:53:00.373 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (16m) 14m ago 30m 97.9M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T04:53:00.373 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (15m) 14m ago 30m 85.3M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T04:53:00.373 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (22m) 15m ago 34m 53.7M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:53:00.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.375+0000 7f0afae00700 1 -- 172.21.15.123:0/1666447740 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0ad806cc30 msgr2=0x7f0ad806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:53:00.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.375+0000 7f0afae00700 1 --2- 172.21.15.123:0/1666447740 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0ad806cc30 0x7f0ad806f0e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f0af4111590 tx=0x7f0ae0009ee0 comp rx=0 tx=0).stop 2024-03-30T04:53:00.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.375+0000 7f0afae00700 1 -- 172.21.15.123:0/1666447740 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0af4102e00 msgr2=0x7f0af4115cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:53:00.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.375+0000 7f0afae00700 1 --2- 172.21.15.123:0/1666447740 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0af4102e00 0x7f0af4115cf0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f0ae4009390 tx=0x7f0ae4009e30 comp rx=0 tx=0).stop 2024-03-30T04:53:00.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.375+0000 7f0afae00700 1 -- 172.21.15.123:0/1666447740 shutdown_connections 2024-03-30T04:53:00.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.375+0000 7f0afae00700 1 --2- 172.21.15.123:0/1666447740 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0ad806cc30 0x7f0ad806f0e0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:00.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.375+0000 7f0afae00700 1 --2- 172.21.15.123:0/1666447740 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0af4102e00 0x7f0af4115cf0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:00.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.375+0000 7f0afae00700 1 --2- 172.21.15.123:0/1666447740 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0af41037a0 0x7f0af4116230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:00.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.375+0000 7f0afae00700 1 -- 172.21.15.123:0/1666447740 >> 172.21.15.123:0/1666447740 conn(0x7f0af40f89b0 msgr2=0x7f0af40706d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:53:00.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.375+0000 7f0afae00700 1 -- 172.21.15.123:0/1666447740 shutdown_connections 2024-03-30T04:53:00.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.376+0000 7f0afae00700 1 -- 172.21.15.123:0/1666447740 wait complete. 2024-03-30T04:53:00.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.507+0000 7fce1c200700 1 -- 172.21.15.123:0/3092987242 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce14102e20 msgr2=0x7fce141031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:53:00.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.507+0000 7fce1c200700 1 --2- 172.21.15.123:0/3092987242 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce14102e20 0x7fce141031f0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fce08009ce0 tx=0x7fce08004d10 comp rx=0 tx=0).stop 2024-03-30T04:53:00.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.507+0000 7fce1c200700 1 -- 172.21.15.123:0/3092987242 shutdown_connections 2024-03-30T04:53:00.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.507+0000 7fce1c200700 1 --2- 172.21.15.123:0/3092987242 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fce141037c0 0x7fce140fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:00.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.507+0000 7fce1c200700 1 --2- 172.21.15.123:0/3092987242 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce14102e20 0x7fce141031f0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:00.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.507+0000 7fce1c200700 1 -- 172.21.15.123:0/3092987242 >> 172.21.15.123:0/3092987242 conn(0x7fce140f8970 msgr2=0x7fce140fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:53:00.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.507+0000 7fce1c200700 1 -- 172.21.15.123:0/3092987242 shutdown_connections 2024-03-30T04:53:00.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.507+0000 7fce1c200700 1 -- 172.21.15.123:0/3092987242 wait complete. 2024-03-30T04:53:00.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.508+0000 7fce1c200700 1 Processor -- start 2024-03-30T04:53:00.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.508+0000 7fce1c200700 1 -- start start 2024-03-30T04:53:00.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.509+0000 7fce1c200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fce14102e20 0x7fce14115ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:53:00.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.509+0000 7fce1c200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce141037c0 0x7fce141163e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:53:00.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.509+0000 7fce1c200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce14116ac0 con 0x7fce141037c0 2024-03-30T04:53:00.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.509+0000 7fce1c200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce1410fe00 con 0x7fce14102e20 2024-03-30T04:53:00.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.509+0000 7fce18e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce141037c0 0x7fce141163e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:53:00.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.509+0000 7fce18e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce141037c0 0x7fce141163e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49592/0 (socket says 172.21.15.123:49592) 2024-03-30T04:53:00.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.509+0000 7fce18e00700 1 -- 172.21.15.123:0/6239039 learned_addr learned my addr 172.21.15.123:0/6239039 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:53:00.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.509+0000 7fce19800700 1 --2- 172.21.15.123:0/6239039 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fce14102e20 0x7fce14115ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:53:00.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.509+0000 7fce18e00700 1 -- 172.21.15.123:0/6239039 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fce14102e20 msgr2=0x7fce14115ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:53:00.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.509+0000 7fce18e00700 1 --2- 172.21.15.123:0/6239039 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fce14102e20 0x7fce14115ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:00.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.509+0000 7fce18e00700 1 -- 172.21.15.123:0/6239039 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce08009940 con 0x7fce141037c0 2024-03-30T04:53:00.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.510+0000 7fce18e00700 1 --2- 172.21.15.123:0/6239039 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce141037c0 0x7fce141163e0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fce0000b790 tx=0x7fce00009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:53:00.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.510+0000 7fce11600700 1 -- 172.21.15.123:0/6239039 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce0000f6f0 con 0x7fce141037c0 2024-03-30T04:53:00.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.510+0000 7fce11600700 1 -- 172.21.15.123:0/6239039 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fce0000fd00 con 0x7fce141037c0 2024-03-30T04:53:00.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.510+0000 7fce1c200700 1 -- 172.21.15.123:0/6239039 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce14110080 con 0x7fce141037c0 2024-03-30T04:53:00.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.510+0000 7fce11600700 1 -- 172.21.15.123:0/6239039 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce000103a0 con 0x7fce141037c0 2024-03-30T04:53:00.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.510+0000 7fce1c200700 1 -- 172.21.15.123:0/6239039 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce14110640 con 0x7fce141037c0 2024-03-30T04:53:00.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.511+0000 7fce1c200700 1 -- 172.21.15.123:0/6239039 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce140fe570 con 0x7fce141037c0 2024-03-30T04:53:00.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.512+0000 7fce11600700 1 -- 172.21.15.123:0/6239039 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fce0000f850 con 0x7fce141037c0 2024-03-30T04:53:00.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.513+0000 7fce11600700 1 --2- 172.21.15.123:0/6239039 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fcdfc06cc30 0x7fcdfc06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:53:00.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.513+0000 7fce11600700 1 -- 172.21.15.123:0/6239039 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fce0008bf60 con 0x7fce141037c0 2024-03-30T04:53:00.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.514+0000 7fce19800700 1 --2- 172.21.15.123:0/6239039 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fcdfc06cc30 0x7fcdfc06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:53:00.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.514+0000 7fce19800700 1 --2- 172.21.15.123:0/6239039 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fcdfc06cc30 0x7fcdfc06f0e0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fce0800b740 tx=0x7fce08004d10 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:53:00.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.517+0000 7fce11600700 1 -- 172.21.15.123:0/6239039 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fce00059930 con 0x7fce141037c0 2024-03-30T04:53:00.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.804+0000 7fce1c200700 1 -- 172.21.15.123:0/6239039 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fce1404bed0 con 0x7fce141037c0 2024-03-30T04:53:00.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.805+0000 7fce11600700 1 -- 172.21.15.123:0/6239039 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7fce000594c0 con 0x7fce141037c0 2024-03-30T04:53:00.803 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:53:00.803 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:53:00.803 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:53:00.803 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:53:00.804 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:53:00.804 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:53:00.804 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:53:00.804 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:53:00.804 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T04:53:00.804 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:53:00.804 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:53:00.804 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:53:00.804 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 3 2024-03-30T04:53:00.804 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:53:00.804 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:53:00.804 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 17.2.7-797-g5c024335 (5c02433542ab1af3975314808d208cabd22abc39) quincy (stable)": 1, 2024-03-30T04:53:00.804 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 13 2024-03-30T04:53:00.804 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:53:00.804 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:53:00.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.807+0000 7fce1c200700 1 -- 172.21.15.123:0/6239039 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fcdfc06cc30 msgr2=0x7fcdfc06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:53:00.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.807+0000 7fce1c200700 1 --2- 172.21.15.123:0/6239039 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fcdfc06cc30 0x7fcdfc06f0e0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fce0800b740 tx=0x7fce08004d10 comp rx=0 tx=0).stop 2024-03-30T04:53:00.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.807+0000 7fce1c200700 1 -- 172.21.15.123:0/6239039 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce141037c0 msgr2=0x7fce141163e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:53:00.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.807+0000 7fce1c200700 1 --2- 172.21.15.123:0/6239039 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce141037c0 0x7fce141163e0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fce0000b790 tx=0x7fce00009220 comp rx=0 tx=0).stop 2024-03-30T04:53:00.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.807+0000 7fce1c200700 1 -- 172.21.15.123:0/6239039 shutdown_connections 2024-03-30T04:53:00.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.807+0000 7fce1c200700 1 --2- 172.21.15.123:0/6239039 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fcdfc06cc30 0x7fcdfc06f0e0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:00.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.807+0000 7fce1c200700 1 --2- 172.21.15.123:0/6239039 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fce14102e20 0x7fce14115ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:00.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.807+0000 7fce1c200700 1 --2- 172.21.15.123:0/6239039 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce141037c0 0x7fce141163e0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:00.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.807+0000 7fce1c200700 1 -- 172.21.15.123:0/6239039 >> 172.21.15.123:0/6239039 conn(0x7fce140f8970 msgr2=0x7fce140f99a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:53:00.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.807+0000 7fce1c200700 1 -- 172.21.15.123:0/6239039 shutdown_connections 2024-03-30T04:53:00.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.807+0000 7fce1c200700 1 -- 172.21.15.123:0/6239039 wait complete. 2024-03-30T04:53:00.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.941+0000 7fac70200700 1 -- 172.21.15.123:0/1856848386 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac681037a0 msgr2=0x7fac680fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:53:00.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.941+0000 7fac70200700 1 --2- 172.21.15.123:0/1856848386 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac681037a0 0x7fac680fce10 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fac54004940 tx=0x7fac54009e30 comp rx=0 tx=0).stop 2024-03-30T04:53:00.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.941+0000 7fac70200700 1 -- 172.21.15.123:0/1856848386 shutdown_connections 2024-03-30T04:53:00.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.941+0000 7fac70200700 1 --2- 172.21.15.123:0/1856848386 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac681037a0 0x7fac680fce10 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:00.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.941+0000 7fac70200700 1 --2- 172.21.15.123:0/1856848386 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac68102e00 0x7fac681031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:00.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.941+0000 7fac70200700 1 -- 172.21.15.123:0/1856848386 >> 172.21.15.123:0/1856848386 conn(0x7fac680f89b0 msgr2=0x7fac680fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:53:00.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.941+0000 7fac70200700 1 -- 172.21.15.123:0/1856848386 shutdown_connections 2024-03-30T04:53:00.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.941+0000 7fac70200700 1 -- 172.21.15.123:0/1856848386 wait complete. 2024-03-30T04:53:00.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.942+0000 7fac70200700 1 Processor -- start 2024-03-30T04:53:00.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.942+0000 7fac70200700 1 -- start start 2024-03-30T04:53:00.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.942+0000 7fac70200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac68102e00 0x7fac68113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:53:00.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.942+0000 7fac70200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac681037a0 0x7fac68114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:53:00.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.942+0000 7fac70200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac68114700 con 0x7fac681037a0 2024-03-30T04:53:00.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.942+0000 7fac70200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac6810da60 con 0x7fac68102e00 2024-03-30T04:53:00.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.943+0000 7fac6ce00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac681037a0 0x7fac68114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:53:00.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.943+0000 7fac6ce00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac681037a0 0x7fac68114020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49608/0 (socket says 172.21.15.123:49608) 2024-03-30T04:53:00.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.943+0000 7fac6ce00700 1 -- 172.21.15.123:0/3381841925 learned_addr learned my addr 172.21.15.123:0/3381841925 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:53:00.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.943+0000 7fac6ce00700 1 -- 172.21.15.123:0/3381841925 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac68102e00 msgr2=0x7fac68113ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:53:00.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.943+0000 7fac6ce00700 1 --2- 172.21.15.123:0/3381841925 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac68102e00 0x7fac68113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:00.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.943+0000 7fac6ce00700 1 -- 172.21.15.123:0/3381841925 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac54005040 con 0x7fac681037a0 2024-03-30T04:53:00.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.943+0000 7fac6ce00700 1 --2- 172.21.15.123:0/3381841925 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac681037a0 0x7fac68114020 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fac54009390 tx=0x7fac54009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:53:00.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.944+0000 7fac65600700 1 -- 172.21.15.123:0/3381841925 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac5400e040 con 0x7fac681037a0 2024-03-30T04:53:00.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.944+0000 7fac65600700 1 -- 172.21.15.123:0/3381841925 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fac54013420 con 0x7fac681037a0 2024-03-30T04:53:00.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.944+0000 7fac70200700 1 -- 172.21.15.123:0/3381841925 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac6810dce0 con 0x7fac681037a0 2024-03-30T04:53:00.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.944+0000 7fac65600700 1 -- 172.21.15.123:0/3381841925 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac5401c630 con 0x7fac681037a0 2024-03-30T04:53:00.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.944+0000 7fac70200700 1 -- 172.21.15.123:0/3381841925 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac6810e2a0 con 0x7fac681037a0 2024-03-30T04:53:00.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.945+0000 7fac70200700 1 -- 172.21.15.123:0/3381841925 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac680fe550 con 0x7fac681037a0 2024-03-30T04:53:00.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.946+0000 7fac65600700 1 -- 172.21.15.123:0/3381841925 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fac540139e0 con 0x7fac681037a0 2024-03-30T04:53:00.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.946+0000 7fac65600700 1 --2- 172.21.15.123:0/3381841925 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fac5006cc30 0x7fac5006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:53:00.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.947+0000 7fac65600700 1 -- 172.21.15.123:0/3381841925 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fac5408d2a0 con 0x7fac681037a0 2024-03-30T04:53:00.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.947+0000 7fac6d800700 1 --2- 172.21.15.123:0/3381841925 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fac5006cc30 0x7fac5006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:53:00.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.947+0000 7fac6d800700 1 --2- 172.21.15.123:0/3381841925 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fac5006cc30 0x7fac5006f0e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fac5c006fd0 tx=0x7fac5c006db0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:53:00.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:00.951+0000 7fac65600700 1 -- 172.21.15.123:0/3381841925 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fac5405ad70 con 0x7fac681037a0 2024-03-30T04:53:01.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:00 smithi135 ceph-mon[106011]: from='client.35214 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:53:01.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:00 smithi135 ceph-mon[106011]: pgmap v654: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:01.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:00 smithi135 ceph-mon[106011]: from='client.35218 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:53:01.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.197+0000 7fac70200700 1 -- 172.21.15.123:0/3381841925 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fac6804bed0 con 0x7fac681037a0 2024-03-30T04:53:01.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.197+0000 7fac65600700 1 -- 172.21.15.123:0/3381841925 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1843 (secure 0 0 0) 0x7fac5405a900 con 0x7fac681037a0 2024-03-30T04:53:01.196 INFO:teuthology.orchestra.run.smithi123.stdout:e32 2024-03-30T04:53:01.196 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:53:01.196 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:53:01.196 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:53:01.196 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:53:01.196 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:53:01.196 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:53:01.196 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T04:53:01.196 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T04:53:01.197 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:53:01.197 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T04:53:01.197 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:53:01.197 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:53:01.197 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:53:01.197 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:53:01.197 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:53:01.197 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T04:53:01.197 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:53:01.197 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:53:01.197 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T04:53:01.197 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:53:01.198 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T04:53:01.198 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T04:53:01.198 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T04:53:01.198 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T04:53:01.198 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:53:01.198 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:53:01.198 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:53:01.198 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:53:01.198 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:53:01.198 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:53:01.198 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T04:53:01.198 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T04:53:01.199 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:53:01.199 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:53:01.199 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T04:53:01.199 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:53:01.199 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:53:01.199 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:53:01.199 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44315} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/1741352336,v1:172.21.15.135:6827/1741352336] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:53:01.199 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:53:01.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.200+0000 7fac70200700 1 -- 172.21.15.123:0/3381841925 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fac5006cc30 msgr2=0x7fac5006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:53:01.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.200+0000 7fac70200700 1 --2- 172.21.15.123:0/3381841925 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fac5006cc30 0x7fac5006f0e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fac5c006fd0 tx=0x7fac5c006db0 comp rx=0 tx=0).stop 2024-03-30T04:53:01.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.200+0000 7fac70200700 1 -- 172.21.15.123:0/3381841925 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac681037a0 msgr2=0x7fac68114020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:53:01.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.200+0000 7fac70200700 1 --2- 172.21.15.123:0/3381841925 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac681037a0 0x7fac68114020 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fac54009390 tx=0x7fac54009e30 comp rx=0 tx=0).stop 2024-03-30T04:53:01.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.200+0000 7fac70200700 1 -- 172.21.15.123:0/3381841925 shutdown_connections 2024-03-30T04:53:01.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.200+0000 7fac70200700 1 --2- 172.21.15.123:0/3381841925 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fac5006cc30 0x7fac5006f0e0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:01.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.200+0000 7fac70200700 1 --2- 172.21.15.123:0/3381841925 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac68102e00 0x7fac68113ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:01.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.200+0000 7fac70200700 1 --2- 172.21.15.123:0/3381841925 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac681037a0 0x7fac68114020 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:01.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.200+0000 7fac70200700 1 -- 172.21.15.123:0/3381841925 >> 172.21.15.123:0/3381841925 conn(0x7fac680f89b0 msgr2=0x7fac680f99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:53:01.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.200+0000 7fac70200700 1 -- 172.21.15.123:0/3381841925 shutdown_connections 2024-03-30T04:53:01.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.200+0000 7fac70200700 1 -- 172.21.15.123:0/3381841925 wait complete. 2024-03-30T04:53:01.201 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 32 2024-03-30T04:53:01.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:00 smithi123 ceph-mon[139369]: from='client.35214 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:53:01.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:00 smithi123 ceph-mon[139369]: pgmap v654: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:01.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:00 smithi123 ceph-mon[139369]: from='client.35218 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:53:01.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.334+0000 7f20df000700 1 -- 172.21.15.123:0/3038251245 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20d81037a0 msgr2=0x7f20d80fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:53:01.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.334+0000 7f20df000700 1 --2- 172.21.15.123:0/3038251245 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20d81037a0 0x7f20d80fce10 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f20c4004940 tx=0x7f20c4009e30 comp rx=0 tx=0).stop 2024-03-30T04:53:01.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.334+0000 7f20df000700 1 -- 172.21.15.123:0/3038251245 shutdown_connections 2024-03-30T04:53:01.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.334+0000 7f20df000700 1 --2- 172.21.15.123:0/3038251245 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20d81037a0 0x7f20d80fce10 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:01.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.334+0000 7f20df000700 1 --2- 172.21.15.123:0/3038251245 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f20d8102e00 0x7f20d81031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:01.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.334+0000 7f20df000700 1 -- 172.21.15.123:0/3038251245 >> 172.21.15.123:0/3038251245 conn(0x7f20d80f89b0 msgr2=0x7f20d80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:53:01.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.335+0000 7f20df000700 1 -- 172.21.15.123:0/3038251245 shutdown_connections 2024-03-30T04:53:01.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.335+0000 7f20df000700 1 -- 172.21.15.123:0/3038251245 wait complete. 2024-03-30T04:53:01.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.335+0000 7f20df000700 1 Processor -- start 2024-03-30T04:53:01.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.336+0000 7f20df000700 1 -- start start 2024-03-30T04:53:01.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.336+0000 7f20df000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f20d8102e00 0x7f20d80702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:53:01.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.336+0000 7f20df000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20d81037a0 0x7f20d8070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:53:01.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.336+0000 7f20df000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d8070d90 con 0x7f20d81037a0 2024-03-30T04:53:01.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.336+0000 7f20df000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d8070ed0 con 0x7f20d8102e00 2024-03-30T04:53:01.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.336+0000 7f20d7400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20d81037a0 0x7f20d8070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:53:01.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.336+0000 7f20d7400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20d81037a0 0x7f20d8070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49640/0 (socket says 172.21.15.123:49640) 2024-03-30T04:53:01.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.336+0000 7f20d7400700 1 -- 172.21.15.123:0/1713693421 learned_addr learned my addr 172.21.15.123:0/1713693421 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:53:01.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.336+0000 7f20d7e00700 1 --2- 172.21.15.123:0/1713693421 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f20d8102e00 0x7f20d80702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:53:01.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.337+0000 7f20d7400700 1 -- 172.21.15.123:0/1713693421 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f20d8102e00 msgr2=0x7f20d80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:53:01.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.337+0000 7f20d7400700 1 --2- 172.21.15.123:0/1713693421 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f20d8102e00 0x7f20d80702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:01.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.337+0000 7f20d7400700 1 -- 172.21.15.123:0/1713693421 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20c4005040 con 0x7f20d81037a0 2024-03-30T04:53:01.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.337+0000 7f20d7400700 1 --2- 172.21.15.123:0/1713693421 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20d81037a0 0x7f20d8070800 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f20c4009390 tx=0x7f20c4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:53:01.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.337+0000 7f20cfe00700 1 -- 172.21.15.123:0/1713693421 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f20c400e040 con 0x7f20d81037a0 2024-03-30T04:53:01.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.337+0000 7f20cfe00700 1 -- 172.21.15.123:0/1713693421 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f20c4013420 con 0x7f20d81037a0 2024-03-30T04:53:01.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.337+0000 7f20df000700 1 -- 172.21.15.123:0/1713693421 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20d8071120 con 0x7f20d81037a0 2024-03-30T04:53:01.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.337+0000 7f20cfe00700 1 -- 172.21.15.123:0/1713693421 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f20c401c630 con 0x7f20d81037a0 2024-03-30T04:53:01.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.337+0000 7f20df000700 1 -- 172.21.15.123:0/1713693421 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20d811a250 con 0x7f20d81037a0 2024-03-30T04:53:01.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.338+0000 7f20df000700 1 -- 172.21.15.123:0/1713693421 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20d80fe550 con 0x7f20d81037a0 2024-03-30T04:53:01.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.339+0000 7f20cfe00700 1 -- 172.21.15.123:0/1713693421 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f20c40139e0 con 0x7f20d81037a0 2024-03-30T04:53:01.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.340+0000 7f20cfe00700 1 --2- 172.21.15.123:0/1713693421 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f20bc06cc30 0x7f20bc06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:53:01.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.340+0000 7f20cfe00700 1 -- 172.21.15.123:0/1713693421 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f20c408d2b0 con 0x7f20d81037a0 2024-03-30T04:53:01.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.340+0000 7f20d7e00700 1 --2- 172.21.15.123:0/1713693421 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f20bc06cc30 0x7f20bc06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:53:01.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.341+0000 7f20d7e00700 1 --2- 172.21.15.123:0/1713693421 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f20bc06cc30 0x7f20bc06f0e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f20c8009fd0 tx=0x7f20c8009f20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:53:01.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.344+0000 7f20cfe00700 1 -- 172.21.15.123:0/1713693421 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f20c40572f0 con 0x7f20d81037a0 2024-03-30T04:53:01.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.562+0000 7f20df000700 1 -- 172.21.15.123:0/1713693421 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f20d804bed0 con 0x7f20bc06cc30 2024-03-30T04:53:01.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.563+0000 7f20cfe00700 1 -- 172.21.15.123:0/1713693421 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f20d804bed0 con 0x7f20bc06cc30 2024-03-30T04:53:01.562 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:53:01.562 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:53:01.562 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:53:01.562 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T04:53:01.562 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:53:01.562 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T04:53:01.562 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T04:53:01.562 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T04:53:01.562 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T04:53:01.562 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:53:01.563 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "15/21 daemons upgraded", 2024-03-30T04:53:01.563 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T04:53:01.563 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:53:01.563 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:53:01.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.566+0000 7f20df000700 1 -- 172.21.15.123:0/1713693421 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f20bc06cc30 msgr2=0x7f20bc06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:53:01.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.566+0000 7f20df000700 1 --2- 172.21.15.123:0/1713693421 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f20bc06cc30 0x7f20bc06f0e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f20c8009fd0 tx=0x7f20c8009f20 comp rx=0 tx=0).stop 2024-03-30T04:53:01.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.566+0000 7f20df000700 1 -- 172.21.15.123:0/1713693421 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20d81037a0 msgr2=0x7f20d8070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:53:01.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.566+0000 7f20df000700 1 --2- 172.21.15.123:0/1713693421 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20d81037a0 0x7f20d8070800 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f20c4009390 tx=0x7f20c4009e30 comp rx=0 tx=0).stop 2024-03-30T04:53:01.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.566+0000 7f20df000700 1 -- 172.21.15.123:0/1713693421 shutdown_connections 2024-03-30T04:53:01.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.566+0000 7f20df000700 1 --2- 172.21.15.123:0/1713693421 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f20bc06cc30 0x7f20bc06f0e0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:01.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.566+0000 7f20df000700 1 --2- 172.21.15.123:0/1713693421 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f20d8102e00 0x7f20d80702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:01.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.566+0000 7f20df000700 1 --2- 172.21.15.123:0/1713693421 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20d81037a0 0x7f20d8070800 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:01.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.566+0000 7f20df000700 1 -- 172.21.15.123:0/1713693421 >> 172.21.15.123:0/1713693421 conn(0x7f20d80f89b0 msgr2=0x7f20d80f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:53:01.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.566+0000 7f20df000700 1 -- 172.21.15.123:0/1713693421 shutdown_connections 2024-03-30T04:53:01.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.566+0000 7f20df000700 1 -- 172.21.15.123:0/1713693421 wait complete. 2024-03-30T04:53:01.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.700+0000 7f0010400700 1 -- 172.21.15.123:0/733034436 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0008070300 msgr2=0x7f00080706d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:53:01.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.700+0000 7f0010400700 1 --2- 172.21.15.123:0/733034436 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0008070300 0x7f00080706d0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7efffc004970 tx=0x7efffc009e30 comp rx=0 tx=0).stop 2024-03-30T04:53:01.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.700+0000 7f0010400700 1 -- 172.21.15.123:0/733034436 shutdown_connections 2024-03-30T04:53:01.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.700+0000 7f0010400700 1 --2- 172.21.15.123:0/733034436 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0008070c10 0x7f0008109190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:01.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.700+0000 7f0010400700 1 --2- 172.21.15.123:0/733034436 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0008070300 0x7f00080706d0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:01.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.700+0000 7f0010400700 1 -- 172.21.15.123:0/733034436 >> 172.21.15.123:0/733034436 conn(0x7f00080f89d0 msgr2=0x7f00080fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:53:01.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.700+0000 7f0010400700 1 -- 172.21.15.123:0/733034436 shutdown_connections 2024-03-30T04:53:01.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.700+0000 7f0010400700 1 -- 172.21.15.123:0/733034436 wait complete. 2024-03-30T04:53:01.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.701+0000 7f0010400700 1 Processor -- start 2024-03-30T04:53:01.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.701+0000 7f0010400700 1 -- start start 2024-03-30T04:53:01.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.702+0000 7f0010400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0008070300 0x7f0008115c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:53:01.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.702+0000 7f0010400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0008070c10 0x7f00081161d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:53:01.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.702+0000 7f0010400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0008116820 con 0x7f0008070c10 2024-03-30T04:53:01.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.702+0000 7f0010400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f000810fbf0 con 0x7f0008070300 2024-03-30T04:53:01.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.702+0000 7f000d000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0008070c10 0x7f00081161d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:53:01.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.702+0000 7f000d000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0008070c10 0x7f00081161d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49656/0 (socket says 172.21.15.123:49656) 2024-03-30T04:53:01.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.702+0000 7f000d000700 1 -- 172.21.15.123:0/796641872 learned_addr learned my addr 172.21.15.123:0/796641872 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:53:01.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.702+0000 7f000da00700 1 --2- 172.21.15.123:0/796641872 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0008070300 0x7f0008115c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:53:01.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.703+0000 7f000d000700 1 -- 172.21.15.123:0/796641872 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0008070300 msgr2=0x7f0008115c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:53:01.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.703+0000 7f000d000700 1 --2- 172.21.15.123:0/796641872 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0008070300 0x7f0008115c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:01.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.703+0000 7f000d000700 1 -- 172.21.15.123:0/796641872 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efffc005040 con 0x7f0008070c10 2024-03-30T04:53:01.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.703+0000 7f000d000700 1 --2- 172.21.15.123:0/796641872 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0008070c10 0x7f00081161d0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7efff4002a50 tx=0x7efff40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:53:01.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.703+0000 7f0005600700 1 -- 172.21.15.123:0/796641872 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efff4007cc0 con 0x7f0008070c10 2024-03-30T04:53:01.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.703+0000 7f0005600700 1 -- 172.21.15.123:0/796641872 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efff400f040 con 0x7f0008070c10 2024-03-30T04:53:01.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.703+0000 7f0010400700 1 -- 172.21.15.123:0/796641872 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f000810fed0 con 0x7f0008070c10 2024-03-30T04:53:01.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.703+0000 7f0005600700 1 -- 172.21.15.123:0/796641872 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efff400d9c0 con 0x7f0008070c10 2024-03-30T04:53:01.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.703+0000 7f0010400700 1 -- 172.21.15.123:0/796641872 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0008110410 con 0x7f0008070c10 2024-03-30T04:53:01.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.704+0000 7f0010400700 1 -- 172.21.15.123:0/796641872 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0008071a40 con 0x7f0008070c10 2024-03-30T04:53:01.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.705+0000 7f0005600700 1 -- 172.21.15.123:0/796641872 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7efff400db20 con 0x7f0008070c10 2024-03-30T04:53:01.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.707+0000 7f0005600700 1 --2- 172.21.15.123:0/796641872 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7efff006cc30 0x7efff006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:53:01.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.707+0000 7f0005600700 1 -- 172.21.15.123:0/796641872 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7efff408bde0 con 0x7f0008070c10 2024-03-30T04:53:01.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.707+0000 7f000da00700 1 --2- 172.21.15.123:0/796641872 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7efff006cc30 0x7efff006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:53:01.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.707+0000 7f000da00700 1 --2- 172.21.15.123:0/796641872 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7efff006cc30 0x7efff006f0e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7efffc00b370 tx=0x7efffc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:53:01.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.711+0000 7f0005600700 1 -- 172.21.15.123:0/796641872 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7efff40597b0 con 0x7f0008070c10 2024-03-30T04:53:01.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.997+0000 7f0010400700 1 -- 172.21.15.123:0/796641872 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f000805f5f0 con 0x7f0008070c10 2024-03-30T04:53:01.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.997+0000 7f0005600700 1 -- 172.21.15.123:0/796641872 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7efff4059340 con 0x7f0008070c10 2024-03-30T04:53:01.996 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T04:53:01.996 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T04:53:01.996 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T04:53:01.996 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T04:53:01.996 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T04:53:01.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.999+0000 7f0010400700 1 -- 172.21.15.123:0/796641872 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7efff006cc30 msgr2=0x7efff006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:53:01.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.999+0000 7f0010400700 1 --2- 172.21.15.123:0/796641872 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7efff006cc30 0x7efff006f0e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7efffc00b370 tx=0x7efffc009e30 comp rx=0 tx=0).stop 2024-03-30T04:53:01.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.999+0000 7f0010400700 1 -- 172.21.15.123:0/796641872 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0008070c10 msgr2=0x7f00081161d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:53:01.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.999+0000 7f0010400700 1 --2- 172.21.15.123:0/796641872 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0008070c10 0x7f00081161d0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7efff4002a50 tx=0x7efff40046b0 comp rx=0 tx=0).stop 2024-03-30T04:53:01.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.999+0000 7f0010400700 1 -- 172.21.15.123:0/796641872 shutdown_connections 2024-03-30T04:53:01.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.999+0000 7f0010400700 1 --2- 172.21.15.123:0/796641872 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7efff006cc30 0x7efff006f0e0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:01.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.999+0000 7f0010400700 1 --2- 172.21.15.123:0/796641872 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0008070300 0x7f0008115c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:01.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.999+0000 7f0010400700 1 --2- 172.21.15.123:0/796641872 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0008070c10 0x7f00081161d0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:01.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:01.999+0000 7f0010400700 1 -- 172.21.15.123:0/796641872 >> 172.21.15.123:0/796641872 conn(0x7f00080f89d0 msgr2=0x7f00080f9fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:53:01.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:02.000+0000 7f0010400700 1 -- 172.21.15.123:0/796641872 shutdown_connections 2024-03-30T04:53:01.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:02.000+0000 7f0010400700 1 -- 172.21.15.123:0/796641872 wait complete. 2024-03-30T04:53:02.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:01 smithi135 ceph-mon[106011]: from='client.35222 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:53:02.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:01 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/6239039' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:53:02.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:01 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3381841925' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:53:02.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:01 smithi123 ceph-mon[139369]: from='client.35222 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:53:02.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:01 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/6239039' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:53:02.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:01 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3381841925' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:53:03.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:02 smithi135 ceph-mon[106011]: from='client.35234 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:53:03.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:02 smithi135 ceph-mon[106011]: pgmap v655: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:03.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:02 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/796641872' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:53:03.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:02 smithi123 ceph-mon[139369]: from='client.35234 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:53:03.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:02 smithi123 ceph-mon[139369]: pgmap v655: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:03.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:02 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/796641872' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:53:05.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:04 smithi135 ceph-mon[106011]: pgmap v656: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:05.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:04 smithi123 ceph-mon[139369]: pgmap v656: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:07.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:06 smithi135 ceph-mon[106011]: pgmap v657: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:07.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:06 smithi123 ceph-mon[139369]: pgmap v657: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:09.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:08 smithi135 ceph-mon[106011]: pgmap v658: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:09.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:08 smithi123 ceph-mon[139369]: pgmap v658: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:10.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:10 smithi123 ceph-mon[139369]: pgmap v659: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:11.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:10 smithi135 ceph-mon[106011]: pgmap v659: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:11.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:11 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:53:11.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:11 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:53:11.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:11 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:53:12.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:11 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:53:12.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:11 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:53:12.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:11 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:53:12.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:12 smithi123 ceph-mon[139369]: pgmap v660: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:12.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:12 smithi123 ceph-mon[139369]: Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED) 2024-03-30T04:53:12.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:12 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:53:12.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:12 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:53:12.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:12 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:53:12.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:12 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:53:12.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:12 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:53:12.973 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:12 smithi135 ceph-mon[106011]: pgmap v660: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:12.973 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:12 smithi135 ceph-mon[106011]: Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED) 2024-03-30T04:53:12.973 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:12 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:53:12.973 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:12 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:53:12.973 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:12 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:53:12.973 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:12 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:53:12.973 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:12 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:53:14.806 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:14 smithi135 ceph-mon[106011]: pgmap v661: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:14.806 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:14 smithi135 ceph-mon[106011]: Upgrade: Updating mds.cephfs.smithi135.xwgjvb 2024-03-30T04:53:14.806 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:14 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:53:14.806 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:14 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi135.xwgjvb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T04:53:14.806 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:14 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:53:14.807 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:14 smithi135 ceph-mon[106011]: Deploying daemon mds.cephfs.smithi135.xwgjvb on smithi135 2024-03-30T04:53:14.807 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:14 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:53:14.807 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:14 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:53:14.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:14 smithi123 ceph-mon[139369]: pgmap v661: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:14.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:14 smithi123 ceph-mon[139369]: Upgrade: Updating mds.cephfs.smithi135.xwgjvb 2024-03-30T04:53:14.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:14 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:53:14.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:14 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi135.xwgjvb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T04:53:14.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:14 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:53:14.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:14 smithi123 ceph-mon[139369]: Deploying daemon mds.cephfs.smithi135.xwgjvb on smithi135 2024-03-30T04:53:14.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:14 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:53:14.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:14 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:53:17.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:16 smithi123 ceph-mon[139369]: pgmap v662: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:17.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:16 smithi123 ceph-mon[139369]: osdmap e86: 6 total, 6 up, 6 in 2024-03-30T04:53:17.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:16 smithi123 ceph-mon[139369]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-03-30T04:53:17.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:16 smithi135 ceph-mon[106011]: pgmap v662: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:17.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:16 smithi135 ceph-mon[106011]: osdmap e86: 6 total, 6 up, 6 in 2024-03-30T04:53:17.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:16 smithi135 ceph-mon[106011]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-03-30T04:53:19.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:18 smithi123 ceph-mon[139369]: pgmap v664: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:19.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:18 smithi135 ceph-mon[106011]: pgmap v664: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:21.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:20 smithi123 ceph-mon[139369]: pgmap v665: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:21.255 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:20 smithi135 ceph-mon[106011]: pgmap v665: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:22.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:22 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:53:22.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:22 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:53:22.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:22 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:53:22.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:22 smithi123 ceph-mon[139369]: pgmap v666: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:22.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:22 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:53:22.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:22 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:53:22.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:22 smithi123 ceph-mon[139369]: mds.? [v2:172.21.15.135:6826/3788988014,v1:172.21.15.135:6827/3788988014] up:boot 2024-03-30T04:53:22.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:22 smithi123 ceph-mon[139369]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-03-30T04:53:22.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:22 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:53:22.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:22 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi135.xwgjvb"}]: dispatch 2024-03-30T04:53:22.717 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:22 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:53:22.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:22 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:53:22.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:22 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:53:22.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:22 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:53:22.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:22 smithi135 ceph-mon[106011]: pgmap v666: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:22.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:22 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:53:22.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:22 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:53:22.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:22 smithi135 ceph-mon[106011]: mds.? [v2:172.21.15.135:6826/3788988014,v1:172.21.15.135:6827/3788988014] up:boot 2024-03-30T04:53:22.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:22 smithi135 ceph-mon[106011]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-03-30T04:53:22.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:22 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:53:22.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:22 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi135.xwgjvb"}]: dispatch 2024-03-30T04:53:22.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:22 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:53:24.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:24 smithi135 ceph-mon[106011]: pgmap v667: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:24.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:24 smithi123 ceph-mon[139369]: pgmap v667: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:25.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:25 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:53:25.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:25 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:53:25.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:25 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:53:25.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:25 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:53:26.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:26 smithi135 ceph-mon[106011]: pgmap v668: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:26.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:26 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:53:26.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:26 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:53:27.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:26 smithi123 ceph-mon[139369]: pgmap v668: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:27.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:26 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:53:27.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:26 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:53:28.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:28 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:53:28.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:28 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:53:28.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:28 smithi123 ceph-mon[139369]: pgmap v669: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:28.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:28 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:53:28.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:28 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T04:53:28.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:28 smithi135 ceph-mon[106011]: pgmap v669: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:29.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:29 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:53:29.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:29 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:53:30.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:30 smithi123 ceph-mon[139369]: pgmap v670: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:31.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:30 smithi135 ceph-mon[106011]: pgmap v670: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:32.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.150+0000 7fe269a00700 1 -- 172.21.15.123:0/1589840507 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe264102d80 msgr2=0x7fe264103150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:53:32.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.150+0000 7fe269a00700 1 --2- 172.21.15.123:0/1589840507 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe264102d80 0x7fe264103150 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fe254004940 tx=0x7fe254009e30 comp rx=0 tx=0).stop 2024-03-30T04:53:32.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.150+0000 7fe269a00700 1 -- 172.21.15.123:0/1589840507 shutdown_connections 2024-03-30T04:53:32.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.150+0000 7fe269a00700 1 --2- 172.21.15.123:0/1589840507 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe264103720 0x7fe2640fcd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:32.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.150+0000 7fe269a00700 1 --2- 172.21.15.123:0/1589840507 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe264102d80 0x7fe264103150 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:32.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.150+0000 7fe269a00700 1 -- 172.21.15.123:0/1589840507 >> 172.21.15.123:0/1589840507 conn(0x7fe2640f8930 msgr2=0x7fe2640fad40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:53:32.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.150+0000 7fe269a00700 1 -- 172.21.15.123:0/1589840507 shutdown_connections 2024-03-30T04:53:32.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.150+0000 7fe269a00700 1 -- 172.21.15.123:0/1589840507 wait complete. 2024-03-30T04:53:32.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.151+0000 7fe269a00700 1 Processor -- start 2024-03-30T04:53:32.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.151+0000 7fe269a00700 1 -- start start 2024-03-30T04:53:32.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.151+0000 7fe269a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe264102d80 0x7fe264113a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:53:32.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.151+0000 7fe269a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe264103720 0x7fe264113fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:53:32.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.151+0000 7fe269a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2641145f0 con 0x7fe264102d80 2024-03-30T04:53:32.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.151+0000 7fe269a00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe26410d9e0 con 0x7fe264103720 2024-03-30T04:53:32.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.152+0000 7fe262a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe264102d80 0x7fe264113a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:53:32.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.152+0000 7fe262a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe264102d80 0x7fe264113a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50662/0 (socket says 172.21.15.123:50662) 2024-03-30T04:53:32.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.152+0000 7fe262a00700 1 -- 172.21.15.123:0/819503727 learned_addr learned my addr 172.21.15.123:0/819503727 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:53:32.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.152+0000 7fe262000700 1 --2- 172.21.15.123:0/819503727 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe264103720 0x7fe264113fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:53:32.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.152+0000 7fe262a00700 1 -- 172.21.15.123:0/819503727 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe264103720 msgr2=0x7fe264113fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:53:32.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.152+0000 7fe262a00700 1 --2- 172.21.15.123:0/819503727 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe264103720 0x7fe264113fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:32.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.152+0000 7fe262a00700 1 -- 172.21.15.123:0/819503727 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe254005040 con 0x7fe264102d80 2024-03-30T04:53:32.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.152+0000 7fe262000700 1 --2- 172.21.15.123:0/819503727 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe264103720 0x7fe264113fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:53:32.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.152+0000 7fe262a00700 1 --2- 172.21.15.123:0/819503727 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe264102d80 0x7fe264113a60 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fe254009390 tx=0x7fe254009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:53:32.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.153+0000 7fe25aa00700 1 -- 172.21.15.123:0/819503727 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe25400e040 con 0x7fe264102d80 2024-03-30T04:53:32.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.153+0000 7fe25aa00700 1 -- 172.21.15.123:0/819503727 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe254013420 con 0x7fe264102d80 2024-03-30T04:53:32.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.153+0000 7fe25aa00700 1 -- 172.21.15.123:0/819503727 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe25401c630 con 0x7fe264102d80 2024-03-30T04:53:32.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.153+0000 7fe269a00700 1 -- 172.21.15.123:0/819503727 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe26410dc60 con 0x7fe264102d80 2024-03-30T04:53:32.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.153+0000 7fe269a00700 1 -- 172.21.15.123:0/819503727 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe26410e220 con 0x7fe264102d80 2024-03-30T04:53:32.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.154+0000 7fe269a00700 1 -- 172.21.15.123:0/819503727 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2640fe4d0 con 0x7fe264102d80 2024-03-30T04:53:32.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.155+0000 7fe25aa00700 1 -- 172.21.15.123:0/819503727 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fe2540139e0 con 0x7fe264102d80 2024-03-30T04:53:32.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.156+0000 7fe25aa00700 1 --2- 172.21.15.123:0/819503727 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe24806cc30 0x7fe24806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:53:32.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.156+0000 7fe25aa00700 1 -- 172.21.15.123:0/819503727 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe25408d2f0 con 0x7fe264102d80 2024-03-30T04:53:32.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.157+0000 7fe262000700 1 --2- 172.21.15.123:0/819503727 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe24806cc30 0x7fe24806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:53:32.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.157+0000 7fe262000700 1 --2- 172.21.15.123:0/819503727 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe24806cc30 0x7fe24806f0e0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe250009fd0 tx=0x7fe250009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:53:32.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.161+0000 7fe25aa00700 1 -- 172.21.15.123:0/819503727 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe25405ae70 con 0x7fe264102d80 2024-03-30T04:53:32.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.381+0000 7fe269a00700 1 -- 172.21.15.123:0/819503727 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe264109d40 con 0x7fe24806cc30 2024-03-30T04:53:32.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.383+0000 7fe25aa00700 1 -- 172.21.15.123:0/819503727 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fe264109d40 con 0x7fe24806cc30 2024-03-30T04:53:32.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.385+0000 7fe269a00700 1 -- 172.21.15.123:0/819503727 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe24806cc30 msgr2=0x7fe24806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:53:32.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.385+0000 7fe269a00700 1 --2- 172.21.15.123:0/819503727 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe24806cc30 0x7fe24806f0e0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe250009fd0 tx=0x7fe250009ee0 comp rx=0 tx=0).stop 2024-03-30T04:53:32.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.385+0000 7fe269a00700 1 -- 172.21.15.123:0/819503727 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe264102d80 msgr2=0x7fe264113a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:53:32.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.385+0000 7fe269a00700 1 --2- 172.21.15.123:0/819503727 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe264102d80 0x7fe264113a60 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fe254009390 tx=0x7fe254009e30 comp rx=0 tx=0).stop 2024-03-30T04:53:32.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.385+0000 7fe269a00700 1 -- 172.21.15.123:0/819503727 shutdown_connections 2024-03-30T04:53:32.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.385+0000 7fe269a00700 1 --2- 172.21.15.123:0/819503727 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe24806cc30 0x7fe24806f0e0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:32.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.385+0000 7fe269a00700 1 --2- 172.21.15.123:0/819503727 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe264102d80 0x7fe264113a60 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:32.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.385+0000 7fe269a00700 1 --2- 172.21.15.123:0/819503727 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe264103720 0x7fe264113fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:32.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.385+0000 7fe269a00700 1 -- 172.21.15.123:0/819503727 >> 172.21.15.123:0/819503727 conn(0x7fe2640f8930 msgr2=0x7fe2640f99a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:53:32.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.385+0000 7fe269a00700 1 -- 172.21.15.123:0/819503727 shutdown_connections 2024-03-30T04:53:32.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.385+0000 7fe269a00700 1 -- 172.21.15.123:0/819503727 wait complete. 2024-03-30T04:53:32.398 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:53:32.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.519+0000 7fc3b7e00700 1 -- 172.21.15.123:0/2164491098 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3b81035e0 msgr2=0x7fc3b80fcc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:53:32.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.519+0000 7fc3b7e00700 1 --2- 172.21.15.123:0/2164491098 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3b81035e0 0x7fc3b80fcc50 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fc3a4004940 tx=0x7fc3a4009e30 comp rx=0 tx=0).stop 2024-03-30T04:53:32.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.519+0000 7fc3b7e00700 1 -- 172.21.15.123:0/2164491098 shutdown_connections 2024-03-30T04:53:32.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.519+0000 7fc3b7e00700 1 --2- 172.21.15.123:0/2164491098 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3b81035e0 0x7fc3b80fcc50 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:32.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.519+0000 7fc3b7e00700 1 --2- 172.21.15.123:0/2164491098 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc3b8102c40 0x7fc3b8103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:32.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.519+0000 7fc3b7e00700 1 -- 172.21.15.123:0/2164491098 >> 172.21.15.123:0/2164491098 conn(0x7fc3b80f8830 msgr2=0x7fc3b80fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:53:32.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.519+0000 7fc3b7e00700 1 -- 172.21.15.123:0/2164491098 shutdown_connections 2024-03-30T04:53:32.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.519+0000 7fc3b7e00700 1 -- 172.21.15.123:0/2164491098 wait complete. 2024-03-30T04:53:32.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.520+0000 7fc3b7e00700 1 Processor -- start 2024-03-30T04:53:32.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.520+0000 7fc3b7e00700 1 -- start start 2024-03-30T04:53:32.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.521+0000 7fc3b7e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc3b8102c40 0x7fc3b8115a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:53:32.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.521+0000 7fc3b7e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3b81035e0 0x7fc3b8115fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:53:32.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.521+0000 7fc3b7e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b8116610 con 0x7fc3b81035e0 2024-03-30T04:53:32.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.521+0000 7fc3b7e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b810fa00 con 0x7fc3b8102c40 2024-03-30T04:53:32.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.521+0000 7fc3b6000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3b81035e0 0x7fc3b8115fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:53:32.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.521+0000 7fc3b6000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3b81035e0 0x7fc3b8115fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50692/0 (socket says 172.21.15.123:50692) 2024-03-30T04:53:32.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.521+0000 7fc3b6000700 1 -- 172.21.15.123:0/4158640787 learned_addr learned my addr 172.21.15.123:0/4158640787 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:53:32.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.521+0000 7fc3b6000700 1 -- 172.21.15.123:0/4158640787 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc3b8102c40 msgr2=0x7fc3b8115a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:53:32.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.521+0000 7fc3b6000700 1 --2- 172.21.15.123:0/4158640787 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc3b8102c40 0x7fc3b8115a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:32.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.521+0000 7fc3b6000700 1 -- 172.21.15.123:0/4158640787 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a4005040 con 0x7fc3b81035e0 2024-03-30T04:53:32.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.522+0000 7fc3b6000700 1 --2- 172.21.15.123:0/4158640787 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3b81035e0 0x7fc3b8115fc0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fc3a4009fd0 tx=0x7fc3a4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:53:32.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.522+0000 7fc3aea00700 1 -- 172.21.15.123:0/4158640787 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc3a400e030 con 0x7fc3b81035e0 2024-03-30T04:53:32.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.522+0000 7fc3aea00700 1 -- 172.21.15.123:0/4158640787 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc3a4013e00 con 0x7fc3b81035e0 2024-03-30T04:53:32.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.522+0000 7fc3b7e00700 1 -- 172.21.15.123:0/4158640787 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3b810fc80 con 0x7fc3b81035e0 2024-03-30T04:53:32.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.522+0000 7fc3aea00700 1 -- 172.21.15.123:0/4158640787 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc3a401ccd0 con 0x7fc3b81035e0 2024-03-30T04:53:32.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.522+0000 7fc3b7e00700 1 -- 172.21.15.123:0/4158640787 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3b8110240 con 0x7fc3b81035e0 2024-03-30T04:53:32.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.523+0000 7fc3b7e00700 1 -- 172.21.15.123:0/4158640787 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3b80fe390 con 0x7fc3b81035e0 2024-03-30T04:53:32.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.524+0000 7fc3aea00700 1 -- 172.21.15.123:0/4158640787 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fc3a401c500 con 0x7fc3b81035e0 2024-03-30T04:53:32.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.525+0000 7fc3aea00700 1 --2- 172.21.15.123:0/4158640787 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc39c06cc30 0x7fc39c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:53:32.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.525+0000 7fc3aea00700 1 -- 172.21.15.123:0/4158640787 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc3a408d8b0 con 0x7fc3b81035e0 2024-03-30T04:53:32.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.525+0000 7fc3b6a00700 1 --2- 172.21.15.123:0/4158640787 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc39c06cc30 0x7fc39c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:53:32.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.526+0000 7fc3b6a00700 1 --2- 172.21.15.123:0/4158640787 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc39c06cc30 0x7fc39c06f0e0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc3a8009fd0 tx=0x7fc3a800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:53:32.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.529+0000 7fc3aea00700 1 -- 172.21.15.123:0/4158640787 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc3a405b380 con 0x7fc3b81035e0 2024-03-30T04:53:32.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.750+0000 7fc3b7e00700 1 -- 172.21.15.123:0/4158640787 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc3b8109bd0 con 0x7fc39c06cc30 2024-03-30T04:53:32.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.752+0000 7fc3aea00700 1 -- 172.21.15.123:0/4158640787 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fc3b8109bd0 con 0x7fc39c06cc30 2024-03-30T04:53:32.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.754+0000 7fc3b7e00700 1 -- 172.21.15.123:0/4158640787 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc39c06cc30 msgr2=0x7fc39c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:53:32.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.754+0000 7fc3b7e00700 1 --2- 172.21.15.123:0/4158640787 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc39c06cc30 0x7fc39c06f0e0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc3a8009fd0 tx=0x7fc3a800b040 comp rx=0 tx=0).stop 2024-03-30T04:53:32.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.754+0000 7fc3b7e00700 1 -- 172.21.15.123:0/4158640787 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3b81035e0 msgr2=0x7fc3b8115fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:53:32.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.754+0000 7fc3b7e00700 1 --2- 172.21.15.123:0/4158640787 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3b81035e0 0x7fc3b8115fc0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fc3a4009fd0 tx=0x7fc3a4009e30 comp rx=0 tx=0).stop 2024-03-30T04:53:32.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.754+0000 7fc3b7e00700 1 -- 172.21.15.123:0/4158640787 shutdown_connections 2024-03-30T04:53:32.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.754+0000 7fc3b7e00700 1 --2- 172.21.15.123:0/4158640787 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc39c06cc30 0x7fc39c06f0e0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:32.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.754+0000 7fc3b7e00700 1 --2- 172.21.15.123:0/4158640787 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc3b8102c40 0x7fc3b8115a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:32.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.754+0000 7fc3b7e00700 1 --2- 172.21.15.123:0/4158640787 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3b81035e0 0x7fc3b8115fc0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:32.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.754+0000 7fc3b7e00700 1 -- 172.21.15.123:0/4158640787 >> 172.21.15.123:0/4158640787 conn(0x7fc3b80f8830 msgr2=0x7fc3b80f9810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:53:32.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.754+0000 7fc3b7e00700 1 -- 172.21.15.123:0/4158640787 shutdown_connections 2024-03-30T04:53:32.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.754+0000 7fc3b7e00700 1 -- 172.21.15.123:0/4158640787 wait complete. 2024-03-30T04:53:32.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.888+0000 7f1a3b200700 1 -- 172.21.15.123:0/2051142698 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a34102e00 msgr2=0x7f1a341031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:53:32.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.888+0000 7f1a3b200700 1 --2- 172.21.15.123:0/2051142698 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a34102e00 0x7f1a341031d0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f1a24004940 tx=0x7f1a24009e30 comp rx=0 tx=0).stop 2024-03-30T04:53:32.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.888+0000 7f1a3b200700 1 -- 172.21.15.123:0/2051142698 shutdown_connections 2024-03-30T04:53:32.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.888+0000 7f1a3b200700 1 --2- 172.21.15.123:0/2051142698 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1a341037a0 0x7f1a340fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:32.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.888+0000 7f1a3b200700 1 --2- 172.21.15.123:0/2051142698 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a34102e00 0x7f1a341031d0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:32.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.888+0000 7f1a3b200700 1 -- 172.21.15.123:0/2051142698 >> 172.21.15.123:0/2051142698 conn(0x7f1a340f89b0 msgr2=0x7f1a340fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:53:32.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.888+0000 7f1a3b200700 1 -- 172.21.15.123:0/2051142698 shutdown_connections 2024-03-30T04:53:32.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.888+0000 7f1a3b200700 1 -- 172.21.15.123:0/2051142698 wait complete. 2024-03-30T04:53:32.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.889+0000 7f1a3b200700 1 Processor -- start 2024-03-30T04:53:32.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.889+0000 7f1a3b200700 1 -- start start 2024-03-30T04:53:32.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.889+0000 7f1a3b200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a34102e00 0x7f1a34115d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:53:32.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.889+0000 7f1a3b200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1a341037a0 0x7f1a34116250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:53:32.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.890+0000 7f1a3b200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a341168a0 con 0x7f1a34102e00 2024-03-30T04:53:32.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.890+0000 7f1a3b200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a3410fc70 con 0x7f1a341037a0 2024-03-30T04:53:32.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.890+0000 7f1a33e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a34102e00 0x7f1a34115d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:53:32.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.890+0000 7f1a33e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a34102e00 0x7f1a34115d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50696/0 (socket says 172.21.15.123:50696) 2024-03-30T04:53:32.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.890+0000 7f1a33e00700 1 -- 172.21.15.123:0/1886824648 learned_addr learned my addr 172.21.15.123:0/1886824648 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:53:32.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.890+0000 7f1a33400700 1 --2- 172.21.15.123:0/1886824648 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1a341037a0 0x7f1a34116250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:53:32.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.890+0000 7f1a33e00700 1 -- 172.21.15.123:0/1886824648 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1a341037a0 msgr2=0x7f1a34116250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:53:32.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.890+0000 7f1a33e00700 1 --2- 172.21.15.123:0/1886824648 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1a341037a0 0x7f1a34116250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:32.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.890+0000 7f1a33e00700 1 -- 172.21.15.123:0/1886824648 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a24005040 con 0x7f1a34102e00 2024-03-30T04:53:32.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.890+0000 7f1a33400700 1 --2- 172.21.15.123:0/1886824648 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1a341037a0 0x7f1a34116250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:53:32.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.890+0000 7f1a33e00700 1 --2- 172.21.15.123:0/1886824648 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a34102e00 0x7f1a34115d10 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f1a24009390 tx=0x7f1a24009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:53:32.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.891+0000 7f1a2be00700 1 -- 172.21.15.123:0/1886824648 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a2400e040 con 0x7f1a34102e00 2024-03-30T04:53:32.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.891+0000 7f1a2be00700 1 -- 172.21.15.123:0/1886824648 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1a24013420 con 0x7f1a34102e00 2024-03-30T04:53:32.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.891+0000 7f1a3b200700 1 -- 172.21.15.123:0/1886824648 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a3410fef0 con 0x7f1a34102e00 2024-03-30T04:53:32.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.891+0000 7f1a2be00700 1 -- 172.21.15.123:0/1886824648 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a2401c630 con 0x7f1a34102e00 2024-03-30T04:53:32.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.891+0000 7f1a3b200700 1 -- 172.21.15.123:0/1886824648 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a341104b0 con 0x7f1a34102e00 2024-03-30T04:53:32.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.892+0000 7f1a3b200700 1 -- 172.21.15.123:0/1886824648 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a340fe550 con 0x7f1a34102e00 2024-03-30T04:53:32.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.893+0000 7f1a2be00700 1 -- 172.21.15.123:0/1886824648 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f1a240139e0 con 0x7f1a34102e00 2024-03-30T04:53:32.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.894+0000 7f1a2be00700 1 --2- 172.21.15.123:0/1886824648 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1a1806cc30 0x7f1a1806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:53:32.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.894+0000 7f1a2be00700 1 -- 172.21.15.123:0/1886824648 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f1a2408d320 con 0x7f1a34102e00 2024-03-30T04:53:32.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.894+0000 7f1a33400700 1 --2- 172.21.15.123:0/1886824648 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1a1806cc30 0x7f1a1806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:53:32.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.894+0000 7f1a33400700 1 --2- 172.21.15.123:0/1886824648 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1a1806cc30 0x7f1a1806f0e0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f1a20009fd0 tx=0x7f1a2000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:53:32.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:32.898+0000 7f1a2be00700 1 -- 172.21.15.123:0/1886824648 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f1a2405ae20 con 0x7f1a34102e00 2024-03-30T04:53:32.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:32 smithi123 ceph-mon[139369]: pgmap v671: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:33.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:32 smithi135 ceph-mon[106011]: pgmap v671: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:33.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.115+0000 7f1a3b200700 1 -- 172.21.15.123:0/1886824648 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f1a3404bf40 con 0x7f1a1806cc30 2024-03-30T04:53:33.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.125+0000 7f1a2be00700 1 -- 172.21.15.123:0/1886824648 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f1a3404bf40 con 0x7f1a1806cc30 2024-03-30T04:53:33.123 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:53:33.124 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (22m) 5s ago 35m 14.6M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:53:33.124 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (19m) 5s ago 35m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T04:53:33.124 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (19m) 8s ago 34m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T04:53:33.124 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (22m) 5s ago 34m 73.5M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T04:53:33.124 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (15m) 5s ago 28m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T04:53:33.124 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (16m) 5s ago 28m 16.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T04:53:33.124 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (12s) 8s ago 28m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 87b073a6c9ca 2024-03-30T04:53:33.124 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (15m) 8s ago 28m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T04:53:33.124 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (24m) 5s ago 36m 510M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:53:33.124 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (24m) 8s ago 34m 456M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:53:33.124 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (21m) 5s ago 37m 70.0M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T04:53:33.124 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (20m) 8s ago 34m 51.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T04:53:33.124 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (23m) 5s ago 35m 8636k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:53:33.124 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (23m) 8s ago 34m 8678k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:53:33.124 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (19m) 5s ago 33m 107M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T04:53:33.125 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (19m) 5s ago 32m 97.2M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T04:53:33.125 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (18m) 5s ago 32m 89.8M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T04:53:33.125 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (17m) 8s ago 31m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T04:53:33.125 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (17m) 8s ago 31m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T04:53:33.125 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (16m) 8s ago 30m 88.8M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T04:53:33.125 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (23m) 5s ago 34m 63.9M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:53:33.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.127+0000 7f1a3b200700 1 -- 172.21.15.123:0/1886824648 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1a1806cc30 msgr2=0x7f1a1806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:53:33.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.127+0000 7f1a3b200700 1 --2- 172.21.15.123:0/1886824648 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1a1806cc30 0x7f1a1806f0e0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f1a20009fd0 tx=0x7f1a2000b040 comp rx=0 tx=0).stop 2024-03-30T04:53:33.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.127+0000 7f1a3b200700 1 -- 172.21.15.123:0/1886824648 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a34102e00 msgr2=0x7f1a34115d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:53:33.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.127+0000 7f1a3b200700 1 --2- 172.21.15.123:0/1886824648 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a34102e00 0x7f1a34115d10 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f1a24009390 tx=0x7f1a24009e30 comp rx=0 tx=0).stop 2024-03-30T04:53:33.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.128+0000 7f1a3b200700 1 -- 172.21.15.123:0/1886824648 shutdown_connections 2024-03-30T04:53:33.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.128+0000 7f1a3b200700 1 --2- 172.21.15.123:0/1886824648 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1a1806cc30 0x7f1a1806f0e0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:33.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.128+0000 7f1a3b200700 1 --2- 172.21.15.123:0/1886824648 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a34102e00 0x7f1a34115d10 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:33.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.128+0000 7f1a3b200700 1 --2- 172.21.15.123:0/1886824648 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1a341037a0 0x7f1a34116250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:33.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.128+0000 7f1a3b200700 1 -- 172.21.15.123:0/1886824648 >> 172.21.15.123:0/1886824648 conn(0x7f1a340f89b0 msgr2=0x7f1a340704f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:53:33.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.128+0000 7f1a3b200700 1 -- 172.21.15.123:0/1886824648 shutdown_connections 2024-03-30T04:53:33.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.128+0000 7f1a3b200700 1 -- 172.21.15.123:0/1886824648 wait complete. 2024-03-30T04:53:33.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.261+0000 7f6eb0a00700 1 -- 172.21.15.123:0/2707147950 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6eac102e00 msgr2=0x7f6eac1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:53:33.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.261+0000 7f6eb0a00700 1 --2- 172.21.15.123:0/2707147950 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6eac102e00 0x7f6eac1031d0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c004940 tx=0x7f6e9c009e30 comp rx=0 tx=0).stop 2024-03-30T04:53:33.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.262+0000 7f6eb0a00700 1 -- 172.21.15.123:0/2707147950 shutdown_connections 2024-03-30T04:53:33.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.262+0000 7f6eb0a00700 1 --2- 172.21.15.123:0/2707147950 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6eac1037a0 0x7f6eac0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:33.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.262+0000 7f6eb0a00700 1 --2- 172.21.15.123:0/2707147950 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6eac102e00 0x7f6eac1031d0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:33.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.262+0000 7f6eb0a00700 1 -- 172.21.15.123:0/2707147950 >> 172.21.15.123:0/2707147950 conn(0x7f6eac0f89b0 msgr2=0x7f6eac0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:53:33.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.262+0000 7f6eb0a00700 1 -- 172.21.15.123:0/2707147950 shutdown_connections 2024-03-30T04:53:33.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.262+0000 7f6eb0a00700 1 -- 172.21.15.123:0/2707147950 wait complete. 2024-03-30T04:53:33.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.262+0000 7f6eb0a00700 1 Processor -- start 2024-03-30T04:53:33.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.263+0000 7f6eb0a00700 1 -- start start 2024-03-30T04:53:33.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.263+0000 7f6eb0a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6eac102e00 0x7f6eac115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:53:33.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.263+0000 7f6eb0a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6eac1037a0 0x7f6eac116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:53:33.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.263+0000 7f6eb0a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eac116860 con 0x7f6eac102e00 2024-03-30T04:53:33.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.263+0000 7f6eb0a00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eac10fc30 con 0x7f6eac1037a0 2024-03-30T04:53:33.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.263+0000 7f6ea9e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6eac102e00 0x7f6eac115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:53:33.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.263+0000 7f6ea9e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6eac102e00 0x7f6eac115cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50720/0 (socket says 172.21.15.123:50720) 2024-03-30T04:53:33.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.263+0000 7f6ea9e00700 1 -- 172.21.15.123:0/3651015227 learned_addr learned my addr 172.21.15.123:0/3651015227 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:53:33.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.263+0000 7f6ea9400700 1 --2- 172.21.15.123:0/3651015227 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6eac1037a0 0x7f6eac116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:53:33.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.264+0000 7f6ea9e00700 1 -- 172.21.15.123:0/3651015227 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6eac1037a0 msgr2=0x7f6eac116210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:53:33.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.264+0000 7f6ea9e00700 1 --2- 172.21.15.123:0/3651015227 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6eac1037a0 0x7f6eac116210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:33.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.264+0000 7f6ea9e00700 1 -- 172.21.15.123:0/3651015227 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e9c005040 con 0x7f6eac102e00 2024-03-30T04:53:33.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.264+0000 7f6ea9e00700 1 --2- 172.21.15.123:0/3651015227 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6eac102e00 0x7f6eac115cd0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c009360 tx=0x7f6e9c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:53:33.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.264+0000 7f6ea2000700 1 -- 172.21.15.123:0/3651015227 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e9c00e040 con 0x7f6eac102e00 2024-03-30T04:53:33.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.264+0000 7f6ea2000700 1 -- 172.21.15.123:0/3651015227 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6e9c013420 con 0x7f6eac102e00 2024-03-30T04:53:33.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.264+0000 7f6ea2000700 1 -- 172.21.15.123:0/3651015227 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e9c01c5f0 con 0x7f6eac102e00 2024-03-30T04:53:33.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.264+0000 7f6eb0a00700 1 -- 172.21.15.123:0/3651015227 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6eac10feb0 con 0x7f6eac102e00 2024-03-30T04:53:33.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.264+0000 7f6eb0a00700 1 -- 172.21.15.123:0/3651015227 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6eac110470 con 0x7f6eac102e00 2024-03-30T04:53:33.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.265+0000 7f6eb0a00700 1 -- 172.21.15.123:0/3651015227 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6eac0fe550 con 0x7f6eac102e00 2024-03-30T04:53:33.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.266+0000 7f6ea2000700 1 -- 172.21.15.123:0/3651015227 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f6e9c013590 con 0x7f6eac102e00 2024-03-30T04:53:33.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.267+0000 7f6ea2000700 1 --2- 172.21.15.123:0/3651015227 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6e9006cc30 0x7f6e9006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:53:33.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.267+0000 7f6ea2000700 1 -- 172.21.15.123:0/3651015227 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f6e9c08d240 con 0x7f6eac102e00 2024-03-30T04:53:33.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.268+0000 7f6ea9400700 1 --2- 172.21.15.123:0/3651015227 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6e9006cc30 0x7f6e9006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:53:33.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.268+0000 7f6ea9400700 1 --2- 172.21.15.123:0/3651015227 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6e9006cc30 0x7f6e9006f0e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f6eac111570 tx=0x7f6e98009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:53:33.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.272+0000 7f6ea2000700 1 -- 172.21.15.123:0/3651015227 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6e9c05ad10 con 0x7f6eac102e00 2024-03-30T04:53:33.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.558+0000 7f6eb0a00700 1 -- 172.21.15.123:0/3651015227 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f6eac04bf00 con 0x7f6eac102e00 2024-03-30T04:53:33.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.558+0000 7f6ea2000700 1 -- 172.21.15.123:0/3651015227 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f6e9c05a8a0 con 0x7f6eac102e00 2024-03-30T04:53:33.557 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:53:33.557 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:53:33.557 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:53:33.557 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:53:33.557 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:53:33.557 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:53:33.557 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:53:33.557 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:53:33.557 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T04:53:33.557 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:53:33.557 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:53:33.557 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 4 2024-03-30T04:53:33.558 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:53:33.558 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:53:33.558 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 14 2024-03-30T04:53:33.558 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:53:33.558 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:53:33.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.560+0000 7f6eb0a00700 1 -- 172.21.15.123:0/3651015227 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6e9006cc30 msgr2=0x7f6e9006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:53:33.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.560+0000 7f6eb0a00700 1 --2- 172.21.15.123:0/3651015227 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6e9006cc30 0x7f6e9006f0e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f6eac111570 tx=0x7f6e98009ee0 comp rx=0 tx=0).stop 2024-03-30T04:53:33.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.560+0000 7f6eb0a00700 1 -- 172.21.15.123:0/3651015227 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6eac102e00 msgr2=0x7f6eac115cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:53:33.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.560+0000 7f6eb0a00700 1 --2- 172.21.15.123:0/3651015227 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6eac102e00 0x7f6eac115cd0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c009360 tx=0x7f6e9c009e30 comp rx=0 tx=0).stop 2024-03-30T04:53:33.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.561+0000 7f6eb0a00700 1 -- 172.21.15.123:0/3651015227 shutdown_connections 2024-03-30T04:53:33.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.561+0000 7f6eb0a00700 1 --2- 172.21.15.123:0/3651015227 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6e9006cc30 0x7f6e9006f0e0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:33.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.561+0000 7f6eb0a00700 1 --2- 172.21.15.123:0/3651015227 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6eac102e00 0x7f6eac115cd0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:33.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.561+0000 7f6eb0a00700 1 --2- 172.21.15.123:0/3651015227 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6eac1037a0 0x7f6eac116210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:33.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.561+0000 7f6eb0a00700 1 -- 172.21.15.123:0/3651015227 >> 172.21.15.123:0/3651015227 conn(0x7f6eac0f89b0 msgr2=0x7f6eac0f9980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:53:33.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.561+0000 7f6eb0a00700 1 -- 172.21.15.123:0/3651015227 shutdown_connections 2024-03-30T04:53:33.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.561+0000 7f6eb0a00700 1 -- 172.21.15.123:0/3651015227 wait complete. 2024-03-30T04:53:33.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.697+0000 7f494fe00700 1 -- 172.21.15.123:0/3454926188 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49481037a0 msgr2=0x7f49480fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:53:33.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.697+0000 7f494fe00700 1 --2- 172.21.15.123:0/3454926188 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49481037a0 0x7f49480fce10 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f4934004940 tx=0x7f4934009e30 comp rx=0 tx=0).stop 2024-03-30T04:53:33.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.697+0000 7f494fe00700 1 -- 172.21.15.123:0/3454926188 shutdown_connections 2024-03-30T04:53:33.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.697+0000 7f494fe00700 1 --2- 172.21.15.123:0/3454926188 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49481037a0 0x7f49480fce10 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:33.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.697+0000 7f494fe00700 1 --2- 172.21.15.123:0/3454926188 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4948102e00 0x7f49481031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:33.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.697+0000 7f494fe00700 1 -- 172.21.15.123:0/3454926188 >> 172.21.15.123:0/3454926188 conn(0x7f49480f89b0 msgr2=0x7f49480fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:53:33.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.697+0000 7f494fe00700 1 -- 172.21.15.123:0/3454926188 shutdown_connections 2024-03-30T04:53:33.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.697+0000 7f494fe00700 1 -- 172.21.15.123:0/3454926188 wait complete. 2024-03-30T04:53:33.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.698+0000 7f494fe00700 1 Processor -- start 2024-03-30T04:53:33.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.698+0000 7f494fe00700 1 -- start start 2024-03-30T04:53:33.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.698+0000 7f494fe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4948102e00 0x7f4948113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:53:33.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.698+0000 7f494fe00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49481037a0 0x7f4948114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:53:33.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.698+0000 7f494fe00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49481146b0 con 0x7f4948102e00 2024-03-30T04:53:33.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.698+0000 7f494fe00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f494810daa0 con 0x7f49481037a0 2024-03-30T04:53:33.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.699+0000 7f494d400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4948102e00 0x7f4948113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:53:33.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.699+0000 7f494d400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4948102e00 0x7f4948113b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50744/0 (socket says 172.21.15.123:50744) 2024-03-30T04:53:33.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.699+0000 7f494d400700 1 -- 172.21.15.123:0/2383101490 learned_addr learned my addr 172.21.15.123:0/2383101490 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:53:33.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.699+0000 7f494ca00700 1 --2- 172.21.15.123:0/2383101490 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49481037a0 0x7f4948114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:53:33.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.699+0000 7f494d400700 1 -- 172.21.15.123:0/2383101490 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49481037a0 msgr2=0x7f4948114060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:53:33.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.699+0000 7f494d400700 1 --2- 172.21.15.123:0/2383101490 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49481037a0 0x7f4948114060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:33.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.699+0000 7f494d400700 1 -- 172.21.15.123:0/2383101490 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4934005040 con 0x7f4948102e00 2024-03-30T04:53:33.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.699+0000 7f494d400700 1 --2- 172.21.15.123:0/2383101490 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4948102e00 0x7f4948113b20 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f493c002a50 tx=0x7f493c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:53:33.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.699+0000 7f4945600700 1 -- 172.21.15.123:0/2383101490 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f493c007cc0 con 0x7f4948102e00 2024-03-30T04:53:33.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.699+0000 7f4945600700 1 -- 172.21.15.123:0/2383101490 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f493c00f040 con 0x7f4948102e00 2024-03-30T04:53:33.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.699+0000 7f494fe00700 1 -- 172.21.15.123:0/2383101490 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f494810dd80 con 0x7f4948102e00 2024-03-30T04:53:33.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.699+0000 7f4945600700 1 -- 172.21.15.123:0/2383101490 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f493c00da70 con 0x7f4948102e00 2024-03-30T04:53:33.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.700+0000 7f494fe00700 1 -- 172.21.15.123:0/2383101490 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f494810e340 con 0x7f4948102e00 2024-03-30T04:53:33.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.701+0000 7f494fe00700 1 -- 172.21.15.123:0/2383101490 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49480fe550 con 0x7f4948102e00 2024-03-30T04:53:33.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.701+0000 7f4945600700 1 -- 172.21.15.123:0/2383101490 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f493c00dc10 con 0x7f4948102e00 2024-03-30T04:53:33.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.702+0000 7f4945600700 1 --2- 172.21.15.123:0/2383101490 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f493006cc30 0x7f493006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:53:33.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.702+0000 7f4945600700 1 -- 172.21.15.123:0/2383101490 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f493c08b810 con 0x7f4948102e00 2024-03-30T04:53:33.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.703+0000 7f494ca00700 1 --2- 172.21.15.123:0/2383101490 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f493006cc30 0x7f493006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:53:33.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.703+0000 7f494ca00700 1 --2- 172.21.15.123:0/2383101490 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f493006cc30 0x7f493006f0e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f494810ec60 tx=0x7f4934009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:53:33.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.707+0000 7f4945600700 1 -- 172.21.15.123:0/2383101490 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f493c0591e0 con 0x7f4948102e00 2024-03-30T04:53:33.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.952+0000 7f494fe00700 1 -- 172.21.15.123:0/2383101490 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f4948109dc0 con 0x7f4948102e00 2024-03-30T04:53:33.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.953+0000 7f4945600700 1 -- 172.21.15.123:0/2383101490 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 34 v34) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f493c058d70 con 0x7f4948102e00 2024-03-30T04:53:33.951 INFO:teuthology.orchestra.run.smithi123.stdout:e34 2024-03-30T04:53:33.951 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:53:33.951 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:53:33.951 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:53:33.952 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:53:33.952 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:53:33.952 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:53:33.952 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T04:53:33.952 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T04:53:33.952 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:53:33.952 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T04:53:33.952 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:53:33.952 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:53:33.952 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:53:33.952 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:53:33.952 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:53:33.952 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T04:53:33.952 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:53:33.953 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:53:33.953 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T04:53:33.953 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:53:33.953 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T04:53:33.953 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T04:53:33.953 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T04:53:33.953 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T04:53:33.953 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:53:33.953 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:53:33.953 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:53:33.953 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:53:33.953 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:53:33.953 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:53:33.953 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T04:53:33.953 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T04:53:33.954 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:53:33.954 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:53:33.954 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T04:53:33.954 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:53:33.954 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:53:33.954 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:53:33.954 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:53:33.954 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44975} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/3788988014,v1:172.21.15.135:6827/3788988014] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:53:33.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.955+0000 7f494fe00700 1 -- 172.21.15.123:0/2383101490 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f493006cc30 msgr2=0x7f493006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:53:33.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.955+0000 7f494fe00700 1 --2- 172.21.15.123:0/2383101490 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f493006cc30 0x7f493006f0e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f494810ec60 tx=0x7f4934009e30 comp rx=0 tx=0).stop 2024-03-30T04:53:33.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.955+0000 7f494fe00700 1 -- 172.21.15.123:0/2383101490 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4948102e00 msgr2=0x7f4948113b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:53:33.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.955+0000 7f494fe00700 1 --2- 172.21.15.123:0/2383101490 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4948102e00 0x7f4948113b20 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f493c002a50 tx=0x7f493c0046b0 comp rx=0 tx=0).stop 2024-03-30T04:53:33.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.955+0000 7f494fe00700 1 -- 172.21.15.123:0/2383101490 shutdown_connections 2024-03-30T04:53:33.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.955+0000 7f494fe00700 1 --2- 172.21.15.123:0/2383101490 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f493006cc30 0x7f493006f0e0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:33.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.955+0000 7f494fe00700 1 --2- 172.21.15.123:0/2383101490 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4948102e00 0x7f4948113b20 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:33.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.955+0000 7f494fe00700 1 --2- 172.21.15.123:0/2383101490 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49481037a0 0x7f4948114060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:33.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.955+0000 7f494fe00700 1 -- 172.21.15.123:0/2383101490 >> 172.21.15.123:0/2383101490 conn(0x7f49480f89b0 msgr2=0x7f49480f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:53:33.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.955+0000 7f494fe00700 1 -- 172.21.15.123:0/2383101490 shutdown_connections 2024-03-30T04:53:33.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:33.955+0000 7f494fe00700 1 -- 172.21.15.123:0/2383101490 wait complete. 2024-03-30T04:53:33.956 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 34 2024-03-30T04:53:33.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:33 smithi123 ceph-mon[139369]: from='client.35244 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:53:33.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:33 smithi123 ceph-mon[139369]: from='client.35248 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:53:33.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:33 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3651015227' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:53:34.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.091+0000 7fdd37000700 1 -- 172.21.15.123:0/1447120421 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd30103540 msgr2=0x7fdd300fcbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:53:34.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.091+0000 7fdd37000700 1 --2- 172.21.15.123:0/1447120421 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd30103540 0x7fdd300fcbb0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c004940 tx=0x7fdd1c009e30 comp rx=0 tx=0).stop 2024-03-30T04:53:34.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.091+0000 7fdd37000700 1 -- 172.21.15.123:0/1447120421 shutdown_connections 2024-03-30T04:53:34.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.091+0000 7fdd37000700 1 --2- 172.21.15.123:0/1447120421 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd30103540 0x7fdd300fcbb0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:34.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.091+0000 7fdd37000700 1 --2- 172.21.15.123:0/1447120421 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdd30102ba0 0x7fdd30102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:34.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.091+0000 7fdd37000700 1 -- 172.21.15.123:0/1447120421 >> 172.21.15.123:0/1447120421 conn(0x7fdd300f8750 msgr2=0x7fdd300fab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:53:34.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.091+0000 7fdd37000700 1 -- 172.21.15.123:0/1447120421 shutdown_connections 2024-03-30T04:53:34.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.091+0000 7fdd37000700 1 -- 172.21.15.123:0/1447120421 wait complete. 2024-03-30T04:53:34.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.092+0000 7fdd37000700 1 Processor -- start 2024-03-30T04:53:34.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.092+0000 7fdd37000700 1 -- start start 2024-03-30T04:53:34.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.093+0000 7fdd37000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdd30102ba0 0x7fdd30115a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:53:34.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.093+0000 7fdd37000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd30103540 0x7fdd30115fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:53:34.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.093+0000 7fdd37000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd30116600 con 0x7fdd30103540 2024-03-30T04:53:34.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.093+0000 7fdd37000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3010f9f0 con 0x7fdd30102ba0 2024-03-30T04:53:34.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.093+0000 7fdd35200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd30103540 0x7fdd30115fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:53:34.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.093+0000 7fdd35200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd30103540 0x7fdd30115fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35372/0 (socket says 172.21.15.123:35372) 2024-03-30T04:53:34.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.093+0000 7fdd35200700 1 -- 172.21.15.123:0/829449706 learned_addr learned my addr 172.21.15.123:0/829449706 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:53:34.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.093+0000 7fdd35c00700 1 --2- 172.21.15.123:0/829449706 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdd30102ba0 0x7fdd30115a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:53:34.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.093+0000 7fdd35200700 1 -- 172.21.15.123:0/829449706 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdd30102ba0 msgr2=0x7fdd30115a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:53:34.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.093+0000 7fdd35200700 1 --2- 172.21.15.123:0/829449706 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdd30102ba0 0x7fdd30115a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:34.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.094+0000 7fdd35200700 1 -- 172.21.15.123:0/829449706 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd1c005040 con 0x7fdd30103540 2024-03-30T04:53:34.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.094+0000 7fdd35200700 1 --2- 172.21.15.123:0/829449706 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd30103540 0x7fdd30115fb0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c009390 tx=0x7fdd1c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:53:34.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.094+0000 7fdd2d600700 1 -- 172.21.15.123:0/829449706 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd1c00e040 con 0x7fdd30103540 2024-03-30T04:53:34.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.094+0000 7fdd2d600700 1 -- 172.21.15.123:0/829449706 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdd1c013420 con 0x7fdd30103540 2024-03-30T04:53:34.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.094+0000 7fdd2d600700 1 -- 172.21.15.123:0/829449706 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd1c01c630 con 0x7fdd30103540 2024-03-30T04:53:34.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.094+0000 7fdd37000700 1 -- 172.21.15.123:0/829449706 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd3010fc70 con 0x7fdd30103540 2024-03-30T04:53:34.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.094+0000 7fdd37000700 1 -- 172.21.15.123:0/829449706 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd30110230 con 0x7fdd30103540 2024-03-30T04:53:34.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.095+0000 7fdd37000700 1 -- 172.21.15.123:0/829449706 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd300fe2f0 con 0x7fdd30103540 2024-03-30T04:53:34.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.096+0000 7fdd2d600700 1 -- 172.21.15.123:0/829449706 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fdd1c013a00 con 0x7fdd30103540 2024-03-30T04:53:34.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.097+0000 7fdd2d600700 1 --2- 172.21.15.123:0/829449706 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fdd1806cc30 0x7fdd1806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:53:34.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.097+0000 7fdd2d600700 1 -- 172.21.15.123:0/829449706 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fdd1c08d120 con 0x7fdd30103540 2024-03-30T04:53:34.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.098+0000 7fdd35c00700 1 --2- 172.21.15.123:0/829449706 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fdd1806cc30 0x7fdd1806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:53:34.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.098+0000 7fdd35c00700 1 --2- 172.21.15.123:0/829449706 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fdd1806cc30 0x7fdd1806f0e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fdd24004f40 tx=0x7fdd2400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:53:34.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.102+0000 7fdd2d600700 1 -- 172.21.15.123:0/829449706 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fdd1c05abf0 con 0x7fdd30103540 2024-03-30T04:53:34.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:33 smithi135 ceph-mon[106011]: from='client.35244 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:53:34.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:33 smithi135 ceph-mon[106011]: from='client.35248 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:53:34.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:33 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3651015227' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:53:34.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.322+0000 7fdd37000700 1 -- 172.21.15.123:0/829449706 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdd30109b30 con 0x7fdd1806cc30 2024-03-30T04:53:34.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.323+0000 7fdd2d600700 1 -- 172.21.15.123:0/829449706 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fdd30109b30 con 0x7fdd1806cc30 2024-03-30T04:53:34.321 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:53:34.321 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:53:34.321 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:53:34.322 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T04:53:34.322 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:53:34.322 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T04:53:34.322 INFO:teuthology.orchestra.run.smithi123.stdout: "mds", 2024-03-30T04:53:34.322 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T04:53:34.322 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T04:53:34.322 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T04:53:34.322 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:53:34.322 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "16/21 daemons upgraded", 2024-03-30T04:53:34.322 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T04:53:34.322 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:53:34.322 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:53:34.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.325+0000 7fdd37000700 1 -- 172.21.15.123:0/829449706 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fdd1806cc30 msgr2=0x7fdd1806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:53:34.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.325+0000 7fdd37000700 1 --2- 172.21.15.123:0/829449706 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fdd1806cc30 0x7fdd1806f0e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fdd24004f40 tx=0x7fdd2400b040 comp rx=0 tx=0).stop 2024-03-30T04:53:34.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.325+0000 7fdd37000700 1 -- 172.21.15.123:0/829449706 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd30103540 msgr2=0x7fdd30115fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:53:34.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.325+0000 7fdd37000700 1 --2- 172.21.15.123:0/829449706 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd30103540 0x7fdd30115fb0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c009390 tx=0x7fdd1c009e30 comp rx=0 tx=0).stop 2024-03-30T04:53:34.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.325+0000 7fdd37000700 1 -- 172.21.15.123:0/829449706 shutdown_connections 2024-03-30T04:53:34.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.325+0000 7fdd37000700 1 --2- 172.21.15.123:0/829449706 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fdd1806cc30 0x7fdd1806f0e0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:34.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.325+0000 7fdd37000700 1 --2- 172.21.15.123:0/829449706 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdd30102ba0 0x7fdd30115a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:34.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.325+0000 7fdd37000700 1 --2- 172.21.15.123:0/829449706 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd30103540 0x7fdd30115fb0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:34.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.325+0000 7fdd37000700 1 -- 172.21.15.123:0/829449706 >> 172.21.15.123:0/829449706 conn(0x7fdd300f8750 msgr2=0x7fdd300f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:53:34.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.326+0000 7fdd37000700 1 -- 172.21.15.123:0/829449706 shutdown_connections 2024-03-30T04:53:34.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.326+0000 7fdd37000700 1 -- 172.21.15.123:0/829449706 wait complete. 2024-03-30T04:53:34.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.459+0000 7f4f58e00700 1 -- 172.21.15.123:0/2829940258 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f54102e00 msgr2=0x7f4f541031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:53:34.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.459+0000 7f4f58e00700 1 --2- 172.21.15.123:0/2829940258 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f54102e00 0x7f4f541031d0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f4f44004940 tx=0x7f4f44009e30 comp rx=0 tx=0).stop 2024-03-30T04:53:34.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.459+0000 7f4f58e00700 1 -- 172.21.15.123:0/2829940258 shutdown_connections 2024-03-30T04:53:34.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.459+0000 7f4f58e00700 1 --2- 172.21.15.123:0/2829940258 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4f541037a0 0x7f4f540fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:34.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.459+0000 7f4f58e00700 1 --2- 172.21.15.123:0/2829940258 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f54102e00 0x7f4f541031d0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:34.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.459+0000 7f4f58e00700 1 -- 172.21.15.123:0/2829940258 >> 172.21.15.123:0/2829940258 conn(0x7f4f540f89b0 msgr2=0x7f4f540fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:53:34.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.459+0000 7f4f58e00700 1 -- 172.21.15.123:0/2829940258 shutdown_connections 2024-03-30T04:53:34.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.459+0000 7f4f58e00700 1 -- 172.21.15.123:0/2829940258 wait complete. 2024-03-30T04:53:34.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.460+0000 7f4f58e00700 1 Processor -- start 2024-03-30T04:53:34.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.460+0000 7f4f58e00700 1 -- start start 2024-03-30T04:53:34.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.461+0000 7f4f58e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f54102e00 0x7f4f54113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:53:34.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.461+0000 7f4f58e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4f541037a0 0x7f4f54114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:53:34.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.461+0000 7f4f58e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f541146b0 con 0x7f4f54102e00 2024-03-30T04:53:34.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.461+0000 7f4f58e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5410daa0 con 0x7f4f541037a0 2024-03-30T04:53:34.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.461+0000 7f4f51e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f54102e00 0x7f4f54113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:53:34.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.461+0000 7f4f51e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f54102e00 0x7f4f54113b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35382/0 (socket says 172.21.15.123:35382) 2024-03-30T04:53:34.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.461+0000 7f4f51e00700 1 -- 172.21.15.123:0/369301670 learned_addr learned my addr 172.21.15.123:0/369301670 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:53:34.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.461+0000 7f4f51400700 1 --2- 172.21.15.123:0/369301670 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4f541037a0 0x7f4f54114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:53:34.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.461+0000 7f4f51e00700 1 -- 172.21.15.123:0/369301670 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4f541037a0 msgr2=0x7f4f54114060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:53:34.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.461+0000 7f4f51e00700 1 --2- 172.21.15.123:0/369301670 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4f541037a0 0x7f4f54114060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:34.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.461+0000 7f4f51e00700 1 -- 172.21.15.123:0/369301670 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f44005040 con 0x7f4f54102e00 2024-03-30T04:53:34.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.462+0000 7f4f51e00700 1 --2- 172.21.15.123:0/369301670 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f54102e00 0x7f4f54113b20 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f4f44009390 tx=0x7f4f44009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:53:34.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.462+0000 7f4f4a000700 1 -- 172.21.15.123:0/369301670 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f4400e040 con 0x7f4f54102e00 2024-03-30T04:53:34.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.462+0000 7f4f4a000700 1 -- 172.21.15.123:0/369301670 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4f44013420 con 0x7f4f54102e00 2024-03-30T04:53:34.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.462+0000 7f4f4a000700 1 -- 172.21.15.123:0/369301670 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f4401c630 con 0x7f4f54102e00 2024-03-30T04:53:34.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.462+0000 7f4f58e00700 1 -- 172.21.15.123:0/369301670 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f5410dd20 con 0x7f4f54102e00 2024-03-30T04:53:34.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.462+0000 7f4f58e00700 1 -- 172.21.15.123:0/369301670 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f5410e2e0 con 0x7f4f54102e00 2024-03-30T04:53:34.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.463+0000 7f4f58e00700 1 -- 172.21.15.123:0/369301670 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f540fe550 con 0x7f4f54102e00 2024-03-30T04:53:34.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.464+0000 7f4f4a000700 1 -- 172.21.15.123:0/369301670 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f4f440139e0 con 0x7f4f54102e00 2024-03-30T04:53:34.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.465+0000 7f4f4a000700 1 --2- 172.21.15.123:0/369301670 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4f3806cc30 0x7f4f3806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:53:34.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.465+0000 7f4f4a000700 1 -- 172.21.15.123:0/369301670 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f4f4408d350 con 0x7f4f54102e00 2024-03-30T04:53:34.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.465+0000 7f4f51400700 1 --2- 172.21.15.123:0/369301670 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4f3806cc30 0x7f4f3806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:53:34.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.466+0000 7f4f51400700 1 --2- 172.21.15.123:0/369301670 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4f3806cc30 0x7f4f3806f0e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f4f5410ec60 tx=0x7f4f4000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:53:34.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.470+0000 7f4f4a000700 1 -- 172.21.15.123:0/369301670 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4f4405ae20 con 0x7f4f54102e00 2024-03-30T04:53:34.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.754+0000 7f4f58e00700 1 -- 172.21.15.123:0/369301670 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f4f54109e50 con 0x7f4f54102e00 2024-03-30T04:53:34.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.755+0000 7f4f4a000700 1 -- 172.21.15.123:0/369301670 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f4f4405a9b0 con 0x7f4f54102e00 2024-03-30T04:53:34.753 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T04:53:34.753 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-30T04:53:34.753 INFO:teuthology.orchestra.run.smithi123.stdout: host smithi135 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi135/config 2024-03-30T04:53:34.754 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-30T04:53:34.754 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T04:53:34.754 INFO:teuthology.orchestra.run.smithi123.stdout:Traceback (most recent call last): 2024-03-30T04:53:34.754 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-30T04:53:34.754 INFO:teuthology.orchestra.run.smithi123.stdout: "__main__", mod_spec) 2024-03-30T04:53:34.754 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-30T04:53:34.754 INFO:teuthology.orchestra.run.smithi123.stdout: exec(code, run_globals) 2024-03-30T04:53:34.754 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10889, in 2024-03-30T04:53:34.754 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10877, in main 2024-03-30T04:53:34.754 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-30T04:53:34.754 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-30T04:53:34.754 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-30T04:53:34.754 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-30T04:53:34.755 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-03-30T04:53:34.755 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-30T04:53:34.755 INFO:teuthology.orchestra.run.smithi123.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T04:53:34.755 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T04:53:34.755 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T04:53:34.755 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T04:53:34.755 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T04:53:34.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.757+0000 7f4f58e00700 1 -- 172.21.15.123:0/369301670 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4f3806cc30 msgr2=0x7f4f3806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:53:34.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.757+0000 7f4f58e00700 1 --2- 172.21.15.123:0/369301670 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4f3806cc30 0x7f4f3806f0e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f4f5410ec60 tx=0x7f4f4000b040 comp rx=0 tx=0).stop 2024-03-30T04:53:34.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.757+0000 7f4f58e00700 1 -- 172.21.15.123:0/369301670 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f54102e00 msgr2=0x7f4f54113b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:53:34.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.757+0000 7f4f58e00700 1 --2- 172.21.15.123:0/369301670 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f54102e00 0x7f4f54113b20 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f4f44009390 tx=0x7f4f44009e30 comp rx=0 tx=0).stop 2024-03-30T04:53:34.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.757+0000 7f4f58e00700 1 -- 172.21.15.123:0/369301670 shutdown_connections 2024-03-30T04:53:34.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.757+0000 7f4f58e00700 1 --2- 172.21.15.123:0/369301670 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4f3806cc30 0x7f4f3806f0e0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:34.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.757+0000 7f4f58e00700 1 --2- 172.21.15.123:0/369301670 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f54102e00 0x7f4f54113b20 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:34.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.757+0000 7f4f58e00700 1 --2- 172.21.15.123:0/369301670 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4f541037a0 0x7f4f54114060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:53:34.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.757+0000 7f4f58e00700 1 -- 172.21.15.123:0/369301670 >> 172.21.15.123:0/369301670 conn(0x7f4f540f89b0 msgr2=0x7f4f540f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:53:34.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.757+0000 7f4f58e00700 1 -- 172.21.15.123:0/369301670 shutdown_connections 2024-03-30T04:53:34.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:53:34.757+0000 7f4f58e00700 1 -- 172.21.15.123:0/369301670 wait complete. 2024-03-30T04:53:34.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:34 smithi123 ceph-mon[139369]: from='client.35252 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:53:34.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:34 smithi123 ceph-mon[139369]: pgmap v672: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:34.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:34 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2383101490' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:53:35.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:34 smithi135 ceph-mon[106011]: from='client.35252 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:53:35.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:34 smithi135 ceph-mon[106011]: pgmap v672: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:35.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:34 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2383101490' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:53:35.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:35 smithi123 ceph-mon[139369]: from='client.35264 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:53:35.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:35 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/369301670' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:53:36.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:35 smithi135 ceph-mon[106011]: from='client.35264 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:53:36.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:35 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/369301670' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:53:36.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:36 smithi123 ceph-mon[139369]: pgmap v673: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:37.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:36 smithi135 ceph-mon[106011]: pgmap v673: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:38.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:38 smithi123 ceph-mon[139369]: pgmap v674: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:39.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:38 smithi135 ceph-mon[106011]: pgmap v674: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:40.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:40 smithi123 ceph-mon[139369]: pgmap v675: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:41.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:40 smithi135 ceph-mon[106011]: pgmap v675: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:42.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:42 smithi123 ceph-mon[139369]: pgmap v676: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:43.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:42 smithi135 ceph-mon[106011]: pgmap v676: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:44.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:44 smithi123 ceph-mon[139369]: pgmap v677: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:44.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:53:45.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:44 smithi135 ceph-mon[106011]: pgmap v677: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:45.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:44 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:53:46.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:46 smithi123 ceph-mon[139369]: pgmap v678: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:47.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:46 smithi135 ceph-mon[106011]: pgmap v678: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:48.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:48 smithi123 ceph-mon[139369]: pgmap v679: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:49.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:48 smithi135 ceph-mon[106011]: pgmap v679: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:50.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:50 smithi123 ceph-mon[139369]: pgmap v680: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:51.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:50 smithi135 ceph-mon[106011]: pgmap v680: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:52.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:52 smithi123 ceph-mon[139369]: pgmap v681: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:53.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:52 smithi135 ceph-mon[106011]: pgmap v681: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:54.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:54 smithi123 ceph-mon[139369]: pgmap v682: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:55.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:54 smithi135 ceph-mon[106011]: pgmap v682: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:56.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:56 smithi123 ceph-mon[139369]: pgmap v683: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:57.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:56 smithi135 ceph-mon[106011]: pgmap v683: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:58.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:58 smithi123 ceph-mon[139369]: pgmap v684: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:59.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:58 smithi135 ceph-mon[106011]: pgmap v684: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:53:59.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:53:59 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:54:00.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:53:59 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:54:00.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:54:00 smithi123 ceph-mon[139369]: pgmap v685: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:01.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:54:00 smithi135 ceph-mon[106011]: pgmap v685: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:03.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:54:02 smithi135 ceph-mon[106011]: pgmap v686: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:03.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:54:02 smithi123 ceph-mon[139369]: pgmap v686: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:04.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:04.906+0000 7f49e2800700 1 -- 172.21.15.123:0/3950387137 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49dc1037a0 msgr2=0x7f49dc0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:04.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:04.906+0000 7f49e2800700 1 --2- 172.21.15.123:0/3950387137 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49dc1037a0 0x7f49dc0fce10 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f49c8004940 tx=0x7f49c8009e30 comp rx=0 tx=0).stop 2024-03-30T04:54:04.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:04.906+0000 7f49e2800700 1 -- 172.21.15.123:0/3950387137 shutdown_connections 2024-03-30T04:54:04.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:04.906+0000 7f49e2800700 1 --2- 172.21.15.123:0/3950387137 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49dc1037a0 0x7f49dc0fce10 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:04.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:04.906+0000 7f49e2800700 1 --2- 172.21.15.123:0/3950387137 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49dc102e00 0x7f49dc1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:04.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:04.906+0000 7f49e2800700 1 -- 172.21.15.123:0/3950387137 >> 172.21.15.123:0/3950387137 conn(0x7f49dc0f89b0 msgr2=0x7f49dc0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:54:04.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:04.906+0000 7f49e2800700 1 -- 172.21.15.123:0/3950387137 shutdown_connections 2024-03-30T04:54:04.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:04.906+0000 7f49e2800700 1 -- 172.21.15.123:0/3950387137 wait complete. 2024-03-30T04:54:04.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:04.907+0000 7f49e2800700 1 Processor -- start 2024-03-30T04:54:04.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:04.907+0000 7f49e2800700 1 -- start start 2024-03-30T04:54:04.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:04.908+0000 7f49e2800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49dc102e00 0x7f49dc115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:54:04.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:04.908+0000 7f49e2800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49dc1037a0 0x7f49dc1161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:54:04.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:04.908+0000 7f49e2800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49dc116810 con 0x7f49dc102e00 2024-03-30T04:54:04.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:04.908+0000 7f49e2800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49dc10fbe0 con 0x7f49dc1037a0 2024-03-30T04:54:04.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:04.908+0000 7f49db400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49dc102e00 0x7f49dc115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:54:04.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:04.908+0000 7f49db400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49dc102e00 0x7f49dc115c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34972/0 (socket says 172.21.15.123:34972) 2024-03-30T04:54:04.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:04.908+0000 7f49db400700 1 -- 172.21.15.123:0/913923049 learned_addr learned my addr 172.21.15.123:0/913923049 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:54:04.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:04.908+0000 7f49daa00700 1 --2- 172.21.15.123:0/913923049 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49dc1037a0 0x7f49dc1161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:54:04.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:04.908+0000 7f49db400700 1 -- 172.21.15.123:0/913923049 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49dc1037a0 msgr2=0x7f49dc1161c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:04.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:04.908+0000 7f49db400700 1 --2- 172.21.15.123:0/913923049 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49dc1037a0 0x7f49dc1161c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:04.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:04.908+0000 7f49db400700 1 -- 172.21.15.123:0/913923049 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c8005040 con 0x7f49dc102e00 2024-03-30T04:54:04.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:04.909+0000 7f49db400700 1 --2- 172.21.15.123:0/913923049 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49dc102e00 0x7f49dc115c80 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f49cc00eb40 tx=0x7f49cc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:54:04.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:04.909+0000 7f49d3400700 1 -- 172.21.15.123:0/913923049 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f49cc007cc0 con 0x7f49dc102e00 2024-03-30T04:54:04.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:04.909+0000 7f49d3400700 1 -- 172.21.15.123:0/913923049 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f49cc00f040 con 0x7f49dc102e00 2024-03-30T04:54:04.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:04.909+0000 7f49e2800700 1 -- 172.21.15.123:0/913923049 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49dc10fe60 con 0x7f49dc102e00 2024-03-30T04:54:04.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:04.909+0000 7f49d3400700 1 -- 172.21.15.123:0/913923049 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f49cc00c990 con 0x7f49dc102e00 2024-03-30T04:54:04.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:04.909+0000 7f49e2800700 1 -- 172.21.15.123:0/913923049 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49dc110420 con 0x7f49dc102e00 2024-03-30T04:54:04.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:04.910+0000 7f49e2800700 1 -- 172.21.15.123:0/913923049 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49dc0fe550 con 0x7f49dc102e00 2024-03-30T04:54:04.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:04.911+0000 7f49d3400700 1 -- 172.21.15.123:0/913923049 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f49cc00caf0 con 0x7f49dc102e00 2024-03-30T04:54:04.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:04.913+0000 7f49d3400700 1 --2- 172.21.15.123:0/913923049 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f49c006cc30 0x7f49c006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:54:04.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:04.913+0000 7f49d3400700 1 -- 172.21.15.123:0/913923049 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f49cc08bc00 con 0x7f49dc102e00 2024-03-30T04:54:04.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:04.913+0000 7f49daa00700 1 --2- 172.21.15.123:0/913923049 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f49c006cc30 0x7f49c006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:54:04.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:04.914+0000 7f49daa00700 1 --2- 172.21.15.123:0/913923049 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f49c006cc30 0x7f49c006f0e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f49dc111520 tx=0x7f49c8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:54:04.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:04.918+0000 7f49d3400700 1 -- 172.21.15.123:0/913923049 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f49cc059700 con 0x7f49dc102e00 2024-03-30T04:54:05.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:54:04 smithi135 ceph-mon[106011]: pgmap v687: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:05.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.140+0000 7f49e2800700 1 -- 172.21.15.123:0/913923049 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f49dc04bf40 con 0x7f49c006cc30 2024-03-30T04:54:05.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.142+0000 7f49d3400700 1 -- 172.21.15.123:0/913923049 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f49dc04bf40 con 0x7f49c006cc30 2024-03-30T04:54:05.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.144+0000 7f49e2800700 1 -- 172.21.15.123:0/913923049 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f49c006cc30 msgr2=0x7f49c006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:05.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.144+0000 7f49e2800700 1 --2- 172.21.15.123:0/913923049 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f49c006cc30 0x7f49c006f0e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f49dc111520 tx=0x7f49c8009e30 comp rx=0 tx=0).stop 2024-03-30T04:54:05.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.144+0000 7f49e2800700 1 -- 172.21.15.123:0/913923049 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49dc102e00 msgr2=0x7f49dc115c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:05.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.144+0000 7f49e2800700 1 --2- 172.21.15.123:0/913923049 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49dc102e00 0x7f49dc115c80 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f49cc00eb40 tx=0x7f49cc00b040 comp rx=0 tx=0).stop 2024-03-30T04:54:05.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.144+0000 7f49e2800700 1 -- 172.21.15.123:0/913923049 shutdown_connections 2024-03-30T04:54:05.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.144+0000 7f49e2800700 1 --2- 172.21.15.123:0/913923049 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f49c006cc30 0x7f49c006f0e0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:05.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.144+0000 7f49e2800700 1 --2- 172.21.15.123:0/913923049 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49dc102e00 0x7f49dc115c80 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:05.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.144+0000 7f49e2800700 1 --2- 172.21.15.123:0/913923049 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49dc1037a0 0x7f49dc1161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:05.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.144+0000 7f49e2800700 1 -- 172.21.15.123:0/913923049 >> 172.21.15.123:0/913923049 conn(0x7f49dc0f89b0 msgr2=0x7f49dc0f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:54:05.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.144+0000 7f49e2800700 1 -- 172.21.15.123:0/913923049 shutdown_connections 2024-03-30T04:54:05.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.144+0000 7f49e2800700 1 -- 172.21.15.123:0/913923049 wait complete. 2024-03-30T04:54:05.157 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:54:05.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:54:04 smithi123 ceph-mon[139369]: pgmap v687: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:05.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.279+0000 7fe3dae00700 1 -- 172.21.15.123:0/2238244922 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3d40fdc40 msgr2=0x7fe3d40fe010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:05.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.279+0000 7fe3dae00700 1 --2- 172.21.15.123:0/2238244922 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3d40fdc40 0x7fe3d40fe010 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fe3c8004940 tx=0x7fe3c8009e30 comp rx=0 tx=0).stop 2024-03-30T04:54:05.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.279+0000 7fe3dae00700 1 -- 172.21.15.123:0/2238244922 shutdown_connections 2024-03-30T04:54:05.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.279+0000 7fe3dae00700 1 --2- 172.21.15.123:0/2238244922 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe3d40fe5e0 0x7fe3d4106c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:05.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.279+0000 7fe3dae00700 1 --2- 172.21.15.123:0/2238244922 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3d40fdc40 0x7fe3d40fe010 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:05.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.279+0000 7fe3dae00700 1 -- 172.21.15.123:0/2238244922 >> 172.21.15.123:0/2238244922 conn(0x7fe3d40f7670 msgr2=0x7fe3d40f9a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:54:05.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.279+0000 7fe3dae00700 1 -- 172.21.15.123:0/2238244922 shutdown_connections 2024-03-30T04:54:05.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.279+0000 7fe3dae00700 1 -- 172.21.15.123:0/2238244922 wait complete. 2024-03-30T04:54:05.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.280+0000 7fe3dae00700 1 Processor -- start 2024-03-30T04:54:05.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.280+0000 7fe3dae00700 1 -- start start 2024-03-30T04:54:05.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.280+0000 7fe3dae00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe3d40fdc40 0x7fe3d40fba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:54:05.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.280+0000 7fe3dae00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3d40fe5e0 0x7fe3d40fbfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:54:05.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.280+0000 7fe3dae00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d40fc5d0 con 0x7fe3d40fe5e0 2024-03-30T04:54:05.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.280+0000 7fe3dae00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d40fc710 con 0x7fe3d40fdc40 2024-03-30T04:54:05.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.281+0000 7fe3d9000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3d40fe5e0 0x7fe3d40fbfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:54:05.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.281+0000 7fe3d9000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3d40fe5e0 0x7fe3d40fbfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34990/0 (socket says 172.21.15.123:34990) 2024-03-30T04:54:05.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.281+0000 7fe3d9000700 1 -- 172.21.15.123:0/4044041977 learned_addr learned my addr 172.21.15.123:0/4044041977 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:54:05.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.281+0000 7fe3d9a00700 1 --2- 172.21.15.123:0/4044041977 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe3d40fdc40 0x7fe3d40fba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:54:05.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.281+0000 7fe3d9000700 1 -- 172.21.15.123:0/4044041977 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe3d40fdc40 msgr2=0x7fe3d40fba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:05.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.281+0000 7fe3d9000700 1 --2- 172.21.15.123:0/4044041977 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe3d40fdc40 0x7fe3d40fba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:05.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.281+0000 7fe3d9000700 1 -- 172.21.15.123:0/4044041977 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c8005040 con 0x7fe3d40fe5e0 2024-03-30T04:54:05.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.282+0000 7fe3d9000700 1 --2- 172.21.15.123:0/4044041977 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3d40fe5e0 0x7fe3d40fbfb0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fe3c000b790 tx=0x7fe3c0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:54:05.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.282+0000 7fe3d1600700 1 -- 172.21.15.123:0/4044041977 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3c000f6f0 con 0x7fe3d40fe5e0 2024-03-30T04:54:05.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.282+0000 7fe3d1600700 1 -- 172.21.15.123:0/4044041977 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe3c000fd00 con 0x7fe3d40fe5e0 2024-03-30T04:54:05.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.282+0000 7fe3dae00700 1 -- 172.21.15.123:0/4044041977 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3d4119c10 con 0x7fe3d40fe5e0 2024-03-30T04:54:05.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.282+0000 7fe3d1600700 1 -- 172.21.15.123:0/4044041977 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3c00103a0 con 0x7fe3d40fe5e0 2024-03-30T04:54:05.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.282+0000 7fe3dae00700 1 -- 172.21.15.123:0/4044041977 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3d411a0a0 con 0x7fe3d40fe5e0 2024-03-30T04:54:05.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.283+0000 7fe3dae00700 1 -- 172.21.15.123:0/4044041977 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3d4066930 con 0x7fe3d40fe5e0 2024-03-30T04:54:05.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.284+0000 7fe3d1600700 1 -- 172.21.15.123:0/4044041977 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fe3c000f850 con 0x7fe3d40fe5e0 2024-03-30T04:54:05.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.285+0000 7fe3d1600700 1 --2- 172.21.15.123:0/4044041977 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe3bc06cc20 0x7fe3bc06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:54:05.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.285+0000 7fe3d1600700 1 -- 172.21.15.123:0/4044041977 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe3c008bf20 con 0x7fe3d40fe5e0 2024-03-30T04:54:05.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.285+0000 7fe3d9a00700 1 --2- 172.21.15.123:0/4044041977 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe3bc06cc20 0x7fe3bc06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:54:05.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.286+0000 7fe3d9a00700 1 --2- 172.21.15.123:0/4044041977 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe3bc06cc20 0x7fe3bc06f0d0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fe3c8009fd0 tx=0x7fe3c8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:54:05.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.290+0000 7fe3d1600700 1 -- 172.21.15.123:0/4044041977 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe3c0055ee0 con 0x7fe3d40fe5e0 2024-03-30T04:54:05.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.511+0000 7fe3dae00700 1 -- 172.21.15.123:0/4044041977 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe3d404bed0 con 0x7fe3bc06cc20 2024-03-30T04:54:05.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.512+0000 7fe3d1600700 1 -- 172.21.15.123:0/4044041977 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fe3d404bed0 con 0x7fe3bc06cc20 2024-03-30T04:54:05.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.514+0000 7fe3dae00700 1 -- 172.21.15.123:0/4044041977 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe3bc06cc20 msgr2=0x7fe3bc06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:05.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.514+0000 7fe3dae00700 1 --2- 172.21.15.123:0/4044041977 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe3bc06cc20 0x7fe3bc06f0d0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fe3c8009fd0 tx=0x7fe3c8009e30 comp rx=0 tx=0).stop 2024-03-30T04:54:05.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.515+0000 7fe3dae00700 1 -- 172.21.15.123:0/4044041977 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3d40fe5e0 msgr2=0x7fe3d40fbfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:05.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.515+0000 7fe3dae00700 1 --2- 172.21.15.123:0/4044041977 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3d40fe5e0 0x7fe3d40fbfb0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fe3c000b790 tx=0x7fe3c0009250 comp rx=0 tx=0).stop 2024-03-30T04:54:05.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.515+0000 7fe3dae00700 1 -- 172.21.15.123:0/4044041977 shutdown_connections 2024-03-30T04:54:05.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.515+0000 7fe3dae00700 1 --2- 172.21.15.123:0/4044041977 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe3bc06cc20 0x7fe3bc06f0d0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:05.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.515+0000 7fe3dae00700 1 --2- 172.21.15.123:0/4044041977 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe3d40fdc40 0x7fe3d40fba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:05.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.515+0000 7fe3dae00700 1 --2- 172.21.15.123:0/4044041977 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3d40fe5e0 0x7fe3d40fbfb0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:05.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.515+0000 7fe3dae00700 1 -- 172.21.15.123:0/4044041977 >> 172.21.15.123:0/4044041977 conn(0x7fe3d40f7670 msgr2=0x7fe3d40f8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:54:05.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.515+0000 7fe3dae00700 1 -- 172.21.15.123:0/4044041977 shutdown_connections 2024-03-30T04:54:05.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.515+0000 7fe3dae00700 1 -- 172.21.15.123:0/4044041977 wait complete. 2024-03-30T04:54:05.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.648+0000 7f6e6c800700 1 -- 172.21.15.123:0/2850970870 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6e640fe830 msgr2=0x7f6e64106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:05.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.648+0000 7f6e6c800700 1 --2- 172.21.15.123:0/2850970870 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6e640fe830 0x7f6e64106f00 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f6e54004940 tx=0x7f6e54009e30 comp rx=0 tx=0).stop 2024-03-30T04:54:05.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.649+0000 7f6e6c800700 1 -- 172.21.15.123:0/2850970870 shutdown_connections 2024-03-30T04:54:05.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.649+0000 7f6e6c800700 1 --2- 172.21.15.123:0/2850970870 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6e640fe830 0x7f6e64106f00 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:05.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.649+0000 7f6e6c800700 1 --2- 172.21.15.123:0/2850970870 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6e640fde90 0x7f6e640fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:05.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.649+0000 7f6e6c800700 1 -- 172.21.15.123:0/2850970870 >> 172.21.15.123:0/2850970870 conn(0x7f6e64072bc0 msgr2=0x7f6e64072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:54:05.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.649+0000 7f6e6c800700 1 -- 172.21.15.123:0/2850970870 shutdown_connections 2024-03-30T04:54:05.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.649+0000 7f6e6c800700 1 -- 172.21.15.123:0/2850970870 wait complete. 2024-03-30T04:54:05.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.649+0000 7f6e6c800700 1 Processor -- start 2024-03-30T04:54:05.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.650+0000 7f6e6c800700 1 -- start start 2024-03-30T04:54:05.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.650+0000 7f6e6c800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6e640fde90 0x7f6e640fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:54:05.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.650+0000 7f6e6c800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6e640fe830 0x7f6e640fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:54:05.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.650+0000 7f6e6c800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e640fc820 con 0x7f6e640fe830 2024-03-30T04:54:05.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.650+0000 7f6e6c800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e640fc960 con 0x7f6e640fde90 2024-03-30T04:54:05.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.650+0000 7f6e69400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6e640fe830 0x7f6e640fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:54:05.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.650+0000 7f6e69400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6e640fe830 0x7f6e640fc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35020/0 (socket says 172.21.15.123:35020) 2024-03-30T04:54:05.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.650+0000 7f6e69400700 1 -- 172.21.15.123:0/3710294097 learned_addr learned my addr 172.21.15.123:0/3710294097 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:54:05.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.650+0000 7f6e69e00700 1 --2- 172.21.15.123:0/3710294097 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6e640fde90 0x7f6e640fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:54:05.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.651+0000 7f6e69400700 1 -- 172.21.15.123:0/3710294097 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6e640fde90 msgr2=0x7f6e640fbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:05.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.651+0000 7f6e69400700 1 --2- 172.21.15.123:0/3710294097 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6e640fde90 0x7f6e640fbd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:05.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.651+0000 7f6e69400700 1 -- 172.21.15.123:0/3710294097 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e54005040 con 0x7f6e640fe830 2024-03-30T04:54:05.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.651+0000 7f6e69400700 1 --2- 172.21.15.123:0/3710294097 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6e640fe830 0x7f6e640fc290 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f6e54000c00 tx=0x7f6e54009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:54:05.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.651+0000 7f6e5e000700 1 -- 172.21.15.123:0/3710294097 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e5400e040 con 0x7f6e640fe830 2024-03-30T04:54:05.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.651+0000 7f6e5e000700 1 -- 172.21.15.123:0/3710294097 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6e5400bed0 con 0x7f6e640fe830 2024-03-30T04:54:05.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.651+0000 7f6e6c800700 1 -- 172.21.15.123:0/3710294097 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e640fcbb0 con 0x7f6e640fe830 2024-03-30T04:54:05.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.651+0000 7f6e5e000700 1 -- 172.21.15.123:0/3710294097 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e5401c3f0 con 0x7f6e640fe830 2024-03-30T04:54:05.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.651+0000 7f6e6c800700 1 -- 172.21.15.123:0/3710294097 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e6411a190 con 0x7f6e640fe830 2024-03-30T04:54:05.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.652+0000 7f6e6c800700 1 -- 172.21.15.123:0/3710294097 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e64065e30 con 0x7f6e640fe830 2024-03-30T04:54:05.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.653+0000 7f6e5e000700 1 -- 172.21.15.123:0/3710294097 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f6e54013420 con 0x7f6e640fe830 2024-03-30T04:54:05.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.654+0000 7f6e5e000700 1 --2- 172.21.15.123:0/3710294097 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6e4c06cc30 0x7f6e4c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:54:05.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.654+0000 7f6e5e000700 1 -- 172.21.15.123:0/3710294097 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f6e5408d4c0 con 0x7f6e640fe830 2024-03-30T04:54:05.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.654+0000 7f6e69e00700 1 --2- 172.21.15.123:0/3710294097 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6e4c06cc30 0x7f6e4c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:54:05.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.655+0000 7f6e69e00700 1 --2- 172.21.15.123:0/3710294097 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6e4c06cc30 0x7f6e4c06f0e0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f6e58004f40 tx=0x7f6e5800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:54:05.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.658+0000 7f6e5e000700 1 -- 172.21.15.123:0/3710294097 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6e54057480 con 0x7f6e640fe830 2024-03-30T04:54:05.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.872+0000 7f6e6c800700 1 -- 172.21.15.123:0/3710294097 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f6e64109e70 con 0x7f6e4c06cc30 2024-03-30T04:54:05.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.881+0000 7f6e5e000700 1 -- 172.21.15.123:0/3710294097 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f6e5800f040 con 0x7f6e4c06cc30 2024-03-30T04:54:05.880 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:54:05.880 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (23m) 38s ago 35m 14.6M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:54:05.880 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (20m) 38s ago 36m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T04:54:05.880 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (20m) 41s ago 34m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T04:54:05.880 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (22m) 38s ago 35m 73.5M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T04:54:05.880 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (16m) 38s ago 29m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T04:54:05.880 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (16m) 38s ago 29m 16.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T04:54:05.880 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (45s) 41s ago 29m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 87b073a6c9ca 2024-03-30T04:54:05.880 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (15m) 41s ago 29m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T04:54:05.881 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (25m) 38s ago 37m 510M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:54:05.881 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (24m) 41s ago 34m 456M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:54:05.881 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (21m) 38s ago 37m 70.0M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T04:54:05.881 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (21m) 41s ago 34m 51.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T04:54:05.881 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (24m) 38s ago 35m 8636k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:54:05.881 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (23m) 41s ago 34m 8678k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:54:05.881 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (20m) 38s ago 33m 107M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T04:54:05.881 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (19m) 38s ago 33m 97.2M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T04:54:05.881 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (18m) 38s ago 32m 89.8M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T04:54:05.881 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (18m) 41s ago 32m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T04:54:05.881 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (17m) 41s ago 31m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T04:54:05.882 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (16m) 41s ago 31m 88.8M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T04:54:05.882 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (23m) 38s ago 35m 63.9M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:54:05.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.884+0000 7f6e6c800700 1 -- 172.21.15.123:0/3710294097 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6e4c06cc30 msgr2=0x7f6e4c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:05.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.884+0000 7f6e6c800700 1 --2- 172.21.15.123:0/3710294097 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6e4c06cc30 0x7f6e4c06f0e0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f6e58004f40 tx=0x7f6e5800b040 comp rx=0 tx=0).stop 2024-03-30T04:54:05.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.884+0000 7f6e6c800700 1 -- 172.21.15.123:0/3710294097 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6e640fe830 msgr2=0x7f6e640fc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:05.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.884+0000 7f6e6c800700 1 --2- 172.21.15.123:0/3710294097 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6e640fe830 0x7f6e640fc290 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f6e54000c00 tx=0x7f6e54009e30 comp rx=0 tx=0).stop 2024-03-30T04:54:05.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.884+0000 7f6e6c800700 1 -- 172.21.15.123:0/3710294097 shutdown_connections 2024-03-30T04:54:05.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.884+0000 7f6e6c800700 1 --2- 172.21.15.123:0/3710294097 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6e4c06cc30 0x7f6e4c06f0e0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:05.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.884+0000 7f6e6c800700 1 --2- 172.21.15.123:0/3710294097 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6e640fde90 0x7f6e640fbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:05.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.884+0000 7f6e6c800700 1 --2- 172.21.15.123:0/3710294097 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6e640fe830 0x7f6e640fc290 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:05.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.884+0000 7f6e6c800700 1 -- 172.21.15.123:0/3710294097 >> 172.21.15.123:0/3710294097 conn(0x7f6e64072bc0 msgr2=0x7f6e64103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:54:05.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.884+0000 7f6e6c800700 1 -- 172.21.15.123:0/3710294097 shutdown_connections 2024-03-30T04:54:05.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:05.884+0000 7f6e6c800700 1 -- 172.21.15.123:0/3710294097 wait complete. 2024-03-30T04:54:06.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.017+0000 7f7a84a00700 1 -- 172.21.15.123:0/2355939252 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a801035d0 msgr2=0x7f7a800fcc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:06.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.017+0000 7f7a84a00700 1 --2- 172.21.15.123:0/2355939252 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a801035d0 0x7f7a800fcc40 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c004940 tx=0x7f7a6c009e30 comp rx=0 tx=0).stop 2024-03-30T04:54:06.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.017+0000 7f7a84a00700 1 -- 172.21.15.123:0/2355939252 shutdown_connections 2024-03-30T04:54:06.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.017+0000 7f7a84a00700 1 --2- 172.21.15.123:0/2355939252 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a801035d0 0x7f7a800fcc40 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:06.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.017+0000 7f7a84a00700 1 --2- 172.21.15.123:0/2355939252 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7a80102c30 0x7f7a80103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:06.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.017+0000 7f7a84a00700 1 -- 172.21.15.123:0/2355939252 >> 172.21.15.123:0/2355939252 conn(0x7f7a800f8760 msgr2=0x7f7a800fab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:54:06.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.017+0000 7f7a84a00700 1 -- 172.21.15.123:0/2355939252 shutdown_connections 2024-03-30T04:54:06.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.017+0000 7f7a84a00700 1 -- 172.21.15.123:0/2355939252 wait complete. 2024-03-30T04:54:06.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.018+0000 7f7a84a00700 1 Processor -- start 2024-03-30T04:54:06.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.018+0000 7f7a84a00700 1 -- start start 2024-03-30T04:54:06.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.018+0000 7f7a84a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7a80102c30 0x7f7a80115ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:54:06.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.018+0000 7f7a84a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a801035d0 0x7f7a80116020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:54:06.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.019+0000 7f7a84a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a80116670 con 0x7f7a801035d0 2024-03-30T04:54:06.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.019+0000 7f7a84a00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8010fa60 con 0x7f7a80102c30 2024-03-30T04:54:06.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.019+0000 7f7a7de00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a801035d0 0x7f7a80116020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:54:06.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.019+0000 7f7a7de00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a801035d0 0x7f7a80116020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35038/0 (socket says 172.21.15.123:35038) 2024-03-30T04:54:06.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.019+0000 7f7a7de00700 1 -- 172.21.15.123:0/2245738861 learned_addr learned my addr 172.21.15.123:0/2245738861 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:54:06.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.019+0000 7f7a7e800700 1 --2- 172.21.15.123:0/2245738861 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7a80102c30 0x7f7a80115ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:54:06.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.019+0000 7f7a7de00700 1 -- 172.21.15.123:0/2245738861 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7a80102c30 msgr2=0x7f7a80115ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:06.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.019+0000 7f7a7de00700 1 --2- 172.21.15.123:0/2245738861 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7a80102c30 0x7f7a80115ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:06.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.019+0000 7f7a7de00700 1 -- 172.21.15.123:0/2245738861 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a6c005040 con 0x7f7a801035d0 2024-03-30T04:54:06.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.019+0000 7f7a7de00700 1 --2- 172.21.15.123:0/2245738861 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a801035d0 0x7f7a80116020 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c009390 tx=0x7f7a6c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:54:06.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.020+0000 7f7a76a00700 1 -- 172.21.15.123:0/2245738861 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a6c00e040 con 0x7f7a801035d0 2024-03-30T04:54:06.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.020+0000 7f7a76a00700 1 -- 172.21.15.123:0/2245738861 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7a6c004290 con 0x7f7a801035d0 2024-03-30T04:54:06.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.020+0000 7f7a84a00700 1 -- 172.21.15.123:0/2245738861 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a8010fce0 con 0x7f7a801035d0 2024-03-30T04:54:06.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.020+0000 7f7a76a00700 1 -- 172.21.15.123:0/2245738861 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a6c013e70 con 0x7f7a801035d0 2024-03-30T04:54:06.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.020+0000 7f7a84a00700 1 -- 172.21.15.123:0/2245738861 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a801102a0 con 0x7f7a801035d0 2024-03-30T04:54:06.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.021+0000 7f7a84a00700 1 -- 172.21.15.123:0/2245738861 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a800fe380 con 0x7f7a801035d0 2024-03-30T04:54:06.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.022+0000 7f7a76a00700 1 -- 172.21.15.123:0/2245738861 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f7a6c013690 con 0x7f7a801035d0 2024-03-30T04:54:06.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.023+0000 7f7a76a00700 1 --2- 172.21.15.123:0/2245738861 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7a6406cbc0 0x7f7a6406f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:54:06.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.023+0000 7f7a76a00700 1 -- 172.21.15.123:0/2245738861 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7a6c08d2b0 con 0x7f7a801035d0 2024-03-30T04:54:06.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.023+0000 7f7a7e800700 1 --2- 172.21.15.123:0/2245738861 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7a6406cbc0 0x7f7a6406f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:54:06.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.024+0000 7f7a7e800700 1 --2- 172.21.15.123:0/2245738861 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7a6406cbc0 0x7f7a6406f070 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f7a70007fd0 tx=0x7f7a70009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:54:06.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.027+0000 7f7a76a00700 1 -- 172.21.15.123:0/2245738861 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7a6c05adb0 con 0x7f7a801035d0 2024-03-30T04:54:06.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.309+0000 7f7a84a00700 1 -- 172.21.15.123:0/2245738861 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f7a80109bc0 con 0x7f7a801035d0 2024-03-30T04:54:06.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.310+0000 7f7a76a00700 1 -- 172.21.15.123:0/2245738861 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f7a6c013420 con 0x7f7a801035d0 2024-03-30T04:54:06.308 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:54:06.308 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:54:06.309 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:54:06.309 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:54:06.309 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:54:06.309 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:54:06.309 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:54:06.309 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:54:06.309 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T04:54:06.309 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:54:06.309 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:54:06.309 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 4 2024-03-30T04:54:06.309 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:54:06.309 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:54:06.309 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 14 2024-03-30T04:54:06.309 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:54:06.310 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:54:06.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.312+0000 7f7a84a00700 1 -- 172.21.15.123:0/2245738861 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7a6406cbc0 msgr2=0x7f7a6406f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:06.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.312+0000 7f7a84a00700 1 --2- 172.21.15.123:0/2245738861 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7a6406cbc0 0x7f7a6406f070 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f7a70007fd0 tx=0x7f7a70009040 comp rx=0 tx=0).stop 2024-03-30T04:54:06.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.312+0000 7f7a84a00700 1 -- 172.21.15.123:0/2245738861 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a801035d0 msgr2=0x7f7a80116020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:06.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.312+0000 7f7a84a00700 1 --2- 172.21.15.123:0/2245738861 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a801035d0 0x7f7a80116020 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c009390 tx=0x7f7a6c009e30 comp rx=0 tx=0).stop 2024-03-30T04:54:06.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.312+0000 7f7a84a00700 1 -- 172.21.15.123:0/2245738861 shutdown_connections 2024-03-30T04:54:06.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.312+0000 7f7a84a00700 1 --2- 172.21.15.123:0/2245738861 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7a6406cbc0 0x7f7a6406f070 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:06.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.312+0000 7f7a84a00700 1 --2- 172.21.15.123:0/2245738861 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7a80102c30 0x7f7a80115ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:06.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.312+0000 7f7a84a00700 1 --2- 172.21.15.123:0/2245738861 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a801035d0 0x7f7a80116020 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:06.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.312+0000 7f7a84a00700 1 -- 172.21.15.123:0/2245738861 >> 172.21.15.123:0/2245738861 conn(0x7f7a800f8760 msgr2=0x7f7a800f9830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:54:06.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.313+0000 7f7a84a00700 1 -- 172.21.15.123:0/2245738861 shutdown_connections 2024-03-30T04:54:06.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.313+0000 7f7a84a00700 1 -- 172.21.15.123:0/2245738861 wait complete. 2024-03-30T04:54:06.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.445+0000 7efd1f000700 1 -- 172.21.15.123:0/645031840 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd181035e0 msgr2=0x7efd180fcc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:06.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.445+0000 7efd1f000700 1 --2- 172.21.15.123:0/645031840 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd181035e0 0x7efd180fcc50 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7efd04004940 tx=0x7efd04009e30 comp rx=0 tx=0).stop 2024-03-30T04:54:06.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.445+0000 7efd1f000700 1 -- 172.21.15.123:0/645031840 shutdown_connections 2024-03-30T04:54:06.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.445+0000 7efd1f000700 1 --2- 172.21.15.123:0/645031840 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd181035e0 0x7efd180fcc50 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:06.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.445+0000 7efd1f000700 1 --2- 172.21.15.123:0/645031840 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efd18102c40 0x7efd18103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:06.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.445+0000 7efd1f000700 1 -- 172.21.15.123:0/645031840 >> 172.21.15.123:0/645031840 conn(0x7efd180f8830 msgr2=0x7efd180fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:54:06.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.445+0000 7efd1f000700 1 -- 172.21.15.123:0/645031840 shutdown_connections 2024-03-30T04:54:06.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.445+0000 7efd1f000700 1 -- 172.21.15.123:0/645031840 wait complete. 2024-03-30T04:54:06.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.446+0000 7efd1f000700 1 Processor -- start 2024-03-30T04:54:06.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.446+0000 7efd1f000700 1 -- start start 2024-03-30T04:54:06.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.447+0000 7efd1f000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efd18102c40 0x7efd18115af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:54:06.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.447+0000 7efd1f000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd181035e0 0x7efd18116030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:54:06.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.447+0000 7efd1f000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd18116680 con 0x7efd181035e0 2024-03-30T04:54:06.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.447+0000 7efd1f000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd1810fa70 con 0x7efd18102c40 2024-03-30T04:54:06.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.447+0000 7efd1d200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd181035e0 0x7efd18116030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:54:06.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.447+0000 7efd1d200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd181035e0 0x7efd18116030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35062/0 (socket says 172.21.15.123:35062) 2024-03-30T04:54:06.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.447+0000 7efd1d200700 1 -- 172.21.15.123:0/2051223323 learned_addr learned my addr 172.21.15.123:0/2051223323 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:54:06.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.447+0000 7efd1dc00700 1 --2- 172.21.15.123:0/2051223323 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efd18102c40 0x7efd18115af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:54:06.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.447+0000 7efd1d200700 1 -- 172.21.15.123:0/2051223323 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efd18102c40 msgr2=0x7efd18115af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:06.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.447+0000 7efd1d200700 1 --2- 172.21.15.123:0/2051223323 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efd18102c40 0x7efd18115af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:06.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.447+0000 7efd1d200700 1 -- 172.21.15.123:0/2051223323 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd04005040 con 0x7efd181035e0 2024-03-30T04:54:06.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.448+0000 7efd1d200700 1 --2- 172.21.15.123:0/2051223323 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd181035e0 0x7efd18116030 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7efd04009390 tx=0x7efd04009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:54:06.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.448+0000 7efd15600700 1 -- 172.21.15.123:0/2051223323 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd0400e040 con 0x7efd181035e0 2024-03-30T04:54:06.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.448+0000 7efd1f000700 1 -- 172.21.15.123:0/2051223323 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd1810fcf0 con 0x7efd181035e0 2024-03-30T04:54:06.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.448+0000 7efd15600700 1 -- 172.21.15.123:0/2051223323 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efd04004290 con 0x7efd181035e0 2024-03-30T04:54:06.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.448+0000 7efd15600700 1 -- 172.21.15.123:0/2051223323 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd04013e70 con 0x7efd181035e0 2024-03-30T04:54:06.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.448+0000 7efd1f000700 1 -- 172.21.15.123:0/2051223323 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd181102b0 con 0x7efd181035e0 2024-03-30T04:54:06.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.449+0000 7efd1f000700 1 -- 172.21.15.123:0/2051223323 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd180fe390 con 0x7efd181035e0 2024-03-30T04:54:06.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.450+0000 7efd15600700 1 -- 172.21.15.123:0/2051223323 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7efd04013690 con 0x7efd181035e0 2024-03-30T04:54:06.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.451+0000 7efd15600700 1 --2- 172.21.15.123:0/2051223323 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7efd0006cc30 0x7efd0006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:54:06.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.451+0000 7efd15600700 1 -- 172.21.15.123:0/2051223323 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7efd0408d1a0 con 0x7efd181035e0 2024-03-30T04:54:06.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.452+0000 7efd1dc00700 1 --2- 172.21.15.123:0/2051223323 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7efd0006cc30 0x7efd0006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:54:06.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.452+0000 7efd1dc00700 1 --2- 172.21.15.123:0/2051223323 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7efd0006cc30 0x7efd0006f0e0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7efd0c004f40 tx=0x7efd0c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:54:06.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.455+0000 7efd15600700 1 -- 172.21.15.123:0/2051223323 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7efd0405ad20 con 0x7efd181035e0 2024-03-30T04:54:06.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.699+0000 7efd1f000700 1 -- 172.21.15.123:0/2051223323 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7efd18111140 con 0x7efd181035e0 2024-03-30T04:54:06.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.700+0000 7efd15600700 1 -- 172.21.15.123:0/2051223323 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 34 v34) v1 ==== 76+0+1843 (secure 0 0 0) 0x7efd04013420 con 0x7efd181035e0 2024-03-30T04:54:06.698 INFO:teuthology.orchestra.run.smithi123.stdout:e34 2024-03-30T04:54:06.698 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:54:06.698 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:54:06.698 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:54:06.699 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:54:06.699 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:54:06.699 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:54:06.699 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T04:54:06.699 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T04:54:06.699 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:54:06.699 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T04:54:06.699 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:54:06.699 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:54:06.699 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:54:06.699 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:54:06.699 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:54:06.699 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T04:54:06.699 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:54:06.700 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:54:06.700 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T04:54:06.700 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:54:06.700 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T04:54:06.700 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T04:54:06.700 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T04:54:06.700 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T04:54:06.700 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:54:06.700 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:54:06.700 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:54:06.700 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:54:06.700 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:54:06.700 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:54:06.700 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T04:54:06.700 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T04:54:06.701 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:54:06.701 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:54:06.701 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T04:54:06.701 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:54:06.701 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:54:06.701 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:54:06.701 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:54:06.701 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44975} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/3788988014,v1:172.21.15.135:6827/3788988014] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:54:06.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.702+0000 7efd1f000700 1 -- 172.21.15.123:0/2051223323 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7efd0006cc30 msgr2=0x7efd0006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:06.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.702+0000 7efd1f000700 1 --2- 172.21.15.123:0/2051223323 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7efd0006cc30 0x7efd0006f0e0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7efd0c004f40 tx=0x7efd0c00b040 comp rx=0 tx=0).stop 2024-03-30T04:54:06.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.702+0000 7efd1f000700 1 -- 172.21.15.123:0/2051223323 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd181035e0 msgr2=0x7efd18116030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:06.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.702+0000 7efd1f000700 1 --2- 172.21.15.123:0/2051223323 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd181035e0 0x7efd18116030 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7efd04009390 tx=0x7efd04009e30 comp rx=0 tx=0).stop 2024-03-30T04:54:06.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.702+0000 7efd1f000700 1 -- 172.21.15.123:0/2051223323 shutdown_connections 2024-03-30T04:54:06.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.702+0000 7efd1f000700 1 --2- 172.21.15.123:0/2051223323 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7efd0006cc30 0x7efd0006f0e0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:06.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.702+0000 7efd1f000700 1 --2- 172.21.15.123:0/2051223323 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efd18102c40 0x7efd18115af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:06.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.702+0000 7efd1f000700 1 --2- 172.21.15.123:0/2051223323 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd181035e0 0x7efd18116030 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:06.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.702+0000 7efd1f000700 1 -- 172.21.15.123:0/2051223323 >> 172.21.15.123:0/2051223323 conn(0x7efd180f8830 msgr2=0x7efd180f98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:54:06.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.702+0000 7efd1f000700 1 -- 172.21.15.123:0/2051223323 shutdown_connections 2024-03-30T04:54:06.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.702+0000 7efd1f000700 1 -- 172.21.15.123:0/2051223323 wait complete. 2024-03-30T04:54:06.703 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 34 2024-03-30T04:54:06.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.837+0000 7f369c600700 1 -- 172.21.15.123:0/3427470497 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3694070c10 msgr2=0x7f3694109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:06.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.837+0000 7f369c600700 1 --2- 172.21.15.123:0/3427470497 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3694070c10 0x7f3694109170 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f3680004970 tx=0x7f3680009e30 comp rx=0 tx=0).stop 2024-03-30T04:54:06.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.837+0000 7f369c600700 1 -- 172.21.15.123:0/3427470497 shutdown_connections 2024-03-30T04:54:06.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.837+0000 7f369c600700 1 --2- 172.21.15.123:0/3427470497 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3694070c10 0x7f3694109170 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:06.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.837+0000 7f369c600700 1 --2- 172.21.15.123:0/3427470497 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3694070300 0x7f36940706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:06.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.837+0000 7f369c600700 1 -- 172.21.15.123:0/3427470497 >> 172.21.15.123:0/3427470497 conn(0x7f36940f89b0 msgr2=0x7f36940fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:54:06.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.837+0000 7f369c600700 1 -- 172.21.15.123:0/3427470497 shutdown_connections 2024-03-30T04:54:06.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.837+0000 7f369c600700 1 -- 172.21.15.123:0/3427470497 wait complete. 2024-03-30T04:54:06.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.838+0000 7f369c600700 1 Processor -- start 2024-03-30T04:54:06.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.838+0000 7f369c600700 1 -- start start 2024-03-30T04:54:06.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.839+0000 7f369c600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3694070300 0x7f3694115a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:54:06.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.839+0000 7f369c600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3694070c10 0x7f3694115fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:54:06.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.839+0000 7f369c600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3694116610 con 0x7f3694070300 2024-03-30T04:54:06.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.839+0000 7f369c600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f369410fbf0 con 0x7f3694070c10 2024-03-30T04:54:06.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.839+0000 7f3699c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3694070300 0x7f3694115a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:54:06.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.839+0000 7f3699c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3694070300 0x7f3694115a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35080/0 (socket says 172.21.15.123:35080) 2024-03-30T04:54:06.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.839+0000 7f3699c00700 1 -- 172.21.15.123:0/4054442952 learned_addr learned my addr 172.21.15.123:0/4054442952 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:54:06.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.839+0000 7f3699200700 1 --2- 172.21.15.123:0/4054442952 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3694070c10 0x7f3694115fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:54:06.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.839+0000 7f3699c00700 1 -- 172.21.15.123:0/4054442952 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3694070c10 msgr2=0x7f3694115fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:06.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.839+0000 7f3699c00700 1 --2- 172.21.15.123:0/4054442952 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3694070c10 0x7f3694115fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:06.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.839+0000 7f3699c00700 1 -- 172.21.15.123:0/4054442952 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3680005040 con 0x7f3694070300 2024-03-30T04:54:06.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.839+0000 7f3699c00700 1 --2- 172.21.15.123:0/4054442952 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3694070300 0x7f3694115a80 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f368800ec90 tx=0x7f368800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:54:06.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.840+0000 7f3691600700 1 -- 172.21.15.123:0/4054442952 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3688007b30 con 0x7f3694070300 2024-03-30T04:54:06.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.840+0000 7f3691600700 1 -- 172.21.15.123:0/4054442952 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f368800f040 con 0x7f3694070300 2024-03-30T04:54:06.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.840+0000 7f369c600700 1 -- 172.21.15.123:0/4054442952 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f369410fed0 con 0x7f3694070300 2024-03-30T04:54:06.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.840+0000 7f3691600700 1 -- 172.21.15.123:0/4054442952 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f368800cbd0 con 0x7f3694070300 2024-03-30T04:54:06.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.840+0000 7f369c600700 1 -- 172.21.15.123:0/4054442952 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3694110410 con 0x7f3694070300 2024-03-30T04:54:06.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.841+0000 7f369c600700 1 -- 172.21.15.123:0/4054442952 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3694071a40 con 0x7f3694070300 2024-03-30T04:54:06.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.842+0000 7f3691600700 1 -- 172.21.15.123:0/4054442952 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f3688007c90 con 0x7f3694070300 2024-03-30T04:54:06.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.843+0000 7f3691600700 1 --2- 172.21.15.123:0/4054442952 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f367c06cc20 0x7f367c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:54:06.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.843+0000 7f3691600700 1 -- 172.21.15.123:0/4054442952 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f368808c4e0 con 0x7f3694070300 2024-03-30T04:54:06.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.843+0000 7f3699200700 1 --2- 172.21.15.123:0/4054442952 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f367c06cc20 0x7f367c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:54:06.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.844+0000 7f3699200700 1 --2- 172.21.15.123:0/4054442952 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f367c06cc20 0x7f367c06f0d0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f3694111320 tx=0x7f3680009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:54:06.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:06.847+0000 7f3691600700 1 -- 172.21.15.123:0/4054442952 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3688059f30 con 0x7f3694070300 2024-03-30T04:54:07.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.069+0000 7f369c600700 1 -- 172.21.15.123:0/4054442952 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3694111140 con 0x7f367c06cc20 2024-03-30T04:54:07.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.071+0000 7f3691600700 1 -- 172.21.15.123:0/4054442952 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f3694111140 con 0x7f367c06cc20 2024-03-30T04:54:07.069 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:54:07.069 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:54:07.069 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:54:07.069 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T04:54:07.069 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:54:07.069 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T04:54:07.069 INFO:teuthology.orchestra.run.smithi123.stdout: "mds", 2024-03-30T04:54:07.069 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T04:54:07.070 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T04:54:07.070 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T04:54:07.070 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:54:07.070 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "16/21 daemons upgraded", 2024-03-30T04:54:07.070 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T04:54:07.070 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:54:07.070 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:54:07.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.073+0000 7f369c600700 1 -- 172.21.15.123:0/4054442952 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f367c06cc20 msgr2=0x7f367c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:07.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.073+0000 7f369c600700 1 --2- 172.21.15.123:0/4054442952 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f367c06cc20 0x7f367c06f0d0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f3694111320 tx=0x7f3680009e30 comp rx=0 tx=0).stop 2024-03-30T04:54:07.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.073+0000 7f369c600700 1 -- 172.21.15.123:0/4054442952 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3694070300 msgr2=0x7f3694115a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:07.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.073+0000 7f369c600700 1 --2- 172.21.15.123:0/4054442952 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3694070300 0x7f3694115a80 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f368800ec90 tx=0x7f368800b040 comp rx=0 tx=0).stop 2024-03-30T04:54:07.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.073+0000 7f369c600700 1 -- 172.21.15.123:0/4054442952 shutdown_connections 2024-03-30T04:54:07.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.073+0000 7f369c600700 1 --2- 172.21.15.123:0/4054442952 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f367c06cc20 0x7f367c06f0d0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:07.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.073+0000 7f369c600700 1 --2- 172.21.15.123:0/4054442952 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3694070300 0x7f3694115a80 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:07.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.073+0000 7f369c600700 1 --2- 172.21.15.123:0/4054442952 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3694070c10 0x7f3694115fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:07.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.073+0000 7f369c600700 1 -- 172.21.15.123:0/4054442952 >> 172.21.15.123:0/4054442952 conn(0x7f36940f89b0 msgr2=0x7f3694105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:54:07.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.073+0000 7f369c600700 1 -- 172.21.15.123:0/4054442952 shutdown_connections 2024-03-30T04:54:07.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.073+0000 7f369c600700 1 -- 172.21.15.123:0/4054442952 wait complete. 2024-03-30T04:54:07.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:54:06 smithi135 ceph-mon[106011]: from='client.35272 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:54:07.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:54:06 smithi135 ceph-mon[106011]: pgmap v688: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:07.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:54:06 smithi135 ceph-mon[106011]: from='client.35276 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:54:07.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:54:06 smithi135 ceph-mon[106011]: from='client.35280 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:54:07.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:54:06 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2245738861' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:54:07.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:54:06 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2051223323' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:54:07.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.206+0000 7f055e000700 1 -- 172.21.15.123:0/1081464232 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0558102c40 msgr2=0x7f0558103010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:07.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.206+0000 7f055e000700 1 --2- 172.21.15.123:0/1081464232 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0558102c40 0x7f0558103010 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f054c004940 tx=0x7f054c009e30 comp rx=0 tx=0).stop 2024-03-30T04:54:07.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.206+0000 7f055e000700 1 -- 172.21.15.123:0/1081464232 shutdown_connections 2024-03-30T04:54:07.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.206+0000 7f055e000700 1 --2- 172.21.15.123:0/1081464232 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f05581035e0 0x7f05580fcc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:07.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.206+0000 7f055e000700 1 --2- 172.21.15.123:0/1081464232 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0558102c40 0x7f0558103010 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:07.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.206+0000 7f055e000700 1 -- 172.21.15.123:0/1081464232 >> 172.21.15.123:0/1081464232 conn(0x7f05580f8830 msgr2=0x7f05580fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:54:07.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.206+0000 7f055e000700 1 -- 172.21.15.123:0/1081464232 shutdown_connections 2024-03-30T04:54:07.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.206+0000 7f055e000700 1 -- 172.21.15.123:0/1081464232 wait complete. 2024-03-30T04:54:07.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.207+0000 7f055e000700 1 Processor -- start 2024-03-30T04:54:07.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.207+0000 7f055e000700 1 -- start start 2024-03-30T04:54:07.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.207+0000 7f055e000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0558102c40 0x7f0558113950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:54:07.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.207+0000 7f055e000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05581035e0 0x7f0558113e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:54:07.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.207+0000 7f055e000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05581144e0 con 0x7f05581035e0 2024-03-30T04:54:07.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.207+0000 7f055e000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f055810d8d0 con 0x7f0558102c40 2024-03-30T04:54:07.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.208+0000 7f0557e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05581035e0 0x7f0558113e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:54:07.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.208+0000 7f0557e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05581035e0 0x7f0558113e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35102/0 (socket says 172.21.15.123:35102) 2024-03-30T04:54:07.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.208+0000 7f0557e00700 1 -- 172.21.15.123:0/4046481218 learned_addr learned my addr 172.21.15.123:0/4046481218 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:54:07.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.208+0000 7f0557e00700 1 -- 172.21.15.123:0/4046481218 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0558102c40 msgr2=0x7f0558113950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:07.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.208+0000 7f055cc00700 1 --2- 172.21.15.123:0/4046481218 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0558102c40 0x7f0558113950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:54:07.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.208+0000 7f0557e00700 1 --2- 172.21.15.123:0/4046481218 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0558102c40 0x7f0558113950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:07.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.208+0000 7f0557e00700 1 -- 172.21.15.123:0/4046481218 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f054c005040 con 0x7f05581035e0 2024-03-30T04:54:07.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.208+0000 7f055cc00700 1 --2- 172.21.15.123:0/4046481218 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0558102c40 0x7f0558113950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:54:07.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.208+0000 7f0557e00700 1 --2- 172.21.15.123:0/4046481218 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05581035e0 0x7f0558113e90 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f054400c890 tx=0x7f05440044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:54:07.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.208+0000 7f0554c00700 1 -- 172.21.15.123:0/4046481218 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f054400a7d0 con 0x7f05581035e0 2024-03-30T04:54:07.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.208+0000 7f0554c00700 1 -- 172.21.15.123:0/4046481218 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f054400ade0 con 0x7f05581035e0 2024-03-30T04:54:07.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.208+0000 7f0554c00700 1 -- 172.21.15.123:0/4046481218 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f05440113a0 con 0x7f05581035e0 2024-03-30T04:54:07.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.208+0000 7f055e000700 1 -- 172.21.15.123:0/4046481218 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f055810dbb0 con 0x7f05581035e0 2024-03-30T04:54:07.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.209+0000 7f055e000700 1 -- 172.21.15.123:0/4046481218 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f055810e170 con 0x7f05581035e0 2024-03-30T04:54:07.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.210+0000 7f055e000700 1 -- 172.21.15.123:0/4046481218 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05580fe390 con 0x7f05581035e0 2024-03-30T04:54:07.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.211+0000 7f0554c00700 1 -- 172.21.15.123:0/4046481218 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f0544012400 con 0x7f05581035e0 2024-03-30T04:54:07.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.212+0000 7f0554c00700 1 --2- 172.21.15.123:0/4046481218 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f054006cc20 0x7f054006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:54:07.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.212+0000 7f0554c00700 1 -- 172.21.15.123:0/4046481218 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f054408c0d0 con 0x7f05581035e0 2024-03-30T04:54:07.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.212+0000 7f055cc00700 1 --2- 172.21.15.123:0/4046481218 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f054006cc20 0x7f054006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:54:07.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.213+0000 7f055cc00700 1 --2- 172.21.15.123:0/4046481218 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f054006cc20 0x7f054006f0d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f054c004560 tx=0x7f054c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:54:07.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.216+0000 7f0554c00700 1 -- 172.21.15.123:0/4046481218 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0544059ba0 con 0x7f05581035e0 2024-03-30T04:54:07.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:54:06 smithi123 ceph-mon[139369]: from='client.35272 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:54:07.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:54:06 smithi123 ceph-mon[139369]: pgmap v688: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:07.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:54:06 smithi123 ceph-mon[139369]: from='client.35276 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:54:07.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:54:06 smithi123 ceph-mon[139369]: from='client.35280 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:54:07.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:54:06 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2245738861' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:54:07.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:54:06 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2051223323' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:54:07.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.496+0000 7f055e000700 1 -- 172.21.15.123:0/4046481218 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f0558109bf0 con 0x7f05581035e0 2024-03-30T04:54:07.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.496+0000 7f0554c00700 1 -- 172.21.15.123:0/4046481218 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f0544059730 con 0x7f05581035e0 2024-03-30T04:54:07.495 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T04:54:07.495 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-30T04:54:07.495 INFO:teuthology.orchestra.run.smithi123.stdout: host smithi135 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi135/config 2024-03-30T04:54:07.495 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-30T04:54:07.495 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T04:54:07.495 INFO:teuthology.orchestra.run.smithi123.stdout:Traceback (most recent call last): 2024-03-30T04:54:07.495 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-30T04:54:07.495 INFO:teuthology.orchestra.run.smithi123.stdout: "__main__", mod_spec) 2024-03-30T04:54:07.495 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-30T04:54:07.495 INFO:teuthology.orchestra.run.smithi123.stdout: exec(code, run_globals) 2024-03-30T04:54:07.496 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10889, in 2024-03-30T04:54:07.496 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10877, in main 2024-03-30T04:54:07.496 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-30T04:54:07.496 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-30T04:54:07.496 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-30T04:54:07.496 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-30T04:54:07.496 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-03-30T04:54:07.496 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-30T04:54:07.496 INFO:teuthology.orchestra.run.smithi123.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T04:54:07.496 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T04:54:07.497 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T04:54:07.497 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T04:54:07.497 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T04:54:07.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.498+0000 7f055e000700 1 -- 172.21.15.123:0/4046481218 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f054006cc20 msgr2=0x7f054006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:07.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.498+0000 7f055e000700 1 --2- 172.21.15.123:0/4046481218 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f054006cc20 0x7f054006f0d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f054c004560 tx=0x7f054c009e30 comp rx=0 tx=0).stop 2024-03-30T04:54:07.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.498+0000 7f055e000700 1 -- 172.21.15.123:0/4046481218 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05581035e0 msgr2=0x7f0558113e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:07.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.498+0000 7f055e000700 1 --2- 172.21.15.123:0/4046481218 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05581035e0 0x7f0558113e90 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f054400c890 tx=0x7f05440044c0 comp rx=0 tx=0).stop 2024-03-30T04:54:07.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.498+0000 7f055e000700 1 -- 172.21.15.123:0/4046481218 shutdown_connections 2024-03-30T04:54:07.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.498+0000 7f055e000700 1 --2- 172.21.15.123:0/4046481218 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f054006cc20 0x7f054006f0d0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:07.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.498+0000 7f055e000700 1 --2- 172.21.15.123:0/4046481218 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0558102c40 0x7f0558113950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:07.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.498+0000 7f055e000700 1 --2- 172.21.15.123:0/4046481218 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05581035e0 0x7f0558113e90 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:07.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.498+0000 7f055e000700 1 -- 172.21.15.123:0/4046481218 >> 172.21.15.123:0/4046481218 conn(0x7f05580f8830 msgr2=0x7f05580f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:54:07.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.499+0000 7f055e000700 1 -- 172.21.15.123:0/4046481218 shutdown_connections 2024-03-30T04:54:07.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:07.499+0000 7f055e000700 1 -- 172.21.15.123:0/4046481218 wait complete. 2024-03-30T04:54:08.057 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:54:07 smithi123 ceph-mon[139369]: from='client.35292 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:54:08.058 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:54:07 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/4046481218' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:54:08.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:54:07 smithi135 ceph-mon[106011]: from='client.35292 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:54:08.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:54:07 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/4046481218' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:54:09.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:54:08 smithi135 ceph-mon[106011]: pgmap v689: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:09.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:54:08 smithi123 ceph-mon[139369]: pgmap v689: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:11.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:54:10 smithi135 ceph-mon[106011]: pgmap v690: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:11.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:54:10 smithi123 ceph-mon[139369]: pgmap v690: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:13.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:54:12 smithi135 ceph-mon[106011]: pgmap v691: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:13.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:54:12 smithi123 ceph-mon[139369]: pgmap v691: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:15.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:54:14 smithi135 ceph-mon[106011]: pgmap v692: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:15.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:54:14 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:54:15.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:54:14 smithi123 ceph-mon[139369]: pgmap v692: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:15.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:54:14 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:54:17.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:54:16 smithi135 ceph-mon[106011]: pgmap v693: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:17.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:54:16 smithi123 ceph-mon[139369]: pgmap v693: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:19.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:54:18 smithi135 ceph-mon[106011]: pgmap v694: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:19.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:54:18 smithi123 ceph-mon[139369]: pgmap v694: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:21.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:54:20 smithi135 ceph-mon[106011]: pgmap v695: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:21.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:54:20 smithi123 ceph-mon[139369]: pgmap v695: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:23.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:54:22 smithi135 ceph-mon[106011]: pgmap v696: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:23.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:54:22 smithi123 ceph-mon[139369]: pgmap v696: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:25.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:54:24 smithi135 ceph-mon[106011]: pgmap v697: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:25.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:54:24 smithi123 ceph-mon[139369]: pgmap v697: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:27.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:54:26 smithi135 ceph-mon[106011]: pgmap v698: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:27.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:54:26 smithi123 ceph-mon[139369]: pgmap v698: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:29.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:54:28 smithi135 ceph-mon[106011]: pgmap v699: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:29.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:54:28 smithi123 ceph-mon[139369]: pgmap v699: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:30.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:54:29 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:54:30.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:54:29 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:54:31.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:54:30 smithi135 ceph-mon[106011]: pgmap v700: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:31.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:54:30 smithi123 ceph-mon[139369]: pgmap v700: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:33.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:54:32 smithi135 ceph-mon[106011]: pgmap v701: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:33.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:54:32 smithi123 ceph-mon[139369]: pgmap v701: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:35.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:54:34 smithi135 ceph-mon[106011]: pgmap v702: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:35.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:54:34 smithi123 ceph-mon[139369]: pgmap v702: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:37.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:54:36 smithi135 ceph-mon[106011]: pgmap v703: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:37.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:54:36 smithi123 ceph-mon[139369]: pgmap v703: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:37.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:37.645+0000 7f3c4b200700 1 -- 172.21.15.123:0/4112488439 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3c4c1035e0 msgr2=0x7f3c4c0fcc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:37.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:37.645+0000 7f3c4b200700 1 --2- 172.21.15.123:0/4112488439 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3c4c1035e0 0x7f3c4c0fcc50 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f3c38004940 tx=0x7f3c38009e30 comp rx=0 tx=0).stop 2024-03-30T04:54:37.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:37.645+0000 7f3c4b200700 1 -- 172.21.15.123:0/4112488439 shutdown_connections 2024-03-30T04:54:37.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:37.645+0000 7f3c4b200700 1 --2- 172.21.15.123:0/4112488439 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3c4c1035e0 0x7f3c4c0fcc50 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:37.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:37.645+0000 7f3c4b200700 1 --2- 172.21.15.123:0/4112488439 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3c4c102c40 0x7f3c4c103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:37.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:37.645+0000 7f3c4b200700 1 -- 172.21.15.123:0/4112488439 >> 172.21.15.123:0/4112488439 conn(0x7f3c4c0f8830 msgr2=0x7f3c4c0fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:54:37.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:37.646+0000 7f3c4b200700 1 -- 172.21.15.123:0/4112488439 shutdown_connections 2024-03-30T04:54:37.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:37.646+0000 7f3c4b200700 1 -- 172.21.15.123:0/4112488439 wait complete. 2024-03-30T04:54:37.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:37.646+0000 7f3c4b200700 1 Processor -- start 2024-03-30T04:54:37.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:37.647+0000 7f3c4b200700 1 -- start start 2024-03-30T04:54:37.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:37.647+0000 7f3c4b200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3c4c102c40 0x7f3c4c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:54:37.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:37.647+0000 7f3c4b200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3c4c1035e0 0x7f3c4c070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:54:37.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:37.647+0000 7f3c4b200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c4c070d90 con 0x7f3c4c1035e0 2024-03-30T04:54:37.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:37.647+0000 7f3c4b200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c4c070ed0 con 0x7f3c4c102c40 2024-03-30T04:54:37.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:37.647+0000 7f3c49400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3c4c1035e0 0x7f3c4c070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:54:37.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:37.647+0000 7f3c49400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3c4c1035e0 0x7f3c4c070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42948/0 (socket says 172.21.15.123:42948) 2024-03-30T04:54:37.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:37.647+0000 7f3c49400700 1 -- 172.21.15.123:0/3521582026 learned_addr learned my addr 172.21.15.123:0/3521582026 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:54:37.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:37.647+0000 7f3c49e00700 1 --2- 172.21.15.123:0/3521582026 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3c4c102c40 0x7f3c4c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:54:37.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:37.648+0000 7f3c49400700 1 -- 172.21.15.123:0/3521582026 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3c4c102c40 msgr2=0x7f3c4c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:37.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:37.648+0000 7f3c49400700 1 --2- 172.21.15.123:0/3521582026 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3c4c102c40 0x7f3c4c0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:37.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:37.648+0000 7f3c49400700 1 -- 172.21.15.123:0/3521582026 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c38005040 con 0x7f3c4c1035e0 2024-03-30T04:54:37.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:37.648+0000 7f3c49400700 1 --2- 172.21.15.123:0/3521582026 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3c4c1035e0 0x7f3c4c070800 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f3c380093b0 tx=0x7f3c38009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:54:37.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:37.648+0000 7f3c42000700 1 -- 172.21.15.123:0/3521582026 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c3800e040 con 0x7f3c4c1035e0 2024-03-30T04:54:37.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:37.648+0000 7f3c42000700 1 -- 172.21.15.123:0/3521582026 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3c38013420 con 0x7f3c4c1035e0 2024-03-30T04:54:37.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:37.648+0000 7f3c4b200700 1 -- 172.21.15.123:0/3521582026 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c4c071120 con 0x7f3c4c1035e0 2024-03-30T04:54:37.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:37.648+0000 7f3c42000700 1 -- 172.21.15.123:0/3521582026 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c3801c630 con 0x7f3c4c1035e0 2024-03-30T04:54:37.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:37.648+0000 7f3c4b200700 1 -- 172.21.15.123:0/3521582026 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c4c11a010 con 0x7f3c4c1035e0 2024-03-30T04:54:37.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:37.649+0000 7f3c4b200700 1 -- 172.21.15.123:0/3521582026 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c4c0fe390 con 0x7f3c4c1035e0 2024-03-30T04:54:37.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:37.650+0000 7f3c42000700 1 -- 172.21.15.123:0/3521582026 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f3c380139e0 con 0x7f3c4c1035e0 2024-03-30T04:54:37.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:37.651+0000 7f3c42000700 1 --2- 172.21.15.123:0/3521582026 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3c3006cc20 0x7f3c3006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:54:37.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:37.651+0000 7f3c42000700 1 -- 172.21.15.123:0/3521582026 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f3c3808d1c0 con 0x7f3c4c1035e0 2024-03-30T04:54:37.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:37.651+0000 7f3c49e00700 1 --2- 172.21.15.123:0/3521582026 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3c3006cc20 0x7f3c3006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:54:37.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:37.652+0000 7f3c49e00700 1 --2- 172.21.15.123:0/3521582026 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3c3006cc20 0x7f3c3006f0d0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c007fd0 tx=0x7f3c3c009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:54:37.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:37.656+0000 7f3c42000700 1 -- 172.21.15.123:0/3521582026 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3c380572b0 con 0x7f3c4c1035e0 2024-03-30T04:54:37.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:37.877+0000 7f3c4b200700 1 -- 172.21.15.123:0/3521582026 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3c4c109c40 con 0x7f3c3006cc20 2024-03-30T04:54:37.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:37.879+0000 7f3c42000700 1 -- 172.21.15.123:0/3521582026 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f3c4c109c40 con 0x7f3c3006cc20 2024-03-30T04:54:37.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:37.881+0000 7f3c4b200700 1 -- 172.21.15.123:0/3521582026 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3c3006cc20 msgr2=0x7f3c3006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:37.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:37.881+0000 7f3c4b200700 1 --2- 172.21.15.123:0/3521582026 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3c3006cc20 0x7f3c3006f0d0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c007fd0 tx=0x7f3c3c009040 comp rx=0 tx=0).stop 2024-03-30T04:54:37.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:37.881+0000 7f3c4b200700 1 -- 172.21.15.123:0/3521582026 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3c4c1035e0 msgr2=0x7f3c4c070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:37.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:37.881+0000 7f3c4b200700 1 --2- 172.21.15.123:0/3521582026 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3c4c1035e0 0x7f3c4c070800 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f3c380093b0 tx=0x7f3c38009e30 comp rx=0 tx=0).stop 2024-03-30T04:54:37.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:37.881+0000 7f3c4b200700 1 -- 172.21.15.123:0/3521582026 shutdown_connections 2024-03-30T04:54:37.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:37.881+0000 7f3c4b200700 1 --2- 172.21.15.123:0/3521582026 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3c3006cc20 0x7f3c3006f0d0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:37.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:37.881+0000 7f3c4b200700 1 --2- 172.21.15.123:0/3521582026 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3c4c102c40 0x7f3c4c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:37.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:37.881+0000 7f3c4b200700 1 --2- 172.21.15.123:0/3521582026 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3c4c1035e0 0x7f3c4c070800 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:37.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:37.881+0000 7f3c4b200700 1 -- 172.21.15.123:0/3521582026 >> 172.21.15.123:0/3521582026 conn(0x7f3c4c0f8830 msgr2=0x7f3c4c0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:54:37.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:37.881+0000 7f3c4b200700 1 -- 172.21.15.123:0/3521582026 shutdown_connections 2024-03-30T04:54:37.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:37.882+0000 7f3c4b200700 1 -- 172.21.15.123:0/3521582026 wait complete. 2024-03-30T04:54:37.894 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:54:38.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.015+0000 7f4094200700 1 -- 172.21.15.123:0/248653986 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f408c102e00 msgr2=0x7f408c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:38.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.015+0000 7f4094200700 1 --2- 172.21.15.123:0/248653986 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f408c102e00 0x7f408c1031d0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f4080004940 tx=0x7f4080009e30 comp rx=0 tx=0).stop 2024-03-30T04:54:38.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.015+0000 7f4094200700 1 -- 172.21.15.123:0/248653986 shutdown_connections 2024-03-30T04:54:38.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.015+0000 7f4094200700 1 --2- 172.21.15.123:0/248653986 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f408c1037a0 0x7f408c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:38.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.015+0000 7f4094200700 1 --2- 172.21.15.123:0/248653986 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f408c102e00 0x7f408c1031d0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:38.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.015+0000 7f4094200700 1 -- 172.21.15.123:0/248653986 >> 172.21.15.123:0/248653986 conn(0x7f408c0f89b0 msgr2=0x7f408c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:54:38.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.015+0000 7f4094200700 1 -- 172.21.15.123:0/248653986 shutdown_connections 2024-03-30T04:54:38.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.015+0000 7f4094200700 1 -- 172.21.15.123:0/248653986 wait complete. 2024-03-30T04:54:38.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.016+0000 7f4094200700 1 Processor -- start 2024-03-30T04:54:38.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.016+0000 7f4094200700 1 -- start start 2024-03-30T04:54:38.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.016+0000 7f4094200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f408c102e00 0x7f408c115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:54:38.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.016+0000 7f4094200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f408c1037a0 0x7f408c116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:54:38.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.016+0000 7f4094200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f408c116860 con 0x7f408c102e00 2024-03-30T04:54:38.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.016+0000 7f4094200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f408c10fc30 con 0x7f408c1037a0 2024-03-30T04:54:38.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.016+0000 7f4091800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f408c102e00 0x7f408c115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:54:38.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.016+0000 7f4091800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f408c102e00 0x7f408c115cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42974/0 (socket says 172.21.15.123:42974) 2024-03-30T04:54:38.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.016+0000 7f4091800700 1 -- 172.21.15.123:0/3019708579 learned_addr learned my addr 172.21.15.123:0/3019708579 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:54:38.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.017+0000 7f4090e00700 1 --2- 172.21.15.123:0/3019708579 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f408c1037a0 0x7f408c116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:54:38.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.017+0000 7f4091800700 1 -- 172.21.15.123:0/3019708579 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f408c1037a0 msgr2=0x7f408c116210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:38.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.017+0000 7f4091800700 1 --2- 172.21.15.123:0/3019708579 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f408c1037a0 0x7f408c116210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:38.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.017+0000 7f4091800700 1 -- 172.21.15.123:0/3019708579 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4080005040 con 0x7f408c102e00 2024-03-30T04:54:38.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.017+0000 7f4091800700 1 --2- 172.21.15.123:0/3019708579 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f408c102e00 0x7f408c115cd0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f4080009390 tx=0x7f4080009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:54:38.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.017+0000 7f4089600700 1 -- 172.21.15.123:0/3019708579 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f408000e040 con 0x7f408c102e00 2024-03-30T04:54:38.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.017+0000 7f4089600700 1 -- 172.21.15.123:0/3019708579 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4080013420 con 0x7f408c102e00 2024-03-30T04:54:38.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.017+0000 7f4089600700 1 -- 172.21.15.123:0/3019708579 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f408001c630 con 0x7f408c102e00 2024-03-30T04:54:38.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.017+0000 7f4094200700 1 -- 172.21.15.123:0/3019708579 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f408c10feb0 con 0x7f408c102e00 2024-03-30T04:54:38.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.018+0000 7f4094200700 1 -- 172.21.15.123:0/3019708579 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f408c110420 con 0x7f408c102e00 2024-03-30T04:54:38.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.019+0000 7f4094200700 1 -- 172.21.15.123:0/3019708579 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f408c0fe550 con 0x7f408c102e00 2024-03-30T04:54:38.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.019+0000 7f4089600700 1 -- 172.21.15.123:0/3019708579 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f40800139e0 con 0x7f408c102e00 2024-03-30T04:54:38.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.020+0000 7f4089600700 1 --2- 172.21.15.123:0/3019708579 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f407406cc30 0x7f407406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:54:38.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.020+0000 7f4089600700 1 -- 172.21.15.123:0/3019708579 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f408008d3f0 con 0x7f408c102e00 2024-03-30T04:54:38.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.020+0000 7f4090e00700 1 --2- 172.21.15.123:0/3019708579 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f407406cc30 0x7f407406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:54:38.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.021+0000 7f4090e00700 1 --2- 172.21.15.123:0/3019708579 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f407406cc30 0x7f407406f0e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f408c111570 tx=0x7f407800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:54:38.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.024+0000 7f4089600700 1 -- 172.21.15.123:0/3019708579 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f408005ae40 con 0x7f408c102e00 2024-03-30T04:54:38.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.246+0000 7f4094200700 1 -- 172.21.15.123:0/3019708579 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f408c04bed0 con 0x7f407406cc30 2024-03-30T04:54:38.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.248+0000 7f4089600700 1 -- 172.21.15.123:0/3019708579 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f408c04bed0 con 0x7f407406cc30 2024-03-30T04:54:38.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.250+0000 7f4094200700 1 -- 172.21.15.123:0/3019708579 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f407406cc30 msgr2=0x7f407406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:38.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.250+0000 7f4094200700 1 --2- 172.21.15.123:0/3019708579 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f407406cc30 0x7f407406f0e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f408c111570 tx=0x7f407800b040 comp rx=0 tx=0).stop 2024-03-30T04:54:38.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.250+0000 7f4094200700 1 -- 172.21.15.123:0/3019708579 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f408c102e00 msgr2=0x7f408c115cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:38.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.250+0000 7f4094200700 1 --2- 172.21.15.123:0/3019708579 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f408c102e00 0x7f408c115cd0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f4080009390 tx=0x7f4080009e30 comp rx=0 tx=0).stop 2024-03-30T04:54:38.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.250+0000 7f4094200700 1 -- 172.21.15.123:0/3019708579 shutdown_connections 2024-03-30T04:54:38.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.250+0000 7f4094200700 1 --2- 172.21.15.123:0/3019708579 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f407406cc30 0x7f407406f0e0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:38.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.250+0000 7f4094200700 1 --2- 172.21.15.123:0/3019708579 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f408c102e00 0x7f408c115cd0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:38.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.250+0000 7f4094200700 1 --2- 172.21.15.123:0/3019708579 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f408c1037a0 0x7f408c116210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:38.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.250+0000 7f4094200700 1 -- 172.21.15.123:0/3019708579 >> 172.21.15.123:0/3019708579 conn(0x7f408c0f89b0 msgr2=0x7f408c0f99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:54:38.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.250+0000 7f4094200700 1 -- 172.21.15.123:0/3019708579 shutdown_connections 2024-03-30T04:54:38.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.250+0000 7f4094200700 1 -- 172.21.15.123:0/3019708579 wait complete. 2024-03-30T04:54:38.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.384+0000 7f4ce3400700 1 -- 172.21.15.123:0/366582324 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4cdc102e00 msgr2=0x7f4cdc1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:38.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.384+0000 7f4ce3400700 1 --2- 172.21.15.123:0/366582324 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4cdc102e00 0x7f4cdc1031d0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f4cd0004940 tx=0x7f4cd0009e30 comp rx=0 tx=0).stop 2024-03-30T04:54:38.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.384+0000 7f4ce3400700 1 -- 172.21.15.123:0/366582324 shutdown_connections 2024-03-30T04:54:38.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.384+0000 7f4ce3400700 1 --2- 172.21.15.123:0/366582324 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4cdc1037a0 0x7f4cdc0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:38.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.384+0000 7f4ce3400700 1 --2- 172.21.15.123:0/366582324 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4cdc102e00 0x7f4cdc1031d0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:38.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.384+0000 7f4ce3400700 1 -- 172.21.15.123:0/366582324 >> 172.21.15.123:0/366582324 conn(0x7f4cdc0f89b0 msgr2=0x7f4cdc0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:54:38.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.384+0000 7f4ce3400700 1 -- 172.21.15.123:0/366582324 shutdown_connections 2024-03-30T04:54:38.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.384+0000 7f4ce3400700 1 -- 172.21.15.123:0/366582324 wait complete. 2024-03-30T04:54:38.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.385+0000 7f4ce3400700 1 Processor -- start 2024-03-30T04:54:38.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.385+0000 7f4ce3400700 1 -- start start 2024-03-30T04:54:38.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.385+0000 7f4ce3400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4cdc102e00 0x7f4cdc115c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:54:38.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.385+0000 7f4ce3400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4cdc1037a0 0x7f4cdc116180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:54:38.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.385+0000 7f4ce3400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cdc1167d0 con 0x7f4cdc1037a0 2024-03-30T04:54:38.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.385+0000 7f4ce3400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cdc10fba0 con 0x7f4cdc102e00 2024-03-30T04:54:38.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.386+0000 7f4cdbe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4cdc1037a0 0x7f4cdc116180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:54:38.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.386+0000 7f4cdbe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4cdc1037a0 0x7f4cdc116180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42980/0 (socket says 172.21.15.123:42980) 2024-03-30T04:54:38.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.386+0000 7f4cdbe00700 1 -- 172.21.15.123:0/459894102 learned_addr learned my addr 172.21.15.123:0/459894102 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:54:38.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.386+0000 7f4ce0a00700 1 --2- 172.21.15.123:0/459894102 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4cdc102e00 0x7f4cdc115c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:54:38.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.386+0000 7f4cdbe00700 1 -- 172.21.15.123:0/459894102 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4cdc102e00 msgr2=0x7f4cdc115c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:38.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.386+0000 7f4cdbe00700 1 --2- 172.21.15.123:0/459894102 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4cdc102e00 0x7f4cdc115c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:38.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.386+0000 7f4cdbe00700 1 -- 172.21.15.123:0/459894102 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd0005040 con 0x7f4cdc1037a0 2024-03-30T04:54:38.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.386+0000 7f4ce0a00700 1 --2- 172.21.15.123:0/459894102 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4cdc102e00 0x7f4cdc115c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:54:38.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.386+0000 7f4cdbe00700 1 --2- 172.21.15.123:0/459894102 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4cdc1037a0 0x7f4cdc116180 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f4cc800eb40 tx=0x7f4cc800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:54:38.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.386+0000 7f4cd8c00700 1 -- 172.21.15.123:0/459894102 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4cc8007cc0 con 0x7f4cdc1037a0 2024-03-30T04:54:38.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.386+0000 7f4cd8c00700 1 -- 172.21.15.123:0/459894102 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4cc800f040 con 0x7f4cdc1037a0 2024-03-30T04:54:38.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.386+0000 7f4ce3400700 1 -- 172.21.15.123:0/459894102 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cdc10fe20 con 0x7f4cdc1037a0 2024-03-30T04:54:38.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.387+0000 7f4cd8c00700 1 -- 172.21.15.123:0/459894102 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4cc8009570 con 0x7f4cdc1037a0 2024-03-30T04:54:38.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.387+0000 7f4ce3400700 1 -- 172.21.15.123:0/459894102 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cdc1103e0 con 0x7f4cdc1037a0 2024-03-30T04:54:38.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.388+0000 7f4ce3400700 1 -- 172.21.15.123:0/459894102 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cdc0fe550 con 0x7f4cdc1037a0 2024-03-30T04:54:38.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.389+0000 7f4cd8c00700 1 -- 172.21.15.123:0/459894102 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f4cc80096d0 con 0x7f4cdc1037a0 2024-03-30T04:54:38.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.391+0000 7f4cd8c00700 1 --2- 172.21.15.123:0/459894102 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4cc406cc30 0x7f4cc406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:54:38.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.391+0000 7f4cd8c00700 1 -- 172.21.15.123:0/459894102 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f4cc808bb00 con 0x7f4cdc1037a0 2024-03-30T04:54:38.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.391+0000 7f4ce0a00700 1 --2- 172.21.15.123:0/459894102 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4cc406cc30 0x7f4cc406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:54:38.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.391+0000 7f4ce0a00700 1 --2- 172.21.15.123:0/459894102 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4cc406cc30 0x7f4cc406f0e0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f4cd000b0d0 tx=0x7f4cd0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:54:38.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.395+0000 7f4cd8c00700 1 -- 172.21.15.123:0/459894102 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4cc8059600 con 0x7f4cdc1037a0 2024-03-30T04:54:38.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.606+0000 7f4ce3400700 1 -- 172.21.15.123:0/459894102 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4cdc109e30 con 0x7f4cc406cc30 2024-03-30T04:54:38.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.616+0000 7f4cd8c00700 1 -- 172.21.15.123:0/459894102 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f4cdc109e30 con 0x7f4cc406cc30 2024-03-30T04:54:38.614 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:54:38.614 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (23m) 71s ago 36m 14.6M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:54:38.614 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (21m) 71s ago 36m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T04:54:38.614 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (20m) 74s ago 35m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T04:54:38.615 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (23m) 71s ago 36m 73.5M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T04:54:38.615 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (16m) 71s ago 29m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T04:54:38.615 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (17m) 71s ago 30m 16.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T04:54:38.615 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (77s) 74s ago 29m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 87b073a6c9ca 2024-03-30T04:54:38.615 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (16m) 74s ago 30m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T04:54:38.615 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (26m) 71s ago 37m 510M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:54:38.615 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (25m) 74s ago 35m 456M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:54:38.615 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (22m) 71s ago 38m 70.0M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T04:54:38.615 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (21m) 74s ago 35m 51.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T04:54:38.615 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (24m) 71s ago 36m 8636k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:54:38.615 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (24m) 74s ago 35m 8678k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:54:38.616 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (20m) 71s ago 34m 107M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T04:54:38.616 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (20m) 71s ago 33m 97.2M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T04:54:38.616 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (19m) 71s ago 33m 89.8M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T04:54:38.616 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (18m) 74s ago 32m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T04:54:38.616 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (18m) 74s ago 32m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T04:54:38.616 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (17m) 74s ago 31m 88.8M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T04:54:38.616 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (24m) 71s ago 35m 63.9M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:54:38.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.618+0000 7f4ce3400700 1 -- 172.21.15.123:0/459894102 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4cc406cc30 msgr2=0x7f4cc406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:38.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.618+0000 7f4ce3400700 1 --2- 172.21.15.123:0/459894102 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4cc406cc30 0x7f4cc406f0e0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f4cd000b0d0 tx=0x7f4cd0009e30 comp rx=0 tx=0).stop 2024-03-30T04:54:38.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.618+0000 7f4ce3400700 1 -- 172.21.15.123:0/459894102 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4cdc1037a0 msgr2=0x7f4cdc116180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:38.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.618+0000 7f4ce3400700 1 --2- 172.21.15.123:0/459894102 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4cdc1037a0 0x7f4cdc116180 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f4cc800eb40 tx=0x7f4cc800b040 comp rx=0 tx=0).stop 2024-03-30T04:54:38.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.618+0000 7f4ce3400700 1 -- 172.21.15.123:0/459894102 shutdown_connections 2024-03-30T04:54:38.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.618+0000 7f4ce3400700 1 --2- 172.21.15.123:0/459894102 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4cc406cc30 0x7f4cc406f0e0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:38.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.618+0000 7f4ce3400700 1 --2- 172.21.15.123:0/459894102 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4cdc102e00 0x7f4cdc115c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:38.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.618+0000 7f4ce3400700 1 --2- 172.21.15.123:0/459894102 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4cdc1037a0 0x7f4cdc116180 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:38.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.618+0000 7f4ce3400700 1 -- 172.21.15.123:0/459894102 >> 172.21.15.123:0/459894102 conn(0x7f4cdc0f89b0 msgr2=0x7f4cdc0f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:54:38.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.618+0000 7f4ce3400700 1 -- 172.21.15.123:0/459894102 shutdown_connections 2024-03-30T04:54:38.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.618+0000 7f4ce3400700 1 -- 172.21.15.123:0/459894102 wait complete. 2024-03-30T04:54:38.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.750+0000 7f1115a00700 1 -- 172.21.15.123:0/1023114926 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11100fde90 msgr2=0x7f11100fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:38.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.750+0000 7f1115a00700 1 --2- 172.21.15.123:0/1023114926 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11100fde90 0x7f11100fe260 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f1100004940 tx=0x7f1100009e30 comp rx=0 tx=0).stop 2024-03-30T04:54:38.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.750+0000 7f1115a00700 1 -- 172.21.15.123:0/1023114926 shutdown_connections 2024-03-30T04:54:38.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.750+0000 7f1115a00700 1 --2- 172.21.15.123:0/1023114926 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f11100fe830 0x7f1110106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:38.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.750+0000 7f1115a00700 1 --2- 172.21.15.123:0/1023114926 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11100fde90 0x7f11100fe260 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:38.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.750+0000 7f1115a00700 1 -- 172.21.15.123:0/1023114926 >> 172.21.15.123:0/1023114926 conn(0x7f1110072bc0 msgr2=0x7f1110072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:54:38.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.750+0000 7f1115a00700 1 -- 172.21.15.123:0/1023114926 shutdown_connections 2024-03-30T04:54:38.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.750+0000 7f1115a00700 1 -- 172.21.15.123:0/1023114926 wait complete. 2024-03-30T04:54:38.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.751+0000 7f1115a00700 1 Processor -- start 2024-03-30T04:54:38.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.751+0000 7f1115a00700 1 -- start start 2024-03-30T04:54:38.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.752+0000 7f1115a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11100fde90 0x7f1110113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:54:38.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.752+0000 7f1115a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f11100fe830 0x7f1110113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:54:38.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.752+0000 7f1115a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1110114600 con 0x7f11100fde90 2024-03-30T04:54:38.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.752+0000 7f1115a00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f111010d9f0 con 0x7f11100fe830 2024-03-30T04:54:38.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.752+0000 7f110ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11100fde90 0x7f1110113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:54:38.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.752+0000 7f110ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11100fde90 0x7f1110113a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43004/0 (socket says 172.21.15.123:43004) 2024-03-30T04:54:38.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.752+0000 7f110ea00700 1 -- 172.21.15.123:0/3702912646 learned_addr learned my addr 172.21.15.123:0/3702912646 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:54:38.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.752+0000 7f110e000700 1 --2- 172.21.15.123:0/3702912646 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f11100fe830 0x7f1110113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:54:38.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.752+0000 7f110ea00700 1 -- 172.21.15.123:0/3702912646 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f11100fe830 msgr2=0x7f1110113fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:38.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.752+0000 7f110ea00700 1 --2- 172.21.15.123:0/3702912646 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f11100fe830 0x7f1110113fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:38.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.752+0000 7f110ea00700 1 -- 172.21.15.123:0/3702912646 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1100005040 con 0x7f11100fde90 2024-03-30T04:54:38.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.752+0000 7f110ea00700 1 --2- 172.21.15.123:0/3702912646 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11100fde90 0x7f1110113a70 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f1100000c00 tx=0x7f1100009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:54:38.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.753+0000 7f1106a00700 1 -- 172.21.15.123:0/3702912646 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1100021070 con 0x7f11100fde90 2024-03-30T04:54:38.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.753+0000 7f1106a00700 1 -- 172.21.15.123:0/3702912646 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1100013e00 con 0x7f11100fde90 2024-03-30T04:54:38.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.753+0000 7f1115a00700 1 -- 172.21.15.123:0/3702912646 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f111010dc70 con 0x7f11100fde90 2024-03-30T04:54:38.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.753+0000 7f1106a00700 1 -- 172.21.15.123:0/3702912646 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f110000e070 con 0x7f11100fde90 2024-03-30T04:54:38.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.753+0000 7f1115a00700 1 -- 172.21.15.123:0/3702912646 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f111010e1b0 con 0x7f11100fde90 2024-03-30T04:54:38.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.754+0000 7f1115a00700 1 -- 172.21.15.123:0/3702912646 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1110065e30 con 0x7f11100fde90 2024-03-30T04:54:38.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.755+0000 7f1106a00700 1 -- 172.21.15.123:0/3702912646 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f1100013950 con 0x7f11100fde90 2024-03-30T04:54:38.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.756+0000 7f1106a00700 1 --2- 172.21.15.123:0/3702912646 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f10f406cc30 0x7f10f406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:54:38.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.756+0000 7f1106a00700 1 -- 172.21.15.123:0/3702912646 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f110008cff0 con 0x7f11100fde90 2024-03-30T04:54:38.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.756+0000 7f110e000700 1 --2- 172.21.15.123:0/3702912646 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f10f406cc30 0x7f10f406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:54:38.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.756+0000 7f110e000700 1 --2- 172.21.15.123:0/3702912646 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f10f406cc30 0x7f10f406f0e0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f111010ebb0 tx=0x7f10fc00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:54:38.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:38.760+0000 7f1106a00700 1 -- 172.21.15.123:0/3702912646 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f11000c09f0 con 0x7f11100fde90 2024-03-30T04:54:39.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.045+0000 7f1115a00700 1 -- 172.21.15.123:0/3702912646 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f111005f5f0 con 0x7f11100fde90 2024-03-30T04:54:39.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.046+0000 7f1106a00700 1 -- 172.21.15.123:0/3702912646 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f110005aac0 con 0x7f11100fde90 2024-03-30T04:54:39.044 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:54:39.044 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:54:39.044 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:54:39.044 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:54:39.044 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:54:39.044 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:54:39.044 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:54:39.045 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:54:39.045 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T04:54:39.045 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:54:39.045 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:54:39.045 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 4 2024-03-30T04:54:39.045 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:54:39.045 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:54:39.045 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 14 2024-03-30T04:54:39.045 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:54:39.045 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:54:39.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.048+0000 7f1115a00700 1 -- 172.21.15.123:0/3702912646 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f10f406cc30 msgr2=0x7f10f406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:39.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.048+0000 7f1115a00700 1 --2- 172.21.15.123:0/3702912646 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f10f406cc30 0x7f10f406f0e0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f111010ebb0 tx=0x7f10fc00b040 comp rx=0 tx=0).stop 2024-03-30T04:54:39.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.048+0000 7f1115a00700 1 -- 172.21.15.123:0/3702912646 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11100fde90 msgr2=0x7f1110113a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:39.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.048+0000 7f1115a00700 1 --2- 172.21.15.123:0/3702912646 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11100fde90 0x7f1110113a70 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f1100000c00 tx=0x7f1100009e30 comp rx=0 tx=0).stop 2024-03-30T04:54:39.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.048+0000 7f1115a00700 1 -- 172.21.15.123:0/3702912646 shutdown_connections 2024-03-30T04:54:39.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.048+0000 7f1115a00700 1 --2- 172.21.15.123:0/3702912646 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f10f406cc30 0x7f10f406f0e0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:39.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.048+0000 7f1115a00700 1 --2- 172.21.15.123:0/3702912646 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11100fde90 0x7f1110113a70 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:39.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.048+0000 7f1115a00700 1 --2- 172.21.15.123:0/3702912646 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f11100fe830 0x7f1110113fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:39.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.048+0000 7f1115a00700 1 -- 172.21.15.123:0/3702912646 >> 172.21.15.123:0/3702912646 conn(0x7f1110072bc0 msgr2=0x7f11101037c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:54:39.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.048+0000 7f1115a00700 1 -- 172.21.15.123:0/3702912646 shutdown_connections 2024-03-30T04:54:39.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.048+0000 7f1115a00700 1 -- 172.21.15.123:0/3702912646 wait complete. 2024-03-30T04:54:39.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:54:38 smithi135 ceph-mon[106011]: pgmap v704: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:39.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:54:38 smithi135 ceph-mon[106011]: from='client.35300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:54:39.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.181+0000 7f0b7b200700 1 -- 172.21.15.123:0/1950522393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b740708a0 msgr2=0x7f0b74070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:39.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.181+0000 7f0b7b200700 1 --2- 172.21.15.123:0/1950522393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b740708a0 0x7f0b74070d10 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f0b60004970 tx=0x7f0b60009e30 comp rx=0 tx=0).stop 2024-03-30T04:54:39.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.181+0000 7f0b7b200700 1 -- 172.21.15.123:0/1950522393 shutdown_connections 2024-03-30T04:54:39.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.181+0000 7f0b7b200700 1 --2- 172.21.15.123:0/1950522393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b740708a0 0x7f0b74070d10 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:39.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.181+0000 7f0b7b200700 1 --2- 172.21.15.123:0/1950522393 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0b74071f90 0x7f0b74070360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:39.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.181+0000 7f0b7b200700 1 -- 172.21.15.123:0/1950522393 >> 172.21.15.123:0/1950522393 conn(0x7f0b740f89b0 msgr2=0x7f0b740fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:54:39.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.181+0000 7f0b7b200700 1 -- 172.21.15.123:0/1950522393 shutdown_connections 2024-03-30T04:54:39.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.181+0000 7f0b7b200700 1 -- 172.21.15.123:0/1950522393 wait complete. 2024-03-30T04:54:39.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.182+0000 7f0b7b200700 1 Processor -- start 2024-03-30T04:54:39.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.182+0000 7f0b7b200700 1 -- start start 2024-03-30T04:54:39.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.183+0000 7f0b7b200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0b740708a0 0x7f0b7411a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:54:39.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.183+0000 7f0b7b200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b74071f90 0x7f0b7411a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:54:39.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.183+0000 7f0b7b200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7411ac40 con 0x7f0b74071f90 2024-03-30T04:54:39.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.183+0000 7f0b7b200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b74114010 con 0x7f0b740708a0 2024-03-30T04:54:39.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.183+0000 7f0b73400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b74071f90 0x7f0b7411a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:54:39.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.183+0000 7f0b73400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b74071f90 0x7f0b7411a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43032/0 (socket says 172.21.15.123:43032) 2024-03-30T04:54:39.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.183+0000 7f0b73400700 1 -- 172.21.15.123:0/1503628496 learned_addr learned my addr 172.21.15.123:0/1503628496 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:54:39.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.183+0000 7f0b73e00700 1 --2- 172.21.15.123:0/1503628496 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0b740708a0 0x7f0b7411a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:54:39.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.183+0000 7f0b73400700 1 -- 172.21.15.123:0/1503628496 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0b740708a0 msgr2=0x7f0b7411a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:39.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.183+0000 7f0b73400700 1 --2- 172.21.15.123:0/1503628496 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0b740708a0 0x7f0b7411a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:39.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.183+0000 7f0b73400700 1 -- 172.21.15.123:0/1503628496 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b60005040 con 0x7f0b74071f90 2024-03-30T04:54:39.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.184+0000 7f0b73400700 1 --2- 172.21.15.123:0/1503628496 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b74071f90 0x7f0b7411a5f0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f0b600043d0 tx=0x7f0b60009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:54:39.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.184+0000 7f0b6be00700 1 -- 172.21.15.123:0/1503628496 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b6000e040 con 0x7f0b74071f90 2024-03-30T04:54:39.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.184+0000 7f0b6be00700 1 -- 172.21.15.123:0/1503628496 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0b60013420 con 0x7f0b74071f90 2024-03-30T04:54:39.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.184+0000 7f0b7b200700 1 -- 172.21.15.123:0/1503628496 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b74114290 con 0x7f0b74071f90 2024-03-30T04:54:39.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.184+0000 7f0b6be00700 1 -- 172.21.15.123:0/1503628496 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b6001c630 con 0x7f0b74071f90 2024-03-30T04:54:39.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.184+0000 7f0b7b200700 1 -- 172.21.15.123:0/1503628496 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b741147d0 con 0x7f0b74071f90 2024-03-30T04:54:39.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.185+0000 7f0b7b200700 1 -- 172.21.15.123:0/1503628496 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b74071aa0 con 0x7f0b74071f90 2024-03-30T04:54:39.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.186+0000 7f0b6be00700 1 -- 172.21.15.123:0/1503628496 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f0b600139e0 con 0x7f0b74071f90 2024-03-30T04:54:39.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.187+0000 7f0b6be00700 1 --2- 172.21.15.123:0/1503628496 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0b5806cc20 0x7f0b5806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:54:39.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.187+0000 7f0b6be00700 1 -- 172.21.15.123:0/1503628496 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f0b6008d2c0 con 0x7f0b74071f90 2024-03-30T04:54:39.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.187+0000 7f0b73e00700 1 --2- 172.21.15.123:0/1503628496 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0b5806cc20 0x7f0b5806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:54:39.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.188+0000 7f0b73e00700 1 --2- 172.21.15.123:0/1503628496 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0b5806cc20 0x7f0b5806f0d0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f0b64004f40 tx=0x7f0b6400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:54:39.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.192+0000 7f0b6be00700 1 -- 172.21.15.123:0/1503628496 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0b6005ad90 con 0x7f0b74071f90 2024-03-30T04:54:39.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:54:38 smithi123 ceph-mon[139369]: pgmap v704: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:39.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:54:38 smithi123 ceph-mon[139369]: from='client.35300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:54:39.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.433+0000 7f0b7b200700 1 -- 172.21.15.123:0/1503628496 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0b7404bed0 con 0x7f0b74071f90 2024-03-30T04:54:39.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.434+0000 7f0b6be00700 1 -- 172.21.15.123:0/1503628496 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 34 v34) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f0b6005a920 con 0x7f0b74071f90 2024-03-30T04:54:39.432 INFO:teuthology.orchestra.run.smithi123.stdout:e34 2024-03-30T04:54:39.432 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:54:39.432 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:54:39.432 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:54:39.433 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:54:39.433 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:54:39.433 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:54:39.433 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T04:54:39.433 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T04:54:39.433 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:54:39.433 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T04:54:39.433 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:54:39.433 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:54:39.433 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:54:39.433 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:54:39.433 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:54:39.433 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T04:54:39.433 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:54:39.434 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:54:39.434 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T04:54:39.434 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:54:39.434 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T04:54:39.434 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T04:54:39.434 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T04:54:39.434 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T04:54:39.434 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:54:39.434 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:54:39.435 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:54:39.435 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:54:39.435 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:54:39.435 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:54:39.435 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T04:54:39.435 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T04:54:39.435 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:54:39.435 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:54:39.435 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T04:54:39.435 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:54:39.435 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:54:39.436 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:54:39.436 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:54:39.436 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44975} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/3788988014,v1:172.21.15.135:6827/3788988014] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:54:39.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.436+0000 7f0b7b200700 1 -- 172.21.15.123:0/1503628496 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0b5806cc20 msgr2=0x7f0b5806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:39.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.436+0000 7f0b7b200700 1 --2- 172.21.15.123:0/1503628496 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0b5806cc20 0x7f0b5806f0d0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f0b64004f40 tx=0x7f0b6400b040 comp rx=0 tx=0).stop 2024-03-30T04:54:39.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.436+0000 7f0b7b200700 1 -- 172.21.15.123:0/1503628496 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b74071f90 msgr2=0x7f0b7411a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:39.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.436+0000 7f0b7b200700 1 --2- 172.21.15.123:0/1503628496 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b74071f90 0x7f0b7411a5f0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f0b600043d0 tx=0x7f0b60009e30 comp rx=0 tx=0).stop 2024-03-30T04:54:39.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.436+0000 7f0b7b200700 1 -- 172.21.15.123:0/1503628496 shutdown_connections 2024-03-30T04:54:39.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.436+0000 7f0b7b200700 1 --2- 172.21.15.123:0/1503628496 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0b5806cc20 0x7f0b5806f0d0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:39.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.436+0000 7f0b7b200700 1 --2- 172.21.15.123:0/1503628496 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0b740708a0 0x7f0b7411a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:39.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.436+0000 7f0b7b200700 1 --2- 172.21.15.123:0/1503628496 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b74071f90 0x7f0b7411a5f0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:39.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.436+0000 7f0b7b200700 1 -- 172.21.15.123:0/1503628496 >> 172.21.15.123:0/1503628496 conn(0x7f0b740f89b0 msgr2=0x7f0b740f9b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:54:39.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.436+0000 7f0b7b200700 1 -- 172.21.15.123:0/1503628496 shutdown_connections 2024-03-30T04:54:39.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.436+0000 7f0b7b200700 1 -- 172.21.15.123:0/1503628496 wait complete. 2024-03-30T04:54:39.438 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 34 2024-03-30T04:54:39.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.569+0000 7fed2c200700 1 -- 172.21.15.123:0/4036726614 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed24102e00 msgr2=0x7fed241031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:39.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.569+0000 7fed2c200700 1 --2- 172.21.15.123:0/4036726614 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed24102e00 0x7fed241031d0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fed18004940 tx=0x7fed18009e30 comp rx=0 tx=0).stop 2024-03-30T04:54:39.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.569+0000 7fed2c200700 1 -- 172.21.15.123:0/4036726614 shutdown_connections 2024-03-30T04:54:39.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.569+0000 7fed2c200700 1 --2- 172.21.15.123:0/4036726614 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fed241037a0 0x7fed240fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:39.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.569+0000 7fed2c200700 1 --2- 172.21.15.123:0/4036726614 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed24102e00 0x7fed241031d0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:39.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.569+0000 7fed2c200700 1 -- 172.21.15.123:0/4036726614 >> 172.21.15.123:0/4036726614 conn(0x7fed240f89b0 msgr2=0x7fed240fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:54:39.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.570+0000 7fed2c200700 1 -- 172.21.15.123:0/4036726614 shutdown_connections 2024-03-30T04:54:39.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.570+0000 7fed2c200700 1 -- 172.21.15.123:0/4036726614 wait complete. 2024-03-30T04:54:39.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.570+0000 7fed2c200700 1 Processor -- start 2024-03-30T04:54:39.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.570+0000 7fed2c200700 1 -- start start 2024-03-30T04:54:39.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.571+0000 7fed2c200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fed24102e00 0x7fed2410af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:54:39.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.571+0000 7fed2c200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed241037a0 0x7fed2410b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:54:39.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.571+0000 7fed2c200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed2410f0d0 con 0x7fed241037a0 2024-03-30T04:54:39.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.571+0000 7fed2c200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed2410b9c0 con 0x7fed24102e00 2024-03-30T04:54:39.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.571+0000 7fed28e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed241037a0 0x7fed2410b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:54:39.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.571+0000 7fed28e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed241037a0 0x7fed2410b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43052/0 (socket says 172.21.15.123:43052) 2024-03-30T04:54:39.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.571+0000 7fed28e00700 1 -- 172.21.15.123:0/1006139844 learned_addr learned my addr 172.21.15.123:0/1006139844 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:54:39.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.571+0000 7fed29800700 1 --2- 172.21.15.123:0/1006139844 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fed24102e00 0x7fed2410af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:54:39.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.572+0000 7fed28e00700 1 -- 172.21.15.123:0/1006139844 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fed24102e00 msgr2=0x7fed2410af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:39.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.572+0000 7fed28e00700 1 --2- 172.21.15.123:0/1006139844 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fed24102e00 0x7fed2410af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:39.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.572+0000 7fed28e00700 1 -- 172.21.15.123:0/1006139844 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed18005040 con 0x7fed241037a0 2024-03-30T04:54:39.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.572+0000 7fed28e00700 1 --2- 172.21.15.123:0/1006139844 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed241037a0 0x7fed2410b480 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fed10002a50 tx=0x7fed10004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:54:39.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.572+0000 7fed21600700 1 -- 172.21.15.123:0/1006139844 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed10007cc0 con 0x7fed241037a0 2024-03-30T04:54:39.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.572+0000 7fed21600700 1 -- 172.21.15.123:0/1006139844 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fed1000f040 con 0x7fed241037a0 2024-03-30T04:54:39.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.572+0000 7fed2c200700 1 -- 172.21.15.123:0/1006139844 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed2410bca0 con 0x7fed241037a0 2024-03-30T04:54:39.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.572+0000 7fed21600700 1 -- 172.21.15.123:0/1006139844 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed1000d920 con 0x7fed241037a0 2024-03-30T04:54:39.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.572+0000 7fed2c200700 1 -- 172.21.15.123:0/1006139844 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed2411e6c0 con 0x7fed241037a0 2024-03-30T04:54:39.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.574+0000 7fed2c200700 1 -- 172.21.15.123:0/1006139844 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed240fe550 con 0x7fed241037a0 2024-03-30T04:54:39.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.574+0000 7fed21600700 1 -- 172.21.15.123:0/1006139844 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fed10009680 con 0x7fed241037a0 2024-03-30T04:54:39.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.576+0000 7fed21600700 1 --2- 172.21.15.123:0/1006139844 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fed0c06cc30 0x7fed0c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:54:39.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.576+0000 7fed21600700 1 -- 172.21.15.123:0/1006139844 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fed1008bcf0 con 0x7fed241037a0 2024-03-30T04:54:39.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.576+0000 7fed29800700 1 --2- 172.21.15.123:0/1006139844 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fed0c06cc30 0x7fed0c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:54:39.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.577+0000 7fed29800700 1 --2- 172.21.15.123:0/1006139844 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fed0c06cc30 0x7fed0c06f0e0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fed18009fd0 tx=0x7fed18009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:54:39.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.580+0000 7fed21600700 1 -- 172.21.15.123:0/1006139844 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fed10055cb0 con 0x7fed241037a0 2024-03-30T04:54:39.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.801+0000 7fed2c200700 1 -- 172.21.15.123:0/1006139844 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fed2404bed0 con 0x7fed0c06cc30 2024-03-30T04:54:39.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.802+0000 7fed21600700 1 -- 172.21.15.123:0/1006139844 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fed2404bed0 con 0x7fed0c06cc30 2024-03-30T04:54:39.803 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:54:39.803 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:54:39.803 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:54:39.803 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T04:54:39.803 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:54:39.803 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T04:54:39.803 INFO:teuthology.orchestra.run.smithi123.stdout: "mds", 2024-03-30T04:54:39.803 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T04:54:39.803 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T04:54:39.803 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T04:54:39.803 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:54:39.803 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "16/21 daemons upgraded", 2024-03-30T04:54:39.804 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T04:54:39.804 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:54:39.804 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:54:39.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.804+0000 7fed2c200700 1 -- 172.21.15.123:0/1006139844 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fed0c06cc30 msgr2=0x7fed0c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:39.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.804+0000 7fed2c200700 1 --2- 172.21.15.123:0/1006139844 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fed0c06cc30 0x7fed0c06f0e0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fed18009fd0 tx=0x7fed18009e30 comp rx=0 tx=0).stop 2024-03-30T04:54:39.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.804+0000 7fed2c200700 1 -- 172.21.15.123:0/1006139844 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed241037a0 msgr2=0x7fed2410b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:39.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.804+0000 7fed2c200700 1 --2- 172.21.15.123:0/1006139844 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed241037a0 0x7fed2410b480 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fed10002a50 tx=0x7fed10004600 comp rx=0 tx=0).stop 2024-03-30T04:54:39.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.804+0000 7fed2c200700 1 -- 172.21.15.123:0/1006139844 shutdown_connections 2024-03-30T04:54:39.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.804+0000 7fed2c200700 1 --2- 172.21.15.123:0/1006139844 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fed0c06cc30 0x7fed0c06f0e0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:39.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.804+0000 7fed2c200700 1 --2- 172.21.15.123:0/1006139844 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fed24102e00 0x7fed2410af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:39.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.804+0000 7fed2c200700 1 --2- 172.21.15.123:0/1006139844 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed241037a0 0x7fed2410b480 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:39.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.804+0000 7fed2c200700 1 -- 172.21.15.123:0/1006139844 >> 172.21.15.123:0/1006139844 conn(0x7fed240f89b0 msgr2=0x7fed240706d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:54:39.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.805+0000 7fed2c200700 1 -- 172.21.15.123:0/1006139844 shutdown_connections 2024-03-30T04:54:39.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.805+0000 7fed2c200700 1 -- 172.21.15.123:0/1006139844 wait complete. 2024-03-30T04:54:39.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.941+0000 7f1836800700 1 -- 172.21.15.123:0/2457372809 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1830102e00 msgr2=0x7f18301031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:39.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.941+0000 7f1836800700 1 --2- 172.21.15.123:0/2457372809 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1830102e00 0x7f18301031d0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f1820004940 tx=0x7f1820009e30 comp rx=0 tx=0).stop 2024-03-30T04:54:39.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.941+0000 7f1836800700 1 -- 172.21.15.123:0/2457372809 shutdown_connections 2024-03-30T04:54:39.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.941+0000 7f1836800700 1 --2- 172.21.15.123:0/2457372809 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f18301037a0 0x7f18300fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:39.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.941+0000 7f1836800700 1 --2- 172.21.15.123:0/2457372809 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1830102e00 0x7f18301031d0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:39.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.941+0000 7f1836800700 1 -- 172.21.15.123:0/2457372809 >> 172.21.15.123:0/2457372809 conn(0x7f18300f89b0 msgr2=0x7f18300fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:54:39.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.941+0000 7f1836800700 1 -- 172.21.15.123:0/2457372809 shutdown_connections 2024-03-30T04:54:39.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.941+0000 7f1836800700 1 -- 172.21.15.123:0/2457372809 wait complete. 2024-03-30T04:54:39.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.942+0000 7f1836800700 1 Processor -- start 2024-03-30T04:54:39.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.942+0000 7f1836800700 1 -- start start 2024-03-30T04:54:39.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.943+0000 7f1836800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1830102e00 0x7f18300702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:54:39.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.943+0000 7f1836800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18301037a0 0x7f1830070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:54:39.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.943+0000 7f1836800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1830070d90 con 0x7f18301037a0 2024-03-30T04:54:39.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.943+0000 7f1836800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1830070ed0 con 0x7f1830102e00 2024-03-30T04:54:39.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.943+0000 7f182ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18301037a0 0x7f1830070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:54:39.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.943+0000 7f182ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18301037a0 0x7f1830070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43076/0 (socket says 172.21.15.123:43076) 2024-03-30T04:54:39.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.943+0000 7f182ea00700 1 -- 172.21.15.123:0/310119103 learned_addr learned my addr 172.21.15.123:0/310119103 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:54:39.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.943+0000 7f182f400700 1 --2- 172.21.15.123:0/310119103 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1830102e00 0x7f18300702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:54:39.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.944+0000 7f182ea00700 1 -- 172.21.15.123:0/310119103 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1830102e00 msgr2=0x7f18300702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:39.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.944+0000 7f182ea00700 1 --2- 172.21.15.123:0/310119103 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1830102e00 0x7f18300702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:39.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.944+0000 7f182ea00700 1 -- 172.21.15.123:0/310119103 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1820005040 con 0x7f18301037a0 2024-03-30T04:54:39.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.944+0000 7f182ea00700 1 --2- 172.21.15.123:0/310119103 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18301037a0 0x7f1830070800 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f181c002a50 tx=0x7f181c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:54:39.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.944+0000 7f1827400700 1 -- 172.21.15.123:0/310119103 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f181c007cc0 con 0x7f18301037a0 2024-03-30T04:54:39.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.944+0000 7f1827400700 1 -- 172.21.15.123:0/310119103 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f181c00f040 con 0x7f18301037a0 2024-03-30T04:54:39.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.944+0000 7f1836800700 1 -- 172.21.15.123:0/310119103 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1830119e20 con 0x7f18301037a0 2024-03-30T04:54:39.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.944+0000 7f1827400700 1 -- 172.21.15.123:0/310119103 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f181c00d920 con 0x7f18301037a0 2024-03-30T04:54:39.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.944+0000 7f1836800700 1 -- 172.21.15.123:0/310119103 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f183011a2a0 con 0x7f18301037a0 2024-03-30T04:54:39.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.945+0000 7f1836800700 1 -- 172.21.15.123:0/310119103 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18300fe550 con 0x7f18301037a0 2024-03-30T04:54:39.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.947+0000 7f1827400700 1 -- 172.21.15.123:0/310119103 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f181c009680 con 0x7f18301037a0 2024-03-30T04:54:39.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.947+0000 7f1827400700 1 --2- 172.21.15.123:0/310119103 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f181406cc30 0x7f181406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:54:39.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.947+0000 7f1827400700 1 -- 172.21.15.123:0/310119103 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f181c08c3c0 con 0x7f18301037a0 2024-03-30T04:54:39.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.948+0000 7f182f400700 1 --2- 172.21.15.123:0/310119103 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f181406cc30 0x7f181406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:54:39.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.948+0000 7f182f400700 1 --2- 172.21.15.123:0/310119103 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f181406cc30 0x7f181406f0e0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f1820009fd0 tx=0x7f1820009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:54:39.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:39.951+0000 7f1827400700 1 -- 172.21.15.123:0/310119103 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f181c056400 con 0x7f18301037a0 2024-03-30T04:54:40.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:54:39 smithi135 ceph-mon[106011]: from='client.35304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:54:40.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:54:39 smithi135 ceph-mon[106011]: from='client.35308 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:54:40.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:54:39 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3702912646' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:54:40.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:54:39 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/1503628496' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:54:40.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:54:39 smithi123 ceph-mon[139369]: from='client.35304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:54:40.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:54:39 smithi123 ceph-mon[139369]: from='client.35308 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:54:40.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:54:39 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3702912646' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:54:40.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:54:39 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/1503628496' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:54:40.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:40.235+0000 7f1836800700 1 -- 172.21.15.123:0/310119103 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f183005f5f0 con 0x7f18301037a0 2024-03-30T04:54:40.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:40.236+0000 7f1827400700 1 -- 172.21.15.123:0/310119103 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f181c059a20 con 0x7f18301037a0 2024-03-30T04:54:40.234 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T04:54:40.234 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-30T04:54:40.234 INFO:teuthology.orchestra.run.smithi123.stdout: host smithi135 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi135/config 2024-03-30T04:54:40.235 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-30T04:54:40.235 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T04:54:40.235 INFO:teuthology.orchestra.run.smithi123.stdout:Traceback (most recent call last): 2024-03-30T04:54:40.235 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-30T04:54:40.235 INFO:teuthology.orchestra.run.smithi123.stdout: "__main__", mod_spec) 2024-03-30T04:54:40.235 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-30T04:54:40.235 INFO:teuthology.orchestra.run.smithi123.stdout: exec(code, run_globals) 2024-03-30T04:54:40.235 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10889, in 2024-03-30T04:54:40.235 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10877, in main 2024-03-30T04:54:40.235 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-30T04:54:40.235 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-30T04:54:40.235 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-30T04:54:40.235 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-30T04:54:40.235 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-03-30T04:54:40.236 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-30T04:54:40.236 INFO:teuthology.orchestra.run.smithi123.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T04:54:40.236 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T04:54:40.236 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T04:54:40.236 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T04:54:40.236 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T04:54:40.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:40.238+0000 7f1836800700 1 -- 172.21.15.123:0/310119103 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f181406cc30 msgr2=0x7f181406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:40.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:40.238+0000 7f1836800700 1 --2- 172.21.15.123:0/310119103 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f181406cc30 0x7f181406f0e0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f1820009fd0 tx=0x7f1820009e30 comp rx=0 tx=0).stop 2024-03-30T04:54:40.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:40.238+0000 7f1836800700 1 -- 172.21.15.123:0/310119103 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18301037a0 msgr2=0x7f1830070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:54:40.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:40.238+0000 7f1836800700 1 --2- 172.21.15.123:0/310119103 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18301037a0 0x7f1830070800 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f181c002a50 tx=0x7f181c004600 comp rx=0 tx=0).stop 2024-03-30T04:54:40.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:40.238+0000 7f1836800700 1 -- 172.21.15.123:0/310119103 shutdown_connections 2024-03-30T04:54:40.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:40.238+0000 7f1836800700 1 --2- 172.21.15.123:0/310119103 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f181406cc30 0x7f181406f0e0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:40.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:40.238+0000 7f1836800700 1 --2- 172.21.15.123:0/310119103 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1830102e00 0x7f18300702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:40.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:40.238+0000 7f1836800700 1 --2- 172.21.15.123:0/310119103 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18301037a0 0x7f1830070800 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:54:40.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:40.238+0000 7f1836800700 1 -- 172.21.15.123:0/310119103 >> 172.21.15.123:0/310119103 conn(0x7f18300f89b0 msgr2=0x7f18300f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:54:40.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:40.239+0000 7f1836800700 1 -- 172.21.15.123:0/310119103 shutdown_connections 2024-03-30T04:54:40.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:54:40.239+0000 7f1836800700 1 -- 172.21.15.123:0/310119103 wait complete. 2024-03-30T04:54:41.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:54:40 smithi135 ceph-mon[106011]: pgmap v705: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:41.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:54:40 smithi135 ceph-mon[106011]: from='client.35320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:54:41.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:54:40 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/310119103' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:54:41.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:54:40 smithi123 ceph-mon[139369]: pgmap v705: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:41.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:54:40 smithi123 ceph-mon[139369]: from='client.35320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:54:41.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:54:40 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/310119103' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:54:43.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:54:42 smithi135 ceph-mon[106011]: pgmap v706: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:43.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:54:42 smithi123 ceph-mon[139369]: pgmap v706: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:45.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:54:44 smithi135 ceph-mon[106011]: pgmap v707: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:45.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:54:44 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:54:45.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:54:44 smithi123 ceph-mon[139369]: pgmap v707: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:45.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:54:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:54:47.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:54:46 smithi135 ceph-mon[106011]: pgmap v708: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:47.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:54:46 smithi123 ceph-mon[139369]: pgmap v708: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:49.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:54:48 smithi135 ceph-mon[106011]: pgmap v709: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:49.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:54:48 smithi123 ceph-mon[139369]: pgmap v709: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:51.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:54:50 smithi135 ceph-mon[106011]: pgmap v710: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:51.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:54:50 smithi123 ceph-mon[139369]: pgmap v710: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:53.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:54:52 smithi135 ceph-mon[106011]: pgmap v711: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:53.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:54:52 smithi123 ceph-mon[139369]: pgmap v711: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:55.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:54:54 smithi135 ceph-mon[106011]: pgmap v712: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:55.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:54:54 smithi123 ceph-mon[139369]: pgmap v712: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:57.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:54:56 smithi135 ceph-mon[106011]: pgmap v713: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:57.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:54:56 smithi123 ceph-mon[139369]: pgmap v713: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:59.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:54:58 smithi135 ceph-mon[106011]: pgmap v714: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:54:59.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:54:58 smithi123 ceph-mon[139369]: pgmap v714: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:00.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:54:59 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:55:00.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:54:59 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:55:01.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:55:00 smithi135 ceph-mon[106011]: pgmap v715: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:01.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:55:00 smithi123 ceph-mon[139369]: pgmap v715: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:03.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:55:02 smithi123 ceph-mon[139369]: pgmap v716: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:03.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:55:02 smithi135 ceph-mon[106011]: pgmap v716: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:05.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:55:04 smithi123 ceph-mon[139369]: pgmap v717: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:05.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:55:04 smithi135 ceph-mon[106011]: pgmap v717: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:07.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:55:06 smithi123 ceph-mon[139369]: pgmap v718: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:07.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:55:06 smithi135 ceph-mon[106011]: pgmap v718: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:09.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:55:08 smithi123 ceph-mon[139369]: pgmap v719: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:09.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:55:08 smithi135 ceph-mon[106011]: pgmap v719: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:10.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.393+0000 7f60e3000700 1 -- 172.21.15.123:0/1503551679 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60dc1037a0 msgr2=0x7f60dc0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:10.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.393+0000 7f60e3000700 1 --2- 172.21.15.123:0/1503551679 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60dc1037a0 0x7f60dc0fce10 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f60c8004940 tx=0x7f60c8009e30 comp rx=0 tx=0).stop 2024-03-30T04:55:10.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.393+0000 7f60e3000700 1 -- 172.21.15.123:0/1503551679 shutdown_connections 2024-03-30T04:55:10.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.393+0000 7f60e3000700 1 --2- 172.21.15.123:0/1503551679 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60dc1037a0 0x7f60dc0fce10 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:10.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.393+0000 7f60e3000700 1 --2- 172.21.15.123:0/1503551679 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f60dc102e00 0x7f60dc1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:10.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.393+0000 7f60e3000700 1 -- 172.21.15.123:0/1503551679 >> 172.21.15.123:0/1503551679 conn(0x7f60dc0f89b0 msgr2=0x7f60dc0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:55:10.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.393+0000 7f60e3000700 1 -- 172.21.15.123:0/1503551679 shutdown_connections 2024-03-30T04:55:10.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.393+0000 7f60e3000700 1 -- 172.21.15.123:0/1503551679 wait complete. 2024-03-30T04:55:10.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.394+0000 7f60e3000700 1 Processor -- start 2024-03-30T04:55:10.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.394+0000 7f60e3000700 1 -- start start 2024-03-30T04:55:10.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.394+0000 7f60e3000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60dc102e00 0x7f60dc0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:55:10.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.394+0000 7f60e3000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f60dc1037a0 0x7f60dc070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:55:10.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.394+0000 7f60e3000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60dc070d90 con 0x7f60dc102e00 2024-03-30T04:55:10.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.394+0000 7f60e3000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60dc070ed0 con 0x7f60dc1037a0 2024-03-30T04:55:10.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.394+0000 7f60dbe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60dc102e00 0x7f60dc0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:55:10.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.394+0000 7f60dbe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60dc102e00 0x7f60dc0702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45246/0 (socket says 172.21.15.123:45246) 2024-03-30T04:55:10.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.394+0000 7f60dbe00700 1 -- 172.21.15.123:0/1102944041 learned_addr learned my addr 172.21.15.123:0/1102944041 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:55:10.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.395+0000 7f60dbe00700 1 -- 172.21.15.123:0/1102944041 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f60dc1037a0 msgr2=0x7f60dc070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:10.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.395+0000 7f60dbe00700 1 --2- 172.21.15.123:0/1102944041 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f60dc1037a0 0x7f60dc070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:10.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.395+0000 7f60dbe00700 1 -- 172.21.15.123:0/1102944041 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60c8005040 con 0x7f60dc102e00 2024-03-30T04:55:10.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.395+0000 7f60dbe00700 1 --2- 172.21.15.123:0/1102944041 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60dc102e00 0x7f60dc0702c0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f60cc002a50 tx=0x7f60cc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:55:10.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.395+0000 7f60d3e00700 1 -- 172.21.15.123:0/1102944041 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f60cc007cc0 con 0x7f60dc102e00 2024-03-30T04:55:10.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.395+0000 7f60d3e00700 1 -- 172.21.15.123:0/1102944041 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f60cc00f040 con 0x7f60dc102e00 2024-03-30T04:55:10.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.395+0000 7f60e3000700 1 -- 172.21.15.123:0/1102944041 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60dc119e00 con 0x7f60dc102e00 2024-03-30T04:55:10.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.395+0000 7f60d3e00700 1 -- 172.21.15.123:0/1102944041 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f60cc009540 con 0x7f60dc102e00 2024-03-30T04:55:10.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.395+0000 7f60e3000700 1 -- 172.21.15.123:0/1102944041 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60dc11a280 con 0x7f60dc102e00 2024-03-30T04:55:10.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.397+0000 7f60e3000700 1 -- 172.21.15.123:0/1102944041 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60dc0fe550 con 0x7f60dc102e00 2024-03-30T04:55:10.395 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.397+0000 7f60d3e00700 1 -- 172.21.15.123:0/1102944041 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f60cc0096a0 con 0x7f60dc102e00 2024-03-30T04:55:10.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.398+0000 7f60d3e00700 1 --2- 172.21.15.123:0/1102944041 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f60c006cc30 0x7f60c006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:55:10.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.398+0000 7f60d3e00700 1 -- 172.21.15.123:0/1102944041 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f60cc08be00 con 0x7f60dc102e00 2024-03-30T04:55:10.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.399+0000 7f60db400700 1 --2- 172.21.15.123:0/1102944041 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f60c006cc30 0x7f60c006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:55:10.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.399+0000 7f60db400700 1 --2- 172.21.15.123:0/1102944041 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f60c006cc30 0x7f60c006f0e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f60c8004f90 tx=0x7f60c8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:55:10.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.403+0000 7f60d3e00700 1 -- 172.21.15.123:0/1102944041 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f60cc055dc0 con 0x7f60dc102e00 2024-03-30T04:55:10.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.627+0000 7f60e3000700 1 -- 172.21.15.123:0/1102944041 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f60dc04bed0 con 0x7f60c006cc30 2024-03-30T04:55:10.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.629+0000 7f60d3e00700 1 -- 172.21.15.123:0/1102944041 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f60dc04bed0 con 0x7f60c006cc30 2024-03-30T04:55:10.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.631+0000 7f60e3000700 1 -- 172.21.15.123:0/1102944041 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f60c006cc30 msgr2=0x7f60c006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:10.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.631+0000 7f60e3000700 1 --2- 172.21.15.123:0/1102944041 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f60c006cc30 0x7f60c006f0e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f60c8004f90 tx=0x7f60c8009e30 comp rx=0 tx=0).stop 2024-03-30T04:55:10.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.631+0000 7f60e3000700 1 -- 172.21.15.123:0/1102944041 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60dc102e00 msgr2=0x7f60dc0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:10.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.631+0000 7f60e3000700 1 --2- 172.21.15.123:0/1102944041 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60dc102e00 0x7f60dc0702c0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f60cc002a50 tx=0x7f60cc0046b0 comp rx=0 tx=0).stop 2024-03-30T04:55:10.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.631+0000 7f60e3000700 1 -- 172.21.15.123:0/1102944041 shutdown_connections 2024-03-30T04:55:10.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.631+0000 7f60e3000700 1 --2- 172.21.15.123:0/1102944041 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f60c006cc30 0x7f60c006f0e0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:10.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.631+0000 7f60e3000700 1 --2- 172.21.15.123:0/1102944041 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60dc102e00 0x7f60dc0702c0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:10.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.631+0000 7f60e3000700 1 --2- 172.21.15.123:0/1102944041 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f60dc1037a0 0x7f60dc070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:10.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.631+0000 7f60e3000700 1 -- 172.21.15.123:0/1102944041 >> 172.21.15.123:0/1102944041 conn(0x7f60dc0f89b0 msgr2=0x7f60dc0f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:55:10.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.632+0000 7f60e3000700 1 -- 172.21.15.123:0/1102944041 shutdown_connections 2024-03-30T04:55:10.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.632+0000 7f60e3000700 1 -- 172.21.15.123:0/1102944041 wait complete. 2024-03-30T04:55:10.644 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:55:10.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.766+0000 7f92e9a00700 1 -- 172.21.15.123:0/15578227 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92e41037a0 msgr2=0x7f92e40fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:10.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.766+0000 7f92e9a00700 1 --2- 172.21.15.123:0/15578227 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92e41037a0 0x7f92e40fce10 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f92d0004940 tx=0x7f92d0009e30 comp rx=0 tx=0).stop 2024-03-30T04:55:10.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.766+0000 7f92e9a00700 1 -- 172.21.15.123:0/15578227 shutdown_connections 2024-03-30T04:55:10.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.766+0000 7f92e9a00700 1 --2- 172.21.15.123:0/15578227 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92e41037a0 0x7f92e40fce10 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:10.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.766+0000 7f92e9a00700 1 --2- 172.21.15.123:0/15578227 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f92e4102e00 0x7f92e41031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:10.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.766+0000 7f92e9a00700 1 -- 172.21.15.123:0/15578227 >> 172.21.15.123:0/15578227 conn(0x7f92e40f89b0 msgr2=0x7f92e40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:55:10.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.766+0000 7f92e9a00700 1 -- 172.21.15.123:0/15578227 shutdown_connections 2024-03-30T04:55:10.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.766+0000 7f92e9a00700 1 -- 172.21.15.123:0/15578227 wait complete. 2024-03-30T04:55:10.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.767+0000 7f92e9a00700 1 Processor -- start 2024-03-30T04:55:10.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.767+0000 7f92e9a00700 1 -- start start 2024-03-30T04:55:10.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.768+0000 7f92e9a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92e4102e00 0x7f92e40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:55:10.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.768+0000 7f92e9a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f92e41037a0 0x7f92e4070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:55:10.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.768+0000 7f92e9a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92e4070d90 con 0x7f92e4102e00 2024-03-30T04:55:10.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.768+0000 7f92e9a00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92e4070ed0 con 0x7f92e41037a0 2024-03-30T04:55:10.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.768+0000 7f92e2a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92e4102e00 0x7f92e40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:55:10.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.768+0000 7f92e2a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92e4102e00 0x7f92e40702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45264/0 (socket says 172.21.15.123:45264) 2024-03-30T04:55:10.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.768+0000 7f92e2a00700 1 -- 172.21.15.123:0/78588412 learned_addr learned my addr 172.21.15.123:0/78588412 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:55:10.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.768+0000 7f92e2000700 1 --2- 172.21.15.123:0/78588412 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f92e41037a0 0x7f92e4070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:55:10.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.768+0000 7f92e2a00700 1 -- 172.21.15.123:0/78588412 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f92e41037a0 msgr2=0x7f92e4070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:10.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.768+0000 7f92e2a00700 1 --2- 172.21.15.123:0/78588412 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f92e41037a0 0x7f92e4070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:10.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.768+0000 7f92e2a00700 1 -- 172.21.15.123:0/78588412 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92d0005040 con 0x7f92e4102e00 2024-03-30T04:55:10.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.769+0000 7f92e2a00700 1 --2- 172.21.15.123:0/78588412 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92e4102e00 0x7f92e40702c0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f92d400eb70 tx=0x7f92d400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:55:10.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.769+0000 7f92daa00700 1 -- 172.21.15.123:0/78588412 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f92d4007cc0 con 0x7f92e4102e00 2024-03-30T04:55:10.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.769+0000 7f92daa00700 1 -- 172.21.15.123:0/78588412 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f92d400f040 con 0x7f92e4102e00 2024-03-30T04:55:10.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.769+0000 7f92e9a00700 1 -- 172.21.15.123:0/78588412 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92e4071120 con 0x7f92e4102e00 2024-03-30T04:55:10.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.769+0000 7f92daa00700 1 -- 172.21.15.123:0/78588412 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f92d400c9f0 con 0x7f92e4102e00 2024-03-30T04:55:10.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.769+0000 7f92e9a00700 1 -- 172.21.15.123:0/78588412 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92e411a1a0 con 0x7f92e4102e00 2024-03-30T04:55:10.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.770+0000 7f92e9a00700 1 -- 172.21.15.123:0/78588412 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92e40fe550 con 0x7f92e4102e00 2024-03-30T04:55:10.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.771+0000 7f92daa00700 1 -- 172.21.15.123:0/78588412 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f92d400cb50 con 0x7f92e4102e00 2024-03-30T04:55:10.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.772+0000 7f92daa00700 1 --2- 172.21.15.123:0/78588412 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f92c806cc30 0x7f92c806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:55:10.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.772+0000 7f92daa00700 1 -- 172.21.15.123:0/78588412 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f92d408bca0 con 0x7f92e4102e00 2024-03-30T04:55:10.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.772+0000 7f92e2000700 1 --2- 172.21.15.123:0/78588412 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f92c806cc30 0x7f92c806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:55:10.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.772+0000 7f92e2000700 1 --2- 172.21.15.123:0/78588412 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f92c806cc30 0x7f92c806f0e0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f92e4071c00 tx=0x7f92d0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:55:10.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.776+0000 7f92daa00700 1 -- 172.21.15.123:0/78588412 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f92d4055be0 con 0x7f92e4102e00 2024-03-30T04:55:10.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:10.999+0000 7f92e9a00700 1 -- 172.21.15.123:0/78588412 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f92e40719b0 con 0x7f92c806cc30 2024-03-30T04:55:10.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.000+0000 7f92daa00700 1 -- 172.21.15.123:0/78588412 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f92e40719b0 con 0x7f92c806cc30 2024-03-30T04:55:11.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.002+0000 7f92e9a00700 1 -- 172.21.15.123:0/78588412 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f92c806cc30 msgr2=0x7f92c806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:11.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.002+0000 7f92e9a00700 1 --2- 172.21.15.123:0/78588412 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f92c806cc30 0x7f92c806f0e0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f92e4071c00 tx=0x7f92d0009e30 comp rx=0 tx=0).stop 2024-03-30T04:55:11.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.002+0000 7f92e9a00700 1 -- 172.21.15.123:0/78588412 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92e4102e00 msgr2=0x7f92e40702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:11.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.002+0000 7f92e9a00700 1 --2- 172.21.15.123:0/78588412 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92e4102e00 0x7f92e40702c0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f92d400eb70 tx=0x7f92d400b040 comp rx=0 tx=0).stop 2024-03-30T04:55:11.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.002+0000 7f92e9a00700 1 -- 172.21.15.123:0/78588412 shutdown_connections 2024-03-30T04:55:11.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.002+0000 7f92e9a00700 1 --2- 172.21.15.123:0/78588412 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f92c806cc30 0x7f92c806f0e0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:11.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.002+0000 7f92e9a00700 1 --2- 172.21.15.123:0/78588412 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92e4102e00 0x7f92e40702c0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:11.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.002+0000 7f92e9a00700 1 --2- 172.21.15.123:0/78588412 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f92e41037a0 0x7f92e4070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:11.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.003+0000 7f92e9a00700 1 -- 172.21.15.123:0/78588412 >> 172.21.15.123:0/78588412 conn(0x7f92e40f89b0 msgr2=0x7f92e40f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:55:11.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.003+0000 7f92e9a00700 1 -- 172.21.15.123:0/78588412 shutdown_connections 2024-03-30T04:55:11.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.003+0000 7f92e9a00700 1 -- 172.21.15.123:0/78588412 wait complete. 2024-03-30T04:55:11.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.137+0000 7f99a3e00700 1 -- 172.21.15.123:0/4027840585 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f999c070c70 msgr2=0x7f999c109110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:11.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.137+0000 7f99a3e00700 1 --2- 172.21.15.123:0/4027840585 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f999c070c70 0x7f999c109110 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f9988004970 tx=0x7f9988009e30 comp rx=0 tx=0).stop 2024-03-30T04:55:11.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.137+0000 7f99a3e00700 1 -- 172.21.15.123:0/4027840585 shutdown_connections 2024-03-30T04:55:11.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.137+0000 7f99a3e00700 1 --2- 172.21.15.123:0/4027840585 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f999c070c70 0x7f999c109110 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:11.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.137+0000 7f99a3e00700 1 --2- 172.21.15.123:0/4027840585 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f999c070360 0x7f999c070730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:11.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.137+0000 7f99a3e00700 1 -- 172.21.15.123:0/4027840585 >> 172.21.15.123:0/4027840585 conn(0x7f999c0f89b0 msgr2=0x7f999c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:55:11.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.137+0000 7f99a3e00700 1 -- 172.21.15.123:0/4027840585 shutdown_connections 2024-03-30T04:55:11.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.137+0000 7f99a3e00700 1 -- 172.21.15.123:0/4027840585 wait complete. 2024-03-30T04:55:11.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.138+0000 7f99a3e00700 1 Processor -- start 2024-03-30T04:55:11.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.138+0000 7f99a3e00700 1 -- start start 2024-03-30T04:55:11.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.138+0000 7f99a3e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f999c070360 0x7f999c115ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:55:11.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.138+0000 7f99a3e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f999c070c70 0x7f999c1161e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:55:11.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.138+0000 7f99a3e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f999c116830 con 0x7f999c070360 2024-03-30T04:55:11.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.138+0000 7f99a3e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f999c10fc00 con 0x7f999c070c70 2024-03-30T04:55:11.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.138+0000 7f99a1400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f999c070360 0x7f999c115ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:55:11.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.138+0000 7f99a1400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f999c070360 0x7f999c115ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45274/0 (socket says 172.21.15.123:45274) 2024-03-30T04:55:11.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.138+0000 7f99a1400700 1 -- 172.21.15.123:0/4202056062 learned_addr learned my addr 172.21.15.123:0/4202056062 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:55:11.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.139+0000 7f99a0a00700 1 --2- 172.21.15.123:0/4202056062 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f999c070c70 0x7f999c1161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:55:11.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.139+0000 7f99a1400700 1 -- 172.21.15.123:0/4202056062 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f999c070c70 msgr2=0x7f999c1161e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:11.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.139+0000 7f99a1400700 1 --2- 172.21.15.123:0/4202056062 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f999c070c70 0x7f999c1161e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:11.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.139+0000 7f99a1400700 1 -- 172.21.15.123:0/4202056062 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9988005040 con 0x7f999c070360 2024-03-30T04:55:11.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.139+0000 7f99a0a00700 1 --2- 172.21.15.123:0/4202056062 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f999c070c70 0x7f999c1161e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:55:11.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.139+0000 7f99a1400700 1 --2- 172.21.15.123:0/4202056062 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f999c070360 0x7f999c115ca0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f9990002a50 tx=0x7f99900046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:55:11.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.139+0000 7f9999600700 1 -- 172.21.15.123:0/4202056062 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9990007cc0 con 0x7f999c070360 2024-03-30T04:55:11.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.139+0000 7f9999600700 1 -- 172.21.15.123:0/4202056062 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f999000f040 con 0x7f999c070360 2024-03-30T04:55:11.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.139+0000 7f99a3e00700 1 -- 172.21.15.123:0/4202056062 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f999c10fee0 con 0x7f999c070360 2024-03-30T04:55:11.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.139+0000 7f9999600700 1 -- 172.21.15.123:0/4202056062 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9990009540 con 0x7f999c070360 2024-03-30T04:55:11.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.139+0000 7f99a3e00700 1 -- 172.21.15.123:0/4202056062 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f999c1103f0 con 0x7f999c070360 2024-03-30T04:55:11.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.141+0000 7f99a3e00700 1 -- 172.21.15.123:0/4202056062 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f999c103bd0 con 0x7f999c070360 2024-03-30T04:55:11.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.142+0000 7f9999600700 1 -- 172.21.15.123:0/4202056062 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f99900096a0 con 0x7f999c070360 2024-03-30T04:55:11.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.142+0000 7f9999600700 1 --2- 172.21.15.123:0/4202056062 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f998406cc30 0x7f998406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:55:11.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.143+0000 7f9999600700 1 -- 172.21.15.123:0/4202056062 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f999008bde0 con 0x7f999c070360 2024-03-30T04:55:11.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.143+0000 7f99a0a00700 1 --2- 172.21.15.123:0/4202056062 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f998406cc30 0x7f998406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:55:11.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.143+0000 7f99a0a00700 1 --2- 172.21.15.123:0/4202056062 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f998406cc30 0x7f998406f0e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f9988009fd0 tx=0x7f9988009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:55:11.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.147+0000 7f9999600700 1 -- 172.21.15.123:0/4202056062 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9990059830 con 0x7f999c070360 2024-03-30T04:55:11.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:55:10 smithi123 ceph-mon[139369]: pgmap v720: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:11.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:55:10 smithi135 ceph-mon[106011]: pgmap v720: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:11.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.360+0000 7f99a3e00700 1 -- 172.21.15.123:0/4202056062 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f999c10c050 con 0x7f998406cc30 2024-03-30T04:55:11.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.369+0000 7f9999600700 1 -- 172.21.15.123:0/4202056062 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f9988008e90 con 0x7f998406cc30 2024-03-30T04:55:11.367 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:55:11.367 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (24m) 103s ago 36m 14.6M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:55:11.367 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (21m) 103s ago 37m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T04:55:11.367 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (21m) 106s ago 35m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T04:55:11.367 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (24m) 103s ago 36m 73.5M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T04:55:11.367 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (17m) 103s ago 30m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T04:55:11.367 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (17m) 103s ago 30m 16.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T04:55:11.368 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (110s) 106s ago 30m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 87b073a6c9ca 2024-03-30T04:55:11.368 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (17m) 106s ago 30m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T04:55:11.368 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (26m) 103s ago 38m 510M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:55:11.368 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (25m) 106s ago 35m 456M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:55:11.368 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (22m) 103s ago 38m 70.0M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T04:55:11.368 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (22m) 106s ago 35m 51.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T04:55:11.368 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (25m) 103s ago 37m 8636k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:55:11.368 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (25m) 106s ago 35m 8678k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:55:11.368 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (21m) 103s ago 34m 107M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T04:55:11.368 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (20m) 103s ago 34m 97.2M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T04:55:11.368 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (19m) 103s ago 33m 89.8M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T04:55:11.368 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (19m) 106s ago 33m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T04:55:11.368 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (18m) 106s ago 32m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T04:55:11.369 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (18m) 106s ago 32m 88.8M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T04:55:11.369 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (24m) 103s ago 36m 63.9M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:55:11.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.372+0000 7f99a3e00700 1 -- 172.21.15.123:0/4202056062 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f998406cc30 msgr2=0x7f998406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:11.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.372+0000 7f99a3e00700 1 --2- 172.21.15.123:0/4202056062 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f998406cc30 0x7f998406f0e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f9988009fd0 tx=0x7f9988009e30 comp rx=0 tx=0).stop 2024-03-30T04:55:11.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.372+0000 7f99a3e00700 1 -- 172.21.15.123:0/4202056062 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f999c070360 msgr2=0x7f999c115ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:11.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.372+0000 7f99a3e00700 1 --2- 172.21.15.123:0/4202056062 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f999c070360 0x7f999c115ca0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f9990002a50 tx=0x7f99900046b0 comp rx=0 tx=0).stop 2024-03-30T04:55:11.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.372+0000 7f99a3e00700 1 -- 172.21.15.123:0/4202056062 shutdown_connections 2024-03-30T04:55:11.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.372+0000 7f99a3e00700 1 --2- 172.21.15.123:0/4202056062 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f998406cc30 0x7f998406f0e0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:11.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.372+0000 7f99a3e00700 1 --2- 172.21.15.123:0/4202056062 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f999c070360 0x7f999c115ca0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:11.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.372+0000 7f99a3e00700 1 --2- 172.21.15.123:0/4202056062 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f999c070c70 0x7f999c1161e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:11.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.372+0000 7f99a3e00700 1 -- 172.21.15.123:0/4202056062 >> 172.21.15.123:0/4202056062 conn(0x7f999c0f89b0 msgr2=0x7f999c1015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:55:11.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.372+0000 7f99a3e00700 1 -- 172.21.15.123:0/4202056062 shutdown_connections 2024-03-30T04:55:11.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.372+0000 7f99a3e00700 1 -- 172.21.15.123:0/4202056062 wait complete. 2024-03-30T04:55:11.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.505+0000 7f6c9d800700 1 -- 172.21.15.123:0/2659605809 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c980fde90 msgr2=0x7f6c980fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:11.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.505+0000 7f6c9d800700 1 --2- 172.21.15.123:0/2659605809 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c980fde90 0x7f6c980fe260 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f6c88004940 tx=0x7f6c88009e30 comp rx=0 tx=0).stop 2024-03-30T04:55:11.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.505+0000 7f6c9d800700 1 -- 172.21.15.123:0/2659605809 shutdown_connections 2024-03-30T04:55:11.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.505+0000 7f6c9d800700 1 --2- 172.21.15.123:0/2659605809 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6c980fe830 0x7f6c98106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:11.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.505+0000 7f6c9d800700 1 --2- 172.21.15.123:0/2659605809 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c980fde90 0x7f6c980fe260 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:11.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.505+0000 7f6c9d800700 1 -- 172.21.15.123:0/2659605809 >> 172.21.15.123:0/2659605809 conn(0x7f6c98072bc0 msgr2=0x7f6c98072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:55:11.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.506+0000 7f6c9d800700 1 -- 172.21.15.123:0/2659605809 shutdown_connections 2024-03-30T04:55:11.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.506+0000 7f6c9d800700 1 -- 172.21.15.123:0/2659605809 wait complete. 2024-03-30T04:55:11.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.506+0000 7f6c9d800700 1 Processor -- start 2024-03-30T04:55:11.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.506+0000 7f6c9d800700 1 -- start start 2024-03-30T04:55:11.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.507+0000 7f6c9d800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6c980fde90 0x7f6c98115d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:55:11.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.507+0000 7f6c9d800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c980fe830 0x7f6c98116240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:55:11.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.507+0000 7f6c9d800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c98116920 con 0x7f6c980fe830 2024-03-30T04:55:11.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.507+0000 7f6c9d800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9810fc60 con 0x7f6c980fde90 2024-03-30T04:55:11.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.507+0000 7f6c96000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c980fe830 0x7f6c98116240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:55:11.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.507+0000 7f6c96000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c980fe830 0x7f6c98116240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45286/0 (socket says 172.21.15.123:45286) 2024-03-30T04:55:11.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.507+0000 7f6c96000700 1 -- 172.21.15.123:0/1380300479 learned_addr learned my addr 172.21.15.123:0/1380300479 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:55:11.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.507+0000 7f6c96000700 1 -- 172.21.15.123:0/1380300479 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6c980fde90 msgr2=0x7f6c98115d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:11.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.507+0000 7f6c96a00700 1 --2- 172.21.15.123:0/1380300479 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6c980fde90 0x7f6c98115d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:55:11.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.507+0000 7f6c96000700 1 --2- 172.21.15.123:0/1380300479 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6c980fde90 0x7f6c98115d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:11.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.507+0000 7f6c96000700 1 -- 172.21.15.123:0/1380300479 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c88005040 con 0x7f6c980fe830 2024-03-30T04:55:11.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.507+0000 7f6c96a00700 1 --2- 172.21.15.123:0/1380300479 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6c980fde90 0x7f6c98115d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:55:11.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.508+0000 7f6c96000700 1 --2- 172.21.15.123:0/1380300479 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c980fe830 0x7f6c98116240 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f6c8400b790 tx=0x7f6c84009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:55:11.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.508+0000 7f6c8ea00700 1 -- 172.21.15.123:0/1380300479 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c8400f770 con 0x7f6c980fe830 2024-03-30T04:55:11.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.508+0000 7f6c8ea00700 1 -- 172.21.15.123:0/1380300479 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6c8400fd80 con 0x7f6c980fe830 2024-03-30T04:55:11.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.508+0000 7f6c9d800700 1 -- 172.21.15.123:0/1380300479 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c9810fee0 con 0x7f6c980fe830 2024-03-30T04:55:11.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.508+0000 7f6c8ea00700 1 -- 172.21.15.123:0/1380300479 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c8400d310 con 0x7f6c980fe830 2024-03-30T04:55:11.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.508+0000 7f6c9d800700 1 -- 172.21.15.123:0/1380300479 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c98110420 con 0x7f6c980fe830 2024-03-30T04:55:11.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.509+0000 7f6c9d800700 1 -- 172.21.15.123:0/1380300479 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c98106a20 con 0x7f6c980fe830 2024-03-30T04:55:11.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.510+0000 7f6c8ea00700 1 -- 172.21.15.123:0/1380300479 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f6c84010420 con 0x7f6c980fe830 2024-03-30T04:55:11.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.511+0000 7f6c8ea00700 1 --2- 172.21.15.123:0/1380300479 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6c7c06cc30 0x7f6c7c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:55:11.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.511+0000 7f6c8ea00700 1 -- 172.21.15.123:0/1380300479 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f6c8408c0d0 con 0x7f6c980fe830 2024-03-30T04:55:11.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.511+0000 7f6c96a00700 1 --2- 172.21.15.123:0/1380300479 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6c7c06cc30 0x7f6c7c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:55:11.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.512+0000 7f6c96a00700 1 --2- 172.21.15.123:0/1380300479 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6c7c06cc30 0x7f6c7c06f0e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f6c8800b3b0 tx=0x7f6c88009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:55:11.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.516+0000 7f6c8ea00700 1 -- 172.21.15.123:0/1380300479 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6c84059b20 con 0x7f6c980fe830 2024-03-30T04:55:11.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.797+0000 7f6c9d800700 1 -- 172.21.15.123:0/1380300479 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f6c981113b0 con 0x7f6c980fe830 2024-03-30T04:55:11.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.798+0000 7f6c8ea00700 1 -- 172.21.15.123:0/1380300479 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f6c840596b0 con 0x7f6c980fe830 2024-03-30T04:55:11.796 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:55:11.796 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:55:11.797 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:55:11.797 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:55:11.797 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:55:11.797 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:55:11.797 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:55:11.797 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:55:11.797 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T04:55:11.797 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:55:11.797 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:55:11.797 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 4 2024-03-30T04:55:11.797 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:55:11.797 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:55:11.797 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 14 2024-03-30T04:55:11.797 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:55:11.798 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:55:11.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.800+0000 7f6c9d800700 1 -- 172.21.15.123:0/1380300479 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6c7c06cc30 msgr2=0x7f6c7c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:11.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.800+0000 7f6c9d800700 1 --2- 172.21.15.123:0/1380300479 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6c7c06cc30 0x7f6c7c06f0e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f6c8800b3b0 tx=0x7f6c88009e30 comp rx=0 tx=0).stop 2024-03-30T04:55:11.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.800+0000 7f6c9d800700 1 -- 172.21.15.123:0/1380300479 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c980fe830 msgr2=0x7f6c98116240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:11.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.800+0000 7f6c9d800700 1 --2- 172.21.15.123:0/1380300479 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c980fe830 0x7f6c98116240 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f6c8400b790 tx=0x7f6c84009250 comp rx=0 tx=0).stop 2024-03-30T04:55:11.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.800+0000 7f6c9d800700 1 -- 172.21.15.123:0/1380300479 shutdown_connections 2024-03-30T04:55:11.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.800+0000 7f6c9d800700 1 --2- 172.21.15.123:0/1380300479 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6c7c06cc30 0x7f6c7c06f0e0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:11.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.800+0000 7f6c9d800700 1 --2- 172.21.15.123:0/1380300479 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6c980fde90 0x7f6c98115d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:11.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.800+0000 7f6c9d800700 1 --2- 172.21.15.123:0/1380300479 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c980fe830 0x7f6c98116240 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:11.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.800+0000 7f6c9d800700 1 -- 172.21.15.123:0/1380300479 >> 172.21.15.123:0/1380300479 conn(0x7f6c98072bc0 msgr2=0x7f6c980fb7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:55:11.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.800+0000 7f6c9d800700 1 -- 172.21.15.123:0/1380300479 shutdown_connections 2024-03-30T04:55:11.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.801+0000 7f6c9d800700 1 -- 172.21.15.123:0/1380300479 wait complete. 2024-03-30T04:55:11.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.938+0000 7f929e000700 1 -- 172.21.15.123:0/2157613028 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9298070300 msgr2=0x7f92980706d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:11.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.938+0000 7f929e000700 1 --2- 172.21.15.123:0/2157613028 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9298070300 0x7f92980706d0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f9288004970 tx=0x7f9288009e30 comp rx=0 tx=0).stop 2024-03-30T04:55:11.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.938+0000 7f929e000700 1 -- 172.21.15.123:0/2157613028 shutdown_connections 2024-03-30T04:55:11.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.938+0000 7f929e000700 1 --2- 172.21.15.123:0/2157613028 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9298070c10 0x7f9298109170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:11.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.938+0000 7f929e000700 1 --2- 172.21.15.123:0/2157613028 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9298070300 0x7f92980706d0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:11.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.938+0000 7f929e000700 1 -- 172.21.15.123:0/2157613028 >> 172.21.15.123:0/2157613028 conn(0x7f92980f89b0 msgr2=0x7f92980fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:55:11.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.938+0000 7f929e000700 1 -- 172.21.15.123:0/2157613028 shutdown_connections 2024-03-30T04:55:11.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.938+0000 7f929e000700 1 -- 172.21.15.123:0/2157613028 wait complete. 2024-03-30T04:55:11.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.939+0000 7f929e000700 1 Processor -- start 2024-03-30T04:55:11.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.939+0000 7f929e000700 1 -- start start 2024-03-30T04:55:11.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.939+0000 7f929e000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9298070300 0x7f9298115a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:55:11.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.940+0000 7f929e000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9298070c10 0x7f9298115fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:55:11.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.940+0000 7f929e000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9298116600 con 0x7f9298070300 2024-03-30T04:55:11.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.940+0000 7f929e000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f929810fbe0 con 0x7f9298070c10 2024-03-30T04:55:11.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.940+0000 7f9297400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9298070300 0x7f9298115a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:55:11.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.940+0000 7f9297400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9298070300 0x7f9298115a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45308/0 (socket says 172.21.15.123:45308) 2024-03-30T04:55:11.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.940+0000 7f9297400700 1 -- 172.21.15.123:0/1322134253 learned_addr learned my addr 172.21.15.123:0/1322134253 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:55:11.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.940+0000 7f9297400700 1 -- 172.21.15.123:0/1322134253 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9298070c10 msgr2=0x7f9298115fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:11.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.940+0000 7f9297400700 1 --2- 172.21.15.123:0/1322134253 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9298070c10 0x7f9298115fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:11.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.940+0000 7f9297400700 1 -- 172.21.15.123:0/1322134253 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9288005040 con 0x7f9298070300 2024-03-30T04:55:11.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.940+0000 7f9297400700 1 --2- 172.21.15.123:0/1322134253 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9298070300 0x7f9298115a70 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f9288009360 tx=0x7f9288009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:55:11.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.941+0000 7f928f400700 1 -- 172.21.15.123:0/1322134253 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f928800e040 con 0x7f9298070300 2024-03-30T04:55:11.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.941+0000 7f929e000700 1 -- 172.21.15.123:0/1322134253 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f929810fe60 con 0x7f9298070300 2024-03-30T04:55:11.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.941+0000 7f928f400700 1 -- 172.21.15.123:0/1322134253 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9288004290 con 0x7f9298070300 2024-03-30T04:55:11.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.941+0000 7f928f400700 1 -- 172.21.15.123:0/1322134253 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f928801c3f0 con 0x7f9298070300 2024-03-30T04:55:11.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.941+0000 7f929e000700 1 -- 172.21.15.123:0/1322134253 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92981103a0 con 0x7f9298070300 2024-03-30T04:55:11.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.942+0000 7f929e000700 1 -- 172.21.15.123:0/1322134253 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9298103bf0 con 0x7f9298070300 2024-03-30T04:55:11.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.943+0000 7f928f400700 1 -- 172.21.15.123:0/1322134253 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f9288013420 con 0x7f9298070300 2024-03-30T04:55:11.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.944+0000 7f928f400700 1 --2- 172.21.15.123:0/1322134253 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f927c06cc20 0x7f927c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:55:11.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.944+0000 7f928f400700 1 -- 172.21.15.123:0/1322134253 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f928808d4d0 con 0x7f9298070300 2024-03-30T04:55:11.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.944+0000 7f9296a00700 1 --2- 172.21.15.123:0/1322134253 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f927c06cc20 0x7f927c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:55:11.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.944+0000 7f9296a00700 1 --2- 172.21.15.123:0/1322134253 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f927c06cc20 0x7f927c06f0d0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f9298111310 tx=0x7f9284007cc0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:55:11.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:11.948+0000 7f928f400700 1 -- 172.21.15.123:0/1322134253 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f928805afd0 con 0x7f9298070300 2024-03-30T04:55:12.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.196+0000 7f929e000700 1 -- 172.21.15.123:0/1322134253 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f929804c740 con 0x7f9298070300 2024-03-30T04:55:12.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.196+0000 7f928f400700 1 -- 172.21.15.123:0/1322134253 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 34 v34) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f928805ab60 con 0x7f9298070300 2024-03-30T04:55:12.195 INFO:teuthology.orchestra.run.smithi123.stdout:e34 2024-03-30T04:55:12.195 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:55:12.195 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:55:12.195 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:55:12.195 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:55:12.195 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:55:12.196 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:55:12.196 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T04:55:12.196 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T04:55:12.196 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:55:12.196 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T04:55:12.196 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:55:12.196 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:55:12.196 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:55:12.196 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:55:12.196 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:55:12.196 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T04:55:12.197 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:55:12.197 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:55:12.197 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T04:55:12.197 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:55:12.197 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T04:55:12.197 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T04:55:12.197 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T04:55:12.197 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T04:55:12.197 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:55:12.197 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:55:12.198 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:55:12.198 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:55:12.198 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:55:12.198 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:55:12.198 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T04:55:12.198 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T04:55:12.198 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:55:12.198 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:55:12.198 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T04:55:12.198 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:55:12.199 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:55:12.199 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:55:12.199 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:55:12.199 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44975} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/3788988014,v1:172.21.15.135:6827/3788988014] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:55:12.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.199+0000 7f929e000700 1 -- 172.21.15.123:0/1322134253 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f927c06cc20 msgr2=0x7f927c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:12.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.199+0000 7f929e000700 1 --2- 172.21.15.123:0/1322134253 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f927c06cc20 0x7f927c06f0d0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f9298111310 tx=0x7f9284007cc0 comp rx=0 tx=0).stop 2024-03-30T04:55:12.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.199+0000 7f929e000700 1 -- 172.21.15.123:0/1322134253 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9298070300 msgr2=0x7f9298115a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:12.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.199+0000 7f929e000700 1 --2- 172.21.15.123:0/1322134253 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9298070300 0x7f9298115a70 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f9288009360 tx=0x7f9288009e30 comp rx=0 tx=0).stop 2024-03-30T04:55:12.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.199+0000 7f929e000700 1 -- 172.21.15.123:0/1322134253 shutdown_connections 2024-03-30T04:55:12.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.199+0000 7f929e000700 1 --2- 172.21.15.123:0/1322134253 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f927c06cc20 0x7f927c06f0d0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:12.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.199+0000 7f929e000700 1 --2- 172.21.15.123:0/1322134253 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9298070300 0x7f9298115a70 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:12.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.199+0000 7f929e000700 1 --2- 172.21.15.123:0/1322134253 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9298070c10 0x7f9298115fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:12.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.199+0000 7f929e000700 1 -- 172.21.15.123:0/1322134253 >> 172.21.15.123:0/1322134253 conn(0x7f92980f89b0 msgr2=0x7f9298101650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:55:12.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.199+0000 7f929e000700 1 -- 172.21.15.123:0/1322134253 shutdown_connections 2024-03-30T04:55:12.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.200+0000 7f929e000700 1 -- 172.21.15.123:0/1322134253 wait complete. 2024-03-30T04:55:12.201 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 34 2024-03-30T04:55:12.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:55:11 smithi123 ceph-mon[139369]: from='client.35328 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:55:12.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:55:11 smithi123 ceph-mon[139369]: from='client.35332 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:55:12.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:55:11 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/1380300479' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:55:12.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.333+0000 7f7b5be00700 1 -- 172.21.15.123:0/3732776673 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b54102e00 msgr2=0x7f7b541031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:12.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.333+0000 7f7b5be00700 1 --2- 172.21.15.123:0/3732776673 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b54102e00 0x7f7b541031d0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f7b48004940 tx=0x7f7b48009e30 comp rx=0 tx=0).stop 2024-03-30T04:55:12.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.333+0000 7f7b5be00700 1 -- 172.21.15.123:0/3732776673 shutdown_connections 2024-03-30T04:55:12.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.333+0000 7f7b5be00700 1 --2- 172.21.15.123:0/3732776673 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b541037a0 0x7f7b540fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:12.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.333+0000 7f7b5be00700 1 --2- 172.21.15.123:0/3732776673 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b54102e00 0x7f7b541031d0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:12.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.333+0000 7f7b5be00700 1 -- 172.21.15.123:0/3732776673 >> 172.21.15.123:0/3732776673 conn(0x7f7b540f89b0 msgr2=0x7f7b540fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:55:12.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.334+0000 7f7b5be00700 1 -- 172.21.15.123:0/3732776673 shutdown_connections 2024-03-30T04:55:12.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.334+0000 7f7b5be00700 1 -- 172.21.15.123:0/3732776673 wait complete. 2024-03-30T04:55:12.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.334+0000 7f7b5be00700 1 Processor -- start 2024-03-30T04:55:12.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.334+0000 7f7b5be00700 1 -- start start 2024-03-30T04:55:12.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.335+0000 7f7b5be00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b54102e00 0x7f7b54113a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:55:12.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.335+0000 7f7b5be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b541037a0 0x7f7b54113f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:55:12.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.335+0000 7f7b5be00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b541145d0 con 0x7f7b541037a0 2024-03-30T04:55:12.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.335+0000 7f7b5be00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b5410d9c0 con 0x7f7b54102e00 2024-03-30T04:55:12.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.335+0000 7f7b58a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b541037a0 0x7f7b54113f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:55:12.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.335+0000 7f7b59400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b54102e00 0x7f7b54113a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:55:12.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.335+0000 7f7b58a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b541037a0 0x7f7b54113f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45320/0 (socket says 172.21.15.123:45320) 2024-03-30T04:55:12.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.335+0000 7f7b58a00700 1 -- 172.21.15.123:0/371640403 learned_addr learned my addr 172.21.15.123:0/371640403 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:55:12.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.335+0000 7f7b58a00700 1 -- 172.21.15.123:0/371640403 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b54102e00 msgr2=0x7f7b54113a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:12.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.336+0000 7f7b58a00700 1 --2- 172.21.15.123:0/371640403 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b54102e00 0x7f7b54113a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:12.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.336+0000 7f7b58a00700 1 -- 172.21.15.123:0/371640403 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b48005040 con 0x7f7b541037a0 2024-03-30T04:55:12.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.336+0000 7f7b58a00700 1 --2- 172.21.15.123:0/371640403 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b541037a0 0x7f7b54113f80 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f7b4000eb40 tx=0x7f7b4000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:55:12.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.336+0000 7f7b51600700 1 -- 172.21.15.123:0/371640403 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b40007cc0 con 0x7f7b541037a0 2024-03-30T04:55:12.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.336+0000 7f7b51600700 1 -- 172.21.15.123:0/371640403 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7b4000f040 con 0x7f7b541037a0 2024-03-30T04:55:12.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.336+0000 7f7b51600700 1 -- 172.21.15.123:0/371640403 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b4000c8f0 con 0x7f7b541037a0 2024-03-30T04:55:12.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.336+0000 7f7b5be00700 1 -- 172.21.15.123:0/371640403 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b5410dc40 con 0x7f7b541037a0 2024-03-30T04:55:12.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.336+0000 7f7b5be00700 1 -- 172.21.15.123:0/371640403 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b5410e200 con 0x7f7b541037a0 2024-03-30T04:55:12.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.337+0000 7f7b5be00700 1 -- 172.21.15.123:0/371640403 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b540fe550 con 0x7f7b541037a0 2024-03-30T04:55:12.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.338+0000 7f7b51600700 1 -- 172.21.15.123:0/371640403 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f7b400090d0 con 0x7f7b541037a0 2024-03-30T04:55:12.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.339+0000 7f7b51600700 1 --2- 172.21.15.123:0/371640403 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7b3c06cc30 0x7f7b3c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:55:12.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.340+0000 7f7b51600700 1 -- 172.21.15.123:0/371640403 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7b4008ba50 con 0x7f7b541037a0 2024-03-30T04:55:12.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.340+0000 7f7b59400700 1 --2- 172.21.15.123:0/371640403 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7b3c06cc30 0x7f7b3c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:55:12.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.340+0000 7f7b59400700 1 --2- 172.21.15.123:0/371640403 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7b3c06cc30 0x7f7b3c06f0e0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f7b4800b0d0 tx=0x7f7b48009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:55:12.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.344+0000 7f7b51600700 1 -- 172.21.15.123:0/371640403 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7b40059550 con 0x7f7b541037a0 2024-03-30T04:55:12.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:55:11 smithi135 ceph-mon[106011]: from='client.35328 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:55:12.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:55:11 smithi135 ceph-mon[106011]: from='client.35332 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:55:12.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:55:11 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/1380300479' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:55:12.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.565+0000 7f7b5be00700 1 -- 172.21.15.123:0/371640403 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7b54109e60 con 0x7f7b3c06cc30 2024-03-30T04:55:12.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.567+0000 7f7b51600700 1 -- 172.21.15.123:0/371640403 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f7b480096f0 con 0x7f7b3c06cc30 2024-03-30T04:55:12.565 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:55:12.565 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:55:12.565 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:55:12.565 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T04:55:12.565 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:55:12.565 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T04:55:12.565 INFO:teuthology.orchestra.run.smithi123.stdout: "mds", 2024-03-30T04:55:12.566 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T04:55:12.566 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T04:55:12.566 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T04:55:12.566 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:55:12.566 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "16/21 daemons upgraded", 2024-03-30T04:55:12.566 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T04:55:12.566 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:55:12.566 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:55:12.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.569+0000 7f7b5be00700 1 -- 172.21.15.123:0/371640403 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7b3c06cc30 msgr2=0x7f7b3c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:12.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.569+0000 7f7b5be00700 1 --2- 172.21.15.123:0/371640403 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7b3c06cc30 0x7f7b3c06f0e0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f7b4800b0d0 tx=0x7f7b48009e30 comp rx=0 tx=0).stop 2024-03-30T04:55:12.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.569+0000 7f7b5be00700 1 -- 172.21.15.123:0/371640403 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b541037a0 msgr2=0x7f7b54113f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:12.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.569+0000 7f7b5be00700 1 --2- 172.21.15.123:0/371640403 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b541037a0 0x7f7b54113f80 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f7b4000eb40 tx=0x7f7b4000b040 comp rx=0 tx=0).stop 2024-03-30T04:55:12.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.569+0000 7f7b5be00700 1 -- 172.21.15.123:0/371640403 shutdown_connections 2024-03-30T04:55:12.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.569+0000 7f7b5be00700 1 --2- 172.21.15.123:0/371640403 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7b3c06cc30 0x7f7b3c06f0e0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:12.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.569+0000 7f7b5be00700 1 --2- 172.21.15.123:0/371640403 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b54102e00 0x7f7b54113a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:12.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.569+0000 7f7b5be00700 1 --2- 172.21.15.123:0/371640403 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b541037a0 0x7f7b54113f80 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:12.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.569+0000 7f7b5be00700 1 -- 172.21.15.123:0/371640403 >> 172.21.15.123:0/371640403 conn(0x7f7b540f89b0 msgr2=0x7f7b540f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:55:12.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.569+0000 7f7b5be00700 1 -- 172.21.15.123:0/371640403 shutdown_connections 2024-03-30T04:55:12.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.569+0000 7f7b5be00700 1 -- 172.21.15.123:0/371640403 wait complete. 2024-03-30T04:55:12.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.702+0000 7f7f3f400700 1 -- 172.21.15.123:0/1838355738 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f38102e00 msgr2=0x7f7f381031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:12.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.702+0000 7f7f3f400700 1 --2- 172.21.15.123:0/1838355738 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f38102e00 0x7f7f381031d0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c004940 tx=0x7f7f2c009e30 comp rx=0 tx=0).stop 2024-03-30T04:55:12.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.702+0000 7f7f3f400700 1 -- 172.21.15.123:0/1838355738 shutdown_connections 2024-03-30T04:55:12.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.702+0000 7f7f3f400700 1 --2- 172.21.15.123:0/1838355738 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f381037a0 0x7f7f380fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:12.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.702+0000 7f7f3f400700 1 --2- 172.21.15.123:0/1838355738 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f38102e00 0x7f7f381031d0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:12.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.702+0000 7f7f3f400700 1 -- 172.21.15.123:0/1838355738 >> 172.21.15.123:0/1838355738 conn(0x7f7f380f89b0 msgr2=0x7f7f380fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:55:12.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.702+0000 7f7f3f400700 1 -- 172.21.15.123:0/1838355738 shutdown_connections 2024-03-30T04:55:12.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.702+0000 7f7f3f400700 1 -- 172.21.15.123:0/1838355738 wait complete. 2024-03-30T04:55:12.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.703+0000 7f7f3f400700 1 Processor -- start 2024-03-30T04:55:12.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.703+0000 7f7f3f400700 1 -- start start 2024-03-30T04:55:12.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.703+0000 7f7f3f400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f38102e00 0x7f7f38115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:55:12.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.703+0000 7f7f3f400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f381037a0 0x7f7f381161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:55:12.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.703+0000 7f7f3f400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f38116800 con 0x7f7f381037a0 2024-03-30T04:55:12.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.703+0000 7f7f3f400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3810fbd0 con 0x7f7f38102e00 2024-03-30T04:55:12.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.703+0000 7f7f37e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f381037a0 0x7f7f381161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:55:12.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.703+0000 7f7f37e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f381037a0 0x7f7f381161b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45344/0 (socket says 172.21.15.123:45344) 2024-03-30T04:55:12.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.704+0000 7f7f37e00700 1 -- 172.21.15.123:0/2500798109 learned_addr learned my addr 172.21.15.123:0/2500798109 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:55:12.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.704+0000 7f7f37e00700 1 -- 172.21.15.123:0/2500798109 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f38102e00 msgr2=0x7f7f38115c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:12.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.704+0000 7f7f3ca00700 1 --2- 172.21.15.123:0/2500798109 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f38102e00 0x7f7f38115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:55:12.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.704+0000 7f7f37e00700 1 --2- 172.21.15.123:0/2500798109 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f38102e00 0x7f7f38115c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:12.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.704+0000 7f7f37e00700 1 -- 172.21.15.123:0/2500798109 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f2c005040 con 0x7f7f381037a0 2024-03-30T04:55:12.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.704+0000 7f7f3ca00700 1 --2- 172.21.15.123:0/2500798109 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f38102e00 0x7f7f38115c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:55:12.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.704+0000 7f7f37e00700 1 --2- 172.21.15.123:0/2500798109 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f381037a0 0x7f7f381161b0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f7f2400eb40 tx=0x7f7f2400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:55:12.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.704+0000 7f7f34c00700 1 -- 172.21.15.123:0/2500798109 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f24007cc0 con 0x7f7f381037a0 2024-03-30T04:55:12.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.704+0000 7f7f34c00700 1 -- 172.21.15.123:0/2500798109 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7f2400f040 con 0x7f7f381037a0 2024-03-30T04:55:12.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.704+0000 7f7f34c00700 1 -- 172.21.15.123:0/2500798109 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f24009570 con 0x7f7f381037a0 2024-03-30T04:55:12.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.704+0000 7f7f3f400700 1 -- 172.21.15.123:0/2500798109 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f3810fe50 con 0x7f7f381037a0 2024-03-30T04:55:12.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.705+0000 7f7f3f400700 1 -- 172.21.15.123:0/2500798109 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f38110410 con 0x7f7f381037a0 2024-03-30T04:55:12.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.706+0000 7f7f3f400700 1 -- 172.21.15.123:0/2500798109 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f380fe550 con 0x7f7f381037a0 2024-03-30T04:55:12.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.707+0000 7f7f34c00700 1 -- 172.21.15.123:0/2500798109 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f7f240096d0 con 0x7f7f381037a0 2024-03-30T04:55:12.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.708+0000 7f7f34c00700 1 --2- 172.21.15.123:0/2500798109 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7f2006cc30 0x7f7f2006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:55:12.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.708+0000 7f7f34c00700 1 -- 172.21.15.123:0/2500798109 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7f2408bb90 con 0x7f7f381037a0 2024-03-30T04:55:12.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.708+0000 7f7f3ca00700 1 --2- 172.21.15.123:0/2500798109 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7f2006cc30 0x7f7f2006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:55:12.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.709+0000 7f7f3ca00700 1 --2- 172.21.15.123:0/2500798109 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7f2006cc30 0x7f7f2006f0e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c004560 tx=0x7f7f2c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:55:12.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.712+0000 7f7f34c00700 1 -- 172.21.15.123:0/2500798109 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7f24059690 con 0x7f7f381037a0 2024-03-30T04:55:12.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.995+0000 7f7f3f400700 1 -- 172.21.15.123:0/2500798109 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f7f3804bed0 con 0x7f7f381037a0 2024-03-30T04:55:12.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.995+0000 7f7f34c00700 1 -- 172.21.15.123:0/2500798109 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f7f24019e10 con 0x7f7f381037a0 2024-03-30T04:55:12.994 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T04:55:12.994 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-30T04:55:12.994 INFO:teuthology.orchestra.run.smithi123.stdout: host smithi135 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi135/config 2024-03-30T04:55:12.994 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-30T04:55:12.994 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T04:55:12.994 INFO:teuthology.orchestra.run.smithi123.stdout:Traceback (most recent call last): 2024-03-30T04:55:12.994 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-30T04:55:12.994 INFO:teuthology.orchestra.run.smithi123.stdout: "__main__", mod_spec) 2024-03-30T04:55:12.994 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-30T04:55:12.994 INFO:teuthology.orchestra.run.smithi123.stdout: exec(code, run_globals) 2024-03-30T04:55:12.995 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10889, in 2024-03-30T04:55:12.995 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10877, in main 2024-03-30T04:55:12.995 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-30T04:55:12.995 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-30T04:55:12.995 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-30T04:55:12.995 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-30T04:55:12.995 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-03-30T04:55:12.995 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-30T04:55:12.995 INFO:teuthology.orchestra.run.smithi123.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T04:55:12.995 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T04:55:12.995 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T04:55:12.995 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T04:55:12.995 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T04:55:12.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.997+0000 7f7f3f400700 1 -- 172.21.15.123:0/2500798109 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7f2006cc30 msgr2=0x7f7f2006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:12.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.997+0000 7f7f3f400700 1 --2- 172.21.15.123:0/2500798109 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7f2006cc30 0x7f7f2006f0e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c004560 tx=0x7f7f2c009e30 comp rx=0 tx=0).stop 2024-03-30T04:55:12.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.997+0000 7f7f3f400700 1 -- 172.21.15.123:0/2500798109 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f381037a0 msgr2=0x7f7f381161b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:12.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.997+0000 7f7f3f400700 1 --2- 172.21.15.123:0/2500798109 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f381037a0 0x7f7f381161b0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f7f2400eb40 tx=0x7f7f2400b040 comp rx=0 tx=0).stop 2024-03-30T04:55:12.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.998+0000 7f7f3f400700 1 -- 172.21.15.123:0/2500798109 shutdown_connections 2024-03-30T04:55:12.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.998+0000 7f7f3f400700 1 --2- 172.21.15.123:0/2500798109 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7f2006cc30 0x7f7f2006f0e0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:12.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.998+0000 7f7f3f400700 1 --2- 172.21.15.123:0/2500798109 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f38102e00 0x7f7f38115c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:12.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.998+0000 7f7f3f400700 1 --2- 172.21.15.123:0/2500798109 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f381037a0 0x7f7f381161b0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:12.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.998+0000 7f7f3f400700 1 -- 172.21.15.123:0/2500798109 >> 172.21.15.123:0/2500798109 conn(0x7f7f380f89b0 msgr2=0x7f7f380f9960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:55:12.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.998+0000 7f7f3f400700 1 -- 172.21.15.123:0/2500798109 shutdown_connections 2024-03-30T04:55:12.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:12.998+0000 7f7f3f400700 1 -- 172.21.15.123:0/2500798109 wait complete. 2024-03-30T04:55:13.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:55:12 smithi123 ceph-mon[139369]: from='client.35336 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:55:13.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:55:12 smithi123 ceph-mon[139369]: pgmap v721: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:13.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:55:12 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/1322134253' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:55:13.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:55:12 smithi135 ceph-mon[106011]: from='client.35336 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:55:13.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:55:12 smithi135 ceph-mon[106011]: pgmap v721: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:13.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:55:12 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/1322134253' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:55:14.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:55:13 smithi123 ceph-mon[139369]: from='client.35348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:55:14.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:55:13 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2500798109' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:55:14.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:55:13 smithi135 ceph-mon[106011]: from='client.35348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:55:14.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:55:13 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2500798109' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:55:15.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:55:14 smithi123 ceph-mon[139369]: pgmap v722: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:15.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:55:14 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:55:15.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:55:14 smithi135 ceph-mon[106011]: pgmap v722: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:15.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:55:14 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:55:17.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:55:16 smithi123 ceph-mon[139369]: pgmap v723: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:17.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:55:16 smithi135 ceph-mon[106011]: pgmap v723: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:19.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:55:18 smithi123 ceph-mon[139369]: pgmap v724: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:19.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:55:18 smithi135 ceph-mon[106011]: pgmap v724: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:21.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:55:20 smithi123 ceph-mon[139369]: pgmap v725: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:21.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:55:20 smithi135 ceph-mon[106011]: pgmap v725: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:23.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:55:22 smithi123 ceph-mon[139369]: pgmap v726: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:23.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:55:22 smithi135 ceph-mon[106011]: pgmap v726: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:25.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:55:24 smithi123 ceph-mon[139369]: pgmap v727: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:25.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:55:24 smithi135 ceph-mon[106011]: pgmap v727: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:27.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:55:26 smithi123 ceph-mon[139369]: pgmap v728: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:27.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:55:26 smithi135 ceph-mon[106011]: pgmap v728: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:29.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:55:28 smithi123 ceph-mon[139369]: pgmap v729: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:29.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:55:28 smithi135 ceph-mon[106011]: pgmap v729: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:30.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:55:29 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:55:30.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:55:29 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:55:31.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:55:30 smithi123 ceph-mon[139369]: pgmap v730: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:31.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:55:30 smithi135 ceph-mon[106011]: pgmap v730: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:33.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:55:32 smithi123 ceph-mon[139369]: pgmap v731: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:33.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:55:32 smithi135 ceph-mon[106011]: pgmap v731: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:35.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:55:34 smithi123 ceph-mon[139369]: pgmap v732: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:35.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:55:34 smithi135 ceph-mon[106011]: pgmap v732: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:37.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:55:36 smithi123 ceph-mon[139369]: pgmap v733: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:37.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:55:36 smithi135 ceph-mon[106011]: pgmap v733: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:39.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:55:38 smithi123 ceph-mon[139369]: pgmap v734: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:39.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:55:38 smithi135 ceph-mon[106011]: pgmap v734: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:41.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:55:40 smithi123 ceph-mon[139369]: pgmap v735: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:41.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:55:40 smithi135 ceph-mon[106011]: pgmap v735: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:43.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.146+0000 7f9216600700 1 -- 172.21.15.123:0/2477818859 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92101037c0 msgr2=0x7f92100fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:43.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.146+0000 7f9216600700 1 --2- 172.21.15.123:0/2477818859 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92101037c0 0x7f92100fce30 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f91fc004940 tx=0x7f91fc009e30 comp rx=0 tx=0).stop 2024-03-30T04:55:43.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.146+0000 7f9216600700 1 -- 172.21.15.123:0/2477818859 shutdown_connections 2024-03-30T04:55:43.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.146+0000 7f9216600700 1 --2- 172.21.15.123:0/2477818859 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92101037c0 0x7f92100fce30 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:43.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.146+0000 7f9216600700 1 --2- 172.21.15.123:0/2477818859 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9210102e20 0x7f92101031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:43.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.146+0000 7f9216600700 1 -- 172.21.15.123:0/2477818859 >> 172.21.15.123:0/2477818859 conn(0x7f92100f89d0 msgr2=0x7f92100fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:55:43.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.146+0000 7f9216600700 1 -- 172.21.15.123:0/2477818859 shutdown_connections 2024-03-30T04:55:43.146 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.146+0000 7f9216600700 1 -- 172.21.15.123:0/2477818859 wait complete. 2024-03-30T04:55:43.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.147+0000 7f9216600700 1 Processor -- start 2024-03-30T04:55:43.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.147+0000 7f9216600700 1 -- start start 2024-03-30T04:55:43.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.148+0000 7f9216600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9210102e20 0x7f9210113ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:55:43.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.148+0000 7f9216600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f92101037c0 0x7f9210113ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:55:43.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.148+0000 7f9216600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92101146d0 con 0x7f9210102e20 2024-03-30T04:55:43.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.148+0000 7f9216600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f921010da30 con 0x7f92101037c0 2024-03-30T04:55:43.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.148+0000 7f920f400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9210102e20 0x7f9210113ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:55:43.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.148+0000 7f920f400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9210102e20 0x7f9210113ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43096/0 (socket says 172.21.15.123:43096) 2024-03-30T04:55:43.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.148+0000 7f920f400700 1 -- 172.21.15.123:0/3831999807 learned_addr learned my addr 172.21.15.123:0/3831999807 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:55:43.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.148+0000 7f920ea00700 1 --2- 172.21.15.123:0/3831999807 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f92101037c0 0x7f9210113ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:55:43.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.148+0000 7f920f400700 1 -- 172.21.15.123:0/3831999807 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f92101037c0 msgr2=0x7f9210113ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:43.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.148+0000 7f920f400700 1 --2- 172.21.15.123:0/3831999807 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f92101037c0 0x7f9210113ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:43.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.148+0000 7f920f400700 1 -- 172.21.15.123:0/3831999807 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91fc005040 con 0x7f9210102e20 2024-03-30T04:55:43.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.148+0000 7f920ea00700 1 --2- 172.21.15.123:0/3831999807 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f92101037c0 0x7f9210113ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:55:43.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.149+0000 7f920f400700 1 --2- 172.21.15.123:0/3831999807 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9210102e20 0x7f9210113ab0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f920000b790 tx=0x7f9200009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:55:43.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.149+0000 7f9207400700 1 -- 172.21.15.123:0/3831999807 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f920000f770 con 0x7f9210102e20 2024-03-30T04:55:43.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.149+0000 7f9207400700 1 -- 172.21.15.123:0/3831999807 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f920000fd80 con 0x7f9210102e20 2024-03-30T04:55:43.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.149+0000 7f9216600700 1 -- 172.21.15.123:0/3831999807 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f921010dd10 con 0x7f9210102e20 2024-03-30T04:55:43.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.149+0000 7f9207400700 1 -- 172.21.15.123:0/3831999807 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f920000d310 con 0x7f9210102e20 2024-03-30T04:55:43.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.149+0000 7f9216600700 1 -- 172.21.15.123:0/3831999807 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f921010e2d0 con 0x7f9210102e20 2024-03-30T04:55:43.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.150+0000 7f9216600700 1 -- 172.21.15.123:0/3831999807 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92100fe570 con 0x7f9210102e20 2024-03-30T04:55:43.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.151+0000 7f9207400700 1 -- 172.21.15.123:0/3831999807 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f9200010420 con 0x7f9210102e20 2024-03-30T04:55:43.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.152+0000 7f9207400700 1 --2- 172.21.15.123:0/3831999807 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f91f406cc30 0x7f91f406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:55:43.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.152+0000 7f9207400700 1 -- 172.21.15.123:0/3831999807 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f920008bfe0 con 0x7f9210102e20 2024-03-30T04:55:43.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.152+0000 7f920ea00700 1 --2- 172.21.15.123:0/3831999807 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f91f406cc30 0x7f91f406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:55:43.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.153+0000 7f920ea00700 1 --2- 172.21.15.123:0/3831999807 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f91f406cc30 0x7f91f406f0e0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f921010ebf0 tx=0x7f91fc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:55:43.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.156+0000 7f9207400700 1 -- 172.21.15.123:0/3831999807 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f92000bf9f0 con 0x7f9210102e20 2024-03-30T04:55:43.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:55:42 smithi123 ceph-mon[139369]: pgmap v736: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:43.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:55:42 smithi135 ceph-mon[106011]: pgmap v736: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:43.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.374+0000 7f9216600700 1 -- 172.21.15.123:0/3831999807 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9210109ea0 con 0x7f91f406cc30 2024-03-30T04:55:43.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.375+0000 7f9207400700 1 -- 172.21.15.123:0/3831999807 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f9210109ea0 con 0x7f91f406cc30 2024-03-30T04:55:43.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.378+0000 7f9216600700 1 -- 172.21.15.123:0/3831999807 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f91f406cc30 msgr2=0x7f91f406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:43.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.378+0000 7f9216600700 1 --2- 172.21.15.123:0/3831999807 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f91f406cc30 0x7f91f406f0e0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f921010ebf0 tx=0x7f91fc009e30 comp rx=0 tx=0).stop 2024-03-30T04:55:43.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.378+0000 7f9216600700 1 -- 172.21.15.123:0/3831999807 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9210102e20 msgr2=0x7f9210113ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:43.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.378+0000 7f9216600700 1 --2- 172.21.15.123:0/3831999807 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9210102e20 0x7f9210113ab0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f920000b790 tx=0x7f9200009250 comp rx=0 tx=0).stop 2024-03-30T04:55:43.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.378+0000 7f9216600700 1 -- 172.21.15.123:0/3831999807 shutdown_connections 2024-03-30T04:55:43.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.378+0000 7f9216600700 1 --2- 172.21.15.123:0/3831999807 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f91f406cc30 0x7f91f406f0e0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:43.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.378+0000 7f9216600700 1 --2- 172.21.15.123:0/3831999807 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9210102e20 0x7f9210113ab0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:43.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.378+0000 7f9216600700 1 --2- 172.21.15.123:0/3831999807 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f92101037c0 0x7f9210113ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:43.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.378+0000 7f9216600700 1 -- 172.21.15.123:0/3831999807 >> 172.21.15.123:0/3831999807 conn(0x7f92100f89d0 msgr2=0x7f92100f9a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:55:43.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.378+0000 7f9216600700 1 -- 172.21.15.123:0/3831999807 shutdown_connections 2024-03-30T04:55:43.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.378+0000 7f9216600700 1 -- 172.21.15.123:0/3831999807 wait complete. 2024-03-30T04:55:43.391 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:55:43.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.514+0000 7f258be00700 1 -- 172.21.15.123:0/3271591327 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25840708a0 msgr2=0x7f2584070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:43.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.514+0000 7f258be00700 1 --2- 172.21.15.123:0/3271591327 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25840708a0 0x7f2584070d10 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f2570004970 tx=0x7f2570009e30 comp rx=0 tx=0).stop 2024-03-30T04:55:43.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.514+0000 7f258be00700 1 -- 172.21.15.123:0/3271591327 shutdown_connections 2024-03-30T04:55:43.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.514+0000 7f258be00700 1 --2- 172.21.15.123:0/3271591327 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25840708a0 0x7f2584070d10 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:43.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.514+0000 7f258be00700 1 --2- 172.21.15.123:0/3271591327 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2584071f90 0x7f2584070360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:43.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.514+0000 7f258be00700 1 -- 172.21.15.123:0/3271591327 >> 172.21.15.123:0/3271591327 conn(0x7f25840f89b0 msgr2=0x7f25840fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:55:43.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.514+0000 7f258be00700 1 -- 172.21.15.123:0/3271591327 shutdown_connections 2024-03-30T04:55:43.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.514+0000 7f258be00700 1 -- 172.21.15.123:0/3271591327 wait complete. 2024-03-30T04:55:43.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.515+0000 7f258be00700 1 Processor -- start 2024-03-30T04:55:43.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.515+0000 7f258be00700 1 -- start start 2024-03-30T04:55:43.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.515+0000 7f258be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25840708a0 0x7f258411a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:55:43.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.515+0000 7f258be00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2584071f90 0x7f258411a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:55:43.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.515+0000 7f258be00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f258411ab90 con 0x7f25840708a0 2024-03-30T04:55:43.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.515+0000 7f258be00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2584114010 con 0x7f2584071f90 2024-03-30T04:55:43.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.515+0000 7f2589400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25840708a0 0x7f258411a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:55:43.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.516+0000 7f2589400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25840708a0 0x7f258411a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43116/0 (socket says 172.21.15.123:43116) 2024-03-30T04:55:43.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.516+0000 7f2589400700 1 -- 172.21.15.123:0/4220374136 learned_addr learned my addr 172.21.15.123:0/4220374136 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:55:43.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.516+0000 7f2588a00700 1 --2- 172.21.15.123:0/4220374136 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2584071f90 0x7f258411a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:55:43.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.516+0000 7f2589400700 1 -- 172.21.15.123:0/4220374136 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2584071f90 msgr2=0x7f258411a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:43.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.516+0000 7f2589400700 1 --2- 172.21.15.123:0/4220374136 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2584071f90 0x7f258411a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:43.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.516+0000 7f2589400700 1 -- 172.21.15.123:0/4220374136 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2570005040 con 0x7f25840708a0 2024-03-30T04:55:43.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.516+0000 7f2589400700 1 --2- 172.21.15.123:0/4220374136 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25840708a0 0x7f258411a000 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f2578002a50 tx=0x7f25780046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:55:43.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.516+0000 7f2581600700 1 -- 172.21.15.123:0/4220374136 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2578007cc0 con 0x7f25840708a0 2024-03-30T04:55:43.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.516+0000 7f2581600700 1 -- 172.21.15.123:0/4220374136 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f257800f040 con 0x7f25840708a0 2024-03-30T04:55:43.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.516+0000 7f258be00700 1 -- 172.21.15.123:0/4220374136 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25841142f0 con 0x7f25840708a0 2024-03-30T04:55:43.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.516+0000 7f2581600700 1 -- 172.21.15.123:0/4220374136 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f257800d9c0 con 0x7f25840708a0 2024-03-30T04:55:43.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.517+0000 7f258be00700 1 -- 172.21.15.123:0/4220374136 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2584114800 con 0x7f25840708a0 2024-03-30T04:55:43.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.518+0000 7f258be00700 1 -- 172.21.15.123:0/4220374136 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2584071aa0 con 0x7f25840708a0 2024-03-30T04:55:43.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.518+0000 7f2581600700 1 -- 172.21.15.123:0/4220374136 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f257800db20 con 0x7f25840708a0 2024-03-30T04:55:43.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.519+0000 7f2581600700 1 --2- 172.21.15.123:0/4220374136 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f256c06cc30 0x7f256c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:55:43.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.519+0000 7f2581600700 1 -- 172.21.15.123:0/4220374136 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f257808bd20 con 0x7f25840708a0 2024-03-30T04:55:43.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.520+0000 7f2588a00700 1 --2- 172.21.15.123:0/4220374136 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f256c06cc30 0x7f256c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:55:43.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.520+0000 7f2588a00700 1 --2- 172.21.15.123:0/4220374136 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f256c06cc30 0x7f256c06f0e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f2570009fd0 tx=0x7f2570009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:55:43.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.524+0000 7f2581600700 1 -- 172.21.15.123:0/4220374136 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2578055ce0 con 0x7f25840708a0 2024-03-30T04:55:43.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.748+0000 7f258be00700 1 -- 172.21.15.123:0/4220374136 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f25841103e0 con 0x7f256c06cc30 2024-03-30T04:55:43.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.750+0000 7f2581600700 1 -- 172.21.15.123:0/4220374136 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f25841103e0 con 0x7f256c06cc30 2024-03-30T04:55:43.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.752+0000 7f258be00700 1 -- 172.21.15.123:0/4220374136 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f256c06cc30 msgr2=0x7f256c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:43.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.752+0000 7f258be00700 1 --2- 172.21.15.123:0/4220374136 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f256c06cc30 0x7f256c06f0e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f2570009fd0 tx=0x7f2570009e30 comp rx=0 tx=0).stop 2024-03-30T04:55:43.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.752+0000 7f258be00700 1 -- 172.21.15.123:0/4220374136 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25840708a0 msgr2=0x7f258411a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:43.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.752+0000 7f258be00700 1 --2- 172.21.15.123:0/4220374136 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25840708a0 0x7f258411a000 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f2578002a50 tx=0x7f25780046b0 comp rx=0 tx=0).stop 2024-03-30T04:55:43.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.752+0000 7f258be00700 1 -- 172.21.15.123:0/4220374136 shutdown_connections 2024-03-30T04:55:43.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.752+0000 7f258be00700 1 --2- 172.21.15.123:0/4220374136 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f256c06cc30 0x7f256c06f0e0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:43.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.752+0000 7f258be00700 1 --2- 172.21.15.123:0/4220374136 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25840708a0 0x7f258411a000 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:43.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.752+0000 7f258be00700 1 --2- 172.21.15.123:0/4220374136 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2584071f90 0x7f258411a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:43.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.752+0000 7f258be00700 1 -- 172.21.15.123:0/4220374136 >> 172.21.15.123:0/4220374136 conn(0x7f25840f89b0 msgr2=0x7f25840f9b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:55:43.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.752+0000 7f258be00700 1 -- 172.21.15.123:0/4220374136 shutdown_connections 2024-03-30T04:55:43.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.752+0000 7f258be00700 1 -- 172.21.15.123:0/4220374136 wait complete. 2024-03-30T04:55:43.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.887+0000 7f1412600700 1 -- 172.21.15.123:0/4141872377 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f140c102e20 msgr2=0x7f140c1031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:43.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.887+0000 7f1412600700 1 --2- 172.21.15.123:0/4141872377 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f140c102e20 0x7f140c1031f0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f13fc004940 tx=0x7f13fc009e30 comp rx=0 tx=0).stop 2024-03-30T04:55:43.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.887+0000 7f1412600700 1 -- 172.21.15.123:0/4141872377 shutdown_connections 2024-03-30T04:55:43.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.887+0000 7f1412600700 1 --2- 172.21.15.123:0/4141872377 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f140c1037c0 0x7f140c0fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:43.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.887+0000 7f1412600700 1 --2- 172.21.15.123:0/4141872377 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f140c102e20 0x7f140c1031f0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:43.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.887+0000 7f1412600700 1 -- 172.21.15.123:0/4141872377 >> 172.21.15.123:0/4141872377 conn(0x7f140c0f89d0 msgr2=0x7f140c0fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:55:43.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.887+0000 7f1412600700 1 -- 172.21.15.123:0/4141872377 shutdown_connections 2024-03-30T04:55:43.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.887+0000 7f1412600700 1 -- 172.21.15.123:0/4141872377 wait complete. 2024-03-30T04:55:43.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.888+0000 7f1412600700 1 Processor -- start 2024-03-30T04:55:43.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.888+0000 7f1412600700 1 -- start start 2024-03-30T04:55:43.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.889+0000 7f1412600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f140c102e20 0x7f140c115c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:55:43.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.889+0000 7f1412600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f140c1037c0 0x7f140c116190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:55:43.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.889+0000 7f1412600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f140c1167e0 con 0x7f140c102e20 2024-03-30T04:55:43.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.889+0000 7f1412600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f140c10fbf0 con 0x7f140c1037c0 2024-03-30T04:55:43.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.889+0000 7f140b400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f140c102e20 0x7f140c115c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:55:43.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.889+0000 7f140b400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f140c102e20 0x7f140c115c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60342/0 (socket says 172.21.15.123:60342) 2024-03-30T04:55:43.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.889+0000 7f140b400700 1 -- 172.21.15.123:0/3536856536 learned_addr learned my addr 172.21.15.123:0/3536856536 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:55:43.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.889+0000 7f140aa00700 1 --2- 172.21.15.123:0/3536856536 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f140c1037c0 0x7f140c116190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:55:43.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.889+0000 7f140b400700 1 -- 172.21.15.123:0/3536856536 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f140c1037c0 msgr2=0x7f140c116190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:43.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.889+0000 7f140b400700 1 --2- 172.21.15.123:0/3536856536 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f140c1037c0 0x7f140c116190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:43.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.889+0000 7f140b400700 1 -- 172.21.15.123:0/3536856536 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13fc005040 con 0x7f140c102e20 2024-03-30T04:55:43.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.889+0000 7f140aa00700 1 --2- 172.21.15.123:0/3536856536 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f140c1037c0 0x7f140c116190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:55:43.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.889+0000 7f140b400700 1 --2- 172.21.15.123:0/3536856536 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f140c102e20 0x7f140c115c50 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f13fc007f50 tx=0x7f13fc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:55:43.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.890+0000 7f1403400700 1 -- 172.21.15.123:0/3536856536 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13fc021070 con 0x7f140c102e20 2024-03-30T04:55:43.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.890+0000 7f1412600700 1 -- 172.21.15.123:0/3536856536 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f140c10fe70 con 0x7f140c102e20 2024-03-30T04:55:43.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.890+0000 7f1403400700 1 -- 172.21.15.123:0/3536856536 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f13fc013e00 con 0x7f140c102e20 2024-03-30T04:55:43.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.890+0000 7f1403400700 1 -- 172.21.15.123:0/3536856536 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13fc00e070 con 0x7f140c102e20 2024-03-30T04:55:43.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.890+0000 7f1412600700 1 -- 172.21.15.123:0/3536856536 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f140c110430 con 0x7f140c102e20 2024-03-30T04:55:43.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.891+0000 7f1412600700 1 -- 172.21.15.123:0/3536856536 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f140c0fe570 con 0x7f140c102e20 2024-03-30T04:55:43.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.892+0000 7f1403400700 1 -- 172.21.15.123:0/3536856536 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f13fc013950 con 0x7f140c102e20 2024-03-30T04:55:43.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.893+0000 7f1403400700 1 --2- 172.21.15.123:0/3536856536 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f13f006cc30 0x7f13f006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:55:43.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.893+0000 7f1403400700 1 -- 172.21.15.123:0/3536856536 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f13fc08d350 con 0x7f140c102e20 2024-03-30T04:55:43.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.893+0000 7f140aa00700 1 --2- 172.21.15.123:0/3536856536 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f13f006cc30 0x7f13f006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:55:43.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.894+0000 7f140aa00700 1 --2- 172.21.15.123:0/3536856536 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f13f006cc30 0x7f13f006f0e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f13f8004600 tx=0x7f13f800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:55:43.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:43.898+0000 7f1403400700 1 -- 172.21.15.123:0/3536856536 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f13fc057310 con 0x7f140c102e20 2024-03-30T04:55:44.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.109+0000 7f1412600700 1 -- 172.21.15.123:0/3536856536 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f140c04bf40 con 0x7f13f006cc30 2024-03-30T04:55:44.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.119+0000 7f1403400700 1 -- 172.21.15.123:0/3536856536 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f140c04bf40 con 0x7f13f006cc30 2024-03-30T04:55:44.117 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:55:44.118 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (24m) 2m ago 37m 14.6M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:55:44.118 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (22m) 2m ago 37m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T04:55:44.118 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (22m) 2m ago 36m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T04:55:44.118 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (24m) 2m ago 37m 73.5M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T04:55:44.118 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (17m) 2m ago 31m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T04:55:44.118 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (18m) 2m ago 31m 16.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T04:55:44.118 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (2m) 2m ago 30m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 87b073a6c9ca 2024-03-30T04:55:44.118 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (17m) 2m ago 31m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T04:55:44.118 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (27m) 2m ago 39m 510M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:55:44.118 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (26m) 2m ago 36m 456M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:55:44.118 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (23m) 2m ago 39m 70.0M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T04:55:44.118 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (22m) 2m ago 36m 51.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T04:55:44.118 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (25m) 2m ago 37m 8636k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:55:44.119 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (25m) 2m ago 36m 8678k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:55:44.119 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (21m) 2m ago 35m 107M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T04:55:44.119 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (21m) 2m ago 34m 97.2M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T04:55:44.119 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (20m) 2m ago 34m 89.8M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T04:55:44.119 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (19m) 2m ago 33m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T04:55:44.119 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (19m) 2m ago 33m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T04:55:44.119 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (18m) 2m ago 32m 88.8M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T04:55:44.119 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (25m) 2m ago 36m 63.9M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:55:44.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.121+0000 7f1412600700 1 -- 172.21.15.123:0/3536856536 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f13f006cc30 msgr2=0x7f13f006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:44.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.121+0000 7f1412600700 1 --2- 172.21.15.123:0/3536856536 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f13f006cc30 0x7f13f006f0e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f13f8004600 tx=0x7f13f800b040 comp rx=0 tx=0).stop 2024-03-30T04:55:44.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.121+0000 7f1412600700 1 -- 172.21.15.123:0/3536856536 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f140c102e20 msgr2=0x7f140c115c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:44.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.121+0000 7f1412600700 1 --2- 172.21.15.123:0/3536856536 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f140c102e20 0x7f140c115c50 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f13fc007f50 tx=0x7f13fc009e30 comp rx=0 tx=0).stop 2024-03-30T04:55:44.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.122+0000 7f1412600700 1 -- 172.21.15.123:0/3536856536 shutdown_connections 2024-03-30T04:55:44.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.122+0000 7f1412600700 1 --2- 172.21.15.123:0/3536856536 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f13f006cc30 0x7f13f006f0e0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:44.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.122+0000 7f1412600700 1 --2- 172.21.15.123:0/3536856536 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f140c102e20 0x7f140c115c50 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:44.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.122+0000 7f1412600700 1 --2- 172.21.15.123:0/3536856536 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f140c1037c0 0x7f140c116190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:44.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.122+0000 7f1412600700 1 -- 172.21.15.123:0/3536856536 >> 172.21.15.123:0/3536856536 conn(0x7f140c0f89d0 msgr2=0x7f140c0f9990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:55:44.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.122+0000 7f1412600700 1 -- 172.21.15.123:0/3536856536 shutdown_connections 2024-03-30T04:55:44.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.122+0000 7f1412600700 1 -- 172.21.15.123:0/3536856536 wait complete. 2024-03-30T04:55:44.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.254+0000 7fbcdf000700 1 -- 172.21.15.123:0/3484831571 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbcd8102e00 msgr2=0x7fbcd81031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:44.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.254+0000 7fbcdf000700 1 --2- 172.21.15.123:0/3484831571 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbcd8102e00 0x7fbcd81031d0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fbcc8004940 tx=0x7fbcc8009e30 comp rx=0 tx=0).stop 2024-03-30T04:55:44.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.254+0000 7fbcdf000700 1 -- 172.21.15.123:0/3484831571 shutdown_connections 2024-03-30T04:55:44.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.254+0000 7fbcdf000700 1 --2- 172.21.15.123:0/3484831571 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbcd81037a0 0x7fbcd80fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:44.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.254+0000 7fbcdf000700 1 --2- 172.21.15.123:0/3484831571 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbcd8102e00 0x7fbcd81031d0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:44.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.254+0000 7fbcdf000700 1 -- 172.21.15.123:0/3484831571 >> 172.21.15.123:0/3484831571 conn(0x7fbcd80f89b0 msgr2=0x7fbcd80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:55:44.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.254+0000 7fbcdf000700 1 -- 172.21.15.123:0/3484831571 shutdown_connections 2024-03-30T04:55:44.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.254+0000 7fbcdf000700 1 -- 172.21.15.123:0/3484831571 wait complete. 2024-03-30T04:55:44.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.255+0000 7fbcdf000700 1 Processor -- start 2024-03-30T04:55:44.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.256+0000 7fbcdf000700 1 -- start start 2024-03-30T04:55:44.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.256+0000 7fbcdf000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbcd8102e00 0x7fbcd80702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:55:44.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.256+0000 7fbcdf000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbcd81037a0 0x7fbcd8070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:55:44.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.256+0000 7fbcdf000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd8070d90 con 0x7fbcd81037a0 2024-03-30T04:55:44.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.256+0000 7fbcdf000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd8070ed0 con 0x7fbcd8102e00 2024-03-30T04:55:44.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.256+0000 7fbcd7400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbcd81037a0 0x7fbcd8070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:55:44.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.256+0000 7fbcd7400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbcd81037a0 0x7fbcd8070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60362/0 (socket says 172.21.15.123:60362) 2024-03-30T04:55:44.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.257+0000 7fbcd7400700 1 -- 172.21.15.123:0/1528238602 learned_addr learned my addr 172.21.15.123:0/1528238602 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:55:44.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.257+0000 7fbcd7e00700 1 --2- 172.21.15.123:0/1528238602 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbcd8102e00 0x7fbcd80702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:55:44.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.257+0000 7fbcd7400700 1 -- 172.21.15.123:0/1528238602 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbcd8102e00 msgr2=0x7fbcd80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:44.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.257+0000 7fbcd7400700 1 --2- 172.21.15.123:0/1528238602 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbcd8102e00 0x7fbcd80702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:44.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.257+0000 7fbcd7400700 1 -- 172.21.15.123:0/1528238602 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcc8005040 con 0x7fbcd81037a0 2024-03-30T04:55:44.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.257+0000 7fbcd7400700 1 --2- 172.21.15.123:0/1528238602 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbcd81037a0 0x7fbcd8070800 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4002a50 tx=0x7fbcc4004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:55:44.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.257+0000 7fbccfe00700 1 -- 172.21.15.123:0/1528238602 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbcc4007cc0 con 0x7fbcd81037a0 2024-03-30T04:55:44.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.257+0000 7fbccfe00700 1 -- 172.21.15.123:0/1528238602 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbcc400f040 con 0x7fbcd81037a0 2024-03-30T04:55:44.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.257+0000 7fbcdf000700 1 -- 172.21.15.123:0/1528238602 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcd8119e20 con 0x7fbcd81037a0 2024-03-30T04:55:44.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.257+0000 7fbccfe00700 1 -- 172.21.15.123:0/1528238602 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbcc400d920 con 0x7fbcd81037a0 2024-03-30T04:55:44.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.257+0000 7fbcdf000700 1 -- 172.21.15.123:0/1528238602 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcd811a2a0 con 0x7fbcd81037a0 2024-03-30T04:55:44.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.258+0000 7fbcdf000700 1 -- 172.21.15.123:0/1528238602 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcd80fe550 con 0x7fbcd81037a0 2024-03-30T04:55:44.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.260+0000 7fbccfe00700 1 -- 172.21.15.123:0/1528238602 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fbcc4009680 con 0x7fbcd81037a0 2024-03-30T04:55:44.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.261+0000 7fbccfe00700 1 --2- 172.21.15.123:0/1528238602 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbcbc06cc30 0x7fbcbc06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:55:44.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.261+0000 7fbccfe00700 1 -- 172.21.15.123:0/1528238602 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fbcc408bcf0 con 0x7fbcd81037a0 2024-03-30T04:55:44.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.261+0000 7fbcd7e00700 1 --2- 172.21.15.123:0/1528238602 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbcbc06cc30 0x7fbcbc06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:55:44.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.261+0000 7fbcd7e00700 1 --2- 172.21.15.123:0/1528238602 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbcbc06cc30 0x7fbcbc06f0e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fbcc8009fd0 tx=0x7fbcc8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:55:44.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.265+0000 7fbccfe00700 1 -- 172.21.15.123:0/1528238602 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fbcc4055cb0 con 0x7fbcd81037a0 2024-03-30T04:55:44.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.550+0000 7fbcdf000700 1 -- 172.21.15.123:0/1528238602 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fbcd804bed0 con 0x7fbcd81037a0 2024-03-30T04:55:44.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.551+0000 7fbccfe00700 1 -- 172.21.15.123:0/1528238602 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7fbcc40592d0 con 0x7fbcd81037a0 2024-03-30T04:55:44.550 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:55:44.550 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:55:44.550 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:55:44.550 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:55:44.550 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:55:44.550 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:55:44.551 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:55:44.551 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:55:44.551 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T04:55:44.551 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:55:44.551 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:55:44.551 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 4 2024-03-30T04:55:44.551 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:55:44.551 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:55:44.551 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 14 2024-03-30T04:55:44.552 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:55:44.552 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:55:44.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.553+0000 7fbcdf000700 1 -- 172.21.15.123:0/1528238602 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbcbc06cc30 msgr2=0x7fbcbc06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:44.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.553+0000 7fbcdf000700 1 --2- 172.21.15.123:0/1528238602 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbcbc06cc30 0x7fbcbc06f0e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fbcc8009fd0 tx=0x7fbcc8009e30 comp rx=0 tx=0).stop 2024-03-30T04:55:44.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.553+0000 7fbcdf000700 1 -- 172.21.15.123:0/1528238602 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbcd81037a0 msgr2=0x7fbcd8070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:44.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.553+0000 7fbcdf000700 1 --2- 172.21.15.123:0/1528238602 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbcd81037a0 0x7fbcd8070800 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4002a50 tx=0x7fbcc4004600 comp rx=0 tx=0).stop 2024-03-30T04:55:44.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.553+0000 7fbcdf000700 1 -- 172.21.15.123:0/1528238602 shutdown_connections 2024-03-30T04:55:44.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.553+0000 7fbcdf000700 1 --2- 172.21.15.123:0/1528238602 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbcbc06cc30 0x7fbcbc06f0e0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:44.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.553+0000 7fbcdf000700 1 --2- 172.21.15.123:0/1528238602 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbcd8102e00 0x7fbcd80702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:44.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.553+0000 7fbcdf000700 1 --2- 172.21.15.123:0/1528238602 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbcd81037a0 0x7fbcd8070800 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:44.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.553+0000 7fbcdf000700 1 -- 172.21.15.123:0/1528238602 >> 172.21.15.123:0/1528238602 conn(0x7fbcd80f89b0 msgr2=0x7fbcd80f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:55:44.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.553+0000 7fbcdf000700 1 -- 172.21.15.123:0/1528238602 shutdown_connections 2024-03-30T04:55:44.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.553+0000 7fbcdf000700 1 -- 172.21.15.123:0/1528238602 wait complete. 2024-03-30T04:55:44.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.688+0000 7fac29a00700 1 -- 172.21.15.123:0/2874886787 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac241037c0 msgr2=0x7fac240fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:44.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.688+0000 7fac29a00700 1 --2- 172.21.15.123:0/2874886787 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac241037c0 0x7fac240fce30 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fac10004940 tx=0x7fac10009e30 comp rx=0 tx=0).stop 2024-03-30T04:55:44.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.688+0000 7fac29a00700 1 -- 172.21.15.123:0/2874886787 shutdown_connections 2024-03-30T04:55:44.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.688+0000 7fac29a00700 1 --2- 172.21.15.123:0/2874886787 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac241037c0 0x7fac240fce30 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:44.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.688+0000 7fac29a00700 1 --2- 172.21.15.123:0/2874886787 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac24102e20 0x7fac241031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:44.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.688+0000 7fac29a00700 1 -- 172.21.15.123:0/2874886787 >> 172.21.15.123:0/2874886787 conn(0x7fac240f89d0 msgr2=0x7fac240fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:55:44.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.688+0000 7fac29a00700 1 -- 172.21.15.123:0/2874886787 shutdown_connections 2024-03-30T04:55:44.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.688+0000 7fac29a00700 1 -- 172.21.15.123:0/2874886787 wait complete. 2024-03-30T04:55:44.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.689+0000 7fac29a00700 1 Processor -- start 2024-03-30T04:55:44.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.689+0000 7fac29a00700 1 -- start start 2024-03-30T04:55:44.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.689+0000 7fac29a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac24102e20 0x7fac24115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:55:44.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.689+0000 7fac29a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac241037c0 0x7fac24116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:55:44.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.690+0000 7fac29a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac24116860 con 0x7fac24102e20 2024-03-30T04:55:44.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.690+0000 7fac29a00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac2410fc30 con 0x7fac241037c0 2024-03-30T04:55:44.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.690+0000 7fac22a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac24102e20 0x7fac24115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:55:44.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.690+0000 7fac22a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac24102e20 0x7fac24115cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60376/0 (socket says 172.21.15.123:60376) 2024-03-30T04:55:44.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.690+0000 7fac22a00700 1 -- 172.21.15.123:0/1355704351 learned_addr learned my addr 172.21.15.123:0/1355704351 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:55:44.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.690+0000 7fac22000700 1 --2- 172.21.15.123:0/1355704351 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac241037c0 0x7fac24116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:55:44.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.690+0000 7fac22a00700 1 -- 172.21.15.123:0/1355704351 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac241037c0 msgr2=0x7fac24116210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:44.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.690+0000 7fac22a00700 1 --2- 172.21.15.123:0/1355704351 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac241037c0 0x7fac24116210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:44.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.690+0000 7fac22a00700 1 -- 172.21.15.123:0/1355704351 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac10005040 con 0x7fac24102e20 2024-03-30T04:55:44.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.690+0000 7fac22000700 1 --2- 172.21.15.123:0/1355704351 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac241037c0 0x7fac24116210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:55:44.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.690+0000 7fac22a00700 1 --2- 172.21.15.123:0/1355704351 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac24102e20 0x7fac24115cd0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fac14002a50 tx=0x7fac140046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:55:44.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.691+0000 7fac1aa00700 1 -- 172.21.15.123:0/1355704351 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac14007cc0 con 0x7fac24102e20 2024-03-30T04:55:44.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.691+0000 7fac1aa00700 1 -- 172.21.15.123:0/1355704351 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fac1400f040 con 0x7fac24102e20 2024-03-30T04:55:44.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.691+0000 7fac1aa00700 1 -- 172.21.15.123:0/1355704351 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac14009540 con 0x7fac24102e20 2024-03-30T04:55:44.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.691+0000 7fac29a00700 1 -- 172.21.15.123:0/1355704351 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac2410ff10 con 0x7fac24102e20 2024-03-30T04:55:44.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.691+0000 7fac29a00700 1 -- 172.21.15.123:0/1355704351 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac241104d0 con 0x7fac24102e20 2024-03-30T04:55:44.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.692+0000 7fac29a00700 1 -- 172.21.15.123:0/1355704351 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac240fe570 con 0x7fac24102e20 2024-03-30T04:55:44.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.693+0000 7fac1aa00700 1 -- 172.21.15.123:0/1355704351 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fac140096a0 con 0x7fac24102e20 2024-03-30T04:55:44.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.694+0000 7fac1aa00700 1 --2- 172.21.15.123:0/1355704351 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fac0806cc30 0x7fac0806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:55:44.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.694+0000 7fac1aa00700 1 -- 172.21.15.123:0/1355704351 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fac1408bde0 con 0x7fac24102e20 2024-03-30T04:55:44.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.694+0000 7fac22000700 1 --2- 172.21.15.123:0/1355704351 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fac0806cc30 0x7fac0806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:55:44.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.695+0000 7fac22000700 1 --2- 172.21.15.123:0/1355704351 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fac0806cc30 0x7fac0806f0e0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fac24111570 tx=0x7fac10009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:55:44.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.699+0000 7fac1aa00700 1 -- 172.21.15.123:0/1355704351 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fac14059830 con 0x7fac24102e20 2024-03-30T04:55:44.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.942+0000 7fac29a00700 1 -- 172.21.15.123:0/1355704351 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fac2404bed0 con 0x7fac24102e20 2024-03-30T04:55:44.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.943+0000 7fac1aa00700 1 -- 172.21.15.123:0/1355704351 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 34 v34) v1 ==== 76+0+1843 (secure 0 0 0) 0x7fac140593c0 con 0x7fac24102e20 2024-03-30T04:55:44.942 INFO:teuthology.orchestra.run.smithi123.stdout:e34 2024-03-30T04:55:44.942 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:55:44.942 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:55:44.942 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:55:44.942 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:55:44.942 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:55:44.942 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:55:44.942 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T04:55:44.943 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T04:55:44.943 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:55:44.943 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T04:55:44.943 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:55:44.943 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:55:44.943 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:55:44.943 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:55:44.943 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:55:44.944 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T04:55:44.944 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:55:44.944 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:55:44.944 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T04:55:44.944 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:55:44.944 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T04:55:44.944 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T04:55:44.944 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T04:55:44.945 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T04:55:44.945 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:55:44.945 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:55:44.945 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:55:44.945 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:55:44.945 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:55:44.945 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:55:44.945 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T04:55:44.946 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T04:55:44.946 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:55:44.946 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:55:44.946 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T04:55:44.946 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:55:44.946 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:55:44.946 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:55:44.946 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:55:44.946 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44975} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/3788988014,v1:172.21.15.135:6827/3788988014] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:55:44.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.945+0000 7fac29a00700 1 -- 172.21.15.123:0/1355704351 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fac0806cc30 msgr2=0x7fac0806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:44.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.945+0000 7fac29a00700 1 --2- 172.21.15.123:0/1355704351 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fac0806cc30 0x7fac0806f0e0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fac24111570 tx=0x7fac10009e30 comp rx=0 tx=0).stop 2024-03-30T04:55:44.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.945+0000 7fac29a00700 1 -- 172.21.15.123:0/1355704351 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac24102e20 msgr2=0x7fac24115cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:44.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.945+0000 7fac29a00700 1 --2- 172.21.15.123:0/1355704351 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac24102e20 0x7fac24115cd0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fac14002a50 tx=0x7fac140046b0 comp rx=0 tx=0).stop 2024-03-30T04:55:44.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.945+0000 7fac29a00700 1 -- 172.21.15.123:0/1355704351 shutdown_connections 2024-03-30T04:55:44.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.945+0000 7fac29a00700 1 --2- 172.21.15.123:0/1355704351 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fac0806cc30 0x7fac0806f0e0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:44.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.945+0000 7fac29a00700 1 --2- 172.21.15.123:0/1355704351 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac24102e20 0x7fac24115cd0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:44.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.945+0000 7fac29a00700 1 --2- 172.21.15.123:0/1355704351 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac241037c0 0x7fac24116210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:44.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.945+0000 7fac29a00700 1 -- 172.21.15.123:0/1355704351 >> 172.21.15.123:0/1355704351 conn(0x7fac240f89d0 msgr2=0x7fac240706a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:55:44.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.945+0000 7fac29a00700 1 -- 172.21.15.123:0/1355704351 shutdown_connections 2024-03-30T04:55:44.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:44.946+0000 7fac29a00700 1 -- 172.21.15.123:0/1355704351 wait complete. 2024-03-30T04:55:44.949 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 34 2024-03-30T04:55:45.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.083+0000 7f386da00700 1 -- 172.21.15.123:0/164682081 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3868102e10 msgr2=0x7f38681031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:45.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.083+0000 7f386da00700 1 --2- 172.21.15.123:0/164682081 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3868102e10 0x7f38681031e0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f3858004940 tx=0x7f3858009e30 comp rx=0 tx=0).stop 2024-03-30T04:55:45.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.083+0000 7f386da00700 1 -- 172.21.15.123:0/164682081 shutdown_connections 2024-03-30T04:55:45.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.083+0000 7f386da00700 1 --2- 172.21.15.123:0/164682081 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f38681037b0 0x7f38680fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:45.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.083+0000 7f386da00700 1 --2- 172.21.15.123:0/164682081 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3868102e10 0x7f38681031e0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:45.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.083+0000 7f386da00700 1 -- 172.21.15.123:0/164682081 >> 172.21.15.123:0/164682081 conn(0x7f38680f89a0 msgr2=0x7f38680fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:55:45.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.084+0000 7f386da00700 1 -- 172.21.15.123:0/164682081 shutdown_connections 2024-03-30T04:55:45.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.084+0000 7f386da00700 1 -- 172.21.15.123:0/164682081 wait complete. 2024-03-30T04:55:45.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.085+0000 7f386da00700 1 Processor -- start 2024-03-30T04:55:45.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.085+0000 7f386da00700 1 -- start start 2024-03-30T04:55:45.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.085+0000 7f386da00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3868102e10 0x7f38680702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:55:45.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.085+0000 7f386da00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f38681037b0 0x7f3868070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:55:45.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.085+0000 7f386da00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3868070d90 con 0x7f3868102e10 2024-03-30T04:55:45.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.085+0000 7f386da00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3868070ed0 con 0x7f38681037b0 2024-03-30T04:55:45.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.086+0000 7f3866a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3868102e10 0x7f38680702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:55:45.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.086+0000 7f3866a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3868102e10 0x7f38680702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60396/0 (socket says 172.21.15.123:60396) 2024-03-30T04:55:45.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.086+0000 7f3866a00700 1 -- 172.21.15.123:0/634106935 learned_addr learned my addr 172.21.15.123:0/634106935 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:55:45.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.086+0000 7f3866a00700 1 -- 172.21.15.123:0/634106935 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f38681037b0 msgr2=0x7f3868070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:45.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.086+0000 7f3866a00700 1 --2- 172.21.15.123:0/634106935 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f38681037b0 0x7f3868070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:45.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.086+0000 7f3866a00700 1 -- 172.21.15.123:0/634106935 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3858005040 con 0x7f3868102e10 2024-03-30T04:55:45.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.086+0000 7f3866a00700 1 --2- 172.21.15.123:0/634106935 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3868102e10 0x7f38680702c0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f3858007f50 tx=0x7f3858009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:55:45.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.087+0000 7f385ea00700 1 -- 172.21.15.123:0/634106935 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3858021070 con 0x7f3868102e10 2024-03-30T04:55:45.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.087+0000 7f386da00700 1 -- 172.21.15.123:0/634106935 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3868071120 con 0x7f3868102e10 2024-03-30T04:55:45.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.087+0000 7f385ea00700 1 -- 172.21.15.123:0/634106935 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3858013e00 con 0x7f3868102e10 2024-03-30T04:55:45.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.087+0000 7f385ea00700 1 -- 172.21.15.123:0/634106935 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f385800e070 con 0x7f3868102e10 2024-03-30T04:55:45.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.087+0000 7f386da00700 1 -- 172.21.15.123:0/634106935 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f386811a180 con 0x7f3868102e10 2024-03-30T04:55:45.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.088+0000 7f386da00700 1 -- 172.21.15.123:0/634106935 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38680fe560 con 0x7f3868102e10 2024-03-30T04:55:45.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.089+0000 7f385ea00700 1 -- 172.21.15.123:0/634106935 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f385801c500 con 0x7f3868102e10 2024-03-30T04:55:45.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.090+0000 7f385ea00700 1 --2- 172.21.15.123:0/634106935 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f384c06cc30 0x7f384c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:55:45.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.090+0000 7f385ea00700 1 -- 172.21.15.123:0/634106935 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f385808d260 con 0x7f3868102e10 2024-03-30T04:55:45.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.090+0000 7f3866000700 1 --2- 172.21.15.123:0/634106935 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f384c06cc30 0x7f384c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:55:45.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.091+0000 7f3866000700 1 --2- 172.21.15.123:0/634106935 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f384c06cc30 0x7f384c06f0e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f3868071c00 tx=0x7f385400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:55:45.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.094+0000 7f385ea00700 1 -- 172.21.15.123:0/634106935 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f38580571a0 con 0x7f3868102e10 2024-03-30T04:55:45.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:55:44 smithi123 ceph-mon[139369]: from='client.35356 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:55:45.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:55:44 smithi123 ceph-mon[139369]: pgmap v737: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:45.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:55:44 smithi123 ceph-mon[139369]: from='client.35360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:55:45.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:55:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:55:45.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:55:44 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/1528238602' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:55:45.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:55:44 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/1355704351' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:55:45.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.317+0000 7f386da00700 1 -- 172.21.15.123:0/634106935 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f386804bf40 con 0x7f384c06cc30 2024-03-30T04:55:45.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.318+0000 7f385ea00700 1 -- 172.21.15.123:0/634106935 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f386804bf40 con 0x7f384c06cc30 2024-03-30T04:55:45.317 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:55:45.317 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:55:45.317 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:55:45.317 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T04:55:45.317 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:55:45.317 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T04:55:45.317 INFO:teuthology.orchestra.run.smithi123.stdout: "mds", 2024-03-30T04:55:45.317 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T04:55:45.317 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T04:55:45.318 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T04:55:45.318 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:55:45.318 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "16/21 daemons upgraded", 2024-03-30T04:55:45.318 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T04:55:45.318 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:55:45.318 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:55:45.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.321+0000 7f386da00700 1 -- 172.21.15.123:0/634106935 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f384c06cc30 msgr2=0x7f384c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:45.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.321+0000 7f386da00700 1 --2- 172.21.15.123:0/634106935 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f384c06cc30 0x7f384c06f0e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f3868071c00 tx=0x7f385400b040 comp rx=0 tx=0).stop 2024-03-30T04:55:45.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.321+0000 7f386da00700 1 -- 172.21.15.123:0/634106935 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3868102e10 msgr2=0x7f38680702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:45.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.321+0000 7f386da00700 1 --2- 172.21.15.123:0/634106935 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3868102e10 0x7f38680702c0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f3858007f50 tx=0x7f3858009e30 comp rx=0 tx=0).stop 2024-03-30T04:55:45.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.321+0000 7f386da00700 1 -- 172.21.15.123:0/634106935 shutdown_connections 2024-03-30T04:55:45.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.321+0000 7f386da00700 1 --2- 172.21.15.123:0/634106935 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f384c06cc30 0x7f384c06f0e0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:45.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.321+0000 7f386da00700 1 --2- 172.21.15.123:0/634106935 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3868102e10 0x7f38680702c0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:45.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.321+0000 7f386da00700 1 --2- 172.21.15.123:0/634106935 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f38681037b0 0x7f3868070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:45.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.321+0000 7f386da00700 1 -- 172.21.15.123:0/634106935 >> 172.21.15.123:0/634106935 conn(0x7f38680f89a0 msgr2=0x7f38680f99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:55:45.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.321+0000 7f386da00700 1 -- 172.21.15.123:0/634106935 shutdown_connections 2024-03-30T04:55:45.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.321+0000 7f386da00700 1 -- 172.21.15.123:0/634106935 wait complete. 2024-03-30T04:55:45.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:55:44 smithi135 ceph-mon[106011]: from='client.35356 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:55:45.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:55:44 smithi135 ceph-mon[106011]: pgmap v737: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:45.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:55:44 smithi135 ceph-mon[106011]: from='client.35360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:55:45.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:55:44 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:55:45.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:55:44 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/1528238602' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:55:45.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:55:44 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/1355704351' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:55:45.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.455+0000 7f4003a00700 1 -- 172.21.15.123:0/4169391262 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ffc0fe830 msgr2=0x7f3ffc106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:45.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.455+0000 7f4003a00700 1 --2- 172.21.15.123:0/4169391262 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ffc0fe830 0x7f3ffc106f00 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f3fe8004940 tx=0x7f3fe8009e30 comp rx=0 tx=0).stop 2024-03-30T04:55:45.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.455+0000 7f4003a00700 1 -- 172.21.15.123:0/4169391262 shutdown_connections 2024-03-30T04:55:45.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.455+0000 7f4003a00700 1 --2- 172.21.15.123:0/4169391262 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ffc0fe830 0x7f3ffc106f00 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:45.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.455+0000 7f4003a00700 1 --2- 172.21.15.123:0/4169391262 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3ffc0fde90 0x7f3ffc0fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:45.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.455+0000 7f4003a00700 1 -- 172.21.15.123:0/4169391262 >> 172.21.15.123:0/4169391262 conn(0x7f3ffc072bc0 msgr2=0x7f3ffc072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:55:45.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.455+0000 7f4003a00700 1 -- 172.21.15.123:0/4169391262 shutdown_connections 2024-03-30T04:55:45.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.455+0000 7f4003a00700 1 -- 172.21.15.123:0/4169391262 wait complete. 2024-03-30T04:55:45.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.456+0000 7f4003a00700 1 Processor -- start 2024-03-30T04:55:45.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.456+0000 7f4003a00700 1 -- start start 2024-03-30T04:55:45.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.457+0000 7f4003a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3ffc0fde90 0x7f3ffc0fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:55:45.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.457+0000 7f4003a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ffc0fe830 0x7f3ffc0fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:55:45.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.457+0000 7f4003a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ffc0fc820 con 0x7f3ffc0fe830 2024-03-30T04:55:45.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.457+0000 7f4003a00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ffc0fc960 con 0x7f3ffc0fde90 2024-03-30T04:55:45.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.457+0000 7f3ffbe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ffc0fe830 0x7f3ffc0fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:55:45.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.457+0000 7f3ffbe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ffc0fe830 0x7f3ffc0fc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:60402/0 (socket says 172.21.15.123:60402) 2024-03-30T04:55:45.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.457+0000 7f3ffbe00700 1 -- 172.21.15.123:0/2609426477 learned_addr learned my addr 172.21.15.123:0/2609426477 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:55:45.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.457+0000 7f4001000700 1 --2- 172.21.15.123:0/2609426477 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3ffc0fde90 0x7f3ffc0fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:55:45.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.457+0000 7f3ffbe00700 1 -- 172.21.15.123:0/2609426477 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3ffc0fde90 msgr2=0x7f3ffc0fbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:45.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.457+0000 7f3ffbe00700 1 --2- 172.21.15.123:0/2609426477 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3ffc0fde90 0x7f3ffc0fbd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:45.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.457+0000 7f3ffbe00700 1 -- 172.21.15.123:0/2609426477 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe8005040 con 0x7f3ffc0fe830 2024-03-30T04:55:45.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.458+0000 7f3ffbe00700 1 --2- 172.21.15.123:0/2609426477 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ffc0fe830 0x7f3ffc0fc290 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f3fe8000c00 tx=0x7f3fe8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:55:45.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.458+0000 7f3ff8c00700 1 -- 172.21.15.123:0/2609426477 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3fe800e040 con 0x7f3ffc0fe830 2024-03-30T04:55:45.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.458+0000 7f3ff8c00700 1 -- 172.21.15.123:0/2609426477 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3fe800bed0 con 0x7f3ffc0fe830 2024-03-30T04:55:45.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.458+0000 7f3ff8c00700 1 -- 172.21.15.123:0/2609426477 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3fe801c3f0 con 0x7f3ffc0fe830 2024-03-30T04:55:45.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.458+0000 7f4003a00700 1 -- 172.21.15.123:0/2609426477 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ffc0fcbb0 con 0x7f3ffc0fe830 2024-03-30T04:55:45.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.458+0000 7f4003a00700 1 -- 172.21.15.123:0/2609426477 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ffc11a140 con 0x7f3ffc0fe830 2024-03-30T04:55:45.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.459+0000 7f4003a00700 1 -- 172.21.15.123:0/2609426477 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ffc065e30 con 0x7f3ffc0fe830 2024-03-30T04:55:45.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.460+0000 7f3ff8c00700 1 -- 172.21.15.123:0/2609426477 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f3fe8013420 con 0x7f3ffc0fe830 2024-03-30T04:55:45.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.461+0000 7f3ff8c00700 1 --2- 172.21.15.123:0/2609426477 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3fe406cc30 0x7f3fe406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:55:45.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.461+0000 7f3ff8c00700 1 -- 172.21.15.123:0/2609426477 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f3fe808d4c0 con 0x7f3ffc0fe830 2024-03-30T04:55:45.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.461+0000 7f4001000700 1 --2- 172.21.15.123:0/2609426477 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3fe406cc30 0x7f3fe406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:55:45.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.462+0000 7f4001000700 1 --2- 172.21.15.123:0/2609426477 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3fe406cc30 0x7f3fe406f0e0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f3ff0004f40 tx=0x7f3ff000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:55:45.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.466+0000 7f3ff8c00700 1 -- 172.21.15.123:0/2609426477 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3fe8057480 con 0x7f3ffc0fe830 2024-03-30T04:55:45.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.750+0000 7f4003a00700 1 -- 172.21.15.123:0/2609426477 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f3ffc05f5f0 con 0x7f3ffc0fe830 2024-03-30T04:55:45.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.751+0000 7f3ff8c00700 1 -- 172.21.15.123:0/2609426477 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f3fe805aaa0 con 0x7f3ffc0fe830 2024-03-30T04:55:45.749 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T04:55:45.750 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-30T04:55:45.750 INFO:teuthology.orchestra.run.smithi123.stdout: host smithi135 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi135/config 2024-03-30T04:55:45.750 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-30T04:55:45.750 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T04:55:45.750 INFO:teuthology.orchestra.run.smithi123.stdout:Traceback (most recent call last): 2024-03-30T04:55:45.750 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-30T04:55:45.750 INFO:teuthology.orchestra.run.smithi123.stdout: "__main__", mod_spec) 2024-03-30T04:55:45.750 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-30T04:55:45.750 INFO:teuthology.orchestra.run.smithi123.stdout: exec(code, run_globals) 2024-03-30T04:55:45.751 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10889, in 2024-03-30T04:55:45.751 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10877, in main 2024-03-30T04:55:45.751 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-30T04:55:45.751 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-30T04:55:45.751 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-30T04:55:45.751 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-30T04:55:45.751 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-03-30T04:55:45.751 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-30T04:55:45.751 INFO:teuthology.orchestra.run.smithi123.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T04:55:45.751 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T04:55:45.751 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T04:55:45.751 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T04:55:45.752 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T04:55:45.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.753+0000 7f4003a00700 1 -- 172.21.15.123:0/2609426477 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3fe406cc30 msgr2=0x7f3fe406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:45.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.753+0000 7f4003a00700 1 --2- 172.21.15.123:0/2609426477 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3fe406cc30 0x7f3fe406f0e0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f3ff0004f40 tx=0x7f3ff000b040 comp rx=0 tx=0).stop 2024-03-30T04:55:45.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.753+0000 7f4003a00700 1 -- 172.21.15.123:0/2609426477 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ffc0fe830 msgr2=0x7f3ffc0fc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:55:45.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.753+0000 7f4003a00700 1 --2- 172.21.15.123:0/2609426477 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ffc0fe830 0x7f3ffc0fc290 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f3fe8000c00 tx=0x7f3fe8009e30 comp rx=0 tx=0).stop 2024-03-30T04:55:45.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.753+0000 7f4003a00700 1 -- 172.21.15.123:0/2609426477 shutdown_connections 2024-03-30T04:55:45.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.753+0000 7f4003a00700 1 --2- 172.21.15.123:0/2609426477 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3fe406cc30 0x7f3fe406f0e0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:45.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.753+0000 7f4003a00700 1 --2- 172.21.15.123:0/2609426477 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3ffc0fde90 0x7f3ffc0fbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:45.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.753+0000 7f4003a00700 1 --2- 172.21.15.123:0/2609426477 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ffc0fe830 0x7f3ffc0fc290 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:55:45.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.753+0000 7f4003a00700 1 -- 172.21.15.123:0/2609426477 >> 172.21.15.123:0/2609426477 conn(0x7f3ffc072bc0 msgr2=0x7f3ffc103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:55:45.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.753+0000 7f4003a00700 1 -- 172.21.15.123:0/2609426477 shutdown_connections 2024-03-30T04:55:45.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:55:45.753+0000 7f4003a00700 1 -- 172.21.15.123:0/2609426477 wait complete. 2024-03-30T04:55:46.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:55:45 smithi123 ceph-mon[139369]: from='client.35364 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:55:46.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:55:45 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2609426477' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:55:46.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:55:45 smithi135 ceph-mon[106011]: from='client.35364 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:55:46.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:55:45 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2609426477' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:55:47.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:55:46 smithi123 ceph-mon[139369]: from='client.35376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:55:47.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:55:46 smithi123 ceph-mon[139369]: pgmap v738: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:47.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:55:46 smithi135 ceph-mon[106011]: from='client.35376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:55:47.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:55:46 smithi135 ceph-mon[106011]: pgmap v738: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:49.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:55:48 smithi135 ceph-mon[106011]: pgmap v739: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:49.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:55:48 smithi123 ceph-mon[139369]: pgmap v739: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:51.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:55:50 smithi135 ceph-mon[106011]: pgmap v740: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:51.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:55:50 smithi123 ceph-mon[139369]: pgmap v740: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:53.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:55:52 smithi135 ceph-mon[106011]: pgmap v741: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:53.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:55:52 smithi123 ceph-mon[139369]: pgmap v741: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:55.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:55:54 smithi135 ceph-mon[106011]: pgmap v742: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:55.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:55:54 smithi123 ceph-mon[139369]: pgmap v742: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:57.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:55:56 smithi135 ceph-mon[106011]: pgmap v743: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:57.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:55:56 smithi123 ceph-mon[139369]: pgmap v743: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:59.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:55:58 smithi135 ceph-mon[106011]: pgmap v744: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:55:59.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:55:58 smithi123 ceph-mon[139369]: pgmap v744: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:00.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:55:59 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:56:00.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:55:59 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:56:01.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:56:01 smithi135 ceph-mon[106011]: pgmap v745: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:01.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:56:01 smithi123 ceph-mon[139369]: pgmap v745: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:03.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:56:03 smithi135 ceph-mon[106011]: pgmap v746: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:03.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:56:03 smithi123 ceph-mon[139369]: pgmap v746: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:05.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:56:05 smithi135 ceph-mon[106011]: pgmap v747: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:05.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:56:05 smithi123 ceph-mon[139369]: pgmap v747: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:07.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:56:07 smithi135 ceph-mon[106011]: pgmap v748: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:07.469 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:56:07 smithi123 ceph-mon[139369]: pgmap v748: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:09.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:56:09 smithi135 ceph-mon[106011]: pgmap v749: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:09.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:56:09 smithi123 ceph-mon[139369]: pgmap v749: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:11.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:56:11 smithi135 ceph-mon[106011]: pgmap v750: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:11.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:56:11 smithi123 ceph-mon[139369]: pgmap v750: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:13.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:56:13 smithi135 ceph-mon[106011]: pgmap v751: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:13.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:56:13 smithi123 ceph-mon[139369]: pgmap v751: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:15.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:56:15 smithi135 ceph-mon[106011]: pgmap v752: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:15.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:56:15 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:56:15.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:56:15 smithi123 ceph-mon[139369]: pgmap v752: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:15.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:56:15 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:56:15.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:15.902+0000 7f3dedc00700 1 -- 172.21.15.123:0/427085461 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3de80fe830 msgr2=0x7f3de8106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:15.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:15.902+0000 7f3dedc00700 1 --2- 172.21.15.123:0/427085461 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3de80fe830 0x7f3de8106f00 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f3dd4004940 tx=0x7f3dd4009e30 comp rx=0 tx=0).stop 2024-03-30T04:56:15.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:15.902+0000 7f3dedc00700 1 -- 172.21.15.123:0/427085461 shutdown_connections 2024-03-30T04:56:15.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:15.902+0000 7f3dedc00700 1 --2- 172.21.15.123:0/427085461 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3de80fe830 0x7f3de8106f00 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:15.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:15.902+0000 7f3dedc00700 1 --2- 172.21.15.123:0/427085461 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3de80fde90 0x7f3de80fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:15.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:15.902+0000 7f3dedc00700 1 -- 172.21.15.123:0/427085461 >> 172.21.15.123:0/427085461 conn(0x7f3de8072bc0 msgr2=0x7f3de8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:56:15.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:15.903+0000 7f3dedc00700 1 -- 172.21.15.123:0/427085461 shutdown_connections 2024-03-30T04:56:15.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:15.903+0000 7f3dedc00700 1 -- 172.21.15.123:0/427085461 wait complete. 2024-03-30T04:56:15.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:15.903+0000 7f3dedc00700 1 Processor -- start 2024-03-30T04:56:15.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:15.904+0000 7f3dedc00700 1 -- start start 2024-03-30T04:56:15.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:15.904+0000 7f3dedc00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3de80fde90 0x7f3de8115d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:56:15.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:15.904+0000 7f3dedc00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3de80fe830 0x7f3de8116270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:56:15.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:15.904+0000 7f3dedc00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de81168c0 con 0x7f3de80fde90 2024-03-30T04:56:15.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:15.904+0000 7f3dedc00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de810fc90 con 0x7f3de80fe830 2024-03-30T04:56:15.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:15.904+0000 7f3de6a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3de80fde90 0x7f3de8115d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:56:15.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:15.904+0000 7f3de6a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3de80fde90 0x7f3de8115d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49038/0 (socket says 172.21.15.123:49038) 2024-03-30T04:56:15.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:15.904+0000 7f3de6a00700 1 -- 172.21.15.123:0/1734449705 learned_addr learned my addr 172.21.15.123:0/1734449705 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:56:15.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:15.904+0000 7f3de6000700 1 --2- 172.21.15.123:0/1734449705 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3de80fe830 0x7f3de8116270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:56:15.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:15.905+0000 7f3de6a00700 1 -- 172.21.15.123:0/1734449705 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3de80fe830 msgr2=0x7f3de8116270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:15.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:15.905+0000 7f3de6a00700 1 --2- 172.21.15.123:0/1734449705 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3de80fe830 0x7f3de8116270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:15.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:15.905+0000 7f3de6a00700 1 -- 172.21.15.123:0/1734449705 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dd4005040 con 0x7f3de80fde90 2024-03-30T04:56:15.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:15.905+0000 7f3de6a00700 1 --2- 172.21.15.123:0/1734449705 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3de80fde90 0x7f3de8115d30 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f3dd8002a50 tx=0x7f3dd80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:56:15.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:15.905+0000 7f3ddea00700 1 -- 172.21.15.123:0/1734449705 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3dd8007cc0 con 0x7f3de80fde90 2024-03-30T04:56:15.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:15.905+0000 7f3ddea00700 1 -- 172.21.15.123:0/1734449705 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3dd800f040 con 0x7f3de80fde90 2024-03-30T04:56:15.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:15.905+0000 7f3ddea00700 1 -- 172.21.15.123:0/1734449705 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3dd8009540 con 0x7f3de80fde90 2024-03-30T04:56:15.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:15.905+0000 7f3dedc00700 1 -- 172.21.15.123:0/1734449705 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3de810ff70 con 0x7f3de80fde90 2024-03-30T04:56:15.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:15.905+0000 7f3dedc00700 1 -- 172.21.15.123:0/1734449705 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3de81104b0 con 0x7f3de80fde90 2024-03-30T04:56:15.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:15.906+0000 7f3dedc00700 1 -- 172.21.15.123:0/1734449705 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3de8106a20 con 0x7f3de80fde90 2024-03-30T04:56:15.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:15.907+0000 7f3ddea00700 1 -- 172.21.15.123:0/1734449705 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f3dd80096a0 con 0x7f3de80fde90 2024-03-30T04:56:15.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:15.908+0000 7f3ddea00700 1 --2- 172.21.15.123:0/1734449705 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3dcc06cc30 0x7f3dcc06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:56:15.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:15.908+0000 7f3ddea00700 1 -- 172.21.15.123:0/1734449705 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f3dd808bde0 con 0x7f3de80fde90 2024-03-30T04:56:15.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:15.908+0000 7f3de6000700 1 --2- 172.21.15.123:0/1734449705 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3dcc06cc30 0x7f3dcc06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:56:15.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:15.909+0000 7f3de6000700 1 --2- 172.21.15.123:0/1734449705 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3dcc06cc30 0x7f3dcc06f0e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f3de81115d0 tx=0x7f3dd4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:56:15.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:15.913+0000 7f3ddea00700 1 -- 172.21.15.123:0/1734449705 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3dd8059830 con 0x7f3de80fde90 2024-03-30T04:56:16.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.134+0000 7f3dedc00700 1 -- 172.21.15.123:0/1734449705 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3de8109e40 con 0x7f3dcc06cc30 2024-03-30T04:56:16.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.135+0000 7f3ddea00700 1 -- 172.21.15.123:0/1734449705 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f3de8109e40 con 0x7f3dcc06cc30 2024-03-30T04:56:16.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.137+0000 7f3dedc00700 1 -- 172.21.15.123:0/1734449705 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3dcc06cc30 msgr2=0x7f3dcc06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:16.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.137+0000 7f3dedc00700 1 --2- 172.21.15.123:0/1734449705 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3dcc06cc30 0x7f3dcc06f0e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f3de81115d0 tx=0x7f3dd4009e30 comp rx=0 tx=0).stop 2024-03-30T04:56:16.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.138+0000 7f3dedc00700 1 -- 172.21.15.123:0/1734449705 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3de80fde90 msgr2=0x7f3de8115d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:16.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.138+0000 7f3dedc00700 1 --2- 172.21.15.123:0/1734449705 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3de80fde90 0x7f3de8115d30 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f3dd8002a50 tx=0x7f3dd80046b0 comp rx=0 tx=0).stop 2024-03-30T04:56:16.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.138+0000 7f3dedc00700 1 -- 172.21.15.123:0/1734449705 shutdown_connections 2024-03-30T04:56:16.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.138+0000 7f3dedc00700 1 --2- 172.21.15.123:0/1734449705 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3dcc06cc30 0x7f3dcc06f0e0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:16.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.138+0000 7f3dedc00700 1 --2- 172.21.15.123:0/1734449705 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3de80fde90 0x7f3de8115d30 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:16.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.138+0000 7f3dedc00700 1 --2- 172.21.15.123:0/1734449705 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3de80fe830 0x7f3de8116270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:16.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.138+0000 7f3dedc00700 1 -- 172.21.15.123:0/1734449705 >> 172.21.15.123:0/1734449705 conn(0x7f3de8072bc0 msgr2=0x7f3de80fb840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:56:16.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.138+0000 7f3dedc00700 1 -- 172.21.15.123:0/1734449705 shutdown_connections 2024-03-30T04:56:16.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.138+0000 7f3dedc00700 1 -- 172.21.15.123:0/1734449705 wait complete. 2024-03-30T04:56:16.150 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:56:16.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.272+0000 7fd37fe00700 1 -- 172.21.15.123:0/4025673402 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3801035e0 msgr2=0x7fd3800fcc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:16.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.272+0000 7fd37fe00700 1 --2- 172.21.15.123:0/4025673402 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3801035e0 0x7fd3800fcc50 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fd36c004940 tx=0x7fd36c009e30 comp rx=0 tx=0).stop 2024-03-30T04:56:16.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.272+0000 7fd37fe00700 1 -- 172.21.15.123:0/4025673402 shutdown_connections 2024-03-30T04:56:16.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.272+0000 7fd37fe00700 1 --2- 172.21.15.123:0/4025673402 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3801035e0 0x7fd3800fcc50 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:16.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.272+0000 7fd37fe00700 1 --2- 172.21.15.123:0/4025673402 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd380102c40 0x7fd380103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:16.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.272+0000 7fd37fe00700 1 -- 172.21.15.123:0/4025673402 >> 172.21.15.123:0/4025673402 conn(0x7fd3800f8830 msgr2=0x7fd3800fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:56:16.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.273+0000 7fd37fe00700 1 -- 172.21.15.123:0/4025673402 shutdown_connections 2024-03-30T04:56:16.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.273+0000 7fd37fe00700 1 -- 172.21.15.123:0/4025673402 wait complete. 2024-03-30T04:56:16.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.273+0000 7fd37fe00700 1 Processor -- start 2024-03-30T04:56:16.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.273+0000 7fd37fe00700 1 -- start start 2024-03-30T04:56:16.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.274+0000 7fd37fe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd380102c40 0x7fd380115b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:56:16.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.274+0000 7fd37fe00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd3801035e0 0x7fd380116040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:56:16.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.274+0000 7fd37fe00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd380116690 con 0x7fd380102c40 2024-03-30T04:56:16.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.274+0000 7fd37fe00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd38010fa80 con 0x7fd3801035e0 2024-03-30T04:56:16.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.274+0000 7fd37ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd380102c40 0x7fd380115b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:56:16.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.274+0000 7fd37ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd380102c40 0x7fd380115b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49054/0 (socket says 172.21.15.123:49054) 2024-03-30T04:56:16.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.274+0000 7fd37ea00700 1 -- 172.21.15.123:0/1939009487 learned_addr learned my addr 172.21.15.123:0/1939009487 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:56:16.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.274+0000 7fd37e000700 1 --2- 172.21.15.123:0/1939009487 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd3801035e0 0x7fd380116040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:56:16.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.274+0000 7fd37ea00700 1 -- 172.21.15.123:0/1939009487 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd3801035e0 msgr2=0x7fd380116040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:16.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.274+0000 7fd37ea00700 1 --2- 172.21.15.123:0/1939009487 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd3801035e0 0x7fd380116040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:16.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.274+0000 7fd37ea00700 1 -- 172.21.15.123:0/1939009487 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd36c005040 con 0x7fd380102c40 2024-03-30T04:56:16.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.275+0000 7fd37ea00700 1 --2- 172.21.15.123:0/1939009487 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd380102c40 0x7fd380115b00 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fd370002a50 tx=0x7fd3700046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:56:16.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.275+0000 7fd376a00700 1 -- 172.21.15.123:0/1939009487 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd370007cc0 con 0x7fd380102c40 2024-03-30T04:56:16.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.275+0000 7fd376a00700 1 -- 172.21.15.123:0/1939009487 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd37000f040 con 0x7fd380102c40 2024-03-30T04:56:16.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.275+0000 7fd376a00700 1 -- 172.21.15.123:0/1939009487 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd37000d9c0 con 0x7fd380102c40 2024-03-30T04:56:16.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.275+0000 7fd37fe00700 1 -- 172.21.15.123:0/1939009487 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd38010fd60 con 0x7fd380102c40 2024-03-30T04:56:16.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.275+0000 7fd37fe00700 1 -- 172.21.15.123:0/1939009487 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3801102f0 con 0x7fd380102c40 2024-03-30T04:56:16.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.276+0000 7fd37fe00700 1 -- 172.21.15.123:0/1939009487 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3800fe390 con 0x7fd380102c40 2024-03-30T04:56:16.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.277+0000 7fd376a00700 1 -- 172.21.15.123:0/1939009487 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fd37000db60 con 0x7fd380102c40 2024-03-30T04:56:16.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.278+0000 7fd376a00700 1 --2- 172.21.15.123:0/1939009487 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd36406cc30 0x7fd36406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:56:16.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.279+0000 7fd376a00700 1 -- 172.21.15.123:0/1939009487 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd37008b680 con 0x7fd380102c40 2024-03-30T04:56:16.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.279+0000 7fd37e000700 1 --2- 172.21.15.123:0/1939009487 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd36406cc30 0x7fd36406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:56:16.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.279+0000 7fd37e000700 1 --2- 172.21.15.123:0/1939009487 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd36406cc30 0x7fd36406f0e0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fd380110d60 tx=0x7fd36c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:56:16.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.283+0000 7fd376a00700 1 -- 172.21.15.123:0/1939009487 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd3700590d0 con 0x7fd380102c40 2024-03-30T04:56:16.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.502+0000 7fd37fe00700 1 -- 172.21.15.123:0/1939009487 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd380109c40 con 0x7fd36406cc30 2024-03-30T04:56:16.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.503+0000 7fd376a00700 1 -- 172.21.15.123:0/1939009487 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fd380109c40 con 0x7fd36406cc30 2024-03-30T04:56:16.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.505+0000 7fd37fe00700 1 -- 172.21.15.123:0/1939009487 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd36406cc30 msgr2=0x7fd36406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:16.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.505+0000 7fd37fe00700 1 --2- 172.21.15.123:0/1939009487 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd36406cc30 0x7fd36406f0e0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fd380110d60 tx=0x7fd36c009e30 comp rx=0 tx=0).stop 2024-03-30T04:56:16.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.505+0000 7fd37fe00700 1 -- 172.21.15.123:0/1939009487 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd380102c40 msgr2=0x7fd380115b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:16.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.505+0000 7fd37fe00700 1 --2- 172.21.15.123:0/1939009487 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd380102c40 0x7fd380115b00 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fd370002a50 tx=0x7fd3700046b0 comp rx=0 tx=0).stop 2024-03-30T04:56:16.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.505+0000 7fd37fe00700 1 -- 172.21.15.123:0/1939009487 shutdown_connections 2024-03-30T04:56:16.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.505+0000 7fd37fe00700 1 --2- 172.21.15.123:0/1939009487 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd36406cc30 0x7fd36406f0e0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:16.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.505+0000 7fd37fe00700 1 --2- 172.21.15.123:0/1939009487 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd380102c40 0x7fd380115b00 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:16.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.505+0000 7fd37fe00700 1 --2- 172.21.15.123:0/1939009487 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd3801035e0 0x7fd380116040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:16.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.505+0000 7fd37fe00700 1 -- 172.21.15.123:0/1939009487 >> 172.21.15.123:0/1939009487 conn(0x7fd3800f8830 msgr2=0x7fd3800f98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:56:16.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.506+0000 7fd37fe00700 1 -- 172.21.15.123:0/1939009487 shutdown_connections 2024-03-30T04:56:16.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.506+0000 7fd37fe00700 1 -- 172.21.15.123:0/1939009487 wait complete. 2024-03-30T04:56:16.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.639+0000 7f8b61400700 1 -- 172.21.15.123:0/445159038 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b5c102e00 msgr2=0x7f8b5c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:16.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.639+0000 7f8b61400700 1 --2- 172.21.15.123:0/445159038 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b5c102e00 0x7f8b5c1031d0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c004940 tx=0x7f8b4c009e30 comp rx=0 tx=0).stop 2024-03-30T04:56:16.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.639+0000 7f8b61400700 1 -- 172.21.15.123:0/445159038 shutdown_connections 2024-03-30T04:56:16.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.639+0000 7f8b61400700 1 --2- 172.21.15.123:0/445159038 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8b5c1037a0 0x7f8b5c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:16.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.639+0000 7f8b61400700 1 --2- 172.21.15.123:0/445159038 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b5c102e00 0x7f8b5c1031d0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:16.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.639+0000 7f8b61400700 1 -- 172.21.15.123:0/445159038 >> 172.21.15.123:0/445159038 conn(0x7f8b5c0f89b0 msgr2=0x7f8b5c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:56:16.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.639+0000 7f8b61400700 1 -- 172.21.15.123:0/445159038 shutdown_connections 2024-03-30T04:56:16.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.639+0000 7f8b61400700 1 -- 172.21.15.123:0/445159038 wait complete. 2024-03-30T04:56:16.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.640+0000 7f8b61400700 1 Processor -- start 2024-03-30T04:56:16.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.640+0000 7f8b61400700 1 -- start start 2024-03-30T04:56:16.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.640+0000 7f8b61400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8b5c102e00 0x7f8b5c115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:56:16.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.640+0000 7f8b61400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b5c1037a0 0x7f8b5c116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:56:16.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.640+0000 7f8b61400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b5c116860 con 0x7f8b5c1037a0 2024-03-30T04:56:16.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.640+0000 7f8b61400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b5c10fc30 con 0x7f8b5c102e00 2024-03-30T04:56:16.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.641+0000 7f8b53e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b5c1037a0 0x7f8b5c116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:56:16.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.641+0000 7f8b53e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b5c1037a0 0x7f8b5c116210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49080/0 (socket says 172.21.15.123:49080) 2024-03-30T04:56:16.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.641+0000 7f8b53e00700 1 -- 172.21.15.123:0/1150134571 learned_addr learned my addr 172.21.15.123:0/1150134571 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:56:16.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.641+0000 7f8b5a800700 1 --2- 172.21.15.123:0/1150134571 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8b5c102e00 0x7f8b5c115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:56:16.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.641+0000 7f8b53e00700 1 -- 172.21.15.123:0/1150134571 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8b5c102e00 msgr2=0x7f8b5c115cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:16.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.641+0000 7f8b53e00700 1 --2- 172.21.15.123:0/1150134571 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8b5c102e00 0x7f8b5c115cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:16.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.641+0000 7f8b53e00700 1 -- 172.21.15.123:0/1150134571 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b4c005040 con 0x7f8b5c1037a0 2024-03-30T04:56:16.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.641+0000 7f8b53e00700 1 --2- 172.21.15.123:0/1150134571 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b5c1037a0 0x7f8b5c116210 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f8b44002a50 tx=0x7f8b44004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:56:16.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.641+0000 7f8b52a00700 1 -- 172.21.15.123:0/1150134571 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b44007cc0 con 0x7f8b5c1037a0 2024-03-30T04:56:16.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.642+0000 7f8b52a00700 1 -- 172.21.15.123:0/1150134571 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8b4400f040 con 0x7f8b5c1037a0 2024-03-30T04:56:16.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.642+0000 7f8b61400700 1 -- 172.21.15.123:0/1150134571 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b5c10ff10 con 0x7f8b5c1037a0 2024-03-30T04:56:16.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.642+0000 7f8b52a00700 1 -- 172.21.15.123:0/1150134571 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b4400d920 con 0x7f8b5c1037a0 2024-03-30T04:56:16.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.642+0000 7f8b61400700 1 -- 172.21.15.123:0/1150134571 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b5c1104d0 con 0x7f8b5c1037a0 2024-03-30T04:56:16.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.643+0000 7f8b61400700 1 -- 172.21.15.123:0/1150134571 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b5c0fe550 con 0x7f8b5c1037a0 2024-03-30T04:56:16.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.644+0000 7f8b52a00700 1 -- 172.21.15.123:0/1150134571 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f8b44009680 con 0x7f8b5c1037a0 2024-03-30T04:56:16.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.644+0000 7f8b52a00700 1 --2- 172.21.15.123:0/1150134571 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8b4006cc30 0x7f8b4006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:56:16.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.645+0000 7f8b52a00700 1 -- 172.21.15.123:0/1150134571 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f8b4408bdb0 con 0x7f8b5c1037a0 2024-03-30T04:56:16.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.645+0000 7f8b5a800700 1 --2- 172.21.15.123:0/1150134571 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8b4006cc30 0x7f8b4006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:56:16.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.645+0000 7f8b5a800700 1 --2- 172.21.15.123:0/1150134571 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8b4006cc30 0x7f8b4006f0e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c009fd0 tx=0x7f8b4c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:56:16.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.649+0000 7f8b52a00700 1 -- 172.21.15.123:0/1150134571 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8b44059780 con 0x7f8b5c1037a0 2024-03-30T04:56:16.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.859+0000 7f8b61400700 1 -- 172.21.15.123:0/1150134571 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f8b5c109e30 con 0x7f8b4006cc30 2024-03-30T04:56:16.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.869+0000 7f8b52a00700 1 -- 172.21.15.123:0/1150134571 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f8b5c109e30 con 0x7f8b4006cc30 2024-03-30T04:56:16.867 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:56:16.867 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (25m) 2m ago 37m 14.6M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:56:16.867 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (22m) 2m ago 38m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T04:56:16.867 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (22m) 2m ago 37m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T04:56:16.868 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (25m) 2m ago 37m 73.5M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T04:56:16.868 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (18m) 2m ago 31m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T04:56:16.868 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (18m) 2m ago 31m 16.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T04:56:16.868 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (2m) 2m ago 31m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 87b073a6c9ca 2024-03-30T04:56:16.868 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (18m) 2m ago 31m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T04:56:16.868 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (27m) 2m ago 39m 510M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:56:16.868 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (26m) 2m ago 36m 456M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:56:16.868 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (23m) 2m ago 39m 70.0M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T04:56:16.868 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (23m) 2m ago 36m 51.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T04:56:16.868 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (26m) 2m ago 38m 8636k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:56:16.868 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (26m) 2m ago 36m 8678k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:56:16.868 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (22m) 2m ago 35m 107M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T04:56:16.868 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (21m) 2m ago 35m 97.2M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T04:56:16.868 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (21m) 2m ago 34m 89.8M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T04:56:16.869 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (20m) 2m ago 34m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T04:56:16.869 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (19m) 2m ago 33m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T04:56:16.869 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (19m) 2m ago 33m 88.8M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T04:56:16.869 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (25m) 2m ago 37m 63.9M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:56:16.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.871+0000 7f8b61400700 1 -- 172.21.15.123:0/1150134571 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8b4006cc30 msgr2=0x7f8b4006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:16.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.871+0000 7f8b61400700 1 --2- 172.21.15.123:0/1150134571 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8b4006cc30 0x7f8b4006f0e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c009fd0 tx=0x7f8b4c009e30 comp rx=0 tx=0).stop 2024-03-30T04:56:16.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.871+0000 7f8b61400700 1 -- 172.21.15.123:0/1150134571 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b5c1037a0 msgr2=0x7f8b5c116210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:16.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.871+0000 7f8b61400700 1 --2- 172.21.15.123:0/1150134571 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b5c1037a0 0x7f8b5c116210 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f8b44002a50 tx=0x7f8b44004600 comp rx=0 tx=0).stop 2024-03-30T04:56:16.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.871+0000 7f8b61400700 1 -- 172.21.15.123:0/1150134571 shutdown_connections 2024-03-30T04:56:16.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.871+0000 7f8b61400700 1 --2- 172.21.15.123:0/1150134571 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8b4006cc30 0x7f8b4006f0e0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:16.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.871+0000 7f8b61400700 1 --2- 172.21.15.123:0/1150134571 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8b5c102e00 0x7f8b5c115cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:16.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.871+0000 7f8b61400700 1 --2- 172.21.15.123:0/1150134571 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b5c1037a0 0x7f8b5c116210 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:16.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.871+0000 7f8b61400700 1 -- 172.21.15.123:0/1150134571 >> 172.21.15.123:0/1150134571 conn(0x7f8b5c0f89b0 msgr2=0x7f8b5c0f9980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:56:16.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.872+0000 7f8b61400700 1 -- 172.21.15.123:0/1150134571 shutdown_connections 2024-03-30T04:56:16.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:16.872+0000 7f8b61400700 1 -- 172.21.15.123:0/1150134571 wait complete. 2024-03-30T04:56:17.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.004+0000 7f0a60e00700 1 -- 172.21.15.123:0/2147698716 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a5c1037a0 msgr2=0x7f0a5c0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:17.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.004+0000 7f0a60e00700 1 --2- 172.21.15.123:0/2147698716 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a5c1037a0 0x7f0a5c0fce10 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f0a48004940 tx=0x7f0a48009e30 comp rx=0 tx=0).stop 2024-03-30T04:56:17.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.004+0000 7f0a60e00700 1 -- 172.21.15.123:0/2147698716 shutdown_connections 2024-03-30T04:56:17.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.004+0000 7f0a60e00700 1 --2- 172.21.15.123:0/2147698716 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a5c1037a0 0x7f0a5c0fce10 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:17.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.004+0000 7f0a60e00700 1 --2- 172.21.15.123:0/2147698716 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0a5c102e00 0x7f0a5c1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:17.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.004+0000 7f0a60e00700 1 -- 172.21.15.123:0/2147698716 >> 172.21.15.123:0/2147698716 conn(0x7f0a5c0f89b0 msgr2=0x7f0a5c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:56:17.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.004+0000 7f0a60e00700 1 -- 172.21.15.123:0/2147698716 shutdown_connections 2024-03-30T04:56:17.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.004+0000 7f0a60e00700 1 -- 172.21.15.123:0/2147698716 wait complete. 2024-03-30T04:56:17.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.005+0000 7f0a60e00700 1 Processor -- start 2024-03-30T04:56:17.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.005+0000 7f0a60e00700 1 -- start start 2024-03-30T04:56:17.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.006+0000 7f0a60e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0a5c102e00 0x7f0a5c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:56:17.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.006+0000 7f0a60e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a5c1037a0 0x7f0a5c070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:56:17.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.006+0000 7f0a60e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a5c070d90 con 0x7f0a5c1037a0 2024-03-30T04:56:17.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.006+0000 7f0a60e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a5c070ed0 con 0x7f0a5c102e00 2024-03-30T04:56:17.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.006+0000 7f0a59400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a5c1037a0 0x7f0a5c070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:56:17.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.006+0000 7f0a59400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a5c1037a0 0x7f0a5c070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49086/0 (socket says 172.21.15.123:49086) 2024-03-30T04:56:17.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.006+0000 7f0a59400700 1 -- 172.21.15.123:0/285390690 learned_addr learned my addr 172.21.15.123:0/285390690 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:56:17.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.006+0000 7f0a59e00700 1 --2- 172.21.15.123:0/285390690 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0a5c102e00 0x7f0a5c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:56:17.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.007+0000 7f0a59400700 1 -- 172.21.15.123:0/285390690 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0a5c102e00 msgr2=0x7f0a5c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:17.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.007+0000 7f0a59400700 1 --2- 172.21.15.123:0/285390690 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0a5c102e00 0x7f0a5c0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:17.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.007+0000 7f0a59400700 1 -- 172.21.15.123:0/285390690 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a48005040 con 0x7f0a5c1037a0 2024-03-30T04:56:17.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.007+0000 7f0a59e00700 1 --2- 172.21.15.123:0/285390690 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0a5c102e00 0x7f0a5c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:56:17.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.007+0000 7f0a59400700 1 --2- 172.21.15.123:0/285390690 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a5c1037a0 0x7f0a5c070800 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f0a48009390 tx=0x7f0a48009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:56:17.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.007+0000 7f0a52000700 1 -- 172.21.15.123:0/285390690 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a4800e040 con 0x7f0a5c1037a0 2024-03-30T04:56:17.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.007+0000 7f0a52000700 1 -- 172.21.15.123:0/285390690 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0a48013420 con 0x7f0a5c1037a0 2024-03-30T04:56:17.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.007+0000 7f0a60e00700 1 -- 172.21.15.123:0/285390690 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a5c071120 con 0x7f0a5c1037a0 2024-03-30T04:56:17.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.007+0000 7f0a52000700 1 -- 172.21.15.123:0/285390690 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a4801c630 con 0x7f0a5c1037a0 2024-03-30T04:56:17.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.007+0000 7f0a60e00700 1 -- 172.21.15.123:0/285390690 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a5c11a230 con 0x7f0a5c1037a0 2024-03-30T04:56:17.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.008+0000 7f0a60e00700 1 -- 172.21.15.123:0/285390690 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a5c0fe550 con 0x7f0a5c1037a0 2024-03-30T04:56:17.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.009+0000 7f0a52000700 1 -- 172.21.15.123:0/285390690 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f0a48013a00 con 0x7f0a5c1037a0 2024-03-30T04:56:17.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.010+0000 7f0a52000700 1 --2- 172.21.15.123:0/285390690 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0a4006cc30 0x7f0a4006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:56:17.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.010+0000 7f0a52000700 1 -- 172.21.15.123:0/285390690 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f0a4808d360 con 0x7f0a5c1037a0 2024-03-30T04:56:17.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.010+0000 7f0a59e00700 1 --2- 172.21.15.123:0/285390690 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0a4006cc30 0x7f0a4006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:56:17.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.011+0000 7f0a59e00700 1 --2- 172.21.15.123:0/285390690 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0a4006cc30 0x7f0a4006f0e0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c009fd0 tx=0x7f0a4c009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:56:17.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.014+0000 7f0a52000700 1 -- 172.21.15.123:0/285390690 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0a48057320 con 0x7f0a5c1037a0 2024-03-30T04:56:17.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.300+0000 7f0a60e00700 1 -- 172.21.15.123:0/285390690 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0a5c109dd0 con 0x7f0a5c1037a0 2024-03-30T04:56:17.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.301+0000 7f0a52000700 1 -- 172.21.15.123:0/285390690 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f0a4805a940 con 0x7f0a5c1037a0 2024-03-30T04:56:17.299 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:56:17.300 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:56:17.300 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:56:17.300 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:56:17.300 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:56:17.300 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:56:17.300 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:56:17.300 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:56:17.300 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T04:56:17.300 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:56:17.300 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:56:17.300 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 4 2024-03-30T04:56:17.300 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:56:17.300 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:56:17.301 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 14 2024-03-30T04:56:17.301 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:56:17.301 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:56:17.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.303+0000 7f0a60e00700 1 -- 172.21.15.123:0/285390690 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0a4006cc30 msgr2=0x7f0a4006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:17.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.303+0000 7f0a60e00700 1 --2- 172.21.15.123:0/285390690 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0a4006cc30 0x7f0a4006f0e0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c009fd0 tx=0x7f0a4c009ee0 comp rx=0 tx=0).stop 2024-03-30T04:56:17.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.303+0000 7f0a60e00700 1 -- 172.21.15.123:0/285390690 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a5c1037a0 msgr2=0x7f0a5c070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:17.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.303+0000 7f0a60e00700 1 --2- 172.21.15.123:0/285390690 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a5c1037a0 0x7f0a5c070800 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f0a48009390 tx=0x7f0a48009e30 comp rx=0 tx=0).stop 2024-03-30T04:56:17.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.304+0000 7f0a60e00700 1 -- 172.21.15.123:0/285390690 shutdown_connections 2024-03-30T04:56:17.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.304+0000 7f0a60e00700 1 --2- 172.21.15.123:0/285390690 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0a4006cc30 0x7f0a4006f0e0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:17.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.304+0000 7f0a60e00700 1 --2- 172.21.15.123:0/285390690 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0a5c102e00 0x7f0a5c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:17.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.304+0000 7f0a60e00700 1 --2- 172.21.15.123:0/285390690 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a5c1037a0 0x7f0a5c070800 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:17.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.304+0000 7f0a60e00700 1 -- 172.21.15.123:0/285390690 >> 172.21.15.123:0/285390690 conn(0x7f0a5c0f89b0 msgr2=0x7f0a5c0f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:56:17.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.304+0000 7f0a60e00700 1 -- 172.21.15.123:0/285390690 shutdown_connections 2024-03-30T04:56:17.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.304+0000 7f0a60e00700 1 -- 172.21.15.123:0/285390690 wait complete. 2024-03-30T04:56:17.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:56:17 smithi135 ceph-mon[106011]: pgmap v753: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:17.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.436+0000 7fc452200700 1 -- 172.21.15.123:0/3377519836 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc44c1037a0 msgr2=0x7fc44c0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:17.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.436+0000 7fc452200700 1 --2- 172.21.15.123:0/3377519836 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc44c1037a0 0x7fc44c0fce10 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fc438004940 tx=0x7fc438009e30 comp rx=0 tx=0).stop 2024-03-30T04:56:17.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.436+0000 7fc452200700 1 -- 172.21.15.123:0/3377519836 shutdown_connections 2024-03-30T04:56:17.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.436+0000 7fc452200700 1 --2- 172.21.15.123:0/3377519836 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc44c1037a0 0x7fc44c0fce10 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:17.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.436+0000 7fc452200700 1 --2- 172.21.15.123:0/3377519836 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc44c102e00 0x7fc44c1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:17.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.436+0000 7fc452200700 1 -- 172.21.15.123:0/3377519836 >> 172.21.15.123:0/3377519836 conn(0x7fc44c0f89b0 msgr2=0x7fc44c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:56:17.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.436+0000 7fc452200700 1 -- 172.21.15.123:0/3377519836 shutdown_connections 2024-03-30T04:56:17.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.436+0000 7fc452200700 1 -- 172.21.15.123:0/3377519836 wait complete. 2024-03-30T04:56:17.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.437+0000 7fc452200700 1 Processor -- start 2024-03-30T04:56:17.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.437+0000 7fc452200700 1 -- start start 2024-03-30T04:56:17.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.437+0000 7fc452200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc44c102e00 0x7fc44c115cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:56:17.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.437+0000 7fc452200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc44c1037a0 0x7fc44c1161f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:56:17.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.437+0000 7fc452200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc44c116840 con 0x7fc44c1037a0 2024-03-30T04:56:17.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.437+0000 7fc452200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc44c10fc10 con 0x7fc44c102e00 2024-03-30T04:56:17.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.437+0000 7fc44aa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc44c1037a0 0x7fc44c1161f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:56:17.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.437+0000 7fc44aa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc44c1037a0 0x7fc44c1161f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49098/0 (socket says 172.21.15.123:49098) 2024-03-30T04:56:17.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.437+0000 7fc44aa00700 1 -- 172.21.15.123:0/3870393790 learned_addr learned my addr 172.21.15.123:0/3870393790 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:56:17.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.438+0000 7fc44b400700 1 --2- 172.21.15.123:0/3870393790 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc44c102e00 0x7fc44c115cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:56:17.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.438+0000 7fc44aa00700 1 -- 172.21.15.123:0/3870393790 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc44c102e00 msgr2=0x7fc44c115cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:17.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.438+0000 7fc44aa00700 1 --2- 172.21.15.123:0/3870393790 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc44c102e00 0x7fc44c115cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:17.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.438+0000 7fc44aa00700 1 -- 172.21.15.123:0/3870393790 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc438005040 con 0x7fc44c1037a0 2024-03-30T04:56:17.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.438+0000 7fc44aa00700 1 --2- 172.21.15.123:0/3870393790 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc44c1037a0 0x7fc44c1161f0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fc438009fd0 tx=0x7fc438009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:56:17.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.438+0000 7fc443400700 1 -- 172.21.15.123:0/3870393790 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc43800e030 con 0x7fc44c1037a0 2024-03-30T04:56:17.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.438+0000 7fc443400700 1 -- 172.21.15.123:0/3870393790 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc438013e00 con 0x7fc44c1037a0 2024-03-30T04:56:17.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.438+0000 7fc452200700 1 -- 172.21.15.123:0/3870393790 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc44c10fe90 con 0x7fc44c1037a0 2024-03-30T04:56:17.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.438+0000 7fc443400700 1 -- 172.21.15.123:0/3870393790 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc43801ccd0 con 0x7fc44c1037a0 2024-03-30T04:56:17.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.438+0000 7fc452200700 1 -- 172.21.15.123:0/3870393790 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc44c110450 con 0x7fc44c1037a0 2024-03-30T04:56:17.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.439+0000 7fc452200700 1 -- 172.21.15.123:0/3870393790 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc44c0fe550 con 0x7fc44c1037a0 2024-03-30T04:56:17.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.440+0000 7fc443400700 1 -- 172.21.15.123:0/3870393790 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fc43801c500 con 0x7fc44c1037a0 2024-03-30T04:56:17.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.441+0000 7fc443400700 1 --2- 172.21.15.123:0/3870393790 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc43006cc30 0x7fc43006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:56:17.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.441+0000 7fc443400700 1 -- 172.21.15.123:0/3870393790 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc43808d980 con 0x7fc44c1037a0 2024-03-30T04:56:17.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.441+0000 7fc44b400700 1 --2- 172.21.15.123:0/3870393790 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc43006cc30 0x7fc43006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:56:17.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.442+0000 7fc44b400700 1 --2- 172.21.15.123:0/3870393790 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc43006cc30 0x7fc43006f0e0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fc43c009fd0 tx=0x7fc43c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:56:17.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.445+0000 7fc443400700 1 -- 172.21.15.123:0/3870393790 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc43805b3d0 con 0x7fc44c1037a0 2024-03-30T04:56:17.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:56:17 smithi123 ceph-mon[139369]: pgmap v753: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:17.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.689+0000 7fc452200700 1 -- 172.21.15.123:0/3870393790 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fc44c04bed0 con 0x7fc44c1037a0 2024-03-30T04:56:17.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.690+0000 7fc443400700 1 -- 172.21.15.123:0/3870393790 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 34 v34) v1 ==== 76+0+1843 (secure 0 0 0) 0x7fc438004490 con 0x7fc44c1037a0 2024-03-30T04:56:17.688 INFO:teuthology.orchestra.run.smithi123.stdout:e34 2024-03-30T04:56:17.688 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:56:17.688 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:56:17.689 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:56:17.689 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:56:17.689 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:56:17.689 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:56:17.689 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T04:56:17.689 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T04:56:17.689 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:56:17.689 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T04:56:17.689 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:56:17.689 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:56:17.689 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:56:17.689 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:56:17.689 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:56:17.690 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T04:56:17.690 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:56:17.690 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:56:17.690 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T04:56:17.690 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:56:17.690 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T04:56:17.690 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T04:56:17.690 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T04:56:17.690 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T04:56:17.690 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:56:17.690 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:56:17.690 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:56:17.690 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:56:17.690 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:56:17.690 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:56:17.691 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T04:56:17.691 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T04:56:17.691 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:56:17.691 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:56:17.691 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T04:56:17.691 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:56:17.691 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:56:17.691 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:56:17.691 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:56:17.691 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44975} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/3788988014,v1:172.21.15.135:6827/3788988014] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:56:17.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.692+0000 7fc452200700 1 -- 172.21.15.123:0/3870393790 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc43006cc30 msgr2=0x7fc43006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:17.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.692+0000 7fc452200700 1 --2- 172.21.15.123:0/3870393790 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc43006cc30 0x7fc43006f0e0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fc43c009fd0 tx=0x7fc43c00b040 comp rx=0 tx=0).stop 2024-03-30T04:56:17.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.692+0000 7fc452200700 1 -- 172.21.15.123:0/3870393790 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc44c1037a0 msgr2=0x7fc44c1161f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:17.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.692+0000 7fc452200700 1 --2- 172.21.15.123:0/3870393790 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc44c1037a0 0x7fc44c1161f0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fc438009fd0 tx=0x7fc438009e30 comp rx=0 tx=0).stop 2024-03-30T04:56:17.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.692+0000 7fc452200700 1 -- 172.21.15.123:0/3870393790 shutdown_connections 2024-03-30T04:56:17.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.692+0000 7fc452200700 1 --2- 172.21.15.123:0/3870393790 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc43006cc30 0x7fc43006f0e0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:17.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.692+0000 7fc452200700 1 --2- 172.21.15.123:0/3870393790 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc44c102e00 0x7fc44c115cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:17.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.692+0000 7fc452200700 1 --2- 172.21.15.123:0/3870393790 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc44c1037a0 0x7fc44c1161f0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:17.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.692+0000 7fc452200700 1 -- 172.21.15.123:0/3870393790 >> 172.21.15.123:0/3870393790 conn(0x7fc44c0f89b0 msgr2=0x7fc44c070480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:56:17.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.693+0000 7fc452200700 1 -- 172.21.15.123:0/3870393790 shutdown_connections 2024-03-30T04:56:17.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.693+0000 7fc452200700 1 -- 172.21.15.123:0/3870393790 wait complete. 2024-03-30T04:56:17.693 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 34 2024-03-30T04:56:17.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.827+0000 7f1c04200700 1 -- 172.21.15.123:0/2320276269 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1bfc1037a0 msgr2=0x7f1bfc0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:17.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.827+0000 7f1c04200700 1 --2- 172.21.15.123:0/2320276269 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1bfc1037a0 0x7f1bfc0fce10 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f1be8004940 tx=0x7f1be8009e30 comp rx=0 tx=0).stop 2024-03-30T04:56:17.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.827+0000 7f1c04200700 1 -- 172.21.15.123:0/2320276269 shutdown_connections 2024-03-30T04:56:17.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.827+0000 7f1c04200700 1 --2- 172.21.15.123:0/2320276269 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1bfc1037a0 0x7f1bfc0fce10 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:17.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.827+0000 7f1c04200700 1 --2- 172.21.15.123:0/2320276269 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1bfc102e00 0x7f1bfc1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:17.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.827+0000 7f1c04200700 1 -- 172.21.15.123:0/2320276269 >> 172.21.15.123:0/2320276269 conn(0x7f1bfc0f89b0 msgr2=0x7f1bfc0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:56:17.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.827+0000 7f1c04200700 1 -- 172.21.15.123:0/2320276269 shutdown_connections 2024-03-30T04:56:17.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.828+0000 7f1c04200700 1 -- 172.21.15.123:0/2320276269 wait complete. 2024-03-30T04:56:17.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.828+0000 7f1c04200700 1 Processor -- start 2024-03-30T04:56:17.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.828+0000 7f1c04200700 1 -- start start 2024-03-30T04:56:17.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.829+0000 7f1c04200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1bfc102e00 0x7f1bfc0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:56:17.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.829+0000 7f1c04200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1bfc1037a0 0x7f1bfc070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:56:17.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.829+0000 7f1c04200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bfc0720b0 con 0x7f1bfc102e00 2024-03-30T04:56:17.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.829+0000 7f1c04200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bfc072220 con 0x7f1bfc1037a0 2024-03-30T04:56:17.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.829+0000 7f1c01800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1bfc102e00 0x7f1bfc0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:56:17.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.829+0000 7f1c01800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1bfc102e00 0x7f1bfc0702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49120/0 (socket says 172.21.15.123:49120) 2024-03-30T04:56:17.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.829+0000 7f1c01800700 1 -- 172.21.15.123:0/879188175 learned_addr learned my addr 172.21.15.123:0/879188175 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:56:17.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.829+0000 7f1c00e00700 1 --2- 172.21.15.123:0/879188175 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1bfc1037a0 0x7f1bfc070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:56:17.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.829+0000 7f1c01800700 1 -- 172.21.15.123:0/879188175 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1bfc1037a0 msgr2=0x7f1bfc070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:17.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.829+0000 7f1c01800700 1 --2- 172.21.15.123:0/879188175 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1bfc1037a0 0x7f1bfc070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:17.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.830+0000 7f1c01800700 1 -- 172.21.15.123:0/879188175 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1be8005040 con 0x7f1bfc102e00 2024-03-30T04:56:17.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.830+0000 7f1c01800700 1 --2- 172.21.15.123:0/879188175 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1bfc102e00 0x7f1bfc0702c0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f1bf0002a80 tx=0x7f1bf00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:56:17.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.830+0000 7f1bf9600700 1 -- 172.21.15.123:0/879188175 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1bf0007cc0 con 0x7f1bfc102e00 2024-03-30T04:56:17.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.830+0000 7f1bf9600700 1 -- 172.21.15.123:0/879188175 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1bf000f040 con 0x7f1bfc102e00 2024-03-30T04:56:17.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.830+0000 7f1c04200700 1 -- 172.21.15.123:0/879188175 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bfc070e60 con 0x7f1bfc102e00 2024-03-30T04:56:17.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.830+0000 7f1bf9600700 1 -- 172.21.15.123:0/879188175 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1bf000da60 con 0x7f1bfc102e00 2024-03-30T04:56:17.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.830+0000 7f1c04200700 1 -- 172.21.15.123:0/879188175 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bfc11e700 con 0x7f1bfc102e00 2024-03-30T04:56:17.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.831+0000 7f1c04200700 1 -- 172.21.15.123:0/879188175 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bfc0fe550 con 0x7f1bfc102e00 2024-03-30T04:56:17.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.832+0000 7f1bf9600700 1 -- 172.21.15.123:0/879188175 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f1bf000dbc0 con 0x7f1bfc102e00 2024-03-30T04:56:17.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.833+0000 7f1bf9600700 1 --2- 172.21.15.123:0/879188175 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1be406cc30 0x7f1be406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:56:17.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.833+0000 7f1bf9600700 1 -- 172.21.15.123:0/879188175 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f1bf008bc70 con 0x7f1bfc102e00 2024-03-30T04:56:17.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.833+0000 7f1c00e00700 1 --2- 172.21.15.123:0/879188175 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1be406cc30 0x7f1be406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:56:17.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.834+0000 7f1c00e00700 1 --2- 172.21.15.123:0/879188175 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1be406cc30 0x7f1be406f0e0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc071c00 tx=0x7f1be8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:56:17.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:17.837+0000 7f1bf9600700 1 -- 172.21.15.123:0/879188175 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f1bf0092080 con 0x7f1bfc102e00 2024-03-30T04:56:18.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.062+0000 7f1c04200700 1 -- 172.21.15.123:0/879188175 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1bfc04bf70 con 0x7f1be406cc30 2024-03-30T04:56:18.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.064+0000 7f1bf9600700 1 -- 172.21.15.123:0/879188175 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f1be80096f0 con 0x7f1be406cc30 2024-03-30T04:56:18.062 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:56:18.062 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:56:18.062 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:56:18.062 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T04:56:18.062 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:56:18.063 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T04:56:18.063 INFO:teuthology.orchestra.run.smithi123.stdout: "mds", 2024-03-30T04:56:18.063 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T04:56:18.063 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T04:56:18.063 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T04:56:18.063 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:56:18.063 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "16/21 daemons upgraded", 2024-03-30T04:56:18.063 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T04:56:18.063 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:56:18.063 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:56:18.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.066+0000 7f1c04200700 1 -- 172.21.15.123:0/879188175 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1be406cc30 msgr2=0x7f1be406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:18.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.066+0000 7f1c04200700 1 --2- 172.21.15.123:0/879188175 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1be406cc30 0x7f1be406f0e0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc071c00 tx=0x7f1be8009e30 comp rx=0 tx=0).stop 2024-03-30T04:56:18.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.066+0000 7f1c04200700 1 -- 172.21.15.123:0/879188175 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1bfc102e00 msgr2=0x7f1bfc0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:18.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.066+0000 7f1c04200700 1 --2- 172.21.15.123:0/879188175 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1bfc102e00 0x7f1bfc0702c0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f1bf0002a80 tx=0x7f1bf00046b0 comp rx=0 tx=0).stop 2024-03-30T04:56:18.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.066+0000 7f1c04200700 1 -- 172.21.15.123:0/879188175 shutdown_connections 2024-03-30T04:56:18.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.066+0000 7f1c04200700 1 --2- 172.21.15.123:0/879188175 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1be406cc30 0x7f1be406f0e0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:18.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.066+0000 7f1c04200700 1 --2- 172.21.15.123:0/879188175 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1bfc102e00 0x7f1bfc0702c0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:18.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.066+0000 7f1c04200700 1 --2- 172.21.15.123:0/879188175 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1bfc1037a0 0x7f1bfc070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:18.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.066+0000 7f1c04200700 1 -- 172.21.15.123:0/879188175 >> 172.21.15.123:0/879188175 conn(0x7f1bfc0f89b0 msgr2=0x7f1bfc0f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:56:18.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.066+0000 7f1c04200700 1 -- 172.21.15.123:0/879188175 shutdown_connections 2024-03-30T04:56:18.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.066+0000 7f1c04200700 1 -- 172.21.15.123:0/879188175 wait complete. 2024-03-30T04:56:18.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.199+0000 7fb0a7600700 1 -- 172.21.15.123:0/3234362179 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0a0102e00 msgr2=0x7fb0a01031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:18.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.199+0000 7fb0a7600700 1 --2- 172.21.15.123:0/3234362179 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0a0102e00 0x7fb0a01031d0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fb094004940 tx=0x7fb094009e30 comp rx=0 tx=0).stop 2024-03-30T04:56:18.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.199+0000 7fb0a7600700 1 -- 172.21.15.123:0/3234362179 shutdown_connections 2024-03-30T04:56:18.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.199+0000 7fb0a7600700 1 --2- 172.21.15.123:0/3234362179 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb0a01037a0 0x7fb0a00fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:18.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.199+0000 7fb0a7600700 1 --2- 172.21.15.123:0/3234362179 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0a0102e00 0x7fb0a01031d0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:18.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.199+0000 7fb0a7600700 1 -- 172.21.15.123:0/3234362179 >> 172.21.15.123:0/3234362179 conn(0x7fb0a00f89b0 msgr2=0x7fb0a00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:56:18.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.200+0000 7fb0a7600700 1 -- 172.21.15.123:0/3234362179 shutdown_connections 2024-03-30T04:56:18.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.200+0000 7fb0a7600700 1 -- 172.21.15.123:0/3234362179 wait complete. 2024-03-30T04:56:18.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.200+0000 7fb0a7600700 1 Processor -- start 2024-03-30T04:56:18.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.201+0000 7fb0a7600700 1 -- start start 2024-03-30T04:56:18.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.201+0000 7fb0a7600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb0a0102e00 0x7fb0a0115d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:56:18.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.201+0000 7fb0a7600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0a01037a0 0x7fb0a0116250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:56:18.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.201+0000 7fb0a7600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a01168a0 con 0x7fb0a01037a0 2024-03-30T04:56:18.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.201+0000 7fb0a7600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a010fc70 con 0x7fb0a0102e00 2024-03-30T04:56:18.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.201+0000 7fb09fe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0a01037a0 0x7fb0a0116250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:56:18.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.201+0000 7fb09fe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0a01037a0 0x7fb0a0116250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49146/0 (socket says 172.21.15.123:49146) 2024-03-30T04:56:18.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.201+0000 7fb09fe00700 1 -- 172.21.15.123:0/101073881 learned_addr learned my addr 172.21.15.123:0/101073881 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:56:18.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.201+0000 7fb0a4c00700 1 --2- 172.21.15.123:0/101073881 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb0a0102e00 0x7fb0a0115d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:56:18.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.202+0000 7fb09fe00700 1 -- 172.21.15.123:0/101073881 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb0a0102e00 msgr2=0x7fb0a0115d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:18.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.202+0000 7fb09fe00700 1 --2- 172.21.15.123:0/101073881 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb0a0102e00 0x7fb0a0115d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:18.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.202+0000 7fb09fe00700 1 -- 172.21.15.123:0/101073881 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb094005040 con 0x7fb0a01037a0 2024-03-30T04:56:18.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.202+0000 7fb09fe00700 1 --2- 172.21.15.123:0/101073881 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0a01037a0 0x7fb0a0116250 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fb08c002a50 tx=0x7fb08c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:56:18.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.202+0000 7fb09cc00700 1 -- 172.21.15.123:0/101073881 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb08c007cc0 con 0x7fb0a01037a0 2024-03-30T04:56:18.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.202+0000 7fb09cc00700 1 -- 172.21.15.123:0/101073881 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb08c00f040 con 0x7fb0a01037a0 2024-03-30T04:56:18.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.202+0000 7fb09cc00700 1 -- 172.21.15.123:0/101073881 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb08c00d920 con 0x7fb0a01037a0 2024-03-30T04:56:18.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.202+0000 7fb0a7600700 1 -- 172.21.15.123:0/101073881 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0a010ff50 con 0x7fb0a01037a0 2024-03-30T04:56:18.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.202+0000 7fb0a7600700 1 -- 172.21.15.123:0/101073881 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0a0110510 con 0x7fb0a01037a0 2024-03-30T04:56:18.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.203+0000 7fb0a7600700 1 -- 172.21.15.123:0/101073881 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0a00fe550 con 0x7fb0a01037a0 2024-03-30T04:56:18.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.204+0000 7fb09cc00700 1 -- 172.21.15.123:0/101073881 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fb08c009680 con 0x7fb0a01037a0 2024-03-30T04:56:18.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.206+0000 7fb09cc00700 1 --2- 172.21.15.123:0/101073881 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb08806cc30 0x7fb08806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:56:18.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.206+0000 7fb09cc00700 1 -- 172.21.15.123:0/101073881 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb08c08bd80 con 0x7fb0a01037a0 2024-03-30T04:56:18.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.206+0000 7fb0a4c00700 1 --2- 172.21.15.123:0/101073881 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb08806cc30 0x7fb08806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:56:18.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.206+0000 7fb0a4c00700 1 --2- 172.21.15.123:0/101073881 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb08806cc30 0x7fb08806f0e0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fb094009fd0 tx=0x7fb094009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:56:18.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.210+0000 7fb09cc00700 1 -- 172.21.15.123:0/101073881 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb08c0597d0 con 0x7fb0a01037a0 2024-03-30T04:56:18.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:56:18 smithi135 ceph-mon[106011]: from='client.35384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:56:18.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:56:18 smithi135 ceph-mon[106011]: from='client.35388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:56:18.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:56:18 smithi135 ceph-mon[106011]: from='client.35392 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:56:18.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:56:18 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/285390690' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:56:18.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:56:18 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3870393790' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:56:18.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:56:18 smithi123 ceph-mon[139369]: from='client.35384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:56:18.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:56:18 smithi123 ceph-mon[139369]: from='client.35388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:56:18.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:56:18 smithi123 ceph-mon[139369]: from='client.35392 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:56:18.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:56:18 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/285390690' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:56:18.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:56:18 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3870393790' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:56:18.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.490+0000 7fb0a7600700 1 -- 172.21.15.123:0/101073881 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fb0a004bed0 con 0x7fb0a01037a0 2024-03-30T04:56:18.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.491+0000 7fb09cc00700 1 -- 172.21.15.123:0/101073881 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7fb08c059360 con 0x7fb0a01037a0 2024-03-30T04:56:18.489 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T04:56:18.489 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-30T04:56:18.489 INFO:teuthology.orchestra.run.smithi123.stdout: host smithi135 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi135/config 2024-03-30T04:56:18.489 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-30T04:56:18.489 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T04:56:18.490 INFO:teuthology.orchestra.run.smithi123.stdout:Traceback (most recent call last): 2024-03-30T04:56:18.490 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-30T04:56:18.490 INFO:teuthology.orchestra.run.smithi123.stdout: "__main__", mod_spec) 2024-03-30T04:56:18.490 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-30T04:56:18.490 INFO:teuthology.orchestra.run.smithi123.stdout: exec(code, run_globals) 2024-03-30T04:56:18.490 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10889, in 2024-03-30T04:56:18.490 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10877, in main 2024-03-30T04:56:18.490 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-30T04:56:18.490 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-30T04:56:18.490 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-30T04:56:18.490 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-30T04:56:18.490 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-03-30T04:56:18.490 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-30T04:56:18.490 INFO:teuthology.orchestra.run.smithi123.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T04:56:18.491 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T04:56:18.491 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T04:56:18.491 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T04:56:18.491 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T04:56:18.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.493+0000 7fb0a7600700 1 -- 172.21.15.123:0/101073881 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb08806cc30 msgr2=0x7fb08806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:18.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.493+0000 7fb0a7600700 1 --2- 172.21.15.123:0/101073881 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb08806cc30 0x7fb08806f0e0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fb094009fd0 tx=0x7fb094009e30 comp rx=0 tx=0).stop 2024-03-30T04:56:18.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.493+0000 7fb0a7600700 1 -- 172.21.15.123:0/101073881 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0a01037a0 msgr2=0x7fb0a0116250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:18.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.493+0000 7fb0a7600700 1 --2- 172.21.15.123:0/101073881 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0a01037a0 0x7fb0a0116250 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fb08c002a50 tx=0x7fb08c004600 comp rx=0 tx=0).stop 2024-03-30T04:56:18.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.493+0000 7fb0a7600700 1 -- 172.21.15.123:0/101073881 shutdown_connections 2024-03-30T04:56:18.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.493+0000 7fb0a7600700 1 --2- 172.21.15.123:0/101073881 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb08806cc30 0x7fb08806f0e0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:18.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.493+0000 7fb0a7600700 1 --2- 172.21.15.123:0/101073881 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb0a0102e00 0x7fb0a0115d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:18.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.493+0000 7fb0a7600700 1 --2- 172.21.15.123:0/101073881 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0a01037a0 0x7fb0a0116250 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:18.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.493+0000 7fb0a7600700 1 -- 172.21.15.123:0/101073881 >> 172.21.15.123:0/101073881 conn(0x7fb0a00f89b0 msgr2=0x7fb0a00704f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:56:18.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.493+0000 7fb0a7600700 1 -- 172.21.15.123:0/101073881 shutdown_connections 2024-03-30T04:56:18.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:18.493+0000 7fb0a7600700 1 -- 172.21.15.123:0/101073881 wait complete. 2024-03-30T04:56:19.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:56:19 smithi135 ceph-mon[106011]: pgmap v754: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:19.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:56:19 smithi135 ceph-mon[106011]: from='client.35404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:56:19.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:56:19 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/101073881' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:56:19.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:56:19 smithi123 ceph-mon[139369]: pgmap v754: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:19.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:56:19 smithi123 ceph-mon[139369]: from='client.35404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:56:19.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:56:19 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/101073881' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:56:21.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:56:21 smithi135 ceph-mon[106011]: pgmap v755: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:21.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:56:21 smithi123 ceph-mon[139369]: pgmap v755: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:23.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:56:23 smithi135 ceph-mon[106011]: pgmap v756: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:23.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:56:23 smithi123 ceph-mon[139369]: pgmap v756: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:25.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:56:25 smithi135 ceph-mon[106011]: pgmap v757: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:25.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:56:25 smithi123 ceph-mon[139369]: pgmap v757: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:27.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:56:27 smithi135 ceph-mon[106011]: pgmap v758: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:27.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:56:27 smithi123 ceph-mon[139369]: pgmap v758: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:29.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:56:29 smithi135 ceph-mon[106011]: pgmap v759: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:29.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:56:29 smithi123 ceph-mon[139369]: pgmap v759: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:30.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:56:30 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:56:30.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:56:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:56:31.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:56:31 smithi135 ceph-mon[106011]: pgmap v760: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:31.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:56:31 smithi123 ceph-mon[139369]: pgmap v760: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:33.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:56:33 smithi135 ceph-mon[106011]: pgmap v761: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:33.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:56:33 smithi123 ceph-mon[139369]: pgmap v761: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:35.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:56:35 smithi135 ceph-mon[106011]: pgmap v762: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:35.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:56:35 smithi123 ceph-mon[139369]: pgmap v762: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:37.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:56:37 smithi135 ceph-mon[106011]: pgmap v763: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:37.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:56:37 smithi123 ceph-mon[139369]: pgmap v763: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:39.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:56:39 smithi135 ceph-mon[106011]: pgmap v764: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:39.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:56:39 smithi123 ceph-mon[139369]: pgmap v764: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:41.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:56:41 smithi135 ceph-mon[106011]: pgmap v765: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:41.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:56:41 smithi123 ceph-mon[139369]: pgmap v765: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:43.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:56:43 smithi135 ceph-mon[106011]: pgmap v766: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:43.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:56:43 smithi123 ceph-mon[139369]: pgmap v766: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:45.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:56:45 smithi135 ceph-mon[106011]: pgmap v767: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:45.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:56:45 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:56:45.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:56:45 smithi123 ceph-mon[139369]: pgmap v767: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:45.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:56:45 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:56:47.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:56:47 smithi123 ceph-mon[139369]: pgmap v768: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:47.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:56:47 smithi135 ceph-mon[106011]: pgmap v768: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:48.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:48.643+0000 7f186d800700 1 -- 172.21.15.123:0/3763606884 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1868102e00 msgr2=0x7f18681031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:48.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:48.643+0000 7f186d800700 1 --2- 172.21.15.123:0/3763606884 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1868102e00 0x7f18681031d0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f1858004940 tx=0x7f1858009e30 comp rx=0 tx=0).stop 2024-03-30T04:56:48.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:48.643+0000 7f186d800700 1 -- 172.21.15.123:0/3763606884 shutdown_connections 2024-03-30T04:56:48.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:48.643+0000 7f186d800700 1 --2- 172.21.15.123:0/3763606884 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f18681037a0 0x7f18680fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:48.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:48.643+0000 7f186d800700 1 --2- 172.21.15.123:0/3763606884 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1868102e00 0x7f18681031d0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:48.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:48.643+0000 7f186d800700 1 -- 172.21.15.123:0/3763606884 >> 172.21.15.123:0/3763606884 conn(0x7f18680f89b0 msgr2=0x7f18680fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:56:48.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:48.643+0000 7f186d800700 1 -- 172.21.15.123:0/3763606884 shutdown_connections 2024-03-30T04:56:48.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:48.643+0000 7f186d800700 1 -- 172.21.15.123:0/3763606884 wait complete. 2024-03-30T04:56:48.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:48.644+0000 7f186d800700 1 Processor -- start 2024-03-30T04:56:48.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:48.644+0000 7f186d800700 1 -- start start 2024-03-30T04:56:48.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:48.645+0000 7f186d800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1868102e00 0x7f1868113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:56:48.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:48.645+0000 7f186d800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18681037a0 0x7f1868114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:56:48.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:48.645+0000 7f186d800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18681146b0 con 0x7f18681037a0 2024-03-30T04:56:48.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:48.645+0000 7f186d800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f186810daa0 con 0x7f1868102e00 2024-03-30T04:56:48.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:48.645+0000 7f1866000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18681037a0 0x7f1868114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:56:48.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:48.645+0000 7f1866000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18681037a0 0x7f1868114060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35248/0 (socket says 172.21.15.123:35248) 2024-03-30T04:56:48.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:48.645+0000 7f1866000700 1 -- 172.21.15.123:0/4007602239 learned_addr learned my addr 172.21.15.123:0/4007602239 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:56:48.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:48.645+0000 7f1866a00700 1 --2- 172.21.15.123:0/4007602239 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1868102e00 0x7f1868113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:56:48.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:48.645+0000 7f1866000700 1 -- 172.21.15.123:0/4007602239 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1868102e00 msgr2=0x7f1868113b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:48.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:48.645+0000 7f1866000700 1 --2- 172.21.15.123:0/4007602239 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1868102e00 0x7f1868113b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:48.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:48.645+0000 7f1866000700 1 -- 172.21.15.123:0/4007602239 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1858005040 con 0x7f18681037a0 2024-03-30T04:56:48.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:48.646+0000 7f1866000700 1 --2- 172.21.15.123:0/4007602239 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18681037a0 0x7f1868114060 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f1854002a50 tx=0x7f1854004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:56:48.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:48.646+0000 7f185ea00700 1 -- 172.21.15.123:0/4007602239 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1854007cc0 con 0x7f18681037a0 2024-03-30T04:56:48.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:48.646+0000 7f185ea00700 1 -- 172.21.15.123:0/4007602239 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f185400f040 con 0x7f18681037a0 2024-03-30T04:56:48.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:48.646+0000 7f186d800700 1 -- 172.21.15.123:0/4007602239 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f186810dd80 con 0x7f18681037a0 2024-03-30T04:56:48.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:48.646+0000 7f185ea00700 1 -- 172.21.15.123:0/4007602239 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f185400d920 con 0x7f18681037a0 2024-03-30T04:56:48.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:48.646+0000 7f186d800700 1 -- 172.21.15.123:0/4007602239 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f186810e340 con 0x7f18681037a0 2024-03-30T04:56:48.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:48.647+0000 7f186d800700 1 -- 172.21.15.123:0/4007602239 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18680fe550 con 0x7f18681037a0 2024-03-30T04:56:48.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:48.648+0000 7f185ea00700 1 -- 172.21.15.123:0/4007602239 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f1854009680 con 0x7f18681037a0 2024-03-30T04:56:48.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:48.649+0000 7f185ea00700 1 --2- 172.21.15.123:0/4007602239 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f184c06cc30 0x7f184c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:56:48.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:48.649+0000 7f185ea00700 1 -- 172.21.15.123:0/4007602239 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f185408bdb0 con 0x7f18681037a0 2024-03-30T04:56:48.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:48.649+0000 7f1866a00700 1 --2- 172.21.15.123:0/4007602239 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f184c06cc30 0x7f184c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:56:48.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:48.650+0000 7f1866a00700 1 --2- 172.21.15.123:0/4007602239 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f184c06cc30 0x7f184c06f0e0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f1858009fd0 tx=0x7f1858009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:56:48.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:48.653+0000 7f185ea00700 1 -- 172.21.15.123:0/4007602239 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f1854059780 con 0x7f18681037a0 2024-03-30T04:56:48.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:48.873+0000 7f186d800700 1 -- 172.21.15.123:0/4007602239 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1868109dc0 con 0x7f184c06cc30 2024-03-30T04:56:48.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:48.874+0000 7f185ea00700 1 -- 172.21.15.123:0/4007602239 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f1868109dc0 con 0x7f184c06cc30 2024-03-30T04:56:48.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:48.876+0000 7f186d800700 1 -- 172.21.15.123:0/4007602239 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f184c06cc30 msgr2=0x7f184c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:48.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:48.876+0000 7f186d800700 1 --2- 172.21.15.123:0/4007602239 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f184c06cc30 0x7f184c06f0e0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f1858009fd0 tx=0x7f1858009e30 comp rx=0 tx=0).stop 2024-03-30T04:56:48.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:48.876+0000 7f186d800700 1 -- 172.21.15.123:0/4007602239 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18681037a0 msgr2=0x7f1868114060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:48.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:48.876+0000 7f186d800700 1 --2- 172.21.15.123:0/4007602239 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18681037a0 0x7f1868114060 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f1854002a50 tx=0x7f1854004600 comp rx=0 tx=0).stop 2024-03-30T04:56:48.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:48.876+0000 7f186d800700 1 -- 172.21.15.123:0/4007602239 shutdown_connections 2024-03-30T04:56:48.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:48.876+0000 7f186d800700 1 --2- 172.21.15.123:0/4007602239 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f184c06cc30 0x7f184c06f0e0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:48.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:48.876+0000 7f186d800700 1 --2- 172.21.15.123:0/4007602239 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1868102e00 0x7f1868113b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:48.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:48.876+0000 7f186d800700 1 --2- 172.21.15.123:0/4007602239 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18681037a0 0x7f1868114060 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:48.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:48.876+0000 7f186d800700 1 -- 172.21.15.123:0/4007602239 >> 172.21.15.123:0/4007602239 conn(0x7f18680f89b0 msgr2=0x7f18680f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:56:48.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:48.877+0000 7f186d800700 1 -- 172.21.15.123:0/4007602239 shutdown_connections 2024-03-30T04:56:48.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:48.877+0000 7f186d800700 1 -- 172.21.15.123:0/4007602239 wait complete. 2024-03-30T04:56:48.890 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:56:49.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.012+0000 7fe30e000700 1 -- 172.21.15.123:0/1952364811 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3081037a0 msgr2=0x7fe3080fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:49.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.012+0000 7fe30e000700 1 --2- 172.21.15.123:0/1952364811 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3081037a0 0x7fe3080fce10 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fe2f4004940 tx=0x7fe2f4009e30 comp rx=0 tx=0).stop 2024-03-30T04:56:49.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.012+0000 7fe30e000700 1 -- 172.21.15.123:0/1952364811 shutdown_connections 2024-03-30T04:56:49.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.012+0000 7fe30e000700 1 --2- 172.21.15.123:0/1952364811 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3081037a0 0x7fe3080fce10 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:49.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.012+0000 7fe30e000700 1 --2- 172.21.15.123:0/1952364811 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe308102e00 0x7fe3081031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:49.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.012+0000 7fe30e000700 1 -- 172.21.15.123:0/1952364811 >> 172.21.15.123:0/1952364811 conn(0x7fe3080f89b0 msgr2=0x7fe3080fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:56:49.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.012+0000 7fe30e000700 1 -- 172.21.15.123:0/1952364811 shutdown_connections 2024-03-30T04:56:49.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.012+0000 7fe30e000700 1 -- 172.21.15.123:0/1952364811 wait complete. 2024-03-30T04:56:49.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.013+0000 7fe30e000700 1 Processor -- start 2024-03-30T04:56:49.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.013+0000 7fe30e000700 1 -- start start 2024-03-30T04:56:49.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.014+0000 7fe30e000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe308102e00 0x7fe308113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:56:49.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.014+0000 7fe30e000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3081037a0 0x7fe308113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:56:49.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.014+0000 7fe30e000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe308114600 con 0x7fe3081037a0 2024-03-30T04:56:49.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.014+0000 7fe30e000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe30810d9f0 con 0x7fe308102e00 2024-03-30T04:56:49.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.014+0000 7fe306a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3081037a0 0x7fe308113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:56:49.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.014+0000 7fe306a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3081037a0 0x7fe308113fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35268/0 (socket says 172.21.15.123:35268) 2024-03-30T04:56:49.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.014+0000 7fe306a00700 1 -- 172.21.15.123:0/3846324002 learned_addr learned my addr 172.21.15.123:0/3846324002 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:56:49.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.014+0000 7fe307400700 1 --2- 172.21.15.123:0/3846324002 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe308102e00 0x7fe308113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:56:49.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.014+0000 7fe306a00700 1 -- 172.21.15.123:0/3846324002 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe308102e00 msgr2=0x7fe308113a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:49.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.014+0000 7fe306a00700 1 --2- 172.21.15.123:0/3846324002 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe308102e00 0x7fe308113a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:49.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.014+0000 7fe306a00700 1 -- 172.21.15.123:0/3846324002 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2f4005040 con 0x7fe3081037a0 2024-03-30T04:56:49.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.015+0000 7fe306a00700 1 --2- 172.21.15.123:0/3846324002 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3081037a0 0x7fe308113fb0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fe2f4008d50 tx=0x7fe2f4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:56:49.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.015+0000 7fe2ff400700 1 -- 172.21.15.123:0/3846324002 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe2f400e030 con 0x7fe3081037a0 2024-03-30T04:56:49.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.015+0000 7fe2ff400700 1 -- 172.21.15.123:0/3846324002 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe2f40093b0 con 0x7fe3081037a0 2024-03-30T04:56:49.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.015+0000 7fe30e000700 1 -- 172.21.15.123:0/3846324002 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe30810dc70 con 0x7fe3081037a0 2024-03-30T04:56:49.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.015+0000 7fe2ff400700 1 -- 172.21.15.123:0/3846324002 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe2f401cbb0 con 0x7fe3081037a0 2024-03-30T04:56:49.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.015+0000 7fe30e000700 1 -- 172.21.15.123:0/3846324002 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe30810e230 con 0x7fe3081037a0 2024-03-30T04:56:49.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.016+0000 7fe30e000700 1 -- 172.21.15.123:0/3846324002 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3080fe550 con 0x7fe3081037a0 2024-03-30T04:56:49.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.017+0000 7fe2ff400700 1 -- 172.21.15.123:0/3846324002 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fe2f4013930 con 0x7fe3081037a0 2024-03-30T04:56:49.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.018+0000 7fe2ff400700 1 --2- 172.21.15.123:0/3846324002 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe2ec06cc30 0x7fe2ec06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:56:49.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.018+0000 7fe2ff400700 1 -- 172.21.15.123:0/3846324002 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe2f408d8e0 con 0x7fe3081037a0 2024-03-30T04:56:49.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.018+0000 7fe307400700 1 --2- 172.21.15.123:0/3846324002 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe2ec06cc30 0x7fe2ec06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:56:49.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.019+0000 7fe307400700 1 --2- 172.21.15.123:0/3846324002 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe2ec06cc30 0x7fe2ec06f0e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fe2f8009b80 tx=0x7fe2f800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:56:49.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.022+0000 7fe2ff400700 1 -- 172.21.15.123:0/3846324002 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe2f405b2b0 con 0x7fe3081037a0 2024-03-30T04:56:49.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.243+0000 7fe30e000700 1 -- 172.21.15.123:0/3846324002 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe308109da0 con 0x7fe2ec06cc30 2024-03-30T04:56:49.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.244+0000 7fe2ff400700 1 -- 172.21.15.123:0/3846324002 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fe308109da0 con 0x7fe2ec06cc30 2024-03-30T04:56:49.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.247+0000 7fe30e000700 1 -- 172.21.15.123:0/3846324002 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe2ec06cc30 msgr2=0x7fe2ec06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:49.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.247+0000 7fe30e000700 1 --2- 172.21.15.123:0/3846324002 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe2ec06cc30 0x7fe2ec06f0e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fe2f8009b80 tx=0x7fe2f800b040 comp rx=0 tx=0).stop 2024-03-30T04:56:49.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.247+0000 7fe30e000700 1 -- 172.21.15.123:0/3846324002 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3081037a0 msgr2=0x7fe308113fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:49.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.247+0000 7fe30e000700 1 --2- 172.21.15.123:0/3846324002 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3081037a0 0x7fe308113fb0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fe2f4008d50 tx=0x7fe2f4009e30 comp rx=0 tx=0).stop 2024-03-30T04:56:49.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.247+0000 7fe30e000700 1 -- 172.21.15.123:0/3846324002 shutdown_connections 2024-03-30T04:56:49.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.247+0000 7fe30e000700 1 --2- 172.21.15.123:0/3846324002 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe2ec06cc30 0x7fe2ec06f0e0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:49.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.247+0000 7fe30e000700 1 --2- 172.21.15.123:0/3846324002 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe308102e00 0x7fe308113a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:49.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.247+0000 7fe30e000700 1 --2- 172.21.15.123:0/3846324002 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3081037a0 0x7fe308113fb0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:49.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.247+0000 7fe30e000700 1 -- 172.21.15.123:0/3846324002 >> 172.21.15.123:0/3846324002 conn(0x7fe3080f89b0 msgr2=0x7fe3080f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:56:49.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.247+0000 7fe30e000700 1 -- 172.21.15.123:0/3846324002 shutdown_connections 2024-03-30T04:56:49.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.247+0000 7fe30e000700 1 -- 172.21.15.123:0/3846324002 wait complete. 2024-03-30T04:56:49.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.381+0000 7f9052a00700 1 -- 172.21.15.123:0/3398267525 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f904c1035e0 msgr2=0x7f904c0fcc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:49.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.381+0000 7f9052a00700 1 --2- 172.21.15.123:0/3398267525 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f904c1035e0 0x7f904c0fcc50 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f9038004940 tx=0x7f9038009e30 comp rx=0 tx=0).stop 2024-03-30T04:56:49.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.381+0000 7f9052a00700 1 -- 172.21.15.123:0/3398267525 shutdown_connections 2024-03-30T04:56:49.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.381+0000 7f9052a00700 1 --2- 172.21.15.123:0/3398267525 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f904c1035e0 0x7f904c0fcc50 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:49.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.381+0000 7f9052a00700 1 --2- 172.21.15.123:0/3398267525 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f904c102c40 0x7f904c103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:49.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.381+0000 7f9052a00700 1 -- 172.21.15.123:0/3398267525 >> 172.21.15.123:0/3398267525 conn(0x7f904c0f8830 msgr2=0x7f904c0fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:56:49.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.381+0000 7f9052a00700 1 -- 172.21.15.123:0/3398267525 shutdown_connections 2024-03-30T04:56:49.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.381+0000 7f9052a00700 1 -- 172.21.15.123:0/3398267525 wait complete. 2024-03-30T04:56:49.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.382+0000 7f9052a00700 1 Processor -- start 2024-03-30T04:56:49.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.382+0000 7f9052a00700 1 -- start start 2024-03-30T04:56:49.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.382+0000 7f9052a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f904c102c40 0x7f904c115a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:56:49.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.382+0000 7f9052a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f904c1035e0 0x7f904c115fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:56:49.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.382+0000 7f9052a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f904c116610 con 0x7f904c102c40 2024-03-30T04:56:49.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.382+0000 7f9052a00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f904c10fa00 con 0x7f904c1035e0 2024-03-30T04:56:49.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.382+0000 7f9051600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f904c102c40 0x7f904c115a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:56:49.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.382+0000 7f9051600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f904c102c40 0x7f904c115a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35286/0 (socket says 172.21.15.123:35286) 2024-03-30T04:56:49.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.382+0000 7f9051600700 1 -- 172.21.15.123:0/1714686881 learned_addr learned my addr 172.21.15.123:0/1714686881 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:56:49.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.383+0000 7f9050c00700 1 --2- 172.21.15.123:0/1714686881 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f904c1035e0 0x7f904c115fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:56:49.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.383+0000 7f9051600700 1 -- 172.21.15.123:0/1714686881 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f904c1035e0 msgr2=0x7f904c115fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:49.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.383+0000 7f9051600700 1 --2- 172.21.15.123:0/1714686881 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f904c1035e0 0x7f904c115fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:49.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.383+0000 7f9051600700 1 -- 172.21.15.123:0/1714686881 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9038005040 con 0x7f904c102c40 2024-03-30T04:56:49.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.383+0000 7f9050c00700 1 --2- 172.21.15.123:0/1714686881 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f904c1035e0 0x7f904c115fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:56:49.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.383+0000 7f9051600700 1 --2- 172.21.15.123:0/1714686881 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f904c102c40 0x7f904c115a80 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f9040002a80 tx=0x7f90400046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:56:49.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.383+0000 7f9049600700 1 -- 172.21.15.123:0/1714686881 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9040007cc0 con 0x7f904c102c40 2024-03-30T04:56:49.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.383+0000 7f9049600700 1 -- 172.21.15.123:0/1714686881 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f904000f040 con 0x7f904c102c40 2024-03-30T04:56:49.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.383+0000 7f9052a00700 1 -- 172.21.15.123:0/1714686881 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f904c10fce0 con 0x7f904c102c40 2024-03-30T04:56:49.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.383+0000 7f9049600700 1 -- 172.21.15.123:0/1714686881 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9040009640 con 0x7f904c102c40 2024-03-30T04:56:49.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.383+0000 7f9052a00700 1 -- 172.21.15.123:0/1714686881 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f904c1102a0 con 0x7f904c102c40 2024-03-30T04:56:49.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.384+0000 7f9052a00700 1 -- 172.21.15.123:0/1714686881 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f904c0fe390 con 0x7f904c102c40 2024-03-30T04:56:49.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.386+0000 7f9049600700 1 -- 172.21.15.123:0/1714686881 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f90400097a0 con 0x7f904c102c40 2024-03-30T04:56:49.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.387+0000 7f9049600700 1 --2- 172.21.15.123:0/1714686881 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f903406cc30 0x7f903406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:56:49.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.387+0000 7f9049600700 1 -- 172.21.15.123:0/1714686881 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f904008b8f0 con 0x7f904c102c40 2024-03-30T04:56:49.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.387+0000 7f9050c00700 1 --2- 172.21.15.123:0/1714686881 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f903406cc30 0x7f903406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:56:49.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.388+0000 7f9050c00700 1 --2- 172.21.15.123:0/1714686881 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f903406cc30 0x7f903406f0e0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f9038009fd0 tx=0x7f9038009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:56:49.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.391+0000 7f9049600700 1 -- 172.21.15.123:0/1714686881 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f90400592c0 con 0x7f904c102c40 2024-03-30T04:56:49.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:56:49 smithi123 ceph-mon[139369]: pgmap v769: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:49.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:56:49 smithi135 ceph-mon[106011]: pgmap v769: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:49.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.605+0000 7f9052a00700 1 -- 172.21.15.123:0/1714686881 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f904c04bed0 con 0x7f903406cc30 2024-03-30T04:56:49.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.616+0000 7f9049600700 1 -- 172.21.15.123:0/1714686881 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f904c04bed0 con 0x7f903406cc30 2024-03-30T04:56:49.614 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:56:49.615 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (25m) 3m ago 38m 14.6M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:56:49.615 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (23m) 3m ago 38m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T04:56:49.615 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (23m) 3m ago 37m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T04:56:49.615 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (25m) 3m ago 38m 73.5M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T04:56:49.615 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (19m) 3m ago 32m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T04:56:49.616 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (19m) 3m ago 32m 16.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T04:56:49.616 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (3m) 3m ago 32m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 87b073a6c9ca 2024-03-30T04:56:49.616 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (18m) 3m ago 32m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T04:56:49.616 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (28m) 3m ago 40m 510M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:56:49.617 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (27m) 3m ago 37m 456M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:56:49.617 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (24m) 3m ago 40m 70.0M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T04:56:49.617 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (24m) 3m ago 37m 51.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T04:56:49.617 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (26m) 3m ago 38m 8636k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:56:49.618 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (26m) 3m ago 37m 8678k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:56:49.618 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (22m) 3m ago 36m 107M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T04:56:49.618 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (22m) 3m ago 35m 97.2M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T04:56:49.621 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (21m) 3m ago 35m 89.8M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T04:56:49.621 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (20m) 3m ago 34m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T04:56:49.621 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (20m) 3m ago 34m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T04:56:49.621 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (19m) 3m ago 33m 88.8M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T04:56:49.621 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (26m) 3m ago 38m 63.9M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:56:49.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.618+0000 7f9052a00700 1 -- 172.21.15.123:0/1714686881 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f903406cc30 msgr2=0x7f903406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:49.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.618+0000 7f9052a00700 1 --2- 172.21.15.123:0/1714686881 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f903406cc30 0x7f903406f0e0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f9038009fd0 tx=0x7f9038009e30 comp rx=0 tx=0).stop 2024-03-30T04:56:49.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.618+0000 7f9052a00700 1 -- 172.21.15.123:0/1714686881 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f904c102c40 msgr2=0x7f904c115a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:49.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.618+0000 7f9052a00700 1 --2- 172.21.15.123:0/1714686881 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f904c102c40 0x7f904c115a80 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f9040002a80 tx=0x7f90400046b0 comp rx=0 tx=0).stop 2024-03-30T04:56:49.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.618+0000 7f9052a00700 1 -- 172.21.15.123:0/1714686881 shutdown_connections 2024-03-30T04:56:49.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.618+0000 7f9052a00700 1 --2- 172.21.15.123:0/1714686881 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f903406cc30 0x7f903406f0e0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:49.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.618+0000 7f9052a00700 1 --2- 172.21.15.123:0/1714686881 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f904c102c40 0x7f904c115a80 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:49.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.618+0000 7f9052a00700 1 --2- 172.21.15.123:0/1714686881 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f904c1035e0 0x7f904c115fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:49.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.618+0000 7f9052a00700 1 -- 172.21.15.123:0/1714686881 >> 172.21.15.123:0/1714686881 conn(0x7f904c0f8830 msgr2=0x7f904c0f9810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:56:49.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.618+0000 7f9052a00700 1 -- 172.21.15.123:0/1714686881 shutdown_connections 2024-03-30T04:56:49.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.618+0000 7f9052a00700 1 -- 172.21.15.123:0/1714686881 wait complete. 2024-03-30T04:56:49.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.752+0000 7f4923a00700 1 -- 172.21.15.123:0/2103038655 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f491c102e00 msgr2=0x7f491c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:49.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.752+0000 7f4923a00700 1 --2- 172.21.15.123:0/2103038655 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f491c102e00 0x7f491c1031d0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f4910004940 tx=0x7f4910009e30 comp rx=0 tx=0).stop 2024-03-30T04:56:49.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.752+0000 7f4923a00700 1 -- 172.21.15.123:0/2103038655 shutdown_connections 2024-03-30T04:56:49.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.752+0000 7f4923a00700 1 --2- 172.21.15.123:0/2103038655 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f491c1037a0 0x7f491c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:49.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.752+0000 7f4923a00700 1 --2- 172.21.15.123:0/2103038655 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f491c102e00 0x7f491c1031d0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:49.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.752+0000 7f4923a00700 1 -- 172.21.15.123:0/2103038655 >> 172.21.15.123:0/2103038655 conn(0x7f491c0f89b0 msgr2=0x7f491c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:56:49.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.752+0000 7f4923a00700 1 -- 172.21.15.123:0/2103038655 shutdown_connections 2024-03-30T04:56:49.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.752+0000 7f4923a00700 1 -- 172.21.15.123:0/2103038655 wait complete. 2024-03-30T04:56:49.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.753+0000 7f4923a00700 1 Processor -- start 2024-03-30T04:56:49.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.753+0000 7f4923a00700 1 -- start start 2024-03-30T04:56:49.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.753+0000 7f4923a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f491c102e00 0x7f491c115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:56:49.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.753+0000 7f4923a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f491c1037a0 0x7f491c116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:56:49.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.753+0000 7f4923a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f491c116860 con 0x7f491c102e00 2024-03-30T04:56:49.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.753+0000 7f4923a00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f491c10fc30 con 0x7f491c1037a0 2024-03-30T04:56:49.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.754+0000 7f4921000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f491c102e00 0x7f491c115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:56:49.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.754+0000 7f4921000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f491c102e00 0x7f491c115cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35292/0 (socket says 172.21.15.123:35292) 2024-03-30T04:56:49.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.754+0000 7f4921000700 1 -- 172.21.15.123:0/426813696 learned_addr learned my addr 172.21.15.123:0/426813696 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:56:49.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.754+0000 7f491be00700 1 --2- 172.21.15.123:0/426813696 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f491c1037a0 0x7f491c116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:56:49.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.754+0000 7f4921000700 1 -- 172.21.15.123:0/426813696 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f491c1037a0 msgr2=0x7f491c116210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:49.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.754+0000 7f4921000700 1 --2- 172.21.15.123:0/426813696 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f491c1037a0 0x7f491c116210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:49.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.754+0000 7f4921000700 1 -- 172.21.15.123:0/426813696 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4910005040 con 0x7f491c102e00 2024-03-30T04:56:49.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.754+0000 7f4921000700 1 --2- 172.21.15.123:0/426813696 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f491c102e00 0x7f491c115cd0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f4910009390 tx=0x7f4910009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:56:49.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.754+0000 7f4918c00700 1 -- 172.21.15.123:0/426813696 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f491000e040 con 0x7f491c102e00 2024-03-30T04:56:49.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.755+0000 7f4918c00700 1 -- 172.21.15.123:0/426813696 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4910013420 con 0x7f491c102e00 2024-03-30T04:56:49.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.755+0000 7f4923a00700 1 -- 172.21.15.123:0/426813696 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f491c10feb0 con 0x7f491c102e00 2024-03-30T04:56:49.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.755+0000 7f4918c00700 1 -- 172.21.15.123:0/426813696 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f491001c630 con 0x7f491c102e00 2024-03-30T04:56:49.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.755+0000 7f4923a00700 1 -- 172.21.15.123:0/426813696 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f491c110470 con 0x7f491c102e00 2024-03-30T04:56:49.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.756+0000 7f4923a00700 1 -- 172.21.15.123:0/426813696 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f491c0fe550 con 0x7f491c102e00 2024-03-30T04:56:49.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.757+0000 7f4918c00700 1 -- 172.21.15.123:0/426813696 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f49100139e0 con 0x7f491c102e00 2024-03-30T04:56:49.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.758+0000 7f4918c00700 1 --2- 172.21.15.123:0/426813696 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f490406cc30 0x7f490406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:56:49.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.758+0000 7f4918c00700 1 -- 172.21.15.123:0/426813696 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f491008d250 con 0x7f491c102e00 2024-03-30T04:56:49.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.758+0000 7f491be00700 1 --2- 172.21.15.123:0/426813696 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f490406cc30 0x7f490406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:56:49.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.758+0000 7f491be00700 1 --2- 172.21.15.123:0/426813696 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f490406cc30 0x7f490406f0e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f4908007fd0 tx=0x7f4908009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:56:49.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:49.762+0000 7f4918c00700 1 -- 172.21.15.123:0/426813696 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f491005add0 con 0x7f491c102e00 2024-03-30T04:56:50.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.047+0000 7f4923a00700 1 -- 172.21.15.123:0/426813696 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f491c04bed0 con 0x7f491c102e00 2024-03-30T04:56:50.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.047+0000 7f4918c00700 1 -- 172.21.15.123:0/426813696 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f491005a960 con 0x7f491c102e00 2024-03-30T04:56:50.046 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:56:50.046 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:56:50.046 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:56:50.046 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:56:50.046 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:56:50.046 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:56:50.046 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:56:50.046 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:56:50.046 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T04:56:50.047 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:56:50.047 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:56:50.047 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 4 2024-03-30T04:56:50.047 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:56:50.047 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:56:50.047 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 14 2024-03-30T04:56:50.047 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:56:50.047 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:56:50.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.050+0000 7f4923a00700 1 -- 172.21.15.123:0/426813696 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f490406cc30 msgr2=0x7f490406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:50.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.050+0000 7f4923a00700 1 --2- 172.21.15.123:0/426813696 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f490406cc30 0x7f490406f0e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f4908007fd0 tx=0x7f4908009040 comp rx=0 tx=0).stop 2024-03-30T04:56:50.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.050+0000 7f4923a00700 1 -- 172.21.15.123:0/426813696 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f491c102e00 msgr2=0x7f491c115cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:50.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.050+0000 7f4923a00700 1 --2- 172.21.15.123:0/426813696 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f491c102e00 0x7f491c115cd0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f4910009390 tx=0x7f4910009e30 comp rx=0 tx=0).stop 2024-03-30T04:56:50.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.050+0000 7f4923a00700 1 -- 172.21.15.123:0/426813696 shutdown_connections 2024-03-30T04:56:50.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.050+0000 7f4923a00700 1 --2- 172.21.15.123:0/426813696 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f490406cc30 0x7f490406f0e0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:50.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.050+0000 7f4923a00700 1 --2- 172.21.15.123:0/426813696 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f491c102e00 0x7f491c115cd0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:50.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.050+0000 7f4923a00700 1 --2- 172.21.15.123:0/426813696 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f491c1037a0 0x7f491c116210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:50.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.050+0000 7f4923a00700 1 -- 172.21.15.123:0/426813696 >> 172.21.15.123:0/426813696 conn(0x7f491c0f89b0 msgr2=0x7f491c0f99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:56:50.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.050+0000 7f4923a00700 1 -- 172.21.15.123:0/426813696 shutdown_connections 2024-03-30T04:56:50.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.050+0000 7f4923a00700 1 -- 172.21.15.123:0/426813696 wait complete. 2024-03-30T04:56:50.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.186+0000 7efc65800700 1 -- 172.21.15.123:0/4093773401 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc601037a0 msgr2=0x7efc600fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:50.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.186+0000 7efc65800700 1 --2- 172.21.15.123:0/4093773401 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc601037a0 0x7efc600fce10 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7efc4c004940 tx=0x7efc4c009e30 comp rx=0 tx=0).stop 2024-03-30T04:56:50.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.186+0000 7efc65800700 1 -- 172.21.15.123:0/4093773401 shutdown_connections 2024-03-30T04:56:50.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.186+0000 7efc65800700 1 --2- 172.21.15.123:0/4093773401 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc601037a0 0x7efc600fce10 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:50.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.186+0000 7efc65800700 1 --2- 172.21.15.123:0/4093773401 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc60102e00 0x7efc601031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:50.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.186+0000 7efc65800700 1 -- 172.21.15.123:0/4093773401 >> 172.21.15.123:0/4093773401 conn(0x7efc600f89b0 msgr2=0x7efc600fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:56:50.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.186+0000 7efc65800700 1 -- 172.21.15.123:0/4093773401 shutdown_connections 2024-03-30T04:56:50.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.186+0000 7efc65800700 1 -- 172.21.15.123:0/4093773401 wait complete. 2024-03-30T04:56:50.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.187+0000 7efc65800700 1 Processor -- start 2024-03-30T04:56:50.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.187+0000 7efc65800700 1 -- start start 2024-03-30T04:56:50.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.187+0000 7efc65800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc60102e00 0x7efc60113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:56:50.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.187+0000 7efc65800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc601037a0 0x7efc60113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:56:50.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.187+0000 7efc65800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc60114600 con 0x7efc60102e00 2024-03-30T04:56:50.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.187+0000 7efc65800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc6010d9f0 con 0x7efc601037a0 2024-03-30T04:56:50.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.187+0000 7efc5ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc60102e00 0x7efc60113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:56:50.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.188+0000 7efc5ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc60102e00 0x7efc60113a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35304/0 (socket says 172.21.15.123:35304) 2024-03-30T04:56:50.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.188+0000 7efc5ea00700 1 -- 172.21.15.123:0/787750064 learned_addr learned my addr 172.21.15.123:0/787750064 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:56:50.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.188+0000 7efc5e000700 1 --2- 172.21.15.123:0/787750064 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc601037a0 0x7efc60113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:56:50.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.188+0000 7efc5ea00700 1 -- 172.21.15.123:0/787750064 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc601037a0 msgr2=0x7efc60113fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:50.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.188+0000 7efc5ea00700 1 --2- 172.21.15.123:0/787750064 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc601037a0 0x7efc60113fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:50.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.188+0000 7efc5ea00700 1 -- 172.21.15.123:0/787750064 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc4c005040 con 0x7efc60102e00 2024-03-30T04:56:50.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.188+0000 7efc5ea00700 1 --2- 172.21.15.123:0/787750064 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc60102e00 0x7efc60113a70 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7efc5000eb40 tx=0x7efc5000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:56:50.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.188+0000 7efc56a00700 1 -- 172.21.15.123:0/787750064 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc50007cc0 con 0x7efc60102e00 2024-03-30T04:56:50.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.188+0000 7efc56a00700 1 -- 172.21.15.123:0/787750064 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efc5000f040 con 0x7efc60102e00 2024-03-30T04:56:50.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.188+0000 7efc65800700 1 -- 172.21.15.123:0/787750064 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc6010dc70 con 0x7efc60102e00 2024-03-30T04:56:50.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.188+0000 7efc56a00700 1 -- 172.21.15.123:0/787750064 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc50009570 con 0x7efc60102e00 2024-03-30T04:56:50.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.189+0000 7efc65800700 1 -- 172.21.15.123:0/787750064 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc6010e230 con 0x7efc60102e00 2024-03-30T04:56:50.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.189+0000 7efc65800700 1 -- 172.21.15.123:0/787750064 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc600fe550 con 0x7efc60102e00 2024-03-30T04:56:50.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.190+0000 7efc56a00700 1 -- 172.21.15.123:0/787750064 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7efc500096d0 con 0x7efc60102e00 2024-03-30T04:56:50.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.191+0000 7efc56a00700 1 --2- 172.21.15.123:0/787750064 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7efc4406cc30 0x7efc4406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:56:50.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.191+0000 7efc56a00700 1 -- 172.21.15.123:0/787750064 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7efc5008bcb0 con 0x7efc60102e00 2024-03-30T04:56:50.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.192+0000 7efc5e000700 1 --2- 172.21.15.123:0/787750064 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7efc4406cc30 0x7efc4406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:56:50.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.192+0000 7efc5e000700 1 --2- 172.21.15.123:0/787750064 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7efc4406cc30 0x7efc4406f0e0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7efc4c00b0d0 tx=0x7efc4c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:56:50.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.196+0000 7efc56a00700 1 -- 172.21.15.123:0/787750064 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7efc500597b0 con 0x7efc60102e00 2024-03-30T04:56:50.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.443+0000 7efc65800700 1 -- 172.21.15.123:0/787750064 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7efc60109da0 con 0x7efc60102e00 2024-03-30T04:56:50.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.444+0000 7efc56a00700 1 -- 172.21.15.123:0/787750064 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 34 v34) v1 ==== 76+0+1843 (secure 0 0 0) 0x7efc50019750 con 0x7efc60102e00 2024-03-30T04:56:50.442 INFO:teuthology.orchestra.run.smithi123.stdout:e34 2024-03-30T04:56:50.442 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:56:50.442 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:56:50.442 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:56:50.442 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:56:50.442 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:56:50.443 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:56:50.443 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T04:56:50.443 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T04:56:50.443 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:56:50.443 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T04:56:50.443 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:56:50.443 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:56:50.443 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:56:50.443 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:56:50.443 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:56:50.443 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T04:56:50.443 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:56:50.443 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:56:50.443 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T04:56:50.444 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:56:50.444 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T04:56:50.444 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T04:56:50.444 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T04:56:50.444 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T04:56:50.444 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:56:50.444 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:56:50.444 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:56:50.444 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:56:50.444 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:56:50.444 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:56:50.445 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T04:56:50.445 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T04:56:50.445 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:56:50.445 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:56:50.445 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T04:56:50.445 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:56:50.445 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:56:50.445 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:56:50.445 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:56:50.445 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44975} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/3788988014,v1:172.21.15.135:6827/3788988014] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:56:50.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.446+0000 7efc65800700 1 -- 172.21.15.123:0/787750064 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7efc4406cc30 msgr2=0x7efc4406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:50.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.446+0000 7efc65800700 1 --2- 172.21.15.123:0/787750064 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7efc4406cc30 0x7efc4406f0e0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7efc4c00b0d0 tx=0x7efc4c009e30 comp rx=0 tx=0).stop 2024-03-30T04:56:50.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.446+0000 7efc65800700 1 -- 172.21.15.123:0/787750064 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc60102e00 msgr2=0x7efc60113a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:50.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.446+0000 7efc65800700 1 --2- 172.21.15.123:0/787750064 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc60102e00 0x7efc60113a70 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7efc5000eb40 tx=0x7efc5000b040 comp rx=0 tx=0).stop 2024-03-30T04:56:50.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.446+0000 7efc65800700 1 -- 172.21.15.123:0/787750064 shutdown_connections 2024-03-30T04:56:50.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.446+0000 7efc65800700 1 --2- 172.21.15.123:0/787750064 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7efc4406cc30 0x7efc4406f0e0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:50.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.446+0000 7efc65800700 1 --2- 172.21.15.123:0/787750064 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc60102e00 0x7efc60113a70 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:50.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.446+0000 7efc65800700 1 --2- 172.21.15.123:0/787750064 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc601037a0 0x7efc60113fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:50.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.446+0000 7efc65800700 1 -- 172.21.15.123:0/787750064 >> 172.21.15.123:0/787750064 conn(0x7efc600f89b0 msgr2=0x7efc600f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:56:50.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.446+0000 7efc65800700 1 -- 172.21.15.123:0/787750064 shutdown_connections 2024-03-30T04:56:50.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.446+0000 7efc65800700 1 -- 172.21.15.123:0/787750064 wait complete. 2024-03-30T04:56:50.447 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 34 2024-03-30T04:56:50.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:56:50 smithi123 ceph-mon[139369]: from='client.35412 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:56:50.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:56:50 smithi123 ceph-mon[139369]: from='client.35416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:56:50.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:56:50 smithi123 ceph-mon[139369]: pgmap v770: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:50.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:56:50 smithi123 ceph-mon[139369]: from='client.35420 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:56:50.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:56:50 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/426813696' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:56:50.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.580+0000 7f8aeea00700 1 -- 172.21.15.123:0/2755780095 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ae8070360 msgr2=0x7f8ae8070730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:50.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.580+0000 7f8aeea00700 1 --2- 172.21.15.123:0/2755780095 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ae8070360 0x7f8ae8070730 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f8adc004970 tx=0x7f8adc009e30 comp rx=0 tx=0).stop 2024-03-30T04:56:50.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.580+0000 7f8aeea00700 1 -- 172.21.15.123:0/2755780095 shutdown_connections 2024-03-30T04:56:50.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.580+0000 7f8aeea00700 1 --2- 172.21.15.123:0/2755780095 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8ae8070c70 0x7f8ae8108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:50.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.580+0000 7f8aeea00700 1 --2- 172.21.15.123:0/2755780095 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ae8070360 0x7f8ae8070730 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:50.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.580+0000 7f8aeea00700 1 -- 172.21.15.123:0/2755780095 >> 172.21.15.123:0/2755780095 conn(0x7f8ae80f8830 msgr2=0x7f8ae80fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:56:50.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.580+0000 7f8aeea00700 1 -- 172.21.15.123:0/2755780095 shutdown_connections 2024-03-30T04:56:50.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.580+0000 7f8aeea00700 1 -- 172.21.15.123:0/2755780095 wait complete. 2024-03-30T04:56:50.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.581+0000 7f8aeea00700 1 Processor -- start 2024-03-30T04:56:50.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.581+0000 7f8aeea00700 1 -- start start 2024-03-30T04:56:50.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.581+0000 7f8aeea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ae8070360 0x7f8ae8115ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:56:50.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.581+0000 7f8aeea00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8ae8070c70 0x7f8ae8115ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:56:50.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.581+0000 7f8aeea00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae8116640 con 0x7f8ae8070360 2024-03-30T04:56:50.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.581+0000 7f8aeea00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae810fa70 con 0x7f8ae8070c70 2024-03-30T04:56:50.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.581+0000 7f8aed600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ae8070360 0x7f8ae8115ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:56:50.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.581+0000 7f8aed600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ae8070360 0x7f8ae8115ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35328/0 (socket says 172.21.15.123:35328) 2024-03-30T04:56:50.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.581+0000 7f8aed600700 1 -- 172.21.15.123:0/53861165 learned_addr learned my addr 172.21.15.123:0/53861165 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:56:50.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.582+0000 7f8aecc00700 1 --2- 172.21.15.123:0/53861165 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8ae8070c70 0x7f8ae8115ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:56:50.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.582+0000 7f8aed600700 1 -- 172.21.15.123:0/53861165 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8ae8070c70 msgr2=0x7f8ae8115ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:50.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.582+0000 7f8aed600700 1 --2- 172.21.15.123:0/53861165 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8ae8070c70 0x7f8ae8115ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:50.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.582+0000 7f8aed600700 1 -- 172.21.15.123:0/53861165 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8adc005040 con 0x7f8ae8070360 2024-03-30T04:56:50.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.582+0000 7f8aed600700 1 --2- 172.21.15.123:0/53861165 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ae8070360 0x7f8ae8115ab0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f8adc005010 tx=0x7f8adc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:56:50.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.582+0000 7f8ae5600700 1 -- 172.21.15.123:0/53861165 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8adc00e040 con 0x7f8ae8070360 2024-03-30T04:56:50.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.582+0000 7f8ae5600700 1 -- 172.21.15.123:0/53861165 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8adc009930 con 0x7f8ae8070360 2024-03-30T04:56:50.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.582+0000 7f8aeea00700 1 -- 172.21.15.123:0/53861165 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ae810fcf0 con 0x7f8ae8070360 2024-03-30T04:56:50.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.582+0000 7f8ae5600700 1 -- 172.21.15.123:0/53861165 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8adc01c3f0 con 0x7f8ae8070360 2024-03-30T04:56:50.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.582+0000 7f8aeea00700 1 -- 172.21.15.123:0/53861165 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ae8110230 con 0x7f8ae8070360 2024-03-30T04:56:50.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.583+0000 7f8aeea00700 1 -- 172.21.15.123:0/53861165 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ae8071aa0 con 0x7f8ae8070360 2024-03-30T04:56:50.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.584+0000 7f8ae5600700 1 -- 172.21.15.123:0/53861165 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f8adc009480 con 0x7f8ae8070360 2024-03-30T04:56:50.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.586+0000 7f8ae5600700 1 --2- 172.21.15.123:0/53861165 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8ad006cc20 0x7f8ad006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:56:50.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.586+0000 7f8ae5600700 1 -- 172.21.15.123:0/53861165 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f8adc08d360 con 0x7f8ae8070360 2024-03-30T04:56:50.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.586+0000 7f8aecc00700 1 --2- 172.21.15.123:0/53861165 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8ad006cc20 0x7f8ad006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:56:50.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.586+0000 7f8aecc00700 1 --2- 172.21.15.123:0/53861165 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8ad006cc20 0x7f8ad006f0d0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f8ad4004600 tx=0x7f8ad400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:56:50.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.590+0000 7f8ae5600700 1 -- 172.21.15.123:0/53861165 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8adc05adb0 con 0x7f8ae8070360 2024-03-30T04:56:50.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:56:50 smithi135 ceph-mon[106011]: from='client.35412 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:56:50.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:56:50 smithi135 ceph-mon[106011]: from='client.35416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:56:50.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:56:50 smithi135 ceph-mon[106011]: pgmap v770: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:50.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:56:50 smithi135 ceph-mon[106011]: from='client.35420 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:56:50.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:56:50 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/426813696' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:56:50.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.810+0000 7f8aeea00700 1 -- 172.21.15.123:0/53861165 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8ae804bed0 con 0x7f8ad006cc20 2024-03-30T04:56:50.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.811+0000 7f8ae5600700 1 -- 172.21.15.123:0/53861165 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f8ae804bed0 con 0x7f8ad006cc20 2024-03-30T04:56:50.810 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:56:50.810 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:56:50.810 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:56:50.810 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T04:56:50.810 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:56:50.810 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T04:56:50.810 INFO:teuthology.orchestra.run.smithi123.stdout: "mds", 2024-03-30T04:56:50.810 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T04:56:50.810 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T04:56:50.811 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T04:56:50.811 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:56:50.811 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "16/21 daemons upgraded", 2024-03-30T04:56:50.811 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T04:56:50.811 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:56:50.811 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:56:50.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.814+0000 7f8aeea00700 1 -- 172.21.15.123:0/53861165 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8ad006cc20 msgr2=0x7f8ad006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:50.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.814+0000 7f8aeea00700 1 --2- 172.21.15.123:0/53861165 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8ad006cc20 0x7f8ad006f0d0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f8ad4004600 tx=0x7f8ad400b040 comp rx=0 tx=0).stop 2024-03-30T04:56:50.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.814+0000 7f8aeea00700 1 -- 172.21.15.123:0/53861165 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ae8070360 msgr2=0x7f8ae8115ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:50.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.814+0000 7f8aeea00700 1 --2- 172.21.15.123:0/53861165 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ae8070360 0x7f8ae8115ab0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f8adc005010 tx=0x7f8adc009e30 comp rx=0 tx=0).stop 2024-03-30T04:56:50.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.814+0000 7f8aeea00700 1 -- 172.21.15.123:0/53861165 shutdown_connections 2024-03-30T04:56:50.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.814+0000 7f8aeea00700 1 --2- 172.21.15.123:0/53861165 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8ad006cc20 0x7f8ad006f0d0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:50.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.814+0000 7f8aeea00700 1 --2- 172.21.15.123:0/53861165 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ae8070360 0x7f8ae8115ab0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:50.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.814+0000 7f8aeea00700 1 --2- 172.21.15.123:0/53861165 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8ae8070c70 0x7f8ae8115ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:50.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.814+0000 7f8aeea00700 1 -- 172.21.15.123:0/53861165 >> 172.21.15.123:0/53861165 conn(0x7f8ae80f8830 msgr2=0x7f8ae80f9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:56:50.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.814+0000 7f8aeea00700 1 -- 172.21.15.123:0/53861165 shutdown_connections 2024-03-30T04:56:50.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.814+0000 7f8aeea00700 1 -- 172.21.15.123:0/53861165 wait complete. 2024-03-30T04:56:50.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.946+0000 7fc604600700 1 -- 172.21.15.123:0/1820727302 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5fc0fe830 msgr2=0x7fc5fc106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:50.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.946+0000 7fc604600700 1 --2- 172.21.15.123:0/1820727302 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5fc0fe830 0x7fc5fc106f00 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fc5e8004940 tx=0x7fc5e8009e30 comp rx=0 tx=0).stop 2024-03-30T04:56:50.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.946+0000 7fc604600700 1 -- 172.21.15.123:0/1820727302 shutdown_connections 2024-03-30T04:56:50.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.946+0000 7fc604600700 1 --2- 172.21.15.123:0/1820727302 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5fc0fe830 0x7fc5fc106f00 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:50.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.946+0000 7fc604600700 1 --2- 172.21.15.123:0/1820727302 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc5fc0fde90 0x7fc5fc0fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:50.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.946+0000 7fc604600700 1 -- 172.21.15.123:0/1820727302 >> 172.21.15.123:0/1820727302 conn(0x7fc5fc072bc0 msgr2=0x7fc5fc072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:56:50.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.946+0000 7fc604600700 1 -- 172.21.15.123:0/1820727302 shutdown_connections 2024-03-30T04:56:50.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.946+0000 7fc604600700 1 -- 172.21.15.123:0/1820727302 wait complete. 2024-03-30T04:56:50.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.947+0000 7fc604600700 1 Processor -- start 2024-03-30T04:56:50.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.947+0000 7fc604600700 1 -- start start 2024-03-30T04:56:50.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.947+0000 7fc604600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5fc0fde90 0x7fc5fc115d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:56:50.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.947+0000 7fc604600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc5fc0fe830 0x7fc5fc116270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:56:50.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.947+0000 7fc604600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5fc1168c0 con 0x7fc5fc0fde90 2024-03-30T04:56:50.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.947+0000 7fc604600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5fc10fc90 con 0x7fc5fc0fe830 2024-03-30T04:56:50.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.948+0000 7fc601c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5fc0fde90 0x7fc5fc115d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:56:50.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.948+0000 7fc601c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5fc0fde90 0x7fc5fc115d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35344/0 (socket says 172.21.15.123:35344) 2024-03-30T04:56:50.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.948+0000 7fc601c00700 1 -- 172.21.15.123:0/4095278636 learned_addr learned my addr 172.21.15.123:0/4095278636 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:56:50.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.948+0000 7fc601200700 1 --2- 172.21.15.123:0/4095278636 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc5fc0fe830 0x7fc5fc116270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:56:50.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.948+0000 7fc601c00700 1 -- 172.21.15.123:0/4095278636 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc5fc0fe830 msgr2=0x7fc5fc116270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:50.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.948+0000 7fc601c00700 1 --2- 172.21.15.123:0/4095278636 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc5fc0fe830 0x7fc5fc116270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:50.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.948+0000 7fc601c00700 1 -- 172.21.15.123:0/4095278636 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e8005040 con 0x7fc5fc0fde90 2024-03-30T04:56:50.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.948+0000 7fc601c00700 1 --2- 172.21.15.123:0/4095278636 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5fc0fde90 0x7fc5fc115d30 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fc5f0002a50 tx=0x7fc5f00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:56:50.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.949+0000 7fc5f9600700 1 -- 172.21.15.123:0/4095278636 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc5f0007cc0 con 0x7fc5fc0fde90 2024-03-30T04:56:50.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.949+0000 7fc5f9600700 1 -- 172.21.15.123:0/4095278636 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc5f000f040 con 0x7fc5fc0fde90 2024-03-30T04:56:50.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.949+0000 7fc5f9600700 1 -- 172.21.15.123:0/4095278636 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc5f000d9c0 con 0x7fc5fc0fde90 2024-03-30T04:56:50.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.949+0000 7fc604600700 1 -- 172.21.15.123:0/4095278636 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5fc10ff70 con 0x7fc5fc0fde90 2024-03-30T04:56:50.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.949+0000 7fc604600700 1 -- 172.21.15.123:0/4095278636 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5fc1104b0 con 0x7fc5fc0fde90 2024-03-30T04:56:50.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.950+0000 7fc604600700 1 -- 172.21.15.123:0/4095278636 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5fc106a20 con 0x7fc5fc0fde90 2024-03-30T04:56:50.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.951+0000 7fc5f9600700 1 -- 172.21.15.123:0/4095278636 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fc5f000db20 con 0x7fc5fc0fde90 2024-03-30T04:56:50.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.952+0000 7fc5f9600700 1 --2- 172.21.15.123:0/4095278636 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc5e406cc30 0x7fc5e406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:56:50.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.952+0000 7fc5f9600700 1 -- 172.21.15.123:0/4095278636 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc5f008bde0 con 0x7fc5fc0fde90 2024-03-30T04:56:50.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.953+0000 7fc601200700 1 --2- 172.21.15.123:0/4095278636 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc5e406cc30 0x7fc5e406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:56:50.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.953+0000 7fc601200700 1 --2- 172.21.15.123:0/4095278636 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc5e406cc30 0x7fc5e406f0e0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc1115d0 tx=0x7fc5e8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:56:50.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:50.957+0000 7fc5f9600700 1 -- 172.21.15.123:0/4095278636 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc5f00597b0 con 0x7fc5fc0fde90 2024-03-30T04:56:51.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:51.241+0000 7fc604600700 1 -- 172.21.15.123:0/4095278636 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc5fc04bed0 con 0x7fc5fc0fde90 2024-03-30T04:56:51.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:51.241+0000 7fc5f9600700 1 -- 172.21.15.123:0/4095278636 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7fc5f0059340 con 0x7fc5fc0fde90 2024-03-30T04:56:51.240 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T04:56:51.240 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-30T04:56:51.240 INFO:teuthology.orchestra.run.smithi123.stdout: host smithi135 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi135/config 2024-03-30T04:56:51.240 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-30T04:56:51.240 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T04:56:51.240 INFO:teuthology.orchestra.run.smithi123.stdout:Traceback (most recent call last): 2024-03-30T04:56:51.240 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-30T04:56:51.240 INFO:teuthology.orchestra.run.smithi123.stdout: "__main__", mod_spec) 2024-03-30T04:56:51.240 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-30T04:56:51.240 INFO:teuthology.orchestra.run.smithi123.stdout: exec(code, run_globals) 2024-03-30T04:56:51.241 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10889, in 2024-03-30T04:56:51.241 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10877, in main 2024-03-30T04:56:51.241 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-30T04:56:51.241 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-30T04:56:51.241 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-30T04:56:51.241 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-30T04:56:51.241 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-03-30T04:56:51.241 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-30T04:56:51.241 INFO:teuthology.orchestra.run.smithi123.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T04:56:51.241 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T04:56:51.241 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T04:56:51.241 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T04:56:51.241 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T04:56:51.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:51.243+0000 7fc604600700 1 -- 172.21.15.123:0/4095278636 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc5e406cc30 msgr2=0x7fc5e406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:51.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:51.243+0000 7fc604600700 1 --2- 172.21.15.123:0/4095278636 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc5e406cc30 0x7fc5e406f0e0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc1115d0 tx=0x7fc5e8009e30 comp rx=0 tx=0).stop 2024-03-30T04:56:51.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:51.244+0000 7fc604600700 1 -- 172.21.15.123:0/4095278636 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5fc0fde90 msgr2=0x7fc5fc115d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:56:51.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:51.244+0000 7fc604600700 1 --2- 172.21.15.123:0/4095278636 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5fc0fde90 0x7fc5fc115d30 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fc5f0002a50 tx=0x7fc5f00046b0 comp rx=0 tx=0).stop 2024-03-30T04:56:51.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:51.244+0000 7fc604600700 1 -- 172.21.15.123:0/4095278636 shutdown_connections 2024-03-30T04:56:51.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:51.244+0000 7fc604600700 1 --2- 172.21.15.123:0/4095278636 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc5e406cc30 0x7fc5e406f0e0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:51.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:51.244+0000 7fc604600700 1 --2- 172.21.15.123:0/4095278636 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5fc0fde90 0x7fc5fc115d30 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:51.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:51.244+0000 7fc604600700 1 --2- 172.21.15.123:0/4095278636 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc5fc0fe830 0x7fc5fc116270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:56:51.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:51.244+0000 7fc604600700 1 -- 172.21.15.123:0/4095278636 >> 172.21.15.123:0/4095278636 conn(0x7fc5fc072bc0 msgr2=0x7fc5fc0fb840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:56:51.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:51.244+0000 7fc604600700 1 -- 172.21.15.123:0/4095278636 shutdown_connections 2024-03-30T04:56:51.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:56:51.244+0000 7fc604600700 1 -- 172.21.15.123:0/4095278636 wait complete. 2024-03-30T04:56:51.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:56:51 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/787750064' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:56:51.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:56:51 smithi123 ceph-mon[139369]: from='client.35432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:56:51.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:56:51 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/787750064' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:56:51.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:56:51 smithi135 ceph-mon[106011]: from='client.35432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:56:52.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:56:52 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/4095278636' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:56:52.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:56:52 smithi123 ceph-mon[139369]: pgmap v771: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:52.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:56:52 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/4095278636' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:56:52.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:56:52 smithi135 ceph-mon[106011]: pgmap v771: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:54.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:56:54 smithi135 ceph-mon[106011]: pgmap v772: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:54.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:56:54 smithi123 ceph-mon[139369]: pgmap v772: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:56.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:56:56 smithi123 ceph-mon[139369]: pgmap v773: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:57.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:56:56 smithi135 ceph-mon[106011]: pgmap v773: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:58.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:56:58 smithi123 ceph-mon[139369]: pgmap v774: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:59.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:56:58 smithi135 ceph-mon[106011]: pgmap v774: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:56:59.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:56:59 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:57:00.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:56:59 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:57:00.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:57:00 smithi123 ceph-mon[139369]: pgmap v775: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:01.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:57:00 smithi135 ceph-mon[106011]: pgmap v775: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:02.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:57:02 smithi123 ceph-mon[139369]: pgmap v776: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:03.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:57:02 smithi135 ceph-mon[106011]: pgmap v776: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:04.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:57:04 smithi123 ceph-mon[139369]: pgmap v777: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:05.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:57:04 smithi135 ceph-mon[106011]: pgmap v777: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:07.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:57:06 smithi135 ceph-mon[106011]: pgmap v778: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:07.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:57:06 smithi123 ceph-mon[139369]: pgmap v778: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:09.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:57:08 smithi135 ceph-mon[106011]: pgmap v779: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:09.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:57:08 smithi123 ceph-mon[139369]: pgmap v779: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:11.060 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:57:10 smithi135 ceph-mon[106011]: pgmap v780: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:11.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:57:10 smithi123 ceph-mon[139369]: pgmap v780: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:13.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:57:12 smithi135 ceph-mon[106011]: pgmap v781: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:13.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:57:12 smithi123 ceph-mon[139369]: pgmap v781: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:15.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:57:14 smithi135 ceph-mon[106011]: pgmap v782: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:15.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:57:14 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:57:15.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:57:14 smithi123 ceph-mon[139369]: pgmap v782: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:15.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:57:14 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:57:17.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:57:16 smithi135 ceph-mon[106011]: pgmap v783: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:17.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:57:16 smithi123 ceph-mon[139369]: pgmap v783: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:19.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:57:18 smithi135 ceph-mon[106011]: pgmap v784: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:19.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:57:18 smithi123 ceph-mon[139369]: pgmap v784: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:21.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:57:20 smithi135 ceph-mon[106011]: pgmap v785: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:21.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:57:20 smithi123 ceph-mon[139369]: pgmap v785: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:21.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.391+0000 7f9d5a000700 1 -- 172.21.15.123:0/2057457940 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d54102e20 msgr2=0x7f9d541031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:21.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.391+0000 7f9d5a000700 1 --2- 172.21.15.123:0/2057457940 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d54102e20 0x7f9d541031f0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f9d44004940 tx=0x7f9d44009e30 comp rx=0 tx=0).stop 2024-03-30T04:57:21.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.392+0000 7f9d5a000700 1 -- 172.21.15.123:0/2057457940 shutdown_connections 2024-03-30T04:57:21.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.392+0000 7f9d5a000700 1 --2- 172.21.15.123:0/2057457940 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9d541037c0 0x7f9d540fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:21.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.392+0000 7f9d5a000700 1 --2- 172.21.15.123:0/2057457940 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d54102e20 0x7f9d541031f0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:21.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.392+0000 7f9d5a000700 1 -- 172.21.15.123:0/2057457940 >> 172.21.15.123:0/2057457940 conn(0x7f9d540f89d0 msgr2=0x7f9d540fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:57:21.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.392+0000 7f9d5a000700 1 -- 172.21.15.123:0/2057457940 shutdown_connections 2024-03-30T04:57:21.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.392+0000 7f9d5a000700 1 -- 172.21.15.123:0/2057457940 wait complete. 2024-03-30T04:57:21.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.392+0000 7f9d5a000700 1 Processor -- start 2024-03-30T04:57:21.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.393+0000 7f9d5a000700 1 -- start start 2024-03-30T04:57:21.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.393+0000 7f9d5a000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9d54102e20 0x7f9d54113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:57:21.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.393+0000 7f9d5a000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d541037c0 0x7f9d54114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:57:21.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.393+0000 7f9d5a000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d54114690 con 0x7f9d541037c0 2024-03-30T04:57:21.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.393+0000 7f9d5a000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5410da80 con 0x7f9d54102e20 2024-03-30T04:57:21.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.393+0000 7f9d4be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d541037c0 0x7f9d54114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:57:21.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.393+0000 7f9d4be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d541037c0 0x7f9d54114040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36638/0 (socket says 172.21.15.123:36638) 2024-03-30T04:57:21.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.393+0000 7f9d4be00700 1 -- 172.21.15.123:0/311854787 learned_addr learned my addr 172.21.15.123:0/311854787 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:57:21.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.393+0000 7f9d53400700 1 --2- 172.21.15.123:0/311854787 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9d54102e20 0x7f9d54113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:57:21.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.394+0000 7f9d4be00700 1 -- 172.21.15.123:0/311854787 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9d54102e20 msgr2=0x7f9d54113b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:21.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.394+0000 7f9d4be00700 1 --2- 172.21.15.123:0/311854787 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9d54102e20 0x7f9d54113b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:21.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.394+0000 7f9d4be00700 1 -- 172.21.15.123:0/311854787 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d44005040 con 0x7f9d541037c0 2024-03-30T04:57:21.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.394+0000 7f9d4be00700 1 --2- 172.21.15.123:0/311854787 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d541037c0 0x7f9d54114040 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c002a50 tx=0x7f9d3c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:57:21.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.394+0000 7f9d4b400700 1 -- 172.21.15.123:0/311854787 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d3c007cc0 con 0x7f9d541037c0 2024-03-30T04:57:21.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.394+0000 7f9d4b400700 1 -- 172.21.15.123:0/311854787 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9d3c00f040 con 0x7f9d541037c0 2024-03-30T04:57:21.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.394+0000 7f9d4b400700 1 -- 172.21.15.123:0/311854787 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d3c00d920 con 0x7f9d541037c0 2024-03-30T04:57:21.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.394+0000 7f9d5a000700 1 -- 172.21.15.123:0/311854787 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d5410dd60 con 0x7f9d541037c0 2024-03-30T04:57:21.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.394+0000 7f9d5a000700 1 -- 172.21.15.123:0/311854787 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d5410e320 con 0x7f9d541037c0 2024-03-30T04:57:21.395 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.396+0000 7f9d5a000700 1 -- 172.21.15.123:0/311854787 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d540fe570 con 0x7f9d541037c0 2024-03-30T04:57:21.395 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.397+0000 7f9d4b400700 1 -- 172.21.15.123:0/311854787 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f9d3c009680 con 0x7f9d541037c0 2024-03-30T04:57:21.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.397+0000 7f9d4b400700 1 --2- 172.21.15.123:0/311854787 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9d3806cc20 0x7f9d3806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:57:21.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.398+0000 7f9d4b400700 1 -- 172.21.15.123:0/311854787 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f9d3c08bd10 con 0x7f9d541037c0 2024-03-30T04:57:21.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.398+0000 7f9d53400700 1 --2- 172.21.15.123:0/311854787 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9d3806cc20 0x7f9d3806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:57:21.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.398+0000 7f9d53400700 1 --2- 172.21.15.123:0/311854787 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9d3806cc20 0x7f9d3806f0d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f9d44004f90 tx=0x7f9d44009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:57:21.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.402+0000 7f9d4b400700 1 -- 172.21.15.123:0/311854787 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9d3c059760 con 0x7f9d541037c0 2024-03-30T04:57:21.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.625+0000 7f9d5a000700 1 -- 172.21.15.123:0/311854787 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9d54109e80 con 0x7f9d3806cc20 2024-03-30T04:57:21.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.627+0000 7f9d4b400700 1 -- 172.21.15.123:0/311854787 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f9d44007d70 con 0x7f9d3806cc20 2024-03-30T04:57:21.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.629+0000 7f9d5a000700 1 -- 172.21.15.123:0/311854787 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9d3806cc20 msgr2=0x7f9d3806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:21.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.629+0000 7f9d5a000700 1 --2- 172.21.15.123:0/311854787 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9d3806cc20 0x7f9d3806f0d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f9d44004f90 tx=0x7f9d44009e30 comp rx=0 tx=0).stop 2024-03-30T04:57:21.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.629+0000 7f9d5a000700 1 -- 172.21.15.123:0/311854787 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d541037c0 msgr2=0x7f9d54114040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:21.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.629+0000 7f9d5a000700 1 --2- 172.21.15.123:0/311854787 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d541037c0 0x7f9d54114040 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c002a50 tx=0x7f9d3c004600 comp rx=0 tx=0).stop 2024-03-30T04:57:21.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.629+0000 7f9d5a000700 1 -- 172.21.15.123:0/311854787 shutdown_connections 2024-03-30T04:57:21.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.629+0000 7f9d5a000700 1 --2- 172.21.15.123:0/311854787 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9d3806cc20 0x7f9d3806f0d0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:21.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.629+0000 7f9d5a000700 1 --2- 172.21.15.123:0/311854787 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9d54102e20 0x7f9d54113b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:21.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.629+0000 7f9d5a000700 1 --2- 172.21.15.123:0/311854787 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d541037c0 0x7f9d54114040 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:21.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.629+0000 7f9d5a000700 1 -- 172.21.15.123:0/311854787 >> 172.21.15.123:0/311854787 conn(0x7f9d540f89d0 msgr2=0x7f9d540f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:57:21.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.629+0000 7f9d5a000700 1 -- 172.21.15.123:0/311854787 shutdown_connections 2024-03-30T04:57:21.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.630+0000 7f9d5a000700 1 -- 172.21.15.123:0/311854787 wait complete. 2024-03-30T04:57:21.642 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:57:21.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.764+0000 7f0ac4200700 1 -- 172.21.15.123:0/3249489493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0abc102e00 msgr2=0x7f0abc1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:21.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.764+0000 7f0ac4200700 1 --2- 172.21.15.123:0/3249489493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0abc102e00 0x7f0abc1031d0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0004940 tx=0x7f0ab0009e30 comp rx=0 tx=0).stop 2024-03-30T04:57:21.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.764+0000 7f0ac4200700 1 -- 172.21.15.123:0/3249489493 shutdown_connections 2024-03-30T04:57:21.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.764+0000 7f0ac4200700 1 --2- 172.21.15.123:0/3249489493 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0abc1037a0 0x7f0abc0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:21.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.764+0000 7f0ac4200700 1 --2- 172.21.15.123:0/3249489493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0abc102e00 0x7f0abc1031d0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:21.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.764+0000 7f0ac4200700 1 -- 172.21.15.123:0/3249489493 >> 172.21.15.123:0/3249489493 conn(0x7f0abc0f89b0 msgr2=0x7f0abc0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:57:21.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.764+0000 7f0ac4200700 1 -- 172.21.15.123:0/3249489493 shutdown_connections 2024-03-30T04:57:21.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.764+0000 7f0ac4200700 1 -- 172.21.15.123:0/3249489493 wait complete. 2024-03-30T04:57:21.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.765+0000 7f0ac4200700 1 Processor -- start 2024-03-30T04:57:21.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.766+0000 7f0ac4200700 1 -- start start 2024-03-30T04:57:21.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.766+0000 7f0ac4200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0abc102e00 0x7f0abc115cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:57:21.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.766+0000 7f0ac4200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0abc1037a0 0x7f0abc116200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:57:21.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.766+0000 7f0ac4200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0abc1168e0 con 0x7f0abc1037a0 2024-03-30T04:57:21.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.766+0000 7f0ac4200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0abc10fc20 con 0x7f0abc102e00 2024-03-30T04:57:21.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.766+0000 7f0ac0e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0abc1037a0 0x7f0abc116200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:57:21.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.767+0000 7f0ac0e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0abc1037a0 0x7f0abc116200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36652/0 (socket says 172.21.15.123:36652) 2024-03-30T04:57:21.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.767+0000 7f0ac0e00700 1 -- 172.21.15.123:0/4281138231 learned_addr learned my addr 172.21.15.123:0/4281138231 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:57:21.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.767+0000 7f0ac1800700 1 --2- 172.21.15.123:0/4281138231 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0abc102e00 0x7f0abc115cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:57:21.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.767+0000 7f0ac0e00700 1 -- 172.21.15.123:0/4281138231 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0abc102e00 msgr2=0x7f0abc115cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:21.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.767+0000 7f0ac0e00700 1 --2- 172.21.15.123:0/4281138231 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0abc102e00 0x7f0abc115cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:21.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.767+0000 7f0ac0e00700 1 -- 172.21.15.123:0/4281138231 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab0005040 con 0x7f0abc1037a0 2024-03-30T04:57:21.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.767+0000 7f0ac0e00700 1 --2- 172.21.15.123:0/4281138231 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0abc1037a0 0x7f0abc116200 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f0aa800b790 tx=0x7f0aa8009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:57:21.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.767+0000 7f0ab9600700 1 -- 172.21.15.123:0/4281138231 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0aa800f6f0 con 0x7f0abc1037a0 2024-03-30T04:57:21.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.767+0000 7f0ab9600700 1 -- 172.21.15.123:0/4281138231 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0aa800fd00 con 0x7f0abc1037a0 2024-03-30T04:57:21.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.767+0000 7f0ac4200700 1 -- 172.21.15.123:0/4281138231 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0abc10fea0 con 0x7f0abc1037a0 2024-03-30T04:57:21.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.767+0000 7f0ab9600700 1 -- 172.21.15.123:0/4281138231 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0aa80103a0 con 0x7f0abc1037a0 2024-03-30T04:57:21.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.768+0000 7f0ac4200700 1 -- 172.21.15.123:0/4281138231 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0abc110460 con 0x7f0abc1037a0 2024-03-30T04:57:21.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.769+0000 7f0ac4200700 1 -- 172.21.15.123:0/4281138231 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0abc0fe550 con 0x7f0abc1037a0 2024-03-30T04:57:21.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.769+0000 7f0ab9600700 1 -- 172.21.15.123:0/4281138231 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f0aa800f850 con 0x7f0abc1037a0 2024-03-30T04:57:21.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.770+0000 7f0ab9600700 1 --2- 172.21.15.123:0/4281138231 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0aa406cc30 0x7f0aa406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:57:21.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.771+0000 7f0ab9600700 1 -- 172.21.15.123:0/4281138231 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f0aa808be10 con 0x7f0abc1037a0 2024-03-30T04:57:21.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.771+0000 7f0ac1800700 1 --2- 172.21.15.123:0/4281138231 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0aa406cc30 0x7f0aa406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:57:21.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.771+0000 7f0ac1800700 1 --2- 172.21.15.123:0/4281138231 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0aa406cc30 0x7f0aa406f0e0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f0ab000b840 tx=0x7f0ab0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:57:21.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.775+0000 7f0ab9600700 1 -- 172.21.15.123:0/4281138231 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0aa80598e0 con 0x7f0abc1037a0 2024-03-30T04:57:21.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.994+0000 7f0ac4200700 1 -- 172.21.15.123:0/4281138231 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0abc109ea0 con 0x7f0aa406cc30 2024-03-30T04:57:21.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.996+0000 7f0ab9600700 1 -- 172.21.15.123:0/4281138231 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f0abc109ea0 con 0x7f0aa406cc30 2024-03-30T04:57:21.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.998+0000 7f0ac4200700 1 -- 172.21.15.123:0/4281138231 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0aa406cc30 msgr2=0x7f0aa406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:21.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.998+0000 7f0ac4200700 1 --2- 172.21.15.123:0/4281138231 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0aa406cc30 0x7f0aa406f0e0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f0ab000b840 tx=0x7f0ab0009e30 comp rx=0 tx=0).stop 2024-03-30T04:57:21.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.998+0000 7f0ac4200700 1 -- 172.21.15.123:0/4281138231 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0abc1037a0 msgr2=0x7f0abc116200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:21.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.998+0000 7f0ac4200700 1 --2- 172.21.15.123:0/4281138231 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0abc1037a0 0x7f0abc116200 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f0aa800b790 tx=0x7f0aa8009220 comp rx=0 tx=0).stop 2024-03-30T04:57:21.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.998+0000 7f0ac4200700 1 -- 172.21.15.123:0/4281138231 shutdown_connections 2024-03-30T04:57:21.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.998+0000 7f0ac4200700 1 --2- 172.21.15.123:0/4281138231 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0aa406cc30 0x7f0aa406f0e0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:21.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.998+0000 7f0ac4200700 1 --2- 172.21.15.123:0/4281138231 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0abc102e00 0x7f0abc115cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:21.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.998+0000 7f0ac4200700 1 --2- 172.21.15.123:0/4281138231 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0abc1037a0 0x7f0abc116200 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:21.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.998+0000 7f0ac4200700 1 -- 172.21.15.123:0/4281138231 >> 172.21.15.123:0/4281138231 conn(0x7f0abc0f89b0 msgr2=0x7f0abc071ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:57:21.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.998+0000 7f0ac4200700 1 -- 172.21.15.123:0/4281138231 shutdown_connections 2024-03-30T04:57:21.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:21.998+0000 7f0ac4200700 1 -- 172.21.15.123:0/4281138231 wait complete. 2024-03-30T04:57:22.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.132+0000 7f3011600700 1 -- 172.21.15.123:0/2468518112 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f300c065630 msgr2=0x7f300c065a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:22.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.132+0000 7f3011600700 1 --2- 172.21.15.123:0/2468518112 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f300c065630 0x7f300c065a80 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f2ff8004940 tx=0x7f2ff8009e30 comp rx=0 tx=0).stop 2024-03-30T04:57:22.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.132+0000 7f3011600700 1 -- 172.21.15.123:0/2468518112 shutdown_connections 2024-03-30T04:57:22.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.132+0000 7f3011600700 1 --2- 172.21.15.123:0/2468518112 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f300c065630 0x7f300c065a80 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:22.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.132+0000 7f3011600700 1 --2- 172.21.15.123:0/2468518112 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f300c102ca0 0x7f300c103070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:22.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.132+0000 7f3011600700 1 -- 172.21.15.123:0/2468518112 >> 172.21.15.123:0/2468518112 conn(0x7f300c071b70 msgr2=0x7f300c071f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:57:22.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.132+0000 7f3011600700 1 -- 172.21.15.123:0/2468518112 shutdown_connections 2024-03-30T04:57:22.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.132+0000 7f3011600700 1 -- 172.21.15.123:0/2468518112 wait complete. 2024-03-30T04:57:22.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.133+0000 7f3011600700 1 Processor -- start 2024-03-30T04:57:22.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.134+0000 7f3011600700 1 -- start start 2024-03-30T04:57:22.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.134+0000 7f3011600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f300c065630 0x7f300c1138c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:57:22.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.134+0000 7f3011600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f300c102ca0 0x7f300c113e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:57:22.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.134+0000 7f3011600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f300c114450 con 0x7f300c065630 2024-03-30T04:57:22.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.134+0000 7f3011600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f300c10d840 con 0x7f300c102ca0 2024-03-30T04:57:22.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.134+0000 7f300be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f300c065630 0x7f300c1138c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:57:22.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.134+0000 7f300be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f300c065630 0x7f300c1138c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36656/0 (socket says 172.21.15.123:36656) 2024-03-30T04:57:22.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.134+0000 7f300be00700 1 -- 172.21.15.123:0/204481571 learned_addr learned my addr 172.21.15.123:0/204481571 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:57:22.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.135+0000 7f300be00700 1 -- 172.21.15.123:0/204481571 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f300c102ca0 msgr2=0x7f300c113e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:22.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.135+0000 7f300be00700 1 --2- 172.21.15.123:0/204481571 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f300c102ca0 0x7f300c113e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:22.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.135+0000 7f300be00700 1 -- 172.21.15.123:0/204481571 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff8005040 con 0x7f300c065630 2024-03-30T04:57:22.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.135+0000 7f300be00700 1 --2- 172.21.15.123:0/204481571 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f300c065630 0x7f300c1138c0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc002a50 tx=0x7f2ffc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:57:22.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.135+0000 7f3003e00700 1 -- 172.21.15.123:0/204481571 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ffc007cc0 con 0x7f300c065630 2024-03-30T04:57:22.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.135+0000 7f3003e00700 1 -- 172.21.15.123:0/204481571 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2ffc00f040 con 0x7f300c065630 2024-03-30T04:57:22.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.135+0000 7f3011600700 1 -- 172.21.15.123:0/204481571 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f300c10db20 con 0x7f300c065630 2024-03-30T04:57:22.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.135+0000 7f3003e00700 1 -- 172.21.15.123:0/204481571 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ffc009540 con 0x7f300c065630 2024-03-30T04:57:22.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.135+0000 7f3011600700 1 -- 172.21.15.123:0/204481571 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f300c10e0e0 con 0x7f300c065630 2024-03-30T04:57:22.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.136+0000 7f3011600700 1 -- 172.21.15.123:0/204481571 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f300c0fe380 con 0x7f300c065630 2024-03-30T04:57:22.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.137+0000 7f3003e00700 1 -- 172.21.15.123:0/204481571 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f2ffc0096a0 con 0x7f300c065630 2024-03-30T04:57:22.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.138+0000 7f3003e00700 1 --2- 172.21.15.123:0/204481571 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2ff006cc30 0x7f2ff006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:57:22.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.138+0000 7f3003e00700 1 -- 172.21.15.123:0/204481571 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f2ffc08bee0 con 0x7f300c065630 2024-03-30T04:57:22.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.138+0000 7f300b400700 1 --2- 172.21.15.123:0/204481571 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2ff006cc30 0x7f2ff006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:57:22.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.139+0000 7f300b400700 1 --2- 172.21.15.123:0/204481571 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2ff006cc30 0x7f2ff006f0e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f2ff8004f90 tx=0x7f2ff8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:57:22.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.143+0000 7f3003e00700 1 -- 172.21.15.123:0/204481571 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2ffc0598b0 con 0x7f300c065630 2024-03-30T04:57:22.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.354+0000 7f3011600700 1 -- 172.21.15.123:0/204481571 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f300c109c10 con 0x7f2ff006cc30 2024-03-30T04:57:22.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.364+0000 7f3003e00700 1 -- 172.21.15.123:0/204481571 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f300c109c10 con 0x7f2ff006cc30 2024-03-30T04:57:22.362 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:57:22.363 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (26m) 3m ago 39m 14.6M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:57:22.363 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (23m) 3m ago 39m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T04:57:22.363 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (23m) 3m ago 38m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T04:57:22.363 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (26m) 3m ago 38m 73.5M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T04:57:22.363 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (19m) 3m ago 32m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T04:57:22.363 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (19m) 3m ago 32m 16.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T04:57:22.363 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (4m) 3m ago 32m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 87b073a6c9ca 2024-03-30T04:57:22.363 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (19m) 3m ago 32m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T04:57:22.363 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (28m) 3m ago 40m 510M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:57:22.363 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (27m) 3m ago 37m 456M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:57:22.363 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (25m) 3m ago 40m 70.0M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T04:57:22.363 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (24m) 3m ago 37m 51.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T04:57:22.363 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (27m) 3m ago 39m 8636k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:57:22.364 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (27m) 3m ago 38m 8678k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:57:22.364 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (23m) 3m ago 37m 107M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T04:57:22.364 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (22m) 3m ago 36m 97.2M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T04:57:22.364 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (22m) 3m ago 35m 89.8M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T04:57:22.364 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (21m) 3m ago 35m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T04:57:22.364 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (20m) 3m ago 34m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T04:57:22.364 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (20m) 3m ago 34m 88.8M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T04:57:22.364 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (26m) 3m ago 38m 63.9M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:57:22.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.367+0000 7f3011600700 1 -- 172.21.15.123:0/204481571 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2ff006cc30 msgr2=0x7f2ff006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:22.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.367+0000 7f3011600700 1 --2- 172.21.15.123:0/204481571 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2ff006cc30 0x7f2ff006f0e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f2ff8004f90 tx=0x7f2ff8009e30 comp rx=0 tx=0).stop 2024-03-30T04:57:22.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.367+0000 7f3011600700 1 -- 172.21.15.123:0/204481571 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f300c065630 msgr2=0x7f300c1138c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:22.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.367+0000 7f3011600700 1 --2- 172.21.15.123:0/204481571 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f300c065630 0x7f300c1138c0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc002a50 tx=0x7f2ffc0046b0 comp rx=0 tx=0).stop 2024-03-30T04:57:22.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.367+0000 7f3011600700 1 -- 172.21.15.123:0/204481571 shutdown_connections 2024-03-30T04:57:22.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.367+0000 7f3011600700 1 --2- 172.21.15.123:0/204481571 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2ff006cc30 0x7f2ff006f0e0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:22.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.367+0000 7f3011600700 1 --2- 172.21.15.123:0/204481571 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f300c065630 0x7f300c1138c0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:22.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.367+0000 7f3011600700 1 --2- 172.21.15.123:0/204481571 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f300c102ca0 0x7f300c113e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:22.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.367+0000 7f3011600700 1 -- 172.21.15.123:0/204481571 >> 172.21.15.123:0/204481571 conn(0x7f300c071b70 msgr2=0x7f300c1059d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:57:22.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.367+0000 7f3011600700 1 -- 172.21.15.123:0/204481571 shutdown_connections 2024-03-30T04:57:22.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.367+0000 7f3011600700 1 -- 172.21.15.123:0/204481571 wait complete. 2024-03-30T04:57:22.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.500+0000 7f01dea00700 1 -- 172.21.15.123:0/654879801 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01d8102e00 msgr2=0x7f01d81031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:22.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.500+0000 7f01dea00700 1 --2- 172.21.15.123:0/654879801 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01d8102e00 0x7f01d81031d0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f01c8004940 tx=0x7f01c8009e30 comp rx=0 tx=0).stop 2024-03-30T04:57:22.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.500+0000 7f01dea00700 1 -- 172.21.15.123:0/654879801 shutdown_connections 2024-03-30T04:57:22.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.500+0000 7f01dea00700 1 --2- 172.21.15.123:0/654879801 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f01d81037a0 0x7f01d80fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:22.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.500+0000 7f01dea00700 1 --2- 172.21.15.123:0/654879801 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01d8102e00 0x7f01d81031d0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:22.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.500+0000 7f01dea00700 1 -- 172.21.15.123:0/654879801 >> 172.21.15.123:0/654879801 conn(0x7f01d80f89b0 msgr2=0x7f01d80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:57:22.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.500+0000 7f01dea00700 1 -- 172.21.15.123:0/654879801 shutdown_connections 2024-03-30T04:57:22.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.500+0000 7f01dea00700 1 -- 172.21.15.123:0/654879801 wait complete. 2024-03-30T04:57:22.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.501+0000 7f01dea00700 1 Processor -- start 2024-03-30T04:57:22.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.501+0000 7f01dea00700 1 -- start start 2024-03-30T04:57:22.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.502+0000 7f01dea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01d8102e00 0x7f01d8115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:57:22.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.502+0000 7f01dea00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f01d81037a0 0x7f01d8116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:57:22.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.502+0000 7f01dea00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d8116880 con 0x7f01d8102e00 2024-03-30T04:57:22.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.502+0000 7f01dea00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d810fc50 con 0x7f01d81037a0 2024-03-30T04:57:22.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.502+0000 7f01d7e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01d8102e00 0x7f01d8115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:57:22.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.502+0000 7f01d7e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01d8102e00 0x7f01d8115cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36674/0 (socket says 172.21.15.123:36674) 2024-03-30T04:57:22.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.502+0000 7f01d7e00700 1 -- 172.21.15.123:0/548891741 learned_addr learned my addr 172.21.15.123:0/548891741 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:57:22.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.502+0000 7f01d7400700 1 --2- 172.21.15.123:0/548891741 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f01d81037a0 0x7f01d8116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:57:22.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.502+0000 7f01d7e00700 1 -- 172.21.15.123:0/548891741 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f01d81037a0 msgr2=0x7f01d8116230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:22.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.502+0000 7f01d7e00700 1 --2- 172.21.15.123:0/548891741 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f01d81037a0 0x7f01d8116230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:22.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.502+0000 7f01d7e00700 1 -- 172.21.15.123:0/548891741 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01c8005040 con 0x7f01d8102e00 2024-03-30T04:57:22.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.503+0000 7f01d7e00700 1 --2- 172.21.15.123:0/548891741 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01d8102e00 0x7f01d8115cf0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f01c8009390 tx=0x7f01c8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:57:22.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.503+0000 7f01cfe00700 1 -- 172.21.15.123:0/548891741 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f01c800e040 con 0x7f01d8102e00 2024-03-30T04:57:22.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.503+0000 7f01cfe00700 1 -- 172.21.15.123:0/548891741 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f01c8013420 con 0x7f01d8102e00 2024-03-30T04:57:22.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.503+0000 7f01cfe00700 1 -- 172.21.15.123:0/548891741 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f01c801c630 con 0x7f01d8102e00 2024-03-30T04:57:22.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.503+0000 7f01dea00700 1 -- 172.21.15.123:0/548891741 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01d810fed0 con 0x7f01d8102e00 2024-03-30T04:57:22.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.503+0000 7f01dea00700 1 -- 172.21.15.123:0/548891741 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01d8110490 con 0x7f01d8102e00 2024-03-30T04:57:22.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.504+0000 7f01dea00700 1 -- 172.21.15.123:0/548891741 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01d80fe550 con 0x7f01d8102e00 2024-03-30T04:57:22.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.505+0000 7f01cfe00700 1 -- 172.21.15.123:0/548891741 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f01c80139e0 con 0x7f01d8102e00 2024-03-30T04:57:22.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.506+0000 7f01cfe00700 1 --2- 172.21.15.123:0/548891741 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f01bc06cc30 0x7f01bc06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:57:22.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.506+0000 7f01cfe00700 1 -- 172.21.15.123:0/548891741 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f01c808d250 con 0x7f01d8102e00 2024-03-30T04:57:22.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.506+0000 7f01d7400700 1 --2- 172.21.15.123:0/548891741 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f01bc06cc30 0x7f01bc06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:57:22.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.507+0000 7f01d7400700 1 --2- 172.21.15.123:0/548891741 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f01bc06cc30 0x7f01bc06f0e0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f01c4007fd0 tx=0x7f01c4009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:57:22.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.510+0000 7f01cfe00700 1 -- 172.21.15.123:0/548891741 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f01c805add0 con 0x7f01d8102e00 2024-03-30T04:57:22.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.795+0000 7f01dea00700 1 -- 172.21.15.123:0/548891741 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f01d804bed0 con 0x7f01d8102e00 2024-03-30T04:57:22.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.795+0000 7f01cfe00700 1 -- 172.21.15.123:0/548891741 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f01c805a960 con 0x7f01d8102e00 2024-03-30T04:57:22.794 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:57:22.794 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:57:22.794 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:57:22.794 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:57:22.794 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:57:22.794 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:57:22.794 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:57:22.794 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:57:22.794 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T04:57:22.794 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:57:22.794 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:57:22.795 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 4 2024-03-30T04:57:22.795 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:57:22.795 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:57:22.795 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 14 2024-03-30T04:57:22.795 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:57:22.795 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:57:22.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.798+0000 7f01dea00700 1 -- 172.21.15.123:0/548891741 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f01bc06cc30 msgr2=0x7f01bc06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:22.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.798+0000 7f01dea00700 1 --2- 172.21.15.123:0/548891741 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f01bc06cc30 0x7f01bc06f0e0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f01c4007fd0 tx=0x7f01c4009040 comp rx=0 tx=0).stop 2024-03-30T04:57:22.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.798+0000 7f01dea00700 1 -- 172.21.15.123:0/548891741 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01d8102e00 msgr2=0x7f01d8115cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:22.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.798+0000 7f01dea00700 1 --2- 172.21.15.123:0/548891741 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01d8102e00 0x7f01d8115cf0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f01c8009390 tx=0x7f01c8009e30 comp rx=0 tx=0).stop 2024-03-30T04:57:22.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.798+0000 7f01dea00700 1 -- 172.21.15.123:0/548891741 shutdown_connections 2024-03-30T04:57:22.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.798+0000 7f01dea00700 1 --2- 172.21.15.123:0/548891741 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f01bc06cc30 0x7f01bc06f0e0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:22.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.798+0000 7f01dea00700 1 --2- 172.21.15.123:0/548891741 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01d8102e00 0x7f01d8115cf0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:22.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.798+0000 7f01dea00700 1 --2- 172.21.15.123:0/548891741 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f01d81037a0 0x7f01d8116230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:22.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.798+0000 7f01dea00700 1 -- 172.21.15.123:0/548891741 >> 172.21.15.123:0/548891741 conn(0x7f01d80f89b0 msgr2=0x7f01d80706d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:57:22.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.798+0000 7f01dea00700 1 -- 172.21.15.123:0/548891741 shutdown_connections 2024-03-30T04:57:22.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.798+0000 7f01dea00700 1 -- 172.21.15.123:0/548891741 wait complete. 2024-03-30T04:57:22.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.931+0000 7f4074a00700 1 -- 172.21.15.123:0/367140288 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4070070c10 msgr2=0x7f4070109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:22.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.931+0000 7f4074a00700 1 --2- 172.21.15.123:0/367140288 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4070070c10 0x7f4070109190 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f405c004970 tx=0x7f405c009e30 comp rx=0 tx=0).stop 2024-03-30T04:57:22.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.931+0000 7f4074a00700 1 -- 172.21.15.123:0/367140288 shutdown_connections 2024-03-30T04:57:22.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.931+0000 7f4074a00700 1 --2- 172.21.15.123:0/367140288 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4070070c10 0x7f4070109190 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:22.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.931+0000 7f4074a00700 1 --2- 172.21.15.123:0/367140288 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4070070300 0x7f40700706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:22.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.931+0000 7f4074a00700 1 -- 172.21.15.123:0/367140288 >> 172.21.15.123:0/367140288 conn(0x7f40700f8970 msgr2=0x7f40700fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:57:22.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.931+0000 7f4074a00700 1 -- 172.21.15.123:0/367140288 shutdown_connections 2024-03-30T04:57:22.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.931+0000 7f4074a00700 1 -- 172.21.15.123:0/367140288 wait complete. 2024-03-30T04:57:22.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.932+0000 7f4074a00700 1 Processor -- start 2024-03-30T04:57:22.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.932+0000 7f4074a00700 1 -- start start 2024-03-30T04:57:22.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.933+0000 7f4074a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4070070300 0x7f4070115cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:57:22.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.933+0000 7f4074a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4070070c10 0x7f4070116200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:57:22.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.933+0000 7f4074a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4070116850 con 0x7f4070070300 2024-03-30T04:57:22.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.933+0000 7f4074a00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f407010fc20 con 0x7f4070070c10 2024-03-30T04:57:22.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.933+0000 7f406de00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4070070300 0x7f4070115cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:57:22.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.933+0000 7f406de00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4070070300 0x7f4070115cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36696/0 (socket says 172.21.15.123:36696) 2024-03-30T04:57:22.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.933+0000 7f406de00700 1 -- 172.21.15.123:0/398493620 learned_addr learned my addr 172.21.15.123:0/398493620 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:57:22.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.933+0000 7f406d400700 1 --2- 172.21.15.123:0/398493620 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4070070c10 0x7f4070116200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:57:22.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.933+0000 7f406de00700 1 -- 172.21.15.123:0/398493620 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4070070c10 msgr2=0x7f4070116200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:22.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.933+0000 7f406de00700 1 --2- 172.21.15.123:0/398493620 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4070070c10 0x7f4070116200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:22.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.933+0000 7f406de00700 1 -- 172.21.15.123:0/398493620 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f405c005040 con 0x7f4070070300 2024-03-30T04:57:22.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.934+0000 7f406de00700 1 --2- 172.21.15.123:0/398493620 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4070070300 0x7f4070115cc0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f4060002a50 tx=0x7f40600046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:57:22.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.934+0000 7f4066000700 1 -- 172.21.15.123:0/398493620 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4060007cc0 con 0x7f4070070300 2024-03-30T04:57:22.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.934+0000 7f4066000700 1 -- 172.21.15.123:0/398493620 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f406000f040 con 0x7f4070070300 2024-03-30T04:57:22.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.934+0000 7f4066000700 1 -- 172.21.15.123:0/398493620 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4060009540 con 0x7f4070070300 2024-03-30T04:57:22.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.934+0000 7f4074a00700 1 -- 172.21.15.123:0/398493620 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f407010ff00 con 0x7f4070070300 2024-03-30T04:57:22.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.934+0000 7f4074a00700 1 -- 172.21.15.123:0/398493620 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4070110410 con 0x7f4070070300 2024-03-30T04:57:22.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.935+0000 7f4074a00700 1 -- 172.21.15.123:0/398493620 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4070071a40 con 0x7f4070070300 2024-03-30T04:57:22.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.936+0000 7f4066000700 1 -- 172.21.15.123:0/398493620 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f40600096a0 con 0x7f4070070300 2024-03-30T04:57:22.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.937+0000 7f4066000700 1 --2- 172.21.15.123:0/398493620 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f405406cc30 0x7f405406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:57:22.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.937+0000 7f4066000700 1 -- 172.21.15.123:0/398493620 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f406008bde0 con 0x7f4070070300 2024-03-30T04:57:22.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.937+0000 7f406d400700 1 --2- 172.21.15.123:0/398493620 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f405406cc30 0x7f405406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:57:22.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.938+0000 7f406d400700 1 --2- 172.21.15.123:0/398493620 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f405406cc30 0x7f405406f0e0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f405c00b370 tx=0x7f405c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:57:22.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:22.941+0000 7f4066000700 1 -- 172.21.15.123:0/398493620 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4060059830 con 0x7f4070070300 2024-03-30T04:57:23.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:57:22 smithi135 ceph-mon[106011]: pgmap v786: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:23.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:57:22 smithi135 ceph-mon[106011]: from='client.35440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:57:23.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:57:22 smithi135 ceph-mon[106011]: from='client.35444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:57:23.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.188+0000 7f4074a00700 1 -- 172.21.15.123:0/398493620 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f407004bed0 con 0x7f4070070300 2024-03-30T04:57:23.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.189+0000 7f4066000700 1 -- 172.21.15.123:0/398493620 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 34 v34) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f40600593c0 con 0x7f4070070300 2024-03-30T04:57:23.187 INFO:teuthology.orchestra.run.smithi123.stdout:e34 2024-03-30T04:57:23.187 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:57:23.187 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:57:23.187 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:57:23.187 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:57:23.187 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:57:23.187 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:57:23.188 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T04:57:23.188 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T04:57:23.188 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:57:23.188 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T04:57:23.188 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:57:23.188 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:57:23.188 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:57:23.188 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:57:23.188 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:57:23.188 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T04:57:23.188 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:57:23.188 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:57:23.188 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T04:57:23.188 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:57:23.189 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T04:57:23.189 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T04:57:23.189 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T04:57:23.189 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T04:57:23.189 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:57:23.189 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:57:23.189 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:57:23.189 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:57:23.189 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:57:23.189 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:57:23.189 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T04:57:23.189 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T04:57:23.189 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:57:23.189 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:57:23.190 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T04:57:23.190 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:57:23.190 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:57:23.190 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:57:23.190 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:57:23.190 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44975} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/3788988014,v1:172.21.15.135:6827/3788988014] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:57:23.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.191+0000 7f4074a00700 1 -- 172.21.15.123:0/398493620 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f405406cc30 msgr2=0x7f405406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:23.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.191+0000 7f4074a00700 1 --2- 172.21.15.123:0/398493620 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f405406cc30 0x7f405406f0e0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f405c00b370 tx=0x7f405c009e30 comp rx=0 tx=0).stop 2024-03-30T04:57:23.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.191+0000 7f4074a00700 1 -- 172.21.15.123:0/398493620 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4070070300 msgr2=0x7f4070115cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:23.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.191+0000 7f4074a00700 1 --2- 172.21.15.123:0/398493620 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4070070300 0x7f4070115cc0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f4060002a50 tx=0x7f40600046b0 comp rx=0 tx=0).stop 2024-03-30T04:57:23.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.191+0000 7f4074a00700 1 -- 172.21.15.123:0/398493620 shutdown_connections 2024-03-30T04:57:23.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.191+0000 7f4074a00700 1 --2- 172.21.15.123:0/398493620 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f405406cc30 0x7f405406f0e0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:23.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.191+0000 7f4074a00700 1 --2- 172.21.15.123:0/398493620 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4070070300 0x7f4070115cc0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:23.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.191+0000 7f4074a00700 1 --2- 172.21.15.123:0/398493620 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4070070c10 0x7f4070116200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:23.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.191+0000 7f4074a00700 1 -- 172.21.15.123:0/398493620 >> 172.21.15.123:0/398493620 conn(0x7f40700f8970 msgr2=0x7f40700f9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:57:23.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.191+0000 7f4074a00700 1 -- 172.21.15.123:0/398493620 shutdown_connections 2024-03-30T04:57:23.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.191+0000 7f4074a00700 1 -- 172.21.15.123:0/398493620 wait complete. 2024-03-30T04:57:23.192 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 34 2024-03-30T04:57:23.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:57:22 smithi123 ceph-mon[139369]: pgmap v786: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:23.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:57:22 smithi123 ceph-mon[139369]: from='client.35440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:57:23.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:57:22 smithi123 ceph-mon[139369]: from='client.35444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:57:23.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.325+0000 7f572b200700 1 -- 172.21.15.123:0/1459293134 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f572c1035d0 msgr2=0x7f572c0fcc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:23.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.325+0000 7f572b200700 1 --2- 172.21.15.123:0/1459293134 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f572c1035d0 0x7f572c0fcc40 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f5718004940 tx=0x7f5718009e30 comp rx=0 tx=0).stop 2024-03-30T04:57:23.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.325+0000 7f572b200700 1 -- 172.21.15.123:0/1459293134 shutdown_connections 2024-03-30T04:57:23.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.325+0000 7f572b200700 1 --2- 172.21.15.123:0/1459293134 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f572c1035d0 0x7f572c0fcc40 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:23.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.325+0000 7f572b200700 1 --2- 172.21.15.123:0/1459293134 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f572c102c30 0x7f572c103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:23.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.325+0000 7f572b200700 1 -- 172.21.15.123:0/1459293134 >> 172.21.15.123:0/1459293134 conn(0x7f572c0f8760 msgr2=0x7f572c0fab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:57:23.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.325+0000 7f572b200700 1 -- 172.21.15.123:0/1459293134 shutdown_connections 2024-03-30T04:57:23.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.325+0000 7f572b200700 1 -- 172.21.15.123:0/1459293134 wait complete. 2024-03-30T04:57:23.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.326+0000 7f572b200700 1 Processor -- start 2024-03-30T04:57:23.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.327+0000 7f572b200700 1 -- start start 2024-03-30T04:57:23.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.327+0000 7f572b200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f572c102c30 0x7f572c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:57:23.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.327+0000 7f572b200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f572c1035d0 0x7f572c070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:57:23.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.327+0000 7f572b200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f572c070d90 con 0x7f572c1035d0 2024-03-30T04:57:23.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.327+0000 7f572b200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f572c070ed0 con 0x7f572c102c30 2024-03-30T04:57:23.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.328+0000 7f5729400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f572c1035d0 0x7f572c070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:57:23.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.328+0000 7f5729400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f572c1035d0 0x7f572c070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36702/0 (socket says 172.21.15.123:36702) 2024-03-30T04:57:23.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.328+0000 7f5729400700 1 -- 172.21.15.123:0/819074430 learned_addr learned my addr 172.21.15.123:0/819074430 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:57:23.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.328+0000 7f5729400700 1 -- 172.21.15.123:0/819074430 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f572c102c30 msgr2=0x7f572c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:23.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.328+0000 7f5729e00700 1 --2- 172.21.15.123:0/819074430 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f572c102c30 0x7f572c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:57:23.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.328+0000 7f5729400700 1 --2- 172.21.15.123:0/819074430 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f572c102c30 0x7f572c0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:23.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.328+0000 7f5729400700 1 -- 172.21.15.123:0/819074430 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5718005040 con 0x7f572c1035d0 2024-03-30T04:57:23.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.328+0000 7f5729e00700 1 --2- 172.21.15.123:0/819074430 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f572c102c30 0x7f572c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:57:23.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.328+0000 7f5729400700 1 --2- 172.21.15.123:0/819074430 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f572c1035d0 0x7f572c070800 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f5718009390 tx=0x7f5718009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:57:23.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.328+0000 7f5722000700 1 -- 172.21.15.123:0/819074430 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f571800e040 con 0x7f572c1035d0 2024-03-30T04:57:23.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.329+0000 7f5722000700 1 -- 172.21.15.123:0/819074430 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5718013420 con 0x7f572c1035d0 2024-03-30T04:57:23.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.329+0000 7f5722000700 1 -- 172.21.15.123:0/819074430 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f571801c630 con 0x7f572c1035d0 2024-03-30T04:57:23.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.329+0000 7f572b200700 1 -- 172.21.15.123:0/819074430 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f572c071120 con 0x7f572c1035d0 2024-03-30T04:57:23.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.329+0000 7f572b200700 1 -- 172.21.15.123:0/819074430 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f572c11a050 con 0x7f572c1035d0 2024-03-30T04:57:23.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.330+0000 7f572b200700 1 -- 172.21.15.123:0/819074430 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f572c0fe380 con 0x7f572c1035d0 2024-03-30T04:57:23.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.331+0000 7f5722000700 1 -- 172.21.15.123:0/819074430 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f57180139e0 con 0x7f572c1035d0 2024-03-30T04:57:23.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.331+0000 7f5722000700 1 --2- 172.21.15.123:0/819074430 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f571006cc20 0x7f571006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:57:23.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.332+0000 7f5722000700 1 -- 172.21.15.123:0/819074430 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f571808d360 con 0x7f572c1035d0 2024-03-30T04:57:23.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.332+0000 7f5729e00700 1 --2- 172.21.15.123:0/819074430 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f571006cc20 0x7f571006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:57:23.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.332+0000 7f5729e00700 1 --2- 172.21.15.123:0/819074430 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f571006cc20 0x7f571006f0d0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f571c009fd0 tx=0x7f571c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:57:23.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.336+0000 7f5722000700 1 -- 172.21.15.123:0/819074430 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5718057320 con 0x7f572c1035d0 2024-03-30T04:57:23.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.554+0000 7f572b200700 1 -- 172.21.15.123:0/819074430 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f572c04bed0 con 0x7f571006cc20 2024-03-30T04:57:23.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.556+0000 7f5722000700 1 -- 172.21.15.123:0/819074430 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f572c04bed0 con 0x7f571006cc20 2024-03-30T04:57:23.554 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:57:23.554 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:57:23.554 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:57:23.554 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T04:57:23.554 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:57:23.554 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T04:57:23.555 INFO:teuthology.orchestra.run.smithi123.stdout: "mds", 2024-03-30T04:57:23.555 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T04:57:23.555 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T04:57:23.555 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T04:57:23.555 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:57:23.555 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "16/21 daemons upgraded", 2024-03-30T04:57:23.555 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T04:57:23.555 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:57:23.556 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:57:23.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.558+0000 7f572b200700 1 -- 172.21.15.123:0/819074430 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f571006cc20 msgr2=0x7f571006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:23.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.558+0000 7f572b200700 1 --2- 172.21.15.123:0/819074430 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f571006cc20 0x7f571006f0d0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f571c009fd0 tx=0x7f571c00b040 comp rx=0 tx=0).stop 2024-03-30T04:57:23.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.558+0000 7f572b200700 1 -- 172.21.15.123:0/819074430 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f572c1035d0 msgr2=0x7f572c070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:23.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.558+0000 7f572b200700 1 --2- 172.21.15.123:0/819074430 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f572c1035d0 0x7f572c070800 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f5718009390 tx=0x7f5718009e30 comp rx=0 tx=0).stop 2024-03-30T04:57:23.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.558+0000 7f572b200700 1 -- 172.21.15.123:0/819074430 shutdown_connections 2024-03-30T04:57:23.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.558+0000 7f572b200700 1 --2- 172.21.15.123:0/819074430 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f571006cc20 0x7f571006f0d0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:23.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.558+0000 7f572b200700 1 --2- 172.21.15.123:0/819074430 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f572c102c30 0x7f572c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:23.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.558+0000 7f572b200700 1 --2- 172.21.15.123:0/819074430 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f572c1035d0 0x7f572c070800 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:23.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.558+0000 7f572b200700 1 -- 172.21.15.123:0/819074430 >> 172.21.15.123:0/819074430 conn(0x7f572c0f8760 msgr2=0x7f572c0f9850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:57:23.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.558+0000 7f572b200700 1 -- 172.21.15.123:0/819074430 shutdown_connections 2024-03-30T04:57:23.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.558+0000 7f572b200700 1 -- 172.21.15.123:0/819074430 wait complete. 2024-03-30T04:57:23.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.693+0000 7fc0d6600700 1 -- 172.21.15.123:0/4053498496 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0d01037a0 msgr2=0x7fc0d00fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:23.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.693+0000 7fc0d6600700 1 --2- 172.21.15.123:0/4053498496 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0d01037a0 0x7fc0d00fce10 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc004940 tx=0x7fc0bc009e30 comp rx=0 tx=0).stop 2024-03-30T04:57:23.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.693+0000 7fc0d6600700 1 -- 172.21.15.123:0/4053498496 shutdown_connections 2024-03-30T04:57:23.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.693+0000 7fc0d6600700 1 --2- 172.21.15.123:0/4053498496 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0d01037a0 0x7fc0d00fce10 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:23.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.693+0000 7fc0d6600700 1 --2- 172.21.15.123:0/4053498496 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc0d0102e00 0x7fc0d01031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:23.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.693+0000 7fc0d6600700 1 -- 172.21.15.123:0/4053498496 >> 172.21.15.123:0/4053498496 conn(0x7fc0d00f89b0 msgr2=0x7fc0d00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:57:23.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.693+0000 7fc0d6600700 1 -- 172.21.15.123:0/4053498496 shutdown_connections 2024-03-30T04:57:23.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.694+0000 7fc0d6600700 1 -- 172.21.15.123:0/4053498496 wait complete. 2024-03-30T04:57:23.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.695+0000 7fc0d6600700 1 Processor -- start 2024-03-30T04:57:23.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.695+0000 7fc0d6600700 1 -- start start 2024-03-30T04:57:23.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.695+0000 7fc0d6600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc0d0102e00 0x7fc0d0115cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:57:23.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.695+0000 7fc0d6600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0d01037a0 0x7fc0d01161f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:57:23.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.695+0000 7fc0d6600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d01168d0 con 0x7fc0d01037a0 2024-03-30T04:57:23.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.695+0000 7fc0d6600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d010fc10 con 0x7fc0d0102e00 2024-03-30T04:57:23.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.696+0000 7fc0cea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0d01037a0 0x7fc0d01161f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:57:23.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.696+0000 7fc0cea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0d01037a0 0x7fc0d01161f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36722/0 (socket says 172.21.15.123:36722) 2024-03-30T04:57:23.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.696+0000 7fc0cea00700 1 -- 172.21.15.123:0/657484272 learned_addr learned my addr 172.21.15.123:0/657484272 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:57:23.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.696+0000 7fc0cea00700 1 -- 172.21.15.123:0/657484272 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc0d0102e00 msgr2=0x7fc0d0115cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:23.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.696+0000 7fc0cea00700 1 --2- 172.21.15.123:0/657484272 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc0d0102e00 0x7fc0d0115cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:23.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.696+0000 7fc0cea00700 1 -- 172.21.15.123:0/657484272 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0bc005040 con 0x7fc0d01037a0 2024-03-30T04:57:23.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.696+0000 7fc0cea00700 1 --2- 172.21.15.123:0/657484272 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0d01037a0 0x7fc0d01161f0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc009390 tx=0x7fc0bc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:57:23.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.696+0000 7fc0c7400700 1 -- 172.21.15.123:0/657484272 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc0bc00e040 con 0x7fc0d01037a0 2024-03-30T04:57:23.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.696+0000 7fc0d6600700 1 -- 172.21.15.123:0/657484272 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0d010fe30 con 0x7fc0d01037a0 2024-03-30T04:57:23.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.697+0000 7fc0c7400700 1 -- 172.21.15.123:0/657484272 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc0bc013420 con 0x7fc0d01037a0 2024-03-30T04:57:23.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.697+0000 7fc0c7400700 1 -- 172.21.15.123:0/657484272 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc0bc01c630 con 0x7fc0d01037a0 2024-03-30T04:57:23.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.697+0000 7fc0d6600700 1 -- 172.21.15.123:0/657484272 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0d01103f0 con 0x7fc0d01037a0 2024-03-30T04:57:23.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.698+0000 7fc0d6600700 1 -- 172.21.15.123:0/657484272 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0d00fe550 con 0x7fc0d01037a0 2024-03-30T04:57:23.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.699+0000 7fc0c7400700 1 -- 172.21.15.123:0/657484272 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fc0bc013a00 con 0x7fc0d01037a0 2024-03-30T04:57:23.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.699+0000 7fc0c7400700 1 --2- 172.21.15.123:0/657484272 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc0b406cc30 0x7fc0b406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:57:23.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.700+0000 7fc0c7400700 1 -- 172.21.15.123:0/657484272 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc0bc08d390 con 0x7fc0d01037a0 2024-03-30T04:57:23.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.700+0000 7fc0cf400700 1 --2- 172.21.15.123:0/657484272 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc0b406cc30 0x7fc0b406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:57:23.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.700+0000 7fc0cf400700 1 --2- 172.21.15.123:0/657484272 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc0b406cc30 0x7fc0b406f0e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fc0c0006fd0 tx=0x7fc0c0006db0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:57:23.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.704+0000 7fc0c7400700 1 -- 172.21.15.123:0/657484272 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc0bc05ade0 con 0x7fc0d01037a0 2024-03-30T04:57:23.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.992+0000 7fc0d6600700 1 -- 172.21.15.123:0/657484272 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc0d004bed0 con 0x7fc0d01037a0 2024-03-30T04:57:23.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.993+0000 7fc0c7400700 1 -- 172.21.15.123:0/657484272 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7fc0bc05a970 con 0x7fc0d01037a0 2024-03-30T04:57:23.991 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T04:57:23.991 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-30T04:57:23.992 INFO:teuthology.orchestra.run.smithi123.stdout: host smithi135 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi135/config 2024-03-30T04:57:23.992 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-30T04:57:23.992 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T04:57:23.992 INFO:teuthology.orchestra.run.smithi123.stdout:Traceback (most recent call last): 2024-03-30T04:57:23.992 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-30T04:57:23.992 INFO:teuthology.orchestra.run.smithi123.stdout: "__main__", mod_spec) 2024-03-30T04:57:23.992 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-30T04:57:23.992 INFO:teuthology.orchestra.run.smithi123.stdout: exec(code, run_globals) 2024-03-30T04:57:23.992 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10889, in 2024-03-30T04:57:23.992 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10877, in main 2024-03-30T04:57:23.992 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-30T04:57:23.993 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-30T04:57:23.993 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-30T04:57:23.993 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-30T04:57:23.993 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-03-30T04:57:23.993 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-30T04:57:23.993 INFO:teuthology.orchestra.run.smithi123.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T04:57:23.993 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T04:57:23.993 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T04:57:23.993 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T04:57:23.993 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T04:57:23.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.995+0000 7fc0d6600700 1 -- 172.21.15.123:0/657484272 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc0b406cc30 msgr2=0x7fc0b406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:23.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.995+0000 7fc0d6600700 1 --2- 172.21.15.123:0/657484272 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc0b406cc30 0x7fc0b406f0e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fc0c0006fd0 tx=0x7fc0c0006db0 comp rx=0 tx=0).stop 2024-03-30T04:57:23.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.995+0000 7fc0d6600700 1 -- 172.21.15.123:0/657484272 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0d01037a0 msgr2=0x7fc0d01161f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:23.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.995+0000 7fc0d6600700 1 --2- 172.21.15.123:0/657484272 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0d01037a0 0x7fc0d01161f0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc009390 tx=0x7fc0bc009e30 comp rx=0 tx=0).stop 2024-03-30T04:57:23.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.995+0000 7fc0d6600700 1 -- 172.21.15.123:0/657484272 shutdown_connections 2024-03-30T04:57:23.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.995+0000 7fc0d6600700 1 --2- 172.21.15.123:0/657484272 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc0b406cc30 0x7fc0b406f0e0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:23.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.995+0000 7fc0d6600700 1 --2- 172.21.15.123:0/657484272 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc0d0102e00 0x7fc0d0115cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:23.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.995+0000 7fc0d6600700 1 --2- 172.21.15.123:0/657484272 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0d01037a0 0x7fc0d01161f0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:23.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.995+0000 7fc0d6600700 1 -- 172.21.15.123:0/657484272 >> 172.21.15.123:0/657484272 conn(0x7fc0d00f89b0 msgr2=0x7fc0d00f9910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:57:23.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.995+0000 7fc0d6600700 1 -- 172.21.15.123:0/657484272 shutdown_connections 2024-03-30T04:57:23.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:23.995+0000 7fc0d6600700 1 -- 172.21.15.123:0/657484272 wait complete. 2024-03-30T04:57:24.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:57:23 smithi135 ceph-mon[106011]: from='client.35448 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:57:24.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:57:23 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/548891741' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:57:24.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:57:23 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/398493620' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:57:24.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:57:23 smithi123 ceph-mon[139369]: from='client.35448 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:57:24.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:57:23 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/548891741' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:57:24.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:57:23 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/398493620' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:57:25.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:57:24 smithi135 ceph-mon[106011]: pgmap v787: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:25.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:57:24 smithi135 ceph-mon[106011]: from='client.35460 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:57:25.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:57:24 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/657484272' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:57:25.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:57:24 smithi123 ceph-mon[139369]: pgmap v787: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:25.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:57:24 smithi123 ceph-mon[139369]: from='client.35460 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:57:25.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:57:24 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/657484272' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:57:27.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:57:26 smithi135 ceph-mon[106011]: pgmap v788: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:27.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:57:26 smithi123 ceph-mon[139369]: pgmap v788: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:29.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:57:28 smithi135 ceph-mon[106011]: pgmap v789: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:29.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:57:28 smithi123 ceph-mon[139369]: pgmap v789: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:30.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:57:29 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:57:30.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:57:29 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:57:31.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:57:30 smithi135 ceph-mon[106011]: pgmap v790: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:31.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:57:30 smithi123 ceph-mon[139369]: pgmap v790: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:33.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:57:32 smithi135 ceph-mon[106011]: pgmap v791: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:33.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:57:32 smithi123 ceph-mon[139369]: pgmap v791: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:35.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:57:34 smithi135 ceph-mon[106011]: pgmap v792: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:35.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:57:34 smithi123 ceph-mon[139369]: pgmap v792: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:37.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:57:36 smithi135 ceph-mon[106011]: pgmap v793: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:37.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:57:36 smithi123 ceph-mon[139369]: pgmap v793: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:39.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:57:38 smithi135 ceph-mon[106011]: pgmap v794: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:39.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:57:38 smithi123 ceph-mon[139369]: pgmap v794: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:41.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:57:40 smithi135 ceph-mon[106011]: pgmap v795: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:41.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:57:40 smithi123 ceph-mon[139369]: pgmap v795: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:43.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:57:42 smithi135 ceph-mon[106011]: pgmap v796: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:43.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:57:42 smithi123 ceph-mon[139369]: pgmap v796: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:45.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:57:44 smithi135 ceph-mon[106011]: pgmap v797: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:45.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:57:44 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:57:45.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:57:44 smithi123 ceph-mon[139369]: pgmap v797: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:45.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:57:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:57:47.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:57:46 smithi135 ceph-mon[106011]: pgmap v798: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:47.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:57:46 smithi123 ceph-mon[139369]: pgmap v798: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:49.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:57:48 smithi135 ceph-mon[106011]: pgmap v799: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:49.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:57:48 smithi123 ceph-mon[139369]: pgmap v799: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:51.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:57:50 smithi135 ceph-mon[106011]: pgmap v800: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:51.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:57:50 smithi123 ceph-mon[139369]: pgmap v800: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:53.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:57:52 smithi135 ceph-mon[106011]: pgmap v801: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:53.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:57:52 smithi123 ceph-mon[139369]: pgmap v801: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:54.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.142+0000 7f7525000700 1 -- 172.21.15.123:0/261157452 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7520105000 msgr2=0x7f75201053d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:54.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.142+0000 7f7525000700 1 --2- 172.21.15.123:0/261157452 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7520105000 0x7f75201053d0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f7510004940 tx=0x7f7510009e30 comp rx=0 tx=0).stop 2024-03-30T04:57:54.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.142+0000 7f7525000700 1 -- 172.21.15.123:0/261157452 shutdown_connections 2024-03-30T04:57:54.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.142+0000 7f7525000700 1 --2- 172.21.15.123:0/261157452 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f75201059a0 0x7f75200ff010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:54.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.142+0000 7f7525000700 1 --2- 172.21.15.123:0/261157452 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7520105000 0x7f75201053d0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:54.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.142+0000 7f7525000700 1 -- 172.21.15.123:0/261157452 >> 172.21.15.123:0/261157452 conn(0x7f75200fabd0 msgr2=0x7f75200fcfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:57:54.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.142+0000 7f7525000700 1 -- 172.21.15.123:0/261157452 shutdown_connections 2024-03-30T04:57:54.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.142+0000 7f7525000700 1 -- 172.21.15.123:0/261157452 wait complete. 2024-03-30T04:57:54.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.143+0000 7f7525000700 1 Processor -- start 2024-03-30T04:57:54.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.143+0000 7f7525000700 1 -- start start 2024-03-30T04:57:54.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.144+0000 7f7525000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7520105000 0x7f7520115cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:57:54.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.144+0000 7f7525000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f75201059a0 0x7f7520116200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:57:54.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.144+0000 7f7525000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7520116850 con 0x7f7520105000 2024-03-30T04:57:54.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.144+0000 7f7525000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f752010fc20 con 0x7f75201059a0 2024-03-30T04:57:54.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.144+0000 7f751de00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7520105000 0x7f7520115cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:57:54.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.144+0000 7f751de00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7520105000 0x7f7520115cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36362/0 (socket says 172.21.15.123:36362) 2024-03-30T04:57:54.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.144+0000 7f751de00700 1 -- 172.21.15.123:0/1802221404 learned_addr learned my addr 172.21.15.123:0/1802221404 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:57:54.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.144+0000 7f751de00700 1 -- 172.21.15.123:0/1802221404 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f75201059a0 msgr2=0x7f7520116200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:54.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.144+0000 7f751d400700 1 --2- 172.21.15.123:0/1802221404 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f75201059a0 0x7f7520116200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:57:54.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.144+0000 7f751de00700 1 --2- 172.21.15.123:0/1802221404 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f75201059a0 0x7f7520116200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:54.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.144+0000 7f751de00700 1 -- 172.21.15.123:0/1802221404 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7510005040 con 0x7f7520105000 2024-03-30T04:57:54.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.144+0000 7f751d400700 1 --2- 172.21.15.123:0/1802221404 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f75201059a0 0x7f7520116200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:57:54.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.145+0000 7f751de00700 1 --2- 172.21.15.123:0/1802221404 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7520105000 0x7f7520115cc0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f7510009390 tx=0x7f7510009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:57:54.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.145+0000 7f7516000700 1 -- 172.21.15.123:0/1802221404 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f751000e040 con 0x7f7520105000 2024-03-30T04:57:54.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.145+0000 7f7516000700 1 -- 172.21.15.123:0/1802221404 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7510013420 con 0x7f7520105000 2024-03-30T04:57:54.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.145+0000 7f7516000700 1 -- 172.21.15.123:0/1802221404 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f751001c630 con 0x7f7520105000 2024-03-30T04:57:54.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.145+0000 7f7525000700 1 -- 172.21.15.123:0/1802221404 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f752010fea0 con 0x7f7520105000 2024-03-30T04:57:54.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.145+0000 7f7525000700 1 -- 172.21.15.123:0/1802221404 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7520110460 con 0x7f7520105000 2024-03-30T04:57:54.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.146+0000 7f7525000700 1 -- 172.21.15.123:0/1802221404 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7520100750 con 0x7f7520105000 2024-03-30T04:57:54.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.147+0000 7f7516000700 1 -- 172.21.15.123:0/1802221404 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f75100139e0 con 0x7f7520105000 2024-03-30T04:57:54.146 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.148+0000 7f7516000700 1 --2- 172.21.15.123:0/1802221404 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f750406cc30 0x7f750406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:57:54.146 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.148+0000 7f7516000700 1 -- 172.21.15.123:0/1802221404 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f751008d230 con 0x7f7520105000 2024-03-30T04:57:54.146 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.148+0000 7f751d400700 1 --2- 172.21.15.123:0/1802221404 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f750406cc30 0x7f750406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:57:54.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.149+0000 7f751d400700 1 --2- 172.21.15.123:0/1802221404 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f750406cc30 0x7f750406f0e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f750c009fd0 tx=0x7f750c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:57:54.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.153+0000 7f7516000700 1 -- 172.21.15.123:0/1802221404 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f751005ae70 con 0x7f7520105000 2024-03-30T04:57:54.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.377+0000 7f7525000700 1 -- 172.21.15.123:0/1802221404 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f752004bed0 con 0x7f750406cc30 2024-03-30T04:57:54.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.378+0000 7f7516000700 1 -- 172.21.15.123:0/1802221404 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f752004bed0 con 0x7f750406cc30 2024-03-30T04:57:54.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.381+0000 7f7525000700 1 -- 172.21.15.123:0/1802221404 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f750406cc30 msgr2=0x7f750406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:54.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.381+0000 7f7525000700 1 --2- 172.21.15.123:0/1802221404 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f750406cc30 0x7f750406f0e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f750c009fd0 tx=0x7f750c00b040 comp rx=0 tx=0).stop 2024-03-30T04:57:54.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.381+0000 7f7525000700 1 -- 172.21.15.123:0/1802221404 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7520105000 msgr2=0x7f7520115cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:54.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.381+0000 7f7525000700 1 --2- 172.21.15.123:0/1802221404 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7520105000 0x7f7520115cc0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f7510009390 tx=0x7f7510009e30 comp rx=0 tx=0).stop 2024-03-30T04:57:54.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.381+0000 7f7525000700 1 -- 172.21.15.123:0/1802221404 shutdown_connections 2024-03-30T04:57:54.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.381+0000 7f7525000700 1 --2- 172.21.15.123:0/1802221404 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f750406cc30 0x7f750406f0e0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:54.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.381+0000 7f7525000700 1 --2- 172.21.15.123:0/1802221404 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7520105000 0x7f7520115cc0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:54.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.381+0000 7f7525000700 1 --2- 172.21.15.123:0/1802221404 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f75201059a0 0x7f7520116200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:54.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.381+0000 7f7525000700 1 -- 172.21.15.123:0/1802221404 >> 172.21.15.123:0/1802221404 conn(0x7f75200fabd0 msgr2=0x7f75200fbd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:57:54.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.381+0000 7f7525000700 1 -- 172.21.15.123:0/1802221404 shutdown_connections 2024-03-30T04:57:54.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.381+0000 7f7525000700 1 -- 172.21.15.123:0/1802221404 wait complete. 2024-03-30T04:57:54.394 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:57:54.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.517+0000 7f10a1400700 1 -- 172.21.15.123:0/1373414472 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f109c0fe830 msgr2=0x7f109c106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:54.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.517+0000 7f10a1400700 1 --2- 172.21.15.123:0/1373414472 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f109c0fe830 0x7f109c106f00 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f1084004940 tx=0x7f1084009e30 comp rx=0 tx=0).stop 2024-03-30T04:57:54.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.517+0000 7f10a1400700 1 -- 172.21.15.123:0/1373414472 shutdown_connections 2024-03-30T04:57:54.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.517+0000 7f10a1400700 1 --2- 172.21.15.123:0/1373414472 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f109c0fe830 0x7f109c106f00 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:54.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.517+0000 7f10a1400700 1 --2- 172.21.15.123:0/1373414472 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f109c0fde90 0x7f109c0fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:54.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.517+0000 7f10a1400700 1 -- 172.21.15.123:0/1373414472 >> 172.21.15.123:0/1373414472 conn(0x7f109c072bc0 msgr2=0x7f109c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:57:54.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.517+0000 7f10a1400700 1 -- 172.21.15.123:0/1373414472 shutdown_connections 2024-03-30T04:57:54.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.517+0000 7f10a1400700 1 -- 172.21.15.123:0/1373414472 wait complete. 2024-03-30T04:57:54.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.518+0000 7f10a1400700 1 Processor -- start 2024-03-30T04:57:54.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.518+0000 7f10a1400700 1 -- start start 2024-03-30T04:57:54.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.518+0000 7f10a1400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f109c0fde90 0x7f109c115d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:57:54.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.518+0000 7f10a1400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f109c0fe830 0x7f109c116250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:57:54.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.518+0000 7f10a1400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f109c1168a0 con 0x7f109c0fde90 2024-03-30T04:57:54.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.518+0000 7f10a1400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f109c10fc70 con 0x7f109c0fe830 2024-03-30T04:57:54.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.518+0000 7f109a800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f109c0fde90 0x7f109c115d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:57:54.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.518+0000 7f109a800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f109c0fde90 0x7f109c115d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36380/0 (socket says 172.21.15.123:36380) 2024-03-30T04:57:54.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.518+0000 7f109a800700 1 -- 172.21.15.123:0/95363169 learned_addr learned my addr 172.21.15.123:0/95363169 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:57:54.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.519+0000 7f1093e00700 1 --2- 172.21.15.123:0/95363169 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f109c0fe830 0x7f109c116250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:57:54.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.519+0000 7f109a800700 1 -- 172.21.15.123:0/95363169 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f109c0fe830 msgr2=0x7f109c116250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:54.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.519+0000 7f109a800700 1 --2- 172.21.15.123:0/95363169 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f109c0fe830 0x7f109c116250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:54.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.519+0000 7f109a800700 1 -- 172.21.15.123:0/95363169 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1084005040 con 0x7f109c0fde90 2024-03-30T04:57:54.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.519+0000 7f109a800700 1 --2- 172.21.15.123:0/95363169 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f109c0fde90 0x7f109c115d10 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f108c002a50 tx=0x7f108c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:57:54.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.519+0000 7f1092a00700 1 -- 172.21.15.123:0/95363169 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f108c007cc0 con 0x7f109c0fde90 2024-03-30T04:57:54.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.519+0000 7f1092a00700 1 -- 172.21.15.123:0/95363169 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f108c00f040 con 0x7f109c0fde90 2024-03-30T04:57:54.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.519+0000 7f10a1400700 1 -- 172.21.15.123:0/95363169 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f109c10ff50 con 0x7f109c0fde90 2024-03-30T04:57:54.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.519+0000 7f1092a00700 1 -- 172.21.15.123:0/95363169 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f108c00d9c0 con 0x7f109c0fde90 2024-03-30T04:57:54.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.519+0000 7f10a1400700 1 -- 172.21.15.123:0/95363169 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f109c110490 con 0x7f109c0fde90 2024-03-30T04:57:54.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.520+0000 7f10a1400700 1 -- 172.21.15.123:0/95363169 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f109c106a20 con 0x7f109c0fde90 2024-03-30T04:57:54.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.521+0000 7f1092a00700 1 -- 172.21.15.123:0/95363169 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f108c00db20 con 0x7f109c0fde90 2024-03-30T04:57:54.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.522+0000 7f1092a00700 1 --2- 172.21.15.123:0/95363169 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f108006cc30 0x7f108006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:57:54.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.522+0000 7f1092a00700 1 -- 172.21.15.123:0/95363169 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f108c08bd20 con 0x7f109c0fde90 2024-03-30T04:57:54.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.523+0000 7f1093e00700 1 --2- 172.21.15.123:0/95363169 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f108006cc30 0x7f108006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:57:54.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.523+0000 7f1093e00700 1 --2- 172.21.15.123:0/95363169 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f108006cc30 0x7f108006f0e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f109c1115b0 tx=0x7f1084009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:57:54.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.527+0000 7f1092a00700 1 -- 172.21.15.123:0/95363169 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f108c059770 con 0x7f109c0fde90 2024-03-30T04:57:54.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.749+0000 7f10a1400700 1 -- 172.21.15.123:0/95363169 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f109c109e40 con 0x7f108006cc30 2024-03-30T04:57:54.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.750+0000 7f1092a00700 1 -- 172.21.15.123:0/95363169 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f109c109e40 con 0x7f108006cc30 2024-03-30T04:57:54.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.752+0000 7f10a1400700 1 -- 172.21.15.123:0/95363169 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f108006cc30 msgr2=0x7f108006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:54.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.752+0000 7f10a1400700 1 --2- 172.21.15.123:0/95363169 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f108006cc30 0x7f108006f0e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f109c1115b0 tx=0x7f1084009e30 comp rx=0 tx=0).stop 2024-03-30T04:57:54.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.752+0000 7f10a1400700 1 -- 172.21.15.123:0/95363169 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f109c0fde90 msgr2=0x7f109c115d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:54.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.752+0000 7f10a1400700 1 --2- 172.21.15.123:0/95363169 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f109c0fde90 0x7f109c115d10 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f108c002a50 tx=0x7f108c0046b0 comp rx=0 tx=0).stop 2024-03-30T04:57:54.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.752+0000 7f10a1400700 1 -- 172.21.15.123:0/95363169 shutdown_connections 2024-03-30T04:57:54.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.752+0000 7f10a1400700 1 --2- 172.21.15.123:0/95363169 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f108006cc30 0x7f108006f0e0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:54.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.752+0000 7f10a1400700 1 --2- 172.21.15.123:0/95363169 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f109c0fde90 0x7f109c115d10 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:54.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.752+0000 7f10a1400700 1 --2- 172.21.15.123:0/95363169 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f109c0fe830 0x7f109c116250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:54.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.752+0000 7f10a1400700 1 -- 172.21.15.123:0/95363169 >> 172.21.15.123:0/95363169 conn(0x7f109c072bc0 msgr2=0x7f109c0fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:57:54.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.752+0000 7f10a1400700 1 -- 172.21.15.123:0/95363169 shutdown_connections 2024-03-30T04:57:54.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.753+0000 7f10a1400700 1 -- 172.21.15.123:0/95363169 wait complete. 2024-03-30T04:57:54.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.887+0000 7f611c600700 1 -- 172.21.15.123:0/653852392 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61141037c0 msgr2=0x7f61140fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:54.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.887+0000 7f611c600700 1 --2- 172.21.15.123:0/653852392 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61141037c0 0x7f61140fce30 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f6100004940 tx=0x7f6100009e30 comp rx=0 tx=0).stop 2024-03-30T04:57:54.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.887+0000 7f611c600700 1 -- 172.21.15.123:0/653852392 shutdown_connections 2024-03-30T04:57:54.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.887+0000 7f611c600700 1 --2- 172.21.15.123:0/653852392 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61141037c0 0x7f61140fce30 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:54.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.887+0000 7f611c600700 1 --2- 172.21.15.123:0/653852392 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6114102e20 0x7f61141031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:54.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.887+0000 7f611c600700 1 -- 172.21.15.123:0/653852392 >> 172.21.15.123:0/653852392 conn(0x7f61140f89d0 msgr2=0x7f61140fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:57:54.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.887+0000 7f611c600700 1 -- 172.21.15.123:0/653852392 shutdown_connections 2024-03-30T04:57:54.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.887+0000 7f611c600700 1 -- 172.21.15.123:0/653852392 wait complete. 2024-03-30T04:57:54.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.888+0000 7f611c600700 1 Processor -- start 2024-03-30T04:57:54.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.888+0000 7f611c600700 1 -- start start 2024-03-30T04:57:54.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.888+0000 7f611c600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6114102e20 0x7f61140702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:57:54.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.888+0000 7f611c600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61141037c0 0x7f6114070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:57:54.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.888+0000 7f611c600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6114070d90 con 0x7f61141037c0 2024-03-30T04:57:54.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.888+0000 7f611c600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6114070ed0 con 0x7f6114102e20 2024-03-30T04:57:54.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.889+0000 7f6119200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61141037c0 0x7f6114070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:57:54.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.889+0000 7f6119200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61141037c0 0x7f6114070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36394/0 (socket says 172.21.15.123:36394) 2024-03-30T04:57:54.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.889+0000 7f6119200700 1 -- 172.21.15.123:0/1651629559 learned_addr learned my addr 172.21.15.123:0/1651629559 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:57:54.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.889+0000 7f6119c00700 1 --2- 172.21.15.123:0/1651629559 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6114102e20 0x7f61140702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:57:54.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.889+0000 7f6119200700 1 -- 172.21.15.123:0/1651629559 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6114102e20 msgr2=0x7f61140702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:54.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.889+0000 7f6119200700 1 --2- 172.21.15.123:0/1651629559 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6114102e20 0x7f61140702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:54.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.889+0000 7f6119200700 1 -- 172.21.15.123:0/1651629559 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6100005040 con 0x7f61141037c0 2024-03-30T04:57:54.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.889+0000 7f6119c00700 1 --2- 172.21.15.123:0/1651629559 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6114102e20 0x7f61140702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:57:54.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.889+0000 7f6119200700 1 --2- 172.21.15.123:0/1651629559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61141037c0 0x7f6114070800 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f6100009390 tx=0x7f6100009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:57:54.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.889+0000 7f6111600700 1 -- 172.21.15.123:0/1651629559 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f610000e040 con 0x7f61141037c0 2024-03-30T04:57:54.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.889+0000 7f6111600700 1 -- 172.21.15.123:0/1651629559 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6100013420 con 0x7f61141037c0 2024-03-30T04:57:54.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.889+0000 7f6111600700 1 -- 172.21.15.123:0/1651629559 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f610001c630 con 0x7f61141037c0 2024-03-30T04:57:54.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.890+0000 7f611c600700 1 -- 172.21.15.123:0/1651629559 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6114071120 con 0x7f61141037c0 2024-03-30T04:57:54.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.890+0000 7f611c600700 1 -- 172.21.15.123:0/1651629559 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f611411a200 con 0x7f61141037c0 2024-03-30T04:57:54.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.891+0000 7f611c600700 1 -- 172.21.15.123:0/1651629559 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61140fe570 con 0x7f61141037c0 2024-03-30T04:57:54.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.892+0000 7f6111600700 1 -- 172.21.15.123:0/1651629559 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f6100013a00 con 0x7f61141037c0 2024-03-30T04:57:54.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.893+0000 7f6111600700 1 --2- 172.21.15.123:0/1651629559 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f60fc06cc30 0x7f60fc06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:57:54.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.893+0000 7f6111600700 1 -- 172.21.15.123:0/1651629559 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f610008d370 con 0x7f61141037c0 2024-03-30T04:57:54.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.893+0000 7f6119c00700 1 --2- 172.21.15.123:0/1651629559 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f60fc06cc30 0x7f60fc06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:57:54.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.893+0000 7f6119c00700 1 --2- 172.21.15.123:0/1651629559 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f60fc06cc30 0x7f60fc06f0e0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f6108009fd0 tx=0x7f6108009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:57:54.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:54.897+0000 7f6111600700 1 -- 172.21.15.123:0/1651629559 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6100057330 con 0x7f61141037c0 2024-03-30T04:57:55.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:57:54 smithi135 ceph-mon[106011]: pgmap v802: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:55.102 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.104+0000 7f611c600700 1 -- 172.21.15.123:0/1651629559 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f6114109e30 con 0x7f60fc06cc30 2024-03-30T04:57:55.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.114+0000 7f6111600700 1 -- 172.21.15.123:0/1651629559 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f6114109e30 con 0x7f60fc06cc30 2024-03-30T04:57:55.112 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:57:55.112 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (27m) 4m ago 39m 14.6M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:57:55.112 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (24m) 4m ago 39m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T04:57:55.112 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (24m) 4m ago 38m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T04:57:55.112 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (26m) 4m ago 39m 73.5M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T04:57:55.112 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (20m) 4m ago 33m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T04:57:55.112 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (20m) 4m ago 33m 16.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T04:57:55.112 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (4m) 4m ago 33m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 87b073a6c9ca 2024-03-30T04:57:55.112 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (19m) 4m ago 33m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T04:57:55.113 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (29m) 4m ago 41m 510M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:57:55.113 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (28m) 4m ago 38m 456M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:57:55.113 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (25m) 4m ago 41m 70.0M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T04:57:55.113 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (25m) 4m ago 38m 51.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T04:57:55.113 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (27m) 4m ago 39m 8636k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:57:55.113 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (27m) 4m ago 38m 8678k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:57:55.113 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (23m) 4m ago 37m 107M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T04:57:55.113 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (23m) 4m ago 37m 97.2M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T04:57:55.113 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (22m) 4m ago 36m 89.8M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T04:57:55.113 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (22m) 4m ago 35m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T04:57:55.113 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (21m) 4m ago 35m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T04:57:55.113 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (20m) 4m ago 34m 88.8M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T04:57:55.113 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (27m) 4m ago 39m 63.9M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:57:55.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.117+0000 7f611c600700 1 -- 172.21.15.123:0/1651629559 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f60fc06cc30 msgr2=0x7f60fc06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:55.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.117+0000 7f611c600700 1 --2- 172.21.15.123:0/1651629559 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f60fc06cc30 0x7f60fc06f0e0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f6108009fd0 tx=0x7f6108009ee0 comp rx=0 tx=0).stop 2024-03-30T04:57:55.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.117+0000 7f611c600700 1 -- 172.21.15.123:0/1651629559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61141037c0 msgr2=0x7f6114070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:55.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.117+0000 7f611c600700 1 --2- 172.21.15.123:0/1651629559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61141037c0 0x7f6114070800 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f6100009390 tx=0x7f6100009e30 comp rx=0 tx=0).stop 2024-03-30T04:57:55.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.117+0000 7f611c600700 1 -- 172.21.15.123:0/1651629559 shutdown_connections 2024-03-30T04:57:55.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.117+0000 7f611c600700 1 --2- 172.21.15.123:0/1651629559 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f60fc06cc30 0x7f60fc06f0e0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:55.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.117+0000 7f611c600700 1 --2- 172.21.15.123:0/1651629559 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6114102e20 0x7f61140702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:55.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.117+0000 7f611c600700 1 --2- 172.21.15.123:0/1651629559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61141037c0 0x7f6114070800 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:55.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.117+0000 7f611c600700 1 -- 172.21.15.123:0/1651629559 >> 172.21.15.123:0/1651629559 conn(0x7f61140f89d0 msgr2=0x7f61140f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:57:55.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.117+0000 7f611c600700 1 -- 172.21.15.123:0/1651629559 shutdown_connections 2024-03-30T04:57:55.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.117+0000 7f611c600700 1 -- 172.21.15.123:0/1651629559 wait complete. 2024-03-30T04:57:55.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:57:54 smithi123 ceph-mon[139369]: pgmap v802: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:55.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.249+0000 7f3d68e00700 1 -- 172.21.15.123:0/3185315994 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d640708a0 msgr2=0x7f3d64070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:55.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.249+0000 7f3d68e00700 1 --2- 172.21.15.123:0/3185315994 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d640708a0 0x7f3d64070d10 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f3d50004970 tx=0x7f3d50009e30 comp rx=0 tx=0).stop 2024-03-30T04:57:55.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.249+0000 7f3d68e00700 1 -- 172.21.15.123:0/3185315994 shutdown_connections 2024-03-30T04:57:55.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.249+0000 7f3d68e00700 1 --2- 172.21.15.123:0/3185315994 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d640708a0 0x7f3d64070d10 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:55.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.249+0000 7f3d68e00700 1 --2- 172.21.15.123:0/3185315994 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3d64071f90 0x7f3d64070360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:55.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.249+0000 7f3d68e00700 1 -- 172.21.15.123:0/3185315994 >> 172.21.15.123:0/3185315994 conn(0x7f3d640f89d0 msgr2=0x7f3d640fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:57:55.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.250+0000 7f3d68e00700 1 -- 172.21.15.123:0/3185315994 shutdown_connections 2024-03-30T04:57:55.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.250+0000 7f3d68e00700 1 -- 172.21.15.123:0/3185315994 wait complete. 2024-03-30T04:57:55.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.250+0000 7f3d68e00700 1 Processor -- start 2024-03-30T04:57:55.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.251+0000 7f3d68e00700 1 -- start start 2024-03-30T04:57:55.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.251+0000 7f3d68e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3d640708a0 0x7f3d6411a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:57:55.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.251+0000 7f3d68e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d64071f90 0x7f3d6411a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:57:55.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.251+0000 7f3d68e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6411ac30 con 0x7f3d64071f90 2024-03-30T04:57:55.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.251+0000 7f3d68e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d64114000 con 0x7f3d640708a0 2024-03-30T04:57:55.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.251+0000 7f3d61400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d64071f90 0x7f3d6411a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:57:55.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.251+0000 7f3d61400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d64071f90 0x7f3d6411a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36398/0 (socket says 172.21.15.123:36398) 2024-03-30T04:57:55.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.251+0000 7f3d61400700 1 -- 172.21.15.123:0/3543617045 learned_addr learned my addr 172.21.15.123:0/3543617045 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:57:55.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.251+0000 7f3d61e00700 1 --2- 172.21.15.123:0/3543617045 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3d640708a0 0x7f3d6411a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:57:55.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.252+0000 7f3d61400700 1 -- 172.21.15.123:0/3543617045 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3d640708a0 msgr2=0x7f3d6411a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:55.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.252+0000 7f3d61400700 1 --2- 172.21.15.123:0/3543617045 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3d640708a0 0x7f3d6411a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:55.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.252+0000 7f3d61400700 1 -- 172.21.15.123:0/3543617045 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d50005040 con 0x7f3d64071f90 2024-03-30T04:57:55.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.252+0000 7f3d61400700 1 --2- 172.21.15.123:0/3543617045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d64071f90 0x7f3d6411a5e0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f3d50005010 tx=0x7f3d50009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:57:55.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.252+0000 7f3d5a000700 1 -- 172.21.15.123:0/3543617045 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d5000e040 con 0x7f3d64071f90 2024-03-30T04:57:55.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.252+0000 7f3d5a000700 1 -- 172.21.15.123:0/3543617045 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3d500098b0 con 0x7f3d64071f90 2024-03-30T04:57:55.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.252+0000 7f3d68e00700 1 -- 172.21.15.123:0/3543617045 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d64114280 con 0x7f3d64071f90 2024-03-30T04:57:55.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.252+0000 7f3d5a000700 1 -- 172.21.15.123:0/3543617045 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d50013e80 con 0x7f3d64071f90 2024-03-30T04:57:55.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.252+0000 7f3d68e00700 1 -- 172.21.15.123:0/3543617045 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d641147c0 con 0x7f3d64071f90 2024-03-30T04:57:55.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.253+0000 7f3d68e00700 1 -- 172.21.15.123:0/3543617045 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d64071aa0 con 0x7f3d64071f90 2024-03-30T04:57:55.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.254+0000 7f3d5a000700 1 -- 172.21.15.123:0/3543617045 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f3d500136b0 con 0x7f3d64071f90 2024-03-30T04:57:55.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.255+0000 7f3d5a000700 1 --2- 172.21.15.123:0/3543617045 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3d4806cbc0 0x7f3d4806f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:57:55.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.255+0000 7f3d5a000700 1 -- 172.21.15.123:0/3543617045 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f3d5008d350 con 0x7f3d64071f90 2024-03-30T04:57:55.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.256+0000 7f3d61e00700 1 --2- 172.21.15.123:0/3543617045 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3d4806cbc0 0x7f3d4806f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:57:55.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.256+0000 7f3d61e00700 1 --2- 172.21.15.123:0/3543617045 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3d4806cbc0 0x7f3d4806f070 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f3d54009b80 tx=0x7f3d5400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:57:55.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.260+0000 7f3d5a000700 1 -- 172.21.15.123:0/3543617045 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3d5005ada0 con 0x7f3d64071f90 2024-03-30T04:57:55.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.544+0000 7f3d68e00700 1 -- 172.21.15.123:0/3543617045 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f3d6405f5f0 con 0x7f3d64071f90 2024-03-30T04:57:55.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.544+0000 7f3d5a000700 1 -- 172.21.15.123:0/3543617045 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f3d5005a930 con 0x7f3d64071f90 2024-03-30T04:57:55.542 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:57:55.542 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:57:55.542 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:57:55.542 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:57:55.542 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:57:55.542 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:57:55.542 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:57:55.543 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:57:55.543 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T04:57:55.543 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:57:55.543 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:57:55.543 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 4 2024-03-30T04:57:55.543 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:57:55.543 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:57:55.543 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 14 2024-03-30T04:57:55.543 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:57:55.543 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:57:55.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.547+0000 7f3d68e00700 1 -- 172.21.15.123:0/3543617045 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3d4806cbc0 msgr2=0x7f3d4806f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:55.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.547+0000 7f3d68e00700 1 --2- 172.21.15.123:0/3543617045 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3d4806cbc0 0x7f3d4806f070 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f3d54009b80 tx=0x7f3d5400b040 comp rx=0 tx=0).stop 2024-03-30T04:57:55.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.547+0000 7f3d68e00700 1 -- 172.21.15.123:0/3543617045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d64071f90 msgr2=0x7f3d6411a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:55.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.547+0000 7f3d68e00700 1 --2- 172.21.15.123:0/3543617045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d64071f90 0x7f3d6411a5e0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f3d50005010 tx=0x7f3d50009e30 comp rx=0 tx=0).stop 2024-03-30T04:57:55.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.547+0000 7f3d68e00700 1 -- 172.21.15.123:0/3543617045 shutdown_connections 2024-03-30T04:57:55.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.547+0000 7f3d68e00700 1 --2- 172.21.15.123:0/3543617045 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3d4806cbc0 0x7f3d4806f070 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:55.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.547+0000 7f3d68e00700 1 --2- 172.21.15.123:0/3543617045 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3d640708a0 0x7f3d6411a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:55.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.547+0000 7f3d68e00700 1 --2- 172.21.15.123:0/3543617045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d64071f90 0x7f3d6411a5e0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:55.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.547+0000 7f3d68e00700 1 -- 172.21.15.123:0/3543617045 >> 172.21.15.123:0/3543617045 conn(0x7f3d640f89d0 msgr2=0x7f3d640f9c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:57:55.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.547+0000 7f3d68e00700 1 -- 172.21.15.123:0/3543617045 shutdown_connections 2024-03-30T04:57:55.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.547+0000 7f3d68e00700 1 -- 172.21.15.123:0/3543617045 wait complete. 2024-03-30T04:57:55.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.681+0000 7f30a1600700 1 -- 172.21.15.123:0/2156311460 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f309c070c70 msgr2=0x7f309c108f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:55.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.681+0000 7f30a1600700 1 --2- 172.21.15.123:0/2156311460 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f309c070c70 0x7f309c108f20 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f3088004970 tx=0x7f3088009e30 comp rx=0 tx=0).stop 2024-03-30T04:57:55.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.681+0000 7f30a1600700 1 -- 172.21.15.123:0/2156311460 shutdown_connections 2024-03-30T04:57:55.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.681+0000 7f30a1600700 1 --2- 172.21.15.123:0/2156311460 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f309c070c70 0x7f309c108f20 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:55.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.681+0000 7f30a1600700 1 --2- 172.21.15.123:0/2156311460 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f309c070360 0x7f309c070730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:55.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.681+0000 7f30a1600700 1 -- 172.21.15.123:0/2156311460 >> 172.21.15.123:0/2156311460 conn(0x7f309c0f8780 msgr2=0x7f309c0fab90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:57:55.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.681+0000 7f30a1600700 1 -- 172.21.15.123:0/2156311460 shutdown_connections 2024-03-30T04:57:55.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.681+0000 7f30a1600700 1 -- 172.21.15.123:0/2156311460 wait complete. 2024-03-30T04:57:55.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.682+0000 7f30a1600700 1 Processor -- start 2024-03-30T04:57:55.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.682+0000 7f30a1600700 1 -- start start 2024-03-30T04:57:55.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.682+0000 7f30a1600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f309c070360 0x7f309c1159f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:57:55.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.682+0000 7f30a1600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f309c070c70 0x7f309c115f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:57:55.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.682+0000 7f30a1600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f309c116580 con 0x7f309c070360 2024-03-30T04:57:55.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.682+0000 7f30a1600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f309c10f970 con 0x7f309c070c70 2024-03-30T04:57:55.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.683+0000 7f309be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f309c070360 0x7f309c1159f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:57:55.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.683+0000 7f309be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f309c070360 0x7f309c1159f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36422/0 (socket says 172.21.15.123:36422) 2024-03-30T04:57:55.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.683+0000 7f309be00700 1 -- 172.21.15.123:0/4241133885 learned_addr learned my addr 172.21.15.123:0/4241133885 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:57:55.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.683+0000 7f309be00700 1 -- 172.21.15.123:0/4241133885 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f309c070c70 msgr2=0x7f309c115f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:55.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.683+0000 7f309b400700 1 --2- 172.21.15.123:0/4241133885 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f309c070c70 0x7f309c115f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:57:55.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.683+0000 7f309be00700 1 --2- 172.21.15.123:0/4241133885 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f309c070c70 0x7f309c115f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:55.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.683+0000 7f309be00700 1 -- 172.21.15.123:0/4241133885 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3088005040 con 0x7f309c070360 2024-03-30T04:57:55.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.683+0000 7f309b400700 1 --2- 172.21.15.123:0/4241133885 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f309c070c70 0x7f309c115f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:57:55.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.683+0000 7f309be00700 1 --2- 172.21.15.123:0/4241133885 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f309c070360 0x7f309c1159f0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f308c00c8c0 tx=0x7f308c0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:57:55.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.684+0000 7f3093e00700 1 -- 172.21.15.123:0/4241133885 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f308c00a980 con 0x7f309c070360 2024-03-30T04:57:55.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.684+0000 7f30a1600700 1 -- 172.21.15.123:0/4241133885 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f309c10fc50 con 0x7f309c070360 2024-03-30T04:57:55.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.684+0000 7f3093e00700 1 -- 172.21.15.123:0/4241133885 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f308c00ce60 con 0x7f309c070360 2024-03-30T04:57:55.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.684+0000 7f3093e00700 1 -- 172.21.15.123:0/4241133885 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f308c0113a0 con 0x7f309c070360 2024-03-30T04:57:55.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.684+0000 7f30a1600700 1 -- 172.21.15.123:0/4241133885 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f309c110190 con 0x7f309c070360 2024-03-30T04:57:55.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.685+0000 7f30a1600700 1 -- 172.21.15.123:0/4241133885 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f309c071aa0 con 0x7f309c070360 2024-03-30T04:57:55.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.686+0000 7f3093e00700 1 -- 172.21.15.123:0/4241133885 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f308c012400 con 0x7f309c070360 2024-03-30T04:57:55.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.687+0000 7f3093e00700 1 --2- 172.21.15.123:0/4241133885 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f308006cc20 0x7f308006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:57:55.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.687+0000 7f3093e00700 1 -- 172.21.15.123:0/4241133885 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f308c08c1d0 con 0x7f309c070360 2024-03-30T04:57:55.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.687+0000 7f309b400700 1 --2- 172.21.15.123:0/4241133885 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f308006cc20 0x7f308006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:57:55.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.687+0000 7f309b400700 1 --2- 172.21.15.123:0/4241133885 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f308006cc20 0x7f308006f0d0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f3088009fd0 tx=0x7f3088009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:57:55.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.691+0000 7f3093e00700 1 -- 172.21.15.123:0/4241133885 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f308c059c20 con 0x7f309c070360 2024-03-30T04:57:55.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.939+0000 7f30a1600700 1 -- 172.21.15.123:0/4241133885 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f309c10bdc0 con 0x7f309c070360 2024-03-30T04:57:55.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.940+0000 7f3093e00700 1 -- 172.21.15.123:0/4241133885 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 34 v34) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f308c0597b0 con 0x7f309c070360 2024-03-30T04:57:55.937 INFO:teuthology.orchestra.run.smithi123.stdout:e34 2024-03-30T04:57:55.937 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:57:55.937 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:57:55.938 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:57:55.938 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:57:55.938 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:57:55.938 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:57:55.938 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T04:57:55.938 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T04:57:55.938 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:57:55.938 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T04:57:55.938 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:57:55.938 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:57:55.938 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:57:55.938 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:57:55.938 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:57:55.939 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T04:57:55.939 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:57:55.939 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:57:55.939 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T04:57:55.939 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:57:55.939 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T04:57:55.939 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T04:57:55.939 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T04:57:55.939 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T04:57:55.939 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:57:55.939 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:57:55.939 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:57:55.940 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:57:55.940 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:57:55.940 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:57:55.940 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T04:57:55.940 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T04:57:55.940 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:57:55.940 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:57:55.940 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T04:57:55.940 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:57:55.940 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:57:55.940 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:57:55.940 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:57:55.941 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44975} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/3788988014,v1:172.21.15.135:6827/3788988014] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:57:55.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.942+0000 7f30a1600700 1 -- 172.21.15.123:0/4241133885 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f308006cc20 msgr2=0x7f308006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:55.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.942+0000 7f30a1600700 1 --2- 172.21.15.123:0/4241133885 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f308006cc20 0x7f308006f0d0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f3088009fd0 tx=0x7f3088009e30 comp rx=0 tx=0).stop 2024-03-30T04:57:55.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.942+0000 7f30a1600700 1 -- 172.21.15.123:0/4241133885 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f309c070360 msgr2=0x7f309c1159f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:55.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.942+0000 7f30a1600700 1 --2- 172.21.15.123:0/4241133885 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f309c070360 0x7f309c1159f0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f308c00c8c0 tx=0x7f308c0044c0 comp rx=0 tx=0).stop 2024-03-30T04:57:55.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.942+0000 7f30a1600700 1 -- 172.21.15.123:0/4241133885 shutdown_connections 2024-03-30T04:57:55.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.942+0000 7f30a1600700 1 --2- 172.21.15.123:0/4241133885 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f308006cc20 0x7f308006f0d0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:55.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.942+0000 7f30a1600700 1 --2- 172.21.15.123:0/4241133885 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f309c070360 0x7f309c1159f0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:55.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.942+0000 7f30a1600700 1 --2- 172.21.15.123:0/4241133885 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f309c070c70 0x7f309c115f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:55.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.942+0000 7f30a1600700 1 -- 172.21.15.123:0/4241133885 >> 172.21.15.123:0/4241133885 conn(0x7f309c0f8780 msgr2=0x7f309c0f9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:57:55.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.942+0000 7f30a1600700 1 -- 172.21.15.123:0/4241133885 shutdown_connections 2024-03-30T04:57:55.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:55.943+0000 7f30a1600700 1 -- 172.21.15.123:0/4241133885 wait complete. 2024-03-30T04:57:55.942 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 34 2024-03-30T04:57:56.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.075+0000 7f7d0f200700 1 -- 172.21.15.123:0/1702607061 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d101035e0 msgr2=0x7f7d100fcc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:56.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.075+0000 7f7d0f200700 1 --2- 172.21.15.123:0/1702607061 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d101035e0 0x7f7d100fcc50 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc004940 tx=0x7f7cfc009e30 comp rx=0 tx=0).stop 2024-03-30T04:57:56.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.075+0000 7f7d0f200700 1 -- 172.21.15.123:0/1702607061 shutdown_connections 2024-03-30T04:57:56.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.075+0000 7f7d0f200700 1 --2- 172.21.15.123:0/1702607061 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d101035e0 0x7f7d100fcc50 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:56.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.075+0000 7f7d0f200700 1 --2- 172.21.15.123:0/1702607061 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d10102c40 0x7f7d10103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:56.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.075+0000 7f7d0f200700 1 -- 172.21.15.123:0/1702607061 >> 172.21.15.123:0/1702607061 conn(0x7f7d100f8830 msgr2=0x7f7d100fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:57:56.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.075+0000 7f7d0f200700 1 -- 172.21.15.123:0/1702607061 shutdown_connections 2024-03-30T04:57:56.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.075+0000 7f7d0f200700 1 -- 172.21.15.123:0/1702607061 wait complete. 2024-03-30T04:57:56.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.076+0000 7f7d0f200700 1 Processor -- start 2024-03-30T04:57:56.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.076+0000 7f7d0f200700 1 -- start start 2024-03-30T04:57:56.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.076+0000 7f7d0f200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d10102c40 0x7f7d10115af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:57:56.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.076+0000 7f7d0f200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d101035e0 0x7f7d10116030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:57:56.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.076+0000 7f7d0f200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d10116680 con 0x7f7d10102c40 2024-03-30T04:57:56.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.076+0000 7f7d0f200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d1010fa70 con 0x7f7d101035e0 2024-03-30T04:57:56.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.077+0000 7f7d0de00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d10102c40 0x7f7d10115af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:57:56.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.077+0000 7f7d0de00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d10102c40 0x7f7d10115af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36436/0 (socket says 172.21.15.123:36436) 2024-03-30T04:57:56.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.077+0000 7f7d0de00700 1 -- 172.21.15.123:0/831576435 learned_addr learned my addr 172.21.15.123:0/831576435 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:57:56.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.077+0000 7f7d0d400700 1 --2- 172.21.15.123:0/831576435 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d101035e0 0x7f7d10116030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:57:56.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.077+0000 7f7d0de00700 1 -- 172.21.15.123:0/831576435 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d101035e0 msgr2=0x7f7d10116030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:56.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.077+0000 7f7d0de00700 1 --2- 172.21.15.123:0/831576435 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d101035e0 0x7f7d10116030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:56.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.077+0000 7f7d0de00700 1 -- 172.21.15.123:0/831576435 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cfc005040 con 0x7f7d10102c40 2024-03-30T04:57:56.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.077+0000 7f7d0de00700 1 --2- 172.21.15.123:0/831576435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d10102c40 0x7f7d10115af0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f7d0000c8c0 tx=0x7f7d000044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:57:56.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.077+0000 7f7d06000700 1 -- 172.21.15.123:0/831576435 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d0000a980 con 0x7f7d10102c40 2024-03-30T04:57:56.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.078+0000 7f7d0f200700 1 -- 172.21.15.123:0/831576435 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d1010fd50 con 0x7f7d10102c40 2024-03-30T04:57:56.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.078+0000 7f7d06000700 1 -- 172.21.15.123:0/831576435 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7d0000ce60 con 0x7f7d10102c40 2024-03-30T04:57:56.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.078+0000 7f7d06000700 1 -- 172.21.15.123:0/831576435 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d000113a0 con 0x7f7d10102c40 2024-03-30T04:57:56.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.078+0000 7f7d0f200700 1 -- 172.21.15.123:0/831576435 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d101102e0 con 0x7f7d10102c40 2024-03-30T04:57:56.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.079+0000 7f7d0f200700 1 -- 172.21.15.123:0/831576435 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d100fe390 con 0x7f7d10102c40 2024-03-30T04:57:56.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.080+0000 7f7d06000700 1 -- 172.21.15.123:0/831576435 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f7d00012400 con 0x7f7d10102c40 2024-03-30T04:57:56.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.080+0000 7f7d06000700 1 --2- 172.21.15.123:0/831576435 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7cf406cc30 0x7f7cf406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:57:56.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.081+0000 7f7d06000700 1 -- 172.21.15.123:0/831576435 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7d0008c1d0 con 0x7f7d10102c40 2024-03-30T04:57:56.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.081+0000 7f7d0d400700 1 --2- 172.21.15.123:0/831576435 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7cf406cc30 0x7f7cf406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:57:56.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.081+0000 7f7d0d400700 1 --2- 172.21.15.123:0/831576435 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7cf406cc30 0x7f7cf406f0e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f7d10110d50 tx=0x7f7cfc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:57:56.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.085+0000 7f7d06000700 1 -- 172.21.15.123:0/831576435 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7d00059c20 con 0x7f7d10102c40 2024-03-30T04:57:56.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:57:55 smithi135 ceph-mon[106011]: from='client.35468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:57:56.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:57:55 smithi135 ceph-mon[106011]: from='client.35472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:57:56.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:57:55 smithi135 ceph-mon[106011]: from='client.35476 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:57:56.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:57:55 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3543617045' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:57:56.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:57:55 smithi123 ceph-mon[139369]: from='client.35468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:57:56.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:57:55 smithi123 ceph-mon[139369]: from='client.35472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:57:56.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:57:55 smithi123 ceph-mon[139369]: from='client.35476 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:57:56.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:57:55 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3543617045' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:57:56.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.307+0000 7f7d0f200700 1 -- 172.21.15.123:0/831576435 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7d10109bd0 con 0x7f7cf406cc30 2024-03-30T04:57:56.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.309+0000 7f7d06000700 1 -- 172.21.15.123:0/831576435 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f7d10109bd0 con 0x7f7cf406cc30 2024-03-30T04:57:56.306 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:57:56.306 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:57:56.306 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:57:56.306 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T04:57:56.307 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:57:56.307 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T04:57:56.307 INFO:teuthology.orchestra.run.smithi123.stdout: "mds", 2024-03-30T04:57:56.307 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T04:57:56.307 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T04:57:56.307 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T04:57:56.307 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:57:56.307 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "16/21 daemons upgraded", 2024-03-30T04:57:56.307 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T04:57:56.307 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:57:56.307 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:57:56.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.311+0000 7f7d0f200700 1 -- 172.21.15.123:0/831576435 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7cf406cc30 msgr2=0x7f7cf406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:56.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.311+0000 7f7d0f200700 1 --2- 172.21.15.123:0/831576435 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7cf406cc30 0x7f7cf406f0e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f7d10110d50 tx=0x7f7cfc009e30 comp rx=0 tx=0).stop 2024-03-30T04:57:56.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.311+0000 7f7d0f200700 1 -- 172.21.15.123:0/831576435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d10102c40 msgr2=0x7f7d10115af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:56.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.311+0000 7f7d0f200700 1 --2- 172.21.15.123:0/831576435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d10102c40 0x7f7d10115af0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f7d0000c8c0 tx=0x7f7d000044c0 comp rx=0 tx=0).stop 2024-03-30T04:57:56.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.311+0000 7f7d0f200700 1 -- 172.21.15.123:0/831576435 shutdown_connections 2024-03-30T04:57:56.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.311+0000 7f7d0f200700 1 --2- 172.21.15.123:0/831576435 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7cf406cc30 0x7f7cf406f0e0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:56.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.311+0000 7f7d0f200700 1 --2- 172.21.15.123:0/831576435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d10102c40 0x7f7d10115af0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:56.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.311+0000 7f7d0f200700 1 --2- 172.21.15.123:0/831576435 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d101035e0 0x7f7d10116030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:56.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.311+0000 7f7d0f200700 1 -- 172.21.15.123:0/831576435 >> 172.21.15.123:0/831576435 conn(0x7f7d100f8830 msgr2=0x7f7d100f9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:57:56.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.311+0000 7f7d0f200700 1 -- 172.21.15.123:0/831576435 shutdown_connections 2024-03-30T04:57:56.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.311+0000 7f7d0f200700 1 -- 172.21.15.123:0/831576435 wait complete. 2024-03-30T04:57:56.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.444+0000 7f1209c00700 1 -- 172.21.15.123:0/731014459 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1204102e00 msgr2=0x7f12041031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:56.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.444+0000 7f1209c00700 1 --2- 172.21.15.123:0/731014459 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1204102e00 0x7f12041031d0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f11f4004940 tx=0x7f11f4009e30 comp rx=0 tx=0).stop 2024-03-30T04:57:56.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.445+0000 7f1209c00700 1 -- 172.21.15.123:0/731014459 shutdown_connections 2024-03-30T04:57:56.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.445+0000 7f1209c00700 1 --2- 172.21.15.123:0/731014459 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f12041037a0 0x7f12040fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:56.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.445+0000 7f1209c00700 1 --2- 172.21.15.123:0/731014459 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1204102e00 0x7f12041031d0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:56.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.445+0000 7f1209c00700 1 -- 172.21.15.123:0/731014459 >> 172.21.15.123:0/731014459 conn(0x7f12040f89b0 msgr2=0x7f12040fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:57:56.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.445+0000 7f1209c00700 1 -- 172.21.15.123:0/731014459 shutdown_connections 2024-03-30T04:57:56.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.445+0000 7f1209c00700 1 -- 172.21.15.123:0/731014459 wait complete. 2024-03-30T04:57:56.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.445+0000 7f1209c00700 1 Processor -- start 2024-03-30T04:57:56.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.446+0000 7f1209c00700 1 -- start start 2024-03-30T04:57:56.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.446+0000 7f1209c00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1204102e00 0x7f1204115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:57:56.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.446+0000 7f1209c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12041037a0 0x7f12041161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:57:56.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.446+0000 7f1209c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1204116810 con 0x7f12041037a0 2024-03-30T04:57:56.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.446+0000 7f1209c00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f120410fbe0 con 0x7f1204102e00 2024-03-30T04:57:56.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.446+0000 7f1202000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12041037a0 0x7f12041161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:57:56.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.446+0000 7f1202000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12041037a0 0x7f12041161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36450/0 (socket says 172.21.15.123:36450) 2024-03-30T04:57:56.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.446+0000 7f1202000700 1 -- 172.21.15.123:0/3507223090 learned_addr learned my addr 172.21.15.123:0/3507223090 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:57:56.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.446+0000 7f1202a00700 1 --2- 172.21.15.123:0/3507223090 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1204102e00 0x7f1204115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:57:56.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.447+0000 7f1202000700 1 -- 172.21.15.123:0/3507223090 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1204102e00 msgr2=0x7f1204115c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:56.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.447+0000 7f1202000700 1 --2- 172.21.15.123:0/3507223090 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1204102e00 0x7f1204115c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:56.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.447+0000 7f1202000700 1 -- 172.21.15.123:0/3507223090 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11f4005040 con 0x7f12041037a0 2024-03-30T04:57:56.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.447+0000 7f1202000700 1 --2- 172.21.15.123:0/3507223090 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12041037a0 0x7f12041161c0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f11f0002a50 tx=0x7f11f00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:57:56.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.447+0000 7f11faa00700 1 -- 172.21.15.123:0/3507223090 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11f0007cc0 con 0x7f12041037a0 2024-03-30T04:57:56.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.447+0000 7f1209c00700 1 -- 172.21.15.123:0/3507223090 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f120410fec0 con 0x7f12041037a0 2024-03-30T04:57:56.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.447+0000 7f11faa00700 1 -- 172.21.15.123:0/3507223090 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f11f000f040 con 0x7f12041037a0 2024-03-30T04:57:56.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.447+0000 7f11faa00700 1 -- 172.21.15.123:0/3507223090 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11f000da30 con 0x7f12041037a0 2024-03-30T04:57:56.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.447+0000 7f1209c00700 1 -- 172.21.15.123:0/3507223090 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1204110480 con 0x7f12041037a0 2024-03-30T04:57:56.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.448+0000 7f1209c00700 1 -- 172.21.15.123:0/3507223090 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12040fe550 con 0x7f12041037a0 2024-03-30T04:57:56.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.449+0000 7f11faa00700 1 -- 172.21.15.123:0/3507223090 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f11f000db90 con 0x7f12041037a0 2024-03-30T04:57:56.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.450+0000 7f11faa00700 1 --2- 172.21.15.123:0/3507223090 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f11e806cc30 0x7f11e806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:57:56.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.450+0000 7f11faa00700 1 -- 172.21.15.123:0/3507223090 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f11f008bca0 con 0x7f12041037a0 2024-03-30T04:57:56.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.451+0000 7f1202a00700 1 --2- 172.21.15.123:0/3507223090 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f11e806cc30 0x7f11e806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:57:56.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.451+0000 7f1202a00700 1 --2- 172.21.15.123:0/3507223090 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f11e806cc30 0x7f11e806f0e0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f11f4009fd0 tx=0x7f11f4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:57:56.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.455+0000 7f11faa00700 1 -- 172.21.15.123:0/3507223090 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f11f00596f0 con 0x7f12041037a0 2024-03-30T04:57:56.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.741+0000 7f1209c00700 1 -- 172.21.15.123:0/3507223090 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f120404bed0 con 0x7f12041037a0 2024-03-30T04:57:56.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.741+0000 7f11faa00700 1 -- 172.21.15.123:0/3507223090 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f11f0059280 con 0x7f12041037a0 2024-03-30T04:57:56.740 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T04:57:56.740 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-30T04:57:56.740 INFO:teuthology.orchestra.run.smithi123.stdout: host smithi135 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi135/config 2024-03-30T04:57:56.740 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-30T04:57:56.740 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T04:57:56.740 INFO:teuthology.orchestra.run.smithi123.stdout:Traceback (most recent call last): 2024-03-30T04:57:56.740 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-30T04:57:56.741 INFO:teuthology.orchestra.run.smithi123.stdout: "__main__", mod_spec) 2024-03-30T04:57:56.741 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-30T04:57:56.741 INFO:teuthology.orchestra.run.smithi123.stdout: exec(code, run_globals) 2024-03-30T04:57:56.741 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10889, in 2024-03-30T04:57:56.741 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10877, in main 2024-03-30T04:57:56.741 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-30T04:57:56.741 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-30T04:57:56.741 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-30T04:57:56.741 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-30T04:57:56.741 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-03-30T04:57:56.741 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-30T04:57:56.742 INFO:teuthology.orchestra.run.smithi123.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T04:57:56.742 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T04:57:56.742 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T04:57:56.742 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T04:57:56.742 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T04:57:56.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.744+0000 7f1209c00700 1 -- 172.21.15.123:0/3507223090 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f11e806cc30 msgr2=0x7f11e806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:56.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.744+0000 7f1209c00700 1 --2- 172.21.15.123:0/3507223090 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f11e806cc30 0x7f11e806f0e0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f11f4009fd0 tx=0x7f11f4009e30 comp rx=0 tx=0).stop 2024-03-30T04:57:56.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.744+0000 7f1209c00700 1 -- 172.21.15.123:0/3507223090 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12041037a0 msgr2=0x7f12041161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:57:56.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.744+0000 7f1209c00700 1 --2- 172.21.15.123:0/3507223090 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12041037a0 0x7f12041161c0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f11f0002a50 tx=0x7f11f00046b0 comp rx=0 tx=0).stop 2024-03-30T04:57:56.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.744+0000 7f1209c00700 1 -- 172.21.15.123:0/3507223090 shutdown_connections 2024-03-30T04:57:56.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.744+0000 7f1209c00700 1 --2- 172.21.15.123:0/3507223090 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f11e806cc30 0x7f11e806f0e0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:56.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.744+0000 7f1209c00700 1 --2- 172.21.15.123:0/3507223090 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1204102e00 0x7f1204115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:56.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.744+0000 7f1209c00700 1 --2- 172.21.15.123:0/3507223090 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12041037a0 0x7f12041161c0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:57:56.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.744+0000 7f1209c00700 1 -- 172.21.15.123:0/3507223090 >> 172.21.15.123:0/3507223090 conn(0x7f12040f89b0 msgr2=0x7f12040f9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:57:56.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.744+0000 7f1209c00700 1 -- 172.21.15.123:0/3507223090 shutdown_connections 2024-03-30T04:57:56.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:57:56.744+0000 7f1209c00700 1 -- 172.21.15.123:0/3507223090 wait complete. 2024-03-30T04:57:57.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:57:56 smithi135 ceph-mon[106011]: pgmap v803: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:57.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:57:56 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/4241133885' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:57:57.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:57:56 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3507223090' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:57:57.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:57:56 smithi123 ceph-mon[139369]: pgmap v803: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:57.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:57:56 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/4241133885' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:57:57.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:57:56 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3507223090' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:57:58.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:57:57 smithi135 ceph-mon[106011]: from='client.35488 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:57:58.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:57:57 smithi123 ceph-mon[139369]: from='client.35488 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:57:59.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:57:58 smithi135 ceph-mon[106011]: pgmap v804: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:57:59.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:57:58 smithi123 ceph-mon[139369]: pgmap v804: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:00.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:57:59 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:58:00.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:57:59 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:58:01.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:58:00 smithi135 ceph-mon[106011]: pgmap v805: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:01.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:58:00 smithi123 ceph-mon[139369]: pgmap v805: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:03.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:58:02 smithi135 ceph-mon[106011]: pgmap v806: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:03.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:58:02 smithi123 ceph-mon[139369]: pgmap v806: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:05.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:58:04 smithi123 ceph-mon[139369]: pgmap v807: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:05.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:58:04 smithi135 ceph-mon[106011]: pgmap v807: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:07.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:58:06 smithi123 ceph-mon[139369]: pgmap v808: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:07.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:58:06 smithi135 ceph-mon[106011]: pgmap v808: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:09.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:58:08 smithi123 ceph-mon[139369]: pgmap v809: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:09.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:58:08 smithi135 ceph-mon[106011]: pgmap v809: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:11.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:58:10 smithi123 ceph-mon[139369]: pgmap v810: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:11.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:58:10 smithi135 ceph-mon[106011]: pgmap v810: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:13.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:58:12 smithi123 ceph-mon[139369]: pgmap v811: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:13.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:58:12 smithi135 ceph-mon[106011]: pgmap v811: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:15.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:58:14 smithi123 ceph-mon[139369]: pgmap v812: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:15.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:58:14 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:58:15.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:58:14 smithi135 ceph-mon[106011]: pgmap v812: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:15.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:58:14 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:58:17.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:58:16 smithi123 ceph-mon[139369]: pgmap v813: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:17.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:58:16 smithi135 ceph-mon[106011]: pgmap v813: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:19.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:58:18 smithi123 ceph-mon[139369]: pgmap v814: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:19.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:58:18 smithi135 ceph-mon[106011]: pgmap v814: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:21.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:58:20 smithi123 ceph-mon[139369]: pgmap v815: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:21.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:58:20 smithi135 ceph-mon[106011]: pgmap v815: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:23.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:58:22 smithi123 ceph-mon[139369]: pgmap v816: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:23.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:58:22 smithi135 ceph-mon[106011]: pgmap v816: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:25.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:58:24 smithi123 ceph-mon[139369]: pgmap v817: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:25.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:58:24 smithi135 ceph-mon[106011]: pgmap v817: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:26.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:26.895+0000 7f9a08800700 1 -- 172.21.15.123:0/257533608 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a00102e20 msgr2=0x7f9a001031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:58:26.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:26.895+0000 7f9a08800700 1 --2- 172.21.15.123:0/257533608 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a00102e20 0x7f9a001031f0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f99f4004940 tx=0x7f99f4009e30 comp rx=0 tx=0).stop 2024-03-30T04:58:26.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:26.896+0000 7f9a08800700 1 -- 172.21.15.123:0/257533608 shutdown_connections 2024-03-30T04:58:26.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:26.896+0000 7f9a08800700 1 --2- 172.21.15.123:0/257533608 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9a001037c0 0x7f9a000fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:58:26.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:26.896+0000 7f9a08800700 1 --2- 172.21.15.123:0/257533608 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a00102e20 0x7f9a001031f0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:58:26.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:26.896+0000 7f9a08800700 1 -- 172.21.15.123:0/257533608 >> 172.21.15.123:0/257533608 conn(0x7f9a000f8970 msgr2=0x7f9a000fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:58:26.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:26.896+0000 7f9a08800700 1 -- 172.21.15.123:0/257533608 shutdown_connections 2024-03-30T04:58:26.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:26.896+0000 7f9a08800700 1 -- 172.21.15.123:0/257533608 wait complete. 2024-03-30T04:58:26.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:26.897+0000 7f9a08800700 1 Processor -- start 2024-03-30T04:58:26.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:26.897+0000 7f9a08800700 1 -- start start 2024-03-30T04:58:26.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:26.897+0000 7f9a08800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a00102e20 0x7f9a00115d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:58:26.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:26.897+0000 7f9a08800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9a001037c0 0x7f9a001162a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:58:26.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:26.897+0000 7f9a08800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a001168f0 con 0x7f9a00102e20 2024-03-30T04:58:26.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:26.897+0000 7f9a08800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a0010fcc0 con 0x7f9a001037c0 2024-03-30T04:58:26.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:26.897+0000 7f9a05e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a00102e20 0x7f9a00115d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:58:26.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:26.897+0000 7f9a05e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a00102e20 0x7f9a00115d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54298/0 (socket says 172.21.15.123:54298) 2024-03-30T04:58:26.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:26.897+0000 7f9a05e00700 1 -- 172.21.15.123:0/2037907093 learned_addr learned my addr 172.21.15.123:0/2037907093 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:58:26.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:26.898+0000 7f9a05400700 1 --2- 172.21.15.123:0/2037907093 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9a001037c0 0x7f9a001162a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:58:26.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:26.898+0000 7f9a05e00700 1 -- 172.21.15.123:0/2037907093 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9a001037c0 msgr2=0x7f9a001162a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:58:26.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:26.898+0000 7f9a05e00700 1 --2- 172.21.15.123:0/2037907093 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9a001037c0 0x7f9a001162a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:58:26.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:26.898+0000 7f9a05e00700 1 -- 172.21.15.123:0/2037907093 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f4005040 con 0x7f9a00102e20 2024-03-30T04:58:26.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:26.898+0000 7f9a05e00700 1 --2- 172.21.15.123:0/2037907093 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a00102e20 0x7f9a00115d60 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f99f4009390 tx=0x7f99f4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:58:26.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:26.898+0000 7f99fa000700 1 -- 172.21.15.123:0/2037907093 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99f400e040 con 0x7f9a00102e20 2024-03-30T04:58:26.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:26.898+0000 7f99fa000700 1 -- 172.21.15.123:0/2037907093 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f99f4013420 con 0x7f9a00102e20 2024-03-30T04:58:26.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:26.898+0000 7f9a08800700 1 -- 172.21.15.123:0/2037907093 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a0010ff40 con 0x7f9a00102e20 2024-03-30T04:58:26.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:26.898+0000 7f99fa000700 1 -- 172.21.15.123:0/2037907093 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99f401c630 con 0x7f9a00102e20 2024-03-30T04:58:26.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:26.898+0000 7f9a08800700 1 -- 172.21.15.123:0/2037907093 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a00110480 con 0x7f9a00102e20 2024-03-30T04:58:26.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:26.900+0000 7f9a08800700 1 -- 172.21.15.123:0/2037907093 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a000fe570 con 0x7f9a00102e20 2024-03-30T04:58:26.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:26.901+0000 7f99fa000700 1 -- 172.21.15.123:0/2037907093 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f99f40139e0 con 0x7f9a00102e20 2024-03-30T04:58:26.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:26.902+0000 7f99fa000700 1 --2- 172.21.15.123:0/2037907093 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f99e806cc30 0x7f99e806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:58:26.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:26.902+0000 7f99fa000700 1 -- 172.21.15.123:0/2037907093 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f99f408d2a0 con 0x7f9a00102e20 2024-03-30T04:58:26.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:26.902+0000 7f9a05400700 1 --2- 172.21.15.123:0/2037907093 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f99e806cc30 0x7f99e806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:58:26.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:26.902+0000 7f9a05400700 1 --2- 172.21.15.123:0/2037907093 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f99e806cc30 0x7f99e806f0e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f9a00111600 tx=0x7f99f0009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:58:26.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:26.907+0000 7f99fa000700 1 -- 172.21.15.123:0/2037907093 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f99f405ae20 con 0x7f9a00102e20 2024-03-30T04:58:27.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.129+0000 7f9a08800700 1 -- 172.21.15.123:0/2037907093 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9a0004bf40 con 0x7f99e806cc30 2024-03-30T04:58:27.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.130+0000 7f99fa000700 1 -- 172.21.15.123:0/2037907093 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f9a0004bf40 con 0x7f99e806cc30 2024-03-30T04:58:27.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.133+0000 7f9a08800700 1 -- 172.21.15.123:0/2037907093 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f99e806cc30 msgr2=0x7f99e806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:58:27.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.133+0000 7f9a08800700 1 --2- 172.21.15.123:0/2037907093 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f99e806cc30 0x7f99e806f0e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f9a00111600 tx=0x7f99f0009ee0 comp rx=0 tx=0).stop 2024-03-30T04:58:27.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.133+0000 7f9a08800700 1 -- 172.21.15.123:0/2037907093 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a00102e20 msgr2=0x7f9a00115d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:58:27.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.133+0000 7f9a08800700 1 --2- 172.21.15.123:0/2037907093 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a00102e20 0x7f9a00115d60 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f99f4009390 tx=0x7f99f4009e30 comp rx=0 tx=0).stop 2024-03-30T04:58:27.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.133+0000 7f9a08800700 1 -- 172.21.15.123:0/2037907093 shutdown_connections 2024-03-30T04:58:27.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.133+0000 7f9a08800700 1 --2- 172.21.15.123:0/2037907093 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f99e806cc30 0x7f99e806f0e0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:58:27.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.133+0000 7f9a08800700 1 --2- 172.21.15.123:0/2037907093 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a00102e20 0x7f9a00115d60 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:58:27.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.133+0000 7f9a08800700 1 --2- 172.21.15.123:0/2037907093 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9a001037c0 0x7f9a001162a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:58:27.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.133+0000 7f9a08800700 1 -- 172.21.15.123:0/2037907093 >> 172.21.15.123:0/2037907093 conn(0x7f9a000f8970 msgr2=0x7f9a000f99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:58:27.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.133+0000 7f9a08800700 1 -- 172.21.15.123:0/2037907093 shutdown_connections 2024-03-30T04:58:27.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.133+0000 7f9a08800700 1 -- 172.21.15.123:0/2037907093 wait complete. 2024-03-30T04:58:27.146 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:58:27.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:58:26 smithi123 ceph-mon[139369]: pgmap v818: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:27.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.267+0000 7fdc12200700 1 -- 172.21.15.123:0/811933944 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc0c102e00 msgr2=0x7fdc0c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:58:27.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.267+0000 7fdc12200700 1 --2- 172.21.15.123:0/811933944 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc0c102e00 0x7fdc0c1031d0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc004940 tx=0x7fdbfc009e30 comp rx=0 tx=0).stop 2024-03-30T04:58:27.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.267+0000 7fdc12200700 1 -- 172.21.15.123:0/811933944 shutdown_connections 2024-03-30T04:58:27.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.267+0000 7fdc12200700 1 --2- 172.21.15.123:0/811933944 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdc0c1037a0 0x7fdc0c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:58:27.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.267+0000 7fdc12200700 1 --2- 172.21.15.123:0/811933944 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc0c102e00 0x7fdc0c1031d0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:58:27.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.267+0000 7fdc12200700 1 -- 172.21.15.123:0/811933944 >> 172.21.15.123:0/811933944 conn(0x7fdc0c0f89b0 msgr2=0x7fdc0c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:58:27.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.268+0000 7fdc12200700 1 -- 172.21.15.123:0/811933944 shutdown_connections 2024-03-30T04:58:27.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.268+0000 7fdc12200700 1 -- 172.21.15.123:0/811933944 wait complete. 2024-03-30T04:58:27.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.268+0000 7fdc12200700 1 Processor -- start 2024-03-30T04:58:27.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.268+0000 7fdc12200700 1 -- start start 2024-03-30T04:58:27.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.269+0000 7fdc12200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdc0c102e00 0x7fdc0c115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:58:27.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.269+0000 7fdc12200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc0c1037a0 0x7fdc0c116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:58:27.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.269+0000 7fdc12200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c116860 con 0x7fdc0c1037a0 2024-03-30T04:58:27.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.269+0000 7fdc12200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c10fc30 con 0x7fdc0c102e00 2024-03-30T04:58:27.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.269+0000 7fdc0aa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc0c1037a0 0x7fdc0c116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:58:27.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.269+0000 7fdc0aa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc0c1037a0 0x7fdc0c116210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54308/0 (socket says 172.21.15.123:54308) 2024-03-30T04:58:27.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.269+0000 7fdc0aa00700 1 -- 172.21.15.123:0/1768607701 learned_addr learned my addr 172.21.15.123:0/1768607701 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:58:27.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.269+0000 7fdc0b400700 1 --2- 172.21.15.123:0/1768607701 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdc0c102e00 0x7fdc0c115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:58:27.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.270+0000 7fdc0aa00700 1 -- 172.21.15.123:0/1768607701 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdc0c102e00 msgr2=0x7fdc0c115cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:58:27.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.270+0000 7fdc0aa00700 1 --2- 172.21.15.123:0/1768607701 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdc0c102e00 0x7fdc0c115cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:58:27.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.270+0000 7fdc0aa00700 1 -- 172.21.15.123:0/1768607701 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbfc005040 con 0x7fdc0c1037a0 2024-03-30T04:58:27.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.270+0000 7fdc0aa00700 1 --2- 172.21.15.123:0/1768607701 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc0c1037a0 0x7fdc0c116210 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8002a50 tx=0x7fdbf8004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:58:27.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.270+0000 7fdc03400700 1 -- 172.21.15.123:0/1768607701 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdbf8007cc0 con 0x7fdc0c1037a0 2024-03-30T04:58:27.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.270+0000 7fdc03400700 1 -- 172.21.15.123:0/1768607701 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdbf800f040 con 0x7fdc0c1037a0 2024-03-30T04:58:27.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.270+0000 7fdc03400700 1 -- 172.21.15.123:0/1768607701 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdbf800d920 con 0x7fdc0c1037a0 2024-03-30T04:58:27.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.270+0000 7fdc12200700 1 -- 172.21.15.123:0/1768607701 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc0c10ff10 con 0x7fdc0c1037a0 2024-03-30T04:58:27.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.270+0000 7fdc12200700 1 -- 172.21.15.123:0/1768607701 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc0c1104d0 con 0x7fdc0c1037a0 2024-03-30T04:58:27.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.271+0000 7fdc12200700 1 -- 172.21.15.123:0/1768607701 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc0c0fe550 con 0x7fdc0c1037a0 2024-03-30T04:58:27.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.272+0000 7fdc03400700 1 -- 172.21.15.123:0/1768607701 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fdbf8009680 con 0x7fdc0c1037a0 2024-03-30T04:58:27.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.273+0000 7fdc03400700 1 --2- 172.21.15.123:0/1768607701 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fdbf006cc30 0x7fdbf006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:58:27.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.274+0000 7fdc03400700 1 -- 172.21.15.123:0/1768607701 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fdbf808bd10 con 0x7fdc0c1037a0 2024-03-30T04:58:27.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.274+0000 7fdc0b400700 1 --2- 172.21.15.123:0/1768607701 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fdbf006cc30 0x7fdbf006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:58:27.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.274+0000 7fdc0b400700 1 --2- 172.21.15.123:0/1768607701 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fdbf006cc30 0x7fdbf006f0e0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc009fd0 tx=0x7fdbfc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:58:27.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.278+0000 7fdc03400700 1 -- 172.21.15.123:0/1768607701 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fdbf8059760 con 0x7fdc0c1037a0 2024-03-30T04:58:27.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:58:26 smithi135 ceph-mon[106011]: pgmap v818: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:27.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.497+0000 7fdc12200700 1 -- 172.21.15.123:0/1768607701 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdc0c109e30 con 0x7fdbf006cc30 2024-03-30T04:58:27.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.498+0000 7fdc03400700 1 -- 172.21.15.123:0/1768607701 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fdc0c109e30 con 0x7fdbf006cc30 2024-03-30T04:58:27.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.501+0000 7fdc12200700 1 -- 172.21.15.123:0/1768607701 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fdbf006cc30 msgr2=0x7fdbf006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:58:27.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.501+0000 7fdc12200700 1 --2- 172.21.15.123:0/1768607701 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fdbf006cc30 0x7fdbf006f0e0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc009fd0 tx=0x7fdbfc009e30 comp rx=0 tx=0).stop 2024-03-30T04:58:27.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.501+0000 7fdc12200700 1 -- 172.21.15.123:0/1768607701 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc0c1037a0 msgr2=0x7fdc0c116210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:58:27.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.501+0000 7fdc12200700 1 --2- 172.21.15.123:0/1768607701 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc0c1037a0 0x7fdc0c116210 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8002a50 tx=0x7fdbf8004600 comp rx=0 tx=0).stop 2024-03-30T04:58:27.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.501+0000 7fdc12200700 1 -- 172.21.15.123:0/1768607701 shutdown_connections 2024-03-30T04:58:27.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.501+0000 7fdc12200700 1 --2- 172.21.15.123:0/1768607701 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fdbf006cc30 0x7fdbf006f0e0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:58:27.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.501+0000 7fdc12200700 1 --2- 172.21.15.123:0/1768607701 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdc0c102e00 0x7fdc0c115cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:58:27.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.501+0000 7fdc12200700 1 --2- 172.21.15.123:0/1768607701 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc0c1037a0 0x7fdc0c116210 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:58:27.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.501+0000 7fdc12200700 1 -- 172.21.15.123:0/1768607701 >> 172.21.15.123:0/1768607701 conn(0x7fdc0c0f89b0 msgr2=0x7fdc0c0f9980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:58:27.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.501+0000 7fdc12200700 1 -- 172.21.15.123:0/1768607701 shutdown_connections 2024-03-30T04:58:27.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.501+0000 7fdc12200700 1 -- 172.21.15.123:0/1768607701 wait complete. 2024-03-30T04:58:27.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.635+0000 7f4180000700 1 -- 172.21.15.123:0/3459924501 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41781037a0 msgr2=0x7f41780fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:58:27.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.635+0000 7f4180000700 1 --2- 172.21.15.123:0/3459924501 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41781037a0 0x7f41780fce10 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f4164004940 tx=0x7f4164009e30 comp rx=0 tx=0).stop 2024-03-30T04:58:27.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.635+0000 7f4180000700 1 -- 172.21.15.123:0/3459924501 shutdown_connections 2024-03-30T04:58:27.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.635+0000 7f4180000700 1 --2- 172.21.15.123:0/3459924501 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41781037a0 0x7f41780fce10 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:58:27.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.635+0000 7f4180000700 1 --2- 172.21.15.123:0/3459924501 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4178102e00 0x7f41781031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:58:27.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.635+0000 7f4180000700 1 -- 172.21.15.123:0/3459924501 >> 172.21.15.123:0/3459924501 conn(0x7f41780f89b0 msgr2=0x7f41780fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:58:27.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.635+0000 7f4180000700 1 -- 172.21.15.123:0/3459924501 shutdown_connections 2024-03-30T04:58:27.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.635+0000 7f4180000700 1 -- 172.21.15.123:0/3459924501 wait complete. 2024-03-30T04:58:27.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.636+0000 7f4180000700 1 Processor -- start 2024-03-30T04:58:27.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.636+0000 7f4180000700 1 -- start start 2024-03-30T04:58:27.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.637+0000 7f4180000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4178102e00 0x7f4178115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:58:27.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.637+0000 7f4180000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41781037a0 0x7f4178116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:58:27.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.637+0000 7f4180000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4178116880 con 0x7f41781037a0 2024-03-30T04:58:27.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.637+0000 7f4180000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f417810fc50 con 0x7f4178102e00 2024-03-30T04:58:27.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.637+0000 7f417cc00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41781037a0 0x7f4178116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:58:27.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.637+0000 7f417cc00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41781037a0 0x7f4178116230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54326/0 (socket says 172.21.15.123:54326) 2024-03-30T04:58:27.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.637+0000 7f417cc00700 1 -- 172.21.15.123:0/604153081 learned_addr learned my addr 172.21.15.123:0/604153081 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:58:27.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.637+0000 7f417d600700 1 --2- 172.21.15.123:0/604153081 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4178102e00 0x7f4178115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:58:27.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.637+0000 7f417cc00700 1 -- 172.21.15.123:0/604153081 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4178102e00 msgr2=0x7f4178115cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:58:27.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.637+0000 7f417cc00700 1 --2- 172.21.15.123:0/604153081 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4178102e00 0x7f4178115cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:58:27.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.637+0000 7f417cc00700 1 -- 172.21.15.123:0/604153081 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4164005040 con 0x7f41781037a0 2024-03-30T04:58:27.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.637+0000 7f417cc00700 1 --2- 172.21.15.123:0/604153081 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41781037a0 0x7f4178116230 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f4164009390 tx=0x7f4164009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:58:27.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.638+0000 7f4175600700 1 -- 172.21.15.123:0/604153081 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f416400e040 con 0x7f41781037a0 2024-03-30T04:58:27.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.638+0000 7f4175600700 1 -- 172.21.15.123:0/604153081 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4164013420 con 0x7f41781037a0 2024-03-30T04:58:27.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.638+0000 7f4180000700 1 -- 172.21.15.123:0/604153081 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f417810fed0 con 0x7f41781037a0 2024-03-30T04:58:27.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.638+0000 7f4175600700 1 -- 172.21.15.123:0/604153081 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f416401c630 con 0x7f41781037a0 2024-03-30T04:58:27.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.638+0000 7f4180000700 1 -- 172.21.15.123:0/604153081 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4178110490 con 0x7f41781037a0 2024-03-30T04:58:27.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.639+0000 7f4180000700 1 -- 172.21.15.123:0/604153081 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41780fe550 con 0x7f41781037a0 2024-03-30T04:58:27.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.640+0000 7f4175600700 1 -- 172.21.15.123:0/604153081 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f4164013a00 con 0x7f41781037a0 2024-03-30T04:58:27.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.641+0000 7f4175600700 1 --2- 172.21.15.123:0/604153081 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f416006cc30 0x7f416006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:58:27.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.641+0000 7f4175600700 1 -- 172.21.15.123:0/604153081 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f416408d120 con 0x7f41781037a0 2024-03-30T04:58:27.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.641+0000 7f417d600700 1 --2- 172.21.15.123:0/604153081 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f416006cc30 0x7f416006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:58:27.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.642+0000 7f417d600700 1 --2- 172.21.15.123:0/604153081 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f416006cc30 0x7f416006f0e0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f416c004f40 tx=0x7f416c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:58:27.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.645+0000 7f4175600700 1 -- 172.21.15.123:0/604153081 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f416405abf0 con 0x7f41781037a0 2024-03-30T04:58:27.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.856+0000 7f4180000700 1 -- 172.21.15.123:0/604153081 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f41781113e0 con 0x7f416006cc30 2024-03-30T04:58:27.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.866+0000 7f4175600700 1 -- 172.21.15.123:0/604153081 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f41781113e0 con 0x7f416006cc30 2024-03-30T04:58:27.864 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:58:27.864 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (27m) 5m ago 40m 14.6M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:58:27.864 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (24m) 5m ago 40m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T04:58:27.864 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (24m) 5m ago 39m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T04:58:27.864 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (27m) 5m ago 39m 73.5M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T04:58:27.864 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (20m) 5m ago 33m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T04:58:27.864 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (20m) 5m ago 33m 16.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T04:58:27.865 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (5m) 5m ago 33m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 87b073a6c9ca 2024-03-30T04:58:27.865 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (20m) 5m ago 33m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T04:58:27.865 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (29m) 5m ago 41m 510M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:58:27.865 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (28m) 5m ago 39m 456M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:58:27.865 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (26m) 5m ago 41m 70.0M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T04:58:27.865 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (25m) 5m ago 38m 51.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T04:58:27.865 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (28m) 5m ago 40m 8636k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:58:27.865 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (28m) 5m ago 39m 8678k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:58:27.865 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (24m) 5m ago 38m 107M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T04:58:27.865 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (23m) 5m ago 37m 97.2M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T04:58:27.865 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (23m) 5m ago 37m 89.8M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T04:58:27.865 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (22m) 5m ago 36m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T04:58:27.865 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (21m) 5m ago 35m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T04:58:27.865 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (21m) 5m ago 35m 88.8M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T04:58:27.865 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (27m) 5m ago 39m 63.9M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:58:27.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.868+0000 7f4180000700 1 -- 172.21.15.123:0/604153081 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f416006cc30 msgr2=0x7f416006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:58:27.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.868+0000 7f4180000700 1 --2- 172.21.15.123:0/604153081 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f416006cc30 0x7f416006f0e0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f416c004f40 tx=0x7f416c00b040 comp rx=0 tx=0).stop 2024-03-30T04:58:27.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.868+0000 7f4180000700 1 -- 172.21.15.123:0/604153081 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41781037a0 msgr2=0x7f4178116230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:58:27.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.868+0000 7f4180000700 1 --2- 172.21.15.123:0/604153081 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41781037a0 0x7f4178116230 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f4164009390 tx=0x7f4164009e30 comp rx=0 tx=0).stop 2024-03-30T04:58:27.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.868+0000 7f4180000700 1 -- 172.21.15.123:0/604153081 shutdown_connections 2024-03-30T04:58:27.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.868+0000 7f4180000700 1 --2- 172.21.15.123:0/604153081 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f416006cc30 0x7f416006f0e0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:58:27.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.868+0000 7f4180000700 1 --2- 172.21.15.123:0/604153081 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4178102e00 0x7f4178115cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:58:27.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.868+0000 7f4180000700 1 --2- 172.21.15.123:0/604153081 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41781037a0 0x7f4178116230 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:58:27.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.868+0000 7f4180000700 1 -- 172.21.15.123:0/604153081 >> 172.21.15.123:0/604153081 conn(0x7f41780f89b0 msgr2=0x7f41780f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:58:27.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.869+0000 7f4180000700 1 -- 172.21.15.123:0/604153081 shutdown_connections 2024-03-30T04:58:27.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:27.869+0000 7f4180000700 1 -- 172.21.15.123:0/604153081 wait complete. 2024-03-30T04:58:28.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.003+0000 7f3b22e00700 1 -- 172.21.15.123:0/2294729068 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b1c1037a0 msgr2=0x7f3b1c0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:58:28.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.003+0000 7f3b22e00700 1 --2- 172.21.15.123:0/2294729068 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b1c1037a0 0x7f3b1c0fce10 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f3b08004940 tx=0x7f3b08009e30 comp rx=0 tx=0).stop 2024-03-30T04:58:28.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.003+0000 7f3b22e00700 1 -- 172.21.15.123:0/2294729068 shutdown_connections 2024-03-30T04:58:28.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.003+0000 7f3b22e00700 1 --2- 172.21.15.123:0/2294729068 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b1c1037a0 0x7f3b1c0fce10 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:58:28.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.003+0000 7f3b22e00700 1 --2- 172.21.15.123:0/2294729068 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b1c102e00 0x7f3b1c1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:58:28.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.003+0000 7f3b22e00700 1 -- 172.21.15.123:0/2294729068 >> 172.21.15.123:0/2294729068 conn(0x7f3b1c0f89b0 msgr2=0x7f3b1c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:58:28.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.003+0000 7f3b22e00700 1 -- 172.21.15.123:0/2294729068 shutdown_connections 2024-03-30T04:58:28.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.003+0000 7f3b22e00700 1 -- 172.21.15.123:0/2294729068 wait complete. 2024-03-30T04:58:28.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.004+0000 7f3b22e00700 1 Processor -- start 2024-03-30T04:58:28.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.004+0000 7f3b22e00700 1 -- start start 2024-03-30T04:58:28.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.005+0000 7f3b22e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b1c102e00 0x7f3b1c115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:58:28.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.005+0000 7f3b22e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b1c1037a0 0x7f3b1c1161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:58:28.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.005+0000 7f3b22e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1c116810 con 0x7f3b1c102e00 2024-03-30T04:58:28.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.005+0000 7f3b22e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1c10fbe0 con 0x7f3b1c1037a0 2024-03-30T04:58:28.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.005+0000 7f3b1be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b1c102e00 0x7f3b1c115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:58:28.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.005+0000 7f3b1be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b1c102e00 0x7f3b1c115c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54332/0 (socket says 172.21.15.123:54332) 2024-03-30T04:58:28.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.005+0000 7f3b1be00700 1 -- 172.21.15.123:0/3670093557 learned_addr learned my addr 172.21.15.123:0/3670093557 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:58:28.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.005+0000 7f3b1b400700 1 --2- 172.21.15.123:0/3670093557 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b1c1037a0 0x7f3b1c1161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:58:28.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.005+0000 7f3b1be00700 1 -- 172.21.15.123:0/3670093557 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b1c1037a0 msgr2=0x7f3b1c1161c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:58:28.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.005+0000 7f3b1be00700 1 --2- 172.21.15.123:0/3670093557 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b1c1037a0 0x7f3b1c1161c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:58:28.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.005+0000 7f3b1be00700 1 -- 172.21.15.123:0/3670093557 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b08005040 con 0x7f3b1c102e00 2024-03-30T04:58:28.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.006+0000 7f3b1be00700 1 --2- 172.21.15.123:0/3670093557 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b1c102e00 0x7f3b1c115c80 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c00eb40 tx=0x7f3b0c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:58:28.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.006+0000 7f3b13e00700 1 -- 172.21.15.123:0/3670093557 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b0c007cc0 con 0x7f3b1c102e00 2024-03-30T04:58:28.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.006+0000 7f3b13e00700 1 -- 172.21.15.123:0/3670093557 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3b0c00f040 con 0x7f3b1c102e00 2024-03-30T04:58:28.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.006+0000 7f3b22e00700 1 -- 172.21.15.123:0/3670093557 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b1c10fe60 con 0x7f3b1c102e00 2024-03-30T04:58:28.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.006+0000 7f3b13e00700 1 -- 172.21.15.123:0/3670093557 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b0c00c990 con 0x7f3b1c102e00 2024-03-30T04:58:28.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.006+0000 7f3b22e00700 1 -- 172.21.15.123:0/3670093557 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b1c110420 con 0x7f3b1c102e00 2024-03-30T04:58:28.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.007+0000 7f3b22e00700 1 -- 172.21.15.123:0/3670093557 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b1c0fe550 con 0x7f3b1c102e00 2024-03-30T04:58:28.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.008+0000 7f3b13e00700 1 -- 172.21.15.123:0/3670093557 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f3b0c00cc00 con 0x7f3b1c102e00 2024-03-30T04:58:28.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.009+0000 7f3b13e00700 1 --2- 172.21.15.123:0/3670093557 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3b0006cc30 0x7f3b0006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:58:28.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.009+0000 7f3b13e00700 1 -- 172.21.15.123:0/3670093557 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f3b0c08bcd0 con 0x7f3b1c102e00 2024-03-30T04:58:28.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.009+0000 7f3b1b400700 1 --2- 172.21.15.123:0/3670093557 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3b0006cc30 0x7f3b0006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:58:28.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.010+0000 7f3b1b400700 1 --2- 172.21.15.123:0/3670093557 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3b0006cc30 0x7f3b0006f0e0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c111520 tx=0x7f3b08009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:58:28.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.013+0000 7f3b13e00700 1 -- 172.21.15.123:0/3670093557 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3b0c059750 con 0x7f3b1c102e00 2024-03-30T04:58:28.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.296+0000 7f3b22e00700 1 -- 172.21.15.123:0/3670093557 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f3b1c04bed0 con 0x7f3b1c102e00 2024-03-30T04:58:28.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.296+0000 7f3b13e00700 1 -- 172.21.15.123:0/3670093557 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f3b0c0592e0 con 0x7f3b1c102e00 2024-03-30T04:58:28.295 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:58:28.295 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:58:28.295 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:58:28.295 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:58:28.295 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:58:28.295 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:58:28.295 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:58:28.295 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:58:28.296 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T04:58:28.296 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:58:28.296 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:58:28.296 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 4 2024-03-30T04:58:28.296 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:58:28.296 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:58:28.296 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 14 2024-03-30T04:58:28.296 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:58:28.296 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:58:28.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.299+0000 7f3b22e00700 1 -- 172.21.15.123:0/3670093557 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3b0006cc30 msgr2=0x7f3b0006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:58:28.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.299+0000 7f3b22e00700 1 --2- 172.21.15.123:0/3670093557 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3b0006cc30 0x7f3b0006f0e0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c111520 tx=0x7f3b08009e30 comp rx=0 tx=0).stop 2024-03-30T04:58:28.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.299+0000 7f3b22e00700 1 -- 172.21.15.123:0/3670093557 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b1c102e00 msgr2=0x7f3b1c115c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:58:28.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.299+0000 7f3b22e00700 1 --2- 172.21.15.123:0/3670093557 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b1c102e00 0x7f3b1c115c80 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c00eb40 tx=0x7f3b0c00b040 comp rx=0 tx=0).stop 2024-03-30T04:58:28.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.299+0000 7f3b22e00700 1 -- 172.21.15.123:0/3670093557 shutdown_connections 2024-03-30T04:58:28.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.299+0000 7f3b22e00700 1 --2- 172.21.15.123:0/3670093557 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3b0006cc30 0x7f3b0006f0e0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:58:28.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.299+0000 7f3b22e00700 1 --2- 172.21.15.123:0/3670093557 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b1c102e00 0x7f3b1c115c80 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:58:28.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.299+0000 7f3b22e00700 1 --2- 172.21.15.123:0/3670093557 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b1c1037a0 0x7f3b1c1161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:58:28.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.299+0000 7f3b22e00700 1 -- 172.21.15.123:0/3670093557 >> 172.21.15.123:0/3670093557 conn(0x7f3b1c0f89b0 msgr2=0x7f3b1c0f99a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:58:28.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.299+0000 7f3b22e00700 1 -- 172.21.15.123:0/3670093557 shutdown_connections 2024-03-30T04:58:28.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.299+0000 7f3b22e00700 1 -- 172.21.15.123:0/3670093557 wait complete. 2024-03-30T04:58:28.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.433+0000 7f5c77e00700 1 -- 172.21.15.123:0/608764055 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c701037a0 msgr2=0x7f5c700fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:58:28.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.433+0000 7f5c77e00700 1 --2- 172.21.15.123:0/608764055 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c701037a0 0x7f5c700fce10 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c004940 tx=0x7f5c5c009e30 comp rx=0 tx=0).stop 2024-03-30T04:58:28.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.433+0000 7f5c77e00700 1 -- 172.21.15.123:0/608764055 shutdown_connections 2024-03-30T04:58:28.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.433+0000 7f5c77e00700 1 --2- 172.21.15.123:0/608764055 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c701037a0 0x7f5c700fce10 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:58:28.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.433+0000 7f5c77e00700 1 --2- 172.21.15.123:0/608764055 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5c70102e00 0x7f5c701031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:58:28.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.433+0000 7f5c77e00700 1 -- 172.21.15.123:0/608764055 >> 172.21.15.123:0/608764055 conn(0x7f5c700f89b0 msgr2=0x7f5c700fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:58:28.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.433+0000 7f5c77e00700 1 -- 172.21.15.123:0/608764055 shutdown_connections 2024-03-30T04:58:28.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.433+0000 7f5c77e00700 1 -- 172.21.15.123:0/608764055 wait complete. 2024-03-30T04:58:28.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.434+0000 7f5c77e00700 1 Processor -- start 2024-03-30T04:58:28.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.434+0000 7f5c77e00700 1 -- start start 2024-03-30T04:58:28.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.434+0000 7f5c77e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5c70102e00 0x7f5c700702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:58:28.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.434+0000 7f5c77e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c701037a0 0x7f5c70070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:58:28.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.434+0000 7f5c77e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c70070d90 con 0x7f5c701037a0 2024-03-30T04:58:28.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.435+0000 7f5c77e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c70070ed0 con 0x7f5c70102e00 2024-03-30T04:58:28.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.435+0000 7f5c74a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c701037a0 0x7f5c70070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:58:28.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.435+0000 7f5c74a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c701037a0 0x7f5c70070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54350/0 (socket says 172.21.15.123:54350) 2024-03-30T04:58:28.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.435+0000 7f5c74a00700 1 -- 172.21.15.123:0/731337192 learned_addr learned my addr 172.21.15.123:0/731337192 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:58:28.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.435+0000 7f5c75400700 1 --2- 172.21.15.123:0/731337192 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5c70102e00 0x7f5c700702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:58:28.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.435+0000 7f5c74a00700 1 -- 172.21.15.123:0/731337192 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5c70102e00 msgr2=0x7f5c700702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:58:28.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.435+0000 7f5c74a00700 1 --2- 172.21.15.123:0/731337192 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5c70102e00 0x7f5c700702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:58:28.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.435+0000 7f5c74a00700 1 -- 172.21.15.123:0/731337192 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c5c005040 con 0x7f5c701037a0 2024-03-30T04:58:28.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.435+0000 7f5c74a00700 1 --2- 172.21.15.123:0/731337192 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c701037a0 0x7f5c70070800 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c009390 tx=0x7f5c5c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:58:28.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.436+0000 7f5c6d600700 1 -- 172.21.15.123:0/731337192 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c5c00e040 con 0x7f5c701037a0 2024-03-30T04:58:28.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.436+0000 7f5c6d600700 1 -- 172.21.15.123:0/731337192 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5c5c013420 con 0x7f5c701037a0 2024-03-30T04:58:28.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.436+0000 7f5c77e00700 1 -- 172.21.15.123:0/731337192 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c70071120 con 0x7f5c701037a0 2024-03-30T04:58:28.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.436+0000 7f5c6d600700 1 -- 172.21.15.123:0/731337192 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c5c01c630 con 0x7f5c701037a0 2024-03-30T04:58:28.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.436+0000 7f5c77e00700 1 -- 172.21.15.123:0/731337192 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c7011a230 con 0x7f5c701037a0 2024-03-30T04:58:28.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.437+0000 7f5c77e00700 1 -- 172.21.15.123:0/731337192 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c700fe550 con 0x7f5c701037a0 2024-03-30T04:58:28.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.438+0000 7f5c6d600700 1 -- 172.21.15.123:0/731337192 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f5c5c013a00 con 0x7f5c701037a0 2024-03-30T04:58:28.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.439+0000 7f5c6d600700 1 --2- 172.21.15.123:0/731337192 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5c5806cc30 0x7f5c5806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:58:28.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.439+0000 7f5c6d600700 1 -- 172.21.15.123:0/731337192 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5c5c08d0e0 con 0x7f5c701037a0 2024-03-30T04:58:28.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.439+0000 7f5c75400700 1 --2- 172.21.15.123:0/731337192 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5c5806cc30 0x7f5c5806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:58:28.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.440+0000 7f5c75400700 1 --2- 172.21.15.123:0/731337192 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5c5806cc30 0x7f5c5806f0e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f5c64004f40 tx=0x7f5c6400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:58:28.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.443+0000 7f5c6d600700 1 -- 172.21.15.123:0/731337192 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5c5c057120 con 0x7f5c701037a0 2024-03-30T04:58:28.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.690+0000 7f5c77e00700 1 -- 172.21.15.123:0/731337192 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f5c70109dd0 con 0x7f5c701037a0 2024-03-30T04:58:28.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.691+0000 7f5c6d600700 1 -- 172.21.15.123:0/731337192 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 34 v34) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f5c5c05a740 con 0x7f5c701037a0 2024-03-30T04:58:28.689 INFO:teuthology.orchestra.run.smithi123.stdout:e34 2024-03-30T04:58:28.690 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:58:28.690 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:58:28.690 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:58:28.690 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:58:28.690 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:58:28.690 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:58:28.690 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T04:58:28.690 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T04:58:28.690 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:58:28.690 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T04:58:28.690 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:58:28.690 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:58:28.691 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:58:28.691 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:58:28.691 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:58:28.691 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T04:58:28.691 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:58:28.691 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:58:28.691 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T04:58:28.691 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:58:28.691 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T04:58:28.691 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T04:58:28.691 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T04:58:28.691 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T04:58:28.691 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:58:28.691 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:58:28.692 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:58:28.692 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:58:28.692 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:58:28.692 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:58:28.692 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T04:58:28.692 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T04:58:28.692 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:58:28.692 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:58:28.692 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T04:58:28.692 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:58:28.692 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:58:28.692 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:58:28.692 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:58:28.692 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44975} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/3788988014,v1:172.21.15.135:6827/3788988014] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:58:28.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.693+0000 7f5c77e00700 1 -- 172.21.15.123:0/731337192 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5c5806cc30 msgr2=0x7f5c5806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:58:28.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.693+0000 7f5c77e00700 1 --2- 172.21.15.123:0/731337192 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5c5806cc30 0x7f5c5806f0e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f5c64004f40 tx=0x7f5c6400b040 comp rx=0 tx=0).stop 2024-03-30T04:58:28.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.693+0000 7f5c77e00700 1 -- 172.21.15.123:0/731337192 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c701037a0 msgr2=0x7f5c70070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:58:28.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.693+0000 7f5c77e00700 1 --2- 172.21.15.123:0/731337192 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c701037a0 0x7f5c70070800 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c009390 tx=0x7f5c5c009e30 comp rx=0 tx=0).stop 2024-03-30T04:58:28.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.693+0000 7f5c77e00700 1 -- 172.21.15.123:0/731337192 shutdown_connections 2024-03-30T04:58:28.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.693+0000 7f5c77e00700 1 --2- 172.21.15.123:0/731337192 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5c5806cc30 0x7f5c5806f0e0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:58:28.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.693+0000 7f5c77e00700 1 --2- 172.21.15.123:0/731337192 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5c70102e00 0x7f5c700702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:58:28.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.693+0000 7f5c77e00700 1 --2- 172.21.15.123:0/731337192 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c701037a0 0x7f5c70070800 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:58:28.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.693+0000 7f5c77e00700 1 -- 172.21.15.123:0/731337192 >> 172.21.15.123:0/731337192 conn(0x7f5c700f89b0 msgr2=0x7f5c700f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:58:28.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.693+0000 7f5c77e00700 1 -- 172.21.15.123:0/731337192 shutdown_connections 2024-03-30T04:58:28.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.694+0000 7f5c77e00700 1 -- 172.21.15.123:0/731337192 wait complete. 2024-03-30T04:58:28.694 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 34 2024-03-30T04:58:28.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.828+0000 7f9752c00700 1 -- 172.21.15.123:0/231284650 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f974c102e00 msgr2=0x7f974c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:58:28.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.828+0000 7f9752c00700 1 --2- 172.21.15.123:0/231284650 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f974c102e00 0x7f974c1031d0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f973c004940 tx=0x7f973c009e30 comp rx=0 tx=0).stop 2024-03-30T04:58:28.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.828+0000 7f9752c00700 1 -- 172.21.15.123:0/231284650 shutdown_connections 2024-03-30T04:58:28.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.828+0000 7f9752c00700 1 --2- 172.21.15.123:0/231284650 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f974c1037a0 0x7f974c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:58:28.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.828+0000 7f9752c00700 1 --2- 172.21.15.123:0/231284650 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f974c102e00 0x7f974c1031d0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:58:28.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.828+0000 7f9752c00700 1 -- 172.21.15.123:0/231284650 >> 172.21.15.123:0/231284650 conn(0x7f974c0f89b0 msgr2=0x7f974c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:58:28.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.828+0000 7f9752c00700 1 -- 172.21.15.123:0/231284650 shutdown_connections 2024-03-30T04:58:28.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.828+0000 7f9752c00700 1 -- 172.21.15.123:0/231284650 wait complete. 2024-03-30T04:58:28.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.829+0000 7f9752c00700 1 Processor -- start 2024-03-30T04:58:28.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.829+0000 7f9752c00700 1 -- start start 2024-03-30T04:58:28.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.830+0000 7f9752c00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f974c102e00 0x7f974c115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:58:28.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.830+0000 7f9752c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f974c1037a0 0x7f974c116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:58:28.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.830+0000 7f9752c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f974c116880 con 0x7f974c1037a0 2024-03-30T04:58:28.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.830+0000 7f9752c00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f974c10fc50 con 0x7f974c102e00 2024-03-30T04:58:28.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.830+0000 7f974b400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f974c1037a0 0x7f974c116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:58:28.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.830+0000 7f974b400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f974c1037a0 0x7f974c116230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54370/0 (socket says 172.21.15.123:54370) 2024-03-30T04:58:28.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.830+0000 7f974b400700 1 -- 172.21.15.123:0/365838760 learned_addr learned my addr 172.21.15.123:0/365838760 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:58:28.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.830+0000 7f974be00700 1 --2- 172.21.15.123:0/365838760 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f974c102e00 0x7f974c115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:58:28.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.830+0000 7f974b400700 1 -- 172.21.15.123:0/365838760 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f974c102e00 msgr2=0x7f974c115cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:58:28.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.830+0000 7f974b400700 1 --2- 172.21.15.123:0/365838760 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f974c102e00 0x7f974c115cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:58:28.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.830+0000 7f974b400700 1 -- 172.21.15.123:0/365838760 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f973c005040 con 0x7f974c1037a0 2024-03-30T04:58:28.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.830+0000 7f974b400700 1 --2- 172.21.15.123:0/365838760 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f974c1037a0 0x7f974c116230 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f9738002a50 tx=0x7f9738004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:58:28.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.831+0000 7f9743e00700 1 -- 172.21.15.123:0/365838760 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9738007cc0 con 0x7f974c1037a0 2024-03-30T04:58:28.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.831+0000 7f9743e00700 1 -- 172.21.15.123:0/365838760 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f973800f040 con 0x7f974c1037a0 2024-03-30T04:58:28.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.831+0000 7f9752c00700 1 -- 172.21.15.123:0/365838760 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f974c10ff30 con 0x7f974c1037a0 2024-03-30T04:58:28.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.831+0000 7f9743e00700 1 -- 172.21.15.123:0/365838760 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f973800d920 con 0x7f974c1037a0 2024-03-30T04:58:28.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.831+0000 7f9752c00700 1 -- 172.21.15.123:0/365838760 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f974c1104f0 con 0x7f974c1037a0 2024-03-30T04:58:28.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.832+0000 7f9752c00700 1 -- 172.21.15.123:0/365838760 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f974c0fe550 con 0x7f974c1037a0 2024-03-30T04:58:28.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.833+0000 7f9743e00700 1 -- 172.21.15.123:0/365838760 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f9738009680 con 0x7f974c1037a0 2024-03-30T04:58:28.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.834+0000 7f9743e00700 1 --2- 172.21.15.123:0/365838760 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f973006cc30 0x7f973006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:58:28.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.834+0000 7f9743e00700 1 -- 172.21.15.123:0/365838760 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f973808bd00 con 0x7f974c1037a0 2024-03-30T04:58:28.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.835+0000 7f974be00700 1 --2- 172.21.15.123:0/365838760 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f973006cc30 0x7f973006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:58:28.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.835+0000 7f974be00700 1 --2- 172.21.15.123:0/365838760 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f973006cc30 0x7f973006f0e0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f973c009fd0 tx=0x7f973c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:58:28.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:28.839+0000 7f9743e00700 1 -- 172.21.15.123:0/365838760 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f97380597d0 con 0x7f974c1037a0 2024-03-30T04:58:29.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.061+0000 7f9752c00700 1 -- 172.21.15.123:0/365838760 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f974c109e30 con 0x7f973006cc30 2024-03-30T04:58:29.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.062+0000 7f9743e00700 1 -- 172.21.15.123:0/365838760 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f974c109e30 con 0x7f973006cc30 2024-03-30T04:58:29.061 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:58:29.061 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:58:29.061 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:58:29.061 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T04:58:29.061 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:58:29.061 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T04:58:29.061 INFO:teuthology.orchestra.run.smithi123.stdout: "mds", 2024-03-30T04:58:29.061 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T04:58:29.061 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T04:58:29.061 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T04:58:29.061 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:58:29.061 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "16/21 daemons upgraded", 2024-03-30T04:58:29.061 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T04:58:29.061 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:58:29.062 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:58:29.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.064+0000 7f9752c00700 1 -- 172.21.15.123:0/365838760 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f973006cc30 msgr2=0x7f973006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:58:29.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.064+0000 7f9752c00700 1 --2- 172.21.15.123:0/365838760 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f973006cc30 0x7f973006f0e0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f973c009fd0 tx=0x7f973c009e30 comp rx=0 tx=0).stop 2024-03-30T04:58:29.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.064+0000 7f9752c00700 1 -- 172.21.15.123:0/365838760 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f974c1037a0 msgr2=0x7f974c116230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:58:29.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.064+0000 7f9752c00700 1 --2- 172.21.15.123:0/365838760 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f974c1037a0 0x7f974c116230 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f9738002a50 tx=0x7f9738004600 comp rx=0 tx=0).stop 2024-03-30T04:58:29.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.065+0000 7f9752c00700 1 -- 172.21.15.123:0/365838760 shutdown_connections 2024-03-30T04:58:29.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.065+0000 7f9752c00700 1 --2- 172.21.15.123:0/365838760 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f973006cc30 0x7f973006f0e0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:58:29.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.065+0000 7f9752c00700 1 --2- 172.21.15.123:0/365838760 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f974c102e00 0x7f974c115cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:58:29.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.065+0000 7f9752c00700 1 --2- 172.21.15.123:0/365838760 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f974c1037a0 0x7f974c116230 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:58:29.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.065+0000 7f9752c00700 1 -- 172.21.15.123:0/365838760 >> 172.21.15.123:0/365838760 conn(0x7f974c0f89b0 msgr2=0x7f974c0706d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:58:29.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.065+0000 7f9752c00700 1 -- 172.21.15.123:0/365838760 shutdown_connections 2024-03-30T04:58:29.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.065+0000 7f9752c00700 1 -- 172.21.15.123:0/365838760 wait complete. 2024-03-30T04:58:29.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.198+0000 7fa556a00700 1 -- 172.21.15.123:0/1123262522 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa550102e00 msgr2=0x7fa5501031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:58:29.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.198+0000 7fa556a00700 1 --2- 172.21.15.123:0/1123262522 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa550102e00 0x7fa5501031d0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fa540004940 tx=0x7fa540009e30 comp rx=0 tx=0).stop 2024-03-30T04:58:29.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.198+0000 7fa556a00700 1 -- 172.21.15.123:0/1123262522 shutdown_connections 2024-03-30T04:58:29.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.198+0000 7fa556a00700 1 --2- 172.21.15.123:0/1123262522 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa5501037a0 0x7fa5500fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:58:29.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.198+0000 7fa556a00700 1 --2- 172.21.15.123:0/1123262522 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa550102e00 0x7fa5501031d0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:58:29.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.198+0000 7fa556a00700 1 -- 172.21.15.123:0/1123262522 >> 172.21.15.123:0/1123262522 conn(0x7fa5500f89b0 msgr2=0x7fa5500fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:58:29.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.198+0000 7fa556a00700 1 -- 172.21.15.123:0/1123262522 shutdown_connections 2024-03-30T04:58:29.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.199+0000 7fa556a00700 1 -- 172.21.15.123:0/1123262522 wait complete. 2024-03-30T04:58:29.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.199+0000 7fa556a00700 1 Processor -- start 2024-03-30T04:58:29.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.199+0000 7fa556a00700 1 -- start start 2024-03-30T04:58:29.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.200+0000 7fa556a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa550102e00 0x7fa550115c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:58:29.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.200+0000 7fa556a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa5501037a0 0x7fa550116180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:58:29.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.200+0000 7fa556a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5501167d0 con 0x7fa550102e00 2024-03-30T04:58:29.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.200+0000 7fa556a00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa55010fba0 con 0x7fa5501037a0 2024-03-30T04:58:29.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.200+0000 7fa54fe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa550102e00 0x7fa550115c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:58:29.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.200+0000 7fa54fe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa550102e00 0x7fa550115c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54378/0 (socket says 172.21.15.123:54378) 2024-03-30T04:58:29.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.200+0000 7fa54fe00700 1 -- 172.21.15.123:0/2244051048 learned_addr learned my addr 172.21.15.123:0/2244051048 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:58:29.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.200+0000 7fa54f400700 1 --2- 172.21.15.123:0/2244051048 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa5501037a0 0x7fa550116180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:58:29.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.200+0000 7fa54fe00700 1 -- 172.21.15.123:0/2244051048 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa5501037a0 msgr2=0x7fa550116180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:58:29.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.200+0000 7fa54fe00700 1 --2- 172.21.15.123:0/2244051048 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa5501037a0 0x7fa550116180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:58:29.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.200+0000 7fa54fe00700 1 -- 172.21.15.123:0/2244051048 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa540005040 con 0x7fa550102e00 2024-03-30T04:58:29.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.201+0000 7fa54f400700 1 --2- 172.21.15.123:0/2244051048 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa5501037a0 0x7fa550116180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:58:29.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.201+0000 7fa54fe00700 1 --2- 172.21.15.123:0/2244051048 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa550102e00 0x7fa550115c40 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fa540000c00 tx=0x7fa540009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:58:29.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.201+0000 7fa547e00700 1 -- 172.21.15.123:0/2244051048 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa540021070 con 0x7fa550102e00 2024-03-30T04:58:29.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.201+0000 7fa547e00700 1 -- 172.21.15.123:0/2244051048 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa540013de0 con 0x7fa550102e00 2024-03-30T04:58:29.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.201+0000 7fa547e00700 1 -- 172.21.15.123:0/2244051048 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa54000e070 con 0x7fa550102e00 2024-03-30T04:58:29.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.201+0000 7fa556a00700 1 -- 172.21.15.123:0/2244051048 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa55010fe20 con 0x7fa550102e00 2024-03-30T04:58:29.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.201+0000 7fa556a00700 1 -- 172.21.15.123:0/2244051048 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa550110380 con 0x7fa550102e00 2024-03-30T04:58:29.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.202+0000 7fa556a00700 1 -- 172.21.15.123:0/2244051048 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5500fe550 con 0x7fa550102e00 2024-03-30T04:58:29.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.203+0000 7fa547e00700 1 -- 172.21.15.123:0/2244051048 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fa54001c600 con 0x7fa550102e00 2024-03-30T04:58:29.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.204+0000 7fa547e00700 1 --2- 172.21.15.123:0/2244051048 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa53406cc20 0x7fa53406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:58:29.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.204+0000 7fa547e00700 1 -- 172.21.15.123:0/2244051048 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa54008d0c0 con 0x7fa550102e00 2024-03-30T04:58:29.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.204+0000 7fa54f400700 1 --2- 172.21.15.123:0/2244051048 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa53406cc20 0x7fa53406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:58:29.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.205+0000 7fa54f400700 1 --2- 172.21.15.123:0/2244051048 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa53406cc20 0x7fa53406f0d0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fa53c004600 tx=0x7fa53c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:58:29.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.208+0000 7fa547e00700 1 -- 172.21.15.123:0/2244051048 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa54005ab90 con 0x7fa550102e00 2024-03-30T04:58:29.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:58:28 smithi123 ceph-mon[139369]: from='client.35496 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:58:29.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:58:28 smithi123 ceph-mon[139369]: from='client.35500 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:58:29.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:58:28 smithi123 ceph-mon[139369]: pgmap v819: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:29.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:58:28 smithi123 ceph-mon[139369]: from='client.35504 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:58:29.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:58:28 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3670093557' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:58:29.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:58:28 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/731337192' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:58:29.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:58:28 smithi135 ceph-mon[106011]: from='client.35496 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:58:29.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:58:28 smithi135 ceph-mon[106011]: from='client.35500 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:58:29.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:58:28 smithi135 ceph-mon[106011]: pgmap v819: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:29.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:58:28 smithi135 ceph-mon[106011]: from='client.35504 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:58:29.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:58:28 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3670093557' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:58:29.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:58:28 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/731337192' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:58:29.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.497+0000 7fa556a00700 1 -- 172.21.15.123:0/2244051048 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fa55004bed0 con 0x7fa550102e00 2024-03-30T04:58:29.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.497+0000 7fa547e00700 1 -- 172.21.15.123:0/2244051048 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7fa54005a720 con 0x7fa550102e00 2024-03-30T04:58:29.496 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T04:58:29.496 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-30T04:58:29.496 INFO:teuthology.orchestra.run.smithi123.stdout: host smithi135 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi135/config 2024-03-30T04:58:29.496 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-30T04:58:29.496 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T04:58:29.496 INFO:teuthology.orchestra.run.smithi123.stdout:Traceback (most recent call last): 2024-03-30T04:58:29.496 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-30T04:58:29.496 INFO:teuthology.orchestra.run.smithi123.stdout: "__main__", mod_spec) 2024-03-30T04:58:29.496 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-30T04:58:29.497 INFO:teuthology.orchestra.run.smithi123.stdout: exec(code, run_globals) 2024-03-30T04:58:29.497 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10889, in 2024-03-30T04:58:29.497 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10877, in main 2024-03-30T04:58:29.497 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-30T04:58:29.497 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-30T04:58:29.497 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-30T04:58:29.497 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-30T04:58:29.497 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-03-30T04:58:29.497 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-30T04:58:29.497 INFO:teuthology.orchestra.run.smithi123.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T04:58:29.497 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T04:58:29.497 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T04:58:29.497 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T04:58:29.497 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T04:58:29.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.500+0000 7fa556a00700 1 -- 172.21.15.123:0/2244051048 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa53406cc20 msgr2=0x7fa53406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:58:29.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.500+0000 7fa556a00700 1 --2- 172.21.15.123:0/2244051048 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa53406cc20 0x7fa53406f0d0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fa53c004600 tx=0x7fa53c00b040 comp rx=0 tx=0).stop 2024-03-30T04:58:29.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.500+0000 7fa556a00700 1 -- 172.21.15.123:0/2244051048 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa550102e00 msgr2=0x7fa550115c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:58:29.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.500+0000 7fa556a00700 1 --2- 172.21.15.123:0/2244051048 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa550102e00 0x7fa550115c40 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fa540000c00 tx=0x7fa540009e30 comp rx=0 tx=0).stop 2024-03-30T04:58:29.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.500+0000 7fa556a00700 1 -- 172.21.15.123:0/2244051048 shutdown_connections 2024-03-30T04:58:29.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.500+0000 7fa556a00700 1 --2- 172.21.15.123:0/2244051048 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa53406cc20 0x7fa53406f0d0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:58:29.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.500+0000 7fa556a00700 1 --2- 172.21.15.123:0/2244051048 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa550102e00 0x7fa550115c40 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:58:29.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.500+0000 7fa556a00700 1 --2- 172.21.15.123:0/2244051048 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa5501037a0 0x7fa550116180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:58:29.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.500+0000 7fa556a00700 1 -- 172.21.15.123:0/2244051048 >> 172.21.15.123:0/2244051048 conn(0x7fa5500f89b0 msgr2=0x7fa5500f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:58:29.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.500+0000 7fa556a00700 1 -- 172.21.15.123:0/2244051048 shutdown_connections 2024-03-30T04:58:29.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:29.500+0000 7fa556a00700 1 -- 172.21.15.123:0/2244051048 wait complete. 2024-03-30T04:58:30.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:58:29 smithi123 ceph-mon[139369]: from='client.35516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:58:30.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:58:29 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:58:30.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:58:29 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2244051048' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:58:30.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:58:29 smithi135 ceph-mon[106011]: from='client.35516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:58:30.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:58:29 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:58:30.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:58:29 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2244051048' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:58:31.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:58:30 smithi123 ceph-mon[139369]: pgmap v820: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:31.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:58:30 smithi135 ceph-mon[106011]: pgmap v820: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:33.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:58:32 smithi123 ceph-mon[139369]: pgmap v821: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:33.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:58:32 smithi135 ceph-mon[106011]: pgmap v821: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:35.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:58:34 smithi123 ceph-mon[139369]: pgmap v822: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:35.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:58:34 smithi135 ceph-mon[106011]: pgmap v822: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:37.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:58:36 smithi123 ceph-mon[139369]: pgmap v823: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:37.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:58:36 smithi135 ceph-mon[106011]: pgmap v823: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:39.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:58:38 smithi123 ceph-mon[139369]: pgmap v824: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:39.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:58:38 smithi135 ceph-mon[106011]: pgmap v824: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:41.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:58:40 smithi123 ceph-mon[139369]: pgmap v825: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:41.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:58:40 smithi135 ceph-mon[106011]: pgmap v825: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:43.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:58:42 smithi123 ceph-mon[139369]: pgmap v826: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:43.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:58:42 smithi135 ceph-mon[106011]: pgmap v826: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:45.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:58:44 smithi123 ceph-mon[139369]: pgmap v827: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:45.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:58:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:58:45.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:58:44 smithi135 ceph-mon[106011]: pgmap v827: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:45.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:58:44 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:58:47.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:58:46 smithi123 ceph-mon[139369]: pgmap v828: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:47.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:58:46 smithi135 ceph-mon[106011]: pgmap v828: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:49.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:58:48 smithi123 ceph-mon[139369]: pgmap v829: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:49.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:58:48 smithi135 ceph-mon[106011]: pgmap v829: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:51.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:58:50 smithi123 ceph-mon[139369]: pgmap v830: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:51.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:58:50 smithi135 ceph-mon[106011]: pgmap v830: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:53.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:58:52 smithi123 ceph-mon[139369]: pgmap v831: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:53.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:58:52 smithi135 ceph-mon[106011]: pgmap v831: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:55.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:58:54 smithi135 ceph-mon[106011]: pgmap v832: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:55.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:58:54 smithi123 ceph-mon[139369]: pgmap v832: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:57.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:58:56 smithi135 ceph-mon[106011]: pgmap v833: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:57.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:58:56 smithi123 ceph-mon[139369]: pgmap v833: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:59.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:58:58 smithi135 ceph-mon[106011]: pgmap v834: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:59.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:58:58 smithi123 ceph-mon[139369]: pgmap v834: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:58:59.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:59.655+0000 7f91f6e00700 1 -- 172.21.15.123:0/611275289 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91f00fe830 msgr2=0x7f91f0106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:58:59.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:59.655+0000 7f91f6e00700 1 --2- 172.21.15.123:0/611275289 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91f00fe830 0x7f91f0106f00 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f91dc004940 tx=0x7f91dc009e30 comp rx=0 tx=0).stop 2024-03-30T04:58:59.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:59.655+0000 7f91f6e00700 1 -- 172.21.15.123:0/611275289 shutdown_connections 2024-03-30T04:58:59.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:59.655+0000 7f91f6e00700 1 --2- 172.21.15.123:0/611275289 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91f00fe830 0x7f91f0106f00 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:58:59.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:59.655+0000 7f91f6e00700 1 --2- 172.21.15.123:0/611275289 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91f00fde90 0x7f91f00fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:58:59.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:59.655+0000 7f91f6e00700 1 -- 172.21.15.123:0/611275289 >> 172.21.15.123:0/611275289 conn(0x7f91f0072bc0 msgr2=0x7f91f0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:58:59.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:59.655+0000 7f91f6e00700 1 -- 172.21.15.123:0/611275289 shutdown_connections 2024-03-30T04:58:59.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:59.655+0000 7f91f6e00700 1 -- 172.21.15.123:0/611275289 wait complete. 2024-03-30T04:58:59.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:59.656+0000 7f91f6e00700 1 Processor -- start 2024-03-30T04:58:59.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:59.656+0000 7f91f6e00700 1 -- start start 2024-03-30T04:58:59.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:59.657+0000 7f91f6e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91f00fde90 0x7f91f0115d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:58:59.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:59.657+0000 7f91f6e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91f00fe830 0x7f91f0116280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:58:59.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:59.657+0000 7f91f6e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f0116960 con 0x7f91f00fde90 2024-03-30T04:58:59.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:59.657+0000 7f91f6e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f010fca0 con 0x7f91f00fe830 2024-03-30T04:58:59.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:59.657+0000 7f91efe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91f00fde90 0x7f91f0115d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:58:59.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:59.657+0000 7f91efe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91f00fde90 0x7f91f0115d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:59882/0 (socket says 172.21.15.123:59882) 2024-03-30T04:58:59.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:59.657+0000 7f91efe00700 1 -- 172.21.15.123:0/4019741210 learned_addr learned my addr 172.21.15.123:0/4019741210 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:58:59.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:59.657+0000 7f91ef400700 1 --2- 172.21.15.123:0/4019741210 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91f00fe830 0x7f91f0116280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:58:59.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:59.657+0000 7f91efe00700 1 -- 172.21.15.123:0/4019741210 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91f00fe830 msgr2=0x7f91f0116280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:58:59.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:59.657+0000 7f91efe00700 1 --2- 172.21.15.123:0/4019741210 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91f00fe830 0x7f91f0116280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:58:59.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:59.657+0000 7f91efe00700 1 -- 172.21.15.123:0/4019741210 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91dc005040 con 0x7f91f00fde90 2024-03-30T04:58:59.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:59.658+0000 7f91efe00700 1 --2- 172.21.15.123:0/4019741210 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91f00fde90 0x7f91f0115d40 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f91e000b790 tx=0x7f91e0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:58:59.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:59.658+0000 7f91e7e00700 1 -- 172.21.15.123:0/4019741210 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f91e000f6f0 con 0x7f91f00fde90 2024-03-30T04:58:59.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:59.658+0000 7f91e7e00700 1 -- 172.21.15.123:0/4019741210 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f91e000fd00 con 0x7f91f00fde90 2024-03-30T04:58:59.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:59.658+0000 7f91f6e00700 1 -- 172.21.15.123:0/4019741210 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91f010ff20 con 0x7f91f00fde90 2024-03-30T04:58:59.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:59.658+0000 7f91e7e00700 1 -- 172.21.15.123:0/4019741210 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f91e00103a0 con 0x7f91f00fde90 2024-03-30T04:58:59.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:59.658+0000 7f91f6e00700 1 -- 172.21.15.123:0/4019741210 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91f0110460 con 0x7f91f00fde90 2024-03-30T04:58:59.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:59.659+0000 7f91f6e00700 1 -- 172.21.15.123:0/4019741210 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91f0106a20 con 0x7f91f00fde90 2024-03-30T04:58:59.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:59.660+0000 7f91e7e00700 1 -- 172.21.15.123:0/4019741210 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f91e000f850 con 0x7f91f00fde90 2024-03-30T04:58:59.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:59.662+0000 7f91e7e00700 1 --2- 172.21.15.123:0/4019741210 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f91d406cc20 0x7f91d406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:58:59.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:59.662+0000 7f91e7e00700 1 -- 172.21.15.123:0/4019741210 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f91e008beb0 con 0x7f91f00fde90 2024-03-30T04:58:59.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:59.662+0000 7f91ef400700 1 --2- 172.21.15.123:0/4019741210 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f91d406cc20 0x7f91d406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:58:59.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:59.662+0000 7f91ef400700 1 --2- 172.21.15.123:0/4019741210 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f91d406cc20 0x7f91d406f0d0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f91dc00b7e0 tx=0x7f91dc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:58:59.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:59.667+0000 7f91e7e00700 1 -- 172.21.15.123:0/4019741210 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f91e0059980 con 0x7f91f00fde90 2024-03-30T04:58:59.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:59.888+0000 7f91f6e00700 1 -- 172.21.15.123:0/4019741210 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f91f0109eb0 con 0x7f91d406cc20 2024-03-30T04:58:59.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:59.889+0000 7f91e7e00700 1 -- 172.21.15.123:0/4019741210 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f91f0109eb0 con 0x7f91d406cc20 2024-03-30T04:58:59.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:59.891+0000 7f91f6e00700 1 -- 172.21.15.123:0/4019741210 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f91d406cc20 msgr2=0x7f91d406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:58:59.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:59.891+0000 7f91f6e00700 1 --2- 172.21.15.123:0/4019741210 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f91d406cc20 0x7f91d406f0d0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f91dc00b7e0 tx=0x7f91dc009e30 comp rx=0 tx=0).stop 2024-03-30T04:58:59.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:59.891+0000 7f91f6e00700 1 -- 172.21.15.123:0/4019741210 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91f00fde90 msgr2=0x7f91f0115d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:58:59.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:59.891+0000 7f91f6e00700 1 --2- 172.21.15.123:0/4019741210 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91f00fde90 0x7f91f0115d40 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f91e000b790 tx=0x7f91e0009250 comp rx=0 tx=0).stop 2024-03-30T04:58:59.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:59.891+0000 7f91f6e00700 1 -- 172.21.15.123:0/4019741210 shutdown_connections 2024-03-30T04:58:59.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:59.891+0000 7f91f6e00700 1 --2- 172.21.15.123:0/4019741210 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f91d406cc20 0x7f91d406f0d0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:58:59.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:59.891+0000 7f91f6e00700 1 --2- 172.21.15.123:0/4019741210 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91f00fde90 0x7f91f0115d40 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:58:59.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:59.891+0000 7f91f6e00700 1 --2- 172.21.15.123:0/4019741210 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91f00fe830 0x7f91f0116280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:58:59.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:59.891+0000 7f91f6e00700 1 -- 172.21.15.123:0/4019741210 >> 172.21.15.123:0/4019741210 conn(0x7f91f0072bc0 msgr2=0x7f91f00fb740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:58:59.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:59.892+0000 7f91f6e00700 1 -- 172.21.15.123:0/4019741210 shutdown_connections 2024-03-30T04:58:59.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:58:59.892+0000 7f91f6e00700 1 -- 172.21.15.123:0/4019741210 wait complete. 2024-03-30T04:58:59.904 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:59:00.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.030+0000 7f3606000700 1 -- 172.21.15.123:0/1619623847 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3600070360 msgr2=0x7f3600070730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:59:00.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.030+0000 7f3606000700 1 --2- 172.21.15.123:0/1619623847 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3600070360 0x7f3600070730 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f35f0004970 tx=0x7f35f0009e30 comp rx=0 tx=0).stop 2024-03-30T04:59:00.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.030+0000 7f3606000700 1 -- 172.21.15.123:0/1619623847 shutdown_connections 2024-03-30T04:59:00.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.030+0000 7f3606000700 1 --2- 172.21.15.123:0/1619623847 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3600070c70 0x7f3600109110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:00.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.030+0000 7f3606000700 1 --2- 172.21.15.123:0/1619623847 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3600070360 0x7f3600070730 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:00.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.030+0000 7f3606000700 1 -- 172.21.15.123:0/1619623847 >> 172.21.15.123:0/1619623847 conn(0x7f36000f89b0 msgr2=0x7f36000fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:59:00.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.030+0000 7f3606000700 1 -- 172.21.15.123:0/1619623847 shutdown_connections 2024-03-30T04:59:00.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.030+0000 7f3606000700 1 -- 172.21.15.123:0/1619623847 wait complete. 2024-03-30T04:59:00.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.031+0000 7f3606000700 1 Processor -- start 2024-03-30T04:59:00.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.031+0000 7f3606000700 1 -- start start 2024-03-30T04:59:00.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.031+0000 7f3606000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3600070360 0x7f3600115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:59:00.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.031+0000 7f3606000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3600070c70 0x7f36001161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:59:00.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.031+0000 7f3606000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3600116810 con 0x7f3600070c70 2024-03-30T04:59:00.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.031+0000 7f3606000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f360010fbe0 con 0x7f3600070360 2024-03-30T04:59:00.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.031+0000 7f35fea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3600070c70 0x7f36001161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:59:00.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.032+0000 7f35fea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3600070c70 0x7f36001161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:59902/0 (socket says 172.21.15.123:59902) 2024-03-30T04:59:00.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.032+0000 7f35fea00700 1 -- 172.21.15.123:0/612864916 learned_addr learned my addr 172.21.15.123:0/612864916 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:59:00.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.032+0000 7f35ff400700 1 --2- 172.21.15.123:0/612864916 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3600070360 0x7f3600115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:59:00.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.032+0000 7f35fea00700 1 -- 172.21.15.123:0/612864916 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3600070360 msgr2=0x7f3600115c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:59:00.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.032+0000 7f35fea00700 1 --2- 172.21.15.123:0/612864916 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3600070360 0x7f3600115c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:00.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.032+0000 7f35fea00700 1 -- 172.21.15.123:0/612864916 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f0005040 con 0x7f3600070c70 2024-03-30T04:59:00.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.032+0000 7f35fea00700 1 --2- 172.21.15.123:0/612864916 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3600070c70 0x7f36001161c0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f35ec002a50 tx=0x7f35ec0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:59:00.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.032+0000 7f35f7400700 1 -- 172.21.15.123:0/612864916 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f35ec007cc0 con 0x7f3600070c70 2024-03-30T04:59:00.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.032+0000 7f35f7400700 1 -- 172.21.15.123:0/612864916 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f35ec00f040 con 0x7f3600070c70 2024-03-30T04:59:00.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.032+0000 7f35f7400700 1 -- 172.21.15.123:0/612864916 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f35ec009540 con 0x7f3600070c70 2024-03-30T04:59:00.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.032+0000 7f3606000700 1 -- 172.21.15.123:0/612864916 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f360010fec0 con 0x7f3600070c70 2024-03-30T04:59:00.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.032+0000 7f3606000700 1 -- 172.21.15.123:0/612864916 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3600110400 con 0x7f3600070c70 2024-03-30T04:59:00.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.034+0000 7f3606000700 1 -- 172.21.15.123:0/612864916 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3600103b90 con 0x7f3600070c70 2024-03-30T04:59:00.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.034+0000 7f35f7400700 1 -- 172.21.15.123:0/612864916 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f35ec0096a0 con 0x7f3600070c70 2024-03-30T04:59:00.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.035+0000 7f35f7400700 1 --2- 172.21.15.123:0/612864916 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f35e406cc20 0x7f35e406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:59:00.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.035+0000 7f35f7400700 1 -- 172.21.15.123:0/612864916 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f35ec08bee0 con 0x7f3600070c70 2024-03-30T04:59:00.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.036+0000 7f35ff400700 1 --2- 172.21.15.123:0/612864916 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f35e406cc20 0x7f35e406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:59:00.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.036+0000 7f35ff400700 1 --2- 172.21.15.123:0/612864916 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f35e406cc20 0x7f35e406f0d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f35f0004f50 tx=0x7f35f0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:59:00.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.040+0000 7f35f7400700 1 -- 172.21.15.123:0/612864916 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f35ec0598b0 con 0x7f3600070c70 2024-03-30T04:59:00.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.262+0000 7f3606000700 1 -- 172.21.15.123:0/612864916 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f360010bfb0 con 0x7f35e406cc20 2024-03-30T04:59:00.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.263+0000 7f35f7400700 1 -- 172.21.15.123:0/612864916 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f360010bfb0 con 0x7f35e406cc20 2024-03-30T04:59:00.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.265+0000 7f3606000700 1 -- 172.21.15.123:0/612864916 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f35e406cc20 msgr2=0x7f35e406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:59:00.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.265+0000 7f3606000700 1 --2- 172.21.15.123:0/612864916 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f35e406cc20 0x7f35e406f0d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f35f0004f50 tx=0x7f35f0009e30 comp rx=0 tx=0).stop 2024-03-30T04:59:00.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.265+0000 7f3606000700 1 -- 172.21.15.123:0/612864916 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3600070c70 msgr2=0x7f36001161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:59:00.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.265+0000 7f3606000700 1 --2- 172.21.15.123:0/612864916 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3600070c70 0x7f36001161c0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f35ec002a50 tx=0x7f35ec0046b0 comp rx=0 tx=0).stop 2024-03-30T04:59:00.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.265+0000 7f3606000700 1 -- 172.21.15.123:0/612864916 shutdown_connections 2024-03-30T04:59:00.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.265+0000 7f3606000700 1 --2- 172.21.15.123:0/612864916 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f35e406cc20 0x7f35e406f0d0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:00.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.265+0000 7f3606000700 1 --2- 172.21.15.123:0/612864916 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3600070360 0x7f3600115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:00.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.265+0000 7f3606000700 1 --2- 172.21.15.123:0/612864916 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3600070c70 0x7f36001161c0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:00.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.266+0000 7f3606000700 1 -- 172.21.15.123:0/612864916 >> 172.21.15.123:0/612864916 conn(0x7f36000f89b0 msgr2=0x7f36001015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:59:00.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.266+0000 7f3606000700 1 -- 172.21.15.123:0/612864916 shutdown_connections 2024-03-30T04:59:00.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.266+0000 7f3606000700 1 -- 172.21.15.123:0/612864916 wait complete. 2024-03-30T04:59:00.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:58:59 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:59:00.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.398+0000 7f9430000700 1 -- 172.21.15.123:0/4115443865 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9428102e00 msgr2=0x7f94281031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:59:00.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.398+0000 7f9430000700 1 --2- 172.21.15.123:0/4115443865 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9428102e00 0x7f94281031d0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f941c004940 tx=0x7f941c009e30 comp rx=0 tx=0).stop 2024-03-30T04:59:00.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.398+0000 7f9430000700 1 -- 172.21.15.123:0/4115443865 shutdown_connections 2024-03-30T04:59:00.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.398+0000 7f9430000700 1 --2- 172.21.15.123:0/4115443865 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f94281037a0 0x7f94280fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:00.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.398+0000 7f9430000700 1 --2- 172.21.15.123:0/4115443865 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9428102e00 0x7f94281031d0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:00.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.398+0000 7f9430000700 1 -- 172.21.15.123:0/4115443865 >> 172.21.15.123:0/4115443865 conn(0x7f94280f89b0 msgr2=0x7f94280fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:59:00.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.398+0000 7f9430000700 1 -- 172.21.15.123:0/4115443865 shutdown_connections 2024-03-30T04:59:00.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.398+0000 7f9430000700 1 -- 172.21.15.123:0/4115443865 wait complete. 2024-03-30T04:59:00.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.399+0000 7f9430000700 1 Processor -- start 2024-03-30T04:59:00.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.399+0000 7f9430000700 1 -- start start 2024-03-30T04:59:00.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.400+0000 7f9430000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9428102e00 0x7f9428115cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:59:00.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.400+0000 7f9430000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f94281037a0 0x7f9428116200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:59:00.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.400+0000 7f9430000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94281168e0 con 0x7f9428102e00 2024-03-30T04:59:00.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.400+0000 7f9430000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f942810fc20 con 0x7f94281037a0 2024-03-30T04:59:00.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.400+0000 7f942d600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9428102e00 0x7f9428115cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:59:00.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.400+0000 7f942d600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9428102e00 0x7f9428115cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:59922/0 (socket says 172.21.15.123:59922) 2024-03-30T04:59:00.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.400+0000 7f942d600700 1 -- 172.21.15.123:0/3267684061 learned_addr learned my addr 172.21.15.123:0/3267684061 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:59:00.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.400+0000 7f942cc00700 1 --2- 172.21.15.123:0/3267684061 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f94281037a0 0x7f9428116200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:59:00.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.400+0000 7f942d600700 1 -- 172.21.15.123:0/3267684061 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f94281037a0 msgr2=0x7f9428116200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:59:00.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.400+0000 7f942d600700 1 --2- 172.21.15.123:0/3267684061 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f94281037a0 0x7f9428116200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:00.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.400+0000 7f942d600700 1 -- 172.21.15.123:0/3267684061 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f941c005040 con 0x7f9428102e00 2024-03-30T04:59:00.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.400+0000 7f942cc00700 1 --2- 172.21.15.123:0/3267684061 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f94281037a0 0x7f9428116200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:59:00.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.400+0000 7f942d600700 1 --2- 172.21.15.123:0/3267684061 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9428102e00 0x7f9428115cc0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f941c009390 tx=0x7f941c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:59:00.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.401+0000 7f9425600700 1 -- 172.21.15.123:0/3267684061 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f941c00e040 con 0x7f9428102e00 2024-03-30T04:59:00.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.401+0000 7f9425600700 1 -- 172.21.15.123:0/3267684061 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f941c013420 con 0x7f9428102e00 2024-03-30T04:59:00.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.401+0000 7f9425600700 1 -- 172.21.15.123:0/3267684061 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f941c01c630 con 0x7f9428102e00 2024-03-30T04:59:00.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.401+0000 7f9430000700 1 -- 172.21.15.123:0/3267684061 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f942810fe40 con 0x7f9428102e00 2024-03-30T04:59:00.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.401+0000 7f9430000700 1 -- 172.21.15.123:0/3267684061 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9428110400 con 0x7f9428102e00 2024-03-30T04:59:00.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.402+0000 7f9430000700 1 -- 172.21.15.123:0/3267684061 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94280fe550 con 0x7f9428102e00 2024-03-30T04:59:00.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.403+0000 7f9425600700 1 -- 172.21.15.123:0/3267684061 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f941c0139e0 con 0x7f9428102e00 2024-03-30T04:59:00.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.404+0000 7f9425600700 1 --2- 172.21.15.123:0/3267684061 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f941006cc30 0x7f941006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:59:00.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.404+0000 7f9425600700 1 -- 172.21.15.123:0/3267684061 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f941c08d250 con 0x7f9428102e00 2024-03-30T04:59:00.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.404+0000 7f942cc00700 1 --2- 172.21.15.123:0/3267684061 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f941006cc30 0x7f941006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:59:00.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.404+0000 7f942cc00700 1 --2- 172.21.15.123:0/3267684061 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f941006cc30 0x7f941006f0e0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f9428111560 tx=0x7f9414008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:59:00.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.408+0000 7f9425600700 1 -- 172.21.15.123:0/3267684061 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f941c05add0 con 0x7f9428102e00 2024-03-30T04:59:00.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:58:59 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:59:00.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.621+0000 7f9430000700 1 -- 172.21.15.123:0/3267684061 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f942804bf70 con 0x7f941006cc30 2024-03-30T04:59:00.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.631+0000 7f9425600700 1 -- 172.21.15.123:0/3267684061 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f9414009790 con 0x7f941006cc30 2024-03-30T04:59:00.629 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:59:00.629 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (28m) 5m ago 40m 14.6M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:59:00.630 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (25m) 5m ago 40m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T04:59:00.630 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (25m) 5m ago 39m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T04:59:00.630 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (27m) 5m ago 40m 73.5M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T04:59:00.630 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (21m) 5m ago 34m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T04:59:00.630 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (21m) 5m ago 34m 16.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T04:59:00.630 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (5m) 5m ago 34m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 87b073a6c9ca 2024-03-30T04:59:00.630 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (20m) 5m ago 34m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T04:59:00.630 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (30m) 5m ago 42m 510M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:59:00.630 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (29m) 5m ago 39m 456M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:59:00.630 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (26m) 5m ago 42m 70.0M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T04:59:00.630 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (26m) 5m ago 39m 51.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T04:59:00.630 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (28m) 5m ago 40m 8636k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:59:00.630 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (28m) 5m ago 39m 8678k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:59:00.631 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (24m) 5m ago 38m 107M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T04:59:00.631 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (24m) 5m ago 38m 97.2M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T04:59:00.631 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (23m) 5m ago 37m 89.8M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T04:59:00.631 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (23m) 5m ago 37m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T04:59:00.631 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (22m) 5m ago 36m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T04:59:00.631 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (21m) 5m ago 36m 88.8M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T04:59:00.631 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (28m) 5m ago 40m 63.9M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:59:00.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.633+0000 7f9430000700 1 -- 172.21.15.123:0/3267684061 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f941006cc30 msgr2=0x7f941006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:59:00.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.633+0000 7f9430000700 1 --2- 172.21.15.123:0/3267684061 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f941006cc30 0x7f941006f0e0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f9428111560 tx=0x7f9414008040 comp rx=0 tx=0).stop 2024-03-30T04:59:00.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.633+0000 7f9430000700 1 -- 172.21.15.123:0/3267684061 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9428102e00 msgr2=0x7f9428115cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:59:00.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.633+0000 7f9430000700 1 --2- 172.21.15.123:0/3267684061 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9428102e00 0x7f9428115cc0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f941c009390 tx=0x7f941c009e30 comp rx=0 tx=0).stop 2024-03-30T04:59:00.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.633+0000 7f9430000700 1 -- 172.21.15.123:0/3267684061 shutdown_connections 2024-03-30T04:59:00.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.634+0000 7f9430000700 1 --2- 172.21.15.123:0/3267684061 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f941006cc30 0x7f941006f0e0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:00.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.634+0000 7f9430000700 1 --2- 172.21.15.123:0/3267684061 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9428102e00 0x7f9428115cc0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:00.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.634+0000 7f9430000700 1 --2- 172.21.15.123:0/3267684061 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f94281037a0 0x7f9428116200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:00.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.634+0000 7f9430000700 1 -- 172.21.15.123:0/3267684061 >> 172.21.15.123:0/3267684061 conn(0x7f94280f89b0 msgr2=0x7f9428071ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:59:00.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.634+0000 7f9430000700 1 -- 172.21.15.123:0/3267684061 shutdown_connections 2024-03-30T04:59:00.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.634+0000 7f9430000700 1 -- 172.21.15.123:0/3267684061 wait complete. 2024-03-30T04:59:00.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.766+0000 7f94fc400700 1 -- 172.21.15.123:0/2306874787 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f94f4102e00 msgr2=0x7f94f41031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:59:00.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.766+0000 7f94fc400700 1 --2- 172.21.15.123:0/2306874787 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f94f4102e00 0x7f94f41031d0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f94e8004940 tx=0x7f94e8009e30 comp rx=0 tx=0).stop 2024-03-30T04:59:00.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.766+0000 7f94fc400700 1 -- 172.21.15.123:0/2306874787 shutdown_connections 2024-03-30T04:59:00.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.766+0000 7f94fc400700 1 --2- 172.21.15.123:0/2306874787 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f94f41037a0 0x7f94f40fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:00.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.766+0000 7f94fc400700 1 --2- 172.21.15.123:0/2306874787 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f94f4102e00 0x7f94f41031d0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:00.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.766+0000 7f94fc400700 1 -- 172.21.15.123:0/2306874787 >> 172.21.15.123:0/2306874787 conn(0x7f94f40f89b0 msgr2=0x7f94f40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:59:00.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.767+0000 7f94fc400700 1 -- 172.21.15.123:0/2306874787 shutdown_connections 2024-03-30T04:59:00.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.767+0000 7f94fc400700 1 -- 172.21.15.123:0/2306874787 wait complete. 2024-03-30T04:59:00.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.767+0000 7f94fc400700 1 Processor -- start 2024-03-30T04:59:00.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.767+0000 7f94fc400700 1 -- start start 2024-03-30T04:59:00.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.768+0000 7f94fc400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f94f4102e00 0x7f94f40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:59:00.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.768+0000 7f94fc400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f94f41037a0 0x7f94f4070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:59:00.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.768+0000 7f94fc400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f4070d90 con 0x7f94f4102e00 2024-03-30T04:59:00.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.768+0000 7f94fc400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f4070ed0 con 0x7f94f41037a0 2024-03-30T04:59:00.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.768+0000 7f94f9a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f94f4102e00 0x7f94f40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:59:00.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.768+0000 7f94f9a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f94f4102e00 0x7f94f40702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:59940/0 (socket says 172.21.15.123:59940) 2024-03-30T04:59:00.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.768+0000 7f94f9a00700 1 -- 172.21.15.123:0/2541596299 learned_addr learned my addr 172.21.15.123:0/2541596299 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:59:00.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.768+0000 7f94f9000700 1 --2- 172.21.15.123:0/2541596299 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f94f41037a0 0x7f94f4070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:59:00.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.768+0000 7f94f9a00700 1 -- 172.21.15.123:0/2541596299 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f94f41037a0 msgr2=0x7f94f4070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:59:00.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.768+0000 7f94f9a00700 1 --2- 172.21.15.123:0/2541596299 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f94f41037a0 0x7f94f4070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:00.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.768+0000 7f94f9a00700 1 -- 172.21.15.123:0/2541596299 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94e8005040 con 0x7f94f4102e00 2024-03-30T04:59:00.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.769+0000 7f94f9a00700 1 --2- 172.21.15.123:0/2541596299 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f94f4102e00 0x7f94f40702c0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f94e8000c00 tx=0x7f94e8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:59:00.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.769+0000 7f94f1600700 1 -- 172.21.15.123:0/2541596299 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f94e8021070 con 0x7f94f4102e00 2024-03-30T04:59:00.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.769+0000 7f94f1600700 1 -- 172.21.15.123:0/2541596299 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f94e8013de0 con 0x7f94f4102e00 2024-03-30T04:59:00.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.769+0000 7f94fc400700 1 -- 172.21.15.123:0/2541596299 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94f4071120 con 0x7f94f4102e00 2024-03-30T04:59:00.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.769+0000 7f94f1600700 1 -- 172.21.15.123:0/2541596299 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f94e800e070 con 0x7f94f4102e00 2024-03-30T04:59:00.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.769+0000 7f94fc400700 1 -- 172.21.15.123:0/2541596299 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94f411a110 con 0x7f94f4102e00 2024-03-30T04:59:00.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.770+0000 7f94fc400700 1 -- 172.21.15.123:0/2541596299 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94f40fe550 con 0x7f94f4102e00 2024-03-30T04:59:00.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.771+0000 7f94f1600700 1 -- 172.21.15.123:0/2541596299 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f94e801c600 con 0x7f94f4102e00 2024-03-30T04:59:00.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.772+0000 7f94f1600700 1 --2- 172.21.15.123:0/2541596299 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f94dc06cc20 0x7f94dc06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:59:00.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.772+0000 7f94f1600700 1 -- 172.21.15.123:0/2541596299 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f94e808d1a0 con 0x7f94f4102e00 2024-03-30T04:59:00.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.772+0000 7f94f9000700 1 --2- 172.21.15.123:0/2541596299 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f94dc06cc20 0x7f94dc06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:59:00.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.773+0000 7f94f9000700 1 --2- 172.21.15.123:0/2541596299 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f94dc06cc20 0x7f94dc06f0d0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f94e0004600 tx=0x7f94e000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:59:00.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:00.776+0000 7f94f1600700 1 -- 172.21.15.123:0/2541596299 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f94e8057160 con 0x7f94f4102e00 2024-03-30T04:59:01.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.058+0000 7f94fc400700 1 -- 172.21.15.123:0/2541596299 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f94f40719b0 con 0x7f94f4102e00 2024-03-30T04:59:01.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.059+0000 7f94f1600700 1 -- 172.21.15.123:0/2541596299 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f94e805a780 con 0x7f94f4102e00 2024-03-30T04:59:01.057 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:59:01.057 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:59:01.057 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:59:01.057 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:59:01.057 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:59:01.057 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:59:01.058 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:59:01.058 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:59:01.058 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T04:59:01.058 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:59:01.058 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:59:01.058 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 4 2024-03-30T04:59:01.058 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:59:01.058 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:59:01.058 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 14 2024-03-30T04:59:01.058 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:59:01.058 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:59:01.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.061+0000 7f94fc400700 1 -- 172.21.15.123:0/2541596299 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f94dc06cc20 msgr2=0x7f94dc06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:59:01.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.061+0000 7f94fc400700 1 --2- 172.21.15.123:0/2541596299 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f94dc06cc20 0x7f94dc06f0d0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f94e0004600 tx=0x7f94e000b040 comp rx=0 tx=0).stop 2024-03-30T04:59:01.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.061+0000 7f94fc400700 1 -- 172.21.15.123:0/2541596299 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f94f4102e00 msgr2=0x7f94f40702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:59:01.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.061+0000 7f94fc400700 1 --2- 172.21.15.123:0/2541596299 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f94f4102e00 0x7f94f40702c0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f94e8000c00 tx=0x7f94e8009e30 comp rx=0 tx=0).stop 2024-03-30T04:59:01.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.061+0000 7f94fc400700 1 -- 172.21.15.123:0/2541596299 shutdown_connections 2024-03-30T04:59:01.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.061+0000 7f94fc400700 1 --2- 172.21.15.123:0/2541596299 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f94dc06cc20 0x7f94dc06f0d0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:01.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.061+0000 7f94fc400700 1 --2- 172.21.15.123:0/2541596299 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f94f4102e00 0x7f94f40702c0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:01.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.061+0000 7f94fc400700 1 --2- 172.21.15.123:0/2541596299 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f94f41037a0 0x7f94f4070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:01.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.061+0000 7f94fc400700 1 -- 172.21.15.123:0/2541596299 >> 172.21.15.123:0/2541596299 conn(0x7f94f40f89b0 msgr2=0x7f94f40f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:59:01.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.061+0000 7f94fc400700 1 -- 172.21.15.123:0/2541596299 shutdown_connections 2024-03-30T04:59:01.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.061+0000 7f94fc400700 1 -- 172.21.15.123:0/2541596299 wait complete. 2024-03-30T04:59:01.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.196+0000 7f843b400700 1 -- 172.21.15.123:0/2989283247 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84341037a0 msgr2=0x7f84340fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:59:01.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.196+0000 7f843b400700 1 --2- 172.21.15.123:0/2989283247 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84341037a0 0x7f84340fce10 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f8420004940 tx=0x7f8420009e30 comp rx=0 tx=0).stop 2024-03-30T04:59:01.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.196+0000 7f843b400700 1 -- 172.21.15.123:0/2989283247 shutdown_connections 2024-03-30T04:59:01.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.196+0000 7f843b400700 1 --2- 172.21.15.123:0/2989283247 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84341037a0 0x7f84340fce10 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:01.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.196+0000 7f843b400700 1 --2- 172.21.15.123:0/2989283247 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8434102e00 0x7f84341031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:01.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.196+0000 7f843b400700 1 -- 172.21.15.123:0/2989283247 >> 172.21.15.123:0/2989283247 conn(0x7f84340f89b0 msgr2=0x7f84340fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:59:01.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.196+0000 7f843b400700 1 -- 172.21.15.123:0/2989283247 shutdown_connections 2024-03-30T04:59:01.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.196+0000 7f843b400700 1 -- 172.21.15.123:0/2989283247 wait complete. 2024-03-30T04:59:01.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.197+0000 7f843b400700 1 Processor -- start 2024-03-30T04:59:01.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.197+0000 7f843b400700 1 -- start start 2024-03-30T04:59:01.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.197+0000 7f843b400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8434102e00 0x7f84340702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:59:01.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.197+0000 7f843b400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f84341037a0 0x7f8434070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:59:01.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.197+0000 7f843b400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8434070e20 con 0x7f8434102e00 2024-03-30T04:59:01.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.197+0000 7f843b400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8434070f60 con 0x7f84341037a0 2024-03-30T04:59:01.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.197+0000 7f8438a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8434102e00 0x7f84340702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:59:01.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.198+0000 7f8438a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8434102e00 0x7f84340702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:59952/0 (socket says 172.21.15.123:59952) 2024-03-30T04:59:01.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.198+0000 7f8438a00700 1 -- 172.21.15.123:0/1390646558 learned_addr learned my addr 172.21.15.123:0/1390646558 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:59:01.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.198+0000 7f8433e00700 1 --2- 172.21.15.123:0/1390646558 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f84341037a0 0x7f8434070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:59:01.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.198+0000 7f8438a00700 1 -- 172.21.15.123:0/1390646558 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f84341037a0 msgr2=0x7f8434070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:59:01.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.198+0000 7f8438a00700 1 --2- 172.21.15.123:0/1390646558 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f84341037a0 0x7f8434070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:01.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.198+0000 7f8438a00700 1 -- 172.21.15.123:0/1390646558 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8420005040 con 0x7f8434102e00 2024-03-30T04:59:01.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.198+0000 7f8438a00700 1 --2- 172.21.15.123:0/1390646558 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8434102e00 0x7f84340702c0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f842800b790 tx=0x7f8428009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:59:01.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.198+0000 7f8430c00700 1 -- 172.21.15.123:0/1390646558 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f842800f770 con 0x7f8434102e00 2024-03-30T04:59:01.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.198+0000 7f8430c00700 1 -- 172.21.15.123:0/1390646558 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f842800fd80 con 0x7f8434102e00 2024-03-30T04:59:01.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.198+0000 7f843b400700 1 -- 172.21.15.123:0/1390646558 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8434119de0 con 0x7f8434102e00 2024-03-30T04:59:01.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.199+0000 7f8430c00700 1 -- 172.21.15.123:0/1390646558 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f842800d310 con 0x7f8434102e00 2024-03-30T04:59:01.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.199+0000 7f843b400700 1 -- 172.21.15.123:0/1390646558 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f843411a2f0 con 0x7f8434102e00 2024-03-30T04:59:01.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.200+0000 7f843b400700 1 -- 172.21.15.123:0/1390646558 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84340fe550 con 0x7f8434102e00 2024-03-30T04:59:01.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.201+0000 7f8430c00700 1 -- 172.21.15.123:0/1390646558 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f8428010420 con 0x7f8434102e00 2024-03-30T04:59:01.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.202+0000 7f8430c00700 1 --2- 172.21.15.123:0/1390646558 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f841c06cc30 0x7f841c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:59:01.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.202+0000 7f8430c00700 1 -- 172.21.15.123:0/1390646558 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f842808c080 con 0x7f8434102e00 2024-03-30T04:59:01.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.202+0000 7f8433e00700 1 --2- 172.21.15.123:0/1390646558 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f841c06cc30 0x7f841c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:59:01.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.203+0000 7f8433e00700 1 --2- 172.21.15.123:0/1390646558 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f841c06cc30 0x7f841c06f0e0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f842000b020 tx=0x7f8420009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:59:01.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.206+0000 7f8430c00700 1 -- 172.21.15.123:0/1390646558 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f84280560c0 con 0x7f8434102e00 2024-03-30T04:59:01.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:59:00 smithi135 ceph-mon[106011]: pgmap v835: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:01.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:59:00 smithi135 ceph-mon[106011]: from='client.35524 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:59:01.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.452+0000 7f843b400700 1 -- 172.21.15.123:0/1390646558 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f84340719b0 con 0x7f8434102e00 2024-03-30T04:59:01.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.452+0000 7f8430c00700 1 -- 172.21.15.123:0/1390646558 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 34 v34) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f84280596e0 con 0x7f8434102e00 2024-03-30T04:59:01.451 INFO:teuthology.orchestra.run.smithi123.stdout:e34 2024-03-30T04:59:01.451 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:59:01.451 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:59:01.451 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:59:01.451 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:59:01.451 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:59:01.451 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:59:01.451 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T04:59:01.452 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T04:59:01.452 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:59:01.452 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T04:59:01.452 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:59:01.452 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:59:01.452 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:59:01.452 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:59:01.452 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:59:01.452 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T04:59:01.452 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:59:01.452 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:59:01.452 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T04:59:01.452 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:59:01.452 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T04:59:01.453 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T04:59:01.453 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T04:59:01.453 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T04:59:01.453 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:59:01.453 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:59:01.453 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:59:01.453 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:59:01.453 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:59:01.453 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:59:01.453 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T04:59:01.453 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T04:59:01.453 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:59:01.453 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:59:01.453 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T04:59:01.453 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:59:01.454 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:59:01.454 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:59:01.454 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:59:01.454 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44975} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/3788988014,v1:172.21.15.135:6827/3788988014] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:59:01.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.455+0000 7f843b400700 1 -- 172.21.15.123:0/1390646558 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f841c06cc30 msgr2=0x7f841c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:59:01.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.455+0000 7f843b400700 1 --2- 172.21.15.123:0/1390646558 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f841c06cc30 0x7f841c06f0e0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f842000b020 tx=0x7f8420009e30 comp rx=0 tx=0).stop 2024-03-30T04:59:01.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.455+0000 7f843b400700 1 -- 172.21.15.123:0/1390646558 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8434102e00 msgr2=0x7f84340702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:59:01.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.455+0000 7f843b400700 1 --2- 172.21.15.123:0/1390646558 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8434102e00 0x7f84340702c0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f842800b790 tx=0x7f8428009250 comp rx=0 tx=0).stop 2024-03-30T04:59:01.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.455+0000 7f843b400700 1 -- 172.21.15.123:0/1390646558 shutdown_connections 2024-03-30T04:59:01.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.455+0000 7f843b400700 1 --2- 172.21.15.123:0/1390646558 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f841c06cc30 0x7f841c06f0e0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:01.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.455+0000 7f843b400700 1 --2- 172.21.15.123:0/1390646558 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8434102e00 0x7f84340702c0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:01.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.455+0000 7f843b400700 1 --2- 172.21.15.123:0/1390646558 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f84341037a0 0x7f8434070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:01.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.455+0000 7f843b400700 1 -- 172.21.15.123:0/1390646558 >> 172.21.15.123:0/1390646558 conn(0x7f84340f89b0 msgr2=0x7f84340f99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:59:01.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.455+0000 7f843b400700 1 -- 172.21.15.123:0/1390646558 shutdown_connections 2024-03-30T04:59:01.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.455+0000 7f843b400700 1 -- 172.21.15.123:0/1390646558 wait complete. 2024-03-30T04:59:01.455 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 34 2024-03-30T04:59:01.465 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:59:00 smithi123 ceph-mon[139369]: pgmap v835: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:01.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:59:00 smithi123 ceph-mon[139369]: from='client.35524 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:59:01.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.590+0000 7f9351e00700 1 -- 172.21.15.123:0/3190873855 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f934c1037a0 msgr2=0x7f934c0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:59:01.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.590+0000 7f9351e00700 1 --2- 172.21.15.123:0/3190873855 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f934c1037a0 0x7f934c0fce10 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f9338004940 tx=0x7f9338009e30 comp rx=0 tx=0).stop 2024-03-30T04:59:01.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.590+0000 7f9351e00700 1 -- 172.21.15.123:0/3190873855 shutdown_connections 2024-03-30T04:59:01.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.590+0000 7f9351e00700 1 --2- 172.21.15.123:0/3190873855 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f934c1037a0 0x7f934c0fce10 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:01.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.590+0000 7f9351e00700 1 --2- 172.21.15.123:0/3190873855 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f934c102e00 0x7f934c1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:01.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.590+0000 7f9351e00700 1 -- 172.21.15.123:0/3190873855 >> 172.21.15.123:0/3190873855 conn(0x7f934c0f89b0 msgr2=0x7f934c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:59:01.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.590+0000 7f9351e00700 1 -- 172.21.15.123:0/3190873855 shutdown_connections 2024-03-30T04:59:01.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.590+0000 7f9351e00700 1 -- 172.21.15.123:0/3190873855 wait complete. 2024-03-30T04:59:01.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.591+0000 7f9351e00700 1 Processor -- start 2024-03-30T04:59:01.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.591+0000 7f9351e00700 1 -- start start 2024-03-30T04:59:01.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.592+0000 7f9351e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f934c102e00 0x7f934c115d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:59:01.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.592+0000 7f9351e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f934c1037a0 0x7f934c116260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:59:01.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.592+0000 7f9351e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f934c1168b0 con 0x7f934c1037a0 2024-03-30T04:59:01.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.592+0000 7f9351e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f934c10fc80 con 0x7f934c102e00 2024-03-30T04:59:01.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.592+0000 7f934a000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f934c1037a0 0x7f934c116260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:59:01.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.592+0000 7f934a000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f934c1037a0 0x7f934c116260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:59976/0 (socket says 172.21.15.123:59976) 2024-03-30T04:59:01.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.592+0000 7f934a000700 1 -- 172.21.15.123:0/3773882113 learned_addr learned my addr 172.21.15.123:0/3773882113 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:59:01.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.592+0000 7f934aa00700 1 --2- 172.21.15.123:0/3773882113 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f934c102e00 0x7f934c115d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:59:01.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.592+0000 7f934a000700 1 -- 172.21.15.123:0/3773882113 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f934c102e00 msgr2=0x7f934c115d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:59:01.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.592+0000 7f934a000700 1 --2- 172.21.15.123:0/3773882113 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f934c102e00 0x7f934c115d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:01.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.592+0000 7f934a000700 1 -- 172.21.15.123:0/3773882113 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9338005040 con 0x7f934c1037a0 2024-03-30T04:59:01.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.592+0000 7f934a000700 1 --2- 172.21.15.123:0/3773882113 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f934c1037a0 0x7f934c116260 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f9338009390 tx=0x7f9338009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:59:01.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.593+0000 7f9342a00700 1 -- 172.21.15.123:0/3773882113 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f933800e040 con 0x7f934c1037a0 2024-03-30T04:59:01.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.593+0000 7f9342a00700 1 -- 172.21.15.123:0/3773882113 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9338013420 con 0x7f934c1037a0 2024-03-30T04:59:01.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.593+0000 7f9342a00700 1 -- 172.21.15.123:0/3773882113 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f933801c630 con 0x7f934c1037a0 2024-03-30T04:59:01.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.593+0000 7f9351e00700 1 -- 172.21.15.123:0/3773882113 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f934c10ff00 con 0x7f934c1037a0 2024-03-30T04:59:01.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.593+0000 7f9351e00700 1 -- 172.21.15.123:0/3773882113 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f934c1104c0 con 0x7f934c1037a0 2024-03-30T04:59:01.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.594+0000 7f9351e00700 1 -- 172.21.15.123:0/3773882113 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f934c0fe550 con 0x7f934c1037a0 2024-03-30T04:59:01.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.595+0000 7f9342a00700 1 -- 172.21.15.123:0/3773882113 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f9338013a00 con 0x7f934c1037a0 2024-03-30T04:59:01.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.596+0000 7f9342a00700 1 --2- 172.21.15.123:0/3773882113 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f933006cc30 0x7f933006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:59:01.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.596+0000 7f9342a00700 1 -- 172.21.15.123:0/3773882113 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f933808d120 con 0x7f934c1037a0 2024-03-30T04:59:01.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.596+0000 7f934aa00700 1 --2- 172.21.15.123:0/3773882113 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f933006cc30 0x7f933006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:59:01.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.597+0000 7f934aa00700 1 --2- 172.21.15.123:0/3773882113 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f933006cc30 0x7f933006f0e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f933c004f60 tx=0x7f933c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:59:01.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.600+0000 7f9342a00700 1 -- 172.21.15.123:0/3773882113 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f933805abf0 con 0x7f934c1037a0 2024-03-30T04:59:01.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.821+0000 7f9351e00700 1 -- 172.21.15.123:0/3773882113 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f934c111410 con 0x7f933006cc30 2024-03-30T04:59:01.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.823+0000 7f9342a00700 1 -- 172.21.15.123:0/3773882113 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f934c111410 con 0x7f933006cc30 2024-03-30T04:59:01.821 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:59:01.821 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:59:01.821 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:59:01.821 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T04:59:01.821 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:59:01.821 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T04:59:01.821 INFO:teuthology.orchestra.run.smithi123.stdout: "mds", 2024-03-30T04:59:01.822 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T04:59:01.822 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T04:59:01.822 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T04:59:01.822 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:59:01.822 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "16/21 daemons upgraded", 2024-03-30T04:59:01.822 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T04:59:01.822 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:59:01.822 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:59:01.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.825+0000 7f9351e00700 1 -- 172.21.15.123:0/3773882113 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f933006cc30 msgr2=0x7f933006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:59:01.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.825+0000 7f9351e00700 1 --2- 172.21.15.123:0/3773882113 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f933006cc30 0x7f933006f0e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f933c004f60 tx=0x7f933c00b040 comp rx=0 tx=0).stop 2024-03-30T04:59:01.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.825+0000 7f9351e00700 1 -- 172.21.15.123:0/3773882113 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f934c1037a0 msgr2=0x7f934c116260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:59:01.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.825+0000 7f9351e00700 1 --2- 172.21.15.123:0/3773882113 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f934c1037a0 0x7f934c116260 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f9338009390 tx=0x7f9338009e30 comp rx=0 tx=0).stop 2024-03-30T04:59:01.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.825+0000 7f9351e00700 1 -- 172.21.15.123:0/3773882113 shutdown_connections 2024-03-30T04:59:01.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.825+0000 7f9351e00700 1 --2- 172.21.15.123:0/3773882113 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f933006cc30 0x7f933006f0e0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:01.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.825+0000 7f9351e00700 1 --2- 172.21.15.123:0/3773882113 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f934c102e00 0x7f934c115d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:01.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.825+0000 7f9351e00700 1 --2- 172.21.15.123:0/3773882113 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f934c1037a0 0x7f934c116260 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:01.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.825+0000 7f9351e00700 1 -- 172.21.15.123:0/3773882113 >> 172.21.15.123:0/3773882113 conn(0x7f934c0f89b0 msgr2=0x7f934c070750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:59:01.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.825+0000 7f9351e00700 1 -- 172.21.15.123:0/3773882113 shutdown_connections 2024-03-30T04:59:01.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.825+0000 7f9351e00700 1 -- 172.21.15.123:0/3773882113 wait complete. 2024-03-30T04:59:01.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.961+0000 7f7da2400700 1 -- 172.21.15.123:0/1649274389 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d9c1035e0 msgr2=0x7f7d9c0fcc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:59:01.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.961+0000 7f7da2400700 1 --2- 172.21.15.123:0/1649274389 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d9c1035e0 0x7f7d9c0fcc50 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f7d88004940 tx=0x7f7d88009e30 comp rx=0 tx=0).stop 2024-03-30T04:59:01.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.961+0000 7f7da2400700 1 -- 172.21.15.123:0/1649274389 shutdown_connections 2024-03-30T04:59:01.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.961+0000 7f7da2400700 1 --2- 172.21.15.123:0/1649274389 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d9c1035e0 0x7f7d9c0fcc50 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:01.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.961+0000 7f7da2400700 1 --2- 172.21.15.123:0/1649274389 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d9c102c40 0x7f7d9c103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:01.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.961+0000 7f7da2400700 1 -- 172.21.15.123:0/1649274389 >> 172.21.15.123:0/1649274389 conn(0x7f7d9c0f8830 msgr2=0x7f7d9c0fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:59:01.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.961+0000 7f7da2400700 1 -- 172.21.15.123:0/1649274389 shutdown_connections 2024-03-30T04:59:01.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.961+0000 7f7da2400700 1 -- 172.21.15.123:0/1649274389 wait complete. 2024-03-30T04:59:01.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.962+0000 7f7da2400700 1 Processor -- start 2024-03-30T04:59:01.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.962+0000 7f7da2400700 1 -- start start 2024-03-30T04:59:01.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.963+0000 7f7da2400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d9c102c40 0x7f7d9c115af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:59:01.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.963+0000 7f7da2400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d9c1035e0 0x7f7d9c116030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:59:01.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.963+0000 7f7da2400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9c116680 con 0x7f7d9c1035e0 2024-03-30T04:59:01.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.963+0000 7f7da2400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9c10fa70 con 0x7f7d9c102c40 2024-03-30T04:59:01.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.963+0000 7f7d9be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d9c1035e0 0x7f7d9c116030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:59:01.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.963+0000 7f7d9be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d9c1035e0 0x7f7d9c116030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:59982/0 (socket says 172.21.15.123:59982) 2024-03-30T04:59:01.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.963+0000 7f7d9be00700 1 -- 172.21.15.123:0/1619807691 learned_addr learned my addr 172.21.15.123:0/1619807691 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:59:01.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.963+0000 7f7da1000700 1 --2- 172.21.15.123:0/1619807691 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d9c102c40 0x7f7d9c115af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:59:01.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.963+0000 7f7d9be00700 1 -- 172.21.15.123:0/1619807691 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d9c102c40 msgr2=0x7f7d9c115af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:59:01.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.963+0000 7f7d9be00700 1 --2- 172.21.15.123:0/1619807691 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d9c102c40 0x7f7d9c115af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:01.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.963+0000 7f7d9be00700 1 -- 172.21.15.123:0/1619807691 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d88005040 con 0x7f7d9c1035e0 2024-03-30T04:59:01.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.964+0000 7f7d9be00700 1 --2- 172.21.15.123:0/1619807691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d9c1035e0 0x7f7d9c116030 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f7d88009390 tx=0x7f7d88009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:59:01.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.964+0000 7f7d98c00700 1 -- 172.21.15.123:0/1619807691 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d8800e040 con 0x7f7d9c1035e0 2024-03-30T04:59:01.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.964+0000 7f7da2400700 1 -- 172.21.15.123:0/1619807691 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d9c10fcf0 con 0x7f7d9c1035e0 2024-03-30T04:59:01.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.964+0000 7f7d98c00700 1 -- 172.21.15.123:0/1619807691 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7d88013420 con 0x7f7d9c1035e0 2024-03-30T04:59:01.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.964+0000 7f7d98c00700 1 -- 172.21.15.123:0/1619807691 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d8801c630 con 0x7f7d9c1035e0 2024-03-30T04:59:01.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.964+0000 7f7da2400700 1 -- 172.21.15.123:0/1619807691 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d9c1102b0 con 0x7f7d9c1035e0 2024-03-30T04:59:01.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.965+0000 7f7da2400700 1 -- 172.21.15.123:0/1619807691 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d9c0fe390 con 0x7f7d9c1035e0 2024-03-30T04:59:01.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.966+0000 7f7d98c00700 1 -- 172.21.15.123:0/1619807691 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f7d88013a00 con 0x7f7d9c1035e0 2024-03-30T04:59:01.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.967+0000 7f7d98c00700 1 --2- 172.21.15.123:0/1619807691 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7d8406cc30 0x7f7d8406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:59:01.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.967+0000 7f7d98c00700 1 -- 172.21.15.123:0/1619807691 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7d8808d390 con 0x7f7d9c1035e0 2024-03-30T04:59:01.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.967+0000 7f7da1000700 1 --2- 172.21.15.123:0/1619807691 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7d8406cc30 0x7f7d8406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:59:01.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.968+0000 7f7da1000700 1 --2- 172.21.15.123:0/1619807691 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7d8406cc30 0x7f7d8406f0e0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f7d90004990 tx=0x7f7d90009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:59:01.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:01.971+0000 7f7d98c00700 1 -- 172.21.15.123:0/1619807691 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7d8805ade0 con 0x7f7d9c1035e0 2024-03-30T04:59:02.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:02.258+0000 7f7da2400700 1 -- 172.21.15.123:0/1619807691 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f7d9c109bd0 con 0x7f7d9c1035e0 2024-03-30T04:59:02.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:02.259+0000 7f7d98c00700 1 -- 172.21.15.123:0/1619807691 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f7d8805a970 con 0x7f7d9c1035e0 2024-03-30T04:59:02.257 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T04:59:02.257 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-30T04:59:02.257 INFO:teuthology.orchestra.run.smithi123.stdout: host smithi135 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi135/config 2024-03-30T04:59:02.258 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-30T04:59:02.258 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T04:59:02.258 INFO:teuthology.orchestra.run.smithi123.stdout:Traceback (most recent call last): 2024-03-30T04:59:02.258 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-30T04:59:02.258 INFO:teuthology.orchestra.run.smithi123.stdout: "__main__", mod_spec) 2024-03-30T04:59:02.258 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-30T04:59:02.258 INFO:teuthology.orchestra.run.smithi123.stdout: exec(code, run_globals) 2024-03-30T04:59:02.258 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10889, in 2024-03-30T04:59:02.258 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10877, in main 2024-03-30T04:59:02.258 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-30T04:59:02.258 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-30T04:59:02.258 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-30T04:59:02.258 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-30T04:59:02.258 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-03-30T04:59:02.259 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-30T04:59:02.259 INFO:teuthology.orchestra.run.smithi123.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T04:59:02.259 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T04:59:02.259 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T04:59:02.259 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T04:59:02.259 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T04:59:02.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:02.261+0000 7f7da2400700 1 -- 172.21.15.123:0/1619807691 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7d8406cc30 msgr2=0x7f7d8406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:59:02.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:02.261+0000 7f7da2400700 1 --2- 172.21.15.123:0/1619807691 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7d8406cc30 0x7f7d8406f0e0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f7d90004990 tx=0x7f7d90009ee0 comp rx=0 tx=0).stop 2024-03-30T04:59:02.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:02.261+0000 7f7da2400700 1 -- 172.21.15.123:0/1619807691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d9c1035e0 msgr2=0x7f7d9c116030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:59:02.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:02.261+0000 7f7da2400700 1 --2- 172.21.15.123:0/1619807691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d9c1035e0 0x7f7d9c116030 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f7d88009390 tx=0x7f7d88009e30 comp rx=0 tx=0).stop 2024-03-30T04:59:02.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:02.261+0000 7f7da2400700 1 -- 172.21.15.123:0/1619807691 shutdown_connections 2024-03-30T04:59:02.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:02.261+0000 7f7da2400700 1 --2- 172.21.15.123:0/1619807691 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7d8406cc30 0x7f7d8406f0e0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:02.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:02.261+0000 7f7da2400700 1 --2- 172.21.15.123:0/1619807691 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d9c102c40 0x7f7d9c115af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:02.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:02.261+0000 7f7da2400700 1 --2- 172.21.15.123:0/1619807691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d9c1035e0 0x7f7d9c116030 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:02.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:02.261+0000 7f7da2400700 1 -- 172.21.15.123:0/1619807691 >> 172.21.15.123:0/1619807691 conn(0x7f7d9c0f8830 msgr2=0x7f7d9c0f9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:59:02.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:02.261+0000 7f7da2400700 1 -- 172.21.15.123:0/1619807691 shutdown_connections 2024-03-30T04:59:02.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:02.261+0000 7f7da2400700 1 -- 172.21.15.123:0/1619807691 wait complete. 2024-03-30T04:59:02.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:59:01 smithi135 ceph-mon[106011]: from='client.35528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:59:02.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:59:01 smithi135 ceph-mon[106011]: from='client.35532 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:59:02.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:59:01 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2541596299' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:59:02.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:59:01 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/1390646558' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:59:02.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:59:01 smithi123 ceph-mon[139369]: from='client.35528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:59:02.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:59:01 smithi123 ceph-mon[139369]: from='client.35532 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:59:02.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:59:01 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2541596299' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:59:02.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:59:01 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/1390646558' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:59:03.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:59:02 smithi135 ceph-mon[106011]: pgmap v836: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:03.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:59:02 smithi135 ceph-mon[106011]: from='client.35544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:59:03.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:59:02 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/1619807691' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:59:03.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:59:02 smithi123 ceph-mon[139369]: pgmap v836: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:03.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:59:02 smithi123 ceph-mon[139369]: from='client.35544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:59:03.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:59:02 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/1619807691' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:59:05.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:59:04 smithi135 ceph-mon[106011]: pgmap v837: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:05.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:59:04 smithi123 ceph-mon[139369]: pgmap v837: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:07.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:59:07 smithi135 ceph-mon[106011]: pgmap v838: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:07.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:59:07 smithi123 ceph-mon[139369]: pgmap v838: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:09.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:59:09 smithi135 ceph-mon[106011]: pgmap v839: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:09.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:59:09 smithi123 ceph-mon[139369]: pgmap v839: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:11.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:59:11 smithi135 ceph-mon[106011]: pgmap v840: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:11.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:59:11 smithi123 ceph-mon[139369]: pgmap v840: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:13.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:59:13 smithi135 ceph-mon[106011]: pgmap v841: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:13.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:59:13 smithi123 ceph-mon[139369]: pgmap v841: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:15.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:59:15 smithi135 ceph-mon[106011]: pgmap v842: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:15.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:59:15 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:59:15.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:59:15 smithi123 ceph-mon[139369]: pgmap v842: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:15.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:59:15 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:59:17.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:59:17 smithi135 ceph-mon[106011]: pgmap v843: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:17.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:59:17 smithi123 ceph-mon[139369]: pgmap v843: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:19.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:59:19 smithi135 ceph-mon[106011]: pgmap v844: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:19.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:59:19 smithi123 ceph-mon[139369]: pgmap v844: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:21.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:59:21 smithi135 ceph-mon[106011]: pgmap v845: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:21.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:59:21 smithi123 ceph-mon[139369]: pgmap v845: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:23.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:59:23 smithi135 ceph-mon[106011]: pgmap v846: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:23.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:59:23 smithi123 ceph-mon[139369]: pgmap v846: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:25.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:59:25 smithi135 ceph-mon[106011]: pgmap v847: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:25.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:59:25 smithi123 ceph-mon[139369]: pgmap v847: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:27.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:59:27 smithi135 ceph-mon[106011]: pgmap v848: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:27.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:59:27 smithi123 ceph-mon[139369]: pgmap v848: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:29.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:59:29 smithi135 ceph-mon[106011]: pgmap v849: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:29.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:59:29 smithi123 ceph-mon[139369]: pgmap v849: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:30.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:59:30 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:59:30.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:59:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:59:31.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:59:31 smithi135 ceph-mon[106011]: pgmap v850: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:31.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:59:31 smithi123 ceph-mon[139369]: pgmap v850: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:32.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.409+0000 7f6d8f000700 1 -- 172.21.15.123:0/2316636269 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d88102e00 msgr2=0x7f6d881031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:59:32.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.409+0000 7f6d8f000700 1 --2- 172.21.15.123:0/2316636269 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d88102e00 0x7f6d881031d0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f6d78004940 tx=0x7f6d78009e30 comp rx=0 tx=0).stop 2024-03-30T04:59:32.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.409+0000 7f6d8f000700 1 -- 172.21.15.123:0/2316636269 shutdown_connections 2024-03-30T04:59:32.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.409+0000 7f6d8f000700 1 --2- 172.21.15.123:0/2316636269 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6d881037a0 0x7f6d880fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:32.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.409+0000 7f6d8f000700 1 --2- 172.21.15.123:0/2316636269 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d88102e00 0x7f6d881031d0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:32.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.409+0000 7f6d8f000700 1 -- 172.21.15.123:0/2316636269 >> 172.21.15.123:0/2316636269 conn(0x7f6d880f89b0 msgr2=0x7f6d880fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:59:32.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.409+0000 7f6d8f000700 1 -- 172.21.15.123:0/2316636269 shutdown_connections 2024-03-30T04:59:32.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.409+0000 7f6d8f000700 1 -- 172.21.15.123:0/2316636269 wait complete. 2024-03-30T04:59:32.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.410+0000 7f6d8f000700 1 Processor -- start 2024-03-30T04:59:32.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.410+0000 7f6d8f000700 1 -- start start 2024-03-30T04:59:32.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.410+0000 7f6d8f000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6d88102e00 0x7f6d88115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:59:32.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.410+0000 7f6d8f000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d881037a0 0x7f6d88116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:59:32.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.410+0000 7f6d8f000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d88116860 con 0x7f6d881037a0 2024-03-30T04:59:32.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.410+0000 7f6d8f000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8810fc30 con 0x7f6d88102e00 2024-03-30T04:59:32.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.411+0000 7f6d87400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d881037a0 0x7f6d88116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:59:32.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.411+0000 7f6d87400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d881037a0 0x7f6d88116210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48704/0 (socket says 172.21.15.123:48704) 2024-03-30T04:59:32.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.411+0000 7f6d87400700 1 -- 172.21.15.123:0/385749700 learned_addr learned my addr 172.21.15.123:0/385749700 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:59:32.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.411+0000 7f6d87e00700 1 --2- 172.21.15.123:0/385749700 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6d88102e00 0x7f6d88115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:59:32.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.411+0000 7f6d87400700 1 -- 172.21.15.123:0/385749700 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6d88102e00 msgr2=0x7f6d88115cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:59:32.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.411+0000 7f6d87400700 1 --2- 172.21.15.123:0/385749700 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6d88102e00 0x7f6d88115cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:32.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.411+0000 7f6d87400700 1 -- 172.21.15.123:0/385749700 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d78005040 con 0x7f6d881037a0 2024-03-30T04:59:32.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.411+0000 7f6d87400700 1 --2- 172.21.15.123:0/385749700 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d881037a0 0x7f6d88116210 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f6d74002a50 tx=0x7f6d74004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:59:32.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.411+0000 7f6d7fe00700 1 -- 172.21.15.123:0/385749700 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d74007cc0 con 0x7f6d881037a0 2024-03-30T04:59:32.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.412+0000 7f6d7fe00700 1 -- 172.21.15.123:0/385749700 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6d7400f040 con 0x7f6d881037a0 2024-03-30T04:59:32.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.412+0000 7f6d8f000700 1 -- 172.21.15.123:0/385749700 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d8810ff10 con 0x7f6d881037a0 2024-03-30T04:59:32.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.412+0000 7f6d7fe00700 1 -- 172.21.15.123:0/385749700 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d7400d920 con 0x7f6d881037a0 2024-03-30T04:59:32.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.412+0000 7f6d8f000700 1 -- 172.21.15.123:0/385749700 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d881104d0 con 0x7f6d881037a0 2024-03-30T04:59:32.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.413+0000 7f6d8f000700 1 -- 172.21.15.123:0/385749700 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d880fe550 con 0x7f6d881037a0 2024-03-30T04:59:32.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.414+0000 7f6d7fe00700 1 -- 172.21.15.123:0/385749700 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f6d74009680 con 0x7f6d881037a0 2024-03-30T04:59:32.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.415+0000 7f6d7fe00700 1 --2- 172.21.15.123:0/385749700 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6d6c06cc30 0x7f6d6c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:59:32.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.415+0000 7f6d7fe00700 1 -- 172.21.15.123:0/385749700 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f6d7408bd10 con 0x7f6d881037a0 2024-03-30T04:59:32.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.415+0000 7f6d87e00700 1 --2- 172.21.15.123:0/385749700 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6d6c06cc30 0x7f6d6c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:59:32.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.416+0000 7f6d87e00700 1 --2- 172.21.15.123:0/385749700 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6d6c06cc30 0x7f6d6c06f0e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f6d7800b840 tx=0x7f6d78009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:59:32.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.420+0000 7f6d7fe00700 1 -- 172.21.15.123:0/385749700 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6d74059760 con 0x7f6d881037a0 2024-03-30T04:59:32.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.643+0000 7f6d8f000700 1 -- 172.21.15.123:0/385749700 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6d88109e60 con 0x7f6d6c06cc30 2024-03-30T04:59:32.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.644+0000 7f6d7fe00700 1 -- 172.21.15.123:0/385749700 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f6d78008ee0 con 0x7f6d6c06cc30 2024-03-30T04:59:32.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.647+0000 7f6d8f000700 1 -- 172.21.15.123:0/385749700 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6d6c06cc30 msgr2=0x7f6d6c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:59:32.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.647+0000 7f6d8f000700 1 --2- 172.21.15.123:0/385749700 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6d6c06cc30 0x7f6d6c06f0e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f6d7800b840 tx=0x7f6d78009e30 comp rx=0 tx=0).stop 2024-03-30T04:59:32.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.647+0000 7f6d8f000700 1 -- 172.21.15.123:0/385749700 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d881037a0 msgr2=0x7f6d88116210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:59:32.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.647+0000 7f6d8f000700 1 --2- 172.21.15.123:0/385749700 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d881037a0 0x7f6d88116210 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f6d74002a50 tx=0x7f6d74004600 comp rx=0 tx=0).stop 2024-03-30T04:59:32.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.647+0000 7f6d8f000700 1 -- 172.21.15.123:0/385749700 shutdown_connections 2024-03-30T04:59:32.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.647+0000 7f6d8f000700 1 --2- 172.21.15.123:0/385749700 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6d6c06cc30 0x7f6d6c06f0e0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:32.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.647+0000 7f6d8f000700 1 --2- 172.21.15.123:0/385749700 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6d88102e00 0x7f6d88115cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:32.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.647+0000 7f6d8f000700 1 --2- 172.21.15.123:0/385749700 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d881037a0 0x7f6d88116210 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:32.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.647+0000 7f6d8f000700 1 -- 172.21.15.123:0/385749700 >> 172.21.15.123:0/385749700 conn(0x7f6d880f89b0 msgr2=0x7f6d880f99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:59:32.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.647+0000 7f6d8f000700 1 -- 172.21.15.123:0/385749700 shutdown_connections 2024-03-30T04:59:32.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.647+0000 7f6d8f000700 1 -- 172.21.15.123:0/385749700 wait complete. 2024-03-30T04:59:32.660 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T04:59:32.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.781+0000 7f3ee4a00700 1 -- 172.21.15.123:0/249631791 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ee0102e00 msgr2=0x7f3ee01031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:59:32.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.781+0000 7f3ee4a00700 1 --2- 172.21.15.123:0/249631791 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ee0102e00 0x7f3ee01031d0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f3ed0004940 tx=0x7f3ed0009e30 comp rx=0 tx=0).stop 2024-03-30T04:59:32.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.782+0000 7f3ee4a00700 1 -- 172.21.15.123:0/249631791 shutdown_connections 2024-03-30T04:59:32.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.782+0000 7f3ee4a00700 1 --2- 172.21.15.123:0/249631791 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3ee01037a0 0x7f3ee00fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:32.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.782+0000 7f3ee4a00700 1 --2- 172.21.15.123:0/249631791 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ee0102e00 0x7f3ee01031d0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:32.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.782+0000 7f3ee4a00700 1 -- 172.21.15.123:0/249631791 >> 172.21.15.123:0/249631791 conn(0x7f3ee00f89b0 msgr2=0x7f3ee00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:59:32.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.782+0000 7f3ee4a00700 1 -- 172.21.15.123:0/249631791 shutdown_connections 2024-03-30T04:59:32.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.782+0000 7f3ee4a00700 1 -- 172.21.15.123:0/249631791 wait complete. 2024-03-30T04:59:32.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.783+0000 7f3ee4a00700 1 Processor -- start 2024-03-30T04:59:32.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.783+0000 7f3ee4a00700 1 -- start start 2024-03-30T04:59:32.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.783+0000 7f3ee4a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3ee0102e00 0x7f3ee0113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:59:32.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.783+0000 7f3ee4a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ee01037a0 0x7f3ee0114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:59:32.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.783+0000 7f3ee4a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee01146b0 con 0x7f3ee01037a0 2024-03-30T04:59:32.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.783+0000 7f3ee4a00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee010daa0 con 0x7f3ee0102e00 2024-03-30T04:59:32.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.783+0000 7f3edd400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ee01037a0 0x7f3ee0114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:59:32.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.783+0000 7f3edd400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ee01037a0 0x7f3ee0114060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48712/0 (socket says 172.21.15.123:48712) 2024-03-30T04:59:32.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.783+0000 7f3edd400700 1 -- 172.21.15.123:0/2915139376 learned_addr learned my addr 172.21.15.123:0/2915139376 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:59:32.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.783+0000 7f3edde00700 1 --2- 172.21.15.123:0/2915139376 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3ee0102e00 0x7f3ee0113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:59:32.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.784+0000 7f3edd400700 1 -- 172.21.15.123:0/2915139376 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3ee0102e00 msgr2=0x7f3ee0113b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:59:32.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.784+0000 7f3edd400700 1 --2- 172.21.15.123:0/2915139376 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3ee0102e00 0x7f3ee0113b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:32.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.784+0000 7f3edd400700 1 -- 172.21.15.123:0/2915139376 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed0005040 con 0x7f3ee01037a0 2024-03-30T04:59:32.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.784+0000 7f3edd400700 1 --2- 172.21.15.123:0/2915139376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ee01037a0 0x7f3ee0114060 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc002a50 tx=0x7f3ecc004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:59:32.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.784+0000 7f3ed6000700 1 -- 172.21.15.123:0/2915139376 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ecc007cc0 con 0x7f3ee01037a0 2024-03-30T04:59:32.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.784+0000 7f3ed6000700 1 -- 172.21.15.123:0/2915139376 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3ecc00f040 con 0x7f3ee01037a0 2024-03-30T04:59:32.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.784+0000 7f3ee4a00700 1 -- 172.21.15.123:0/2915139376 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ee010dd80 con 0x7f3ee01037a0 2024-03-30T04:59:32.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.784+0000 7f3ed6000700 1 -- 172.21.15.123:0/2915139376 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ecc00d920 con 0x7f3ee01037a0 2024-03-30T04:59:32.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.784+0000 7f3ee4a00700 1 -- 172.21.15.123:0/2915139376 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ee010e340 con 0x7f3ee01037a0 2024-03-30T04:59:32.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.785+0000 7f3ee4a00700 1 -- 172.21.15.123:0/2915139376 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ee00fe550 con 0x7f3ee01037a0 2024-03-30T04:59:32.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.786+0000 7f3ed6000700 1 -- 172.21.15.123:0/2915139376 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f3ecc009680 con 0x7f3ee01037a0 2024-03-30T04:59:32.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.787+0000 7f3ed6000700 1 --2- 172.21.15.123:0/2915139376 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3ec406cc30 0x7f3ec406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:59:32.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.787+0000 7f3ed6000700 1 -- 172.21.15.123:0/2915139376 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f3ecc08bd80 con 0x7f3ee01037a0 2024-03-30T04:59:32.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.787+0000 7f3edde00700 1 --2- 172.21.15.123:0/2915139376 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3ec406cc30 0x7f3ec406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:59:32.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.788+0000 7f3edde00700 1 --2- 172.21.15.123:0/2915139376 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3ec406cc30 0x7f3ec406f0e0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f3ed0009fd0 tx=0x7f3ed0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:59:32.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:32.792+0000 7f3ed6000700 1 -- 172.21.15.123:0/2915139376 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3ecc0597d0 con 0x7f3ee01037a0 2024-03-30T04:59:33.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.019+0000 7f3ee4a00700 1 -- 172.21.15.123:0/2915139376 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3ee0109dc0 con 0x7f3ec406cc30 2024-03-30T04:59:33.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.021+0000 7f3ed6000700 1 -- 172.21.15.123:0/2915139376 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f3ee0109dc0 con 0x7f3ec406cc30 2024-03-30T04:59:33.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.023+0000 7f3ee4a00700 1 -- 172.21.15.123:0/2915139376 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3ec406cc30 msgr2=0x7f3ec406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:59:33.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.023+0000 7f3ee4a00700 1 --2- 172.21.15.123:0/2915139376 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3ec406cc30 0x7f3ec406f0e0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f3ed0009fd0 tx=0x7f3ed0009e30 comp rx=0 tx=0).stop 2024-03-30T04:59:33.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.023+0000 7f3ee4a00700 1 -- 172.21.15.123:0/2915139376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ee01037a0 msgr2=0x7f3ee0114060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:59:33.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.023+0000 7f3ee4a00700 1 --2- 172.21.15.123:0/2915139376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ee01037a0 0x7f3ee0114060 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc002a50 tx=0x7f3ecc004600 comp rx=0 tx=0).stop 2024-03-30T04:59:33.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.023+0000 7f3ee4a00700 1 -- 172.21.15.123:0/2915139376 shutdown_connections 2024-03-30T04:59:33.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.023+0000 7f3ee4a00700 1 --2- 172.21.15.123:0/2915139376 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3ec406cc30 0x7f3ec406f0e0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:33.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.023+0000 7f3ee4a00700 1 --2- 172.21.15.123:0/2915139376 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3ee0102e00 0x7f3ee0113b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:33.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.023+0000 7f3ee4a00700 1 --2- 172.21.15.123:0/2915139376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ee01037a0 0x7f3ee0114060 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:33.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.023+0000 7f3ee4a00700 1 -- 172.21.15.123:0/2915139376 >> 172.21.15.123:0/2915139376 conn(0x7f3ee00f89b0 msgr2=0x7f3ee00f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:59:33.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.023+0000 7f3ee4a00700 1 -- 172.21.15.123:0/2915139376 shutdown_connections 2024-03-30T04:59:33.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.023+0000 7f3ee4a00700 1 -- 172.21.15.123:0/2915139376 wait complete. 2024-03-30T04:59:33.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.160+0000 7f2f1d400700 1 -- 172.21.15.123:0/743353943 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f18102e00 msgr2=0x7f2f181031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:59:33.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.160+0000 7f2f1d400700 1 --2- 172.21.15.123:0/743353943 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f18102e00 0x7f2f181031d0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f2f08004940 tx=0x7f2f08009e30 comp rx=0 tx=0).stop 2024-03-30T04:59:33.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.160+0000 7f2f1d400700 1 -- 172.21.15.123:0/743353943 shutdown_connections 2024-03-30T04:59:33.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.160+0000 7f2f1d400700 1 --2- 172.21.15.123:0/743353943 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2f181037a0 0x7f2f180fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:33.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.160+0000 7f2f1d400700 1 --2- 172.21.15.123:0/743353943 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f18102e00 0x7f2f181031d0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:33.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.160+0000 7f2f1d400700 1 -- 172.21.15.123:0/743353943 >> 172.21.15.123:0/743353943 conn(0x7f2f180f89b0 msgr2=0x7f2f180fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:59:33.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.160+0000 7f2f1d400700 1 -- 172.21.15.123:0/743353943 shutdown_connections 2024-03-30T04:59:33.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.160+0000 7f2f1d400700 1 -- 172.21.15.123:0/743353943 wait complete. 2024-03-30T04:59:33.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.161+0000 7f2f1d400700 1 Processor -- start 2024-03-30T04:59:33.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.161+0000 7f2f1d400700 1 -- start start 2024-03-30T04:59:33.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.162+0000 7f2f1d400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f18102e00 0x7f2f180702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:59:33.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.162+0000 7f2f1d400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2f181037a0 0x7f2f18070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:59:33.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.162+0000 7f2f1d400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f18070d90 con 0x7f2f18102e00 2024-03-30T04:59:33.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.162+0000 7f2f1d400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f18070ed0 con 0x7f2f181037a0 2024-03-30T04:59:33.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.162+0000 7f2f16800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f18102e00 0x7f2f180702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:59:33.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.162+0000 7f2f16800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f18102e00 0x7f2f180702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48726/0 (socket says 172.21.15.123:48726) 2024-03-30T04:59:33.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.162+0000 7f2f16800700 1 -- 172.21.15.123:0/2253591134 learned_addr learned my addr 172.21.15.123:0/2253591134 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:59:33.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.162+0000 7f2f15e00700 1 --2- 172.21.15.123:0/2253591134 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2f181037a0 0x7f2f18070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:59:33.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.162+0000 7f2f16800700 1 -- 172.21.15.123:0/2253591134 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2f181037a0 msgr2=0x7f2f18070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:59:33.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.162+0000 7f2f16800700 1 --2- 172.21.15.123:0/2253591134 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2f181037a0 0x7f2f18070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:33.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.162+0000 7f2f16800700 1 -- 172.21.15.123:0/2253591134 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f08005040 con 0x7f2f18102e00 2024-03-30T04:59:33.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.163+0000 7f2f16800700 1 --2- 172.21.15.123:0/2253591134 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f18102e00 0x7f2f180702c0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f2f08009390 tx=0x7f2f08009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:59:33.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.163+0000 7f2f0ea00700 1 -- 172.21.15.123:0/2253591134 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f0800e040 con 0x7f2f18102e00 2024-03-30T04:59:33.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.163+0000 7f2f0ea00700 1 -- 172.21.15.123:0/2253591134 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2f08013420 con 0x7f2f18102e00 2024-03-30T04:59:33.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.163+0000 7f2f0ea00700 1 -- 172.21.15.123:0/2253591134 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f0801c630 con 0x7f2f18102e00 2024-03-30T04:59:33.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.163+0000 7f2f1d400700 1 -- 172.21.15.123:0/2253591134 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f18071120 con 0x7f2f18102e00 2024-03-30T04:59:33.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.163+0000 7f2f1d400700 1 -- 172.21.15.123:0/2253591134 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f1811a250 con 0x7f2f18102e00 2024-03-30T04:59:33.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.164+0000 7f2f1d400700 1 -- 172.21.15.123:0/2253591134 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f180fe550 con 0x7f2f18102e00 2024-03-30T04:59:33.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.165+0000 7f2f0ea00700 1 -- 172.21.15.123:0/2253591134 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f2f080139e0 con 0x7f2f18102e00 2024-03-30T04:59:33.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.166+0000 7f2f0ea00700 1 --2- 172.21.15.123:0/2253591134 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2efc06cc30 0x7f2efc06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:59:33.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.166+0000 7f2f0ea00700 1 -- 172.21.15.123:0/2253591134 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f2f0808d260 con 0x7f2f18102e00 2024-03-30T04:59:33.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.166+0000 7f2f15e00700 1 --2- 172.21.15.123:0/2253591134 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2efc06cc30 0x7f2efc06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:59:33.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.167+0000 7f2f15e00700 1 --2- 172.21.15.123:0/2253591134 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2efc06cc30 0x7f2efc06f0e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f2f18071c00 tx=0x7f2f04009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:59:33.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.170+0000 7f2f0ea00700 1 -- 172.21.15.123:0/2253591134 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2f080572a0 con 0x7f2f18102e00 2024-03-30T04:59:33.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:59:33 smithi135 ceph-mon[106011]: pgmap v851: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:33.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.380+0000 7f2f1d400700 1 -- 172.21.15.123:0/2253591134 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2f180719b0 con 0x7f2efc06cc30 2024-03-30T04:59:33.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.389+0000 7f2f0ea00700 1 -- 172.21.15.123:0/2253591134 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f2f180719b0 con 0x7f2efc06cc30 2024-03-30T04:59:33.388 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:59:33.388 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (28m) 6m ago 41m 14.6M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T04:59:33.388 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (25m) 6m ago 41m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T04:59:33.388 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (25m) 6m ago 40m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T04:59:33.388 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (28m) 6m ago 40m 73.5M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T04:59:33.388 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (21m) 6m ago 34m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T04:59:33.388 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (22m) 6m ago 34m 16.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T04:59:33.388 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (6m) 6m ago 34m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 87b073a6c9ca 2024-03-30T04:59:33.388 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (21m) 6m ago 34m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T04:59:33.388 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (31m) 6m ago 42m 510M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T04:59:33.389 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (30m) 6m ago 40m 456M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T04:59:33.389 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (27m) 6m ago 43m 70.0M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T04:59:33.389 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (26m) 6m ago 40m 51.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T04:59:33.389 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (29m) 6m ago 41m 8636k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T04:59:33.389 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (29m) 6m ago 40m 8678k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T04:59:33.389 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (25m) 6m ago 39m 107M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T04:59:33.389 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (25m) 6m ago 38m 97.2M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T04:59:33.389 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (24m) 6m ago 38m 89.8M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T04:59:33.389 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (23m) 6m ago 37m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T04:59:33.389 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (23m) 6m ago 37m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T04:59:33.389 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (22m) 6m ago 36m 88.8M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T04:59:33.389 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (29m) 6m ago 40m 63.9M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T04:59:33.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.392+0000 7f2f1d400700 1 -- 172.21.15.123:0/2253591134 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2efc06cc30 msgr2=0x7f2efc06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:59:33.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.392+0000 7f2f1d400700 1 --2- 172.21.15.123:0/2253591134 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2efc06cc30 0x7f2efc06f0e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f2f18071c00 tx=0x7f2f04009ee0 comp rx=0 tx=0).stop 2024-03-30T04:59:33.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.392+0000 7f2f1d400700 1 -- 172.21.15.123:0/2253591134 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f18102e00 msgr2=0x7f2f180702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:59:33.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.392+0000 7f2f1d400700 1 --2- 172.21.15.123:0/2253591134 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f18102e00 0x7f2f180702c0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f2f08009390 tx=0x7f2f08009e30 comp rx=0 tx=0).stop 2024-03-30T04:59:33.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.392+0000 7f2f1d400700 1 -- 172.21.15.123:0/2253591134 shutdown_connections 2024-03-30T04:59:33.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.392+0000 7f2f1d400700 1 --2- 172.21.15.123:0/2253591134 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2efc06cc30 0x7f2efc06f0e0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:33.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.392+0000 7f2f1d400700 1 --2- 172.21.15.123:0/2253591134 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f18102e00 0x7f2f180702c0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:33.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.392+0000 7f2f1d400700 1 --2- 172.21.15.123:0/2253591134 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2f181037a0 0x7f2f18070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:33.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.392+0000 7f2f1d400700 1 -- 172.21.15.123:0/2253591134 >> 172.21.15.123:0/2253591134 conn(0x7f2f180f89b0 msgr2=0x7f2f180f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:59:33.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.392+0000 7f2f1d400700 1 -- 172.21.15.123:0/2253591134 shutdown_connections 2024-03-30T04:59:33.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.392+0000 7f2f1d400700 1 -- 172.21.15.123:0/2253591134 wait complete. 2024-03-30T04:59:33.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:59:33 smithi123 ceph-mon[139369]: pgmap v851: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:33.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.525+0000 7f2120a00700 1 -- 172.21.15.123:0/1273754030 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f211c102e00 msgr2=0x7f211c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:59:33.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.525+0000 7f2120a00700 1 --2- 172.21.15.123:0/1273754030 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f211c102e00 0x7f211c1031d0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f210c004940 tx=0x7f210c009e30 comp rx=0 tx=0).stop 2024-03-30T04:59:33.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.525+0000 7f2120a00700 1 -- 172.21.15.123:0/1273754030 shutdown_connections 2024-03-30T04:59:33.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.525+0000 7f2120a00700 1 --2- 172.21.15.123:0/1273754030 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f211c1037a0 0x7f211c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:33.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.525+0000 7f2120a00700 1 --2- 172.21.15.123:0/1273754030 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f211c102e00 0x7f211c1031d0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:33.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.525+0000 7f2120a00700 1 -- 172.21.15.123:0/1273754030 >> 172.21.15.123:0/1273754030 conn(0x7f211c0f89b0 msgr2=0x7f211c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:59:33.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.525+0000 7f2120a00700 1 -- 172.21.15.123:0/1273754030 shutdown_connections 2024-03-30T04:59:33.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.525+0000 7f2120a00700 1 -- 172.21.15.123:0/1273754030 wait complete. 2024-03-30T04:59:33.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.526+0000 7f2120a00700 1 Processor -- start 2024-03-30T04:59:33.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.526+0000 7f2120a00700 1 -- start start 2024-03-30T04:59:33.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.526+0000 7f2120a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f211c102e00 0x7f211c115cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:59:33.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.526+0000 7f2120a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f211c1037a0 0x7f211c1161f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:59:33.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.526+0000 7f2120a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f211c1168d0 con 0x7f211c102e00 2024-03-30T04:59:33.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.526+0000 7f2120a00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f211c10fc10 con 0x7f211c1037a0 2024-03-30T04:59:33.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.526+0000 7f2119e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f211c102e00 0x7f211c115cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:59:33.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.526+0000 7f2119e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f211c102e00 0x7f211c115cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48740/0 (socket says 172.21.15.123:48740) 2024-03-30T04:59:33.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.526+0000 7f2119e00700 1 -- 172.21.15.123:0/4082993882 learned_addr learned my addr 172.21.15.123:0/4082993882 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:59:33.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.527+0000 7f2119400700 1 --2- 172.21.15.123:0/4082993882 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f211c1037a0 0x7f211c1161f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:59:33.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.527+0000 7f2119e00700 1 -- 172.21.15.123:0/4082993882 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f211c1037a0 msgr2=0x7f211c1161f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:59:33.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.527+0000 7f2119e00700 1 --2- 172.21.15.123:0/4082993882 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f211c1037a0 0x7f211c1161f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:33.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.527+0000 7f2119e00700 1 -- 172.21.15.123:0/4082993882 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f210c005040 con 0x7f211c102e00 2024-03-30T04:59:33.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.527+0000 7f2119e00700 1 --2- 172.21.15.123:0/4082993882 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f211c102e00 0x7f211c115cb0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f210c009360 tx=0x7f210c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:59:33.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.527+0000 7f2112000700 1 -- 172.21.15.123:0/4082993882 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f210c00e040 con 0x7f211c102e00 2024-03-30T04:59:33.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.527+0000 7f2112000700 1 -- 172.21.15.123:0/4082993882 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f210c013420 con 0x7f211c102e00 2024-03-30T04:59:33.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.527+0000 7f2120a00700 1 -- 172.21.15.123:0/4082993882 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f211c10fe30 con 0x7f211c102e00 2024-03-30T04:59:33.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.527+0000 7f2112000700 1 -- 172.21.15.123:0/4082993882 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f210c01c5f0 con 0x7f211c102e00 2024-03-30T04:59:33.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.527+0000 7f2120a00700 1 -- 172.21.15.123:0/4082993882 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f211c1103f0 con 0x7f211c102e00 2024-03-30T04:59:33.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.529+0000 7f2120a00700 1 -- 172.21.15.123:0/4082993882 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f211c0fe550 con 0x7f211c102e00 2024-03-30T04:59:33.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.529+0000 7f2112000700 1 -- 172.21.15.123:0/4082993882 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f210c013590 con 0x7f211c102e00 2024-03-30T04:59:33.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.531+0000 7f2112000700 1 --2- 172.21.15.123:0/4082993882 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f210006cc30 0x7f210006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:59:33.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.531+0000 7f2112000700 1 -- 172.21.15.123:0/4082993882 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f210c08d240 con 0x7f211c102e00 2024-03-30T04:59:33.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.531+0000 7f2119400700 1 --2- 172.21.15.123:0/4082993882 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f210006cc30 0x7f210006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:59:33.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.531+0000 7f2119400700 1 --2- 172.21.15.123:0/4082993882 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f210006cc30 0x7f210006f0e0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f2108006fd0 tx=0x7f2108008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:59:33.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.535+0000 7f2112000700 1 -- 172.21.15.123:0/4082993882 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f210c05ad10 con 0x7f211c102e00 2024-03-30T04:59:33.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.822+0000 7f2120a00700 1 -- 172.21.15.123:0/4082993882 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f211c05f5f0 con 0x7f211c102e00 2024-03-30T04:59:33.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.823+0000 7f2112000700 1 -- 172.21.15.123:0/4082993882 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f210c05a8a0 con 0x7f211c102e00 2024-03-30T04:59:33.821 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:59:33.822 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T04:59:33.822 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:59:33.822 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:59:33.822 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T04:59:33.822 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T04:59:33.822 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:59:33.822 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T04:59:33.822 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T04:59:33.822 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:59:33.822 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T04:59:33.822 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 4 2024-03-30T04:59:33.822 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T04:59:33.822 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T04:59:33.823 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 14 2024-03-30T04:59:33.823 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T04:59:33.823 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:59:33.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.825+0000 7f2120a00700 1 -- 172.21.15.123:0/4082993882 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f210006cc30 msgr2=0x7f210006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:59:33.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.825+0000 7f2120a00700 1 --2- 172.21.15.123:0/4082993882 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f210006cc30 0x7f210006f0e0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f2108006fd0 tx=0x7f2108008040 comp rx=0 tx=0).stop 2024-03-30T04:59:33.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.825+0000 7f2120a00700 1 -- 172.21.15.123:0/4082993882 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f211c102e00 msgr2=0x7f211c115cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:59:33.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.825+0000 7f2120a00700 1 --2- 172.21.15.123:0/4082993882 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f211c102e00 0x7f211c115cb0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f210c009360 tx=0x7f210c009e30 comp rx=0 tx=0).stop 2024-03-30T04:59:33.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.825+0000 7f2120a00700 1 -- 172.21.15.123:0/4082993882 shutdown_connections 2024-03-30T04:59:33.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.825+0000 7f2120a00700 1 --2- 172.21.15.123:0/4082993882 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f210006cc30 0x7f210006f0e0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:33.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.825+0000 7f2120a00700 1 --2- 172.21.15.123:0/4082993882 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f211c102e00 0x7f211c115cb0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:33.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.825+0000 7f2120a00700 1 --2- 172.21.15.123:0/4082993882 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f211c1037a0 0x7f211c1161f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:33.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.825+0000 7f2120a00700 1 -- 172.21.15.123:0/4082993882 >> 172.21.15.123:0/4082993882 conn(0x7f211c0f89b0 msgr2=0x7f211c0f9a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:59:33.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.825+0000 7f2120a00700 1 -- 172.21.15.123:0/4082993882 shutdown_connections 2024-03-30T04:59:33.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.826+0000 7f2120a00700 1 -- 172.21.15.123:0/4082993882 wait complete. 2024-03-30T04:59:33.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.959+0000 7f9231200700 1 -- 172.21.15.123:0/1269866204 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f922c1035e0 msgr2=0x7f922c0fcc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:59:33.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.959+0000 7f9231200700 1 --2- 172.21.15.123:0/1269866204 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f922c1035e0 0x7f922c0fcc50 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f9218004940 tx=0x7f9218009e30 comp rx=0 tx=0).stop 2024-03-30T04:59:33.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.959+0000 7f9231200700 1 -- 172.21.15.123:0/1269866204 shutdown_connections 2024-03-30T04:59:33.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.959+0000 7f9231200700 1 --2- 172.21.15.123:0/1269866204 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f922c1035e0 0x7f922c0fcc50 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:33.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.959+0000 7f9231200700 1 --2- 172.21.15.123:0/1269866204 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f922c102c40 0x7f922c103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:33.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.959+0000 7f9231200700 1 -- 172.21.15.123:0/1269866204 >> 172.21.15.123:0/1269866204 conn(0x7f922c0f8830 msgr2=0x7f922c0fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:59:33.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.959+0000 7f9231200700 1 -- 172.21.15.123:0/1269866204 shutdown_connections 2024-03-30T04:59:33.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.959+0000 7f9231200700 1 -- 172.21.15.123:0/1269866204 wait complete. 2024-03-30T04:59:33.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.960+0000 7f9231200700 1 Processor -- start 2024-03-30T04:59:33.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.960+0000 7f9231200700 1 -- start start 2024-03-30T04:59:33.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.961+0000 7f9231200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f922c102c40 0x7f922c115b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:59:33.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.961+0000 7f9231200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f922c1035e0 0x7f922c116040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:59:33.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.961+0000 7f9231200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f922c116720 con 0x7f922c1035e0 2024-03-30T04:59:33.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.961+0000 7f9231200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f922c10fa80 con 0x7f922c102c40 2024-03-30T04:59:33.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.961+0000 7f922aa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f922c1035e0 0x7f922c116040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:59:33.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.961+0000 7f922aa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f922c1035e0 0x7f922c116040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53526/0 (socket says 172.21.15.123:53526) 2024-03-30T04:59:33.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.961+0000 7f922aa00700 1 -- 172.21.15.123:0/1414735276 learned_addr learned my addr 172.21.15.123:0/1414735276 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:59:33.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.961+0000 7f922b400700 1 --2- 172.21.15.123:0/1414735276 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f922c102c40 0x7f922c115b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:59:33.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.961+0000 7f922aa00700 1 -- 172.21.15.123:0/1414735276 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f922c102c40 msgr2=0x7f922c115b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:59:33.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.961+0000 7f922aa00700 1 --2- 172.21.15.123:0/1414735276 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f922c102c40 0x7f922c115b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:33.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.961+0000 7f922aa00700 1 -- 172.21.15.123:0/1414735276 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9218005040 con 0x7f922c1035e0 2024-03-30T04:59:33.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.961+0000 7f922aa00700 1 --2- 172.21.15.123:0/1414735276 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f922c1035e0 0x7f922c116040 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f9218009390 tx=0x7f9218009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:59:33.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.962+0000 7f9223400700 1 -- 172.21.15.123:0/1414735276 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f921800e040 con 0x7f922c1035e0 2024-03-30T04:59:33.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.962+0000 7f9223400700 1 -- 172.21.15.123:0/1414735276 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9218013420 con 0x7f922c1035e0 2024-03-30T04:59:33.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.962+0000 7f9223400700 1 -- 172.21.15.123:0/1414735276 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f921801c630 con 0x7f922c1035e0 2024-03-30T04:59:33.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.962+0000 7f9231200700 1 -- 172.21.15.123:0/1414735276 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f922c10fd00 con 0x7f922c1035e0 2024-03-30T04:59:33.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.962+0000 7f9231200700 1 -- 172.21.15.123:0/1414735276 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f922c1102c0 con 0x7f922c1035e0 2024-03-30T04:59:33.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.963+0000 7f9231200700 1 -- 172.21.15.123:0/1414735276 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f922c0fe390 con 0x7f922c1035e0 2024-03-30T04:59:33.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.964+0000 7f9223400700 1 -- 172.21.15.123:0/1414735276 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f9218013a00 con 0x7f922c1035e0 2024-03-30T04:59:33.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.965+0000 7f9223400700 1 --2- 172.21.15.123:0/1414735276 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f921006cc30 0x7f921006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:59:33.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.965+0000 7f9223400700 1 -- 172.21.15.123:0/1414735276 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f921808d2c0 con 0x7f922c1035e0 2024-03-30T04:59:33.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.965+0000 7f922b400700 1 --2- 172.21.15.123:0/1414735276 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f921006cc30 0x7f921006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:59:33.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.965+0000 7f922b400700 1 --2- 172.21.15.123:0/1414735276 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f921006cc30 0x7f921006f0e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f921c006fd0 tx=0x7f921c008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:59:33.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:33.969+0000 7f9223400700 1 -- 172.21.15.123:0/1414735276 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f921805ad90 con 0x7f922c1035e0 2024-03-30T04:59:34.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.215+0000 7f9231200700 1 -- 172.21.15.123:0/1414735276 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f922c109c50 con 0x7f922c1035e0 2024-03-30T04:59:34.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.216+0000 7f9223400700 1 -- 172.21.15.123:0/1414735276 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 34 v34) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f921805a920 con 0x7f922c1035e0 2024-03-30T04:59:34.214 INFO:teuthology.orchestra.run.smithi123.stdout:e34 2024-03-30T04:59:34.214 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:59:34.214 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:59:34.214 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T04:59:34.214 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:59:34.214 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:59:34.215 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T04:59:34.215 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T04:59:34.215 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T04:59:34.215 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T04:59:34.215 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T04:59:34.215 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T04:59:34.215 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T04:59:34.215 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T04:59:34.215 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T04:59:34.215 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T04:59:34.215 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T04:59:34.215 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T04:59:34.215 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T04:59:34.215 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T04:59:34.216 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:59:34.216 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T04:59:34.216 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T04:59:34.216 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T04:59:34.216 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T04:59:34.216 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T04:59:34.216 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T04:59:34.216 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T04:59:34.216 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T04:59:34.216 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T04:59:34.216 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T04:59:34.216 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T04:59:34.216 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T04:59:34.216 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:59:34.217 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:59:34.217 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T04:59:34.217 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T04:59:34.217 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:59:34.217 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:59:34.217 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:59:34.217 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44975} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/3788988014,v1:172.21.15.135:6827/3788988014] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:59:34.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.218+0000 7f9231200700 1 -- 172.21.15.123:0/1414735276 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f921006cc30 msgr2=0x7f921006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:59:34.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.218+0000 7f9231200700 1 --2- 172.21.15.123:0/1414735276 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f921006cc30 0x7f921006f0e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f921c006fd0 tx=0x7f921c008040 comp rx=0 tx=0).stop 2024-03-30T04:59:34.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.218+0000 7f9231200700 1 -- 172.21.15.123:0/1414735276 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f922c1035e0 msgr2=0x7f922c116040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:59:34.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.218+0000 7f9231200700 1 --2- 172.21.15.123:0/1414735276 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f922c1035e0 0x7f922c116040 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f9218009390 tx=0x7f9218009e30 comp rx=0 tx=0).stop 2024-03-30T04:59:34.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.218+0000 7f9231200700 1 -- 172.21.15.123:0/1414735276 shutdown_connections 2024-03-30T04:59:34.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.218+0000 7f9231200700 1 --2- 172.21.15.123:0/1414735276 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f921006cc30 0x7f921006f0e0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:34.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.218+0000 7f9231200700 1 --2- 172.21.15.123:0/1414735276 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f922c102c40 0x7f922c115b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:34.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.218+0000 7f9231200700 1 --2- 172.21.15.123:0/1414735276 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f922c1035e0 0x7f922c116040 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:34.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.218+0000 7f9231200700 1 -- 172.21.15.123:0/1414735276 >> 172.21.15.123:0/1414735276 conn(0x7f922c0f8830 msgr2=0x7f922c0f9810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:59:34.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.218+0000 7f9231200700 1 -- 172.21.15.123:0/1414735276 shutdown_connections 2024-03-30T04:59:34.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.218+0000 7f9231200700 1 -- 172.21.15.123:0/1414735276 wait complete. 2024-03-30T04:59:34.219 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 34 2024-03-30T04:59:34.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:59:34 smithi135 ceph-mon[106011]: from='client.35552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:59:34.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:59:34 smithi135 ceph-mon[106011]: from='client.35556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:59:34.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:59:34 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/4082993882' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:59:34.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.353+0000 7f9c9de00700 1 -- 172.21.15.123:0/3204414571 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c981037a0 msgr2=0x7f9c980fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:59:34.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.353+0000 7f9c9de00700 1 --2- 172.21.15.123:0/3204414571 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c981037a0 0x7f9c980fce10 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f9c84004940 tx=0x7f9c84009e30 comp rx=0 tx=0).stop 2024-03-30T04:59:34.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.353+0000 7f9c9de00700 1 -- 172.21.15.123:0/3204414571 shutdown_connections 2024-03-30T04:59:34.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.353+0000 7f9c9de00700 1 --2- 172.21.15.123:0/3204414571 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c981037a0 0x7f9c980fce10 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:34.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.353+0000 7f9c9de00700 1 --2- 172.21.15.123:0/3204414571 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c98102e00 0x7f9c981031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:34.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.353+0000 7f9c9de00700 1 -- 172.21.15.123:0/3204414571 >> 172.21.15.123:0/3204414571 conn(0x7f9c980f89b0 msgr2=0x7f9c980fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:59:34.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.353+0000 7f9c9de00700 1 -- 172.21.15.123:0/3204414571 shutdown_connections 2024-03-30T04:59:34.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.353+0000 7f9c9de00700 1 -- 172.21.15.123:0/3204414571 wait complete. 2024-03-30T04:59:34.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.354+0000 7f9c9de00700 1 Processor -- start 2024-03-30T04:59:34.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.354+0000 7f9c9de00700 1 -- start start 2024-03-30T04:59:34.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.354+0000 7f9c9de00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c98102e00 0x7f9c980702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:59:34.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.354+0000 7f9c9de00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c981037a0 0x7f9c98070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:59:34.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.354+0000 7f9c9de00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c98070d90 con 0x7f9c981037a0 2024-03-30T04:59:34.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.354+0000 7f9c9de00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c98070ed0 con 0x7f9c98102e00 2024-03-30T04:59:34.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.355+0000 7f9c96000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c981037a0 0x7f9c98070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:59:34.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.355+0000 7f9c96000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c981037a0 0x7f9c98070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53544/0 (socket says 172.21.15.123:53544) 2024-03-30T04:59:34.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.355+0000 7f9c96000700 1 -- 172.21.15.123:0/643494147 learned_addr learned my addr 172.21.15.123:0/643494147 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:59:34.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.355+0000 7f9c96a00700 1 --2- 172.21.15.123:0/643494147 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c98102e00 0x7f9c980702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:59:34.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.355+0000 7f9c96000700 1 -- 172.21.15.123:0/643494147 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c98102e00 msgr2=0x7f9c980702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:59:34.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.355+0000 7f9c96000700 1 --2- 172.21.15.123:0/643494147 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c98102e00 0x7f9c980702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:34.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.355+0000 7f9c96000700 1 -- 172.21.15.123:0/643494147 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c84005040 con 0x7f9c981037a0 2024-03-30T04:59:34.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.355+0000 7f9c96000700 1 --2- 172.21.15.123:0/643494147 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c981037a0 0x7f9c98070800 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f9c8400b0d0 tx=0x7f9c84009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:59:34.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.356+0000 7f9c8ea00700 1 -- 172.21.15.123:0/643494147 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c84021070 con 0x7f9c981037a0 2024-03-30T04:59:34.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.356+0000 7f9c8ea00700 1 -- 172.21.15.123:0/643494147 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9c84013e00 con 0x7f9c981037a0 2024-03-30T04:59:34.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.356+0000 7f9c9de00700 1 -- 172.21.15.123:0/643494147 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c98071120 con 0x7f9c981037a0 2024-03-30T04:59:34.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.356+0000 7f9c8ea00700 1 -- 172.21.15.123:0/643494147 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c8400e070 con 0x7f9c981037a0 2024-03-30T04:59:34.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.356+0000 7f9c9de00700 1 -- 172.21.15.123:0/643494147 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c9811a140 con 0x7f9c981037a0 2024-03-30T04:59:34.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.357+0000 7f9c9de00700 1 -- 172.21.15.123:0/643494147 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c980fe550 con 0x7f9c981037a0 2024-03-30T04:59:34.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.358+0000 7f9c8ea00700 1 -- 172.21.15.123:0/643494147 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f9c8401c400 con 0x7f9c981037a0 2024-03-30T04:59:34.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.359+0000 7f9c8ea00700 1 --2- 172.21.15.123:0/643494147 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9c7c06cc20 0x7f9c7c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:59:34.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.359+0000 7f9c8ea00700 1 -- 172.21.15.123:0/643494147 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f9c8408d1a0 con 0x7f9c981037a0 2024-03-30T04:59:34.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.359+0000 7f9c96a00700 1 --2- 172.21.15.123:0/643494147 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9c7c06cc20 0x7f9c7c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:59:34.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.359+0000 7f9c96a00700 1 --2- 172.21.15.123:0/643494147 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9c7c06cc20 0x7f9c7c06f0d0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f9c88009fd0 tx=0x7f9c8800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:59:34.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.363+0000 7f9c8ea00700 1 -- 172.21.15.123:0/643494147 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9c84057160 con 0x7f9c981037a0 2024-03-30T04:59:34.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:59:34 smithi123 ceph-mon[139369]: from='client.35552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:59:34.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:59:34 smithi123 ceph-mon[139369]: from='client.35556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:59:34.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:59:34 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/4082993882' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T04:59:34.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.586+0000 7f9c9de00700 1 -- 172.21.15.123:0/643494147 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9c98109e40 con 0x7f9c7c06cc20 2024-03-30T04:59:34.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.588+0000 7f9c8ea00700 1 -- 172.21.15.123:0/643494147 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f9c8800f040 con 0x7f9c7c06cc20 2024-03-30T04:59:34.586 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T04:59:34.586 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T04:59:34.586 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T04:59:34.586 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T04:59:34.586 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T04:59:34.586 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T04:59:34.586 INFO:teuthology.orchestra.run.smithi123.stdout: "mds", 2024-03-30T04:59:34.587 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T04:59:34.587 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T04:59:34.587 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T04:59:34.587 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T04:59:34.587 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "16/21 daemons upgraded", 2024-03-30T04:59:34.587 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T04:59:34.587 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T04:59:34.587 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T04:59:34.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.590+0000 7f9c9de00700 1 -- 172.21.15.123:0/643494147 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9c7c06cc20 msgr2=0x7f9c7c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:59:34.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.590+0000 7f9c9de00700 1 --2- 172.21.15.123:0/643494147 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9c7c06cc20 0x7f9c7c06f0d0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f9c88009fd0 tx=0x7f9c8800b040 comp rx=0 tx=0).stop 2024-03-30T04:59:34.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.590+0000 7f9c9de00700 1 -- 172.21.15.123:0/643494147 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c981037a0 msgr2=0x7f9c98070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:59:34.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.590+0000 7f9c9de00700 1 --2- 172.21.15.123:0/643494147 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c981037a0 0x7f9c98070800 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f9c8400b0d0 tx=0x7f9c84009e30 comp rx=0 tx=0).stop 2024-03-30T04:59:34.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.590+0000 7f9c9de00700 1 -- 172.21.15.123:0/643494147 shutdown_connections 2024-03-30T04:59:34.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.590+0000 7f9c9de00700 1 --2- 172.21.15.123:0/643494147 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9c7c06cc20 0x7f9c7c06f0d0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:34.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.590+0000 7f9c9de00700 1 --2- 172.21.15.123:0/643494147 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c98102e00 0x7f9c980702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:34.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.590+0000 7f9c9de00700 1 --2- 172.21.15.123:0/643494147 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c981037a0 0x7f9c98070800 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:34.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.590+0000 7f9c9de00700 1 -- 172.21.15.123:0/643494147 >> 172.21.15.123:0/643494147 conn(0x7f9c980f89b0 msgr2=0x7f9c980f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:59:34.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.591+0000 7f9c9de00700 1 -- 172.21.15.123:0/643494147 shutdown_connections 2024-03-30T04:59:34.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.591+0000 7f9c9de00700 1 -- 172.21.15.123:0/643494147 wait complete. 2024-03-30T04:59:34.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.724+0000 7f44b2800700 1 -- 172.21.15.123:0/350967560 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44ac102e00 msgr2=0x7f44ac1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:59:34.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.724+0000 7f44b2800700 1 --2- 172.21.15.123:0/350967560 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44ac102e00 0x7f44ac1031d0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f449c004940 tx=0x7f449c009e30 comp rx=0 tx=0).stop 2024-03-30T04:59:34.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.724+0000 7f44b2800700 1 -- 172.21.15.123:0/350967560 shutdown_connections 2024-03-30T04:59:34.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.724+0000 7f44b2800700 1 --2- 172.21.15.123:0/350967560 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f44ac1037a0 0x7f44ac0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:34.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.724+0000 7f44b2800700 1 --2- 172.21.15.123:0/350967560 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44ac102e00 0x7f44ac1031d0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:34.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.724+0000 7f44b2800700 1 -- 172.21.15.123:0/350967560 >> 172.21.15.123:0/350967560 conn(0x7f44ac0f89b0 msgr2=0x7f44ac0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:59:34.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.724+0000 7f44b2800700 1 -- 172.21.15.123:0/350967560 shutdown_connections 2024-03-30T04:59:34.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.724+0000 7f44b2800700 1 -- 172.21.15.123:0/350967560 wait complete. 2024-03-30T04:59:34.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.725+0000 7f44b2800700 1 Processor -- start 2024-03-30T04:59:34.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.725+0000 7f44b2800700 1 -- start start 2024-03-30T04:59:34.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.726+0000 7f44b2800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44ac102e00 0x7f44ac113a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:59:34.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.726+0000 7f44b2800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f44ac1037a0 0x7f44ac113fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:59:34.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.726+0000 7f44b2800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44ac1145f0 con 0x7f44ac102e00 2024-03-30T04:59:34.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.726+0000 7f44b2800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44ac10d9e0 con 0x7f44ac1037a0 2024-03-30T04:59:34.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.726+0000 7f44ab400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44ac102e00 0x7f44ac113a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:59:34.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.726+0000 7f44ab400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44ac102e00 0x7f44ac113a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53566/0 (socket says 172.21.15.123:53566) 2024-03-30T04:59:34.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.726+0000 7f44ab400700 1 -- 172.21.15.123:0/1069874932 learned_addr learned my addr 172.21.15.123:0/1069874932 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T04:59:34.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.726+0000 7f44aaa00700 1 --2- 172.21.15.123:0/1069874932 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f44ac1037a0 0x7f44ac113fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:59:34.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.726+0000 7f44ab400700 1 -- 172.21.15.123:0/1069874932 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f44ac1037a0 msgr2=0x7f44ac113fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:59:34.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.726+0000 7f44ab400700 1 --2- 172.21.15.123:0/1069874932 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f44ac1037a0 0x7f44ac113fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:34.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.726+0000 7f44ab400700 1 -- 172.21.15.123:0/1069874932 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f449c005040 con 0x7f44ac102e00 2024-03-30T04:59:34.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.726+0000 7f44aaa00700 1 --2- 172.21.15.123:0/1069874932 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f44ac1037a0 0x7f44ac113fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:59:34.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.726+0000 7f44ab400700 1 --2- 172.21.15.123:0/1069874932 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44ac102e00 0x7f44ac113a60 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f449c007f50 tx=0x7f449c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:59:34.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.727+0000 7f44a3400700 1 -- 172.21.15.123:0/1069874932 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f449c021070 con 0x7f44ac102e00 2024-03-30T04:59:34.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.727+0000 7f44a3400700 1 -- 172.21.15.123:0/1069874932 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f449c013e00 con 0x7f44ac102e00 2024-03-30T04:59:34.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.727+0000 7f44b2800700 1 -- 172.21.15.123:0/1069874932 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44ac10dc60 con 0x7f44ac102e00 2024-03-30T04:59:34.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.727+0000 7f44a3400700 1 -- 172.21.15.123:0/1069874932 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f449c00e070 con 0x7f44ac102e00 2024-03-30T04:59:34.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.727+0000 7f44b2800700 1 -- 172.21.15.123:0/1069874932 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44ac10e220 con 0x7f44ac102e00 2024-03-30T04:59:34.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.728+0000 7f44b2800700 1 -- 172.21.15.123:0/1069874932 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44ac0fe550 con 0x7f44ac102e00 2024-03-30T04:59:34.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.729+0000 7f44a3400700 1 -- 172.21.15.123:0/1069874932 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f449c013950 con 0x7f44ac102e00 2024-03-30T04:59:34.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.731+0000 7f44a3400700 1 --2- 172.21.15.123:0/1069874932 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f449006cc30 0x7f449006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:59:34.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.731+0000 7f44a3400700 1 -- 172.21.15.123:0/1069874932 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f449c08d2b0 con 0x7f44ac102e00 2024-03-30T04:59:34.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.731+0000 7f44aaa00700 1 --2- 172.21.15.123:0/1069874932 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f449006cc30 0x7f449006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:59:34.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.731+0000 7f44aaa00700 1 --2- 172.21.15.123:0/1069874932 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f449006cc30 0x7f449006f0e0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f4498004600 tx=0x7f449800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:59:34.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:34.736+0000 7f44a3400700 1 -- 172.21.15.123:0/1069874932 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f449c05ad80 con 0x7f44ac102e00 2024-03-30T04:59:35.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:35.014+0000 7f44b2800700 1 -- 172.21.15.123:0/1069874932 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f44ac109de0 con 0x7f44ac102e00 2024-03-30T04:59:35.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:35.015+0000 7f44a3400700 1 -- 172.21.15.123:0/1069874932 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f449c05a910 con 0x7f44ac102e00 2024-03-30T04:59:35.013 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T04:59:35.013 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-30T04:59:35.013 INFO:teuthology.orchestra.run.smithi123.stdout: host smithi135 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi135/config 2024-03-30T04:59:35.013 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-30T04:59:35.013 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T04:59:35.013 INFO:teuthology.orchestra.run.smithi123.stdout:Traceback (most recent call last): 2024-03-30T04:59:35.014 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-30T04:59:35.014 INFO:teuthology.orchestra.run.smithi123.stdout: "__main__", mod_spec) 2024-03-30T04:59:35.014 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-30T04:59:35.014 INFO:teuthology.orchestra.run.smithi123.stdout: exec(code, run_globals) 2024-03-30T04:59:35.014 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10889, in 2024-03-30T04:59:35.014 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10877, in main 2024-03-30T04:59:35.014 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-30T04:59:35.014 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-30T04:59:35.014 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-30T04:59:35.015 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-30T04:59:35.015 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-03-30T04:59:35.015 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-30T04:59:35.015 INFO:teuthology.orchestra.run.smithi123.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T04:59:35.015 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T04:59:35.015 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T04:59:35.015 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T04:59:35.015 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T04:59:35.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:35.017+0000 7f44b2800700 1 -- 172.21.15.123:0/1069874932 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f449006cc30 msgr2=0x7f449006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:59:35.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:35.017+0000 7f44b2800700 1 --2- 172.21.15.123:0/1069874932 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f449006cc30 0x7f449006f0e0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f4498004600 tx=0x7f449800b040 comp rx=0 tx=0).stop 2024-03-30T04:59:35.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:35.017+0000 7f44b2800700 1 -- 172.21.15.123:0/1069874932 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44ac102e00 msgr2=0x7f44ac113a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:59:35.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:35.017+0000 7f44b2800700 1 --2- 172.21.15.123:0/1069874932 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44ac102e00 0x7f44ac113a60 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f449c007f50 tx=0x7f449c009e30 comp rx=0 tx=0).stop 2024-03-30T04:59:35.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:35.017+0000 7f44b2800700 1 -- 172.21.15.123:0/1069874932 shutdown_connections 2024-03-30T04:59:35.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:35.017+0000 7f44b2800700 1 --2- 172.21.15.123:0/1069874932 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f449006cc30 0x7f449006f0e0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:35.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:35.017+0000 7f44b2800700 1 --2- 172.21.15.123:0/1069874932 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44ac102e00 0x7f44ac113a60 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:35.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:35.017+0000 7f44b2800700 1 --2- 172.21.15.123:0/1069874932 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f44ac1037a0 0x7f44ac113fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:59:35.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:35.017+0000 7f44b2800700 1 -- 172.21.15.123:0/1069874932 >> 172.21.15.123:0/1069874932 conn(0x7f44ac0f89b0 msgr2=0x7f44ac0f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:59:35.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:35.017+0000 7f44b2800700 1 -- 172.21.15.123:0/1069874932 shutdown_connections 2024-03-30T04:59:35.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T04:59:35.017+0000 7f44b2800700 1 -- 172.21.15.123:0/1069874932 wait complete. 2024-03-30T04:59:35.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:59:35 smithi135 ceph-mon[106011]: from='client.35560 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:59:35.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:59:35 smithi135 ceph-mon[106011]: pgmap v852: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:35.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:59:35 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/1414735276' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:59:35.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:59:35 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/1069874932' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:59:35.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:59:35 smithi123 ceph-mon[139369]: from='client.35560 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:59:35.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:59:35 smithi123 ceph-mon[139369]: pgmap v852: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:35.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:59:35 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/1414735276' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:59:35.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:59:35 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/1069874932' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T04:59:36.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:59:36 smithi135 ceph-mon[106011]: from='client.35572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:59:36.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:59:36 smithi123 ceph-mon[139369]: from='client.35572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:59:37.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:59:37 smithi135 ceph-mon[106011]: pgmap v853: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:37.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:59:37 smithi123 ceph-mon[139369]: pgmap v853: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:39.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:59:39 smithi135 ceph-mon[106011]: pgmap v854: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:39.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:59:39 smithi123 ceph-mon[139369]: pgmap v854: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:41.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:59:41 smithi135 ceph-mon[106011]: pgmap v855: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:41.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:59:41 smithi123 ceph-mon[139369]: pgmap v855: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:43.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:59:43 smithi135 ceph-mon[106011]: pgmap v856: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:43.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:59:43 smithi123 ceph-mon[139369]: pgmap v856: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:45.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:59:45 smithi135 ceph-mon[106011]: pgmap v857: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:45.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:59:45 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:59:45.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:59:45 smithi123 ceph-mon[139369]: pgmap v857: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:45.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:59:45 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T04:59:47.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:59:47 smithi123 ceph-mon[139369]: pgmap v858: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:47.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:59:47 smithi135 ceph-mon[106011]: pgmap v858: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:49.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:59:49 smithi123 ceph-mon[139369]: pgmap v859: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:49.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:59:49 smithi135 ceph-mon[106011]: pgmap v859: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:51.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:59:51 smithi123 ceph-mon[139369]: pgmap v860: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:51.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:59:51 smithi135 ceph-mon[106011]: pgmap v860: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:53.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:59:53 smithi123 ceph-mon[139369]: pgmap v861: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:53.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:59:53 smithi135 ceph-mon[106011]: pgmap v861: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:55.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:59:55 smithi123 ceph-mon[139369]: pgmap v862: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:55.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:59:55 smithi135 ceph-mon[106011]: pgmap v862: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:57.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:59:57 smithi123 ceph-mon[139369]: pgmap v863: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:57.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:59:57 smithi135 ceph-mon[106011]: pgmap v863: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:59.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 04:59:59 smithi123 ceph-mon[139369]: pgmap v864: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T04:59:59.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 04:59:59 smithi135 ceph-mon[106011]: pgmap v864: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:00.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:00 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-mon-smithi123[139346]: 2024-03-30T05:00:00.000+0000 7f50e7e00700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T05:00:00.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:00 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-mon-smithi123[139346]: 2024-03-30T05:00:00.000+0000 7f50e7e00700 -1 log_channel(cluster) log [ERR] : [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-30T05:00:00.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:00 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-mon-smithi123[139346]: 2024-03-30T05:00:00.000+0000 7f50e7e00700 -1 log_channel(cluster) log [ERR] : host smithi135 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi135/config 2024-03-30T05:00:00.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:00 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-mon-smithi123[139346]: 2024-03-30T05:00:00.000+0000 7f50e7e00700 -1 log_channel(cluster) log [ERR] : /usr/bin/podman: timeout after 895 seconds 2024-03-30T05:00:00.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:00 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-mon-smithi123[139346]: 2024-03-30T05:00:00.000+0000 7f50e7e00700 -1 log_channel(cluster) log [ERR] : Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T05:00:00.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:00 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-mon-smithi123[139346]: 2024-03-30T05:00:00.000+0000 7f50e7e00700 -1 log_channel(cluster) log [ERR] : Traceback (most recent call last): 2024-03-30T05:00:00.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:00 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-mon-smithi123[139346]: 2024-03-30T05:00:00.000+0000 7f50e7e00700 -1 log_channel(cluster) log [ERR] : File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-30T05:00:00.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:00 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-mon-smithi123[139346]: 2024-03-30T05:00:00.000+0000 7f50e7e00700 -1 log_channel(cluster) log [ERR] : "__main__", mod_spec) 2024-03-30T05:00:00.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:00 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-mon-smithi123[139346]: 2024-03-30T05:00:00.000+0000 7f50e7e00700 -1 log_channel(cluster) log [ERR] : File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-30T05:00:00.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:00 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-mon-smithi123[139346]: 2024-03-30T05:00:00.000+0000 7f50e7e00700 -1 log_channel(cluster) log [ERR] : exec(code, run_globals) 2024-03-30T05:00:00.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:00 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-mon-smithi123[139346]: 2024-03-30T05:00:00.000+0000 7f50e7e00700 -1 log_channel(cluster) log [ERR] : File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10889, in 2024-03-30T05:00:00.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:00 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-mon-smithi123[139346]: 2024-03-30T05:00:00.000+0000 7f50e7e00700 -1 log_channel(cluster) log [ERR] : File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10877, in main 2024-03-30T05:00:00.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:00 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-mon-smithi123[139346]: 2024-03-30T05:00:00.000+0000 7f50e7e00700 -1 log_channel(cluster) log [ERR] : File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-30T05:00:00.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:00 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-mon-smithi123[139346]: 2024-03-30T05:00:00.000+0000 7f50e7e00700 -1 log_channel(cluster) log [ERR] : File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-30T05:00:00.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:00 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-mon-smithi123[139346]: 2024-03-30T05:00:00.000+0000 7f50e7e00700 -1 log_channel(cluster) log [ERR] : File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-30T05:00:00.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:00 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-mon-smithi123[139346]: 2024-03-30T05:00:00.000+0000 7f50e7e00700 -1 log_channel(cluster) log [ERR] : File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-30T05:00:00.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:00 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-mon-smithi123[139346]: 2024-03-30T05:00:00.000+0000 7f50e7e00700 -1 log_channel(cluster) log [ERR] : File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-03-30T05:00:00.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:00 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-mon-smithi123[139346]: 2024-03-30T05:00:00.000+0000 7f50e7e00700 -1 log_channel(cluster) log [ERR] : File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-30T05:00:00.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:00 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-mon-smithi123[139346]: 2024-03-30T05:00:00.000+0000 7f50e7e00700 -1 log_channel(cluster) log [ERR] : RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T05:00:00.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:00 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-mon-smithi123[139346]: 2024-03-30T05:00:00.000+0000 7f50e7e00700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T05:00:00.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:00 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-mon-smithi123[139346]: 2024-03-30T05:00:00.000+0000 7f50e7e00700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2024-03-30T05:00:00.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:00 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-mon-smithi123[139346]: 2024-03-30T05:00:00.000+0000 7f50e7e00700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T05:00:00.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:00 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-mon-smithi123[139346]: 2024-03-30T05:00:00.000+0000 7f50e7e00700 -1 log_channel(cluster) log [ERR] : fs cephfs is offline because no MDS is active for it. 2024-03-30T05:00:00.469 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:00 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:00:00.469 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:00 smithi123 ceph-mon[139369]: Health detail: HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T05:00:00.469 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:00 smithi123 ceph-mon[139369]: [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-30T05:00:00.469 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:00 smithi123 ceph-mon[139369]: host smithi135 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi135/config 2024-03-30T05:00:00.469 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:00 smithi123 ceph-mon[139369]: /usr/bin/podman: timeout after 895 seconds 2024-03-30T05:00:00.469 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:00 smithi123 ceph-mon[139369]: Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T05:00:00.469 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:00 smithi123 ceph-mon[139369]: Traceback (most recent call last): 2024-03-30T05:00:00.469 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:00 smithi123 ceph-mon[139369]: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-30T05:00:00.469 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:00 smithi123 ceph-mon[139369]: "__main__", mod_spec) 2024-03-30T05:00:00.469 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:00 smithi123 ceph-mon[139369]: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-30T05:00:00.469 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:00 smithi123 ceph-mon[139369]: exec(code, run_globals) 2024-03-30T05:00:00.469 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:00 smithi123 ceph-mon[139369]: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10889, in 2024-03-30T05:00:00.469 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:00 smithi123 ceph-mon[139369]: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10877, in main 2024-03-30T05:00:00.470 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:00 smithi123 ceph-mon[139369]: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-30T05:00:00.470 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:00 smithi123 ceph-mon[139369]: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-30T05:00:00.470 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:00 smithi123 ceph-mon[139369]: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-30T05:00:00.470 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:00 smithi123 ceph-mon[139369]: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-30T05:00:00.470 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:00 smithi123 ceph-mon[139369]: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-03-30T05:00:00.470 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:00 smithi123 ceph-mon[139369]: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-30T05:00:00.470 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:00 smithi123 ceph-mon[139369]: RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T05:00:00.470 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:00 smithi123 ceph-mon[139369]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T05:00:00.470 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:00 smithi123 ceph-mon[139369]: fs cephfs is degraded 2024-03-30T05:00:00.470 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:00 smithi123 ceph-mon[139369]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T05:00:00.470 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:00 smithi123 ceph-mon[139369]: fs cephfs is offline because no MDS is active for it. 2024-03-30T05:00:00.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:00 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:00:00.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:00 smithi135 ceph-mon[106011]: Health detail: HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T05:00:00.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:00 smithi135 ceph-mon[106011]: [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-30T05:00:00.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:00 smithi135 ceph-mon[106011]: host smithi135 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi135/config 2024-03-30T05:00:00.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:00 smithi135 ceph-mon[106011]: /usr/bin/podman: timeout after 895 seconds 2024-03-30T05:00:00.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:00 smithi135 ceph-mon[106011]: Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T05:00:00.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:00 smithi135 ceph-mon[106011]: Traceback (most recent call last): 2024-03-30T05:00:00.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:00 smithi135 ceph-mon[106011]: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-30T05:00:00.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:00 smithi135 ceph-mon[106011]: "__main__", mod_spec) 2024-03-30T05:00:00.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:00 smithi135 ceph-mon[106011]: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-30T05:00:00.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:00 smithi135 ceph-mon[106011]: exec(code, run_globals) 2024-03-30T05:00:00.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:00 smithi135 ceph-mon[106011]: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10889, in 2024-03-30T05:00:00.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:00 smithi135 ceph-mon[106011]: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10877, in main 2024-03-30T05:00:00.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:00 smithi135 ceph-mon[106011]: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-30T05:00:00.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:00 smithi135 ceph-mon[106011]: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-30T05:00:00.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:00 smithi135 ceph-mon[106011]: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-30T05:00:00.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:00 smithi135 ceph-mon[106011]: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-30T05:00:00.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:00 smithi135 ceph-mon[106011]: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-03-30T05:00:00.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:00 smithi135 ceph-mon[106011]: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-30T05:00:00.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:00 smithi135 ceph-mon[106011]: RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T05:00:00.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:00 smithi135 ceph-mon[106011]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T05:00:00.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:00 smithi135 ceph-mon[106011]: fs cephfs is degraded 2024-03-30T05:00:00.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:00 smithi135 ceph-mon[106011]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T05:00:00.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:00 smithi135 ceph-mon[106011]: fs cephfs is offline because no MDS is active for it. 2024-03-30T05:00:01.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:01 smithi123 ceph-mon[139369]: pgmap v865: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:01.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:01 smithi135 ceph-mon[106011]: pgmap v865: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:02.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:02 smithi123 ceph-mon[139369]: pgmap v866: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:02.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:02 smithi135 ceph-mon[106011]: pgmap v866: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:04.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:04 smithi123 ceph-mon[139369]: pgmap v867: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:05.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:04 smithi135 ceph-mon[106011]: pgmap v867: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:05.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.164+0000 7f7f5ea00700 1 -- 172.21.15.123:0/1246265143 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f581037c0 msgr2=0x7f7f580fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:05.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.164+0000 7f7f5ea00700 1 --2- 172.21.15.123:0/1246265143 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f581037c0 0x7f7f580fce30 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f7f44004940 tx=0x7f7f44009e30 comp rx=0 tx=0).stop 2024-03-30T05:00:05.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.164+0000 7f7f5ea00700 1 -- 172.21.15.123:0/1246265143 shutdown_connections 2024-03-30T05:00:05.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.164+0000 7f7f5ea00700 1 --2- 172.21.15.123:0/1246265143 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f581037c0 0x7f7f580fce30 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:05.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.164+0000 7f7f5ea00700 1 --2- 172.21.15.123:0/1246265143 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f58102e20 0x7f7f581031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:05.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.164+0000 7f7f5ea00700 1 -- 172.21.15.123:0/1246265143 >> 172.21.15.123:0/1246265143 conn(0x7f7f580f89d0 msgr2=0x7f7f580fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:00:05.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.165+0000 7f7f5ea00700 1 -- 172.21.15.123:0/1246265143 shutdown_connections 2024-03-30T05:00:05.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.165+0000 7f7f5ea00700 1 -- 172.21.15.123:0/1246265143 wait complete. 2024-03-30T05:00:05.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.165+0000 7f7f5ea00700 1 Processor -- start 2024-03-30T05:00:05.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.165+0000 7f7f5ea00700 1 -- start start 2024-03-30T05:00:05.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.166+0000 7f7f5ea00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f58102e20 0x7f7f580702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:00:05.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.166+0000 7f7f5ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f581037c0 0x7f7f58070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:00:05.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.166+0000 7f7f5ea00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f58070d90 con 0x7f7f581037c0 2024-03-30T05:00:05.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.166+0000 7f7f5ea00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f58070ed0 con 0x7f7f58102e20 2024-03-30T05:00:05.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.166+0000 7f7f57400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f581037c0 0x7f7f58070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:00:05.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.166+0000 7f7f57400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f581037c0 0x7f7f58070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52644/0 (socket says 172.21.15.123:52644) 2024-03-30T05:00:05.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.166+0000 7f7f57400700 1 -- 172.21.15.123:0/775028866 learned_addr learned my addr 172.21.15.123:0/775028866 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:00:05.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.167+0000 7f7f57e00700 1 --2- 172.21.15.123:0/775028866 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f58102e20 0x7f7f580702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:00:05.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.167+0000 7f7f57400700 1 -- 172.21.15.123:0/775028866 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f58102e20 msgr2=0x7f7f580702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:05.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.167+0000 7f7f57400700 1 --2- 172.21.15.123:0/775028866 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f58102e20 0x7f7f580702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:05.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.167+0000 7f7f57400700 1 -- 172.21.15.123:0/775028866 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f44005040 con 0x7f7f581037c0 2024-03-30T05:00:05.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.167+0000 7f7f57400700 1 --2- 172.21.15.123:0/775028866 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f581037c0 0x7f7f58070800 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f7f44009390 tx=0x7f7f44009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:00:05.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.167+0000 7f7f4fe00700 1 -- 172.21.15.123:0/775028866 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f4400e040 con 0x7f7f581037c0 2024-03-30T05:00:05.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.167+0000 7f7f4fe00700 1 -- 172.21.15.123:0/775028866 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7f44013420 con 0x7f7f581037c0 2024-03-30T05:00:05.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.167+0000 7f7f4fe00700 1 -- 172.21.15.123:0/775028866 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f4401c630 con 0x7f7f581037c0 2024-03-30T05:00:05.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.167+0000 7f7f5ea00700 1 -- 172.21.15.123:0/775028866 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f58071120 con 0x7f7f581037c0 2024-03-30T05:00:05.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.167+0000 7f7f5ea00700 1 -- 172.21.15.123:0/775028866 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f5811a200 con 0x7f7f581037c0 2024-03-30T05:00:05.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.169+0000 7f7f5ea00700 1 -- 172.21.15.123:0/775028866 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f580fe570 con 0x7f7f581037c0 2024-03-30T05:00:05.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.169+0000 7f7f4fe00700 1 -- 172.21.15.123:0/775028866 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f7f44013a00 con 0x7f7f581037c0 2024-03-30T05:00:05.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.170+0000 7f7f4fe00700 1 --2- 172.21.15.123:0/775028866 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7f3c06cc30 0x7f7f3c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:00:05.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.170+0000 7f7f4fe00700 1 -- 172.21.15.123:0/775028866 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7f4408d2e0 con 0x7f7f581037c0 2024-03-30T05:00:05.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.171+0000 7f7f57e00700 1 --2- 172.21.15.123:0/775028866 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7f3c06cc30 0x7f7f3c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:00:05.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.171+0000 7f7f57e00700 1 --2- 172.21.15.123:0/775028866 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7f3c06cc30 0x7f7f3c06f0e0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f7f48004990 tx=0x7f7f48009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:00:05.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.175+0000 7f7f4fe00700 1 -- 172.21.15.123:0/775028866 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7f44057320 con 0x7f7f581037c0 2024-03-30T05:00:05.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.395+0000 7f7f5ea00700 1 -- 172.21.15.123:0/775028866 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7f58109e60 con 0x7f7f3c06cc30 2024-03-30T05:00:05.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.396+0000 7f7f4fe00700 1 -- 172.21.15.123:0/775028866 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f7f48004e60 con 0x7f7f3c06cc30 2024-03-30T05:00:05.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.398+0000 7f7f5ea00700 1 -- 172.21.15.123:0/775028866 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7f3c06cc30 msgr2=0x7f7f3c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:05.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.398+0000 7f7f5ea00700 1 --2- 172.21.15.123:0/775028866 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7f3c06cc30 0x7f7f3c06f0e0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f7f48004990 tx=0x7f7f48009ee0 comp rx=0 tx=0).stop 2024-03-30T05:00:05.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.398+0000 7f7f5ea00700 1 -- 172.21.15.123:0/775028866 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f581037c0 msgr2=0x7f7f58070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:05.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.398+0000 7f7f5ea00700 1 --2- 172.21.15.123:0/775028866 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f581037c0 0x7f7f58070800 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f7f44009390 tx=0x7f7f44009e30 comp rx=0 tx=0).stop 2024-03-30T05:00:05.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.399+0000 7f7f5ea00700 1 -- 172.21.15.123:0/775028866 shutdown_connections 2024-03-30T05:00:05.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.399+0000 7f7f5ea00700 1 --2- 172.21.15.123:0/775028866 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7f3c06cc30 0x7f7f3c06f0e0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:05.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.399+0000 7f7f5ea00700 1 --2- 172.21.15.123:0/775028866 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f58102e20 0x7f7f580702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:05.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.399+0000 7f7f5ea00700 1 --2- 172.21.15.123:0/775028866 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f581037c0 0x7f7f58070800 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:05.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.399+0000 7f7f5ea00700 1 -- 172.21.15.123:0/775028866 >> 172.21.15.123:0/775028866 conn(0x7f7f580f89d0 msgr2=0x7f7f580f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:00:05.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.399+0000 7f7f5ea00700 1 -- 172.21.15.123:0/775028866 shutdown_connections 2024-03-30T05:00:05.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.399+0000 7f7f5ea00700 1 -- 172.21.15.123:0/775028866 wait complete. 2024-03-30T05:00:05.411 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T05:00:05.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.533+0000 7f8c4f400700 1 -- 172.21.15.123:0/2257786033 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c481037a0 msgr2=0x7f8c480fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:05.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.533+0000 7f8c4f400700 1 --2- 172.21.15.123:0/2257786033 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c481037a0 0x7f8c480fce10 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f8c34004940 tx=0x7f8c34009e30 comp rx=0 tx=0).stop 2024-03-30T05:00:05.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.533+0000 7f8c4f400700 1 -- 172.21.15.123:0/2257786033 shutdown_connections 2024-03-30T05:00:05.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.533+0000 7f8c4f400700 1 --2- 172.21.15.123:0/2257786033 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c481037a0 0x7f8c480fce10 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:05.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.533+0000 7f8c4f400700 1 --2- 172.21.15.123:0/2257786033 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8c48102e00 0x7f8c481031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:05.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.533+0000 7f8c4f400700 1 -- 172.21.15.123:0/2257786033 >> 172.21.15.123:0/2257786033 conn(0x7f8c480f89b0 msgr2=0x7f8c480fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:00:05.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.533+0000 7f8c4f400700 1 -- 172.21.15.123:0/2257786033 shutdown_connections 2024-03-30T05:00:05.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.533+0000 7f8c4f400700 1 -- 172.21.15.123:0/2257786033 wait complete. 2024-03-30T05:00:05.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.534+0000 7f8c4f400700 1 Processor -- start 2024-03-30T05:00:05.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.534+0000 7f8c4f400700 1 -- start start 2024-03-30T05:00:05.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.534+0000 7f8c4f400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8c48102e00 0x7f8c48113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:00:05.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.534+0000 7f8c4f400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c481037a0 0x7f8c48113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:00:05.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.534+0000 7f8c4f400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c48114600 con 0x7f8c481037a0 2024-03-30T05:00:05.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.534+0000 7f8c4f400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4810d9f0 con 0x7f8c48102e00 2024-03-30T05:00:05.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.534+0000 7f8c47e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c481037a0 0x7f8c48113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:00:05.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.534+0000 7f8c47e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c481037a0 0x7f8c48113fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52666/0 (socket says 172.21.15.123:52666) 2024-03-30T05:00:05.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.534+0000 7f8c47e00700 1 -- 172.21.15.123:0/3014328869 learned_addr learned my addr 172.21.15.123:0/3014328869 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:00:05.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.535+0000 7f8c47e00700 1 -- 172.21.15.123:0/3014328869 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8c48102e00 msgr2=0x7f8c48113a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:05.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.535+0000 7f8c47e00700 1 --2- 172.21.15.123:0/3014328869 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8c48102e00 0x7f8c48113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:05.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.535+0000 7f8c47e00700 1 -- 172.21.15.123:0/3014328869 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c34005040 con 0x7f8c481037a0 2024-03-30T05:00:05.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.535+0000 7f8c47e00700 1 --2- 172.21.15.123:0/3014328869 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c481037a0 0x7f8c48113fb0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f8c34008d50 tx=0x7f8c34009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:00:05.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.535+0000 7f8c44c00700 1 -- 172.21.15.123:0/3014328869 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c3400e030 con 0x7f8c481037a0 2024-03-30T05:00:05.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.535+0000 7f8c44c00700 1 -- 172.21.15.123:0/3014328869 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8c34013de0 con 0x7f8c481037a0 2024-03-30T05:00:05.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.535+0000 7f8c44c00700 1 -- 172.21.15.123:0/3014328869 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c3401cd90 con 0x7f8c481037a0 2024-03-30T05:00:05.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.535+0000 7f8c4f400700 1 -- 172.21.15.123:0/3014328869 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c4810dc70 con 0x7f8c481037a0 2024-03-30T05:00:05.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.535+0000 7f8c4f400700 1 -- 172.21.15.123:0/3014328869 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c4810e230 con 0x7f8c481037a0 2024-03-30T05:00:05.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.537+0000 7f8c4f400700 1 -- 172.21.15.123:0/3014328869 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c480fe550 con 0x7f8c481037a0 2024-03-30T05:00:05.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.537+0000 7f8c44c00700 1 -- 172.21.15.123:0/3014328869 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f8c34013930 con 0x7f8c481037a0 2024-03-30T05:00:05.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.539+0000 7f8c44c00700 1 --2- 172.21.15.123:0/3014328869 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8c3006cc30 0x7f8c3006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:00:05.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.539+0000 7f8c44c00700 1 -- 172.21.15.123:0/3014328869 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f8c3408d980 con 0x7f8c481037a0 2024-03-30T05:00:05.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.539+0000 7f8c4ca00700 1 --2- 172.21.15.123:0/3014328869 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8c3006cc30 0x7f8c3006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:00:05.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.540+0000 7f8c4ca00700 1 --2- 172.21.15.123:0/3014328869 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8c3006cc30 0x7f8c3006f0e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c009fd0 tx=0x7f8c3c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:00:05.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.543+0000 7f8c44c00700 1 -- 172.21.15.123:0/3014328869 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8c3405b3d0 con 0x7f8c481037a0 2024-03-30T05:00:05.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.763+0000 7f8c4f400700 1 -- 172.21.15.123:0/3014328869 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8c48109da0 con 0x7f8c3006cc30 2024-03-30T05:00:05.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.765+0000 7f8c44c00700 1 -- 172.21.15.123:0/3014328869 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f8c48109da0 con 0x7f8c3006cc30 2024-03-30T05:00:05.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.767+0000 7f8c4f400700 1 -- 172.21.15.123:0/3014328869 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8c3006cc30 msgr2=0x7f8c3006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:05.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.767+0000 7f8c4f400700 1 --2- 172.21.15.123:0/3014328869 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8c3006cc30 0x7f8c3006f0e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c009fd0 tx=0x7f8c3c00b040 comp rx=0 tx=0).stop 2024-03-30T05:00:05.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.767+0000 7f8c4f400700 1 -- 172.21.15.123:0/3014328869 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c481037a0 msgr2=0x7f8c48113fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:05.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.767+0000 7f8c4f400700 1 --2- 172.21.15.123:0/3014328869 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c481037a0 0x7f8c48113fb0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f8c34008d50 tx=0x7f8c34009e30 comp rx=0 tx=0).stop 2024-03-30T05:00:05.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.767+0000 7f8c4f400700 1 -- 172.21.15.123:0/3014328869 shutdown_connections 2024-03-30T05:00:05.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.767+0000 7f8c4f400700 1 --2- 172.21.15.123:0/3014328869 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8c3006cc30 0x7f8c3006f0e0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:05.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.767+0000 7f8c4f400700 1 --2- 172.21.15.123:0/3014328869 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8c48102e00 0x7f8c48113a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:05.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.767+0000 7f8c4f400700 1 --2- 172.21.15.123:0/3014328869 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c481037a0 0x7f8c48113fb0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:05.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.767+0000 7f8c4f400700 1 -- 172.21.15.123:0/3014328869 >> 172.21.15.123:0/3014328869 conn(0x7f8c480f89b0 msgr2=0x7f8c480f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:00:05.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.767+0000 7f8c4f400700 1 -- 172.21.15.123:0/3014328869 shutdown_connections 2024-03-30T05:00:05.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.767+0000 7f8c4f400700 1 -- 172.21.15.123:0/3014328869 wait complete. 2024-03-30T05:00:05.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.903+0000 7fcda3c00700 1 -- 172.21.15.123:0/3427729419 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd9c101d70 msgr2=0x7fcd9c102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:05.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.903+0000 7fcda3c00700 1 --2- 172.21.15.123:0/3427729419 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd9c101d70 0x7fcd9c102140 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fcd90004940 tx=0x7fcd90009e30 comp rx=0 tx=0).stop 2024-03-30T05:00:05.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.903+0000 7fcda3c00700 1 -- 172.21.15.123:0/3427729419 shutdown_connections 2024-03-30T05:00:05.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.903+0000 7fcda3c00700 1 --2- 172.21.15.123:0/3427729419 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcd9c102710 0x7fcd9c0fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:05.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.903+0000 7fcda3c00700 1 --2- 172.21.15.123:0/3427729419 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd9c101d70 0x7fcd9c102140 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:05.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.903+0000 7fcda3c00700 1 -- 172.21.15.123:0/3427729419 >> 172.21.15.123:0/3427729419 conn(0x7fcd9c072bc0 msgr2=0x7fcd9c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:00:05.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.904+0000 7fcda3c00700 1 -- 172.21.15.123:0/3427729419 shutdown_connections 2024-03-30T05:00:05.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.904+0000 7fcda3c00700 1 -- 172.21.15.123:0/3427729419 wait complete. 2024-03-30T05:00:05.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.905+0000 7fcda3c00700 1 Processor -- start 2024-03-30T05:00:05.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.905+0000 7fcda3c00700 1 -- start start 2024-03-30T05:00:05.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.905+0000 7fcda3c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd9c101d70 0x7fcd9c113ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:00:05.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.906+0000 7fcda3c00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcd9c102710 0x7fcd9c114000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:00:05.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.906+0000 7fcda3c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9c114650 con 0x7fcd9c101d70 2024-03-30T05:00:05.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.906+0000 7fcda3c00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9c10da40 con 0x7fcd9c102710 2024-03-30T05:00:05.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.906+0000 7fcda1200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd9c101d70 0x7fcd9c113ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:00:05.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.906+0000 7fcda1200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd9c101d70 0x7fcd9c113ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52686/0 (socket says 172.21.15.123:52686) 2024-03-30T05:00:05.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.906+0000 7fcda1200700 1 -- 172.21.15.123:0/1041177015 learned_addr learned my addr 172.21.15.123:0/1041177015 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:00:05.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.906+0000 7fcd9be00700 1 --2- 172.21.15.123:0/1041177015 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcd9c102710 0x7fcd9c114000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:00:05.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.906+0000 7fcda1200700 1 -- 172.21.15.123:0/1041177015 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcd9c102710 msgr2=0x7fcd9c114000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:05.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.906+0000 7fcda1200700 1 --2- 172.21.15.123:0/1041177015 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcd9c102710 0x7fcd9c114000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:05.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.906+0000 7fcda1200700 1 -- 172.21.15.123:0/1041177015 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd90005040 con 0x7fcd9c101d70 2024-03-30T05:00:05.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.906+0000 7fcda1200700 1 --2- 172.21.15.123:0/1041177015 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd9c101d70 0x7fcd9c113ac0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fcd90009390 tx=0x7fcd90009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:00:05.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.907+0000 7fcd98c00700 1 -- 172.21.15.123:0/1041177015 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcd9000e040 con 0x7fcd9c101d70 2024-03-30T05:00:05.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.907+0000 7fcd98c00700 1 -- 172.21.15.123:0/1041177015 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcd90013420 con 0x7fcd9c101d70 2024-03-30T05:00:05.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.907+0000 7fcda3c00700 1 -- 172.21.15.123:0/1041177015 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd9c10dcc0 con 0x7fcd9c101d70 2024-03-30T05:00:05.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.907+0000 7fcd98c00700 1 -- 172.21.15.123:0/1041177015 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcd9001c630 con 0x7fcd9c101d70 2024-03-30T05:00:05.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.907+0000 7fcda3c00700 1 -- 172.21.15.123:0/1041177015 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd9c10e280 con 0x7fcd9c101d70 2024-03-30T05:00:05.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.908+0000 7fcda3c00700 1 -- 172.21.15.123:0/1041177015 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd9c0fd4c0 con 0x7fcd9c101d70 2024-03-30T05:00:05.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.909+0000 7fcd98c00700 1 -- 172.21.15.123:0/1041177015 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fcd900139e0 con 0x7fcd9c101d70 2024-03-30T05:00:05.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.909+0000 7fcd98c00700 1 --2- 172.21.15.123:0/1041177015 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fcd8406cc30 0x7fcd8406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:00:05.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.909+0000 7fcd98c00700 1 -- 172.21.15.123:0/1041177015 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fcd9008d250 con 0x7fcd9c101d70 2024-03-30T05:00:05.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.910+0000 7fcd9be00700 1 --2- 172.21.15.123:0/1041177015 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fcd8406cc30 0x7fcd8406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:00:05.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.910+0000 7fcd9be00700 1 --2- 172.21.15.123:0/1041177015 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fcd8406cc30 0x7fcd8406f0e0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fcd88007fd0 tx=0x7fcd88009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:00:05.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:05.914+0000 7fcd98c00700 1 -- 172.21.15.123:0/1041177015 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fcd9005add0 con 0x7fcd9c101d70 2024-03-30T05:00:06.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.126+0000 7fcda3c00700 1 -- 172.21.15.123:0/1041177015 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fcd9c108da0 con 0x7fcd8406cc30 2024-03-30T05:00:06.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.136+0000 7fcd98c00700 1 -- 172.21.15.123:0/1041177015 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fcd9c108da0 con 0x7fcd8406cc30 2024-03-30T05:00:06.134 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T05:00:06.134 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (29m) 6m ago 41m 14.6M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T05:00:06.134 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (26m) 6m ago 42m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T05:00:06.135 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (26m) 6m ago 40m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T05:00:06.135 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (28m) 6m ago 41m 73.5M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T05:00:06.135 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (22m) 6m ago 35m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T05:00:06.135 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (22m) 6m ago 35m 16.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T05:00:06.135 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (6m) 6m ago 35m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 87b073a6c9ca 2024-03-30T05:00:06.135 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (21m) 6m ago 35m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T05:00:06.135 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (31m) 6m ago 43m 510M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T05:00:06.135 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (30m) 6m ago 40m 456M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T05:00:06.135 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (27m) 6m ago 43m 70.0M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T05:00:06.135 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (27m) 6m ago 40m 51.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T05:00:06.135 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (30m) 6m ago 41m 8636k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T05:00:06.135 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (29m) 6m ago 40m 8678k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T05:00:06.135 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (26m) 6m ago 39m 107M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T05:00:06.135 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (25m) 6m ago 39m 97.2M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T05:00:06.136 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (24m) 6m ago 38m 89.8M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T05:00:06.136 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (24m) 6m ago 38m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T05:00:06.136 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (23m) 6m ago 37m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T05:00:06.136 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (22m) 6m ago 37m 88.8M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T05:00:06.136 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (29m) 6m ago 41m 63.9M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T05:00:06.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.138+0000 7fcda3c00700 1 -- 172.21.15.123:0/1041177015 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fcd8406cc30 msgr2=0x7fcd8406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:06.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.138+0000 7fcda3c00700 1 --2- 172.21.15.123:0/1041177015 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fcd8406cc30 0x7fcd8406f0e0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fcd88007fd0 tx=0x7fcd88009040 comp rx=0 tx=0).stop 2024-03-30T05:00:06.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.138+0000 7fcda3c00700 1 -- 172.21.15.123:0/1041177015 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd9c101d70 msgr2=0x7fcd9c113ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:06.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.138+0000 7fcda3c00700 1 --2- 172.21.15.123:0/1041177015 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd9c101d70 0x7fcd9c113ac0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fcd90009390 tx=0x7fcd90009e30 comp rx=0 tx=0).stop 2024-03-30T05:00:06.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.138+0000 7fcda3c00700 1 -- 172.21.15.123:0/1041177015 shutdown_connections 2024-03-30T05:00:06.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.138+0000 7fcda3c00700 1 --2- 172.21.15.123:0/1041177015 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fcd8406cc30 0x7fcd8406f0e0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:06.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.138+0000 7fcda3c00700 1 --2- 172.21.15.123:0/1041177015 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd9c101d70 0x7fcd9c113ac0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:06.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.138+0000 7fcda3c00700 1 --2- 172.21.15.123:0/1041177015 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcd9c102710 0x7fcd9c114000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:06.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.138+0000 7fcda3c00700 1 -- 172.21.15.123:0/1041177015 >> 172.21.15.123:0/1041177015 conn(0x7fcd9c072bc0 msgr2=0x7fcd9c0659a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:00:06.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.138+0000 7fcda3c00700 1 -- 172.21.15.123:0/1041177015 shutdown_connections 2024-03-30T05:00:06.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.138+0000 7fcda3c00700 1 -- 172.21.15.123:0/1041177015 wait complete. 2024-03-30T05:00:06.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.271+0000 7fb358000700 1 -- 172.21.15.123:0/3413801388 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb350102e00 msgr2=0x7fb3501031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:06.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.271+0000 7fb358000700 1 --2- 172.21.15.123:0/3413801388 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb350102e00 0x7fb3501031d0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fb344004940 tx=0x7fb344009e30 comp rx=0 tx=0).stop 2024-03-30T05:00:06.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.272+0000 7fb358000700 1 -- 172.21.15.123:0/3413801388 shutdown_connections 2024-03-30T05:00:06.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.272+0000 7fb358000700 1 --2- 172.21.15.123:0/3413801388 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb3501037a0 0x7fb3500fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:06.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.272+0000 7fb358000700 1 --2- 172.21.15.123:0/3413801388 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb350102e00 0x7fb3501031d0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:06.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.272+0000 7fb358000700 1 -- 172.21.15.123:0/3413801388 >> 172.21.15.123:0/3413801388 conn(0x7fb3500f89b0 msgr2=0x7fb3500fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:00:06.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.272+0000 7fb358000700 1 -- 172.21.15.123:0/3413801388 shutdown_connections 2024-03-30T05:00:06.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.272+0000 7fb358000700 1 -- 172.21.15.123:0/3413801388 wait complete. 2024-03-30T05:00:06.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.273+0000 7fb358000700 1 Processor -- start 2024-03-30T05:00:06.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.273+0000 7fb358000700 1 -- start start 2024-03-30T05:00:06.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.273+0000 7fb358000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb350102e00 0x7fb3500702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:00:06.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.273+0000 7fb358000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb3501037a0 0x7fb350070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:00:06.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.273+0000 7fb358000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb350070d90 con 0x7fb350102e00 2024-03-30T05:00:06.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.273+0000 7fb358000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb350070ed0 con 0x7fb3501037a0 2024-03-30T05:00:06.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.273+0000 7fb355600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb350102e00 0x7fb3500702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:00:06.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.273+0000 7fb355600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb350102e00 0x7fb3500702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52700/0 (socket says 172.21.15.123:52700) 2024-03-30T05:00:06.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.273+0000 7fb355600700 1 -- 172.21.15.123:0/2959088788 learned_addr learned my addr 172.21.15.123:0/2959088788 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:00:06.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.274+0000 7fb354c00700 1 --2- 172.21.15.123:0/2959088788 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb3501037a0 0x7fb350070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:00:06.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.274+0000 7fb355600700 1 -- 172.21.15.123:0/2959088788 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb3501037a0 msgr2=0x7fb350070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:06.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.274+0000 7fb355600700 1 --2- 172.21.15.123:0/2959088788 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb3501037a0 0x7fb350070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:06.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.274+0000 7fb355600700 1 -- 172.21.15.123:0/2959088788 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb344005040 con 0x7fb350102e00 2024-03-30T05:00:06.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.274+0000 7fb354c00700 1 --2- 172.21.15.123:0/2959088788 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb3501037a0 0x7fb350070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T05:00:06.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.274+0000 7fb355600700 1 --2- 172.21.15.123:0/2959088788 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb350102e00 0x7fb3500702c0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fb344009390 tx=0x7fb344009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:00:06.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.274+0000 7fb34d600700 1 -- 172.21.15.123:0/2959088788 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb34400e040 con 0x7fb350102e00 2024-03-30T05:00:06.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.274+0000 7fb34d600700 1 -- 172.21.15.123:0/2959088788 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb344013420 con 0x7fb350102e00 2024-03-30T05:00:06.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.274+0000 7fb34d600700 1 -- 172.21.15.123:0/2959088788 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb34401c630 con 0x7fb350102e00 2024-03-30T05:00:06.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.274+0000 7fb358000700 1 -- 172.21.15.123:0/2959088788 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb350071120 con 0x7fb350102e00 2024-03-30T05:00:06.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.275+0000 7fb358000700 1 -- 172.21.15.123:0/2959088788 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb35011a250 con 0x7fb350102e00 2024-03-30T05:00:06.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.275+0000 7fb358000700 1 -- 172.21.15.123:0/2959088788 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3500fe550 con 0x7fb350102e00 2024-03-30T05:00:06.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.276+0000 7fb34d600700 1 -- 172.21.15.123:0/2959088788 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fb3440139e0 con 0x7fb350102e00 2024-03-30T05:00:06.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.277+0000 7fb34d600700 1 --2- 172.21.15.123:0/2959088788 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb33806cc30 0x7fb33806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:00:06.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.277+0000 7fb34d600700 1 -- 172.21.15.123:0/2959088788 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb34408d310 con 0x7fb350102e00 2024-03-30T05:00:06.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.278+0000 7fb354c00700 1 --2- 172.21.15.123:0/2959088788 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb33806cc30 0x7fb33806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:00:06.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.278+0000 7fb354c00700 1 --2- 172.21.15.123:0/2959088788 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb33806cc30 0x7fb33806f0e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fb33c009fd0 tx=0x7fb33c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:00:06.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.282+0000 7fb34d600700 1 -- 172.21.15.123:0/2959088788 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb3440572d0 con 0x7fb350102e00 2024-03-30T05:00:06.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.564+0000 7fb358000700 1 -- 172.21.15.123:0/2959088788 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fb3500719b0 con 0x7fb350102e00 2024-03-30T05:00:06.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.565+0000 7fb34d600700 1 -- 172.21.15.123:0/2959088788 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7fb34405a8f0 con 0x7fb350102e00 2024-03-30T05:00:06.563 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T05:00:06.563 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T05:00:06.563 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T05:00:06.563 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:00:06.563 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T05:00:06.564 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T05:00:06.564 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:00:06.564 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T05:00:06.564 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T05:00:06.564 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:00:06.564 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T05:00:06.564 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 4 2024-03-30T05:00:06.564 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:00:06.564 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T05:00:06.564 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 14 2024-03-30T05:00:06.564 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T05:00:06.564 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T05:00:06.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.567+0000 7fb358000700 1 -- 172.21.15.123:0/2959088788 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb33806cc30 msgr2=0x7fb33806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:06.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.567+0000 7fb358000700 1 --2- 172.21.15.123:0/2959088788 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb33806cc30 0x7fb33806f0e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fb33c009fd0 tx=0x7fb33c00b040 comp rx=0 tx=0).stop 2024-03-30T05:00:06.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.567+0000 7fb358000700 1 -- 172.21.15.123:0/2959088788 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb350102e00 msgr2=0x7fb3500702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:06.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.567+0000 7fb358000700 1 --2- 172.21.15.123:0/2959088788 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb350102e00 0x7fb3500702c0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fb344009390 tx=0x7fb344009e30 comp rx=0 tx=0).stop 2024-03-30T05:00:06.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.567+0000 7fb358000700 1 -- 172.21.15.123:0/2959088788 shutdown_connections 2024-03-30T05:00:06.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.567+0000 7fb358000700 1 --2- 172.21.15.123:0/2959088788 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb33806cc30 0x7fb33806f0e0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:06.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.567+0000 7fb358000700 1 --2- 172.21.15.123:0/2959088788 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb350102e00 0x7fb3500702c0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:06.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.567+0000 7fb358000700 1 --2- 172.21.15.123:0/2959088788 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb3501037a0 0x7fb350070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:06.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.567+0000 7fb358000700 1 -- 172.21.15.123:0/2959088788 >> 172.21.15.123:0/2959088788 conn(0x7fb3500f89b0 msgr2=0x7fb3500f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:00:06.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.567+0000 7fb358000700 1 -- 172.21.15.123:0/2959088788 shutdown_connections 2024-03-30T05:00:06.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.567+0000 7fb358000700 1 -- 172.21.15.123:0/2959088788 wait complete. 2024-03-30T05:00:06.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.701+0000 7fd13fe00700 1 -- 172.21.15.123:0/2685254268 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1401057f0 msgr2=0x7fd1400fee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:06.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.701+0000 7fd13fe00700 1 --2- 172.21.15.123:0/2685254268 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1401057f0 0x7fd1400fee60 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fd12c004940 tx=0x7fd12c009e30 comp rx=0 tx=0).stop 2024-03-30T05:00:06.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.701+0000 7fd13fe00700 1 -- 172.21.15.123:0/2685254268 shutdown_connections 2024-03-30T05:00:06.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.701+0000 7fd13fe00700 1 --2- 172.21.15.123:0/2685254268 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1401057f0 0x7fd1400fee60 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:06.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.701+0000 7fd13fe00700 1 --2- 172.21.15.123:0/2685254268 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd140104e50 0x7fd140105220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:06.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.701+0000 7fd13fe00700 1 -- 172.21.15.123:0/2685254268 >> 172.21.15.123:0/2685254268 conn(0x7fd1400fab10 msgr2=0x7fd1400fcef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:00:06.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.701+0000 7fd13fe00700 1 -- 172.21.15.123:0/2685254268 shutdown_connections 2024-03-30T05:00:06.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.701+0000 7fd13fe00700 1 -- 172.21.15.123:0/2685254268 wait complete. 2024-03-30T05:00:06.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.702+0000 7fd13fe00700 1 Processor -- start 2024-03-30T05:00:06.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.702+0000 7fd13fe00700 1 -- start start 2024-03-30T05:00:06.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.702+0000 7fd13fe00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd140104e50 0x7fd140115b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:00:06.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.702+0000 7fd13fe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1401057f0 0x7fd140116050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:00:06.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.702+0000 7fd13fe00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1401166a0 con 0x7fd1401057f0 2024-03-30T05:00:06.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.702+0000 7fd13fe00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd14010fa90 con 0x7fd140104e50 2024-03-30T05:00:06.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.702+0000 7fd13e000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1401057f0 0x7fd140116050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:00:06.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.703+0000 7fd13e000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1401057f0 0x7fd140116050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52724/0 (socket says 172.21.15.123:52724) 2024-03-30T05:00:06.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.703+0000 7fd13e000700 1 -- 172.21.15.123:0/2224942447 learned_addr learned my addr 172.21.15.123:0/2224942447 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:00:06.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.703+0000 7fd13ea00700 1 --2- 172.21.15.123:0/2224942447 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd140104e50 0x7fd140115b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:00:06.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.703+0000 7fd13e000700 1 -- 172.21.15.123:0/2224942447 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd140104e50 msgr2=0x7fd140115b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:06.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.703+0000 7fd13e000700 1 --2- 172.21.15.123:0/2224942447 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd140104e50 0x7fd140115b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:06.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.703+0000 7fd13e000700 1 -- 172.21.15.123:0/2224942447 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd12c005040 con 0x7fd1401057f0 2024-03-30T05:00:06.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.703+0000 7fd13e000700 1 --2- 172.21.15.123:0/2224942447 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1401057f0 0x7fd140116050 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fd12c009390 tx=0x7fd12c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:00:06.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.703+0000 7fd136a00700 1 -- 172.21.15.123:0/2224942447 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd12c00e040 con 0x7fd1401057f0 2024-03-30T05:00:06.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.703+0000 7fd136a00700 1 -- 172.21.15.123:0/2224942447 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd12c013420 con 0x7fd1401057f0 2024-03-30T05:00:06.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.703+0000 7fd13fe00700 1 -- 172.21.15.123:0/2224942447 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd14010fd10 con 0x7fd1401057f0 2024-03-30T05:00:06.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.703+0000 7fd136a00700 1 -- 172.21.15.123:0/2224942447 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd12c01c630 con 0x7fd1401057f0 2024-03-30T05:00:06.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.704+0000 7fd13fe00700 1 -- 172.21.15.123:0/2224942447 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1401102d0 con 0x7fd1401057f0 2024-03-30T05:00:06.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.704+0000 7fd13fe00700 1 -- 172.21.15.123:0/2224942447 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1401005a0 con 0x7fd1401057f0 2024-03-30T05:00:06.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.706+0000 7fd136a00700 1 -- 172.21.15.123:0/2224942447 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fd12c013a00 con 0x7fd1401057f0 2024-03-30T05:00:06.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.706+0000 7fd136a00700 1 --2- 172.21.15.123:0/2224942447 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd12406cc20 0x7fd12406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:00:06.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.706+0000 7fd136a00700 1 -- 172.21.15.123:0/2224942447 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd12c08d1b0 con 0x7fd1401057f0 2024-03-30T05:00:06.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.707+0000 7fd13ea00700 1 --2- 172.21.15.123:0/2224942447 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd12406cc20 0x7fd12406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:00:06.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.707+0000 7fd13ea00700 1 --2- 172.21.15.123:0/2224942447 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd12406cc20 0x7fd12406f0d0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fd130004f60 tx=0x7fd13000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:00:06.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.711+0000 7fd136a00700 1 -- 172.21.15.123:0/2224942447 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd12c05ac00 con 0x7fd1401057f0 2024-03-30T05:00:06.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.957+0000 7fd13fe00700 1 -- 172.21.15.123:0/2224942447 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fd1400fe610 con 0x7fd1401057f0 2024-03-30T05:00:06.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.958+0000 7fd136a00700 1 -- 172.21.15.123:0/2224942447 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 34 v34) v1 ==== 76+0+1843 (secure 0 0 0) 0x7fd12c05a790 con 0x7fd1401057f0 2024-03-30T05:00:06.956 INFO:teuthology.orchestra.run.smithi123.stdout:e34 2024-03-30T05:00:06.956 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T05:00:06.956 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T05:00:06.956 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T05:00:06.957 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:00:06.957 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T05:00:06.957 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T05:00:06.957 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T05:00:06.957 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T05:00:06.957 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T05:00:06.957 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T05:00:06.957 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T05:00:06.957 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T05:00:06.957 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T05:00:06.957 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T05:00:06.957 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T05:00:06.957 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T05:00:06.957 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T05:00:06.957 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T05:00:06.958 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T05:00:06.958 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T05:00:06.958 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T05:00:06.958 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T05:00:06.958 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T05:00:06.958 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T05:00:06.958 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T05:00:06.958 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T05:00:06.958 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T05:00:06.958 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T05:00:06.958 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T05:00:06.958 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T05:00:06.959 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T05:00:06.959 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T05:00:06.959 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:00:06.959 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:00:06.959 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T05:00:06.959 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:00:06.959 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:00:06.959 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:00:06.959 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:00:06.959 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44975} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/3788988014,v1:172.21.15.135:6827/3788988014] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:00:06.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.960+0000 7fd13fe00700 1 -- 172.21.15.123:0/2224942447 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd12406cc20 msgr2=0x7fd12406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:06.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.960+0000 7fd13fe00700 1 --2- 172.21.15.123:0/2224942447 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd12406cc20 0x7fd12406f0d0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fd130004f60 tx=0x7fd13000b040 comp rx=0 tx=0).stop 2024-03-30T05:00:06.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.960+0000 7fd13fe00700 1 -- 172.21.15.123:0/2224942447 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1401057f0 msgr2=0x7fd140116050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:06.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.960+0000 7fd13fe00700 1 --2- 172.21.15.123:0/2224942447 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1401057f0 0x7fd140116050 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fd12c009390 tx=0x7fd12c009e30 comp rx=0 tx=0).stop 2024-03-30T05:00:06.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.960+0000 7fd13fe00700 1 -- 172.21.15.123:0/2224942447 shutdown_connections 2024-03-30T05:00:06.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.960+0000 7fd13fe00700 1 --2- 172.21.15.123:0/2224942447 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd12406cc20 0x7fd12406f0d0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:06.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.960+0000 7fd13fe00700 1 --2- 172.21.15.123:0/2224942447 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd140104e50 0x7fd140115b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:06.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.960+0000 7fd13fe00700 1 --2- 172.21.15.123:0/2224942447 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1401057f0 0x7fd140116050 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:06.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.960+0000 7fd13fe00700 1 -- 172.21.15.123:0/2224942447 >> 172.21.15.123:0/2224942447 conn(0x7fd1400fab10 msgr2=0x7fd1400fcba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:00:06.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.960+0000 7fd13fe00700 1 -- 172.21.15.123:0/2224942447 shutdown_connections 2024-03-30T05:00:06.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:06.960+0000 7fd13fe00700 1 -- 172.21.15.123:0/2224942447 wait complete. 2024-03-30T05:00:06.962 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 34 2024-03-30T05:00:07.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.093+0000 7fb363800700 1 -- 172.21.15.123:0/1838368049 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb35c102e00 msgr2=0x7fb35c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:07.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.093+0000 7fb363800700 1 --2- 172.21.15.123:0/1838368049 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb35c102e00 0x7fb35c1031d0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fb350004940 tx=0x7fb350009e30 comp rx=0 tx=0).stop 2024-03-30T05:00:07.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.093+0000 7fb363800700 1 -- 172.21.15.123:0/1838368049 shutdown_connections 2024-03-30T05:00:07.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.093+0000 7fb363800700 1 --2- 172.21.15.123:0/1838368049 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb35c1037a0 0x7fb35c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:07.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.093+0000 7fb363800700 1 --2- 172.21.15.123:0/1838368049 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb35c102e00 0x7fb35c1031d0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:07.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.094+0000 7fb363800700 1 -- 172.21.15.123:0/1838368049 >> 172.21.15.123:0/1838368049 conn(0x7fb35c0f89b0 msgr2=0x7fb35c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:00:07.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.094+0000 7fb363800700 1 -- 172.21.15.123:0/1838368049 shutdown_connections 2024-03-30T05:00:07.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.094+0000 7fb363800700 1 -- 172.21.15.123:0/1838368049 wait complete. 2024-03-30T05:00:07.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.094+0000 7fb363800700 1 Processor -- start 2024-03-30T05:00:07.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.095+0000 7fb363800700 1 -- start start 2024-03-30T05:00:07.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.095+0000 7fb363800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb35c102e00 0x7fb35c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:00:07.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.095+0000 7fb363800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb35c1037a0 0x7fb35c070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:00:07.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.095+0000 7fb363800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb35c070d90 con 0x7fb35c1037a0 2024-03-30T05:00:07.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.095+0000 7fb363800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb35c070ed0 con 0x7fb35c102e00 2024-03-30T05:00:07.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.095+0000 7fb35be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb35c1037a0 0x7fb35c070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:00:07.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.095+0000 7fb35be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb35c1037a0 0x7fb35c070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52728/0 (socket says 172.21.15.123:52728) 2024-03-30T05:00:07.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.095+0000 7fb35be00700 1 -- 172.21.15.123:0/380645205 learned_addr learned my addr 172.21.15.123:0/380645205 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:00:07.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.095+0000 7fb360e00700 1 --2- 172.21.15.123:0/380645205 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb35c102e00 0x7fb35c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:00:07.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.096+0000 7fb35be00700 1 -- 172.21.15.123:0/380645205 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb35c102e00 msgr2=0x7fb35c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:07.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.096+0000 7fb35be00700 1 --2- 172.21.15.123:0/380645205 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb35c102e00 0x7fb35c0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:07.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.096+0000 7fb35be00700 1 -- 172.21.15.123:0/380645205 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb350005040 con 0x7fb35c1037a0 2024-03-30T05:00:07.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.096+0000 7fb35be00700 1 --2- 172.21.15.123:0/380645205 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb35c1037a0 0x7fb35c070800 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fb348002a50 tx=0x7fb348004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:00:07.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.096+0000 7fb358c00700 1 -- 172.21.15.123:0/380645205 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb348007cc0 con 0x7fb35c1037a0 2024-03-30T05:00:07.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.096+0000 7fb358c00700 1 -- 172.21.15.123:0/380645205 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb34800f040 con 0x7fb35c1037a0 2024-03-30T05:00:07.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.096+0000 7fb363800700 1 -- 172.21.15.123:0/380645205 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb35c119e20 con 0x7fb35c1037a0 2024-03-30T05:00:07.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.096+0000 7fb358c00700 1 -- 172.21.15.123:0/380645205 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb34800d920 con 0x7fb35c1037a0 2024-03-30T05:00:07.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.096+0000 7fb363800700 1 -- 172.21.15.123:0/380645205 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb35c11a2a0 con 0x7fb35c1037a0 2024-03-30T05:00:07.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.098+0000 7fb363800700 1 -- 172.21.15.123:0/380645205 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb35c0fe550 con 0x7fb35c1037a0 2024-03-30T05:00:07.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.098+0000 7fb358c00700 1 -- 172.21.15.123:0/380645205 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fb348009680 con 0x7fb35c1037a0 2024-03-30T05:00:07.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.099+0000 7fb358c00700 1 --2- 172.21.15.123:0/380645205 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb34406cc30 0x7fb34406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:00:07.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.100+0000 7fb358c00700 1 -- 172.21.15.123:0/380645205 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb34808bd60 con 0x7fb35c1037a0 2024-03-30T05:00:07.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.100+0000 7fb360e00700 1 --2- 172.21.15.123:0/380645205 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb34406cc30 0x7fb34406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:00:07.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.100+0000 7fb360e00700 1 --2- 172.21.15.123:0/380645205 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb34406cc30 0x7fb34406f0e0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fb350009fd0 tx=0x7fb350009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:00:07.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:06 smithi135 ceph-mon[106011]: from='client.35580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:00:07.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:06 smithi135 ceph-mon[106011]: pgmap v868: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:07.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:06 smithi135 ceph-mon[106011]: from='client.35584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:00:07.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:06 smithi135 ceph-mon[106011]: from='client.35588 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:00:07.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:06 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2959088788' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:00:07.102 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.104+0000 7fb358c00700 1 -- 172.21.15.123:0/380645205 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb348090070 con 0x7fb35c1037a0 2024-03-30T05:00:07.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:06 smithi123 ceph-mon[139369]: from='client.35580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:00:07.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:06 smithi123 ceph-mon[139369]: pgmap v868: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:07.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:06 smithi123 ceph-mon[139369]: from='client.35584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:00:07.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:06 smithi123 ceph-mon[139369]: from='client.35588 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:00:07.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:06 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2959088788' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:00:07.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.326+0000 7fb363800700 1 -- 172.21.15.123:0/380645205 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb35c04bed0 con 0x7fb34406cc30 2024-03-30T05:00:07.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.328+0000 7fb358c00700 1 -- 172.21.15.123:0/380645205 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fb35c04bed0 con 0x7fb34406cc30 2024-03-30T05:00:07.326 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T05:00:07.326 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T05:00:07.326 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T05:00:07.326 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T05:00:07.326 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T05:00:07.326 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T05:00:07.326 INFO:teuthology.orchestra.run.smithi123.stdout: "mds", 2024-03-30T05:00:07.327 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T05:00:07.327 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T05:00:07.327 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T05:00:07.327 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T05:00:07.327 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "16/21 daemons upgraded", 2024-03-30T05:00:07.327 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T05:00:07.327 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T05:00:07.327 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T05:00:07.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.330+0000 7fb363800700 1 -- 172.21.15.123:0/380645205 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb34406cc30 msgr2=0x7fb34406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:07.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.330+0000 7fb363800700 1 --2- 172.21.15.123:0/380645205 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb34406cc30 0x7fb34406f0e0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fb350009fd0 tx=0x7fb350009e30 comp rx=0 tx=0).stop 2024-03-30T05:00:07.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.330+0000 7fb363800700 1 -- 172.21.15.123:0/380645205 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb35c1037a0 msgr2=0x7fb35c070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:07.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.330+0000 7fb363800700 1 --2- 172.21.15.123:0/380645205 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb35c1037a0 0x7fb35c070800 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fb348002a50 tx=0x7fb348004600 comp rx=0 tx=0).stop 2024-03-30T05:00:07.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.330+0000 7fb363800700 1 -- 172.21.15.123:0/380645205 shutdown_connections 2024-03-30T05:00:07.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.330+0000 7fb363800700 1 --2- 172.21.15.123:0/380645205 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb34406cc30 0x7fb34406f0e0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:07.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.330+0000 7fb363800700 1 --2- 172.21.15.123:0/380645205 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb35c102e00 0x7fb35c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:07.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.330+0000 7fb363800700 1 --2- 172.21.15.123:0/380645205 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb35c1037a0 0x7fb35c070800 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:07.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.330+0000 7fb363800700 1 -- 172.21.15.123:0/380645205 >> 172.21.15.123:0/380645205 conn(0x7fb35c0f89b0 msgr2=0x7fb35c0f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:00:07.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.330+0000 7fb363800700 1 -- 172.21.15.123:0/380645205 shutdown_connections 2024-03-30T05:00:07.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.330+0000 7fb363800700 1 -- 172.21.15.123:0/380645205 wait complete. 2024-03-30T05:00:07.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.462+0000 7f91c7c00700 1 -- 172.21.15.123:0/3646422365 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91c0102e00 msgr2=0x7f91c01031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:07.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.462+0000 7f91c7c00700 1 --2- 172.21.15.123:0/3646422365 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91c0102e00 0x7f91c01031d0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f91b4004940 tx=0x7f91b4009e30 comp rx=0 tx=0).stop 2024-03-30T05:00:07.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.462+0000 7f91c7c00700 1 -- 172.21.15.123:0/3646422365 shutdown_connections 2024-03-30T05:00:07.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.462+0000 7f91c7c00700 1 --2- 172.21.15.123:0/3646422365 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91c01037a0 0x7f91c00fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:07.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.462+0000 7f91c7c00700 1 --2- 172.21.15.123:0/3646422365 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91c0102e00 0x7f91c01031d0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:07.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.462+0000 7f91c7c00700 1 -- 172.21.15.123:0/3646422365 >> 172.21.15.123:0/3646422365 conn(0x7f91c00f89b0 msgr2=0x7f91c00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:00:07.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.462+0000 7f91c7c00700 1 -- 172.21.15.123:0/3646422365 shutdown_connections 2024-03-30T05:00:07.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.462+0000 7f91c7c00700 1 -- 172.21.15.123:0/3646422365 wait complete. 2024-03-30T05:00:07.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.463+0000 7f91c7c00700 1 Processor -- start 2024-03-30T05:00:07.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.463+0000 7f91c7c00700 1 -- start start 2024-03-30T05:00:07.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.464+0000 7f91c7c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91c0102e00 0x7f91c0115d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:00:07.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.464+0000 7f91c7c00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91c01037a0 0x7f91c0116250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:00:07.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.464+0000 7f91c7c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91c01168a0 con 0x7f91c0102e00 2024-03-30T05:00:07.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.464+0000 7f91c7c00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91c010fc70 con 0x7f91c01037a0 2024-03-30T05:00:07.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.464+0000 7f91c5200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91c0102e00 0x7f91c0115d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:00:07.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.464+0000 7f91c5200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91c0102e00 0x7f91c0115d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52748/0 (socket says 172.21.15.123:52748) 2024-03-30T05:00:07.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.464+0000 7f91c5200700 1 -- 172.21.15.123:0/2024172489 learned_addr learned my addr 172.21.15.123:0/2024172489 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:00:07.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.464+0000 7f91bfe00700 1 --2- 172.21.15.123:0/2024172489 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91c01037a0 0x7f91c0116250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:00:07.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.464+0000 7f91c5200700 1 -- 172.21.15.123:0/2024172489 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91c01037a0 msgr2=0x7f91c0116250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:07.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.464+0000 7f91c5200700 1 --2- 172.21.15.123:0/2024172489 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91c01037a0 0x7f91c0116250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:07.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.464+0000 7f91c5200700 1 -- 172.21.15.123:0/2024172489 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91b4005040 con 0x7f91c0102e00 2024-03-30T05:00:07.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.464+0000 7f91bfe00700 1 --2- 172.21.15.123:0/2024172489 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91c01037a0 0x7f91c0116250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T05:00:07.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.465+0000 7f91c5200700 1 --2- 172.21.15.123:0/2024172489 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91c0102e00 0x7f91c0115d10 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f91b4009390 tx=0x7f91b4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:00:07.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.465+0000 7f91bcc00700 1 -- 172.21.15.123:0/2024172489 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f91b400e040 con 0x7f91c0102e00 2024-03-30T05:00:07.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.465+0000 7f91bcc00700 1 -- 172.21.15.123:0/2024172489 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f91b4013420 con 0x7f91c0102e00 2024-03-30T05:00:07.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.465+0000 7f91c7c00700 1 -- 172.21.15.123:0/2024172489 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91c010fef0 con 0x7f91c0102e00 2024-03-30T05:00:07.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.465+0000 7f91bcc00700 1 -- 172.21.15.123:0/2024172489 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f91b401c630 con 0x7f91c0102e00 2024-03-30T05:00:07.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.465+0000 7f91c7c00700 1 -- 172.21.15.123:0/2024172489 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91c01104b0 con 0x7f91c0102e00 2024-03-30T05:00:07.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.466+0000 7f91c7c00700 1 -- 172.21.15.123:0/2024172489 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91c00fe550 con 0x7f91c0102e00 2024-03-30T05:00:07.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.467+0000 7f91bcc00700 1 -- 172.21.15.123:0/2024172489 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f91b40139e0 con 0x7f91c0102e00 2024-03-30T05:00:07.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.468+0000 7f91bcc00700 1 --2- 172.21.15.123:0/2024172489 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f91a806cc30 0x7f91a806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:00:07.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.468+0000 7f91bcc00700 1 -- 172.21.15.123:0/2024172489 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f91b408d320 con 0x7f91c0102e00 2024-03-30T05:00:07.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.468+0000 7f91bfe00700 1 --2- 172.21.15.123:0/2024172489 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f91a806cc30 0x7f91a806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:00:07.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.469+0000 7f91bfe00700 1 --2- 172.21.15.123:0/2024172489 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f91a806cc30 0x7f91a806f0e0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f91ac009fd0 tx=0x7f91ac009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:00:07.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.473+0000 7f91bcc00700 1 -- 172.21.15.123:0/2024172489 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f91b405ae20 con 0x7f91c0102e00 2024-03-30T05:00:07.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.762+0000 7f91c7c00700 1 -- 172.21.15.123:0/2024172489 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f91c004bed0 con 0x7f91c0102e00 2024-03-30T05:00:07.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.762+0000 7f91bcc00700 1 -- 172.21.15.123:0/2024172489 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f91b405a9b0 con 0x7f91c0102e00 2024-03-30T05:00:07.760 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T05:00:07.761 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-30T05:00:07.761 INFO:teuthology.orchestra.run.smithi123.stdout: host smithi135 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi135/config 2024-03-30T05:00:07.761 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-30T05:00:07.761 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T05:00:07.761 INFO:teuthology.orchestra.run.smithi123.stdout:Traceback (most recent call last): 2024-03-30T05:00:07.761 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-30T05:00:07.761 INFO:teuthology.orchestra.run.smithi123.stdout: "__main__", mod_spec) 2024-03-30T05:00:07.761 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-30T05:00:07.761 INFO:teuthology.orchestra.run.smithi123.stdout: exec(code, run_globals) 2024-03-30T05:00:07.761 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10889, in 2024-03-30T05:00:07.761 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10877, in main 2024-03-30T05:00:07.761 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-30T05:00:07.762 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-30T05:00:07.762 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-30T05:00:07.762 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-30T05:00:07.762 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-03-30T05:00:07.762 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-30T05:00:07.762 INFO:teuthology.orchestra.run.smithi123.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T05:00:07.762 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T05:00:07.762 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T05:00:07.762 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T05:00:07.762 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T05:00:07.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.764+0000 7f91c7c00700 1 -- 172.21.15.123:0/2024172489 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f91a806cc30 msgr2=0x7f91a806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:07.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.764+0000 7f91c7c00700 1 --2- 172.21.15.123:0/2024172489 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f91a806cc30 0x7f91a806f0e0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f91ac009fd0 tx=0x7f91ac009ee0 comp rx=0 tx=0).stop 2024-03-30T05:00:07.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.764+0000 7f91c7c00700 1 -- 172.21.15.123:0/2024172489 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91c0102e00 msgr2=0x7f91c0115d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:07.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.764+0000 7f91c7c00700 1 --2- 172.21.15.123:0/2024172489 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91c0102e00 0x7f91c0115d10 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f91b4009390 tx=0x7f91b4009e30 comp rx=0 tx=0).stop 2024-03-30T05:00:07.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.764+0000 7f91c7c00700 1 -- 172.21.15.123:0/2024172489 shutdown_connections 2024-03-30T05:00:07.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.764+0000 7f91c7c00700 1 --2- 172.21.15.123:0/2024172489 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f91a806cc30 0x7f91a806f0e0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:07.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.764+0000 7f91c7c00700 1 --2- 172.21.15.123:0/2024172489 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91c0102e00 0x7f91c0115d10 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:07.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.764+0000 7f91c7c00700 1 --2- 172.21.15.123:0/2024172489 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91c01037a0 0x7f91c0116250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:07.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.764+0000 7f91c7c00700 1 -- 172.21.15.123:0/2024172489 >> 172.21.15.123:0/2024172489 conn(0x7f91c00f89b0 msgr2=0x7f91c00704f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:00:07.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.765+0000 7f91c7c00700 1 -- 172.21.15.123:0/2024172489 shutdown_connections 2024-03-30T05:00:07.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:07.765+0000 7f91c7c00700 1 -- 172.21.15.123:0/2024172489 wait complete. 2024-03-30T05:00:08.057 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:07 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2224942447' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T05:00:08.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:07 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2224942447' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T05:00:09.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:08 smithi135 ceph-mon[106011]: from='client.35600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:00:09.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:08 smithi135 ceph-mon[106011]: pgmap v869: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:09.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:08 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2024172489' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T05:00:09.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:08 smithi123 ceph-mon[139369]: from='client.35600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:00:09.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:08 smithi123 ceph-mon[139369]: pgmap v869: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:09.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:08 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2024172489' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T05:00:11.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:10 smithi135 ceph-mon[106011]: pgmap v870: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:11.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:10 smithi123 ceph-mon[139369]: pgmap v870: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:13.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:12 smithi135 ceph-mon[106011]: pgmap v871: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:13.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:12 smithi123 ceph-mon[139369]: pgmap v871: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:15.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:14 smithi135 ceph-mon[106011]: pgmap v872: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:15.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:14 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:00:15.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:14 smithi123 ceph-mon[139369]: pgmap v872: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:15.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:14 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:00:17.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:16 smithi135 ceph-mon[106011]: pgmap v873: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:17.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:16 smithi123 ceph-mon[139369]: pgmap v873: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:19.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:18 smithi135 ceph-mon[106011]: pgmap v874: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:19.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:18 smithi123 ceph-mon[139369]: pgmap v874: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:21.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:20 smithi135 ceph-mon[106011]: pgmap v875: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:21.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:20 smithi123 ceph-mon[139369]: pgmap v875: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:23.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:22 smithi135 ceph-mon[106011]: pgmap v876: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:23.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:22 smithi123 ceph-mon[139369]: pgmap v876: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:25.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:24 smithi135 ceph-mon[106011]: pgmap v877: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:25.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:24 smithi123 ceph-mon[139369]: pgmap v877: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:27.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:26 smithi135 ceph-mon[106011]: pgmap v878: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:27.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:26 smithi123 ceph-mon[139369]: pgmap v878: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:29.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:28 smithi135 ceph-mon[106011]: pgmap v879: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:29.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:28 smithi123 ceph-mon[139369]: pgmap v879: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:30.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:29 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:00:30.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:29 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:00:31.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:30 smithi135 ceph-mon[106011]: pgmap v880: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:31.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:30 smithi123 ceph-mon[139369]: pgmap v880: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:33.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:32 smithi135 ceph-mon[106011]: pgmap v881: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:33.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:32 smithi123 ceph-mon[139369]: pgmap v881: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:35.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:34 smithi135 ceph-mon[106011]: pgmap v882: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:35.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:34 smithi123 ceph-mon[139369]: pgmap v882: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:37.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:36 smithi135 ceph-mon[106011]: pgmap v883: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:37.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:36 smithi123 ceph-mon[139369]: pgmap v883: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:37.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:37.913+0000 7fa6ede00700 1 -- 172.21.15.123:0/1477171910 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6e8102e00 msgr2=0x7fa6e81031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:37.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:37.913+0000 7fa6ede00700 1 --2- 172.21.15.123:0/1477171910 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6e8102e00 0x7fa6e81031d0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fa6d8004940 tx=0x7fa6d8009e30 comp rx=0 tx=0).stop 2024-03-30T05:00:37.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:37.913+0000 7fa6ede00700 1 -- 172.21.15.123:0/1477171910 shutdown_connections 2024-03-30T05:00:37.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:37.913+0000 7fa6ede00700 1 --2- 172.21.15.123:0/1477171910 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa6e81037a0 0x7fa6e80fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:37.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:37.913+0000 7fa6ede00700 1 --2- 172.21.15.123:0/1477171910 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6e8102e00 0x7fa6e81031d0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:37.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:37.913+0000 7fa6ede00700 1 -- 172.21.15.123:0/1477171910 >> 172.21.15.123:0/1477171910 conn(0x7fa6e80f89b0 msgr2=0x7fa6e80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:00:37.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:37.913+0000 7fa6ede00700 1 -- 172.21.15.123:0/1477171910 shutdown_connections 2024-03-30T05:00:37.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:37.914+0000 7fa6ede00700 1 -- 172.21.15.123:0/1477171910 wait complete. 2024-03-30T05:00:37.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:37.914+0000 7fa6ede00700 1 Processor -- start 2024-03-30T05:00:37.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:37.914+0000 7fa6ede00700 1 -- start start 2024-03-30T05:00:37.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:37.915+0000 7fa6ede00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6e8102e00 0x7fa6e8115d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:00:37.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:37.915+0000 7fa6ede00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa6e81037a0 0x7fa6e8116250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:00:37.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:37.915+0000 7fa6ede00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e81168a0 con 0x7fa6e8102e00 2024-03-30T05:00:37.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:37.915+0000 7fa6ede00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e810fc70 con 0x7fa6e81037a0 2024-03-30T05:00:37.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:37.915+0000 7fa6e6a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6e8102e00 0x7fa6e8115d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:00:37.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:37.915+0000 7fa6e6a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6e8102e00 0x7fa6e8115d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52630/0 (socket says 172.21.15.123:52630) 2024-03-30T05:00:37.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:37.915+0000 7fa6e6a00700 1 -- 172.21.15.123:0/3182124049 learned_addr learned my addr 172.21.15.123:0/3182124049 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:00:37.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:37.915+0000 7fa6e6000700 1 --2- 172.21.15.123:0/3182124049 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa6e81037a0 0x7fa6e8116250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:00:37.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:37.915+0000 7fa6e6a00700 1 -- 172.21.15.123:0/3182124049 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa6e81037a0 msgr2=0x7fa6e8116250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:37.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:37.915+0000 7fa6e6a00700 1 --2- 172.21.15.123:0/3182124049 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa6e81037a0 0x7fa6e8116250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:37.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:37.915+0000 7fa6e6a00700 1 -- 172.21.15.123:0/3182124049 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d8005040 con 0x7fa6e8102e00 2024-03-30T05:00:37.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:37.915+0000 7fa6e6000700 1 --2- 172.21.15.123:0/3182124049 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa6e81037a0 0x7fa6e8116250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T05:00:37.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:37.916+0000 7fa6e6a00700 1 --2- 172.21.15.123:0/3182124049 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6e8102e00 0x7fa6e8115d10 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fa6d8009390 tx=0x7fa6d8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:00:37.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:37.916+0000 7fa6dea00700 1 -- 172.21.15.123:0/3182124049 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa6d800e040 con 0x7fa6e8102e00 2024-03-30T05:00:37.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:37.916+0000 7fa6ede00700 1 -- 172.21.15.123:0/3182124049 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6e810fef0 con 0x7fa6e8102e00 2024-03-30T05:00:37.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:37.916+0000 7fa6dea00700 1 -- 172.21.15.123:0/3182124049 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa6d8013420 con 0x7fa6e8102e00 2024-03-30T05:00:37.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:37.916+0000 7fa6dea00700 1 -- 172.21.15.123:0/3182124049 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa6d801c630 con 0x7fa6e8102e00 2024-03-30T05:00:37.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:37.916+0000 7fa6ede00700 1 -- 172.21.15.123:0/3182124049 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6e81104b0 con 0x7fa6e8102e00 2024-03-30T05:00:37.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:37.917+0000 7fa6ede00700 1 -- 172.21.15.123:0/3182124049 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6e80fe550 con 0x7fa6e8102e00 2024-03-30T05:00:37.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:37.918+0000 7fa6dea00700 1 -- 172.21.15.123:0/3182124049 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fa6d80139e0 con 0x7fa6e8102e00 2024-03-30T05:00:37.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:37.919+0000 7fa6dea00700 1 --2- 172.21.15.123:0/3182124049 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa6cc06cc30 0x7fa6cc06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:00:37.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:37.919+0000 7fa6dea00700 1 -- 172.21.15.123:0/3182124049 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa6d808d320 con 0x7fa6e8102e00 2024-03-30T05:00:37.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:37.919+0000 7fa6e6000700 1 --2- 172.21.15.123:0/3182124049 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa6cc06cc30 0x7fa6cc06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:00:37.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:37.920+0000 7fa6e6000700 1 --2- 172.21.15.123:0/3182124049 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa6cc06cc30 0x7fa6cc06f0e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fa6d4009fd0 tx=0x7fa6d4009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:00:37.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:37.924+0000 7fa6dea00700 1 -- 172.21.15.123:0/3182124049 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa6d805ae20 con 0x7fa6e8102e00 2024-03-30T05:00:38.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.146+0000 7fa6ede00700 1 -- 172.21.15.123:0/3182124049 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa6e804bf40 con 0x7fa6cc06cc30 2024-03-30T05:00:38.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.147+0000 7fa6dea00700 1 -- 172.21.15.123:0/3182124049 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa6e804bf40 con 0x7fa6cc06cc30 2024-03-30T05:00:38.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.150+0000 7fa6ede00700 1 -- 172.21.15.123:0/3182124049 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa6cc06cc30 msgr2=0x7fa6cc06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:38.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.150+0000 7fa6ede00700 1 --2- 172.21.15.123:0/3182124049 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa6cc06cc30 0x7fa6cc06f0e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fa6d4009fd0 tx=0x7fa6d4009ee0 comp rx=0 tx=0).stop 2024-03-30T05:00:38.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.150+0000 7fa6ede00700 1 -- 172.21.15.123:0/3182124049 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6e8102e00 msgr2=0x7fa6e8115d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:38.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.150+0000 7fa6ede00700 1 --2- 172.21.15.123:0/3182124049 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6e8102e00 0x7fa6e8115d10 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fa6d8009390 tx=0x7fa6d8009e30 comp rx=0 tx=0).stop 2024-03-30T05:00:38.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.150+0000 7fa6ede00700 1 -- 172.21.15.123:0/3182124049 shutdown_connections 2024-03-30T05:00:38.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.150+0000 7fa6ede00700 1 --2- 172.21.15.123:0/3182124049 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa6cc06cc30 0x7fa6cc06f0e0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:38.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.150+0000 7fa6ede00700 1 --2- 172.21.15.123:0/3182124049 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6e8102e00 0x7fa6e8115d10 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:38.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.150+0000 7fa6ede00700 1 --2- 172.21.15.123:0/3182124049 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa6e81037a0 0x7fa6e8116250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:38.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.150+0000 7fa6ede00700 1 -- 172.21.15.123:0/3182124049 >> 172.21.15.123:0/3182124049 conn(0x7fa6e80f89b0 msgr2=0x7fa6e80704f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:00:38.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.150+0000 7fa6ede00700 1 -- 172.21.15.123:0/3182124049 shutdown_connections 2024-03-30T05:00:38.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.150+0000 7fa6ede00700 1 -- 172.21.15.123:0/3182124049 wait complete. 2024-03-30T05:00:38.163 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T05:00:38.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.285+0000 7fdb3ba00700 1 -- 172.21.15.123:0/913623739 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb34102e00 msgr2=0x7fdb341031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:38.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.285+0000 7fdb3ba00700 1 --2- 172.21.15.123:0/913623739 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb34102e00 0x7fdb341031d0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fdb28004940 tx=0x7fdb28009e30 comp rx=0 tx=0).stop 2024-03-30T05:00:38.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.285+0000 7fdb3ba00700 1 -- 172.21.15.123:0/913623739 shutdown_connections 2024-03-30T05:00:38.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.285+0000 7fdb3ba00700 1 --2- 172.21.15.123:0/913623739 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdb341037a0 0x7fdb340fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:38.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.285+0000 7fdb3ba00700 1 --2- 172.21.15.123:0/913623739 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb34102e00 0x7fdb341031d0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:38.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.285+0000 7fdb3ba00700 1 -- 172.21.15.123:0/913623739 >> 172.21.15.123:0/913623739 conn(0x7fdb340f89b0 msgr2=0x7fdb340fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:00:38.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.285+0000 7fdb3ba00700 1 -- 172.21.15.123:0/913623739 shutdown_connections 2024-03-30T05:00:38.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.285+0000 7fdb3ba00700 1 -- 172.21.15.123:0/913623739 wait complete. 2024-03-30T05:00:38.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.286+0000 7fdb3ba00700 1 Processor -- start 2024-03-30T05:00:38.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.286+0000 7fdb3ba00700 1 -- start start 2024-03-30T05:00:38.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.287+0000 7fdb3ba00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdb34102e00 0x7fdb34115c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:00:38.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.287+0000 7fdb3ba00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb341037a0 0x7fdb34116190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:00:38.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.287+0000 7fdb3ba00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb341167e0 con 0x7fdb341037a0 2024-03-30T05:00:38.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.287+0000 7fdb3ba00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3410fbb0 con 0x7fdb34102e00 2024-03-30T05:00:38.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.287+0000 7fdb33e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb341037a0 0x7fdb34116190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:00:38.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.287+0000 7fdb33e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb341037a0 0x7fdb34116190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52646/0 (socket says 172.21.15.123:52646) 2024-03-30T05:00:38.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.287+0000 7fdb33e00700 1 -- 172.21.15.123:0/4067547347 learned_addr learned my addr 172.21.15.123:0/4067547347 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:00:38.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.287+0000 7fdb39000700 1 --2- 172.21.15.123:0/4067547347 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdb34102e00 0x7fdb34115c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:00:38.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.287+0000 7fdb33e00700 1 -- 172.21.15.123:0/4067547347 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdb34102e00 msgr2=0x7fdb34115c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:38.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.287+0000 7fdb33e00700 1 --2- 172.21.15.123:0/4067547347 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdb34102e00 0x7fdb34115c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:38.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.287+0000 7fdb33e00700 1 -- 172.21.15.123:0/4067547347 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb28005040 con 0x7fdb341037a0 2024-03-30T05:00:38.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.287+0000 7fdb39000700 1 --2- 172.21.15.123:0/4067547347 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdb34102e00 0x7fdb34115c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T05:00:38.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.288+0000 7fdb33e00700 1 --2- 172.21.15.123:0/4067547347 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb341037a0 0x7fdb34116190 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fdb2000eb40 tx=0x7fdb2000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:00:38.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.288+0000 7fdb30c00700 1 -- 172.21.15.123:0/4067547347 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb20007cc0 con 0x7fdb341037a0 2024-03-30T05:00:38.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.288+0000 7fdb30c00700 1 -- 172.21.15.123:0/4067547347 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdb2000f040 con 0x7fdb341037a0 2024-03-30T05:00:38.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.288+0000 7fdb30c00700 1 -- 172.21.15.123:0/4067547347 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb20009570 con 0x7fdb341037a0 2024-03-30T05:00:38.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.288+0000 7fdb3ba00700 1 -- 172.21.15.123:0/4067547347 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb3410fe30 con 0x7fdb341037a0 2024-03-30T05:00:38.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.288+0000 7fdb3ba00700 1 -- 172.21.15.123:0/4067547347 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb341103f0 con 0x7fdb341037a0 2024-03-30T05:00:38.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.289+0000 7fdb3ba00700 1 -- 172.21.15.123:0/4067547347 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb340fe550 con 0x7fdb341037a0 2024-03-30T05:00:38.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.290+0000 7fdb30c00700 1 -- 172.21.15.123:0/4067547347 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fdb200096d0 con 0x7fdb341037a0 2024-03-30T05:00:38.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.291+0000 7fdb30c00700 1 --2- 172.21.15.123:0/4067547347 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fdb1c06cd00 0x7fdb1c06f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:00:38.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.291+0000 7fdb30c00700 1 -- 172.21.15.123:0/4067547347 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fdb2008bb00 con 0x7fdb341037a0 2024-03-30T05:00:38.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.291+0000 7fdb39000700 1 --2- 172.21.15.123:0/4067547347 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fdb1c06cd00 0x7fdb1c06f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:00:38.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.292+0000 7fdb39000700 1 --2- 172.21.15.123:0/4067547347 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fdb1c06cd00 0x7fdb1c06f1b0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fdb280136f0 tx=0x7fdb28009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:00:38.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.295+0000 7fdb30c00700 1 -- 172.21.15.123:0/4067547347 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fdb20059600 con 0x7fdb341037a0 2024-03-30T05:00:38.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.515+0000 7fdb3ba00700 1 -- 172.21.15.123:0/4067547347 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdb34109dc0 con 0x7fdb1c06cd00 2024-03-30T05:00:38.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.517+0000 7fdb30c00700 1 -- 172.21.15.123:0/4067547347 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fdb34109dc0 con 0x7fdb1c06cd00 2024-03-30T05:00:38.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.519+0000 7fdb3ba00700 1 -- 172.21.15.123:0/4067547347 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fdb1c06cd00 msgr2=0x7fdb1c06f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:38.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.519+0000 7fdb3ba00700 1 --2- 172.21.15.123:0/4067547347 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fdb1c06cd00 0x7fdb1c06f1b0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fdb280136f0 tx=0x7fdb28009e30 comp rx=0 tx=0).stop 2024-03-30T05:00:38.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.519+0000 7fdb3ba00700 1 -- 172.21.15.123:0/4067547347 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb341037a0 msgr2=0x7fdb34116190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:38.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.519+0000 7fdb3ba00700 1 --2- 172.21.15.123:0/4067547347 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb341037a0 0x7fdb34116190 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fdb2000eb40 tx=0x7fdb2000b040 comp rx=0 tx=0).stop 2024-03-30T05:00:38.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.519+0000 7fdb3ba00700 1 -- 172.21.15.123:0/4067547347 shutdown_connections 2024-03-30T05:00:38.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.519+0000 7fdb3ba00700 1 --2- 172.21.15.123:0/4067547347 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fdb1c06cd00 0x7fdb1c06f1b0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:38.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.519+0000 7fdb3ba00700 1 --2- 172.21.15.123:0/4067547347 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdb34102e00 0x7fdb34115c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:38.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.519+0000 7fdb3ba00700 1 --2- 172.21.15.123:0/4067547347 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb341037a0 0x7fdb34116190 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:38.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.519+0000 7fdb3ba00700 1 -- 172.21.15.123:0/4067547347 >> 172.21.15.123:0/4067547347 conn(0x7fdb340f89b0 msgr2=0x7fdb340f9900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:00:38.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.519+0000 7fdb3ba00700 1 -- 172.21.15.123:0/4067547347 shutdown_connections 2024-03-30T05:00:38.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.519+0000 7fdb3ba00700 1 -- 172.21.15.123:0/4067547347 wait complete. 2024-03-30T05:00:38.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.653+0000 7f1b40200700 1 -- 172.21.15.123:0/1651893009 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b38070360 msgr2=0x7f1b38070730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:38.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.653+0000 7f1b40200700 1 --2- 172.21.15.123:0/1651893009 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b38070360 0x7f1b38070730 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c004970 tx=0x7f1b2c009e30 comp rx=0 tx=0).stop 2024-03-30T05:00:38.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.653+0000 7f1b40200700 1 -- 172.21.15.123:0/1651893009 shutdown_connections 2024-03-30T05:00:38.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.653+0000 7f1b40200700 1 --2- 172.21.15.123:0/1651893009 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1b38070c70 0x7f1b38109110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:38.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.653+0000 7f1b40200700 1 --2- 172.21.15.123:0/1651893009 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b38070360 0x7f1b38070730 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:38.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.653+0000 7f1b40200700 1 -- 172.21.15.123:0/1651893009 >> 172.21.15.123:0/1651893009 conn(0x7f1b380f89b0 msgr2=0x7f1b380fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:00:38.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.653+0000 7f1b40200700 1 -- 172.21.15.123:0/1651893009 shutdown_connections 2024-03-30T05:00:38.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.653+0000 7f1b40200700 1 -- 172.21.15.123:0/1651893009 wait complete. 2024-03-30T05:00:38.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.653+0000 7f1b40200700 1 Processor -- start 2024-03-30T05:00:38.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.654+0000 7f1b40200700 1 -- start start 2024-03-30T05:00:38.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.654+0000 7f1b40200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1b38070360 0x7f1b38115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:00:38.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.654+0000 7f1b40200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b38070c70 0x7f1b381161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:00:38.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.654+0000 7f1b40200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b38116810 con 0x7f1b38070c70 2024-03-30T05:00:38.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.654+0000 7f1b40200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3810fbe0 con 0x7f1b38070360 2024-03-30T05:00:38.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.654+0000 7f1b3ce00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b38070c70 0x7f1b381161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:00:38.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.654+0000 7f1b3ce00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b38070c70 0x7f1b381161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52656/0 (socket says 172.21.15.123:52656) 2024-03-30T05:00:38.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.654+0000 7f1b3ce00700 1 -- 172.21.15.123:0/3739015351 learned_addr learned my addr 172.21.15.123:0/3739015351 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:00:38.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.654+0000 7f1b3d800700 1 --2- 172.21.15.123:0/3739015351 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1b38070360 0x7f1b38115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:00:38.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.655+0000 7f1b3ce00700 1 -- 172.21.15.123:0/3739015351 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1b38070360 msgr2=0x7f1b38115c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:38.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.655+0000 7f1b3ce00700 1 --2- 172.21.15.123:0/3739015351 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1b38070360 0x7f1b38115c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:38.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.655+0000 7f1b3ce00700 1 -- 172.21.15.123:0/3739015351 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b2c005040 con 0x7f1b38070c70 2024-03-30T05:00:38.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.655+0000 7f1b3ce00700 1 --2- 172.21.15.123:0/3739015351 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b38070c70 0x7f1b381161c0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f1b24002a80 tx=0x7f1b240046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:00:38.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.655+0000 7f1b35600700 1 -- 172.21.15.123:0/3739015351 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b24007cc0 con 0x7f1b38070c70 2024-03-30T05:00:38.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.655+0000 7f1b35600700 1 -- 172.21.15.123:0/3739015351 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1b2400f040 con 0x7f1b38070c70 2024-03-30T05:00:38.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.655+0000 7f1b40200700 1 -- 172.21.15.123:0/3739015351 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b3810fec0 con 0x7f1b38070c70 2024-03-30T05:00:38.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.655+0000 7f1b35600700 1 -- 172.21.15.123:0/3739015351 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b2400da60 con 0x7f1b38070c70 2024-03-30T05:00:38.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.655+0000 7f1b40200700 1 -- 172.21.15.123:0/3739015351 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b38110400 con 0x7f1b38070c70 2024-03-30T05:00:38.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.656+0000 7f1b40200700 1 -- 172.21.15.123:0/3739015351 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b38103b90 con 0x7f1b38070c70 2024-03-30T05:00:38.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.657+0000 7f1b35600700 1 -- 172.21.15.123:0/3739015351 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f1b2400dbc0 con 0x7f1b38070c70 2024-03-30T05:00:38.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.659+0000 7f1b35600700 1 --2- 172.21.15.123:0/3739015351 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1b2006cc30 0x7f1b2006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:00:38.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.659+0000 7f1b35600700 1 -- 172.21.15.123:0/3739015351 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f1b2408bc70 con 0x7f1b38070c70 2024-03-30T05:00:38.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.659+0000 7f1b3d800700 1 --2- 172.21.15.123:0/3739015351 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1b2006cc30 0x7f1b2006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:00:38.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.660+0000 7f1b3d800700 1 --2- 172.21.15.123:0/3739015351 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1b2006cc30 0x7f1b2006f0e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c00b370 tx=0x7f1b2c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:00:38.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.663+0000 7f1b35600700 1 -- 172.21.15.123:0/3739015351 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f1b24092080 con 0x7f1b38070c70 2024-03-30T05:00:38.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.875+0000 7f1b40200700 1 -- 172.21.15.123:0/3739015351 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f1b3810bfb0 con 0x7f1b2006cc30 2024-03-30T05:00:38.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.886+0000 7f1b35600700 1 -- 172.21.15.123:0/3739015351 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f1b3810bfb0 con 0x7f1b2006cc30 2024-03-30T05:00:38.884 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T05:00:38.884 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (29m) 7m ago 42m 14.6M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T05:00:38.884 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (27m) 7m ago 42m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T05:00:38.884 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (26m) 7m ago 41m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T05:00:38.884 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (29m) 7m ago 42m 73.5M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T05:00:38.884 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (22m) 7m ago 35m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T05:00:38.884 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (23m) 7m ago 36m 16.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T05:00:38.884 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (7m) 7m ago 35m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 87b073a6c9ca 2024-03-30T05:00:38.884 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (22m) 7m ago 36m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T05:00:38.885 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (32m) 7m ago 43m 510M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T05:00:38.885 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (31m) 7m ago 41m 456M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T05:00:38.885 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (28m) 7m ago 44m 70.0M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T05:00:38.885 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (27m) 7m ago 41m 51.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T05:00:38.885 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (30m) 7m ago 42m 8636k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T05:00:38.885 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (30m) 7m ago 41m 8678k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T05:00:38.885 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (26m) 7m ago 40m 107M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T05:00:38.885 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (26m) 7m ago 39m 97.2M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T05:00:38.885 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (25m) 7m ago 39m 89.8M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T05:00:38.885 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (24m) 7m ago 38m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T05:00:38.885 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (24m) 7m ago 38m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T05:00:38.885 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (23m) 7m ago 37m 88.8M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T05:00:38.885 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (30m) 7m ago 41m 63.9M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T05:00:38.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.888+0000 7f1b40200700 1 -- 172.21.15.123:0/3739015351 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1b2006cc30 msgr2=0x7f1b2006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:38.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.888+0000 7f1b40200700 1 --2- 172.21.15.123:0/3739015351 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1b2006cc30 0x7f1b2006f0e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c00b370 tx=0x7f1b2c009e30 comp rx=0 tx=0).stop 2024-03-30T05:00:38.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.888+0000 7f1b40200700 1 -- 172.21.15.123:0/3739015351 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b38070c70 msgr2=0x7f1b381161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:38.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.888+0000 7f1b40200700 1 --2- 172.21.15.123:0/3739015351 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b38070c70 0x7f1b381161c0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f1b24002a80 tx=0x7f1b240046b0 comp rx=0 tx=0).stop 2024-03-30T05:00:38.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.888+0000 7f1b40200700 1 -- 172.21.15.123:0/3739015351 shutdown_connections 2024-03-30T05:00:38.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.888+0000 7f1b40200700 1 --2- 172.21.15.123:0/3739015351 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1b2006cc30 0x7f1b2006f0e0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:38.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.888+0000 7f1b40200700 1 --2- 172.21.15.123:0/3739015351 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1b38070360 0x7f1b38115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:38.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.888+0000 7f1b40200700 1 --2- 172.21.15.123:0/3739015351 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b38070c70 0x7f1b381161c0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:38.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.888+0000 7f1b40200700 1 -- 172.21.15.123:0/3739015351 >> 172.21.15.123:0/3739015351 conn(0x7f1b380f89b0 msgr2=0x7f1b381015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:00:38.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.888+0000 7f1b40200700 1 -- 172.21.15.123:0/3739015351 shutdown_connections 2024-03-30T05:00:38.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:38.888+0000 7f1b40200700 1 -- 172.21.15.123:0/3739015351 wait complete. 2024-03-30T05:00:39.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.020+0000 7f2136c00700 1 -- 172.21.15.123:0/1433195709 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21301037a0 msgr2=0x7f21300fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:39.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.020+0000 7f2136c00700 1 --2- 172.21.15.123:0/1433195709 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21301037a0 0x7f21300fce10 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f211c004940 tx=0x7f211c009e30 comp rx=0 tx=0).stop 2024-03-30T05:00:39.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.020+0000 7f2136c00700 1 -- 172.21.15.123:0/1433195709 shutdown_connections 2024-03-30T05:00:39.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.020+0000 7f2136c00700 1 --2- 172.21.15.123:0/1433195709 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21301037a0 0x7f21300fce10 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:39.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.020+0000 7f2136c00700 1 --2- 172.21.15.123:0/1433195709 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2130102e00 0x7f21301031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:39.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.020+0000 7f2136c00700 1 -- 172.21.15.123:0/1433195709 >> 172.21.15.123:0/1433195709 conn(0x7f21300f89b0 msgr2=0x7f21300fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:00:39.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.020+0000 7f2136c00700 1 -- 172.21.15.123:0/1433195709 shutdown_connections 2024-03-30T05:00:39.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.020+0000 7f2136c00700 1 -- 172.21.15.123:0/1433195709 wait complete. 2024-03-30T05:00:39.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.021+0000 7f2136c00700 1 Processor -- start 2024-03-30T05:00:39.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.021+0000 7f2136c00700 1 -- start start 2024-03-30T05:00:39.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.021+0000 7f2136c00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2130102e00 0x7f2130115d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:00:39.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.021+0000 7f2136c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21301037a0 0x7f2130116240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:00:39.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.021+0000 7f2136c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2130116890 con 0x7f21301037a0 2024-03-30T05:00:39.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.021+0000 7f2136c00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f213010fc60 con 0x7f2130102e00 2024-03-30T05:00:39.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.022+0000 7f212f400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21301037a0 0x7f2130116240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:00:39.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.022+0000 7f212f400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21301037a0 0x7f2130116240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52688/0 (socket says 172.21.15.123:52688) 2024-03-30T05:00:39.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.022+0000 7f212f400700 1 -- 172.21.15.123:0/4152915878 learned_addr learned my addr 172.21.15.123:0/4152915878 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:00:39.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.022+0000 7f212f400700 1 -- 172.21.15.123:0/4152915878 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2130102e00 msgr2=0x7f2130115d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:39.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.022+0000 7f212f400700 1 --2- 172.21.15.123:0/4152915878 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2130102e00 0x7f2130115d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:39.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.022+0000 7f212f400700 1 -- 172.21.15.123:0/4152915878 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f211c005040 con 0x7f21301037a0 2024-03-30T05:00:39.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.022+0000 7f212f400700 1 --2- 172.21.15.123:0/4152915878 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21301037a0 0x7f2130116240 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f211c009390 tx=0x7f211c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:00:39.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.022+0000 7f2127e00700 1 -- 172.21.15.123:0/4152915878 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f211c00e040 con 0x7f21301037a0 2024-03-30T05:00:39.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.022+0000 7f2127e00700 1 -- 172.21.15.123:0/4152915878 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f211c013420 con 0x7f21301037a0 2024-03-30T05:00:39.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.023+0000 7f2136c00700 1 -- 172.21.15.123:0/4152915878 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f213010fee0 con 0x7f21301037a0 2024-03-30T05:00:39.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.023+0000 7f2127e00700 1 -- 172.21.15.123:0/4152915878 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f211c01c630 con 0x7f21301037a0 2024-03-30T05:00:39.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.023+0000 7f2136c00700 1 -- 172.21.15.123:0/4152915878 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21301104a0 con 0x7f21301037a0 2024-03-30T05:00:39.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.024+0000 7f2136c00700 1 -- 172.21.15.123:0/4152915878 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21300fe550 con 0x7f21301037a0 2024-03-30T05:00:39.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.025+0000 7f2127e00700 1 -- 172.21.15.123:0/4152915878 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f211c013a00 con 0x7f21301037a0 2024-03-30T05:00:39.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.025+0000 7f2127e00700 1 --2- 172.21.15.123:0/4152915878 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f211406cc30 0x7f211406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:00:39.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.026+0000 7f2127e00700 1 -- 172.21.15.123:0/4152915878 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f211c08d2c0 con 0x7f21301037a0 2024-03-30T05:00:39.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.026+0000 7f212fe00700 1 --2- 172.21.15.123:0/4152915878 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f211406cc30 0x7f211406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:00:39.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.026+0000 7f212fe00700 1 --2- 172.21.15.123:0/4152915878 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f211406cc30 0x7f211406f0e0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f2120004d00 tx=0x7f2120009ce0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:00:39.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.030+0000 7f2127e00700 1 -- 172.21.15.123:0/4152915878 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f211c05ad90 con 0x7f21301037a0 2024-03-30T05:00:39.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:38 smithi135 ceph-mon[106011]: pgmap v884: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:39.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:38 smithi123 ceph-mon[139369]: pgmap v884: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:39.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.315+0000 7f2136c00700 1 -- 172.21.15.123:0/4152915878 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f213004bed0 con 0x7f21301037a0 2024-03-30T05:00:39.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.316+0000 7f2127e00700 1 -- 172.21.15.123:0/4152915878 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f211c05a920 con 0x7f21301037a0 2024-03-30T05:00:39.314 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T05:00:39.314 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T05:00:39.314 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T05:00:39.314 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:00:39.314 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T05:00:39.314 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T05:00:39.314 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:00:39.314 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T05:00:39.315 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T05:00:39.315 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:00:39.315 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T05:00:39.315 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 4 2024-03-30T05:00:39.315 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:00:39.315 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T05:00:39.315 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 14 2024-03-30T05:00:39.315 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T05:00:39.315 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T05:00:39.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.318+0000 7f2136c00700 1 -- 172.21.15.123:0/4152915878 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f211406cc30 msgr2=0x7f211406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:39.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.318+0000 7f2136c00700 1 --2- 172.21.15.123:0/4152915878 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f211406cc30 0x7f211406f0e0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f2120004d00 tx=0x7f2120009ce0 comp rx=0 tx=0).stop 2024-03-30T05:00:39.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.318+0000 7f2136c00700 1 -- 172.21.15.123:0/4152915878 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21301037a0 msgr2=0x7f2130116240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:39.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.318+0000 7f2136c00700 1 --2- 172.21.15.123:0/4152915878 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21301037a0 0x7f2130116240 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f211c009390 tx=0x7f211c009e30 comp rx=0 tx=0).stop 2024-03-30T05:00:39.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.318+0000 7f2136c00700 1 -- 172.21.15.123:0/4152915878 shutdown_connections 2024-03-30T05:00:39.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.318+0000 7f2136c00700 1 --2- 172.21.15.123:0/4152915878 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f211406cc30 0x7f211406f0e0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:39.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.318+0000 7f2136c00700 1 --2- 172.21.15.123:0/4152915878 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2130102e00 0x7f2130115d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:39.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.318+0000 7f2136c00700 1 --2- 172.21.15.123:0/4152915878 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21301037a0 0x7f2130116240 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:39.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.318+0000 7f2136c00700 1 -- 172.21.15.123:0/4152915878 >> 172.21.15.123:0/4152915878 conn(0x7f21300f89b0 msgr2=0x7f21300f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:00:39.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.318+0000 7f2136c00700 1 -- 172.21.15.123:0/4152915878 shutdown_connections 2024-03-30T05:00:39.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.318+0000 7f2136c00700 1 -- 172.21.15.123:0/4152915878 wait complete. 2024-03-30T05:00:39.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.451+0000 7f32f8400700 1 -- 172.21.15.123:0/2198816481 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32f00fe850 msgr2=0x7f32f0106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:39.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.451+0000 7f32f8400700 1 --2- 172.21.15.123:0/2198816481 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32f00fe850 0x7f32f0106f20 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f32dc004970 tx=0x7f32dc009e30 comp rx=0 tx=0).stop 2024-03-30T05:00:39.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.451+0000 7f32f8400700 1 -- 172.21.15.123:0/2198816481 shutdown_connections 2024-03-30T05:00:39.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.451+0000 7f32f8400700 1 --2- 172.21.15.123:0/2198816481 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32f00fe850 0x7f32f0106f20 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:39.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.451+0000 7f32f8400700 1 --2- 172.21.15.123:0/2198816481 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f32f00fdeb0 0x7f32f00fe280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:39.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.451+0000 7f32f8400700 1 -- 172.21.15.123:0/2198816481 >> 172.21.15.123:0/2198816481 conn(0x7f32f0072bc0 msgr2=0x7f32f0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:00:39.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.451+0000 7f32f8400700 1 -- 172.21.15.123:0/2198816481 shutdown_connections 2024-03-30T05:00:39.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.452+0000 7f32f8400700 1 -- 172.21.15.123:0/2198816481 wait complete. 2024-03-30T05:00:39.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.452+0000 7f32f8400700 1 Processor -- start 2024-03-30T05:00:39.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.452+0000 7f32f8400700 1 -- start start 2024-03-30T05:00:39.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.453+0000 7f32f8400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32f00fdeb0 0x7f32f00fbd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:00:39.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.453+0000 7f32f8400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f32f00fe850 0x7f32f00fc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:00:39.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.453+0000 7f32f8400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f00fc840 con 0x7f32f00fdeb0 2024-03-30T05:00:39.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.453+0000 7f32f8400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f00fc980 con 0x7f32f00fe850 2024-03-30T05:00:39.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.453+0000 7f32f5a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32f00fdeb0 0x7f32f00fbd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:00:39.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.453+0000 7f32f5a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32f00fdeb0 0x7f32f00fbd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52698/0 (socket says 172.21.15.123:52698) 2024-03-30T05:00:39.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.453+0000 7f32f5a00700 1 -- 172.21.15.123:0/562846052 learned_addr learned my addr 172.21.15.123:0/562846052 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:00:39.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.453+0000 7f32f5000700 1 --2- 172.21.15.123:0/562846052 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f32f00fe850 0x7f32f00fc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:00:39.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.453+0000 7f32f5a00700 1 -- 172.21.15.123:0/562846052 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f32f00fe850 msgr2=0x7f32f00fc2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:39.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.453+0000 7f32f5a00700 1 --2- 172.21.15.123:0/562846052 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f32f00fe850 0x7f32f00fc2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:39.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.453+0000 7f32f5a00700 1 -- 172.21.15.123:0/562846052 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32dc005040 con 0x7f32f00fdeb0 2024-03-30T05:00:39.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.454+0000 7f32f5a00700 1 --2- 172.21.15.123:0/562846052 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32f00fdeb0 0x7f32f00fbd70 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f32e400eb40 tx=0x7f32e400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:00:39.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.454+0000 7f32ed600700 1 -- 172.21.15.123:0/562846052 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32e4007cc0 con 0x7f32f00fdeb0 2024-03-30T05:00:39.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.454+0000 7f32ed600700 1 -- 172.21.15.123:0/562846052 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f32e400f040 con 0x7f32f00fdeb0 2024-03-30T05:00:39.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.454+0000 7f32ed600700 1 -- 172.21.15.123:0/562846052 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32e4009570 con 0x7f32f00fdeb0 2024-03-30T05:00:39.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.454+0000 7f32f8400700 1 -- 172.21.15.123:0/562846052 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32f00fcbd0 con 0x7f32f00fdeb0 2024-03-30T05:00:39.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.454+0000 7f32f8400700 1 -- 172.21.15.123:0/562846052 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32f011a0c0 con 0x7f32f00fdeb0 2024-03-30T05:00:39.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.455+0000 7f32f8400700 1 -- 172.21.15.123:0/562846052 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32f0065e10 con 0x7f32f00fdeb0 2024-03-30T05:00:39.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.456+0000 7f32ed600700 1 -- 172.21.15.123:0/562846052 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f32e40096d0 con 0x7f32f00fdeb0 2024-03-30T05:00:39.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.457+0000 7f32ed600700 1 --2- 172.21.15.123:0/562846052 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f32d806cc20 0x7f32d806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:00:39.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.458+0000 7f32ed600700 1 -- 172.21.15.123:0/562846052 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f32e408bca0 con 0x7f32f00fdeb0 2024-03-30T05:00:39.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.458+0000 7f32f5000700 1 --2- 172.21.15.123:0/562846052 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f32d806cc20 0x7f32d806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:00:39.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.458+0000 7f32f5000700 1 --2- 172.21.15.123:0/562846052 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f32d806cc20 0x7f32d806f0d0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f32f00fd6b0 tx=0x7f32dc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:00:39.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.462+0000 7f32ed600700 1 -- 172.21.15.123:0/562846052 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f32e4055d10 con 0x7f32f00fdeb0 2024-03-30T05:00:39.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.708+0000 7f32f8400700 1 -- 172.21.15.123:0/562846052 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f32f00fd460 con 0x7f32f00fdeb0 2024-03-30T05:00:39.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.708+0000 7f32ed600700 1 -- 172.21.15.123:0/562846052 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 34 v34) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f32e4019750 con 0x7f32f00fdeb0 2024-03-30T05:00:39.706 INFO:teuthology.orchestra.run.smithi123.stdout:e34 2024-03-30T05:00:39.707 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T05:00:39.707 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T05:00:39.707 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T05:00:39.707 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:00:39.707 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T05:00:39.707 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T05:00:39.707 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T05:00:39.707 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T05:00:39.707 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T05:00:39.707 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T05:00:39.707 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T05:00:39.707 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T05:00:39.708 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T05:00:39.708 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T05:00:39.708 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T05:00:39.708 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T05:00:39.708 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T05:00:39.708 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T05:00:39.708 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T05:00:39.708 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T05:00:39.708 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T05:00:39.708 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T05:00:39.708 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T05:00:39.708 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T05:00:39.708 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T05:00:39.708 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T05:00:39.709 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T05:00:39.709 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T05:00:39.709 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T05:00:39.709 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T05:00:39.709 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T05:00:39.709 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T05:00:39.709 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:00:39.709 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:00:39.709 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T05:00:39.709 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:00:39.709 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:00:39.709 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:00:39.709 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:00:39.710 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44975} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/3788988014,v1:172.21.15.135:6827/3788988014] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:00:39.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.710+0000 7f32f8400700 1 -- 172.21.15.123:0/562846052 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f32d806cc20 msgr2=0x7f32d806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:39.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.710+0000 7f32f8400700 1 --2- 172.21.15.123:0/562846052 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f32d806cc20 0x7f32d806f0d0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f32f00fd6b0 tx=0x7f32dc009e30 comp rx=0 tx=0).stop 2024-03-30T05:00:39.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.710+0000 7f32f8400700 1 -- 172.21.15.123:0/562846052 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32f00fdeb0 msgr2=0x7f32f00fbd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:39.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.710+0000 7f32f8400700 1 --2- 172.21.15.123:0/562846052 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32f00fdeb0 0x7f32f00fbd70 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f32e400eb40 tx=0x7f32e400b040 comp rx=0 tx=0).stop 2024-03-30T05:00:39.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.710+0000 7f32f8400700 1 -- 172.21.15.123:0/562846052 shutdown_connections 2024-03-30T05:00:39.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.710+0000 7f32f8400700 1 --2- 172.21.15.123:0/562846052 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f32d806cc20 0x7f32d806f0d0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:39.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.710+0000 7f32f8400700 1 --2- 172.21.15.123:0/562846052 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32f00fdeb0 0x7f32f00fbd70 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:39.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.710+0000 7f32f8400700 1 --2- 172.21.15.123:0/562846052 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f32f00fe850 0x7f32f00fc2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:39.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.710+0000 7f32f8400700 1 -- 172.21.15.123:0/562846052 >> 172.21.15.123:0/562846052 conn(0x7f32f0072bc0 msgr2=0x7f32f0103780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:00:39.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.711+0000 7f32f8400700 1 -- 172.21.15.123:0/562846052 shutdown_connections 2024-03-30T05:00:39.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.711+0000 7f32f8400700 1 -- 172.21.15.123:0/562846052 wait complete. 2024-03-30T05:00:39.711 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 34 2024-03-30T05:00:39.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.848+0000 7fbdc0200700 1 -- 172.21.15.123:0/3624384527 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdb8070c10 msgr2=0x7fbdb8109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:39.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.848+0000 7fbdc0200700 1 --2- 172.21.15.123:0/3624384527 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdb8070c10 0x7fbdb8109170 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fbda4004970 tx=0x7fbda4009e30 comp rx=0 tx=0).stop 2024-03-30T05:00:39.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.848+0000 7fbdc0200700 1 -- 172.21.15.123:0/3624384527 shutdown_connections 2024-03-30T05:00:39.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.848+0000 7fbdc0200700 1 --2- 172.21.15.123:0/3624384527 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdb8070c10 0x7fbdb8109170 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:39.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.848+0000 7fbdc0200700 1 --2- 172.21.15.123:0/3624384527 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbdb8070300 0x7fbdb80706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:39.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.848+0000 7fbdc0200700 1 -- 172.21.15.123:0/3624384527 >> 172.21.15.123:0/3624384527 conn(0x7fbdb80f89b0 msgr2=0x7fbdb80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:00:39.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.848+0000 7fbdc0200700 1 -- 172.21.15.123:0/3624384527 shutdown_connections 2024-03-30T05:00:39.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.848+0000 7fbdc0200700 1 -- 172.21.15.123:0/3624384527 wait complete. 2024-03-30T05:00:39.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.849+0000 7fbdc0200700 1 Processor -- start 2024-03-30T05:00:39.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.849+0000 7fbdc0200700 1 -- start start 2024-03-30T05:00:39.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.850+0000 7fbdc0200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbdb8070300 0x7fbdb8115a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:00:39.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.850+0000 7fbdc0200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdb8070c10 0x7fbdb8115fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:00:39.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.850+0000 7fbdc0200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb8116610 con 0x7fbdb8070c10 2024-03-30T05:00:39.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.850+0000 7fbdc0200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb810fbf0 con 0x7fbdb8070300 2024-03-30T05:00:39.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.850+0000 7fbdbce00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdb8070c10 0x7fbdb8115fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:00:39.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.850+0000 7fbdbce00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdb8070c10 0x7fbdb8115fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52704/0 (socket says 172.21.15.123:52704) 2024-03-30T05:00:39.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.850+0000 7fbdbce00700 1 -- 172.21.15.123:0/2133075867 learned_addr learned my addr 172.21.15.123:0/2133075867 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:00:39.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.850+0000 7fbdbd800700 1 --2- 172.21.15.123:0/2133075867 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbdb8070300 0x7fbdb8115a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:00:39.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.851+0000 7fbdbce00700 1 -- 172.21.15.123:0/2133075867 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbdb8070300 msgr2=0x7fbdb8115a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:39.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.851+0000 7fbdbce00700 1 --2- 172.21.15.123:0/2133075867 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbdb8070300 0x7fbdb8115a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:39.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.851+0000 7fbdbce00700 1 -- 172.21.15.123:0/2133075867 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbda4005040 con 0x7fbdb8070c10 2024-03-30T05:00:39.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.851+0000 7fbdbce00700 1 --2- 172.21.15.123:0/2133075867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdb8070c10 0x7fbdb8115fc0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fbda4009380 tx=0x7fbda4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:00:39.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.851+0000 7fbdb5600700 1 -- 172.21.15.123:0/2133075867 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbda400e040 con 0x7fbdb8070c10 2024-03-30T05:00:39.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.851+0000 7fbdb5600700 1 -- 172.21.15.123:0/2133075867 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbda4013420 con 0x7fbdb8070c10 2024-03-30T05:00:39.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.851+0000 7fbdc0200700 1 -- 172.21.15.123:0/2133075867 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdb810fe70 con 0x7fbdb8070c10 2024-03-30T05:00:39.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.851+0000 7fbdb5600700 1 -- 172.21.15.123:0/2133075867 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbda401c630 con 0x7fbdb8070c10 2024-03-30T05:00:39.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.851+0000 7fbdc0200700 1 -- 172.21.15.123:0/2133075867 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdb81103b0 con 0x7fbdb8070c10 2024-03-30T05:00:39.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.852+0000 7fbdc0200700 1 -- 172.21.15.123:0/2133075867 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdb8071a40 con 0x7fbdb8070c10 2024-03-30T05:00:39.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.853+0000 7fbdb5600700 1 -- 172.21.15.123:0/2133075867 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fbda4013990 con 0x7fbdb8070c10 2024-03-30T05:00:39.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.854+0000 7fbdb5600700 1 --2- 172.21.15.123:0/2133075867 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbda006cc20 0x7fbda006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:00:39.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.854+0000 7fbdb5600700 1 -- 172.21.15.123:0/2133075867 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fbda408d2c0 con 0x7fbdb8070c10 2024-03-30T05:00:39.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.854+0000 7fbdbd800700 1 --2- 172.21.15.123:0/2133075867 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbda006cc20 0x7fbda006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:00:39.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.855+0000 7fbdbd800700 1 --2- 172.21.15.123:0/2133075867 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbda006cc20 0x7fbda006f0d0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fbdac004f40 tx=0x7fbdac00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:00:39.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:39.860+0000 7fbdb5600700 1 -- 172.21.15.123:0/2133075867 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fbda405ad10 con 0x7fbdb8070c10 2024-03-30T05:00:40.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.082+0000 7fbdc0200700 1 -- 172.21.15.123:0/2133075867 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbdb810c060 con 0x7fbda006cc20 2024-03-30T05:00:40.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.083+0000 7fbdb5600700 1 -- 172.21.15.123:0/2133075867 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fbdac00f040 con 0x7fbda006cc20 2024-03-30T05:00:40.080 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T05:00:40.080 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T05:00:40.081 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T05:00:40.081 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T05:00:40.081 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T05:00:40.081 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T05:00:40.081 INFO:teuthology.orchestra.run.smithi123.stdout: "mds", 2024-03-30T05:00:40.081 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T05:00:40.081 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T05:00:40.081 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T05:00:40.081 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T05:00:40.081 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "16/21 daemons upgraded", 2024-03-30T05:00:40.081 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T05:00:40.081 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T05:00:40.081 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T05:00:40.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.085+0000 7fbdc0200700 1 -- 172.21.15.123:0/2133075867 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbda006cc20 msgr2=0x7fbda006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:40.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.085+0000 7fbdc0200700 1 --2- 172.21.15.123:0/2133075867 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbda006cc20 0x7fbda006f0d0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fbdac004f40 tx=0x7fbdac00b040 comp rx=0 tx=0).stop 2024-03-30T05:00:40.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.085+0000 7fbdc0200700 1 -- 172.21.15.123:0/2133075867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdb8070c10 msgr2=0x7fbdb8115fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:40.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.085+0000 7fbdc0200700 1 --2- 172.21.15.123:0/2133075867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdb8070c10 0x7fbdb8115fc0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fbda4009380 tx=0x7fbda4009e30 comp rx=0 tx=0).stop 2024-03-30T05:00:40.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.086+0000 7fbdc0200700 1 -- 172.21.15.123:0/2133075867 shutdown_connections 2024-03-30T05:00:40.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.086+0000 7fbdc0200700 1 --2- 172.21.15.123:0/2133075867 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbda006cc20 0x7fbda006f0d0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:40.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.086+0000 7fbdc0200700 1 --2- 172.21.15.123:0/2133075867 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbdb8070300 0x7fbdb8115a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:40.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.086+0000 7fbdc0200700 1 --2- 172.21.15.123:0/2133075867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdb8070c10 0x7fbdb8115fc0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:40.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.086+0000 7fbdc0200700 1 -- 172.21.15.123:0/2133075867 >> 172.21.15.123:0/2133075867 conn(0x7fbdb80f89b0 msgr2=0x7fbdb8105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:00:40.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.086+0000 7fbdc0200700 1 -- 172.21.15.123:0/2133075867 shutdown_connections 2024-03-30T05:00:40.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.086+0000 7fbdc0200700 1 -- 172.21.15.123:0/2133075867 wait complete. 2024-03-30T05:00:40.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:39 smithi135 ceph-mon[106011]: from='client.35608 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:00:40.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:39 smithi135 ceph-mon[106011]: from='client.35612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:00:40.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:39 smithi135 ceph-mon[106011]: from='client.35616 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:00:40.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:39 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/4152915878' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:00:40.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:39 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/562846052' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T05:00:40.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:39 smithi123 ceph-mon[139369]: from='client.35608 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:00:40.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:39 smithi123 ceph-mon[139369]: from='client.35612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:00:40.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:39 smithi123 ceph-mon[139369]: from='client.35616 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:00:40.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:39 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/4152915878' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:00:40.217 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:39 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/562846052' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T05:00:40.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.219+0000 7f7aece00700 1 -- 172.21.15.123:0/2607204674 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ae81037a0 msgr2=0x7f7ae80fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:40.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.219+0000 7f7aece00700 1 --2- 172.21.15.123:0/2607204674 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ae81037a0 0x7f7ae80fce10 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f7ad4004940 tx=0x7f7ad4009e30 comp rx=0 tx=0).stop 2024-03-30T05:00:40.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.219+0000 7f7aece00700 1 -- 172.21.15.123:0/2607204674 shutdown_connections 2024-03-30T05:00:40.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.219+0000 7f7aece00700 1 --2- 172.21.15.123:0/2607204674 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ae81037a0 0x7f7ae80fce10 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:40.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.219+0000 7f7aece00700 1 --2- 172.21.15.123:0/2607204674 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7ae8102e00 0x7f7ae81031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:40.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.219+0000 7f7aece00700 1 -- 172.21.15.123:0/2607204674 >> 172.21.15.123:0/2607204674 conn(0x7f7ae80f89b0 msgr2=0x7f7ae80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:00:40.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.219+0000 7f7aece00700 1 -- 172.21.15.123:0/2607204674 shutdown_connections 2024-03-30T05:00:40.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.219+0000 7f7aece00700 1 -- 172.21.15.123:0/2607204674 wait complete. 2024-03-30T05:00:40.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.220+0000 7f7aece00700 1 Processor -- start 2024-03-30T05:00:40.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.220+0000 7f7aece00700 1 -- start start 2024-03-30T05:00:40.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.220+0000 7f7aece00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7ae8102e00 0x7f7ae8115d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:00:40.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.220+0000 7f7aece00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ae81037a0 0x7f7ae8116260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:00:40.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.220+0000 7f7aece00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae81168b0 con 0x7f7ae81037a0 2024-03-30T05:00:40.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.220+0000 7f7aece00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae810fc80 con 0x7f7ae8102e00 2024-03-30T05:00:40.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.220+0000 7f7ae5400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ae81037a0 0x7f7ae8116260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:00:40.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.221+0000 7f7ae5400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ae81037a0 0x7f7ae8116260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52730/0 (socket says 172.21.15.123:52730) 2024-03-30T05:00:40.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.221+0000 7f7ae5400700 1 -- 172.21.15.123:0/2512799290 learned_addr learned my addr 172.21.15.123:0/2512799290 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:00:40.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.221+0000 7f7ae5e00700 1 --2- 172.21.15.123:0/2512799290 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7ae8102e00 0x7f7ae8115d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:00:40.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.221+0000 7f7ae5400700 1 -- 172.21.15.123:0/2512799290 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7ae8102e00 msgr2=0x7f7ae8115d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:40.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.221+0000 7f7ae5400700 1 --2- 172.21.15.123:0/2512799290 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7ae8102e00 0x7f7ae8115d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:40.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.221+0000 7f7ae5400700 1 -- 172.21.15.123:0/2512799290 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ad4005040 con 0x7f7ae81037a0 2024-03-30T05:00:40.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.221+0000 7f7ae5e00700 1 --2- 172.21.15.123:0/2512799290 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7ae8102e00 0x7f7ae8115d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T05:00:40.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.221+0000 7f7ae5400700 1 --2- 172.21.15.123:0/2512799290 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ae81037a0 0x7f7ae8116260 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f7ad4009390 tx=0x7f7ad4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:00:40.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.221+0000 7f7ade000700 1 -- 172.21.15.123:0/2512799290 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ad400e040 con 0x7f7ae81037a0 2024-03-30T05:00:40.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.221+0000 7f7ade000700 1 -- 172.21.15.123:0/2512799290 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7ad4013420 con 0x7f7ae81037a0 2024-03-30T05:00:40.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.221+0000 7f7ade000700 1 -- 172.21.15.123:0/2512799290 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ad401c630 con 0x7f7ae81037a0 2024-03-30T05:00:40.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.221+0000 7f7aece00700 1 -- 172.21.15.123:0/2512799290 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ae810ff00 con 0x7f7ae81037a0 2024-03-30T05:00:40.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.222+0000 7f7aece00700 1 -- 172.21.15.123:0/2512799290 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ae8110470 con 0x7f7ae81037a0 2024-03-30T05:00:40.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.223+0000 7f7aece00700 1 -- 172.21.15.123:0/2512799290 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ae80fe550 con 0x7f7ae81037a0 2024-03-30T05:00:40.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.223+0000 7f7ade000700 1 -- 172.21.15.123:0/2512799290 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f7ad4013a00 con 0x7f7ae81037a0 2024-03-30T05:00:40.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.224+0000 7f7ade000700 1 --2- 172.21.15.123:0/2512799290 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7acc06cc30 0x7f7acc06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:00:40.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.224+0000 7f7ade000700 1 -- 172.21.15.123:0/2512799290 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7ad408d390 con 0x7f7ae81037a0 2024-03-30T05:00:40.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.225+0000 7f7ae5e00700 1 --2- 172.21.15.123:0/2512799290 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7acc06cc30 0x7f7acc06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:00:40.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.225+0000 7f7ae5e00700 1 --2- 172.21.15.123:0/2512799290 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7acc06cc30 0x7f7acc06f0e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f7ad8009fd0 tx=0x7f7ad800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:00:40.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.229+0000 7f7ade000700 1 -- 172.21.15.123:0/2512799290 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7ad405ade0 con 0x7f7ae81037a0 2024-03-30T05:00:40.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.508+0000 7f7aece00700 1 -- 172.21.15.123:0/2512799290 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f7ae804bed0 con 0x7f7ae81037a0 2024-03-30T05:00:40.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.509+0000 7f7ade000700 1 -- 172.21.15.123:0/2512799290 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f7ad405a970 con 0x7f7ae81037a0 2024-03-30T05:00:40.506 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T05:00:40.506 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-30T05:00:40.506 INFO:teuthology.orchestra.run.smithi123.stdout: host smithi135 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi135/config 2024-03-30T05:00:40.507 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-30T05:00:40.507 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T05:00:40.507 INFO:teuthology.orchestra.run.smithi123.stdout:Traceback (most recent call last): 2024-03-30T05:00:40.507 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-30T05:00:40.507 INFO:teuthology.orchestra.run.smithi123.stdout: "__main__", mod_spec) 2024-03-30T05:00:40.507 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-30T05:00:40.507 INFO:teuthology.orchestra.run.smithi123.stdout: exec(code, run_globals) 2024-03-30T05:00:40.507 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10889, in 2024-03-30T05:00:40.507 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10877, in main 2024-03-30T05:00:40.507 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-30T05:00:40.507 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-30T05:00:40.507 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-30T05:00:40.507 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-30T05:00:40.507 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-03-30T05:00:40.508 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-30T05:00:40.508 INFO:teuthology.orchestra.run.smithi123.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T05:00:40.508 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T05:00:40.508 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T05:00:40.508 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T05:00:40.508 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T05:00:40.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.511+0000 7f7aece00700 1 -- 172.21.15.123:0/2512799290 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7acc06cc30 msgr2=0x7f7acc06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:40.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.511+0000 7f7aece00700 1 --2- 172.21.15.123:0/2512799290 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7acc06cc30 0x7f7acc06f0e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f7ad8009fd0 tx=0x7f7ad800b040 comp rx=0 tx=0).stop 2024-03-30T05:00:40.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.511+0000 7f7aece00700 1 -- 172.21.15.123:0/2512799290 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ae81037a0 msgr2=0x7f7ae8116260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:00:40.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.511+0000 7f7aece00700 1 --2- 172.21.15.123:0/2512799290 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ae81037a0 0x7f7ae8116260 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f7ad4009390 tx=0x7f7ad4009e30 comp rx=0 tx=0).stop 2024-03-30T05:00:40.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.511+0000 7f7aece00700 1 -- 172.21.15.123:0/2512799290 shutdown_connections 2024-03-30T05:00:40.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.511+0000 7f7aece00700 1 --2- 172.21.15.123:0/2512799290 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7acc06cc30 0x7f7acc06f0e0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:40.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.511+0000 7f7aece00700 1 --2- 172.21.15.123:0/2512799290 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7ae8102e00 0x7f7ae8115d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:40.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.511+0000 7f7aece00700 1 --2- 172.21.15.123:0/2512799290 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ae81037a0 0x7f7ae8116260 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:00:40.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.511+0000 7f7aece00700 1 -- 172.21.15.123:0/2512799290 >> 172.21.15.123:0/2512799290 conn(0x7f7ae80f89b0 msgr2=0x7f7ae80f9a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:00:40.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.511+0000 7f7aece00700 1 -- 172.21.15.123:0/2512799290 shutdown_connections 2024-03-30T05:00:40.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:00:40.511+0000 7f7aece00700 1 -- 172.21.15.123:0/2512799290 wait complete. 2024-03-30T05:00:41.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:40 smithi135 ceph-mon[106011]: pgmap v885: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:41.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:40 smithi135 ceph-mon[106011]: from='client.35628 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:00:41.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:40 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2512799290' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T05:00:41.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:40 smithi123 ceph-mon[139369]: pgmap v885: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:41.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:40 smithi123 ceph-mon[139369]: from='client.35628 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:00:41.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:40 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2512799290' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T05:00:43.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:42 smithi135 ceph-mon[106011]: pgmap v886: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:43.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:42 smithi123 ceph-mon[139369]: pgmap v886: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:45.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:44 smithi135 ceph-mon[106011]: pgmap v887: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:45.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:44 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:00:45.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:44 smithi123 ceph-mon[139369]: pgmap v887: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:45.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:00:47.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:46 smithi135 ceph-mon[106011]: pgmap v888: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:47.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:46 smithi123 ceph-mon[139369]: pgmap v888: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:49.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:48 smithi135 ceph-mon[106011]: pgmap v889: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:49.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:48 smithi123 ceph-mon[139369]: pgmap v889: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:51.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:50 smithi135 ceph-mon[106011]: pgmap v890: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:51.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:50 smithi123 ceph-mon[139369]: pgmap v890: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:53.103 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:52 smithi135 ceph-mon[106011]: pgmap v891: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:53.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:52 smithi123 ceph-mon[139369]: pgmap v891: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:55.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:54 smithi123 ceph-mon[139369]: pgmap v892: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:55.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:54 smithi135 ceph-mon[106011]: pgmap v892: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:57.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:56 smithi123 ceph-mon[139369]: pgmap v893: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:57.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:56 smithi135 ceph-mon[106011]: pgmap v893: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:59.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:58 smithi123 ceph-mon[139369]: pgmap v894: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:00:59.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:58 smithi135 ceph-mon[106011]: pgmap v894: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:00.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:00:59 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:01:00.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:00:59 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:01:01.212 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:01:00 smithi135 ceph-mon[106011]: pgmap v895: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:01.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:01:00 smithi123 ceph-mon[139369]: pgmap v895: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:03.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:01:02 smithi123 ceph-mon[139369]: pgmap v896: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:03.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:01:02 smithi135 ceph-mon[106011]: pgmap v896: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:05.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:01:04 smithi123 ceph-mon[139369]: pgmap v897: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:05.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:01:04 smithi135 ceph-mon[106011]: pgmap v897: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:07.219 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:01:06 smithi123 ceph-mon[139369]: pgmap v898: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:07.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:01:06 smithi135 ceph-mon[106011]: pgmap v898: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:09.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:01:08 smithi123 ceph-mon[139369]: pgmap v899: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:09.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:01:08 smithi135 ceph-mon[106011]: pgmap v899: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:10.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:10.658+0000 7fdb2a600700 1 -- 172.21.15.123:0/2615628113 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb24102e00 msgr2=0x7fdb241031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:10.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:10.658+0000 7fdb2a600700 1 --2- 172.21.15.123:0/2615628113 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb24102e00 0x7fdb241031d0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fdb14004940 tx=0x7fdb14009e30 comp rx=0 tx=0).stop 2024-03-30T05:01:10.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:10.658+0000 7fdb2a600700 1 -- 172.21.15.123:0/2615628113 shutdown_connections 2024-03-30T05:01:10.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:10.658+0000 7fdb2a600700 1 --2- 172.21.15.123:0/2615628113 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdb241037a0 0x7fdb240fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:10.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:10.658+0000 7fdb2a600700 1 --2- 172.21.15.123:0/2615628113 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb24102e00 0x7fdb241031d0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:10.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:10.658+0000 7fdb2a600700 1 -- 172.21.15.123:0/2615628113 >> 172.21.15.123:0/2615628113 conn(0x7fdb240f89b0 msgr2=0x7fdb240fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:01:10.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:10.658+0000 7fdb2a600700 1 -- 172.21.15.123:0/2615628113 shutdown_connections 2024-03-30T05:01:10.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:10.658+0000 7fdb2a600700 1 -- 172.21.15.123:0/2615628113 wait complete. 2024-03-30T05:01:10.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:10.659+0000 7fdb2a600700 1 Processor -- start 2024-03-30T05:01:10.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:10.659+0000 7fdb2a600700 1 -- start start 2024-03-30T05:01:10.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:10.660+0000 7fdb2a600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdb24102e00 0x7fdb240702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:01:10.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:10.660+0000 7fdb2a600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb241037a0 0x7fdb24070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:01:10.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:10.660+0000 7fdb2a600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb24070d90 con 0x7fdb241037a0 2024-03-30T05:01:10.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:10.660+0000 7fdb2a600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb24070ed0 con 0x7fdb24102e00 2024-03-30T05:01:10.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:10.660+0000 7fdb22a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb241037a0 0x7fdb24070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:01:10.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:10.660+0000 7fdb22a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb241037a0 0x7fdb24070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58294/0 (socket says 172.21.15.123:58294) 2024-03-30T05:01:10.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:10.660+0000 7fdb22a00700 1 -- 172.21.15.123:0/2144236464 learned_addr learned my addr 172.21.15.123:0/2144236464 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:01:10.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:10.660+0000 7fdb23400700 1 --2- 172.21.15.123:0/2144236464 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdb24102e00 0x7fdb240702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:01:10.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:10.660+0000 7fdb22a00700 1 -- 172.21.15.123:0/2144236464 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdb24102e00 msgr2=0x7fdb240702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:10.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:10.660+0000 7fdb22a00700 1 --2- 172.21.15.123:0/2144236464 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdb24102e00 0x7fdb240702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:10.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:10.660+0000 7fdb22a00700 1 -- 172.21.15.123:0/2144236464 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb14005040 con 0x7fdb241037a0 2024-03-30T05:01:10.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:10.661+0000 7fdb22a00700 1 --2- 172.21.15.123:0/2144236464 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb241037a0 0x7fdb24070800 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fdb10002a50 tx=0x7fdb10004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:01:10.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:10.661+0000 7fdb1b400700 1 -- 172.21.15.123:0/2144236464 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb10007cc0 con 0x7fdb241037a0 2024-03-30T05:01:10.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:10.661+0000 7fdb1b400700 1 -- 172.21.15.123:0/2144236464 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdb1000f040 con 0x7fdb241037a0 2024-03-30T05:01:10.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:10.661+0000 7fdb2a600700 1 -- 172.21.15.123:0/2144236464 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb24119e20 con 0x7fdb241037a0 2024-03-30T05:01:10.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:10.661+0000 7fdb1b400700 1 -- 172.21.15.123:0/2144236464 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb1000d920 con 0x7fdb241037a0 2024-03-30T05:01:10.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:10.661+0000 7fdb2a600700 1 -- 172.21.15.123:0/2144236464 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb2411a2a0 con 0x7fdb241037a0 2024-03-30T05:01:10.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:10.662+0000 7fdb2a600700 1 -- 172.21.15.123:0/2144236464 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb240fe550 con 0x7fdb241037a0 2024-03-30T05:01:10.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:10.663+0000 7fdb1b400700 1 -- 172.21.15.123:0/2144236464 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fdb10009680 con 0x7fdb241037a0 2024-03-30T05:01:10.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:10.664+0000 7fdb1b400700 1 --2- 172.21.15.123:0/2144236464 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fdb0806cc30 0x7fdb0806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:01:10.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:10.664+0000 7fdb1b400700 1 -- 172.21.15.123:0/2144236464 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fdb1008bcf0 con 0x7fdb241037a0 2024-03-30T05:01:10.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:10.664+0000 7fdb23400700 1 --2- 172.21.15.123:0/2144236464 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fdb0806cc30 0x7fdb0806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:01:10.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:10.665+0000 7fdb23400700 1 --2- 172.21.15.123:0/2144236464 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fdb0806cc30 0x7fdb0806f0e0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fdb14009fd0 tx=0x7fdb14009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:01:10.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:10.669+0000 7fdb1b400700 1 -- 172.21.15.123:0/2144236464 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fdb10055cb0 con 0x7fdb241037a0 2024-03-30T05:01:10.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:10.891+0000 7fdb2a600700 1 -- 172.21.15.123:0/2144236464 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdb2404bed0 con 0x7fdb0806cc30 2024-03-30T05:01:10.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:10.892+0000 7fdb1b400700 1 -- 172.21.15.123:0/2144236464 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fdb2404bed0 con 0x7fdb0806cc30 2024-03-30T05:01:10.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:10.894+0000 7fdb2a600700 1 -- 172.21.15.123:0/2144236464 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fdb0806cc30 msgr2=0x7fdb0806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:10.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:10.894+0000 7fdb2a600700 1 --2- 172.21.15.123:0/2144236464 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fdb0806cc30 0x7fdb0806f0e0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fdb14009fd0 tx=0x7fdb14009e30 comp rx=0 tx=0).stop 2024-03-30T05:01:10.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:10.895+0000 7fdb2a600700 1 -- 172.21.15.123:0/2144236464 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb241037a0 msgr2=0x7fdb24070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:10.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:10.895+0000 7fdb2a600700 1 --2- 172.21.15.123:0/2144236464 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb241037a0 0x7fdb24070800 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fdb10002a50 tx=0x7fdb10004600 comp rx=0 tx=0).stop 2024-03-30T05:01:10.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:10.895+0000 7fdb2a600700 1 -- 172.21.15.123:0/2144236464 shutdown_connections 2024-03-30T05:01:10.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:10.895+0000 7fdb2a600700 1 --2- 172.21.15.123:0/2144236464 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fdb0806cc30 0x7fdb0806f0e0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:10.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:10.895+0000 7fdb2a600700 1 --2- 172.21.15.123:0/2144236464 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdb24102e00 0x7fdb240702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:10.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:10.895+0000 7fdb2a600700 1 --2- 172.21.15.123:0/2144236464 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb241037a0 0x7fdb24070800 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:10.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:10.895+0000 7fdb2a600700 1 -- 172.21.15.123:0/2144236464 >> 172.21.15.123:0/2144236464 conn(0x7fdb240f89b0 msgr2=0x7fdb240f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:01:10.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:10.895+0000 7fdb2a600700 1 -- 172.21.15.123:0/2144236464 shutdown_connections 2024-03-30T05:01:10.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:10.895+0000 7fdb2a600700 1 -- 172.21.15.123:0/2144236464 wait complete. 2024-03-30T05:01:10.908 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T05:01:11.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.030+0000 7f9f2f000700 1 -- 172.21.15.123:0/3080071433 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f281037c0 msgr2=0x7f9f280fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:11.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.030+0000 7f9f2f000700 1 --2- 172.21.15.123:0/3080071433 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f281037c0 0x7f9f280fce30 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f9f14004940 tx=0x7f9f14009e30 comp rx=0 tx=0).stop 2024-03-30T05:01:11.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.030+0000 7f9f2f000700 1 -- 172.21.15.123:0/3080071433 shutdown_connections 2024-03-30T05:01:11.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.030+0000 7f9f2f000700 1 --2- 172.21.15.123:0/3080071433 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f281037c0 0x7f9f280fce30 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:11.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.030+0000 7f9f2f000700 1 --2- 172.21.15.123:0/3080071433 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9f28102e20 0x7f9f281031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:11.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.030+0000 7f9f2f000700 1 -- 172.21.15.123:0/3080071433 >> 172.21.15.123:0/3080071433 conn(0x7f9f280f89d0 msgr2=0x7f9f280fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:01:11.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.030+0000 7f9f2f000700 1 -- 172.21.15.123:0/3080071433 shutdown_connections 2024-03-30T05:01:11.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.030+0000 7f9f2f000700 1 -- 172.21.15.123:0/3080071433 wait complete. 2024-03-30T05:01:11.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.031+0000 7f9f2f000700 1 Processor -- start 2024-03-30T05:01:11.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.032+0000 7f9f2f000700 1 -- start start 2024-03-30T05:01:11.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.032+0000 7f9f2f000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f28102e20 0x7f9f280702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:01:11.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.032+0000 7f9f2f000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9f281037c0 0x7f9f28070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:01:11.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.032+0000 7f9f2f000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f28070d90 con 0x7f9f28102e20 2024-03-30T05:01:11.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.032+0000 7f9f2f000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f28070ed0 con 0x7f9f281037c0 2024-03-30T05:01:11.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.032+0000 7f9f27e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f28102e20 0x7f9f280702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:01:11.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.032+0000 7f9f27e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f28102e20 0x7f9f280702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58306/0 (socket says 172.21.15.123:58306) 2024-03-30T05:01:11.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.032+0000 7f9f27e00700 1 -- 172.21.15.123:0/962972118 learned_addr learned my addr 172.21.15.123:0/962972118 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:01:11.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.033+0000 7f9f27e00700 1 -- 172.21.15.123:0/962972118 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9f281037c0 msgr2=0x7f9f28070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:11.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.033+0000 7f9f27400700 1 --2- 172.21.15.123:0/962972118 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9f281037c0 0x7f9f28070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:01:11.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.033+0000 7f9f27e00700 1 --2- 172.21.15.123:0/962972118 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9f281037c0 0x7f9f28070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:11.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.033+0000 7f9f27e00700 1 -- 172.21.15.123:0/962972118 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f14005040 con 0x7f9f28102e20 2024-03-30T05:01:11.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.033+0000 7f9f27400700 1 --2- 172.21.15.123:0/962972118 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9f281037c0 0x7f9f28070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T05:01:11.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.033+0000 7f9f27e00700 1 --2- 172.21.15.123:0/962972118 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f28102e20 0x7f9f280702c0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f9f18002a50 tx=0x7f9f180046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:01:11.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.033+0000 7f9f1fe00700 1 -- 172.21.15.123:0/962972118 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f18007cc0 con 0x7f9f28102e20 2024-03-30T05:01:11.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.033+0000 7f9f1fe00700 1 -- 172.21.15.123:0/962972118 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9f1800f040 con 0x7f9f28102e20 2024-03-30T05:01:11.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.033+0000 7f9f2f000700 1 -- 172.21.15.123:0/962972118 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f28119e20 con 0x7f9f28102e20 2024-03-30T05:01:11.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.033+0000 7f9f1fe00700 1 -- 172.21.15.123:0/962972118 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f1800da70 con 0x7f9f28102e20 2024-03-30T05:01:11.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.034+0000 7f9f2f000700 1 -- 172.21.15.123:0/962972118 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f2811a2a0 con 0x7f9f28102e20 2024-03-30T05:01:11.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.035+0000 7f9f2f000700 1 -- 172.21.15.123:0/962972118 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f280fe570 con 0x7f9f28102e20 2024-03-30T05:01:11.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.035+0000 7f9f1fe00700 1 -- 172.21.15.123:0/962972118 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f9f1800dc10 con 0x7f9f28102e20 2024-03-30T05:01:11.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.036+0000 7f9f1fe00700 1 --2- 172.21.15.123:0/962972118 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9f0c06cc20 0x7f9f0c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:01:11.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.036+0000 7f9f1fe00700 1 -- 172.21.15.123:0/962972118 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f9f1808b760 con 0x7f9f28102e20 2024-03-30T05:01:11.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.037+0000 7f9f27400700 1 --2- 172.21.15.123:0/962972118 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9f0c06cc20 0x7f9f0c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:01:11.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.037+0000 7f9f27400700 1 --2- 172.21.15.123:0/962972118 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9f0c06cc20 0x7f9f0c06f0d0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f9f14009fd0 tx=0x7f9f14009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:01:11.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.041+0000 7f9f1fe00700 1 -- 172.21.15.123:0/962972118 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9f18055720 con 0x7f9f28102e20 2024-03-30T05:01:11.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:01:10 smithi123 ceph-mon[139369]: pgmap v900: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:11.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.262+0000 7f9f2f000700 1 -- 172.21.15.123:0/962972118 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9f280719b0 con 0x7f9f0c06cc20 2024-03-30T05:01:11.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.263+0000 7f9f1fe00700 1 -- 172.21.15.123:0/962972118 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f9f280719b0 con 0x7f9f0c06cc20 2024-03-30T05:01:11.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.265+0000 7f9f2f000700 1 -- 172.21.15.123:0/962972118 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9f0c06cc20 msgr2=0x7f9f0c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:11.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.265+0000 7f9f2f000700 1 --2- 172.21.15.123:0/962972118 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9f0c06cc20 0x7f9f0c06f0d0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f9f14009fd0 tx=0x7f9f14009e30 comp rx=0 tx=0).stop 2024-03-30T05:01:11.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.265+0000 7f9f2f000700 1 -- 172.21.15.123:0/962972118 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f28102e20 msgr2=0x7f9f280702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:11.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.265+0000 7f9f2f000700 1 --2- 172.21.15.123:0/962972118 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f28102e20 0x7f9f280702c0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f9f18002a50 tx=0x7f9f180046b0 comp rx=0 tx=0).stop 2024-03-30T05:01:11.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.266+0000 7f9f2f000700 1 -- 172.21.15.123:0/962972118 shutdown_connections 2024-03-30T05:01:11.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.266+0000 7f9f2f000700 1 --2- 172.21.15.123:0/962972118 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9f0c06cc20 0x7f9f0c06f0d0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:11.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.266+0000 7f9f2f000700 1 --2- 172.21.15.123:0/962972118 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f28102e20 0x7f9f280702c0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:11.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.266+0000 7f9f2f000700 1 --2- 172.21.15.123:0/962972118 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9f281037c0 0x7f9f28070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:11.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.266+0000 7f9f2f000700 1 -- 172.21.15.123:0/962972118 >> 172.21.15.123:0/962972118 conn(0x7f9f280f89d0 msgr2=0x7f9f280f9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:01:11.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.266+0000 7f9f2f000700 1 -- 172.21.15.123:0/962972118 shutdown_connections 2024-03-30T05:01:11.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.266+0000 7f9f2f000700 1 -- 172.21.15.123:0/962972118 wait complete. 2024-03-30T05:01:11.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:01:10 smithi135 ceph-mon[106011]: pgmap v900: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:11.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.400+0000 7fd4b7200700 1 -- 172.21.15.123:0/3079743070 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd4b8102c40 msgr2=0x7fd4b8103010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:11.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.400+0000 7fd4b7200700 1 --2- 172.21.15.123:0/3079743070 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd4b8102c40 0x7fd4b8103010 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fd4a8004940 tx=0x7fd4a8009e30 comp rx=0 tx=0).stop 2024-03-30T05:01:11.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.400+0000 7fd4b7200700 1 -- 172.21.15.123:0/3079743070 shutdown_connections 2024-03-30T05:01:11.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.400+0000 7fd4b7200700 1 --2- 172.21.15.123:0/3079743070 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd4b81035e0 0x7fd4b80fcc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:11.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.400+0000 7fd4b7200700 1 --2- 172.21.15.123:0/3079743070 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd4b8102c40 0x7fd4b8103010 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:11.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.400+0000 7fd4b7200700 1 -- 172.21.15.123:0/3079743070 >> 172.21.15.123:0/3079743070 conn(0x7fd4b80f8830 msgr2=0x7fd4b80fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:01:11.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.400+0000 7fd4b7200700 1 -- 172.21.15.123:0/3079743070 shutdown_connections 2024-03-30T05:01:11.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.400+0000 7fd4b7200700 1 -- 172.21.15.123:0/3079743070 wait complete. 2024-03-30T05:01:11.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.401+0000 7fd4b7200700 1 Processor -- start 2024-03-30T05:01:11.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.402+0000 7fd4b7200700 1 -- start start 2024-03-30T05:01:11.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.402+0000 7fd4b7200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd4b8102c40 0x7fd4b8113950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:01:11.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.402+0000 7fd4b7200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd4b81035e0 0x7fd4b8113e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:01:11.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.402+0000 7fd4b7200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b81144e0 con 0x7fd4b8102c40 2024-03-30T05:01:11.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.402+0000 7fd4b7200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b810d8d0 con 0x7fd4b81035e0 2024-03-30T05:01:11.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.403+0000 7fd4b5e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd4b8102c40 0x7fd4b8113950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:01:11.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.403+0000 7fd4b5e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd4b8102c40 0x7fd4b8113950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58332/0 (socket says 172.21.15.123:58332) 2024-03-30T05:01:11.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.403+0000 7fd4b5e00700 1 -- 172.21.15.123:0/2424976220 learned_addr learned my addr 172.21.15.123:0/2424976220 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:01:11.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.403+0000 7fd4b5400700 1 --2- 172.21.15.123:0/2424976220 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd4b81035e0 0x7fd4b8113e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:01:11.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.403+0000 7fd4b5e00700 1 -- 172.21.15.123:0/2424976220 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd4b81035e0 msgr2=0x7fd4b8113e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:11.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.403+0000 7fd4b5e00700 1 --2- 172.21.15.123:0/2424976220 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd4b81035e0 0x7fd4b8113e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:11.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.403+0000 7fd4b5e00700 1 -- 172.21.15.123:0/2424976220 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4a8005040 con 0x7fd4b8102c40 2024-03-30T05:01:11.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.403+0000 7fd4b5400700 1 --2- 172.21.15.123:0/2424976220 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd4b81035e0 0x7fd4b8113e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T05:01:11.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.403+0000 7fd4b5e00700 1 --2- 172.21.15.123:0/2424976220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd4b8102c40 0x7fd4b8113950 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fd4a8009390 tx=0x7fd4a8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:01:11.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.403+0000 7fd4ae000700 1 -- 172.21.15.123:0/2424976220 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd4a800e040 con 0x7fd4b8102c40 2024-03-30T05:01:11.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.403+0000 7fd4ae000700 1 -- 172.21.15.123:0/2424976220 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd4a8013420 con 0x7fd4b8102c40 2024-03-30T05:01:11.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.404+0000 7fd4b7200700 1 -- 172.21.15.123:0/2424976220 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4b810db50 con 0x7fd4b8102c40 2024-03-30T05:01:11.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.404+0000 7fd4ae000700 1 -- 172.21.15.123:0/2424976220 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd4a801c630 con 0x7fd4b8102c40 2024-03-30T05:01:11.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.404+0000 7fd4b7200700 1 -- 172.21.15.123:0/2424976220 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4b810e110 con 0x7fd4b8102c40 2024-03-30T05:01:11.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.405+0000 7fd4b7200700 1 -- 172.21.15.123:0/2424976220 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4b80fe390 con 0x7fd4b8102c40 2024-03-30T05:01:11.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.406+0000 7fd4ae000700 1 -- 172.21.15.123:0/2424976220 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fd4a80139e0 con 0x7fd4b8102c40 2024-03-30T05:01:11.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.406+0000 7fd4ae000700 1 --2- 172.21.15.123:0/2424976220 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd49c06cc30 0x7fd49c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:01:11.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.406+0000 7fd4ae000700 1 -- 172.21.15.123:0/2424976220 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd4a808d2e0 con 0x7fd4b8102c40 2024-03-30T05:01:11.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.407+0000 7fd4b5400700 1 --2- 172.21.15.123:0/2424976220 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd49c06cc30 0x7fd49c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:01:11.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.407+0000 7fd4b5400700 1 --2- 172.21.15.123:0/2424976220 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd49c06cc30 0x7fd49c06f0e0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fd4a4009fd0 tx=0x7fd4a400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:01:11.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.411+0000 7fd4ae000700 1 -- 172.21.15.123:0/2424976220 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd4a805ae60 con 0x7fd4b8102c40 2024-03-30T05:01:11.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.621+0000 7fd4b7200700 1 -- 172.21.15.123:0/2424976220 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd4b8109bf0 con 0x7fd49c06cc30 2024-03-30T05:01:11.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.630+0000 7fd4ae000700 1 -- 172.21.15.123:0/2424976220 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fd4b8109bf0 con 0x7fd49c06cc30 2024-03-30T05:01:11.629 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T05:01:11.629 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (30m) 7m ago 42m 14.6M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T05:01:11.629 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (27m) 7m ago 43m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T05:01:11.629 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (27m) 7m ago 41m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T05:01:11.629 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (30m) 7m ago 42m 73.5M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T05:01:11.629 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (23m) 7m ago 36m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T05:01:11.629 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (23m) 7m ago 36m 16.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T05:01:11.629 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (7m) 7m ago 36m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 87b073a6c9ca 2024-03-30T05:01:11.629 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (23m) 7m ago 36m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T05:01:11.629 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (32m) 7m ago 44m 510M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T05:01:11.629 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (31m) 7m ago 41m 456M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T05:01:11.629 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (28m) 7m ago 44m 70.0M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T05:01:11.630 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (28m) 7m ago 41m 51.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T05:01:11.630 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (31m) 7m ago 43m 8636k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T05:01:11.630 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (31m) 7m ago 41m 8678k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T05:01:11.630 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (27m) 7m ago 40m 107M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T05:01:11.630 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (26m) 7m ago 40m 97.2M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T05:01:11.630 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (25m) 7m ago 39m 89.8M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T05:01:11.630 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (25m) 7m ago 39m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T05:01:11.630 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (24m) 7m ago 38m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T05:01:11.630 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (24m) 7m ago 38m 88.8M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T05:01:11.630 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (30m) 7m ago 42m 63.9M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T05:01:11.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.633+0000 7fd4b7200700 1 -- 172.21.15.123:0/2424976220 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd49c06cc30 msgr2=0x7fd49c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:11.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.633+0000 7fd4b7200700 1 --2- 172.21.15.123:0/2424976220 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd49c06cc30 0x7fd49c06f0e0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fd4a4009fd0 tx=0x7fd4a400b040 comp rx=0 tx=0).stop 2024-03-30T05:01:11.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.633+0000 7fd4b7200700 1 -- 172.21.15.123:0/2424976220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd4b8102c40 msgr2=0x7fd4b8113950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:11.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.633+0000 7fd4b7200700 1 --2- 172.21.15.123:0/2424976220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd4b8102c40 0x7fd4b8113950 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fd4a8009390 tx=0x7fd4a8009e30 comp rx=0 tx=0).stop 2024-03-30T05:01:11.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.633+0000 7fd4b7200700 1 -- 172.21.15.123:0/2424976220 shutdown_connections 2024-03-30T05:01:11.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.633+0000 7fd4b7200700 1 --2- 172.21.15.123:0/2424976220 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd49c06cc30 0x7fd49c06f0e0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:11.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.633+0000 7fd4b7200700 1 --2- 172.21.15.123:0/2424976220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd4b8102c40 0x7fd4b8113950 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:11.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.633+0000 7fd4b7200700 1 --2- 172.21.15.123:0/2424976220 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd4b81035e0 0x7fd4b8113e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:11.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.633+0000 7fd4b7200700 1 -- 172.21.15.123:0/2424976220 >> 172.21.15.123:0/2424976220 conn(0x7fd4b80f8830 msgr2=0x7fd4b80f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:01:11.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.633+0000 7fd4b7200700 1 -- 172.21.15.123:0/2424976220 shutdown_connections 2024-03-30T05:01:11.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.633+0000 7fd4b7200700 1 -- 172.21.15.123:0/2424976220 wait complete. 2024-03-30T05:01:11.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.766+0000 7f4200000700 1 -- 172.21.15.123:0/455225694 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41f81037a0 msgr2=0x7f41f80fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:11.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.766+0000 7f4200000700 1 --2- 172.21.15.123:0/455225694 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41f81037a0 0x7f41f80fce10 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f41e4004940 tx=0x7f41e4009e30 comp rx=0 tx=0).stop 2024-03-30T05:01:11.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.766+0000 7f4200000700 1 -- 172.21.15.123:0/455225694 shutdown_connections 2024-03-30T05:01:11.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.766+0000 7f4200000700 1 --2- 172.21.15.123:0/455225694 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41f81037a0 0x7f41f80fce10 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:11.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.766+0000 7f4200000700 1 --2- 172.21.15.123:0/455225694 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f41f8102e00 0x7f41f81031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:11.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.766+0000 7f4200000700 1 -- 172.21.15.123:0/455225694 >> 172.21.15.123:0/455225694 conn(0x7f41f80f89b0 msgr2=0x7f41f80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:01:11.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.766+0000 7f4200000700 1 -- 172.21.15.123:0/455225694 shutdown_connections 2024-03-30T05:01:11.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.766+0000 7f4200000700 1 -- 172.21.15.123:0/455225694 wait complete. 2024-03-30T05:01:11.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.767+0000 7f4200000700 1 Processor -- start 2024-03-30T05:01:11.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.767+0000 7f4200000700 1 -- start start 2024-03-30T05:01:11.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.767+0000 7f4200000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41f8102e00 0x7f41f8115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:01:11.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.767+0000 7f4200000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f41f81037a0 0x7f41f81161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:01:11.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.767+0000 7f4200000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41f81167f0 con 0x7f41f8102e00 2024-03-30T05:01:11.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.767+0000 7f4200000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41f810fbc0 con 0x7f41f81037a0 2024-03-30T05:01:11.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.768+0000 7f41fd600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41f8102e00 0x7f41f8115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:01:11.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.768+0000 7f41fd600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41f8102e00 0x7f41f8115c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58348/0 (socket says 172.21.15.123:58348) 2024-03-30T05:01:11.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.768+0000 7f41fd600700 1 -- 172.21.15.123:0/1974636208 learned_addr learned my addr 172.21.15.123:0/1974636208 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:01:11.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.768+0000 7f41fcc00700 1 --2- 172.21.15.123:0/1974636208 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f41f81037a0 0x7f41f81161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:01:11.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.768+0000 7f41fd600700 1 -- 172.21.15.123:0/1974636208 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f41f81037a0 msgr2=0x7f41f81161a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:11.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.768+0000 7f41fd600700 1 --2- 172.21.15.123:0/1974636208 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f41f81037a0 0x7f41f81161a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:11.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.768+0000 7f41fd600700 1 -- 172.21.15.123:0/1974636208 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41e4005040 con 0x7f41f8102e00 2024-03-30T05:01:11.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.768+0000 7f41fd600700 1 --2- 172.21.15.123:0/1974636208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41f8102e00 0x7f41f8115c60 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f41ec00eb40 tx=0x7f41ec00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:01:11.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.768+0000 7f41f5600700 1 -- 172.21.15.123:0/1974636208 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f41ec007cc0 con 0x7f41f8102e00 2024-03-30T05:01:11.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.769+0000 7f41f5600700 1 -- 172.21.15.123:0/1974636208 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f41ec00f040 con 0x7f41f8102e00 2024-03-30T05:01:11.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.769+0000 7f4200000700 1 -- 172.21.15.123:0/1974636208 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41f810fe40 con 0x7f41f8102e00 2024-03-30T05:01:11.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.769+0000 7f41f5600700 1 -- 172.21.15.123:0/1974636208 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f41ec009570 con 0x7f41f8102e00 2024-03-30T05:01:11.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.769+0000 7f4200000700 1 -- 172.21.15.123:0/1974636208 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41f8110400 con 0x7f41f8102e00 2024-03-30T05:01:11.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.770+0000 7f4200000700 1 -- 172.21.15.123:0/1974636208 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41f80fe550 con 0x7f41f8102e00 2024-03-30T05:01:11.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.771+0000 7f41f5600700 1 -- 172.21.15.123:0/1974636208 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f41ec0096d0 con 0x7f41f8102e00 2024-03-30T05:01:11.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.771+0000 7f41f5600700 1 --2- 172.21.15.123:0/1974636208 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f41e006cc30 0x7f41e006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:01:11.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.772+0000 7f41f5600700 1 -- 172.21.15.123:0/1974636208 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f41ec08bda0 con 0x7f41f8102e00 2024-03-30T05:01:11.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.772+0000 7f41fcc00700 1 --2- 172.21.15.123:0/1974636208 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f41e006cc30 0x7f41e006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:01:11.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.772+0000 7f41fcc00700 1 --2- 172.21.15.123:0/1974636208 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f41e006cc30 0x7f41e006f0e0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f41e4009fd0 tx=0x7f41e4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:01:11.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:11.776+0000 7f41f5600700 1 -- 172.21.15.123:0/1974636208 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f41ec059820 con 0x7f41f8102e00 2024-03-30T05:01:12.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.070+0000 7f4200000700 1 -- 172.21.15.123:0/1974636208 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f41f804bed0 con 0x7f41f8102e00 2024-03-30T05:01:12.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.070+0000 7f41f5600700 1 -- 172.21.15.123:0/1974636208 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f41ec0593b0 con 0x7f41f8102e00 2024-03-30T05:01:12.069 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T05:01:12.069 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T05:01:12.069 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T05:01:12.069 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:01:12.069 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T05:01:12.069 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T05:01:12.069 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:01:12.069 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T05:01:12.069 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T05:01:12.069 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:01:12.069 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T05:01:12.070 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 4 2024-03-30T05:01:12.070 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:01:12.070 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T05:01:12.070 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 14 2024-03-30T05:01:12.070 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T05:01:12.070 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T05:01:12.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.073+0000 7f4200000700 1 -- 172.21.15.123:0/1974636208 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f41e006cc30 msgr2=0x7f41e006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:12.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.073+0000 7f4200000700 1 --2- 172.21.15.123:0/1974636208 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f41e006cc30 0x7f41e006f0e0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f41e4009fd0 tx=0x7f41e4009e30 comp rx=0 tx=0).stop 2024-03-30T05:01:12.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.073+0000 7f4200000700 1 -- 172.21.15.123:0/1974636208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41f8102e00 msgr2=0x7f41f8115c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:12.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.073+0000 7f4200000700 1 --2- 172.21.15.123:0/1974636208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41f8102e00 0x7f41f8115c60 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f41ec00eb40 tx=0x7f41ec00b040 comp rx=0 tx=0).stop 2024-03-30T05:01:12.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.073+0000 7f4200000700 1 -- 172.21.15.123:0/1974636208 shutdown_connections 2024-03-30T05:01:12.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.073+0000 7f4200000700 1 --2- 172.21.15.123:0/1974636208 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f41e006cc30 0x7f41e006f0e0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:12.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.073+0000 7f4200000700 1 --2- 172.21.15.123:0/1974636208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41f8102e00 0x7f41f8115c60 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:12.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.073+0000 7f4200000700 1 --2- 172.21.15.123:0/1974636208 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f41f81037a0 0x7f41f81161a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:12.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.073+0000 7f4200000700 1 -- 172.21.15.123:0/1974636208 >> 172.21.15.123:0/1974636208 conn(0x7f41f80f89b0 msgr2=0x7f41f80f99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:01:12.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.073+0000 7f4200000700 1 -- 172.21.15.123:0/1974636208 shutdown_connections 2024-03-30T05:01:12.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.073+0000 7f4200000700 1 -- 172.21.15.123:0/1974636208 wait complete. 2024-03-30T05:01:12.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.207+0000 7f1652800700 1 -- 172.21.15.123:0/3301893086 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f164c102e00 msgr2=0x7f164c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:12.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.207+0000 7f1652800700 1 --2- 172.21.15.123:0/3301893086 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f164c102e00 0x7f164c1031d0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f163c004940 tx=0x7f163c009e30 comp rx=0 tx=0).stop 2024-03-30T05:01:12.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.207+0000 7f1652800700 1 -- 172.21.15.123:0/3301893086 shutdown_connections 2024-03-30T05:01:12.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.207+0000 7f1652800700 1 --2- 172.21.15.123:0/3301893086 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f164c1037a0 0x7f164c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:12.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.207+0000 7f1652800700 1 --2- 172.21.15.123:0/3301893086 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f164c102e00 0x7f164c1031d0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:12.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.207+0000 7f1652800700 1 -- 172.21.15.123:0/3301893086 >> 172.21.15.123:0/3301893086 conn(0x7f164c0f89b0 msgr2=0x7f164c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:01:12.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.207+0000 7f1652800700 1 -- 172.21.15.123:0/3301893086 shutdown_connections 2024-03-30T05:01:12.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.207+0000 7f1652800700 1 -- 172.21.15.123:0/3301893086 wait complete. 2024-03-30T05:01:12.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.208+0000 7f1652800700 1 Processor -- start 2024-03-30T05:01:12.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.208+0000 7f1652800700 1 -- start start 2024-03-30T05:01:12.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.208+0000 7f1652800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f164c102e00 0x7f164c113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:01:12.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.208+0000 7f1652800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f164c1037a0 0x7f164c114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:01:12.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.208+0000 7f1652800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f164c1146b0 con 0x7f164c1037a0 2024-03-30T05:01:12.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.208+0000 7f1652800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f164c10daa0 con 0x7f164c102e00 2024-03-30T05:01:12.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.208+0000 7f164aa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f164c1037a0 0x7f164c114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:01:12.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.208+0000 7f164aa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f164c1037a0 0x7f164c114060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58364/0 (socket says 172.21.15.123:58364) 2024-03-30T05:01:12.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.208+0000 7f164aa00700 1 -- 172.21.15.123:0/1870461242 learned_addr learned my addr 172.21.15.123:0/1870461242 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:01:12.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.208+0000 7f164b400700 1 --2- 172.21.15.123:0/1870461242 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f164c102e00 0x7f164c113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:01:12.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.209+0000 7f164aa00700 1 -- 172.21.15.123:0/1870461242 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f164c102e00 msgr2=0x7f164c113b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:12.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.209+0000 7f164aa00700 1 --2- 172.21.15.123:0/1870461242 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f164c102e00 0x7f164c113b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:12.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.209+0000 7f164aa00700 1 -- 172.21.15.123:0/1870461242 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f163c005040 con 0x7f164c1037a0 2024-03-30T05:01:12.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.209+0000 7f164aa00700 1 --2- 172.21.15.123:0/1870461242 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f164c1037a0 0x7f164c114060 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f1638002a50 tx=0x7f1638004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:01:12.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.209+0000 7f1643400700 1 -- 172.21.15.123:0/1870461242 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1638007cc0 con 0x7f164c1037a0 2024-03-30T05:01:12.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.209+0000 7f1643400700 1 -- 172.21.15.123:0/1870461242 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f163800f040 con 0x7f164c1037a0 2024-03-30T05:01:12.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.209+0000 7f1652800700 1 -- 172.21.15.123:0/1870461242 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f164c10dd80 con 0x7f164c1037a0 2024-03-30T05:01:12.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.209+0000 7f1643400700 1 -- 172.21.15.123:0/1870461242 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f163800d920 con 0x7f164c1037a0 2024-03-30T05:01:12.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.209+0000 7f1652800700 1 -- 172.21.15.123:0/1870461242 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f164c10e340 con 0x7f164c1037a0 2024-03-30T05:01:12.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.210+0000 7f1652800700 1 -- 172.21.15.123:0/1870461242 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f164c0fe550 con 0x7f164c1037a0 2024-03-30T05:01:12.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.211+0000 7f1643400700 1 -- 172.21.15.123:0/1870461242 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f1638009680 con 0x7f164c1037a0 2024-03-30T05:01:12.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.212+0000 7f1643400700 1 --2- 172.21.15.123:0/1870461242 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f163006cc30 0x7f163006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:01:12.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.212+0000 7f1643400700 1 -- 172.21.15.123:0/1870461242 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f163808bd10 con 0x7f164c1037a0 2024-03-30T05:01:12.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.212+0000 7f164b400700 1 --2- 172.21.15.123:0/1870461242 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f163006cc30 0x7f163006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:01:12.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.213+0000 7f164b400700 1 --2- 172.21.15.123:0/1870461242 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f163006cc30 0x7f163006f0e0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f163c009fd0 tx=0x7f163c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:01:12.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.217+0000 7f1643400700 1 -- 172.21.15.123:0/1870461242 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f1638059760 con 0x7f164c1037a0 2024-03-30T05:01:12.215 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:01:11 smithi123 ceph-mon[139369]: from='client.35636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:01:12.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:01:11 smithi135 ceph-mon[106011]: from='client.35636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:01:12.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.461+0000 7f1652800700 1 -- 172.21.15.123:0/1870461242 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f164c109dc0 con 0x7f164c1037a0 2024-03-30T05:01:12.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.462+0000 7f1643400700 1 -- 172.21.15.123:0/1870461242 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 34 v34) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f16380592f0 con 0x7f164c1037a0 2024-03-30T05:01:12.460 INFO:teuthology.orchestra.run.smithi123.stdout:e34 2024-03-30T05:01:12.461 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T05:01:12.461 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T05:01:12.461 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T05:01:12.461 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:01:12.461 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T05:01:12.461 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T05:01:12.461 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T05:01:12.461 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T05:01:12.461 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T05:01:12.461 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T05:01:12.461 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T05:01:12.461 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T05:01:12.461 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T05:01:12.462 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T05:01:12.462 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T05:01:12.462 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T05:01:12.462 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T05:01:12.462 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T05:01:12.462 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T05:01:12.462 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T05:01:12.462 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T05:01:12.462 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T05:01:12.462 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T05:01:12.462 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T05:01:12.462 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T05:01:12.462 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T05:01:12.463 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T05:01:12.463 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T05:01:12.463 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T05:01:12.463 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T05:01:12.463 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T05:01:12.463 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T05:01:12.463 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:01:12.463 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:01:12.463 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T05:01:12.463 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:01:12.463 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:01:12.463 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:01:12.463 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:01:12.463 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44975} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/3788988014,v1:172.21.15.135:6827/3788988014] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:01:12.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.464+0000 7f1652800700 1 -- 172.21.15.123:0/1870461242 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f163006cc30 msgr2=0x7f163006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:12.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.464+0000 7f1652800700 1 --2- 172.21.15.123:0/1870461242 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f163006cc30 0x7f163006f0e0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f163c009fd0 tx=0x7f163c009e30 comp rx=0 tx=0).stop 2024-03-30T05:01:12.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.464+0000 7f1652800700 1 -- 172.21.15.123:0/1870461242 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f164c1037a0 msgr2=0x7f164c114060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:12.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.464+0000 7f1652800700 1 --2- 172.21.15.123:0/1870461242 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f164c1037a0 0x7f164c114060 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f1638002a50 tx=0x7f1638004600 comp rx=0 tx=0).stop 2024-03-30T05:01:12.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.465+0000 7f1652800700 1 -- 172.21.15.123:0/1870461242 shutdown_connections 2024-03-30T05:01:12.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.465+0000 7f1652800700 1 --2- 172.21.15.123:0/1870461242 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f163006cc30 0x7f163006f0e0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:12.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.465+0000 7f1652800700 1 --2- 172.21.15.123:0/1870461242 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f164c102e00 0x7f164c113b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:12.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.465+0000 7f1652800700 1 --2- 172.21.15.123:0/1870461242 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f164c1037a0 0x7f164c114060 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:12.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.465+0000 7f1652800700 1 -- 172.21.15.123:0/1870461242 >> 172.21.15.123:0/1870461242 conn(0x7f164c0f89b0 msgr2=0x7f164c0f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:01:12.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.465+0000 7f1652800700 1 -- 172.21.15.123:0/1870461242 shutdown_connections 2024-03-30T05:01:12.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.465+0000 7f1652800700 1 -- 172.21.15.123:0/1870461242 wait complete. 2024-03-30T05:01:12.465 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 34 2024-03-30T05:01:12.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.599+0000 7f07f3200700 1 -- 172.21.15.123:0/3933009883 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07ec1037c0 msgr2=0x7f07ec0fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:12.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.599+0000 7f07f3200700 1 --2- 172.21.15.123:0/3933009883 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07ec1037c0 0x7f07ec0fce30 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f07d8004940 tx=0x7f07d8009e30 comp rx=0 tx=0).stop 2024-03-30T05:01:12.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.599+0000 7f07f3200700 1 -- 172.21.15.123:0/3933009883 shutdown_connections 2024-03-30T05:01:12.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.600+0000 7f07f3200700 1 --2- 172.21.15.123:0/3933009883 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07ec1037c0 0x7f07ec0fce30 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:12.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.600+0000 7f07f3200700 1 --2- 172.21.15.123:0/3933009883 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f07ec102e20 0x7f07ec1031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:12.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.600+0000 7f07f3200700 1 -- 172.21.15.123:0/3933009883 >> 172.21.15.123:0/3933009883 conn(0x7f07ec0f8970 msgr2=0x7f07ec0fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:01:12.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.600+0000 7f07f3200700 1 -- 172.21.15.123:0/3933009883 shutdown_connections 2024-03-30T05:01:12.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.600+0000 7f07f3200700 1 -- 172.21.15.123:0/3933009883 wait complete. 2024-03-30T05:01:12.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.600+0000 7f07f3200700 1 Processor -- start 2024-03-30T05:01:12.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.601+0000 7f07f3200700 1 -- start start 2024-03-30T05:01:12.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.601+0000 7f07f3200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07ec102e20 0x7f07ec115cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:01:12.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.601+0000 7f07f3200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f07ec1037c0 0x7f07ec116200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:01:12.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.601+0000 7f07f3200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07ec116850 con 0x7f07ec102e20 2024-03-30T05:01:12.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.601+0000 7f07f3200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07ec10fc20 con 0x7f07ec1037c0 2024-03-30T05:01:12.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.601+0000 7f07ebe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07ec102e20 0x7f07ec115cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:01:12.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.601+0000 7f07ebe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07ec102e20 0x7f07ec115cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58380/0 (socket says 172.21.15.123:58380) 2024-03-30T05:01:12.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.601+0000 7f07ebe00700 1 -- 172.21.15.123:0/1135603089 learned_addr learned my addr 172.21.15.123:0/1135603089 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:01:12.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.601+0000 7f07eb400700 1 --2- 172.21.15.123:0/1135603089 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f07ec1037c0 0x7f07ec116200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:01:12.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.602+0000 7f07ebe00700 1 -- 172.21.15.123:0/1135603089 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f07ec1037c0 msgr2=0x7f07ec116200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:12.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.602+0000 7f07ebe00700 1 --2- 172.21.15.123:0/1135603089 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f07ec1037c0 0x7f07ec116200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:12.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.602+0000 7f07ebe00700 1 -- 172.21.15.123:0/1135603089 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07d8005040 con 0x7f07ec102e20 2024-03-30T05:01:12.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.602+0000 7f07ebe00700 1 --2- 172.21.15.123:0/1135603089 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07ec102e20 0x7f07ec115cc0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f07dc002a50 tx=0x7f07dc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:01:12.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.602+0000 7f07e3e00700 1 -- 172.21.15.123:0/1135603089 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07dc007cc0 con 0x7f07ec102e20 2024-03-30T05:01:12.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.602+0000 7f07e3e00700 1 -- 172.21.15.123:0/1135603089 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f07dc00f040 con 0x7f07ec102e20 2024-03-30T05:01:12.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.602+0000 7f07f3200700 1 -- 172.21.15.123:0/1135603089 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07ec10ff00 con 0x7f07ec102e20 2024-03-30T05:01:12.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.602+0000 7f07e3e00700 1 -- 172.21.15.123:0/1135603089 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07dc00d9c0 con 0x7f07ec102e20 2024-03-30T05:01:12.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.602+0000 7f07f3200700 1 -- 172.21.15.123:0/1135603089 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07ec1104c0 con 0x7f07ec102e20 2024-03-30T05:01:12.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.603+0000 7f07f3200700 1 -- 172.21.15.123:0/1135603089 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07ec0fe570 con 0x7f07ec102e20 2024-03-30T05:01:12.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.604+0000 7f07e3e00700 1 -- 172.21.15.123:0/1135603089 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f07dc00db20 con 0x7f07ec102e20 2024-03-30T05:01:12.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.605+0000 7f07e3e00700 1 --2- 172.21.15.123:0/1135603089 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f07d006cc30 0x7f07d006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:01:12.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.605+0000 7f07e3e00700 1 -- 172.21.15.123:0/1135603089 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f07dc08bde0 con 0x7f07ec102e20 2024-03-30T05:01:12.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.605+0000 7f07eb400700 1 --2- 172.21.15.123:0/1135603089 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f07d006cc30 0x7f07d006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:01:12.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.606+0000 7f07eb400700 1 --2- 172.21.15.123:0/1135603089 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f07d006cc30 0x7f07d006f0e0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f07ec111560 tx=0x7f07d8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:01:12.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.609+0000 7f07e3e00700 1 -- 172.21.15.123:0/1135603089 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f07dc0597b0 con 0x7f07ec102e20 2024-03-30T05:01:12.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.831+0000 7f07f3200700 1 -- 172.21.15.123:0/1135603089 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f07ec04bf40 con 0x7f07d006cc30 2024-03-30T05:01:12.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.833+0000 7f07e3e00700 1 -- 172.21.15.123:0/1135603089 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f07ec04bf40 con 0x7f07d006cc30 2024-03-30T05:01:12.831 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T05:01:12.831 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T05:01:12.831 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T05:01:12.831 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T05:01:12.831 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T05:01:12.831 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T05:01:12.831 INFO:teuthology.orchestra.run.smithi123.stdout: "mds", 2024-03-30T05:01:12.832 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T05:01:12.832 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T05:01:12.832 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T05:01:12.832 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T05:01:12.832 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "16/21 daemons upgraded", 2024-03-30T05:01:12.832 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T05:01:12.832 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T05:01:12.832 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T05:01:12.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.835+0000 7f07f3200700 1 -- 172.21.15.123:0/1135603089 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f07d006cc30 msgr2=0x7f07d006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:12.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.835+0000 7f07f3200700 1 --2- 172.21.15.123:0/1135603089 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f07d006cc30 0x7f07d006f0e0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f07ec111560 tx=0x7f07d8009e30 comp rx=0 tx=0).stop 2024-03-30T05:01:12.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.835+0000 7f07f3200700 1 -- 172.21.15.123:0/1135603089 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07ec102e20 msgr2=0x7f07ec115cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:12.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.835+0000 7f07f3200700 1 --2- 172.21.15.123:0/1135603089 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07ec102e20 0x7f07ec115cc0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f07dc002a50 tx=0x7f07dc0046b0 comp rx=0 tx=0).stop 2024-03-30T05:01:12.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.835+0000 7f07f3200700 1 -- 172.21.15.123:0/1135603089 shutdown_connections 2024-03-30T05:01:12.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.835+0000 7f07f3200700 1 --2- 172.21.15.123:0/1135603089 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f07d006cc30 0x7f07d006f0e0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:12.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.835+0000 7f07f3200700 1 --2- 172.21.15.123:0/1135603089 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07ec102e20 0x7f07ec115cc0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:12.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.835+0000 7f07f3200700 1 --2- 172.21.15.123:0/1135603089 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f07ec1037c0 0x7f07ec116200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:12.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.835+0000 7f07f3200700 1 -- 172.21.15.123:0/1135603089 >> 172.21.15.123:0/1135603089 conn(0x7f07ec0f8970 msgr2=0x7f07ec0f99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:01:12.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.835+0000 7f07f3200700 1 -- 172.21.15.123:0/1135603089 shutdown_connections 2024-03-30T05:01:12.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.835+0000 7f07f3200700 1 -- 172.21.15.123:0/1135603089 wait complete. 2024-03-30T05:01:12.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.969+0000 7f835a000700 1 -- 172.21.15.123:0/1932941665 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8354102e00 msgr2=0x7f83541031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:12.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.969+0000 7f835a000700 1 --2- 172.21.15.123:0/1932941665 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8354102e00 0x7f83541031d0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f8344004940 tx=0x7f8344009e30 comp rx=0 tx=0).stop 2024-03-30T05:01:12.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.969+0000 7f835a000700 1 -- 172.21.15.123:0/1932941665 shutdown_connections 2024-03-30T05:01:12.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.969+0000 7f835a000700 1 --2- 172.21.15.123:0/1932941665 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f83541037a0 0x7f83540fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:12.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.969+0000 7f835a000700 1 --2- 172.21.15.123:0/1932941665 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8354102e00 0x7f83541031d0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:12.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.969+0000 7f835a000700 1 -- 172.21.15.123:0/1932941665 >> 172.21.15.123:0/1932941665 conn(0x7f83540f89b0 msgr2=0x7f83540fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:01:12.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.969+0000 7f835a000700 1 -- 172.21.15.123:0/1932941665 shutdown_connections 2024-03-30T05:01:12.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.969+0000 7f835a000700 1 -- 172.21.15.123:0/1932941665 wait complete. 2024-03-30T05:01:12.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.970+0000 7f835a000700 1 Processor -- start 2024-03-30T05:01:12.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.970+0000 7f835a000700 1 -- start start 2024-03-30T05:01:12.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.970+0000 7f835a000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8354102e00 0x7f83540702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:01:12.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.970+0000 7f835a000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f83541037a0 0x7f8354070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:01:12.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.970+0000 7f835a000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8354070d90 con 0x7f8354102e00 2024-03-30T05:01:12.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.970+0000 7f835a000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8354070ed0 con 0x7f83541037a0 2024-03-30T05:01:12.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.970+0000 7f8353400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8354102e00 0x7f83540702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:01:12.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.970+0000 7f8353400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8354102e00 0x7f83540702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58392/0 (socket says 172.21.15.123:58392) 2024-03-30T05:01:12.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.970+0000 7f8353400700 1 -- 172.21.15.123:0/595479455 learned_addr learned my addr 172.21.15.123:0/595479455 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:01:12.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.971+0000 7f8352a00700 1 --2- 172.21.15.123:0/595479455 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f83541037a0 0x7f8354070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:01:12.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.971+0000 7f8353400700 1 -- 172.21.15.123:0/595479455 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f83541037a0 msgr2=0x7f8354070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:12.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.971+0000 7f8353400700 1 --2- 172.21.15.123:0/595479455 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f83541037a0 0x7f8354070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:12.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.971+0000 7f8353400700 1 -- 172.21.15.123:0/595479455 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8344005040 con 0x7f8354102e00 2024-03-30T05:01:12.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.971+0000 7f8353400700 1 --2- 172.21.15.123:0/595479455 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8354102e00 0x7f83540702c0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f8344008d80 tx=0x7f8344009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:01:12.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.971+0000 7f834b400700 1 -- 172.21.15.123:0/595479455 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f834400e030 con 0x7f8354102e00 2024-03-30T05:01:12.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.971+0000 7f834b400700 1 -- 172.21.15.123:0/595479455 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8344013de0 con 0x7f8354102e00 2024-03-30T05:01:12.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.971+0000 7f834b400700 1 -- 172.21.15.123:0/595479455 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f834401cd60 con 0x7f8354102e00 2024-03-30T05:01:12.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.971+0000 7f835a000700 1 -- 172.21.15.123:0/595479455 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8354071120 con 0x7f8354102e00 2024-03-30T05:01:12.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.972+0000 7f835a000700 1 -- 172.21.15.123:0/595479455 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f835411a190 con 0x7f8354102e00 2024-03-30T05:01:12.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.972+0000 7f835a000700 1 -- 172.21.15.123:0/595479455 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83540fe550 con 0x7f8354102e00 2024-03-30T05:01:12.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.974+0000 7f834b400700 1 -- 172.21.15.123:0/595479455 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f834401c590 con 0x7f8354102e00 2024-03-30T05:01:12.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.974+0000 7f834b400700 1 --2- 172.21.15.123:0/595479455 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f833806cc30 0x7f833806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:01:12.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.974+0000 7f834b400700 1 -- 172.21.15.123:0/595479455 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f834408d7f0 con 0x7f8354102e00 2024-03-30T05:01:12.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.975+0000 7f8352a00700 1 --2- 172.21.15.123:0/595479455 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f833806cc30 0x7f833806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:01:12.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.975+0000 7f8352a00700 1 --2- 172.21.15.123:0/595479455 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f833806cc30 0x7f833806f0e0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f8340004600 tx=0x7f834000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:01:12.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:12.979+0000 7f834b400700 1 -- 172.21.15.123:0/595479455 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f83440578f0 con 0x7f8354102e00 2024-03-30T05:01:13.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:01:12 smithi123 ceph-mon[139369]: from='client.35640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:01:13.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:01:12 smithi123 ceph-mon[139369]: pgmap v901: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:13.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:01:12 smithi123 ceph-mon[139369]: from='client.35644 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:01:13.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:01:12 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/1974636208' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:01:13.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:01:12 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/1870461242' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T05:01:13.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:13.262+0000 7f835a000700 1 -- 172.21.15.123:0/595479455 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f835404bf60 con 0x7f8354102e00 2024-03-30T05:01:13.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:13.262+0000 7f834b400700 1 -- 172.21.15.123:0/595479455 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f834405af10 con 0x7f8354102e00 2024-03-30T05:01:13.260 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T05:01:13.261 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-30T05:01:13.261 INFO:teuthology.orchestra.run.smithi123.stdout: host smithi135 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi135/config 2024-03-30T05:01:13.261 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-30T05:01:13.261 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T05:01:13.261 INFO:teuthology.orchestra.run.smithi123.stdout:Traceback (most recent call last): 2024-03-30T05:01:13.261 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-30T05:01:13.261 INFO:teuthology.orchestra.run.smithi123.stdout: "__main__", mod_spec) 2024-03-30T05:01:13.261 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-30T05:01:13.261 INFO:teuthology.orchestra.run.smithi123.stdout: exec(code, run_globals) 2024-03-30T05:01:13.261 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10889, in 2024-03-30T05:01:13.261 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10877, in main 2024-03-30T05:01:13.261 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-30T05:01:13.261 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-30T05:01:13.262 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-30T05:01:13.262 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-30T05:01:13.262 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-03-30T05:01:13.262 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-30T05:01:13.262 INFO:teuthology.orchestra.run.smithi123.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T05:01:13.262 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T05:01:13.262 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T05:01:13.262 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T05:01:13.262 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T05:01:13.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:13.264+0000 7f835a000700 1 -- 172.21.15.123:0/595479455 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f833806cc30 msgr2=0x7f833806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:13.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:13.264+0000 7f835a000700 1 --2- 172.21.15.123:0/595479455 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f833806cc30 0x7f833806f0e0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f8340004600 tx=0x7f834000b040 comp rx=0 tx=0).stop 2024-03-30T05:01:13.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:13.264+0000 7f835a000700 1 -- 172.21.15.123:0/595479455 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8354102e00 msgr2=0x7f83540702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:13.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:13.264+0000 7f835a000700 1 --2- 172.21.15.123:0/595479455 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8354102e00 0x7f83540702c0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f8344008d80 tx=0x7f8344009e30 comp rx=0 tx=0).stop 2024-03-30T05:01:13.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:13.264+0000 7f835a000700 1 -- 172.21.15.123:0/595479455 shutdown_connections 2024-03-30T05:01:13.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:13.264+0000 7f835a000700 1 --2- 172.21.15.123:0/595479455 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f833806cc30 0x7f833806f0e0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:13.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:13.264+0000 7f835a000700 1 --2- 172.21.15.123:0/595479455 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8354102e00 0x7f83540702c0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:13.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:13.264+0000 7f835a000700 1 --2- 172.21.15.123:0/595479455 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f83541037a0 0x7f8354070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:13.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:13.264+0000 7f835a000700 1 -- 172.21.15.123:0/595479455 >> 172.21.15.123:0/595479455 conn(0x7f83540f89b0 msgr2=0x7f83540f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:01:13.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:13.265+0000 7f835a000700 1 -- 172.21.15.123:0/595479455 shutdown_connections 2024-03-30T05:01:13.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:13.265+0000 7f835a000700 1 -- 172.21.15.123:0/595479455 wait complete. 2024-03-30T05:01:13.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:01:12 smithi135 ceph-mon[106011]: from='client.35640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:01:13.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:01:12 smithi135 ceph-mon[106011]: pgmap v901: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:13.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:01:12 smithi135 ceph-mon[106011]: from='client.35644 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:01:13.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:01:12 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/1974636208' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:01:13.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:01:12 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/1870461242' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T05:01:14.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:01:13 smithi123 ceph-mon[139369]: from='client.35656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:01:14.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:01:13 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/595479455' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T05:01:14.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:01:13 smithi135 ceph-mon[106011]: from='client.35656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:01:14.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:01:13 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/595479455' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T05:01:15.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:01:14 smithi123 ceph-mon[139369]: pgmap v902: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:15.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:01:14 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:01:15.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:01:14 smithi135 ceph-mon[106011]: pgmap v902: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:15.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:01:14 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:01:17.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:01:16 smithi123 ceph-mon[139369]: pgmap v903: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:17.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:01:16 smithi135 ceph-mon[106011]: pgmap v903: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:19.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:01:18 smithi123 ceph-mon[139369]: pgmap v904: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:19.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:01:18 smithi135 ceph-mon[106011]: pgmap v904: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:21.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:01:20 smithi123 ceph-mon[139369]: pgmap v905: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:21.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:01:20 smithi135 ceph-mon[106011]: pgmap v905: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:23.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:01:22 smithi123 ceph-mon[139369]: pgmap v906: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:23.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:01:22 smithi135 ceph-mon[106011]: pgmap v906: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:25.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:01:24 smithi123 ceph-mon[139369]: pgmap v907: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:25.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:01:24 smithi135 ceph-mon[106011]: pgmap v907: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:27.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:01:26 smithi123 ceph-mon[139369]: pgmap v908: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:27.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:01:26 smithi135 ceph-mon[106011]: pgmap v908: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:29.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:01:28 smithi123 ceph-mon[139369]: pgmap v909: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:29.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:01:28 smithi135 ceph-mon[106011]: pgmap v909: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:30.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:01:29 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:01:30.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:01:29 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:01:31.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:01:30 smithi123 ceph-mon[139369]: pgmap v910: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:31.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:01:30 smithi135 ceph-mon[106011]: pgmap v910: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:33.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:01:32 smithi123 ceph-mon[139369]: pgmap v911: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:33.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:01:32 smithi135 ceph-mon[106011]: pgmap v911: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:35.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:01:34 smithi123 ceph-mon[139369]: pgmap v912: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:35.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:01:34 smithi135 ceph-mon[106011]: pgmap v912: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:37.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:01:36 smithi123 ceph-mon[139369]: pgmap v913: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:37.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:01:36 smithi135 ceph-mon[106011]: pgmap v913: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:39.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:01:38 smithi123 ceph-mon[139369]: pgmap v914: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:39.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:01:38 smithi135 ceph-mon[106011]: pgmap v914: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:41.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:01:40 smithi123 ceph-mon[139369]: pgmap v915: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:41.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:01:40 smithi135 ceph-mon[106011]: pgmap v915: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:43.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:01:42 smithi135 ceph-mon[106011]: pgmap v916: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:43.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.415+0000 7f1cd3e00700 1 -- 172.21.15.123:0/3033929195 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ccc070360 msgr2=0x7f1ccc070730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:43.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.415+0000 7f1cd3e00700 1 --2- 172.21.15.123:0/3033929195 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ccc070360 0x7f1ccc070730 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f1cc0004970 tx=0x7f1cc0009e30 comp rx=0 tx=0).stop 2024-03-30T05:01:43.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.415+0000 7f1cd3e00700 1 -- 172.21.15.123:0/3033929195 shutdown_connections 2024-03-30T05:01:43.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.415+0000 7f1cd3e00700 1 --2- 172.21.15.123:0/3033929195 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1ccc070c70 0x7f1ccc109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:43.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.415+0000 7f1cd3e00700 1 --2- 172.21.15.123:0/3033929195 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ccc070360 0x7f1ccc070730 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:43.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.415+0000 7f1cd3e00700 1 -- 172.21.15.123:0/3033929195 >> 172.21.15.123:0/3033929195 conn(0x7f1ccc0f89d0 msgr2=0x7f1ccc0fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:01:43.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.415+0000 7f1cd3e00700 1 -- 172.21.15.123:0/3033929195 shutdown_connections 2024-03-30T05:01:43.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.415+0000 7f1cd3e00700 1 -- 172.21.15.123:0/3033929195 wait complete. 2024-03-30T05:01:43.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.416+0000 7f1cd3e00700 1 Processor -- start 2024-03-30T05:01:43.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.416+0000 7f1cd3e00700 1 -- start start 2024-03-30T05:01:43.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.416+0000 7f1cd3e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ccc070360 0x7f1ccc115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:01:43.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.416+0000 7f1cd3e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1ccc070c70 0x7f1ccc1161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:01:43.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.416+0000 7f1cd3e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ccc116800 con 0x7f1ccc070360 2024-03-30T05:01:43.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.416+0000 7f1cd3e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ccc10fbd0 con 0x7f1ccc070c70 2024-03-30T05:01:43.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.416+0000 7f1cd1400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ccc070360 0x7f1ccc115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:01:43.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.416+0000 7f1cd1400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ccc070360 0x7f1ccc115c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:51032/0 (socket says 172.21.15.123:51032) 2024-03-30T05:01:43.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.417+0000 7f1cd1400700 1 -- 172.21.15.123:0/3517743198 learned_addr learned my addr 172.21.15.123:0/3517743198 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:01:43.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.417+0000 7f1cd1400700 1 -- 172.21.15.123:0/3517743198 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1ccc070c70 msgr2=0x7f1ccc1161b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:43.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.417+0000 7f1cd0a00700 1 --2- 172.21.15.123:0/3517743198 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1ccc070c70 0x7f1ccc1161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:01:43.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.417+0000 7f1cd1400700 1 --2- 172.21.15.123:0/3517743198 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1ccc070c70 0x7f1ccc1161b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:43.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.417+0000 7f1cd1400700 1 -- 172.21.15.123:0/3517743198 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cc0005040 con 0x7f1ccc070360 2024-03-30T05:01:43.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.417+0000 7f1cd1400700 1 --2- 172.21.15.123:0/3517743198 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ccc070360 0x7f1ccc115c70 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f1cc0004f50 tx=0x7f1cc0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:01:43.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.417+0000 7f1cc9600700 1 -- 172.21.15.123:0/3517743198 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1cc000e040 con 0x7f1ccc070360 2024-03-30T05:01:43.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.417+0000 7f1cc9600700 1 -- 172.21.15.123:0/3517743198 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1cc0013420 con 0x7f1ccc070360 2024-03-30T05:01:43.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.417+0000 7f1cd3e00700 1 -- 172.21.15.123:0/3517743198 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ccc10fe50 con 0x7f1ccc070360 2024-03-30T05:01:43.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.417+0000 7f1cc9600700 1 -- 172.21.15.123:0/3517743198 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1cc001c610 con 0x7f1ccc070360 2024-03-30T05:01:43.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.418+0000 7f1cd3e00700 1 -- 172.21.15.123:0/3517743198 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ccc110390 con 0x7f1ccc070360 2024-03-30T05:01:43.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.419+0000 7f1cd3e00700 1 -- 172.21.15.123:0/3517743198 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ccc071aa0 con 0x7f1ccc070360 2024-03-30T05:01:43.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.420+0000 7f1cc9600700 1 -- 172.21.15.123:0/3517743198 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f1cc0013590 con 0x7f1ccc070360 2024-03-30T05:01:43.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.420+0000 7f1cc9600700 1 --2- 172.21.15.123:0/3517743198 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1cb406cc30 0x7f1cb406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:01:43.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.421+0000 7f1cc9600700 1 -- 172.21.15.123:0/3517743198 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f1cc008d310 con 0x7f1ccc070360 2024-03-30T05:01:43.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.421+0000 7f1cd0a00700 1 --2- 172.21.15.123:0/3517743198 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1cb406cc30 0x7f1cb406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:01:43.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.421+0000 7f1cd0a00700 1 --2- 172.21.15.123:0/3517743198 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1cb406cc30 0x7f1cb406f0e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc111510 tx=0x7f1cb800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:01:43.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.425+0000 7f1cc9600700 1 -- 172.21.15.123:0/3517743198 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f1cc005ace0 con 0x7f1ccc070360 2024-03-30T05:01:43.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:01:42 smithi123 ceph-mon[139369]: pgmap v916: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:43.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.647+0000 7f1cd3e00700 1 -- 172.21.15.123:0/3517743198 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1ccc04bf70 con 0x7f1cb406cc30 2024-03-30T05:01:43.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.648+0000 7f1cc9600700 1 -- 172.21.15.123:0/3517743198 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f1cb800f040 con 0x7f1cb406cc30 2024-03-30T05:01:43.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.651+0000 7f1cd3e00700 1 -- 172.21.15.123:0/3517743198 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1cb406cc30 msgr2=0x7f1cb406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:43.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.651+0000 7f1cd3e00700 1 --2- 172.21.15.123:0/3517743198 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1cb406cc30 0x7f1cb406f0e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc111510 tx=0x7f1cb800b040 comp rx=0 tx=0).stop 2024-03-30T05:01:43.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.651+0000 7f1cd3e00700 1 -- 172.21.15.123:0/3517743198 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ccc070360 msgr2=0x7f1ccc115c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:43.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.651+0000 7f1cd3e00700 1 --2- 172.21.15.123:0/3517743198 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ccc070360 0x7f1ccc115c70 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f1cc0004f50 tx=0x7f1cc0009e30 comp rx=0 tx=0).stop 2024-03-30T05:01:43.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.651+0000 7f1cd3e00700 1 -- 172.21.15.123:0/3517743198 shutdown_connections 2024-03-30T05:01:43.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.651+0000 7f1cd3e00700 1 --2- 172.21.15.123:0/3517743198 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1cb406cc30 0x7f1cb406f0e0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:43.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.651+0000 7f1cd3e00700 1 --2- 172.21.15.123:0/3517743198 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ccc070360 0x7f1ccc115c70 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:43.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.651+0000 7f1cd3e00700 1 --2- 172.21.15.123:0/3517743198 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1ccc070c70 0x7f1ccc1161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:43.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.651+0000 7f1cd3e00700 1 -- 172.21.15.123:0/3517743198 >> 172.21.15.123:0/3517743198 conn(0x7f1ccc0f89d0 msgr2=0x7f1ccc0f9f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:01:43.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.651+0000 7f1cd3e00700 1 -- 172.21.15.123:0/3517743198 shutdown_connections 2024-03-30T05:01:43.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.651+0000 7f1cd3e00700 1 -- 172.21.15.123:0/3517743198 wait complete. 2024-03-30T05:01:43.663 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T05:01:43.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.785+0000 7f7441c00700 1 -- 172.21.15.123:0/3789308208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f743c102e00 msgr2=0x7f743c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:43.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.785+0000 7f7441c00700 1 --2- 172.21.15.123:0/3789308208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f743c102e00 0x7f743c1031d0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f742c004940 tx=0x7f742c009e30 comp rx=0 tx=0).stop 2024-03-30T05:01:43.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.785+0000 7f7441c00700 1 -- 172.21.15.123:0/3789308208 shutdown_connections 2024-03-30T05:01:43.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.785+0000 7f7441c00700 1 --2- 172.21.15.123:0/3789308208 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f743c1037a0 0x7f743c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:43.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.785+0000 7f7441c00700 1 --2- 172.21.15.123:0/3789308208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f743c102e00 0x7f743c1031d0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:43.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.785+0000 7f7441c00700 1 -- 172.21.15.123:0/3789308208 >> 172.21.15.123:0/3789308208 conn(0x7f743c0f89b0 msgr2=0x7f743c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:01:43.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.785+0000 7f7441c00700 1 -- 172.21.15.123:0/3789308208 shutdown_connections 2024-03-30T05:01:43.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.785+0000 7f7441c00700 1 -- 172.21.15.123:0/3789308208 wait complete. 2024-03-30T05:01:43.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.786+0000 7f7441c00700 1 Processor -- start 2024-03-30T05:01:43.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.786+0000 7f7441c00700 1 -- start start 2024-03-30T05:01:43.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.787+0000 7f7441c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f743c102e00 0x7f743c115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:01:43.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.787+0000 7f7441c00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f743c1037a0 0x7f743c116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:01:43.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.787+0000 7f7441c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f743c116860 con 0x7f743c102e00 2024-03-30T05:01:43.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.787+0000 7f7441c00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f743c10fc30 con 0x7f743c1037a0 2024-03-30T05:01:43.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.787+0000 7f743aa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f743c102e00 0x7f743c115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:01:43.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.787+0000 7f743aa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f743c102e00 0x7f743c115cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:51052/0 (socket says 172.21.15.123:51052) 2024-03-30T05:01:43.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.787+0000 7f743aa00700 1 -- 172.21.15.123:0/2389906067 learned_addr learned my addr 172.21.15.123:0/2389906067 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:01:43.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.787+0000 7f743a000700 1 --2- 172.21.15.123:0/2389906067 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f743c1037a0 0x7f743c116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:01:43.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.787+0000 7f743aa00700 1 -- 172.21.15.123:0/2389906067 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f743c1037a0 msgr2=0x7f743c116210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:43.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.787+0000 7f743aa00700 1 --2- 172.21.15.123:0/2389906067 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f743c1037a0 0x7f743c116210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:43.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.787+0000 7f743aa00700 1 -- 172.21.15.123:0/2389906067 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f742c005040 con 0x7f743c102e00 2024-03-30T05:01:43.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.788+0000 7f743aa00700 1 --2- 172.21.15.123:0/2389906067 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f743c102e00 0x7f743c115cd0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f742c009360 tx=0x7f742c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:01:43.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.788+0000 7f7432a00700 1 -- 172.21.15.123:0/2389906067 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f742c00e040 con 0x7f743c102e00 2024-03-30T05:01:43.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.788+0000 7f7432a00700 1 -- 172.21.15.123:0/2389906067 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f742c013420 con 0x7f743c102e00 2024-03-30T05:01:43.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.788+0000 7f7432a00700 1 -- 172.21.15.123:0/2389906067 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f742c01c5d0 con 0x7f743c102e00 2024-03-30T05:01:43.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.788+0000 7f7441c00700 1 -- 172.21.15.123:0/2389906067 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f743c10feb0 con 0x7f743c102e00 2024-03-30T05:01:43.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.788+0000 7f7441c00700 1 -- 172.21.15.123:0/2389906067 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f743c110470 con 0x7f743c102e00 2024-03-30T05:01:43.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.789+0000 7f7441c00700 1 -- 172.21.15.123:0/2389906067 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f743c0fe550 con 0x7f743c102e00 2024-03-30T05:01:43.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.790+0000 7f7432a00700 1 -- 172.21.15.123:0/2389906067 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f742c013590 con 0x7f743c102e00 2024-03-30T05:01:43.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.791+0000 7f7432a00700 1 --2- 172.21.15.123:0/2389906067 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f742006cc30 0x7f742006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:01:43.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.791+0000 7f7432a00700 1 -- 172.21.15.123:0/2389906067 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f742c08d1b0 con 0x7f743c102e00 2024-03-30T05:01:43.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.791+0000 7f743a000700 1 --2- 172.21.15.123:0/2389906067 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f742006cc30 0x7f742006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:01:43.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.792+0000 7f743a000700 1 --2- 172.21.15.123:0/2389906067 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f742006cc30 0x7f742006f0e0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f743c111570 tx=0x7f742800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:01:43.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:43.795+0000 7f7432a00700 1 -- 172.21.15.123:0/2389906067 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f742c05ac00 con 0x7f743c102e00 2024-03-30T05:01:44.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.017+0000 7f7441c00700 1 -- 172.21.15.123:0/2389906067 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f743c04bed0 con 0x7f742006cc30 2024-03-30T05:01:44.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.019+0000 7f7432a00700 1 -- 172.21.15.123:0/2389906067 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f743c04bed0 con 0x7f742006cc30 2024-03-30T05:01:44.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.021+0000 7f7441c00700 1 -- 172.21.15.123:0/2389906067 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f742006cc30 msgr2=0x7f742006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:44.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.021+0000 7f7441c00700 1 --2- 172.21.15.123:0/2389906067 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f742006cc30 0x7f742006f0e0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f743c111570 tx=0x7f742800b040 comp rx=0 tx=0).stop 2024-03-30T05:01:44.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.021+0000 7f7441c00700 1 -- 172.21.15.123:0/2389906067 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f743c102e00 msgr2=0x7f743c115cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:44.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.021+0000 7f7441c00700 1 --2- 172.21.15.123:0/2389906067 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f743c102e00 0x7f743c115cd0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f742c009360 tx=0x7f742c009e30 comp rx=0 tx=0).stop 2024-03-30T05:01:44.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.021+0000 7f7441c00700 1 -- 172.21.15.123:0/2389906067 shutdown_connections 2024-03-30T05:01:44.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.021+0000 7f7441c00700 1 --2- 172.21.15.123:0/2389906067 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f742006cc30 0x7f742006f0e0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:44.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.021+0000 7f7441c00700 1 --2- 172.21.15.123:0/2389906067 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f743c102e00 0x7f743c115cd0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:44.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.021+0000 7f7441c00700 1 --2- 172.21.15.123:0/2389906067 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f743c1037a0 0x7f743c116210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:44.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.021+0000 7f7441c00700 1 -- 172.21.15.123:0/2389906067 >> 172.21.15.123:0/2389906067 conn(0x7f743c0f89b0 msgr2=0x7f743c0f9980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:01:44.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.021+0000 7f7441c00700 1 -- 172.21.15.123:0/2389906067 shutdown_connections 2024-03-30T05:01:44.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.021+0000 7f7441c00700 1 -- 172.21.15.123:0/2389906067 wait complete. 2024-03-30T05:01:44.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.156+0000 7f8cdec00700 1 -- 172.21.15.123:0/1367435696 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8cd81037a0 msgr2=0x7f8cd80fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:44.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.156+0000 7f8cdec00700 1 --2- 172.21.15.123:0/1367435696 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8cd81037a0 0x7f8cd80fce10 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f8cc4004940 tx=0x7f8cc4009e30 comp rx=0 tx=0).stop 2024-03-30T05:01:44.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.156+0000 7f8cdec00700 1 -- 172.21.15.123:0/1367435696 shutdown_connections 2024-03-30T05:01:44.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.156+0000 7f8cdec00700 1 --2- 172.21.15.123:0/1367435696 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8cd81037a0 0x7f8cd80fce10 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:44.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.156+0000 7f8cdec00700 1 --2- 172.21.15.123:0/1367435696 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8cd8102e00 0x7f8cd81031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:44.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.156+0000 7f8cdec00700 1 -- 172.21.15.123:0/1367435696 >> 172.21.15.123:0/1367435696 conn(0x7f8cd80f89b0 msgr2=0x7f8cd80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:01:44.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.156+0000 7f8cdec00700 1 -- 172.21.15.123:0/1367435696 shutdown_connections 2024-03-30T05:01:44.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.156+0000 7f8cdec00700 1 -- 172.21.15.123:0/1367435696 wait complete. 2024-03-30T05:01:44.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.157+0000 7f8cdec00700 1 Processor -- start 2024-03-30T05:01:44.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.157+0000 7f8cdec00700 1 -- start start 2024-03-30T05:01:44.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.157+0000 7f8cdec00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8cd8102e00 0x7f8cd8115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:01:44.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.157+0000 7f8cdec00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8cd81037a0 0x7f8cd81161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:01:44.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.157+0000 7f8cdec00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd81167f0 con 0x7f8cd81037a0 2024-03-30T05:01:44.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.157+0000 7f8cdec00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd810fbc0 con 0x7f8cd8102e00 2024-03-30T05:01:44.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.157+0000 7f8cd7400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8cd81037a0 0x7f8cd81161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:01:44.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.157+0000 7f8cd7400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8cd81037a0 0x7f8cd81161a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40960/0 (socket says 172.21.15.123:40960) 2024-03-30T05:01:44.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.158+0000 7f8cd7400700 1 -- 172.21.15.123:0/2889248053 learned_addr learned my addr 172.21.15.123:0/2889248053 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:01:44.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.158+0000 7f8cd7e00700 1 --2- 172.21.15.123:0/2889248053 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8cd8102e00 0x7f8cd8115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:01:44.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.158+0000 7f8cd7400700 1 -- 172.21.15.123:0/2889248053 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8cd8102e00 msgr2=0x7f8cd8115c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:44.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.158+0000 7f8cd7400700 1 --2- 172.21.15.123:0/2889248053 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8cd8102e00 0x7f8cd8115c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:44.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.158+0000 7f8cd7400700 1 -- 172.21.15.123:0/2889248053 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc4005040 con 0x7f8cd81037a0 2024-03-30T05:01:44.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.158+0000 7f8cd7e00700 1 --2- 172.21.15.123:0/2889248053 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8cd8102e00 0x7f8cd8115c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T05:01:44.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.158+0000 7f8cd7400700 1 --2- 172.21.15.123:0/2889248053 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8cd81037a0 0x7f8cd81161a0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f8cc40093b0 tx=0x7f8cc4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:01:44.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.158+0000 7f8ccfe00700 1 -- 172.21.15.123:0/2889248053 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8cc400e040 con 0x7f8cd81037a0 2024-03-30T05:01:44.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.158+0000 7f8ccfe00700 1 -- 172.21.15.123:0/2889248053 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8cc4013420 con 0x7f8cd81037a0 2024-03-30T05:01:44.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.158+0000 7f8cdec00700 1 -- 172.21.15.123:0/2889248053 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cd810fe40 con 0x7f8cd81037a0 2024-03-30T05:01:44.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.158+0000 7f8ccfe00700 1 -- 172.21.15.123:0/2889248053 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8cc401c630 con 0x7f8cd81037a0 2024-03-30T05:01:44.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.159+0000 7f8cdec00700 1 -- 172.21.15.123:0/2889248053 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cd8110400 con 0x7f8cd81037a0 2024-03-30T05:01:44.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.160+0000 7f8cdec00700 1 -- 172.21.15.123:0/2889248053 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cd80fe550 con 0x7f8cd81037a0 2024-03-30T05:01:44.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.160+0000 7f8ccfe00700 1 -- 172.21.15.123:0/2889248053 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f8cc40139e0 con 0x7f8cd81037a0 2024-03-30T05:01:44.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.161+0000 7f8ccfe00700 1 --2- 172.21.15.123:0/2889248053 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8cbc06cc20 0x7f8cbc06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:01:44.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.161+0000 7f8ccfe00700 1 -- 172.21.15.123:0/2889248053 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f8cc408d1c0 con 0x7f8cd81037a0 2024-03-30T05:01:44.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.162+0000 7f8cd7e00700 1 --2- 172.21.15.123:0/2889248053 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8cbc06cc20 0x7f8cbc06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:01:44.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.162+0000 7f8cd7e00700 1 --2- 172.21.15.123:0/2889248053 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8cbc06cc20 0x7f8cbc06f0d0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f8cc8007fd0 tx=0x7f8cc8009f60 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:01:44.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.166+0000 7f8ccfe00700 1 -- 172.21.15.123:0/2889248053 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8cc405ac90 con 0x7f8cd81037a0 2024-03-30T05:01:44.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.379+0000 7f8cdec00700 1 -- 172.21.15.123:0/2889248053 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f8cd8109e10 con 0x7f8cbc06cc20 2024-03-30T05:01:44.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.389+0000 7f8ccfe00700 1 -- 172.21.15.123:0/2889248053 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f8cd8109e10 con 0x7f8cbc06cc20 2024-03-30T05:01:44.387 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T05:01:44.387 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (30m) 8m ago 43m 14.6M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T05:01:44.387 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (28m) 8m ago 43m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T05:01:44.387 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (28m) 8m ago 42m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T05:01:44.387 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (30m) 8m ago 43m 73.5M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T05:01:44.387 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (23m) 8m ago 37m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T05:01:44.387 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (24m) 8m ago 37m 16.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T05:01:44.387 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (8m) 8m ago 36m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 87b073a6c9ca 2024-03-30T05:01:44.388 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (23m) 8m ago 37m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T05:01:44.388 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (33m) 8m ago 45m 510M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T05:01:44.388 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (32m) 8m ago 42m 456M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T05:01:44.388 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (29m) 8m ago 45m 70.0M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T05:01:44.388 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (28m) 8m ago 42m 51.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T05:01:44.388 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (31m) 8m ago 43m 8636k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T05:01:44.388 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (31m) 8m ago 42m 8678k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T05:01:44.388 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (27m) 8m ago 41m 107M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T05:01:44.388 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (27m) 8m ago 40m 97.2M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T05:01:44.388 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (26m) 8m ago 40m 89.8M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T05:01:44.388 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (25m) 8m ago 39m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T05:01:44.388 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (25m) 8m ago 39m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T05:01:44.388 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (24m) 8m ago 38m 88.8M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T05:01:44.388 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (31m) 8m ago 42m 63.9M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T05:01:44.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.391+0000 7f8cdec00700 1 -- 172.21.15.123:0/2889248053 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8cbc06cc20 msgr2=0x7f8cbc06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:44.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.391+0000 7f8cdec00700 1 --2- 172.21.15.123:0/2889248053 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8cbc06cc20 0x7f8cbc06f0d0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f8cc8007fd0 tx=0x7f8cc8009f60 comp rx=0 tx=0).stop 2024-03-30T05:01:44.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.391+0000 7f8cdec00700 1 -- 172.21.15.123:0/2889248053 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8cd81037a0 msgr2=0x7f8cd81161a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:44.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.391+0000 7f8cdec00700 1 --2- 172.21.15.123:0/2889248053 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8cd81037a0 0x7f8cd81161a0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f8cc40093b0 tx=0x7f8cc4009e30 comp rx=0 tx=0).stop 2024-03-30T05:01:44.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.391+0000 7f8cdec00700 1 -- 172.21.15.123:0/2889248053 shutdown_connections 2024-03-30T05:01:44.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.391+0000 7f8cdec00700 1 --2- 172.21.15.123:0/2889248053 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8cbc06cc20 0x7f8cbc06f0d0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:44.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.391+0000 7f8cdec00700 1 --2- 172.21.15.123:0/2889248053 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8cd8102e00 0x7f8cd8115c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:44.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.391+0000 7f8cdec00700 1 --2- 172.21.15.123:0/2889248053 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8cd81037a0 0x7f8cd81161a0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:44.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.391+0000 7f8cdec00700 1 -- 172.21.15.123:0/2889248053 >> 172.21.15.123:0/2889248053 conn(0x7f8cd80f89b0 msgr2=0x7f8cd80f99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:01:44.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.391+0000 7f8cdec00700 1 -- 172.21.15.123:0/2889248053 shutdown_connections 2024-03-30T05:01:44.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.391+0000 7f8cdec00700 1 -- 172.21.15.123:0/2889248053 wait complete. 2024-03-30T05:01:44.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.525+0000 7fea39800700 1 -- 172.21.15.123:0/355984471 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea340fde90 msgr2=0x7fea340fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:44.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.525+0000 7fea39800700 1 --2- 172.21.15.123:0/355984471 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea340fde90 0x7fea340fe260 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fea24004940 tx=0x7fea24009e30 comp rx=0 tx=0).stop 2024-03-30T05:01:44.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.525+0000 7fea39800700 1 -- 172.21.15.123:0/355984471 shutdown_connections 2024-03-30T05:01:44.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.525+0000 7fea39800700 1 --2- 172.21.15.123:0/355984471 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea340fe830 0x7fea34106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:44.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.525+0000 7fea39800700 1 --2- 172.21.15.123:0/355984471 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea340fde90 0x7fea340fe260 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:44.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.525+0000 7fea39800700 1 -- 172.21.15.123:0/355984471 >> 172.21.15.123:0/355984471 conn(0x7fea34072bc0 msgr2=0x7fea34072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:01:44.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.525+0000 7fea39800700 1 -- 172.21.15.123:0/355984471 shutdown_connections 2024-03-30T05:01:44.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.525+0000 7fea39800700 1 -- 172.21.15.123:0/355984471 wait complete. 2024-03-30T05:01:44.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.526+0000 7fea39800700 1 Processor -- start 2024-03-30T05:01:44.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.527+0000 7fea39800700 1 -- start start 2024-03-30T05:01:44.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.527+0000 7fea39800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea340fde90 0x7fea34113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:01:44.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.527+0000 7fea39800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea340fe830 0x7fea34114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:01:44.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.527+0000 7fea39800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea34114680 con 0x7fea340fe830 2024-03-30T05:01:44.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.527+0000 7fea39800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea3410da70 con 0x7fea340fde90 2024-03-30T05:01:44.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.528+0000 7fea32000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea340fe830 0x7fea34114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:01:44.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.528+0000 7fea32000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea340fe830 0x7fea34114030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40980/0 (socket says 172.21.15.123:40980) 2024-03-30T05:01:44.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.528+0000 7fea32000700 1 -- 172.21.15.123:0/704179122 learned_addr learned my addr 172.21.15.123:0/704179122 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:01:44.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.528+0000 7fea32a00700 1 --2- 172.21.15.123:0/704179122 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea340fde90 0x7fea34113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:01:44.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.528+0000 7fea32000700 1 -- 172.21.15.123:0/704179122 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea340fde90 msgr2=0x7fea34113af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:44.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.528+0000 7fea32000700 1 --2- 172.21.15.123:0/704179122 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea340fde90 0x7fea34113af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:44.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.528+0000 7fea32000700 1 -- 172.21.15.123:0/704179122 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea24005040 con 0x7fea340fe830 2024-03-30T05:01:44.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.528+0000 7fea32000700 1 --2- 172.21.15.123:0/704179122 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea340fe830 0x7fea34114030 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fea20002a50 tx=0x7fea200046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:01:44.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.528+0000 7fea2aa00700 1 -- 172.21.15.123:0/704179122 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea20007cc0 con 0x7fea340fe830 2024-03-30T05:01:44.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.528+0000 7fea2aa00700 1 -- 172.21.15.123:0/704179122 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fea2000f040 con 0x7fea340fe830 2024-03-30T05:01:44.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.528+0000 7fea39800700 1 -- 172.21.15.123:0/704179122 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea3410dd50 con 0x7fea340fe830 2024-03-30T05:01:44.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.528+0000 7fea2aa00700 1 -- 172.21.15.123:0/704179122 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea20009540 con 0x7fea340fe830 2024-03-30T05:01:44.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.529+0000 7fea39800700 1 -- 172.21.15.123:0/704179122 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea3410e290 con 0x7fea340fe830 2024-03-30T05:01:44.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.530+0000 7fea39800700 1 -- 172.21.15.123:0/704179122 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea341019b0 con 0x7fea340fe830 2024-03-30T05:01:44.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.531+0000 7fea2aa00700 1 -- 172.21.15.123:0/704179122 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fea200096a0 con 0x7fea340fe830 2024-03-30T05:01:44.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.531+0000 7fea2aa00700 1 --2- 172.21.15.123:0/704179122 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fea1806cc30 0x7fea1806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:01:44.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.532+0000 7fea2aa00700 1 -- 172.21.15.123:0/704179122 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fea2008be60 con 0x7fea340fe830 2024-03-30T05:01:44.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.532+0000 7fea32a00700 1 --2- 172.21.15.123:0/704179122 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fea1806cc30 0x7fea1806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:01:44.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.532+0000 7fea32a00700 1 --2- 172.21.15.123:0/704179122 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fea1806cc30 0x7fea1806f0e0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fea2400b3b0 tx=0x7fea24009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:01:44.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.536+0000 7fea2aa00700 1 -- 172.21.15.123:0/704179122 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fea200598b0 con 0x7fea340fe830 2024-03-30T05:01:44.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.821+0000 7fea39800700 1 -- 172.21.15.123:0/704179122 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fea3410f140 con 0x7fea340fe830 2024-03-30T05:01:44.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.821+0000 7fea2aa00700 1 -- 172.21.15.123:0/704179122 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7fea20059440 con 0x7fea340fe830 2024-03-30T05:01:44.819 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T05:01:44.820 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T05:01:44.820 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T05:01:44.820 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:01:44.820 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T05:01:44.820 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T05:01:44.820 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:01:44.820 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T05:01:44.820 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T05:01:44.820 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:01:44.820 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T05:01:44.820 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 4 2024-03-30T05:01:44.820 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:01:44.820 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T05:01:44.820 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 14 2024-03-30T05:01:44.821 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T05:01:44.821 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T05:01:44.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.823+0000 7fea39800700 1 -- 172.21.15.123:0/704179122 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fea1806cc30 msgr2=0x7fea1806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:44.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.823+0000 7fea39800700 1 --2- 172.21.15.123:0/704179122 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fea1806cc30 0x7fea1806f0e0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fea2400b3b0 tx=0x7fea24009e30 comp rx=0 tx=0).stop 2024-03-30T05:01:44.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.823+0000 7fea39800700 1 -- 172.21.15.123:0/704179122 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea340fe830 msgr2=0x7fea34114030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:44.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.823+0000 7fea39800700 1 --2- 172.21.15.123:0/704179122 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea340fe830 0x7fea34114030 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fea20002a50 tx=0x7fea200046b0 comp rx=0 tx=0).stop 2024-03-30T05:01:44.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.824+0000 7fea39800700 1 -- 172.21.15.123:0/704179122 shutdown_connections 2024-03-30T05:01:44.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.824+0000 7fea39800700 1 --2- 172.21.15.123:0/704179122 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fea1806cc30 0x7fea1806f0e0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:44.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.824+0000 7fea39800700 1 --2- 172.21.15.123:0/704179122 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea340fde90 0x7fea34113af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:44.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.824+0000 7fea39800700 1 --2- 172.21.15.123:0/704179122 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea340fe830 0x7fea34114030 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:44.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.824+0000 7fea39800700 1 -- 172.21.15.123:0/704179122 >> 172.21.15.123:0/704179122 conn(0x7fea34072bc0 msgr2=0x7fea340ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:01:44.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.824+0000 7fea39800700 1 -- 172.21.15.123:0/704179122 shutdown_connections 2024-03-30T05:01:44.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.824+0000 7fea39800700 1 -- 172.21.15.123:0/704179122 wait complete. 2024-03-30T05:01:44.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.957+0000 7f52fe800700 1 -- 172.21.15.123:0/902085868 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52f8070300 msgr2=0x7f52f80706d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:44.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.957+0000 7f52fe800700 1 --2- 172.21.15.123:0/902085868 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52f8070300 0x7f52f80706d0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f52e8004970 tx=0x7f52e8009e30 comp rx=0 tx=0).stop 2024-03-30T05:01:44.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.957+0000 7f52fe800700 1 -- 172.21.15.123:0/902085868 shutdown_connections 2024-03-30T05:01:44.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.957+0000 7f52fe800700 1 --2- 172.21.15.123:0/902085868 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f52f8070c10 0x7f52f8109190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:44.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.957+0000 7f52fe800700 1 --2- 172.21.15.123:0/902085868 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52f8070300 0x7f52f80706d0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:44.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.957+0000 7f52fe800700 1 -- 172.21.15.123:0/902085868 >> 172.21.15.123:0/902085868 conn(0x7f52f80f89d0 msgr2=0x7f52f80fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:01:44.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.957+0000 7f52fe800700 1 -- 172.21.15.123:0/902085868 shutdown_connections 2024-03-30T05:01:44.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.957+0000 7f52fe800700 1 -- 172.21.15.123:0/902085868 wait complete. 2024-03-30T05:01:44.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.958+0000 7f52fe800700 1 Processor -- start 2024-03-30T05:01:44.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.958+0000 7f52fe800700 1 -- start start 2024-03-30T05:01:44.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.958+0000 7f52fe800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f52f8070300 0x7f52f8115c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:01:44.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.958+0000 7f52fe800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52f8070c10 0x7f52f81161d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:01:44.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.958+0000 7f52fe800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f8116820 con 0x7f52f8070c10 2024-03-30T05:01:44.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.958+0000 7f52fe800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f810fbf0 con 0x7f52f8070300 2024-03-30T05:01:44.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.958+0000 7f52f6a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52f8070c10 0x7f52f81161d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:01:44.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.958+0000 7f52f6a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52f8070c10 0x7f52f81161d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40994/0 (socket says 172.21.15.123:40994) 2024-03-30T05:01:44.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.958+0000 7f52f6a00700 1 -- 172.21.15.123:0/915935221 learned_addr learned my addr 172.21.15.123:0/915935221 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:01:44.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.959+0000 7f52f7400700 1 --2- 172.21.15.123:0/915935221 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f52f8070300 0x7f52f8115c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:01:44.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.959+0000 7f52f6a00700 1 -- 172.21.15.123:0/915935221 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f52f8070300 msgr2=0x7f52f8115c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:44.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.959+0000 7f52f6a00700 1 --2- 172.21.15.123:0/915935221 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f52f8070300 0x7f52f8115c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:44.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.959+0000 7f52f6a00700 1 -- 172.21.15.123:0/915935221 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52e8005040 con 0x7f52f8070c10 2024-03-30T05:01:44.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.959+0000 7f52f6a00700 1 --2- 172.21.15.123:0/915935221 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52f8070c10 0x7f52f81161d0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f52e4002a50 tx=0x7f52e40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:01:44.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.959+0000 7f52ef400700 1 -- 172.21.15.123:0/915935221 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f52e4007cc0 con 0x7f52f8070c10 2024-03-30T05:01:44.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.959+0000 7f52ef400700 1 -- 172.21.15.123:0/915935221 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f52e400f040 con 0x7f52f8070c10 2024-03-30T05:01:44.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.959+0000 7f52fe800700 1 -- 172.21.15.123:0/915935221 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52f810fed0 con 0x7f52f8070c10 2024-03-30T05:01:44.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.959+0000 7f52ef400700 1 -- 172.21.15.123:0/915935221 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f52e4009540 con 0x7f52f8070c10 2024-03-30T05:01:44.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.959+0000 7f52fe800700 1 -- 172.21.15.123:0/915935221 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52f8110410 con 0x7f52f8070c10 2024-03-30T05:01:44.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.960+0000 7f52fe800700 1 -- 172.21.15.123:0/915935221 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52f8071a40 con 0x7f52f8070c10 2024-03-30T05:01:44.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.961+0000 7f52ef400700 1 -- 172.21.15.123:0/915935221 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f52e40096a0 con 0x7f52f8070c10 2024-03-30T05:01:44.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.962+0000 7f52ef400700 1 --2- 172.21.15.123:0/915935221 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f52dc06cc30 0x7f52dc06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:01:44.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.962+0000 7f52ef400700 1 -- 172.21.15.123:0/915935221 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f52e408bee0 con 0x7f52f8070c10 2024-03-30T05:01:44.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.963+0000 7f52f7400700 1 --2- 172.21.15.123:0/915935221 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f52dc06cc30 0x7f52dc06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:01:44.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.963+0000 7f52f7400700 1 --2- 172.21.15.123:0/915935221 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f52dc06cc30 0x7f52dc06f0e0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f52e8009fd0 tx=0x7f52e8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:01:44.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:44.967+0000 7f52ef400700 1 -- 172.21.15.123:0/915935221 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f52e40598b0 con 0x7f52f8070c10 2024-03-30T05:01:45.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.215+0000 7f52fe800700 1 -- 172.21.15.123:0/915935221 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f52f805f5f0 con 0x7f52f8070c10 2024-03-30T05:01:45.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.216+0000 7f52ef400700 1 -- 172.21.15.123:0/915935221 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 34 v34) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f52e4059440 con 0x7f52f8070c10 2024-03-30T05:01:45.214 INFO:teuthology.orchestra.run.smithi123.stdout:e34 2024-03-30T05:01:45.214 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T05:01:45.215 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T05:01:45.215 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T05:01:45.215 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:01:45.215 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T05:01:45.215 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T05:01:45.215 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T05:01:45.215 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T05:01:45.215 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T05:01:45.215 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T05:01:45.215 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T05:01:45.215 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T05:01:45.215 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T05:01:45.215 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T05:01:45.215 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T05:01:45.215 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T05:01:45.215 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T05:01:45.216 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T05:01:45.216 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T05:01:45.216 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T05:01:45.216 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T05:01:45.216 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T05:01:45.216 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T05:01:45.216 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T05:01:45.216 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T05:01:45.216 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T05:01:45.216 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T05:01:45.216 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T05:01:45.216 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T05:01:45.216 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T05:01:45.216 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T05:01:45.217 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T05:01:45.217 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:01:45.217 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:01:45.217 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T05:01:45.217 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:01:45.217 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:01:45.217 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:01:45.217 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:01:45.217 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44975} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/3788988014,v1:172.21.15.135:6827/3788988014] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:01:45.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.218+0000 7f52fe800700 1 -- 172.21.15.123:0/915935221 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f52dc06cc30 msgr2=0x7f52dc06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:45.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.218+0000 7f52fe800700 1 --2- 172.21.15.123:0/915935221 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f52dc06cc30 0x7f52dc06f0e0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f52e8009fd0 tx=0x7f52e8009e30 comp rx=0 tx=0).stop 2024-03-30T05:01:45.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.218+0000 7f52fe800700 1 -- 172.21.15.123:0/915935221 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52f8070c10 msgr2=0x7f52f81161d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:45.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.218+0000 7f52fe800700 1 --2- 172.21.15.123:0/915935221 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52f8070c10 0x7f52f81161d0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f52e4002a50 tx=0x7f52e40046b0 comp rx=0 tx=0).stop 2024-03-30T05:01:45.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.218+0000 7f52fe800700 1 -- 172.21.15.123:0/915935221 shutdown_connections 2024-03-30T05:01:45.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.218+0000 7f52fe800700 1 --2- 172.21.15.123:0/915935221 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f52dc06cc30 0x7f52dc06f0e0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:45.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.218+0000 7f52fe800700 1 --2- 172.21.15.123:0/915935221 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f52f8070300 0x7f52f8115c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:45.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.218+0000 7f52fe800700 1 --2- 172.21.15.123:0/915935221 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52f8070c10 0x7f52f81161d0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:45.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.218+0000 7f52fe800700 1 -- 172.21.15.123:0/915935221 >> 172.21.15.123:0/915935221 conn(0x7f52f80f89d0 msgr2=0x7f52f80f9fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:01:45.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.218+0000 7f52fe800700 1 -- 172.21.15.123:0/915935221 shutdown_connections 2024-03-30T05:01:45.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.218+0000 7f52fe800700 1 -- 172.21.15.123:0/915935221 wait complete. 2024-03-30T05:01:45.219 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 34 2024-03-30T05:01:45.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:01:44 smithi135 ceph-mon[106011]: pgmap v917: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:45.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:01:44 smithi135 ceph-mon[106011]: from='client.35664 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:01:45.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:01:44 smithi135 ceph-mon[106011]: from='client.35668 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:01:45.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:01:44 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:01:45.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:01:44 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/704179122' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:01:45.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.355+0000 7fbd02a00700 1 -- 172.21.15.123:0/1219299825 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbcfc1037a0 msgr2=0x7fbcfc0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:45.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.355+0000 7fbd02a00700 1 --2- 172.21.15.123:0/1219299825 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbcfc1037a0 0x7fbcfc0fce10 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fbce8004940 tx=0x7fbce8009e30 comp rx=0 tx=0).stop 2024-03-30T05:01:45.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.355+0000 7fbd02a00700 1 -- 172.21.15.123:0/1219299825 shutdown_connections 2024-03-30T05:01:45.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.355+0000 7fbd02a00700 1 --2- 172.21.15.123:0/1219299825 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbcfc1037a0 0x7fbcfc0fce10 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:45.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.355+0000 7fbd02a00700 1 --2- 172.21.15.123:0/1219299825 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbcfc102e00 0x7fbcfc1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:45.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.355+0000 7fbd02a00700 1 -- 172.21.15.123:0/1219299825 >> 172.21.15.123:0/1219299825 conn(0x7fbcfc0f89b0 msgr2=0x7fbcfc0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:01:45.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.355+0000 7fbd02a00700 1 -- 172.21.15.123:0/1219299825 shutdown_connections 2024-03-30T05:01:45.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.355+0000 7fbd02a00700 1 -- 172.21.15.123:0/1219299825 wait complete. 2024-03-30T05:01:45.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.356+0000 7fbd02a00700 1 Processor -- start 2024-03-30T05:01:45.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.356+0000 7fbd02a00700 1 -- start start 2024-03-30T05:01:45.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.356+0000 7fbd02a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbcfc102e00 0x7fbcfc115da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:01:45.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.356+0000 7fbd02a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbcfc1037a0 0x7fbcfc1162e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:01:45.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.356+0000 7fbd02a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcfc116930 con 0x7fbcfc1037a0 2024-03-30T05:01:45.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.356+0000 7fbd02a00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcfc10fd00 con 0x7fbcfc102e00 2024-03-30T05:01:45.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.357+0000 7fbcfb400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbcfc1037a0 0x7fbcfc1162e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:01:45.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.357+0000 7fbcfb400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbcfc1037a0 0x7fbcfc1162e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41014/0 (socket says 172.21.15.123:41014) 2024-03-30T05:01:45.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.357+0000 7fbcfb400700 1 -- 172.21.15.123:0/3458011295 learned_addr learned my addr 172.21.15.123:0/3458011295 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:01:45.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.357+0000 7fbcfbe00700 1 --2- 172.21.15.123:0/3458011295 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbcfc102e00 0x7fbcfc115da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:01:45.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.357+0000 7fbcfb400700 1 -- 172.21.15.123:0/3458011295 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbcfc102e00 msgr2=0x7fbcfc115da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:45.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.357+0000 7fbcfb400700 1 --2- 172.21.15.123:0/3458011295 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbcfc102e00 0x7fbcfc115da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:45.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.357+0000 7fbcfb400700 1 -- 172.21.15.123:0/3458011295 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbce8005040 con 0x7fbcfc1037a0 2024-03-30T05:01:45.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.357+0000 7fbcfbe00700 1 --2- 172.21.15.123:0/3458011295 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbcfc102e00 0x7fbcfc115da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T05:01:45.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.357+0000 7fbcfb400700 1 --2- 172.21.15.123:0/3458011295 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbcfc1037a0 0x7fbcfc1162e0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fbce8009390 tx=0x7fbce8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:01:45.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.357+0000 7fbcf3e00700 1 -- 172.21.15.123:0/3458011295 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbce800e040 con 0x7fbcfc1037a0 2024-03-30T05:01:45.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.358+0000 7fbcf3e00700 1 -- 172.21.15.123:0/3458011295 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbce8013420 con 0x7fbcfc1037a0 2024-03-30T05:01:45.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.358+0000 7fbd02a00700 1 -- 172.21.15.123:0/3458011295 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcfc10ff80 con 0x7fbcfc1037a0 2024-03-30T05:01:45.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.358+0000 7fbcf3e00700 1 -- 172.21.15.123:0/3458011295 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbce801c630 con 0x7fbcfc1037a0 2024-03-30T05:01:45.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.358+0000 7fbd02a00700 1 -- 172.21.15.123:0/3458011295 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcfc1104c0 con 0x7fbcfc1037a0 2024-03-30T05:01:45.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.359+0000 7fbd02a00700 1 -- 172.21.15.123:0/3458011295 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcfc0fe550 con 0x7fbcfc1037a0 2024-03-30T05:01:45.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.360+0000 7fbcf3e00700 1 -- 172.21.15.123:0/3458011295 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fbce80139e0 con 0x7fbcfc1037a0 2024-03-30T05:01:45.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.360+0000 7fbcf3e00700 1 --2- 172.21.15.123:0/3458011295 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbce006cc30 0x7fbce006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:01:45.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.361+0000 7fbcf3e00700 1 -- 172.21.15.123:0/3458011295 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fbce808d360 con 0x7fbcfc1037a0 2024-03-30T05:01:45.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.361+0000 7fbcfbe00700 1 --2- 172.21.15.123:0/3458011295 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbce006cc30 0x7fbce006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:01:45.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.361+0000 7fbcfbe00700 1 --2- 172.21.15.123:0/3458011295 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbce006cc30 0x7fbce006f0e0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fbcec007fd0 tx=0x7fbcec009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:01:45.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.365+0000 7fbcf3e00700 1 -- 172.21.15.123:0/3458011295 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fbce805adb0 con 0x7fbcfc1037a0 2024-03-30T05:01:45.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:01:44 smithi123 ceph-mon[139369]: pgmap v917: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:45.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:01:44 smithi123 ceph-mon[139369]: from='client.35664 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:01:45.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:01:44 smithi123 ceph-mon[139369]: from='client.35668 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:01:45.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:01:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:01:45.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:01:44 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/704179122' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:01:45.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.588+0000 7fbd02a00700 1 -- 172.21.15.123:0/3458011295 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbcfc111460 con 0x7fbce006cc30 2024-03-30T05:01:45.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.589+0000 7fbcf3e00700 1 -- 172.21.15.123:0/3458011295 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fbcfc111460 con 0x7fbce006cc30 2024-03-30T05:01:45.587 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T05:01:45.587 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T05:01:45.588 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T05:01:45.588 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T05:01:45.588 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T05:01:45.588 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T05:01:45.588 INFO:teuthology.orchestra.run.smithi123.stdout: "mds", 2024-03-30T05:01:45.588 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T05:01:45.589 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T05:01:45.589 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T05:01:45.589 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T05:01:45.589 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "16/21 daemons upgraded", 2024-03-30T05:01:45.589 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T05:01:45.589 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T05:01:45.589 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T05:01:45.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.591+0000 7fbd02a00700 1 -- 172.21.15.123:0/3458011295 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbce006cc30 msgr2=0x7fbce006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:45.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.591+0000 7fbd02a00700 1 --2- 172.21.15.123:0/3458011295 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbce006cc30 0x7fbce006f0e0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fbcec007fd0 tx=0x7fbcec009040 comp rx=0 tx=0).stop 2024-03-30T05:01:45.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.591+0000 7fbd02a00700 1 -- 172.21.15.123:0/3458011295 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbcfc1037a0 msgr2=0x7fbcfc1162e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:45.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.591+0000 7fbd02a00700 1 --2- 172.21.15.123:0/3458011295 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbcfc1037a0 0x7fbcfc1162e0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fbce8009390 tx=0x7fbce8009e30 comp rx=0 tx=0).stop 2024-03-30T05:01:45.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.591+0000 7fbd02a00700 1 -- 172.21.15.123:0/3458011295 shutdown_connections 2024-03-30T05:01:45.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.591+0000 7fbd02a00700 1 --2- 172.21.15.123:0/3458011295 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbce006cc30 0x7fbce006f0e0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:45.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.591+0000 7fbd02a00700 1 --2- 172.21.15.123:0/3458011295 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbcfc102e00 0x7fbcfc115da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:45.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.591+0000 7fbd02a00700 1 --2- 172.21.15.123:0/3458011295 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbcfc1037a0 0x7fbcfc1162e0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:45.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.591+0000 7fbd02a00700 1 -- 172.21.15.123:0/3458011295 >> 172.21.15.123:0/3458011295 conn(0x7fbcfc0f89b0 msgr2=0x7fbcfc0f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:01:45.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.592+0000 7fbd02a00700 1 -- 172.21.15.123:0/3458011295 shutdown_connections 2024-03-30T05:01:45.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.592+0000 7fbd02a00700 1 -- 172.21.15.123:0/3458011295 wait complete. 2024-03-30T05:01:45.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.725+0000 7f34c8600700 1 -- 172.21.15.123:0/890635794 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34c0102e00 msgr2=0x7f34c01031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:45.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.725+0000 7f34c8600700 1 --2- 172.21.15.123:0/890635794 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34c0102e00 0x7f34c01031d0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f34b4004940 tx=0x7f34b4009e30 comp rx=0 tx=0).stop 2024-03-30T05:01:45.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.725+0000 7f34c8600700 1 -- 172.21.15.123:0/890635794 shutdown_connections 2024-03-30T05:01:45.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.725+0000 7f34c8600700 1 --2- 172.21.15.123:0/890635794 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f34c01037a0 0x7f34c00fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:45.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.725+0000 7f34c8600700 1 --2- 172.21.15.123:0/890635794 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34c0102e00 0x7f34c01031d0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:45.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.726+0000 7f34c8600700 1 -- 172.21.15.123:0/890635794 >> 172.21.15.123:0/890635794 conn(0x7f34c00f89b0 msgr2=0x7f34c00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:01:45.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.726+0000 7f34c8600700 1 -- 172.21.15.123:0/890635794 shutdown_connections 2024-03-30T05:01:45.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.726+0000 7f34c8600700 1 -- 172.21.15.123:0/890635794 wait complete. 2024-03-30T05:01:45.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.727+0000 7f34c8600700 1 Processor -- start 2024-03-30T05:01:45.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.727+0000 7f34c8600700 1 -- start start 2024-03-30T05:01:45.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.728+0000 7f34c8600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34c0102e00 0x7f34c0113a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:01:45.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.728+0000 7f34c8600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f34c01037a0 0x7f34c0113f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:01:45.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.728+0000 7f34c8600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c01145d0 con 0x7f34c0102e00 2024-03-30T05:01:45.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.728+0000 7f34c8600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c010d9c0 con 0x7f34c01037a0 2024-03-30T05:01:45.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.728+0000 7f34c5c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34c0102e00 0x7f34c0113a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:01:45.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.728+0000 7f34c5c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34c0102e00 0x7f34c0113a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41044/0 (socket says 172.21.15.123:41044) 2024-03-30T05:01:45.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.728+0000 7f34c5c00700 1 -- 172.21.15.123:0/2257075154 learned_addr learned my addr 172.21.15.123:0/2257075154 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:01:45.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.728+0000 7f34c5200700 1 --2- 172.21.15.123:0/2257075154 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f34c01037a0 0x7f34c0113f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:01:45.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.728+0000 7f34c5c00700 1 -- 172.21.15.123:0/2257075154 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f34c01037a0 msgr2=0x7f34c0113f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:45.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.728+0000 7f34c5c00700 1 --2- 172.21.15.123:0/2257075154 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f34c01037a0 0x7f34c0113f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:45.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.728+0000 7f34c5c00700 1 -- 172.21.15.123:0/2257075154 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34b4005040 con 0x7f34c0102e00 2024-03-30T05:01:45.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.728+0000 7f34c5c00700 1 --2- 172.21.15.123:0/2257075154 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34c0102e00 0x7f34c0113a40 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f34b4007f50 tx=0x7f34b4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:01:45.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.729+0000 7f34bd600700 1 -- 172.21.15.123:0/2257075154 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f34b4021070 con 0x7f34c0102e00 2024-03-30T05:01:45.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.729+0000 7f34bd600700 1 -- 172.21.15.123:0/2257075154 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f34b4013e00 con 0x7f34c0102e00 2024-03-30T05:01:45.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.729+0000 7f34c8600700 1 -- 172.21.15.123:0/2257075154 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34c010dc40 con 0x7f34c0102e00 2024-03-30T05:01:45.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.729+0000 7f34bd600700 1 -- 172.21.15.123:0/2257075154 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f34b400e070 con 0x7f34c0102e00 2024-03-30T05:01:45.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.729+0000 7f34c8600700 1 -- 172.21.15.123:0/2257075154 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34c010e200 con 0x7f34c0102e00 2024-03-30T05:01:45.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.730+0000 7f34c8600700 1 -- 172.21.15.123:0/2257075154 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34c00fe550 con 0x7f34c0102e00 2024-03-30T05:01:45.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.731+0000 7f34bd600700 1 -- 172.21.15.123:0/2257075154 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f34b4013950 con 0x7f34c0102e00 2024-03-30T05:01:45.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.732+0000 7f34bd600700 1 --2- 172.21.15.123:0/2257075154 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f34a806cc30 0x7f34a806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:01:45.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.732+0000 7f34bd600700 1 -- 172.21.15.123:0/2257075154 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f34b408d350 con 0x7f34c0102e00 2024-03-30T05:01:45.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.732+0000 7f34c5200700 1 --2- 172.21.15.123:0/2257075154 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f34a806cc30 0x7f34a806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:01:45.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.733+0000 7f34c5200700 1 --2- 172.21.15.123:0/2257075154 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f34a806cc30 0x7f34a806f0e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f34ac004600 tx=0x7f34ac00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:01:45.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:45.737+0000 7f34bd600700 1 -- 172.21.15.123:0/2257075154 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f34b405ada0 con 0x7f34c0102e00 2024-03-30T05:01:46.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:46.018+0000 7f34c8600700 1 -- 172.21.15.123:0/2257075154 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f34c0109e50 con 0x7f34c0102e00 2024-03-30T05:01:46.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:46.019+0000 7f34bd600700 1 -- 172.21.15.123:0/2257075154 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f34b405a930 con 0x7f34c0102e00 2024-03-30T05:01:46.018 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T05:01:46.018 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-30T05:01:46.018 INFO:teuthology.orchestra.run.smithi123.stdout: host smithi135 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi135/config 2024-03-30T05:01:46.018 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-30T05:01:46.018 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T05:01:46.018 INFO:teuthology.orchestra.run.smithi123.stdout:Traceback (most recent call last): 2024-03-30T05:01:46.018 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-30T05:01:46.018 INFO:teuthology.orchestra.run.smithi123.stdout: "__main__", mod_spec) 2024-03-30T05:01:46.018 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-30T05:01:46.019 INFO:teuthology.orchestra.run.smithi123.stdout: exec(code, run_globals) 2024-03-30T05:01:46.019 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10889, in 2024-03-30T05:01:46.019 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10877, in main 2024-03-30T05:01:46.019 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-30T05:01:46.019 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-30T05:01:46.019 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-30T05:01:46.019 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-30T05:01:46.019 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-03-30T05:01:46.019 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-30T05:01:46.019 INFO:teuthology.orchestra.run.smithi123.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T05:01:46.019 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T05:01:46.020 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T05:01:46.020 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T05:01:46.020 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T05:01:46.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:46.021+0000 7f34c8600700 1 -- 172.21.15.123:0/2257075154 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f34a806cc30 msgr2=0x7f34a806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:46.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:46.021+0000 7f34c8600700 1 --2- 172.21.15.123:0/2257075154 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f34a806cc30 0x7f34a806f0e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f34ac004600 tx=0x7f34ac00b040 comp rx=0 tx=0).stop 2024-03-30T05:01:46.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:46.021+0000 7f34c8600700 1 -- 172.21.15.123:0/2257075154 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34c0102e00 msgr2=0x7f34c0113a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:01:46.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:46.021+0000 7f34c8600700 1 --2- 172.21.15.123:0/2257075154 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34c0102e00 0x7f34c0113a40 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f34b4007f50 tx=0x7f34b4009e30 comp rx=0 tx=0).stop 2024-03-30T05:01:46.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:46.021+0000 7f34c8600700 1 -- 172.21.15.123:0/2257075154 shutdown_connections 2024-03-30T05:01:46.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:46.021+0000 7f34c8600700 1 --2- 172.21.15.123:0/2257075154 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f34a806cc30 0x7f34a806f0e0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:46.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:46.021+0000 7f34c8600700 1 --2- 172.21.15.123:0/2257075154 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34c0102e00 0x7f34c0113a40 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:46.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:46.021+0000 7f34c8600700 1 --2- 172.21.15.123:0/2257075154 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f34c01037a0 0x7f34c0113f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:01:46.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:46.021+0000 7f34c8600700 1 -- 172.21.15.123:0/2257075154 >> 172.21.15.123:0/2257075154 conn(0x7f34c00f89b0 msgr2=0x7f34c00f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:01:46.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:46.021+0000 7f34c8600700 1 -- 172.21.15.123:0/2257075154 shutdown_connections 2024-03-30T05:01:46.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:01:46.022+0000 7f34c8600700 1 -- 172.21.15.123:0/2257075154 wait complete. 2024-03-30T05:01:46.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:01:45 smithi135 ceph-mon[106011]: from='client.35672 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:01:46.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:01:45 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/915935221' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T05:01:46.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:01:45 smithi123 ceph-mon[139369]: from='client.35672 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:01:46.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:01:45 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/915935221' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T05:01:47.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:01:46 smithi135 ceph-mon[106011]: from='client.35684 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:01:47.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:01:46 smithi135 ceph-mon[106011]: pgmap v918: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:47.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:01:46 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2257075154' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T05:01:47.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:01:46 smithi123 ceph-mon[139369]: from='client.35684 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:01:47.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:01:46 smithi123 ceph-mon[139369]: pgmap v918: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:47.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:01:46 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2257075154' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T05:01:49.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:01:48 smithi135 ceph-mon[106011]: pgmap v919: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:49.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:01:48 smithi123 ceph-mon[139369]: pgmap v919: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:51.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:01:50 smithi135 ceph-mon[106011]: pgmap v920: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:51.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:01:50 smithi123 ceph-mon[139369]: pgmap v920: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:53.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:01:52 smithi135 ceph-mon[106011]: pgmap v921: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:53.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:01:52 smithi123 ceph-mon[139369]: pgmap v921: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:55.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:01:55 smithi135 ceph-mon[106011]: pgmap v922: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:55.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:01:54 smithi123 ceph-mon[139369]: pgmap v922: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:57.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:01:57 smithi135 ceph-mon[106011]: pgmap v923: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:57.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:01:57 smithi123 ceph-mon[139369]: pgmap v923: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:59.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:01:59 smithi135 ceph-mon[106011]: pgmap v924: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:01:59.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:01:59 smithi123 ceph-mon[139369]: pgmap v924: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:00.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:02:00 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:02:00.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:02:00 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:02:01.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:02:01 smithi135 ceph-mon[106011]: pgmap v925: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:01.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:02:01 smithi123 ceph-mon[139369]: pgmap v925: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:03.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:02:03 smithi135 ceph-mon[106011]: pgmap v926: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:03.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:02:03 smithi123 ceph-mon[139369]: pgmap v926: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:05.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:02:05 smithi135 ceph-mon[106011]: pgmap v927: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:05.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:02:05 smithi123 ceph-mon[139369]: pgmap v927: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:07.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:02:07 smithi135 ceph-mon[106011]: pgmap v928: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:07.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:02:07 smithi123 ceph-mon[139369]: pgmap v928: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:09.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:02:09 smithi135 ceph-mon[106011]: pgmap v929: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:09.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:02:09 smithi123 ceph-mon[139369]: pgmap v929: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:11.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:02:11 smithi135 ceph-mon[106011]: pgmap v930: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:11.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:02:11 smithi123 ceph-mon[139369]: pgmap v930: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:13.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:02:13 smithi135 ceph-mon[106011]: pgmap v931: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:13.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:02:13 smithi123 ceph-mon[139369]: pgmap v931: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:15.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:02:15 smithi135 ceph-mon[106011]: pgmap v932: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:15.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:02:15 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:02:15.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:02:15 smithi123 ceph-mon[139369]: pgmap v932: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:15.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:02:15 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:02:16.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.172+0000 7f9511e00700 1 -- 172.21.15.123:0/622552968 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f950c1037a0 msgr2=0x7f950c0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:16.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.172+0000 7f9511e00700 1 --2- 172.21.15.123:0/622552968 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f950c1037a0 0x7f950c0fce10 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f94f8004940 tx=0x7f94f8009e30 comp rx=0 tx=0).stop 2024-03-30T05:02:16.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.172+0000 7f9511e00700 1 -- 172.21.15.123:0/622552968 shutdown_connections 2024-03-30T05:02:16.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.172+0000 7f9511e00700 1 --2- 172.21.15.123:0/622552968 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f950c1037a0 0x7f950c0fce10 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:16.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.172+0000 7f9511e00700 1 --2- 172.21.15.123:0/622552968 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f950c102e00 0x7f950c1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:16.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.172+0000 7f9511e00700 1 -- 172.21.15.123:0/622552968 >> 172.21.15.123:0/622552968 conn(0x7f950c0f89b0 msgr2=0x7f950c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:02:16.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.172+0000 7f9511e00700 1 -- 172.21.15.123:0/622552968 shutdown_connections 2024-03-30T05:02:16.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.172+0000 7f9511e00700 1 -- 172.21.15.123:0/622552968 wait complete. 2024-03-30T05:02:16.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.173+0000 7f9511e00700 1 Processor -- start 2024-03-30T05:02:16.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.173+0000 7f9511e00700 1 -- start start 2024-03-30T05:02:16.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.174+0000 7f9511e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f950c102e00 0x7f950c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:02:16.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.174+0000 7f9511e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f950c1037a0 0x7f950c070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:02:16.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.174+0000 7f9511e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950c070d90 con 0x7f950c102e00 2024-03-30T05:02:16.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.174+0000 7f9511e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950c070ed0 con 0x7f950c1037a0 2024-03-30T05:02:16.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.174+0000 7f950aa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f950c102e00 0x7f950c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:02:16.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.174+0000 7f950aa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f950c102e00 0x7f950c0702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48502/0 (socket says 172.21.15.123:48502) 2024-03-30T05:02:16.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.174+0000 7f950aa00700 1 -- 172.21.15.123:0/3751825786 learned_addr learned my addr 172.21.15.123:0/3751825786 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:02:16.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.174+0000 7f950aa00700 1 -- 172.21.15.123:0/3751825786 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f950c1037a0 msgr2=0x7f950c070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:16.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.174+0000 7f950aa00700 1 --2- 172.21.15.123:0/3751825786 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f950c1037a0 0x7f950c070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:16.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.174+0000 7f950aa00700 1 -- 172.21.15.123:0/3751825786 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f8005040 con 0x7f950c102e00 2024-03-30T05:02:16.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.175+0000 7f950aa00700 1 --2- 172.21.15.123:0/3751825786 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f950c102e00 0x7f950c0702c0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f94fc002a80 tx=0x7f94fc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:02:16.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.175+0000 7f9502a00700 1 -- 172.21.15.123:0/3751825786 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f94fc007cc0 con 0x7f950c102e00 2024-03-30T05:02:16.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.175+0000 7f9502a00700 1 -- 172.21.15.123:0/3751825786 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f94fc00f040 con 0x7f950c102e00 2024-03-30T05:02:16.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.175+0000 7f9511e00700 1 -- 172.21.15.123:0/3751825786 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f950c119d70 con 0x7f950c102e00 2024-03-30T05:02:16.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.175+0000 7f9502a00700 1 -- 172.21.15.123:0/3751825786 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f94fc009640 con 0x7f950c102e00 2024-03-30T05:02:16.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.175+0000 7f9511e00700 1 -- 172.21.15.123:0/3751825786 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f950c11a1f0 con 0x7f950c102e00 2024-03-30T05:02:16.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.176+0000 7f9511e00700 1 -- 172.21.15.123:0/3751825786 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f950c0fe550 con 0x7f950c102e00 2024-03-30T05:02:16.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.177+0000 7f9502a00700 1 -- 172.21.15.123:0/3751825786 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f94fc0097a0 con 0x7f950c102e00 2024-03-30T05:02:16.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.178+0000 7f9502a00700 1 --2- 172.21.15.123:0/3751825786 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f94f006cc20 0x7f94f006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:02:16.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.178+0000 7f9502a00700 1 -- 172.21.15.123:0/3751825786 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f94fc08b7e0 con 0x7f950c102e00 2024-03-30T05:02:16.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.178+0000 7f950a000700 1 --2- 172.21.15.123:0/3751825786 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f94f006cc20 0x7f94f006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:02:16.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.179+0000 7f950a000700 1 --2- 172.21.15.123:0/3751825786 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f94f006cc20 0x7f94f006f0d0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f94f8004f90 tx=0x7f94f8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:02:16.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.183+0000 7f9502a00700 1 -- 172.21.15.123:0/3751825786 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f94fc0557a0 con 0x7f950c102e00 2024-03-30T05:02:16.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.403+0000 7f9511e00700 1 -- 172.21.15.123:0/3751825786 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f950c04bed0 con 0x7f94f006cc20 2024-03-30T05:02:16.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.404+0000 7f9502a00700 1 -- 172.21.15.123:0/3751825786 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f950c04bed0 con 0x7f94f006cc20 2024-03-30T05:02:16.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.406+0000 7f9511e00700 1 -- 172.21.15.123:0/3751825786 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f94f006cc20 msgr2=0x7f94f006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:16.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.406+0000 7f9511e00700 1 --2- 172.21.15.123:0/3751825786 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f94f006cc20 0x7f94f006f0d0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f94f8004f90 tx=0x7f94f8009e30 comp rx=0 tx=0).stop 2024-03-30T05:02:16.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.406+0000 7f9511e00700 1 -- 172.21.15.123:0/3751825786 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f950c102e00 msgr2=0x7f950c0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:16.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.406+0000 7f9511e00700 1 --2- 172.21.15.123:0/3751825786 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f950c102e00 0x7f950c0702c0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f94fc002a80 tx=0x7f94fc0046b0 comp rx=0 tx=0).stop 2024-03-30T05:02:16.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.407+0000 7f9511e00700 1 -- 172.21.15.123:0/3751825786 shutdown_connections 2024-03-30T05:02:16.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.407+0000 7f9511e00700 1 --2- 172.21.15.123:0/3751825786 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f94f006cc20 0x7f94f006f0d0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:16.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.407+0000 7f9511e00700 1 --2- 172.21.15.123:0/3751825786 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f950c102e00 0x7f950c0702c0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:16.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.407+0000 7f9511e00700 1 --2- 172.21.15.123:0/3751825786 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f950c1037a0 0x7f950c070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:16.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.407+0000 7f9511e00700 1 -- 172.21.15.123:0/3751825786 >> 172.21.15.123:0/3751825786 conn(0x7f950c0f89b0 msgr2=0x7f950c0f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:02:16.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.407+0000 7f9511e00700 1 -- 172.21.15.123:0/3751825786 shutdown_connections 2024-03-30T05:02:16.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.407+0000 7f9511e00700 1 -- 172.21.15.123:0/3751825786 wait complete. 2024-03-30T05:02:16.419 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T05:02:16.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.541+0000 7f6914600700 1 -- 172.21.15.123:0/2396090948 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f690c1037a0 msgr2=0x7f690c0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:16.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.541+0000 7f6914600700 1 --2- 172.21.15.123:0/2396090948 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f690c1037a0 0x7f690c0fce10 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f68f8004940 tx=0x7f68f8009e30 comp rx=0 tx=0).stop 2024-03-30T05:02:16.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.541+0000 7f6914600700 1 -- 172.21.15.123:0/2396090948 shutdown_connections 2024-03-30T05:02:16.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.541+0000 7f6914600700 1 --2- 172.21.15.123:0/2396090948 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f690c1037a0 0x7f690c0fce10 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:16.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.541+0000 7f6914600700 1 --2- 172.21.15.123:0/2396090948 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f690c102e00 0x7f690c1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:16.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.541+0000 7f6914600700 1 -- 172.21.15.123:0/2396090948 >> 172.21.15.123:0/2396090948 conn(0x7f690c0f89b0 msgr2=0x7f690c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:02:16.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.541+0000 7f6914600700 1 -- 172.21.15.123:0/2396090948 shutdown_connections 2024-03-30T05:02:16.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.541+0000 7f6914600700 1 -- 172.21.15.123:0/2396090948 wait complete. 2024-03-30T05:02:16.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.542+0000 7f6914600700 1 Processor -- start 2024-03-30T05:02:16.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.542+0000 7f6914600700 1 -- start start 2024-03-30T05:02:16.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.542+0000 7f6914600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f690c102e00 0x7f690c10af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:02:16.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.542+0000 7f6914600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f690c1037a0 0x7f690c10b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:02:16.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.542+0000 7f6914600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f690c10ba10 con 0x7f690c1037a0 2024-03-30T05:02:16.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.542+0000 7f6914600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f690c10bb50 con 0x7f690c102e00 2024-03-30T05:02:16.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.543+0000 7f6911200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f690c1037a0 0x7f690c10b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:02:16.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.543+0000 7f6911200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f690c1037a0 0x7f690c10b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48514/0 (socket says 172.21.15.123:48514) 2024-03-30T05:02:16.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.543+0000 7f6911200700 1 -- 172.21.15.123:0/3352348186 learned_addr learned my addr 172.21.15.123:0/3352348186 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:02:16.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.543+0000 7f6911c00700 1 --2- 172.21.15.123:0/3352348186 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f690c102e00 0x7f690c10af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:02:16.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.543+0000 7f6911200700 1 -- 172.21.15.123:0/3352348186 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f690c102e00 msgr2=0x7f690c10af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:16.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.543+0000 7f6911200700 1 --2- 172.21.15.123:0/3352348186 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f690c102e00 0x7f690c10af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:16.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.543+0000 7f6911200700 1 -- 172.21.15.123:0/3352348186 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68f8005040 con 0x7f690c1037a0 2024-03-30T05:02:16.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.543+0000 7f6911200700 1 --2- 172.21.15.123:0/3352348186 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f690c1037a0 0x7f690c10b4d0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f68f8009390 tx=0x7f68f8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:02:16.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.544+0000 7f6909600700 1 -- 172.21.15.123:0/3352348186 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f68f800e040 con 0x7f690c1037a0 2024-03-30T05:02:16.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.544+0000 7f6914600700 1 -- 172.21.15.123:0/3352348186 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f690c10bdd0 con 0x7f690c1037a0 2024-03-30T05:02:16.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.544+0000 7f6909600700 1 -- 172.21.15.123:0/3352348186 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f68f8013420 con 0x7f690c1037a0 2024-03-30T05:02:16.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.544+0000 7f6909600700 1 -- 172.21.15.123:0/3352348186 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f68f801c630 con 0x7f690c1037a0 2024-03-30T05:02:16.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.544+0000 7f6914600700 1 -- 172.21.15.123:0/3352348186 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f690c11e6e0 con 0x7f690c1037a0 2024-03-30T05:02:16.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.545+0000 7f6914600700 1 -- 172.21.15.123:0/3352348186 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f690c0fe550 con 0x7f690c1037a0 2024-03-30T05:02:16.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.546+0000 7f6909600700 1 -- 172.21.15.123:0/3352348186 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f68f8013a00 con 0x7f690c1037a0 2024-03-30T05:02:16.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.547+0000 7f6909600700 1 --2- 172.21.15.123:0/3352348186 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f68f406cc30 0x7f68f406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:02:16.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.547+0000 7f6909600700 1 -- 172.21.15.123:0/3352348186 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f68f808d360 con 0x7f690c1037a0 2024-03-30T05:02:16.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.547+0000 7f6911c00700 1 --2- 172.21.15.123:0/3352348186 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f68f406cc30 0x7f68f406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:02:16.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.548+0000 7f6911c00700 1 --2- 172.21.15.123:0/3352348186 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f68f406cc30 0x7f68f406f0e0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f6900006fd0 tx=0x7f6900008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:02:16.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.551+0000 7f6909600700 1 -- 172.21.15.123:0/3352348186 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f68f8057320 con 0x7f690c1037a0 2024-03-30T05:02:16.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.771+0000 7f6914600700 1 -- 172.21.15.123:0/3352348186 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f690c109e80 con 0x7f68f406cc30 2024-03-30T05:02:16.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.773+0000 7f6909600700 1 -- 172.21.15.123:0/3352348186 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f690c109e80 con 0x7f68f406cc30 2024-03-30T05:02:16.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.775+0000 7f6914600700 1 -- 172.21.15.123:0/3352348186 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f68f406cc30 msgr2=0x7f68f406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:16.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.775+0000 7f6914600700 1 --2- 172.21.15.123:0/3352348186 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f68f406cc30 0x7f68f406f0e0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f6900006fd0 tx=0x7f6900008040 comp rx=0 tx=0).stop 2024-03-30T05:02:16.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.775+0000 7f6914600700 1 -- 172.21.15.123:0/3352348186 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f690c1037a0 msgr2=0x7f690c10b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:16.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.775+0000 7f6914600700 1 --2- 172.21.15.123:0/3352348186 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f690c1037a0 0x7f690c10b4d0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f68f8009390 tx=0x7f68f8009e30 comp rx=0 tx=0).stop 2024-03-30T05:02:16.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.775+0000 7f6914600700 1 -- 172.21.15.123:0/3352348186 shutdown_connections 2024-03-30T05:02:16.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.775+0000 7f6914600700 1 --2- 172.21.15.123:0/3352348186 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f68f406cc30 0x7f68f406f0e0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:16.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.775+0000 7f6914600700 1 --2- 172.21.15.123:0/3352348186 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f690c102e00 0x7f690c10af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:16.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.775+0000 7f6914600700 1 --2- 172.21.15.123:0/3352348186 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f690c1037a0 0x7f690c10b4d0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:16.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.775+0000 7f6914600700 1 -- 172.21.15.123:0/3352348186 >> 172.21.15.123:0/3352348186 conn(0x7f690c0f89b0 msgr2=0x7f690c0f9910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:02:16.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.775+0000 7f6914600700 1 -- 172.21.15.123:0/3352348186 shutdown_connections 2024-03-30T05:02:16.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.775+0000 7f6914600700 1 -- 172.21.15.123:0/3352348186 wait complete. 2024-03-30T05:02:16.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.910+0000 7f31f3c00700 1 -- 172.21.15.123:0/3896260192 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31ec102e00 msgr2=0x7f31ec1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:16.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.910+0000 7f31f3c00700 1 --2- 172.21.15.123:0/3896260192 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31ec102e00 0x7f31ec1031d0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f31e0004940 tx=0x7f31e0009e30 comp rx=0 tx=0).stop 2024-03-30T05:02:16.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.910+0000 7f31f3c00700 1 -- 172.21.15.123:0/3896260192 shutdown_connections 2024-03-30T05:02:16.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.910+0000 7f31f3c00700 1 --2- 172.21.15.123:0/3896260192 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31ec1037a0 0x7f31ec0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:16.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.910+0000 7f31f3c00700 1 --2- 172.21.15.123:0/3896260192 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31ec102e00 0x7f31ec1031d0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:16.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.910+0000 7f31f3c00700 1 -- 172.21.15.123:0/3896260192 >> 172.21.15.123:0/3896260192 conn(0x7f31ec0f89b0 msgr2=0x7f31ec0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:02:16.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.910+0000 7f31f3c00700 1 -- 172.21.15.123:0/3896260192 shutdown_connections 2024-03-30T05:02:16.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.910+0000 7f31f3c00700 1 -- 172.21.15.123:0/3896260192 wait complete. 2024-03-30T05:02:16.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.911+0000 7f31f3c00700 1 Processor -- start 2024-03-30T05:02:16.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.911+0000 7f31f3c00700 1 -- start start 2024-03-30T05:02:16.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.912+0000 7f31f3c00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31ec102e00 0x7f31ec115d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:02:16.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.912+0000 7f31f3c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31ec1037a0 0x7f31ec116250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:02:16.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.912+0000 7f31f3c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31ec1168a0 con 0x7f31ec1037a0 2024-03-30T05:02:16.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.912+0000 7f31f3c00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31ec10fc70 con 0x7f31ec102e00 2024-03-30T05:02:16.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.912+0000 7f31ebe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31ec1037a0 0x7f31ec116250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:02:16.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.912+0000 7f31ebe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31ec1037a0 0x7f31ec116250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48532/0 (socket says 172.21.15.123:48532) 2024-03-30T05:02:16.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.912+0000 7f31ebe00700 1 -- 172.21.15.123:0/3001671453 learned_addr learned my addr 172.21.15.123:0/3001671453 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:02:16.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.912+0000 7f31f1200700 1 --2- 172.21.15.123:0/3001671453 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31ec102e00 0x7f31ec115d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:02:16.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.912+0000 7f31ebe00700 1 -- 172.21.15.123:0/3001671453 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31ec102e00 msgr2=0x7f31ec115d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:16.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.912+0000 7f31ebe00700 1 --2- 172.21.15.123:0/3001671453 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31ec102e00 0x7f31ec115d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:16.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.912+0000 7f31ebe00700 1 -- 172.21.15.123:0/3001671453 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31e0005040 con 0x7f31ec1037a0 2024-03-30T05:02:16.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.913+0000 7f31ebe00700 1 --2- 172.21.15.123:0/3001671453 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31ec1037a0 0x7f31ec116250 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f31d8002a50 tx=0x7f31d8004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:02:16.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.913+0000 7f31e8c00700 1 -- 172.21.15.123:0/3001671453 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31d8007cc0 con 0x7f31ec1037a0 2024-03-30T05:02:16.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.913+0000 7f31e8c00700 1 -- 172.21.15.123:0/3001671453 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f31d800f040 con 0x7f31ec1037a0 2024-03-30T05:02:16.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.913+0000 7f31f3c00700 1 -- 172.21.15.123:0/3001671453 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31ec10ff50 con 0x7f31ec1037a0 2024-03-30T05:02:16.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.913+0000 7f31e8c00700 1 -- 172.21.15.123:0/3001671453 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31d800d920 con 0x7f31ec1037a0 2024-03-30T05:02:16.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.913+0000 7f31f3c00700 1 -- 172.21.15.123:0/3001671453 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31ec110510 con 0x7f31ec1037a0 2024-03-30T05:02:16.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.914+0000 7f31f3c00700 1 -- 172.21.15.123:0/3001671453 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31ec0fe550 con 0x7f31ec1037a0 2024-03-30T05:02:16.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.915+0000 7f31e8c00700 1 -- 172.21.15.123:0/3001671453 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f31d8009680 con 0x7f31ec1037a0 2024-03-30T05:02:16.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.916+0000 7f31e8c00700 1 --2- 172.21.15.123:0/3001671453 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f31d406cc30 0x7f31d406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:02:16.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.916+0000 7f31e8c00700 1 -- 172.21.15.123:0/3001671453 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f31d808bdb0 con 0x7f31ec1037a0 2024-03-30T05:02:16.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.916+0000 7f31f1200700 1 --2- 172.21.15.123:0/3001671453 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f31d406cc30 0x7f31d406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:02:16.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.917+0000 7f31f1200700 1 --2- 172.21.15.123:0/3001671453 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f31d406cc30 0x7f31d406f0e0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f31e0009fd0 tx=0x7f31e0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:02:16.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:16.920+0000 7f31e8c00700 1 -- 172.21.15.123:0/3001671453 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f31d8059780 con 0x7f31ec1037a0 2024-03-30T05:02:17.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.135+0000 7f31f3c00700 1 -- 172.21.15.123:0/3001671453 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f31ec109e30 con 0x7f31d406cc30 2024-03-30T05:02:17.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.145+0000 7f31e8c00700 1 -- 172.21.15.123:0/3001671453 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f31ec109e30 con 0x7f31d406cc30 2024-03-30T05:02:17.143 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T05:02:17.143 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (31m) 8m ago 43m 14.6M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T05:02:17.143 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (28m) 8m ago 44m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T05:02:17.143 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (28m) 8m ago 43m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T05:02:17.143 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (31m) 8m ago 43m 73.5M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T05:02:17.143 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (24m) 8m ago 37m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T05:02:17.144 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (24m) 8m ago 37m 16.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T05:02:17.144 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (8m) 8m ago 37m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 87b073a6c9ca 2024-03-30T05:02:17.144 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (24m) 8m ago 37m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T05:02:17.144 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (33m) 8m ago 45m 510M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T05:02:17.144 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (32m) 8m ago 42m 456M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T05:02:17.144 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (29m) 8m ago 45m 70.0M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T05:02:17.144 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (29m) 8m ago 42m 51.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T05:02:17.144 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (32m) 8m ago 44m 8636k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T05:02:17.144 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (32m) 8m ago 42m 8678k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T05:02:17.144 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (28m) 8m ago 41m 107M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T05:02:17.144 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (27m) 8m ago 41m 97.2M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T05:02:17.144 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (27m) 8m ago 40m 89.8M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T05:02:17.144 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (26m) 8m ago 40m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T05:02:17.145 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (25m) 8m ago 39m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T05:02:17.145 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (25m) 8m ago 39m 88.8M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T05:02:17.145 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (31m) 8m ago 43m 63.9M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T05:02:17.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.147+0000 7f31f3c00700 1 -- 172.21.15.123:0/3001671453 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f31d406cc30 msgr2=0x7f31d406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:17.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.147+0000 7f31f3c00700 1 --2- 172.21.15.123:0/3001671453 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f31d406cc30 0x7f31d406f0e0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f31e0009fd0 tx=0x7f31e0009e30 comp rx=0 tx=0).stop 2024-03-30T05:02:17.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.147+0000 7f31f3c00700 1 -- 172.21.15.123:0/3001671453 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31ec1037a0 msgr2=0x7f31ec116250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:17.146 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.147+0000 7f31f3c00700 1 --2- 172.21.15.123:0/3001671453 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31ec1037a0 0x7f31ec116250 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f31d8002a50 tx=0x7f31d8004600 comp rx=0 tx=0).stop 2024-03-30T05:02:17.146 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.147+0000 7f31f3c00700 1 -- 172.21.15.123:0/3001671453 shutdown_connections 2024-03-30T05:02:17.146 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.147+0000 7f31f3c00700 1 --2- 172.21.15.123:0/3001671453 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f31d406cc30 0x7f31d406f0e0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:17.146 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.147+0000 7f31f3c00700 1 --2- 172.21.15.123:0/3001671453 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31ec102e00 0x7f31ec115d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:17.146 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.147+0000 7f31f3c00700 1 --2- 172.21.15.123:0/3001671453 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31ec1037a0 0x7f31ec116250 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:17.146 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.147+0000 7f31f3c00700 1 -- 172.21.15.123:0/3001671453 >> 172.21.15.123:0/3001671453 conn(0x7f31ec0f89b0 msgr2=0x7f31ec0704f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:02:17.146 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.147+0000 7f31f3c00700 1 -- 172.21.15.123:0/3001671453 shutdown_connections 2024-03-30T05:02:17.146 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.147+0000 7f31f3c00700 1 -- 172.21.15.123:0/3001671453 wait complete. 2024-03-30T05:02:17.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.281+0000 7fc2f0c00700 1 -- 172.21.15.123:0/3947342264 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2ec070c70 msgr2=0x7fc2ec108f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:17.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.281+0000 7fc2f0c00700 1 --2- 172.21.15.123:0/3947342264 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2ec070c70 0x7fc2ec108f50 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fc2d8004970 tx=0x7fc2d8009e30 comp rx=0 tx=0).stop 2024-03-30T05:02:17.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.281+0000 7fc2f0c00700 1 -- 172.21.15.123:0/3947342264 shutdown_connections 2024-03-30T05:02:17.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.281+0000 7fc2f0c00700 1 --2- 172.21.15.123:0/3947342264 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2ec070c70 0x7fc2ec108f50 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:17.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.281+0000 7fc2f0c00700 1 --2- 172.21.15.123:0/3947342264 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc2ec070360 0x7fc2ec070730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:17.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.281+0000 7fc2f0c00700 1 -- 172.21.15.123:0/3947342264 >> 172.21.15.123:0/3947342264 conn(0x7fc2ec0f8790 msgr2=0x7fc2ec0faba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:02:17.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.281+0000 7fc2f0c00700 1 -- 172.21.15.123:0/3947342264 shutdown_connections 2024-03-30T05:02:17.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.281+0000 7fc2f0c00700 1 -- 172.21.15.123:0/3947342264 wait complete. 2024-03-30T05:02:17.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.282+0000 7fc2f0c00700 1 Processor -- start 2024-03-30T05:02:17.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.282+0000 7fc2f0c00700 1 -- start start 2024-03-30T05:02:17.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.282+0000 7fc2f0c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2ec070360 0x7fc2ec115b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:02:17.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.282+0000 7fc2f0c00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc2ec070c70 0x7fc2ec116060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:02:17.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.282+0000 7fc2f0c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2ec1166b0 con 0x7fc2ec070360 2024-03-30T05:02:17.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.282+0000 7fc2f0c00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2ec10faa0 con 0x7fc2ec070c70 2024-03-30T05:02:17.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.282+0000 7fc2eb400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2ec070360 0x7fc2ec115b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:02:17.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.282+0000 7fc2eb400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2ec070360 0x7fc2ec115b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48548/0 (socket says 172.21.15.123:48548) 2024-03-30T05:02:17.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.282+0000 7fc2eb400700 1 -- 172.21.15.123:0/2012140527 learned_addr learned my addr 172.21.15.123:0/2012140527 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:02:17.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.283+0000 7fc2eaa00700 1 --2- 172.21.15.123:0/2012140527 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc2ec070c70 0x7fc2ec116060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:02:17.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.283+0000 7fc2eb400700 1 -- 172.21.15.123:0/2012140527 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc2ec070c70 msgr2=0x7fc2ec116060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:17.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.283+0000 7fc2eb400700 1 --2- 172.21.15.123:0/2012140527 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc2ec070c70 0x7fc2ec116060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:17.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.283+0000 7fc2eb400700 1 -- 172.21.15.123:0/2012140527 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d8005040 con 0x7fc2ec070360 2024-03-30T05:02:17.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.283+0000 7fc2eb400700 1 --2- 172.21.15.123:0/2012140527 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2ec070360 0x7fc2ec115b20 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc00eb40 tx=0x7fc2dc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:02:17.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.283+0000 7fc2e3400700 1 -- 172.21.15.123:0/2012140527 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc2dc007cc0 con 0x7fc2ec070360 2024-03-30T05:02:17.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.283+0000 7fc2e3400700 1 -- 172.21.15.123:0/2012140527 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc2dc00f040 con 0x7fc2ec070360 2024-03-30T05:02:17.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.283+0000 7fc2f0c00700 1 -- 172.21.15.123:0/2012140527 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2ec10fd20 con 0x7fc2ec070360 2024-03-30T05:02:17.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.283+0000 7fc2e3400700 1 -- 172.21.15.123:0/2012140527 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc2dc009570 con 0x7fc2ec070360 2024-03-30T05:02:17.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.283+0000 7fc2f0c00700 1 -- 172.21.15.123:0/2012140527 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2ec110230 con 0x7fc2ec070360 2024-03-30T05:02:17.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.284+0000 7fc2f0c00700 1 -- 172.21.15.123:0/2012140527 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2ec071aa0 con 0x7fc2ec070360 2024-03-30T05:02:17.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.285+0000 7fc2e3400700 1 -- 172.21.15.123:0/2012140527 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fc2dc0096d0 con 0x7fc2ec070360 2024-03-30T05:02:17.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.286+0000 7fc2e3400700 1 --2- 172.21.15.123:0/2012140527 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc2d006cc20 0x7fc2d006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:02:17.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.286+0000 7fc2e3400700 1 -- 172.21.15.123:0/2012140527 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc2dc08bda0 con 0x7fc2ec070360 2024-03-30T05:02:17.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.286+0000 7fc2eaa00700 1 --2- 172.21.15.123:0/2012140527 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc2d006cc20 0x7fc2d006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:02:17.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.287+0000 7fc2eaa00700 1 --2- 172.21.15.123:0/2012140527 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc2d006cc20 0x7fc2d006f0d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fc2d8009fd0 tx=0x7fc2d8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:02:17.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.291+0000 7fc2e3400700 1 -- 172.21.15.123:0/2012140527 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc2dc059820 con 0x7fc2ec070360 2024-03-30T05:02:17.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:02:17 smithi135 ceph-mon[106011]: pgmap v933: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:17.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:02:17 smithi123 ceph-mon[139369]: pgmap v933: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:17.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.577+0000 7fc2f0c00700 1 -- 172.21.15.123:0/2012140527 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc2ec10bdf0 con 0x7fc2ec070360 2024-03-30T05:02:17.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.578+0000 7fc2e3400700 1 -- 172.21.15.123:0/2012140527 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7fc2dc0593b0 con 0x7fc2ec070360 2024-03-30T05:02:17.576 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T05:02:17.577 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T05:02:17.577 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T05:02:17.577 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:02:17.577 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T05:02:17.577 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T05:02:17.577 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:02:17.577 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T05:02:17.577 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T05:02:17.577 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:02:17.577 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T05:02:17.578 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 4 2024-03-30T05:02:17.578 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:02:17.578 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T05:02:17.578 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 14 2024-03-30T05:02:17.578 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T05:02:17.578 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T05:02:17.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.580+0000 7fc2f0c00700 1 -- 172.21.15.123:0/2012140527 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc2d006cc20 msgr2=0x7fc2d006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:17.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.580+0000 7fc2f0c00700 1 --2- 172.21.15.123:0/2012140527 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc2d006cc20 0x7fc2d006f0d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fc2d8009fd0 tx=0x7fc2d8009e30 comp rx=0 tx=0).stop 2024-03-30T05:02:17.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.580+0000 7fc2f0c00700 1 -- 172.21.15.123:0/2012140527 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2ec070360 msgr2=0x7fc2ec115b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:17.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.580+0000 7fc2f0c00700 1 --2- 172.21.15.123:0/2012140527 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2ec070360 0x7fc2ec115b20 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc00eb40 tx=0x7fc2dc00b040 comp rx=0 tx=0).stop 2024-03-30T05:02:17.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.580+0000 7fc2f0c00700 1 -- 172.21.15.123:0/2012140527 shutdown_connections 2024-03-30T05:02:17.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.580+0000 7fc2f0c00700 1 --2- 172.21.15.123:0/2012140527 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc2d006cc20 0x7fc2d006f0d0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:17.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.580+0000 7fc2f0c00700 1 --2- 172.21.15.123:0/2012140527 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2ec070360 0x7fc2ec115b20 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:17.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.580+0000 7fc2f0c00700 1 --2- 172.21.15.123:0/2012140527 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc2ec070c70 0x7fc2ec116060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:17.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.580+0000 7fc2f0c00700 1 -- 172.21.15.123:0/2012140527 >> 172.21.15.123:0/2012140527 conn(0x7fc2ec0f8790 msgr2=0x7fc2ec0f9d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:02:17.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.581+0000 7fc2f0c00700 1 -- 172.21.15.123:0/2012140527 shutdown_connections 2024-03-30T05:02:17.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.581+0000 7fc2f0c00700 1 -- 172.21.15.123:0/2012140527 wait complete. 2024-03-30T05:02:17.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.716+0000 7f31cec00700 1 -- 172.21.15.123:0/2338579411 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31c8102c40 msgr2=0x7f31c8103010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:17.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.716+0000 7f31cec00700 1 --2- 172.21.15.123:0/2338579411 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31c8102c40 0x7f31c8103010 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f31bc004940 tx=0x7f31bc009e30 comp rx=0 tx=0).stop 2024-03-30T05:02:17.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.716+0000 7f31cec00700 1 -- 172.21.15.123:0/2338579411 shutdown_connections 2024-03-30T05:02:17.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.716+0000 7f31cec00700 1 --2- 172.21.15.123:0/2338579411 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31c81035e0 0x7f31c80fcc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:17.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.716+0000 7f31cec00700 1 --2- 172.21.15.123:0/2338579411 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31c8102c40 0x7f31c8103010 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:17.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.716+0000 7f31cec00700 1 -- 172.21.15.123:0/2338579411 >> 172.21.15.123:0/2338579411 conn(0x7f31c80f8830 msgr2=0x7f31c80fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:02:17.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.716+0000 7f31cec00700 1 -- 172.21.15.123:0/2338579411 shutdown_connections 2024-03-30T05:02:17.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.716+0000 7f31cec00700 1 -- 172.21.15.123:0/2338579411 wait complete. 2024-03-30T05:02:17.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.717+0000 7f31cec00700 1 Processor -- start 2024-03-30T05:02:17.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.717+0000 7f31cec00700 1 -- start start 2024-03-30T05:02:17.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.717+0000 7f31cec00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31c8102c40 0x7f31c80702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:02:17.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.717+0000 7f31cec00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31c81035e0 0x7f31c8070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:02:17.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.717+0000 7f31cec00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c8070d90 con 0x7f31c81035e0 2024-03-30T05:02:17.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.717+0000 7f31cec00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c8070ed0 con 0x7f31c8102c40 2024-03-30T05:02:17.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.717+0000 7f31cce00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31c81035e0 0x7f31c8070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:02:17.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.717+0000 7f31cce00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31c81035e0 0x7f31c8070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48570/0 (socket says 172.21.15.123:48570) 2024-03-30T05:02:17.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.717+0000 7f31cce00700 1 -- 172.21.15.123:0/33645686 learned_addr learned my addr 172.21.15.123:0/33645686 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:02:17.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.718+0000 7f31cd800700 1 --2- 172.21.15.123:0/33645686 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31c8102c40 0x7f31c80702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:02:17.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.718+0000 7f31cce00700 1 -- 172.21.15.123:0/33645686 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31c8102c40 msgr2=0x7f31c80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:17.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.718+0000 7f31cce00700 1 --2- 172.21.15.123:0/33645686 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31c8102c40 0x7f31c80702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:17.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.718+0000 7f31cce00700 1 -- 172.21.15.123:0/33645686 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31bc005040 con 0x7f31c81035e0 2024-03-30T05:02:17.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.718+0000 7f31cce00700 1 --2- 172.21.15.123:0/33645686 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31c81035e0 0x7f31c8070800 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f31b400c890 tx=0x7f31b40044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:02:17.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.718+0000 7f31c5600700 1 -- 172.21.15.123:0/33645686 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31b400a700 con 0x7f31c81035e0 2024-03-30T05:02:17.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.718+0000 7f31c5600700 1 -- 172.21.15.123:0/33645686 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f31b400ad10 con 0x7f31c81035e0 2024-03-30T05:02:17.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.718+0000 7f31c5600700 1 -- 172.21.15.123:0/33645686 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31b4012e70 con 0x7f31c81035e0 2024-03-30T05:02:17.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.718+0000 7f31cec00700 1 -- 172.21.15.123:0/33645686 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31c8119c50 con 0x7f31c81035e0 2024-03-30T05:02:17.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.719+0000 7f31cec00700 1 -- 172.21.15.123:0/33645686 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31c811a070 con 0x7f31c81035e0 2024-03-30T05:02:17.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.719+0000 7f31cec00700 1 -- 172.21.15.123:0/33645686 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31c80fe390 con 0x7f31c81035e0 2024-03-30T05:02:17.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.720+0000 7f31c5600700 1 -- 172.21.15.123:0/33645686 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f31b400a860 con 0x7f31c81035e0 2024-03-30T05:02:17.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.722+0000 7f31c5600700 1 --2- 172.21.15.123:0/33645686 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f31b006cc30 0x7f31b006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:02:17.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.722+0000 7f31c5600700 1 -- 172.21.15.123:0/33645686 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f31b408c050 con 0x7f31c81035e0 2024-03-30T05:02:17.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.722+0000 7f31cd800700 1 --2- 172.21.15.123:0/33645686 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f31b006cc30 0x7f31b006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:02:17.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.723+0000 7f31cd800700 1 --2- 172.21.15.123:0/33645686 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f31b006cc30 0x7f31b006f0e0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f31bc00b840 tx=0x7f31bc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:02:17.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.726+0000 7f31c5600700 1 -- 172.21.15.123:0/33645686 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f31b4056090 con 0x7f31c81035e0 2024-03-30T05:02:17.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.970+0000 7f31cec00700 1 -- 172.21.15.123:0/33645686 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f31c8109bf0 con 0x7f31c81035e0 2024-03-30T05:02:17.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.970+0000 7f31c5600700 1 -- 172.21.15.123:0/33645686 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 34 v34) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f31b40596b0 con 0x7f31c81035e0 2024-03-30T05:02:17.969 INFO:teuthology.orchestra.run.smithi123.stdout:e34 2024-03-30T05:02:17.969 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T05:02:17.969 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T05:02:17.969 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T05:02:17.969 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:02:17.969 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T05:02:17.969 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T05:02:17.969 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T05:02:17.969 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T05:02:17.969 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T05:02:17.970 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T05:02:17.970 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T05:02:17.970 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T05:02:17.970 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T05:02:17.970 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T05:02:17.970 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T05:02:17.970 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T05:02:17.970 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T05:02:17.970 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T05:02:17.970 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T05:02:17.970 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T05:02:17.970 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T05:02:17.971 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T05:02:17.971 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T05:02:17.971 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T05:02:17.971 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T05:02:17.971 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T05:02:17.971 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T05:02:17.971 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T05:02:17.971 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T05:02:17.971 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T05:02:17.971 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T05:02:17.971 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T05:02:17.971 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:02:17.971 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:02:17.972 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T05:02:17.972 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:02:17.972 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:02:17.972 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:02:17.972 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:02:17.972 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44975} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/3788988014,v1:172.21.15.135:6827/3788988014] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:02:17.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.973+0000 7f31cec00700 1 -- 172.21.15.123:0/33645686 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f31b006cc30 msgr2=0x7f31b006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:17.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.973+0000 7f31cec00700 1 --2- 172.21.15.123:0/33645686 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f31b006cc30 0x7f31b006f0e0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f31bc00b840 tx=0x7f31bc009e30 comp rx=0 tx=0).stop 2024-03-30T05:02:17.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.973+0000 7f31cec00700 1 -- 172.21.15.123:0/33645686 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31c81035e0 msgr2=0x7f31c8070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:17.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.973+0000 7f31cec00700 1 --2- 172.21.15.123:0/33645686 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31c81035e0 0x7f31c8070800 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f31b400c890 tx=0x7f31b40044c0 comp rx=0 tx=0).stop 2024-03-30T05:02:17.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.973+0000 7f31cec00700 1 -- 172.21.15.123:0/33645686 shutdown_connections 2024-03-30T05:02:17.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.973+0000 7f31cec00700 1 --2- 172.21.15.123:0/33645686 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f31b006cc30 0x7f31b006f0e0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:17.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.973+0000 7f31cec00700 1 --2- 172.21.15.123:0/33645686 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31c8102c40 0x7f31c80702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:17.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.973+0000 7f31cec00700 1 --2- 172.21.15.123:0/33645686 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31c81035e0 0x7f31c8070800 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:17.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.973+0000 7f31cec00700 1 -- 172.21.15.123:0/33645686 >> 172.21.15.123:0/33645686 conn(0x7f31c80f8830 msgr2=0x7f31c80f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:02:17.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.973+0000 7f31cec00700 1 -- 172.21.15.123:0/33645686 shutdown_connections 2024-03-30T05:02:17.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:17.973+0000 7f31cec00700 1 -- 172.21.15.123:0/33645686 wait complete. 2024-03-30T05:02:17.974 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 34 2024-03-30T05:02:18.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.111+0000 7fc799c00700 1 -- 172.21.15.123:0/1180818722 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc794102e00 msgr2=0x7fc7941031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:18.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.111+0000 7fc799c00700 1 --2- 172.21.15.123:0/1180818722 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc794102e00 0x7fc7941031d0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fc784004940 tx=0x7fc784009e30 comp rx=0 tx=0).stop 2024-03-30T05:02:18.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.111+0000 7fc799c00700 1 -- 172.21.15.123:0/1180818722 shutdown_connections 2024-03-30T05:02:18.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.111+0000 7fc799c00700 1 --2- 172.21.15.123:0/1180818722 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7941037a0 0x7fc7940fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:18.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.111+0000 7fc799c00700 1 --2- 172.21.15.123:0/1180818722 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc794102e00 0x7fc7941031d0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:18.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.111+0000 7fc799c00700 1 -- 172.21.15.123:0/1180818722 >> 172.21.15.123:0/1180818722 conn(0x7fc7940f89b0 msgr2=0x7fc7940fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:02:18.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.112+0000 7fc799c00700 1 -- 172.21.15.123:0/1180818722 shutdown_connections 2024-03-30T05:02:18.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.112+0000 7fc799c00700 1 -- 172.21.15.123:0/1180818722 wait complete. 2024-03-30T05:02:18.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.112+0000 7fc799c00700 1 Processor -- start 2024-03-30T05:02:18.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.112+0000 7fc799c00700 1 -- start start 2024-03-30T05:02:18.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.113+0000 7fc799c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc794102e00 0x7fc794115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:02:18.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.113+0000 7fc799c00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7941037a0 0x7fc794116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:02:18.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.113+0000 7fc799c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc794116860 con 0x7fc794102e00 2024-03-30T05:02:18.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.113+0000 7fc799c00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc79410fc30 con 0x7fc7941037a0 2024-03-30T05:02:18.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.113+0000 7fc792a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc794102e00 0x7fc794115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:02:18.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.113+0000 7fc792a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc794102e00 0x7fc794115cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48592/0 (socket says 172.21.15.123:48592) 2024-03-30T05:02:18.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.113+0000 7fc792a00700 1 -- 172.21.15.123:0/268271876 learned_addr learned my addr 172.21.15.123:0/268271876 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:02:18.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.113+0000 7fc792000700 1 --2- 172.21.15.123:0/268271876 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7941037a0 0x7fc794116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:02:18.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.113+0000 7fc792a00700 1 -- 172.21.15.123:0/268271876 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7941037a0 msgr2=0x7fc794116210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:18.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.113+0000 7fc792a00700 1 --2- 172.21.15.123:0/268271876 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7941037a0 0x7fc794116210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:18.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.113+0000 7fc792a00700 1 -- 172.21.15.123:0/268271876 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc784005040 con 0x7fc794102e00 2024-03-30T05:02:18.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.114+0000 7fc792a00700 1 --2- 172.21.15.123:0/268271876 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc794102e00 0x7fc794115cd0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fc784009390 tx=0x7fc784009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:02:18.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.114+0000 7fc78aa00700 1 -- 172.21.15.123:0/268271876 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc78400e040 con 0x7fc794102e00 2024-03-30T05:02:18.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.114+0000 7fc78aa00700 1 -- 172.21.15.123:0/268271876 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc784013420 con 0x7fc794102e00 2024-03-30T05:02:18.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.114+0000 7fc799c00700 1 -- 172.21.15.123:0/268271876 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc79410feb0 con 0x7fc794102e00 2024-03-30T05:02:18.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.114+0000 7fc78aa00700 1 -- 172.21.15.123:0/268271876 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc78401c630 con 0x7fc794102e00 2024-03-30T05:02:18.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.114+0000 7fc799c00700 1 -- 172.21.15.123:0/268271876 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc794110470 con 0x7fc794102e00 2024-03-30T05:02:18.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.115+0000 7fc799c00700 1 -- 172.21.15.123:0/268271876 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7940fe550 con 0x7fc794102e00 2024-03-30T05:02:18.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.116+0000 7fc78aa00700 1 -- 172.21.15.123:0/268271876 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fc7840139e0 con 0x7fc794102e00 2024-03-30T05:02:18.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.117+0000 7fc78aa00700 1 --2- 172.21.15.123:0/268271876 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc77806cc30 0x7fc77806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:02:18.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.117+0000 7fc78aa00700 1 -- 172.21.15.123:0/268271876 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc78408d250 con 0x7fc794102e00 2024-03-30T05:02:18.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.117+0000 7fc792000700 1 --2- 172.21.15.123:0/268271876 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc77806cc30 0x7fc77806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:02:18.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.118+0000 7fc792000700 1 --2- 172.21.15.123:0/268271876 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc77806cc30 0x7fc77806f0e0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fc794111570 tx=0x7fc780009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:02:18.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.121+0000 7fc78aa00700 1 -- 172.21.15.123:0/268271876 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc78405add0 con 0x7fc794102e00 2024-03-30T05:02:18.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.345+0000 7fc799c00700 1 -- 172.21.15.123:0/268271876 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc79404bf40 con 0x7fc77806cc30 2024-03-30T05:02:18.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.347+0000 7fc78aa00700 1 -- 172.21.15.123:0/268271876 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fc79404bf40 con 0x7fc77806cc30 2024-03-30T05:02:18.345 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T05:02:18.345 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T05:02:18.345 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T05:02:18.345 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T05:02:18.345 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T05:02:18.346 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T05:02:18.346 INFO:teuthology.orchestra.run.smithi123.stdout: "mds", 2024-03-30T05:02:18.346 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T05:02:18.346 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T05:02:18.346 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T05:02:18.346 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T05:02:18.346 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "16/21 daemons upgraded", 2024-03-30T05:02:18.346 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T05:02:18.346 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T05:02:18.346 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T05:02:18.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.349+0000 7fc799c00700 1 -- 172.21.15.123:0/268271876 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc77806cc30 msgr2=0x7fc77806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:18.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.349+0000 7fc799c00700 1 --2- 172.21.15.123:0/268271876 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc77806cc30 0x7fc77806f0e0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fc794111570 tx=0x7fc780009ee0 comp rx=0 tx=0).stop 2024-03-30T05:02:18.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.349+0000 7fc799c00700 1 -- 172.21.15.123:0/268271876 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc794102e00 msgr2=0x7fc794115cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:18.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.349+0000 7fc799c00700 1 --2- 172.21.15.123:0/268271876 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc794102e00 0x7fc794115cd0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fc784009390 tx=0x7fc784009e30 comp rx=0 tx=0).stop 2024-03-30T05:02:18.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.349+0000 7fc799c00700 1 -- 172.21.15.123:0/268271876 shutdown_connections 2024-03-30T05:02:18.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.349+0000 7fc799c00700 1 --2- 172.21.15.123:0/268271876 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc77806cc30 0x7fc77806f0e0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:18.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.349+0000 7fc799c00700 1 --2- 172.21.15.123:0/268271876 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc794102e00 0x7fc794115cd0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:18.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.349+0000 7fc799c00700 1 --2- 172.21.15.123:0/268271876 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7941037a0 0x7fc794116210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:18.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.349+0000 7fc799c00700 1 -- 172.21.15.123:0/268271876 >> 172.21.15.123:0/268271876 conn(0x7fc7940f89b0 msgr2=0x7fc7940f99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:02:18.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.349+0000 7fc799c00700 1 -- 172.21.15.123:0/268271876 shutdown_connections 2024-03-30T05:02:18.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.349+0000 7fc799c00700 1 -- 172.21.15.123:0/268271876 wait complete. 2024-03-30T05:02:18.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:02:18 smithi135 ceph-mon[106011]: from='client.35692 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:02:18.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:02:18 smithi135 ceph-mon[106011]: from='client.35696 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:02:18.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:02:18 smithi135 ceph-mon[106011]: from='client.35700 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:02:18.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:02:18 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2012140527' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:02:18.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:02:18 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/33645686' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T05:02:18.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:02:18 smithi123 ceph-mon[139369]: from='client.35692 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:02:18.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:02:18 smithi123 ceph-mon[139369]: from='client.35696 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:02:18.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:02:18 smithi123 ceph-mon[139369]: from='client.35700 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:02:18.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:02:18 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2012140527' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:02:18.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:02:18 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/33645686' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T05:02:18.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.481+0000 7fd532800700 1 -- 172.21.15.123:0/2280364063 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd52c102e00 msgr2=0x7fd52c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:18.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.481+0000 7fd532800700 1 --2- 172.21.15.123:0/2280364063 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd52c102e00 0x7fd52c1031d0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fd51c004940 tx=0x7fd51c009e30 comp rx=0 tx=0).stop 2024-03-30T05:02:18.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.481+0000 7fd532800700 1 -- 172.21.15.123:0/2280364063 shutdown_connections 2024-03-30T05:02:18.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.481+0000 7fd532800700 1 --2- 172.21.15.123:0/2280364063 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd52c1037a0 0x7fd52c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:18.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.481+0000 7fd532800700 1 --2- 172.21.15.123:0/2280364063 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd52c102e00 0x7fd52c1031d0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:18.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.481+0000 7fd532800700 1 -- 172.21.15.123:0/2280364063 >> 172.21.15.123:0/2280364063 conn(0x7fd52c0f89b0 msgr2=0x7fd52c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:02:18.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.482+0000 7fd532800700 1 -- 172.21.15.123:0/2280364063 shutdown_connections 2024-03-30T05:02:18.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.482+0000 7fd532800700 1 -- 172.21.15.123:0/2280364063 wait complete. 2024-03-30T05:02:18.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.482+0000 7fd532800700 1 Processor -- start 2024-03-30T05:02:18.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.483+0000 7fd532800700 1 -- start start 2024-03-30T05:02:18.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.483+0000 7fd532800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd52c102e00 0x7fd52c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:02:18.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.483+0000 7fd532800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd52c1037a0 0x7fd52c070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:02:18.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.483+0000 7fd532800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd52c070d90 con 0x7fd52c1037a0 2024-03-30T05:02:18.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.483+0000 7fd532800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd52c070ed0 con 0x7fd52c102e00 2024-03-30T05:02:18.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.483+0000 7fd52aa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd52c1037a0 0x7fd52c070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:02:18.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.483+0000 7fd52aa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd52c1037a0 0x7fd52c070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48610/0 (socket says 172.21.15.123:48610) 2024-03-30T05:02:18.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.483+0000 7fd52aa00700 1 -- 172.21.15.123:0/2745072698 learned_addr learned my addr 172.21.15.123:0/2745072698 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:02:18.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.483+0000 7fd52aa00700 1 -- 172.21.15.123:0/2745072698 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd52c102e00 msgr2=0x7fd52c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:18.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.484+0000 7fd52aa00700 1 --2- 172.21.15.123:0/2745072698 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd52c102e00 0x7fd52c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:18.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.484+0000 7fd52aa00700 1 -- 172.21.15.123:0/2745072698 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd51c005040 con 0x7fd52c1037a0 2024-03-30T05:02:18.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.484+0000 7fd52aa00700 1 --2- 172.21.15.123:0/2745072698 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd52c1037a0 0x7fd52c070800 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fd518002a50 tx=0x7fd518004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:02:18.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.484+0000 7fd523400700 1 -- 172.21.15.123:0/2745072698 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd518007cc0 con 0x7fd52c1037a0 2024-03-30T05:02:18.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.484+0000 7fd523400700 1 -- 172.21.15.123:0/2745072698 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd51800f040 con 0x7fd52c1037a0 2024-03-30T05:02:18.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.484+0000 7fd532800700 1 -- 172.21.15.123:0/2745072698 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd52c119e20 con 0x7fd52c1037a0 2024-03-30T05:02:18.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.484+0000 7fd523400700 1 -- 172.21.15.123:0/2745072698 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd51800d9a0 con 0x7fd52c1037a0 2024-03-30T05:02:18.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.484+0000 7fd532800700 1 -- 172.21.15.123:0/2745072698 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd52c11a2a0 con 0x7fd52c1037a0 2024-03-30T05:02:18.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.485+0000 7fd532800700 1 -- 172.21.15.123:0/2745072698 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd52c0fe550 con 0x7fd52c1037a0 2024-03-30T05:02:18.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.486+0000 7fd523400700 1 -- 172.21.15.123:0/2745072698 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fd51800dc60 con 0x7fd52c1037a0 2024-03-30T05:02:18.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.487+0000 7fd523400700 1 --2- 172.21.15.123:0/2745072698 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd51006cc20 0x7fd51006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:02:18.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.487+0000 7fd523400700 1 -- 172.21.15.123:0/2745072698 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd51808bef0 con 0x7fd52c1037a0 2024-03-30T05:02:18.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.487+0000 7fd52b400700 1 --2- 172.21.15.123:0/2745072698 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd51006cc20 0x7fd51006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:02:18.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.488+0000 7fd52b400700 1 --2- 172.21.15.123:0/2745072698 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd51006cc20 0x7fd51006f0d0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fd51c004f90 tx=0x7fd51c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:02:18.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.492+0000 7fd523400700 1 -- 172.21.15.123:0/2745072698 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd518055e30 con 0x7fd52c1037a0 2024-03-30T05:02:18.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.775+0000 7fd532800700 1 -- 172.21.15.123:0/2745072698 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fd52c05f5f0 con 0x7fd52c1037a0 2024-03-30T05:02:18.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.776+0000 7fd523400700 1 -- 172.21.15.123:0/2745072698 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7fd518059450 con 0x7fd52c1037a0 2024-03-30T05:02:18.774 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T05:02:18.774 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-30T05:02:18.774 INFO:teuthology.orchestra.run.smithi123.stdout: host smithi135 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi135/config 2024-03-30T05:02:18.775 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-30T05:02:18.775 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T05:02:18.775 INFO:teuthology.orchestra.run.smithi123.stdout:Traceback (most recent call last): 2024-03-30T05:02:18.775 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-30T05:02:18.775 INFO:teuthology.orchestra.run.smithi123.stdout: "__main__", mod_spec) 2024-03-30T05:02:18.775 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-30T05:02:18.775 INFO:teuthology.orchestra.run.smithi123.stdout: exec(code, run_globals) 2024-03-30T05:02:18.775 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10889, in 2024-03-30T05:02:18.775 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10877, in main 2024-03-30T05:02:18.775 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-30T05:02:18.775 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-30T05:02:18.775 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-30T05:02:18.775 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-30T05:02:18.775 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-03-30T05:02:18.775 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-30T05:02:18.776 INFO:teuthology.orchestra.run.smithi123.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T05:02:18.776 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T05:02:18.776 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T05:02:18.776 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T05:02:18.776 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T05:02:18.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.778+0000 7fd532800700 1 -- 172.21.15.123:0/2745072698 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd51006cc20 msgr2=0x7fd51006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:18.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.778+0000 7fd532800700 1 --2- 172.21.15.123:0/2745072698 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd51006cc20 0x7fd51006f0d0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fd51c004f90 tx=0x7fd51c009e30 comp rx=0 tx=0).stop 2024-03-30T05:02:18.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.778+0000 7fd532800700 1 -- 172.21.15.123:0/2745072698 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd52c1037a0 msgr2=0x7fd52c070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:18.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.778+0000 7fd532800700 1 --2- 172.21.15.123:0/2745072698 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd52c1037a0 0x7fd52c070800 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fd518002a50 tx=0x7fd518004600 comp rx=0 tx=0).stop 2024-03-30T05:02:18.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.778+0000 7fd532800700 1 -- 172.21.15.123:0/2745072698 shutdown_connections 2024-03-30T05:02:18.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.778+0000 7fd532800700 1 --2- 172.21.15.123:0/2745072698 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd51006cc20 0x7fd51006f0d0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:18.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.778+0000 7fd532800700 1 --2- 172.21.15.123:0/2745072698 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd52c102e00 0x7fd52c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:18.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.778+0000 7fd532800700 1 --2- 172.21.15.123:0/2745072698 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd52c1037a0 0x7fd52c070800 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:18.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.778+0000 7fd532800700 1 -- 172.21.15.123:0/2745072698 >> 172.21.15.123:0/2745072698 conn(0x7fd52c0f89b0 msgr2=0x7fd52c0f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:02:18.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.778+0000 7fd532800700 1 -- 172.21.15.123:0/2745072698 shutdown_connections 2024-03-30T05:02:18.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:18.778+0000 7fd532800700 1 -- 172.21.15.123:0/2745072698 wait complete. 2024-03-30T05:02:19.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:02:19 smithi135 ceph-mon[106011]: pgmap v934: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:19.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:02:19 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2745072698' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T05:02:19.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:02:19 smithi123 ceph-mon[139369]: pgmap v934: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:19.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:02:19 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2745072698' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T05:02:20.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:02:20 smithi135 ceph-mon[106011]: from='client.35712 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:02:20.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:02:20 smithi123 ceph-mon[139369]: from='client.35712 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:02:21.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:02:21 smithi135 ceph-mon[106011]: pgmap v935: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:21.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:02:21 smithi123 ceph-mon[139369]: pgmap v935: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:23.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:02:23 smithi135 ceph-mon[106011]: pgmap v936: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:23.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:02:23 smithi123 ceph-mon[139369]: pgmap v936: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:25.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:02:25 smithi135 ceph-mon[106011]: pgmap v937: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:25.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:02:25 smithi123 ceph-mon[139369]: pgmap v937: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:27.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:02:27 smithi135 ceph-mon[106011]: pgmap v938: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:27.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:02:27 smithi123 ceph-mon[139369]: pgmap v938: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:29.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:02:29 smithi135 ceph-mon[106011]: pgmap v939: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:29.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:02:29 smithi123 ceph-mon[139369]: pgmap v939: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:30.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:02:30 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:02:30.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:02:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:02:31.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:02:31 smithi135 ceph-mon[106011]: pgmap v940: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:31.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:02:31 smithi123 ceph-mon[139369]: pgmap v940: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:33.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:02:33 smithi135 ceph-mon[106011]: pgmap v941: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:33.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:02:33 smithi123 ceph-mon[139369]: pgmap v941: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:35.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:02:35 smithi135 ceph-mon[106011]: pgmap v942: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:35.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:02:35 smithi123 ceph-mon[139369]: pgmap v942: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:37.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:02:37 smithi135 ceph-mon[106011]: pgmap v943: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:37.474 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:02:37 smithi123 ceph-mon[139369]: pgmap v943: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:39.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:02:39 smithi123 ceph-mon[139369]: pgmap v944: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:39.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:02:39 smithi135 ceph-mon[106011]: pgmap v944: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:41.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:02:41 smithi123 ceph-mon[139369]: pgmap v945: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:41.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:02:41 smithi135 ceph-mon[106011]: pgmap v945: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:43.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:02:43 smithi123 ceph-mon[139369]: pgmap v946: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:43.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:02:43 smithi135 ceph-mon[106011]: pgmap v946: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:45.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:02:45 smithi123 ceph-mon[139369]: pgmap v947: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:45.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:02:45 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:02:45.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:02:45 smithi135 ceph-mon[106011]: pgmap v947: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:45.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:02:45 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:02:47.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:02:47 smithi123 ceph-mon[139369]: pgmap v948: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:47.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:02:47 smithi135 ceph-mon[106011]: pgmap v948: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:48.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:48.926+0000 7f18b4400700 1 -- 172.21.15.123:0/862983609 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18ac102e20 msgr2=0x7f18ac1031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:48.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:48.926+0000 7f18b4400700 1 --2- 172.21.15.123:0/862983609 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18ac102e20 0x7f18ac1031f0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f18a0004940 tx=0x7f18a0009e30 comp rx=0 tx=0).stop 2024-03-30T05:02:48.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:48.926+0000 7f18b4400700 1 -- 172.21.15.123:0/862983609 shutdown_connections 2024-03-30T05:02:48.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:48.926+0000 7f18b4400700 1 --2- 172.21.15.123:0/862983609 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f18ac1037c0 0x7f18ac0fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:48.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:48.926+0000 7f18b4400700 1 --2- 172.21.15.123:0/862983609 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18ac102e20 0x7f18ac1031f0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:48.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:48.926+0000 7f18b4400700 1 -- 172.21.15.123:0/862983609 >> 172.21.15.123:0/862983609 conn(0x7f18ac0f89d0 msgr2=0x7f18ac0fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:02:48.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:48.926+0000 7f18b4400700 1 -- 172.21.15.123:0/862983609 shutdown_connections 2024-03-30T05:02:48.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:48.926+0000 7f18b4400700 1 -- 172.21.15.123:0/862983609 wait complete. 2024-03-30T05:02:48.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:48.927+0000 7f18b4400700 1 Processor -- start 2024-03-30T05:02:48.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:48.927+0000 7f18b4400700 1 -- start start 2024-03-30T05:02:48.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:48.928+0000 7f18b4400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18ac102e20 0x7f18ac115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:02:48.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:48.928+0000 7f18b4400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f18ac1037c0 0x7f18ac116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:02:48.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:48.928+0000 7f18b4400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18ac1168f0 con 0x7f18ac102e20 2024-03-30T05:02:48.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:48.928+0000 7f18b4400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18ac10fc30 con 0x7f18ac1037c0 2024-03-30T05:02:48.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:48.928+0000 7f18b1a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18ac102e20 0x7f18ac115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:02:48.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:48.928+0000 7f18b1a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18ac102e20 0x7f18ac115cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43384/0 (socket says 172.21.15.123:43384) 2024-03-30T05:02:48.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:48.928+0000 7f18b1a00700 1 -- 172.21.15.123:0/749478382 learned_addr learned my addr 172.21.15.123:0/749478382 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:02:48.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:48.928+0000 7f18b1000700 1 --2- 172.21.15.123:0/749478382 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f18ac1037c0 0x7f18ac116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:02:48.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:48.928+0000 7f18b1a00700 1 -- 172.21.15.123:0/749478382 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f18ac1037c0 msgr2=0x7f18ac116210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:48.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:48.928+0000 7f18b1a00700 1 --2- 172.21.15.123:0/749478382 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f18ac1037c0 0x7f18ac116210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:48.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:48.928+0000 7f18b1a00700 1 -- 172.21.15.123:0/749478382 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18a0005040 con 0x7f18ac102e20 2024-03-30T05:02:48.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:48.928+0000 7f18b1a00700 1 --2- 172.21.15.123:0/749478382 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18ac102e20 0x7f18ac115cd0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f18a0009390 tx=0x7f18a0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:02:48.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:48.929+0000 7f18a9600700 1 -- 172.21.15.123:0/749478382 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f18a000e040 con 0x7f18ac102e20 2024-03-30T05:02:48.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:48.929+0000 7f18a9600700 1 -- 172.21.15.123:0/749478382 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f18a0013420 con 0x7f18ac102e20 2024-03-30T05:02:48.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:48.929+0000 7f18b4400700 1 -- 172.21.15.123:0/749478382 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18ac10fe50 con 0x7f18ac102e20 2024-03-30T05:02:48.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:48.929+0000 7f18a9600700 1 -- 172.21.15.123:0/749478382 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f18a001c630 con 0x7f18ac102e20 2024-03-30T05:02:48.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:48.929+0000 7f18b4400700 1 -- 172.21.15.123:0/749478382 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18ac110410 con 0x7f18ac102e20 2024-03-30T05:02:48.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:48.930+0000 7f18b4400700 1 -- 172.21.15.123:0/749478382 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18ac0fe570 con 0x7f18ac102e20 2024-03-30T05:02:48.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:48.931+0000 7f18a9600700 1 -- 172.21.15.123:0/749478382 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f18a00139e0 con 0x7f18ac102e20 2024-03-30T05:02:48.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:48.932+0000 7f18a9600700 1 --2- 172.21.15.123:0/749478382 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f189406cc30 0x7f189406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:02:48.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:48.932+0000 7f18a9600700 1 -- 172.21.15.123:0/749478382 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f18a008d320 con 0x7f18ac102e20 2024-03-30T05:02:48.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:48.932+0000 7f18b1000700 1 --2- 172.21.15.123:0/749478382 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f189406cc30 0x7f189406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:02:48.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:48.933+0000 7f18b1000700 1 --2- 172.21.15.123:0/749478382 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f189406cc30 0x7f189406f0e0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f1898006fd0 tx=0x7f1898008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:02:48.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:48.937+0000 7f18a9600700 1 -- 172.21.15.123:0/749478382 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f18a005ae20 con 0x7f18ac102e20 2024-03-30T05:02:49.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.156+0000 7f18b4400700 1 -- 172.21.15.123:0/749478382 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f18ac111320 con 0x7f189406cc30 2024-03-30T05:02:49.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.157+0000 7f18a9600700 1 -- 172.21.15.123:0/749478382 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f18ac111320 con 0x7f189406cc30 2024-03-30T05:02:49.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.160+0000 7f18b4400700 1 -- 172.21.15.123:0/749478382 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f189406cc30 msgr2=0x7f189406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:49.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.160+0000 7f18b4400700 1 --2- 172.21.15.123:0/749478382 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f189406cc30 0x7f189406f0e0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f1898006fd0 tx=0x7f1898008040 comp rx=0 tx=0).stop 2024-03-30T05:02:49.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.160+0000 7f18b4400700 1 -- 172.21.15.123:0/749478382 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18ac102e20 msgr2=0x7f18ac115cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:49.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.160+0000 7f18b4400700 1 --2- 172.21.15.123:0/749478382 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18ac102e20 0x7f18ac115cd0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f18a0009390 tx=0x7f18a0009e30 comp rx=0 tx=0).stop 2024-03-30T05:02:49.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.160+0000 7f18b4400700 1 -- 172.21.15.123:0/749478382 shutdown_connections 2024-03-30T05:02:49.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.160+0000 7f18b4400700 1 --2- 172.21.15.123:0/749478382 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f189406cc30 0x7f189406f0e0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:49.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.160+0000 7f18b4400700 1 --2- 172.21.15.123:0/749478382 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18ac102e20 0x7f18ac115cd0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:49.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.160+0000 7f18b4400700 1 --2- 172.21.15.123:0/749478382 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f18ac1037c0 0x7f18ac116210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:49.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.160+0000 7f18b4400700 1 -- 172.21.15.123:0/749478382 >> 172.21.15.123:0/749478382 conn(0x7f18ac0f89d0 msgr2=0x7f18ac0f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:02:49.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.160+0000 7f18b4400700 1 -- 172.21.15.123:0/749478382 shutdown_connections 2024-03-30T05:02:49.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.160+0000 7f18b4400700 1 -- 172.21.15.123:0/749478382 wait complete. 2024-03-30T05:02:49.173 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T05:02:49.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.295+0000 7efe03800700 1 -- 172.21.15.123:0/3190200426 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efdfc1037a0 msgr2=0x7efdfc0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:49.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.295+0000 7efe03800700 1 --2- 172.21.15.123:0/3190200426 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efdfc1037a0 0x7efdfc0fce10 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7efde8004940 tx=0x7efde8009e30 comp rx=0 tx=0).stop 2024-03-30T05:02:49.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.295+0000 7efe03800700 1 -- 172.21.15.123:0/3190200426 shutdown_connections 2024-03-30T05:02:49.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.295+0000 7efe03800700 1 --2- 172.21.15.123:0/3190200426 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efdfc1037a0 0x7efdfc0fce10 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:49.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.295+0000 7efe03800700 1 --2- 172.21.15.123:0/3190200426 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efdfc102e00 0x7efdfc1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:49.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.295+0000 7efe03800700 1 -- 172.21.15.123:0/3190200426 >> 172.21.15.123:0/3190200426 conn(0x7efdfc0f89b0 msgr2=0x7efdfc0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:02:49.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.295+0000 7efe03800700 1 -- 172.21.15.123:0/3190200426 shutdown_connections 2024-03-30T05:02:49.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.295+0000 7efe03800700 1 -- 172.21.15.123:0/3190200426 wait complete. 2024-03-30T05:02:49.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.296+0000 7efe03800700 1 Processor -- start 2024-03-30T05:02:49.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.296+0000 7efe03800700 1 -- start start 2024-03-30T05:02:49.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.297+0000 7efe03800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efdfc102e00 0x7efdfc113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:02:49.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.297+0000 7efe03800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efdfc1037a0 0x7efdfc114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:02:49.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.297+0000 7efe03800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdfc114690 con 0x7efdfc1037a0 2024-03-30T05:02:49.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.297+0000 7efe03800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdfc10da80 con 0x7efdfc102e00 2024-03-30T05:02:49.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.297+0000 7efdfbe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efdfc1037a0 0x7efdfc114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:02:49.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.297+0000 7efdfbe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efdfc1037a0 0x7efdfc114040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43408/0 (socket says 172.21.15.123:43408) 2024-03-30T05:02:49.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.297+0000 7efdfbe00700 1 -- 172.21.15.123:0/1631307667 learned_addr learned my addr 172.21.15.123:0/1631307667 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:02:49.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.297+0000 7efe00e00700 1 --2- 172.21.15.123:0/1631307667 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efdfc102e00 0x7efdfc113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:02:49.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.297+0000 7efdfbe00700 1 -- 172.21.15.123:0/1631307667 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efdfc102e00 msgr2=0x7efdfc113b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:49.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.297+0000 7efdfbe00700 1 --2- 172.21.15.123:0/1631307667 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efdfc102e00 0x7efdfc113b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:49.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.297+0000 7efdfbe00700 1 -- 172.21.15.123:0/1631307667 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efde8005040 con 0x7efdfc1037a0 2024-03-30T05:02:49.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.298+0000 7efdfbe00700 1 --2- 172.21.15.123:0/1631307667 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efdfc1037a0 0x7efdfc114040 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7efde8009390 tx=0x7efde8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:02:49.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.298+0000 7efdf8c00700 1 -- 172.21.15.123:0/1631307667 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efde800e040 con 0x7efdfc1037a0 2024-03-30T05:02:49.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.298+0000 7efdf8c00700 1 -- 172.21.15.123:0/1631307667 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efde8013420 con 0x7efdfc1037a0 2024-03-30T05:02:49.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.298+0000 7efe03800700 1 -- 172.21.15.123:0/1631307667 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdfc10dd00 con 0x7efdfc1037a0 2024-03-30T05:02:49.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.298+0000 7efdf8c00700 1 -- 172.21.15.123:0/1631307667 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efde801c630 con 0x7efdfc1037a0 2024-03-30T05:02:49.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.298+0000 7efe03800700 1 -- 172.21.15.123:0/1631307667 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdfc10e2c0 con 0x7efdfc1037a0 2024-03-30T05:02:49.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.299+0000 7efe03800700 1 -- 172.21.15.123:0/1631307667 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdfc0fe550 con 0x7efdfc1037a0 2024-03-30T05:02:49.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.300+0000 7efdf8c00700 1 -- 172.21.15.123:0/1631307667 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7efde8013a00 con 0x7efdfc1037a0 2024-03-30T05:02:49.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.301+0000 7efdf8c00700 1 --2- 172.21.15.123:0/1631307667 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7efde406cc30 0x7efde406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:02:49.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.301+0000 7efdf8c00700 1 -- 172.21.15.123:0/1631307667 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7efde808d390 con 0x7efdfc1037a0 2024-03-30T05:02:49.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.302+0000 7efe00e00700 1 --2- 172.21.15.123:0/1631307667 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7efde406cc30 0x7efde406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:02:49.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.302+0000 7efe00e00700 1 --2- 172.21.15.123:0/1631307667 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7efde406cc30 0x7efde406f0e0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7efdf0004990 tx=0x7efdf0009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:02:49.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.306+0000 7efdf8c00700 1 -- 172.21.15.123:0/1631307667 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7efde805ade0 con 0x7efdfc1037a0 2024-03-30T05:02:49.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:02:49 smithi123 ceph-mon[139369]: pgmap v949: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:49.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.533+0000 7efe03800700 1 -- 172.21.15.123:0/1631307667 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7efdfc10f1c0 con 0x7efde406cc30 2024-03-30T05:02:49.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.534+0000 7efdf8c00700 1 -- 172.21.15.123:0/1631307667 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7efdfc10f1c0 con 0x7efde406cc30 2024-03-30T05:02:49.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.537+0000 7efe03800700 1 -- 172.21.15.123:0/1631307667 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7efde406cc30 msgr2=0x7efde406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:49.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.537+0000 7efe03800700 1 --2- 172.21.15.123:0/1631307667 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7efde406cc30 0x7efde406f0e0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7efdf0004990 tx=0x7efdf0009ee0 comp rx=0 tx=0).stop 2024-03-30T05:02:49.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.537+0000 7efe03800700 1 -- 172.21.15.123:0/1631307667 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efdfc1037a0 msgr2=0x7efdfc114040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:49.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.537+0000 7efe03800700 1 --2- 172.21.15.123:0/1631307667 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efdfc1037a0 0x7efdfc114040 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7efde8009390 tx=0x7efde8009e30 comp rx=0 tx=0).stop 2024-03-30T05:02:49.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.537+0000 7efe03800700 1 -- 172.21.15.123:0/1631307667 shutdown_connections 2024-03-30T05:02:49.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.537+0000 7efe03800700 1 --2- 172.21.15.123:0/1631307667 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7efde406cc30 0x7efde406f0e0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:49.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.537+0000 7efe03800700 1 --2- 172.21.15.123:0/1631307667 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efdfc102e00 0x7efdfc113b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:49.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.537+0000 7efe03800700 1 --2- 172.21.15.123:0/1631307667 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efdfc1037a0 0x7efdfc114040 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:49.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.537+0000 7efe03800700 1 -- 172.21.15.123:0/1631307667 >> 172.21.15.123:0/1631307667 conn(0x7efdfc0f89b0 msgr2=0x7efdfc0f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:02:49.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.537+0000 7efe03800700 1 -- 172.21.15.123:0/1631307667 shutdown_connections 2024-03-30T05:02:49.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.537+0000 7efe03800700 1 -- 172.21.15.123:0/1631307667 wait complete. 2024-03-30T05:02:49.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:02:49 smithi135 ceph-mon[106011]: pgmap v949: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:49.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.670+0000 7f8774a00700 1 -- 172.21.15.123:0/1354411690 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8770102c40 msgr2=0x7f8770103010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:49.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.670+0000 7f8774a00700 1 --2- 172.21.15.123:0/1354411690 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8770102c40 0x7f8770103010 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f8760004940 tx=0x7f8760009e30 comp rx=0 tx=0).stop 2024-03-30T05:02:49.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.670+0000 7f8774a00700 1 -- 172.21.15.123:0/1354411690 shutdown_connections 2024-03-30T05:02:49.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.670+0000 7f8774a00700 1 --2- 172.21.15.123:0/1354411690 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f87701035e0 0x7f87700fcc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:49.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.670+0000 7f8774a00700 1 --2- 172.21.15.123:0/1354411690 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8770102c40 0x7f8770103010 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:49.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.670+0000 7f8774a00700 1 -- 172.21.15.123:0/1354411690 >> 172.21.15.123:0/1354411690 conn(0x7f87700f8830 msgr2=0x7f87700fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:02:49.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.670+0000 7f8774a00700 1 -- 172.21.15.123:0/1354411690 shutdown_connections 2024-03-30T05:02:49.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.670+0000 7f8774a00700 1 -- 172.21.15.123:0/1354411690 wait complete. 2024-03-30T05:02:49.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.671+0000 7f8774a00700 1 Processor -- start 2024-03-30T05:02:49.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.671+0000 7f8774a00700 1 -- start start 2024-03-30T05:02:49.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.672+0000 7f8774a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8770102c40 0x7f8770115b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:02:49.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.672+0000 7f8774a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87701035e0 0x7f8770116070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:02:49.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.672+0000 7f8774a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87701166c0 con 0x7f87701035e0 2024-03-30T05:02:49.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.672+0000 7f8774a00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f877010fab0 con 0x7f8770102c40 2024-03-30T05:02:49.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.672+0000 7f8767e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87701035e0 0x7f8770116070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:02:49.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.672+0000 7f8767e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87701035e0 0x7f8770116070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43422/0 (socket says 172.21.15.123:43422) 2024-03-30T05:02:49.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.672+0000 7f8767e00700 1 -- 172.21.15.123:0/253257493 learned_addr learned my addr 172.21.15.123:0/253257493 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:02:49.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.672+0000 7f876e800700 1 --2- 172.21.15.123:0/253257493 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8770102c40 0x7f8770115b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:02:49.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.672+0000 7f8767e00700 1 -- 172.21.15.123:0/253257493 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8770102c40 msgr2=0x7f8770115b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:49.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.672+0000 7f8767e00700 1 --2- 172.21.15.123:0/253257493 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8770102c40 0x7f8770115b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:49.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.672+0000 7f8767e00700 1 -- 172.21.15.123:0/253257493 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8760005040 con 0x7f87701035e0 2024-03-30T05:02:49.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.672+0000 7f876e800700 1 --2- 172.21.15.123:0/253257493 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8770102c40 0x7f8770115b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T05:02:49.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.672+0000 7f8767e00700 1 --2- 172.21.15.123:0/253257493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87701035e0 0x7f8770116070 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f8758002a50 tx=0x7f8758004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:02:49.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.673+0000 7f8766a00700 1 -- 172.21.15.123:0/253257493 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8758007cc0 con 0x7f87701035e0 2024-03-30T05:02:49.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.673+0000 7f8766a00700 1 -- 172.21.15.123:0/253257493 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f875800f040 con 0x7f87701035e0 2024-03-30T05:02:49.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.673+0000 7f8774a00700 1 -- 172.21.15.123:0/253257493 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f877010fd90 con 0x7f87701035e0 2024-03-30T05:02:49.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.673+0000 7f8766a00700 1 -- 172.21.15.123:0/253257493 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f875800d9a0 con 0x7f87701035e0 2024-03-30T05:02:49.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.673+0000 7f8774a00700 1 -- 172.21.15.123:0/253257493 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8770110350 con 0x7f87701035e0 2024-03-30T05:02:49.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.674+0000 7f8774a00700 1 -- 172.21.15.123:0/253257493 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87700fe390 con 0x7f87701035e0 2024-03-30T05:02:49.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.675+0000 7f8766a00700 1 -- 172.21.15.123:0/253257493 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f875800dc60 con 0x7f87701035e0 2024-03-30T05:02:49.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.676+0000 7f8766a00700 1 --2- 172.21.15.123:0/253257493 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f875406cc30 0x7f875406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:02:49.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.676+0000 7f8766a00700 1 -- 172.21.15.123:0/253257493 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f875808bef0 con 0x7f87701035e0 2024-03-30T05:02:49.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.676+0000 7f876e800700 1 --2- 172.21.15.123:0/253257493 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f875406cc30 0x7f875406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:02:49.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.676+0000 7f876e800700 1 --2- 172.21.15.123:0/253257493 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f875406cc30 0x7f875406f0e0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f876000b840 tx=0x7f8760009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:02:49.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.680+0000 7f8766a00700 1 -- 172.21.15.123:0/253257493 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f87580598c0 con 0x7f87701035e0 2024-03-30T05:02:49.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.892+0000 7f8774a00700 1 -- 172.21.15.123:0/253257493 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f8770109c60 con 0x7f875406cc30 2024-03-30T05:02:49.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.902+0000 7f8766a00700 1 -- 172.21.15.123:0/253257493 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f8770109c60 con 0x7f875406cc30 2024-03-30T05:02:49.900 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T05:02:49.900 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (31m) 9m ago 44m 14.6M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T05:02:49.900 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (29m) 9m ago 44m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T05:02:49.900 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (29m) 9m ago 43m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T05:02:49.900 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (31m) 9m ago 44m 73.5M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T05:02:49.900 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (25m) 9m ago 38m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T05:02:49.900 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (25m) 9m ago 38m 16.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T05:02:49.900 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (9m) 9m ago 38m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 87b073a6c9ca 2024-03-30T05:02:49.901 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (24m) 9m ago 38m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T05:02:49.901 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (34m) 9m ago 46m 510M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T05:02:49.901 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (33m) 9m ago 43m 456M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T05:02:49.901 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (30m) 9m ago 46m 70.0M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T05:02:49.901 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (30m) 9m ago 43m 51.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T05:02:49.901 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (32m) 9m ago 44m 8636k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T05:02:49.901 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (32m) 9m ago 43m 8678k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T05:02:49.901 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (28m) 9m ago 42m 107M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T05:02:49.901 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (28m) 9m ago 41m 97.2M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T05:02:49.901 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (27m) 9m ago 41m 89.8M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T05:02:49.901 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (26m) 9m ago 40m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T05:02:49.901 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (26m) 9m ago 40m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T05:02:49.901 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (25m) 9m ago 39m 88.8M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T05:02:49.901 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (32m) 9m ago 44m 63.9M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T05:02:49.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.904+0000 7f8774a00700 1 -- 172.21.15.123:0/253257493 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f875406cc30 msgr2=0x7f875406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:49.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.904+0000 7f8774a00700 1 --2- 172.21.15.123:0/253257493 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f875406cc30 0x7f875406f0e0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f876000b840 tx=0x7f8760009e30 comp rx=0 tx=0).stop 2024-03-30T05:02:49.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.904+0000 7f8774a00700 1 -- 172.21.15.123:0/253257493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87701035e0 msgr2=0x7f8770116070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:49.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.904+0000 7f8774a00700 1 --2- 172.21.15.123:0/253257493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87701035e0 0x7f8770116070 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f8758002a50 tx=0x7f8758004600 comp rx=0 tx=0).stop 2024-03-30T05:02:49.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.904+0000 7f8774a00700 1 -- 172.21.15.123:0/253257493 shutdown_connections 2024-03-30T05:02:49.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.904+0000 7f8774a00700 1 --2- 172.21.15.123:0/253257493 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f875406cc30 0x7f875406f0e0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:49.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.904+0000 7f8774a00700 1 --2- 172.21.15.123:0/253257493 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8770102c40 0x7f8770115b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:49.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.904+0000 7f8774a00700 1 --2- 172.21.15.123:0/253257493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87701035e0 0x7f8770116070 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:49.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.904+0000 7f8774a00700 1 -- 172.21.15.123:0/253257493 >> 172.21.15.123:0/253257493 conn(0x7f87700f8830 msgr2=0x7f87700f9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:02:49.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.904+0000 7f8774a00700 1 -- 172.21.15.123:0/253257493 shutdown_connections 2024-03-30T05:02:49.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:49.904+0000 7f8774a00700 1 -- 172.21.15.123:0/253257493 wait complete. 2024-03-30T05:02:50.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.038+0000 7fc969200700 1 -- 172.21.15.123:0/2029426473 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc9640fde90 msgr2=0x7fc9640fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:50.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.038+0000 7fc969200700 1 --2- 172.21.15.123:0/2029426473 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc9640fde90 0x7fc9640fe260 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fc954004940 tx=0x7fc954009e30 comp rx=0 tx=0).stop 2024-03-30T05:02:50.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.038+0000 7fc969200700 1 -- 172.21.15.123:0/2029426473 shutdown_connections 2024-03-30T05:02:50.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.038+0000 7fc969200700 1 --2- 172.21.15.123:0/2029426473 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc9640fe830 0x7fc964106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:50.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.038+0000 7fc969200700 1 --2- 172.21.15.123:0/2029426473 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc9640fde90 0x7fc9640fe260 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:50.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.038+0000 7fc969200700 1 -- 172.21.15.123:0/2029426473 >> 172.21.15.123:0/2029426473 conn(0x7fc964072bc0 msgr2=0x7fc964072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:02:50.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.038+0000 7fc969200700 1 -- 172.21.15.123:0/2029426473 shutdown_connections 2024-03-30T05:02:50.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.038+0000 7fc969200700 1 -- 172.21.15.123:0/2029426473 wait complete. 2024-03-30T05:02:50.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.039+0000 7fc969200700 1 Processor -- start 2024-03-30T05:02:50.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.039+0000 7fc969200700 1 -- start start 2024-03-30T05:02:50.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.039+0000 7fc969200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc9640fde90 0x7fc9640fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:02:50.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.039+0000 7fc969200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc9640fe830 0x7fc9640fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:02:50.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.039+0000 7fc969200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9640fc820 con 0x7fc9640fe830 2024-03-30T05:02:50.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.039+0000 7fc969200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9640fc960 con 0x7fc9640fde90 2024-03-30T05:02:50.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.040+0000 7fc961400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc9640fe830 0x7fc9640fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:02:50.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.040+0000 7fc961400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc9640fe830 0x7fc9640fc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43428/0 (socket says 172.21.15.123:43428) 2024-03-30T05:02:50.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.040+0000 7fc961400700 1 -- 172.21.15.123:0/1579428993 learned_addr learned my addr 172.21.15.123:0/1579428993 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:02:50.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.040+0000 7fc961e00700 1 --2- 172.21.15.123:0/1579428993 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc9640fde90 0x7fc9640fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:02:50.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.040+0000 7fc961400700 1 -- 172.21.15.123:0/1579428993 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc9640fde90 msgr2=0x7fc9640fbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:50.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.040+0000 7fc961400700 1 --2- 172.21.15.123:0/1579428993 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc9640fde90 0x7fc9640fbd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:50.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.040+0000 7fc961400700 1 -- 172.21.15.123:0/1579428993 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc954005040 con 0x7fc9640fe830 2024-03-30T05:02:50.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.040+0000 7fc961400700 1 --2- 172.21.15.123:0/1579428993 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc9640fe830 0x7fc9640fc290 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fc95000c8c0 tx=0x7fc9500044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:02:50.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.040+0000 7fc95a000700 1 -- 172.21.15.123:0/1579428993 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc95000a980 con 0x7fc9640fe830 2024-03-30T05:02:50.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.040+0000 7fc95a000700 1 -- 172.21.15.123:0/1579428993 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc95000ce60 con 0x7fc9640fe830 2024-03-30T05:02:50.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.041+0000 7fc969200700 1 -- 172.21.15.123:0/1579428993 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc964119df0 con 0x7fc9640fe830 2024-03-30T05:02:50.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.041+0000 7fc95a000700 1 -- 172.21.15.123:0/1579428993 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc9500113a0 con 0x7fc9640fe830 2024-03-30T05:02:50.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.041+0000 7fc969200700 1 -- 172.21.15.123:0/1579428993 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc96411a1f0 con 0x7fc9640fe830 2024-03-30T05:02:50.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.042+0000 7fc969200700 1 -- 172.21.15.123:0/1579428993 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc964065e30 con 0x7fc9640fe830 2024-03-30T05:02:50.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.043+0000 7fc95a000700 1 -- 172.21.15.123:0/1579428993 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fc950012400 con 0x7fc9640fe830 2024-03-30T05:02:50.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.043+0000 7fc95a000700 1 --2- 172.21.15.123:0/1579428993 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc94806cc30 0x7fc94806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:02:50.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.044+0000 7fc95a000700 1 -- 172.21.15.123:0/1579428993 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc95008c1e0 con 0x7fc9640fe830 2024-03-30T05:02:50.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.044+0000 7fc961e00700 1 --2- 172.21.15.123:0/1579428993 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc94806cc30 0x7fc94806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:02:50.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.044+0000 7fc961e00700 1 --2- 172.21.15.123:0/1579428993 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc94806cc30 0x7fc94806f0e0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fc95400b3b0 tx=0x7fc954009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:02:50.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.048+0000 7fc95a000700 1 -- 172.21.15.123:0/1579428993 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc9500561a0 con 0x7fc9640fe830 2024-03-30T05:02:50.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.338+0000 7fc969200700 1 -- 172.21.15.123:0/1579428993 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc96404bed0 con 0x7fc9640fe830 2024-03-30T05:02:50.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.338+0000 7fc95a000700 1 -- 172.21.15.123:0/1579428993 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7fc9500597c0 con 0x7fc9640fe830 2024-03-30T05:02:50.337 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T05:02:50.337 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T05:02:50.337 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T05:02:50.337 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:02:50.337 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T05:02:50.337 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T05:02:50.337 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:02:50.337 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T05:02:50.337 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T05:02:50.338 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:02:50.338 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T05:02:50.338 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 4 2024-03-30T05:02:50.338 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:02:50.338 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T05:02:50.338 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 14 2024-03-30T05:02:50.338 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T05:02:50.338 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T05:02:50.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.341+0000 7fc969200700 1 -- 172.21.15.123:0/1579428993 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc94806cc30 msgr2=0x7fc94806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:50.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.341+0000 7fc969200700 1 --2- 172.21.15.123:0/1579428993 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc94806cc30 0x7fc94806f0e0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fc95400b3b0 tx=0x7fc954009e30 comp rx=0 tx=0).stop 2024-03-30T05:02:50.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.341+0000 7fc969200700 1 -- 172.21.15.123:0/1579428993 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc9640fe830 msgr2=0x7fc9640fc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:50.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.341+0000 7fc969200700 1 --2- 172.21.15.123:0/1579428993 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc9640fe830 0x7fc9640fc290 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fc95000c8c0 tx=0x7fc9500044c0 comp rx=0 tx=0).stop 2024-03-30T05:02:50.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.341+0000 7fc969200700 1 -- 172.21.15.123:0/1579428993 shutdown_connections 2024-03-30T05:02:50.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.341+0000 7fc969200700 1 --2- 172.21.15.123:0/1579428993 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc94806cc30 0x7fc94806f0e0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:50.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.341+0000 7fc969200700 1 --2- 172.21.15.123:0/1579428993 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc9640fde90 0x7fc9640fbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:50.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.341+0000 7fc969200700 1 --2- 172.21.15.123:0/1579428993 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc9640fe830 0x7fc9640fc290 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:50.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.341+0000 7fc969200700 1 -- 172.21.15.123:0/1579428993 >> 172.21.15.123:0/1579428993 conn(0x7fc964072bc0 msgr2=0x7fc9640ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:02:50.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.341+0000 7fc969200700 1 -- 172.21.15.123:0/1579428993 shutdown_connections 2024-03-30T05:02:50.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.341+0000 7fc969200700 1 -- 172.21.15.123:0/1579428993 wait complete. 2024-03-30T05:02:50.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:02:50 smithi123 ceph-mon[139369]: from='client.35720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:02:50.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.474+0000 7f3de0c00700 1 -- 172.21.15.123:0/953156098 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ddc102e00 msgr2=0x7f3ddc1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:50.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.474+0000 7f3de0c00700 1 --2- 172.21.15.123:0/953156098 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ddc102e00 0x7f3ddc1031d0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc004940 tx=0x7f3dcc009e30 comp rx=0 tx=0).stop 2024-03-30T05:02:50.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.474+0000 7f3de0c00700 1 -- 172.21.15.123:0/953156098 shutdown_connections 2024-03-30T05:02:50.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.474+0000 7f3de0c00700 1 --2- 172.21.15.123:0/953156098 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3ddc1037a0 0x7f3ddc0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:50.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.474+0000 7f3de0c00700 1 --2- 172.21.15.123:0/953156098 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ddc102e00 0x7f3ddc1031d0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:50.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.474+0000 7f3de0c00700 1 -- 172.21.15.123:0/953156098 >> 172.21.15.123:0/953156098 conn(0x7f3ddc0f89b0 msgr2=0x7f3ddc0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:02:50.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.475+0000 7f3de0c00700 1 -- 172.21.15.123:0/953156098 shutdown_connections 2024-03-30T05:02:50.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.475+0000 7f3de0c00700 1 -- 172.21.15.123:0/953156098 wait complete. 2024-03-30T05:02:50.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.475+0000 7f3de0c00700 1 Processor -- start 2024-03-30T05:02:50.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.476+0000 7f3de0c00700 1 -- start start 2024-03-30T05:02:50.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.476+0000 7f3de0c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ddc102e00 0x7f3ddc115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:02:50.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.476+0000 7f3de0c00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3ddc1037a0 0x7f3ddc116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:02:50.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.476+0000 7f3de0c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ddc116860 con 0x7f3ddc102e00 2024-03-30T05:02:50.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.476+0000 7f3de0c00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ddc10fc30 con 0x7f3ddc1037a0 2024-03-30T05:02:50.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.476+0000 7f3dd9e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ddc102e00 0x7f3ddc115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:02:50.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.476+0000 7f3dd9e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ddc102e00 0x7f3ddc115cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43432/0 (socket says 172.21.15.123:43432) 2024-03-30T05:02:50.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.476+0000 7f3dd9e00700 1 -- 172.21.15.123:0/2496480396 learned_addr learned my addr 172.21.15.123:0/2496480396 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:02:50.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.476+0000 7f3dd9400700 1 --2- 172.21.15.123:0/2496480396 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3ddc1037a0 0x7f3ddc116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:02:50.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.477+0000 7f3dd9e00700 1 -- 172.21.15.123:0/2496480396 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3ddc1037a0 msgr2=0x7f3ddc116210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:50.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.477+0000 7f3dd9e00700 1 --2- 172.21.15.123:0/2496480396 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3ddc1037a0 0x7f3ddc116210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:50.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.477+0000 7f3dd9e00700 1 -- 172.21.15.123:0/2496480396 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dcc005040 con 0x7f3ddc102e00 2024-03-30T05:02:50.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.477+0000 7f3dd9e00700 1 --2- 172.21.15.123:0/2496480396 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ddc102e00 0x7f3ddc115cd0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc009360 tx=0x7f3dcc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:02:50.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.477+0000 7f3dd2000700 1 -- 172.21.15.123:0/2496480396 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3dcc00e040 con 0x7f3ddc102e00 2024-03-30T05:02:50.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.477+0000 7f3dd2000700 1 -- 172.21.15.123:0/2496480396 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3dcc013420 con 0x7f3ddc102e00 2024-03-30T05:02:50.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.477+0000 7f3de0c00700 1 -- 172.21.15.123:0/2496480396 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ddc10feb0 con 0x7f3ddc102e00 2024-03-30T05:02:50.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.477+0000 7f3dd2000700 1 -- 172.21.15.123:0/2496480396 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3dcc01c5d0 con 0x7f3ddc102e00 2024-03-30T05:02:50.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.477+0000 7f3de0c00700 1 -- 172.21.15.123:0/2496480396 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ddc110470 con 0x7f3ddc102e00 2024-03-30T05:02:50.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.478+0000 7f3de0c00700 1 -- 172.21.15.123:0/2496480396 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ddc0fe550 con 0x7f3ddc102e00 2024-03-30T05:02:50.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.479+0000 7f3dd2000700 1 -- 172.21.15.123:0/2496480396 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f3dcc013590 con 0x7f3ddc102e00 2024-03-30T05:02:50.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.480+0000 7f3dd2000700 1 --2- 172.21.15.123:0/2496480396 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3dc006cc30 0x7f3dc006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:02:50.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.480+0000 7f3dd2000700 1 -- 172.21.15.123:0/2496480396 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f3dcc08d250 con 0x7f3ddc102e00 2024-03-30T05:02:50.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.480+0000 7f3dd9400700 1 --2- 172.21.15.123:0/2496480396 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3dc006cc30 0x7f3dc006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:02:50.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.481+0000 7f3dd9400700 1 --2- 172.21.15.123:0/2496480396 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3dc006cc30 0x7f3dc006f0e0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc111570 tx=0x7f3dc800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:02:50.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.485+0000 7f3dd2000700 1 -- 172.21.15.123:0/2496480396 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3dcc05ac20 con 0x7f3ddc102e00 2024-03-30T05:02:50.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:02:50 smithi135 ceph-mon[106011]: from='client.35720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:02:50.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.730+0000 7f3de0c00700 1 -- 172.21.15.123:0/2496480396 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f3ddc04bf00 con 0x7f3ddc102e00 2024-03-30T05:02:50.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.730+0000 7f3dd2000700 1 -- 172.21.15.123:0/2496480396 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 34 v34) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f3dcc05a7b0 con 0x7f3ddc102e00 2024-03-30T05:02:50.729 INFO:teuthology.orchestra.run.smithi123.stdout:e34 2024-03-30T05:02:50.729 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T05:02:50.729 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T05:02:50.729 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T05:02:50.729 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:02:50.729 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T05:02:50.729 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T05:02:50.729 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T05:02:50.729 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T05:02:50.729 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T05:02:50.729 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T05:02:50.729 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T05:02:50.729 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T05:02:50.730 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T05:02:50.730 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T05:02:50.730 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T05:02:50.730 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T05:02:50.730 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T05:02:50.730 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T05:02:50.730 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T05:02:50.730 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T05:02:50.730 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T05:02:50.730 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T05:02:50.730 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T05:02:50.730 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T05:02:50.730 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T05:02:50.731 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T05:02:50.731 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T05:02:50.731 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T05:02:50.731 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T05:02:50.731 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T05:02:50.731 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T05:02:50.731 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T05:02:50.731 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:02:50.731 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:02:50.731 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T05:02:50.731 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:02:50.731 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:02:50.731 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:02:50.731 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:02:50.732 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44975} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/3788988014,v1:172.21.15.135:6827/3788988014] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:02:50.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.732+0000 7f3de0c00700 1 -- 172.21.15.123:0/2496480396 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3dc006cc30 msgr2=0x7f3dc006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:50.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.732+0000 7f3de0c00700 1 --2- 172.21.15.123:0/2496480396 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3dc006cc30 0x7f3dc006f0e0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc111570 tx=0x7f3dc800b040 comp rx=0 tx=0).stop 2024-03-30T05:02:50.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.732+0000 7f3de0c00700 1 -- 172.21.15.123:0/2496480396 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ddc102e00 msgr2=0x7f3ddc115cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:50.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.732+0000 7f3de0c00700 1 --2- 172.21.15.123:0/2496480396 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ddc102e00 0x7f3ddc115cd0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc009360 tx=0x7f3dcc009e30 comp rx=0 tx=0).stop 2024-03-30T05:02:50.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.733+0000 7f3de0c00700 1 -- 172.21.15.123:0/2496480396 shutdown_connections 2024-03-30T05:02:50.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.733+0000 7f3de0c00700 1 --2- 172.21.15.123:0/2496480396 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3dc006cc30 0x7f3dc006f0e0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:50.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.733+0000 7f3de0c00700 1 --2- 172.21.15.123:0/2496480396 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ddc102e00 0x7f3ddc115cd0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:50.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.733+0000 7f3de0c00700 1 --2- 172.21.15.123:0/2496480396 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3ddc1037a0 0x7f3ddc116210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:50.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.733+0000 7f3de0c00700 1 -- 172.21.15.123:0/2496480396 >> 172.21.15.123:0/2496480396 conn(0x7f3ddc0f89b0 msgr2=0x7f3ddc0f99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:02:50.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.733+0000 7f3de0c00700 1 -- 172.21.15.123:0/2496480396 shutdown_connections 2024-03-30T05:02:50.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.733+0000 7f3de0c00700 1 -- 172.21.15.123:0/2496480396 wait complete. 2024-03-30T05:02:50.733 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 34 2024-03-30T05:02:50.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.867+0000 7f9c4a400700 1 -- 172.21.15.123:0/3946778464 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c44070c10 msgr2=0x7f9c44109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:50.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.867+0000 7f9c4a400700 1 --2- 172.21.15.123:0/3946778464 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c44070c10 0x7f9c44109190 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f9c30004970 tx=0x7f9c30009e30 comp rx=0 tx=0).stop 2024-03-30T05:02:50.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.867+0000 7f9c4a400700 1 -- 172.21.15.123:0/3946778464 shutdown_connections 2024-03-30T05:02:50.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.867+0000 7f9c4a400700 1 --2- 172.21.15.123:0/3946778464 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c44070c10 0x7f9c44109190 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:50.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.867+0000 7f9c4a400700 1 --2- 172.21.15.123:0/3946778464 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c44070300 0x7f9c440706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:50.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.867+0000 7f9c4a400700 1 -- 172.21.15.123:0/3946778464 >> 172.21.15.123:0/3946778464 conn(0x7f9c440f89d0 msgr2=0x7f9c440fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:02:50.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.867+0000 7f9c4a400700 1 -- 172.21.15.123:0/3946778464 shutdown_connections 2024-03-30T05:02:50.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.867+0000 7f9c4a400700 1 -- 172.21.15.123:0/3946778464 wait complete. 2024-03-30T05:02:50.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.868+0000 7f9c4a400700 1 Processor -- start 2024-03-30T05:02:50.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.868+0000 7f9c4a400700 1 -- start start 2024-03-30T05:02:50.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.868+0000 7f9c4a400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c44070300 0x7f9c44115ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:02:50.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.868+0000 7f9c4a400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c44070c10 0x7f9c441161e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:02:50.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.868+0000 7f9c4a400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c44116830 con 0x7f9c44070300 2024-03-30T05:02:50.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.868+0000 7f9c4a400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4410fc00 con 0x7f9c44070c10 2024-03-30T05:02:50.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.869+0000 7f9c43400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c44070300 0x7f9c44115ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:02:50.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.869+0000 7f9c43400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c44070300 0x7f9c44115ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43454/0 (socket says 172.21.15.123:43454) 2024-03-30T05:02:50.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.869+0000 7f9c43400700 1 -- 172.21.15.123:0/3937310754 learned_addr learned my addr 172.21.15.123:0/3937310754 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:02:50.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.869+0000 7f9c42a00700 1 --2- 172.21.15.123:0/3937310754 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c44070c10 0x7f9c441161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:02:50.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.869+0000 7f9c43400700 1 -- 172.21.15.123:0/3937310754 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c44070c10 msgr2=0x7f9c441161e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:50.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.869+0000 7f9c43400700 1 --2- 172.21.15.123:0/3937310754 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c44070c10 0x7f9c441161e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:50.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.869+0000 7f9c43400700 1 -- 172.21.15.123:0/3937310754 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c30005040 con 0x7f9c44070300 2024-03-30T05:02:50.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.869+0000 7f9c43400700 1 --2- 172.21.15.123:0/3937310754 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c44070300 0x7f9c44115ca0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f9c3400c830 tx=0x7f9c34004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:02:50.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.869+0000 7f9c3b400700 1 -- 172.21.15.123:0/3937310754 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c340106f0 con 0x7f9c44070300 2024-03-30T05:02:50.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.870+0000 7f9c3b400700 1 -- 172.21.15.123:0/3937310754 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9c34010d00 con 0x7f9c44070300 2024-03-30T05:02:50.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.870+0000 7f9c3b400700 1 -- 172.21.15.123:0/3937310754 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c34012e90 con 0x7f9c44070300 2024-03-30T05:02:50.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.870+0000 7f9c4a400700 1 -- 172.21.15.123:0/3937310754 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c4410fee0 con 0x7f9c44070300 2024-03-30T05:02:50.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.870+0000 7f9c4a400700 1 -- 172.21.15.123:0/3937310754 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c441103f0 con 0x7f9c44070300 2024-03-30T05:02:50.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.871+0000 7f9c4a400700 1 -- 172.21.15.123:0/3937310754 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c44071a40 con 0x7f9c44070300 2024-03-30T05:02:50.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.872+0000 7f9c3b400700 1 -- 172.21.15.123:0/3937310754 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f9c34010850 con 0x7f9c44070300 2024-03-30T05:02:50.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.873+0000 7f9c3b400700 1 --2- 172.21.15.123:0/3937310754 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9c2806cc30 0x7f9c2806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:02:50.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.873+0000 7f9c3b400700 1 -- 172.21.15.123:0/3937310754 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f9c3408c0b0 con 0x7f9c44070300 2024-03-30T05:02:50.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.873+0000 7f9c42a00700 1 --2- 172.21.15.123:0/3937310754 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9c2806cc30 0x7f9c2806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:02:50.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.873+0000 7f9c42a00700 1 --2- 172.21.15.123:0/3937310754 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9c2806cc30 0x7f9c2806f0e0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f9c30009fd0 tx=0x7f9c30009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:02:50.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:50.877+0000 7f9c3b400700 1 -- 172.21.15.123:0/3937310754 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9c34059b00 con 0x7f9c44070300 2024-03-30T05:02:51.102 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.104+0000 7f9c4a400700 1 -- 172.21.15.123:0/3937310754 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9c4404bf40 con 0x7f9c2806cc30 2024-03-30T05:02:51.104 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.106+0000 7f9c3b400700 1 -- 172.21.15.123:0/3937310754 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f9c4404bf40 con 0x7f9c2806cc30 2024-03-30T05:02:51.104 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T05:02:51.104 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T05:02:51.104 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T05:02:51.104 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T05:02:51.104 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T05:02:51.104 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T05:02:51.104 INFO:teuthology.orchestra.run.smithi123.stdout: "mds", 2024-03-30T05:02:51.104 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T05:02:51.105 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T05:02:51.105 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T05:02:51.105 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T05:02:51.105 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "16/21 daemons upgraded", 2024-03-30T05:02:51.105 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T05:02:51.105 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T05:02:51.105 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T05:02:51.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.108+0000 7f9c4a400700 1 -- 172.21.15.123:0/3937310754 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9c2806cc30 msgr2=0x7f9c2806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:51.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.108+0000 7f9c4a400700 1 --2- 172.21.15.123:0/3937310754 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9c2806cc30 0x7f9c2806f0e0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f9c30009fd0 tx=0x7f9c30009e30 comp rx=0 tx=0).stop 2024-03-30T05:02:51.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.108+0000 7f9c4a400700 1 -- 172.21.15.123:0/3937310754 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c44070300 msgr2=0x7f9c44115ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:51.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.108+0000 7f9c4a400700 1 --2- 172.21.15.123:0/3937310754 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c44070300 0x7f9c44115ca0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f9c3400c830 tx=0x7f9c34004470 comp rx=0 tx=0).stop 2024-03-30T05:02:51.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.108+0000 7f9c4a400700 1 -- 172.21.15.123:0/3937310754 shutdown_connections 2024-03-30T05:02:51.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.108+0000 7f9c4a400700 1 --2- 172.21.15.123:0/3937310754 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9c2806cc30 0x7f9c2806f0e0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:51.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.108+0000 7f9c4a400700 1 --2- 172.21.15.123:0/3937310754 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c44070300 0x7f9c44115ca0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:51.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.108+0000 7f9c4a400700 1 --2- 172.21.15.123:0/3937310754 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c44070c10 0x7f9c441161e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:51.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.108+0000 7f9c4a400700 1 -- 172.21.15.123:0/3937310754 >> 172.21.15.123:0/3937310754 conn(0x7f9c440f89d0 msgr2=0x7f9c440f9f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:02:51.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.108+0000 7f9c4a400700 1 -- 172.21.15.123:0/3937310754 shutdown_connections 2024-03-30T05:02:51.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.108+0000 7f9c4a400700 1 -- 172.21.15.123:0/3937310754 wait complete. 2024-03-30T05:02:51.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.244+0000 7fde7f200700 1 -- 172.21.15.123:0/1949327747 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde781037a0 msgr2=0x7fde780fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:51.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.244+0000 7fde7f200700 1 --2- 172.21.15.123:0/1949327747 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde781037a0 0x7fde780fce10 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fde64004940 tx=0x7fde64009e30 comp rx=0 tx=0).stop 2024-03-30T05:02:51.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.244+0000 7fde7f200700 1 -- 172.21.15.123:0/1949327747 shutdown_connections 2024-03-30T05:02:51.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.244+0000 7fde7f200700 1 --2- 172.21.15.123:0/1949327747 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde781037a0 0x7fde780fce10 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:51.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.244+0000 7fde7f200700 1 --2- 172.21.15.123:0/1949327747 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fde78102e00 0x7fde781031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:51.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.244+0000 7fde7f200700 1 -- 172.21.15.123:0/1949327747 >> 172.21.15.123:0/1949327747 conn(0x7fde780f89b0 msgr2=0x7fde780fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:02:51.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.244+0000 7fde7f200700 1 -- 172.21.15.123:0/1949327747 shutdown_connections 2024-03-30T05:02:51.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.244+0000 7fde7f200700 1 -- 172.21.15.123:0/1949327747 wait complete. 2024-03-30T05:02:51.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.245+0000 7fde7f200700 1 Processor -- start 2024-03-30T05:02:51.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.245+0000 7fde7f200700 1 -- start start 2024-03-30T05:02:51.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.245+0000 7fde7f200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde78102e00 0x7fde780702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:02:51.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.245+0000 7fde7f200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fde781037a0 0x7fde78070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:02:51.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.245+0000 7fde7f200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde78070d90 con 0x7fde78102e00 2024-03-30T05:02:51.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.245+0000 7fde7f200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde78070ed0 con 0x7fde781037a0 2024-03-30T05:02:51.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.245+0000 7fde77e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde78102e00 0x7fde780702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:02:51.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.245+0000 7fde77e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde78102e00 0x7fde780702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43474/0 (socket says 172.21.15.123:43474) 2024-03-30T05:02:51.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.245+0000 7fde77e00700 1 -- 172.21.15.123:0/1674178243 learned_addr learned my addr 172.21.15.123:0/1674178243 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:02:51.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.246+0000 7fde77400700 1 --2- 172.21.15.123:0/1674178243 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fde781037a0 0x7fde78070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:02:51.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.246+0000 7fde77e00700 1 -- 172.21.15.123:0/1674178243 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fde781037a0 msgr2=0x7fde78070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:51.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.246+0000 7fde77e00700 1 --2- 172.21.15.123:0/1674178243 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fde781037a0 0x7fde78070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:51.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.246+0000 7fde77e00700 1 -- 172.21.15.123:0/1674178243 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde64005040 con 0x7fde78102e00 2024-03-30T05:02:51.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.246+0000 7fde77e00700 1 --2- 172.21.15.123:0/1674178243 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde78102e00 0x7fde780702c0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fde68002a50 tx=0x7fde680046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:02:51.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.246+0000 7fde6fe00700 1 -- 172.21.15.123:0/1674178243 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde68007cc0 con 0x7fde78102e00 2024-03-30T05:02:51.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.246+0000 7fde6fe00700 1 -- 172.21.15.123:0/1674178243 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fde6800f040 con 0x7fde78102e00 2024-03-30T05:02:51.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.246+0000 7fde7f200700 1 -- 172.21.15.123:0/1674178243 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde78119e00 con 0x7fde78102e00 2024-03-30T05:02:51.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.246+0000 7fde6fe00700 1 -- 172.21.15.123:0/1674178243 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde6800d9c0 con 0x7fde78102e00 2024-03-30T05:02:51.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.246+0000 7fde7f200700 1 -- 172.21.15.123:0/1674178243 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde7811a280 con 0x7fde78102e00 2024-03-30T05:02:51.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.248+0000 7fde7f200700 1 -- 172.21.15.123:0/1674178243 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde780fe550 con 0x7fde78102e00 2024-03-30T05:02:51.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.249+0000 7fde6fe00700 1 -- 172.21.15.123:0/1674178243 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fde6800db20 con 0x7fde78102e00 2024-03-30T05:02:51.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.249+0000 7fde6fe00700 1 --2- 172.21.15.123:0/1674178243 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fde5c06cc30 0x7fde5c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:02:51.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.250+0000 7fde6fe00700 1 -- 172.21.15.123:0/1674178243 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fde6808bd00 con 0x7fde78102e00 2024-03-30T05:02:51.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.250+0000 7fde77400700 1 --2- 172.21.15.123:0/1674178243 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fde5c06cc30 0x7fde5c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:02:51.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.250+0000 7fde77400700 1 --2- 172.21.15.123:0/1674178243 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fde5c06cc30 0x7fde5c06f0e0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fde78071c00 tx=0x7fde64009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:02:51.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.254+0000 7fde6fe00700 1 -- 172.21.15.123:0/1674178243 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fde68055cc0 con 0x7fde78102e00 2024-03-30T05:02:51.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:02:51 smithi123 ceph-mon[139369]: from='client.35724 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:02:51.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:02:51 smithi123 ceph-mon[139369]: pgmap v950: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:51.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:02:51 smithi123 ceph-mon[139369]: from='client.35728 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:02:51.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:02:51 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/1579428993' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:02:51.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:02:51 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2496480396' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T05:02:51.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.536+0000 7fde7f200700 1 -- 172.21.15.123:0/1674178243 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fde7805f5f0 con 0x7fde78102e00 2024-03-30T05:02:51.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.536+0000 7fde6fe00700 1 -- 172.21.15.123:0/1674178243 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7fde7805f5f0 con 0x7fde78102e00 2024-03-30T05:02:51.535 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T05:02:51.535 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-30T05:02:51.535 INFO:teuthology.orchestra.run.smithi123.stdout: host smithi135 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi135/config 2024-03-30T05:02:51.535 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-30T05:02:51.535 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T05:02:51.535 INFO:teuthology.orchestra.run.smithi123.stdout:Traceback (most recent call last): 2024-03-30T05:02:51.535 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-30T05:02:51.535 INFO:teuthology.orchestra.run.smithi123.stdout: "__main__", mod_spec) 2024-03-30T05:02:51.535 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-30T05:02:51.535 INFO:teuthology.orchestra.run.smithi123.stdout: exec(code, run_globals) 2024-03-30T05:02:51.536 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10889, in 2024-03-30T05:02:51.536 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10877, in main 2024-03-30T05:02:51.536 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-30T05:02:51.536 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-30T05:02:51.536 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-30T05:02:51.536 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-30T05:02:51.536 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-03-30T05:02:51.536 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-30T05:02:51.536 INFO:teuthology.orchestra.run.smithi123.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T05:02:51.536 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T05:02:51.536 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T05:02:51.536 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T05:02:51.536 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T05:02:51.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.539+0000 7fde7f200700 1 -- 172.21.15.123:0/1674178243 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fde5c06cc30 msgr2=0x7fde5c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:51.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.539+0000 7fde7f200700 1 --2- 172.21.15.123:0/1674178243 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fde5c06cc30 0x7fde5c06f0e0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fde78071c00 tx=0x7fde64009e30 comp rx=0 tx=0).stop 2024-03-30T05:02:51.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.539+0000 7fde7f200700 1 -- 172.21.15.123:0/1674178243 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde78102e00 msgr2=0x7fde780702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:02:51.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.539+0000 7fde7f200700 1 --2- 172.21.15.123:0/1674178243 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde78102e00 0x7fde780702c0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fde68002a50 tx=0x7fde680046b0 comp rx=0 tx=0).stop 2024-03-30T05:02:51.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.539+0000 7fde7f200700 1 -- 172.21.15.123:0/1674178243 shutdown_connections 2024-03-30T05:02:51.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.539+0000 7fde7f200700 1 --2- 172.21.15.123:0/1674178243 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fde5c06cc30 0x7fde5c06f0e0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:51.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.539+0000 7fde7f200700 1 --2- 172.21.15.123:0/1674178243 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde78102e00 0x7fde780702c0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:51.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.539+0000 7fde7f200700 1 --2- 172.21.15.123:0/1674178243 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fde781037a0 0x7fde78070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:02:51.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.539+0000 7fde7f200700 1 -- 172.21.15.123:0/1674178243 >> 172.21.15.123:0/1674178243 conn(0x7fde780f89b0 msgr2=0x7fde780f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:02:51.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.539+0000 7fde7f200700 1 -- 172.21.15.123:0/1674178243 shutdown_connections 2024-03-30T05:02:51.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:02:51.539+0000 7fde7f200700 1 -- 172.21.15.123:0/1674178243 wait complete. 2024-03-30T05:02:51.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:02:51 smithi135 ceph-mon[106011]: from='client.35724 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:02:51.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:02:51 smithi135 ceph-mon[106011]: pgmap v950: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:51.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:02:51 smithi135 ceph-mon[106011]: from='client.35728 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:02:51.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:02:51 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/1579428993' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:02:51.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:02:51 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2496480396' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T05:02:52.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:02:52 smithi123 ceph-mon[139369]: from='client.35740 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:02:52.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:02:52 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/1674178243' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T05:02:52.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:02:52 smithi135 ceph-mon[106011]: from='client.35740 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:02:52.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:02:52 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/1674178243' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T05:02:53.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:02:53 smithi123 ceph-mon[139369]: pgmap v951: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:53.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:02:53 smithi135 ceph-mon[106011]: pgmap v951: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:55.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:02:55 smithi123 ceph-mon[139369]: pgmap v952: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:55.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:02:55 smithi135 ceph-mon[106011]: pgmap v952: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:57.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:02:57 smithi123 ceph-mon[139369]: pgmap v953: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:57.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:02:57 smithi135 ceph-mon[106011]: pgmap v953: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:59.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:02:59 smithi123 ceph-mon[139369]: pgmap v954: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:02:59.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:02:59 smithi135 ceph-mon[106011]: pgmap v954: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:00.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:03:00 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:03:00.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:03:00 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:03:01.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:03:01 smithi123 ceph-mon[139369]: pgmap v955: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:01.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:03:01 smithi135 ceph-mon[106011]: pgmap v955: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:03.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:03:03 smithi123 ceph-mon[139369]: pgmap v956: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:03.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:03:03 smithi135 ceph-mon[106011]: pgmap v956: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:05.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:03:05 smithi123 ceph-mon[139369]: pgmap v957: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:05.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:03:05 smithi135 ceph-mon[106011]: pgmap v957: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:06.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:03:06 smithi135 ceph-mon[106011]: pgmap v958: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:06.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:03:06 smithi123 ceph-mon[139369]: pgmap v958: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:08.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:03:08 smithi123 ceph-mon[139369]: pgmap v959: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:09.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:03:08 smithi135 ceph-mon[106011]: pgmap v959: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:11.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:03:10 smithi135 ceph-mon[106011]: pgmap v960: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:11.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:03:10 smithi123 ceph-mon[139369]: pgmap v960: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:13.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:03:12 smithi135 ceph-mon[106011]: pgmap v961: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:13.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:03:12 smithi123 ceph-mon[139369]: pgmap v961: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:15.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:03:14 smithi135 ceph-mon[106011]: pgmap v962: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:15.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:03:14 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:03:15.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:03:14 smithi123 ceph-mon[139369]: pgmap v962: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:15.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:03:14 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:03:17.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:03:16 smithi135 ceph-mon[106011]: pgmap v963: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:17.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:03:16 smithi123 ceph-mon[139369]: pgmap v963: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:19.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:03:18 smithi135 ceph-mon[106011]: pgmap v964: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:19.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:03:18 smithi123 ceph-mon[139369]: pgmap v964: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:21.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:03:20 smithi135 ceph-mon[106011]: pgmap v965: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:21.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:03:20 smithi123 ceph-mon[139369]: pgmap v965: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:21.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:21.690+0000 7fd52ce00700 1 -- 172.21.15.123:0/2625596624 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5281037a0 msgr2=0x7fd5280fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:21.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:21.690+0000 7fd52ce00700 1 --2- 172.21.15.123:0/2625596624 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5281037a0 0x7fd5280fce10 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fd514004940 tx=0x7fd514009e30 comp rx=0 tx=0).stop 2024-03-30T05:03:21.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:21.690+0000 7fd52ce00700 1 -- 172.21.15.123:0/2625596624 shutdown_connections 2024-03-30T05:03:21.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:21.690+0000 7fd52ce00700 1 --2- 172.21.15.123:0/2625596624 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5281037a0 0x7fd5280fce10 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:21.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:21.690+0000 7fd52ce00700 1 --2- 172.21.15.123:0/2625596624 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd528102e00 0x7fd5281031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:21.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:21.690+0000 7fd52ce00700 1 -- 172.21.15.123:0/2625596624 >> 172.21.15.123:0/2625596624 conn(0x7fd5280f89b0 msgr2=0x7fd5280fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:03:21.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:21.690+0000 7fd52ce00700 1 -- 172.21.15.123:0/2625596624 shutdown_connections 2024-03-30T05:03:21.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:21.690+0000 7fd52ce00700 1 -- 172.21.15.123:0/2625596624 wait complete. 2024-03-30T05:03:21.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:21.691+0000 7fd52ce00700 1 Processor -- start 2024-03-30T05:03:21.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:21.691+0000 7fd52ce00700 1 -- start start 2024-03-30T05:03:21.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:21.691+0000 7fd52ce00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd528102e00 0x7fd5280702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:03:21.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:21.691+0000 7fd52ce00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd5281037a0 0x7fd528070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:03:21.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:21.691+0000 7fd52ce00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd528070e20 con 0x7fd528102e00 2024-03-30T05:03:21.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:21.691+0000 7fd52ce00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd528070f60 con 0x7fd5281037a0 2024-03-30T05:03:21.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:21.692+0000 7fd525e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd528102e00 0x7fd5280702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:03:21.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:21.692+0000 7fd525e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd528102e00 0x7fd5280702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47796/0 (socket says 172.21.15.123:47796) 2024-03-30T05:03:21.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:21.692+0000 7fd525e00700 1 -- 172.21.15.123:0/3741934980 learned_addr learned my addr 172.21.15.123:0/3741934980 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:03:21.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:21.692+0000 7fd525400700 1 --2- 172.21.15.123:0/3741934980 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd5281037a0 0x7fd528070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:03:21.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:21.692+0000 7fd525e00700 1 -- 172.21.15.123:0/3741934980 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd5281037a0 msgr2=0x7fd528070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:21.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:21.692+0000 7fd525e00700 1 --2- 172.21.15.123:0/3741934980 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd5281037a0 0x7fd528070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:21.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:21.692+0000 7fd525e00700 1 -- 172.21.15.123:0/3741934980 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd514005040 con 0x7fd528102e00 2024-03-30T05:03:21.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:21.692+0000 7fd525400700 1 --2- 172.21.15.123:0/3741934980 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd5281037a0 0x7fd528070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T05:03:21.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:21.692+0000 7fd525e00700 1 --2- 172.21.15.123:0/3741934980 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd528102e00 0x7fd5280702c0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fd51800b790 tx=0x7fd518009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:03:21.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:21.693+0000 7fd51e000700 1 -- 172.21.15.123:0/3741934980 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd51800f770 con 0x7fd528102e00 2024-03-30T05:03:21.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:21.693+0000 7fd51e000700 1 -- 172.21.15.123:0/3741934980 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd51800fd80 con 0x7fd528102e00 2024-03-30T05:03:21.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:21.693+0000 7fd52ce00700 1 -- 172.21.15.123:0/3741934980 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd528119de0 con 0x7fd528102e00 2024-03-30T05:03:21.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:21.693+0000 7fd51e000700 1 -- 172.21.15.123:0/3741934980 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd51800d310 con 0x7fd528102e00 2024-03-30T05:03:21.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:21.693+0000 7fd52ce00700 1 -- 172.21.15.123:0/3741934980 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd52811a2f0 con 0x7fd528102e00 2024-03-30T05:03:21.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:21.694+0000 7fd52ce00700 1 -- 172.21.15.123:0/3741934980 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5280fe550 con 0x7fd528102e00 2024-03-30T05:03:21.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:21.695+0000 7fd51e000700 1 -- 172.21.15.123:0/3741934980 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fd518010420 con 0x7fd528102e00 2024-03-30T05:03:21.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:21.696+0000 7fd51e000700 1 --2- 172.21.15.123:0/3741934980 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd50c06cc30 0x7fd50c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:03:21.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:21.696+0000 7fd51e000700 1 -- 172.21.15.123:0/3741934980 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd51808c080 con 0x7fd528102e00 2024-03-30T05:03:21.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:21.697+0000 7fd525400700 1 --2- 172.21.15.123:0/3741934980 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd50c06cc30 0x7fd50c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:03:21.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:21.697+0000 7fd525400700 1 --2- 172.21.15.123:0/3741934980 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd50c06cc30 0x7fd50c06f0e0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fd51400b3b0 tx=0x7fd514009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:03:21.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:21.701+0000 7fd51e000700 1 -- 172.21.15.123:0/3741934980 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd5180560c0 con 0x7fd528102e00 2024-03-30T05:03:21.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:21.921+0000 7fd52ce00700 1 -- 172.21.15.123:0/3741934980 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd5280719b0 con 0x7fd50c06cc30 2024-03-30T05:03:21.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:21.922+0000 7fd51e000700 1 -- 172.21.15.123:0/3741934980 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fd5280719b0 con 0x7fd50c06cc30 2024-03-30T05:03:21.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:21.925+0000 7fd52ce00700 1 -- 172.21.15.123:0/3741934980 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd50c06cc30 msgr2=0x7fd50c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:21.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:21.925+0000 7fd52ce00700 1 --2- 172.21.15.123:0/3741934980 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd50c06cc30 0x7fd50c06f0e0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fd51400b3b0 tx=0x7fd514009e30 comp rx=0 tx=0).stop 2024-03-30T05:03:21.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:21.925+0000 7fd52ce00700 1 -- 172.21.15.123:0/3741934980 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd528102e00 msgr2=0x7fd5280702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:21.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:21.925+0000 7fd52ce00700 1 --2- 172.21.15.123:0/3741934980 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd528102e00 0x7fd5280702c0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fd51800b790 tx=0x7fd518009250 comp rx=0 tx=0).stop 2024-03-30T05:03:21.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:21.925+0000 7fd52ce00700 1 -- 172.21.15.123:0/3741934980 shutdown_connections 2024-03-30T05:03:21.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:21.925+0000 7fd52ce00700 1 --2- 172.21.15.123:0/3741934980 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd50c06cc30 0x7fd50c06f0e0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:21.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:21.925+0000 7fd52ce00700 1 --2- 172.21.15.123:0/3741934980 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd528102e00 0x7fd5280702c0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:21.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:21.925+0000 7fd52ce00700 1 --2- 172.21.15.123:0/3741934980 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd5281037a0 0x7fd528070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:21.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:21.925+0000 7fd52ce00700 1 -- 172.21.15.123:0/3741934980 >> 172.21.15.123:0/3741934980 conn(0x7fd5280f89b0 msgr2=0x7fd5280f99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:03:21.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:21.925+0000 7fd52ce00700 1 -- 172.21.15.123:0/3741934980 shutdown_connections 2024-03-30T05:03:21.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:21.925+0000 7fd52ce00700 1 -- 172.21.15.123:0/3741934980 wait complete. 2024-03-30T05:03:21.937 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T05:03:22.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.058+0000 7f7f53800700 1 -- 172.21.15.123:0/867635326 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f4c1037a0 msgr2=0x7f7f4c0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:22.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.058+0000 7f7f53800700 1 --2- 172.21.15.123:0/867635326 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f4c1037a0 0x7f7f4c0fce10 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f7f38004940 tx=0x7f7f38009e30 comp rx=0 tx=0).stop 2024-03-30T05:03:22.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.058+0000 7f7f53800700 1 -- 172.21.15.123:0/867635326 shutdown_connections 2024-03-30T05:03:22.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.058+0000 7f7f53800700 1 --2- 172.21.15.123:0/867635326 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f4c1037a0 0x7f7f4c0fce10 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:22.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.058+0000 7f7f53800700 1 --2- 172.21.15.123:0/867635326 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f4c102e00 0x7f7f4c1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:22.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.058+0000 7f7f53800700 1 -- 172.21.15.123:0/867635326 >> 172.21.15.123:0/867635326 conn(0x7f7f4c0f89b0 msgr2=0x7f7f4c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:03:22.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.058+0000 7f7f53800700 1 -- 172.21.15.123:0/867635326 shutdown_connections 2024-03-30T05:03:22.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.058+0000 7f7f53800700 1 -- 172.21.15.123:0/867635326 wait complete. 2024-03-30T05:03:22.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.059+0000 7f7f53800700 1 Processor -- start 2024-03-30T05:03:22.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.059+0000 7f7f53800700 1 -- start start 2024-03-30T05:03:22.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.060+0000 7f7f53800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f4c102e00 0x7f7f4c115d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:03:22.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.060+0000 7f7f53800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f4c1037a0 0x7f7f4c116260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:03:22.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.060+0000 7f7f53800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4c1168b0 con 0x7f7f4c1037a0 2024-03-30T05:03:22.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.060+0000 7f7f53800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4c10fc80 con 0x7f7f4c102e00 2024-03-30T05:03:22.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.060+0000 7f7f4be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f4c1037a0 0x7f7f4c116260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:03:22.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.060+0000 7f7f4be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f4c1037a0 0x7f7f4c116260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47814/0 (socket says 172.21.15.123:47814) 2024-03-30T05:03:22.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.060+0000 7f7f4be00700 1 -- 172.21.15.123:0/1060791369 learned_addr learned my addr 172.21.15.123:0/1060791369 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:03:22.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.060+0000 7f7f50e00700 1 --2- 172.21.15.123:0/1060791369 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f4c102e00 0x7f7f4c115d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:03:22.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.060+0000 7f7f4be00700 1 -- 172.21.15.123:0/1060791369 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f4c102e00 msgr2=0x7f7f4c115d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:22.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.060+0000 7f7f4be00700 1 --2- 172.21.15.123:0/1060791369 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f4c102e00 0x7f7f4c115d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:22.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.060+0000 7f7f4be00700 1 -- 172.21.15.123:0/1060791369 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f38005040 con 0x7f7f4c1037a0 2024-03-30T05:03:22.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.061+0000 7f7f4be00700 1 --2- 172.21.15.123:0/1060791369 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f4c1037a0 0x7f7f4c116260 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f7f38009390 tx=0x7f7f38009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:03:22.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.061+0000 7f7f48c00700 1 -- 172.21.15.123:0/1060791369 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f3800e040 con 0x7f7f4c1037a0 2024-03-30T05:03:22.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.061+0000 7f7f48c00700 1 -- 172.21.15.123:0/1060791369 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7f38013420 con 0x7f7f4c1037a0 2024-03-30T05:03:22.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.061+0000 7f7f53800700 1 -- 172.21.15.123:0/1060791369 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f4c10ff00 con 0x7f7f4c1037a0 2024-03-30T05:03:22.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.061+0000 7f7f48c00700 1 -- 172.21.15.123:0/1060791369 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f3801c630 con 0x7f7f4c1037a0 2024-03-30T05:03:22.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.061+0000 7f7f53800700 1 -- 172.21.15.123:0/1060791369 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f4c1104c0 con 0x7f7f4c1037a0 2024-03-30T05:03:22.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.062+0000 7f7f53800700 1 -- 172.21.15.123:0/1060791369 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f4c0fe550 con 0x7f7f4c1037a0 2024-03-30T05:03:22.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.063+0000 7f7f48c00700 1 -- 172.21.15.123:0/1060791369 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f7f38013a00 con 0x7f7f4c1037a0 2024-03-30T05:03:22.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.064+0000 7f7f48c00700 1 --2- 172.21.15.123:0/1060791369 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7f3406cc30 0x7f7f3406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:03:22.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.064+0000 7f7f48c00700 1 -- 172.21.15.123:0/1060791369 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7f3808d120 con 0x7f7f4c1037a0 2024-03-30T05:03:22.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.064+0000 7f7f50e00700 1 --2- 172.21.15.123:0/1060791369 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7f3406cc30 0x7f7f3406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:03:22.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.065+0000 7f7f50e00700 1 --2- 172.21.15.123:0/1060791369 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7f3406cc30 0x7f7f3406f0e0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f7f40004f40 tx=0x7f7f4000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:03:22.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.068+0000 7f7f48c00700 1 -- 172.21.15.123:0/1060791369 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7f3805abf0 con 0x7f7f4c1037a0 2024-03-30T05:03:22.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.290+0000 7f7f53800700 1 -- 172.21.15.123:0/1060791369 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7f4c109e10 con 0x7f7f3406cc30 2024-03-30T05:03:22.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.291+0000 7f7f48c00700 1 -- 172.21.15.123:0/1060791369 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f7f4c109e10 con 0x7f7f3406cc30 2024-03-30T05:03:22.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.293+0000 7f7f53800700 1 -- 172.21.15.123:0/1060791369 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7f3406cc30 msgr2=0x7f7f3406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:22.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.293+0000 7f7f53800700 1 --2- 172.21.15.123:0/1060791369 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7f3406cc30 0x7f7f3406f0e0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f7f40004f40 tx=0x7f7f4000b040 comp rx=0 tx=0).stop 2024-03-30T05:03:22.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.293+0000 7f7f53800700 1 -- 172.21.15.123:0/1060791369 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f4c1037a0 msgr2=0x7f7f4c116260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:22.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.293+0000 7f7f53800700 1 --2- 172.21.15.123:0/1060791369 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f4c1037a0 0x7f7f4c116260 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f7f38009390 tx=0x7f7f38009e30 comp rx=0 tx=0).stop 2024-03-30T05:03:22.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.293+0000 7f7f53800700 1 -- 172.21.15.123:0/1060791369 shutdown_connections 2024-03-30T05:03:22.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.293+0000 7f7f53800700 1 --2- 172.21.15.123:0/1060791369 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7f3406cc30 0x7f7f3406f0e0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:22.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.293+0000 7f7f53800700 1 --2- 172.21.15.123:0/1060791369 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7f4c102e00 0x7f7f4c115d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:22.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.293+0000 7f7f53800700 1 --2- 172.21.15.123:0/1060791369 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f4c1037a0 0x7f7f4c116260 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:22.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.293+0000 7f7f53800700 1 -- 172.21.15.123:0/1060791369 >> 172.21.15.123:0/1060791369 conn(0x7f7f4c0f89b0 msgr2=0x7f7f4c0f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:03:22.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.294+0000 7f7f53800700 1 -- 172.21.15.123:0/1060791369 shutdown_connections 2024-03-30T05:03:22.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.294+0000 7f7f53800700 1 -- 172.21.15.123:0/1060791369 wait complete. 2024-03-30T05:03:22.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.426+0000 7f4c37a00700 1 -- 172.21.15.123:0/796103859 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c30102e00 msgr2=0x7f4c301031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:22.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.426+0000 7f4c37a00700 1 --2- 172.21.15.123:0/796103859 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c30102e00 0x7f4c301031d0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f4c24004940 tx=0x7f4c24009e30 comp rx=0 tx=0).stop 2024-03-30T05:03:22.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.426+0000 7f4c37a00700 1 -- 172.21.15.123:0/796103859 shutdown_connections 2024-03-30T05:03:22.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.426+0000 7f4c37a00700 1 --2- 172.21.15.123:0/796103859 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c301037a0 0x7f4c300fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:22.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.426+0000 7f4c37a00700 1 --2- 172.21.15.123:0/796103859 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c30102e00 0x7f4c301031d0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:22.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.426+0000 7f4c37a00700 1 -- 172.21.15.123:0/796103859 >> 172.21.15.123:0/796103859 conn(0x7f4c300f89b0 msgr2=0x7f4c300fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:03:22.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.426+0000 7f4c37a00700 1 -- 172.21.15.123:0/796103859 shutdown_connections 2024-03-30T05:03:22.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.426+0000 7f4c37a00700 1 -- 172.21.15.123:0/796103859 wait complete. 2024-03-30T05:03:22.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.427+0000 7f4c37a00700 1 Processor -- start 2024-03-30T05:03:22.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.427+0000 7f4c37a00700 1 -- start start 2024-03-30T05:03:22.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.428+0000 7f4c37a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c30102e00 0x7f4c30115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:03:22.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.428+0000 7f4c37a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c301037a0 0x7f4c30116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:03:22.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.428+0000 7f4c37a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c30116880 con 0x7f4c301037a0 2024-03-30T05:03:22.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.428+0000 7f4c37a00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c3010fc50 con 0x7f4c30102e00 2024-03-30T05:03:22.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.428+0000 7f4c2fe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c301037a0 0x7f4c30116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:03:22.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.428+0000 7f4c2fe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c301037a0 0x7f4c30116230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47838/0 (socket says 172.21.15.123:47838) 2024-03-30T05:03:22.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.428+0000 7f4c2fe00700 1 -- 172.21.15.123:0/2795872953 learned_addr learned my addr 172.21.15.123:0/2795872953 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:03:22.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.428+0000 7f4c35000700 1 --2- 172.21.15.123:0/2795872953 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c30102e00 0x7f4c30115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:03:22.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.428+0000 7f4c2fe00700 1 -- 172.21.15.123:0/2795872953 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c30102e00 msgr2=0x7f4c30115cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:22.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.428+0000 7f4c2fe00700 1 --2- 172.21.15.123:0/2795872953 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c30102e00 0x7f4c30115cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:22.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.428+0000 7f4c2fe00700 1 -- 172.21.15.123:0/2795872953 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c24005040 con 0x7f4c301037a0 2024-03-30T05:03:22.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.428+0000 7f4c35000700 1 --2- 172.21.15.123:0/2795872953 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c30102e00 0x7f4c30115cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T05:03:22.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.429+0000 7f4c2fe00700 1 --2- 172.21.15.123:0/2795872953 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c301037a0 0x7f4c30116230 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c00c890 tx=0x7f4c1c0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:03:22.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.429+0000 7f4c2cc00700 1 -- 172.21.15.123:0/2795872953 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c1c00a7d0 con 0x7f4c301037a0 2024-03-30T05:03:22.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.429+0000 7f4c2cc00700 1 -- 172.21.15.123:0/2795872953 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4c1c00ade0 con 0x7f4c301037a0 2024-03-30T05:03:22.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.429+0000 7f4c37a00700 1 -- 172.21.15.123:0/2795872953 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c3010ff30 con 0x7f4c301037a0 2024-03-30T05:03:22.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.429+0000 7f4c2cc00700 1 -- 172.21.15.123:0/2795872953 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c1c0113a0 con 0x7f4c301037a0 2024-03-30T05:03:22.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.429+0000 7f4c37a00700 1 -- 172.21.15.123:0/2795872953 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c301104f0 con 0x7f4c301037a0 2024-03-30T05:03:22.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.430+0000 7f4c37a00700 1 -- 172.21.15.123:0/2795872953 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c300fe550 con 0x7f4c301037a0 2024-03-30T05:03:22.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.431+0000 7f4c2cc00700 1 -- 172.21.15.123:0/2795872953 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f4c1c012400 con 0x7f4c301037a0 2024-03-30T05:03:22.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.433+0000 7f4c2cc00700 1 --2- 172.21.15.123:0/2795872953 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4c1806cc30 0x7f4c1806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:03:22.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.433+0000 7f4c2cc00700 1 -- 172.21.15.123:0/2795872953 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f4c1c08c1a0 con 0x7f4c301037a0 2024-03-30T05:03:22.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.433+0000 7f4c35000700 1 --2- 172.21.15.123:0/2795872953 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4c1806cc30 0x7f4c1806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:03:22.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.434+0000 7f4c35000700 1 --2- 172.21.15.123:0/2795872953 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4c1806cc30 0x7f4c1806f0e0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f4c24000c00 tx=0x7f4c24009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:03:22.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.437+0000 7f4c2cc00700 1 -- 172.21.15.123:0/2795872953 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4c1c059bf0 con 0x7f4c301037a0 2024-03-30T05:03:22.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.651+0000 7f4c37a00700 1 -- 172.21.15.123:0/2795872953 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4c30109e30 con 0x7f4c1806cc30 2024-03-30T05:03:22.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.661+0000 7f4c2cc00700 1 -- 172.21.15.123:0/2795872953 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f4c30109e30 con 0x7f4c1806cc30 2024-03-30T05:03:22.659 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T05:03:22.659 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (32m) 9m ago 45m 14.6M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T05:03:22.660 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (29m) 9m ago 45m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T05:03:22.660 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (29m) 9m ago 44m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T05:03:22.660 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (32m) 9m ago 44m 73.5M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T05:03:22.660 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (25m) 9m ago 38m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T05:03:22.660 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (25m) 9m ago 38m 16.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T05:03:22.660 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (10m) 9m ago 38m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 87b073a6c9ca 2024-03-30T05:03:22.660 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (25m) 9m ago 38m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T05:03:22.660 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (34m) 9m ago 46m 510M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T05:03:22.660 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (33m) 9m ago 43m 456M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T05:03:22.660 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (31m) 9m ago 46m 70.0M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T05:03:22.660 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (30m) 9m ago 43m 51.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T05:03:22.660 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (33m) 9m ago 45m 8636k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T05:03:22.661 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (33m) 9m ago 44m 8678k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T05:03:22.661 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (29m) 9m ago 43m 107M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T05:03:22.661 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (28m) 9m ago 42m 97.2M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T05:03:22.661 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (28m) 9m ago 41m 89.8M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T05:03:22.661 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (27m) 9m ago 41m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T05:03:22.661 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (26m) 9m ago 40m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T05:03:22.661 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (26m) 9m ago 40m 88.8M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T05:03:22.661 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (32m) 9m ago 44m 63.9M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T05:03:22.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.663+0000 7f4c37a00700 1 -- 172.21.15.123:0/2795872953 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4c1806cc30 msgr2=0x7f4c1806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:22.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.663+0000 7f4c37a00700 1 --2- 172.21.15.123:0/2795872953 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4c1806cc30 0x7f4c1806f0e0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f4c24000c00 tx=0x7f4c24009e30 comp rx=0 tx=0).stop 2024-03-30T05:03:22.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.663+0000 7f4c37a00700 1 -- 172.21.15.123:0/2795872953 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c301037a0 msgr2=0x7f4c30116230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:22.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.663+0000 7f4c37a00700 1 --2- 172.21.15.123:0/2795872953 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c301037a0 0x7f4c30116230 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c00c890 tx=0x7f4c1c0044c0 comp rx=0 tx=0).stop 2024-03-30T05:03:22.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.664+0000 7f4c37a00700 1 -- 172.21.15.123:0/2795872953 shutdown_connections 2024-03-30T05:03:22.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.664+0000 7f4c37a00700 1 --2- 172.21.15.123:0/2795872953 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4c1806cc30 0x7f4c1806f0e0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:22.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.664+0000 7f4c37a00700 1 --2- 172.21.15.123:0/2795872953 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c30102e00 0x7f4c30115cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:22.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.664+0000 7f4c37a00700 1 --2- 172.21.15.123:0/2795872953 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c301037a0 0x7f4c30116230 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:22.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.664+0000 7f4c37a00700 1 -- 172.21.15.123:0/2795872953 >> 172.21.15.123:0/2795872953 conn(0x7f4c300f89b0 msgr2=0x7f4c300706d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:03:22.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.664+0000 7f4c37a00700 1 -- 172.21.15.123:0/2795872953 shutdown_connections 2024-03-30T05:03:22.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.664+0000 7f4c37a00700 1 -- 172.21.15.123:0/2795872953 wait complete. 2024-03-30T05:03:22.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.797+0000 7f6d69200700 1 -- 172.21.15.123:0/542746655 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d640fde90 msgr2=0x7f6d640fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:22.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.797+0000 7f6d69200700 1 --2- 172.21.15.123:0/542746655 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d640fde90 0x7f6d640fe260 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f6d54004940 tx=0x7f6d54009e30 comp rx=0 tx=0).stop 2024-03-30T05:03:22.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.797+0000 7f6d69200700 1 -- 172.21.15.123:0/542746655 shutdown_connections 2024-03-30T05:03:22.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.797+0000 7f6d69200700 1 --2- 172.21.15.123:0/542746655 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6d640fe830 0x7f6d64106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:22.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.797+0000 7f6d69200700 1 --2- 172.21.15.123:0/542746655 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d640fde90 0x7f6d640fe260 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:22.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.797+0000 7f6d69200700 1 -- 172.21.15.123:0/542746655 >> 172.21.15.123:0/542746655 conn(0x7f6d64072bc0 msgr2=0x7f6d64072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:03:22.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.797+0000 7f6d69200700 1 -- 172.21.15.123:0/542746655 shutdown_connections 2024-03-30T05:03:22.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.797+0000 7f6d69200700 1 -- 172.21.15.123:0/542746655 wait complete. 2024-03-30T05:03:22.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.798+0000 7f6d69200700 1 Processor -- start 2024-03-30T05:03:22.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.798+0000 7f6d69200700 1 -- start start 2024-03-30T05:03:22.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.798+0000 7f6d69200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6d640fde90 0x7f6d64115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:03:22.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.798+0000 7f6d69200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d640fe830 0x7f6d641161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:03:22.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.798+0000 7f6d69200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d64116810 con 0x7f6d640fe830 2024-03-30T05:03:22.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.798+0000 7f6d69200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d6410fbe0 con 0x7f6d640fde90 2024-03-30T05:03:22.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.798+0000 7f6d61400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d640fe830 0x7f6d641161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:03:22.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.799+0000 7f6d61400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d640fe830 0x7f6d641161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47868/0 (socket says 172.21.15.123:47868) 2024-03-30T05:03:22.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.799+0000 7f6d61400700 1 -- 172.21.15.123:0/1310697106 learned_addr learned my addr 172.21.15.123:0/1310697106 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:03:22.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.799+0000 7f6d61e00700 1 --2- 172.21.15.123:0/1310697106 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6d640fde90 0x7f6d64115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:03:22.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.799+0000 7f6d61400700 1 -- 172.21.15.123:0/1310697106 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6d640fde90 msgr2=0x7f6d64115c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:22.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.799+0000 7f6d61400700 1 --2- 172.21.15.123:0/1310697106 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6d640fde90 0x7f6d64115c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:22.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.799+0000 7f6d61400700 1 -- 172.21.15.123:0/1310697106 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d54005040 con 0x7f6d640fe830 2024-03-30T05:03:22.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.799+0000 7f6d61400700 1 --2- 172.21.15.123:0/1310697106 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d640fe830 0x7f6d641161c0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f6d5000eb40 tx=0x7f6d5000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:03:22.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.799+0000 7f6d5a000700 1 -- 172.21.15.123:0/1310697106 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d50007cc0 con 0x7f6d640fe830 2024-03-30T05:03:22.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.799+0000 7f6d5a000700 1 -- 172.21.15.123:0/1310697106 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6d5000f040 con 0x7f6d640fe830 2024-03-30T05:03:22.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.799+0000 7f6d69200700 1 -- 172.21.15.123:0/1310697106 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d6410fe60 con 0x7f6d640fe830 2024-03-30T05:03:22.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.799+0000 7f6d5a000700 1 -- 172.21.15.123:0/1310697106 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d50009570 con 0x7f6d640fe830 2024-03-30T05:03:22.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.800+0000 7f6d69200700 1 -- 172.21.15.123:0/1310697106 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d641103a0 con 0x7f6d640fe830 2024-03-30T05:03:22.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.801+0000 7f6d69200700 1 -- 172.21.15.123:0/1310697106 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d64106a20 con 0x7f6d640fe830 2024-03-30T05:03:22.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.802+0000 7f6d5a000700 1 -- 172.21.15.123:0/1310697106 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f6d500096d0 con 0x7f6d640fe830 2024-03-30T05:03:22.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.802+0000 7f6d5a000700 1 --2- 172.21.15.123:0/1310697106 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6d4806cc30 0x7f6d4806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:03:22.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.802+0000 7f6d5a000700 1 -- 172.21.15.123:0/1310697106 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f6d5008bcb0 con 0x7f6d640fe830 2024-03-30T05:03:22.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.803+0000 7f6d61e00700 1 --2- 172.21.15.123:0/1310697106 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6d4806cc30 0x7f6d4806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:03:22.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.803+0000 7f6d61e00700 1 --2- 172.21.15.123:0/1310697106 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6d4806cc30 0x7f6d4806f0e0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f6d5400b0d0 tx=0x7f6d54009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:03:22.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:22.807+0000 7f6d5a000700 1 -- 172.21.15.123:0/1310697106 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6d500597b0 con 0x7f6d640fe830 2024-03-30T05:03:23.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.094+0000 7f6d69200700 1 -- 172.21.15.123:0/1310697106 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f6d64065e30 con 0x7f6d640fe830 2024-03-30T05:03:23.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.094+0000 7f6d5a000700 1 -- 172.21.15.123:0/1310697106 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f6d50059340 con 0x7f6d640fe830 2024-03-30T05:03:23.093 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T05:03:23.093 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T05:03:23.093 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T05:03:23.093 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:03:23.093 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T05:03:23.093 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T05:03:23.093 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:03:23.093 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T05:03:23.093 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T05:03:23.093 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:03:23.093 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T05:03:23.094 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 4 2024-03-30T05:03:23.094 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:03:23.094 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T05:03:23.094 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 14 2024-03-30T05:03:23.094 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T05:03:23.094 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T05:03:23.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.097+0000 7f6d69200700 1 -- 172.21.15.123:0/1310697106 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6d4806cc30 msgr2=0x7f6d4806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:23.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.097+0000 7f6d69200700 1 --2- 172.21.15.123:0/1310697106 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6d4806cc30 0x7f6d4806f0e0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f6d5400b0d0 tx=0x7f6d54009e30 comp rx=0 tx=0).stop 2024-03-30T05:03:23.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.097+0000 7f6d69200700 1 -- 172.21.15.123:0/1310697106 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d640fe830 msgr2=0x7f6d641161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:23.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.097+0000 7f6d69200700 1 --2- 172.21.15.123:0/1310697106 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d640fe830 0x7f6d641161c0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f6d5000eb40 tx=0x7f6d5000b040 comp rx=0 tx=0).stop 2024-03-30T05:03:23.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.097+0000 7f6d69200700 1 -- 172.21.15.123:0/1310697106 shutdown_connections 2024-03-30T05:03:23.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.097+0000 7f6d69200700 1 --2- 172.21.15.123:0/1310697106 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f6d4806cc30 0x7f6d4806f0e0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:23.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.097+0000 7f6d69200700 1 --2- 172.21.15.123:0/1310697106 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6d640fde90 0x7f6d64115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:23.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.097+0000 7f6d69200700 1 --2- 172.21.15.123:0/1310697106 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d640fe830 0x7f6d641161c0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:23.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.097+0000 7f6d69200700 1 -- 172.21.15.123:0/1310697106 >> 172.21.15.123:0/1310697106 conn(0x7f6d64072bc0 msgr2=0x7f6d640fb7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:03:23.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.097+0000 7f6d69200700 1 -- 172.21.15.123:0/1310697106 shutdown_connections 2024-03-30T05:03:23.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.097+0000 7f6d69200700 1 -- 172.21.15.123:0/1310697106 wait complete. 2024-03-30T05:03:23.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:03:22 smithi135 ceph-mon[106011]: pgmap v966: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:23.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:03:22 smithi135 ceph-mon[106011]: from='client.35748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:03:23.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:03:22 smithi123 ceph-mon[139369]: pgmap v966: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:23.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:03:22 smithi123 ceph-mon[139369]: from='client.35748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:03:23.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.229+0000 7f9d23e00700 1 -- 172.21.15.123:0/1921760312 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d1c102e00 msgr2=0x7f9d1c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:23.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.229+0000 7f9d23e00700 1 --2- 172.21.15.123:0/1921760312 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d1c102e00 0x7f9d1c1031d0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f9d10004940 tx=0x7f9d10009e30 comp rx=0 tx=0).stop 2024-03-30T05:03:23.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.229+0000 7f9d23e00700 1 -- 172.21.15.123:0/1921760312 shutdown_connections 2024-03-30T05:03:23.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.229+0000 7f9d23e00700 1 --2- 172.21.15.123:0/1921760312 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9d1c1037a0 0x7f9d1c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:23.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.229+0000 7f9d23e00700 1 --2- 172.21.15.123:0/1921760312 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d1c102e00 0x7f9d1c1031d0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:23.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.229+0000 7f9d23e00700 1 -- 172.21.15.123:0/1921760312 >> 172.21.15.123:0/1921760312 conn(0x7f9d1c0f89b0 msgr2=0x7f9d1c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:03:23.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.229+0000 7f9d23e00700 1 -- 172.21.15.123:0/1921760312 shutdown_connections 2024-03-30T05:03:23.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.229+0000 7f9d23e00700 1 -- 172.21.15.123:0/1921760312 wait complete. 2024-03-30T05:03:23.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.230+0000 7f9d23e00700 1 Processor -- start 2024-03-30T05:03:23.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.230+0000 7f9d23e00700 1 -- start start 2024-03-30T05:03:23.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.230+0000 7f9d23e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d1c102e00 0x7f9d1c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:03:23.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.230+0000 7f9d23e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9d1c1037a0 0x7f9d1c070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:03:23.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.230+0000 7f9d23e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d1c070d90 con 0x7f9d1c102e00 2024-03-30T05:03:23.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.230+0000 7f9d23e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d1c070ed0 con 0x7f9d1c1037a0 2024-03-30T05:03:23.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.231+0000 7f9d21400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d1c102e00 0x7f9d1c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:03:23.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.231+0000 7f9d21400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d1c102e00 0x7f9d1c0702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47884/0 (socket says 172.21.15.123:47884) 2024-03-30T05:03:23.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.231+0000 7f9d21400700 1 -- 172.21.15.123:0/3853201310 learned_addr learned my addr 172.21.15.123:0/3853201310 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:03:23.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.231+0000 7f9d20a00700 1 --2- 172.21.15.123:0/3853201310 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9d1c1037a0 0x7f9d1c070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:03:23.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.231+0000 7f9d21400700 1 -- 172.21.15.123:0/3853201310 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9d1c1037a0 msgr2=0x7f9d1c070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:23.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.231+0000 7f9d21400700 1 --2- 172.21.15.123:0/3853201310 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9d1c1037a0 0x7f9d1c070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:23.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.231+0000 7f9d21400700 1 -- 172.21.15.123:0/3853201310 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d10005040 con 0x7f9d1c102e00 2024-03-30T05:03:23.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.231+0000 7f9d21400700 1 --2- 172.21.15.123:0/3853201310 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d1c102e00 0x7f9d1c0702c0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f9d10009390 tx=0x7f9d10009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:03:23.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.232+0000 7f9d19600700 1 -- 172.21.15.123:0/3853201310 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d1000e040 con 0x7f9d1c102e00 2024-03-30T05:03:23.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.232+0000 7f9d19600700 1 -- 172.21.15.123:0/3853201310 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9d10013420 con 0x7f9d1c102e00 2024-03-30T05:03:23.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.232+0000 7f9d23e00700 1 -- 172.21.15.123:0/3853201310 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d1c071120 con 0x7f9d1c102e00 2024-03-30T05:03:23.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.232+0000 7f9d19600700 1 -- 172.21.15.123:0/3853201310 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d1001c630 con 0x7f9d1c102e00 2024-03-30T05:03:23.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.232+0000 7f9d23e00700 1 -- 172.21.15.123:0/3853201310 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d1c11a250 con 0x7f9d1c102e00 2024-03-30T05:03:23.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.233+0000 7f9d23e00700 1 -- 172.21.15.123:0/3853201310 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d1c0fe550 con 0x7f9d1c102e00 2024-03-30T05:03:23.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.234+0000 7f9d19600700 1 -- 172.21.15.123:0/3853201310 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f9d100139e0 con 0x7f9d1c102e00 2024-03-30T05:03:23.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.234+0000 7f9d19600700 1 --2- 172.21.15.123:0/3853201310 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9d0406cc30 0x7f9d0406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:03:23.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.235+0000 7f9d19600700 1 -- 172.21.15.123:0/3853201310 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f9d1008d260 con 0x7f9d1c102e00 2024-03-30T05:03:23.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.235+0000 7f9d20a00700 1 --2- 172.21.15.123:0/3853201310 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9d0406cc30 0x7f9d0406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:03:23.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.235+0000 7f9d20a00700 1 --2- 172.21.15.123:0/3853201310 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9d0406cc30 0x7f9d0406f0e0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c071c00 tx=0x7f9d08009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:03:23.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.239+0000 7f9d19600700 1 -- 172.21.15.123:0/3853201310 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9d100572a0 con 0x7f9d1c102e00 2024-03-30T05:03:23.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.485+0000 7f9d23e00700 1 -- 172.21.15.123:0/3853201310 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f9d1c0719b0 con 0x7f9d1c102e00 2024-03-30T05:03:23.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.485+0000 7f9d19600700 1 -- 172.21.15.123:0/3853201310 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 34 v34) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f9d1005a8c0 con 0x7f9d1c102e00 2024-03-30T05:03:23.484 INFO:teuthology.orchestra.run.smithi123.stdout:e34 2024-03-30T05:03:23.484 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T05:03:23.484 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T05:03:23.484 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T05:03:23.484 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:03:23.484 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T05:03:23.484 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T05:03:23.484 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T05:03:23.484 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T05:03:23.484 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T05:03:23.484 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T05:03:23.485 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T05:03:23.485 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T05:03:23.485 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T05:03:23.485 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T05:03:23.485 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T05:03:23.485 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T05:03:23.485 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T05:03:23.485 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T05:03:23.485 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T05:03:23.486 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T05:03:23.486 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T05:03:23.486 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T05:03:23.486 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T05:03:23.486 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T05:03:23.487 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T05:03:23.487 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T05:03:23.487 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T05:03:23.487 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T05:03:23.487 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T05:03:23.487 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T05:03:23.487 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T05:03:23.487 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T05:03:23.487 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:03:23.487 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:03:23.488 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T05:03:23.488 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:03:23.488 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:03:23.488 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:03:23.488 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:03:23.488 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44975} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/3788988014,v1:172.21.15.135:6827/3788988014] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:03:23.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.487+0000 7f9d23e00700 1 -- 172.21.15.123:0/3853201310 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9d0406cc30 msgr2=0x7f9d0406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:23.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.487+0000 7f9d23e00700 1 --2- 172.21.15.123:0/3853201310 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9d0406cc30 0x7f9d0406f0e0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c071c00 tx=0x7f9d08009ee0 comp rx=0 tx=0).stop 2024-03-30T05:03:23.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.487+0000 7f9d23e00700 1 -- 172.21.15.123:0/3853201310 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d1c102e00 msgr2=0x7f9d1c0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:23.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.487+0000 7f9d23e00700 1 --2- 172.21.15.123:0/3853201310 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d1c102e00 0x7f9d1c0702c0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f9d10009390 tx=0x7f9d10009e30 comp rx=0 tx=0).stop 2024-03-30T05:03:23.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.488+0000 7f9d23e00700 1 -- 172.21.15.123:0/3853201310 shutdown_connections 2024-03-30T05:03:23.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.488+0000 7f9d23e00700 1 --2- 172.21.15.123:0/3853201310 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9d0406cc30 0x7f9d0406f0e0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:23.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.488+0000 7f9d23e00700 1 --2- 172.21.15.123:0/3853201310 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d1c102e00 0x7f9d1c0702c0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:23.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.488+0000 7f9d23e00700 1 --2- 172.21.15.123:0/3853201310 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9d1c1037a0 0x7f9d1c070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:23.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.488+0000 7f9d23e00700 1 -- 172.21.15.123:0/3853201310 >> 172.21.15.123:0/3853201310 conn(0x7f9d1c0f89b0 msgr2=0x7f9d1c0f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:03:23.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.488+0000 7f9d23e00700 1 -- 172.21.15.123:0/3853201310 shutdown_connections 2024-03-30T05:03:23.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.488+0000 7f9d23e00700 1 -- 172.21.15.123:0/3853201310 wait complete. 2024-03-30T05:03:23.490 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 34 2024-03-30T05:03:23.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.625+0000 7fcc2fc00700 1 -- 172.21.15.123:0/945447980 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc28070300 msgr2=0x7fcc280706d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:23.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.625+0000 7fcc2fc00700 1 --2- 172.21.15.123:0/945447980 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc28070300 0x7fcc280706d0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c004970 tx=0x7fcc1c009e30 comp rx=0 tx=0).stop 2024-03-30T05:03:23.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.625+0000 7fcc2fc00700 1 -- 172.21.15.123:0/945447980 shutdown_connections 2024-03-30T05:03:23.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.626+0000 7fcc2fc00700 1 --2- 172.21.15.123:0/945447980 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcc28070c10 0x7fcc28109190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:23.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.626+0000 7fcc2fc00700 1 --2- 172.21.15.123:0/945447980 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc28070300 0x7fcc280706d0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:23.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.626+0000 7fcc2fc00700 1 -- 172.21.15.123:0/945447980 >> 172.21.15.123:0/945447980 conn(0x7fcc280f89d0 msgr2=0x7fcc280fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:03:23.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.626+0000 7fcc2fc00700 1 -- 172.21.15.123:0/945447980 shutdown_connections 2024-03-30T05:03:23.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.626+0000 7fcc2fc00700 1 -- 172.21.15.123:0/945447980 wait complete. 2024-03-30T05:03:23.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.626+0000 7fcc2fc00700 1 Processor -- start 2024-03-30T05:03:23.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.627+0000 7fcc2fc00700 1 -- start start 2024-03-30T05:03:23.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.627+0000 7fcc2fc00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc28070300 0x7fcc28115cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:03:23.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.627+0000 7fcc2fc00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcc28070c10 0x7fcc281161f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:03:23.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.627+0000 7fcc2fc00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc28116840 con 0x7fcc28070300 2024-03-30T05:03:23.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.627+0000 7fcc2fc00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2810fc10 con 0x7fcc28070c10 2024-03-30T05:03:23.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.627+0000 7fcc2d200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc28070300 0x7fcc28115cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:03:23.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.627+0000 7fcc2d200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc28070300 0x7fcc28115cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47906/0 (socket says 172.21.15.123:47906) 2024-03-30T05:03:23.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.627+0000 7fcc2d200700 1 -- 172.21.15.123:0/2748193601 learned_addr learned my addr 172.21.15.123:0/2748193601 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:03:23.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.627+0000 7fcc27e00700 1 --2- 172.21.15.123:0/2748193601 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcc28070c10 0x7fcc281161f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:03:23.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.627+0000 7fcc2d200700 1 -- 172.21.15.123:0/2748193601 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcc28070c10 msgr2=0x7fcc281161f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:23.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.627+0000 7fcc2d200700 1 --2- 172.21.15.123:0/2748193601 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcc28070c10 0x7fcc281161f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:23.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.627+0000 7fcc2d200700 1 -- 172.21.15.123:0/2748193601 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc1c005040 con 0x7fcc28070300 2024-03-30T05:03:23.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.628+0000 7fcc2d200700 1 --2- 172.21.15.123:0/2748193601 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc28070300 0x7fcc28115cb0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c004f50 tx=0x7fcc1c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:03:23.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.628+0000 7fcc24c00700 1 -- 172.21.15.123:0/2748193601 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc1c00e040 con 0x7fcc28070300 2024-03-30T05:03:23.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.628+0000 7fcc24c00700 1 -- 172.21.15.123:0/2748193601 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcc1c013420 con 0x7fcc28070300 2024-03-30T05:03:23.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.628+0000 7fcc24c00700 1 -- 172.21.15.123:0/2748193601 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc1c01c610 con 0x7fcc28070300 2024-03-30T05:03:23.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.628+0000 7fcc2fc00700 1 -- 172.21.15.123:0/2748193601 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc2810fe90 con 0x7fcc28070300 2024-03-30T05:03:23.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.628+0000 7fcc2fc00700 1 -- 172.21.15.123:0/2748193601 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc28110380 con 0x7fcc28070300 2024-03-30T05:03:23.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.629+0000 7fcc2fc00700 1 -- 172.21.15.123:0/2748193601 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc28071a40 con 0x7fcc28070300 2024-03-30T05:03:23.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.631+0000 7fcc24c00700 1 -- 172.21.15.123:0/2748193601 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fcc1c013590 con 0x7fcc28070300 2024-03-30T05:03:23.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.631+0000 7fcc24c00700 1 --2- 172.21.15.123:0/2748193601 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fcc1006cc30 0x7fcc1006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:03:23.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.631+0000 7fcc24c00700 1 -- 172.21.15.123:0/2748193601 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fcc1c08d310 con 0x7fcc28070300 2024-03-30T05:03:23.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.632+0000 7fcc27e00700 1 --2- 172.21.15.123:0/2748193601 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fcc1006cc30 0x7fcc1006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:03:23.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.632+0000 7fcc27e00700 1 --2- 172.21.15.123:0/2748193601 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fcc1006cc30 0x7fcc1006f0e0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fcc14009fd0 tx=0x7fcc14009f20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:03:23.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.636+0000 7fcc24c00700 1 -- 172.21.15.123:0/2748193601 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fcc1c05ace0 con 0x7fcc28070300 2024-03-30T05:03:23.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.862+0000 7fcc2fc00700 1 -- 172.21.15.123:0/2748193601 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcc2804bf40 con 0x7fcc1006cc30 2024-03-30T05:03:23.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.863+0000 7fcc24c00700 1 -- 172.21.15.123:0/2748193601 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fcc2804bf40 con 0x7fcc1006cc30 2024-03-30T05:03:23.861 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T05:03:23.861 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T05:03:23.861 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T05:03:23.861 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T05:03:23.861 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T05:03:23.861 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T05:03:23.861 INFO:teuthology.orchestra.run.smithi123.stdout: "mds", 2024-03-30T05:03:23.862 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T05:03:23.862 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T05:03:23.862 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T05:03:23.862 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T05:03:23.862 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "16/21 daemons upgraded", 2024-03-30T05:03:23.862 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T05:03:23.862 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T05:03:23.862 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T05:03:23.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.865+0000 7fcc2fc00700 1 -- 172.21.15.123:0/2748193601 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fcc1006cc30 msgr2=0x7fcc1006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:23.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.865+0000 7fcc2fc00700 1 --2- 172.21.15.123:0/2748193601 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fcc1006cc30 0x7fcc1006f0e0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fcc14009fd0 tx=0x7fcc14009f20 comp rx=0 tx=0).stop 2024-03-30T05:03:23.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.865+0000 7fcc2fc00700 1 -- 172.21.15.123:0/2748193601 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc28070300 msgr2=0x7fcc28115cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:23.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.865+0000 7fcc2fc00700 1 --2- 172.21.15.123:0/2748193601 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc28070300 0x7fcc28115cb0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c004f50 tx=0x7fcc1c009e30 comp rx=0 tx=0).stop 2024-03-30T05:03:23.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.865+0000 7fcc2fc00700 1 -- 172.21.15.123:0/2748193601 shutdown_connections 2024-03-30T05:03:23.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.865+0000 7fcc2fc00700 1 --2- 172.21.15.123:0/2748193601 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fcc1006cc30 0x7fcc1006f0e0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:23.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.865+0000 7fcc2fc00700 1 --2- 172.21.15.123:0/2748193601 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc28070300 0x7fcc28115cb0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:23.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.865+0000 7fcc2fc00700 1 --2- 172.21.15.123:0/2748193601 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcc28070c10 0x7fcc281161f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:23.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.865+0000 7fcc2fc00700 1 -- 172.21.15.123:0/2748193601 >> 172.21.15.123:0/2748193601 conn(0x7fcc280f89d0 msgr2=0x7fcc280f9f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:03:23.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.865+0000 7fcc2fc00700 1 -- 172.21.15.123:0/2748193601 shutdown_connections 2024-03-30T05:03:23.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.866+0000 7fcc2fc00700 1 -- 172.21.15.123:0/2748193601 wait complete. 2024-03-30T05:03:23.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.999+0000 7fb2eba00700 1 -- 172.21.15.123:0/695115017 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2e41037a0 msgr2=0x7fb2e40fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:23.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.999+0000 7fb2eba00700 1 --2- 172.21.15.123:0/695115017 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2e41037a0 0x7fb2e40fce10 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fb2d0004940 tx=0x7fb2d0009e30 comp rx=0 tx=0).stop 2024-03-30T05:03:23.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.999+0000 7fb2eba00700 1 -- 172.21.15.123:0/695115017 shutdown_connections 2024-03-30T05:03:23.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.999+0000 7fb2eba00700 1 --2- 172.21.15.123:0/695115017 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2e41037a0 0x7fb2e40fce10 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:23.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.999+0000 7fb2eba00700 1 --2- 172.21.15.123:0/695115017 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb2e4102e00 0x7fb2e41031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:23.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.999+0000 7fb2eba00700 1 -- 172.21.15.123:0/695115017 >> 172.21.15.123:0/695115017 conn(0x7fb2e40f89b0 msgr2=0x7fb2e40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:03:23.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.999+0000 7fb2eba00700 1 -- 172.21.15.123:0/695115017 shutdown_connections 2024-03-30T05:03:23.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:23.999+0000 7fb2eba00700 1 -- 172.21.15.123:0/695115017 wait complete. 2024-03-30T05:03:23.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:24.000+0000 7fb2eba00700 1 Processor -- start 2024-03-30T05:03:23.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:24.000+0000 7fb2eba00700 1 -- start start 2024-03-30T05:03:23.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:24.001+0000 7fb2eba00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2e4102e00 0x7fb2e40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:03:23.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:24.001+0000 7fb2eba00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb2e41037a0 0x7fb2e4070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:03:23.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:24.001+0000 7fb2eba00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e40720b0 con 0x7fb2e4102e00 2024-03-30T05:03:23.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:24.001+0000 7fb2eba00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e4072220 con 0x7fb2e41037a0 2024-03-30T05:03:24.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:24.001+0000 7fb2e9000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2e4102e00 0x7fb2e40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:03:24.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:24.001+0000 7fb2e9000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2e4102e00 0x7fb2e40702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56872/0 (socket says 172.21.15.123:56872) 2024-03-30T05:03:24.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:24.001+0000 7fb2e9000700 1 -- 172.21.15.123:0/3775353160 learned_addr learned my addr 172.21.15.123:0/3775353160 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:03:24.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:24.001+0000 7fb2e3e00700 1 --2- 172.21.15.123:0/3775353160 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb2e41037a0 0x7fb2e4070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:03:24.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:24.001+0000 7fb2e9000700 1 -- 172.21.15.123:0/3775353160 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb2e41037a0 msgr2=0x7fb2e4070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:24.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:24.001+0000 7fb2e9000700 1 --2- 172.21.15.123:0/3775353160 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb2e41037a0 0x7fb2e4070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:24.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:24.001+0000 7fb2e9000700 1 -- 172.21.15.123:0/3775353160 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d0005040 con 0x7fb2e4102e00 2024-03-30T05:03:24.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:24.001+0000 7fb2e3e00700 1 --2- 172.21.15.123:0/3775353160 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb2e41037a0 0x7fb2e4070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T05:03:24.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:24.002+0000 7fb2e9000700 1 --2- 172.21.15.123:0/3775353160 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2e4102e00 0x7fb2e40702c0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fb2d800eb40 tx=0x7fb2d800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:03:24.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:24.002+0000 7fb2e0c00700 1 -- 172.21.15.123:0/3775353160 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb2d8007cc0 con 0x7fb2e4102e00 2024-03-30T05:03:24.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:24.002+0000 7fb2e0c00700 1 -- 172.21.15.123:0/3775353160 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb2d800f040 con 0x7fb2e4102e00 2024-03-30T05:03:24.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:24.002+0000 7fb2e0c00700 1 -- 172.21.15.123:0/3775353160 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb2d8009570 con 0x7fb2e4102e00 2024-03-30T05:03:24.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:24.002+0000 7fb2eba00700 1 -- 172.21.15.123:0/3775353160 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2e4070e00 con 0x7fb2e4102e00 2024-03-30T05:03:24.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:24.002+0000 7fb2eba00700 1 -- 172.21.15.123:0/3775353160 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2e4071130 con 0x7fb2e4102e00 2024-03-30T05:03:24.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:24.003+0000 7fb2eba00700 1 -- 172.21.15.123:0/3775353160 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2e40fe550 con 0x7fb2e4102e00 2024-03-30T05:03:24.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:24.004+0000 7fb2e0c00700 1 -- 172.21.15.123:0/3775353160 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fb2d80096d0 con 0x7fb2e4102e00 2024-03-30T05:03:24.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:24.005+0000 7fb2e0c00700 1 --2- 172.21.15.123:0/3775353160 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb2cc06cc30 0x7fb2cc06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:03:24.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:24.005+0000 7fb2e0c00700 1 -- 172.21.15.123:0/3775353160 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb2d808bbe0 con 0x7fb2e4102e00 2024-03-30T05:03:24.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:24.006+0000 7fb2e3e00700 1 --2- 172.21.15.123:0/3775353160 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb2cc06cc30 0x7fb2cc06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:03:24.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:24.006+0000 7fb2e3e00700 1 --2- 172.21.15.123:0/3775353160 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb2cc06cc30 0x7fb2cc06f0e0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fb2d0000c00 tx=0x7fb2d0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:03:24.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:24.012+0000 7fb2e0c00700 1 -- 172.21.15.123:0/3775353160 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb2d8059660 con 0x7fb2e4102e00 2024-03-30T05:03:24.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:03:23 smithi135 ceph-mon[106011]: from='client.35752 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:03:24.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:03:23 smithi135 ceph-mon[106011]: from='client.35756 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:03:24.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:03:23 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/1310697106' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:03:24.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:03:23 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3853201310' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T05:03:24.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:03:23 smithi123 ceph-mon[139369]: from='client.35752 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:03:24.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:03:23 smithi123 ceph-mon[139369]: from='client.35756 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:03:24.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:03:23 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/1310697106' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:03:24.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:03:23 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3853201310' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T05:03:24.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:24.295+0000 7fb2eba00700 1 -- 172.21.15.123:0/3775353160 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fb2e4071130 con 0x7fb2e4102e00 2024-03-30T05:03:24.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:24.296+0000 7fb2e0c00700 1 -- 172.21.15.123:0/3775353160 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7fb2d8019e10 con 0x7fb2e4102e00 2024-03-30T05:03:24.294 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T05:03:24.294 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-30T05:03:24.294 INFO:teuthology.orchestra.run.smithi123.stdout: host smithi135 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi135/config 2024-03-30T05:03:24.294 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-30T05:03:24.295 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T05:03:24.295 INFO:teuthology.orchestra.run.smithi123.stdout:Traceback (most recent call last): 2024-03-30T05:03:24.295 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-30T05:03:24.295 INFO:teuthology.orchestra.run.smithi123.stdout: "__main__", mod_spec) 2024-03-30T05:03:24.295 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-30T05:03:24.295 INFO:teuthology.orchestra.run.smithi123.stdout: exec(code, run_globals) 2024-03-30T05:03:24.295 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10889, in 2024-03-30T05:03:24.295 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10877, in main 2024-03-30T05:03:24.295 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-30T05:03:24.295 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-30T05:03:24.295 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-30T05:03:24.295 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-30T05:03:24.295 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-03-30T05:03:24.296 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-30T05:03:24.296 INFO:teuthology.orchestra.run.smithi123.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T05:03:24.296 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T05:03:24.296 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T05:03:24.296 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T05:03:24.296 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T05:03:24.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:24.298+0000 7fb2eba00700 1 -- 172.21.15.123:0/3775353160 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb2cc06cc30 msgr2=0x7fb2cc06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:24.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:24.298+0000 7fb2eba00700 1 --2- 172.21.15.123:0/3775353160 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb2cc06cc30 0x7fb2cc06f0e0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fb2d0000c00 tx=0x7fb2d0009e30 comp rx=0 tx=0).stop 2024-03-30T05:03:24.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:24.298+0000 7fb2eba00700 1 -- 172.21.15.123:0/3775353160 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2e4102e00 msgr2=0x7fb2e40702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:24.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:24.298+0000 7fb2eba00700 1 --2- 172.21.15.123:0/3775353160 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2e4102e00 0x7fb2e40702c0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fb2d800eb40 tx=0x7fb2d800b040 comp rx=0 tx=0).stop 2024-03-30T05:03:24.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:24.298+0000 7fb2eba00700 1 -- 172.21.15.123:0/3775353160 shutdown_connections 2024-03-30T05:03:24.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:24.298+0000 7fb2eba00700 1 --2- 172.21.15.123:0/3775353160 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb2cc06cc30 0x7fb2cc06f0e0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:24.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:24.298+0000 7fb2eba00700 1 --2- 172.21.15.123:0/3775353160 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2e4102e00 0x7fb2e40702c0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:24.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:24.298+0000 7fb2eba00700 1 --2- 172.21.15.123:0/3775353160 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb2e41037a0 0x7fb2e4070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:24.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:24.298+0000 7fb2eba00700 1 -- 172.21.15.123:0/3775353160 >> 172.21.15.123:0/3775353160 conn(0x7fb2e40f89b0 msgr2=0x7fb2e40f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:03:24.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:24.298+0000 7fb2eba00700 1 -- 172.21.15.123:0/3775353160 shutdown_connections 2024-03-30T05:03:24.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:24.298+0000 7fb2eba00700 1 -- 172.21.15.123:0/3775353160 wait complete. 2024-03-30T05:03:25.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:03:24 smithi135 ceph-mon[106011]: pgmap v967: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:25.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:03:24 smithi135 ceph-mon[106011]: from='client.35768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:03:25.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:03:24 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3775353160' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T05:03:25.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:03:24 smithi123 ceph-mon[139369]: pgmap v967: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:25.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:03:24 smithi123 ceph-mon[139369]: from='client.35768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:03:25.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:03:24 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3775353160' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T05:03:27.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:03:26 smithi135 ceph-mon[106011]: pgmap v968: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:27.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:03:26 smithi123 ceph-mon[139369]: pgmap v968: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:29.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:03:28 smithi135 ceph-mon[106011]: pgmap v969: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:29.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:03:28 smithi123 ceph-mon[139369]: pgmap v969: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:30.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:03:29 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:03:30.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:03:29 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:03:31.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:03:30 smithi135 ceph-mon[106011]: pgmap v970: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:31.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:03:30 smithi123 ceph-mon[139369]: pgmap v970: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:33.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:03:32 smithi123 ceph-mon[139369]: pgmap v971: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:33.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:03:32 smithi135 ceph-mon[106011]: pgmap v971: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:35.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:03:34 smithi123 ceph-mon[139369]: pgmap v972: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:35.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:03:34 smithi135 ceph-mon[106011]: pgmap v972: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:37.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:03:36 smithi123 ceph-mon[139369]: pgmap v973: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:37.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:03:36 smithi135 ceph-mon[106011]: pgmap v973: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:39.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:03:38 smithi123 ceph-mon[139369]: pgmap v974: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:39.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:03:38 smithi135 ceph-mon[106011]: pgmap v974: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:41.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:03:40 smithi123 ceph-mon[139369]: pgmap v975: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:41.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:03:40 smithi135 ceph-mon[106011]: pgmap v975: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:43.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:03:42 smithi123 ceph-mon[139369]: pgmap v976: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:43.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:03:42 smithi135 ceph-mon[106011]: pgmap v976: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:45.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:03:44 smithi123 ceph-mon[139369]: pgmap v977: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:45.218 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:03:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:03:45.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:03:44 smithi135 ceph-mon[106011]: pgmap v977: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:45.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:03:44 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:03:47.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:03:46 smithi123 ceph-mon[139369]: pgmap v978: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:47.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:03:46 smithi135 ceph-mon[106011]: pgmap v978: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:49.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:03:48 smithi123 ceph-mon[139369]: pgmap v979: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:49.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:03:48 smithi135 ceph-mon[106011]: pgmap v979: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:51.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:03:50 smithi123 ceph-mon[139369]: pgmap v980: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:51.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:03:50 smithi135 ceph-mon[106011]: pgmap v980: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:53.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:03:52 smithi123 ceph-mon[139369]: pgmap v981: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:53.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:03:52 smithi135 ceph-mon[106011]: pgmap v981: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:54.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.449+0000 7f2246800700 1 -- 172.21.15.123:0/2746516135 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f22401037a0 msgr2=0x7f22400fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:54.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.449+0000 7f2246800700 1 --2- 172.21.15.123:0/2746516135 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f22401037a0 0x7f22400fce10 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f222c004940 tx=0x7f222c009e30 comp rx=0 tx=0).stop 2024-03-30T05:03:54.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.450+0000 7f2246800700 1 -- 172.21.15.123:0/2746516135 shutdown_connections 2024-03-30T05:03:54.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.450+0000 7f2246800700 1 --2- 172.21.15.123:0/2746516135 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f22401037a0 0x7f22400fce10 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:54.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.450+0000 7f2246800700 1 --2- 172.21.15.123:0/2746516135 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2240102e00 0x7f22401031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:54.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.450+0000 7f2246800700 1 -- 172.21.15.123:0/2746516135 >> 172.21.15.123:0/2746516135 conn(0x7f22400f89b0 msgr2=0x7f22400fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:03:54.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.450+0000 7f2246800700 1 -- 172.21.15.123:0/2746516135 shutdown_connections 2024-03-30T05:03:54.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.450+0000 7f2246800700 1 -- 172.21.15.123:0/2746516135 wait complete. 2024-03-30T05:03:54.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.450+0000 7f2246800700 1 Processor -- start 2024-03-30T05:03:54.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.451+0000 7f2246800700 1 -- start start 2024-03-30T05:03:54.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.451+0000 7f2246800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2240102e00 0x7f2240115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:03:54.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.451+0000 7f2246800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f22401037a0 0x7f22401161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:03:54.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.451+0000 7f2246800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2240116810 con 0x7f22401037a0 2024-03-30T05:03:54.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.451+0000 7f2246800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f224010fbe0 con 0x7f2240102e00 2024-03-30T05:03:54.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.451+0000 7f223ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f22401037a0 0x7f22401161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:03:54.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.451+0000 7f223ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f22401037a0 0x7f22401161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45348/0 (socket says 172.21.15.123:45348) 2024-03-30T05:03:54.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.451+0000 7f223ea00700 1 -- 172.21.15.123:0/629550058 learned_addr learned my addr 172.21.15.123:0/629550058 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:03:54.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.451+0000 7f223f400700 1 --2- 172.21.15.123:0/629550058 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2240102e00 0x7f2240115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:03:54.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.452+0000 7f223ea00700 1 -- 172.21.15.123:0/629550058 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2240102e00 msgr2=0x7f2240115c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:54.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.452+0000 7f223ea00700 1 --2- 172.21.15.123:0/629550058 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2240102e00 0x7f2240115c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:54.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.452+0000 7f223ea00700 1 -- 172.21.15.123:0/629550058 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f222c005040 con 0x7f22401037a0 2024-03-30T05:03:54.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.452+0000 7f223ea00700 1 --2- 172.21.15.123:0/629550058 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f22401037a0 0x7f22401161c0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f222c008d50 tx=0x7f222c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:03:54.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.452+0000 7f2237400700 1 -- 172.21.15.123:0/629550058 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f222c00e030 con 0x7f22401037a0 2024-03-30T05:03:54.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.452+0000 7f2237400700 1 -- 172.21.15.123:0/629550058 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f222c0093b0 con 0x7f22401037a0 2024-03-30T05:03:54.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.452+0000 7f2237400700 1 -- 172.21.15.123:0/629550058 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f222c01cbb0 con 0x7f22401037a0 2024-03-30T05:03:54.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.452+0000 7f2246800700 1 -- 172.21.15.123:0/629550058 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f224010fe60 con 0x7f22401037a0 2024-03-30T05:03:54.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.452+0000 7f2246800700 1 -- 172.21.15.123:0/629550058 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2240110420 con 0x7f22401037a0 2024-03-30T05:03:54.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.454+0000 7f2246800700 1 -- 172.21.15.123:0/629550058 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22400fe550 con 0x7f22401037a0 2024-03-30T05:03:54.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.455+0000 7f2237400700 1 -- 172.21.15.123:0/629550058 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f222c013930 con 0x7f22401037a0 2024-03-30T05:03:54.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.456+0000 7f2237400700 1 --2- 172.21.15.123:0/629550058 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f222406cc30 0x7f222406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:03:54.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.456+0000 7f2237400700 1 -- 172.21.15.123:0/629550058 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f222c08d680 con 0x7f22401037a0 2024-03-30T05:03:54.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.456+0000 7f223f400700 1 --2- 172.21.15.123:0/629550058 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f222406cc30 0x7f222406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:03:54.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.457+0000 7f223f400700 1 --2- 172.21.15.123:0/629550058 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f222406cc30 0x7f222406f0e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f2230009b80 tx=0x7f223000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:03:54.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.461+0000 7f2237400700 1 -- 172.21.15.123:0/629550058 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f222c05b0d0 con 0x7f22401037a0 2024-03-30T05:03:54.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.683+0000 7f2246800700 1 -- 172.21.15.123:0/629550058 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2240111370 con 0x7f222406cc30 2024-03-30T05:03:54.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.684+0000 7f2237400700 1 -- 172.21.15.123:0/629550058 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f2240111370 con 0x7f222406cc30 2024-03-30T05:03:54.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.687+0000 7f2246800700 1 -- 172.21.15.123:0/629550058 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f222406cc30 msgr2=0x7f222406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:54.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.687+0000 7f2246800700 1 --2- 172.21.15.123:0/629550058 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f222406cc30 0x7f222406f0e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f2230009b80 tx=0x7f223000b040 comp rx=0 tx=0).stop 2024-03-30T05:03:54.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.687+0000 7f2246800700 1 -- 172.21.15.123:0/629550058 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f22401037a0 msgr2=0x7f22401161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:54.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.687+0000 7f2246800700 1 --2- 172.21.15.123:0/629550058 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f22401037a0 0x7f22401161c0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f222c008d50 tx=0x7f222c009e30 comp rx=0 tx=0).stop 2024-03-30T05:03:54.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.687+0000 7f2246800700 1 -- 172.21.15.123:0/629550058 shutdown_connections 2024-03-30T05:03:54.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.687+0000 7f2246800700 1 --2- 172.21.15.123:0/629550058 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f222406cc30 0x7f222406f0e0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:54.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.687+0000 7f2246800700 1 --2- 172.21.15.123:0/629550058 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2240102e00 0x7f2240115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:54.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.687+0000 7f2246800700 1 --2- 172.21.15.123:0/629550058 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f22401037a0 0x7f22401161c0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:54.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.687+0000 7f2246800700 1 -- 172.21.15.123:0/629550058 >> 172.21.15.123:0/629550058 conn(0x7f22400f89b0 msgr2=0x7f22400f99a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:03:54.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.687+0000 7f2246800700 1 -- 172.21.15.123:0/629550058 shutdown_connections 2024-03-30T05:03:54.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.687+0000 7f2246800700 1 -- 172.21.15.123:0/629550058 wait complete. 2024-03-30T05:03:54.700 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T05:03:54.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.822+0000 7f096dc00700 1 -- 172.21.15.123:0/3607741402 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0968102e00 msgr2=0x7f09681031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:54.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.822+0000 7f096dc00700 1 --2- 172.21.15.123:0/3607741402 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0968102e00 0x7f09681031d0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f0958004940 tx=0x7f0958009e30 comp rx=0 tx=0).stop 2024-03-30T05:03:54.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.822+0000 7f096dc00700 1 -- 172.21.15.123:0/3607741402 shutdown_connections 2024-03-30T05:03:54.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.822+0000 7f096dc00700 1 --2- 172.21.15.123:0/3607741402 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f09681037a0 0x7f09680fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:54.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.822+0000 7f096dc00700 1 --2- 172.21.15.123:0/3607741402 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0968102e00 0x7f09681031d0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:54.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.822+0000 7f096dc00700 1 -- 172.21.15.123:0/3607741402 >> 172.21.15.123:0/3607741402 conn(0x7f09680f89b0 msgr2=0x7f09680fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:03:54.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.822+0000 7f096dc00700 1 -- 172.21.15.123:0/3607741402 shutdown_connections 2024-03-30T05:03:54.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.822+0000 7f096dc00700 1 -- 172.21.15.123:0/3607741402 wait complete. 2024-03-30T05:03:54.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.823+0000 7f096dc00700 1 Processor -- start 2024-03-30T05:03:54.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.823+0000 7f096dc00700 1 -- start start 2024-03-30T05:03:54.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.823+0000 7f096dc00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0968102e00 0x7f0968115d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:03:54.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.823+0000 7f096dc00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09681037a0 0x7f0968116250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:03:54.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.823+0000 7f096dc00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09681168a0 con 0x7f09681037a0 2024-03-30T05:03:54.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.823+0000 7f096dc00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f096810fc70 con 0x7f0968102e00 2024-03-30T05:03:54.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.824+0000 7f0966000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09681037a0 0x7f0968116250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:03:54.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.824+0000 7f0966000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09681037a0 0x7f0968116250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45372/0 (socket says 172.21.15.123:45372) 2024-03-30T05:03:54.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.824+0000 7f0966000700 1 -- 172.21.15.123:0/2502534937 learned_addr learned my addr 172.21.15.123:0/2502534937 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:03:54.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.824+0000 7f0966a00700 1 --2- 172.21.15.123:0/2502534937 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0968102e00 0x7f0968115d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:03:54.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.824+0000 7f0966000700 1 -- 172.21.15.123:0/2502534937 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0968102e00 msgr2=0x7f0968115d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:54.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.824+0000 7f0966000700 1 --2- 172.21.15.123:0/2502534937 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0968102e00 0x7f0968115d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:54.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.824+0000 7f0966000700 1 -- 172.21.15.123:0/2502534937 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0958005040 con 0x7f09681037a0 2024-03-30T05:03:54.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.824+0000 7f0966000700 1 --2- 172.21.15.123:0/2502534937 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09681037a0 0x7f0968116250 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f0954002a50 tx=0x7f0954004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:03:54.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.825+0000 7f095ea00700 1 -- 172.21.15.123:0/2502534937 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0954007cc0 con 0x7f09681037a0 2024-03-30T05:03:54.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.825+0000 7f095ea00700 1 -- 172.21.15.123:0/2502534937 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f095400f040 con 0x7f09681037a0 2024-03-30T05:03:54.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.825+0000 7f095ea00700 1 -- 172.21.15.123:0/2502534937 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f095400d920 con 0x7f09681037a0 2024-03-30T05:03:54.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.825+0000 7f096dc00700 1 -- 172.21.15.123:0/2502534937 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f096810ff50 con 0x7f09681037a0 2024-03-30T05:03:54.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.825+0000 7f096dc00700 1 -- 172.21.15.123:0/2502534937 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0968110510 con 0x7f09681037a0 2024-03-30T05:03:54.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.826+0000 7f096dc00700 1 -- 172.21.15.123:0/2502534937 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09680fe550 con 0x7f09681037a0 2024-03-30T05:03:54.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.827+0000 7f095ea00700 1 -- 172.21.15.123:0/2502534937 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f0954009680 con 0x7f09681037a0 2024-03-30T05:03:54.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.828+0000 7f095ea00700 1 --2- 172.21.15.123:0/2502534937 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f094c06cc30 0x7f094c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:03:54.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.828+0000 7f095ea00700 1 -- 172.21.15.123:0/2502534937 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f095408bc90 con 0x7f09681037a0 2024-03-30T05:03:54.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.828+0000 7f0966a00700 1 --2- 172.21.15.123:0/2502534937 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f094c06cc30 0x7f094c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:03:54.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.828+0000 7f0966a00700 1 --2- 172.21.15.123:0/2502534937 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f094c06cc30 0x7f094c06f0e0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f0958009fd0 tx=0x7f0958009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:03:54.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:54.832+0000 7f095ea00700 1 -- 172.21.15.123:0/2502534937 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0954059760 con 0x7f09681037a0 2024-03-30T05:03:55.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.054+0000 7f096dc00700 1 -- 172.21.15.123:0/2502534937 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0968109e30 con 0x7f094c06cc30 2024-03-30T05:03:55.771 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:03:54 smithi135 ceph-mon[106011]: pgmap v982: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:55.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.056+0000 7f095ea00700 1 -- 172.21.15.123:0/2502534937 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f0968109e30 con 0x7f094c06cc30 2024-03-30T05:03:55.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.058+0000 7f096dc00700 1 -- 172.21.15.123:0/2502534937 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f094c06cc30 msgr2=0x7f094c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:55.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.058+0000 7f096dc00700 1 --2- 172.21.15.123:0/2502534937 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f094c06cc30 0x7f094c06f0e0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f0958009fd0 tx=0x7f0958009e30 comp rx=0 tx=0).stop 2024-03-30T05:03:55.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.058+0000 7f096dc00700 1 -- 172.21.15.123:0/2502534937 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09681037a0 msgr2=0x7f0968116250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:55.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.058+0000 7f096dc00700 1 --2- 172.21.15.123:0/2502534937 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09681037a0 0x7f0968116250 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f0954002a50 tx=0x7f0954004600 comp rx=0 tx=0).stop 2024-03-30T05:03:55.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.058+0000 7f096dc00700 1 -- 172.21.15.123:0/2502534937 shutdown_connections 2024-03-30T05:03:55.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.058+0000 7f096dc00700 1 --2- 172.21.15.123:0/2502534937 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f094c06cc30 0x7f094c06f0e0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:55.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.058+0000 7f096dc00700 1 --2- 172.21.15.123:0/2502534937 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0968102e00 0x7f0968115d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:55.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.058+0000 7f096dc00700 1 --2- 172.21.15.123:0/2502534937 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09681037a0 0x7f0968116250 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:55.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.058+0000 7f096dc00700 1 -- 172.21.15.123:0/2502534937 >> 172.21.15.123:0/2502534937 conn(0x7f09680f89b0 msgr2=0x7f09680704f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:03:55.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.058+0000 7f096dc00700 1 -- 172.21.15.123:0/2502534937 shutdown_connections 2024-03-30T05:03:55.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.059+0000 7f096dc00700 1 -- 172.21.15.123:0/2502534937 wait complete. 2024-03-30T05:03:55.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.192+0000 7f5778000700 1 -- 172.21.15.123:0/2357783847 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57701037a0 msgr2=0x7f57700fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:55.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.192+0000 7f5778000700 1 --2- 172.21.15.123:0/2357783847 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57701037a0 0x7f57700fce10 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f575c004940 tx=0x7f575c009e30 comp rx=0 tx=0).stop 2024-03-30T05:03:55.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.192+0000 7f5778000700 1 -- 172.21.15.123:0/2357783847 shutdown_connections 2024-03-30T05:03:55.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.192+0000 7f5778000700 1 --2- 172.21.15.123:0/2357783847 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57701037a0 0x7f57700fce10 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:55.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.192+0000 7f5778000700 1 --2- 172.21.15.123:0/2357783847 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5770102e00 0x7f57701031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:55.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.192+0000 7f5778000700 1 -- 172.21.15.123:0/2357783847 >> 172.21.15.123:0/2357783847 conn(0x7f57700f89b0 msgr2=0x7f57700fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:03:55.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.192+0000 7f5778000700 1 -- 172.21.15.123:0/2357783847 shutdown_connections 2024-03-30T05:03:55.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.192+0000 7f5778000700 1 -- 172.21.15.123:0/2357783847 wait complete. 2024-03-30T05:03:55.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.193+0000 7f5778000700 1 Processor -- start 2024-03-30T05:03:55.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.193+0000 7f5778000700 1 -- start start 2024-03-30T05:03:55.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.194+0000 7f5778000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5770102e00 0x7f5770115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:03:55.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.194+0000 7f5778000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f57701037a0 0x7f57701161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:03:55.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.194+0000 7f5778000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5770116810 con 0x7f5770102e00 2024-03-30T05:03:55.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.194+0000 7f5778000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f577010fbe0 con 0x7f57701037a0 2024-03-30T05:03:55.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.194+0000 7f5775600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5770102e00 0x7f5770115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:03:55.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.194+0000 7f5775600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5770102e00 0x7f5770115c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45396/0 (socket says 172.21.15.123:45396) 2024-03-30T05:03:55.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.194+0000 7f5775600700 1 -- 172.21.15.123:0/4010359748 learned_addr learned my addr 172.21.15.123:0/4010359748 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:03:55.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.194+0000 7f5774c00700 1 --2- 172.21.15.123:0/4010359748 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f57701037a0 0x7f57701161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:03:55.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.194+0000 7f5775600700 1 -- 172.21.15.123:0/4010359748 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f57701037a0 msgr2=0x7f57701161c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:55.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.194+0000 7f5775600700 1 --2- 172.21.15.123:0/4010359748 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f57701037a0 0x7f57701161c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:55.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.194+0000 7f5775600700 1 -- 172.21.15.123:0/4010359748 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f575c005040 con 0x7f5770102e00 2024-03-30T05:03:55.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.195+0000 7f5775600700 1 --2- 172.21.15.123:0/4010359748 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5770102e00 0x7f5770115c80 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f5764002a80 tx=0x7f57640046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:03:55.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.195+0000 7f576d600700 1 -- 172.21.15.123:0/4010359748 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5764007cc0 con 0x7f5770102e00 2024-03-30T05:03:55.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.195+0000 7f576d600700 1 -- 172.21.15.123:0/4010359748 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f576400f040 con 0x7f5770102e00 2024-03-30T05:03:55.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.195+0000 7f5778000700 1 -- 172.21.15.123:0/4010359748 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f577010fec0 con 0x7f5770102e00 2024-03-30T05:03:55.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.195+0000 7f576d600700 1 -- 172.21.15.123:0/4010359748 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5764009640 con 0x7f5770102e00 2024-03-30T05:03:55.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.195+0000 7f5778000700 1 -- 172.21.15.123:0/4010359748 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5770110480 con 0x7f5770102e00 2024-03-30T05:03:55.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.196+0000 7f5778000700 1 -- 172.21.15.123:0/4010359748 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57700fe550 con 0x7f5770102e00 2024-03-30T05:03:55.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.197+0000 7f576d600700 1 -- 172.21.15.123:0/4010359748 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f57640097a0 con 0x7f5770102e00 2024-03-30T05:03:55.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.198+0000 7f576d600700 1 --2- 172.21.15.123:0/4010359748 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f575806cc30 0x7f575806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:03:55.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.198+0000 7f576d600700 1 -- 172.21.15.123:0/4010359748 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f576408b7a0 con 0x7f5770102e00 2024-03-30T05:03:55.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.198+0000 7f5774c00700 1 --2- 172.21.15.123:0/4010359748 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f575806cc30 0x7f575806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:03:55.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.199+0000 7f5774c00700 1 --2- 172.21.15.123:0/4010359748 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f575806cc30 0x7f575806f0e0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f575c00b020 tx=0x7f575c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:03:55.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.202+0000 7f576d600700 1 -- 172.21.15.123:0/4010359748 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5764059230 con 0x7f5770102e00 2024-03-30T05:03:55.778 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:03:54 smithi123 ceph-mon[139369]: pgmap v982: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:55.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.415+0000 7f5778000700 1 -- 172.21.15.123:0/4010359748 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f577004bf40 con 0x7f575806cc30 2024-03-30T05:03:55.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.425+0000 7f576d600700 1 -- 172.21.15.123:0/4010359748 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f577004bf40 con 0x7f575806cc30 2024-03-30T05:03:55.778 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T05:03:55.779 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (33m) 10m ago 45m 14.6M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T05:03:55.779 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (30m) 10m ago 45m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T05:03:55.779 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (30m) 10m ago 44m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T05:03:55.779 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (32m) 10m ago 45m 73.5M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T05:03:55.779 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (26m) 10m ago 39m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T05:03:55.779 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (26m) 10m ago 39m 16.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T05:03:55.779 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (10m) 10m ago 39m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 87b073a6c9ca 2024-03-30T05:03:55.779 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (25m) 10m ago 39m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T05:03:55.779 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (35m) 10m ago 47m 510M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T05:03:55.779 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (34m) 10m ago 44m 456M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T05:03:55.779 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (31m) 10m ago 47m 70.0M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T05:03:55.779 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (31m) 10m ago 44m 51.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T05:03:55.779 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (33m) 10m ago 45m 8636k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T05:03:55.780 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (33m) 10m ago 44m 8678k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T05:03:55.780 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (29m) 10m ago 43m 107M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T05:03:55.780 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (29m) 10m ago 43m 97.2M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T05:03:55.780 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (28m) 10m ago 42m 89.8M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T05:03:55.780 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (28m) 10m ago 41m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T05:03:55.780 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (27m) 10m ago 41m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T05:03:55.780 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (26m) 10m ago 40m 88.8M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T05:03:55.780 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (33m) 10m ago 45m 63.9M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T05:03:55.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.427+0000 7f5778000700 1 -- 172.21.15.123:0/4010359748 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f575806cc30 msgr2=0x7f575806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:55.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.427+0000 7f5778000700 1 --2- 172.21.15.123:0/4010359748 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f575806cc30 0x7f575806f0e0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f575c00b020 tx=0x7f575c009e30 comp rx=0 tx=0).stop 2024-03-30T05:03:55.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.427+0000 7f5778000700 1 -- 172.21.15.123:0/4010359748 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5770102e00 msgr2=0x7f5770115c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:55.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.427+0000 7f5778000700 1 --2- 172.21.15.123:0/4010359748 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5770102e00 0x7f5770115c80 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f5764002a80 tx=0x7f57640046b0 comp rx=0 tx=0).stop 2024-03-30T05:03:55.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.427+0000 7f5778000700 1 -- 172.21.15.123:0/4010359748 shutdown_connections 2024-03-30T05:03:55.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.427+0000 7f5778000700 1 --2- 172.21.15.123:0/4010359748 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f575806cc30 0x7f575806f0e0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:55.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.427+0000 7f5778000700 1 --2- 172.21.15.123:0/4010359748 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5770102e00 0x7f5770115c80 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:55.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.427+0000 7f5778000700 1 --2- 172.21.15.123:0/4010359748 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f57701037a0 0x7f57701161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:55.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.427+0000 7f5778000700 1 -- 172.21.15.123:0/4010359748 >> 172.21.15.123:0/4010359748 conn(0x7f57700f89b0 msgr2=0x7f57700f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:03:55.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.427+0000 7f5778000700 1 -- 172.21.15.123:0/4010359748 shutdown_connections 2024-03-30T05:03:55.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.427+0000 7f5778000700 1 -- 172.21.15.123:0/4010359748 wait complete. 2024-03-30T05:03:55.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.558+0000 7f1afac00700 1 -- 172.21.15.123:0/374950739 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1af41037a0 msgr2=0x7f1af40fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:55.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.558+0000 7f1afac00700 1 --2- 172.21.15.123:0/374950739 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1af41037a0 0x7f1af40fce10 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0004940 tx=0x7f1ae0009e30 comp rx=0 tx=0).stop 2024-03-30T05:03:55.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.558+0000 7f1afac00700 1 -- 172.21.15.123:0/374950739 shutdown_connections 2024-03-30T05:03:55.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.559+0000 7f1afac00700 1 --2- 172.21.15.123:0/374950739 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1af41037a0 0x7f1af40fce10 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:55.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.559+0000 7f1afac00700 1 --2- 172.21.15.123:0/374950739 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1af4102e00 0x7f1af41031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:55.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.559+0000 7f1afac00700 1 -- 172.21.15.123:0/374950739 >> 172.21.15.123:0/374950739 conn(0x7f1af40f89b0 msgr2=0x7f1af40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:03:55.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.559+0000 7f1afac00700 1 -- 172.21.15.123:0/374950739 shutdown_connections 2024-03-30T05:03:55.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.559+0000 7f1afac00700 1 -- 172.21.15.123:0/374950739 wait complete. 2024-03-30T05:03:55.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.559+0000 7f1afac00700 1 Processor -- start 2024-03-30T05:03:55.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.560+0000 7f1afac00700 1 -- start start 2024-03-30T05:03:55.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.560+0000 7f1afac00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1af4102e00 0x7f1af4115d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:03:55.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.560+0000 7f1afac00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1af41037a0 0x7f1af4116240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:03:55.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.560+0000 7f1afac00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1af4116890 con 0x7f1af4102e00 2024-03-30T05:03:55.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.560+0000 7f1afac00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1af410fc60 con 0x7f1af41037a0 2024-03-30T05:03:55.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.560+0000 7f1af3e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1af4102e00 0x7f1af4115d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:03:55.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.560+0000 7f1af3e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1af4102e00 0x7f1af4115d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45412/0 (socket says 172.21.15.123:45412) 2024-03-30T05:03:55.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.560+0000 7f1af3e00700 1 -- 172.21.15.123:0/892770967 learned_addr learned my addr 172.21.15.123:0/892770967 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:03:55.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.560+0000 7f1af3400700 1 --2- 172.21.15.123:0/892770967 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1af41037a0 0x7f1af4116240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:03:55.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.561+0000 7f1af3e00700 1 -- 172.21.15.123:0/892770967 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1af41037a0 msgr2=0x7f1af4116240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:55.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.561+0000 7f1af3e00700 1 --2- 172.21.15.123:0/892770967 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1af41037a0 0x7f1af4116240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:55.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.561+0000 7f1af3e00700 1 -- 172.21.15.123:0/892770967 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae0005040 con 0x7f1af4102e00 2024-03-30T05:03:55.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.561+0000 7f1af3e00700 1 --2- 172.21.15.123:0/892770967 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1af4102e00 0x7f1af4115d00 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f1ae4002a50 tx=0x7f1ae40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:03:55.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.561+0000 7f1aebe00700 1 -- 172.21.15.123:0/892770967 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ae4007cc0 con 0x7f1af4102e00 2024-03-30T05:03:55.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.561+0000 7f1afac00700 1 -- 172.21.15.123:0/892770967 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1af410ff40 con 0x7f1af4102e00 2024-03-30T05:03:55.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.561+0000 7f1aebe00700 1 -- 172.21.15.123:0/892770967 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1ae400f040 con 0x7f1af4102e00 2024-03-30T05:03:55.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.561+0000 7f1aebe00700 1 -- 172.21.15.123:0/892770967 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ae400d9c0 con 0x7f1af4102e00 2024-03-30T05:03:55.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.561+0000 7f1afac00700 1 -- 172.21.15.123:0/892770967 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1af4110500 con 0x7f1af4102e00 2024-03-30T05:03:55.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.562+0000 7f1afac00700 1 -- 172.21.15.123:0/892770967 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1af40fe550 con 0x7f1af4102e00 2024-03-30T05:03:55.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.564+0000 7f1aebe00700 1 -- 172.21.15.123:0/892770967 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f1ae400db20 con 0x7f1af4102e00 2024-03-30T05:03:55.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.564+0000 7f1aebe00700 1 --2- 172.21.15.123:0/892770967 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1ad806cc30 0x7f1ad806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:03:55.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.565+0000 7f1aebe00700 1 -- 172.21.15.123:0/892770967 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f1ae408c450 con 0x7f1af4102e00 2024-03-30T05:03:55.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.565+0000 7f1af3400700 1 --2- 172.21.15.123:0/892770967 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1ad806cc30 0x7f1ad806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:03:55.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.565+0000 7f1af3400700 1 --2- 172.21.15.123:0/892770967 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1ad806cc30 0x7f1ad806f0e0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f1af41115a0 tx=0x7f1ae0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:03:55.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.568+0000 7f1aebe00700 1 -- 172.21.15.123:0/892770967 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f1ae4059f50 con 0x7f1af4102e00 2024-03-30T05:03:55.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.854+0000 7f1afac00700 1 -- 172.21.15.123:0/892770967 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f1af404bed0 con 0x7f1af4102e00 2024-03-30T05:03:55.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.854+0000 7f1aebe00700 1 -- 172.21.15.123:0/892770967 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f1ae4059ae0 con 0x7f1af4102e00 2024-03-30T05:03:55.853 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T05:03:55.853 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T05:03:55.853 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T05:03:55.853 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:03:55.853 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T05:03:55.853 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T05:03:55.853 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:03:55.853 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T05:03:55.853 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T05:03:55.853 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:03:55.853 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T05:03:55.853 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 4 2024-03-30T05:03:55.854 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:03:55.854 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T05:03:55.854 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 14 2024-03-30T05:03:55.854 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T05:03:55.854 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T05:03:55.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.857+0000 7f1afac00700 1 -- 172.21.15.123:0/892770967 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1ad806cc30 msgr2=0x7f1ad806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:55.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.857+0000 7f1afac00700 1 --2- 172.21.15.123:0/892770967 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1ad806cc30 0x7f1ad806f0e0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f1af41115a0 tx=0x7f1ae0009e30 comp rx=0 tx=0).stop 2024-03-30T05:03:55.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.857+0000 7f1afac00700 1 -- 172.21.15.123:0/892770967 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1af4102e00 msgr2=0x7f1af4115d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:55.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.857+0000 7f1afac00700 1 --2- 172.21.15.123:0/892770967 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1af4102e00 0x7f1af4115d00 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f1ae4002a50 tx=0x7f1ae40046b0 comp rx=0 tx=0).stop 2024-03-30T05:03:55.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.857+0000 7f1afac00700 1 -- 172.21.15.123:0/892770967 shutdown_connections 2024-03-30T05:03:55.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.857+0000 7f1afac00700 1 --2- 172.21.15.123:0/892770967 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1ad806cc30 0x7f1ad806f0e0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:55.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.857+0000 7f1afac00700 1 --2- 172.21.15.123:0/892770967 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1af4102e00 0x7f1af4115d00 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:55.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.857+0000 7f1afac00700 1 --2- 172.21.15.123:0/892770967 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1af41037a0 0x7f1af4116240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:55.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.857+0000 7f1afac00700 1 -- 172.21.15.123:0/892770967 >> 172.21.15.123:0/892770967 conn(0x7f1af40f89b0 msgr2=0x7f1af40f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:03:55.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.857+0000 7f1afac00700 1 -- 172.21.15.123:0/892770967 shutdown_connections 2024-03-30T05:03:55.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.857+0000 7f1afac00700 1 -- 172.21.15.123:0/892770967 wait complete. 2024-03-30T05:03:55.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.992+0000 7f9c19c00700 1 -- 172.21.15.123:0/76677214 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c14102e00 msgr2=0x7f9c141031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:55.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.992+0000 7f9c19c00700 1 --2- 172.21.15.123:0/76677214 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c14102e00 0x7f9c141031d0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f9c04004940 tx=0x7f9c04009e30 comp rx=0 tx=0).stop 2024-03-30T05:03:55.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.992+0000 7f9c19c00700 1 -- 172.21.15.123:0/76677214 shutdown_connections 2024-03-30T05:03:55.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.992+0000 7f9c19c00700 1 --2- 172.21.15.123:0/76677214 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c141037a0 0x7f9c140fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:55.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.992+0000 7f9c19c00700 1 --2- 172.21.15.123:0/76677214 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c14102e00 0x7f9c141031d0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:55.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.992+0000 7f9c19c00700 1 -- 172.21.15.123:0/76677214 >> 172.21.15.123:0/76677214 conn(0x7f9c140f89b0 msgr2=0x7f9c140fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:03:55.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.993+0000 7f9c19c00700 1 -- 172.21.15.123:0/76677214 shutdown_connections 2024-03-30T05:03:55.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.993+0000 7f9c19c00700 1 -- 172.21.15.123:0/76677214 wait complete. 2024-03-30T05:03:55.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.994+0000 7f9c19c00700 1 Processor -- start 2024-03-30T05:03:55.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.994+0000 7f9c19c00700 1 -- start start 2024-03-30T05:03:55.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.994+0000 7f9c19c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c14102e00 0x7f9c14115d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:03:55.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.994+0000 7f9c19c00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c141037a0 0x7f9c14116250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:03:55.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.994+0000 7f9c19c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c141168a0 con 0x7f9c14102e00 2024-03-30T05:03:55.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.994+0000 7f9c19c00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c1410fc70 con 0x7f9c141037a0 2024-03-30T05:03:55.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.995+0000 7f9c12a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c14102e00 0x7f9c14115d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:03:55.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.995+0000 7f9c12a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c14102e00 0x7f9c14115d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45432/0 (socket says 172.21.15.123:45432) 2024-03-30T05:03:55.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.995+0000 7f9c12a00700 1 -- 172.21.15.123:0/357810131 learned_addr learned my addr 172.21.15.123:0/357810131 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:03:55.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.995+0000 7f9c12000700 1 --2- 172.21.15.123:0/357810131 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c141037a0 0x7f9c14116250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:03:55.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.995+0000 7f9c12a00700 1 -- 172.21.15.123:0/357810131 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c141037a0 msgr2=0x7f9c14116250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:55.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.995+0000 7f9c12a00700 1 --2- 172.21.15.123:0/357810131 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c141037a0 0x7f9c14116250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:55.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.995+0000 7f9c12a00700 1 -- 172.21.15.123:0/357810131 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c04005040 con 0x7f9c14102e00 2024-03-30T05:03:55.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.995+0000 7f9c12a00700 1 --2- 172.21.15.123:0/357810131 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c14102e00 0x7f9c14115d10 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f9c04009390 tx=0x7f9c04009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:03:55.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.996+0000 7f9c0aa00700 1 -- 172.21.15.123:0/357810131 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c0400e040 con 0x7f9c14102e00 2024-03-30T05:03:55.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.996+0000 7f9c0aa00700 1 -- 172.21.15.123:0/357810131 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9c04013420 con 0x7f9c14102e00 2024-03-30T05:03:55.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.996+0000 7f9c19c00700 1 -- 172.21.15.123:0/357810131 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c1410fef0 con 0x7f9c14102e00 2024-03-30T05:03:55.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.996+0000 7f9c0aa00700 1 -- 172.21.15.123:0/357810131 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c0401c630 con 0x7f9c14102e00 2024-03-30T05:03:55.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.996+0000 7f9c19c00700 1 -- 172.21.15.123:0/357810131 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c141104b0 con 0x7f9c14102e00 2024-03-30T05:03:55.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.997+0000 7f9c19c00700 1 -- 172.21.15.123:0/357810131 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c140fe550 con 0x7f9c14102e00 2024-03-30T05:03:55.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.998+0000 7f9c0aa00700 1 -- 172.21.15.123:0/357810131 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f9c040139e0 con 0x7f9c14102e00 2024-03-30T05:03:55.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.998+0000 7f9c0aa00700 1 --2- 172.21.15.123:0/357810131 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9bf806cc30 0x7f9bf806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:03:55.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.999+0000 7f9c0aa00700 1 -- 172.21.15.123:0/357810131 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f9c0408d320 con 0x7f9c14102e00 2024-03-30T05:03:55.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.999+0000 7f9c12000700 1 --2- 172.21.15.123:0/357810131 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9bf806cc30 0x7f9bf806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:03:55.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:55.999+0000 7f9c12000700 1 --2- 172.21.15.123:0/357810131 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9bf806cc30 0x7f9bf806f0e0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f9c00007fd0 tx=0x7f9c00009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:03:56.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.003+0000 7f9c0aa00700 1 -- 172.21.15.123:0/357810131 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9c0405ae20 con 0x7f9c14102e00 2024-03-30T05:03:56.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:03:55 smithi123 ceph-mon[139369]: from='client.35776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:03:56.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:03:55 smithi123 ceph-mon[139369]: from='client.35780 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:03:56.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:03:55 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/892770967' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:03:56.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.250+0000 7f9c19c00700 1 -- 172.21.15.123:0/357810131 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f9c1405f5f0 con 0x7f9c14102e00 2024-03-30T05:03:56.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.250+0000 7f9c0aa00700 1 -- 172.21.15.123:0/357810131 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 34 v34) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f9c0405a9b0 con 0x7f9c14102e00 2024-03-30T05:03:56.248 INFO:teuthology.orchestra.run.smithi123.stdout:e34 2024-03-30T05:03:56.249 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T05:03:56.249 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T05:03:56.249 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T05:03:56.249 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:03:56.249 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T05:03:56.249 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T05:03:56.249 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T05:03:56.249 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T05:03:56.249 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T05:03:56.249 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T05:03:56.249 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T05:03:56.249 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T05:03:56.249 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T05:03:56.249 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T05:03:56.250 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T05:03:56.250 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T05:03:56.250 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T05:03:56.250 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T05:03:56.250 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T05:03:56.250 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T05:03:56.250 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T05:03:56.250 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T05:03:56.250 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T05:03:56.250 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T05:03:56.250 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T05:03:56.250 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T05:03:56.250 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T05:03:56.250 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T05:03:56.251 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T05:03:56.251 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T05:03:56.251 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T05:03:56.251 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T05:03:56.251 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:03:56.251 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:03:56.251 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T05:03:56.251 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:03:56.251 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:03:56.251 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:03:56.251 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:03:56.251 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44975} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/3788988014,v1:172.21.15.135:6827/3788988014] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:03:56.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.252+0000 7f9c19c00700 1 -- 172.21.15.123:0/357810131 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9bf806cc30 msgr2=0x7f9bf806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:56.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.252+0000 7f9c19c00700 1 --2- 172.21.15.123:0/357810131 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9bf806cc30 0x7f9bf806f0e0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f9c00007fd0 tx=0x7f9c00009040 comp rx=0 tx=0).stop 2024-03-30T05:03:56.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.252+0000 7f9c19c00700 1 -- 172.21.15.123:0/357810131 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c14102e00 msgr2=0x7f9c14115d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:56.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.252+0000 7f9c19c00700 1 --2- 172.21.15.123:0/357810131 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c14102e00 0x7f9c14115d10 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f9c04009390 tx=0x7f9c04009e30 comp rx=0 tx=0).stop 2024-03-30T05:03:56.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.253+0000 7f9c19c00700 1 -- 172.21.15.123:0/357810131 shutdown_connections 2024-03-30T05:03:56.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.253+0000 7f9c19c00700 1 --2- 172.21.15.123:0/357810131 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9bf806cc30 0x7f9bf806f0e0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:56.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.253+0000 7f9c19c00700 1 --2- 172.21.15.123:0/357810131 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c14102e00 0x7f9c14115d10 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:56.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.253+0000 7f9c19c00700 1 --2- 172.21.15.123:0/357810131 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c141037a0 0x7f9c14116250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:56.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.253+0000 7f9c19c00700 1 -- 172.21.15.123:0/357810131 >> 172.21.15.123:0/357810131 conn(0x7f9c140f89b0 msgr2=0x7f9c140704f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:03:56.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.253+0000 7f9c19c00700 1 -- 172.21.15.123:0/357810131 shutdown_connections 2024-03-30T05:03:56.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.253+0000 7f9c19c00700 1 -- 172.21.15.123:0/357810131 wait complete. 2024-03-30T05:03:56.253 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 34 2024-03-30T05:03:56.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:03:55 smithi135 ceph-mon[106011]: from='client.35776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:03:56.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:03:55 smithi135 ceph-mon[106011]: from='client.35780 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:03:56.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:03:55 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/892770967' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:03:56.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.386+0000 7fc22a000700 1 -- 172.21.15.123:0/2735336561 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc224102e20 msgr2=0x7fc2241031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:56.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.386+0000 7fc22a000700 1 --2- 172.21.15.123:0/2735336561 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc224102e20 0x7fc2241031f0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fc214004940 tx=0x7fc214009e30 comp rx=0 tx=0).stop 2024-03-30T05:03:56.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.386+0000 7fc22a000700 1 -- 172.21.15.123:0/2735336561 shutdown_connections 2024-03-30T05:03:56.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.386+0000 7fc22a000700 1 --2- 172.21.15.123:0/2735336561 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc2241037c0 0x7fc2240fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:56.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.386+0000 7fc22a000700 1 --2- 172.21.15.123:0/2735336561 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc224102e20 0x7fc2241031f0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:56.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.386+0000 7fc22a000700 1 -- 172.21.15.123:0/2735336561 >> 172.21.15.123:0/2735336561 conn(0x7fc2240f89d0 msgr2=0x7fc2240fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:03:56.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.386+0000 7fc22a000700 1 -- 172.21.15.123:0/2735336561 shutdown_connections 2024-03-30T05:03:56.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.386+0000 7fc22a000700 1 -- 172.21.15.123:0/2735336561 wait complete. 2024-03-30T05:03:56.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.387+0000 7fc22a000700 1 Processor -- start 2024-03-30T05:03:56.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.388+0000 7fc22a000700 1 -- start start 2024-03-30T05:03:56.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.388+0000 7fc22a000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc224102e20 0x7fc2240702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:03:56.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.388+0000 7fc22a000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2241037c0 0x7fc224070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:03:56.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.388+0000 7fc22a000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc224070d90 con 0x7fc2241037c0 2024-03-30T05:03:56.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.388+0000 7fc22a000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc224070ed0 con 0x7fc224102e20 2024-03-30T05:03:56.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.388+0000 7fc222a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2241037c0 0x7fc224070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:03:56.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.388+0000 7fc222a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2241037c0 0x7fc224070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45444/0 (socket says 172.21.15.123:45444) 2024-03-30T05:03:56.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.388+0000 7fc222a00700 1 -- 172.21.15.123:0/2093900746 learned_addr learned my addr 172.21.15.123:0/2093900746 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:03:56.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.389+0000 7fc223400700 1 --2- 172.21.15.123:0/2093900746 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc224102e20 0x7fc2240702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:03:56.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.389+0000 7fc222a00700 1 -- 172.21.15.123:0/2093900746 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc224102e20 msgr2=0x7fc2240702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:56.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.389+0000 7fc222a00700 1 --2- 172.21.15.123:0/2093900746 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc224102e20 0x7fc2240702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:56.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.389+0000 7fc222a00700 1 -- 172.21.15.123:0/2093900746 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc214005040 con 0x7fc2241037c0 2024-03-30T05:03:56.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.389+0000 7fc222a00700 1 --2- 172.21.15.123:0/2093900746 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2241037c0 0x7fc224070800 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fc210002a50 tx=0x7fc210004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:03:56.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.389+0000 7fc21b400700 1 -- 172.21.15.123:0/2093900746 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc210007cc0 con 0x7fc2241037c0 2024-03-30T05:03:56.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.389+0000 7fc21b400700 1 -- 172.21.15.123:0/2093900746 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc21000f040 con 0x7fc2241037c0 2024-03-30T05:03:56.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.389+0000 7fc22a000700 1 -- 172.21.15.123:0/2093900746 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc224119e00 con 0x7fc2241037c0 2024-03-30T05:03:56.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.389+0000 7fc21b400700 1 -- 172.21.15.123:0/2093900746 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc21000d920 con 0x7fc2241037c0 2024-03-30T05:03:56.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.389+0000 7fc22a000700 1 -- 172.21.15.123:0/2093900746 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc22411a280 con 0x7fc2241037c0 2024-03-30T05:03:56.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.390+0000 7fc22a000700 1 -- 172.21.15.123:0/2093900746 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2240fe570 con 0x7fc2241037c0 2024-03-30T05:03:56.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.391+0000 7fc21b400700 1 -- 172.21.15.123:0/2093900746 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fc210009680 con 0x7fc2241037c0 2024-03-30T05:03:56.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.392+0000 7fc21b400700 1 --2- 172.21.15.123:0/2093900746 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc20806cc30 0x7fc20806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:03:56.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.392+0000 7fc21b400700 1 -- 172.21.15.123:0/2093900746 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc21008be30 con 0x7fc2241037c0 2024-03-30T05:03:56.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.392+0000 7fc223400700 1 --2- 172.21.15.123:0/2093900746 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc20806cc30 0x7fc20806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:03:56.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.393+0000 7fc223400700 1 --2- 172.21.15.123:0/2093900746 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc20806cc30 0x7fc20806f0e0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fc214009fd0 tx=0x7fc214009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:03:56.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.397+0000 7fc21b400700 1 -- 172.21.15.123:0/2093900746 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc210055d70 con 0x7fc2241037c0 2024-03-30T05:03:56.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.616+0000 7fc22a000700 1 -- 172.21.15.123:0/2093900746 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc22404bed0 con 0x7fc20806cc30 2024-03-30T05:03:56.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.617+0000 7fc21b400700 1 -- 172.21.15.123:0/2093900746 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fc22404bed0 con 0x7fc20806cc30 2024-03-30T05:03:56.615 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T05:03:56.615 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T05:03:56.615 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T05:03:56.615 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T05:03:56.615 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T05:03:56.616 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T05:03:56.616 INFO:teuthology.orchestra.run.smithi123.stdout: "mds", 2024-03-30T05:03:56.616 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T05:03:56.616 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T05:03:56.616 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T05:03:56.616 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T05:03:56.616 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "16/21 daemons upgraded", 2024-03-30T05:03:56.616 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T05:03:56.616 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T05:03:56.616 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T05:03:56.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.619+0000 7fc22a000700 1 -- 172.21.15.123:0/2093900746 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc20806cc30 msgr2=0x7fc20806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:56.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.619+0000 7fc22a000700 1 --2- 172.21.15.123:0/2093900746 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc20806cc30 0x7fc20806f0e0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fc214009fd0 tx=0x7fc214009e30 comp rx=0 tx=0).stop 2024-03-30T05:03:56.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.619+0000 7fc22a000700 1 -- 172.21.15.123:0/2093900746 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2241037c0 msgr2=0x7fc224070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:56.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.619+0000 7fc22a000700 1 --2- 172.21.15.123:0/2093900746 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2241037c0 0x7fc224070800 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fc210002a50 tx=0x7fc210004600 comp rx=0 tx=0).stop 2024-03-30T05:03:56.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.619+0000 7fc22a000700 1 -- 172.21.15.123:0/2093900746 shutdown_connections 2024-03-30T05:03:56.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.619+0000 7fc22a000700 1 --2- 172.21.15.123:0/2093900746 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc20806cc30 0x7fc20806f0e0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:56.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.619+0000 7fc22a000700 1 --2- 172.21.15.123:0/2093900746 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc224102e20 0x7fc2240702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:56.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.619+0000 7fc22a000700 1 --2- 172.21.15.123:0/2093900746 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2241037c0 0x7fc224070800 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:56.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.619+0000 7fc22a000700 1 -- 172.21.15.123:0/2093900746 >> 172.21.15.123:0/2093900746 conn(0x7fc2240f89d0 msgr2=0x7fc2240f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:03:56.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.620+0000 7fc22a000700 1 -- 172.21.15.123:0/2093900746 shutdown_connections 2024-03-30T05:03:56.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.620+0000 7fc22a000700 1 -- 172.21.15.123:0/2093900746 wait complete. 2024-03-30T05:03:56.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.753+0000 7fa0f6400700 1 -- 172.21.15.123:0/600495538 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0f0102e00 msgr2=0x7fa0f01031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:56.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.753+0000 7fa0f6400700 1 --2- 172.21.15.123:0/600495538 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0f0102e00 0x7fa0f01031d0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fa0e0004940 tx=0x7fa0e0009e30 comp rx=0 tx=0).stop 2024-03-30T05:03:56.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.753+0000 7fa0f6400700 1 -- 172.21.15.123:0/600495538 shutdown_connections 2024-03-30T05:03:56.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.753+0000 7fa0f6400700 1 --2- 172.21.15.123:0/600495538 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa0f01037a0 0x7fa0f00fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:56.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.753+0000 7fa0f6400700 1 --2- 172.21.15.123:0/600495538 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0f0102e00 0x7fa0f01031d0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:56.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.753+0000 7fa0f6400700 1 -- 172.21.15.123:0/600495538 >> 172.21.15.123:0/600495538 conn(0x7fa0f00f89b0 msgr2=0x7fa0f00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:03:56.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.753+0000 7fa0f6400700 1 -- 172.21.15.123:0/600495538 shutdown_connections 2024-03-30T05:03:56.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.753+0000 7fa0f6400700 1 -- 172.21.15.123:0/600495538 wait complete. 2024-03-30T05:03:56.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.754+0000 7fa0f6400700 1 Processor -- start 2024-03-30T05:03:56.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.754+0000 7fa0f6400700 1 -- start start 2024-03-30T05:03:56.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.754+0000 7fa0f6400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa0f0102e00 0x7fa0f0115d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:03:56.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.754+0000 7fa0f6400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0f01037a0 0x7fa0f0116250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:03:56.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.754+0000 7fa0f6400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f01168a0 con 0x7fa0f01037a0 2024-03-30T05:03:56.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.754+0000 7fa0f6400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f010fc70 con 0x7fa0f0102e00 2024-03-30T05:03:56.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.755+0000 7fa0ef400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa0f0102e00 0x7fa0f0115d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:03:56.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.755+0000 7fa0ef400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa0f0102e00 0x7fa0f0115d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.123:60748/0 (socket says 172.21.15.123:60748) 2024-03-30T05:03:56.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.755+0000 7fa0ef400700 1 -- 172.21.15.123:0/3287629776 learned_addr learned my addr 172.21.15.123:0/3287629776 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:03:56.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.755+0000 7fa0e7e00700 1 --2- 172.21.15.123:0/3287629776 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0f01037a0 0x7fa0f0116250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:03:56.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.755+0000 7fa0ef400700 1 -- 172.21.15.123:0/3287629776 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0f01037a0 msgr2=0x7fa0f0116250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:56.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.755+0000 7fa0ef400700 1 --2- 172.21.15.123:0/3287629776 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0f01037a0 0x7fa0f0116250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:56.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.755+0000 7fa0ef400700 1 -- 172.21.15.123:0/3287629776 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0e0005040 con 0x7fa0f0102e00 2024-03-30T05:03:56.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.755+0000 7fa0ef400700 1 --2- 172.21.15.123:0/3287629776 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa0f0102e00 0x7fa0f0115d10 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa0e0009390 tx=0x7fa0e0009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:03:56.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.756+0000 7fa0e7400700 1 -- 172.21.15.123:0/3287629776 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0e000e040 con 0x7fa0f0102e00 2024-03-30T05:03:56.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.756+0000 7fa0e7400700 1 -- 172.21.15.123:0/3287629776 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa0e0013420 con 0x7fa0f0102e00 2024-03-30T05:03:56.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.756+0000 7fa0f6400700 1 -- 172.21.15.123:0/3287629776 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0f010fef0 con 0x7fa0f0102e00 2024-03-30T05:03:56.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.756+0000 7fa0e7400700 1 -- 172.21.15.123:0/3287629776 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0e001c630 con 0x7fa0f0102e00 2024-03-30T05:03:56.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.756+0000 7fa0f6400700 1 -- 172.21.15.123:0/3287629776 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0f01104b0 con 0x7fa0f0102e00 2024-03-30T05:03:56.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.757+0000 7fa0f6400700 1 -- 172.21.15.123:0/3287629776 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0f00fe550 con 0x7fa0f0102e00 2024-03-30T05:03:56.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.758+0000 7fa0e7400700 1 -- 172.21.15.123:0/3287629776 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fa0e00139e0 con 0x7fa0f0102e00 2024-03-30T05:03:56.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.759+0000 7fa0e7400700 1 --2- 172.21.15.123:0/3287629776 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa0d406cc30 0x7fa0d406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:03:56.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.759+0000 7fa0e7400700 1 -- 172.21.15.123:0/3287629776 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa0e008d320 con 0x7fa0f0102e00 2024-03-30T05:03:56.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.759+0000 7fa0e7e00700 1 --2- 172.21.15.123:0/3287629776 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa0d406cc30 0x7fa0d406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:03:56.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.760+0000 7fa0e7e00700 1 --2- 172.21.15.123:0/3287629776 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa0d406cc30 0x7fa0d406f0e0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fa0f01115b0 tx=0x7fa0d8009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:03:56.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:56.765+0000 7fa0e7400700 1 -- 172.21.15.123:0/3287629776 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa0e005ae20 con 0x7fa0f0102e00 2024-03-30T05:03:57.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:57.047+0000 7fa0f6400700 1 -- 172.21.15.123:0/3287629776 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fa0f005f5f0 con 0x7fa0f0102e00 2024-03-30T05:03:57.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:57.048+0000 7fa0e7400700 1 -- 172.21.15.123:0/3287629776 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7fa0e005a9b0 con 0x7fa0f0102e00 2024-03-30T05:03:57.046 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T05:03:57.047 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-30T05:03:57.047 INFO:teuthology.orchestra.run.smithi123.stdout: host smithi135 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi135/config 2024-03-30T05:03:57.047 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-30T05:03:57.047 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T05:03:57.047 INFO:teuthology.orchestra.run.smithi123.stdout:Traceback (most recent call last): 2024-03-30T05:03:57.047 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-30T05:03:57.047 INFO:teuthology.orchestra.run.smithi123.stdout: "__main__", mod_spec) 2024-03-30T05:03:57.047 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-30T05:03:57.047 INFO:teuthology.orchestra.run.smithi123.stdout: exec(code, run_globals) 2024-03-30T05:03:57.047 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10889, in 2024-03-30T05:03:57.047 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10877, in main 2024-03-30T05:03:57.047 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-30T05:03:57.047 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-30T05:03:57.048 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-30T05:03:57.048 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-30T05:03:57.048 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-03-30T05:03:57.048 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-30T05:03:57.048 INFO:teuthology.orchestra.run.smithi123.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T05:03:57.048 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T05:03:57.048 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T05:03:57.048 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T05:03:57.048 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T05:03:57.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:57.051+0000 7fa0f6400700 1 -- 172.21.15.123:0/3287629776 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa0d406cc30 msgr2=0x7fa0d406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:57.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:57.051+0000 7fa0f6400700 1 --2- 172.21.15.123:0/3287629776 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa0d406cc30 0x7fa0d406f0e0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fa0f01115b0 tx=0x7fa0d8009ee0 comp rx=0 tx=0).stop 2024-03-30T05:03:57.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:57.051+0000 7fa0f6400700 1 -- 172.21.15.123:0/3287629776 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa0f0102e00 msgr2=0x7fa0f0115d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:03:57.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:57.051+0000 7fa0f6400700 1 --2- 172.21.15.123:0/3287629776 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa0f0102e00 0x7fa0f0115d10 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa0e0009390 tx=0x7fa0e0009e30 comp rx=0 tx=0).stop 2024-03-30T05:03:57.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:57.051+0000 7fa0f6400700 1 -- 172.21.15.123:0/3287629776 shutdown_connections 2024-03-30T05:03:57.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:57.051+0000 7fa0f6400700 1 --2- 172.21.15.123:0/3287629776 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa0d406cc30 0x7fa0d406f0e0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:57.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:57.051+0000 7fa0f6400700 1 --2- 172.21.15.123:0/3287629776 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa0f0102e00 0x7fa0f0115d10 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:57.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:57.051+0000 7fa0f6400700 1 --2- 172.21.15.123:0/3287629776 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0f01037a0 0x7fa0f0116250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:03:57.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:57.051+0000 7fa0f6400700 1 -- 172.21.15.123:0/3287629776 >> 172.21.15.123:0/3287629776 conn(0x7fa0f00f89b0 msgr2=0x7fa0f00704f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:03:57.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:57.051+0000 7fa0f6400700 1 -- 172.21.15.123:0/3287629776 shutdown_connections 2024-03-30T05:03:57.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:03:57.051+0000 7fa0f6400700 1 -- 172.21.15.123:0/3287629776 wait complete. 2024-03-30T05:03:57.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:03:56 smithi123 ceph-mon[139369]: from='client.35784 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:03:57.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:03:56 smithi123 ceph-mon[139369]: pgmap v983: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:57.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:03:56 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/357810131' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T05:03:57.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:03:56 smithi135 ceph-mon[106011]: from='client.35784 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:03:57.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:03:56 smithi135 ceph-mon[106011]: pgmap v983: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:57.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:03:56 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/357810131' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T05:03:58.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:03:57 smithi123 ceph-mon[139369]: from='client.35796 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:03:58.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:03:57 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3287629776' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T05:03:58.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:03:57 smithi135 ceph-mon[106011]: from='client.35796 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:03:58.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:03:57 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3287629776' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T05:03:59.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:03:58 smithi123 ceph-mon[139369]: pgmap v984: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:03:59.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:03:58 smithi135 ceph-mon[106011]: pgmap v984: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:00.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:03:59 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:04:00.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:03:59 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:04:01.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:04:00 smithi123 ceph-mon[139369]: pgmap v985: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:01.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:04:00 smithi135 ceph-mon[106011]: pgmap v985: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:03.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:04:02 smithi123 ceph-mon[139369]: pgmap v986: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:03.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:04:02 smithi135 ceph-mon[106011]: pgmap v986: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:05.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:04:04 smithi123 ceph-mon[139369]: pgmap v987: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:05.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:04:04 smithi135 ceph-mon[106011]: pgmap v987: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:07.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:04:06 smithi123 ceph-mon[139369]: pgmap v988: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:07.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:04:06 smithi135 ceph-mon[106011]: pgmap v988: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:09.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:04:08 smithi123 ceph-mon[139369]: pgmap v989: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:09.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:04:08 smithi135 ceph-mon[106011]: pgmap v989: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:11.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:04:10 smithi123 ceph-mon[139369]: pgmap v990: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:11.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:04:10 smithi135 ceph-mon[106011]: pgmap v990: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:13.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:04:12 smithi123 ceph-mon[139369]: pgmap v991: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:13.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:04:12 smithi135 ceph-mon[106011]: pgmap v991: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:15.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:04:14 smithi123 ceph-mon[139369]: pgmap v992: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:15.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:04:14 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:04:15.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:04:14 smithi135 ceph-mon[106011]: pgmap v992: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:15.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:04:14 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:04:17.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:04:16 smithi123 ceph-mon[139369]: pgmap v993: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:17.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:04:16 smithi135 ceph-mon[106011]: pgmap v993: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:19.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:04:18 smithi123 ceph-mon[139369]: pgmap v994: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:19.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:04:18 smithi135 ceph-mon[106011]: pgmap v994: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:21.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:04:20 smithi123 ceph-mon[139369]: pgmap v995: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:21.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:04:20 smithi135 ceph-mon[106011]: pgmap v995: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:23.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:04:22 smithi123 ceph-mon[139369]: pgmap v996: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:23.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:04:22 smithi135 ceph-mon[106011]: pgmap v996: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:25.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:04:24 smithi135 ceph-mon[106011]: pgmap v997: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:25.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:04:24 smithi123 ceph-mon[139369]: pgmap v997: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:27.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.197+0000 7f840f200700 1 -- 172.21.15.123:0/910485780 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8408102e00 msgr2=0x7f84081031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:04:27.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.197+0000 7f840f200700 1 --2- 172.21.15.123:0/910485780 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8408102e00 0x7f84081031d0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f83f8004940 tx=0x7f83f8009e30 comp rx=0 tx=0).stop 2024-03-30T05:04:27.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.197+0000 7f840f200700 1 -- 172.21.15.123:0/910485780 shutdown_connections 2024-03-30T05:04:27.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.197+0000 7f840f200700 1 --2- 172.21.15.123:0/910485780 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f84081037a0 0x7f84080fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:04:27.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.197+0000 7f840f200700 1 --2- 172.21.15.123:0/910485780 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8408102e00 0x7f84081031d0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:04:27.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.197+0000 7f840f200700 1 -- 172.21.15.123:0/910485780 >> 172.21.15.123:0/910485780 conn(0x7f84080f89b0 msgr2=0x7f84080fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:04:27.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.198+0000 7f840f200700 1 -- 172.21.15.123:0/910485780 shutdown_connections 2024-03-30T05:04:27.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.198+0000 7f840f200700 1 -- 172.21.15.123:0/910485780 wait complete. 2024-03-30T05:04:27.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.198+0000 7f840f200700 1 Processor -- start 2024-03-30T05:04:27.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.198+0000 7f840f200700 1 -- start start 2024-03-30T05:04:27.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.199+0000 7f840f200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8408102e00 0x7f84080702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:04:27.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.199+0000 7f840f200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f84081037a0 0x7f8408070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:04:27.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.199+0000 7f840f200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8408070d90 con 0x7f8408102e00 2024-03-30T05:04:27.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.199+0000 7f840f200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8408070ed0 con 0x7f84081037a0 2024-03-30T05:04:27.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.199+0000 7f8407e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8408102e00 0x7f84080702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:04:27.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.199+0000 7f8407e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8408102e00 0x7f84080702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45244/0 (socket says 172.21.15.123:45244) 2024-03-30T05:04:27.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.199+0000 7f8407e00700 1 -- 172.21.15.123:0/2948214875 learned_addr learned my addr 172.21.15.123:0/2948214875 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:04:27.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.199+0000 7f8407400700 1 --2- 172.21.15.123:0/2948214875 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f84081037a0 0x7f8408070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:04:27.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.199+0000 7f8407e00700 1 -- 172.21.15.123:0/2948214875 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f84081037a0 msgr2=0x7f8408070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:04:27.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.199+0000 7f8407e00700 1 --2- 172.21.15.123:0/2948214875 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f84081037a0 0x7f8408070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:04:27.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.199+0000 7f8407e00700 1 -- 172.21.15.123:0/2948214875 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83f8005040 con 0x7f8408102e00 2024-03-30T05:04:27.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.200+0000 7f8407e00700 1 --2- 172.21.15.123:0/2948214875 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8408102e00 0x7f84080702c0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f83f8009360 tx=0x7f83f8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:04:27.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.200+0000 7f83ffe00700 1 -- 172.21.15.123:0/2948214875 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f83f800e040 con 0x7f8408102e00 2024-03-30T05:04:27.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.200+0000 7f840f200700 1 -- 172.21.15.123:0/2948214875 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8408071120 con 0x7f8408102e00 2024-03-30T05:04:27.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.200+0000 7f83ffe00700 1 -- 172.21.15.123:0/2948214875 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f83f8013420 con 0x7f8408102e00 2024-03-30T05:04:27.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.200+0000 7f83ffe00700 1 -- 172.21.15.123:0/2948214875 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f83f801c5d0 con 0x7f8408102e00 2024-03-30T05:04:27.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.200+0000 7f840f200700 1 -- 172.21.15.123:0/2948214875 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f840811a250 con 0x7f8408102e00 2024-03-30T05:04:27.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.201+0000 7f840f200700 1 -- 172.21.15.123:0/2948214875 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84080fe550 con 0x7f8408102e00 2024-03-30T05:04:27.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.202+0000 7f83ffe00700 1 -- 172.21.15.123:0/2948214875 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f83f8013590 con 0x7f8408102e00 2024-03-30T05:04:27.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.203+0000 7f83ffe00700 1 --2- 172.21.15.123:0/2948214875 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f83ec06cc30 0x7f83ec06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:04:27.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.203+0000 7f83ffe00700 1 -- 172.21.15.123:0/2948214875 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f83f808d250 con 0x7f8408102e00 2024-03-30T05:04:27.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.203+0000 7f8407400700 1 --2- 172.21.15.123:0/2948214875 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f83ec06cc30 0x7f83ec06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:04:27.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.204+0000 7f8407400700 1 --2- 172.21.15.123:0/2948214875 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f83ec06cc30 0x7f83ec06f0e0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f8408071c00 tx=0x7f83f4009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:04:27.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.208+0000 7f83ffe00700 1 -- 172.21.15.123:0/2948214875 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f83f8057190 con 0x7f8408102e00 2024-03-30T05:04:27.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:04:26 smithi135 ceph-mon[106011]: pgmap v998: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:27.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.431+0000 7f840f200700 1 -- 172.21.15.123:0/2948214875 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f84080719b0 con 0x7f83ec06cc30 2024-03-30T05:04:27.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.432+0000 7f83ffe00700 1 -- 172.21.15.123:0/2948214875 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f84080719b0 con 0x7f83ec06cc30 2024-03-30T05:04:27.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.434+0000 7f840f200700 1 -- 172.21.15.123:0/2948214875 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f83ec06cc30 msgr2=0x7f83ec06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:04:27.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.434+0000 7f840f200700 1 --2- 172.21.15.123:0/2948214875 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f83ec06cc30 0x7f83ec06f0e0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f8408071c00 tx=0x7f83f4009040 comp rx=0 tx=0).stop 2024-03-30T05:04:27.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.434+0000 7f840f200700 1 -- 172.21.15.123:0/2948214875 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8408102e00 msgr2=0x7f84080702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:04:27.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.434+0000 7f840f200700 1 --2- 172.21.15.123:0/2948214875 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8408102e00 0x7f84080702c0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f83f8009360 tx=0x7f83f8009e30 comp rx=0 tx=0).stop 2024-03-30T05:04:27.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.434+0000 7f840f200700 1 -- 172.21.15.123:0/2948214875 shutdown_connections 2024-03-30T05:04:27.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.434+0000 7f840f200700 1 --2- 172.21.15.123:0/2948214875 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f83ec06cc30 0x7f83ec06f0e0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:04:27.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.434+0000 7f840f200700 1 --2- 172.21.15.123:0/2948214875 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8408102e00 0x7f84080702c0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:04:27.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.434+0000 7f840f200700 1 --2- 172.21.15.123:0/2948214875 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f84081037a0 0x7f8408070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:04:27.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.434+0000 7f840f200700 1 -- 172.21.15.123:0/2948214875 >> 172.21.15.123:0/2948214875 conn(0x7f84080f89b0 msgr2=0x7f84080f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:04:27.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.434+0000 7f840f200700 1 -- 172.21.15.123:0/2948214875 shutdown_connections 2024-03-30T05:04:27.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.435+0000 7f840f200700 1 -- 172.21.15.123:0/2948214875 wait complete. 2024-03-30T05:04:27.448 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T05:04:27.465 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:04:26 smithi123 ceph-mon[139369]: pgmap v998: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:27.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.568+0000 7f31bec00700 1 -- 172.21.15.123:0/1787938463 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31b8102e00 msgr2=0x7f31b81031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:04:27.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.568+0000 7f31bec00700 1 --2- 172.21.15.123:0/1787938463 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31b8102e00 0x7f31b81031d0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f31a8004940 tx=0x7f31a8009e30 comp rx=0 tx=0).stop 2024-03-30T05:04:27.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.568+0000 7f31bec00700 1 -- 172.21.15.123:0/1787938463 shutdown_connections 2024-03-30T05:04:27.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.568+0000 7f31bec00700 1 --2- 172.21.15.123:0/1787938463 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31b81037a0 0x7f31b80fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:04:27.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.568+0000 7f31bec00700 1 --2- 172.21.15.123:0/1787938463 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31b8102e00 0x7f31b81031d0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:04:27.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.568+0000 7f31bec00700 1 -- 172.21.15.123:0/1787938463 >> 172.21.15.123:0/1787938463 conn(0x7f31b80f89b0 msgr2=0x7f31b80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:04:27.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.568+0000 7f31bec00700 1 -- 172.21.15.123:0/1787938463 shutdown_connections 2024-03-30T05:04:27.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.569+0000 7f31bec00700 1 -- 172.21.15.123:0/1787938463 wait complete. 2024-03-30T05:04:27.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.569+0000 7f31bec00700 1 Processor -- start 2024-03-30T05:04:27.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.569+0000 7f31bec00700 1 -- start start 2024-03-30T05:04:27.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.570+0000 7f31bec00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31b8102e00 0x7f31b8115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:04:27.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.570+0000 7f31bec00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31b81037a0 0x7f31b8116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:04:27.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.570+0000 7f31bec00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b8116860 con 0x7f31b81037a0 2024-03-30T05:04:27.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.570+0000 7f31bec00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b810fc30 con 0x7f31b8102e00 2024-03-30T05:04:27.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.570+0000 7f31b7400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31b81037a0 0x7f31b8116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:04:27.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.570+0000 7f31b7e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31b8102e00 0x7f31b8115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:04:27.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.570+0000 7f31b7400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31b81037a0 0x7f31b8116210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45256/0 (socket says 172.21.15.123:45256) 2024-03-30T05:04:27.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.570+0000 7f31b7400700 1 -- 172.21.15.123:0/2337211579 learned_addr learned my addr 172.21.15.123:0/2337211579 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:04:27.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.570+0000 7f31b7400700 1 -- 172.21.15.123:0/2337211579 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31b8102e00 msgr2=0x7f31b8115cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:04:27.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.570+0000 7f31b7400700 1 --2- 172.21.15.123:0/2337211579 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31b8102e00 0x7f31b8115cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:04:27.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.570+0000 7f31b7400700 1 -- 172.21.15.123:0/2337211579 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a8005040 con 0x7f31b81037a0 2024-03-30T05:04:27.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.571+0000 7f31b7400700 1 --2- 172.21.15.123:0/2337211579 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31b81037a0 0x7f31b8116210 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f31a400c890 tx=0x7f31a40044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:04:27.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.571+0000 7f31afe00700 1 -- 172.21.15.123:0/2337211579 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31a400a700 con 0x7f31b81037a0 2024-03-30T05:04:27.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.571+0000 7f31afe00700 1 -- 172.21.15.123:0/2337211579 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f31a400ad10 con 0x7f31b81037a0 2024-03-30T05:04:27.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.571+0000 7f31afe00700 1 -- 172.21.15.123:0/2337211579 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31a4012e70 con 0x7f31b81037a0 2024-03-30T05:04:27.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.571+0000 7f31bec00700 1 -- 172.21.15.123:0/2337211579 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31b810ff10 con 0x7f31b81037a0 2024-03-30T05:04:27.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.571+0000 7f31bec00700 1 -- 172.21.15.123:0/2337211579 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31b8110480 con 0x7f31b81037a0 2024-03-30T05:04:27.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.572+0000 7f31bec00700 1 -- 172.21.15.123:0/2337211579 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31b80fe550 con 0x7f31b81037a0 2024-03-30T05:04:27.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.574+0000 7f31afe00700 1 -- 172.21.15.123:0/2337211579 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f31a400a860 con 0x7f31b81037a0 2024-03-30T05:04:27.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.574+0000 7f31afe00700 1 --2- 172.21.15.123:0/2337211579 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f319c06cc20 0x7f319c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:04:27.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.574+0000 7f31afe00700 1 -- 172.21.15.123:0/2337211579 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f31a408bf90 con 0x7f31b81037a0 2024-03-30T05:04:27.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.575+0000 7f31b7e00700 1 --2- 172.21.15.123:0/2337211579 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f319c06cc20 0x7f319c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:04:27.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.575+0000 7f31b7e00700 1 --2- 172.21.15.123:0/2337211579 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f319c06cc20 0x7f319c06f0d0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f31a8004f90 tx=0x7f31a8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:04:27.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.579+0000 7f31afe00700 1 -- 172.21.15.123:0/2337211579 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f31a4059b20 con 0x7f31b81037a0 2024-03-30T05:04:27.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.804+0000 7f31bec00700 1 -- 172.21.15.123:0/2337211579 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f31b8109e30 con 0x7f319c06cc20 2024-03-30T05:04:27.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.806+0000 7f31afe00700 1 -- 172.21.15.123:0/2337211579 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f31b8109e30 con 0x7f319c06cc20 2024-03-30T05:04:27.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.808+0000 7f31bec00700 1 -- 172.21.15.123:0/2337211579 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f319c06cc20 msgr2=0x7f319c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:04:27.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.808+0000 7f31bec00700 1 --2- 172.21.15.123:0/2337211579 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f319c06cc20 0x7f319c06f0d0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f31a8004f90 tx=0x7f31a8009e30 comp rx=0 tx=0).stop 2024-03-30T05:04:27.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.808+0000 7f31bec00700 1 -- 172.21.15.123:0/2337211579 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31b81037a0 msgr2=0x7f31b8116210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:04:27.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.808+0000 7f31bec00700 1 --2- 172.21.15.123:0/2337211579 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31b81037a0 0x7f31b8116210 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f31a400c890 tx=0x7f31a40044c0 comp rx=0 tx=0).stop 2024-03-30T05:04:27.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.808+0000 7f31bec00700 1 -- 172.21.15.123:0/2337211579 shutdown_connections 2024-03-30T05:04:27.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.808+0000 7f31bec00700 1 --2- 172.21.15.123:0/2337211579 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f319c06cc20 0x7f319c06f0d0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:04:27.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.808+0000 7f31bec00700 1 --2- 172.21.15.123:0/2337211579 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31b8102e00 0x7f31b8115cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:04:27.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.808+0000 7f31bec00700 1 --2- 172.21.15.123:0/2337211579 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31b81037a0 0x7f31b8116210 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:04:27.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.808+0000 7f31bec00700 1 -- 172.21.15.123:0/2337211579 >> 172.21.15.123:0/2337211579 conn(0x7f31b80f89b0 msgr2=0x7f31b80f99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:04:27.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.808+0000 7f31bec00700 1 -- 172.21.15.123:0/2337211579 shutdown_connections 2024-03-30T05:04:27.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.809+0000 7f31bec00700 1 -- 172.21.15.123:0/2337211579 wait complete. 2024-03-30T05:04:27.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.942+0000 7fb245c00700 1 -- 172.21.15.123:0/2971769113 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb240070c70 msgr2=0x7fb240109110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:04:27.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.942+0000 7fb245c00700 1 --2- 172.21.15.123:0/2971769113 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb240070c70 0x7fb240109110 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fb22c004970 tx=0x7fb22c009e30 comp rx=0 tx=0).stop 2024-03-30T05:04:27.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.942+0000 7fb245c00700 1 -- 172.21.15.123:0/2971769113 shutdown_connections 2024-03-30T05:04:27.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.942+0000 7fb245c00700 1 --2- 172.21.15.123:0/2971769113 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb240070c70 0x7fb240109110 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:04:27.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.942+0000 7fb245c00700 1 --2- 172.21.15.123:0/2971769113 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb240070360 0x7fb240070730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:04:27.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.942+0000 7fb245c00700 1 -- 172.21.15.123:0/2971769113 >> 172.21.15.123:0/2971769113 conn(0x7fb2400f89b0 msgr2=0x7fb2400fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:04:27.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.942+0000 7fb245c00700 1 -- 172.21.15.123:0/2971769113 shutdown_connections 2024-03-30T05:04:27.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.942+0000 7fb245c00700 1 -- 172.21.15.123:0/2971769113 wait complete. 2024-03-30T05:04:27.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.943+0000 7fb245c00700 1 Processor -- start 2024-03-30T05:04:27.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.943+0000 7fb245c00700 1 -- start start 2024-03-30T05:04:27.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.943+0000 7fb245c00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb240070360 0x7fb240115ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:04:27.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.943+0000 7fb245c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb240070c70 0x7fb2401161e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:04:27.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.943+0000 7fb245c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb240116830 con 0x7fb240070c70 2024-03-30T05:04:27.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.943+0000 7fb245c00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb24010fc00 con 0x7fb240070360 2024-03-30T05:04:27.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.943+0000 7fb23e000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb240070c70 0x7fb2401161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:04:27.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.943+0000 7fb23e000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb240070c70 0x7fb2401161e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45262/0 (socket says 172.21.15.123:45262) 2024-03-30T05:04:27.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.944+0000 7fb23e000700 1 -- 172.21.15.123:0/3449723022 learned_addr learned my addr 172.21.15.123:0/3449723022 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:04:27.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.944+0000 7fb23ea00700 1 --2- 172.21.15.123:0/3449723022 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb240070360 0x7fb240115ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:04:27.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.944+0000 7fb23e000700 1 -- 172.21.15.123:0/3449723022 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb240070360 msgr2=0x7fb240115ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:04:27.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.944+0000 7fb23e000700 1 --2- 172.21.15.123:0/3449723022 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb240070360 0x7fb240115ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:04:27.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.944+0000 7fb23e000700 1 -- 172.21.15.123:0/3449723022 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb22c005040 con 0x7fb240070c70 2024-03-30T05:04:27.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.944+0000 7fb23e000700 1 --2- 172.21.15.123:0/3449723022 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb240070c70 0x7fb2401161e0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fb22c004f50 tx=0x7fb22c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:04:27.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.944+0000 7fb236a00700 1 -- 172.21.15.123:0/3449723022 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb22c00e040 con 0x7fb240070c70 2024-03-30T05:04:27.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.944+0000 7fb245c00700 1 -- 172.21.15.123:0/3449723022 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb24010fe80 con 0x7fb240070c70 2024-03-30T05:04:27.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.944+0000 7fb236a00700 1 -- 172.21.15.123:0/3449723022 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb22c013420 con 0x7fb240070c70 2024-03-30T05:04:27.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.944+0000 7fb236a00700 1 -- 172.21.15.123:0/3449723022 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb22c01c630 con 0x7fb240070c70 2024-03-30T05:04:27.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.945+0000 7fb245c00700 1 -- 172.21.15.123:0/3449723022 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2401103c0 con 0x7fb240070c70 2024-03-30T05:04:27.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.946+0000 7fb245c00700 1 -- 172.21.15.123:0/3449723022 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb240103bd0 con 0x7fb240070c70 2024-03-30T05:04:27.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.947+0000 7fb236a00700 1 -- 172.21.15.123:0/3449723022 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fb22c013590 con 0x7fb240070c70 2024-03-30T05:04:27.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.947+0000 7fb236a00700 1 --2- 172.21.15.123:0/3449723022 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb22406cc30 0x7fb22406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:04:27.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.947+0000 7fb236a00700 1 -- 172.21.15.123:0/3449723022 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb22c08d280 con 0x7fb240070c70 2024-03-30T05:04:27.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.948+0000 7fb23ea00700 1 --2- 172.21.15.123:0/3449723022 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb22406cc30 0x7fb22406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:04:27.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.948+0000 7fb23ea00700 1 --2- 172.21.15.123:0/3449723022 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb22406cc30 0x7fb22406f0e0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fb230004f40 tx=0x7fb23000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:04:27.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:27.952+0000 7fb236a00700 1 -- 172.21.15.123:0/3449723022 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb22c05acd0 con 0x7fb240070c70 2024-03-30T05:04:28.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.163+0000 7fb245c00700 1 -- 172.21.15.123:0/3449723022 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb2401112f0 con 0x7fb22406cc30 2024-03-30T05:04:28.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.173+0000 7fb236a00700 1 -- 172.21.15.123:0/3449723022 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fb2401112f0 con 0x7fb22406cc30 2024-03-30T05:04:28.171 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T05:04:28.171 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (33m) 11m ago 46m 14.6M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T05:04:28.171 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (30m) 11m ago 46m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T05:04:28.171 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (30m) 11m ago 45m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T05:04:28.172 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (33m) 11m ago 45m 73.5M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T05:04:28.172 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (26m) 11m ago 39m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T05:04:28.172 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (26m) 11m ago 39m 16.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T05:04:28.172 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (11m) 11m ago 39m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 87b073a6c9ca 2024-03-30T05:04:28.172 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (26m) 11m ago 39m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T05:04:28.172 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (35m) 11m ago 47m 510M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T05:04:28.172 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (34m) 11m ago 45m 456M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T05:04:28.172 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (32m) 11m ago 47m 70.0M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T05:04:28.172 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (31m) 11m ago 44m 51.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T05:04:28.172 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (34m) 11m ago 46m 8636k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T05:04:28.172 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (34m) 11m ago 45m 8678k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T05:04:28.172 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (30m) 11m ago 44m 107M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T05:04:28.172 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (29m) 11m ago 43m 97.2M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T05:04:28.172 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (29m) 11m ago 43m 89.8M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T05:04:28.173 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (28m) 11m ago 42m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T05:04:28.173 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (27m) 11m ago 41m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T05:04:28.173 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (27m) 11m ago 41m 88.8M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T05:04:28.173 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (33m) 11m ago 45m 63.9M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T05:04:28.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.175+0000 7fb245c00700 1 -- 172.21.15.123:0/3449723022 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb22406cc30 msgr2=0x7fb22406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:04:28.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.175+0000 7fb245c00700 1 --2- 172.21.15.123:0/3449723022 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb22406cc30 0x7fb22406f0e0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fb230004f40 tx=0x7fb23000b040 comp rx=0 tx=0).stop 2024-03-30T05:04:28.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.175+0000 7fb245c00700 1 -- 172.21.15.123:0/3449723022 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb240070c70 msgr2=0x7fb2401161e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:04:28.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.175+0000 7fb245c00700 1 --2- 172.21.15.123:0/3449723022 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb240070c70 0x7fb2401161e0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fb22c004f50 tx=0x7fb22c009e30 comp rx=0 tx=0).stop 2024-03-30T05:04:28.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.176+0000 7fb245c00700 1 -- 172.21.15.123:0/3449723022 shutdown_connections 2024-03-30T05:04:28.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.176+0000 7fb245c00700 1 --2- 172.21.15.123:0/3449723022 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb22406cc30 0x7fb22406f0e0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:04:28.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.176+0000 7fb245c00700 1 --2- 172.21.15.123:0/3449723022 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb240070360 0x7fb240115ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:04:28.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.176+0000 7fb245c00700 1 --2- 172.21.15.123:0/3449723022 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb240070c70 0x7fb2401161e0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:04:28.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.176+0000 7fb245c00700 1 -- 172.21.15.123:0/3449723022 >> 172.21.15.123:0/3449723022 conn(0x7fb2400f89b0 msgr2=0x7fb2401015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:04:28.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.176+0000 7fb245c00700 1 -- 172.21.15.123:0/3449723022 shutdown_connections 2024-03-30T05:04:28.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.176+0000 7fb245c00700 1 -- 172.21.15.123:0/3449723022 wait complete. 2024-03-30T05:04:28.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.309+0000 7f2a6ec00700 1 -- 172.21.15.123:0/3585320538 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2a68102e00 msgr2=0x7f2a681031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:04:28.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.309+0000 7f2a6ec00700 1 --2- 172.21.15.123:0/3585320538 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2a68102e00 0x7f2a681031d0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f2a58004940 tx=0x7f2a58009e30 comp rx=0 tx=0).stop 2024-03-30T05:04:28.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.309+0000 7f2a6ec00700 1 -- 172.21.15.123:0/3585320538 shutdown_connections 2024-03-30T05:04:28.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.309+0000 7f2a6ec00700 1 --2- 172.21.15.123:0/3585320538 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2a681037a0 0x7f2a680fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:04:28.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.309+0000 7f2a6ec00700 1 --2- 172.21.15.123:0/3585320538 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2a68102e00 0x7f2a681031d0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:04:28.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.309+0000 7f2a6ec00700 1 -- 172.21.15.123:0/3585320538 >> 172.21.15.123:0/3585320538 conn(0x7f2a680f89b0 msgr2=0x7f2a680fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:04:28.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.309+0000 7f2a6ec00700 1 -- 172.21.15.123:0/3585320538 shutdown_connections 2024-03-30T05:04:28.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.309+0000 7f2a6ec00700 1 -- 172.21.15.123:0/3585320538 wait complete. 2024-03-30T05:04:28.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.310+0000 7f2a6ec00700 1 Processor -- start 2024-03-30T05:04:28.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.310+0000 7f2a6ec00700 1 -- start start 2024-03-30T05:04:28.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.311+0000 7f2a6ec00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2a68102e00 0x7f2a680702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:04:28.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.311+0000 7f2a6ec00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2a681037a0 0x7f2a68070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:04:28.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.311+0000 7f2a6ec00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a68070d90 con 0x7f2a68102e00 2024-03-30T05:04:28.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.311+0000 7f2a6ec00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a68070ed0 con 0x7f2a681037a0 2024-03-30T05:04:28.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.311+0000 7f2a67e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2a68102e00 0x7f2a680702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:04:28.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.311+0000 7f2a67e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2a68102e00 0x7f2a680702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45286/0 (socket says 172.21.15.123:45286) 2024-03-30T05:04:28.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.311+0000 7f2a67e00700 1 -- 172.21.15.123:0/3671091859 learned_addr learned my addr 172.21.15.123:0/3671091859 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:04:28.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.311+0000 7f2a67400700 1 --2- 172.21.15.123:0/3671091859 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2a681037a0 0x7f2a68070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:04:28.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.312+0000 7f2a67e00700 1 -- 172.21.15.123:0/3671091859 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2a681037a0 msgr2=0x7f2a68070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:04:28.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.312+0000 7f2a67e00700 1 --2- 172.21.15.123:0/3671091859 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2a681037a0 0x7f2a68070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:04:28.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.312+0000 7f2a67e00700 1 -- 172.21.15.123:0/3671091859 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a58005040 con 0x7f2a68102e00 2024-03-30T05:04:28.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.312+0000 7f2a67e00700 1 --2- 172.21.15.123:0/3671091859 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2a68102e00 0x7f2a680702c0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f2a58009390 tx=0x7f2a58009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:04:28.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.312+0000 7f2a5fe00700 1 -- 172.21.15.123:0/3671091859 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a5800e040 con 0x7f2a68102e00 2024-03-30T05:04:28.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.312+0000 7f2a5fe00700 1 -- 172.21.15.123:0/3671091859 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2a58013420 con 0x7f2a68102e00 2024-03-30T05:04:28.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.312+0000 7f2a6ec00700 1 -- 172.21.15.123:0/3671091859 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a68071120 con 0x7f2a68102e00 2024-03-30T05:04:28.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.312+0000 7f2a5fe00700 1 -- 172.21.15.123:0/3671091859 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a5801c630 con 0x7f2a68102e00 2024-03-30T05:04:28.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.312+0000 7f2a6ec00700 1 -- 172.21.15.123:0/3671091859 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a6811a250 con 0x7f2a68102e00 2024-03-30T05:04:28.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.313+0000 7f2a6ec00700 1 -- 172.21.15.123:0/3671091859 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a680fe550 con 0x7f2a68102e00 2024-03-30T05:04:28.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.314+0000 7f2a5fe00700 1 -- 172.21.15.123:0/3671091859 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f2a580139e0 con 0x7f2a68102e00 2024-03-30T05:04:28.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.315+0000 7f2a5fe00700 1 --2- 172.21.15.123:0/3671091859 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2a4c06cd00 0x7f2a4c06f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:04:28.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.315+0000 7f2a5fe00700 1 -- 172.21.15.123:0/3671091859 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f2a5808d310 con 0x7f2a68102e00 2024-03-30T05:04:28.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.315+0000 7f2a67400700 1 --2- 172.21.15.123:0/3671091859 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2a4c06cd00 0x7f2a4c06f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:04:28.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.316+0000 7f2a67400700 1 --2- 172.21.15.123:0/3671091859 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2a4c06cd00 0x7f2a4c06f1b0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f2a68071c00 tx=0x7f2a5400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:04:28.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.319+0000 7f2a5fe00700 1 -- 172.21.15.123:0/3671091859 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2a58057350 con 0x7f2a68102e00 2024-03-30T05:04:28.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.605+0000 7f2a6ec00700 1 -- 172.21.15.123:0/3671091859 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f2a680719b0 con 0x7f2a68102e00 2024-03-30T05:04:28.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.605+0000 7f2a5fe00700 1 -- 172.21.15.123:0/3671091859 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f2a5805a970 con 0x7f2a68102e00 2024-03-30T05:04:28.604 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T05:04:28.604 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T05:04:28.604 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T05:04:28.604 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:04:28.604 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T05:04:28.604 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T05:04:28.604 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:04:28.604 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T05:04:28.604 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T05:04:28.604 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:04:28.604 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T05:04:28.605 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 4 2024-03-30T05:04:28.605 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:04:28.605 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T05:04:28.605 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 14 2024-03-30T05:04:28.605 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T05:04:28.605 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T05:04:28.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.608+0000 7f2a6ec00700 1 -- 172.21.15.123:0/3671091859 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2a4c06cd00 msgr2=0x7f2a4c06f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:04:28.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.608+0000 7f2a6ec00700 1 --2- 172.21.15.123:0/3671091859 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2a4c06cd00 0x7f2a4c06f1b0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f2a68071c00 tx=0x7f2a5400b040 comp rx=0 tx=0).stop 2024-03-30T05:04:28.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.608+0000 7f2a6ec00700 1 -- 172.21.15.123:0/3671091859 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2a68102e00 msgr2=0x7f2a680702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:04:28.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.608+0000 7f2a6ec00700 1 --2- 172.21.15.123:0/3671091859 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2a68102e00 0x7f2a680702c0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f2a58009390 tx=0x7f2a58009e30 comp rx=0 tx=0).stop 2024-03-30T05:04:28.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.608+0000 7f2a6ec00700 1 -- 172.21.15.123:0/3671091859 shutdown_connections 2024-03-30T05:04:28.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.608+0000 7f2a6ec00700 1 --2- 172.21.15.123:0/3671091859 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2a4c06cd00 0x7f2a4c06f1b0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:04:28.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.608+0000 7f2a6ec00700 1 --2- 172.21.15.123:0/3671091859 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2a68102e00 0x7f2a680702c0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:04:28.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.608+0000 7f2a6ec00700 1 --2- 172.21.15.123:0/3671091859 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2a681037a0 0x7f2a68070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:04:28.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.608+0000 7f2a6ec00700 1 -- 172.21.15.123:0/3671091859 >> 172.21.15.123:0/3671091859 conn(0x7f2a680f89b0 msgr2=0x7f2a680f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:04:28.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.608+0000 7f2a6ec00700 1 -- 172.21.15.123:0/3671091859 shutdown_connections 2024-03-30T05:04:28.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.608+0000 7f2a6ec00700 1 -- 172.21.15.123:0/3671091859 wait complete. 2024-03-30T05:04:28.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.741+0000 7f30f2000700 1 -- 172.21.15.123:0/2114664514 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30ec070c10 msgr2=0x7f30ec108fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:04:28.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.741+0000 7f30f2000700 1 --2- 172.21.15.123:0/2114664514 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30ec070c10 0x7f30ec108fa0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f30d8004970 tx=0x7f30d8009e30 comp rx=0 tx=0).stop 2024-03-30T05:04:28.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.741+0000 7f30f2000700 1 -- 172.21.15.123:0/2114664514 shutdown_connections 2024-03-30T05:04:28.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.741+0000 7f30f2000700 1 --2- 172.21.15.123:0/2114664514 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30ec070c10 0x7f30ec108fa0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:04:28.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.741+0000 7f30f2000700 1 --2- 172.21.15.123:0/2114664514 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f30ec070300 0x7f30ec0706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:04:28.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.741+0000 7f30f2000700 1 -- 172.21.15.123:0/2114664514 >> 172.21.15.123:0/2114664514 conn(0x7f30ec0f87a0 msgr2=0x7f30ec0fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:04:28.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.741+0000 7f30f2000700 1 -- 172.21.15.123:0/2114664514 shutdown_connections 2024-03-30T05:04:28.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.741+0000 7f30f2000700 1 -- 172.21.15.123:0/2114664514 wait complete. 2024-03-30T05:04:28.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.742+0000 7f30f2000700 1 Processor -- start 2024-03-30T05:04:28.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.742+0000 7f30f2000700 1 -- start start 2024-03-30T05:04:28.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.742+0000 7f30f2000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f30ec070300 0x7f30ec115a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:04:28.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.743+0000 7f30f2000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30ec070c10 0x7f30ec115fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:04:28.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.743+0000 7f30f2000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30ec116610 con 0x7f30ec070c10 2024-03-30T05:04:28.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.743+0000 7f30f2000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30ec10fa00 con 0x7f30ec070300 2024-03-30T05:04:28.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.743+0000 7f30ebe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30ec070c10 0x7f30ec115fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:04:28.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.743+0000 7f30ebe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30ec070c10 0x7f30ec115fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45310/0 (socket says 172.21.15.123:45310) 2024-03-30T05:04:28.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.743+0000 7f30ebe00700 1 -- 172.21.15.123:0/313824880 learned_addr learned my addr 172.21.15.123:0/313824880 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:04:28.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.743+0000 7f30f0c00700 1 --2- 172.21.15.123:0/313824880 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f30ec070300 0x7f30ec115a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:04:28.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.743+0000 7f30ebe00700 1 -- 172.21.15.123:0/313824880 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f30ec070300 msgr2=0x7f30ec115a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:04:28.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.743+0000 7f30ebe00700 1 --2- 172.21.15.123:0/313824880 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f30ec070300 0x7f30ec115a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:04:28.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.743+0000 7f30ebe00700 1 -- 172.21.15.123:0/313824880 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30d8005040 con 0x7f30ec070c10 2024-03-30T05:04:28.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.743+0000 7f30ebe00700 1 --2- 172.21.15.123:0/313824880 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30ec070c10 0x7f30ec115fc0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f30d8004f50 tx=0x7f30d8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:04:28.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.744+0000 7f30e8c00700 1 -- 172.21.15.123:0/313824880 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f30d800e040 con 0x7f30ec070c10 2024-03-30T05:04:28.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.744+0000 7f30e8c00700 1 -- 172.21.15.123:0/313824880 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f30d8013420 con 0x7f30ec070c10 2024-03-30T05:04:28.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.744+0000 7f30f2000700 1 -- 172.21.15.123:0/313824880 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30ec10fc80 con 0x7f30ec070c10 2024-03-30T05:04:28.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.744+0000 7f30e8c00700 1 -- 172.21.15.123:0/313824880 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f30d801c630 con 0x7f30ec070c10 2024-03-30T05:04:28.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.744+0000 7f30f2000700 1 -- 172.21.15.123:0/313824880 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30ec1101c0 con 0x7f30ec070c10 2024-03-30T05:04:28.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.745+0000 7f30f2000700 1 -- 172.21.15.123:0/313824880 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30ec071a40 con 0x7f30ec070c10 2024-03-30T05:04:28.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.746+0000 7f30e8c00700 1 -- 172.21.15.123:0/313824880 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f30d8013590 con 0x7f30ec070c10 2024-03-30T05:04:28.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.747+0000 7f30e8c00700 1 --2- 172.21.15.123:0/313824880 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f30d406cc30 0x7f30d406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:04:28.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.747+0000 7f30e8c00700 1 -- 172.21.15.123:0/313824880 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f30d808d1c0 con 0x7f30ec070c10 2024-03-30T05:04:28.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.747+0000 7f30f0c00700 1 --2- 172.21.15.123:0/313824880 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f30d406cc30 0x7f30d406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:04:28.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.748+0000 7f30f0c00700 1 --2- 172.21.15.123:0/313824880 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f30d406cc30 0x7f30d406f0e0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f30e0004f40 tx=0x7f30e000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:04:28.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.751+0000 7f30e8c00700 1 -- 172.21.15.123:0/313824880 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f30d805ac90 con 0x7f30ec070c10 2024-03-30T05:04:28.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.995+0000 7f30f2000700 1 -- 172.21.15.123:0/313824880 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f30ec10bdf0 con 0x7f30ec070c10 2024-03-30T05:04:28.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.996+0000 7f30e8c00700 1 -- 172.21.15.123:0/313824880 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 34 v34) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f30d805a820 con 0x7f30ec070c10 2024-03-30T05:04:28.994 INFO:teuthology.orchestra.run.smithi123.stdout:e34 2024-03-30T05:04:28.994 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T05:04:28.994 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T05:04:28.994 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T05:04:28.994 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:04:28.995 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T05:04:28.995 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T05:04:28.995 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T05:04:28.995 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T05:04:28.995 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T05:04:28.995 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T05:04:28.995 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T05:04:28.995 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T05:04:28.995 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T05:04:28.995 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T05:04:28.995 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T05:04:28.995 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T05:04:28.995 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T05:04:28.995 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T05:04:28.996 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T05:04:28.996 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T05:04:28.996 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T05:04:28.996 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T05:04:28.996 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T05:04:28.996 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T05:04:28.996 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T05:04:28.996 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T05:04:28.996 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T05:04:28.996 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T05:04:28.996 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T05:04:28.996 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T05:04:28.996 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T05:04:28.996 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T05:04:28.997 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:04:28.997 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:04:28.997 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T05:04:28.997 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:04:28.997 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:04:28.997 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:04:28.997 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:04:28.997 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44975} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/3788988014,v1:172.21.15.135:6827/3788988014] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:04:28.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.998+0000 7f30f2000700 1 -- 172.21.15.123:0/313824880 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f30d406cc30 msgr2=0x7f30d406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:04:28.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.998+0000 7f30f2000700 1 --2- 172.21.15.123:0/313824880 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f30d406cc30 0x7f30d406f0e0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f30e0004f40 tx=0x7f30e000b040 comp rx=0 tx=0).stop 2024-03-30T05:04:28.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.998+0000 7f30f2000700 1 -- 172.21.15.123:0/313824880 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30ec070c10 msgr2=0x7f30ec115fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:04:28.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.998+0000 7f30f2000700 1 --2- 172.21.15.123:0/313824880 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30ec070c10 0x7f30ec115fc0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f30d8004f50 tx=0x7f30d8009e30 comp rx=0 tx=0).stop 2024-03-30T05:04:28.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.998+0000 7f30f2000700 1 -- 172.21.15.123:0/313824880 shutdown_connections 2024-03-30T05:04:28.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.998+0000 7f30f2000700 1 --2- 172.21.15.123:0/313824880 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f30d406cc30 0x7f30d406f0e0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:04:28.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.998+0000 7f30f2000700 1 --2- 172.21.15.123:0/313824880 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f30ec070300 0x7f30ec115a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:04:28.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.998+0000 7f30f2000700 1 --2- 172.21.15.123:0/313824880 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30ec070c10 0x7f30ec115fc0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:04:28.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.998+0000 7f30f2000700 1 -- 172.21.15.123:0/313824880 >> 172.21.15.123:0/313824880 conn(0x7f30ec0f87a0 msgr2=0x7f30ec0f9cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:04:28.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.998+0000 7f30f2000700 1 -- 172.21.15.123:0/313824880 shutdown_connections 2024-03-30T05:04:28.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:28.999+0000 7f30f2000700 1 -- 172.21.15.123:0/313824880 wait complete. 2024-03-30T05:04:28.999 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 34 2024-03-30T05:04:29.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.132+0000 7f90d8600700 1 -- 172.21.15.123:0/3070649249 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90d0102e00 msgr2=0x7f90d01031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:04:29.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.132+0000 7f90d8600700 1 --2- 172.21.15.123:0/3070649249 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90d0102e00 0x7f90d01031d0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f90c4004940 tx=0x7f90c4009e30 comp rx=0 tx=0).stop 2024-03-30T05:04:29.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.132+0000 7f90d8600700 1 -- 172.21.15.123:0/3070649249 shutdown_connections 2024-03-30T05:04:29.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.132+0000 7f90d8600700 1 --2- 172.21.15.123:0/3070649249 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f90d01037a0 0x7f90d00fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:04:29.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.132+0000 7f90d8600700 1 --2- 172.21.15.123:0/3070649249 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90d0102e00 0x7f90d01031d0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:04:29.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.132+0000 7f90d8600700 1 -- 172.21.15.123:0/3070649249 >> 172.21.15.123:0/3070649249 conn(0x7f90d00f89b0 msgr2=0x7f90d00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:04:29.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.132+0000 7f90d8600700 1 -- 172.21.15.123:0/3070649249 shutdown_connections 2024-03-30T05:04:29.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.132+0000 7f90d8600700 1 -- 172.21.15.123:0/3070649249 wait complete. 2024-03-30T05:04:29.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.133+0000 7f90d8600700 1 Processor -- start 2024-03-30T05:04:29.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.133+0000 7f90d8600700 1 -- start start 2024-03-30T05:04:29.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.133+0000 7f90d8600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90d0102e00 0x7f90d00702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:04:29.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.133+0000 7f90d8600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f90d01037a0 0x7f90d0070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:04:29.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.133+0000 7f90d8600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d0070d90 con 0x7f90d0102e00 2024-03-30T05:04:29.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.133+0000 7f90d8600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d0070ed0 con 0x7f90d01037a0 2024-03-30T05:04:29.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.134+0000 7f90d5c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90d0102e00 0x7f90d00702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:04:29.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.134+0000 7f90d5c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90d0102e00 0x7f90d00702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45328/0 (socket says 172.21.15.123:45328) 2024-03-30T05:04:29.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.134+0000 7f90d5c00700 1 -- 172.21.15.123:0/1269254125 learned_addr learned my addr 172.21.15.123:0/1269254125 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:04:29.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.134+0000 7f90d5c00700 1 -- 172.21.15.123:0/1269254125 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f90d01037a0 msgr2=0x7f90d0070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:04:29.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.134+0000 7f90d5c00700 1 --2- 172.21.15.123:0/1269254125 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f90d01037a0 0x7f90d0070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:04:29.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.134+0000 7f90d5c00700 1 -- 172.21.15.123:0/1269254125 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90c4005040 con 0x7f90d0102e00 2024-03-30T05:04:29.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.134+0000 7f90d5c00700 1 --2- 172.21.15.123:0/1269254125 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90d0102e00 0x7f90d00702c0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f90c4009390 tx=0x7f90c4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:04:29.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.134+0000 7f90cd600700 1 -- 172.21.15.123:0/1269254125 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90c400e040 con 0x7f90d0102e00 2024-03-30T05:04:29.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.135+0000 7f90cd600700 1 -- 172.21.15.123:0/1269254125 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f90c4013420 con 0x7f90d0102e00 2024-03-30T05:04:29.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.135+0000 7f90cd600700 1 -- 172.21.15.123:0/1269254125 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90c401c630 con 0x7f90d0102e00 2024-03-30T05:04:29.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.135+0000 7f90d8600700 1 -- 172.21.15.123:0/1269254125 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90d0071120 con 0x7f90d0102e00 2024-03-30T05:04:29.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.135+0000 7f90d8600700 1 -- 172.21.15.123:0/1269254125 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90d011a270 con 0x7f90d0102e00 2024-03-30T05:04:29.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.136+0000 7f90d8600700 1 -- 172.21.15.123:0/1269254125 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90d00fe550 con 0x7f90d0102e00 2024-03-30T05:04:29.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.137+0000 7f90cd600700 1 -- 172.21.15.123:0/1269254125 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f90c40139e0 con 0x7f90d0102e00 2024-03-30T05:04:29.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.138+0000 7f90cd600700 1 --2- 172.21.15.123:0/1269254125 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f90b806cc30 0x7f90b806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:04:29.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.138+0000 7f90cd600700 1 -- 172.21.15.123:0/1269254125 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f90c408d260 con 0x7f90d0102e00 2024-03-30T05:04:29.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.138+0000 7f90d5200700 1 --2- 172.21.15.123:0/1269254125 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f90b806cc30 0x7f90b806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:04:29.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.138+0000 7f90d5200700 1 --2- 172.21.15.123:0/1269254125 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f90b806cc30 0x7f90b806f0e0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f90d0071c00 tx=0x7f90bc007d40 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:04:29.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.142+0000 7f90cd600700 1 -- 172.21.15.123:0/1269254125 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f90c40572a0 con 0x7f90d0102e00 2024-03-30T05:04:29.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:04:28 smithi135 ceph-mon[106011]: from='client.35804 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:04:29.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:04:28 smithi135 ceph-mon[106011]: pgmap v999: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:29.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:04:28 smithi135 ceph-mon[106011]: from='client.35808 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:04:29.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:04:28 smithi135 ceph-mon[106011]: from='client.35812 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:04:29.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:04:28 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3671091859' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:04:29.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.363+0000 7f90d8600700 1 -- 172.21.15.123:0/1269254125 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f90d004bed0 con 0x7f90b806cc30 2024-03-30T05:04:29.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.364+0000 7f90cd600700 1 -- 172.21.15.123:0/1269254125 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f90d004bed0 con 0x7f90b806cc30 2024-03-30T05:04:29.362 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T05:04:29.362 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T05:04:29.362 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T05:04:29.362 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T05:04:29.363 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T05:04:29.363 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T05:04:29.363 INFO:teuthology.orchestra.run.smithi123.stdout: "mds", 2024-03-30T05:04:29.363 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T05:04:29.363 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T05:04:29.363 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T05:04:29.363 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T05:04:29.363 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "16/21 daemons upgraded", 2024-03-30T05:04:29.363 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T05:04:29.363 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T05:04:29.363 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T05:04:29.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.366+0000 7f90d8600700 1 -- 172.21.15.123:0/1269254125 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f90b806cc30 msgr2=0x7f90b806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:04:29.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.366+0000 7f90d8600700 1 --2- 172.21.15.123:0/1269254125 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f90b806cc30 0x7f90b806f0e0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f90d0071c00 tx=0x7f90bc007d40 comp rx=0 tx=0).stop 2024-03-30T05:04:29.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.366+0000 7f90d8600700 1 -- 172.21.15.123:0/1269254125 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90d0102e00 msgr2=0x7f90d00702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:04:29.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.366+0000 7f90d8600700 1 --2- 172.21.15.123:0/1269254125 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90d0102e00 0x7f90d00702c0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f90c4009390 tx=0x7f90c4009e30 comp rx=0 tx=0).stop 2024-03-30T05:04:29.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.366+0000 7f90d8600700 1 -- 172.21.15.123:0/1269254125 shutdown_connections 2024-03-30T05:04:29.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.366+0000 7f90d8600700 1 --2- 172.21.15.123:0/1269254125 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f90b806cc30 0x7f90b806f0e0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:04:29.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.366+0000 7f90d8600700 1 --2- 172.21.15.123:0/1269254125 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90d0102e00 0x7f90d00702c0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:04:29.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.366+0000 7f90d8600700 1 --2- 172.21.15.123:0/1269254125 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f90d01037a0 0x7f90d0070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:04:29.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.366+0000 7f90d8600700 1 -- 172.21.15.123:0/1269254125 >> 172.21.15.123:0/1269254125 conn(0x7f90d00f89b0 msgr2=0x7f90d00f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:04:29.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.367+0000 7f90d8600700 1 -- 172.21.15.123:0/1269254125 shutdown_connections 2024-03-30T05:04:29.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.367+0000 7f90d8600700 1 -- 172.21.15.123:0/1269254125 wait complete. 2024-03-30T05:04:29.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:04:28 smithi123 ceph-mon[139369]: from='client.35804 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:04:29.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:04:28 smithi123 ceph-mon[139369]: pgmap v999: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:29.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:04:28 smithi123 ceph-mon[139369]: from='client.35808 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:04:29.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:04:28 smithi123 ceph-mon[139369]: from='client.35812 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:04:29.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:04:28 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3671091859' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:04:29.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.501+0000 7fc118200700 1 -- 172.21.15.123:0/2863315238 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc1101037a0 msgr2=0x7fc1100fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:04:29.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.501+0000 7fc118200700 1 --2- 172.21.15.123:0/2863315238 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc1101037a0 0x7fc1100fce10 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc004940 tx=0x7fc0fc009e30 comp rx=0 tx=0).stop 2024-03-30T05:04:29.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.501+0000 7fc118200700 1 -- 172.21.15.123:0/2863315238 shutdown_connections 2024-03-30T05:04:29.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.501+0000 7fc118200700 1 --2- 172.21.15.123:0/2863315238 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc1101037a0 0x7fc1100fce10 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:04:29.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.501+0000 7fc118200700 1 --2- 172.21.15.123:0/2863315238 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc110102e00 0x7fc1101031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:04:29.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.501+0000 7fc118200700 1 -- 172.21.15.123:0/2863315238 >> 172.21.15.123:0/2863315238 conn(0x7fc1100f89b0 msgr2=0x7fc1100fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:04:29.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.501+0000 7fc118200700 1 -- 172.21.15.123:0/2863315238 shutdown_connections 2024-03-30T05:04:29.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.501+0000 7fc118200700 1 -- 172.21.15.123:0/2863315238 wait complete. 2024-03-30T05:04:29.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.502+0000 7fc118200700 1 Processor -- start 2024-03-30T05:04:29.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.503+0000 7fc118200700 1 -- start start 2024-03-30T05:04:29.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.503+0000 7fc118200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc110102e00 0x7fc110113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:04:29.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.503+0000 7fc118200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc1101037a0 0x7fc110114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:04:29.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.503+0000 7fc118200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc110114690 con 0x7fc110102e00 2024-03-30T05:04:29.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.503+0000 7fc118200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc11010da80 con 0x7fc1101037a0 2024-03-30T05:04:29.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.503+0000 7fc115800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc110102e00 0x7fc110113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:04:29.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.504+0000 7fc115800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc110102e00 0x7fc110113b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45350/0 (socket says 172.21.15.123:45350) 2024-03-30T05:04:29.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.504+0000 7fc115800700 1 -- 172.21.15.123:0/3528916297 learned_addr learned my addr 172.21.15.123:0/3528916297 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:04:29.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.504+0000 7fc114e00700 1 --2- 172.21.15.123:0/3528916297 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc1101037a0 0x7fc110114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:04:29.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.504+0000 7fc115800700 1 -- 172.21.15.123:0/3528916297 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc1101037a0 msgr2=0x7fc110114040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:04:29.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.504+0000 7fc115800700 1 --2- 172.21.15.123:0/3528916297 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc1101037a0 0x7fc110114040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:04:29.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.504+0000 7fc115800700 1 -- 172.21.15.123:0/3528916297 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0fc005040 con 0x7fc110102e00 2024-03-30T05:04:29.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.504+0000 7fc115800700 1 --2- 172.21.15.123:0/3528916297 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc110102e00 0x7fc110113b00 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fc10400c8c0 tx=0x7fc1040044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:04:29.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.504+0000 7fc10d600700 1 -- 172.21.15.123:0/3528916297 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc10400a760 con 0x7fc110102e00 2024-03-30T05:04:29.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.504+0000 7fc10d600700 1 -- 172.21.15.123:0/3528916297 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc10400ad70 con 0x7fc110102e00 2024-03-30T05:04:29.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.504+0000 7fc118200700 1 -- 172.21.15.123:0/3528916297 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc11010dd60 con 0x7fc110102e00 2024-03-30T05:04:29.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.504+0000 7fc10d600700 1 -- 172.21.15.123:0/3528916297 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc1040113a0 con 0x7fc110102e00 2024-03-30T05:04:29.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.505+0000 7fc118200700 1 -- 172.21.15.123:0/3528916297 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc11010e2f0 con 0x7fc110102e00 2024-03-30T05:04:29.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.506+0000 7fc118200700 1 -- 172.21.15.123:0/3528916297 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1100fe550 con 0x7fc110102e00 2024-03-30T05:04:29.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.507+0000 7fc10d600700 1 -- 172.21.15.123:0/3528916297 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fc104012400 con 0x7fc110102e00 2024-03-30T05:04:29.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.508+0000 7fc10d600700 1 --2- 172.21.15.123:0/3528916297 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc0f806cc80 0x7fc0f806f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:04:29.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.508+0000 7fc10d600700 1 -- 172.21.15.123:0/3528916297 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc10408c100 con 0x7fc110102e00 2024-03-30T05:04:29.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.508+0000 7fc114e00700 1 --2- 172.21.15.123:0/3528916297 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc0f806cc80 0x7fc0f806f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:04:29.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.509+0000 7fc114e00700 1 --2- 172.21.15.123:0/3528916297 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc0f806cc80 0x7fc0f806f130 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fc11010ed60 tx=0x7fc0fc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:04:29.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.512+0000 7fc10d600700 1 -- 172.21.15.123:0/3528916297 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc104059b50 con 0x7fc110102e00 2024-03-30T05:04:29.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.795+0000 7fc118200700 1 -- 172.21.15.123:0/3528916297 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc110109da0 con 0x7fc110102e00 2024-03-30T05:04:29.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.796+0000 7fc10d600700 1 -- 172.21.15.123:0/3528916297 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7fc1040596e0 con 0x7fc110102e00 2024-03-30T05:04:29.794 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T05:04:29.794 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-30T05:04:29.794 INFO:teuthology.orchestra.run.smithi123.stdout: host smithi135 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi135/config 2024-03-30T05:04:29.795 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-30T05:04:29.795 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T05:04:29.795 INFO:teuthology.orchestra.run.smithi123.stdout:Traceback (most recent call last): 2024-03-30T05:04:29.795 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-30T05:04:29.795 INFO:teuthology.orchestra.run.smithi123.stdout: "__main__", mod_spec) 2024-03-30T05:04:29.795 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-30T05:04:29.795 INFO:teuthology.orchestra.run.smithi123.stdout: exec(code, run_globals) 2024-03-30T05:04:29.795 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10889, in 2024-03-30T05:04:29.795 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10877, in main 2024-03-30T05:04:29.795 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-30T05:04:29.795 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-30T05:04:29.795 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-30T05:04:29.796 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-30T05:04:29.796 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-03-30T05:04:29.796 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-30T05:04:29.796 INFO:teuthology.orchestra.run.smithi123.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T05:04:29.796 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T05:04:29.796 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T05:04:29.796 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T05:04:29.796 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T05:04:29.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.798+0000 7fc118200700 1 -- 172.21.15.123:0/3528916297 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc0f806cc80 msgr2=0x7fc0f806f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:04:29.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.798+0000 7fc118200700 1 --2- 172.21.15.123:0/3528916297 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc0f806cc80 0x7fc0f806f130 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fc11010ed60 tx=0x7fc0fc009e30 comp rx=0 tx=0).stop 2024-03-30T05:04:29.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.798+0000 7fc118200700 1 -- 172.21.15.123:0/3528916297 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc110102e00 msgr2=0x7fc110113b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:04:29.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.798+0000 7fc118200700 1 --2- 172.21.15.123:0/3528916297 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc110102e00 0x7fc110113b00 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fc10400c8c0 tx=0x7fc1040044c0 comp rx=0 tx=0).stop 2024-03-30T05:04:29.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.798+0000 7fc118200700 1 -- 172.21.15.123:0/3528916297 shutdown_connections 2024-03-30T05:04:29.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.798+0000 7fc118200700 1 --2- 172.21.15.123:0/3528916297 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc0f806cc80 0x7fc0f806f130 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:04:29.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.798+0000 7fc118200700 1 --2- 172.21.15.123:0/3528916297 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc110102e00 0x7fc110113b00 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:04:29.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.798+0000 7fc118200700 1 --2- 172.21.15.123:0/3528916297 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc1101037a0 0x7fc110114040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:04:29.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.798+0000 7fc118200700 1 -- 172.21.15.123:0/3528916297 >> 172.21.15.123:0/3528916297 conn(0x7fc1100f89b0 msgr2=0x7fc1100f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:04:29.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.798+0000 7fc118200700 1 -- 172.21.15.123:0/3528916297 shutdown_connections 2024-03-30T05:04:29.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:29.799+0000 7fc118200700 1 -- 172.21.15.123:0/3528916297 wait complete. 2024-03-30T05:04:30.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:04:29 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/313824880' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T05:04:30.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:04:29 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:04:30.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:04:29 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3528916297' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T05:04:30.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:04:29 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/313824880' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T05:04:30.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:04:29 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:04:30.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:04:29 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3528916297' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T05:04:31.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:04:30 smithi135 ceph-mon[106011]: from='client.35824 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:04:31.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:04:30 smithi135 ceph-mon[106011]: pgmap v1000: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:31.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:04:30 smithi123 ceph-mon[139369]: from='client.35824 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:04:31.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:04:30 smithi123 ceph-mon[139369]: pgmap v1000: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:33.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:04:32 smithi135 ceph-mon[106011]: pgmap v1001: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:33.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:04:32 smithi123 ceph-mon[139369]: pgmap v1001: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:35.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:04:34 smithi135 ceph-mon[106011]: pgmap v1002: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:35.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:04:34 smithi123 ceph-mon[139369]: pgmap v1002: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:37.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:04:37 smithi135 ceph-mon[106011]: pgmap v1003: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:37.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:04:37 smithi123 ceph-mon[139369]: pgmap v1003: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:39.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:04:39 smithi135 ceph-mon[106011]: pgmap v1004: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:39.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:04:39 smithi123 ceph-mon[139369]: pgmap v1004: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:41.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:04:41 smithi135 ceph-mon[106011]: pgmap v1005: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:41.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:04:41 smithi123 ceph-mon[139369]: pgmap v1005: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:43.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:04:43 smithi135 ceph-mon[106011]: pgmap v1006: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:43.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:04:43 smithi123 ceph-mon[139369]: pgmap v1006: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:45.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:04:45 smithi135 ceph-mon[106011]: pgmap v1007: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:45.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:04:45 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:04:45.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:04:45 smithi123 ceph-mon[139369]: pgmap v1007: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:45.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:04:45 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:04:47.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:04:47 smithi135 ceph-mon[106011]: pgmap v1008: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:47.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:04:47 smithi123 ceph-mon[139369]: pgmap v1008: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:49.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:04:49 smithi135 ceph-mon[106011]: pgmap v1009: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:49.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:04:49 smithi123 ceph-mon[139369]: pgmap v1009: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:51.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:04:51 smithi135 ceph-mon[106011]: pgmap v1010: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:51.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:04:51 smithi123 ceph-mon[139369]: pgmap v1010: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:53.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:04:53 smithi135 ceph-mon[106011]: pgmap v1011: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:53.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:04:53 smithi123 ceph-mon[139369]: pgmap v1011: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:55.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:04:55 smithi135 ceph-mon[106011]: pgmap v1012: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:55.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:04:55 smithi123 ceph-mon[139369]: pgmap v1012: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:57.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:04:57 smithi135 ceph-mon[106011]: pgmap v1013: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:57.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:04:57 smithi123 ceph-mon[139369]: pgmap v1013: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:59.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:04:59 smithi135 ceph-mon[106011]: pgmap v1014: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:59.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:04:59 smithi123 ceph-mon[139369]: pgmap v1014: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:04:59.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:59.946+0000 7f4852e00700 1 -- 172.21.15.123:0/1311245423 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f484c070360 msgr2=0x7f484c070730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:04:59.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:59.946+0000 7f4852e00700 1 --2- 172.21.15.123:0/1311245423 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f484c070360 0x7f484c070730 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f483c004970 tx=0x7f483c009e30 comp rx=0 tx=0).stop 2024-03-30T05:04:59.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:59.946+0000 7f4852e00700 1 -- 172.21.15.123:0/1311245423 shutdown_connections 2024-03-30T05:04:59.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:59.946+0000 7f4852e00700 1 --2- 172.21.15.123:0/1311245423 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f484c070c70 0x7f484c109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:04:59.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:59.946+0000 7f4852e00700 1 --2- 172.21.15.123:0/1311245423 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f484c070360 0x7f484c070730 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:04:59.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:59.946+0000 7f4852e00700 1 -- 172.21.15.123:0/1311245423 >> 172.21.15.123:0/1311245423 conn(0x7f484c0f89d0 msgr2=0x7f484c0fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:04:59.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:59.946+0000 7f4852e00700 1 -- 172.21.15.123:0/1311245423 shutdown_connections 2024-03-30T05:04:59.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:59.946+0000 7f4852e00700 1 -- 172.21.15.123:0/1311245423 wait complete. 2024-03-30T05:04:59.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:59.947+0000 7f4852e00700 1 Processor -- start 2024-03-30T05:04:59.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:59.947+0000 7f4852e00700 1 -- start start 2024-03-30T05:04:59.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:59.948+0000 7f4852e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f484c070360 0x7f484c115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:04:59.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:59.948+0000 7f4852e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f484c070c70 0x7f484c1161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:04:59.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:59.948+0000 7f4852e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484c116800 con 0x7f484c070360 2024-03-30T05:04:59.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:59.948+0000 7f4852e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484c10fbd0 con 0x7f484c070c70 2024-03-30T05:04:59.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:59.948+0000 7f4843e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f484c070360 0x7f484c115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:04:59.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:59.948+0000 7f4843e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f484c070360 0x7f484c115c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45288/0 (socket says 172.21.15.123:45288) 2024-03-30T05:04:59.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:59.948+0000 7f4843e00700 1 -- 172.21.15.123:0/2171146969 learned_addr learned my addr 172.21.15.123:0/2171146969 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:04:59.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:59.948+0000 7f484be00700 1 --2- 172.21.15.123:0/2171146969 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f484c070c70 0x7f484c1161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:04:59.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:59.949+0000 7f4843e00700 1 -- 172.21.15.123:0/2171146969 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f484c070c70 msgr2=0x7f484c1161b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:04:59.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:59.949+0000 7f4843e00700 1 --2- 172.21.15.123:0/2171146969 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f484c070c70 0x7f484c1161b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:04:59.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:59.949+0000 7f4843e00700 1 -- 172.21.15.123:0/2171146969 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f483c005040 con 0x7f484c070360 2024-03-30T05:04:59.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:59.949+0000 7f4843e00700 1 --2- 172.21.15.123:0/2171146969 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f484c070360 0x7f484c115c70 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f483c004f50 tx=0x7f483c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:04:59.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:59.949+0000 7f4848c00700 1 -- 172.21.15.123:0/2171146969 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f483c00e040 con 0x7f484c070360 2024-03-30T05:04:59.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:59.949+0000 7f4848c00700 1 -- 172.21.15.123:0/2171146969 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f483c013420 con 0x7f484c070360 2024-03-30T05:04:59.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:59.949+0000 7f4852e00700 1 -- 172.21.15.123:0/2171146969 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f484c10fe50 con 0x7f484c070360 2024-03-30T05:04:59.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:59.949+0000 7f4848c00700 1 -- 172.21.15.123:0/2171146969 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f483c01c630 con 0x7f484c070360 2024-03-30T05:04:59.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:59.949+0000 7f4852e00700 1 -- 172.21.15.123:0/2171146969 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f484c110390 con 0x7f484c070360 2024-03-30T05:04:59.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:59.950+0000 7f4852e00700 1 -- 172.21.15.123:0/2171146969 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f484c071aa0 con 0x7f484c070360 2024-03-30T05:04:59.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:59.951+0000 7f4848c00700 1 -- 172.21.15.123:0/2171146969 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f483c013590 con 0x7f484c070360 2024-03-30T05:04:59.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:59.952+0000 7f4848c00700 1 --2- 172.21.15.123:0/2171146969 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f483006cc30 0x7f483006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:04:59.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:59.952+0000 7f4848c00700 1 -- 172.21.15.123:0/2171146969 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f483c08d1c0 con 0x7f484c070360 2024-03-30T05:04:59.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:59.952+0000 7f484be00700 1 --2- 172.21.15.123:0/2171146969 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f483006cc30 0x7f483006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:04:59.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:59.953+0000 7f484be00700 1 --2- 172.21.15.123:0/2171146969 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f483006cc30 0x7f483006f0e0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f484c111510 tx=0x7f483400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:04:59.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:04:59.957+0000 7f4848c00700 1 -- 172.21.15.123:0/2171146969 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f483c05ac90 con 0x7f484c070360 2024-03-30T05:05:00.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.184+0000 7f4852e00700 1 -- 172.21.15.123:0/2171146969 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f484c04bf70 con 0x7f483006cc30 2024-03-30T05:05:00.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.186+0000 7f4848c00700 1 -- 172.21.15.123:0/2171146969 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f483400f040 con 0x7f483006cc30 2024-03-30T05:05:00.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.188+0000 7f4852e00700 1 -- 172.21.15.123:0/2171146969 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f483006cc30 msgr2=0x7f483006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:00.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.188+0000 7f4852e00700 1 --2- 172.21.15.123:0/2171146969 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f483006cc30 0x7f483006f0e0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f484c111510 tx=0x7f483400b040 comp rx=0 tx=0).stop 2024-03-30T05:05:00.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.188+0000 7f4852e00700 1 -- 172.21.15.123:0/2171146969 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f484c070360 msgr2=0x7f484c115c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:00.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.188+0000 7f4852e00700 1 --2- 172.21.15.123:0/2171146969 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f484c070360 0x7f484c115c70 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f483c004f50 tx=0x7f483c009e30 comp rx=0 tx=0).stop 2024-03-30T05:05:00.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.188+0000 7f4852e00700 1 -- 172.21.15.123:0/2171146969 shutdown_connections 2024-03-30T05:05:00.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.188+0000 7f4852e00700 1 --2- 172.21.15.123:0/2171146969 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f483006cc30 0x7f483006f0e0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:00.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.188+0000 7f4852e00700 1 --2- 172.21.15.123:0/2171146969 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f484c070360 0x7f484c115c70 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:00.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.188+0000 7f4852e00700 1 --2- 172.21.15.123:0/2171146969 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f484c070c70 0x7f484c1161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:00.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.188+0000 7f4852e00700 1 -- 172.21.15.123:0/2171146969 >> 172.21.15.123:0/2171146969 conn(0x7f484c0f89d0 msgr2=0x7f484c0f9f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:05:00.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.188+0000 7f4852e00700 1 -- 172.21.15.123:0/2171146969 shutdown_connections 2024-03-30T05:05:00.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.189+0000 7f4852e00700 1 -- 172.21.15.123:0/2171146969 wait complete. 2024-03-30T05:05:00.201 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T05:05:00.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.322+0000 7f5b69000700 1 -- 172.21.15.123:0/594257919 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b64102c40 msgr2=0x7f5b64103010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:00.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.322+0000 7f5b69000700 1 --2- 172.21.15.123:0/594257919 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b64102c40 0x7f5b64103010 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f5b54004940 tx=0x7f5b54009e30 comp rx=0 tx=0).stop 2024-03-30T05:05:00.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.322+0000 7f5b69000700 1 -- 172.21.15.123:0/594257919 shutdown_connections 2024-03-30T05:05:00.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.322+0000 7f5b69000700 1 --2- 172.21.15.123:0/594257919 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5b641035e0 0x7f5b640fcc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:00.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.322+0000 7f5b69000700 1 --2- 172.21.15.123:0/594257919 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b64102c40 0x7f5b64103010 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:00.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.322+0000 7f5b69000700 1 -- 172.21.15.123:0/594257919 >> 172.21.15.123:0/594257919 conn(0x7f5b640f8830 msgr2=0x7f5b640fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:05:00.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.322+0000 7f5b69000700 1 -- 172.21.15.123:0/594257919 shutdown_connections 2024-03-30T05:05:00.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.322+0000 7f5b69000700 1 -- 172.21.15.123:0/594257919 wait complete. 2024-03-30T05:05:00.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.323+0000 7f5b69000700 1 Processor -- start 2024-03-30T05:05:00.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.323+0000 7f5b69000700 1 -- start start 2024-03-30T05:05:00.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.324+0000 7f5b69000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b64102c40 0x7f5b64113950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:05:00.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.324+0000 7f5b69000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5b641035e0 0x7f5b64113e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:05:00.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.324+0000 7f5b69000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b641144e0 con 0x7f5b64102c40 2024-03-30T05:05:00.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.324+0000 7f5b69000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6410d8d0 con 0x7f5b641035e0 2024-03-30T05:05:00.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.324+0000 7f5b63400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b64102c40 0x7f5b64113950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:05:00.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.324+0000 7f5b63400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b64102c40 0x7f5b64113950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45304/0 (socket says 172.21.15.123:45304) 2024-03-30T05:05:00.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.324+0000 7f5b63400700 1 -- 172.21.15.123:0/3285812704 learned_addr learned my addr 172.21.15.123:0/3285812704 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:05:00.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.324+0000 7f5b62a00700 1 --2- 172.21.15.123:0/3285812704 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5b641035e0 0x7f5b64113e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:05:00.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.324+0000 7f5b63400700 1 -- 172.21.15.123:0/3285812704 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5b641035e0 msgr2=0x7f5b64113e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:00.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.324+0000 7f5b63400700 1 --2- 172.21.15.123:0/3285812704 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5b641035e0 0x7f5b64113e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:00.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.324+0000 7f5b63400700 1 -- 172.21.15.123:0/3285812704 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b54005040 con 0x7f5b64102c40 2024-03-30T05:05:00.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.325+0000 7f5b63400700 1 --2- 172.21.15.123:0/3285812704 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b64102c40 0x7f5b64113950 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f5b5400b3b0 tx=0x7f5b54009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:05:00.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.325+0000 7f5b5b400700 1 -- 172.21.15.123:0/3285812704 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b5400e040 con 0x7f5b64102c40 2024-03-30T05:05:00.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.325+0000 7f5b5b400700 1 -- 172.21.15.123:0/3285812704 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5b54013420 con 0x7f5b64102c40 2024-03-30T05:05:00.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.325+0000 7f5b69000700 1 -- 172.21.15.123:0/3285812704 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b6410db50 con 0x7f5b64102c40 2024-03-30T05:05:00.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.325+0000 7f5b5b400700 1 -- 172.21.15.123:0/3285812704 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b5401c610 con 0x7f5b64102c40 2024-03-30T05:05:00.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.325+0000 7f5b69000700 1 -- 172.21.15.123:0/3285812704 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b6410e110 con 0x7f5b64102c40 2024-03-30T05:05:00.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.326+0000 7f5b69000700 1 -- 172.21.15.123:0/3285812704 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b640fe390 con 0x7f5b64102c40 2024-03-30T05:05:00.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.327+0000 7f5b5b400700 1 -- 172.21.15.123:0/3285812704 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f5b54013a00 con 0x7f5b64102c40 2024-03-30T05:05:00.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.328+0000 7f5b5b400700 1 --2- 172.21.15.123:0/3285812704 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5b4806cc20 0x7f5b4806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:05:00.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.328+0000 7f5b5b400700 1 -- 172.21.15.123:0/3285812704 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5b5408d140 con 0x7f5b64102c40 2024-03-30T05:05:00.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.328+0000 7f5b62a00700 1 --2- 172.21.15.123:0/3285812704 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5b4806cc20 0x7f5b4806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:05:00.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.328+0000 7f5b62a00700 1 --2- 172.21.15.123:0/3285812704 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5b4806cc20 0x7f5b4806f0d0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f5b6410ea90 tx=0x7f5b5000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:05:00.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.332+0000 7f5b5b400700 1 -- 172.21.15.123:0/3285812704 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5b5405ac10 con 0x7f5b64102c40 2024-03-30T05:05:00.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:05:00 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:05:00.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:05:00 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:05:00.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.552+0000 7f5b69000700 1 -- 172.21.15.123:0/3285812704 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5b64109c40 con 0x7f5b4806cc20 2024-03-30T05:05:00.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.553+0000 7f5b5b400700 1 -- 172.21.15.123:0/3285812704 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f5b64109c40 con 0x7f5b4806cc20 2024-03-30T05:05:00.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.555+0000 7f5b69000700 1 -- 172.21.15.123:0/3285812704 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5b4806cc20 msgr2=0x7f5b4806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:00.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.555+0000 7f5b69000700 1 --2- 172.21.15.123:0/3285812704 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5b4806cc20 0x7f5b4806f0d0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f5b6410ea90 tx=0x7f5b5000b040 comp rx=0 tx=0).stop 2024-03-30T05:05:00.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.555+0000 7f5b69000700 1 -- 172.21.15.123:0/3285812704 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b64102c40 msgr2=0x7f5b64113950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:00.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.555+0000 7f5b69000700 1 --2- 172.21.15.123:0/3285812704 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b64102c40 0x7f5b64113950 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f5b5400b3b0 tx=0x7f5b54009e30 comp rx=0 tx=0).stop 2024-03-30T05:05:00.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.555+0000 7f5b69000700 1 -- 172.21.15.123:0/3285812704 shutdown_connections 2024-03-30T05:05:00.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.555+0000 7f5b69000700 1 --2- 172.21.15.123:0/3285812704 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5b4806cc20 0x7f5b4806f0d0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:00.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.555+0000 7f5b69000700 1 --2- 172.21.15.123:0/3285812704 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b64102c40 0x7f5b64113950 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:00.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.555+0000 7f5b69000700 1 --2- 172.21.15.123:0/3285812704 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5b641035e0 0x7f5b64113e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:00.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.555+0000 7f5b69000700 1 -- 172.21.15.123:0/3285812704 >> 172.21.15.123:0/3285812704 conn(0x7f5b640f8830 msgr2=0x7f5b640f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:05:00.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.556+0000 7f5b69000700 1 -- 172.21.15.123:0/3285812704 shutdown_connections 2024-03-30T05:05:00.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.556+0000 7f5b69000700 1 -- 172.21.15.123:0/3285812704 wait complete. 2024-03-30T05:05:00.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.690+0000 7fe8b9600700 1 -- 172.21.15.123:0/4164274072 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe8b4102e20 msgr2=0x7fe8b41031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:00.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.690+0000 7fe8b9600700 1 --2- 172.21.15.123:0/4164274072 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe8b4102e20 0x7fe8b41031f0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fe8a4004940 tx=0x7fe8a4009e30 comp rx=0 tx=0).stop 2024-03-30T05:05:00.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.690+0000 7fe8b9600700 1 -- 172.21.15.123:0/4164274072 shutdown_connections 2024-03-30T05:05:00.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.690+0000 7fe8b9600700 1 --2- 172.21.15.123:0/4164274072 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe8b41037c0 0x7fe8b40fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:00.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.690+0000 7fe8b9600700 1 --2- 172.21.15.123:0/4164274072 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe8b4102e20 0x7fe8b41031f0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:00.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.690+0000 7fe8b9600700 1 -- 172.21.15.123:0/4164274072 >> 172.21.15.123:0/4164274072 conn(0x7fe8b40f89d0 msgr2=0x7fe8b40fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:05:00.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.690+0000 7fe8b9600700 1 -- 172.21.15.123:0/4164274072 shutdown_connections 2024-03-30T05:05:00.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.690+0000 7fe8b9600700 1 -- 172.21.15.123:0/4164274072 wait complete. 2024-03-30T05:05:00.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.691+0000 7fe8b9600700 1 Processor -- start 2024-03-30T05:05:00.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.691+0000 7fe8b9600700 1 -- start start 2024-03-30T05:05:00.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.692+0000 7fe8b9600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe8b4102e20 0x7fe8b4113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:05:00.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.692+0000 7fe8b9600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe8b41037c0 0x7fe8b4114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:05:00.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.692+0000 7fe8b9600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b4114690 con 0x7fe8b41037c0 2024-03-30T05:05:00.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.692+0000 7fe8b9600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b410da80 con 0x7fe8b4102e20 2024-03-30T05:05:00.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.692+0000 7fe8b2000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe8b41037c0 0x7fe8b4114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:05:00.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.692+0000 7fe8b2000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe8b41037c0 0x7fe8b4114040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45326/0 (socket says 172.21.15.123:45326) 2024-03-30T05:05:00.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.692+0000 7fe8b2000700 1 -- 172.21.15.123:0/3324096744 learned_addr learned my addr 172.21.15.123:0/3324096744 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:05:00.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.692+0000 7fe8b2a00700 1 --2- 172.21.15.123:0/3324096744 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe8b4102e20 0x7fe8b4113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:05:00.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.692+0000 7fe8b2000700 1 -- 172.21.15.123:0/3324096744 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe8b4102e20 msgr2=0x7fe8b4113b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:00.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.692+0000 7fe8b2000700 1 --2- 172.21.15.123:0/3324096744 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe8b4102e20 0x7fe8b4113b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:00.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.693+0000 7fe8b2000700 1 -- 172.21.15.123:0/3324096744 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8a4005040 con 0x7fe8b41037c0 2024-03-30T05:05:00.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.693+0000 7fe8b2000700 1 --2- 172.21.15.123:0/3324096744 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe8b41037c0 0x7fe8b4114040 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fe8a0002a50 tx=0x7fe8a0004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:05:00.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.693+0000 7fe8aaa00700 1 -- 172.21.15.123:0/3324096744 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe8a0007cc0 con 0x7fe8b41037c0 2024-03-30T05:05:00.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.693+0000 7fe8aaa00700 1 -- 172.21.15.123:0/3324096744 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe8a000f040 con 0x7fe8b41037c0 2024-03-30T05:05:00.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.693+0000 7fe8b9600700 1 -- 172.21.15.123:0/3324096744 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8b410dd60 con 0x7fe8b41037c0 2024-03-30T05:05:00.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.693+0000 7fe8aaa00700 1 -- 172.21.15.123:0/3324096744 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe8a000d920 con 0x7fe8b41037c0 2024-03-30T05:05:00.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.693+0000 7fe8b9600700 1 -- 172.21.15.123:0/3324096744 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8b410e320 con 0x7fe8b41037c0 2024-03-30T05:05:00.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.694+0000 7fe8b9600700 1 -- 172.21.15.123:0/3324096744 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8b40fe570 con 0x7fe8b41037c0 2024-03-30T05:05:00.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.695+0000 7fe8aaa00700 1 -- 172.21.15.123:0/3324096744 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fe8a0009680 con 0x7fe8b41037c0 2024-03-30T05:05:00.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.696+0000 7fe8aaa00700 1 --2- 172.21.15.123:0/3324096744 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe89806cc30 0x7fe89806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:05:00.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.696+0000 7fe8aaa00700 1 -- 172.21.15.123:0/3324096744 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe8a008bdb0 con 0x7fe8b41037c0 2024-03-30T05:05:00.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.696+0000 7fe8b2a00700 1 --2- 172.21.15.123:0/3324096744 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe89806cc30 0x7fe89806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:05:00.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.697+0000 7fe8b2a00700 1 --2- 172.21.15.123:0/3324096744 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe89806cc30 0x7fe89806f0e0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fe8a4009fd0 tx=0x7fe8a4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:05:00.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.701+0000 7fe8aaa00700 1 -- 172.21.15.123:0/3324096744 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe8a0059780 con 0x7fe8b41037c0 2024-03-30T05:05:00.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.915+0000 7fe8b9600700 1 -- 172.21.15.123:0/3324096744 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fe8b4109de0 con 0x7fe89806cc30 2024-03-30T05:05:00.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.925+0000 7fe8aaa00700 1 -- 172.21.15.123:0/3324096744 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fe8b4109de0 con 0x7fe89806cc30 2024-03-30T05:05:00.924 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T05:05:00.924 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (34m) 11m ago 46m 14.6M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T05:05:00.924 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (31m) 11m ago 46m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T05:05:00.924 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (31m) 11m ago 45m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T05:05:00.924 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (33m) 11m ago 46m 73.5M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T05:05:00.924 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (27m) 11m ago 40m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T05:05:00.924 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (27m) 11m ago 40m 16.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T05:05:00.924 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (11m) 11m ago 40m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 87b073a6c9ca 2024-03-30T05:05:00.924 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (26m) 11m ago 40m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T05:05:00.924 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (36m) 11m ago 48m 510M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T05:05:00.924 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (35m) 11m ago 45m 456M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T05:05:00.924 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (32m) 11m ago 48m 70.0M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T05:05:00.924 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (32m) 11m ago 45m 51.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T05:05:00.925 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (34m) 11m ago 46m 8636k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T05:05:00.925 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (34m) 11m ago 45m 8678k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T05:05:00.925 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (30m) 11m ago 44m 107M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T05:05:00.925 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (30m) 11m ago 44m 97.2M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T05:05:00.925 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (29m) 11m ago 43m 89.8M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T05:05:00.925 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (29m) 11m ago 43m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T05:05:00.925 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (28m) 11m ago 42m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T05:05:00.925 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (27m) 11m ago 42m 88.8M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T05:05:00.925 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (34m) 11m ago 46m 63.9M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T05:05:00.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.928+0000 7fe8b9600700 1 -- 172.21.15.123:0/3324096744 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe89806cc30 msgr2=0x7fe89806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:00.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.928+0000 7fe8b9600700 1 --2- 172.21.15.123:0/3324096744 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe89806cc30 0x7fe89806f0e0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fe8a4009fd0 tx=0x7fe8a4009e30 comp rx=0 tx=0).stop 2024-03-30T05:05:00.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.928+0000 7fe8b9600700 1 -- 172.21.15.123:0/3324096744 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe8b41037c0 msgr2=0x7fe8b4114040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:00.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.928+0000 7fe8b9600700 1 --2- 172.21.15.123:0/3324096744 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe8b41037c0 0x7fe8b4114040 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fe8a0002a50 tx=0x7fe8a0004600 comp rx=0 tx=0).stop 2024-03-30T05:05:00.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.928+0000 7fe8b9600700 1 -- 172.21.15.123:0/3324096744 shutdown_connections 2024-03-30T05:05:00.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.928+0000 7fe8b9600700 1 --2- 172.21.15.123:0/3324096744 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe89806cc30 0x7fe89806f0e0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:00.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.928+0000 7fe8b9600700 1 --2- 172.21.15.123:0/3324096744 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe8b4102e20 0x7fe8b4113b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:00.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.928+0000 7fe8b9600700 1 --2- 172.21.15.123:0/3324096744 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe8b41037c0 0x7fe8b4114040 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:00.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.928+0000 7fe8b9600700 1 -- 172.21.15.123:0/3324096744 >> 172.21.15.123:0/3324096744 conn(0x7fe8b40f89d0 msgr2=0x7fe8b40f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:05:00.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.928+0000 7fe8b9600700 1 -- 172.21.15.123:0/3324096744 shutdown_connections 2024-03-30T05:05:00.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:00.928+0000 7fe8b9600700 1 -- 172.21.15.123:0/3324096744 wait complete. 2024-03-30T05:05:01.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.062+0000 7f8e26e00700 1 -- 172.21.15.123:0/3599397212 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e201037a0 msgr2=0x7f8e200fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:01.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.062+0000 7f8e26e00700 1 --2- 172.21.15.123:0/3599397212 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e201037a0 0x7f8e200fce10 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c004940 tx=0x7f8e0c009e30 comp rx=0 tx=0).stop 2024-03-30T05:05:01.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.062+0000 7f8e26e00700 1 -- 172.21.15.123:0/3599397212 shutdown_connections 2024-03-30T05:05:01.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.062+0000 7f8e26e00700 1 --2- 172.21.15.123:0/3599397212 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e201037a0 0x7f8e200fce10 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:01.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.062+0000 7f8e26e00700 1 --2- 172.21.15.123:0/3599397212 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8e20102e00 0x7f8e201031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:01.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.062+0000 7f8e26e00700 1 -- 172.21.15.123:0/3599397212 >> 172.21.15.123:0/3599397212 conn(0x7f8e200f89b0 msgr2=0x7f8e200fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:05:01.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.062+0000 7f8e26e00700 1 -- 172.21.15.123:0/3599397212 shutdown_connections 2024-03-30T05:05:01.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.062+0000 7f8e26e00700 1 -- 172.21.15.123:0/3599397212 wait complete. 2024-03-30T05:05:01.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.063+0000 7f8e26e00700 1 Processor -- start 2024-03-30T05:05:01.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.063+0000 7f8e26e00700 1 -- start start 2024-03-30T05:05:01.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.063+0000 7f8e26e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8e20102e00 0x7f8e20115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:05:01.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.063+0000 7f8e26e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e201037a0 0x7f8e201161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:05:01.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.063+0000 7f8e26e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e20116810 con 0x7f8e201037a0 2024-03-30T05:05:01.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.063+0000 7f8e26e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e2010fbe0 con 0x7f8e20102e00 2024-03-30T05:05:01.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.064+0000 7f8e1f400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e201037a0 0x7f8e201161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:05:01.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.064+0000 7f8e1f400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e201037a0 0x7f8e201161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45332/0 (socket says 172.21.15.123:45332) 2024-03-30T05:05:01.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.064+0000 7f8e1f400700 1 -- 172.21.15.123:0/1700650423 learned_addr learned my addr 172.21.15.123:0/1700650423 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:05:01.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.064+0000 7f8e1fe00700 1 --2- 172.21.15.123:0/1700650423 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8e20102e00 0x7f8e20115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:05:01.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.064+0000 7f8e1f400700 1 -- 172.21.15.123:0/1700650423 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8e20102e00 msgr2=0x7f8e20115c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:01.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.064+0000 7f8e1f400700 1 --2- 172.21.15.123:0/1700650423 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8e20102e00 0x7f8e20115c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:01.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.064+0000 7f8e1f400700 1 -- 172.21.15.123:0/1700650423 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e0c005040 con 0x7f8e201037a0 2024-03-30T05:05:01.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.064+0000 7f8e1f400700 1 --2- 172.21.15.123:0/1700650423 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e201037a0 0x7f8e201161c0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c009fd0 tx=0x7f8e0c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:05:01.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.064+0000 7f8e17e00700 1 -- 172.21.15.123:0/1700650423 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e0c00e030 con 0x7f8e201037a0 2024-03-30T05:05:01.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.064+0000 7f8e17e00700 1 -- 172.21.15.123:0/1700650423 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8e0c0093b0 con 0x7f8e201037a0 2024-03-30T05:05:01.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.064+0000 7f8e26e00700 1 -- 172.21.15.123:0/1700650423 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e2010fe60 con 0x7f8e201037a0 2024-03-30T05:05:01.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.064+0000 7f8e17e00700 1 -- 172.21.15.123:0/1700650423 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e0c01cc80 con 0x7f8e201037a0 2024-03-30T05:05:01.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.065+0000 7f8e26e00700 1 -- 172.21.15.123:0/1700650423 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e20110420 con 0x7f8e201037a0 2024-03-30T05:05:01.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.066+0000 7f8e26e00700 1 -- 172.21.15.123:0/1700650423 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e200fe550 con 0x7f8e201037a0 2024-03-30T05:05:01.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.067+0000 7f8e17e00700 1 -- 172.21.15.123:0/1700650423 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f8e0c01c400 con 0x7f8e201037a0 2024-03-30T05:05:01.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.068+0000 7f8e17e00700 1 --2- 172.21.15.123:0/1700650423 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8e0406cc30 0x7f8e0406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:05:01.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.068+0000 7f8e17e00700 1 -- 172.21.15.123:0/1700650423 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f8e0c08d7c0 con 0x7f8e201037a0 2024-03-30T05:05:01.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.068+0000 7f8e1fe00700 1 --2- 172.21.15.123:0/1700650423 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8e0406cc30 0x7f8e0406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:05:01.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.069+0000 7f8e1fe00700 1 --2- 172.21.15.123:0/1700650423 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8e0406cc30 0x7f8e0406f0e0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f8e10009b80 tx=0x7f8e1000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:05:01.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.072+0000 7f8e17e00700 1 -- 172.21.15.123:0/1700650423 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8e0c05b290 con 0x7f8e201037a0 2024-03-30T05:05:01.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:05:01 smithi135 ceph-mon[106011]: pgmap v1015: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:01.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:05:01 smithi135 ceph-mon[106011]: from='client.35832 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:05:01.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.357+0000 7f8e26e00700 1 -- 172.21.15.123:0/1700650423 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f8e2004bed0 con 0x7f8e201037a0 2024-03-30T05:05:01.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.358+0000 7f8e17e00700 1 -- 172.21.15.123:0/1700650423 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f8e0c05ae20 con 0x7f8e201037a0 2024-03-30T05:05:01.356 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T05:05:01.356 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T05:05:01.356 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T05:05:01.356 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:05:01.356 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T05:05:01.356 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T05:05:01.356 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:05:01.357 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T05:05:01.357 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T05:05:01.357 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:05:01.357 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T05:05:01.357 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 4 2024-03-30T05:05:01.357 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:05:01.357 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T05:05:01.357 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 14 2024-03-30T05:05:01.357 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T05:05:01.357 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T05:05:01.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.360+0000 7f8e26e00700 1 -- 172.21.15.123:0/1700650423 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8e0406cc30 msgr2=0x7f8e0406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:01.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.360+0000 7f8e26e00700 1 --2- 172.21.15.123:0/1700650423 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8e0406cc30 0x7f8e0406f0e0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f8e10009b80 tx=0x7f8e1000b040 comp rx=0 tx=0).stop 2024-03-30T05:05:01.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.360+0000 7f8e26e00700 1 -- 172.21.15.123:0/1700650423 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e201037a0 msgr2=0x7f8e201161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:01.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.360+0000 7f8e26e00700 1 --2- 172.21.15.123:0/1700650423 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e201037a0 0x7f8e201161c0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c009fd0 tx=0x7f8e0c009e30 comp rx=0 tx=0).stop 2024-03-30T05:05:01.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.360+0000 7f8e26e00700 1 -- 172.21.15.123:0/1700650423 shutdown_connections 2024-03-30T05:05:01.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.360+0000 7f8e26e00700 1 --2- 172.21.15.123:0/1700650423 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f8e0406cc30 0x7f8e0406f0e0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:01.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.360+0000 7f8e26e00700 1 --2- 172.21.15.123:0/1700650423 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8e20102e00 0x7f8e20115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:01.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.360+0000 7f8e26e00700 1 --2- 172.21.15.123:0/1700650423 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e201037a0 0x7f8e201161c0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:01.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.360+0000 7f8e26e00700 1 -- 172.21.15.123:0/1700650423 >> 172.21.15.123:0/1700650423 conn(0x7f8e200f89b0 msgr2=0x7f8e200f99a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:05:01.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.360+0000 7f8e26e00700 1 -- 172.21.15.123:0/1700650423 shutdown_connections 2024-03-30T05:05:01.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.360+0000 7f8e26e00700 1 -- 172.21.15.123:0/1700650423 wait complete. 2024-03-30T05:05:01.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:05:01 smithi123 ceph-mon[139369]: pgmap v1015: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:01.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:05:01 smithi123 ceph-mon[139369]: from='client.35832 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:05:01.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.494+0000 7f0e3e800700 1 -- 172.21.15.123:0/715123168 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e38102e00 msgr2=0x7f0e381031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:01.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.494+0000 7f0e3e800700 1 --2- 172.21.15.123:0/715123168 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e38102e00 0x7f0e381031d0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f0e28004940 tx=0x7f0e28009e30 comp rx=0 tx=0).stop 2024-03-30T05:05:01.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.495+0000 7f0e3e800700 1 -- 172.21.15.123:0/715123168 shutdown_connections 2024-03-30T05:05:01.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.495+0000 7f0e3e800700 1 --2- 172.21.15.123:0/715123168 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0e381037a0 0x7f0e380fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:01.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.495+0000 7f0e3e800700 1 --2- 172.21.15.123:0/715123168 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e38102e00 0x7f0e381031d0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:01.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.495+0000 7f0e3e800700 1 -- 172.21.15.123:0/715123168 >> 172.21.15.123:0/715123168 conn(0x7f0e380f89b0 msgr2=0x7f0e380fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:05:01.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.495+0000 7f0e3e800700 1 -- 172.21.15.123:0/715123168 shutdown_connections 2024-03-30T05:05:01.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.495+0000 7f0e3e800700 1 -- 172.21.15.123:0/715123168 wait complete. 2024-03-30T05:05:01.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.496+0000 7f0e3e800700 1 Processor -- start 2024-03-30T05:05:01.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.496+0000 7f0e3e800700 1 -- start start 2024-03-30T05:05:01.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.496+0000 7f0e3e800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0e38102e00 0x7f0e38113a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:05:01.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.496+0000 7f0e3e800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e381037a0 0x7f0e38113f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:05:01.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.496+0000 7f0e3e800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e381145d0 con 0x7f0e381037a0 2024-03-30T05:05:01.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.496+0000 7f0e3e800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3810d9c0 con 0x7f0e38102e00 2024-03-30T05:05:01.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.496+0000 7f0e36a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e381037a0 0x7f0e38113f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:05:01.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.496+0000 7f0e36a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e381037a0 0x7f0e38113f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45354/0 (socket says 172.21.15.123:45354) 2024-03-30T05:05:01.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.496+0000 7f0e36a00700 1 -- 172.21.15.123:0/2121168952 learned_addr learned my addr 172.21.15.123:0/2121168952 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:05:01.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.497+0000 7f0e37400700 1 --2- 172.21.15.123:0/2121168952 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0e38102e00 0x7f0e38113a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:05:01.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.497+0000 7f0e36a00700 1 -- 172.21.15.123:0/2121168952 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0e38102e00 msgr2=0x7f0e38113a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:01.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.497+0000 7f0e36a00700 1 --2- 172.21.15.123:0/2121168952 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0e38102e00 0x7f0e38113a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:01.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.497+0000 7f0e36a00700 1 -- 172.21.15.123:0/2121168952 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e28005040 con 0x7f0e381037a0 2024-03-30T05:05:01.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.497+0000 7f0e36a00700 1 --2- 172.21.15.123:0/2121168952 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e381037a0 0x7f0e38113f80 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f0e2400eb40 tx=0x7f0e2400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:05:01.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.497+0000 7f0e2f400700 1 -- 172.21.15.123:0/2121168952 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e24007cc0 con 0x7f0e381037a0 2024-03-30T05:05:01.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.497+0000 7f0e2f400700 1 -- 172.21.15.123:0/2121168952 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0e2400f040 con 0x7f0e381037a0 2024-03-30T05:05:01.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.497+0000 7f0e3e800700 1 -- 172.21.15.123:0/2121168952 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e3810dc40 con 0x7f0e381037a0 2024-03-30T05:05:01.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.497+0000 7f0e2f400700 1 -- 172.21.15.123:0/2121168952 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e2400c8f0 con 0x7f0e381037a0 2024-03-30T05:05:01.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.497+0000 7f0e3e800700 1 -- 172.21.15.123:0/2121168952 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e3810e200 con 0x7f0e381037a0 2024-03-30T05:05:01.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.498+0000 7f0e3e800700 1 -- 172.21.15.123:0/2121168952 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e380fe550 con 0x7f0e381037a0 2024-03-30T05:05:01.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.499+0000 7f0e2f400700 1 -- 172.21.15.123:0/2121168952 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f0e2400cb60 con 0x7f0e381037a0 2024-03-30T05:05:01.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.500+0000 7f0e2f400700 1 --2- 172.21.15.123:0/2121168952 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0e1c06cc30 0x7f0e1c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:05:01.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.500+0000 7f0e2f400700 1 -- 172.21.15.123:0/2121168952 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f0e2408bb50 con 0x7f0e381037a0 2024-03-30T05:05:01.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.500+0000 7f0e37400700 1 --2- 172.21.15.123:0/2121168952 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0e1c06cc30 0x7f0e1c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:05:01.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.501+0000 7f0e37400700 1 --2- 172.21.15.123:0/2121168952 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0e1c06cc30 0x7f0e1c06f0e0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f0e2800b0d0 tx=0x7f0e28009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:05:01.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.504+0000 7f0e2f400700 1 -- 172.21.15.123:0/2121168952 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0e240595d0 con 0x7f0e381037a0 2024-03-30T05:05:01.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.749+0000 7f0e3e800700 1 -- 172.21.15.123:0/2121168952 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0e38109dc0 con 0x7f0e381037a0 2024-03-30T05:05:01.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.750+0000 7f0e2f400700 1 -- 172.21.15.123:0/2121168952 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 34 v34) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f0e24059160 con 0x7f0e381037a0 2024-03-30T05:05:01.748 INFO:teuthology.orchestra.run.smithi123.stdout:e34 2024-03-30T05:05:01.748 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T05:05:01.748 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T05:05:01.748 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T05:05:01.748 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:05:01.748 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T05:05:01.748 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T05:05:01.749 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T05:05:01.749 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T05:05:01.749 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T05:05:01.749 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T05:05:01.749 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T05:05:01.749 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T05:05:01.749 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T05:05:01.749 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T05:05:01.749 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T05:05:01.749 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T05:05:01.749 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T05:05:01.749 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T05:05:01.749 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T05:05:01.749 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T05:05:01.749 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T05:05:01.750 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T05:05:01.750 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T05:05:01.750 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T05:05:01.750 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T05:05:01.750 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T05:05:01.750 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T05:05:01.750 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T05:05:01.750 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T05:05:01.750 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T05:05:01.750 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T05:05:01.750 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T05:05:01.750 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:05:01.750 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:05:01.751 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T05:05:01.751 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:05:01.751 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:05:01.751 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:05:01.751 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:05:01.751 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44975} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/3788988014,v1:172.21.15.135:6827/3788988014] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:05:01.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.752+0000 7f0e3e800700 1 -- 172.21.15.123:0/2121168952 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0e1c06cc30 msgr2=0x7f0e1c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:01.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.752+0000 7f0e3e800700 1 --2- 172.21.15.123:0/2121168952 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0e1c06cc30 0x7f0e1c06f0e0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f0e2800b0d0 tx=0x7f0e28009e30 comp rx=0 tx=0).stop 2024-03-30T05:05:01.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.752+0000 7f0e3e800700 1 -- 172.21.15.123:0/2121168952 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e381037a0 msgr2=0x7f0e38113f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:01.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.752+0000 7f0e3e800700 1 --2- 172.21.15.123:0/2121168952 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e381037a0 0x7f0e38113f80 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f0e2400eb40 tx=0x7f0e2400b040 comp rx=0 tx=0).stop 2024-03-30T05:05:01.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.752+0000 7f0e3e800700 1 -- 172.21.15.123:0/2121168952 shutdown_connections 2024-03-30T05:05:01.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.752+0000 7f0e3e800700 1 --2- 172.21.15.123:0/2121168952 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0e1c06cc30 0x7f0e1c06f0e0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:01.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.752+0000 7f0e3e800700 1 --2- 172.21.15.123:0/2121168952 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0e38102e00 0x7f0e38113a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:01.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.752+0000 7f0e3e800700 1 --2- 172.21.15.123:0/2121168952 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e381037a0 0x7f0e38113f80 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:01.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.752+0000 7f0e3e800700 1 -- 172.21.15.123:0/2121168952 >> 172.21.15.123:0/2121168952 conn(0x7f0e380f89b0 msgr2=0x7f0e380f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:05:01.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.752+0000 7f0e3e800700 1 -- 172.21.15.123:0/2121168952 shutdown_connections 2024-03-30T05:05:01.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.752+0000 7f0e3e800700 1 -- 172.21.15.123:0/2121168952 wait complete. 2024-03-30T05:05:01.752 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 34 2024-03-30T05:05:01.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.886+0000 7fc761400700 1 -- 172.21.15.123:0/1566856710 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc75c102e00 msgr2=0x7fc75c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:01.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.886+0000 7fc761400700 1 --2- 172.21.15.123:0/1566856710 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc75c102e00 0x7fc75c1031d0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fc74c004940 tx=0x7fc74c009e30 comp rx=0 tx=0).stop 2024-03-30T05:05:01.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.886+0000 7fc761400700 1 -- 172.21.15.123:0/1566856710 shutdown_connections 2024-03-30T05:05:01.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.886+0000 7fc761400700 1 --2- 172.21.15.123:0/1566856710 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc75c1037a0 0x7fc75c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:01.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.886+0000 7fc761400700 1 --2- 172.21.15.123:0/1566856710 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc75c102e00 0x7fc75c1031d0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:01.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.886+0000 7fc761400700 1 -- 172.21.15.123:0/1566856710 >> 172.21.15.123:0/1566856710 conn(0x7fc75c0f89b0 msgr2=0x7fc75c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:05:01.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.886+0000 7fc761400700 1 -- 172.21.15.123:0/1566856710 shutdown_connections 2024-03-30T05:05:01.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.886+0000 7fc761400700 1 -- 172.21.15.123:0/1566856710 wait complete. 2024-03-30T05:05:01.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.887+0000 7fc761400700 1 Processor -- start 2024-03-30T05:05:01.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.887+0000 7fc761400700 1 -- start start 2024-03-30T05:05:01.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.887+0000 7fc761400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc75c102e00 0x7fc75c115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:05:01.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.887+0000 7fc761400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc75c1037a0 0x7fc75c1161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:05:01.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.887+0000 7fc761400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc75c116800 con 0x7fc75c1037a0 2024-03-30T05:05:01.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.887+0000 7fc761400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc75c10fbd0 con 0x7fc75c102e00 2024-03-30T05:05:01.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.888+0000 7fc759e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc75c1037a0 0x7fc75c1161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:05:01.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.888+0000 7fc759e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc75c1037a0 0x7fc75c1161b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45378/0 (socket says 172.21.15.123:45378) 2024-03-30T05:05:01.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.888+0000 7fc759e00700 1 -- 172.21.15.123:0/2696659809 learned_addr learned my addr 172.21.15.123:0/2696659809 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:05:01.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.888+0000 7fc75a800700 1 --2- 172.21.15.123:0/2696659809 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc75c102e00 0x7fc75c115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:05:01.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.888+0000 7fc759e00700 1 -- 172.21.15.123:0/2696659809 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc75c102e00 msgr2=0x7fc75c115c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:01.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.888+0000 7fc759e00700 1 --2- 172.21.15.123:0/2696659809 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc75c102e00 0x7fc75c115c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:01.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.888+0000 7fc759e00700 1 -- 172.21.15.123:0/2696659809 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc74c005040 con 0x7fc75c1037a0 2024-03-30T05:05:01.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.888+0000 7fc759e00700 1 --2- 172.21.15.123:0/2696659809 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc75c1037a0 0x7fc75c1161b0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fc74800eb40 tx=0x7fc74800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:05:01.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.888+0000 7fc752a00700 1 -- 172.21.15.123:0/2696659809 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc748007cc0 con 0x7fc75c1037a0 2024-03-30T05:05:01.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.888+0000 7fc752a00700 1 -- 172.21.15.123:0/2696659809 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc74800f040 con 0x7fc75c1037a0 2024-03-30T05:05:01.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.889+0000 7fc761400700 1 -- 172.21.15.123:0/2696659809 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc75c10fe50 con 0x7fc75c1037a0 2024-03-30T05:05:01.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.889+0000 7fc752a00700 1 -- 172.21.15.123:0/2696659809 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc748009570 con 0x7fc75c1037a0 2024-03-30T05:05:01.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.889+0000 7fc761400700 1 -- 172.21.15.123:0/2696659809 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc75c110410 con 0x7fc75c1037a0 2024-03-30T05:05:01.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.890+0000 7fc761400700 1 -- 172.21.15.123:0/2696659809 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc75c0fe550 con 0x7fc75c1037a0 2024-03-30T05:05:01.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.891+0000 7fc752a00700 1 -- 172.21.15.123:0/2696659809 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fc7480096d0 con 0x7fc75c1037a0 2024-03-30T05:05:01.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.891+0000 7fc752a00700 1 --2- 172.21.15.123:0/2696659809 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc74006cc30 0x7fc74006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:05:01.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.892+0000 7fc752a00700 1 -- 172.21.15.123:0/2696659809 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc74808bb00 con 0x7fc75c1037a0 2024-03-30T05:05:01.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.892+0000 7fc75a800700 1 --2- 172.21.15.123:0/2696659809 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc74006cc30 0x7fc74006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:05:01.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.892+0000 7fc75a800700 1 --2- 172.21.15.123:0/2696659809 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc74006cc30 0x7fc74006f0e0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fc74c004560 tx=0x7fc74c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:05:01.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:01.896+0000 7fc752a00700 1 -- 172.21.15.123:0/2696659809 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc748059600 con 0x7fc75c1037a0 2024-03-30T05:05:02.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.120+0000 7fc761400700 1 -- 172.21.15.123:0/2696659809 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc75c109e30 con 0x7fc74006cc30 2024-03-30T05:05:02.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.122+0000 7fc752a00700 1 -- 172.21.15.123:0/2696659809 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fc75c109e30 con 0x7fc74006cc30 2024-03-30T05:05:02.120 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T05:05:02.120 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T05:05:02.120 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T05:05:02.120 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T05:05:02.120 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T05:05:02.120 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T05:05:02.120 INFO:teuthology.orchestra.run.smithi123.stdout: "mds", 2024-03-30T05:05:02.120 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T05:05:02.120 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T05:05:02.120 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T05:05:02.121 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T05:05:02.121 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "16/21 daemons upgraded", 2024-03-30T05:05:02.121 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T05:05:02.121 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T05:05:02.121 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T05:05:02.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.124+0000 7fc761400700 1 -- 172.21.15.123:0/2696659809 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc74006cc30 msgr2=0x7fc74006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:02.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.124+0000 7fc761400700 1 --2- 172.21.15.123:0/2696659809 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc74006cc30 0x7fc74006f0e0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fc74c004560 tx=0x7fc74c009e30 comp rx=0 tx=0).stop 2024-03-30T05:05:02.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.124+0000 7fc761400700 1 -- 172.21.15.123:0/2696659809 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc75c1037a0 msgr2=0x7fc75c1161b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:02.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.124+0000 7fc761400700 1 --2- 172.21.15.123:0/2696659809 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc75c1037a0 0x7fc75c1161b0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fc74800eb40 tx=0x7fc74800b040 comp rx=0 tx=0).stop 2024-03-30T05:05:02.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.124+0000 7fc761400700 1 -- 172.21.15.123:0/2696659809 shutdown_connections 2024-03-30T05:05:02.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.124+0000 7fc761400700 1 --2- 172.21.15.123:0/2696659809 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc74006cc30 0x7fc74006f0e0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:02.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.124+0000 7fc761400700 1 --2- 172.21.15.123:0/2696659809 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc75c102e00 0x7fc75c115c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:02.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.124+0000 7fc761400700 1 --2- 172.21.15.123:0/2696659809 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc75c1037a0 0x7fc75c1161b0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:02.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.124+0000 7fc761400700 1 -- 172.21.15.123:0/2696659809 >> 172.21.15.123:0/2696659809 conn(0x7fc75c0f89b0 msgr2=0x7fc75c0f9960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:05:02.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.124+0000 7fc761400700 1 -- 172.21.15.123:0/2696659809 shutdown_connections 2024-03-30T05:05:02.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.124+0000 7fc761400700 1 -- 172.21.15.123:0/2696659809 wait complete. 2024-03-30T05:05:02.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.258+0000 7f614ac00700 1 -- 172.21.15.123:0/120373967 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61441037a0 msgr2=0x7f61440fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:02.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.258+0000 7f614ac00700 1 --2- 172.21.15.123:0/120373967 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61441037a0 0x7f61440fce10 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f6130004940 tx=0x7f6130009e30 comp rx=0 tx=0).stop 2024-03-30T05:05:02.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.258+0000 7f614ac00700 1 -- 172.21.15.123:0/120373967 shutdown_connections 2024-03-30T05:05:02.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.258+0000 7f614ac00700 1 --2- 172.21.15.123:0/120373967 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61441037a0 0x7f61440fce10 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:02.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.258+0000 7f614ac00700 1 --2- 172.21.15.123:0/120373967 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6144102e00 0x7f61441031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:02.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.258+0000 7f614ac00700 1 -- 172.21.15.123:0/120373967 >> 172.21.15.123:0/120373967 conn(0x7f61440f89b0 msgr2=0x7f61440fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:05:02.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.258+0000 7f614ac00700 1 -- 172.21.15.123:0/120373967 shutdown_connections 2024-03-30T05:05:02.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.258+0000 7f614ac00700 1 -- 172.21.15.123:0/120373967 wait complete. 2024-03-30T05:05:02.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.259+0000 7f614ac00700 1 Processor -- start 2024-03-30T05:05:02.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.259+0000 7f614ac00700 1 -- start start 2024-03-30T05:05:02.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.259+0000 7f614ac00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6144102e00 0x7f6144115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:05:02.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.259+0000 7f614ac00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61441037a0 0x7f61441161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:05:02.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.259+0000 7f614ac00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6144116810 con 0x7f61441037a0 2024-03-30T05:05:02.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.259+0000 7f614ac00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f614410fbe0 con 0x7f6144102e00 2024-03-30T05:05:02.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.260+0000 7f6143400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61441037a0 0x7f61441161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:05:02.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.260+0000 7f6143400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61441037a0 0x7f61441161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45386/0 (socket says 172.21.15.123:45386) 2024-03-30T05:05:02.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.260+0000 7f6143400700 1 -- 172.21.15.123:0/1305065875 learned_addr learned my addr 172.21.15.123:0/1305065875 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:05:02.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.260+0000 7f6143e00700 1 --2- 172.21.15.123:0/1305065875 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6144102e00 0x7f6144115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:05:02.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.260+0000 7f6143400700 1 -- 172.21.15.123:0/1305065875 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6144102e00 msgr2=0x7f6144115c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:02.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.260+0000 7f6143400700 1 --2- 172.21.15.123:0/1305065875 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6144102e00 0x7f6144115c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:02.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.260+0000 7f6143400700 1 -- 172.21.15.123:0/1305065875 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6130005040 con 0x7f61441037a0 2024-03-30T05:05:02.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.260+0000 7f6143400700 1 --2- 172.21.15.123:0/1305065875 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61441037a0 0x7f61441161c0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f6130009390 tx=0x7f6130009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:05:02.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.260+0000 7f613be00700 1 -- 172.21.15.123:0/1305065875 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f613000e040 con 0x7f61441037a0 2024-03-30T05:05:02.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.260+0000 7f613be00700 1 -- 172.21.15.123:0/1305065875 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6130013420 con 0x7f61441037a0 2024-03-30T05:05:02.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.260+0000 7f614ac00700 1 -- 172.21.15.123:0/1305065875 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f614410fe60 con 0x7f61441037a0 2024-03-30T05:05:02.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.260+0000 7f613be00700 1 -- 172.21.15.123:0/1305065875 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f613001c630 con 0x7f61441037a0 2024-03-30T05:05:02.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.261+0000 7f614ac00700 1 -- 172.21.15.123:0/1305065875 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6144110420 con 0x7f61441037a0 2024-03-30T05:05:02.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.262+0000 7f614ac00700 1 -- 172.21.15.123:0/1305065875 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61440fe550 con 0x7f61441037a0 2024-03-30T05:05:02.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.262+0000 7f613be00700 1 -- 172.21.15.123:0/1305065875 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f6130013a00 con 0x7f61441037a0 2024-03-30T05:05:02.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.264+0000 7f613be00700 1 --2- 172.21.15.123:0/1305065875 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f612806cc30 0x7f612806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:05:02.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.264+0000 7f613be00700 1 -- 172.21.15.123:0/1305065875 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f613008d0e0 con 0x7f61441037a0 2024-03-30T05:05:02.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.264+0000 7f6143e00700 1 --2- 172.21.15.123:0/1305065875 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f612806cc30 0x7f612806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:05:02.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.265+0000 7f6143e00700 1 --2- 172.21.15.123:0/1305065875 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f612806cc30 0x7f612806f0e0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f6134009fd0 tx=0x7f613400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:05:02.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.269+0000 7f613be00700 1 -- 172.21.15.123:0/1305065875 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f613005abb0 con 0x7f61441037a0 2024-03-30T05:05:02.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:05:02 smithi135 ceph-mon[106011]: from='client.35836 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:05:02.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:05:02 smithi135 ceph-mon[106011]: from='client.35840 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:05:02.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:05:02 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/1700650423' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:05:02.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:05:02 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2121168952' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T05:05:02.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:05:02 smithi123 ceph-mon[139369]: from='client.35836 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:05:02.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:05:02 smithi123 ceph-mon[139369]: from='client.35840 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:05:02.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:05:02 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/1700650423' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:05:02.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:05:02 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2121168952' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T05:05:02.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.550+0000 7f614ac00700 1 -- 172.21.15.123:0/1305065875 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f614404bed0 con 0x7f61441037a0 2024-03-30T05:05:02.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.551+0000 7f613be00700 1 -- 172.21.15.123:0/1305065875 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f613005a740 con 0x7f61441037a0 2024-03-30T05:05:02.550 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T05:05:02.550 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-30T05:05:02.550 INFO:teuthology.orchestra.run.smithi123.stdout: host smithi135 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi135/config 2024-03-30T05:05:02.550 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-30T05:05:02.550 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T05:05:02.550 INFO:teuthology.orchestra.run.smithi123.stdout:Traceback (most recent call last): 2024-03-30T05:05:02.550 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-30T05:05:02.550 INFO:teuthology.orchestra.run.smithi123.stdout: "__main__", mod_spec) 2024-03-30T05:05:02.550 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-30T05:05:02.550 INFO:teuthology.orchestra.run.smithi123.stdout: exec(code, run_globals) 2024-03-30T05:05:02.551 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10889, in 2024-03-30T05:05:02.551 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10877, in main 2024-03-30T05:05:02.551 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-30T05:05:02.551 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-30T05:05:02.551 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-30T05:05:02.551 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-30T05:05:02.551 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-03-30T05:05:02.551 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-30T05:05:02.551 INFO:teuthology.orchestra.run.smithi123.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T05:05:02.551 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T05:05:02.551 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T05:05:02.551 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T05:05:02.551 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T05:05:02.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.553+0000 7f614ac00700 1 -- 172.21.15.123:0/1305065875 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f612806cc30 msgr2=0x7f612806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:02.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.553+0000 7f614ac00700 1 --2- 172.21.15.123:0/1305065875 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f612806cc30 0x7f612806f0e0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f6134009fd0 tx=0x7f613400b040 comp rx=0 tx=0).stop 2024-03-30T05:05:02.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.553+0000 7f614ac00700 1 -- 172.21.15.123:0/1305065875 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61441037a0 msgr2=0x7f61441161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:02.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.553+0000 7f614ac00700 1 --2- 172.21.15.123:0/1305065875 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61441037a0 0x7f61441161c0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f6130009390 tx=0x7f6130009e30 comp rx=0 tx=0).stop 2024-03-30T05:05:02.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.553+0000 7f614ac00700 1 -- 172.21.15.123:0/1305065875 shutdown_connections 2024-03-30T05:05:02.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.553+0000 7f614ac00700 1 --2- 172.21.15.123:0/1305065875 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f612806cc30 0x7f612806f0e0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:02.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.553+0000 7f614ac00700 1 --2- 172.21.15.123:0/1305065875 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6144102e00 0x7f6144115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:02.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.553+0000 7f614ac00700 1 --2- 172.21.15.123:0/1305065875 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61441037a0 0x7f61441161c0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:02.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.553+0000 7f614ac00700 1 -- 172.21.15.123:0/1305065875 >> 172.21.15.123:0/1305065875 conn(0x7f61440f89b0 msgr2=0x7f61440f99a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:05:02.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.553+0000 7f614ac00700 1 -- 172.21.15.123:0/1305065875 shutdown_connections 2024-03-30T05:05:02.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:02.554+0000 7f614ac00700 1 -- 172.21.15.123:0/1305065875 wait complete. 2024-03-30T05:05:03.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:05:03 smithi135 ceph-mon[106011]: pgmap v1016: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:03.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:05:03 smithi135 ceph-mon[106011]: from='client.35852 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:05:03.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:05:03 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/1305065875' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T05:05:03.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:05:03 smithi123 ceph-mon[139369]: pgmap v1016: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:03.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:05:03 smithi123 ceph-mon[139369]: from='client.35852 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:05:03.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:05:03 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/1305065875' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T05:05:05.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:05:05 smithi135 ceph-mon[106011]: pgmap v1017: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:05.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:05:05 smithi123 ceph-mon[139369]: pgmap v1017: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:07.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:05:07 smithi135 ceph-mon[106011]: pgmap v1018: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:07.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:05:07 smithi123 ceph-mon[139369]: pgmap v1018: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:09.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:05:09 smithi135 ceph-mon[106011]: pgmap v1019: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:09.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:05:09 smithi123 ceph-mon[139369]: pgmap v1019: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:11.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:05:11 smithi135 ceph-mon[106011]: pgmap v1020: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:11.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:05:11 smithi123 ceph-mon[139369]: pgmap v1020: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:13.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:05:13 smithi135 ceph-mon[106011]: pgmap v1021: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:13.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:05:13 smithi123 ceph-mon[139369]: pgmap v1021: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:15.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:05:15 smithi135 ceph-mon[106011]: pgmap v1022: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:15.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:05:15 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:05:15.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:05:15 smithi123 ceph-mon[139369]: pgmap v1022: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:15.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:05:15 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:05:17.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:05:17 smithi135 ceph-mon[106011]: pgmap v1023: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:17.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:05:17 smithi123 ceph-mon[139369]: pgmap v1023: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:19.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:05:19 smithi135 ceph-mon[106011]: pgmap v1024: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:19.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:05:19 smithi123 ceph-mon[139369]: pgmap v1024: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:21.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:05:21 smithi123 ceph-mon[139369]: pgmap v1025: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:21.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:05:21 smithi135 ceph-mon[106011]: pgmap v1025: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:23.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:05:23 smithi123 ceph-mon[139369]: pgmap v1026: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:23.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:05:23 smithi135 ceph-mon[106011]: pgmap v1026: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:25.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:05:25 smithi123 ceph-mon[139369]: pgmap v1027: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:25.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:05:25 smithi135 ceph-mon[106011]: pgmap v1027: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:27.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:05:27 smithi123 ceph-mon[139369]: pgmap v1028: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:27.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:05:27 smithi135 ceph-mon[106011]: pgmap v1028: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:29.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:05:29 smithi123 ceph-mon[139369]: pgmap v1029: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:29.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:05:29 smithi135 ceph-mon[106011]: pgmap v1029: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:30.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:05:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:05:30.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:05:30 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:05:31.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:05:31 smithi123 ceph-mon[139369]: pgmap v1030: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:31.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:05:31 smithi135 ceph-mon[106011]: pgmap v1030: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:32.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:32.700+0000 7fd10da00700 1 -- 172.21.15.123:0/2890050127 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd108102e00 msgr2=0x7fd1081031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:32.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:32.700+0000 7fd10da00700 1 --2- 172.21.15.123:0/2890050127 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd108102e00 0x7fd1081031d0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fd0f8004940 tx=0x7fd0f8009e30 comp rx=0 tx=0).stop 2024-03-30T05:05:32.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:32.700+0000 7fd10da00700 1 -- 172.21.15.123:0/2890050127 shutdown_connections 2024-03-30T05:05:32.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:32.700+0000 7fd10da00700 1 --2- 172.21.15.123:0/2890050127 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd1081037a0 0x7fd1080fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:32.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:32.700+0000 7fd10da00700 1 --2- 172.21.15.123:0/2890050127 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd108102e00 0x7fd1081031d0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:32.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:32.700+0000 7fd10da00700 1 -- 172.21.15.123:0/2890050127 >> 172.21.15.123:0/2890050127 conn(0x7fd1080f89b0 msgr2=0x7fd1080fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:05:32.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:32.701+0000 7fd10da00700 1 -- 172.21.15.123:0/2890050127 shutdown_connections 2024-03-30T05:05:32.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:32.701+0000 7fd10da00700 1 -- 172.21.15.123:0/2890050127 wait complete. 2024-03-30T05:05:32.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:32.701+0000 7fd10da00700 1 Processor -- start 2024-03-30T05:05:32.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:32.701+0000 7fd10da00700 1 -- start start 2024-03-30T05:05:32.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:32.702+0000 7fd10da00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd108102e00 0x7fd108115c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:05:32.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:32.702+0000 7fd10da00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd1081037a0 0x7fd108116190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:05:32.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:32.702+0000 7fd10da00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1081167e0 con 0x7fd108102e00 2024-03-30T05:05:32.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:32.702+0000 7fd10da00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd10810fbb0 con 0x7fd1081037a0 2024-03-30T05:05:32.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:32.702+0000 7fd106a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd108102e00 0x7fd108115c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:05:32.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:32.702+0000 7fd106a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd108102e00 0x7fd108115c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56938/0 (socket says 172.21.15.123:56938) 2024-03-30T05:05:32.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:32.702+0000 7fd106a00700 1 -- 172.21.15.123:0/1472610038 learned_addr learned my addr 172.21.15.123:0/1472610038 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:05:32.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:32.702+0000 7fd106000700 1 --2- 172.21.15.123:0/1472610038 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd1081037a0 0x7fd108116190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:05:32.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:32.702+0000 7fd106a00700 1 -- 172.21.15.123:0/1472610038 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd1081037a0 msgr2=0x7fd108116190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:32.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:32.702+0000 7fd106a00700 1 --2- 172.21.15.123:0/1472610038 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd1081037a0 0x7fd108116190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:32.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:32.703+0000 7fd106a00700 1 -- 172.21.15.123:0/1472610038 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0f8005040 con 0x7fd108102e00 2024-03-30T05:05:32.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:32.703+0000 7fd106000700 1 --2- 172.21.15.123:0/1472610038 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd1081037a0 0x7fd108116190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T05:05:32.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:32.703+0000 7fd106a00700 1 --2- 172.21.15.123:0/1472610038 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd108102e00 0x7fd108115c50 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fd0f8009390 tx=0x7fd0f8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:05:32.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:32.703+0000 7fd0fea00700 1 -- 172.21.15.123:0/1472610038 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd0f800e040 con 0x7fd108102e00 2024-03-30T05:05:32.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:32.703+0000 7fd0fea00700 1 -- 172.21.15.123:0/1472610038 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd0f8013420 con 0x7fd108102e00 2024-03-30T05:05:32.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:32.703+0000 7fd10da00700 1 -- 172.21.15.123:0/1472610038 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd10810fe30 con 0x7fd108102e00 2024-03-30T05:05:32.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:32.703+0000 7fd0fea00700 1 -- 172.21.15.123:0/1472610038 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd0f801c630 con 0x7fd108102e00 2024-03-30T05:05:32.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:32.703+0000 7fd10da00700 1 -- 172.21.15.123:0/1472610038 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1081103f0 con 0x7fd108102e00 2024-03-30T05:05:32.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:32.704+0000 7fd10da00700 1 -- 172.21.15.123:0/1472610038 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1080fe550 con 0x7fd108102e00 2024-03-30T05:05:32.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:32.705+0000 7fd0fea00700 1 -- 172.21.15.123:0/1472610038 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fd0f80139e0 con 0x7fd108102e00 2024-03-30T05:05:32.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:32.706+0000 7fd0fea00700 1 --2- 172.21.15.123:0/1472610038 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd0ec06cc30 0x7fd0ec06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:05:32.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:32.706+0000 7fd0fea00700 1 -- 172.21.15.123:0/1472610038 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd0f808d250 con 0x7fd108102e00 2024-03-30T05:05:32.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:32.706+0000 7fd106000700 1 --2- 172.21.15.123:0/1472610038 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd0ec06cc30 0x7fd0ec06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:05:32.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:32.707+0000 7fd106000700 1 --2- 172.21.15.123:0/1472610038 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd0ec06cc30 0x7fd0ec06f0e0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fd0f4007fd0 tx=0x7fd0f4009f60 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:05:32.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:32.711+0000 7fd0fea00700 1 -- 172.21.15.123:0/1472610038 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd0f805add0 con 0x7fd108102e00 2024-03-30T05:05:32.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:32.931+0000 7fd10da00700 1 -- 172.21.15.123:0/1472610038 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd10804bf40 con 0x7fd0ec06cc30 2024-03-30T05:05:32.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:32.931+0000 7fd0fea00700 1 -- 172.21.15.123:0/1472610038 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fd10804bf40 con 0x7fd0ec06cc30 2024-03-30T05:05:32.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:32.933+0000 7fd10da00700 1 -- 172.21.15.123:0/1472610038 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd0ec06cc30 msgr2=0x7fd0ec06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:32.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:32.933+0000 7fd10da00700 1 --2- 172.21.15.123:0/1472610038 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd0ec06cc30 0x7fd0ec06f0e0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fd0f4007fd0 tx=0x7fd0f4009f60 comp rx=0 tx=0).stop 2024-03-30T05:05:32.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:32.933+0000 7fd10da00700 1 -- 172.21.15.123:0/1472610038 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd108102e00 msgr2=0x7fd108115c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:32.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:32.933+0000 7fd10da00700 1 --2- 172.21.15.123:0/1472610038 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd108102e00 0x7fd108115c50 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fd0f8009390 tx=0x7fd0f8009e30 comp rx=0 tx=0).stop 2024-03-30T05:05:32.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:32.933+0000 7fd10da00700 1 -- 172.21.15.123:0/1472610038 shutdown_connections 2024-03-30T05:05:32.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:32.933+0000 7fd10da00700 1 --2- 172.21.15.123:0/1472610038 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd0ec06cc30 0x7fd0ec06f0e0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:32.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:32.933+0000 7fd10da00700 1 --2- 172.21.15.123:0/1472610038 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd108102e00 0x7fd108115c50 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:32.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:32.933+0000 7fd10da00700 1 --2- 172.21.15.123:0/1472610038 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd1081037a0 0x7fd108116190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:32.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:32.933+0000 7fd10da00700 1 -- 172.21.15.123:0/1472610038 >> 172.21.15.123:0/1472610038 conn(0x7fd1080f89b0 msgr2=0x7fd1080f9900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:05:32.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:32.935+0000 7fd10da00700 1 -- 172.21.15.123:0/1472610038 shutdown_connections 2024-03-30T05:05:32.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:32.935+0000 7fd10da00700 1 -- 172.21.15.123:0/1472610038 wait complete. 2024-03-30T05:05:32.946 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T05:05:33.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.076+0000 7f0b3f200700 1 -- 172.21.15.123:0/1137634998 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b38070c70 msgr2=0x7f0b38109110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:33.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.076+0000 7f0b3f200700 1 --2- 172.21.15.123:0/1137634998 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b38070c70 0x7f0b38109110 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f0b24004970 tx=0x7f0b24009e30 comp rx=0 tx=0).stop 2024-03-30T05:05:33.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.076+0000 7f0b3f200700 1 -- 172.21.15.123:0/1137634998 shutdown_connections 2024-03-30T05:05:33.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.076+0000 7f0b3f200700 1 --2- 172.21.15.123:0/1137634998 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b38070c70 0x7f0b38109110 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:33.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.076+0000 7f0b3f200700 1 --2- 172.21.15.123:0/1137634998 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0b38070360 0x7f0b38070730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:33.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.077+0000 7f0b3f200700 1 -- 172.21.15.123:0/1137634998 >> 172.21.15.123:0/1137634998 conn(0x7f0b380f89b0 msgr2=0x7f0b380fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:05:33.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.077+0000 7f0b3f200700 1 -- 172.21.15.123:0/1137634998 shutdown_connections 2024-03-30T05:05:33.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.077+0000 7f0b3f200700 1 -- 172.21.15.123:0/1137634998 wait complete. 2024-03-30T05:05:33.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.077+0000 7f0b3f200700 1 Processor -- start 2024-03-30T05:05:33.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.078+0000 7f0b3f200700 1 -- start start 2024-03-30T05:05:33.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.078+0000 7f0b3f200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0b38070360 0x7f0b38115a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:05:33.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.078+0000 7f0b3f200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b38070c70 0x7f0b38115fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:05:33.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.078+0000 7f0b3f200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b38116600 con 0x7f0b38070c70 2024-03-30T05:05:33.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.078+0000 7f0b3f200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3810fbe0 con 0x7f0b38070360 2024-03-30T05:05:33.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.078+0000 7f0b37400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b38070c70 0x7f0b38115fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:05:33.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.078+0000 7f0b37400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b38070c70 0x7f0b38115fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56944/0 (socket says 172.21.15.123:56944) 2024-03-30T05:05:33.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.078+0000 7f0b37400700 1 -- 172.21.15.123:0/2012879412 learned_addr learned my addr 172.21.15.123:0/2012879412 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:05:33.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.078+0000 7f0b37e00700 1 --2- 172.21.15.123:0/2012879412 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0b38070360 0x7f0b38115a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:05:33.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.079+0000 7f0b37400700 1 -- 172.21.15.123:0/2012879412 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0b38070360 msgr2=0x7f0b38115a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:33.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.079+0000 7f0b37400700 1 --2- 172.21.15.123:0/2012879412 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0b38070360 0x7f0b38115a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:33.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.079+0000 7f0b37400700 1 -- 172.21.15.123:0/2012879412 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b24005040 con 0x7f0b38070c70 2024-03-30T05:05:33.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.079+0000 7f0b37400700 1 --2- 172.21.15.123:0/2012879412 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b38070c70 0x7f0b38115fb0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f0b24005010 tx=0x7f0b24009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:05:33.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.079+0000 7f0b2fe00700 1 -- 172.21.15.123:0/2012879412 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b2400e040 con 0x7f0b38070c70 2024-03-30T05:05:33.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.079+0000 7f0b2fe00700 1 -- 172.21.15.123:0/2012879412 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0b240098b0 con 0x7f0b38070c70 2024-03-30T05:05:33.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.079+0000 7f0b3f200700 1 -- 172.21.15.123:0/2012879412 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b3810fe60 con 0x7f0b38070c70 2024-03-30T05:05:33.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.079+0000 7f0b2fe00700 1 -- 172.21.15.123:0/2012879412 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b24013e80 con 0x7f0b38070c70 2024-03-30T05:05:33.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.079+0000 7f0b3f200700 1 -- 172.21.15.123:0/2012879412 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b381103a0 con 0x7f0b38070c70 2024-03-30T05:05:33.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.080+0000 7f0b3f200700 1 -- 172.21.15.123:0/2012879412 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b38071aa0 con 0x7f0b38070c70 2024-03-30T05:05:33.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.081+0000 7f0b2fe00700 1 -- 172.21.15.123:0/2012879412 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f0b240136b0 con 0x7f0b38070c70 2024-03-30T05:05:33.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.082+0000 7f0b2fe00700 1 --2- 172.21.15.123:0/2012879412 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0b1c06cc20 0x7f0b1c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:05:33.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.082+0000 7f0b2fe00700 1 -- 172.21.15.123:0/2012879412 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f0b2408d180 con 0x7f0b38070c70 2024-03-30T05:05:33.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.082+0000 7f0b37e00700 1 --2- 172.21.15.123:0/2012879412 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0b1c06cc20 0x7f0b1c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:05:33.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.083+0000 7f0b37e00700 1 --2- 172.21.15.123:0/2012879412 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0b1c06cc20 0x7f0b1c06f0d0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f0b28009b80 tx=0x7f0b2800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:05:33.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.086+0000 7f0b2fe00700 1 -- 172.21.15.123:0/2012879412 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0b2405ad00 con 0x7f0b38070c70 2024-03-30T05:05:33.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.315+0000 7f0b3f200700 1 -- 172.21.15.123:0/2012879412 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0b3810c020 con 0x7f0b1c06cc20 2024-03-30T05:05:33.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.316+0000 7f0b2fe00700 1 -- 172.21.15.123:0/2012879412 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f0b3810c020 con 0x7f0b1c06cc20 2024-03-30T05:05:33.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.319+0000 7f0b3f200700 1 -- 172.21.15.123:0/2012879412 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0b1c06cc20 msgr2=0x7f0b1c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:33.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.319+0000 7f0b3f200700 1 --2- 172.21.15.123:0/2012879412 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0b1c06cc20 0x7f0b1c06f0d0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f0b28009b80 tx=0x7f0b2800b040 comp rx=0 tx=0).stop 2024-03-30T05:05:33.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.319+0000 7f0b3f200700 1 -- 172.21.15.123:0/2012879412 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b38070c70 msgr2=0x7f0b38115fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:33.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.319+0000 7f0b3f200700 1 --2- 172.21.15.123:0/2012879412 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b38070c70 0x7f0b38115fb0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f0b24005010 tx=0x7f0b24009e30 comp rx=0 tx=0).stop 2024-03-30T05:05:33.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.319+0000 7f0b3f200700 1 -- 172.21.15.123:0/2012879412 shutdown_connections 2024-03-30T05:05:33.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.319+0000 7f0b3f200700 1 --2- 172.21.15.123:0/2012879412 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0b1c06cc20 0x7f0b1c06f0d0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:33.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.319+0000 7f0b3f200700 1 --2- 172.21.15.123:0/2012879412 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0b38070360 0x7f0b38115a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:33.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.319+0000 7f0b3f200700 1 --2- 172.21.15.123:0/2012879412 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b38070c70 0x7f0b38115fb0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:33.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.319+0000 7f0b3f200700 1 -- 172.21.15.123:0/2012879412 >> 172.21.15.123:0/2012879412 conn(0x7f0b380f89b0 msgr2=0x7f0b38105a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:05:33.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.319+0000 7f0b3f200700 1 -- 172.21.15.123:0/2012879412 shutdown_connections 2024-03-30T05:05:33.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.319+0000 7f0b3f200700 1 -- 172.21.15.123:0/2012879412 wait complete. 2024-03-30T05:05:33.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.453+0000 7f77d5400700 1 -- 172.21.15.123:0/3780542979 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77d0102e00 msgr2=0x7f77d01031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:33.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.453+0000 7f77d5400700 1 --2- 172.21.15.123:0/3780542979 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77d0102e00 0x7f77d01031d0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f77c0004940 tx=0x7f77c0009e30 comp rx=0 tx=0).stop 2024-03-30T05:05:33.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.453+0000 7f77d5400700 1 -- 172.21.15.123:0/3780542979 shutdown_connections 2024-03-30T05:05:33.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.453+0000 7f77d5400700 1 --2- 172.21.15.123:0/3780542979 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f77d01037a0 0x7f77d00fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:33.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.453+0000 7f77d5400700 1 --2- 172.21.15.123:0/3780542979 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77d0102e00 0x7f77d01031d0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:33.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.453+0000 7f77d5400700 1 -- 172.21.15.123:0/3780542979 >> 172.21.15.123:0/3780542979 conn(0x7f77d00f89b0 msgr2=0x7f77d00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:05:33.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.453+0000 7f77d5400700 1 -- 172.21.15.123:0/3780542979 shutdown_connections 2024-03-30T05:05:33.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.453+0000 7f77d5400700 1 -- 172.21.15.123:0/3780542979 wait complete. 2024-03-30T05:05:33.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.454+0000 7f77d5400700 1 Processor -- start 2024-03-30T05:05:33.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.454+0000 7f77d5400700 1 -- start start 2024-03-30T05:05:33.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.455+0000 7f77d5400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f77d0102e00 0x7f77d00702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:05:33.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.455+0000 7f77d5400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77d01037a0 0x7f77d0070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:05:33.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.455+0000 7f77d5400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d0070d90 con 0x7f77d01037a0 2024-03-30T05:05:33.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.455+0000 7f77d5400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d0070ed0 con 0x7f77d0102e00 2024-03-30T05:05:33.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.455+0000 7f77cde00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77d01037a0 0x7f77d0070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:05:33.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.455+0000 7f77cde00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77d01037a0 0x7f77d0070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56974/0 (socket says 172.21.15.123:56974) 2024-03-30T05:05:33.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.455+0000 7f77cde00700 1 -- 172.21.15.123:0/1496891553 learned_addr learned my addr 172.21.15.123:0/1496891553 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:05:33.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.455+0000 7f77ce800700 1 --2- 172.21.15.123:0/1496891553 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f77d0102e00 0x7f77d00702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:05:33.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.455+0000 7f77cde00700 1 -- 172.21.15.123:0/1496891553 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f77d0102e00 msgr2=0x7f77d00702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:33.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.455+0000 7f77cde00700 1 --2- 172.21.15.123:0/1496891553 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f77d0102e00 0x7f77d00702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:33.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.455+0000 7f77cde00700 1 -- 172.21.15.123:0/1496891553 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77c0005040 con 0x7f77d01037a0 2024-03-30T05:05:33.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.456+0000 7f77cde00700 1 --2- 172.21.15.123:0/1496891553 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77d01037a0 0x7f77d0070800 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f77bc002a50 tx=0x7f77bc004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:05:33.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.456+0000 7f77c6a00700 1 -- 172.21.15.123:0/1496891553 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f77bc007cc0 con 0x7f77d01037a0 2024-03-30T05:05:33.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.456+0000 7f77c6a00700 1 -- 172.21.15.123:0/1496891553 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f77bc00f040 con 0x7f77d01037a0 2024-03-30T05:05:33.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.456+0000 7f77c6a00700 1 -- 172.21.15.123:0/1496891553 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f77bc00d920 con 0x7f77d01037a0 2024-03-30T05:05:33.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.456+0000 7f77d5400700 1 -- 172.21.15.123:0/1496891553 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77d0119e20 con 0x7f77d01037a0 2024-03-30T05:05:33.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.456+0000 7f77d5400700 1 -- 172.21.15.123:0/1496891553 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77d011a2a0 con 0x7f77d01037a0 2024-03-30T05:05:33.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.457+0000 7f77d5400700 1 -- 172.21.15.123:0/1496891553 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77d00fe550 con 0x7f77d01037a0 2024-03-30T05:05:33.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.458+0000 7f77c6a00700 1 -- 172.21.15.123:0/1496891553 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f77bc009680 con 0x7f77d01037a0 2024-03-30T05:05:33.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.459+0000 7f77c6a00700 1 --2- 172.21.15.123:0/1496891553 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f77b406cc30 0x7f77b406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:05:33.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.459+0000 7f77c6a00700 1 -- 172.21.15.123:0/1496891553 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f77bc08bcf0 con 0x7f77d01037a0 2024-03-30T05:05:33.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.459+0000 7f77ce800700 1 --2- 172.21.15.123:0/1496891553 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f77b406cc30 0x7f77b406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:05:33.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.459+0000 7f77ce800700 1 --2- 172.21.15.123:0/1496891553 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f77b406cc30 0x7f77b406f0e0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f77c0009fd0 tx=0x7f77c0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:05:33.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.463+0000 7f77c6a00700 1 -- 172.21.15.123:0/1496891553 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f77bc055cb0 con 0x7f77d01037a0 2024-03-30T05:05:33.465 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:05:33 smithi123 ceph-mon[139369]: pgmap v1031: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:33.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:05:33 smithi135 ceph-mon[106011]: pgmap v1031: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:33.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.677+0000 7f77d5400700 1 -- 172.21.15.123:0/1496891553 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f77d004bed0 con 0x7f77b406cc30 2024-03-30T05:05:33.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.687+0000 7f77c6a00700 1 -- 172.21.15.123:0/1496891553 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f77d004bed0 con 0x7f77b406cc30 2024-03-30T05:05:33.684 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T05:05:33.684 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (34m) 12m ago 47m 14.6M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T05:05:33.684 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (31m) 12m ago 47m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T05:05:33.684 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (31m) 12m ago 46m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T05:05:33.684 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (34m) 12m ago 46m 73.5M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T05:05:33.684 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (27m) 12m ago 40m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T05:05:33.684 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (28m) 12m ago 40m 16.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T05:05:33.684 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (12m) 12m ago 40m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 87b073a6c9ca 2024-03-30T05:05:33.684 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (27m) 12m ago 40m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T05:05:33.685 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (37m) 12m ago 48m 510M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T05:05:33.685 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (36m) 12m ago 46m 456M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T05:05:33.685 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (33m) 12m ago 49m 70.0M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T05:05:33.685 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (32m) 12m ago 46m 51.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T05:05:33.685 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (35m) 12m ago 47m 8636k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T05:05:33.685 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (35m) 12m ago 46m 8678k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T05:05:33.685 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (31m) 12m ago 45m 107M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T05:05:33.685 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (31m) 12m ago 44m 97.2M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T05:05:33.685 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (30m) 12m ago 44m 89.8M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T05:05:33.685 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (29m) 12m ago 43m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T05:05:33.685 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (29m) 12m ago 43m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T05:05:33.685 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (28m) 12m ago 42m 88.8M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T05:05:33.685 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (35m) 12m ago 46m 63.9M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T05:05:33.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.689+0000 7f77d5400700 1 -- 172.21.15.123:0/1496891553 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f77b406cc30 msgr2=0x7f77b406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:33.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.689+0000 7f77d5400700 1 --2- 172.21.15.123:0/1496891553 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f77b406cc30 0x7f77b406f0e0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f77c0009fd0 tx=0x7f77c0009e30 comp rx=0 tx=0).stop 2024-03-30T05:05:33.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.689+0000 7f77d5400700 1 -- 172.21.15.123:0/1496891553 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77d01037a0 msgr2=0x7f77d0070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:33.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.689+0000 7f77d5400700 1 --2- 172.21.15.123:0/1496891553 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77d01037a0 0x7f77d0070800 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f77bc002a50 tx=0x7f77bc004600 comp rx=0 tx=0).stop 2024-03-30T05:05:33.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.689+0000 7f77d5400700 1 -- 172.21.15.123:0/1496891553 shutdown_connections 2024-03-30T05:05:33.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.689+0000 7f77d5400700 1 --2- 172.21.15.123:0/1496891553 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f77b406cc30 0x7f77b406f0e0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:33.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.689+0000 7f77d5400700 1 --2- 172.21.15.123:0/1496891553 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f77d0102e00 0x7f77d00702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:33.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.689+0000 7f77d5400700 1 --2- 172.21.15.123:0/1496891553 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77d01037a0 0x7f77d0070800 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:33.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.689+0000 7f77d5400700 1 -- 172.21.15.123:0/1496891553 >> 172.21.15.123:0/1496891553 conn(0x7f77d00f89b0 msgr2=0x7f77d00f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:05:33.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.690+0000 7f77d5400700 1 -- 172.21.15.123:0/1496891553 shutdown_connections 2024-03-30T05:05:33.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.690+0000 7f77d5400700 1 -- 172.21.15.123:0/1496891553 wait complete. 2024-03-30T05:05:33.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.822+0000 7f51b0200700 1 -- 172.21.15.123:0/3227135483 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51a8102e00 msgr2=0x7f51a81031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:33.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.822+0000 7f51b0200700 1 --2- 172.21.15.123:0/3227135483 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51a8102e00 0x7f51a81031d0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f519c004940 tx=0x7f519c009e30 comp rx=0 tx=0).stop 2024-03-30T05:05:33.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.822+0000 7f51b0200700 1 -- 172.21.15.123:0/3227135483 shutdown_connections 2024-03-30T05:05:33.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.822+0000 7f51b0200700 1 --2- 172.21.15.123:0/3227135483 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f51a81037a0 0x7f51a80fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:33.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.822+0000 7f51b0200700 1 --2- 172.21.15.123:0/3227135483 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51a8102e00 0x7f51a81031d0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:33.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.822+0000 7f51b0200700 1 -- 172.21.15.123:0/3227135483 >> 172.21.15.123:0/3227135483 conn(0x7f51a80f89b0 msgr2=0x7f51a80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:05:33.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.822+0000 7f51b0200700 1 -- 172.21.15.123:0/3227135483 shutdown_connections 2024-03-30T05:05:33.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.822+0000 7f51b0200700 1 -- 172.21.15.123:0/3227135483 wait complete. 2024-03-30T05:05:33.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.823+0000 7f51b0200700 1 Processor -- start 2024-03-30T05:05:33.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.823+0000 7f51b0200700 1 -- start start 2024-03-30T05:05:33.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.823+0000 7f51b0200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51a8102e00 0x7f51a8115c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:05:33.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.823+0000 7f51b0200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f51a81037a0 0x7f51a8116180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:05:33.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.823+0000 7f51b0200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51a81167d0 con 0x7f51a8102e00 2024-03-30T05:05:33.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.823+0000 7f51b0200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51a810fba0 con 0x7f51a81037a0 2024-03-30T05:05:33.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.823+0000 7f51ad800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51a8102e00 0x7f51a8115c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:05:33.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.824+0000 7f51ad800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51a8102e00 0x7f51a8115c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37590/0 (socket says 172.21.15.123:37590) 2024-03-30T05:05:33.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.824+0000 7f51ad800700 1 -- 172.21.15.123:0/2908544558 learned_addr learned my addr 172.21.15.123:0/2908544558 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:05:33.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.824+0000 7f51ace00700 1 --2- 172.21.15.123:0/2908544558 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f51a81037a0 0x7f51a8116180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:05:33.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.824+0000 7f51ad800700 1 -- 172.21.15.123:0/2908544558 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f51a81037a0 msgr2=0x7f51a8116180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:33.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.824+0000 7f51ad800700 1 --2- 172.21.15.123:0/2908544558 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f51a81037a0 0x7f51a8116180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:33.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.824+0000 7f51ad800700 1 -- 172.21.15.123:0/2908544558 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f519c005040 con 0x7f51a8102e00 2024-03-30T05:05:33.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.824+0000 7f51ad800700 1 --2- 172.21.15.123:0/2908544558 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51a8102e00 0x7f51a8115c40 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f519c007f50 tx=0x7f519c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:05:33.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.824+0000 7f51a5600700 1 -- 172.21.15.123:0/2908544558 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f519c021070 con 0x7f51a8102e00 2024-03-30T05:05:33.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.824+0000 7f51a5600700 1 -- 172.21.15.123:0/2908544558 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f519c013e00 con 0x7f51a8102e00 2024-03-30T05:05:33.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.824+0000 7f51b0200700 1 -- 172.21.15.123:0/2908544558 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51a810fe20 con 0x7f51a8102e00 2024-03-30T05:05:33.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.824+0000 7f51a5600700 1 -- 172.21.15.123:0/2908544558 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f519c00e070 con 0x7f51a8102e00 2024-03-30T05:05:33.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.825+0000 7f51b0200700 1 -- 172.21.15.123:0/2908544558 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51a81103e0 con 0x7f51a8102e00 2024-03-30T05:05:33.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.826+0000 7f51b0200700 1 -- 172.21.15.123:0/2908544558 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51a80fe550 con 0x7f51a8102e00 2024-03-30T05:05:33.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.827+0000 7f51a5600700 1 -- 172.21.15.123:0/2908544558 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f519c013950 con 0x7f51a8102e00 2024-03-30T05:05:33.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.827+0000 7f51a5600700 1 --2- 172.21.15.123:0/2908544558 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f519006cc30 0x7f519006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:05:33.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.828+0000 7f51a5600700 1 -- 172.21.15.123:0/2908544558 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f519c08d250 con 0x7f51a8102e00 2024-03-30T05:05:33.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.828+0000 7f51ace00700 1 --2- 172.21.15.123:0/2908544558 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f519006cc30 0x7f519006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:05:33.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.828+0000 7f51ace00700 1 --2- 172.21.15.123:0/2908544558 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f519006cc30 0x7f519006f0e0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f5194004600 tx=0x7f519400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:05:33.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:33.832+0000 7f51a5600700 1 -- 172.21.15.123:0/2908544558 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f519c05ad20 con 0x7f51a8102e00 2024-03-30T05:05:34.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.116+0000 7f51b0200700 1 -- 172.21.15.123:0/2908544558 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f51a804bf20 con 0x7f51a8102e00 2024-03-30T05:05:34.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.117+0000 7f51a5600700 1 -- 172.21.15.123:0/2908544558 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f519c05a8b0 con 0x7f51a8102e00 2024-03-30T05:05:34.114 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T05:05:34.114 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T05:05:34.114 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T05:05:34.114 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:05:34.115 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T05:05:34.115 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T05:05:34.115 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:05:34.115 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T05:05:34.115 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T05:05:34.115 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:05:34.115 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T05:05:34.115 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 4 2024-03-30T05:05:34.115 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:05:34.115 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T05:05:34.115 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 14 2024-03-30T05:05:34.115 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T05:05:34.115 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T05:05:34.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.119+0000 7f51b0200700 1 -- 172.21.15.123:0/2908544558 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f519006cc30 msgr2=0x7f519006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:34.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.119+0000 7f51b0200700 1 --2- 172.21.15.123:0/2908544558 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f519006cc30 0x7f519006f0e0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f5194004600 tx=0x7f519400b040 comp rx=0 tx=0).stop 2024-03-30T05:05:34.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.119+0000 7f51b0200700 1 -- 172.21.15.123:0/2908544558 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51a8102e00 msgr2=0x7f51a8115c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:34.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.119+0000 7f51b0200700 1 --2- 172.21.15.123:0/2908544558 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51a8102e00 0x7f51a8115c40 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f519c007f50 tx=0x7f519c009e30 comp rx=0 tx=0).stop 2024-03-30T05:05:34.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.119+0000 7f51b0200700 1 -- 172.21.15.123:0/2908544558 shutdown_connections 2024-03-30T05:05:34.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.119+0000 7f51b0200700 1 --2- 172.21.15.123:0/2908544558 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f519006cc30 0x7f519006f0e0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:34.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.119+0000 7f51b0200700 1 --2- 172.21.15.123:0/2908544558 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51a8102e00 0x7f51a8115c40 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:34.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.119+0000 7f51b0200700 1 --2- 172.21.15.123:0/2908544558 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f51a81037a0 0x7f51a8116180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:34.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.119+0000 7f51b0200700 1 -- 172.21.15.123:0/2908544558 >> 172.21.15.123:0/2908544558 conn(0x7f51a80f89b0 msgr2=0x7f51a80f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:05:34.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.119+0000 7f51b0200700 1 -- 172.21.15.123:0/2908544558 shutdown_connections 2024-03-30T05:05:34.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.119+0000 7f51b0200700 1 -- 172.21.15.123:0/2908544558 wait complete. 2024-03-30T05:05:34.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.252+0000 7faa8fa00700 1 -- 172.21.15.123:0/796871772 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa88102e00 msgr2=0x7faa881031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:34.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.252+0000 7faa8fa00700 1 --2- 172.21.15.123:0/796871772 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa88102e00 0x7faa881031d0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7faa7c004940 tx=0x7faa7c009e30 comp rx=0 tx=0).stop 2024-03-30T05:05:34.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.252+0000 7faa8fa00700 1 -- 172.21.15.123:0/796871772 shutdown_connections 2024-03-30T05:05:34.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.252+0000 7faa8fa00700 1 --2- 172.21.15.123:0/796871772 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa881037a0 0x7faa880fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:34.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.252+0000 7faa8fa00700 1 --2- 172.21.15.123:0/796871772 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa88102e00 0x7faa881031d0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:34.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.252+0000 7faa8fa00700 1 -- 172.21.15.123:0/796871772 >> 172.21.15.123:0/796871772 conn(0x7faa880f89b0 msgr2=0x7faa880fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:05:34.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.253+0000 7faa8fa00700 1 -- 172.21.15.123:0/796871772 shutdown_connections 2024-03-30T05:05:34.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.253+0000 7faa8fa00700 1 -- 172.21.15.123:0/796871772 wait complete. 2024-03-30T05:05:34.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.253+0000 7faa8fa00700 1 Processor -- start 2024-03-30T05:05:34.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.253+0000 7faa8fa00700 1 -- start start 2024-03-30T05:05:34.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.254+0000 7faa8fa00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa88102e00 0x7faa88115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:05:34.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.254+0000 7faa8fa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa881037a0 0x7faa881161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:05:34.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.254+0000 7faa8fa00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa88116810 con 0x7faa881037a0 2024-03-30T05:05:34.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.254+0000 7faa8fa00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa8810fbe0 con 0x7faa88102e00 2024-03-30T05:05:34.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.254+0000 7faa87e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa881037a0 0x7faa881161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:05:34.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.254+0000 7faa87e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa881037a0 0x7faa881161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37604/0 (socket says 172.21.15.123:37604) 2024-03-30T05:05:34.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.254+0000 7faa87e00700 1 -- 172.21.15.123:0/2737143436 learned_addr learned my addr 172.21.15.123:0/2737143436 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:05:34.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.254+0000 7faa8d000700 1 --2- 172.21.15.123:0/2737143436 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa88102e00 0x7faa88115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:05:34.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.254+0000 7faa87e00700 1 -- 172.21.15.123:0/2737143436 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa88102e00 msgr2=0x7faa88115c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:34.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.254+0000 7faa87e00700 1 --2- 172.21.15.123:0/2737143436 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa88102e00 0x7faa88115c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:34.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.254+0000 7faa87e00700 1 -- 172.21.15.123:0/2737143436 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa7c005040 con 0x7faa881037a0 2024-03-30T05:05:34.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.255+0000 7faa87e00700 1 --2- 172.21.15.123:0/2737143436 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa881037a0 0x7faa881161c0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7faa7400eb40 tx=0x7faa7400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:05:34.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.255+0000 7faa84c00700 1 -- 172.21.15.123:0/2737143436 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa74007cc0 con 0x7faa881037a0 2024-03-30T05:05:34.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.255+0000 7faa84c00700 1 -- 172.21.15.123:0/2737143436 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faa7400f040 con 0x7faa881037a0 2024-03-30T05:05:34.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.255+0000 7faa8fa00700 1 -- 172.21.15.123:0/2737143436 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa8810fe60 con 0x7faa881037a0 2024-03-30T05:05:34.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.255+0000 7faa84c00700 1 -- 172.21.15.123:0/2737143436 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa7400c8f0 con 0x7faa881037a0 2024-03-30T05:05:34.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.255+0000 7faa8fa00700 1 -- 172.21.15.123:0/2737143436 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa88110420 con 0x7faa881037a0 2024-03-30T05:05:34.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.256+0000 7faa8fa00700 1 -- 172.21.15.123:0/2737143436 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa880fe550 con 0x7faa881037a0 2024-03-30T05:05:34.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.257+0000 7faa84c00700 1 -- 172.21.15.123:0/2737143436 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7faa740090d0 con 0x7faa881037a0 2024-03-30T05:05:34.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.258+0000 7faa84c00700 1 --2- 172.21.15.123:0/2737143436 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7faa7006cc30 0x7faa7006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:05:34.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.258+0000 7faa84c00700 1 -- 172.21.15.123:0/2737143436 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7faa7408ba50 con 0x7faa881037a0 2024-03-30T05:05:34.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.259+0000 7faa8d000700 1 --2- 172.21.15.123:0/2737143436 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7faa7006cc30 0x7faa7006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:05:34.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.259+0000 7faa8d000700 1 --2- 172.21.15.123:0/2737143436 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7faa7006cc30 0x7faa7006f0e0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7faa7c0045a0 tx=0x7faa7c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:05:34.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.262+0000 7faa84c00700 1 -- 172.21.15.123:0/2737143436 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7faa74059550 con 0x7faa881037a0 2024-03-30T05:05:34.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:05:34 smithi123 ceph-mon[139369]: from='client.35860 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:05:34.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:05:34 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2908544558' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:05:34.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.508+0000 7faa8fa00700 1 -- 172.21.15.123:0/2737143436 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7faa8804bed0 con 0x7faa881037a0 2024-03-30T05:05:34.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.508+0000 7faa84c00700 1 -- 172.21.15.123:0/2737143436 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 34 v34) v1 ==== 76+0+1843 (secure 0 0 0) 0x7faa740590e0 con 0x7faa881037a0 2024-03-30T05:05:34.505 INFO:teuthology.orchestra.run.smithi123.stdout:e34 2024-03-30T05:05:34.505 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T05:05:34.506 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T05:05:34.506 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T05:05:34.506 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:05:34.506 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T05:05:34.506 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T05:05:34.506 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T05:05:34.506 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T05:05:34.506 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T05:05:34.506 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T05:05:34.506 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T05:05:34.506 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T05:05:34.507 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T05:05:34.507 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T05:05:34.507 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T05:05:34.507 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T05:05:34.507 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T05:05:34.507 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T05:05:34.507 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T05:05:34.507 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T05:05:34.507 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T05:05:34.507 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T05:05:34.507 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T05:05:34.507 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T05:05:34.507 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T05:05:34.507 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T05:05:34.508 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T05:05:34.508 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T05:05:34.508 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T05:05:34.508 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T05:05:34.508 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T05:05:34.508 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T05:05:34.508 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:05:34.508 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:05:34.508 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T05:05:34.508 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:05:34.508 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:05:34.508 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:05:34.508 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:05:34.508 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44975} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/3788988014,v1:172.21.15.135:6827/3788988014] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:05:34.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.510+0000 7faa8fa00700 1 -- 172.21.15.123:0/2737143436 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7faa7006cc30 msgr2=0x7faa7006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:34.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.510+0000 7faa8fa00700 1 --2- 172.21.15.123:0/2737143436 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7faa7006cc30 0x7faa7006f0e0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7faa7c0045a0 tx=0x7faa7c009e30 comp rx=0 tx=0).stop 2024-03-30T05:05:34.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.510+0000 7faa8fa00700 1 -- 172.21.15.123:0/2737143436 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa881037a0 msgr2=0x7faa881161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:34.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.510+0000 7faa8fa00700 1 --2- 172.21.15.123:0/2737143436 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa881037a0 0x7faa881161c0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7faa7400eb40 tx=0x7faa7400b040 comp rx=0 tx=0).stop 2024-03-30T05:05:34.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.510+0000 7faa8fa00700 1 -- 172.21.15.123:0/2737143436 shutdown_connections 2024-03-30T05:05:34.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.510+0000 7faa8fa00700 1 --2- 172.21.15.123:0/2737143436 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7faa7006cc30 0x7faa7006f0e0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:34.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.510+0000 7faa8fa00700 1 --2- 172.21.15.123:0/2737143436 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa88102e00 0x7faa88115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:34.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.510+0000 7faa8fa00700 1 --2- 172.21.15.123:0/2737143436 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa881037a0 0x7faa881161c0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:34.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.510+0000 7faa8fa00700 1 -- 172.21.15.123:0/2737143436 >> 172.21.15.123:0/2737143436 conn(0x7faa880f89b0 msgr2=0x7faa880f9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:05:34.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.510+0000 7faa8fa00700 1 -- 172.21.15.123:0/2737143436 shutdown_connections 2024-03-30T05:05:34.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.511+0000 7faa8fa00700 1 -- 172.21.15.123:0/2737143436 wait complete. 2024-03-30T05:05:34.510 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 34 2024-03-30T05:05:34.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:05:34 smithi135 ceph-mon[106011]: from='client.35860 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:05:34.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:05:34 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2908544558' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:05:34.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.644+0000 7fa735a00700 1 -- 172.21.15.123:0/906279649 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa730070c70 msgr2=0x7fa730109110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:34.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.644+0000 7fa735a00700 1 --2- 172.21.15.123:0/906279649 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa730070c70 0x7fa730109110 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fa71c004970 tx=0x7fa71c009e30 comp rx=0 tx=0).stop 2024-03-30T05:05:34.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.644+0000 7fa735a00700 1 -- 172.21.15.123:0/906279649 shutdown_connections 2024-03-30T05:05:34.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.644+0000 7fa735a00700 1 --2- 172.21.15.123:0/906279649 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa730070c70 0x7fa730109110 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:34.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.644+0000 7fa735a00700 1 --2- 172.21.15.123:0/906279649 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa730070360 0x7fa730070730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:34.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.644+0000 7fa735a00700 1 -- 172.21.15.123:0/906279649 >> 172.21.15.123:0/906279649 conn(0x7fa7300f89b0 msgr2=0x7fa7300fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:05:34.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.644+0000 7fa735a00700 1 -- 172.21.15.123:0/906279649 shutdown_connections 2024-03-30T05:05:34.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.644+0000 7fa735a00700 1 -- 172.21.15.123:0/906279649 wait complete. 2024-03-30T05:05:34.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.645+0000 7fa735a00700 1 Processor -- start 2024-03-30T05:05:34.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.645+0000 7fa735a00700 1 -- start start 2024-03-30T05:05:34.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.645+0000 7fa735a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa730070360 0x7fa730115cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:05:34.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.645+0000 7fa735a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa730070c70 0x7fa7301161f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:05:34.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.645+0000 7fa735a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa730116840 con 0x7fa730070c70 2024-03-30T05:05:34.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.645+0000 7fa735a00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa73010fc10 con 0x7fa730070360 2024-03-30T05:05:34.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.646+0000 7fa72e000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa730070c70 0x7fa7301161f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:05:34.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.646+0000 7fa72e000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa730070c70 0x7fa7301161f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37634/0 (socket says 172.21.15.123:37634) 2024-03-30T05:05:34.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.646+0000 7fa72e000700 1 -- 172.21.15.123:0/1343022103 learned_addr learned my addr 172.21.15.123:0/1343022103 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:05:34.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.646+0000 7fa72ea00700 1 --2- 172.21.15.123:0/1343022103 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa730070360 0x7fa730115cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:05:34.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.646+0000 7fa72e000700 1 -- 172.21.15.123:0/1343022103 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa730070360 msgr2=0x7fa730115cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:34.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.646+0000 7fa72e000700 1 --2- 172.21.15.123:0/1343022103 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa730070360 0x7fa730115cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:34.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.646+0000 7fa72e000700 1 -- 172.21.15.123:0/1343022103 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa71c005040 con 0x7fa730070c70 2024-03-30T05:05:34.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.646+0000 7fa72e000700 1 --2- 172.21.15.123:0/1343022103 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa730070c70 0x7fa7301161f0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fa71c005010 tx=0x7fa71c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:05:34.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.647+0000 7fa726a00700 1 -- 172.21.15.123:0/1343022103 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa71c00e040 con 0x7fa730070c70 2024-03-30T05:05:34.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.647+0000 7fa726a00700 1 -- 172.21.15.123:0/1343022103 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa71c004290 con 0x7fa730070c70 2024-03-30T05:05:34.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.647+0000 7fa735a00700 1 -- 172.21.15.123:0/1343022103 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa73010fe90 con 0x7fa730070c70 2024-03-30T05:05:34.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.647+0000 7fa726a00700 1 -- 172.21.15.123:0/1343022103 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa71c013d00 con 0x7fa730070c70 2024-03-30T05:05:34.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.647+0000 7fa735a00700 1 -- 172.21.15.123:0/1343022103 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7301103d0 con 0x7fa730070c70 2024-03-30T05:05:34.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.648+0000 7fa735a00700 1 -- 172.21.15.123:0/1343022103 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa730071aa0 con 0x7fa730070c70 2024-03-30T05:05:34.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.649+0000 7fa726a00700 1 -- 172.21.15.123:0/1343022103 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fa71c0134b0 con 0x7fa730070c70 2024-03-30T05:05:34.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.650+0000 7fa726a00700 1 --2- 172.21.15.123:0/1343022103 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa71406cc30 0x7fa71406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:05:34.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.650+0000 7fa726a00700 1 -- 172.21.15.123:0/1343022103 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa71c08d1d0 con 0x7fa730070c70 2024-03-30T05:05:34.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.650+0000 7fa72ea00700 1 --2- 172.21.15.123:0/1343022103 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa71406cc30 0x7fa71406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:05:34.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.651+0000 7fa72ea00700 1 --2- 172.21.15.123:0/1343022103 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa71406cc30 0x7fa71406f0e0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fa720009b80 tx=0x7fa72000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:05:34.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.654+0000 7fa726a00700 1 -- 172.21.15.123:0/1343022103 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa71c05ac20 con 0x7fa730070c70 2024-03-30T05:05:34.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.874+0000 7fa735a00700 1 -- 172.21.15.123:0/1343022103 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa73010c040 con 0x7fa71406cc30 2024-03-30T05:05:34.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.875+0000 7fa726a00700 1 -- 172.21.15.123:0/1343022103 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa73010c040 con 0x7fa71406cc30 2024-03-30T05:05:34.873 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T05:05:34.873 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T05:05:34.873 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T05:05:34.874 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T05:05:34.874 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T05:05:34.874 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T05:05:34.874 INFO:teuthology.orchestra.run.smithi123.stdout: "mds", 2024-03-30T05:05:34.874 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T05:05:34.874 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T05:05:34.874 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T05:05:34.874 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T05:05:34.874 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "16/21 daemons upgraded", 2024-03-30T05:05:34.874 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T05:05:34.874 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T05:05:34.874 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T05:05:34.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.878+0000 7fa735a00700 1 -- 172.21.15.123:0/1343022103 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa71406cc30 msgr2=0x7fa71406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:34.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.878+0000 7fa735a00700 1 --2- 172.21.15.123:0/1343022103 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa71406cc30 0x7fa71406f0e0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fa720009b80 tx=0x7fa72000b040 comp rx=0 tx=0).stop 2024-03-30T05:05:34.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.878+0000 7fa735a00700 1 -- 172.21.15.123:0/1343022103 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa730070c70 msgr2=0x7fa7301161f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:34.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.878+0000 7fa735a00700 1 --2- 172.21.15.123:0/1343022103 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa730070c70 0x7fa7301161f0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fa71c005010 tx=0x7fa71c009e30 comp rx=0 tx=0).stop 2024-03-30T05:05:34.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.878+0000 7fa735a00700 1 -- 172.21.15.123:0/1343022103 shutdown_connections 2024-03-30T05:05:34.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.878+0000 7fa735a00700 1 --2- 172.21.15.123:0/1343022103 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa71406cc30 0x7fa71406f0e0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:34.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.878+0000 7fa735a00700 1 --2- 172.21.15.123:0/1343022103 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa730070360 0x7fa730115cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:34.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.878+0000 7fa735a00700 1 --2- 172.21.15.123:0/1343022103 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa730070c70 0x7fa7301161f0 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:34.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.878+0000 7fa735a00700 1 -- 172.21.15.123:0/1343022103 >> 172.21.15.123:0/1343022103 conn(0x7fa7300f89b0 msgr2=0x7fa7300f9eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:05:34.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.878+0000 7fa735a00700 1 -- 172.21.15.123:0/1343022103 shutdown_connections 2024-03-30T05:05:34.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:34.878+0000 7fa735a00700 1 -- 172.21.15.123:0/1343022103 wait complete. 2024-03-30T05:05:35.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:35.011+0000 7f3870200700 1 -- 172.21.15.123:0/101543840 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38681037a0 msgr2=0x7f38680fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:35.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:35.011+0000 7f3870200700 1 --2- 172.21.15.123:0/101543840 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38681037a0 0x7f38680fce10 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f3854004940 tx=0x7f3854009e30 comp rx=0 tx=0).stop 2024-03-30T05:05:35.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:35.011+0000 7f3870200700 1 -- 172.21.15.123:0/101543840 shutdown_connections 2024-03-30T05:05:35.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:35.011+0000 7f3870200700 1 --2- 172.21.15.123:0/101543840 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38681037a0 0x7f38680fce10 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:35.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:35.011+0000 7f3870200700 1 --2- 172.21.15.123:0/101543840 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3868102e00 0x7f38681031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:35.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:35.011+0000 7f3870200700 1 -- 172.21.15.123:0/101543840 >> 172.21.15.123:0/101543840 conn(0x7f38680f89b0 msgr2=0x7f38680fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:05:35.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:35.011+0000 7f3870200700 1 -- 172.21.15.123:0/101543840 shutdown_connections 2024-03-30T05:05:35.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:35.011+0000 7f3870200700 1 -- 172.21.15.123:0/101543840 wait complete. 2024-03-30T05:05:35.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:35.012+0000 7f3870200700 1 Processor -- start 2024-03-30T05:05:35.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:35.012+0000 7f3870200700 1 -- start start 2024-03-30T05:05:35.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:35.012+0000 7f3870200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3868102e00 0x7f3868115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:05:35.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:35.012+0000 7f3870200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38681037a0 0x7f3868116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:05:35.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:35.012+0000 7f3870200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3868116880 con 0x7f38681037a0 2024-03-30T05:05:35.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:35.012+0000 7f3870200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f386810fc50 con 0x7f3868102e00 2024-03-30T05:05:35.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:35.012+0000 7f386ce00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38681037a0 0x7f3868116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:05:35.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:35.013+0000 7f386ce00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38681037a0 0x7f3868116230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37648/0 (socket says 172.21.15.123:37648) 2024-03-30T05:05:35.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:35.013+0000 7f386ce00700 1 -- 172.21.15.123:0/824779075 learned_addr learned my addr 172.21.15.123:0/824779075 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:05:35.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:35.013+0000 7f386d800700 1 --2- 172.21.15.123:0/824779075 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3868102e00 0x7f3868115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:05:35.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:35.013+0000 7f386ce00700 1 -- 172.21.15.123:0/824779075 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3868102e00 msgr2=0x7f3868115cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:35.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:35.013+0000 7f386ce00700 1 --2- 172.21.15.123:0/824779075 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3868102e00 0x7f3868115cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:35.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:35.013+0000 7f386ce00700 1 -- 172.21.15.123:0/824779075 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3854005040 con 0x7f38681037a0 2024-03-30T05:05:35.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:35.013+0000 7f386d800700 1 --2- 172.21.15.123:0/824779075 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3868102e00 0x7f3868115cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T05:05:35.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:35.013+0000 7f386ce00700 1 --2- 172.21.15.123:0/824779075 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38681037a0 0x7f3868116230 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f3854009390 tx=0x7f3854009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:05:35.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:35.013+0000 7f3865600700 1 -- 172.21.15.123:0/824779075 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f385400e040 con 0x7f38681037a0 2024-03-30T05:05:35.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:35.013+0000 7f3865600700 1 -- 172.21.15.123:0/824779075 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3854013420 con 0x7f38681037a0 2024-03-30T05:05:35.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:35.013+0000 7f3870200700 1 -- 172.21.15.123:0/824779075 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f386810fed0 con 0x7f38681037a0 2024-03-30T05:05:35.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:35.013+0000 7f3865600700 1 -- 172.21.15.123:0/824779075 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f385401c630 con 0x7f38681037a0 2024-03-30T05:05:35.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:35.014+0000 7f3870200700 1 -- 172.21.15.123:0/824779075 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3868110490 con 0x7f38681037a0 2024-03-30T05:05:35.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:35.014+0000 7f3870200700 1 -- 172.21.15.123:0/824779075 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38680fe550 con 0x7f38681037a0 2024-03-30T05:05:35.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:35.016+0000 7f3865600700 1 -- 172.21.15.123:0/824779075 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f3854013a00 con 0x7f38681037a0 2024-03-30T05:05:35.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:35.017+0000 7f3865600700 1 --2- 172.21.15.123:0/824779075 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f385006cc30 0x7f385006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:05:35.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:35.017+0000 7f3865600700 1 -- 172.21.15.123:0/824779075 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f385408dda0 con 0x7f38681037a0 2024-03-30T05:05:35.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:35.017+0000 7f386d800700 1 --2- 172.21.15.123:0/824779075 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f385006cc30 0x7f385006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:05:35.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:35.018+0000 7f386d800700 1 --2- 172.21.15.123:0/824779075 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f385006cc30 0x7f385006f0e0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f385c009fd0 tx=0x7f385c009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:05:35.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:35.021+0000 7f3865600700 1 -- 172.21.15.123:0/824779075 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f385405b870 con 0x7f38681037a0 2024-03-30T05:05:35.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:35.306+0000 7f3870200700 1 -- 172.21.15.123:0/824779075 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f386805f5f0 con 0x7f38681037a0 2024-03-30T05:05:35.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:35.307+0000 7f3865600700 1 -- 172.21.15.123:0/824779075 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f385405b400 con 0x7f38681037a0 2024-03-30T05:05:35.305 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T05:05:35.305 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-30T05:05:35.305 INFO:teuthology.orchestra.run.smithi123.stdout: host smithi135 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi135/config 2024-03-30T05:05:35.305 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-30T05:05:35.305 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T05:05:35.305 INFO:teuthology.orchestra.run.smithi123.stdout:Traceback (most recent call last): 2024-03-30T05:05:35.306 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-30T05:05:35.306 INFO:teuthology.orchestra.run.smithi123.stdout: "__main__", mod_spec) 2024-03-30T05:05:35.306 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-30T05:05:35.306 INFO:teuthology.orchestra.run.smithi123.stdout: exec(code, run_globals) 2024-03-30T05:05:35.306 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10889, in 2024-03-30T05:05:35.306 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10877, in main 2024-03-30T05:05:35.306 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-30T05:05:35.306 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-30T05:05:35.306 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-30T05:05:35.307 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-30T05:05:35.307 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-03-30T05:05:35.307 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-30T05:05:35.307 INFO:teuthology.orchestra.run.smithi123.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T05:05:35.307 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T05:05:35.307 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T05:05:35.307 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T05:05:35.307 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T05:05:35.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:35.309+0000 7f3870200700 1 -- 172.21.15.123:0/824779075 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f385006cc30 msgr2=0x7f385006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:35.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:35.309+0000 7f3870200700 1 --2- 172.21.15.123:0/824779075 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f385006cc30 0x7f385006f0e0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f385c009fd0 tx=0x7f385c009ee0 comp rx=0 tx=0).stop 2024-03-30T05:05:35.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:35.309+0000 7f3870200700 1 -- 172.21.15.123:0/824779075 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38681037a0 msgr2=0x7f3868116230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:05:35.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:35.309+0000 7f3870200700 1 --2- 172.21.15.123:0/824779075 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38681037a0 0x7f3868116230 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f3854009390 tx=0x7f3854009e30 comp rx=0 tx=0).stop 2024-03-30T05:05:35.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:35.309+0000 7f3870200700 1 -- 172.21.15.123:0/824779075 shutdown_connections 2024-03-30T05:05:35.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:35.309+0000 7f3870200700 1 --2- 172.21.15.123:0/824779075 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f385006cc30 0x7f385006f0e0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:35.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:35.309+0000 7f3870200700 1 --2- 172.21.15.123:0/824779075 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3868102e00 0x7f3868115cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:35.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:35.309+0000 7f3870200700 1 --2- 172.21.15.123:0/824779075 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38681037a0 0x7f3868116230 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:05:35.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:35.309+0000 7f3870200700 1 -- 172.21.15.123:0/824779075 >> 172.21.15.123:0/824779075 conn(0x7f38680f89b0 msgr2=0x7f38680f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:05:35.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:35.309+0000 7f3870200700 1 -- 172.21.15.123:0/824779075 shutdown_connections 2024-03-30T05:05:35.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:05:35.309+0000 7f3870200700 1 -- 172.21.15.123:0/824779075 wait complete. 2024-03-30T05:05:35.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:05:35 smithi123 ceph-mon[139369]: from='client.35864 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:05:35.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:05:35 smithi123 ceph-mon[139369]: pgmap v1032: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:35.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:05:35 smithi123 ceph-mon[139369]: from='client.35868 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:05:35.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:05:35 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2737143436' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T05:05:35.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:05:35 smithi135 ceph-mon[106011]: from='client.35864 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:05:35.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:05:35 smithi135 ceph-mon[106011]: pgmap v1032: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:35.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:05:35 smithi135 ceph-mon[106011]: from='client.35868 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:05:35.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:05:35 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2737143436' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T05:05:36.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:05:36 smithi123 ceph-mon[139369]: from='client.35880 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:05:36.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:05:36 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/824779075' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T05:05:36.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:05:36 smithi135 ceph-mon[106011]: from='client.35880 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:05:36.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:05:36 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/824779075' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T05:05:37.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:05:37 smithi123 ceph-mon[139369]: pgmap v1033: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:37.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:05:37 smithi135 ceph-mon[106011]: pgmap v1033: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:39.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:05:39 smithi123 ceph-mon[139369]: pgmap v1034: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:39.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:05:39 smithi135 ceph-mon[106011]: pgmap v1034: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:41.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:05:41 smithi123 ceph-mon[139369]: pgmap v1035: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:41.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:05:41 smithi135 ceph-mon[106011]: pgmap v1035: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:43.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:05:43 smithi123 ceph-mon[139369]: pgmap v1036: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:43.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:05:43 smithi135 ceph-mon[106011]: pgmap v1036: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:45.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:05:45 smithi123 ceph-mon[139369]: pgmap v1037: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:45.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:05:45 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:05:45.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:05:45 smithi135 ceph-mon[106011]: pgmap v1037: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:45.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:05:45 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:05:47.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:05:47 smithi123 ceph-mon[139369]: pgmap v1038: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:47.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:05:47 smithi135 ceph-mon[106011]: pgmap v1038: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:49.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:05:49 smithi123 ceph-mon[139369]: pgmap v1039: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:49.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:05:49 smithi135 ceph-mon[106011]: pgmap v1039: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:51.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:05:51 smithi123 ceph-mon[139369]: pgmap v1040: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:51.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:05:51 smithi135 ceph-mon[106011]: pgmap v1040: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:53.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:05:53 smithi123 ceph-mon[139369]: pgmap v1041: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:53.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:05:53 smithi135 ceph-mon[106011]: pgmap v1041: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:55.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:05:55 smithi123 ceph-mon[139369]: pgmap v1042: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:55.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:05:55 smithi135 ceph-mon[106011]: pgmap v1042: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:57.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:05:57 smithi123 ceph-mon[139369]: pgmap v1043: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:57.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:05:57 smithi135 ceph-mon[106011]: pgmap v1043: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:59.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:05:59 smithi123 ceph-mon[139369]: pgmap v1044: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:05:59.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:05:59 smithi135 ceph-mon[106011]: pgmap v1044: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:00.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:06:00 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:06:00.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:06:00 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:06:01.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:06:01 smithi123 ceph-mon[139369]: pgmap v1045: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:01.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:06:01 smithi135 ceph-mon[106011]: pgmap v1045: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:02.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:06:02 smithi123 ceph-mon[139369]: pgmap v1046: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:02.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:06:02 smithi135 ceph-mon[106011]: pgmap v1046: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:05.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:06:04 smithi135 ceph-mon[106011]: pgmap v1047: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:05.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:06:04 smithi123 ceph-mon[139369]: pgmap v1047: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:05.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.456+0000 7f2e97400700 1 -- 172.21.15.123:0/186926395 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e90102e00 msgr2=0x7f2e901031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:05.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.456+0000 7f2e97400700 1 --2- 172.21.15.123:0/186926395 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e90102e00 0x7f2e901031d0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f2e84004940 tx=0x7f2e84009e30 comp rx=0 tx=0).stop 2024-03-30T05:06:05.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.456+0000 7f2e97400700 1 -- 172.21.15.123:0/186926395 shutdown_connections 2024-03-30T05:06:05.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.456+0000 7f2e97400700 1 --2- 172.21.15.123:0/186926395 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2e901037a0 0x7f2e900fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:05.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.456+0000 7f2e97400700 1 --2- 172.21.15.123:0/186926395 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e90102e00 0x7f2e901031d0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:05.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.456+0000 7f2e97400700 1 -- 172.21.15.123:0/186926395 >> 172.21.15.123:0/186926395 conn(0x7f2e900f89b0 msgr2=0x7f2e900fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:06:05.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.456+0000 7f2e97400700 1 -- 172.21.15.123:0/186926395 shutdown_connections 2024-03-30T05:06:05.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.456+0000 7f2e97400700 1 -- 172.21.15.123:0/186926395 wait complete. 2024-03-30T05:06:05.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.457+0000 7f2e97400700 1 Processor -- start 2024-03-30T05:06:05.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.457+0000 7f2e97400700 1 -- start start 2024-03-30T05:06:05.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.458+0000 7f2e97400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e90102e00 0x7f2e900702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:06:05.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.458+0000 7f2e97400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2e901037a0 0x7f2e90070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:06:05.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.458+0000 7f2e97400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e90070d90 con 0x7f2e90102e00 2024-03-30T05:06:05.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.458+0000 7f2e97400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e90070ed0 con 0x7f2e901037a0 2024-03-30T05:06:05.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.458+0000 7f2e94a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e90102e00 0x7f2e900702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:06:05.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.458+0000 7f2e94a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e90102e00 0x7f2e900702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:32964/0 (socket says 172.21.15.123:32964) 2024-03-30T05:06:05.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.458+0000 7f2e94a00700 1 -- 172.21.15.123:0/879438384 learned_addr learned my addr 172.21.15.123:0/879438384 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:06:05.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.458+0000 7f2e8fe00700 1 --2- 172.21.15.123:0/879438384 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2e901037a0 0x7f2e90070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:06:05.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.458+0000 7f2e94a00700 1 -- 172.21.15.123:0/879438384 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2e901037a0 msgr2=0x7f2e90070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:05.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.458+0000 7f2e94a00700 1 --2- 172.21.15.123:0/879438384 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2e901037a0 0x7f2e90070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:05.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.458+0000 7f2e94a00700 1 -- 172.21.15.123:0/879438384 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e84005040 con 0x7f2e90102e00 2024-03-30T05:06:05.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.458+0000 7f2e8fe00700 1 --2- 172.21.15.123:0/879438384 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2e901037a0 0x7f2e90070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T05:06:05.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.459+0000 7f2e94a00700 1 --2- 172.21.15.123:0/879438384 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e90102e00 0x7f2e900702c0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f2e84008d80 tx=0x7f2e84009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:06:05.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.459+0000 7f2e8cc00700 1 -- 172.21.15.123:0/879438384 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e8400e030 con 0x7f2e90102e00 2024-03-30T05:06:05.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.459+0000 7f2e8cc00700 1 -- 172.21.15.123:0/879438384 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2e84013de0 con 0x7f2e90102e00 2024-03-30T05:06:05.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.459+0000 7f2e97400700 1 -- 172.21.15.123:0/879438384 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e90071120 con 0x7f2e90102e00 2024-03-30T05:06:05.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.459+0000 7f2e8cc00700 1 -- 172.21.15.123:0/879438384 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e8401cd60 con 0x7f2e90102e00 2024-03-30T05:06:05.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.459+0000 7f2e97400700 1 -- 172.21.15.123:0/879438384 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e9011a170 con 0x7f2e90102e00 2024-03-30T05:06:05.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.460+0000 7f2e97400700 1 -- 172.21.15.123:0/879438384 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e900fe550 con 0x7f2e90102e00 2024-03-30T05:06:05.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.461+0000 7f2e8cc00700 1 -- 172.21.15.123:0/879438384 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f2e8401c590 con 0x7f2e90102e00 2024-03-30T05:06:05.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.462+0000 7f2e8cc00700 1 --2- 172.21.15.123:0/879438384 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2e7806cc30 0x7f2e7806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:06:05.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.462+0000 7f2e8cc00700 1 -- 172.21.15.123:0/879438384 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f2e8408d970 con 0x7f2e90102e00 2024-03-30T05:06:05.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.462+0000 7f2e8fe00700 1 --2- 172.21.15.123:0/879438384 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2e7806cc30 0x7f2e7806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:06:05.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.463+0000 7f2e8fe00700 1 --2- 172.21.15.123:0/879438384 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2e7806cc30 0x7f2e7806f0e0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c004600 tx=0x7f2e7c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:06:05.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.467+0000 7f2e8cc00700 1 -- 172.21.15.123:0/879438384 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2e84057930 con 0x7f2e90102e00 2024-03-30T05:06:05.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.686+0000 7f2e97400700 1 -- 172.21.15.123:0/879438384 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2e900719b0 con 0x7f2e7806cc30 2024-03-30T05:06:05.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.687+0000 7f2e8cc00700 1 -- 172.21.15.123:0/879438384 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f2e900719b0 con 0x7f2e7806cc30 2024-03-30T05:06:05.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.689+0000 7f2e97400700 1 -- 172.21.15.123:0/879438384 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2e7806cc30 msgr2=0x7f2e7806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:05.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.689+0000 7f2e97400700 1 --2- 172.21.15.123:0/879438384 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2e7806cc30 0x7f2e7806f0e0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c004600 tx=0x7f2e7c00b040 comp rx=0 tx=0).stop 2024-03-30T05:06:05.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.689+0000 7f2e97400700 1 -- 172.21.15.123:0/879438384 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e90102e00 msgr2=0x7f2e900702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:05.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.689+0000 7f2e97400700 1 --2- 172.21.15.123:0/879438384 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e90102e00 0x7f2e900702c0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f2e84008d80 tx=0x7f2e84009e30 comp rx=0 tx=0).stop 2024-03-30T05:06:05.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.689+0000 7f2e97400700 1 -- 172.21.15.123:0/879438384 shutdown_connections 2024-03-30T05:06:05.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.689+0000 7f2e97400700 1 --2- 172.21.15.123:0/879438384 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2e7806cc30 0x7f2e7806f0e0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:05.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.689+0000 7f2e97400700 1 --2- 172.21.15.123:0/879438384 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e90102e00 0x7f2e900702c0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:05.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.689+0000 7f2e97400700 1 --2- 172.21.15.123:0/879438384 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2e901037a0 0x7f2e90070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:05.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.689+0000 7f2e97400700 1 -- 172.21.15.123:0/879438384 >> 172.21.15.123:0/879438384 conn(0x7f2e900f89b0 msgr2=0x7f2e900f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:06:05.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.690+0000 7f2e97400700 1 -- 172.21.15.123:0/879438384 shutdown_connections 2024-03-30T05:06:05.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.690+0000 7f2e97400700 1 -- 172.21.15.123:0/879438384 wait complete. 2024-03-30T05:06:05.702 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T05:06:05.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.824+0000 7f68ab200700 1 -- 172.21.15.123:0/319007820 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68a4102e00 msgr2=0x7f68a41031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:05.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.824+0000 7f68ab200700 1 --2- 172.21.15.123:0/319007820 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68a4102e00 0x7f68a41031d0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f6894004940 tx=0x7f6894009e30 comp rx=0 tx=0).stop 2024-03-30T05:06:05.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.824+0000 7f68ab200700 1 -- 172.21.15.123:0/319007820 shutdown_connections 2024-03-30T05:06:05.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.824+0000 7f68ab200700 1 --2- 172.21.15.123:0/319007820 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f68a41037a0 0x7f68a40fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:05.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.824+0000 7f68ab200700 1 --2- 172.21.15.123:0/319007820 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68a4102e00 0x7f68a41031d0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:05.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.824+0000 7f68ab200700 1 -- 172.21.15.123:0/319007820 >> 172.21.15.123:0/319007820 conn(0x7f68a40f89b0 msgr2=0x7f68a40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:06:05.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.824+0000 7f68ab200700 1 -- 172.21.15.123:0/319007820 shutdown_connections 2024-03-30T05:06:05.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.824+0000 7f68ab200700 1 -- 172.21.15.123:0/319007820 wait complete. 2024-03-30T05:06:05.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.825+0000 7f68ab200700 1 Processor -- start 2024-03-30T05:06:05.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.825+0000 7f68ab200700 1 -- start start 2024-03-30T05:06:05.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.825+0000 7f68ab200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f68a4102e00 0x7f68a4115d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:06:05.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.825+0000 7f68ab200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68a41037a0 0x7f68a4116250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:06:05.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.825+0000 7f68ab200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a41168a0 con 0x7f68a41037a0 2024-03-30T05:06:05.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.825+0000 7f68ab200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a410fc70 con 0x7f68a4102e00 2024-03-30T05:06:05.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.826+0000 7f68a3400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68a41037a0 0x7f68a4116250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:06:05.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.826+0000 7f68a3400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68a41037a0 0x7f68a4116250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:32980/0 (socket says 172.21.15.123:32980) 2024-03-30T05:06:05.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.826+0000 7f68a3400700 1 -- 172.21.15.123:0/3536059832 learned_addr learned my addr 172.21.15.123:0/3536059832 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:06:05.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.826+0000 7f68a3e00700 1 --2- 172.21.15.123:0/3536059832 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f68a4102e00 0x7f68a4115d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:06:05.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.826+0000 7f68a3400700 1 -- 172.21.15.123:0/3536059832 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f68a4102e00 msgr2=0x7f68a4115d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:05.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.826+0000 7f68a3400700 1 --2- 172.21.15.123:0/3536059832 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f68a4102e00 0x7f68a4115d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:05.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.826+0000 7f68a3400700 1 -- 172.21.15.123:0/3536059832 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6894005040 con 0x7f68a41037a0 2024-03-30T05:06:05.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.826+0000 7f68a3e00700 1 --2- 172.21.15.123:0/3536059832 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f68a4102e00 0x7f68a4115d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T05:06:05.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.826+0000 7f68a3400700 1 --2- 172.21.15.123:0/3536059832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68a41037a0 0x7f68a4116250 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f6890002a50 tx=0x7f6890004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:06:05.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.826+0000 7f689be00700 1 -- 172.21.15.123:0/3536059832 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6890007cc0 con 0x7f68a41037a0 2024-03-30T05:06:05.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.826+0000 7f689be00700 1 -- 172.21.15.123:0/3536059832 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f689000f040 con 0x7f68a41037a0 2024-03-30T05:06:05.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.826+0000 7f68ab200700 1 -- 172.21.15.123:0/3536059832 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68a410ff50 con 0x7f68a41037a0 2024-03-30T05:06:05.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.827+0000 7f689be00700 1 -- 172.21.15.123:0/3536059832 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f689000d9a0 con 0x7f68a41037a0 2024-03-30T05:06:05.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.827+0000 7f68ab200700 1 -- 172.21.15.123:0/3536059832 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68a4110510 con 0x7f68a41037a0 2024-03-30T05:06:05.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.828+0000 7f68ab200700 1 -- 172.21.15.123:0/3536059832 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68a40fe550 con 0x7f68a41037a0 2024-03-30T05:06:05.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.829+0000 7f689be00700 1 -- 172.21.15.123:0/3536059832 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f689000dc60 con 0x7f68a41037a0 2024-03-30T05:06:05.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.830+0000 7f689be00700 1 --2- 172.21.15.123:0/3536059832 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f688806cc30 0x7f688806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:06:05.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.830+0000 7f689be00700 1 -- 172.21.15.123:0/3536059832 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f689008be70 con 0x7f68a41037a0 2024-03-30T05:06:05.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.830+0000 7f68a3e00700 1 --2- 172.21.15.123:0/3536059832 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f688806cc30 0x7f688806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:06:05.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.831+0000 7f68a3e00700 1 --2- 172.21.15.123:0/3536059832 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f688806cc30 0x7f688806f0e0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f689400b3b0 tx=0x7f6894009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:06:05.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:05.834+0000 7f689be00700 1 -- 172.21.15.123:0/3536059832 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f68900598c0 con 0x7f68a41037a0 2024-03-30T05:06:06.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.055+0000 7f68ab200700 1 -- 172.21.15.123:0/3536059832 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f68a4109e30 con 0x7f688806cc30 2024-03-30T05:06:06.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.056+0000 7f689be00700 1 -- 172.21.15.123:0/3536059832 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f68a4109e30 con 0x7f688806cc30 2024-03-30T05:06:06.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.058+0000 7f68ab200700 1 -- 172.21.15.123:0/3536059832 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f688806cc30 msgr2=0x7f688806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:06.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.058+0000 7f68ab200700 1 --2- 172.21.15.123:0/3536059832 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f688806cc30 0x7f688806f0e0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f689400b3b0 tx=0x7f6894009e30 comp rx=0 tx=0).stop 2024-03-30T05:06:06.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.058+0000 7f68ab200700 1 -- 172.21.15.123:0/3536059832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68a41037a0 msgr2=0x7f68a4116250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:06.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.058+0000 7f68ab200700 1 --2- 172.21.15.123:0/3536059832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68a41037a0 0x7f68a4116250 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f6890002a50 tx=0x7f6890004600 comp rx=0 tx=0).stop 2024-03-30T05:06:06.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.059+0000 7f68ab200700 1 -- 172.21.15.123:0/3536059832 shutdown_connections 2024-03-30T05:06:06.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.059+0000 7f68ab200700 1 --2- 172.21.15.123:0/3536059832 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f688806cc30 0x7f688806f0e0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:06.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.059+0000 7f68ab200700 1 --2- 172.21.15.123:0/3536059832 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f68a4102e00 0x7f68a4115d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:06.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.059+0000 7f68ab200700 1 --2- 172.21.15.123:0/3536059832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68a41037a0 0x7f68a4116250 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:06.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.059+0000 7f68ab200700 1 -- 172.21.15.123:0/3536059832 >> 172.21.15.123:0/3536059832 conn(0x7f68a40f89b0 msgr2=0x7f68a40704f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:06:06.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.059+0000 7f68ab200700 1 -- 172.21.15.123:0/3536059832 shutdown_connections 2024-03-30T05:06:06.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.059+0000 7f68ab200700 1 -- 172.21.15.123:0/3536059832 wait complete. 2024-03-30T05:06:06.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.193+0000 7fac49a00700 1 -- 172.21.15.123:0/1719235780 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac441037a0 msgr2=0x7fac440fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:06.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.193+0000 7fac49a00700 1 --2- 172.21.15.123:0/1719235780 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac441037a0 0x7fac440fce10 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fac30004940 tx=0x7fac30009e30 comp rx=0 tx=0).stop 2024-03-30T05:06:06.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.193+0000 7fac49a00700 1 -- 172.21.15.123:0/1719235780 shutdown_connections 2024-03-30T05:06:06.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.193+0000 7fac49a00700 1 --2- 172.21.15.123:0/1719235780 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac441037a0 0x7fac440fce10 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:06.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.193+0000 7fac49a00700 1 --2- 172.21.15.123:0/1719235780 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac44102e00 0x7fac441031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:06.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.193+0000 7fac49a00700 1 -- 172.21.15.123:0/1719235780 >> 172.21.15.123:0/1719235780 conn(0x7fac440f89b0 msgr2=0x7fac440fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:06:06.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.193+0000 7fac49a00700 1 -- 172.21.15.123:0/1719235780 shutdown_connections 2024-03-30T05:06:06.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.193+0000 7fac49a00700 1 -- 172.21.15.123:0/1719235780 wait complete. 2024-03-30T05:06:06.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.194+0000 7fac49a00700 1 Processor -- start 2024-03-30T05:06:06.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.194+0000 7fac49a00700 1 -- start start 2024-03-30T05:06:06.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.194+0000 7fac49a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac44102e00 0x7fac44115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:06:06.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.194+0000 7fac49a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac441037a0 0x7fac441161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:06:06.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.194+0000 7fac49a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac44116810 con 0x7fac441037a0 2024-03-30T05:06:06.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.194+0000 7fac49a00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac4410fbe0 con 0x7fac44102e00 2024-03-30T05:06:06.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.194+0000 7fac42000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac441037a0 0x7fac441161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:06:06.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.194+0000 7fac42000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac441037a0 0x7fac441161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:32994/0 (socket says 172.21.15.123:32994) 2024-03-30T05:06:06.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.195+0000 7fac42000700 1 -- 172.21.15.123:0/2500681835 learned_addr learned my addr 172.21.15.123:0/2500681835 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:06:06.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.195+0000 7fac42000700 1 -- 172.21.15.123:0/2500681835 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac44102e00 msgr2=0x7fac44115c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:06.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.195+0000 7fac42000700 1 --2- 172.21.15.123:0/2500681835 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac44102e00 0x7fac44115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:06.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.195+0000 7fac42000700 1 -- 172.21.15.123:0/2500681835 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac30005040 con 0x7fac441037a0 2024-03-30T05:06:06.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.195+0000 7fac42000700 1 --2- 172.21.15.123:0/2500681835 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac441037a0 0x7fac441161c0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fac30009fd0 tx=0x7fac30009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:06:06.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.195+0000 7fac3aa00700 1 -- 172.21.15.123:0/2500681835 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac3000e030 con 0x7fac441037a0 2024-03-30T05:06:06.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.195+0000 7fac3aa00700 1 -- 172.21.15.123:0/2500681835 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fac30013e00 con 0x7fac441037a0 2024-03-30T05:06:06.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.195+0000 7fac49a00700 1 -- 172.21.15.123:0/2500681835 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac4410fe60 con 0x7fac441037a0 2024-03-30T05:06:06.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.195+0000 7fac3aa00700 1 -- 172.21.15.123:0/2500681835 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac3001ccd0 con 0x7fac441037a0 2024-03-30T05:06:06.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.196+0000 7fac49a00700 1 -- 172.21.15.123:0/2500681835 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac44110420 con 0x7fac441037a0 2024-03-30T05:06:06.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.197+0000 7fac49a00700 1 -- 172.21.15.123:0/2500681835 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac440fe550 con 0x7fac441037a0 2024-03-30T05:06:06.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.197+0000 7fac3aa00700 1 -- 172.21.15.123:0/2500681835 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fac3001c500 con 0x7fac441037a0 2024-03-30T05:06:06.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.198+0000 7fac3aa00700 1 --2- 172.21.15.123:0/2500681835 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fac2806cc30 0x7fac2806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:06:06.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.198+0000 7fac3aa00700 1 -- 172.21.15.123:0/2500681835 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fac3008d8a0 con 0x7fac441037a0 2024-03-30T05:06:06.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.199+0000 7fac42a00700 1 --2- 172.21.15.123:0/2500681835 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fac2806cc30 0x7fac2806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:06:06.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.199+0000 7fac42a00700 1 --2- 172.21.15.123:0/2500681835 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fac2806cc30 0x7fac2806f0e0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fac34009fd0 tx=0x7fac3400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:06:06.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.203+0000 7fac3aa00700 1 -- 172.21.15.123:0/2500681835 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fac3005b370 con 0x7fac441037a0 2024-03-30T05:06:06.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.414+0000 7fac49a00700 1 -- 172.21.15.123:0/2500681835 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fac44109da0 con 0x7fac2806cc30 2024-03-30T05:06:06.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.423+0000 7fac3aa00700 1 -- 172.21.15.123:0/2500681835 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fac44109da0 con 0x7fac2806cc30 2024-03-30T05:06:06.422 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T05:06:06.422 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (35m) 12m ago 47m 14.6M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T05:06:06.422 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (32m) 12m ago 48m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T05:06:06.422 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (32m) 12m ago 46m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T05:06:06.422 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (34m) 12m ago 47m 73.5M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T05:06:06.422 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (28m) 12m ago 41m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T05:06:06.422 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (28m) 12m ago 41m 16.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T05:06:06.422 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (12m) 12m ago 41m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 87b073a6c9ca 2024-03-30T05:06:06.422 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (27m) 12m ago 41m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T05:06:06.423 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (37m) 12m ago 49m 510M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T05:06:06.423 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (36m) 12m ago 46m 456M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T05:06:06.423 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (33m) 12m ago 49m 70.0M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T05:06:06.423 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (33m) 12m ago 46m 51.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T05:06:06.423 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (36m) 12m ago 48m 8636k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T05:06:06.423 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (35m) 12m ago 46m 8678k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T05:06:06.423 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (32m) 12m ago 45m 107M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T05:06:06.423 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (31m) 12m ago 45m 97.2M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T05:06:06.423 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (30m) 12m ago 44m 89.8M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T05:06:06.423 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (30m) 12m ago 44m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T05:06:06.423 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (29m) 12m ago 43m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T05:06:06.423 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (28m) 12m ago 43m 88.8M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T05:06:06.423 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (35m) 12m ago 47m 63.9M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T05:06:06.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.426+0000 7fac49a00700 1 -- 172.21.15.123:0/2500681835 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fac2806cc30 msgr2=0x7fac2806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:06.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.426+0000 7fac49a00700 1 --2- 172.21.15.123:0/2500681835 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fac2806cc30 0x7fac2806f0e0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fac34009fd0 tx=0x7fac3400b040 comp rx=0 tx=0).stop 2024-03-30T05:06:06.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.426+0000 7fac49a00700 1 -- 172.21.15.123:0/2500681835 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac441037a0 msgr2=0x7fac441161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:06.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.426+0000 7fac49a00700 1 --2- 172.21.15.123:0/2500681835 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac441037a0 0x7fac441161c0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fac30009fd0 tx=0x7fac30009e30 comp rx=0 tx=0).stop 2024-03-30T05:06:06.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.426+0000 7fac49a00700 1 -- 172.21.15.123:0/2500681835 shutdown_connections 2024-03-30T05:06:06.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.426+0000 7fac49a00700 1 --2- 172.21.15.123:0/2500681835 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fac2806cc30 0x7fac2806f0e0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:06.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.426+0000 7fac49a00700 1 --2- 172.21.15.123:0/2500681835 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fac44102e00 0x7fac44115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:06.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.426+0000 7fac49a00700 1 --2- 172.21.15.123:0/2500681835 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac441037a0 0x7fac441161c0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:06.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.426+0000 7fac49a00700 1 -- 172.21.15.123:0/2500681835 >> 172.21.15.123:0/2500681835 conn(0x7fac440f89b0 msgr2=0x7fac440f99a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:06:06.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.426+0000 7fac49a00700 1 -- 172.21.15.123:0/2500681835 shutdown_connections 2024-03-30T05:06:06.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.426+0000 7fac49a00700 1 -- 172.21.15.123:0/2500681835 wait complete. 2024-03-30T05:06:06.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.557+0000 7fdba0a00700 1 -- 172.21.15.123:0/3023286705 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb9c0656a0 msgr2=0x7fdb9c065af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:06.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.557+0000 7fdba0a00700 1 --2- 172.21.15.123:0/3023286705 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb9c0656a0 0x7fdb9c065af0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fdb88004940 tx=0x7fdb88009e30 comp rx=0 tx=0).stop 2024-03-30T05:06:06.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.557+0000 7fdba0a00700 1 -- 172.21.15.123:0/3023286705 shutdown_connections 2024-03-30T05:06:06.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.557+0000 7fdba0a00700 1 --2- 172.21.15.123:0/3023286705 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb9c0656a0 0x7fdb9c065af0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:06.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.557+0000 7fdba0a00700 1 --2- 172.21.15.123:0/3023286705 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdb9c102de0 0x7fdb9c1031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:06.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.557+0000 7fdba0a00700 1 -- 172.21.15.123:0/3023286705 >> 172.21.15.123:0/3023286705 conn(0x7fdb9c072bc0 msgr2=0x7fdb9c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:06:06.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.557+0000 7fdba0a00700 1 -- 172.21.15.123:0/3023286705 shutdown_connections 2024-03-30T05:06:06.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.557+0000 7fdba0a00700 1 -- 172.21.15.123:0/3023286705 wait complete. 2024-03-30T05:06:06.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.558+0000 7fdba0a00700 1 Processor -- start 2024-03-30T05:06:06.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.558+0000 7fdba0a00700 1 -- start start 2024-03-30T05:06:06.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.558+0000 7fdba0a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdb9c0656a0 0x7fdb9c115c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:06:06.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.559+0000 7fdba0a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb9c102de0 0x7fdb9c116160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:06:06.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.559+0000 7fdba0a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9c1167b0 con 0x7fdb9c102de0 2024-03-30T05:06:06.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.559+0000 7fdba0a00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9c10fba0 con 0x7fdb9c0656a0 2024-03-30T05:06:06.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.559+0000 7fdb99400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb9c102de0 0x7fdb9c116160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:06:06.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.559+0000 7fdb99400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb9c102de0 0x7fdb9c116160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33014/0 (socket says 172.21.15.123:33014) 2024-03-30T05:06:06.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.559+0000 7fdb99400700 1 -- 172.21.15.123:0/4291572443 learned_addr learned my addr 172.21.15.123:0/4291572443 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:06:06.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.559+0000 7fdb99e00700 1 --2- 172.21.15.123:0/4291572443 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdb9c0656a0 0x7fdb9c115c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:06:06.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.559+0000 7fdb99400700 1 -- 172.21.15.123:0/4291572443 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdb9c0656a0 msgr2=0x7fdb9c115c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:06.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.559+0000 7fdb99400700 1 --2- 172.21.15.123:0/4291572443 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdb9c0656a0 0x7fdb9c115c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:06.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.559+0000 7fdb99400700 1 -- 172.21.15.123:0/4291572443 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb88005040 con 0x7fdb9c102de0 2024-03-30T05:06:06.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.559+0000 7fdb99400700 1 --2- 172.21.15.123:0/4291572443 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb9c102de0 0x7fdb9c116160 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fdb88009390 tx=0x7fdb88009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:06:06.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.560+0000 7fdb92000700 1 -- 172.21.15.123:0/4291572443 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb8800e040 con 0x7fdb9c102de0 2024-03-30T05:06:06.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.560+0000 7fdb92000700 1 -- 172.21.15.123:0/4291572443 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdb88013420 con 0x7fdb9c102de0 2024-03-30T05:06:06.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.560+0000 7fdba0a00700 1 -- 172.21.15.123:0/4291572443 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb9c10fe20 con 0x7fdb9c102de0 2024-03-30T05:06:06.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.560+0000 7fdb92000700 1 -- 172.21.15.123:0/4291572443 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb8801c630 con 0x7fdb9c102de0 2024-03-30T05:06:06.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.560+0000 7fdba0a00700 1 -- 172.21.15.123:0/4291572443 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb9c1103e0 con 0x7fdb9c102de0 2024-03-30T05:06:06.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.561+0000 7fdba0a00700 1 -- 172.21.15.123:0/4291572443 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb9c106980 con 0x7fdb9c102de0 2024-03-30T05:06:06.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.562+0000 7fdb92000700 1 -- 172.21.15.123:0/4291572443 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fdb880139e0 con 0x7fdb9c102de0 2024-03-30T05:06:06.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.563+0000 7fdb92000700 1 --2- 172.21.15.123:0/4291572443 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fdb8006cc30 0x7fdb8006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:06:06.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.563+0000 7fdb92000700 1 -- 172.21.15.123:0/4291572443 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fdb8808d3f0 con 0x7fdb9c102de0 2024-03-30T05:06:06.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.563+0000 7fdb99e00700 1 --2- 172.21.15.123:0/4291572443 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fdb8006cc30 0x7fdb8006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:06:06.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.563+0000 7fdb99e00700 1 --2- 172.21.15.123:0/4291572443 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fdb8006cc30 0x7fdb8006f0e0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c007fd0 tx=0x7fdb8c009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:06:06.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.567+0000 7fdb92000700 1 -- 172.21.15.123:0/4291572443 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fdb8805ae40 con 0x7fdb9c102de0 2024-03-30T05:06:06.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.853+0000 7fdba0a00700 1 -- 172.21.15.123:0/4291572443 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fdb9c111200 con 0x7fdb9c102de0 2024-03-30T05:06:06.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.853+0000 7fdb92000700 1 -- 172.21.15.123:0/4291572443 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7fdb8805a9d0 con 0x7fdb9c102de0 2024-03-30T05:06:06.852 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T05:06:06.852 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T05:06:06.852 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T05:06:06.853 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:06:06.853 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T05:06:06.853 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T05:06:06.853 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:06:06.853 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T05:06:06.854 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T05:06:06.854 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:06:06.854 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T05:06:06.854 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 4 2024-03-30T05:06:06.854 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:06:06.854 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T05:06:06.855 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 14 2024-03-30T05:06:06.855 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T05:06:06.855 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T05:06:06.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.856+0000 7fdba0a00700 1 -- 172.21.15.123:0/4291572443 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fdb8006cc30 msgr2=0x7fdb8006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:06.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.856+0000 7fdba0a00700 1 --2- 172.21.15.123:0/4291572443 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fdb8006cc30 0x7fdb8006f0e0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c007fd0 tx=0x7fdb8c009040 comp rx=0 tx=0).stop 2024-03-30T05:06:06.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.856+0000 7fdba0a00700 1 -- 172.21.15.123:0/4291572443 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb9c102de0 msgr2=0x7fdb9c116160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:06.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.856+0000 7fdba0a00700 1 --2- 172.21.15.123:0/4291572443 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb9c102de0 0x7fdb9c116160 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fdb88009390 tx=0x7fdb88009e30 comp rx=0 tx=0).stop 2024-03-30T05:06:06.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.856+0000 7fdba0a00700 1 -- 172.21.15.123:0/4291572443 shutdown_connections 2024-03-30T05:06:06.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.856+0000 7fdba0a00700 1 --2- 172.21.15.123:0/4291572443 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fdb8006cc30 0x7fdb8006f0e0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:06.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.856+0000 7fdba0a00700 1 --2- 172.21.15.123:0/4291572443 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdb9c0656a0 0x7fdb9c115c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:06.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.856+0000 7fdba0a00700 1 --2- 172.21.15.123:0/4291572443 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb9c102de0 0x7fdb9c116160 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:06.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.856+0000 7fdba0a00700 1 -- 172.21.15.123:0/4291572443 >> 172.21.15.123:0/4291572443 conn(0x7fdb9c072bc0 msgr2=0x7fdb9c103b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:06:06.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.856+0000 7fdba0a00700 1 -- 172.21.15.123:0/4291572443 shutdown_connections 2024-03-30T05:06:06.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.856+0000 7fdba0a00700 1 -- 172.21.15.123:0/4291572443 wait complete. 2024-03-30T05:06:06.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.989+0000 7f9d1d400700 1 -- 172.21.15.123:0/3020847988 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d18102e00 msgr2=0x7f9d181031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:06.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.989+0000 7f9d1d400700 1 --2- 172.21.15.123:0/3020847988 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d18102e00 0x7f9d181031d0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f9d08004940 tx=0x7f9d08009e30 comp rx=0 tx=0).stop 2024-03-30T05:06:06.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.989+0000 7f9d1d400700 1 -- 172.21.15.123:0/3020847988 shutdown_connections 2024-03-30T05:06:06.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.989+0000 7f9d1d400700 1 --2- 172.21.15.123:0/3020847988 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9d181037a0 0x7f9d180fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:06.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.989+0000 7f9d1d400700 1 --2- 172.21.15.123:0/3020847988 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d18102e00 0x7f9d181031d0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:06.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.989+0000 7f9d1d400700 1 -- 172.21.15.123:0/3020847988 >> 172.21.15.123:0/3020847988 conn(0x7f9d180f89b0 msgr2=0x7f9d180fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:06:06.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.989+0000 7f9d1d400700 1 -- 172.21.15.123:0/3020847988 shutdown_connections 2024-03-30T05:06:06.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.989+0000 7f9d1d400700 1 -- 172.21.15.123:0/3020847988 wait complete. 2024-03-30T05:06:06.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.990+0000 7f9d1d400700 1 Processor -- start 2024-03-30T05:06:06.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.991+0000 7f9d1d400700 1 -- start start 2024-03-30T05:06:06.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.991+0000 7f9d1d400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9d18102e00 0x7f9d18113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:06:06.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.991+0000 7f9d1d400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d181037a0 0x7f9d18114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:06:06.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.991+0000 7f9d1d400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d181146b0 con 0x7f9d181037a0 2024-03-30T05:06:06.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.991+0000 7f9d1d400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d1810daa0 con 0x7f9d18102e00 2024-03-30T05:06:06.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.991+0000 7f9d15e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d181037a0 0x7f9d18114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:06:06.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.991+0000 7f9d15e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d181037a0 0x7f9d18114060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33036/0 (socket says 172.21.15.123:33036) 2024-03-30T05:06:06.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.991+0000 7f9d15e00700 1 -- 172.21.15.123:0/3787618324 learned_addr learned my addr 172.21.15.123:0/3787618324 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:06:06.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.991+0000 7f9d16800700 1 --2- 172.21.15.123:0/3787618324 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9d18102e00 0x7f9d18113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:06:06.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.992+0000 7f9d15e00700 1 -- 172.21.15.123:0/3787618324 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9d18102e00 msgr2=0x7f9d18113b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:06.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.992+0000 7f9d15e00700 1 --2- 172.21.15.123:0/3787618324 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9d18102e00 0x7f9d18113b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:06.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.992+0000 7f9d15e00700 1 -- 172.21.15.123:0/3787618324 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d08005040 con 0x7f9d181037a0 2024-03-30T05:06:06.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.992+0000 7f9d15e00700 1 --2- 172.21.15.123:0/3787618324 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d181037a0 0x7f9d18114060 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f9d04002a50 tx=0x7f9d04004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:06:06.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.992+0000 7f9d0ea00700 1 -- 172.21.15.123:0/3787618324 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d04007cc0 con 0x7f9d181037a0 2024-03-30T05:06:06.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.992+0000 7f9d0ea00700 1 -- 172.21.15.123:0/3787618324 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9d0400f040 con 0x7f9d181037a0 2024-03-30T05:06:06.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.992+0000 7f9d1d400700 1 -- 172.21.15.123:0/3787618324 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d1810dd80 con 0x7f9d181037a0 2024-03-30T05:06:06.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.992+0000 7f9d0ea00700 1 -- 172.21.15.123:0/3787618324 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d0400d920 con 0x7f9d181037a0 2024-03-30T05:06:06.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.992+0000 7f9d1d400700 1 -- 172.21.15.123:0/3787618324 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d1810e340 con 0x7f9d181037a0 2024-03-30T05:06:06.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.993+0000 7f9d1d400700 1 -- 172.21.15.123:0/3787618324 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d180fe550 con 0x7f9d181037a0 2024-03-30T05:06:06.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.994+0000 7f9d0ea00700 1 -- 172.21.15.123:0/3787618324 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f9d04009680 con 0x7f9d181037a0 2024-03-30T05:06:06.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.995+0000 7f9d0ea00700 1 --2- 172.21.15.123:0/3787618324 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9cfc06cc30 0x7f9cfc06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:06:06.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.995+0000 7f9d0ea00700 1 -- 172.21.15.123:0/3787618324 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f9d0408bd10 con 0x7f9d181037a0 2024-03-30T05:06:06.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.996+0000 7f9d16800700 1 --2- 172.21.15.123:0/3787618324 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9cfc06cc30 0x7f9cfc06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:06:06.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:06.996+0000 7f9d16800700 1 --2- 172.21.15.123:0/3787618324 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9cfc06cc30 0x7f9cfc06f0e0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f9d08009fd0 tx=0x7f9d08009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:06:06.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.000+0000 7f9d0ea00700 1 -- 172.21.15.123:0/3787618324 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9d04059760 con 0x7f9d181037a0 2024-03-30T05:06:07.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:06:06 smithi135 ceph-mon[106011]: pgmap v1048: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:07.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:06:06 smithi135 ceph-mon[106011]: from='client.35888 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:06:07.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:06:06 smithi135 ceph-mon[106011]: from='client.35892 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:06:07.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:06:06 smithi123 ceph-mon[139369]: pgmap v1048: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:07.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:06:06 smithi123 ceph-mon[139369]: from='client.35888 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:06:07.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:06:06 smithi123 ceph-mon[139369]: from='client.35892 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:06:07.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.247+0000 7f9d1d400700 1 -- 172.21.15.123:0/3787618324 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f9d18109df0 con 0x7f9d181037a0 2024-03-30T05:06:07.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.247+0000 7f9d0ea00700 1 -- 172.21.15.123:0/3787618324 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 34 v34) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f9d040592f0 con 0x7f9d181037a0 2024-03-30T05:06:07.246 INFO:teuthology.orchestra.run.smithi123.stdout:e34 2024-03-30T05:06:07.246 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T05:06:07.246 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T05:06:07.246 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T05:06:07.247 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:06:07.247 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T05:06:07.247 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T05:06:07.247 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T05:06:07.247 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T05:06:07.247 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T05:06:07.247 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T05:06:07.248 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T05:06:07.248 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T05:06:07.248 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T05:06:07.248 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T05:06:07.248 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T05:06:07.248 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T05:06:07.249 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T05:06:07.249 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T05:06:07.249 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T05:06:07.249 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T05:06:07.249 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T05:06:07.250 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T05:06:07.250 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T05:06:07.250 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T05:06:07.250 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T05:06:07.250 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T05:06:07.250 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T05:06:07.250 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T05:06:07.251 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T05:06:07.251 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T05:06:07.251 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T05:06:07.251 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T05:06:07.251 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:06:07.251 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:06:07.251 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T05:06:07.252 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:06:07.252 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:06:07.252 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:06:07.252 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:06:07.252 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44975} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/3788988014,v1:172.21.15.135:6827/3788988014] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:06:07.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.250+0000 7f9d1d400700 1 -- 172.21.15.123:0/3787618324 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9cfc06cc30 msgr2=0x7f9cfc06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:07.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.250+0000 7f9d1d400700 1 --2- 172.21.15.123:0/3787618324 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9cfc06cc30 0x7f9cfc06f0e0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f9d08009fd0 tx=0x7f9d08009e30 comp rx=0 tx=0).stop 2024-03-30T05:06:07.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.250+0000 7f9d1d400700 1 -- 172.21.15.123:0/3787618324 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d181037a0 msgr2=0x7f9d18114060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:07.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.250+0000 7f9d1d400700 1 --2- 172.21.15.123:0/3787618324 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d181037a0 0x7f9d18114060 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f9d04002a50 tx=0x7f9d04004600 comp rx=0 tx=0).stop 2024-03-30T05:06:07.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.250+0000 7f9d1d400700 1 -- 172.21.15.123:0/3787618324 shutdown_connections 2024-03-30T05:06:07.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.250+0000 7f9d1d400700 1 --2- 172.21.15.123:0/3787618324 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9cfc06cc30 0x7f9cfc06f0e0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:07.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.250+0000 7f9d1d400700 1 --2- 172.21.15.123:0/3787618324 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9d18102e00 0x7f9d18113b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:07.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.250+0000 7f9d1d400700 1 --2- 172.21.15.123:0/3787618324 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d181037a0 0x7f9d18114060 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:07.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.250+0000 7f9d1d400700 1 -- 172.21.15.123:0/3787618324 >> 172.21.15.123:0/3787618324 conn(0x7f9d180f89b0 msgr2=0x7f9d180f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:06:07.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.250+0000 7f9d1d400700 1 -- 172.21.15.123:0/3787618324 shutdown_connections 2024-03-30T05:06:07.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.250+0000 7f9d1d400700 1 -- 172.21.15.123:0/3787618324 wait complete. 2024-03-30T05:06:07.255 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 34 2024-03-30T05:06:07.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.383+0000 7fa521200700 1 -- 172.21.15.123:0/2433069902 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa51c1037a0 msgr2=0x7fa51c0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:07.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.383+0000 7fa521200700 1 --2- 172.21.15.123:0/2433069902 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa51c1037a0 0x7fa51c0fce10 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fa508004940 tx=0x7fa508009e30 comp rx=0 tx=0).stop 2024-03-30T05:06:07.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.383+0000 7fa521200700 1 -- 172.21.15.123:0/2433069902 shutdown_connections 2024-03-30T05:06:07.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.383+0000 7fa521200700 1 --2- 172.21.15.123:0/2433069902 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa51c1037a0 0x7fa51c0fce10 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:07.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.383+0000 7fa521200700 1 --2- 172.21.15.123:0/2433069902 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa51c102e00 0x7fa51c1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:07.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.383+0000 7fa521200700 1 -- 172.21.15.123:0/2433069902 >> 172.21.15.123:0/2433069902 conn(0x7fa51c0f89b0 msgr2=0x7fa51c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:06:07.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.383+0000 7fa521200700 1 -- 172.21.15.123:0/2433069902 shutdown_connections 2024-03-30T05:06:07.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.383+0000 7fa521200700 1 -- 172.21.15.123:0/2433069902 wait complete. 2024-03-30T05:06:07.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.384+0000 7fa521200700 1 Processor -- start 2024-03-30T05:06:07.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.385+0000 7fa521200700 1 -- start start 2024-03-30T05:06:07.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.385+0000 7fa521200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa51c102e00 0x7fa51c113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:06:07.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.385+0000 7fa521200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa51c1037a0 0x7fa51c113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:06:07.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.385+0000 7fa521200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa51c114600 con 0x7fa51c1037a0 2024-03-30T05:06:07.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.385+0000 7fa521200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa51c10d9f0 con 0x7fa51c102e00 2024-03-30T05:06:07.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.385+0000 7fa519400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa51c1037a0 0x7fa51c113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:06:07.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.385+0000 7fa519400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa51c1037a0 0x7fa51c113fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33048/0 (socket says 172.21.15.123:33048) 2024-03-30T05:06:07.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.386+0000 7fa519400700 1 -- 172.21.15.123:0/3581347098 learned_addr learned my addr 172.21.15.123:0/3581347098 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:06:07.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.386+0000 7fa519e00700 1 --2- 172.21.15.123:0/3581347098 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa51c102e00 0x7fa51c113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:06:07.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.386+0000 7fa519400700 1 -- 172.21.15.123:0/3581347098 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa51c102e00 msgr2=0x7fa51c113a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:07.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.386+0000 7fa519400700 1 --2- 172.21.15.123:0/3581347098 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa51c102e00 0x7fa51c113a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:07.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.386+0000 7fa519400700 1 -- 172.21.15.123:0/3581347098 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa508005040 con 0x7fa51c1037a0 2024-03-30T05:06:07.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.386+0000 7fa519400700 1 --2- 172.21.15.123:0/3581347098 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa51c1037a0 0x7fa51c113fb0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fa508008d50 tx=0x7fa508009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:06:07.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.386+0000 7fa512000700 1 -- 172.21.15.123:0/3581347098 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa50800e030 con 0x7fa51c1037a0 2024-03-30T05:06:07.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.386+0000 7fa512000700 1 -- 172.21.15.123:0/3581347098 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa5080093b0 con 0x7fa51c1037a0 2024-03-30T05:06:07.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.386+0000 7fa521200700 1 -- 172.21.15.123:0/3581347098 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa51c10dc70 con 0x7fa51c1037a0 2024-03-30T05:06:07.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.386+0000 7fa512000700 1 -- 172.21.15.123:0/3581347098 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa50801cbb0 con 0x7fa51c1037a0 2024-03-30T05:06:07.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.387+0000 7fa521200700 1 -- 172.21.15.123:0/3581347098 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa51c10e230 con 0x7fa51c1037a0 2024-03-30T05:06:07.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.388+0000 7fa521200700 1 -- 172.21.15.123:0/3581347098 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa51c0fe550 con 0x7fa51c1037a0 2024-03-30T05:06:07.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.389+0000 7fa512000700 1 -- 172.21.15.123:0/3581347098 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fa508013930 con 0x7fa51c1037a0 2024-03-30T05:06:07.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.389+0000 7fa512000700 1 --2- 172.21.15.123:0/3581347098 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa50006cc30 0x7fa50006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:06:07.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.390+0000 7fa512000700 1 -- 172.21.15.123:0/3581347098 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa50808d850 con 0x7fa51c1037a0 2024-03-30T05:06:07.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.390+0000 7fa519e00700 1 --2- 172.21.15.123:0/3581347098 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa50006cc30 0x7fa50006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:06:07.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.390+0000 7fa519e00700 1 --2- 172.21.15.123:0/3581347098 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa50006cc30 0x7fa50006f0e0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fa50c009b80 tx=0x7fa50c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:06:07.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.394+0000 7fa512000700 1 -- 172.21.15.123:0/3581347098 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa50805b2a0 con 0x7fa51c1037a0 2024-03-30T05:06:07.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.613+0000 7fa521200700 1 -- 172.21.15.123:0/3581347098 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa51c109da0 con 0x7fa50006cc30 2024-03-30T05:06:07.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.615+0000 7fa512000700 1 -- 172.21.15.123:0/3581347098 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa51c109da0 con 0x7fa50006cc30 2024-03-30T05:06:07.613 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T05:06:07.613 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T05:06:07.613 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T05:06:07.613 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T05:06:07.613 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T05:06:07.613 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T05:06:07.613 INFO:teuthology.orchestra.run.smithi123.stdout: "mds", 2024-03-30T05:06:07.613 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T05:06:07.613 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T05:06:07.613 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T05:06:07.613 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T05:06:07.614 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "16/21 daemons upgraded", 2024-03-30T05:06:07.614 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T05:06:07.614 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T05:06:07.614 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T05:06:07.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.617+0000 7fa521200700 1 -- 172.21.15.123:0/3581347098 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa50006cc30 msgr2=0x7fa50006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:07.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.617+0000 7fa521200700 1 --2- 172.21.15.123:0/3581347098 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa50006cc30 0x7fa50006f0e0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fa50c009b80 tx=0x7fa50c00b040 comp rx=0 tx=0).stop 2024-03-30T05:06:07.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.617+0000 7fa521200700 1 -- 172.21.15.123:0/3581347098 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa51c1037a0 msgr2=0x7fa51c113fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:07.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.617+0000 7fa521200700 1 --2- 172.21.15.123:0/3581347098 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa51c1037a0 0x7fa51c113fb0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fa508008d50 tx=0x7fa508009e30 comp rx=0 tx=0).stop 2024-03-30T05:06:07.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.617+0000 7fa521200700 1 -- 172.21.15.123:0/3581347098 shutdown_connections 2024-03-30T05:06:07.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.617+0000 7fa521200700 1 --2- 172.21.15.123:0/3581347098 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fa50006cc30 0x7fa50006f0e0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:07.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.617+0000 7fa521200700 1 --2- 172.21.15.123:0/3581347098 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa51c102e00 0x7fa51c113a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:07.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.617+0000 7fa521200700 1 --2- 172.21.15.123:0/3581347098 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa51c1037a0 0x7fa51c113fb0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:07.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.617+0000 7fa521200700 1 -- 172.21.15.123:0/3581347098 >> 172.21.15.123:0/3581347098 conn(0x7fa51c0f89b0 msgr2=0x7fa51c0f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:06:07.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.617+0000 7fa521200700 1 -- 172.21.15.123:0/3581347098 shutdown_connections 2024-03-30T05:06:07.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.617+0000 7fa521200700 1 -- 172.21.15.123:0/3581347098 wait complete. 2024-03-30T05:06:07.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.751+0000 7f3f6c000700 1 -- 172.21.15.123:0/496788180 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f640fe830 msgr2=0x7f3f64106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:07.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.751+0000 7f3f6c000700 1 --2- 172.21.15.123:0/496788180 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f640fe830 0x7f3f64106f00 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f3f50004940 tx=0x7f3f50009e30 comp rx=0 tx=0).stop 2024-03-30T05:06:07.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.751+0000 7f3f6c000700 1 -- 172.21.15.123:0/496788180 shutdown_connections 2024-03-30T05:06:07.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.751+0000 7f3f6c000700 1 --2- 172.21.15.123:0/496788180 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f640fe830 0x7f3f64106f00 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:07.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.751+0000 7f3f6c000700 1 --2- 172.21.15.123:0/496788180 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3f640fde90 0x7f3f640fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:07.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.751+0000 7f3f6c000700 1 -- 172.21.15.123:0/496788180 >> 172.21.15.123:0/496788180 conn(0x7f3f64072bc0 msgr2=0x7f3f64072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:06:07.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.751+0000 7f3f6c000700 1 -- 172.21.15.123:0/496788180 shutdown_connections 2024-03-30T05:06:07.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.751+0000 7f3f6c000700 1 -- 172.21.15.123:0/496788180 wait complete. 2024-03-30T05:06:07.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.752+0000 7f3f6c000700 1 Processor -- start 2024-03-30T05:06:07.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.752+0000 7f3f6c000700 1 -- start start 2024-03-30T05:06:07.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.753+0000 7f3f6c000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f640fde90 0x7f3f64115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:06:07.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.753+0000 7f3f6c000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3f640fe830 0x7f3f641161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:06:07.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.753+0000 7f3f6c000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f64116800 con 0x7f3f640fde90 2024-03-30T05:06:07.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.753+0000 7f3f6c000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6410fbd0 con 0x7f3f640fe830 2024-03-30T05:06:07.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.753+0000 7f3f69600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f640fde90 0x7f3f64115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:06:07.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.753+0000 7f3f69600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f640fde90 0x7f3f64115c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33070/0 (socket says 172.21.15.123:33070) 2024-03-30T05:06:07.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.753+0000 7f3f69600700 1 -- 172.21.15.123:0/548748516 learned_addr learned my addr 172.21.15.123:0/548748516 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:06:07.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.753+0000 7f3f68c00700 1 --2- 172.21.15.123:0/548748516 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3f640fe830 0x7f3f641161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:06:07.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.753+0000 7f3f69600700 1 -- 172.21.15.123:0/548748516 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3f640fe830 msgr2=0x7f3f641161b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:07.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.753+0000 7f3f69600700 1 --2- 172.21.15.123:0/548748516 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3f640fe830 0x7f3f641161b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:07.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.753+0000 7f3f69600700 1 -- 172.21.15.123:0/548748516 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f50005040 con 0x7f3f640fde90 2024-03-30T05:06:07.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.754+0000 7f3f69600700 1 --2- 172.21.15.123:0/548748516 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f640fde90 0x7f3f64115c70 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f3f5800eb40 tx=0x7f3f5800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:06:07.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.754+0000 7f3f61600700 1 -- 172.21.15.123:0/548748516 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f58007cc0 con 0x7f3f640fde90 2024-03-30T05:06:07.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.754+0000 7f3f61600700 1 -- 172.21.15.123:0/548748516 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3f5800f040 con 0x7f3f640fde90 2024-03-30T05:06:07.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.754+0000 7f3f61600700 1 -- 172.21.15.123:0/548748516 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f5800c990 con 0x7f3f640fde90 2024-03-30T05:06:07.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.754+0000 7f3f6c000700 1 -- 172.21.15.123:0/548748516 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f6410fe50 con 0x7f3f640fde90 2024-03-30T05:06:07.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.754+0000 7f3f6c000700 1 -- 172.21.15.123:0/548748516 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f64110390 con 0x7f3f640fde90 2024-03-30T05:06:07.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.755+0000 7f3f6c000700 1 -- 172.21.15.123:0/548748516 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f64106a20 con 0x7f3f640fde90 2024-03-30T05:06:07.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.756+0000 7f3f61600700 1 -- 172.21.15.123:0/548748516 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f3f5800caf0 con 0x7f3f640fde90 2024-03-30T05:06:07.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.757+0000 7f3f61600700 1 --2- 172.21.15.123:0/548748516 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3f4c06cc30 0x7f3f4c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:06:07.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.757+0000 7f3f61600700 1 -- 172.21.15.123:0/548748516 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f3f5808bc00 con 0x7f3f640fde90 2024-03-30T05:06:07.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.757+0000 7f3f68c00700 1 --2- 172.21.15.123:0/548748516 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3f4c06cc30 0x7f3f4c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:06:07.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.758+0000 7f3f68c00700 1 --2- 172.21.15.123:0/548748516 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3f4c06cc30 0x7f3f4c06f0e0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f3f64111510 tx=0x7f3f50009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:06:07.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:07.761+0000 7f3f61600700 1 -- 172.21.15.123:0/548748516 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3f58059700 con 0x7f3f640fde90 2024-03-30T05:06:08.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:08.048+0000 7f3f6c000700 1 -- 172.21.15.123:0/548748516 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f3f6404bed0 con 0x7f3f640fde90 2024-03-30T05:06:08.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:08.048+0000 7f3f61600700 1 -- 172.21.15.123:0/548748516 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f3f58019e10 con 0x7f3f640fde90 2024-03-30T05:06:08.048 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T05:06:08.048 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-30T05:06:08.048 INFO:teuthology.orchestra.run.smithi123.stdout: host smithi135 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi135/config 2024-03-30T05:06:08.048 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-30T05:06:08.048 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T05:06:08.048 INFO:teuthology.orchestra.run.smithi123.stdout:Traceback (most recent call last): 2024-03-30T05:06:08.048 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-30T05:06:08.048 INFO:teuthology.orchestra.run.smithi123.stdout: "__main__", mod_spec) 2024-03-30T05:06:08.048 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-30T05:06:08.048 INFO:teuthology.orchestra.run.smithi123.stdout: exec(code, run_globals) 2024-03-30T05:06:08.049 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10889, in 2024-03-30T05:06:08.049 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10877, in main 2024-03-30T05:06:08.049 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-30T05:06:08.049 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-30T05:06:08.049 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-30T05:06:08.049 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-30T05:06:08.049 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-03-30T05:06:08.049 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-30T05:06:08.049 INFO:teuthology.orchestra.run.smithi123.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T05:06:08.049 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T05:06:08.049 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T05:06:08.049 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T05:06:08.049 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T05:06:08.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:08.051+0000 7f3f6c000700 1 -- 172.21.15.123:0/548748516 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3f4c06cc30 msgr2=0x7f3f4c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:08.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:08.051+0000 7f3f6c000700 1 --2- 172.21.15.123:0/548748516 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3f4c06cc30 0x7f3f4c06f0e0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f3f64111510 tx=0x7f3f50009e30 comp rx=0 tx=0).stop 2024-03-30T05:06:08.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:08.051+0000 7f3f6c000700 1 -- 172.21.15.123:0/548748516 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f640fde90 msgr2=0x7f3f64115c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:08.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:08.051+0000 7f3f6c000700 1 --2- 172.21.15.123:0/548748516 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f640fde90 0x7f3f64115c70 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f3f5800eb40 tx=0x7f3f5800b040 comp rx=0 tx=0).stop 2024-03-30T05:06:08.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:08.051+0000 7f3f6c000700 1 -- 172.21.15.123:0/548748516 shutdown_connections 2024-03-30T05:06:08.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:08.051+0000 7f3f6c000700 1 --2- 172.21.15.123:0/548748516 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3f4c06cc30 0x7f3f4c06f0e0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:08.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:08.051+0000 7f3f6c000700 1 --2- 172.21.15.123:0/548748516 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f640fde90 0x7f3f64115c70 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:08.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:08.051+0000 7f3f6c000700 1 --2- 172.21.15.123:0/548748516 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3f640fe830 0x7f3f641161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:08.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:08.051+0000 7f3f6c000700 1 -- 172.21.15.123:0/548748516 >> 172.21.15.123:0/548748516 conn(0x7f3f64072bc0 msgr2=0x7f3f640fb6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:06:08.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:08.051+0000 7f3f6c000700 1 -- 172.21.15.123:0/548748516 shutdown_connections 2024-03-30T05:06:08.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:08.051+0000 7f3f6c000700 1 -- 172.21.15.123:0/548748516 wait complete. 2024-03-30T05:06:08.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:06:07 smithi135 ceph-mon[106011]: from='client.35896 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:06:08.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:06:07 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/4291572443' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:06:08.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:06:07 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3787618324' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T05:06:08.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:06:07 smithi123 ceph-mon[139369]: from='client.35896 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:06:08.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:06:07 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/4291572443' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:06:08.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:06:07 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3787618324' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T05:06:09.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:06:08 smithi135 ceph-mon[106011]: from='client.35908 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:06:09.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:06:08 smithi135 ceph-mon[106011]: pgmap v1049: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:09.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:06:08 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/548748516' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T05:06:09.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:06:08 smithi123 ceph-mon[139369]: from='client.35908 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:06:09.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:06:08 smithi123 ceph-mon[139369]: pgmap v1049: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:09.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:06:08 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/548748516' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T05:06:11.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:06:10 smithi135 ceph-mon[106011]: pgmap v1050: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:11.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:06:10 smithi123 ceph-mon[139369]: pgmap v1050: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:13.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:06:12 smithi123 ceph-mon[139369]: pgmap v1051: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:13.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:06:12 smithi135 ceph-mon[106011]: pgmap v1051: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:15.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:06:14 smithi123 ceph-mon[139369]: pgmap v1052: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:15.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:06:14 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:06:15.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:06:14 smithi135 ceph-mon[106011]: pgmap v1052: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:15.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:06:14 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:06:17.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:06:16 smithi123 ceph-mon[139369]: pgmap v1053: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:17.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:06:16 smithi135 ceph-mon[106011]: pgmap v1053: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:19.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:06:18 smithi123 ceph-mon[139369]: pgmap v1054: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:19.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:06:18 smithi135 ceph-mon[106011]: pgmap v1054: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:21.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:06:20 smithi123 ceph-mon[139369]: pgmap v1055: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:21.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:06:20 smithi135 ceph-mon[106011]: pgmap v1055: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:23.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:06:22 smithi123 ceph-mon[139369]: pgmap v1056: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:23.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:06:22 smithi135 ceph-mon[106011]: pgmap v1056: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:25.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:06:24 smithi123 ceph-mon[139369]: pgmap v1057: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:25.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:06:24 smithi135 ceph-mon[106011]: pgmap v1057: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:27.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:06:26 smithi123 ceph-mon[139369]: pgmap v1058: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:27.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:06:26 smithi135 ceph-mon[106011]: pgmap v1058: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:29.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:06:28 smithi123 ceph-mon[139369]: pgmap v1059: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:29.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:06:28 smithi135 ceph-mon[106011]: pgmap v1059: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:30.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:06:29 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:06:30.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:06:29 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:06:31.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:06:30 smithi123 ceph-mon[139369]: pgmap v1060: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:31.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:06:30 smithi135 ceph-mon[106011]: pgmap v1060: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:33.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:06:32 smithi123 ceph-mon[139369]: pgmap v1061: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:33.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:06:32 smithi135 ceph-mon[106011]: pgmap v1061: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:35.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:06:34 smithi123 ceph-mon[139369]: pgmap v1062: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:35.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:06:34 smithi135 ceph-mon[106011]: pgmap v1062: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:37.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:06:36 smithi123 ceph-mon[139369]: pgmap v1063: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:37.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:06:36 smithi135 ceph-mon[106011]: pgmap v1063: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:38.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.203+0000 7f53df200700 1 -- 172.21.15.123:0/4264726082 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53d8102e20 msgr2=0x7f53d81031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:38.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.203+0000 7f53df200700 1 --2- 172.21.15.123:0/4264726082 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53d8102e20 0x7f53d81031f0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f53c8004940 tx=0x7f53c8009e30 comp rx=0 tx=0).stop 2024-03-30T05:06:38.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.203+0000 7f53df200700 1 -- 172.21.15.123:0/4264726082 shutdown_connections 2024-03-30T05:06:38.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.203+0000 7f53df200700 1 --2- 172.21.15.123:0/4264726082 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53d81037c0 0x7f53d80fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:38.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.203+0000 7f53df200700 1 --2- 172.21.15.123:0/4264726082 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53d8102e20 0x7f53d81031f0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:38.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.203+0000 7f53df200700 1 -- 172.21.15.123:0/4264726082 >> 172.21.15.123:0/4264726082 conn(0x7f53d80f89d0 msgr2=0x7f53d80fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:06:38.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.203+0000 7f53df200700 1 -- 172.21.15.123:0/4264726082 shutdown_connections 2024-03-30T05:06:38.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.203+0000 7f53df200700 1 -- 172.21.15.123:0/4264726082 wait complete. 2024-03-30T05:06:38.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.204+0000 7f53df200700 1 Processor -- start 2024-03-30T05:06:38.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.204+0000 7f53df200700 1 -- start start 2024-03-30T05:06:38.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.205+0000 7f53df200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53d8102e20 0x7f53d8113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:06:38.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.205+0000 7f53df200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53d81037c0 0x7f53d8114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:06:38.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.205+0000 7f53df200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53d8114690 con 0x7f53d8102e20 2024-03-30T05:06:38.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.205+0000 7f53df200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53d810da80 con 0x7f53d81037c0 2024-03-30T05:06:38.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.205+0000 7f53d7e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53d8102e20 0x7f53d8113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:06:38.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.205+0000 7f53d7e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53d8102e20 0x7f53d8113b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38098/0 (socket says 172.21.15.123:38098) 2024-03-30T05:06:38.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.205+0000 7f53d7e00700 1 -- 172.21.15.123:0/2664491217 learned_addr learned my addr 172.21.15.123:0/2664491217 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:06:38.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.205+0000 7f53d7400700 1 --2- 172.21.15.123:0/2664491217 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53d81037c0 0x7f53d8114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:06:38.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.205+0000 7f53d7e00700 1 -- 172.21.15.123:0/2664491217 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53d81037c0 msgr2=0x7f53d8114040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:38.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.205+0000 7f53d7e00700 1 --2- 172.21.15.123:0/2664491217 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53d81037c0 0x7f53d8114040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:38.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.205+0000 7f53d7e00700 1 -- 172.21.15.123:0/2664491217 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53c8005040 con 0x7f53d8102e20 2024-03-30T05:06:38.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.206+0000 7f53d7e00700 1 --2- 172.21.15.123:0/2664491217 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53d8102e20 0x7f53d8113b00 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f53c8009360 tx=0x7f53c8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:06:38.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.206+0000 7f53cfe00700 1 -- 172.21.15.123:0/2664491217 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f53c800e040 con 0x7f53d8102e20 2024-03-30T05:06:38.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.206+0000 7f53cfe00700 1 -- 172.21.15.123:0/2664491217 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f53c8013420 con 0x7f53d8102e20 2024-03-30T05:06:38.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.206+0000 7f53df200700 1 -- 172.21.15.123:0/2664491217 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53d810dd00 con 0x7f53d8102e20 2024-03-30T05:06:38.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.206+0000 7f53cfe00700 1 -- 172.21.15.123:0/2664491217 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f53c801c5d0 con 0x7f53d8102e20 2024-03-30T05:06:38.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.206+0000 7f53df200700 1 -- 172.21.15.123:0/2664491217 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53d810e2c0 con 0x7f53d8102e20 2024-03-30T05:06:38.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.207+0000 7f53df200700 1 -- 172.21.15.123:0/2664491217 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53d80fe570 con 0x7f53d8102e20 2024-03-30T05:06:38.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.208+0000 7f53cfe00700 1 -- 172.21.15.123:0/2664491217 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f53c8013590 con 0x7f53d8102e20 2024-03-30T05:06:38.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.209+0000 7f53cfe00700 1 --2- 172.21.15.123:0/2664491217 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f53bc06cc30 0x7f53bc06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:06:38.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.209+0000 7f53cfe00700 1 -- 172.21.15.123:0/2664491217 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f53c808d1b0 con 0x7f53d8102e20 2024-03-30T05:06:38.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.209+0000 7f53d7400700 1 --2- 172.21.15.123:0/2664491217 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f53bc06cc30 0x7f53bc06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:06:38.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.210+0000 7f53d7400700 1 --2- 172.21.15.123:0/2664491217 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f53bc06cc30 0x7f53bc06f0e0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f53d810ec40 tx=0x7f53c400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:06:38.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.214+0000 7f53cfe00700 1 -- 172.21.15.123:0/2664491217 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f53c805ac00 con 0x7f53d8102e20 2024-03-30T05:06:38.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.435+0000 7f53df200700 1 -- 172.21.15.123:0/2664491217 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f53d8109de0 con 0x7f53bc06cc30 2024-03-30T05:06:38.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.436+0000 7f53cfe00700 1 -- 172.21.15.123:0/2664491217 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f53d8109de0 con 0x7f53bc06cc30 2024-03-30T05:06:38.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.438+0000 7f53df200700 1 -- 172.21.15.123:0/2664491217 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f53bc06cc30 msgr2=0x7f53bc06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:38.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.438+0000 7f53df200700 1 --2- 172.21.15.123:0/2664491217 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f53bc06cc30 0x7f53bc06f0e0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f53d810ec40 tx=0x7f53c400b040 comp rx=0 tx=0).stop 2024-03-30T05:06:38.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.438+0000 7f53df200700 1 -- 172.21.15.123:0/2664491217 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53d8102e20 msgr2=0x7f53d8113b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:38.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.438+0000 7f53df200700 1 --2- 172.21.15.123:0/2664491217 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53d8102e20 0x7f53d8113b00 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f53c8009360 tx=0x7f53c8009e30 comp rx=0 tx=0).stop 2024-03-30T05:06:38.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.438+0000 7f53df200700 1 -- 172.21.15.123:0/2664491217 shutdown_connections 2024-03-30T05:06:38.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.438+0000 7f53df200700 1 --2- 172.21.15.123:0/2664491217 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f53bc06cc30 0x7f53bc06f0e0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:38.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.438+0000 7f53df200700 1 --2- 172.21.15.123:0/2664491217 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53d8102e20 0x7f53d8113b00 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:38.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.438+0000 7f53df200700 1 --2- 172.21.15.123:0/2664491217 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53d81037c0 0x7f53d8114040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:38.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.438+0000 7f53df200700 1 -- 172.21.15.123:0/2664491217 >> 172.21.15.123:0/2664491217 conn(0x7f53d80f89d0 msgr2=0x7f53d80f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:06:38.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.439+0000 7f53df200700 1 -- 172.21.15.123:0/2664491217 shutdown_connections 2024-03-30T05:06:38.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.439+0000 7f53df200700 1 -- 172.21.15.123:0/2664491217 wait complete. 2024-03-30T05:06:38.451 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T05:06:38.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.573+0000 7fcf94e00700 1 -- 172.21.15.123:0/4052296336 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf901037a0 msgr2=0x7fcf900fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:38.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.573+0000 7fcf94e00700 1 --2- 172.21.15.123:0/4052296336 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf901037a0 0x7fcf900fce10 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c004940 tx=0x7fcf7c009e30 comp rx=0 tx=0).stop 2024-03-30T05:06:38.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.574+0000 7fcf94e00700 1 -- 172.21.15.123:0/4052296336 shutdown_connections 2024-03-30T05:06:38.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.574+0000 7fcf94e00700 1 --2- 172.21.15.123:0/4052296336 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf901037a0 0x7fcf900fce10 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:38.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.574+0000 7fcf94e00700 1 --2- 172.21.15.123:0/4052296336 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcf90102e00 0x7fcf901031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:38.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.574+0000 7fcf94e00700 1 -- 172.21.15.123:0/4052296336 >> 172.21.15.123:0/4052296336 conn(0x7fcf900f89b0 msgr2=0x7fcf900fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:06:38.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.574+0000 7fcf94e00700 1 -- 172.21.15.123:0/4052296336 shutdown_connections 2024-03-30T05:06:38.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.574+0000 7fcf94e00700 1 -- 172.21.15.123:0/4052296336 wait complete. 2024-03-30T05:06:38.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.574+0000 7fcf94e00700 1 Processor -- start 2024-03-30T05:06:38.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.575+0000 7fcf94e00700 1 -- start start 2024-03-30T05:06:38.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.575+0000 7fcf94e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcf90102e00 0x7fcf900702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:06:38.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.575+0000 7fcf94e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf901037a0 0x7fcf90070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:06:38.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.575+0000 7fcf94e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf90070d90 con 0x7fcf901037a0 2024-03-30T05:06:38.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.575+0000 7fcf94e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf90070ed0 con 0x7fcf90102e00 2024-03-30T05:06:38.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.575+0000 7fcf8d400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf901037a0 0x7fcf90070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:06:38.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.575+0000 7fcf8d400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf901037a0 0x7fcf90070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38124/0 (socket says 172.21.15.123:38124) 2024-03-30T05:06:38.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.575+0000 7fcf8d400700 1 -- 172.21.15.123:0/3372862169 learned_addr learned my addr 172.21.15.123:0/3372862169 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:06:38.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.576+0000 7fcf8de00700 1 --2- 172.21.15.123:0/3372862169 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcf90102e00 0x7fcf900702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:06:38.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.576+0000 7fcf8d400700 1 -- 172.21.15.123:0/3372862169 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcf90102e00 msgr2=0x7fcf900702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:38.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.576+0000 7fcf8d400700 1 --2- 172.21.15.123:0/3372862169 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcf90102e00 0x7fcf900702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:38.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.576+0000 7fcf8d400700 1 -- 172.21.15.123:0/3372862169 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf7c005040 con 0x7fcf901037a0 2024-03-30T05:06:38.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.576+0000 7fcf8d400700 1 --2- 172.21.15.123:0/3372862169 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf901037a0 0x7fcf90070800 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c009380 tx=0x7fcf7c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:06:38.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.576+0000 7fcf86000700 1 -- 172.21.15.123:0/3372862169 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf7c00e040 con 0x7fcf901037a0 2024-03-30T05:06:38.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.576+0000 7fcf94e00700 1 -- 172.21.15.123:0/3372862169 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf90071120 con 0x7fcf901037a0 2024-03-30T05:06:38.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.576+0000 7fcf86000700 1 -- 172.21.15.123:0/3372862169 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcf7c013420 con 0x7fcf901037a0 2024-03-30T05:06:38.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.576+0000 7fcf86000700 1 -- 172.21.15.123:0/3372862169 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf7c01c630 con 0x7fcf901037a0 2024-03-30T05:06:38.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.576+0000 7fcf94e00700 1 -- 172.21.15.123:0/3372862169 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf9011a230 con 0x7fcf901037a0 2024-03-30T05:06:38.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.577+0000 7fcf94e00700 1 -- 172.21.15.123:0/3372862169 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf900fe550 con 0x7fcf901037a0 2024-03-30T05:06:38.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.578+0000 7fcf86000700 1 -- 172.21.15.123:0/3372862169 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fcf7c0139e0 con 0x7fcf901037a0 2024-03-30T05:06:38.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.579+0000 7fcf86000700 1 --2- 172.21.15.123:0/3372862169 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fcf7406cc20 0x7fcf7406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:06:38.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.579+0000 7fcf86000700 1 -- 172.21.15.123:0/3372862169 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fcf7c08d280 con 0x7fcf901037a0 2024-03-30T05:06:38.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.579+0000 7fcf8de00700 1 --2- 172.21.15.123:0/3372862169 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fcf7406cc20 0x7fcf7406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:06:38.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.580+0000 7fcf8de00700 1 --2- 172.21.15.123:0/3372862169 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fcf7406cc20 0x7fcf7406f0d0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fcf80004990 tx=0x7fcf80009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:06:38.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.583+0000 7fcf86000700 1 -- 172.21.15.123:0/3372862169 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fcf7c057240 con 0x7fcf901037a0 2024-03-30T05:06:38.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.804+0000 7fcf94e00700 1 -- 172.21.15.123:0/3372862169 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcf90109e10 con 0x7fcf7406cc20 2024-03-30T05:06:38.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.805+0000 7fcf86000700 1 -- 172.21.15.123:0/3372862169 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fcf90109e10 con 0x7fcf7406cc20 2024-03-30T05:06:38.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.807+0000 7fcf94e00700 1 -- 172.21.15.123:0/3372862169 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fcf7406cc20 msgr2=0x7fcf7406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:38.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.807+0000 7fcf94e00700 1 --2- 172.21.15.123:0/3372862169 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fcf7406cc20 0x7fcf7406f0d0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fcf80004990 tx=0x7fcf80009ee0 comp rx=0 tx=0).stop 2024-03-30T05:06:38.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.807+0000 7fcf94e00700 1 -- 172.21.15.123:0/3372862169 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf901037a0 msgr2=0x7fcf90070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:38.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.807+0000 7fcf94e00700 1 --2- 172.21.15.123:0/3372862169 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf901037a0 0x7fcf90070800 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c009380 tx=0x7fcf7c009e30 comp rx=0 tx=0).stop 2024-03-30T05:06:38.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.807+0000 7fcf94e00700 1 -- 172.21.15.123:0/3372862169 shutdown_connections 2024-03-30T05:06:38.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.807+0000 7fcf94e00700 1 --2- 172.21.15.123:0/3372862169 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fcf7406cc20 0x7fcf7406f0d0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:38.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.807+0000 7fcf94e00700 1 --2- 172.21.15.123:0/3372862169 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcf90102e00 0x7fcf900702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:38.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.807+0000 7fcf94e00700 1 --2- 172.21.15.123:0/3372862169 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf901037a0 0x7fcf90070800 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:38.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.807+0000 7fcf94e00700 1 -- 172.21.15.123:0/3372862169 >> 172.21.15.123:0/3372862169 conn(0x7fcf900f89b0 msgr2=0x7fcf900f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:06:38.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.808+0000 7fcf94e00700 1 -- 172.21.15.123:0/3372862169 shutdown_connections 2024-03-30T05:06:38.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.808+0000 7fcf94e00700 1 -- 172.21.15.123:0/3372862169 wait complete. 2024-03-30T05:06:38.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.942+0000 7fb4ee000700 1 -- 172.21.15.123:0/3199482216 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb4e8102e20 msgr2=0x7fb4e81031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:38.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.942+0000 7fb4ee000700 1 --2- 172.21.15.123:0/3199482216 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb4e8102e20 0x7fb4e81031f0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fb4d8004940 tx=0x7fb4d8009e30 comp rx=0 tx=0).stop 2024-03-30T05:06:38.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.943+0000 7fb4ee000700 1 -- 172.21.15.123:0/3199482216 shutdown_connections 2024-03-30T05:06:38.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.943+0000 7fb4ee000700 1 --2- 172.21.15.123:0/3199482216 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb4e81037c0 0x7fb4e80fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:38.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.943+0000 7fb4ee000700 1 --2- 172.21.15.123:0/3199482216 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb4e8102e20 0x7fb4e81031f0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:38.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.943+0000 7fb4ee000700 1 -- 172.21.15.123:0/3199482216 >> 172.21.15.123:0/3199482216 conn(0x7fb4e80f89d0 msgr2=0x7fb4e80fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:06:38.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.943+0000 7fb4ee000700 1 -- 172.21.15.123:0/3199482216 shutdown_connections 2024-03-30T05:06:38.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.943+0000 7fb4ee000700 1 -- 172.21.15.123:0/3199482216 wait complete. 2024-03-30T05:06:38.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.943+0000 7fb4ee000700 1 Processor -- start 2024-03-30T05:06:38.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.944+0000 7fb4ee000700 1 -- start start 2024-03-30T05:06:38.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.944+0000 7fb4ee000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb4e8102e20 0x7fb4e8113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:06:38.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.944+0000 7fb4ee000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb4e81037c0 0x7fb4e8114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:06:38.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.944+0000 7fb4ee000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e8114690 con 0x7fb4e8102e20 2024-03-30T05:06:38.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.944+0000 7fb4ee000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e810da80 con 0x7fb4e81037c0 2024-03-30T05:06:38.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.944+0000 7fb4e7400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb4e8102e20 0x7fb4e8113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:06:38.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.944+0000 7fb4e7400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb4e8102e20 0x7fb4e8113b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38156/0 (socket says 172.21.15.123:38156) 2024-03-30T05:06:38.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.944+0000 7fb4e7400700 1 -- 172.21.15.123:0/2744846987 learned_addr learned my addr 172.21.15.123:0/2744846987 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:06:38.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.944+0000 7fb4e6a00700 1 --2- 172.21.15.123:0/2744846987 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb4e81037c0 0x7fb4e8114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:06:38.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.945+0000 7fb4e7400700 1 -- 172.21.15.123:0/2744846987 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb4e81037c0 msgr2=0x7fb4e8114040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:38.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.945+0000 7fb4e7400700 1 --2- 172.21.15.123:0/2744846987 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb4e81037c0 0x7fb4e8114040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:38.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.945+0000 7fb4e7400700 1 -- 172.21.15.123:0/2744846987 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4d8005040 con 0x7fb4e8102e20 2024-03-30T05:06:38.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.945+0000 7fb4e6a00700 1 --2- 172.21.15.123:0/2744846987 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb4e81037c0 0x7fb4e8114040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T05:06:38.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.945+0000 7fb4e7400700 1 --2- 172.21.15.123:0/2744846987 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb4e8102e20 0x7fb4e8113b00 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fb4d8009390 tx=0x7fb4d8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:06:38.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.945+0000 7fb4df400700 1 -- 172.21.15.123:0/2744846987 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4d800e040 con 0x7fb4e8102e20 2024-03-30T05:06:38.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.945+0000 7fb4df400700 1 -- 172.21.15.123:0/2744846987 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb4d8013420 con 0x7fb4e8102e20 2024-03-30T05:06:38.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.945+0000 7fb4ee000700 1 -- 172.21.15.123:0/2744846987 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4e810dd00 con 0x7fb4e8102e20 2024-03-30T05:06:38.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.945+0000 7fb4df400700 1 -- 172.21.15.123:0/2744846987 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4d801c630 con 0x7fb4e8102e20 2024-03-30T05:06:38.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.945+0000 7fb4ee000700 1 -- 172.21.15.123:0/2744846987 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4e810e2c0 con 0x7fb4e8102e20 2024-03-30T05:06:38.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.946+0000 7fb4ee000700 1 -- 172.21.15.123:0/2744846987 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4e80fe570 con 0x7fb4e8102e20 2024-03-30T05:06:38.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.947+0000 7fb4df400700 1 -- 172.21.15.123:0/2744846987 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fb4d80139e0 con 0x7fb4e8102e20 2024-03-30T05:06:38.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.948+0000 7fb4df400700 1 --2- 172.21.15.123:0/2744846987 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb4cc06cc30 0x7fb4cc06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:06:38.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.948+0000 7fb4df400700 1 -- 172.21.15.123:0/2744846987 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb4d808d250 con 0x7fb4e8102e20 2024-03-30T05:06:38.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.948+0000 7fb4e6a00700 1 --2- 172.21.15.123:0/2744846987 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb4cc06cc30 0x7fb4cc06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:06:38.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.949+0000 7fb4e6a00700 1 --2- 172.21.15.123:0/2744846987 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb4cc06cc30 0x7fb4cc06f0e0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fb4d4007fd0 tx=0x7fb4d4009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:06:38.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:38.953+0000 7fb4df400700 1 -- 172.21.15.123:0/2744846987 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb4d805add0 con 0x7fb4e8102e20 2024-03-30T05:06:39.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.165+0000 7fb4ee000700 1 -- 172.21.15.123:0/2744846987 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb4e8109e30 con 0x7fb4cc06cc30 2024-03-30T05:06:39.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.175+0000 7fb4df400700 1 -- 172.21.15.123:0/2744846987 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fb4e8109e30 con 0x7fb4cc06cc30 2024-03-30T05:06:39.173 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T05:06:39.173 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (35m) 13m ago 48m 14.6M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T05:06:39.173 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (33m) 13m ago 48m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T05:06:39.173 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (32m) 13m ago 47m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T05:06:39.173 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (35m) 13m ago 48m 73.5M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T05:06:39.173 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (28m) 13m ago 41m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T05:06:39.173 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (29m) 13m ago 42m 16.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T05:06:39.173 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (13m) 13m ago 41m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 87b073a6c9ca 2024-03-30T05:06:39.174 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (28m) 13m ago 42m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T05:06:39.174 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (38m) 13m ago 49m 510M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T05:06:39.174 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (37m) 13m ago 47m 456M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T05:06:39.174 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (34m) 13m ago 50m 70.0M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T05:06:39.174 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (33m) 13m ago 47m 51.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T05:06:39.174 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (36m) 13m ago 48m 8636k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T05:06:39.174 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (36m) 13m ago 47m 8678k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T05:06:39.174 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (32m) 13m ago 46m 107M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T05:06:39.174 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (32m) 13m ago 45m 97.2M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T05:06:39.174 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (31m) 13m ago 45m 89.8M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T05:06:39.174 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (30m) 13m ago 44m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T05:06:39.174 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (30m) 13m ago 44m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T05:06:39.174 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (29m) 13m ago 43m 88.8M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T05:06:39.175 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (36m) 13m ago 47m 63.9M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T05:06:39.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.177+0000 7fb4ee000700 1 -- 172.21.15.123:0/2744846987 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb4cc06cc30 msgr2=0x7fb4cc06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:39.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.177+0000 7fb4ee000700 1 --2- 172.21.15.123:0/2744846987 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb4cc06cc30 0x7fb4cc06f0e0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fb4d4007fd0 tx=0x7fb4d4009040 comp rx=0 tx=0).stop 2024-03-30T05:06:39.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.177+0000 7fb4ee000700 1 -- 172.21.15.123:0/2744846987 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb4e8102e20 msgr2=0x7fb4e8113b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:39.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.177+0000 7fb4ee000700 1 --2- 172.21.15.123:0/2744846987 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb4e8102e20 0x7fb4e8113b00 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fb4d8009390 tx=0x7fb4d8009e30 comp rx=0 tx=0).stop 2024-03-30T05:06:39.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.177+0000 7fb4ee000700 1 -- 172.21.15.123:0/2744846987 shutdown_connections 2024-03-30T05:06:39.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.177+0000 7fb4ee000700 1 --2- 172.21.15.123:0/2744846987 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb4cc06cc30 0x7fb4cc06f0e0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:39.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.177+0000 7fb4ee000700 1 --2- 172.21.15.123:0/2744846987 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb4e8102e20 0x7fb4e8113b00 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:39.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.177+0000 7fb4ee000700 1 --2- 172.21.15.123:0/2744846987 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb4e81037c0 0x7fb4e8114040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:39.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.177+0000 7fb4ee000700 1 -- 172.21.15.123:0/2744846987 >> 172.21.15.123:0/2744846987 conn(0x7fb4e80f89d0 msgr2=0x7fb4e80f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:06:39.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.177+0000 7fb4ee000700 1 -- 172.21.15.123:0/2744846987 shutdown_connections 2024-03-30T05:06:39.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.178+0000 7fb4ee000700 1 -- 172.21.15.123:0/2744846987 wait complete. 2024-03-30T05:06:39.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:06:38 smithi123 ceph-mon[139369]: pgmap v1064: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:39.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.310+0000 7f4b84000700 1 -- 172.21.15.123:0/935241950 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b7c102e00 msgr2=0x7f4b7c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:39.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.310+0000 7f4b84000700 1 --2- 172.21.15.123:0/935241950 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b7c102e00 0x7f4b7c1031d0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f4b70004940 tx=0x7f4b70009e30 comp rx=0 tx=0).stop 2024-03-30T05:06:39.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.310+0000 7f4b84000700 1 -- 172.21.15.123:0/935241950 shutdown_connections 2024-03-30T05:06:39.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.310+0000 7f4b84000700 1 --2- 172.21.15.123:0/935241950 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4b7c1037a0 0x7f4b7c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:39.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.310+0000 7f4b84000700 1 --2- 172.21.15.123:0/935241950 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b7c102e00 0x7f4b7c1031d0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:39.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.310+0000 7f4b84000700 1 -- 172.21.15.123:0/935241950 >> 172.21.15.123:0/935241950 conn(0x7f4b7c0f89b0 msgr2=0x7f4b7c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:06:39.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.310+0000 7f4b84000700 1 -- 172.21.15.123:0/935241950 shutdown_connections 2024-03-30T05:06:39.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.310+0000 7f4b84000700 1 -- 172.21.15.123:0/935241950 wait complete. 2024-03-30T05:06:39.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.311+0000 7f4b84000700 1 Processor -- start 2024-03-30T05:06:39.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.311+0000 7f4b84000700 1 -- start start 2024-03-30T05:06:39.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.311+0000 7f4b84000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4b7c102e00 0x7f4b7c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:06:39.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.311+0000 7f4b84000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b7c1037a0 0x7f4b7c070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:06:39.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.311+0000 7f4b84000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7c070d90 con 0x7f4b7c1037a0 2024-03-30T05:06:39.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.311+0000 7f4b84000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7c070ed0 con 0x7f4b7c102e00 2024-03-30T05:06:39.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.311+0000 7f4b80c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b7c1037a0 0x7f4b7c070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:06:39.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.312+0000 7f4b80c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b7c1037a0 0x7f4b7c070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38168/0 (socket says 172.21.15.123:38168) 2024-03-30T05:06:39.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.312+0000 7f4b80c00700 1 -- 172.21.15.123:0/3539917659 learned_addr learned my addr 172.21.15.123:0/3539917659 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:06:39.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.312+0000 7f4b81600700 1 --2- 172.21.15.123:0/3539917659 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4b7c102e00 0x7f4b7c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:06:39.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.312+0000 7f4b80c00700 1 -- 172.21.15.123:0/3539917659 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4b7c102e00 msgr2=0x7f4b7c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:39.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.312+0000 7f4b80c00700 1 --2- 172.21.15.123:0/3539917659 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4b7c102e00 0x7f4b7c0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:39.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.312+0000 7f4b80c00700 1 -- 172.21.15.123:0/3539917659 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b70005040 con 0x7f4b7c1037a0 2024-03-30T05:06:39.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.312+0000 7f4b81600700 1 --2- 172.21.15.123:0/3539917659 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4b7c102e00 0x7f4b7c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T05:06:39.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.312+0000 7f4b80c00700 1 --2- 172.21.15.123:0/3539917659 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b7c1037a0 0x7f4b7c070800 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f4b68002a80 tx=0x7f4b68004630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:06:39.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.312+0000 7f4b79600700 1 -- 172.21.15.123:0/3539917659 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b68007cc0 con 0x7f4b7c1037a0 2024-03-30T05:06:39.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.312+0000 7f4b79600700 1 -- 172.21.15.123:0/3539917659 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4b6800f040 con 0x7f4b7c1037a0 2024-03-30T05:06:39.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.312+0000 7f4b84000700 1 -- 172.21.15.123:0/3539917659 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b7c119e40 con 0x7f4b7c1037a0 2024-03-30T05:06:39.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.312+0000 7f4b79600700 1 -- 172.21.15.123:0/3539917659 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b6800da40 con 0x7f4b7c1037a0 2024-03-30T05:06:39.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.313+0000 7f4b84000700 1 -- 172.21.15.123:0/3539917659 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b7c11a2c0 con 0x7f4b7c1037a0 2024-03-30T05:06:39.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.314+0000 7f4b84000700 1 -- 172.21.15.123:0/3539917659 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b7c0fe550 con 0x7f4b7c1037a0 2024-03-30T05:06:39.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.314+0000 7f4b79600700 1 -- 172.21.15.123:0/3539917659 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f4b6800dbe0 con 0x7f4b7c1037a0 2024-03-30T05:06:39.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.316+0000 7f4b79600700 1 --2- 172.21.15.123:0/3539917659 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4b6406cc30 0x7f4b6406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:06:39.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.316+0000 7f4b79600700 1 -- 172.21.15.123:0/3539917659 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f4b6808b7c0 con 0x7f4b7c1037a0 2024-03-30T05:06:39.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.316+0000 7f4b81600700 1 --2- 172.21.15.123:0/3539917659 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4b6406cc30 0x7f4b6406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:06:39.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.316+0000 7f4b81600700 1 --2- 172.21.15.123:0/3539917659 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4b6406cc30 0x7f4b6406f0e0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f4b70000c00 tx=0x7f4b70009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:06:39.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.320+0000 7f4b79600700 1 -- 172.21.15.123:0/3539917659 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4b68055700 con 0x7f4b7c1037a0 2024-03-30T05:06:39.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:06:38 smithi135 ceph-mon[106011]: pgmap v1064: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:39.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.600+0000 7f4b84000700 1 -- 172.21.15.123:0/3539917659 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f4b7c04bed0 con 0x7f4b7c1037a0 2024-03-30T05:06:39.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.601+0000 7f4b79600700 1 -- 172.21.15.123:0/3539917659 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f4b68058d20 con 0x7f4b7c1037a0 2024-03-30T05:06:39.599 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T05:06:39.599 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T05:06:39.599 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T05:06:39.599 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:06:39.599 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T05:06:39.599 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T05:06:39.599 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:06:39.599 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T05:06:39.599 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T05:06:39.599 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:06:39.600 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T05:06:39.600 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 4 2024-03-30T05:06:39.600 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:06:39.600 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T05:06:39.600 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 14 2024-03-30T05:06:39.600 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T05:06:39.600 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T05:06:39.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.603+0000 7f4b84000700 1 -- 172.21.15.123:0/3539917659 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4b6406cc30 msgr2=0x7f4b6406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:39.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.603+0000 7f4b84000700 1 --2- 172.21.15.123:0/3539917659 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4b6406cc30 0x7f4b6406f0e0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f4b70000c00 tx=0x7f4b70009e30 comp rx=0 tx=0).stop 2024-03-30T05:06:39.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.603+0000 7f4b84000700 1 -- 172.21.15.123:0/3539917659 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b7c1037a0 msgr2=0x7f4b7c070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:39.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.603+0000 7f4b84000700 1 --2- 172.21.15.123:0/3539917659 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b7c1037a0 0x7f4b7c070800 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f4b68002a80 tx=0x7f4b68004630 comp rx=0 tx=0).stop 2024-03-30T05:06:39.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.603+0000 7f4b84000700 1 -- 172.21.15.123:0/3539917659 shutdown_connections 2024-03-30T05:06:39.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.603+0000 7f4b84000700 1 --2- 172.21.15.123:0/3539917659 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4b6406cc30 0x7f4b6406f0e0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:39.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.603+0000 7f4b84000700 1 --2- 172.21.15.123:0/3539917659 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4b7c102e00 0x7f4b7c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:39.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.603+0000 7f4b84000700 1 --2- 172.21.15.123:0/3539917659 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b7c1037a0 0x7f4b7c070800 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:39.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.603+0000 7f4b84000700 1 -- 172.21.15.123:0/3539917659 >> 172.21.15.123:0/3539917659 conn(0x7f4b7c0f89b0 msgr2=0x7f4b7c0f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:06:39.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.603+0000 7f4b84000700 1 -- 172.21.15.123:0/3539917659 shutdown_connections 2024-03-30T05:06:39.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.603+0000 7f4b84000700 1 -- 172.21.15.123:0/3539917659 wait complete. 2024-03-30T05:06:39.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.736+0000 7fd846e00700 1 -- 172.21.15.123:0/1560087234 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8400708a0 msgr2=0x7fd840070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:39.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.736+0000 7fd846e00700 1 --2- 172.21.15.123:0/1560087234 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8400708a0 0x7fd840070d10 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fd82c004970 tx=0x7fd82c009e30 comp rx=0 tx=0).stop 2024-03-30T05:06:39.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.736+0000 7fd846e00700 1 -- 172.21.15.123:0/1560087234 shutdown_connections 2024-03-30T05:06:39.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.736+0000 7fd846e00700 1 --2- 172.21.15.123:0/1560087234 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8400708a0 0x7fd840070d10 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:39.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.736+0000 7fd846e00700 1 --2- 172.21.15.123:0/1560087234 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd840071f90 0x7fd840070360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:39.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.736+0000 7fd846e00700 1 -- 172.21.15.123:0/1560087234 >> 172.21.15.123:0/1560087234 conn(0x7fd8400f89b0 msgr2=0x7fd8400fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:06:39.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.736+0000 7fd846e00700 1 -- 172.21.15.123:0/1560087234 shutdown_connections 2024-03-30T05:06:39.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.736+0000 7fd846e00700 1 -- 172.21.15.123:0/1560087234 wait complete. 2024-03-30T05:06:39.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.737+0000 7fd846e00700 1 Processor -- start 2024-03-30T05:06:39.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.737+0000 7fd846e00700 1 -- start start 2024-03-30T05:06:39.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.738+0000 7fd846e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8400708a0 0x7fd84011a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:06:39.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.738+0000 7fd846e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd840071f90 0x7fd84011a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:06:39.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.738+0000 7fd846e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd84011ac40 con 0x7fd8400708a0 2024-03-30T05:06:39.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.738+0000 7fd846e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd840114010 con 0x7fd840071f90 2024-03-30T05:06:39.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.738+0000 7fd83fe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8400708a0 0x7fd84011a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:06:39.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.738+0000 7fd83fe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8400708a0 0x7fd84011a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38186/0 (socket says 172.21.15.123:38186) 2024-03-30T05:06:39.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.738+0000 7fd83fe00700 1 -- 172.21.15.123:0/1534673266 learned_addr learned my addr 172.21.15.123:0/1534673266 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:06:39.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.738+0000 7fd83f400700 1 --2- 172.21.15.123:0/1534673266 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd840071f90 0x7fd84011a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:06:39.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.738+0000 7fd83fe00700 1 -- 172.21.15.123:0/1534673266 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd840071f90 msgr2=0x7fd84011a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:39.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.738+0000 7fd83fe00700 1 --2- 172.21.15.123:0/1534673266 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd840071f90 0x7fd84011a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:39.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.738+0000 7fd83fe00700 1 -- 172.21.15.123:0/1534673266 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd82c005040 con 0x7fd8400708a0 2024-03-30T05:06:39.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.739+0000 7fd83fe00700 1 --2- 172.21.15.123:0/1534673266 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8400708a0 0x7fd84011a0b0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fd830002a50 tx=0x7fd8300046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:06:39.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.739+0000 7fd837e00700 1 -- 172.21.15.123:0/1534673266 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd830007cc0 con 0x7fd8400708a0 2024-03-30T05:06:39.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.739+0000 7fd846e00700 1 -- 172.21.15.123:0/1534673266 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8401142f0 con 0x7fd8400708a0 2024-03-30T05:06:39.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.739+0000 7fd837e00700 1 -- 172.21.15.123:0/1534673266 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd83000f040 con 0x7fd8400708a0 2024-03-30T05:06:39.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.739+0000 7fd837e00700 1 -- 172.21.15.123:0/1534673266 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd83000d9c0 con 0x7fd8400708a0 2024-03-30T05:06:39.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.739+0000 7fd846e00700 1 -- 172.21.15.123:0/1534673266 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd840114800 con 0x7fd8400708a0 2024-03-30T05:06:39.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.740+0000 7fd846e00700 1 -- 172.21.15.123:0/1534673266 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd840071aa0 con 0x7fd8400708a0 2024-03-30T05:06:39.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.741+0000 7fd837e00700 1 -- 172.21.15.123:0/1534673266 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fd83000db20 con 0x7fd8400708a0 2024-03-30T05:06:39.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.742+0000 7fd837e00700 1 --2- 172.21.15.123:0/1534673266 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd82406cc80 0x7fd82406f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:06:39.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.742+0000 7fd837e00700 1 -- 172.21.15.123:0/1534673266 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd83008bde0 con 0x7fd8400708a0 2024-03-30T05:06:39.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.742+0000 7fd83f400700 1 --2- 172.21.15.123:0/1534673266 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd82406cc80 0x7fd82406f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:06:39.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.743+0000 7fd83f400700 1 --2- 172.21.15.123:0/1534673266 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd82406cc80 0x7fd82406f130 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fd82c00b370 tx=0x7fd82c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:06:39.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.746+0000 7fd837e00700 1 -- 172.21.15.123:0/1534673266 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd8300597b0 con 0x7fd8400708a0 2024-03-30T05:06:39.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.996+0000 7fd846e00700 1 -- 172.21.15.123:0/1534673266 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fd84004bed0 con 0x7fd8400708a0 2024-03-30T05:06:39.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.996+0000 7fd837e00700 1 -- 172.21.15.123:0/1534673266 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 34 v34) v1 ==== 76+0+1843 (secure 0 0 0) 0x7fd830059340 con 0x7fd8400708a0 2024-03-30T05:06:39.995 INFO:teuthology.orchestra.run.smithi123.stdout:e34 2024-03-30T05:06:39.995 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T05:06:39.995 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T05:06:39.995 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T05:06:39.995 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:06:39.995 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T05:06:39.995 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T05:06:39.995 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T05:06:39.995 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T05:06:39.995 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T05:06:39.995 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T05:06:39.996 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T05:06:39.996 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T05:06:39.996 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T05:06:39.996 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T05:06:39.996 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T05:06:39.996 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T05:06:39.996 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T05:06:39.996 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T05:06:39.996 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T05:06:39.996 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T05:06:39.996 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T05:06:39.996 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T05:06:39.996 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T05:06:39.996 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T05:06:39.996 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T05:06:39.997 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T05:06:39.997 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T05:06:39.997 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T05:06:39.997 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T05:06:39.997 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T05:06:39.997 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T05:06:39.997 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T05:06:39.997 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:06:39.997 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:06:39.997 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T05:06:39.997 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:06:39.997 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:06:39.997 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:06:39.997 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:06:39.998 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44975} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/3788988014,v1:172.21.15.135:6827/3788988014] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:06:39.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.999+0000 7fd846e00700 1 -- 172.21.15.123:0/1534673266 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd82406cc80 msgr2=0x7fd82406f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:39.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.999+0000 7fd846e00700 1 --2- 172.21.15.123:0/1534673266 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd82406cc80 0x7fd82406f130 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fd82c00b370 tx=0x7fd82c009e30 comp rx=0 tx=0).stop 2024-03-30T05:06:39.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.999+0000 7fd846e00700 1 -- 172.21.15.123:0/1534673266 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8400708a0 msgr2=0x7fd84011a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:39.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.999+0000 7fd846e00700 1 --2- 172.21.15.123:0/1534673266 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8400708a0 0x7fd84011a0b0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fd830002a50 tx=0x7fd8300046b0 comp rx=0 tx=0).stop 2024-03-30T05:06:39.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.999+0000 7fd846e00700 1 -- 172.21.15.123:0/1534673266 shutdown_connections 2024-03-30T05:06:39.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.999+0000 7fd846e00700 1 --2- 172.21.15.123:0/1534673266 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd82406cc80 0x7fd82406f130 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:39.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.999+0000 7fd846e00700 1 --2- 172.21.15.123:0/1534673266 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8400708a0 0x7fd84011a0b0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:39.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.999+0000 7fd846e00700 1 --2- 172.21.15.123:0/1534673266 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd840071f90 0x7fd84011a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:39.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.999+0000 7fd846e00700 1 -- 172.21.15.123:0/1534673266 >> 172.21.15.123:0/1534673266 conn(0x7fd8400f89b0 msgr2=0x7fd8400f9b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:06:39.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.999+0000 7fd846e00700 1 -- 172.21.15.123:0/1534673266 shutdown_connections 2024-03-30T05:06:39.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:39.999+0000 7fd846e00700 1 -- 172.21.15.123:0/1534673266 wait complete. 2024-03-30T05:06:39.999 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 34 2024-03-30T05:06:40.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.133+0000 7f5287800700 1 -- 172.21.15.123:0/1179651324 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52801037a0 msgr2=0x7f52800fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:40.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.133+0000 7f5287800700 1 --2- 172.21.15.123:0/1179651324 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52801037a0 0x7f52800fce10 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f526c004940 tx=0x7f526c009e30 comp rx=0 tx=0).stop 2024-03-30T05:06:40.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.133+0000 7f5287800700 1 -- 172.21.15.123:0/1179651324 shutdown_connections 2024-03-30T05:06:40.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.133+0000 7f5287800700 1 --2- 172.21.15.123:0/1179651324 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52801037a0 0x7f52800fce10 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:40.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.133+0000 7f5287800700 1 --2- 172.21.15.123:0/1179651324 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5280102e00 0x7f52801031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:40.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.133+0000 7f5287800700 1 -- 172.21.15.123:0/1179651324 >> 172.21.15.123:0/1179651324 conn(0x7f52800f89b0 msgr2=0x7f52800fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:06:40.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.133+0000 7f5287800700 1 -- 172.21.15.123:0/1179651324 shutdown_connections 2024-03-30T05:06:40.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.134+0000 7f5287800700 1 -- 172.21.15.123:0/1179651324 wait complete. 2024-03-30T05:06:40.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.134+0000 7f5287800700 1 Processor -- start 2024-03-30T05:06:40.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.134+0000 7f5287800700 1 -- start start 2024-03-30T05:06:40.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.135+0000 7f5287800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5280102e00 0x7f5280115d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:06:40.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.135+0000 7f5287800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52801037a0 0x7f5280116240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:06:40.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.135+0000 7f5287800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5280116890 con 0x7f52801037a0 2024-03-30T05:06:40.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.135+0000 7f5287800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f528010fc60 con 0x7f5280102e00 2024-03-30T05:06:40.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.135+0000 7f527fe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52801037a0 0x7f5280116240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:06:40.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.135+0000 7f527fe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52801037a0 0x7f5280116240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38216/0 (socket says 172.21.15.123:38216) 2024-03-30T05:06:40.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.135+0000 7f527fe00700 1 -- 172.21.15.123:0/462107570 learned_addr learned my addr 172.21.15.123:0/462107570 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:06:40.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.135+0000 7f527fe00700 1 -- 172.21.15.123:0/462107570 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5280102e00 msgr2=0x7f5280115d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:40.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.135+0000 7f5284e00700 1 --2- 172.21.15.123:0/462107570 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5280102e00 0x7f5280115d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:06:40.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.135+0000 7f527fe00700 1 --2- 172.21.15.123:0/462107570 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5280102e00 0x7f5280115d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:40.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.135+0000 7f527fe00700 1 -- 172.21.15.123:0/462107570 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f526c005040 con 0x7f52801037a0 2024-03-30T05:06:40.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.135+0000 7f5284e00700 1 --2- 172.21.15.123:0/462107570 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5280102e00 0x7f5280115d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T05:06:40.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.136+0000 7f527fe00700 1 --2- 172.21.15.123:0/462107570 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52801037a0 0x7f5280116240 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f526c009390 tx=0x7f526c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:06:40.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.136+0000 7f527cc00700 1 -- 172.21.15.123:0/462107570 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f526c00e040 con 0x7f52801037a0 2024-03-30T05:06:40.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.136+0000 7f527cc00700 1 -- 172.21.15.123:0/462107570 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f526c013420 con 0x7f52801037a0 2024-03-30T05:06:40.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.136+0000 7f5287800700 1 -- 172.21.15.123:0/462107570 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f528010fee0 con 0x7f52801037a0 2024-03-30T05:06:40.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.136+0000 7f527cc00700 1 -- 172.21.15.123:0/462107570 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f526c01c630 con 0x7f52801037a0 2024-03-30T05:06:40.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.136+0000 7f5287800700 1 -- 172.21.15.123:0/462107570 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52801104a0 con 0x7f52801037a0 2024-03-30T05:06:40.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.137+0000 7f5287800700 1 -- 172.21.15.123:0/462107570 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52800fe550 con 0x7f52801037a0 2024-03-30T05:06:40.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.138+0000 7f527cc00700 1 -- 172.21.15.123:0/462107570 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f526c013a00 con 0x7f52801037a0 2024-03-30T05:06:40.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.139+0000 7f527cc00700 1 --2- 172.21.15.123:0/462107570 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f526806cc30 0x7f526806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:06:40.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.139+0000 7f527cc00700 1 -- 172.21.15.123:0/462107570 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f526c08d2d0 con 0x7f52801037a0 2024-03-30T05:06:40.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.139+0000 7f5284e00700 1 --2- 172.21.15.123:0/462107570 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f526806cc30 0x7f526806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:06:40.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.140+0000 7f5284e00700 1 --2- 172.21.15.123:0/462107570 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f526806cc30 0x7f526806f0e0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f5274007fd0 tx=0x7f5274009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:06:40.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.144+0000 7f527cc00700 1 -- 172.21.15.123:0/462107570 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f526c05ada0 con 0x7f52801037a0 2024-03-30T05:06:40.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:06:39 smithi123 ceph-mon[139369]: from='client.35916 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:06:40.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:06:39 smithi123 ceph-mon[139369]: from='client.35920 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:06:40.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:06:39 smithi123 ceph-mon[139369]: from='client.35924 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:06:40.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:06:39 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3539917659' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:06:40.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:06:39 smithi135 ceph-mon[106011]: from='client.35916 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:06:40.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:06:39 smithi135 ceph-mon[106011]: from='client.35920 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:06:40.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:06:39 smithi135 ceph-mon[106011]: from='client.35924 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:06:40.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:06:39 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3539917659' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:06:40.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.367+0000 7f5287800700 1 -- 172.21.15.123:0/462107570 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5280109da0 con 0x7f526806cc30 2024-03-30T05:06:40.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.368+0000 7f527cc00700 1 -- 172.21.15.123:0/462107570 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f5280109da0 con 0x7f526806cc30 2024-03-30T05:06:40.366 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T05:06:40.366 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T05:06:40.367 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T05:06:40.367 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T05:06:40.367 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T05:06:40.367 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T05:06:40.367 INFO:teuthology.orchestra.run.smithi123.stdout: "mds", 2024-03-30T05:06:40.367 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T05:06:40.367 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T05:06:40.367 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T05:06:40.367 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T05:06:40.367 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "16/21 daemons upgraded", 2024-03-30T05:06:40.367 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T05:06:40.367 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T05:06:40.367 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T05:06:40.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.371+0000 7f5287800700 1 -- 172.21.15.123:0/462107570 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f526806cc30 msgr2=0x7f526806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:40.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.371+0000 7f5287800700 1 --2- 172.21.15.123:0/462107570 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f526806cc30 0x7f526806f0e0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f5274007fd0 tx=0x7f5274009040 comp rx=0 tx=0).stop 2024-03-30T05:06:40.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.371+0000 7f5287800700 1 -- 172.21.15.123:0/462107570 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52801037a0 msgr2=0x7f5280116240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:40.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.371+0000 7f5287800700 1 --2- 172.21.15.123:0/462107570 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52801037a0 0x7f5280116240 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f526c009390 tx=0x7f526c009e30 comp rx=0 tx=0).stop 2024-03-30T05:06:40.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.371+0000 7f5287800700 1 -- 172.21.15.123:0/462107570 shutdown_connections 2024-03-30T05:06:40.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.371+0000 7f5287800700 1 --2- 172.21.15.123:0/462107570 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f526806cc30 0x7f526806f0e0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:40.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.371+0000 7f5287800700 1 --2- 172.21.15.123:0/462107570 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5280102e00 0x7f5280115d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:40.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.371+0000 7f5287800700 1 --2- 172.21.15.123:0/462107570 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52801037a0 0x7f5280116240 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:40.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.371+0000 7f5287800700 1 -- 172.21.15.123:0/462107570 >> 172.21.15.123:0/462107570 conn(0x7f52800f89b0 msgr2=0x7f52800f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:06:40.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.371+0000 7f5287800700 1 -- 172.21.15.123:0/462107570 shutdown_connections 2024-03-30T05:06:40.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.371+0000 7f5287800700 1 -- 172.21.15.123:0/462107570 wait complete. 2024-03-30T05:06:40.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.505+0000 7f616ba00700 1 -- 172.21.15.123:0/845779853 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6164102e00 msgr2=0x7f61641031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:40.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.505+0000 7f616ba00700 1 --2- 172.21.15.123:0/845779853 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6164102e00 0x7f61641031d0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f6158004940 tx=0x7f6158009e30 comp rx=0 tx=0).stop 2024-03-30T05:06:40.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.505+0000 7f616ba00700 1 -- 172.21.15.123:0/845779853 shutdown_connections 2024-03-30T05:06:40.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.505+0000 7f616ba00700 1 --2- 172.21.15.123:0/845779853 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f61641037a0 0x7f61640fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:40.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.505+0000 7f616ba00700 1 --2- 172.21.15.123:0/845779853 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6164102e00 0x7f61641031d0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:40.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.505+0000 7f616ba00700 1 -- 172.21.15.123:0/845779853 >> 172.21.15.123:0/845779853 conn(0x7f61640f89b0 msgr2=0x7f61640fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:06:40.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.506+0000 7f616ba00700 1 -- 172.21.15.123:0/845779853 shutdown_connections 2024-03-30T05:06:40.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.506+0000 7f616ba00700 1 -- 172.21.15.123:0/845779853 wait complete. 2024-03-30T05:06:40.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.506+0000 7f616ba00700 1 Processor -- start 2024-03-30T05:06:40.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.507+0000 7f616ba00700 1 -- start start 2024-03-30T05:06:40.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.507+0000 7f616ba00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6164102e00 0x7f61640702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:06:40.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.507+0000 7f616ba00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61641037a0 0x7f6164070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:06:40.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.507+0000 7f616ba00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6164070d90 con 0x7f61641037a0 2024-03-30T05:06:40.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.507+0000 7f616ba00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6164070ed0 con 0x7f6164102e00 2024-03-30T05:06:40.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.507+0000 7f6163e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61641037a0 0x7f6164070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:06:40.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.507+0000 7f6163e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61641037a0 0x7f6164070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38246/0 (socket says 172.21.15.123:38246) 2024-03-30T05:06:40.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.507+0000 7f6163e00700 1 -- 172.21.15.123:0/1751716374 learned_addr learned my addr 172.21.15.123:0/1751716374 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:06:40.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.507+0000 7f6169000700 1 --2- 172.21.15.123:0/1751716374 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6164102e00 0x7f61640702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:06:40.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.508+0000 7f6163e00700 1 -- 172.21.15.123:0/1751716374 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6164102e00 msgr2=0x7f61640702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:40.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.508+0000 7f6163e00700 1 --2- 172.21.15.123:0/1751716374 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6164102e00 0x7f61640702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:40.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.508+0000 7f6163e00700 1 -- 172.21.15.123:0/1751716374 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6158005040 con 0x7f61641037a0 2024-03-30T05:06:40.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.508+0000 7f6163e00700 1 --2- 172.21.15.123:0/1751716374 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61641037a0 0x7f6164070800 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f6150002a50 tx=0x7f6150004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:06:40.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.508+0000 7f6160c00700 1 -- 172.21.15.123:0/1751716374 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6150007cc0 con 0x7f61641037a0 2024-03-30T05:06:40.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.508+0000 7f6160c00700 1 -- 172.21.15.123:0/1751716374 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f615000f040 con 0x7f61641037a0 2024-03-30T05:06:40.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.508+0000 7f616ba00700 1 -- 172.21.15.123:0/1751716374 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6164119e20 con 0x7f61641037a0 2024-03-30T05:06:40.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.508+0000 7f6160c00700 1 -- 172.21.15.123:0/1751716374 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f615000d9a0 con 0x7f61641037a0 2024-03-30T05:06:40.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.508+0000 7f616ba00700 1 -- 172.21.15.123:0/1751716374 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f616411a2a0 con 0x7f61641037a0 2024-03-30T05:06:40.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.509+0000 7f616ba00700 1 -- 172.21.15.123:0/1751716374 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61640fe550 con 0x7f61641037a0 2024-03-30T05:06:40.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.510+0000 7f6160c00700 1 -- 172.21.15.123:0/1751716374 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f615000dc60 con 0x7f61641037a0 2024-03-30T05:06:40.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.511+0000 7f6160c00700 1 --2- 172.21.15.123:0/1751716374 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f614c06cc30 0x7f614c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:06:40.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.512+0000 7f6160c00700 1 -- 172.21.15.123:0/1751716374 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f615008bef0 con 0x7f61641037a0 2024-03-30T05:06:40.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.512+0000 7f6169000700 1 --2- 172.21.15.123:0/1751716374 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f614c06cc30 0x7f614c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:06:40.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.512+0000 7f6169000700 1 --2- 172.21.15.123:0/1751716374 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f614c06cc30 0x7f614c06f0e0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f615800b3b0 tx=0x7f6158009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:06:40.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.516+0000 7f6160c00700 1 -- 172.21.15.123:0/1751716374 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6150055e30 con 0x7f61641037a0 2024-03-30T05:06:40.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.794+0000 7f616ba00700 1 -- 172.21.15.123:0/1751716374 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f616405f5f0 con 0x7f61641037a0 2024-03-30T05:06:40.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.795+0000 7f6160c00700 1 -- 172.21.15.123:0/1751716374 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f616405f5f0 con 0x7f61641037a0 2024-03-30T05:06:40.793 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T05:06:40.793 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-30T05:06:40.793 INFO:teuthology.orchestra.run.smithi123.stdout: host smithi135 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi135/config 2024-03-30T05:06:40.794 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-30T05:06:40.794 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T05:06:40.794 INFO:teuthology.orchestra.run.smithi123.stdout:Traceback (most recent call last): 2024-03-30T05:06:40.794 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-30T05:06:40.794 INFO:teuthology.orchestra.run.smithi123.stdout: "__main__", mod_spec) 2024-03-30T05:06:40.794 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-30T05:06:40.794 INFO:teuthology.orchestra.run.smithi123.stdout: exec(code, run_globals) 2024-03-30T05:06:40.794 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10889, in 2024-03-30T05:06:40.794 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10877, in main 2024-03-30T05:06:40.794 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-30T05:06:40.794 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-30T05:06:40.794 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-30T05:06:40.794 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-30T05:06:40.795 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-03-30T05:06:40.795 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-30T05:06:40.795 INFO:teuthology.orchestra.run.smithi123.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T05:06:40.795 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T05:06:40.795 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T05:06:40.795 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T05:06:40.795 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T05:06:40.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.798+0000 7f616ba00700 1 -- 172.21.15.123:0/1751716374 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f614c06cc30 msgr2=0x7f614c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:40.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.798+0000 7f616ba00700 1 --2- 172.21.15.123:0/1751716374 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f614c06cc30 0x7f614c06f0e0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f615800b3b0 tx=0x7f6158009e30 comp rx=0 tx=0).stop 2024-03-30T05:06:40.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.798+0000 7f616ba00700 1 -- 172.21.15.123:0/1751716374 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61641037a0 msgr2=0x7f6164070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:06:40.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.798+0000 7f616ba00700 1 --2- 172.21.15.123:0/1751716374 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61641037a0 0x7f6164070800 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f6150002a50 tx=0x7f6150004600 comp rx=0 tx=0).stop 2024-03-30T05:06:40.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.798+0000 7f616ba00700 1 -- 172.21.15.123:0/1751716374 shutdown_connections 2024-03-30T05:06:40.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.798+0000 7f616ba00700 1 --2- 172.21.15.123:0/1751716374 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f614c06cc30 0x7f614c06f0e0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:40.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.798+0000 7f616ba00700 1 --2- 172.21.15.123:0/1751716374 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6164102e00 0x7f61640702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:40.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.798+0000 7f616ba00700 1 --2- 172.21.15.123:0/1751716374 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61641037a0 0x7f6164070800 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:06:40.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.798+0000 7f616ba00700 1 -- 172.21.15.123:0/1751716374 >> 172.21.15.123:0/1751716374 conn(0x7f61640f89b0 msgr2=0x7f61640f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:06:40.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.798+0000 7f616ba00700 1 -- 172.21.15.123:0/1751716374 shutdown_connections 2024-03-30T05:06:40.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:06:40.798+0000 7f616ba00700 1 -- 172.21.15.123:0/1751716374 wait complete. 2024-03-30T05:06:41.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:06:40 smithi123 ceph-mon[139369]: pgmap v1065: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:41.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:06:40 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/1534673266' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T05:06:41.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:06:40 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/1751716374' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T05:06:41.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:06:40 smithi135 ceph-mon[106011]: pgmap v1065: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:41.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:06:40 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/1534673266' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T05:06:41.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:06:40 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/1751716374' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T05:06:42.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:06:41 smithi123 ceph-mon[139369]: from='client.35936 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:06:42.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:06:41 smithi135 ceph-mon[106011]: from='client.35936 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:06:43.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:06:42 smithi123 ceph-mon[139369]: pgmap v1066: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:43.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:06:42 smithi135 ceph-mon[106011]: pgmap v1066: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:45.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:06:44 smithi123 ceph-mon[139369]: pgmap v1067: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:45.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:06:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:06:45.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:06:44 smithi135 ceph-mon[106011]: pgmap v1067: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:45.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:06:44 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:06:47.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:06:46 smithi123 ceph-mon[139369]: pgmap v1068: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:47.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:06:46 smithi135 ceph-mon[106011]: pgmap v1068: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:49.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:06:48 smithi123 ceph-mon[139369]: pgmap v1069: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:49.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:06:48 smithi135 ceph-mon[106011]: pgmap v1069: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:51.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:06:50 smithi123 ceph-mon[139369]: pgmap v1070: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:51.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:06:50 smithi135 ceph-mon[106011]: pgmap v1070: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:53.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:06:52 smithi123 ceph-mon[139369]: pgmap v1071: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:53.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:06:52 smithi135 ceph-mon[106011]: pgmap v1071: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:55.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:06:54 smithi123 ceph-mon[139369]: pgmap v1072: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:55.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:06:54 smithi135 ceph-mon[106011]: pgmap v1072: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:57.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:06:56 smithi123 ceph-mon[139369]: pgmap v1073: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:57.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:06:56 smithi135 ceph-mon[106011]: pgmap v1073: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:59.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:06:58 smithi123 ceph-mon[139369]: pgmap v1074: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:06:59.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:06:58 smithi135 ceph-mon[106011]: pgmap v1074: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:00.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:06:59 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:07:00.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:06:59 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:07:01.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:07:00 smithi135 ceph-mon[106011]: pgmap v1075: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:01.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:07:00 smithi123 ceph-mon[139369]: pgmap v1075: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:03.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:07:02 smithi135 ceph-mon[106011]: pgmap v1076: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:03.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:07:02 smithi123 ceph-mon[139369]: pgmap v1076: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:05.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:07:04 smithi135 ceph-mon[106011]: pgmap v1077: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:05.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:07:04 smithi123 ceph-mon[139369]: pgmap v1077: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:07.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:07:06 smithi135 ceph-mon[106011]: pgmap v1078: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:07.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:07:06 smithi123 ceph-mon[139369]: pgmap v1078: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:09.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:07:08 smithi135 ceph-mon[106011]: pgmap v1079: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:09.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:07:08 smithi123 ceph-mon[139369]: pgmap v1079: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:10.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:10.951+0000 7f5e07e00700 1 -- 172.21.15.123:0/3993449847 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e00102e00 msgr2=0x7f5e001031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:10.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:10.951+0000 7f5e07e00700 1 --2- 172.21.15.123:0/3993449847 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e00102e00 0x7f5e001031d0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f5df4004940 tx=0x7f5df4009e30 comp rx=0 tx=0).stop 2024-03-30T05:07:10.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:10.951+0000 7f5e07e00700 1 -- 172.21.15.123:0/3993449847 shutdown_connections 2024-03-30T05:07:10.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:10.951+0000 7f5e07e00700 1 --2- 172.21.15.123:0/3993449847 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5e001037a0 0x7f5e000fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:10.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:10.951+0000 7f5e07e00700 1 --2- 172.21.15.123:0/3993449847 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e00102e00 0x7f5e001031d0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:10.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:10.951+0000 7f5e07e00700 1 -- 172.21.15.123:0/3993449847 >> 172.21.15.123:0/3993449847 conn(0x7f5e000f89b0 msgr2=0x7f5e000fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:07:10.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:10.951+0000 7f5e07e00700 1 -- 172.21.15.123:0/3993449847 shutdown_connections 2024-03-30T05:07:10.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:10.951+0000 7f5e07e00700 1 -- 172.21.15.123:0/3993449847 wait complete. 2024-03-30T05:07:10.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:10.953+0000 7f5e07e00700 1 Processor -- start 2024-03-30T05:07:10.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:10.953+0000 7f5e07e00700 1 -- start start 2024-03-30T05:07:10.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:10.953+0000 7f5e07e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5e00102e00 0x7f5e000702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:07:10.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:10.953+0000 7f5e07e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e001037a0 0x7f5e00070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:07:10.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:10.953+0000 7f5e07e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e00070d90 con 0x7f5e001037a0 2024-03-30T05:07:10.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:10.953+0000 7f5e07e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e00070ed0 con 0x7f5e00102e00 2024-03-30T05:07:10.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:10.954+0000 7f5e04a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e001037a0 0x7f5e00070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:07:10.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:10.954+0000 7f5e04a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e001037a0 0x7f5e00070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58864/0 (socket says 172.21.15.123:58864) 2024-03-30T05:07:10.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:10.954+0000 7f5e04a00700 1 -- 172.21.15.123:0/49185730 learned_addr learned my addr 172.21.15.123:0/49185730 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:07:10.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:10.954+0000 7f5e05400700 1 --2- 172.21.15.123:0/49185730 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5e00102e00 0x7f5e000702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:07:10.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:10.954+0000 7f5e04a00700 1 -- 172.21.15.123:0/49185730 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5e00102e00 msgr2=0x7f5e000702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:10.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:10.954+0000 7f5e04a00700 1 --2- 172.21.15.123:0/49185730 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5e00102e00 0x7f5e000702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:10.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:10.954+0000 7f5e04a00700 1 -- 172.21.15.123:0/49185730 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5df4005040 con 0x7f5e001037a0 2024-03-30T05:07:10.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:10.954+0000 7f5e04a00700 1 --2- 172.21.15.123:0/49185730 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e001037a0 0x7f5e00070800 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f5dec00eb40 tx=0x7f5dec00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:07:10.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:10.954+0000 7f5dfd600700 1 -- 172.21.15.123:0/49185730 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5dec007cc0 con 0x7f5e001037a0 2024-03-30T05:07:10.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:10.955+0000 7f5dfd600700 1 -- 172.21.15.123:0/49185730 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5dec00f040 con 0x7f5e001037a0 2024-03-30T05:07:10.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:10.955+0000 7f5e07e00700 1 -- 172.21.15.123:0/49185730 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e00071120 con 0x7f5e001037a0 2024-03-30T05:07:10.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:10.955+0000 7f5dfd600700 1 -- 172.21.15.123:0/49185730 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5dec009570 con 0x7f5e001037a0 2024-03-30T05:07:10.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:10.955+0000 7f5e07e00700 1 -- 172.21.15.123:0/49185730 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e0011a170 con 0x7f5e001037a0 2024-03-30T05:07:10.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:10.956+0000 7f5e07e00700 1 -- 172.21.15.123:0/49185730 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e000fe550 con 0x7f5e001037a0 2024-03-30T05:07:10.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:10.957+0000 7f5dfd600700 1 -- 172.21.15.123:0/49185730 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f5dec0096d0 con 0x7f5e001037a0 2024-03-30T05:07:10.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:10.958+0000 7f5dfd600700 1 --2- 172.21.15.123:0/49185730 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5de806cc20 0x7f5de806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:07:10.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:10.958+0000 7f5dfd600700 1 -- 172.21.15.123:0/49185730 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5dec08ba90 con 0x7f5e001037a0 2024-03-30T05:07:10.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:10.958+0000 7f5e05400700 1 --2- 172.21.15.123:0/49185730 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5de806cc20 0x7f5de806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:07:10.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:10.959+0000 7f5e05400700 1 --2- 172.21.15.123:0/49185730 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5de806cc20 0x7f5de806f0d0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f5df400b0d0 tx=0x7f5df4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:07:10.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:10.962+0000 7f5dfd600700 1 -- 172.21.15.123:0/49185730 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5dec055b80 con 0x7f5e001037a0 2024-03-30T05:07:11.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.183+0000 7f5e07e00700 1 -- 172.21.15.123:0/49185730 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5e0004bed0 con 0x7f5de806cc20 2024-03-30T05:07:11.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.184+0000 7f5dfd600700 1 -- 172.21.15.123:0/49185730 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f5e0004bed0 con 0x7f5de806cc20 2024-03-30T05:07:11.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.186+0000 7f5e07e00700 1 -- 172.21.15.123:0/49185730 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5de806cc20 msgr2=0x7f5de806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:11.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.186+0000 7f5e07e00700 1 --2- 172.21.15.123:0/49185730 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5de806cc20 0x7f5de806f0d0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f5df400b0d0 tx=0x7f5df4009e30 comp rx=0 tx=0).stop 2024-03-30T05:07:11.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.186+0000 7f5e07e00700 1 -- 172.21.15.123:0/49185730 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e001037a0 msgr2=0x7f5e00070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:11.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.186+0000 7f5e07e00700 1 --2- 172.21.15.123:0/49185730 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e001037a0 0x7f5e00070800 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f5dec00eb40 tx=0x7f5dec00b040 comp rx=0 tx=0).stop 2024-03-30T05:07:11.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.187+0000 7f5e07e00700 1 -- 172.21.15.123:0/49185730 shutdown_connections 2024-03-30T05:07:11.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.187+0000 7f5e07e00700 1 --2- 172.21.15.123:0/49185730 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5de806cc20 0x7f5de806f0d0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:11.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.187+0000 7f5e07e00700 1 --2- 172.21.15.123:0/49185730 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5e00102e00 0x7f5e000702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:11.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.187+0000 7f5e07e00700 1 --2- 172.21.15.123:0/49185730 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e001037a0 0x7f5e00070800 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:11.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.187+0000 7f5e07e00700 1 -- 172.21.15.123:0/49185730 >> 172.21.15.123:0/49185730 conn(0x7f5e000f89b0 msgr2=0x7f5e000f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:07:11.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.187+0000 7f5e07e00700 1 -- 172.21.15.123:0/49185730 shutdown_connections 2024-03-30T05:07:11.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.187+0000 7f5e07e00700 1 -- 172.21.15.123:0/49185730 wait complete. 2024-03-30T05:07:11.199 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T05:07:11.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.321+0000 7f66b1800700 1 -- 172.21.15.123:0/2012447242 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66ac1037a0 msgr2=0x7f66ac0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:11.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.321+0000 7f66b1800700 1 --2- 172.21.15.123:0/2012447242 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66ac1037a0 0x7f66ac0fce10 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f6698004940 tx=0x7f6698009e30 comp rx=0 tx=0).stop 2024-03-30T05:07:11.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.321+0000 7f66b1800700 1 -- 172.21.15.123:0/2012447242 shutdown_connections 2024-03-30T05:07:11.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.321+0000 7f66b1800700 1 --2- 172.21.15.123:0/2012447242 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66ac1037a0 0x7f66ac0fce10 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:11.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.321+0000 7f66b1800700 1 --2- 172.21.15.123:0/2012447242 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f66ac102e00 0x7f66ac1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:11.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.321+0000 7f66b1800700 1 -- 172.21.15.123:0/2012447242 >> 172.21.15.123:0/2012447242 conn(0x7f66ac0f89b0 msgr2=0x7f66ac0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:07:11.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.321+0000 7f66b1800700 1 -- 172.21.15.123:0/2012447242 shutdown_connections 2024-03-30T05:07:11.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.321+0000 7f66b1800700 1 -- 172.21.15.123:0/2012447242 wait complete. 2024-03-30T05:07:11.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.322+0000 7f66b1800700 1 Processor -- start 2024-03-30T05:07:11.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.322+0000 7f66b1800700 1 -- start start 2024-03-30T05:07:11.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.322+0000 7f66b1800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f66ac102e00 0x7f66ac115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:07:11.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.322+0000 7f66b1800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66ac1037a0 0x7f66ac1161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:07:11.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.322+0000 7f66b1800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66ac116810 con 0x7f66ac1037a0 2024-03-30T05:07:11.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.322+0000 7f66b1800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66ac10fbe0 con 0x7f66ac102e00 2024-03-30T05:07:11.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.322+0000 7f66aa000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66ac1037a0 0x7f66ac1161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:07:11.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.322+0000 7f66aa000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66ac1037a0 0x7f66ac1161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58882/0 (socket says 172.21.15.123:58882) 2024-03-30T05:07:11.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.322+0000 7f66aa000700 1 -- 172.21.15.123:0/2915930985 learned_addr learned my addr 172.21.15.123:0/2915930985 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:07:11.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.323+0000 7f66aaa00700 1 --2- 172.21.15.123:0/2915930985 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f66ac102e00 0x7f66ac115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:07:11.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.323+0000 7f66aa000700 1 -- 172.21.15.123:0/2915930985 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f66ac102e00 msgr2=0x7f66ac115c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:11.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.323+0000 7f66aa000700 1 --2- 172.21.15.123:0/2915930985 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f66ac102e00 0x7f66ac115c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:11.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.323+0000 7f66aa000700 1 -- 172.21.15.123:0/2915930985 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6698005040 con 0x7f66ac1037a0 2024-03-30T05:07:11.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.323+0000 7f66aa000700 1 --2- 172.21.15.123:0/2915930985 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66ac1037a0 0x7f66ac1161c0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f6698008d50 tx=0x7f6698009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:07:11.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.323+0000 7f66a2a00700 1 -- 172.21.15.123:0/2915930985 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f669800e030 con 0x7f66ac1037a0 2024-03-30T05:07:11.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.323+0000 7f66a2a00700 1 -- 172.21.15.123:0/2915930985 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6698013de0 con 0x7f66ac1037a0 2024-03-30T05:07:11.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.323+0000 7f66b1800700 1 -- 172.21.15.123:0/2915930985 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66ac10fe60 con 0x7f66ac1037a0 2024-03-30T05:07:11.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.323+0000 7f66a2a00700 1 -- 172.21.15.123:0/2915930985 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f669801cd90 con 0x7f66ac1037a0 2024-03-30T05:07:11.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.324+0000 7f66b1800700 1 -- 172.21.15.123:0/2915930985 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66ac110420 con 0x7f66ac1037a0 2024-03-30T05:07:11.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.325+0000 7f66b1800700 1 -- 172.21.15.123:0/2915930985 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66ac0fe550 con 0x7f66ac1037a0 2024-03-30T05:07:11.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.326+0000 7f66a2a00700 1 -- 172.21.15.123:0/2915930985 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f6698013930 con 0x7f66ac1037a0 2024-03-30T05:07:11.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.327+0000 7f66a2a00700 1 --2- 172.21.15.123:0/2915930985 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f669006cc30 0x7f669006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:07:11.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.327+0000 7f66a2a00700 1 -- 172.21.15.123:0/2915930985 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f669808dd60 con 0x7f66ac1037a0 2024-03-30T05:07:11.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.327+0000 7f66aaa00700 1 --2- 172.21.15.123:0/2915930985 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f669006cc30 0x7f669006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:07:11.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.328+0000 7f66aaa00700 1 --2- 172.21.15.123:0/2915930985 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f669006cc30 0x7f669006f0e0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f669c004600 tx=0x7f669c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:07:11.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.330+0000 7f66a2a00700 1 -- 172.21.15.123:0/2915930985 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f669805b830 con 0x7f66ac1037a0 2024-03-30T05:07:11.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:07:10 smithi135 ceph-mon[106011]: pgmap v1080: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:11.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:07:10 smithi123 ceph-mon[139369]: pgmap v1080: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:11.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.552+0000 7f66b1800700 1 -- 172.21.15.123:0/2915930985 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f66ac109e10 con 0x7f669006cc30 2024-03-30T05:07:11.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.553+0000 7f66a2a00700 1 -- 172.21.15.123:0/2915930985 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f66ac109e10 con 0x7f669006cc30 2024-03-30T05:07:11.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.555+0000 7f66b1800700 1 -- 172.21.15.123:0/2915930985 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f669006cc30 msgr2=0x7f669006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:11.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.555+0000 7f66b1800700 1 --2- 172.21.15.123:0/2915930985 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f669006cc30 0x7f669006f0e0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f669c004600 tx=0x7f669c00b040 comp rx=0 tx=0).stop 2024-03-30T05:07:11.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.555+0000 7f66b1800700 1 -- 172.21.15.123:0/2915930985 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66ac1037a0 msgr2=0x7f66ac1161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:11.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.555+0000 7f66b1800700 1 --2- 172.21.15.123:0/2915930985 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66ac1037a0 0x7f66ac1161c0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f6698008d50 tx=0x7f6698009e30 comp rx=0 tx=0).stop 2024-03-30T05:07:11.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.555+0000 7f66b1800700 1 -- 172.21.15.123:0/2915930985 shutdown_connections 2024-03-30T05:07:11.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.555+0000 7f66b1800700 1 --2- 172.21.15.123:0/2915930985 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f669006cc30 0x7f669006f0e0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:11.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.555+0000 7f66b1800700 1 --2- 172.21.15.123:0/2915930985 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f66ac102e00 0x7f66ac115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:11.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.555+0000 7f66b1800700 1 --2- 172.21.15.123:0/2915930985 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66ac1037a0 0x7f66ac1161c0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:11.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.555+0000 7f66b1800700 1 -- 172.21.15.123:0/2915930985 >> 172.21.15.123:0/2915930985 conn(0x7f66ac0f89b0 msgr2=0x7f66ac0f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:07:11.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.555+0000 7f66b1800700 1 -- 172.21.15.123:0/2915930985 shutdown_connections 2024-03-30T05:07:11.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.555+0000 7f66b1800700 1 -- 172.21.15.123:0/2915930985 wait complete. 2024-03-30T05:07:11.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.688+0000 7f0f50000700 1 -- 172.21.15.123:0/217032784 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f481037a0 msgr2=0x7f0f480fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:11.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.688+0000 7f0f50000700 1 --2- 172.21.15.123:0/217032784 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f481037a0 0x7f0f480fce10 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f0f34004940 tx=0x7f0f34009e30 comp rx=0 tx=0).stop 2024-03-30T05:07:11.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.688+0000 7f0f50000700 1 -- 172.21.15.123:0/217032784 shutdown_connections 2024-03-30T05:07:11.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.688+0000 7f0f50000700 1 --2- 172.21.15.123:0/217032784 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f481037a0 0x7f0f480fce10 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:11.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.688+0000 7f0f50000700 1 --2- 172.21.15.123:0/217032784 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0f48102e00 0x7f0f481031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:11.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.688+0000 7f0f50000700 1 -- 172.21.15.123:0/217032784 >> 172.21.15.123:0/217032784 conn(0x7f0f480f89b0 msgr2=0x7f0f480fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:07:11.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.688+0000 7f0f50000700 1 -- 172.21.15.123:0/217032784 shutdown_connections 2024-03-30T05:07:11.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.688+0000 7f0f50000700 1 -- 172.21.15.123:0/217032784 wait complete. 2024-03-30T05:07:11.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.689+0000 7f0f50000700 1 Processor -- start 2024-03-30T05:07:11.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.689+0000 7f0f50000700 1 -- start start 2024-03-30T05:07:11.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.689+0000 7f0f50000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f48102e00 0x7f0f480702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:07:11.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.689+0000 7f0f50000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0f481037a0 0x7f0f48070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:07:11.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.689+0000 7f0f50000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f480720b0 con 0x7f0f48102e00 2024-03-30T05:07:11.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.689+0000 7f0f50000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f48072220 con 0x7f0f481037a0 2024-03-30T05:07:11.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.690+0000 7f0f4d600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f48102e00 0x7f0f480702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:07:11.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.690+0000 7f0f4d600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f48102e00 0x7f0f480702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58898/0 (socket says 172.21.15.123:58898) 2024-03-30T05:07:11.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.690+0000 7f0f4d600700 1 -- 172.21.15.123:0/1843262234 learned_addr learned my addr 172.21.15.123:0/1843262234 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:07:11.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.690+0000 7f0f4cc00700 1 --2- 172.21.15.123:0/1843262234 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0f481037a0 0x7f0f48070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:07:11.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.690+0000 7f0f4d600700 1 -- 172.21.15.123:0/1843262234 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0f481037a0 msgr2=0x7f0f48070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:11.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.690+0000 7f0f4d600700 1 --2- 172.21.15.123:0/1843262234 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0f481037a0 0x7f0f48070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:11.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.690+0000 7f0f4d600700 1 -- 172.21.15.123:0/1843262234 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f34005040 con 0x7f0f48102e00 2024-03-30T05:07:11.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.690+0000 7f0f4d600700 1 --2- 172.21.15.123:0/1843262234 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f48102e00 0x7f0f480702c0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c00eb70 tx=0x7f0f3c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:07:11.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.690+0000 7f0f45600700 1 -- 172.21.15.123:0/1843262234 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f3c007cc0 con 0x7f0f48102e00 2024-03-30T05:07:11.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.691+0000 7f0f45600700 1 -- 172.21.15.123:0/1843262234 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0f3c00f040 con 0x7f0f48102e00 2024-03-30T05:07:11.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.691+0000 7f0f50000700 1 -- 172.21.15.123:0/1843262234 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f48070e00 con 0x7f0f48102e00 2024-03-30T05:07:11.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.691+0000 7f0f45600700 1 -- 172.21.15.123:0/1843262234 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f3c00c9f0 con 0x7f0f48102e00 2024-03-30T05:07:11.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.691+0000 7f0f50000700 1 -- 172.21.15.123:0/1843262234 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f4811e700 con 0x7f0f48102e00 2024-03-30T05:07:11.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.692+0000 7f0f50000700 1 -- 172.21.15.123:0/1843262234 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f480fe550 con 0x7f0f48102e00 2024-03-30T05:07:11.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.693+0000 7f0f45600700 1 -- 172.21.15.123:0/1843262234 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f0f3c00cb50 con 0x7f0f48102e00 2024-03-30T05:07:11.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.693+0000 7f0f45600700 1 --2- 172.21.15.123:0/1843262234 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0f3006cc30 0x7f0f3006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:07:11.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.694+0000 7f0f45600700 1 -- 172.21.15.123:0/1843262234 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f0f3c08bbe0 con 0x7f0f48102e00 2024-03-30T05:07:11.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.694+0000 7f0f4cc00700 1 --2- 172.21.15.123:0/1843262234 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0f3006cc30 0x7f0f3006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:07:11.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.694+0000 7f0f4cc00700 1 --2- 172.21.15.123:0/1843262234 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0f3006cc30 0x7f0f3006f0e0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f0f48071c00 tx=0x7f0f34009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:07:11.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.698+0000 7f0f45600700 1 -- 172.21.15.123:0/1843262234 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0f3c059630 con 0x7f0f48102e00 2024-03-30T05:07:11.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.914+0000 7f0f50000700 1 -- 172.21.15.123:0/1843262234 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f0f4804bf00 con 0x7f0f3006cc30 2024-03-30T05:07:11.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.924+0000 7f0f45600700 1 -- 172.21.15.123:0/1843262234 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f0f4804bf00 con 0x7f0f3006cc30 2024-03-30T05:07:11.922 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T05:07:11.922 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (36m) 13m ago 48m 14.6M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T05:07:11.922 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (33m) 13m ago 49m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T05:07:11.922 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (33m) 13m ago 47m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T05:07:11.922 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (36m) 13m ago 48m 73.5M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T05:07:11.922 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (29m) 13m ago 42m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T05:07:11.922 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (29m) 13m ago 42m 16.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T05:07:11.923 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (13m) 13m ago 42m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 87b073a6c9ca 2024-03-30T05:07:11.923 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (29m) 13m ago 42m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T05:07:11.923 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (38m) 13m ago 50m 510M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T05:07:11.923 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (37m) 13m ago 47m 456M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T05:07:11.923 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (34m) 13m ago 50m 70.0M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T05:07:11.923 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (34m) 13m ago 47m 51.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T05:07:11.923 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (37m) 13m ago 49m 8636k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T05:07:11.923 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (37m) 13m ago 47m 8678k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T05:07:11.923 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (33m) 13m ago 46m 107M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T05:07:11.923 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (32m) 13m ago 46m 97.2M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T05:07:11.923 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (31m) 13m ago 45m 89.8M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T05:07:11.923 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (31m) 13m ago 45m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T05:07:11.923 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (30m) 13m ago 44m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T05:07:11.923 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (30m) 13m ago 44m 88.8M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T05:07:11.924 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (36m) 13m ago 48m 63.9M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T05:07:11.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.926+0000 7f0f50000700 1 -- 172.21.15.123:0/1843262234 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0f3006cc30 msgr2=0x7f0f3006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:11.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.926+0000 7f0f50000700 1 --2- 172.21.15.123:0/1843262234 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0f3006cc30 0x7f0f3006f0e0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f0f48071c00 tx=0x7f0f34009e30 comp rx=0 tx=0).stop 2024-03-30T05:07:11.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.926+0000 7f0f50000700 1 -- 172.21.15.123:0/1843262234 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f48102e00 msgr2=0x7f0f480702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:11.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.926+0000 7f0f50000700 1 --2- 172.21.15.123:0/1843262234 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f48102e00 0x7f0f480702c0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c00eb70 tx=0x7f0f3c00b040 comp rx=0 tx=0).stop 2024-03-30T05:07:11.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.926+0000 7f0f50000700 1 -- 172.21.15.123:0/1843262234 shutdown_connections 2024-03-30T05:07:11.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.926+0000 7f0f50000700 1 --2- 172.21.15.123:0/1843262234 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0f3006cc30 0x7f0f3006f0e0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:11.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.926+0000 7f0f50000700 1 --2- 172.21.15.123:0/1843262234 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f48102e00 0x7f0f480702c0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:11.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.926+0000 7f0f50000700 1 --2- 172.21.15.123:0/1843262234 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0f481037a0 0x7f0f48070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:11.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.926+0000 7f0f50000700 1 -- 172.21.15.123:0/1843262234 >> 172.21.15.123:0/1843262234 conn(0x7f0f480f89b0 msgr2=0x7f0f480f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:07:11.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.926+0000 7f0f50000700 1 -- 172.21.15.123:0/1843262234 shutdown_connections 2024-03-30T05:07:11.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:11.926+0000 7f0f50000700 1 -- 172.21.15.123:0/1843262234 wait complete. 2024-03-30T05:07:12.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.062+0000 7faff3000700 1 -- 172.21.15.123:0/4156029358 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafec102e00 msgr2=0x7fafec1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:12.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.062+0000 7faff3000700 1 --2- 172.21.15.123:0/4156029358 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafec102e00 0x7fafec1031d0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fafdc004940 tx=0x7fafdc009e30 comp rx=0 tx=0).stop 2024-03-30T05:07:12.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.062+0000 7faff3000700 1 -- 172.21.15.123:0/4156029358 shutdown_connections 2024-03-30T05:07:12.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.062+0000 7faff3000700 1 --2- 172.21.15.123:0/4156029358 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fafec1037a0 0x7fafec0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:12.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.062+0000 7faff3000700 1 --2- 172.21.15.123:0/4156029358 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafec102e00 0x7fafec1031d0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:12.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.062+0000 7faff3000700 1 -- 172.21.15.123:0/4156029358 >> 172.21.15.123:0/4156029358 conn(0x7fafec0f89b0 msgr2=0x7fafec0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:07:12.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.062+0000 7faff3000700 1 -- 172.21.15.123:0/4156029358 shutdown_connections 2024-03-30T05:07:12.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.062+0000 7faff3000700 1 -- 172.21.15.123:0/4156029358 wait complete. 2024-03-30T05:07:12.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.063+0000 7faff3000700 1 Processor -- start 2024-03-30T05:07:12.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.063+0000 7faff3000700 1 -- start start 2024-03-30T05:07:12.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.063+0000 7faff3000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafec102e00 0x7fafec115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:07:12.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.063+0000 7faff3000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fafec1037a0 0x7fafec116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:07:12.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.063+0000 7faff3000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafec116880 con 0x7fafec102e00 2024-03-30T05:07:12.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.063+0000 7faff3000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafec10fc50 con 0x7fafec1037a0 2024-03-30T05:07:12.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.064+0000 7fafebe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafec102e00 0x7fafec115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:07:12.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.064+0000 7fafebe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafec102e00 0x7fafec115cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58924/0 (socket says 172.21.15.123:58924) 2024-03-30T05:07:12.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.064+0000 7fafebe00700 1 -- 172.21.15.123:0/1652547211 learned_addr learned my addr 172.21.15.123:0/1652547211 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:07:12.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.064+0000 7fafebe00700 1 -- 172.21.15.123:0/1652547211 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fafec1037a0 msgr2=0x7fafec116230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:12.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.064+0000 7fafebe00700 1 --2- 172.21.15.123:0/1652547211 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fafec1037a0 0x7fafec116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:12.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.064+0000 7fafebe00700 1 -- 172.21.15.123:0/1652547211 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafdc005040 con 0x7fafec102e00 2024-03-30T05:07:12.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.064+0000 7fafebe00700 1 --2- 172.21.15.123:0/1652547211 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafec102e00 0x7fafec115cf0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fafdc009390 tx=0x7fafdc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:07:12.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.064+0000 7fafe3e00700 1 -- 172.21.15.123:0/1652547211 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fafdc00e040 con 0x7fafec102e00 2024-03-30T05:07:12.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.064+0000 7fafe3e00700 1 -- 172.21.15.123:0/1652547211 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fafdc013420 con 0x7fafec102e00 2024-03-30T05:07:12.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.064+0000 7faff3000700 1 -- 172.21.15.123:0/1652547211 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafec10fed0 con 0x7fafec102e00 2024-03-30T05:07:12.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.064+0000 7fafe3e00700 1 -- 172.21.15.123:0/1652547211 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fafdc01c630 con 0x7fafec102e00 2024-03-30T05:07:12.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.065+0000 7faff3000700 1 -- 172.21.15.123:0/1652547211 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafec110490 con 0x7fafec102e00 2024-03-30T05:07:12.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.066+0000 7faff3000700 1 -- 172.21.15.123:0/1652547211 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafec0fe550 con 0x7fafec102e00 2024-03-30T05:07:12.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.066+0000 7fafe3e00700 1 -- 172.21.15.123:0/1652547211 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fafdc0139e0 con 0x7fafec102e00 2024-03-30T05:07:12.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.067+0000 7fafe3e00700 1 --2- 172.21.15.123:0/1652547211 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fafd006cc30 0x7fafd006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:07:12.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.067+0000 7fafe3e00700 1 -- 172.21.15.123:0/1652547211 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fafdc08d320 con 0x7fafec102e00 2024-03-30T05:07:12.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.068+0000 7fafeb400700 1 --2- 172.21.15.123:0/1652547211 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fafd006cc30 0x7fafd006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:07:12.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.068+0000 7fafeb400700 1 --2- 172.21.15.123:0/1652547211 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fafd006cc30 0x7fafd006f0e0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fafec111590 tx=0x7fafd8009ce0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:07:12.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.072+0000 7fafe3e00700 1 -- 172.21.15.123:0/1652547211 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fafdc05ae20 con 0x7fafec102e00 2024-03-30T05:07:12.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:07:11 smithi135 ceph-mon[106011]: from='client.35944 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:07:12.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.360+0000 7faff3000700 1 -- 172.21.15.123:0/1652547211 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fafec109e30 con 0x7fafec102e00 2024-03-30T05:07:12.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.360+0000 7fafe3e00700 1 -- 172.21.15.123:0/1652547211 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7fafdc05a9b0 con 0x7fafec102e00 2024-03-30T05:07:12.358 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T05:07:12.358 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T05:07:12.359 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T05:07:12.359 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:07:12.359 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T05:07:12.359 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T05:07:12.359 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:07:12.359 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T05:07:12.359 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T05:07:12.359 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:07:12.359 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T05:07:12.359 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 4 2024-03-30T05:07:12.359 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:07:12.359 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T05:07:12.359 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 14 2024-03-30T05:07:12.360 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T05:07:12.360 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T05:07:12.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.363+0000 7faff3000700 1 -- 172.21.15.123:0/1652547211 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fafd006cc30 msgr2=0x7fafd006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:12.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.363+0000 7faff3000700 1 --2- 172.21.15.123:0/1652547211 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fafd006cc30 0x7fafd006f0e0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fafec111590 tx=0x7fafd8009ce0 comp rx=0 tx=0).stop 2024-03-30T05:07:12.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.363+0000 7faff3000700 1 -- 172.21.15.123:0/1652547211 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafec102e00 msgr2=0x7fafec115cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:12.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.363+0000 7faff3000700 1 --2- 172.21.15.123:0/1652547211 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafec102e00 0x7fafec115cf0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fafdc009390 tx=0x7fafdc009e30 comp rx=0 tx=0).stop 2024-03-30T05:07:12.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.363+0000 7faff3000700 1 -- 172.21.15.123:0/1652547211 shutdown_connections 2024-03-30T05:07:12.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.363+0000 7faff3000700 1 --2- 172.21.15.123:0/1652547211 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fafd006cc30 0x7fafd006f0e0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:12.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.363+0000 7faff3000700 1 --2- 172.21.15.123:0/1652547211 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafec102e00 0x7fafec115cf0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:12.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.363+0000 7faff3000700 1 --2- 172.21.15.123:0/1652547211 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fafec1037a0 0x7fafec116230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:12.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.363+0000 7faff3000700 1 -- 172.21.15.123:0/1652547211 >> 172.21.15.123:0/1652547211 conn(0x7fafec0f89b0 msgr2=0x7fafec0f99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:07:12.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.363+0000 7faff3000700 1 -- 172.21.15.123:0/1652547211 shutdown_connections 2024-03-30T05:07:12.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.363+0000 7faff3000700 1 -- 172.21.15.123:0/1652547211 wait complete. 2024-03-30T05:07:12.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:07:11 smithi123 ceph-mon[139369]: from='client.35944 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:07:12.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.495+0000 7fc4d4800700 1 -- 172.21.15.123:0/44033953 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4cc1037c0 msgr2=0x7fc4cc0fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:12.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.495+0000 7fc4d4800700 1 --2- 172.21.15.123:0/44033953 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4cc1037c0 0x7fc4cc0fce30 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fc4bc004940 tx=0x7fc4bc009e30 comp rx=0 tx=0).stop 2024-03-30T05:07:12.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.495+0000 7fc4d4800700 1 -- 172.21.15.123:0/44033953 shutdown_connections 2024-03-30T05:07:12.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.495+0000 7fc4d4800700 1 --2- 172.21.15.123:0/44033953 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4cc1037c0 0x7fc4cc0fce30 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:12.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.495+0000 7fc4d4800700 1 --2- 172.21.15.123:0/44033953 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc4cc102e20 0x7fc4cc1031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:12.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.495+0000 7fc4d4800700 1 -- 172.21.15.123:0/44033953 >> 172.21.15.123:0/44033953 conn(0x7fc4cc0f89d0 msgr2=0x7fc4cc0fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:07:12.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.495+0000 7fc4d4800700 1 -- 172.21.15.123:0/44033953 shutdown_connections 2024-03-30T05:07:12.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.495+0000 7fc4d4800700 1 -- 172.21.15.123:0/44033953 wait complete. 2024-03-30T05:07:12.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.496+0000 7fc4d4800700 1 Processor -- start 2024-03-30T05:07:12.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.496+0000 7fc4d4800700 1 -- start start 2024-03-30T05:07:12.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.497+0000 7fc4d4800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4cc102e20 0x7fc4cc0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:07:12.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.497+0000 7fc4d4800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc4cc1037c0 0x7fc4cc070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:07:12.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.497+0000 7fc4d4800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4cc070d90 con 0x7fc4cc102e20 2024-03-30T05:07:12.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.497+0000 7fc4d4800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4cc070ed0 con 0x7fc4cc1037c0 2024-03-30T05:07:12.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.497+0000 7fc4d1e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4cc102e20 0x7fc4cc0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:07:12.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.497+0000 7fc4d1e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4cc102e20 0x7fc4cc0702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58946/0 (socket says 172.21.15.123:58946) 2024-03-30T05:07:12.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.497+0000 7fc4d1e00700 1 -- 172.21.15.123:0/3363331375 learned_addr learned my addr 172.21.15.123:0/3363331375 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:07:12.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.497+0000 7fc4d1400700 1 --2- 172.21.15.123:0/3363331375 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc4cc1037c0 0x7fc4cc070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:07:12.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.498+0000 7fc4d1e00700 1 -- 172.21.15.123:0/3363331375 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc4cc1037c0 msgr2=0x7fc4cc070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:12.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.498+0000 7fc4d1e00700 1 --2- 172.21.15.123:0/3363331375 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc4cc1037c0 0x7fc4cc070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:12.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.498+0000 7fc4d1e00700 1 -- 172.21.15.123:0/3363331375 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4bc005040 con 0x7fc4cc102e20 2024-03-30T05:07:12.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.498+0000 7fc4d1e00700 1 --2- 172.21.15.123:0/3363331375 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4cc102e20 0x7fc4cc0702c0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fc4c000eb40 tx=0x7fc4c000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:07:12.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.498+0000 7fc4c6000700 1 -- 172.21.15.123:0/3363331375 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc4c0007cc0 con 0x7fc4cc102e20 2024-03-30T05:07:12.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.498+0000 7fc4c6000700 1 -- 172.21.15.123:0/3363331375 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc4c000f040 con 0x7fc4cc102e20 2024-03-30T05:07:12.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.498+0000 7fc4d4800700 1 -- 172.21.15.123:0/3363331375 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4cc071120 con 0x7fc4cc102e20 2024-03-30T05:07:12.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.498+0000 7fc4c6000700 1 -- 172.21.15.123:0/3363331375 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc4c0009570 con 0x7fc4cc102e20 2024-03-30T05:07:12.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.498+0000 7fc4d4800700 1 -- 172.21.15.123:0/3363331375 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4cc11a170 con 0x7fc4cc102e20 2024-03-30T05:07:12.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.499+0000 7fc4d4800700 1 -- 172.21.15.123:0/3363331375 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4cc0fe570 con 0x7fc4cc102e20 2024-03-30T05:07:12.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.500+0000 7fc4c6000700 1 -- 172.21.15.123:0/3363331375 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fc4c00096d0 con 0x7fc4cc102e20 2024-03-30T05:07:12.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.501+0000 7fc4c6000700 1 --2- 172.21.15.123:0/3363331375 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc4b406cc30 0x7fc4b406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:07:12.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.501+0000 7fc4c6000700 1 -- 172.21.15.123:0/3363331375 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc4c008bca0 con 0x7fc4cc102e20 2024-03-30T05:07:12.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.501+0000 7fc4d1400700 1 --2- 172.21.15.123:0/3363331375 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc4b406cc30 0x7fc4b406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:07:12.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.502+0000 7fc4d1400700 1 --2- 172.21.15.123:0/3363331375 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc4b406cc30 0x7fc4b406f0e0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fc4bc009fd0 tx=0x7fc4bc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:07:12.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.505+0000 7fc4c6000700 1 -- 172.21.15.123:0/3363331375 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc4c0055d10 con 0x7fc4cc102e20 2024-03-30T05:07:12.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.750+0000 7fc4d4800700 1 -- 172.21.15.123:0/3363331375 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fc4cc04bed0 con 0x7fc4cc102e20 2024-03-30T05:07:12.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.750+0000 7fc4c6000700 1 -- 172.21.15.123:0/3363331375 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 34 v34) v1 ==== 76+0+1843 (secure 0 0 0) 0x7fc4c0059330 con 0x7fc4cc102e20 2024-03-30T05:07:12.749 INFO:teuthology.orchestra.run.smithi123.stdout:e34 2024-03-30T05:07:12.749 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T05:07:12.749 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T05:07:12.749 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T05:07:12.749 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:07:12.749 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T05:07:12.749 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T05:07:12.749 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T05:07:12.749 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T05:07:12.749 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T05:07:12.749 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T05:07:12.749 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T05:07:12.749 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T05:07:12.750 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T05:07:12.750 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T05:07:12.750 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T05:07:12.750 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T05:07:12.750 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T05:07:12.750 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T05:07:12.750 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T05:07:12.750 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T05:07:12.750 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T05:07:12.750 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T05:07:12.750 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T05:07:12.750 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T05:07:12.750 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T05:07:12.751 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T05:07:12.751 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T05:07:12.751 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T05:07:12.751 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T05:07:12.751 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T05:07:12.751 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T05:07:12.751 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T05:07:12.751 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:07:12.751 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:07:12.751 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T05:07:12.751 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:07:12.751 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:07:12.751 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:07:12.751 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:07:12.751 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44975} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/3788988014,v1:172.21.15.135:6827/3788988014] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:07:12.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.753+0000 7fc4d4800700 1 -- 172.21.15.123:0/3363331375 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc4b406cc30 msgr2=0x7fc4b406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:12.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.753+0000 7fc4d4800700 1 --2- 172.21.15.123:0/3363331375 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc4b406cc30 0x7fc4b406f0e0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fc4bc009fd0 tx=0x7fc4bc009e30 comp rx=0 tx=0).stop 2024-03-30T05:07:12.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.753+0000 7fc4d4800700 1 -- 172.21.15.123:0/3363331375 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4cc102e20 msgr2=0x7fc4cc0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:12.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.753+0000 7fc4d4800700 1 --2- 172.21.15.123:0/3363331375 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4cc102e20 0x7fc4cc0702c0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fc4c000eb40 tx=0x7fc4c000b040 comp rx=0 tx=0).stop 2024-03-30T05:07:12.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.753+0000 7fc4d4800700 1 -- 172.21.15.123:0/3363331375 shutdown_connections 2024-03-30T05:07:12.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.753+0000 7fc4d4800700 1 --2- 172.21.15.123:0/3363331375 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc4b406cc30 0x7fc4b406f0e0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:12.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.753+0000 7fc4d4800700 1 --2- 172.21.15.123:0/3363331375 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4cc102e20 0x7fc4cc0702c0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:12.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.753+0000 7fc4d4800700 1 --2- 172.21.15.123:0/3363331375 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc4cc1037c0 0x7fc4cc070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:12.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.753+0000 7fc4d4800700 1 -- 172.21.15.123:0/3363331375 >> 172.21.15.123:0/3363331375 conn(0x7fc4cc0f89d0 msgr2=0x7fc4cc0f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:07:12.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.753+0000 7fc4d4800700 1 -- 172.21.15.123:0/3363331375 shutdown_connections 2024-03-30T05:07:12.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.753+0000 7fc4d4800700 1 -- 172.21.15.123:0/3363331375 wait complete. 2024-03-30T05:07:12.753 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 34 2024-03-30T05:07:12.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.888+0000 7f9c3ea00700 1 -- 172.21.15.123:0/1187107835 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c38102e00 msgr2=0x7f9c381031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:12.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.888+0000 7f9c3ea00700 1 --2- 172.21.15.123:0/1187107835 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c38102e00 0x7f9c381031d0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f9c28004940 tx=0x7f9c28009e30 comp rx=0 tx=0).stop 2024-03-30T05:07:12.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.888+0000 7f9c3ea00700 1 -- 172.21.15.123:0/1187107835 shutdown_connections 2024-03-30T05:07:12.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.888+0000 7f9c3ea00700 1 --2- 172.21.15.123:0/1187107835 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c381037a0 0x7f9c380fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:12.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.888+0000 7f9c3ea00700 1 --2- 172.21.15.123:0/1187107835 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c38102e00 0x7f9c381031d0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:12.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.888+0000 7f9c3ea00700 1 -- 172.21.15.123:0/1187107835 >> 172.21.15.123:0/1187107835 conn(0x7f9c380f89b0 msgr2=0x7f9c380fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:07:12.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.888+0000 7f9c3ea00700 1 -- 172.21.15.123:0/1187107835 shutdown_connections 2024-03-30T05:07:12.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.888+0000 7f9c3ea00700 1 -- 172.21.15.123:0/1187107835 wait complete. 2024-03-30T05:07:12.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.889+0000 7f9c3ea00700 1 Processor -- start 2024-03-30T05:07:12.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.889+0000 7f9c3ea00700 1 -- start start 2024-03-30T05:07:12.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.889+0000 7f9c3ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c38102e00 0x7f9c380702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:07:12.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.889+0000 7f9c3ea00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c381037a0 0x7f9c38070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:07:12.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.889+0000 7f9c3ea00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c38070d90 con 0x7f9c38102e00 2024-03-30T05:07:12.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.889+0000 7f9c3ea00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c38070ed0 con 0x7f9c381037a0 2024-03-30T05:07:12.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.889+0000 7f9c37e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c38102e00 0x7f9c380702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:07:12.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.889+0000 7f9c37e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c38102e00 0x7f9c380702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58974/0 (socket says 172.21.15.123:58974) 2024-03-30T05:07:12.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.889+0000 7f9c37e00700 1 -- 172.21.15.123:0/3937979932 learned_addr learned my addr 172.21.15.123:0/3937979932 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:07:12.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.890+0000 7f9c37e00700 1 -- 172.21.15.123:0/3937979932 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c381037a0 msgr2=0x7f9c38070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:12.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.890+0000 7f9c37e00700 1 --2- 172.21.15.123:0/3937979932 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c381037a0 0x7f9c38070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:12.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.890+0000 7f9c37e00700 1 -- 172.21.15.123:0/3937979932 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c28005040 con 0x7f9c38102e00 2024-03-30T05:07:12.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.890+0000 7f9c37e00700 1 --2- 172.21.15.123:0/3937979932 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c38102e00 0x7f9c380702c0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f9c28009390 tx=0x7f9c28009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:07:12.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.890+0000 7f9c2fe00700 1 -- 172.21.15.123:0/3937979932 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c2800e040 con 0x7f9c38102e00 2024-03-30T05:07:12.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.890+0000 7f9c2fe00700 1 -- 172.21.15.123:0/3937979932 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9c28013420 con 0x7f9c38102e00 2024-03-30T05:07:12.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.890+0000 7f9c3ea00700 1 -- 172.21.15.123:0/3937979932 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c38071120 con 0x7f9c38102e00 2024-03-30T05:07:12.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.890+0000 7f9c2fe00700 1 -- 172.21.15.123:0/3937979932 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c2801c630 con 0x7f9c38102e00 2024-03-30T05:07:12.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.890+0000 7f9c3ea00700 1 -- 172.21.15.123:0/3937979932 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c3811a170 con 0x7f9c38102e00 2024-03-30T05:07:12.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.891+0000 7f9c3ea00700 1 -- 172.21.15.123:0/3937979932 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c380fe550 con 0x7f9c38102e00 2024-03-30T05:07:12.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.892+0000 7f9c2fe00700 1 -- 172.21.15.123:0/3937979932 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f9c280139e0 con 0x7f9c38102e00 2024-03-30T05:07:12.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.893+0000 7f9c2fe00700 1 --2- 172.21.15.123:0/3937979932 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9c1c06cc30 0x7f9c1c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:07:12.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.893+0000 7f9c2fe00700 1 -- 172.21.15.123:0/3937979932 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f9c2808d260 con 0x7f9c38102e00 2024-03-30T05:07:12.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.893+0000 7f9c37400700 1 --2- 172.21.15.123:0/3937979932 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9c1c06cc30 0x7f9c1c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:07:12.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.894+0000 7f9c37400700 1 --2- 172.21.15.123:0/3937979932 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9c1c06cc30 0x7f9c1c06f0e0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f9c24009b80 tx=0x7f9c24009d60 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:07:12.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:12.898+0000 7f9c2fe00700 1 -- 172.21.15.123:0/3937979932 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9c280572a0 con 0x7f9c38102e00 2024-03-30T05:07:13.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.126+0000 7f9c3ea00700 1 -- 172.21.15.123:0/3937979932 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9c3804bed0 con 0x7f9c1c06cc30 2024-03-30T05:07:13.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.127+0000 7f9c2fe00700 1 -- 172.21.15.123:0/3937979932 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f9c3804bed0 con 0x7f9c1c06cc30 2024-03-30T05:07:13.125 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T05:07:13.125 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T05:07:13.126 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T05:07:13.126 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T05:07:13.126 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T05:07:13.126 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T05:07:13.126 INFO:teuthology.orchestra.run.smithi123.stdout: "mds", 2024-03-30T05:07:13.126 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T05:07:13.126 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T05:07:13.126 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T05:07:13.126 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T05:07:13.126 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "16/21 daemons upgraded", 2024-03-30T05:07:13.126 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T05:07:13.126 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T05:07:13.126 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T05:07:13.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.130+0000 7f9c3ea00700 1 -- 172.21.15.123:0/3937979932 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9c1c06cc30 msgr2=0x7f9c1c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:13.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.130+0000 7f9c3ea00700 1 --2- 172.21.15.123:0/3937979932 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9c1c06cc30 0x7f9c1c06f0e0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f9c24009b80 tx=0x7f9c24009d60 comp rx=0 tx=0).stop 2024-03-30T05:07:13.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.130+0000 7f9c3ea00700 1 -- 172.21.15.123:0/3937979932 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c38102e00 msgr2=0x7f9c380702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:13.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.130+0000 7f9c3ea00700 1 --2- 172.21.15.123:0/3937979932 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c38102e00 0x7f9c380702c0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f9c28009390 tx=0x7f9c28009e30 comp rx=0 tx=0).stop 2024-03-30T05:07:13.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.130+0000 7f9c3ea00700 1 -- 172.21.15.123:0/3937979932 shutdown_connections 2024-03-30T05:07:13.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.130+0000 7f9c3ea00700 1 --2- 172.21.15.123:0/3937979932 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9c1c06cc30 0x7f9c1c06f0e0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:13.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.130+0000 7f9c3ea00700 1 --2- 172.21.15.123:0/3937979932 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c38102e00 0x7f9c380702c0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:13.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.130+0000 7f9c3ea00700 1 --2- 172.21.15.123:0/3937979932 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c381037a0 0x7f9c38070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:13.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.130+0000 7f9c3ea00700 1 -- 172.21.15.123:0/3937979932 >> 172.21.15.123:0/3937979932 conn(0x7f9c380f89b0 msgr2=0x7f9c380f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:07:13.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.130+0000 7f9c3ea00700 1 -- 172.21.15.123:0/3937979932 shutdown_connections 2024-03-30T05:07:13.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.130+0000 7f9c3ea00700 1 -- 172.21.15.123:0/3937979932 wait complete. 2024-03-30T05:07:13.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.262+0000 7f5883e00700 1 -- 172.21.15.123:0/30944475 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f587c1037a0 msgr2=0x7f587c0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:13.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.262+0000 7f5883e00700 1 --2- 172.21.15.123:0/30944475 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f587c1037a0 0x7f587c0fce10 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f5868004940 tx=0x7f5868009e30 comp rx=0 tx=0).stop 2024-03-30T05:07:13.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.262+0000 7f5883e00700 1 -- 172.21.15.123:0/30944475 shutdown_connections 2024-03-30T05:07:13.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.262+0000 7f5883e00700 1 --2- 172.21.15.123:0/30944475 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f587c1037a0 0x7f587c0fce10 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:13.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.262+0000 7f5883e00700 1 --2- 172.21.15.123:0/30944475 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f587c102e00 0x7f587c1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:13.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.262+0000 7f5883e00700 1 -- 172.21.15.123:0/30944475 >> 172.21.15.123:0/30944475 conn(0x7f587c0f89b0 msgr2=0x7f587c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:07:13.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.262+0000 7f5883e00700 1 -- 172.21.15.123:0/30944475 shutdown_connections 2024-03-30T05:07:13.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.262+0000 7f5883e00700 1 -- 172.21.15.123:0/30944475 wait complete. 2024-03-30T05:07:13.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.263+0000 7f5883e00700 1 Processor -- start 2024-03-30T05:07:13.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.263+0000 7f5883e00700 1 -- start start 2024-03-30T05:07:13.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.263+0000 7f5883e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f587c102e00 0x7f587c113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:07:13.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.263+0000 7f5883e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f587c1037a0 0x7f587c113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:07:13.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.263+0000 7f5883e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f587c114600 con 0x7f587c102e00 2024-03-30T05:07:13.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.263+0000 7f5883e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f587c10d9f0 con 0x7f587c1037a0 2024-03-30T05:07:13.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.263+0000 7f5881400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f587c102e00 0x7f587c113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:07:13.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.263+0000 7f5881400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f587c102e00 0x7f587c113a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58990/0 (socket says 172.21.15.123:58990) 2024-03-30T05:07:13.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.263+0000 7f5881400700 1 -- 172.21.15.123:0/958804061 learned_addr learned my addr 172.21.15.123:0/958804061 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:07:13.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.264+0000 7f5880a00700 1 --2- 172.21.15.123:0/958804061 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f587c1037a0 0x7f587c113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:07:13.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.264+0000 7f5881400700 1 -- 172.21.15.123:0/958804061 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f587c1037a0 msgr2=0x7f587c113fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:13.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.264+0000 7f5881400700 1 --2- 172.21.15.123:0/958804061 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f587c1037a0 0x7f587c113fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:13.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.264+0000 7f5881400700 1 -- 172.21.15.123:0/958804061 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5868005040 con 0x7f587c102e00 2024-03-30T05:07:13.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.264+0000 7f5881400700 1 --2- 172.21.15.123:0/958804061 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f587c102e00 0x7f587c113a70 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f587000eb40 tx=0x7f587000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:07:13.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.264+0000 7f5879600700 1 -- 172.21.15.123:0/958804061 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5870007cc0 con 0x7f587c102e00 2024-03-30T05:07:13.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.264+0000 7f5879600700 1 -- 172.21.15.123:0/958804061 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f587000f040 con 0x7f587c102e00 2024-03-30T05:07:13.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.264+0000 7f5883e00700 1 -- 172.21.15.123:0/958804061 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f587c10dc70 con 0x7f587c102e00 2024-03-30T05:07:13.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.264+0000 7f5879600700 1 -- 172.21.15.123:0/958804061 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5870009570 con 0x7f587c102e00 2024-03-30T05:07:13.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.264+0000 7f5883e00700 1 -- 172.21.15.123:0/958804061 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f587c10e230 con 0x7f587c102e00 2024-03-30T05:07:13.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.265+0000 7f5883e00700 1 -- 172.21.15.123:0/958804061 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f587c0fe550 con 0x7f587c102e00 2024-03-30T05:07:13.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.266+0000 7f5879600700 1 -- 172.21.15.123:0/958804061 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f58700096d0 con 0x7f587c102e00 2024-03-30T05:07:13.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.267+0000 7f5879600700 1 --2- 172.21.15.123:0/958804061 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f586406cc30 0x7f586406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:07:13.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.267+0000 7f5879600700 1 -- 172.21.15.123:0/958804061 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f587008bda0 con 0x7f587c102e00 2024-03-30T05:07:13.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.268+0000 7f5880a00700 1 --2- 172.21.15.123:0/958804061 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f586406cc30 0x7f586406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:07:13.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.268+0000 7f5880a00700 1 --2- 172.21.15.123:0/958804061 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f586406cc30 0x7f586406f0e0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f586800b0d0 tx=0x7f5868009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:07:13.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.272+0000 7f5879600700 1 -- 172.21.15.123:0/958804061 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5870059820 con 0x7f587c102e00 2024-03-30T05:07:13.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:07:13 smithi135 ceph-mon[106011]: from='client.35948 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:07:13.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:07:13 smithi135 ceph-mon[106011]: pgmap v1081: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:13.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:07:13 smithi135 ceph-mon[106011]: from='client.35952 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:07:13.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:07:13 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/1652547211' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:07:13.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:07:13 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3363331375' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T05:07:13.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:07:12 smithi123 ceph-mon[139369]: from='client.35948 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:07:13.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:07:12 smithi123 ceph-mon[139369]: pgmap v1081: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:13.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:07:12 smithi123 ceph-mon[139369]: from='client.35952 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:07:13.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:07:12 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/1652547211' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:07:13.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:07:12 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3363331375' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T05:07:13.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.558+0000 7f5883e00700 1 -- 172.21.15.123:0/958804061 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f587c109da0 con 0x7f587c102e00 2024-03-30T05:07:13.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.558+0000 7f5879600700 1 -- 172.21.15.123:0/958804061 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f5870019e10 con 0x7f587c102e00 2024-03-30T05:07:13.556 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T05:07:13.557 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-30T05:07:13.557 INFO:teuthology.orchestra.run.smithi123.stdout: host smithi135 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi135/config 2024-03-30T05:07:13.557 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-30T05:07:13.557 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T05:07:13.557 INFO:teuthology.orchestra.run.smithi123.stdout:Traceback (most recent call last): 2024-03-30T05:07:13.557 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-30T05:07:13.557 INFO:teuthology.orchestra.run.smithi123.stdout: "__main__", mod_spec) 2024-03-30T05:07:13.557 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-30T05:07:13.557 INFO:teuthology.orchestra.run.smithi123.stdout: exec(code, run_globals) 2024-03-30T05:07:13.557 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10889, in 2024-03-30T05:07:13.557 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10877, in main 2024-03-30T05:07:13.557 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-30T05:07:13.557 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-30T05:07:13.557 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-30T05:07:13.558 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-30T05:07:13.558 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-03-30T05:07:13.558 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-30T05:07:13.558 INFO:teuthology.orchestra.run.smithi123.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T05:07:13.558 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T05:07:13.558 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T05:07:13.558 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T05:07:13.558 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T05:07:13.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.560+0000 7f5883e00700 1 -- 172.21.15.123:0/958804061 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f586406cc30 msgr2=0x7f586406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:13.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.560+0000 7f5883e00700 1 --2- 172.21.15.123:0/958804061 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f586406cc30 0x7f586406f0e0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f586800b0d0 tx=0x7f5868009e30 comp rx=0 tx=0).stop 2024-03-30T05:07:13.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.560+0000 7f5883e00700 1 -- 172.21.15.123:0/958804061 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f587c102e00 msgr2=0x7f587c113a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:13.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.560+0000 7f5883e00700 1 --2- 172.21.15.123:0/958804061 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f587c102e00 0x7f587c113a70 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f587000eb40 tx=0x7f587000b040 comp rx=0 tx=0).stop 2024-03-30T05:07:13.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.561+0000 7f5883e00700 1 -- 172.21.15.123:0/958804061 shutdown_connections 2024-03-30T05:07:13.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.561+0000 7f5883e00700 1 --2- 172.21.15.123:0/958804061 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f586406cc30 0x7f586406f0e0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:13.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.561+0000 7f5883e00700 1 --2- 172.21.15.123:0/958804061 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f587c102e00 0x7f587c113a70 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:13.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.561+0000 7f5883e00700 1 --2- 172.21.15.123:0/958804061 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f587c1037a0 0x7f587c113fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:13.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.561+0000 7f5883e00700 1 -- 172.21.15.123:0/958804061 >> 172.21.15.123:0/958804061 conn(0x7f587c0f89b0 msgr2=0x7f587c0f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:07:13.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.561+0000 7f5883e00700 1 -- 172.21.15.123:0/958804061 shutdown_connections 2024-03-30T05:07:13.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:13.561+0000 7f5883e00700 1 -- 172.21.15.123:0/958804061 wait complete. 2024-03-30T05:07:14.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:07:14 smithi135 ceph-mon[106011]: from='client.35964 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:07:14.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:07:14 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/958804061' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T05:07:14.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:07:14 smithi123 ceph-mon[139369]: from='client.35964 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:07:14.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:07:14 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/958804061' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T05:07:15.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:07:15 smithi135 ceph-mon[106011]: pgmap v1082: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:15.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:07:15 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:07:15.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:07:15 smithi123 ceph-mon[139369]: pgmap v1082: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:15.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:07:15 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:07:17.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:07:17 smithi135 ceph-mon[106011]: pgmap v1083: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:17.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:07:17 smithi123 ceph-mon[139369]: pgmap v1083: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:19.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:07:19 smithi135 ceph-mon[106011]: pgmap v1084: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:19.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:07:19 smithi123 ceph-mon[139369]: pgmap v1084: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:21.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:07:21 smithi135 ceph-mon[106011]: pgmap v1085: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:21.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:07:21 smithi123 ceph-mon[139369]: pgmap v1085: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:23.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:07:23 smithi135 ceph-mon[106011]: pgmap v1086: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:23.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:07:23 smithi123 ceph-mon[139369]: pgmap v1086: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:25.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:07:25 smithi135 ceph-mon[106011]: pgmap v1087: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:25.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:07:25 smithi123 ceph-mon[139369]: pgmap v1087: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:27.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:07:27 smithi135 ceph-mon[106011]: pgmap v1088: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:27.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:07:27 smithi123 ceph-mon[139369]: pgmap v1088: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:29.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:07:29 smithi135 ceph-mon[106011]: pgmap v1089: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:29.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:07:29 smithi123 ceph-mon[139369]: pgmap v1089: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:30.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:07:30 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:07:30.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:07:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:07:31.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:07:31 smithi135 ceph-mon[106011]: pgmap v1090: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:31.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:07:31 smithi123 ceph-mon[139369]: pgmap v1090: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:33.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:07:33 smithi135 ceph-mon[106011]: pgmap v1091: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:33.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:07:33 smithi123 ceph-mon[139369]: pgmap v1091: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:35.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:07:35 smithi135 ceph-mon[106011]: pgmap v1092: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:35.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:07:35 smithi123 ceph-mon[139369]: pgmap v1092: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:37.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:07:37 smithi135 ceph-mon[106011]: pgmap v1093: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:37.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:07:37 smithi123 ceph-mon[139369]: pgmap v1093: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:39.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:07:39 smithi135 ceph-mon[106011]: pgmap v1094: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:39.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:07:39 smithi123 ceph-mon[139369]: pgmap v1094: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:41.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:07:41 smithi135 ceph-mon[106011]: pgmap v1095: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:41.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:07:41 smithi123 ceph-mon[139369]: pgmap v1095: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:43.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:07:43 smithi135 ceph-mon[106011]: pgmap v1096: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:43.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:07:43 smithi123 ceph-mon[139369]: pgmap v1096: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:43.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:43.708+0000 7fbdb9600700 1 -- 172.21.15.123:0/1157598718 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdb40708a0 msgr2=0x7fbdb4070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:43.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:43.708+0000 7fbdb9600700 1 --2- 172.21.15.123:0/1157598718 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdb40708a0 0x7fbdb4070d10 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fbda0004970 tx=0x7fbda0009e30 comp rx=0 tx=0).stop 2024-03-30T05:07:43.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:43.708+0000 7fbdb9600700 1 -- 172.21.15.123:0/1157598718 shutdown_connections 2024-03-30T05:07:43.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:43.708+0000 7fbdb9600700 1 --2- 172.21.15.123:0/1157598718 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdb40708a0 0x7fbdb4070d10 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:43.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:43.708+0000 7fbdb9600700 1 --2- 172.21.15.123:0/1157598718 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbdb4071f90 0x7fbdb4070360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:43.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:43.708+0000 7fbdb9600700 1 -- 172.21.15.123:0/1157598718 >> 172.21.15.123:0/1157598718 conn(0x7fbdb40f89b0 msgr2=0x7fbdb40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:07:43.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:43.708+0000 7fbdb9600700 1 -- 172.21.15.123:0/1157598718 shutdown_connections 2024-03-30T05:07:43.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:43.708+0000 7fbdb9600700 1 -- 172.21.15.123:0/1157598718 wait complete. 2024-03-30T05:07:43.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:43.709+0000 7fbdb9600700 1 Processor -- start 2024-03-30T05:07:43.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:43.709+0000 7fbdb9600700 1 -- start start 2024-03-30T05:07:43.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:43.710+0000 7fbdb9600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbdb40708a0 0x7fbdb411a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:07:43.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:43.710+0000 7fbdb9600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdb4071f90 0x7fbdb411a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:07:43.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:43.710+0000 7fbdb9600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb411ac40 con 0x7fbdb4071f90 2024-03-30T05:07:43.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:43.710+0000 7fbdb9600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb4114010 con 0x7fbdb40708a0 2024-03-30T05:07:43.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:43.710+0000 7fbdb2000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdb4071f90 0x7fbdb411a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:07:43.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:43.710+0000 7fbdb2000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdb4071f90 0x7fbdb411a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43242/0 (socket says 172.21.15.123:43242) 2024-03-30T05:07:43.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:43.710+0000 7fbdb2000700 1 -- 172.21.15.123:0/1769795620 learned_addr learned my addr 172.21.15.123:0/1769795620 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:07:43.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:43.710+0000 7fbdb2a00700 1 --2- 172.21.15.123:0/1769795620 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbdb40708a0 0x7fbdb411a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:07:43.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:43.710+0000 7fbdb2000700 1 -- 172.21.15.123:0/1769795620 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbdb40708a0 msgr2=0x7fbdb411a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:43.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:43.710+0000 7fbdb2000700 1 --2- 172.21.15.123:0/1769795620 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbdb40708a0 0x7fbdb411a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:43.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:43.710+0000 7fbdb2000700 1 -- 172.21.15.123:0/1769795620 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbda0005040 con 0x7fbdb4071f90 2024-03-30T05:07:43.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:43.710+0000 7fbdb2a00700 1 --2- 172.21.15.123:0/1769795620 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbdb40708a0 0x7fbdb411a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T05:07:43.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:43.711+0000 7fbdb2000700 1 --2- 172.21.15.123:0/1769795620 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdb4071f90 0x7fbdb411a5f0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fbda0009360 tx=0x7fbda0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:07:43.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:43.711+0000 7fbdaaa00700 1 -- 172.21.15.123:0/1769795620 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbda000e040 con 0x7fbdb4071f90 2024-03-30T05:07:43.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:43.711+0000 7fbdaaa00700 1 -- 172.21.15.123:0/1769795620 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbda0013420 con 0x7fbdb4071f90 2024-03-30T05:07:43.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:43.711+0000 7fbdb9600700 1 -- 172.21.15.123:0/1769795620 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdb4114290 con 0x7fbdb4071f90 2024-03-30T05:07:43.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:43.711+0000 7fbdaaa00700 1 -- 172.21.15.123:0/1769795620 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbda001c630 con 0x7fbdb4071f90 2024-03-30T05:07:43.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:43.711+0000 7fbdb9600700 1 -- 172.21.15.123:0/1769795620 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdb41147d0 con 0x7fbdb4071f90 2024-03-30T05:07:43.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:43.712+0000 7fbdb9600700 1 -- 172.21.15.123:0/1769795620 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdb4071aa0 con 0x7fbdb4071f90 2024-03-30T05:07:43.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:43.713+0000 7fbdaaa00700 1 -- 172.21.15.123:0/1769795620 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fbda00139e0 con 0x7fbdb4071f90 2024-03-30T05:07:43.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:43.714+0000 7fbdaaa00700 1 --2- 172.21.15.123:0/1769795620 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbd9806cc30 0x7fbd9806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:07:43.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:43.714+0000 7fbdaaa00700 1 -- 172.21.15.123:0/1769795620 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fbda008d2c0 con 0x7fbdb4071f90 2024-03-30T05:07:43.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:43.714+0000 7fbdb2a00700 1 --2- 172.21.15.123:0/1769795620 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbd9806cc30 0x7fbd9806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:07:43.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:43.715+0000 7fbdb2a00700 1 --2- 172.21.15.123:0/1769795620 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbd9806cc30 0x7fbd9806f0e0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fbda4007fd0 tx=0x7fbda4009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:07:43.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:43.719+0000 7fbdaaa00700 1 -- 172.21.15.123:0/1769795620 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fbda005ad90 con 0x7fbdb4071f90 2024-03-30T05:07:43.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:43.944+0000 7fbdb9600700 1 -- 172.21.15.123:0/1769795620 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbdb41103e0 con 0x7fbd9806cc30 2024-03-30T05:07:43.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:43.945+0000 7fbdaaa00700 1 -- 172.21.15.123:0/1769795620 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fbdb41103e0 con 0x7fbd9806cc30 2024-03-30T05:07:43.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:43.948+0000 7fbdb9600700 1 -- 172.21.15.123:0/1769795620 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbd9806cc30 msgr2=0x7fbd9806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:43.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:43.948+0000 7fbdb9600700 1 --2- 172.21.15.123:0/1769795620 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbd9806cc30 0x7fbd9806f0e0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fbda4007fd0 tx=0x7fbda4009040 comp rx=0 tx=0).stop 2024-03-30T05:07:43.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:43.948+0000 7fbdb9600700 1 -- 172.21.15.123:0/1769795620 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdb4071f90 msgr2=0x7fbdb411a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:43.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:43.948+0000 7fbdb9600700 1 --2- 172.21.15.123:0/1769795620 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdb4071f90 0x7fbdb411a5f0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fbda0009360 tx=0x7fbda0009e30 comp rx=0 tx=0).stop 2024-03-30T05:07:43.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:43.948+0000 7fbdb9600700 1 -- 172.21.15.123:0/1769795620 shutdown_connections 2024-03-30T05:07:43.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:43.948+0000 7fbdb9600700 1 --2- 172.21.15.123:0/1769795620 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbd9806cc30 0x7fbd9806f0e0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:43.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:43.948+0000 7fbdb9600700 1 --2- 172.21.15.123:0/1769795620 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbdb40708a0 0x7fbdb411a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:43.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:43.948+0000 7fbdb9600700 1 --2- 172.21.15.123:0/1769795620 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdb4071f90 0x7fbdb411a5f0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:43.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:43.948+0000 7fbdb9600700 1 -- 172.21.15.123:0/1769795620 >> 172.21.15.123:0/1769795620 conn(0x7fbdb40f89b0 msgr2=0x7fbdb40f9b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:07:43.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:43.948+0000 7fbdb9600700 1 -- 172.21.15.123:0/1769795620 shutdown_connections 2024-03-30T05:07:43.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:43.948+0000 7fbdb9600700 1 -- 172.21.15.123:0/1769795620 wait complete. 2024-03-30T05:07:43.961 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T05:07:44.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.082+0000 7fe3b8a00700 1 -- 172.21.15.123:0/1530592312 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3b4105990 msgr2=0x7fe3b40ff000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:44.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.083+0000 7fe3b8a00700 1 --2- 172.21.15.123:0/1530592312 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3b4105990 0x7fe3b40ff000 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0004940 tx=0x7fe3a0009e30 comp rx=0 tx=0).stop 2024-03-30T05:07:44.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.083+0000 7fe3b8a00700 1 -- 172.21.15.123:0/1530592312 shutdown_connections 2024-03-30T05:07:44.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.083+0000 7fe3b8a00700 1 --2- 172.21.15.123:0/1530592312 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3b4105990 0x7fe3b40ff000 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:44.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.083+0000 7fe3b8a00700 1 --2- 172.21.15.123:0/1530592312 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe3b4104ff0 0x7fe3b41053c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:44.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.083+0000 7fe3b8a00700 1 -- 172.21.15.123:0/1530592312 >> 172.21.15.123:0/1530592312 conn(0x7fe3b40faba0 msgr2=0x7fe3b40fcfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:07:44.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.083+0000 7fe3b8a00700 1 -- 172.21.15.123:0/1530592312 shutdown_connections 2024-03-30T05:07:44.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.083+0000 7fe3b8a00700 1 -- 172.21.15.123:0/1530592312 wait complete. 2024-03-30T05:07:44.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.083+0000 7fe3b8a00700 1 Processor -- start 2024-03-30T05:07:44.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.084+0000 7fe3b8a00700 1 -- start start 2024-03-30T05:07:44.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.084+0000 7fe3b8a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3b4104ff0 0x7fe3b4115b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:07:44.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.084+0000 7fe3b8a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe3b4105990 0x7fe3b4116080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:07:44.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.084+0000 7fe3b8a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b41166d0 con 0x7fe3b4104ff0 2024-03-30T05:07:44.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.084+0000 7fe3b8a00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b410fcf0 con 0x7fe3b4105990 2024-03-30T05:07:44.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.084+0000 7fe3b1e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3b4104ff0 0x7fe3b4115b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:07:44.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.084+0000 7fe3b1e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3b4104ff0 0x7fe3b4115b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39608/0 (socket says 172.21.15.123:39608) 2024-03-30T05:07:44.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.084+0000 7fe3b1e00700 1 -- 172.21.15.123:0/601240053 learned_addr learned my addr 172.21.15.123:0/601240053 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:07:44.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.084+0000 7fe3b1400700 1 --2- 172.21.15.123:0/601240053 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe3b4105990 0x7fe3b4116080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:07:44.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.085+0000 7fe3b1e00700 1 -- 172.21.15.123:0/601240053 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe3b4105990 msgr2=0x7fe3b4116080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:44.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.085+0000 7fe3b1e00700 1 --2- 172.21.15.123:0/601240053 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe3b4105990 0x7fe3b4116080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:44.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.085+0000 7fe3b1e00700 1 -- 172.21.15.123:0/601240053 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a0005040 con 0x7fe3b4104ff0 2024-03-30T05:07:44.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.085+0000 7fe3b1e00700 1 --2- 172.21.15.123:0/601240053 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3b4104ff0 0x7fe3b4115b40 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fe3a4002a50 tx=0x7fe3a40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:07:44.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.085+0000 7fe3aa000700 1 -- 172.21.15.123:0/601240053 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3a4007cc0 con 0x7fe3b4104ff0 2024-03-30T05:07:44.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.085+0000 7fe3aa000700 1 -- 172.21.15.123:0/601240053 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe3a400f040 con 0x7fe3b4104ff0 2024-03-30T05:07:44.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.085+0000 7fe3b8a00700 1 -- 172.21.15.123:0/601240053 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3b410ffd0 con 0x7fe3b4104ff0 2024-03-30T05:07:44.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.085+0000 7fe3aa000700 1 -- 172.21.15.123:0/601240053 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3a400d9c0 con 0x7fe3b4104ff0 2024-03-30T05:07:44.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.085+0000 7fe3b8a00700 1 -- 172.21.15.123:0/601240053 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3b4110510 con 0x7fe3b4104ff0 2024-03-30T05:07:44.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.086+0000 7fe3b8a00700 1 -- 172.21.15.123:0/601240053 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3b4102300 con 0x7fe3b4104ff0 2024-03-30T05:07:44.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.087+0000 7fe3aa000700 1 -- 172.21.15.123:0/601240053 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fe3a400db20 con 0x7fe3b4104ff0 2024-03-30T05:07:44.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.088+0000 7fe3aa000700 1 --2- 172.21.15.123:0/601240053 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe39806cc20 0x7fe39806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:07:44.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.088+0000 7fe3aa000700 1 -- 172.21.15.123:0/601240053 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe3a408bde0 con 0x7fe3b4104ff0 2024-03-30T05:07:44.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.089+0000 7fe3b1400700 1 --2- 172.21.15.123:0/601240053 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe39806cc20 0x7fe39806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:07:44.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.089+0000 7fe3b1400700 1 --2- 172.21.15.123:0/601240053 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe39806cc20 0x7fe39806f0d0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4111420 tx=0x7fe3a0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:07:44.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.093+0000 7fe3aa000700 1 -- 172.21.15.123:0/601240053 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe3a4055d20 con 0x7fe3b4104ff0 2024-03-30T05:07:44.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.315+0000 7fe3b8a00700 1 -- 172.21.15.123:0/601240053 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe3b410bf90 con 0x7fe39806cc20 2024-03-30T05:07:44.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.316+0000 7fe3aa000700 1 -- 172.21.15.123:0/601240053 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fe3b410bf90 con 0x7fe39806cc20 2024-03-30T05:07:44.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.319+0000 7fe3b8a00700 1 -- 172.21.15.123:0/601240053 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe39806cc20 msgr2=0x7fe39806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:44.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.319+0000 7fe3b8a00700 1 --2- 172.21.15.123:0/601240053 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe39806cc20 0x7fe39806f0d0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4111420 tx=0x7fe3a0009e30 comp rx=0 tx=0).stop 2024-03-30T05:07:44.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.319+0000 7fe3b8a00700 1 -- 172.21.15.123:0/601240053 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3b4104ff0 msgr2=0x7fe3b4115b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:44.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.319+0000 7fe3b8a00700 1 --2- 172.21.15.123:0/601240053 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3b4104ff0 0x7fe3b4115b40 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fe3a4002a50 tx=0x7fe3a40046b0 comp rx=0 tx=0).stop 2024-03-30T05:07:44.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.319+0000 7fe3b8a00700 1 -- 172.21.15.123:0/601240053 shutdown_connections 2024-03-30T05:07:44.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.319+0000 7fe3b8a00700 1 --2- 172.21.15.123:0/601240053 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe39806cc20 0x7fe39806f0d0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:44.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.319+0000 7fe3b8a00700 1 --2- 172.21.15.123:0/601240053 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3b4104ff0 0x7fe3b4115b40 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:44.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.319+0000 7fe3b8a00700 1 --2- 172.21.15.123:0/601240053 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe3b4105990 0x7fe3b4116080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:44.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.319+0000 7fe3b8a00700 1 -- 172.21.15.123:0/601240053 >> 172.21.15.123:0/601240053 conn(0x7fe3b40faba0 msgr2=0x7fe3b40fcad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:07:44.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.319+0000 7fe3b8a00700 1 -- 172.21.15.123:0/601240053 shutdown_connections 2024-03-30T05:07:44.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.319+0000 7fe3b8a00700 1 -- 172.21.15.123:0/601240053 wait complete. 2024-03-30T05:07:44.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.451+0000 7fd8fbe00700 1 -- 172.21.15.123:0/3695809200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8fc1035e0 msgr2=0x7fd8fc0fcc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:44.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.451+0000 7fd8fbe00700 1 --2- 172.21.15.123:0/3695809200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8fc1035e0 0x7fd8fc0fcc50 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fd8e8004940 tx=0x7fd8e8009e30 comp rx=0 tx=0).stop 2024-03-30T05:07:44.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.451+0000 7fd8fbe00700 1 -- 172.21.15.123:0/3695809200 shutdown_connections 2024-03-30T05:07:44.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.451+0000 7fd8fbe00700 1 --2- 172.21.15.123:0/3695809200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8fc1035e0 0x7fd8fc0fcc50 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:44.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.451+0000 7fd8fbe00700 1 --2- 172.21.15.123:0/3695809200 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd8fc102c40 0x7fd8fc103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:44.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.451+0000 7fd8fbe00700 1 -- 172.21.15.123:0/3695809200 >> 172.21.15.123:0/3695809200 conn(0x7fd8fc0f8830 msgr2=0x7fd8fc0fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:07:44.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.451+0000 7fd8fbe00700 1 -- 172.21.15.123:0/3695809200 shutdown_connections 2024-03-30T05:07:44.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.451+0000 7fd8fbe00700 1 -- 172.21.15.123:0/3695809200 wait complete. 2024-03-30T05:07:44.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.452+0000 7fd8fbe00700 1 Processor -- start 2024-03-30T05:07:44.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.452+0000 7fd8fbe00700 1 -- start start 2024-03-30T05:07:44.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.453+0000 7fd8fbe00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd8fc102c40 0x7fd8fc115ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:07:44.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.453+0000 7fd8fbe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8fc1035e0 0x7fd8fc116020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:07:44.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.453+0000 7fd8fbe00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8fc116670 con 0x7fd8fc1035e0 2024-03-30T05:07:44.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.453+0000 7fd8fbe00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8fc10fa60 con 0x7fd8fc102c40 2024-03-30T05:07:44.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.453+0000 7fd8fa000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8fc1035e0 0x7fd8fc116020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:07:44.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.453+0000 7fd8fa000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8fc1035e0 0x7fd8fc116020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39624/0 (socket says 172.21.15.123:39624) 2024-03-30T05:07:44.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.453+0000 7fd8fa000700 1 -- 172.21.15.123:0/3825344357 learned_addr learned my addr 172.21.15.123:0/3825344357 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:07:44.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.453+0000 7fd8faa00700 1 --2- 172.21.15.123:0/3825344357 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd8fc102c40 0x7fd8fc115ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:07:44.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.454+0000 7fd8fa000700 1 -- 172.21.15.123:0/3825344357 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd8fc102c40 msgr2=0x7fd8fc115ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:44.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.454+0000 7fd8fa000700 1 --2- 172.21.15.123:0/3825344357 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd8fc102c40 0x7fd8fc115ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:44.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.454+0000 7fd8fa000700 1 -- 172.21.15.123:0/3825344357 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8e8005040 con 0x7fd8fc1035e0 2024-03-30T05:07:44.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.454+0000 7fd8fa000700 1 --2- 172.21.15.123:0/3825344357 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8fc1035e0 0x7fd8fc116020 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fd8e8009390 tx=0x7fd8e8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:07:44.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.454+0000 7fd8f2a00700 1 -- 172.21.15.123:0/3825344357 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd8e800e040 con 0x7fd8fc1035e0 2024-03-30T05:07:44.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.454+0000 7fd8f2a00700 1 -- 172.21.15.123:0/3825344357 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd8e8013420 con 0x7fd8fc1035e0 2024-03-30T05:07:44.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.454+0000 7fd8f2a00700 1 -- 172.21.15.123:0/3825344357 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd8e801c630 con 0x7fd8fc1035e0 2024-03-30T05:07:44.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.454+0000 7fd8fbe00700 1 -- 172.21.15.123:0/3825344357 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8fc10fce0 con 0x7fd8fc1035e0 2024-03-30T05:07:44.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.454+0000 7fd8fbe00700 1 -- 172.21.15.123:0/3825344357 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8fc1102a0 con 0x7fd8fc1035e0 2024-03-30T05:07:44.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.455+0000 7fd8fbe00700 1 -- 172.21.15.123:0/3825344357 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8fc0fe390 con 0x7fd8fc1035e0 2024-03-30T05:07:44.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.457+0000 7fd8f2a00700 1 -- 172.21.15.123:0/3825344357 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fd8e8013a00 con 0x7fd8fc1035e0 2024-03-30T05:07:44.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.457+0000 7fd8f2a00700 1 --2- 172.21.15.123:0/3825344357 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd8e006cc30 0x7fd8e006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:07:44.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.457+0000 7fd8f2a00700 1 -- 172.21.15.123:0/3825344357 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd8e808d120 con 0x7fd8fc1035e0 2024-03-30T05:07:44.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.458+0000 7fd8faa00700 1 --2- 172.21.15.123:0/3825344357 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd8e006cc30 0x7fd8e006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:07:44.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.458+0000 7fd8faa00700 1 --2- 172.21.15.123:0/3825344357 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd8e006cc30 0x7fd8e006f0e0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec004f40 tx=0x7fd8ec00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:07:44.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.462+0000 7fd8f2a00700 1 -- 172.21.15.123:0/3825344357 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd8e805abf0 con 0x7fd8fc1035e0 2024-03-30T05:07:44.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.678+0000 7fd8fbe00700 1 -- 172.21.15.123:0/3825344357 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd8fc109bd0 con 0x7fd8e006cc30 2024-03-30T05:07:44.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.687+0000 7fd8f2a00700 1 -- 172.21.15.123:0/3825344357 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fd8fc109bd0 con 0x7fd8e006cc30 2024-03-30T05:07:44.685 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T05:07:44.686 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (36m) 14m ago 49m 14.6M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T05:07:44.686 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (34m) 14m ago 49m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T05:07:44.686 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (34m) 14m ago 48m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T05:07:44.686 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (36m) 14m ago 49m 73.5M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T05:07:44.686 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (29m) 14m ago 43m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T05:07:44.686 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (30m) 14m ago 43m 16.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T05:07:44.686 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (14m) 14m ago 42m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 87b073a6c9ca 2024-03-30T05:07:44.686 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (29m) 14m ago 43m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T05:07:44.686 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (39m) 14m ago 51m 510M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T05:07:44.686 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (38m) 14m ago 48m 456M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T05:07:44.686 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (35m) 14m ago 51m 70.0M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T05:07:44.686 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (34m) 14m ago 48m 51.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T05:07:44.686 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (37m) 14m ago 49m 8636k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T05:07:44.687 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (37m) 14m ago 48m 8678k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T05:07:44.687 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (33m) 14m ago 47m 107M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T05:07:44.687 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (33m) 14m ago 46m 97.2M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T05:07:44.687 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (32m) 14m ago 46m 89.8M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T05:07:44.687 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (31m) 14m ago 45m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T05:07:44.687 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (31m) 14m ago 45m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T05:07:44.687 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (30m) 14m ago 44m 88.8M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T05:07:44.687 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (37m) 14m ago 48m 63.9M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T05:07:44.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.690+0000 7fd8fbe00700 1 -- 172.21.15.123:0/3825344357 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd8e006cc30 msgr2=0x7fd8e006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:44.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.690+0000 7fd8fbe00700 1 --2- 172.21.15.123:0/3825344357 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd8e006cc30 0x7fd8e006f0e0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec004f40 tx=0x7fd8ec00b040 comp rx=0 tx=0).stop 2024-03-30T05:07:44.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.690+0000 7fd8fbe00700 1 -- 172.21.15.123:0/3825344357 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8fc1035e0 msgr2=0x7fd8fc116020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:44.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.690+0000 7fd8fbe00700 1 --2- 172.21.15.123:0/3825344357 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8fc1035e0 0x7fd8fc116020 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fd8e8009390 tx=0x7fd8e8009e30 comp rx=0 tx=0).stop 2024-03-30T05:07:44.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.690+0000 7fd8fbe00700 1 -- 172.21.15.123:0/3825344357 shutdown_connections 2024-03-30T05:07:44.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.690+0000 7fd8fbe00700 1 --2- 172.21.15.123:0/3825344357 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd8e006cc30 0x7fd8e006f0e0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:44.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.690+0000 7fd8fbe00700 1 --2- 172.21.15.123:0/3825344357 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd8fc102c40 0x7fd8fc115ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:44.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.690+0000 7fd8fbe00700 1 --2- 172.21.15.123:0/3825344357 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8fc1035e0 0x7fd8fc116020 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:44.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.690+0000 7fd8fbe00700 1 -- 172.21.15.123:0/3825344357 >> 172.21.15.123:0/3825344357 conn(0x7fd8fc0f8830 msgr2=0x7fd8fc0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:07:44.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.690+0000 7fd8fbe00700 1 -- 172.21.15.123:0/3825344357 shutdown_connections 2024-03-30T05:07:44.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.690+0000 7fd8fbe00700 1 -- 172.21.15.123:0/3825344357 wait complete. 2024-03-30T05:07:44.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.822+0000 7fbf7ea00700 1 -- 172.21.15.123:0/1887554945 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf781037a0 msgr2=0x7fbf780fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:44.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.822+0000 7fbf7ea00700 1 --2- 172.21.15.123:0/1887554945 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf781037a0 0x7fbf780fce10 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fbf64004940 tx=0x7fbf64009e30 comp rx=0 tx=0).stop 2024-03-30T05:07:44.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.822+0000 7fbf7ea00700 1 -- 172.21.15.123:0/1887554945 shutdown_connections 2024-03-30T05:07:44.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.822+0000 7fbf7ea00700 1 --2- 172.21.15.123:0/1887554945 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf781037a0 0x7fbf780fce10 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:44.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.822+0000 7fbf7ea00700 1 --2- 172.21.15.123:0/1887554945 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbf78102e00 0x7fbf781031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:44.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.822+0000 7fbf7ea00700 1 -- 172.21.15.123:0/1887554945 >> 172.21.15.123:0/1887554945 conn(0x7fbf780f89b0 msgr2=0x7fbf780fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:07:44.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.822+0000 7fbf7ea00700 1 -- 172.21.15.123:0/1887554945 shutdown_connections 2024-03-30T05:07:44.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.822+0000 7fbf7ea00700 1 -- 172.21.15.123:0/1887554945 wait complete. 2024-03-30T05:07:44.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.823+0000 7fbf7ea00700 1 Processor -- start 2024-03-30T05:07:44.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.823+0000 7fbf7ea00700 1 -- start start 2024-03-30T05:07:44.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.824+0000 7fbf7ea00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbf78102e00 0x7fbf78115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:07:44.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.824+0000 7fbf7ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf781037a0 0x7fbf781161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:07:44.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.824+0000 7fbf7ea00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf78116810 con 0x7fbf781037a0 2024-03-30T05:07:44.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.824+0000 7fbf7ea00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf7810fbe0 con 0x7fbf78102e00 2024-03-30T05:07:44.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.824+0000 7fbf77400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf781037a0 0x7fbf781161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:07:44.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.824+0000 7fbf77400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf781037a0 0x7fbf781161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39630/0 (socket says 172.21.15.123:39630) 2024-03-30T05:07:44.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.824+0000 7fbf77400700 1 -- 172.21.15.123:0/1716008479 learned_addr learned my addr 172.21.15.123:0/1716008479 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:07:44.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.825+0000 7fbf77e00700 1 --2- 172.21.15.123:0/1716008479 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbf78102e00 0x7fbf78115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:07:44.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.825+0000 7fbf77400700 1 -- 172.21.15.123:0/1716008479 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbf78102e00 msgr2=0x7fbf78115c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:44.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.825+0000 7fbf77400700 1 --2- 172.21.15.123:0/1716008479 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbf78102e00 0x7fbf78115c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:44.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.825+0000 7fbf77400700 1 -- 172.21.15.123:0/1716008479 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf64005040 con 0x7fbf781037a0 2024-03-30T05:07:44.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.825+0000 7fbf77400700 1 --2- 172.21.15.123:0/1716008479 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf781037a0 0x7fbf781161c0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fbf64009fd0 tx=0x7fbf64009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:07:44.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.825+0000 7fbf6fe00700 1 -- 172.21.15.123:0/1716008479 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf6400e030 con 0x7fbf781037a0 2024-03-30T05:07:44.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.825+0000 7fbf7ea00700 1 -- 172.21.15.123:0/1716008479 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf7810fe60 con 0x7fbf781037a0 2024-03-30T05:07:44.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.825+0000 7fbf6fe00700 1 -- 172.21.15.123:0/1716008479 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbf64013e00 con 0x7fbf781037a0 2024-03-30T05:07:44.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.825+0000 7fbf6fe00700 1 -- 172.21.15.123:0/1716008479 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf6401ccd0 con 0x7fbf781037a0 2024-03-30T05:07:44.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.825+0000 7fbf7ea00700 1 -- 172.21.15.123:0/1716008479 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf78110420 con 0x7fbf781037a0 2024-03-30T05:07:44.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.826+0000 7fbf7ea00700 1 -- 172.21.15.123:0/1716008479 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf780fe550 con 0x7fbf781037a0 2024-03-30T05:07:44.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.828+0000 7fbf6fe00700 1 -- 172.21.15.123:0/1716008479 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fbf6401c500 con 0x7fbf781037a0 2024-03-30T05:07:44.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.828+0000 7fbf6fe00700 1 --2- 172.21.15.123:0/1716008479 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbf5c06cc30 0x7fbf5c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:07:44.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.828+0000 7fbf6fe00700 1 -- 172.21.15.123:0/1716008479 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fbf6408d7f0 con 0x7fbf781037a0 2024-03-30T05:07:44.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.829+0000 7fbf77e00700 1 --2- 172.21.15.123:0/1716008479 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbf5c06cc30 0x7fbf5c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:07:44.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.829+0000 7fbf77e00700 1 --2- 172.21.15.123:0/1716008479 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbf5c06cc30 0x7fbf5c06f0e0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fbf68009fd0 tx=0x7fbf6800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:07:44.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:44.833+0000 7fbf6fe00700 1 -- 172.21.15.123:0/1716008479 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fbf6405b380 con 0x7fbf781037a0 2024-03-30T05:07:45.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.117+0000 7fbf7ea00700 1 -- 172.21.15.123:0/1716008479 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fbf7804bed0 con 0x7fbf781037a0 2024-03-30T05:07:45.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.117+0000 7fbf6fe00700 1 -- 172.21.15.123:0/1716008479 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7fbf6405af10 con 0x7fbf781037a0 2024-03-30T05:07:45.115 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T05:07:45.116 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T05:07:45.116 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T05:07:45.116 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:07:45.116 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T05:07:45.116 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T05:07:45.116 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:07:45.116 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T05:07:45.116 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T05:07:45.116 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:07:45.116 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T05:07:45.116 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 4 2024-03-30T05:07:45.116 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:07:45.116 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T05:07:45.116 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 14 2024-03-30T05:07:45.117 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T05:07:45.117 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T05:07:45.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.120+0000 7fbf7ea00700 1 -- 172.21.15.123:0/1716008479 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbf5c06cc30 msgr2=0x7fbf5c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:45.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.120+0000 7fbf7ea00700 1 --2- 172.21.15.123:0/1716008479 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbf5c06cc30 0x7fbf5c06f0e0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fbf68009fd0 tx=0x7fbf6800b040 comp rx=0 tx=0).stop 2024-03-30T05:07:45.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.120+0000 7fbf7ea00700 1 -- 172.21.15.123:0/1716008479 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf781037a0 msgr2=0x7fbf781161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:45.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.120+0000 7fbf7ea00700 1 --2- 172.21.15.123:0/1716008479 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf781037a0 0x7fbf781161c0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fbf64009fd0 tx=0x7fbf64009e30 comp rx=0 tx=0).stop 2024-03-30T05:07:45.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.120+0000 7fbf7ea00700 1 -- 172.21.15.123:0/1716008479 shutdown_connections 2024-03-30T05:07:45.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.120+0000 7fbf7ea00700 1 --2- 172.21.15.123:0/1716008479 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbf5c06cc30 0x7fbf5c06f0e0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:45.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.120+0000 7fbf7ea00700 1 --2- 172.21.15.123:0/1716008479 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbf78102e00 0x7fbf78115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:45.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.120+0000 7fbf7ea00700 1 --2- 172.21.15.123:0/1716008479 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf781037a0 0x7fbf781161c0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:45.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.120+0000 7fbf7ea00700 1 -- 172.21.15.123:0/1716008479 >> 172.21.15.123:0/1716008479 conn(0x7fbf780f89b0 msgr2=0x7fbf780f99a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:07:45.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.120+0000 7fbf7ea00700 1 -- 172.21.15.123:0/1716008479 shutdown_connections 2024-03-30T05:07:45.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.120+0000 7fbf7ea00700 1 -- 172.21.15.123:0/1716008479 wait complete. 2024-03-30T05:07:45.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.253+0000 7f7afd200700 1 -- 172.21.15.123:0/1001097018 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7af81037a0 msgr2=0x7f7af80fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:45.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.253+0000 7f7afd200700 1 --2- 172.21.15.123:0/1001097018 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7af81037a0 0x7f7af80fce10 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f7ae4004940 tx=0x7f7ae4009e30 comp rx=0 tx=0).stop 2024-03-30T05:07:45.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.253+0000 7f7afd200700 1 -- 172.21.15.123:0/1001097018 shutdown_connections 2024-03-30T05:07:45.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.253+0000 7f7afd200700 1 --2- 172.21.15.123:0/1001097018 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7af81037a0 0x7f7af80fce10 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:45.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.253+0000 7f7afd200700 1 --2- 172.21.15.123:0/1001097018 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7af8102e00 0x7f7af81031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:45.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.253+0000 7f7afd200700 1 -- 172.21.15.123:0/1001097018 >> 172.21.15.123:0/1001097018 conn(0x7f7af80f89b0 msgr2=0x7f7af80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:07:45.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.253+0000 7f7afd200700 1 -- 172.21.15.123:0/1001097018 shutdown_connections 2024-03-30T05:07:45.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.253+0000 7f7afd200700 1 -- 172.21.15.123:0/1001097018 wait complete. 2024-03-30T05:07:45.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.254+0000 7f7afd200700 1 Processor -- start 2024-03-30T05:07:45.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.254+0000 7f7afd200700 1 -- start start 2024-03-30T05:07:45.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.254+0000 7f7afd200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7af8102e00 0x7f7af8115da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:07:45.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.254+0000 7f7afd200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7af81037a0 0x7f7af81162e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:07:45.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.254+0000 7f7afd200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af8116930 con 0x7f7af81037a0 2024-03-30T05:07:45.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.254+0000 7f7afd200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af810fd00 con 0x7f7af8102e00 2024-03-30T05:07:45.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.254+0000 7f7af5400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7af81037a0 0x7f7af81162e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:07:45.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.255+0000 7f7af5400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7af81037a0 0x7f7af81162e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39656/0 (socket says 172.21.15.123:39656) 2024-03-30T05:07:45.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.255+0000 7f7af5400700 1 -- 172.21.15.123:0/2172196444 learned_addr learned my addr 172.21.15.123:0/2172196444 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:07:45.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.255+0000 7f7af5400700 1 -- 172.21.15.123:0/2172196444 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7af8102e00 msgr2=0x7f7af8115da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:45.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.255+0000 7f7af5e00700 1 --2- 172.21.15.123:0/2172196444 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7af8102e00 0x7f7af8115da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:07:45.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.255+0000 7f7af5400700 1 --2- 172.21.15.123:0/2172196444 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7af8102e00 0x7f7af8115da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:45.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.255+0000 7f7af5400700 1 -- 172.21.15.123:0/2172196444 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ae4005040 con 0x7f7af81037a0 2024-03-30T05:07:45.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.255+0000 7f7af5400700 1 --2- 172.21.15.123:0/2172196444 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7af81037a0 0x7f7af81162e0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f7ae4009390 tx=0x7f7ae4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:07:45.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.255+0000 7f7aee000700 1 -- 172.21.15.123:0/2172196444 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ae400e040 con 0x7f7af81037a0 2024-03-30T05:07:45.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.255+0000 7f7aee000700 1 -- 172.21.15.123:0/2172196444 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7ae4013420 con 0x7f7af81037a0 2024-03-30T05:07:45.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.255+0000 7f7afd200700 1 -- 172.21.15.123:0/2172196444 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7af810ff80 con 0x7f7af81037a0 2024-03-30T05:07:45.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.255+0000 7f7aee000700 1 -- 172.21.15.123:0/2172196444 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ae401c630 con 0x7f7af81037a0 2024-03-30T05:07:45.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.256+0000 7f7afd200700 1 -- 172.21.15.123:0/2172196444 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7af81104c0 con 0x7f7af81037a0 2024-03-30T05:07:45.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.256+0000 7f7afd200700 1 -- 172.21.15.123:0/2172196444 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7af80fe550 con 0x7f7af81037a0 2024-03-30T05:07:45.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.258+0000 7f7aee000700 1 -- 172.21.15.123:0/2172196444 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f7ae4013a00 con 0x7f7af81037a0 2024-03-30T05:07:45.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.259+0000 7f7aee000700 1 --2- 172.21.15.123:0/2172196444 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7adc06cc30 0x7f7adc06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:07:45.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.259+0000 7f7aee000700 1 -- 172.21.15.123:0/2172196444 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7ae408d390 con 0x7f7af81037a0 2024-03-30T05:07:45.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.259+0000 7f7af5e00700 1 --2- 172.21.15.123:0/2172196444 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7adc06cc30 0x7f7adc06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:07:45.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.260+0000 7f7af5e00700 1 --2- 172.21.15.123:0/2172196444 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7adc06cc30 0x7f7adc06f0e0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f7ae8004d60 tx=0x7f7ae800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:07:45.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.264+0000 7f7aee000700 1 -- 172.21.15.123:0/2172196444 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7ae405ade0 con 0x7f7af81037a0 2024-03-30T05:07:45.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:07:45 smithi135 ceph-mon[106011]: pgmap v1097: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:45.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:07:45 smithi135 ceph-mon[106011]: from='client.35972 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:07:45.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:07:45 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:07:45.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:07:45 smithi123 ceph-mon[139369]: pgmap v1097: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:45.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:07:45 smithi123 ceph-mon[139369]: from='client.35972 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:07:45.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:07:45 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:07:45.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.511+0000 7f7afd200700 1 -- 172.21.15.123:0/2172196444 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f7af804bed0 con 0x7f7af81037a0 2024-03-30T05:07:45.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.512+0000 7f7aee000700 1 -- 172.21.15.123:0/2172196444 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 34 v34) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f7ae405a970 con 0x7f7af81037a0 2024-03-30T05:07:45.510 INFO:teuthology.orchestra.run.smithi123.stdout:e34 2024-03-30T05:07:45.510 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T05:07:45.510 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T05:07:45.510 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T05:07:45.511 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:07:45.511 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T05:07:45.511 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T05:07:45.511 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T05:07:45.511 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T05:07:45.511 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T05:07:45.511 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T05:07:45.511 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T05:07:45.511 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T05:07:45.511 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T05:07:45.511 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T05:07:45.511 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T05:07:45.511 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T05:07:45.511 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T05:07:45.512 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T05:07:45.512 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T05:07:45.512 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T05:07:45.512 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T05:07:45.512 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T05:07:45.512 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T05:07:45.512 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T05:07:45.512 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T05:07:45.512 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T05:07:45.512 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T05:07:45.512 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T05:07:45.512 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T05:07:45.512 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T05:07:45.513 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T05:07:45.513 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T05:07:45.513 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:07:45.513 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:07:45.513 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T05:07:45.513 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:07:45.513 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:07:45.513 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:07:45.513 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:07:45.513 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44975} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/3788988014,v1:172.21.15.135:6827/3788988014] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:07:45.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.514+0000 7f7afd200700 1 -- 172.21.15.123:0/2172196444 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7adc06cc30 msgr2=0x7f7adc06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:45.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.514+0000 7f7afd200700 1 --2- 172.21.15.123:0/2172196444 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7adc06cc30 0x7f7adc06f0e0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f7ae8004d60 tx=0x7f7ae800b040 comp rx=0 tx=0).stop 2024-03-30T05:07:45.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.514+0000 7f7afd200700 1 -- 172.21.15.123:0/2172196444 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7af81037a0 msgr2=0x7f7af81162e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:45.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.514+0000 7f7afd200700 1 --2- 172.21.15.123:0/2172196444 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7af81037a0 0x7f7af81162e0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f7ae4009390 tx=0x7f7ae4009e30 comp rx=0 tx=0).stop 2024-03-30T05:07:45.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.514+0000 7f7afd200700 1 -- 172.21.15.123:0/2172196444 shutdown_connections 2024-03-30T05:07:45.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.514+0000 7f7afd200700 1 --2- 172.21.15.123:0/2172196444 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7adc06cc30 0x7f7adc06f0e0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:45.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.514+0000 7f7afd200700 1 --2- 172.21.15.123:0/2172196444 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7af8102e00 0x7f7af8115da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:45.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.514+0000 7f7afd200700 1 --2- 172.21.15.123:0/2172196444 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7af81037a0 0x7f7af81162e0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:45.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.514+0000 7f7afd200700 1 -- 172.21.15.123:0/2172196444 >> 172.21.15.123:0/2172196444 conn(0x7f7af80f89b0 msgr2=0x7f7af80f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:07:45.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.515+0000 7f7afd200700 1 -- 172.21.15.123:0/2172196444 shutdown_connections 2024-03-30T05:07:45.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.515+0000 7f7afd200700 1 -- 172.21.15.123:0/2172196444 wait complete. 2024-03-30T05:07:45.515 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 34 2024-03-30T05:07:45.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.649+0000 7f7045400700 1 -- 172.21.15.123:0/17830239 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70401037c0 msgr2=0x7f70400fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:45.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.649+0000 7f7045400700 1 --2- 172.21.15.123:0/17830239 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70401037c0 0x7f70400fce30 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f702c004940 tx=0x7f702c009e30 comp rx=0 tx=0).stop 2024-03-30T05:07:45.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.649+0000 7f7045400700 1 -- 172.21.15.123:0/17830239 shutdown_connections 2024-03-30T05:07:45.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.649+0000 7f7045400700 1 --2- 172.21.15.123:0/17830239 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70401037c0 0x7f70400fce30 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:45.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.649+0000 7f7045400700 1 --2- 172.21.15.123:0/17830239 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7040102e20 0x7f70401031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:45.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.649+0000 7f7045400700 1 -- 172.21.15.123:0/17830239 >> 172.21.15.123:0/17830239 conn(0x7f70400f89d0 msgr2=0x7f70400fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:07:45.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.649+0000 7f7045400700 1 -- 172.21.15.123:0/17830239 shutdown_connections 2024-03-30T05:07:45.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.649+0000 7f7045400700 1 -- 172.21.15.123:0/17830239 wait complete. 2024-03-30T05:07:45.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.650+0000 7f7045400700 1 Processor -- start 2024-03-30T05:07:45.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.650+0000 7f7045400700 1 -- start start 2024-03-30T05:07:45.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.651+0000 7f7045400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7040102e20 0x7f7040113ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:07:45.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.651+0000 7f7045400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70401037c0 0x7f7040114010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:07:45.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.651+0000 7f7045400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7040114660 con 0x7f70401037c0 2024-03-30T05:07:45.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.651+0000 7f7045400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f704010da50 con 0x7f7040102e20 2024-03-30T05:07:45.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.651+0000 7f703de00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70401037c0 0x7f7040114010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:07:45.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.651+0000 7f703de00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70401037c0 0x7f7040114010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39674/0 (socket says 172.21.15.123:39674) 2024-03-30T05:07:45.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.651+0000 7f703de00700 1 -- 172.21.15.123:0/3664429239 learned_addr learned my addr 172.21.15.123:0/3664429239 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:07:45.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.651+0000 7f703e800700 1 --2- 172.21.15.123:0/3664429239 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7040102e20 0x7f7040113ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:07:45.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.651+0000 7f703de00700 1 -- 172.21.15.123:0/3664429239 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7040102e20 msgr2=0x7f7040113ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:45.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.651+0000 7f703de00700 1 --2- 172.21.15.123:0/3664429239 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7040102e20 0x7f7040113ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:45.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.651+0000 7f703de00700 1 -- 172.21.15.123:0/3664429239 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f702c005040 con 0x7f70401037c0 2024-03-30T05:07:45.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.651+0000 7f703de00700 1 --2- 172.21.15.123:0/3664429239 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70401037c0 0x7f7040114010 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f702c009390 tx=0x7f702c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:07:45.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.652+0000 7f7036a00700 1 -- 172.21.15.123:0/3664429239 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f702c00e040 con 0x7f70401037c0 2024-03-30T05:07:45.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.652+0000 7f7036a00700 1 -- 172.21.15.123:0/3664429239 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f702c013420 con 0x7f70401037c0 2024-03-30T05:07:45.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.652+0000 7f7045400700 1 -- 172.21.15.123:0/3664429239 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f704010dcd0 con 0x7f70401037c0 2024-03-30T05:07:45.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.652+0000 7f7036a00700 1 -- 172.21.15.123:0/3664429239 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f702c01c630 con 0x7f70401037c0 2024-03-30T05:07:45.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.652+0000 7f7045400700 1 -- 172.21.15.123:0/3664429239 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f704010e290 con 0x7f70401037c0 2024-03-30T05:07:45.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.653+0000 7f7045400700 1 -- 172.21.15.123:0/3664429239 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70400fe570 con 0x7f70401037c0 2024-03-30T05:07:45.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.654+0000 7f7036a00700 1 -- 172.21.15.123:0/3664429239 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f702c0139e0 con 0x7f70401037c0 2024-03-30T05:07:45.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.655+0000 7f7036a00700 1 --2- 172.21.15.123:0/3664429239 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f702406cc30 0x7f702406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:07:45.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.655+0000 7f7036a00700 1 -- 172.21.15.123:0/3664429239 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f702c08d2a0 con 0x7f70401037c0 2024-03-30T05:07:45.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.655+0000 7f703e800700 1 --2- 172.21.15.123:0/3664429239 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f702406cc30 0x7f702406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:07:45.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.656+0000 7f703e800700 1 --2- 172.21.15.123:0/3664429239 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f702406cc30 0x7f702406f0e0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f7030007fd0 tx=0x7f7030009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:07:45.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.659+0000 7f7036a00700 1 -- 172.21.15.123:0/3664429239 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f702c05ad70 con 0x7f70401037c0 2024-03-30T05:07:45.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.880+0000 7f7045400700 1 -- 172.21.15.123:0/3664429239 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7040109dc0 con 0x7f702406cc30 2024-03-30T05:07:45.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.882+0000 7f7036a00700 1 -- 172.21.15.123:0/3664429239 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f7040109dc0 con 0x7f702406cc30 2024-03-30T05:07:45.880 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T05:07:45.880 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T05:07:45.880 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T05:07:45.880 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T05:07:45.880 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T05:07:45.880 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T05:07:45.881 INFO:teuthology.orchestra.run.smithi123.stdout: "mds", 2024-03-30T05:07:45.881 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T05:07:45.881 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T05:07:45.881 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T05:07:45.881 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T05:07:45.881 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "16/21 daemons upgraded", 2024-03-30T05:07:45.881 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T05:07:45.881 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T05:07:45.881 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T05:07:45.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.884+0000 7f7045400700 1 -- 172.21.15.123:0/3664429239 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f702406cc30 msgr2=0x7f702406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:45.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.884+0000 7f7045400700 1 --2- 172.21.15.123:0/3664429239 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f702406cc30 0x7f702406f0e0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f7030007fd0 tx=0x7f7030009040 comp rx=0 tx=0).stop 2024-03-30T05:07:45.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.884+0000 7f7045400700 1 -- 172.21.15.123:0/3664429239 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70401037c0 msgr2=0x7f7040114010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:45.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.884+0000 7f7045400700 1 --2- 172.21.15.123:0/3664429239 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70401037c0 0x7f7040114010 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f702c009390 tx=0x7f702c009e30 comp rx=0 tx=0).stop 2024-03-30T05:07:45.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.884+0000 7f7045400700 1 -- 172.21.15.123:0/3664429239 shutdown_connections 2024-03-30T05:07:45.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.884+0000 7f7045400700 1 --2- 172.21.15.123:0/3664429239 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f702406cc30 0x7f702406f0e0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:45.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.884+0000 7f7045400700 1 --2- 172.21.15.123:0/3664429239 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7040102e20 0x7f7040113ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:45.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.884+0000 7f7045400700 1 --2- 172.21.15.123:0/3664429239 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70401037c0 0x7f7040114010 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:45.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.884+0000 7f7045400700 1 -- 172.21.15.123:0/3664429239 >> 172.21.15.123:0/3664429239 conn(0x7f70400f89d0 msgr2=0x7f70400f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:07:45.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.884+0000 7f7045400700 1 -- 172.21.15.123:0/3664429239 shutdown_connections 2024-03-30T05:07:45.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:45.884+0000 7f7045400700 1 -- 172.21.15.123:0/3664429239 wait complete. 2024-03-30T05:07:46.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:46.017+0000 7f9b92400700 1 -- 172.21.15.123:0/3100589266 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b8c0fe830 msgr2=0x7f9b8c106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:46.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:46.017+0000 7f9b92400700 1 --2- 172.21.15.123:0/3100589266 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b8c0fe830 0x7f9b8c106f00 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f9b78004940 tx=0x7f9b78009e30 comp rx=0 tx=0).stop 2024-03-30T05:07:46.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:46.017+0000 7f9b92400700 1 -- 172.21.15.123:0/3100589266 shutdown_connections 2024-03-30T05:07:46.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:46.017+0000 7f9b92400700 1 --2- 172.21.15.123:0/3100589266 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b8c0fe830 0x7f9b8c106f00 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:46.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:46.017+0000 7f9b92400700 1 --2- 172.21.15.123:0/3100589266 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9b8c0fde90 0x7f9b8c0fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:46.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:46.017+0000 7f9b92400700 1 -- 172.21.15.123:0/3100589266 >> 172.21.15.123:0/3100589266 conn(0x7f9b8c072bc0 msgr2=0x7f9b8c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:07:46.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:46.018+0000 7f9b92400700 1 -- 172.21.15.123:0/3100589266 shutdown_connections 2024-03-30T05:07:46.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:46.018+0000 7f9b92400700 1 -- 172.21.15.123:0/3100589266 wait complete. 2024-03-30T05:07:46.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:46.018+0000 7f9b92400700 1 Processor -- start 2024-03-30T05:07:46.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:46.019+0000 7f9b92400700 1 -- start start 2024-03-30T05:07:46.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:46.019+0000 7f9b92400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b8c0fde90 0x7f9b8c115d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:07:46.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:46.019+0000 7f9b92400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9b8c0fe830 0x7f9b8c116270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:07:46.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:46.019+0000 7f9b92400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8c1168c0 con 0x7f9b8c0fde90 2024-03-30T05:07:46.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:46.019+0000 7f9b92400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8c10fc90 con 0x7f9b8c0fe830 2024-03-30T05:07:46.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:46.019+0000 7f9b8b400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b8c0fde90 0x7f9b8c115d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:07:46.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:46.019+0000 7f9b8b400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b8c0fde90 0x7f9b8c115d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39688/0 (socket says 172.21.15.123:39688) 2024-03-30T05:07:46.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:46.019+0000 7f9b8b400700 1 -- 172.21.15.123:0/4291410623 learned_addr learned my addr 172.21.15.123:0/4291410623 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:07:46.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:46.019+0000 7f9b8aa00700 1 --2- 172.21.15.123:0/4291410623 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9b8c0fe830 0x7f9b8c116270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:07:46.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:46.020+0000 7f9b8b400700 1 -- 172.21.15.123:0/4291410623 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9b8c0fe830 msgr2=0x7f9b8c116270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:46.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:46.020+0000 7f9b8b400700 1 --2- 172.21.15.123:0/4291410623 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9b8c0fe830 0x7f9b8c116270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:46.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:46.020+0000 7f9b8b400700 1 -- 172.21.15.123:0/4291410623 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b78005040 con 0x7f9b8c0fde90 2024-03-30T05:07:46.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:46.020+0000 7f9b8b400700 1 --2- 172.21.15.123:0/4291410623 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b8c0fde90 0x7f9b8c115d30 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c002a50 tx=0x7f9b7c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:07:46.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:46.020+0000 7f9b83400700 1 -- 172.21.15.123:0/4291410623 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9b7c007cc0 con 0x7f9b8c0fde90 2024-03-30T05:07:46.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:46.020+0000 7f9b83400700 1 -- 172.21.15.123:0/4291410623 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9b7c00f040 con 0x7f9b8c0fde90 2024-03-30T05:07:46.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:46.020+0000 7f9b92400700 1 -- 172.21.15.123:0/4291410623 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b8c10ff70 con 0x7f9b8c0fde90 2024-03-30T05:07:46.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:46.020+0000 7f9b83400700 1 -- 172.21.15.123:0/4291410623 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9b7c00d9c0 con 0x7f9b8c0fde90 2024-03-30T05:07:46.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:46.020+0000 7f9b92400700 1 -- 172.21.15.123:0/4291410623 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b8c1104b0 con 0x7f9b8c0fde90 2024-03-30T05:07:46.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:46.021+0000 7f9b92400700 1 -- 172.21.15.123:0/4291410623 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b8c106a20 con 0x7f9b8c0fde90 2024-03-30T05:07:46.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:46.023+0000 7f9b83400700 1 -- 172.21.15.123:0/4291410623 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f9b7c00db20 con 0x7f9b8c0fde90 2024-03-30T05:07:46.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:46.023+0000 7f9b83400700 1 --2- 172.21.15.123:0/4291410623 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9b7006cc80 0x7f9b7006f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:07:46.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:46.023+0000 7f9b83400700 1 -- 172.21.15.123:0/4291410623 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f9b7c08bde0 con 0x7f9b8c0fde90 2024-03-30T05:07:46.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:46.024+0000 7f9b8aa00700 1 --2- 172.21.15.123:0/4291410623 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9b7006cc80 0x7f9b7006f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:07:46.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:46.024+0000 7f9b8aa00700 1 --2- 172.21.15.123:0/4291410623 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9b7006cc80 0x7f9b7006f130 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c1115d0 tx=0x7f9b78009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:07:46.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:46.028+0000 7f9b83400700 1 -- 172.21.15.123:0/4291410623 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9b7c0597b0 con 0x7f9b8c0fde90 2024-03-30T05:07:46.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:46.312+0000 7f9b92400700 1 -- 172.21.15.123:0/4291410623 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f9b8c04bed0 con 0x7f9b8c0fde90 2024-03-30T05:07:46.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:46.312+0000 7f9b83400700 1 -- 172.21.15.123:0/4291410623 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f9b7c059340 con 0x7f9b8c0fde90 2024-03-30T05:07:46.310 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T05:07:46.311 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-30T05:07:46.311 INFO:teuthology.orchestra.run.smithi123.stdout: host smithi135 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi135/config 2024-03-30T05:07:46.311 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-30T05:07:46.311 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T05:07:46.311 INFO:teuthology.orchestra.run.smithi123.stdout:Traceback (most recent call last): 2024-03-30T05:07:46.311 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-30T05:07:46.311 INFO:teuthology.orchestra.run.smithi123.stdout: "__main__", mod_spec) 2024-03-30T05:07:46.311 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-30T05:07:46.311 INFO:teuthology.orchestra.run.smithi123.stdout: exec(code, run_globals) 2024-03-30T05:07:46.311 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10889, in 2024-03-30T05:07:46.311 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10877, in main 2024-03-30T05:07:46.311 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-30T05:07:46.312 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-30T05:07:46.312 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-30T05:07:46.312 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-30T05:07:46.312 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-03-30T05:07:46.312 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-30T05:07:46.312 INFO:teuthology.orchestra.run.smithi123.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T05:07:46.312 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T05:07:46.312 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T05:07:46.312 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T05:07:46.312 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T05:07:46.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:46.314+0000 7f9b92400700 1 -- 172.21.15.123:0/4291410623 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9b7006cc80 msgr2=0x7f9b7006f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:46.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:46.314+0000 7f9b92400700 1 --2- 172.21.15.123:0/4291410623 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9b7006cc80 0x7f9b7006f130 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c1115d0 tx=0x7f9b78009e30 comp rx=0 tx=0).stop 2024-03-30T05:07:46.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:46.314+0000 7f9b92400700 1 -- 172.21.15.123:0/4291410623 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b8c0fde90 msgr2=0x7f9b8c115d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:07:46.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:46.314+0000 7f9b92400700 1 --2- 172.21.15.123:0/4291410623 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b8c0fde90 0x7f9b8c115d30 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c002a50 tx=0x7f9b7c0046b0 comp rx=0 tx=0).stop 2024-03-30T05:07:46.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:46.314+0000 7f9b92400700 1 -- 172.21.15.123:0/4291410623 shutdown_connections 2024-03-30T05:07:46.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:46.314+0000 7f9b92400700 1 --2- 172.21.15.123:0/4291410623 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9b7006cc80 0x7f9b7006f130 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:46.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:46.314+0000 7f9b92400700 1 --2- 172.21.15.123:0/4291410623 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b8c0fde90 0x7f9b8c115d30 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:46.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:46.314+0000 7f9b92400700 1 --2- 172.21.15.123:0/4291410623 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9b8c0fe830 0x7f9b8c116270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:07:46.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:46.314+0000 7f9b92400700 1 -- 172.21.15.123:0/4291410623 >> 172.21.15.123:0/4291410623 conn(0x7f9b8c072bc0 msgr2=0x7f9b8c0fb840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:07:46.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:46.315+0000 7f9b92400700 1 -- 172.21.15.123:0/4291410623 shutdown_connections 2024-03-30T05:07:46.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:07:46.315+0000 7f9b92400700 1 -- 172.21.15.123:0/4291410623 wait complete. 2024-03-30T05:07:46.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:07:46 smithi135 ceph-mon[106011]: from='client.35976 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:07:46.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:07:46 smithi135 ceph-mon[106011]: from='client.35980 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:07:46.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:07:46 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/1716008479' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:07:46.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:07:46 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2172196444' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T05:07:46.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:07:46 smithi123 ceph-mon[139369]: from='client.35976 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:07:46.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:07:46 smithi123 ceph-mon[139369]: from='client.35980 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:07:46.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:07:46 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/1716008479' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:07:46.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:07:46 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2172196444' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T05:07:47.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:07:47 smithi135 ceph-mon[106011]: pgmap v1098: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:47.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:07:47 smithi135 ceph-mon[106011]: from='client.35992 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:07:47.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:07:47 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/4291410623' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T05:07:47.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:07:47 smithi123 ceph-mon[139369]: pgmap v1098: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:47.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:07:47 smithi123 ceph-mon[139369]: from='client.35992 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:07:47.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:07:47 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/4291410623' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T05:07:49.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:07:49 smithi135 ceph-mon[106011]: pgmap v1099: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:49.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:07:49 smithi123 ceph-mon[139369]: pgmap v1099: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:51.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:07:51 smithi135 ceph-mon[106011]: pgmap v1100: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:51.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:07:51 smithi123 ceph-mon[139369]: pgmap v1100: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:53.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:07:53 smithi135 ceph-mon[106011]: pgmap v1101: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:53.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:07:53 smithi123 ceph-mon[139369]: pgmap v1101: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:55.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:07:55 smithi135 ceph-mon[106011]: pgmap v1102: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:55.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:07:55 smithi123 ceph-mon[139369]: pgmap v1102: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:57.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:07:57 smithi135 ceph-mon[106011]: pgmap v1103: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:57.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:07:57 smithi123 ceph-mon[139369]: pgmap v1103: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:59.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:07:59 smithi123 ceph-mon[139369]: pgmap v1104: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:07:59.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:07:59 smithi135 ceph-mon[106011]: pgmap v1104: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:08:00.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:00 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:08:00.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:00 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:08:01.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:01 smithi123 ceph-mon[139369]: pgmap v1105: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:08:01.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:01 smithi135 ceph-mon[106011]: pgmap v1105: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:08:03.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:03 smithi123 ceph-mon[139369]: pgmap v1106: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:08:03.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:03 smithi135 ceph-mon[106011]: pgmap v1106: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:08:05.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:05 smithi123 ceph-mon[139369]: pgmap v1107: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:08:05.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:05 smithi135 ceph-mon[106011]: pgmap v1107: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:08:07.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:07 smithi123 ceph-mon[139369]: pgmap v1108: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:08:07.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:07 smithi135 ceph-mon[106011]: pgmap v1108: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:08:09.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:09 smithi123 ceph-mon[139369]: pgmap v1109: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:08:09.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:09 smithi135 ceph-mon[106011]: pgmap v1109: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:08:11.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:11 smithi123 ceph-mon[139369]: pgmap v1110: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:08:11.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:11 smithi135 ceph-mon[106011]: pgmap v1110: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:08:13.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:13 smithi123 ceph-mon[139369]: pgmap v1111: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:08:13.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:13 smithi135 ceph-mon[106011]: pgmap v1111: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:08:15.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:15 smithi123 ceph-mon[139369]: pgmap v1112: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:08:15.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:15 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:08:15.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:15 smithi135 ceph-mon[106011]: pgmap v1112: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:08:15.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:15 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:08:16.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.462+0000 7f3866400700 1 -- 172.21.15.123:0/3233794106 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38600708a0 msgr2=0x7f3860070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:08:16.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.462+0000 7f3866400700 1 --2- 172.21.15.123:0/3233794106 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38600708a0 0x7f3860070d10 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f384c004970 tx=0x7f384c009e30 comp rx=0 tx=0).stop 2024-03-30T05:08:16.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.462+0000 7f3866400700 1 -- 172.21.15.123:0/3233794106 shutdown_connections 2024-03-30T05:08:16.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.462+0000 7f3866400700 1 --2- 172.21.15.123:0/3233794106 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38600708a0 0x7f3860070d10 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:16.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.462+0000 7f3866400700 1 --2- 172.21.15.123:0/3233794106 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3860071f90 0x7f3860070360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:16.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.462+0000 7f3866400700 1 -- 172.21.15.123:0/3233794106 >> 172.21.15.123:0/3233794106 conn(0x7f38600f89b0 msgr2=0x7f38600fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:08:16.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.462+0000 7f3866400700 1 -- 172.21.15.123:0/3233794106 shutdown_connections 2024-03-30T05:08:16.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.462+0000 7f3866400700 1 -- 172.21.15.123:0/3233794106 wait complete. 2024-03-30T05:08:16.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.463+0000 7f3866400700 1 Processor -- start 2024-03-30T05:08:16.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.463+0000 7f3866400700 1 -- start start 2024-03-30T05:08:16.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.463+0000 7f3866400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38600708a0 0x7f386011a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:08:16.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.463+0000 7f3866400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3860071f90 0x7f386011a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:08:16.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.463+0000 7f3866400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f386011ac40 con 0x7f38600708a0 2024-03-30T05:08:16.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.463+0000 7f3866400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3860114010 con 0x7f3860071f90 2024-03-30T05:08:16.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.464+0000 7f385f400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38600708a0 0x7f386011a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:08:16.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.464+0000 7f385f400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38600708a0 0x7f386011a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36980/0 (socket says 172.21.15.123:36980) 2024-03-30T05:08:16.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.464+0000 7f385f400700 1 -- 172.21.15.123:0/235607879 learned_addr learned my addr 172.21.15.123:0/235607879 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:08:16.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.464+0000 7f385f400700 1 -- 172.21.15.123:0/235607879 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3860071f90 msgr2=0x7f386011a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:08:16.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.464+0000 7f385f400700 1 --2- 172.21.15.123:0/235607879 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3860071f90 0x7f386011a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:16.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.464+0000 7f385f400700 1 -- 172.21.15.123:0/235607879 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f384c005040 con 0x7f38600708a0 2024-03-30T05:08:16.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.464+0000 7f385f400700 1 --2- 172.21.15.123:0/235607879 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38600708a0 0x7f386011a0b0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f3850002a80 tx=0x7f38500046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:08:16.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.464+0000 7f3857400700 1 -- 172.21.15.123:0/235607879 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3850007cc0 con 0x7f38600708a0 2024-03-30T05:08:16.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.464+0000 7f3857400700 1 -- 172.21.15.123:0/235607879 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f385000f040 con 0x7f38600708a0 2024-03-30T05:08:16.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.464+0000 7f3866400700 1 -- 172.21.15.123:0/235607879 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38601142f0 con 0x7f38600708a0 2024-03-30T05:08:16.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.464+0000 7f3857400700 1 -- 172.21.15.123:0/235607879 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f385000da20 con 0x7f38600708a0 2024-03-30T05:08:16.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.465+0000 7f3866400700 1 -- 172.21.15.123:0/235607879 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3860114830 con 0x7f38600708a0 2024-03-30T05:08:16.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.466+0000 7f3866400700 1 -- 172.21.15.123:0/235607879 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3860071aa0 con 0x7f38600708a0 2024-03-30T05:08:16.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.466+0000 7f3857400700 1 -- 172.21.15.123:0/235607879 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f385000dba0 con 0x7f38600708a0 2024-03-30T05:08:16.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.467+0000 7f3857400700 1 --2- 172.21.15.123:0/235607879 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f384406cc30 0x7f384406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:08:16.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.467+0000 7f3857400700 1 -- 172.21.15.123:0/235607879 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f385008b840 con 0x7f38600708a0 2024-03-30T05:08:16.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.468+0000 7f385ea00700 1 --2- 172.21.15.123:0/235607879 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f384406cc30 0x7f384406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:08:16.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.468+0000 7f385ea00700 1 --2- 172.21.15.123:0/235607879 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f384406cc30 0x7f384406f0e0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f384c000c00 tx=0x7f384c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:08:16.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.472+0000 7f3857400700 1 -- 172.21.15.123:0/235607879 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3850059210 con 0x7f38600708a0 2024-03-30T05:08:16.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.692+0000 7f3866400700 1 -- 172.21.15.123:0/235607879 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f38601103e0 con 0x7f384406cc30 2024-03-30T05:08:16.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.694+0000 7f3857400700 1 -- 172.21.15.123:0/235607879 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f38601103e0 con 0x7f384406cc30 2024-03-30T05:08:16.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.696+0000 7f3866400700 1 -- 172.21.15.123:0/235607879 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f384406cc30 msgr2=0x7f384406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:08:16.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.696+0000 7f3866400700 1 --2- 172.21.15.123:0/235607879 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f384406cc30 0x7f384406f0e0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f384c000c00 tx=0x7f384c009e30 comp rx=0 tx=0).stop 2024-03-30T05:08:16.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.696+0000 7f3866400700 1 -- 172.21.15.123:0/235607879 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38600708a0 msgr2=0x7f386011a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:08:16.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.696+0000 7f3866400700 1 --2- 172.21.15.123:0/235607879 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38600708a0 0x7f386011a0b0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f3850002a80 tx=0x7f38500046b0 comp rx=0 tx=0).stop 2024-03-30T05:08:16.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.696+0000 7f3866400700 1 -- 172.21.15.123:0/235607879 shutdown_connections 2024-03-30T05:08:16.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.696+0000 7f3866400700 1 --2- 172.21.15.123:0/235607879 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f384406cc30 0x7f384406f0e0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:16.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.696+0000 7f3866400700 1 --2- 172.21.15.123:0/235607879 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38600708a0 0x7f386011a0b0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:16.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.696+0000 7f3866400700 1 --2- 172.21.15.123:0/235607879 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3860071f90 0x7f386011a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:16.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.696+0000 7f3866400700 1 -- 172.21.15.123:0/235607879 >> 172.21.15.123:0/235607879 conn(0x7f38600f89b0 msgr2=0x7f38600f9b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:08:16.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.696+0000 7f3866400700 1 -- 172.21.15.123:0/235607879 shutdown_connections 2024-03-30T05:08:16.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.696+0000 7f3866400700 1 -- 172.21.15.123:0/235607879 wait complete. 2024-03-30T05:08:16.709 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T05:08:16.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.831+0000 7fc07f400700 1 -- 172.21.15.123:0/1107426368 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0781037a0 msgr2=0x7fc0780fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:08:16.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.831+0000 7fc07f400700 1 --2- 172.21.15.123:0/1107426368 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0781037a0 0x7fc0780fce10 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fc064004940 tx=0x7fc064009e30 comp rx=0 tx=0).stop 2024-03-30T05:08:16.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.831+0000 7fc07f400700 1 -- 172.21.15.123:0/1107426368 shutdown_connections 2024-03-30T05:08:16.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.831+0000 7fc07f400700 1 --2- 172.21.15.123:0/1107426368 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0781037a0 0x7fc0780fce10 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:16.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.831+0000 7fc07f400700 1 --2- 172.21.15.123:0/1107426368 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc078102e00 0x7fc0781031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:16.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.831+0000 7fc07f400700 1 -- 172.21.15.123:0/1107426368 >> 172.21.15.123:0/1107426368 conn(0x7fc0780f89b0 msgr2=0x7fc0780fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:08:16.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.831+0000 7fc07f400700 1 -- 172.21.15.123:0/1107426368 shutdown_connections 2024-03-30T05:08:16.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.831+0000 7fc07f400700 1 -- 172.21.15.123:0/1107426368 wait complete. 2024-03-30T05:08:16.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.832+0000 7fc07f400700 1 Processor -- start 2024-03-30T05:08:16.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.832+0000 7fc07f400700 1 -- start start 2024-03-30T05:08:16.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.832+0000 7fc07f400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc078102e00 0x7fc0780702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:08:16.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.832+0000 7fc07f400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0781037a0 0x7fc078070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:08:16.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.832+0000 7fc07f400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc078070d90 con 0x7fc0781037a0 2024-03-30T05:08:16.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.832+0000 7fc07f400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc078070ed0 con 0x7fc078102e00 2024-03-30T05:08:16.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.832+0000 7fc077e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0781037a0 0x7fc078070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:08:16.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.832+0000 7fc077e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0781037a0 0x7fc078070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36996/0 (socket says 172.21.15.123:36996) 2024-03-30T05:08:16.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.832+0000 7fc077e00700 1 -- 172.21.15.123:0/2814659966 learned_addr learned my addr 172.21.15.123:0/2814659966 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:08:16.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.833+0000 7fc07ca00700 1 --2- 172.21.15.123:0/2814659966 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc078102e00 0x7fc0780702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:08:16.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.833+0000 7fc077e00700 1 -- 172.21.15.123:0/2814659966 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc078102e00 msgr2=0x7fc0780702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:08:16.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.833+0000 7fc077e00700 1 --2- 172.21.15.123:0/2814659966 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc078102e00 0x7fc0780702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:16.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.833+0000 7fc077e00700 1 -- 172.21.15.123:0/2814659966 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc064005040 con 0x7fc0781037a0 2024-03-30T05:08:16.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.833+0000 7fc07ca00700 1 --2- 172.21.15.123:0/2814659966 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc078102e00 0x7fc0780702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T05:08:16.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.833+0000 7fc077e00700 1 --2- 172.21.15.123:0/2814659966 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0781037a0 0x7fc078070800 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fc064009fd0 tx=0x7fc064009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:08:16.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.833+0000 7fc074c00700 1 -- 172.21.15.123:0/2814659966 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc06400e030 con 0x7fc0781037a0 2024-03-30T05:08:16.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.833+0000 7fc074c00700 1 -- 172.21.15.123:0/2814659966 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc064013e00 con 0x7fc0781037a0 2024-03-30T05:08:16.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.833+0000 7fc07f400700 1 -- 172.21.15.123:0/2814659966 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc078071120 con 0x7fc0781037a0 2024-03-30T05:08:16.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.833+0000 7fc074c00700 1 -- 172.21.15.123:0/2814659966 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc06401ccd0 con 0x7fc0781037a0 2024-03-30T05:08:16.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.834+0000 7fc07f400700 1 -- 172.21.15.123:0/2814659966 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc07811a1a0 con 0x7fc0781037a0 2024-03-30T05:08:16.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.835+0000 7fc07f400700 1 -- 172.21.15.123:0/2814659966 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0780fe550 con 0x7fc0781037a0 2024-03-30T05:08:16.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.835+0000 7fc074c00700 1 -- 172.21.15.123:0/2814659966 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fc06401c500 con 0x7fc0781037a0 2024-03-30T05:08:16.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.836+0000 7fc074c00700 1 --2- 172.21.15.123:0/2814659966 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc06006cc30 0x7fc06006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:08:16.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.836+0000 7fc074c00700 1 -- 172.21.15.123:0/2814659966 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc06408d8b0 con 0x7fc0781037a0 2024-03-30T05:08:16.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.836+0000 7fc07ca00700 1 --2- 172.21.15.123:0/2814659966 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc06006cc30 0x7fc06006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:08:16.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.837+0000 7fc07ca00700 1 --2- 172.21.15.123:0/2814659966 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc06006cc30 0x7fc06006f0e0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fc06c009fd0 tx=0x7fc06c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:08:16.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:16.841+0000 7fc074c00700 1 -- 172.21.15.123:0/2814659966 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc0640578f0 con 0x7fc0781037a0 2024-03-30T05:08:17.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.060+0000 7fc07f400700 1 -- 172.21.15.123:0/2814659966 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc078109e40 con 0x7fc06006cc30 2024-03-30T05:08:17.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.061+0000 7fc074c00700 1 -- 172.21.15.123:0/2814659966 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fc06c009420 con 0x7fc06006cc30 2024-03-30T05:08:17.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.063+0000 7fc07f400700 1 -- 172.21.15.123:0/2814659966 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc06006cc30 msgr2=0x7fc06006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:08:17.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.063+0000 7fc07f400700 1 --2- 172.21.15.123:0/2814659966 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc06006cc30 0x7fc06006f0e0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fc06c009fd0 tx=0x7fc06c00b040 comp rx=0 tx=0).stop 2024-03-30T05:08:17.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.063+0000 7fc07f400700 1 -- 172.21.15.123:0/2814659966 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0781037a0 msgr2=0x7fc078070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:08:17.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.063+0000 7fc07f400700 1 --2- 172.21.15.123:0/2814659966 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0781037a0 0x7fc078070800 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fc064009fd0 tx=0x7fc064009e30 comp rx=0 tx=0).stop 2024-03-30T05:08:17.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.063+0000 7fc07f400700 1 -- 172.21.15.123:0/2814659966 shutdown_connections 2024-03-30T05:08:17.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.063+0000 7fc07f400700 1 --2- 172.21.15.123:0/2814659966 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc06006cc30 0x7fc06006f0e0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:17.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.063+0000 7fc07f400700 1 --2- 172.21.15.123:0/2814659966 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc078102e00 0x7fc0780702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:17.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.063+0000 7fc07f400700 1 --2- 172.21.15.123:0/2814659966 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0781037a0 0x7fc078070800 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:17.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.063+0000 7fc07f400700 1 -- 172.21.15.123:0/2814659966 >> 172.21.15.123:0/2814659966 conn(0x7fc0780f89b0 msgr2=0x7fc0780f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:08:17.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.064+0000 7fc07f400700 1 -- 172.21.15.123:0/2814659966 shutdown_connections 2024-03-30T05:08:17.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.064+0000 7fc07f400700 1 -- 172.21.15.123:0/2814659966 wait complete. 2024-03-30T05:08:17.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.196+0000 7f1f38e00700 1 -- 172.21.15.123:0/3051819481 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f34102e00 msgr2=0x7f1f341031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:08:17.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.196+0000 7f1f38e00700 1 --2- 172.21.15.123:0/3051819481 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f34102e00 0x7f1f341031d0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f1f24004940 tx=0x7f1f24009e30 comp rx=0 tx=0).stop 2024-03-30T05:08:17.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.196+0000 7f1f38e00700 1 -- 172.21.15.123:0/3051819481 shutdown_connections 2024-03-30T05:08:17.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.196+0000 7f1f38e00700 1 --2- 172.21.15.123:0/3051819481 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1f341037a0 0x7f1f340fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:17.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.196+0000 7f1f38e00700 1 --2- 172.21.15.123:0/3051819481 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f34102e00 0x7f1f341031d0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:17.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.196+0000 7f1f38e00700 1 -- 172.21.15.123:0/3051819481 >> 172.21.15.123:0/3051819481 conn(0x7f1f340f89b0 msgr2=0x7f1f340fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:08:17.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.196+0000 7f1f38e00700 1 -- 172.21.15.123:0/3051819481 shutdown_connections 2024-03-30T05:08:17.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.197+0000 7f1f38e00700 1 -- 172.21.15.123:0/3051819481 wait complete. 2024-03-30T05:08:17.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.197+0000 7f1f38e00700 1 Processor -- start 2024-03-30T05:08:17.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.197+0000 7f1f38e00700 1 -- start start 2024-03-30T05:08:17.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.198+0000 7f1f38e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1f34102e00 0x7f1f34115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:08:17.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.198+0000 7f1f38e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f341037a0 0x7f1f34116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:08:17.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.198+0000 7f1f38e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f34116860 con 0x7f1f341037a0 2024-03-30T05:08:17.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.198+0000 7f1f38e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3410fc30 con 0x7f1f34102e00 2024-03-30T05:08:17.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.198+0000 7f1f31400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f341037a0 0x7f1f34116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:08:17.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.198+0000 7f1f31400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f341037a0 0x7f1f34116210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37024/0 (socket says 172.21.15.123:37024) 2024-03-30T05:08:17.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.198+0000 7f1f31400700 1 -- 172.21.15.123:0/332473896 learned_addr learned my addr 172.21.15.123:0/332473896 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:08:17.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.198+0000 7f1f31e00700 1 --2- 172.21.15.123:0/332473896 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1f34102e00 0x7f1f34115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:08:17.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.198+0000 7f1f31400700 1 -- 172.21.15.123:0/332473896 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1f34102e00 msgr2=0x7f1f34115cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:08:17.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.198+0000 7f1f31400700 1 --2- 172.21.15.123:0/332473896 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1f34102e00 0x7f1f34115cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:17.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.198+0000 7f1f31400700 1 -- 172.21.15.123:0/332473896 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f24005040 con 0x7f1f341037a0 2024-03-30T05:08:17.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.199+0000 7f1f31400700 1 --2- 172.21.15.123:0/332473896 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f341037a0 0x7f1f34116210 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f1f20002a50 tx=0x7f1f20004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:08:17.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.199+0000 7f1f2a000700 1 -- 172.21.15.123:0/332473896 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f20007cc0 con 0x7f1f341037a0 2024-03-30T05:08:17.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.199+0000 7f1f2a000700 1 -- 172.21.15.123:0/332473896 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1f2000f040 con 0x7f1f341037a0 2024-03-30T05:08:17.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.199+0000 7f1f2a000700 1 -- 172.21.15.123:0/332473896 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f2000d920 con 0x7f1f341037a0 2024-03-30T05:08:17.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.199+0000 7f1f38e00700 1 -- 172.21.15.123:0/332473896 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f3410ff10 con 0x7f1f341037a0 2024-03-30T05:08:17.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.199+0000 7f1f38e00700 1 -- 172.21.15.123:0/332473896 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f341104d0 con 0x7f1f341037a0 2024-03-30T05:08:17.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.200+0000 7f1f38e00700 1 -- 172.21.15.123:0/332473896 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f340fe550 con 0x7f1f341037a0 2024-03-30T05:08:17.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.201+0000 7f1f2a000700 1 -- 172.21.15.123:0/332473896 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f1f20009680 con 0x7f1f341037a0 2024-03-30T05:08:17.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.202+0000 7f1f2a000700 1 --2- 172.21.15.123:0/332473896 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1f1806cc30 0x7f1f1806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:08:17.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.202+0000 7f1f2a000700 1 -- 172.21.15.123:0/332473896 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f1f2008baf0 con 0x7f1f341037a0 2024-03-30T05:08:17.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.202+0000 7f1f31e00700 1 --2- 172.21.15.123:0/332473896 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1f1806cc30 0x7f1f1806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:08:17.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.203+0000 7f1f31e00700 1 --2- 172.21.15.123:0/332473896 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1f1806cc30 0x7f1f1806f0e0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f1f2400b840 tx=0x7f1f24009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:08:17.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.206+0000 7f1f2a000700 1 -- 172.21.15.123:0/332473896 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f1f20059540 con 0x7f1f341037a0 2024-03-30T05:08:17.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.422+0000 7f1f38e00700 1 -- 172.21.15.123:0/332473896 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f1f34109e60 con 0x7f1f1806cc30 2024-03-30T05:08:17.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.432+0000 7f1f2a000700 1 -- 172.21.15.123:0/332473896 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f1f24008ee0 con 0x7f1f1806cc30 2024-03-30T05:08:17.430 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T05:08:17.430 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (37m) 14m ago 49m 14.6M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T05:08:17.430 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (34m) 14m ago 50m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T05:08:17.431 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (34m) 14m ago 49m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T05:08:17.431 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (37m) 14m ago 49m 73.5M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T05:08:17.431 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (30m) 14m ago 43m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T05:08:17.431 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (30m) 14m ago 43m 16.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T05:08:17.431 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (14m) 14m ago 43m 10.7M - 18.2.2-1018-g470100a4 e7c444a08cf7 87b073a6c9ca 2024-03-30T05:08:17.431 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (30m) 14m ago 43m 16.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T05:08:17.431 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (39m) 14m ago 51m 510M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T05:08:17.431 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (38m) 14m ago 48m 456M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T05:08:17.431 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (35m) 14m ago 51m 70.0M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T05:08:17.431 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (35m) 14m ago 48m 51.1M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T05:08:17.432 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (38m) 14m ago 50m 8636k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T05:08:17.432 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (38m) 14m ago 48m 8678k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T05:08:17.432 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (34m) 14m ago 47m 107M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T05:08:17.432 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (33m) 14m ago 47m 97.2M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T05:08:17.432 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (33m) 14m ago 46m 89.8M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T05:08:17.432 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (32m) 14m ago 46m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T05:08:17.432 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (31m) 14m ago 45m 101M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T05:08:17.432 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (31m) 14m ago 45m 88.8M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T05:08:17.432 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (37m) 14m ago 49m 63.9M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T05:08:17.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.434+0000 7f1f38e00700 1 -- 172.21.15.123:0/332473896 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1f1806cc30 msgr2=0x7f1f1806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:08:17.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.434+0000 7f1f38e00700 1 --2- 172.21.15.123:0/332473896 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1f1806cc30 0x7f1f1806f0e0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f1f2400b840 tx=0x7f1f24009e30 comp rx=0 tx=0).stop 2024-03-30T05:08:17.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.434+0000 7f1f38e00700 1 -- 172.21.15.123:0/332473896 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f341037a0 msgr2=0x7f1f34116210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:08:17.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.434+0000 7f1f38e00700 1 --2- 172.21.15.123:0/332473896 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f341037a0 0x7f1f34116210 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f1f20002a50 tx=0x7f1f20004600 comp rx=0 tx=0).stop 2024-03-30T05:08:17.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.434+0000 7f1f38e00700 1 -- 172.21.15.123:0/332473896 shutdown_connections 2024-03-30T05:08:17.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.434+0000 7f1f38e00700 1 --2- 172.21.15.123:0/332473896 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1f1806cc30 0x7f1f1806f0e0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:17.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.434+0000 7f1f38e00700 1 --2- 172.21.15.123:0/332473896 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1f34102e00 0x7f1f34115cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:17.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.434+0000 7f1f38e00700 1 --2- 172.21.15.123:0/332473896 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f341037a0 0x7f1f34116210 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:17.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.434+0000 7f1f38e00700 1 -- 172.21.15.123:0/332473896 >> 172.21.15.123:0/332473896 conn(0x7f1f340f89b0 msgr2=0x7f1f340f99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:08:17.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.435+0000 7f1f38e00700 1 -- 172.21.15.123:0/332473896 shutdown_connections 2024-03-30T05:08:17.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.435+0000 7f1f38e00700 1 -- 172.21.15.123:0/332473896 wait complete. 2024-03-30T05:08:17.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:17 smithi123 ceph-mon[139369]: pgmap v1113: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:08:17.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.567+0000 7fd767400700 1 -- 172.21.15.123:0/605420548 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd760102e00 msgr2=0x7fd7601031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:08:17.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.567+0000 7fd767400700 1 --2- 172.21.15.123:0/605420548 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd760102e00 0x7fd7601031d0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fd754004940 tx=0x7fd754009e30 comp rx=0 tx=0).stop 2024-03-30T05:08:17.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.568+0000 7fd767400700 1 -- 172.21.15.123:0/605420548 shutdown_connections 2024-03-30T05:08:17.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.568+0000 7fd767400700 1 --2- 172.21.15.123:0/605420548 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd7601037a0 0x7fd7600fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:17.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.568+0000 7fd767400700 1 --2- 172.21.15.123:0/605420548 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd760102e00 0x7fd7601031d0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:17.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.568+0000 7fd767400700 1 -- 172.21.15.123:0/605420548 >> 172.21.15.123:0/605420548 conn(0x7fd7600f89b0 msgr2=0x7fd7600fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:08:17.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.568+0000 7fd767400700 1 -- 172.21.15.123:0/605420548 shutdown_connections 2024-03-30T05:08:17.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.568+0000 7fd767400700 1 -- 172.21.15.123:0/605420548 wait complete. 2024-03-30T05:08:17.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.569+0000 7fd767400700 1 Processor -- start 2024-03-30T05:08:17.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.569+0000 7fd767400700 1 -- start start 2024-03-30T05:08:17.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.569+0000 7fd767400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd760102e00 0x7fd760115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:08:17.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.569+0000 7fd767400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7601037a0 0x7fd760116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:08:17.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.569+0000 7fd767400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd760116880 con 0x7fd7601037a0 2024-03-30T05:08:17.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.569+0000 7fd767400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd76010fc50 con 0x7fd760102e00 2024-03-30T05:08:17.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.569+0000 7fd75fe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7601037a0 0x7fd760116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:08:17.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.569+0000 7fd75fe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7601037a0 0x7fd760116230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37046/0 (socket says 172.21.15.123:37046) 2024-03-30T05:08:17.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.569+0000 7fd75fe00700 1 -- 172.21.15.123:0/3701881575 learned_addr learned my addr 172.21.15.123:0/3701881575 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:08:17.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.570+0000 7fd764a00700 1 --2- 172.21.15.123:0/3701881575 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd760102e00 0x7fd760115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:08:17.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.570+0000 7fd75fe00700 1 -- 172.21.15.123:0/3701881575 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd760102e00 msgr2=0x7fd760115cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:08:17.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.570+0000 7fd75fe00700 1 --2- 172.21.15.123:0/3701881575 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd760102e00 0x7fd760115cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:17.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.570+0000 7fd75fe00700 1 -- 172.21.15.123:0/3701881575 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd754005040 con 0x7fd7601037a0 2024-03-30T05:08:17.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.570+0000 7fd75fe00700 1 --2- 172.21.15.123:0/3701881575 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7601037a0 0x7fd760116230 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fd74c00c890 tx=0x7fd74c0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:08:17.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.570+0000 7fd75cc00700 1 -- 172.21.15.123:0/3701881575 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd74c00a7d0 con 0x7fd7601037a0 2024-03-30T05:08:17.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.570+0000 7fd75cc00700 1 -- 172.21.15.123:0/3701881575 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd74c00ade0 con 0x7fd7601037a0 2024-03-30T05:08:17.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.570+0000 7fd767400700 1 -- 172.21.15.123:0/3701881575 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd76010ff30 con 0x7fd7601037a0 2024-03-30T05:08:17.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.570+0000 7fd75cc00700 1 -- 172.21.15.123:0/3701881575 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd74c0113a0 con 0x7fd7601037a0 2024-03-30T05:08:17.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.571+0000 7fd767400700 1 -- 172.21.15.123:0/3701881575 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7601104f0 con 0x7fd7601037a0 2024-03-30T05:08:17.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.571+0000 7fd767400700 1 -- 172.21.15.123:0/3701881575 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7600fe550 con 0x7fd7601037a0 2024-03-30T05:08:17.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.572+0000 7fd75cc00700 1 -- 172.21.15.123:0/3701881575 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fd74c012400 con 0x7fd7601037a0 2024-03-30T05:08:17.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.574+0000 7fd75cc00700 1 --2- 172.21.15.123:0/3701881575 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd74806cc30 0x7fd74806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:08:17.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.575+0000 7fd75cc00700 1 -- 172.21.15.123:0/3701881575 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd74c08c1f0 con 0x7fd7601037a0 2024-03-30T05:08:17.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.575+0000 7fd764a00700 1 --2- 172.21.15.123:0/3701881575 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd74806cc30 0x7fd74806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:08:17.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.575+0000 7fd764a00700 1 --2- 172.21.15.123:0/3701881575 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd74806cc30 0x7fd74806f0e0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fd75400b3b0 tx=0x7fd754009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:08:17.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.579+0000 7fd75cc00700 1 -- 172.21.15.123:0/3701881575 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd74c059c40 con 0x7fd7601037a0 2024-03-30T05:08:17.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:17 smithi135 ceph-mon[106011]: pgmap v1113: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:08:17.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.864+0000 7fd767400700 1 -- 172.21.15.123:0/3701881575 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fd760109dc0 con 0x7fd7601037a0 2024-03-30T05:08:17.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.865+0000 7fd75cc00700 1 -- 172.21.15.123:0/3701881575 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7fd74c0597d0 con 0x7fd7601037a0 2024-03-30T05:08:17.863 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T05:08:17.863 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T05:08:17.863 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T05:08:17.864 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:08:17.864 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T05:08:17.864 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T05:08:17.864 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:08:17.864 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T05:08:17.864 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T05:08:17.864 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:08:17.864 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T05:08:17.864 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 4 2024-03-30T05:08:17.864 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:08:17.864 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T05:08:17.864 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 14 2024-03-30T05:08:17.864 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T05:08:17.864 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T05:08:17.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.867+0000 7fd767400700 1 -- 172.21.15.123:0/3701881575 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd74806cc30 msgr2=0x7fd74806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:08:17.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.867+0000 7fd767400700 1 --2- 172.21.15.123:0/3701881575 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd74806cc30 0x7fd74806f0e0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fd75400b3b0 tx=0x7fd754009e30 comp rx=0 tx=0).stop 2024-03-30T05:08:17.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.867+0000 7fd767400700 1 -- 172.21.15.123:0/3701881575 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7601037a0 msgr2=0x7fd760116230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:08:17.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.867+0000 7fd767400700 1 --2- 172.21.15.123:0/3701881575 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7601037a0 0x7fd760116230 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fd74c00c890 tx=0x7fd74c0044c0 comp rx=0 tx=0).stop 2024-03-30T05:08:17.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.867+0000 7fd767400700 1 -- 172.21.15.123:0/3701881575 shutdown_connections 2024-03-30T05:08:17.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.867+0000 7fd767400700 1 --2- 172.21.15.123:0/3701881575 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd74806cc30 0x7fd74806f0e0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:17.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.867+0000 7fd767400700 1 --2- 172.21.15.123:0/3701881575 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd760102e00 0x7fd760115cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:17.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.867+0000 7fd767400700 1 --2- 172.21.15.123:0/3701881575 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7601037a0 0x7fd760116230 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:17.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.867+0000 7fd767400700 1 -- 172.21.15.123:0/3701881575 >> 172.21.15.123:0/3701881575 conn(0x7fd7600f89b0 msgr2=0x7fd7600706d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:08:17.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.868+0000 7fd767400700 1 -- 172.21.15.123:0/3701881575 shutdown_connections 2024-03-30T05:08:17.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:17.868+0000 7fd767400700 1 -- 172.21.15.123:0/3701881575 wait complete. 2024-03-30T05:08:18.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.002+0000 7f7661e00700 1 -- 172.21.15.123:0/583658857 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f765c102d80 msgr2=0x7f765c103150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:08:18.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.002+0000 7f7661e00700 1 --2- 172.21.15.123:0/583658857 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f765c102d80 0x7f765c103150 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f764c004940 tx=0x7f764c009e30 comp rx=0 tx=0).stop 2024-03-30T05:08:18.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.002+0000 7f7661e00700 1 -- 172.21.15.123:0/583658857 shutdown_connections 2024-03-30T05:08:18.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.002+0000 7f7661e00700 1 --2- 172.21.15.123:0/583658857 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f765c103720 0x7f765c0fcd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:18.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.002+0000 7f7661e00700 1 --2- 172.21.15.123:0/583658857 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f765c102d80 0x7f765c103150 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:18.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.002+0000 7f7661e00700 1 -- 172.21.15.123:0/583658857 >> 172.21.15.123:0/583658857 conn(0x7f765c0f8930 msgr2=0x7f765c0fad40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:08:18.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.002+0000 7f7661e00700 1 -- 172.21.15.123:0/583658857 shutdown_connections 2024-03-30T05:08:18.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.002+0000 7f7661e00700 1 -- 172.21.15.123:0/583658857 wait complete. 2024-03-30T05:08:18.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.003+0000 7f7661e00700 1 Processor -- start 2024-03-30T05:08:18.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.003+0000 7f7661e00700 1 -- start start 2024-03-30T05:08:18.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.004+0000 7f7661e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f765c102d80 0x7f765c115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:08:18.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.004+0000 7f7661e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f765c103720 0x7f765c1161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:08:18.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.004+0000 7f7661e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765c116800 con 0x7f765c103720 2024-03-30T05:08:18.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.004+0000 7f7661e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765c10fbd0 con 0x7f765c102d80 2024-03-30T05:08:18.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.004+0000 7f765a000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f765c103720 0x7f765c1161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:08:18.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.004+0000 7f765a000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f765c103720 0x7f765c1161b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37056/0 (socket says 172.21.15.123:37056) 2024-03-30T05:08:18.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.004+0000 7f765a000700 1 -- 172.21.15.123:0/251245532 learned_addr learned my addr 172.21.15.123:0/251245532 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:08:18.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.004+0000 7f765aa00700 1 --2- 172.21.15.123:0/251245532 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f765c102d80 0x7f765c115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:08:18.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.004+0000 7f765a000700 1 -- 172.21.15.123:0/251245532 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f765c102d80 msgr2=0x7f765c115c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:08:18.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.004+0000 7f765a000700 1 --2- 172.21.15.123:0/251245532 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f765c102d80 0x7f765c115c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:18.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.004+0000 7f765a000700 1 -- 172.21.15.123:0/251245532 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f764c005040 con 0x7f765c103720 2024-03-30T05:08:18.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.005+0000 7f765a000700 1 --2- 172.21.15.123:0/251245532 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f765c103720 0x7f765c1161b0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f764800c890 tx=0x7f76480044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:08:18.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.005+0000 7f7652a00700 1 -- 172.21.15.123:0/251245532 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f764800a7d0 con 0x7f765c103720 2024-03-30T05:08:18.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.005+0000 7f7652a00700 1 -- 172.21.15.123:0/251245532 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f764800ade0 con 0x7f765c103720 2024-03-30T05:08:18.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.005+0000 7f7661e00700 1 -- 172.21.15.123:0/251245532 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f765c10feb0 con 0x7f765c103720 2024-03-30T05:08:18.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.005+0000 7f7652a00700 1 -- 172.21.15.123:0/251245532 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f76480113a0 con 0x7f765c103720 2024-03-30T05:08:18.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.005+0000 7f7661e00700 1 -- 172.21.15.123:0/251245532 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f765c110470 con 0x7f765c103720 2024-03-30T05:08:18.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.006+0000 7f7661e00700 1 -- 172.21.15.123:0/251245532 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f765c0fe4d0 con 0x7f765c103720 2024-03-30T05:08:18.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.007+0000 7f7652a00700 1 -- 172.21.15.123:0/251245532 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f7648012400 con 0x7f765c103720 2024-03-30T05:08:18.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.008+0000 7f7652a00700 1 --2- 172.21.15.123:0/251245532 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f764006cc30 0x7f764006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:08:18.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.008+0000 7f7652a00700 1 -- 172.21.15.123:0/251245532 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f764808c1f0 con 0x7f765c103720 2024-03-30T05:08:18.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.009+0000 7f765aa00700 1 --2- 172.21.15.123:0/251245532 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f764006cc30 0x7f764006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:08:18.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.009+0000 7f765aa00700 1 --2- 172.21.15.123:0/251245532 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f764006cc30 0x7f764006f0e0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f764c00b020 tx=0x7f764c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:08:18.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.013+0000 7f7652a00700 1 -- 172.21.15.123:0/251245532 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7648059c40 con 0x7f765c103720 2024-03-30T05:08:18.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.262+0000 7f7661e00700 1 -- 172.21.15.123:0/251245532 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f765c04bed0 con 0x7f765c103720 2024-03-30T05:08:18.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.262+0000 7f7652a00700 1 -- 172.21.15.123:0/251245532 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 34 v34) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f76480597d0 con 0x7f765c103720 2024-03-30T05:08:18.260 INFO:teuthology.orchestra.run.smithi123.stdout:e34 2024-03-30T05:08:18.261 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T05:08:18.261 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T05:08:18.261 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T05:08:18.261 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:08:18.261 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T05:08:18.261 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T05:08:18.261 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 25 2024-03-30T05:08:18.261 INFO:teuthology.orchestra.run.smithi123.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-30T05:08:18.261 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T05:08:18.261 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T04:37:19.516763+0000 2024-03-30T05:08:18.261 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T05:08:18.261 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T05:08:18.261 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T05:08:18.261 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T05:08:18.262 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T05:08:18.262 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T05:08:18.262 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T05:08:18.262 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T05:08:18.262 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T05:08:18.262 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T05:08:18.262 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T05:08:18.262 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T05:08:18.262 INFO:teuthology.orchestra.run.smithi123.stdout:up {} 2024-03-30T05:08:18.262 INFO:teuthology.orchestra.run.smithi123.stdout:failed 0,1 2024-03-30T05:08:18.262 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T05:08:18.262 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T05:08:18.263 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T05:08:18.263 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T05:08:18.263 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T05:08:18.263 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T05:08:18.263 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T05:08:18.263 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T05:08:18.263 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:08:18.263 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:08:18.263 INFO:teuthology.orchestra.run.smithi123.stdout:Standby daemons: 2024-03-30T05:08:18.263 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:08:18.263 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:08:18.263 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{-1:34452} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:08:18.263 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{-1:44355} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:08:18.263 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{-1:44975} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/3788988014,v1:172.21.15.135:6827/3788988014] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:08:18.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.265+0000 7f7661e00700 1 -- 172.21.15.123:0/251245532 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f764006cc30 msgr2=0x7f764006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:08:18.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.265+0000 7f7661e00700 1 --2- 172.21.15.123:0/251245532 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f764006cc30 0x7f764006f0e0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f764c00b020 tx=0x7f764c009e30 comp rx=0 tx=0).stop 2024-03-30T05:08:18.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.265+0000 7f7661e00700 1 -- 172.21.15.123:0/251245532 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f765c103720 msgr2=0x7f765c1161b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:08:18.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.265+0000 7f7661e00700 1 --2- 172.21.15.123:0/251245532 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f765c103720 0x7f765c1161b0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f764800c890 tx=0x7f76480044c0 comp rx=0 tx=0).stop 2024-03-30T05:08:18.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.265+0000 7f7661e00700 1 -- 172.21.15.123:0/251245532 shutdown_connections 2024-03-30T05:08:18.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.265+0000 7f7661e00700 1 --2- 172.21.15.123:0/251245532 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f764006cc30 0x7f764006f0e0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:18.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.265+0000 7f7661e00700 1 --2- 172.21.15.123:0/251245532 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f765c102d80 0x7f765c115c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:18.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.265+0000 7f7661e00700 1 --2- 172.21.15.123:0/251245532 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f765c103720 0x7f765c1161b0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:18.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.265+0000 7f7661e00700 1 -- 172.21.15.123:0/251245532 >> 172.21.15.123:0/251245532 conn(0x7f765c0f8930 msgr2=0x7f765c070560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:08:18.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.265+0000 7f7661e00700 1 -- 172.21.15.123:0/251245532 shutdown_connections 2024-03-30T05:08:18.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.265+0000 7f7661e00700 1 -- 172.21.15.123:0/251245532 wait complete. 2024-03-30T05:08:18.265 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 34 2024-03-30T05:08:18.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.398+0000 7f4f3d000700 1 -- 172.21.15.123:0/3989181620 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f38070300 msgr2=0x7f4f380706d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:08:18.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.398+0000 7f4f3d000700 1 --2- 172.21.15.123:0/3989181620 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f38070300 0x7f4f380706d0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f4f28004970 tx=0x7f4f28009e30 comp rx=0 tx=0).stop 2024-03-30T05:08:18.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.398+0000 7f4f3d000700 1 -- 172.21.15.123:0/3989181620 shutdown_connections 2024-03-30T05:08:18.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.398+0000 7f4f3d000700 1 --2- 172.21.15.123:0/3989181620 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4f38070c10 0x7f4f38109190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:18.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.398+0000 7f4f3d000700 1 --2- 172.21.15.123:0/3989181620 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f38070300 0x7f4f380706d0 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:18.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.398+0000 7f4f3d000700 1 -- 172.21.15.123:0/3989181620 >> 172.21.15.123:0/3989181620 conn(0x7f4f380f8970 msgr2=0x7f4f380fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:08:18.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.399+0000 7f4f3d000700 1 -- 172.21.15.123:0/3989181620 shutdown_connections 2024-03-30T05:08:18.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.399+0000 7f4f3d000700 1 -- 172.21.15.123:0/3989181620 wait complete. 2024-03-30T05:08:18.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.399+0000 7f4f3d000700 1 Processor -- start 2024-03-30T05:08:18.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.399+0000 7f4f3d000700 1 -- start start 2024-03-30T05:08:18.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.400+0000 7f4f3d000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4f38070300 0x7f4f38115c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:08:18.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.400+0000 7f4f3d000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f38070c10 0x7f4f381161d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:08:18.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.400+0000 7f4f3d000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f38116820 con 0x7f4f38070c10 2024-03-30T05:08:18.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.400+0000 7f4f3d000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3810fc30 con 0x7f4f38070300 2024-03-30T05:08:18.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.400+0000 7f4f35400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f38070c10 0x7f4f381161d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:08:18.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.400+0000 7f4f35400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f38070c10 0x7f4f381161d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37072/0 (socket says 172.21.15.123:37072) 2024-03-30T05:08:18.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.400+0000 7f4f35400700 1 -- 172.21.15.123:0/2019761571 learned_addr learned my addr 172.21.15.123:0/2019761571 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:08:18.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.400+0000 7f4f35e00700 1 --2- 172.21.15.123:0/2019761571 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4f38070300 0x7f4f38115c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:08:18.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.401+0000 7f4f35400700 1 -- 172.21.15.123:0/2019761571 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4f38070300 msgr2=0x7f4f38115c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:08:18.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.401+0000 7f4f35400700 1 --2- 172.21.15.123:0/2019761571 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4f38070300 0x7f4f38115c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:18.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.401+0000 7f4f35400700 1 -- 172.21.15.123:0/2019761571 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f28005040 con 0x7f4f38070c10 2024-03-30T05:08:18.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.401+0000 7f4f35400700 1 --2- 172.21.15.123:0/2019761571 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f38070c10 0x7f4f381161d0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f4f24002a80 tx=0x7f4f240046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:08:18.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.401+0000 7f4f2e000700 1 -- 172.21.15.123:0/2019761571 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f24007cc0 con 0x7f4f38070c10 2024-03-30T05:08:18.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.401+0000 7f4f2e000700 1 -- 172.21.15.123:0/2019761571 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4f2400f040 con 0x7f4f38070c10 2024-03-30T05:08:18.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.401+0000 7f4f3d000700 1 -- 172.21.15.123:0/2019761571 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f3810ff10 con 0x7f4f38070c10 2024-03-30T05:08:18.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.401+0000 7f4f2e000700 1 -- 172.21.15.123:0/2019761571 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f2400da60 con 0x7f4f38070c10 2024-03-30T05:08:18.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.401+0000 7f4f3d000700 1 -- 172.21.15.123:0/2019761571 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f38110450 con 0x7f4f38070c10 2024-03-30T05:08:18.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.402+0000 7f4f3d000700 1 -- 172.21.15.123:0/2019761571 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f38071a40 con 0x7f4f38070c10 2024-03-30T05:08:18.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.403+0000 7f4f2e000700 1 -- 172.21.15.123:0/2019761571 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f4f2400dbc0 con 0x7f4f38070c10 2024-03-30T05:08:18.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.404+0000 7f4f2e000700 1 --2- 172.21.15.123:0/2019761571 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4f1c06cc30 0x7f4f1c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:08:18.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.404+0000 7f4f2e000700 1 -- 172.21.15.123:0/2019761571 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f4f2408bc70 con 0x7f4f38070c10 2024-03-30T05:08:18.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.404+0000 7f4f35e00700 1 --2- 172.21.15.123:0/2019761571 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4f1c06cc30 0x7f4f1c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:08:18.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.405+0000 7f4f35e00700 1 --2- 172.21.15.123:0/2019761571 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4f1c06cc30 0x7f4f1c06f0e0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f4f2800b370 tx=0x7f4f28009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:08:18.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.408+0000 7f4f2e000700 1 -- 172.21.15.123:0/2019761571 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4f24092080 con 0x7f4f38070c10 2024-03-30T05:08:18.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:18 smithi123 ceph-mon[139369]: from='client.36000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:08:18.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:18 smithi123 ceph-mon[139369]: from='client.36004 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:08:18.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:18 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3701881575' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:08:18.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:18 smithi135 ceph-mon[106011]: from='client.36000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:08:18.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:18 smithi135 ceph-mon[106011]: from='client.36004 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:08:18.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:18 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3701881575' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:08:18.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.630+0000 7f4f3d000700 1 -- 172.21.15.123:0/2019761571 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4f3810bfd0 con 0x7f4f1c06cc30 2024-03-30T05:08:18.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.631+0000 7f4f2e000700 1 -- 172.21.15.123:0/2019761571 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f4f3810bfd0 con 0x7f4f1c06cc30 2024-03-30T05:08:18.630 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T05:08:18.630 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35", 2024-03-30T05:08:18.630 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": true, 2024-03-30T05:08:18.630 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-30T05:08:18.630 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [ 2024-03-30T05:08:18.630 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-03-30T05:08:18.630 INFO:teuthology.orchestra.run.smithi123.stdout: "mds", 2024-03-30T05:08:18.630 INFO:teuthology.orchestra.run.smithi123.stdout: "osd", 2024-03-30T05:08:18.630 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr", 2024-03-30T05:08:18.630 INFO:teuthology.orchestra.run.smithi123.stdout: "mon" 2024-03-30T05:08:18.630 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-03-30T05:08:18.631 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": "16/21 daemons upgraded", 2024-03-30T05:08:18.631 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "Currently upgrading mds daemons", 2024-03-30T05:08:18.631 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T05:08:18.631 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T05:08:18.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.634+0000 7f4f3d000700 1 -- 172.21.15.123:0/2019761571 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4f1c06cc30 msgr2=0x7f4f1c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:08:18.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.634+0000 7f4f3d000700 1 --2- 172.21.15.123:0/2019761571 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4f1c06cc30 0x7f4f1c06f0e0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f4f2800b370 tx=0x7f4f28009e30 comp rx=0 tx=0).stop 2024-03-30T05:08:18.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.634+0000 7f4f3d000700 1 -- 172.21.15.123:0/2019761571 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f38070c10 msgr2=0x7f4f381161d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:08:18.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.634+0000 7f4f3d000700 1 --2- 172.21.15.123:0/2019761571 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f38070c10 0x7f4f381161d0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f4f24002a80 tx=0x7f4f240046b0 comp rx=0 tx=0).stop 2024-03-30T05:08:18.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.634+0000 7f4f3d000700 1 -- 172.21.15.123:0/2019761571 shutdown_connections 2024-03-30T05:08:18.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.634+0000 7f4f3d000700 1 --2- 172.21.15.123:0/2019761571 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f4f1c06cc30 0x7f4f1c06f0e0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:18.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.634+0000 7f4f3d000700 1 --2- 172.21.15.123:0/2019761571 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4f38070300 0x7f4f38115c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:18.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.634+0000 7f4f3d000700 1 --2- 172.21.15.123:0/2019761571 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f38070c10 0x7f4f381161d0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:18.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.634+0000 7f4f3d000700 1 -- 172.21.15.123:0/2019761571 >> 172.21.15.123:0/2019761571 conn(0x7f4f380f8970 msgr2=0x7f4f380f9ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:08:18.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.634+0000 7f4f3d000700 1 -- 172.21.15.123:0/2019761571 shutdown_connections 2024-03-30T05:08:18.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.634+0000 7f4f3d000700 1 -- 172.21.15.123:0/2019761571 wait complete. 2024-03-30T05:08:18.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.767+0000 7effb0400700 1 -- 172.21.15.123:0/3382474904 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7effa81037a0 msgr2=0x7effa80fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:08:18.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.767+0000 7effb0400700 1 --2- 172.21.15.123:0/3382474904 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7effa81037a0 0x7effa80fce10 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7eff94004940 tx=0x7eff94009e30 comp rx=0 tx=0).stop 2024-03-30T05:08:18.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.767+0000 7effb0400700 1 -- 172.21.15.123:0/3382474904 shutdown_connections 2024-03-30T05:08:18.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.767+0000 7effb0400700 1 --2- 172.21.15.123:0/3382474904 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7effa81037a0 0x7effa80fce10 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:18.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.767+0000 7effb0400700 1 --2- 172.21.15.123:0/3382474904 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7effa8102e00 0x7effa81031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:18.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.767+0000 7effb0400700 1 -- 172.21.15.123:0/3382474904 >> 172.21.15.123:0/3382474904 conn(0x7effa80f89b0 msgr2=0x7effa80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:08:18.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.767+0000 7effb0400700 1 -- 172.21.15.123:0/3382474904 shutdown_connections 2024-03-30T05:08:18.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.767+0000 7effb0400700 1 -- 172.21.15.123:0/3382474904 wait complete. 2024-03-30T05:08:18.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.768+0000 7effb0400700 1 Processor -- start 2024-03-30T05:08:18.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.768+0000 7effb0400700 1 -- start start 2024-03-30T05:08:18.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.768+0000 7effb0400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7effa8102e00 0x7effa8115da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:08:18.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.768+0000 7effb0400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7effa81037a0 0x7effa81162e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:08:18.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.768+0000 7effb0400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa8116930 con 0x7effa81037a0 2024-03-30T05:08:18.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.768+0000 7effb0400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa810fd00 con 0x7effa8102e00 2024-03-30T05:08:18.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.769+0000 7effad000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7effa81037a0 0x7effa81162e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:08:18.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.769+0000 7effad000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7effa81037a0 0x7effa81162e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37092/0 (socket says 172.21.15.123:37092) 2024-03-30T05:08:18.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.769+0000 7effad000700 1 -- 172.21.15.123:0/2308672860 learned_addr learned my addr 172.21.15.123:0/2308672860 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:08:18.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.769+0000 7effada00700 1 --2- 172.21.15.123:0/2308672860 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7effa8102e00 0x7effa8115da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:08:18.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.769+0000 7effad000700 1 -- 172.21.15.123:0/2308672860 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7effa8102e00 msgr2=0x7effa8115da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:08:18.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.769+0000 7effad000700 1 --2- 172.21.15.123:0/2308672860 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7effa8102e00 0x7effa8115da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:18.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.769+0000 7effad000700 1 -- 172.21.15.123:0/2308672860 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff94005040 con 0x7effa81037a0 2024-03-30T05:08:18.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.769+0000 7effada00700 1 --2- 172.21.15.123:0/2308672860 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7effa8102e00 0x7effa8115da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T05:08:18.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.769+0000 7effad000700 1 --2- 172.21.15.123:0/2308672860 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7effa81037a0 0x7effa81162e0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7eff94009390 tx=0x7eff94009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:08:18.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.770+0000 7effa5600700 1 -- 172.21.15.123:0/2308672860 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff9400e040 con 0x7effa81037a0 2024-03-30T05:08:18.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.770+0000 7effa5600700 1 -- 172.21.15.123:0/2308672860 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7eff94013420 con 0x7effa81037a0 2024-03-30T05:08:18.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.770+0000 7effa5600700 1 -- 172.21.15.123:0/2308672860 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff9401c630 con 0x7effa81037a0 2024-03-30T05:08:18.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.770+0000 7effb0400700 1 -- 172.21.15.123:0/2308672860 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effa810ff80 con 0x7effa81037a0 2024-03-30T05:08:18.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.770+0000 7effb0400700 1 -- 172.21.15.123:0/2308672860 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effa81104c0 con 0x7effa81037a0 2024-03-30T05:08:18.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.771+0000 7effb0400700 1 -- 172.21.15.123:0/2308672860 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effa80fe550 con 0x7effa81037a0 2024-03-30T05:08:18.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.772+0000 7effa5600700 1 -- 172.21.15.123:0/2308672860 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7eff94013a00 con 0x7effa81037a0 2024-03-30T05:08:18.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.773+0000 7effa5600700 1 --2- 172.21.15.123:0/2308672860 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7eff9006cc30 0x7eff9006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:08:18.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.773+0000 7effa5600700 1 -- 172.21.15.123:0/2308672860 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7eff9408d2c0 con 0x7effa81037a0 2024-03-30T05:08:18.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.773+0000 7effada00700 1 --2- 172.21.15.123:0/2308672860 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7eff9006cc30 0x7eff9006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:08:18.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.774+0000 7effada00700 1 --2- 172.21.15.123:0/2308672860 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7eff9006cc30 0x7eff9006f0e0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7eff9c009fd0 tx=0x7eff9c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:08:18.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:18.777+0000 7effa5600700 1 -- 172.21.15.123:0/2308672860 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7eff9405ad90 con 0x7effa81037a0 2024-03-30T05:08:19.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:19.060+0000 7effb0400700 1 -- 172.21.15.123:0/2308672860 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7effa805f5f0 con 0x7effa81037a0 2024-03-30T05:08:19.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:19.060+0000 7effa5600700 1 -- 172.21.15.123:0/2308672860 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7eff9405a920 con 0x7effa81037a0 2024-03-30T05:08:19.058 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-30T05:08:19.058 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-30T05:08:19.058 INFO:teuthology.orchestra.run.smithi123.stdout: host smithi135 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi135/config 2024-03-30T05:08:19.059 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-30T05:08:19.059 INFO:teuthology.orchestra.run.smithi123.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T05:08:19.059 INFO:teuthology.orchestra.run.smithi123.stdout:Traceback (most recent call last): 2024-03-30T05:08:19.059 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-30T05:08:19.059 INFO:teuthology.orchestra.run.smithi123.stdout: "__main__", mod_spec) 2024-03-30T05:08:19.059 INFO:teuthology.orchestra.run.smithi123.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-30T05:08:19.059 INFO:teuthology.orchestra.run.smithi123.stdout: exec(code, run_globals) 2024-03-30T05:08:19.059 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10889, in 2024-03-30T05:08:19.059 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 10877, in main 2024-03-30T05:08:19.059 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2576, in _infer_config 2024-03-30T05:08:19.059 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-03-30T05:08:19.059 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2604, in _infer_image 2024-03-30T05:08:19.059 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-03-30T05:08:19.059 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-03-30T05:08:19.060 INFO:teuthology.orchestra.run.smithi123.stdout: File "/tmp/tmp835nu59t.cephadm.build/__main__.py", line 2267, in call_throws 2024-03-30T05:08:19.060 INFO:teuthology.orchestra.run.smithi123.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 -e NODE_NAME=smithi135 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcbq3wtqr:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35 inventory --format=json-pretty --filter-for-batch 2024-03-30T05:08:19.060 INFO:teuthology.orchestra.run.smithi123.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-30T05:08:19.060 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is degraded 2024-03-30T05:08:19.060 INFO:teuthology.orchestra.run.smithi123.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-30T05:08:19.060 INFO:teuthology.orchestra.run.smithi123.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-30T05:08:19.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:19.062+0000 7effb0400700 1 -- 172.21.15.123:0/2308672860 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7eff9006cc30 msgr2=0x7eff9006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:08:19.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:19.062+0000 7effb0400700 1 --2- 172.21.15.123:0/2308672860 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7eff9006cc30 0x7eff9006f0e0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7eff9c009fd0 tx=0x7eff9c00b040 comp rx=0 tx=0).stop 2024-03-30T05:08:19.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:19.062+0000 7effb0400700 1 -- 172.21.15.123:0/2308672860 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7effa81037a0 msgr2=0x7effa81162e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:08:19.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:19.062+0000 7effb0400700 1 --2- 172.21.15.123:0/2308672860 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7effa81037a0 0x7effa81162e0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7eff94009390 tx=0x7eff94009e30 comp rx=0 tx=0).stop 2024-03-30T05:08:19.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:19.063+0000 7effb0400700 1 -- 172.21.15.123:0/2308672860 shutdown_connections 2024-03-30T05:08:19.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:19.063+0000 7effb0400700 1 --2- 172.21.15.123:0/2308672860 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7eff9006cc30 0x7eff9006f0e0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:19.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:19.063+0000 7effb0400700 1 --2- 172.21.15.123:0/2308672860 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7effa8102e00 0x7effa8115da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:19.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:19.063+0000 7effb0400700 1 --2- 172.21.15.123:0/2308672860 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7effa81037a0 0x7effa81162e0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:19.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:19.063+0000 7effb0400700 1 -- 172.21.15.123:0/2308672860 >> 172.21.15.123:0/2308672860 conn(0x7effa80f89b0 msgr2=0x7effa80f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:08:19.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:19.063+0000 7effb0400700 1 -- 172.21.15.123:0/2308672860 shutdown_connections 2024-03-30T05:08:19.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:19.063+0000 7effb0400700 1 -- 172.21.15.123:0/2308672860 wait complete. 2024-03-30T05:08:19.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:19 smithi123 ceph-mon[139369]: from='client.36008 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:08:19.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:19 smithi123 ceph-mon[139369]: pgmap v1114: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:08:19.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:19 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/251245532' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T05:08:19.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:19 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2308672860' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T05:08:19.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:19 smithi135 ceph-mon[106011]: from='client.36008 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:08:19.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:19 smithi135 ceph-mon[106011]: pgmap v1114: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:08:19.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:19 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/251245532' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T05:08:19.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:19 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2308672860' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T05:08:20.469 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:20 smithi123 ceph-mon[139369]: from='client.36020 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:08:20.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:20 smithi135 ceph-mon[106011]: from='client.36020 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:08:21.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:21 smithi123 ceph-mon[139369]: pgmap v1115: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:08:21.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:21 smithi135 ceph-mon[106011]: pgmap v1115: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:08:23.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:23 smithi123 ceph-mon[139369]: pgmap v1116: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:08:23.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:23 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T05:08:23.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:23 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T05:08:23.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:23 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:23.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:23 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T05:08:23.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:23 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:08:23.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:23 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:08:23.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:23 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:08:23.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:23 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:08:23.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:23 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:08:23.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:23 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:23.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:23 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi123.dvtind"}]: dispatch 2024-03-30T05:08:23.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:23 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi123.dvtind"}]': finished 2024-03-30T05:08:23.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:23 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi123.tqxvzr"}]: dispatch 2024-03-30T05:08:23.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:23 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi123.tqxvzr"}]': finished 2024-03-30T05:08:23.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:23 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi135.xwgjvb"}]: dispatch 2024-03-30T05:08:23.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:23 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi135.xwgjvb"}]': finished 2024-03-30T05:08:23.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:23 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi135.zvnlap"}]: dispatch 2024-03-30T05:08:23.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:23 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi135.zvnlap"}]': finished 2024-03-30T05:08:23.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:23 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]: dispatch 2024-03-30T05:08:23.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:23 smithi135 ceph-mon[106011]: pgmap v1116: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:08:23.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:23 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T05:08:23.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:23 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T05:08:23.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:23 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:23.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:23 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T05:08:23.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:23 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:08:23.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:23 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:08:23.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:23 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:08:23.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:23 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:08:23.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:23 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:08:23.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:23 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:23.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:23 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi123.dvtind"}]: dispatch 2024-03-30T05:08:23.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:23 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi123.dvtind"}]': finished 2024-03-30T05:08:23.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:23 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi123.tqxvzr"}]: dispatch 2024-03-30T05:08:23.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:23 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi123.tqxvzr"}]': finished 2024-03-30T05:08:23.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:23 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi135.xwgjvb"}]: dispatch 2024-03-30T05:08:23.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:23 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi135.xwgjvb"}]': finished 2024-03-30T05:08:23.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:23 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi135.zvnlap"}]: dispatch 2024-03-30T05:08:23.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:23 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi135.zvnlap"}]': finished 2024-03-30T05:08:23.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:23 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]: dispatch 2024-03-30T05:08:24.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:24 smithi123 ceph-mon[139369]: pgmap v1117: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:08:24.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:24 smithi123 ceph-mon[139369]: pgmap v1118: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:08:24.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:24 smithi123 ceph-mon[139369]: Upgrade: Setting container_image for all mds 2024-03-30T05:08:24.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:24 smithi123 ceph-mon[139369]: Upgrade: Setting filesystem cephfs Joinable 2024-03-30T05:08:24.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:24 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]': finished 2024-03-30T05:08:24.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:24 smithi123 ceph-mon[139369]: Standby daemon mds.cephfs.smithi123.tqxvzr assigned to filesystem cephfs as rank 0 2024-03-30T05:08:24.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:24 smithi123 ceph-mon[139369]: Standby daemon mds.cephfs.smithi123.dvtind assigned to filesystem cephfs as rank 1 2024-03-30T05:08:24.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:24 smithi123 ceph-mon[139369]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-30T05:08:24.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:24 smithi123 ceph-mon[139369]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-03-30T05:08:24.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:24 smithi123 ceph-mon[139369]: fsmap cephfs:2/2 {0=cephfs.smithi123.tqxvzr=up:replay,1=cephfs.smithi123.dvtind=up:replay} 2 up:standby 2024-03-30T05:08:24.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:24 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "1"}]: dispatch 2024-03-30T05:08:24.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:24 smithi135 ceph-mon[106011]: pgmap v1117: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:08:24.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:24 smithi135 ceph-mon[106011]: pgmap v1118: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail 2024-03-30T05:08:24.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:24 smithi135 ceph-mon[106011]: Upgrade: Setting container_image for all mds 2024-03-30T05:08:24.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:24 smithi135 ceph-mon[106011]: Upgrade: Setting filesystem cephfs Joinable 2024-03-30T05:08:24.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:24 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]': finished 2024-03-30T05:08:24.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:24 smithi135 ceph-mon[106011]: Standby daemon mds.cephfs.smithi123.tqxvzr assigned to filesystem cephfs as rank 0 2024-03-30T05:08:24.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:24 smithi135 ceph-mon[106011]: Standby daemon mds.cephfs.smithi123.dvtind assigned to filesystem cephfs as rank 1 2024-03-30T05:08:24.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:24 smithi135 ceph-mon[106011]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-30T05:08:24.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:24 smithi135 ceph-mon[106011]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-03-30T05:08:24.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:24 smithi135 ceph-mon[106011]: fsmap cephfs:2/2 {0=cephfs.smithi123.tqxvzr=up:replay,1=cephfs.smithi123.dvtind=up:replay} 2 up:standby 2024-03-30T05:08:24.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:24 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "1"}]: dispatch 2024-03-30T05:08:25.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:25 smithi123 ceph-mon[139369]: Upgrade: Enabling allow_standby_replay on filesystem cephfs 2024-03-30T05:08:25.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:25 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "1"}]': finished 2024-03-30T05:08:25.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:25 smithi123 ceph-mon[139369]: fsmap cephfs:2/2 {0=cephfs.smithi123.tqxvzr=up:replay,1=cephfs.smithi123.dvtind=up:replay} 2 up:standby 2024-03-30T05:08:25.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:25 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:25.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:25 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:08:25.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:25 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:25.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:25 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:08:25.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:25 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:25.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:25 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:08:25.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:25 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:08:25.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:25 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:25.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:25 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:08:25.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:25 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:25.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:25 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:08:25.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:25 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:25.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:25 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:08:25.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:25 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:25.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:25 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:08:25.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:25 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:08:25.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:25 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:08:25.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:25 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:08:25.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:25 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:08:25.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:25 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:08:25.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:25 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-30T05:08:25.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:25 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-30T05:08:25.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:25 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-30T05:08:25.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:25 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-30T05:08:25.469 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:25 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-30T05:08:25.469 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:25 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-30T05:08:25.469 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:25 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-30T05:08:25.469 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:25 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-03-30T05:08:25.469 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:25 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-30T05:08:25.469 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:25 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-30T05:08:25.469 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:25 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-30T05:08:25.469 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:25 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-03-30T05:08:25.469 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:25 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-30T05:08:25.469 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:25 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-30T05:08:25.469 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:25 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-30T05:08:25.469 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:25 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-30T05:08:25.469 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:25 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-30T05:08:25.469 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:25 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-30T05:08:25.469 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:25 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-03-30T05:08:25.470 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:25 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-30T05:08:25.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:25 smithi135 ceph-mon[106011]: Upgrade: Enabling allow_standby_replay on filesystem cephfs 2024-03-30T05:08:25.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:25 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "1"}]': finished 2024-03-30T05:08:25.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:25 smithi135 ceph-mon[106011]: fsmap cephfs:2/2 {0=cephfs.smithi123.tqxvzr=up:replay,1=cephfs.smithi123.dvtind=up:replay} 2 up:standby 2024-03-30T05:08:25.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:25 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:25.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:25 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:08:25.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:25 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:25.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:25 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:08:25.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:25 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:25.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:25 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:08:25.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:25 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:08:25.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:25 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:25.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:25 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:08:25.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:25 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:25.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:25 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:08:25.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:25 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:25.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:25 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:08:25.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:25 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:25.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:25 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:08:25.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:25 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:08:25.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:25 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:08:25.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:25 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:08:25.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:25 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:08:25.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:25 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:08:25.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:25 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-30T05:08:25.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:25 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-30T05:08:25.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:25 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-30T05:08:25.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:25 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-30T05:08:25.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:25 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-30T05:08:25.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:25 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-30T05:08:25.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:25 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-30T05:08:25.603 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:25 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-03-30T05:08:25.603 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:25 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-30T05:08:25.603 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:25 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-30T05:08:25.603 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:25 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-30T05:08:25.603 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:25 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-03-30T05:08:25.603 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:25 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-30T05:08:25.603 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:25 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-30T05:08:25.603 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:25 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-30T05:08:25.603 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:25 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-30T05:08:25.603 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:25 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-30T05:08:25.603 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:25 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-30T05:08:25.603 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:25 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-03-30T05:08:25.603 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:25 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-30T05:08:26.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:26 smithi123 ceph-mon[139369]: pgmap v1119: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 4.9 MiB/s rd, 1 op/s 2024-03-30T05:08:26.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:26 smithi123 ceph-mon[139369]: Upgrade: Setting container_image for all rgw 2024-03-30T05:08:26.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:26 smithi123 ceph-mon[139369]: Upgrade: Setting container_image for all rbd-mirror 2024-03-30T05:08:26.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:26 smithi123 ceph-mon[139369]: Upgrade: Setting container_image for all ceph-exporter 2024-03-30T05:08:26.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:26 smithi123 ceph-mon[139369]: Upgrade: Setting container_image for all iscsi 2024-03-30T05:08:26.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:26 smithi123 ceph-mon[139369]: Upgrade: Setting container_image for all nfs 2024-03-30T05:08:26.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:26 smithi123 ceph-mon[139369]: Upgrade: Setting container_image for all nvmeof 2024-03-30T05:08:26.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:26 smithi123 ceph-mon[139369]: Upgrade: Finalizing container_image settings 2024-03-30T05:08:26.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:26 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-30T05:08:26.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:26 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-30T05:08:26.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:26 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-30T05:08:26.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:26 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-30T05:08:26.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:26 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-30T05:08:26.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:26 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-30T05:08:26.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:26 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-30T05:08:26.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:26 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-30T05:08:26.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:26 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-30T05:08:26.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:26 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-03-30T05:08:26.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:26 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2024-03-30T05:08:26.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:26 smithi123 ceph-mon[139369]: Upgrade: Complete! 2024-03-30T05:08:26.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:26 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-30T05:08:26.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:26 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-30T05:08:26.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:26 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T05:08:26.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:26 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:26.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:26 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:26.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:26 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:26.468 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:26 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:26.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:26 smithi135 ceph-mon[106011]: pgmap v1119: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 4.9 MiB/s rd, 1 op/s 2024-03-30T05:08:26.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:26 smithi135 ceph-mon[106011]: Upgrade: Setting container_image for all rgw 2024-03-30T05:08:26.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:26 smithi135 ceph-mon[106011]: Upgrade: Setting container_image for all rbd-mirror 2024-03-30T05:08:26.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:26 smithi135 ceph-mon[106011]: Upgrade: Setting container_image for all ceph-exporter 2024-03-30T05:08:26.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:26 smithi135 ceph-mon[106011]: Upgrade: Setting container_image for all iscsi 2024-03-30T05:08:26.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:26 smithi135 ceph-mon[106011]: Upgrade: Setting container_image for all nfs 2024-03-30T05:08:26.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:26 smithi135 ceph-mon[106011]: Upgrade: Setting container_image for all nvmeof 2024-03-30T05:08:26.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:26 smithi135 ceph-mon[106011]: Upgrade: Finalizing container_image settings 2024-03-30T05:08:26.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:26 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-30T05:08:26.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:26 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-30T05:08:26.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:26 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-30T05:08:26.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:26 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-30T05:08:26.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:26 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-30T05:08:26.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:26 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-30T05:08:26.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:26 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-30T05:08:26.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:26 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-30T05:08:26.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:26 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-30T05:08:26.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:26 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-03-30T05:08:26.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:26 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2024-03-30T05:08:26.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:26 smithi135 ceph-mon[106011]: Upgrade: Complete! 2024-03-30T05:08:26.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:26 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-30T05:08:26.603 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:26 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-30T05:08:26.603 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:26 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T05:08:26.603 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:26 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:26.603 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:26 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:26.603 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:26 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:26.603 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:26 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:28.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:28 smithi123 ceph-mon[139369]: pgmap v1120: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 6.6 MiB/s rd, 2 op/s 2024-03-30T05:08:28.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:28 smithi135 ceph-mon[106011]: pgmap v1120: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 6.6 MiB/s rd, 2 op/s 2024-03-30T05:08:29.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:29 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:29.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:29 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:29.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:29 smithi123 ceph-mon[139369]: pgmap v1121: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 20 MiB/s rd, 7 op/s 2024-03-30T05:08:29.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:29 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:29.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:29 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:29.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:29 smithi135 ceph-mon[106011]: pgmap v1121: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 20 MiB/s rd, 7 op/s 2024-03-30T05:08:30.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:30 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:30.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:30 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:08:30.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:30 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:30.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:30.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:08:30.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:31.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:31 smithi135 ceph-mon[106011]: pgmap v1122: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 27 MiB/s rd, 9 op/s 2024-03-30T05:08:31.965 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:31 smithi123 ceph-mon[139369]: pgmap v1122: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 27 MiB/s rd, 9 op/s 2024-03-30T05:08:32.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:32 smithi135 ceph-mon[106011]: mds.? [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] up:resolve 2024-03-30T05:08:32.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:32 smithi135 ceph-mon[106011]: mds.? [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] up:resolve 2024-03-30T05:08:32.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:32 smithi135 ceph-mon[106011]: fsmap cephfs:2/2 {0=cephfs.smithi123.tqxvzr=up:resolve,1=cephfs.smithi123.dvtind=up:resolve} 2 up:standby 2024-03-30T05:08:32.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:32 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:32.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:32 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:32.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:32 smithi123 ceph-mon[139369]: mds.? [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] up:resolve 2024-03-30T05:08:32.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:32 smithi123 ceph-mon[139369]: mds.? [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] up:resolve 2024-03-30T05:08:32.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:32 smithi123 ceph-mon[139369]: fsmap cephfs:2/2 {0=cephfs.smithi123.tqxvzr=up:resolve,1=cephfs.smithi123.dvtind=up:resolve} 2 up:standby 2024-03-30T05:08:32.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:32 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:32.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:32 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:33.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:33 smithi135 ceph-mon[106011]: reconnect by client.14540 192.168.0.1:0/1247659015 after 0.00199995 2024-03-30T05:08:33.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:33 smithi135 ceph-mon[106011]: reconnect by client.14540 192.168.0.1:0/1247659015 after 0.00199995 2024-03-30T05:08:33.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:33 smithi135 ceph-mon[106011]: reconnect by client.24369 192.168.0.1:0/585251847 after 0.00199995 2024-03-30T05:08:33.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:33 smithi135 ceph-mon[106011]: reconnect by client.24369 192.168.0.1:0/585251847 after 0.00199995 2024-03-30T05:08:33.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:33 smithi135 ceph-mon[106011]: mds.? [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] up:reconnect 2024-03-30T05:08:33.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:33 smithi135 ceph-mon[106011]: mds.? [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] up:reconnect 2024-03-30T05:08:33.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:33 smithi135 ceph-mon[106011]: fsmap cephfs:2/2 {0=cephfs.smithi123.tqxvzr=up:reconnect,1=cephfs.smithi123.dvtind=up:reconnect} 2 up:standby 2024-03-30T05:08:33.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:33 smithi135 ceph-mon[106011]: pgmap v1123: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 25 MiB/s rd, 8 op/s 2024-03-30T05:08:33.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:33 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:33.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:33 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:33.950 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:33 smithi123 ceph-mon[139369]: reconnect by client.14540 192.168.0.1:0/1247659015 after 0.00199995 2024-03-30T05:08:33.950 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:33 smithi123 ceph-mon[139369]: reconnect by client.14540 192.168.0.1:0/1247659015 after 0.00199995 2024-03-30T05:08:33.950 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:33 smithi123 ceph-mon[139369]: reconnect by client.24369 192.168.0.1:0/585251847 after 0.00199995 2024-03-30T05:08:33.950 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:33 smithi123 ceph-mon[139369]: reconnect by client.24369 192.168.0.1:0/585251847 after 0.00199995 2024-03-30T05:08:33.950 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:33 smithi123 ceph-mon[139369]: mds.? [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] up:reconnect 2024-03-30T05:08:33.950 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:33 smithi123 ceph-mon[139369]: mds.? [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] up:reconnect 2024-03-30T05:08:33.950 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:33 smithi123 ceph-mon[139369]: fsmap cephfs:2/2 {0=cephfs.smithi123.tqxvzr=up:reconnect,1=cephfs.smithi123.dvtind=up:reconnect} 2 up:standby 2024-03-30T05:08:33.950 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:33 smithi123 ceph-mon[139369]: pgmap v1123: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 25 MiB/s rd, 8 op/s 2024-03-30T05:08:33.950 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:33 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:33.950 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:33 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:34.801 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:34 smithi123 ceph-mon[139369]: mds.? [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] up:rejoin 2024-03-30T05:08:34.801 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:34 smithi123 ceph-mon[139369]: mds.? [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] up:rejoin 2024-03-30T05:08:34.801 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:34 smithi123 ceph-mon[139369]: fsmap cephfs:2/2 {0=cephfs.smithi123.tqxvzr=up:rejoin,1=cephfs.smithi123.dvtind=up:rejoin} 2 up:standby 2024-03-30T05:08:34.801 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:34 smithi123 ceph-mon[139369]: daemon mds.cephfs.smithi123.tqxvzr is now active in filesystem cephfs as rank 0 2024-03-30T05:08:34.801 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:34 smithi123 ceph-mon[139369]: daemon mds.cephfs.smithi123.dvtind is now active in filesystem cephfs as rank 1 2024-03-30T05:08:34.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:34 smithi135 ceph-mon[106011]: mds.? [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] up:rejoin 2024-03-30T05:08:34.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:34 smithi135 ceph-mon[106011]: mds.? [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] up:rejoin 2024-03-30T05:08:34.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:34 smithi135 ceph-mon[106011]: fsmap cephfs:2/2 {0=cephfs.smithi123.tqxvzr=up:rejoin,1=cephfs.smithi123.dvtind=up:rejoin} 2 up:standby 2024-03-30T05:08:34.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:34 smithi135 ceph-mon[106011]: daemon mds.cephfs.smithi123.tqxvzr is now active in filesystem cephfs as rank 0 2024-03-30T05:08:34.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:34 smithi135 ceph-mon[106011]: daemon mds.cephfs.smithi123.dvtind is now active in filesystem cephfs as rank 1 2024-03-30T05:08:35.571 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:35 smithi135 ceph-mon[106011]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-30T05:08:35.571 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:35 smithi135 ceph-mon[106011]: mds.? [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] up:active 2024-03-30T05:08:35.571 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:35 smithi135 ceph-mon[106011]: mds.? [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] up:active 2024-03-30T05:08:35.571 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:35 smithi135 ceph-mon[106011]: fsmap cephfs:2 {0=cephfs.smithi123.tqxvzr=up:active,1=cephfs.smithi123.dvtind=up:active} 2 up:standby 2024-03-30T05:08:35.571 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:35 smithi135 ceph-mon[106011]: fsmap cephfs:2 {0=cephfs.smithi123.tqxvzr=up:active,1=cephfs.smithi123.dvtind=up:active} 2 up:standby-replay 2024-03-30T05:08:35.571 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:35 smithi135 ceph-mon[106011]: pgmap v1124: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 21 MiB/s rd, 341 B/s wr, 7 op/s 2024-03-30T05:08:35.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:35 smithi123 ceph-mon[139369]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-30T05:08:35.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:35 smithi123 ceph-mon[139369]: mds.? [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] up:active 2024-03-30T05:08:35.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:35 smithi123 ceph-mon[139369]: mds.? [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] up:active 2024-03-30T05:08:35.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:35 smithi123 ceph-mon[139369]: fsmap cephfs:2 {0=cephfs.smithi123.tqxvzr=up:active,1=cephfs.smithi123.dvtind=up:active} 2 up:standby 2024-03-30T05:08:35.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:35 smithi123 ceph-mon[139369]: fsmap cephfs:2 {0=cephfs.smithi123.tqxvzr=up:active,1=cephfs.smithi123.dvtind=up:active} 2 up:standby-replay 2024-03-30T05:08:35.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:35 smithi123 ceph-mon[139369]: pgmap v1124: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 21 MiB/s rd, 341 B/s wr, 7 op/s 2024-03-30T05:08:37.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:37 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:37.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:37 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:37.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:37 smithi123 ceph-mon[139369]: pgmap v1125: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 19 MiB/s rd, 426 B/s wr, 7 op/s 2024-03-30T05:08:37.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:37 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:37.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:37 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:37.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:37 smithi135 ceph-mon[106011]: pgmap v1125: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 19 MiB/s rd, 426 B/s wr, 7 op/s 2024-03-30T05:08:40.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:39 smithi135 ceph-mon[106011]: pgmap v1126: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 28 MiB/s rd, 597 B/s wr, 13 op/s 2024-03-30T05:08:40.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:39 smithi123 ceph-mon[139369]: pgmap v1126: 65 pgs: 65 active+clean; 259 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 28 MiB/s rd, 597 B/s wr, 13 op/s 2024-03-30T05:08:42.129 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:41 smithi123 ceph-mon[139369]: pgmap v1127: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 27 MiB/s rd, 4.6 KiB/s wr, 16 op/s 2024-03-30T05:08:42.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:41 smithi135 ceph-mon[106011]: pgmap v1127: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 27 MiB/s rd, 4.6 KiB/s wr, 16 op/s 2024-03-30T05:08:43.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:43 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:43.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:43 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:43.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:43 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T05:08:43.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:43 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T05:08:43.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:43 smithi135 ceph-mon[106011]: pgmap v1128: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 23 MiB/s rd, 4.7 KiB/s wr, 15 op/s 2024-03-30T05:08:43.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:43 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:43.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:43 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T05:08:43.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:43 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T05:08:43.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:43 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T05:08:43.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:43 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:43.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:43 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:43.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:43 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:43.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:43 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T05:08:43.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:43 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T05:08:43.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:43 smithi123 ceph-mon[139369]: pgmap v1128: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 23 MiB/s rd, 4.7 KiB/s wr, 15 op/s 2024-03-30T05:08:43.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:43 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:43.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:43 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T05:08:43.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:43 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T05:08:43.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:43 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T05:08:43.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:43 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:44.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:44 smithi135 ceph-mon[106011]: Health check cleared: CEPHADM_REFRESH_FAILED (was: failed to probe daemons or devices) 2024-03-30T05:08:44.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:44 smithi135 ceph-mon[106011]: Cluster is now healthy 2024-03-30T05:08:44.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:44 smithi123 ceph-mon[139369]: Health check cleared: CEPHADM_REFRESH_FAILED (was: failed to probe daemons or devices) 2024-03-30T05:08:44.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:44 smithi123 ceph-mon[139369]: Cluster is now healthy 2024-03-30T05:08:45.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:45 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:45.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:45 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:08:45.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:45 smithi135 ceph-mon[106011]: pgmap v1129: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 22 MiB/s rd, 8.9 KiB/s wr, 18 op/s 2024-03-30T05:08:45.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:45 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:08:45.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:45 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:08:45.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:45 smithi123 ceph-mon[139369]: pgmap v1129: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 22 MiB/s rd, 8.9 KiB/s wr, 18 op/s 2024-03-30T05:08:47.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:47 smithi135 ceph-mon[106011]: pgmap v1130: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 22 MiB/s rd, 8.5 KiB/s wr, 18 op/s 2024-03-30T05:08:47.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:47 smithi123 ceph-mon[139369]: pgmap v1130: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 22 MiB/s rd, 8.5 KiB/s wr, 18 op/s 2024-03-30T05:08:49.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:49.212+0000 7f9322a00700 1 -- 172.21.15.123:0/983766287 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f931c102e00 msgr2=0x7f931c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:08:49.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:49.212+0000 7f9322a00700 1 --2- 172.21.15.123:0/983766287 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f931c102e00 0x7f931c1031d0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f930c004940 tx=0x7f930c009e30 comp rx=0 tx=0).stop 2024-03-30T05:08:49.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:49.212+0000 7f9322a00700 1 -- 172.21.15.123:0/983766287 shutdown_connections 2024-03-30T05:08:49.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:49.212+0000 7f9322a00700 1 --2- 172.21.15.123:0/983766287 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f931c1037a0 0x7f931c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:49.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:49.212+0000 7f9322a00700 1 --2- 172.21.15.123:0/983766287 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f931c102e00 0x7f931c1031d0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:49.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:49.212+0000 7f9322a00700 1 -- 172.21.15.123:0/983766287 >> 172.21.15.123:0/983766287 conn(0x7f931c0f89b0 msgr2=0x7f931c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:08:49.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:49.212+0000 7f9322a00700 1 -- 172.21.15.123:0/983766287 shutdown_connections 2024-03-30T05:08:49.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:49.212+0000 7f9322a00700 1 -- 172.21.15.123:0/983766287 wait complete. 2024-03-30T05:08:49.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:49.213+0000 7f9322a00700 1 Processor -- start 2024-03-30T05:08:49.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:49.213+0000 7f9322a00700 1 -- start start 2024-03-30T05:08:49.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:49.214+0000 7f9322a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f931c102e00 0x7f931c115d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:08:49.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:49.214+0000 7f9322a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f931c1037a0 0x7f931c116250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:08:49.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:49.214+0000 7f9322a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f931c1168a0 con 0x7f931c102e00 2024-03-30T05:08:49.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:49.214+0000 7f9322a00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f931c10fc70 con 0x7f931c1037a0 2024-03-30T05:08:49.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:49.214+0000 7f931be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f931c102e00 0x7f931c115d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:08:49.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:49.214+0000 7f931be00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f931c102e00 0x7f931c115d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45390/0 (socket says 172.21.15.123:45390) 2024-03-30T05:08:49.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:49.214+0000 7f931be00700 1 -- 172.21.15.123:0/4121379105 learned_addr learned my addr 172.21.15.123:0/4121379105 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:08:49.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:49.214+0000 7f931b400700 1 --2- 172.21.15.123:0/4121379105 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f931c1037a0 0x7f931c116250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:08:49.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:49.214+0000 7f931be00700 1 -- 172.21.15.123:0/4121379105 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f931c1037a0 msgr2=0x7f931c116250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:08:49.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:49.214+0000 7f931be00700 1 --2- 172.21.15.123:0/4121379105 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f931c1037a0 0x7f931c116250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:49.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:49.214+0000 7f931be00700 1 -- 172.21.15.123:0/4121379105 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f930c005040 con 0x7f931c102e00 2024-03-30T05:08:49.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:49.215+0000 7f931be00700 1 --2- 172.21.15.123:0/4121379105 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f931c102e00 0x7f931c115d10 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f930c009390 tx=0x7f930c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:08:49.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:49.215+0000 7f9313e00700 1 -- 172.21.15.123:0/4121379105 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f930c00e040 con 0x7f931c102e00 2024-03-30T05:08:49.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:49.215+0000 7f9313e00700 1 -- 172.21.15.123:0/4121379105 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f930c013420 con 0x7f931c102e00 2024-03-30T05:08:49.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:49.215+0000 7f9313e00700 1 -- 172.21.15.123:0/4121379105 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f930c01c630 con 0x7f931c102e00 2024-03-30T05:08:49.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:49.215+0000 7f9322a00700 1 -- 172.21.15.123:0/4121379105 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f931c10fef0 con 0x7f931c102e00 2024-03-30T05:08:49.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:49.215+0000 7f9322a00700 1 -- 172.21.15.123:0/4121379105 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f931c1104b0 con 0x7f931c102e00 2024-03-30T05:08:49.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:49.216+0000 7f9322a00700 1 -- 172.21.15.123:0/4121379105 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f931c0fe550 con 0x7f931c102e00 2024-03-30T05:08:49.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:49.217+0000 7f9313e00700 1 -- 172.21.15.123:0/4121379105 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f930c0139e0 con 0x7f931c102e00 2024-03-30T05:08:49.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:49.218+0000 7f9313e00700 1 --2- 172.21.15.123:0/4121379105 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f930006cc30 0x7f930006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:08:49.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:49.218+0000 7f9313e00700 1 -- 172.21.15.123:0/4121379105 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f930c08d240 con 0x7f931c102e00 2024-03-30T05:08:49.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:49.218+0000 7f931b400700 1 --2- 172.21.15.123:0/4121379105 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f930006cc30 0x7f930006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:08:49.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:49.219+0000 7f931b400700 1 --2- 172.21.15.123:0/4121379105 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f930006cc30 0x7f930006f0e0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f931c1115b0 tx=0x7f9308009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:08:49.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:49.222+0000 7f9313e00700 1 -- 172.21.15.123:0/4121379105 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f930c05adc0 con 0x7f931c102e00 2024-03-30T05:08:49.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:49.442+0000 7f9322a00700 1 -- 172.21.15.123:0/4121379105 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f931c04bed0 con 0x7f930006cc30 2024-03-30T05:08:49.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:49.443+0000 7f9313e00700 1 -- 172.21.15.123:0/4121379105 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7f931c04bed0 con 0x7f930006cc30 2024-03-30T05:08:49.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:49.446+0000 7f9322a00700 1 -- 172.21.15.123:0/4121379105 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f930006cc30 msgr2=0x7f930006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:08:49.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:49.446+0000 7f9322a00700 1 --2- 172.21.15.123:0/4121379105 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f930006cc30 0x7f930006f0e0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f931c1115b0 tx=0x7f9308009ee0 comp rx=0 tx=0).stop 2024-03-30T05:08:49.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:49.446+0000 7f9322a00700 1 -- 172.21.15.123:0/4121379105 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f931c102e00 msgr2=0x7f931c115d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:08:49.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:49.446+0000 7f9322a00700 1 --2- 172.21.15.123:0/4121379105 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f931c102e00 0x7f931c115d10 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f930c009390 tx=0x7f930c009e30 comp rx=0 tx=0).stop 2024-03-30T05:08:49.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:49.446+0000 7f9322a00700 1 -- 172.21.15.123:0/4121379105 shutdown_connections 2024-03-30T05:08:49.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:49.446+0000 7f9322a00700 1 --2- 172.21.15.123:0/4121379105 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f930006cc30 0x7f930006f0e0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:49.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:49.446+0000 7f9322a00700 1 --2- 172.21.15.123:0/4121379105 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f931c102e00 0x7f931c115d10 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:49.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:49.446+0000 7f9322a00700 1 --2- 172.21.15.123:0/4121379105 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f931c1037a0 0x7f931c116250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:49.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:49.446+0000 7f9322a00700 1 -- 172.21.15.123:0/4121379105 >> 172.21.15.123:0/4121379105 conn(0x7f931c0f89b0 msgr2=0x7f931c0704f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:08:49.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:49.446+0000 7f9322a00700 1 -- 172.21.15.123:0/4121379105 shutdown_connections 2024-03-30T05:08:49.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:49.446+0000 7f9322a00700 1 -- 172.21.15.123:0/4121379105 wait complete. 2024-03-30T05:08:49.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:49 smithi135 ceph-mon[106011]: pgmap v1131: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 21 MiB/s rd, 8.4 KiB/s wr, 16 op/s 2024-03-30T05:08:49.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:49 smithi123 ceph-mon[139369]: pgmap v1131: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 21 MiB/s rd, 8.4 KiB/s wr, 16 op/s 2024-03-30T05:08:50.377 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -e sha1=470100a4ccbcf7769052bf09a5413881ccf1b97f -- bash -c 'ceph orch ps' 2024-03-30T05:08:50.805 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T05:08:51.160 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:50 smithi123 ceph-mon[139369]: from='client.36028 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:08:51.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:50 smithi135 ceph-mon[106011]: from='client.36028 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:08:51.967 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:51 smithi123 ceph-mon[139369]: pgmap v1132: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 9.8 MiB/s rd, 8.3 KiB/s wr, 12 op/s 2024-03-30T05:08:52.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:51 smithi135 ceph-mon[106011]: pgmap v1132: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 9.8 MiB/s rd, 8.3 KiB/s wr, 12 op/s 2024-03-30T05:08:52.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:52.761+0000 7f201a200700 1 -- 172.21.15.123:0/739733611 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2014102e00 msgr2=0x7f20141031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:08:52.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:52.761+0000 7f201a200700 1 --2- 172.21.15.123:0/739733611 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2014102e00 0x7f20141031d0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f2004004940 tx=0x7f2004009e30 comp rx=0 tx=0).stop 2024-03-30T05:08:52.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:52.761+0000 7f201a200700 1 -- 172.21.15.123:0/739733611 shutdown_connections 2024-03-30T05:08:52.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:52.761+0000 7f201a200700 1 --2- 172.21.15.123:0/739733611 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f20141037a0 0x7f20140fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:52.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:52.761+0000 7f201a200700 1 --2- 172.21.15.123:0/739733611 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2014102e00 0x7f20141031d0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:52.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:52.761+0000 7f201a200700 1 -- 172.21.15.123:0/739733611 >> 172.21.15.123:0/739733611 conn(0x7f20140f89b0 msgr2=0x7f20140fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:08:52.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:52.761+0000 7f201a200700 1 -- 172.21.15.123:0/739733611 shutdown_connections 2024-03-30T05:08:52.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:52.761+0000 7f201a200700 1 -- 172.21.15.123:0/739733611 wait complete. 2024-03-30T05:08:52.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:52.762+0000 7f201a200700 1 Processor -- start 2024-03-30T05:08:52.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:52.762+0000 7f201a200700 1 -- start start 2024-03-30T05:08:52.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:52.762+0000 7f201a200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2014102e00 0x7f2014113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:08:52.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:52.762+0000 7f201a200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20141037a0 0x7f2014114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:08:52.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:52.762+0000 7f201a200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20141146b0 con 0x7f20141037a0 2024-03-30T05:08:52.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:52.762+0000 7f201a200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f201410daa0 con 0x7f2014102e00 2024-03-30T05:08:52.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:52.763+0000 7f2012a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20141037a0 0x7f2014114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:08:52.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:52.763+0000 7f2012a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20141037a0 0x7f2014114060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45412/0 (socket says 172.21.15.123:45412) 2024-03-30T05:08:52.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:52.763+0000 7f2012a00700 1 -- 172.21.15.123:0/3065370027 learned_addr learned my addr 172.21.15.123:0/3065370027 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:08:52.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:52.763+0000 7f2013400700 1 --2- 172.21.15.123:0/3065370027 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2014102e00 0x7f2014113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:08:52.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:52.763+0000 7f2012a00700 1 -- 172.21.15.123:0/3065370027 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2014102e00 msgr2=0x7f2014113b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:08:52.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:52.763+0000 7f2012a00700 1 --2- 172.21.15.123:0/3065370027 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2014102e00 0x7f2014113b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:52.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:52.763+0000 7f2012a00700 1 -- 172.21.15.123:0/3065370027 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2004005040 con 0x7f20141037a0 2024-03-30T05:08:52.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:52.763+0000 7f2012a00700 1 --2- 172.21.15.123:0/3065370027 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20141037a0 0x7f2014114060 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f200000c800 tx=0x7f2000004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:08:52.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:52.764+0000 7f200b400700 1 -- 172.21.15.123:0/3065370027 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f20000106f0 con 0x7f20141037a0 2024-03-30T05:08:52.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:52.764+0000 7f200b400700 1 -- 172.21.15.123:0/3065370027 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2000010d00 con 0x7f20141037a0 2024-03-30T05:08:52.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:52.764+0000 7f201a200700 1 -- 172.21.15.123:0/3065370027 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f201410dd80 con 0x7f20141037a0 2024-03-30T05:08:52.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:52.764+0000 7f200b400700 1 -- 172.21.15.123:0/3065370027 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f20000113a0 con 0x7f20141037a0 2024-03-30T05:08:52.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:52.764+0000 7f201a200700 1 -- 172.21.15.123:0/3065370027 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f201410e340 con 0x7f20141037a0 2024-03-30T05:08:52.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:52.765+0000 7f201a200700 1 -- 172.21.15.123:0/3065370027 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20140fe550 con 0x7f20141037a0 2024-03-30T05:08:52.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:52.766+0000 7f200b400700 1 -- 172.21.15.123:0/3065370027 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f2000010850 con 0x7f20141037a0 2024-03-30T05:08:52.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:52.767+0000 7f200b400700 1 --2- 172.21.15.123:0/3065370027 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1ff806cc30 0x7f1ff806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:08:52.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:52.767+0000 7f200b400700 1 -- 172.21.15.123:0/3065370027 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f200008bff0 con 0x7f20141037a0 2024-03-30T05:08:52.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:52.768+0000 7f2013400700 1 --2- 172.21.15.123:0/3065370027 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1ff806cc30 0x7f1ff806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:08:52.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:52.768+0000 7f2013400700 1 --2- 172.21.15.123:0/3065370027 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1ff806cc30 0x7f1ff806f0e0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f2004009fd0 tx=0x7f2004009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:08:52.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:52.772+0000 7f200b400700 1 -- 172.21.15.123:0/3065370027 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2000059ac0 con 0x7f20141037a0 2024-03-30T05:08:52.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:52.985+0000 7f201a200700 1 -- 172.21.15.123:0/3065370027 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2014109dc0 con 0x7f1ff806cc30 2024-03-30T05:08:52.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:52.995+0000 7f200b400700 1 -- 172.21.15.123:0/3065370027 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f2014109dc0 con 0x7f1ff806cc30 2024-03-30T05:08:52.993 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T05:08:52.993 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (38m) 20s ago 50m 14.4M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T05:08:52.993 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (35m) 20s ago 50m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T05:08:52.993 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (35m) 24s ago 49m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T05:08:52.993 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (37m) 20s ago 50m 76.9M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T05:08:52.993 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (31m) 20s ago 44m 120M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T05:08:52.994 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (31m) 20s ago 44m 110M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T05:08:52.994 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (15m) 24s ago 44m 16.3M - 18.2.2-1018-g470100a4 e7c444a08cf7 87b073a6c9ca 2024-03-30T05:08:52.994 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (30m) 24s ago 44m 21.5M - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T05:08:52.994 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (40m) 20s ago 52m 514M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T05:08:52.994 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (39m) 24s ago 49m 457M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T05:08:52.994 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (36m) 20s ago 52m 84.9M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T05:08:52.994 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (36m) 24s ago 49m 57.9M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T05:08:52.994 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (38m) 20s ago 50m 8368k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T05:08:52.994 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (38m) 24s ago 49m 8669k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T05:08:52.994 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (34m) 20s ago 48m 111M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T05:08:52.994 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (34m) 20s ago 47m 104M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T05:08:52.994 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (33m) 20s ago 47m 94.2M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T05:08:52.994 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (33m) 24s ago 46m 107M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T05:08:52.994 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (32m) 24s ago 46m 106M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T05:08:52.995 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (31m) 24s ago 45m 94.8M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T05:08:52.995 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (38m) 20s ago 50m 66.2M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T05:08:52.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:52.997+0000 7f201a200700 1 -- 172.21.15.123:0/3065370027 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1ff806cc30 msgr2=0x7f1ff806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:08:52.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:52.997+0000 7f201a200700 1 --2- 172.21.15.123:0/3065370027 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1ff806cc30 0x7f1ff806f0e0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f2004009fd0 tx=0x7f2004009e30 comp rx=0 tx=0).stop 2024-03-30T05:08:52.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:52.997+0000 7f201a200700 1 -- 172.21.15.123:0/3065370027 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20141037a0 msgr2=0x7f2014114060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:08:52.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:52.997+0000 7f201a200700 1 --2- 172.21.15.123:0/3065370027 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20141037a0 0x7f2014114060 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f200000c800 tx=0x7f2000004470 comp rx=0 tx=0).stop 2024-03-30T05:08:52.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:52.997+0000 7f201a200700 1 -- 172.21.15.123:0/3065370027 shutdown_connections 2024-03-30T05:08:52.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:52.997+0000 7f201a200700 1 --2- 172.21.15.123:0/3065370027 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1ff806cc30 0x7f1ff806f0e0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:52.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:52.998+0000 7f201a200700 1 --2- 172.21.15.123:0/3065370027 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2014102e00 0x7f2014113b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:52.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:52.998+0000 7f201a200700 1 --2- 172.21.15.123:0/3065370027 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20141037a0 0x7f2014114060 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:52.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:52.998+0000 7f201a200700 1 -- 172.21.15.123:0/3065370027 >> 172.21.15.123:0/3065370027 conn(0x7f20140f89b0 msgr2=0x7f20140f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:08:52.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:52.998+0000 7f201a200700 1 -- 172.21.15.123:0/3065370027 shutdown_connections 2024-03-30T05:08:52.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:52.998+0000 7f201a200700 1 -- 172.21.15.123:0/3065370027 wait complete. 2024-03-30T05:08:53.800 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -e sha1=470100a4ccbcf7769052bf09a5413881ccf1b97f -- bash -c 'ceph orch upgrade status' 2024-03-30T05:08:54.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:53 smithi123 ceph-mon[139369]: pgmap v1133: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 707 KiB/s rd, 4.2 KiB/s wr, 5 op/s 2024-03-30T05:08:54.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:53 smithi123 ceph-mon[139369]: from='client.36032 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:08:54.228 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T05:08:54.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:53 smithi135 ceph-mon[106011]: pgmap v1133: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 707 KiB/s rd, 4.2 KiB/s wr, 5 op/s 2024-03-30T05:08:54.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:53 smithi135 ceph-mon[106011]: from='client.36032 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:08:56.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:55 smithi123 ceph-mon[139369]: pgmap v1134: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 8.6 KiB/s rd, 4.1 KiB/s wr, 6 op/s 2024-03-30T05:08:56.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:56.270+0000 7f8235000700 1 -- 172.21.15.123:0/1981256851 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8230102e00 msgr2=0x7f82301031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:08:56.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:56.270+0000 7f8235000700 1 --2- 172.21.15.123:0/1981256851 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8230102e00 0x7f82301031d0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f8220004940 tx=0x7f8220009e30 comp rx=0 tx=0).stop 2024-03-30T05:08:56.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:56.270+0000 7f8235000700 1 -- 172.21.15.123:0/1981256851 shutdown_connections 2024-03-30T05:08:56.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:56.270+0000 7f8235000700 1 --2- 172.21.15.123:0/1981256851 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f82301037a0 0x7f82300fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:56.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:56.270+0000 7f8235000700 1 --2- 172.21.15.123:0/1981256851 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8230102e00 0x7f82301031d0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:56.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:56.270+0000 7f8235000700 1 -- 172.21.15.123:0/1981256851 >> 172.21.15.123:0/1981256851 conn(0x7f82300f89b0 msgr2=0x7f82300fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:08:56.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:56.270+0000 7f8235000700 1 -- 172.21.15.123:0/1981256851 shutdown_connections 2024-03-30T05:08:56.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:56.270+0000 7f8235000700 1 -- 172.21.15.123:0/1981256851 wait complete. 2024-03-30T05:08:56.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:56.271+0000 7f8235000700 1 Processor -- start 2024-03-30T05:08:56.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:56.271+0000 7f8235000700 1 -- start start 2024-03-30T05:08:56.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:56.271+0000 7f8235000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8230102e00 0x7f82300702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:08:56.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:56.271+0000 7f8235000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f82301037a0 0x7f8230070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:08:56.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:56.271+0000 7f8235000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8230070d90 con 0x7f8230102e00 2024-03-30T05:08:56.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:56.271+0000 7f8235000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8230070ed0 con 0x7f82301037a0 2024-03-30T05:08:56.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:56.272+0000 7f822de00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8230102e00 0x7f82300702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:08:56.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:56.272+0000 7f822de00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8230102e00 0x7f82300702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40300/0 (socket says 172.21.15.123:40300) 2024-03-30T05:08:56.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:56.272+0000 7f822de00700 1 -- 172.21.15.123:0/3108198251 learned_addr learned my addr 172.21.15.123:0/3108198251 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:08:56.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:56.272+0000 7f822de00700 1 -- 172.21.15.123:0/3108198251 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f82301037a0 msgr2=0x7f8230070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:08:56.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:56.272+0000 7f822de00700 1 --2- 172.21.15.123:0/3108198251 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f82301037a0 0x7f8230070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:56.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:56.272+0000 7f822de00700 1 -- 172.21.15.123:0/3108198251 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8220005040 con 0x7f8230102e00 2024-03-30T05:08:56.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:56.272+0000 7f822de00700 1 --2- 172.21.15.123:0/3108198251 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8230102e00 0x7f82300702c0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f8220009390 tx=0x7f8220009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:08:56.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:56.272+0000 7f8226000700 1 -- 172.21.15.123:0/3108198251 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f822000e040 con 0x7f8230102e00 2024-03-30T05:08:56.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:56.273+0000 7f8226000700 1 -- 172.21.15.123:0/3108198251 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8220013420 con 0x7f8230102e00 2024-03-30T05:08:56.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:56.273+0000 7f8235000700 1 -- 172.21.15.123:0/3108198251 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8230071120 con 0x7f8230102e00 2024-03-30T05:08:56.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:56.273+0000 7f8226000700 1 -- 172.21.15.123:0/3108198251 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f822001c630 con 0x7f8230102e00 2024-03-30T05:08:56.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:56.273+0000 7f8235000700 1 -- 172.21.15.123:0/3108198251 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f823011a250 con 0x7f8230102e00 2024-03-30T05:08:56.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:56.274+0000 7f8235000700 1 -- 172.21.15.123:0/3108198251 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82300fe550 con 0x7f8230102e00 2024-03-30T05:08:56.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:56.275+0000 7f8226000700 1 -- 172.21.15.123:0/3108198251 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f82200139e0 con 0x7f8230102e00 2024-03-30T05:08:56.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:56.275+0000 7f8226000700 1 --2- 172.21.15.123:0/3108198251 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f821406cc30 0x7f821406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:08:56.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:56.275+0000 7f8226000700 1 -- 172.21.15.123:0/3108198251 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f822008d1a0 con 0x7f8230102e00 2024-03-30T05:08:56.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:56.276+0000 7f822d400700 1 --2- 172.21.15.123:0/3108198251 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f821406cc30 0x7f821406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:08:56.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:56.276+0000 7f822d400700 1 --2- 172.21.15.123:0/3108198251 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f821406cc30 0x7f821406f0e0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f8230071c00 tx=0x7f821c009ce0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:08:56.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:56.280+0000 7f8226000700 1 -- 172.21.15.123:0/3108198251 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f82200572a0 con 0x7f8230102e00 2024-03-30T05:08:56.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:55 smithi135 ceph-mon[106011]: pgmap v1134: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 8.6 KiB/s rd, 4.1 KiB/s wr, 6 op/s 2024-03-30T05:08:56.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:56.500+0000 7f8235000700 1 -- 172.21.15.123:0/3108198251 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f823004bed0 con 0x7f821406cc30 2024-03-30T05:08:56.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:56.501+0000 7f8226000700 1 -- 172.21.15.123:0/3108198251 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7f823004bed0 con 0x7f821406cc30 2024-03-30T05:08:56.500 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T05:08:56.500 INFO:teuthology.orchestra.run.smithi123.stdout: "target_image": null, 2024-03-30T05:08:56.500 INFO:teuthology.orchestra.run.smithi123.stdout: "in_progress": false, 2024-03-30T05:08:56.500 INFO:teuthology.orchestra.run.smithi123.stdout: "which": "", 2024-03-30T05:08:56.500 INFO:teuthology.orchestra.run.smithi123.stdout: "services_complete": [], 2024-03-30T05:08:56.500 INFO:teuthology.orchestra.run.smithi123.stdout: "progress": null, 2024-03-30T05:08:56.500 INFO:teuthology.orchestra.run.smithi123.stdout: "message": "", 2024-03-30T05:08:56.500 INFO:teuthology.orchestra.run.smithi123.stdout: "is_paused": false 2024-03-30T05:08:56.500 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T05:08:56.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:56.504+0000 7f8235000700 1 -- 172.21.15.123:0/3108198251 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f821406cc30 msgr2=0x7f821406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:08:56.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:56.504+0000 7f8235000700 1 --2- 172.21.15.123:0/3108198251 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f821406cc30 0x7f821406f0e0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f8230071c00 tx=0x7f821c009ce0 comp rx=0 tx=0).stop 2024-03-30T05:08:56.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:56.504+0000 7f8235000700 1 -- 172.21.15.123:0/3108198251 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8230102e00 msgr2=0x7f82300702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:08:56.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:56.504+0000 7f8235000700 1 --2- 172.21.15.123:0/3108198251 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8230102e00 0x7f82300702c0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f8220009390 tx=0x7f8220009e30 comp rx=0 tx=0).stop 2024-03-30T05:08:56.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:56.504+0000 7f8235000700 1 -- 172.21.15.123:0/3108198251 shutdown_connections 2024-03-30T05:08:56.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:56.504+0000 7f8235000700 1 --2- 172.21.15.123:0/3108198251 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f821406cc30 0x7f821406f0e0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:56.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:56.504+0000 7f8235000700 1 --2- 172.21.15.123:0/3108198251 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8230102e00 0x7f82300702c0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:56.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:56.504+0000 7f8235000700 1 --2- 172.21.15.123:0/3108198251 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f82301037a0 0x7f8230070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:56.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:56.504+0000 7f8235000700 1 -- 172.21.15.123:0/3108198251 >> 172.21.15.123:0/3108198251 conn(0x7f82300f89b0 msgr2=0x7f82300f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:08:56.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:56.504+0000 7f8235000700 1 -- 172.21.15.123:0/3108198251 shutdown_connections 2024-03-30T05:08:56.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:56.504+0000 7f8235000700 1 -- 172.21.15.123:0/3108198251 wait complete. 2024-03-30T05:08:57.247 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -e sha1=470100a4ccbcf7769052bf09a5413881ccf1b97f -- bash -c 'ceph health detail' 2024-03-30T05:08:57.671 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T05:08:58.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:57 smithi123 ceph-mon[139369]: pgmap v1135: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:08:58.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:57 smithi123 ceph-mon[139369]: from='client.36036 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:08:58.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:57 smithi135 ceph-mon[106011]: pgmap v1135: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:08:58.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:57 smithi135 ceph-mon[106011]: from='client.36036 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:08:59.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:59.596+0000 7f2ac0200700 1 -- 172.21.15.123:0/3313167821 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ab8102e00 msgr2=0x7f2ab81031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:08:59.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:59.596+0000 7f2ac0200700 1 --2- 172.21.15.123:0/3313167821 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ab8102e00 0x7f2ab81031d0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f2aac004940 tx=0x7f2aac009e30 comp rx=0 tx=0).stop 2024-03-30T05:08:59.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:59.597+0000 7f2ac0200700 1 -- 172.21.15.123:0/3313167821 shutdown_connections 2024-03-30T05:08:59.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:59.597+0000 7f2ac0200700 1 --2- 172.21.15.123:0/3313167821 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2ab81037a0 0x7f2ab80fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:59.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:59.597+0000 7f2ac0200700 1 --2- 172.21.15.123:0/3313167821 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ab8102e00 0x7f2ab81031d0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:59.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:59.597+0000 7f2ac0200700 1 -- 172.21.15.123:0/3313167821 >> 172.21.15.123:0/3313167821 conn(0x7f2ab80f89b0 msgr2=0x7f2ab80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:08:59.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:59.597+0000 7f2ac0200700 1 -- 172.21.15.123:0/3313167821 shutdown_connections 2024-03-30T05:08:59.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:59.597+0000 7f2ac0200700 1 -- 172.21.15.123:0/3313167821 wait complete. 2024-03-30T05:08:59.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:59.598+0000 7f2ac0200700 1 Processor -- start 2024-03-30T05:08:59.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:59.598+0000 7f2ac0200700 1 -- start start 2024-03-30T05:08:59.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:59.599+0000 7f2ac0200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2ab8102e00 0x7f2ab8115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:08:59.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:59.599+0000 7f2ac0200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ab81037a0 0x7f2ab8116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:08:59.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:59.599+0000 7f2ac0200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab8116860 con 0x7f2ab81037a0 2024-03-30T05:08:59.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:59.599+0000 7f2ac0200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab810fc30 con 0x7f2ab8102e00 2024-03-30T05:08:59.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:59.599+0000 7f2abce00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ab81037a0 0x7f2ab8116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:08:59.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:59.599+0000 7f2abce00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ab81037a0 0x7f2ab8116210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40316/0 (socket says 172.21.15.123:40316) 2024-03-30T05:08:59.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:59.599+0000 7f2abce00700 1 -- 172.21.15.123:0/566544742 learned_addr learned my addr 172.21.15.123:0/566544742 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:08:59.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:59.599+0000 7f2abd800700 1 --2- 172.21.15.123:0/566544742 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2ab8102e00 0x7f2ab8115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:08:59.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:59.599+0000 7f2abce00700 1 -- 172.21.15.123:0/566544742 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2ab8102e00 msgr2=0x7f2ab8115cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:08:59.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:59.599+0000 7f2abce00700 1 --2- 172.21.15.123:0/566544742 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2ab8102e00 0x7f2ab8115cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:59.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:59.599+0000 7f2abce00700 1 -- 172.21.15.123:0/566544742 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aac005040 con 0x7f2ab81037a0 2024-03-30T05:08:59.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:59.600+0000 7f2abce00700 1 --2- 172.21.15.123:0/566544742 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ab81037a0 0x7f2ab8116210 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f2aa4002a50 tx=0x7f2aa4004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:08:59.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:59.600+0000 7f2ab5600700 1 -- 172.21.15.123:0/566544742 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2aa4007cc0 con 0x7f2ab81037a0 2024-03-30T05:08:59.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:59.600+0000 7f2ac0200700 1 -- 172.21.15.123:0/566544742 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ab810ff10 con 0x7f2ab81037a0 2024-03-30T05:08:59.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:59.600+0000 7f2ab5600700 1 -- 172.21.15.123:0/566544742 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2aa400f040 con 0x7f2ab81037a0 2024-03-30T05:08:59.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:59.600+0000 7f2ab5600700 1 -- 172.21.15.123:0/566544742 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2aa400d920 con 0x7f2ab81037a0 2024-03-30T05:08:59.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:59.600+0000 7f2ac0200700 1 -- 172.21.15.123:0/566544742 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ab81104d0 con 0x7f2ab81037a0 2024-03-30T05:08:59.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:59.601+0000 7f2ac0200700 1 -- 172.21.15.123:0/566544742 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ab80fe550 con 0x7f2ab81037a0 2024-03-30T05:08:59.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:59.602+0000 7f2ab5600700 1 -- 172.21.15.123:0/566544742 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f2aa4009680 con 0x7f2ab81037a0 2024-03-30T05:08:59.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:59.603+0000 7f2ab5600700 1 --2- 172.21.15.123:0/566544742 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2aa006cc30 0x7f2aa006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:08:59.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:59.603+0000 7f2ab5600700 1 -- 172.21.15.123:0/566544742 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f2aa408bdb0 con 0x7f2ab81037a0 2024-03-30T05:08:59.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:59.603+0000 7f2abd800700 1 --2- 172.21.15.123:0/566544742 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2aa006cc30 0x7f2aa006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:08:59.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:59.604+0000 7f2abd800700 1 --2- 172.21.15.123:0/566544742 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2aa006cc30 0x7f2aa006f0e0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f2aac009fd0 tx=0x7f2aac009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:08:59.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:59.607+0000 7f2ab5600700 1 -- 172.21.15.123:0/566544742 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2aa4059780 con 0x7f2ab81037a0 2024-03-30T05:08:59.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:59.888+0000 7f2ac0200700 1 -- 172.21.15.123:0/566544742 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f2ab804bed0 con 0x7f2ab81037a0 2024-03-30T05:08:59.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:59.888+0000 7f2ab5600700 1 -- 172.21.15.123:0/566544742 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f2aa4059310 con 0x7f2ab81037a0 2024-03-30T05:08:59.886 INFO:teuthology.orchestra.run.smithi123.stdout:HEALTH_OK 2024-03-30T05:08:59.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:59.890+0000 7f2ac0200700 1 -- 172.21.15.123:0/566544742 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2aa006cc30 msgr2=0x7f2aa006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:08:59.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:59.890+0000 7f2ac0200700 1 --2- 172.21.15.123:0/566544742 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2aa006cc30 0x7f2aa006f0e0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f2aac009fd0 tx=0x7f2aac009e30 comp rx=0 tx=0).stop 2024-03-30T05:08:59.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:59.890+0000 7f2ac0200700 1 -- 172.21.15.123:0/566544742 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ab81037a0 msgr2=0x7f2ab8116210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:08:59.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:59.890+0000 7f2ac0200700 1 --2- 172.21.15.123:0/566544742 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ab81037a0 0x7f2ab8116210 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f2aa4002a50 tx=0x7f2aa4004600 comp rx=0 tx=0).stop 2024-03-30T05:08:59.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:59.891+0000 7f2ac0200700 1 -- 172.21.15.123:0/566544742 shutdown_connections 2024-03-30T05:08:59.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:59.891+0000 7f2ac0200700 1 --2- 172.21.15.123:0/566544742 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2aa006cc30 0x7f2aa006f0e0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:59.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:59.891+0000 7f2ac0200700 1 --2- 172.21.15.123:0/566544742 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2ab8102e00 0x7f2ab8115cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:59.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:59.891+0000 7f2ac0200700 1 --2- 172.21.15.123:0/566544742 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ab81037a0 0x7f2ab8116210 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:08:59.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:59.891+0000 7f2ac0200700 1 -- 172.21.15.123:0/566544742 >> 172.21.15.123:0/566544742 conn(0x7f2ab80f89b0 msgr2=0x7f2ab80f99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:08:59.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:59.891+0000 7f2ac0200700 1 -- 172.21.15.123:0/566544742 shutdown_connections 2024-03-30T05:08:59.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:08:59.891+0000 7f2ac0200700 1 -- 172.21.15.123:0/566544742 wait complete. 2024-03-30T05:09:00.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:59 smithi123 ceph-mon[139369]: pgmap v1136: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:09:00.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:59 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:09:00.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:08:59 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/566544742' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T05:09:00.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:59 smithi135 ceph-mon[106011]: pgmap v1136: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:09:00.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:59 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:09:00.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:08:59 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/566544742' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-30T05:09:00.633 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -e sha1=470100a4ccbcf7769052bf09a5413881ccf1b97f -- bash -c 'ceph versions' 2024-03-30T05:09:01.057 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T05:09:02.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:09:01 smithi123 ceph-mon[139369]: pgmap v1137: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:09:02.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:09:01 smithi135 ceph-mon[106011]: pgmap v1137: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:09:03.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:09:03.334+0000 7fc4c9e00700 1 -- 172.21.15.123:0/2120622433 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4c4102e00 msgr2=0x7fc4c41031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:09:03.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:09:03.334+0000 7fc4c9e00700 1 --2- 172.21.15.123:0/2120622433 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4c4102e00 0x7fc4c41031d0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fc4b4004940 tx=0x7fc4b4009e30 comp rx=0 tx=0).stop 2024-03-30T05:09:03.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:09:03.334+0000 7fc4c9e00700 1 -- 172.21.15.123:0/2120622433 shutdown_connections 2024-03-30T05:09:03.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:09:03.334+0000 7fc4c9e00700 1 --2- 172.21.15.123:0/2120622433 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc4c41037a0 0x7fc4c40fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:09:03.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:09:03.334+0000 7fc4c9e00700 1 --2- 172.21.15.123:0/2120622433 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4c4102e00 0x7fc4c41031d0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:09:03.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:09:03.334+0000 7fc4c9e00700 1 -- 172.21.15.123:0/2120622433 >> 172.21.15.123:0/2120622433 conn(0x7fc4c40f89b0 msgr2=0x7fc4c40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:09:03.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:09:03.335+0000 7fc4c9e00700 1 -- 172.21.15.123:0/2120622433 shutdown_connections 2024-03-30T05:09:03.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:09:03.335+0000 7fc4c9e00700 1 -- 172.21.15.123:0/2120622433 wait complete. 2024-03-30T05:09:03.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:09:03.335+0000 7fc4c9e00700 1 Processor -- start 2024-03-30T05:09:03.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:09:03.336+0000 7fc4c9e00700 1 -- start start 2024-03-30T05:09:03.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:09:03.336+0000 7fc4c9e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4c4102e00 0x7fc4c4115d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:09:03.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:09:03.336+0000 7fc4c9e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc4c41037a0 0x7fc4c4116250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:09:03.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:09:03.336+0000 7fc4c9e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c41168a0 con 0x7fc4c4102e00 2024-03-30T05:09:03.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:09:03.336+0000 7fc4c9e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c410fc70 con 0x7fc4c41037a0 2024-03-30T05:09:03.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:09:03.336+0000 7fc4c2a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4c4102e00 0x7fc4c4115d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:09:03.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:09:03.336+0000 7fc4c2a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4c4102e00 0x7fc4c4115d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40342/0 (socket says 172.21.15.123:40342) 2024-03-30T05:09:03.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:09:03.336+0000 7fc4c2a00700 1 -- 172.21.15.123:0/397127802 learned_addr learned my addr 172.21.15.123:0/397127802 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:09:03.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:09:03.336+0000 7fc4c2a00700 1 -- 172.21.15.123:0/397127802 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc4c41037a0 msgr2=0x7fc4c4116250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:09:03.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:09:03.336+0000 7fc4c2a00700 1 --2- 172.21.15.123:0/397127802 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc4c41037a0 0x7fc4c4116250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:09:03.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:09:03.337+0000 7fc4c2a00700 1 -- 172.21.15.123:0/397127802 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4b4005040 con 0x7fc4c4102e00 2024-03-30T05:09:03.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:09:03.337+0000 7fc4c2a00700 1 --2- 172.21.15.123:0/397127802 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4c4102e00 0x7fc4c4115d10 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fc4b4009360 tx=0x7fc4b4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:09:03.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:09:03.337+0000 7fc4baa00700 1 -- 172.21.15.123:0/397127802 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc4b400e040 con 0x7fc4c4102e00 2024-03-30T05:09:03.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:09:03.337+0000 7fc4baa00700 1 -- 172.21.15.123:0/397127802 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc4b4013420 con 0x7fc4c4102e00 2024-03-30T05:09:03.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:09:03.337+0000 7fc4c9e00700 1 -- 172.21.15.123:0/397127802 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4c410fef0 con 0x7fc4c4102e00 2024-03-30T05:09:03.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:09:03.337+0000 7fc4baa00700 1 -- 172.21.15.123:0/397127802 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc4b401c5f0 con 0x7fc4c4102e00 2024-03-30T05:09:03.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:09:03.337+0000 7fc4c9e00700 1 -- 172.21.15.123:0/397127802 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4c41104b0 con 0x7fc4c4102e00 2024-03-30T05:09:03.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:09:03.338+0000 7fc4c9e00700 1 -- 172.21.15.123:0/397127802 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4c40fe550 con 0x7fc4c4102e00 2024-03-30T05:09:03.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:09:03.339+0000 7fc4baa00700 1 -- 172.21.15.123:0/397127802 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fc4b4013590 con 0x7fc4c4102e00 2024-03-30T05:09:03.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:09:03.340+0000 7fc4baa00700 1 --2- 172.21.15.123:0/397127802 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc4a806cc30 0x7fc4a806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:09:03.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:09:03.340+0000 7fc4baa00700 1 -- 172.21.15.123:0/397127802 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc4b408d340 con 0x7fc4c4102e00 2024-03-30T05:09:03.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:09:03.340+0000 7fc4c2000700 1 --2- 172.21.15.123:0/397127802 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc4a806cc30 0x7fc4a806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:09:03.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:09:03.341+0000 7fc4c2000700 1 --2- 172.21.15.123:0/397127802 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc4a806cc30 0x7fc4a806f0e0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fc4c41115b0 tx=0x7fc4b0007cc0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:09:03.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:09:03.345+0000 7fc4baa00700 1 -- 172.21.15.123:0/397127802 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc4b405ad90 con 0x7fc4c4102e00 2024-03-30T05:09:03.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:09:03.632+0000 7fc4c9e00700 1 -- 172.21.15.123:0/397127802 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc4c4109dc0 con 0x7fc4c4102e00 2024-03-30T05:09:03.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:09:03.632+0000 7fc4baa00700 1 -- 172.21.15.123:0/397127802 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7fc4b405a920 con 0x7fc4c4102e00 2024-03-30T05:09:03.629 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T05:09:03.629 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T05:09:03.629 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T05:09:03.630 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:09:03.630 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T05:09:03.630 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T05:09:03.630 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:09:03.630 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T05:09:03.630 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T05:09:03.630 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:09:03.630 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T05:09:03.630 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 4 2024-03-30T05:09:03.630 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:09:03.630 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T05:09:03.630 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 14 2024-03-30T05:09:03.630 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T05:09:03.630 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T05:09:03.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:09:03.635+0000 7fc4c9e00700 1 -- 172.21.15.123:0/397127802 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc4a806cc30 msgr2=0x7fc4a806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:09:03.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:09:03.635+0000 7fc4c9e00700 1 --2- 172.21.15.123:0/397127802 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc4a806cc30 0x7fc4a806f0e0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fc4c41115b0 tx=0x7fc4b0007cc0 comp rx=0 tx=0).stop 2024-03-30T05:09:03.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:09:03.635+0000 7fc4c9e00700 1 -- 172.21.15.123:0/397127802 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4c4102e00 msgr2=0x7fc4c4115d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:09:03.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:09:03.635+0000 7fc4c9e00700 1 --2- 172.21.15.123:0/397127802 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4c4102e00 0x7fc4c4115d10 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fc4b4009360 tx=0x7fc4b4009e30 comp rx=0 tx=0).stop 2024-03-30T05:09:03.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:09:03.635+0000 7fc4c9e00700 1 -- 172.21.15.123:0/397127802 shutdown_connections 2024-03-30T05:09:03.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:09:03.635+0000 7fc4c9e00700 1 --2- 172.21.15.123:0/397127802 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc4a806cc30 0x7fc4a806f0e0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:09:03.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:09:03.635+0000 7fc4c9e00700 1 --2- 172.21.15.123:0/397127802 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4c4102e00 0x7fc4c4115d10 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:09:03.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:09:03.635+0000 7fc4c9e00700 1 --2- 172.21.15.123:0/397127802 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc4c41037a0 0x7fc4c4116250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:09:03.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:09:03.635+0000 7fc4c9e00700 1 -- 172.21.15.123:0/397127802 >> 172.21.15.123:0/397127802 conn(0x7fc4c40f89b0 msgr2=0x7fc4c40704f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:09:03.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:09:03.635+0000 7fc4c9e00700 1 -- 172.21.15.123:0/397127802 shutdown_connections 2024-03-30T05:09:03.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:09:03.635+0000 7fc4c9e00700 1 -- 172.21.15.123:0/397127802 wait complete. 2024-03-30T05:09:04.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:09:03 smithi123 ceph-mon[139369]: pgmap v1138: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:09:04.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:09:03 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/397127802' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:09:04.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:09:03 smithi135 ceph-mon[106011]: pgmap v1138: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:09:04.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:09:03 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/397127802' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:09:04.356 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -e sha1=470100a4ccbcf7769052bf09a5413881ccf1b97f -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-03-30T05:09:04.784 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T05:09:05.966 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:09:05 smithi123 ceph-mon[139369]: pgmap v1139: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:09:06.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:09:05 smithi135 ceph-mon[106011]: pgmap v1139: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:09:06.654 INFO:teuthology.orchestra.run.smithi123.stdout:wait for servicemap items w/ changing names to refresh 2024-03-30T05:09:07.370 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -e sha1=470100a4ccbcf7769052bf09a5413881ccf1b97f -- bash -c 'sleep 60' 2024-03-30T05:09:07.796 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T05:09:08.167 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:09:07 smithi123 ceph-mon[139369]: pgmap v1140: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:09:08.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:09:07 smithi135 ceph-mon[106011]: pgmap v1140: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:09:10.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:09:09 smithi123 ceph-mon[139369]: pgmap v1141: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:09:10.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:09:09 smithi135 ceph-mon[106011]: pgmap v1141: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:09:12.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:09:11 smithi123 ceph-mon[139369]: pgmap v1142: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:09:12.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:09:11 smithi135 ceph-mon[106011]: pgmap v1142: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:09:14.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:09:13 smithi123 ceph-mon[139369]: pgmap v1143: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:09:14.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:09:13 smithi135 ceph-mon[106011]: pgmap v1143: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:09:15.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:09:14 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:09:15.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:09:14 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:09:16.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:09:15 smithi123 ceph-mon[139369]: pgmap v1144: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:09:16.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:09:15 smithi135 ceph-mon[106011]: pgmap v1144: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:09:18.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:09:17 smithi123 ceph-mon[139369]: pgmap v1145: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:09:18.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:09:17 smithi135 ceph-mon[106011]: pgmap v1145: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:09:20.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:09:19 smithi123 ceph-mon[139369]: pgmap v1146: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:09:20.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:09:19 smithi135 ceph-mon[106011]: pgmap v1146: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:09:22.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:09:21 smithi123 ceph-mon[139369]: pgmap v1147: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:09:22.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:09:21 smithi135 ceph-mon[106011]: pgmap v1147: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:09:24.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:09:23 smithi123 ceph-mon[139369]: pgmap v1148: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:09:24.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:09:23 smithi135 ceph-mon[106011]: pgmap v1148: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:09:26.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:09:25 smithi123 ceph-mon[139369]: pgmap v1149: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:09:26.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:09:25 smithi135 ceph-mon[106011]: pgmap v1149: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:09:28.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:09:27 smithi123 ceph-mon[139369]: pgmap v1150: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:09:28.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:09:27 smithi135 ceph-mon[106011]: pgmap v1150: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:09:30.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:09:29 smithi135 ceph-mon[106011]: pgmap v1151: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:09:30.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:09:29 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:09:30.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:09:29 smithi123 ceph-mon[139369]: pgmap v1151: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:09:30.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:09:29 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:09:32.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:09:31 smithi135 ceph-mon[106011]: pgmap v1152: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:09:32.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:09:31 smithi123 ceph-mon[139369]: pgmap v1152: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:09:34.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:09:33 smithi135 ceph-mon[106011]: pgmap v1153: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:09:34.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:09:33 smithi123 ceph-mon[139369]: pgmap v1153: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:09:36.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:09:35 smithi135 ceph-mon[106011]: pgmap v1154: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:09:36.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:09:35 smithi123 ceph-mon[139369]: pgmap v1154: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:09:38.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:09:37 smithi135 ceph-mon[106011]: pgmap v1155: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:09:38.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:09:37 smithi123 ceph-mon[139369]: pgmap v1155: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:09:40.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:09:39 smithi135 ceph-mon[106011]: pgmap v1156: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:09:40.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:09:39 smithi123 ceph-mon[139369]: pgmap v1156: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:09:42.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:09:41 smithi135 ceph-mon[106011]: pgmap v1157: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:09:42.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:09:41 smithi123 ceph-mon[139369]: pgmap v1157: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:09:43.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:09:42 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T05:09:43.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:09:42 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T05:09:44.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:09:44 smithi135 ceph-mon[106011]: pgmap v1158: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:09:44.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:09:44 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T05:09:44.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:09:44 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T05:09:44.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:09:44 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:09:44.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:09:44 smithi123 ceph-mon[139369]: pgmap v1158: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:09:44.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:09:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T05:09:44.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:09:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T05:09:44.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:09:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:09:45.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:09:45 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:09:45.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:09:45 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:09:46.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:09:46 smithi135 ceph-mon[106011]: pgmap v1159: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:09:46.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:09:46 smithi123 ceph-mon[139369]: pgmap v1159: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:09:48.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:09:48 smithi135 ceph-mon[106011]: pgmap v1160: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:09:48.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:09:48 smithi123 ceph-mon[139369]: pgmap v1160: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:09:50.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:09:50 smithi135 ceph-mon[106011]: pgmap v1161: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:09:50.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:09:50 smithi123 ceph-mon[139369]: pgmap v1161: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:09:52.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:09:52 smithi135 ceph-mon[106011]: pgmap v1162: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:09:52.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:09:52 smithi123 ceph-mon[139369]: pgmap v1162: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:09:54.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:09:54 smithi135 ceph-mon[106011]: pgmap v1163: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:09:54.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:09:54 smithi123 ceph-mon[139369]: pgmap v1163: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:09:56.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:09:56 smithi135 ceph-mon[106011]: pgmap v1164: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:09:56.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:09:56 smithi123 ceph-mon[139369]: pgmap v1164: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:09:58.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:09:58 smithi135 ceph-mon[106011]: pgmap v1165: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:09:58.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:09:58 smithi123 ceph-mon[139369]: pgmap v1165: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:10:00.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:00 smithi135 ceph-mon[106011]: pgmap v1166: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:10:00.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:00 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:10:00.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:00 smithi135 ceph-mon[106011]: overall HEALTH_OK 2024-03-30T05:10:00.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:00 smithi123 ceph-mon[139369]: pgmap v1166: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:10:00.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:00 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:10:00.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:00 smithi123 ceph-mon[139369]: overall HEALTH_OK 2024-03-30T05:10:02.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:02 smithi135 ceph-mon[106011]: pgmap v1167: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:10:02.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:02 smithi123 ceph-mon[139369]: pgmap v1167: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:10:04.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:04 smithi135 ceph-mon[106011]: pgmap v1168: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:10:04.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:04 smithi123 ceph-mon[139369]: pgmap v1168: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:10:06.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:06 smithi135 ceph-mon[106011]: pgmap v1169: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:10:06.465 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:06 smithi123 ceph-mon[139369]: pgmap v1169: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:10:08.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:08 smithi135 ceph-mon[106011]: pgmap v1170: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:10:08.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:08 smithi123 ceph-mon[139369]: pgmap v1170: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:10:10.303 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:10 smithi123 ceph-mon[139369]: pgmap v1171: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:10:10.335 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -e sha1=470100a4ccbcf7769052bf09a5413881ccf1b97f -- bash -c 'ceph orch ps' 2024-03-30T05:10:10.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:10 smithi135 ceph-mon[106011]: pgmap v1171: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:10:10.766 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T05:10:12.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:12 smithi135 ceph-mon[106011]: pgmap v1172: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:10:12.396 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:12 smithi123 ceph-mon[139369]: pgmap v1172: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:10:12.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:12.684+0000 7f84c5600700 1 -- 172.21.15.123:0/3174526022 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84c0102e00 msgr2=0x7f84c01031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:12.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:12.684+0000 7f84c5600700 1 --2- 172.21.15.123:0/3174526022 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84c0102e00 0x7f84c01031d0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f84b0004940 tx=0x7f84b0009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:12.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:12.684+0000 7f84c5600700 1 -- 172.21.15.123:0/3174526022 shutdown_connections 2024-03-30T05:10:12.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:12.684+0000 7f84c5600700 1 --2- 172.21.15.123:0/3174526022 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f84c01037a0 0x7f84c00fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:12.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:12.684+0000 7f84c5600700 1 --2- 172.21.15.123:0/3174526022 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84c0102e00 0x7f84c01031d0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:12.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:12.684+0000 7f84c5600700 1 -- 172.21.15.123:0/3174526022 >> 172.21.15.123:0/3174526022 conn(0x7f84c00f89b0 msgr2=0x7f84c00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:12.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:12.684+0000 7f84c5600700 1 -- 172.21.15.123:0/3174526022 shutdown_connections 2024-03-30T05:10:12.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:12.684+0000 7f84c5600700 1 -- 172.21.15.123:0/3174526022 wait complete. 2024-03-30T05:10:12.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:12.685+0000 7f84c5600700 1 Processor -- start 2024-03-30T05:10:12.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:12.685+0000 7f84c5600700 1 -- start start 2024-03-30T05:10:12.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:12.685+0000 7f84c5600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f84c0102e00 0x7f84c0115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:12.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:12.685+0000 7f84c5600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84c01037a0 0x7f84c0116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:12.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:12.685+0000 7f84c5600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c0116860 con 0x7f84c01037a0 2024-03-30T05:10:12.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:12.685+0000 7f84c5600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c010fc30 con 0x7f84c0102e00 2024-03-30T05:10:12.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:12.685+0000 7f84b7e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84c01037a0 0x7f84c0116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:12.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:12.685+0000 7f84b7e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84c01037a0 0x7f84c0116210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42204/0 (socket says 172.21.15.123:42204) 2024-03-30T05:10:12.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:12.685+0000 7f84b7e00700 1 -- 172.21.15.123:0/2391811275 learned_addr learned my addr 172.21.15.123:0/2391811275 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:10:12.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:12.686+0000 7f84bea00700 1 --2- 172.21.15.123:0/2391811275 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f84c0102e00 0x7f84c0115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:12.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:12.686+0000 7f84b7e00700 1 -- 172.21.15.123:0/2391811275 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f84c0102e00 msgr2=0x7f84c0115cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:12.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:12.686+0000 7f84b7e00700 1 --2- 172.21.15.123:0/2391811275 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f84c0102e00 0x7f84c0115cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:12.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:12.686+0000 7f84b7e00700 1 -- 172.21.15.123:0/2391811275 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84b0005040 con 0x7f84c01037a0 2024-03-30T05:10:12.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:12.686+0000 7f84bea00700 1 --2- 172.21.15.123:0/2391811275 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f84c0102e00 0x7f84c0115cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T05:10:12.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:12.686+0000 7f84b7e00700 1 --2- 172.21.15.123:0/2391811275 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84c01037a0 0x7f84c0116210 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f84a800c890 tx=0x7f84a80044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:12.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:12.686+0000 7f84b6a00700 1 -- 172.21.15.123:0/2391811275 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f84a800a7d0 con 0x7f84c01037a0 2024-03-30T05:10:12.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:12.686+0000 7f84b6a00700 1 -- 172.21.15.123:0/2391811275 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f84a800ade0 con 0x7f84c01037a0 2024-03-30T05:10:12.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:12.686+0000 7f84b6a00700 1 -- 172.21.15.123:0/2391811275 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f84a80113a0 con 0x7f84c01037a0 2024-03-30T05:10:12.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:12.686+0000 7f84c5600700 1 -- 172.21.15.123:0/2391811275 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84c010ff10 con 0x7f84c01037a0 2024-03-30T05:10:12.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:12.686+0000 7f84c5600700 1 -- 172.21.15.123:0/2391811275 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84c01104d0 con 0x7f84c01037a0 2024-03-30T05:10:12.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:12.687+0000 7f84c5600700 1 -- 172.21.15.123:0/2391811275 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84c00fe550 con 0x7f84c01037a0 2024-03-30T05:10:12.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:12.688+0000 7f84b6a00700 1 -- 172.21.15.123:0/2391811275 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f84a8012400 con 0x7f84c01037a0 2024-03-30T05:10:12.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:12.689+0000 7f84b6a00700 1 --2- 172.21.15.123:0/2391811275 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f84a406cc30 0x7f84a406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:12.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:12.689+0000 7f84b6a00700 1 -- 172.21.15.123:0/2391811275 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f84a808c1a0 con 0x7f84c01037a0 2024-03-30T05:10:12.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:12.690+0000 7f84bea00700 1 --2- 172.21.15.123:0/2391811275 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f84a406cc30 0x7f84a406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:12.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:12.690+0000 7f84bea00700 1 --2- 172.21.15.123:0/2391811275 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f84a406cc30 0x7f84a406f0e0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f84b000b3b0 tx=0x7f84b0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:12.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:12.694+0000 7f84b6a00700 1 -- 172.21.15.123:0/2391811275 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f84a8059bf0 con 0x7f84c01037a0 2024-03-30T05:10:12.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:12.906+0000 7f84c5600700 1 -- 172.21.15.123:0/2391811275 --> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f84c0109e30 con 0x7f84a406cc30 2024-03-30T05:10:12.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:12.916+0000 7f84b6a00700 1 -- 172.21.15.123:0/2391811275 <== mgr.34104 v2:172.21.15.123:6800/4244519131 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f84c0109e30 con 0x7f84a406cc30 2024-03-30T05:10:12.914 INFO:teuthology.orchestra.run.smithi123.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T05:10:12.914 INFO:teuthology.orchestra.run.smithi123.stdout:alertmanager.smithi123 smithi123 *:9093,9094 running (39m) 100s ago 51m 14.4M - 0.25.0 c8568f914cd2 22473830d06b 2024-03-30T05:10:12.914 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi123 smithi123 running (36m) 100s ago 52m 6656k - 18.2.2-1018-g470100a4 e7c444a08cf7 7c83493966b9 2024-03-30T05:10:12.914 INFO:teuthology.orchestra.run.smithi123.stdout:crash.smithi135 smithi135 running (36m) 104s ago 50m 6660k - 18.2.2-1018-g470100a4 e7c444a08cf7 5e1dee72c7f8 2024-03-30T05:10:12.914 INFO:teuthology.orchestra.run.smithi123.stdout:grafana.smithi123 smithi123 *:3000 running (39m) 100s ago 51m 76.9M - 9.4.7 954c08fa6188 c43e663f8a18 2024-03-30T05:10:12.914 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.dvtind smithi123 running (32m) 100s ago 45m 120M - 18.2.2-1018-g470100a4 e7c444a08cf7 08eb25ca7457 2024-03-30T05:10:12.914 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi123.tqxvzr smithi123 running (32m) 100s ago 45m 110M - 18.2.2-1018-g470100a4 e7c444a08cf7 c700acd7a242 2024-03-30T05:10:12.914 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.xwgjvb smithi135 running (16m) 104s ago 45m 16.3M - 18.2.2-1018-g470100a4 e7c444a08cf7 87b073a6c9ca 2024-03-30T05:10:12.914 INFO:teuthology.orchestra.run.smithi123.stdout:mds.cephfs.smithi135.zvnlap smithi135 running (32m) 104s ago 45m 21.5M - 18.2.2-1018-g470100a4 e7c444a08cf7 6e96c4ff02a3 2024-03-30T05:10:12.914 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi123.xgksqq smithi123 *:8443,9283,8765 running (41m) 100s ago 53m 514M - 18.2.2-1018-g470100a4 e7c444a08cf7 74bc39dd0171 2024-03-30T05:10:12.914 INFO:teuthology.orchestra.run.smithi123.stdout:mgr.smithi135.kyblli smithi135 *:8443,9283,8765 running (40m) 104s ago 50m 457M - 18.2.2-1018-g470100a4 e7c444a08cf7 1fade164a9aa 2024-03-30T05:10:12.914 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi123 smithi123 running (37m) 100s ago 53m 84.9M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 2b3ed6979ed2 2024-03-30T05:10:12.915 INFO:teuthology.orchestra.run.smithi123.stdout:mon.smithi135 smithi135 running (37m) 104s ago 50m 57.9M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 a66acfbe9e0a 2024-03-30T05:10:12.915 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi123 smithi123 *:9100 running (40m) 100s ago 52m 8368k - 1.5.0 0da6a335fe13 9430333d4610 2024-03-30T05:10:12.915 INFO:teuthology.orchestra.run.smithi123.stdout:node-exporter.smithi135 smithi135 *:9100 running (40m) 104s ago 50m 8669k - 1.5.0 0da6a335fe13 91059eac81a0 2024-03-30T05:10:12.917 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 smithi123 running (36m) 100s ago 49m 111M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 f8d62db1437b 2024-03-30T05:10:12.917 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 smithi123 running (35m) 100s ago 49m 104M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 5d21109ef266 2024-03-30T05:10:12.917 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 smithi123 running (35m) 100s ago 48m 94.2M 1618M 18.2.2-1018-g470100a4 e7c444a08cf7 06673576e943 2024-03-30T05:10:12.917 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 smithi135 running (34m) 104s ago 48m 107M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 acbbc38441e4 2024-03-30T05:10:12.917 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 smithi135 running (33m) 104s ago 47m 106M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 c7b2e84c0f81 2024-03-30T05:10:12.917 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 smithi135 running (33m) 104s ago 47m 94.8M 2642M 18.2.2-1018-g470100a4 e7c444a08cf7 27615b610231 2024-03-30T05:10:12.917 INFO:teuthology.orchestra.run.smithi123.stdout:prometheus.smithi123 smithi123 *:9095 running (39m) 100s ago 51m 66.2M - 2.43.0 a07b618ecd1d 536bb98877ba 2024-03-30T05:10:12.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:12.918+0000 7f84c5600700 1 -- 172.21.15.123:0/2391811275 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f84a406cc30 msgr2=0x7f84a406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:12.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:12.918+0000 7f84c5600700 1 --2- 172.21.15.123:0/2391811275 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f84a406cc30 0x7f84a406f0e0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f84b000b3b0 tx=0x7f84b0009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:12.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:12.918+0000 7f84c5600700 1 -- 172.21.15.123:0/2391811275 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84c01037a0 msgr2=0x7f84c0116210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:12.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:12.918+0000 7f84c5600700 1 --2- 172.21.15.123:0/2391811275 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84c01037a0 0x7f84c0116210 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f84a800c890 tx=0x7f84a80044c0 comp rx=0 tx=0).stop 2024-03-30T05:10:12.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:12.918+0000 7f84c5600700 1 -- 172.21.15.123:0/2391811275 shutdown_connections 2024-03-30T05:10:12.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:12.918+0000 7f84c5600700 1 --2- 172.21.15.123:0/2391811275 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f84a406cc30 0x7f84a406f0e0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:12.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:12.918+0000 7f84c5600700 1 --2- 172.21.15.123:0/2391811275 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f84c0102e00 0x7f84c0115cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:12.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:12.918+0000 7f84c5600700 1 --2- 172.21.15.123:0/2391811275 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84c01037a0 0x7f84c0116210 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:12.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:12.918+0000 7f84c5600700 1 -- 172.21.15.123:0/2391811275 >> 172.21.15.123:0/2391811275 conn(0x7f84c00f89b0 msgr2=0x7f84c00f99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:12.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:12.918+0000 7f84c5600700 1 -- 172.21.15.123:0/2391811275 shutdown_connections 2024-03-30T05:10:12.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:12.918+0000 7f84c5600700 1 -- 172.21.15.123:0/2391811275 wait complete. 2024-03-30T05:10:13.804 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -e sha1=470100a4ccbcf7769052bf09a5413881ccf1b97f -- bash -c 'ceph versions' 2024-03-30T05:10:14.231 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T05:10:14.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:14 smithi135 ceph-mon[106011]: pgmap v1173: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:10:14.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:14 smithi135 ceph-mon[106011]: from='client.36048 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:10:14.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:14 smithi123 ceph-mon[139369]: pgmap v1173: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:10:14.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:14 smithi123 ceph-mon[139369]: from='client.36048 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T05:10:15.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:15 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:10:15.463 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:15 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:10:16.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:16.187+0000 7fe9dd000700 1 -- 172.21.15.123:0/30502718 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9d8102e00 msgr2=0x7fe9d81031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:16.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:16.187+0000 7fe9dd000700 1 --2- 172.21.15.123:0/30502718 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9d8102e00 0x7fe9d81031d0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fe9c8004940 tx=0x7fe9c8009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:16.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:16.187+0000 7fe9dd000700 1 -- 172.21.15.123:0/30502718 shutdown_connections 2024-03-30T05:10:16.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:16.187+0000 7fe9dd000700 1 --2- 172.21.15.123:0/30502718 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe9d81037a0 0x7fe9d80fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:16.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:16.187+0000 7fe9dd000700 1 --2- 172.21.15.123:0/30502718 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9d8102e00 0x7fe9d81031d0 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:16.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:16.187+0000 7fe9dd000700 1 -- 172.21.15.123:0/30502718 >> 172.21.15.123:0/30502718 conn(0x7fe9d80f89b0 msgr2=0x7fe9d80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:16.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:16.187+0000 7fe9dd000700 1 -- 172.21.15.123:0/30502718 shutdown_connections 2024-03-30T05:10:16.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:16.187+0000 7fe9dd000700 1 -- 172.21.15.123:0/30502718 wait complete. 2024-03-30T05:10:16.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:16.188+0000 7fe9dd000700 1 Processor -- start 2024-03-30T05:10:16.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:16.188+0000 7fe9dd000700 1 -- start start 2024-03-30T05:10:16.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:16.188+0000 7fe9dd000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9d8102e00 0x7fe9d81138b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:16.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:16.188+0000 7fe9dd000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe9d81037a0 0x7fe9d8113df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:16.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:16.189+0000 7fe9dd000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d8114440 con 0x7fe9d8102e00 2024-03-30T05:10:16.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:16.189+0000 7fe9dd000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d810da40 con 0x7fe9d81037a0 2024-03-30T05:10:16.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:16.189+0000 7fe9d5e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9d8102e00 0x7fe9d81138b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:16.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:16.189+0000 7fe9d5e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9d8102e00 0x7fe9d81138b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34840/0 (socket says 172.21.15.123:34840) 2024-03-30T05:10:16.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:16.189+0000 7fe9d5e00700 1 -- 172.21.15.123:0/2154321394 learned_addr learned my addr 172.21.15.123:0/2154321394 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:10:16.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:16.189+0000 7fe9d5400700 1 --2- 172.21.15.123:0/2154321394 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe9d81037a0 0x7fe9d8113df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:16.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:16.189+0000 7fe9d5e00700 1 -- 172.21.15.123:0/2154321394 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe9d81037a0 msgr2=0x7fe9d8113df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:16.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:16.189+0000 7fe9d5e00700 1 --2- 172.21.15.123:0/2154321394 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe9d81037a0 0x7fe9d8113df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:16.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:16.189+0000 7fe9d5e00700 1 -- 172.21.15.123:0/2154321394 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c8005040 con 0x7fe9d8102e00 2024-03-30T05:10:16.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:16.189+0000 7fe9d5e00700 1 --2- 172.21.15.123:0/2154321394 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9d8102e00 0x7fe9d81138b0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fe9c8009360 tx=0x7fe9c8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:16.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:16.190+0000 7fe9ce000700 1 -- 172.21.15.123:0/2154321394 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9c800e040 con 0x7fe9d8102e00 2024-03-30T05:10:16.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:16.190+0000 7fe9dd000700 1 -- 172.21.15.123:0/2154321394 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9d810dcc0 con 0x7fe9d8102e00 2024-03-30T05:10:16.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:16.190+0000 7fe9ce000700 1 -- 172.21.15.123:0/2154321394 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe9c8013420 con 0x7fe9d8102e00 2024-03-30T05:10:16.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:16.190+0000 7fe9ce000700 1 -- 172.21.15.123:0/2154321394 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9c801c5f0 con 0x7fe9d8102e00 2024-03-30T05:10:16.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:16.190+0000 7fe9dd000700 1 -- 172.21.15.123:0/2154321394 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9d810e280 con 0x7fe9d8102e00 2024-03-30T05:10:16.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:16.191+0000 7fe9dd000700 1 -- 172.21.15.123:0/2154321394 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9d80fe550 con 0x7fe9d8102e00 2024-03-30T05:10:16.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:16.192+0000 7fe9ce000700 1 -- 172.21.15.123:0/2154321394 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fe9c8013590 con 0x7fe9d8102e00 2024-03-30T05:10:16.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:16.193+0000 7fe9ce000700 1 --2- 172.21.15.123:0/2154321394 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe9bc06cc20 0x7fe9bc06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:16.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:16.193+0000 7fe9ce000700 1 -- 172.21.15.123:0/2154321394 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe9c808d2c0 con 0x7fe9d8102e00 2024-03-30T05:10:16.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:16.193+0000 7fe9d5400700 1 --2- 172.21.15.123:0/2154321394 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe9bc06cc20 0x7fe9bc06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:16.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:16.193+0000 7fe9d5400700 1 --2- 172.21.15.123:0/2154321394 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe9bc06cc20 0x7fe9bc06f0d0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fe9c4007fd0 tx=0x7fe9c4009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:16.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:16.197+0000 7fe9ce000700 1 -- 172.21.15.123:0/2154321394 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe9c805ad90 con 0x7fe9d8102e00 2024-03-30T05:10:16.215 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:16 smithi123 ceph-mon[139369]: pgmap v1174: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:10:16.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:16 smithi135 ceph-mon[106011]: pgmap v1174: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:10:16.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:16.481+0000 7fe9dd000700 1 -- 172.21.15.123:0/2154321394 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe9d8109dc0 con 0x7fe9d8102e00 2024-03-30T05:10:16.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:16.482+0000 7fe9ce000700 1 -- 172.21.15.123:0/2154321394 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7fe9c805a920 con 0x7fe9d8102e00 2024-03-30T05:10:16.480 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-03-30T05:10:16.480 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": { 2024-03-30T05:10:16.480 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T05:10:16.480 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:10:16.480 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": { 2024-03-30T05:10:16.480 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 2 2024-03-30T05:10:16.480 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:10:16.480 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": { 2024-03-30T05:10:16.480 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 6 2024-03-30T05:10:16.480 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:10:16.480 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": { 2024-03-30T05:10:16.480 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 4 2024-03-30T05:10:16.481 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-03-30T05:10:16.481 INFO:teuthology.orchestra.run.smithi123.stdout: "overall": { 2024-03-30T05:10:16.481 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)": 14 2024-03-30T05:10:16.481 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-03-30T05:10:16.481 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-03-30T05:10:16.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:16.484+0000 7fe9dd000700 1 -- 172.21.15.123:0/2154321394 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe9bc06cc20 msgr2=0x7fe9bc06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:16.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:16.484+0000 7fe9dd000700 1 --2- 172.21.15.123:0/2154321394 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe9bc06cc20 0x7fe9bc06f0d0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fe9c4007fd0 tx=0x7fe9c4009040 comp rx=0 tx=0).stop 2024-03-30T05:10:16.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:16.484+0000 7fe9dd000700 1 -- 172.21.15.123:0/2154321394 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9d8102e00 msgr2=0x7fe9d81138b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:16.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:16.484+0000 7fe9dd000700 1 --2- 172.21.15.123:0/2154321394 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9d8102e00 0x7fe9d81138b0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fe9c8009360 tx=0x7fe9c8009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:16.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:16.484+0000 7fe9dd000700 1 -- 172.21.15.123:0/2154321394 shutdown_connections 2024-03-30T05:10:16.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:16.484+0000 7fe9dd000700 1 --2- 172.21.15.123:0/2154321394 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe9bc06cc20 0x7fe9bc06f0d0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:16.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:16.484+0000 7fe9dd000700 1 --2- 172.21.15.123:0/2154321394 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9d8102e00 0x7fe9d81138b0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:16.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:16.484+0000 7fe9dd000700 1 --2- 172.21.15.123:0/2154321394 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe9d81037a0 0x7fe9d8113df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:16.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:16.484+0000 7fe9dd000700 1 -- 172.21.15.123:0/2154321394 >> 172.21.15.123:0/2154321394 conn(0x7fe9d80f89b0 msgr2=0x7fe9d80f9a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:16.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:16.484+0000 7fe9dd000700 1 -- 172.21.15.123:0/2154321394 shutdown_connections 2024-03-30T05:10:16.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:16.484+0000 7fe9dd000700 1 -- 172.21.15.123:0/2154321394 wait complete. 2024-03-30T05:10:17.083 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -e sha1=470100a4ccbcf7769052bf09a5413881ccf1b97f -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-03-30T05:10:17.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:17 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2154321394' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:10:17.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:17 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2154321394' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:10:17.510 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T05:10:18.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:18 smithi135 ceph-mon[106011]: pgmap v1175: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:10:18.447 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:18 smithi123 ceph-mon[139369]: pgmap v1175: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:10:19.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:19.416+0000 7fe3f7c00700 1 -- 172.21.15.123:0/3884985332 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3f0102e00 msgr2=0x7fe3f01031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:19.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:19.416+0000 7fe3f7c00700 1 --2- 172.21.15.123:0/3884985332 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3f0102e00 0x7fe3f01031d0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fe3e4004940 tx=0x7fe3e4009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:19.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:19.416+0000 7fe3f7c00700 1 -- 172.21.15.123:0/3884985332 shutdown_connections 2024-03-30T05:10:19.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:19.416+0000 7fe3f7c00700 1 --2- 172.21.15.123:0/3884985332 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe3f01037a0 0x7fe3f00fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:19.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:19.416+0000 7fe3f7c00700 1 --2- 172.21.15.123:0/3884985332 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3f0102e00 0x7fe3f01031d0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:19.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:19.416+0000 7fe3f7c00700 1 -- 172.21.15.123:0/3884985332 >> 172.21.15.123:0/3884985332 conn(0x7fe3f00f89b0 msgr2=0x7fe3f00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:19.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:19.417+0000 7fe3f7c00700 1 -- 172.21.15.123:0/3884985332 shutdown_connections 2024-03-30T05:10:19.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:19.417+0000 7fe3f7c00700 1 -- 172.21.15.123:0/3884985332 wait complete. 2024-03-30T05:10:19.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:19.417+0000 7fe3f7c00700 1 Processor -- start 2024-03-30T05:10:19.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:19.418+0000 7fe3f7c00700 1 -- start start 2024-03-30T05:10:19.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:19.418+0000 7fe3f7c00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe3f0102e00 0x7fe3f0115d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:19.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:19.418+0000 7fe3f7c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3f01037a0 0x7fe3f0116250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:19.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:19.418+0000 7fe3f7c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f01168a0 con 0x7fe3f01037a0 2024-03-30T05:10:19.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:19.418+0000 7fe3f7c00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f010fc70 con 0x7fe3f0102e00 2024-03-30T05:10:19.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:19.418+0000 7fe3efe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3f01037a0 0x7fe3f0116250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:19.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:19.418+0000 7fe3efe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3f01037a0 0x7fe3f0116250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34870/0 (socket says 172.21.15.123:34870) 2024-03-30T05:10:19.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:19.418+0000 7fe3efe00700 1 -- 172.21.15.123:0/3192055871 learned_addr learned my addr 172.21.15.123:0/3192055871 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:10:19.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:19.418+0000 7fe3f5200700 1 --2- 172.21.15.123:0/3192055871 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe3f0102e00 0x7fe3f0115d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:19.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:19.419+0000 7fe3efe00700 1 -- 172.21.15.123:0/3192055871 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe3f0102e00 msgr2=0x7fe3f0115d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:19.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:19.419+0000 7fe3efe00700 1 --2- 172.21.15.123:0/3192055871 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe3f0102e00 0x7fe3f0115d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:19.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:19.419+0000 7fe3efe00700 1 -- 172.21.15.123:0/3192055871 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3e4005040 con 0x7fe3f01037a0 2024-03-30T05:10:19.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:19.419+0000 7fe3efe00700 1 --2- 172.21.15.123:0/3192055871 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3f01037a0 0x7fe3f0116250 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc00c800 tx=0x7fe3dc004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:19.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:19.419+0000 7fe3ecc00700 1 -- 172.21.15.123:0/3192055871 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3dc00a760 con 0x7fe3f01037a0 2024-03-30T05:10:19.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:19.419+0000 7fe3ecc00700 1 -- 172.21.15.123:0/3192055871 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe3dc00ad70 con 0x7fe3f01037a0 2024-03-30T05:10:19.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:19.419+0000 7fe3f7c00700 1 -- 172.21.15.123:0/3192055871 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3f010ff50 con 0x7fe3f01037a0 2024-03-30T05:10:19.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:19.419+0000 7fe3ecc00700 1 -- 172.21.15.123:0/3192055871 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3dc0113a0 con 0x7fe3f01037a0 2024-03-30T05:10:19.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:19.419+0000 7fe3f7c00700 1 -- 172.21.15.123:0/3192055871 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3f0110510 con 0x7fe3f01037a0 2024-03-30T05:10:19.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:19.420+0000 7fe3f7c00700 1 -- 172.21.15.123:0/3192055871 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3f00fe550 con 0x7fe3f01037a0 2024-03-30T05:10:19.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:19.421+0000 7fe3ecc00700 1 -- 172.21.15.123:0/3192055871 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fe3dc012400 con 0x7fe3f01037a0 2024-03-30T05:10:19.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:19.422+0000 7fe3ecc00700 1 --2- 172.21.15.123:0/3192055871 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe3d806cc30 0x7fe3d806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:19.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:19.423+0000 7fe3ecc00700 1 -- 172.21.15.123:0/3192055871 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe3dc08c0d0 con 0x7fe3f01037a0 2024-03-30T05:10:19.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:19.423+0000 7fe3f5200700 1 --2- 172.21.15.123:0/3192055871 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe3d806cc30 0x7fe3d806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:19.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:19.423+0000 7fe3f5200700 1 --2- 172.21.15.123:0/3192055871 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe3d806cc30 0x7fe3d806f0e0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fe3e400b020 tx=0x7fe3e4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:19.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:19.427+0000 7fe3ecc00700 1 -- 172.21.15.123:0/3192055871 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe3dc059b20 con 0x7fe3f01037a0 2024-03-30T05:10:19.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:19.718+0000 7fe3f7c00700 1 -- 172.21.15.123:0/3192055871 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe3f004bed0 con 0x7fe3f01037a0 2024-03-30T05:10:19.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:19.718+0000 7fe3ecc00700 1 -- 172.21.15.123:0/3192055871 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7fe3dc0596b0 con 0x7fe3f01037a0 2024-03-30T05:10:19.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:19.720+0000 7fe3f7c00700 1 -- 172.21.15.123:0/3192055871 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe3d806cc30 msgr2=0x7fe3d806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:19.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:19.720+0000 7fe3f7c00700 1 --2- 172.21.15.123:0/3192055871 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe3d806cc30 0x7fe3d806f0e0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fe3e400b020 tx=0x7fe3e4009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:19.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:19.720+0000 7fe3f7c00700 1 -- 172.21.15.123:0/3192055871 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3f01037a0 msgr2=0x7fe3f0116250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:19.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:19.720+0000 7fe3f7c00700 1 --2- 172.21.15.123:0/3192055871 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3f01037a0 0x7fe3f0116250 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc00c800 tx=0x7fe3dc004470 comp rx=0 tx=0).stop 2024-03-30T05:10:19.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:19.721+0000 7fe3f7c00700 1 -- 172.21.15.123:0/3192055871 shutdown_connections 2024-03-30T05:10:19.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:19.721+0000 7fe3f7c00700 1 --2- 172.21.15.123:0/3192055871 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe3d806cc30 0x7fe3d806f0e0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:19.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:19.721+0000 7fe3f7c00700 1 --2- 172.21.15.123:0/3192055871 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe3f0102e00 0x7fe3f0115d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:19.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:19.721+0000 7fe3f7c00700 1 --2- 172.21.15.123:0/3192055871 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3f01037a0 0x7fe3f0116250 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:19.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:19.721+0000 7fe3f7c00700 1 -- 172.21.15.123:0/3192055871 >> 172.21.15.123:0/3192055871 conn(0x7fe3f00f89b0 msgr2=0x7fe3f00704f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:19.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:19.721+0000 7fe3f7c00700 1 -- 172.21.15.123:0/3192055871 shutdown_connections 2024-03-30T05:10:19.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:19.721+0000 7fe3f7c00700 1 -- 172.21.15.123:0/3192055871 wait complete. 2024-03-30T05:10:19.734 INFO:teuthology.orchestra.run.smithi123.stdout:true 2024-03-30T05:10:20.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:20 smithi135 ceph-mon[106011]: pgmap v1176: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:10:20.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:20 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3192055871' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:10:20.416 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:20 smithi123 ceph-mon[139369]: pgmap v1176: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:10:20.416 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:20 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3192055871' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:10:20.442 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -e sha1=470100a4ccbcf7769052bf09a5413881ccf1b97f -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-03-30T05:10:20.868 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T05:10:22.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:22 smithi123 ceph-mon[139369]: pgmap v1177: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:10:22.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:22 smithi135 ceph-mon[106011]: pgmap v1177: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:10:22.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:22.978+0000 7fedf2200700 1 -- 172.21.15.123:0/4055723513 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedec1037a0 msgr2=0x7fedec0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:22.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:22.978+0000 7fedf2200700 1 --2- 172.21.15.123:0/4055723513 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedec1037a0 0x7fedec0fce10 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fedd8004940 tx=0x7fedd8009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:22.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:22.978+0000 7fedf2200700 1 -- 172.21.15.123:0/4055723513 shutdown_connections 2024-03-30T05:10:22.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:22.978+0000 7fedf2200700 1 --2- 172.21.15.123:0/4055723513 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedec1037a0 0x7fedec0fce10 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:22.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:22.978+0000 7fedf2200700 1 --2- 172.21.15.123:0/4055723513 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fedec102e00 0x7fedec1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:22.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:22.978+0000 7fedf2200700 1 -- 172.21.15.123:0/4055723513 >> 172.21.15.123:0/4055723513 conn(0x7fedec0f89b0 msgr2=0x7fedec0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:22.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:22.978+0000 7fedf2200700 1 -- 172.21.15.123:0/4055723513 shutdown_connections 2024-03-30T05:10:22.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:22.978+0000 7fedf2200700 1 -- 172.21.15.123:0/4055723513 wait complete. 2024-03-30T05:10:22.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:22.979+0000 7fedf2200700 1 Processor -- start 2024-03-30T05:10:22.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:22.979+0000 7fedf2200700 1 -- start start 2024-03-30T05:10:22.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:22.980+0000 7fedf2200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedec102e00 0x7fedec115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:22.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:22.980+0000 7fedf2200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fedec1037a0 0x7fedec116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:22.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:22.980+0000 7fedf2200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedec116880 con 0x7fedec102e00 2024-03-30T05:10:22.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:22.980+0000 7fedf2200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedec10fc50 con 0x7fedec1037a0 2024-03-30T05:10:22.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:22.980+0000 7fedeb400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedec102e00 0x7fedec115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:22.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:22.980+0000 7fedeb400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedec102e00 0x7fedec115cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34888/0 (socket says 172.21.15.123:34888) 2024-03-30T05:10:22.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:22.980+0000 7fedeb400700 1 -- 172.21.15.123:0/945445139 learned_addr learned my addr 172.21.15.123:0/945445139 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:10:22.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:22.980+0000 7fedeaa00700 1 --2- 172.21.15.123:0/945445139 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fedec1037a0 0x7fedec116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:22.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:22.980+0000 7fedeb400700 1 -- 172.21.15.123:0/945445139 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fedec1037a0 msgr2=0x7fedec116230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:22.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:22.980+0000 7fedeb400700 1 --2- 172.21.15.123:0/945445139 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fedec1037a0 0x7fedec116230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:22.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:22.980+0000 7fedeb400700 1 -- 172.21.15.123:0/945445139 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd8005040 con 0x7fedec102e00 2024-03-30T05:10:22.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:22.981+0000 7fedeb400700 1 --2- 172.21.15.123:0/945445139 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedec102e00 0x7fedec115cf0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7feddc002a50 tx=0x7feddc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:22.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:22.981+0000 7fede3400700 1 -- 172.21.15.123:0/945445139 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feddc007cc0 con 0x7fedec102e00 2024-03-30T05:10:22.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:22.981+0000 7fede3400700 1 -- 172.21.15.123:0/945445139 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feddc00f040 con 0x7fedec102e00 2024-03-30T05:10:22.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:22.981+0000 7fedf2200700 1 -- 172.21.15.123:0/945445139 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedec10ff30 con 0x7fedec102e00 2024-03-30T05:10:22.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:22.981+0000 7fede3400700 1 -- 172.21.15.123:0/945445139 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feddc009540 con 0x7fedec102e00 2024-03-30T05:10:22.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:22.981+0000 7fedf2200700 1 -- 172.21.15.123:0/945445139 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedec1104f0 con 0x7fedec102e00 2024-03-30T05:10:22.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:22.982+0000 7fedf2200700 1 -- 172.21.15.123:0/945445139 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedec0fe550 con 0x7fedec102e00 2024-03-30T05:10:22.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:22.983+0000 7fede3400700 1 -- 172.21.15.123:0/945445139 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7feddc0096a0 con 0x7fedec102e00 2024-03-30T05:10:22.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:22.984+0000 7fede3400700 1 --2- 172.21.15.123:0/945445139 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fedd006cc30 0x7fedd006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:22.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:22.984+0000 7fede3400700 1 -- 172.21.15.123:0/945445139 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7feddc08bde0 con 0x7fedec102e00 2024-03-30T05:10:22.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:22.984+0000 7fedeaa00700 1 --2- 172.21.15.123:0/945445139 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fedd006cc30 0x7fedd006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:22.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:22.985+0000 7fedeaa00700 1 --2- 172.21.15.123:0/945445139 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fedd006cc30 0x7fedd006f0e0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fedd800b020 tx=0x7fedd8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:22.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:22.989+0000 7fede3400700 1 -- 172.21.15.123:0/945445139 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7feddc059830 con 0x7fedec102e00 2024-03-30T05:10:23.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:23.274+0000 7fedf2200700 1 -- 172.21.15.123:0/945445139 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fedec04bed0 con 0x7fedec102e00 2024-03-30T05:10:23.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:23.275+0000 7fede3400700 1 -- 172.21.15.123:0/945445139 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7feddc0593c0 con 0x7fedec102e00 2024-03-30T05:10:23.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:23.277+0000 7fedf2200700 1 -- 172.21.15.123:0/945445139 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fedd006cc30 msgr2=0x7fedd006f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:23.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:23.277+0000 7fedf2200700 1 --2- 172.21.15.123:0/945445139 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fedd006cc30 0x7fedd006f0e0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fedd800b020 tx=0x7fedd8009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:23.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:23.277+0000 7fedf2200700 1 -- 172.21.15.123:0/945445139 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedec102e00 msgr2=0x7fedec115cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:23.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:23.277+0000 7fedf2200700 1 --2- 172.21.15.123:0/945445139 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedec102e00 0x7fedec115cf0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7feddc002a50 tx=0x7feddc0046b0 comp rx=0 tx=0).stop 2024-03-30T05:10:23.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:23.277+0000 7fedf2200700 1 -- 172.21.15.123:0/945445139 shutdown_connections 2024-03-30T05:10:23.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:23.277+0000 7fedf2200700 1 --2- 172.21.15.123:0/945445139 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fedd006cc30 0x7fedd006f0e0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:23.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:23.277+0000 7fedf2200700 1 --2- 172.21.15.123:0/945445139 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedec102e00 0x7fedec115cf0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:23.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:23.277+0000 7fedf2200700 1 --2- 172.21.15.123:0/945445139 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fedec1037a0 0x7fedec116230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:23.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:23.277+0000 7fedf2200700 1 -- 172.21.15.123:0/945445139 >> 172.21.15.123:0/945445139 conn(0x7fedec0f89b0 msgr2=0x7fedec0f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:23.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:23.277+0000 7fedf2200700 1 -- 172.21.15.123:0/945445139 shutdown_connections 2024-03-30T05:10:23.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:23.277+0000 7fedf2200700 1 -- 172.21.15.123:0/945445139 wait complete. 2024-03-30T05:10:23.289 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)" 2024-03-30T05:10:24.038 DEBUG:teuthology.parallel:result is None 2024-03-30T05:10:24.038 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-30T05:10:24.051 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi123.front.sepia.ceph.com 2024-03-30T05:10:24.051 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 -- bash -c 'ceph fs dump' 2024-03-30T05:10:24.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:24 smithi135 ceph-mon[106011]: pgmap v1178: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:10:24.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:24 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/945445139' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:10:24.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:24 smithi123 ceph-mon[139369]: pgmap v1178: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:10:24.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:24 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/945445139' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-30T05:10:24.479 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/mon.smithi123/config 2024-03-30T05:10:26.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:26 smithi135 ceph-mon[106011]: pgmap v1179: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:10:26.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:26 smithi123 ceph-mon[139369]: pgmap v1179: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:10:26.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:26.500+0000 7fb56f200700 1 -- 172.21.15.123:0/2626890871 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb570102c50 msgr2=0x7fb570103020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:26.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:26.500+0000 7fb56f200700 1 --2- 172.21.15.123:0/2626890871 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb570102c50 0x7fb570103020 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fb560004940 tx=0x7fb560009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:26.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:26.500+0000 7fb56f200700 1 -- 172.21.15.123:0/2626890871 shutdown_connections 2024-03-30T05:10:26.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:26.500+0000 7fb56f200700 1 --2- 172.21.15.123:0/2626890871 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb5701035f0 0x7fb5700fcc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:26.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:26.500+0000 7fb56f200700 1 --2- 172.21.15.123:0/2626890871 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb570102c50 0x7fb570103020 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:26.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:26.500+0000 7fb56f200700 1 -- 172.21.15.123:0/2626890871 >> 172.21.15.123:0/2626890871 conn(0x7fb5700f87e0 msgr2=0x7fb5700fabf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:26.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:26.500+0000 7fb56f200700 1 -- 172.21.15.123:0/2626890871 shutdown_connections 2024-03-30T05:10:26.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:26.500+0000 7fb56f200700 1 -- 172.21.15.123:0/2626890871 wait complete. 2024-03-30T05:10:26.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:26.501+0000 7fb56f200700 1 Processor -- start 2024-03-30T05:10:26.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:26.501+0000 7fb56f200700 1 -- start start 2024-03-30T05:10:26.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:26.502+0000 7fb56f200700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb570102c50 0x7fb570115b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:26.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:26.502+0000 7fb56f200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5701035f0 0x7fb570116050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:26.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:26.502+0000 7fb56f200700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5701166a0 con 0x7fb5701035f0 2024-03-30T05:10:26.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:26.502+0000 7fb56f200700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb57010fa90 con 0x7fb570102c50 2024-03-30T05:10:26.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:26.502+0000 7fb56d400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5701035f0 0x7fb570116050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:26.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:26.502+0000 7fb56d400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5701035f0 0x7fb570116050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41206/0 (socket says 172.21.15.123:41206) 2024-03-30T05:10:26.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:26.502+0000 7fb56d400700 1 -- 172.21.15.123:0/3573217832 learned_addr learned my addr 172.21.15.123:0/3573217832 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:10:26.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:26.502+0000 7fb56de00700 1 --2- 172.21.15.123:0/3573217832 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb570102c50 0x7fb570115b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:26.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:26.502+0000 7fb56d400700 1 -- 172.21.15.123:0/3573217832 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb570102c50 msgr2=0x7fb570115b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:26.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:26.502+0000 7fb56d400700 1 --2- 172.21.15.123:0/3573217832 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb570102c50 0x7fb570115b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:26.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:26.502+0000 7fb56d400700 1 -- 172.21.15.123:0/3573217832 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb560005040 con 0x7fb5701035f0 2024-03-30T05:10:26.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:26.503+0000 7fb56d400700 1 --2- 172.21.15.123:0/3573217832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5701035f0 0x7fb570116050 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fb55c002a50 tx=0x7fb55c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:26.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:26.503+0000 7fb566000700 1 -- 172.21.15.123:0/3573217832 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb55c007cc0 con 0x7fb5701035f0 2024-03-30T05:10:26.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:26.503+0000 7fb566000700 1 -- 172.21.15.123:0/3573217832 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb55c00f040 con 0x7fb5701035f0 2024-03-30T05:10:26.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:26.503+0000 7fb56f200700 1 -- 172.21.15.123:0/3573217832 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb57010fd70 con 0x7fb5701035f0 2024-03-30T05:10:26.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:26.503+0000 7fb566000700 1 -- 172.21.15.123:0/3573217832 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb55c00d920 con 0x7fb5701035f0 2024-03-30T05:10:26.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:26.503+0000 7fb56f200700 1 -- 172.21.15.123:0/3573217832 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb570110330 con 0x7fb5701035f0 2024-03-30T05:10:26.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:26.504+0000 7fb56f200700 1 -- 172.21.15.123:0/3573217832 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5700fe3a0 con 0x7fb5701035f0 2024-03-30T05:10:26.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:26.505+0000 7fb566000700 1 -- 172.21.15.123:0/3573217832 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fb55c009680 con 0x7fb5701035f0 2024-03-30T05:10:26.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:26.506+0000 7fb566000700 1 --2- 172.21.15.123:0/3573217832 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb55406cc30 0x7fb55406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:26.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:26.506+0000 7fb566000700 1 -- 172.21.15.123:0/3573217832 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb55c08bd10 con 0x7fb5701035f0 2024-03-30T05:10:26.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:26.506+0000 7fb56de00700 1 --2- 172.21.15.123:0/3573217832 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb55406cc30 0x7fb55406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:26.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:26.507+0000 7fb56de00700 1 --2- 172.21.15.123:0/3573217832 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb55406cc30 0x7fb55406f0e0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fb56000b840 tx=0x7fb560009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:26.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:26.511+0000 7fb566000700 1 -- 172.21.15.123:0/3573217832 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb55c059760 con 0x7fb5701035f0 2024-03-30T05:10:26.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:26.756+0000 7fb56f200700 1 -- 172.21.15.123:0/3573217832 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fb570109c00 con 0x7fb5701035f0 2024-03-30T05:10:26.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:26.756+0000 7fb566000700 1 -- 172.21.15.123:0/3573217832 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 42 v42) v1 ==== 76+0+1878 (secure 0 0 0) 0x7fb55c0592f0 con 0x7fb5701035f0 2024-03-30T05:10:26.754 INFO:teuthology.orchestra.run.smithi123.stdout:e42 2024-03-30T05:10:26.755 INFO:teuthology.orchestra.run.smithi123.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T05:10:26.755 INFO:teuthology.orchestra.run.smithi123.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T05:10:26.755 INFO:teuthology.orchestra.run.smithi123.stdout:legacy client fscid: 1 2024-03-30T05:10:26.755 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:10:26.755 INFO:teuthology.orchestra.run.smithi123.stdout:Filesystem 'cephfs' (1) 2024-03-30T05:10:26.755 INFO:teuthology.orchestra.run.smithi123.stdout:fs_name cephfs 2024-03-30T05:10:26.755 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 42 2024-03-30T05:10:26.755 INFO:teuthology.orchestra.run.smithi123.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-30T05:10:26.755 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-03-30T04:24:30.211769+0000 2024-03-30T05:10:26.755 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2024-03-30T05:08:34.514493+0000 2024-03-30T05:10:26.755 INFO:teuthology.orchestra.run.smithi123.stdout:tableserver 0 2024-03-30T05:10:26.755 INFO:teuthology.orchestra.run.smithi123.stdout:root 0 2024-03-30T05:10:26.756 INFO:teuthology.orchestra.run.smithi123.stdout:session_timeout 60 2024-03-30T05:10:26.756 INFO:teuthology.orchestra.run.smithi123.stdout:session_autoclose 300 2024-03-30T05:10:26.756 INFO:teuthology.orchestra.run.smithi123.stdout:max_file_size 1099511627776 2024-03-30T05:10:26.756 INFO:teuthology.orchestra.run.smithi123.stdout:max_xattr_size 65536 2024-03-30T05:10:26.756 INFO:teuthology.orchestra.run.smithi123.stdout:required_client_features {} 2024-03-30T05:10:26.756 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure 0 2024-03-30T05:10:26.756 INFO:teuthology.orchestra.run.smithi123.stdout:last_failure_osd_epoch 82 2024-03-30T05:10:26.756 INFO:teuthology.orchestra.run.smithi123.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T05:10:26.756 INFO:teuthology.orchestra.run.smithi123.stdout:max_mds 2 2024-03-30T05:10:26.756 INFO:teuthology.orchestra.run.smithi123.stdout:in 0,1 2024-03-30T05:10:26.756 INFO:teuthology.orchestra.run.smithi123.stdout:up {0=34420,1=34452} 2024-03-30T05:10:26.756 INFO:teuthology.orchestra.run.smithi123.stdout:failed 2024-03-30T05:10:26.756 INFO:teuthology.orchestra.run.smithi123.stdout:damaged 2024-03-30T05:10:26.756 INFO:teuthology.orchestra.run.smithi123.stdout:stopped 2024-03-30T05:10:26.757 INFO:teuthology.orchestra.run.smithi123.stdout:data_pools [3] 2024-03-30T05:10:26.757 INFO:teuthology.orchestra.run.smithi123.stdout:metadata_pool 2 2024-03-30T05:10:26.757 INFO:teuthology.orchestra.run.smithi123.stdout:inline_data disabled 2024-03-30T05:10:26.757 INFO:teuthology.orchestra.run.smithi123.stdout:balancer 2024-03-30T05:10:26.757 INFO:teuthology.orchestra.run.smithi123.stdout:bal_rank_mask -1 2024-03-30T05:10:26.757 INFO:teuthology.orchestra.run.smithi123.stdout:standby_count_wanted 1 2024-03-30T05:10:26.757 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.tqxvzr{0:34420} state up:active seq 470 join_fscid=1 addr [v2:172.21.15.123:6826/1741131598,v1:172.21.15.123:6827/1741131598] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:10:26.757 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.zvnlap{0:44355} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.135:6824/978884711,v1:172.21.15.135:6825/978884711] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:10:26.757 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi123.dvtind{1:34452} state up:active seq 465 join_fscid=1 addr [v2:172.21.15.123:6828/4108919307,v1:172.21.15.123:6829/4108919307] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:10:26.757 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.cephfs.smithi135.xwgjvb{1:44975} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.135:6826/3788988014,v1:172.21.15.135:6827/3788988014] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T05:10:26.757 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:10:26.757 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:10:26.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:26.759+0000 7fb56f200700 1 -- 172.21.15.123:0/3573217832 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb55406cc30 msgr2=0x7fb55406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:26.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:26.759+0000 7fb56f200700 1 --2- 172.21.15.123:0/3573217832 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb55406cc30 0x7fb55406f0e0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fb56000b840 tx=0x7fb560009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:26.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:26.759+0000 7fb56f200700 1 -- 172.21.15.123:0/3573217832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5701035f0 msgr2=0x7fb570116050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:26.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:26.759+0000 7fb56f200700 1 --2- 172.21.15.123:0/3573217832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5701035f0 0x7fb570116050 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fb55c002a50 tx=0x7fb55c004600 comp rx=0 tx=0).stop 2024-03-30T05:10:26.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:26.759+0000 7fb56f200700 1 -- 172.21.15.123:0/3573217832 shutdown_connections 2024-03-30T05:10:26.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:26.759+0000 7fb56f200700 1 --2- 172.21.15.123:0/3573217832 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb55406cc30 0x7fb55406f0e0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:26.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:26.759+0000 7fb56f200700 1 --2- 172.21.15.123:0/3573217832 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb570102c50 0x7fb570115b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:26.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:26.759+0000 7fb56f200700 1 --2- 172.21.15.123:0/3573217832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5701035f0 0x7fb570116050 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:26.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:26.759+0000 7fb56f200700 1 -- 172.21.15.123:0/3573217832 >> 172.21.15.123:0/3573217832 conn(0x7fb5700f87e0 msgr2=0x7fb5700f97e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:26.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:26.759+0000 7fb56f200700 1 -- 172.21.15.123:0/3573217832 shutdown_connections 2024-03-30T05:10:26.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:26.759+0000 7fb56f200700 1 -- 172.21.15.123:0/3573217832 wait complete. 2024-03-30T05:10:26.759 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 42 2024-03-30T05:10:27.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:27 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3573217832' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T05:10:27.465 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:27 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3573217832' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T05:10:27.641 INFO:teuthology.run_tasks:Running task fs.post_upgrade_checks... 2024-03-30T05:10:27.652 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T05:10:27.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:27.849+0000 7fcce2c00700 1 -- 172.21.15.123:0/595478442 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccdc1036a0 msgr2=0x7fccdc0fcd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:27.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:27.849+0000 7fcce2c00700 1 --2- 172.21.15.123:0/595478442 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccdc1036a0 0x7fccdc0fcd10 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fccc8004940 tx=0x7fccc8009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:27.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:27.849+0000 7fcce2c00700 1 -- 172.21.15.123:0/595478442 shutdown_connections 2024-03-30T05:10:27.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:27.849+0000 7fcce2c00700 1 --2- 172.21.15.123:0/595478442 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccdc1036a0 0x7fccdc0fcd10 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:27.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:27.849+0000 7fcce2c00700 1 --2- 172.21.15.123:0/595478442 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fccdc102d00 0x7fccdc1030d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:27.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:27.849+0000 7fcce2c00700 1 -- 172.21.15.123:0/595478442 >> 172.21.15.123:0/595478442 conn(0x7fccdc0f8850 msgr2=0x7fccdc0fac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:27.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:27.849+0000 7fcce2c00700 1 -- 172.21.15.123:0/595478442 shutdown_connections 2024-03-30T05:10:27.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:27.849+0000 7fcce2c00700 1 -- 172.21.15.123:0/595478442 wait complete. 2024-03-30T05:10:27.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:27.850+0000 7fcce2c00700 1 Processor -- start 2024-03-30T05:10:27.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:27.850+0000 7fcce2c00700 1 -- start start 2024-03-30T05:10:27.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:27.850+0000 7fcce2c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccdc102d00 0x7fccdc115940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:27.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:27.850+0000 7fcce2c00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fccdc1036a0 0x7fccdc115e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:27.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:27.850+0000 7fcce2c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccdc1164d0 con 0x7fccdc102d00 2024-03-30T05:10:27.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:27.850+0000 7fcce2c00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccdc10fad0 con 0x7fccdc1036a0 2024-03-30T05:10:27.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:27.851+0000 7fccdbe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccdc102d00 0x7fccdc115940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:27.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:27.851+0000 7fccdbe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccdc102d00 0x7fccdc115940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41224/0 (socket says 172.21.15.123:41224) 2024-03-30T05:10:27.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:27.851+0000 7fccdbe00700 1 -- 172.21.15.123:0/3208285765 learned_addr learned my addr 172.21.15.123:0/3208285765 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:10:27.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:27.851+0000 7fccdb400700 1 --2- 172.21.15.123:0/3208285765 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fccdc1036a0 0x7fccdc115e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:27.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:27.851+0000 7fccdbe00700 1 -- 172.21.15.123:0/3208285765 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fccdc1036a0 msgr2=0x7fccdc115e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:27.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:27.851+0000 7fccdbe00700 1 --2- 172.21.15.123:0/3208285765 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fccdc1036a0 0x7fccdc115e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:27.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:27.851+0000 7fccdbe00700 1 -- 172.21.15.123:0/3208285765 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc8005040 con 0x7fccdc102d00 2024-03-30T05:10:27.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:27.851+0000 7fccdb400700 1 --2- 172.21.15.123:0/3208285765 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fccdc1036a0 0x7fccdc115e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T05:10:27.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:27.851+0000 7fccdbe00700 1 --2- 172.21.15.123:0/3208285765 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccdc102d00 0x7fccdc115940 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fcccc00eb40 tx=0x7fcccc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:27.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:27.852+0000 7fccd3e00700 1 -- 172.21.15.123:0/3208285765 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcccc007cc0 con 0x7fccdc102d00 2024-03-30T05:10:27.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:27.852+0000 7fcce2c00700 1 -- 172.21.15.123:0/3208285765 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccdc10fd50 con 0x7fccdc102d00 2024-03-30T05:10:27.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:27.852+0000 7fccd3e00700 1 -- 172.21.15.123:0/3208285765 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcccc00f040 con 0x7fccdc102d00 2024-03-30T05:10:27.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:27.852+0000 7fccd3e00700 1 -- 172.21.15.123:0/3208285765 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcccc009570 con 0x7fccdc102d00 2024-03-30T05:10:27.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:27.852+0000 7fcce2c00700 1 -- 172.21.15.123:0/3208285765 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccdc110310 con 0x7fccdc102d00 2024-03-30T05:10:27.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:27.853+0000 7fcce2c00700 1 -- 172.21.15.123:0/3208285765 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccdc0fe450 con 0x7fccdc102d00 2024-03-30T05:10:27.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:27.853+0000 7fccd3e00700 1 -- 172.21.15.123:0/3208285765 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fcccc0096d0 con 0x7fccdc102d00 2024-03-30T05:10:27.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:27.854+0000 7fccd3e00700 1 --2- 172.21.15.123:0/3208285765 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fccc006cc20 0x7fccc006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:27.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:27.855+0000 7fccd3e00700 1 -- 172.21.15.123:0/3208285765 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fcccc08bda0 con 0x7fccdc102d00 2024-03-30T05:10:27.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:27.855+0000 7fccdb400700 1 --2- 172.21.15.123:0/3208285765 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fccc006cc20 0x7fccc006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:27.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:27.855+0000 7fccdb400700 1 --2- 172.21.15.123:0/3208285765 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fccc006cc20 0x7fccc006f0d0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fccc8009fd0 tx=0x7fccc8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:27.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:27.859+0000 7fccd3e00700 1 -- 172.21.15.123:0/3208285765 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fcccc059820 con 0x7fccdc102d00 2024-03-30T05:10:28.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.065+0000 7fcce2c00700 1 -- 172.21.15.123:0/3208285765 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fccdc109ca0 con 0x7fccdc102d00 2024-03-30T05:10:28.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.066+0000 7fccd3e00700 1 -- 172.21.15.123:0/3208285765 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+16625 (secure 0 0 0) 0x7fcccc0593b0 con 0x7fccdc102d00 2024-03-30T05:10:28.063 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:10:28.064 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":86,"fsid":"2e77a2be-ee4c-11ee-b647-cb9ed24678a4","created":"2024-03-30T04:16:30.949442+0000","modified":"2024-03-30T04:53:15.872216+0000","last_up_change":"2024-03-30T04:37:14.509879+0000","last_in_change":"2024-03-30T04:22:44.504478+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T04:21:34.842970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-03-30T04:24:29.050323+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-03-30T04:24:29.704111+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3968446b-8696-4eec-b640-c97973ca054d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":49,"up_from":53,"up_thru":78,"down_at":50,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":1500077196},{"type":"v1","addr":"172.21.15.123:6803","nonce":1500077196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":1500077196},{"type":"v1","addr":"172.21.15.123:6805","nonce":1500077196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1500077196},{"type":"v1","addr":"172.21.15.123:6809","nonce":1500077196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":1500077196},{"type":"v1","addr":"172.21.15.123:6807","nonce":1500077196}]},"public_addr":"172.21.15.123:6803/1500077196","cluster_addr":"172.21.15.123:6805/1500077196","heartbeat_back_addr":"172.21.15.123:6809/1500077196","heartbeat_front_addr":"172.21.15.123:6807/1500077196","state":["exists","up"]},{"osd":1,"uuid":"336be34b-bb3c-46ab-8439-af04a1366db5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":54,"up_from":58,"up_thru":78,"down_at":55,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":4217392586},{"type":"v1","addr":"172.21.15.123:6811","nonce":4217392586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":4217392586},{"type":"v1","addr":"172.21.15.123:6813","nonce":4217392586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4217392586},{"type":"v1","addr":"172.21.15.123:6817","nonce":4217392586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":4217392586},{"type":"v1","addr":"172.21.15.123:6815","nonce":4217392586}]},"public_addr":"172.21.15.123:6811/4217392586","cluster_addr":"172.21.15.123:6813/4217392586","heartbeat_back_addr":"172.21.15.123:6817/4217392586","heartbeat_front_addr":"172.21.15.123:6815/4217392586","state":["exists","up"]},{"osd":2,"uuid":"4386fc8a-e9ed-413e-bd3a-bb84a0e62827","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":59,"up_from":63,"up_thru":78,"down_at":60,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":3356591260},{"type":"v1","addr":"172.21.15.123:6819","nonce":3356591260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":3356591260},{"type":"v1","addr":"172.21.15.123:6821","nonce":3356591260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":3356591260},{"type":"v1","addr":"172.21.15.123:6825","nonce":3356591260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":3356591260},{"type":"v1","addr":"172.21.15.123:6823","nonce":3356591260}]},"public_addr":"172.21.15.123:6819/3356591260","cluster_addr":"172.21.15.123:6821/3356591260","heartbeat_back_addr":"172.21.15.123:6825/3356591260","heartbeat_front_addr":"172.21.15.123:6823/3356591260","state":["exists","up"]},{"osd":3,"uuid":"086d86d4-ff1a-47dd-8416-447ccf9064ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":64,"up_from":68,"up_thru":78,"down_at":65,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":978771295},{"type":"v1","addr":"172.21.15.135:6801","nonce":978771295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":978771295},{"type":"v1","addr":"172.21.15.135:6803","nonce":978771295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":978771295},{"type":"v1","addr":"172.21.15.135:6807","nonce":978771295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":978771295},{"type":"v1","addr":"172.21.15.135:6805","nonce":978771295}]},"public_addr":"172.21.15.135:6801/978771295","cluster_addr":"172.21.15.135:6803/978771295","heartbeat_back_addr":"172.21.15.135:6807/978771295","heartbeat_front_addr":"172.21.15.135:6805/978771295","state":["exists","up"]},{"osd":4,"uuid":"31868cec-dd4d-44dd-a150-7ebdbd61ea4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":34,"last_clean_end":69,"up_from":73,"up_thru":78,"down_at":70,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":4133828133},{"type":"v1","addr":"172.21.15.135:6809","nonce":4133828133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":4133828133},{"type":"v1","addr":"172.21.15.135:6811","nonce":4133828133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":4133828133},{"type":"v1","addr":"172.21.15.135:6815","nonce":4133828133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":4133828133},{"type":"v1","addr":"172.21.15.135:6813","nonce":4133828133}]},"public_addr":"172.21.15.135:6809/4133828133","cluster_addr":"172.21.15.135:6811/4133828133","heartbeat_back_addr":"172.21.15.135:6815/4133828133","heartbeat_front_addr":"172.21.15.135:6813/4133828133","state":["exists","up"]},{"osd":5,"uuid":"dc6e11b9-4729-4504-8917-51a3bb5e2b7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":39,"last_clean_end":74,"up_from":78,"up_thru":78,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":1977885887},{"type":"v1","addr":"172.21.15.135:6817","nonce":1977885887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":1977885887},{"type":"v1","addr":"172.21.15.135:6819","nonce":1977885887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":1977885887},{"type":"v1","addr":"172.21.15.135:6823","nonce":1977885887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":1977885887},{"type":"v1","addr":"172.21.15.135:6821","nonce":1977885887}]},"public_addr":"172.21.15.135:6817/1977885887","cluster_addr":"172.21.15.135:6819/1977885887","heartbeat_back_addr":"172.21.15.135:6823/1977885887","heartbeat_front_addr":"172.21.15.135:6821/1977885887","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T04:20:25.138939+0000","dead_epoch":49},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T04:20:58.657719+0000","dead_epoch":54},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T04:21:32.397235+0000","dead_epoch":59},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T04:22:02.245820+0000","dead_epoch":64},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T04:22:33.842250+0000","dead_epoch":69},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T04:23:04.503576+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.123:6829/3037849622":"2024-03-31T04:37:19.516180+0000","172.21.15.123:6828/3037849622":"2024-03-31T04:37:19.516180+0000","172.21.15.123:0/1275706223":"2024-03-31T04:18:56.235750+0000","172.21.15.123:6827/2393193575":"2024-03-31T04:24:47.075751+0000","172.21.15.135:0/1604630130":"2024-03-31T04:28:49.824398+0000","172.21.15.123:0/3228440712":"2024-03-31T04:32:29.347901+0000","172.21.15.123:0/3037148222":"2024-03-31T04:17:41.459573+0000","172.21.15.123:0/2358069559":"2024-03-31T04:17:41.459573+0000","172.21.15.135:6828/3601553667":"2024-03-31T04:28:49.824398+0000","172.21.15.123:0/4218181674":"2024-03-31T04:26:59.096603+0000","172.21.15.123:0/2447404972":"2024-03-31T04:26:59.096603+0000","172.21.15.123:0/3593359364":"2024-03-31T04:18:56.235750+0000","172.21.15.135:6825/4050541062":"2024-03-31T04:37:19.516195+0000","172.21.15.135:0/1748591175":"2024-03-31T04:28:49.824398+0000","172.21.15.123:0/518566709":"2024-03-31T04:18:56.235750+0000","172.21.15.135:0/952436162":"2024-03-31T04:28:49.824398+0000","172.21.15.135:6829/3601553667":"2024-03-31T04:28:49.824398+0000","172.21.15.123:6800/1291592055":"2024-03-31T04:17:05.107043+0000","172.21.15.135:0/3366037413":"2024-03-31T04:28:49.824398+0000","172.21.15.123:6826/2393193575":"2024-03-31T04:24:47.075751+0000","172.21.15.123:6801/1776532690":"2024-03-31T04:17:41.459573+0000","172.21.15.123:0/1064275350":"2024-03-31T04:26:59.096603+0000","172.21.15.123:0/72410915":"2024-03-31T04:26:59.096603+0000","172.21.15.123:6800/2125774379":"2024-03-31T04:32:29.347901+0000","172.21.15.123:6801/1291592055":"2024-03-31T04:17:05.107043+0000","172.21.15.123:6800/1564935593":"2024-03-31T04:26:59.096603+0000","172.21.15.123:0/2756071515":"2024-03-31T04:17:05.107043+0000","172.21.15.123:0/133647565":"2024-03-31T04:17:41.459573+0000","172.21.15.123:6801/2125774379":"2024-03-31T04:32:29.347901+0000","172.21.15.123:6801/1564935593":"2024-03-31T04:26:59.096603+0000","172.21.15.123:6800/1776532690":"2024-03-31T04:17:41.459573+0000","172.21.15.123:6800/70490328":"2024-03-31T04:18:56.235750+0000","172.21.15.123:0/2366611723":"2024-03-31T04:17:05.107043+0000","172.21.15.123:6801/70490328":"2024-03-31T04:18:56.235750+0000","172.21.15.135:6824/4050541062":"2024-03-31T04:37:19.516195+0000","172.21.15.123:0/250978570":"2024-03-31T04:17:05.107043+0000","172.21.15.123:0/2743438158":"2024-03-31T04:32:29.347901+0000","172.21.15.123:0/1159868778":"2024-03-31T04:32:29.347901+0000","172.21.15.123:0/2182269825":"2024-03-31T04:32:29.347901+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T05:10:28.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.069+0000 7fcce2c00700 1 -- 172.21.15.123:0/3208285765 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fccc006cc20 msgr2=0x7fccc006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:28.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.069+0000 7fcce2c00700 1 --2- 172.21.15.123:0/3208285765 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fccc006cc20 0x7fccc006f0d0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fccc8009fd0 tx=0x7fccc8009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:28.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.069+0000 7fcce2c00700 1 -- 172.21.15.123:0/3208285765 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccdc102d00 msgr2=0x7fccdc115940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:28.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.069+0000 7fcce2c00700 1 --2- 172.21.15.123:0/3208285765 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccdc102d00 0x7fccdc115940 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fcccc00eb40 tx=0x7fcccc00b040 comp rx=0 tx=0).stop 2024-03-30T05:10:28.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.069+0000 7fcce2c00700 1 -- 172.21.15.123:0/3208285765 shutdown_connections 2024-03-30T05:10:28.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.069+0000 7fcce2c00700 1 --2- 172.21.15.123:0/3208285765 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fccc006cc20 0x7fccc006f0d0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:28.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.069+0000 7fcce2c00700 1 --2- 172.21.15.123:0/3208285765 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccdc102d00 0x7fccdc115940 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:28.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.069+0000 7fcce2c00700 1 --2- 172.21.15.123:0/3208285765 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fccdc1036a0 0x7fccdc115e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:28.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.069+0000 7fcce2c00700 1 -- 172.21.15.123:0/3208285765 >> 172.21.15.123:0/3208285765 conn(0x7fccdc0f8850 msgr2=0x7fccdc0f9770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:28.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.069+0000 7fcce2c00700 1 -- 172.21.15.123:0/3208285765 shutdown_connections 2024-03-30T05:10:28.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.069+0000 7fcce2c00700 1 -- 172.21.15.123:0/3208285765 wait complete. 2024-03-30T05:10:28.083 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T04:21:34.842970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-03-30T04:24:29.050323+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-03-30T04:24:29.704111+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-30T05:10:28.084 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-30T05:10:28.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.279+0000 7fb749000700 1 -- 172.21.15.123:0/2479121128 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb744102ce0 msgr2=0x7fb7441030b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:28.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.279+0000 7fb749000700 1 --2- 172.21.15.123:0/2479121128 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb744102ce0 0x7fb7441030b0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fb734004940 tx=0x7fb734009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:28.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.279+0000 7fb749000700 1 -- 172.21.15.123:0/2479121128 shutdown_connections 2024-03-30T05:10:28.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.279+0000 7fb749000700 1 --2- 172.21.15.123:0/2479121128 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb744103680 0x7fb7440fccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:28.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.279+0000 7fb749000700 1 --2- 172.21.15.123:0/2479121128 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb744102ce0 0x7fb7441030b0 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:28.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.279+0000 7fb749000700 1 -- 172.21.15.123:0/2479121128 >> 172.21.15.123:0/2479121128 conn(0x7fb7440f8830 msgr2=0x7fb7440fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:28.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.279+0000 7fb749000700 1 -- 172.21.15.123:0/2479121128 shutdown_connections 2024-03-30T05:10:28.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.279+0000 7fb749000700 1 -- 172.21.15.123:0/2479121128 wait complete. 2024-03-30T05:10:28.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.280+0000 7fb749000700 1 Processor -- start 2024-03-30T05:10:28.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.280+0000 7fb749000700 1 -- start start 2024-03-30T05:10:28.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.281+0000 7fb749000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb744102ce0 0x7fb7440702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:28.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.281+0000 7fb749000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb744103680 0x7fb744070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:28.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.281+0000 7fb749000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7440722a0 con 0x7fb744103680 2024-03-30T05:10:28.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.281+0000 7fb749000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb744070d40 con 0x7fb744102ce0 2024-03-30T05:10:28.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.281+0000 7fb741400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb744103680 0x7fb744070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:28.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.281+0000 7fb741400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb744103680 0x7fb744070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41240/0 (socket says 172.21.15.123:41240) 2024-03-30T05:10:28.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.281+0000 7fb741400700 1 -- 172.21.15.123:0/2992984371 learned_addr learned my addr 172.21.15.123:0/2992984371 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:10:28.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.281+0000 7fb741e00700 1 --2- 172.21.15.123:0/2992984371 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb744102ce0 0x7fb7440702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:28.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.281+0000 7fb741400700 1 -- 172.21.15.123:0/2992984371 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb744102ce0 msgr2=0x7fb7440702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:28.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.281+0000 7fb741400700 1 --2- 172.21.15.123:0/2992984371 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb744102ce0 0x7fb7440702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:28.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.281+0000 7fb741400700 1 -- 172.21.15.123:0/2992984371 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb734005040 con 0x7fb744103680 2024-03-30T05:10:28.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.282+0000 7fb741400700 1 --2- 172.21.15.123:0/2992984371 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb744103680 0x7fb744070800 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fb73000b790 tx=0x7fb730009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:28.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.282+0000 7fb73a000700 1 -- 172.21.15.123:0/2992984371 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb73000f6f0 con 0x7fb744103680 2024-03-30T05:10:28.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.282+0000 7fb749000700 1 -- 172.21.15.123:0/2992984371 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb744071020 con 0x7fb744103680 2024-03-30T05:10:28.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.282+0000 7fb73a000700 1 -- 172.21.15.123:0/2992984371 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb73000fd00 con 0x7fb744103680 2024-03-30T05:10:28.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.282+0000 7fb73a000700 1 -- 172.21.15.123:0/2992984371 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7300103a0 con 0x7fb744103680 2024-03-30T05:10:28.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.282+0000 7fb749000700 1 -- 172.21.15.123:0/2992984371 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb74411a010 con 0x7fb744103680 2024-03-30T05:10:28.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.283+0000 7fb749000700 1 -- 172.21.15.123:0/2992984371 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7440fe430 con 0x7fb744103680 2024-03-30T05:10:28.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.284+0000 7fb73a000700 1 -- 172.21.15.123:0/2992984371 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fb73000f850 con 0x7fb744103680 2024-03-30T05:10:28.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.285+0000 7fb73a000700 1 --2- 172.21.15.123:0/2992984371 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb72806cc20 0x7fb72806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:28.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.285+0000 7fb73a000700 1 -- 172.21.15.123:0/2992984371 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb73008bec0 con 0x7fb744103680 2024-03-30T05:10:28.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.285+0000 7fb741e00700 1 --2- 172.21.15.123:0/2992984371 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb72806cc20 0x7fb72806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:28.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.286+0000 7fb741e00700 1 --2- 172.21.15.123:0/2992984371 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb72806cc20 0x7fb72806f0d0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fb734009fd0 tx=0x7fb734009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:28.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.289+0000 7fb73a000700 1 -- 172.21.15.123:0/2992984371 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb730055e80 con 0x7fb744103680 2024-03-30T05:10:28.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:28 smithi123 ceph-mon[139369]: pgmap v1180: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:10:28.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:28 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3208285765' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T05:10:28.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.492+0000 7fb749000700 1 -- 172.21.15.123:0/2992984371 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb744109d10 con 0x7fb744103680 2024-03-30T05:10:28.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.493+0000 7fb73a000700 1 -- 172.21.15.123:0/2992984371 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v86) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb7300594a0 con 0x7fb744103680 2024-03-30T05:10:28.490 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 1 2024-03-30T05:10:28.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.496+0000 7fb749000700 1 -- 172.21.15.123:0/2992984371 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb72806cc20 msgr2=0x7fb72806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:28.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.496+0000 7fb749000700 1 --2- 172.21.15.123:0/2992984371 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb72806cc20 0x7fb72806f0d0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fb734009fd0 tx=0x7fb734009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:28.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.496+0000 7fb749000700 1 -- 172.21.15.123:0/2992984371 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb744103680 msgr2=0x7fb744070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:28.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.496+0000 7fb749000700 1 --2- 172.21.15.123:0/2992984371 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb744103680 0x7fb744070800 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fb73000b790 tx=0x7fb730009220 comp rx=0 tx=0).stop 2024-03-30T05:10:28.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.496+0000 7fb749000700 1 -- 172.21.15.123:0/2992984371 shutdown_connections 2024-03-30T05:10:28.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.496+0000 7fb749000700 1 --2- 172.21.15.123:0/2992984371 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb72806cc20 0x7fb72806f0d0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:28.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.496+0000 7fb749000700 1 --2- 172.21.15.123:0/2992984371 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb744102ce0 0x7fb7440702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:28.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.496+0000 7fb749000700 1 --2- 172.21.15.123:0/2992984371 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb744103680 0x7fb744070800 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:28.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.496+0000 7fb749000700 1 -- 172.21.15.123:0/2992984371 >> 172.21.15.123:0/2992984371 conn(0x7fb7440f8830 msgr2=0x7fb7440f9830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:28.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.496+0000 7fb749000700 1 -- 172.21.15.123:0/2992984371 shutdown_connections 2024-03-30T05:10:28.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.496+0000 7fb749000700 1 -- 172.21.15.123:0/2992984371 wait complete. 2024-03-30T05:10:28.511 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-03-30T05:10:28.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:28 smithi135 ceph-mon[106011]: pgmap v1180: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:10:28.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:28 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3208285765' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T05:10:28.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.705+0000 7f563b400700 1 -- 172.21.15.123:0/1511230370 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5634102ce0 msgr2=0x7f56341030b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:28.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.705+0000 7f563b400700 1 --2- 172.21.15.123:0/1511230370 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5634102ce0 0x7f56341030b0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f5628004940 tx=0x7f5628009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:28.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.705+0000 7f563b400700 1 -- 172.21.15.123:0/1511230370 shutdown_connections 2024-03-30T05:10:28.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.705+0000 7f563b400700 1 --2- 172.21.15.123:0/1511230370 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5634103680 0x7f56340fccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:28.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.705+0000 7f563b400700 1 --2- 172.21.15.123:0/1511230370 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5634102ce0 0x7f56341030b0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:28.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.705+0000 7f563b400700 1 -- 172.21.15.123:0/1511230370 >> 172.21.15.123:0/1511230370 conn(0x7f56340f8830 msgr2=0x7f56340fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:28.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.705+0000 7f563b400700 1 -- 172.21.15.123:0/1511230370 shutdown_connections 2024-03-30T05:10:28.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.705+0000 7f563b400700 1 -- 172.21.15.123:0/1511230370 wait complete. 2024-03-30T05:10:28.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.706+0000 7f563b400700 1 Processor -- start 2024-03-30T05:10:28.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.706+0000 7f563b400700 1 -- start start 2024-03-30T05:10:28.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.707+0000 7f563b400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5634102ce0 0x7f56340702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:28.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.707+0000 7f563b400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5634103680 0x7f5634070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:28.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.707+0000 7f563b400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5634072210 con 0x7f5634103680 2024-03-30T05:10:28.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.707+0000 7f563b400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5634070d40 con 0x7f5634102ce0 2024-03-30T05:10:28.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.707+0000 7f5633e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5634103680 0x7f5634070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:28.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.707+0000 7f5633e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5634103680 0x7f5634070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41250/0 (socket says 172.21.15.123:41250) 2024-03-30T05:10:28.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.707+0000 7f5633e00700 1 -- 172.21.15.123:0/2937657366 learned_addr learned my addr 172.21.15.123:0/2937657366 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:10:28.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.707+0000 7f5633e00700 1 -- 172.21.15.123:0/2937657366 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5634102ce0 msgr2=0x7f56340702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:28.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.707+0000 7f5638a00700 1 --2- 172.21.15.123:0/2937657366 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5634102ce0 0x7f56340702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:28.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.707+0000 7f5633e00700 1 --2- 172.21.15.123:0/2937657366 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5634102ce0 0x7f56340702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:28.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.707+0000 7f5633e00700 1 -- 172.21.15.123:0/2937657366 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5628005040 con 0x7f5634103680 2024-03-30T05:10:28.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.707+0000 7f5638a00700 1 --2- 172.21.15.123:0/2937657366 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5634102ce0 0x7f56340702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T05:10:28.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.707+0000 7f5633e00700 1 --2- 172.21.15.123:0/2937657366 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5634103680 0x7f5634070800 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f562000eb40 tx=0x7f562000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:28.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.708+0000 7f5630c00700 1 -- 172.21.15.123:0/2937657366 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5620007cc0 con 0x7f5634103680 2024-03-30T05:10:28.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.708+0000 7f563b400700 1 -- 172.21.15.123:0/2937657366 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5634070fc0 con 0x7f5634103680 2024-03-30T05:10:28.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.708+0000 7f5630c00700 1 -- 172.21.15.123:0/2937657366 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f562000f040 con 0x7f5634103680 2024-03-30T05:10:28.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.708+0000 7f5630c00700 1 -- 172.21.15.123:0/2937657366 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5620009570 con 0x7f5634103680 2024-03-30T05:10:28.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.708+0000 7f563b400700 1 -- 172.21.15.123:0/2937657366 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5634119ea0 con 0x7f5634103680 2024-03-30T05:10:28.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.709+0000 7f563b400700 1 -- 172.21.15.123:0/2937657366 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56340fe430 con 0x7f5634103680 2024-03-30T05:10:28.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.710+0000 7f5630c00700 1 -- 172.21.15.123:0/2937657366 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f56200096d0 con 0x7f5634103680 2024-03-30T05:10:28.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.711+0000 7f5630c00700 1 --2- 172.21.15.123:0/2937657366 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f561c06cc20 0x7f561c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:28.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.711+0000 7f5630c00700 1 -- 172.21.15.123:0/2937657366 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f562008ba90 con 0x7f5634103680 2024-03-30T05:10:28.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.711+0000 7f5638a00700 1 --2- 172.21.15.123:0/2937657366 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f561c06cc20 0x7f561c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:28.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.712+0000 7f5638a00700 1 --2- 172.21.15.123:0/2937657366 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f561c06cc20 0x7f561c06f0d0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f5628004560 tx=0x7f5628009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:28.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.715+0000 7f5630c00700 1 -- 172.21.15.123:0/2937657366 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5620055b80 con 0x7f5634103680 2024-03-30T05:10:28.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.917+0000 7f563b400700 1 -- 172.21.15.123:0/2937657366 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f5634064970 con 0x7f5634103680 2024-03-30T05:10:28.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.917+0000 7f5630c00700 1 -- 172.21.15.123:0/2937657366 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v86) v1 ==== 107+0+11 (secure 0 0 0) 0x7f5620019020 con 0x7f5634103680 2024-03-30T05:10:28.914 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 32 2024-03-30T05:10:28.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.920+0000 7f563b400700 1 -- 172.21.15.123:0/2937657366 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f561c06cc20 msgr2=0x7f561c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:28.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.920+0000 7f563b400700 1 --2- 172.21.15.123:0/2937657366 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f561c06cc20 0x7f561c06f0d0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f5628004560 tx=0x7f5628009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:28.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.920+0000 7f563b400700 1 -- 172.21.15.123:0/2937657366 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5634103680 msgr2=0x7f5634070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:28.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.920+0000 7f563b400700 1 --2- 172.21.15.123:0/2937657366 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5634103680 0x7f5634070800 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f562000eb40 tx=0x7f562000b040 comp rx=0 tx=0).stop 2024-03-30T05:10:28.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.920+0000 7f563b400700 1 -- 172.21.15.123:0/2937657366 shutdown_connections 2024-03-30T05:10:28.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.920+0000 7f563b400700 1 --2- 172.21.15.123:0/2937657366 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f561c06cc20 0x7f561c06f0d0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:28.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.920+0000 7f563b400700 1 --2- 172.21.15.123:0/2937657366 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5634102ce0 0x7f56340702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:28.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.920+0000 7f563b400700 1 --2- 172.21.15.123:0/2937657366 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5634103680 0x7f5634070800 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:28.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.920+0000 7f563b400700 1 -- 172.21.15.123:0/2937657366 >> 172.21.15.123:0/2937657366 conn(0x7f56340f8830 msgr2=0x7f56340f9880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:28.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.920+0000 7f563b400700 1 -- 172.21.15.123:0/2937657366 shutdown_connections 2024-03-30T05:10:28.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:28.920+0000 7f563b400700 1 -- 172.21.15.123:0/2937657366 wait complete. 2024-03-30T05:10:28.935 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-03-30T05:10:29.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.132+0000 7f3e6dc00700 1 -- 172.21.15.123:0/443531818 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e68103670 msgr2=0x7f3e680fcce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:29.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.132+0000 7f3e6dc00700 1 --2- 172.21.15.123:0/443531818 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e68103670 0x7f3e680fcce0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f3e54004940 tx=0x7f3e54009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:29.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.132+0000 7f3e6dc00700 1 -- 172.21.15.123:0/443531818 shutdown_connections 2024-03-30T05:10:29.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.132+0000 7f3e6dc00700 1 --2- 172.21.15.123:0/443531818 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e68103670 0x7f3e680fcce0 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:29.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.132+0000 7f3e6dc00700 1 --2- 172.21.15.123:0/443531818 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3e68102cd0 0x7f3e681030a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:29.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.132+0000 7f3e6dc00700 1 -- 172.21.15.123:0/443531818 >> 172.21.15.123:0/443531818 conn(0x7f3e680f8820 msgr2=0x7f3e680fac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:29.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.132+0000 7f3e6dc00700 1 -- 172.21.15.123:0/443531818 shutdown_connections 2024-03-30T05:10:29.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.132+0000 7f3e6dc00700 1 -- 172.21.15.123:0/443531818 wait complete. 2024-03-30T05:10:29.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.132+0000 7f3e6dc00700 1 Processor -- start 2024-03-30T05:10:29.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.133+0000 7f3e6dc00700 1 -- start start 2024-03-30T05:10:29.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.133+0000 7f3e6dc00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3e68102cd0 0x7f3e68115a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:29.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.133+0000 7f3e6dc00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e68103670 0x7f3e68115f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:29.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.133+0000 7f3e6dc00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e681165a0 con 0x7f3e68103670 2024-03-30T05:10:29.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.133+0000 7f3e6dc00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e6810fba0 con 0x7f3e68102cd0 2024-03-30T05:10:29.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.133+0000 7f3e66000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e68103670 0x7f3e68115f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:29.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.133+0000 7f3e66000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e68103670 0x7f3e68115f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41256/0 (socket says 172.21.15.123:41256) 2024-03-30T05:10:29.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.133+0000 7f3e66000700 1 -- 172.21.15.123:0/3099228947 learned_addr learned my addr 172.21.15.123:0/3099228947 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:10:29.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.133+0000 7f3e66000700 1 -- 172.21.15.123:0/3099228947 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3e68102cd0 msgr2=0x7f3e68115a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:29.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.134+0000 7f3e66a00700 1 --2- 172.21.15.123:0/3099228947 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3e68102cd0 0x7f3e68115a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:29.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.134+0000 7f3e66000700 1 --2- 172.21.15.123:0/3099228947 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3e68102cd0 0x7f3e68115a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:29.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.134+0000 7f3e66000700 1 -- 172.21.15.123:0/3099228947 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e54005040 con 0x7f3e68103670 2024-03-30T05:10:29.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.134+0000 7f3e66a00700 1 --2- 172.21.15.123:0/3099228947 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3e68102cd0 0x7f3e68115a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T05:10:29.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.134+0000 7f3e66000700 1 --2- 172.21.15.123:0/3099228947 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e68103670 0x7f3e68115f50 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f3e54009390 tx=0x7f3e54009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:29.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.134+0000 7f3e5ea00700 1 -- 172.21.15.123:0/3099228947 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e5400e040 con 0x7f3e68103670 2024-03-30T05:10:29.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.134+0000 7f3e5ea00700 1 -- 172.21.15.123:0/3099228947 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3e54013420 con 0x7f3e68103670 2024-03-30T05:10:29.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.134+0000 7f3e6dc00700 1 -- 172.21.15.123:0/3099228947 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e6810fe20 con 0x7f3e68103670 2024-03-30T05:10:29.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.134+0000 7f3e5ea00700 1 -- 172.21.15.123:0/3099228947 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e5401c630 con 0x7f3e68103670 2024-03-30T05:10:29.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.134+0000 7f3e6dc00700 1 -- 172.21.15.123:0/3099228947 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e68110360 con 0x7f3e68103670 2024-03-30T05:10:29.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.135+0000 7f3e6dc00700 1 -- 172.21.15.123:0/3099228947 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e680fe420 con 0x7f3e68103670 2024-03-30T05:10:29.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.136+0000 7f3e5ea00700 1 -- 172.21.15.123:0/3099228947 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f3e54013a00 con 0x7f3e68103670 2024-03-30T05:10:29.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.137+0000 7f3e5ea00700 1 --2- 172.21.15.123:0/3099228947 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3e4c06cc20 0x7f3e4c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:29.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.137+0000 7f3e5ea00700 1 -- 172.21.15.123:0/3099228947 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f3e5408d2d0 con 0x7f3e68103670 2024-03-30T05:10:29.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.137+0000 7f3e66a00700 1 --2- 172.21.15.123:0/3099228947 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3e4c06cc20 0x7f3e4c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:29.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.138+0000 7f3e66a00700 1 --2- 172.21.15.123:0/3099228947 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3e4c06cc20 0x7f3e4c06f0d0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f3e58007fd0 tx=0x7f3e58009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:29.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.142+0000 7f3e5ea00700 1 -- 172.21.15.123:0/3099228947 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3e5405ada0 con 0x7f3e68103670 2024-03-30T05:10:29.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:29 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2992984371' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T05:10:29.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:29 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2937657366' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-30T05:10:29.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.346+0000 7f3e6dc00700 1 -- 172.21.15.123:0/3099228947 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f3e68109c70 con 0x7f3e68103670 2024-03-30T05:10:29.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.346+0000 7f3e5ea00700 1 -- 172.21.15.123:0/3099228947 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v86) v1 ==== 107+0+11 (secure 0 0 0) 0x7f3e5405a930 con 0x7f3e68103670 2024-03-30T05:10:29.344 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 32 2024-03-30T05:10:29.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.349+0000 7f3e6dc00700 1 -- 172.21.15.123:0/3099228947 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3e4c06cc20 msgr2=0x7f3e4c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:29.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.349+0000 7f3e6dc00700 1 --2- 172.21.15.123:0/3099228947 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3e4c06cc20 0x7f3e4c06f0d0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f3e58007fd0 tx=0x7f3e58009040 comp rx=0 tx=0).stop 2024-03-30T05:10:29.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.349+0000 7f3e6dc00700 1 -- 172.21.15.123:0/3099228947 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e68103670 msgr2=0x7f3e68115f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:29.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.349+0000 7f3e6dc00700 1 --2- 172.21.15.123:0/3099228947 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e68103670 0x7f3e68115f50 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f3e54009390 tx=0x7f3e54009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:29.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.349+0000 7f3e6dc00700 1 -- 172.21.15.123:0/3099228947 shutdown_connections 2024-03-30T05:10:29.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.349+0000 7f3e6dc00700 1 --2- 172.21.15.123:0/3099228947 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f3e4c06cc20 0x7f3e4c06f0d0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:29.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.349+0000 7f3e6dc00700 1 --2- 172.21.15.123:0/3099228947 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3e68102cd0 0x7f3e68115a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:29.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.349+0000 7f3e6dc00700 1 --2- 172.21.15.123:0/3099228947 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e68103670 0x7f3e68115f50 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:29.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.349+0000 7f3e6dc00700 1 -- 172.21.15.123:0/3099228947 >> 172.21.15.123:0/3099228947 conn(0x7f3e680f8820 msgr2=0x7f3e680f9880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:29.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.349+0000 7f3e6dc00700 1 -- 172.21.15.123:0/3099228947 shutdown_connections 2024-03-30T05:10:29.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.349+0000 7f3e6dc00700 1 -- 172.21.15.123:0/3099228947 wait complete. 2024-03-30T05:10:29.364 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T05:10:29.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.561+0000 7f5fe2c00700 1 -- 172.21.15.123:0/3501527929 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fdc070c70 msgr2=0x7f5fdc109010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:29.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.561+0000 7f5fe2c00700 1 --2- 172.21.15.123:0/3501527929 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fdc070c70 0x7f5fdc109010 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f5fc8004970 tx=0x7f5fc8009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:29.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.561+0000 7f5fe2c00700 1 -- 172.21.15.123:0/3501527929 shutdown_connections 2024-03-30T05:10:29.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.561+0000 7f5fe2c00700 1 --2- 172.21.15.123:0/3501527929 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fdc070c70 0x7f5fdc109010 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:29.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.561+0000 7f5fe2c00700 1 --2- 172.21.15.123:0/3501527929 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5fdc070360 0x7f5fdc070730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:29.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.561+0000 7f5fe2c00700 1 -- 172.21.15.123:0/3501527929 >> 172.21.15.123:0/3501527929 conn(0x7f5fdc0f8830 msgr2=0x7f5fdc0fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:29.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.561+0000 7f5fe2c00700 1 -- 172.21.15.123:0/3501527929 shutdown_connections 2024-03-30T05:10:29.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.561+0000 7f5fe2c00700 1 -- 172.21.15.123:0/3501527929 wait complete. 2024-03-30T05:10:29.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.562+0000 7f5fe2c00700 1 Processor -- start 2024-03-30T05:10:29.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.562+0000 7f5fe2c00700 1 -- start start 2024-03-30T05:10:29.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.562+0000 7f5fe2c00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5fdc070360 0x7f5fdc115a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:29.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.562+0000 7f5fe2c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fdc070c70 0x7f5fdc115f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:29.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.563+0000 7f5fe2c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fdc1165b0 con 0x7f5fdc070c70 2024-03-30T05:10:29.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.563+0000 7f5fe2c00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fdc10fb90 con 0x7f5fdc070360 2024-03-30T05:10:29.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.563+0000 7f5fdb400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fdc070c70 0x7f5fdc115f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:29.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.563+0000 7f5fdbe00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5fdc070360 0x7f5fdc115a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:29.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.563+0000 7f5fdb400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fdc070c70 0x7f5fdc115f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41282/0 (socket says 172.21.15.123:41282) 2024-03-30T05:10:29.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.563+0000 7f5fdb400700 1 -- 172.21.15.123:0/2028365190 learned_addr learned my addr 172.21.15.123:0/2028365190 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:10:29.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.563+0000 7f5fdb400700 1 -- 172.21.15.123:0/2028365190 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5fdc070360 msgr2=0x7f5fdc115a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:29.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.563+0000 7f5fdb400700 1 --2- 172.21.15.123:0/2028365190 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5fdc070360 0x7f5fdc115a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:29.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.563+0000 7f5fdb400700 1 -- 172.21.15.123:0/2028365190 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fc8005040 con 0x7f5fdc070c70 2024-03-30T05:10:29.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.563+0000 7f5fdb400700 1 --2- 172.21.15.123:0/2028365190 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fdc070c70 0x7f5fdc115f60 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f5fc8004f50 tx=0x7f5fc8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:29.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.564+0000 7f5fd3e00700 1 -- 172.21.15.123:0/2028365190 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5fc800e040 con 0x7f5fdc070c70 2024-03-30T05:10:29.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.564+0000 7f5fd3e00700 1 -- 172.21.15.123:0/2028365190 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5fc8013420 con 0x7f5fdc070c70 2024-03-30T05:10:29.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.564+0000 7f5fe2c00700 1 -- 172.21.15.123:0/2028365190 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fdc10fe10 con 0x7f5fdc070c70 2024-03-30T05:10:29.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.564+0000 7f5fd3e00700 1 -- 172.21.15.123:0/2028365190 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5fc801c630 con 0x7f5fdc070c70 2024-03-30T05:10:29.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.564+0000 7f5fe2c00700 1 -- 172.21.15.123:0/2028365190 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fdc110350 con 0x7f5fdc070c70 2024-03-30T05:10:29.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.565+0000 7f5fe2c00700 1 -- 172.21.15.123:0/2028365190 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fdc071aa0 con 0x7f5fdc070c70 2024-03-30T05:10:29.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.566+0000 7f5fd3e00700 1 -- 172.21.15.123:0/2028365190 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f5fc8013590 con 0x7f5fdc070c70 2024-03-30T05:10:29.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.567+0000 7f5fd3e00700 1 --2- 172.21.15.123:0/2028365190 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5fc006cc20 0x7f5fc006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:29.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.567+0000 7f5fd3e00700 1 -- 172.21.15.123:0/2028365190 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5fc808d1c0 con 0x7f5fdc070c70 2024-03-30T05:10:29.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.567+0000 7f5fdbe00700 1 --2- 172.21.15.123:0/2028365190 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5fc006cc20 0x7f5fc006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:29.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.568+0000 7f5fdbe00700 1 --2- 172.21.15.123:0/2028365190 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5fc006cc20 0x7f5fc006f0d0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc004f40 tx=0x7f5fcc00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:29.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.571+0000 7f5fd3e00700 1 -- 172.21.15.123:0/2028365190 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5fc805ac90 con 0x7f5fdc070c70 2024-03-30T05:10:29.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:29 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2992984371' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T05:10:29.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:29 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2937657366' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-30T05:10:29.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.824+0000 7f5fe2c00700 1 -- 172.21.15.123:0/2028365190 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5fdc064970 con 0x7f5fdc070c70 2024-03-30T05:10:29.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.825+0000 7f5fd3e00700 1 -- 172.21.15.123:0/2028365190 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4870 (secure 0 0 0) 0x7f5fc805a820 con 0x7f5fdc070c70 2024-03-30T05:10:29.823 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:10:29.823 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":42,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T04:24:30.211769+0000","modified":"2024-03-30T05:08:34.514493+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34420,"mds_1":34452},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34420":{"gid":34420,"name":"cephfs.smithi123.tqxvzr","rank":0,"incarnation":36,"state":"up:active","state_seq":470,"addr":"172.21.15.123:6827/1741131598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1741131598},{"type":"v1","addr":"172.21.15.123:6827","nonce":1741131598}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_34452":{"gid":34452,"name":"cephfs.smithi123.dvtind","rank":1,"incarnation":36,"state":"up:active","state_seq":465,"addr":"172.21.15.123:6829/4108919307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":4108919307},{"type":"v1","addr":"172.21.15.123:6829","nonce":4108919307}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_44355":{"gid":44355,"name":"cephfs.smithi135.zvnlap","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.135:6825/978884711","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":978884711},{"type":"v1","addr":"172.21.15.135:6825","nonce":978884711}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_44975":{"gid":44975,"name":"cephfs.smithi135.xwgjvb","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.135:6827/3788988014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3788988014},{"type":"v1","addr":"172.21.15.135:6827","nonce":3788988014}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-30T05:10:29.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.827+0000 7f5fe2c00700 1 -- 172.21.15.123:0/2028365190 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5fc006cc20 msgr2=0x7f5fc006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:29.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.827+0000 7f5fe2c00700 1 --2- 172.21.15.123:0/2028365190 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5fc006cc20 0x7f5fc006f0d0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc004f40 tx=0x7f5fcc00b040 comp rx=0 tx=0).stop 2024-03-30T05:10:29.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.827+0000 7f5fe2c00700 1 -- 172.21.15.123:0/2028365190 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fdc070c70 msgr2=0x7f5fdc115f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:29.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.827+0000 7f5fe2c00700 1 --2- 172.21.15.123:0/2028365190 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fdc070c70 0x7f5fdc115f60 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f5fc8004f50 tx=0x7f5fc8009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:29.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.827+0000 7f5fe2c00700 1 -- 172.21.15.123:0/2028365190 shutdown_connections 2024-03-30T05:10:29.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.827+0000 7f5fe2c00700 1 --2- 172.21.15.123:0/2028365190 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f5fc006cc20 0x7f5fc006f0d0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:29.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.827+0000 7f5fe2c00700 1 --2- 172.21.15.123:0/2028365190 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5fdc070360 0x7f5fdc115a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:29.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.827+0000 7f5fe2c00700 1 --2- 172.21.15.123:0/2028365190 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fdc070c70 0x7f5fdc115f60 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:29.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.828+0000 7f5fe2c00700 1 -- 172.21.15.123:0/2028365190 >> 172.21.15.123:0/2028365190 conn(0x7f5fdc0f8830 msgr2=0x7f5fdc0f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:29.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.828+0000 7f5fe2c00700 1 -- 172.21.15.123:0/2028365190 shutdown_connections 2024-03-30T05:10:29.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:29.828+0000 7f5fe2c00700 1 -- 172.21.15.123:0/2028365190 wait complete. 2024-03-30T05:10:29.827 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 42 2024-03-30T05:10:29.844 DEBUG:tasks.fs:checking fs fscid=1,name=cephfs state = {'epoch': 19, 'max_mds': 2, 'flags': 50} 2024-03-30T05:10:29.844 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 20 2024-03-30T05:10:30.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.042+0000 7f1601000700 1 -- 172.21.15.123:0/1891894317 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15fc102cd0 msgr2=0x7f15fc1030a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:30.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.042+0000 7f1601000700 1 --2- 172.21.15.123:0/1891894317 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15fc102cd0 0x7f15fc1030a0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f15ec004940 tx=0x7f15ec009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:30.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.042+0000 7f1601000700 1 -- 172.21.15.123:0/1891894317 shutdown_connections 2024-03-30T05:10:30.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.042+0000 7f1601000700 1 --2- 172.21.15.123:0/1891894317 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f15fc103670 0x7f15fc0fcce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:30.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.042+0000 7f1601000700 1 --2- 172.21.15.123:0/1891894317 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15fc102cd0 0x7f15fc1030a0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:30.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.042+0000 7f1601000700 1 -- 172.21.15.123:0/1891894317 >> 172.21.15.123:0/1891894317 conn(0x7f15fc0f8820 msgr2=0x7f15fc0fac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:30.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.042+0000 7f1601000700 1 -- 172.21.15.123:0/1891894317 shutdown_connections 2024-03-30T05:10:30.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.042+0000 7f1601000700 1 -- 172.21.15.123:0/1891894317 wait complete. 2024-03-30T05:10:30.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.043+0000 7f1601000700 1 Processor -- start 2024-03-30T05:10:30.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.043+0000 7f1601000700 1 -- start start 2024-03-30T05:10:30.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.043+0000 7f1601000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f15fc102cd0 0x7f15fc0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:30.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.043+0000 7f1601000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15fc103670 0x7f15fc070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:30.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.043+0000 7f1601000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15fc072210 con 0x7f15fc103670 2024-03-30T05:10:30.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.043+0000 7f1601000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15fc070d40 con 0x7f15fc102cd0 2024-03-30T05:10:30.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.044+0000 7f15f9400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15fc103670 0x7f15fc070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:30.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.044+0000 7f15f9400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15fc103670 0x7f15fc070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41306/0 (socket says 172.21.15.123:41306) 2024-03-30T05:10:30.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.044+0000 7f15f9400700 1 -- 172.21.15.123:0/3102097442 learned_addr learned my addr 172.21.15.123:0/3102097442 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:10:30.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.044+0000 7f15f9e00700 1 --2- 172.21.15.123:0/3102097442 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f15fc102cd0 0x7f15fc0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:30.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.044+0000 7f15f9400700 1 -- 172.21.15.123:0/3102097442 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f15fc102cd0 msgr2=0x7f15fc0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:30.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.044+0000 7f15f9400700 1 --2- 172.21.15.123:0/3102097442 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f15fc102cd0 0x7f15fc0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:30.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.044+0000 7f15f9400700 1 -- 172.21.15.123:0/3102097442 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15ec005040 con 0x7f15fc103670 2024-03-30T05:10:30.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.044+0000 7f15f9e00700 1 --2- 172.21.15.123:0/3102097442 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f15fc102cd0 0x7f15fc0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T05:10:30.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.044+0000 7f15f9400700 1 --2- 172.21.15.123:0/3102097442 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15fc103670 0x7f15fc070800 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f15e800eb40 tx=0x7f15e800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:30.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.044+0000 7f15f2000700 1 -- 172.21.15.123:0/3102097442 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f15e8007cc0 con 0x7f15fc103670 2024-03-30T05:10:30.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.044+0000 7f15f2000700 1 -- 172.21.15.123:0/3102097442 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f15e800f040 con 0x7f15fc103670 2024-03-30T05:10:30.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.044+0000 7f1601000700 1 -- 172.21.15.123:0/3102097442 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15fc070fc0 con 0x7f15fc103670 2024-03-30T05:10:30.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.044+0000 7f15f2000700 1 -- 172.21.15.123:0/3102097442 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f15e8009570 con 0x7f15fc103670 2024-03-30T05:10:30.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.044+0000 7f1601000700 1 -- 172.21.15.123:0/3102097442 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15fc119e90 con 0x7f15fc103670 2024-03-30T05:10:30.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.045+0000 7f1601000700 1 -- 172.21.15.123:0/3102097442 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15fc0fe420 con 0x7f15fc103670 2024-03-30T05:10:30.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.047+0000 7f15f2000700 1 -- 172.21.15.123:0/3102097442 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f15e80096d0 con 0x7f15fc103670 2024-03-30T05:10:30.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.048+0000 7f15f2000700 1 --2- 172.21.15.123:0/3102097442 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f15e006cc20 0x7f15e006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:30.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.048+0000 7f15f2000700 1 -- 172.21.15.123:0/3102097442 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f15e808bc70 con 0x7f15fc103670 2024-03-30T05:10:30.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.049+0000 7f15f9e00700 1 --2- 172.21.15.123:0/3102097442 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f15e006cc20 0x7f15e006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:30.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.049+0000 7f15f9e00700 1 --2- 172.21.15.123:0/3102097442 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f15e006cc20 0x7f15e006f0d0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f15ec00b0d0 tx=0x7f15ec009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:30.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.052+0000 7f15f2000700 1 -- 172.21.15.123:0/3102097442 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f15e8055e20 con 0x7f15fc103670 2024-03-30T05:10:30.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.301+0000 7f1601000700 1 -- 172.21.15.123:0/3102097442 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 20, "format": "json"} v 0) v1 -- 0x7f15fc11a0e0 con 0x7f15fc103670 2024-03-30T05:10:30.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.302+0000 7f15f2000700 1 -- 172.21.15.123:0/3102097442 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 20, "format": "json"}]=0 dumped fsmap epoch 20 v42) v1 ==== 107+0+4869 (secure 0 0 0) 0x7f15e8059440 con 0x7f15fc103670 2024-03-30T05:10:30.300 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:10:30.300 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T04:24:30.211769+0000","modified":"2024-03-30T04:26:57.768733+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14488,"mds_1":24309},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14488":{"gid":14488,"name":"cephfs.smithi123.dvtind","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.123:6829/3037849622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3037849622},{"type":"v1","addr":"172.21.15.123:6829","nonce":3037849622}]},"join_fscid":1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14500":{"gid":14500,"name":"cephfs.smithi123.tqxvzr","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6827/2433675995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":2433675995},{"type":"v1","addr":"172.21.15.123:6827","nonce":2433675995}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24309":{"gid":24309,"name":"cephfs.smithi135.zvnlap","rank":1,"incarnation":7,"state":"up:active","state_seq":4,"addr":"172.21.15.135:6825/4050541062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":4050541062},{"type":"v1","addr":"172.21.15.135:6825","nonce":4050541062}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24319":{"gid":24319,"name":"cephfs.smithi135.xwgjvb","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.135:6827/1096986122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":1096986122},{"type":"v1","addr":"172.21.15.135:6827","nonce":1096986122}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-30T05:10:30.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.305+0000 7f1601000700 1 -- 172.21.15.123:0/3102097442 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f15e006cc20 msgr2=0x7f15e006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:30.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.305+0000 7f1601000700 1 --2- 172.21.15.123:0/3102097442 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f15e006cc20 0x7f15e006f0d0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f15ec00b0d0 tx=0x7f15ec009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:30.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.305+0000 7f1601000700 1 -- 172.21.15.123:0/3102097442 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15fc103670 msgr2=0x7f15fc070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:30.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.305+0000 7f1601000700 1 --2- 172.21.15.123:0/3102097442 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15fc103670 0x7f15fc070800 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f15e800eb40 tx=0x7f15e800b040 comp rx=0 tx=0).stop 2024-03-30T05:10:30.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.305+0000 7f1601000700 1 -- 172.21.15.123:0/3102097442 shutdown_connections 2024-03-30T05:10:30.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.305+0000 7f1601000700 1 --2- 172.21.15.123:0/3102097442 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f15e006cc20 0x7f15e006f0d0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:30.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.305+0000 7f1601000700 1 --2- 172.21.15.123:0/3102097442 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f15fc102cd0 0x7f15fc0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:30.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.305+0000 7f1601000700 1 --2- 172.21.15.123:0/3102097442 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15fc103670 0x7f15fc070800 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:30.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.305+0000 7f1601000700 1 -- 172.21.15.123:0/3102097442 >> 172.21.15.123:0/3102097442 conn(0x7f15fc0f8820 msgr2=0x7f15fc0f98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:30.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.305+0000 7f1601000700 1 -- 172.21.15.123:0/3102097442 shutdown_connections 2024-03-30T05:10:30.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.305+0000 7f1601000700 1 -- 172.21.15.123:0/3102097442 wait complete. 2024-03-30T05:10:30.304 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 20 2024-03-30T05:10:30.321 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 21 2024-03-30T05:10:30.379 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:30 smithi123 ceph-mon[139369]: pgmap v1181: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:10:30.379 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:30 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3099228947' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-30T05:10:30.379 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:10:30.379 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:30 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2028365190' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T05:10:30.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.510+0000 7f6767600700 1 -- 172.21.15.123:0/1187598757 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6760103670 msgr2=0x7f67600fcce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:30.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.510+0000 7f6767600700 1 --2- 172.21.15.123:0/1187598757 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6760103670 0x7f67600fcce0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f674c004940 tx=0x7f674c009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:30.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.510+0000 7f6767600700 1 -- 172.21.15.123:0/1187598757 shutdown_connections 2024-03-30T05:10:30.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.510+0000 7f6767600700 1 --2- 172.21.15.123:0/1187598757 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6760103670 0x7f67600fcce0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:30.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.510+0000 7f6767600700 1 --2- 172.21.15.123:0/1187598757 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6760102cd0 0x7f67601030a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:30.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.510+0000 7f6767600700 1 -- 172.21.15.123:0/1187598757 >> 172.21.15.123:0/1187598757 conn(0x7f67600f8820 msgr2=0x7f67600fac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:30.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.510+0000 7f6767600700 1 -- 172.21.15.123:0/1187598757 shutdown_connections 2024-03-30T05:10:30.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.511+0000 7f6767600700 1 -- 172.21.15.123:0/1187598757 wait complete. 2024-03-30T05:10:30.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.511+0000 7f6767600700 1 Processor -- start 2024-03-30T05:10:30.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.511+0000 7f6767600700 1 -- start start 2024-03-30T05:10:30.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.512+0000 7f6767600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6760102cd0 0x7f6760115970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:30.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.512+0000 7f6767600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6760103670 0x7f6760115eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:30.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.512+0000 7f6767600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6760116500 con 0x7f6760103670 2024-03-30T05:10:30.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.512+0000 7f6767600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f676010fb00 con 0x7f6760102cd0 2024-03-30T05:10:30.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.512+0000 7f675fe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6760103670 0x7f6760115eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:30.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.512+0000 7f675fe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6760103670 0x7f6760115eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41336/0 (socket says 172.21.15.123:41336) 2024-03-30T05:10:30.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.512+0000 7f675fe00700 1 -- 172.21.15.123:0/4053963880 learned_addr learned my addr 172.21.15.123:0/4053963880 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:10:30.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.512+0000 7f6764c00700 1 --2- 172.21.15.123:0/4053963880 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6760102cd0 0x7f6760115970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:30.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.512+0000 7f675fe00700 1 -- 172.21.15.123:0/4053963880 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6760102cd0 msgr2=0x7f6760115970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:30.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.512+0000 7f675fe00700 1 --2- 172.21.15.123:0/4053963880 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6760102cd0 0x7f6760115970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:30.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.512+0000 7f675fe00700 1 -- 172.21.15.123:0/4053963880 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f674c005040 con 0x7f6760103670 2024-03-30T05:10:30.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.512+0000 7f675fe00700 1 --2- 172.21.15.123:0/4053963880 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6760103670 0x7f6760115eb0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f674c0093b0 tx=0x7f674c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:30.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.513+0000 7f675cc00700 1 -- 172.21.15.123:0/4053963880 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f674c00e040 con 0x7f6760103670 2024-03-30T05:10:30.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.513+0000 7f675cc00700 1 -- 172.21.15.123:0/4053963880 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f674c013420 con 0x7f6760103670 2024-03-30T05:10:30.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.513+0000 7f675cc00700 1 -- 172.21.15.123:0/4053963880 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f674c01c630 con 0x7f6760103670 2024-03-30T05:10:30.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.513+0000 7f6767600700 1 -- 172.21.15.123:0/4053963880 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f676010fd80 con 0x7f6760103670 2024-03-30T05:10:30.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.513+0000 7f6767600700 1 -- 172.21.15.123:0/4053963880 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6760110340 con 0x7f6760103670 2024-03-30T05:10:30.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.514+0000 7f6767600700 1 -- 172.21.15.123:0/4053963880 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67600fe420 con 0x7f6760103670 2024-03-30T05:10:30.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.515+0000 7f675cc00700 1 -- 172.21.15.123:0/4053963880 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f674c0139e0 con 0x7f6760103670 2024-03-30T05:10:30.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.516+0000 7f675cc00700 1 --2- 172.21.15.123:0/4053963880 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f674806cc20 0x7f674806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:30.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.516+0000 7f675cc00700 1 -- 172.21.15.123:0/4053963880 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f674c08d300 con 0x7f6760103670 2024-03-30T05:10:30.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.516+0000 7f6764c00700 1 --2- 172.21.15.123:0/4053963880 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f674806cc20 0x7f674806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:30.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.516+0000 7f6764c00700 1 --2- 172.21.15.123:0/4053963880 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f674806cc20 0x7f674806f0d0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f6754007fd0 tx=0x7f6754009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:30.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.520+0000 7f675cc00700 1 -- 172.21.15.123:0/4053963880 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f674c05ae00 con 0x7f6760103670 2024-03-30T05:10:30.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:30 smithi135 ceph-mon[106011]: pgmap v1181: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:10:30.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:30 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3099228947' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-30T05:10:30.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:30 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:10:30.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:30 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2028365190' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T05:10:30.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.769+0000 7f6767600700 1 -- 172.21.15.123:0/4053963880 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 21, "format": "json"} v 0) v1 -- 0x7f6760064970 con 0x7f6760103670 2024-03-30T05:10:30.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.769+0000 7f675cc00700 1 -- 172.21.15.123:0/4053963880 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 21, "format": "json"}]=0 dumped fsmap epoch 21 v42) v1 ==== 107+0+4870 (secure 0 0 0) 0x7f674c05a990 con 0x7f6760103670 2024-03-30T05:10:30.768 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:10:30.768 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T04:24:30.211769+0000","modified":"2024-03-30T04:27:53.205465+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14488,"mds_1":24309},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14488":{"gid":14488,"name":"cephfs.smithi123.dvtind","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.123:6829/3037849622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3037849622},{"type":"v1","addr":"172.21.15.123:6829","nonce":3037849622}]},"join_fscid":1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14500":{"gid":14500,"name":"cephfs.smithi123.tqxvzr","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6827/2433675995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":2433675995},{"type":"v1","addr":"172.21.15.123:6827","nonce":2433675995}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24309":{"gid":24309,"name":"cephfs.smithi135.zvnlap","rank":1,"incarnation":7,"state":"up:active","state_seq":4,"addr":"172.21.15.135:6825/4050541062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":4050541062},{"type":"v1","addr":"172.21.15.135:6825","nonce":4050541062}]},"join_fscid":1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24319":{"gid":24319,"name":"cephfs.smithi135.xwgjvb","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.135:6827/1096986122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":1096986122},{"type":"v1","addr":"172.21.15.135:6827","nonce":1096986122}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-30T05:10:30.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.772+0000 7f6767600700 1 -- 172.21.15.123:0/4053963880 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f674806cc20 msgr2=0x7f674806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:30.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.772+0000 7f6767600700 1 --2- 172.21.15.123:0/4053963880 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f674806cc20 0x7f674806f0d0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f6754007fd0 tx=0x7f6754009040 comp rx=0 tx=0).stop 2024-03-30T05:10:30.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.772+0000 7f6767600700 1 -- 172.21.15.123:0/4053963880 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6760103670 msgr2=0x7f6760115eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:30.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.772+0000 7f6767600700 1 --2- 172.21.15.123:0/4053963880 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6760103670 0x7f6760115eb0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f674c0093b0 tx=0x7f674c009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:30.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.772+0000 7f6767600700 1 -- 172.21.15.123:0/4053963880 shutdown_connections 2024-03-30T05:10:30.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.772+0000 7f6767600700 1 --2- 172.21.15.123:0/4053963880 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f674806cc20 0x7f674806f0d0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:30.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.772+0000 7f6767600700 1 --2- 172.21.15.123:0/4053963880 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6760102cd0 0x7f6760115970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:30.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.772+0000 7f6767600700 1 --2- 172.21.15.123:0/4053963880 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6760103670 0x7f6760115eb0 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:30.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.772+0000 7f6767600700 1 -- 172.21.15.123:0/4053963880 >> 172.21.15.123:0/4053963880 conn(0x7f67600f8820 msgr2=0x7f67600f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:30.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.772+0000 7f6767600700 1 -- 172.21.15.123:0/4053963880 shutdown_connections 2024-03-30T05:10:30.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.772+0000 7f6767600700 1 -- 172.21.15.123:0/4053963880 wait complete. 2024-03-30T05:10:30.772 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 21 2024-03-30T05:10:30.788 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 22 2024-03-30T05:10:30.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.978+0000 7eff37000700 1 -- 172.21.15.123:0/1056288587 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff300702e0 msgr2=0x7eff300706b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:30.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.978+0000 7eff37000700 1 --2- 172.21.15.123:0/1056288587 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff300702e0 0x7eff300706b0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7eff20004940 tx=0x7eff20009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:30.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.978+0000 7eff37000700 1 -- 172.21.15.123:0/1056288587 shutdown_connections 2024-03-30T05:10:30.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.978+0000 7eff37000700 1 --2- 172.21.15.123:0/1056288587 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7eff30070bf0 0x7eff30109060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:30.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.978+0000 7eff37000700 1 --2- 172.21.15.123:0/1056288587 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff300702e0 0x7eff300706b0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:30.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.978+0000 7eff37000700 1 -- 172.21.15.123:0/1056288587 >> 172.21.15.123:0/1056288587 conn(0x7eff300f8840 msgr2=0x7eff300fac50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:30.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.978+0000 7eff37000700 1 -- 172.21.15.123:0/1056288587 shutdown_connections 2024-03-30T05:10:30.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.978+0000 7eff37000700 1 -- 172.21.15.123:0/1056288587 wait complete. 2024-03-30T05:10:30.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.979+0000 7eff37000700 1 Processor -- start 2024-03-30T05:10:30.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.979+0000 7eff37000700 1 -- start start 2024-03-30T05:10:30.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.980+0000 7eff37000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff300702e0 0x7eff30115980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:30.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.980+0000 7eff37000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7eff30070bf0 0x7eff30115ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:30.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.980+0000 7eff37000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff30116510 con 0x7eff300702e0 2024-03-30T05:10:30.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.980+0000 7eff37000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff3010fb30 con 0x7eff30070bf0 2024-03-30T05:10:30.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.980+0000 7eff2fe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff300702e0 0x7eff30115980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:30.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.980+0000 7eff2fe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff300702e0 0x7eff30115980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41342/0 (socket says 172.21.15.123:41342) 2024-03-30T05:10:30.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.980+0000 7eff2fe00700 1 -- 172.21.15.123:0/3387525433 learned_addr learned my addr 172.21.15.123:0/3387525433 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:10:30.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.980+0000 7eff2f400700 1 --2- 172.21.15.123:0/3387525433 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7eff30070bf0 0x7eff30115ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:30.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.980+0000 7eff2fe00700 1 -- 172.21.15.123:0/3387525433 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7eff30070bf0 msgr2=0x7eff30115ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:30.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.980+0000 7eff2fe00700 1 --2- 172.21.15.123:0/3387525433 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7eff30070bf0 0x7eff30115ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:30.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.980+0000 7eff2fe00700 1 -- 172.21.15.123:0/3387525433 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff20005040 con 0x7eff300702e0 2024-03-30T05:10:30.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.980+0000 7eff2fe00700 1 --2- 172.21.15.123:0/3387525433 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff300702e0 0x7eff30115980 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7eff20009390 tx=0x7eff20009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:30.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.980+0000 7eff27e00700 1 -- 172.21.15.123:0/3387525433 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff2000e040 con 0x7eff300702e0 2024-03-30T05:10:30.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.981+0000 7eff27e00700 1 -- 172.21.15.123:0/3387525433 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7eff20013420 con 0x7eff300702e0 2024-03-30T05:10:30.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.981+0000 7eff37000700 1 -- 172.21.15.123:0/3387525433 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff3010fdb0 con 0x7eff300702e0 2024-03-30T05:10:30.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.981+0000 7eff27e00700 1 -- 172.21.15.123:0/3387525433 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff2001c630 con 0x7eff300702e0 2024-03-30T05:10:30.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.981+0000 7eff37000700 1 -- 172.21.15.123:0/3387525433 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff301102f0 con 0x7eff300702e0 2024-03-30T05:10:30.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.982+0000 7eff37000700 1 -- 172.21.15.123:0/3387525433 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff30071a20 con 0x7eff300702e0 2024-03-30T05:10:30.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.982+0000 7eff27e00700 1 -- 172.21.15.123:0/3387525433 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7eff200139e0 con 0x7eff300702e0 2024-03-30T05:10:30.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.983+0000 7eff27e00700 1 --2- 172.21.15.123:0/3387525433 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7eff1406cc20 0x7eff1406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:30.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.983+0000 7eff27e00700 1 -- 172.21.15.123:0/3387525433 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7eff2008d320 con 0x7eff300702e0 2024-03-30T05:10:30.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.984+0000 7eff2f400700 1 --2- 172.21.15.123:0/3387525433 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7eff1406cc20 0x7eff1406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:30.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.984+0000 7eff2f400700 1 --2- 172.21.15.123:0/3387525433 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7eff1406cc20 0x7eff1406f0d0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7eff30111220 tx=0x7eff1c009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:30.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:30.988+0000 7eff27e00700 1 -- 172.21.15.123:0/3387525433 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7eff20057390 con 0x7eff300702e0 2024-03-30T05:10:31.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.237+0000 7eff37000700 1 -- 172.21.15.123:0/3387525433 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 22, "format": "json"} v 0) v1 -- 0x7eff30064dc0 con 0x7eff300702e0 2024-03-30T05:10:31.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.238+0000 7eff27e00700 1 -- 172.21.15.123:0/3387525433 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 22, "format": "json"}]=0 dumped fsmap epoch 22 v42) v1 ==== 107+0+4869 (secure 0 0 0) 0x7eff2005a9b0 con 0x7eff300702e0 2024-03-30T05:10:31.236 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:10:31.236 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T04:24:30.211769+0000","modified":"2024-03-30T04:31:23.532125+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14488,"mds_1":24309},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14488":{"gid":14488,"name":"cephfs.smithi123.dvtind","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.123:6829/3037849622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3037849622},{"type":"v1","addr":"172.21.15.123:6829","nonce":3037849622}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14500":{"gid":14500,"name":"cephfs.smithi123.tqxvzr","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6827/2433675995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":2433675995},{"type":"v1","addr":"172.21.15.123:6827","nonce":2433675995}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24309":{"gid":24309,"name":"cephfs.smithi135.zvnlap","rank":1,"incarnation":7,"state":"up:active","state_seq":4,"addr":"172.21.15.135:6825/4050541062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":4050541062},{"type":"v1","addr":"172.21.15.135:6825","nonce":4050541062}]},"join_fscid":1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24319":{"gid":24319,"name":"cephfs.smithi135.xwgjvb","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.135:6827/1096986122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":1096986122},{"type":"v1","addr":"172.21.15.135:6827","nonce":1096986122}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-30T05:10:31.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.240+0000 7eff37000700 1 -- 172.21.15.123:0/3387525433 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7eff1406cc20 msgr2=0x7eff1406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:31.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.240+0000 7eff37000700 1 --2- 172.21.15.123:0/3387525433 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7eff1406cc20 0x7eff1406f0d0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7eff30111220 tx=0x7eff1c009ee0 comp rx=0 tx=0).stop 2024-03-30T05:10:31.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.240+0000 7eff37000700 1 -- 172.21.15.123:0/3387525433 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff300702e0 msgr2=0x7eff30115980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:31.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.240+0000 7eff37000700 1 --2- 172.21.15.123:0/3387525433 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff300702e0 0x7eff30115980 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7eff20009390 tx=0x7eff20009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:31.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.240+0000 7eff37000700 1 -- 172.21.15.123:0/3387525433 shutdown_connections 2024-03-30T05:10:31.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.240+0000 7eff37000700 1 --2- 172.21.15.123:0/3387525433 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7eff1406cc20 0x7eff1406f0d0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:31.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.240+0000 7eff37000700 1 --2- 172.21.15.123:0/3387525433 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff300702e0 0x7eff30115980 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:31.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.240+0000 7eff37000700 1 --2- 172.21.15.123:0/3387525433 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7eff30070bf0 0x7eff30115ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:31.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.240+0000 7eff37000700 1 -- 172.21.15.123:0/3387525433 >> 172.21.15.123:0/3387525433 conn(0x7eff300f8840 msgr2=0x7eff300f9d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:31.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.240+0000 7eff37000700 1 -- 172.21.15.123:0/3387525433 shutdown_connections 2024-03-30T05:10:31.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.240+0000 7eff37000700 1 -- 172.21.15.123:0/3387525433 wait complete. 2024-03-30T05:10:31.240 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 22 2024-03-30T05:10:31.256 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 23 2024-03-30T05:10:31.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.449+0000 7f51e4a00700 1 -- 172.21.15.123:0/917826340 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51e0102ce0 msgr2=0x7f51e01030b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:31.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.449+0000 7f51e4a00700 1 --2- 172.21.15.123:0/917826340 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51e0102ce0 0x7f51e01030b0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f51d0004940 tx=0x7f51d0009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:31.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.449+0000 7f51e4a00700 1 -- 172.21.15.123:0/917826340 shutdown_connections 2024-03-30T05:10:31.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.449+0000 7f51e4a00700 1 --2- 172.21.15.123:0/917826340 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f51e0103680 0x7f51e00fccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:31.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.449+0000 7f51e4a00700 1 --2- 172.21.15.123:0/917826340 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51e0102ce0 0x7f51e01030b0 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:31.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.449+0000 7f51e4a00700 1 -- 172.21.15.123:0/917826340 >> 172.21.15.123:0/917826340 conn(0x7f51e00f8830 msgr2=0x7f51e00fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:31.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.450+0000 7f51e4a00700 1 -- 172.21.15.123:0/917826340 shutdown_connections 2024-03-30T05:10:31.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.450+0000 7f51e4a00700 1 -- 172.21.15.123:0/917826340 wait complete. 2024-03-30T05:10:31.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.450+0000 7f51e4a00700 1 Processor -- start 2024-03-30T05:10:31.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.450+0000 7f51e4a00700 1 -- start start 2024-03-30T05:10:31.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.451+0000 7f51e4a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51e0102ce0 0x7f51e00702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:31.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.451+0000 7f51e4a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f51e0103680 0x7f51e0070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:31.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.451+0000 7f51e4a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51e0072210 con 0x7f51e0102ce0 2024-03-30T05:10:31.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.451+0000 7f51e4a00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51e0070d40 con 0x7f51e0103680 2024-03-30T05:10:31.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.451+0000 7f51dde00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51e0102ce0 0x7f51e00702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:31.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.451+0000 7f51dde00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51e0102ce0 0x7f51e00702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41360/0 (socket says 172.21.15.123:41360) 2024-03-30T05:10:31.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.451+0000 7f51dde00700 1 -- 172.21.15.123:0/3959052474 learned_addr learned my addr 172.21.15.123:0/3959052474 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:10:31.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.451+0000 7f51dd400700 1 --2- 172.21.15.123:0/3959052474 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f51e0103680 0x7f51e0070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:31.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.451+0000 7f51dde00700 1 -- 172.21.15.123:0/3959052474 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f51e0103680 msgr2=0x7f51e0070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:31.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.451+0000 7f51dde00700 1 --2- 172.21.15.123:0/3959052474 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f51e0103680 0x7f51e0070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:31.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.451+0000 7f51dde00700 1 -- 172.21.15.123:0/3959052474 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d0005040 con 0x7f51e0102ce0 2024-03-30T05:10:31.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.452+0000 7f51dde00700 1 --2- 172.21.15.123:0/3959052474 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51e0102ce0 0x7f51e00702c0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f51d0009390 tx=0x7f51d0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:31.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.452+0000 7f51d6000700 1 -- 172.21.15.123:0/3959052474 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f51d000e040 con 0x7f51e0102ce0 2024-03-30T05:10:31.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.452+0000 7f51d6000700 1 -- 172.21.15.123:0/3959052474 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f51d0013420 con 0x7f51e0102ce0 2024-03-30T05:10:31.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.452+0000 7f51e4a00700 1 -- 172.21.15.123:0/3959052474 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51e0070fc0 con 0x7f51e0102ce0 2024-03-30T05:10:31.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.452+0000 7f51d6000700 1 -- 172.21.15.123:0/3959052474 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f51d001c630 con 0x7f51e0102ce0 2024-03-30T05:10:31.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.452+0000 7f51e4a00700 1 -- 172.21.15.123:0/3959052474 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51e0119f60 con 0x7f51e0102ce0 2024-03-30T05:10:31.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.453+0000 7f51e4a00700 1 -- 172.21.15.123:0/3959052474 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51e00fe430 con 0x7f51e0102ce0 2024-03-30T05:10:31.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.454+0000 7f51d6000700 1 -- 172.21.15.123:0/3959052474 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f51d00139e0 con 0x7f51e0102ce0 2024-03-30T05:10:31.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.455+0000 7f51d6000700 1 --2- 172.21.15.123:0/3959052474 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f51c406cc20 0x7f51c406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:31.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.455+0000 7f51d6000700 1 -- 172.21.15.123:0/3959052474 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f51d008d310 con 0x7f51e0102ce0 2024-03-30T05:10:31.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.455+0000 7f51dd400700 1 --2- 172.21.15.123:0/3959052474 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f51c406cc20 0x7f51c406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:31.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.456+0000 7f51dd400700 1 --2- 172.21.15.123:0/3959052474 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f51c406cc20 0x7f51c406f0d0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f51cc007fd0 tx=0x7f51cc009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:31.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.459+0000 7f51d6000700 1 -- 172.21.15.123:0/3959052474 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f51d00572d0 con 0x7f51e0102ce0 2024-03-30T05:10:31.465 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:31 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3102097442' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 20, "format": "json"}]: dispatch 2024-03-30T05:10:31.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:31 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/4053963880' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 21, "format": "json"}]: dispatch 2024-03-30T05:10:31.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:31 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3102097442' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 20, "format": "json"}]: dispatch 2024-03-30T05:10:31.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:31 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/4053963880' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 21, "format": "json"}]: dispatch 2024-03-30T05:10:31.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.706+0000 7f51e4a00700 1 -- 172.21.15.123:0/3959052474 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 23, "format": "json"} v 0) v1 -- 0x7f51e0064970 con 0x7f51e0102ce0 2024-03-30T05:10:31.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.707+0000 7f51d6000700 1 -- 172.21.15.123:0/3959052474 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 23, "format": "json"}]=0 dumped fsmap epoch 23 v42) v1 ==== 107+0+4868 (secure 0 0 0) 0x7f51d005a8f0 con 0x7f51e0102ce0 2024-03-30T05:10:31.705 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:10:31.705 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T04:24:30.211769+0000","modified":"2024-03-30T04:32:48.448604+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14488,"mds_1":24309},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14488":{"gid":14488,"name":"cephfs.smithi123.dvtind","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.123:6829/3037849622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3037849622},{"type":"v1","addr":"172.21.15.123:6829","nonce":3037849622}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14500":{"gid":14500,"name":"cephfs.smithi123.tqxvzr","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.123:6827/2433675995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":2433675995},{"type":"v1","addr":"172.21.15.123:6827","nonce":2433675995}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24309":{"gid":24309,"name":"cephfs.smithi135.zvnlap","rank":1,"incarnation":7,"state":"up:active","state_seq":4,"addr":"172.21.15.135:6825/4050541062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":4050541062},{"type":"v1","addr":"172.21.15.135:6825","nonce":4050541062}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24319":{"gid":24319,"name":"cephfs.smithi135.xwgjvb","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.135:6827/1096986122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":1096986122},{"type":"v1","addr":"172.21.15.135:6827","nonce":1096986122}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-30T05:10:31.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.709+0000 7f51e4a00700 1 -- 172.21.15.123:0/3959052474 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f51c406cc20 msgr2=0x7f51c406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:31.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.709+0000 7f51e4a00700 1 --2- 172.21.15.123:0/3959052474 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f51c406cc20 0x7f51c406f0d0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f51cc007fd0 tx=0x7f51cc009040 comp rx=0 tx=0).stop 2024-03-30T05:10:31.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.709+0000 7f51e4a00700 1 -- 172.21.15.123:0/3959052474 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51e0102ce0 msgr2=0x7f51e00702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:31.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.709+0000 7f51e4a00700 1 --2- 172.21.15.123:0/3959052474 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51e0102ce0 0x7f51e00702c0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f51d0009390 tx=0x7f51d0009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:31.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.709+0000 7f51e4a00700 1 -- 172.21.15.123:0/3959052474 shutdown_connections 2024-03-30T05:10:31.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.709+0000 7f51e4a00700 1 --2- 172.21.15.123:0/3959052474 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f51c406cc20 0x7f51c406f0d0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:31.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.709+0000 7f51e4a00700 1 --2- 172.21.15.123:0/3959052474 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51e0102ce0 0x7f51e00702c0 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:31.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.709+0000 7f51e4a00700 1 --2- 172.21.15.123:0/3959052474 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f51e0103680 0x7f51e0070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:31.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.709+0000 7f51e4a00700 1 -- 172.21.15.123:0/3959052474 >> 172.21.15.123:0/3959052474 conn(0x7f51e00f8830 msgr2=0x7f51e00f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:31.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.710+0000 7f51e4a00700 1 -- 172.21.15.123:0/3959052474 shutdown_connections 2024-03-30T05:10:31.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.710+0000 7f51e4a00700 1 -- 172.21.15.123:0/3959052474 wait complete. 2024-03-30T05:10:31.709 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 23 2024-03-30T05:10:31.725 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 24 2024-03-30T05:10:31.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.915+0000 7fc07a000700 1 -- 172.21.15.123:0/748115271 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc074103680 msgr2=0x7fc0740fccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:31.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.915+0000 7fc07a000700 1 --2- 172.21.15.123:0/748115271 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc074103680 0x7fc0740fccf0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fc060004940 tx=0x7fc060009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:31.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.915+0000 7fc07a000700 1 -- 172.21.15.123:0/748115271 shutdown_connections 2024-03-30T05:10:31.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.915+0000 7fc07a000700 1 --2- 172.21.15.123:0/748115271 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc074103680 0x7fc0740fccf0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:31.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.915+0000 7fc07a000700 1 --2- 172.21.15.123:0/748115271 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc074102ce0 0x7fc0741030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:31.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.915+0000 7fc07a000700 1 -- 172.21.15.123:0/748115271 >> 172.21.15.123:0/748115271 conn(0x7fc0740f8830 msgr2=0x7fc0740fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:31.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.915+0000 7fc07a000700 1 -- 172.21.15.123:0/748115271 shutdown_connections 2024-03-30T05:10:31.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.915+0000 7fc07a000700 1 -- 172.21.15.123:0/748115271 wait complete. 2024-03-30T05:10:31.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.916+0000 7fc07a000700 1 Processor -- start 2024-03-30T05:10:31.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.916+0000 7fc07a000700 1 -- start start 2024-03-30T05:10:31.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.917+0000 7fc07a000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc074102ce0 0x7fc074115980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:31.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.917+0000 7fc07a000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc074103680 0x7fc074115ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:31.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.917+0000 7fc07a000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc074116510 con 0x7fc074103680 2024-03-30T05:10:31.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.917+0000 7fc07a000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc07410fb10 con 0x7fc074102ce0 2024-03-30T05:10:31.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.917+0000 7fc073400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc074102ce0 0x7fc074115980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:31.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.917+0000 7fc072a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc074103680 0x7fc074115ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:31.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.917+0000 7fc072a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc074103680 0x7fc074115ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41378/0 (socket says 172.21.15.123:41378) 2024-03-30T05:10:31.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.917+0000 7fc072a00700 1 -- 172.21.15.123:0/1299126656 learned_addr learned my addr 172.21.15.123:0/1299126656 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:10:31.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.918+0000 7fc072a00700 1 -- 172.21.15.123:0/1299126656 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc074102ce0 msgr2=0x7fc074115980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:31.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.918+0000 7fc072a00700 1 --2- 172.21.15.123:0/1299126656 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc074102ce0 0x7fc074115980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:31.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.918+0000 7fc072a00700 1 -- 172.21.15.123:0/1299126656 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc060005040 con 0x7fc074103680 2024-03-30T05:10:31.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.919+0000 7fc072a00700 1 --2- 172.21.15.123:0/1299126656 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc074103680 0x7fc074115ec0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fc060009390 tx=0x7fc060009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:31.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.919+0000 7fc06b400700 1 -- 172.21.15.123:0/1299126656 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc06000e040 con 0x7fc074103680 2024-03-30T05:10:31.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.919+0000 7fc06b400700 1 -- 172.21.15.123:0/1299126656 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc060013420 con 0x7fc074103680 2024-03-30T05:10:31.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.919+0000 7fc06b400700 1 -- 172.21.15.123:0/1299126656 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc06001c630 con 0x7fc074103680 2024-03-30T05:10:31.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.919+0000 7fc07a000700 1 -- 172.21.15.123:0/1299126656 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc07410fd90 con 0x7fc074103680 2024-03-30T05:10:31.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.919+0000 7fc07a000700 1 -- 172.21.15.123:0/1299126656 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc074110350 con 0x7fc074103680 2024-03-30T05:10:31.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.920+0000 7fc07a000700 1 -- 172.21.15.123:0/1299126656 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0740fe430 con 0x7fc074103680 2024-03-30T05:10:31.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.921+0000 7fc06b400700 1 -- 172.21.15.123:0/1299126656 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fc060013a00 con 0x7fc074103680 2024-03-30T05:10:31.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.922+0000 7fc06b400700 1 --2- 172.21.15.123:0/1299126656 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc05806cc20 0x7fc05806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:31.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.922+0000 7fc06b400700 1 -- 172.21.15.123:0/1299126656 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc06008d130 con 0x7fc074103680 2024-03-30T05:10:31.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.922+0000 7fc073400700 1 --2- 172.21.15.123:0/1299126656 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc05806cc20 0x7fc05806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:31.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.923+0000 7fc073400700 1 --2- 172.21.15.123:0/1299126656 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc05806cc20 0x7fc05806f0d0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fc064004f40 tx=0x7fc06400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:31.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:31.926+0000 7fc06b400700 1 -- 172.21.15.123:0/1299126656 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc06005ac00 con 0x7fc074103680 2024-03-30T05:10:32.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.174+0000 7fc07a000700 1 -- 172.21.15.123:0/1299126656 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 24, "format": "json"} v 0) v1 -- 0x7fc074109c80 con 0x7fc074103680 2024-03-30T05:10:32.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.175+0000 7fc06b400700 1 -- 172.21.15.123:0/1299126656 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 24, "format": "json"}]=0 dumped fsmap epoch 24 v42) v1 ==== 107+0+3290 (secure 0 0 0) 0x7fc06005a790 con 0x7fc074103680 2024-03-30T05:10:32.173 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:10:32.173 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T04:24:30.211769+0000","modified":"2024-03-30T04:37:19.249398+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14488,"mds_1":24309},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14488":{"gid":14488,"name":"cephfs.smithi123.dvtind","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.123:6829/3037849622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3037849622},{"type":"v1","addr":"172.21.15.123:6829","nonce":3037849622}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24309":{"gid":24309,"name":"cephfs.smithi135.zvnlap","rank":1,"incarnation":7,"state":"up:active","state_seq":4,"addr":"172.21.15.135:6825/4050541062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":4050541062},{"type":"v1","addr":"172.21.15.135:6825","nonce":4050541062}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-30T05:10:32.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.177+0000 7fc07a000700 1 -- 172.21.15.123:0/1299126656 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc05806cc20 msgr2=0x7fc05806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:32.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.177+0000 7fc07a000700 1 --2- 172.21.15.123:0/1299126656 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc05806cc20 0x7fc05806f0d0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fc064004f40 tx=0x7fc06400b040 comp rx=0 tx=0).stop 2024-03-30T05:10:32.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.177+0000 7fc07a000700 1 -- 172.21.15.123:0/1299126656 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc074103680 msgr2=0x7fc074115ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:32.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.177+0000 7fc07a000700 1 --2- 172.21.15.123:0/1299126656 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc074103680 0x7fc074115ec0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fc060009390 tx=0x7fc060009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:32.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.178+0000 7fc07a000700 1 -- 172.21.15.123:0/1299126656 shutdown_connections 2024-03-30T05:10:32.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.178+0000 7fc07a000700 1 --2- 172.21.15.123:0/1299126656 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fc05806cc20 0x7fc05806f0d0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:32.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.178+0000 7fc07a000700 1 --2- 172.21.15.123:0/1299126656 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc074102ce0 0x7fc074115980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:32.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.178+0000 7fc07a000700 1 --2- 172.21.15.123:0/1299126656 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc074103680 0x7fc074115ec0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:32.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.178+0000 7fc07a000700 1 -- 172.21.15.123:0/1299126656 >> 172.21.15.123:0/1299126656 conn(0x7fc0740f8830 msgr2=0x7fc0740f9900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:32.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.178+0000 7fc07a000700 1 -- 172.21.15.123:0/1299126656 shutdown_connections 2024-03-30T05:10:32.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.178+0000 7fc07a000700 1 -- 172.21.15.123:0/1299126656 wait complete. 2024-03-30T05:10:32.177 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 24 2024-03-30T05:10:32.194 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 24 2024-03-30T05:10:32.194 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 25 2024-03-30T05:10:32.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.385+0000 7f0801e00700 1 -- 172.21.15.123:0/2592389385 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07fc103680 msgr2=0x7f07fc0fccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:32.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.385+0000 7f0801e00700 1 --2- 172.21.15.123:0/2592389385 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07fc103680 0x7f07fc0fccf0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f07e8004940 tx=0x7f07e8009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:32.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.385+0000 7f0801e00700 1 -- 172.21.15.123:0/2592389385 shutdown_connections 2024-03-30T05:10:32.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.385+0000 7f0801e00700 1 --2- 172.21.15.123:0/2592389385 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07fc103680 0x7f07fc0fccf0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:32.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.385+0000 7f0801e00700 1 --2- 172.21.15.123:0/2592389385 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f07fc102ce0 0x7f07fc1030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:32.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.385+0000 7f0801e00700 1 -- 172.21.15.123:0/2592389385 >> 172.21.15.123:0/2592389385 conn(0x7f07fc0f8830 msgr2=0x7f07fc0fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:32.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.386+0000 7f0801e00700 1 -- 172.21.15.123:0/2592389385 shutdown_connections 2024-03-30T05:10:32.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.386+0000 7f0801e00700 1 -- 172.21.15.123:0/2592389385 wait complete. 2024-03-30T05:10:32.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.386+0000 7f0801e00700 1 Processor -- start 2024-03-30T05:10:32.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.386+0000 7f0801e00700 1 -- start start 2024-03-30T05:10:32.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.387+0000 7f0801e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07fc102ce0 0x7f07fc113790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:32.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.387+0000 7f0801e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f07fc103680 0x7f07fc113cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:32.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.387+0000 7f0801e00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07fc1143b0 con 0x7f07fc102ce0 2024-03-30T05:10:32.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.387+0000 7f0801e00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07fc10d920 con 0x7f07fc103680 2024-03-30T05:10:32.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.387+0000 7f07faa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07fc102ce0 0x7f07fc113790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:32.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.387+0000 7f07faa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07fc102ce0 0x7f07fc113790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41390/0 (socket says 172.21.15.123:41390) 2024-03-30T05:10:32.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.387+0000 7f07faa00700 1 -- 172.21.15.123:0/1995945459 learned_addr learned my addr 172.21.15.123:0/1995945459 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:10:32.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.387+0000 7f07fa000700 1 --2- 172.21.15.123:0/1995945459 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f07fc103680 0x7f07fc113cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:32.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.387+0000 7f07faa00700 1 -- 172.21.15.123:0/1995945459 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f07fc103680 msgr2=0x7f07fc113cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:32.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.387+0000 7f07faa00700 1 --2- 172.21.15.123:0/1995945459 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f07fc103680 0x7f07fc113cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:32.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.387+0000 7f07faa00700 1 -- 172.21.15.123:0/1995945459 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07e8005040 con 0x7f07fc102ce0 2024-03-30T05:10:32.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.388+0000 7f07faa00700 1 --2- 172.21.15.123:0/1995945459 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07fc102ce0 0x7f07fc113790 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f07ec00b790 tx=0x7f07ec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:32.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.388+0000 7f07f2a00700 1 -- 172.21.15.123:0/1995945459 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07ec00f6f0 con 0x7f07fc102ce0 2024-03-30T05:10:32.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.388+0000 7f07f2a00700 1 -- 172.21.15.123:0/1995945459 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f07ec00fd00 con 0x7f07fc102ce0 2024-03-30T05:10:32.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.388+0000 7f0801e00700 1 -- 172.21.15.123:0/1995945459 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07fc10dc00 con 0x7f07fc102ce0 2024-03-30T05:10:32.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.388+0000 7f07f2a00700 1 -- 172.21.15.123:0/1995945459 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07ec0103a0 con 0x7f07fc102ce0 2024-03-30T05:10:32.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.388+0000 7f0801e00700 1 -- 172.21.15.123:0/1995945459 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07fc10e190 con 0x7f07fc102ce0 2024-03-30T05:10:32.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.389+0000 7f0801e00700 1 -- 172.21.15.123:0/1995945459 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07fc0fe430 con 0x7f07fc102ce0 2024-03-30T05:10:32.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.390+0000 7f07f2a00700 1 -- 172.21.15.123:0/1995945459 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f07ec00f850 con 0x7f07fc102ce0 2024-03-30T05:10:32.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.391+0000 7f07f2a00700 1 --2- 172.21.15.123:0/1995945459 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f07e006cc20 0x7f07e006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:32.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.391+0000 7f07f2a00700 1 -- 172.21.15.123:0/1995945459 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f07ec08bf30 con 0x7f07fc102ce0 2024-03-30T05:10:32.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.391+0000 7f07fa000700 1 --2- 172.21.15.123:0/1995945459 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f07e006cc20 0x7f07e006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:32.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.392+0000 7f07fa000700 1 --2- 172.21.15.123:0/1995945459 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f07e006cc20 0x7f07e006f0d0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f07fc10ec00 tx=0x7f07e8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:32.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.395+0000 7f07f2a00700 1 -- 172.21.15.123:0/1995945459 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f07ec059980 con 0x7f07fc102ce0 2024-03-30T05:10:32.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:32 smithi123 ceph-mon[139369]: pgmap v1182: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:10:32.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:32 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3387525433' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 22, "format": "json"}]: dispatch 2024-03-30T05:10:32.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:32 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3959052474' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 23, "format": "json"}]: dispatch 2024-03-30T05:10:32.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:32 smithi135 ceph-mon[106011]: pgmap v1182: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:10:32.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:32 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3387525433' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 22, "format": "json"}]: dispatch 2024-03-30T05:10:32.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:32 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3959052474' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 23, "format": "json"}]: dispatch 2024-03-30T05:10:32.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.646+0000 7f0801e00700 1 -- 172.21.15.123:0/1995945459 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 25, "format": "json"} v 0) v1 -- 0x7f07fc109cf0 con 0x7f07fc102ce0 2024-03-30T05:10:32.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.646+0000 7f07f2a00700 1 -- 172.21.15.123:0/1995945459 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 25, "format": "json"}]=0 dumped fsmap epoch 25 v42) v1 ==== 107+0+3269 (secure 0 0 0) 0x7f07ec059510 con 0x7f07fc102ce0 2024-03-30T05:10:32.645 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:10:32.645 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34408,"name":"cephfs.smithi123.tqxvzr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6827/2891461179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":2891461179},{"type":"v1","addr":"172.21.15.123:6827","nonce":2891461179}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25},{"gid":44315,"name":"cephfs.smithi135.xwgjvb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6827/1741352336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":1741352336},{"type":"v1","addr":"172.21.15.135:6827","nonce":1741352336}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":25,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T04:24:30.211769+0000","modified":"2024-03-30T04:37:19.516763+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-30T05:10:32.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.649+0000 7f0801e00700 1 -- 172.21.15.123:0/1995945459 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f07e006cc20 msgr2=0x7f07e006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:32.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.649+0000 7f0801e00700 1 --2- 172.21.15.123:0/1995945459 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f07e006cc20 0x7f07e006f0d0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f07fc10ec00 tx=0x7f07e8009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:32.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.649+0000 7f0801e00700 1 -- 172.21.15.123:0/1995945459 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07fc102ce0 msgr2=0x7f07fc113790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:32.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.649+0000 7f0801e00700 1 --2- 172.21.15.123:0/1995945459 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07fc102ce0 0x7f07fc113790 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f07ec00b790 tx=0x7f07ec009250 comp rx=0 tx=0).stop 2024-03-30T05:10:32.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.649+0000 7f0801e00700 1 -- 172.21.15.123:0/1995945459 shutdown_connections 2024-03-30T05:10:32.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.649+0000 7f0801e00700 1 --2- 172.21.15.123:0/1995945459 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f07e006cc20 0x7f07e006f0d0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:32.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.649+0000 7f0801e00700 1 --2- 172.21.15.123:0/1995945459 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07fc102ce0 0x7f07fc113790 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:32.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.649+0000 7f0801e00700 1 --2- 172.21.15.123:0/1995945459 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f07fc103680 0x7f07fc113cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:32.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.649+0000 7f0801e00700 1 -- 172.21.15.123:0/1995945459 >> 172.21.15.123:0/1995945459 conn(0x7f07fc0f8830 msgr2=0x7f07fc0f9850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:32.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.649+0000 7f0801e00700 1 -- 172.21.15.123:0/1995945459 shutdown_connections 2024-03-30T05:10:32.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.649+0000 7f0801e00700 1 -- 172.21.15.123:0/1995945459 wait complete. 2024-03-30T05:10:32.648 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 25 2024-03-30T05:10:32.665 DEBUG:tasks.fs:max_mds not reduced in epoch 25 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-03-30T05:10:32.665 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 25 2024-03-30T05:10:32.665 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 26 2024-03-30T05:10:32.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.856+0000 7f733e400700 1 -- 172.21.15.123:0/1775822046 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7338070c70 msgr2=0x7f7338108fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:32.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.856+0000 7f733e400700 1 --2- 172.21.15.123:0/1775822046 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7338070c70 0x7f7338108fe0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f7324004970 tx=0x7f7324009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:32.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.856+0000 7f733e400700 1 -- 172.21.15.123:0/1775822046 shutdown_connections 2024-03-30T05:10:32.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.856+0000 7f733e400700 1 --2- 172.21.15.123:0/1775822046 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7338070c70 0x7f7338108fe0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:32.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.856+0000 7f733e400700 1 --2- 172.21.15.123:0/1775822046 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7338070360 0x7f7338070730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:32.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.856+0000 7f733e400700 1 -- 172.21.15.123:0/1775822046 >> 172.21.15.123:0/1775822046 conn(0x7f73380f8820 msgr2=0x7f73380fac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:32.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.856+0000 7f733e400700 1 -- 172.21.15.123:0/1775822046 shutdown_connections 2024-03-30T05:10:32.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.856+0000 7f733e400700 1 -- 172.21.15.123:0/1775822046 wait complete. 2024-03-30T05:10:32.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.857+0000 7f733e400700 1 Processor -- start 2024-03-30T05:10:32.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.857+0000 7f733e400700 1 -- start start 2024-03-30T05:10:32.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.857+0000 7f733e400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7338070360 0x7f7338115920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:32.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.857+0000 7f733e400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7338070c70 0x7f7338115e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:32.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.857+0000 7f733e400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73381164b0 con 0x7f7338070360 2024-03-30T05:10:32.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.857+0000 7f733e400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f733810fab0 con 0x7f7338070c70 2024-03-30T05:10:32.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.857+0000 7f7337400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7338070360 0x7f7338115920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:32.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.858+0000 7f7337400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7338070360 0x7f7338115920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41414/0 (socket says 172.21.15.123:41414) 2024-03-30T05:10:32.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.858+0000 7f7337400700 1 -- 172.21.15.123:0/1921553958 learned_addr learned my addr 172.21.15.123:0/1921553958 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:10:32.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.858+0000 7f7336a00700 1 --2- 172.21.15.123:0/1921553958 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7338070c70 0x7f7338115e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:32.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.858+0000 7f7337400700 1 -- 172.21.15.123:0/1921553958 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7338070c70 msgr2=0x7f7338115e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:32.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.858+0000 7f7337400700 1 --2- 172.21.15.123:0/1921553958 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7338070c70 0x7f7338115e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:32.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.858+0000 7f7337400700 1 -- 172.21.15.123:0/1921553958 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7324005040 con 0x7f7338070360 2024-03-30T05:10:32.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.858+0000 7f7337400700 1 --2- 172.21.15.123:0/1921553958 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7338070360 0x7f7338115920 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f7328002a50 tx=0x7f73280046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:32.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.858+0000 7f732f400700 1 -- 172.21.15.123:0/1921553958 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7328007cc0 con 0x7f7338070360 2024-03-30T05:10:32.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.858+0000 7f732f400700 1 -- 172.21.15.123:0/1921553958 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f732800f040 con 0x7f7338070360 2024-03-30T05:10:32.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.858+0000 7f733e400700 1 -- 172.21.15.123:0/1921553958 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f733810fd90 con 0x7f7338070360 2024-03-30T05:10:32.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.858+0000 7f732f400700 1 -- 172.21.15.123:0/1921553958 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f732800d9c0 con 0x7f7338070360 2024-03-30T05:10:32.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.858+0000 7f733e400700 1 -- 172.21.15.123:0/1921553958 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73381102a0 con 0x7f7338070360 2024-03-30T05:10:32.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.860+0000 7f733e400700 1 -- 172.21.15.123:0/1921553958 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7338071aa0 con 0x7f7338070360 2024-03-30T05:10:32.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.861+0000 7f732f400700 1 -- 172.21.15.123:0/1921553958 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f732800db20 con 0x7f7338070360 2024-03-30T05:10:32.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.861+0000 7f732f400700 1 --2- 172.21.15.123:0/1921553958 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f731c06cc20 0x7f731c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:32.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.862+0000 7f732f400700 1 -- 172.21.15.123:0/1921553958 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f732808bd60 con 0x7f7338070360 2024-03-30T05:10:32.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.862+0000 7f7336a00700 1 --2- 172.21.15.123:0/1921553958 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f731c06cc20 0x7f731c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:32.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.862+0000 7f7336a00700 1 --2- 172.21.15.123:0/1921553958 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f731c06cc20 0x7f731c06f0d0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f732400b370 tx=0x7f7324009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:32.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:32.866+0000 7f732f400700 1 -- 172.21.15.123:0/1921553958 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f73280597b0 con 0x7f7338070360 2024-03-30T05:10:33.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.114+0000 7f733e400700 1 -- 172.21.15.123:0/1921553958 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 26, "format": "json"} v 0) v1 -- 0x7f7338064970 con 0x7f7338070360 2024-03-30T05:10:33.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.115+0000 7f732f400700 1 -- 172.21.15.123:0/1921553958 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 26, "format": "json"}]=0 dumped fsmap epoch 26 v42) v1 ==== 107+0+4833 (secure 0 0 0) 0x7f7328059340 con 0x7f7338070360 2024-03-30T05:10:33.113 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:10:33.113 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34408,"name":"cephfs.smithi123.tqxvzr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6827/2891461179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":2891461179},{"type":"v1","addr":"172.21.15.123:6827","nonce":2891461179}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25},{"gid":44315,"name":"cephfs.smithi135.xwgjvb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6827/1741352336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":1741352336},{"type":"v1","addr":"172.21.15.135:6827","nonce":1741352336}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25},{"gid":44323,"name":"cephfs.smithi135.zvnlap","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6825/2944944016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":2944944016},{"type":"v1","addr":"172.21.15.135:6825","nonce":2944944016}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26},{"gid":44325,"name":"cephfs.smithi123.dvtind","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6829/3902778632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3902778632},{"type":"v1","addr":"172.21.15.123:6829","nonce":3902778632}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T04:24:30.211769+0000","modified":"2024-03-30T04:37:19.516763+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-30T05:10:33.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.117+0000 7f733e400700 1 -- 172.21.15.123:0/1921553958 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f731c06cc20 msgr2=0x7f731c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:33.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.117+0000 7f733e400700 1 --2- 172.21.15.123:0/1921553958 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f731c06cc20 0x7f731c06f0d0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f732400b370 tx=0x7f7324009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:33.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.117+0000 7f733e400700 1 -- 172.21.15.123:0/1921553958 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7338070360 msgr2=0x7f7338115920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:33.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.117+0000 7f733e400700 1 --2- 172.21.15.123:0/1921553958 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7338070360 0x7f7338115920 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f7328002a50 tx=0x7f73280046b0 comp rx=0 tx=0).stop 2024-03-30T05:10:33.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.118+0000 7f733e400700 1 -- 172.21.15.123:0/1921553958 shutdown_connections 2024-03-30T05:10:33.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.118+0000 7f733e400700 1 --2- 172.21.15.123:0/1921553958 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f731c06cc20 0x7f731c06f0d0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:33.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.118+0000 7f733e400700 1 --2- 172.21.15.123:0/1921553958 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7338070360 0x7f7338115920 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:33.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.118+0000 7f733e400700 1 --2- 172.21.15.123:0/1921553958 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7338070c70 0x7f7338115e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:33.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.118+0000 7f733e400700 1 -- 172.21.15.123:0/1921553958 >> 172.21.15.123:0/1921553958 conn(0x7f73380f8820 msgr2=0x7f73380f9d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:33.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.118+0000 7f733e400700 1 -- 172.21.15.123:0/1921553958 shutdown_connections 2024-03-30T05:10:33.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.118+0000 7f733e400700 1 -- 172.21.15.123:0/1921553958 wait complete. 2024-03-30T05:10:33.117 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 26 2024-03-30T05:10:33.134 DEBUG:tasks.fs:max_mds not reduced in epoch 26 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-03-30T05:10:33.134 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 26 2024-03-30T05:10:33.134 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 27 2024-03-30T05:10:33.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.328+0000 7f9910a00700 1 -- 172.21.15.123:0/2902434422 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f990c103680 msgr2=0x7f990c064e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:33.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.328+0000 7f9910a00700 1 --2- 172.21.15.123:0/2902434422 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f990c103680 0x7f990c064e40 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f98f8004940 tx=0x7f98f8009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:33.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.328+0000 7f9910a00700 1 -- 172.21.15.123:0/2902434422 shutdown_connections 2024-03-30T05:10:33.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.328+0000 7f9910a00700 1 --2- 172.21.15.123:0/2902434422 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f990c103680 0x7f990c064e40 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:33.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.328+0000 7f9910a00700 1 --2- 172.21.15.123:0/2902434422 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f990c102ce0 0x7f990c1030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:33.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.328+0000 7f9910a00700 1 -- 172.21.15.123:0/2902434422 >> 172.21.15.123:0/2902434422 conn(0x7f990c072ad0 msgr2=0x7f990c072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:33.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.328+0000 7f9910a00700 1 -- 172.21.15.123:0/2902434422 shutdown_connections 2024-03-30T05:10:33.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.328+0000 7f9910a00700 1 -- 172.21.15.123:0/2902434422 wait complete. 2024-03-30T05:10:33.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.329+0000 7f9910a00700 1 Processor -- start 2024-03-30T05:10:33.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.329+0000 7f9910a00700 1 -- start start 2024-03-30T05:10:33.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.329+0000 7f9910a00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f990c102ce0 0x7f990c1137a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:33.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.329+0000 7f9910a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f990c103680 0x7f990c113ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:33.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.329+0000 7f9910a00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f990c1143c0 con 0x7f990c103680 2024-03-30T05:10:33.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.329+0000 7f9910a00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f990c10d930 con 0x7f990c102ce0 2024-03-30T05:10:33.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.329+0000 7f9909400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f990c103680 0x7f990c113ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:33.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.329+0000 7f9909400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f990c103680 0x7f990c113ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41426/0 (socket says 172.21.15.123:41426) 2024-03-30T05:10:33.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.329+0000 7f9909400700 1 -- 172.21.15.123:0/2258364897 learned_addr learned my addr 172.21.15.123:0/2258364897 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:10:33.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.330+0000 7f9909400700 1 -- 172.21.15.123:0/2258364897 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f990c102ce0 msgr2=0x7f990c1137a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:33.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.330+0000 7f9909400700 1 --2- 172.21.15.123:0/2258364897 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f990c102ce0 0x7f990c1137a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:33.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.330+0000 7f9909400700 1 -- 172.21.15.123:0/2258364897 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98f8005040 con 0x7f990c103680 2024-03-30T05:10:33.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.330+0000 7f9909400700 1 --2- 172.21.15.123:0/2258364897 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f990c103680 0x7f990c113ce0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f98f8009390 tx=0x7f98f8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:33.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.330+0000 7f9902000700 1 -- 172.21.15.123:0/2258364897 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98f800e040 con 0x7f990c103680 2024-03-30T05:10:33.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.330+0000 7f9902000700 1 -- 172.21.15.123:0/2258364897 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f98f8013420 con 0x7f990c103680 2024-03-30T05:10:33.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.330+0000 7f9910a00700 1 -- 172.21.15.123:0/2258364897 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f990c10dbb0 con 0x7f990c103680 2024-03-30T05:10:33.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.330+0000 7f9902000700 1 -- 172.21.15.123:0/2258364897 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98f801c630 con 0x7f990c103680 2024-03-30T05:10:33.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.330+0000 7f9910a00700 1 -- 172.21.15.123:0/2258364897 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f990c10e170 con 0x7f990c103680 2024-03-30T05:10:33.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.331+0000 7f9910a00700 1 -- 172.21.15.123:0/2258364897 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f990c0fe430 con 0x7f990c103680 2024-03-30T05:10:33.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.332+0000 7f9902000700 1 -- 172.21.15.123:0/2258364897 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f98f8013a00 con 0x7f990c103680 2024-03-30T05:10:33.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.333+0000 7f9902000700 1 --2- 172.21.15.123:0/2258364897 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f98f006cc20 0x7f98f006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:33.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.333+0000 7f9902000700 1 -- 172.21.15.123:0/2258364897 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f98f808d200 con 0x7f990c103680 2024-03-30T05:10:33.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.333+0000 7f9909e00700 1 --2- 172.21.15.123:0/2258364897 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f98f006cc20 0x7f98f006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:33.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.334+0000 7f9909e00700 1 --2- 172.21.15.123:0/2258364897 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f98f006cc20 0x7f98f006f0d0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f98fc006fd0 tx=0x7f98fc006db0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:33.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.337+0000 7f9902000700 1 -- 172.21.15.123:0/2258364897 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f98f805ad90 con 0x7f990c103680 2024-03-30T05:10:33.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:33 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/1299126656' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 24, "format": "json"}]: dispatch 2024-03-30T05:10:33.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:33 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/1995945459' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 25, "format": "json"}]: dispatch 2024-03-30T05:10:33.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:33 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/1921553958' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 26, "format": "json"}]: dispatch 2024-03-30T05:10:33.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.586+0000 7f9910a00700 1 -- 172.21.15.123:0/2258364897 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 27, "format": "json"} v 0) v1 -- 0x7f990c109cf0 con 0x7f990c103680 2024-03-30T05:10:33.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.586+0000 7f9902000700 1 -- 172.21.15.123:0/2258364897 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 27, "format": "json"}]=0 dumped fsmap epoch 27 v42) v1 ==== 107+0+4051 (secure 0 0 0) 0x7f98f805a920 con 0x7f990c103680 2024-03-30T05:10:33.584 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:10:33.585 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44315,"name":"cephfs.smithi135.xwgjvb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6827/1741352336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":1741352336},{"type":"v1","addr":"172.21.15.135:6827","nonce":1741352336}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25},{"gid":44323,"name":"cephfs.smithi135.zvnlap","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6825/2944944016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":2944944016},{"type":"v1","addr":"172.21.15.135:6825","nonce":2944944016}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26},{"gid":44325,"name":"cephfs.smithi123.dvtind","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6829/3902778632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3902778632},{"type":"v1","addr":"172.21.15.123:6829","nonce":3902778632}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T04:24:30.211769+0000","modified":"2024-03-30T04:37:19.516763+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-30T05:10:33.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.589+0000 7f9910a00700 1 -- 172.21.15.123:0/2258364897 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f98f006cc20 msgr2=0x7f98f006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:33.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.589+0000 7f9910a00700 1 --2- 172.21.15.123:0/2258364897 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f98f006cc20 0x7f98f006f0d0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f98fc006fd0 tx=0x7f98fc006db0 comp rx=0 tx=0).stop 2024-03-30T05:10:33.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.589+0000 7f9910a00700 1 -- 172.21.15.123:0/2258364897 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f990c103680 msgr2=0x7f990c113ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:33.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.589+0000 7f9910a00700 1 --2- 172.21.15.123:0/2258364897 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f990c103680 0x7f990c113ce0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f98f8009390 tx=0x7f98f8009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:33.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.589+0000 7f9910a00700 1 -- 172.21.15.123:0/2258364897 shutdown_connections 2024-03-30T05:10:33.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.589+0000 7f9910a00700 1 --2- 172.21.15.123:0/2258364897 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f98f006cc20 0x7f98f006f0d0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:33.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.589+0000 7f9910a00700 1 --2- 172.21.15.123:0/2258364897 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f990c102ce0 0x7f990c1137a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:33.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.589+0000 7f9910a00700 1 --2- 172.21.15.123:0/2258364897 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f990c103680 0x7f990c113ce0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:33.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.589+0000 7f9910a00700 1 -- 172.21.15.123:0/2258364897 >> 172.21.15.123:0/2258364897 conn(0x7f990c072ad0 msgr2=0x7f990c105a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:33.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.589+0000 7f9910a00700 1 -- 172.21.15.123:0/2258364897 shutdown_connections 2024-03-30T05:10:33.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.589+0000 7f9910a00700 1 -- 172.21.15.123:0/2258364897 wait complete. 2024-03-30T05:10:33.588 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 27 2024-03-30T05:10:33.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:33 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/1299126656' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 24, "format": "json"}]: dispatch 2024-03-30T05:10:33.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:33 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/1995945459' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 25, "format": "json"}]: dispatch 2024-03-30T05:10:33.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:33 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/1921553958' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 26, "format": "json"}]: dispatch 2024-03-30T05:10:33.605 DEBUG:tasks.fs:max_mds not reduced in epoch 27 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-03-30T05:10:33.606 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 27 2024-03-30T05:10:33.606 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 28 2024-03-30T05:10:33.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.798+0000 7f61d2000700 1 -- 172.21.15.123:0/519251080 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61cc103680 msgr2=0x7f61cc0fccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:33.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.798+0000 7f61d2000700 1 --2- 172.21.15.123:0/519251080 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61cc103680 0x7f61cc0fccf0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f61b8004940 tx=0x7f61b8009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:33.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.798+0000 7f61d2000700 1 -- 172.21.15.123:0/519251080 shutdown_connections 2024-03-30T05:10:33.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.798+0000 7f61d2000700 1 --2- 172.21.15.123:0/519251080 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61cc103680 0x7f61cc0fccf0 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:33.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.798+0000 7f61d2000700 1 --2- 172.21.15.123:0/519251080 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f61cc102ce0 0x7f61cc1030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:33.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.798+0000 7f61d2000700 1 -- 172.21.15.123:0/519251080 >> 172.21.15.123:0/519251080 conn(0x7f61cc0f8830 msgr2=0x7f61cc0fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:33.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.798+0000 7f61d2000700 1 -- 172.21.15.123:0/519251080 shutdown_connections 2024-03-30T05:10:33.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.798+0000 7f61d2000700 1 -- 172.21.15.123:0/519251080 wait complete. 2024-03-30T05:10:33.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.799+0000 7f61d2000700 1 Processor -- start 2024-03-30T05:10:33.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.799+0000 7f61d2000700 1 -- start start 2024-03-30T05:10:33.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.800+0000 7f61d2000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61cc102ce0 0x7f61cc1136d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:33.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.800+0000 7f61d2000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f61cc103680 0x7f61cc113c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:33.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.800+0000 7f61d2000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61cc114260 con 0x7f61cc102ce0 2024-03-30T05:10:33.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.800+0000 7f61d2000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61cc10d880 con 0x7f61cc103680 2024-03-30T05:10:33.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.800+0000 7f61cb400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61cc102ce0 0x7f61cc1136d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:33.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.800+0000 7f61cb400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61cc102ce0 0x7f61cc1136d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41448/0 (socket says 172.21.15.123:41448) 2024-03-30T05:10:33.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.800+0000 7f61cb400700 1 -- 172.21.15.123:0/150081290 learned_addr learned my addr 172.21.15.123:0/150081290 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:10:33.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.800+0000 7f61caa00700 1 --2- 172.21.15.123:0/150081290 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f61cc103680 0x7f61cc113c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:33.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.800+0000 7f61cb400700 1 -- 172.21.15.123:0/150081290 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f61cc103680 msgr2=0x7f61cc113c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:33.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.800+0000 7f61cb400700 1 --2- 172.21.15.123:0/150081290 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f61cc103680 0x7f61cc113c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:33.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.800+0000 7f61cb400700 1 -- 172.21.15.123:0/150081290 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b8005040 con 0x7f61cc102ce0 2024-03-30T05:10:33.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.800+0000 7f61caa00700 1 --2- 172.21.15.123:0/150081290 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f61cc103680 0x7f61cc113c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T05:10:33.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.801+0000 7f61cb400700 1 --2- 172.21.15.123:0/150081290 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61cc102ce0 0x7f61cc1136d0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f61bc002a80 tx=0x7f61bc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:33.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.801+0000 7f61c3400700 1 -- 172.21.15.123:0/150081290 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f61bc007cc0 con 0x7f61cc102ce0 2024-03-30T05:10:33.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.801+0000 7f61c3400700 1 -- 172.21.15.123:0/150081290 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f61bc00f040 con 0x7f61cc102ce0 2024-03-30T05:10:33.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.801+0000 7f61c3400700 1 -- 172.21.15.123:0/150081290 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f61bc009640 con 0x7f61cc102ce0 2024-03-30T05:10:33.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.801+0000 7f61d2000700 1 -- 172.21.15.123:0/150081290 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61cc10db60 con 0x7f61cc102ce0 2024-03-30T05:10:33.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.801+0000 7f61d2000700 1 -- 172.21.15.123:0/150081290 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61cc10e0d0 con 0x7f61cc102ce0 2024-03-30T05:10:33.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.802+0000 7f61d2000700 1 -- 172.21.15.123:0/150081290 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61cc0fe430 con 0x7f61cc102ce0 2024-03-30T05:10:33.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.803+0000 7f61c3400700 1 -- 172.21.15.123:0/150081290 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f61bc0097a0 con 0x7f61cc102ce0 2024-03-30T05:10:33.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.804+0000 7f61c3400700 1 --2- 172.21.15.123:0/150081290 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f61b006cc20 0x7f61b006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:33.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.804+0000 7f61c3400700 1 -- 172.21.15.123:0/150081290 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f61bc08b8f0 con 0x7f61cc102ce0 2024-03-30T05:10:33.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.804+0000 7f61caa00700 1 --2- 172.21.15.123:0/150081290 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f61b006cc20 0x7f61b006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:33.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.804+0000 7f61caa00700 1 --2- 172.21.15.123:0/150081290 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f61b006cc20 0x7f61b006f0d0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f61cc10efb0 tx=0x7f61b8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:33.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:33.808+0000 7f61c3400700 1 -- 172.21.15.123:0/150081290 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f61bc0592c0 con 0x7f61cc102ce0 2024-03-30T05:10:34.055 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.057+0000 7f61d2000700 1 -- 172.21.15.123:0/150081290 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 28, "format": "json"} v 0) v1 -- 0x7f61cc064970 con 0x7f61cc102ce0 2024-03-30T05:10:34.055 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.057+0000 7f61c3400700 1 -- 172.21.15.123:0/150081290 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 28, "format": "json"}]=0 dumped fsmap epoch 28 v42) v1 ==== 107+0+4833 (secure 0 0 0) 0x7f61bc058e50 con 0x7f61cc102ce0 2024-03-30T05:10:34.056 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:10:34.056 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34420,"name":"cephfs.smithi123.tqxvzr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6827/1741131598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1741131598},{"type":"v1","addr":"172.21.15.123:6827","nonce":1741131598}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":44315,"name":"cephfs.smithi135.xwgjvb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6827/1741352336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":1741352336},{"type":"v1","addr":"172.21.15.135:6827","nonce":1741352336}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25},{"gid":44323,"name":"cephfs.smithi135.zvnlap","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6825/2944944016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":2944944016},{"type":"v1","addr":"172.21.15.135:6825","nonce":2944944016}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26},{"gid":44325,"name":"cephfs.smithi123.dvtind","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6829/3902778632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":3902778632},{"type":"v1","addr":"172.21.15.123:6829","nonce":3902778632}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T04:24:30.211769+0000","modified":"2024-03-30T04:37:19.516763+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-30T05:10:34.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.060+0000 7f61d2000700 1 -- 172.21.15.123:0/150081290 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f61b006cc20 msgr2=0x7f61b006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:34.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.060+0000 7f61d2000700 1 --2- 172.21.15.123:0/150081290 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f61b006cc20 0x7f61b006f0d0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f61cc10efb0 tx=0x7f61b8009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:34.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.060+0000 7f61d2000700 1 -- 172.21.15.123:0/150081290 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61cc102ce0 msgr2=0x7f61cc1136d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:34.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.060+0000 7f61d2000700 1 --2- 172.21.15.123:0/150081290 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61cc102ce0 0x7f61cc1136d0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f61bc002a80 tx=0x7f61bc0046b0 comp rx=0 tx=0).stop 2024-03-30T05:10:34.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.060+0000 7f61d2000700 1 -- 172.21.15.123:0/150081290 shutdown_connections 2024-03-30T05:10:34.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.060+0000 7f61d2000700 1 --2- 172.21.15.123:0/150081290 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f61b006cc20 0x7f61b006f0d0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:34.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.060+0000 7f61d2000700 1 --2- 172.21.15.123:0/150081290 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61cc102ce0 0x7f61cc1136d0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:34.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.060+0000 7f61d2000700 1 --2- 172.21.15.123:0/150081290 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f61cc103680 0x7f61cc113c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:34.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.060+0000 7f61d2000700 1 -- 172.21.15.123:0/150081290 >> 172.21.15.123:0/150081290 conn(0x7f61cc0f8830 msgr2=0x7f61cc0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:34.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.060+0000 7f61d2000700 1 -- 172.21.15.123:0/150081290 shutdown_connections 2024-03-30T05:10:34.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.060+0000 7f61d2000700 1 -- 172.21.15.123:0/150081290 wait complete. 2024-03-30T05:10:34.059 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 28 2024-03-30T05:10:34.075 DEBUG:tasks.fs:max_mds not reduced in epoch 28 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-03-30T05:10:34.075 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 28 2024-03-30T05:10:34.075 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 29 2024-03-30T05:10:34.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.268+0000 7f99fb800700 1 -- 172.21.15.123:0/963102690 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99f4102ce0 msgr2=0x7f99f41030b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:34.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.268+0000 7f99fb800700 1 --2- 172.21.15.123:0/963102690 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99f4102ce0 0x7f99f41030b0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f99e8004940 tx=0x7f99e8009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:34.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.269+0000 7f99fb800700 1 -- 172.21.15.123:0/963102690 shutdown_connections 2024-03-30T05:10:34.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.269+0000 7f99fb800700 1 --2- 172.21.15.123:0/963102690 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f99f4103680 0x7f99f40fccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:34.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.269+0000 7f99fb800700 1 --2- 172.21.15.123:0/963102690 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99f4102ce0 0x7f99f41030b0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:34.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.269+0000 7f99fb800700 1 -- 172.21.15.123:0/963102690 >> 172.21.15.123:0/963102690 conn(0x7f99f40f8830 msgr2=0x7f99f40fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:34.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.269+0000 7f99fb800700 1 -- 172.21.15.123:0/963102690 shutdown_connections 2024-03-30T05:10:34.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.269+0000 7f99fb800700 1 -- 172.21.15.123:0/963102690 wait complete. 2024-03-30T05:10:34.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.269+0000 7f99fb800700 1 Processor -- start 2024-03-30T05:10:34.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.270+0000 7f99fb800700 1 -- start start 2024-03-30T05:10:34.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.270+0000 7f99fb800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99f4102ce0 0x7f99f40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:34.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.270+0000 7f99fb800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f99f4103680 0x7f99f4070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:34.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.270+0000 7f99fb800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f4072210 con 0x7f99f4102ce0 2024-03-30T05:10:34.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.270+0000 7f99fb800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f4070d40 con 0x7f99f4103680 2024-03-30T05:10:34.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.270+0000 7f99f8e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99f4102ce0 0x7f99f40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:34.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.270+0000 7f99f8e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99f4102ce0 0x7f99f40702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39750/0 (socket says 172.21.15.123:39750) 2024-03-30T05:10:34.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.270+0000 7f99f8e00700 1 -- 172.21.15.123:0/1730611322 learned_addr learned my addr 172.21.15.123:0/1730611322 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:10:34.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.270+0000 7f99f3e00700 1 --2- 172.21.15.123:0/1730611322 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f99f4103680 0x7f99f4070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:34.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.270+0000 7f99f8e00700 1 -- 172.21.15.123:0/1730611322 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f99f4103680 msgr2=0x7f99f4070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:34.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.270+0000 7f99f8e00700 1 --2- 172.21.15.123:0/1730611322 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f99f4103680 0x7f99f4070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:34.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.270+0000 7f99f8e00700 1 -- 172.21.15.123:0/1730611322 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99e8005040 con 0x7f99f4102ce0 2024-03-30T05:10:34.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.271+0000 7f99f8e00700 1 --2- 172.21.15.123:0/1730611322 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99f4102ce0 0x7f99f40702c0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f99e8009360 tx=0x7f99e8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:34.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.271+0000 7f99f0c00700 1 -- 172.21.15.123:0/1730611322 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99e800e040 con 0x7f99f4102ce0 2024-03-30T05:10:34.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.271+0000 7f99f0c00700 1 -- 172.21.15.123:0/1730611322 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f99e8013420 con 0x7f99f4102ce0 2024-03-30T05:10:34.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.271+0000 7f99fb800700 1 -- 172.21.15.123:0/1730611322 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99f4070fc0 con 0x7f99f4102ce0 2024-03-30T05:10:34.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.271+0000 7f99f0c00700 1 -- 172.21.15.123:0/1730611322 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99e801c5f0 con 0x7f99f4102ce0 2024-03-30T05:10:34.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.271+0000 7f99fb800700 1 -- 172.21.15.123:0/1730611322 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99f4119f60 con 0x7f99f4102ce0 2024-03-30T05:10:34.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.272+0000 7f99fb800700 1 -- 172.21.15.123:0/1730611322 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99f40fe430 con 0x7f99f4102ce0 2024-03-30T05:10:34.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.273+0000 7f99f0c00700 1 -- 172.21.15.123:0/1730611322 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f99e8013590 con 0x7f99f4102ce0 2024-03-30T05:10:34.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.274+0000 7f99f0c00700 1 --2- 172.21.15.123:0/1730611322 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f99dc06cc20 0x7f99dc06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:34.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.274+0000 7f99f0c00700 1 -- 172.21.15.123:0/1730611322 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f99e808d260 con 0x7f99f4102ce0 2024-03-30T05:10:34.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.274+0000 7f99f3e00700 1 --2- 172.21.15.123:0/1730611322 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f99dc06cc20 0x7f99dc06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:34.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.275+0000 7f99f3e00700 1 --2- 172.21.15.123:0/1730611322 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f99dc06cc20 0x7f99dc06f0d0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f99f40719f0 tx=0x7f99e0009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:34.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.278+0000 7f99f0c00700 1 -- 172.21.15.123:0/1730611322 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f99e80572a0 con 0x7f99f4102ce0 2024-03-30T05:10:34.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:34 smithi123 ceph-mon[139369]: pgmap v1183: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:10:34.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:34 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2258364897' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 27, "format": "json"}]: dispatch 2024-03-30T05:10:34.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:34 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/150081290' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 28, "format": "json"}]: dispatch 2024-03-30T05:10:34.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.525+0000 7f99fb800700 1 -- 172.21.15.123:0/1730611322 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 29, "format": "json"} v 0) v1 -- 0x7f99f4064970 con 0x7f99f4102ce0 2024-03-30T05:10:34.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.526+0000 7f99f0c00700 1 -- 172.21.15.123:0/1730611322 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 29, "format": "json"}]=0 dumped fsmap epoch 29 v42) v1 ==== 107+0+4051 (secure 0 0 0) 0x7f99e805a8c0 con 0x7f99f4102ce0 2024-03-30T05:10:34.524 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:10:34.524 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34420,"name":"cephfs.smithi123.tqxvzr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6827/1741131598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1741131598},{"type":"v1","addr":"172.21.15.123:6827","nonce":1741131598}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":44315,"name":"cephfs.smithi135.xwgjvb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6827/1741352336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":1741352336},{"type":"v1","addr":"172.21.15.135:6827","nonce":1741352336}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25},{"gid":44323,"name":"cephfs.smithi135.zvnlap","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6825/2944944016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":2944944016},{"type":"v1","addr":"172.21.15.135:6825","nonce":2944944016}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T04:24:30.211769+0000","modified":"2024-03-30T04:37:19.516763+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-30T05:10:34.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.528+0000 7f99fb800700 1 -- 172.21.15.123:0/1730611322 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f99dc06cc20 msgr2=0x7f99dc06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:34.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.528+0000 7f99fb800700 1 --2- 172.21.15.123:0/1730611322 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f99dc06cc20 0x7f99dc06f0d0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f99f40719f0 tx=0x7f99e0009ee0 comp rx=0 tx=0).stop 2024-03-30T05:10:34.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.528+0000 7f99fb800700 1 -- 172.21.15.123:0/1730611322 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99f4102ce0 msgr2=0x7f99f40702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:34.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.528+0000 7f99fb800700 1 --2- 172.21.15.123:0/1730611322 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99f4102ce0 0x7f99f40702c0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f99e8009360 tx=0x7f99e8009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:34.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.528+0000 7f99fb800700 1 -- 172.21.15.123:0/1730611322 shutdown_connections 2024-03-30T05:10:34.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.528+0000 7f99fb800700 1 --2- 172.21.15.123:0/1730611322 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f99dc06cc20 0x7f99dc06f0d0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:34.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.528+0000 7f99fb800700 1 --2- 172.21.15.123:0/1730611322 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99f4102ce0 0x7f99f40702c0 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:34.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.528+0000 7f99fb800700 1 --2- 172.21.15.123:0/1730611322 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f99f4103680 0x7f99f4070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:34.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.528+0000 7f99fb800700 1 -- 172.21.15.123:0/1730611322 >> 172.21.15.123:0/1730611322 conn(0x7f99f40f8830 msgr2=0x7f99f40f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:34.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.529+0000 7f99fb800700 1 -- 172.21.15.123:0/1730611322 shutdown_connections 2024-03-30T05:10:34.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.529+0000 7f99fb800700 1 -- 172.21.15.123:0/1730611322 wait complete. 2024-03-30T05:10:34.528 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 29 2024-03-30T05:10:34.544 DEBUG:tasks.fs:max_mds not reduced in epoch 29 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-03-30T05:10:34.544 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 29 2024-03-30T05:10:34.544 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 30 2024-03-30T05:10:34.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:34 smithi135 ceph-mon[106011]: pgmap v1183: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:10:34.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:34 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2258364897' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 27, "format": "json"}]: dispatch 2024-03-30T05:10:34.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:34 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/150081290' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 28, "format": "json"}]: dispatch 2024-03-30T05:10:34.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.735+0000 7fd15e600700 1 -- 172.21.15.123:0/3599961206 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd158102cd0 msgr2=0x7fd1581030a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:34.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.735+0000 7fd15e600700 1 --2- 172.21.15.123:0/3599961206 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd158102cd0 0x7fd1581030a0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fd148004940 tx=0x7fd148009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:34.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.736+0000 7fd15e600700 1 -- 172.21.15.123:0/3599961206 shutdown_connections 2024-03-30T05:10:34.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.736+0000 7fd15e600700 1 --2- 172.21.15.123:0/3599961206 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd158103670 0x7fd1580fcce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:34.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.736+0000 7fd15e600700 1 --2- 172.21.15.123:0/3599961206 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd158102cd0 0x7fd1581030a0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:34.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.736+0000 7fd15e600700 1 -- 172.21.15.123:0/3599961206 >> 172.21.15.123:0/3599961206 conn(0x7fd1580f8820 msgr2=0x7fd1580fac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:34.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.736+0000 7fd15e600700 1 -- 172.21.15.123:0/3599961206 shutdown_connections 2024-03-30T05:10:34.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.736+0000 7fd15e600700 1 -- 172.21.15.123:0/3599961206 wait complete. 2024-03-30T05:10:34.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.736+0000 7fd15e600700 1 Processor -- start 2024-03-30T05:10:34.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.737+0000 7fd15e600700 1 -- start start 2024-03-30T05:10:34.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.737+0000 7fd15e600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd158102cd0 0x7fd1580702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:34.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.737+0000 7fd15e600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd158103670 0x7fd158070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:34.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.737+0000 7fd15e600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1580722a0 con 0x7fd158102cd0 2024-03-30T05:10:34.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.737+0000 7fd15e600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd158070d40 con 0x7fd158103670 2024-03-30T05:10:34.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.737+0000 7fd157400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd158102cd0 0x7fd1580702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:34.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.737+0000 7fd157400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd158102cd0 0x7fd1580702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39770/0 (socket says 172.21.15.123:39770) 2024-03-30T05:10:34.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.737+0000 7fd157400700 1 -- 172.21.15.123:0/1845902287 learned_addr learned my addr 172.21.15.123:0/1845902287 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:10:34.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.737+0000 7fd157400700 1 -- 172.21.15.123:0/1845902287 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd158103670 msgr2=0x7fd158070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:34.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.737+0000 7fd157400700 1 --2- 172.21.15.123:0/1845902287 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd158103670 0x7fd158070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:34.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.737+0000 7fd157400700 1 -- 172.21.15.123:0/1845902287 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd148005040 con 0x7fd158102cd0 2024-03-30T05:10:34.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.738+0000 7fd157400700 1 --2- 172.21.15.123:0/1845902287 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd158102cd0 0x7fd1580702c0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fd148009390 tx=0x7fd148009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:34.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.738+0000 7fd14f400700 1 -- 172.21.15.123:0/1845902287 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd14800e040 con 0x7fd158102cd0 2024-03-30T05:10:34.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.738+0000 7fd14f400700 1 -- 172.21.15.123:0/1845902287 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd148013420 con 0x7fd158102cd0 2024-03-30T05:10:34.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.738+0000 7fd14f400700 1 -- 172.21.15.123:0/1845902287 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd14801c630 con 0x7fd158102cd0 2024-03-30T05:10:34.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.738+0000 7fd15e600700 1 -- 172.21.15.123:0/1845902287 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd158070fc0 con 0x7fd158102cd0 2024-03-30T05:10:34.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.738+0000 7fd15e600700 1 -- 172.21.15.123:0/1845902287 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd158119fb0 con 0x7fd158102cd0 2024-03-30T05:10:34.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.739+0000 7fd15e600700 1 -- 172.21.15.123:0/1845902287 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1580fe420 con 0x7fd158102cd0 2024-03-30T05:10:34.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.740+0000 7fd14f400700 1 -- 172.21.15.123:0/1845902287 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fd1480139e0 con 0x7fd158102cd0 2024-03-30T05:10:34.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.741+0000 7fd14f400700 1 --2- 172.21.15.123:0/1845902287 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd13c06cc20 0x7fd13c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:34.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.741+0000 7fd14f400700 1 -- 172.21.15.123:0/1845902287 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd14808d310 con 0x7fd158102cd0 2024-03-30T05:10:34.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.741+0000 7fd156a00700 1 --2- 172.21.15.123:0/1845902287 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd13c06cc20 0x7fd13c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:34.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.742+0000 7fd156a00700 1 --2- 172.21.15.123:0/1845902287 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd13c06cc20 0x7fd13c06f0d0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fd144006fd0 tx=0x7fd144006db0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:34.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.745+0000 7fd14f400700 1 -- 172.21.15.123:0/1845902287 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd1480572d0 con 0x7fd158102cd0 2024-03-30T05:10:34.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.995+0000 7fd15e600700 1 -- 172.21.15.123:0/1845902287 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 30, "format": "json"} v 0) v1 -- 0x7fd158064970 con 0x7fd158102cd0 2024-03-30T05:10:34.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.996+0000 7fd14f400700 1 -- 172.21.15.123:0/1845902287 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 30, "format": "json"}]=0 dumped fsmap epoch 30 v42) v1 ==== 107+0+4833 (secure 0 0 0) 0x7fd14805a8f0 con 0x7fd158102cd0 2024-03-30T05:10:34.994 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:10:34.994 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34420,"name":"cephfs.smithi123.tqxvzr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6827/1741131598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1741131598},{"type":"v1","addr":"172.21.15.123:6827","nonce":1741131598}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":34452,"name":"cephfs.smithi123.dvtind","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6829/4108919307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":4108919307},{"type":"v1","addr":"172.21.15.123:6829","nonce":4108919307}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":44315,"name":"cephfs.smithi135.xwgjvb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6827/1741352336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":1741352336},{"type":"v1","addr":"172.21.15.135:6827","nonce":1741352336}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25},{"gid":44323,"name":"cephfs.smithi135.zvnlap","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6825/2944944016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":2944944016},{"type":"v1","addr":"172.21.15.135:6825","nonce":2944944016}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T04:24:30.211769+0000","modified":"2024-03-30T04:37:19.516763+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-30T05:10:34.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.999+0000 7fd15e600700 1 -- 172.21.15.123:0/1845902287 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd13c06cc20 msgr2=0x7fd13c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:34.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.999+0000 7fd15e600700 1 --2- 172.21.15.123:0/1845902287 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd13c06cc20 0x7fd13c06f0d0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fd144006fd0 tx=0x7fd144006db0 comp rx=0 tx=0).stop 2024-03-30T05:10:34.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.999+0000 7fd15e600700 1 -- 172.21.15.123:0/1845902287 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd158102cd0 msgr2=0x7fd1580702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:34.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.999+0000 7fd15e600700 1 --2- 172.21.15.123:0/1845902287 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd158102cd0 0x7fd1580702c0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fd148009390 tx=0x7fd148009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:34.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.999+0000 7fd15e600700 1 -- 172.21.15.123:0/1845902287 shutdown_connections 2024-03-30T05:10:34.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.999+0000 7fd15e600700 1 --2- 172.21.15.123:0/1845902287 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fd13c06cc20 0x7fd13c06f0d0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:34.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.999+0000 7fd15e600700 1 --2- 172.21.15.123:0/1845902287 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd158102cd0 0x7fd1580702c0 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:34.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.999+0000 7fd15e600700 1 --2- 172.21.15.123:0/1845902287 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd158103670 0x7fd158070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:34.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.999+0000 7fd15e600700 1 -- 172.21.15.123:0/1845902287 >> 172.21.15.123:0/1845902287 conn(0x7fd1580f8820 msgr2=0x7fd1580f98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:34.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.999+0000 7fd15e600700 1 -- 172.21.15.123:0/1845902287 shutdown_connections 2024-03-30T05:10:34.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:34.999+0000 7fd15e600700 1 -- 172.21.15.123:0/1845902287 wait complete. 2024-03-30T05:10:34.998 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 30 2024-03-30T05:10:35.015 DEBUG:tasks.fs:max_mds not reduced in epoch 30 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-03-30T05:10:35.015 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 30 2024-03-30T05:10:35.015 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 31 2024-03-30T05:10:35.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.207+0000 7fdab2400700 1 -- 172.21.15.123:0/2912760575 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdaac102ce0 msgr2=0x7fdaac1030b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:35.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.207+0000 7fdab2400700 1 --2- 172.21.15.123:0/2912760575 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdaac102ce0 0x7fdaac1030b0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fda9c004940 tx=0x7fda9c009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:35.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.207+0000 7fdab2400700 1 -- 172.21.15.123:0/2912760575 shutdown_connections 2024-03-30T05:10:35.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.207+0000 7fdab2400700 1 --2- 172.21.15.123:0/2912760575 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdaac103680 0x7fdaac064e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:35.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.207+0000 7fdab2400700 1 --2- 172.21.15.123:0/2912760575 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdaac102ce0 0x7fdaac1030b0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:35.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.207+0000 7fdab2400700 1 -- 172.21.15.123:0/2912760575 >> 172.21.15.123:0/2912760575 conn(0x7fdaac072ad0 msgr2=0x7fdaac072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:35.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.208+0000 7fdab2400700 1 -- 172.21.15.123:0/2912760575 shutdown_connections 2024-03-30T05:10:35.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.208+0000 7fdab2400700 1 -- 172.21.15.123:0/2912760575 wait complete. 2024-03-30T05:10:35.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.208+0000 7fdab2400700 1 Processor -- start 2024-03-30T05:10:35.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.208+0000 7fdab2400700 1 -- start start 2024-03-30T05:10:35.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.209+0000 7fdab2400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdaac102ce0 0x7fdaac0fbc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:35.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.209+0000 7fdab2400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdaac103680 0x7fdaac0fc180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:35.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.209+0000 7fdab2400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaac0fdc20 con 0x7fdaac103680 2024-03-30T05:10:35.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.209+0000 7fdab2400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaac0fc6c0 con 0x7fdaac102ce0 2024-03-30T05:10:35.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.209+0000 7fdaaaa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdaac103680 0x7fdaac0fc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:35.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.209+0000 7fdaaaa00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdaac103680 0x7fdaac0fc180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39784/0 (socket says 172.21.15.123:39784) 2024-03-30T05:10:35.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.209+0000 7fdaaaa00700 1 -- 172.21.15.123:0/941041730 learned_addr learned my addr 172.21.15.123:0/941041730 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:10:35.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.209+0000 7fdaab400700 1 --2- 172.21.15.123:0/941041730 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdaac102ce0 0x7fdaac0fbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:35.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.209+0000 7fdaaaa00700 1 -- 172.21.15.123:0/941041730 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdaac102ce0 msgr2=0x7fdaac0fbc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:35.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.209+0000 7fdaaaa00700 1 --2- 172.21.15.123:0/941041730 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdaac102ce0 0x7fdaac0fbc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:35.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.209+0000 7fdaaaa00700 1 -- 172.21.15.123:0/941041730 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda9c005040 con 0x7fdaac103680 2024-03-30T05:10:35.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.209+0000 7fdaab400700 1 --2- 172.21.15.123:0/941041730 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdaac102ce0 0x7fdaac0fbc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T05:10:35.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.210+0000 7fdaaaa00700 1 --2- 172.21.15.123:0/941041730 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdaac103680 0x7fdaac0fc180 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fda9800b790 tx=0x7fda98009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:35.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.210+0000 7fdaa3400700 1 -- 172.21.15.123:0/941041730 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda9800f6f0 con 0x7fdaac103680 2024-03-30T05:10:35.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.210+0000 7fdaa3400700 1 -- 172.21.15.123:0/941041730 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fda9800fd00 con 0x7fdaac103680 2024-03-30T05:10:35.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.210+0000 7fdab2400700 1 -- 172.21.15.123:0/941041730 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaac0fc9a0 con 0x7fdaac103680 2024-03-30T05:10:35.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.210+0000 7fdaa3400700 1 -- 172.21.15.123:0/941041730 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda980103a0 con 0x7fdaac103680 2024-03-30T05:10:35.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.210+0000 7fdab2400700 1 -- 172.21.15.123:0/941041730 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaac119fe0 con 0x7fdaac103680 2024-03-30T05:10:35.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.211+0000 7fdab2400700 1 -- 172.21.15.123:0/941041730 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaac0fe430 con 0x7fdaac103680 2024-03-30T05:10:35.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.212+0000 7fdaa3400700 1 -- 172.21.15.123:0/941041730 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fda9800f850 con 0x7fdaac103680 2024-03-30T05:10:35.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.213+0000 7fdaa3400700 1 --2- 172.21.15.123:0/941041730 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fda9006cc20 0x7fda9006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:35.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.213+0000 7fdaa3400700 1 -- 172.21.15.123:0/941041730 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fda9808bee0 con 0x7fdaac103680 2024-03-30T05:10:35.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.213+0000 7fdaab400700 1 --2- 172.21.15.123:0/941041730 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fda9006cc20 0x7fda9006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:35.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.214+0000 7fdaab400700 1 --2- 172.21.15.123:0/941041730 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fda9006cc20 0x7fda9006f0d0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fda9c00b840 tx=0x7fda9c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:35.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.217+0000 7fdaa3400700 1 -- 172.21.15.123:0/941041730 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fda98055ea0 con 0x7fdaac103680 2024-03-30T05:10:35.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.466+0000 7fdab2400700 1 -- 172.21.15.123:0/941041730 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 31, "format": "json"} v 0) v1 -- 0x7fdaac064970 con 0x7fdaac103680 2024-03-30T05:10:35.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.467+0000 7fdaa3400700 1 -- 172.21.15.123:0/941041730 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 31, "format": "json"}]=0 dumped fsmap epoch 31 v42) v1 ==== 107+0+4051 (secure 0 0 0) 0x7fda980594c0 con 0x7fdaac103680 2024-03-30T05:10:35.465 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:10:35.465 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34420,"name":"cephfs.smithi123.tqxvzr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6827/1741131598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1741131598},{"type":"v1","addr":"172.21.15.123:6827","nonce":1741131598}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":34452,"name":"cephfs.smithi123.dvtind","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6829/4108919307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":4108919307},{"type":"v1","addr":"172.21.15.123:6829","nonce":4108919307}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":44315,"name":"cephfs.smithi135.xwgjvb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6827/1741352336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":1741352336},{"type":"v1","addr":"172.21.15.135:6827","nonce":1741352336}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":25,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T04:24:30.211769+0000","modified":"2024-03-30T04:37:19.516763+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-30T05:10:35.465 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:35 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/1730611322' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 29, "format": "json"}]: dispatch 2024-03-30T05:10:35.465 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:35 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/1845902287' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 30, "format": "json"}]: dispatch 2024-03-30T05:10:35.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.469+0000 7fdab2400700 1 -- 172.21.15.123:0/941041730 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fda9006cc20 msgr2=0x7fda9006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:35.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.469+0000 7fdab2400700 1 --2- 172.21.15.123:0/941041730 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fda9006cc20 0x7fda9006f0d0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fda9c00b840 tx=0x7fda9c009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:35.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.469+0000 7fdab2400700 1 -- 172.21.15.123:0/941041730 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdaac103680 msgr2=0x7fdaac0fc180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:35.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.469+0000 7fdab2400700 1 --2- 172.21.15.123:0/941041730 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdaac103680 0x7fdaac0fc180 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fda9800b790 tx=0x7fda98009220 comp rx=0 tx=0).stop 2024-03-30T05:10:35.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.469+0000 7fdab2400700 1 -- 172.21.15.123:0/941041730 shutdown_connections 2024-03-30T05:10:35.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.469+0000 7fdab2400700 1 --2- 172.21.15.123:0/941041730 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fda9006cc20 0x7fda9006f0d0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:35.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.469+0000 7fdab2400700 1 --2- 172.21.15.123:0/941041730 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdaac102ce0 0x7fdaac0fbc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:35.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.469+0000 7fdab2400700 1 --2- 172.21.15.123:0/941041730 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdaac103680 0x7fdaac0fc180 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:35.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.469+0000 7fdab2400700 1 -- 172.21.15.123:0/941041730 >> 172.21.15.123:0/941041730 conn(0x7fdaac072ad0 msgr2=0x7fdaac105a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:35.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.469+0000 7fdab2400700 1 -- 172.21.15.123:0/941041730 shutdown_connections 2024-03-30T05:10:35.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.469+0000 7fdab2400700 1 -- 172.21.15.123:0/941041730 wait complete. 2024-03-30T05:10:35.468 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 31 2024-03-30T05:10:35.485 DEBUG:tasks.fs:max_mds not reduced in epoch 31 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-03-30T05:10:35.485 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 31 2024-03-30T05:10:35.485 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 32 2024-03-30T05:10:35.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:35 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/1730611322' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 29, "format": "json"}]: dispatch 2024-03-30T05:10:35.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:35 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/1845902287' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 30, "format": "json"}]: dispatch 2024-03-30T05:10:35.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.677+0000 7f2792000700 1 -- 172.21.15.123:0/326542799 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f278c1036a0 msgr2=0x7f278c0fcd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:35.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.677+0000 7f2792000700 1 --2- 172.21.15.123:0/326542799 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f278c1036a0 0x7f278c0fcd10 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f2778004940 tx=0x7f2778009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:35.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.677+0000 7f2792000700 1 -- 172.21.15.123:0/326542799 shutdown_connections 2024-03-30T05:10:35.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.677+0000 7f2792000700 1 --2- 172.21.15.123:0/326542799 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f278c1036a0 0x7f278c0fcd10 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:35.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.677+0000 7f2792000700 1 --2- 172.21.15.123:0/326542799 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f278c102d00 0x7f278c1030d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:35.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.677+0000 7f2792000700 1 -- 172.21.15.123:0/326542799 >> 172.21.15.123:0/326542799 conn(0x7f278c0f8810 msgr2=0x7f278c0fac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:35.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.677+0000 7f2792000700 1 -- 172.21.15.123:0/326542799 shutdown_connections 2024-03-30T05:10:35.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.677+0000 7f2792000700 1 -- 172.21.15.123:0/326542799 wait complete. 2024-03-30T05:10:35.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.678+0000 7f2792000700 1 Processor -- start 2024-03-30T05:10:35.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.678+0000 7f2792000700 1 -- start start 2024-03-30T05:10:35.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.678+0000 7f2792000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f278c102d00 0x7f278c115930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:35.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.678+0000 7f2792000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f278c1036a0 0x7f278c115e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:35.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.678+0000 7f2792000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f278c1164c0 con 0x7f278c102d00 2024-03-30T05:10:35.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.678+0000 7f2792000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f278c10fac0 con 0x7f278c1036a0 2024-03-30T05:10:35.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.679+0000 7f278b400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f278c102d00 0x7f278c115930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:35.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.679+0000 7f278b400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f278c102d00 0x7f278c115930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39798/0 (socket says 172.21.15.123:39798) 2024-03-30T05:10:35.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.679+0000 7f278b400700 1 -- 172.21.15.123:0/2728138464 learned_addr learned my addr 172.21.15.123:0/2728138464 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:10:35.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.679+0000 7f278b400700 1 -- 172.21.15.123:0/2728138464 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f278c1036a0 msgr2=0x7f278c115e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:35.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.679+0000 7f278b400700 1 --2- 172.21.15.123:0/2728138464 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f278c1036a0 0x7f278c115e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:35.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.679+0000 7f278b400700 1 -- 172.21.15.123:0/2728138464 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2778005040 con 0x7f278c102d00 2024-03-30T05:10:35.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.679+0000 7f278b400700 1 --2- 172.21.15.123:0/2728138464 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f278c102d00 0x7f278c115930 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f277c00eb40 tx=0x7f277c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:35.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.679+0000 7f2783400700 1 -- 172.21.15.123:0/2728138464 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f277c007cc0 con 0x7f278c102d00 2024-03-30T05:10:35.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.679+0000 7f2783400700 1 -- 172.21.15.123:0/2728138464 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f277c00f040 con 0x7f278c102d00 2024-03-30T05:10:35.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.679+0000 7f2792000700 1 -- 172.21.15.123:0/2728138464 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f278c10fd40 con 0x7f278c102d00 2024-03-30T05:10:35.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.679+0000 7f2783400700 1 -- 172.21.15.123:0/2728138464 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f277c009570 con 0x7f278c102d00 2024-03-30T05:10:35.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.680+0000 7f2792000700 1 -- 172.21.15.123:0/2728138464 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f278c110300 con 0x7f278c102d00 2024-03-30T05:10:35.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.681+0000 7f2792000700 1 -- 172.21.15.123:0/2728138464 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f278c0fe450 con 0x7f278c102d00 2024-03-30T05:10:35.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.681+0000 7f2783400700 1 -- 172.21.15.123:0/2728138464 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f277c0096d0 con 0x7f278c102d00 2024-03-30T05:10:35.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.682+0000 7f2783400700 1 --2- 172.21.15.123:0/2728138464 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f277006cc20 0x7f277006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:35.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.682+0000 7f2783400700 1 -- 172.21.15.123:0/2728138464 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f277c08bcb0 con 0x7f278c102d00 2024-03-30T05:10:35.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.682+0000 7f278aa00700 1 --2- 172.21.15.123:0/2728138464 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f277006cc20 0x7f277006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:35.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.683+0000 7f278aa00700 1 --2- 172.21.15.123:0/2728138464 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f277006cc20 0x7f277006f0d0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f277800b0d0 tx=0x7f2778009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:35.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.687+0000 7f2783400700 1 -- 172.21.15.123:0/2728138464 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f277c0597b0 con 0x7f278c102d00 2024-03-30T05:10:35.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.937+0000 7f2792000700 1 -- 172.21.15.123:0/2728138464 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 32, "format": "json"} v 0) v1 -- 0x7f278c109ca0 con 0x7f278c102d00 2024-03-30T05:10:35.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.938+0000 7f2783400700 1 -- 172.21.15.123:0/2728138464 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 32, "format": "json"}]=0 dumped fsmap epoch 32 v42) v1 ==== 107+0+4830 (secure 0 0 0) 0x7f277c059340 con 0x7f278c102d00 2024-03-30T05:10:35.936 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:10:35.936 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34420,"name":"cephfs.smithi123.tqxvzr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6827/1741131598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1741131598},{"type":"v1","addr":"172.21.15.123:6827","nonce":1741131598}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":34452,"name":"cephfs.smithi123.dvtind","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6829/4108919307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":4108919307},{"type":"v1","addr":"172.21.15.123:6829","nonce":4108919307}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":44315,"name":"cephfs.smithi135.xwgjvb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6827/1741352336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":1741352336},{"type":"v1","addr":"172.21.15.135:6827","nonce":1741352336}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25},{"gid":44355,"name":"cephfs.smithi135.zvnlap","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6825/978884711","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":978884711},{"type":"v1","addr":"172.21.15.135:6825","nonce":978884711}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":25,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T04:24:30.211769+0000","modified":"2024-03-30T04:37:19.516763+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-30T05:10:35.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.940+0000 7f2792000700 1 -- 172.21.15.123:0/2728138464 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f277006cc20 msgr2=0x7f277006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:35.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.940+0000 7f2792000700 1 --2- 172.21.15.123:0/2728138464 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f277006cc20 0x7f277006f0d0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f277800b0d0 tx=0x7f2778009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:35.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.940+0000 7f2792000700 1 -- 172.21.15.123:0/2728138464 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f278c102d00 msgr2=0x7f278c115930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:35.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.940+0000 7f2792000700 1 --2- 172.21.15.123:0/2728138464 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f278c102d00 0x7f278c115930 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f277c00eb40 tx=0x7f277c00b040 comp rx=0 tx=0).stop 2024-03-30T05:10:35.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.940+0000 7f2792000700 1 -- 172.21.15.123:0/2728138464 shutdown_connections 2024-03-30T05:10:35.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.940+0000 7f2792000700 1 --2- 172.21.15.123:0/2728138464 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f277006cc20 0x7f277006f0d0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:35.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.940+0000 7f2792000700 1 --2- 172.21.15.123:0/2728138464 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f278c102d00 0x7f278c115930 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:35.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.940+0000 7f2792000700 1 --2- 172.21.15.123:0/2728138464 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f278c1036a0 0x7f278c115e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:35.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.940+0000 7f2792000700 1 -- 172.21.15.123:0/2728138464 >> 172.21.15.123:0/2728138464 conn(0x7f278c0f8810 msgr2=0x7f278c0f9790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:35.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.940+0000 7f2792000700 1 -- 172.21.15.123:0/2728138464 shutdown_connections 2024-03-30T05:10:35.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:35.940+0000 7f2792000700 1 -- 172.21.15.123:0/2728138464 wait complete. 2024-03-30T05:10:35.939 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 32 2024-03-30T05:10:35.957 DEBUG:tasks.fs:max_mds not reduced in epoch 32 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-03-30T05:10:35.957 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 32 2024-03-30T05:10:35.957 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 33 2024-03-30T05:10:36.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.150+0000 7f2548c00700 1 -- 172.21.15.123:0/4065340100 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2544103670 msgr2=0x7f25440fcce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:36.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.150+0000 7f2548c00700 1 --2- 172.21.15.123:0/4065340100 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2544103670 0x7f25440fcce0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f2530004940 tx=0x7f2530009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:36.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.150+0000 7f2548c00700 1 -- 172.21.15.123:0/4065340100 shutdown_connections 2024-03-30T05:10:36.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.150+0000 7f2548c00700 1 --2- 172.21.15.123:0/4065340100 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2544103670 0x7f25440fcce0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:36.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.150+0000 7f2548c00700 1 --2- 172.21.15.123:0/4065340100 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2544102cd0 0x7f25441030a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:36.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.150+0000 7f2548c00700 1 -- 172.21.15.123:0/4065340100 >> 172.21.15.123:0/4065340100 conn(0x7f25440f8820 msgr2=0x7f25440fac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:36.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.150+0000 7f2548c00700 1 -- 172.21.15.123:0/4065340100 shutdown_connections 2024-03-30T05:10:36.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.150+0000 7f2548c00700 1 -- 172.21.15.123:0/4065340100 wait complete. 2024-03-30T05:10:36.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.151+0000 7f2548c00700 1 Processor -- start 2024-03-30T05:10:36.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.151+0000 7f2548c00700 1 -- start start 2024-03-30T05:10:36.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.151+0000 7f2548c00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2544102cd0 0x7f2544113780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:36.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.151+0000 7f2548c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2544103670 0x7f2544113cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:36.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.151+0000 7f2548c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2544114310 con 0x7f2544103670 2024-03-30T05:10:36.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.151+0000 7f2548c00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f254410d910 con 0x7f2544102cd0 2024-03-30T05:10:36.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.151+0000 7f2541400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2544103670 0x7f2544113cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:36.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.151+0000 7f2541e00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2544102cd0 0x7f2544113780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:36.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.151+0000 7f2541400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2544103670 0x7f2544113cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39806/0 (socket says 172.21.15.123:39806) 2024-03-30T05:10:36.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.151+0000 7f2541400700 1 -- 172.21.15.123:0/3873311770 learned_addr learned my addr 172.21.15.123:0/3873311770 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:10:36.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.152+0000 7f2541400700 1 -- 172.21.15.123:0/3873311770 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2544102cd0 msgr2=0x7f2544113780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:36.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.152+0000 7f2541400700 1 --2- 172.21.15.123:0/3873311770 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2544102cd0 0x7f2544113780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:36.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.152+0000 7f2541400700 1 -- 172.21.15.123:0/3873311770 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2530005040 con 0x7f2544103670 2024-03-30T05:10:36.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.152+0000 7f2541400700 1 --2- 172.21.15.123:0/3873311770 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2544103670 0x7f2544113cc0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f2530009390 tx=0x7f2530009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:36.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.152+0000 7f253a000700 1 -- 172.21.15.123:0/3873311770 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f253000e040 con 0x7f2544103670 2024-03-30T05:10:36.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.152+0000 7f253a000700 1 -- 172.21.15.123:0/3873311770 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2530004290 con 0x7f2544103670 2024-03-30T05:10:36.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.152+0000 7f2548c00700 1 -- 172.21.15.123:0/3873311770 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f254410db90 con 0x7f2544103670 2024-03-30T05:10:36.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.152+0000 7f253a000700 1 -- 172.21.15.123:0/3873311770 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2530013e70 con 0x7f2544103670 2024-03-30T05:10:36.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.152+0000 7f2548c00700 1 -- 172.21.15.123:0/3873311770 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f254410e150 con 0x7f2544103670 2024-03-30T05:10:36.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.153+0000 7f2548c00700 1 -- 172.21.15.123:0/3873311770 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25440fe420 con 0x7f2544103670 2024-03-30T05:10:36.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.154+0000 7f253a000700 1 -- 172.21.15.123:0/3873311770 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f2530013690 con 0x7f2544103670 2024-03-30T05:10:36.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.155+0000 7f253a000700 1 --2- 172.21.15.123:0/3873311770 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f252806cbc0 0x7f252806f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:36.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.155+0000 7f253a000700 1 -- 172.21.15.123:0/3873311770 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f253008d2c0 con 0x7f2544103670 2024-03-30T05:10:36.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.155+0000 7f2541e00700 1 --2- 172.21.15.123:0/3873311770 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f252806cbc0 0x7f252806f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:36.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.156+0000 7f2541e00700 1 --2- 172.21.15.123:0/3873311770 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f252806cbc0 0x7f252806f070 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f2534004f40 tx=0x7f253400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:36.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.160+0000 7f253a000700 1 -- 172.21.15.123:0/3873311770 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f253005adc0 con 0x7f2544103670 2024-03-30T05:10:36.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:36 smithi123 ceph-mon[139369]: pgmap v1184: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:10:36.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:36 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/941041730' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 31, "format": "json"}]: dispatch 2024-03-30T05:10:36.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:36 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2728138464' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 32, "format": "json"}]: dispatch 2024-03-30T05:10:36.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.409+0000 7f2548c00700 1 -- 172.21.15.123:0/3873311770 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 33, "format": "json"} v 0) v1 -- 0x7f2544109c70 con 0x7f2544103670 2024-03-30T05:10:36.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.410+0000 7f253a000700 1 -- 172.21.15.123:0/3873311770 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 33, "format": "json"}]=0 dumped fsmap epoch 33 v42) v1 ==== 107+0+4048 (secure 0 0 0) 0x7f2530013420 con 0x7f2544103670 2024-03-30T05:10:36.408 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:10:36.408 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34420,"name":"cephfs.smithi123.tqxvzr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6827/1741131598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1741131598},{"type":"v1","addr":"172.21.15.123:6827","nonce":1741131598}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":34452,"name":"cephfs.smithi123.dvtind","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6829/4108919307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":4108919307},{"type":"v1","addr":"172.21.15.123:6829","nonce":4108919307}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":44355,"name":"cephfs.smithi135.zvnlap","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6825/978884711","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":978884711},{"type":"v1","addr":"172.21.15.135:6825","nonce":978884711}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":25,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T04:24:30.211769+0000","modified":"2024-03-30T04:37:19.516763+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-30T05:10:36.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.412+0000 7f2548c00700 1 -- 172.21.15.123:0/3873311770 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f252806cbc0 msgr2=0x7f252806f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:36.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.412+0000 7f2548c00700 1 --2- 172.21.15.123:0/3873311770 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f252806cbc0 0x7f252806f070 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f2534004f40 tx=0x7f253400b040 comp rx=0 tx=0).stop 2024-03-30T05:10:36.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.412+0000 7f2548c00700 1 -- 172.21.15.123:0/3873311770 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2544103670 msgr2=0x7f2544113cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:36.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.412+0000 7f2548c00700 1 --2- 172.21.15.123:0/3873311770 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2544103670 0x7f2544113cc0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f2530009390 tx=0x7f2530009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:36.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.412+0000 7f2548c00700 1 -- 172.21.15.123:0/3873311770 shutdown_connections 2024-03-30T05:10:36.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.412+0000 7f2548c00700 1 --2- 172.21.15.123:0/3873311770 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f252806cbc0 0x7f252806f070 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:36.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.412+0000 7f2548c00700 1 --2- 172.21.15.123:0/3873311770 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2544102cd0 0x7f2544113780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:36.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.412+0000 7f2548c00700 1 --2- 172.21.15.123:0/3873311770 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2544103670 0x7f2544113cc0 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:36.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.412+0000 7f2548c00700 1 -- 172.21.15.123:0/3873311770 >> 172.21.15.123:0/3873311770 conn(0x7f25440f8820 msgr2=0x7f25440f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:36.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.412+0000 7f2548c00700 1 -- 172.21.15.123:0/3873311770 shutdown_connections 2024-03-30T05:10:36.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.412+0000 7f2548c00700 1 -- 172.21.15.123:0/3873311770 wait complete. 2024-03-30T05:10:36.412 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 33 2024-03-30T05:10:36.428 DEBUG:tasks.fs:max_mds not reduced in epoch 33 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-03-30T05:10:36.428 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 33 2024-03-30T05:10:36.428 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 34 2024-03-30T05:10:36.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:36 smithi135 ceph-mon[106011]: pgmap v1184: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:10:36.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:36 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/941041730' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 31, "format": "json"}]: dispatch 2024-03-30T05:10:36.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:36 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2728138464' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 32, "format": "json"}]: dispatch 2024-03-30T05:10:36.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.620+0000 7f2fc0400700 1 -- 172.21.15.123:0/3709590351 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2fb8070360 msgr2=0x7f2fb8070730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:36.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.620+0000 7f2fc0400700 1 --2- 172.21.15.123:0/3709590351 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2fb8070360 0x7f2fb8070730 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2fac004940 tx=0x7f2fac009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:36.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.620+0000 7f2fc0400700 1 -- 172.21.15.123:0/3709590351 shutdown_connections 2024-03-30T05:10:36.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.620+0000 7f2fc0400700 1 --2- 172.21.15.123:0/3709590351 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fb8070c70 0x7f2fb8108fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:36.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.620+0000 7f2fc0400700 1 --2- 172.21.15.123:0/3709590351 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2fb8070360 0x7f2fb8070730 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:36.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.620+0000 7f2fc0400700 1 -- 172.21.15.123:0/3709590351 >> 172.21.15.123:0/3709590351 conn(0x7f2fb80f8820 msgr2=0x7f2fb80fac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:36.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.620+0000 7f2fc0400700 1 -- 172.21.15.123:0/3709590351 shutdown_connections 2024-03-30T05:10:36.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.620+0000 7f2fc0400700 1 -- 172.21.15.123:0/3709590351 wait complete. 2024-03-30T05:10:36.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.621+0000 7f2fc0400700 1 Processor -- start 2024-03-30T05:10:36.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.621+0000 7f2fc0400700 1 -- start start 2024-03-30T05:10:36.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.621+0000 7f2fc0400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fb8070360 0x7f2fb8115a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:36.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.621+0000 7f2fc0400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2fb8070c70 0x7f2fb8115f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:36.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.622+0000 7f2fc0400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb81165a0 con 0x7f2fb8070360 2024-03-30T05:10:36.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.622+0000 7f2fc0400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb810fba0 con 0x7f2fb8070c70 2024-03-30T05:10:36.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.622+0000 7f2fbda00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fb8070360 0x7f2fb8115a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:36.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.622+0000 7f2fbda00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fb8070360 0x7f2fb8115a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39820/0 (socket says 172.21.15.123:39820) 2024-03-30T05:10:36.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.622+0000 7f2fbda00700 1 -- 172.21.15.123:0/4010921265 learned_addr learned my addr 172.21.15.123:0/4010921265 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:10:36.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.622+0000 7f2fbd000700 1 --2- 172.21.15.123:0/4010921265 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2fb8070c70 0x7f2fb8115f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:36.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.622+0000 7f2fbda00700 1 -- 172.21.15.123:0/4010921265 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2fb8070c70 msgr2=0x7f2fb8115f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:36.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.622+0000 7f2fbda00700 1 --2- 172.21.15.123:0/4010921265 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2fb8070c70 0x7f2fb8115f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:36.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.622+0000 7f2fbda00700 1 -- 172.21.15.123:0/4010921265 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fac005040 con 0x7f2fb8070360 2024-03-30T05:10:36.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.622+0000 7f2fbd000700 1 --2- 172.21.15.123:0/4010921265 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2fb8070c70 0x7f2fb8115f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T05:10:36.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.622+0000 7f2fbda00700 1 --2- 172.21.15.123:0/4010921265 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fb8070360 0x7f2fb8115a10 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f2fac009fd0 tx=0x7f2fac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:36.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.623+0000 7f2fb5600700 1 -- 172.21.15.123:0/4010921265 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2fac00e030 con 0x7f2fb8070360 2024-03-30T05:10:36.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.623+0000 7f2fb5600700 1 -- 172.21.15.123:0/4010921265 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2fac0093b0 con 0x7f2fb8070360 2024-03-30T05:10:36.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.623+0000 7f2fc0400700 1 -- 172.21.15.123:0/4010921265 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fb810fe20 con 0x7f2fb8070360 2024-03-30T05:10:36.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.623+0000 7f2fb5600700 1 -- 172.21.15.123:0/4010921265 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2fac01ccf0 con 0x7f2fb8070360 2024-03-30T05:10:36.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.623+0000 7f2fc0400700 1 -- 172.21.15.123:0/4010921265 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fb8110360 con 0x7f2fb8070360 2024-03-30T05:10:36.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.624+0000 7f2fc0400700 1 -- 172.21.15.123:0/4010921265 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fb8071aa0 con 0x7f2fb8070360 2024-03-30T05:10:36.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.625+0000 7f2fb5600700 1 -- 172.21.15.123:0/4010921265 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f2fac01c500 con 0x7f2fb8070360 2024-03-30T05:10:36.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.626+0000 7f2fb5600700 1 --2- 172.21.15.123:0/4010921265 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2fa006cc20 0x7f2fa006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:36.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.626+0000 7f2fb5600700 1 -- 172.21.15.123:0/4010921265 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f2fac08d7b0 con 0x7f2fb8070360 2024-03-30T05:10:36.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.626+0000 7f2fbd000700 1 --2- 172.21.15.123:0/4010921265 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2fa006cc20 0x7f2fa006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:36.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.626+0000 7f2fbd000700 1 --2- 172.21.15.123:0/4010921265 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2fa006cc20 0x7f2fa006f0d0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f2fa4004600 tx=0x7f2fa400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:36.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.630+0000 7f2fb5600700 1 -- 172.21.15.123:0/4010921265 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2fac05b240 con 0x7f2fb8070360 2024-03-30T05:10:36.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.878+0000 7f2fc0400700 1 -- 172.21.15.123:0/4010921265 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 34, "format": "json"} v 0) v1 -- 0x7f2fb8110ff0 con 0x7f2fb8070360 2024-03-30T05:10:36.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.878+0000 7f2fb5600700 1 -- 172.21.15.123:0/4010921265 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 34, "format": "json"}]=0 dumped fsmap epoch 34 v42) v1 ==== 107+0+4830 (secure 0 0 0) 0x7f2fac05add0 con 0x7f2fb8070360 2024-03-30T05:10:36.876 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:10:36.876 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34420,"name":"cephfs.smithi123.tqxvzr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6827/1741131598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1741131598},{"type":"v1","addr":"172.21.15.123:6827","nonce":1741131598}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":34452,"name":"cephfs.smithi123.dvtind","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6829/4108919307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":4108919307},{"type":"v1","addr":"172.21.15.123:6829","nonce":4108919307}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":44355,"name":"cephfs.smithi135.zvnlap","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6825/978884711","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":978884711},{"type":"v1","addr":"172.21.15.135:6825","nonce":978884711}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":32},{"gid":44975,"name":"cephfs.smithi135.xwgjvb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6827/3788988014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3788988014},{"type":"v1","addr":"172.21.15.135:6827","nonce":3788988014}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":25,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T04:24:30.211769+0000","modified":"2024-03-30T04:37:19.516763+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-30T05:10:36.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.881+0000 7f2fc0400700 1 -- 172.21.15.123:0/4010921265 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2fa006cc20 msgr2=0x7f2fa006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:36.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.881+0000 7f2fc0400700 1 --2- 172.21.15.123:0/4010921265 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2fa006cc20 0x7f2fa006f0d0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f2fa4004600 tx=0x7f2fa400b040 comp rx=0 tx=0).stop 2024-03-30T05:10:36.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.881+0000 7f2fc0400700 1 -- 172.21.15.123:0/4010921265 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fb8070360 msgr2=0x7f2fb8115a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:36.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.881+0000 7f2fc0400700 1 --2- 172.21.15.123:0/4010921265 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fb8070360 0x7f2fb8115a10 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f2fac009fd0 tx=0x7f2fac009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:36.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.881+0000 7f2fc0400700 1 -- 172.21.15.123:0/4010921265 shutdown_connections 2024-03-30T05:10:36.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.881+0000 7f2fc0400700 1 --2- 172.21.15.123:0/4010921265 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2fa006cc20 0x7f2fa006f0d0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:36.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.881+0000 7f2fc0400700 1 --2- 172.21.15.123:0/4010921265 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fb8070360 0x7f2fb8115a10 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:36.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.881+0000 7f2fc0400700 1 --2- 172.21.15.123:0/4010921265 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2fb8070c70 0x7f2fb8115f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:36.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.881+0000 7f2fc0400700 1 -- 172.21.15.123:0/4010921265 >> 172.21.15.123:0/4010921265 conn(0x7f2fb80f8820 msgr2=0x7f2fb80f9e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:36.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.881+0000 7f2fc0400700 1 -- 172.21.15.123:0/4010921265 shutdown_connections 2024-03-30T05:10:36.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:36.881+0000 7f2fc0400700 1 -- 172.21.15.123:0/4010921265 wait complete. 2024-03-30T05:10:36.880 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 34 2024-03-30T05:10:36.897 DEBUG:tasks.fs:max_mds not reduced in epoch 34 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-03-30T05:10:36.897 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 34 2024-03-30T05:10:36.897 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 35 2024-03-30T05:10:37.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.088+0000 7f4842600700 1 -- 172.21.15.123:0/3224766561 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f483c102cd0 msgr2=0x7f483c1030a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:37.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.088+0000 7f4842600700 1 --2- 172.21.15.123:0/3224766561 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f483c102cd0 0x7f483c1030a0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f482c004940 tx=0x7f482c009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:37.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.088+0000 7f4842600700 1 -- 172.21.15.123:0/3224766561 shutdown_connections 2024-03-30T05:10:37.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.088+0000 7f4842600700 1 --2- 172.21.15.123:0/3224766561 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f483c103670 0x7f483c0fcce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:37.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.088+0000 7f4842600700 1 --2- 172.21.15.123:0/3224766561 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f483c102cd0 0x7f483c1030a0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:37.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.088+0000 7f4842600700 1 -- 172.21.15.123:0/3224766561 >> 172.21.15.123:0/3224766561 conn(0x7f483c0f8820 msgr2=0x7f483c0fac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:37.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.088+0000 7f4842600700 1 -- 172.21.15.123:0/3224766561 shutdown_connections 2024-03-30T05:10:37.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.088+0000 7f4842600700 1 -- 172.21.15.123:0/3224766561 wait complete. 2024-03-30T05:10:37.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.089+0000 7f4842600700 1 Processor -- start 2024-03-30T05:10:37.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.089+0000 7f4842600700 1 -- start start 2024-03-30T05:10:37.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.089+0000 7f4842600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f483c102cd0 0x7f483c115990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:37.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.089+0000 7f4842600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f483c103670 0x7f483c115ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:37.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.089+0000 7f4842600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483c116520 con 0x7f483c103670 2024-03-30T05:10:37.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.089+0000 7f4842600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483c10fb40 con 0x7f483c102cd0 2024-03-30T05:10:37.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.090+0000 7f483b400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f483c103670 0x7f483c115ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:37.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.090+0000 7f483b400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f483c103670 0x7f483c115ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39846/0 (socket says 172.21.15.123:39846) 2024-03-30T05:10:37.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.090+0000 7f483b400700 1 -- 172.21.15.123:0/2013431286 learned_addr learned my addr 172.21.15.123:0/2013431286 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:10:37.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.090+0000 7f4833400700 1 --2- 172.21.15.123:0/2013431286 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f483c102cd0 0x7f483c115990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:37.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.090+0000 7f483b400700 1 -- 172.21.15.123:0/2013431286 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f483c102cd0 msgr2=0x7f483c115990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:37.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.090+0000 7f483b400700 1 --2- 172.21.15.123:0/2013431286 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f483c102cd0 0x7f483c115990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:37.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.090+0000 7f483b400700 1 -- 172.21.15.123:0/2013431286 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f482c005040 con 0x7f483c103670 2024-03-30T05:10:37.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.090+0000 7f483b400700 1 --2- 172.21.15.123:0/2013431286 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f483c103670 0x7f483c115ed0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f4824002a50 tx=0x7f4824004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:37.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.090+0000 7f4833e00700 1 -- 172.21.15.123:0/2013431286 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4824007cc0 con 0x7f483c103670 2024-03-30T05:10:37.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.090+0000 7f4833e00700 1 -- 172.21.15.123:0/2013431286 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f482400f040 con 0x7f483c103670 2024-03-30T05:10:37.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.090+0000 7f4842600700 1 -- 172.21.15.123:0/2013431286 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f483c10fe20 con 0x7f483c103670 2024-03-30T05:10:37.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.090+0000 7f4833e00700 1 -- 172.21.15.123:0/2013431286 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f482400d9a0 con 0x7f483c103670 2024-03-30T05:10:37.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.091+0000 7f4842600700 1 -- 172.21.15.123:0/2013431286 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f483c1103e0 con 0x7f483c103670 2024-03-30T05:10:37.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.091+0000 7f4842600700 1 -- 172.21.15.123:0/2013431286 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f483c0fe420 con 0x7f483c103670 2024-03-30T05:10:37.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.093+0000 7f4833e00700 1 -- 172.21.15.123:0/2013431286 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f482400dc10 con 0x7f483c103670 2024-03-30T05:10:37.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.094+0000 7f4833e00700 1 --2- 172.21.15.123:0/2013431286 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f482006cc20 0x7f482006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:37.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.094+0000 7f4833e00700 1 -- 172.21.15.123:0/2013431286 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f482408bdb0 con 0x7f483c103670 2024-03-30T05:10:37.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.094+0000 7f4833400700 1 --2- 172.21.15.123:0/2013431286 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f482006cc20 0x7f482006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:37.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.094+0000 7f4833400700 1 --2- 172.21.15.123:0/2013431286 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f482006cc20 0x7f482006f0d0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f482c000c00 tx=0x7f482c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:37.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.098+0000 7f4833e00700 1 -- 172.21.15.123:0/2013431286 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4824055df0 con 0x7f483c103670 2024-03-30T05:10:37.215 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:37 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3873311770' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 33, "format": "json"}]: dispatch 2024-03-30T05:10:37.215 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:37 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/4010921265' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 34, "format": "json"}]: dispatch 2024-03-30T05:10:37.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.348+0000 7f4842600700 1 -- 172.21.15.123:0/2013431286 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 35, "format": "json"} v 0) v1 -- 0x7f483c110650 con 0x7f483c103670 2024-03-30T05:10:37.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.348+0000 7f4833e00700 1 -- 172.21.15.123:0/2013431286 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 35, "format": "json"}]=0 dumped fsmap epoch 35 v42) v1 ==== 107+0+4829 (secure 0 0 0) 0x7f4824059410 con 0x7f483c103670 2024-03-30T05:10:37.346 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:10:37.347 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34420,"name":"cephfs.smithi123.tqxvzr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6827/1741131598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1741131598},{"type":"v1","addr":"172.21.15.123:6827","nonce":1741131598}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":34452,"name":"cephfs.smithi123.dvtind","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6829/4108919307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":4108919307},{"type":"v1","addr":"172.21.15.123:6829","nonce":4108919307}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":44355,"name":"cephfs.smithi135.zvnlap","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6825/978884711","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":978884711},{"type":"v1","addr":"172.21.15.135:6825","nonce":978884711}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":32},{"gid":44975,"name":"cephfs.smithi135.xwgjvb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6827/3788988014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3788988014},{"type":"v1","addr":"172.21.15.135:6827","nonce":3788988014}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T04:24:30.211769+0000","modified":"2024-03-30T05:08:24.040725+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-30T05:10:37.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.351+0000 7f4842600700 1 -- 172.21.15.123:0/2013431286 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f482006cc20 msgr2=0x7f482006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:37.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.351+0000 7f4842600700 1 --2- 172.21.15.123:0/2013431286 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f482006cc20 0x7f482006f0d0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f482c000c00 tx=0x7f482c009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:37.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.351+0000 7f4842600700 1 -- 172.21.15.123:0/2013431286 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f483c103670 msgr2=0x7f483c115ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:37.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.351+0000 7f4842600700 1 --2- 172.21.15.123:0/2013431286 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f483c103670 0x7f483c115ed0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f4824002a50 tx=0x7f4824004600 comp rx=0 tx=0).stop 2024-03-30T05:10:37.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.351+0000 7f4842600700 1 -- 172.21.15.123:0/2013431286 shutdown_connections 2024-03-30T05:10:37.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.351+0000 7f4842600700 1 --2- 172.21.15.123:0/2013431286 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f482006cc20 0x7f482006f0d0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:37.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.351+0000 7f4842600700 1 --2- 172.21.15.123:0/2013431286 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f483c102cd0 0x7f483c115990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:37.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.351+0000 7f4842600700 1 --2- 172.21.15.123:0/2013431286 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f483c103670 0x7f483c115ed0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:37.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.351+0000 7f4842600700 1 -- 172.21.15.123:0/2013431286 >> 172.21.15.123:0/2013431286 conn(0x7f483c0f8820 msgr2=0x7f483c0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:37.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.351+0000 7f4842600700 1 -- 172.21.15.123:0/2013431286 shutdown_connections 2024-03-30T05:10:37.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.351+0000 7f4842600700 1 -- 172.21.15.123:0/2013431286 wait complete. 2024-03-30T05:10:37.350 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 35 2024-03-30T05:10:37.367 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 35 2024-03-30T05:10:37.367 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 36 2024-03-30T05:10:37.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.559+0000 7fbf8ac00700 1 -- 172.21.15.123:0/284730522 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf84102b70 msgr2=0x7fbf84102f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:37.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.559+0000 7fbf8ac00700 1 --2- 172.21.15.123:0/284730522 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf84102b70 0x7fbf84102f40 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fbf78004940 tx=0x7fbf78009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:37.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.559+0000 7fbf8ac00700 1 -- 172.21.15.123:0/284730522 shutdown_connections 2024-03-30T05:10:37.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.559+0000 7fbf8ac00700 1 --2- 172.21.15.123:0/284730522 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbf84103510 0x7fbf840fcb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:37.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.559+0000 7fbf8ac00700 1 --2- 172.21.15.123:0/284730522 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf84102b70 0x7fbf84102f40 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:37.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.559+0000 7fbf8ac00700 1 -- 172.21.15.123:0/284730522 >> 172.21.15.123:0/284730522 conn(0x7fbf840f86e0 msgr2=0x7fbf840faaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:37.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.559+0000 7fbf8ac00700 1 -- 172.21.15.123:0/284730522 shutdown_connections 2024-03-30T05:10:37.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.559+0000 7fbf8ac00700 1 -- 172.21.15.123:0/284730522 wait complete. 2024-03-30T05:10:37.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.560+0000 7fbf8ac00700 1 Processor -- start 2024-03-30T05:10:37.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.560+0000 7fbf8ac00700 1 -- start start 2024-03-30T05:10:37.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.560+0000 7fbf8ac00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf84102b70 0x7fbf840702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:37.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.560+0000 7fbf8ac00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbf84103510 0x7fbf84070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:37.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.560+0000 7fbf8ac00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf84072210 con 0x7fbf84102b70 2024-03-30T05:10:37.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.560+0000 7fbf8ac00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf84070d40 con 0x7fbf84103510 2024-03-30T05:10:37.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.561+0000 7fbf89800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf84102b70 0x7fbf840702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:37.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.561+0000 7fbf89800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf84102b70 0x7fbf840702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39864/0 (socket says 172.21.15.123:39864) 2024-03-30T05:10:37.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.561+0000 7fbf89800700 1 -- 172.21.15.123:0/30326977 learned_addr learned my addr 172.21.15.123:0/30326977 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:10:37.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.561+0000 7fbf88e00700 1 --2- 172.21.15.123:0/30326977 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbf84103510 0x7fbf84070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:37.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.561+0000 7fbf89800700 1 -- 172.21.15.123:0/30326977 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbf84103510 msgr2=0x7fbf84070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:37.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.561+0000 7fbf89800700 1 --2- 172.21.15.123:0/30326977 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbf84103510 0x7fbf84070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:37.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.561+0000 7fbf89800700 1 -- 172.21.15.123:0/30326977 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf78005040 con 0x7fbf84102b70 2024-03-30T05:10:37.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.561+0000 7fbf88e00700 1 --2- 172.21.15.123:0/30326977 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbf84103510 0x7fbf84070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T05:10:37.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.561+0000 7fbf89800700 1 --2- 172.21.15.123:0/30326977 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf84102b70 0x7fbf840702c0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fbf78009390 tx=0x7fbf78009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:37.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.561+0000 7fbf81600700 1 -- 172.21.15.123:0/30326977 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf7800e040 con 0x7fbf84102b70 2024-03-30T05:10:37.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.561+0000 7fbf81600700 1 -- 172.21.15.123:0/30326977 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbf78013420 con 0x7fbf84102b70 2024-03-30T05:10:37.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.562+0000 7fbf8ac00700 1 -- 172.21.15.123:0/30326977 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf84070fc0 con 0x7fbf84102b70 2024-03-30T05:10:37.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.562+0000 7fbf81600700 1 -- 172.21.15.123:0/30326977 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf7801c630 con 0x7fbf84102b70 2024-03-30T05:10:37.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.562+0000 7fbf8ac00700 1 -- 172.21.15.123:0/30326977 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf84119d30 con 0x7fbf84102b70 2024-03-30T05:10:37.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.563+0000 7fbf8ac00700 1 -- 172.21.15.123:0/30326977 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf840fe2c0 con 0x7fbf84102b70 2024-03-30T05:10:37.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.563+0000 7fbf81600700 1 -- 172.21.15.123:0/30326977 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fbf780139e0 con 0x7fbf84102b70 2024-03-30T05:10:37.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.564+0000 7fbf81600700 1 --2- 172.21.15.123:0/30326977 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbf6c06cc30 0x7fbf6c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:37.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.564+0000 7fbf81600700 1 -- 172.21.15.123:0/30326977 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fbf7808d330 con 0x7fbf84102b70 2024-03-30T05:10:37.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.565+0000 7fbf88e00700 1 --2- 172.21.15.123:0/30326977 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbf6c06cc30 0x7fbf6c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:37.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.565+0000 7fbf88e00700 1 --2- 172.21.15.123:0/30326977 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbf6c06cc30 0x7fbf6c06f0e0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fbf70009fd0 tx=0x7fbf70009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:37.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.569+0000 7fbf81600700 1 -- 172.21.15.123:0/30326977 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fbf780572f0 con 0x7fbf84102b70 2024-03-30T05:10:37.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:37 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3873311770' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 33, "format": "json"}]: dispatch 2024-03-30T05:10:37.602 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:37 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/4010921265' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 34, "format": "json"}]: dispatch 2024-03-30T05:10:37.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.818+0000 7fbf8ac00700 1 -- 172.21.15.123:0/30326977 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 36, "format": "json"} v 0) v1 -- 0x7fbf8404c710 con 0x7fbf84102b70 2024-03-30T05:10:37.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.819+0000 7fbf81600700 1 -- 172.21.15.123:0/30326977 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 36, "format": "json"}]=0 dumped fsmap epoch 36 v42) v1 ==== 107+0+4852 (secure 0 0 0) 0x7fbf8404c710 con 0x7fbf84102b70 2024-03-30T05:10:37.817 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:10:37.817 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44355,"name":"cephfs.smithi135.zvnlap","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6825/978884711","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":978884711},{"type":"v1","addr":"172.21.15.135:6825","nonce":978884711}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":32},{"gid":44975,"name":"cephfs.smithi135.xwgjvb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6827/3788988014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3788988014},{"type":"v1","addr":"172.21.15.135:6827","nonce":3788988014}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T04:24:30.211769+0000","modified":"2024-03-30T05:08:24.045845+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34420,"mds_1":34452},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34420":{"gid":34420,"name":"cephfs.smithi123.tqxvzr","rank":0,"incarnation":36,"state":"up:replay","state_seq":1,"addr":"172.21.15.123:6827/1741131598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1741131598},{"type":"v1","addr":"172.21.15.123:6827","nonce":1741131598}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_34452":{"gid":34452,"name":"cephfs.smithi123.dvtind","rank":1,"incarnation":36,"state":"up:replay","state_seq":1,"addr":"172.21.15.123:6829/4108919307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":4108919307},{"type":"v1","addr":"172.21.15.123:6829","nonce":4108919307}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-30T05:10:37.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.821+0000 7fbf8ac00700 1 -- 172.21.15.123:0/30326977 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbf6c06cc30 msgr2=0x7fbf6c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:37.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.821+0000 7fbf8ac00700 1 --2- 172.21.15.123:0/30326977 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbf6c06cc30 0x7fbf6c06f0e0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fbf70009fd0 tx=0x7fbf70009ee0 comp rx=0 tx=0).stop 2024-03-30T05:10:37.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.821+0000 7fbf8ac00700 1 -- 172.21.15.123:0/30326977 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf84102b70 msgr2=0x7fbf840702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:37.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.821+0000 7fbf8ac00700 1 --2- 172.21.15.123:0/30326977 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf84102b70 0x7fbf840702c0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fbf78009390 tx=0x7fbf78009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:37.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.822+0000 7fbf8ac00700 1 -- 172.21.15.123:0/30326977 shutdown_connections 2024-03-30T05:10:37.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.822+0000 7fbf8ac00700 1 --2- 172.21.15.123:0/30326977 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fbf6c06cc30 0x7fbf6c06f0e0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:37.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.822+0000 7fbf8ac00700 1 --2- 172.21.15.123:0/30326977 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf84102b70 0x7fbf840702c0 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:37.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.822+0000 7fbf8ac00700 1 --2- 172.21.15.123:0/30326977 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbf84103510 0x7fbf84070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:37.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.822+0000 7fbf8ac00700 1 -- 172.21.15.123:0/30326977 >> 172.21.15.123:0/30326977 conn(0x7fbf840f86e0 msgr2=0x7fbf840f9800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:37.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.822+0000 7fbf8ac00700 1 -- 172.21.15.123:0/30326977 shutdown_connections 2024-03-30T05:10:37.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:37.822+0000 7fbf8ac00700 1 -- 172.21.15.123:0/30326977 wait complete. 2024-03-30T05:10:37.821 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 36 2024-03-30T05:10:37.837 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 36 2024-03-30T05:10:37.837 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 37 2024-03-30T05:10:38.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.033+0000 7fe828600700 1 -- 172.21.15.123:0/2983845970 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe820103680 msgr2=0x7fe8200fccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:38.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.033+0000 7fe828600700 1 --2- 172.21.15.123:0/2983845970 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe820103680 0x7fe8200fccf0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fe80c004940 tx=0x7fe80c009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:38.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.033+0000 7fe828600700 1 -- 172.21.15.123:0/2983845970 shutdown_connections 2024-03-30T05:10:38.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.033+0000 7fe828600700 1 --2- 172.21.15.123:0/2983845970 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe820103680 0x7fe8200fccf0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:38.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.033+0000 7fe828600700 1 --2- 172.21.15.123:0/2983845970 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe820102ce0 0x7fe8201030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:38.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.033+0000 7fe828600700 1 -- 172.21.15.123:0/2983845970 >> 172.21.15.123:0/2983845970 conn(0x7fe8200f8830 msgr2=0x7fe8200fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:38.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.033+0000 7fe828600700 1 -- 172.21.15.123:0/2983845970 shutdown_connections 2024-03-30T05:10:38.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.033+0000 7fe828600700 1 -- 172.21.15.123:0/2983845970 wait complete. 2024-03-30T05:10:38.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.034+0000 7fe828600700 1 Processor -- start 2024-03-30T05:10:38.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.034+0000 7fe828600700 1 -- start start 2024-03-30T05:10:38.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.034+0000 7fe828600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe820102ce0 0x7fe820115980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:38.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.034+0000 7fe828600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe820103680 0x7fe820115ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:38.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.034+0000 7fe828600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe820116510 con 0x7fe820103680 2024-03-30T05:10:38.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.034+0000 7fe828600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe82010fb10 con 0x7fe820102ce0 2024-03-30T05:10:38.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.034+0000 7fe825200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe820103680 0x7fe820115ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:38.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.034+0000 7fe825200700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe820103680 0x7fe820115ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39894/0 (socket says 172.21.15.123:39894) 2024-03-30T05:10:38.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.034+0000 7fe825200700 1 -- 172.21.15.123:0/2214035907 learned_addr learned my addr 172.21.15.123:0/2214035907 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:10:38.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.035+0000 7fe825c00700 1 --2- 172.21.15.123:0/2214035907 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe820102ce0 0x7fe820115980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:38.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.035+0000 7fe825200700 1 -- 172.21.15.123:0/2214035907 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe820102ce0 msgr2=0x7fe820115980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:38.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.035+0000 7fe825200700 1 --2- 172.21.15.123:0/2214035907 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe820102ce0 0x7fe820115980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:38.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.035+0000 7fe825200700 1 -- 172.21.15.123:0/2214035907 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe80c005040 con 0x7fe820103680 2024-03-30T05:10:38.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.035+0000 7fe825c00700 1 --2- 172.21.15.123:0/2214035907 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe820102ce0 0x7fe820115980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T05:10:38.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.035+0000 7fe825200700 1 --2- 172.21.15.123:0/2214035907 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe820103680 0x7fe820115ec0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fe80c009390 tx=0x7fe80c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:38.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.035+0000 7fe81d600700 1 -- 172.21.15.123:0/2214035907 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe80c00e040 con 0x7fe820103680 2024-03-30T05:10:38.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.035+0000 7fe81d600700 1 -- 172.21.15.123:0/2214035907 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe80c013420 con 0x7fe820103680 2024-03-30T05:10:38.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.035+0000 7fe828600700 1 -- 172.21.15.123:0/2214035907 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe82010fd90 con 0x7fe820103680 2024-03-30T05:10:38.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.035+0000 7fe81d600700 1 -- 172.21.15.123:0/2214035907 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe80c01c630 con 0x7fe820103680 2024-03-30T05:10:38.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.035+0000 7fe828600700 1 -- 172.21.15.123:0/2214035907 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe820110350 con 0x7fe820103680 2024-03-30T05:10:38.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.036+0000 7fe828600700 1 -- 172.21.15.123:0/2214035907 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8200fe430 con 0x7fe820103680 2024-03-30T05:10:38.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.037+0000 7fe81d600700 1 -- 172.21.15.123:0/2214035907 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fe80c013a00 con 0x7fe820103680 2024-03-30T05:10:38.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.039+0000 7fe81d600700 1 --2- 172.21.15.123:0/2214035907 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe80806cc20 0x7fe80806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:38.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.039+0000 7fe81d600700 1 -- 172.21.15.123:0/2214035907 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe80c08d390 con 0x7fe820103680 2024-03-30T05:10:38.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.039+0000 7fe825c00700 1 --2- 172.21.15.123:0/2214035907 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe80806cc20 0x7fe80806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:38.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.039+0000 7fe825c00700 1 --2- 172.21.15.123:0/2214035907 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe80806cc20 0x7fe80806f0d0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fe814009fd0 tx=0x7fe814009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:38.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.043+0000 7fe81d600700 1 -- 172.21.15.123:0/2214035907 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe80c05ade0 con 0x7fe820103680 2024-03-30T05:10:38.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.296+0000 7fe828600700 1 -- 172.21.15.123:0/2214035907 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 37, "format": "json"} v 0) v1 -- 0x7fe820109c80 con 0x7fe820103680 2024-03-30T05:10:38.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.296+0000 7fe81d600700 1 -- 172.21.15.123:0/2214035907 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 37, "format": "json"}]=0 dumped fsmap epoch 37 v42) v1 ==== 107+0+4851 (secure 0 0 0) 0x7fe80c05a970 con 0x7fe820103680 2024-03-30T05:10:38.294 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:10:38.294 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44355,"name":"cephfs.smithi135.zvnlap","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6825/978884711","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":978884711},{"type":"v1","addr":"172.21.15.135:6825","nonce":978884711}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":32},{"gid":44975,"name":"cephfs.smithi135.xwgjvb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6827/3788988014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3788988014},{"type":"v1","addr":"172.21.15.135:6827","nonce":3788988014}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":37,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T04:24:30.211769+0000","modified":"2024-03-30T05:08:25.047190+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34420,"mds_1":34452},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34420":{"gid":34420,"name":"cephfs.smithi123.tqxvzr","rank":0,"incarnation":36,"state":"up:replay","state_seq":1,"addr":"172.21.15.123:6827/1741131598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1741131598},{"type":"v1","addr":"172.21.15.123:6827","nonce":1741131598}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_34452":{"gid":34452,"name":"cephfs.smithi123.dvtind","rank":1,"incarnation":36,"state":"up:replay","state_seq":1,"addr":"172.21.15.123:6829/4108919307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":4108919307},{"type":"v1","addr":"172.21.15.123:6829","nonce":4108919307}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-30T05:10:38.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.299+0000 7fe828600700 1 -- 172.21.15.123:0/2214035907 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe80806cc20 msgr2=0x7fe80806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:38.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.299+0000 7fe828600700 1 --2- 172.21.15.123:0/2214035907 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe80806cc20 0x7fe80806f0d0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fe814009fd0 tx=0x7fe814009ee0 comp rx=0 tx=0).stop 2024-03-30T05:10:38.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.299+0000 7fe828600700 1 -- 172.21.15.123:0/2214035907 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe820103680 msgr2=0x7fe820115ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:38.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.299+0000 7fe828600700 1 --2- 172.21.15.123:0/2214035907 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe820103680 0x7fe820115ec0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fe80c009390 tx=0x7fe80c009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:38.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.299+0000 7fe828600700 1 -- 172.21.15.123:0/2214035907 shutdown_connections 2024-03-30T05:10:38.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.299+0000 7fe828600700 1 --2- 172.21.15.123:0/2214035907 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe80806cc20 0x7fe80806f0d0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:38.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.299+0000 7fe828600700 1 --2- 172.21.15.123:0/2214035907 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe820102ce0 0x7fe820115980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:38.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.299+0000 7fe828600700 1 --2- 172.21.15.123:0/2214035907 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe820103680 0x7fe820115ec0 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:38.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.299+0000 7fe828600700 1 -- 172.21.15.123:0/2214035907 >> 172.21.15.123:0/2214035907 conn(0x7fe8200f8830 msgr2=0x7fe8200f9920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:38.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.299+0000 7fe828600700 1 -- 172.21.15.123:0/2214035907 shutdown_connections 2024-03-30T05:10:38.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.299+0000 7fe828600700 1 -- 172.21.15.123:0/2214035907 wait complete. 2024-03-30T05:10:38.298 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 37 2024-03-30T05:10:38.315 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 38 2024-03-30T05:10:38.465 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:38 smithi123 ceph-mon[139369]: pgmap v1185: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:10:38.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:38 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2013431286' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 35, "format": "json"}]: dispatch 2024-03-30T05:10:38.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:38 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/30326977' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 36, "format": "json"}]: dispatch 2024-03-30T05:10:38.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.507+0000 7faa39800700 1 -- 172.21.15.123:0/434609651 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa34070c70 msgr2=0x7faa34108fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:38.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.507+0000 7faa39800700 1 --2- 172.21.15.123:0/434609651 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa34070c70 0x7faa34108fe0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7faa20004970 tx=0x7faa20009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:38.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.507+0000 7faa39800700 1 -- 172.21.15.123:0/434609651 shutdown_connections 2024-03-30T05:10:38.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.507+0000 7faa39800700 1 --2- 172.21.15.123:0/434609651 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa34070c70 0x7faa34108fe0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:38.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.507+0000 7faa39800700 1 --2- 172.21.15.123:0/434609651 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa34070360 0x7faa34070730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:38.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.507+0000 7faa39800700 1 -- 172.21.15.123:0/434609651 >> 172.21.15.123:0/434609651 conn(0x7faa340f8820 msgr2=0x7faa340fac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:38.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.507+0000 7faa39800700 1 -- 172.21.15.123:0/434609651 shutdown_connections 2024-03-30T05:10:38.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.507+0000 7faa39800700 1 -- 172.21.15.123:0/434609651 wait complete. 2024-03-30T05:10:38.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.508+0000 7faa39800700 1 Processor -- start 2024-03-30T05:10:38.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.508+0000 7faa39800700 1 -- start start 2024-03-30T05:10:38.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.509+0000 7faa39800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa34070360 0x7faa34115920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:38.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.509+0000 7faa39800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa34070c70 0x7faa34115e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:38.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.509+0000 7faa39800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa341164b0 con 0x7faa34070360 2024-03-30T05:10:38.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.509+0000 7faa39800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa3410fab0 con 0x7faa34070c70 2024-03-30T05:10:38.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.509+0000 7faa32a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa34070360 0x7faa34115920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:38.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.509+0000 7faa32a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa34070360 0x7faa34115920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39906/0 (socket says 172.21.15.123:39906) 2024-03-30T05:10:38.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.509+0000 7faa32a00700 1 -- 172.21.15.123:0/1850519523 learned_addr learned my addr 172.21.15.123:0/1850519523 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:10:38.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.509+0000 7faa32000700 1 --2- 172.21.15.123:0/1850519523 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa34070c70 0x7faa34115e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:38.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.510+0000 7faa32a00700 1 -- 172.21.15.123:0/1850519523 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa34070c70 msgr2=0x7faa34115e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:38.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.510+0000 7faa32a00700 1 --2- 172.21.15.123:0/1850519523 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa34070c70 0x7faa34115e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:38.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.510+0000 7faa32a00700 1 -- 172.21.15.123:0/1850519523 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa20005040 con 0x7faa34070360 2024-03-30T05:10:38.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.510+0000 7faa32a00700 1 --2- 172.21.15.123:0/1850519523 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa34070360 0x7faa34115920 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7faa2400eb40 tx=0x7faa2400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:38.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.510+0000 7faa2aa00700 1 -- 172.21.15.123:0/1850519523 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa24007cc0 con 0x7faa34070360 2024-03-30T05:10:38.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.510+0000 7faa2aa00700 1 -- 172.21.15.123:0/1850519523 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faa2400f040 con 0x7faa34070360 2024-03-30T05:10:38.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.510+0000 7faa39800700 1 -- 172.21.15.123:0/1850519523 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa3410fd30 con 0x7faa34070360 2024-03-30T05:10:38.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.510+0000 7faa2aa00700 1 -- 172.21.15.123:0/1850519523 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa2400c990 con 0x7faa34070360 2024-03-30T05:10:38.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.510+0000 7faa39800700 1 -- 172.21.15.123:0/1850519523 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa34110240 con 0x7faa34070360 2024-03-30T05:10:38.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.511+0000 7faa39800700 1 -- 172.21.15.123:0/1850519523 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa34071aa0 con 0x7faa34070360 2024-03-30T05:10:38.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.512+0000 7faa2aa00700 1 -- 172.21.15.123:0/1850519523 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7faa2400caf0 con 0x7faa34070360 2024-03-30T05:10:38.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.513+0000 7faa2aa00700 1 --2- 172.21.15.123:0/1850519523 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7faa1806cc20 0x7faa1806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:38.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.513+0000 7faa2aa00700 1 -- 172.21.15.123:0/1850519523 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7faa2408bc00 con 0x7faa34070360 2024-03-30T05:10:38.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.513+0000 7faa32000700 1 --2- 172.21.15.123:0/1850519523 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7faa1806cc20 0x7faa1806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:38.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.514+0000 7faa32000700 1 --2- 172.21.15.123:0/1850519523 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7faa1806cc20 0x7faa1806f0d0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7faa34110d90 tx=0x7faa20009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:38.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.517+0000 7faa2aa00700 1 -- 172.21.15.123:0/1850519523 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7faa24059700 con 0x7faa34070360 2024-03-30T05:10:38.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:38 smithi135 ceph-mon[106011]: pgmap v1185: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:10:38.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:38 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2013431286' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 35, "format": "json"}]: dispatch 2024-03-30T05:10:38.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:38 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/30326977' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 36, "format": "json"}]: dispatch 2024-03-30T05:10:38.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.770+0000 7faa39800700 1 -- 172.21.15.123:0/1850519523 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 38, "format": "json"} v 0) v1 -- 0x7faa3410be80 con 0x7faa34070360 2024-03-30T05:10:38.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.771+0000 7faa2aa00700 1 -- 172.21.15.123:0/1850519523 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 38, "format": "json"}]=0 dumped fsmap epoch 38 v42) v1 ==== 107+0+4857 (secure 0 0 0) 0x7faa24059290 con 0x7faa34070360 2024-03-30T05:10:38.769 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:10:38.769 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44355,"name":"cephfs.smithi135.zvnlap","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6825/978884711","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":978884711},{"type":"v1","addr":"172.21.15.135:6825","nonce":978884711}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":32},{"gid":44975,"name":"cephfs.smithi135.xwgjvb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6827/3788988014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3788988014},{"type":"v1","addr":"172.21.15.135:6827","nonce":3788988014}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":38,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T04:24:30.211769+0000","modified":"2024-03-30T05:08:31.480430+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34420,"mds_1":34452},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34420":{"gid":34420,"name":"cephfs.smithi123.tqxvzr","rank":0,"incarnation":36,"state":"up:resolve","state_seq":467,"addr":"172.21.15.123:6827/1741131598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1741131598},{"type":"v1","addr":"172.21.15.123:6827","nonce":1741131598}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_34452":{"gid":34452,"name":"cephfs.smithi123.dvtind","rank":1,"incarnation":36,"state":"up:resolve","state_seq":462,"addr":"172.21.15.123:6829/4108919307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":4108919307},{"type":"v1","addr":"172.21.15.123:6829","nonce":4108919307}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-30T05:10:38.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.773+0000 7faa39800700 1 -- 172.21.15.123:0/1850519523 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7faa1806cc20 msgr2=0x7faa1806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:38.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.773+0000 7faa39800700 1 --2- 172.21.15.123:0/1850519523 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7faa1806cc20 0x7faa1806f0d0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7faa34110d90 tx=0x7faa20009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:38.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.773+0000 7faa39800700 1 -- 172.21.15.123:0/1850519523 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa34070360 msgr2=0x7faa34115920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:38.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.773+0000 7faa39800700 1 --2- 172.21.15.123:0/1850519523 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa34070360 0x7faa34115920 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7faa2400eb40 tx=0x7faa2400b040 comp rx=0 tx=0).stop 2024-03-30T05:10:38.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.774+0000 7faa39800700 1 -- 172.21.15.123:0/1850519523 shutdown_connections 2024-03-30T05:10:38.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.774+0000 7faa39800700 1 --2- 172.21.15.123:0/1850519523 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7faa1806cc20 0x7faa1806f0d0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:38.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.774+0000 7faa39800700 1 --2- 172.21.15.123:0/1850519523 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa34070360 0x7faa34115920 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:38.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.774+0000 7faa39800700 1 --2- 172.21.15.123:0/1850519523 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa34070c70 0x7faa34115e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:38.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.774+0000 7faa39800700 1 -- 172.21.15.123:0/1850519523 >> 172.21.15.123:0/1850519523 conn(0x7faa340f8820 msgr2=0x7faa340f9d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:38.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.774+0000 7faa39800700 1 -- 172.21.15.123:0/1850519523 shutdown_connections 2024-03-30T05:10:38.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.774+0000 7faa39800700 1 -- 172.21.15.123:0/1850519523 wait complete. 2024-03-30T05:10:38.772 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 38 2024-03-30T05:10:38.790 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 39 2024-03-30T05:10:38.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.980+0000 7f785b000700 1 -- 172.21.15.123:0/1069422239 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7854102ce0 msgr2=0x7f78541030b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:38.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.980+0000 7f785b000700 1 --2- 172.21.15.123:0/1069422239 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7854102ce0 0x7f78541030b0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f7844004940 tx=0x7f7844009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:38.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.980+0000 7f785b000700 1 -- 172.21.15.123:0/1069422239 shutdown_connections 2024-03-30T05:10:38.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.980+0000 7f785b000700 1 --2- 172.21.15.123:0/1069422239 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7854103680 0x7f78540fccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:38.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.980+0000 7f785b000700 1 --2- 172.21.15.123:0/1069422239 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7854102ce0 0x7f78541030b0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:38.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.980+0000 7f785b000700 1 -- 172.21.15.123:0/1069422239 >> 172.21.15.123:0/1069422239 conn(0x7f78540f8830 msgr2=0x7f78540fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:38.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.981+0000 7f785b000700 1 -- 172.21.15.123:0/1069422239 shutdown_connections 2024-03-30T05:10:38.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.981+0000 7f785b000700 1 -- 172.21.15.123:0/1069422239 wait complete. 2024-03-30T05:10:38.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.981+0000 7f785b000700 1 Processor -- start 2024-03-30T05:10:38.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.981+0000 7f785b000700 1 -- start start 2024-03-30T05:10:38.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.982+0000 7f785b000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7854102ce0 0x7f78541159a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:38.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.982+0000 7f785b000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7854103680 0x7f7854115ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:38.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.982+0000 7f785b000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7854116530 con 0x7f7854102ce0 2024-03-30T05:10:38.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.982+0000 7f785b000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f785410fb50 con 0x7f7854103680 2024-03-30T05:10:38.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.982+0000 7f7853e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7854102ce0 0x7f78541159a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:38.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.982+0000 7f7853e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7854102ce0 0x7f78541159a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39922/0 (socket says 172.21.15.123:39922) 2024-03-30T05:10:38.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.982+0000 7f7853e00700 1 -- 172.21.15.123:0/2042081265 learned_addr learned my addr 172.21.15.123:0/2042081265 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:10:38.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.982+0000 7f7853400700 1 --2- 172.21.15.123:0/2042081265 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7854103680 0x7f7854115ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:38.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.982+0000 7f7853e00700 1 -- 172.21.15.123:0/2042081265 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7854103680 msgr2=0x7f7854115ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:38.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.982+0000 7f7853e00700 1 --2- 172.21.15.123:0/2042081265 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7854103680 0x7f7854115ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:38.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.982+0000 7f7853e00700 1 -- 172.21.15.123:0/2042081265 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7844005040 con 0x7f7854102ce0 2024-03-30T05:10:38.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.983+0000 7f7853e00700 1 --2- 172.21.15.123:0/2042081265 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7854102ce0 0x7f78541159a0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f7844009390 tx=0x7f7844009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:38.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.983+0000 7f784be00700 1 -- 172.21.15.123:0/2042081265 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f784400e040 con 0x7f7854102ce0 2024-03-30T05:10:38.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.983+0000 7f784be00700 1 -- 172.21.15.123:0/2042081265 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7844013420 con 0x7f7854102ce0 2024-03-30T05:10:38.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.983+0000 7f785b000700 1 -- 172.21.15.123:0/2042081265 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f785410fdd0 con 0x7f7854102ce0 2024-03-30T05:10:38.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.983+0000 7f784be00700 1 -- 172.21.15.123:0/2042081265 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f784401c630 con 0x7f7854102ce0 2024-03-30T05:10:38.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.983+0000 7f785b000700 1 -- 172.21.15.123:0/2042081265 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7854110390 con 0x7f7854102ce0 2024-03-30T05:10:38.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.984+0000 7f785b000700 1 -- 172.21.15.123:0/2042081265 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78540fe430 con 0x7f7854102ce0 2024-03-30T05:10:38.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.985+0000 7f784be00700 1 -- 172.21.15.123:0/2042081265 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f78440139e0 con 0x7f7854102ce0 2024-03-30T05:10:38.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.986+0000 7f784be00700 1 --2- 172.21.15.123:0/2042081265 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f783806cc20 0x7f783806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:38.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.986+0000 7f784be00700 1 -- 172.21.15.123:0/2042081265 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f784408d3d0 con 0x7f7854102ce0 2024-03-30T05:10:38.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.986+0000 7f7853400700 1 --2- 172.21.15.123:0/2042081265 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f783806cc20 0x7f783806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:38.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.987+0000 7f7853400700 1 --2- 172.21.15.123:0/2042081265 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f783806cc20 0x7f783806f0d0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f7854111240 tx=0x7f784000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:38.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:38.990+0000 7f784be00700 1 -- 172.21.15.123:0/2042081265 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7844057390 con 0x7f7854102ce0 2024-03-30T05:10:39.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.244+0000 7f785b000700 1 -- 172.21.15.123:0/2042081265 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 39, "format": "json"} v 0) v1 -- 0x7f7854111020 con 0x7f7854102ce0 2024-03-30T05:10:39.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.245+0000 7f784be00700 1 -- 172.21.15.123:0/2042081265 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 39, "format": "json"}]=0 dumped fsmap epoch 39 v42) v1 ==== 107+0+4861 (secure 0 0 0) 0x7f784405a9b0 con 0x7f7854102ce0 2024-03-30T05:10:39.243 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:10:39.243 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44355,"name":"cephfs.smithi135.zvnlap","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6825/978884711","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":978884711},{"type":"v1","addr":"172.21.15.135:6825","nonce":978884711}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":32},{"gid":44975,"name":"cephfs.smithi135.xwgjvb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6827/3788988014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3788988014},{"type":"v1","addr":"172.21.15.135:6827","nonce":3788988014}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":39,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T04:24:30.211769+0000","modified":"2024-03-30T05:08:32.485244+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34420,"mds_1":34452},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34420":{"gid":34420,"name":"cephfs.smithi123.tqxvzr","rank":0,"incarnation":36,"state":"up:reconnect","state_seq":468,"addr":"172.21.15.123:6827/1741131598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1741131598},{"type":"v1","addr":"172.21.15.123:6827","nonce":1741131598}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_34452":{"gid":34452,"name":"cephfs.smithi123.dvtind","rank":1,"incarnation":36,"state":"up:reconnect","state_seq":463,"addr":"172.21.15.123:6829/4108919307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":4108919307},{"type":"v1","addr":"172.21.15.123:6829","nonce":4108919307}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-30T05:10:39.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.247+0000 7f785b000700 1 -- 172.21.15.123:0/2042081265 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f783806cc20 msgr2=0x7f783806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:39.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.247+0000 7f785b000700 1 --2- 172.21.15.123:0/2042081265 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f783806cc20 0x7f783806f0d0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f7854111240 tx=0x7f784000b040 comp rx=0 tx=0).stop 2024-03-30T05:10:39.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.247+0000 7f785b000700 1 -- 172.21.15.123:0/2042081265 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7854102ce0 msgr2=0x7f78541159a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:39.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.247+0000 7f785b000700 1 --2- 172.21.15.123:0/2042081265 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7854102ce0 0x7f78541159a0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f7844009390 tx=0x7f7844009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:39.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.248+0000 7f785b000700 1 -- 172.21.15.123:0/2042081265 shutdown_connections 2024-03-30T05:10:39.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.248+0000 7f785b000700 1 --2- 172.21.15.123:0/2042081265 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f783806cc20 0x7f783806f0d0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:39.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.248+0000 7f785b000700 1 --2- 172.21.15.123:0/2042081265 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7854102ce0 0x7f78541159a0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:39.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.248+0000 7f785b000700 1 --2- 172.21.15.123:0/2042081265 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7854103680 0x7f7854115ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:39.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.248+0000 7f785b000700 1 -- 172.21.15.123:0/2042081265 >> 172.21.15.123:0/2042081265 conn(0x7f78540f8830 msgr2=0x7f78540f9820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:39.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.248+0000 7f785b000700 1 -- 172.21.15.123:0/2042081265 shutdown_connections 2024-03-30T05:10:39.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.248+0000 7f785b000700 1 -- 172.21.15.123:0/2042081265 wait complete. 2024-03-30T05:10:39.247 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 39 2024-03-30T05:10:39.263 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 40 2024-03-30T05:10:39.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.454+0000 7fda87c00700 1 -- 172.21.15.123:0/1955870160 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda80103670 msgr2=0x7fda800fcce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:39.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.454+0000 7fda87c00700 1 --2- 172.21.15.123:0/1955870160 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda80103670 0x7fda800fcce0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fda6c004940 tx=0x7fda6c009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:39.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.454+0000 7fda87c00700 1 -- 172.21.15.123:0/1955870160 shutdown_connections 2024-03-30T05:10:39.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.454+0000 7fda87c00700 1 --2- 172.21.15.123:0/1955870160 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda80103670 0x7fda800fcce0 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:39.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.454+0000 7fda87c00700 1 --2- 172.21.15.123:0/1955870160 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fda80102cd0 0x7fda801030a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:39.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.454+0000 7fda87c00700 1 -- 172.21.15.123:0/1955870160 >> 172.21.15.123:0/1955870160 conn(0x7fda800f8820 msgr2=0x7fda800fac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:39.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.454+0000 7fda87c00700 1 -- 172.21.15.123:0/1955870160 shutdown_connections 2024-03-30T05:10:39.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.454+0000 7fda87c00700 1 -- 172.21.15.123:0/1955870160 wait complete. 2024-03-30T05:10:39.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.455+0000 7fda87c00700 1 Processor -- start 2024-03-30T05:10:39.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.455+0000 7fda87c00700 1 -- start start 2024-03-30T05:10:39.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.455+0000 7fda87c00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fda80102cd0 0x7fda800702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:39.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.455+0000 7fda87c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda80103670 0x7fda80070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:39.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.455+0000 7fda87c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda80072210 con 0x7fda80103670 2024-03-30T05:10:39.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.455+0000 7fda87c00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda80070d40 con 0x7fda80102cd0 2024-03-30T05:10:39.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.456+0000 7fda7fe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda80103670 0x7fda80070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:39.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.456+0000 7fda7fe00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda80103670 0x7fda80070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39940/0 (socket says 172.21.15.123:39940) 2024-03-30T05:10:39.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.456+0000 7fda7fe00700 1 -- 172.21.15.123:0/3951319831 learned_addr learned my addr 172.21.15.123:0/3951319831 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:10:39.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.456+0000 7fda85200700 1 --2- 172.21.15.123:0/3951319831 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fda80102cd0 0x7fda800702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:39.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.456+0000 7fda7fe00700 1 -- 172.21.15.123:0/3951319831 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fda80102cd0 msgr2=0x7fda800702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:39.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.456+0000 7fda7fe00700 1 --2- 172.21.15.123:0/3951319831 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fda80102cd0 0x7fda800702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:39.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.456+0000 7fda7fe00700 1 -- 172.21.15.123:0/3951319831 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda6c005040 con 0x7fda80103670 2024-03-30T05:10:39.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.456+0000 7fda7fe00700 1 --2- 172.21.15.123:0/3951319831 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda80103670 0x7fda80070800 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fda6c009390 tx=0x7fda6c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:39.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.457+0000 7fda7cc00700 1 -- 172.21.15.123:0/3951319831 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda6c00e040 con 0x7fda80103670 2024-03-30T05:10:39.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.457+0000 7fda7cc00700 1 -- 172.21.15.123:0/3951319831 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fda6c013420 con 0x7fda80103670 2024-03-30T05:10:39.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.457+0000 7fda87c00700 1 -- 172.21.15.123:0/3951319831 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda80070fc0 con 0x7fda80103670 2024-03-30T05:10:39.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.457+0000 7fda7cc00700 1 -- 172.21.15.123:0/3951319831 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda6c01c630 con 0x7fda80103670 2024-03-30T05:10:39.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.457+0000 7fda87c00700 1 -- 172.21.15.123:0/3951319831 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda80119e70 con 0x7fda80103670 2024-03-30T05:10:39.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.458+0000 7fda87c00700 1 -- 172.21.15.123:0/3951319831 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda800fe420 con 0x7fda80103670 2024-03-30T05:10:39.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.459+0000 7fda7cc00700 1 -- 172.21.15.123:0/3951319831 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fda6c013a00 con 0x7fda80103670 2024-03-30T05:10:39.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.460+0000 7fda7cc00700 1 --2- 172.21.15.123:0/3951319831 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fda6806cc20 0x7fda6806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:39.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.460+0000 7fda7cc00700 1 -- 172.21.15.123:0/3951319831 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fda6c08d0e0 con 0x7fda80103670 2024-03-30T05:10:39.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.460+0000 7fda85200700 1 --2- 172.21.15.123:0/3951319831 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fda6806cc20 0x7fda6806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:39.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.461+0000 7fda85200700 1 --2- 172.21.15.123:0/3951319831 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fda6806cc20 0x7fda6806f0d0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fda74009fd0 tx=0x7fda7400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:39.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.464+0000 7fda7cc00700 1 -- 172.21.15.123:0/3951319831 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fda6c057160 con 0x7fda80103670 2024-03-30T05:10:39.465 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:39 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2214035907' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 37, "format": "json"}]: dispatch 2024-03-30T05:10:39.465 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:39 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/1850519523' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 38, "format": "json"}]: dispatch 2024-03-30T05:10:39.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:39 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2214035907' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 37, "format": "json"}]: dispatch 2024-03-30T05:10:39.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:39 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/1850519523' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 38, "format": "json"}]: dispatch 2024-03-30T05:10:39.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.711+0000 7fda87c00700 1 -- 172.21.15.123:0/3951319831 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 40, "format": "json"} v 0) v1 -- 0x7fda80064970 con 0x7fda80103670 2024-03-30T05:10:39.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.712+0000 7fda7cc00700 1 -- 172.21.15.123:0/3951319831 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 40, "format": "json"}]=0 dumped fsmap epoch 40 v42) v1 ==== 107+0+4855 (secure 0 0 0) 0x7fda6c05a780 con 0x7fda80103670 2024-03-30T05:10:39.710 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:10:39.710 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44355,"name":"cephfs.smithi135.zvnlap","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6825/978884711","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":978884711},{"type":"v1","addr":"172.21.15.135:6825","nonce":978884711}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":32},{"gid":44975,"name":"cephfs.smithi135.xwgjvb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6827/3788988014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3788988014},{"type":"v1","addr":"172.21.15.135:6827","nonce":3788988014}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":40,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T04:24:30.211769+0000","modified":"2024-03-30T05:08:33.494863+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34420,"mds_1":34452},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34420":{"gid":34420,"name":"cephfs.smithi123.tqxvzr","rank":0,"incarnation":36,"state":"up:rejoin","state_seq":469,"addr":"172.21.15.123:6827/1741131598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1741131598},{"type":"v1","addr":"172.21.15.123:6827","nonce":1741131598}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_34452":{"gid":34452,"name":"cephfs.smithi123.dvtind","rank":1,"incarnation":36,"state":"up:rejoin","state_seq":464,"addr":"172.21.15.123:6829/4108919307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":4108919307},{"type":"v1","addr":"172.21.15.123:6829","nonce":4108919307}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-30T05:10:39.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.714+0000 7fda87c00700 1 -- 172.21.15.123:0/3951319831 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fda6806cc20 msgr2=0x7fda6806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:39.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.714+0000 7fda87c00700 1 --2- 172.21.15.123:0/3951319831 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fda6806cc20 0x7fda6806f0d0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fda74009fd0 tx=0x7fda7400b040 comp rx=0 tx=0).stop 2024-03-30T05:10:39.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.714+0000 7fda87c00700 1 -- 172.21.15.123:0/3951319831 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda80103670 msgr2=0x7fda80070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:39.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.714+0000 7fda87c00700 1 --2- 172.21.15.123:0/3951319831 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda80103670 0x7fda80070800 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fda6c009390 tx=0x7fda6c009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:39.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.714+0000 7fda87c00700 1 -- 172.21.15.123:0/3951319831 shutdown_connections 2024-03-30T05:10:39.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.714+0000 7fda87c00700 1 --2- 172.21.15.123:0/3951319831 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fda6806cc20 0x7fda6806f0d0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:39.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.714+0000 7fda87c00700 1 --2- 172.21.15.123:0/3951319831 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fda80102cd0 0x7fda800702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:39.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.714+0000 7fda87c00700 1 --2- 172.21.15.123:0/3951319831 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda80103670 0x7fda80070800 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:39.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.715+0000 7fda87c00700 1 -- 172.21.15.123:0/3951319831 >> 172.21.15.123:0/3951319831 conn(0x7fda800f8820 msgr2=0x7fda800f9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:39.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.715+0000 7fda87c00700 1 -- 172.21.15.123:0/3951319831 shutdown_connections 2024-03-30T05:10:39.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.715+0000 7fda87c00700 1 -- 172.21.15.123:0/3951319831 wait complete. 2024-03-30T05:10:39.714 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 40 2024-03-30T05:10:39.731 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 41 2024-03-30T05:10:39.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.923+0000 7f9d2c400700 1 -- 172.21.15.123:0/2716213972 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d24103670 msgr2=0x7f9d240fcce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:39.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.923+0000 7f9d2c400700 1 --2- 172.21.15.123:0/2716213972 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d24103670 0x7f9d240fcce0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f9d10004940 tx=0x7f9d10009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:39.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.923+0000 7f9d2c400700 1 -- 172.21.15.123:0/2716213972 shutdown_connections 2024-03-30T05:10:39.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.923+0000 7f9d2c400700 1 --2- 172.21.15.123:0/2716213972 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d24103670 0x7f9d240fcce0 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:39.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.923+0000 7f9d2c400700 1 --2- 172.21.15.123:0/2716213972 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9d24102cd0 0x7f9d241030a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:39.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.923+0000 7f9d2c400700 1 -- 172.21.15.123:0/2716213972 >> 172.21.15.123:0/2716213972 conn(0x7f9d240f8820 msgr2=0x7f9d240fac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:39.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.923+0000 7f9d2c400700 1 -- 172.21.15.123:0/2716213972 shutdown_connections 2024-03-30T05:10:39.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.923+0000 7f9d2c400700 1 -- 172.21.15.123:0/2716213972 wait complete. 2024-03-30T05:10:39.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.924+0000 7f9d2c400700 1 Processor -- start 2024-03-30T05:10:39.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.924+0000 7f9d2c400700 1 -- start start 2024-03-30T05:10:39.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.924+0000 7f9d2c400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9d24102cd0 0x7f9d24115900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:39.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.925+0000 7f9d2c400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d24103670 0x7f9d24115e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:39.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.925+0000 7f9d2c400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d24116490 con 0x7f9d24103670 2024-03-30T05:10:39.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.925+0000 7f9d2c400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d2410fa90 con 0x7f9d24102cd0 2024-03-30T05:10:39.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.925+0000 7f9d29000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d24103670 0x7f9d24115e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:39.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.925+0000 7f9d29000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d24103670 0x7f9d24115e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39958/0 (socket says 172.21.15.123:39958) 2024-03-30T05:10:39.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.925+0000 7f9d29000700 1 -- 172.21.15.123:0/4192962512 learned_addr learned my addr 172.21.15.123:0/4192962512 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:10:39.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.925+0000 7f9d29a00700 1 --2- 172.21.15.123:0/4192962512 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9d24102cd0 0x7f9d24115900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:39.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.925+0000 7f9d29000700 1 -- 172.21.15.123:0/4192962512 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9d24102cd0 msgr2=0x7f9d24115900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:39.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.925+0000 7f9d29000700 1 --2- 172.21.15.123:0/4192962512 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9d24102cd0 0x7f9d24115900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:39.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.925+0000 7f9d29000700 1 -- 172.21.15.123:0/4192962512 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d10005040 con 0x7f9d24103670 2024-03-30T05:10:39.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.925+0000 7f9d29000700 1 --2- 172.21.15.123:0/4192962512 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d24103670 0x7f9d24115e40 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f9d10008d50 tx=0x7f9d10009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:39.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.926+0000 7f9d21600700 1 -- 172.21.15.123:0/4192962512 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d1000e030 con 0x7f9d24103670 2024-03-30T05:10:39.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.926+0000 7f9d21600700 1 -- 172.21.15.123:0/4192962512 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9d100093b0 con 0x7f9d24103670 2024-03-30T05:10:39.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.926+0000 7f9d2c400700 1 -- 172.21.15.123:0/4192962512 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d2410fd10 con 0x7f9d24103670 2024-03-30T05:10:39.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.926+0000 7f9d21600700 1 -- 172.21.15.123:0/4192962512 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d1001cbb0 con 0x7f9d24103670 2024-03-30T05:10:39.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.926+0000 7f9d2c400700 1 -- 172.21.15.123:0/4192962512 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d241102d0 con 0x7f9d24103670 2024-03-30T05:10:39.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.927+0000 7f9d2c400700 1 -- 172.21.15.123:0/4192962512 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d240fe420 con 0x7f9d24103670 2024-03-30T05:10:39.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.928+0000 7f9d21600700 1 -- 172.21.15.123:0/4192962512 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f9d10013930 con 0x7f9d24103670 2024-03-30T05:10:39.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.928+0000 7f9d21600700 1 --2- 172.21.15.123:0/4192962512 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9d0c06cc20 0x7f9d0c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:39.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.929+0000 7f9d21600700 1 -- 172.21.15.123:0/4192962512 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f9d1008d860 con 0x7f9d24103670 2024-03-30T05:10:39.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.929+0000 7f9d29a00700 1 --2- 172.21.15.123:0/4192962512 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9d0c06cc20 0x7f9d0c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:39.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.929+0000 7f9d29a00700 1 --2- 172.21.15.123:0/4192962512 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9d0c06cc20 0x7f9d0c06f0d0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f9d18009b80 tx=0x7f9d1800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:39.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:39.933+0000 7f9d21600700 1 -- 172.21.15.123:0/4192962512 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9d1005b2b0 con 0x7f9d24103670 2024-03-30T05:10:40.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.186+0000 7f9d2c400700 1 -- 172.21.15.123:0/4192962512 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 41, "format": "json"} v 0) v1 -- 0x7f9d24109c70 con 0x7f9d24103670 2024-03-30T05:10:40.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.186+0000 7f9d21600700 1 -- 172.21.15.123:0/4192962512 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 41, "format": "json"}]=0 dumped fsmap epoch 41 v42) v1 ==== 107+0+4855 (secure 0 0 0) 0x7f9d1005ae40 con 0x7f9d24103670 2024-03-30T05:10:40.184 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:10:40.184 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44355,"name":"cephfs.smithi135.zvnlap","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6825/978884711","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":978884711},{"type":"v1","addr":"172.21.15.135:6825","nonce":978884711}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":32},{"gid":44975,"name":"cephfs.smithi135.xwgjvb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6827/3788988014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3788988014},{"type":"v1","addr":"172.21.15.135:6827","nonce":3788988014}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":41,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T04:24:30.211769+0000","modified":"2024-03-30T05:08:34.502469+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34420,"mds_1":34452},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34420":{"gid":34420,"name":"cephfs.smithi123.tqxvzr","rank":0,"incarnation":36,"state":"up:active","state_seq":470,"addr":"172.21.15.123:6827/1741131598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1741131598},{"type":"v1","addr":"172.21.15.123:6827","nonce":1741131598}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_34452":{"gid":34452,"name":"cephfs.smithi123.dvtind","rank":1,"incarnation":36,"state":"up:active","state_seq":465,"addr":"172.21.15.123:6829/4108919307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":4108919307},{"type":"v1","addr":"172.21.15.123:6829","nonce":4108919307}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-30T05:10:40.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.189+0000 7f9d2c400700 1 -- 172.21.15.123:0/4192962512 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9d0c06cc20 msgr2=0x7f9d0c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:40.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.189+0000 7f9d2c400700 1 --2- 172.21.15.123:0/4192962512 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9d0c06cc20 0x7f9d0c06f0d0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f9d18009b80 tx=0x7f9d1800b040 comp rx=0 tx=0).stop 2024-03-30T05:10:40.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.189+0000 7f9d2c400700 1 -- 172.21.15.123:0/4192962512 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d24103670 msgr2=0x7f9d24115e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:40.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.189+0000 7f9d2c400700 1 --2- 172.21.15.123:0/4192962512 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d24103670 0x7f9d24115e40 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f9d10008d50 tx=0x7f9d10009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:40.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.189+0000 7f9d2c400700 1 -- 172.21.15.123:0/4192962512 shutdown_connections 2024-03-30T05:10:40.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.189+0000 7f9d2c400700 1 --2- 172.21.15.123:0/4192962512 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f9d0c06cc20 0x7f9d0c06f0d0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:40.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.189+0000 7f9d2c400700 1 --2- 172.21.15.123:0/4192962512 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9d24102cd0 0x7f9d24115900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:40.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.189+0000 7f9d2c400700 1 --2- 172.21.15.123:0/4192962512 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d24103670 0x7f9d24115e40 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:40.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.189+0000 7f9d2c400700 1 -- 172.21.15.123:0/4192962512 >> 172.21.15.123:0/4192962512 conn(0x7f9d240f8820 msgr2=0x7f9d24071ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:40.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.189+0000 7f9d2c400700 1 -- 172.21.15.123:0/4192962512 shutdown_connections 2024-03-30T05:10:40.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.189+0000 7f9d2c400700 1 -- 172.21.15.123:0/4192962512 wait complete. 2024-03-30T05:10:40.188 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 41 2024-03-30T05:10:40.205 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-03-30T05:10:40.217 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-03-30T05:10:40.218 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-30T05:10:40.218 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-30T05:10:40.264 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-30T05:10:40.264 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-30T05:10:40.351 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T05:10:40.450 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:40 smithi123 ceph-mon[139369]: pgmap v1186: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:10:40.450 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:40 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2042081265' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 39, "format": "json"}]: dispatch 2024-03-30T05:10:40.450 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:40 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3951319831' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 40, "format": "json"}]: dispatch 2024-03-30T05:10:40.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.581+0000 7f548ea00700 1 -- 172.21.15.123:0/1369951500 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5488070360 msgr2=0x7f5488070730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:40.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.581+0000 7f548ea00700 1 --2- 172.21.15.123:0/1369951500 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5488070360 0x7f5488070730 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f5478004970 tx=0x7f5478009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:40.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.581+0000 7f548ea00700 1 -- 172.21.15.123:0/1369951500 shutdown_connections 2024-03-30T05:10:40.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.581+0000 7f548ea00700 1 --2- 172.21.15.123:0/1369951500 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5488070c70 0x7f5488109010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:40.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.581+0000 7f548ea00700 1 --2- 172.21.15.123:0/1369951500 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5488070360 0x7f5488070730 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:40.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.581+0000 7f548ea00700 1 -- 172.21.15.123:0/1369951500 >> 172.21.15.123:0/1369951500 conn(0x7f54880f8850 msgr2=0x7f54880fac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:40.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.581+0000 7f548ea00700 1 -- 172.21.15.123:0/1369951500 shutdown_connections 2024-03-30T05:10:40.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.581+0000 7f548ea00700 1 -- 172.21.15.123:0/1369951500 wait complete. 2024-03-30T05:10:40.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.582+0000 7f548ea00700 1 Processor -- start 2024-03-30T05:10:40.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.582+0000 7f548ea00700 1 -- start start 2024-03-30T05:10:40.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.582+0000 7f548ea00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5488070360 0x7f5488115a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:40.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.582+0000 7f548ea00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5488070c70 0x7f5488115f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:40.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.582+0000 7f548ea00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54881165b0 con 0x7f5488070c70 2024-03-30T05:10:40.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.582+0000 7f548ea00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548810fb90 con 0x7f5488070360 2024-03-30T05:10:40.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.582+0000 7f5487400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5488070c70 0x7f5488115f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:40.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.582+0000 7f5487400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5488070c70 0x7f5488115f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39980/0 (socket says 172.21.15.123:39980) 2024-03-30T05:10:40.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.582+0000 7f5487400700 1 -- 172.21.15.123:0/2753617568 learned_addr learned my addr 172.21.15.123:0/2753617568 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:10:40.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.583+0000 7f5487400700 1 -- 172.21.15.123:0/2753617568 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5488070360 msgr2=0x7f5488115a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:40.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.583+0000 7f5487400700 1 --2- 172.21.15.123:0/2753617568 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5488070360 0x7f5488115a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:40.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.583+0000 7f5487400700 1 -- 172.21.15.123:0/2753617568 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5478005040 con 0x7f5488070c70 2024-03-30T05:10:40.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.583+0000 7f5487400700 1 --2- 172.21.15.123:0/2753617568 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5488070c70 0x7f5488115f60 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f547400c8c0 tx=0x7f54740044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:40.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.583+0000 7f547fe00700 1 -- 172.21.15.123:0/2753617568 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f547400a980 con 0x7f5488070c70 2024-03-30T05:10:40.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.583+0000 7f547fe00700 1 -- 172.21.15.123:0/2753617568 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f547400ce60 con 0x7f5488070c70 2024-03-30T05:10:40.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.583+0000 7f548ea00700 1 -- 172.21.15.123:0/2753617568 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f548810fe70 con 0x7f5488070c70 2024-03-30T05:10:40.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.583+0000 7f547fe00700 1 -- 172.21.15.123:0/2753617568 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f54740113a0 con 0x7f5488070c70 2024-03-30T05:10:40.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.583+0000 7f548ea00700 1 -- 172.21.15.123:0/2753617568 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54881103b0 con 0x7f5488070c70 2024-03-30T05:10:40.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.584+0000 7f548ea00700 1 -- 172.21.15.123:0/2753617568 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5488071aa0 con 0x7f5488070c70 2024-03-30T05:10:40.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.586+0000 7f547fe00700 1 -- 172.21.15.123:0/2753617568 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f5474012400 con 0x7f5488070c70 2024-03-30T05:10:40.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.587+0000 7f547fe00700 1 --2- 172.21.15.123:0/2753617568 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f546c06cc20 0x7f546c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:40.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.587+0000 7f547fe00700 1 -- 172.21.15.123:0/2753617568 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f547408c1d0 con 0x7f5488070c70 2024-03-30T05:10:40.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.587+0000 7f5487e00700 1 --2- 172.21.15.123:0/2753617568 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f546c06cc20 0x7f546c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:40.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.588+0000 7f5487e00700 1 --2- 172.21.15.123:0/2753617568 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f546c06cc20 0x7f546c06f0d0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f5478000c00 tx=0x7f5478009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:40.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.591+0000 7f547fe00700 1 -- 172.21.15.123:0/2753617568 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5474059c20 con 0x7f5488070c70 2024-03-30T05:10:40.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:40 smithi135 ceph-mon[106011]: pgmap v1186: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:10:40.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:40 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2042081265' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 39, "format": "json"}]: dispatch 2024-03-30T05:10:40.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:40 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3951319831' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 40, "format": "json"}]: dispatch 2024-03-30T05:10:40.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.797+0000 7f548ea00700 1 -- 172.21.15.123:0/2753617568 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5488064970 con 0x7f5488070c70 2024-03-30T05:10:40.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.798+0000 7f547fe00700 1 -- 172.21.15.123:0/2753617568 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+16625 (secure 0 0 0) 0x7f54740597b0 con 0x7f5488070c70 2024-03-30T05:10:40.797 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:10:40.797 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":86,"fsid":"2e77a2be-ee4c-11ee-b647-cb9ed24678a4","created":"2024-03-30T04:16:30.949442+0000","modified":"2024-03-30T04:53:15.872216+0000","last_up_change":"2024-03-30T04:37:14.509879+0000","last_in_change":"2024-03-30T04:22:44.504478+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T04:21:34.842970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-03-30T04:24:29.050323+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-03-30T04:24:29.704111+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3968446b-8696-4eec-b640-c97973ca054d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":49,"up_from":53,"up_thru":78,"down_at":50,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":1500077196},{"type":"v1","addr":"172.21.15.123:6803","nonce":1500077196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":1500077196},{"type":"v1","addr":"172.21.15.123:6805","nonce":1500077196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1500077196},{"type":"v1","addr":"172.21.15.123:6809","nonce":1500077196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":1500077196},{"type":"v1","addr":"172.21.15.123:6807","nonce":1500077196}]},"public_addr":"172.21.15.123:6803/1500077196","cluster_addr":"172.21.15.123:6805/1500077196","heartbeat_back_addr":"172.21.15.123:6809/1500077196","heartbeat_front_addr":"172.21.15.123:6807/1500077196","state":["exists","up"]},{"osd":1,"uuid":"336be34b-bb3c-46ab-8439-af04a1366db5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":54,"up_from":58,"up_thru":78,"down_at":55,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":4217392586},{"type":"v1","addr":"172.21.15.123:6811","nonce":4217392586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":4217392586},{"type":"v1","addr":"172.21.15.123:6813","nonce":4217392586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4217392586},{"type":"v1","addr":"172.21.15.123:6817","nonce":4217392586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":4217392586},{"type":"v1","addr":"172.21.15.123:6815","nonce":4217392586}]},"public_addr":"172.21.15.123:6811/4217392586","cluster_addr":"172.21.15.123:6813/4217392586","heartbeat_back_addr":"172.21.15.123:6817/4217392586","heartbeat_front_addr":"172.21.15.123:6815/4217392586","state":["exists","up"]},{"osd":2,"uuid":"4386fc8a-e9ed-413e-bd3a-bb84a0e62827","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":59,"up_from":63,"up_thru":78,"down_at":60,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":3356591260},{"type":"v1","addr":"172.21.15.123:6819","nonce":3356591260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":3356591260},{"type":"v1","addr":"172.21.15.123:6821","nonce":3356591260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":3356591260},{"type":"v1","addr":"172.21.15.123:6825","nonce":3356591260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":3356591260},{"type":"v1","addr":"172.21.15.123:6823","nonce":3356591260}]},"public_addr":"172.21.15.123:6819/3356591260","cluster_addr":"172.21.15.123:6821/3356591260","heartbeat_back_addr":"172.21.15.123:6825/3356591260","heartbeat_front_addr":"172.21.15.123:6823/3356591260","state":["exists","up"]},{"osd":3,"uuid":"086d86d4-ff1a-47dd-8416-447ccf9064ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":64,"up_from":68,"up_thru":78,"down_at":65,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":978771295},{"type":"v1","addr":"172.21.15.135:6801","nonce":978771295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":978771295},{"type":"v1","addr":"172.21.15.135:6803","nonce":978771295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":978771295},{"type":"v1","addr":"172.21.15.135:6807","nonce":978771295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":978771295},{"type":"v1","addr":"172.21.15.135:6805","nonce":978771295}]},"public_addr":"172.21.15.135:6801/978771295","cluster_addr":"172.21.15.135:6803/978771295","heartbeat_back_addr":"172.21.15.135:6807/978771295","heartbeat_front_addr":"172.21.15.135:6805/978771295","state":["exists","up"]},{"osd":4,"uuid":"31868cec-dd4d-44dd-a150-7ebdbd61ea4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":34,"last_clean_end":69,"up_from":73,"up_thru":78,"down_at":70,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":4133828133},{"type":"v1","addr":"172.21.15.135:6809","nonce":4133828133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":4133828133},{"type":"v1","addr":"172.21.15.135:6811","nonce":4133828133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":4133828133},{"type":"v1","addr":"172.21.15.135:6815","nonce":4133828133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":4133828133},{"type":"v1","addr":"172.21.15.135:6813","nonce":4133828133}]},"public_addr":"172.21.15.135:6809/4133828133","cluster_addr":"172.21.15.135:6811/4133828133","heartbeat_back_addr":"172.21.15.135:6815/4133828133","heartbeat_front_addr":"172.21.15.135:6813/4133828133","state":["exists","up"]},{"osd":5,"uuid":"dc6e11b9-4729-4504-8917-51a3bb5e2b7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":39,"last_clean_end":74,"up_from":78,"up_thru":78,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":1977885887},{"type":"v1","addr":"172.21.15.135:6817","nonce":1977885887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":1977885887},{"type":"v1","addr":"172.21.15.135:6819","nonce":1977885887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":1977885887},{"type":"v1","addr":"172.21.15.135:6823","nonce":1977885887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":1977885887},{"type":"v1","addr":"172.21.15.135:6821","nonce":1977885887}]},"public_addr":"172.21.15.135:6817/1977885887","cluster_addr":"172.21.15.135:6819/1977885887","heartbeat_back_addr":"172.21.15.135:6823/1977885887","heartbeat_front_addr":"172.21.15.135:6821/1977885887","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T04:20:25.138939+0000","dead_epoch":49},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T04:20:58.657719+0000","dead_epoch":54},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T04:21:32.397235+0000","dead_epoch":59},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T04:22:02.245820+0000","dead_epoch":64},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T04:22:33.842250+0000","dead_epoch":69},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T04:23:04.503576+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.123:6829/3037849622":"2024-03-31T04:37:19.516180+0000","172.21.15.123:6828/3037849622":"2024-03-31T04:37:19.516180+0000","172.21.15.123:0/1275706223":"2024-03-31T04:18:56.235750+0000","172.21.15.123:6827/2393193575":"2024-03-31T04:24:47.075751+0000","172.21.15.135:0/1604630130":"2024-03-31T04:28:49.824398+0000","172.21.15.123:0/3228440712":"2024-03-31T04:32:29.347901+0000","172.21.15.123:0/3037148222":"2024-03-31T04:17:41.459573+0000","172.21.15.123:0/2358069559":"2024-03-31T04:17:41.459573+0000","172.21.15.135:6828/3601553667":"2024-03-31T04:28:49.824398+0000","172.21.15.123:0/4218181674":"2024-03-31T04:26:59.096603+0000","172.21.15.123:0/2447404972":"2024-03-31T04:26:59.096603+0000","172.21.15.123:0/3593359364":"2024-03-31T04:18:56.235750+0000","172.21.15.135:6825/4050541062":"2024-03-31T04:37:19.516195+0000","172.21.15.135:0/1748591175":"2024-03-31T04:28:49.824398+0000","172.21.15.123:0/518566709":"2024-03-31T04:18:56.235750+0000","172.21.15.135:0/952436162":"2024-03-31T04:28:49.824398+0000","172.21.15.135:6829/3601553667":"2024-03-31T04:28:49.824398+0000","172.21.15.123:6800/1291592055":"2024-03-31T04:17:05.107043+0000","172.21.15.135:0/3366037413":"2024-03-31T04:28:49.824398+0000","172.21.15.123:6826/2393193575":"2024-03-31T04:24:47.075751+0000","172.21.15.123:6801/1776532690":"2024-03-31T04:17:41.459573+0000","172.21.15.123:0/1064275350":"2024-03-31T04:26:59.096603+0000","172.21.15.123:0/72410915":"2024-03-31T04:26:59.096603+0000","172.21.15.123:6800/2125774379":"2024-03-31T04:32:29.347901+0000","172.21.15.123:6801/1291592055":"2024-03-31T04:17:05.107043+0000","172.21.15.123:6800/1564935593":"2024-03-31T04:26:59.096603+0000","172.21.15.123:0/2756071515":"2024-03-31T04:17:05.107043+0000","172.21.15.123:0/133647565":"2024-03-31T04:17:41.459573+0000","172.21.15.123:6801/2125774379":"2024-03-31T04:32:29.347901+0000","172.21.15.123:6801/1564935593":"2024-03-31T04:26:59.096603+0000","172.21.15.123:6800/1776532690":"2024-03-31T04:17:41.459573+0000","172.21.15.123:6800/70490328":"2024-03-31T04:18:56.235750+0000","172.21.15.123:0/2366611723":"2024-03-31T04:17:05.107043+0000","172.21.15.123:6801/70490328":"2024-03-31T04:18:56.235750+0000","172.21.15.135:6824/4050541062":"2024-03-31T04:37:19.516195+0000","172.21.15.123:0/250978570":"2024-03-31T04:17:05.107043+0000","172.21.15.123:0/2743438158":"2024-03-31T04:32:29.347901+0000","172.21.15.123:0/1159868778":"2024-03-31T04:32:29.347901+0000","172.21.15.123:0/2182269825":"2024-03-31T04:32:29.347901+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T05:10:40.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.801+0000 7f548ea00700 1 -- 172.21.15.123:0/2753617568 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f546c06cc20 msgr2=0x7f546c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:40.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.801+0000 7f548ea00700 1 --2- 172.21.15.123:0/2753617568 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f546c06cc20 0x7f546c06f0d0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f5478000c00 tx=0x7f5478009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:40.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.801+0000 7f548ea00700 1 -- 172.21.15.123:0/2753617568 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5488070c70 msgr2=0x7f5488115f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:40.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.801+0000 7f548ea00700 1 --2- 172.21.15.123:0/2753617568 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5488070c70 0x7f5488115f60 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f547400c8c0 tx=0x7f54740044c0 comp rx=0 tx=0).stop 2024-03-30T05:10:40.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.801+0000 7f548ea00700 1 -- 172.21.15.123:0/2753617568 shutdown_connections 2024-03-30T05:10:40.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.801+0000 7f548ea00700 1 --2- 172.21.15.123:0/2753617568 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f546c06cc20 0x7f546c06f0d0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:40.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.801+0000 7f548ea00700 1 --2- 172.21.15.123:0/2753617568 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5488070360 0x7f5488115a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:40.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.801+0000 7f548ea00700 1 --2- 172.21.15.123:0/2753617568 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5488070c70 0x7f5488115f60 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:40.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.801+0000 7f548ea00700 1 -- 172.21.15.123:0/2753617568 >> 172.21.15.123:0/2753617568 conn(0x7f54880f8850 msgr2=0x7f54880f9d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:40.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.801+0000 7f548ea00700 1 -- 172.21.15.123:0/2753617568 shutdown_connections 2024-03-30T05:10:40.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:40.801+0000 7f548ea00700 1 -- 172.21.15.123:0/2753617568 wait complete. 2024-03-30T05:10:40.817 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T04:21:34.842970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-03-30T04:24:29.050323+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-03-30T04:24:29.704111+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-30T05:10:40.818 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-30T05:10:41.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.011+0000 7f1ad2c00700 1 -- 172.21.15.123:0/4140035061 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1acc103670 msgr2=0x7f1acc0fcce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:41.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.011+0000 7f1ad2c00700 1 --2- 172.21.15.123:0/4140035061 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1acc103670 0x7f1acc0fcce0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f1ab8004940 tx=0x7f1ab8009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:41.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.011+0000 7f1ad2c00700 1 -- 172.21.15.123:0/4140035061 shutdown_connections 2024-03-30T05:10:41.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.011+0000 7f1ad2c00700 1 --2- 172.21.15.123:0/4140035061 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1acc103670 0x7f1acc0fcce0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:41.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.011+0000 7f1ad2c00700 1 --2- 172.21.15.123:0/4140035061 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1acc102cd0 0x7f1acc1030a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:41.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.011+0000 7f1ad2c00700 1 -- 172.21.15.123:0/4140035061 >> 172.21.15.123:0/4140035061 conn(0x7f1acc0f8820 msgr2=0x7f1acc0fac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:41.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.011+0000 7f1ad2c00700 1 -- 172.21.15.123:0/4140035061 shutdown_connections 2024-03-30T05:10:41.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.011+0000 7f1ad2c00700 1 -- 172.21.15.123:0/4140035061 wait complete. 2024-03-30T05:10:41.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.012+0000 7f1ad2c00700 1 Processor -- start 2024-03-30T05:10:41.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.012+0000 7f1ad2c00700 1 -- start start 2024-03-30T05:10:41.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.012+0000 7f1ad2c00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1acc102cd0 0x7f1acc0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:41.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.012+0000 7f1ad2c00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1acc103670 0x7f1acc070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:41.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.012+0000 7f1ad2c00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1acc072210 con 0x7f1acc103670 2024-03-30T05:10:41.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.012+0000 7f1ad2c00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1acc070d40 con 0x7f1acc102cd0 2024-03-30T05:10:41.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.012+0000 7f1acb400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1acc103670 0x7f1acc070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:41.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.012+0000 7f1acb400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1acc103670 0x7f1acc070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39986/0 (socket says 172.21.15.123:39986) 2024-03-30T05:10:41.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.012+0000 7f1acb400700 1 -- 172.21.15.123:0/1930207868 learned_addr learned my addr 172.21.15.123:0/1930207868 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:10:41.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.013+0000 7f1acbe00700 1 --2- 172.21.15.123:0/1930207868 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1acc102cd0 0x7f1acc0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:41.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.013+0000 7f1acb400700 1 -- 172.21.15.123:0/1930207868 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1acc102cd0 msgr2=0x7f1acc0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:41.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.013+0000 7f1acb400700 1 --2- 172.21.15.123:0/1930207868 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1acc102cd0 0x7f1acc0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:41.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.013+0000 7f1acb400700 1 -- 172.21.15.123:0/1930207868 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab8005040 con 0x7f1acc103670 2024-03-30T05:10:41.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.013+0000 7f1acb400700 1 --2- 172.21.15.123:0/1930207868 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1acc103670 0x7f1acc070800 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f1ab8009390 tx=0x7f1ab8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:41.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.013+0000 7f1ac3e00700 1 -- 172.21.15.123:0/1930207868 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ab800e040 con 0x7f1acc103670 2024-03-30T05:10:41.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.013+0000 7f1ac3e00700 1 -- 172.21.15.123:0/1930207868 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1ab8013420 con 0x7f1acc103670 2024-03-30T05:10:41.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.013+0000 7f1ad2c00700 1 -- 172.21.15.123:0/1930207868 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1acc070fc0 con 0x7f1acc103670 2024-03-30T05:10:41.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.013+0000 7f1ac3e00700 1 -- 172.21.15.123:0/1930207868 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ab801c630 con 0x7f1acc103670 2024-03-30T05:10:41.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.013+0000 7f1ad2c00700 1 -- 172.21.15.123:0/1930207868 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1acc119ed0 con 0x7f1acc103670 2024-03-30T05:10:41.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.014+0000 7f1ad2c00700 1 -- 172.21.15.123:0/1930207868 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1acc0fe420 con 0x7f1acc103670 2024-03-30T05:10:41.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.015+0000 7f1ac3e00700 1 -- 172.21.15.123:0/1930207868 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f1ab8013a00 con 0x7f1acc103670 2024-03-30T05:10:41.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.016+0000 7f1ac3e00700 1 --2- 172.21.15.123:0/1930207868 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1ab006cc20 0x7f1ab006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:41.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.016+0000 7f1ac3e00700 1 -- 172.21.15.123:0/1930207868 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f1ab808d2c0 con 0x7f1acc103670 2024-03-30T05:10:41.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.016+0000 7f1acbe00700 1 --2- 172.21.15.123:0/1930207868 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1ab006cc20 0x7f1ab006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:41.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.017+0000 7f1acbe00700 1 --2- 172.21.15.123:0/1930207868 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1ab006cc20 0x7f1ab006f0d0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f1abc007fd0 tx=0x7f1abc009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:41.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.020+0000 7f1ac3e00700 1 -- 172.21.15.123:0/1930207868 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f1ab8057300 con 0x7f1acc103670 2024-03-30T05:10:41.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.227+0000 7f1ad2c00700 1 -- 172.21.15.123:0/1930207868 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1acc109c70 con 0x7f1acc103670 2024-03-30T05:10:41.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.228+0000 7f1ac3e00700 1 -- 172.21.15.123:0/1930207868 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v86) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1ab805a920 con 0x7f1acc103670 2024-03-30T05:10:41.226 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 1 2024-03-30T05:10:41.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.230+0000 7f1ad2c00700 1 -- 172.21.15.123:0/1930207868 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1ab006cc20 msgr2=0x7f1ab006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:41.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.230+0000 7f1ad2c00700 1 --2- 172.21.15.123:0/1930207868 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1ab006cc20 0x7f1ab006f0d0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f1abc007fd0 tx=0x7f1abc009040 comp rx=0 tx=0).stop 2024-03-30T05:10:41.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.230+0000 7f1ad2c00700 1 -- 172.21.15.123:0/1930207868 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1acc103670 msgr2=0x7f1acc070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:41.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.231+0000 7f1ad2c00700 1 --2- 172.21.15.123:0/1930207868 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1acc103670 0x7f1acc070800 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f1ab8009390 tx=0x7f1ab8009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:41.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.231+0000 7f1ad2c00700 1 -- 172.21.15.123:0/1930207868 shutdown_connections 2024-03-30T05:10:41.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.231+0000 7f1ad2c00700 1 --2- 172.21.15.123:0/1930207868 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f1ab006cc20 0x7f1ab006f0d0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:41.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.231+0000 7f1ad2c00700 1 --2- 172.21.15.123:0/1930207868 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1acc102cd0 0x7f1acc0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:41.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.231+0000 7f1ad2c00700 1 --2- 172.21.15.123:0/1930207868 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1acc103670 0x7f1acc070800 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:41.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.231+0000 7f1ad2c00700 1 -- 172.21.15.123:0/1930207868 >> 172.21.15.123:0/1930207868 conn(0x7f1acc0f8820 msgr2=0x7f1acc0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:41.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.231+0000 7f1ad2c00700 1 -- 172.21.15.123:0/1930207868 shutdown_connections 2024-03-30T05:10:41.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.231+0000 7f1ad2c00700 1 -- 172.21.15.123:0/1930207868 wait complete. 2024-03-30T05:10:41.247 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-03-30T05:10:41.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.440+0000 7f2eaf000700 1 -- 172.21.15.123:0/296320966 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ea8102cd0 msgr2=0x7f2ea81030a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:41.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.440+0000 7f2eaf000700 1 --2- 172.21.15.123:0/296320966 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ea8102cd0 0x7f2ea81030a0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f2e98004940 tx=0x7f2e98009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:41.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.440+0000 7f2eaf000700 1 -- 172.21.15.123:0/296320966 shutdown_connections 2024-03-30T05:10:41.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.440+0000 7f2eaf000700 1 --2- 172.21.15.123:0/296320966 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2ea8103670 0x7f2ea80fcce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:41.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.440+0000 7f2eaf000700 1 --2- 172.21.15.123:0/296320966 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ea8102cd0 0x7f2ea81030a0 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:41.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.440+0000 7f2eaf000700 1 -- 172.21.15.123:0/296320966 >> 172.21.15.123:0/296320966 conn(0x7f2ea80f8820 msgr2=0x7f2ea80fac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:41.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.440+0000 7f2eaf000700 1 -- 172.21.15.123:0/296320966 shutdown_connections 2024-03-30T05:10:41.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.440+0000 7f2eaf000700 1 -- 172.21.15.123:0/296320966 wait complete. 2024-03-30T05:10:41.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.441+0000 7f2eaf000700 1 Processor -- start 2024-03-30T05:10:41.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.441+0000 7f2eaf000700 1 -- start start 2024-03-30T05:10:41.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.441+0000 7f2eaf000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ea8102cd0 0x7f2ea80702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:41.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.441+0000 7f2eaf000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2ea8103670 0x7f2ea8070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:41.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.441+0000 7f2eaf000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea8072210 con 0x7f2ea8102cd0 2024-03-30T05:10:41.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.441+0000 7f2eaf000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea8070d40 con 0x7f2ea8103670 2024-03-30T05:10:41.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.442+0000 7f2ea7e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ea8102cd0 0x7f2ea80702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:41.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.442+0000 7f2ea7e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ea8102cd0 0x7f2ea80702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39992/0 (socket says 172.21.15.123:39992) 2024-03-30T05:10:41.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.442+0000 7f2ea7e00700 1 -- 172.21.15.123:0/449998493 learned_addr learned my addr 172.21.15.123:0/449998493 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:10:41.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.442+0000 7f2ea7400700 1 --2- 172.21.15.123:0/449998493 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2ea8103670 0x7f2ea8070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:41.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.442+0000 7f2ea7e00700 1 -- 172.21.15.123:0/449998493 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2ea8103670 msgr2=0x7f2ea8070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:41.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.442+0000 7f2ea7e00700 1 --2- 172.21.15.123:0/449998493 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2ea8103670 0x7f2ea8070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:41.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.442+0000 7f2ea7e00700 1 -- 172.21.15.123:0/449998493 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e98005040 con 0x7f2ea8102cd0 2024-03-30T05:10:41.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.442+0000 7f2ea7400700 1 --2- 172.21.15.123:0/449998493 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2ea8103670 0x7f2ea8070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T05:10:41.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.442+0000 7f2ea7e00700 1 --2- 172.21.15.123:0/449998493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ea8102cd0 0x7f2ea80702c0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f2e98009360 tx=0x7f2e98009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:41.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.442+0000 7f2e9fe00700 1 -- 172.21.15.123:0/449998493 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e9800e040 con 0x7f2ea8102cd0 2024-03-30T05:10:41.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.442+0000 7f2e9fe00700 1 -- 172.21.15.123:0/449998493 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2e98013420 con 0x7f2ea8102cd0 2024-03-30T05:10:41.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.442+0000 7f2eaf000700 1 -- 172.21.15.123:0/449998493 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ea8070fc0 con 0x7f2ea8102cd0 2024-03-30T05:10:41.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.442+0000 7f2e9fe00700 1 -- 172.21.15.123:0/449998493 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e9801c5f0 con 0x7f2ea8102cd0 2024-03-30T05:10:41.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.443+0000 7f2eaf000700 1 -- 172.21.15.123:0/449998493 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ea8119f10 con 0x7f2ea8102cd0 2024-03-30T05:10:41.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.444+0000 7f2eaf000700 1 -- 172.21.15.123:0/449998493 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ea80fe420 con 0x7f2ea8102cd0 2024-03-30T05:10:41.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.444+0000 7f2e9fe00700 1 -- 172.21.15.123:0/449998493 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f2e98013590 con 0x7f2ea8102cd0 2024-03-30T05:10:41.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.445+0000 7f2e9fe00700 1 --2- 172.21.15.123:0/449998493 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2e8c06cc20 0x7f2e8c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:41.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.445+0000 7f2e9fe00700 1 -- 172.21.15.123:0/449998493 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f2e9808d260 con 0x7f2ea8102cd0 2024-03-30T05:10:41.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.445+0000 7f2ea7400700 1 --2- 172.21.15.123:0/449998493 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2e8c06cc20 0x7f2e8c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:41.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.446+0000 7f2ea7400700 1 --2- 172.21.15.123:0/449998493 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2e8c06cc20 0x7f2e8c06f0d0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f2e94009fd0 tx=0x7f2e9400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:41.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.450+0000 7f2e9fe00700 1 -- 172.21.15.123:0/449998493 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2e980572a0 con 0x7f2ea8102cd0 2024-03-30T05:10:41.465 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:41 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/4192962512' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 41, "format": "json"}]: dispatch 2024-03-30T05:10:41.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:41 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/2753617568' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T05:10:41.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:41 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/4192962512' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 41, "format": "json"}]: dispatch 2024-03-30T05:10:41.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:41 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/2753617568' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T05:10:41.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.653+0000 7f2eaf000700 1 -- 172.21.15.123:0/449998493 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f2ea8064970 con 0x7f2ea8102cd0 2024-03-30T05:10:41.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.653+0000 7f2e9fe00700 1 -- 172.21.15.123:0/449998493 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v86) v1 ==== 107+0+11 (secure 0 0 0) 0x7f2e9805a8c0 con 0x7f2ea8102cd0 2024-03-30T05:10:41.651 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 32 2024-03-30T05:10:41.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.656+0000 7f2eaf000700 1 -- 172.21.15.123:0/449998493 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2e8c06cc20 msgr2=0x7f2e8c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:41.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.656+0000 7f2eaf000700 1 --2- 172.21.15.123:0/449998493 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2e8c06cc20 0x7f2e8c06f0d0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f2e94009fd0 tx=0x7f2e9400b040 comp rx=0 tx=0).stop 2024-03-30T05:10:41.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.656+0000 7f2eaf000700 1 -- 172.21.15.123:0/449998493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ea8102cd0 msgr2=0x7f2ea80702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:41.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.656+0000 7f2eaf000700 1 --2- 172.21.15.123:0/449998493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ea8102cd0 0x7f2ea80702c0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f2e98009360 tx=0x7f2e98009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:41.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.656+0000 7f2eaf000700 1 -- 172.21.15.123:0/449998493 shutdown_connections 2024-03-30T05:10:41.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.656+0000 7f2eaf000700 1 --2- 172.21.15.123:0/449998493 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f2e8c06cc20 0x7f2e8c06f0d0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:41.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.656+0000 7f2eaf000700 1 --2- 172.21.15.123:0/449998493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ea8102cd0 0x7f2ea80702c0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:41.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.656+0000 7f2eaf000700 1 --2- 172.21.15.123:0/449998493 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2ea8103670 0x7f2ea8070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:41.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.656+0000 7f2eaf000700 1 -- 172.21.15.123:0/449998493 >> 172.21.15.123:0/449998493 conn(0x7f2ea80f8820 msgr2=0x7f2ea80f98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:41.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.656+0000 7f2eaf000700 1 -- 172.21.15.123:0/449998493 shutdown_connections 2024-03-30T05:10:41.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.656+0000 7f2eaf000700 1 -- 172.21.15.123:0/449998493 wait complete. 2024-03-30T05:10:41.671 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-03-30T05:10:41.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.864+0000 7f7c42800700 1 -- 172.21.15.123:0/927511850 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c3c103680 msgr2=0x7f7c3c064e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:41.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.864+0000 7f7c42800700 1 --2- 172.21.15.123:0/927511850 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c3c103680 0x7f7c3c064e40 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f7c28004940 tx=0x7f7c28009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:41.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.864+0000 7f7c42800700 1 -- 172.21.15.123:0/927511850 shutdown_connections 2024-03-30T05:10:41.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.864+0000 7f7c42800700 1 --2- 172.21.15.123:0/927511850 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c3c103680 0x7f7c3c064e40 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:41.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.864+0000 7f7c42800700 1 --2- 172.21.15.123:0/927511850 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7c3c102ce0 0x7f7c3c1030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:41.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.864+0000 7f7c42800700 1 -- 172.21.15.123:0/927511850 >> 172.21.15.123:0/927511850 conn(0x7f7c3c072ad0 msgr2=0x7f7c3c072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:41.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.865+0000 7f7c42800700 1 -- 172.21.15.123:0/927511850 shutdown_connections 2024-03-30T05:10:41.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.865+0000 7f7c42800700 1 -- 172.21.15.123:0/927511850 wait complete. 2024-03-30T05:10:41.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.865+0000 7f7c42800700 1 Processor -- start 2024-03-30T05:10:41.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.865+0000 7f7c42800700 1 -- start start 2024-03-30T05:10:41.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.866+0000 7f7c42800700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c3c102ce0 0x7f7c3c115980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:41.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.866+0000 7f7c42800700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7c3c103680 0x7f7c3c115ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:41.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.866+0000 7f7c42800700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3c116510 con 0x7f7c3c102ce0 2024-03-30T05:10:41.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.866+0000 7f7c42800700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3c10fb60 con 0x7f7c3c103680 2024-03-30T05:10:41.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.866+0000 7f7c3b400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c3c102ce0 0x7f7c3c115980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:41.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.866+0000 7f7c3b400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c3c102ce0 0x7f7c3c115980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40002/0 (socket says 172.21.15.123:40002) 2024-03-30T05:10:41.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.866+0000 7f7c3b400700 1 -- 172.21.15.123:0/422292252 learned_addr learned my addr 172.21.15.123:0/422292252 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:10:41.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.866+0000 7f7c3b400700 1 -- 172.21.15.123:0/422292252 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7c3c103680 msgr2=0x7f7c3c115ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:41.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.866+0000 7f7c3b400700 1 --2- 172.21.15.123:0/422292252 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7c3c103680 0x7f7c3c115ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:41.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.866+0000 7f7c3b400700 1 -- 172.21.15.123:0/422292252 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c28005040 con 0x7f7c3c102ce0 2024-03-30T05:10:41.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.867+0000 7f7c3b400700 1 --2- 172.21.15.123:0/422292252 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c3c102ce0 0x7f7c3c115980 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f7c2c002a50 tx=0x7f7c2c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:41.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.867+0000 7f7c33400700 1 -- 172.21.15.123:0/422292252 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c2c007cc0 con 0x7f7c3c102ce0 2024-03-30T05:10:41.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.867+0000 7f7c33400700 1 -- 172.21.15.123:0/422292252 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7c2c00f040 con 0x7f7c3c102ce0 2024-03-30T05:10:41.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.867+0000 7f7c42800700 1 -- 172.21.15.123:0/422292252 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c3c10fe40 con 0x7f7c3c102ce0 2024-03-30T05:10:41.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.867+0000 7f7c33400700 1 -- 172.21.15.123:0/422292252 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c2c009540 con 0x7f7c3c102ce0 2024-03-30T05:10:41.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.867+0000 7f7c42800700 1 -- 172.21.15.123:0/422292252 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c3c110400 con 0x7f7c3c102ce0 2024-03-30T05:10:41.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.868+0000 7f7c42800700 1 -- 172.21.15.123:0/422292252 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c3c1068f0 con 0x7f7c3c102ce0 2024-03-30T05:10:41.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.869+0000 7f7c33400700 1 -- 172.21.15.123:0/422292252 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f7c2c0096a0 con 0x7f7c3c102ce0 2024-03-30T05:10:41.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.870+0000 7f7c33400700 1 --2- 172.21.15.123:0/422292252 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7c2006cc20 0x7f7c2006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:41.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.870+0000 7f7c33400700 1 -- 172.21.15.123:0/422292252 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7c2c08bd60 con 0x7f7c3c102ce0 2024-03-30T05:10:41.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.870+0000 7f7c3aa00700 1 --2- 172.21.15.123:0/422292252 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7c2006cc20 0x7f7c2006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:41.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.871+0000 7f7c3aa00700 1 --2- 172.21.15.123:0/422292252 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7c2006cc20 0x7f7c2006f0d0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f7c28004f90 tx=0x7f7c28009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:41.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:41.875+0000 7f7c33400700 1 -- 172.21.15.123:0/422292252 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7c2c059830 con 0x7f7c3c102ce0 2024-03-30T05:10:42.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.080+0000 7f7c42800700 1 -- 172.21.15.123:0/422292252 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f7c3c110fd0 con 0x7f7c3c102ce0 2024-03-30T05:10:42.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.080+0000 7f7c33400700 1 -- 172.21.15.123:0/422292252 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v86) v1 ==== 107+0+11 (secure 0 0 0) 0x7f7c2c0593c0 con 0x7f7c3c102ce0 2024-03-30T05:10:42.078 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 32 2024-03-30T05:10:42.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.082+0000 7f7c42800700 1 -- 172.21.15.123:0/422292252 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7c2006cc20 msgr2=0x7f7c2006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:42.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.082+0000 7f7c42800700 1 --2- 172.21.15.123:0/422292252 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7c2006cc20 0x7f7c2006f0d0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f7c28004f90 tx=0x7f7c28009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:42.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.082+0000 7f7c42800700 1 -- 172.21.15.123:0/422292252 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c3c102ce0 msgr2=0x7f7c3c115980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:42.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.082+0000 7f7c42800700 1 --2- 172.21.15.123:0/422292252 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c3c102ce0 0x7f7c3c115980 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f7c2c002a50 tx=0x7f7c2c0046b0 comp rx=0 tx=0).stop 2024-03-30T05:10:42.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.083+0000 7f7c42800700 1 -- 172.21.15.123:0/422292252 shutdown_connections 2024-03-30T05:10:42.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.083+0000 7f7c42800700 1 --2- 172.21.15.123:0/422292252 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f7c2006cc20 0x7f7c2006f0d0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:42.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.083+0000 7f7c42800700 1 --2- 172.21.15.123:0/422292252 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c3c102ce0 0x7f7c3c115980 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:42.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.083+0000 7f7c42800700 1 --2- 172.21.15.123:0/422292252 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7c3c103680 0x7f7c3c115ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:42.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.083+0000 7f7c42800700 1 -- 172.21.15.123:0/422292252 >> 172.21.15.123:0/422292252 conn(0x7f7c3c072ad0 msgr2=0x7f7c3c103a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:42.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.083+0000 7f7c42800700 1 -- 172.21.15.123:0/422292252 shutdown_connections 2024-03-30T05:10:42.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.083+0000 7f7c42800700 1 -- 172.21.15.123:0/422292252 wait complete. 2024-03-30T05:10:42.098 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-30T05:10:42.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.290+0000 7fe53f600700 1 -- 172.21.15.123:0/3790285586 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5380708a0 msgr2=0x7fe538070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:42.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.290+0000 7fe53f600700 1 --2- 172.21.15.123:0/3790285586 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5380708a0 0x7fe538070d10 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fe524004970 tx=0x7fe524009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:42.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.290+0000 7fe53f600700 1 -- 172.21.15.123:0/3790285586 shutdown_connections 2024-03-30T05:10:42.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.290+0000 7fe53f600700 1 --2- 172.21.15.123:0/3790285586 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5380708a0 0x7fe538070d10 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:42.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.290+0000 7fe53f600700 1 --2- 172.21.15.123:0/3790285586 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe538071f90 0x7fe538070360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:42.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.290+0000 7fe53f600700 1 -- 172.21.15.123:0/3790285586 >> 172.21.15.123:0/3790285586 conn(0x7fe5380f8820 msgr2=0x7fe5380fac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:42.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.290+0000 7fe53f600700 1 -- 172.21.15.123:0/3790285586 shutdown_connections 2024-03-30T05:10:42.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.291+0000 7fe53f600700 1 -- 172.21.15.123:0/3790285586 wait complete. 2024-03-30T05:10:42.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.291+0000 7fe53f600700 1 Processor -- start 2024-03-30T05:10:42.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.291+0000 7fe53f600700 1 -- start start 2024-03-30T05:10:42.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.292+0000 7fe53f600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe5380708a0 0x7fe538119d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:42.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.292+0000 7fe53f600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe538071f90 0x7fe53811a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:42.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.292+0000 7fe53f600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe53811a8c0 con 0x7fe538071f90 2024-03-30T05:10:42.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.292+0000 7fe53f600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe538113ee0 con 0x7fe5380708a0 2024-03-30T05:10:42.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.292+0000 7fe537e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe538071f90 0x7fe53811a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:42.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.292+0000 7fe537e00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe538071f90 0x7fe53811a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40024/0 (socket says 172.21.15.123:40024) 2024-03-30T05:10:42.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.292+0000 7fe537e00700 1 -- 172.21.15.123:0/3002824319 learned_addr learned my addr 172.21.15.123:0/3002824319 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:10:42.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.292+0000 7fe53cc00700 1 --2- 172.21.15.123:0/3002824319 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe5380708a0 0x7fe538119d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:42.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.292+0000 7fe537e00700 1 -- 172.21.15.123:0/3002824319 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe5380708a0 msgr2=0x7fe538119d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:42.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.292+0000 7fe537e00700 1 --2- 172.21.15.123:0/3002824319 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe5380708a0 0x7fe538119d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:42.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.292+0000 7fe537e00700 1 -- 172.21.15.123:0/3002824319 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe524005040 con 0x7fe538071f90 2024-03-30T05:10:42.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.292+0000 7fe53cc00700 1 --2- 172.21.15.123:0/3002824319 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe5380708a0 0x7fe538119d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T05:10:42.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.293+0000 7fe537e00700 1 --2- 172.21.15.123:0/3002824319 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe538071f90 0x7fe53811a270 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fe524004f50 tx=0x7fe524009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:42.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.293+0000 7fe534c00700 1 -- 172.21.15.123:0/3002824319 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe52400e040 con 0x7fe538071f90 2024-03-30T05:10:42.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.293+0000 7fe53f600700 1 -- 172.21.15.123:0/3002824319 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe538114160 con 0x7fe538071f90 2024-03-30T05:10:42.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.293+0000 7fe534c00700 1 -- 172.21.15.123:0/3002824319 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe524013420 con 0x7fe538071f90 2024-03-30T05:10:42.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.293+0000 7fe534c00700 1 -- 172.21.15.123:0/3002824319 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe52401c610 con 0x7fe538071f90 2024-03-30T05:10:42.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.293+0000 7fe53f600700 1 -- 172.21.15.123:0/3002824319 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5381146a0 con 0x7fe538071f90 2024-03-30T05:10:42.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.294+0000 7fe53f600700 1 -- 172.21.15.123:0/3002824319 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe538071aa0 con 0x7fe538071f90 2024-03-30T05:10:42.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.295+0000 7fe534c00700 1 -- 172.21.15.123:0/3002824319 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fe524013590 con 0x7fe538071f90 2024-03-30T05:10:42.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.296+0000 7fe534c00700 1 --2- 172.21.15.123:0/3002824319 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe52006cc20 0x7fe52006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:42.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.296+0000 7fe534c00700 1 -- 172.21.15.123:0/3002824319 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe52408d260 con 0x7fe538071f90 2024-03-30T05:10:42.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.296+0000 7fe53cc00700 1 --2- 172.21.15.123:0/3002824319 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe52006cc20 0x7fe52006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:42.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.297+0000 7fe53cc00700 1 --2- 172.21.15.123:0/3002824319 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe52006cc20 0x7fe52006f0d0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fe52c009b80 tx=0x7fe52c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:42.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.300+0000 7fe534c00700 1 -- 172.21.15.123:0/3002824319 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe524057220 con 0x7fe538071f90 2024-03-30T05:10:42.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:42 smithi123 ceph-mon[139369]: pgmap v1187: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:10:42.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:42 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/1930207868' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T05:10:42.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:42 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/449998493' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-30T05:10:42.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:42 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/422292252' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-30T05:10:42.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.522+0000 7fe53f600700 1 -- 172.21.15.123:0/3002824319 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe5381102b0 con 0x7fe538071f90 2024-03-30T05:10:42.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.522+0000 7fe534c00700 1 -- 172.21.15.123:0/3002824319 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 39 entries v86) v1 ==== 81+0+2345 (secure 0 0 0) 0x7fe52405a840 con 0x7fe538071f90 2024-03-30T05:10:42.520 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:6829/3037849622 2024-03-31T04:37:19.516180+0000 2024-03-30T05:10:42.521 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:6828/3037849622 2024-03-31T04:37:19.516180+0000 2024-03-30T05:10:42.521 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:0/1275706223 2024-03-31T04:18:56.235750+0000 2024-03-30T05:10:42.521 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:6827/2393193575 2024-03-31T04:24:47.075751+0000 2024-03-30T05:10:42.521 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.135:0/1604630130 2024-03-31T04:28:49.824398+0000 2024-03-30T05:10:42.521 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:0/3228440712 2024-03-31T04:32:29.347901+0000 2024-03-30T05:10:42.521 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:0/3037148222 2024-03-31T04:17:41.459573+0000 2024-03-30T05:10:42.521 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:0/2358069559 2024-03-31T04:17:41.459573+0000 2024-03-30T05:10:42.521 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.135:6828/3601553667 2024-03-31T04:28:49.824398+0000 2024-03-30T05:10:42.521 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:0/4218181674 2024-03-31T04:26:59.096603+0000 2024-03-30T05:10:42.521 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:0/2447404972 2024-03-31T04:26:59.096603+0000 2024-03-30T05:10:42.521 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:0/3593359364 2024-03-31T04:18:56.235750+0000 2024-03-30T05:10:42.522 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.135:6825/4050541062 2024-03-31T04:37:19.516195+0000 2024-03-30T05:10:42.522 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.135:0/1748591175 2024-03-31T04:28:49.824398+0000 2024-03-30T05:10:42.522 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:0/518566709 2024-03-31T04:18:56.235750+0000 2024-03-30T05:10:42.522 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.135:0/952436162 2024-03-31T04:28:49.824398+0000 2024-03-30T05:10:42.522 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.135:6829/3601553667 2024-03-31T04:28:49.824398+0000 2024-03-30T05:10:42.522 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:6800/1291592055 2024-03-31T04:17:05.107043+0000 2024-03-30T05:10:42.522 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.135:0/3366037413 2024-03-31T04:28:49.824398+0000 2024-03-30T05:10:42.522 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:6826/2393193575 2024-03-31T04:24:47.075751+0000 2024-03-30T05:10:42.522 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:6801/1776532690 2024-03-31T04:17:41.459573+0000 2024-03-30T05:10:42.522 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:0/1064275350 2024-03-31T04:26:59.096603+0000 2024-03-30T05:10:42.523 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:0/72410915 2024-03-31T04:26:59.096603+0000 2024-03-30T05:10:42.523 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:6800/2125774379 2024-03-31T04:32:29.347901+0000 2024-03-30T05:10:42.523 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:6801/1291592055 2024-03-31T04:17:05.107043+0000 2024-03-30T05:10:42.523 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:6800/1564935593 2024-03-31T04:26:59.096603+0000 2024-03-30T05:10:42.523 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:0/2756071515 2024-03-31T04:17:05.107043+0000 2024-03-30T05:10:42.523 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:0/133647565 2024-03-31T04:17:41.459573+0000 2024-03-30T05:10:42.523 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:6801/2125774379 2024-03-31T04:32:29.347901+0000 2024-03-30T05:10:42.523 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:6801/1564935593 2024-03-31T04:26:59.096603+0000 2024-03-30T05:10:42.523 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:6800/1776532690 2024-03-31T04:17:41.459573+0000 2024-03-30T05:10:42.523 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:6800/70490328 2024-03-31T04:18:56.235750+0000 2024-03-30T05:10:42.523 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:0/2366611723 2024-03-31T04:17:05.107043+0000 2024-03-30T05:10:42.523 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:6801/70490328 2024-03-31T04:18:56.235750+0000 2024-03-30T05:10:42.524 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.135:6824/4050541062 2024-03-31T04:37:19.516195+0000 2024-03-30T05:10:42.524 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:0/250978570 2024-03-31T04:17:05.107043+0000 2024-03-30T05:10:42.524 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:0/2743438158 2024-03-31T04:32:29.347901+0000 2024-03-30T05:10:42.524 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:0/1159868778 2024-03-31T04:32:29.347901+0000 2024-03-30T05:10:42.524 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:0/2182269825 2024-03-31T04:32:29.347901+0000 2024-03-30T05:10:42.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.525+0000 7fe53f600700 1 -- 172.21.15.123:0/3002824319 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe52006cc20 msgr2=0x7fe52006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:42.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.525+0000 7fe53f600700 1 --2- 172.21.15.123:0/3002824319 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe52006cc20 0x7fe52006f0d0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fe52c009b80 tx=0x7fe52c00b040 comp rx=0 tx=0).stop 2024-03-30T05:10:42.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.525+0000 7fe53f600700 1 -- 172.21.15.123:0/3002824319 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe538071f90 msgr2=0x7fe53811a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:42.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.525+0000 7fe53f600700 1 --2- 172.21.15.123:0/3002824319 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe538071f90 0x7fe53811a270 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fe524004f50 tx=0x7fe524009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:42.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.525+0000 7fe53f600700 1 -- 172.21.15.123:0/3002824319 shutdown_connections 2024-03-30T05:10:42.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.525+0000 7fe53f600700 1 --2- 172.21.15.123:0/3002824319 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fe52006cc20 0x7fe52006f0d0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:42.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.525+0000 7fe53f600700 1 --2- 172.21.15.123:0/3002824319 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe5380708a0 0x7fe538119d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:42.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.525+0000 7fe53f600700 1 --2- 172.21.15.123:0/3002824319 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe538071f90 0x7fe53811a270 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:42.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.525+0000 7fe53f600700 1 -- 172.21.15.123:0/3002824319 >> 172.21.15.123:0/3002824319 conn(0x7fe5380f8820 msgr2=0x7fe5380f9a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:42.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.525+0000 7fe53f600700 1 -- 172.21.15.123:0/3002824319 shutdown_connections 2024-03-30T05:10:42.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.525+0000 7fe53f600700 1 -- 172.21.15.123:0/3002824319 wait complete. 2024-03-30T05:10:42.526 INFO:teuthology.orchestra.run.smithi123.stderr:listed 39 entries 2024-03-30T05:10:42.541 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-30T05:10:42.541 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-30T05:10:42.587 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T05:10:42.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:42 smithi135 ceph-mon[106011]: pgmap v1187: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:10:42.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:42 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/1930207868' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T05:10:42.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:42 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/449998493' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-30T05:10:42.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:42 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/422292252' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-30T05:10:42.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.817+0000 7fae47000700 1 -- 172.21.15.123:0/952696336 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae40105110 msgr2=0x7fae401054e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:42.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.817+0000 7fae47000700 1 --2- 172.21.15.123:0/952696336 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae40105110 0x7fae401054e0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fae30004940 tx=0x7fae30009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:42.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.817+0000 7fae47000700 1 -- 172.21.15.123:0/952696336 shutdown_connections 2024-03-30T05:10:42.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.817+0000 7fae47000700 1 --2- 172.21.15.123:0/952696336 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fae40105ab0 0x7fae400ff120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:42.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.817+0000 7fae47000700 1 --2- 172.21.15.123:0/952696336 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae40105110 0x7fae401054e0 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:42.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.817+0000 7fae47000700 1 -- 172.21.15.123:0/952696336 >> 172.21.15.123:0/952696336 conn(0x7fae400fac80 msgr2=0x7fae400fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:42.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.817+0000 7fae47000700 1 -- 172.21.15.123:0/952696336 shutdown_connections 2024-03-30T05:10:42.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.817+0000 7fae47000700 1 -- 172.21.15.123:0/952696336 wait complete. 2024-03-30T05:10:42.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.818+0000 7fae47000700 1 Processor -- start 2024-03-30T05:10:42.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.818+0000 7fae47000700 1 -- start start 2024-03-30T05:10:42.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.819+0000 7fae47000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fae40105110 0x7fae40115af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:42.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.819+0000 7fae47000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae40105ab0 0x7fae40116030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:42.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.819+0000 7fae47000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae40116680 con 0x7fae40105ab0 2024-03-30T05:10:42.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.819+0000 7fae47000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae4010fca0 con 0x7fae40105110 2024-03-30T05:10:42.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.819+0000 7fae3f400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae40105ab0 0x7fae40116030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:42.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.819+0000 7fae3f400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae40105ab0 0x7fae40116030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40042/0 (socket says 172.21.15.123:40042) 2024-03-30T05:10:42.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.819+0000 7fae3f400700 1 -- 172.21.15.123:0/250954368 learned_addr learned my addr 172.21.15.123:0/250954368 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:10:42.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.819+0000 7fae3fe00700 1 --2- 172.21.15.123:0/250954368 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fae40105110 0x7fae40115af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:42.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.819+0000 7fae3f400700 1 -- 172.21.15.123:0/250954368 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fae40105110 msgr2=0x7fae40115af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:42.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.819+0000 7fae3f400700 1 --2- 172.21.15.123:0/250954368 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fae40105110 0x7fae40115af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:42.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.819+0000 7fae3f400700 1 -- 172.21.15.123:0/250954368 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae30005040 con 0x7fae40105ab0 2024-03-30T05:10:42.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.820+0000 7fae3f400700 1 --2- 172.21.15.123:0/250954368 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae40105ab0 0x7fae40116030 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fae2c00eb40 tx=0x7fae2c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:42.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.820+0000 7fae37e00700 1 -- 172.21.15.123:0/250954368 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae2c007cc0 con 0x7fae40105ab0 2024-03-30T05:10:42.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.820+0000 7fae37e00700 1 -- 172.21.15.123:0/250954368 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fae2c00f040 con 0x7fae40105ab0 2024-03-30T05:10:42.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.820+0000 7fae47000700 1 -- 172.21.15.123:0/250954368 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae4010ff20 con 0x7fae40105ab0 2024-03-30T05:10:42.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.820+0000 7fae37e00700 1 -- 172.21.15.123:0/250954368 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae2c00c8f0 con 0x7fae40105ab0 2024-03-30T05:10:42.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.820+0000 7fae47000700 1 -- 172.21.15.123:0/250954368 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae401104e0 con 0x7fae40105ab0 2024-03-30T05:10:42.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.821+0000 7fae47000700 1 -- 172.21.15.123:0/250954368 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae40100860 con 0x7fae40105ab0 2024-03-30T05:10:42.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.822+0000 7fae37e00700 1 -- 172.21.15.123:0/250954368 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fae2c0090d0 con 0x7fae40105ab0 2024-03-30T05:10:42.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.823+0000 7fae37e00700 1 --2- 172.21.15.123:0/250954368 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fae2406cc20 0x7fae2406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:42.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.823+0000 7fae37e00700 1 -- 172.21.15.123:0/250954368 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fae2c08ba50 con 0x7fae40105ab0 2024-03-30T05:10:42.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.824+0000 7fae3fe00700 1 --2- 172.21.15.123:0/250954368 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fae2406cc20 0x7fae2406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:42.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.824+0000 7fae3fe00700 1 --2- 172.21.15.123:0/250954368 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fae2406cc20 0x7fae2406f0d0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fae300045a0 tx=0x7fae30009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:42.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:42.828+0000 7fae37e00700 1 -- 172.21.15.123:0/250954368 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fae2c055ac0 con 0x7fae40105ab0 2024-03-30T05:10:43.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.032+0000 7fae47000700 1 -- 172.21.15.123:0/250954368 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fae40064310 con 0x7fae40105ab0 2024-03-30T05:10:43.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.034+0000 7fae37e00700 1 -- 172.21.15.123:0/250954368 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+16625 (secure 0 0 0) 0x7fae2c0590e0 con 0x7fae40105ab0 2024-03-30T05:10:43.032 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-03-30T05:10:43.032 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":86,"fsid":"2e77a2be-ee4c-11ee-b647-cb9ed24678a4","created":"2024-03-30T04:16:30.949442+0000","modified":"2024-03-30T04:53:15.872216+0000","last_up_change":"2024-03-30T04:37:14.509879+0000","last_in_change":"2024-03-30T04:22:44.504478+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T04:21:34.842970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-03-30T04:24:29.050323+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-03-30T04:24:29.704111+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3968446b-8696-4eec-b640-c97973ca054d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":49,"up_from":53,"up_thru":78,"down_at":50,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":1500077196},{"type":"v1","addr":"172.21.15.123:6803","nonce":1500077196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":1500077196},{"type":"v1","addr":"172.21.15.123:6805","nonce":1500077196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1500077196},{"type":"v1","addr":"172.21.15.123:6809","nonce":1500077196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":1500077196},{"type":"v1","addr":"172.21.15.123:6807","nonce":1500077196}]},"public_addr":"172.21.15.123:6803/1500077196","cluster_addr":"172.21.15.123:6805/1500077196","heartbeat_back_addr":"172.21.15.123:6809/1500077196","heartbeat_front_addr":"172.21.15.123:6807/1500077196","state":["exists","up"]},{"osd":1,"uuid":"336be34b-bb3c-46ab-8439-af04a1366db5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":54,"up_from":58,"up_thru":78,"down_at":55,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":4217392586},{"type":"v1","addr":"172.21.15.123:6811","nonce":4217392586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":4217392586},{"type":"v1","addr":"172.21.15.123:6813","nonce":4217392586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4217392586},{"type":"v1","addr":"172.21.15.123:6817","nonce":4217392586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":4217392586},{"type":"v1","addr":"172.21.15.123:6815","nonce":4217392586}]},"public_addr":"172.21.15.123:6811/4217392586","cluster_addr":"172.21.15.123:6813/4217392586","heartbeat_back_addr":"172.21.15.123:6817/4217392586","heartbeat_front_addr":"172.21.15.123:6815/4217392586","state":["exists","up"]},{"osd":2,"uuid":"4386fc8a-e9ed-413e-bd3a-bb84a0e62827","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":59,"up_from":63,"up_thru":78,"down_at":60,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":3356591260},{"type":"v1","addr":"172.21.15.123:6819","nonce":3356591260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":3356591260},{"type":"v1","addr":"172.21.15.123:6821","nonce":3356591260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":3356591260},{"type":"v1","addr":"172.21.15.123:6825","nonce":3356591260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":3356591260},{"type":"v1","addr":"172.21.15.123:6823","nonce":3356591260}]},"public_addr":"172.21.15.123:6819/3356591260","cluster_addr":"172.21.15.123:6821/3356591260","heartbeat_back_addr":"172.21.15.123:6825/3356591260","heartbeat_front_addr":"172.21.15.123:6823/3356591260","state":["exists","up"]},{"osd":3,"uuid":"086d86d4-ff1a-47dd-8416-447ccf9064ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":64,"up_from":68,"up_thru":78,"down_at":65,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":978771295},{"type":"v1","addr":"172.21.15.135:6801","nonce":978771295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":978771295},{"type":"v1","addr":"172.21.15.135:6803","nonce":978771295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":978771295},{"type":"v1","addr":"172.21.15.135:6807","nonce":978771295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":978771295},{"type":"v1","addr":"172.21.15.135:6805","nonce":978771295}]},"public_addr":"172.21.15.135:6801/978771295","cluster_addr":"172.21.15.135:6803/978771295","heartbeat_back_addr":"172.21.15.135:6807/978771295","heartbeat_front_addr":"172.21.15.135:6805/978771295","state":["exists","up"]},{"osd":4,"uuid":"31868cec-dd4d-44dd-a150-7ebdbd61ea4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":34,"last_clean_end":69,"up_from":73,"up_thru":78,"down_at":70,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":4133828133},{"type":"v1","addr":"172.21.15.135:6809","nonce":4133828133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":4133828133},{"type":"v1","addr":"172.21.15.135:6811","nonce":4133828133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":4133828133},{"type":"v1","addr":"172.21.15.135:6815","nonce":4133828133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":4133828133},{"type":"v1","addr":"172.21.15.135:6813","nonce":4133828133}]},"public_addr":"172.21.15.135:6809/4133828133","cluster_addr":"172.21.15.135:6811/4133828133","heartbeat_back_addr":"172.21.15.135:6815/4133828133","heartbeat_front_addr":"172.21.15.135:6813/4133828133","state":["exists","up"]},{"osd":5,"uuid":"dc6e11b9-4729-4504-8917-51a3bb5e2b7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":39,"last_clean_end":74,"up_from":78,"up_thru":78,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":1977885887},{"type":"v1","addr":"172.21.15.135:6817","nonce":1977885887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":1977885887},{"type":"v1","addr":"172.21.15.135:6819","nonce":1977885887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":1977885887},{"type":"v1","addr":"172.21.15.135:6823","nonce":1977885887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":1977885887},{"type":"v1","addr":"172.21.15.135:6821","nonce":1977885887}]},"public_addr":"172.21.15.135:6817/1977885887","cluster_addr":"172.21.15.135:6819/1977885887","heartbeat_back_addr":"172.21.15.135:6823/1977885887","heartbeat_front_addr":"172.21.15.135:6821/1977885887","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T04:20:25.138939+0000","dead_epoch":49},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T04:20:58.657719+0000","dead_epoch":54},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T04:21:32.397235+0000","dead_epoch":59},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T04:22:02.245820+0000","dead_epoch":64},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T04:22:33.842250+0000","dead_epoch":69},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T04:23:04.503576+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.123:6829/3037849622":"2024-03-31T04:37:19.516180+0000","172.21.15.123:6828/3037849622":"2024-03-31T04:37:19.516180+0000","172.21.15.123:0/1275706223":"2024-03-31T04:18:56.235750+0000","172.21.15.123:6827/2393193575":"2024-03-31T04:24:47.075751+0000","172.21.15.135:0/1604630130":"2024-03-31T04:28:49.824398+0000","172.21.15.123:0/3228440712":"2024-03-31T04:32:29.347901+0000","172.21.15.123:0/3037148222":"2024-03-31T04:17:41.459573+0000","172.21.15.123:0/2358069559":"2024-03-31T04:17:41.459573+0000","172.21.15.135:6828/3601553667":"2024-03-31T04:28:49.824398+0000","172.21.15.123:0/4218181674":"2024-03-31T04:26:59.096603+0000","172.21.15.123:0/2447404972":"2024-03-31T04:26:59.096603+0000","172.21.15.123:0/3593359364":"2024-03-31T04:18:56.235750+0000","172.21.15.135:6825/4050541062":"2024-03-31T04:37:19.516195+0000","172.21.15.135:0/1748591175":"2024-03-31T04:28:49.824398+0000","172.21.15.123:0/518566709":"2024-03-31T04:18:56.235750+0000","172.21.15.135:0/952436162":"2024-03-31T04:28:49.824398+0000","172.21.15.135:6829/3601553667":"2024-03-31T04:28:49.824398+0000","172.21.15.123:6800/1291592055":"2024-03-31T04:17:05.107043+0000","172.21.15.135:0/3366037413":"2024-03-31T04:28:49.824398+0000","172.21.15.123:6826/2393193575":"2024-03-31T04:24:47.075751+0000","172.21.15.123:6801/1776532690":"2024-03-31T04:17:41.459573+0000","172.21.15.123:0/1064275350":"2024-03-31T04:26:59.096603+0000","172.21.15.123:0/72410915":"2024-03-31T04:26:59.096603+0000","172.21.15.123:6800/2125774379":"2024-03-31T04:32:29.347901+0000","172.21.15.123:6801/1291592055":"2024-03-31T04:17:05.107043+0000","172.21.15.123:6800/1564935593":"2024-03-31T04:26:59.096603+0000","172.21.15.123:0/2756071515":"2024-03-31T04:17:05.107043+0000","172.21.15.123:0/133647565":"2024-03-31T04:17:41.459573+0000","172.21.15.123:6801/2125774379":"2024-03-31T04:32:29.347901+0000","172.21.15.123:6801/1564935593":"2024-03-31T04:26:59.096603+0000","172.21.15.123:6800/1776532690":"2024-03-31T04:17:41.459573+0000","172.21.15.123:6800/70490328":"2024-03-31T04:18:56.235750+0000","172.21.15.123:0/2366611723":"2024-03-31T04:17:05.107043+0000","172.21.15.123:6801/70490328":"2024-03-31T04:18:56.235750+0000","172.21.15.135:6824/4050541062":"2024-03-31T04:37:19.516195+0000","172.21.15.123:0/250978570":"2024-03-31T04:17:05.107043+0000","172.21.15.123:0/2743438158":"2024-03-31T04:32:29.347901+0000","172.21.15.123:0/1159868778":"2024-03-31T04:32:29.347901+0000","172.21.15.123:0/2182269825":"2024-03-31T04:32:29.347901+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T05:10:43.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.036+0000 7fae47000700 1 -- 172.21.15.123:0/250954368 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fae2406cc20 msgr2=0x7fae2406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:43.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.036+0000 7fae47000700 1 --2- 172.21.15.123:0/250954368 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fae2406cc20 0x7fae2406f0d0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fae300045a0 tx=0x7fae30009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:43.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.036+0000 7fae47000700 1 -- 172.21.15.123:0/250954368 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae40105ab0 msgr2=0x7fae40116030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:43.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.036+0000 7fae47000700 1 --2- 172.21.15.123:0/250954368 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae40105ab0 0x7fae40116030 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fae2c00eb40 tx=0x7fae2c00b040 comp rx=0 tx=0).stop 2024-03-30T05:10:43.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.036+0000 7fae47000700 1 -- 172.21.15.123:0/250954368 shutdown_connections 2024-03-30T05:10:43.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.036+0000 7fae47000700 1 --2- 172.21.15.123:0/250954368 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fae2406cc20 0x7fae2406f0d0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:43.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.036+0000 7fae47000700 1 --2- 172.21.15.123:0/250954368 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fae40105110 0x7fae40115af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:43.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.036+0000 7fae47000700 1 --2- 172.21.15.123:0/250954368 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae40105ab0 0x7fae40116030 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:43.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.036+0000 7fae47000700 1 -- 172.21.15.123:0/250954368 >> 172.21.15.123:0/250954368 conn(0x7fae400fac80 msgr2=0x7fae400fbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:43.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.036+0000 7fae47000700 1 -- 172.21.15.123:0/250954368 shutdown_connections 2024-03-30T05:10:43.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.037+0000 7fae47000700 1 -- 172.21.15.123:0/250954368 wait complete. 2024-03-30T05:10:43.054 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T04:21:34.842970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-03-30T04:24:29.050323+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-03-30T04:24:29.704111+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-30T05:10:43.054 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-30T05:10:43.173 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:43 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/3002824319' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-30T05:10:43.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.248+0000 7f0d7d000700 1 -- 172.21.15.123:0/3675155974 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d780fe4c0 msgr2=0x7f0d78106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:43.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.248+0000 7f0d7d000700 1 --2- 172.21.15.123:0/3675155974 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d780fe4c0 0x7f0d78106b00 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f0d64004940 tx=0x7f0d64009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:43.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.248+0000 7f0d7d000700 1 -- 172.21.15.123:0/3675155974 shutdown_connections 2024-03-30T05:10:43.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.248+0000 7f0d7d000700 1 --2- 172.21.15.123:0/3675155974 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d780fe4c0 0x7f0d78106b00 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:43.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.248+0000 7f0d7d000700 1 --2- 172.21.15.123:0/3675155974 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0d780fdb20 0x7f0d780fdef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:43.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.248+0000 7f0d7d000700 1 -- 172.21.15.123:0/3675155974 >> 172.21.15.123:0/3675155974 conn(0x7f0d780f7590 msgr2=0x7f0d780f99a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:43.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.248+0000 7f0d7d000700 1 -- 172.21.15.123:0/3675155974 shutdown_connections 2024-03-30T05:10:43.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.248+0000 7f0d7d000700 1 -- 172.21.15.123:0/3675155974 wait complete. 2024-03-30T05:10:43.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.249+0000 7f0d7d000700 1 Processor -- start 2024-03-30T05:10:43.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.249+0000 7f0d7d000700 1 -- start start 2024-03-30T05:10:43.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.249+0000 7f0d7d000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d780fdb20 0x7f0d78113580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:43.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.250+0000 7f0d7d000700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0d780fe4c0 0x7f0d78113ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:43.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.250+0000 7f0d7d000700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d781141a0 con 0x7f0d780fdb20 2024-03-30T05:10:43.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.250+0000 7f0d7d000700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d7810d710 con 0x7f0d780fe4c0 2024-03-30T05:10:43.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.250+0000 7f0d77400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d780fdb20 0x7f0d78113580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:43.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.250+0000 7f0d77400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d780fdb20 0x7f0d78113580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40054/0 (socket says 172.21.15.123:40054) 2024-03-30T05:10:43.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.250+0000 7f0d77400700 1 -- 172.21.15.123:0/1206958624 learned_addr learned my addr 172.21.15.123:0/1206958624 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:10:43.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.250+0000 7f0d77400700 1 -- 172.21.15.123:0/1206958624 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0d780fe4c0 msgr2=0x7f0d78113ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:43.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.250+0000 7f0d77400700 1 --2- 172.21.15.123:0/1206958624 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0d780fe4c0 0x7f0d78113ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:43.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.250+0000 7f0d77400700 1 -- 172.21.15.123:0/1206958624 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d64005040 con 0x7f0d780fdb20 2024-03-30T05:10:43.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.250+0000 7f0d77400700 1 --2- 172.21.15.123:0/1206958624 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d780fdb20 0x7f0d78113580 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f0d6800b790 tx=0x7f0d68009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:43.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.250+0000 7f0d6f400700 1 -- 172.21.15.123:0/1206958624 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d6800f770 con 0x7f0d780fdb20 2024-03-30T05:10:43.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.251+0000 7f0d6f400700 1 -- 172.21.15.123:0/1206958624 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0d6800fd80 con 0x7f0d780fdb20 2024-03-30T05:10:43.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.251+0000 7f0d7d000700 1 -- 172.21.15.123:0/1206958624 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d7810d9f0 con 0x7f0d780fdb20 2024-03-30T05:10:43.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.251+0000 7f0d6f400700 1 -- 172.21.15.123:0/1206958624 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d6800d310 con 0x7f0d780fdb20 2024-03-30T05:10:43.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.251+0000 7f0d7d000700 1 -- 172.21.15.123:0/1206958624 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d7810df30 con 0x7f0d780fdb20 2024-03-30T05:10:43.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.252+0000 7f0d7d000700 1 -- 172.21.15.123:0/1206958624 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d780667e0 con 0x7f0d780fdb20 2024-03-30T05:10:43.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.253+0000 7f0d6f400700 1 -- 172.21.15.123:0/1206958624 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f0d68010420 con 0x7f0d780fdb20 2024-03-30T05:10:43.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.253+0000 7f0d6f400700 1 --2- 172.21.15.123:0/1206958624 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0d5c06cc20 0x7f0d5c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:43.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.253+0000 7f0d6f400700 1 -- 172.21.15.123:0/1206958624 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f0d6808bfe0 con 0x7f0d780fdb20 2024-03-30T05:10:43.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.254+0000 7f0d76a00700 1 --2- 172.21.15.123:0/1206958624 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0d5c06cc20 0x7f0d5c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:43.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.254+0000 7f0d76a00700 1 --2- 172.21.15.123:0/1206958624 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0d5c06cc20 0x7f0d5c06f0d0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f0d64004f90 tx=0x7f0d64009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:43.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.258+0000 7f0d6f400700 1 -- 172.21.15.123:0/1206958624 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0d680bf9f0 con 0x7f0d780fdb20 2024-03-30T05:10:43.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.463+0000 7f0d7d000700 1 -- 172.21.15.123:0/1206958624 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0d7810ebd0 con 0x7f0d780fdb20 2024-03-30T05:10:43.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.463+0000 7f0d6f400700 1 -- 172.21.15.123:0/1206958624 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v86) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0d68059ab0 con 0x7f0d780fdb20 2024-03-30T05:10:43.462 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 1 2024-03-30T05:10:43.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.466+0000 7f0d7d000700 1 -- 172.21.15.123:0/1206958624 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0d5c06cc20 msgr2=0x7f0d5c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:43.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.466+0000 7f0d7d000700 1 --2- 172.21.15.123:0/1206958624 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0d5c06cc20 0x7f0d5c06f0d0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f0d64004f90 tx=0x7f0d64009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:43.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.466+0000 7f0d7d000700 1 -- 172.21.15.123:0/1206958624 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d780fdb20 msgr2=0x7f0d78113580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:43.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.466+0000 7f0d7d000700 1 --2- 172.21.15.123:0/1206958624 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d780fdb20 0x7f0d78113580 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f0d6800b790 tx=0x7f0d68009250 comp rx=0 tx=0).stop 2024-03-30T05:10:43.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.466+0000 7f0d7d000700 1 -- 172.21.15.123:0/1206958624 shutdown_connections 2024-03-30T05:10:43.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.466+0000 7f0d7d000700 1 --2- 172.21.15.123:0/1206958624 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0d5c06cc20 0x7f0d5c06f0d0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:43.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.466+0000 7f0d7d000700 1 --2- 172.21.15.123:0/1206958624 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d780fdb20 0x7f0d78113580 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:43.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.466+0000 7f0d7d000700 1 --2- 172.21.15.123:0/1206958624 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0d780fe4c0 0x7f0d78113ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:43.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.466+0000 7f0d7d000700 1 -- 172.21.15.123:0/1206958624 >> 172.21.15.123:0/1206958624 conn(0x7f0d780f7590 msgr2=0x7f0d780f8ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:43.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.466+0000 7f0d7d000700 1 -- 172.21.15.123:0/1206958624 shutdown_connections 2024-03-30T05:10:43.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.466+0000 7f0d7d000700 1 -- 172.21.15.123:0/1206958624 wait complete. 2024-03-30T05:10:43.481 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-03-30T05:10:43.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:43 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/3002824319' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-30T05:10:43.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:43 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/250954368' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T05:10:43.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.677+0000 7fb70da00700 1 -- 172.21.15.123:0/500266194 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb708102cd0 msgr2=0x7fb7081030a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:43.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.677+0000 7fb70da00700 1 --2- 172.21.15.123:0/500266194 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb708102cd0 0x7fb7081030a0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fb6f8004940 tx=0x7fb6f8009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:43.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.677+0000 7fb70da00700 1 -- 172.21.15.123:0/500266194 shutdown_connections 2024-03-30T05:10:43.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.677+0000 7fb70da00700 1 --2- 172.21.15.123:0/500266194 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb708103670 0x7fb7080fcce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:43.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.677+0000 7fb70da00700 1 --2- 172.21.15.123:0/500266194 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb708102cd0 0x7fb7081030a0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:43.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.677+0000 7fb70da00700 1 -- 172.21.15.123:0/500266194 >> 172.21.15.123:0/500266194 conn(0x7fb7080f8840 msgr2=0x7fb7080fac50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:43.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.677+0000 7fb70da00700 1 -- 172.21.15.123:0/500266194 shutdown_connections 2024-03-30T05:10:43.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.677+0000 7fb70da00700 1 -- 172.21.15.123:0/500266194 wait complete. 2024-03-30T05:10:43.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.678+0000 7fb70da00700 1 Processor -- start 2024-03-30T05:10:43.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.678+0000 7fb70da00700 1 -- start start 2024-03-30T05:10:43.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.679+0000 7fb70da00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb708102cd0 0x7fb708113760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:43.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.679+0000 7fb70da00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb708103670 0x7fb708113ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:43.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.679+0000 7fb70da00700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7081142f0 con 0x7fb708103670 2024-03-30T05:10:43.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.679+0000 7fb70da00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb70810d8f0 con 0x7fb708102cd0 2024-03-30T05:10:43.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.679+0000 7fb706000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb708103670 0x7fb708113ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:43.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.679+0000 7fb706000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb708103670 0x7fb708113ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40086/0 (socket says 172.21.15.123:40086) 2024-03-30T05:10:43.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.679+0000 7fb706000700 1 -- 172.21.15.123:0/1885026379 learned_addr learned my addr 172.21.15.123:0/1885026379 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:10:43.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.679+0000 7fb706a00700 1 --2- 172.21.15.123:0/1885026379 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb708102cd0 0x7fb708113760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:43.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.679+0000 7fb706000700 1 -- 172.21.15.123:0/1885026379 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb708102cd0 msgr2=0x7fb708113760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:43.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.679+0000 7fb706000700 1 --2- 172.21.15.123:0/1885026379 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb708102cd0 0x7fb708113760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:43.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.679+0000 7fb706000700 1 -- 172.21.15.123:0/1885026379 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6f8005040 con 0x7fb708103670 2024-03-30T05:10:43.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.679+0000 7fb706a00700 1 --2- 172.21.15.123:0/1885026379 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb708102cd0 0x7fb708113760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T05:10:43.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.680+0000 7fb706000700 1 --2- 172.21.15.123:0/1885026379 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb708103670 0x7fb708113ca0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fb6f4002a50 tx=0x7fb6f4004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:43.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.680+0000 7fb6fea00700 1 -- 172.21.15.123:0/1885026379 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb6f4007cc0 con 0x7fb708103670 2024-03-30T05:10:43.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.680+0000 7fb6fea00700 1 -- 172.21.15.123:0/1885026379 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb6f400f040 con 0x7fb708103670 2024-03-30T05:10:43.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.680+0000 7fb70da00700 1 -- 172.21.15.123:0/1885026379 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb70810dbd0 con 0x7fb708103670 2024-03-30T05:10:43.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.680+0000 7fb6fea00700 1 -- 172.21.15.123:0/1885026379 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb6f400d9a0 con 0x7fb708103670 2024-03-30T05:10:43.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.680+0000 7fb70da00700 1 -- 172.21.15.123:0/1885026379 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb70810e190 con 0x7fb708103670 2024-03-30T05:10:43.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.681+0000 7fb70da00700 1 -- 172.21.15.123:0/1885026379 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7080fe420 con 0x7fb708103670 2024-03-30T05:10:43.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.682+0000 7fb6fea00700 1 -- 172.21.15.123:0/1885026379 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fb6f400dc60 con 0x7fb708103670 2024-03-30T05:10:43.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.683+0000 7fb6fea00700 1 --2- 172.21.15.123:0/1885026379 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb6ec06cc20 0x7fb6ec06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:43.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.683+0000 7fb6fea00700 1 -- 172.21.15.123:0/1885026379 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb6f408be70 con 0x7fb708103670 2024-03-30T05:10:43.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.683+0000 7fb706a00700 1 --2- 172.21.15.123:0/1885026379 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb6ec06cc20 0x7fb6ec06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:43.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.684+0000 7fb706a00700 1 --2- 172.21.15.123:0/1885026379 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb6ec06cc20 0x7fb6ec06f0d0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fb6f8000c00 tx=0x7fb6f8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:43.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.688+0000 7fb6fea00700 1 -- 172.21.15.123:0/1885026379 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb6f40598c0 con 0x7fb708103670 2024-03-30T05:10:43.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.893+0000 7fb70da00700 1 -- 172.21.15.123:0/1885026379 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7fb708109cc0 con 0x7fb708103670 2024-03-30T05:10:43.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.894+0000 7fb6fea00700 1 -- 172.21.15.123:0/1885026379 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v86) v1 ==== 107+0+11 (secure 0 0 0) 0x7fb6f4059450 con 0x7fb708103670 2024-03-30T05:10:43.892 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 32 2024-03-30T05:10:43.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.896+0000 7fb70da00700 1 -- 172.21.15.123:0/1885026379 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb6ec06cc20 msgr2=0x7fb6ec06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:43.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.896+0000 7fb70da00700 1 --2- 172.21.15.123:0/1885026379 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb6ec06cc20 0x7fb6ec06f0d0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fb6f8000c00 tx=0x7fb6f8009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:43.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.896+0000 7fb70da00700 1 -- 172.21.15.123:0/1885026379 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb708103670 msgr2=0x7fb708113ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:43.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.896+0000 7fb70da00700 1 --2- 172.21.15.123:0/1885026379 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb708103670 0x7fb708113ca0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fb6f4002a50 tx=0x7fb6f4004600 comp rx=0 tx=0).stop 2024-03-30T05:10:43.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.896+0000 7fb70da00700 1 -- 172.21.15.123:0/1885026379 shutdown_connections 2024-03-30T05:10:43.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.896+0000 7fb70da00700 1 --2- 172.21.15.123:0/1885026379 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7fb6ec06cc20 0x7fb6ec06f0d0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:43.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.896+0000 7fb70da00700 1 --2- 172.21.15.123:0/1885026379 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb708102cd0 0x7fb708113760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:43.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.896+0000 7fb70da00700 1 --2- 172.21.15.123:0/1885026379 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb708103670 0x7fb708113ca0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:43.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.896+0000 7fb70da00700 1 -- 172.21.15.123:0/1885026379 >> 172.21.15.123:0/1885026379 conn(0x7fb7080f8840 msgr2=0x7fb7080f98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:43.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.896+0000 7fb70da00700 1 -- 172.21.15.123:0/1885026379 shutdown_connections 2024-03-30T05:10:43.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:43.896+0000 7fb70da00700 1 -- 172.21.15.123:0/1885026379 wait complete. 2024-03-30T05:10:43.910 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:43 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/250954368' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T05:10:43.912 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-03-30T05:10:44.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.111+0000 7f81e8400700 1 -- 172.21.15.123:0/4208423960 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81e0070c70 msgr2=0x7f81e0108ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:44.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.111+0000 7f81e8400700 1 --2- 172.21.15.123:0/4208423960 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81e0070c70 0x7f81e0108ff0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f81cc004970 tx=0x7f81cc009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:44.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.111+0000 7f81e8400700 1 -- 172.21.15.123:0/4208423960 shutdown_connections 2024-03-30T05:10:44.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.111+0000 7f81e8400700 1 --2- 172.21.15.123:0/4208423960 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81e0070c70 0x7f81e0108ff0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:44.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.111+0000 7f81e8400700 1 --2- 172.21.15.123:0/4208423960 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f81e0070360 0x7f81e0070730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:44.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.111+0000 7f81e8400700 1 -- 172.21.15.123:0/4208423960 >> 172.21.15.123:0/4208423960 conn(0x7f81e00f8830 msgr2=0x7f81e00fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:44.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.111+0000 7f81e8400700 1 -- 172.21.15.123:0/4208423960 shutdown_connections 2024-03-30T05:10:44.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.111+0000 7f81e8400700 1 -- 172.21.15.123:0/4208423960 wait complete. 2024-03-30T05:10:44.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.112+0000 7f81e8400700 1 Processor -- start 2024-03-30T05:10:44.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.112+0000 7f81e8400700 1 -- start start 2024-03-30T05:10:44.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.113+0000 7f81e8400700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81e0070360 0x7f81e0115a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:44.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.113+0000 7f81e8400700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f81e0070c70 0x7f81e0115f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:44.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.113+0000 7f81e8400700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81e0116590 con 0x7f81e0070360 2024-03-30T05:10:44.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.113+0000 7f81e8400700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81e010fb90 con 0x7f81e0070c70 2024-03-30T05:10:44.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.113+0000 7f81e5a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81e0070360 0x7f81e0115a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:44.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.113+0000 7f81e5a00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81e0070360 0x7f81e0115a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48984/0 (socket says 172.21.15.123:48984) 2024-03-30T05:10:44.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.113+0000 7f81e5a00700 1 -- 172.21.15.123:0/4165818427 learned_addr learned my addr 172.21.15.123:0/4165818427 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:10:44.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.113+0000 7f81e5000700 1 --2- 172.21.15.123:0/4165818427 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f81e0070c70 0x7f81e0115f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:44.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.113+0000 7f81e5a00700 1 -- 172.21.15.123:0/4165818427 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f81e0070c70 msgr2=0x7f81e0115f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:44.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.113+0000 7f81e5a00700 1 --2- 172.21.15.123:0/4165818427 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f81e0070c70 0x7f81e0115f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:44.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.113+0000 7f81e5a00700 1 -- 172.21.15.123:0/4165818427 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81cc005040 con 0x7f81e0070360 2024-03-30T05:10:44.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.114+0000 7f81e5a00700 1 --2- 172.21.15.123:0/4165818427 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81e0070360 0x7f81e0115a00 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f81d4002a50 tx=0x7f81d40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:44.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.114+0000 7f81dd600700 1 -- 172.21.15.123:0/4165818427 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f81d4007cc0 con 0x7f81e0070360 2024-03-30T05:10:44.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.114+0000 7f81dd600700 1 -- 172.21.15.123:0/4165818427 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f81d400f040 con 0x7f81e0070360 2024-03-30T05:10:44.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.114+0000 7f81e8400700 1 -- 172.21.15.123:0/4165818427 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81e010fe70 con 0x7f81e0070360 2024-03-30T05:10:44.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.114+0000 7f81dd600700 1 -- 172.21.15.123:0/4165818427 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f81d400d9c0 con 0x7f81e0070360 2024-03-30T05:10:44.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.114+0000 7f81e8400700 1 -- 172.21.15.123:0/4165818427 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81e0110380 con 0x7f81e0070360 2024-03-30T05:10:44.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.115+0000 7f81e8400700 1 -- 172.21.15.123:0/4165818427 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81e0071aa0 con 0x7f81e0070360 2024-03-30T05:10:44.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.116+0000 7f81dd600700 1 -- 172.21.15.123:0/4165818427 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f81d400db20 con 0x7f81e0070360 2024-03-30T05:10:44.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.117+0000 7f81dd600700 1 --2- 172.21.15.123:0/4165818427 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f81c806cc20 0x7f81c806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:44.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.117+0000 7f81dd600700 1 -- 172.21.15.123:0/4165818427 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f81d408bd20 con 0x7f81e0070360 2024-03-30T05:10:44.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.117+0000 7f81e5000700 1 --2- 172.21.15.123:0/4165818427 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f81c806cc20 0x7f81c806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:44.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.118+0000 7f81e5000700 1 --2- 172.21.15.123:0/4165818427 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f81c806cc20 0x7f81c806f0d0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f81cc00b370 tx=0x7f81cc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:44.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.122+0000 7f81dd600700 1 -- 172.21.15.123:0/4165818427 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f81d4059770 con 0x7f81e0070360 2024-03-30T05:10:44.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.339+0000 7f81e8400700 1 -- 172.21.15.123:0/4165818427 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f81e0110fe0 con 0x7f81e0070360 2024-03-30T05:10:44.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.339+0000 7f81dd600700 1 -- 172.21.15.123:0/4165818427 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v86) v1 ==== 107+0+11 (secure 0 0 0) 0x7f81d4059300 con 0x7f81e0070360 2024-03-30T05:10:44.337 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 32 2024-03-30T05:10:44.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.342+0000 7f81e8400700 1 -- 172.21.15.123:0/4165818427 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f81c806cc20 msgr2=0x7f81c806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:44.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.342+0000 7f81e8400700 1 --2- 172.21.15.123:0/4165818427 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f81c806cc20 0x7f81c806f0d0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f81cc00b370 tx=0x7f81cc009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:44.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.342+0000 7f81e8400700 1 -- 172.21.15.123:0/4165818427 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81e0070360 msgr2=0x7f81e0115a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:44.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.342+0000 7f81e8400700 1 --2- 172.21.15.123:0/4165818427 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81e0070360 0x7f81e0115a00 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f81d4002a50 tx=0x7f81d40046b0 comp rx=0 tx=0).stop 2024-03-30T05:10:44.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.342+0000 7f81e8400700 1 -- 172.21.15.123:0/4165818427 shutdown_connections 2024-03-30T05:10:44.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.342+0000 7f81e8400700 1 --2- 172.21.15.123:0/4165818427 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f81c806cc20 0x7f81c806f0d0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:44.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.342+0000 7f81e8400700 1 --2- 172.21.15.123:0/4165818427 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81e0070360 0x7f81e0115a00 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:44.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.342+0000 7f81e8400700 1 --2- 172.21.15.123:0/4165818427 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f81e0070c70 0x7f81e0115f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:44.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.342+0000 7f81e8400700 1 -- 172.21.15.123:0/4165818427 >> 172.21.15.123:0/4165818427 conn(0x7f81e00f8830 msgr2=0x7f81e00f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:44.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.342+0000 7f81e8400700 1 -- 172.21.15.123:0/4165818427 shutdown_connections 2024-03-30T05:10:44.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.342+0000 7f81e8400700 1 -- 172.21.15.123:0/4165818427 wait complete. 2024-03-30T05:10:44.358 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-30T05:10:44.419 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:44 smithi123 ceph-mon[139369]: pgmap v1188: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:10:44.419 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:44 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/1206958624' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T05:10:44.420 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:44 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/1885026379' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-30T05:10:44.420 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T05:10:44.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.560+0000 7f0f19600700 1 -- 172.21.15.123:0/1038898599 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f140fdf80 msgr2=0x7f0f140fe350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:44.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.560+0000 7f0f19600700 1 --2- 172.21.15.123:0/1038898599 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f140fdf80 0x7f0f140fe350 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f0f04004970 tx=0x7f0f04009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:44.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.560+0000 7f0f19600700 1 -- 172.21.15.123:0/1038898599 shutdown_connections 2024-03-30T05:10:44.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.560+0000 7f0f19600700 1 --2- 172.21.15.123:0/1038898599 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0f140fe890 0x7f0f14106da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:44.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.560+0000 7f0f19600700 1 --2- 172.21.15.123:0/1038898599 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f140fdf80 0x7f0f140fe350 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:44.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.560+0000 7f0f19600700 1 -- 172.21.15.123:0/1038898599 >> 172.21.15.123:0/1038898599 conn(0x7f0f140f7730 msgr2=0x7f0f140f9b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:44.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.560+0000 7f0f19600700 1 -- 172.21.15.123:0/1038898599 shutdown_connections 2024-03-30T05:10:44.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.560+0000 7f0f19600700 1 -- 172.21.15.123:0/1038898599 wait complete. 2024-03-30T05:10:44.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.560+0000 7f0f19600700 1 Processor -- start 2024-03-30T05:10:44.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.561+0000 7f0f19600700 1 -- start start 2024-03-30T05:10:44.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.561+0000 7f0f19600700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0f140fdf80 0x7f0f14113770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:44.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.561+0000 7f0f19600700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f140fe890 0x7f0f14113cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:44.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.561+0000 7f0f19600700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f14114300 con 0x7f0f140fe890 2024-03-30T05:10:44.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.561+0000 7f0f19600700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1410d900 con 0x7f0f140fdf80 2024-03-30T05:10:44.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.561+0000 7f0f12000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f140fe890 0x7f0f14113cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:44.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.561+0000 7f0f12000700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f140fe890 0x7f0f14113cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49008/0 (socket says 172.21.15.123:49008) 2024-03-30T05:10:44.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.561+0000 7f0f12000700 1 -- 172.21.15.123:0/100693606 learned_addr learned my addr 172.21.15.123:0/100693606 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-03-30T05:10:44.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.561+0000 7f0f12a00700 1 --2- 172.21.15.123:0/100693606 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0f140fdf80 0x7f0f14113770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:44.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.562+0000 7f0f12000700 1 -- 172.21.15.123:0/100693606 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0f140fdf80 msgr2=0x7f0f14113770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:44.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.562+0000 7f0f12000700 1 --2- 172.21.15.123:0/100693606 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0f140fdf80 0x7f0f14113770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:44.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.562+0000 7f0f12000700 1 -- 172.21.15.123:0/100693606 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f04005040 con 0x7f0f140fe890 2024-03-30T05:10:44.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.562+0000 7f0f12000700 1 --2- 172.21.15.123:0/100693606 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f140fe890 0x7f0f14113cb0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f0f00002a50 tx=0x7f0f000046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:44.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.562+0000 7f0f0aa00700 1 -- 172.21.15.123:0/100693606 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f00007cc0 con 0x7f0f140fe890 2024-03-30T05:10:44.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.562+0000 7f0f0aa00700 1 -- 172.21.15.123:0/100693606 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0f0000f040 con 0x7f0f140fe890 2024-03-30T05:10:44.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.562+0000 7f0f19600700 1 -- 172.21.15.123:0/100693606 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f1410dbe0 con 0x7f0f140fe890 2024-03-30T05:10:44.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.562+0000 7f0f0aa00700 1 -- 172.21.15.123:0/100693606 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f0000d9c0 con 0x7f0f140fe890 2024-03-30T05:10:44.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.562+0000 7f0f19600700 1 -- 172.21.15.123:0/100693606 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f1410e120 con 0x7f0f140fe890 2024-03-30T05:10:44.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.563+0000 7f0f19600700 1 -- 172.21.15.123:0/100693606 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f14066950 con 0x7f0f140fe890 2024-03-30T05:10:44.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.564+0000 7f0f0aa00700 1 -- 172.21.15.123:0/100693606 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 38) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f0f0000db20 con 0x7f0f140fe890 2024-03-30T05:10:44.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.566+0000 7f0f0aa00700 1 --2- 172.21.15.123:0/100693606 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0ef806cc20 0x7f0ef806f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T05:10:44.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.566+0000 7f0f0aa00700 1 -- 172.21.15.123:0/100693606 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f0f0008bce0 con 0x7f0f140fe890 2024-03-30T05:10:44.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.566+0000 7f0f12a00700 1 --2- 172.21.15.123:0/100693606 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0ef806cc20 0x7f0ef806f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T05:10:44.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.567+0000 7f0f12a00700 1 --2- 172.21.15.123:0/100693606 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0ef806cc20 0x7f0ef806f0d0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f0f0400b370 tx=0x7f0f04009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T05:10:44.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.570+0000 7f0f0aa00700 1 -- 172.21.15.123:0/100693606 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0f00059770 con 0x7f0f140fe890 2024-03-30T05:10:44.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:44 smithi135 ceph-mon[106011]: pgmap v1188: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:10:44.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:44 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/1206958624' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T05:10:44.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:44 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/1885026379' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-30T05:10:44.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:44 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T05:10:44.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.800+0000 7f0f19600700 1 -- 172.21.15.123:0/100693606 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f0f1410edc0 con 0x7f0f140fe890 2024-03-30T05:10:44.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.800+0000 7f0f0aa00700 1 -- 172.21.15.123:0/100693606 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 39 entries v86) v1 ==== 81+0+2345 (secure 0 0 0) 0x7f0f00059300 con 0x7f0f140fe890 2024-03-30T05:10:44.798 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:6829/3037849622 2024-03-31T04:37:19.516180+0000 2024-03-30T05:10:44.799 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:6828/3037849622 2024-03-31T04:37:19.516180+0000 2024-03-30T05:10:44.799 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:0/1275706223 2024-03-31T04:18:56.235750+0000 2024-03-30T05:10:44.799 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:6827/2393193575 2024-03-31T04:24:47.075751+0000 2024-03-30T05:10:44.799 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.135:0/1604630130 2024-03-31T04:28:49.824398+0000 2024-03-30T05:10:44.799 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:0/3228440712 2024-03-31T04:32:29.347901+0000 2024-03-30T05:10:44.799 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:0/3037148222 2024-03-31T04:17:41.459573+0000 2024-03-30T05:10:44.799 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:0/2358069559 2024-03-31T04:17:41.459573+0000 2024-03-30T05:10:44.799 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.135:6828/3601553667 2024-03-31T04:28:49.824398+0000 2024-03-30T05:10:44.799 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:0/4218181674 2024-03-31T04:26:59.096603+0000 2024-03-30T05:10:44.799 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:0/2447404972 2024-03-31T04:26:59.096603+0000 2024-03-30T05:10:44.799 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:0/3593359364 2024-03-31T04:18:56.235750+0000 2024-03-30T05:10:44.799 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.135:6825/4050541062 2024-03-31T04:37:19.516195+0000 2024-03-30T05:10:44.799 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.135:0/1748591175 2024-03-31T04:28:49.824398+0000 2024-03-30T05:10:44.800 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:0/518566709 2024-03-31T04:18:56.235750+0000 2024-03-30T05:10:44.800 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.135:0/952436162 2024-03-31T04:28:49.824398+0000 2024-03-30T05:10:44.800 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.135:6829/3601553667 2024-03-31T04:28:49.824398+0000 2024-03-30T05:10:44.800 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:6800/1291592055 2024-03-31T04:17:05.107043+0000 2024-03-30T05:10:44.800 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.135:0/3366037413 2024-03-31T04:28:49.824398+0000 2024-03-30T05:10:44.800 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:6826/2393193575 2024-03-31T04:24:47.075751+0000 2024-03-30T05:10:44.800 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:6801/1776532690 2024-03-31T04:17:41.459573+0000 2024-03-30T05:10:44.800 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:0/1064275350 2024-03-31T04:26:59.096603+0000 2024-03-30T05:10:44.800 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:0/72410915 2024-03-31T04:26:59.096603+0000 2024-03-30T05:10:44.800 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:6800/2125774379 2024-03-31T04:32:29.347901+0000 2024-03-30T05:10:44.800 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:6801/1291592055 2024-03-31T04:17:05.107043+0000 2024-03-30T05:10:44.800 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:6800/1564935593 2024-03-31T04:26:59.096603+0000 2024-03-30T05:10:44.801 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:0/2756071515 2024-03-31T04:17:05.107043+0000 2024-03-30T05:10:44.801 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:0/133647565 2024-03-31T04:17:41.459573+0000 2024-03-30T05:10:44.801 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:6801/2125774379 2024-03-31T04:32:29.347901+0000 2024-03-30T05:10:44.801 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:6801/1564935593 2024-03-31T04:26:59.096603+0000 2024-03-30T05:10:44.801 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:6800/1776532690 2024-03-31T04:17:41.459573+0000 2024-03-30T05:10:44.801 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:6800/70490328 2024-03-31T04:18:56.235750+0000 2024-03-30T05:10:44.801 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:0/2366611723 2024-03-31T04:17:05.107043+0000 2024-03-30T05:10:44.801 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:6801/70490328 2024-03-31T04:18:56.235750+0000 2024-03-30T05:10:44.801 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.135:6824/4050541062 2024-03-31T04:37:19.516195+0000 2024-03-30T05:10:44.801 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:0/250978570 2024-03-31T04:17:05.107043+0000 2024-03-30T05:10:44.801 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:0/2743438158 2024-03-31T04:32:29.347901+0000 2024-03-30T05:10:44.801 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:0/1159868778 2024-03-31T04:32:29.347901+0000 2024-03-30T05:10:44.802 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:0/2182269825 2024-03-31T04:32:29.347901+0000 2024-03-30T05:10:44.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.803+0000 7f0f19600700 1 -- 172.21.15.123:0/100693606 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0ef806cc20 msgr2=0x7f0ef806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:44.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.803+0000 7f0f19600700 1 --2- 172.21.15.123:0/100693606 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0ef806cc20 0x7f0ef806f0d0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f0f0400b370 tx=0x7f0f04009e30 comp rx=0 tx=0).stop 2024-03-30T05:10:44.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.803+0000 7f0f19600700 1 -- 172.21.15.123:0/100693606 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f140fe890 msgr2=0x7f0f14113cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T05:10:44.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.803+0000 7f0f19600700 1 --2- 172.21.15.123:0/100693606 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f140fe890 0x7f0f14113cb0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f0f00002a50 tx=0x7f0f000046b0 comp rx=0 tx=0).stop 2024-03-30T05:10:44.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.803+0000 7f0f19600700 1 -- 172.21.15.123:0/100693606 shutdown_connections 2024-03-30T05:10:44.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.803+0000 7f0f19600700 1 --2- 172.21.15.123:0/100693606 >> [v2:172.21.15.123:6800/4244519131,v1:172.21.15.123:6801/4244519131] conn(0x7f0ef806cc20 0x7f0ef806f0d0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:44.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.803+0000 7f0f19600700 1 --2- 172.21.15.123:0/100693606 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0f140fdf80 0x7f0f14113770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:44.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.803+0000 7f0f19600700 1 --2- 172.21.15.123:0/100693606 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f140fe890 0x7f0f14113cb0 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T05:10:44.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.803+0000 7f0f19600700 1 -- 172.21.15.123:0/100693606 >> 172.21.15.123:0/100693606 conn(0x7f0f140f7730 msgr2=0x7f0f140f8bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T05:10:44.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.803+0000 7f0f19600700 1 -- 172.21.15.123:0/100693606 shutdown_connections 2024-03-30T05:10:44.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-03-30T05:10:44.803+0000 7f0f19600700 1 -- 172.21.15.123:0/100693606 wait complete. 2024-03-30T05:10:44.803 INFO:teuthology.orchestra.run.smithi123.stderr:listed 39 entries 2024-03-30T05:10:44.822 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi123.front.sepia.ceph.com... 2024-03-30T05:10:44.822 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T05:10:44.822 DEBUG:teuthology.orchestra.run.smithi123:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-30T05:10:44.892 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T05:10:45.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:45 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/4165818427' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-30T05:10:45.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:45 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:10:45.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:45 smithi123 ceph-mon[139369]: from='client.? 172.21.15.123:0/100693606' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-30T05:10:45.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:45 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T05:10:45.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:45 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T05:10:45.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:45 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:10:45.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:45 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/4165818427' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-30T05:10:45.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:45 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:10:45.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:45 smithi135 ceph-mon[106011]: from='client.? 172.21.15.123:0/100693606' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-30T05:10:45.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:45 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T05:10:45.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:45 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T05:10:45.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:45 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:10:46.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:46 smithi123 ceph-mon[139369]: pgmap v1189: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:10:46.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:46 smithi135 ceph-mon[106011]: pgmap v1189: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:10:48.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:48 smithi123 ceph-mon[139369]: pgmap v1190: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:10:48.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:48 smithi135 ceph-mon[106011]: pgmap v1190: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:10:50.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:50 smithi123 ceph-mon[139369]: pgmap v1191: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:10:50.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:50 smithi135 ceph-mon[106011]: pgmap v1191: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:10:52.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:52 smithi123 ceph-mon[139369]: pgmap v1192: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:10:52.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:52 smithi135 ceph-mon[106011]: pgmap v1192: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:10:54.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:54 smithi123 ceph-mon[139369]: pgmap v1193: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:10:54.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:54 smithi135 ceph-mon[106011]: pgmap v1193: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:10:56.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:56 smithi123 ceph-mon[139369]: pgmap v1194: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:10:56.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:56 smithi135 ceph-mon[106011]: pgmap v1194: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:10:58.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:10:58 smithi123 ceph-mon[139369]: pgmap v1195: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:10:58.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:10:58 smithi135 ceph-mon[106011]: pgmap v1195: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:11:00.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:11:00 smithi123 ceph-mon[139369]: pgmap v1196: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:11:00.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:11:00 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:11:00.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:11:00 smithi135 ceph-mon[106011]: pgmap v1196: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:11:00.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:11:00 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:11:02.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:11:02 smithi123 ceph-mon[139369]: pgmap v1197: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:11:02.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:11:02 smithi135 ceph-mon[106011]: pgmap v1197: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:11:04.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:11:04 smithi123 ceph-mon[139369]: pgmap v1198: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:11:04.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:11:04 smithi135 ceph-mon[106011]: pgmap v1198: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:11:06.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:11:06 smithi135 ceph-mon[106011]: pgmap v1199: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:11:06.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:11:06 smithi123 ceph-mon[139369]: pgmap v1199: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:11:08.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:11:08 smithi123 ceph-mon[139369]: pgmap v1200: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:11:08.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:11:08 smithi135 ceph-mon[106011]: pgmap v1200: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:11:10.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:11:10 smithi135 ceph-mon[106011]: pgmap v1201: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:11:10.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:11:10 smithi123 ceph-mon[139369]: pgmap v1201: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:11:12.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:11:12 smithi135 ceph-mon[106011]: pgmap v1202: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:11:12.715 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:11:12 smithi123 ceph-mon[139369]: pgmap v1202: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:11:14.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:11:14 smithi135 ceph-mon[106011]: pgmap v1203: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:11:14.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:11:14 smithi123 ceph-mon[139369]: pgmap v1203: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:11:15.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:11:15 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:11:15.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:11:15 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:11:16.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:11:16 smithi135 ceph-mon[106011]: pgmap v1204: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:11:16.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:11:16 smithi123 ceph-mon[139369]: pgmap v1204: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:11:18.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:11:18 smithi123 ceph-mon[139369]: pgmap v1205: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:11:18.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:11:18 smithi135 ceph-mon[106011]: pgmap v1205: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:11:20.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:11:20 smithi135 ceph-mon[106011]: pgmap v1206: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:11:20.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:11:20 smithi123 ceph-mon[139369]: pgmap v1206: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:11:21.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:11:21 smithi135 ceph-mon[106011]: pgmap v1207: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:11:21.715 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:11:21 smithi123 ceph-mon[139369]: pgmap v1207: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:11:23.850 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:11:23 smithi135 ceph-mon[106011]: pgmap v1208: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:11:23.965 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:11:23 smithi123 ceph-mon[139369]: pgmap v1208: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:11:26.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:11:25 smithi123 ceph-mon[139369]: pgmap v1209: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:11:26.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:11:25 smithi135 ceph-mon[106011]: pgmap v1209: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:11:28.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:11:27 smithi123 ceph-mon[139369]: pgmap v1210: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:11:28.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:11:27 smithi135 ceph-mon[106011]: pgmap v1210: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:11:30.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:11:29 smithi123 ceph-mon[139369]: pgmap v1211: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:11:30.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:11:29 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:11:30.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:11:29 smithi135 ceph-mon[106011]: pgmap v1211: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:11:30.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:11:29 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:11:32.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:11:31 smithi123 ceph-mon[139369]: pgmap v1212: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:11:32.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:11:31 smithi135 ceph-mon[106011]: pgmap v1212: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:11:34.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:11:33 smithi123 ceph-mon[139369]: pgmap v1213: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:11:34.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:11:33 smithi135 ceph-mon[106011]: pgmap v1213: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:11:36.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:11:35 smithi123 ceph-mon[139369]: pgmap v1214: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:11:36.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:11:35 smithi135 ceph-mon[106011]: pgmap v1214: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:11:38.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:11:37 smithi123 ceph-mon[139369]: pgmap v1215: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:11:38.351 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:11:37 smithi135 ceph-mon[106011]: pgmap v1215: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:11:40.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:11:39 smithi123 ceph-mon[139369]: pgmap v1216: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:11:40.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:11:39 smithi135 ceph-mon[106011]: pgmap v1216: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:11:42.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:11:41 smithi123 ceph-mon[139369]: pgmap v1217: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:11:42.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:11:41 smithi135 ceph-mon[106011]: pgmap v1217: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:11:44.216 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:11:43 smithi123 ceph-mon[139369]: pgmap v1218: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:11:44.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:11:43 smithi135 ceph-mon[106011]: pgmap v1218: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:11:45.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:11:44 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:11:45.465 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:11:44 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:11:46.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:11:45 smithi135 ceph-mon[106011]: pgmap v1219: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:11:46.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:11:45 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T05:11:46.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:11:45 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T05:11:46.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:11:45 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T05:11:46.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:11:45 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:11:46.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:11:45 smithi123 ceph-mon[139369]: pgmap v1219: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:11:46.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:11:45 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T05:11:46.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:11:45 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T05:11:46.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:11:45 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T05:11:46.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:11:45 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:11:48.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:11:47 smithi135 ceph-mon[106011]: pgmap v1220: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:11:48.465 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:11:47 smithi123 ceph-mon[139369]: pgmap v1220: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:11:50.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:11:49 smithi135 ceph-mon[106011]: pgmap v1221: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:11:50.465 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:11:49 smithi123 ceph-mon[139369]: pgmap v1221: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:11:52.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:11:51 smithi135 ceph-mon[106011]: pgmap v1222: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:11:52.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:11:51 smithi123 ceph-mon[139369]: pgmap v1222: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:11:54.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:11:53 smithi135 ceph-mon[106011]: pgmap v1223: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:11:54.465 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:11:53 smithi123 ceph-mon[139369]: pgmap v1223: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:11:56.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:11:55 smithi135 ceph-mon[106011]: pgmap v1224: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:11:56.465 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:11:55 smithi123 ceph-mon[139369]: pgmap v1224: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:11:58.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:11:57 smithi135 ceph-mon[106011]: pgmap v1225: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:11:58.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:11:57 smithi123 ceph-mon[139369]: pgmap v1225: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:12:00.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:12:00 smithi135 ceph-mon[106011]: pgmap v1226: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:12:00.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:12:00 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:12:00.465 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:12:00 smithi123 ceph-mon[139369]: pgmap v1226: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:12:00.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:12:00 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:12:02.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:12:02 smithi135 ceph-mon[106011]: pgmap v1227: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:12:02.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:12:02 smithi123 ceph-mon[139369]: pgmap v1227: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:12:04.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:12:04 smithi135 ceph-mon[106011]: pgmap v1228: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:12:04.465 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:12:04 smithi123 ceph-mon[139369]: pgmap v1228: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:12:06.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:12:06 smithi135 ceph-mon[106011]: pgmap v1229: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:12:06.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:12:06 smithi123 ceph-mon[139369]: pgmap v1229: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:12:08.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:12:08 smithi135 ceph-mon[106011]: pgmap v1230: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:12:08.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:12:08 smithi123 ceph-mon[139369]: pgmap v1230: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:12:10.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:12:10 smithi135 ceph-mon[106011]: pgmap v1231: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:12:10.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:12:10 smithi123 ceph-mon[139369]: pgmap v1231: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:12:12.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:12:12 smithi135 ceph-mon[106011]: pgmap v1232: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:12:12.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:12:12 smithi123 ceph-mon[139369]: pgmap v1232: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:12:14.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:12:14 smithi135 ceph-mon[106011]: pgmap v1233: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:12:14.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:12:14 smithi123 ceph-mon[139369]: pgmap v1233: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:12:15.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:12:15 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:12:15.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:12:15 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:12:16.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:12:16 smithi135 ceph-mon[106011]: pgmap v1234: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:12:16.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:12:16 smithi123 ceph-mon[139369]: pgmap v1234: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:12:18.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:12:18 smithi135 ceph-mon[106011]: pgmap v1235: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:12:18.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:12:18 smithi123 ceph-mon[139369]: pgmap v1235: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:12:20.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:12:20 smithi135 ceph-mon[106011]: pgmap v1236: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:12:20.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:12:20 smithi123 ceph-mon[139369]: pgmap v1236: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:12:22.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:12:22 smithi135 ceph-mon[106011]: pgmap v1237: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:12:22.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:12:22 smithi123 ceph-mon[139369]: pgmap v1237: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:12:24.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:12:24 smithi135 ceph-mon[106011]: pgmap v1238: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:12:24.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:12:24 smithi123 ceph-mon[139369]: pgmap v1238: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:12:26.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:12:26 smithi135 ceph-mon[106011]: pgmap v1239: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:12:26.470 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:12:26 smithi123 ceph-mon[139369]: pgmap v1239: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:12:28.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:12:28 smithi135 ceph-mon[106011]: pgmap v1240: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:12:28.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:12:28 smithi123 ceph-mon[139369]: pgmap v1240: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:12:30.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:12:30 smithi135 ceph-mon[106011]: pgmap v1241: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:12:30.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:12:30 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:12:30.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:12:30 smithi123 ceph-mon[139369]: pgmap v1241: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:12:30.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:12:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:12:32.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:12:32 smithi135 ceph-mon[106011]: pgmap v1242: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:12:32.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:12:32 smithi123 ceph-mon[139369]: pgmap v1242: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:12:34.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:12:34 smithi135 ceph-mon[106011]: pgmap v1243: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:12:34.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:12:34 smithi123 ceph-mon[139369]: pgmap v1243: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:12:36.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:12:36 smithi135 ceph-mon[106011]: pgmap v1244: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:12:36.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:12:36 smithi123 ceph-mon[139369]: pgmap v1244: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:12:38.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:12:38 smithi135 ceph-mon[106011]: pgmap v1245: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:12:38.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:12:38 smithi123 ceph-mon[139369]: pgmap v1245: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:12:40.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:12:40 smithi135 ceph-mon[106011]: pgmap v1246: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:12:40.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:12:40 smithi123 ceph-mon[139369]: pgmap v1246: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:12:42.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:12:42 smithi135 ceph-mon[106011]: pgmap v1247: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:12:42.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:12:42 smithi123 ceph-mon[139369]: pgmap v1247: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:12:44.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:12:44 smithi135 ceph-mon[106011]: pgmap v1248: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:12:44.465 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:12:44 smithi123 ceph-mon[139369]: pgmap v1248: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:12:45.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:12:45 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:12:45.465 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:12:45 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:12:46.465 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:12:46 smithi123 ceph-mon[139369]: pgmap v1249: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:12:46.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:12:46 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T05:12:46.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:12:46 smithi135 ceph-mon[106011]: pgmap v1249: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:12:46.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:12:46 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T05:12:47.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:12:47 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T05:12:47.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:12:47 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T05:12:47.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:12:47 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:12:47.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:12:47 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T05:12:47.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:12:47 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T05:12:47.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:12:47 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:12:48.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:12:48 smithi123 ceph-mon[139369]: pgmap v1250: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:12:48.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:12:48 smithi135 ceph-mon[106011]: pgmap v1250: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:12:50.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:12:50 smithi123 ceph-mon[139369]: pgmap v1251: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:12:50.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:12:50 smithi135 ceph-mon[106011]: pgmap v1251: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:12:52.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:12:52 smithi123 ceph-mon[139369]: pgmap v1252: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-30T05:12:52.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:12:52 smithi135 ceph-mon[106011]: pgmap v1252: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-30T05:12:54.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:12:54 smithi123 ceph-mon[139369]: pgmap v1253: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-30T05:12:54.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:12:54 smithi135 ceph-mon[106011]: pgmap v1253: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-30T05:12:56.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:12:56 smithi123 ceph-mon[139369]: pgmap v1254: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-30T05:12:56.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:12:56 smithi135 ceph-mon[106011]: pgmap v1254: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-30T05:12:58.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:12:58 smithi123 ceph-mon[139369]: pgmap v1255: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-30T05:12:58.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:12:58 smithi135 ceph-mon[106011]: pgmap v1255: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-30T05:13:00.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:13:00 smithi123 ceph-mon[139369]: pgmap v1256: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-30T05:13:00.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:13:00 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:13:00.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:13:00 smithi135 ceph-mon[106011]: pgmap v1256: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-30T05:13:00.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:13:00 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:13:02.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:13:02 smithi123 ceph-mon[139369]: pgmap v1257: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-30T05:13:02.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:13:02 smithi135 ceph-mon[106011]: pgmap v1257: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-30T05:13:04.465 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:13:04 smithi123 ceph-mon[139369]: pgmap v1258: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:13:04.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:13:04 smithi135 ceph-mon[106011]: pgmap v1258: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:13:06.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:13:06 smithi123 ceph-mon[139369]: pgmap v1259: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:13:06.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:13:06 smithi135 ceph-mon[106011]: pgmap v1259: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:13:08.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:13:08 smithi123 ceph-mon[139369]: pgmap v1260: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:13:08.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:13:08 smithi135 ceph-mon[106011]: pgmap v1260: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:13:10.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:13:10 smithi123 ceph-mon[139369]: pgmap v1261: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:13:10.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:13:10 smithi135 ceph-mon[106011]: pgmap v1261: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:13:12.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:13:12 smithi123 ceph-mon[139369]: pgmap v1262: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:13:12.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:13:12 smithi135 ceph-mon[106011]: pgmap v1262: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:13:14.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:13:14 smithi123 ceph-mon[139369]: pgmap v1263: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:13:14.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:13:14 smithi135 ceph-mon[106011]: pgmap v1263: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:13:15.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:13:15 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:13:15.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:13:15 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:13:16.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:13:16 smithi123 ceph-mon[139369]: pgmap v1264: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:13:16.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:13:16 smithi135 ceph-mon[106011]: pgmap v1264: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:13:18.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:13:18 smithi123 ceph-mon[139369]: pgmap v1265: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:13:18.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:13:18 smithi135 ceph-mon[106011]: pgmap v1265: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:13:20.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:13:20 smithi123 ceph-mon[139369]: pgmap v1266: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:13:20.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:13:20 smithi135 ceph-mon[106011]: pgmap v1266: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:13:22.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:13:22 smithi123 ceph-mon[139369]: pgmap v1267: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:13:22.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:13:22 smithi135 ceph-mon[106011]: pgmap v1267: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:13:24.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:13:24 smithi123 ceph-mon[139369]: pgmap v1268: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:13:24.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:13:24 smithi135 ceph-mon[106011]: pgmap v1268: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:13:26.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:13:26 smithi123 ceph-mon[139369]: pgmap v1269: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:13:26.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:13:26 smithi135 ceph-mon[106011]: pgmap v1269: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:13:28.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:13:28 smithi123 ceph-mon[139369]: pgmap v1270: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:13:28.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:13:28 smithi135 ceph-mon[106011]: pgmap v1270: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:13:30.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:13:30 smithi123 ceph-mon[139369]: pgmap v1271: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:13:30.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:13:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:13:30.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:13:30 smithi135 ceph-mon[106011]: pgmap v1271: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:13:30.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:13:30 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:13:32.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:13:32 smithi123 ceph-mon[139369]: pgmap v1272: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:13:32.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:13:32 smithi135 ceph-mon[106011]: pgmap v1272: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:13:34.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:13:34 smithi123 ceph-mon[139369]: pgmap v1273: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:13:34.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:13:34 smithi135 ceph-mon[106011]: pgmap v1273: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:13:36.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:13:36 smithi123 ceph-mon[139369]: pgmap v1274: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:13:36.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:13:36 smithi135 ceph-mon[106011]: pgmap v1274: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:13:38.465 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:13:38 smithi123 ceph-mon[139369]: pgmap v1275: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:13:38.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:13:38 smithi135 ceph-mon[106011]: pgmap v1275: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:13:40.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:13:40 smithi135 ceph-mon[106011]: pgmap v1276: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:13:40.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:13:40 smithi123 ceph-mon[139369]: pgmap v1276: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:13:42.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:13:42 smithi135 ceph-mon[106011]: pgmap v1277: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:13:42.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:13:42 smithi123 ceph-mon[139369]: pgmap v1277: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:13:44.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:13:44 smithi135 ceph-mon[106011]: pgmap v1278: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:13:44.715 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:13:44 smithi123 ceph-mon[139369]: pgmap v1278: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:13:45.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:13:45 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:13:45.715 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:13:45 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:13:46.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:13:46 smithi135 ceph-mon[106011]: pgmap v1279: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:13:46.715 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:13:46 smithi123 ceph-mon[139369]: pgmap v1279: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:13:47.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:13:47 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T05:13:47.715 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:13:47 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T05:13:48.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:13:48 smithi123 ceph-mon[139369]: pgmap v1280: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:13:48.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:13:48 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T05:13:48.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:13:48 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T05:13:48.467 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:13:48 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:13:48.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:13:48 smithi135 ceph-mon[106011]: pgmap v1280: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:13:48.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:13:48 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T05:13:48.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:13:48 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T05:13:48.601 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:13:48 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:13:50.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:13:50 smithi135 ceph-mon[106011]: pgmap v1281: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:13:50.715 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:13:50 smithi123 ceph-mon[139369]: pgmap v1281: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:13:52.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:13:52 smithi135 ceph-mon[106011]: pgmap v1282: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:13:52.715 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:13:52 smithi123 ceph-mon[139369]: pgmap v1282: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:13:54.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:13:54 smithi135 ceph-mon[106011]: pgmap v1283: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:13:54.715 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:13:54 smithi123 ceph-mon[139369]: pgmap v1283: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:13:56.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:13:56 smithi135 ceph-mon[106011]: pgmap v1284: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:13:56.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:13:56 smithi123 ceph-mon[139369]: pgmap v1284: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:13:58.465 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:13:58 smithi123 ceph-mon[139369]: pgmap v1285: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:13:58.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:13:58 smithi135 ceph-mon[106011]: pgmap v1285: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:14:00.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:14:00 smithi135 ceph-mon[106011]: pgmap v1286: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:14:00.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:14:00 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:14:00.715 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:14:00 smithi123 ceph-mon[139369]: pgmap v1286: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:14:00.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:14:00 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:14:02.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:14:02 smithi135 ceph-mon[106011]: pgmap v1287: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:14:02.715 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:14:02 smithi123 ceph-mon[139369]: pgmap v1287: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:14:04.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:14:04 smithi135 ceph-mon[106011]: pgmap v1288: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:14:04.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:14:04 smithi123 ceph-mon[139369]: pgmap v1288: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:14:06.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:14:06 smithi135 ceph-mon[106011]: pgmap v1289: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:14:06.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:14:06 smithi123 ceph-mon[139369]: pgmap v1289: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:14:07.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:14:07 smithi135 ceph-mon[106011]: pgmap v1290: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:14:07.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:14:07 smithi123 ceph-mon[139369]: pgmap v1290: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:14:09.965 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:14:09 smithi123 ceph-mon[139369]: pgmap v1291: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:14:10.100 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:14:09 smithi135 ceph-mon[106011]: pgmap v1291: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:14:12.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:14:11 smithi135 ceph-mon[106011]: pgmap v1292: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:14:12.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:14:11 smithi123 ceph-mon[139369]: pgmap v1292: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:14:14.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:14:13 smithi135 ceph-mon[106011]: pgmap v1293: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:14:14.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:14:13 smithi123 ceph-mon[139369]: pgmap v1293: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:14:15.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:14:14 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:14:15.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:14:14 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:14:16.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:14:15 smithi135 ceph-mon[106011]: pgmap v1294: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:14:16.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:14:15 smithi123 ceph-mon[139369]: pgmap v1294: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:14:18.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:14:17 smithi135 ceph-mon[106011]: pgmap v1295: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:14:18.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:14:17 smithi123 ceph-mon[139369]: pgmap v1295: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:14:20.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:14:19 smithi135 ceph-mon[106011]: pgmap v1296: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:14:20.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:14:19 smithi123 ceph-mon[139369]: pgmap v1296: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:14:22.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:14:21 smithi135 ceph-mon[106011]: pgmap v1297: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:14:22.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:14:21 smithi123 ceph-mon[139369]: pgmap v1297: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:14:24.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:14:24 smithi135 ceph-mon[106011]: pgmap v1298: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:14:24.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:14:23 smithi123 ceph-mon[139369]: pgmap v1298: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:14:26.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:14:26 smithi135 ceph-mon[106011]: pgmap v1299: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:14:26.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:14:26 smithi123 ceph-mon[139369]: pgmap v1299: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:14:28.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:14:28 smithi135 ceph-mon[106011]: pgmap v1300: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:14:28.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:14:28 smithi123 ceph-mon[139369]: pgmap v1300: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:14:30.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:14:30 smithi135 ceph-mon[106011]: pgmap v1301: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:14:30.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:14:30 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:14:30.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:14:30 smithi123 ceph-mon[139369]: pgmap v1301: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:14:30.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:14:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:14:32.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:14:32 smithi135 ceph-mon[106011]: pgmap v1302: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:14:32.465 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:14:32 smithi123 ceph-mon[139369]: pgmap v1302: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:14:34.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:14:34 smithi135 ceph-mon[106011]: pgmap v1303: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:14:34.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:14:34 smithi123 ceph-mon[139369]: pgmap v1303: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:14:36.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:14:36 smithi135 ceph-mon[106011]: pgmap v1304: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:14:36.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:14:36 smithi123 ceph-mon[139369]: pgmap v1304: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:14:38.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:14:38 smithi135 ceph-mon[106011]: pgmap v1305: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:14:38.465 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:14:38 smithi123 ceph-mon[139369]: pgmap v1305: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:14:40.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:14:40 smithi135 ceph-mon[106011]: pgmap v1306: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:14:40.465 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:14:40 smithi123 ceph-mon[139369]: pgmap v1306: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:14:42.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:14:42 smithi135 ceph-mon[106011]: pgmap v1307: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:14:42.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:14:42 smithi123 ceph-mon[139369]: pgmap v1307: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:14:44.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:14:44 smithi135 ceph-mon[106011]: pgmap v1308: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:14:44.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:14:44 smithi123 ceph-mon[139369]: pgmap v1308: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:14:45.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:14:45 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:14:45.465 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:14:45 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:14:46.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:14:46 smithi135 ceph-mon[106011]: pgmap v1309: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:14:46.465 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:14:46 smithi123 ceph-mon[139369]: pgmap v1309: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:14:48.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:14:48 smithi135 ceph-mon[106011]: pgmap v1310: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:14:48.465 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:14:48 smithi123 ceph-mon[139369]: pgmap v1310: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:14:49.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:14:49 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T05:14:49.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:14:49 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T05:14:49.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:14:49 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T05:14:49.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:14:49 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:14:49.465 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:14:49 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T05:14:49.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:14:49 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T05:14:49.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:14:49 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T05:14:49.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:14:49 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' 2024-03-30T05:14:50.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:14:50 smithi135 ceph-mon[106011]: pgmap v1311: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:14:50.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:14:50 smithi123 ceph-mon[139369]: pgmap v1311: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:14:52.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:14:52 smithi135 ceph-mon[106011]: pgmap v1312: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:14:52.465 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:14:52 smithi123 ceph-mon[139369]: pgmap v1312: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:14:54.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:14:54 smithi135 ceph-mon[106011]: pgmap v1313: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:14:54.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:14:54 smithi123 ceph-mon[139369]: pgmap v1313: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:14:56.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:14:56 smithi135 ceph-mon[106011]: pgmap v1314: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:14:56.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:14:56 smithi123 ceph-mon[139369]: pgmap v1314: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:14:58.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:14:58 smithi135 ceph-mon[106011]: pgmap v1315: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:14:58.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:14:58 smithi123 ceph-mon[139369]: pgmap v1315: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:15:00.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:15:00 smithi135 ceph-mon[106011]: pgmap v1316: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:15:00.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:15:00 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:15:00.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:15:00 smithi123 ceph-mon[139369]: pgmap v1316: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:15:00.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:15:00 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:15:02.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:15:02 smithi135 ceph-mon[106011]: pgmap v1317: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:15:02.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:15:02 smithi123 ceph-mon[139369]: pgmap v1317: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:15:04.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:15:04 smithi135 ceph-mon[106011]: pgmap v1318: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:15:04.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:15:04 smithi123 ceph-mon[139369]: pgmap v1318: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:15:06.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:15:06 smithi135 ceph-mon[106011]: pgmap v1319: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:15:06.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:15:06 smithi123 ceph-mon[139369]: pgmap v1319: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:15:08.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:15:08 smithi135 ceph-mon[106011]: pgmap v1320: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:15:08.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:15:08 smithi123 ceph-mon[139369]: pgmap v1320: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:15:10.465 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:15:10 smithi123 ceph-mon[139369]: pgmap v1321: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:15:10.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:15:10 smithi135 ceph-mon[106011]: pgmap v1321: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:15:12.465 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:15:12 smithi123 ceph-mon[139369]: pgmap v1322: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:15:12.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:15:12 smithi135 ceph-mon[106011]: pgmap v1322: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:15:14.465 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:15:14 smithi123 ceph-mon[139369]: pgmap v1323: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:15:14.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:15:14 smithi135 ceph-mon[106011]: pgmap v1323: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:15:15.465 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:15:15 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:15:15.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:15:15 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:15:16.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:15:16 smithi123 ceph-mon[139369]: pgmap v1324: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:15:16.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:15:16 smithi135 ceph-mon[106011]: pgmap v1324: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:15:18.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:15:18 smithi123 ceph-mon[139369]: pgmap v1325: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:15:18.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:15:18 smithi135 ceph-mon[106011]: pgmap v1325: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:15:20.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:15:20 smithi123 ceph-mon[139369]: pgmap v1326: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:15:20.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:15:20 smithi135 ceph-mon[106011]: pgmap v1326: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:15:22.465 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:15:22 smithi123 ceph-mon[139369]: pgmap v1327: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:15:22.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:15:22 smithi135 ceph-mon[106011]: pgmap v1327: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:15:24.465 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:15:24 smithi123 ceph-mon[139369]: pgmap v1328: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:15:24.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:15:24 smithi135 ceph-mon[106011]: pgmap v1328: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:15:26.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:15:26 smithi123 ceph-mon[139369]: pgmap v1329: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:15:26.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:15:26 smithi135 ceph-mon[106011]: pgmap v1329: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:15:28.465 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:15:28 smithi123 ceph-mon[139369]: pgmap v1330: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:15:28.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:15:28 smithi135 ceph-mon[106011]: pgmap v1330: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:15:30.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:15:30 smithi123 ceph-mon[139369]: pgmap v1331: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:15:30.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:15:30 smithi123 ceph-mon[139369]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:15:30.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:15:30 smithi135 ceph-mon[106011]: pgmap v1331: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:15:30.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:15:30 smithi135 ceph-mon[106011]: from='mgr.34104 172.21.15.123:0/3910741067' entity='mgr.smithi123.xgksqq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T05:15:32.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:15:32 smithi123 ceph-mon[139369]: pgmap v1332: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:15:32.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:15:32 smithi135 ceph-mon[106011]: pgmap v1332: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:15:34.465 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:15:34 smithi123 ceph-mon[139369]: pgmap v1333: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:15:34.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:15:34 smithi135 ceph-mon[106011]: pgmap v1333: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:15:36.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:15:36 smithi123 ceph-mon[139369]: pgmap v1334: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:15:36.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:15:36 smithi135 ceph-mon[106011]: pgmap v1334: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:15:38.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:15:38 smithi123 ceph-mon[139369]: pgmap v1335: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:15:38.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:15:38 smithi135 ceph-mon[106011]: pgmap v1335: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:15:40.466 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:15:40 smithi123 ceph-mon[139369]: pgmap v1336: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:15:40.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:15:40 smithi135 ceph-mon[106011]: pgmap v1336: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:15:42.465 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:15:42 smithi123 ceph-mon[139369]: pgmap v1337: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:15:42.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:15:42 smithi135 ceph-mon[106011]: pgmap v1337: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-30T05:15:43.964 ERROR:tasks.cephfs.fuse_mount:process failed to terminate after unmount. This probably indicates a bug within ceph-fuse. 2024-03-30T05:15:43.965 ERROR:teuthology.run_tasks:Manager failed: ceph-fuse Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_470100a4ccbcf7769052bf09a5413881ccf1b97f/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/git.ceph.com_ceph-c_470100a4ccbcf7769052bf09a5413881ccf1b97f/qa/tasks/cephfs/fuse_mount.py", line 400, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-03-30T05:15:43.967 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-30T05:15:43.978 INFO:tasks.cephadm:Teardown begin 2024-03-30T05:15:43.978 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_470100a4ccbcf7769052bf09a5413881ccf1b97f/qa/tasks/cephadm.py", line 1750, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_470100a4ccbcf7769052bf09a5413881ccf1b97f/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/git.ceph.com_ceph-c_470100a4ccbcf7769052bf09a5413881ccf1b97f/qa/tasks/cephfs/fuse_mount.py", line 400, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-03-30T05:15:44.030 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-30T05:15:44.112 DEBUG:teuthology.orchestra.run.smithi135:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-30T05:15:44.192 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-30T05:15:44.192 DEBUG:teuthology.orchestra.run.smithi123:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-30T05:15:44.244 DEBUG:teuthology.orchestra.run.smithi135:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-30T05:15:44.289 INFO:tasks.cephadm:Stopping all daemons... 2024-03-30T05:15:44.290 INFO:tasks.cephadm.mon.smithi123:Stopping mon.smithi123... 2024-03-30T05:15:44.290 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4@mon.smithi123 2024-03-30T05:15:44.356 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:15:44 smithi123 ceph-mon[139369]: pgmap v1338: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:15:44.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:15:44 smithi135 ceph-mon[106011]: pgmap v1338: 65 pgs: 65 active+clean; 260 MiB data, 997 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-30T05:15:44.716 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:15:44 smithi123 systemd[1]: Stopping Ceph mon.smithi123 for 2e77a2be-ee4c-11ee-b647-cb9ed24678a4... 2024-03-30T05:15:45.091 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:15:44 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-mon-smithi123[139346]: 2024-03-30T05:15:44.740+0000 7f50eba00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi123 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T05:15:45.091 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:15:44 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-mon-smithi123[139346]: 2024-03-30T05:15:44.740+0000 7f50eba00700 -1 mon.smithi123@0(leader) e3 *** Got Signal Terminated *** 2024-03-30T05:15:45.465 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Mar 30 05:15:45 smithi123 bash[205616]: ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-mon-smithi123 2024-03-30T05:15:45.763 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4@mon.smithi123.service' 2024-03-30T05:15:45.857 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T05:15:45.857 INFO:tasks.cephadm.mon.smithi123:Stopped mon.smithi123 2024-03-30T05:15:45.857 INFO:tasks.cephadm.mon.smithi135:Stopping mon.smithi135... 2024-03-30T05:15:45.857 DEBUG:teuthology.orchestra.run.smithi135:> sudo systemctl stop ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4@mon.smithi135 2024-03-30T05:15:46.334 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:15:45 smithi135 systemd[1]: Stopping Ceph mon.smithi135 for 2e77a2be-ee4c-11ee-b647-cb9ed24678a4... 2024-03-30T05:15:46.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:15:46 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-mon-smithi135[105988]: 2024-03-30T05:15:46.334+0000 7f8500e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi135 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T05:15:46.600 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:15:46 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-mon-smithi135[105988]: 2024-03-30T05:15:46.334+0000 7f8500e00700 -1 mon.smithi135@1(peon) e3 *** Got Signal Terminated *** 2024-03-30T05:15:47.350 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Mar 30 05:15:47 smithi135 bash[153175]: ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-mon-smithi135 2024-03-30T05:15:47.522 DEBUG:teuthology.orchestra.run.smithi135:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4@mon.smithi135.service' 2024-03-30T05:15:47.622 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T05:15:47.622 INFO:tasks.cephadm.mon.smithi135:Stopped mon.smithi135 2024-03-30T05:15:47.623 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-30T05:15:47.623 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4@osd.0 2024-03-30T05:15:47.966 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 30 05:15:47 smithi123 systemd[1]: Stopping Ceph osd.0 for 2e77a2be-ee4c-11ee-b647-cb9ed24678a4... 2024-03-30T05:15:48.465 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 30 05:15:48 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-0[153827]: 2024-03-30T05:15:48.020+0000 7f270c800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T05:15:48.466 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 30 05:15:48 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-0[153827]: 2024-03-30T05:15:48.020+0000 7f270c800700 -1 osd.0 86 *** Got signal Terminated *** 2024-03-30T05:15:48.466 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 30 05:15:48 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-0[153827]: 2024-03-30T05:15:48.020+0000 7f270c800700 -1 osd.0 86 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T05:15:54.216 INFO:journalctl@ceph.osd.0.smithi123.stdout:Mar 30 05:15:53 smithi123 bash[205775]: ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-0 2024-03-30T05:15:55.916 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4@osd.0.service' 2024-03-30T05:15:56.005 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T05:15:56.005 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-30T05:15:56.005 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-30T05:15:56.005 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4@osd.1 2024-03-30T05:15:56.216 INFO:journalctl@ceph.osd.1.smithi123.stdout:Mar 30 05:15:56 smithi123 systemd[1]: Stopping Ceph osd.1 for 2e77a2be-ee4c-11ee-b647-cb9ed24678a4... 2024-03-30T05:15:56.716 INFO:journalctl@ceph.osd.1.smithi123.stdout:Mar 30 05:15:56 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-1[160401]: 2024-03-30T05:15:56.422+0000 7f5a5a800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T05:15:56.716 INFO:journalctl@ceph.osd.1.smithi123.stdout:Mar 30 05:15:56 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-1[160401]: 2024-03-30T05:15:56.422+0000 7f5a5a800700 -1 osd.1 86 *** Got signal Terminated *** 2024-03-30T05:15:56.716 INFO:journalctl@ceph.osd.1.smithi123.stdout:Mar 30 05:15:56 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-1[160401]: 2024-03-30T05:15:56.422+0000 7f5a5a800700 -1 osd.1 86 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T05:16:02.466 INFO:journalctl@ceph.osd.1.smithi123.stdout:Mar 30 05:16:02 smithi123 bash[206040]: ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-1 2024-03-30T05:16:04.208 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4@osd.1.service' 2024-03-30T05:16:04.296 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T05:16:04.296 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-30T05:16:04.296 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-30T05:16:04.297 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4@osd.2 2024-03-30T05:16:04.466 INFO:journalctl@ceph.osd.2.smithi123.stdout:Mar 30 05:16:04 smithi123 systemd[1]: Stopping Ceph osd.2 for 2e77a2be-ee4c-11ee-b647-cb9ed24678a4... 2024-03-30T05:16:04.965 INFO:journalctl@ceph.osd.2.smithi123.stdout:Mar 30 05:16:04 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-2[166988]: 2024-03-30T05:16:04.699+0000 7fd308c00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T05:16:04.966 INFO:journalctl@ceph.osd.2.smithi123.stdout:Mar 30 05:16:04 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-2[166988]: 2024-03-30T05:16:04.699+0000 7fd308c00700 -1 osd.2 86 *** Got signal Terminated *** 2024-03-30T05:16:04.966 INFO:journalctl@ceph.osd.2.smithi123.stdout:Mar 30 05:16:04 smithi123 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-2[166988]: 2024-03-30T05:16:04.699+0000 7fd308c00700 -1 osd.2 86 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T05:16:10.716 INFO:journalctl@ceph.osd.2.smithi123.stdout:Mar 30 05:16:10 smithi123 bash[206302]: ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-2 2024-03-30T05:16:12.576 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4@osd.2.service' 2024-03-30T05:16:12.663 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T05:16:12.663 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-30T05:16:12.663 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-30T05:16:12.663 DEBUG:teuthology.orchestra.run.smithi135:> sudo systemctl stop ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4@osd.3 2024-03-30T05:16:13.061 INFO:journalctl@ceph.osd.3.smithi135.stdout:Mar 30 05:16:12 smithi135 systemd[1]: Stopping Ceph osd.3 for 2e77a2be-ee4c-11ee-b647-cb9ed24678a4... 2024-03-30T05:16:13.350 INFO:journalctl@ceph.osd.3.smithi135.stdout:Mar 30 05:16:13 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-3[118739]: 2024-03-30T05:16:13.062+0000 7f1e50400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T05:16:13.350 INFO:journalctl@ceph.osd.3.smithi135.stdout:Mar 30 05:16:13 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-3[118739]: 2024-03-30T05:16:13.062+0000 7f1e50400700 -1 osd.3 86 *** Got signal Terminated *** 2024-03-30T05:16:13.350 INFO:journalctl@ceph.osd.3.smithi135.stdout:Mar 30 05:16:13 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-3[118739]: 2024-03-30T05:16:13.062+0000 7f1e50400700 -1 osd.3 86 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T05:16:15.600 INFO:journalctl@ceph.osd.3.smithi135.stdout:Mar 30 05:16:15 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-3[118739]: 2024-03-30T05:16:15.155+0000 7f1e4c600700 -1 osd.3 86 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-30T05:15:48.806558+0000 front 2024-03-30T05:15:48.806474+0000 (oldest deadline 2024-03-30T05:16:14.706222+0000) 2024-03-30T05:16:15.600 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 05:16:15 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-5[131567]: 2024-03-30T05:16:15.193+0000 7fadcf800700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-30T05:15:52.642193+0000 front 2024-03-30T05:15:52.642174+0000 (oldest deadline 2024-03-30T05:16:14.942168+0000) 2024-03-30T05:16:16.600 INFO:journalctl@ceph.osd.3.smithi135.stdout:Mar 30 05:16:16 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-3[118739]: 2024-03-30T05:16:16.128+0000 7f1e4c600700 -1 osd.3 86 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-30T05:15:48.806558+0000 front 2024-03-30T05:15:48.806474+0000 (oldest deadline 2024-03-30T05:16:14.706222+0000) 2024-03-30T05:16:16.600 INFO:journalctl@ceph.osd.4.smithi135.stdout:Mar 30 05:16:16 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-4[125510]: 2024-03-30T05:16:16.211+0000 7fa632a00700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-30T05:15:50.572182+0000 front 2024-03-30T05:15:50.572261+0000 (oldest deadline 2024-03-30T05:16:15.872204+0000) 2024-03-30T05:16:16.600 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 05:16:16 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-5[131567]: 2024-03-30T05:16:16.151+0000 7fadcf800700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-30T05:15:52.642193+0000 front 2024-03-30T05:15:52.642174+0000 (oldest deadline 2024-03-30T05:16:14.942168+0000) 2024-03-30T05:16:17.600 INFO:journalctl@ceph.osd.3.smithi135.stdout:Mar 30 05:16:17 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-3[118739]: 2024-03-30T05:16:17.148+0000 7f1e4c600700 -1 osd.3 86 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-30T05:15:48.806558+0000 front 2024-03-30T05:15:48.806474+0000 (oldest deadline 2024-03-30T05:16:14.706222+0000) 2024-03-30T05:16:17.600 INFO:journalctl@ceph.osd.4.smithi135.stdout:Mar 30 05:16:17 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-4[125510]: 2024-03-30T05:16:17.161+0000 7fa632a00700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-30T05:15:50.572182+0000 front 2024-03-30T05:15:50.572261+0000 (oldest deadline 2024-03-30T05:16:15.872204+0000) 2024-03-30T05:16:17.600 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 05:16:17 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-5[131567]: 2024-03-30T05:16:17.198+0000 7fadcf800700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-30T05:15:52.642193+0000 front 2024-03-30T05:15:52.642174+0000 (oldest deadline 2024-03-30T05:16:14.942168+0000) 2024-03-30T05:16:18.600 INFO:journalctl@ceph.osd.4.smithi135.stdout:Mar 30 05:16:18 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-4[125510]: 2024-03-30T05:16:18.151+0000 7fa632a00700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-30T05:15:50.572182+0000 front 2024-03-30T05:15:50.572261+0000 (oldest deadline 2024-03-30T05:16:15.872204+0000) 2024-03-30T05:16:18.600 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 05:16:18 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-5[131567]: 2024-03-30T05:16:18.194+0000 7fadcf800700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-30T05:15:52.642193+0000 front 2024-03-30T05:15:52.642174+0000 (oldest deadline 2024-03-30T05:16:14.942168+0000) 2024-03-30T05:16:19.100 INFO:journalctl@ceph.osd.3.smithi135.stdout:Mar 30 05:16:18 smithi135 bash[153321]: ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-3 2024-03-30T05:16:19.600 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 05:16:19 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-5[131567]: 2024-03-30T05:16:19.181+0000 7fadcf800700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-30T05:15:52.642193+0000 front 2024-03-30T05:15:52.642174+0000 (oldest deadline 2024-03-30T05:16:14.942168+0000) 2024-03-30T05:16:19.600 INFO:journalctl@ceph.osd.4.smithi135.stdout:Mar 30 05:16:19 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-4[125510]: 2024-03-30T05:16:19.201+0000 7fa632a00700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-30T05:15:50.572182+0000 front 2024-03-30T05:15:50.572261+0000 (oldest deadline 2024-03-30T05:16:15.872204+0000) 2024-03-30T05:16:20.350 INFO:journalctl@ceph.osd.4.smithi135.stdout:Mar 30 05:16:20 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-4[125510]: 2024-03-30T05:16:20.208+0000 7fa632a00700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-30T05:15:50.572182+0000 front 2024-03-30T05:15:50.572261+0000 (oldest deadline 2024-03-30T05:16:15.872204+0000) 2024-03-30T05:16:20.350 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 05:16:20 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-5[131567]: 2024-03-30T05:16:20.156+0000 7fadcf800700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-30T05:15:52.642193+0000 front 2024-03-30T05:15:52.642174+0000 (oldest deadline 2024-03-30T05:16:14.942168+0000) 2024-03-30T05:16:21.192 DEBUG:teuthology.orchestra.run.smithi135:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4@osd.3.service' 2024-03-30T05:16:21.278 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T05:16:21.279 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-30T05:16:21.279 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-30T05:16:21.279 DEBUG:teuthology.orchestra.run.smithi135:> sudo systemctl stop ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4@osd.4 2024-03-30T05:16:21.600 INFO:journalctl@ceph.osd.4.smithi135.stdout:Mar 30 05:16:21 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-4[125510]: 2024-03-30T05:16:21.186+0000 7fa632a00700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-30T05:15:50.572182+0000 front 2024-03-30T05:15:50.572261+0000 (oldest deadline 2024-03-30T05:16:15.872204+0000) 2024-03-30T05:16:21.600 INFO:journalctl@ceph.osd.4.smithi135.stdout:Mar 30 05:16:21 smithi135 systemd[1]: Stopping Ceph osd.4 for 2e77a2be-ee4c-11ee-b647-cb9ed24678a4... 2024-03-30T05:16:21.600 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 05:16:21 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-5[131567]: 2024-03-30T05:16:21.169+0000 7fadcf800700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-30T05:15:52.642193+0000 front 2024-03-30T05:15:52.642174+0000 (oldest deadline 2024-03-30T05:16:14.942168+0000) 2024-03-30T05:16:22.100 INFO:journalctl@ceph.osd.4.smithi135.stdout:Mar 30 05:16:21 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-4[125510]: 2024-03-30T05:16:21.680+0000 7fa63b800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T05:16:22.100 INFO:journalctl@ceph.osd.4.smithi135.stdout:Mar 30 05:16:21 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-4[125510]: 2024-03-30T05:16:21.680+0000 7fa63b800700 -1 osd.4 86 *** Got signal Terminated *** 2024-03-30T05:16:22.100 INFO:journalctl@ceph.osd.4.smithi135.stdout:Mar 30 05:16:21 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-4[125510]: 2024-03-30T05:16:21.680+0000 7fa63b800700 -1 osd.4 86 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T05:16:22.600 INFO:journalctl@ceph.osd.4.smithi135.stdout:Mar 30 05:16:22 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-4[125510]: 2024-03-30T05:16:22.219+0000 7fa632a00700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-30T05:15:50.572182+0000 front 2024-03-30T05:15:50.572261+0000 (oldest deadline 2024-03-30T05:16:15.872204+0000) 2024-03-30T05:16:22.600 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 05:16:22 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-5[131567]: 2024-03-30T05:16:22.210+0000 7fadcf800700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-30T05:15:52.642193+0000 front 2024-03-30T05:15:52.642174+0000 (oldest deadline 2024-03-30T05:16:14.942168+0000) 2024-03-30T05:16:23.600 INFO:journalctl@ceph.osd.4.smithi135.stdout:Mar 30 05:16:23 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-4[125510]: 2024-03-30T05:16:23.239+0000 7fa632a00700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-30T05:15:50.572182+0000 front 2024-03-30T05:15:50.572261+0000 (oldest deadline 2024-03-30T05:16:15.872204+0000) 2024-03-30T05:16:23.601 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 05:16:23 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-5[131567]: 2024-03-30T05:16:23.199+0000 7fadcf800700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-30T05:15:52.642193+0000 front 2024-03-30T05:15:52.642174+0000 (oldest deadline 2024-03-30T05:16:14.942168+0000) 2024-03-30T05:16:23.601 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 05:16:23 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-5[131567]: 2024-03-30T05:16:23.199+0000 7fadcf800700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.123:6814 osd.1 since back 2024-03-30T05:16:00.743668+0000 front 2024-03-30T05:16:00.743686+0000 (oldest deadline 2024-03-30T05:16:23.043601+0000) 2024-03-30T05:16:24.600 INFO:journalctl@ceph.osd.4.smithi135.stdout:Mar 30 05:16:24 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-4[125510]: 2024-03-30T05:16:24.223+0000 7fa632a00700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-30T05:15:50.572182+0000 front 2024-03-30T05:15:50.572261+0000 (oldest deadline 2024-03-30T05:16:15.872204+0000) 2024-03-30T05:16:24.600 INFO:journalctl@ceph.osd.4.smithi135.stdout:Mar 30 05:16:24 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-4[125510]: 2024-03-30T05:16:24.223+0000 7fa632a00700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.123:6814 osd.1 since back 2024-03-30T05:15:58.773510+0000 front 2024-03-30T05:15:58.773493+0000 (oldest deadline 2024-03-30T05:16:24.073239+0000) 2024-03-30T05:16:24.600 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 05:16:24 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-5[131567]: 2024-03-30T05:16:24.155+0000 7fadcf800700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-30T05:15:52.642193+0000 front 2024-03-30T05:15:52.642174+0000 (oldest deadline 2024-03-30T05:16:14.942168+0000) 2024-03-30T05:16:24.600 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 05:16:24 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-5[131567]: 2024-03-30T05:16:24.155+0000 7fadcf800700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.123:6814 osd.1 since back 2024-03-30T05:16:00.743668+0000 front 2024-03-30T05:16:00.743686+0000 (oldest deadline 2024-03-30T05:16:23.043601+0000) 2024-03-30T05:16:25.600 INFO:journalctl@ceph.osd.4.smithi135.stdout:Mar 30 05:16:25 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-4[125510]: 2024-03-30T05:16:25.189+0000 7fa632a00700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-30T05:15:50.572182+0000 front 2024-03-30T05:15:50.572261+0000 (oldest deadline 2024-03-30T05:16:15.872204+0000) 2024-03-30T05:16:25.600 INFO:journalctl@ceph.osd.4.smithi135.stdout:Mar 30 05:16:25 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-4[125510]: 2024-03-30T05:16:25.189+0000 7fa632a00700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.123:6814 osd.1 since back 2024-03-30T05:15:58.773510+0000 front 2024-03-30T05:15:58.773493+0000 (oldest deadline 2024-03-30T05:16:24.073239+0000) 2024-03-30T05:16:25.601 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 05:16:25 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-5[131567]: 2024-03-30T05:16:25.134+0000 7fadcf800700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-30T05:15:52.642193+0000 front 2024-03-30T05:15:52.642174+0000 (oldest deadline 2024-03-30T05:16:14.942168+0000) 2024-03-30T05:16:25.601 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 05:16:25 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-5[131567]: 2024-03-30T05:16:25.134+0000 7fadcf800700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.123:6814 osd.1 since back 2024-03-30T05:16:00.743668+0000 front 2024-03-30T05:16:00.743686+0000 (oldest deadline 2024-03-30T05:16:23.043601+0000) 2024-03-30T05:16:26.600 INFO:journalctl@ceph.osd.4.smithi135.stdout:Mar 30 05:16:26 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-4[125510]: 2024-03-30T05:16:26.235+0000 7fa632a00700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-30T05:15:50.572182+0000 front 2024-03-30T05:15:50.572261+0000 (oldest deadline 2024-03-30T05:16:15.872204+0000) 2024-03-30T05:16:26.600 INFO:journalctl@ceph.osd.4.smithi135.stdout:Mar 30 05:16:26 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-4[125510]: 2024-03-30T05:16:26.235+0000 7fa632a00700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.123:6814 osd.1 since back 2024-03-30T05:15:58.773510+0000 front 2024-03-30T05:15:58.773493+0000 (oldest deadline 2024-03-30T05:16:24.073239+0000) 2024-03-30T05:16:26.600 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 05:16:26 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-5[131567]: 2024-03-30T05:16:26.127+0000 7fadcf800700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-30T05:15:52.642193+0000 front 2024-03-30T05:15:52.642174+0000 (oldest deadline 2024-03-30T05:16:14.942168+0000) 2024-03-30T05:16:26.600 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 05:16:26 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-5[131567]: 2024-03-30T05:16:26.127+0000 7fadcf800700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.123:6814 osd.1 since back 2024-03-30T05:16:00.743668+0000 front 2024-03-30T05:16:00.743686+0000 (oldest deadline 2024-03-30T05:16:23.043601+0000) 2024-03-30T05:16:27.421 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 05:16:27 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-5[131567]: 2024-03-30T05:16:27.129+0000 7fadcf800700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-30T05:15:52.642193+0000 front 2024-03-30T05:15:52.642174+0000 (oldest deadline 2024-03-30T05:16:14.942168+0000) 2024-03-30T05:16:27.421 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 05:16:27 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-5[131567]: 2024-03-30T05:16:27.129+0000 7fadcf800700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.123:6814 osd.1 since back 2024-03-30T05:16:00.743668+0000 front 2024-03-30T05:16:00.743686+0000 (oldest deadline 2024-03-30T05:16:23.043601+0000) 2024-03-30T05:16:27.850 INFO:journalctl@ceph.osd.4.smithi135.stdout:Mar 30 05:16:27 smithi135 bash[153585]: ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-4 2024-03-30T05:16:28.350 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 05:16:28 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-5[131567]: 2024-03-30T05:16:28.091+0000 7fadcf800700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-30T05:15:52.642193+0000 front 2024-03-30T05:15:52.642174+0000 (oldest deadline 2024-03-30T05:16:14.942168+0000) 2024-03-30T05:16:28.350 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 05:16:28 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-5[131567]: 2024-03-30T05:16:28.091+0000 7fadcf800700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.123:6814 osd.1 since back 2024-03-30T05:16:00.743668+0000 front 2024-03-30T05:16:00.743686+0000 (oldest deadline 2024-03-30T05:16:23.043601+0000) 2024-03-30T05:16:29.350 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 05:16:29 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-5[131567]: 2024-03-30T05:16:29.090+0000 7fadcf800700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-30T05:15:52.642193+0000 front 2024-03-30T05:15:52.642174+0000 (oldest deadline 2024-03-30T05:16:14.942168+0000) 2024-03-30T05:16:29.350 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 05:16:29 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-5[131567]: 2024-03-30T05:16:29.090+0000 7fadcf800700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.123:6814 osd.1 since back 2024-03-30T05:16:00.743668+0000 front 2024-03-30T05:16:00.743686+0000 (oldest deadline 2024-03-30T05:16:23.043601+0000) 2024-03-30T05:16:29.810 DEBUG:teuthology.orchestra.run.smithi135:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4@osd.4.service' 2024-03-30T05:16:29.896 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T05:16:29.896 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-30T05:16:29.896 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-30T05:16:29.897 DEBUG:teuthology.orchestra.run.smithi135:> sudo systemctl stop ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4@osd.5 2024-03-30T05:16:30.093 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 05:16:29 smithi135 systemd[1]: Stopping Ceph osd.5 for 2e77a2be-ee4c-11ee-b647-cb9ed24678a4... 2024-03-30T05:16:30.350 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 05:16:30 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-5[131567]: 2024-03-30T05:16:30.094+0000 7fadcf800700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-30T05:15:52.642193+0000 front 2024-03-30T05:15:52.642174+0000 (oldest deadline 2024-03-30T05:16:14.942168+0000) 2024-03-30T05:16:30.350 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 05:16:30 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-5[131567]: 2024-03-30T05:16:30.094+0000 7fadcf800700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.123:6814 osd.1 since back 2024-03-30T05:16:00.743668+0000 front 2024-03-30T05:16:00.743686+0000 (oldest deadline 2024-03-30T05:16:23.043601+0000) 2024-03-30T05:16:30.350 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 05:16:30 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-5[131567]: 2024-03-30T05:16:30.266+0000 7fadd3600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T05:16:30.350 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 05:16:30 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-5[131567]: 2024-03-30T05:16:30.266+0000 7fadd3600700 -1 osd.5 86 *** Got signal Terminated *** 2024-03-30T05:16:30.350 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 05:16:30 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-5[131567]: 2024-03-30T05:16:30.266+0000 7fadd3600700 -1 osd.5 86 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T05:16:31.600 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 05:16:31 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-5[131567]: 2024-03-30T05:16:31.107+0000 7fadcf800700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-30T05:15:52.642193+0000 front 2024-03-30T05:15:52.642174+0000 (oldest deadline 2024-03-30T05:16:14.942168+0000) 2024-03-30T05:16:31.600 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 05:16:31 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-5[131567]: 2024-03-30T05:16:31.107+0000 7fadcf800700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.123:6814 osd.1 since back 2024-03-30T05:16:00.743668+0000 front 2024-03-30T05:16:00.743686+0000 (oldest deadline 2024-03-30T05:16:23.043601+0000) 2024-03-30T05:16:32.600 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 05:16:32 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-5[131567]: 2024-03-30T05:16:32.136+0000 7fadcf800700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-30T05:15:52.642193+0000 front 2024-03-30T05:15:52.642174+0000 (oldest deadline 2024-03-30T05:16:14.942168+0000) 2024-03-30T05:16:32.600 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 05:16:32 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-5[131567]: 2024-03-30T05:16:32.136+0000 7fadcf800700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.123:6814 osd.1 since back 2024-03-30T05:16:00.743668+0000 front 2024-03-30T05:16:00.743686+0000 (oldest deadline 2024-03-30T05:16:23.043601+0000) 2024-03-30T05:16:33.600 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 05:16:33 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-5[131567]: 2024-03-30T05:16:33.171+0000 7fadcf800700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-30T05:15:52.642193+0000 front 2024-03-30T05:15:52.642174+0000 (oldest deadline 2024-03-30T05:16:14.942168+0000) 2024-03-30T05:16:33.600 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 05:16:33 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-5[131567]: 2024-03-30T05:16:33.171+0000 7fadcf800700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.123:6814 osd.1 since back 2024-03-30T05:16:00.743668+0000 front 2024-03-30T05:16:00.743686+0000 (oldest deadline 2024-03-30T05:16:23.043601+0000) 2024-03-30T05:16:33.600 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 05:16:33 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-5[131567]: 2024-03-30T05:16:33.171+0000 7fadcf800700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.123:6822 osd.2 since back 2024-03-30T05:16:07.144902+0000 front 2024-03-30T05:16:07.144838+0000 (oldest deadline 2024-03-30T05:16:32.444784+0000) 2024-03-30T05:16:34.600 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 05:16:34 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-5[131567]: 2024-03-30T05:16:34.201+0000 7fadcf800700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-30T05:15:52.642193+0000 front 2024-03-30T05:15:52.642174+0000 (oldest deadline 2024-03-30T05:16:14.942168+0000) 2024-03-30T05:16:34.600 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 05:16:34 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-5[131567]: 2024-03-30T05:16:34.201+0000 7fadcf800700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.123:6814 osd.1 since back 2024-03-30T05:16:00.743668+0000 front 2024-03-30T05:16:00.743686+0000 (oldest deadline 2024-03-30T05:16:23.043601+0000) 2024-03-30T05:16:34.600 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 05:16:34 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-5[131567]: 2024-03-30T05:16:34.201+0000 7fadcf800700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.123:6822 osd.2 since back 2024-03-30T05:16:07.144902+0000 front 2024-03-30T05:16:07.144838+0000 (oldest deadline 2024-03-30T05:16:32.444784+0000) 2024-03-30T05:16:35.600 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 05:16:35 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-5[131567]: 2024-03-30T05:16:35.157+0000 7fadcf800700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.123:6806 osd.0 since back 2024-03-30T05:15:52.642193+0000 front 2024-03-30T05:15:52.642174+0000 (oldest deadline 2024-03-30T05:16:14.942168+0000) 2024-03-30T05:16:35.601 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 05:16:35 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-5[131567]: 2024-03-30T05:16:35.157+0000 7fadcf800700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.123:6814 osd.1 since back 2024-03-30T05:16:00.743668+0000 front 2024-03-30T05:16:00.743686+0000 (oldest deadline 2024-03-30T05:16:23.043601+0000) 2024-03-30T05:16:35.601 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 05:16:35 smithi135 ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-5[131567]: 2024-03-30T05:16:35.157+0000 7fadcf800700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.123:6822 osd.2 since back 2024-03-30T05:16:07.144902+0000 front 2024-03-30T05:16:07.144838+0000 (oldest deadline 2024-03-30T05:16:32.444784+0000) 2024-03-30T05:16:36.350 INFO:journalctl@ceph.osd.5.smithi135.stdout:Mar 30 05:16:36 smithi135 bash[153854]: ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4-osd-5 2024-03-30T05:16:38.264 DEBUG:teuthology.orchestra.run.smithi135:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2e77a2be-ee4c-11ee-b647-cb9ed24678a4@osd.5.service' 2024-03-30T05:16:38.350 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T05:16:38.350 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-30T05:16:38.350 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 --force --keep-logs 2024-03-30T05:16:42.197 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi123.stderr:ceph-fuse[94945]: fuse finished with error 0 and tester_r 0 2024-03-30T05:17:00.345 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 --force --keep-logs 2024-03-30T05:17:15.195 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-30T05:17:15.273 DEBUG:teuthology.orchestra.run.smithi135:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-30T05:17:15.344 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-30T05:17:15.346 DEBUG:teuthology.misc:Transferring archived files from smithi123:/var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash to /home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629473/remote/smithi123/crash 2024-03-30T05:17:15.347 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar c -f - -C /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash -- . 2024-03-30T05:17:15.414 INFO:teuthology.orchestra.run.smithi123.stderr:tar: /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash: Cannot open: No such file or directory 2024-03-30T05:17:15.414 INFO:teuthology.orchestra.run.smithi123.stderr:tar: Error is not recoverable: exiting now 2024-03-30T05:17:15.417 DEBUG:teuthology.misc:Transferring archived files from smithi135:/var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash to /home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629473/remote/smithi135/crash 2024-03-30T05:17:15.418 DEBUG:teuthology.orchestra.run.smithi135:> sudo tar c -f - -C /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash -- . 2024-03-30T05:17:15.482 INFO:teuthology.orchestra.run.smithi135.stderr:tar: /var/lib/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/crash: Cannot open: No such file or directory 2024-03-30T05:17:15.482 INFO:teuthology.orchestra.run.smithi135.stderr:tar: Error is not recoverable: exiting now 2024-03-30T05:17:15.485 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-30T05:17:15.485 DEBUG:teuthology.orchestra.run.smithi123:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-03-30T05:17:15.561 INFO:teuthology.orchestra.run.smithi123.stderr:grep: /var/log/ceph/2e77a2be-ee4c-11ee-b647-cb9ed24678a4/ceph.log: No such file or directory 2024-03-30T05:17:15.566 INFO:tasks.cephadm:Compressing logs... 2024-03-30T05:17:15.566 DEBUG:teuthology.orchestra.run.smithi123:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-30T05:17:15.570 DEBUG:teuthology.orchestra.run.smithi135:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-30T05:17:15.631 INFO:teuthology.orchestra.run.smithi123.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-30T05:17:15.635 INFO:teuthology.orchestra.run.smithi135.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-30T05:19:04.035 INFO:teuthology.orchestra.run.smithi135.stderr:gzip: /var/log/ceph/ceph-client.1.log: file size changed while zipping 2024-03-30T05:19:04.039 INFO:tasks.cephadm:Archiving logs... 2024-03-30T05:19:04.039 DEBUG:teuthology.misc:Transferring archived files from smithi123:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629473/remote/smithi123/log 2024-03-30T05:19:04.040 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-30T05:19:15.709 DEBUG:teuthology.misc:Transferring archived files from smithi135:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629473/remote/smithi135/log 2024-03-30T05:19:15.710 DEBUG:teuthology.orchestra.run.smithi135:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-30T05:19:28.046 INFO:tasks.cephadm:Removing cluster... 2024-03-30T05:19:28.046 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 --force 2024-03-30T05:19:29.005 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2e77a2be-ee4c-11ee-b647-cb9ed24678a4 --force 2024-03-30T05:19:29.948 INFO:tasks.cephadm:Removing cephadm ... 2024-03-30T05:19:29.949 DEBUG:teuthology.orchestra.run.smithi123:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-30T05:19:29.999 DEBUG:teuthology.orchestra.run.smithi135:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-30T05:19:30.043 INFO:tasks.cephadm:Teardown complete 2024-03-30T05:19:30.043 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-30T05:19:30.055 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_470100a4ccbcf7769052bf09a5413881ccf1b97f/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/git.ceph.com_ceph-c_470100a4ccbcf7769052bf09a5413881ccf1b97f/qa/tasks/cephfs/fuse_mount.py", line 400, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-03-30T05:19:30.056 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-30T05:19:30.056 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-30T05:19:30.060 DEBUG:teuthology.orchestra.run.smithi135:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-30T05:19:30.217 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-30T05:19:30.228 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-30T05:19:30.228 DEBUG:teuthology.orchestra.run.smithi123:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-30T05:19:30.232 DEBUG:teuthology.orchestra.run.smithi135:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-30T05:19:30.271 INFO:teuthology.orchestra.run.smithi123.stderr:bash: ntpq: command not found 2024-03-30T05:19:30.275 INFO:teuthology.orchestra.run.smithi135.stderr:bash: ntpq: command not found 2024-03-30T05:19:30.279 INFO:teuthology.orchestra.run.smithi123.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-30T05:19:30.279 INFO:teuthology.orchestra.run.smithi123.stdout:=============================================================================== 2024-03-30T05:19:30.279 INFO:teuthology.orchestra.run.smithi123.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 49 +435us[ +448us] +/- 86ms 2024-03-30T05:19:30.279 INFO:teuthology.orchestra.run.smithi123.stdout:^* hv02.front.sepia.ceph.com 3 6 377 45 -920us[ -906us] +/- 66ms 2024-03-30T05:19:30.279 INFO:teuthology.orchestra.run.smithi123.stdout:^+ hv03.front.sepia.ceph.com 3 8 377 48 +622us[ +636us] +/- 79ms 2024-03-30T05:19:30.280 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T05:19:30.282 INFO:teuthology.orchestra.run.smithi135.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-30T05:19:30.282 INFO:teuthology.orchestra.run.smithi135.stdout:=============================================================================== 2024-03-30T05:19:30.282 INFO:teuthology.orchestra.run.smithi135.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 48 +491us[ +496us] +/- 86ms 2024-03-30T05:19:30.282 INFO:teuthology.orchestra.run.smithi135.stdout:^* hv02.front.sepia.ceph.com 3 7 377 42 -912us[ -907us] +/- 66ms 2024-03-30T05:19:30.282 INFO:teuthology.orchestra.run.smithi135.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 48 +653us[ +659us] +/- 79ms 2024-03-30T05:19:30.283 INFO:teuthology.orchestra.run.smithi135.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T05:19:30.283 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-30T05:19:30.294 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-30T05:19:30.295 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-30T05:19:30.306 DEBUG:teuthology.orchestra.run.smithi123:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-30T05:19:30.458 DEBUG:teuthology.orchestra.run.smithi135:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-30T05:19:30.591 DEBUG:teuthology.orchestra.run.smithi123:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-30T05:19:30.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T05:19:30.663 DEBUG:teuthology.orchestra.run.smithi135:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-30T05:19:30.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T05:19:30.734 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-30T05:19:30.746 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-30T05:19:30.762 INFO:teuthology.task.internal:Duration was 4495.593842 seconds 2024-03-30T05:19:30.762 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-30T05:19:30.773 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-30T05:19:30.773 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-30T05:19:30.778 DEBUG:teuthology.orchestra.run.smithi135:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-30T05:19:30.885 INFO:teuthology.orchestra.run.smithi123.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-30T05:19:30.889 INFO:teuthology.orchestra.run.smithi135.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-30T05:19:31.133 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-30T05:19:31.133 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi123.front.sepia.ceph.com 2024-03-30T05:19:31.133 DEBUG:teuthology.orchestra.run.smithi123:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-30T05:19:31.192 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi135.front.sepia.ceph.com 2024-03-30T05:19:31.192 DEBUG:teuthology.orchestra.run.smithi135:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-30T05:19:31.251 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-30T05:19:31.251 DEBUG:teuthology.orchestra.run.smithi123:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-30T05:19:31.255 DEBUG:teuthology.orchestra.run.smithi135:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-30T05:19:31.364 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-30T05:19:31.597 DEBUG:teuthology.orchestra.run.smithi123:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-30T05:19:31.602 DEBUG:teuthology.orchestra.run.smithi135:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-30T05:19:31.797 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-30T05:19:31.913 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-30T05:19:31.914 DEBUG:teuthology.orchestra.run.smithi123:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-30T05:19:31.981 DEBUG:teuthology.orchestra.run.smithi135:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-30T05:19:32.048 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-30T05:19:32.061 DEBUG:teuthology.orchestra.run.smithi123:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-30T05:19:32.065 DEBUG:teuthology.orchestra.run.smithi135:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-30T05:19:32.127 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern = core 2024-03-30T05:19:32.131 INFO:teuthology.orchestra.run.smithi135.stdout:kernel.core_pattern = core 2024-03-30T05:19:32.175 DEBUG:teuthology.orchestra.run.smithi123:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-30T05:19:32.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T05:19:32.218 DEBUG:teuthology.orchestra.run.smithi135:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-30T05:19:32.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T05:19:32.261 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-30T05:19:32.273 INFO:teuthology.task.internal:Transferring archived files... 2024-03-30T05:19:32.275 DEBUG:teuthology.misc:Transferring archived files from smithi123:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629473/remote/smithi123 2024-03-30T05:19:32.275 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-30T05:19:32.362 DEBUG:teuthology.misc:Transferring archived files from smithi135:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629473/remote/smithi135 2024-03-30T05:19:32.362 DEBUG:teuthology.orchestra.run.smithi135:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-30T05:19:32.444 INFO:teuthology.task.internal:Removing archive directory... 2024-03-30T05:19:32.445 DEBUG:teuthology.orchestra.run.smithi123:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-30T05:19:32.449 DEBUG:teuthology.orchestra.run.smithi135:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-30T05:19:32.493 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-30T05:19:32.507 INFO:teuthology.task.internal:Not uploading archives. 2024-03-30T05:19:32.507 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-30T05:19:32.518 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-30T05:19:32.519 DEBUG:teuthology.orchestra.run.smithi123:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-30T05:19:32.534 DEBUG:teuthology.orchestra.run.smithi135:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-30T05:19:32.574 INFO:teuthology.orchestra.run.smithi123.stdout: 524301 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 30 05:19 /home/ubuntu/cephtest 2024-03-30T05:19:32.575 INFO:teuthology.orchestra.run.smithi123.stdout: 943200 4 d--------- 2 ubuntu ubuntu 4096 Mar 30 04:25 /home/ubuntu/cephtest/mnt.0 2024-03-30T05:19:32.575 INFO:teuthology.orchestra.run.smithi123.stderr:find: ‘/home/ubuntu/cephtest/mnt.0’: Permission denied 2024-03-30T05:19:32.576 INFO:teuthology.orchestra.run.smithi123.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-03-30T05:19:32.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T05:19:32.578 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_470100a4ccbcf7769052bf09a5413881ccf1b97f/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/git.ceph.com_ceph-c_470100a4ccbcf7769052bf09a5413881ccf1b97f/qa/tasks/cephfs/fuse_mount.py", line 400, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi123 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-03-30T05:19:32.578 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-30T05:19:32.591 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-30T05:19:32.605 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-03-30T05:19:32.605 INFO:teuthology.nuke:Checking targets against current locks 2024-03-30T05:19:32.644 DEBUG:teuthology.nuke:shortname: smithi123 2024-03-30T05:19:32.645 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-30T05:19:32.687 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi123.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629473', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-30 03:54:40.477689', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL84LCzYPCo3lePnhqemzPds+HKP3NxrBEFTSpPOsTCV3NO0gMHX/66ja2Ywr7NGE8XNXFLYQm8xOm1D+BY4hgs='} 2024-03-30T05:19:32.699 DEBUG:teuthology.nuke:shortname: smithi135 2024-03-30T05:19:32.699 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-30T05:19:32.739 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi135.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629473', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-30 03:54:40.479399', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCvGdlm5XODb+4nlZVWzc1Z3AkK8rNHpQllvV8Q7wuxG/Kk3+oIkPF97vCgUgdi2GPWH4MMiGsU+M9UZ74G+SB4='} 2024-03-30T05:19:32.783 INFO:teuthology.orchestra.console.smithi123:Power off 2024-03-30T05:19:32.783 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-30T05:19:32.810 INFO:teuthology.orchestra.console.smithi135:Power off 2024-03-30T05:19:32.811 DEBUG:teuthology.orchestra.console.smithi135:pexpect command: ipmitool -H smithi135.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-30T05:19:32.848 DEBUG:teuthology.orchestra.console.smithi123:power off output: Chassis Power Control: Down/Off 2024-03-30T05:19:32.849 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T05:19:32.863 DEBUG:teuthology.orchestra.console.smithi135:power off output: Chassis Power Control: Down/Off 2024-03-30T05:19:32.863 DEBUG:teuthology.orchestra.console.smithi135:pexpect command: ipmitool -H smithi135.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T05:19:32.915 DEBUG:teuthology.orchestra.console.smithi123:check power output: Chassis Power is on 2024-03-30T05:19:32.932 DEBUG:teuthology.orchestra.console.smithi135:check power output: Chassis Power is on 2024-03-30T05:19:36.916 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T05:19:36.932 DEBUG:teuthology.orchestra.console.smithi135:pexpect command: ipmitool -H smithi135.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T05:19:37.029 DEBUG:teuthology.orchestra.console.smithi123:check power output: Chassis Power is on 2024-03-30T05:19:37.044 DEBUG:teuthology.orchestra.console.smithi135:check power output: Chassis Power is on 2024-03-30T05:19:41.030 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T05:19:41.045 DEBUG:teuthology.orchestra.console.smithi135:pexpect command: ipmitool -H smithi135.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T05:19:41.142 DEBUG:teuthology.orchestra.console.smithi123:check power output: Chassis Power is on 2024-03-30T05:19:41.155 DEBUG:teuthology.orchestra.console.smithi135:check power output: Chassis Power is on 2024-03-30T05:19:45.146 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T05:19:45.156 DEBUG:teuthology.orchestra.console.smithi135:pexpect command: ipmitool -H smithi135.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T05:19:45.256 DEBUG:teuthology.orchestra.console.smithi123:check power output: Chassis Power is off 2024-03-30T05:19:45.264 DEBUG:teuthology.orchestra.console.smithi135:check power output: Chassis Power is off 2024-03-30T05:19:45.358 INFO:teuthology.orchestra.console.smithi123:Power off completed 2024-03-30T05:19:45.365 INFO:teuthology.orchestra.console.smithi135:Power off completed 2024-03-30T05:19:45.470 INFO:teuthology.run:Summary data: description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 4495.593841791153 failure_reason: reached maximum tries (51) after waiting for 300 seconds flavor: default host.a-kernel-sha1: 1df0c7e8a2354c42d738e5454ba83ea7332deac2 host.b-kernel-sha1: 1df0c7e8a2354c42d738e5454ba83ea7332deac2 owner: scheduled_rishabh@teuthology status: fail success: false 2024-03-30T05:19:45.470 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-30T05:19:45.564 INFO:teuthology.run:FAIL